2024-04-25T01:14:30.487 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T01:14:30.487 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} --verbose --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672318 --name teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672318/orig.config.yaml 2024-04-25T01:14:30.551 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T01:14:30.646 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672318 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7672318' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8528 sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 sleep_before_teardown: 0 subset: 122/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJWrI1AU39Ehj+BPwjpra1E9YULUzTCooYs+kB1+xCfgP6TVzlhSp7f7vNLD6OONrnhgKdlg6YDNqieCCjMM/uk= smithi032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJo598f+DFL12M7JLBu+5RWkNkJDbaAe24CITX+6UcqLPsY1Btmt2mVg2KgAsVbYxO1Ol6W+6K2MpacokIehziI= smithi130.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA77Z/E59dQe95txHq0s2gt1KnvpwydFgDYbS3FaliTlfdWbDXBjLbi7wTkeU5lYdWvCLQa9jXQBIqryTCcoXPs= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/iozone.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T01:14:30.646 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa; will attempt to use it 2024-04-25T01:14:30.647 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks 2024-04-25T01:14:30.647 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T01:14:30.649 INFO:teuthology.task.internal:Checking packages... 2024-04-25T01:14:30.680 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd2c45d977142e024a59b68a8f1dce8f0b9840551' 2024-04-25T01:14:30.680 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T01:14:30.680 INFO:teuthology.packaging:ref: None 2024-04-25T01:14:30.680 INFO:teuthology.packaging:tag: None 2024-04-25T01:14:30.680 INFO:teuthology.packaging:branch: squid 2024-04-25T01:14:30.681 INFO:teuthology.packaging:sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:14:30.681 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-25T01:14:30.901 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2445.gd2c45d97 2024-04-25T01:14:30.903 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T01:14:30.910 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T01:14:30.910 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T01:14:30.950 INFO:teuthology.task.internal:Saving configuration 2024-04-25T01:14:30.975 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T01:14:30.983 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T01:14:31.007 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672318', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:09:45.404126', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJWrI1AU39Ehj+BPwjpra1E9YULUzTCooYs+kB1+xCfgP6TVzlhSp7f7vNLD6OONrnhgKdlg6YDNqieCCjMM/uk='} 2024-04-25T01:14:31.028 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672318', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:09:45.405591', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJo598f+DFL12M7JLBu+5RWkNkJDbaAe24CITX+6UcqLPsY1Btmt2mVg2KgAsVbYxO1Ol6W+6K2MpacokIehziI='} 2024-04-25T01:14:31.049 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672318', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:09:45.406837', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA77Z/E59dQe95txHq0s2gt1KnvpwydFgDYbS3FaliTlfdWbDXBjLbi7wTkeU5lYdWvCLQa9jXQBIqryTCcoXPs='} 2024-04-25T01:14:31.049 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T01:14:31.055 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-25T01:14:31.055 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-25T01:14:31.055 INFO:teuthology.task.internal:roles: ubuntu@smithi130.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-25T01:14:31.055 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T01:14:31.177 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f85b3a27160>, signals=[15]) 2024-04-25T01:14:31.177 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T01:14:31.184 INFO:teuthology.task.internal:Opening connections... 2024-04-25T01:14:31.184 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T01:14:31.186 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:14:31.272 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2024-04-25T01:14:31.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:14:31.352 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi130.front.sepia.ceph.com 2024-04-25T01:14:31.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:14:31.425 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T01:14:31.432 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-04-25T01:14:31.450 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-04-25T01:14:31.451 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-04-25T01:14:31.509 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-04-25T01:14:31.509 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-04-25T01:14:31.509 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-04-25T01:14:31.509 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-04-25T01:14:31.509 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-04-25T01:14:31.509 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-04-25T01:14:31.509 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T01:14:31.510 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-04-25T01:14:31.510 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-04-25T01:14:31.510 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T01:14:31.510 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-04-25T01:14:31.510 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T01:14:31.510 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T01:14:31.510 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T01:14:31.511 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-04-25T01:14:31.535 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2024-04-25T01:14:31.553 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2024-04-25T01:14:31.554 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2024-04-25T01:14:31.612 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="CentOS Stream" 2024-04-25T01:14:31.612 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="9" 2024-04-25T01:14:31.612 INFO:teuthology.orchestra.run.smithi032.stdout:ID="centos" 2024-04-25T01:14:31.612 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="rhel fedora" 2024-04-25T01:14:31.612 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="9" 2024-04-25T01:14:31.612 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el9" 2024-04-25T01:14:31.612 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T01:14:31.613 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2024-04-25T01:14:31.613 INFO:teuthology.orchestra.run.smithi032.stdout:LOGO="fedora-logo-icon" 2024-04-25T01:14:31.613 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T01:14:31.613 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://centos.org/" 2024-04-25T01:14:31.613 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T01:14:31.613 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T01:14:31.613 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T01:14:31.613 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2024-04-25T01:14:31.635 DEBUG:teuthology.orchestra.run.smithi130:> uname -m 2024-04-25T01:14:31.653 INFO:teuthology.orchestra.run.smithi130.stdout:x86_64 2024-04-25T01:14:31.653 DEBUG:teuthology.orchestra.run.smithi130:> cat /etc/os-release 2024-04-25T01:14:31.710 INFO:teuthology.orchestra.run.smithi130.stdout:NAME="CentOS Stream" 2024-04-25T01:14:31.711 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION="9" 2024-04-25T01:14:31.711 INFO:teuthology.orchestra.run.smithi130.stdout:ID="centos" 2024-04-25T01:14:31.711 INFO:teuthology.orchestra.run.smithi130.stdout:ID_LIKE="rhel fedora" 2024-04-25T01:14:31.711 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION_ID="9" 2024-04-25T01:14:31.711 INFO:teuthology.orchestra.run.smithi130.stdout:PLATFORM_ID="platform:el9" 2024-04-25T01:14:31.711 INFO:teuthology.orchestra.run.smithi130.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T01:14:31.711 INFO:teuthology.orchestra.run.smithi130.stdout:ANSI_COLOR="0;31" 2024-04-25T01:14:31.711 INFO:teuthology.orchestra.run.smithi130.stdout:LOGO="fedora-logo-icon" 2024-04-25T01:14:31.711 INFO:teuthology.orchestra.run.smithi130.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T01:14:31.711 INFO:teuthology.orchestra.run.smithi130.stdout:HOME_URL="https://centos.org/" 2024-04-25T01:14:31.711 INFO:teuthology.orchestra.run.smithi130.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T01:14:31.711 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T01:14:31.711 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T01:14:31.712 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server 2024-04-25T01:14:31.734 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T01:14:31.749 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T01:14:31.818 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T01:14:31.818 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-04-25T01:14:31.821 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2024-04-25T01:14:31.824 DEBUG:teuthology.orchestra.run.smithi130:> test '!' -e /home/ubuntu/cephtest 2024-04-25T01:14:31.840 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T01:14:31.846 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T01:14:31.847 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-04-25T01:14:31.878 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2024-04-25T01:14:31.881 DEBUG:teuthology.orchestra.run.smithi130:> test -z $(ls -A /var/lib/ceph) 2024-04-25T01:14:31.964 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T01:14:32.043 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T01:14:32.057 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-04-25T01:14:32.057 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-25T01:14:32.057 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-25T01:14:32.058 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T01:14:32.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:14:32.076 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-04-25T01:14:32.134 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-437.el9.x86_64 2024-04-25T01:14:32.134 INFO:teuthology.task.kernel:Running kernel on smithi028: 5.14.0-437.el9.x86_64 2024-04-25T01:14:32.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum install -y kernel 2024-04-25T01:14:35.523 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.0 MB 00:01 2024-04-25T01:14:37.624 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-04-25T01:14:41.941 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-25T01:14:43.417 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-04-25T01:14:45.271 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-25T01:14:50.476 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-04-25T01:14:52.433 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T01:14:52.433 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T01:14:52.525 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-25T01:14:52.526 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-04-25T01:14:52.526 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-25T01:14:52.655 DEBUG:teuthology.orchestra.run.smithi028:> echo no | sudo yum reinstall kernel || true 2024-04-25T01:14:53.178 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 01:14:50 AM UTC. 2024-04-25T01:14:53.306 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-25T01:14:53.315 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-25T01:14:53.315 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-04-25T01:14:53.315 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-25T01:14:53.315 INFO:teuthology.orchestra.run.smithi028.stdout:Reinstalling: 2024-04-25T01:14:53.315 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:14:53.315 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:14:53.315 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-25T01:14:53.315 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-25T01:14:53.315 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:14:53.315 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 255 k 2024-04-25T01:14:53.315 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 0 2024-04-25T01:14:53.316 INFO:teuthology.orchestra.run.smithi028.stderr:Operation aborted. 2024-04-25T01:14:53.375 INFO:teuthology.orchestra.run.smithi028.stdout:Is this ok [y/N]: 2024-04-25T01:14:53.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum reinstall -y kernel || true 2024-04-25T01:14:53.890 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 01:14:50 AM UTC. 2024-04-25T01:14:54.010 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-25T01:14:54.011 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-25T01:14:54.011 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-04-25T01:14:54.011 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-25T01:14:54.011 INFO:teuthology.orchestra.run.smithi028.stdout:Reinstalling: 2024-04-25T01:14:54.012 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T01:14:54.012 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:14:54.012 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-25T01:14:54.012 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-25T01:14:54.012 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:14:54.012 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 255 k 2024-04-25T01:14:54.012 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 0 2024-04-25T01:14:54.012 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-25T01:14:54.995 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-437.el9.x86_64.rpm 435 kB/s | 255 kB 00:00 2024-04-25T01:14:54.995 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:14:54.995 INFO:teuthology.orchestra.run.smithi028.stdout:Total 259 kB/s | 255 kB 00:00 2024-04-25T01:14:55.012 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-25T01:14:55.199 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-25T01:14:55.200 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-25T01:14:55.205 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-25T01:14:55.206 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-25T01:14:55.327 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-25T01:14:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:14:55.396 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:14:56.040 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:14:56.040 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T01:14:56.726 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T01:14:56.726 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:14:56.726 INFO:teuthology.orchestra.run.smithi028.stdout:Reinstalled: 2024-04-25T01:14:56.727 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T01:14:56.727 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:14:56.727 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-25T01:14:56.892 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T01:14:56.940 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T01:14:56.940 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T01:14:56.940 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T01:14:56.940 DEBUG:teuthology.parallel:result is None 2024-04-25T01:14:56.941 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T01:14:56.948 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T01:14:56.948 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T01:14:56.951 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T01:14:56.953 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T01:14:56.971 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T01:14:57.057 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T01:14:57.188 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T01:14:57.190 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T01:14:57.200 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T01:14:57.207 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T01:14:57.321 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T01:14:57.333 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T01:14:57.333 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T01:14:57.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:14:57.353 DEBUG:teuthology.orchestra.run.smithi130:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T01:14:57.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:14:57.372 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T01:14:57.377 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T01:14:57.397 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T01:14:57.410 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:14:57.435 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:14:57.440 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:14:57.452 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:14:57.457 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:14:57.497 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:14:57.499 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T01:14:57.506 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T01:14:57.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T01:14:57.509 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T01:14:57.512 DEBUG:teuthology.orchestra.run.smithi130:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T01:14:57.566 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T01:14:57.575 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T01:14:57.575 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T01:14:57.577 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T01:14:57.580 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T01:14:57.622 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:14:57.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:14:57.726 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:14:57.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:14:57.847 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:14:57.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T01:14:57.912 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:14:57.958 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:14:58.004 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:14:58.093 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:14:58.134 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-25T01:14:58.134 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T01:14:58.202 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:14:58.248 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:14:58.297 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:14:58.380 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:14:58.422 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-25T01:14:58.423 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T01:14:58.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-04-25T01:14:58.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2024-04-25T01:14:58.494 DEBUG:teuthology.orchestra.run.smithi130:> sudo service rsyslog restart 2024-04-25T01:14:58.527 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:14:58.533 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:14:58.573 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:14:59.028 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T01:14:59.036 INFO:teuthology.task.internal:Starting timer... 2024-04-25T01:14:59.036 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T01:14:59.101 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T01:14:59.218 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-25T01:14:59.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-04-25T01:14:59.278 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-04-25T01:14:59.280 DEBUG:teuthology.orchestra.run.smithi032:> sudo service auditd rotate 2024-04-25T01:14:59.356 INFO:teuthology.orchestra.run.smithi032.stdout:Rotating logs: 2024-04-25T01:14:59.358 DEBUG:teuthology.orchestra.run.smithi130:> sudo service auditd rotate 2024-04-25T01:14:59.427 INFO:teuthology.orchestra.run.smithi130.stdout:Rotating logs: 2024-04-25T01:14:59.429 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T01:14:59.429 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-04-25T01:14:59.454 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-04-25T01:14:59.455 DEBUG:teuthology.orchestra.run.smithi032:> /usr/sbin/getenforce 2024-04-25T01:14:59.481 INFO:teuthology.orchestra.run.smithi032.stdout:Permissive 2024-04-25T01:14:59.481 DEBUG:teuthology.orchestra.run.smithi130:> /usr/sbin/getenforce 2024-04-25T01:14:59.520 INFO:teuthology.orchestra.run.smithi130.stdout:Permissive 2024-04-25T01:14:59.520 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi032.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi130.front.sepia.ceph.com': 'permissive'} 2024-04-25T01:14:59.521 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:14:59.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:14:59.551 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:14:59.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:14:59.580 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:14:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:14:59.608 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T01:14:59.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-04-25T01:14:59.634 DEBUG:teuthology.orchestra.run.smithi032:> sudo /usr/sbin/setenforce permissive 2024-04-25T01:14:59.660 DEBUG:teuthology.orchestra.run.smithi130:> sudo /usr/sbin/setenforce permissive 2024-04-25T01:14:59.686 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T01:14:59.695 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T01:14:59.713 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T01:14:59.714 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi032.front.sepia.ceph.com,smithi130.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T01:18:53.957 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi032.front.sepia.ceph.com'), Remote(name='ubuntu@smithi130.front.sepia.ceph.com')] 2024-04-25T01:18:53.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-25T01:18:53.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:18:54.038 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-04-25T01:18:54.119 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-25T01:18:54.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-25T01:18:54.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:18:54.197 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-04-25T01:18:54.278 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-25T01:18:54.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-04-25T01:18:54.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:18:54.353 DEBUG:teuthology.orchestra.run.smithi130:> true 2024-04-25T01:18:54.438 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-04-25T01:18:54.439 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T01:18:54.488 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T01:18:54.489 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T01:18:54.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:18:54.492 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T01:18:54.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:18:54.494 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T01:18:54.494 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:18:54.523 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T01:18:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T01:18:54.524 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T01:18:54.537 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T01:18:54.539 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T01:18:54.539 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T01:18:54.563 INFO:teuthology.orchestra.run.smithi130.stderr:sudo: ntpd: command not found 2024-04-25T01:18:54.563 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: ntpd: command not found 2024-04-25T01:18:54.564 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-04-25T01:18:54.573 INFO:teuthology.orchestra.run.smithi130.stdout:506 Cannot talk to daemon 2024-04-25T01:18:54.574 INFO:teuthology.orchestra.run.smithi032.stdout:506 Cannot talk to daemon 2024-04-25T01:18:54.576 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-04-25T01:18:54.587 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T01:18:54.588 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T01:18:54.590 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T01:18:54.600 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T01:18:54.602 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T01:18:54.605 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T01:18:54.652 INFO:teuthology.orchestra.run.smithi130.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:18:54.657 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:18:54.657 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2024-04-25T01:18:54.657 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:54.657 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:54.657 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:54.657 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:54.665 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:18:54.668 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:18:54.668 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-04-25T01:18:54.669 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:54.669 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:54.669 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:54.669 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:18:54.677 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:18:54.677 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-04-25T01:18:54.677 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:54.677 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:54.677 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:54.677 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:18:54.678 INFO:teuthology.run_tasks:Running task install... 2024-04-25T01:18:54.703 DEBUG:teuthology.task.install:project ceph 2024-04-25T01:18:54.704 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}} 2024-04-25T01:18:54.704 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'} 2024-04-25T01:18:54.704 INFO:teuthology.task.install:Using flavor: default 2024-04-25T01:18:54.716 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T01:18:54.716 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T01:18:54.716 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T01:18:54.717 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:18:54.720 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T01:18:54.720 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:18:54.722 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T01:18:54.722 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:18:54.886 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T01:18:54.886 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T01:18:55.016 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T01:18:55.017 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T01:18:55.022 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T01:18:55.023 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T01:18:55.098 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T01:18:55.098 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-25T01:18:55.098 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T01:18:55.123 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T01:18:55.124 DEBUG:teuthology.orchestra.run.smithi130:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T01:18:55.190 DEBUG:teuthology.orchestra.run.smithi130:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T01:18:55.212 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T01:18:55.212 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:18:55.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T01:18:55.215 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T01:18:55.216 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-25T01:18:55.216 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T01:18:55.240 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T01:18:55.240 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T01:18:55.243 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T01:18:55.243 DEBUG:teuthology.orchestra.run.smithi032:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T01:18:55.262 DEBUG:teuthology.orchestra.run.smithi130:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T01:18:55.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T01:18:55.311 DEBUG:teuthology.orchestra.run.smithi032:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T01:18:55.325 INFO:teuthology.orchestra.run.smithi130.stdout:check_obsoletes = 1 2024-04-25T01:18:55.327 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum clean all 2024-04-25T01:18:55.383 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T01:18:55.387 DEBUG:teuthology.orchestra.run.smithi032:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T01:18:55.448 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-04-25T01:18:55.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-04-25T01:18:55.452 INFO:teuthology.orchestra.run.smithi032.stdout:check_obsoletes = 1 2024-04-25T01:18:55.454 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum clean all 2024-04-25T01:18:55.551 INFO:teuthology.orchestra.run.smithi130.stdout:48 files removed 2024-04-25T01:18:55.579 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T01:18:55.685 INFO:teuthology.orchestra.run.smithi028.stdout:48 files removed 2024-04-25T01:18:55.689 INFO:teuthology.orchestra.run.smithi032.stdout:48 files removed 2024-04-25T01:18:55.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T01:18:55.716 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T01:18:56.258 INFO:teuthology.orchestra.run.smithi130.stdout:ceph packages for x86_64 199 kB/s | 83 kB 00:00 2024-04-25T01:18:56.528 INFO:teuthology.orchestra.run.smithi032.stdout:ceph packages for x86_64 155 kB/s | 83 kB 00:00 2024-04-25T01:18:56.533 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 154 kB/s | 83 kB 00:00 2024-04-25T01:18:56.626 INFO:teuthology.orchestra.run.smithi130.stdout:ceph noarch packages 35 kB/s | 11 kB 00:00 2024-04-25T01:18:56.992 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 28 kB/s | 11 kB 00:00 2024-04-25T01:18:57.051 INFO:teuthology.orchestra.run.smithi130.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-04-25T01:18:57.062 INFO:teuthology.orchestra.run.smithi032.stdout:ceph noarch packages 24 kB/s | 11 kB 00:00 2024-04-25T01:18:57.560 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 3.6 kB/s | 1.9 kB 00:00 2024-04-25T01:18:57.621 INFO:teuthology.orchestra.run.smithi032.stdout:ceph source packages 3.6 kB/s | 1.9 kB 00:00 2024-04-25T01:18:57.910 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - BaseOS 9.8 MB/s | 8.0 MB 00:00 2024-04-25T01:18:58.518 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 8.7 MB/s | 8.0 MB 00:00 2024-04-25T01:19:00.104 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 3.3 MB/s | 8.0 MB 00:02 2024-04-25T01:19:01.911 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 8.3 MB/s | 19 MB 00:02 2024-04-25T01:19:03.539 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 8.1 MB/s | 19 MB 00:02 2024-04-25T01:19:08.211 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 6.2 MB 00:02 2024-04-25T01:19:08.798 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.2 MB 00:01 2024-04-25T01:19:09.888 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-25T01:19:10.307 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 16 kB 00:00 2024-04-25T01:19:11.183 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-04-25T01:19:12.179 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-25T01:19:12.629 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 19 MB 00:13 2024-04-25T01:19:16.489 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-25T01:19:17.230 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - CRB 5.7 MB/s | 6.2 MB 00:01 2024-04-25T01:19:17.533 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-25T01:19:18.661 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-04-25T01:19:18.682 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:19:18.682 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:19:18.694 INFO:teuthology.orchestra.run.smithi028.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T01:19:18.694 INFO:teuthology.orchestra.run.smithi028.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T01:19:18.695 INFO:teuthology.orchestra.run.smithi028.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T01:19:18.696 INFO:teuthology.orchestra.run.smithi028.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T01:19:18.698 INFO:teuthology.orchestra.run.smithi028.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T01:19:18.700 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T01:19:18.701 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T01:19:18.765 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-25T01:19:18.775 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-25T01:19:18.775 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-04-25T01:19:18.775 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-25T01:19:18.775 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-25T01:19:18.775 INFO:teuthology.orchestra.run.smithi028.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T01:19:18.775 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T01:19:18.775 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T01:19:18.775 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T01:19:18.775 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T01:19:18.775 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T01:19:18.776 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T01:19:18.776 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T01:19:18.776 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T01:19:18.776 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T01:19:18.776 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T01:19:18.776 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T01:19:18.776 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T01:19:18.776 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T01:19:18.776 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T01:19:18.776 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T01:19:18.776 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T01:19:18.777 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T01:19:18.777 INFO:teuthology.orchestra.run.smithi028.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T01:19:18.777 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T01:19:18.777 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T01:19:18.777 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T01:19:18.777 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T01:19:18.777 INFO:teuthology.orchestra.run.smithi028.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T01:19:18.777 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T01:19:18.777 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T01:19:18.777 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T01:19:18.777 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T01:19:18.778 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T01:19:18.778 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T01:19:18.778 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T01:19:18.778 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T01:19:18.778 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T01:19:18.778 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T01:19:18.778 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:19:18.778 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T01:19:18.778 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T01:19:18.778 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-04-25T01:19:18.778 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T01:19:18.779 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:19:18.779 INFO:teuthology.orchestra.run.smithi028.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T01:19:18.779 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T01:19:18.779 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-04-25T01:19:18.779 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T01:19:18.779 INFO:teuthology.orchestra.run.smithi028.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T01:19:18.779 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T01:19:18.779 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T01:19:18.779 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T01:19:18.779 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T01:19:18.780 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T01:19:18.780 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T01:19:18.780 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T01:19:18.780 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T01:19:18.780 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T01:19:18.780 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T01:19:18.780 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T01:19:18.780 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T01:19:18.780 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T01:19:18.780 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T01:19:18.780 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T01:19:18.781 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T01:19:18.781 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T01:19:18.781 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T01:19:18.781 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T01:19:18.781 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T01:19:18.781 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T01:19:18.781 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T01:19:18.781 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T01:19:18.781 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T01:19:18.781 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T01:19:18.781 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T01:19:18.781 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T01:19:18.782 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T01:19:18.782 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T01:19:18.782 INFO:teuthology.orchestra.run.smithi028.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T01:19:18.782 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T01:19:18.782 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T01:19:18.782 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T01:19:18.782 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T01:19:18.782 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T01:19:18.782 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T01:19:18.782 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T01:19:18.782 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T01:19:18.783 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T01:19:18.783 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T01:19:18.783 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T01:19:18.783 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T01:19:18.783 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T01:19:18.783 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T01:19:18.783 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T01:19:18.783 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T01:19:18.783 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T01:19:18.783 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T01:19:18.783 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T01:19:18.783 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T01:19:18.784 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T01:19:18.784 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T01:19:18.784 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T01:19:18.784 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T01:19:18.784 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T01:19:18.784 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T01:19:18.784 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T01:19:18.784 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T01:19:18.784 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T01:19:18.784 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T01:19:18.784 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T01:19:18.784 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T01:19:18.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T01:19:18.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T01:19:18.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T01:19:18.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T01:19:18.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T01:19:18.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T01:19:18.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T01:19:18.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T01:19:18.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T01:19:18.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T01:19:18.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T01:19:18.785 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T01:19:18.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T01:19:18.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T01:19:18.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T01:19:18.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T01:19:18.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T01:19:18.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T01:19:18.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T01:19:18.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T01:19:18.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T01:19:18.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T01:19:18.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T01:19:18.786 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T01:19:18.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T01:19:18.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T01:19:18.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T01:19:18.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T01:19:18.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T01:19:18.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T01:19:18.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T01:19:18.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T01:19:18.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T01:19:18.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T01:19:18.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T01:19:18.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T01:19:18.787 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T01:19:18.788 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T01:19:18.788 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T01:19:18.788 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T01:19:18.788 INFO:teuthology.orchestra.run.smithi028.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T01:19:18.788 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T01:19:18.788 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T01:19:18.788 INFO:teuthology.orchestra.run.smithi028.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T01:19:18.788 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T01:19:18.788 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T01:19:18.788 INFO:teuthology.orchestra.run.smithi028.stdout:Installing weak dependencies: 2024-04-25T01:19:18.788 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T01:19:18.788 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:19:18.789 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-25T01:19:18.789 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-25T01:19:18.789 INFO:teuthology.orchestra.run.smithi028.stdout:Install 148 Packages 2024-04-25T01:19:18.789 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 4 Packages 2024-04-25T01:19:18.789 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:19:18.789 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 220 M 2024-04-25T01:19:18.790 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-25T01:19:19.736 INFO:teuthology.orchestra.run.smithi032.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:19:19.737 INFO:teuthology.orchestra.run.smithi032.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:19:19.749 INFO:teuthology.orchestra.run.smithi032.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T01:19:19.749 INFO:teuthology.orchestra.run.smithi032.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T01:19:19.750 INFO:teuthology.orchestra.run.smithi032.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T01:19:19.751 INFO:teuthology.orchestra.run.smithi032.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T01:19:19.753 INFO:teuthology.orchestra.run.smithi032.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T01:19:19.755 INFO:teuthology.orchestra.run.smithi032.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T01:19:19.756 INFO:teuthology.orchestra.run.smithi032.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T01:19:19.823 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-25T01:19:19.844 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================= 2024-04-25T01:19:19.844 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-25T01:19:19.844 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================= 2024-04-25T01:19:19.844 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-25T01:19:19.844 INFO:teuthology.orchestra.run.smithi032.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T01:19:19.844 INFO:teuthology.orchestra.run.smithi032.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T01:19:19.845 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T01:19:19.845 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T01:19:19.845 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T01:19:19.845 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T01:19:19.845 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T01:19:19.845 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T01:19:19.845 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T01:19:19.845 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T01:19:19.845 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T01:19:19.845 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T01:19:19.845 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T01:19:19.845 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T01:19:19.845 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T01:19:19.846 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T01:19:19.846 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T01:19:19.846 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T01:19:19.846 INFO:teuthology.orchestra.run.smithi032.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T01:19:19.846 INFO:teuthology.orchestra.run.smithi032.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T01:19:19.846 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T01:19:19.846 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T01:19:19.846 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T01:19:19.846 INFO:teuthology.orchestra.run.smithi032.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T01:19:19.846 INFO:teuthology.orchestra.run.smithi032.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T01:19:19.846 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T01:19:19.846 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T01:19:19.846 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T01:19:19.846 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T01:19:19.847 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T01:19:19.847 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T01:19:19.847 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T01:19:19.847 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T01:19:19.847 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T01:19:19.847 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:19:19.847 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T01:19:19.847 INFO:teuthology.orchestra.run.smithi032.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T01:19:19.847 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrading: 2024-04-25T01:19:19.848 INFO:teuthology.orchestra.run.smithi032.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T01:19:19.848 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:19:19.848 INFO:teuthology.orchestra.run.smithi032.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T01:19:19.848 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T01:19:19.848 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-25T01:19:19.848 INFO:teuthology.orchestra.run.smithi032.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T01:19:19.848 INFO:teuthology.orchestra.run.smithi032.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T01:19:19.848 INFO:teuthology.orchestra.run.smithi032.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T01:19:19.848 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T01:19:19.848 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T01:19:19.848 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T01:19:19.848 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T01:19:19.848 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T01:19:19.848 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T01:19:19.849 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T01:19:19.849 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T01:19:19.849 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T01:19:19.849 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T01:19:19.849 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T01:19:19.849 INFO:teuthology.orchestra.run.smithi032.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T01:19:19.849 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T01:19:19.849 INFO:teuthology.orchestra.run.smithi032.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T01:19:19.849 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T01:19:19.849 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T01:19:19.849 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T01:19:19.849 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T01:19:19.849 INFO:teuthology.orchestra.run.smithi032.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T01:19:19.849 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T01:19:19.849 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T01:19:19.850 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T01:19:19.850 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T01:19:19.850 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T01:19:19.850 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T01:19:19.850 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T01:19:19.850 INFO:teuthology.orchestra.run.smithi032.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T01:19:19.850 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T01:19:19.850 INFO:teuthology.orchestra.run.smithi032.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T01:19:19.850 INFO:teuthology.orchestra.run.smithi032.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T01:19:19.850 INFO:teuthology.orchestra.run.smithi032.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T01:19:19.850 INFO:teuthology.orchestra.run.smithi032.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T01:19:19.850 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T01:19:19.850 INFO:teuthology.orchestra.run.smithi032.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T01:19:19.851 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T01:19:19.851 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T01:19:19.851 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T01:19:19.851 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T01:19:19.851 INFO:teuthology.orchestra.run.smithi032.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T01:19:19.851 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T01:19:19.851 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T01:19:19.851 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T01:19:19.851 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T01:19:19.851 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T01:19:19.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T01:19:19.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T01:19:19.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T01:19:19.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T01:19:19.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T01:19:19.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T01:19:19.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T01:19:19.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T01:19:19.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T01:19:19.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T01:19:19.852 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T01:19:19.853 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T01:19:19.853 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T01:19:19.853 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T01:19:19.853 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T01:19:19.853 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T01:19:19.853 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T01:19:19.853 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T01:19:19.853 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T01:19:19.853 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T01:19:19.853 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T01:19:19.853 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T01:19:19.853 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T01:19:19.853 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T01:19:19.853 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T01:19:19.854 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T01:19:19.854 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T01:19:19.854 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T01:19:19.854 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T01:19:19.854 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T01:19:19.854 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T01:19:19.854 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T01:19:19.854 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T01:19:19.854 INFO:teuthology.orchestra.run.smithi032.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T01:19:19.854 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T01:19:19.855 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T01:19:19.855 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T01:19:19.855 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T01:19:19.855 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T01:19:19.855 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T01:19:19.855 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T01:19:19.855 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T01:19:19.855 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T01:19:19.855 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T01:19:19.855 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T01:19:19.855 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T01:19:19.856 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T01:19:19.856 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T01:19:19.856 INFO:teuthology.orchestra.run.smithi032.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T01:19:19.856 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T01:19:19.856 INFO:teuthology.orchestra.run.smithi032.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T01:19:19.856 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T01:19:19.856 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T01:19:19.856 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T01:19:19.856 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T01:19:19.856 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T01:19:19.856 INFO:teuthology.orchestra.run.smithi032.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T01:19:19.856 INFO:teuthology.orchestra.run.smithi032.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T01:19:19.856 INFO:teuthology.orchestra.run.smithi032.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T01:19:19.857 INFO:teuthology.orchestra.run.smithi032.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T01:19:19.857 INFO:teuthology.orchestra.run.smithi032.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T01:19:19.857 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T01:19:19.857 INFO:teuthology.orchestra.run.smithi032.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T01:19:19.857 INFO:teuthology.orchestra.run.smithi032.stdout:Installing weak dependencies: 2024-04-25T01:19:19.857 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T01:19:19.857 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-25T01:19:19.857 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-25T01:19:19.857 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================= 2024-04-25T01:19:19.857 INFO:teuthology.orchestra.run.smithi032.stdout:Install 148 Packages 2024-04-25T01:19:19.857 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrade 4 Packages 2024-04-25T01:19:19.857 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-25T01:19:19.858 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 220 M 2024-04-25T01:19:19.858 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-25T01:19:20.115 INFO:teuthology.orchestra.run.smithi028.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-04-25T01:19:20.440 INFO:teuthology.orchestra.run.smithi028.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 2.7 MB/s | 890 kB 00:00 2024-04-25T01:19:20.524 INFO:teuthology.orchestra.run.smithi028.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 1.7 MB/s | 145 kB 00:00 2024-04-25T01:19:20.824 INFO:teuthology.orchestra.run.smithi028.stdout:(4/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 5.5 MB/s | 5.4 MB 00:00 2024-04-25T01:19:20.843 INFO:teuthology.orchestra.run.smithi032.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-04-25T01:19:21.066 INFO:teuthology.orchestra.run.smithi028.stdout:(5/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 6.2 MB/s | 1.5 MB 00:00 2024-04-25T01:19:21.133 INFO:teuthology.orchestra.run.smithi028.stdout:(6/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 3.9 MB/s | 2.4 MB 00:00 2024-04-25T01:19:21.444 INFO:teuthology.orchestra.run.smithi032.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.4 MB/s | 890 kB 00:00 2024-04-25T01:19:21.544 INFO:teuthology.orchestra.run.smithi032.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 1.4 MB/s | 145 kB 00:00 2024-04-25T01:19:21.978 INFO:teuthology.orchestra.run.smithi032.stdout:(4/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 5.5 MB/s | 2.4 MB 00:00 2024-04-25T01:19:22.025 INFO:teuthology.orchestra.run.smithi028.stdout:(7/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 4.9 MB/s | 4.7 MB 00:00 2024-04-25T01:19:22.211 INFO:teuthology.orchestra.run.smithi032.stdout:(5/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 6.4 MB/s | 1.5 MB 00:00 2024-04-25T01:19:22.895 INFO:teuthology.orchestra.run.smithi032.stdout:(6/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 6.9 MB/s | 4.7 MB 00:00 2024-04-25T01:19:23.621 INFO:teuthology.orchestra.run.smithi130.stdout:Extra Packages for Enterprise Linux 4.3 MB/s | 21 MB 00:04 2024-04-25T01:19:23.742 INFO:teuthology.orchestra.run.smithi028.stdout:(8/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el9 6.0 MB/s | 10 MB 00:01 2024-04-25T01:19:23.817 INFO:teuthology.orchestra.run.smithi028.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 336 kB/s | 25 kB 00:00 2024-04-25T01:19:25.003 INFO:teuthology.orchestra.run.smithi032.stdout:(7/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x86 8.0 MB/s | 17 MB 00:02 2024-04-25T01:19:25.225 INFO:teuthology.orchestra.run.smithi028.stdout:(10/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x8 4.1 MB/s | 17 MB 00:04 2024-04-25T01:19:25.325 INFO:teuthology.orchestra.run.smithi028.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 2.2 MB/s | 222 kB 00:00 2024-04-25T01:19:25.417 INFO:teuthology.orchestra.run.smithi028.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 295 kB/s | 27 kB 00:00 2024-04-25T01:19:25.518 INFO:teuthology.orchestra.run.smithi028.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 317 kB/s | 32 kB 00:00 2024-04-25T01:19:25.768 INFO:teuthology.orchestra.run.smithi028.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 2.9 MB/s | 745 kB 00:00 2024-04-25T01:19:25.860 INFO:teuthology.orchestra.run.smithi028.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 1.7 MB/s | 163 kB 00:00 2024-04-25T01:19:25.960 INFO:teuthology.orchestra.run.smithi028.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 1.2 MB/s | 127 kB 00:00 2024-04-25T01:19:26.119 INFO:teuthology.orchestra.run.smithi028.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 3.1 MB/s | 501 kB 00:00 2024-04-25T01:19:26.187 INFO:teuthology.orchestra.run.smithi032.stdout:(8/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el9 8.7 MB/s | 10 MB 00:01 2024-04-25T01:19:26.287 INFO:teuthology.orchestra.run.smithi032.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 252 kB/s | 25 kB 00:00 2024-04-25T01:19:27.553 INFO:teuthology.orchestra.run.smithi028.stdout:(18/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 3.7 MB/s | 5.3 MB 00:01 2024-04-25T01:19:27.619 INFO:teuthology.orchestra.run.smithi028.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 680 kB/s | 45 kB 00:00 2024-04-25T01:19:27.703 INFO:teuthology.orchestra.run.smithi028.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 1.6 MB/s | 133 kB 00:00 2024-04-25T01:19:27.795 INFO:teuthology.orchestra.run.smithi028.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 1.7 MB/s | 162 kB 00:00 2024-04-25T01:19:27.962 INFO:teuthology.orchestra.run.smithi028.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 1.9 MB/s | 321 kB 00:00 2024-04-25T01:19:28.112 INFO:teuthology.orchestra.run.smithi028.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 1.9 MB/s | 299 kB 00:00 2024-04-25T01:19:28.196 INFO:teuthology.orchestra.run.smithi028.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.2 MB/s | 99 kB 00:00 2024-04-25T01:19:28.288 INFO:teuthology.orchestra.run.smithi028.stdout:(25/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 931 kB/s | 85 kB 00:00 2024-04-25T01:19:28.687 INFO:teuthology.orchestra.run.smithi130.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-04-25T01:19:29.220 INFO:teuthology.orchestra.run.smithi032.stdout:(10/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x 621 kB/s | 5.4 MB 00:08 2024-04-25T01:19:29.247 INFO:teuthology.orchestra.run.smithi028.stdout:(26/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 3.2 MB/s | 3.1 MB 00:00 2024-04-25T01:19:29.322 INFO:teuthology.orchestra.run.smithi028.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 2.2 MB/s | 171 kB 00:00 2024-04-25T01:19:29.389 INFO:teuthology.orchestra.run.smithi028.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 476 kB/s | 32 kB 00:00 2024-04-25T01:19:29.464 INFO:teuthology.orchestra.run.smithi028.stdout:(29/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 1.8 MB/s | 141 kB 00:00 2024-04-25T01:19:29.504 INFO:teuthology.orchestra.run.smithi032.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 784 kB/s | 222 kB 00:00 2024-04-25T01:19:29.621 INFO:teuthology.orchestra.run.smithi032.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 232 kB/s | 27 kB 00:00 2024-04-25T01:19:29.759 INFO:teuthology.orchestra.run.smithi032.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 229 kB/s | 32 kB 00:00 2024-04-25T01:19:30.306 INFO:teuthology.orchestra.run.smithi028.stdout:(30/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 4.2 MB/s | 3.5 MB 00:00 2024-04-25T01:19:30.379 INFO:teuthology.orchestra.run.smithi032.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 1.2 MB/s | 745 kB 00:00 2024-04-25T01:19:30.589 INFO:teuthology.orchestra.run.smithi032.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 783 kB/s | 163 kB 00:00 2024-04-25T01:19:30.697 INFO:teuthology.orchestra.run.smithi032.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 1.1 MB/s | 127 kB 00:00 2024-04-25T01:19:30.873 INFO:teuthology.orchestra.run.smithi130.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:19:30.874 INFO:teuthology.orchestra.run.smithi130.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:19:30.884 INFO:teuthology.orchestra.run.smithi130.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T01:19:30.886 INFO:teuthology.orchestra.run.smithi130.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T01:19:30.886 INFO:teuthology.orchestra.run.smithi130.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T01:19:30.887 INFO:teuthology.orchestra.run.smithi130.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T01:19:30.889 INFO:teuthology.orchestra.run.smithi130.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T01:19:30.891 INFO:teuthology.orchestra.run.smithi130.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T01:19:30.892 INFO:teuthology.orchestra.run.smithi130.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T01:19:30.953 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-04-25T01:19:30.973 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================= 2024-04-25T01:19:30.973 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-04-25T01:19:30.973 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================= 2024-04-25T01:19:30.973 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-04-25T01:19:30.973 INFO:teuthology.orchestra.run.smithi130.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T01:19:30.973 INFO:teuthology.orchestra.run.smithi130.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T01:19:30.973 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T01:19:30.973 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T01:19:30.973 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T01:19:30.973 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T01:19:30.974 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T01:19:30.974 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T01:19:30.974 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T01:19:30.974 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T01:19:30.974 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T01:19:30.974 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T01:19:30.974 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T01:19:30.974 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T01:19:30.974 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T01:19:30.974 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T01:19:30.974 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T01:19:30.975 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T01:19:30.975 INFO:teuthology.orchestra.run.smithi130.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T01:19:30.975 INFO:teuthology.orchestra.run.smithi130.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T01:19:30.975 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T01:19:30.975 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T01:19:30.975 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T01:19:30.975 INFO:teuthology.orchestra.run.smithi130.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T01:19:30.975 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T01:19:30.975 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T01:19:30.975 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T01:19:30.975 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T01:19:30.976 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T01:19:30.976 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T01:19:30.976 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T01:19:30.976 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T01:19:30.976 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T01:19:30.976 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T01:19:30.976 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:19:30.976 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T01:19:30.976 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T01:19:30.976 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrading: 2024-04-25T01:19:30.976 INFO:teuthology.orchestra.run.smithi130.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T01:19:30.977 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:19:30.977 INFO:teuthology.orchestra.run.smithi130.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T01:19:30.977 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T01:19:30.977 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-04-25T01:19:30.977 INFO:teuthology.orchestra.run.smithi130.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T01:19:30.977 INFO:teuthology.orchestra.run.smithi130.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T01:19:30.977 INFO:teuthology.orchestra.run.smithi130.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T01:19:30.977 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T01:19:30.977 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T01:19:30.977 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T01:19:30.977 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T01:19:30.977 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T01:19:30.978 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T01:19:30.978 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T01:19:30.978 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T01:19:30.978 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T01:19:30.978 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T01:19:30.978 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T01:19:30.978 INFO:teuthology.orchestra.run.smithi130.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T01:19:30.978 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T01:19:30.979 INFO:teuthology.orchestra.run.smithi130.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T01:19:30.979 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T01:19:30.979 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T01:19:30.979 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T01:19:30.979 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T01:19:30.979 INFO:teuthology.orchestra.run.smithi130.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T01:19:30.979 INFO:teuthology.orchestra.run.smithi130.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T01:19:30.979 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T01:19:30.979 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T01:19:30.979 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T01:19:30.980 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T01:19:30.980 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T01:19:30.980 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T01:19:30.980 INFO:teuthology.orchestra.run.smithi130.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T01:19:30.980 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T01:19:30.980 INFO:teuthology.orchestra.run.smithi130.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T01:19:30.980 INFO:teuthology.orchestra.run.smithi130.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T01:19:30.980 INFO:teuthology.orchestra.run.smithi130.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T01:19:30.980 INFO:teuthology.orchestra.run.smithi130.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T01:19:30.981 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T01:19:30.981 INFO:teuthology.orchestra.run.smithi130.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T01:19:30.981 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T01:19:30.981 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T01:19:30.981 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T01:19:30.981 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T01:19:30.981 INFO:teuthology.orchestra.run.smithi130.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T01:19:30.981 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T01:19:30.981 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T01:19:30.982 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T01:19:30.982 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T01:19:30.982 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T01:19:30.982 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T01:19:30.982 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T01:19:30.982 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T01:19:30.982 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T01:19:30.982 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T01:19:30.982 INFO:teuthology.orchestra.run.smithi130.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T01:19:30.982 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T01:19:30.982 INFO:teuthology.orchestra.run.smithi130.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T01:19:30.982 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T01:19:30.982 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T01:19:30.983 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T01:19:30.983 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T01:19:30.983 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T01:19:30.983 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T01:19:30.983 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T01:19:30.983 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T01:19:30.983 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T01:19:30.983 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T01:19:30.983 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T01:19:30.983 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T01:19:30.983 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T01:19:30.983 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T01:19:30.983 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T01:19:30.983 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T01:19:30.984 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T01:19:30.984 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T01:19:30.984 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T01:19:30.984 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T01:19:30.984 INFO:teuthology.orchestra.run.smithi130.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T01:19:30.984 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T01:19:30.984 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T01:19:30.984 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T01:19:30.984 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T01:19:30.984 INFO:teuthology.orchestra.run.smithi130.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T01:19:30.984 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T01:19:30.984 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T01:19:30.984 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T01:19:30.984 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T01:19:30.984 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T01:19:30.985 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T01:19:30.985 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T01:19:30.985 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T01:19:30.985 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T01:19:30.985 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T01:19:30.985 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T01:19:30.985 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T01:19:30.985 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T01:19:30.985 INFO:teuthology.orchestra.run.smithi130.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T01:19:30.985 INFO:teuthology.orchestra.run.smithi130.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T01:19:30.985 INFO:teuthology.orchestra.run.smithi130.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T01:19:30.985 INFO:teuthology.orchestra.run.smithi130.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T01:19:30.985 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T01:19:30.985 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T01:19:30.986 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T01:19:30.986 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T01:19:30.986 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T01:19:30.986 INFO:teuthology.orchestra.run.smithi130.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T01:19:30.986 INFO:teuthology.orchestra.run.smithi130.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T01:19:30.986 INFO:teuthology.orchestra.run.smithi130.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T01:19:30.986 INFO:teuthology.orchestra.run.smithi130.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T01:19:30.986 INFO:teuthology.orchestra.run.smithi130.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T01:19:30.986 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T01:19:30.986 INFO:teuthology.orchestra.run.smithi130.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T01:19:30.986 INFO:teuthology.orchestra.run.smithi130.stdout:Installing weak dependencies: 2024-04-25T01:19:30.986 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T01:19:30.986 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-25T01:19:30.987 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-04-25T01:19:30.987 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================= 2024-04-25T01:19:30.987 INFO:teuthology.orchestra.run.smithi130.stdout:Install 148 Packages 2024-04-25T01:19:30.987 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrade 4 Packages 2024-04-25T01:19:30.987 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-25T01:19:30.987 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 220 M 2024-04-25T01:19:30.987 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-04-25T01:19:31.039 INFO:teuthology.orchestra.run.smithi032.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 1.4 MB/s | 501 kB 00:00 2024-04-25T01:19:31.931 INFO:teuthology.orchestra.run.smithi130.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-04-25T01:19:31.931 INFO:teuthology.orchestra.run.smithi028.stdout:(31/152): ceph-mgr-diskprediction-local-19.0.0- 4.5 MB/s | 7.4 MB 00:01 2024-04-25T01:19:31.998 INFO:teuthology.orchestra.run.smithi028.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 3.7 MB/s | 249 kB 00:00 2024-04-25T01:19:32.031 INFO:teuthology.orchestra.run.smithi032.stdout:(18/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 8.4 MB/s | 48 MB 00:05 2024-04-25T01:19:32.057 INFO:teuthology.orchestra.run.smithi028.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 847 kB/s | 49 kB 00:00 2024-04-25T01:19:32.123 INFO:teuthology.orchestra.run.smithi032.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 494 kB/s | 45 kB 00:00 2024-04-25T01:19:32.132 INFO:teuthology.orchestra.run.smithi028.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 204 kB/s | 15 kB 00:00 2024-04-25T01:19:32.215 INFO:teuthology.orchestra.run.smithi032.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 1.4 MB/s | 133 kB 00:00 2024-04-25T01:19:32.216 INFO:teuthology.orchestra.run.smithi028.stdout:(35/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 3.1 MB/s | 264 kB 00:00 2024-04-25T01:19:32.315 INFO:teuthology.orchestra.run.smithi032.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 1.6 MB/s | 162 kB 00:00 2024-04-25T01:19:32.432 INFO:teuthology.orchestra.run.smithi032.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 2.7 MB/s | 321 kB 00:00 2024-04-25T01:19:32.449 INFO:teuthology.orchestra.run.smithi028.stdout:(36/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 3.2 MB/s | 762 kB 00:00 2024-04-25T01:19:32.524 INFO:teuthology.orchestra.run.smithi032.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 3.2 MB/s | 299 kB 00:00 2024-04-25T01:19:32.608 INFO:teuthology.orchestra.run.smithi032.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.2 MB/s | 99 kB 00:00 2024-04-25T01:19:32.684 INFO:teuthology.orchestra.run.smithi032.stdout:(25/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.1 MB/s | 85 kB 00:00 2024-04-25T01:19:32.822 INFO:teuthology.orchestra.run.smithi130.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 998 kB/s | 890 kB 00:00 2024-04-25T01:19:32.866 INFO:teuthology.orchestra.run.smithi028.stdout:(37/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 5.3 MB/s | 48 MB 00:09 2024-04-25T01:19:32.891 INFO:teuthology.orchestra.run.smithi028.stdout:(38/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 41 kB/s | 18 kB 00:00 2024-04-25T01:19:32.989 INFO:teuthology.orchestra.run.smithi130.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 869 kB/s | 145 kB 00:00 2024-04-25T01:19:33.050 INFO:teuthology.orchestra.run.smithi028.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 438 kB/s | 80 kB 00:00 2024-04-25T01:19:33.090 INFO:teuthology.orchestra.run.smithi130.stdout:(4/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 3.6 MB/s | 5.4 MB 00:01 2024-04-25T01:19:33.200 INFO:teuthology.orchestra.run.smithi032.stdout:(26/152): ceph-common-19.0.0-2445.gd2c45d97.el9 1.6 MB/s | 21 MB 00:12 2024-04-25T01:19:33.215 INFO:teuthology.orchestra.run.smithi130.stdout:(5/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 12 MB/s | 1.5 MB 00:00 2024-04-25T01:19:33.250 INFO:teuthology.orchestra.run.smithi028.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 976 kB/s | 195 kB 00:00 2024-04-25T01:19:33.259 INFO:teuthology.orchestra.run.smithi032.stdout:(27/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 5.4 MB/s | 3.1 MB 00:00 2024-04-25T01:19:33.351 INFO:teuthology.orchestra.run.smithi028.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-04-25T01:19:33.459 INFO:teuthology.orchestra.run.smithi028.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-04-25T01:19:33.535 INFO:teuthology.orchestra.run.smithi032.stdout:(28/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 2.1 MB/s | 5.3 MB 00:02 2024-04-25T01:19:33.560 INFO:teuthology.orchestra.run.smithi028.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 106 kB 00:00 2024-04-25T01:19:33.599 INFO:teuthology.orchestra.run.smithi130.stdout:(6/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 12 MB/s | 4.7 MB 00:00 2024-04-25T01:19:33.643 INFO:teuthology.orchestra.run.smithi032.stdout:(29/152): ceph-grafana-dashboards-19.0.0-2445.g 83 kB/s | 32 kB 00:00 2024-04-25T01:19:33.668 INFO:teuthology.orchestra.run.smithi032.stdout:(30/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 366 kB/s | 171 kB 00:00 2024-04-25T01:19:33.694 INFO:teuthology.orchestra.run.smithi032.stdout:(31/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 906 kB/s | 141 kB 00:00 2024-04-25T01:19:33.701 INFO:teuthology.orchestra.run.smithi028.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 954 kB/s | 135 kB 00:00 2024-04-25T01:19:33.770 INFO:teuthology.orchestra.run.smithi028.stdout:(45/152): python3-cryptography-36.0.1-4.el9.x86 2.9 MB/s | 1.2 MB 00:00 2024-04-25T01:19:33.794 INFO:teuthology.orchestra.run.smithi028.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-04-25T01:19:33.861 INFO:teuthology.orchestra.run.smithi028.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.3 MB/s | 215 kB 00:00 2024-04-25T01:19:33.870 INFO:teuthology.orchestra.run.smithi032.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 1.4 MB/s | 249 kB 00:00 2024-04-25T01:19:33.995 INFO:teuthology.orchestra.run.smithi032.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 395 kB/s | 49 kB 00:00 2024-04-25T01:19:34.145 INFO:teuthology.orchestra.run.smithi032.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 102 kB/s | 15 kB 00:00 2024-04-25T01:19:34.224 INFO:teuthology.orchestra.run.smithi130.stdout:(7/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 1.9 MB/s | 2.4 MB 00:01 2024-04-25T01:19:34.270 INFO:teuthology.orchestra.run.smithi032.stdout:(35/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 5.6 MB/s | 3.5 MB 00:00 2024-04-25T01:19:34.295 INFO:teuthology.orchestra.run.smithi032.stdout:(36/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 1.7 MB/s | 264 kB 00:00 2024-04-25T01:19:34.395 INFO:teuthology.orchestra.run.smithi032.stdout:(37/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 5.9 MB/s | 762 kB 00:00 2024-04-25T01:19:34.420 INFO:teuthology.orchestra.run.smithi032.stdout:(38/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 145 kB/s | 18 kB 00:00 2024-04-25T01:19:34.461 INFO:teuthology.orchestra.run.smithi028.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 1.0 MB/s | 689 kB 00:00 2024-04-25T01:19:34.711 INFO:teuthology.orchestra.run.smithi028.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 1.1 MB/s | 943 kB 00:00 2024-04-25T01:19:34.712 INFO:teuthology.orchestra.run.smithi032.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 253 kB/s | 80 kB 00:00 2024-04-25T01:19:34.728 INFO:teuthology.orchestra.run.smithi028.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 400 kB/s | 107 kB 00:00 2024-04-25T01:19:34.862 INFO:teuthology.orchestra.run.smithi028.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 143 kB/s | 21 kB 00:00 2024-04-25T01:19:34.941 INFO:teuthology.orchestra.run.smithi130.stdout:(8/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x86 13 MB/s | 17 MB 00:01 2024-04-25T01:19:34.991 INFO:teuthology.orchestra.run.smithi130.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 504 kB/s | 25 kB 00:00 2024-04-25T01:19:35.038 INFO:teuthology.orchestra.run.smithi032.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 601 kB/s | 195 kB 00:00 2024-04-25T01:19:35.079 INFO:teuthology.orchestra.run.smithi028.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 151 kB/s | 33 kB 00:00 2024-04-25T01:19:35.113 INFO:teuthology.orchestra.run.smithi032.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-04-25T01:19:35.254 INFO:teuthology.orchestra.run.smithi028.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 597 kB/s | 313 kB 00:00 2024-04-25T01:19:35.679 INFO:teuthology.orchestra.run.smithi028.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 43 kB/s | 18 kB 00:00 2024-04-25T01:19:35.854 INFO:teuthology.orchestra.run.smithi028.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 64 kB/s | 11 kB 00:00 2024-04-25T01:19:35.930 INFO:teuthology.orchestra.run.smithi032.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 1.5 MB/s | 1.2 MB 00:00 2024-04-25T01:19:35.997 INFO:teuthology.orchestra.run.smithi032.stdout:(43/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 264 kB/s | 253 kB 00:00 2024-04-25T01:19:36.339 INFO:teuthology.orchestra.run.smithi032.stdout:(44/152): ceph-mgr-diskprediction-local-19.0.0- 2.8 MB/s | 7.4 MB 00:02 2024-04-25T01:19:36.364 INFO:teuthology.orchestra.run.smithi032.stdout:(45/152): python3-ply-3.11-14.el9.noarch.rpm 245 kB/s | 106 kB 00:00 2024-04-25T01:19:36.440 INFO:teuthology.orchestra.run.smithi032.stdout:(46/152): python3-pycparser-2.20-6.el9.noarch.r 305 kB/s | 135 kB 00:00 2024-04-25T01:19:36.455 INFO:teuthology.orchestra.run.smithi028.stdout:(56/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.2 MB/s | 3.0 MB 00:01 2024-04-25T01:19:36.555 INFO:teuthology.orchestra.run.smithi028.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 229 kB/s | 160 kB 00:00 2024-04-25T01:19:36.615 INFO:teuthology.orchestra.run.smithi032.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 3.8 MB/s | 689 kB 00:00 2024-04-25T01:19:36.639 INFO:teuthology.orchestra.run.smithi028.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 246 kB/s | 45 kB 00:00 2024-04-25T01:19:36.673 INFO:teuthology.orchestra.run.smithi032.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 16 MB/s | 943 kB 00:00 2024-04-25T01:19:36.699 INFO:teuthology.orchestra.run.smithi032.stdout:(49/152): python3-requests-2.25.1-8.el9.noarch. 347 kB/s | 125 kB 00:00 2024-04-25T01:19:36.724 INFO:teuthology.orchestra.run.smithi032.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-04-25T01:19:36.766 INFO:teuthology.orchestra.run.smithi032.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 7.4 MB/s | 313 kB 00:00 2024-04-25T01:19:36.792 INFO:teuthology.orchestra.run.smithi032.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 233 kB/s | 21 kB 00:00 2024-04-25T01:19:36.817 INFO:teuthology.orchestra.run.smithi032.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 651 kB/s | 33 kB 00:00 2024-04-25T01:19:36.859 INFO:teuthology.orchestra.run.smithi032.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 439 kB/s | 18 kB 00:00 2024-04-25T01:19:36.900 INFO:teuthology.orchestra.run.smithi032.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 269 kB/s | 11 kB 00:00 2024-04-25T01:19:36.943 INFO:teuthology.orchestra.run.smithi032.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 160 kB 00:00 2024-04-25T01:19:36.986 INFO:teuthology.orchestra.run.smithi032.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-04-25T01:19:37.035 INFO:teuthology.orchestra.run.smithi032.stdout:(58/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-04-25T01:19:37.068 INFO:teuthology.orchestra.run.smithi032.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 662 kB 00:00 2024-04-25T01:19:37.094 INFO:teuthology.orchestra.run.smithi032.stdout:(60/152): python3-urllib3-1.26.5-5.el9.noarch.r 294 kB/s | 215 kB 00:00 2024-04-25T01:19:37.119 INFO:teuthology.orchestra.run.smithi032.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 506 kB/s | 25 kB 00:00 2024-04-25T01:19:37.153 INFO:teuthology.orchestra.run.smithi032.stdout:(62/152): libtool-2.4.6-45.el9.x86_64.rpm 4.8 MB/s | 582 kB 00:00 2024-04-25T01:19:37.180 INFO:teuthology.orchestra.run.smithi028.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 1.0 MB/s | 582 kB 00:00 2024-04-25T01:19:37.186 INFO:teuthology.orchestra.run.smithi032.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-04-25T01:19:37.212 INFO:teuthology.orchestra.run.smithi032.stdout:(64/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 403 kB/s | 47 kB 00:00 2024-04-25T01:19:37.237 INFO:teuthology.orchestra.run.smithi032.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 701 kB/s | 35 kB 00:00 2024-04-25T01:19:37.262 INFO:teuthology.orchestra.run.smithi032.stdout:(66/152): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-25T01:19:37.323 INFO:teuthology.orchestra.run.smithi028.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 179 kB/s | 25 kB 00:00 2024-04-25T01:19:37.329 INFO:teuthology.orchestra.run.smithi032.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-04-25T01:19:37.398 INFO:teuthology.orchestra.run.smithi032.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 28 MB/s | 4.4 MB 00:00 2024-04-25T01:19:37.430 INFO:teuthology.orchestra.run.smithi032.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 8.9 MB/s | 911 kB 00:00 2024-04-25T01:19:37.455 INFO:teuthology.orchestra.run.smithi032.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-25T01:19:37.539 INFO:teuthology.orchestra.run.smithi032.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-04-25T01:19:37.548 INFO:teuthology.orchestra.run.smithi028.stdout:(61/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 211 kB/s | 47 kB 00:00 2024-04-25T01:19:37.581 INFO:teuthology.orchestra.run.smithi032.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 8.1 MB/s | 1.0 MB 00:00 2024-04-25T01:19:37.606 INFO:teuthology.orchestra.run.smithi032.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-25T01:19:37.706 INFO:teuthology.orchestra.run.smithi032.stdout:(74/152): postgresql-server-13.14-1.el9.x86_64. 21 MB/s | 5.8 MB 00:00 2024-04-25T01:19:37.731 INFO:teuthology.orchestra.run.smithi032.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.9 MB/s | 244 kB 00:00 2024-04-25T01:19:37.757 INFO:teuthology.orchestra.run.smithi032.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-04-25T01:19:37.784 INFO:teuthology.orchestra.run.smithi032.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 490 kB/s | 26 kB 00:00 2024-04-25T01:19:37.810 INFO:teuthology.orchestra.run.smithi032.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 365 kB/s | 19 kB 00:00 2024-04-25T01:19:37.835 INFO:teuthology.orchestra.run.smithi032.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-04-25T01:19:37.840 INFO:teuthology.orchestra.run.smithi028.stdout:(62/152): librdkafka-1.6.1-102.el9.x86_64.rpm 516 kB/s | 662 kB 00:01 2024-04-25T01:19:37.860 INFO:teuthology.orchestra.run.smithi032.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 693 kB/s | 35 kB 00:00 2024-04-25T01:19:37.874 INFO:teuthology.orchestra.run.smithi130.stdout:(10/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 17 MB/s | 48 MB 00:02 2024-04-25T01:19:37.890 INFO:teuthology.orchestra.run.smithi028.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 855 kB/s | 292 kB 00:00 2024-04-25T01:19:37.952 INFO:teuthology.orchestra.run.smithi032.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-04-25T01:19:37.977 INFO:teuthology.orchestra.run.smithi032.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-04-25T01:19:38.003 INFO:teuthology.orchestra.run.smithi032.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-04-25T01:19:38.028 INFO:teuthology.orchestra.run.smithi032.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-25T01:19:38.032 INFO:teuthology.orchestra.run.smithi028.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 249 kB/s | 35 kB 00:00 2024-04-25T01:19:38.053 INFO:teuthology.orchestra.run.smithi032.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 841 kB/s | 42 kB 00:00 2024-04-25T01:19:38.087 INFO:teuthology.orchestra.run.smithi032.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-04-25T01:19:38.099 INFO:teuthology.orchestra.run.smithi130.stdout:(11/152): ceph-common-19.0.0-2445.gd2c45d97.el9 3.2 MB/s | 21 MB 00:06 2024-04-25T01:19:38.112 INFO:teuthology.orchestra.run.smithi032.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-04-25T01:19:38.124 INFO:teuthology.orchestra.run.smithi130.stdout:(12/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 888 kB/s | 222 kB 00:00 2024-04-25T01:19:38.137 INFO:teuthology.orchestra.run.smithi032.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-25T01:19:38.163 INFO:teuthology.orchestra.run.smithi032.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-25T01:19:38.166 INFO:teuthology.orchestra.run.smithi130.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 761 kB/s | 32 kB 00:00 2024-04-25T01:19:38.188 INFO:teuthology.orchestra.run.smithi032.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-25T01:19:38.218 INFO:teuthology.orchestra.run.smithi130.stdout:(14/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 231 kB/s | 27 kB 00:00 2024-04-25T01:19:38.250 INFO:teuthology.orchestra.run.smithi130.stdout:(15/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 8.7 MB/s | 745 kB 00:00 2024-04-25T01:19:38.255 INFO:teuthology.orchestra.run.smithi032.stdout:(91/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-04-25T01:19:38.265 INFO:teuthology.orchestra.run.smithi028.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 443 kB/s | 188 kB 00:00 2024-04-25T01:19:38.280 INFO:teuthology.orchestra.run.smithi032.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 452 kB/s | 42 kB 00:00 2024-04-25T01:19:38.292 INFO:teuthology.orchestra.run.smithi130.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 3.0 MB/s | 127 kB 00:00 2024-04-25T01:19:38.306 INFO:teuthology.orchestra.run.smithi032.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 881 kB/s | 44 kB 00:00 2024-04-25T01:19:38.339 INFO:teuthology.orchestra.run.smithi032.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-04-25T01:19:38.343 INFO:teuthology.orchestra.run.smithi130.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 9.7 MB/s | 501 kB 00:00 2024-04-25T01:19:38.365 INFO:teuthology.orchestra.run.smithi032.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 613 kB/s | 36 kB 00:00 2024-04-25T01:19:38.390 INFO:teuthology.orchestra.run.smithi032.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-04-25T01:19:38.393 INFO:teuthology.orchestra.run.smithi130.stdout:(18/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 926 kB/s | 163 kB 00:00 2024-04-25T01:19:38.415 INFO:teuthology.orchestra.run.smithi032.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 305 kB 00:00 2024-04-25T01:19:38.440 INFO:teuthology.orchestra.run.smithi032.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-25T01:19:38.466 INFO:teuthology.orchestra.run.smithi032.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-25T01:19:38.491 INFO:teuthology.orchestra.run.smithi032.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-25T01:19:38.502 INFO:teuthology.orchestra.run.smithi130.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 416 kB/s | 45 kB 00:00 2024-04-25T01:19:38.516 INFO:teuthology.orchestra.run.smithi032.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 894 kB/s | 45 kB 00:00 2024-04-25T01:19:38.542 INFO:teuthology.orchestra.run.smithi032.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 443 kB/s | 22 kB 00:00 2024-04-25T01:19:38.575 INFO:teuthology.orchestra.run.smithi032.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 14 MB/s | 863 kB 00:00 2024-04-25T01:19:38.644 INFO:teuthology.orchestra.run.smithi130.stdout:(20/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 18 MB/s | 5.3 MB 00:00 2024-04-25T01:19:38.669 INFO:teuthology.orchestra.run.smithi130.stdout:(21/152): python3-ceph-common-19.0.0-2445.gd2c4 797 kB/s | 133 kB 00:00 2024-04-25T01:19:38.695 INFO:teuthology.orchestra.run.smithi130.stdout:(22/152): python3-cephfs-19.0.0-2445.gd2c45d97. 3.2 MB/s | 162 kB 00:00 2024-04-25T01:19:38.745 INFO:teuthology.orchestra.run.smithi130.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 5.9 MB/s | 299 kB 00:00 2024-04-25T01:19:38.787 INFO:teuthology.orchestra.run.smithi130.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 2.3 MB/s | 99 kB 00:00 2024-04-25T01:19:38.792 INFO:teuthology.orchestra.run.smithi032.stdout:(104/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-04-25T01:19:38.816 INFO:teuthology.orchestra.run.smithi028.stdout:(66/152): ceph-common-19.0.0-2445.gd2c45d97.el9 1.1 MB/s | 21 MB 00:18 2024-04-25T01:19:38.829 INFO:teuthology.orchestra.run.smithi130.stdout:(25/152): python3-rados-19.0.0-2445.gd2c45d97.e 2.0 MB/s | 321 kB 00:00 2024-04-25T01:19:38.854 INFO:teuthology.orchestra.run.smithi130.stdout:(26/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.2 MB/s | 85 kB 00:00 2024-04-25T01:19:38.885 INFO:teuthology.orchestra.run.smithi032.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-04-25T01:19:38.905 INFO:teuthology.orchestra.run.smithi130.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 3.3 MB/s | 171 kB 00:00 2024-04-25T01:19:38.910 INFO:teuthology.orchestra.run.smithi032.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 166 kB/s | 19 kB 00:00 2024-04-25T01:19:38.935 INFO:teuthology.orchestra.run.smithi032.stdout:(107/152): gperftools-libs-2.9.1-2.el9.x86_64.r 859 kB/s | 309 kB 00:00 2024-04-25T01:19:38.947 INFO:teuthology.orchestra.run.smithi130.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 763 kB/s | 32 kB 00:00 2024-04-25T01:19:38.997 INFO:teuthology.orchestra.run.smithi130.stdout:(29/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 2.8 MB/s | 141 kB 00:00 2024-04-25T01:19:39.027 INFO:teuthology.orchestra.run.smithi032.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 530 kB/s | 49 kB 00:00 2024-04-25T01:19:39.052 INFO:teuthology.orchestra.run.smithi032.stdout:(109/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 183 kB/s | 26 kB 00:00 2024-04-25T01:19:39.119 INFO:teuthology.orchestra.run.smithi032.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 732 kB/s | 67 kB 00:00 2024-04-25T01:19:39.203 INFO:teuthology.orchestra.run.smithi032.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-04-25T01:19:39.214 INFO:teuthology.orchestra.run.smithi130.stdout:(30/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 16 MB/s | 3.5 MB 00:00 2024-04-25T01:19:39.320 INFO:teuthology.orchestra.run.smithi032.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-04-25T01:19:39.387 INFO:teuthology.orchestra.run.smithi032.stdout:(113/152): libarrow-9.0.0-10.el9.x86_64.rpm 8.8 MB/s | 4.4 MB 00:00 2024-04-25T01:19:39.412 INFO:teuthology.orchestra.run.smithi032.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 472 kB/s | 43 kB 00:00 2024-04-25T01:19:39.437 INFO:teuthology.orchestra.run.smithi032.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 643 kB/s | 32 kB 00:00 2024-04-25T01:19:39.463 INFO:teuthology.orchestra.run.smithi032.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-04-25T01:19:39.505 INFO:teuthology.orchestra.run.smithi032.stdout:(117/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.1 MB/s | 838 kB 00:00 2024-04-25T01:19:39.539 INFO:teuthology.orchestra.run.smithi032.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-04-25T01:19:39.564 INFO:teuthology.orchestra.run.smithi032.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-04-25T01:19:39.597 INFO:teuthology.orchestra.run.smithi032.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 604 kB/s | 35 kB 00:00 2024-04-25T01:19:39.606 INFO:teuthology.orchestra.run.smithi130.stdout:(31/152): ceph-mgr-diskprediction-local-19.0.0- 19 MB/s | 7.4 MB 00:00 2024-04-25T01:19:39.623 INFO:teuthology.orchestra.run.smithi032.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 3.4 MB/s | 206 kB 00:00 2024-04-25T01:19:39.656 INFO:teuthology.orchestra.run.smithi130.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 4.9 MB/s | 249 kB 00:00 2024-04-25T01:19:39.664 INFO:teuthology.orchestra.run.smithi032.stdout:(122/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.9 MB/s | 303 kB 00:00 2024-04-25T01:19:39.690 INFO:teuthology.orchestra.run.smithi032.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-04-25T01:19:39.706 INFO:teuthology.orchestra.run.smithi130.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 991 kB/s | 49 kB 00:00 2024-04-25T01:19:39.715 INFO:teuthology.orchestra.run.smithi032.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-04-25T01:19:39.741 INFO:teuthology.orchestra.run.smithi032.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 351 kB/s | 18 kB 00:00 2024-04-25T01:19:39.748 INFO:teuthology.orchestra.run.smithi130.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 366 kB/s | 15 kB 00:00 2024-04-25T01:19:39.766 INFO:teuthology.orchestra.run.smithi032.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-04-25T01:19:39.791 INFO:teuthology.orchestra.run.smithi032.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-04-25T01:19:39.807 INFO:teuthology.orchestra.run.smithi130.stdout:(35/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 4.4 MB/s | 264 kB 00:00 2024-04-25T01:19:39.816 INFO:teuthology.orchestra.run.smithi032.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-04-25T01:19:39.875 INFO:teuthology.orchestra.run.smithi032.stdout:(129/152): python3-grpcio-1.46.7-10.el9.x86_64. 7.3 MB/s | 2.0 MB 00:00 2024-04-25T01:19:39.890 INFO:teuthology.orchestra.run.smithi130.stdout:(36/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 8.9 MB/s | 762 kB 00:00 2024-04-25T01:19:39.900 INFO:teuthology.orchestra.run.smithi032.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 555 kB/s | 46 kB 00:00 2024-04-25T01:19:39.933 INFO:teuthology.orchestra.run.smithi130.stdout:(37/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 437 kB/s | 18 kB 00:00 2024-04-25T01:19:39.942 INFO:teuthology.orchestra.run.smithi032.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.1 MB/s | 79 kB 00:00 2024-04-25T01:19:39.967 INFO:teuthology.orchestra.run.smithi032.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 859 kB/s | 58 kB 00:00 2024-04-25T01:19:40.008 INFO:teuthology.orchestra.run.smithi130.stdout:(38/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 2.6 MB/s | 3.1 MB 00:01 2024-04-25T01:19:40.009 INFO:teuthology.orchestra.run.smithi032.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-04-25T01:19:40.016 INFO:teuthology.orchestra.run.smithi028.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 1.3 MB/s | 1.6 MB 00:01 2024-04-25T01:19:40.035 INFO:teuthology.orchestra.run.smithi032.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 245 kB/s | 16 kB 00:00 2024-04-25T01:19:40.077 INFO:teuthology.orchestra.run.smithi032.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-04-25T01:19:40.102 INFO:teuthology.orchestra.run.smithi032.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 422 kB/s | 28 kB 00:00 2024-04-25T01:19:40.127 INFO:teuthology.orchestra.run.smithi032.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 615 kB/s | 31 kB 00:00 2024-04-25T01:19:40.153 INFO:teuthology.orchestra.run.smithi032.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-04-25T01:19:40.158 INFO:teuthology.orchestra.run.smithi130.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 357 kB/s | 80 kB 00:00 2024-04-25T01:19:40.178 INFO:teuthology.orchestra.run.smithi032.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-25T01:19:40.203 INFO:teuthology.orchestra.run.smithi032.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 716 kB/s | 36 kB 00:00 2024-04-25T01:19:40.237 INFO:teuthology.orchestra.run.smithi032.stdout:(141/152): python3-kubernetes-26.1.0-2.el9.noar 2.3 MB/s | 1.0 MB 00:00 2024-04-25T01:19:40.263 INFO:teuthology.orchestra.run.smithi032.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 608 kB/s | 51 kB 00:00 2024-04-25T01:19:40.296 INFO:teuthology.orchestra.run.smithi032.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-25T01:19:40.308 INFO:teuthology.orchestra.run.smithi130.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-04-25T01:19:40.321 INFO:teuthology.orchestra.run.smithi032.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-04-25T01:19:40.334 INFO:teuthology.orchestra.run.smithi130.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-04-25T01:19:40.346 INFO:teuthology.orchestra.run.smithi032.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-04-25T01:19:40.372 INFO:teuthology.orchestra.run.smithi032.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-04-25T01:19:40.376 INFO:teuthology.orchestra.run.smithi130.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 253 kB 00:00 2024-04-25T01:19:40.468 INFO:teuthology.orchestra.run.smithi130.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 106 kB 00:00 2024-04-25T01:19:40.489 INFO:teuthology.orchestra.run.smithi032.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-04-25T01:19:40.535 INFO:teuthology.orchestra.run.smithi130.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-04-25T01:19:40.539 INFO:teuthology.orchestra.run.smithi032.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 8.2 MB/s | 1.6 MB 00:00 2024-04-25T01:19:40.585 INFO:teuthology.orchestra.run.smithi130.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-25T01:19:40.635 INFO:teuthology.orchestra.run.smithi130.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T01:19:40.749 INFO:teuthology.orchestra.run.smithi028.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 1.2 MB/s | 911 kB 00:00 2024-04-25T01:19:40.794 INFO:teuthology.orchestra.run.smithi130.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 4.2 MB/s | 689 kB 00:00 2024-04-25T01:19:40.853 INFO:teuthology.orchestra.run.smithi130.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 16 MB/s | 943 kB 00:00 2024-04-25T01:19:40.928 INFO:teuthology.orchestra.run.smithi130.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 107 kB 00:00 2024-04-25T01:19:40.956 INFO:teuthology.orchestra.run.smithi032.stdout:(149/152): librados2-19.0.0-2445.gd2c45d97.el9. 5.8 MB/s | 3.4 MB 00:00 2024-04-25T01:19:40.970 INFO:teuthology.orchestra.run.smithi130.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 519 kB/s | 21 kB 00:00 2024-04-25T01:19:41.012 INFO:teuthology.orchestra.run.smithi130.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 7.3 MB/s | 313 kB 00:00 2024-04-25T01:19:41.046 INFO:teuthology.orchestra.run.smithi130.stdout:(52/152): python3-cryptography-36.0.1-4.el9.x86 1.7 MB/s | 1.2 MB 00:00 2024-04-25T01:19:41.063 INFO:teuthology.orchestra.run.smithi130.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 652 kB/s | 33 kB 00:00 2024-04-25T01:19:41.075 INFO:teuthology.orchestra.run.smithi028.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 410 kB/s | 133 kB 00:00 2024-04-25T01:19:41.096 INFO:teuthology.orchestra.run.smithi130.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 550 kB/s | 18 kB 00:00 2024-04-25T01:19:41.130 INFO:teuthology.orchestra.run.smithi130.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 333 kB/s | 11 kB 00:00 2024-04-25T01:19:41.172 INFO:teuthology.orchestra.run.smithi130.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 160 kB 00:00 2024-04-25T01:19:41.206 INFO:teuthology.orchestra.run.smithi130.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-04-25T01:19:41.256 INFO:teuthology.orchestra.run.smithi130.stdout:(58/152): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-04-25T01:19:41.315 INFO:teuthology.orchestra.run.smithi130.stdout:(59/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-04-25T01:19:41.323 INFO:teuthology.orchestra.run.smithi032.stdout:(150/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 3.8 MB/s | 3.1 MB 00:00 2024-04-25T01:19:41.423 INFO:teuthology.orchestra.run.smithi130.stdout:(60/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 1.4 MB/s | 10 MB 00:07 2024-04-25T01:19:41.448 INFO:teuthology.orchestra.run.smithi130.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 190 kB/s | 25 kB 00:00 2024-04-25T01:19:41.482 INFO:teuthology.orchestra.run.smithi130.stdout:(62/152): libtool-2.4.6-45.el9.x86_64.rpm 2.5 MB/s | 582 kB 00:00 2024-04-25T01:19:41.516 INFO:teuthology.orchestra.run.smithi130.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-04-25T01:19:41.541 INFO:teuthology.orchestra.run.smithi130.stdout:(64/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 405 kB/s | 47 kB 00:00 2024-04-25T01:19:41.566 INFO:teuthology.orchestra.run.smithi130.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-04-25T01:19:41.591 INFO:teuthology.orchestra.run.smithi130.stdout:(66/152): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-25T01:19:41.683 INFO:teuthology.orchestra.run.smithi130.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 38 MB/s | 4.4 MB 00:00 2024-04-25T01:19:41.725 INFO:teuthology.orchestra.run.smithi130.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-25T01:19:41.759 INFO:teuthology.orchestra.run.smithi130.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 911 kB 00:00 2024-04-25T01:19:41.784 INFO:teuthology.orchestra.run.smithi130.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-25T01:19:41.860 INFO:teuthology.orchestra.run.smithi130.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-04-25T01:19:41.901 INFO:teuthology.orchestra.run.smithi130.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 8.6 MB/s | 1.0 MB 00:00 2024-04-25T01:19:41.926 INFO:teuthology.orchestra.run.smithi130.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-25T01:19:42.007 INFO:teuthology.orchestra.run.smithi032.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 942 kB/s | 1.3 MB 00:01 2024-04-25T01:19:42.010 INFO:teuthology.orchestra.run.smithi130.stdout:(74/152): postgresql-server-13.14-1.el9.x86_64. 23 MB/s | 5.8 MB 00:00 2024-04-25T01:19:42.035 INFO:teuthology.orchestra.run.smithi130.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.2 MB/s | 244 kB 00:00 2024-04-25T01:19:42.061 INFO:teuthology.orchestra.run.smithi130.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-04-25T01:19:42.086 INFO:teuthology.orchestra.run.smithi130.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 509 kB/s | 26 kB 00:00 2024-04-25T01:19:42.111 INFO:teuthology.orchestra.run.smithi130.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-04-25T01:19:42.137 INFO:teuthology.orchestra.run.smithi130.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-04-25T01:19:42.164 INFO:teuthology.orchestra.run.smithi130.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 663 kB/s | 35 kB 00:00 2024-04-25T01:19:42.248 INFO:teuthology.orchestra.run.smithi130.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-04-25T01:19:42.273 INFO:teuthology.orchestra.run.smithi130.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-04-25T01:19:42.298 INFO:teuthology.orchestra.run.smithi130.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-04-25T01:19:42.315 INFO:teuthology.orchestra.run.smithi130.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.8 MB/s | 77 kB 00:00 2024-04-25T01:19:42.340 INFO:teuthology.orchestra.run.smithi130.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-04-25T01:19:42.366 INFO:teuthology.orchestra.run.smithi130.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-04-25T01:19:42.373 INFO:teuthology.orchestra.run.smithi032.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 1.7 MB/s | 2.4 MB 00:01 2024-04-25T01:19:42.377 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:19:42.377 INFO:teuthology.orchestra.run.smithi032.stdout:Total 9.8 MB/s | 220 MB 00:22 2024-04-25T01:19:42.391 INFO:teuthology.orchestra.run.smithi130.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-25T01:19:42.416 INFO:teuthology.orchestra.run.smithi130.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-25T01:19:42.442 INFO:teuthology.orchestra.run.smithi130.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-25T01:19:42.467 INFO:teuthology.orchestra.run.smithi130.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-25T01:19:42.534 INFO:teuthology.orchestra.run.smithi130.stdout:(91/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 5.1 MB 00:00 2024-04-25T01:19:42.559 INFO:teuthology.orchestra.run.smithi130.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 453 kB/s | 42 kB 00:00 2024-04-25T01:19:42.584 INFO:teuthology.orchestra.run.smithi130.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 882 kB/s | 44 kB 00:00 2024-04-25T01:19:42.610 INFO:teuthology.orchestra.run.smithi130.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-25T01:19:42.635 INFO:teuthology.orchestra.run.smithi130.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 715 kB/s | 36 kB 00:00 2024-04-25T01:19:42.660 INFO:teuthology.orchestra.run.smithi130.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 946 kB/s | 47 kB 00:00 2024-04-25T01:19:42.685 INFO:teuthology.orchestra.run.smithi130.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-04-25T01:19:42.711 INFO:teuthology.orchestra.run.smithi130.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-25T01:19:42.736 INFO:teuthology.orchestra.run.smithi130.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-25T01:19:42.758 INFO:teuthology.orchestra.run.smithi028.stdout:(70/152): openblas-openmp-0.3.21-2.el9.x86_64.r 1.0 MB/s | 4.7 MB 00:04 2024-04-25T01:19:42.761 INFO:teuthology.orchestra.run.smithi130.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-25T01:19:42.786 INFO:teuthology.orchestra.run.smithi130.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 891 kB/s | 45 kB 00:00 2024-04-25T01:19:42.825 INFO:teuthology.orchestra.run.smithi028.stdout:(71/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 994 kB/s | 4.4 MB 00:04 2024-04-25T01:19:42.879 INFO:teuthology.orchestra.run.smithi130.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-04-25T01:19:43.029 INFO:teuthology.orchestra.run.smithi130.stdout:(103/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-04-25T01:19:43.109 INFO:teuthology.orchestra.run.smithi028.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 309 kB/s | 87 kB 00:00 2024-04-25T01:19:43.137 INFO:teuthology.orchestra.run.smithi130.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.4 MB/s | 863 kB 00:00 2024-04-25T01:19:43.396 INFO:teuthology.orchestra.run.smithi130.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-04-25T01:19:43.513 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-25T01:19:43.571 INFO:teuthology.orchestra.run.smithi130.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 570 kB/s | 309 kB 00:00 2024-04-25T01:19:43.626 INFO:teuthology.orchestra.run.smithi028.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 1.2 MB/s | 1.0 MB 00:00 2024-04-25T01:19:43.630 INFO:teuthology.orchestra.run.smithi130.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 730 kB/s | 548 kB 00:00 2024-04-25T01:19:43.672 INFO:teuthology.orchestra.run.smithi130.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 259 kB/s | 26 kB 00:00 2024-04-25T01:19:43.673 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-25T01:19:43.673 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-25T01:19:43.739 INFO:teuthology.orchestra.run.smithi130.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 446 kB/s | 49 kB 00:00 2024-04-25T01:19:43.768 INFO:teuthology.orchestra.run.smithi028.stdout:(74/152): postgresql-server-13.14-1.el9.x86_64. 2.1 MB/s | 5.8 MB 00:02 2024-04-25T01:19:43.781 INFO:teuthology.orchestra.run.smithi130.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 619 kB/s | 67 kB 00:00 2024-04-25T01:19:43.848 INFO:teuthology.orchestra.run.smithi130.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-04-25T01:19:43.968 INFO:teuthology.orchestra.run.smithi028.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 714 kB/s | 244 kB 00:00 2024-04-25T01:19:43.998 INFO:teuthology.orchestra.run.smithi130.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 3.6 MB/s | 548 kB 00:00 2024-04-25T01:19:44.023 INFO:teuthology.orchestra.run.smithi130.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 838 kB 00:00 2024-04-25T01:19:44.110 INFO:teuthology.orchestra.run.smithi028.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 725 kB/s | 248 kB 00:00 2024-04-25T01:19:44.127 INFO:teuthology.orchestra.run.smithi028.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 162 kB/s | 26 kB 00:00 2024-04-25T01:19:44.174 INFO:teuthology.orchestra.run.smithi130.stdout:(114/152): libarrow-9.0.0-10.el9.x86_64.rpm 5.7 MB/s | 4.4 MB 00:00 2024-04-25T01:19:44.199 INFO:teuthology.orchestra.run.smithi130.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 217 kB/s | 43 kB 00:00 2024-04-25T01:19:44.224 INFO:teuthology.orchestra.run.smithi130.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 160 kB/s | 32 kB 00:00 2024-04-25T01:19:44.275 INFO:teuthology.orchestra.run.smithi130.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-04-25T01:19:44.300 INFO:teuthology.orchestra.run.smithi130.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-04-25T01:19:44.310 INFO:teuthology.orchestra.run.smithi028.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 95 kB/s | 19 kB 00:00 2024-04-25T01:19:44.342 INFO:teuthology.orchestra.run.smithi130.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-04-25T01:19:44.375 INFO:teuthology.orchestra.run.smithi130.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.9 MB/s | 303 kB 00:00 2024-04-25T01:19:44.392 INFO:teuthology.orchestra.run.smithi130.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 383 kB/s | 35 kB 00:00 2024-04-25T01:19:44.451 INFO:teuthology.orchestra.run.smithi130.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 1.8 MB/s | 206 kB 00:00 2024-04-25T01:19:44.510 INFO:teuthology.orchestra.run.smithi028.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 174 kB/s | 35 kB 00:00 2024-04-25T01:19:44.518 INFO:teuthology.orchestra.run.smithi130.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-04-25T01:19:44.543 INFO:teuthology.orchestra.run.smithi130.stdout:(124/152): python3-grpcio-tools-1.46.7-10.el9.x 959 kB/s | 144 kB 00:00 2024-04-25T01:19:44.544 INFO:teuthology.orchestra.run.smithi028.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 412 kB/s | 172 kB 00:00 2024-04-25T01:19:44.560 INFO:teuthology.orchestra.run.smithi130.stdout:(125/152): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-04-25T01:19:44.619 INFO:teuthology.orchestra.run.smithi130.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 175 kB/s | 18 kB 00:00 2024-04-25T01:19:44.644 INFO:teuthology.orchestra.run.smithi130.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 230 kB/s | 23 kB 00:00 2024-04-25T01:19:44.669 INFO:teuthology.orchestra.run.smithi130.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 178 kB/s | 19 kB 00:00 2024-04-25T01:19:44.720 INFO:teuthology.orchestra.run.smithi130.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-04-25T01:19:44.770 INFO:teuthology.orchestra.run.smithi130.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 462 kB/s | 46 kB 00:00 2024-04-25T01:19:44.812 INFO:teuthology.orchestra.run.smithi130.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 857 kB/s | 79 kB 00:00 2024-04-25T01:19:44.846 INFO:teuthology.orchestra.run.smithi130.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 5.1 MB/s | 1.0 MB 00:00 2024-04-25T01:19:44.871 INFO:teuthology.orchestra.run.smithi130.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 574 kB/s | 58 kB 00:00 2024-04-25T01:19:44.895 INFO:teuthology.orchestra.run.smithi028.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 715 kB/s | 250 kB 00:00 2024-04-25T01:19:44.913 INFO:teuthology.orchestra.run.smithi130.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-04-25T01:19:44.946 INFO:teuthology.orchestra.run.smithi130.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 163 kB/s | 16 kB 00:00 2024-04-25T01:19:44.971 INFO:teuthology.orchestra.run.smithi130.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 896 kB/s | 90 kB 00:00 2024-04-25T01:19:45.014 INFO:teuthology.orchestra.run.smithi130.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 281 kB/s | 28 kB 00:00 2024-04-25T01:19:45.047 INFO:teuthology.orchestra.run.smithi130.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 305 kB/s | 31 kB 00:00 2024-04-25T01:19:45.081 INFO:teuthology.orchestra.run.smithi130.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-04-25T01:19:45.115 INFO:teuthology.orchestra.run.smithi130.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 586 kB/s | 59 kB 00:00 2024-04-25T01:19:45.148 INFO:teuthology.orchestra.run.smithi130.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-04-25T01:19:45.182 INFO:teuthology.orchestra.run.smithi130.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 508 kB/s | 51 kB 00:00 2024-04-25T01:19:45.215 INFO:teuthology.orchestra.run.smithi130.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-25T01:19:45.249 INFO:teuthology.orchestra.run.smithi130.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 893 kB/s | 90 kB 00:00 2024-04-25T01:19:45.278 INFO:teuthology.orchestra.run.smithi028.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 580 kB/s | 222 kB 00:00 2024-04-25T01:19:45.308 INFO:teuthology.orchestra.run.smithi130.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-04-25T01:19:45.333 INFO:teuthology.orchestra.run.smithi130.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 170 kB/s | 20 kB 00:00 2024-04-25T01:19:45.354 INFO:teuthology.orchestra.run.smithi028.stdout:(83/152): python3-babel-2.9.1-2.el9.noarch.rpm 2.7 MB/s | 6.0 MB 00:02 2024-04-25T01:19:45.358 INFO:teuthology.orchestra.run.smithi130.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-04-25T01:19:45.508 INFO:teuthology.orchestra.run.smithi130.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-04-25T01:19:45.527 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-25T01:19:45.527 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-25T01:19:45.562 INFO:teuthology.orchestra.run.smithi028.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 272 kB/s | 77 kB 00:00 2024-04-25T01:19:45.646 INFO:teuthology.orchestra.run.smithi028.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 145 kB/s | 42 kB 00:00 2024-04-25T01:19:45.650 INFO:teuthology.orchestra.run.smithi130.stdout:(149/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 11 MB/s | 3.1 MB 00:00 2024-04-25T01:19:46.096 INFO:teuthology.orchestra.run.smithi028.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 353 kB/s | 159 kB 00:00 2024-04-25T01:19:46.121 INFO:teuthology.orchestra.run.smithi028.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 482 kB/s | 269 kB 00:00 2024-04-25T01:19:46.297 INFO:teuthology.orchestra.run.smithi028.stdout:(88/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.9 MB/s | 5.1 MB 00:01 2024-04-25T01:19:46.309 INFO:teuthology.orchestra.run.smithi130.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 3.6 MB/s | 2.4 MB 00:00 2024-04-25T01:19:46.338 INFO:teuthology.orchestra.run.smithi028.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 235 kB/s | 51 kB 00:00 2024-04-25T01:19:46.434 INFO:teuthology.orchestra.run.smithi130.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 1.5 MB/s | 1.3 MB 00:00 2024-04-25T01:19:46.480 INFO:teuthology.orchestra.run.smithi028.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 292 kB/s | 54 kB 00:00 2024-04-25T01:19:46.547 INFO:teuthology.orchestra.run.smithi028.stdout:(91/152): python3-pyasn1-modules-0.4.8-6.el9.no 618 kB/s | 279 kB 00:00 2024-04-25T01:19:46.664 INFO:teuthology.orchestra.run.smithi028.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 227 kB/s | 42 kB 00:00 2024-04-25T01:19:46.739 INFO:teuthology.orchestra.run.smithi028.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 230 kB/s | 44 kB 00:00 2024-04-25T01:19:46.882 INFO:teuthology.orchestra.run.smithi028.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 253 kB/s | 36 kB 00:00 2024-04-25T01:19:47.032 INFO:teuthology.orchestra.run.smithi028.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 601 kB/s | 221 kB 00:00 2024-04-25T01:19:47.057 INFO:teuthology.orchestra.run.smithi028.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 270 kB/s | 47 kB 00:00 2024-04-25T01:19:47.258 INFO:teuthology.orchestra.run.smithi028.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 289 kB/s | 58 kB 00:00 2024-04-25T01:19:47.474 INFO:teuthology.orchestra.run.smithi028.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 433 kB/s | 94 kB 00:00 2024-04-25T01:19:47.541 INFO:teuthology.orchestra.run.smithi028.stdout:(99/152): socat-1.7.4.1-5.el9.x86_64.rpm 599 kB/s | 305 kB 00:00 2024-04-25T01:19:47.650 INFO:teuthology.orchestra.run.smithi028.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 363 kB/s | 64 kB 00:00 2024-04-25T01:19:47.834 INFO:teuthology.orchestra.run.smithi130.stdout:(152/152): librados2-19.0.0-2445.gd2c45d97.el9. 1.4 MB/s | 3.4 MB 00:02 2024-04-25T01:19:47.838 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:19:47.839 INFO:teuthology.orchestra.run.smithi130.stdout:Total 13 MB/s | 220 MB 00:16 2024-04-25T01:19:47.891 INFO:teuthology.orchestra.run.smithi028.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 93 kB/s | 22 kB 00:00 2024-04-25T01:19:47.926 INFO:teuthology.orchestra.run.smithi028.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 116 kB/s | 45 kB 00:00 2024-04-25T01:19:48.026 INFO:teuthology.orchestra.run.smithi028.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 6.3 MB/s | 863 kB 00:00 2024-04-25T01:19:48.301 INFO:teuthology.orchestra.run.smithi028.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.4 MB/s | 548 kB 00:00 2024-04-25T01:19:48.385 INFO:teuthology.orchestra.run.smithi028.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 234 kB/s | 19 kB 00:00 2024-04-25T01:19:48.418 INFO:teuthology.orchestra.run.smithi028.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 787 kB/s | 309 kB 00:00 2024-04-25T01:19:48.502 INFO:teuthology.orchestra.run.smithi028.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 312 kB/s | 26 kB 00:00 2024-04-25T01:19:48.594 INFO:teuthology.orchestra.run.smithi028.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 531 kB/s | 49 kB 00:00 2024-04-25T01:19:48.711 INFO:teuthology.orchestra.run.smithi028.stdout:(109/152): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-04-25T01:19:48.736 INFO:teuthology.orchestra.run.smithi028.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 474 kB/s | 67 kB 00:00 2024-04-25T01:19:48.803 INFO:teuthology.orchestra.run.smithi028.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-04-25T01:19:48.879 INFO:teuthology.orchestra.run.smithi028.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-04-25T01:19:48.920 INFO:teuthology.orchestra.run.smithi028.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-04-25T01:19:48.955 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-04-25T01:19:48.962 INFO:teuthology.orchestra.run.smithi028.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 519 kB/s | 43 kB 00:00 2024-04-25T01:19:49.013 INFO:teuthology.orchestra.run.smithi028.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 350 kB/s | 32 kB 00:00 2024-04-25T01:19:49.071 INFO:teuthology.orchestra.run.smithi028.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-04-25T01:19:49.099 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-04-25T01:19:49.099 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-04-25T01:19:49.105 INFO:teuthology.orchestra.run.smithi028.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-04-25T01:19:49.155 INFO:teuthology.orchestra.run.smithi028.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-04-25T01:19:49.197 INFO:teuthology.orchestra.run.smithi028.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-04-25T01:19:49.222 INFO:teuthology.orchestra.run.smithi028.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 528 kB/s | 35 kB 00:00 2024-04-25T01:19:49.306 INFO:teuthology.orchestra.run.smithi028.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 1.9 MB/s | 206 kB 00:00 2024-04-25T01:19:49.407 INFO:teuthology.orchestra.run.smithi028.stdout:(122/152): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-04-25T01:19:49.432 INFO:teuthology.orchestra.run.smithi028.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-25T01:19:49.482 INFO:teuthology.orchestra.run.smithi028.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-04-25T01:19:49.507 INFO:teuthology.orchestra.run.smithi028.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 235 kB/s | 18 kB 00:00 2024-04-25T01:19:49.566 INFO:teuthology.orchestra.run.smithi028.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 277 kB/s | 23 kB 00:00 2024-04-25T01:19:49.591 INFO:teuthology.orchestra.run.smithi028.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 233 kB/s | 19 kB 00:00 2024-04-25T01:19:49.641 INFO:teuthology.orchestra.run.smithi028.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-04-25T01:19:49.717 INFO:teuthology.orchestra.run.smithi028.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 8.2 MB/s | 1.0 MB 00:00 2024-04-25T01:19:49.751 INFO:teuthology.orchestra.run.smithi028.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 425 kB/s | 46 kB 00:00 2024-04-25T01:19:49.784 INFO:teuthology.orchestra.run.smithi028.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.1 MB/s | 79 kB 00:00 2024-04-25T01:19:49.843 INFO:teuthology.orchestra.run.smithi028.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 629 kB/s | 58 kB 00:00 2024-04-25T01:19:49.885 INFO:teuthology.orchestra.run.smithi028.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-04-25T01:19:49.927 INFO:teuthology.orchestra.run.smithi028.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 197 kB/s | 16 kB 00:00 2024-04-25T01:19:50.103 INFO:teuthology.orchestra.run.smithi028.stdout:(135/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.3 MB/s | 16 MB 00:03 2024-04-25T01:19:50.127 INFO:teuthology.orchestra.run.smithi028.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 141 kB/s | 28 kB 00:00 2024-04-25T01:19:50.152 INFO:teuthology.orchestra.run.smithi028.stdout:(137/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 336 kB/s | 90 kB 00:00 2024-04-25T01:19:50.219 INFO:teuthology.orchestra.run.smithi028.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-04-25T01:19:50.244 INFO:teuthology.orchestra.run.smithi028.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 644 kB/s | 59 kB 00:00 2024-04-25T01:19:50.295 INFO:teuthology.orchestra.run.smithi028.stdout:(140/152): python3-repoze-lru-0.7-16.el9.noarch 160 kB/s | 31 kB 00:00 2024-04-25T01:19:50.320 INFO:teuthology.orchestra.run.smithi028.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-04-25T01:19:50.354 INFO:teuthology.orchestra.run.smithi028.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 468 kB/s | 51 kB 00:00 2024-04-25T01:19:50.404 INFO:teuthology.orchestra.run.smithi028.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-04-25T01:19:50.479 INFO:teuthology.orchestra.run.smithi028.stdout:(144/152): python3-zc-lockfile-2.0-10.el9.noarc 266 kB/s | 20 kB 00:00 2024-04-25T01:19:50.513 INFO:teuthology.orchestra.run.smithi028.stdout:(145/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.0 MB/s | 230 kB 00:00 2024-04-25T01:19:50.563 INFO:teuthology.orchestra.run.smithi028.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 2.0 MB/s | 427 kB 00:00 2024-04-25T01:19:50.605 INFO:teuthology.orchestra.run.smithi028.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-04-25T01:19:50.739 INFO:teuthology.orchestra.run.smithi028.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2024-04-25T01:19:50.925 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-04-25T01:19:50.926 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-04-25T01:19:50.964 INFO:teuthology.orchestra.run.smithi028.stdout:(149/152): librados2-19.0.0-2445.gd2c45d97.el9. 8.5 MB/s | 3.4 MB 00:00 2024-04-25T01:19:51.014 INFO:teuthology.orchestra.run.smithi028.stdout:(150/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 7.7 MB/s | 3.1 MB 00:00 2024-04-25T01:19:51.373 INFO:teuthology.orchestra.run.smithi028.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 2.1 MB/s | 1.3 MB 00:00 2024-04-25T01:19:51.548 INFO:teuthology.orchestra.run.smithi028.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 4.1 MB/s | 2.4 MB 00:00 2024-04-25T01:19:51.551 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:19:51.551 INFO:teuthology.orchestra.run.smithi028.stdout:Total 6.7 MB/s | 220 MB 00:32 2024-04-25T01:19:51.857 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-25T01:19:51.969 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T01:19:52.008 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T01:19:52.062 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T01:19:52.095 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T01:19:52.121 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T01:19:52.164 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T01:19:52.519 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T01:19:52.630 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:19:52.689 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-25T01:19:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:19:52.806 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:19:52.848 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-25T01:19:52.848 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-25T01:19:52.899 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:19:52.935 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:19:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T01:19:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T01:19:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:19:53.265 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:19:53.295 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:19:53.358 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:19:53.379 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:19:53.401 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:19:53.614 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:19:53.646 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:19:53.755 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T01:19:53.852 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T01:19:53.895 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T01:19:53.931 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T01:19:54.005 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T01:19:54.046 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T01:19:54.099 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T01:19:54.139 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T01:19:54.167 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T01:19:54.238 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T01:19:54.299 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T01:19:54.345 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T01:19:54.380 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T01:19:54.459 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T01:19:54.491 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T01:19:54.535 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T01:19:54.563 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T01:19:54.644 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T01:19:54.695 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T01:19:54.719 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-25T01:19:54.720 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-25T01:19:54.774 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T01:19:54.917 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T01:19:54.974 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T01:19:55.533 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T01:19:55.622 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T01:19:55.656 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T01:19:55.699 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T01:19:55.737 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T01:19:55.780 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T01:19:55.930 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T01:19:55.982 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T01:19:56.047 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T01:19:56.138 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T01:19:56.175 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T01:19:56.212 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T01:19:56.260 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T01:19:56.295 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T01:19:56.338 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T01:19:56.381 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T01:19:56.526 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T01:19:56.569 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T01:19:56.607 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T01:19:56.658 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T01:19:56.763 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T01:19:56.807 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T01:19:56.848 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T01:19:56.881 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T01:19:56.915 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T01:19:56.927 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-04-25T01:19:56.994 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T01:19:57.012 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T01:19:57.051 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T01:19:57.104 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T01:19:57.121 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T01:19:57.147 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T01:19:57.164 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T01:19:57.172 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T01:19:57.192 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T01:19:57.215 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T01:19:57.227 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T01:19:57.274 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T01:19:57.323 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T01:19:57.352 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T01:19:57.408 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T01:19:57.445 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T01:19:57.488 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T01:19:57.521 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T01:19:57.536 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T01:19:57.612 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T01:19:57.640 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:19:57.661 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T01:19:57.692 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T01:19:57.748 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T01:19:57.761 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:19:57.799 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:19:57.868 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T01:19:57.882 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:19:57.912 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:19:57.928 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T01:19:58.058 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T01:19:58.101 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T01:19:58.129 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:19:58.231 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:19:58.255 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:19:58.318 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:19:58.339 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:19:58.360 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:19:58.548 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:19:58.580 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:19:58.687 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T01:19:58.696 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T01:19:58.749 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T01:19:58.783 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T01:19:58.784 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T01:19:58.830 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T01:19:58.864 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T01:19:58.887 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T01:19:58.929 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T01:19:58.961 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T01:19:58.963 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T01:19:59.007 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T01:19:59.039 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T01:19:59.057 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T01:19:59.067 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T01:19:59.094 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T01:19:59.137 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T01:19:59.190 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T01:19:59.220 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T01:19:59.256 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T01:19:59.276 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T01:19:59.308 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T01:19:59.332 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T01:19:59.387 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T01:19:59.584 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T01:19:59.644 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T01:19:59.672 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T01:19:59.745 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T01:19:59.787 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T01:19:59.862 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T01:19:59.994 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T01:20:00.056 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T01:20:00.064 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T01:20:00.185 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T01:20:00.573 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T01:20:00.660 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T01:20:00.689 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T01:20:00.731 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T01:20:00.755 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-25T01:20:00.770 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T01:20:00.811 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T01:20:00.850 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T01:20:00.893 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T01:20:00.959 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T01:20:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T01:20:01.005 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T01:20:01.014 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T01:20:01.030 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T01:20:01.069 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T01:20:01.073 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T01:20:01.142 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T01:20:01.175 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T01:20:01.210 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T01:20:01.257 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T01:20:01.293 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T01:20:01.336 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T01:20:01.378 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T01:20:01.421 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T01:20:01.473 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T01:20:01.514 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T01:20:01.531 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:20:01.546 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T01:20:01.567 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T01:20:01.584 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T01:20:01.598 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T01:20:01.616 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T01:20:01.639 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T01:20:01.647 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:20:01.674 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:20:01.734 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T01:20:01.749 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:20:01.772 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T01:20:01.778 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:20:01.813 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T01:20:01.838 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T01:20:01.879 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T01:20:01.936 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T01:20:01.943 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T01:20:01.954 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T01:20:01.984 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T01:20:02.011 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:20:02.021 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:20:02.090 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T01:20:02.104 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:20:02.120 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T01:20:02.124 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:20:02.134 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T01:20:02.140 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T01:20:02.154 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:20:02.171 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T01:20:02.174 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T01:20:02.213 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T01:20:02.216 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:20:02.238 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:20:02.245 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T01:20:02.269 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:20:02.275 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T01:20:02.310 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T01:20:02.343 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T01:20:02.393 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T01:20:02.428 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T01:20:02.481 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:20:02.513 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:20:02.514 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T01:20:02.558 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T01:20:02.598 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T01:20:02.624 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T01:20:02.645 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T01:20:02.675 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T01:20:02.707 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:20:02.743 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T01:20:02.778 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T01:20:02.779 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:20:02.791 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T01:20:02.809 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T01:20:02.820 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T01:20:02.856 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T01:20:02.931 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T01:20:02.971 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T01:20:03.015 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T01:20:03.047 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T01:20:03.074 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T01:20:03.147 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T01:20:03.206 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T01:20:03.236 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T01:20:03.271 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T01:20:03.350 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T01:20:03.382 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T01:20:03.427 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T01:20:03.454 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T01:20:03.506 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T01:20:03.526 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T01:20:03.555 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T01:20:03.569 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T01:20:03.589 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T01:20:03.647 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T01:20:03.687 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T01:20:03.747 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T01:20:03.804 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T01:20:03.833 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T01:20:03.866 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T01:20:03.872 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T01:20:04.042 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T01:20:04.072 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T01:20:04.150 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T01:20:04.429 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T01:20:04.519 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T01:20:04.554 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T01:20:04.597 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T01:20:04.635 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T01:20:04.677 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T01:20:04.794 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T01:20:04.851 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T01:20:04.896 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T01:20:04.903 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T01:20:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T01:20:05.035 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T01:20:05.071 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T01:20:05.108 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T01:20:05.155 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T01:20:05.171 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:20:05.191 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T01:20:05.214 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:20:05.234 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T01:20:05.276 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T01:20:05.419 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T01:20:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T01:20:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T01:20:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T01:20:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T01:20:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T01:20:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T01:20:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T01:20:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T01:20:05.783 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:20:05.826 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:20:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T01:20:05.918 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:20:05.918 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T01:20:05.918 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T01:20:05.918 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-25T01:20:05.952 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:20:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T01:20:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T01:20:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T01:20:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T01:20:06.132 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T01:20:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T01:20:06.202 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T01:20:06.221 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T01:20:06.246 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T01:20:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T01:20:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T01:20:06.448 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T01:20:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T01:20:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T01:20:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T01:20:06.557 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T01:20:06.601 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:20:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T01:20:06.692 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:20:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T01:20:06.723 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T01:20:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T01:20:06.759 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T01:20:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T01:20:06.885 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T01:20:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T01:20:07.229 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T01:20:07.254 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:20:07.325 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:20:07.356 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T01:20:07.695 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T01:20:07.742 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T01:20:07.776 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T01:20:07.878 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T01:20:07.926 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T01:20:08.016 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T01:20:08.052 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T01:20:08.237 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T01:20:08.275 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T01:20:08.355 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T01:20:09.033 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T01:20:09.151 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T01:20:09.635 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:20:09.669 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:20:10.208 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:20:10.240 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:20:10.335 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:20:10.335 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T01:20:10.335 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T01:20:10.335 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-25T01:20:10.366 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:20:10.441 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T01:20:10.512 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T01:20:10.540 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T01:20:10.573 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T01:20:10.901 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T01:20:10.936 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:20:11.028 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:20:11.058 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T01:20:11.094 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T01:20:11.589 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T01:20:11.614 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:20:11.677 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:20:11.707 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T01:20:14.047 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:20:14.079 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:20:14.646 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:20:14.682 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:20:14.765 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:20:14.765 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T01:20:14.765 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T01:20:14.765 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:20:14.791 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:20:17.843 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:20:17.843 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /sys 2024-04-25T01:20:17.843 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /proc 2024-04-25T01:20:17.843 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /mnt 2024-04-25T01:20:17.843 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /var/tmp 2024-04-25T01:20:17.843 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /home 2024-04-25T01:20:17.844 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /root 2024-04-25T01:20:17.844 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /tmp 2024-04-25T01:20:17.844 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-25T01:20:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:20:18.032 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:20:18.033 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:18.033 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T01:20:18.033 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-25T01:20:19.561 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:20:19.589 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:20:19.589 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:19.589 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T01:20:19.589 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:20:19.589 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:20:19.589 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-25T01:20:19.829 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:20:19.853 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:20:19.854 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:19.854 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T01:20:19.854 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:20:19.854 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:20:19.854 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-25T01:20:20.313 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:20:20.337 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:20:20.337 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:20.337 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T01:20:20.337 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:20:20.337 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:20:20.337 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-25T01:20:20.461 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T01:20:20.518 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T01:20:20.591 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T01:20:20.792 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T01:20:20.848 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T01:20:20.928 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T01:20:21.019 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:20:21.170 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:20:21.215 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T01:20:21.247 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T01:20:21.274 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T01:20:21.717 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:20:21.748 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:20:21.780 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T01:20:21.845 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T01:20:21.857 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:20:21.857 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /sys 2024-04-25T01:20:21.857 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /proc 2024-04-25T01:20:21.857 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /mnt 2024-04-25T01:20:21.857 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /var/tmp 2024-04-25T01:20:21.857 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /home 2024-04-25T01:20:21.857 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /root 2024-04-25T01:20:21.857 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /tmp 2024-04-25T01:20:21.857 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-25T01:20:21.871 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:20:22.020 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:20:22.036 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:20:22.036 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:22.036 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T01:20:22.036 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-25T01:20:22.957 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:20:22.998 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:20:23.524 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:20:23.550 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:20:23.550 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:23.550 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T01:20:23.550 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:20:23.550 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:20:23.551 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-25T01:20:23.784 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:20:23.807 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:20:23.807 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:23.807 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T01:20:23.807 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:20:23.807 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:20:23.807 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-25T01:20:24.011 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:20:24.049 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:20:24.158 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:20:24.251 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:20:24.274 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:20:24.274 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:24.274 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T01:20:24.274 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:20:24.274 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:20:24.274 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-25T01:20:24.327 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T01:20:24.368 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:20:24.389 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T01:20:24.392 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:20:24.392 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:24.393 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T01:20:24.393 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:20:24.393 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:20:24.393 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-25T01:20:24.439 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T01:20:24.443 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:20:24.455 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:20:24.494 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T01:20:24.556 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T01:20:24.602 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T01:20:24.680 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T01:20:24.727 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T01:20:24.805 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T01:20:24.894 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:20:25.025 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:20:25.061 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T01:20:25.093 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T01:20:25.121 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T01:20:25.486 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:20:25.511 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:20:25.543 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T01:20:25.545 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T01:20:25.580 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:20:25.604 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:20:25.604 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:25.604 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T01:20:25.604 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:20:25.604 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:20:25.605 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-25T01:20:25.605 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T01:20:25.634 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:20:25.655 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:20:25.679 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:20:25.679 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:25.679 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T01:20:25.679 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-25T01:20:25.731 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:20:25.754 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:20:25.754 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:25.755 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T01:20:25.755 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:20:25.755 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:20:25.755 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-25T01:20:26.073 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:20:26.098 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:20:26.098 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:26.098 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T01:20:26.098 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:20:26.098 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:20:26.098 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-25T01:20:26.660 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:20:26.703 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:20:26.910 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:20:26.910 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-04-25T01:20:26.910 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-04-25T01:20:26.910 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-04-25T01:20:26.910 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-04-25T01:20:26.910 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-04-25T01:20:26.910 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-04-25T01:20:26.910 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-04-25T01:20:26.911 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:20:27.098 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:20:27.115 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:20:27.115 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:27.115 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T01:20:27.115 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:20:27.661 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:20:27.695 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:20:27.796 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:20:27.958 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T01:20:27.997 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:20:28.020 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:20:28.020 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:28.020 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T01:20:28.020 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:20:28.020 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:20:28.020 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-25T01:20:28.064 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:20:28.076 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:20:28.175 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T01:20:28.229 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T01:20:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:20:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:20:28.669 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:28.669 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T01:20:28.669 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:20:28.669 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:20:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:20:28.908 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:20:28.932 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:20:28.932 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:28.932 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T01:20:28.932 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:20:28.932 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:20:28.932 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:20:29.128 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T01:20:29.167 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:20:29.190 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:20:29.190 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:29.190 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T01:20:29.190 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:20:29.190 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:20:29.190 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-25T01:20:29.226 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:20:29.248 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:20:29.248 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:29.248 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T01:20:29.248 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-25T01:20:29.293 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:20:29.316 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:20:29.316 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:29.316 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T01:20:29.316 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:20:29.316 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:20:29.316 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-25T01:20:29.400 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:20:29.424 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:20:29.424 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:29.424 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T01:20:29.425 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:20:29.425 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:20:29.425 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:20:29.523 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T01:20:29.581 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T01:20:29.619 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:20:29.636 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T01:20:29.641 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:20:29.641 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:29.641 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T01:20:29.641 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:20:29.641 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:20:29.641 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-25T01:20:29.829 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T01:20:29.877 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T01:20:29.957 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T01:20:30.046 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:20:30.196 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:20:30.234 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T01:20:30.267 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T01:20:30.294 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T01:20:30.746 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:20:30.776 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:20:30.808 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T01:20:30.845 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T01:20:30.873 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T01:20:30.886 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T01:20:30.899 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:20:30.928 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T01:20:30.959 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T01:20:30.986 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T01:20:31.180 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T01:20:31.232 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T01:20:31.262 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T01:20:31.856 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:20:31.893 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:20:31.915 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:20:31.986 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:20:32.017 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:20:32.843 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T01:20:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T01:20:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T01:20:33.031 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:20:33.051 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:20:33.052 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T01:20:33.087 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T01:20:33.087 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:20:33.159 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:20:33.188 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:20:33.237 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:20:33.238 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:20:33.327 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:20:33.330 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T01:20:33.370 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:20:33.394 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:20:33.394 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:33.394 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T01:20:33.394 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:20:33.395 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:20:33.395 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:20:33.397 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:20:33.446 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:20:33.457 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:20:33.558 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T01:20:33.612 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T01:20:34.148 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T01:20:34.189 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T01:20:34.230 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T01:20:34.262 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T01:20:34.289 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T01:20:34.471 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T01:20:34.519 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T01:20:34.541 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T01:20:34.565 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T01:20:34.598 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:20:34.622 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:20:34.622 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:34.622 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T01:20:34.622 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:20:34.622 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:20:34.622 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:20:34.665 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:20:34.689 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:20:34.689 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:34.689 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T01:20:34.689 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:20:34.733 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:20:34.756 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:20:34.756 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:34.756 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T01:20:34.757 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:20:34.757 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:20:34.757 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:20:35.083 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:20:35.108 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:20:35.108 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:20:35.108 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T01:20:35.108 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:20:35.108 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:20:35.108 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:20:35.161 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:20:35.205 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:20:35.225 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:20:36.065 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T01:20:36.152 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T01:20:36.193 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T01:20:36.277 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T01:20:36.306 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T01:20:36.307 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:20:36.391 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:20:36.423 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:20:36.424 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:20:36.521 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:20:36.590 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:20:39.298 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:20:39.298 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T01:20:39.298 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T01:20:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T01:20:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T01:20:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T01:20:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T01:20:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T01:20:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:20:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:20:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:20:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T01:20:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T01:20:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:20:39.299 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T01:20:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T01:20:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T01:20:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T01:20:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T01:20:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T01:20:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T01:20:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T01:20:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T01:20:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T01:20:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T01:20:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T01:20:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T01:20:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T01:20:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T01:20:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T01:20:39.300 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T01:20:39.301 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T01:20:39.301 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T01:20:39.312 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T01:20:39.312 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T01:20:39.312 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T01:20:39.312 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T01:20:39.312 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T01:20:39.313 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T01:20:39.313 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T01:20:39.313 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T01:20:39.313 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T01:20:39.313 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T01:20:39.313 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T01:20:39.313 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T01:20:39.313 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T01:20:39.313 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T01:20:39.313 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T01:20:39.313 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T01:20:39.313 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T01:20:39.313 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T01:20:39.314 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T01:20:39.314 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T01:20:39.314 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T01:20:39.314 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T01:20:39.314 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T01:20:39.314 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T01:20:39.314 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T01:20:39.314 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T01:20:39.314 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T01:20:39.314 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T01:20:39.315 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T01:20:39.315 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T01:20:39.315 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T01:20:39.315 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T01:20:39.315 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T01:20:39.315 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T01:20:39.315 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T01:20:39.315 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T01:20:39.315 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T01:20:39.315 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T01:20:39.315 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T01:20:39.315 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T01:20:39.315 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T01:20:39.315 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T01:20:39.316 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T01:20:39.316 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T01:20:39.316 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T01:20:39.316 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T01:20:39.316 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T01:20:39.316 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T01:20:39.316 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T01:20:39.316 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T01:20:39.316 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T01:20:39.316 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T01:20:39.316 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T01:20:39.317 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T01:20:39.317 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T01:20:39.317 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T01:20:39.317 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T01:20:39.317 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T01:20:39.317 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T01:20:39.317 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T01:20:39.317 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T01:20:39.317 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T01:20:39.317 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T01:20:39.317 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T01:20:39.317 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T01:20:39.317 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T01:20:39.317 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T01:20:39.318 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T01:20:39.318 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T01:20:39.318 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T01:20:39.318 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T01:20:39.318 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T01:20:39.318 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T01:20:39.318 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T01:20:39.318 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T01:20:39.318 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T01:20:39.318 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T01:20:39.319 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T01:20:39.319 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T01:20:39.319 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T01:20:39.319 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T01:20:39.319 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T01:20:39.319 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T01:20:39.319 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T01:20:39.319 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T01:20:39.319 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:20:39.319 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:20:39.319 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T01:20:39.319 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T01:20:39.320 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T01:20:39.320 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T01:20:39.320 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T01:20:39.320 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T01:20:39.320 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T01:20:39.320 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T01:20:39.320 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T01:20:39.320 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T01:20:39.320 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T01:20:39.320 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T01:20:39.320 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T01:20:39.320 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T01:20:39.320 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T01:20:39.321 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T01:20:39.321 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T01:20:39.321 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T01:20:39.321 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T01:20:39.321 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T01:20:39.321 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T01:20:39.321 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T01:20:39.321 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T01:20:39.321 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T01:20:39.321 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T01:20:39.321 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T01:20:39.322 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T01:20:39.322 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T01:20:39.322 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T01:20:39.322 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T01:20:39.322 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T01:20:39.322 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T01:20:39.322 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T01:20:39.322 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T01:20:39.322 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:20:39.322 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T01:20:39.871 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T01:20:39.913 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T01:20:39.954 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T01:20:39.985 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T01:20:40.012 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T01:20:40.205 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T01:20:40.250 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T01:20:40.272 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T01:20:40.850 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:20:40.894 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:20:40.915 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:20:41.446 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:20:41.446 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-25T01:20:41.447 INFO:teuthology.orchestra.run.smithi032.stdout:Upgraded: 2024-04-25T01:20:41.447 INFO:teuthology.orchestra.run.smithi032.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.447 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.447 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T01:20:41.447 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T01:20:41.447 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-25T01:20:41.447 INFO:teuthology.orchestra.run.smithi032.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T01:20:41.447 INFO:teuthology.orchestra.run.smithi032.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T01:20:41.447 INFO:teuthology.orchestra.run.smithi032.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T01:20:41.447 INFO:teuthology.orchestra.run.smithi032.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T01:20:41.447 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.447 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.447 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.448 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.448 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:41.448 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.448 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.448 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.448 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:41.448 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:41.448 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:41.448 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:41.448 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:41.448 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.448 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.448 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:41.448 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.449 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.449 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.449 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:41.449 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:41.449 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.449 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.449 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:41.449 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T01:20:41.449 INFO:teuthology.orchestra.run.smithi032.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T01:20:41.449 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T01:20:41.449 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T01:20:41.449 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T01:20:41.449 INFO:teuthology.orchestra.run.smithi032.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T01:20:41.449 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T01:20:41.450 INFO:teuthology.orchestra.run.smithi032.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T01:20:41.450 INFO:teuthology.orchestra.run.smithi032.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T01:20:41.450 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T01:20:41.450 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T01:20:41.450 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.450 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.450 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.450 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T01:20:41.450 INFO:teuthology.orchestra.run.smithi032.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T01:20:41.450 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T01:20:41.450 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T01:20:41.450 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T01:20:41.450 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.451 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.451 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T01:20:41.451 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.451 INFO:teuthology.orchestra.run.smithi032.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T01:20:41.451 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T01:20:41.451 INFO:teuthology.orchestra.run.smithi032.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T01:20:41.451 INFO:teuthology.orchestra.run.smithi032.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T01:20:41.451 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T01:20:41.451 INFO:teuthology.orchestra.run.smithi032.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T01:20:41.451 INFO:teuthology.orchestra.run.smithi032.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T01:20:41.451 INFO:teuthology.orchestra.run.smithi032.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T01:20:41.451 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T01:20:41.451 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T01:20:41.452 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T01:20:41.452 INFO:teuthology.orchestra.run.smithi032.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T01:20:41.452 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T01:20:41.452 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T01:20:41.452 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T01:20:41.452 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T01:20:41.452 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T01:20:41.452 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T01:20:41.452 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T01:20:41.452 INFO:teuthology.orchestra.run.smithi032.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T01:20:41.452 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T01:20:41.452 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T01:20:41.452 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T01:20:41.452 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.453 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.453 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.453 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T01:20:41.453 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T01:20:41.453 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T01:20:41.453 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T01:20:41.453 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T01:20:41.453 INFO:teuthology.orchestra.run.smithi032.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T01:20:41.453 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T01:20:41.453 INFO:teuthology.orchestra.run.smithi032.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T01:20:41.453 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T01:20:41.453 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T01:20:41.453 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T01:20:41.454 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T01:20:41.454 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T01:20:41.454 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T01:20:41.454 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T01:20:41.454 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T01:20:41.454 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T01:20:41.454 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T01:20:41.454 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T01:20:41.454 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T01:20:41.454 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T01:20:41.454 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T01:20:41.454 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T01:20:41.454 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T01:20:41.455 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T01:20:41.455 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T01:20:41.455 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T01:20:41.455 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T01:20:41.455 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T01:20:41.455 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T01:20:41.455 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T01:20:41.455 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T01:20:41.455 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T01:20:41.455 INFO:teuthology.orchestra.run.smithi032.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T01:20:41.455 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T01:20:41.455 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T01:20:41.455 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T01:20:41.456 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T01:20:41.456 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T01:20:41.456 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T01:20:41.456 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.456 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.456 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T01:20:41.456 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T01:20:41.456 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T01:20:41.456 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.456 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T01:20:41.456 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T01:20:41.456 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T01:20:41.456 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T01:20:41.456 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T01:20:41.456 INFO:teuthology.orchestra.run.smithi032.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T01:20:41.457 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T01:20:41.457 INFO:teuthology.orchestra.run.smithi032.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T01:20:41.457 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T01:20:41.457 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T01:20:41.457 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T01:20:41.457 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T01:20:41.457 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T01:20:41.457 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T01:20:41.457 INFO:teuthology.orchestra.run.smithi032.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T01:20:41.457 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.457 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.457 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:41.457 INFO:teuthology.orchestra.run.smithi032.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T01:20:41.457 INFO:teuthology.orchestra.run.smithi032.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T01:20:41.458 INFO:teuthology.orchestra.run.smithi032.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T01:20:41.458 INFO:teuthology.orchestra.run.smithi032.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T01:20:41.458 INFO:teuthology.orchestra.run.smithi032.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T01:20:41.458 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T01:20:41.458 INFO:teuthology.orchestra.run.smithi032.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T01:20:41.458 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-25T01:20:41.458 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-25T01:20:41.813 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T01:20:41.832 DEBUG:teuthology.parallel:result is None 2024-04-25T01:20:41.899 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T01:20:41.941 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T01:20:42.025 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T01:20:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T01:20:42.053 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:20:42.121 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:20:42.162 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:20:42.163 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:20:42.234 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:20:42.293 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:20:42.293 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T01:20:42.293 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T01:20:42.294 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T01:20:42.294 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T01:20:42.294 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T01:20:42.294 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T01:20:42.294 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T01:20:42.294 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:20:42.294 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:20:42.294 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:20:42.294 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T01:20:42.294 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T01:20:42.294 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:20:42.294 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T01:20:42.294 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T01:20:42.294 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T01:20:42.295 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T01:20:42.295 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T01:20:42.295 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T01:20:42.295 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T01:20:42.295 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T01:20:42.295 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T01:20:42.295 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T01:20:42.295 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T01:20:42.295 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T01:20:42.295 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T01:20:42.295 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T01:20:42.307 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T01:20:42.307 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T01:20:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T01:20:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T01:20:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T01:20:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T01:20:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T01:20:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T01:20:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T01:20:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T01:20:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T01:20:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T01:20:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T01:20:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T01:20:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T01:20:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T01:20:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T01:20:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T01:20:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T01:20:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T01:20:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T01:20:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T01:20:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T01:20:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T01:20:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T01:20:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T01:20:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T01:20:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T01:20:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T01:20:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T01:20:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T01:20:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T01:20:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T01:20:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T01:20:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T01:20:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T01:20:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T01:20:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T01:20:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T01:20:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T01:20:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T01:20:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T01:20:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T01:20:42.311 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T01:20:42.311 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T01:20:42.311 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T01:20:42.311 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T01:20:42.311 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T01:20:42.311 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T01:20:42.311 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T01:20:42.311 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T01:20:42.311 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T01:20:42.311 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T01:20:42.311 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T01:20:42.311 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T01:20:42.311 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T01:20:42.311 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T01:20:42.311 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T01:20:42.312 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T01:20:42.312 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T01:20:42.312 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T01:20:42.312 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T01:20:42.312 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T01:20:42.312 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T01:20:42.312 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T01:20:42.312 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T01:20:42.312 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T01:20:42.312 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T01:20:42.312 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T01:20:42.312 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T01:20:42.312 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T01:20:42.312 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T01:20:42.313 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T01:20:42.313 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T01:20:42.313 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T01:20:42.313 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T01:20:42.313 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T01:20:42.313 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T01:20:42.313 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T01:20:42.313 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T01:20:42.313 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T01:20:42.313 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T01:20:42.313 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T01:20:42.313 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T01:20:42.313 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T01:20:42.313 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T01:20:42.313 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T01:20:42.314 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T01:20:42.314 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T01:20:42.314 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T01:20:42.314 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:20:42.314 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:20:42.314 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T01:20:42.314 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T01:20:42.314 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T01:20:42.314 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T01:20:42.314 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T01:20:42.314 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T01:20:42.314 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T01:20:42.314 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T01:20:42.314 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T01:20:42.315 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T01:20:42.315 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T01:20:42.315 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T01:20:42.315 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T01:20:42.315 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T01:20:42.315 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T01:20:42.315 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T01:20:42.315 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T01:20:42.315 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T01:20:42.315 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T01:20:42.315 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T01:20:42.315 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T01:20:42.315 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T01:20:42.315 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T01:20:42.316 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T01:20:42.316 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T01:20:42.316 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T01:20:42.316 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T01:20:42.316 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T01:20:42.316 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T01:20:42.316 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T01:20:42.316 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T01:20:42.316 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T01:20:42.316 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T01:20:42.316 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T01:20:42.316 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:20:42.316 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T01:20:42.317 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:20:44.422 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:20:44.422 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-25T01:20:44.423 INFO:teuthology.orchestra.run.smithi130.stdout:Upgraded: 2024-04-25T01:20:44.423 INFO:teuthology.orchestra.run.smithi130.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.423 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.423 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T01:20:44.423 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T01:20:44.423 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-04-25T01:20:44.423 INFO:teuthology.orchestra.run.smithi130.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T01:20:44.423 INFO:teuthology.orchestra.run.smithi130.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T01:20:44.423 INFO:teuthology.orchestra.run.smithi130.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T01:20:44.423 INFO:teuthology.orchestra.run.smithi130.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T01:20:44.423 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.423 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.423 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.424 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.424 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:44.424 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.424 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.424 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.424 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:44.424 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:44.424 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:44.424 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:44.424 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:44.424 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.424 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.424 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:44.424 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.425 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.425 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.425 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:44.425 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:44.425 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.425 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.425 INFO:teuthology.orchestra.run.smithi130.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:44.425 INFO:teuthology.orchestra.run.smithi130.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T01:20:44.425 INFO:teuthology.orchestra.run.smithi130.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T01:20:44.425 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T01:20:44.425 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T01:20:44.425 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T01:20:44.425 INFO:teuthology.orchestra.run.smithi130.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T01:20:44.426 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T01:20:44.426 INFO:teuthology.orchestra.run.smithi130.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T01:20:44.426 INFO:teuthology.orchestra.run.smithi130.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T01:20:44.426 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T01:20:44.426 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T01:20:44.426 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.426 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.426 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.426 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T01:20:44.426 INFO:teuthology.orchestra.run.smithi130.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T01:20:44.426 INFO:teuthology.orchestra.run.smithi130.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T01:20:44.426 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T01:20:44.426 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T01:20:44.427 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.427 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.427 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T01:20:44.427 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.427 INFO:teuthology.orchestra.run.smithi130.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T01:20:44.427 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T01:20:44.427 INFO:teuthology.orchestra.run.smithi130.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T01:20:44.427 INFO:teuthology.orchestra.run.smithi130.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T01:20:44.427 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T01:20:44.427 INFO:teuthology.orchestra.run.smithi130.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T01:20:44.427 INFO:teuthology.orchestra.run.smithi130.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T01:20:44.427 INFO:teuthology.orchestra.run.smithi130.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T01:20:44.427 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T01:20:44.428 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T01:20:44.428 INFO:teuthology.orchestra.run.smithi130.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T01:20:44.428 INFO:teuthology.orchestra.run.smithi130.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T01:20:44.428 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T01:20:44.428 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T01:20:44.428 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T01:20:44.428 INFO:teuthology.orchestra.run.smithi130.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T01:20:44.428 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T01:20:44.428 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T01:20:44.428 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T01:20:44.428 INFO:teuthology.orchestra.run.smithi130.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T01:20:44.428 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T01:20:44.428 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T01:20:44.429 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T01:20:44.429 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.429 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.429 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.429 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T01:20:44.429 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T01:20:44.429 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T01:20:44.429 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T01:20:44.429 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T01:20:44.429 INFO:teuthology.orchestra.run.smithi130.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T01:20:44.429 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T01:20:44.429 INFO:teuthology.orchestra.run.smithi130.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T01:20:44.429 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T01:20:44.430 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T01:20:44.430 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T01:20:44.430 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T01:20:44.430 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T01:20:44.430 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T01:20:44.430 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T01:20:44.430 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T01:20:44.430 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T01:20:44.430 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T01:20:44.430 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T01:20:44.430 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T01:20:44.430 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T01:20:44.431 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T01:20:44.431 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T01:20:44.431 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T01:20:44.431 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T01:20:44.431 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T01:20:44.431 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T01:20:44.431 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T01:20:44.431 INFO:teuthology.orchestra.run.smithi130.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T01:20:44.431 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T01:20:44.431 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T01:20:44.431 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T01:20:44.431 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T01:20:44.431 INFO:teuthology.orchestra.run.smithi130.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T01:20:44.432 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T01:20:44.432 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T01:20:44.432 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T01:20:44.432 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T01:20:44.432 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T01:20:44.432 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T01:20:44.432 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.432 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.432 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T01:20:44.432 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T01:20:44.432 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T01:20:44.432 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.432 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T01:20:44.432 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T01:20:44.432 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T01:20:44.433 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T01:20:44.433 INFO:teuthology.orchestra.run.smithi130.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T01:20:44.433 INFO:teuthology.orchestra.run.smithi130.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T01:20:44.433 INFO:teuthology.orchestra.run.smithi130.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T01:20:44.433 INFO:teuthology.orchestra.run.smithi130.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T01:20:44.433 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T01:20:44.433 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T01:20:44.433 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T01:20:44.433 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T01:20:44.433 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T01:20:44.433 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T01:20:44.433 INFO:teuthology.orchestra.run.smithi130.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T01:20:44.433 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.433 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.434 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:44.434 INFO:teuthology.orchestra.run.smithi130.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T01:20:44.434 INFO:teuthology.orchestra.run.smithi130.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T01:20:44.434 INFO:teuthology.orchestra.run.smithi130.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T01:20:44.434 INFO:teuthology.orchestra.run.smithi130.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T01:20:44.434 INFO:teuthology.orchestra.run.smithi130.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T01:20:44.434 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T01:20:44.434 INFO:teuthology.orchestra.run.smithi130.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T01:20:44.435 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-25T01:20:44.435 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-04-25T01:20:44.826 DEBUG:teuthology.parallel:result is None 2024-04-25T01:20:48.029 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:20:48.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T01:20:48.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T01:20:48.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T01:20:48.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T01:20:48.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T01:20:48.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T01:20:48.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T01:20:48.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:20:48.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:20:48.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:20:48.029 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T01:20:48.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T01:20:48.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:20:48.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T01:20:48.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T01:20:48.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T01:20:48.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T01:20:48.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T01:20:48.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T01:20:48.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T01:20:48.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T01:20:48.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T01:20:48.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T01:20:48.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T01:20:48.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T01:20:48.031 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T01:20:48.031 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T01:20:48.031 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T01:20:48.031 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T01:20:48.042 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T01:20:48.042 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T01:20:48.042 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T01:20:48.042 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T01:20:48.042 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T01:20:48.042 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T01:20:48.042 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T01:20:48.042 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T01:20:48.042 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T01:20:48.043 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T01:20:48.043 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T01:20:48.043 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T01:20:48.043 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T01:20:48.043 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T01:20:48.043 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T01:20:48.043 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T01:20:48.043 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T01:20:48.043 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T01:20:48.043 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T01:20:48.043 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T01:20:48.043 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T01:20:48.043 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T01:20:48.043 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T01:20:48.043 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T01:20:48.043 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T01:20:48.044 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T01:20:48.044 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T01:20:48.044 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T01:20:48.044 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T01:20:48.044 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T01:20:48.044 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T01:20:48.044 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T01:20:48.044 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T01:20:48.044 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T01:20:48.044 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T01:20:48.044 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T01:20:48.044 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T01:20:48.044 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T01:20:48.044 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T01:20:48.044 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T01:20:48.045 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T01:20:48.045 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T01:20:48.045 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T01:20:48.045 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T01:20:48.045 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T01:20:48.045 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T01:20:48.045 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T01:20:48.045 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T01:20:48.045 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T01:20:48.045 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T01:20:48.045 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T01:20:48.045 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T01:20:48.045 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T01:20:48.045 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T01:20:48.045 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T01:20:48.046 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T01:20:48.046 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T01:20:48.046 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T01:20:48.046 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T01:20:48.046 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T01:20:48.046 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T01:20:48.046 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T01:20:48.046 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T01:20:48.046 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T01:20:48.046 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T01:20:48.046 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T01:20:48.046 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T01:20:48.046 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T01:20:48.046 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T01:20:48.047 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T01:20:48.047 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T01:20:48.047 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T01:20:48.047 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T01:20:48.047 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T01:20:48.047 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T01:20:48.047 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T01:20:48.047 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T01:20:48.047 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T01:20:48.047 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T01:20:48.047 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T01:20:48.047 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T01:20:48.047 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T01:20:48.047 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T01:20:48.047 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T01:20:48.047 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T01:20:48.048 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T01:20:48.048 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T01:20:48.048 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T01:20:48.048 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:20:48.048 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:20:48.048 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T01:20:48.048 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T01:20:48.048 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T01:20:48.048 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T01:20:48.048 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T01:20:48.048 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T01:20:48.048 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T01:20:48.048 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T01:20:48.048 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T01:20:48.049 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T01:20:48.049 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T01:20:48.049 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T01:20:48.049 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T01:20:48.049 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T01:20:48.049 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T01:20:48.049 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T01:20:48.049 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T01:20:48.049 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T01:20:48.049 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T01:20:48.049 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T01:20:48.049 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T01:20:48.049 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T01:20:48.049 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T01:20:48.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T01:20:48.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T01:20:48.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T01:20:48.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T01:20:48.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T01:20:48.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T01:20:48.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T01:20:48.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T01:20:48.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T01:20:48.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T01:20:48.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T01:20:48.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:20:48.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T01:20:50.286 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:20:50.287 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:20:50.287 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-04-25T01:20:50.287 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.287 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.287 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T01:20:50.287 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T01:20:50.287 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-25T01:20:50.287 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T01:20:50.287 INFO:teuthology.orchestra.run.smithi028.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T01:20:50.287 INFO:teuthology.orchestra.run.smithi028.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T01:20:50.287 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T01:20:50.287 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.288 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.288 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.288 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.288 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:50.288 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.288 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.288 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.288 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:50.288 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:50.288 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:50.288 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:50.288 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:50.289 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.289 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.289 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:50.289 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.289 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.289 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.289 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:50.289 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:50.289 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.289 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.289 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:20:50.289 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T01:20:50.289 INFO:teuthology.orchestra.run.smithi028.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T01:20:50.290 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T01:20:50.290 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T01:20:50.290 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T01:20:50.290 INFO:teuthology.orchestra.run.smithi028.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T01:20:50.290 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T01:20:50.290 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T01:20:50.290 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T01:20:50.290 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T01:20:50.290 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T01:20:50.291 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.291 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.291 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.291 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T01:20:50.291 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T01:20:50.291 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T01:20:50.291 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T01:20:50.291 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T01:20:50.291 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.291 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.291 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T01:20:50.291 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.291 INFO:teuthology.orchestra.run.smithi028.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T01:20:50.291 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T01:20:50.291 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T01:20:50.292 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T01:20:50.292 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T01:20:50.292 INFO:teuthology.orchestra.run.smithi028.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T01:20:50.292 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T01:20:50.292 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T01:20:50.292 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T01:20:50.292 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T01:20:50.292 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T01:20:50.292 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T01:20:50.292 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T01:20:50.292 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T01:20:50.292 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T01:20:50.292 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T01:20:50.293 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T01:20:50.293 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T01:20:50.293 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T01:20:50.293 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T01:20:50.293 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T01:20:50.293 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T01:20:50.293 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T01:20:50.293 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.293 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.293 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.293 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T01:20:50.293 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T01:20:50.293 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T01:20:50.294 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T01:20:50.294 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T01:20:50.294 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T01:20:50.294 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T01:20:50.294 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T01:20:50.294 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T01:20:50.294 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T01:20:50.294 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T01:20:50.294 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T01:20:50.294 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T01:20:50.294 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T01:20:50.294 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T01:20:50.295 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T01:20:50.295 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T01:20:50.295 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T01:20:50.295 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T01:20:50.295 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T01:20:50.295 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T01:20:50.295 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T01:20:50.295 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T01:20:50.295 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T01:20:50.295 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T01:20:50.295 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T01:20:50.295 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T01:20:50.296 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T01:20:50.296 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T01:20:50.296 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T01:20:50.296 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T01:20:50.296 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T01:20:50.296 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T01:20:50.296 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T01:20:50.296 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T01:20:50.296 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T01:20:50.296 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T01:20:50.296 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T01:20:50.296 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T01:20:50.296 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T01:20:50.297 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.297 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.297 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T01:20:50.297 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T01:20:50.297 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T01:20:50.297 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.297 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T01:20:50.297 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T01:20:50.297 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T01:20:50.297 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T01:20:50.297 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T01:20:50.297 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T01:20:50.297 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T01:20:50.297 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T01:20:50.298 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T01:20:50.298 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T01:20:50.298 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T01:20:50.298 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T01:20:50.298 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T01:20:50.298 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T01:20:50.298 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T01:20:50.298 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.298 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.298 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:20:50.298 INFO:teuthology.orchestra.run.smithi028.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T01:20:50.298 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T01:20:50.298 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T01:20:50.298 INFO:teuthology.orchestra.run.smithi028.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T01:20:50.299 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T01:20:50.299 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T01:20:50.299 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T01:20:50.299 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:20:50.299 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-25T01:20:50.656 DEBUG:teuthology.parallel:result is None 2024-04-25T01:20:50.656 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:20:50.878 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T01:20:50.900 INFO:teuthology.orchestra.run.smithi028.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T01:20:50.900 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T01:20:50.900 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T01:20:50.902 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:20:51.119 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T01:20:51.145 INFO:teuthology.orchestra.run.smithi032.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T01:20:51.145 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T01:20:51.145 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T01:20:51.147 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:20:51.288 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T01:20:51.313 INFO:teuthology.orchestra.run.smithi130.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T01:20:51.313 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T01:20:51.314 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T01:20:51.315 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T01:20:51.315 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:20:51.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T01:20:51.344 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-25T01:20:51.344 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T01:20:51.377 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-25T01:20:51.377 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T01:20:51.407 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T01:20:51.407 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:20:51.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T01:20:51.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T01:20:51.500 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-25T01:20:51.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T01:20:51.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T01:20:51.596 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-25T01:20:51.596 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T01:20:51.627 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T01:20:51.693 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T01:20:51.694 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:20:51.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T01:20:51.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T01:20:51.784 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-25T01:20:51.784 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T01:20:51.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T01:20:51.880 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-25T01:20:51.880 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T01:20:51.908 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T01:20:51.975 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T01:20:51.975 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:20:51.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T01:20:52.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T01:20:52.068 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-25T01:20:52.068 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T01:20:52.097 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T01:20:52.166 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-25T01:20:52.167 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T01:20:52.193 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T01:20:52.259 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-25T01:20:52.358 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-25T01:20:52.358 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:20:52.359 INFO:tasks.cephadm:Cluster fsid is 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:20:52.359 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-25T01:20:52.359 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.28', 'mon.b': '172.21.15.32', 'mon.c': '172.21.15.130'} 2024-04-25T01:20:52.359 INFO:tasks.cephadm:First mon is mon.a on smithi028 2024-04-25T01:20:52.359 INFO:tasks.cephadm:First mgr is x 2024-04-25T01:20:52.359 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-25T01:20:52.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo hostname $(hostname -s) 2024-04-25T01:20:52.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo hostname $(hostname -s) 2024-04-25T01:20:52.414 DEBUG:teuthology.orchestra.run.smithi130:> sudo hostname $(hostname -s) 2024-04-25T01:20:52.442 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-25T01:20:52.443 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:20:52.712 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54987/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2445-gd2c45d97', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2445.gd2c45d97'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-24 22:37:46.422067', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54981/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2445-gd2c45d97', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2445.gd2c45d97'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-24 17:00:00.872117', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-25T01:20:52.848 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 d2c45d977142e024a59b68a8f1dce8f0b9840551 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:20:52.850 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm 2024-04-25T01:20:52.851 DEBUG:teuthology.orchestra.run.smithi028:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T01:20:53.796 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 25 01:20 /home/ubuntu/cephtest/cephadm 2024-04-25T01:20:53.796 DEBUG:teuthology.orchestra.run.smithi032:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T01:20:54.531 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 25 01:20 /home/ubuntu/cephtest/cephadm 2024-04-25T01:20:54.532 DEBUG:teuthology.orchestra.run.smithi130:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T01:20:54.967 INFO:teuthology.orchestra.run.smithi130.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 25 01:20 /home/ubuntu/cephtest/cephadm 2024-04-25T01:20:54.967 DEBUG:teuthology.orchestra.run.smithi028:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T01:20:54.986 DEBUG:teuthology.orchestra.run.smithi032:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T01:20:55.006 DEBUG:teuthology.orchestra.run.smithi130:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T01:20:55.042 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 on all hosts... 2024-04-25T01:20:55.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-25T01:20:55.045 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-25T01:20:55.049 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-25T01:20:56.592 INFO:teuthology.orchestra.run.smithi032.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:20:56.603 INFO:teuthology.orchestra.run.smithi028.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:20:56.706 INFO:teuthology.orchestra.run.smithi130.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:22:09.744 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-04-25T01:22:09.745 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:22:09.745 INFO:teuthology.orchestra.run.smithi130.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-25T01:22:09.745 INFO:teuthology.orchestra.run.smithi130.stdout: "repo_digests": [ 2024-04-25T01:22:09.745 INFO:teuthology.orchestra.run.smithi130.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-25T01:22:09.745 INFO:teuthology.orchestra.run.smithi130.stdout: ] 2024-04-25T01:22:09.745 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-04-25T01:22:11.231 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:22:11.231 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:22:11.231 INFO:teuthology.orchestra.run.smithi028.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-25T01:22:11.231 INFO:teuthology.orchestra.run.smithi028.stdout: "repo_digests": [ 2024-04-25T01:22:11.231 INFO:teuthology.orchestra.run.smithi028.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-25T01:22:11.231 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-25T01:22:11.231 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:22:12.870 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-25T01:22:12.870 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:22:12.870 INFO:teuthology.orchestra.run.smithi032.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-25T01:22:12.870 INFO:teuthology.orchestra.run.smithi032.stdout: "repo_digests": [ 2024-04-25T01:22:12.870 INFO:teuthology.orchestra.run.smithi032.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-25T01:22:12.870 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-25T01:22:12.870 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-25T01:22:12.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph 2024-04-25T01:22:12.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph 2024-04-25T01:22:12.957 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /etc/ceph 2024-04-25T01:22:12.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /etc/ceph 2024-04-25T01:22:13.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /etc/ceph 2024-04-25T01:22:13.041 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 777 /etc/ceph 2024-04-25T01:22:13.070 INFO:tasks.cephadm:Writing seed config... 2024-04-25T01:22:13.071 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-25T01:22:13.071 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-25T01:22:13.071 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-25T01:22:13.071 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-25T01:22:13.071 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-25T01:22:13.072 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-25T01:22:13.072 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-25T01:22:13.072 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-25T01:22:13.072 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-25T01:22:13.072 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-25T01:22:13.072 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-25T01:22:13.072 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-25T01:22:13.072 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-25T01:22:13.072 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-25T01:22:13.072 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-25T01:22:13.072 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-25T01:22:13.072 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-25T01:22:13.072 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-25T01:22:13.072 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-25T01:22:13.072 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-25T01:22:13.072 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-25T01:22:13.073 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-25T01:22:13.073 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-25T01:22:13.073 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-25T01:22:13.073 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-25T01:22:13.073 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-25T01:22:13.073 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-25T01:22:13.073 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-25T01:22:13.073 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-25T01:22:13.073 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-25T01:22:13.073 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-25T01:22:13.073 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-25T01:22:13.073 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-25T01:22:13.073 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-25T01:22:13.073 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-25T01:22:13.073 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-25T01:22:13.074 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-25T01:22:13.074 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-25T01:22:13.074 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-25T01:22:13.074 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-25T01:22:13.074 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-25T01:22:13.074 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-25T01:22:13.074 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-25T01:22:13.074 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-25T01:22:13.074 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-04-25T01:22:13.075 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:22:13.075 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-25T01:22:13.093 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 0e88f60a-02a2-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-25T01:22:13.094 DEBUG:teuthology.orchestra.run.smithi028:mon.a> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mon.a.service 2024-04-25T01:22:13.137 DEBUG:teuthology.orchestra.run.smithi028:mgr.x> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mgr.x.service 2024-04-25T01:22:13.180 INFO:tasks.cephadm:Bootstrapping... 2024-04-25T01:22:13.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 -v bootstrap --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.28 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:22:13.384 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:22:13.384 INFO:teuthology.orchestra.run.smithi028.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551', '-v', 'bootstrap', '--fsid', '0e88f60a-02a2-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.28', '--skip-admin-label'] 2024-04-25T01:22:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-25T01:22:13.384 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying podman|docker is present... 2024-04-25T01:22:13.607 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T01:22:13.607 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying lvm2 is present... 2024-04-25T01:22:13.607 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying time synchronization is in place... 2024-04-25T01:22:13.615 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T01:22:13.615 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T01:22:13.621 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T01:22:13.621 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-25T01:22:13.628 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-04-25T01:22:13.634 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-04-25T01:22:13.634 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-04-25T01:22:13.634 INFO:teuthology.orchestra.run.smithi028.stdout:Repeating the final host check... 2024-04-25T01:22:13.658 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T01:22:13.659 INFO:teuthology.orchestra.run.smithi028.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-25T01:22:13.659 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl is present 2024-04-25T01:22:13.659 INFO:teuthology.orchestra.run.smithi028.stdout:lvcreate is present 2024-04-25T01:22:13.665 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T01:22:13.665 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T01:22:13.671 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T01:22:13.671 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-25T01:22:13.678 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-04-25T01:22:13.685 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-04-25T01:22:13.685 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-04-25T01:22:13.685 INFO:teuthology.orchestra.run.smithi028.stdout:Host looks OK 2024-04-25T01:22:13.685 INFO:teuthology.orchestra.run.smithi028.stdout:Cluster fsid: 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:22:13.685 INFO:teuthology.orchestra.run.smithi028.stdout:Acquiring lock 140700813581856 on /run/cephadm/0e88f60a-02a2-11ef-bc93-c7b262605968.lock 2024-04-25T01:22:13.685 INFO:teuthology.orchestra.run.smithi028.stdout:Lock 140700813581856 acquired on /run/cephadm/0e88f60a-02a2-11ef-bc93-c7b262605968.lock 2024-04-25T01:22:13.685 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 3300 ... 2024-04-25T01:22:13.686 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 6789 ... 2024-04-25T01:22:13.686 INFO:teuthology.orchestra.run.smithi028.stdout:Base mon IP(s) is [172.21.15.28:3300, 172.21.15.28:6789], mon addrv is [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-04-25T01:22:13.689 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.28 metric 100 2024-04-25T01:22:13.689 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.28 metric 100 2024-04-25T01:22:13.691 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-25T01:22:13.692 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-25T01:22:13.692 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-25T01:22:13.694 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-25T01:22:13.694 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-25T01:22:13.694 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T01:22:13.694 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-25T01:22:13.694 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-04-25T01:22:13.694 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T01:22:13.695 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-04-25T01:22:13.695 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-04-25T01:22:13.695 INFO:teuthology.orchestra.run.smithi028.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-25T01:22:13.695 INFO:teuthology.orchestra.run.smithi028.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-25T01:22:13.696 INFO:teuthology.orchestra.run.smithi028.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:22:14.403 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e 2024-04-25T01:22:14.403 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:22:14.403 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Getting image source signatures 2024-04-25T01:22:14.404 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:58a99251f41e8a467bafa0ec849428709568906ca732f10972304f37056be6c5 2024-04-25T01:22:14.404 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-25T01:22:14.404 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying config sha256:f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e 2024-04-25T01:22:14.404 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-25T01:22:15.409 INFO:teuthology.orchestra.run.smithi028.stdout:ceph: stdout ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev) 2024-04-25T01:22:15.409 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph version: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev) 2024-04-25T01:22:15.409 INFO:teuthology.orchestra.run.smithi028.stdout:Extracting ceph user uid/gid from container image... 2024-04-25T01:22:16.271 INFO:teuthology.orchestra.run.smithi028.stdout:stat: stdout 167 167 2024-04-25T01:22:16.272 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial keys... 2024-04-25T01:22:17.335 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQBIsClmH8FWJBAAm+5O22z9B0+GbKmQsk47GQ== 2024-04-25T01:22:18.141 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQBJsClmSmcqKRAAJdh0jBdJ3buTYAFup5UhtQ== 2024-04-25T01:22:19.056 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQBKsClmr7kSHhAAC7lxWLsNAEHKskR6Yz9zOg== 2024-04-25T01:22:19.057 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial monmap... 2024-04-25T01:22:23.900 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T01:22:23.900 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-25T01:22:23.900 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:22:23.900 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T01:22:23.900 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool for a [v2:172.21.15.28:3300,v1:172.21.15.28:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T01:22:23.900 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = quincy 2024-04-25T01:22:23.900 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: set fsid to 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:22:23.900 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T01:22:23.900 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:22:23.900 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mon... 2024-04-25T01:22:24.795 INFO:teuthology.orchestra.run.smithi028.stdout:create mon.a on 2024-04-25T01:22:25.010 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-25T01:22:25.162 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-25T01:22:25.344 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-0e88f60a-02a2-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-0e88f60a-02a2-11ef-bc93-c7b262605968.target. 2024-04-25T01:22:25.344 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-0e88f60a-02a2-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-0e88f60a-02a2-11ef-bc93-c7b262605968.target. 2024-04-25T01:22:25.573 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mon.a 2024-04-25T01:22:25.573 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-25T01:22:25.749 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0e88f60a-02a2-11ef-bc93-c7b262605968.target.wants/ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@.service. 2024-04-25T01:22:25.801 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:25 smithi028 systemd[1]: Starting Ceph mon.a for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:22:26.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:26 smithi028 podman[29000]: 2024-04-25 01:22:26.013128923 +0000 UTC m=+0.118753797 container create 6857711b7e708a22486284f1b870abbced4e6312f4df28fe10157eb3f906a660 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-04-25T01:22:26.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:26 smithi028 podman[29000]: 2024-04-25 01:22:25.918572611 +0000 UTC m=+0.024197487 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:22:26.326 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:22:26.326 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-25T01:22:26.333 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:22:26.334 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-25T01:22:26.334 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-25T01:22:26.334 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T01:22:26.334 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon to start... 2024-04-25T01:22:26.334 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon... 2024-04-25T01:22:26.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:26 smithi028 podman[29000]: 2024-04-25 01:22:26.256786068 +0000 UTC m=+0.362410946 container init 6857711b7e708a22486284f1b870abbced4e6312f4df28fe10157eb3f906a660 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-25T01:22:26.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:26 smithi028 podman[29000]: 2024-04-25 01:22:26.259804823 +0000 UTC m=+0.365429700 container start 6857711b7e708a22486284f1b870abbced4e6312f4df28fe10157eb3f906a660 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:22:26.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:26 smithi028 ceph-mon[29020]: mkfs 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:22:26.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:26 smithi028 ceph-mon[29020]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T01:22:26.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:26 smithi028 bash[29000]: 6857711b7e708a22486284f1b870abbced4e6312f4df28fe10157eb3f906a660 2024-04-25T01:22:26.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:26 smithi028 systemd[1]: Started Ceph mon.a for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:22:27.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:27 smithi028 ceph-mon[29020]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T01:22:27.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:27 smithi028 ceph-mon[29020]: monmap epoch 1 2024-04-25T01:22:27.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:27 smithi028 ceph-mon[29020]: fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:22:27.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:27 smithi028 ceph-mon[29020]: last_changed 2024-04-25T01:22:20.217202+0000 2024-04-25T01:22:27.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:27 smithi028 ceph-mon[29020]: created 2024-04-25T01:22:20.217202+0000 2024-04-25T01:22:27.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:27 smithi028 ceph-mon[29020]: min_mon_release 19 (squid) 2024-04-25T01:22:27.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:27 smithi028 ceph-mon[29020]: election_strategy: 1 2024-04-25T01:22:27.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:27 smithi028 ceph-mon[29020]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-25T01:22:27.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:27 smithi028 ceph-mon[29020]: fsmap 2024-04-25T01:22:27.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:27 smithi028 ceph-mon[29020]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T01:22:27.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:27 smithi028 ceph-mon[29020]: mgrmap e1: no daemons active 2024-04-25T01:22:27.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:27 smithi028 ceph-mon[29020]: from='client.? 172.21.15.28:0/1975039268' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-25T01:22:27.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout cluster: 2024-04-25T01:22:27.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout id: 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:22:27.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-25T01:22:27.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-25T01:22:27.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout services: 2024-04-25T01:22:27.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.558505s) 2024-04-25T01:22:27.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-25T01:22:27.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-25T01:22:27.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-25T01:22:27.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout data: 2024-04-25T01:22:27.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-25T01:22:27.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-25T01:22:27.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-25T01:22:27.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pgs: 2024-04-25T01:22:27.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-25T01:22:27.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.776+0000 7f7097fd9700 1 Processor -- start 2024-04-25T01:22:27.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.776+0000 7f7097fd9700 1 -- start start 2024-04-25T01:22:27.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.776+0000 7f7097fd9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7090108680 0x7f709010aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.776+0000 7f7097fd9700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f709007be90 con 0x7f7090108680 2024-04-25T01:22:27.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.777+0000 7f7095d75700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7090108680 0x7f709010aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.777+0000 7f7095d75700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7090108680 0x7f709010aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46754/0 (socket says 172.21.15.28:46754) 2024-04-25T01:22:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.777+0000 7f7095d75700 1 -- 172.21.15.28:0/3477121084 learned_addr learned my addr 172.21.15.28:0/3477121084 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.777+0000 7f7095d75700 1 -- 172.21.15.28:0/3477121084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f709010afb0 con 0x7f7090108680 2024-04-25T01:22:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.777+0000 7f7095d75700 1 --2- 172.21.15.28:0/3477121084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7090108680 0x7f709010aa70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f708c009e80 tx=0x7f708c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5604007e696a3ee5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.777+0000 7f7094d73700 1 -- 172.21.15.28:0/3477121084 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f708c017070 con 0x7f7090108680 2024-04-25T01:22:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.777+0000 7f7094d73700 1 -- 172.21.15.28:0/3477121084 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708c003a20 con 0x7f7090108680 2024-04-25T01:22:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.777+0000 7f7094d73700 1 -- 172.21.15.28:0/3477121084 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f708c003d20 con 0x7f7090108680 2024-04-25T01:22:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.778+0000 7f7097fd9700 1 -- 172.21.15.28:0/3477121084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7090108680 msgr2=0x7f709010aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.778+0000 7f7097fd9700 1 --2- 172.21.15.28:0/3477121084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7090108680 0x7f709010aa70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f708c009e80 tx=0x7f708c00b330 comp rx=0 tx=0).stop 2024-04-25T01:22:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.778+0000 7f7097fd9700 1 -- 172.21.15.28:0/3477121084 shutdown_connections 2024-04-25T01:22:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.778+0000 7f7097fd9700 1 --2- 172.21.15.28:0/3477121084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7090108680 0x7f709010aa70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.778+0000 7f7097fd9700 1 -- 172.21.15.28:0/3477121084 >> 172.21.15.28:0/3477121084 conn(0x7f7090102050 msgr2=0x7f70901044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:27.384 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.778+0000 7f7097fd9700 1 -- 172.21.15.28:0/3477121084 shutdown_connections 2024-04-25T01:22:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.778+0000 7f7097fd9700 1 -- 172.21.15.28:0/3477121084 wait complete. 2024-04-25T01:22:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.779+0000 7f7097fd9700 1 Processor -- start 2024-04-25T01:22:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.779+0000 7f7097fd9700 1 -- start start 2024-04-25T01:22:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.779+0000 7f7097fd9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7090108680 0x7f709019b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.779+0000 7f7097fd9700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f709010b420 con 0x7f7090108680 2024-04-25T01:22:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.779+0000 7f7095d75700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7090108680 0x7f709019b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.779+0000 7f7095d75700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7090108680 0x7f709019b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46770/0 (socket says 172.21.15.28:46770) 2024-04-25T01:22:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.779+0000 7f7095d75700 1 -- 172.21.15.28:0/1975039268 learned_addr learned my addr 172.21.15.28:0/1975039268 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.779+0000 7f7095d75700 1 -- 172.21.15.28:0/1975039268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f709019b820 con 0x7f7090108680 2024-04-25T01:22:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.779+0000 7f7095d75700 1 --2- 172.21.15.28:0/1975039268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7090108680 0x7f709019b2e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f709007b2d0 tx=0x7f708c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.780+0000 7f7086ffd700 1 -- 172.21.15.28:0/1975039268 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f708c022020 con 0x7f7090108680 2024-04-25T01:22:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.780+0000 7f7086ffd700 1 -- 172.21.15.28:0/1975039268 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708c018040 con 0x7f7090108680 2024-04-25T01:22:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.780+0000 7f7086ffd700 1 -- 172.21.15.28:0/1975039268 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f708c017070 con 0x7f7090108680 2024-04-25T01:22:27.385 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.780+0000 7f7097fd9700 1 -- 172.21.15.28:0/1975039268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f709019bab0 con 0x7f7090108680 2024-04-25T01:22:27.386 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.780+0000 7f7097fd9700 1 -- 172.21.15.28:0/1975039268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f709019e7a0 con 0x7f7090108680 2024-04-25T01:22:27.386 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.780+0000 7f7086ffd700 1 -- 172.21.15.28:0/1975039268 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f708c025020 con 0x7f7090108680 2024-04-25T01:22:27.386 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.780+0000 7f7086ffd700 1 -- 172.21.15.28:0/1975039268 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f708c00f050 con 0x7f7090108680 2024-04-25T01:22:27.386 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.781+0000 7f7097fd9700 1 -- 172.21.15.28:0/1975039268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7090063890 con 0x7f7090108680 2024-04-25T01:22:27.386 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.783+0000 7f7086ffd700 1 -- 172.21.15.28:0/1975039268 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f708c02c030 con 0x7f7090108680 2024-04-25T01:22:27.386 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.856+0000 7f7097fd9700 1 -- 172.21.15.28:0/1975039268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f709002dce0 con 0x7f7090108680 2024-04-25T01:22:27.386 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.856+0000 7f7086ffd700 1 -- 172.21.15.28:0/1975039268 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f708c01e480 con 0x7f7090108680 2024-04-25T01:22:27.386 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.857+0000 7f7097fd9700 1 -- 172.21.15.28:0/1975039268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7090108680 msgr2=0x7f709019b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:27.386 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.857+0000 7f7097fd9700 1 --2- 172.21.15.28:0/1975039268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7090108680 0x7f709019b2e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f709007b2d0 tx=0x7f708c004ad0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.386 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.857+0000 7f7097fd9700 1 -- 172.21.15.28:0/1975039268 shutdown_connections 2024-04-25T01:22:27.386 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.857+0000 7f7097fd9700 1 --2- 172.21.15.28:0/1975039268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7090108680 0x7f709019b2e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:27.386 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.857+0000 7f7097fd9700 1 -- 172.21.15.28:0/1975039268 >> 172.21.15.28:0/1975039268 conn(0x7f7090102050 msgr2=0x7f70901043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:27.386 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.857+0000 7f7097fd9700 1 -- 172.21.15.28:0/1975039268 shutdown_connections 2024-04-25T01:22:27.386 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:26.857+0000 7f7097fd9700 1 -- 172.21.15.28:0/1975039268 wait complete. 2024-04-25T01:22:27.387 INFO:teuthology.orchestra.run.smithi028.stdout:mon is available 2024-04-25T01:22:27.387 INFO:teuthology.orchestra.run.smithi028.stdout:Assimilating anything we can from ceph.conf... 2024-04-25T01:22:28.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:28 smithi028 ceph-mon[29020]: from='client.? 172.21.15.28:0/3593511794' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T01:22:28.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:28 smithi028 ceph-mon[29020]: from='client.? 172.21.15.28:0/3593511794' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-25T01:22:28.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-25T01:22:28.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T01:22:28.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:22:28.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-04-25T01:22:28.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T01:22:28.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T01:22:28.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T01:22:28.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T01:22:28.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-25T01:22:28.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T01:22:28.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T01:22:28.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-25T01:22:28.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T01:22:28.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T01:22:28.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T01:22:28.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T01:22:28.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T01:22:28.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T01:22:28.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.803+0000 7f7ccb6f3700 1 Processor -- start 2024-04-25T01:22:28.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.803+0000 7f7ccb6f3700 1 -- start start 2024-04-25T01:22:28.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.803+0000 7f7ccb6f3700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc4107bf0 0x7f7cc4108010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.803+0000 7f7ccb6f3700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc41085e0 con 0x7f7cc4107bf0 2024-04-25T01:22:28.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.803+0000 7f7cc948f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc4107bf0 0x7f7cc4108010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:28.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.803+0000 7f7cc948f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc4107bf0 0x7f7cc4108010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46778/0 (socket says 172.21.15.28:46778) 2024-04-25T01:22:28.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.803+0000 7f7cc948f700 1 -- 172.21.15.28:0/2488789182 learned_addr learned my addr 172.21.15.28:0/2488789182 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:28.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.804+0000 7f7cc948f700 1 -- 172.21.15.28:0/2488789182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc4108da0 con 0x7f7cc4107bf0 2024-04-25T01:22:28.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.804+0000 7f7cc948f700 1 --2- 172.21.15.28:0/2488789182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc4107bf0 0x7f7cc4108010 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7cc00098a0 tx=0x7f7cc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d183b5d8f630524 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:28.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.804+0000 7f7cbbfff700 1 -- 172.21.15.28:0/2488789182 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cc0004160 con 0x7f7cc4107bf0 2024-04-25T01:22:28.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.804+0000 7f7cbbfff700 1 -- 172.21.15.28:0/2488789182 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cc0004300 con 0x7f7cc4107bf0 2024-04-25T01:22:28.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.804+0000 7f7cbbfff700 1 -- 172.21.15.28:0/2488789182 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cc00036a0 con 0x7f7cc4107bf0 2024-04-25T01:22:28.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.804+0000 7f7ccb6f3700 1 -- 172.21.15.28:0/2488789182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc4107bf0 msgr2=0x7f7cc4108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.804+0000 7f7ccb6f3700 1 --2- 172.21.15.28:0/2488789182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc4107bf0 0x7f7cc4108010 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7cc00098a0 tx=0x7f7cc0009bb0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.805+0000 7f7ccb6f3700 1 -- 172.21.15.28:0/2488789182 shutdown_connections 2024-04-25T01:22:28.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.805+0000 7f7ccb6f3700 1 --2- 172.21.15.28:0/2488789182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc4107bf0 0x7f7cc4108010 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.805+0000 7f7ccb6f3700 1 -- 172.21.15.28:0/2488789182 >> 172.21.15.28:0/2488789182 conn(0x7f7cc4103170 msgr2=0x7f7cc41055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:28.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.805+0000 7f7ccb6f3700 1 -- 172.21.15.28:0/2488789182 shutdown_connections 2024-04-25T01:22:28.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.805+0000 7f7ccb6f3700 1 -- 172.21.15.28:0/2488789182 wait complete. 2024-04-25T01:22:28.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.805+0000 7f7ccb6f3700 1 Processor -- start 2024-04-25T01:22:28.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.805+0000 7f7ccb6f3700 1 -- start start 2024-04-25T01:22:28.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.805+0000 7f7ccb6f3700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc4107bf0 0x7f7cc419b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:28.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.805+0000 7f7ccb6f3700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc41092d0 con 0x7f7cc4107bf0 2024-04-25T01:22:28.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.806+0000 7f7cc948f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc4107bf0 0x7f7cc419b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:28.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.806+0000 7f7cc948f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc4107bf0 0x7f7cc419b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46792/0 (socket says 172.21.15.28:46792) 2024-04-25T01:22:28.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.806+0000 7f7cc948f700 1 -- 172.21.15.28:0/3593511794 learned_addr learned my addr 172.21.15.28:0/3593511794 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:28.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.806+0000 7f7cc948f700 1 -- 172.21.15.28:0/3593511794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc419b7d0 con 0x7f7cc4107bf0 2024-04-25T01:22:28.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.806+0000 7f7cc948f700 1 --2- 172.21.15.28:0/3593511794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc4107bf0 0x7f7cc419b290 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0000c00 tx=0x7f7cc0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:28.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.806+0000 7f7cba7fc700 1 -- 172.21.15.28:0/3593511794 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cc0003db0 con 0x7f7cc4107bf0 2024-04-25T01:22:28.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.806+0000 7f7cba7fc700 1 -- 172.21.15.28:0/3593511794 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cc0011630 con 0x7f7cc4107bf0 2024-04-25T01:22:28.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.806+0000 7f7ccb6f3700 1 -- 172.21.15.28:0/3593511794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cc419ba60 con 0x7f7cc4107bf0 2024-04-25T01:22:28.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.806+0000 7f7cba7fc700 1 -- 172.21.15.28:0/3593511794 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cc0011950 con 0x7f7cc4107bf0 2024-04-25T01:22:28.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.806+0000 7f7ccb6f3700 1 -- 172.21.15.28:0/3593511794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cc419e750 con 0x7f7cc4107bf0 2024-04-25T01:22:28.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.806+0000 7f7cba7fc700 1 -- 172.21.15.28:0/3593511794 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7cc0011af0 con 0x7f7cc4107bf0 2024-04-25T01:22:28.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.806+0000 7f7cba7fc700 1 -- 172.21.15.28:0/3593511794 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7cc001c450 con 0x7f7cc4107bf0 2024-04-25T01:22:28.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.807+0000 7f7ccb6f3700 1 -- 172.21.15.28:0/3593511794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cc4063890 con 0x7f7cc4107bf0 2024-04-25T01:22:28.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.810+0000 7f7cba7fc700 1 -- 172.21.15.28:0/3593511794 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7cc0004260 con 0x7f7cc4107bf0 2024-04-25T01:22:28.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.881+0000 7f7ccb6f3700 1 -- 172.21.15.28:0/3593511794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f7cc402dcb0 con 0x7f7cc4107bf0 2024-04-25T01:22:28.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.884+0000 7f7cba7fc700 1 -- 172.21.15.28:0/3593511794 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f7cc001b450 con 0x7f7cc4107bf0 2024-04-25T01:22:28.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.885+0000 7f7ccb6f3700 1 -- 172.21.15.28:0/3593511794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc4107bf0 msgr2=0x7f7cc419b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:28.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.885+0000 7f7ccb6f3700 1 --2- 172.21.15.28:0/3593511794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc4107bf0 0x7f7cc419b290 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0000c00 tx=0x7f7cc0004750 comp rx=0 tx=0).stop 2024-04-25T01:22:28.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.885+0000 7f7cba7fc700 1 -- 172.21.15.28:0/3593511794 <== mon.0 v2:172.21.15.28:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7cc001b630 con 0x7f7cc4107bf0 2024-04-25T01:22:28.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.885+0000 7f7ccb6f3700 1 -- 172.21.15.28:0/3593511794 shutdown_connections 2024-04-25T01:22:28.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.885+0000 7f7ccb6f3700 1 --2- 172.21.15.28:0/3593511794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc4107bf0 0x7f7cc419b290 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:28.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.885+0000 7f7ccb6f3700 1 -- 172.21.15.28:0/3593511794 >> 172.21.15.28:0/3593511794 conn(0x7f7cc4103170 msgr2=0x7f7cc4191f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:28.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.885+0000 7f7ccb6f3700 1 -- 172.21.15.28:0/3593511794 shutdown_connections 2024-04-25T01:22:28.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:27.885+0000 7f7ccb6f3700 1 -- 172.21.15.28:0/3593511794 wait complete. 2024-04-25T01:22:28.409 INFO:teuthology.orchestra.run.smithi028.stdout:Generating new minimal ceph.conf... 2024-04-25T01:22:29.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:29 smithi028 ceph-mon[29020]: from='client.? 172.21.15.28:0/2158685584' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:22:29.896 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.835+0000 7fd73319c700 1 Processor -- start 2024-04-25T01:22:29.896 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.835+0000 7fd73319c700 1 -- start start 2024-04-25T01:22:29.896 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.835+0000 7fd73319c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c107c10 0x7fd72c108030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.896 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.835+0000 7fd73319c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c108600 con 0x7fd72c107c10 2024-04-25T01:22:29.896 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.836+0000 7fd730f38700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c107c10 0x7fd72c108030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.896 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.836+0000 7fd730f38700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c107c10 0x7fd72c108030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46794/0 (socket says 172.21.15.28:46794) 2024-04-25T01:22:29.896 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.836+0000 7fd730f38700 1 -- 172.21.15.28:0/4287806880 learned_addr learned my addr 172.21.15.28:0/4287806880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:29.896 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.836+0000 7fd730f38700 1 -- 172.21.15.28:0/4287806880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72c108d70 con 0x7fd72c107c10 2024-04-25T01:22:29.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.836+0000 7fd730f38700 1 --2- 172.21.15.28:0/4287806880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c107c10 0x7fd72c108030 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd7200098a0 tx=0x7fd720009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48204a5cdf6424d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:29.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.836+0000 7fd72b7fe700 1 -- 172.21.15.28:0/4287806880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd720004160 con 0x7fd72c107c10 2024-04-25T01:22:29.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.836+0000 7fd72b7fe700 1 -- 172.21.15.28:0/4287806880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd720004300 con 0x7fd72c107c10 2024-04-25T01:22:29.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.836+0000 7fd72b7fe700 1 -- 172.21.15.28:0/4287806880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd720011b50 con 0x7fd72c107c10 2024-04-25T01:22:29.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.837+0000 7fd73319c700 1 -- 172.21.15.28:0/4287806880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c107c10 msgr2=0x7fd72c108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.837+0000 7fd73319c700 1 --2- 172.21.15.28:0/4287806880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c107c10 0x7fd72c108030 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd7200098a0 tx=0x7fd720009bb0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.837+0000 7fd73319c700 1 -- 172.21.15.28:0/4287806880 shutdown_connections 2024-04-25T01:22:29.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.837+0000 7fd73319c700 1 --2- 172.21.15.28:0/4287806880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c107c10 0x7fd72c108030 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.837+0000 7fd73319c700 1 -- 172.21.15.28:0/4287806880 >> 172.21.15.28:0/4287806880 conn(0x7fd72c103230 msgr2=0x7fd72c105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:29.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.837+0000 7fd73319c700 1 -- 172.21.15.28:0/4287806880 shutdown_connections 2024-04-25T01:22:29.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.837+0000 7fd73319c700 1 -- 172.21.15.28:0/4287806880 wait complete. 2024-04-25T01:22:29.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.837+0000 7fd73319c700 1 Processor -- start 2024-04-25T01:22:29.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.837+0000 7fd73319c700 1 -- start start 2024-04-25T01:22:29.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.838+0000 7fd73319c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c107c10 0x7fd72c19bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:29.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.838+0000 7fd73319c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c10c120 con 0x7fd72c107c10 2024-04-25T01:22:29.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.838+0000 7fd730f38700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c107c10 0x7fd72c19bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:29.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.838+0000 7fd730f38700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c107c10 0x7fd72c19bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46810/0 (socket says 172.21.15.28:46810) 2024-04-25T01:22:29.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.838+0000 7fd730f38700 1 -- 172.21.15.28:0/2158685584 learned_addr learned my addr 172.21.15.28:0/2158685584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:29.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.838+0000 7fd730f38700 1 -- 172.21.15.28:0/2158685584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72c19c0d0 con 0x7fd72c107c10 2024-04-25T01:22:29.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.838+0000 7fd730f38700 1 --2- 172.21.15.28:0/2158685584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c107c10 0x7fd72c19bb90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd720011850 tx=0x7fd720011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:29.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.838+0000 7fd729ffb700 1 -- 172.21.15.28:0/2158685584 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd720003af0 con 0x7fd72c107c10 2024-04-25T01:22:29.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.838+0000 7fd729ffb700 1 -- 172.21.15.28:0/2158685584 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd720020070 con 0x7fd72c107c10 2024-04-25T01:22:29.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.838+0000 7fd73319c700 1 -- 172.21.15.28:0/2158685584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd72c19c360 con 0x7fd72c107c10 2024-04-25T01:22:29.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.838+0000 7fd729ffb700 1 -- 172.21.15.28:0/2158685584 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7200195a0 con 0x7fd72c107c10 2024-04-25T01:22:29.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.839+0000 7fd73319c700 1 -- 172.21.15.28:0/2158685584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd72c19f050 con 0x7fd72c107c10 2024-04-25T01:22:29.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.839+0000 7fd729ffb700 1 -- 172.21.15.28:0/2158685584 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd720019be0 con 0x7fd72c107c10 2024-04-25T01:22:29.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.839+0000 7fd729ffb700 1 -- 172.21.15.28:0/2158685584 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd720011e00 con 0x7fd72c107c10 2024-04-25T01:22:29.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.839+0000 7fd73319c700 1 -- 172.21.15.28:0/2158685584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd72c063890 con 0x7fd72c107c10 2024-04-25T01:22:29.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.842+0000 7fd729ffb700 1 -- 172.21.15.28:0/2158685584 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd720003c90 con 0x7fd72c107c10 2024-04-25T01:22:29.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.913+0000 7fd73319c700 1 -- 172.21.15.28:0/2158685584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fd72c19f220 con 0x7fd72c107c10 2024-04-25T01:22:29.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.913+0000 7fd729ffb700 1 -- 172.21.15.28:0/2158685584 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fd720025460 con 0x7fd72c107c10 2024-04-25T01:22:29.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.914+0000 7fd73319c700 1 -- 172.21.15.28:0/2158685584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c107c10 msgr2=0x7fd72c19bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:29.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.914+0000 7fd73319c700 1 --2- 172.21.15.28:0/2158685584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c107c10 0x7fd72c19bb90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd720011850 tx=0x7fd720011c10 comp rx=0 tx=0).stop 2024-04-25T01:22:29.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.914+0000 7fd73319c700 1 -- 172.21.15.28:0/2158685584 shutdown_connections 2024-04-25T01:22:29.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.914+0000 7fd73319c700 1 --2- 172.21.15.28:0/2158685584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd72c107c10 0x7fd72c19bb90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:29.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.914+0000 7fd73319c700 1 -- 172.21.15.28:0/2158685584 >> 172.21.15.28:0/2158685584 conn(0x7fd72c103230 msgr2=0x7fd72c103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:29.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.914+0000 7fd73319c700 1 -- 172.21.15.28:0/2158685584 shutdown_connections 2024-04-25T01:22:29.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:28.914+0000 7fd73319c700 1 -- 172.21.15.28:0/2158685584 wait complete. 2024-04-25T01:22:29.899 INFO:teuthology.orchestra.run.smithi028.stdout:Restarting the monitor... 2024-04-25T01:22:30.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:29 smithi028 systemd[1]: Stopping Ceph mon.a for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:22:30.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:30 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-a[29016]: 2024-04-25T01:22:30.186+0000 7f2cc7f93700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:22:30.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:30 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-a[29016]: 2024-04-25T01:22:30.186+0000 7f2cc7f93700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-25T01:22:30.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:30 smithi028 podman[29265]: 2024-04-25 01:22:30.421529307 +0000 UTC m=+0.331415634 container died 6857711b7e708a22486284f1b870abbced4e6312f4df28fe10157eb3f906a660 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9) 2024-04-25T01:22:30.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:30 smithi028 podman[29265]: 2024-04-25 01:22:30.793373049 +0000 UTC m=+0.703259371 container cleanup 6857711b7e708a22486284f1b870abbced4e6312f4df28fe10157eb3f906a660 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T01:22:30.992 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:30 smithi028 bash[29265]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-a 2024-04-25T01:22:31.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 podman[29279]: 2024-04-25 01:22:31.011426072 +0000 UTC m=+0.590379661 container remove 6857711b7e708a22486284f1b870abbced4e6312f4df28fe10157eb3f906a660 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:22:31.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 systemd[1]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-25T01:22:31.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 systemd[1]: Stopped Ceph mon.a for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:22:31.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 systemd[1]: Starting Ceph mon.a for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:22:31.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 podman[29371]: 2024-04-25 01:22:31.370133786 +0000 UTC m=+0.106755603 container create 8b32243128b3bd12db0cb949a7f1e4f95331c5318b93c36d3ba29cd333fb85b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424) 2024-04-25T01:22:31.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 podman[29371]: 2024-04-25 01:22:31.281776805 +0000 UTC m=+0.018398638 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:22:31.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 podman[29371]: 2024-04-25 01:22:31.565362728 +0000 UTC m=+0.301984546 container init 8b32243128b3bd12db0cb949a7f1e4f95331c5318b93c36d3ba29cd333fb85b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:22:31.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 podman[29371]: 2024-04-25 01:22:31.568525889 +0000 UTC m=+0.305147715 container start 8b32243128b3bd12db0cb949a7f1e4f95331c5318b93c36d3ba29cd333fb85b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T01:22:31.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T01:22:31.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev), process ceph-mon, pid 2 2024-04-25T01:22:31.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: pidfile_write: ignore empty --pid-file 2024-04-25T01:22:31.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: load: jerasure load: lrc 2024-04-25T01:22:31.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: RocksDB version: 7.9.2 2024-04-25T01:22:31.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Git sha 0 2024-04-25T01:22:31.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Compile date 2024-04-24 16:41:23 2024-04-25T01:22:31.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: DB SUMMARY 2024-04-25T01:22:31.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: DB Session ID: DNIXFJ6JA0DURYTDDX0O 2024-04-25T01:22:31.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: CURRENT file: CURRENT 2024-04-25T01:22:31.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T01:22:31.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-25T01:22:31.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-25T01:22:31.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104160 ; 2024-04-25T01:22:31.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.error_if_exists: 0 2024-04-25T01:22:31.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.create_if_missing: 0 2024-04-25T01:22:31.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.paranoid_checks: 1 2024-04-25T01:22:31.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T01:22:31.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T01:22:31.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T01:22:31.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.env: 0x559cc033ac60 2024-04-25T01:22:31.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T01:22:31.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.info_log: 0x559cc2187aa0 2024-04-25T01:22:31.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T01:22:31.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.statistics: (nil) 2024-04-25T01:22:31.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.use_fsync: 0 2024-04-25T01:22:31.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_log_file_size: 0 2024-04-25T01:22:31.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T01:22:31.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T01:22:31.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T01:22:31.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T01:22:31.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.allow_fallocate: 1 2024-04-25T01:22:31.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T01:22:31.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T01:22:31.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.use_direct_reads: 0 2024-04-25T01:22:31.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T01:22:31.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T01:22:31.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.db_log_dir: 2024-04-25T01:22:31.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.wal_dir: 2024-04-25T01:22:31.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T01:22:31.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T01:22:31.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T01:22:31.642 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T01:22:31.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T01:22:31.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T01:22:31.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T01:22:31.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T01:22:31.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.write_buffer_manager: 0x559cc215a1e0 2024-04-25T01:22:31.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T01:22:31.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T01:22:31.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T01:22:31.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T01:22:31.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T01:22:31.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T01:22:31.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T01:22:31.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T01:22:31.643 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.unordered_write: 0 2024-04-25T01:22:31.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T01:22:31.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T01:22:31.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T01:22:31.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T01:22:31.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.row_cache: None 2024-04-25T01:22:31.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.wal_filter: None 2024-04-25T01:22:31.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T01:22:31.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T01:22:31.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.two_write_queues: 0 2024-04-25T01:22:31.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T01:22:31.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.wal_compression: 0 2024-04-25T01:22:31.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.atomic_flush: 0 2024-04-25T01:22:31.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T01:22:31.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T01:22:31.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T01:22:31.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.log_readahead_size: 0 2024-04-25T01:22:31.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T01:22:31.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T01:22:31.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T01:22:31.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T01:22:31.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T01:22:31.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T01:22:31.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T01:22:31.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_background_jobs: 2 2024-04-25T01:22:31.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_background_compactions: -1 2024-04-25T01:22:31.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_subcompactions: 1 2024-04-25T01:22:31.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T01:22:31.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T01:22:31.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T01:22:31.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T01:22:31.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T01:22:31.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T01:22:31.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T01:22:31.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T01:22:31.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_open_files: -1 2024-04-25T01:22:31.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T01:22:31.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T01:22:31.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T01:22:31.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T01:22:31.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_background_flushes: -1 2024-04-25T01:22:31.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Compression algorithms supported: 2024-04-25T01:22:31.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T01:22:31.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: kZSTD supported: 0 2024-04-25T01:22:31.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: kXpressCompression supported: 0 2024-04-25T01:22:31.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T01:22:31.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: kZlibCompression supported: 1 2024-04-25T01:22:31.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: kSnappyCompression supported: 1 2024-04-25T01:22:31.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: kLZ4Compression supported: 1 2024-04-25T01:22:31.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: kBZip2Compression supported: 0 2024-04-25T01:22:31.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T01:22:31.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T01:22:31.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-25T01:22:31.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T01:22:31.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T01:22:31.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.merge_operator: 2024-04-25T01:22:31.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compaction_filter: None 2024-04-25T01:22:31.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T01:22:31.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T01:22:31.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T01:22:31.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T01:22:31.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559cc2187ba0) 2024-04-25T01:22:31.648 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-04-25T01:22:31.648 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T01:22:31.648 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T01:22:31.648 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-04-25T01:22:31.648 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_type: 0 2024-04-25T01:22:31.648 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_index_type: 0 2024-04-25T01:22:31.648 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_shortening: 1 2024-04-25T01:22:31.648 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T01:22:31.649 INFO:journalctl@ceph.mon.a.smithi028.stdout: checksum: 4 2024-04-25T01:22:31.649 INFO:journalctl@ceph.mon.a.smithi028.stdout: no_block_cache: 0 2024-04-25T01:22:31.649 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache: 0x559cc218b090 2024-04-25T01:22:31.649 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-04-25T01:22:31.649 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_options: 2024-04-25T01:22:31.649 INFO:journalctl@ceph.mon.a.smithi028.stdout: capacity : 536870912 2024-04-25T01:22:31.649 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_shard_bits : 4 2024-04-25T01:22:31.649 INFO:journalctl@ceph.mon.a.smithi028.stdout: strict_capacity_limit : 0 2024-04-25T01:22:31.649 INFO:journalctl@ceph.mon.a.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-04-25T01:22:31.649 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_compressed: (nil) 2024-04-25T01:22:31.649 INFO:journalctl@ceph.mon.a.smithi028.stdout: persistent_cache: (nil) 2024-04-25T01:22:31.649 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size: 4096 2024-04-25T01:22:31.649 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size_deviation: 10 2024-04-25T01:22:31.649 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_restart_interval: 16 2024-04-25T01:22:31.649 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_block_restart_interval: 1 2024-04-25T01:22:31.650 INFO:journalctl@ceph.mon.a.smithi028.stdout: metadata_block_size: 4096 2024-04-25T01:22:31.650 INFO:journalctl@ceph.mon.a.smithi028.stdout: partition_filters: 0 2024-04-25T01:22:31.650 INFO:journalctl@ceph.mon.a.smithi028.stdout: use_delta_encoding: 1 2024-04-25T01:22:31.650 INFO:journalctl@ceph.mon.a.smithi028.stdout: filter_policy: bloomfilter 2024-04-25T01:22:31.650 INFO:journalctl@ceph.mon.a.smithi028.stdout: whole_key_filtering: 1 2024-04-25T01:22:31.650 INFO:journalctl@ceph.mon.a.smithi028.stdout: verify_compression: 0 2024-04-25T01:22:31.650 INFO:journalctl@ceph.mon.a.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-04-25T01:22:31.650 INFO:journalctl@ceph.mon.a.smithi028.stdout: format_version: 5 2024-04-25T01:22:31.650 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_index_compression: 1 2024-04-25T01:22:31.650 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_align: 0 2024-04-25T01:22:31.650 INFO:journalctl@ceph.mon.a.smithi028.stdout: max_auto_readahead_size: 262144 2024-04-25T01:22:31.650 INFO:journalctl@ceph.mon.a.smithi028.stdout: prepopulate_block_cache: 0 2024-04-25T01:22:31.650 INFO:journalctl@ceph.mon.a.smithi028.stdout: initial_auto_readahead_size: 8192 2024-04-25T01:22:31.650 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T01:22:31.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T01:22:31.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T01:22:31.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compression: NoCompression 2024-04-25T01:22:31.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T01:22:31.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T01:22:31.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T01:22:31.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.num_levels: 7 2024-04-25T01:22:31.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T01:22:31.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T01:22:31.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T01:22:31.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T01:22:31.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T01:22:31.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T01:22:31.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T01:22:31.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T01:22:31.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T01:22:31.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T01:22:31.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T01:22:31.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T01:22:31.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T01:22:31.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T01:22:31.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T01:22:31.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T01:22:31.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T01:22:31.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T01:22:31.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T01:22:31.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T01:22:31.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T01:22:31.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T01:22:31.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T01:22:31.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T01:22:31.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T01:22:31.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T01:22:31.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T01:22:31.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T01:22:31.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T01:22:31.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T01:22:31.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T01:22:31.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T01:22:31.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T01:22:31.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T01:22:31.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T01:22:31.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T01:22:31.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T01:22:31.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T01:22:31.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T01:22:31.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T01:22:31.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T01:22:31.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T01:22:31.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T01:22:31.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T01:22:31.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T01:22:31.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T01:22:31.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T01:22:31.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T01:22:31.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T01:22:31.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T01:22:31.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T01:22:31.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T01:22:31.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T01:22:31.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.table_properties_collectors: 2024-04-25T01:22:31.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.inplace_update_support: 0 2024-04-25T01:22:31.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T01:22:31.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T01:22:31.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T01:22:31.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T01:22:31.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.bloom_locality: 0 2024-04-25T01:22:31.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.max_successive_merges: 0 2024-04-25T01:22:31.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T01:22:31.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T01:22:31.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T01:22:31.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T01:22:31.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.ttl: 2592000 2024-04-25T01:22:31.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T01:22:31.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T01:22:31.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T01:22:31.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.enable_blob_files: false 2024-04-25T01:22:31.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.min_blob_size: 0 2024-04-25T01:22:31.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T01:22:31.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T01:22:31.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T01:22:31.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T01:22:31.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T01:22:31.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T01:22:31.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T01:22:31.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T01:22:31.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-25T01:22:31.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-25T01:22:31.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5824f6f8-5330-43be-879d-9f277ec4e387 2024-04-25T01:22:31.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714008151597310, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-25T01:22:31.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-25T01:22:31.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714008151598304, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99365, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 97285, "index_size": 298, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14034, "raw_average_key_size": 49, "raw_value_size": 89621, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714008151, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5824f6f8-5330-43be-879d-9f277ec4e387", "db_session_id": "DNIXFJ6JA0DURYTDDX0O", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-25T01:22:31.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714008151598393, "job": 1, "event": "recovery_finished"} 2024-04-25T01:22:31.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-25T01:22:31.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T01:22:31.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559cc227c000 2024-04-25T01:22:31.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: DB pointer 0x559cc2266000 2024-04-25T01:22:31.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T01:22:31.657 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T01:22:31.657 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** DB Stats ** 2024-04-25T01:22:31.657 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T01:22:31.657 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T01:22:31.657 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T01:22:31.657 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T01:22:31.657 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T01:22:31.657 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T01:22:31.658 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T01:22:31.658 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-25T01:22:31.658 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-04-25T01:22:31.658 INFO:journalctl@ceph.mon.a.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T01:22:31.658 INFO:journalctl@ceph.mon.a.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T01:22:31.658 INFO:journalctl@ceph.mon.a.smithi028.stdout: L0 2/0 98.87 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:22:31.658 INFO:journalctl@ceph.mon.a.smithi028.stdout: Sum 2/0 98.87 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:22:31.658 INFO:journalctl@ceph.mon.a.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:22:31.658 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-25T01:22:31.658 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-04-25T01:22:31.658 INFO:journalctl@ceph.mon.a.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T01:22:31.658 INFO:journalctl@ceph.mon.a.smithi028.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T01:22:31.658 INFO:journalctl@ceph.mon.a.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 157.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:22:31.658 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-25T01:22:31.659 INFO:journalctl@ceph.mon.a.smithi028.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T01:22:31.659 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-25T01:22:31.659 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T01:22:31.659 INFO:journalctl@ceph.mon.a.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T01:22:31.659 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T01:22:31.659 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T01:22:31.659 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T01:22:31.659 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T01:22:31.659 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative compaction: 0.00 GB write, 30.73 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T01:22:31.659 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval compaction: 0.00 GB write, 30.73 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T01:22:31.659 INFO:journalctl@ceph.mon.a.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T01:22:31.659 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache BinnedLRUCache@0x559cc218b090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-25T01:22:31.659 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-25T01:22:31.659 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-25T01:22:31.659 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T01:22:31.659 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: starting mon.a rank 0 at public addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] at bind addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:22:31.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: mon.a@-1(???) e1 preinit fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:22:31.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: mon.a@-1(???).mds e1 new map 2024-04-25T01:22:31.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: mon.a@-1(???).mds e1 print_map 2024-04-25T01:22:31.660 INFO:journalctl@ceph.mon.a.smithi028.stdout: e1 2024-04-25T01:22:31.660 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T01:22:31.660 INFO:journalctl@ceph.mon.a.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:22:31.660 INFO:journalctl@ceph.mon.a.smithi028.stdout: legacy client fscid: -1 2024-04-25T01:22:31.660 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-25T01:22:31.660 INFO:journalctl@ceph.mon.a.smithi028.stdout: No filesystems configured 2024-04-25T01:22:31.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T01:22:31.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:22:31.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:22:31.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:22:31.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-25T01:22:31.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T01:22:31.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: monmap epoch 1 2024-04-25T01:22:31.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:22:31.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: last_changed 2024-04-25T01:22:20.217202+0000 2024-04-25T01:22:31.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: created 2024-04-25T01:22:20.217202+0000 2024-04-25T01:22:31.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: min_mon_release 19 (squid) 2024-04-25T01:22:31.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: election_strategy: 1 2024-04-25T01:22:31.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-25T01:22:31.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: fsmap 2024-04-25T01:22:31.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T01:22:31.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 ceph-mon[29390]: mgrmap e1: no daemons active 2024-04-25T01:22:31.674 INFO:teuthology.orchestra.run.smithi028.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-25T01:22:31.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 bash[29371]: 8b32243128b3bd12db0cb949a7f1e4f95331c5318b93c36d3ba29cd333fb85b5 2024-04-25T01:22:31.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:31 smithi028 systemd[1]: Started Ceph mon.a for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:22:32.723 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.121+0000 7f5827ec0700 1 Processor -- start 2024-04-25T01:22:32.724 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.121+0000 7f5827ec0700 1 -- start start 2024-04-25T01:22:32.724 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.122+0000 7f5827ec0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58201064a0 0x7f58201068c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.724 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.122+0000 7f5827ec0700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f582007be70 con 0x7f58201064a0 2024-04-25T01:22:32.724 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.122+0000 7f5825c5c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58201064a0 0x7f58201068c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.724 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.122+0000 7f5825c5c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58201064a0 0x7f58201068c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46814/0 (socket says 172.21.15.28:46814) 2024-04-25T01:22:32.724 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.122+0000 7f5825c5c700 1 -- 172.21.15.28:0/3554533210 learned_addr learned my addr 172.21.15.28:0/3554533210 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:32.724 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.122+0000 7f5825c5c700 1 -- 172.21.15.28:0/3554533210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5820106e00 con 0x7f58201064a0 2024-04-25T01:22:32.724 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.122+0000 7f5825c5c700 1 --2- 172.21.15.28:0/3554533210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58201064a0 0x7f58201068c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f581c0098a0 tx=0x7f581c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a21cf924afe06587 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:32.724 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.123+0000 7f5824c5a700 1 -- 172.21.15.28:0/3554533210 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f581c004160 con 0x7f58201064a0 2024-04-25T01:22:32.724 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.123+0000 7f5824c5a700 1 -- 172.21.15.28:0/3554533210 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f581c004300 con 0x7f58201064a0 2024-04-25T01:22:32.724 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.123+0000 7f5824c5a700 1 -- 172.21.15.28:0/3554533210 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f581c011b50 con 0x7f58201064a0 2024-04-25T01:22:32.724 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.123+0000 7f5827ec0700 1 -- 172.21.15.28:0/3554533210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58201064a0 msgr2=0x7f58201068c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.724 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.123+0000 7f5827ec0700 1 --2- 172.21.15.28:0/3554533210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58201064a0 0x7f58201068c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f581c0098a0 tx=0x7f581c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.725 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.123+0000 7f5827ec0700 1 -- 172.21.15.28:0/3554533210 shutdown_connections 2024-04-25T01:22:32.725 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.123+0000 7f5827ec0700 1 --2- 172.21.15.28:0/3554533210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58201064a0 0x7f58201068c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.725 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.123+0000 7f5827ec0700 1 -- 172.21.15.28:0/3554533210 >> 172.21.15.28:0/3554533210 conn(0x7f58201020a0 msgr2=0x7f58201044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:32.725 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.123+0000 7f5827ec0700 1 -- 172.21.15.28:0/3554533210 shutdown_connections 2024-04-25T01:22:32.725 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.124+0000 7f5827ec0700 1 -- 172.21.15.28:0/3554533210 wait complete. 2024-04-25T01:22:32.725 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.124+0000 7f5827ec0700 1 Processor -- start 2024-04-25T01:22:32.725 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.124+0000 7f5827ec0700 1 -- start start 2024-04-25T01:22:32.725 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.124+0000 7f5827ec0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58201064a0 0x7f582019dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:32.725 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.124+0000 7f5827ec0700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5820107e90 con 0x7f58201064a0 2024-04-25T01:22:32.725 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.124+0000 7f5825c5c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58201064a0 0x7f582019dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:32.725 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.124+0000 7f5825c5c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58201064a0 0x7f582019dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46828/0 (socket says 172.21.15.28:46828) 2024-04-25T01:22:32.725 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.124+0000 7f5825c5c700 1 -- 172.21.15.28:0/2326448839 learned_addr learned my addr 172.21.15.28:0/2326448839 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:32.725 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.125+0000 7f5825c5c700 1 -- 172.21.15.28:0/2326448839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f582019e290 con 0x7f58201064a0 2024-04-25T01:22:32.725 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.125+0000 7f5825c5c700 1 --2- 172.21.15.28:0/2326448839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58201064a0 0x7f582019dd50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f581c011b60 tx=0x7f581c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:32.726 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.125+0000 7f5816ffd700 1 -- 172.21.15.28:0/2326448839 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f581c003b20 con 0x7f58201064a0 2024-04-25T01:22:32.726 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.125+0000 7f5816ffd700 1 -- 172.21.15.28:0/2326448839 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f581c003cc0 con 0x7f58201064a0 2024-04-25T01:22:32.726 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.125+0000 7f5827ec0700 1 -- 172.21.15.28:0/2326448839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f582019e520 con 0x7f58201064a0 2024-04-25T01:22:32.726 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.125+0000 7f5816ffd700 1 -- 172.21.15.28:0/2326448839 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f581c024840 con 0x7f58201064a0 2024-04-25T01:22:32.726 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.125+0000 7f5827ec0700 1 -- 172.21.15.28:0/2326448839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58201a1210 con 0x7f58201064a0 2024-04-25T01:22:32.726 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.125+0000 7f5816ffd700 1 -- 172.21.15.28:0/2326448839 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f581c022020 con 0x7f58201064a0 2024-04-25T01:22:32.726 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.125+0000 7f5816ffd700 1 -- 172.21.15.28:0/2326448839 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f581c011e00 con 0x7f58201064a0 2024-04-25T01:22:32.726 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.126+0000 7f5827ec0700 1 -- 172.21.15.28:0/2326448839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5820063890 con 0x7f58201064a0 2024-04-25T01:22:32.726 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.128+0000 7f5816ffd700 1 -- 172.21.15.28:0/2326448839 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f581c043dd0 con 0x7f58201064a0 2024-04-25T01:22:32.726 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.199+0000 7f5827ec0700 1 -- 172.21.15.28:0/2326448839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f582002dd30 con 0x7f58201064a0 2024-04-25T01:22:32.726 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.201+0000 7f5816ffd700 1 -- 172.21.15.28:0/2326448839 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f581c032030 con 0x7f58201064a0 2024-04-25T01:22:32.726 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.201+0000 7f5827ec0700 1 -- 172.21.15.28:0/2326448839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58201064a0 msgr2=0x7f582019dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:32.726 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.201+0000 7f5827ec0700 1 --2- 172.21.15.28:0/2326448839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58201064a0 0x7f582019dd50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f581c011b60 tx=0x7f581c011c40 comp rx=0 tx=0).stop 2024-04-25T01:22:32.726 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.201+0000 7f5827ec0700 1 -- 172.21.15.28:0/2326448839 shutdown_connections 2024-04-25T01:22:32.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.201+0000 7f5827ec0700 1 --2- 172.21.15.28:0/2326448839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58201064a0 0x7f582019dd50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:32.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.201+0000 7f5827ec0700 1 -- 172.21.15.28:0/2326448839 >> 172.21.15.28:0/2326448839 conn(0x7f58201020a0 msgr2=0x7f5820194a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:32.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.201+0000 7f5827ec0700 1 -- 172.21.15.28:0/2326448839 shutdown_connections 2024-04-25T01:22:32.727 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:32.202+0000 7f5827ec0700 1 -- 172.21.15.28:0/2326448839 wait complete. 2024-04-25T01:22:32.727 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-25T01:22:32.749 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:22:32.750 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mgr... 2024-04-25T01:22:32.750 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-25T01:22:32.750 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-25T01:22:32.988 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mgr.x 2024-04-25T01:22:32.989 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-25T01:22:33.140 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0e88f60a-02a2-11ef-bc93-c7b262605968.target.wants/ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@.service. 2024-04-25T01:22:33.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:33 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2326448839' entity='client.admin' 2024-04-25T01:22:33.646 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:33 smithi028 podman[29602]: 2024-04-25 01:22:33.610938632 +0000 UTC m=+0.329103453 container start a137f351b2ca9b22bd1d139d127a8e8d14ff825ef5912f4adb93c7c9a7f55162 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-25T01:22:33.659 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:22:33.659 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-25T01:22:33.668 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:22:33.668 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-25T01:22:33.668 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-25T01:22:33.668 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T01:22:33.676 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:22:33.676 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-25T01:22:33.684 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:22:33.684 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-25T01:22:33.684 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-25T01:22:33.684 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-25T01:22:33.684 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr to start... 2024-04-25T01:22:33.684 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr... 2024-04-25T01:22:33.942 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:33 smithi028 bash[29602]: a137f351b2ca9b22bd1d139d127a8e8d14ff825ef5912f4adb93c7c9a7f55162 2024-04-25T01:22:33.942 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:33 smithi028 systemd[1]: Started Ceph mgr.x for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:22:33.943 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:33 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:33.768+0000 7f0587079200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:22:34.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:34 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2303282974' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:22:34.315 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:33 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:33.961+0000 7f0587079200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:22:34.315 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:34 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:34.070+0000 7f0587079200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:22:34.718 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:34 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:34.403+0000 7f0587079200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:22:34.719 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:34 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:34.491+0000 7f0587079200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:22:34.722 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-25T01:22:34.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:22:34.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "0e88f60a-02a2-11ef-bc93-c7b262605968", 2024-04-25T01:22:34.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:22:34.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:22:34.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:22:34.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:22:34.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:34.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:22:34.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:22:34.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:22:34.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:22:34.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:22:34.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:22:34.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:22:34.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-25T01:22:34.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:22:34.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:22:34.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:22:34.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:22:34.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:34.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:22:34.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:22:34.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:22:34.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:22:34.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:22:34.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:22:34.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:22:34.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:22:34.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:34.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:22:34.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:22:34.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:22:34.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:22:34.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:22:34.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:22:34.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:22:34.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:22:34.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:22:34.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:34.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:22:34.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:22:34.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:22:34.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:22:34.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:34.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:22:34.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T01:22:34.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:22:34.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:22:34.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:22:34.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:22:34.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:22:34.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:22:34.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:22:34.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:34.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:22:34.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:22:34.738 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:22:26.300124+0000", 2024-04-25T01:22:34.738 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:22:34.738 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:34.738 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:22:34.738 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:22:34.738 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.097+0000 7fd480cd9700 1 Processor -- start 2024-04-25T01:22:34.738 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.097+0000 7fd480cd9700 1 -- start start 2024-04-25T01:22:34.738 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.097+0000 7fd480cd9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c108720 0x7fd47c10ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:34.738 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.097+0000 7fd480cd9700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c07bdf0 con 0x7fd47c108720 2024-04-25T01:22:34.738 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.097+0000 7fd47a59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c108720 0x7fd47c10ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:34.738 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.097+0000 7fd47a59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c108720 0x7fd47c10ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46848/0 (socket says 172.21.15.28:46848) 2024-04-25T01:22:34.738 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.097+0000 7fd47a59c700 1 -- 172.21.15.28:0/3492056955 learned_addr learned my addr 172.21.15.28:0/3492056955 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:34.738 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.098+0000 7fd47a59c700 1 -- 172.21.15.28:0/3492056955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd47c10b050 con 0x7fd47c108720 2024-04-25T01:22:34.738 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.098+0000 7fd47a59c700 1 --2- 172.21.15.28:0/3492056955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c108720 0x7fd47c10ab10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd46c0098a0 tx=0x7fd46c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6be76ca49f10e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:34.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.098+0000 7fd47959a700 1 -- 172.21.15.28:0/3492056955 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd46c004160 con 0x7fd47c108720 2024-04-25T01:22:34.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.098+0000 7fd47959a700 1 -- 172.21.15.28:0/3492056955 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd46c004300 con 0x7fd47c108720 2024-04-25T01:22:34.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.098+0000 7fd47959a700 1 -- 172.21.15.28:0/3492056955 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd46c011b50 con 0x7fd47c108720 2024-04-25T01:22:34.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.099+0000 7fd480cd9700 1 -- 172.21.15.28:0/3492056955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c108720 msgr2=0x7fd47c10ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:34.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.099+0000 7fd480cd9700 1 --2- 172.21.15.28:0/3492056955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c108720 0x7fd47c10ab10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd46c0098a0 tx=0x7fd46c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.099+0000 7fd480cd9700 1 -- 172.21.15.28:0/3492056955 shutdown_connections 2024-04-25T01:22:34.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.099+0000 7fd480cd9700 1 --2- 172.21.15.28:0/3492056955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c108720 0x7fd47c10ab10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.099+0000 7fd480cd9700 1 -- 172.21.15.28:0/3492056955 >> 172.21.15.28:0/3492056955 conn(0x7fd47c102090 msgr2=0x7fd47c1044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:34.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.099+0000 7fd480cd9700 1 -- 172.21.15.28:0/3492056955 shutdown_connections 2024-04-25T01:22:34.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.099+0000 7fd480cd9700 1 -- 172.21.15.28:0/3492056955 wait complete. 2024-04-25T01:22:34.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.100+0000 7fd480cd9700 1 Processor -- start 2024-04-25T01:22:34.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.100+0000 7fd480cd9700 1 -- start start 2024-04-25T01:22:34.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.100+0000 7fd480cd9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c108720 0x7fd47c19bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:34.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.100+0000 7fd480cd9700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c10bfb0 con 0x7fd47c108720 2024-04-25T01:22:34.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.100+0000 7fd47a59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c108720 0x7fd47c19bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:34.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.100+0000 7fd47a59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c108720 0x7fd47c19bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46852/0 (socket says 172.21.15.28:46852) 2024-04-25T01:22:34.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.100+0000 7fd47a59c700 1 -- 172.21.15.28:0/2303282974 learned_addr learned my addr 172.21.15.28:0/2303282974 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:34.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.101+0000 7fd47a59c700 1 -- 172.21.15.28:0/2303282974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd47c19c060 con 0x7fd47c108720 2024-04-25T01:22:34.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.101+0000 7fd47a59c700 1 --2- 172.21.15.28:0/2303282974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c108720 0x7fd47c19bb20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd46c011670 tx=0x7fd46c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:34.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.101+0000 7fd46b7fe700 1 -- 172.21.15.28:0/2303282974 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd46c0039c0 con 0x7fd47c108720 2024-04-25T01:22:34.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.101+0000 7fd46b7fe700 1 -- 172.21.15.28:0/2303282974 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd46c011e50 con 0x7fd47c108720 2024-04-25T01:22:34.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.101+0000 7fd480cd9700 1 -- 172.21.15.28:0/2303282974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd47c19c2f0 con 0x7fd47c108720 2024-04-25T01:22:34.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.101+0000 7fd46b7fe700 1 -- 172.21.15.28:0/2303282974 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd46c01a5c0 con 0x7fd47c108720 2024-04-25T01:22:34.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.101+0000 7fd480cd9700 1 -- 172.21.15.28:0/2303282974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd47c19efe0 con 0x7fd47c108720 2024-04-25T01:22:34.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.101+0000 7fd46b7fe700 1 -- 172.21.15.28:0/2303282974 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd46c01ac00 con 0x7fd47c108720 2024-04-25T01:22:34.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.102+0000 7fd46b7fe700 1 -- 172.21.15.28:0/2303282974 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd46c019700 con 0x7fd47c108720 2024-04-25T01:22:34.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.102+0000 7fd480cd9700 1 -- 172.21.15.28:0/2303282974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd47c063890 con 0x7fd47c108720 2024-04-25T01:22:34.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.106+0000 7fd46b7fe700 1 -- 172.21.15.28:0/2303282974 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd46c003b60 con 0x7fd47c108720 2024-04-25T01:22:34.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.180+0000 7fd480cd9700 1 -- 172.21.15.28:0/2303282974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fd47c19f1b0 con 0x7fd47c108720 2024-04-25T01:22:34.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.181+0000 7fd46b7fe700 1 -- 172.21.15.28:0/2303282974 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fd46c0199a0 con 0x7fd47c108720 2024-04-25T01:22:34.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.182+0000 7fd480cd9700 1 -- 172.21.15.28:0/2303282974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c108720 msgr2=0x7fd47c19bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:34.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.182+0000 7fd480cd9700 1 --2- 172.21.15.28:0/2303282974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c108720 0x7fd47c19bb20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd46c011670 tx=0x7fd46c011b60 comp rx=0 tx=0).stop 2024-04-25T01:22:34.741 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.182+0000 7fd480cd9700 1 -- 172.21.15.28:0/2303282974 shutdown_connections 2024-04-25T01:22:34.741 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.182+0000 7fd480cd9700 1 --2- 172.21.15.28:0/2303282974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c108720 0x7fd47c19bb20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:34.741 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.182+0000 7fd480cd9700 1 -- 172.21.15.28:0/2303282974 >> 172.21.15.28:0/2303282974 conn(0x7fd47c102090 msgr2=0x7fd47c102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:34.741 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.183+0000 7fd480cd9700 1 -- 172.21.15.28:0/2303282974 shutdown_connections 2024-04-25T01:22:34.741 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:34.183+0000 7fd480cd9700 1 -- 172.21.15.28:0/2303282974 wait complete. 2024-04-25T01:22:34.741 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (1/15)... 2024-04-25T01:22:35.933 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:35 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:35.777+0000 7f0587079200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:22:35.933 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:35 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:35.953+0000 7f0587079200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:22:36.315 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:36 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:36.129+0000 7f0587079200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:22:36.712 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:36 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:36.426+0000 7f0587079200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:22:36.713 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:36 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:36.522+0000 7f0587079200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:22:36.713 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:36 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:36.732+0000 7f0587079200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:22:36.967 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:36 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:36.937+0000 7f0587079200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:22:37.519 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:37 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/574224216' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:22:37.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-25T01:22:37.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:22:37.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "0e88f60a-02a2-11ef-bc93-c7b262605968", 2024-04-25T01:22:37.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:22:37.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:22:37.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:22:37.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:22:37.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:37.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:22:37.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:22:37.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:22:37.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:22:37.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:22:37.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:22:37.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:22:37.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-25T01:22:37.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:22:37.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:22:37.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:22:37.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:22:37.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:37.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:22:37.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:22:37.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:22:37.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:22:37.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:22:37.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:22:37.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:22:37.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:22:37.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:37.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:22:37.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:22:37.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:22:37.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:22:37.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:22:37.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:22:37.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:22:37.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:22:37.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:22:37.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:37.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:22:37.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:22:37.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:22:37.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:22:37.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:37.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:22:37.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T01:22:37.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:22:37.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:22:37.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:22:37.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:22:37.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:22:37.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:22:37.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:22:37.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:37.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:22:37.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:22:37.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:22:26.300124+0000", 2024-04-25T01:22:37.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:22:37.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:37.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:22:37.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:22:37.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.167+0000 7fe3cc04d700 1 Processor -- start 2024-04-25T01:22:37.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.167+0000 7fe3cc04d700 1 -- start start 2024-04-25T01:22:37.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.168+0000 7fe3cc04d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3c4108660 0x7fe3c410aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.168+0000 7fe3cc04d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c407be90 con 0x7fe3c4108660 2024-04-25T01:22:37.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.168+0000 7fe3c9de9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3c4108660 0x7fe3c410aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.168+0000 7fe3c9de9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3c4108660 0x7fe3c410aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39896/0 (socket says 172.21.15.28:39896) 2024-04-25T01:22:37.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.168+0000 7fe3c9de9700 1 -- 172.21.15.28:0/241007520 learned_addr learned my addr 172.21.15.28:0/241007520 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:37.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.168+0000 7fe3c9de9700 1 -- 172.21.15.28:0/241007520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c410af90 con 0x7fe3c4108660 2024-04-25T01:22:37.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.168+0000 7fe3c9de9700 1 --2- 172.21.15.28:0/241007520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3c4108660 0x7fe3c410aa50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00098a0 tx=0x7fe3c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=141e37a4c735dafc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:37.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.169+0000 7fe3c8de7700 1 -- 172.21.15.28:0/241007520 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3c0004160 con 0x7fe3c4108660 2024-04-25T01:22:37.733 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.169+0000 7fe3c8de7700 1 -- 172.21.15.28:0/241007520 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe3c0004300 con 0x7fe3c4108660 2024-04-25T01:22:37.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.169+0000 7fe3c8de7700 1 -- 172.21.15.28:0/241007520 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3c0011b50 con 0x7fe3c4108660 2024-04-25T01:22:37.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.169+0000 7fe3cc04d700 1 -- 172.21.15.28:0/241007520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3c4108660 msgr2=0x7fe3c410aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.169+0000 7fe3cc04d700 1 --2- 172.21.15.28:0/241007520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3c4108660 0x7fe3c410aa50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00098a0 tx=0x7fe3c0009bb0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.169+0000 7fe3cc04d700 1 -- 172.21.15.28:0/241007520 shutdown_connections 2024-04-25T01:22:37.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.169+0000 7fe3cc04d700 1 --2- 172.21.15.28:0/241007520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3c4108660 0x7fe3c410aa50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.169+0000 7fe3cc04d700 1 -- 172.21.15.28:0/241007520 >> 172.21.15.28:0/241007520 conn(0x7fe3c4102070 msgr2=0x7fe3c4104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:37.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.169+0000 7fe3cc04d700 1 -- 172.21.15.28:0/241007520 shutdown_connections 2024-04-25T01:22:37.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.170+0000 7fe3cc04d700 1 -- 172.21.15.28:0/241007520 wait complete. 2024-04-25T01:22:37.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.170+0000 7fe3cc04d700 1 Processor -- start 2024-04-25T01:22:37.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.170+0000 7fe3cc04d700 1 -- start start 2024-04-25T01:22:37.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.170+0000 7fe3cc04d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3c4108660 0x7fe3c4107ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:37.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.170+0000 7fe3cc04d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c410bfb0 con 0x7fe3c4108660 2024-04-25T01:22:37.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.170+0000 7fe3c9de9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3c4108660 0x7fe3c4107ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:37.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.170+0000 7fe3c9de9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3c4108660 0x7fe3c4107ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39910/0 (socket says 172.21.15.28:39910) 2024-04-25T01:22:37.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.170+0000 7fe3c9de9700 1 -- 172.21.15.28:0/574224216 learned_addr learned my addr 172.21.15.28:0/574224216 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:37.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.171+0000 7fe3c9de9700 1 -- 172.21.15.28:0/574224216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c4108420 con 0x7fe3c4108660 2024-04-25T01:22:37.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.171+0000 7fe3c9de9700 1 --2- 172.21.15.28:0/574224216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3c4108660 0x7fe3c4107ee0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0011670 tx=0x7fe3c0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:37.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.171+0000 7fe3baffd700 1 -- 172.21.15.28:0/574224216 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3c00039c0 con 0x7fe3c4108660 2024-04-25T01:22:37.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.171+0000 7fe3cc04d700 1 -- 172.21.15.28:0/574224216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c4106610 con 0x7fe3c4108660 2024-04-25T01:22:37.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.171+0000 7fe3baffd700 1 -- 172.21.15.28:0/574224216 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe3c0011e50 con 0x7fe3c4108660 2024-04-25T01:22:37.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.171+0000 7fe3baffd700 1 -- 172.21.15.28:0/574224216 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3c001a5c0 con 0x7fe3c4108660 2024-04-25T01:22:37.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.171+0000 7fe3cc04d700 1 -- 172.21.15.28:0/574224216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c4106b50 con 0x7fe3c4108660 2024-04-25T01:22:37.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.171+0000 7fe3baffd700 1 -- 172.21.15.28:0/574224216 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe3c001ac50 con 0x7fe3c4108660 2024-04-25T01:22:37.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.171+0000 7fe3baffd700 1 -- 172.21.15.28:0/574224216 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe3c0019700 con 0x7fe3c4108660 2024-04-25T01:22:37.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.172+0000 7fe3cc04d700 1 -- 172.21.15.28:0/574224216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3c4063890 con 0x7fe3c4108660 2024-04-25T01:22:37.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.176+0000 7fe3baffd700 1 -- 172.21.15.28:0/574224216 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe3c0003b60 con 0x7fe3c4108660 2024-04-25T01:22:37.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.250+0000 7fe3cc04d700 1 -- 172.21.15.28:0/574224216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fe3c4107080 con 0x7fe3c4108660 2024-04-25T01:22:37.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.250+0000 7fe3baffd700 1 -- 172.21.15.28:0/574224216 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fe3c00199a0 con 0x7fe3c4108660 2024-04-25T01:22:37.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.252+0000 7fe3cc04d700 1 -- 172.21.15.28:0/574224216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3c4108660 msgr2=0x7fe3c4107ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:37.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.252+0000 7fe3cc04d700 1 --2- 172.21.15.28:0/574224216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3c4108660 0x7fe3c4107ee0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0011670 tx=0x7fe3c0011b60 comp rx=0 tx=0).stop 2024-04-25T01:22:37.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.252+0000 7fe3cc04d700 1 -- 172.21.15.28:0/574224216 shutdown_connections 2024-04-25T01:22:37.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.252+0000 7fe3cc04d700 1 --2- 172.21.15.28:0/574224216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3c4108660 0x7fe3c4107ee0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:37.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.252+0000 7fe3cc04d700 1 -- 172.21.15.28:0/574224216 >> 172.21.15.28:0/574224216 conn(0x7fe3c4102070 msgr2=0x7fe3c4102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:37.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.252+0000 7fe3cc04d700 1 -- 172.21.15.28:0/574224216 shutdown_connections 2024-04-25T01:22:37.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:37.252+0000 7fe3cc04d700 1 -- 172.21.15.28:0/574224216 wait complete. 2024-04-25T01:22:37.736 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (2/15)... 2024-04-25T01:22:37.815 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:37 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:37.587+0000 7f0587079200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:22:38.119 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:38 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:38.037+0000 7f0587079200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:22:38.119 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:38 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:38.137+0000 7f0587079200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:22:38.494 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:38 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:38.347+0000 7f0587079200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:22:39.220 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:38 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:38.893+0000 7f0587079200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:22:39.220 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:38 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:38.988+0000 7f0587079200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:22:39.565 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:39 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:39.238+0000 7f0587079200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:22:39.565 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:39 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:39.377+0000 7f0587079200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:22:39.565 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:39 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:39.475+0000 7f0587079200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:22:39.855 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:39 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:39.595+0000 7f0587079200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:22:39.855 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:39 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:39.690+0000 7f0587079200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:22:40.531 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:40 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2818533912' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:22:40.815 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:40 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:40.549+0000 7f0587079200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:22:40.815 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:40 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:40.773+0000 7f0587079200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:22:40.869 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-25T01:22:40.869 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:22:40.869 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "0e88f60a-02a2-11ef-bc93-c7b262605968", 2024-04-25T01:22:40.869 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:22:40.870 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:22:40.870 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:22:40.870 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:22:40.870 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:40.870 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:22:40.870 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:22:40.870 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:22:40.870 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:22:40.870 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:22:40.870 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:22:40.870 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:22:40.870 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-25T01:22:40.870 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:22:40.870 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:22:40.871 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:22:40.871 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:22:40.871 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:40.871 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:22:40.871 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:22:40.871 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:22:40.871 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:22:40.871 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:22:40.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:22:40.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:22:40.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:22:40.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:40.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:22:40.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:22:40.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:22:40.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:22:40.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:22:40.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:22:40.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:22:40.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:22:40.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:22:40.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:40.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:22:40.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:22:40.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:22:40.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:22:40.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:40.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:22:40.881 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T01:22:40.881 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:22:40.881 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:22:40.881 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:22:40.881 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:22:40.881 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:22:40.881 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:22:40.881 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:22:40.881 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:40.881 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:22:40.881 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:22:40.881 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:22:26.300124+0000", 2024-04-25T01:22:40.881 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:22:40.881 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:40.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:22:40.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:22:40.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.173+0000 7eff09a0a700 1 Processor -- start 2024-04-25T01:22:40.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.173+0000 7eff09a0a700 1 -- start start 2024-04-25T01:22:40.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.173+0000 7eff09a0a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff04105ec0 0x7eff041062e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:40.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.173+0000 7eff09a0a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff041068b0 con 0x7eff04105ec0 2024-04-25T01:22:40.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.173+0000 7eff02ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff04105ec0 0x7eff041062e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:40.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.173+0000 7eff02ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff04105ec0 0x7eff041062e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39922/0 (socket says 172.21.15.28:39922) 2024-04-25T01:22:40.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.173+0000 7eff02ffd700 1 -- 172.21.15.28:0/409780115 learned_addr learned my addr 172.21.15.28:0/409780115 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:40.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.173+0000 7eff02ffd700 1 -- 172.21.15.28:0/409780115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff04108e50 con 0x7eff04105ec0 2024-04-25T01:22:40.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.174+0000 7eff02ffd700 1 --2- 172.21.15.28:0/409780115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff04105ec0 0x7eff041062e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efef40098a0 tx=0x7efef4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b104a6800c8e63c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:40.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.174+0000 7eff01ffb700 1 -- 172.21.15.28:0/409780115 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efef4004160 con 0x7eff04105ec0 2024-04-25T01:22:40.882 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.174+0000 7eff01ffb700 1 -- 172.21.15.28:0/409780115 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efef4004300 con 0x7eff04105ec0 2024-04-25T01:22:40.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.174+0000 7eff01ffb700 1 -- 172.21.15.28:0/409780115 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efef4011b50 con 0x7eff04105ec0 2024-04-25T01:22:40.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.174+0000 7eff09a0a700 1 -- 172.21.15.28:0/409780115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff04105ec0 msgr2=0x7eff041062e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:40.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.174+0000 7eff09a0a700 1 --2- 172.21.15.28:0/409780115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff04105ec0 0x7eff041062e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efef40098a0 tx=0x7efef4009bb0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.174+0000 7eff09a0a700 1 -- 172.21.15.28:0/409780115 shutdown_connections 2024-04-25T01:22:40.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.174+0000 7eff09a0a700 1 --2- 172.21.15.28:0/409780115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff04105ec0 0x7eff041062e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.174+0000 7eff09a0a700 1 -- 172.21.15.28:0/409780115 >> 172.21.15.28:0/409780115 conn(0x7eff04101830 msgr2=0x7eff04103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:40.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.174+0000 7eff09a0a700 1 -- 172.21.15.28:0/409780115 shutdown_connections 2024-04-25T01:22:40.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.174+0000 7eff09a0a700 1 -- 172.21.15.28:0/409780115 wait complete. 2024-04-25T01:22:40.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.175+0000 7eff09a0a700 1 Processor -- start 2024-04-25T01:22:40.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.175+0000 7eff09a0a700 1 -- start start 2024-04-25T01:22:40.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.175+0000 7eff09a0a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff04105ec0 0x7eff0419bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:40.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.175+0000 7eff09a0a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff0410c200 con 0x7eff04105ec0 2024-04-25T01:22:40.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.176+0000 7eff02ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff04105ec0 0x7eff0419bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:40.883 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.176+0000 7eff02ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff04105ec0 0x7eff0419bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39938/0 (socket says 172.21.15.28:39938) 2024-04-25T01:22:40.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.176+0000 7eff02ffd700 1 -- 172.21.15.28:0/2818533912 learned_addr learned my addr 172.21.15.28:0/2818533912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:40.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.176+0000 7eff02ffd700 1 -- 172.21.15.28:0/2818533912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff0419c0e0 con 0x7eff04105ec0 2024-04-25T01:22:40.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.176+0000 7eff02ffd700 1 --2- 172.21.15.28:0/2818533912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff04105ec0 0x7eff0419bba0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efef4011670 tx=0x7efef4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:40.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.176+0000 7eff08a08700 1 -- 172.21.15.28:0/2818533912 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efef40039c0 con 0x7eff04105ec0 2024-04-25T01:22:40.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.176+0000 7eff08a08700 1 -- 172.21.15.28:0/2818533912 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efef4011e50 con 0x7eff04105ec0 2024-04-25T01:22:40.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.176+0000 7eff09a0a700 1 -- 172.21.15.28:0/2818533912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff0419c370 con 0x7eff04105ec0 2024-04-25T01:22:40.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.177+0000 7eff08a08700 1 -- 172.21.15.28:0/2818533912 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efef401a5c0 con 0x7eff04105ec0 2024-04-25T01:22:40.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.177+0000 7eff09a0a700 1 -- 172.21.15.28:0/2818533912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff0419f060 con 0x7eff04105ec0 2024-04-25T01:22:40.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.177+0000 7eff08a08700 1 -- 172.21.15.28:0/2818533912 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efef401ac50 con 0x7eff04105ec0 2024-04-25T01:22:40.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.177+0000 7eff08a08700 1 -- 172.21.15.28:0/2818533912 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efef40247c0 con 0x7eff04105ec0 2024-04-25T01:22:40.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.178+0000 7eff09a0a700 1 -- 172.21.15.28:0/2818533912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff04063890 con 0x7eff04105ec0 2024-04-25T01:22:40.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.180+0000 7eff08a08700 1 -- 172.21.15.28:0/2818533912 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7efef4003b60 con 0x7eff04105ec0 2024-04-25T01:22:40.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.255+0000 7eff09a0a700 1 -- 172.21.15.28:0/2818533912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7eff0419f2d0 con 0x7eff04105ec0 2024-04-25T01:22:40.884 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.255+0000 7eff08a08700 1 -- 172.21.15.28:0/2818533912 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7efef400b660 con 0x7eff04105ec0 2024-04-25T01:22:40.885 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.256+0000 7eff09a0a700 1 -- 172.21.15.28:0/2818533912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff04105ec0 msgr2=0x7eff0419bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:40.885 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.256+0000 7eff09a0a700 1 --2- 172.21.15.28:0/2818533912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff04105ec0 0x7eff0419bba0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efef4011670 tx=0x7efef4011b60 comp rx=0 tx=0).stop 2024-04-25T01:22:40.885 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.256+0000 7eff09a0a700 1 -- 172.21.15.28:0/2818533912 shutdown_connections 2024-04-25T01:22:40.885 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.256+0000 7eff09a0a700 1 --2- 172.21.15.28:0/2818533912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff04105ec0 0x7eff0419bba0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:40.885 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.256+0000 7eff09a0a700 1 -- 172.21.15.28:0/2818533912 >> 172.21.15.28:0/2818533912 conn(0x7eff04101830 msgr2=0x7eff04102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:40.885 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.256+0000 7eff09a0a700 1 -- 172.21.15.28:0/2818533912 shutdown_connections 2024-04-25T01:22:40.885 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:40.256+0000 7eff09a0a700 1 -- 172.21.15.28:0/2818533912 wait complete. 2024-04-25T01:22:40.885 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (3/15)... 2024-04-25T01:22:41.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:41 smithi028 ceph-mon[29390]: Activating manager daemon x 2024-04-25T01:22:41.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:41 smithi028 ceph-mon[29390]: mgrmap e2: x(active, starting, since 0.0033809s) 2024-04-25T01:22:41.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:41 smithi028 ceph-mon[29390]: from='mgr.14100 172.21.15.28:0/1764507134' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:22:41.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:41 smithi028 ceph-mon[29390]: from='mgr.14100 172.21.15.28:0/1764507134' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:22:41.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:41 smithi028 ceph-mon[29390]: from='mgr.14100 172.21.15.28:0/1764507134' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:22:41.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:41 smithi028 ceph-mon[29390]: from='mgr.14100 172.21.15.28:0/1764507134' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:22:41.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:41 smithi028 ceph-mon[29390]: from='mgr.14100 172.21.15.28:0/1764507134' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:22:41.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:41 smithi028 ceph-mon[29390]: Manager daemon x is now available 2024-04-25T01:22:41.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:41 smithi028 ceph-mon[29390]: from='mgr.14100 172.21.15.28:0/1764507134' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:22:41.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:41 smithi028 ceph-mon[29390]: from='mgr.14100 172.21.15.28:0/1764507134' entity='mgr.x' 2024-04-25T01:22:41.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:41 smithi028 ceph-mon[29390]: from='mgr.14100 172.21.15.28:0/1764507134' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:22:41.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:41 smithi028 ceph-mon[29390]: from='mgr.14100 172.21.15.28:0/1764507134' entity='mgr.x' 2024-04-25T01:22:41.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:41 smithi028 ceph-mon[29390]: from='mgr.14100 172.21.15.28:0/1764507134' entity='mgr.x' 2024-04-25T01:22:43.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:42 smithi028 ceph-mon[29390]: mgrmap e3: x(active, since 1.00746s) 2024-04-25T01:22:43.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:43 smithi028 ceph-mon[29390]: mgrmap e4: x(active, since 2s) 2024-04-25T01:22:43.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:43 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2472392076' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:22:44.130 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-25T01:22:44.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:22:44.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "0e88f60a-02a2-11ef-bc93-c7b262605968", 2024-04-25T01:22:44.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:22:44.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:22:44.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:22:44.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:22:44.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:44.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:22:44.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:22:44.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:22:44.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:22:44.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:22:44.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:22:44.133 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:22:44.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-04-25T01:22:44.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:22:44.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:22:44.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:22:44.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:22:44.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:44.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:22:44.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:22:44.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:22:44.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:22:44.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:22:44.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:22:44.134 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:22:44.143 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:22:44.143 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:44.143 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:22:44.143 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:22:44.143 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:22:44.143 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:22:44.143 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:22:44.143 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:22:44.143 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:22:44.143 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:22:44.143 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:22:44.143 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:44.144 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:22:44.144 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:22:44.144 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:22:44.144 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:22:44.144 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:44.144 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:22:44.144 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T01:22:44.144 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:22:44.144 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:22:44.144 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:22:44.144 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:22:44.144 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:22:44.144 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:22:44.144 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:22:44.144 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:44.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:22:44.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:22:44.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:22:26.300124+0000", 2024-04-25T01:22:44.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:22:44.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:22:44.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:22:44.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:22:44.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.279+0000 7f014b816700 1 Processor -- start 2024-04-25T01:22:44.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.279+0000 7f014b816700 1 -- start start 2024-04-25T01:22:44.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.279+0000 7f014b816700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0144107bf0 0x7f0144108010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.279+0000 7f014b816700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01441085e0 con 0x7f0144107bf0 2024-04-25T01:22:44.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.279+0000 7f01495b2700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0144107bf0 0x7f0144108010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.280+0000 7f01495b2700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0144107bf0 0x7f0144108010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39996/0 (socket says 172.21.15.28:39996) 2024-04-25T01:22:44.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.280+0000 7f01495b2700 1 -- 172.21.15.28:0/311946961 learned_addr learned my addr 172.21.15.28:0/311946961 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:44.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.280+0000 7f01495b2700 1 -- 172.21.15.28:0/311946961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0144108da0 con 0x7f0144107bf0 2024-04-25T01:22:44.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.280+0000 7f01495b2700 1 --2- 172.21.15.28:0/311946961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0144107bf0 0x7f0144108010 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f013c009e80 tx=0x7f013c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=712702202b5eb98f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:44.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.280+0000 7f0143fff700 1 -- 172.21.15.28:0/311946961 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f013c017070 con 0x7f0144107bf0 2024-04-25T01:22:44.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.280+0000 7f0143fff700 1 -- 172.21.15.28:0/311946961 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f013c00b8c0 con 0x7f0144107bf0 2024-04-25T01:22:44.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.280+0000 7f0143fff700 1 -- 172.21.15.28:0/311946961 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f013c0045c0 con 0x7f0144107bf0 2024-04-25T01:22:44.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.281+0000 7f014b816700 1 -- 172.21.15.28:0/311946961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0144107bf0 msgr2=0x7f0144108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.281+0000 7f014b816700 1 --2- 172.21.15.28:0/311946961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0144107bf0 0x7f0144108010 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f013c009e80 tx=0x7f013c00b330 comp rx=0 tx=0).stop 2024-04-25T01:22:44.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.281+0000 7f014b816700 1 -- 172.21.15.28:0/311946961 shutdown_connections 2024-04-25T01:22:44.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.281+0000 7f014b816700 1 --2- 172.21.15.28:0/311946961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0144107bf0 0x7f0144108010 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.281+0000 7f014b816700 1 -- 172.21.15.28:0/311946961 >> 172.21.15.28:0/311946961 conn(0x7f0144103170 msgr2=0x7f01441055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:44.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.281+0000 7f014b816700 1 -- 172.21.15.28:0/311946961 shutdown_connections 2024-04-25T01:22:44.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.281+0000 7f014b816700 1 -- 172.21.15.28:0/311946961 wait complete. 2024-04-25T01:22:44.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.281+0000 7f014b816700 1 Processor -- start 2024-04-25T01:22:44.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.282+0000 7f014b816700 1 -- start start 2024-04-25T01:22:44.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.282+0000 7f014b816700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0144107bf0 0x7f014407db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.282+0000 7f014b816700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f014410c150 con 0x7f0144107bf0 2024-04-25T01:22:44.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.282+0000 7f01495b2700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0144107bf0 0x7f014407db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.282+0000 7f01495b2700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0144107bf0 0x7f014407db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40004/0 (socket says 172.21.15.28:40004) 2024-04-25T01:22:44.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.282+0000 7f01495b2700 1 -- 172.21.15.28:0/2472392076 learned_addr learned my addr 172.21.15.28:0/2472392076 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:44.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.282+0000 7f01495b2700 1 -- 172.21.15.28:0/2472392076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f014407e0d0 con 0x7f0144107bf0 2024-04-25T01:22:44.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.282+0000 7f01495b2700 1 --2- 172.21.15.28:0/2472392076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0144107bf0 0x7f014407db90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f013c0072a0 tx=0x7f013c00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:44.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.282+0000 7f01427fc700 1 -- 172.21.15.28:0/2472392076 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f013c020020 con 0x7f0144107bf0 2024-04-25T01:22:44.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.282+0000 7f014b816700 1 -- 172.21.15.28:0/2472392076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f014407c2c0 con 0x7f0144107bf0 2024-04-25T01:22:44.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.282+0000 7f01427fc700 1 -- 172.21.15.28:0/2472392076 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f013c003b50 con 0x7f0144107bf0 2024-04-25T01:22:44.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.282+0000 7f01427fc700 1 -- 172.21.15.28:0/2472392076 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f013c017040 con 0x7f0144107bf0 2024-04-25T01:22:44.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.283+0000 7f014b816700 1 -- 172.21.15.28:0/2472392076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f014407c800 con 0x7f0144107bf0 2024-04-25T01:22:44.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.283+0000 7f01427fc700 1 -- 172.21.15.28:0/2472392076 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f013c023020 con 0x7f0144107bf0 2024-04-25T01:22:44.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.283+0000 7f01427fc700 1 --2- 172.21.15.28:0/2472392076 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f012c039000 0x7f012c03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:44.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.283+0000 7f01427fc700 1 -- 172.21.15.28:0/2472392076 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f013c00f050 con 0x7f0144107bf0 2024-04-25T01:22:44.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.283+0000 7f014b816700 1 -- 172.21.15.28:0/2472392076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0144063890 con 0x7f0144107bf0 2024-04-25T01:22:44.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.284+0000 7f0148db1700 1 --2- 172.21.15.28:0/2472392076 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f012c039000 0x7f012c03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:44.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.284+0000 7f0148db1700 1 --2- 172.21.15.28:0/2472392076 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f012c039000 0x7f012c03b4c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0138006fd0 tx=0x7f0138006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:44.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.289+0000 7f01427fc700 1 -- 172.21.15.28:0/2472392076 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f013c02a020 con 0x7f0144107bf0 2024-04-25T01:22:44.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.579+0000 7f014b816700 1 -- 172.21.15.28:0/2472392076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f014407c9d0 con 0x7f0144107bf0 2024-04-25T01:22:44.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.580+0000 7f01427fc700 1 -- 172.21.15.28:0/2472392076 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f013c003d70 con 0x7f0144107bf0 2024-04-25T01:22:44.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.582+0000 7f014b816700 1 -- 172.21.15.28:0/2472392076 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f012c039000 msgr2=0x7f012c03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.582+0000 7f014b816700 1 --2- 172.21.15.28:0/2472392076 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f012c039000 0x7f012c03b4c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0138006fd0 tx=0x7f0138006e40 comp rx=0 tx=0).stop 2024-04-25T01:22:44.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.582+0000 7f014b816700 1 -- 172.21.15.28:0/2472392076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0144107bf0 msgr2=0x7f014407db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:44.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.582+0000 7f014b816700 1 --2- 172.21.15.28:0/2472392076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0144107bf0 0x7f014407db90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f013c0072a0 tx=0x7f013c00bd50 comp rx=0 tx=0).stop 2024-04-25T01:22:44.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.582+0000 7f014b816700 1 -- 172.21.15.28:0/2472392076 shutdown_connections 2024-04-25T01:22:44.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.582+0000 7f014b816700 1 --2- 172.21.15.28:0/2472392076 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f012c039000 0x7f012c03b4c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.582+0000 7f014b816700 1 --2- 172.21.15.28:0/2472392076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0144107bf0 0x7f014407db90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:44.149 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.582+0000 7f014b816700 1 -- 172.21.15.28:0/2472392076 >> 172.21.15.28:0/2472392076 conn(0x7f0144103170 msgr2=0x7f0144103ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:44.149 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.582+0000 7f014b816700 1 -- 172.21.15.28:0/2472392076 shutdown_connections 2024-04-25T01:22:44.149 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:43.582+0000 7f014b816700 1 -- 172.21.15.28:0/2472392076 wait complete. 2024-04-25T01:22:44.149 INFO:teuthology.orchestra.run.smithi028.stdout:mgr is available 2024-04-25T01:22:45.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:44 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1498158195' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T01:22:45.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-25T01:22:45.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T01:22:45.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:22:45.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T01:22:45.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T01:22:45.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T01:22:45.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T01:22:45.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-25T01:22:45.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T01:22:45.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T01:22:45.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-25T01:22:45.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T01:22:45.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T01:22:45.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T01:22:45.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T01:22:45.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T01:22:45.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T01:22:45.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.600+0000 7f57a72de700 1 Processor -- start 2024-04-25T01:22:45.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.600+0000 7f57a72de700 1 -- start start 2024-04-25T01:22:45.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.600+0000 7f57a72de700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a007de50 0x7f57a007c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.600+0000 7f57a72de700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57a007c840 con 0x7f57a007de50 2024-04-25T01:22:45.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.601+0000 7f57a507a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a007de50 0x7f57a007c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:45.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.601+0000 7f57a507a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a007de50 0x7f57a007c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40016/0 (socket says 172.21.15.28:40016) 2024-04-25T01:22:45.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.601+0000 7f57a507a700 1 -- 172.21.15.28:0/3439937763 learned_addr learned my addr 172.21.15.28:0/3439937763 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:45.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.601+0000 7f57a507a700 1 -- 172.21.15.28:0/3439937763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57a007c9c0 con 0x7f57a007de50 2024-04-25T01:22:45.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.601+0000 7f57a507a700 1 --2- 172.21.15.28:0/3439937763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a007de50 0x7f57a007c300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f579c0098a0 tx=0x7f579c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61fcb83525520b13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:45.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.601+0000 7f5797fff700 1 -- 172.21.15.28:0/3439937763 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f579c004160 con 0x7f57a007de50 2024-04-25T01:22:45.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.601+0000 7f5797fff700 1 -- 172.21.15.28:0/3439937763 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f579c004300 con 0x7f57a007de50 2024-04-25T01:22:45.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.601+0000 7f5797fff700 1 -- 172.21.15.28:0/3439937763 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f579c011b50 con 0x7f57a007de50 2024-04-25T01:22:45.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.602+0000 7f57a72de700 1 -- 172.21.15.28:0/3439937763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a007de50 msgr2=0x7f57a007c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.602+0000 7f57a72de700 1 --2- 172.21.15.28:0/3439937763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a007de50 0x7f57a007c300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f579c0098a0 tx=0x7f579c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.602+0000 7f57a72de700 1 -- 172.21.15.28:0/3439937763 shutdown_connections 2024-04-25T01:22:45.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.602+0000 7f57a72de700 1 --2- 172.21.15.28:0/3439937763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a007de50 0x7f57a007c300 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.602+0000 7f57a72de700 1 -- 172.21.15.28:0/3439937763 >> 172.21.15.28:0/3439937763 conn(0x7f57a0103170 msgr2=0x7f57a01055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:45.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.602+0000 7f57a72de700 1 -- 172.21.15.28:0/3439937763 shutdown_connections 2024-04-25T01:22:45.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.602+0000 7f57a72de700 1 -- 172.21.15.28:0/3439937763 wait complete. 2024-04-25T01:22:45.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.602+0000 7f57a72de700 1 Processor -- start 2024-04-25T01:22:45.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.603+0000 7f57a72de700 1 -- start start 2024-04-25T01:22:45.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.603+0000 7f57a72de700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a007de50 0x7f57a019dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.603+0000 7f57a72de700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57a010e1c0 con 0x7f57a007de50 2024-04-25T01:22:45.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.603+0000 7f57a507a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a007de50 0x7f57a019dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:45.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.603+0000 7f57a507a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a007de50 0x7f57a019dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40022/0 (socket says 172.21.15.28:40022) 2024-04-25T01:22:45.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.603+0000 7f57a507a700 1 -- 172.21.15.28:0/1498158195 learned_addr learned my addr 172.21.15.28:0/1498158195 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:45.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.603+0000 7f57a507a700 1 -- 172.21.15.28:0/1498158195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57a019e270 con 0x7f57a007de50 2024-04-25T01:22:45.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.603+0000 7f57a507a700 1 --2- 172.21.15.28:0/1498158195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a007de50 0x7f57a019dd30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f579c011670 tx=0x7f579c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:45.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.603+0000 7f57967fc700 1 -- 172.21.15.28:0/1498158195 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f579c0039c0 con 0x7f57a007de50 2024-04-25T01:22:45.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.603+0000 7f57967fc700 1 -- 172.21.15.28:0/1498158195 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f579c011e50 con 0x7f57a007de50 2024-04-25T01:22:45.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.603+0000 7f57a72de700 1 -- 172.21.15.28:0/1498158195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57a019e500 con 0x7f57a007de50 2024-04-25T01:22:45.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.603+0000 7f57967fc700 1 -- 172.21.15.28:0/1498158195 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f579c01a5c0 con 0x7f57a007de50 2024-04-25T01:22:45.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.603+0000 7f57a72de700 1 -- 172.21.15.28:0/1498158195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57a01a11f0 con 0x7f57a007de50 2024-04-25T01:22:45.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.604+0000 7f57967fc700 1 -- 172.21.15.28:0/1498158195 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f579c024430 con 0x7f57a007de50 2024-04-25T01:22:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.605+0000 7f57a72de700 1 -- 172.21.15.28:0/1498158195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57a0063890 con 0x7f57a007de50 2024-04-25T01:22:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.605+0000 7f57967fc700 1 --2- 172.21.15.28:0/1498158195 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f578c038fb0 0x7f578c03b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.605+0000 7f57967fc700 1 -- 172.21.15.28:0/1498158195 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f579c04d500 con 0x7f57a007de50 2024-04-25T01:22:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.605+0000 7f57a4879700 1 --2- 172.21.15.28:0/1498158195 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f578c038fb0 0x7f578c03b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.605+0000 7f57a4879700 1 --2- 172.21.15.28:0/1498158195 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f578c038fb0 0x7f578c03b470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5790006fd0 tx=0x7f5790006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.610+0000 7f57967fc700 1 -- 172.21.15.28:0/1498158195 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f579c04cf20 con 0x7f57a007de50 2024-04-25T01:22:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.815+0000 7f57a72de700 1 -- 172.21.15.28:0/1498158195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f57a01a13c0 con 0x7f57a007de50 2024-04-25T01:22:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.816+0000 7f57967fc700 1 -- 172.21.15.28:0/1498158195 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f57a01a13c0 con 0x7f57a007de50 2024-04-25T01:22:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.818+0000 7f57a72de700 1 -- 172.21.15.28:0/1498158195 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f578c038fb0 msgr2=0x7f578c03b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.818+0000 7f57a72de700 1 --2- 172.21.15.28:0/1498158195 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f578c038fb0 0x7f578c03b470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5790006fd0 tx=0x7f5790006e40 comp rx=0 tx=0).stop 2024-04-25T01:22:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.818+0000 7f57a72de700 1 -- 172.21.15.28:0/1498158195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a007de50 msgr2=0x7f57a019dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.818+0000 7f57a72de700 1 --2- 172.21.15.28:0/1498158195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a007de50 0x7f57a019dd30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f579c011670 tx=0x7f579c011b60 comp rx=0 tx=0).stop 2024-04-25T01:22:45.363 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.818+0000 7f57a72de700 1 -- 172.21.15.28:0/1498158195 shutdown_connections 2024-04-25T01:22:45.364 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.818+0000 7f57a72de700 1 --2- 172.21.15.28:0/1498158195 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f578c038fb0 0x7f578c03b470 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.364 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.818+0000 7f57a72de700 1 --2- 172.21.15.28:0/1498158195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57a007de50 0x7f57a019dd30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:45.364 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.818+0000 7f57a72de700 1 -- 172.21.15.28:0/1498158195 >> 172.21.15.28:0/1498158195 conn(0x7f57a0103170 msgr2=0x7f57a0103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:45.364 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.818+0000 7f57a72de700 1 -- 172.21.15.28:0/1498158195 shutdown_connections 2024-04-25T01:22:45.364 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:44.818+0000 7f57a72de700 1 -- 172.21.15.28:0/1498158195 wait complete. 2024-04-25T01:22:45.364 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling cephadm module... 2024-04-25T01:22:46.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:46 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2153192565' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-25T01:22:46.431 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:46 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: ignoring --setuser ceph since I am not root 2024-04-25T01:22:46.431 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:46 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: ignoring --setgroup ceph since I am not root 2024-04-25T01:22:46.431 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:46 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:46.321+0000 7f0a0472c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:22:46.707 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:46 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:46.513+0000 7f0a0472c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:22:46.707 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:46 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:46.618+0000 7f0a0472c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:22:46.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.819+0000 7f63c6b04700 1 Processor -- start 2024-04-25T01:22:46.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.819+0000 7f63c6b04700 1 -- start start 2024-04-25T01:22:46.715 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.819+0000 7f63c6b04700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c0106440 0x7f63c0106860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.819+0000 7f63c6b04700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f63c007beb0 con 0x7f63c0106440 2024-04-25T01:22:46.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.819+0000 7f63c48a0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c0106440 0x7f63c0106860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.819+0000 7f63c48a0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c0106440 0x7f63c0106860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40026/0 (socket says 172.21.15.28:40026) 2024-04-25T01:22:46.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.819+0000 7f63c48a0700 1 -- 172.21.15.28:0/2344869393 learned_addr learned my addr 172.21.15.28:0/2344869393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:46.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.820+0000 7f63c48a0700 1 -- 172.21.15.28:0/2344869393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c0106da0 con 0x7f63c0106440 2024-04-25T01:22:46.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.820+0000 7f63c48a0700 1 --2- 172.21.15.28:0/2344869393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c0106440 0x7f63c0106860 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f63b80098a0 tx=0x7f63b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3707ebb4f620e7c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:46.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.820+0000 7f63bf7fe700 1 -- 172.21.15.28:0/2344869393 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63b8004160 con 0x7f63c0106440 2024-04-25T01:22:46.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.820+0000 7f63bf7fe700 1 -- 172.21.15.28:0/2344869393 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f63b8004300 con 0x7f63c0106440 2024-04-25T01:22:46.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.820+0000 7f63bf7fe700 1 -- 172.21.15.28:0/2344869393 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63b8011b50 con 0x7f63c0106440 2024-04-25T01:22:46.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.821+0000 7f63c6b04700 1 -- 172.21.15.28:0/2344869393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c0106440 msgr2=0x7f63c0106860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.821+0000 7f63c6b04700 1 --2- 172.21.15.28:0/2344869393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c0106440 0x7f63c0106860 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f63b80098a0 tx=0x7f63b8009bb0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.821+0000 7f63c6b04700 1 -- 172.21.15.28:0/2344869393 shutdown_connections 2024-04-25T01:22:46.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.821+0000 7f63c6b04700 1 --2- 172.21.15.28:0/2344869393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c0106440 0x7f63c0106860 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.716 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.821+0000 7f63c6b04700 1 -- 172.21.15.28:0/2344869393 >> 172.21.15.28:0/2344869393 conn(0x7f63c0102040 msgr2=0x7f63c01044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:46.717 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.821+0000 7f63c6b04700 1 -- 172.21.15.28:0/2344869393 shutdown_connections 2024-04-25T01:22:46.717 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.821+0000 7f63c6b04700 1 -- 172.21.15.28:0/2344869393 wait complete. 2024-04-25T01:22:46.717 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.821+0000 7f63c6b04700 1 Processor -- start 2024-04-25T01:22:46.717 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.821+0000 7f63c6b04700 1 -- start start 2024-04-25T01:22:46.717 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.821+0000 7f63c6b04700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c0106440 0x7f63c019dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.717 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.821+0000 7f63c6b04700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f63c0107e30 con 0x7f63c0106440 2024-04-25T01:22:46.717 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.821+0000 7f63c48a0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c0106440 0x7f63c019dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.717 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.822+0000 7f63c48a0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c0106440 0x7f63c019dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40038/0 (socket says 172.21.15.28:40038) 2024-04-25T01:22:46.717 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.822+0000 7f63c48a0700 1 -- 172.21.15.28:0/2153192565 learned_addr learned my addr 172.21.15.28:0/2153192565 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:46.717 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.822+0000 7f63c48a0700 1 -- 172.21.15.28:0/2153192565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c019e270 con 0x7f63c0106440 2024-04-25T01:22:46.717 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.822+0000 7f63c48a0700 1 --2- 172.21.15.28:0/2153192565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c0106440 0x7f63c019dd30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f63b8011b60 tx=0x7f63b8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:46.717 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.822+0000 7f63bdffb700 1 -- 172.21.15.28:0/2153192565 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63b8003b20 con 0x7f63c0106440 2024-04-25T01:22:46.717 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.822+0000 7f63bdffb700 1 -- 172.21.15.28:0/2153192565 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f63b8003cc0 con 0x7f63c0106440 2024-04-25T01:22:46.718 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.822+0000 7f63c6b04700 1 -- 172.21.15.28:0/2153192565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c019e500 con 0x7f63c0106440 2024-04-25T01:22:46.718 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.822+0000 7f63bdffb700 1 -- 172.21.15.28:0/2153192565 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63b8024840 con 0x7f63c0106440 2024-04-25T01:22:46.718 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.822+0000 7f63c6b04700 1 -- 172.21.15.28:0/2153192565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c01a11f0 con 0x7f63c0106440 2024-04-25T01:22:46.718 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.823+0000 7f63bdffb700 1 -- 172.21.15.28:0/2153192565 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f63b8022020 con 0x7f63c0106440 2024-04-25T01:22:46.718 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.823+0000 7f63c6b04700 1 -- 172.21.15.28:0/2153192565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63c0063890 con 0x7f63c0106440 2024-04-25T01:22:46.718 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.823+0000 7f63bdffb700 1 --2- 172.21.15.28:0/2153192565 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f63ac039050 0x7f63ac03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:46.718 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.823+0000 7f63bdffb700 1 -- 172.21.15.28:0/2153192565 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f63b8011e00 con 0x7f63c0106440 2024-04-25T01:22:46.718 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.823+0000 7f63bffff700 1 --2- 172.21.15.28:0/2153192565 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f63ac039050 0x7f63ac03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:46.718 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.824+0000 7f63bffff700 1 --2- 172.21.15.28:0/2153192565 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f63ac039050 0x7f63ac03b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f63b000ad30 tx=0x7f63b00093f0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:46.718 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:45.828+0000 7f63bdffb700 1 -- 172.21.15.28:0/2153192565 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f63b8053080 con 0x7f63c0106440 2024-04-25T01:22:46.718 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:46.096+0000 7f63c6b04700 1 -- 172.21.15.28:0/2153192565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f63c00008d0 con 0x7f63c0106440 2024-04-25T01:22:46.718 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:46.148+0000 7f63bdffb700 1 -- 172.21.15.28:0/2153192565 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f63b800bc90 con 0x7f63c0106440 2024-04-25T01:22:46.718 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:46.149+0000 7f63bdffb700 1 -- 172.21.15.28:0/2153192565 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f63b802d770 con 0x7f63c0106440 2024-04-25T01:22:46.718 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:46.151+0000 7f63c6b04700 1 -- 172.21.15.28:0/2153192565 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f63ac039050 msgr2=0x7f63ac03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.719 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:46.151+0000 7f63c6b04700 1 --2- 172.21.15.28:0/2153192565 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f63ac039050 0x7f63ac03b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f63b000ad30 tx=0x7f63b00093f0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.719 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:46.151+0000 7f63c6b04700 1 -- 172.21.15.28:0/2153192565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c0106440 msgr2=0x7f63c019dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:46.719 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:46.151+0000 7f63c6b04700 1 --2- 172.21.15.28:0/2153192565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c0106440 0x7f63c019dd30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f63b8011b60 tx=0x7f63b8011c40 comp rx=0 tx=0).stop 2024-04-25T01:22:46.719 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:46.151+0000 7f63c6b04700 1 -- 172.21.15.28:0/2153192565 shutdown_connections 2024-04-25T01:22:46.719 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:46.151+0000 7f63c6b04700 1 --2- 172.21.15.28:0/2153192565 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f63ac039050 0x7f63ac03b510 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.719 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:46.151+0000 7f63c6b04700 1 --2- 172.21.15.28:0/2153192565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63c0106440 0x7f63c019dd30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:46.719 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:46.151+0000 7f63c6b04700 1 -- 172.21.15.28:0/2153192565 >> 172.21.15.28:0/2153192565 conn(0x7f63c0102040 msgr2=0x7f63c01949f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:46.719 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:46.151+0000 7f63c6b04700 1 -- 172.21.15.28:0/2153192565 shutdown_connections 2024-04-25T01:22:46.719 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:46.151+0000 7f63c6b04700 1 -- 172.21.15.28:0/2153192565 wait complete. 2024-04-25T01:22:47.016 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:46 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:46.945+0000 7f0a0472c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:22:47.016 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:47 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:47.034+0000 7f0a0472c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:22:47.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:47 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2153192565' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-25T01:22:47.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:47 smithi028 ceph-mon[29390]: mgrmap e5: x(active, since 5s) 2024-04-25T01:22:48.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:22:48.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-25T01:22:48.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T01:22:48.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T01:22:48.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T01:22:48.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:22:48.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.159+0000 7f07bf00f700 1 Processor -- start 2024-04-25T01:22:48.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.159+0000 7f07bf00f700 1 -- start start 2024-04-25T01:22:48.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.159+0000 7f07bf00f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07b8108660 0x7f07b810aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:48.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.159+0000 7f07bf00f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07b807bed0 con 0x7f07b8108660 2024-04-25T01:22:48.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.160+0000 7f07bcdab700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07b8108660 0x7f07b810aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:48.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.160+0000 7f07bcdab700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07b8108660 0x7f07b810aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56382/0 (socket says 172.21.15.28:56382) 2024-04-25T01:22:48.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.160+0000 7f07bcdab700 1 -- 172.21.15.28:0/2898899286 learned_addr learned my addr 172.21.15.28:0/2898899286 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:48.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.160+0000 7f07bcdab700 1 -- 172.21.15.28:0/2898899286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b810af90 con 0x7f07b8108660 2024-04-25T01:22:48.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.160+0000 7f07bcdab700 1 --2- 172.21.15.28:0/2898899286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07b8108660 0x7f07b810aa50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f07ac0098a0 tx=0x7f07ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54df5535d12aa144 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:48.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.160+0000 7f07b77fe700 1 -- 172.21.15.28:0/2898899286 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f07ac004160 con 0x7f07b8108660 2024-04-25T01:22:48.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.160+0000 7f07b77fe700 1 -- 172.21.15.28:0/2898899286 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f07ac004300 con 0x7f07b8108660 2024-04-25T01:22:48.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.160+0000 7f07b77fe700 1 -- 172.21.15.28:0/2898899286 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f07ac011b50 con 0x7f07b8108660 2024-04-25T01:22:48.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.161+0000 7f07bf00f700 1 -- 172.21.15.28:0/2898899286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07b8108660 msgr2=0x7f07b810aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:48.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.161+0000 7f07bf00f700 1 --2- 172.21.15.28:0/2898899286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07b8108660 0x7f07b810aa50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f07ac0098a0 tx=0x7f07ac009bb0 comp rx=0 tx=0).stop 2024-04-25T01:22:48.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.161+0000 7f07bf00f700 1 -- 172.21.15.28:0/2898899286 shutdown_connections 2024-04-25T01:22:48.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.161+0000 7f07bf00f700 1 --2- 172.21.15.28:0/2898899286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07b8108660 0x7f07b810aa50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:48.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.161+0000 7f07bf00f700 1 -- 172.21.15.28:0/2898899286 >> 172.21.15.28:0/2898899286 conn(0x7f07b81020d0 msgr2=0x7f07b81044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:48.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.161+0000 7f07bf00f700 1 -- 172.21.15.28:0/2898899286 shutdown_connections 2024-04-25T01:22:48.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.161+0000 7f07bf00f700 1 -- 172.21.15.28:0/2898899286 wait complete. 2024-04-25T01:22:48.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.162+0000 7f07bf00f700 1 Processor -- start 2024-04-25T01:22:48.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.162+0000 7f07bf00f700 1 -- start start 2024-04-25T01:22:48.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.162+0000 7f07bf00f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07b8108660 0x7f07b819bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:48.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.162+0000 7f07bf00f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07b810bfc0 con 0x7f07b8108660 2024-04-25T01:22:48.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.162+0000 7f07bcdab700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07b8108660 0x7f07b819bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:48.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.162+0000 7f07bcdab700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07b8108660 0x7f07b819bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56388/0 (socket says 172.21.15.28:56388) 2024-04-25T01:22:48.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.162+0000 7f07bcdab700 1 -- 172.21.15.28:0/3693123241 learned_addr learned my addr 172.21.15.28:0/3693123241 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:48.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.162+0000 7f07bcdab700 1 -- 172.21.15.28:0/3693123241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b819c0a0 con 0x7f07b8108660 2024-04-25T01:22:48.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.162+0000 7f07bcdab700 1 --2- 172.21.15.28:0/3693123241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07b8108660 0x7f07b819bb60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f07ac011670 tx=0x7f07ac011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:48.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.162+0000 7f07b5ffb700 1 -- 172.21.15.28:0/3693123241 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f07ac0039c0 con 0x7f07b8108660 2024-04-25T01:22:48.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.163+0000 7f07b5ffb700 1 -- 172.21.15.28:0/3693123241 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f07ac011e50 con 0x7f07b8108660 2024-04-25T01:22:48.232 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.163+0000 7f07bf00f700 1 -- 172.21.15.28:0/3693123241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07b819c330 con 0x7f07b8108660 2024-04-25T01:22:48.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.163+0000 7f07b5ffb700 1 -- 172.21.15.28:0/3693123241 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f07ac01a5c0 con 0x7f07b8108660 2024-04-25T01:22:48.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.163+0000 7f07bf00f700 1 -- 172.21.15.28:0/3693123241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07b819f020 con 0x7f07b8108660 2024-04-25T01:22:48.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.163+0000 7f07b5ffb700 1 -- 172.21.15.28:0/3693123241 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f07ac01a870 con 0x7f07b8108660 2024-04-25T01:22:48.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.164+0000 7f07b5ffb700 1 --2- 172.21.15.28:0/3693123241 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f07a8039050 0x7f07a803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:48.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.164+0000 7f07b5ffb700 1 -- 172.21.15.28:0/3693123241 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f07ac04d2c0 con 0x7f07b8108660 2024-04-25T01:22:48.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.164+0000 7f07b7fff700 1 -- 172.21.15.28:0/3693123241 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f07a8039050 msgr2=0x7f07a803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3973667708 2024-04-25T01:22:48.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.164+0000 7f07b7fff700 1 --2- 172.21.15.28:0/3693123241 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f07a8039050 0x7f07a803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:22:48.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.164+0000 7f07bf00f700 1 -- 172.21.15.28:0/3693123241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07b8063890 con 0x7f07b8108660 2024-04-25T01:22:48.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.169+0000 7f07b5ffb700 1 -- 172.21.15.28:0/3693123241 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f07ac019c70 con 0x7f07b8108660 2024-04-25T01:22:48.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.365+0000 7f07b7fff700 1 -- 172.21.15.28:0/3693123241 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f07a8039050 msgr2=0x7f07a803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3973667708 2024-04-25T01:22:48.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.365+0000 7f07b7fff700 1 --2- 172.21.15.28:0/3693123241 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f07a8039050 0x7f07a803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:22:48.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.457+0000 7f07bf00f700 1 -- 172.21.15.28:0/3693123241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f07b819f1f0 con 0x7f07b8108660 2024-04-25T01:22:48.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.458+0000 7f07b5ffb700 1 -- 172.21.15.28:0/3693123241 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f07ac003b60 con 0x7f07b8108660 2024-04-25T01:22:48.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.463+0000 7f07bf00f700 1 -- 172.21.15.28:0/3693123241 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f07a8039050 msgr2=0x7f07a803b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:22:48.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.463+0000 7f07bf00f700 1 --2- 172.21.15.28:0/3693123241 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f07a8039050 0x7f07a803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:48.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.463+0000 7f07bf00f700 1 -- 172.21.15.28:0/3693123241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07b8108660 msgr2=0x7f07b819bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:48.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.463+0000 7f07bf00f700 1 --2- 172.21.15.28:0/3693123241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07b8108660 0x7f07b819bb60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f07ac011670 tx=0x7f07ac011b60 comp rx=0 tx=0).stop 2024-04-25T01:22:48.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.463+0000 7f07bf00f700 1 -- 172.21.15.28:0/3693123241 shutdown_connections 2024-04-25T01:22:48.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.463+0000 7f07bf00f700 1 --2- 172.21.15.28:0/3693123241 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7f07a8039050 0x7f07a803b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:48.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.463+0000 7f07bf00f700 1 --2- 172.21.15.28:0/3693123241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07b8108660 0x7f07b819bb60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:48.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.463+0000 7f07bf00f700 1 -- 172.21.15.28:0/3693123241 >> 172.21.15.28:0/3693123241 conn(0x7f07b81020d0 msgr2=0x7f07b8102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:48.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.463+0000 7f07bf00f700 1 -- 172.21.15.28:0/3693123241 shutdown_connections 2024-04-25T01:22:48.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:47.463+0000 7f07bf00f700 1 -- 172.21.15.28:0/3693123241 wait complete. 2024-04-25T01:22:48.234 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-04-25T01:22:48.234 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 5... 2024-04-25T01:22:48.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:48 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3693123241' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T01:22:48.456 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:48 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:48.295+0000 7f0a0472c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:22:48.456 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:48 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:48.474+0000 7f0a0472c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:22:48.706 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:48 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:48.656+0000 7f0a0472c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:22:49.037 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:48 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:48.959+0000 7f0a0472c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:22:49.038 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:49 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:49.055+0000 7f0a0472c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:22:49.308 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:49 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:49.278+0000 7f0a0472c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:22:49.565 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:49 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:49.478+0000 7f0a0472c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:22:50.520 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:50 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:50.101+0000 7f0a0472c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:22:50.815 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:50 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:50.538+0000 7f0a0472c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:22:50.815 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:50 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:50.638+0000 7f0a0472c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:22:51.315 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:50 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:50.846+0000 7f0a0472c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:22:51.711 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:51 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:51.391+0000 7f0a0472c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:22:51.711 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:51 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:51.486+0000 7f0a0472c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:22:52.065 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:51 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:51.736+0000 7f0a0472c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:22:52.065 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:51 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:51.875+0000 7f0a0472c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:22:52.065 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:51 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:51.973+0000 7f0a0472c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:22:52.565 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:52 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:52.093+0000 7f0a0472c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:22:52.565 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:52 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:52.188+0000 7f0a0472c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:22:53.288 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:53 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:53.031+0000 7f0a0472c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:22:53.288 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:53 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:22:53.254+0000 7f0a0472c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:22:53.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:53 smithi028 ceph-mon[29390]: Active manager daemon x restarted 2024-04-25T01:22:53.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:53 smithi028 ceph-mon[29390]: Activating manager daemon x 2024-04-25T01:22:53.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:53 smithi028 ceph-mon[29390]: osdmap e2: 0 total, 0 up, 0 in 2024-04-25T01:22:53.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:53 smithi028 ceph-mon[29390]: mgrmap e6: x(active, starting, since 0.00346878s) 2024-04-25T01:22:53.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:53 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:22:53.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:53 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:22:53.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:53 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:22:53.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:53 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:22:53.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:53 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:22:53.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:53 smithi028 ceph-mon[29390]: Manager daemon x is now available 2024-04-25T01:22:53.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:53 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' 2024-04-25T01:22:53.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:53 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' 2024-04-25T01:22:53.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:53 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:53.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:53 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:53.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:53 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:22:54.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:54 smithi028 ceph-mon[29390]: Found migration_current of "None". Setting to last migration. 2024-04-25T01:22:54.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:54 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:22:54.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:54 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' 2024-04-25T01:22:54.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:54 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' 2024-04-25T01:22:54.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:54 smithi028 ceph-mon[29390]: [25/Apr/2024:01:22:54] ENGINE Bus STARTING 2024-04-25T01:22:54.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:54 smithi028 ceph-mon[29390]: [25/Apr/2024:01:22:54] ENGINE Serving on https://172.21.15.28:7150 2024-04-25T01:22:54.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:54 smithi028 ceph-mon[29390]: mgrmap e7: x(active, since 1.00599s) 2024-04-25T01:22:54.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:54 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:54.794 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:22:54.795 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-25T01:22:54.795 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T01:22:54.795 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:22:54.795 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.720+0000 7fc78f9b3700 1 Processor -- start 2024-04-25T01:22:54.795 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.720+0000 7fc78f9b3700 1 -- start start 2024-04-25T01:22:54.795 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.720+0000 7fc78f9b3700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78807de40 0x7fc78807c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.795 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.720+0000 7fc78f9b3700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc78807c880 con 0x7fc78807de40 2024-04-25T01:22:54.795 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.720+0000 7fc78d74f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78807de40 0x7fc78807c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.795 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.720+0000 7fc78d74f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78807de40 0x7fc78807c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56400/0 (socket says 172.21.15.28:56400) 2024-04-25T01:22:54.795 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.720+0000 7fc78d74f700 1 -- 172.21.15.28:0/3220490465 learned_addr learned my addr 172.21.15.28:0/3220490465 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:54.795 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.720+0000 7fc78d74f700 1 -- 172.21.15.28:0/3220490465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc78807ca00 con 0x7fc78807de40 2024-04-25T01:22:54.795 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.721+0000 7fc78d74f700 1 --2- 172.21.15.28:0/3220490465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78807de40 0x7fc78807c340 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc7840098a0 tx=0x7fc784009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d185e20414067f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:54.795 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.721+0000 7fc77ffff700 1 -- 172.21.15.28:0/3220490465 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc784004160 con 0x7fc78807de40 2024-04-25T01:22:54.796 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.721+0000 7fc77ffff700 1 -- 172.21.15.28:0/3220490465 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc784004300 con 0x7fc78807de40 2024-04-25T01:22:54.796 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.721+0000 7fc77ffff700 1 -- 172.21.15.28:0/3220490465 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc784011b50 con 0x7fc78807de40 2024-04-25T01:22:54.796 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.722+0000 7fc78f9b3700 1 -- 172.21.15.28:0/3220490465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78807de40 msgr2=0x7fc78807c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.796 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.722+0000 7fc78f9b3700 1 --2- 172.21.15.28:0/3220490465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78807de40 0x7fc78807c340 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc7840098a0 tx=0x7fc784009bb0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.796 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.722+0000 7fc78f9b3700 1 -- 172.21.15.28:0/3220490465 shutdown_connections 2024-04-25T01:22:54.796 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.722+0000 7fc78f9b3700 1 --2- 172.21.15.28:0/3220490465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78807de40 0x7fc78807c340 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.796 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.722+0000 7fc78f9b3700 1 -- 172.21.15.28:0/3220490465 >> 172.21.15.28:0/3220490465 conn(0x7fc788103230 msgr2=0x7fc788105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:54.796 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.722+0000 7fc78f9b3700 1 -- 172.21.15.28:0/3220490465 shutdown_connections 2024-04-25T01:22:54.796 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.722+0000 7fc78f9b3700 1 -- 172.21.15.28:0/3220490465 wait complete. 2024-04-25T01:22:54.796 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.722+0000 7fc78f9b3700 1 Processor -- start 2024-04-25T01:22:54.796 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.722+0000 7fc78f9b3700 1 -- start start 2024-04-25T01:22:54.796 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.723+0000 7fc78f9b3700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78807de40 0x7fc78819dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.796 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.723+0000 7fc78f9b3700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc78810e1e0 con 0x7fc78807de40 2024-04-25T01:22:54.796 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.723+0000 7fc78d74f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78807de40 0x7fc78819dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.796 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.723+0000 7fc78d74f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78807de40 0x7fc78819dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56406/0 (socket says 172.21.15.28:56406) 2024-04-25T01:22:54.797 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.723+0000 7fc78d74f700 1 -- 172.21.15.28:0/1929998211 learned_addr learned my addr 172.21.15.28:0/1929998211 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:54.797 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.723+0000 7fc78d74f700 1 -- 172.21.15.28:0/1929998211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc78819e2e0 con 0x7fc78807de40 2024-04-25T01:22:54.797 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.723+0000 7fc78d74f700 1 --2- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78807de40 0x7fc78819dda0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc784011b60 tx=0x7fc784011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:54.797 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.723+0000 7fc77e7fc700 1 -- 172.21.15.28:0/1929998211 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc784003970 con 0x7fc78807de40 2024-04-25T01:22:54.797 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.723+0000 7fc77e7fc700 1 -- 172.21.15.28:0/1929998211 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc784003b10 con 0x7fc78807de40 2024-04-25T01:22:54.797 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.723+0000 7fc78f9b3700 1 -- 172.21.15.28:0/1929998211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc78819e570 con 0x7fc78807de40 2024-04-25T01:22:54.797 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.723+0000 7fc77e7fc700 1 -- 172.21.15.28:0/1929998211 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc784024610 con 0x7fc78807de40 2024-04-25T01:22:54.797 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.723+0000 7fc78f9b3700 1 -- 172.21.15.28:0/1929998211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7881a1260 con 0x7fc78807de40 2024-04-25T01:22:54.797 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.724+0000 7fc77e7fc700 1 -- 172.21.15.28:0/1929998211 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7fc7840248c0 con 0x7fc78807de40 2024-04-25T01:22:54.797 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.724+0000 7fc77e7fc700 1 --2- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7fc7740390a0 0x7fc77403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.797 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.724+0000 7fc77e7fc700 1 -- 172.21.15.28:0/1929998211 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc784011e00 con 0x7fc78807de40 2024-04-25T01:22:54.797 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.724+0000 7fc78f9b3700 1 -- 172.21.15.28:0/1929998211 --> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc788063890 con 0x7fc7740390a0 2024-04-25T01:22:54.797 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.724+0000 7fc78cf4e700 1 -- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7fc7740390a0 msgr2=0x7fc77403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3973667708 2024-04-25T01:22:54.797 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.724+0000 7fc78cf4e700 1 --2- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7fc7740390a0 0x7fc77403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:22:54.798 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.926+0000 7fc78cf4e700 1 -- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7fc7740390a0 msgr2=0x7fc77403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3973667708 2024-04-25T01:22:54.798 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:48.926+0000 7fc78cf4e700 1 --2- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7fc7740390a0 0x7fc77403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:22:54.798 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:49.327+0000 7fc78cf4e700 1 -- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7fc7740390a0 msgr2=0x7fc77403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3973667708 2024-04-25T01:22:54.798 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:49.327+0000 7fc78cf4e700 1 --2- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7fc7740390a0 0x7fc77403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T01:22:54.798 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:50.128+0000 7fc78cf4e700 1 -- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7fc7740390a0 msgr2=0x7fc77403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3973667708 2024-04-25T01:22:54.798 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:50.128+0000 7fc78cf4e700 1 --2- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7fc7740390a0 0x7fc77403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T01:22:54.798 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:51.729+0000 7fc78cf4e700 1 -- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7fc7740390a0 msgr2=0x7fc77403b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3973667708 2024-04-25T01:22:54.798 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:51.729+0000 7fc78cf4e700 1 --2- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7fc7740390a0 0x7fc77403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T01:22:54.798 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:53.257+0000 7fc77e7fc700 1 -- 172.21.15.28:0/1929998211 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7fc78402d910 con 0x7fc78807de40 2024-04-25T01:22:54.798 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:53.257+0000 7fc77e7fc700 1 -- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7fc7740390a0 msgr2=0x7fc77403b560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:22:54.798 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:53.257+0000 7fc77e7fc700 1 --2- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7fc7740390a0 0x7fc77403b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.798 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:54.259+0000 7fc77e7fc700 1 -- 172.21.15.28:0/1929998211 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7fc78404ea70 con 0x7fc78807de40 2024-04-25T01:22:54.798 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:54.259+0000 7fc77e7fc700 1 --2- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fc77403c7e0 0x7fc77403ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:54.798 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:54.259+0000 7fc77e7fc700 1 -- 172.21.15.28:0/1929998211 --> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc788063890 con 0x7fc77403c7e0 2024-04-25T01:22:54.799 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:54.259+0000 7fc78cf4e700 1 --2- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fc77403c7e0 0x7fc77403ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:54.799 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:54.260+0000 7fc78cf4e700 1 --2- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fc77403c7e0 0x7fc77403ebd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc778003a10 tx=0x7fc7780092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:54.799 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:54.260+0000 7fc77e7fc700 1 -- 172.21.15.28:0/1929998211 <== mgr.14120 v2:172.21.15.28:6800/2407469300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fc788063890 con 0x7fc77403c7e0 2024-04-25T01:22:54.799 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:54.267+0000 7fc78f9b3700 1 -- 172.21.15.28:0/1929998211 --> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fc788063890 con 0x7fc77403c7e0 2024-04-25T01:22:54.799 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:54.268+0000 7fc77e7fc700 1 -- 172.21.15.28:0/1929998211 <== mgr.14120 v2:172.21.15.28:6800/2407469300 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fc788063890 con 0x7fc77403c7e0 2024-04-25T01:22:54.799 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:54.268+0000 7fc78f9b3700 1 -- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fc77403c7e0 msgr2=0x7fc77403ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.799 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:54.268+0000 7fc78f9b3700 1 --2- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fc77403c7e0 0x7fc77403ebd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc778003a10 tx=0x7fc7780092b0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.799 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:54.268+0000 7fc78f9b3700 1 -- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78807de40 msgr2=0x7fc78819dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:54.799 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:54.268+0000 7fc78f9b3700 1 --2- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78807de40 0x7fc78819dda0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc784011b60 tx=0x7fc784011c40 comp rx=0 tx=0).stop 2024-04-25T01:22:54.799 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:54.268+0000 7fc78f9b3700 1 -- 172.21.15.28:0/1929998211 shutdown_connections 2024-04-25T01:22:54.799 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:54.268+0000 7fc78f9b3700 1 --2- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fc77403c7e0 0x7fc77403ebd0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.799 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:54.268+0000 7fc78f9b3700 1 --2- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78807de40 0x7fc78819dda0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.799 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:54.268+0000 7fc78f9b3700 1 --2- 172.21.15.28:0/1929998211 >> [v2:172.21.15.28:6800/3973667708,v1:172.21.15.28:6801/3973667708] conn(0x7fc7740390a0 0x7fc77403b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:54.799 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:54.268+0000 7fc78f9b3700 1 -- 172.21.15.28:0/1929998211 >> 172.21.15.28:0/1929998211 conn(0x7fc788103230 msgr2=0x7fc788103dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:54.800 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:54.268+0000 7fc78f9b3700 1 -- 172.21.15.28:0/1929998211 shutdown_connections 2024-04-25T01:22:54.800 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:54.268+0000 7fc78f9b3700 1 -- 172.21.15.28:0/1929998211 wait complete. 2024-04-25T01:22:54.800 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 5 is available 2024-04-25T01:22:54.800 INFO:teuthology.orchestra.run.smithi028.stdout:Setting orchestrator backend to cephadm... 2024-04-25T01:22:55.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:55 smithi028 ceph-mon[29390]: [25/Apr/2024:01:22:54] ENGINE Serving on http://172.21.15.28:8765 2024-04-25T01:22:55.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:55 smithi028 ceph-mon[29390]: [25/Apr/2024:01:22:54] ENGINE Bus STARTED 2024-04-25T01:22:55.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:55 smithi028 ceph-mon[29390]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T01:22:55.462 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:55 smithi028 ceph-mon[29390]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T01:22:56.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.245+0000 7f4e390b1700 1 Processor -- start 2024-04-25T01:22:56.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.245+0000 7f4e390b1700 1 -- start start 2024-04-25T01:22:56.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.246+0000 7f4e390b1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e34107bf0 0x7f4e34108010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:56.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.246+0000 7f4e390b1700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4e341085e0 con 0x7f4e34107bf0 2024-04-25T01:22:56.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.246+0000 7f4e32d9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e34107bf0 0x7f4e34108010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:56.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.246+0000 7f4e32d9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e34107bf0 0x7f4e34108010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56502/0 (socket says 172.21.15.28:56502) 2024-04-25T01:22:56.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.246+0000 7f4e32d9d700 1 -- 172.21.15.28:0/1707313931 learned_addr learned my addr 172.21.15.28:0/1707313931 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:56.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.246+0000 7f4e32d9d700 1 -- 172.21.15.28:0/1707313931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e34108da0 con 0x7f4e34107bf0 2024-04-25T01:22:56.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.246+0000 7f4e32d9d700 1 --2- 172.21.15.28:0/1707313931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e34107bf0 0x7f4e34108010 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4e240098a0 tx=0x7f4e24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67ba991678878d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:56.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.246+0000 7f4e31d9b700 1 -- 172.21.15.28:0/1707313931 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e24004160 con 0x7f4e34107bf0 2024-04-25T01:22:56.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.247+0000 7f4e31d9b700 1 -- 172.21.15.28:0/1707313931 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4e24004300 con 0x7f4e34107bf0 2024-04-25T01:22:56.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.247+0000 7f4e31d9b700 1 -- 172.21.15.28:0/1707313931 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e24011b50 con 0x7f4e34107bf0 2024-04-25T01:22:56.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.247+0000 7f4e390b1700 1 -- 172.21.15.28:0/1707313931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e34107bf0 msgr2=0x7f4e34108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:56.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.247+0000 7f4e390b1700 1 --2- 172.21.15.28:0/1707313931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e34107bf0 0x7f4e34108010 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4e240098a0 tx=0x7f4e24009bb0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.247+0000 7f4e390b1700 1 -- 172.21.15.28:0/1707313931 shutdown_connections 2024-04-25T01:22:56.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.247+0000 7f4e390b1700 1 --2- 172.21.15.28:0/1707313931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e34107bf0 0x7f4e34108010 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.247+0000 7f4e390b1700 1 -- 172.21.15.28:0/1707313931 >> 172.21.15.28:0/1707313931 conn(0x7f4e34103170 msgr2=0x7f4e341055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:56.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.247+0000 7f4e390b1700 1 -- 172.21.15.28:0/1707313931 shutdown_connections 2024-04-25T01:22:56.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.247+0000 7f4e390b1700 1 -- 172.21.15.28:0/1707313931 wait complete. 2024-04-25T01:22:56.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.248+0000 7f4e390b1700 1 Processor -- start 2024-04-25T01:22:56.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.248+0000 7f4e390b1700 1 -- start start 2024-04-25T01:22:56.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.248+0000 7f4e390b1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e34107bf0 0x7f4e3419bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:56.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.248+0000 7f4e390b1700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3410c150 con 0x7f4e34107bf0 2024-04-25T01:22:56.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.249+0000 7f4e32d9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e34107bf0 0x7f4e3419bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:56.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.249+0000 7f4e32d9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e34107bf0 0x7f4e3419bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56508/0 (socket says 172.21.15.28:56508) 2024-04-25T01:22:56.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.249+0000 7f4e32d9d700 1 -- 172.21.15.28:0/2883746737 learned_addr learned my addr 172.21.15.28:0/2883746737 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:56.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.249+0000 7f4e32d9d700 1 -- 172.21.15.28:0/2883746737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e3419c0b0 con 0x7f4e34107bf0 2024-04-25T01:22:56.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.249+0000 7f4e32d9d700 1 --2- 172.21.15.28:0/2883746737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e34107bf0 0x7f4e3419bb70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4e24000c00 tx=0x7f4e24011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:56.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.249+0000 7f4e1bfff700 1 -- 172.21.15.28:0/2883746737 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e24003b20 con 0x7f4e34107bf0 2024-04-25T01:22:56.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.249+0000 7f4e1bfff700 1 -- 172.21.15.28:0/2883746737 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4e24003cc0 con 0x7f4e34107bf0 2024-04-25T01:22:56.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.249+0000 7f4e390b1700 1 -- 172.21.15.28:0/2883746737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e3419c340 con 0x7f4e34107bf0 2024-04-25T01:22:56.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.249+0000 7f4e1bfff700 1 -- 172.21.15.28:0/2883746737 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e24024910 con 0x7f4e34107bf0 2024-04-25T01:22:56.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.249+0000 7f4e390b1700 1 -- 172.21.15.28:0/2883746737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e3419f030 con 0x7f4e34107bf0 2024-04-25T01:22:56.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.250+0000 7f4e1bfff700 1 -- 172.21.15.28:0/2883746737 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f4e24022020 con 0x7f4e34107bf0 2024-04-25T01:22:56.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.250+0000 7f4e390b1700 1 -- 172.21.15.28:0/2883746737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e34063890 con 0x7f4e34107bf0 2024-04-25T01:22:56.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.250+0000 7f4e1bfff700 1 --2- 172.21.15.28:0/2883746737 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f4e1c038f80 0x7f4e1c03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:56.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.250+0000 7f4e1bfff700 1 -- 172.21.15.28:0/2883746737 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4e24011e00 con 0x7f4e34107bf0 2024-04-25T01:22:56.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.251+0000 7f4e3259c700 1 --2- 172.21.15.28:0/2883746737 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f4e1c038f80 0x7f4e1c03b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:56.060 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.251+0000 7f4e3259c700 1 --2- 172.21.15.28:0/2883746737 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f4e1c038f80 0x7f4e1c03b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4e28006fd0 tx=0x7f4e28006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:56.060 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.256+0000 7f4e1bfff700 1 -- 172.21.15.28:0/2883746737 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4e2402cdf0 con 0x7f4e34107bf0 2024-04-25T01:22:56.060 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.310+0000 7f4e1bfff700 1 -- 172.21.15.28:0/2883746737 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f4e2401acd0 con 0x7f4e34107bf0 2024-04-25T01:22:56.060 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.480+0000 7f4e390b1700 1 -- 172.21.15.28:0/2883746737 --> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f4e34104400 con 0x7f4e1c038f80 2024-04-25T01:22:56.060 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.485+0000 7f4e1bfff700 1 -- 172.21.15.28:0/2883746737 <== mgr.14120 v2:172.21.15.28:6800/2407469300 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4e34104400 con 0x7f4e1c038f80 2024-04-25T01:22:56.060 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.487+0000 7f4e390b1700 1 -- 172.21.15.28:0/2883746737 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f4e1c038f80 msgr2=0x7f4e1c03b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:56.060 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.487+0000 7f4e390b1700 1 --2- 172.21.15.28:0/2883746737 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f4e1c038f80 0x7f4e1c03b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4e28006fd0 tx=0x7f4e28006e40 comp rx=0 tx=0).stop 2024-04-25T01:22:56.060 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.487+0000 7f4e390b1700 1 -- 172.21.15.28:0/2883746737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e34107bf0 msgr2=0x7f4e3419bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:56.060 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.487+0000 7f4e390b1700 1 --2- 172.21.15.28:0/2883746737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e34107bf0 0x7f4e3419bb70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4e24000c00 tx=0x7f4e24011c10 comp rx=0 tx=0).stop 2024-04-25T01:22:56.060 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.487+0000 7f4e390b1700 1 -- 172.21.15.28:0/2883746737 shutdown_connections 2024-04-25T01:22:56.060 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.488+0000 7f4e390b1700 1 --2- 172.21.15.28:0/2883746737 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f4e1c038f80 0x7f4e1c03b440 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.060 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.488+0000 7f4e390b1700 1 --2- 172.21.15.28:0/2883746737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e34107bf0 0x7f4e3419bb70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:56.060 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.488+0000 7f4e390b1700 1 -- 172.21.15.28:0/2883746737 >> 172.21.15.28:0/2883746737 conn(0x7f4e34103170 msgr2=0x7f4e34103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:56.060 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.488+0000 7f4e390b1700 1 -- 172.21.15.28:0/2883746737 shutdown_connections 2024-04-25T01:22:56.060 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:55.488+0000 7f4e390b1700 1 -- 172.21.15.28:0/2883746737 wait complete. 2024-04-25T01:22:56.301 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:56 smithi028 ceph-mon[29390]: mgrmap e8: x(active, since 2s) 2024-04-25T01:22:56.301 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:56 smithi028 ceph-mon[29390]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:22:56.301 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:56 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' 2024-04-25T01:22:56.301 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:56 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:22:57.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-25T01:22:57.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.513+0000 7f6aaf768700 1 Processor -- start 2024-04-25T01:22:57.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.514+0000 7f6aaf768700 1 -- start start 2024-04-25T01:22:57.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.514+0000 7f6aaf768700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6aa8107bf0 0x7f6aa8108010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:57.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.514+0000 7f6aaf768700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa81085e0 con 0x7f6aa8107bf0 2024-04-25T01:22:57.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.514+0000 7f6aad504700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6aa8107bf0 0x7f6aa8108010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:57.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.514+0000 7f6aad504700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6aa8107bf0 0x7f6aa8108010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56510/0 (socket says 172.21.15.28:56510) 2024-04-25T01:22:57.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.514+0000 7f6aad504700 1 -- 172.21.15.28:0/4157786347 learned_addr learned my addr 172.21.15.28:0/4157786347 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:57.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.514+0000 7f6aad504700 1 -- 172.21.15.28:0/4157786347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa8108da0 con 0x7f6aa8107bf0 2024-04-25T01:22:57.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.514+0000 7f6aad504700 1 --2- 172.21.15.28:0/4157786347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6aa8107bf0 0x7f6aa8108010 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6aa40098a0 tx=0x7f6aa4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c0a1451c35a761 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:57.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.515+0000 7f6a9ffff700 1 -- 172.21.15.28:0/4157786347 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6aa4004160 con 0x7f6aa8107bf0 2024-04-25T01:22:57.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.515+0000 7f6a9ffff700 1 -- 172.21.15.28:0/4157786347 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6aa4004300 con 0x7f6aa8107bf0 2024-04-25T01:22:57.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.515+0000 7f6a9ffff700 1 -- 172.21.15.28:0/4157786347 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6aa4011b50 con 0x7f6aa8107bf0 2024-04-25T01:22:57.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.515+0000 7f6aaf768700 1 -- 172.21.15.28:0/4157786347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6aa8107bf0 msgr2=0x7f6aa8108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:57.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.515+0000 7f6aaf768700 1 --2- 172.21.15.28:0/4157786347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6aa8107bf0 0x7f6aa8108010 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6aa40098a0 tx=0x7f6aa4009bb0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.515+0000 7f6aaf768700 1 -- 172.21.15.28:0/4157786347 shutdown_connections 2024-04-25T01:22:57.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.515+0000 7f6aaf768700 1 --2- 172.21.15.28:0/4157786347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6aa8107bf0 0x7f6aa8108010 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.515+0000 7f6aaf768700 1 -- 172.21.15.28:0/4157786347 >> 172.21.15.28:0/4157786347 conn(0x7f6aa8103170 msgr2=0x7f6aa81055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:57.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.515+0000 7f6aaf768700 1 -- 172.21.15.28:0/4157786347 shutdown_connections 2024-04-25T01:22:57.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.515+0000 7f6aaf768700 1 -- 172.21.15.28:0/4157786347 wait complete. 2024-04-25T01:22:57.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.516+0000 7f6aaf768700 1 Processor -- start 2024-04-25T01:22:57.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.516+0000 7f6aaf768700 1 -- start start 2024-04-25T01:22:57.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.516+0000 7f6aaf768700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6aa8107bf0 0x7f6aa819bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:57.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.516+0000 7f6aaf768700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa810c150 con 0x7f6aa8107bf0 2024-04-25T01:22:57.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.516+0000 7f6aad504700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6aa8107bf0 0x7f6aa819bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:57.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.516+0000 7f6aad504700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6aa8107bf0 0x7f6aa819bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56526/0 (socket says 172.21.15.28:56526) 2024-04-25T01:22:57.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.517+0000 7f6aad504700 1 -- 172.21.15.28:0/115551732 learned_addr learned my addr 172.21.15.28:0/115551732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:57.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.517+0000 7f6aad504700 1 -- 172.21.15.28:0/115551732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa819c100 con 0x7f6aa8107bf0 2024-04-25T01:22:57.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.517+0000 7f6aad504700 1 --2- 172.21.15.28:0/115551732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6aa8107bf0 0x7f6aa819bbc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4000c00 tx=0x7f6aa4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:57.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.517+0000 7f6a9e7fc700 1 -- 172.21.15.28:0/115551732 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6aa4003b20 con 0x7f6aa8107bf0 2024-04-25T01:22:57.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.517+0000 7f6a9e7fc700 1 -- 172.21.15.28:0/115551732 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6aa4003cc0 con 0x7f6aa8107bf0 2024-04-25T01:22:57.282 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.517+0000 7f6aaf768700 1 -- 172.21.15.28:0/115551732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aa819c390 con 0x7f6aa8107bf0 2024-04-25T01:22:57.282 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.517+0000 7f6a9e7fc700 1 -- 172.21.15.28:0/115551732 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6aa4024910 con 0x7f6aa8107bf0 2024-04-25T01:22:57.282 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.517+0000 7f6aaf768700 1 -- 172.21.15.28:0/115551732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aa819f030 con 0x7f6aa8107bf0 2024-04-25T01:22:57.282 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.518+0000 7f6a9e7fc700 1 -- 172.21.15.28:0/115551732 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f6aa4022020 con 0x7f6aa8107bf0 2024-04-25T01:22:57.282 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.518+0000 7f6aaf768700 1 -- 172.21.15.28:0/115551732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6aa8063890 con 0x7f6aa8107bf0 2024-04-25T01:22:57.282 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.518+0000 7f6a9e7fc700 1 --2- 172.21.15.28:0/115551732 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f6a940390a0 0x7f6a9403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:57.282 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.518+0000 7f6a9e7fc700 1 -- 172.21.15.28:0/115551732 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6aa4011e00 con 0x7f6aa8107bf0 2024-04-25T01:22:57.282 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.518+0000 7f6aacd03700 1 --2- 172.21.15.28:0/115551732 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f6a940390a0 0x7f6a9403b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:57.282 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.519+0000 7f6aacd03700 1 --2- 172.21.15.28:0/115551732 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f6a940390a0 0x7f6a9403b560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6a98006fd0 tx=0x7f6a98006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:57.282 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.524+0000 7f6a9e7fc700 1 -- 172.21.15.28:0/115551732 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6aa401acd0 con 0x7f6aa8107bf0 2024-04-25T01:22:57.282 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.732+0000 7f6aaf768700 1 -- 172.21.15.28:0/115551732 --> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f6aa8104400 con 0x7f6a940390a0 2024-04-25T01:22:57.282 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.733+0000 7f6a9e7fc700 1 -- 172.21.15.28:0/115551732 <== mgr.14120 v2:172.21.15.28:6800/2407469300 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f6aa8104400 con 0x7f6a940390a0 2024-04-25T01:22:57.282 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.735+0000 7f6aaf768700 1 -- 172.21.15.28:0/115551732 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f6a940390a0 msgr2=0x7f6a9403b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:57.282 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.735+0000 7f6aaf768700 1 --2- 172.21.15.28:0/115551732 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f6a940390a0 0x7f6a9403b560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6a98006fd0 tx=0x7f6a98006e40 comp rx=0 tx=0).stop 2024-04-25T01:22:57.283 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.735+0000 7f6aaf768700 1 -- 172.21.15.28:0/115551732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6aa8107bf0 msgr2=0x7f6aa819bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:57.283 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.735+0000 7f6aaf768700 1 --2- 172.21.15.28:0/115551732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6aa8107bf0 0x7f6aa819bbc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4000c00 tx=0x7f6aa4011c10 comp rx=0 tx=0).stop 2024-04-25T01:22:57.283 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.735+0000 7f6aaf768700 1 -- 172.21.15.28:0/115551732 shutdown_connections 2024-04-25T01:22:57.283 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.735+0000 7f6aaf768700 1 --2- 172.21.15.28:0/115551732 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f6a940390a0 0x7f6a9403b560 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.283 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.735+0000 7f6aaf768700 1 --2- 172.21.15.28:0/115551732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6aa8107bf0 0x7f6aa819bbc0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:57.283 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.735+0000 7f6aaf768700 1 -- 172.21.15.28:0/115551732 >> 172.21.15.28:0/115551732 conn(0x7f6aa8103170 msgr2=0x7f6aa8103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:57.283 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.735+0000 7f6aaf768700 1 -- 172.21.15.28:0/115551732 shutdown_connections 2024-04-25T01:22:57.283 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:56.735+0000 7f6aaf768700 1 -- 172.21.15.28:0/115551732 wait complete. 2024-04-25T01:22:57.283 INFO:teuthology.orchestra.run.smithi028.stdout:Generating ssh key... 2024-04-25T01:22:57.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:57 smithi028 ceph-mon[29390]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:22:58.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:58 smithi028 ceph-mon[29390]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:22:58.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:58 smithi028 ceph-mon[29390]: Generating ssh key... 2024-04-25T01:22:58.565 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: Generating public/private rsa key pair. 2024-04-25T01:22:58.565 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: Your identification has been saved in /tmp/tmpcghvxehu/key. 2024-04-25T01:22:58.565 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: Your public key has been saved in /tmp/tmpcghvxehu/key.pub. 2024-04-25T01:22:58.565 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: The key fingerprint is: 2024-04-25T01:22:58.566 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: SHA256:feJMAE3oY8SoghhmRZx7tlOnyNvsFO0GqnkDnGBjg/4 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:22:58.566 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: The key's randomart image is: 2024-04-25T01:22:58.566 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: +---[RSA 3072]----+ 2024-04-25T01:22:58.566 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: | +o.o.+. | 2024-04-25T01:22:58.566 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: |.o o. +.. | 2024-04-25T01:22:58.566 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: |*. ..o . | 2024-04-25T01:22:58.566 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: |=*.. o+..+ | 2024-04-25T01:22:58.566 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: |+.= =.++S.+ . | 2024-04-25T01:22:58.566 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: | . + =..++ o | 2024-04-25T01:22:58.566 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: | . ..=. oo | 2024-04-25T01:22:58.566 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: | Eo+.o. | 2024-04-25T01:22:58.566 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: | o. o. | 2024-04-25T01:22:58.566 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:22:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: +----[SHA256]-----+ 2024-04-25T01:22:58.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.706+0000 7f2787d5b700 1 Processor -- start 2024-04-25T01:22:58.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.706+0000 7f2787d5b700 1 -- start start 2024-04-25T01:22:58.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.706+0000 7f2787d5b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27801080a0 0x7f27801084c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.706+0000 7f2787d5b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2780108a90 con 0x7f27801080a0 2024-04-25T01:22:58.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.707+0000 7f2785af7700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27801080a0 0x7f27801084c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.707+0000 7f2785af7700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27801080a0 0x7f27801084c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46408/0 (socket says 172.21.15.28:46408) 2024-04-25T01:22:58.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.707+0000 7f2785af7700 1 -- 172.21.15.28:0/2389013545 learned_addr learned my addr 172.21.15.28:0/2389013545 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:58.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.707+0000 7f2785af7700 1 -- 172.21.15.28:0/2389013545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2780108c10 con 0x7f27801080a0 2024-04-25T01:22:58.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.707+0000 7f2785af7700 1 --2- 172.21.15.28:0/2389013545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27801080a0 0x7f27801084c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f27700098a0 tx=0x7f2770009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=574cabe57821c2b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:58.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.707+0000 7f2784af5700 1 -- 172.21.15.28:0/2389013545 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2770004160 con 0x7f27801080a0 2024-04-25T01:22:58.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.707+0000 7f2784af5700 1 -- 172.21.15.28:0/2389013545 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2770004300 con 0x7f27801080a0 2024-04-25T01:22:58.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.707+0000 7f2784af5700 1 -- 172.21.15.28:0/2389013545 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2770011b50 con 0x7f27801080a0 2024-04-25T01:22:58.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.708+0000 7f2787d5b700 1 -- 172.21.15.28:0/2389013545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27801080a0 msgr2=0x7f27801084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.708+0000 7f2787d5b700 1 --2- 172.21.15.28:0/2389013545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27801080a0 0x7f27801084c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f27700098a0 tx=0x7f2770009bb0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.708+0000 7f2787d5b700 1 -- 172.21.15.28:0/2389013545 shutdown_connections 2024-04-25T01:22:58.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.708+0000 7f2787d5b700 1 --2- 172.21.15.28:0/2389013545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27801080a0 0x7f27801084c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.708+0000 7f2787d5b700 1 -- 172.21.15.28:0/2389013545 >> 172.21.15.28:0/2389013545 conn(0x7f2780101830 msgr2=0x7f2780103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:58.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.708+0000 7f2787d5b700 1 -- 172.21.15.28:0/2389013545 shutdown_connections 2024-04-25T01:22:58.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.708+0000 7f2787d5b700 1 -- 172.21.15.28:0/2389013545 wait complete. 2024-04-25T01:22:58.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.709+0000 7f2787d5b700 1 Processor -- start 2024-04-25T01:22:58.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.709+0000 7f2787d5b700 1 -- start start 2024-04-25T01:22:58.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.709+0000 7f2787d5b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27801080a0 0x7f278019dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.709+0000 7f2787d5b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27801069a0 con 0x7f27801080a0 2024-04-25T01:22:58.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.709+0000 7f2785af7700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27801080a0 0x7f278019dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.709+0000 7f2785af7700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27801080a0 0x7f278019dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46424/0 (socket says 172.21.15.28:46424) 2024-04-25T01:22:58.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.709+0000 7f2785af7700 1 -- 172.21.15.28:0/2381100364 learned_addr learned my addr 172.21.15.28:0/2381100364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:22:58.928 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.709+0000 7f2785af7700 1 -- 172.21.15.28:0/2381100364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278019e1e0 con 0x7f27801080a0 2024-04-25T01:22:58.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.709+0000 7f2785af7700 1 --2- 172.21.15.28:0/2381100364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27801080a0 0x7f278019dca0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2770011670 tx=0x7f2770011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:58.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.710+0000 7f2776ffd700 1 -- 172.21.15.28:0/2381100364 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27700039c0 con 0x7f27801080a0 2024-04-25T01:22:58.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.710+0000 7f2787d5b700 1 -- 172.21.15.28:0/2381100364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f278019e4d0 con 0x7f27801080a0 2024-04-25T01:22:58.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.710+0000 7f2776ffd700 1 -- 172.21.15.28:0/2381100364 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2770011e50 con 0x7f27801080a0 2024-04-25T01:22:58.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.710+0000 7f2776ffd700 1 -- 172.21.15.28:0/2381100364 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f277001a5c0 con 0x7f27801080a0 2024-04-25T01:22:58.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.710+0000 7f2787d5b700 1 -- 172.21.15.28:0/2381100364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27801a1160 con 0x7f27801080a0 2024-04-25T01:22:58.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.710+0000 7f2776ffd700 1 -- 172.21.15.28:0/2381100364 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f2770024430 con 0x7f27801080a0 2024-04-25T01:22:58.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.711+0000 7f2787d5b700 1 -- 172.21.15.28:0/2381100364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2780063890 con 0x7f27801080a0 2024-04-25T01:22:58.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.711+0000 7f2776ffd700 1 --2- 172.21.15.28:0/2381100364 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f276c039000 0x7f276c03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:22:58.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.711+0000 7f2776ffd700 1 -- 172.21.15.28:0/2381100364 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f277004d5b0 con 0x7f27801080a0 2024-04-25T01:22:58.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.711+0000 7f27852f6700 1 --2- 172.21.15.28:0/2381100364 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f276c039000 0x7f276c03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:22:58.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.711+0000 7f27852f6700 1 --2- 172.21.15.28:0/2381100364 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f276c039000 0x7f276c03b4c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f277c006fd0 tx=0x7f277c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:22:58.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.716+0000 7f2776ffd700 1 -- 172.21.15.28:0/2381100364 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f27700198e0 con 0x7f27801080a0 2024-04-25T01:22:58.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:57.925+0000 7f2787d5b700 1 -- 172.21.15.28:0/2381100364 --> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f2780062670 con 0x7f276c039000 2024-04-25T01:22:58.929 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:58.376+0000 7f2776ffd700 1 -- 172.21.15.28:0/2381100364 <== mgr.14120 v2:172.21.15.28:6800/2407469300 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2780062670 con 0x7f276c039000 2024-04-25T01:22:58.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:58.378+0000 7f2787d5b700 1 -- 172.21.15.28:0/2381100364 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f276c039000 msgr2=0x7f276c03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:58.378+0000 7f2787d5b700 1 --2- 172.21.15.28:0/2381100364 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f276c039000 0x7f276c03b4c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f277c006fd0 tx=0x7f277c006e40 comp rx=0 tx=0).stop 2024-04-25T01:22:58.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:58.378+0000 7f2787d5b700 1 -- 172.21.15.28:0/2381100364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27801080a0 msgr2=0x7f278019dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:22:58.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:58.378+0000 7f2787d5b700 1 --2- 172.21.15.28:0/2381100364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27801080a0 0x7f278019dca0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2770011670 tx=0x7f2770011b60 comp rx=0 tx=0).stop 2024-04-25T01:22:58.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:58.378+0000 7f2787d5b700 1 -- 172.21.15.28:0/2381100364 shutdown_connections 2024-04-25T01:22:58.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:58.378+0000 7f2787d5b700 1 --2- 172.21.15.28:0/2381100364 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f276c039000 0x7f276c03b4c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:58.378+0000 7f2787d5b700 1 --2- 172.21.15.28:0/2381100364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27801080a0 0x7f278019dca0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:22:58.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:58.378+0000 7f2787d5b700 1 -- 172.21.15.28:0/2381100364 >> 172.21.15.28:0/2381100364 conn(0x7f2780101830 msgr2=0x7f27801942c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:22:58.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:58.378+0000 7f2787d5b700 1 -- 172.21.15.28:0/2381100364 shutdown_connections 2024-04-25T01:22:58.930 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:58.378+0000 7f2787d5b700 1 -- 172.21.15.28:0/2381100364 wait complete. 2024-04-25T01:22:59.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:59 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' 2024-04-25T01:22:59.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:22:59 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' 2024-04-25T01:23:00.091 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNDiUVBwg9V5g9uZkL2ovljFZpXg/5LNJQeAahyRKa/yhLU5xnPA77zjGxO90Vx5yC51qBABAchCR63eCTZntbBLakWrAmTQ+QsmgVWX6flcsNx8RJ0zw6lc2QBgx0JM+dC6ES7Wt4d9leCQKd/eXNHujSg0pgbBVS4EMkKbSL7jC2Lcvtywmb1QW/+FvtK/QrQ9fKLOP6Ago+ueDLKK7Ve20YRBozUwrnn+oorKGdL3hWutMOlqKjlz0PUfE3/q4tGydD8n5FbdylSNPB3S1qsMClL2IYGbLzlhg9FpplxT58fjyLTwUQAXkrJ3NA0SQfFcVV3bfvm9WG2ZjCF9A6X1w6WOoktnu66rkWwJ0utd6tVeOrqRb8m5dFv36e7Ykhh0ihsiAmsBO5oyDO6zYsZ0No0ZddJ9nyahva9M6DXNRvCGwmJY26v5sSc4dJB778RHXxGBtcQ3H1ABOffBmLR4Km2MAstTPntRfen1lzpl70/YyDU7G0zKgZEG8iW10= ceph-0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:23:00.091 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.347+0000 7f61e1eaf700 1 Processor -- start 2024-04-25T01:23:00.091 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.347+0000 7f61e1eaf700 1 -- start start 2024-04-25T01:23:00.091 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.347+0000 7f61e1eaf700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61dc108670 0x7f61dc10aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.091 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.347+0000 7f61e1eaf700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc07be90 con 0x7f61dc108670 2024-04-25T01:23:00.092 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.347+0000 7f61db7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61dc108670 0x7f61dc10aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.092 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.347+0000 7f61db7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61dc108670 0x7f61dc10aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46436/0 (socket says 172.21.15.28:46436) 2024-04-25T01:23:00.092 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.347+0000 7f61db7fe700 1 -- 172.21.15.28:0/1250243051 learned_addr learned my addr 172.21.15.28:0/1250243051 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:00.092 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.348+0000 7f61db7fe700 1 -- 172.21.15.28:0/1250243051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61dc10afa0 con 0x7f61dc108670 2024-04-25T01:23:00.092 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.348+0000 7f61db7fe700 1 --2- 172.21.15.28:0/1250243051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61dc108670 0x7f61dc10aa60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f61cc0098a0 tx=0x7f61cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee002392de4247c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:00.092 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.348+0000 7f61da7fc700 1 -- 172.21.15.28:0/1250243051 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61cc004160 con 0x7f61dc108670 2024-04-25T01:23:00.092 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.348+0000 7f61da7fc700 1 -- 172.21.15.28:0/1250243051 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f61cc004300 con 0x7f61dc108670 2024-04-25T01:23:00.092 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.348+0000 7f61da7fc700 1 -- 172.21.15.28:0/1250243051 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61cc011b50 con 0x7f61dc108670 2024-04-25T01:23:00.092 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.349+0000 7f61e1eaf700 1 -- 172.21.15.28:0/1250243051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61dc108670 msgr2=0x7f61dc10aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.092 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.349+0000 7f61e1eaf700 1 --2- 172.21.15.28:0/1250243051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61dc108670 0x7f61dc10aa60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f61cc0098a0 tx=0x7f61cc009bb0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.092 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.349+0000 7f61e1eaf700 1 -- 172.21.15.28:0/1250243051 shutdown_connections 2024-04-25T01:23:00.092 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.349+0000 7f61e1eaf700 1 --2- 172.21.15.28:0/1250243051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61dc108670 0x7f61dc10aa60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.092 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.349+0000 7f61e1eaf700 1 -- 172.21.15.28:0/1250243051 >> 172.21.15.28:0/1250243051 conn(0x7f61dc102040 msgr2=0x7f61dc1044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:00.092 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.349+0000 7f61e1eaf700 1 -- 172.21.15.28:0/1250243051 shutdown_connections 2024-04-25T01:23:00.093 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.349+0000 7f61e1eaf700 1 -- 172.21.15.28:0/1250243051 wait complete. 2024-04-25T01:23:00.093 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.350+0000 7f61e1eaf700 1 Processor -- start 2024-04-25T01:23:00.093 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.350+0000 7f61e1eaf700 1 -- start start 2024-04-25T01:23:00.093 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.350+0000 7f61e1eaf700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61dc108670 0x7f61dc19bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.093 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.350+0000 7f61e1eaf700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc10bfc0 con 0x7f61dc108670 2024-04-25T01:23:00.093 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.350+0000 7f61db7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61dc108670 0x7f61dc19bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.093 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.350+0000 7f61db7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61dc108670 0x7f61dc19bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46448/0 (socket says 172.21.15.28:46448) 2024-04-25T01:23:00.093 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.350+0000 7f61db7fe700 1 -- 172.21.15.28:0/1850584832 learned_addr learned my addr 172.21.15.28:0/1850584832 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:00.093 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.350+0000 7f61db7fe700 1 -- 172.21.15.28:0/1850584832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61dc19c070 con 0x7f61dc108670 2024-04-25T01:23:00.093 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.351+0000 7f61db7fe700 1 --2- 172.21.15.28:0/1850584832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61dc108670 0x7f61dc19bb30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f61cc011b60 tx=0x7f61cc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:00.093 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.351+0000 7f61d8ff9700 1 -- 172.21.15.28:0/1850584832 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61cc003b20 con 0x7f61dc108670 2024-04-25T01:23:00.093 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.351+0000 7f61d8ff9700 1 -- 172.21.15.28:0/1850584832 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f61cc003cc0 con 0x7f61dc108670 2024-04-25T01:23:00.093 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.351+0000 7f61d8ff9700 1 -- 172.21.15.28:0/1850584832 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61cc024840 con 0x7f61dc108670 2024-04-25T01:23:00.093 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.351+0000 7f61e1eaf700 1 -- 172.21.15.28:0/1850584832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61dc19c300 con 0x7f61dc108670 2024-04-25T01:23:00.094 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.351+0000 7f61e1eaf700 1 -- 172.21.15.28:0/1850584832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61dc19eff0 con 0x7f61dc108670 2024-04-25T01:23:00.094 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.352+0000 7f61d8ff9700 1 -- 172.21.15.28:0/1850584832 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f61cc022020 con 0x7f61dc108670 2024-04-25T01:23:00.094 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.352+0000 7f61e1eaf700 1 -- 172.21.15.28:0/1850584832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61dc063890 con 0x7f61dc108670 2024-04-25T01:23:00.094 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.352+0000 7f61d8ff9700 1 --2- 172.21.15.28:0/1850584832 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f61c40390a0 0x7f61c403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:00.094 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.352+0000 7f61d8ff9700 1 -- 172.21.15.28:0/1850584832 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f61cc011e00 con 0x7f61dc108670 2024-04-25T01:23:00.094 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.352+0000 7f61daffd700 1 --2- 172.21.15.28:0/1850584832 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f61c40390a0 0x7f61c403b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:00.094 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.352+0000 7f61daffd700 1 --2- 172.21.15.28:0/1850584832 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f61c40390a0 0x7f61c403b560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f61d0006fd0 tx=0x7f61d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:00.094 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.357+0000 7f61d8ff9700 1 -- 172.21.15.28:0/1850584832 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f61cc00ba30 con 0x7f61dc108670 2024-04-25T01:23:00.094 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.564+0000 7f61e1eaf700 1 -- 172.21.15.28:0/1850584832 --> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f61dc103260 con 0x7f61c40390a0 2024-04-25T01:23:00.094 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.565+0000 7f61d8ff9700 1 -- 172.21.15.28:0/1850584832 <== mgr.14120 v2:172.21.15.28:6800/2407469300 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f61dc103260 con 0x7f61c40390a0 2024-04-25T01:23:00.094 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.567+0000 7f61e1eaf700 1 -- 172.21.15.28:0/1850584832 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f61c40390a0 msgr2=0x7f61c403b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.094 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.567+0000 7f61e1eaf700 1 --2- 172.21.15.28:0/1850584832 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f61c40390a0 0x7f61c403b560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f61d0006fd0 tx=0x7f61d0006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:00.094 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.567+0000 7f61e1eaf700 1 -- 172.21.15.28:0/1850584832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61dc108670 msgr2=0x7f61dc19bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:00.095 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.567+0000 7f61e1eaf700 1 --2- 172.21.15.28:0/1850584832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61dc108670 0x7f61dc19bb30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f61cc011b60 tx=0x7f61cc011c40 comp rx=0 tx=0).stop 2024-04-25T01:23:00.095 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.567+0000 7f61e1eaf700 1 -- 172.21.15.28:0/1850584832 shutdown_connections 2024-04-25T01:23:00.095 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.567+0000 7f61e1eaf700 1 --2- 172.21.15.28:0/1850584832 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f61c40390a0 0x7f61c403b560 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.095 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.567+0000 7f61e1eaf700 1 --2- 172.21.15.28:0/1850584832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61dc108670 0x7f61dc19bb30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:00.095 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.567+0000 7f61e1eaf700 1 -- 172.21.15.28:0/1850584832 >> 172.21.15.28:0/1850584832 conn(0x7f61dc102040 msgr2=0x7f61dc102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:00.095 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.567+0000 7f61e1eaf700 1 -- 172.21.15.28:0/1850584832 shutdown_connections 2024-04-25T01:23:00.095 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:22:59.567+0000 7f61e1eaf700 1 -- 172.21.15.28:0/1850584832 wait complete. 2024-04-25T01:23:00.095 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-25T01:23:00.095 INFO:teuthology.orchestra.run.smithi028.stdout:Adding key to root@localhost authorized_keys... 2024-04-25T01:23:00.095 INFO:teuthology.orchestra.run.smithi028.stdout:Adding host smithi028... 2024-04-25T01:23:00.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:00 smithi028 ceph-mon[29390]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:01.616 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:01 smithi028 ceph-mon[29390]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:02.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:02 smithi028 ceph-mon[29390]: Deploying cephadm binary to smithi028 2024-04-25T01:23:05.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Added host 'smithi028' with addr '172.21.15.28' 2024-04-25T01:23:05.489 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.529+0000 7fa378e5f700 1 Processor -- start 2024-04-25T01:23:05.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.530+0000 7fa378e5f700 1 -- start start 2024-04-25T01:23:05.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.530+0000 7fa378e5f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa374107bf0 0x7fa374108010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.530+0000 7fa378e5f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa3741085e0 con 0x7fa374107bf0 2024-04-25T01:23:05.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.530+0000 7fa37259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa374107bf0 0x7fa374108010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.530+0000 7fa37259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa374107bf0 0x7fa374108010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46458/0 (socket says 172.21.15.28:46458) 2024-04-25T01:23:05.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.530+0000 7fa37259c700 1 -- 172.21.15.28:0/2012050003 learned_addr learned my addr 172.21.15.28:0/2012050003 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:05.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.530+0000 7fa37259c700 1 -- 172.21.15.28:0/2012050003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa374108da0 con 0x7fa374107bf0 2024-04-25T01:23:05.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.531+0000 7fa37259c700 1 --2- 172.21.15.28:0/2012050003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa374107bf0 0x7fa374108010 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa3640098a0 tx=0x7fa364009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad1afcbc592442be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:05.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.531+0000 7fa37159a700 1 -- 172.21.15.28:0/2012050003 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa364004160 con 0x7fa374107bf0 2024-04-25T01:23:05.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.531+0000 7fa37159a700 1 -- 172.21.15.28:0/2012050003 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa364004300 con 0x7fa374107bf0 2024-04-25T01:23:05.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.531+0000 7fa37159a700 1 -- 172.21.15.28:0/2012050003 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa364011b50 con 0x7fa374107bf0 2024-04-25T01:23:05.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.531+0000 7fa378e5f700 1 -- 172.21.15.28:0/2012050003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa374107bf0 msgr2=0x7fa374108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.531+0000 7fa378e5f700 1 --2- 172.21.15.28:0/2012050003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa374107bf0 0x7fa374108010 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa3640098a0 tx=0x7fa364009bb0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.490 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.531+0000 7fa378e5f700 1 -- 172.21.15.28:0/2012050003 shutdown_connections 2024-04-25T01:23:05.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.531+0000 7fa378e5f700 1 --2- 172.21.15.28:0/2012050003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa374107bf0 0x7fa374108010 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.531+0000 7fa378e5f700 1 -- 172.21.15.28:0/2012050003 >> 172.21.15.28:0/2012050003 conn(0x7fa374103170 msgr2=0x7fa3741055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:05.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.531+0000 7fa378e5f700 1 -- 172.21.15.28:0/2012050003 shutdown_connections 2024-04-25T01:23:05.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.532+0000 7fa378e5f700 1 -- 172.21.15.28:0/2012050003 wait complete. 2024-04-25T01:23:05.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.532+0000 7fa378e5f700 1 Processor -- start 2024-04-25T01:23:05.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.532+0000 7fa378e5f700 1 -- start start 2024-04-25T01:23:05.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.532+0000 7fa378e5f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa374107bf0 0x7fa37419bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.532+0000 7fa378e5f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa37410c150 con 0x7fa374107bf0 2024-04-25T01:23:05.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.533+0000 7fa37259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa374107bf0 0x7fa37419bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.533+0000 7fa37259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa374107bf0 0x7fa37419bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46472/0 (socket says 172.21.15.28:46472) 2024-04-25T01:23:05.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.533+0000 7fa37259c700 1 -- 172.21.15.28:0/1073946515 learned_addr learned my addr 172.21.15.28:0/1073946515 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:05.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.533+0000 7fa37259c700 1 -- 172.21.15.28:0/1073946515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa37419c100 con 0x7fa374107bf0 2024-04-25T01:23:05.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.533+0000 7fa37259c700 1 --2- 172.21.15.28:0/1073946515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa374107bf0 0x7fa37419bbc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa364011670 tx=0x7fa364011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:05.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.533+0000 7fa3637fe700 1 -- 172.21.15.28:0/1073946515 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa3640039c0 con 0x7fa374107bf0 2024-04-25T01:23:05.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.533+0000 7fa3637fe700 1 -- 172.21.15.28:0/1073946515 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa364011e50 con 0x7fa374107bf0 2024-04-25T01:23:05.491 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.533+0000 7fa378e5f700 1 -- 172.21.15.28:0/1073946515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa37419c390 con 0x7fa374107bf0 2024-04-25T01:23:05.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.533+0000 7fa3637fe700 1 -- 172.21.15.28:0/1073946515 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa36401a5c0 con 0x7fa374107bf0 2024-04-25T01:23:05.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.533+0000 7fa378e5f700 1 -- 172.21.15.28:0/1073946515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa37419f030 con 0x7fa374107bf0 2024-04-25T01:23:05.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.534+0000 7fa3637fe700 1 -- 172.21.15.28:0/1073946515 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fa364024430 con 0x7fa374107bf0 2024-04-25T01:23:05.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.534+0000 7fa378e5f700 1 -- 172.21.15.28:0/1073946515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa374063890 con 0x7fa374107bf0 2024-04-25T01:23:05.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.534+0000 7fa3637fe700 1 --2- 172.21.15.28:0/1073946515 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fa35c039000 0x7fa35c03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:05.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.534+0000 7fa3637fe700 1 -- 172.21.15.28:0/1073946515 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa36404d510 con 0x7fa374107bf0 2024-04-25T01:23:05.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.534+0000 7fa371d9b700 1 --2- 172.21.15.28:0/1073946515 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fa35c039000 0x7fa35c03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:05.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.535+0000 7fa371d9b700 1 --2- 172.21.15.28:0/1073946515 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fa35c039000 0x7fa35c03b4c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa368006fd0 tx=0x7fa368006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:05.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.540+0000 7fa3637fe700 1 -- 172.21.15.28:0/1073946515 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa364024720 con 0x7fa374107bf0 2024-04-25T01:23:05.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:00.752+0000 7fa378e5f700 1 -- 172.21.15.28:0/1073946515 --> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}) -- 0x7fa374104400 con 0x7fa35c039000 2024-04-25T01:23:05.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:04.962+0000 7fa3637fe700 1 -- 172.21.15.28:0/1073946515 <== mgr.14120 v2:172.21.15.28:6800/2407469300 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fa374104400 con 0x7fa35c039000 2024-04-25T01:23:05.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:04.964+0000 7fa378e5f700 1 -- 172.21.15.28:0/1073946515 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fa35c039000 msgr2=0x7fa35c03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:04.964+0000 7fa378e5f700 1 --2- 172.21.15.28:0/1073946515 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fa35c039000 0x7fa35c03b4c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa368006fd0 tx=0x7fa368006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:05.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:04.964+0000 7fa378e5f700 1 -- 172.21.15.28:0/1073946515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa374107bf0 msgr2=0x7fa37419bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:05.492 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:04.964+0000 7fa378e5f700 1 --2- 172.21.15.28:0/1073946515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa374107bf0 0x7fa37419bbc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa364011670 tx=0x7fa364011b60 comp rx=0 tx=0).stop 2024-04-25T01:23:05.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:04.964+0000 7fa378e5f700 1 -- 172.21.15.28:0/1073946515 shutdown_connections 2024-04-25T01:23:05.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:04.964+0000 7fa378e5f700 1 --2- 172.21.15.28:0/1073946515 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fa35c039000 0x7fa35c03b4c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:04.964+0000 7fa378e5f700 1 --2- 172.21.15.28:0/1073946515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa374107bf0 0x7fa37419bbc0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:05.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:04.964+0000 7fa378e5f700 1 -- 172.21.15.28:0/1073946515 >> 172.21.15.28:0/1073946515 conn(0x7fa374103170 msgr2=0x7fa374103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:05.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:04.964+0000 7fa378e5f700 1 -- 172.21.15.28:0/1073946515 shutdown_connections 2024-04-25T01:23:05.493 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:04.964+0000 7fa378e5f700 1 -- 172.21.15.28:0/1073946515 wait complete. 2024-04-25T01:23:05.493 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mon service... 2024-04-25T01:23:06.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:05 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' 2024-04-25T01:23:06.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:05 smithi028 ceph-mon[29390]: Added host smithi028 2024-04-25T01:23:06.227 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:05 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:07.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-25T01:23:07.152 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.011+0000 7f3d0e52e700 1 Processor -- start 2024-04-25T01:23:07.152 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.012+0000 7f3d0e52e700 1 -- start start 2024-04-25T01:23:07.152 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.012+0000 7f3d0e52e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d08108070 0x7f3d08108490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:07.152 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.012+0000 7f3d0e52e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d08108a60 con 0x7f3d08108070 2024-04-25T01:23:07.152 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.012+0000 7f3d07fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d08108070 0x7f3d08108490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:07.152 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.012+0000 7f3d07fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d08108070 0x7f3d08108490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46478/0 (socket says 172.21.15.28:46478) 2024-04-25T01:23:07.152 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.012+0000 7f3d07fff700 1 -- 172.21.15.28:0/475705340 learned_addr learned my addr 172.21.15.28:0/475705340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:07.152 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.012+0000 7f3d07fff700 1 -- 172.21.15.28:0/475705340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d08106c80 con 0x7f3d08108070 2024-04-25T01:23:07.152 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.013+0000 7f3d07fff700 1 --2- 172.21.15.28:0/475705340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d08108070 0x7f3d08108490 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80098a0 tx=0x7f3cf8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a10499fbe63392c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:07.152 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.013+0000 7f3d06ffd700 1 -- 172.21.15.28:0/475705340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cf8004160 con 0x7f3d08108070 2024-04-25T01:23:07.152 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.013+0000 7f3d06ffd700 1 -- 172.21.15.28:0/475705340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3cf8004300 con 0x7f3d08108070 2024-04-25T01:23:07.152 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.013+0000 7f3d06ffd700 1 -- 172.21.15.28:0/475705340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cf8011b50 con 0x7f3d08108070 2024-04-25T01:23:07.152 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.013+0000 7f3d0e52e700 1 -- 172.21.15.28:0/475705340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d08108070 msgr2=0x7f3d08108490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:07.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.013+0000 7f3d0e52e700 1 --2- 172.21.15.28:0/475705340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d08108070 0x7f3d08108490 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80098a0 tx=0x7f3cf8009bb0 comp rx=0 tx=0).stop 2024-04-25T01:23:07.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.013+0000 7f3d0e52e700 1 -- 172.21.15.28:0/475705340 shutdown_connections 2024-04-25T01:23:07.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.013+0000 7f3d0e52e700 1 --2- 172.21.15.28:0/475705340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d08108070 0x7f3d08108490 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:07.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.013+0000 7f3d0e52e700 1 -- 172.21.15.28:0/475705340 >> 172.21.15.28:0/475705340 conn(0x7f3d081017d0 msgr2=0x7f3d08103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:07.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.013+0000 7f3d0e52e700 1 -- 172.21.15.28:0/475705340 shutdown_connections 2024-04-25T01:23:07.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.013+0000 7f3d0e52e700 1 -- 172.21.15.28:0/475705340 wait complete. 2024-04-25T01:23:07.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.014+0000 7f3d0e52e700 1 Processor -- start 2024-04-25T01:23:07.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.014+0000 7f3d0e52e700 1 -- start start 2024-04-25T01:23:07.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.014+0000 7f3d0e52e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d08108070 0x7f3d0819bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:07.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.014+0000 7f3d0e52e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d0810b040 con 0x7f3d08108070 2024-04-25T01:23:07.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.014+0000 7f3d07fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d08108070 0x7f3d0819bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:07.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.014+0000 7f3d07fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d08108070 0x7f3d0819bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46494/0 (socket says 172.21.15.28:46494) 2024-04-25T01:23:07.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.014+0000 7f3d07fff700 1 -- 172.21.15.28:0/2029355959 learned_addr learned my addr 172.21.15.28:0/2029355959 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:07.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.015+0000 7f3d07fff700 1 -- 172.21.15.28:0/2029355959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d0819c090 con 0x7f3d08108070 2024-04-25T01:23:07.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.015+0000 7f3d07fff700 1 --2- 172.21.15.28:0/2029355959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d08108070 0x7f3d0819bb50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8011670 tx=0x7f3cf8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:07.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.015+0000 7f3d057fa700 1 -- 172.21.15.28:0/2029355959 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cf80039c0 con 0x7f3d08108070 2024-04-25T01:23:07.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.015+0000 7f3d057fa700 1 -- 172.21.15.28:0/2029355959 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3cf8011e50 con 0x7f3d08108070 2024-04-25T01:23:07.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.015+0000 7f3d0e52e700 1 -- 172.21.15.28:0/2029355959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d0819c320 con 0x7f3d08108070 2024-04-25T01:23:07.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.015+0000 7f3d057fa700 1 -- 172.21.15.28:0/2029355959 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cf801a5c0 con 0x7f3d08108070 2024-04-25T01:23:07.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.015+0000 7f3d0e52e700 1 -- 172.21.15.28:0/2029355959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d0819f010 con 0x7f3d08108070 2024-04-25T01:23:07.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.016+0000 7f3d057fa700 1 -- 172.21.15.28:0/2029355959 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f3cf801a870 con 0x7f3d08108070 2024-04-25T01:23:07.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.016+0000 7f3d0e52e700 1 -- 172.21.15.28:0/2029355959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d08063890 con 0x7f3d08108070 2024-04-25T01:23:07.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.016+0000 7f3d057fa700 1 --2- 172.21.15.28:0/2029355959 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f3cf4039050 0x7f3cf403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:07.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.016+0000 7f3d057fa700 1 -- 172.21.15.28:0/2029355959 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3cf804d420 con 0x7f3d08108070 2024-04-25T01:23:07.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.016+0000 7f3d077fe700 1 --2- 172.21.15.28:0/2029355959 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f3cf4039050 0x7f3cf403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:07.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.017+0000 7f3d077fe700 1 --2- 172.21.15.28:0/2029355959 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f3cf4039050 0x7f3cf403b510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0006fd0 tx=0x7f3cf0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:07.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.021+0000 7f3d057fa700 1 -- 172.21.15.28:0/2029355959 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3cf8019c70 con 0x7f3d08108070 2024-04-25T01:23:07.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.245+0000 7f3d0e52e700 1 -- 172.21.15.28:0/2029355959 --> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f3d080008d0 con 0x7f3cf4039050 2024-04-25T01:23:07.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.251+0000 7f3d057fa700 1 -- 172.21.15.28:0/2029355959 <== mgr.14120 v2:172.21.15.28:6800/2407469300 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f3d080008d0 con 0x7f3cf4039050 2024-04-25T01:23:07.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.253+0000 7f3d0e52e700 1 -- 172.21.15.28:0/2029355959 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f3cf4039050 msgr2=0x7f3cf403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:07.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.253+0000 7f3d0e52e700 1 --2- 172.21.15.28:0/2029355959 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f3cf4039050 0x7f3cf403b510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0006fd0 tx=0x7f3cf0006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:07.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.253+0000 7f3d0e52e700 1 -- 172.21.15.28:0/2029355959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d08108070 msgr2=0x7f3d0819bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:07.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.253+0000 7f3d0e52e700 1 --2- 172.21.15.28:0/2029355959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d08108070 0x7f3d0819bb50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8011670 tx=0x7f3cf8011b60 comp rx=0 tx=0).stop 2024-04-25T01:23:07.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.254+0000 7f3d0e52e700 1 -- 172.21.15.28:0/2029355959 shutdown_connections 2024-04-25T01:23:07.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.254+0000 7f3d0e52e700 1 --2- 172.21.15.28:0/2029355959 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f3cf4039050 0x7f3cf403b510 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:07.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.254+0000 7f3d0e52e700 1 --2- 172.21.15.28:0/2029355959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d08108070 0x7f3d0819bb50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:07.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.254+0000 7f3d0e52e700 1 -- 172.21.15.28:0/2029355959 >> 172.21.15.28:0/2029355959 conn(0x7f3d081017d0 msgr2=0x7f3d08102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:07.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.254+0000 7f3d0e52e700 1 -- 172.21.15.28:0/2029355959 shutdown_connections 2024-04-25T01:23:07.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:06.254+0000 7f3d0e52e700 1 -- 172.21.15.28:0/2029355959 wait complete. 2024-04-25T01:23:07.155 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mgr service... 2024-04-25T01:23:07.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:07 smithi028 ceph-mon[29390]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:07.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:07 smithi028 ceph-mon[29390]: Saving service mon spec with placement count:5 2024-04-25T01:23:07.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:07 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' 2024-04-25T01:23:08.984 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-25T01:23:08.984 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.230+0000 7fdd88030700 1 Processor -- start 2024-04-25T01:23:08.984 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.230+0000 7fdd88030700 1 -- start start 2024-04-25T01:23:08.984 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.230+0000 7fdd88030700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd80106af0 0x7fdd80106f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:08.984 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.230+0000 7fdd88030700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd801074e0 con 0x7fdd80106af0 2024-04-25T01:23:08.984 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.231+0000 7fdd85dcc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd80106af0 0x7fdd80106f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:08.984 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.231+0000 7fdd85dcc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd80106af0 0x7fdd80106f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36448/0 (socket says 172.21.15.28:36448) 2024-04-25T01:23:08.984 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.231+0000 7fdd85dcc700 1 -- 172.21.15.28:0/913577704 learned_addr learned my addr 172.21.15.28:0/913577704 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:08.984 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.231+0000 7fdd85dcc700 1 -- 172.21.15.28:0/913577704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd80107cf0 con 0x7fdd80106af0 2024-04-25T01:23:08.984 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.231+0000 7fdd85dcc700 1 --2- 172.21.15.28:0/913577704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd80106af0 0x7fdd80106f10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c0098a0 tx=0x7fdd7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7381fe5368b9fd5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:08.984 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.232+0000 7fdd84dca700 1 -- 172.21.15.28:0/913577704 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdd7c004160 con 0x7fdd80106af0 2024-04-25T01:23:08.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.232+0000 7fdd84dca700 1 -- 172.21.15.28:0/913577704 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd7c004300 con 0x7fdd80106af0 2024-04-25T01:23:08.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.232+0000 7fdd84dca700 1 -- 172.21.15.28:0/913577704 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdd7c011b20 con 0x7fdd80106af0 2024-04-25T01:23:08.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.232+0000 7fdd88030700 1 -- 172.21.15.28:0/913577704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd80106af0 msgr2=0x7fdd80106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:08.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.232+0000 7fdd88030700 1 --2- 172.21.15.28:0/913577704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd80106af0 0x7fdd80106f10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c0098a0 tx=0x7fdd7c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:23:08.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.232+0000 7fdd88030700 1 -- 172.21.15.28:0/913577704 shutdown_connections 2024-04-25T01:23:08.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.232+0000 7fdd88030700 1 --2- 172.21.15.28:0/913577704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd80106af0 0x7fdd80106f10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:08.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.232+0000 7fdd88030700 1 -- 172.21.15.28:0/913577704 >> 172.21.15.28:0/913577704 conn(0x7fdd801020d0 msgr2=0x7fdd801044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:08.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.233+0000 7fdd88030700 1 -- 172.21.15.28:0/913577704 shutdown_connections 2024-04-25T01:23:08.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.233+0000 7fdd88030700 1 -- 172.21.15.28:0/913577704 wait complete. 2024-04-25T01:23:08.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.233+0000 7fdd88030700 1 Processor -- start 2024-04-25T01:23:08.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.233+0000 7fdd88030700 1 -- start start 2024-04-25T01:23:08.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.233+0000 7fdd88030700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd80106af0 0x7fdd8019bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:08.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.233+0000 7fdd88030700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8010b0a0 con 0x7fdd80106af0 2024-04-25T01:23:08.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.233+0000 7fdd85dcc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd80106af0 0x7fdd8019bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:08.986 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.233+0000 7fdd85dcc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd80106af0 0x7fdd8019bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36464/0 (socket says 172.21.15.28:36464) 2024-04-25T01:23:08.986 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.233+0000 7fdd85dcc700 1 -- 172.21.15.28:0/1560205557 learned_addr learned my addr 172.21.15.28:0/1560205557 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:08.986 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.234+0000 7fdd85dcc700 1 -- 172.21.15.28:0/1560205557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8019c040 con 0x7fdd80106af0 2024-04-25T01:23:08.986 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.234+0000 7fdd85dcc700 1 --2- 172.21.15.28:0/1560205557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd80106af0 0x7fdd8019bb00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c0117e0 tx=0x7fdd7c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:08.986 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.234+0000 7fdd76ffd700 1 -- 172.21.15.28:0/1560205557 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdd7c003bf0 con 0x7fdd80106af0 2024-04-25T01:23:08.986 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.234+0000 7fdd76ffd700 1 -- 172.21.15.28:0/1560205557 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd7c01a910 con 0x7fdd80106af0 2024-04-25T01:23:08.986 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.234+0000 7fdd88030700 1 -- 172.21.15.28:0/1560205557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd8019c2d0 con 0x7fdd80106af0 2024-04-25T01:23:08.986 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.234+0000 7fdd76ffd700 1 -- 172.21.15.28:0/1560205557 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdd7c0249a0 con 0x7fdd80106af0 2024-04-25T01:23:08.986 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.234+0000 7fdd88030700 1 -- 172.21.15.28:0/1560205557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd8019c730 con 0x7fdd80106af0 2024-04-25T01:23:08.986 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.235+0000 7fdd76ffd700 1 -- 172.21.15.28:0/1560205557 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fdd7c022020 con 0x7fdd80106af0 2024-04-25T01:23:08.986 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.235+0000 7fdd88030700 1 -- 172.21.15.28:0/1560205557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd80063890 con 0x7fdd80106af0 2024-04-25T01:23:08.986 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.236+0000 7fdd76ffd700 1 --2- 172.21.15.28:0/1560205557 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fdd6c039050 0x7fdd6c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:08.986 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.236+0000 7fdd76ffd700 1 -- 172.21.15.28:0/1560205557 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdd7c04d310 con 0x7fdd80106af0 2024-04-25T01:23:08.987 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.236+0000 7fdd855cb700 1 --2- 172.21.15.28:0/1560205557 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fdd6c039050 0x7fdd6c03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:08.987 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.236+0000 7fdd855cb700 1 --2- 172.21.15.28:0/1560205557 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fdd6c039050 0x7fdd6c03b510 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdd70006fd0 tx=0x7fdd70006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:08.987 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.241+0000 7fdd76ffd700 1 -- 172.21.15.28:0/1560205557 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdd7c011e20 con 0x7fdd80106af0 2024-04-25T01:23:08.987 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.461+0000 7fdd88030700 1 -- 172.21.15.28:0/1560205557 --> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fdd80062670 con 0x7fdd6c039050 2024-04-25T01:23:08.987 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.467+0000 7fdd76ffd700 1 -- 172.21.15.28:0/1560205557 <== mgr.14120 v2:172.21.15.28:6800/2407469300 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fdd80062670 con 0x7fdd6c039050 2024-04-25T01:23:08.987 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.469+0000 7fdd88030700 1 -- 172.21.15.28:0/1560205557 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fdd6c039050 msgr2=0x7fdd6c03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:08.987 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.469+0000 7fdd88030700 1 --2- 172.21.15.28:0/1560205557 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fdd6c039050 0x7fdd6c03b510 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdd70006fd0 tx=0x7fdd70006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:08.987 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.469+0000 7fdd88030700 1 -- 172.21.15.28:0/1560205557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd80106af0 msgr2=0x7fdd8019bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:08.987 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.469+0000 7fdd88030700 1 --2- 172.21.15.28:0/1560205557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd80106af0 0x7fdd8019bb00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c0117e0 tx=0x7fdd7c011c10 comp rx=0 tx=0).stop 2024-04-25T01:23:08.987 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.469+0000 7fdd88030700 1 -- 172.21.15.28:0/1560205557 shutdown_connections 2024-04-25T01:23:08.987 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.469+0000 7fdd88030700 1 --2- 172.21.15.28:0/1560205557 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fdd6c039050 0x7fdd6c03b510 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:08.987 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.469+0000 7fdd88030700 1 --2- 172.21.15.28:0/1560205557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd80106af0 0x7fdd8019bb00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:08.987 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.469+0000 7fdd88030700 1 -- 172.21.15.28:0/1560205557 >> 172.21.15.28:0/1560205557 conn(0x7fdd801020d0 msgr2=0x7fdd80102ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:08.987 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.469+0000 7fdd88030700 1 -- 172.21.15.28:0/1560205557 shutdown_connections 2024-04-25T01:23:08.987 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:08.469+0000 7fdd88030700 1 -- 172.21.15.28:0/1560205557 wait complete. 2024-04-25T01:23:09.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:09 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' 2024-04-25T01:23:09.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:09 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' 2024-04-25T01:23:10.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:10 smithi028 ceph-mon[29390]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:10.307 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:10 smithi028 ceph-mon[29390]: Saving service mgr spec with placement count:2 2024-04-25T01:23:10.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:10 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' 2024-04-25T01:23:10.308 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:10 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/885204357' entity='client.admin' 2024-04-25T01:23:10.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.463+0000 7f7dfe8c8700 1 Processor -- start 2024-04-25T01:23:10.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.463+0000 7f7dfe8c8700 1 -- start start 2024-04-25T01:23:10.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.463+0000 7f7dfe8c8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df8108690 0x7f7df810aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.463+0000 7f7dfe8c8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7df807be90 con 0x7f7df8108690 2024-04-25T01:23:10.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.464+0000 7f7df7fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df8108690 0x7f7df810aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.464+0000 7f7df7fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df8108690 0x7f7df810aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36466/0 (socket says 172.21.15.28:36466) 2024-04-25T01:23:10.315 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.464+0000 7f7df7fff700 1 -- 172.21.15.28:0/4140910762 learned_addr learned my addr 172.21.15.28:0/4140910762 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:10.316 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.464+0000 7f7df7fff700 1 -- 172.21.15.28:0/4140910762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df810afc0 con 0x7f7df8108690 2024-04-25T01:23:10.316 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.464+0000 7f7df7fff700 1 --2- 172.21.15.28:0/4140910762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df8108690 0x7f7df810aa80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7de80098a0 tx=0x7f7de8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab7da20b5c3f76ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.316 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.464+0000 7f7df6ffd700 1 -- 172.21.15.28:0/4140910762 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7de8004160 con 0x7f7df8108690 2024-04-25T01:23:10.316 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.464+0000 7f7df6ffd700 1 -- 172.21.15.28:0/4140910762 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7de8004300 con 0x7f7df8108690 2024-04-25T01:23:10.316 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.464+0000 7f7df6ffd700 1 -- 172.21.15.28:0/4140910762 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7de8011b20 con 0x7f7df8108690 2024-04-25T01:23:10.316 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.465+0000 7f7dfe8c8700 1 -- 172.21.15.28:0/4140910762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df8108690 msgr2=0x7f7df810aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.316 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.465+0000 7f7dfe8c8700 1 --2- 172.21.15.28:0/4140910762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df8108690 0x7f7df810aa80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7de80098a0 tx=0x7f7de8009bb0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.316 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.465+0000 7f7dfe8c8700 1 -- 172.21.15.28:0/4140910762 shutdown_connections 2024-04-25T01:23:10.316 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.465+0000 7f7dfe8c8700 1 --2- 172.21.15.28:0/4140910762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df8108690 0x7f7df810aa80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.316 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.465+0000 7f7dfe8c8700 1 -- 172.21.15.28:0/4140910762 >> 172.21.15.28:0/4140910762 conn(0x7f7df81020a0 msgr2=0x7f7df81044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:10.316 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.465+0000 7f7dfe8c8700 1 -- 172.21.15.28:0/4140910762 shutdown_connections 2024-04-25T01:23:10.316 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.465+0000 7f7dfe8c8700 1 -- 172.21.15.28:0/4140910762 wait complete. 2024-04-25T01:23:10.316 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.466+0000 7f7dfe8c8700 1 Processor -- start 2024-04-25T01:23:10.316 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.466+0000 7f7dfe8c8700 1 -- start start 2024-04-25T01:23:10.316 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.466+0000 7f7dfe8c8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df8108690 0x7f7df819bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.466+0000 7f7dfe8c8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7df810c020 con 0x7f7df8108690 2024-04-25T01:23:10.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.466+0000 7f7df7fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df8108690 0x7f7df819bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.466+0000 7f7df7fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df8108690 0x7f7df819bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36478/0 (socket says 172.21.15.28:36478) 2024-04-25T01:23:10.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.466+0000 7f7df7fff700 1 -- 172.21.15.28:0/885204357 learned_addr learned my addr 172.21.15.28:0/885204357 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:10.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.466+0000 7f7df7fff700 1 -- 172.21.15.28:0/885204357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df819c050 con 0x7f7df8108690 2024-04-25T01:23:10.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.467+0000 7f7df7fff700 1 --2- 172.21.15.28:0/885204357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df8108690 0x7f7df819bb10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7de80117e0 tx=0x7f7de8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.467+0000 7f7df57fa700 1 -- 172.21.15.28:0/885204357 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7de8003bf0 con 0x7f7df8108690 2024-04-25T01:23:10.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.467+0000 7f7df57fa700 1 -- 172.21.15.28:0/885204357 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7de801a910 con 0x7f7df8108690 2024-04-25T01:23:10.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.467+0000 7f7dfe8c8700 1 -- 172.21.15.28:0/885204357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7df819c2e0 con 0x7f7df8108690 2024-04-25T01:23:10.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.467+0000 7f7df57fa700 1 -- 172.21.15.28:0/885204357 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7de8024970 con 0x7f7df8108690 2024-04-25T01:23:10.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.467+0000 7f7dfe8c8700 1 -- 172.21.15.28:0/885204357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7df819c740 con 0x7f7df8108690 2024-04-25T01:23:10.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.468+0000 7f7df57fa700 1 -- 172.21.15.28:0/885204357 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f7de8022020 con 0x7f7df8108690 2024-04-25T01:23:10.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.468+0000 7f7dfe8c8700 1 -- 172.21.15.28:0/885204357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7df8063890 con 0x7f7df8108690 2024-04-25T01:23:10.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.468+0000 7f7df57fa700 1 --2- 172.21.15.28:0/885204357 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f7de0039050 0x7f7de003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:10.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.468+0000 7f7df57fa700 1 -- 172.21.15.28:0/885204357 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7de804d370 con 0x7f7df8108690 2024-04-25T01:23:10.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.468+0000 7f7df77fe700 1 --2- 172.21.15.28:0/885204357 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f7de0039050 0x7f7de003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:10.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.469+0000 7f7df77fe700 1 --2- 172.21.15.28:0/885204357 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f7de0039050 0x7f7de003b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7dec006fd0 tx=0x7f7dec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:10.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.474+0000 7f7df57fa700 1 -- 172.21.15.28:0/885204357 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7de8011e20 con 0x7f7df8108690 2024-04-25T01:23:10.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.691+0000 7f7dfe8c8700 1 -- 172.21.15.28:0/885204357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f7df819efd0 con 0x7f7df8108690 2024-04-25T01:23:10.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.693+0000 7f7df57fa700 1 -- 172.21.15.28:0/885204357 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f7de800b3e0 con 0x7f7df8108690 2024-04-25T01:23:10.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.695+0000 7f7dfe8c8700 1 -- 172.21.15.28:0/885204357 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f7de0039050 msgr2=0x7f7de003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.695+0000 7f7dfe8c8700 1 --2- 172.21.15.28:0/885204357 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f7de0039050 0x7f7de003b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7dec006fd0 tx=0x7f7dec006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:10.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.695+0000 7f7dfe8c8700 1 -- 172.21.15.28:0/885204357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df8108690 msgr2=0x7f7df819bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:10.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.695+0000 7f7dfe8c8700 1 --2- 172.21.15.28:0/885204357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df8108690 0x7f7df819bb10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7de80117e0 tx=0x7f7de8011c10 comp rx=0 tx=0).stop 2024-04-25T01:23:10.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.695+0000 7f7dfe8c8700 1 -- 172.21.15.28:0/885204357 shutdown_connections 2024-04-25T01:23:10.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.695+0000 7f7dfe8c8700 1 --2- 172.21.15.28:0/885204357 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f7de0039050 0x7f7de003b510 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.695+0000 7f7dfe8c8700 1 --2- 172.21.15.28:0/885204357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7df8108690 0x7f7df819bb10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:10.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.695+0000 7f7dfe8c8700 1 -- 172.21.15.28:0/885204357 >> 172.21.15.28:0/885204357 conn(0x7f7df81020a0 msgr2=0x7f7df8102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:10.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.695+0000 7f7dfe8c8700 1 -- 172.21.15.28:0/885204357 shutdown_connections 2024-04-25T01:23:10.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:09.696+0000 7f7dfe8c8700 1 -- 172.21.15.28:0/885204357 wait complete. 2024-04-25T01:23:12.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:11 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3608217111' entity='client.admin' 2024-04-25T01:23:12.234 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:11 smithi028 ceph-mon[29390]: from='mgr.14120 172.21.15.28:0/1059139565' entity='mgr.x' 2024-04-25T01:23:12.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.770+0000 7fc48b40a700 1 Processor -- start 2024-04-25T01:23:12.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.770+0000 7fc48b40a700 1 -- start start 2024-04-25T01:23:12.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.770+0000 7fc48b40a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484107c50 0x7fc484108070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.770+0000 7fc48b40a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc484108640 con 0x7fc484107c50 2024-04-25T01:23:12.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.771+0000 7fc4891a6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484107c50 0x7fc484108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.771+0000 7fc4891a6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484107c50 0x7fc484108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36490/0 (socket says 172.21.15.28:36490) 2024-04-25T01:23:12.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.771+0000 7fc4891a6700 1 -- 172.21.15.28:0/2702695223 learned_addr learned my addr 172.21.15.28:0/2702695223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:12.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.771+0000 7fc4891a6700 1 -- 172.21.15.28:0/2702695223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc484108db0 con 0x7fc484107c50 2024-04-25T01:23:12.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.771+0000 7fc4891a6700 1 --2- 172.21.15.28:0/2702695223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484107c50 0x7fc484108070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc47c009e80 tx=0x7fc47c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f07d2ceb922f4764 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:12.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.771+0000 7fc483fff700 1 -- 172.21.15.28:0/2702695223 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc47c017070 con 0x7fc484107c50 2024-04-25T01:23:12.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.772+0000 7fc483fff700 1 -- 172.21.15.28:0/2702695223 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc47c00b8c0 con 0x7fc484107c50 2024-04-25T01:23:12.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.772+0000 7fc483fff700 1 -- 172.21.15.28:0/2702695223 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc47c0045c0 con 0x7fc484107c50 2024-04-25T01:23:12.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.772+0000 7fc48b40a700 1 -- 172.21.15.28:0/2702695223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484107c50 msgr2=0x7fc484108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.772+0000 7fc48b40a700 1 --2- 172.21.15.28:0/2702695223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484107c50 0x7fc484108070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc47c009e80 tx=0x7fc47c00b330 comp rx=0 tx=0).stop 2024-04-25T01:23:12.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.772+0000 7fc48b40a700 1 -- 172.21.15.28:0/2702695223 shutdown_connections 2024-04-25T01:23:12.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.772+0000 7fc48b40a700 1 --2- 172.21.15.28:0/2702695223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484107c50 0x7fc484108070 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.772+0000 7fc48b40a700 1 -- 172.21.15.28:0/2702695223 >> 172.21.15.28:0/2702695223 conn(0x7fc484103230 msgr2=0x7fc484105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:12.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.772+0000 7fc48b40a700 1 -- 172.21.15.28:0/2702695223 shutdown_connections 2024-04-25T01:23:12.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.772+0000 7fc48b40a700 1 -- 172.21.15.28:0/2702695223 wait complete. 2024-04-25T01:23:12.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.773+0000 7fc48b40a700 1 Processor -- start 2024-04-25T01:23:12.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.773+0000 7fc48b40a700 1 -- start start 2024-04-25T01:23:12.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.773+0000 7fc48b40a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484107c50 0x7fc48419bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.773+0000 7fc48b40a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc48410c160 con 0x7fc484107c50 2024-04-25T01:23:12.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.773+0000 7fc4891a6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484107c50 0x7fc48419bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.773+0000 7fc4891a6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484107c50 0x7fc48419bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36496/0 (socket says 172.21.15.28:36496) 2024-04-25T01:23:12.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.773+0000 7fc4891a6700 1 -- 172.21.15.28:0/3608217111 learned_addr learned my addr 172.21.15.28:0/3608217111 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:12.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.773+0000 7fc4891a6700 1 -- 172.21.15.28:0/3608217111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc48419c0e0 con 0x7fc484107c50 2024-04-25T01:23:12.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.774+0000 7fc4891a6700 1 --2- 172.21.15.28:0/3608217111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484107c50 0x7fc48419bba0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc47c003fa0 tx=0x7fc47c004080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:12.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.774+0000 7fc4827fc700 1 -- 172.21.15.28:0/3608217111 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc47c020020 con 0x7fc484107c50 2024-04-25T01:23:12.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.774+0000 7fc48b40a700 1 -- 172.21.15.28:0/3608217111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc48419c370 con 0x7fc484107c50 2024-04-25T01:23:12.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.774+0000 7fc4827fc700 1 -- 172.21.15.28:0/3608217111 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc47c00bba0 con 0x7fc484107c50 2024-04-25T01:23:12.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.774+0000 7fc4827fc700 1 -- 172.21.15.28:0/3608217111 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc47c017040 con 0x7fc484107c50 2024-04-25T01:23:12.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.774+0000 7fc48b40a700 1 -- 172.21.15.28:0/3608217111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc48419f060 con 0x7fc484107c50 2024-04-25T01:23:12.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.775+0000 7fc4827fc700 1 -- 172.21.15.28:0/3608217111 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fc47c023020 con 0x7fc484107c50 2024-04-25T01:23:12.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.775+0000 7fc48b40a700 1 -- 172.21.15.28:0/3608217111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc484063890 con 0x7fc484107c50 2024-04-25T01:23:12.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.775+0000 7fc4827fc700 1 --2- 172.21.15.28:0/3608217111 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fc46c039000 0x7fc46c03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:12.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.775+0000 7fc4827fc700 1 -- 172.21.15.28:0/3608217111 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc47c00f050 con 0x7fc484107c50 2024-04-25T01:23:12.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.775+0000 7fc4889a5700 1 --2- 172.21.15.28:0/3608217111 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fc46c039000 0x7fc46c03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:12.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.775+0000 7fc4889a5700 1 --2- 172.21.15.28:0/3608217111 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fc46c039000 0x7fc46c03b4c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc478006fd0 tx=0x7fc478006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:12.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.780+0000 7fc4827fc700 1 -- 172.21.15.28:0/3608217111 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc47c02a020 con 0x7fc484107c50 2024-04-25T01:23:12.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:10.997+0000 7fc48b40a700 1 -- 172.21.15.28:0/3608217111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fc48419f230 con 0x7fc484107c50 2024-04-25T01:23:12.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:11.000+0000 7fc4827fc700 1 -- 172.21.15.28:0/3608217111 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fc48419f230 con 0x7fc484107c50 2024-04-25T01:23:12.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:11.002+0000 7fc48b40a700 1 -- 172.21.15.28:0/3608217111 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fc46c039000 msgr2=0x7fc46c03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:11.002+0000 7fc48b40a700 1 --2- 172.21.15.28:0/3608217111 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fc46c039000 0x7fc46c03b4c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc478006fd0 tx=0x7fc478006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:12.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:11.002+0000 7fc48b40a700 1 -- 172.21.15.28:0/3608217111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484107c50 msgr2=0x7fc48419bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:12.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:11.002+0000 7fc48b40a700 1 --2- 172.21.15.28:0/3608217111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484107c50 0x7fc48419bba0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc47c003fa0 tx=0x7fc47c004080 comp rx=0 tx=0).stop 2024-04-25T01:23:12.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:11.002+0000 7fc48b40a700 1 -- 172.21.15.28:0/3608217111 shutdown_connections 2024-04-25T01:23:12.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:11.002+0000 7fc48b40a700 1 --2- 172.21.15.28:0/3608217111 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7fc46c039000 0x7fc46c03b4c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:11.002+0000 7fc48b40a700 1 --2- 172.21.15.28:0/3608217111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484107c50 0x7fc48419bba0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:12.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:11.002+0000 7fc48b40a700 1 -- 172.21.15.28:0/3608217111 >> 172.21.15.28:0/3608217111 conn(0x7fc484103230 msgr2=0x7fc484103d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:12.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:11.002+0000 7fc48b40a700 1 -- 172.21.15.28:0/3608217111 shutdown_connections 2024-04-25T01:23:12.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:11.002+0000 7fc48b40a700 1 -- 172.21.15.28:0/3608217111 wait complete. 2024-04-25T01:23:12.246 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling the dashboard module... 2024-04-25T01:23:13.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:12 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3396151816' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-25T01:23:13.231 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:13 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: ignoring --setuser ceph since I am not root 2024-04-25T01:23:13.231 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:13 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: ignoring --setgroup ceph since I am not root 2024-04-25T01:23:13.232 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:13 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:13.192+0000 7f9040de4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:23:13.564 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:13 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:13.390+0000 7f9040de4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:23:13.565 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:13 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:13.497+0000 7f9040de4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:23:13.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.705+0000 7f8159f0a700 1 Processor -- start 2024-04-25T01:23:13.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.705+0000 7f8159f0a700 1 -- start start 2024-04-25T01:23:13.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.706+0000 7f8159f0a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815407de20 0x7f815407c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.706+0000 7f8159f0a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f815407c860 con 0x7f815407de20 2024-04-25T01:23:13.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.706+0000 7f81537fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815407de20 0x7f815407c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.706+0000 7f81537fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815407de20 0x7f815407c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36506/0 (socket says 172.21.15.28:36506) 2024-04-25T01:23:13.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.706+0000 7f81537fe700 1 -- 172.21.15.28:0/2207849017 learned_addr learned my addr 172.21.15.28:0/2207849017 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:13.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.706+0000 7f81537fe700 1 -- 172.21.15.28:0/2207849017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f815407c9e0 con 0x7f815407de20 2024-04-25T01:23:13.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.706+0000 7f81537fe700 1 --2- 172.21.15.28:0/2207849017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815407de20 0x7f815407c320 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f81440098a0 tx=0x7f8144009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d457cc784f18e70 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:13.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.706+0000 7f8152ffd700 1 -- 172.21.15.28:0/2207849017 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8144004160 con 0x7f815407de20 2024-04-25T01:23:13.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.706+0000 7f8152ffd700 1 -- 172.21.15.28:0/2207849017 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8144004300 con 0x7f815407de20 2024-04-25T01:23:13.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.706+0000 7f8152ffd700 1 -- 172.21.15.28:0/2207849017 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8144011b20 con 0x7f815407de20 2024-04-25T01:23:13.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.707+0000 7f8159f0a700 1 -- 172.21.15.28:0/2207849017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815407de20 msgr2=0x7f815407c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.707+0000 7f8159f0a700 1 --2- 172.21.15.28:0/2207849017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815407de20 0x7f815407c320 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f81440098a0 tx=0x7f8144009bb0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.707+0000 7f8159f0a700 1 -- 172.21.15.28:0/2207849017 shutdown_connections 2024-04-25T01:23:13.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.707+0000 7f8159f0a700 1 --2- 172.21.15.28:0/2207849017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815407de20 0x7f815407c320 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.707+0000 7f8159f0a700 1 -- 172.21.15.28:0/2207849017 >> 172.21.15.28:0/2207849017 conn(0x7f8154103230 msgr2=0x7f8154105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:13.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.707+0000 7f8159f0a700 1 -- 172.21.15.28:0/2207849017 shutdown_connections 2024-04-25T01:23:13.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.707+0000 7f8159f0a700 1 -- 172.21.15.28:0/2207849017 wait complete. 2024-04-25T01:23:13.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.708+0000 7f8159f0a700 1 Processor -- start 2024-04-25T01:23:13.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.708+0000 7f8159f0a700 1 -- start start 2024-04-25T01:23:13.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.708+0000 7f8159f0a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815407de20 0x7f815419dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.708+0000 7f8159f0a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f815410e1b0 con 0x7f815407de20 2024-04-25T01:23:13.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.708+0000 7f81537fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815407de20 0x7f815419dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.708+0000 7f81537fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815407de20 0x7f815419dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36510/0 (socket says 172.21.15.28:36510) 2024-04-25T01:23:13.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.708+0000 7f81537fe700 1 -- 172.21.15.28:0/3396151816 learned_addr learned my addr 172.21.15.28:0/3396151816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:13.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.709+0000 7f81537fe700 1 -- 172.21.15.28:0/3396151816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f815419e230 con 0x7f815407de20 2024-04-25T01:23:13.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.709+0000 7f81537fe700 1 --2- 172.21.15.28:0/3396151816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815407de20 0x7f815419dcf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8144011b60 tx=0x7f8144011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:13.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.709+0000 7f81517fa700 1 -- 172.21.15.28:0/3396151816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8144003b60 con 0x7f815407de20 2024-04-25T01:23:13.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.709+0000 7f8159f0a700 1 -- 172.21.15.28:0/3396151816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f815419e4c0 con 0x7f815407de20 2024-04-25T01:23:13.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.709+0000 7f81517fa700 1 -- 172.21.15.28:0/3396151816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f814401a910 con 0x7f815407de20 2024-04-25T01:23:13.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.709+0000 7f81517fa700 1 -- 172.21.15.28:0/3396151816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8144024930 con 0x7f815407de20 2024-04-25T01:23:13.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.709+0000 7f8159f0a700 1 -- 172.21.15.28:0/3396151816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f815419e920 con 0x7f815407de20 2024-04-25T01:23:13.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.710+0000 7f81517fa700 1 -- 172.21.15.28:0/3396151816 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f8144022020 con 0x7f815407de20 2024-04-25T01:23:13.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.710+0000 7f8159f0a700 1 -- 172.21.15.28:0/3396151816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8154063890 con 0x7f815407de20 2024-04-25T01:23:13.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.711+0000 7f81517fa700 1 --2- 172.21.15.28:0/3396151816 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f8140039050 0x7f814003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:13.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.711+0000 7f81517fa700 1 -- 172.21.15.28:0/3396151816 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f814404d270 con 0x7f815407de20 2024-04-25T01:23:13.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.711+0000 7f814affd700 1 --2- 172.21.15.28:0/3396151816 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f8140039050 0x7f814003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:13.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.711+0000 7f814affd700 1 --2- 172.21.15.28:0/3396151816 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f8140039050 0x7f814003b510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f813c006fd0 tx=0x7f813c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:13.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.716+0000 7f81517fa700 1 -- 172.21.15.28:0/3396151816 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8144019c70 con 0x7f815407de20 2024-04-25T01:23:13.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:12.983+0000 7f8159f0a700 1 -- 172.21.15.28:0/3396151816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f81541a1380 con 0x7f815407de20 2024-04-25T01:23:13.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:13.000+0000 7f81517fa700 1 -- 172.21.15.28:0/3396151816 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f81440172b0 con 0x7f815407de20 2024-04-25T01:23:13.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:13.002+0000 7f81517fa700 1 -- 172.21.15.28:0/3396151816 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f814402d910 con 0x7f815407de20 2024-04-25T01:23:13.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:13.003+0000 7f8159f0a700 1 -- 172.21.15.28:0/3396151816 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f8140039050 msgr2=0x7f814003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:13.003+0000 7f8159f0a700 1 --2- 172.21.15.28:0/3396151816 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f8140039050 0x7f814003b510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f813c006fd0 tx=0x7f813c006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:13.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:13.003+0000 7f8159f0a700 1 -- 172.21.15.28:0/3396151816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815407de20 msgr2=0x7f815419dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:13.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:13.003+0000 7f8159f0a700 1 --2- 172.21.15.28:0/3396151816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815407de20 0x7f815419dcf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8144011b60 tx=0x7f8144011c40 comp rx=0 tx=0).stop 2024-04-25T01:23:13.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:13.004+0000 7f8159f0a700 1 -- 172.21.15.28:0/3396151816 shutdown_connections 2024-04-25T01:23:13.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:13.004+0000 7f8159f0a700 1 --2- 172.21.15.28:0/3396151816 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f8140039050 0x7f814003b510 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.643 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:13.004+0000 7f8159f0a700 1 --2- 172.21.15.28:0/3396151816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815407de20 0x7f815419dcf0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:13.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:13.004+0000 7f8159f0a700 1 -- 172.21.15.28:0/3396151816 >> 172.21.15.28:0/3396151816 conn(0x7f8154103230 msgr2=0x7f8154103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:13.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:13.004+0000 7f8159f0a700 1 -- 172.21.15.28:0/3396151816 shutdown_connections 2024-04-25T01:23:13.644 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:13.004+0000 7f8159f0a700 1 -- 172.21.15.28:0/3396151816 wait complete. 2024-04-25T01:23:13.897 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:13 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:13.826+0000 7f9040de4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:23:13.897 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:13 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:13.916+0000 7f9040de4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:23:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:14 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3396151816' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-25T01:23:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:14 smithi028 ceph-mon[29390]: mgrmap e9: x(active, since 19s) 2024-04-25T01:23:15.049 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:23:15.055 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-25T01:23:15.055 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T01:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T01:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T01:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.124+0000 7f13212bf700 1 Processor -- start 2024-04-25T01:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.125+0000 7f13212bf700 1 -- start start 2024-04-25T01:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.125+0000 7f13212bf700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f131c107bf0 0x7f131c108010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.125+0000 7f13212bf700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f131c1085e0 con 0x7f131c107bf0 2024-04-25T01:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.125+0000 7f131affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f131c107bf0 0x7f131c108010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.125+0000 7f131affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f131c107bf0 0x7f131c108010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36526/0 (socket says 172.21.15.28:36526) 2024-04-25T01:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.125+0000 7f131affd700 1 -- 172.21.15.28:0/672006255 learned_addr learned my addr 172.21.15.28:0/672006255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.125+0000 7f131affd700 1 -- 172.21.15.28:0/672006255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131c108da0 con 0x7f131c107bf0 2024-04-25T01:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.126+0000 7f131affd700 1 --2- 172.21.15.28:0/672006255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f131c107bf0 0x7f131c108010 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f130c0098a0 tx=0x7f130c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed610b87470a6b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.126+0000 7f1319ffb700 1 -- 172.21.15.28:0/672006255 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f130c004160 con 0x7f131c107bf0 2024-04-25T01:23:15.056 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.126+0000 7f1319ffb700 1 -- 172.21.15.28:0/672006255 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f130c004300 con 0x7f131c107bf0 2024-04-25T01:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.126+0000 7f1319ffb700 1 -- 172.21.15.28:0/672006255 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f130c011b20 con 0x7f131c107bf0 2024-04-25T01:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.126+0000 7f13212bf700 1 -- 172.21.15.28:0/672006255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f131c107bf0 msgr2=0x7f131c108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.126+0000 7f13212bf700 1 --2- 172.21.15.28:0/672006255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f131c107bf0 0x7f131c108010 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f130c0098a0 tx=0x7f130c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.126+0000 7f13212bf700 1 -- 172.21.15.28:0/672006255 shutdown_connections 2024-04-25T01:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.126+0000 7f13212bf700 1 --2- 172.21.15.28:0/672006255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f131c107bf0 0x7f131c108010 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.126+0000 7f13212bf700 1 -- 172.21.15.28:0/672006255 >> 172.21.15.28:0/672006255 conn(0x7f131c103170 msgr2=0x7f131c1055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.127+0000 7f13212bf700 1 -- 172.21.15.28:0/672006255 shutdown_connections 2024-04-25T01:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.127+0000 7f13212bf700 1 -- 172.21.15.28:0/672006255 wait complete. 2024-04-25T01:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.127+0000 7f13212bf700 1 Processor -- start 2024-04-25T01:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.127+0000 7f13212bf700 1 -- start start 2024-04-25T01:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.127+0000 7f13212bf700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f131c107bf0 0x7f131c19bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.127+0000 7f13212bf700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f131c10c150 con 0x7f131c107bf0 2024-04-25T01:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.128+0000 7f131affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f131c107bf0 0x7f131c19bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:15.057 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.128+0000 7f131affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f131c107bf0 0x7f131c19bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36542/0 (socket says 172.21.15.28:36542) 2024-04-25T01:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.128+0000 7f131affd700 1 -- 172.21.15.28:0/1105297205 learned_addr learned my addr 172.21.15.28:0/1105297205 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.128+0000 7f131affd700 1 -- 172.21.15.28:0/1105297205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131c19c050 con 0x7f131c107bf0 2024-04-25T01:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.128+0000 7f131affd700 1 --2- 172.21.15.28:0/1105297205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f131c107bf0 0x7f131c19bb10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f130c0117e0 tx=0x7f130c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.128+0000 7f1303fff700 1 -- 172.21.15.28:0/1105297205 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f130c003bf0 con 0x7f131c107bf0 2024-04-25T01:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.128+0000 7f1303fff700 1 -- 172.21.15.28:0/1105297205 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f130c01a910 con 0x7f131c107bf0 2024-04-25T01:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.128+0000 7f13212bf700 1 -- 172.21.15.28:0/1105297205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f131c19c2e0 con 0x7f131c107bf0 2024-04-25T01:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.128+0000 7f1303fff700 1 -- 172.21.15.28:0/1105297205 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f130c0249a0 con 0x7f131c107bf0 2024-04-25T01:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.128+0000 7f13212bf700 1 -- 172.21.15.28:0/1105297205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f131c19c740 con 0x7f131c107bf0 2024-04-25T01:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.129+0000 7f1303fff700 1 -- 172.21.15.28:0/1105297205 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f130c022020 con 0x7f131c107bf0 2024-04-25T01:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.129+0000 7f1303fff700 1 --2- 172.21.15.28:0/1105297205 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f13040390b0 0x7f130403b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.129+0000 7f1303fff700 1 -- 172.21.15.28:0/1105297205 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f130c04d350 con 0x7f131c107bf0 2024-04-25T01:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.129+0000 7f131a7fc700 1 -- 172.21.15.28:0/1105297205 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f13040390b0 msgr2=0x7f130403b570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2407469300 2024-04-25T01:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.129+0000 7f131a7fc700 1 --2- 172.21.15.28:0/1105297205 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f13040390b0 0x7f130403b570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:23:15.058 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.130+0000 7f13212bf700 1 -- 172.21.15.28:0/1105297205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f131c063890 con 0x7f131c107bf0 2024-04-25T01:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.135+0000 7f1303fff700 1 -- 172.21.15.28:0/1105297205 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f130c011e20 con 0x7f131c107bf0 2024-04-25T01:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.330+0000 7f131a7fc700 1 -- 172.21.15.28:0/1105297205 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f13040390b0 msgr2=0x7f130403b570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2407469300 2024-04-25T01:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.330+0000 7f131a7fc700 1 --2- 172.21.15.28:0/1105297205 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f13040390b0 0x7f130403b570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.520+0000 7f13212bf700 1 -- 172.21.15.28:0/1105297205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f131c19f070 con 0x7f131c107bf0 2024-04-25T01:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.520+0000 7f1303fff700 1 -- 172.21.15.28:0/1105297205 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f130c00ba00 con 0x7f131c107bf0 2024-04-25T01:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.522+0000 7f13212bf700 1 -- 172.21.15.28:0/1105297205 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f13040390b0 msgr2=0x7f130403b570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.522+0000 7f13212bf700 1 --2- 172.21.15.28:0/1105297205 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f13040390b0 0x7f130403b570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.522+0000 7f13212bf700 1 -- 172.21.15.28:0/1105297205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f131c107bf0 msgr2=0x7f131c19bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.522+0000 7f13212bf700 1 --2- 172.21.15.28:0/1105297205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f131c107bf0 0x7f131c19bb10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f130c0117e0 tx=0x7f130c011c10 comp rx=0 tx=0).stop 2024-04-25T01:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.523+0000 7f13212bf700 1 -- 172.21.15.28:0/1105297205 shutdown_connections 2024-04-25T01:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.523+0000 7f13212bf700 1 --2- 172.21.15.28:0/1105297205 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f13040390b0 0x7f130403b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.523+0000 7f13212bf700 1 --2- 172.21.15.28:0/1105297205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f131c107bf0 0x7f131c19bb10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.523+0000 7f13212bf700 1 -- 172.21.15.28:0/1105297205 >> 172.21.15.28:0/1105297205 conn(0x7f131c103170 msgr2=0x7f131c103c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.523+0000 7f13212bf700 1 -- 172.21.15.28:0/1105297205 shutdown_connections 2024-04-25T01:23:15.059 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:14.523+0000 7f13212bf700 1 -- 172.21.15.28:0/1105297205 wait complete. 2024-04-25T01:23:15.060 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-04-25T01:23:15.060 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 9... 2024-04-25T01:23:15.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:15 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1105297205' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T01:23:15.416 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:15 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:15.251+0000 7f9040de4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:23:15.416 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:15 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:15.435+0000 7f9040de4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:23:15.676 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:15 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:15.615+0000 7f9040de4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:23:15.995 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:15 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:15.917+0000 7f9040de4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:23:15.995 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:16 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:16.014+0000 7f9040de4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:23:16.314 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:16 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:16.224+0000 7f9040de4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:23:16.814 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:16 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:16.426+0000 7f9040de4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:23:17.314 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:17 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:17.055+0000 7f9040de4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:23:17.789 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:17 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:17.497+0000 7f9040de4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:23:17.790 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:17 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:17.598+0000 7f9040de4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:23:17.790 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:17 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:17.808+0000 7f9040de4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:23:18.684 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:18 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:18.354+0000 7f9040de4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:23:18.684 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:18 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:18.450+0000 7f9040de4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:23:19.050 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:18 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:18.703+0000 7f9040de4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:23:19.050 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:18 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:18.848+0000 7f9040de4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:23:19.050 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:18 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:18.948+0000 7f9040de4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:23:19.050 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:19 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:19.068+0000 7f9040de4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:23:19.315 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:19 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:19.165+0000 7f9040de4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:23:20.276 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:20 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:20.016+0000 7f9040de4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:23:20.276 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:23:20 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:23:20.242+0000 7f9040de4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:23:20.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:20 smithi028 ceph-mon[29390]: Active manager daemon x restarted 2024-04-25T01:23:20.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:20 smithi028 ceph-mon[29390]: Activating manager daemon x 2024-04-25T01:23:20.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:20 smithi028 ceph-mon[29390]: osdmap e3: 0 total, 0 up, 0 in 2024-04-25T01:23:20.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:20 smithi028 ceph-mon[29390]: mgrmap e10: x(active, starting, since 0.00299243s) 2024-04-25T01:23:20.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:23:20.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:23:20.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:23:20.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:23:20.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:23:20.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:20 smithi028 ceph-mon[29390]: Manager daemon x is now available 2024-04-25T01:23:20.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:21.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:21 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:23:21.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:21 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:23:21.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:21 smithi028 ceph-mon[29390]: [25/Apr/2024:01:23:20] ENGINE Bus STARTING 2024-04-25T01:23:21.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:21 smithi028 ceph-mon[29390]: mgrmap e11: x(active, since 1.00592s) 2024-04-25T01:23:21.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:23:21.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-25T01:23:21.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T01:23:21.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:23:21.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.489+0000 7f0762f9c700 1 Processor -- start 2024-04-25T01:23:21.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.489+0000 7f0762f9c700 1 -- start start 2024-04-25T01:23:21.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.489+0000 7f0762f9c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075c1080b0 0x7f075c1084d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.489+0000 7f0762f9c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f075c108aa0 con 0x7f075c1080b0 2024-04-25T01:23:21.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.489+0000 7f0760d38700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075c1080b0 0x7f075c1084d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:21.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.489+0000 7f0760d38700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075c1080b0 0x7f075c1084d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36548/0 (socket says 172.21.15.28:36548) 2024-04-25T01:23:21.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.489+0000 7f0760d38700 1 -- 172.21.15.28:0/840269231 learned_addr learned my addr 172.21.15.28:0/840269231 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:21.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.490+0000 7f0760d38700 1 -- 172.21.15.28:0/840269231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f075c106cc0 con 0x7f075c1080b0 2024-04-25T01:23:21.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.490+0000 7f0760d38700 1 --2- 172.21.15.28:0/840269231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075c1080b0 0x7f075c1084d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f07500098a0 tx=0x7f0750009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe38071a27d87410 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:21.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.490+0000 7f075b7fe700 1 -- 172.21.15.28:0/840269231 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0750004160 con 0x7f075c1080b0 2024-04-25T01:23:21.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.490+0000 7f075b7fe700 1 -- 172.21.15.28:0/840269231 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0750004300 con 0x7f075c1080b0 2024-04-25T01:23:21.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.490+0000 7f075b7fe700 1 -- 172.21.15.28:0/840269231 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0750011b20 con 0x7f075c1080b0 2024-04-25T01:23:21.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.491+0000 7f0762f9c700 1 -- 172.21.15.28:0/840269231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075c1080b0 msgr2=0x7f075c1084d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:21.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.491+0000 7f0762f9c700 1 --2- 172.21.15.28:0/840269231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075c1080b0 0x7f075c1084d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f07500098a0 tx=0x7f0750009bb0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.491+0000 7f0762f9c700 1 -- 172.21.15.28:0/840269231 shutdown_connections 2024-04-25T01:23:21.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.491+0000 7f0762f9c700 1 --2- 172.21.15.28:0/840269231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075c1080b0 0x7f075c1084d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.491+0000 7f0762f9c700 1 -- 172.21.15.28:0/840269231 >> 172.21.15.28:0/840269231 conn(0x7f075c101830 msgr2=0x7f075c103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:21.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.491+0000 7f0762f9c700 1 -- 172.21.15.28:0/840269231 shutdown_connections 2024-04-25T01:23:21.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.491+0000 7f0762f9c700 1 -- 172.21.15.28:0/840269231 wait complete. 2024-04-25T01:23:21.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.492+0000 7f0762f9c700 1 Processor -- start 2024-04-25T01:23:21.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.492+0000 7f0762f9c700 1 -- start start 2024-04-25T01:23:21.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.492+0000 7f0762f9c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075c1080b0 0x7f075c191c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.492+0000 7f0762f9c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f075c10b080 con 0x7f075c1080b0 2024-04-25T01:23:21.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.492+0000 7f0760d38700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075c1080b0 0x7f075c191c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:21.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.492+0000 7f0760d38700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075c1080b0 0x7f075c191c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36554/0 (socket says 172.21.15.28:36554) 2024-04-25T01:23:21.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.492+0000 7f0760d38700 1 -- 172.21.15.28:0/2667821502 learned_addr learned my addr 172.21.15.28:0/2667821502 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:21.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.492+0000 7f0760d38700 1 -- 172.21.15.28:0/2667821502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f075c1921c0 con 0x7f075c1080b0 2024-04-25T01:23:21.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.492+0000 7f0760d38700 1 --2- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075c1080b0 0x7f075c191c80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f07500117e0 tx=0x7f0750011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:21.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.492+0000 7f0759ffb700 1 -- 172.21.15.28:0/2667821502 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0750003bf0 con 0x7f075c1080b0 2024-04-25T01:23:21.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.493+0000 7f0762f9c700 1 -- 172.21.15.28:0/2667821502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f075c1903b0 con 0x7f075c1080b0 2024-04-25T01:23:21.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.493+0000 7f0759ffb700 1 -- 172.21.15.28:0/2667821502 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f075001a910 con 0x7f075c1080b0 2024-04-25T01:23:21.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.493+0000 7f0759ffb700 1 -- 172.21.15.28:0/2667821502 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0750024970 con 0x7f075c1080b0 2024-04-25T01:23:21.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.493+0000 7f0762f9c700 1 -- 172.21.15.28:0/2667821502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f075c190870 con 0x7f075c1080b0 2024-04-25T01:23:21.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.494+0000 7f0759ffb700 1 -- 172.21.15.28:0/2667821502 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f0750022020 con 0x7f075c1080b0 2024-04-25T01:23:21.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.494+0000 7f0759ffb700 1 --2- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f074c0390a0 0x7f074c03b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.494+0000 7f0759ffb700 1 -- 172.21.15.28:0/2667821502 --> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f074c03bc70 con 0x7f074c0390a0 2024-04-25T01:23:21.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.494+0000 7f0759ffb700 1 -- 172.21.15.28:0/2667821502 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f075004d870 con 0x7f075c1080b0 2024-04-25T01:23:21.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.494+0000 7f075bfff700 1 -- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f074c0390a0 msgr2=0x7f074c03b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2407469300 2024-04-25T01:23:21.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.494+0000 7f075bfff700 1 --2- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f074c0390a0 0x7f074c03b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:23:21.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.695+0000 7f075bfff700 1 -- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f074c0390a0 msgr2=0x7f074c03b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2407469300 2024-04-25T01:23:21.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:15.695+0000 7f075bfff700 1 --2- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f074c0390a0 0x7f074c03b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:23:21.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:16.096+0000 7f075bfff700 1 -- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f074c0390a0 msgr2=0x7f074c03b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2407469300 2024-04-25T01:23:21.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:16.096+0000 7f075bfff700 1 --2- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f074c0390a0 0x7f074c03b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T01:23:21.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:16.897+0000 7f075bfff700 1 -- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f074c0390a0 msgr2=0x7f074c03b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2407469300 2024-04-25T01:23:21.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:16.897+0000 7f075bfff700 1 --2- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f074c0390a0 0x7f074c03b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T01:23:21.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:18.498+0000 7f075bfff700 1 -- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f074c0390a0 msgr2=0x7f074c03b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2407469300 2024-04-25T01:23:21.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:18.498+0000 7f075bfff700 1 --2- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f074c0390a0 0x7f074c03b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T01:23:21.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:20.245+0000 7f0759ffb700 1 -- 172.21.15.28:0/2667821502 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7f075002d3f0 con 0x7f075c1080b0 2024-04-25T01:23:21.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:20.245+0000 7f0759ffb700 1 -- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f074c0390a0 msgr2=0x7f074c03b560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:23:21.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:20.245+0000 7f0759ffb700 1 --2- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f074c0390a0 0x7f074c03b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:21.247+0000 7f0759ffb700 1 -- 172.21.15.28:0/2667821502 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f075002d770 con 0x7f075c1080b0 2024-04-25T01:23:21.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:21.247+0000 7f0759ffb700 1 --2- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f074c03cb60 0x7f074c03ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:21.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:21.247+0000 7f0759ffb700 1 -- 172.21.15.28:0/2667821502 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f074c03bc70 con 0x7f074c03cb60 2024-04-25T01:23:21.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:21.248+0000 7f075bfff700 1 --2- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f074c03cb60 0x7f074c03ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:21.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:21.248+0000 7f075bfff700 1 --2- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f074c03cb60 0x7f074c03ef50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0748003a10 tx=0x7f07480092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:21.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:21.249+0000 7f0759ffb700 1 -- 172.21.15.28:0/2667821502 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f074c03bc70 con 0x7f074c03cb60 2024-04-25T01:23:21.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:21.256+0000 7f0762f9c700 1 -- 172.21.15.28:0/2667821502 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f075c063890 con 0x7f074c03cb60 2024-04-25T01:23:21.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:21.256+0000 7f0759ffb700 1 -- 172.21.15.28:0/2667821502 <== mgr.14152 v2:172.21.15.28:6800/1149553567 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f075c063890 con 0x7f074c03cb60 2024-04-25T01:23:21.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:21.256+0000 7f0762f9c700 1 -- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f074c03cb60 msgr2=0x7f074c03ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:21.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:21.256+0000 7f0762f9c700 1 --2- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f074c03cb60 0x7f074c03ef50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0748003a10 tx=0x7f07480092b0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:21.256+0000 7f0762f9c700 1 -- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075c1080b0 msgr2=0x7f075c191c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:21.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:21.256+0000 7f0762f9c700 1 --2- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075c1080b0 0x7f075c191c80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f07500117e0 tx=0x7f0750011ba0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:21.257+0000 7f0762f9c700 1 -- 172.21.15.28:0/2667821502 shutdown_connections 2024-04-25T01:23:21.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:21.257+0000 7f0762f9c700 1 --2- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f074c03cb60 0x7f074c03ef50 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:21.257+0000 7f0762f9c700 1 --2- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f075c1080b0 0x7f075c191c80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:21.257+0000 7f0762f9c700 1 --2- 172.21.15.28:0/2667821502 >> [v2:172.21.15.28:6800/2407469300,v1:172.21.15.28:6801/2407469300] conn(0x7f074c0390a0 0x7f074c03b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:21.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:21.257+0000 7f0762f9c700 1 -- 172.21.15.28:0/2667821502 >> 172.21.15.28:0/2667821502 conn(0x7f075c101830 msgr2=0x7f075c102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:21.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:21.257+0000 7f0762f9c700 1 -- 172.21.15.28:0/2667821502 shutdown_connections 2024-04-25T01:23:21.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:21.257+0000 7f0762f9c700 1 -- 172.21.15.28:0/2667821502 wait complete. 2024-04-25T01:23:21.961 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 9 is available 2024-04-25T01:23:21.961 INFO:teuthology.orchestra.run.smithi028.stdout:Generating a dashboard self-signed certificate... 2024-04-25T01:23:22.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:22 smithi028 ceph-mon[29390]: [25/Apr/2024:01:23:21] ENGINE Serving on http://172.21.15.28:8765 2024-04-25T01:23:22.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:22 smithi028 ceph-mon[29390]: [25/Apr/2024:01:23:21] ENGINE Serving on https://172.21.15.28:7150 2024-04-25T01:23:22.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:22 smithi028 ceph-mon[29390]: [25/Apr/2024:01:23:21] ENGINE Bus STARTED 2024-04-25T01:23:22.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:22 smithi028 ceph-mon[29390]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T01:23:22.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:22 smithi028 ceph-mon[29390]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T01:23:23.344 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-25T01:23:23.344 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.444+0000 7fd2652be700 1 Processor -- start 2024-04-25T01:23:23.344 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.444+0000 7fd2652be700 1 -- start start 2024-04-25T01:23:23.344 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.445+0000 7fd2652be700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd260106490 0x7fd2601068b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:23.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.445+0000 7fd2652be700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd26007be70 con 0x7fd260106490 2024-04-25T01:23:23.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.445+0000 7fd25effd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd260106490 0x7fd2601068b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:23.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.445+0000 7fd25effd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd260106490 0x7fd2601068b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35722/0 (socket says 172.21.15.28:35722) 2024-04-25T01:23:23.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.445+0000 7fd25effd700 1 -- 172.21.15.28:0/4030732012 learned_addr learned my addr 172.21.15.28:0/4030732012 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:23.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.445+0000 7fd25effd700 1 -- 172.21.15.28:0/4030732012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd260106df0 con 0x7fd260106490 2024-04-25T01:23:23.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.445+0000 7fd25effd700 1 --2- 172.21.15.28:0/4030732012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd260106490 0x7fd2601068b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd2500098a0 tx=0x7fd250009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df7ddbaa01d39dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:23.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.446+0000 7fd25dffb700 1 -- 172.21.15.28:0/4030732012 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd250004160 con 0x7fd260106490 2024-04-25T01:23:23.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.446+0000 7fd25dffb700 1 -- 172.21.15.28:0/4030732012 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd250004300 con 0x7fd260106490 2024-04-25T01:23:23.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.446+0000 7fd25dffb700 1 -- 172.21.15.28:0/4030732012 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd250011b20 con 0x7fd260106490 2024-04-25T01:23:23.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.446+0000 7fd2652be700 1 -- 172.21.15.28:0/4030732012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd260106490 msgr2=0x7fd2601068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.446+0000 7fd2652be700 1 --2- 172.21.15.28:0/4030732012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd260106490 0x7fd2601068b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd2500098a0 tx=0x7fd250009bb0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.446+0000 7fd2652be700 1 -- 172.21.15.28:0/4030732012 shutdown_connections 2024-04-25T01:23:23.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.446+0000 7fd2652be700 1 --2- 172.21.15.28:0/4030732012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd260106490 0x7fd2601068b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.446+0000 7fd2652be700 1 -- 172.21.15.28:0/4030732012 >> 172.21.15.28:0/4030732012 conn(0x7fd260102050 msgr2=0x7fd2601044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:23.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.446+0000 7fd2652be700 1 -- 172.21.15.28:0/4030732012 shutdown_connections 2024-04-25T01:23:23.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.446+0000 7fd2652be700 1 -- 172.21.15.28:0/4030732012 wait complete. 2024-04-25T01:23:23.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.447+0000 7fd2652be700 1 Processor -- start 2024-04-25T01:23:23.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.447+0000 7fd2652be700 1 -- start start 2024-04-25T01:23:23.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.447+0000 7fd2652be700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd260106490 0x7fd26019dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:23.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.447+0000 7fd2652be700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd260107ec0 con 0x7fd260106490 2024-04-25T01:23:23.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.448+0000 7fd25effd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd260106490 0x7fd26019dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:23.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.448+0000 7fd25effd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd260106490 0x7fd26019dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35724/0 (socket says 172.21.15.28:35724) 2024-04-25T01:23:23.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.448+0000 7fd25effd700 1 -- 172.21.15.28:0/3262906802 learned_addr learned my addr 172.21.15.28:0/3262906802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:23.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.448+0000 7fd25effd700 1 -- 172.21.15.28:0/3262906802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd26019e200 con 0x7fd260106490 2024-04-25T01:23:23.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.448+0000 7fd25effd700 1 --2- 172.21.15.28:0/3262906802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd260106490 0x7fd26019dcc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd2500117e0 tx=0x7fd250011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:23.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.448+0000 7fd247fff700 1 -- 172.21.15.28:0/3262906802 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd250003bf0 con 0x7fd260106490 2024-04-25T01:23:23.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.448+0000 7fd2652be700 1 -- 172.21.15.28:0/3262906802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd26019e490 con 0x7fd260106490 2024-04-25T01:23:23.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.448+0000 7fd247fff700 1 -- 172.21.15.28:0/3262906802 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd25001a910 con 0x7fd260106490 2024-04-25T01:23:23.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.448+0000 7fd247fff700 1 -- 172.21.15.28:0/3262906802 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd250024970 con 0x7fd260106490 2024-04-25T01:23:23.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.448+0000 7fd2652be700 1 -- 172.21.15.28:0/3262906802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd26019e8f0 con 0x7fd260106490 2024-04-25T01:23:23.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.449+0000 7fd247fff700 1 -- 172.21.15.28:0/3262906802 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7fd250022020 con 0x7fd260106490 2024-04-25T01:23:23.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.449+0000 7fd2652be700 1 -- 172.21.15.28:0/3262906802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd260063890 con 0x7fd260106490 2024-04-25T01:23:23.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.449+0000 7fd247fff700 1 --2- 172.21.15.28:0/3262906802 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd248039050 0x7fd24803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:23.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.449+0000 7fd247fff700 1 -- 172.21.15.28:0/3262906802 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd25004d7e0 con 0x7fd260106490 2024-04-25T01:23:23.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.450+0000 7fd25e7fc700 1 --2- 172.21.15.28:0/3262906802 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd248039050 0x7fd24803b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:23.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.450+0000 7fd25e7fc700 1 --2- 172.21.15.28:0/3262906802 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd248039050 0x7fd24803b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd254006fd0 tx=0x7fd254006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:23.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.455+0000 7fd247fff700 1 -- 172.21.15.28:0/3262906802 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd250011e20 con 0x7fd260106490 2024-04-25T01:23:23.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.668+0000 7fd2652be700 1 -- 172.21.15.28:0/3262906802 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fd260062670 con 0x7fd248039050 2024-04-25T01:23:23.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.817+0000 7fd247fff700 1 -- 172.21.15.28:0/3262906802 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fd260062670 con 0x7fd248039050 2024-04-25T01:23:23.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.819+0000 7fd2652be700 1 -- 172.21.15.28:0/3262906802 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd248039050 msgr2=0x7fd24803b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.819+0000 7fd2652be700 1 --2- 172.21.15.28:0/3262906802 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd248039050 0x7fd24803b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd254006fd0 tx=0x7fd254006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:23.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.819+0000 7fd2652be700 1 -- 172.21.15.28:0/3262906802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd260106490 msgr2=0x7fd26019dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:23.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.819+0000 7fd2652be700 1 --2- 172.21.15.28:0/3262906802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd260106490 0x7fd26019dcc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd2500117e0 tx=0x7fd250011c10 comp rx=0 tx=0).stop 2024-04-25T01:23:23.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.819+0000 7fd2652be700 1 -- 172.21.15.28:0/3262906802 shutdown_connections 2024-04-25T01:23:23.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.819+0000 7fd2652be700 1 --2- 172.21.15.28:0/3262906802 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd248039050 0x7fd24803b510 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.819+0000 7fd2652be700 1 --2- 172.21.15.28:0/3262906802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd260106490 0x7fd26019dcc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:23.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.819+0000 7fd2652be700 1 -- 172.21.15.28:0/3262906802 >> 172.21.15.28:0/3262906802 conn(0x7fd260102050 msgr2=0x7fd2601949a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:23.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.819+0000 7fd2652be700 1 -- 172.21.15.28:0/3262906802 shutdown_connections 2024-04-25T01:23:23.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:22.819+0000 7fd2652be700 1 -- 172.21.15.28:0/3262906802 wait complete. 2024-04-25T01:23:23.348 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial admin user... 2024-04-25T01:23:23.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:23 smithi028 ceph-mon[29390]: mgrmap e12: x(active, since 2s) 2024-04-25T01:23:23.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:23 smithi028 ceph-mon[29390]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:23.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:23.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:23.453 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:24.775 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$T6GWY0F.MXTgs.v9dnxS7umdiFvp/Hy/yRHFt6I3iKWRrZyqyqZfC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714008204, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-25T01:23:24.775 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.766+0000 7f38c78a1700 1 Processor -- start 2024-04-25T01:23:24.775 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.767+0000 7f38c78a1700 1 -- start start 2024-04-25T01:23:24.775 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.767+0000 7f38c78a1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c0108620 0x7f38c010aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.775 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.767+0000 7f38c78a1700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f38c007bed0 con 0x7f38c0108620 2024-04-25T01:23:24.775 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.767+0000 7f38c563d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c0108620 0x7f38c010aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.775 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.767+0000 7f38c563d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c0108620 0x7f38c010aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35732/0 (socket says 172.21.15.28:35732) 2024-04-25T01:23:24.775 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.767+0000 7f38c563d700 1 -- 172.21.15.28:0/1154351469 learned_addr learned my addr 172.21.15.28:0/1154351469 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:24.775 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.767+0000 7f38c563d700 1 -- 172.21.15.28:0/1154351469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c010af50 con 0x7f38c0108620 2024-04-25T01:23:24.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.767+0000 7f38c563d700 1 --2- 172.21.15.28:0/1154351469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c0108620 0x7f38c010aa10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f38bc0098a0 tx=0x7f38bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe212b1eeeab2ac5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:24.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.767+0000 7f38b7fff700 1 -- 172.21.15.28:0/1154351469 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f38bc004160 con 0x7f38c0108620 2024-04-25T01:23:24.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.768+0000 7f38b7fff700 1 -- 172.21.15.28:0/1154351469 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38bc004300 con 0x7f38c0108620 2024-04-25T01:23:24.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.768+0000 7f38b7fff700 1 -- 172.21.15.28:0/1154351469 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f38bc011b20 con 0x7f38c0108620 2024-04-25T01:23:24.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.768+0000 7f38c78a1700 1 -- 172.21.15.28:0/1154351469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c0108620 msgr2=0x7f38c010aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.768+0000 7f38c78a1700 1 --2- 172.21.15.28:0/1154351469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c0108620 0x7f38c010aa10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f38bc0098a0 tx=0x7f38bc009bb0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.768+0000 7f38c78a1700 1 -- 172.21.15.28:0/1154351469 shutdown_connections 2024-04-25T01:23:24.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.768+0000 7f38c78a1700 1 --2- 172.21.15.28:0/1154351469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c0108620 0x7f38c010aa10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.768+0000 7f38c78a1700 1 -- 172.21.15.28:0/1154351469 >> 172.21.15.28:0/1154351469 conn(0x7f38c0102070 msgr2=0x7f38c0104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:24.776 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.768+0000 7f38c78a1700 1 -- 172.21.15.28:0/1154351469 shutdown_connections 2024-04-25T01:23:24.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.768+0000 7f38c78a1700 1 -- 172.21.15.28:0/1154351469 wait complete. 2024-04-25T01:23:24.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.769+0000 7f38c78a1700 1 Processor -- start 2024-04-25T01:23:24.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.769+0000 7f38c78a1700 1 -- start start 2024-04-25T01:23:24.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.769+0000 7f38c78a1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c0108620 0x7f38c0107ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.769+0000 7f38c78a1700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f38c010bfc0 con 0x7f38c0108620 2024-04-25T01:23:24.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.769+0000 7f38c563d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c0108620 0x7f38c0107ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.769+0000 7f38c563d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c0108620 0x7f38c0107ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35738/0 (socket says 172.21.15.28:35738) 2024-04-25T01:23:24.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.769+0000 7f38c563d700 1 -- 172.21.15.28:0/3993475657 learned_addr learned my addr 172.21.15.28:0/3993475657 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:24.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.770+0000 7f38c563d700 1 -- 172.21.15.28:0/3993475657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c01083e0 con 0x7f38c0108620 2024-04-25T01:23:24.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.770+0000 7f38c563d700 1 --2- 172.21.15.28:0/3993475657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c0108620 0x7f38c0107ea0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f38bc0117e0 tx=0x7f38bc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:24.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.770+0000 7f38b67fc700 1 -- 172.21.15.28:0/3993475657 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f38bc003bf0 con 0x7f38c0108620 2024-04-25T01:23:24.777 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.770+0000 7f38c78a1700 1 -- 172.21.15.28:0/3993475657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38c01065d0 con 0x7f38c0108620 2024-04-25T01:23:24.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.770+0000 7f38b67fc700 1 -- 172.21.15.28:0/3993475657 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38bc01a910 con 0x7f38c0108620 2024-04-25T01:23:24.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.770+0000 7f38c78a1700 1 -- 172.21.15.28:0/3993475657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38c0106a90 con 0x7f38c0108620 2024-04-25T01:23:24.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.770+0000 7f38b67fc700 1 -- 172.21.15.28:0/3993475657 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f38bc024970 con 0x7f38c0108620 2024-04-25T01:23:24.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.771+0000 7f38b67fc700 1 -- 172.21.15.28:0/3993475657 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f38bc022020 con 0x7f38c0108620 2024-04-25T01:23:24.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.771+0000 7f38b67fc700 1 --2- 172.21.15.28:0/3993475657 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f38ac0390b0 0x7f38ac03b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:24.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.771+0000 7f38b67fc700 1 -- 172.21.15.28:0/3993475657 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f38bc04d580 con 0x7f38c0108620 2024-04-25T01:23:24.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.771+0000 7f38c78a1700 1 -- 172.21.15.28:0/3993475657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38c0063890 con 0x7f38c0108620 2024-04-25T01:23:24.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.771+0000 7f38c4e3c700 1 --2- 172.21.15.28:0/3993475657 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f38ac0390b0 0x7f38ac03b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:24.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.772+0000 7f38c4e3c700 1 --2- 172.21.15.28:0/3993475657 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f38ac0390b0 0x7f38ac03b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f38b0006fd0 tx=0x7f38b0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:24.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:23.781+0000 7f38b67fc700 1 -- 172.21.15.28:0/3993475657 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f38bc011e20 con 0x7f38c0108620 2024-04-25T01:23:24.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:24.009+0000 7f38c78a1700 1 -- 172.21.15.28:0/3993475657 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f38c0106d10 con 0x7f38ac0390b0 2024-04-25T01:23:24.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:24.255+0000 7f38b67fc700 1 -- 172.21.15.28:0/3993475657 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f38c0106d10 con 0x7f38ac0390b0 2024-04-25T01:23:24.778 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:24.257+0000 7f38c78a1700 1 -- 172.21.15.28:0/3993475657 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f38ac0390b0 msgr2=0x7f38ac03b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:24.257+0000 7f38c78a1700 1 --2- 172.21.15.28:0/3993475657 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f38ac0390b0 0x7f38ac03b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f38b0006fd0 tx=0x7f38b0006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:24.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:24.257+0000 7f38c78a1700 1 -- 172.21.15.28:0/3993475657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c0108620 msgr2=0x7f38c0107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:24.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:24.257+0000 7f38c78a1700 1 --2- 172.21.15.28:0/3993475657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c0108620 0x7f38c0107ea0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f38bc0117e0 tx=0x7f38bc011c10 comp rx=0 tx=0).stop 2024-04-25T01:23:24.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:24.257+0000 7f38c78a1700 1 -- 172.21.15.28:0/3993475657 shutdown_connections 2024-04-25T01:23:24.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:24.257+0000 7f38c78a1700 1 --2- 172.21.15.28:0/3993475657 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f38ac0390b0 0x7f38ac03b570 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:24.257+0000 7f38c78a1700 1 --2- 172.21.15.28:0/3993475657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38c0108620 0x7f38c0107ea0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:24.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:24.257+0000 7f38c78a1700 1 -- 172.21.15.28:0/3993475657 >> 172.21.15.28:0/3993475657 conn(0x7f38c0102070 msgr2=0x7f38c0102af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:24.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:24.258+0000 7f38c78a1700 1 -- 172.21.15.28:0/3993475657 shutdown_connections 2024-04-25T01:23:24.779 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:24.258+0000 7f38c78a1700 1 -- 172.21.15.28:0/3993475657 wait complete. 2024-04-25T01:23:24.779 INFO:teuthology.orchestra.run.smithi028.stdout:Fetching dashboard port number... 2024-04-25T01:23:25.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:25 smithi028 ceph-mon[29390]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:25.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:25 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:25.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:25 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:26.024 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 8443 2024-04-25T01:23:26.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.185+0000 7faa885d8700 1 Processor -- start 2024-04-25T01:23:26.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.185+0000 7faa885d8700 1 -- start start 2024-04-25T01:23:26.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.186+0000 7faa885d8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa80109e10 0x7faa8010a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.186+0000 7faa885d8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa8010a800 con 0x7faa80109e10 2024-04-25T01:23:26.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.186+0000 7faa86374700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa80109e10 0x7faa8010a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.186+0000 7faa86374700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa80109e10 0x7faa8010a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35742/0 (socket says 172.21.15.28:35742) 2024-04-25T01:23:26.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.186+0000 7faa86374700 1 -- 172.21.15.28:0/297466869 learned_addr learned my addr 172.21.15.28:0/297466869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:26.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.186+0000 7faa86374700 1 -- 172.21.15.28:0/297466869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8010afc0 con 0x7faa80109e10 2024-04-25T01:23:26.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.186+0000 7faa86374700 1 --2- 172.21.15.28:0/297466869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa80109e10 0x7faa8010a230 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0098a0 tx=0x7faa7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f3a2751a9128bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:26.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.186+0000 7faa85372700 1 -- 172.21.15.28:0/297466869 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa7c004160 con 0x7faa80109e10 2024-04-25T01:23:26.025 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.186+0000 7faa85372700 1 -- 172.21.15.28:0/297466869 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa7c004300 con 0x7faa80109e10 2024-04-25T01:23:26.026 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.186+0000 7faa85372700 1 -- 172.21.15.28:0/297466869 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa7c011b20 con 0x7faa80109e10 2024-04-25T01:23:26.026 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.187+0000 7faa885d8700 1 -- 172.21.15.28:0/297466869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa80109e10 msgr2=0x7faa8010a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.026 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.187+0000 7faa885d8700 1 --2- 172.21.15.28:0/297466869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa80109e10 0x7faa8010a230 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0098a0 tx=0x7faa7c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.026 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.187+0000 7faa885d8700 1 -- 172.21.15.28:0/297466869 shutdown_connections 2024-04-25T01:23:26.026 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.187+0000 7faa885d8700 1 --2- 172.21.15.28:0/297466869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa80109e10 0x7faa8010a230 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.026 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.187+0000 7faa885d8700 1 -- 172.21.15.28:0/297466869 >> 172.21.15.28:0/297466869 conn(0x7faa8007d280 msgr2=0x7faa8007d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:26.026 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.187+0000 7faa885d8700 1 -- 172.21.15.28:0/297466869 shutdown_connections 2024-04-25T01:23:26.026 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.187+0000 7faa885d8700 1 -- 172.21.15.28:0/297466869 wait complete. 2024-04-25T01:23:26.026 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.188+0000 7faa885d8700 1 Processor -- start 2024-04-25T01:23:26.026 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.188+0000 7faa885d8700 1 -- start start 2024-04-25T01:23:26.026 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.188+0000 7faa885d8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa80109e10 0x7faa8019dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.026 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.188+0000 7faa885d8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa8010e370 con 0x7faa80109e10 2024-04-25T01:23:26.026 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.188+0000 7faa86374700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa80109e10 0x7faa8019dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.026 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.188+0000 7faa86374700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa80109e10 0x7faa8019dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35756/0 (socket says 172.21.15.28:35756) 2024-04-25T01:23:26.027 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.189+0000 7faa86374700 1 -- 172.21.15.28:0/1272695117 learned_addr learned my addr 172.21.15.28:0/1272695117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:26.027 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.189+0000 7faa86374700 1 -- 172.21.15.28:0/1272695117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8019e230 con 0x7faa80109e10 2024-04-25T01:23:26.027 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.189+0000 7faa86374700 1 --2- 172.21.15.28:0/1272695117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa80109e10 0x7faa8019dcf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0117e0 tx=0x7faa7c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:26.027 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.189+0000 7faa777fe700 1 -- 172.21.15.28:0/1272695117 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa7c003bf0 con 0x7faa80109e10 2024-04-25T01:23:26.027 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.189+0000 7faa777fe700 1 -- 172.21.15.28:0/1272695117 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa7c01a910 con 0x7faa80109e10 2024-04-25T01:23:26.027 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.189+0000 7faa885d8700 1 -- 172.21.15.28:0/1272695117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa8019e4c0 con 0x7faa80109e10 2024-04-25T01:23:26.027 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.189+0000 7faa777fe700 1 -- 172.21.15.28:0/1272695117 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa7c024970 con 0x7faa80109e10 2024-04-25T01:23:26.027 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.189+0000 7faa885d8700 1 -- 172.21.15.28:0/1272695117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa8019e920 con 0x7faa80109e10 2024-04-25T01:23:26.027 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.190+0000 7faa777fe700 1 -- 172.21.15.28:0/1272695117 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7faa7c022020 con 0x7faa80109e10 2024-04-25T01:23:26.027 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.191+0000 7faa885d8700 1 -- 172.21.15.28:0/1272695117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa80063890 con 0x7faa80109e10 2024-04-25T01:23:26.027 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.191+0000 7faa777fe700 1 --2- 172.21.15.28:0/1272695117 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7faa6c0390a0 0x7faa6c03b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:26.027 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.191+0000 7faa777fe700 1 -- 172.21.15.28:0/1272695117 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7faa7c04daa0 con 0x7faa80109e10 2024-04-25T01:23:26.027 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.191+0000 7faa85b73700 1 --2- 172.21.15.28:0/1272695117 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7faa6c0390a0 0x7faa6c03b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:26.027 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.191+0000 7faa85b73700 1 --2- 172.21.15.28:0/1272695117 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7faa6c0390a0 0x7faa6c03b560 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7faa70006fd0 tx=0x7faa70006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:26.028 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.197+0000 7faa777fe700 1 -- 172.21.15.28:0/1272695117 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faa7c02dce0 con 0x7faa80109e10 2024-04-25T01:23:26.028 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.410+0000 7faa885d8700 1 -- 172.21.15.28:0/1272695117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7faa801a1440 con 0x7faa80109e10 2024-04-25T01:23:26.028 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.411+0000 7faa777fe700 1 -- 172.21.15.28:0/1272695117 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7faa7c018350 con 0x7faa80109e10 2024-04-25T01:23:26.028 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.413+0000 7faa885d8700 1 -- 172.21.15.28:0/1272695117 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7faa6c0390a0 msgr2=0x7faa6c03b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.028 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.413+0000 7faa885d8700 1 --2- 172.21.15.28:0/1272695117 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7faa6c0390a0 0x7faa6c03b560 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7faa70006fd0 tx=0x7faa70006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:26.028 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.413+0000 7faa885d8700 1 -- 172.21.15.28:0/1272695117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa80109e10 msgr2=0x7faa8019dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:26.028 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.413+0000 7faa885d8700 1 --2- 172.21.15.28:0/1272695117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa80109e10 0x7faa8019dcf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0117e0 tx=0x7faa7c011ba0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.028 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.413+0000 7faa885d8700 1 -- 172.21.15.28:0/1272695117 shutdown_connections 2024-04-25T01:23:26.028 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.413+0000 7faa885d8700 1 --2- 172.21.15.28:0/1272695117 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7faa6c0390a0 0x7faa6c03b560 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.028 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.413+0000 7faa885d8700 1 --2- 172.21.15.28:0/1272695117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa80109e10 0x7faa8019dcf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:26.028 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.413+0000 7faa885d8700 1 -- 172.21.15.28:0/1272695117 >> 172.21.15.28:0/1272695117 conn(0x7faa8007d280 msgr2=0x7faa80107480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:26.028 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.413+0000 7faa885d8700 1 -- 172.21.15.28:0/1272695117 shutdown_connections 2024-04-25T01:23:26.028 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:25.413+0000 7faa885d8700 1 -- 172.21.15.28:0/1272695117 wait complete. 2024-04-25T01:23:26.033 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:23:26.033 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-25T01:23:26.041 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:23:26.041 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-25T01:23:26.041 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-25T01:23:26.041 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-25T01:23:26.043 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph Dashboard is now available at: 2024-04-25T01:23:26.043 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:23:26.043 INFO:teuthology.orchestra.run.smithi028.stdout: URL: https://smithi028:8443/ 2024-04-25T01:23:26.043 INFO:teuthology.orchestra.run.smithi028.stdout: User: admin 2024-04-25T01:23:26.043 INFO:teuthology.orchestra.run.smithi028.stdout: Password: dulrk1au6j 2024-04-25T01:23:26.043 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:23:26.043 INFO:teuthology.orchestra.run.smithi028.stdout:Saving cluster configuration to /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config directory 2024-04-25T01:23:26.044 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling autotune for osd_memory_target 2024-04-25T01:23:26.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:26 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1272695117' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-25T01:23:27.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.480+0000 7f6d99037700 1 Processor -- start 2024-04-25T01:23:27.876 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.480+0000 7f6d99037700 1 -- start start 2024-04-25T01:23:27.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.480+0000 7f6d99037700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94107bf0 0x7f6d94108010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:27.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.480+0000 7f6d99037700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d941085e0 con 0x7f6d94107bf0 2024-04-25T01:23:27.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.480+0000 7f6d92d9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94107bf0 0x7f6d94108010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:27.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.480+0000 7f6d92d9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94107bf0 0x7f6d94108010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35770/0 (socket says 172.21.15.28:35770) 2024-04-25T01:23:27.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.480+0000 7f6d92d9d700 1 -- 172.21.15.28:0/553049812 learned_addr learned my addr 172.21.15.28:0/553049812 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:27.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.480+0000 7f6d92d9d700 1 -- 172.21.15.28:0/553049812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d94108da0 con 0x7f6d94107bf0 2024-04-25T01:23:27.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.481+0000 7f6d92d9d700 1 --2- 172.21.15.28:0/553049812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94107bf0 0x7f6d94108010 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6d840098a0 tx=0x7f6d84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d1ae0e7ab11718 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:27.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.481+0000 7f6d91d9b700 1 -- 172.21.15.28:0/553049812 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d84004160 con 0x7f6d94107bf0 2024-04-25T01:23:27.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.481+0000 7f6d91d9b700 1 -- 172.21.15.28:0/553049812 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d84004300 con 0x7f6d94107bf0 2024-04-25T01:23:27.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.481+0000 7f6d91d9b700 1 -- 172.21.15.28:0/553049812 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d84011b20 con 0x7f6d94107bf0 2024-04-25T01:23:27.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.481+0000 7f6d99037700 1 -- 172.21.15.28:0/553049812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94107bf0 msgr2=0x7f6d94108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:27.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.481+0000 7f6d99037700 1 --2- 172.21.15.28:0/553049812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94107bf0 0x7f6d94108010 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6d840098a0 tx=0x7f6d84009bb0 comp rx=0 tx=0).stop 2024-04-25T01:23:27.877 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.481+0000 7f6d99037700 1 -- 172.21.15.28:0/553049812 shutdown_connections 2024-04-25T01:23:27.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.481+0000 7f6d99037700 1 --2- 172.21.15.28:0/553049812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94107bf0 0x7f6d94108010 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:27.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.481+0000 7f6d99037700 1 -- 172.21.15.28:0/553049812 >> 172.21.15.28:0/553049812 conn(0x7f6d94103170 msgr2=0x7f6d941055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:27.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.481+0000 7f6d99037700 1 -- 172.21.15.28:0/553049812 shutdown_connections 2024-04-25T01:23:27.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.481+0000 7f6d99037700 1 -- 172.21.15.28:0/553049812 wait complete. 2024-04-25T01:23:27.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.482+0000 7f6d99037700 1 Processor -- start 2024-04-25T01:23:27.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.482+0000 7f6d99037700 1 -- start start 2024-04-25T01:23:27.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.482+0000 7f6d99037700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94107bf0 0x7f6d9419bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:27.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.482+0000 7f6d99037700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9410c150 con 0x7f6d94107bf0 2024-04-25T01:23:27.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.483+0000 7f6d92d9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94107bf0 0x7f6d9419bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:27.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.483+0000 7f6d92d9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94107bf0 0x7f6d9419bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35774/0 (socket says 172.21.15.28:35774) 2024-04-25T01:23:27.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.483+0000 7f6d92d9d700 1 -- 172.21.15.28:0/1830795581 learned_addr learned my addr 172.21.15.28:0/1830795581 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:27.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.483+0000 7f6d92d9d700 1 -- 172.21.15.28:0/1830795581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d9419c050 con 0x7f6d94107bf0 2024-04-25T01:23:27.878 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.483+0000 7f6d92d9d700 1 --2- 172.21.15.28:0/1830795581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94107bf0 0x7f6d9419bb10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6d840117e0 tx=0x7f6d84011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:27.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.483+0000 7f6d7bfff700 1 -- 172.21.15.28:0/1830795581 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d84003bf0 con 0x7f6d94107bf0 2024-04-25T01:23:27.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.483+0000 7f6d7bfff700 1 -- 172.21.15.28:0/1830795581 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d8401a910 con 0x7f6d94107bf0 2024-04-25T01:23:27.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.483+0000 7f6d99037700 1 -- 172.21.15.28:0/1830795581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d9419c2e0 con 0x7f6d94107bf0 2024-04-25T01:23:27.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.483+0000 7f6d7bfff700 1 -- 172.21.15.28:0/1830795581 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d84024970 con 0x7f6d94107bf0 2024-04-25T01:23:27.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.483+0000 7f6d99037700 1 -- 172.21.15.28:0/1830795581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d9419c740 con 0x7f6d94107bf0 2024-04-25T01:23:27.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.484+0000 7f6d7bfff700 1 -- 172.21.15.28:0/1830795581 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f6d84022020 con 0x7f6d94107bf0 2024-04-25T01:23:27.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.484+0000 7f6d99037700 1 -- 172.21.15.28:0/1830795581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d94063890 con 0x7f6d94107bf0 2024-04-25T01:23:27.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.484+0000 7f6d7bfff700 1 --2- 172.21.15.28:0/1830795581 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6d7c039100 0x7f6d7c03b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:27.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.484+0000 7f6d7bfff700 1 -- 172.21.15.28:0/1830795581 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6d8404d5c0 con 0x7f6d94107bf0 2024-04-25T01:23:27.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.485+0000 7f6d9259c700 1 --2- 172.21.15.28:0/1830795581 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6d7c039100 0x7f6d7c03b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:27.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.485+0000 7f6d9259c700 1 --2- 172.21.15.28:0/1830795581 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6d7c039100 0x7f6d7c03b5c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6d88006fd0 tx=0x7f6d88006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:27.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.490+0000 7f6d7bfff700 1 -- 172.21.15.28:0/1830795581 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6d84020c10 con 0x7f6d94107bf0 2024-04-25T01:23:27.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.701+0000 7f6d99037700 1 -- 172.21.15.28:0/1830795581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f6d9419f110 con 0x7f6d94107bf0 2024-04-25T01:23:27.879 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.702+0000 7f6d7bfff700 1 -- 172.21.15.28:0/1830795581 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f6d8401abe0 con 0x7f6d94107bf0 2024-04-25T01:23:27.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.704+0000 7f6d99037700 1 -- 172.21.15.28:0/1830795581 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6d7c039100 msgr2=0x7f6d7c03b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:27.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.704+0000 7f6d99037700 1 --2- 172.21.15.28:0/1830795581 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6d7c039100 0x7f6d7c03b5c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6d88006fd0 tx=0x7f6d88006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:27.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.704+0000 7f6d99037700 1 -- 172.21.15.28:0/1830795581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94107bf0 msgr2=0x7f6d9419bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:27.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.704+0000 7f6d99037700 1 --2- 172.21.15.28:0/1830795581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94107bf0 0x7f6d9419bb10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6d840117e0 tx=0x7f6d84011ba0 comp rx=0 tx=0).stop 2024-04-25T01:23:27.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.704+0000 7f6d99037700 1 -- 172.21.15.28:0/1830795581 shutdown_connections 2024-04-25T01:23:27.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.704+0000 7f6d99037700 1 --2- 172.21.15.28:0/1830795581 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6d7c039100 0x7f6d7c03b5c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:27.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.704+0000 7f6d99037700 1 --2- 172.21.15.28:0/1830795581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94107bf0 0x7f6d9419bb10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:27.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.704+0000 7f6d99037700 1 -- 172.21.15.28:0/1830795581 >> 172.21.15.28:0/1830795581 conn(0x7f6d94103170 msgr2=0x7f6d94103c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:27.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.704+0000 7f6d99037700 1 -- 172.21.15.28:0/1830795581 shutdown_connections 2024-04-25T01:23:27.880 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:26.704+0000 7f6d99037700 1 -- 172.21.15.28:0/1830795581 wait complete. 2024-04-25T01:23:28.119 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:28 smithi028 ceph-mon[29390]: mgrmap e13: x(active, since 6s) 2024-04-25T01:23:29.206 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.334+0000 7efdcc771700 1 Processor -- start 2024-04-25T01:23:29.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.334+0000 7efdcc771700 1 -- start start 2024-04-25T01:23:29.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.334+0000 7efdcc771700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdc4107bf0 0x7efdc4108010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.334+0000 7efdcc771700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efdc41085e0 con 0x7efdc4107bf0 2024-04-25T01:23:29.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.334+0000 7efdca50d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdc4107bf0 0x7efdc4108010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.334+0000 7efdca50d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdc4107bf0 0x7efdc4108010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36852/0 (socket says 172.21.15.28:36852) 2024-04-25T01:23:29.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.334+0000 7efdca50d700 1 -- 172.21.15.28:0/1508526550 learned_addr learned my addr 172.21.15.28:0/1508526550 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:29.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.335+0000 7efdca50d700 1 -- 172.21.15.28:0/1508526550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc4108da0 con 0x7efdc4107bf0 2024-04-25T01:23:29.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.335+0000 7efdca50d700 1 --2- 172.21.15.28:0/1508526550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdc4107bf0 0x7efdc4108010 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efdc00098a0 tx=0x7efdc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=525fc8ffd38c7518 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:29.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.335+0000 7efdc950b700 1 -- 172.21.15.28:0/1508526550 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdc0004160 con 0x7efdc4107bf0 2024-04-25T01:23:29.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.335+0000 7efdc950b700 1 -- 172.21.15.28:0/1508526550 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdc0004300 con 0x7efdc4107bf0 2024-04-25T01:23:29.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.335+0000 7efdc950b700 1 -- 172.21.15.28:0/1508526550 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdc0011b20 con 0x7efdc4107bf0 2024-04-25T01:23:29.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.336+0000 7efdcc771700 1 -- 172.21.15.28:0/1508526550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdc4107bf0 msgr2=0x7efdc4108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.207 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.336+0000 7efdcc771700 1 --2- 172.21.15.28:0/1508526550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdc4107bf0 0x7efdc4108010 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efdc00098a0 tx=0x7efdc0009bb0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.336+0000 7efdcc771700 1 -- 172.21.15.28:0/1508526550 shutdown_connections 2024-04-25T01:23:29.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.336+0000 7efdcc771700 1 --2- 172.21.15.28:0/1508526550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdc4107bf0 0x7efdc4108010 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.336+0000 7efdcc771700 1 -- 172.21.15.28:0/1508526550 >> 172.21.15.28:0/1508526550 conn(0x7efdc4103170 msgr2=0x7efdc41055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:29.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.336+0000 7efdcc771700 1 -- 172.21.15.28:0/1508526550 shutdown_connections 2024-04-25T01:23:29.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.336+0000 7efdcc771700 1 -- 172.21.15.28:0/1508526550 wait complete. 2024-04-25T01:23:29.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.336+0000 7efdcc771700 1 Processor -- start 2024-04-25T01:23:29.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.336+0000 7efdcc771700 1 -- start start 2024-04-25T01:23:29.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.337+0000 7efdcc771700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdc4107bf0 0x7efdc419bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.337+0000 7efdcc771700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efdc410c150 con 0x7efdc4107bf0 2024-04-25T01:23:29.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.337+0000 7efdca50d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdc4107bf0 0x7efdc419bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.337+0000 7efdca50d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdc4107bf0 0x7efdc419bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36864/0 (socket says 172.21.15.28:36864) 2024-04-25T01:23:29.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.337+0000 7efdca50d700 1 -- 172.21.15.28:0/636640175 learned_addr learned my addr 172.21.15.28:0/636640175 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:29.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.337+0000 7efdca50d700 1 -- 172.21.15.28:0/636640175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc419c040 con 0x7efdc4107bf0 2024-04-25T01:23:29.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.337+0000 7efdca50d700 1 --2- 172.21.15.28:0/636640175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdc4107bf0 0x7efdc419bb00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efdc00117e0 tx=0x7efdc0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:29.208 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.337+0000 7efdbb7fe700 1 -- 172.21.15.28:0/636640175 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdc0003bf0 con 0x7efdc4107bf0 2024-04-25T01:23:29.209 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.337+0000 7efdbb7fe700 1 -- 172.21.15.28:0/636640175 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdc001a910 con 0x7efdc4107bf0 2024-04-25T01:23:29.209 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.337+0000 7efdcc771700 1 -- 172.21.15.28:0/636640175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdc419c2d0 con 0x7efdc4107bf0 2024-04-25T01:23:29.209 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.337+0000 7efdbb7fe700 1 -- 172.21.15.28:0/636640175 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdc0024970 con 0x7efdc4107bf0 2024-04-25T01:23:29.209 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.338+0000 7efdcc771700 1 -- 172.21.15.28:0/636640175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdc419c730 con 0x7efdc4107bf0 2024-04-25T01:23:29.209 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.338+0000 7efdbb7fe700 1 -- 172.21.15.28:0/636640175 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7efdc0022020 con 0x7efdc4107bf0 2024-04-25T01:23:29.209 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.339+0000 7efdbb7fe700 1 --2- 172.21.15.28:0/636640175 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7efdb0039140 0x7efdb003b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:29.209 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.339+0000 7efdcc771700 1 -- 172.21.15.28:0/636640175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdc4063890 con 0x7efdc4107bf0 2024-04-25T01:23:29.209 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.339+0000 7efdbb7fe700 1 -- 172.21.15.28:0/636640175 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7efdc004da00 con 0x7efdc4107bf0 2024-04-25T01:23:29.209 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.339+0000 7efdc9d0c700 1 --2- 172.21.15.28:0/636640175 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7efdb0039140 0x7efdb003b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:29.209 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.339+0000 7efdc9d0c700 1 --2- 172.21.15.28:0/636640175 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7efdb0039140 0x7efdb003b600 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efdb4006fd0 tx=0x7efdb4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:29.209 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.344+0000 7efdbb7fe700 1 -- 172.21.15.28:0/636640175 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efdc0010380 con 0x7efdc4107bf0 2024-04-25T01:23:29.209 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.664+0000 7efdcc771700 1 -- 172.21.15.28:0/636640175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7efdc419f100 con 0x7efdc4107bf0 2024-04-25T01:23:29.209 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.666+0000 7efdbb7fe700 1 -- 172.21.15.28:0/636640175 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7efdc0032030 con 0x7efdc4107bf0 2024-04-25T01:23:29.209 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.668+0000 7efdcc771700 1 -- 172.21.15.28:0/636640175 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7efdb0039140 msgr2=0x7efdb003b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.210 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.668+0000 7efdcc771700 1 --2- 172.21.15.28:0/636640175 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7efdb0039140 0x7efdb003b600 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efdb4006fd0 tx=0x7efdb4006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:29.210 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.668+0000 7efdcc771700 1 -- 172.21.15.28:0/636640175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdc4107bf0 msgr2=0x7efdc419bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:29.210 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.668+0000 7efdcc771700 1 --2- 172.21.15.28:0/636640175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdc4107bf0 0x7efdc419bb00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efdc00117e0 tx=0x7efdc0011ba0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.210 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.668+0000 7efdcc771700 1 -- 172.21.15.28:0/636640175 shutdown_connections 2024-04-25T01:23:29.210 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.668+0000 7efdcc771700 1 --2- 172.21.15.28:0/636640175 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7efdb0039140 0x7efdb003b600 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.210 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.668+0000 7efdcc771700 1 --2- 172.21.15.28:0/636640175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdc4107bf0 0x7efdc419bb00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:29.210 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.668+0000 7efdcc771700 1 -- 172.21.15.28:0/636640175 >> 172.21.15.28:0/636640175 conn(0x7efdc4103170 msgr2=0x7efdc4103c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:29.210 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.668+0000 7efdcc771700 1 -- 172.21.15.28:0/636640175 shutdown_connections 2024-04-25T01:23:29.210 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-25T01:23:28.668+0000 7efdcc771700 1 -- 172.21.15.28:0/636640175 wait complete. 2024-04-25T01:23:29.210 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-25T01:23:29.210 INFO:teuthology.orchestra.run.smithi028.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-25T01:23:29.210 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:23:29.210 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:23:29.210 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:23:29.210 INFO:teuthology.orchestra.run.smithi028.stdout:Or, if you are only running a single cluster on this host: 2024-04-25T01:23:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:23:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-25T01:23:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:23:29.211 INFO:teuthology.orchestra.run.smithi028.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-25T01:23:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:23:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: ceph telemetry on 2024-04-25T01:23:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:23:29.211 INFO:teuthology.orchestra.run.smithi028.stdout:For more information see: 2024-04-25T01:23:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:23:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-25T01:23:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:23:29.211 INFO:teuthology.orchestra.run.smithi028.stdout:Bootstrap complete. 2024-04-25T01:23:29.247 INFO:tasks.cephadm:Fetching config... 2024-04-25T01:23:29.247 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:23:29.247 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-25T01:23:29.263 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-25T01:23:29.264 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:23:29.264 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-25T01:23:29.319 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-25T01:23:29.319 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:23:29.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-25T01:23:29.386 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-25T01:23:29.387 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:23:29.387 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-25T01:23:29.442 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-25T01:23:29.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNDiUVBwg9V5g9uZkL2ovljFZpXg/5LNJQeAahyRKa/yhLU5xnPA77zjGxO90Vx5yC51qBABAchCR63eCTZntbBLakWrAmTQ+QsmgVWX6flcsNx8RJ0zw6lc2QBgx0JM+dC6ES7Wt4d9leCQKd/eXNHujSg0pgbBVS4EMkKbSL7jC2Lcvtywmb1QW/+FvtK/QrQ9fKLOP6Ago+ueDLKK7Ve20YRBozUwrnn+oorKGdL3hWutMOlqKjlz0PUfE3/q4tGydD8n5FbdylSNPB3S1qsMClL2IYGbLzlhg9FpplxT58fjyLTwUQAXkrJ3NA0SQfFcVV3bfvm9WG2ZjCF9A6X1w6WOoktnu66rkWwJ0utd6tVeOrqRb8m5dFv36e7Ykhh0ihsiAmsBO5oyDO6zYsZ0No0ZddJ9nyahva9M6DXNRvCGwmJY26v5sSc4dJB778RHXxGBtcQ3H1ABOffBmLR4Km2MAstTPntRfen1lzpl70/YyDU7G0zKgZEG8iW10= ceph-0e88f60a-02a2-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T01:23:29.529 INFO:teuthology.orchestra.run.smithi028.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNDiUVBwg9V5g9uZkL2ovljFZpXg/5LNJQeAahyRKa/yhLU5xnPA77zjGxO90Vx5yC51qBABAchCR63eCTZntbBLakWrAmTQ+QsmgVWX6flcsNx8RJ0zw6lc2QBgx0JM+dC6ES7Wt4d9leCQKd/eXNHujSg0pgbBVS4EMkKbSL7jC2Lcvtywmb1QW/+FvtK/QrQ9fKLOP6Ago+ueDLKK7Ve20YRBozUwrnn+oorKGdL3hWutMOlqKjlz0PUfE3/q4tGydD8n5FbdylSNPB3S1qsMClL2IYGbLzlhg9FpplxT58fjyLTwUQAXkrJ3NA0SQfFcVV3bfvm9WG2ZjCF9A6X1w6WOoktnu66rkWwJ0utd6tVeOrqRb8m5dFv36e7Ykhh0ihsiAmsBO5oyDO6zYsZ0No0ZddJ9nyahva9M6DXNRvCGwmJY26v5sSc4dJB778RHXxGBtcQ3H1ABOffBmLR4Km2MAstTPntRfen1lzpl70/YyDU7G0zKgZEG8iW10= ceph-0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:23:29.542 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNDiUVBwg9V5g9uZkL2ovljFZpXg/5LNJQeAahyRKa/yhLU5xnPA77zjGxO90Vx5yC51qBABAchCR63eCTZntbBLakWrAmTQ+QsmgVWX6flcsNx8RJ0zw6lc2QBgx0JM+dC6ES7Wt4d9leCQKd/eXNHujSg0pgbBVS4EMkKbSL7jC2Lcvtywmb1QW/+FvtK/QrQ9fKLOP6Ago+ueDLKK7Ve20YRBozUwrnn+oorKGdL3hWutMOlqKjlz0PUfE3/q4tGydD8n5FbdylSNPB3S1qsMClL2IYGbLzlhg9FpplxT58fjyLTwUQAXkrJ3NA0SQfFcVV3bfvm9WG2ZjCF9A6X1w6WOoktnu66rkWwJ0utd6tVeOrqRb8m5dFv36e7Ykhh0ihsiAmsBO5oyDO6zYsZ0No0ZddJ9nyahva9M6DXNRvCGwmJY26v5sSc4dJB778RHXxGBtcQ3H1ABOffBmLR4Km2MAstTPntRfen1lzpl70/YyDU7G0zKgZEG8iW10= ceph-0e88f60a-02a2-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T01:23:29.582 INFO:teuthology.orchestra.run.smithi032.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNDiUVBwg9V5g9uZkL2ovljFZpXg/5LNJQeAahyRKa/yhLU5xnPA77zjGxO90Vx5yC51qBABAchCR63eCTZntbBLakWrAmTQ+QsmgVWX6flcsNx8RJ0zw6lc2QBgx0JM+dC6ES7Wt4d9leCQKd/eXNHujSg0pgbBVS4EMkKbSL7jC2Lcvtywmb1QW/+FvtK/QrQ9fKLOP6Ago+ueDLKK7Ve20YRBozUwrnn+oorKGdL3hWutMOlqKjlz0PUfE3/q4tGydD8n5FbdylSNPB3S1qsMClL2IYGbLzlhg9FpplxT58fjyLTwUQAXkrJ3NA0SQfFcVV3bfvm9WG2ZjCF9A6X1w6WOoktnu66rkWwJ0utd6tVeOrqRb8m5dFv36e7Ykhh0ihsiAmsBO5oyDO6zYsZ0No0ZddJ9nyahva9M6DXNRvCGwmJY26v5sSc4dJB778RHXxGBtcQ3H1ABOffBmLR4Km2MAstTPntRfen1lzpl70/YyDU7G0zKgZEG8iW10= ceph-0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:23:29.595 DEBUG:teuthology.orchestra.run.smithi130:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNDiUVBwg9V5g9uZkL2ovljFZpXg/5LNJQeAahyRKa/yhLU5xnPA77zjGxO90Vx5yC51qBABAchCR63eCTZntbBLakWrAmTQ+QsmgVWX6flcsNx8RJ0zw6lc2QBgx0JM+dC6ES7Wt4d9leCQKd/eXNHujSg0pgbBVS4EMkKbSL7jC2Lcvtywmb1QW/+FvtK/QrQ9fKLOP6Ago+ueDLKK7Ve20YRBozUwrnn+oorKGdL3hWutMOlqKjlz0PUfE3/q4tGydD8n5FbdylSNPB3S1qsMClL2IYGbLzlhg9FpplxT58fjyLTwUQAXkrJ3NA0SQfFcVV3bfvm9WG2ZjCF9A6X1w6WOoktnu66rkWwJ0utd6tVeOrqRb8m5dFv36e7Ykhh0ihsiAmsBO5oyDO6zYsZ0No0ZddJ9nyahva9M6DXNRvCGwmJY26v5sSc4dJB778RHXxGBtcQ3H1ABOffBmLR4Km2MAstTPntRfen1lzpl70/YyDU7G0zKgZEG8iW10= ceph-0e88f60a-02a2-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T01:23:29.633 INFO:teuthology.orchestra.run.smithi130.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNDiUVBwg9V5g9uZkL2ovljFZpXg/5LNJQeAahyRKa/yhLU5xnPA77zjGxO90Vx5yC51qBABAchCR63eCTZntbBLakWrAmTQ+QsmgVWX6flcsNx8RJ0zw6lc2QBgx0JM+dC6ES7Wt4d9leCQKd/eXNHujSg0pgbBVS4EMkKbSL7jC2Lcvtywmb1QW/+FvtK/QrQ9fKLOP6Ago+ueDLKK7Ve20YRBozUwrnn+oorKGdL3hWutMOlqKjlz0PUfE3/q4tGydD8n5FbdylSNPB3S1qsMClL2IYGbLzlhg9FpplxT58fjyLTwUQAXkrJ3NA0SQfFcVV3bfvm9WG2ZjCF9A6X1w6WOoktnu66rkWwJ0utd6tVeOrqRb8m5dFv36e7Ykhh0ihsiAmsBO5oyDO6zYsZ0No0ZddJ9nyahva9M6DXNRvCGwmJY26v5sSc4dJB778RHXxGBtcQ3H1ABOffBmLR4Km2MAstTPntRfen1lzpl70/YyDU7G0zKgZEG8iW10= ceph-0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:23:29.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-25T01:23:29.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:29 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/636640175' entity='client.admin' 2024-04-25T01:23:29.888 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.083+0000 7f121745c700 1 -- 172.21.15.28:0/2081431255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12101033a0 msgr2=0x7f12101037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.083+0000 7f121745c700 1 --2- 172.21.15.28:0/2081431255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12101033a0 0x7f12101037c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f120c009910 tx=0x7f120c009c20 comp rx=0 tx=0).stop 2024-04-25T01:23:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.083+0000 7f121745c700 1 -- 172.21.15.28:0/2081431255 shutdown_connections 2024-04-25T01:23:32.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.083+0000 7f121745c700 1 --2- 172.21.15.28:0/2081431255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12101033a0 0x7f12101037c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.083+0000 7f121745c700 1 -- 172.21.15.28:0/2081431255 >> 172.21.15.28:0/2081431255 conn(0x7f12100fd280 msgr2=0x7f12100ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:32.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.083+0000 7f121745c700 1 -- 172.21.15.28:0/2081431255 shutdown_connections 2024-04-25T01:23:32.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.083+0000 7f121745c700 1 -- 172.21.15.28:0/2081431255 wait complete. 2024-04-25T01:23:32.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.084+0000 7f121745c700 1 Processor -- start 2024-04-25T01:23:32.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.084+0000 7f121745c700 1 -- start start 2024-04-25T01:23:32.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.084+0000 7f121745c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12101033a0 0x7f1210197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.084+0000 7f121745c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1210107e60 con 0x7f12101033a0 2024-04-25T01:23:32.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.084+0000 7f12151f8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12101033a0 0x7f1210197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.084+0000 7f12151f8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12101033a0 0x7f1210197530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36868/0 (socket says 172.21.15.28:36868) 2024-04-25T01:23:32.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.084+0000 7f12151f8700 1 -- 172.21.15.28:0/3183886873 learned_addr learned my addr 172.21.15.28:0/3183886873 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:32.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.085+0000 7f12151f8700 1 -- 172.21.15.28:0/3183886873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1210197a70 con 0x7f12101033a0 2024-04-25T01:23:32.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.085+0000 7f12151f8700 1 --2- 172.21.15.28:0/3183886873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12101033a0 0x7f1210197530 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f120c005850 tx=0x7f120c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:32.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.085+0000 7f12067fc700 1 -- 172.21.15.28:0/3183886873 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f120c026070 con 0x7f12101033a0 2024-04-25T01:23:32.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.085+0000 7f121745c700 1 -- 172.21.15.28:0/3183886873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1210197d00 con 0x7f12101033a0 2024-04-25T01:23:32.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.085+0000 7f12067fc700 1 -- 172.21.15.28:0/3183886873 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f120c0044a0 con 0x7f12101033a0 2024-04-25T01:23:32.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.085+0000 7f12067fc700 1 -- 172.21.15.28:0/3183886873 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f120c003ab0 con 0x7f12101033a0 2024-04-25T01:23:32.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.085+0000 7f121745c700 1 -- 172.21.15.28:0/3183886873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f121019a9a0 con 0x7f12101033a0 2024-04-25T01:23:32.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.086+0000 7f12067fc700 1 -- 172.21.15.28:0/3183886873 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f120c029070 con 0x7f12101033a0 2024-04-25T01:23:32.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.086+0000 7f121745c700 1 -- 172.21.15.28:0/3183886873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12101085e0 con 0x7f12101033a0 2024-04-25T01:23:32.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.086+0000 7f12067fc700 1 --2- 172.21.15.28:0/3183886873 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f11fc0390f0 0x7f11fc03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:32.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.086+0000 7f12067fc700 1 -- 172.21.15.28:0/3183886873 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f120c01e070 con 0x7f12101033a0 2024-04-25T01:23:32.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.086+0000 7f12149f7700 1 --2- 172.21.15.28:0/3183886873 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f11fc0390f0 0x7f11fc03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:32.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.087+0000 7f12149f7700 1 --2- 172.21.15.28:0/3183886873 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f11fc0390f0 0x7f11fc03b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1200006fd0 tx=0x7f1200006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.091+0000 7f12067fc700 1 -- 172.21.15.28:0/3183886873 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f120c013070 con 0x7f12101033a0 2024-04-25T01:23:32.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.298+0000 7f121745c700 1 -- 172.21.15.28:0/3183886873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f121002db30 con 0x7f12101033a0 2024-04-25T01:23:32.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.300+0000 7f12067fc700 1 -- 172.21.15.28:0/3183886873 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f120c030020 con 0x7f12101033a0 2024-04-25T01:23:32.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.302+0000 7f121745c700 1 -- 172.21.15.28:0/3183886873 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f11fc0390f0 msgr2=0x7f11fc03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.302+0000 7f121745c700 1 --2- 172.21.15.28:0/3183886873 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f11fc0390f0 0x7f11fc03b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1200006fd0 tx=0x7f1200006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:32.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.302+0000 7f121745c700 1 -- 172.21.15.28:0/3183886873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12101033a0 msgr2=0x7f1210197530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:32.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.302+0000 7f121745c700 1 --2- 172.21.15.28:0/3183886873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12101033a0 0x7f1210197530 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f120c005850 tx=0x7f120c005030 comp rx=0 tx=0).stop 2024-04-25T01:23:32.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.302+0000 7f121745c700 1 -- 172.21.15.28:0/3183886873 shutdown_connections 2024-04-25T01:23:32.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.302+0000 7f121745c700 1 --2- 172.21.15.28:0/3183886873 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f11fc0390f0 0x7f11fc03b5b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.302+0000 7f121745c700 1 --2- 172.21.15.28:0/3183886873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12101033a0 0x7f1210197530 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:32.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.302+0000 7f121745c700 1 -- 172.21.15.28:0/3183886873 >> 172.21.15.28:0/3183886873 conn(0x7f12100fd280 msgr2=0x7f12100fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:32.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.302+0000 7f121745c700 1 -- 172.21.15.28:0/3183886873 shutdown_connections 2024-04-25T01:23:32.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:32.302+0000 7f121745c700 1 -- 172.21.15.28:0/3183886873 wait complete. 2024-04-25T01:23:32.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:32 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:32.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:32 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:32.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:32 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:32.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:32 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:32.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:32 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:32.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:32 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:32.838 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-25T01:23:32.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-25T01:23:33.081 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:33.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:33 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3183886873' entity='client.admin' 2024-04-25T01:23:33.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:33.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:34.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.604+0000 7ff50b32c700 1 -- 172.21.15.28:0/1596797745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff504101cc0 msgr2=0x7ff5041020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:34.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.604+0000 7ff50b32c700 1 --2- 172.21.15.28:0/1596797745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff504101cc0 0x7ff5041020e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff500009910 tx=0x7ff500009c20 comp rx=0 tx=0).stop 2024-04-25T01:23:34.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.605+0000 7ff50b32c700 1 -- 172.21.15.28:0/1596797745 shutdown_connections 2024-04-25T01:23:34.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.605+0000 7ff50b32c700 1 --2- 172.21.15.28:0/1596797745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff504101cc0 0x7ff5041020e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:34.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.605+0000 7ff50b32c700 1 -- 172.21.15.28:0/1596797745 >> 172.21.15.28:0/1596797745 conn(0x7ff5040fd260 msgr2=0x7ff5040ff6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:34.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.605+0000 7ff50b32c700 1 -- 172.21.15.28:0/1596797745 shutdown_connections 2024-04-25T01:23:34.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.605+0000 7ff50b32c700 1 -- 172.21.15.28:0/1596797745 wait complete. 2024-04-25T01:23:34.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.605+0000 7ff50b32c700 1 Processor -- start 2024-04-25T01:23:34.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.605+0000 7ff50b32c700 1 -- start start 2024-04-25T01:23:34.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.606+0000 7ff50b32c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff504101cc0 0x7ff504075ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:34.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.606+0000 7ff50b32c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff504107d50 con 0x7ff504101cc0 2024-04-25T01:23:34.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.606+0000 7ff5090c8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff504101cc0 0x7ff504075ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:34.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.606+0000 7ff5090c8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff504101cc0 0x7ff504075ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36882/0 (socket says 172.21.15.28:36882) 2024-04-25T01:23:34.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.606+0000 7ff5090c8700 1 -- 172.21.15.28:0/2174441013 learned_addr learned my addr 172.21.15.28:0/2174441013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:34.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.606+0000 7ff5090c8700 1 -- 172.21.15.28:0/2174441013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff504076430 con 0x7ff504101cc0 2024-04-25T01:23:34.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.606+0000 7ff5090c8700 1 --2- 172.21.15.28:0/2174441013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff504101cc0 0x7ff504075ef0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff500006010 tx=0x7ff500005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:34.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.606+0000 7ff4fa7fc700 1 -- 172.21.15.28:0/2174441013 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff500027070 con 0x7ff504101cc0 2024-04-25T01:23:34.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.606+0000 7ff4fa7fc700 1 -- 172.21.15.28:0/2174441013 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff50000be40 con 0x7ff504101cc0 2024-04-25T01:23:34.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.606+0000 7ff4fa7fc700 1 -- 172.21.15.28:0/2174441013 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff50000f460 con 0x7ff504101cc0 2024-04-25T01:23:34.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.606+0000 7ff50b32c700 1 -- 172.21.15.28:0/2174441013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff504074620 con 0x7ff504101cc0 2024-04-25T01:23:34.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.606+0000 7ff50b32c700 1 -- 172.21.15.28:0/2174441013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff504074b60 con 0x7ff504101cc0 2024-04-25T01:23:34.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.607+0000 7ff4fa7fc700 1 -- 172.21.15.28:0/2174441013 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7ff50002a070 con 0x7ff504101cc0 2024-04-25T01:23:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.607+0000 7ff50b32c700 1 -- 172.21.15.28:0/2174441013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5041084d0 con 0x7ff504101cc0 2024-04-25T01:23:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.607+0000 7ff4fa7fc700 1 --2- 172.21.15.28:0/2174441013 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4f00390f0 0x7ff4f003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.608+0000 7ff4fa7fc700 1 -- 172.21.15.28:0/2174441013 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff50001e030 con 0x7ff504101cc0 2024-04-25T01:23:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.608+0000 7ff5088c7700 1 --2- 172.21.15.28:0/2174441013 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4f00390f0 0x7ff4f003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.608+0000 7ff5088c7700 1 --2- 172.21.15.28:0/2174441013 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4f00390f0 0x7ff4f003b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff4f4006fd0 tx=0x7ff4f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:34.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.612+0000 7ff4fa7fc700 1 -- 172.21.15.28:0/2174441013 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff500013070 con 0x7ff504101cc0 2024-04-25T01:23:34.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.824+0000 7ff50b32c700 1 -- 172.21.15.28:0/2174441013 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7ff5041907d0 con 0x7ff4f00390f0 2024-04-25T01:23:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.827+0000 7ff4fa7fc700 1 -- 172.21.15.28:0/2174441013 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff5041907d0 con 0x7ff4f00390f0 2024-04-25T01:23:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.829+0000 7ff50b32c700 1 -- 172.21.15.28:0/2174441013 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4f00390f0 msgr2=0x7ff4f003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.829+0000 7ff50b32c700 1 --2- 172.21.15.28:0/2174441013 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4f00390f0 0x7ff4f003b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff4f4006fd0 tx=0x7ff4f4006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.829+0000 7ff50b32c700 1 -- 172.21.15.28:0/2174441013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff504101cc0 msgr2=0x7ff504075ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.829+0000 7ff50b32c700 1 --2- 172.21.15.28:0/2174441013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff504101cc0 0x7ff504075ef0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff500006010 tx=0x7ff500005050 comp rx=0 tx=0).stop 2024-04-25T01:23:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.829+0000 7ff50b32c700 1 -- 172.21.15.28:0/2174441013 shutdown_connections 2024-04-25T01:23:34.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.829+0000 7ff50b32c700 1 --2- 172.21.15.28:0/2174441013 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4f00390f0 0x7ff4f003b5b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:34.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.829+0000 7ff50b32c700 1 --2- 172.21.15.28:0/2174441013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff504101cc0 0x7ff504075ef0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:34.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.829+0000 7ff50b32c700 1 -- 172.21.15.28:0/2174441013 >> 172.21.15.28:0/2174441013 conn(0x7ff5040fd260 msgr2=0x7ff5040fdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:34.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.829+0000 7ff50b32c700 1 -- 172.21.15.28:0/2174441013 shutdown_connections 2024-04-25T01:23:34.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:34.829+0000 7ff50b32c700 1 -- 172.21.15.28:0/2174441013 wait complete. 2024-04-25T01:23:35.486 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi032 2024-04-25T01:23:35.486 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-25T01:23:35.486 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/etc/ceph/ceph.conf 2024-04-25T01:23:35.506 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-25T01:23:35.506 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:23:35.565 INFO:tasks.cephadm:Adding host smithi032 to orchestrator... 2024-04-25T01:23:35.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch host add smithi032 2024-04-25T01:23:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:36.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:35 smithi028 ceph-mon[29390]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:36.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:36.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:36.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:36.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:36.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:35 smithi028 ceph-mon[29390]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-25T01:23:36.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:36.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:36.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:36.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:36 smithi028 ceph-mon[29390]: Updating smithi028:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:23:36.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:36 smithi028 ceph-mon[29390]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:23:36.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:36 smithi028 ceph-mon[29390]: Updating smithi028:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:23:37.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.376+0000 7f9d077c9700 1 -- 172.21.15.28:0/2468445163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00101690 msgr2=0x7f9d00101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:37.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.376+0000 7f9d077c9700 1 --2- 172.21.15.28:0/2468445163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00101690 0x7f9d00101a70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00b0a0 tx=0x7f9cfc00b3b0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.376+0000 7f9d077c9700 1 -- 172.21.15.28:0/2468445163 shutdown_connections 2024-04-25T01:23:37.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.376+0000 7f9d077c9700 1 --2- 172.21.15.28:0/2468445163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00101690 0x7f9d00101a70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:37.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.376+0000 7f9d077c9700 1 -- 172.21.15.28:0/2468445163 >> 172.21.15.28:0/2468445163 conn(0x7f9d000780c0 msgr2=0x7f9d000784d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:37.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.376+0000 7f9d077c9700 1 -- 172.21.15.28:0/2468445163 shutdown_connections 2024-04-25T01:23:37.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.376+0000 7f9d077c9700 1 -- 172.21.15.28:0/2468445163 wait complete. 2024-04-25T01:23:37.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.377+0000 7f9d077c9700 1 Processor -- start 2024-04-25T01:23:37.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.377+0000 7f9d077c9700 1 -- start start 2024-04-25T01:23:37.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.377+0000 7f9d077c9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00101690 0x7f9d0019d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.377+0000 7f9d077c9700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0010b1f0 con 0x7f9d00101690 2024-04-25T01:23:37.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.377+0000 7f9d05565700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00101690 0x7f9d0019d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.377+0000 7f9d05565700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00101690 0x7f9d0019d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40224/0 (socket says 172.21.15.28:40224) 2024-04-25T01:23:37.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.377+0000 7f9d05565700 1 -- 172.21.15.28:0/425579019 learned_addr learned my addr 172.21.15.28:0/425579019 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:37.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.378+0000 7f9d05565700 1 -- 172.21.15.28:0/425579019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d0019d9a0 con 0x7f9d00101690 2024-04-25T01:23:37.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.378+0000 7f9d05565700 1 --2- 172.21.15.28:0/425579019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00101690 0x7f9d0019d460 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc015040 tx=0x7f9cfc003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:37.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.378+0000 7f9cf67fc700 1 -- 172.21.15.28:0/425579019 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9cfc017040 con 0x7f9d00101690 2024-04-25T01:23:37.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.378+0000 7f9cf67fc700 1 -- 172.21.15.28:0/425579019 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cfc022070 con 0x7f9d00101690 2024-04-25T01:23:37.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.378+0000 7f9cf67fc700 1 -- 172.21.15.28:0/425579019 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9cfc01d5d0 con 0x7f9d00101690 2024-04-25T01:23:37.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.378+0000 7f9d077c9700 1 -- 172.21.15.28:0/425579019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d0019dc30 con 0x7f9d00101690 2024-04-25T01:23:37.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.378+0000 7f9d077c9700 1 -- 172.21.15.28:0/425579019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d00197670 con 0x7f9d00101690 2024-04-25T01:23:37.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.379+0000 7f9cf67fc700 1 -- 172.21.15.28:0/425579019 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f9cfc004000 con 0x7f9d00101690 2024-04-25T01:23:37.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.379+0000 7f9d077c9700 1 -- 172.21.15.28:0/425579019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d00102440 con 0x7f9d00101690 2024-04-25T01:23:37.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.379+0000 7f9cf67fc700 1 --2- 172.21.15.28:0/425579019 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9cec039150 0x7f9cec03b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:37.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.379+0000 7f9cf67fc700 1 -- 172.21.15.28:0/425579019 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9cfc00f050 con 0x7f9d00101690 2024-04-25T01:23:37.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.379+0000 7f9d04d64700 1 --2- 172.21.15.28:0/425579019 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9cec039150 0x7f9cec03b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:37.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.380+0000 7f9d04d64700 1 --2- 172.21.15.28:0/425579019 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9cec039150 0x7f9cec03b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0006fd0 tx=0x7f9cf0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:37.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.384+0000 7f9cf67fc700 1 -- 172.21.15.28:0/425579019 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9cfc007e20 con 0x7f9d00101690 2024-04-25T01:23:37.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:37.595+0000 7f9d077c9700 1 -- 172.21.15.28:0/425579019 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi032", "target": ["mon-mgr", ""]}) -- 0x7f9d00063890 con 0x7f9cec039150 2024-04-25T01:23:38.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:38 smithi028 ceph-mon[29390]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi032", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:39.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:39 smithi028 ceph-mon[29390]: Deploying cephadm binary to smithi032 2024-04-25T01:23:41.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:41 smithi028 ceph-mon[29390]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:23:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:41.897+0000 7f9cf67fc700 1 -- 172.21.15.28:0/425579019 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f9d00063890 con 0x7f9cec039150 2024-04-25T01:23:41.878 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi032' with addr '172.21.15.32' 2024-04-25T01:23:41.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:41.900+0000 7f9d077c9700 1 -- 172.21.15.28:0/425579019 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9cec039150 msgr2=0x7f9cec03b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:41.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:41.900+0000 7f9d077c9700 1 --2- 172.21.15.28:0/425579019 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9cec039150 0x7f9cec03b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0006fd0 tx=0x7f9cf0006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:41.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:41.900+0000 7f9d077c9700 1 -- 172.21.15.28:0/425579019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00101690 msgr2=0x7f9d0019d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:41.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:41.900+0000 7f9d077c9700 1 --2- 172.21.15.28:0/425579019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00101690 0x7f9d0019d460 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc015040 tx=0x7f9cfc003d50 comp rx=0 tx=0).stop 2024-04-25T01:23:41.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:41.900+0000 7f9d077c9700 1 -- 172.21.15.28:0/425579019 shutdown_connections 2024-04-25T01:23:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:41.900+0000 7f9d077c9700 1 --2- 172.21.15.28:0/425579019 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9cec039150 0x7f9cec03b610 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:41.900+0000 7f9d077c9700 1 --2- 172.21.15.28:0/425579019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d00101690 0x7f9d0019d460 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:41.900+0000 7f9d077c9700 1 -- 172.21.15.28:0/425579019 >> 172.21.15.28:0/425579019 conn(0x7f9d000780c0 msgr2=0x7f9d000ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:41.900+0000 7f9d077c9700 1 -- 172.21.15.28:0/425579019 shutdown_connections 2024-04-25T01:23:41.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:41.900+0000 7f9d077c9700 1 -- 172.21.15.28:0/425579019 wait complete. 2024-04-25T01:23:42.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T01:23:42.723 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:43.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:42 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:43.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:42 smithi028 ceph-mon[29390]: Added host smithi032 2024-04-25T01:23:43.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:42 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:43.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:42 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.262+0000 7f72ed76b700 1 -- 172.21.15.28:0/2343941370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8101ea0 msgr2=0x7f72e8102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.262+0000 7f72ed76b700 1 --2- 172.21.15.28:0/2343941370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8101ea0 0x7f72e8102280 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009910 tx=0x7f72d8009c20 comp rx=0 tx=0).stop 2024-04-25T01:23:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.262+0000 7f72ed76b700 1 -- 172.21.15.28:0/2343941370 shutdown_connections 2024-04-25T01:23:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.262+0000 7f72ed76b700 1 --2- 172.21.15.28:0/2343941370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8101ea0 0x7f72e8102280 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.262+0000 7f72ed76b700 1 -- 172.21.15.28:0/2343941370 >> 172.21.15.28:0/2343941370 conn(0x7f72e80fd770 msgr2=0x7f72e80ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.262+0000 7f72ed76b700 1 -- 172.21.15.28:0/2343941370 shutdown_connections 2024-04-25T01:23:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.262+0000 7f72ed76b700 1 -- 172.21.15.28:0/2343941370 wait complete. 2024-04-25T01:23:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.262+0000 7f72ed76b700 1 Processor -- start 2024-04-25T01:23:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.263+0000 7f72ed76b700 1 -- start start 2024-04-25T01:23:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.263+0000 7f72ed76b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8101ea0 0x7f72e8074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.263+0000 7f72ed76b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f72e810c270 con 0x7f72e8101ea0 2024-04-25T01:23:44.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.263+0000 7f72e6ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8101ea0 0x7f72e8074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:44.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.263+0000 7f72e6ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8101ea0 0x7f72e8074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40232/0 (socket says 172.21.15.28:40232) 2024-04-25T01:23:44.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.263+0000 7f72e6ffd700 1 -- 172.21.15.28:0/2197568988 learned_addr learned my addr 172.21.15.28:0/2197568988 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:44.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.263+0000 7f72e6ffd700 1 -- 172.21.15.28:0/2197568988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e8076420 con 0x7f72e8101ea0 2024-04-25T01:23:44.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.263+0000 7f72e6ffd700 1 --2- 172.21.15.28:0/2197568988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8101ea0 0x7f72e8074530 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f72d8005850 tx=0x7f72d8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.263+0000 7f72dffff700 1 -- 172.21.15.28:0/2197568988 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72d8026070 con 0x7f72e8101ea0 2024-04-25T01:23:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.264+0000 7f72ed76b700 1 -- 172.21.15.28:0/2197568988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72e8074ad0 con 0x7f72e8101ea0 2024-04-25T01:23:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.264+0000 7f72dffff700 1 -- 172.21.15.28:0/2197568988 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72d800bc30 con 0x7f72e8101ea0 2024-04-25T01:23:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.264+0000 7f72dffff700 1 -- 172.21.15.28:0/2197568988 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72d8003a20 con 0x7f72e8101ea0 2024-04-25T01:23:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.264+0000 7f72ed76b700 1 -- 172.21.15.28:0/2197568988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72e8074f90 con 0x7f72e8101ea0 2024-04-25T01:23:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.264+0000 7f72dffff700 1 -- 172.21.15.28:0/2197568988 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f72d8029070 con 0x7f72e8101ea0 2024-04-25T01:23:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.264+0000 7f72ed76b700 1 -- 172.21.15.28:0/2197568988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72e8103380 con 0x7f72e8101ea0 2024-04-25T01:23:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.265+0000 7f72dffff700 1 --2- 172.21.15.28:0/2197568988 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f72d40390f0 0x7f72d403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.265+0000 7f72dffff700 1 -- 172.21.15.28:0/2197568988 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f72d801e070 con 0x7f72e8101ea0 2024-04-25T01:23:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.265+0000 7f72e67fc700 1 --2- 172.21.15.28:0/2197568988 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f72d40390f0 0x7f72d403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.265+0000 7f72e67fc700 1 --2- 172.21.15.28:0/2197568988 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f72d40390f0 0x7f72d403b5b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f72d0006fd0 tx=0x7f72d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.270+0000 7f72dffff700 1 -- 172.21.15.28:0/2197568988 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f72d8013070 con 0x7f72e8101ea0 2024-04-25T01:23:44.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:44 smithi028 ceph-mon[29390]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:23:44.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:44 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:44.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.485+0000 7f72ed76b700 1 -- 172.21.15.28:0/2197568988 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f72e8075a60 con 0x7f72d40390f0 2024-04-25T01:23:44.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.486+0000 7f72dffff700 1 -- 172.21.15.28:0/2197568988 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f72e8075a60 con 0x7f72d40390f0 2024-04-25T01:23:44.466 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:23:44.466 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.32", "hostname": "smithi032", "labels": [], "status": ""}] 2024-04-25T01:23:44.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.488+0000 7f72ed76b700 1 -- 172.21.15.28:0/2197568988 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f72d40390f0 msgr2=0x7f72d403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:44.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.488+0000 7f72ed76b700 1 --2- 172.21.15.28:0/2197568988 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f72d40390f0 0x7f72d403b5b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f72d0006fd0 tx=0x7f72d0006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:44.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.488+0000 7f72ed76b700 1 -- 172.21.15.28:0/2197568988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8101ea0 msgr2=0x7f72e8074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:44.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.488+0000 7f72ed76b700 1 --2- 172.21.15.28:0/2197568988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8101ea0 0x7f72e8074530 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f72d8005850 tx=0x7f72d8005030 comp rx=0 tx=0).stop 2024-04-25T01:23:44.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.488+0000 7f72ed76b700 1 -- 172.21.15.28:0/2197568988 shutdown_connections 2024-04-25T01:23:44.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.488+0000 7f72ed76b700 1 --2- 172.21.15.28:0/2197568988 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f72d40390f0 0x7f72d403b5b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:44.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.488+0000 7f72ed76b700 1 --2- 172.21.15.28:0/2197568988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72e8101ea0 0x7f72e8074530 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:44.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.488+0000 7f72ed76b700 1 -- 172.21.15.28:0/2197568988 >> 172.21.15.28:0/2197568988 conn(0x7f72e80fd770 msgr2=0x7f72e8104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:44.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.489+0000 7f72ed76b700 1 -- 172.21.15.28:0/2197568988 shutdown_connections 2024-04-25T01:23:44.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:44.489+0000 7f72ed76b700 1 -- 172.21.15.28:0/2197568988 wait complete. 2024-04-25T01:23:45.033 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi130 2024-04-25T01:23:45.033 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-25T01:23:45.033 DEBUG:teuthology.orchestra.run.smithi130:> dd of=/etc/ceph/ceph.conf 2024-04-25T01:23:45.053 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-25T01:23:45.053 DEBUG:teuthology.orchestra.run.smithi130:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:23:45.110 INFO:tasks.cephadm:Adding host smithi130 to orchestrator... 2024-04-25T01:23:45.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch host add smithi130 2024-04-25T01:23:45.350 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:45.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:45 smithi028 ceph-mon[29390]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:23:45.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:45 smithi028 ceph-mon[29390]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:23:45.717 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:45 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:46.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.951+0000 7f347a183700 1 -- 172.21.15.28:0/3412691684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3474100da0 msgr2=0x7f3474101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:46.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.951+0000 7f347a183700 1 --2- 172.21.15.28:0/3412691684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3474100da0 0x7f3474101180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3464009910 tx=0x7f3464009c20 comp rx=0 tx=0).stop 2024-04-25T01:23:46.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.951+0000 7f347a183700 1 -- 172.21.15.28:0/3412691684 shutdown_connections 2024-04-25T01:23:46.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.951+0000 7f347a183700 1 --2- 172.21.15.28:0/3412691684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3474100da0 0x7f3474101180 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:46.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.951+0000 7f347a183700 1 -- 172.21.15.28:0/3412691684 >> 172.21.15.28:0/3412691684 conn(0x7f34740fc630 msgr2=0x7f34740fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:46.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.951+0000 7f347a183700 1 -- 172.21.15.28:0/3412691684 shutdown_connections 2024-04-25T01:23:46.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.951+0000 7f347a183700 1 -- 172.21.15.28:0/3412691684 wait complete. 2024-04-25T01:23:46.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.951+0000 7f347a183700 1 Processor -- start 2024-04-25T01:23:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.952+0000 7f347a183700 1 -- start start 2024-04-25T01:23:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.952+0000 7f347a183700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3474100da0 0x7f3474102840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.952+0000 7f347a183700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f347410b170 con 0x7f3474100da0 2024-04-25T01:23:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.952+0000 7f34737fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3474100da0 0x7f3474102840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.952+0000 7f34737fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3474100da0 0x7f3474102840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49930/0 (socket says 172.21.15.28:49930) 2024-04-25T01:23:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.952+0000 7f34737fe700 1 -- 172.21.15.28:0/2425522609 learned_addr learned my addr 172.21.15.28:0/2425522609 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.952+0000 7f34737fe700 1 -- 172.21.15.28:0/2425522609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3474104730 con 0x7f3474100da0 2024-04-25T01:23:46.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.952+0000 7f34737fe700 1 --2- 172.21.15.28:0/2425522609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3474100da0 0x7f3474102840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3464005850 tx=0x7f3464005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:46.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.953+0000 7f3470ff9700 1 -- 172.21.15.28:0/2425522609 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3464026070 con 0x7f3474100da0 2024-04-25T01:23:46.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.953+0000 7f347a183700 1 -- 172.21.15.28:0/2425522609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3474102de0 con 0x7f3474100da0 2024-04-25T01:23:46.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.953+0000 7f3470ff9700 1 -- 172.21.15.28:0/2425522609 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f346400bc30 con 0x7f3474100da0 2024-04-25T01:23:46.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.953+0000 7f3470ff9700 1 -- 172.21.15.28:0/2425522609 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3464003a20 con 0x7f3474100da0 2024-04-25T01:23:46.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.953+0000 7f347a183700 1 -- 172.21.15.28:0/2425522609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34741032a0 con 0x7f3474100da0 2024-04-25T01:23:46.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.954+0000 7f3470ff9700 1 -- 172.21.15.28:0/2425522609 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f3464029070 con 0x7f3474100da0 2024-04-25T01:23:46.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.954+0000 7f347a183700 1 -- 172.21.15.28:0/2425522609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3474102280 con 0x7f3474100da0 2024-04-25T01:23:46.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.954+0000 7f3470ff9700 1 --2- 172.21.15.28:0/2425522609 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f345c0390f0 0x7f345c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:46.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.954+0000 7f3470ff9700 1 -- 172.21.15.28:0/2425522609 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f346401e070 con 0x7f3474100da0 2024-04-25T01:23:46.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.954+0000 7f3472ffd700 1 --2- 172.21.15.28:0/2425522609 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f345c0390f0 0x7f345c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:46.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.954+0000 7f3472ffd700 1 --2- 172.21.15.28:0/2425522609 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f345c0390f0 0x7f345c03b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3468006fd0 tx=0x7f3468006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:46.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:46.959+0000 7f3470ff9700 1 -- 172.21.15.28:0/2425522609 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3464013070 con 0x7f3474100da0 2024-04-25T01:23:47.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:47.168+0000 7f347a183700 1 -- 172.21.15.28:0/2425522609 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi130", "target": ["mon-mgr", ""]}) -- 0x7f3474103530 con 0x7f345c0390f0 2024-04-25T01:23:48.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:47 smithi028 ceph-mon[29390]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:23:49.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:48 smithi028 ceph-mon[29390]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi130", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:23:49.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:48 smithi028 ceph-mon[29390]: Deploying cephadm binary to smithi130 2024-04-25T01:23:49.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:48 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:49.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:48 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:49.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:48 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:49.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:48 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:49.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:48 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi032", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:49.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:48 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:49.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:48 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:49.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:48 smithi028 ceph-mon[29390]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-25T01:23:50.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:49 smithi028 ceph-mon[29390]: Updating smithi032:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:23:50.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:49 smithi028 ceph-mon[29390]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:23:50.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:49 smithi028 ceph-mon[29390]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:23:51.405 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi130' with addr '172.21.15.130' 2024-04-25T01:23:51.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:51.424+0000 7f3470ff9700 1 -- 172.21.15.28:0/2425522609 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f3474103530 con 0x7f345c0390f0 2024-04-25T01:23:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:51.426+0000 7f347a183700 1 -- 172.21.15.28:0/2425522609 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f345c0390f0 msgr2=0x7f345c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:51.426+0000 7f347a183700 1 --2- 172.21.15.28:0/2425522609 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f345c0390f0 0x7f345c03b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3468006fd0 tx=0x7f3468006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:51.426+0000 7f347a183700 1 -- 172.21.15.28:0/2425522609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3474100da0 msgr2=0x7f3474102840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:51.426+0000 7f347a183700 1 --2- 172.21.15.28:0/2425522609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3474100da0 0x7f3474102840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3464005850 tx=0x7f3464005030 comp rx=0 tx=0).stop 2024-04-25T01:23:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:51.426+0000 7f347a183700 1 -- 172.21.15.28:0/2425522609 shutdown_connections 2024-04-25T01:23:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:51.426+0000 7f347a183700 1 --2- 172.21.15.28:0/2425522609 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f345c0390f0 0x7f345c03b5b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:51.426+0000 7f347a183700 1 --2- 172.21.15.28:0/2425522609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3474100da0 0x7f3474102840 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:51.426+0000 7f347a183700 1 -- 172.21.15.28:0/2425522609 >> 172.21.15.28:0/2425522609 conn(0x7f34740fc630 msgr2=0x7f347406a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:51.427+0000 7f347a183700 1 -- 172.21.15.28:0/2425522609 shutdown_connections 2024-04-25T01:23:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:51.427+0000 7f347a183700 1 -- 172.21.15.28:0/2425522609 wait complete. 2024-04-25T01:23:51.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:51 smithi028 ceph-mon[29390]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:23:51.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:51 smithi028 ceph-mon[29390]: Updating smithi032:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:23:51.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:51 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:51.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:51 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:51.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:51 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:51.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:51 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:51.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:51 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:23:52.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T01:23:52.244 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:53.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:52 smithi028 ceph-mon[29390]: Added host smithi130 2024-04-25T01:23:53.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:52 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:53.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:52 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:53.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.693+0000 7f95bf83d700 1 -- 172.21.15.28:0/48688251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b8101ea0 msgr2=0x7f95b8102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:53.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.693+0000 7f95bf83d700 1 --2- 172.21.15.28:0/48688251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b8101ea0 0x7f95b8102280 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f95b4009910 tx=0x7f95b4009c20 comp rx=0 tx=0).stop 2024-04-25T01:23:53.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.693+0000 7f95bf83d700 1 -- 172.21.15.28:0/48688251 shutdown_connections 2024-04-25T01:23:53.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.693+0000 7f95bf83d700 1 --2- 172.21.15.28:0/48688251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b8101ea0 0x7f95b8102280 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:53.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.693+0000 7f95bf83d700 1 -- 172.21.15.28:0/48688251 >> 172.21.15.28:0/48688251 conn(0x7f95b80fd770 msgr2=0x7f95b80ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:53.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.694+0000 7f95bf83d700 1 -- 172.21.15.28:0/48688251 shutdown_connections 2024-04-25T01:23:53.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.694+0000 7f95bf83d700 1 -- 172.21.15.28:0/48688251 wait complete. 2024-04-25T01:23:53.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.694+0000 7f95bf83d700 1 Processor -- start 2024-04-25T01:23:53.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.694+0000 7f95bf83d700 1 -- start start 2024-04-25T01:23:53.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.694+0000 7f95bf83d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b8101ea0 0x7f95b8074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:53.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.694+0000 7f95bf83d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f95b810c270 con 0x7f95b8101ea0 2024-04-25T01:23:53.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.694+0000 7f95bd5d9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b8101ea0 0x7f95b8074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:53.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.694+0000 7f95bd5d9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b8101ea0 0x7f95b8074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49948/0 (socket says 172.21.15.28:49948) 2024-04-25T01:23:53.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.694+0000 7f95bd5d9700 1 -- 172.21.15.28:0/1778894564 learned_addr learned my addr 172.21.15.28:0/1778894564 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:53.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.695+0000 7f95bd5d9700 1 -- 172.21.15.28:0/1778894564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95b8076420 con 0x7f95b8101ea0 2024-04-25T01:23:53.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.695+0000 7f95bd5d9700 1 --2- 172.21.15.28:0/1778894564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b8101ea0 0x7f95b8074530 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f95b4006010 tx=0x7f95b4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:53.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.695+0000 7f95ae7fc700 1 -- 172.21.15.28:0/1778894564 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95b4026070 con 0x7f95b8101ea0 2024-04-25T01:23:53.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.695+0000 7f95ae7fc700 1 -- 172.21.15.28:0/1778894564 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95b400bc30 con 0x7f95b8101ea0 2024-04-25T01:23:53.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.695+0000 7f95bf83d700 1 -- 172.21.15.28:0/1778894564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95b8074ad0 con 0x7f95b8101ea0 2024-04-25T01:23:53.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.695+0000 7f95ae7fc700 1 -- 172.21.15.28:0/1778894564 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95b40039d0 con 0x7f95b8101ea0 2024-04-25T01:23:53.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.695+0000 7f95bf83d700 1 -- 172.21.15.28:0/1778894564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95b8074f90 con 0x7f95b8101ea0 2024-04-25T01:23:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.696+0000 7f95ae7fc700 1 -- 172.21.15.28:0/1778894564 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f95b4029070 con 0x7f95b8101ea0 2024-04-25T01:23:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.696+0000 7f95bf83d700 1 -- 172.21.15.28:0/1778894564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95b8103380 con 0x7f95b8101ea0 2024-04-25T01:23:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.696+0000 7f95ae7fc700 1 --2- 172.21.15.28:0/1778894564 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f95a40390f0 0x7f95a403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.696+0000 7f95ae7fc700 1 -- 172.21.15.28:0/1778894564 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f95b401e070 con 0x7f95b8101ea0 2024-04-25T01:23:53.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.696+0000 7f95bcdd8700 1 --2- 172.21.15.28:0/1778894564 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f95a40390f0 0x7f95a403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:53.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.697+0000 7f95bcdd8700 1 --2- 172.21.15.28:0/1778894564 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f95a40390f0 0x7f95a403b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f95a8006fd0 tx=0x7f95a8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:53.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.701+0000 7f95ae7fc700 1 -- 172.21.15.28:0/1778894564 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f95b4013070 con 0x7f95b8101ea0 2024-04-25T01:23:53.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.916+0000 7f95bf83d700 1 -- 172.21.15.28:0/1778894564 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f95b8075a60 con 0x7f95a40390f0 2024-04-25T01:23:53.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.917+0000 7f95ae7fc700 1 -- 172.21.15.28:0/1778894564 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7f95b8075a60 con 0x7f95a40390f0 2024-04-25T01:23:53.898 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:23:53.898 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.32", "hostname": "smithi032", "labels": [], "status": ""}, {"addr": "172.21.15.130", "hostname": "smithi130", "labels": [], "status": ""}] 2024-04-25T01:23:53.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.919+0000 7f95bf83d700 1 -- 172.21.15.28:0/1778894564 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f95a40390f0 msgr2=0x7f95a403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:53.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.919+0000 7f95bf83d700 1 --2- 172.21.15.28:0/1778894564 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f95a40390f0 0x7f95a403b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f95a8006fd0 tx=0x7f95a8006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:53.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.919+0000 7f95bf83d700 1 -- 172.21.15.28:0/1778894564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b8101ea0 msgr2=0x7f95b8074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:53.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.919+0000 7f95bf83d700 1 --2- 172.21.15.28:0/1778894564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b8101ea0 0x7f95b8074530 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f95b4006010 tx=0x7f95b4005030 comp rx=0 tx=0).stop 2024-04-25T01:23:53.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.919+0000 7f95bf83d700 1 -- 172.21.15.28:0/1778894564 shutdown_connections 2024-04-25T01:23:53.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.919+0000 7f95bf83d700 1 --2- 172.21.15.28:0/1778894564 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f95a40390f0 0x7f95a403b5b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:53.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.919+0000 7f95bf83d700 1 --2- 172.21.15.28:0/1778894564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95b8101ea0 0x7f95b8074530 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:53.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.919+0000 7f95bf83d700 1 -- 172.21.15.28:0/1778894564 >> 172.21.15.28:0/1778894564 conn(0x7f95b80fd770 msgr2=0x7f95b8103870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:53.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.919+0000 7f95bf83d700 1 -- 172.21.15.28:0/1778894564 shutdown_connections 2024-04-25T01:23:53.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:53.919+0000 7f95bf83d700 1 -- 172.21.15.28:0/1778894564 wait complete. 2024-04-25T01:23:54.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:53 smithi028 ceph-mon[29390]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:23:54.466 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-25T01:23:54.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-25T01:23:54.707 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:23:55.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:55 smithi028 ceph-mon[29390]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:23:55.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:55 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:56.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:56 smithi028 ceph-mon[29390]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:23:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.189+0000 7f406a688700 1 -- 172.21.15.28:0/94980246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4064102de0 msgr2=0x7f40641031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.189+0000 7f406a688700 1 --2- 172.21.15.28:0/94980246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4064102de0 0x7f40641031c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4054009910 tx=0x7f4054009c20 comp rx=0 tx=0).stop 2024-04-25T01:23:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.189+0000 7f406a688700 1 -- 172.21.15.28:0/94980246 shutdown_connections 2024-04-25T01:23:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.189+0000 7f406a688700 1 --2- 172.21.15.28:0/94980246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4064102de0 0x7f40641031c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.189+0000 7f406a688700 1 -- 172.21.15.28:0/94980246 >> 172.21.15.28:0/94980246 conn(0x7f40640fc630 msgr2=0x7f40640fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.189+0000 7f406a688700 1 -- 172.21.15.28:0/94980246 shutdown_connections 2024-04-25T01:23:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.189+0000 7f406a688700 1 -- 172.21.15.28:0/94980246 wait complete. 2024-04-25T01:23:56.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.189+0000 7f406a688700 1 Processor -- start 2024-04-25T01:23:56.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.189+0000 7f406a688700 1 -- start start 2024-04-25T01:23:56.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.190+0000 7f406a688700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4064102de0 0x7f406419d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:56.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.190+0000 7f406a688700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f406410c2d0 con 0x7f4064102de0 2024-04-25T01:23:56.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.190+0000 7f4063fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4064102de0 0x7f406419d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:56.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.190+0000 7f4063fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4064102de0 0x7f406419d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49966/0 (socket says 172.21.15.28:49966) 2024-04-25T01:23:56.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.190+0000 7f4063fff700 1 -- 172.21.15.28:0/3346046527 learned_addr learned my addr 172.21.15.28:0/3346046527 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:23:56.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.190+0000 7f4063fff700 1 -- 172.21.15.28:0/3346046527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406419d980 con 0x7f4064102de0 2024-04-25T01:23:56.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.190+0000 7f4063fff700 1 --2- 172.21.15.28:0/3346046527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4064102de0 0x7f406419d440 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4054006010 tx=0x7f4054005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:56.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.190+0000 7f40617fa700 1 -- 172.21.15.28:0/3346046527 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4054026070 con 0x7f4064102de0 2024-04-25T01:23:56.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.190+0000 7f40617fa700 1 -- 172.21.15.28:0/3346046527 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f405400bc30 con 0x7f4064102de0 2024-04-25T01:23:56.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.190+0000 7f406a688700 1 -- 172.21.15.28:0/3346046527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f406419dc10 con 0x7f4064102de0 2024-04-25T01:23:56.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.190+0000 7f40617fa700 1 -- 172.21.15.28:0/3346046527 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4054003ab0 con 0x7f4064102de0 2024-04-25T01:23:56.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.190+0000 7f406a688700 1 -- 172.21.15.28:0/3346046527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4064197600 con 0x7f4064102de0 2024-04-25T01:23:56.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.191+0000 7f40617fa700 1 -- 172.21.15.28:0/3346046527 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f4054029070 con 0x7f4064102de0 2024-04-25T01:23:56.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.191+0000 7f406a688700 1 -- 172.21.15.28:0/3346046527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f406406a7c0 con 0x7f4064102de0 2024-04-25T01:23:56.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.191+0000 7f40617fa700 1 --2- 172.21.15.28:0/3346046527 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4050039150 0x7f405003b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:56.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.192+0000 7f40617fa700 1 -- 172.21.15.28:0/3346046527 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f405401e070 con 0x7f4064102de0 2024-04-25T01:23:56.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.192+0000 7f40637fe700 1 --2- 172.21.15.28:0/3346046527 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4050039150 0x7f405003b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:56.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.192+0000 7f40637fe700 1 --2- 172.21.15.28:0/3346046527 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4050039150 0x7f405003b610 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f404c006fd0 tx=0x7f404c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:56.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.196+0000 7f40617fa700 1 -- 172.21.15.28:0/3346046527 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4054013070 con 0x7f4064102de0 2024-04-25T01:23:56.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:56.410+0000 7f406a688700 1 -- 172.21.15.28:0/3346046527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f4064063890 con 0x7f4064102de0 2024-04-25T01:23:57.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:57.047+0000 7f40617fa700 1 -- 172.21.15.28:0/3346046527 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f4054030090 con 0x7f4064102de0 2024-04-25T01:23:57.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:57.050+0000 7f406a688700 1 -- 172.21.15.28:0/3346046527 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4050039150 msgr2=0x7f405003b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:57.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:57.050+0000 7f406a688700 1 --2- 172.21.15.28:0/3346046527 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4050039150 0x7f405003b610 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f404c006fd0 tx=0x7f404c006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:57.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:57.050+0000 7f406a688700 1 -- 172.21.15.28:0/3346046527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4064102de0 msgr2=0x7f406419d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:57.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:57.050+0000 7f406a688700 1 --2- 172.21.15.28:0/3346046527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4064102de0 0x7f406419d440 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4054006010 tx=0x7f4054005050 comp rx=0 tx=0).stop 2024-04-25T01:23:57.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:57.051+0000 7f406a688700 1 -- 172.21.15.28:0/3346046527 shutdown_connections 2024-04-25T01:23:57.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:57.051+0000 7f406a688700 1 --2- 172.21.15.28:0/3346046527 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4050039150 0x7f405003b610 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:57.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:57.051+0000 7f406a688700 1 --2- 172.21.15.28:0/3346046527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4064102de0 0x7f406419d440 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:57.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:57.051+0000 7f406a688700 1 -- 172.21.15.28:0/3346046527 >> 172.21.15.28:0/3346046527 conn(0x7f40640fc630 msgr2=0x7f4064103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:57.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:57.051+0000 7f406a688700 1 -- 172.21.15.28:0/3346046527 shutdown_connections 2024-04-25T01:23:57.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:23:57.051+0000 7f406a688700 1 -- 172.21.15.28:0/3346046527 wait complete. 2024-04-25T01:23:57.033 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-04-25T01:23:57.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:57 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3346046527' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T01:23:57.688 INFO:tasks.cephadm:Adding mon.a on smithi028 2024-04-25T01:23:57.689 INFO:tasks.cephadm:Adding mon.b on smithi032 2024-04-25T01:23:57.689 INFO:tasks.cephadm:Adding mon.c on smithi130 2024-04-25T01:23:57.689 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi028:172.21.15.28=a;smithi032:172.21.15.32=b;smithi130:172.21.15.130=c' 2024-04-25T01:23:57.923 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:23:58.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:58 smithi028 ceph-mon[29390]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:23:58.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:58 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3346046527' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T01:23:58.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:58 smithi028 ceph-mon[29390]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:23:58.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:58 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:58.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:58 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:58.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:58 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:58.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:58 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:58.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:58 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi130", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:23:58.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:58 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:23:58.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:58 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:23:58.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:58 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:58.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:58 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:58.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:58 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:23:59.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:59 smithi028 ceph-mon[29390]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-25T01:23:59.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:59 smithi028 ceph-mon[29390]: Updating smithi130:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:23:59.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:59 smithi028 ceph-mon[29390]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:23:59.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:23:59 smithi028 ceph-mon[29390]: Updating smithi130:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:23:59.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.450+0000 7f973c715700 1 -- 172.21.15.130:0/3935790961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9734101ea0 msgr2=0x7f9734102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:59.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.450+0000 7f973c715700 1 --2- 172.21.15.130:0/3935790961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9734101ea0 0x7f9734102280 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9730009910 tx=0x7f9730009c20 comp rx=0 tx=0).stop 2024-04-25T01:23:59.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.450+0000 7f973c715700 1 -- 172.21.15.130:0/3935790961 shutdown_connections 2024-04-25T01:23:59.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.450+0000 7f973c715700 1 --2- 172.21.15.130:0/3935790961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9734101ea0 0x7f9734102280 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.450+0000 7f973c715700 1 -- 172.21.15.130:0/3935790961 >> 172.21.15.130:0/3935790961 conn(0x7f97340fd770 msgr2=0x7f97340ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:59.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.450+0000 7f973c715700 1 -- 172.21.15.130:0/3935790961 shutdown_connections 2024-04-25T01:23:59.432 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.450+0000 7f973c715700 1 -- 172.21.15.130:0/3935790961 wait complete. 2024-04-25T01:23:59.433 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.451+0000 7f973c715700 1 Processor -- start 2024-04-25T01:23:59.433 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.451+0000 7f973c715700 1 -- start start 2024-04-25T01:23:59.433 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.451+0000 7f973c715700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9734101ea0 0x7f973419d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:59.433 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.451+0000 7f973c715700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f973410c270 con 0x7f9734101ea0 2024-04-25T01:23:59.433 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.451+0000 7f973a4b1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9734101ea0 0x7f973419d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:59.433 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.451+0000 7f973a4b1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9734101ea0 0x7f973419d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.130:48566/0 (socket says 172.21.15.130:48566) 2024-04-25T01:23:59.433 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.451+0000 7f973a4b1700 1 -- 172.21.15.130:0/2338651148 learned_addr learned my addr 172.21.15.130:0/2338651148 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-25T01:23:59.433 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.451+0000 7f973a4b1700 1 -- 172.21.15.130:0/2338651148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f973419dab0 con 0x7f9734101ea0 2024-04-25T01:23:59.433 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.452+0000 7f973a4b1700 1 --2- 172.21.15.130:0/2338651148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9734101ea0 0x7f973419d570 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9730005850 tx=0x7f9730005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:59.435 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.452+0000 7f972b7fe700 1 -- 172.21.15.130:0/2338651148 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9730026070 con 0x7f9734101ea0 2024-04-25T01:23:59.435 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.452+0000 7f973c715700 1 -- 172.21.15.130:0/2338651148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f973419dda0 con 0x7f9734101ea0 2024-04-25T01:23:59.435 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.452+0000 7f972b7fe700 1 -- 172.21.15.130:0/2338651148 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f973000bbd0 con 0x7f9734101ea0 2024-04-25T01:23:59.435 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.452+0000 7f973c715700 1 -- 172.21.15.130:0/2338651148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9734197810 con 0x7f9734101ea0 2024-04-25T01:23:59.435 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.452+0000 7f972b7fe700 1 -- 172.21.15.130:0/2338651148 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9730003970 con 0x7f9734101ea0 2024-04-25T01:23:59.435 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.453+0000 7f973c715700 1 -- 172.21.15.130:0/2338651148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9734103380 con 0x7f9734101ea0 2024-04-25T01:23:59.435 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.453+0000 7f972b7fe700 1 -- 172.21.15.130:0/2338651148 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f9730029070 con 0x7f9734101ea0 2024-04-25T01:23:59.435 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.453+0000 7f972b7fe700 1 --2- 172.21.15.130:0/2338651148 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f97200390f0 0x7f972003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:23:59.435 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.453+0000 7f972b7fe700 1 -- 172.21.15.130:0/2338651148 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f973001e070 con 0x7f9734101ea0 2024-04-25T01:23:59.435 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.453+0000 7f9739cb0700 1 --2- 172.21.15.130:0/2338651148 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f97200390f0 0x7f972003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:23:59.435 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.454+0000 7f9739cb0700 1 --2- 172.21.15.130:0/2338651148 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f97200390f0 0x7f972003b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9724006fd0 tx=0x7f9724006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:23:59.439 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.459+0000 7f972b7fe700 1 -- 172.21.15.130:0/2338651148 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9730013070 con 0x7f9734101ea0 2024-04-25T01:23:59.647 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.666+0000 7f973c715700 1 -- 172.21.15.130:0/2338651148 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi032:172.21.15.32=b;smithi130:172.21.15.130=c", "target": ["mon-mgr", ""]}) -- 0x7f9734198b70 con 0x7f97200390f0 2024-04-25T01:23:59.653 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.673+0000 7f972b7fe700 1 -- 172.21.15.130:0/2338651148 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f9734198b70 con 0x7f97200390f0 2024-04-25T01:23:59.654 INFO:teuthology.orchestra.run.smithi130.stdout:Scheduled mon update... 2024-04-25T01:23:59.656 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.675+0000 7f973c715700 1 -- 172.21.15.130:0/2338651148 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f97200390f0 msgr2=0x7f972003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:59.656 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.675+0000 7f973c715700 1 --2- 172.21.15.130:0/2338651148 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f97200390f0 0x7f972003b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9724006fd0 tx=0x7f9724006e40 comp rx=0 tx=0).stop 2024-04-25T01:23:59.656 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.675+0000 7f973c715700 1 -- 172.21.15.130:0/2338651148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9734101ea0 msgr2=0x7f973419d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:23:59.656 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.675+0000 7f973c715700 1 --2- 172.21.15.130:0/2338651148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9734101ea0 0x7f973419d570 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9730005850 tx=0x7f9730005030 comp rx=0 tx=0).stop 2024-04-25T01:23:59.656 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.675+0000 7f973c715700 1 -- 172.21.15.130:0/2338651148 shutdown_connections 2024-04-25T01:23:59.656 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.675+0000 7f973c715700 1 --2- 172.21.15.130:0/2338651148 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f97200390f0 0x7f972003b5b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.656 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.675+0000 7f973c715700 1 --2- 172.21.15.130:0/2338651148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9734101ea0 0x7f973419d570 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:23:59.656 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.675+0000 7f973c715700 1 -- 172.21.15.130:0/2338651148 >> 172.21.15.130:0/2338651148 conn(0x7f97340fd770 msgr2=0x7f9734104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:23:59.656 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.675+0000 7f973c715700 1 -- 172.21.15.130:0/2338651148 shutdown_connections 2024-04-25T01:23:59.656 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:23:59.675+0000 7f973c715700 1 -- 172.21.15.130:0/2338651148 wait complete. 2024-04-25T01:24:00.228 DEBUG:teuthology.orchestra.run.smithi032:mon.b> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mon.b.service 2024-04-25T01:24:00.231 DEBUG:teuthology.orchestra.run.smithi130:mon.c> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mon.c.service 2024-04-25T01:24:00.233 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T01:24:00.233 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T01:24:00.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:00 smithi028 ceph-mon[29390]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:00.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:00.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:24:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:01.002 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:24:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:01 smithi028 ceph-mon[29390]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi032:172.21.15.32=b;smithi130:172.21.15.130=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:01 smithi028 ceph-mon[29390]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi032:172.21.15.32=b;smithi130:172.21.15.130=c;count:3 2024-04-25T01:24:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:01 smithi028 ceph-mon[29390]: Deploying daemon mon.c on smithi130 2024-04-25T01:24:02.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:02 smithi028 ceph-mon[29390]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:02.711 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.729+0000 7f1238725700 1 -- 172.21.15.130:0/3727531046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1230107cd0 msgr2=0x7f12301080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:02.711 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.729+0000 7f1238725700 1 --2- 172.21.15.130:0/3727531046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1230107cd0 0x7f12301080b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f122c009910 tx=0x7f122c009c20 comp rx=0 tx=0).stop 2024-04-25T01:24:02.711 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.729+0000 7f1238725700 1 -- 172.21.15.130:0/3727531046 shutdown_connections 2024-04-25T01:24:02.711 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.729+0000 7f1238725700 1 --2- 172.21.15.130:0/3727531046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1230107cd0 0x7f12301080b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:02.711 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.729+0000 7f1238725700 1 -- 172.21.15.130:0/3727531046 >> 172.21.15.130:0/3727531046 conn(0x7f1230076690 msgr2=0x7f1230076aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:02.711 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.730+0000 7f1238725700 1 -- 172.21.15.130:0/3727531046 shutdown_connections 2024-04-25T01:24:02.712 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.730+0000 7f1238725700 1 -- 172.21.15.130:0/3727531046 wait complete. 2024-04-25T01:24:02.712 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.730+0000 7f1238725700 1 Processor -- start 2024-04-25T01:24:02.712 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.730+0000 7f1238725700 1 -- start start 2024-04-25T01:24:02.712 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.730+0000 7f1238725700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1230107cd0 0x7f123019d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.712 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.730+0000 7f1238725700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f123010c260 con 0x7f1230107cd0 2024-04-25T01:24:02.712 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.731+0000 7f12364c1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1230107cd0 0x7f123019d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.712 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.731+0000 7f12364c1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1230107cd0 0x7f123019d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.130:38636/0 (socket says 172.21.15.130:38636) 2024-04-25T01:24:02.712 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.731+0000 7f12364c1700 1 -- 172.21.15.130:0/34666413 learned_addr learned my addr 172.21.15.130:0/34666413 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-25T01:24:02.712 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.731+0000 7f12364c1700 1 -- 172.21.15.130:0/34666413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f123019d910 con 0x7f1230107cd0 2024-04-25T01:24:02.714 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.731+0000 7f12364c1700 1 --2- 172.21.15.130:0/34666413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1230107cd0 0x7f123019d3d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f122c005850 tx=0x7f122c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:02.714 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.731+0000 7f12277fe700 1 -- 172.21.15.130:0/34666413 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f122c026070 con 0x7f1230107cd0 2024-04-25T01:24:02.714 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.731+0000 7f1238725700 1 -- 172.21.15.130:0/34666413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f123019dc00 con 0x7f1230107cd0 2024-04-25T01:24:02.714 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.731+0000 7f12277fe700 1 -- 172.21.15.130:0/34666413 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f122c00bbd0 con 0x7f1230107cd0 2024-04-25T01:24:02.714 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.731+0000 7f12277fe700 1 -- 172.21.15.130:0/34666413 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f122c003970 con 0x7f1230107cd0 2024-04-25T01:24:02.714 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.731+0000 7f1238725700 1 -- 172.21.15.130:0/34666413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1230197590 con 0x7f1230107cd0 2024-04-25T01:24:02.714 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.732+0000 7f1238725700 1 -- 172.21.15.130:0/34666413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1230103380 con 0x7f1230107cd0 2024-04-25T01:24:02.714 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.732+0000 7f12277fe700 1 -- 172.21.15.130:0/34666413 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f122c029070 con 0x7f1230107cd0 2024-04-25T01:24:02.714 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.732+0000 7f12277fe700 1 --2- 172.21.15.130:0/34666413 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f121c0390f0 0x7f121c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:02.714 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.732+0000 7f12277fe700 1 -- 172.21.15.130:0/34666413 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f122c01e070 con 0x7f1230107cd0 2024-04-25T01:24:02.714 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.733+0000 7f1235cc0700 1 --2- 172.21.15.130:0/34666413 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f121c0390f0 0x7f121c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:02.714 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.733+0000 7f1235cc0700 1 --2- 172.21.15.130:0/34666413 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f121c0390f0 0x7f121c03b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1220006fd0 tx=0x7f1220006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:02.718 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:02.738+0000 7f12277fe700 1 -- 172.21.15.130:0/34666413 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f122c013070 con 0x7f1230107cd0 2024-04-25T01:24:03.003 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:03.022+0000 7f1238725700 1 -- 172.21.15.130:0/34666413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f1230063890 con 0x7f1230107cd0 2024-04-25T01:24:03.004 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:03.023+0000 7f12277fe700 1 -- 172.21.15.130:0/34666413 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f122c030020 con 0x7f1230107cd0 2024-04-25T01:24:03.004 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-25T01:24:03.004 INFO:teuthology.orchestra.run.smithi130.stdout:{"epoch":1,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","modified":"2024-04-25T01:22:20.217202Z","created":"2024-04-25T01:22:20.217202Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-25T01:24:03.006 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:03.025+0000 7f1238725700 1 -- 172.21.15.130:0/34666413 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f121c0390f0 msgr2=0x7f121c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:03.006 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:03.025+0000 7f1238725700 1 --2- 172.21.15.130:0/34666413 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f121c0390f0 0x7f121c03b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1220006fd0 tx=0x7f1220006e40 comp rx=0 tx=0).stop 2024-04-25T01:24:03.006 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:03.025+0000 7f1238725700 1 -- 172.21.15.130:0/34666413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1230107cd0 msgr2=0x7f123019d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:03.006 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:03.025+0000 7f1238725700 1 --2- 172.21.15.130:0/34666413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1230107cd0 0x7f123019d3d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f122c005850 tx=0x7f122c005030 comp rx=0 tx=0).stop 2024-04-25T01:24:03.006 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:03.025+0000 7f1238725700 1 -- 172.21.15.130:0/34666413 shutdown_connections 2024-04-25T01:24:03.006 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:03.025+0000 7f1238725700 1 --2- 172.21.15.130:0/34666413 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f121c0390f0 0x7f121c03b5b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:03.006 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:03.025+0000 7f1238725700 1 --2- 172.21.15.130:0/34666413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1230107cd0 0x7f123019d3d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:03.006 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:03.025+0000 7f1238725700 1 -- 172.21.15.130:0/34666413 >> 172.21.15.130:0/34666413 conn(0x7f1230076690 msgr2=0x7f12300fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:03.006 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:03.025+0000 7f1238725700 1 -- 172.21.15.130:0/34666413 shutdown_connections 2024-04-25T01:24:03.007 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:03.025+0000 7f1238725700 1 -- 172.21.15.130:0/34666413 wait complete. 2024-04-25T01:24:03.007 INFO:teuthology.orchestra.run.smithi130.stderr:dumped monmap epoch 1 2024-04-25T01:24:03.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:03 smithi028 ceph-mon[29390]: from='client.? 172.21.15.130:0/34666413' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:24:04.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:04 smithi028 ceph-mon[29390]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:04.584 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T01:24:04.584 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T01:24:04.818 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:24:06.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:06 smithi028 ceph-mon[29390]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:06.372 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.390+0000 7f6b860fe700 1 -- 172.21.15.130:0/520903053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b80100660 msgr2=0x7f6b80100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.372 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.390+0000 7f6b860fe700 1 --2- 172.21.15.130:0/520903053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b80100660 0x7f6b80100a40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009910 tx=0x7f6b70009c20 comp rx=0 tx=0).stop 2024-04-25T01:24:06.372 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.390+0000 7f6b860fe700 1 -- 172.21.15.130:0/520903053 shutdown_connections 2024-04-25T01:24:06.372 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.390+0000 7f6b860fe700 1 --2- 172.21.15.130:0/520903053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b80100660 0x7f6b80100a40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.373 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.390+0000 7f6b860fe700 1 -- 172.21.15.130:0/520903053 >> 172.21.15.130:0/520903053 conn(0x7f6b800770b0 msgr2=0x7f6b800774c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:06.373 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.390+0000 7f6b860fe700 1 -- 172.21.15.130:0/520903053 shutdown_connections 2024-04-25T01:24:06.373 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.390+0000 7f6b860fe700 1 -- 172.21.15.130:0/520903053 wait complete. 2024-04-25T01:24:06.373 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.391+0000 7f6b860fe700 1 Processor -- start 2024-04-25T01:24:06.373 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.391+0000 7f6b860fe700 1 -- start start 2024-04-25T01:24:06.373 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.391+0000 7f6b860fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b80100660 0x7f6b801986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.373 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.391+0000 7f6b860fe700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8010a9e0 con 0x7f6b80100660 2024-04-25T01:24:06.373 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.392+0000 7f6b7f7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b80100660 0x7f6b801986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.373 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.392+0000 7f6b7f7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b80100660 0x7f6b801986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.130:38650/0 (socket says 172.21.15.130:38650) 2024-04-25T01:24:06.373 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.392+0000 7f6b7f7fe700 1 -- 172.21.15.130:0/1950072048 learned_addr learned my addr 172.21.15.130:0/1950072048 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-25T01:24:06.374 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.392+0000 7f6b7f7fe700 1 -- 172.21.15.130:0/1950072048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b80198be0 con 0x7f6b80100660 2024-04-25T01:24:06.375 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.392+0000 7f6b7f7fe700 1 --2- 172.21.15.130:0/1950072048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b80100660 0x7f6b801986a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6b70005850 tx=0x7f6b70005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:06.375 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.392+0000 7f6b7cff9700 1 -- 172.21.15.130:0/1950072048 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b70026070 con 0x7f6b80100660 2024-04-25T01:24:06.375 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.393+0000 7f6b7cff9700 1 -- 172.21.15.130:0/1950072048 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b7000bbd0 con 0x7f6b80100660 2024-04-25T01:24:06.375 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.393+0000 7f6b860fe700 1 -- 172.21.15.130:0/1950072048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b80198ed0 con 0x7f6b80100660 2024-04-25T01:24:06.375 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.393+0000 7f6b7cff9700 1 -- 172.21.15.130:0/1950072048 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b70003970 con 0x7f6b80100660 2024-04-25T01:24:06.375 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.393+0000 7f6b860fe700 1 -- 172.21.15.130:0/1950072048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b80199390 con 0x7f6b80100660 2024-04-25T01:24:06.375 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.394+0000 7f6b7cff9700 1 -- 172.21.15.130:0/1950072048 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f6b70029070 con 0x7f6b80100660 2024-04-25T01:24:06.375 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.394+0000 7f6b860fe700 1 -- 172.21.15.130:0/1950072048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b80101af0 con 0x7f6b80100660 2024-04-25T01:24:06.375 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.394+0000 7f6b7cff9700 1 --2- 172.21.15.130:0/1950072048 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6b680390f0 0x7f6b6803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:06.376 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.394+0000 7f6b7cff9700 1 -- 172.21.15.130:0/1950072048 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6b7001e070 con 0x7f6b80100660 2024-04-25T01:24:06.376 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.394+0000 7f6b7effd700 1 --2- 172.21.15.130:0/1950072048 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6b680390f0 0x7f6b6803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:06.376 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.395+0000 7f6b7effd700 1 --2- 172.21.15.130:0/1950072048 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6b680390f0 0x7f6b6803b5b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6b74006fd0 tx=0x7f6b74006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:06.380 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.399+0000 7f6b7cff9700 1 -- 172.21.15.130:0/1950072048 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6b70013070 con 0x7f6b80100660 2024-04-25T01:24:06.664 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.683+0000 7f6b860fe700 1 -- 172.21.15.130:0/1950072048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f6b800fd9d0 con 0x7f6b80100660 2024-04-25T01:24:06.664 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.683+0000 7f6b7cff9700 1 -- 172.21.15.130:0/1950072048 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f6b70030070 con 0x7f6b80100660 2024-04-25T01:24:06.664 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-25T01:24:06.665 INFO:teuthology.orchestra.run.smithi130.stdout:{"epoch":1,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","modified":"2024-04-25T01:22:20.217202Z","created":"2024-04-25T01:22:20.217202Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-25T01:24:06.666 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.685+0000 7f6b860fe700 1 -- 172.21.15.130:0/1950072048 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6b680390f0 msgr2=0x7f6b6803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.666 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.685+0000 7f6b860fe700 1 --2- 172.21.15.130:0/1950072048 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6b680390f0 0x7f6b6803b5b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6b74006fd0 tx=0x7f6b74006e40 comp rx=0 tx=0).stop 2024-04-25T01:24:06.666 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.685+0000 7f6b860fe700 1 -- 172.21.15.130:0/1950072048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b80100660 msgr2=0x7f6b801986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:06.666 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.685+0000 7f6b860fe700 1 --2- 172.21.15.130:0/1950072048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b80100660 0x7f6b801986a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6b70005850 tx=0x7f6b70005030 comp rx=0 tx=0).stop 2024-04-25T01:24:06.666 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.685+0000 7f6b860fe700 1 -- 172.21.15.130:0/1950072048 shutdown_connections 2024-04-25T01:24:06.667 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.685+0000 7f6b860fe700 1 --2- 172.21.15.130:0/1950072048 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6b680390f0 0x7f6b6803b5b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.667 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.685+0000 7f6b860fe700 1 --2- 172.21.15.130:0/1950072048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b80100660 0x7f6b801986a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:06.667 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.685+0000 7f6b860fe700 1 -- 172.21.15.130:0/1950072048 >> 172.21.15.130:0/1950072048 conn(0x7f6b800770b0 msgr2=0x7f6b80103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:06.667 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.686+0000 7f6b860fe700 1 -- 172.21.15.130:0/1950072048 shutdown_connections 2024-04-25T01:24:06.667 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:06.686+0000 7f6b860fe700 1 -- 172.21.15.130:0/1950072048 wait complete. 2024-04-25T01:24:06.667 INFO:teuthology.orchestra.run.smithi130.stderr:dumped monmap epoch 1 2024-04-25T01:24:06.723 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: mon.c@-1(???) e0 preinit fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:24:07.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: mon.c@-1(synchronizing).mds e1 new map 2024-04-25T01:24:07.086 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-25T01:24:07.086 INFO:journalctl@ceph.mon.c.smithi130.stdout: e1 2024-04-25T01:24:07.086 INFO:journalctl@ceph.mon.c.smithi130.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T01:24:07.087 INFO:journalctl@ceph.mon.c.smithi130.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:24:07.087 INFO:journalctl@ceph.mon.c.smithi130.stdout: legacy client fscid: -1 2024-04-25T01:24:07.087 INFO:journalctl@ceph.mon.c.smithi130.stdout: 2024-04-25T01:24:07.087 INFO:journalctl@ceph.mon.c.smithi130.stdout: No filesystems configured 2024-04-25T01:24:07.087 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-25T01:24:07.087 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-25T01:24:07.087 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-25T01:24:07.087 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-25T01:24:07.087 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-25T01:24:07.087 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-25T01:24:07.087 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T01:24:07.087 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:24:07.087 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:24:07.087 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:24:07.087 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:07.087 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:07.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:24:07.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:07.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi130", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:07.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: Deploying cephadm binary to smithi130 2024-04-25T01:24:07.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi032", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:24:07.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:07.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:07.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-25T01:24:07.088 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: Updating smithi032:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:24:07.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:07.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:24:07.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:07.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: Updating smithi032:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:24:07.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:07.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: Added host smithi130 2024-04-25T01:24:07.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:07.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:24:07.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:07.089 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3346046527' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T01:24:07.090 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:07.090 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3346046527' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T01:24:07.090 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:24:07.090 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.090 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.090 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.090 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.090 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi130", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:24:07.090 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:07.090 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:07.090 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.090 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.090 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.090 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-25T01:24:07.090 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: Updating smithi130:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:24:07.091 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:24:07.091 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: Updating smithi130:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:24:07.091 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:07.091 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.091 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:07.091 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:07.091 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:07.091 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:07.091 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:24:07.091 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:07.091 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi032:172.21.15.32=b;smithi130:172.21.15.130=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:07.091 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi032:172.21.15.32=b;smithi130:172.21.15.130=c;count:3 2024-04-25T01:24:07.091 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: Deploying daemon mon.c on smithi130 2024-04-25T01:24:07.091 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:07.091 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: from='client.? 172.21.15.130:0/34666413' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:24:07.091 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:07.092 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:07.092 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 ceph-mon[32702]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T01:24:07.092 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 bash[32657]: 3cdb8931f7c82603268fc913bd01e49c3b7f81de11887c1ba5a213ef879ac0ef 2024-04-25T01:24:07.092 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:06 smithi130 systemd[1]: Started Ceph mon.c for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:24:08.350 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T01:24:08.350 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T01:24:08.595 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:24:11.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:11.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:24:11.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: mon.a calling monitor election 2024-04-25T01:24:11.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:11.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:11.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:11.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:11.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: mon.c calling monitor election 2024-04-25T01:24:11.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:11.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:11.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:11.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:11.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:24:11.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: monmap epoch 2 2024-04-25T01:24:11.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:24:11.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: last_changed 2024-04-25T01:24:06.764034+0000 2024-04-25T01:24:11.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: created 2024-04-25T01:22:20.217202+0000 2024-04-25T01:24:11.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: min_mon_release 19 (squid) 2024-04-25T01:24:11.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: election_strategy: 1 2024-04-25T01:24:11.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-25T01:24:11.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: 1: [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon.c 2024-04-25T01:24:11.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: fsmap 2024-04-25T01:24:11.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:24:11.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: mgrmap e13: x(active, since 51s) 2024-04-25T01:24:11.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: overall HEALTH_OK 2024-04-25T01:24:11.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:11.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:11 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:12.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:12.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:24:12.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: mon.a calling monitor election 2024-04-25T01:24:12.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:12.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:12.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:12.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:12.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: mon.c calling monitor election 2024-04-25T01:24:12.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:12.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:12.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:12.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:12.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:24:12.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: monmap epoch 2 2024-04-25T01:24:12.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:24:12.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: last_changed 2024-04-25T01:24:06.764034+0000 2024-04-25T01:24:12.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: created 2024-04-25T01:22:20.217202+0000 2024-04-25T01:24:12.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: min_mon_release 19 (squid) 2024-04-25T01:24:12.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: election_strategy: 1 2024-04-25T01:24:12.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-25T01:24:12.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: 1: [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon.c 2024-04-25T01:24:12.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: fsmap 2024-04-25T01:24:12.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:24:12.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: mgrmap e13: x(active, since 51s) 2024-04-25T01:24:12.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: overall HEALTH_OK 2024-04-25T01:24:12.067 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:12.067 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:11 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:12.973 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.992+0000 7f4cfe38a700 1 -- 172.21.15.130:0/1315123893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf8100680 msgr2=0x7f4ce0005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:12.973 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.992+0000 7f4cfe38a700 1 --2- 172.21.15.130:0/1315123893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf8100680 0x7f4ce0005610 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8005fa0 tx=0x7f4ce800ffc0 comp rx=0 tx=0).stop 2024-04-25T01:24:12.973 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.992+0000 7f4cfe38a700 1 -- 172.21.15.130:0/1315123893 shutdown_connections 2024-04-25T01:24:12.973 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.992+0000 7f4cfe38a700 1 --2- 172.21.15.130:0/1315123893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf8100680 0x7f4ce0005610 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:12.973 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.992+0000 7f4cfe38a700 1 -- 172.21.15.130:0/1315123893 >> 172.21.15.130:0/1315123893 conn(0x7f4cf80772e0 msgr2=0x7f4cf80776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:12.974 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.992+0000 7f4cfe38a700 1 -- 172.21.15.130:0/1315123893 shutdown_connections 2024-04-25T01:24:12.974 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.992+0000 7f4cfe38a700 1 -- 172.21.15.130:0/1315123893 wait complete. 2024-04-25T01:24:12.975 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.993+0000 7f4cfe38a700 1 Processor -- start 2024-04-25T01:24:12.976 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.993+0000 7f4cfe38a700 1 -- start start 2024-04-25T01:24:12.976 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.993+0000 7f4cfe38a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf8100680 0x7f4cf819f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:12.976 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.993+0000 7f4cfe38a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cf819fc10 0x7f4cf8199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:12.976 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.993+0000 7f4cfe38a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf8109de0 con 0x7f4cf8100680 2024-04-25T01:24:12.976 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.993+0000 7f4cfe38a700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf8109c60 con 0x7f4cf819fc10 2024-04-25T01:24:12.976 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.994+0000 7f4cf77fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cf819fc10 0x7f4cf8199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:12.976 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.994+0000 7f4cf77fe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cf819fc10 0x7f4cf8199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:32818/0 (socket says 172.21.15.130:32818) 2024-04-25T01:24:12.976 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.994+0000 7f4cf77fe700 1 -- 172.21.15.130:0/779578377 learned_addr learned my addr 172.21.15.130:0/779578377 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-25T01:24:12.976 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.994+0000 7f4cf7fff700 1 --2- 172.21.15.130:0/779578377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf8100680 0x7f4cf819f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:12.976 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.994+0000 7f4cf77fe700 1 -- 172.21.15.130:0/779578377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cf819fc10 msgr2=0x7f4cf8199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-25T01:24:12.976 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.994+0000 7f4cf77fe700 1 -- 172.21.15.130:0/779578377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cf819fc10 msgr2=0x7f4cf8199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T01:24:12.976 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.994+0000 7f4cf77fe700 1 --2- 172.21.15.130:0/779578377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cf819fc10 0x7f4cf8199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T01:24:12.976 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.994+0000 7f4cf77fe700 1 --2- 172.21.15.130:0/779578377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cf819fc10 0x7f4cf8199850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:24:12.977 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.994+0000 7f4cf7fff700 1 -- 172.21.15.130:0/779578377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cf819fc10 msgr2=0x7f4cf8199850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:24:12.977 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.994+0000 7f4cf7fff700 1 --2- 172.21.15.130:0/779578377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cf819fc10 0x7f4cf8199850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:12.977 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.994+0000 7f4cf7fff700 1 -- 172.21.15.130:0/779578377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf81a0250 con 0x7f4cf8100680 2024-04-25T01:24:12.977 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.994+0000 7f4cf7fff700 1 --2- 172.21.15.130:0/779578377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf8100680 0x7f4cf819f6d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8000c00 tx=0x7f4ce8004d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:12.977 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.995+0000 7f4cf57fa700 1 -- 172.21.15.130:0/779578377 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f4ce80053b0 con 0x7f4cf8100680 2024-04-25T01:24:12.977 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.995+0000 7f4cf57fa700 1 -- 172.21.15.130:0/779578377 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ce8005550 con 0x7f4cf8100680 2024-04-25T01:24:12.977 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.995+0000 7f4cf57fa700 1 -- 172.21.15.130:0/779578377 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f4ce8011670 con 0x7f4cf8100680 2024-04-25T01:24:12.977 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.995+0000 7f4cfe38a700 1 -- 172.21.15.130:0/779578377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cf8199df0 con 0x7f4cf8100680 2024-04-25T01:24:12.977 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.995+0000 7f4cfe38a700 1 -- 172.21.15.130:0/779578377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cf819a350 con 0x7f4cf8100680 2024-04-25T01:24:12.978 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.996+0000 7f4cf57fa700 1 -- 172.21.15.130:0/779578377 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f4ce80118d0 con 0x7f4cf8100680 2024-04-25T01:24:12.978 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.996+0000 7f4cfe38a700 1 -- 172.21.15.130:0/779578377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cf806a190 con 0x7f4cf8100680 2024-04-25T01:24:12.979 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.996+0000 7f4cf57fa700 1 --2- 172.21.15.130:0/779578377 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4ce4038eb0 0x7f4ce403b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:12.979 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.996+0000 7f4cf57fa700 1 -- 172.21.15.130:0/779578377 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4ce804e830 con 0x7f4cf8100680 2024-04-25T01:24:12.979 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.996+0000 7f4cf77fe700 1 --2- 172.21.15.130:0/779578377 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4ce4038eb0 0x7f4ce403b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:12.979 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:12.997+0000 7f4cf77fe700 1 --2- 172.21.15.130:0/779578377 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4ce4038eb0 0x7f4ce403b370 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009180 tx=0x7f4cec008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:12.982 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:13.002+0000 7f4cf57fa700 1 -- 172.21.15.130:0/779578377 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4ce8022070 con 0x7f4cf8100680 2024-04-25T01:24:13.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:12 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:13.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:12 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:24:13.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:12 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:13.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:12 smithi028 ceph-mon[29390]: Deploying daemon mon.b on smithi032 2024-04-25T01:24:13.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:12 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:13.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:12 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:13.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:12 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:24:13.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:12 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:13.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:12 smithi130 ceph-mon[32702]: Deploying daemon mon.b on smithi032 2024-04-25T01:24:13.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:12 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:13.262 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:13.281+0000 7f4cfe38a700 1 -- 172.21.15.130:0/779578377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f4cf819a630 con 0x7f4cf8100680 2024-04-25T01:24:13.262 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:13.282+0000 7f4cf57fa700 1 -- 172.21.15.130:0/779578377 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f4ce8019bf0 con 0x7f4cf8100680 2024-04-25T01:24:13.263 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-25T01:24:13.263 INFO:teuthology.orchestra.run.smithi130.stdout:{"epoch":2,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","modified":"2024-04-25T01:24:06.764034Z","created":"2024-04-25T01:22:20.217202Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:3300","nonce":0},{"type":"v1","addr":"172.21.15.130:6789","nonce":0}]},"addr":"172.21.15.130:6789/0","public_addr":"172.21.15.130:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T01:24:13.265 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:13.284+0000 7f4cfe38a700 1 -- 172.21.15.130:0/779578377 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4ce4038eb0 msgr2=0x7f4ce403b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:13.265 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:13.284+0000 7f4cfe38a700 1 --2- 172.21.15.130:0/779578377 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4ce4038eb0 0x7f4ce403b370 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009180 tx=0x7f4cec008040 comp rx=0 tx=0).stop 2024-04-25T01:24:13.265 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:13.284+0000 7f4cfe38a700 1 -- 172.21.15.130:0/779578377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf8100680 msgr2=0x7f4cf819f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:13.265 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:13.284+0000 7f4cfe38a700 1 --2- 172.21.15.130:0/779578377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf8100680 0x7f4cf819f6d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8000c00 tx=0x7f4ce8004d60 comp rx=0 tx=0).stop 2024-04-25T01:24:13.265 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:13.284+0000 7f4cfe38a700 1 -- 172.21.15.130:0/779578377 shutdown_connections 2024-04-25T01:24:13.265 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:13.284+0000 7f4cfe38a700 1 --2- 172.21.15.130:0/779578377 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4ce4038eb0 0x7f4ce403b370 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.265 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:13.284+0000 7f4cfe38a700 1 --2- 172.21.15.130:0/779578377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cf8100680 0x7f4cf819f6d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.265 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:13.284+0000 7f4cfe38a700 1 --2- 172.21.15.130:0/779578377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4cf819fc10 0x7f4cf8199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:13.265 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:13.284+0000 7f4cfe38a700 1 -- 172.21.15.130:0/779578377 >> 172.21.15.130:0/779578377 conn(0x7f4cf80772e0 msgr2=0x7f4cf818f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:13.265 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:13.284+0000 7f4cfe38a700 1 -- 172.21.15.130:0/779578377 shutdown_connections 2024-04-25T01:24:13.265 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:13.284+0000 7f4cfe38a700 1 -- 172.21.15.130:0/779578377 wait complete. 2024-04-25T01:24:13.266 INFO:teuthology.orchestra.run.smithi130.stderr:dumped monmap epoch 2 2024-04-25T01:24:13.950 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:13 smithi130 ceph-mon[32702]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:13.950 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:13 smithi130 ceph-mon[32702]: from='client.? 172.21.15.130:0/779578377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:24:14.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:13 smithi028 ceph-mon[29390]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:14.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:13 smithi028 ceph-mon[29390]: from='client.? 172.21.15.130:0/779578377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:24:14.953 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T01:24:14.953 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T01:24:15.198 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:24:15.892 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: mon.b@-1(synchronizing).mds e1 new map 2024-04-25T01:24:15.892 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-25T01:24:15.892 INFO:journalctl@ceph.mon.b.smithi032.stdout: e1 2024-04-25T01:24:15.892 INFO:journalctl@ceph.mon.b.smithi032.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T01:24:15.892 INFO:journalctl@ceph.mon.b.smithi032.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:24:15.892 INFO:journalctl@ceph.mon.b.smithi032.stdout: legacy client fscid: -1 2024-04-25T01:24:15.892 INFO:journalctl@ceph.mon.b.smithi032.stdout: 2024-04-25T01:24:15.892 INFO:journalctl@ceph.mon.b.smithi032.stdout: No filesystems configured 2024-04-25T01:24:15.892 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-25T01:24:15.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:15 smithi130 ceph-mon[32702]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:16.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-25T01:24:16.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-25T01:24:16.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-25T01:24:16.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-25T01:24:16.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-25T01:24:16.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T01:24:16.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:24:16.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:24:16.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:24:16.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:16.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:16.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:24:16.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:16.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi130", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:16.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: Deploying cephadm binary to smithi130 2024-04-25T01:24:16.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi032", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:24:16.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:16.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:16.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-25T01:24:16.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: Updating smithi032:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:24:16.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:16.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:24:16.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:16.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: Updating smithi032:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:24:16.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:16.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: Added host smithi130 2024-04-25T01:24:16.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:16.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:24:16.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:16.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3346046527' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T01:24:16.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:16.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3346046527' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T01:24:16.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:24:16.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi130", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:24:16.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:16.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:16.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-25T01:24:16.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: Updating smithi130:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:24:16.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:24:16.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: Updating smithi130:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:24:16.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:16.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:16.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:16.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:16.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:24:16.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:16.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi032:172.21.15.32=b;smithi130:172.21.15.130=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:16.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi032:172.21.15.32=b;smithi130:172.21.15.130=c;count:3 2024-04-25T01:24:16.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: Deploying daemon mon.c on smithi130 2024-04-25T01:24:16.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:16.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='client.? 172.21.15.130:0/34666413' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:24:16.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:16.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:16.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:16.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:24:16.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: mon.a calling monitor election 2024-04-25T01:24:16.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:16.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:16.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:16.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:16.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: mon.c calling monitor election 2024-04-25T01:24:16.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:16.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:16.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:16.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:16.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:24:16.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: monmap epoch 2 2024-04-25T01:24:16.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:24:16.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: last_changed 2024-04-25T01:24:06.764034+0000 2024-04-25T01:24:16.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: created 2024-04-25T01:22:20.217202+0000 2024-04-25T01:24:16.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: min_mon_release 19 (squid) 2024-04-25T01:24:16.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: election_strategy: 1 2024-04-25T01:24:16.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-25T01:24:16.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: 1: [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon.c 2024-04-25T01:24:16.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: fsmap 2024-04-25T01:24:16.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:24:16.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: mgrmap e13: x(active, since 51s) 2024-04-25T01:24:16.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: overall HEALTH_OK 2024-04-25T01:24:16.246 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.246 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.246 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:16.246 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:24:16.246 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:16.246 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: Deploying daemon mon.b on smithi032 2024-04-25T01:24:16.246 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:16.246 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:16.246 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: from='client.? 172.21.15.130:0/779578377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:24:16.246 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:16.246 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 ceph-mon[32461]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T01:24:16.246 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 bash[32443]: 9ebdfa9017ebf6ce6cd4a7130fa2d450123d4fb625ee7f7885f2ed05d69a5fdd 2024-04-25T01:24:16.246 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:15 smithi032 systemd[1]: Started Ceph mon.b for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:24:16.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:15 smithi028 ceph-mon[29390]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:20.928 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.946+0000 7f5d59435700 1 -- 172.21.15.130:0/2944154672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d400035c0 msgr2=0x7f5d40005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.928 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.946+0000 7f5d59435700 1 --2- 172.21.15.130:0/2944154672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d400035c0 0x7f5d40005a50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5d54079470 tx=0x7f5d4400fe60 comp rx=0 tx=0).stop 2024-04-25T01:24:20.928 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.946+0000 7f5d59435700 1 -- 172.21.15.130:0/2944154672 shutdown_connections 2024-04-25T01:24:20.928 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.946+0000 7f5d59435700 1 --2- 172.21.15.130:0/2944154672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d400035c0 0x7f5d40005a50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.928 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.946+0000 7f5d59435700 1 -- 172.21.15.130:0/2944154672 >> 172.21.15.130:0/2944154672 conn(0x7f5d540772e0 msgr2=0x7f5d540776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:20.928 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.947+0000 7f5d59435700 1 -- 172.21.15.130:0/2944154672 shutdown_connections 2024-04-25T01:24:20.928 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.947+0000 7f5d59435700 1 -- 172.21.15.130:0/2944154672 wait complete. 2024-04-25T01:24:20.928 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.948+0000 7f5d59435700 1 Processor -- start 2024-04-25T01:24:20.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.948+0000 7f5d59435700 1 -- start start 2024-04-25T01:24:20.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.948+0000 7f5d59435700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d400035c0 0x7f5d541965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.948+0000 7f5d59435700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d54196b10 0x7f5d54196f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.948+0000 7f5d59435700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d5419b620 0x7f5d5419bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.948+0000 7f5d59435700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d54108d20 con 0x7f5d400035c0 2024-04-25T01:24:20.930 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.948+0000 7f5d59435700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5d54108ba0 con 0x7f5d54196b10 2024-04-25T01:24:20.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.948+0000 7f5d59435700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f5d54108ea0 con 0x7f5d5419b620 2024-04-25T01:24:20.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.948+0000 7f5d52ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d400035c0 0x7f5d541965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.948+0000 7f5d52ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d400035c0 0x7f5d541965d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.130:42544/0 (socket says 172.21.15.130:42544) 2024-04-25T01:24:20.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.948+0000 7f5d52ffd700 1 -- 172.21.15.130:0/465818910 learned_addr learned my addr 172.21.15.130:0/465818910 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-25T01:24:20.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.948+0000 7f5d537fe700 1 --2- 172.21.15.130:0/465818910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d5419b620 0x7f5d5419bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.949+0000 7f5d537fe700 1 -- 172.21.15.130:0/465818910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d5419b620 msgr2=0x7f5d5419bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-25T01:24:20.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.949+0000 7f5d537fe700 1 -- 172.21.15.130:0/465818910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d5419b620 msgr2=0x7f5d5419bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T01:24:20.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.949+0000 7f5d537fe700 1 --2- 172.21.15.130:0/465818910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d5419b620 0x7f5d5419bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T01:24:20.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.949+0000 7f5d537fe700 1 --2- 172.21.15.130:0/465818910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d5419b620 0x7f5d5419bab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:24:20.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.949+0000 7f5d527fc700 1 --2- 172.21.15.130:0/465818910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d54196b10 0x7f5d54196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.949+0000 7f5d52ffd700 1 -- 172.21.15.130:0/465818910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d5419b620 msgr2=0x7f5d5419bab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:24:20.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.949+0000 7f5d52ffd700 1 --2- 172.21.15.130:0/465818910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d5419b620 0x7f5d5419bab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.949+0000 7f5d52ffd700 1 -- 172.21.15.130:0/465818910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d54196b10 msgr2=0x7f5d54196f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:20.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.949+0000 7f5d52ffd700 1 --2- 172.21.15.130:0/465818910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d54196b10 0x7f5d54196f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:20.931 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.949+0000 7f5d52ffd700 1 -- 172.21.15.130:0/465818910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d5419c270 con 0x7f5d400035c0 2024-04-25T01:24:20.932 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.949+0000 7f5d52ffd700 1 --2- 172.21.15.130:0/465818910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d400035c0 0x7f5d541965d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5d44020040 tx=0x7f5d44005790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.932 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.949+0000 7f5d4bfff700 1 -- 172.21.15.130:0/465818910 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4402b020 con 0x7f5d400035c0 2024-04-25T01:24:20.932 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.949+0000 7f5d4bfff700 1 -- 172.21.15.130:0/465818910 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d440041b0 con 0x7f5d400035c0 2024-04-25T01:24:20.932 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.949+0000 7f5d59435700 1 -- 172.21.15.130:0/465818910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d5419c500 con 0x7f5d400035c0 2024-04-25T01:24:20.932 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.949+0000 7f5d4bfff700 1 -- 172.21.15.130:0/465818910 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d44022040 con 0x7f5d400035c0 2024-04-25T01:24:20.932 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.949+0000 7f5d59435700 1 -- 172.21.15.130:0/465818910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d541ab430 con 0x7f5d400035c0 2024-04-25T01:24:20.932 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.950+0000 7f5d4bfff700 1 -- 172.21.15.130:0/465818910 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f5d4402e020 con 0x7f5d400035c0 2024-04-25T01:24:20.933 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.951+0000 7f5d4bfff700 1 --2- 172.21.15.130:0/465818910 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5d34039580 0x7f5d3403ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:20.933 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.951+0000 7f5d4bfff700 1 -- 172.21.15.130:0/465818910 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5d4404ec70 con 0x7f5d400035c0 2024-04-25T01:24:20.933 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.951+0000 7f5d59435700 1 -- 172.21.15.130:0/465818910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d54063890 con 0x7f5d400035c0 2024-04-25T01:24:20.933 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.951+0000 7f5d527fc700 1 --2- 172.21.15.130:0/465818910 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5d34039580 0x7f5d3403ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:20.933 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.951+0000 7f5d527fc700 1 --2- 172.21.15.130:0/465818910 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5d34039580 0x7f5d3403ba40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c009850 tx=0x7f5d3c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:20.937 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:20.956+0000 7f5d4bfff700 1 -- 172.21.15.130:0/465818910 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5d44035020 con 0x7f5d400035c0 2024-04-25T01:24:21.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: mon.c calling monitor election 2024-04-25T01:24:21.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:24:21.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: mon.a calling monitor election 2024-04-25T01:24:21.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:21.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:21.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:21.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:21.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:21.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:21.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:21.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:21.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:21.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:24:21.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: monmap epoch 3 2024-04-25T01:24:21.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:24:21.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: last_changed 2024-04-25T01:24:15.917750+0000 2024-04-25T01:24:21.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: created 2024-04-25T01:22:20.217202+0000 2024-04-25T01:24:21.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: min_mon_release 19 (squid) 2024-04-25T01:24:21.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: election_strategy: 1 2024-04-25T01:24:21.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-25T01:24:21.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: 1: [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon.c 2024-04-25T01:24:21.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: 2: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.b 2024-04-25T01:24:21.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: fsmap 2024-04-25T01:24:21.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:24:21.214 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: mgrmap e13: x(active, since 60s) 2024-04-25T01:24:21.214 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: overall HEALTH_OK 2024-04-25T01:24:21.214 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:21.214 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:21.230 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:21.249+0000 7f5d59435700 1 -- 172.21.15.130:0/465818910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f5d541ab600 con 0x7f5d400035c0 2024-04-25T01:24:21.230 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:21.250+0000 7f5d4bfff700 1 -- 172.21.15.130:0/465818910 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f5d44010c70 con 0x7f5d400035c0 2024-04-25T01:24:21.231 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-04-25T01:24:21.231 INFO:teuthology.orchestra.run.smithi130.stdout:{"epoch":3,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","modified":"2024-04-25T01:24:15.917750Z","created":"2024-04-25T01:22:20.217202Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:3300","nonce":0},{"type":"v1","addr":"172.21.15.130:6789","nonce":0}]},"addr":"172.21.15.130:6789/0","public_addr":"172.21.15.130:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T01:24:21.233 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:21.252+0000 7f5d59435700 1 -- 172.21.15.130:0/465818910 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5d34039580 msgr2=0x7f5d3403ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.233 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:21.252+0000 7f5d59435700 1 --2- 172.21.15.130:0/465818910 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5d34039580 0x7f5d3403ba40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c009850 tx=0x7f5d3c008040 comp rx=0 tx=0).stop 2024-04-25T01:24:21.233 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:21.252+0000 7f5d59435700 1 -- 172.21.15.130:0/465818910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d400035c0 msgr2=0x7f5d541965d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:21.233 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:21.252+0000 7f5d59435700 1 --2- 172.21.15.130:0/465818910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d400035c0 0x7f5d541965d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5d44020040 tx=0x7f5d44005790 comp rx=0 tx=0).stop 2024-04-25T01:24:21.233 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:21.252+0000 7f5d59435700 1 -- 172.21.15.130:0/465818910 shutdown_connections 2024-04-25T01:24:21.233 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:21.252+0000 7f5d59435700 1 --2- 172.21.15.130:0/465818910 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5d34039580 0x7f5d3403ba40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.233 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:21.252+0000 7f5d59435700 1 --2- 172.21.15.130:0/465818910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5d5419b620 0x7f5d5419bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.233 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:21.252+0000 7f5d59435700 1 --2- 172.21.15.130:0/465818910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d400035c0 0x7f5d541965d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.233 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:21.252+0000 7f5d59435700 1 --2- 172.21.15.130:0/465818910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d54196b10 0x7f5d54196f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:21.234 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:21.252+0000 7f5d59435700 1 -- 172.21.15.130:0/465818910 >> 172.21.15.130:0/465818910 conn(0x7f5d540772e0 msgr2=0x7f5d541028d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:21.234 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:21.252+0000 7f5d59435700 1 -- 172.21.15.130:0/465818910 shutdown_connections 2024-04-25T01:24:21.234 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:21.252+0000 7f5d59435700 1 -- 172.21.15.130:0/465818910 wait complete. 2024-04-25T01:24:21.234 INFO:teuthology.orchestra.run.smithi130.stderr:dumped monmap epoch 3 2024-04-25T01:24:21.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: mon.c calling monitor election 2024-04-25T01:24:21.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:24:21.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: mon.a calling monitor election 2024-04-25T01:24:21.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:21.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:21.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:21.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:21.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:21.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:21.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:21.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:21.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:21.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:24:21.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: monmap epoch 3 2024-04-25T01:24:21.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:24:21.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: last_changed 2024-04-25T01:24:15.917750+0000 2024-04-25T01:24:21.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: created 2024-04-25T01:22:20.217202+0000 2024-04-25T01:24:21.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: min_mon_release 19 (squid) 2024-04-25T01:24:21.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: election_strategy: 1 2024-04-25T01:24:21.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-25T01:24:21.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: 1: [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon.c 2024-04-25T01:24:21.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: 2: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.b 2024-04-25T01:24:21.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: fsmap 2024-04-25T01:24:21.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:24:21.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: mgrmap e13: x(active, since 60s) 2024-04-25T01:24:21.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: overall HEALTH_OK 2024-04-25T01:24:21.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:21.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:21.868 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-25T01:24:21.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-25T01:24:22.125 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:22.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:21 smithi130 ceph-mon[32702]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:22.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:21 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:22.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:21 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:22.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:21 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:22.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:21 smithi130 ceph-mon[32702]: from='client.? 172.21.15.130:0/465818910' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:24:22.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:21 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:22.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:21 smithi028 ceph-mon[29390]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:22.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:21 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:22.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:21 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:22.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:21 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:22.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:21 smithi028 ceph-mon[29390]: from='client.? 172.21.15.130:0/465818910' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:24:22.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:21 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:23.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: mon.c calling monitor election 2024-04-25T01:24:23.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:24:23.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: mon.a calling monitor election 2024-04-25T01:24:23.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:23.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:24:23.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:23.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:23.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:23.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:23.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:23.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:23.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:23.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:24:23.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: monmap epoch 3 2024-04-25T01:24:23.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:24:23.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: last_changed 2024-04-25T01:24:15.917750+0000 2024-04-25T01:24:23.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: created 2024-04-25T01:22:20.217202+0000 2024-04-25T01:24:23.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: min_mon_release 19 (squid) 2024-04-25T01:24:23.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: election_strategy: 1 2024-04-25T01:24:23.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-25T01:24:23.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: 1: [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon.c 2024-04-25T01:24:23.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: 2: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.b 2024-04-25T01:24:23.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: fsmap 2024-04-25T01:24:23.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:24:23.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: mgrmap e13: x(active, since 60s) 2024-04-25T01:24:23.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: overall HEALTH_OK 2024-04-25T01:24:23.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.239 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:23.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:23.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: from='client.? 172.21.15.130:0/465818910' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:24:23.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:22 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:23.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.857+0000 7f7132872700 1 -- 172.21.15.28:0/562892295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f712c101ea0 msgr2=0x7f712c102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:23.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.857+0000 7f7132872700 1 --2- 172.21.15.28:0/562892295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f712c101ea0 0x7f712c102280 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f711c009910 tx=0x7f711c009c20 comp rx=0 tx=0).stop 2024-04-25T01:24:23.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.857+0000 7f7132872700 1 -- 172.21.15.28:0/562892295 shutdown_connections 2024-04-25T01:24:23.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.857+0000 7f7132872700 1 --2- 172.21.15.28:0/562892295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f712c101ea0 0x7f712c102280 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:23.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.857+0000 7f7132872700 1 -- 172.21.15.28:0/562892295 >> 172.21.15.28:0/562892295 conn(0x7f712c0fd770 msgr2=0x7f712c0ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:23.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.857+0000 7f7132872700 1 -- 172.21.15.28:0/562892295 shutdown_connections 2024-04-25T01:24:23.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.857+0000 7f7132872700 1 -- 172.21.15.28:0/562892295 wait complete. 2024-04-25T01:24:23.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.858+0000 7f7132872700 1 Processor -- start 2024-04-25T01:24:23.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.858+0000 7f7132872700 1 -- start start 2024-04-25T01:24:23.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.858+0000 7f7132872700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f712c101ea0 0x7f712c074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:23.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.858+0000 7f7132872700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f712c074a70 0x7f712c074ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:23.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.858+0000 7f7132872700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f712c1a3bb0 0x7f712c1a5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:23.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.858+0000 7f7132872700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f712c10c270 con 0x7f712c074a70 2024-04-25T01:24:23.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.858+0000 7f7132872700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f712c10c0f0 con 0x7f712c101ea0 2024-04-25T01:24:23.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.858+0000 7f7132872700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f712c10c3f0 con 0x7f712c1a3bb0 2024-04-25T01:24:23.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.858+0000 7f712b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f712c074a70 0x7f712c074ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:23.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.858+0000 7f712b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f712c074a70 0x7f712c074ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36288/0 (socket says 172.21.15.28:36288) 2024-04-25T01:24:23.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.858+0000 7f712b7fe700 1 -- 172.21.15.28:0/743248777 learned_addr learned my addr 172.21.15.28:0/743248777 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:24:23.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.858+0000 7f712bfff700 1 --2- 172.21.15.28:0/743248777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f712c101ea0 0x7f712c074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.859+0000 7f7130e0f700 1 --2- 172.21.15.28:0/743248777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f712c1a3bb0 0x7f712c1a5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.859+0000 7f712b7fe700 1 -- 172.21.15.28:0/743248777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f712c1a3bb0 msgr2=0x7f712c1a5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.859+0000 7f712b7fe700 1 --2- 172.21.15.28:0/743248777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f712c1a3bb0 0x7f712c1a5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.859+0000 7f712b7fe700 1 -- 172.21.15.28:0/743248777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f712c101ea0 msgr2=0x7f712c074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.859+0000 7f712b7fe700 1 --2- 172.21.15.28:0/743248777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f712c101ea0 0x7f712c074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.859+0000 7f712b7fe700 1 -- 172.21.15.28:0/743248777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f712c1a66d0 con 0x7f712c074a70 2024-04-25T01:24:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.859+0000 7f7130e0f700 1 --2- 172.21.15.28:0/743248777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f712c1a3bb0 0x7f712c1a5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:24:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.859+0000 7f712b7fe700 1 --2- 172.21.15.28:0/743248777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f712c074a70 0x7f712c074ef0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f712000b780 tx=0x7f712000ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.859+0000 7f71297fa700 1 -- 172.21.15.28:0/743248777 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7120004230 con 0x7f712c074a70 2024-04-25T01:24:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.859+0000 7f71297fa700 1 -- 172.21.15.28:0/743248777 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71200043d0 con 0x7f712c074a70 2024-04-25T01:24:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.859+0000 7f71297fa700 1 -- 172.21.15.28:0/743248777 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7120011c90 con 0x7f712c074a70 2024-04-25T01:24:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.859+0000 7f7132872700 1 -- 172.21.15.28:0/743248777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f712c1a69c0 con 0x7f712c074a70 2024-04-25T01:24:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.859+0000 7f7132872700 1 -- 172.21.15.28:0/743248777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f712c1a6eb0 con 0x7f712c074a70 2024-04-25T01:24:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.860+0000 7f71297fa700 1 -- 172.21.15.28:0/743248777 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f712001a400 con 0x7f712c074a70 2024-04-25T01:24:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.860+0000 7f71297fa700 1 --2- 172.21.15.28:0/743248777 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f7114039580 0x7f711403ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:23.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.860+0000 7f7132872700 1 -- 172.21.15.28:0/743248777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f712c063890 con 0x7f712c074a70 2024-04-25T01:24:23.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.860+0000 7f71297fa700 1 -- 172.21.15.28:0/743248777 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f712004bcc0 con 0x7f712c074a70 2024-04-25T01:24:23.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.860+0000 7f712bfff700 1 --2- 172.21.15.28:0/743248777 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f7114039580 0x7f711403ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:23.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.861+0000 7f712bfff700 1 --2- 172.21.15.28:0/743248777 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f7114039580 0x7f711403ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f711c006010 tx=0x7f711c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:23.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:23.866+0000 7f71297fa700 1 -- 172.21.15.28:0/743248777 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7120018750 con 0x7f712c074a70 2024-04-25T01:24:23.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: mon.b calling monitor election 2024-04-25T01:24:23.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:23.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-25T01:24:23.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-25T01:24:23.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-25T01:24:23.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: Updating smithi032:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:24:23.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: Updating smithi130:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:24:23.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: Updating smithi028:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:24:23.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: mon.b calling monitor election 2024-04-25T01:24:23.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: mon.a calling monitor election 2024-04-25T01:24:23.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: mon.c calling monitor election 2024-04-25T01:24:23.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T01:24:23.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: monmap epoch 3 2024-04-25T01:24:23.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:24:23.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: last_changed 2024-04-25T01:24:15.917750+0000 2024-04-25T01:24:23.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: created 2024-04-25T01:22:20.217202+0000 2024-04-25T01:24:23.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: min_mon_release 19 (squid) 2024-04-25T01:24:23.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: election_strategy: 1 2024-04-25T01:24:23.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-25T01:24:23.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: 1: [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon.c 2024-04-25T01:24:23.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: 2: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.b 2024-04-25T01:24:23.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: fsmap 2024-04-25T01:24:23.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:24:23.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: mgrmap e13: x(active, since 62s) 2024-04-25T01:24:23.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: overall HEALTH_OK 2024-04-25T01:24:23.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:24:23.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:24:23.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:23.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: mon.b calling monitor election 2024-04-25T01:24:23.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:23.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-25T01:24:23.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-25T01:24:23.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-25T01:24:23.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: Updating smithi032:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:24:23.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: Updating smithi130:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:24:23.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: Updating smithi028:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:24:23.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: mon.b calling monitor election 2024-04-25T01:24:23.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: mon.a calling monitor election 2024-04-25T01:24:23.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: mon.c calling monitor election 2024-04-25T01:24:23.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T01:24:23.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: monmap epoch 3 2024-04-25T01:24:23.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:24:23.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: last_changed 2024-04-25T01:24:15.917750+0000 2024-04-25T01:24:23.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: created 2024-04-25T01:22:20.217202+0000 2024-04-25T01:24:23.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: min_mon_release 19 (squid) 2024-04-25T01:24:23.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: election_strategy: 1 2024-04-25T01:24:23.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-25T01:24:23.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: 1: [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon.c 2024-04-25T01:24:23.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: 2: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.b 2024-04-25T01:24:23.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: fsmap 2024-04-25T01:24:23.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:24:23.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: mgrmap e13: x(active, since 62s) 2024-04-25T01:24:23.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: overall HEALTH_OK 2024-04-25T01:24:23.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:24:23.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:24:23.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:23.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: mon.b calling monitor election 2024-04-25T01:24:23.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:23.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-25T01:24:23.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-25T01:24:23.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-25T01:24:23.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: Updating smithi032:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:24:23.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: Updating smithi130:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:24:23.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: Updating smithi028:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:24:23.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: mon.b calling monitor election 2024-04-25T01:24:23.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: mon.a calling monitor election 2024-04-25T01:24:23.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: mon.c calling monitor election 2024-04-25T01:24:23.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T01:24:23.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: monmap epoch 3 2024-04-25T01:24:23.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:24:23.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: last_changed 2024-04-25T01:24:15.917750+0000 2024-04-25T01:24:23.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: created 2024-04-25T01:22:20.217202+0000 2024-04-25T01:24:23.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: min_mon_release 19 (squid) 2024-04-25T01:24:23.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: election_strategy: 1 2024-04-25T01:24:23.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-25T01:24:23.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: 1: [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] mon.c 2024-04-25T01:24:23.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: 2: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.b 2024-04-25T01:24:23.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: fsmap 2024-04-25T01:24:23.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:24:23.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: mgrmap e13: x(active, since 62s) 2024-04-25T01:24:23.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: overall HEALTH_OK 2024-04-25T01:24:23.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.989 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.989 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.989 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.989 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.989 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.989 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.989 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:23.989 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:24:23.989 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:24:23.989 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:24.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:24.080+0000 7f7132872700 1 -- 172.21.15.28:0/743248777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f712c1a71c0 con 0x7f712c074a70 2024-04-25T01:24:24.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:24.081+0000 7f71297fa700 1 -- 172.21.15.28:0/743248777 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f7120003ca0 con 0x7f712c074a70 2024-04-25T01:24:24.061 INFO:teuthology.orchestra.run.smithi028.stdout:# minimal ceph.conf for 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:24:24.061 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-04-25T01:24:24.062 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:24:24.062 INFO:teuthology.orchestra.run.smithi028.stdout: mon_host = [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] 2024-04-25T01:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:24.083+0000 7f7132872700 1 -- 172.21.15.28:0/743248777 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f7114039580 msgr2=0x7f711403ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:24.083+0000 7f7132872700 1 --2- 172.21.15.28:0/743248777 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f7114039580 0x7f711403ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f711c006010 tx=0x7f711c0058e0 comp rx=0 tx=0).stop 2024-04-25T01:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:24.083+0000 7f7132872700 1 -- 172.21.15.28:0/743248777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f712c074a70 msgr2=0x7f712c074ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:24.083+0000 7f7132872700 1 --2- 172.21.15.28:0/743248777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f712c074a70 0x7f712c074ef0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f712000b780 tx=0x7f712000ba90 comp rx=0 tx=0).stop 2024-04-25T01:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:24.083+0000 7f7132872700 1 -- 172.21.15.28:0/743248777 shutdown_connections 2024-04-25T01:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:24.083+0000 7f7132872700 1 --2- 172.21.15.28:0/743248777 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f7114039580 0x7f711403ba40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:24.083+0000 7f7132872700 1 --2- 172.21.15.28:0/743248777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f712c1a3bb0 0x7f712c1a5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:24.083+0000 7f7132872700 1 --2- 172.21.15.28:0/743248777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f712c101ea0 0x7f712c074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:24.083+0000 7f7132872700 1 --2- 172.21.15.28:0/743248777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f712c074a70 0x7f712c074ef0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:24.083+0000 7f7132872700 1 -- 172.21.15.28:0/743248777 >> 172.21.15.28:0/743248777 conn(0x7f712c0fd770 msgr2=0x7f712c103870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:24.083+0000 7f7132872700 1 -- 172.21.15.28:0/743248777 shutdown_connections 2024-04-25T01:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:24.083+0000 7f7132872700 1 -- 172.21.15.28:0/743248777 wait complete. 2024-04-25T01:24:24.908 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:24 smithi028 ceph-mon[29390]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T01:24:24.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:24 smithi028 ceph-mon[29390]: Reconfiguring daemon mon.a on smithi028 2024-04-25T01:24:24.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:24 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:24.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:24 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/743248777' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:24.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:24 smithi130 ceph-mon[32702]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T01:24:24.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:24 smithi130 ceph-mon[32702]: Reconfiguring daemon mon.a on smithi028 2024-04-25T01:24:24.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:24 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:24.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:24 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/743248777' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:24.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:24 smithi032 ceph-mon[32461]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T01:24:24.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:24 smithi032 ceph-mon[32461]: Reconfiguring daemon mon.a on smithi028 2024-04-25T01:24:24.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:24 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:24:24.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:24 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/743248777' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:25.035 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-25T01:24:25.035 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:24:25.035 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T01:24:25.062 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:24:25.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:24:25.127 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-25T01:24:25.127 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T01:24:25.160 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-25T01:24:25.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:24:25.229 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-25T01:24:25.229 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T01:24:25.260 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-25T01:24:25.260 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:24:25.327 INFO:tasks.cephadm:Adding mgr.x on smithi028 2024-04-25T01:24:25.327 INFO:tasks.cephadm:Adding mgr.y on smithi032 2024-04-25T01:24:25.327 INFO:tasks.cephadm:Adding mgr.z on smithi130 2024-04-25T01:24:25.327 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi028=x;smithi032=y;smithi130=z' 2024-04-25T01:24:25.601 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:24:25.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:25 smithi028 ceph-mon[29390]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:25.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:25 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:25.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:25 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:25.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:25 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:25.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:25 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:24:25.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:25 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:24:25.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:25 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:25.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:25 smithi130 ceph-mon[32702]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:25.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:25 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:25.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:25 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:25.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:25 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:25.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:25 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:24:25.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:25 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:24:25.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:25 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:25.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:25 smithi032 ceph-mon[32461]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:25.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:25 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:25.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:25 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:25.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:25 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:25.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:25 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:24:25.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:25 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:24:25.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:25 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:26.911 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:26 smithi130 ceph-mon[32702]: Reconfiguring mon.b (monmap changed)... 2024-04-25T01:24:26.911 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:26 smithi130 ceph-mon[32702]: Reconfiguring daemon mon.b on smithi032 2024-04-25T01:24:26.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:26 smithi032 ceph-mon[32461]: Reconfiguring mon.b (monmap changed)... 2024-04-25T01:24:26.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:26 smithi032 ceph-mon[32461]: Reconfiguring daemon mon.b on smithi032 2024-04-25T01:24:27.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:26 smithi028 ceph-mon[29390]: Reconfiguring mon.b (monmap changed)... 2024-04-25T01:24:27.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:26 smithi028 ceph-mon[29390]: Reconfiguring daemon mon.b on smithi032 2024-04-25T01:24:27.245 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.263+0000 7f0e3ec30700 1 -- 172.21.15.130:0/3627954361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e381000a0 msgr2=0x7f0e381004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:27.246 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.263+0000 7f0e3ec30700 1 --2- 172.21.15.130:0/3627954361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e381000a0 0x7f0e381004c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c009910 tx=0x7f0e2c009c20 comp rx=0 tx=0).stop 2024-04-25T01:24:27.246 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.263+0000 7f0e3ec30700 1 -- 172.21.15.130:0/3627954361 shutdown_connections 2024-04-25T01:24:27.246 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.263+0000 7f0e3ec30700 1 --2- 172.21.15.130:0/3627954361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e381000a0 0x7f0e381004c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:27.246 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.263+0000 7f0e3ec30700 1 -- 172.21.15.130:0/3627954361 >> 172.21.15.130:0/3627954361 conn(0x7f0e380fba20 msgr2=0x7f0e380fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:27.246 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.263+0000 7f0e3ec30700 1 -- 172.21.15.130:0/3627954361 shutdown_connections 2024-04-25T01:24:27.246 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.263+0000 7f0e3ec30700 1 -- 172.21.15.130:0/3627954361 wait complete. 2024-04-25T01:24:27.246 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e3ec30700 1 Processor -- start 2024-04-25T01:24:27.246 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e3ec30700 1 -- start start 2024-04-25T01:24:27.246 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e3ec30700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e381000a0 0x7f0e38197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:27.246 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e3ec30700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e38197b60 0x7f0e3819cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:27.246 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e3ec30700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e38197fe0 0x7f0e3819f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:27.247 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e3ec30700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e38107e20 con 0x7f0e38197b60 2024-04-25T01:24:27.247 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e3ec30700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0e38107ca0 con 0x7f0e38197fe0 2024-04-25T01:24:27.247 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e3ec30700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0e38107fa0 con 0x7f0e381000a0 2024-04-25T01:24:27.247 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e3c9cc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e381000a0 0x7f0e38197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:27.247 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e3c9cc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e381000a0 0x7f0e38197620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:39308/0 (socket says 172.21.15.130:39308) 2024-04-25T01:24:27.247 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e3c9cc700 1 -- 172.21.15.130:0/1637661878 learned_addr learned my addr 172.21.15.130:0/1637661878 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-25T01:24:27.247 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e37fff700 1 --2- 172.21.15.130:0/1637661878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e38197b60 0x7f0e3819cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:27.247 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e3c9cc700 1 -- 172.21.15.130:0/1637661878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e38197fe0 msgr2=0x7f0e3819f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:27.247 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e3d1cd700 1 --2- 172.21.15.130:0/1637661878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e38197fe0 0x7f0e3819f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:27.247 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e3c9cc700 1 --2- 172.21.15.130:0/1637661878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e38197fe0 0x7f0e3819f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:27.247 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e3c9cc700 1 -- 172.21.15.130:0/1637661878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e38197b60 msgr2=0x7f0e3819cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:27.247 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e3c9cc700 1 --2- 172.21.15.130:0/1637661878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e38197b60 0x7f0e3819cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:27.248 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e3c9cc700 1 -- 172.21.15.130:0/1637661878 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e3819fa30 con 0x7f0e381000a0 2024-04-25T01:24:27.248 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e37fff700 1 --2- 172.21.15.130:0/1637661878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e38197b60 0x7f0e3819cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:24:27.248 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e3d1cd700 1 --2- 172.21.15.130:0/1637661878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e38197fe0 0x7f0e3819f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:24:27.248 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.264+0000 7f0e3c9cc700 1 --2- 172.21.15.130:0/1637661878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e381000a0 0x7f0e38197620 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00be40 tx=0x7f0e2c00be70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:27.248 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.265+0000 7f0e35ffb700 1 -- 172.21.15.130:0/1637661878 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c027070 con 0x7f0e381000a0 2024-04-25T01:24:27.248 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.265+0000 7f0e35ffb700 1 -- 172.21.15.130:0/1637661878 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e2c005da0 con 0x7f0e381000a0 2024-04-25T01:24:27.248 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.265+0000 7f0e3ec30700 1 -- 172.21.15.130:0/1637661878 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e3819fcc0 con 0x7f0e381000a0 2024-04-25T01:24:27.248 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.265+0000 7f0e35ffb700 1 -- 172.21.15.130:0/1637661878 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c00f460 con 0x7f0e381000a0 2024-04-25T01:24:27.248 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.265+0000 7f0e3ec30700 1 -- 172.21.15.130:0/1637661878 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e381a01a0 con 0x7f0e381000a0 2024-04-25T01:24:27.249 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.266+0000 7f0e35ffb700 1 -- 172.21.15.130:0/1637661878 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f0e2c02a070 con 0x7f0e381000a0 2024-04-25T01:24:27.249 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.266+0000 7f0e3ec30700 1 -- 172.21.15.130:0/1637661878 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e38063890 con 0x7f0e381000a0 2024-04-25T01:24:27.249 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.266+0000 7f0e35ffb700 1 --2- 172.21.15.130:0/1637661878 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f0e28039580 0x7f0e2803ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:27.249 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.266+0000 7f0e35ffb700 1 -- 172.21.15.130:0/1637661878 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f0e2c04d410 con 0x7f0e381000a0 2024-04-25T01:24:27.249 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.266+0000 7f0e37fff700 1 --2- 172.21.15.130:0/1637661878 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f0e28039580 0x7f0e2803ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:27.249 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.267+0000 7f0e37fff700 1 --2- 172.21.15.130:0/1637661878 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f0e28039580 0x7f0e2803ba40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009ec0 tx=0x7f0e24009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:27.252 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.271+0000 7f0e35ffb700 1 -- 172.21.15.130:0/1637661878 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0e2c013070 con 0x7f0e381000a0 2024-04-25T01:24:27.468 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.487+0000 7f0e3ec30700 1 -- 172.21.15.130:0/1637661878 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi032=y;smithi130=z", "target": ["mon-mgr", ""]}) -- 0x7f0e38062700 con 0x7f0e28039580 2024-04-25T01:24:27.474 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.493+0000 7f0e35ffb700 1 -- 172.21.15.130:0/1637661878 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0e38062700 con 0x7f0e28039580 2024-04-25T01:24:27.474 INFO:teuthology.orchestra.run.smithi130.stdout:Scheduled mgr update... 2024-04-25T01:24:27.476 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.495+0000 7f0e3ec30700 1 -- 172.21.15.130:0/1637661878 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f0e28039580 msgr2=0x7f0e2803ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:27.476 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.495+0000 7f0e3ec30700 1 --2- 172.21.15.130:0/1637661878 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f0e28039580 0x7f0e2803ba40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009ec0 tx=0x7f0e24009210 comp rx=0 tx=0).stop 2024-04-25T01:24:27.476 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.495+0000 7f0e3ec30700 1 -- 172.21.15.130:0/1637661878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e381000a0 msgr2=0x7f0e38197620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:27.476 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.495+0000 7f0e3ec30700 1 --2- 172.21.15.130:0/1637661878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e381000a0 0x7f0e38197620 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00be40 tx=0x7f0e2c00be70 comp rx=0 tx=0).stop 2024-04-25T01:24:27.476 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.495+0000 7f0e3ec30700 1 -- 172.21.15.130:0/1637661878 shutdown_connections 2024-04-25T01:24:27.476 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.495+0000 7f0e3ec30700 1 --2- 172.21.15.130:0/1637661878 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f0e28039580 0x7f0e2803ba40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:27.476 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.495+0000 7f0e3ec30700 1 --2- 172.21.15.130:0/1637661878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e38197fe0 0x7f0e3819f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:27.476 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.495+0000 7f0e3ec30700 1 --2- 172.21.15.130:0/1637661878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e381000a0 0x7f0e38197620 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:27.476 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.495+0000 7f0e3ec30700 1 --2- 172.21.15.130:0/1637661878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e38197b60 0x7f0e3819cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:27.476 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.495+0000 7f0e3ec30700 1 -- 172.21.15.130:0/1637661878 >> 172.21.15.130:0/1637661878 conn(0x7f0e380fba20 msgr2=0x7f0e380fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:27.477 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.495+0000 7f0e3ec30700 1 -- 172.21.15.130:0/1637661878 shutdown_connections 2024-04-25T01:24:27.477 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:24:27.495+0000 7f0e3ec30700 1 -- 172.21.15.130:0/1637661878 wait complete. 2024-04-25T01:24:27.841 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:27 smithi130 ceph-mon[32702]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:27.841 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:27 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:27.841 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:27 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:27.841 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:27 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:24:27.842 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:27 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:24:27.842 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:27 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:27.842 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:27 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:27.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:27 smithi032 ceph-mon[32461]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:27.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:27 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:27.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:27 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:27.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:27 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:24:27.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:27 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:24:27.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:27 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:27.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:27 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:28.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:27 smithi028 ceph-mon[29390]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:28.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:27 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:28.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:27 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:28.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:27 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:24:28.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:27 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:24:28.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:27 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:28.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:27 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:28.127 DEBUG:teuthology.orchestra.run.smithi032:mgr.y> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mgr.y.service 2024-04-25T01:24:28.129 DEBUG:teuthology.orchestra.run.smithi130:mgr.z> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mgr.z.service 2024-04-25T01:24:28.132 INFO:tasks.cephadm:Deploying OSDs... 2024-04-25T01:24:28.132 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:24:28.132 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T01:24:28.152 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T01:24:28.152 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-04-25T01:24:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T01:24:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:24:28.211 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-25T01:24:28.211 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:24:28.211 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:24:28.211 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-25 01:23:30.226513350 +0000 2024-04-25T01:24:28.211 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-25 01:18:15.609669890 +0000 2024-04-25T01:24:28.211 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-25 01:18:15.609669890 +0000 2024-04-25T01:24:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-25 01:18:15.609669890 +0000 2024-04-25T01:24:28.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T01:24:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-25T01:24:28.279 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-25T01:24:28.279 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000128576 s, 4.0 MB/s 2024-04-25T01:24:28.280 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T01:24:28.340 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-04-25T01:24:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T01:24:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:24:28.399 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T01:24:28.399 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:24:28.399 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:24:28.399 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-25 01:23:30.262512337 +0000 2024-04-25T01:24:28.399 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-25 01:18:15.940660542 +0000 2024-04-25T01:24:28.399 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-25 01:18:15.940660542 +0000 2024-04-25T01:24:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-25 01:18:15.940660542 +0000 2024-04-25T01:24:28.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T01:24:28.466 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-25T01:24:28.466 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-25T01:24:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000127662 s, 4.0 MB/s 2024-04-25T01:24:28.468 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T01:24:28.527 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-04-25T01:24:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T01:24:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:24:28.585 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-25T01:24:28.585 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:24:28.585 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:24:28.585 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-25 01:23:30.297511353 +0000 2024-04-25T01:24:28.585 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-25 01:18:16.272651165 +0000 2024-04-25T01:24:28.585 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-25 01:18:16.272651165 +0000 2024-04-25T01:24:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-25 01:18:16.272651165 +0000 2024-04-25T01:24:28.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T01:24:28.652 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-25T01:24:28.652 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-25T01:24:28.652 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.00010305 s, 5.0 MB/s 2024-04-25T01:24:28.654 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T01:24:28.712 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-04-25T01:24:28.769 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T01:24:28.769 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:24:28.769 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-25T01:24:28.769 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:24:28.769 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:24:28.769 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-25 01:23:30.332510368 +0000 2024-04-25T01:24:28.769 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-25 01:18:16.605641761 +0000 2024-04-25T01:24:28.770 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-25 01:18:16.605641761 +0000 2024-04-25T01:24:28.770 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-25 01:18:16.605641761 +0000 2024-04-25T01:24:28.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T01:24:28.834 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-25T01:24:28.835 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-25T01:24:28.835 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000118966 s, 4.3 MB/s 2024-04-25T01:24:28.836 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T01:24:28.893 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-25T01:24:28.893 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T01:24:28.906 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:28 smithi130 ceph-mon[32702]: Reconfiguring mon.c (monmap changed)... 2024-04-25T01:24:28.906 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:28 smithi130 ceph-mon[32702]: Reconfiguring daemon mon.c on smithi130 2024-04-25T01:24:28.906 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:28 smithi130 ceph-mon[32702]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi032=y;smithi130=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:28.906 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:28 smithi130 ceph-mon[32702]: Saving service mgr spec with placement smithi028=x;smithi032=y;smithi130=z;count:3 2024-04-25T01:24:28.913 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T01:24:28.913 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_1 2024-04-25T01:24:28.971 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T01:24:28.972 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:24:28.972 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-25T01:24:28.972 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:24:28.972 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:24:28.972 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-25 01:24:15.344261061 +0000 2024-04-25T01:24:28.972 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-25 01:18:35.603351766 +0000 2024-04-25T01:24:28.972 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-25 01:18:35.603351766 +0000 2024-04-25T01:24:28.972 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-04-25 01:18:35.603351766 +0000 2024-04-25T01:24:28.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T01:24:28.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:28 smithi032 ceph-mon[32461]: Reconfiguring mon.c (monmap changed)... 2024-04-25T01:24:28.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:28 smithi032 ceph-mon[32461]: Reconfiguring daemon mon.c on smithi130 2024-04-25T01:24:28.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:28 smithi032 ceph-mon[32461]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi032=y;smithi130=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:28.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:28 smithi032 ceph-mon[32461]: Saving service mgr spec with placement smithi028=x;smithi032=y;smithi130=z;count:3 2024-04-25T01:24:29.013 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-25T01:24:29.013 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-25T01:24:29.013 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000128927 s, 4.0 MB/s 2024-04-25T01:24:29.014 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T01:24:29.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:28 smithi028 ceph-mon[29390]: Reconfiguring mon.c (monmap changed)... 2024-04-25T01:24:29.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:28 smithi028 ceph-mon[29390]: Reconfiguring daemon mon.c on smithi130 2024-04-25T01:24:29.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:28 smithi028 ceph-mon[29390]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi032=y;smithi130=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:29.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:28 smithi028 ceph-mon[29390]: Saving service mgr spec with placement smithi028=x;smithi032=y;smithi130=z;count:3 2024-04-25T01:24:29.073 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_2 2024-04-25T01:24:29.130 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T01:24:29.130 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:24:29.130 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-25T01:24:29.130 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:24:29.130 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:24:29.130 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-25 01:24:15.345261041 +0000 2024-04-25T01:24:29.130 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-25 01:18:35.934345133 +0000 2024-04-25T01:24:29.130 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-25 01:18:35.934345133 +0000 2024-04-25T01:24:29.130 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-04-25 01:18:35.934345133 +0000 2024-04-25T01:24:29.131 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T01:24:29.195 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-25T01:24:29.195 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-25T01:24:29.195 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000118832 s, 4.3 MB/s 2024-04-25T01:24:29.197 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T01:24:29.255 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_3 2024-04-25T01:24:29.312 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T01:24:29.312 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:24:29.312 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-25T01:24:29.312 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:24:29.312 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:24:29.312 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-25 01:24:15.345261041 +0000 2024-04-25T01:24:29.312 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-25 01:18:36.277338259 +0000 2024-04-25T01:24:29.312 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-25 01:18:36.277338259 +0000 2024-04-25T01:24:29.312 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-04-25 01:18:36.277338259 +0000 2024-04-25T01:24:29.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T01:24:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-25T01:24:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-25T01:24:29.377 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000114026 s, 4.5 MB/s 2024-04-25T01:24:29.379 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T01:24:29.436 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_4 2024-04-25T01:24:29.492 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T01:24:29.492 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:24:29.492 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-25T01:24:29.492 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:24:29.492 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:24:29.492 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-25 01:24:15.346261021 +0000 2024-04-25T01:24:29.492 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-25 01:18:36.620331386 +0000 2024-04-25T01:24:29.492 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-25 01:18:36.620331386 +0000 2024-04-25T01:24:29.493 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-04-25 01:18:36.620331386 +0000 2024-04-25T01:24:29.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T01:24:29.557 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-25T01:24:29.557 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-25T01:24:29.557 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000115258 s, 4.4 MB/s 2024-04-25T01:24:29.558 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T01:24:29.615 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-04-25T01:24:29.615 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T01:24:29.633 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T01:24:29.634 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_1 2024-04-25T01:24:29.691 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T01:24:29.691 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:24:29.691 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-25T01:24:29.691 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:24:29.691 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:24:29.691 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-04-25 01:24:05.151744747 +0000 2024-04-25T01:24:29.691 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-04-25 01:18:29.340626066 +0000 2024-04-25T01:24:29.691 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-04-25 01:18:29.340626066 +0000 2024-04-25T01:24:29.692 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-04-25 01:18:29.340626066 +0000 2024-04-25T01:24:29.692 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T01:24:29.756 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-04-25T01:24:29.756 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-04-25T01:24:29.757 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000279168 s, 1.8 MB/s 2024-04-25T01:24:29.758 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T01:24:29.815 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_2 2024-04-25T01:24:29.871 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T01:24:29.871 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:24:29.871 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T01:24:29.871 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:24:29.871 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:24:29.871 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-04-25 01:24:05.151744747 +0000 2024-04-25T01:24:29.871 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-04-25 01:18:29.677619917 +0000 2024-04-25T01:24:29.871 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-04-25 01:18:29.677619917 +0000 2024-04-25T01:24:29.871 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-04-25 01:18:29.677619917 +0000 2024-04-25T01:24:29.872 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T01:24:29.932 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:29 smithi130 ceph-mon[32702]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:29.932 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:29 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:29.932 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:29 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:29.933 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:29 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:29.935 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-04-25T01:24:29.935 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-04-25T01:24:29.935 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000185287 s, 2.8 MB/s 2024-04-25T01:24:29.936 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T01:24:29.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:29 smithi032 ceph-mon[32461]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:29.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:29 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:29.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:29 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:29.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:29 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:29.991 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_3 2024-04-25T01:24:30.047 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T01:24:30.047 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:24:30.047 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-25T01:24:30.047 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:24:30.047 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:24:30.047 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-04-25 01:24:05.152744729 +0000 2024-04-25T01:24:30.047 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-04-25 01:18:30.017613713 +0000 2024-04-25T01:24:30.047 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-04-25 01:18:30.017613713 +0000 2024-04-25T01:24:30.047 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-04-25 01:18:30.017613713 +0000 2024-04-25T01:24:30.048 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T01:24:30.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:29 smithi028 ceph-mon[29390]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:30.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:29 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:30.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:29 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:30.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:29 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:30.110 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-04-25T01:24:30.110 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-04-25T01:24:30.110 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000182872 s, 2.8 MB/s 2024-04-25T01:24:30.111 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T01:24:30.166 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_4 2024-04-25T01:24:30.223 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T01:24:30.223 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:24:30.223 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-25T01:24:30.223 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:24:30.223 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:24:30.223 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-04-25 01:24:05.152744729 +0000 2024-04-25T01:24:30.223 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-04-25 01:18:30.368607309 +0000 2024-04-25T01:24:30.223 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-04-25 01:18:30.368607309 +0000 2024-04-25T01:24:30.223 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-04-25 01:18:30.368607309 +0000 2024-04-25T01:24:30.224 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T01:24:30.289 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-04-25T01:24:30.289 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-04-25T01:24:30.289 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000178238 s, 2.9 MB/s 2024-04-25T01:24:30.290 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T01:24:30.346 INFO:tasks.cephadm:Deploying osd.0 on smithi028 with /dev/vg_nvme/lv_4... 2024-04-25T01:24:30.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T01:24:30.591 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:30.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:30 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:30.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:30 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:30.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:30 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:30.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:30 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:24:30.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:30 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:24:30.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:30 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:24:30.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:30 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:30.957 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:30 smithi028 ceph-mon[29390]: Deploying daemon mgr.z on smithi130 2024-04-25T01:24:30.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:30 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:30.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:30 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:30.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:30 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:30.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:30 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:24:30.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:30 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:24:30.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:30 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:24:30.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:30 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:30.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:30 smithi130 ceph-mon[32702]: Deploying daemon mgr.z on smithi130 2024-04-25T01:24:30.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:30 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:30.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:30 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:30.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:30 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:30.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:30 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:24:30.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:30 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:24:30.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:30 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:24:30.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:30 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:30.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:30 smithi032 ceph-mon[32461]: Deploying daemon mgr.z on smithi130 2024-04-25T01:24:31.825 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:31 smithi130 systemd[1]: Starting Ceph mgr.z for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:24:32.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:31 smithi028 ceph-mon[29390]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:32.135 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:31 smithi130 ceph-mon[32702]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:32.135 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:31 smithi130 podman[34553]: 2024-04-25 01:24:31.90103083 +0000 UTC m=+0.017844254 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:24:32.135 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:32 smithi130 podman[34553]: 2024-04-25 01:24:32.042372952 +0000 UTC m=+0.159186412 container create 0e6b2e060ce6cbc264438c176c1ecbeac6b206e41a7cb2e568213e6b5e9db962 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-d2c45d9, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-25T01:24:32.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:31 smithi032 ceph-mon[32461]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:32.446 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:32 smithi130 podman[34553]: 2024-04-25 01:24:32.311468934 +0000 UTC m=+0.428282348 container init 0e6b2e060ce6cbc264438c176c1ecbeac6b206e41a7cb2e568213e6b5e9db962 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=squid-d2c45d9) 2024-04-25T01:24:32.447 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:32 smithi130 podman[34553]: 2024-04-25 01:24:32.314453693 +0000 UTC m=+0.431267109 container start 0e6b2e060ce6cbc264438c176c1ecbeac6b206e41a7cb2e568213e6b5e9db962 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-25T01:24:32.447 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:32 smithi130 ceph-mgr[34572]: -- 172.21.15.130:0/2778876357 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d1e821a5a0 con 0x55d1e81eec00 2024-04-25T01:24:32.447 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:32 smithi130 bash[34553]: 0e6b2e060ce6cbc264438c176c1ecbeac6b206e41a7cb2e568213e6b5e9db962 2024-04-25T01:24:32.447 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:32 smithi130 systemd[1]: Started Ceph mgr.z for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:24:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T01:24:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T01:24:32.475 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-25T01:24:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-25T01:24:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0087651 s, 1.2 GB/s 2024-04-25T01:24:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-25T01:24:32.711 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:32 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:32.466+0000 7f57e116b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:24:32.711 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:32 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:32.651+0000 7f57e116b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:24:33.043 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:32 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:32.754+0000 7f57e116b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:24:33.043 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:33 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:33.062+0000 7f57e116b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:24:33.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_4 2024-04-25T01:24:33.294 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:24:33.435 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:33 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:33.146+0000 7f57e116b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:24:33.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:33 smithi028 ceph-mon[29390]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:33.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:33.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:33.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:33.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:24:33.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:24:33.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:24:33.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:33.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:33 smithi028 ceph-mon[29390]: Deploying daemon mgr.y on smithi032 2024-04-25T01:24:33.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:33 smithi130 ceph-mon[32702]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:33.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:33.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:33.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:33.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:24:33.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:24:33.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:24:33.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:33.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:33 smithi130 ceph-mon[32702]: Deploying daemon mgr.y on smithi032 2024-04-25T01:24:33.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:33 smithi032 ceph-mon[32461]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:33.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:33.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:33.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:33.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:24:33.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:24:33.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:24:33.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:33.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:33 smithi032 ceph-mon[32461]: Deploying daemon mgr.y on smithi032 2024-04-25T01:24:34.654 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:34 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:34.335+0000 7f57e116b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:24:34.655 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:34 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:34.503+0000 7f57e116b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:24:34.655 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:34 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:34.674+0000 7f57e116b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:24:34.737 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:34 smithi032 systemd[1]: Starting Ceph mgr.y for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:24:34.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.775+0000 7fde2d8cc700 1 -- 172.21.15.28:0/2614370336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde281022b0 msgr2=0x7fde281026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:34.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.775+0000 7fde2d8cc700 1 --2- 172.21.15.28:0/2614370336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde281022b0 0x7fde281026d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fde180099c0 tx=0x7fde18009cd0 comp rx=0 tx=0).stop 2024-04-25T01:24:34.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.775+0000 7fde2d8cc700 1 -- 172.21.15.28:0/2614370336 shutdown_connections 2024-04-25T01:24:34.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.775+0000 7fde2d8cc700 1 --2- 172.21.15.28:0/2614370336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde28103e50 0x7fde2810a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:34.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.775+0000 7fde2d8cc700 1 --2- 172.21.15.28:0/2614370336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde281022b0 0x7fde281026d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:34.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.775+0000 7fde2d8cc700 1 --2- 172.21.15.28:0/2614370336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde281034b0 0x7fde28103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:34.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.775+0000 7fde2d8cc700 1 -- 172.21.15.28:0/2614370336 >> 172.21.15.28:0/2614370336 conn(0x7fde280fd850 msgr2=0x7fde280ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:34.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.775+0000 7fde2d8cc700 1 -- 172.21.15.28:0/2614370336 shutdown_connections 2024-04-25T01:24:34.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.775+0000 7fde2d8cc700 1 -- 172.21.15.28:0/2614370336 wait complete. 2024-04-25T01:24:34.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.775+0000 7fde2d8cc700 1 Processor -- start 2024-04-25T01:24:34.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.775+0000 7fde2d8cc700 1 -- start start 2024-04-25T01:24:34.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde2d8cc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde281022b0 0x7fde28075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:34.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde2d8cc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde281034b0 0x7fde28074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:34.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde2d8cc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde28103e50 0x7fde28074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:34.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde2d8cc700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde2810dee0 con 0x7fde281034b0 2024-04-25T01:24:34.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde2d8cc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fde2810dd60 con 0x7fde281022b0 2024-04-25T01:24:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde2d8cc700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fde2810e060 con 0x7fde28103e50 2024-04-25T01:24:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde1ffff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde281034b0 0x7fde28074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde1ffff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde281034b0 0x7fde28074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48148/0 (socket says 172.21.15.28:48148) 2024-04-25T01:24:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde1ffff700 1 -- 172.21.15.28:0/3203922696 learned_addr learned my addr 172.21.15.28:0/3203922696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:24:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde277fe700 1 --2- 172.21.15.28:0/3203922696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde28103e50 0x7fde28074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde26ffd700 1 --2- 172.21.15.28:0/3203922696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde281022b0 0x7fde28075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde1ffff700 1 -- 172.21.15.28:0/3203922696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde28103e50 msgr2=0x7fde28074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde1ffff700 1 --2- 172.21.15.28:0/3203922696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde28103e50 0x7fde28074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde1ffff700 1 -- 172.21.15.28:0/3203922696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde281022b0 msgr2=0x7fde28075fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde1ffff700 1 --2- 172.21.15.28:0/3203922696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde281022b0 0x7fde28075fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde1ffff700 1 -- 172.21.15.28:0/3203922696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde28075410 con 0x7fde281034b0 2024-04-25T01:24:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde1ffff700 1 --2- 172.21.15.28:0/3203922696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde281034b0 0x7fde28074620 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fde1000e9c0 tx=0x7fde1000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde2c8ca700 1 -- 172.21.15.28:0/3203922696 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1000cbd0 con 0x7fde281034b0 2024-04-25T01:24:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde2c8ca700 1 -- 172.21.15.28:0/3203922696 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde1000cd70 con 0x7fde281034b0 2024-04-25T01:24:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde2d8cc700 1 -- 172.21.15.28:0/3203922696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde281a26e0 con 0x7fde281034b0 2024-04-25T01:24:34.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.776+0000 7fde2c8ca700 1 -- 172.21.15.28:0/3203922696 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde10010430 con 0x7fde281034b0 2024-04-25T01:24:34.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.777+0000 7fde2d8cc700 1 -- 172.21.15.28:0/3203922696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde281a2ba0 con 0x7fde281034b0 2024-04-25T01:24:34.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.777+0000 7fde2c8ca700 1 -- 172.21.15.28:0/3203922696 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fde10004830 con 0x7fde281034b0 2024-04-25T01:24:34.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.777+0000 7fde2d8cc700 1 -- 172.21.15.28:0/3203922696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde28068f20 con 0x7fde281034b0 2024-04-25T01:24:34.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.778+0000 7fde2c8ca700 1 --2- 172.21.15.28:0/3203922696 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fde14039330 0x7fde1403b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:24:34.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.778+0000 7fde2c8ca700 1 -- 172.21.15.28:0/3203922696 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fde1004c5b0 con 0x7fde281034b0 2024-04-25T01:24:34.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.778+0000 7fde26ffd700 1 --2- 172.21.15.28:0/3203922696 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fde14039330 0x7fde1403b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:24:34.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.778+0000 7fde26ffd700 1 --2- 172.21.15.28:0/3203922696 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fde14039330 0x7fde1403b7f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fde1800b580 tx=0x7fde18005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:24:34.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:34.783+0000 7fde2c8ca700 1 -- 172.21.15.28:0/3203922696 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fde10016b30 con 0x7fde281034b0 2024-04-25T01:24:34.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:35.006+0000 7fde2d8cc700 1 -- 172.21.15.28:0/3203922696 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fde28062670 con 0x7fde14039330 2024-04-25T01:24:35.052 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:34 smithi032 podman[33837]: 2024-04-25 01:24:34.700177532 +0000 UTC m=+0.018131540 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:24:35.052 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:34 smithi032 podman[33837]: 2024-04-25 01:24:34.803940954 +0000 UTC m=+0.121894974 container create 4db81abec7f938c2b2d232ef3132c2734a39401aa06db2e024424c5e7bd7d707 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-25T01:24:35.052 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:34 smithi032 podman[33837]: 2024-04-25 01:24:34.998444605 +0000 UTC m=+0.316398598 container init 4db81abec7f938c2b2d232ef3132c2734a39401aa06db2e024424c5e7bd7d707 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-25T01:24:35.052 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:35 smithi032 podman[33837]: 2024-04-25 01:24:35.001413587 +0000 UTC m=+0.319367587 container start 4db81abec7f938c2b2d232ef3132c2734a39401aa06db2e024424c5e7bd7d707 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:24:35.052 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:35 smithi032 ceph-mgr[33857]: -- 172.21.15.32:0/951508965 <== mon.2 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564d5b2505a0 con 0x564d5b224800 2024-04-25T01:24:35.211 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:34 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:34.959+0000 7f57e116b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:24:35.211 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:35 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:35.051+0000 7f57e116b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:24:35.342 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:35 smithi032 bash[33837]: 4db81abec7f938c2b2d232ef3132c2734a39401aa06db2e024424c5e7bd7d707 2024-04-25T01:24:35.342 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:35 smithi032 systemd[1]: Started Ceph mgr.y for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:24:35.342 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:35 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:35.169+0000 7f4648237200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:24:35.342 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:35 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:35.360+0000 7f4648237200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:24:35.711 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:35 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:35.256+0000 7f57e116b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:24:35.711 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:35 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:35.454+0000 7f57e116b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:24:35.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:35 smithi130 ceph-mon[32702]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:35.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:35 smithi130 ceph-mon[32702]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:35.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:35 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:24:35.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:35 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:24:35.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:35 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:35.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:35 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:35.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:35 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:35.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:35 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:35.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:35 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:35.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:35 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:35.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:35 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:35.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:35 smithi028 ceph-mon[29390]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:35.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:35 smithi028 ceph-mon[29390]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:35.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:24:35.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:24:35.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:35.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:35.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:35.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:35.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:35.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:35.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:35.736 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:35 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:35.470+0000 7f4648237200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:24:35.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:35 smithi032 ceph-mon[32461]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:35.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:35 smithi032 ceph-mon[32461]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:24:35.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:35 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:24:35.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:35 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:24:35.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:35 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:35.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:35 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:35.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:35 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:35.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:35 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:35.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:35 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:35.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:35 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:35.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:35 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:36.160 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:35 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:35.800+0000 7f4648237200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:24:36.161 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:35 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:35.900+0000 7f4648237200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:24:36.461 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:36 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:36.094+0000 7f57e116b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:24:36.827 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:36 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:36.525+0000 7f57e116b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:24:36.827 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:36 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:36.637+0000 7f57e116b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:24:36.827 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:36 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:36.847+0000 7f57e116b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:24:37.486 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:37 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:37.188+0000 7f4648237200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:24:37.486 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:37 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:37.363+0000 7f4648237200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:24:37.663 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:37 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:37.374+0000 7f57e116b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:24:37.663 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:37 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:37.464+0000 7f57e116b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:24:37.816 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:37 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:37.539+0000 7f4648237200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:24:37.816 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:37 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:37.835+0000 7f4648237200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:24:37.816 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:37 smithi032 ceph-mon[32461]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:37.816 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:37 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:37.816 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:37 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:37.816 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:37 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:37.816 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:37 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:37.816 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:37 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:37.816 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:37 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:37.816 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:37 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:37.816 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:37 smithi032 ceph-mon[32461]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T01:24:37.817 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:37 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:24:37.817 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:37 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:24:37.817 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:37 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:37.817 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:37 smithi032 ceph-mon[32461]: Reconfiguring daemon mgr.x on smithi028 2024-04-25T01:24:37.961 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:37 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:37.703+0000 7f57e116b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:24:37.961 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:37 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:37.833+0000 7f57e116b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:24:37.961 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:37 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:37.927+0000 7f57e116b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:24:37.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:37 smithi130 ceph-mon[32702]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:37.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:37 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:37.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:37 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:37.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:37 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:37.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:37 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:37.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:37 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:37.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:37 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:37.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:37 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:37.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:37 smithi130 ceph-mon[32702]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T01:24:37.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:37 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:24:37.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:37 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:24:37.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:37 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:37.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:37 smithi130 ceph-mon[32702]: Reconfiguring daemon mgr.x on smithi028 2024-04-25T01:24:38.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:37 smithi028 ceph-mon[29390]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:38.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:37 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:38.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:37 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:38.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:37 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:38.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:37 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:38.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:37 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:38.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:37 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:38.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:37 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:38.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:37 smithi028 ceph-mon[29390]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T01:24:38.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:37 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:24:38.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:37 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:24:38.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:37 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:38.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:37 smithi028 ceph-mon[29390]: Reconfiguring daemon mgr.x on smithi028 2024-04-25T01:24:38.120 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:37 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:37.931+0000 7f4648237200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:24:38.120 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:38 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:38.140+0000 7f4648237200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:24:38.461 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:38 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:38.043+0000 7f57e116b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:24:38.461 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:38 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:38.133+0000 7f57e116b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:24:38.486 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:38 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:38.338+0000 7f4648237200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:24:38.942 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:38 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2783759395' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b"}]: dispatch 2024-04-25T01:24:38.942 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:38 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2783759395' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b"}]': finished 2024-04-25T01:24:38.942 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:38 smithi032 ceph-mon[32461]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T01:24:38.943 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:38 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:24:38.943 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:38 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1645092879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:24:38.946 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:38 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2783759395' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b"}]: dispatch 2024-04-25T01:24:38.947 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:38 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2783759395' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b"}]': finished 2024-04-25T01:24:38.947 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:38 smithi130 ceph-mon[32702]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T01:24:38.947 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:38 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:24:38.947 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:38 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1645092879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:24:39.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:38 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2783759395' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b"}]: dispatch 2024-04-25T01:24:39.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:38 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2783759395' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b"}]': finished 2024-04-25T01:24:39.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:38 smithi028 ceph-mon[29390]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T01:24:39.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:38 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:24:39.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:38 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1645092879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:24:39.211 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:38 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:38.966+0000 7f57e116b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:24:39.211 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:24:39 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:24:39.175+0000 7f57e116b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:24:39.236 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:38 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:38.962+0000 7f4648237200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:24:39.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:39.695+0000 7fde2c8ca700 1 -- 172.21.15.28:0/3203922696 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 14) ==== 92060+0+0 (secure 0 0 0) 0x7fde10021960 con 0x7fde281034b0 2024-04-25T01:24:39.678 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:39 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:39.397+0000 7f4648237200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:24:39.678 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:39 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:39.495+0000 7f4648237200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:24:39.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:39 smithi130 ceph-mon[32702]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:39.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:39 smithi130 ceph-mon[32702]: Standby manager daemon z started 2024-04-25T01:24:39.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:39 smithi130 ceph-mon[32702]: from='mgr.? 172.21.15.130:0/3874767478' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:24:39.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:39 smithi130 ceph-mon[32702]: from='mgr.? 172.21.15.130:0/3874767478' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:24:39.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:39 smithi130 ceph-mon[32702]: from='mgr.? 172.21.15.130:0/3874767478' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:24:39.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:39 smithi130 ceph-mon[32702]: from='mgr.? 172.21.15.130:0/3874767478' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:24:39.986 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:39 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:39.703+0000 7f4648237200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:24:39.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:39 smithi032 ceph-mon[32461]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:39.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:39 smithi032 ceph-mon[32461]: Standby manager daemon z started 2024-04-25T01:24:39.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:39 smithi032 ceph-mon[32461]: from='mgr.? 172.21.15.130:0/3874767478' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:24:39.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:39 smithi032 ceph-mon[32461]: from='mgr.? 172.21.15.130:0/3874767478' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:24:39.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:39 smithi032 ceph-mon[32461]: from='mgr.? 172.21.15.130:0/3874767478' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:24:39.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:39 smithi032 ceph-mon[32461]: from='mgr.? 172.21.15.130:0/3874767478' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:24:40.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:39 smithi028 ceph-mon[29390]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:40.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:39 smithi028 ceph-mon[29390]: Standby manager daemon z started 2024-04-25T01:24:40.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:39 smithi028 ceph-mon[29390]: from='mgr.? 172.21.15.130:0/3874767478' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:24:40.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:39 smithi028 ceph-mon[29390]: from='mgr.? 172.21.15.130:0/3874767478' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:24:40.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:39 smithi028 ceph-mon[29390]: from='mgr.? 172.21.15.130:0/3874767478' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:24:40.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:39 smithi028 ceph-mon[29390]: from='mgr.? 172.21.15.130:0/3874767478' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:24:40.486 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:40 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:40.247+0000 7f4648237200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:24:40.486 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:40 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:40.341+0000 7f4648237200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:24:40.926 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:40 smithi032 ceph-mon[32461]: mgrmap e14: x(active, since 79s), standbys: z 2024-04-25T01:24:40.926 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:40 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:24:40.926 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:40 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:40.589+0000 7f4648237200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:24:40.926 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:40 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:40.728+0000 7f4648237200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:24:40.926 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:40 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:40.827+0000 7f4648237200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:24:40.926 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:40 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:40.945+0000 7f4648237200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:24:40.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:40 smithi130 ceph-mon[32702]: mgrmap e14: x(active, since 79s), standbys: z 2024-04-25T01:24:40.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:40 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:24:41.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:40 smithi028 ceph-mon[29390]: mgrmap e14: x(active, since 79s), standbys: z 2024-04-25T01:24:41.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:40 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:24:41.236 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:41 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:41.039+0000 7f4648237200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:24:41.904 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:41 smithi028 ceph-mon[29390]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:41.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:41 smithi130 ceph-mon[32702]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:41.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:41 smithi032 ceph-mon[32461]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:41.986 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:41 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:41.886+0000 7f4648237200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:24:42.486 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:24:42 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:24:42.107+0000 7f4648237200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:24:42.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:42.716+0000 7fde2c8ca700 1 -- 172.21.15.28:0/3203922696 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7fde10021960 con 0x7fde281034b0 2024-04-25T01:24:42.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:42 smithi130 ceph-mon[32702]: Standby manager daemon y started 2024-04-25T01:24:42.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:42 smithi130 ceph-mon[32702]: from='mgr.? 172.21.15.32:0/2775109331' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:24:42.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:42 smithi130 ceph-mon[32702]: from='mgr.? 172.21.15.32:0/2775109331' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:24:42.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:42 smithi130 ceph-mon[32702]: from='mgr.? 172.21.15.32:0/2775109331' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:24:42.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:42 smithi130 ceph-mon[32702]: from='mgr.? 172.21.15.32:0/2775109331' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:24:42.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:42 smithi032 ceph-mon[32461]: Standby manager daemon y started 2024-04-25T01:24:42.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:42 smithi032 ceph-mon[32461]: from='mgr.? 172.21.15.32:0/2775109331' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:24:42.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:42 smithi032 ceph-mon[32461]: from='mgr.? 172.21.15.32:0/2775109331' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:24:42.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:42 smithi032 ceph-mon[32461]: from='mgr.? 172.21.15.32:0/2775109331' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:24:42.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:42 smithi032 ceph-mon[32461]: from='mgr.? 172.21.15.32:0/2775109331' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:24:43.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:42 smithi028 ceph-mon[29390]: Standby manager daemon y started 2024-04-25T01:24:43.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:42 smithi028 ceph-mon[29390]: from='mgr.? 172.21.15.32:0/2775109331' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:24:43.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:42 smithi028 ceph-mon[29390]: from='mgr.? 172.21.15.32:0/2775109331' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:24:43.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:42 smithi028 ceph-mon[29390]: from='mgr.? 172.21.15.32:0/2775109331' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:24:43.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:42 smithi028 ceph-mon[29390]: from='mgr.? 172.21.15.32:0/2775109331' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:24:43.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:43 smithi028 ceph-mon[29390]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:43.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:43 smithi028 ceph-mon[29390]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-04-25T01:24:43.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:43 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:24:43.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:43 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:43.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:43 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:43.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:43 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:43.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:43 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:43.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:43 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:43.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:43 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:43.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:43 smithi130 ceph-mon[32702]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:43.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:43 smithi130 ceph-mon[32702]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-04-25T01:24:43.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:43 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:24:43.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:43 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:43.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:43 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:43.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:43 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:43.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:43 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:43.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:43 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:43.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:43 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:43.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:43 smithi032 ceph-mon[32461]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:43.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:43 smithi032 ceph-mon[32461]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-04-25T01:24:43.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:43 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:24:43.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:43 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:43.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:43 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:43.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:43 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:43.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:43 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:43.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:43 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:43.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:43 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:45.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:45 smithi130 ceph-mon[32702]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:45.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:45 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T01:24:45.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:45 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:45.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:45 smithi032 ceph-mon[32461]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:45.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:45 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T01:24:45.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:45 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:46.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:45 smithi028 ceph-mon[29390]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:46.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:45 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T01:24:46.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:45 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:46.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:46 smithi130 ceph-mon[32702]: Deploying daemon osd.0 on smithi028 2024-04-25T01:24:46.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:46 smithi032 ceph-mon[32461]: Deploying daemon osd.0 on smithi028 2024-04-25T01:24:46.994 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:46 smithi028 ceph-mon[29390]: Deploying daemon osd.0 on smithi028 2024-04-25T01:24:47.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:47 smithi130 ceph-mon[32702]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:47.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:47 smithi032 ceph-mon[32461]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:48.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:47 smithi028 ceph-mon[29390]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:49.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:49 smithi130 ceph-mon[32702]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:49.983 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:49 smithi028 ceph-mon[29390]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:49.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:49 smithi032 ceph-mon[32461]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:51.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:51 smithi130 ceph-mon[32702]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:51.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:51 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:51.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:51 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:51.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:51 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:51.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:51 smithi032 ceph-mon[32461]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:51.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:51 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:51.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:51 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:51.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:51 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:52.008 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:51 smithi028 ceph-mon[29390]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:52.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:51 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:52.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:51 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:52.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:51 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:53.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:53 smithi130 ceph-mon[32702]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:53.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:53.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:53.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:53.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:53.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:53.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:53 smithi032 ceph-mon[32461]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:53.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:53.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:53.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:53.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:53.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:54.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:53 smithi028 ceph-mon[29390]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:54.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:54.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:54.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:24:54.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:24:54.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:54.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:54.981+0000 7fde2c8ca700 1 -- 172.21.15.28:0/3203922696 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fde28062670 con 0x7fde14039330 2024-04-25T01:24:54.961 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 0 on host 'smithi028' 2024-04-25T01:24:54.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:54.983+0000 7fde2d8cc700 1 -- 172.21.15.28:0/3203922696 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fde14039330 msgr2=0x7fde1403b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:54.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:54.983+0000 7fde2d8cc700 1 --2- 172.21.15.28:0/3203922696 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fde14039330 0x7fde1403b7f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fde1800b580 tx=0x7fde18005fb0 comp rx=0 tx=0).stop 2024-04-25T01:24:54.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:54.983+0000 7fde2d8cc700 1 -- 172.21.15.28:0/3203922696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde281034b0 msgr2=0x7fde28074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:24:54.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:54.983+0000 7fde2d8cc700 1 --2- 172.21.15.28:0/3203922696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde281034b0 0x7fde28074620 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fde1000e9c0 tx=0x7fde1000ecd0 comp rx=0 tx=0).stop 2024-04-25T01:24:54.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:54.983+0000 7fde2d8cc700 1 -- 172.21.15.28:0/3203922696 shutdown_connections 2024-04-25T01:24:54.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:54.983+0000 7fde2d8cc700 1 --2- 172.21.15.28:0/3203922696 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fde14039330 0x7fde1403b7f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:54.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:54.983+0000 7fde2d8cc700 1 --2- 172.21.15.28:0/3203922696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fde28103e50 0x7fde28074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:54.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:54.983+0000 7fde2d8cc700 1 --2- 172.21.15.28:0/3203922696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde281022b0 0x7fde28075fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:54.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:54.983+0000 7fde2d8cc700 1 --2- 172.21.15.28:0/3203922696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde281034b0 0x7fde28074620 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:24:54.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:54.984+0000 7fde2d8cc700 1 -- 172.21.15.28:0/3203922696 >> 172.21.15.28:0/3203922696 conn(0x7fde280fd850 msgr2=0x7fde281067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:24:54.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:54.984+0000 7fde2d8cc700 1 -- 172.21.15.28:0/3203922696 shutdown_connections 2024-04-25T01:24:54.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:24:54.984+0000 7fde2d8cc700 1 -- 172.21.15.28:0/3203922696 wait complete. 2024-04-25T01:24:55.631 DEBUG:teuthology.orchestra.run.smithi028:osd.0> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.0.service 2024-04-25T01:24:55.634 INFO:tasks.cephadm:Deploying osd.1 on smithi032 with /dev/vg_nvme/lv_4... 2024-04-25T01:24:55.634 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T01:24:55.887 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:24:55.984 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:55 smithi028 ceph-mon[29390]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:55.984 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:55 smithi028 ceph-mon[29390]: from='osd.0 [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T01:24:55.984 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:55 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:55.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:55 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:55.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:55 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:55.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:55 smithi032 ceph-mon[32461]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:55.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:55 smithi032 ceph-mon[32461]: from='osd.0 [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T01:24:55.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:55 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:55.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:55 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:55.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:55 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:56.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:55 smithi130 ceph-mon[32702]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:56.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:55 smithi130 ceph-mon[32702]: from='osd.0 [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T01:24:56.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:55 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:56.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:55 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:24:56.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:55 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:24:56.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:56 smithi032 ceph-mon[32461]: from='osd.0 [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T01:24:56.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:56 smithi032 ceph-mon[32461]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T01:24:56.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:56 smithi032 ceph-mon[32461]: from='osd.0 [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-25T01:24:56.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:56 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:24:57.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:56 smithi028 ceph-mon[29390]: from='osd.0 [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T01:24:57.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:56 smithi028 ceph-mon[29390]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T01:24:57.028 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:56 smithi028 ceph-mon[29390]: from='osd.0 [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-25T01:24:57.029 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:56 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:24:57.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:56 smithi130 ceph-mon[32702]: from='osd.0 [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T01:24:57.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:56 smithi130 ceph-mon[32702]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T01:24:57.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:56 smithi130 ceph-mon[32702]: from='osd.0 [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-25T01:24:57.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:56 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:24:57.564 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:24:57 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-0[41941]: 2024-04-25T01:24:57.321+0000 7fabd0b61700 -1 osd.0 0 waiting for initial osdmap 2024-04-25T01:24:57.564 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:24:57 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-0[41941]: 2024-04-25T01:24:57.326+0000 7fabc894c700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:24:57.942 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T01:24:57.942 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T01:24:57.942 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-04-25T01:24:57.942 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-04-25T01:24:57.942 INFO:teuthology.orchestra.run.smithi032.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00829595 s, 1.3 GB/s 2024-04-25T01:24:57.942 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-04-25T01:24:57.989 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:57 smithi032 ceph-mon[32461]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:57.990 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:57 smithi032 ceph-mon[32461]: from='osd.0 [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-25T01:24:57.990 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:57 smithi032 ceph-mon[32461]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T01:24:57.990 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:57 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:24:57.990 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:57 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:24:57.990 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:57 smithi032 ceph-mon[32461]: from='osd.0 [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772]' entity='osd.0' 2024-04-25T01:24:58.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:57 smithi028 ceph-mon[29390]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:58.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:57 smithi028 ceph-mon[29390]: from='osd.0 [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-25T01:24:58.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:57 smithi028 ceph-mon[29390]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T01:24:58.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:57 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:24:58.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:57 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:24:58.065 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:57 smithi028 ceph-mon[29390]: from='osd.0 [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772]' entity='osd.0' 2024-04-25T01:24:58.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:57 smithi130 ceph-mon[32702]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:24:58.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:57 smithi130 ceph-mon[32702]: from='osd.0 [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-25T01:24:58.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:57 smithi130 ceph-mon[32702]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T01:24:58.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:57 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:24:58.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:57 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:24:58.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:57 smithi130 ceph-mon[32702]: from='osd.0 [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772]' entity='osd.0' 2024-04-25T01:24:58.593 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_4 2024-04-25T01:24:58.840 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:24:58.890 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:58 smithi032 ceph-mon[32461]: purged_snaps scrub starts 2024-04-25T01:24:58.890 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:58 smithi032 ceph-mon[32461]: purged_snaps scrub ok 2024-04-25T01:24:58.890 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:58 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:24:58.890 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:58 smithi032 ceph-mon[32461]: osd.0 [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772] boot 2024-04-25T01:24:58.890 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:58 smithi032 ceph-mon[32461]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T01:24:58.890 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:58 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:24:59.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:58 smithi028 ceph-mon[29390]: purged_snaps scrub starts 2024-04-25T01:24:59.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:58 smithi028 ceph-mon[29390]: purged_snaps scrub ok 2024-04-25T01:24:59.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:58 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:24:59.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:58 smithi028 ceph-mon[29390]: osd.0 [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772] boot 2024-04-25T01:24:59.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:58 smithi028 ceph-mon[29390]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T01:24:59.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:58 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:24:59.112 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:58 smithi130 ceph-mon[32702]: purged_snaps scrub starts 2024-04-25T01:24:59.112 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:58 smithi130 ceph-mon[32702]: purged_snaps scrub ok 2024-04-25T01:24:59.112 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:58 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:24:59.113 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:58 smithi130 ceph-mon[32702]: osd.0 [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772] boot 2024-04-25T01:24:59.113 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:58 smithi130 ceph-mon[32702]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T01:24:59.113 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:58 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:25:00.204 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:59 smithi032 ceph-mon[32461]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:25:00.205 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:59 smithi032 ceph-mon[32461]: Detected new or changed devices on smithi028 2024-04-25T01:25:00.205 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:59 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:00.205 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:59 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:00.205 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:59 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:00.205 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:59 smithi032 ceph-mon[32461]: Adjusting osd_memory_target on smithi028 to 17093M 2024-04-25T01:25:00.205 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:59 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:00.205 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:59 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:00.205 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:59 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:00.205 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:24:59 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:00.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:59 smithi130 ceph-mon[32702]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:25:00.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:59 smithi130 ceph-mon[32702]: Detected new or changed devices on smithi028 2024-04-25T01:25:00.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:59 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:00.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:59 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:00.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:59 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:00.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:59 smithi130 ceph-mon[32702]: Adjusting osd_memory_target on smithi028 to 17093M 2024-04-25T01:25:00.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:59 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:00.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:59 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:00.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:59 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:00.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:24:59 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:00.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:59 smithi028 ceph-mon[29390]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:25:00.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:59 smithi028 ceph-mon[29390]: Detected new or changed devices on smithi028 2024-04-25T01:25:00.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:59 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:00.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:59 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:00.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:59 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:00.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:59 smithi028 ceph-mon[29390]: Adjusting osd_memory_target on smithi028 to 17093M 2024-04-25T01:25:00.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:59 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:00.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:59 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:59 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:24:59 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:00.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.331+0000 7f2599223700 1 -- 172.21.15.32:0/2551878068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2594102d80 msgr2=0x7f2594105170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:00.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.331+0000 7f2599223700 1 --2- 172.21.15.32:0/2551878068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2594102d80 0x7f2594105170 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2584009990 tx=0x7f2584009ca0 comp rx=0 tx=0).stop 2024-04-25T01:25:00.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.331+0000 7f2599223700 1 -- 172.21.15.32:0/2551878068 shutdown_connections 2024-04-25T01:25:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.331+0000 7f2599223700 1 --2- 172.21.15.32:0/2551878068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2594107fe0 0x7f259410a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.331+0000 7f2599223700 1 --2- 172.21.15.32:0/2551878068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2594102d80 0x7f2594105170 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.331+0000 7f2599223700 1 --2- 172.21.15.32:0/2551878068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25941056b0 0x7f2594107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.331+0000 7f2599223700 1 -- 172.21.15.32:0/2551878068 >> 172.21.15.32:0/2551878068 conn(0x7f25940fc730 msgr2=0x7f25940feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.331+0000 7f2599223700 1 -- 172.21.15.32:0/2551878068 shutdown_connections 2024-04-25T01:25:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.331+0000 7f2599223700 1 -- 172.21.15.32:0/2551878068 wait complete. 2024-04-25T01:25:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.332+0000 7f2599223700 1 Processor -- start 2024-04-25T01:25:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.332+0000 7f2599223700 1 -- start start 2024-04-25T01:25:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.332+0000 7f2599223700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2594102d80 0x7f25941984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.332+0000 7f2599223700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25941056b0 0x7f2594198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.332+0000 7f2599223700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2594107fe0 0x7f259419faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.332+0000 7f2599223700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f259410de70 con 0x7f25941056b0 2024-04-25T01:25:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.332+0000 7f2599223700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f259410dcf0 con 0x7f2594102d80 2024-04-25T01:25:00.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.332+0000 7f2599223700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f259410dff0 con 0x7f2594107fe0 2024-04-25T01:25:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.332+0000 7f2592d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2594102d80 0x7f25941984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.332+0000 7f2592d9d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2594102d80 0x7f25941984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58248/0 (socket says 172.21.15.32:58248) 2024-04-25T01:25:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.332+0000 7f2592d9d700 1 -- 172.21.15.32:0/1040115800 learned_addr learned my addr 172.21.15.32:0/1040115800 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-25T01:25:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.332+0000 7f259359e700 1 --2- 172.21.15.32:0/1040115800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2594107fe0 0x7f259419faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.332+0000 7f259259c700 1 --2- 172.21.15.32:0/1040115800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25941056b0 0x7f2594198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.333+0000 7f259359e700 1 -- 172.21.15.32:0/1040115800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2594102d80 msgr2=0x7f25941984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:00.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.333+0000 7f259359e700 1 --2- 172.21.15.32:0/1040115800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2594102d80 0x7f25941984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.333+0000 7f259359e700 1 -- 172.21.15.32:0/1040115800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25941056b0 msgr2=0x7f2594198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.333+0000 7f259359e700 1 --2- 172.21.15.32:0/1040115800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25941056b0 0x7f2594198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.333+0000 7f259359e700 1 -- 172.21.15.32:0/1040115800 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25941a0140 con 0x7f2594107fe0 2024-04-25T01:25:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.333+0000 7f2592d9d700 1 --2- 172.21.15.32:0/1040115800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2594102d80 0x7f25941984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:25:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.333+0000 7f259359e700 1 --2- 172.21.15.32:0/1040115800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2594107fe0 0x7f259419faa0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f258c00efc0 tx=0x7f258c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.333+0000 7f257bfff700 1 -- 172.21.15.32:0/1040115800 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c0092e0 con 0x7f2594107fe0 2024-04-25T01:25:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.333+0000 7f257bfff700 1 -- 172.21.15.32:0/1040115800 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f258c004bb0 con 0x7f2594107fe0 2024-04-25T01:25:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.333+0000 7f2599223700 1 -- 172.21.15.32:0/1040115800 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25941a03d0 con 0x7f2594107fe0 2024-04-25T01:25:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.333+0000 7f257bfff700 1 -- 172.21.15.32:0/1040115800 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c004280 con 0x7f2594107fe0 2024-04-25T01:25:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.333+0000 7f2599223700 1 -- 172.21.15.32:0/1040115800 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25941a0910 con 0x7f2594107fe0 2024-04-25T01:25:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.334+0000 7f2599223700 1 -- 172.21.15.32:0/1040115800 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2594068f20 con 0x7f2594107fe0 2024-04-25T01:25:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.335+0000 7f257bfff700 1 -- 172.21.15.32:0/1040115800 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f258c01b030 con 0x7f2594107fe0 2024-04-25T01:25:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.336+0000 7f257bfff700 1 --2- 172.21.15.32:0/1040115800 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f257c0a2c10 0x7f257c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.336+0000 7f257bfff700 1 -- 172.21.15.32:0/1040115800 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f258c0cdad0 con 0x7f2594107fe0 2024-04-25T01:25:00.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.336+0000 7f2592d9d700 1 --2- 172.21.15.32:0/1040115800 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f257c0a2c10 0x7f257c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:00.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.337+0000 7f2592d9d700 1 --2- 172.21.15.32:0/1040115800 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f257c0a2c10 0x7f257c0a50d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f258400b7b0 tx=0x7f258400b8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:00.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.339+0000 7f257bfff700 1 -- 172.21.15.32:0/1040115800 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f258c022020 con 0x7f2594107fe0 2024-04-25T01:25:00.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:00.554+0000 7f2599223700 1 -- 172.21.15.32:0/1040115800 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f2594062730 con 0x7f257c0a2c10 2024-04-25T01:25:01.154 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:00 smithi032 ceph-mon[32461]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T01:25:01.155 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:00 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:25:01.155 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:00 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:25:01.155 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:00 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:01.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:00 smithi130 ceph-mon[32702]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T01:25:01.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:00 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:25:01.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:00 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:25:01.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:00 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:00 smithi028 ceph-mon[29390]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T01:25:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:25:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:25:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:02.068 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:01 smithi032 ceph-mon[32461]: pgmap v49: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:02.068 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:01 smithi032 ceph-mon[32461]: from='client.24142 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:02.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:01 smithi130 ceph-mon[32702]: pgmap v49: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:02.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:01 smithi130 ceph-mon[32702]: from='client.24142 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:02.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:01 smithi028 ceph-mon[29390]: pgmap v49: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:02.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:01 smithi028 ceph-mon[29390]: from='client.24142 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:04.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:03 smithi130 ceph-mon[32702]: pgmap v50: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:04.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:03 smithi130 ceph-mon[32702]: from='client.? 172.21.15.32:0/3467824566' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a457e4c-75ec-4b92-9aaa-f66036d4b492"}]: dispatch 2024-04-25T01:25:04.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:03 smithi130 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a457e4c-75ec-4b92-9aaa-f66036d4b492"}]: dispatch 2024-04-25T01:25:04.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:03 smithi130 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a457e4c-75ec-4b92-9aaa-f66036d4b492"}]': finished 2024-04-25T01:25:04.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:03 smithi130 ceph-mon[32702]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T01:25:04.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:03 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:25:04.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:03 smithi130 ceph-mon[32702]: from='client.? 172.21.15.32:0/1484737880' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:25:04.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:03 smithi032 ceph-mon[32461]: pgmap v50: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:04.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:03 smithi032 ceph-mon[32461]: from='client.? 172.21.15.32:0/3467824566' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a457e4c-75ec-4b92-9aaa-f66036d4b492"}]: dispatch 2024-04-25T01:25:04.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:03 smithi032 ceph-mon[32461]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a457e4c-75ec-4b92-9aaa-f66036d4b492"}]: dispatch 2024-04-25T01:25:04.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:03 smithi032 ceph-mon[32461]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a457e4c-75ec-4b92-9aaa-f66036d4b492"}]': finished 2024-04-25T01:25:04.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:03 smithi032 ceph-mon[32461]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T01:25:04.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:03 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:25:04.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:03 smithi032 ceph-mon[32461]: from='client.? 172.21.15.32:0/1484737880' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:25:04.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:03 smithi028 ceph-mon[29390]: pgmap v50: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:04.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:03 smithi028 ceph-mon[29390]: from='client.? 172.21.15.32:0/3467824566' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a457e4c-75ec-4b92-9aaa-f66036d4b492"}]: dispatch 2024-04-25T01:25:04.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:03 smithi028 ceph-mon[29390]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a457e4c-75ec-4b92-9aaa-f66036d4b492"}]: dispatch 2024-04-25T01:25:04.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:03 smithi028 ceph-mon[29390]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a457e4c-75ec-4b92-9aaa-f66036d4b492"}]': finished 2024-04-25T01:25:04.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:03 smithi028 ceph-mon[29390]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T01:25:04.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:03 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:25:04.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:03 smithi028 ceph-mon[29390]: from='client.? 172.21.15.32:0/1484737880' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:25:06.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:05 smithi130 ceph-mon[32702]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:06.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:05 smithi032 ceph-mon[32461]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:06.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:05 smithi028 ceph-mon[29390]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:08.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:07 smithi130 ceph-mon[32702]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:08.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:07 smithi032 ceph-mon[32461]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:08.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:07 smithi028 ceph-mon[29390]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:10.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:09 smithi130 ceph-mon[32702]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:10.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:09 smithi032 ceph-mon[32461]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:10.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:09 smithi028 ceph-mon[29390]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:11.177 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:10 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T01:25:11.177 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:10 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:11.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:10 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T01:25:11.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:10 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:11.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:10 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T01:25:11.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:10 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:12.134 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:11 smithi032 ceph-mon[32461]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:12.134 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:11 smithi032 ceph-mon[32461]: Deploying daemon osd.1 on smithi032 2024-04-25T01:25:12.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:11 smithi130 ceph-mon[32702]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:12.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:11 smithi130 ceph-mon[32702]: Deploying daemon osd.1 on smithi032 2024-04-25T01:25:12.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:11 smithi028 ceph-mon[29390]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:12.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:11 smithi028 ceph-mon[29390]: Deploying daemon osd.1 on smithi032 2024-04-25T01:25:14.112 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:13 smithi032 ceph-mon[32461]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:14.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:13 smithi130 ceph-mon[32702]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:14.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:13 smithi028 ceph-mon[29390]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:16.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:15 smithi130 ceph-mon[32702]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:16.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:15 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:16.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:15 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:16.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:15 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:16.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:15 smithi032 ceph-mon[32461]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:16.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:16.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:16.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:16.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:15 smithi028 ceph-mon[29390]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:16.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:15 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:16.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:15 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:16.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:15 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:18.200 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:17 smithi032 ceph-mon[32461]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:18.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:17 smithi130 ceph-mon[32702]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:18.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:17 smithi028 ceph-mon[29390]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:19.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:19 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:19.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:19 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:19.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:19 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:19.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:19 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:19.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:19 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:19.368 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:19 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:19.368 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:19 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:19.368 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:19 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:19.368 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:19 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:19.368 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:19 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:19.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:19.469+0000 7f257bfff700 1 -- 172.21.15.32:0/1040115800 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2594062730 con 0x7f257c0a2c10 2024-04-25T01:25:19.450 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 1 on host 'smithi032' 2024-04-25T01:25:19.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:19.471+0000 7f2599223700 1 -- 172.21.15.32:0/1040115800 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f257c0a2c10 msgr2=0x7f257c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:19.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:19.471+0000 7f2599223700 1 --2- 172.21.15.32:0/1040115800 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f257c0a2c10 0x7f257c0a50d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f258400b7b0 tx=0x7f258400b8a0 comp rx=0 tx=0).stop 2024-04-25T01:25:19.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:19.471+0000 7f2599223700 1 -- 172.21.15.32:0/1040115800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2594107fe0 msgr2=0x7f259419faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:19.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:19.471+0000 7f2599223700 1 --2- 172.21.15.32:0/1040115800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2594107fe0 0x7f259419faa0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f258c00efc0 tx=0x7f258c00c3a0 comp rx=0 tx=0).stop 2024-04-25T01:25:19.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:19.471+0000 7f2599223700 1 -- 172.21.15.32:0/1040115800 shutdown_connections 2024-04-25T01:25:19.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:19.471+0000 7f2599223700 1 --2- 172.21.15.32:0/1040115800 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f257c0a2c10 0x7f257c0a50d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:19.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:19.471+0000 7f2599223700 1 --2- 172.21.15.32:0/1040115800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2594107fe0 0x7f259419faa0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:19.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:19.471+0000 7f2599223700 1 --2- 172.21.15.32:0/1040115800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2594102d80 0x7f25941984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:19.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:19.471+0000 7f2599223700 1 --2- 172.21.15.32:0/1040115800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25941056b0 0x7f2594198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:19.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:19.471+0000 7f2599223700 1 -- 172.21.15.32:0/1040115800 >> 172.21.15.32:0/1040115800 conn(0x7f25940fc730 msgr2=0x7f2594103800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:19.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:19.471+0000 7f2599223700 1 -- 172.21.15.32:0/1040115800 shutdown_connections 2024-04-25T01:25:19.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:25:19.471+0000 7f2599223700 1 -- 172.21.15.32:0/1040115800 wait complete. 2024-04-25T01:25:19.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:19 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:19.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:19 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:19.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:19 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:19.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:19 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:19.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:19 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:20.081 DEBUG:teuthology.orchestra.run.smithi032:osd.1> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.1.service 2024-04-25T01:25:20.084 INFO:tasks.cephadm:Deploying osd.2 on smithi130 with /dev/vg_nvme/lv_4... 2024-04-25T01:25:20.084 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T01:25:20.313 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:20 smithi032 ceph-mon[32461]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:20.313 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:20 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:20.313 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:20 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:20.313 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:20 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:20.313 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:20 smithi032 ceph-mon[32461]: from='osd.1 [v2:172.21.15.32:6800/2516196299,v1:172.21.15.32:6801/2516196299]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:25:20.313 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:20 smithi032 ceph-mon[32461]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:25:20.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:20 smithi028 ceph-mon[29390]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:20.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:20.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:20.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:20.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:20 smithi028 ceph-mon[29390]: from='osd.1 [v2:172.21.15.32:6800/2516196299,v1:172.21.15.32:6801/2516196299]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:25:20.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:20 smithi028 ceph-mon[29390]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:25:20.329 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:25:20.451 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:20 smithi130 ceph-mon[32702]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:20.451 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:20.452 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:20.452 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:20.452 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:20 smithi130 ceph-mon[32702]: from='osd.1 [v2:172.21.15.32:6800/2516196299,v1:172.21.15.32:6801/2516196299]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:25:20.452 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:20 smithi130 ceph-mon[32702]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:25:21.578 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:21 smithi032 ceph-mon[32461]: pgmap v60: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:21.579 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:21 smithi032 ceph-mon[32461]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T01:25:21.579 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:21 smithi032 ceph-mon[32461]: from='osd.1 [v2:172.21.15.32:6800/2516196299,v1:172.21.15.32:6801/2516196299]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:25:21.579 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:21 smithi032 ceph-mon[32461]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T01:25:21.579 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:21 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:25:21.579 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:21 smithi032 ceph-mon[32461]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:25:21.579 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:25:21 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-1[37059]: 2024-04-25T01:25:21.436+0000 7f552ece3700 -1 osd.1 0 waiting for initial osdmap 2024-04-25T01:25:21.579 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:25:21 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-1[37059]: 2024-04-25T01:25:21.441+0000 7f552aad6700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:25:21.581 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:21 smithi130 ceph-mon[32702]: pgmap v60: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:21.581 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:21 smithi130 ceph-mon[32702]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T01:25:21.581 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:21 smithi130 ceph-mon[32702]: from='osd.1 [v2:172.21.15.32:6800/2516196299,v1:172.21.15.32:6801/2516196299]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:25:21.581 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:21 smithi130 ceph-mon[32702]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T01:25:21.581 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:21 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:25:21.581 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:21 smithi130 ceph-mon[32702]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:25:21.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:21 smithi028 ceph-mon[29390]: pgmap v60: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:21.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:21 smithi028 ceph-mon[29390]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T01:25:21.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:21 smithi028 ceph-mon[29390]: from='osd.1 [v2:172.21.15.32:6800/2516196299,v1:172.21.15.32:6801/2516196299]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:25:21.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:21 smithi028 ceph-mon[29390]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T01:25:21.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:21 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:25:21.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:21 smithi028 ceph-mon[29390]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:25:22.146 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T01:25:22.146 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T01:25:22.146 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-04-25T01:25:22.146 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-04-25T01:25:22.146 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011187 s, 937 MB/s 2024-04-25T01:25:22.146 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-04-25T01:25:22.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:22 smithi130 ceph-mon[32702]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-25T01:25:22.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:22 smithi130 ceph-mon[32702]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T01:25:22.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:22 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:25:22.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:22 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:25:22.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:22 smithi032 ceph-mon[32461]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-25T01:25:22.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:22 smithi032 ceph-mon[32461]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T01:25:22.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:22 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:25:22.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:22 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:25:22.813 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi130:vg_nvme/lv_4 2024-04-25T01:25:22.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:22 smithi028 ceph-mon[29390]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-25T01:25:22.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:22 smithi028 ceph-mon[29390]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T01:25:22.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:22 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:25:22.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:22 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:25:23.054 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:25:23.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:23 smithi130 ceph-mon[32702]: purged_snaps scrub starts 2024-04-25T01:25:23.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:23 smithi130 ceph-mon[32702]: purged_snaps scrub ok 2024-04-25T01:25:23.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:23 smithi130 ceph-mon[32702]: pgmap v63: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:23.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:23 smithi130 ceph-mon[32702]: osd.1 [v2:172.21.15.32:6800/2516196299,v1:172.21.15.32:6801/2516196299] boot 2024-04-25T01:25:23.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:23 smithi130 ceph-mon[32702]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T01:25:23.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:25:23.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:23 smithi032 ceph-mon[32461]: purged_snaps scrub starts 2024-04-25T01:25:23.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:23 smithi032 ceph-mon[32461]: purged_snaps scrub ok 2024-04-25T01:25:23.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:23 smithi032 ceph-mon[32461]: pgmap v63: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:23.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:23 smithi032 ceph-mon[32461]: osd.1 [v2:172.21.15.32:6800/2516196299,v1:172.21.15.32:6801/2516196299] boot 2024-04-25T01:25:23.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:23 smithi032 ceph-mon[32461]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T01:25:23.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:25:23.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:23 smithi028 ceph-mon[29390]: purged_snaps scrub starts 2024-04-25T01:25:23.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:23 smithi028 ceph-mon[29390]: purged_snaps scrub ok 2024-04-25T01:25:23.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:23 smithi028 ceph-mon[29390]: pgmap v63: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:25:23.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:23 smithi028 ceph-mon[29390]: osd.1 [v2:172.21.15.32:6800/2516196299,v1:172.21.15.32:6801/2516196299] boot 2024-04-25T01:25:23.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:23 smithi028 ceph-mon[29390]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T01:25:23.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:25:24.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:24 smithi130 ceph-mon[32702]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T01:25:24.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:24 smithi130 ceph-mon[32702]: Detected new or changed devices on smithi032 2024-04-25T01:25:24.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:24 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:24.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:24 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:24.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:24 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:24.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:24 smithi130 ceph-mon[32702]: Adjusting osd_memory_target on smithi032 to 17093M 2024-04-25T01:25:24.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:24 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:24.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:24 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:24.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:24 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:24.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:24 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:24.499 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.518+0000 7eff96fea700 1 -- 172.21.15.130:0/1345518065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff90102290 msgr2=0x7eff901026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:24.499 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.518+0000 7eff96fea700 1 --2- 172.21.15.130:0/1345518065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff90102290 0x7eff901026b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7eff84009990 tx=0x7eff84009ca0 comp rx=0 tx=0).stop 2024-04-25T01:25:24.499 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.519+0000 7eff96fea700 1 -- 172.21.15.130:0/1345518065 shutdown_connections 2024-04-25T01:25:24.499 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.519+0000 7eff96fea700 1 --2- 172.21.15.130:0/1345518065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff90103e30 0x7eff9010a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:24.499 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.519+0000 7eff96fea700 1 --2- 172.21.15.130:0/1345518065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff90102290 0x7eff901026b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:24.499 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.519+0000 7eff96fea700 1 --2- 172.21.15.130:0/1345518065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff90103490 0x7eff901038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:24.499 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.519+0000 7eff96fea700 1 -- 172.21.15.130:0/1345518065 >> 172.21.15.130:0/1345518065 conn(0x7eff900fd830 msgr2=0x7eff900ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:24.499 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.519+0000 7eff96fea700 1 -- 172.21.15.130:0/1345518065 shutdown_connections 2024-04-25T01:25:24.499 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.519+0000 7eff96fea700 1 -- 172.21.15.130:0/1345518065 wait complete. 2024-04-25T01:25:24.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.519+0000 7eff96fea700 1 Processor -- start 2024-04-25T01:25:24.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.519+0000 7eff96fea700 1 -- start start 2024-04-25T01:25:24.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.519+0000 7eff96fea700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff90102290 0x7eff901985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:24.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.519+0000 7eff96fea700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff90103490 0x7eff90198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:24.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.520+0000 7eff96fea700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff90103e30 0x7eff9019fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:24.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.520+0000 7eff96fea700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff9010ded0 con 0x7eff90103490 2024-04-25T01:25:24.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.520+0000 7eff96fea700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7eff9010dd50 con 0x7eff90103e30 2024-04-25T01:25:24.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.520+0000 7eff96fea700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7eff9010e050 con 0x7eff90102290 2024-04-25T01:25:24.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.520+0000 7eff94d86700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff90102290 0x7eff901985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:24.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.520+0000 7eff94d86700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff90102290 0x7eff901985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:41672/0 (socket says 172.21.15.130:41672) 2024-04-25T01:25:24.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.520+0000 7eff94d86700 1 -- 172.21.15.130:0/3103303177 learned_addr learned my addr 172.21.15.130:0/3103303177 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-25T01:25:24.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.520+0000 7eff94d86700 1 -- 172.21.15.130:0/3103303177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff90103e30 msgr2=0x7eff9019fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:24.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.520+0000 7eff94d86700 1 --2- 172.21.15.130:0/3103303177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff90103e30 0x7eff9019fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:24.503 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.520+0000 7eff94d86700 1 -- 172.21.15.130:0/3103303177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff90103490 msgr2=0x7eff90198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:24.503 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.520+0000 7eff8ffff700 1 --2- 172.21.15.130:0/3103303177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff90103490 0x7eff90198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:24.503 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.520+0000 7eff94d86700 1 --2- 172.21.15.130:0/3103303177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff90103490 0x7eff90198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:24.503 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.520+0000 7eff94d86700 1 -- 172.21.15.130:0/3103303177 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff901a0200 con 0x7eff90102290 2024-04-25T01:25:24.503 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.520+0000 7eff8ffff700 1 --2- 172.21.15.130:0/3103303177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff90103490 0x7eff90198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:24.503 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.520+0000 7eff94d86700 1 --2- 172.21.15.130:0/3103303177 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff90102290 0x7eff901985a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7eff840058e0 tx=0x7eff84004580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:24.503 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.520+0000 7eff8dffb700 1 -- 172.21.15.130:0/3103303177 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff84027070 con 0x7eff90102290 2024-04-25T01:25:24.503 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.520+0000 7eff8dffb700 1 -- 172.21.15.130:0/3103303177 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff84022470 con 0x7eff90102290 2024-04-25T01:25:24.503 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.520+0000 7eff96fea700 1 -- 172.21.15.130:0/3103303177 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff901a0490 con 0x7eff90102290 2024-04-25T01:25:24.503 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.520+0000 7eff8dffb700 1 -- 172.21.15.130:0/3103303177 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8400f6a0 con 0x7eff90102290 2024-04-25T01:25:24.503 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.521+0000 7eff96fea700 1 -- 172.21.15.130:0/3103303177 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff901a09c0 con 0x7eff90102290 2024-04-25T01:25:24.503 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.521+0000 7eff96fea700 1 -- 172.21.15.130:0/3103303177 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff90068f20 con 0x7eff90102290 2024-04-25T01:25:24.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.522+0000 7eff8dffb700 1 -- 172.21.15.130:0/3103303177 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7eff8402a070 con 0x7eff90102290 2024-04-25T01:25:24.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.523+0000 7eff8dffb700 1 --2- 172.21.15.130:0/3103303177 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7eff800a2c10 0x7eff800a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:24.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.523+0000 7eff8dffb700 1 -- 172.21.15.130:0/3103303177 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7eff840ceca0 con 0x7eff90102290 2024-04-25T01:25:24.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.523+0000 7eff8ffff700 1 --2- 172.21.15.130:0/3103303177 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7eff800a2c10 0x7eff800a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:24.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.524+0000 7eff8ffff700 1 --2- 172.21.15.130:0/3103303177 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7eff800a2c10 0x7eff800a50d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7eff7c007900 tx=0x7eff7c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:24.506 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.527+0000 7eff8dffb700 1 -- 172.21.15.130:0/3103303177 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7eff84031070 con 0x7eff90102290 2024-04-25T01:25:24.709 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:24.730+0000 7eff96fea700 1 -- 172.21.15.130:0/3103303177 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7eff90062670 con 0x7eff800a2c10 2024-04-25T01:25:24.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:24 smithi032 ceph-mon[32461]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T01:25:24.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:24 smithi032 ceph-mon[32461]: Detected new or changed devices on smithi032 2024-04-25T01:25:24.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:24 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:24.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:24 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:24.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:24 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:24.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:24 smithi032 ceph-mon[32461]: Adjusting osd_memory_target on smithi032 to 17093M 2024-04-25T01:25:24.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:24 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:24.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:24 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:24.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:24 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:24.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:24 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:24.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:24 smithi028 ceph-mon[29390]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T01:25:24.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:24 smithi028 ceph-mon[29390]: Detected new or changed devices on smithi032 2024-04-25T01:25:24.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:24 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:24.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:24 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:24.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:24 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:24.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:24 smithi028 ceph-mon[29390]: Adjusting osd_memory_target on smithi032 to 17093M 2024-04-25T01:25:24.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:24 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:24.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:24 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:24.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:24 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:24.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:24 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:25.588 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:25 smithi130 ceph-mon[32702]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:25.588 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:25 smithi130 ceph-mon[32702]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:25.588 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:25 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:25:25.588 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:25 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:25:25.588 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:25 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:25.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:25 smithi032 ceph-mon[32461]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:25.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:25 smithi032 ceph-mon[32461]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:25.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:25 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:25:25.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:25 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:25:25.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:25 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:25.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:25 smithi028 ceph-mon[29390]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:25.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:25 smithi028 ceph-mon[29390]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:25.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:25 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:25:25.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:25 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:25:25.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:25 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:27.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:27 smithi130 ceph-mon[32702]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:27.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:27 smithi130 ceph-mon[32702]: from='client.? 172.21.15.130:0/828960206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b6e267e-fe68-4971-8f49-3e51b82d1e40"}]: dispatch 2024-04-25T01:25:27.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:27 smithi130 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b6e267e-fe68-4971-8f49-3e51b82d1e40"}]: dispatch 2024-04-25T01:25:27.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:27 smithi130 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b6e267e-fe68-4971-8f49-3e51b82d1e40"}]': finished 2024-04-25T01:25:27.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:27 smithi130 ceph-mon[32702]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T01:25:27.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:27 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:27.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:27 smithi032 ceph-mon[32461]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:27.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:27 smithi032 ceph-mon[32461]: from='client.? 172.21.15.130:0/828960206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b6e267e-fe68-4971-8f49-3e51b82d1e40"}]: dispatch 2024-04-25T01:25:27.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:27 smithi032 ceph-mon[32461]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b6e267e-fe68-4971-8f49-3e51b82d1e40"}]: dispatch 2024-04-25T01:25:27.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:27 smithi032 ceph-mon[32461]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b6e267e-fe68-4971-8f49-3e51b82d1e40"}]': finished 2024-04-25T01:25:27.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:27 smithi032 ceph-mon[32461]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T01:25:27.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:27 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:27.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:27 smithi028 ceph-mon[29390]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:27.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:27 smithi028 ceph-mon[29390]: from='client.? 172.21.15.130:0/828960206' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b6e267e-fe68-4971-8f49-3e51b82d1e40"}]: dispatch 2024-04-25T01:25:27.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:27 smithi028 ceph-mon[29390]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b6e267e-fe68-4971-8f49-3e51b82d1e40"}]: dispatch 2024-04-25T01:25:27.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:27 smithi028 ceph-mon[29390]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b6e267e-fe68-4971-8f49-3e51b82d1e40"}]': finished 2024-04-25T01:25:27.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:27 smithi028 ceph-mon[29390]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T01:25:27.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:27 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:28.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:28 smithi130 ceph-mon[32702]: from='client.? 172.21.15.130:0/2847159139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:25:28.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:28 smithi032 ceph-mon[32461]: from='client.? 172.21.15.130:0/2847159139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:25:28.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:28 smithi028 ceph-mon[29390]: from='client.? 172.21.15.130:0/2847159139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:25:29.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:29 smithi130 ceph-mon[32702]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:29.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:29 smithi032 ceph-mon[32461]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:29.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:29 smithi028 ceph-mon[29390]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:31.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:31 smithi032 ceph-mon[32461]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:31.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:31 smithi028 ceph-mon[29390]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:31.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:31 smithi130 ceph-mon[32702]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:33.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:33 smithi130 ceph-mon[32702]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:33.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:33 smithi032 ceph-mon[32461]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:33.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:33 smithi028 ceph-mon[29390]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:34.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:34 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T01:25:34.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:34 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:34.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:34 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T01:25:34.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:34 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:34.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:34 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T01:25:34.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:34 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:35.639 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:35 smithi130 ceph-mon[32702]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:35.639 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:35 smithi130 ceph-mon[32702]: Deploying daemon osd.2 on smithi130 2024-04-25T01:25:35.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:35 smithi032 ceph-mon[32461]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:35.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:35 smithi032 ceph-mon[32461]: Deploying daemon osd.2 on smithi130 2024-04-25T01:25:35.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:35 smithi028 ceph-mon[29390]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:35.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:35 smithi028 ceph-mon[29390]: Deploying daemon osd.2 on smithi130 2024-04-25T01:25:37.614 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:37 smithi130 ceph-mon[32702]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:37.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:37 smithi032 ceph-mon[32461]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:37.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:37 smithi028 ceph-mon[29390]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:39.630 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:39 smithi130 ceph-mon[32702]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:39.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:39 smithi032 ceph-mon[32461]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:39.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:39 smithi028 ceph-mon[29390]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:40.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:40 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:40.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:40 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:40.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:40 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:40.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:40 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:40.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:40 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:40.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:40 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:41.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:40 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:41.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:40 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:41.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:40 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:41.822 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:41 smithi130 ceph-mon[32702]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:41.822 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:41 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:41.822 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:41 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:41.822 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:41 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:41.822 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:41 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:41.822 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:41 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:41.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:41 smithi032 ceph-mon[32461]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:41.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:41 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:41.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:41 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:41.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:41 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:41.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:41 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:41.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:41 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:42.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:41 smithi028 ceph-mon[29390]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:42.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:41 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:42.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:41 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:42.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:41 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:42.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:41 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:42.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:41 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:43.381 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 2 on host 'smithi130' 2024-04-25T01:25:43.383 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:43.399+0000 7eff8dffb700 1 -- 172.21.15.130:0/3103303177 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7eff90062670 con 0x7eff800a2c10 2024-04-25T01:25:43.383 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:43.401+0000 7eff96fea700 1 -- 172.21.15.130:0/3103303177 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7eff800a2c10 msgr2=0x7eff800a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:43.383 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:43.401+0000 7eff96fea700 1 --2- 172.21.15.130:0/3103303177 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7eff800a2c10 0x7eff800a50d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7eff7c007900 tx=0x7eff7c008040 comp rx=0 tx=0).stop 2024-04-25T01:25:43.383 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:43.401+0000 7eff96fea700 1 -- 172.21.15.130:0/3103303177 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff90102290 msgr2=0x7eff901985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:43.383 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:43.401+0000 7eff96fea700 1 --2- 172.21.15.130:0/3103303177 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff90102290 0x7eff901985a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7eff840058e0 tx=0x7eff84004580 comp rx=0 tx=0).stop 2024-04-25T01:25:43.383 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:43.401+0000 7eff96fea700 1 -- 172.21.15.130:0/3103303177 shutdown_connections 2024-04-25T01:25:43.383 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:43.401+0000 7eff96fea700 1 --2- 172.21.15.130:0/3103303177 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7eff800a2c10 0x7eff800a50d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:43.384 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:43.401+0000 7eff96fea700 1 --2- 172.21.15.130:0/3103303177 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff90103e30 0x7eff9019fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:43.384 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:43.401+0000 7eff96fea700 1 --2- 172.21.15.130:0/3103303177 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff90102290 0x7eff901985a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:43.384 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:43.401+0000 7eff96fea700 1 --2- 172.21.15.130:0/3103303177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff90103490 0x7eff90198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:43.384 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:43.401+0000 7eff96fea700 1 -- 172.21.15.130:0/3103303177 >> 172.21.15.130:0/3103303177 conn(0x7eff900fd830 msgr2=0x7eff90106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:43.384 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:43.402+0000 7eff96fea700 1 -- 172.21.15.130:0/3103303177 shutdown_connections 2024-04-25T01:25:43.384 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:25:43.402+0000 7eff96fea700 1 -- 172.21.15.130:0/3103303177 wait complete. 2024-04-25T01:25:43.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:43 smithi130 ceph-mon[32702]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:43.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:43 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:43.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:43 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:43.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:43 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:43.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:43 smithi130 ceph-mon[32702]: from='osd.2 [v2:172.21.15.130:6800/3497329111,v1:172.21.15.130:6801/3497329111]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:25:43.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:43 smithi130 ceph-mon[32702]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:25:43.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:43 smithi032 ceph-mon[32461]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:43.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:43 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:43.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:43 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:43.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:43 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:43.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:43 smithi032 ceph-mon[32461]: from='osd.2 [v2:172.21.15.130:6800/3497329111,v1:172.21.15.130:6801/3497329111]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:25:43.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:43 smithi032 ceph-mon[32461]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:25:44.060 DEBUG:teuthology.orchestra.run.smithi130:osd.2> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.2.service 2024-04-25T01:25:44.062 INFO:tasks.cephadm:Deploying osd.3 on smithi028 with /dev/vg_nvme/lv_3... 2024-04-25T01:25:44.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T01:25:44.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:43 smithi028 ceph-mon[29390]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:44.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:43 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:44.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:43 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:44.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:43 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:25:44.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:43 smithi028 ceph-mon[29390]: from='osd.2 [v2:172.21.15.130:6800/3497329111,v1:172.21.15.130:6801/3497329111]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:25:44.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:43 smithi028 ceph-mon[29390]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:25:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:45.640 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:45 smithi130 ceph-mon[32702]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:45.641 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:45 smithi130 ceph-mon[32702]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T01:25:45.641 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:45 smithi130 ceph-mon[32702]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T01:25:45.641 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:45 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:45.641 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:45 smithi130 ceph-mon[32702]: from='osd.2 [v2:172.21.15.130:6800/3497329111,v1:172.21.15.130:6801/3497329111]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:25:45.641 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:45 smithi130 ceph-mon[32702]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:25:45.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:45 smithi028 ceph-mon[29390]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:45.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:45 smithi028 ceph-mon[29390]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T01:25:45.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:45 smithi028 ceph-mon[29390]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T01:25:45.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:45 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:45.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:45 smithi028 ceph-mon[29390]: from='osd.2 [v2:172.21.15.130:6800/3497329111,v1:172.21.15.130:6801/3497329111]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:25:45.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:45 smithi028 ceph-mon[29390]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:25:45.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:45 smithi032 ceph-mon[32461]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:45.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:45 smithi032 ceph-mon[32461]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T01:25:45.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:45 smithi032 ceph-mon[32461]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T01:25:45.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:45 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:45.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:45 smithi032 ceph-mon[32461]: from='osd.2 [v2:172.21.15.130:6800/3497329111,v1:172.21.15.130:6801/3497329111]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:25:45.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:45 smithi032 ceph-mon[32461]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:25:46.349 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:25:46 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-2[37842]: 2024-04-25T01:25:46.055+0000 7f4b4c537700 -1 osd.2 0 waiting for initial osdmap 2024-04-25T01:25:46.349 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:25:46 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-2[37842]: 2024-04-25T01:25:46.060+0000 7f4b44322700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:25:46.396 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T01:25:46.396 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T01:25:46.396 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-25T01:25:46.396 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-25T01:25:46.396 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00875677 s, 1.2 GB/s 2024-04-25T01:25:46.396 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-25T01:25:46.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:46 smithi028 ceph-mon[29390]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-25T01:25:46.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:46 smithi028 ceph-mon[29390]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T01:25:46.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:46 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:46.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:46 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:46.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:46 smithi028 ceph-mon[29390]: from='osd.2 ' entity='osd.2' 2024-04-25T01:25:46.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:46 smithi130 ceph-mon[32702]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-25T01:25:46.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:46 smithi130 ceph-mon[32702]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T01:25:46.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:46 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:46.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:46 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:46.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:46 smithi130 ceph-mon[32702]: from='osd.2 ' entity='osd.2' 2024-04-25T01:25:46.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:46 smithi032 ceph-mon[32461]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-25T01:25:46.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:46 smithi032 ceph-mon[32461]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T01:25:46.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:46 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:46.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:46 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:46.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:46 smithi032 ceph-mon[32461]: from='osd.2 ' entity='osd.2' 2024-04-25T01:25:47.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_3 2024-04-25T01:25:47.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:47 smithi028 ceph-mon[29390]: purged_snaps scrub starts 2024-04-25T01:25:47.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:47 smithi028 ceph-mon[29390]: purged_snaps scrub ok 2024-04-25T01:25:47.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:47 smithi028 ceph-mon[29390]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:47.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:47 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:47.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:47 smithi028 ceph-mon[29390]: osd.2 [v2:172.21.15.130:6800/3497329111,v1:172.21.15.130:6801/3497329111] boot 2024-04-25T01:25:47.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:47 smithi028 ceph-mon[29390]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T01:25:47.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:47 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:47.351 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:47 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:47.380 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:25:47.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:47 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:47.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:47 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:47.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:47 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:47.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:47 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:47.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:47 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:47.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:47 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:47.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:47 smithi130 ceph-mon[32702]: purged_snaps scrub starts 2024-04-25T01:25:47.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:47 smithi130 ceph-mon[32702]: purged_snaps scrub ok 2024-04-25T01:25:47.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:47 smithi130 ceph-mon[32702]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:47.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:47 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:47.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:47 smithi130 ceph-mon[32702]: osd.2 [v2:172.21.15.130:6800/3497329111,v1:172.21.15.130:6801/3497329111] boot 2024-04-25T01:25:47.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:47 smithi130 ceph-mon[32702]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T01:25:47.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:47 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:47.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:47 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:47.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:47 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:47.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:47 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:47.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:47 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:47.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:47 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:47.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:47 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:47.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:47 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:47.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:47 smithi032 ceph-mon[32461]: purged_snaps scrub starts 2024-04-25T01:25:47.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:47 smithi032 ceph-mon[32461]: purged_snaps scrub ok 2024-04-25T01:25:47.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:47 smithi032 ceph-mon[32461]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:25:47.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:47 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:47.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:47 smithi032 ceph-mon[32461]: osd.2 [v2:172.21.15.130:6800/3497329111,v1:172.21.15.130:6801/3497329111] boot 2024-04-25T01:25:47.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:47 smithi032 ceph-mon[32461]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T01:25:47.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:47 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:25:47.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:47 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:47.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:47 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:47.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:47 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:25:47.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:47 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:47.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:47 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:47.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:47 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:25:47.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:47 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:48.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.004+0000 7f9a61a64700 1 -- 172.21.15.28:0/1430869207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c103e30 msgr2=0x7f9a5c10a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:48.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.004+0000 7f9a61a64700 1 --2- 172.21.15.28:0/1430869207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c103e30 0x7f9a5c10a6e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400b0d0 tx=0x7f9a5400b3e0 comp rx=0 tx=0).stop 2024-04-25T01:25:48.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.004+0000 7f9a61a64700 1 -- 172.21.15.28:0/1430869207 shutdown_connections 2024-04-25T01:25:48.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.004+0000 7f9a61a64700 1 --2- 172.21.15.28:0/1430869207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c103e30 0x7f9a5c10a6e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:48.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.004+0000 7f9a61a64700 1 --2- 172.21.15.28:0/1430869207 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a5c102290 0x7f9a5c1026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:48.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.004+0000 7f9a61a64700 1 --2- 172.21.15.28:0/1430869207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a5c103490 0x7f9a5c1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:48.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.004+0000 7f9a61a64700 1 -- 172.21.15.28:0/1430869207 >> 172.21.15.28:0/1430869207 conn(0x7f9a5c0fd830 msgr2=0x7f9a5c0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:25:48.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.004+0000 7f9a61a64700 1 -- 172.21.15.28:0/1430869207 shutdown_connections 2024-04-25T01:25:48.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.004+0000 7f9a61a64700 1 -- 172.21.15.28:0/1430869207 wait complete. 2024-04-25T01:25:48.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.004+0000 7f9a61a64700 1 Processor -- start 2024-04-25T01:25:48.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.005+0000 7f9a61a64700 1 -- start start 2024-04-25T01:25:48.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.005+0000 7f9a61a64700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c102290 0x7f9a5c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:48.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.005+0000 7f9a61a64700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a5c103490 0x7f9a5c198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:48.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.005+0000 7f9a61a64700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a5c103e30 0x7f9a5c19f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:48.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.005+0000 7f9a61a64700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c10de50 con 0x7f9a5c102290 2024-04-25T01:25:48.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.005+0000 7f9a61a64700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c10dcd0 con 0x7f9a5c103e30 2024-04-25T01:25:48.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.005+0000 7f9a61a64700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c10dfd0 con 0x7f9a5c103490 2024-04-25T01:25:48.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.005+0000 7f9a5b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c102290 0x7f9a5c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:48.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.005+0000 7f9a5b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c102290 0x7f9a5c1983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40398/0 (socket says 172.21.15.28:40398) 2024-04-25T01:25:48.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.005+0000 7f9a5b7fe700 1 -- 172.21.15.28:0/3746112088 learned_addr learned my addr 172.21.15.28:0/3746112088 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:25:48.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.005+0000 7f9a5bfff700 1 --2- 172.21.15.28:0/3746112088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a5c103e30 0x7f9a5c19f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:48.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.005+0000 7f9a5affd700 1 --2- 172.21.15.28:0/3746112088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a5c103490 0x7f9a5c198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:48.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.005+0000 7f9a5b7fe700 1 -- 172.21.15.28:0/3746112088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a5c103490 msgr2=0x7f9a5c198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:48.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.005+0000 7f9a5b7fe700 1 --2- 172.21.15.28:0/3746112088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a5c103490 0x7f9a5c198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:48.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.005+0000 7f9a5b7fe700 1 -- 172.21.15.28:0/3746112088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a5c103e30 msgr2=0x7f9a5c19f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:25:48.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.005+0000 7f9a5b7fe700 1 --2- 172.21.15.28:0/3746112088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a5c103e30 0x7f9a5c19f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:25:48.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.005+0000 7f9a5b7fe700 1 -- 172.21.15.28:0/3746112088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a5c1a0030 con 0x7f9a5c102290 2024-04-25T01:25:48.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.005+0000 7f9a5bfff700 1 --2- 172.21.15.28:0/3746112088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a5c103e30 0x7f9a5c19f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:25:48.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.005+0000 7f9a5b7fe700 1 --2- 172.21.15.28:0/3746112088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c102290 0x7f9a5c1983d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c00e9a0 tx=0x7f9a4c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:48.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.006+0000 7f9a58ff9700 1 -- 172.21.15.28:0/3746112088 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c00cbd0 con 0x7f9a5c102290 2024-04-25T01:25:48.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.006+0000 7f9a61a64700 1 -- 172.21.15.28:0/3746112088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a5c1a0320 con 0x7f9a5c102290 2024-04-25T01:25:48.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.006+0000 7f9a58ff9700 1 -- 172.21.15.28:0/3746112088 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a4c004500 con 0x7f9a5c102290 2024-04-25T01:25:48.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.006+0000 7f9a58ff9700 1 -- 172.21.15.28:0/3746112088 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c010430 con 0x7f9a5c102290 2024-04-25T01:25:48.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.006+0000 7f9a61a64700 1 -- 172.21.15.28:0/3746112088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a5c1a08b0 con 0x7f9a5c102290 2024-04-25T01:25:48.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.007+0000 7f9a61a64700 1 -- 172.21.15.28:0/3746112088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a5c068f20 con 0x7f9a5c102290 2024-04-25T01:25:48.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.008+0000 7f9a58ff9700 1 -- 172.21.15.28:0/3746112088 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f9a4c003f40 con 0x7f9a5c102290 2024-04-25T01:25:48.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.008+0000 7f9a58ff9700 1 --2- 172.21.15.28:0/3746112088 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9a440a2b90 0x7f9a440a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:25:48.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.009+0000 7f9a5affd700 1 --2- 172.21.15.28:0/3746112088 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9a440a2b90 0x7f9a440a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:25:48.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.009+0000 7f9a58ff9700 1 -- 172.21.15.28:0/3746112088 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7f9a4c0cdf60 con 0x7f9a5c102290 2024-04-25T01:25:48.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.009+0000 7f9a5affd700 1 --2- 172.21.15.28:0/3746112088 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9a440a2b90 0x7f9a440a5050 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9a50005950 tx=0x7f9a50005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:25:48.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.012+0000 7f9a58ff9700 1 -- 172.21.15.28:0/3746112088 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9a4c01f340 con 0x7f9a5c102290 2024-04-25T01:25:49.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:48 smithi028 ceph-mon[29390]: Detected new or changed devices on smithi130 2024-04-25T01:25:49.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:48 smithi028 ceph-mon[29390]: Adjusting osd_memory_target on smithi130 to 17088M 2024-04-25T01:25:49.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:49.228+0000 7f9a61a64700 1 -- 172.21.15.28:0/3746112088 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f9a5c062670 con 0x7f9a440a2b90 2024-04-25T01:25:49.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:48 smithi130 ceph-mon[32702]: Detected new or changed devices on smithi130 2024-04-25T01:25:49.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:48 smithi130 ceph-mon[32702]: Adjusting osd_memory_target on smithi130 to 17088M 2024-04-25T01:25:49.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:48 smithi032 ceph-mon[32461]: Detected new or changed devices on smithi130 2024-04-25T01:25:49.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:48 smithi032 ceph-mon[32461]: Adjusting osd_memory_target on smithi130 to 17088M 2024-04-25T01:25:50.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:49 smithi028 ceph-mon[29390]: pgmap v82: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:25:50.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:49 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:25:50.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:49 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T01:25:50.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:49 smithi028 ceph-mon[29390]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T01:25:50.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:49 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:25:50.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:49 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:25:50.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:49 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:25:50.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:49 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:50.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:49 smithi130 ceph-mon[32702]: pgmap v82: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:25:50.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:49 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:25:50.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:49 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T01:25:50.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:49 smithi130 ceph-mon[32702]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T01:25:50.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:49 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:25:50.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:49 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:25:50.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:49 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:25:50.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:49 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:50.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:49 smithi032 ceph-mon[32461]: pgmap v82: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:25:50.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:49 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:25:50.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:49 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T01:25:50.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:49 smithi032 ceph-mon[32461]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T01:25:50.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:49 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:25:50.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:49 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:25:50.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:49 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:25:50.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:49 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:25:50.313 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:25:50 smithi028 sudo[46349]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:25:50.313 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:25:50 smithi028 sudo[46349]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:25:50.313 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:25:50 smithi028 sudo[46349]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:25:50.314 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:25:50 smithi028 sudo[46349]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:25:50.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:25:50.789+0000 7f9a58ff9700 1 -- 172.21.15.28:0/3746112088 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9a4c098380 con 0x7f9a5c102290 2024-04-25T01:25:51.001 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:50 smithi028 ceph-mon[29390]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:51.001 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:50 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:25:51.001 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:50 smithi028 ceph-mon[29390]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T01:25:51.002 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:25:50 smithi028 sudo[46355]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:25:51.002 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:25:50 smithi028 sudo[46355]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:25:51.002 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:25:50 smithi028 sudo[46355]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:25:51.002 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:25:50 smithi028 sudo[46355]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:25:51.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:50 smithi130 ceph-mon[32702]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:51.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:50 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:25:51.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:50 smithi130 ceph-mon[32702]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T01:25:51.227 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:50 smithi032 ceph-mon[32461]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:25:51.228 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:50 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:25:51.228 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:50 smithi032 ceph-mon[32461]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T01:25:51.486 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:25:51 smithi032 sudo[41193]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:25:51.486 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:25:51 smithi032 sudo[41193]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:25:51.486 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:25:51 smithi032 sudo[41193]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:25:51.486 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:25:51 smithi032 sudo[41193]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:25:51.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:51 smithi032 ceph-mon[32461]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:25:51.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:51 smithi032 ceph-mon[32461]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T01:25:51.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:51 smithi032 ceph-mon[32461]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T01:25:51.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:51 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:51.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:51 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:51.986 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:25:51 smithi032 sudo[41196]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:25:51.987 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:25:51 smithi032 sudo[41196]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:25:51.987 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:25:51 smithi032 sudo[41196]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:25:51.987 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:25:51 smithi032 sudo[41196]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:25:52.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:51 smithi028 ceph-mon[29390]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:25:52.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:51 smithi028 ceph-mon[29390]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T01:25:52.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:51 smithi028 ceph-mon[29390]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T01:25:52.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:51 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:52.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:51 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:52.211 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:25:51 smithi130 sudo[41453]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:25:52.211 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:25:51 smithi130 sudo[41453]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:25:52.211 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:25:51 smithi130 sudo[41453]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:25:52.211 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:25:51 smithi130 sudo[41453]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:25:52.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:51 smithi130 ceph-mon[32702]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:25:52.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:51 smithi130 ceph-mon[32702]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T01:25:52.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:51 smithi130 ceph-mon[32702]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T01:25:52.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:51 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:52.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:51 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:52.711 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:25:52 smithi130 sudo[41456]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:25:52.711 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:25:52 smithi130 sudo[41456]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:25:52.711 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:25:52 smithi130 sudo[41456]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:25:52.711 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:25:52 smithi130 sudo[41456]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:25:53.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:52 smithi028 sudo[46455]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:25:53.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:52 smithi028 sudo[46455]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:25:53.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:52 smithi028 sudo[46455]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:25:53.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:52 smithi028 sudo[46455]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:25:53.396 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:53 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1236951319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd309ba8-012b-4058-bcc0-57f61f41c810"}]: dispatch 2024-04-25T01:25:53.396 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:53 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1236951319' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd309ba8-012b-4058-bcc0-57f61f41c810"}]': finished 2024-04-25T01:25:53.396 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:53 smithi130 ceph-mon[32702]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T01:25:53.396 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:25:53.397 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:53.397 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:53 smithi130 ceph-mon[32702]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:25:53.397 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:53 smithi130 sudo[41459]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:25:53.397 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:53 smithi130 sudo[41459]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:25:53.397 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:53 smithi130 sudo[41459]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:25:53.397 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:53 smithi130 sudo[41459]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:25:53.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:53 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1236951319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd309ba8-012b-4058-bcc0-57f61f41c810"}]: dispatch 2024-04-25T01:25:53.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:53 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1236951319' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd309ba8-012b-4058-bcc0-57f61f41c810"}]': finished 2024-04-25T01:25:53.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:53 smithi032 ceph-mon[32461]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T01:25:53.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:25:53.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:53.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:53 smithi032 ceph-mon[32461]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:25:53.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:53 smithi028 sudo[46458]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:25:53.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:53 smithi028 sudo[46458]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:25:53.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:53 smithi028 sudo[46458]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:25:53.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:53 smithi028 sudo[46458]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:25:53.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:53 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1236951319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd309ba8-012b-4058-bcc0-57f61f41c810"}]: dispatch 2024-04-25T01:25:53.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:53 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1236951319' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd309ba8-012b-4058-bcc0-57f61f41c810"}]': finished 2024-04-25T01:25:53.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:53 smithi028 ceph-mon[29390]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T01:25:53.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:25:53.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:25:53.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:53 smithi028 ceph-mon[29390]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:25:53.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:53 smithi130 sudo[41462]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:25:53.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:53 smithi130 sudo[41462]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:25:53.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:53 smithi130 sudo[41462]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:25:53.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:53 smithi130 sudo[41462]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:25:54.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 sudo[41199]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:25:54.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 sudo[41199]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:25:54.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 sudo[41199]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:25:54.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 sudo[41199]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:25:54.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:54 smithi028 ceph-mon[29390]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:25:54.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:54 smithi028 ceph-mon[29390]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:25:54.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:54 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:25:54.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:54 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:25:54.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:54 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:25:54.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:54 smithi028 ceph-mon[29390]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:25:54.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:54 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:25:54.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:54 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:25:54.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:54 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:25:54.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:54 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1919435625' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:25:54.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:54 smithi028 ceph-mon[29390]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:25:54.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:54 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:25:54.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:54 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:25:54.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:54 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:25:54.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:54 smithi130 ceph-mon[32702]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:25:54.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:54 smithi130 ceph-mon[32702]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:25:54.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:54 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:25:54.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:54 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:25:54.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:54 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:25:54.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:54 smithi130 ceph-mon[32702]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:25:54.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:54 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:25:54.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:54 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:25:54.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:54 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:25:54.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:54 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1919435625' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:25:54.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:54 smithi130 ceph-mon[32702]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:25:54.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:54 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:25:54.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:54 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:25:54.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:54 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:25:54.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 sudo[41202]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:25:54.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 sudo[41202]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:25:54.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 sudo[41202]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:25:54.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 sudo[41202]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:25:54.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 ceph-mon[32461]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:25:54.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 ceph-mon[32461]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:25:54.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:25:54.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:25:54.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:25:54.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 ceph-mon[32461]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:25:54.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:25:54.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:25:54.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:25:54.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1919435625' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:25:54.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 ceph-mon[32461]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:25:54.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:25:54.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:25:54.988 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:54 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:25:55.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:55 smithi028 ceph-mon[29390]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:25:55.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:55 smithi028 ceph-mon[29390]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:25:55.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:55 smithi028 ceph-mon[29390]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:25:55.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:55 smithi130 ceph-mon[32702]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:25:55.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:55 smithi130 ceph-mon[32702]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:25:55.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:55 smithi130 ceph-mon[32702]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:25:55.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:55 smithi032 ceph-mon[32461]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:25:55.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:55 smithi032 ceph-mon[32461]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:25:55.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:55 smithi032 ceph-mon[32461]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:25:57.807 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:57 smithi028 ceph-mon[29390]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:25:57.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:57 smithi130 ceph-mon[32702]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:25:57.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:57 smithi032 ceph-mon[32461]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:25:59.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:25:59 smithi028 ceph-mon[29390]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:25:59.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:25:59 smithi130 ceph-mon[32702]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:25:59.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:25:59 smithi032 ceph-mon[32461]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:26:00.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T01:26:00.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:00.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:00 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T01:26:00.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:00 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:00.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:00 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T01:26:00.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:00 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:01.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:01 smithi028 ceph-mon[29390]: Deploying daemon osd.3 on smithi028 2024-04-25T01:26:01.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:01 smithi028 ceph-mon[29390]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-04-25T01:26:01.887 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:01 smithi032 ceph-mon[32461]: Deploying daemon osd.3 on smithi028 2024-04-25T01:26:01.887 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:01 smithi032 ceph-mon[32461]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-04-25T01:26:01.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:01 smithi130 ceph-mon[32702]: Deploying daemon osd.3 on smithi028 2024-04-25T01:26:01.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:01 smithi130 ceph-mon[32702]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-04-25T01:26:03.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:03 smithi028 ceph-mon[29390]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-04-25T01:26:03.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:03 smithi130 ceph-mon[32702]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-04-25T01:26:03.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:03 smithi032 ceph-mon[32461]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-04-25T01:26:05.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:05 smithi028 ceph-mon[29390]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-04-25T01:26:05.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:05 smithi130 ceph-mon[32702]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-04-25T01:26:05.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:05 smithi032 ceph-mon[32461]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-04-25T01:26:06.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:06.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:06.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:06.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:06 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:06.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:06 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:06.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:06 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:07.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:06 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:07.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:06 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:07.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:06 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:07.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:07 smithi130 ceph-mon[32702]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-04-25T01:26:07.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:07 smithi032 ceph-mon[32461]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-04-25T01:26:08.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:07 smithi028 ceph-mon[29390]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-04-25T01:26:09.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:09 smithi028 ceph-mon[29390]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-04-25T01:26:09.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:09 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:09.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:09 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:09.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:09 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:09.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:09 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:26:09.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:09 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:26:09.487+0000 7f9a58ff9700 1 -- 172.21.15.28:0/3746112088 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9a5c062670 con 0x7f9a440a2b90 2024-04-25T01:26:09.467 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 3 on host 'smithi028' 2024-04-25T01:26:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:26:09.489+0000 7f9a61a64700 1 -- 172.21.15.28:0/3746112088 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9a440a2b90 msgr2=0x7f9a440a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:26:09.489+0000 7f9a61a64700 1 --2- 172.21.15.28:0/3746112088 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9a440a2b90 0x7f9a440a5050 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9a50005950 tx=0x7f9a50005cf0 comp rx=0 tx=0).stop 2024-04-25T01:26:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:26:09.489+0000 7f9a61a64700 1 -- 172.21.15.28:0/3746112088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c102290 msgr2=0x7f9a5c1983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:26:09.489+0000 7f9a61a64700 1 --2- 172.21.15.28:0/3746112088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c102290 0x7f9a5c1983d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c00e9a0 tx=0x7f9a4c00ed60 comp rx=0 tx=0).stop 2024-04-25T01:26:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:26:09.489+0000 7f9a61a64700 1 -- 172.21.15.28:0/3746112088 shutdown_connections 2024-04-25T01:26:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:26:09.489+0000 7f9a61a64700 1 --2- 172.21.15.28:0/3746112088 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9a440a2b90 0x7f9a440a5050 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:26:09.489+0000 7f9a61a64700 1 --2- 172.21.15.28:0/3746112088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a5c103e30 0x7f9a5c19f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:26:09.489+0000 7f9a61a64700 1 --2- 172.21.15.28:0/3746112088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c102290 0x7f9a5c1983d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:26:09.489+0000 7f9a61a64700 1 --2- 172.21.15.28:0/3746112088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a5c103490 0x7f9a5c198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:26:09.489+0000 7f9a61a64700 1 -- 172.21.15.28:0/3746112088 >> 172.21.15.28:0/3746112088 conn(0x7f9a5c0fd830 msgr2=0x7f9a5c106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:26:09.490+0000 7f9a61a64700 1 -- 172.21.15.28:0/3746112088 shutdown_connections 2024-04-25T01:26:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:26:09.490+0000 7f9a61a64700 1 -- 172.21.15.28:0/3746112088 wait complete. 2024-04-25T01:26:09.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:09 smithi130 ceph-mon[32702]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-04-25T01:26:09.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:09 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:09.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:09 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:09.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:09 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:09.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:09 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:26:09.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:09 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:09.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:09 smithi032 ceph-mon[32461]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-04-25T01:26:09.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:09 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:09.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:09 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:09.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:09 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:09.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:09 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:26:09.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:09 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:10.117 DEBUG:teuthology.orchestra.run.smithi028:osd.3> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.3.service 2024-04-25T01:26:10.120 INFO:tasks.cephadm:Deploying osd.4 on smithi032 with /dev/vg_nvme/lv_3... 2024-04-25T01:26:10.120 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T01:26:10.374 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:26:10.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:10 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:10.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:10 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:10.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:10 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:10.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:10 smithi028 ceph-mon[29390]: from='osd.3 [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T01:26:10.682 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:10 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:10.682 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:10 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:10.682 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:10 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:10.682 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:10 smithi032 ceph-mon[32461]: from='osd.3 [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T01:26:10.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:10 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:10.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:10 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:10.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:10 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:10.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:10 smithi130 ceph-mon[32702]: from='osd.3 [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T01:26:11.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:11 smithi028 ceph-mon[29390]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-04-25T01:26:11.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:11 smithi028 ceph-mon[29390]: from='osd.3 [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T01:26:11.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:11 smithi028 ceph-mon[29390]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T01:26:11.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:11 smithi028 ceph-mon[29390]: from='osd.3 [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-25T01:26:11.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:11 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:26:11.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:11 smithi130 ceph-mon[32702]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-04-25T01:26:11.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:11 smithi130 ceph-mon[32702]: from='osd.3 [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T01:26:11.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:11 smithi130 ceph-mon[32702]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T01:26:11.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:11 smithi130 ceph-mon[32702]: from='osd.3 [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-25T01:26:11.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:11 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:26:11.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:11 smithi032 ceph-mon[32461]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-04-25T01:26:11.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:11 smithi032 ceph-mon[32461]: from='osd.3 [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T01:26:11.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:11 smithi032 ceph-mon[32461]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T01:26:11.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:11 smithi032 ceph-mon[32461]: from='osd.3 [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-25T01:26:11.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:11 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:26:12.313 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:26:12 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-3[48485]: 2024-04-25T01:26:12.029+0000 7fbe59c36700 -1 osd.3 0 waiting for initial osdmap 2024-04-25T01:26:12.314 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:26:12 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-3[48485]: 2024-04-25T01:26:12.034+0000 7fbe51a21700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:26:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T01:26:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T01:26:12.501 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-04-25T01:26:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-04-25T01:26:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00825751 s, 1.3 GB/s 2024-04-25T01:26:12.501 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-04-25T01:26:12.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:12 smithi130 ceph-mon[32702]: from='osd.3 [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-25T01:26:12.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:12 smithi130 ceph-mon[32702]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T01:26:12.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:12 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:26:12.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:12 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:26:12.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:12 smithi130 ceph-mon[32702]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T01:26:12.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:12 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:26:12.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:12 smithi130 ceph-mon[32702]: from='osd.3 [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927]' entity='osd.3' 2024-04-25T01:26:12.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:12 smithi032 ceph-mon[32461]: from='osd.3 [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-25T01:26:12.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:12 smithi032 ceph-mon[32461]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T01:26:12.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:12 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:26:12.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:12 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:26:12.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:12 smithi032 ceph-mon[32461]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T01:26:12.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:12 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:26:12.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:12 smithi032 ceph-mon[32461]: from='osd.3 [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927]' entity='osd.3' 2024-04-25T01:26:12.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:12 smithi028 ceph-mon[29390]: from='osd.3 [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-25T01:26:12.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:12 smithi028 ceph-mon[29390]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T01:26:12.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:12 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:26:12.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:12 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:26:12.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:12 smithi028 ceph-mon[29390]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T01:26:12.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:12 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:26:12.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:12 smithi028 ceph-mon[29390]: from='osd.3 [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927]' entity='osd.3' 2024-04-25T01:26:13.156 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_3 2024-04-25T01:26:13.404 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:26:13.697 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:13 smithi032 ceph-mon[32461]: purged_snaps scrub starts 2024-04-25T01:26:13.698 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:13 smithi032 ceph-mon[32461]: purged_snaps scrub ok 2024-04-25T01:26:13.698 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:13 smithi032 ceph-mon[32461]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:26:13.698 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:26:13.698 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:13 smithi032 ceph-mon[32461]: osd.3 [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927] boot 2024-04-25T01:26:13.698 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:13 smithi032 ceph-mon[32461]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T01:26:13.698 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:26:13.698 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:13.698 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:13.698 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:26:13.698 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:26:13.698 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:13.698 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:13.699 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:26:13.699 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:13.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:13 smithi130 ceph-mon[32702]: purged_snaps scrub starts 2024-04-25T01:26:13.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:13 smithi130 ceph-mon[32702]: purged_snaps scrub ok 2024-04-25T01:26:13.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:13 smithi130 ceph-mon[32702]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:26:13.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:26:13.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:13 smithi130 ceph-mon[32702]: osd.3 [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927] boot 2024-04-25T01:26:13.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:13 smithi130 ceph-mon[32702]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T01:26:13.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:26:13.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:13.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:13.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:26:13.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:26:13.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:13.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:13.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:26:13.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:13.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:13 smithi028 ceph-mon[29390]: purged_snaps scrub starts 2024-04-25T01:26:13.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:13 smithi028 ceph-mon[29390]: purged_snaps scrub ok 2024-04-25T01:26:13.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:13 smithi028 ceph-mon[29390]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:26:13.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:26:13.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:13 smithi028 ceph-mon[29390]: osd.3 [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927] boot 2024-04-25T01:26:13.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:13 smithi028 ceph-mon[29390]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T01:26:13.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:26:13.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:13.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:13.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:26:13.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:26:13.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:13.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:13.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:26:13.815 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:14.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:14 smithi028 ceph-mon[29390]: Detected new or changed devices on smithi028 2024-04-25T01:26:14.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:14 smithi028 ceph-mon[29390]: Adjusting osd_memory_target on smithi028 to 8546M 2024-04-25T01:26:14.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:14 smithi028 ceph-mon[29390]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T01:26:14.877 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:14 smithi032 ceph-mon[32461]: Detected new or changed devices on smithi028 2024-04-25T01:26:14.877 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:14 smithi032 ceph-mon[32461]: Adjusting osd_memory_target on smithi028 to 8546M 2024-04-25T01:26:14.877 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:14 smithi032 ceph-mon[32461]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T01:26:14.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:14 smithi130 ceph-mon[32702]: Detected new or changed devices on smithi028 2024-04-25T01:26:14.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:14 smithi130 ceph-mon[32702]: Adjusting osd_memory_target on smithi028 to 8546M 2024-04-25T01:26:14.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:14 smithi130 ceph-mon[32702]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T01:26:15.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.119+0000 7ff205f1b700 1 -- 172.21.15.32:0/2695825720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff200102d50 msgr2=0x7ff200109600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:15.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.119+0000 7ff205f1b700 1 --2- 172.21.15.32:0/2695825720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff200102d50 0x7ff200109600 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8009fc0 tx=0x7ff1f800b330 comp rx=0 tx=0).stop 2024-04-25T01:26:15.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.119+0000 7ff205f1b700 1 -- 172.21.15.32:0/2695825720 shutdown_connections 2024-04-25T01:26:15.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.119+0000 7ff205f1b700 1 --2- 172.21.15.32:0/2695825720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff200102d50 0x7ff200109600 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:15.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.119+0000 7ff205f1b700 1 --2- 172.21.15.32:0/2695825720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff2001011b0 0x7ff2001015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:15.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.119+0000 7ff205f1b700 1 --2- 172.21.15.32:0/2695825720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2001023b0 0x7ff200102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:15.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.119+0000 7ff205f1b700 1 -- 172.21.15.32:0/2695825720 >> 172.21.15.32:0/2695825720 conn(0x7ff2000fc750 msgr2=0x7ff2000feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:15.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.119+0000 7ff205f1b700 1 -- 172.21.15.32:0/2695825720 shutdown_connections 2024-04-25T01:26:15.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.119+0000 7ff205f1b700 1 -- 172.21.15.32:0/2695825720 wait complete. 2024-04-25T01:26:15.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.119+0000 7ff205f1b700 1 Processor -- start 2024-04-25T01:26:15.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.120+0000 7ff205f1b700 1 -- start start 2024-04-25T01:26:15.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.120+0000 7ff205f1b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2001011b0 0x7ff20006b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:15.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.120+0000 7ff205f1b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2001023b0 0x7ff20019d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:15.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.120+0000 7ff205f1b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff200102d50 0x7ff20019f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:15.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.120+0000 7ff205f1b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff20010cd80 con 0x7ff2001011b0 2024-04-25T01:26:15.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.120+0000 7ff205f1b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff20010cc00 con 0x7ff2001023b0 2024-04-25T01:26:15.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.120+0000 7ff205f1b700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff20010cf00 con 0x7ff200102d50 2024-04-25T01:26:15.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.120+0000 7ff1feffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2001023b0 0x7ff20019d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:15.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.120+0000 7ff1ff7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2001011b0 0x7ff20006b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:15.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.120+0000 7ff1ff7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2001011b0 0x7ff20006b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.32:55392/0 (socket says 172.21.15.32:55392) 2024-04-25T01:26:15.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.120+0000 7ff1feffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2001023b0 0x7ff20019d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50360/0 (socket says 172.21.15.32:50360) 2024-04-25T01:26:15.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.120+0000 7ff1ff7fe700 1 -- 172.21.15.32:0/3024038368 learned_addr learned my addr 172.21.15.32:0/3024038368 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-25T01:26:15.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.120+0000 7ff1fffff700 1 --2- 172.21.15.32:0/3024038368 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff200102d50 0x7ff20019f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:15.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.120+0000 7ff1ff7fe700 1 -- 172.21.15.32:0/3024038368 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff200102d50 msgr2=0x7ff20019f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:15.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.120+0000 7ff1ff7fe700 1 --2- 172.21.15.32:0/3024038368 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff200102d50 0x7ff20019f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:15.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.120+0000 7ff1ff7fe700 1 -- 172.21.15.32:0/3024038368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2001023b0 msgr2=0x7ff20019d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:15.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.120+0000 7ff1ff7fe700 1 --2- 172.21.15.32:0/3024038368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2001023b0 0x7ff20019d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:15.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.120+0000 7ff1ff7fe700 1 -- 172.21.15.32:0/3024038368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2001a0070 con 0x7ff2001011b0 2024-04-25T01:26:15.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.120+0000 7ff1feffd700 1 --2- 172.21.15.32:0/3024038368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2001023b0 0x7ff20019d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:26:15.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.121+0000 7ff1ff7fe700 1 --2- 172.21.15.32:0/3024038368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2001011b0 0x7ff20006b4c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800d980 tx=0x7ff1e800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:15.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.121+0000 7ff1fcff9700 1 -- 172.21.15.32:0/3024038368 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8017030 con 0x7ff2001011b0 2024-04-25T01:26:15.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.121+0000 7ff205f1b700 1 -- 172.21.15.32:0/3024038368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2001a0360 con 0x7ff2001011b0 2024-04-25T01:26:15.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.121+0000 7ff1fcff9700 1 -- 172.21.15.32:0/3024038368 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1e8004500 con 0x7ff2001011b0 2024-04-25T01:26:15.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.121+0000 7ff1fcff9700 1 -- 172.21.15.32:0/3024038368 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8003aa0 con 0x7ff2001011b0 2024-04-25T01:26:15.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.121+0000 7ff205f1b700 1 -- 172.21.15.32:0/3024038368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2001a08a0 con 0x7ff2001011b0 2024-04-25T01:26:15.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.122+0000 7ff205f1b700 1 -- 172.21.15.32:0/3024038368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff200068f20 con 0x7ff2001011b0 2024-04-25T01:26:15.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.123+0000 7ff1fcff9700 1 -- 172.21.15.32:0/3024038368 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff1e801a070 con 0x7ff2001011b0 2024-04-25T01:26:15.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.123+0000 7ff1fcff9700 1 --2- 172.21.15.32:0/3024038368 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1ec0a2ca0 0x7ff1ec0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:15.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.124+0000 7ff1fcff9700 1 -- 172.21.15.32:0/3024038368 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3698+0+0 (secure 0 0 0) 0x7ff1e80cdbb0 con 0x7ff2001011b0 2024-04-25T01:26:15.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.124+0000 7ff1feffd700 1 --2- 172.21.15.32:0/3024038368 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1ec0a2ca0 0x7ff1ec0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:15.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.124+0000 7ff1feffd700 1 --2- 172.21.15.32:0/3024038368 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1ec0a2ca0 0x7ff1ec0a5160 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0009750 tx=0x7ff1f0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:15.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.127+0000 7ff1fcff9700 1 -- 172.21.15.32:0/3024038368 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff1e809bc70 con 0x7ff2001011b0 2024-04-25T01:26:15.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:15.342+0000 7ff205f1b700 1 -- 172.21.15.32:0/3024038368 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7ff200062670 con 0x7ff1ec0a2ca0 2024-04-25T01:26:15.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:15 smithi028 ceph-mon[29390]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:15.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:15 smithi028 ceph-mon[29390]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T01:26:15.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:15 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:26:15.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:15 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:26:15.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:15 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:15.955 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:15 smithi032 ceph-mon[32461]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:15.955 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:15 smithi032 ceph-mon[32461]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T01:26:15.955 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:26:15.955 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:26:15.955 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:15.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:15 smithi130 ceph-mon[32702]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:15.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:15 smithi130 ceph-mon[32702]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T01:26:15.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:15 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:26:15.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:15 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:26:15.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:15 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:16.928 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:16 smithi032 ceph-mon[32461]: from='client.14325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:26:16.928 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:16 smithi032 ceph-mon[32461]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T01:26:16.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:16 smithi130 ceph-mon[32702]: from='client.14325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:26:16.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:16 smithi130 ceph-mon[32702]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T01:26:17.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:16 smithi028 ceph-mon[29390]: from='client.14325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:26:17.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:16 smithi028 ceph-mon[29390]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T01:26:17.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:17 smithi130 ceph-mon[32702]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:17.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:17 smithi130 ceph-mon[32702]: osdmap e30: 4 total, 4 up, 4 in 2024-04-25T01:26:17.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:17 smithi032 ceph-mon[32461]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:17.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:17 smithi032 ceph-mon[32461]: osdmap e30: 4 total, 4 up, 4 in 2024-04-25T01:26:18.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:17 smithi028 ceph-mon[29390]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:18.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:17 smithi028 ceph-mon[29390]: osdmap e30: 4 total, 4 up, 4 in 2024-04-25T01:26:18.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:18 smithi032 ceph-mon[32461]: from='client.? 172.21.15.32:0/62822382' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80f71a46-76da-45eb-b59b-6af112d2e0aa"}]: dispatch 2024-04-25T01:26:18.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:18 smithi032 ceph-mon[32461]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80f71a46-76da-45eb-b59b-6af112d2e0aa"}]: dispatch 2024-04-25T01:26:18.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:18 smithi032 ceph-mon[32461]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80f71a46-76da-45eb-b59b-6af112d2e0aa"}]': finished 2024-04-25T01:26:18.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:18 smithi032 ceph-mon[32461]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T01:26:18.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:18 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:26:18.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:18 smithi130 ceph-mon[32702]: from='client.? 172.21.15.32:0/62822382' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80f71a46-76da-45eb-b59b-6af112d2e0aa"}]: dispatch 2024-04-25T01:26:18.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:18 smithi130 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80f71a46-76da-45eb-b59b-6af112d2e0aa"}]: dispatch 2024-04-25T01:26:18.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:18 smithi130 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80f71a46-76da-45eb-b59b-6af112d2e0aa"}]': finished 2024-04-25T01:26:18.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:18 smithi130 ceph-mon[32702]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T01:26:18.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:18 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:26:19.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:18 smithi028 ceph-mon[29390]: from='client.? 172.21.15.32:0/62822382' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80f71a46-76da-45eb-b59b-6af112d2e0aa"}]: dispatch 2024-04-25T01:26:19.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:18 smithi028 ceph-mon[29390]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80f71a46-76da-45eb-b59b-6af112d2e0aa"}]: dispatch 2024-04-25T01:26:19.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:18 smithi028 ceph-mon[29390]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80f71a46-76da-45eb-b59b-6af112d2e0aa"}]': finished 2024-04-25T01:26:19.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:18 smithi028 ceph-mon[29390]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T01:26:19.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:18 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:26:19.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:19 smithi130 ceph-mon[32702]: pgmap v110: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:19.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:19 smithi130 ceph-mon[32702]: from='client.? 172.21.15.32:0/933220069' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:26:19.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:19 smithi032 ceph-mon[32461]: pgmap v110: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:19.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:19 smithi032 ceph-mon[32461]: from='client.? 172.21.15.32:0/933220069' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:26:20.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:19 smithi028 ceph-mon[29390]: pgmap v110: 1 pgs: 1 remapped+peering; 577 KiB data, 128 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:20.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:19 smithi028 ceph-mon[29390]: from='client.? 172.21.15.32:0/933220069' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:26:21.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:21 smithi130 ceph-mon[32702]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-25T01:26:21.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:21 smithi032 ceph-mon[32461]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-25T01:26:22.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:21 smithi028 ceph-mon[29390]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-25T01:26:23.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:23 smithi032 ceph-mon[32461]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-25T01:26:23.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:23 smithi130 ceph-mon[32702]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-25T01:26:24.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:23 smithi028 ceph-mon[29390]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-25T01:26:25.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:25 smithi130 ceph-mon[32702]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:26:25.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:25 smithi032 ceph-mon[32461]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:26:26.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:25 smithi028 ceph-mon[29390]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:26:26.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:26 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T01:26:26.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:26 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:26.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:26 smithi130 ceph-mon[32702]: Deploying daemon osd.4 on smithi032 2024-04-25T01:26:26.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:26 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T01:26:26.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:26 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:26.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:26 smithi032 ceph-mon[32461]: Deploying daemon osd.4 on smithi032 2024-04-25T01:26:27.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:26 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T01:26:27.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:26 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:27.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:26 smithi028 ceph-mon[29390]: Deploying daemon osd.4 on smithi032 2024-04-25T01:26:27.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:27 smithi130 ceph-mon[32702]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-25T01:26:27.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:27 smithi032 ceph-mon[32461]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-25T01:26:28.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:27 smithi028 ceph-mon[29390]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-25T01:26:29.826 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:29 smithi032 ceph-mon[32461]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-25T01:26:29.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:29 smithi130 ceph-mon[32702]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-25T01:26:30.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:29 smithi028 ceph-mon[29390]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-25T01:26:31.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:31 smithi032 ceph-mon[32461]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T01:26:31.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:31.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:31.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:31.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:31 smithi130 ceph-mon[32702]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T01:26:31.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:31.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:31.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:32.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:31 smithi028 ceph-mon[29390]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T01:26:32.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:32.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:32.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:33.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:33 smithi130 ceph-mon[32702]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:33.967 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:33 smithi032 ceph-mon[32461]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:34.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:33 smithi028 ceph-mon[29390]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:35.191 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 4 on host 'smithi032' 2024-04-25T01:26:35.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:35.211+0000 7ff1fcff9700 1 -- 172.21.15.32:0/3024038368 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff200062670 con 0x7ff1ec0a2ca0 2024-04-25T01:26:35.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:35.213+0000 7ff205f1b700 1 -- 172.21.15.32:0/3024038368 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1ec0a2ca0 msgr2=0x7ff1ec0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:35.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:35.213+0000 7ff205f1b700 1 --2- 172.21.15.32:0/3024038368 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1ec0a2ca0 0x7ff1ec0a5160 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0009750 tx=0x7ff1f0009310 comp rx=0 tx=0).stop 2024-04-25T01:26:35.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:35.213+0000 7ff205f1b700 1 -- 172.21.15.32:0/3024038368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2001011b0 msgr2=0x7ff20006b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:35.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:35.213+0000 7ff205f1b700 1 --2- 172.21.15.32:0/3024038368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2001011b0 0x7ff20006b4c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800d980 tx=0x7ff1e800dc90 comp rx=0 tx=0).stop 2024-04-25T01:26:35.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:35.213+0000 7ff205f1b700 1 -- 172.21.15.32:0/3024038368 shutdown_connections 2024-04-25T01:26:35.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:35.213+0000 7ff205f1b700 1 --2- 172.21.15.32:0/3024038368 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1ec0a2ca0 0x7ff1ec0a5160 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:35.213+0000 7ff205f1b700 1 --2- 172.21.15.32:0/3024038368 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff200102d50 0x7ff20019f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:35.213+0000 7ff205f1b700 1 --2- 172.21.15.32:0/3024038368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2001011b0 0x7ff20006b4c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:35.213+0000 7ff205f1b700 1 --2- 172.21.15.32:0/3024038368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff2001023b0 0x7ff20019d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:35.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:35.213+0000 7ff205f1b700 1 -- 172.21.15.32:0/3024038368 >> 172.21.15.32:0/3024038368 conn(0x7ff2000fc750 msgr2=0x7ff200105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:35.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:35.213+0000 7ff205f1b700 1 -- 172.21.15.32:0/3024038368 shutdown_connections 2024-04-25T01:26:35.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:26:35.213+0000 7ff205f1b700 1 -- 172.21.15.32:0/3024038368 wait complete. 2024-04-25T01:26:35.863 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:35 smithi032 ceph-mon[32461]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:35.864 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:35 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:35.864 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:35 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:35.864 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:35 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:35.864 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:35 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:35.864 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:35 smithi032 ceph-mon[32461]: from='osd.4 [v2:172.21.15.32:6808/3679896058,v1:172.21.15.32:6809/3679896058]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:26:35.864 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:35 smithi032 ceph-mon[32461]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:26:35.889 DEBUG:teuthology.orchestra.run.smithi032:osd.4> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.4.service 2024-04-25T01:26:35.892 INFO:tasks.cephadm:Deploying osd.5 on smithi130 with /dev/vg_nvme/lv_3... 2024-04-25T01:26:35.892 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T01:26:35.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:35 smithi130 ceph-mon[32702]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:35.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:35 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:35.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:35 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:35.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:35 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:35.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:35 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:35.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:35 smithi130 ceph-mon[32702]: from='osd.4 [v2:172.21.15.32:6808/3679896058,v1:172.21.15.32:6809/3679896058]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:26:35.964 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:35 smithi130 ceph-mon[32702]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:26:36.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:35 smithi028 ceph-mon[29390]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:36.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:36.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:36.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:36.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:36.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:35 smithi028 ceph-mon[29390]: from='osd.4 [v2:172.21.15.32:6808/3679896058,v1:172.21.15.32:6809/3679896058]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:26:36.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:35 smithi028 ceph-mon[29390]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:26:36.147 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:26:37.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:37 smithi032 ceph-mon[32461]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:37.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:37 smithi032 ceph-mon[32461]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T01:26:37.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:37 smithi032 ceph-mon[32461]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T01:26:37.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:37 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:26:37.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:37 smithi032 ceph-mon[32461]: from='osd.4 [v2:172.21.15.32:6808/3679896058,v1:172.21.15.32:6809/3679896058]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:26:37.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:37 smithi032 ceph-mon[32461]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:26:37.486 INFO:journalctl@ceph.osd.4.smithi032.stdout:Apr 25 01:26:37 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-4[44020]: 2024-04-25T01:26:37.307+0000 7f805f4c0700 -1 osd.4 0 waiting for initial osdmap 2024-04-25T01:26:37.486 INFO:journalctl@ceph.osd.4.smithi032.stdout:Apr 25 01:26:37 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-4[44020]: 2024-04-25T01:26:37.313+0000 7f805b2b3700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:26:37.558 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:37 smithi130 ceph-mon[32702]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:37.558 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:37 smithi130 ceph-mon[32702]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T01:26:37.558 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:37 smithi130 ceph-mon[32702]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T01:26:37.558 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:37 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:26:37.558 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:37 smithi130 ceph-mon[32702]: from='osd.4 [v2:172.21.15.32:6808/3679896058,v1:172.21.15.32:6809/3679896058]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:26:37.558 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:37 smithi130 ceph-mon[32702]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:26:37.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:37 smithi028 ceph-mon[29390]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:37.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:37 smithi028 ceph-mon[29390]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T01:26:37.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:37 smithi028 ceph-mon[29390]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T01:26:37.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:37 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:26:37.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:37 smithi028 ceph-mon[29390]: from='osd.4 [v2:172.21.15.32:6808/3679896058,v1:172.21.15.32:6809/3679896058]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:26:37.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:37 smithi028 ceph-mon[29390]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:26:38.102 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T01:26:38.103 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T01:26:38.103 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-04-25T01:26:38.103 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-04-25T01:26:38.103 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112506 s, 932 MB/s 2024-04-25T01:26:38.103 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-04-25T01:26:38.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:38 smithi028 ceph-mon[29390]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-25T01:26:38.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:38 smithi028 ceph-mon[29390]: osdmap e33: 5 total, 4 up, 5 in 2024-04-25T01:26:38.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:38 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:26:38.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:38 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:26:38.567 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:38 smithi130 ceph-mon[32702]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-25T01:26:38.567 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:38 smithi130 ceph-mon[32702]: osdmap e33: 5 total, 4 up, 5 in 2024-04-25T01:26:38.567 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:38 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:26:38.567 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:38 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:26:38.695 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:38 smithi032 ceph-mon[32461]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-25T01:26:38.696 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:38 smithi032 ceph-mon[32461]: osdmap e33: 5 total, 4 up, 5 in 2024-04-25T01:26:38.696 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:38 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:26:38.696 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:38 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:26:38.733 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi130:vg_nvme/lv_3 2024-04-25T01:26:38.978 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:26:39.357 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:39 smithi130 ceph-mon[32702]: purged_snaps scrub starts 2024-04-25T01:26:39.357 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:39 smithi130 ceph-mon[32702]: purged_snaps scrub ok 2024-04-25T01:26:39.357 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:39 smithi130 ceph-mon[32702]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:39.357 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:39 smithi130 ceph-mon[32702]: osd.4 [v2:172.21.15.32:6808/3679896058,v1:172.21.15.32:6809/3679896058] boot 2024-04-25T01:26:39.357 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:39 smithi130 ceph-mon[32702]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T01:26:39.357 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:39 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:26:39.357 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:39 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:39.357 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:39 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:39.358 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:39 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:26:39.358 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:39 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:26:39.358 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:39 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:39.358 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:39 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:39.358 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:39 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:26:39.358 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:39 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:39.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:39 smithi028 ceph-mon[29390]: purged_snaps scrub starts 2024-04-25T01:26:39.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:39 smithi028 ceph-mon[29390]: purged_snaps scrub ok 2024-04-25T01:26:39.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:39 smithi028 ceph-mon[29390]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:39.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:39 smithi028 ceph-mon[29390]: osd.4 [v2:172.21.15.32:6808/3679896058,v1:172.21.15.32:6809/3679896058] boot 2024-04-25T01:26:39.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:39 smithi028 ceph-mon[29390]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T01:26:39.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:39 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:26:39.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:39 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:39.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:39 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:39.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:39 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:26:39.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:39 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:26:39.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:39 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:39.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:39 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:39.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:39 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:26:39.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:39 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:39.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:39 smithi032 ceph-mon[32461]: purged_snaps scrub starts 2024-04-25T01:26:39.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:39 smithi032 ceph-mon[32461]: purged_snaps scrub ok 2024-04-25T01:26:39.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:39 smithi032 ceph-mon[32461]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 129 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:26:39.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:39 smithi032 ceph-mon[32461]: osd.4 [v2:172.21.15.32:6808/3679896058,v1:172.21.15.32:6809/3679896058] boot 2024-04-25T01:26:39.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:39 smithi032 ceph-mon[32461]: osdmap e34: 5 total, 5 up, 5 in 2024-04-25T01:26:39.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:39 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:26:39.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:39 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:39.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:39 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:39.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:39 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:26:39.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:39 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:26:39.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:39 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:39.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:39 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:39.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:39 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:26:39.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:39 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:40.441 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:40 smithi130 ceph-mon[32702]: Detected new or changed devices on smithi032 2024-04-25T01:26:40.442 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:40 smithi130 ceph-mon[32702]: Adjusting osd_memory_target on smithi032 to 8546M 2024-04-25T01:26:40.442 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:40 smithi130 ceph-mon[32702]: osdmap e35: 5 total, 5 up, 5 in 2024-04-25T01:26:40.442 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:40 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:40.442 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:40 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:40.442 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:40 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:26:40.443 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:40 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:40.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:40 smithi028 ceph-mon[29390]: Detected new or changed devices on smithi032 2024-04-25T01:26:40.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:40 smithi028 ceph-mon[29390]: Adjusting osd_memory_target on smithi032 to 8546M 2024-04-25T01:26:40.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:40 smithi028 ceph-mon[29390]: osdmap e35: 5 total, 5 up, 5 in 2024-04-25T01:26:40.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:40 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:40.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:40 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:40.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:40 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:26:40.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:40 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:40.673 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.694+0000 7f78c9260700 1 -- 172.21.15.130:0/3496378540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78c4102d30 msgr2=0x7f78c41095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:40.674 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.694+0000 7f78c9260700 1 --2- 172.21.15.130:0/3496378540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78c4102d30 0x7f78c41095e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f78bc00b0a0 tx=0x7f78bc00b3b0 comp rx=0 tx=0).stop 2024-04-25T01:26:40.674 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.694+0000 7f78c9260700 1 -- 172.21.15.130:0/3496378540 shutdown_connections 2024-04-25T01:26:40.674 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.694+0000 7f78c9260700 1 --2- 172.21.15.130:0/3496378540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78c4102d30 0x7f78c41095e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:40.674 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.694+0000 7f78c9260700 1 --2- 172.21.15.130:0/3496378540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78c4101190 0x7f78c41015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:40.674 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.694+0000 7f78c9260700 1 --2- 172.21.15.130:0/3496378540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c4102390 0x7f78c41027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:40.674 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.694+0000 7f78c9260700 1 -- 172.21.15.130:0/3496378540 >> 172.21.15.130:0/3496378540 conn(0x7f78c40fc730 msgr2=0x7f78c40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:26:40.674 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.694+0000 7f78c9260700 1 -- 172.21.15.130:0/3496378540 shutdown_connections 2024-04-25T01:26:40.674 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.694+0000 7f78c9260700 1 -- 172.21.15.130:0/3496378540 wait complete. 2024-04-25T01:26:40.676 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.695+0000 7f78c9260700 1 Processor -- start 2024-04-25T01:26:40.676 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.695+0000 7f78c9260700 1 -- start start 2024-04-25T01:26:40.677 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.695+0000 7f78c9260700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c4101190 0x7f78c406aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:40.677 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.695+0000 7f78c9260700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78c4102390 0x7f78c419d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:40.677 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.695+0000 7f78c9260700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78c4102d30 0x7f78c419f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:40.677 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.695+0000 7f78c9260700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f78c410cde0 con 0x7f78c4101190 2024-04-25T01:26:40.677 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.695+0000 7f78c9260700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f78c410cc60 con 0x7f78c4102d30 2024-04-25T01:26:40.677 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.695+0000 7f78c9260700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f78c410cf60 con 0x7f78c4102390 2024-04-25T01:26:40.677 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.695+0000 7f78c259c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78c4102390 0x7f78c419d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:40.677 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.695+0000 7f78c259c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78c4102390 0x7f78c419d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:44080/0 (socket says 172.21.15.130:44080) 2024-04-25T01:26:40.677 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.695+0000 7f78c259c700 1 -- 172.21.15.130:0/2073790393 learned_addr learned my addr 172.21.15.130:0/2073790393 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-25T01:26:40.677 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.695+0000 7f78c2d9d700 1 --2- 172.21.15.130:0/2073790393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c4101190 0x7f78c406aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:40.677 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.695+0000 7f78c359e700 1 --2- 172.21.15.130:0/2073790393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78c4102d30 0x7f78c419f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:40.677 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.695+0000 7f78c259c700 1 -- 172.21.15.130:0/2073790393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78c4102d30 msgr2=0x7f78c419f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:40.678 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.695+0000 7f78c259c700 1 --2- 172.21.15.130:0/2073790393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78c4102d30 0x7f78c419f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:40.678 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.695+0000 7f78c259c700 1 -- 172.21.15.130:0/2073790393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c4101190 msgr2=0x7f78c406aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:26:40.678 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.695+0000 7f78c259c700 1 --2- 172.21.15.130:0/2073790393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c4101190 0x7f78c406aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:26:40.678 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.695+0000 7f78c259c700 1 -- 172.21.15.130:0/2073790393 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c41a0140 con 0x7f78c4102390 2024-04-25T01:26:40.678 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.695+0000 7f78c2d9d700 1 --2- 172.21.15.130:0/2073790393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c4101190 0x7f78c406aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:26:40.678 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.696+0000 7f78c259c700 1 --2- 172.21.15.130:0/2073790393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78c4102390 0x7f78c419d230 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f78b400b530 tx=0x7f78b400b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:40.678 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.696+0000 7f78abfff700 1 -- 172.21.15.130:0/2073790393 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b4015020 con 0x7f78c4102390 2024-04-25T01:26:40.678 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.696+0000 7f78c9260700 1 -- 172.21.15.130:0/2073790393 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78c41a0430 con 0x7f78c4102390 2024-04-25T01:26:40.678 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.696+0000 7f78abfff700 1 -- 172.21.15.130:0/2073790393 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78b4011d10 con 0x7f78c4102390 2024-04-25T01:26:40.678 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.696+0000 7f78abfff700 1 -- 172.21.15.130:0/2073790393 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b4010e50 con 0x7f78c4102390 2024-04-25T01:26:40.678 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.696+0000 7f78c9260700 1 -- 172.21.15.130:0/2073790393 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78c41a0a40 con 0x7f78c4102390 2024-04-25T01:26:40.678 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.697+0000 7f78c9260700 1 -- 172.21.15.130:0/2073790393 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78c4068f20 con 0x7f78c4102390 2024-04-25T01:26:40.679 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.698+0000 7f78abfff700 1 -- 172.21.15.130:0/2073790393 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f78b4018020 con 0x7f78c4102390 2024-04-25T01:26:40.679 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.698+0000 7f78abfff700 1 --2- 172.21.15.130:0/2073790393 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f78ac0a2dc0 0x7f78ac0a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:26:40.679 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.699+0000 7f78abfff700 1 -- 172.21.15.130:0/2073790393 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7f78b40cddb0 con 0x7f78c4102390 2024-04-25T01:26:40.679 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.699+0000 7f78c2d9d700 1 --2- 172.21.15.130:0/2073790393 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f78ac0a2dc0 0x7f78ac0a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:26:40.679 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.699+0000 7f78c2d9d700 1 --2- 172.21.15.130:0/2073790393 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f78ac0a2dc0 0x7f78ac0a5280 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f78c41021f0 tx=0x7f78b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:26:40.681 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.702+0000 7f78abfff700 1 -- 172.21.15.130:0/2073790393 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f78b401f020 con 0x7f78c4102390 2024-04-25T01:26:40.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:40 smithi032 ceph-mon[32461]: Detected new or changed devices on smithi032 2024-04-25T01:26:40.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:40 smithi032 ceph-mon[32461]: Adjusting osd_memory_target on smithi032 to 8546M 2024-04-25T01:26:40.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:40 smithi032 ceph-mon[32461]: osdmap e35: 5 total, 5 up, 5 in 2024-04-25T01:26:40.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:40 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:40.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:40 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:40.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:40 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:26:40.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:40 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:40.893 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:26:40.913+0000 7f78c9260700 1 -- 172.21.15.130:0/2073790393 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f78c4062670 con 0x7f78ac0a2dc0 2024-04-25T01:26:41.634 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:41 smithi130 ceph-mon[32702]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:41.634 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:41 smithi130 ceph-mon[32702]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:26:41.635 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:41 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:26:41.635 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:41 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:26:41.635 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:41 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:41.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:41 smithi032 ceph-mon[32461]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:41.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:41 smithi032 ceph-mon[32461]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:26:41.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:41 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:26:41.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:41 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:26:41.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:41 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:41.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:41 smithi028 ceph-mon[29390]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:41.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:41 smithi028 ceph-mon[29390]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:26:41.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:41 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:26:41.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:41 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:26:41.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:41 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:43.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:43 smithi032 ceph-mon[32461]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:43.615 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:43 smithi028 ceph-mon[29390]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:43.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:43 smithi130 ceph-mon[32702]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:44.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:44 smithi130 ceph-mon[32702]: from='client.? 172.21.15.130:0/3664895618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0394f4a0-297e-470a-b491-6da6616a7ed8"}]: dispatch 2024-04-25T01:26:44.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:44 smithi130 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0394f4a0-297e-470a-b491-6da6616a7ed8"}]: dispatch 2024-04-25T01:26:44.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:44 smithi130 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0394f4a0-297e-470a-b491-6da6616a7ed8"}]': finished 2024-04-25T01:26:44.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:44 smithi130 ceph-mon[32702]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T01:26:44.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:44 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:26:44.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:44 smithi130 ceph-mon[32702]: from='client.? 172.21.15.130:0/3899134441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:26:44.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:44 smithi032 ceph-mon[32461]: from='client.? 172.21.15.130:0/3664895618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0394f4a0-297e-470a-b491-6da6616a7ed8"}]: dispatch 2024-04-25T01:26:44.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:44 smithi032 ceph-mon[32461]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0394f4a0-297e-470a-b491-6da6616a7ed8"}]: dispatch 2024-04-25T01:26:44.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:44 smithi032 ceph-mon[32461]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0394f4a0-297e-470a-b491-6da6616a7ed8"}]': finished 2024-04-25T01:26:44.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:44 smithi032 ceph-mon[32461]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T01:26:44.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:44 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:26:44.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:44 smithi032 ceph-mon[32461]: from='client.? 172.21.15.130:0/3899134441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:26:44.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:44 smithi028 ceph-mon[29390]: from='client.? 172.21.15.130:0/3664895618' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0394f4a0-297e-470a-b491-6da6616a7ed8"}]: dispatch 2024-04-25T01:26:44.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:44 smithi028 ceph-mon[29390]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0394f4a0-297e-470a-b491-6da6616a7ed8"}]: dispatch 2024-04-25T01:26:44.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:44 smithi028 ceph-mon[29390]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0394f4a0-297e-470a-b491-6da6616a7ed8"}]': finished 2024-04-25T01:26:44.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:44 smithi028 ceph-mon[29390]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T01:26:44.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:44 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:26:44.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:44 smithi028 ceph-mon[29390]: from='client.? 172.21.15.130:0/3899134441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:26:45.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:45 smithi028 ceph-mon[29390]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:45.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:45 smithi130 ceph-mon[32702]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:45.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:45 smithi032 ceph-mon[32461]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:47.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:47 smithi028 ceph-mon[29390]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:47.920 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:47 smithi130 ceph-mon[32702]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:47.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:47 smithi032 ceph-mon[32461]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:49.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:49 smithi130 ceph-mon[32702]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:49.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:49 smithi028 ceph-mon[29390]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:49.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:49 smithi032 ceph-mon[32461]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:51.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:51 smithi028 ceph-mon[29390]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:51.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:51 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T01:26:51.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:51 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:51.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:51 smithi028 ceph-mon[29390]: Deploying daemon osd.5 on smithi130 2024-04-25T01:26:51.853 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:51 smithi130 ceph-mon[32702]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:51.853 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:51 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T01:26:51.853 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:51 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:51.853 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:51 smithi130 ceph-mon[32702]: Deploying daemon osd.5 on smithi130 2024-04-25T01:26:51.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:51 smithi032 ceph-mon[32461]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:51.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:51 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T01:26:51.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:51 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:26:51.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:51 smithi032 ceph-mon[32461]: Deploying daemon osd.5 on smithi130 2024-04-25T01:26:53.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:53 smithi028 ceph-mon[29390]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:53.920 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:53 smithi130 ceph-mon[32702]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:53.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:53 smithi032 ceph-mon[32461]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:55.805 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:55 smithi130 ceph-mon[32702]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:55.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:55 smithi028 ceph-mon[29390]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:55.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:55 smithi032 ceph-mon[32461]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:57.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:57 smithi130 ceph-mon[32702]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:57.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:57 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:57.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:57 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:57.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:57 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:57.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:57 smithi032 ceph-mon[32461]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:57.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:57 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:57.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:57 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:57.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:57 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:57.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:57 smithi028 ceph-mon[29390]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:57.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:57 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:57.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:57 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:26:57.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:57 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:26:59.658 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:26:59 smithi130 ceph-mon[32702]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:59.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:26:59 smithi032 ceph-mon[32461]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:26:59.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:26:59 smithi028 ceph-mon[29390]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:27:00.417 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 5 on host 'smithi130' 2024-04-25T01:27:00.419 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:00.434+0000 7f78abfff700 1 -- 172.21.15.130:0/2073790393 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f78c4062670 con 0x7f78ac0a2dc0 2024-04-25T01:27:00.419 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:00.437+0000 7f78c9260700 1 -- 172.21.15.130:0/2073790393 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f78ac0a2dc0 msgr2=0x7f78ac0a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:00.420 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:00.437+0000 7f78c9260700 1 --2- 172.21.15.130:0/2073790393 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f78ac0a2dc0 0x7f78ac0a5280 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f78c41021f0 tx=0x7f78b8005f90 comp rx=0 tx=0).stop 2024-04-25T01:27:00.420 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:00.437+0000 7f78c9260700 1 -- 172.21.15.130:0/2073790393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78c4102390 msgr2=0x7f78c419d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:00.420 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:00.437+0000 7f78c9260700 1 --2- 172.21.15.130:0/2073790393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78c4102390 0x7f78c419d230 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f78b400b530 tx=0x7f78b400b8f0 comp rx=0 tx=0).stop 2024-04-25T01:27:00.420 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:00.437+0000 7f78c9260700 1 -- 172.21.15.130:0/2073790393 shutdown_connections 2024-04-25T01:27:00.420 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:00.437+0000 7f78c9260700 1 --2- 172.21.15.130:0/2073790393 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f78ac0a2dc0 0x7f78ac0a5280 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:00.420 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:00.437+0000 7f78c9260700 1 --2- 172.21.15.130:0/2073790393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f78c4102d30 0x7f78c419f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:00.420 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:00.437+0000 7f78c9260700 1 --2- 172.21.15.130:0/2073790393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c4101190 0x7f78c406aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:00.420 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:00.437+0000 7f78c9260700 1 --2- 172.21.15.130:0/2073790393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f78c4102390 0x7f78c419d230 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:00.420 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:00.437+0000 7f78c9260700 1 -- 172.21.15.130:0/2073790393 >> 172.21.15.130:0/2073790393 conn(0x7f78c40fc730 msgr2=0x7f78c4105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:00.420 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:00.438+0000 7f78c9260700 1 -- 172.21.15.130:0/2073790393 shutdown_connections 2024-04-25T01:27:00.420 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:00.438+0000 7f78c9260700 1 -- 172.21.15.130:0/2073790393 wait complete. 2024-04-25T01:27:00.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:00 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:00.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:00 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:00.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:00 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:00.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:00 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:00.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:00 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:00.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:00 smithi130 ceph-mon[32702]: from='osd.5 [v2:172.21.15.130:6808/4253601662,v1:172.21.15.130:6809/4253601662]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:27:00.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:00 smithi130 ceph-mon[32702]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:27:00.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:00 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:00.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:00 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:00.712 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:00 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:00.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:00 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:00.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:00 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:00.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:00 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:00.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:00 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:00.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:00 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:00.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:00 smithi032 ceph-mon[32461]: from='osd.5 [v2:172.21.15.130:6808/4253601662,v1:172.21.15.130:6809/4253601662]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:27:00.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:00 smithi032 ceph-mon[32461]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:27:00.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:00 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:00.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:00 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:00.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:00 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:00.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:00.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:00.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:00.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:00.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:00.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:00 smithi028 ceph-mon[29390]: from='osd.5 [v2:172.21.15.130:6808/4253601662,v1:172.21.15.130:6809/4253601662]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:27:00.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:00 smithi028 ceph-mon[29390]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:27:00.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:00.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:00.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:00 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:01.021 DEBUG:teuthology.orchestra.run.smithi130:osd.5> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.5.service 2024-04-25T01:27:01.024 INFO:tasks.cephadm:Deploying osd.6 on smithi028 with /dev/vg_nvme/lv_2... 2024-04-25T01:27:01.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T01:27:01.279 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:27:01.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:01 smithi028 ceph-mon[29390]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:27:01.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:01 smithi028 ceph-mon[29390]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T01:27:01.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:01 smithi028 ceph-mon[29390]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T01:27:01.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:01 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:27:01.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:01 smithi028 ceph-mon[29390]: from='osd.5 [v2:172.21.15.130:6808/4253601662,v1:172.21.15.130:6809/4253601662]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:27:01.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:01 smithi028 ceph-mon[29390]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:27:01.903 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:01 smithi130 ceph-mon[32702]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:27:01.904 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:01 smithi130 ceph-mon[32702]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T01:27:01.904 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:01 smithi130 ceph-mon[32702]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T01:27:01.904 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:01 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:27:01.904 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:01 smithi130 ceph-mon[32702]: from='osd.5 [v2:172.21.15.130:6808/4253601662,v1:172.21.15.130:6809/4253601662]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:27:01.904 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:01 smithi130 ceph-mon[32702]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:27:01.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:01 smithi032 ceph-mon[32461]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:27:01.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:01 smithi032 ceph-mon[32461]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T01:27:01.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:01 smithi032 ceph-mon[32461]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T01:27:01.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:01 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:27:01.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:01 smithi032 ceph-mon[32461]: from='osd.5 [v2:172.21.15.130:6808/4253601662,v1:172.21.15.130:6809/4253601662]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:27:01.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:01 smithi032 ceph-mon[32461]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:27:02.460 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:27:02 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:27:02.185+0000 7f47f238f700 -1 osd.5 0 waiting for initial osdmap 2024-04-25T01:27:02.460 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:27:02 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:27:02.193+0000 7f47ea17a700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:27:02.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:02 smithi028 ceph-mon[29390]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-25T01:27:02.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:02 smithi028 ceph-mon[29390]: osdmap e38: 6 total, 5 up, 6 in 2024-04-25T01:27:02.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:02 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:27:02.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:02 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:27:02.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:02 smithi028 ceph-mon[29390]: osdmap e39: 6 total, 5 up, 6 in 2024-04-25T01:27:02.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:02 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:27:02.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:02 smithi028 ceph-mon[29390]: from='osd.5 ' entity='osd.5' 2024-04-25T01:27:02.879 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:02 smithi130 ceph-mon[32702]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-25T01:27:02.879 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:02 smithi130 ceph-mon[32702]: osdmap e38: 6 total, 5 up, 6 in 2024-04-25T01:27:02.879 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:02 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:27:02.879 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:02 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:27:02.880 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:02 smithi130 ceph-mon[32702]: osdmap e39: 6 total, 5 up, 6 in 2024-04-25T01:27:02.880 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:02 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:27:02.880 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:02 smithi130 ceph-mon[32702]: from='osd.5 ' entity='osd.5' 2024-04-25T01:27:02.931 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:02 smithi032 ceph-mon[32461]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-25T01:27:02.932 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:02 smithi032 ceph-mon[32461]: osdmap e38: 6 total, 5 up, 6 in 2024-04-25T01:27:02.932 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:02 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:27:02.932 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:02 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:27:02.932 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:02 smithi032 ceph-mon[32461]: osdmap e39: 6 total, 5 up, 6 in 2024-04-25T01:27:02.932 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:02 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:27:02.932 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:02 smithi032 ceph-mon[32461]: from='osd.5 ' entity='osd.5' 2024-04-25T01:27:03.217 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T01:27:03.217 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T01:27:03.217 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-25T01:27:03.217 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-25T01:27:03.217 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00856604 s, 1.2 GB/s 2024-04-25T01:27:03.217 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-25T01:27:03.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:03 smithi130 ceph-mon[32702]: purged_snaps scrub starts 2024-04-25T01:27:03.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:03 smithi130 ceph-mon[32702]: purged_snaps scrub ok 2024-04-25T01:27:03.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:03 smithi130 ceph-mon[32702]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:27:03.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:03 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:27:03.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:03 smithi130 ceph-mon[32702]: osd.5 [v2:172.21.15.130:6808/4253601662,v1:172.21.15.130:6809/4253601662] boot 2024-04-25T01:27:03.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:03 smithi130 ceph-mon[32702]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T01:27:03.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:03 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:27:03.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:03 smithi028 ceph-mon[29390]: purged_snaps scrub starts 2024-04-25T01:27:03.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:03 smithi028 ceph-mon[29390]: purged_snaps scrub ok 2024-04-25T01:27:03.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:03 smithi028 ceph-mon[29390]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:27:03.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:03 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:27:03.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:03 smithi028 ceph-mon[29390]: osd.5 [v2:172.21.15.130:6808/4253601662,v1:172.21.15.130:6809/4253601662] boot 2024-04-25T01:27:03.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:03 smithi028 ceph-mon[29390]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T01:27:03.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:03 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:27:03.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_2 2024-04-25T01:27:03.899 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:03 smithi032 ceph-mon[32461]: purged_snaps scrub starts 2024-04-25T01:27:03.899 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:03 smithi032 ceph-mon[32461]: purged_snaps scrub ok 2024-04-25T01:27:03.899 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:03 smithi032 ceph-mon[32461]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:27:03.899 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:03 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:27:03.899 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:03 smithi032 ceph-mon[32461]: osd.5 [v2:172.21.15.130:6808/4253601662,v1:172.21.15.130:6809/4253601662] boot 2024-04-25T01:27:03.899 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:03 smithi032 ceph-mon[32461]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T01:27:03.899 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:03 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:27:04.133 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:27:04.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:04 smithi028 ceph-mon[29390]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T01:27:04.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:04 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:04.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:04 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:04.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:04 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:04.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:04 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:04.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:04 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:04.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:04 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:04.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:04 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:04.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:04 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:04.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:04 smithi130 ceph-mon[32702]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T01:27:04.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:04 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:04.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:04 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:04.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:04 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:04.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:04 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:04.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:04 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:04.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:04 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:04.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:04 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:04.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:04 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:04.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:04 smithi032 ceph-mon[32461]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T01:27:04.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:04 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:04.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:04 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:04.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:04 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:04.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:04 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:04.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:04 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:04.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:04 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:04.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:04 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:04.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:04 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:05.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.714+0000 7f2bc6ce6700 1 -- 172.21.15.28:0/3265592849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc0103490 msgr2=0x7f2bc01038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.714+0000 7f2bc6ce6700 1 --2- 172.21.15.28:0/3265592849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc0103490 0x7f2bc01038f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2bac0099c0 tx=0x7f2bac009cd0 comp rx=0 tx=0).stop 2024-04-25T01:27:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.714+0000 7f2bc6ce6700 1 -- 172.21.15.28:0/3265592849 shutdown_connections 2024-04-25T01:27:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.714+0000 7f2bc6ce6700 1 --2- 172.21.15.28:0/3265592849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bc0103e30 0x7f2bc010a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.714+0000 7f2bc6ce6700 1 --2- 172.21.15.28:0/3265592849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bc0102290 0x7f2bc01026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.714+0000 7f2bc6ce6700 1 --2- 172.21.15.28:0/3265592849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc0103490 0x7f2bc01038f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.714+0000 7f2bc6ce6700 1 -- 172.21.15.28:0/3265592849 >> 172.21.15.28:0/3265592849 conn(0x7f2bc00fd830 msgr2=0x7f2bc00ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.714+0000 7f2bc6ce6700 1 -- 172.21.15.28:0/3265592849 shutdown_connections 2024-04-25T01:27:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.714+0000 7f2bc6ce6700 1 -- 172.21.15.28:0/3265592849 wait complete. 2024-04-25T01:27:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.715+0000 7f2bc6ce6700 1 Processor -- start 2024-04-25T01:27:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.715+0000 7f2bc6ce6700 1 -- start start 2024-04-25T01:27:05.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.715+0000 7f2bc6ce6700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bc0102290 0x7f2bc01984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:05.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.715+0000 7f2bc6ce6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bc0103490 0x7f2bc01989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:05.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.715+0000 7f2bc6ce6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc0103e30 0x7f2bc019fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:05.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.715+0000 7f2bc6ce6700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc010def0 con 0x7f2bc0103e30 2024-04-25T01:27:05.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.715+0000 7f2bc6ce6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc010dd70 con 0x7f2bc0103490 2024-04-25T01:27:05.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.715+0000 7f2bc6ce6700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc010e070 con 0x7f2bc0102290 2024-04-25T01:27:05.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.715+0000 7f2bc5283700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc0103e30 0x7f2bc019fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:05.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.715+0000 7f2bc5283700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc0103e30 0x7f2bc019fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52354/0 (socket says 172.21.15.28:52354) 2024-04-25T01:27:05.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.715+0000 7f2bc5283700 1 -- 172.21.15.28:0/1201786666 learned_addr learned my addr 172.21.15.28:0/1201786666 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:27:05.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.715+0000 7f2bc4a82700 1 --2- 172.21.15.28:0/1201786666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bc0102290 0x7f2bc01984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.715+0000 7f2bbffff700 1 --2- 172.21.15.28:0/1201786666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bc0103490 0x7f2bc01989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:05.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.716+0000 7f2bc5283700 1 -- 172.21.15.28:0/1201786666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bc0102290 msgr2=0x7f2bc01984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:05.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.716+0000 7f2bc5283700 1 --2- 172.21.15.28:0/1201786666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bc0102290 0x7f2bc01984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:05.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.716+0000 7f2bc5283700 1 -- 172.21.15.28:0/1201786666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bc0103490 msgr2=0x7f2bc01989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:05.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.716+0000 7f2bc5283700 1 --2- 172.21.15.28:0/1201786666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bc0103490 0x7f2bc01989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:05.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.716+0000 7f2bc5283700 1 -- 172.21.15.28:0/1201786666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc01a0100 con 0x7f2bc0103e30 2024-04-25T01:27:05.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.716+0000 7f2bc5283700 1 --2- 172.21.15.28:0/1201786666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc0103e30 0x7f2bc019fa60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8007da0 tx=0x7f2bb800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:05.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.716+0000 7f2bbdffb700 1 -- 172.21.15.28:0/1201786666 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb8016020 con 0x7f2bc0103e30 2024-04-25T01:27:05.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.716+0000 7f2bbdffb700 1 -- 172.21.15.28:0/1201786666 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bb800ac50 con 0x7f2bc0103e30 2024-04-25T01:27:05.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.716+0000 7f2bc6ce6700 1 -- 172.21.15.28:0/1201786666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bc01a03f0 con 0x7f2bc0103e30 2024-04-25T01:27:05.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.716+0000 7f2bbdffb700 1 -- 172.21.15.28:0/1201786666 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb80115e0 con 0x7f2bc0103e30 2024-04-25T01:27:05.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.716+0000 7f2bc6ce6700 1 -- 172.21.15.28:0/1201786666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bc01a08e0 con 0x7f2bc0103e30 2024-04-25T01:27:05.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.717+0000 7f2bc6ce6700 1 -- 172.21.15.28:0/1201786666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bc0068f20 con 0x7f2bc0103e30 2024-04-25T01:27:05.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.718+0000 7f2bbdffb700 1 -- 172.21.15.28:0/1201786666 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2bb8019020 con 0x7f2bc0103e30 2024-04-25T01:27:05.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.719+0000 7f2bbdffb700 1 --2- 172.21.15.28:0/1201786666 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f2bb00a2c20 0x7f2bb00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:05.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.719+0000 7f2bc4a82700 1 --2- 172.21.15.28:0/1201786666 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f2bb00a2c20 0x7f2bb00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:05.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.719+0000 7f2bbdffb700 1 -- 172.21.15.28:0/1201786666 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 4533+0+0 (secure 0 0 0) 0x7f2bb80ce100 con 0x7f2bc0103e30 2024-04-25T01:27:05.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.719+0000 7f2bc4a82700 1 --2- 172.21.15.28:0/1201786666 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f2bb00a2c20 0x7f2bb00a50e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2bc01032f0 tx=0x7f2bb4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:05.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.724+0000 7f2bbdffb700 1 -- 172.21.15.28:0/1201786666 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2bb8020020 con 0x7f2bc0103e30 2024-04-25T01:27:05.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:05 smithi028 ceph-mon[29390]: Detected new or changed devices on smithi130 2024-04-25T01:27:05.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:05 smithi028 ceph-mon[29390]: Adjusting osd_memory_target on smithi130 to 8544M 2024-04-25T01:27:05.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:05 smithi028 ceph-mon[29390]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:05.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:05 smithi028 ceph-mon[29390]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T01:27:05.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:05.940+0000 7f2bc6ce6700 1 -- 172.21.15.28:0/1201786666 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f2bc0062670 con 0x7f2bb00a2c20 2024-04-25T01:27:05.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:05 smithi130 ceph-mon[32702]: Detected new or changed devices on smithi130 2024-04-25T01:27:05.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:05 smithi130 ceph-mon[32702]: Adjusting osd_memory_target on smithi130 to 8544M 2024-04-25T01:27:05.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:05 smithi130 ceph-mon[32702]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:05.963 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:05 smithi130 ceph-mon[32702]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T01:27:05.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:05 smithi032 ceph-mon[32461]: Detected new or changed devices on smithi130 2024-04-25T01:27:05.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:05 smithi032 ceph-mon[32461]: Adjusting osd_memory_target on smithi130 to 8544M 2024-04-25T01:27:05.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:05 smithi032 ceph-mon[32461]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:05.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:05 smithi032 ceph-mon[32461]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T01:27:06.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:06 smithi028 ceph-mon[29390]: osdmap e43: 6 total, 6 up, 6 in 2024-04-25T01:27:06.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:06 smithi028 ceph-mon[29390]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:27:06.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:06 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:27:06.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:06 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:27:06.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:06 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:06.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:06 smithi130 ceph-mon[32702]: osdmap e43: 6 total, 6 up, 6 in 2024-04-25T01:27:06.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:06 smithi130 ceph-mon[32702]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:27:06.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:27:06.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:27:06.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:06.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:06 smithi032 ceph-mon[32461]: osdmap e43: 6 total, 6 up, 6 in 2024-04-25T01:27:06.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:06 smithi032 ceph-mon[32461]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:27:06.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:06 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:27:06.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:06 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:27:06.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:06 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:07.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:07 smithi028 ceph-mon[29390]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:07.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:07 smithi028 ceph-mon[29390]: osdmap e44: 6 total, 6 up, 6 in 2024-04-25T01:27:07.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:07 smithi130 ceph-mon[32702]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:07.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:07 smithi130 ceph-mon[32702]: osdmap e44: 6 total, 6 up, 6 in 2024-04-25T01:27:07.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:07 smithi032 ceph-mon[32461]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:07.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:07 smithi032 ceph-mon[32461]: osdmap e44: 6 total, 6 up, 6 in 2024-04-25T01:27:09.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:09 smithi130 ceph-mon[32702]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:09.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:09 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2522350673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84cb9069-9d19-4d0e-8db4-cb0d41d7e48a"}]: dispatch 2024-04-25T01:27:09.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:09 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2522350673' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "84cb9069-9d19-4d0e-8db4-cb0d41d7e48a"}]': finished 2024-04-25T01:27:09.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:09 smithi130 ceph-mon[32702]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T01:27:09.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:09 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:27:09.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:09 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/521830122' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:27:09.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:09 smithi032 ceph-mon[32461]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:09.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:09 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2522350673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84cb9069-9d19-4d0e-8db4-cb0d41d7e48a"}]: dispatch 2024-04-25T01:27:09.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:09 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2522350673' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "84cb9069-9d19-4d0e-8db4-cb0d41d7e48a"}]': finished 2024-04-25T01:27:09.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:09 smithi032 ceph-mon[32461]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T01:27:09.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:09 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:27:09.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:09 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/521830122' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:27:10.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:09 smithi028 ceph-mon[29390]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:10.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:09 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2522350673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "84cb9069-9d19-4d0e-8db4-cb0d41d7e48a"}]: dispatch 2024-04-25T01:27:10.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:09 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2522350673' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "84cb9069-9d19-4d0e-8db4-cb0d41d7e48a"}]': finished 2024-04-25T01:27:10.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:09 smithi028 ceph-mon[29390]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T01:27:10.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:09 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:27:10.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:09 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/521830122' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:27:11.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:11 smithi130 ceph-mon[32702]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-25T01:27:11.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:11 smithi032 ceph-mon[32461]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-25T01:27:12.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:11 smithi028 ceph-mon[29390]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-25T01:27:13.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:13 smithi028 ceph-mon[29390]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-25T01:27:13.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:13 smithi130 ceph-mon[32702]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-25T01:27:13.972 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:13 smithi032 ceph-mon[32461]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-25T01:27:15.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:15 smithi130 ceph-mon[32702]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:27:15.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:15 smithi032 ceph-mon[32461]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:27:16.054 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:15 smithi028 ceph-mon[29390]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:27:16.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T01:27:16.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:16.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T01:27:16.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:16.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T01:27:16.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:17.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:17 smithi028 ceph-mon[29390]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-25T01:27:17.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:17 smithi028 ceph-mon[29390]: Deploying daemon osd.6 on smithi028 2024-04-25T01:27:17.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:17 smithi130 ceph-mon[32702]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-25T01:27:17.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:17 smithi130 ceph-mon[32702]: Deploying daemon osd.6 on smithi028 2024-04-25T01:27:17.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:17 smithi032 ceph-mon[32461]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-25T01:27:17.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:17 smithi032 ceph-mon[32461]: Deploying daemon osd.6 on smithi028 2024-04-25T01:27:19.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:19 smithi028 ceph-mon[29390]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:27:19.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:19 smithi130 ceph-mon[32702]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:27:19.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:19 smithi032 ceph-mon[32461]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:27:21.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:21 smithi028 ceph-mon[29390]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:21.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:21 smithi130 ceph-mon[32702]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:21.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:21 smithi032 ceph-mon[32461]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:23.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:23 smithi028 ceph-mon[29390]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:23.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:23.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:23.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:23.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:23 smithi130 ceph-mon[32702]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:23.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:23.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:23.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:23.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:23 smithi032 ceph-mon[32461]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:23.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:23.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:23.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:25.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:25 smithi028 ceph-mon[29390]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:25.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:25 smithi130 ceph-mon[32702]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:25.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:25 smithi032 ceph-mon[32461]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:26.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:26.749+0000 7f2bbdffb700 1 -- 172.21.15.28:0/1201786666 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2bc0062670 con 0x7f2bb00a2c20 2024-04-25T01:27:26.729 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 6 on host 'smithi028' 2024-04-25T01:27:26.729 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:26 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:26.729 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:26 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:26.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:26 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:26.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:26 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:26.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:26 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:26.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:26 smithi028 ceph-mon[29390]: from='osd.6 [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T01:27:26.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:26.752+0000 7f2bc6ce6700 1 -- 172.21.15.28:0/1201786666 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f2bb00a2c20 msgr2=0x7f2bb00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:26.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:26.752+0000 7f2bc6ce6700 1 --2- 172.21.15.28:0/1201786666 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f2bb00a2c20 0x7f2bb00a50e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2bc01032f0 tx=0x7f2bb4009190 comp rx=0 tx=0).stop 2024-04-25T01:27:26.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:26.752+0000 7f2bc6ce6700 1 -- 172.21.15.28:0/1201786666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc0103e30 msgr2=0x7f2bc019fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:26.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:26.752+0000 7f2bc6ce6700 1 --2- 172.21.15.28:0/1201786666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc0103e30 0x7f2bc019fa60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8007da0 tx=0x7f2bb800a340 comp rx=0 tx=0).stop 2024-04-25T01:27:26.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:26.752+0000 7f2bc6ce6700 1 -- 172.21.15.28:0/1201786666 shutdown_connections 2024-04-25T01:27:26.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:26.752+0000 7f2bc6ce6700 1 --2- 172.21.15.28:0/1201786666 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f2bb00a2c20 0x7f2bb00a50e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:26.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:26.752+0000 7f2bc6ce6700 1 --2- 172.21.15.28:0/1201786666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc0103e30 0x7f2bc019fa60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:26.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:26.752+0000 7f2bc6ce6700 1 --2- 172.21.15.28:0/1201786666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2bc0102290 0x7f2bc01984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:26.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:26.752+0000 7f2bc6ce6700 1 --2- 172.21.15.28:0/1201786666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2bc0103490 0x7f2bc01989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:26.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:26.752+0000 7f2bc6ce6700 1 -- 172.21.15.28:0/1201786666 >> 172.21.15.28:0/1201786666 conn(0x7f2bc00fd830 msgr2=0x7f2bc0106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:26.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:26.752+0000 7f2bc6ce6700 1 -- 172.21.15.28:0/1201786666 shutdown_connections 2024-04-25T01:27:26.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:27:26.752+0000 7f2bc6ce6700 1 -- 172.21.15.28:0/1201786666 wait complete. 2024-04-25T01:27:26.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:26 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:26.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:26 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:26.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:26 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:26.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:26 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:26.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:26 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:26.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:26 smithi130 ceph-mon[32702]: from='osd.6 [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T01:27:26.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:26 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:26.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:26 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:26.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:26 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:26.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:26 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:26.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:26 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:26.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:26 smithi032 ceph-mon[32461]: from='osd.6 [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T01:27:27.457 DEBUG:teuthology.orchestra.run.smithi028:osd.6> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.6.service 2024-04-25T01:27:27.459 INFO:tasks.cephadm:Deploying osd.7 on smithi032 with /dev/vg_nvme/lv_2... 2024-04-25T01:27:27.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T01:27:27.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:27 smithi028 ceph-mon[29390]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:27.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:27 smithi028 ceph-mon[29390]: from='osd.6 [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T01:27:27.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:27 smithi028 ceph-mon[29390]: osdmap e46: 7 total, 6 up, 7 in 2024-04-25T01:27:27.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:27 smithi028 ceph-mon[29390]: from='osd.6 [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-25T01:27:27.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:27 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:27:27.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:27 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:27.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:27 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:27.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:27 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:27.714 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:27:27 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:27:27.614+0000 7fc3e6b7d700 -1 osd.6 0 waiting for initial osdmap 2024-04-25T01:27:27.714 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:27:27 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:27:27.622+0000 7fc3e2970700 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:27:27.719 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:27:27.841 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:27 smithi032 ceph-mon[32461]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:27.841 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:27 smithi032 ceph-mon[32461]: from='osd.6 [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T01:27:27.841 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:27 smithi032 ceph-mon[32461]: osdmap e46: 7 total, 6 up, 7 in 2024-04-25T01:27:27.841 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:27 smithi032 ceph-mon[32461]: from='osd.6 [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-25T01:27:27.842 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:27 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:27:27.842 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:27 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:27.842 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:27 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:27.842 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:27 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:27.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:27 smithi130 ceph-mon[32702]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:27.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:27 smithi130 ceph-mon[32702]: from='osd.6 [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T01:27:27.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:27 smithi130 ceph-mon[32702]: osdmap e46: 7 total, 6 up, 7 in 2024-04-25T01:27:27.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:27 smithi130 ceph-mon[32702]: from='osd.6 [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-25T01:27:27.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:27 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:27:27.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:27 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:27.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:27 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:27.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:27 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:28.803 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:28 smithi028 ceph-mon[29390]: from='osd.6 [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-25T01:27:28.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:28 smithi028 ceph-mon[29390]: osdmap e47: 7 total, 6 up, 7 in 2024-04-25T01:27:28.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:28 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:27:28.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:28 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:27:28.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:28 smithi130 ceph-mon[32702]: from='osd.6 [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-25T01:27:28.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:28 smithi130 ceph-mon[32702]: osdmap e47: 7 total, 6 up, 7 in 2024-04-25T01:27:28.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:28 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:27:28.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:28 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:27:28.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:28 smithi032 ceph-mon[32461]: from='osd.6 [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-25T01:27:28.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:28 smithi032 ceph-mon[32461]: osdmap e47: 7 total, 6 up, 7 in 2024-04-25T01:27:28.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:28 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:27:28.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:28 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:27:29.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:29 smithi032 ceph-mon[32461]: purged_snaps scrub starts 2024-04-25T01:27:29.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:29 smithi032 ceph-mon[32461]: purged_snaps scrub ok 2024-04-25T01:27:29.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:29 smithi032 ceph-mon[32461]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:29.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:29 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:27:29.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:29 smithi032 ceph-mon[32461]: osd.6 [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523] boot 2024-04-25T01:27:29.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:29 smithi032 ceph-mon[32461]: osdmap e48: 7 total, 7 up, 7 in 2024-04-25T01:27:29.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:29 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:27:29.862 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:29 smithi130 ceph-mon[32702]: purged_snaps scrub starts 2024-04-25T01:27:29.862 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:29 smithi130 ceph-mon[32702]: purged_snaps scrub ok 2024-04-25T01:27:29.862 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:29 smithi130 ceph-mon[32702]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:29.862 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:29 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:27:29.862 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:29 smithi130 ceph-mon[32702]: osd.6 [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523] boot 2024-04-25T01:27:29.862 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:29 smithi130 ceph-mon[32702]: osdmap e48: 7 total, 7 up, 7 in 2024-04-25T01:27:29.862 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:29 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:27:29.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:29 smithi028 ceph-mon[29390]: purged_snaps scrub starts 2024-04-25T01:27:29.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:29 smithi028 ceph-mon[29390]: purged_snaps scrub ok 2024-04-25T01:27:29.970 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:29 smithi028 ceph-mon[29390]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:27:29.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:29 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:27:29.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:29 smithi028 ceph-mon[29390]: osd.6 [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523] boot 2024-04-25T01:27:29.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:29 smithi028 ceph-mon[29390]: osdmap e48: 7 total, 7 up, 7 in 2024-04-25T01:27:29.971 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:29 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:27:30.013 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T01:27:30.013 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T01:27:30.014 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-04-25T01:27:30.014 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-04-25T01:27:30.014 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00823909 s, 1.3 GB/s 2024-04-25T01:27:30.014 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-04-25T01:27:30.732 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_2 2024-04-25T01:27:30.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:30 smithi130 ceph-mon[32702]: osdmap e49: 7 total, 7 up, 7 in 2024-04-25T01:27:30.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:30 smithi032 ceph-mon[32461]: osdmap e49: 7 total, 7 up, 7 in 2024-04-25T01:27:30.986 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:27:31.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:30 smithi028 ceph-mon[29390]: osdmap e49: 7 total, 7 up, 7 in 2024-04-25T01:27:32.035 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:31 smithi032 ceph-mon[32461]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:32.035 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:31 smithi032 ceph-mon[32461]: Detected new or changed devices on smithi028 2024-04-25T01:27:32.035 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:32.035 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:32.035 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:32.035 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:32.035 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:32.035 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:31 smithi032 ceph-mon[32461]: Adjusting osd_memory_target on smithi028 to 5697M 2024-04-25T01:27:32.036 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:32.036 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:32.036 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:32.036 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:32.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:31 smithi130 ceph-mon[32702]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:32.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:31 smithi130 ceph-mon[32702]: Detected new or changed devices on smithi028 2024-04-25T01:27:32.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:32.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:32.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:32.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:32.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:32.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:31 smithi130 ceph-mon[32702]: Adjusting osd_memory_target on smithi028 to 5697M 2024-04-25T01:27:32.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:32.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:32.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:32.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:32.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:31 smithi028 ceph-mon[29390]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:32.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:31 smithi028 ceph-mon[29390]: Detected new or changed devices on smithi028 2024-04-25T01:27:32.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:32.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:32.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:32.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:32.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:32.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:31 smithi028 ceph-mon[29390]: Adjusting osd_memory_target on smithi028 to 5697M 2024-04-25T01:27:32.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:32.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:32.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:32.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:32.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.693+0000 7f632ed51700 1 -- 172.21.15.32:0/133647693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6328105650 msgr2=0x7f6328107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:32.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.693+0000 7f632ed51700 1 --2- 172.21.15.32:0/133647693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6328105650 0x7f6328107a40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6314009990 tx=0x7f6314009ca0 comp rx=0 tx=0).stop 2024-04-25T01:27:32.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.693+0000 7f632ed51700 1 -- 172.21.15.32:0/133647693 shutdown_connections 2024-04-25T01:27:32.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.693+0000 7f632ed51700 1 --2- 172.21.15.32:0/133647693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6328107f80 0x7f632810a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:32.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.693+0000 7f632ed51700 1 --2- 172.21.15.32:0/133647693 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6328102d20 0x7f6328105110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.693+0000 7f632ed51700 1 --2- 172.21.15.32:0/133647693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6328105650 0x7f6328107a40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.693+0000 7f632ed51700 1 -- 172.21.15.32:0/133647693 >> 172.21.15.32:0/133647693 conn(0x7f63280fc730 msgr2=0x7f63280feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.693+0000 7f632ed51700 1 -- 172.21.15.32:0/133647693 shutdown_connections 2024-04-25T01:27:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.693+0000 7f632ed51700 1 -- 172.21.15.32:0/133647693 wait complete. 2024-04-25T01:27:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.694+0000 7f632ed51700 1 Processor -- start 2024-04-25T01:27:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.694+0000 7f632ed51700 1 -- start start 2024-04-25T01:27:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.694+0000 7f632ed51700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6328102d20 0x7f6328102680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.694+0000 7f632ed51700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6328105650 0x7f6328100cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.694+0000 7f632ed51700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6328107f80 0x7f6328101210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.694+0000 7f632ed51700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f632810dea0 con 0x7f6328105650 2024-04-25T01:27:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.694+0000 7f632ed51700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f632810dd20 con 0x7f6328102d20 2024-04-25T01:27:32.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.694+0000 7f632ed51700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f632810e020 con 0x7f6328107f80 2024-04-25T01:27:32.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.694+0000 7f632caed700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6328102d20 0x7f6328102680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:32.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.694+0000 7f632caed700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6328102d20 0x7f6328102680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46906/0 (socket says 172.21.15.32:46906) 2024-04-25T01:27:32.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.694+0000 7f632caed700 1 -- 172.21.15.32:0/659317636 learned_addr learned my addr 172.21.15.32:0/659317636 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-25T01:27:32.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.694+0000 7f632d2ee700 1 --2- 172.21.15.32:0/659317636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6328107f80 0x7f6328101210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:32.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.694+0000 7f6327fff700 1 --2- 172.21.15.32:0/659317636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6328105650 0x7f6328100cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:32.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.694+0000 7f632caed700 1 -- 172.21.15.32:0/659317636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6328107f80 msgr2=0x7f6328101210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:32.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.694+0000 7f632caed700 1 --2- 172.21.15.32:0/659317636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6328107f80 0x7f6328101210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:32.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.694+0000 7f632caed700 1 -- 172.21.15.32:0/659317636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6328105650 msgr2=0x7f6328100cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:32.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.694+0000 7f632caed700 1 --2- 172.21.15.32:0/659317636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6328105650 0x7f6328100cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:32.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.694+0000 7f632caed700 1 -- 172.21.15.32:0/659317636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6328101ac0 con 0x7f6328102d20 2024-04-25T01:27:32.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.695+0000 7f632caed700 1 --2- 172.21.15.32:0/659317636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6328102d20 0x7f6328102680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f631c00d980 tx=0x7f631c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:32.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.695+0000 7f6325ffb700 1 -- 172.21.15.32:0/659317636 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631c017070 con 0x7f6328102d20 2024-04-25T01:27:32.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.695+0000 7f632ed51700 1 -- 172.21.15.32:0/659317636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63281a26b0 con 0x7f6328102d20 2024-04-25T01:27:32.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.695+0000 7f6325ffb700 1 -- 172.21.15.32:0/659317636 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f631c005020 con 0x7f6328102d20 2024-04-25T01:27:32.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.695+0000 7f6325ffb700 1 -- 172.21.15.32:0/659317636 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631c009bc0 con 0x7f6328102d20 2024-04-25T01:27:32.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.695+0000 7f632ed51700 1 -- 172.21.15.32:0/659317636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63281a2b70 con 0x7f6328102d20 2024-04-25T01:27:32.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.696+0000 7f632ed51700 1 -- 172.21.15.32:0/659317636 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6328068f20 con 0x7f6328102d20 2024-04-25T01:27:32.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.697+0000 7f6325ffb700 1 -- 172.21.15.32:0/659317636 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f631c0051d0 con 0x7f6328102d20 2024-04-25T01:27:32.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.697+0000 7f6325ffb700 1 --2- 172.21.15.32:0/659317636 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f63180a2c20 0x7f63180a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:32.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.698+0000 7f6325ffb700 1 -- 172.21.15.32:0/659317636 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4965+0+0 (secure 0 0 0) 0x7f631c0ce300 con 0x7f6328102d20 2024-04-25T01:27:32.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.698+0000 7f6327fff700 1 --2- 172.21.15.32:0/659317636 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f63180a2c20 0x7f63180a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:32.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.698+0000 7f6327fff700 1 --2- 172.21.15.32:0/659317636 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f63180a2c20 0x7f63180a50e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f631400b5c0 tx=0x7f6314005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:32.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.702+0000 7f6325ffb700 1 -- 172.21.15.32:0/659317636 <== mon.2 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f631c020020 con 0x7f6328102d20 2024-04-25T01:27:32.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:32.917+0000 7f632ed51700 1 -- 172.21.15.32:0/659317636 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f6328062670 con 0x7f63180a2c20 2024-04-25T01:27:34.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:33 smithi130 ceph-mon[32702]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:34.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:33 smithi130 ceph-mon[32702]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:27:34.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:27:34.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:27:34.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:34.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:33 smithi032 ceph-mon[32461]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:34.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:33 smithi032 ceph-mon[32461]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:27:34.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:27:34.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:27:34.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:34.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:33 smithi028 ceph-mon[29390]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:34.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:33 smithi028 ceph-mon[29390]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:27:34.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:27:34.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:27:34.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:36.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:35 smithi130 ceph-mon[32702]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:36.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:35 smithi032 ceph-mon[32461]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:36.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:35 smithi028 ceph-mon[29390]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:37.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:36 smithi130 ceph-mon[32702]: from='client.? 172.21.15.32:0/4155482292' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea0a98a6-80dc-4c7d-b335-b22f9326de2d"}]: dispatch 2024-04-25T01:27:37.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:36 smithi130 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea0a98a6-80dc-4c7d-b335-b22f9326de2d"}]: dispatch 2024-04-25T01:27:37.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:36 smithi130 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea0a98a6-80dc-4c7d-b335-b22f9326de2d"}]': finished 2024-04-25T01:27:37.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:36 smithi130 ceph-mon[32702]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T01:27:37.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:36 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:27:37.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:36 smithi130 ceph-mon[32702]: from='client.? 172.21.15.32:0/3858782550' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:27:37.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:36 smithi032 ceph-mon[32461]: from='client.? 172.21.15.32:0/4155482292' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea0a98a6-80dc-4c7d-b335-b22f9326de2d"}]: dispatch 2024-04-25T01:27:37.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:36 smithi032 ceph-mon[32461]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea0a98a6-80dc-4c7d-b335-b22f9326de2d"}]: dispatch 2024-04-25T01:27:37.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:36 smithi032 ceph-mon[32461]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea0a98a6-80dc-4c7d-b335-b22f9326de2d"}]': finished 2024-04-25T01:27:37.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:36 smithi032 ceph-mon[32461]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T01:27:37.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:36 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:27:37.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:36 smithi032 ceph-mon[32461]: from='client.? 172.21.15.32:0/3858782550' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:27:37.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:36 smithi028 ceph-mon[29390]: from='client.? 172.21.15.32:0/4155482292' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea0a98a6-80dc-4c7d-b335-b22f9326de2d"}]: dispatch 2024-04-25T01:27:37.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:36 smithi028 ceph-mon[29390]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea0a98a6-80dc-4c7d-b335-b22f9326de2d"}]: dispatch 2024-04-25T01:27:37.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:36 smithi028 ceph-mon[29390]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea0a98a6-80dc-4c7d-b335-b22f9326de2d"}]': finished 2024-04-25T01:27:37.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:36 smithi028 ceph-mon[29390]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T01:27:37.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:36 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:27:37.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:36 smithi028 ceph-mon[29390]: from='client.? 172.21.15.32:0/3858782550' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:27:38.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:37 smithi130 ceph-mon[32702]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:38.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:37 smithi032 ceph-mon[32461]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:38.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:37 smithi028 ceph-mon[29390]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:39.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:39 smithi032 ceph-mon[32461]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:40.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:39 smithi130 ceph-mon[32702]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:40.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:39 smithi028 ceph-mon[29390]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:42.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:41 smithi130 ceph-mon[32702]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:42.213 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:41 smithi032 ceph-mon[32461]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:42.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:41 smithi028 ceph-mon[29390]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:44.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:43 smithi130 ceph-mon[32702]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:44.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:43 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T01:27:44.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:43 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:44.229 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:43 smithi032 ceph-mon[32461]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:44.229 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:43 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T01:27:44.229 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:43 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:44.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:43 smithi028 ceph-mon[29390]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:44.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:43 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T01:27:44.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:43 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:45.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:44 smithi130 ceph-mon[32702]: Deploying daemon osd.7 on smithi032 2024-04-25T01:27:45.227 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:44 smithi032 ceph-mon[32461]: Deploying daemon osd.7 on smithi032 2024-04-25T01:27:45.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:44 smithi028 ceph-mon[29390]: Deploying daemon osd.7 on smithi032 2024-04-25T01:27:46.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:45 smithi130 ceph-mon[32702]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:46.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:45 smithi032 ceph-mon[32461]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:46.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:45 smithi028 ceph-mon[29390]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:48.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:47 smithi130 ceph-mon[32702]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:48.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:47 smithi032 ceph-mon[32461]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:48.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:47 smithi028 ceph-mon[29390]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:50.151 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:49 smithi032 ceph-mon[32461]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:50.151 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:49 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:50.151 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:49 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:50.151 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:49 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:50.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:49 smithi130 ceph-mon[32702]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:50.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:49 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:50.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:49 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:50.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:49 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:50.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:49 smithi028 ceph-mon[29390]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:50.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:49 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:50.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:49 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:50.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:49 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:52.105 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:51 smithi032 ceph-mon[32461]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:52.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:51 smithi130 ceph-mon[32702]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:52.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:51 smithi028 ceph-mon[29390]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:53.167 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 7 on host 'smithi032' 2024-04-25T01:27:53.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:53.186+0000 7f6325ffb700 1 -- 172.21.15.32:0/659317636 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6328062670 con 0x7f63180a2c20 2024-04-25T01:27:53.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:53.189+0000 7f632ed51700 1 -- 172.21.15.32:0/659317636 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f63180a2c20 msgr2=0x7f63180a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:53.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:53.189+0000 7f632ed51700 1 --2- 172.21.15.32:0/659317636 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f63180a2c20 0x7f63180a50e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f631400b5c0 tx=0x7f6314005fb0 comp rx=0 tx=0).stop 2024-04-25T01:27:53.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:53.189+0000 7f632ed51700 1 -- 172.21.15.32:0/659317636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6328102d20 msgr2=0x7f6328102680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:53.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:53.189+0000 7f632ed51700 1 --2- 172.21.15.32:0/659317636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6328102d20 0x7f6328102680 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f631c00d980 tx=0x7f631c00dc90 comp rx=0 tx=0).stop 2024-04-25T01:27:53.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:53.189+0000 7f632ed51700 1 -- 172.21.15.32:0/659317636 shutdown_connections 2024-04-25T01:27:53.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:53.189+0000 7f632ed51700 1 --2- 172.21.15.32:0/659317636 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f63180a2c20 0x7f63180a50e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:53.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:53.189+0000 7f632ed51700 1 --2- 172.21.15.32:0/659317636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6328107f80 0x7f6328101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:53.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:53.189+0000 7f632ed51700 1 --2- 172.21.15.32:0/659317636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6328102d20 0x7f6328102680 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:53.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:53.189+0000 7f632ed51700 1 --2- 172.21.15.32:0/659317636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6328105650 0x7f6328100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:53.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:53.189+0000 7f632ed51700 1 -- 172.21.15.32:0/659317636 >> 172.21.15.32:0/659317636 conn(0x7f63280fc730 msgr2=0x7f6328103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:53.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:53.189+0000 7f632ed51700 1 -- 172.21.15.32:0/659317636 shutdown_connections 2024-04-25T01:27:53.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:27:53.189+0000 7f632ed51700 1 -- 172.21.15.32:0/659317636 wait complete. 2024-04-25T01:27:53.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:53 smithi032 ceph-mon[32461]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:53.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:53.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:53.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:53.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:53.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:53.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:53 smithi032 ceph-mon[32461]: from='osd.7 [v2:172.21.15.32:6816/1373301181,v1:172.21.15.32:6817/1373301181]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:27:53.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:53 smithi032 ceph-mon[32461]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:27:53.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:53.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:53.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:53.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:53 smithi028 ceph-mon[29390]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:53.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:53.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:53.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:53.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:53.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:53.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:53 smithi028 ceph-mon[29390]: from='osd.7 [v2:172.21.15.32:6816/1373301181,v1:172.21.15.32:6817/1373301181]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:27:53.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:53 smithi028 ceph-mon[29390]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:27:53.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:53.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:53.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:53.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:53 smithi130 ceph-mon[32702]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:53.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:53.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:53.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:53.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:53.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:53.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:53 smithi130 ceph-mon[32702]: from='osd.7 [v2:172.21.15.32:6816/1373301181,v1:172.21.15.32:6817/1373301181]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:27:53.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:53 smithi130 ceph-mon[32702]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:27:53.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:53.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:53.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:27:53.908 DEBUG:teuthology.orchestra.run.smithi032:osd.7> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.7.service 2024-04-25T01:27:53.910 INFO:tasks.cephadm:Deploying osd.8 on smithi130 with /dev/vg_nvme/lv_2... 2024-04-25T01:27:53.910 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T01:27:54.160 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:27:55.122 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:54 smithi032 ceph-mon[32461]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T01:27:55.122 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:54 smithi032 ceph-mon[32461]: osdmap e51: 8 total, 7 up, 8 in 2024-04-25T01:27:55.122 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:54 smithi032 ceph-mon[32461]: from='osd.7 [v2:172.21.15.32:6816/1373301181,v1:172.21.15.32:6817/1373301181]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:27:55.122 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:54 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:27:55.122 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:54 smithi032 ceph-mon[32461]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:27:55.123 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:27:54 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:27:54.969+0000 7fe006554700 -1 osd.7 0 waiting for initial osdmap 2024-04-25T01:27:55.123 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:27:54 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:27:54.980+0000 7fe002347700 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:27:55.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:54 smithi130 ceph-mon[32702]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T01:27:55.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:54 smithi130 ceph-mon[32702]: osdmap e51: 8 total, 7 up, 8 in 2024-04-25T01:27:55.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:54 smithi130 ceph-mon[32702]: from='osd.7 [v2:172.21.15.32:6816/1373301181,v1:172.21.15.32:6817/1373301181]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:27:55.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:54 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:27:55.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:54 smithi130 ceph-mon[32702]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:27:55.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:54 smithi028 ceph-mon[29390]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T01:27:55.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:54 smithi028 ceph-mon[29390]: osdmap e51: 8 total, 7 up, 8 in 2024-04-25T01:27:55.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:54 smithi028 ceph-mon[29390]: from='osd.7 [v2:172.21.15.32:6816/1373301181,v1:172.21.15.32:6817/1373301181]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:27:55.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:54 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:27:55.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:54 smithi028 ceph-mon[29390]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:27:56.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:55 smithi130 ceph-mon[32702]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:56.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:55 smithi130 ceph-mon[32702]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-25T01:27:56.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:55 smithi130 ceph-mon[32702]: osdmap e52: 8 total, 7 up, 8 in 2024-04-25T01:27:56.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:55 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:27:56.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:55 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:27:56.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:55 smithi032 ceph-mon[32461]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:56.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:55 smithi032 ceph-mon[32461]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-25T01:27:56.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:55 smithi032 ceph-mon[32461]: osdmap e52: 8 total, 7 up, 8 in 2024-04-25T01:27:56.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:55 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:27:56.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:55 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:27:56.258 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T01:27:56.258 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T01:27:56.258 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-04-25T01:27:56.258 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-04-25T01:27:56.258 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127944 s, 820 MB/s 2024-04-25T01:27:56.258 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-04-25T01:27:56.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:55 smithi028 ceph-mon[29390]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:27:56.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:55 smithi028 ceph-mon[29390]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-25T01:27:56.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:55 smithi028 ceph-mon[29390]: osdmap e52: 8 total, 7 up, 8 in 2024-04-25T01:27:56.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:55 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:27:56.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:55 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:27:56.894 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi130:vg_nvme/lv_2 2024-04-25T01:27:57.144 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:27:57.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:56 smithi130 ceph-mon[32702]: purged_snaps scrub starts 2024-04-25T01:27:57.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:56 smithi130 ceph-mon[32702]: purged_snaps scrub ok 2024-04-25T01:27:57.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:56 smithi130 ceph-mon[32702]: osd.7 [v2:172.21.15.32:6816/1373301181,v1:172.21.15.32:6817/1373301181] boot 2024-04-25T01:27:57.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:56 smithi130 ceph-mon[32702]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T01:27:57.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:56 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:27:57.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:56 smithi130 ceph-mon[32702]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T01:27:57.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:56 smithi032 ceph-mon[32461]: purged_snaps scrub starts 2024-04-25T01:27:57.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:56 smithi032 ceph-mon[32461]: purged_snaps scrub ok 2024-04-25T01:27:57.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:56 smithi032 ceph-mon[32461]: osd.7 [v2:172.21.15.32:6816/1373301181,v1:172.21.15.32:6817/1373301181] boot 2024-04-25T01:27:57.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:56 smithi032 ceph-mon[32461]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T01:27:57.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:56 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:27:57.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:56 smithi032 ceph-mon[32461]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T01:27:57.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:56 smithi028 ceph-mon[29390]: purged_snaps scrub starts 2024-04-25T01:27:57.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:56 smithi028 ceph-mon[29390]: purged_snaps scrub ok 2024-04-25T01:27:57.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:56 smithi028 ceph-mon[29390]: osd.7 [v2:172.21.15.32:6816/1373301181,v1:172.21.15.32:6817/1373301181] boot 2024-04-25T01:27:57.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:56 smithi028 ceph-mon[29390]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T01:27:57.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:56 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:27:57.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:56 smithi028 ceph-mon[29390]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T01:27:58.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:57 smithi130 ceph-mon[32702]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:27:58.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:57 smithi130 ceph-mon[32702]: Detected new or changed devices on smithi032 2024-04-25T01:27:58.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:57 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:58.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:57 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:58.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:57 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:58.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:57 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:58.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:57 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:58.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:57 smithi130 ceph-mon[32702]: Adjusting osd_memory_target on smithi032 to 5697M 2024-04-25T01:27:58.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:57 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:58.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:57 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:58.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:57 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:58.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:57 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:58.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:57 smithi130 ceph-mon[32702]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T01:27:58.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:57 smithi032 ceph-mon[32461]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:27:58.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:57 smithi032 ceph-mon[32461]: Detected new or changed devices on smithi032 2024-04-25T01:27:58.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:57 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:58.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:57 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:58.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:57 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:58.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:57 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:58.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:57 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:58.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:57 smithi032 ceph-mon[32461]: Adjusting osd_memory_target on smithi032 to 5697M 2024-04-25T01:27:58.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:57 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:58.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:57 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:58.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:57 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:58.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:57 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:58.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:57 smithi032 ceph-mon[32461]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T01:27:58.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:57 smithi028 ceph-mon[29390]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:27:58.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:57 smithi028 ceph-mon[29390]: Detected new or changed devices on smithi032 2024-04-25T01:27:58.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:57 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:58.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:57 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:58.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:57 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:58.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:57 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:58.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:57 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:27:58.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:57 smithi028 ceph-mon[29390]: Adjusting osd_memory_target on smithi032 to 5697M 2024-04-25T01:27:58.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:57 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:58.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:57 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:58.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:57 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:27:58.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:57 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:27:58.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:57 smithi028 ceph-mon[29390]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T01:27:58.821 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.842+0000 7f9cb682f700 1 -- 172.21.15.130:0/3807464736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cb0102d30 msgr2=0x7f9cb01095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:58.822 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.842+0000 7f9cb682f700 1 --2- 172.21.15.130:0/3807464736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cb0102d30 0x7f9cb01095e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9ca800b0a0 tx=0x7f9ca800b3b0 comp rx=0 tx=0).stop 2024-04-25T01:27:58.822 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.842+0000 7f9cb682f700 1 -- 172.21.15.130:0/3807464736 shutdown_connections 2024-04-25T01:27:58.822 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.842+0000 7f9cb682f700 1 --2- 172.21.15.130:0/3807464736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cb0102d30 0x7f9cb01095e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:58.822 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.842+0000 7f9cb682f700 1 --2- 172.21.15.130:0/3807464736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cb0101190 0x7f9cb01015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:58.822 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.842+0000 7f9cb682f700 1 --2- 172.21.15.130:0/3807464736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb0102390 0x7f9cb01027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:58.822 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.842+0000 7f9cb682f700 1 -- 172.21.15.130:0/3807464736 >> 172.21.15.130:0/3807464736 conn(0x7f9cb00fc730 msgr2=0x7f9cb00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:27:58.822 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.842+0000 7f9cb682f700 1 -- 172.21.15.130:0/3807464736 shutdown_connections 2024-04-25T01:27:58.822 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.842+0000 7f9cb682f700 1 -- 172.21.15.130:0/3807464736 wait complete. 2024-04-25T01:27:58.822 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.843+0000 7f9cb682f700 1 Processor -- start 2024-04-25T01:27:58.822 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.843+0000 7f9cb682f700 1 -- start start 2024-04-25T01:27:58.824 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.843+0000 7f9cb682f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cb0101190 0x7f9cb006aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:58.824 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.843+0000 7f9cb682f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cb0102390 0x7f9cb019d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:58.824 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.843+0000 7f9cb682f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb0102d30 0x7f9cb019f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:58.824 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.843+0000 7f9cb682f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb010cdf0 con 0x7f9cb0102d30 2024-04-25T01:27:58.824 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.843+0000 7f9cb682f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb010cc70 con 0x7f9cb0102390 2024-04-25T01:27:58.824 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.843+0000 7f9cb682f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb010cf70 con 0x7f9cb0101190 2024-04-25T01:27:58.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.843+0000 7f9caffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cb0101190 0x7f9cb006aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:58.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.843+0000 7f9caffff700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cb0101190 0x7f9cb006aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:45246/0 (socket says 172.21.15.130:45246) 2024-04-25T01:27:58.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.843+0000 7f9caffff700 1 -- 172.21.15.130:0/899147982 learned_addr learned my addr 172.21.15.130:0/899147982 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-25T01:27:58.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.843+0000 7f9caf7fe700 1 --2- 172.21.15.130:0/899147982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cb0102390 0x7f9cb019d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:58.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.843+0000 7f9cb4dcc700 1 --2- 172.21.15.130:0/899147982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb0102d30 0x7f9cb019f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:58.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.843+0000 7f9caffff700 1 -- 172.21.15.130:0/899147982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cb0102390 msgr2=0x7f9cb019d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:58.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.843+0000 7f9caffff700 1 --2- 172.21.15.130:0/899147982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cb0102390 0x7f9cb019d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:58.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.843+0000 7f9caffff700 1 -- 172.21.15.130:0/899147982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb0102d30 msgr2=0x7f9cb019f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:27:58.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.843+0000 7f9caffff700 1 --2- 172.21.15.130:0/899147982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb0102d30 0x7f9cb019f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:27:58.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.844+0000 7f9caffff700 1 -- 172.21.15.130:0/899147982 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cb01a0150 con 0x7f9cb0101190 2024-04-25T01:27:58.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.844+0000 7f9caffff700 1 --2- 172.21.15.130:0/899147982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cb0101190 0x7f9cb006aa70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9ca000b500 tx=0x7f9ca000b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:58.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.844+0000 7f9cad7fa700 1 -- 172.21.15.130:0/899147982 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0015020 con 0x7f9cb0101190 2024-04-25T01:27:58.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.844+0000 7f9cb682f700 1 -- 172.21.15.130:0/899147982 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cb01a0440 con 0x7f9cb0101190 2024-04-25T01:27:58.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.844+0000 7f9cb682f700 1 -- 172.21.15.130:0/899147982 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cb01a0a00 con 0x7f9cb0101190 2024-04-25T01:27:58.826 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.844+0000 7f9cad7fa700 1 -- 172.21.15.130:0/899147982 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ca0011d10 con 0x7f9cb0101190 2024-04-25T01:27:58.826 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.844+0000 7f9cad7fa700 1 -- 172.21.15.130:0/899147982 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca001ad60 con 0x7f9cb0101190 2024-04-25T01:27:58.826 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.845+0000 7f9cb682f700 1 -- 172.21.15.130:0/899147982 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c90005360 con 0x7f9cb0101190 2024-04-25T01:27:58.826 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.847+0000 7f9cad7fa700 1 -- 172.21.15.130:0/899147982 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9ca0018020 con 0x7f9cb0101190 2024-04-25T01:27:58.827 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.847+0000 7f9cad7fa700 1 --2- 172.21.15.130:0/899147982 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9c9c0a7030 0x7f9c9c0a94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:27:58.827 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.848+0000 7f9cad7fa700 1 -- 172.21.15.130:0/899147982 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5397+0+0 (secure 0 0 0) 0x7f9ca00ce0d0 con 0x7f9cb0101190 2024-04-25T01:27:58.827 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.848+0000 7f9caf7fe700 1 --2- 172.21.15.130:0/899147982 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9c9c0a7030 0x7f9c9c0a94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:27:58.827 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.848+0000 7f9caf7fe700 1 --2- 172.21.15.130:0/899147982 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9c9c0a7030 0x7f9c9c0a94f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9cb006aff0 tx=0x7f9c98009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:27:58.830 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:58.851+0000 7f9cad7fa700 1 -- 172.21.15.130:0/899147982 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9ca001f020 con 0x7f9cb0101190 2024-04-25T01:27:59.037 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:27:59.058+0000 7f9cb682f700 1 -- 172.21.15.130:0/899147982 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f9c90000bf0 con 0x7f9c9c0a7030 2024-04-25T01:27:59.754 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:59 smithi130 ceph-mon[32702]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:27:59.754 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:59 smithi130 ceph-mon[32702]: osdmap e56: 8 total, 8 up, 8 in 2024-04-25T01:27:59.754 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:59 smithi130 ceph-mon[32702]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:27:59.754 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:59 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:27:59.754 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:59 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:27:59.754 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:27:59 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:27:59.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:59 smithi032 ceph-mon[32461]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:27:59.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:59 smithi032 ceph-mon[32461]: osdmap e56: 8 total, 8 up, 8 in 2024-04-25T01:27:59.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:59 smithi032 ceph-mon[32461]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:27:59.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:59 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:27:59.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:59 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:27:59.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:27:59 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:00.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:59 smithi028 ceph-mon[29390]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:00.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:59 smithi028 ceph-mon[29390]: osdmap e56: 8 total, 8 up, 8 in 2024-04-25T01:28:00.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:59 smithi028 ceph-mon[29390]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:28:00.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:59 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:28:00.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:59 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:28:00.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:27:59 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:00.922 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:00 smithi130 ceph-mon[32702]: osdmap e57: 8 total, 8 up, 8 in 2024-04-25T01:28:00.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:00 smithi032 ceph-mon[32461]: osdmap e57: 8 total, 8 up, 8 in 2024-04-25T01:28:01.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:00 smithi028 ceph-mon[29390]: osdmap e57: 8 total, 8 up, 8 in 2024-04-25T01:28:01.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:01 smithi130 ceph-mon[32702]: pgmap v187: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:01.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:01 smithi032 ceph-mon[32461]: pgmap v187: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:02.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:01 smithi028 ceph-mon[29390]: pgmap v187: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:02.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:02 smithi130 ceph-mon[32702]: from='client.? 172.21.15.130:0/1211019316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5beef33-36c9-4759-a168-c104ee1b5f75"}]: dispatch 2024-04-25T01:28:02.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:02 smithi130 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5beef33-36c9-4759-a168-c104ee1b5f75"}]: dispatch 2024-04-25T01:28:02.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:02 smithi130 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5beef33-36c9-4759-a168-c104ee1b5f75"}]': finished 2024-04-25T01:28:02.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:02 smithi130 ceph-mon[32702]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T01:28:02.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:02 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:28:02.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:02 smithi130 ceph-mon[32702]: from='client.? 172.21.15.130:0/3089917238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:28:02.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:02 smithi032 ceph-mon[32461]: from='client.? 172.21.15.130:0/1211019316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5beef33-36c9-4759-a168-c104ee1b5f75"}]: dispatch 2024-04-25T01:28:02.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:02 smithi032 ceph-mon[32461]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5beef33-36c9-4759-a168-c104ee1b5f75"}]: dispatch 2024-04-25T01:28:02.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:02 smithi032 ceph-mon[32461]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5beef33-36c9-4759-a168-c104ee1b5f75"}]': finished 2024-04-25T01:28:02.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:02 smithi032 ceph-mon[32461]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T01:28:02.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:02 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:28:02.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:02 smithi032 ceph-mon[32461]: from='client.? 172.21.15.130:0/3089917238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:28:03.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:02 smithi028 ceph-mon[29390]: from='client.? 172.21.15.130:0/1211019316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5beef33-36c9-4759-a168-c104ee1b5f75"}]: dispatch 2024-04-25T01:28:03.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:02 smithi028 ceph-mon[29390]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5beef33-36c9-4759-a168-c104ee1b5f75"}]: dispatch 2024-04-25T01:28:03.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:02 smithi028 ceph-mon[29390]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5beef33-36c9-4759-a168-c104ee1b5f75"}]': finished 2024-04-25T01:28:03.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:02 smithi028 ceph-mon[29390]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T01:28:03.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:02 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:28:03.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:02 smithi028 ceph-mon[29390]: from='client.? 172.21.15.130:0/3089917238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:28:04.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:03 smithi028 ceph-mon[29390]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:04.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:03 smithi130 ceph-mon[32702]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:04.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:03 smithi032 ceph-mon[32461]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:06.042 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:05 smithi130 ceph-mon[32702]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:06.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:05 smithi028 ceph-mon[29390]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:06.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:05 smithi032 ceph-mon[32461]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:08.037 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:07 smithi130 ceph-mon[32702]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:08.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:07 smithi028 ceph-mon[29390]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:08.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:07 smithi032 ceph-mon[32461]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:10.052 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:09 smithi130 ceph-mon[32702]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:10.053 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:09 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T01:28:10.053 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:09 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:10.053 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:09 smithi130 ceph-mon[32702]: Deploying daemon osd.8 on smithi130 2024-04-25T01:28:10.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:09 smithi028 ceph-mon[29390]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:10.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:09 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T01:28:10.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:09 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:10.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:09 smithi028 ceph-mon[29390]: Deploying daemon osd.8 on smithi130 2024-04-25T01:28:10.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:09 smithi032 ceph-mon[32461]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:10.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:09 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T01:28:10.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:09 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:10.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:09 smithi032 ceph-mon[32461]: Deploying daemon osd.8 on smithi130 2024-04-25T01:28:12.038 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:11 smithi130 ceph-mon[32702]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:12.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:11 smithi028 ceph-mon[29390]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:12.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:11 smithi032 ceph-mon[32461]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:14.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:13 smithi028 ceph-mon[29390]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:14.072 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:13 smithi130 ceph-mon[32702]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:14.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:13 smithi032 ceph-mon[32461]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:15.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:15 smithi130 ceph-mon[32702]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:15.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:15 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:15.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:15 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:15.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:15 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:28:15.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:15 smithi032 ceph-mon[32461]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:15.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:15.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:15.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:15 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:28:16.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:15 smithi028 ceph-mon[29390]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:16.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:15 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:16.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:15 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:16.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:15 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:28:17.886 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:17 smithi130 ceph-mon[32702]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:17.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:17 smithi032 ceph-mon[32461]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:18.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:17 smithi028 ceph-mon[29390]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:18.204 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:28:18.225+0000 7f9cad7fa700 1 -- 172.21.15.130:0/899147982 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9c90000bf0 con 0x7f9c9c0a7030 2024-04-25T01:28:18.205 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 8 on host 'smithi130' 2024-04-25T01:28:18.206 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:28:18.227+0000 7f9cb682f700 1 -- 172.21.15.130:0/899147982 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9c9c0a7030 msgr2=0x7f9c9c0a94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:18.206 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:28:18.227+0000 7f9cb682f700 1 --2- 172.21.15.130:0/899147982 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9c9c0a7030 0x7f9c9c0a94f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9cb006aff0 tx=0x7f9c98009260 comp rx=0 tx=0).stop 2024-04-25T01:28:18.206 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:28:18.227+0000 7f9cb682f700 1 -- 172.21.15.130:0/899147982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cb0101190 msgr2=0x7f9cb006aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:18.206 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:28:18.227+0000 7f9cb682f700 1 --2- 172.21.15.130:0/899147982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cb0101190 0x7f9cb006aa70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9ca000b500 tx=0x7f9ca000b810 comp rx=0 tx=0).stop 2024-04-25T01:28:18.207 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:28:18.227+0000 7f9cb682f700 1 -- 172.21.15.130:0/899147982 shutdown_connections 2024-04-25T01:28:18.207 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:28:18.227+0000 7f9cb682f700 1 --2- 172.21.15.130:0/899147982 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9c9c0a7030 0x7f9c9c0a94f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:18.207 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:28:18.227+0000 7f9cb682f700 1 --2- 172.21.15.130:0/899147982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb0102d30 0x7f9cb019f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:18.207 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:28:18.227+0000 7f9cb682f700 1 --2- 172.21.15.130:0/899147982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cb0101190 0x7f9cb006aa70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:18.207 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:28:18.227+0000 7f9cb682f700 1 --2- 172.21.15.130:0/899147982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cb0102390 0x7f9cb019d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:18.207 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:28:18.227+0000 7f9cb682f700 1 -- 172.21.15.130:0/899147982 >> 172.21.15.130:0/899147982 conn(0x7f9cb00fc730 msgr2=0x7f9cb0105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:18.207 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:28:18.227+0000 7f9cb682f700 1 -- 172.21.15.130:0/899147982 shutdown_connections 2024-04-25T01:28:18.207 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:28:18.227+0000 7f9cb682f700 1 -- 172.21.15.130:0/899147982 wait complete. 2024-04-25T01:28:18.825 DEBUG:teuthology.orchestra.run.smithi130:osd.8> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.8.service 2024-04-25T01:28:18.827 INFO:tasks.cephadm:Deploying osd.9 on smithi028 with /dev/vg_nvme/lv_1... 2024-04-25T01:28:18.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T01:28:19.101 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:28:19.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:19 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:19.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:19 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:19.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:19 smithi130 ceph-mon[32702]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:19.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:19 smithi130 ceph-mon[32702]: from='osd.8 [v2:172.21.15.130:6816/2911117678,v1:172.21.15.130:6817/2911117678]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:28:19.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:19 smithi130 ceph-mon[32702]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:28:19.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:19 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:19.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:19 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:19.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:19 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:19.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:19 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:19.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:19 smithi028 ceph-mon[29390]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:19.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:19 smithi028 ceph-mon[29390]: from='osd.8 [v2:172.21.15.130:6816/2911117678,v1:172.21.15.130:6817/2911117678]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:28:19.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:19 smithi028 ceph-mon[29390]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:28:19.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:19 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:19.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:19 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:19.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:19 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:19.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:19 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:19.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:19 smithi032 ceph-mon[32461]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:19.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:19 smithi032 ceph-mon[32461]: from='osd.8 [v2:172.21.15.130:6816/2911117678,v1:172.21.15.130:6817/2911117678]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:28:19.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:19 smithi032 ceph-mon[32461]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:28:19.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:19 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:19.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:19 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:20.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:20 smithi130 ceph-mon[32702]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T01:28:20.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:20 smithi130 ceph-mon[32702]: osdmap e59: 9 total, 8 up, 9 in 2024-04-25T01:28:20.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:28:20.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:20 smithi130 ceph-mon[32702]: from='osd.8 [v2:172.21.15.130:6816/2911117678,v1:172.21.15.130:6817/2911117678]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:28:20.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:20 smithi130 ceph-mon[32702]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:28:20.460 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:28:20 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:28:20.231+0000 7fa723ba8700 -1 osd.8 0 waiting for initial osdmap 2024-04-25T01:28:20.461 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:28:20 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:28:20.241+0000 7fa71f99b700 -1 osd.8 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:28:20.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:20 smithi028 ceph-mon[29390]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T01:28:20.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:20 smithi028 ceph-mon[29390]: osdmap e59: 9 total, 8 up, 9 in 2024-04-25T01:28:20.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:28:20.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:20 smithi028 ceph-mon[29390]: from='osd.8 [v2:172.21.15.130:6816/2911117678,v1:172.21.15.130:6817/2911117678]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:28:20.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:20 smithi028 ceph-mon[29390]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:28:20.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:20 smithi032 ceph-mon[32461]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T01:28:20.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:20 smithi032 ceph-mon[32461]: osdmap e59: 9 total, 8 up, 9 in 2024-04-25T01:28:20.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:20 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:28:20.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:20 smithi032 ceph-mon[32461]: from='osd.8 [v2:172.21.15.130:6816/2911117678,v1:172.21.15.130:6817/2911117678]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:28:20.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:20 smithi032 ceph-mon[32461]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:28:21.347 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:21 smithi028 ceph-mon[29390]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-25T01:28:21.348 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:21 smithi028 ceph-mon[29390]: osdmap e60: 9 total, 8 up, 9 in 2024-04-25T01:28:21.348 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:21 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:28:21.348 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:21 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:28:21.348 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:21 smithi028 ceph-mon[29390]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T01:28:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T01:28:21.348 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-25T01:28:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-25T01:28:21.349 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00838282 s, 1.3 GB/s 2024-04-25T01:28:21.349 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-25T01:28:21.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:21 smithi130 ceph-mon[32702]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-25T01:28:21.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:21 smithi130 ceph-mon[32702]: osdmap e60: 9 total, 8 up, 9 in 2024-04-25T01:28:21.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:21 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:28:21.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:21 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:28:21.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:21 smithi130 ceph-mon[32702]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:21.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:21 smithi032 ceph-mon[32461]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-25T01:28:21.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:21 smithi032 ceph-mon[32461]: osdmap e60: 9 total, 8 up, 9 in 2024-04-25T01:28:21.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:21 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:28:21.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:21 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:28:21.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:21 smithi032 ceph-mon[32461]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:28:22.144 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_1 2024-04-25T01:28:22.390 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:22 smithi130 ceph-mon[32702]: purged_snaps scrub starts 2024-04-25T01:28:22.390 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:22 smithi130 ceph-mon[32702]: purged_snaps scrub ok 2024-04-25T01:28:22.390 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:22 smithi130 ceph-mon[32702]: osd.8 [v2:172.21.15.130:6816/2911117678,v1:172.21.15.130:6817/2911117678] boot 2024-04-25T01:28:22.390 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:22 smithi130 ceph-mon[32702]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T01:28:22.390 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:22 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:28:22.390 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:22 smithi130 ceph-mon[32702]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T01:28:22.399 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:28:22.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:22 smithi032 ceph-mon[32461]: purged_snaps scrub starts 2024-04-25T01:28:22.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:22 smithi032 ceph-mon[32461]: purged_snaps scrub ok 2024-04-25T01:28:22.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:22 smithi032 ceph-mon[32461]: osd.8 [v2:172.21.15.130:6816/2911117678,v1:172.21.15.130:6817/2911117678] boot 2024-04-25T01:28:22.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:22 smithi032 ceph-mon[32461]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T01:28:22.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:22 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:28:22.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:22 smithi032 ceph-mon[32461]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T01:28:22.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:22 smithi028 ceph-mon[29390]: purged_snaps scrub starts 2024-04-25T01:28:22.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:22 smithi028 ceph-mon[29390]: purged_snaps scrub ok 2024-04-25T01:28:22.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:22 smithi028 ceph-mon[29390]: osd.8 [v2:172.21.15.130:6816/2911117678,v1:172.21.15.130:6817/2911117678] boot 2024-04-25T01:28:22.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:22 smithi028 ceph-mon[29390]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T01:28:22.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:22 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:28:22.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:22 smithi028 ceph-mon[29390]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T01:28:23.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:23 smithi130 ceph-mon[32702]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:23.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:23 smithi130 ceph-mon[32702]: Detected new or changed devices on smithi130 2024-04-25T01:28:23.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:23.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:23.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:23.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:23.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:23.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:23 smithi130 ceph-mon[32702]: Adjusting osd_memory_target on smithi130 to 5696M 2024-04-25T01:28:23.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:23.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:23.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:28:23.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:23.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:28:23.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:23.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:28:23.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:23.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:23 smithi130 ceph-mon[32702]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T01:28:23.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:23 smithi032 ceph-mon[32461]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:23.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:23 smithi032 ceph-mon[32461]: Detected new or changed devices on smithi130 2024-04-25T01:28:23.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:23.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:23.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:23.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:23.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:23.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:23 smithi032 ceph-mon[32461]: Adjusting osd_memory_target on smithi130 to 5696M 2024-04-25T01:28:23.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:23.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:23.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:28:23.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:23.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:28:23.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:23.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:28:23.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:23.737 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:23 smithi032 ceph-mon[32461]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T01:28:23.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:23 smithi028 ceph-mon[29390]: pgmap v203: 1 pgs: 1 peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:23.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:23 smithi028 ceph-mon[29390]: Detected new or changed devices on smithi130 2024-04-25T01:28:23.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:23.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:23.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:23.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:23.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:23.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:23 smithi028 ceph-mon[29390]: Adjusting osd_memory_target on smithi130 to 5696M 2024-04-25T01:28:23.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:23.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:23.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:28:23.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:23.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:28:23.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:23.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:28:23.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:23.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:23 smithi028 ceph-mon[29390]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T01:28:24.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.175+0000 7fbd5343c700 1 -- 172.21.15.28:0/340217091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd4c1011b0 msgr2=0x7fbd4c1015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:24.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.175+0000 7fbd5343c700 1 --2- 172.21.15.28:0/340217091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd4c1011b0 0x7fbd4c1015d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbd480099c0 tx=0x7fbd48009cd0 comp rx=0 tx=0).stop 2024-04-25T01:28:24.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.175+0000 7fbd5343c700 1 -- 172.21.15.28:0/340217091 shutdown_connections 2024-04-25T01:28:24.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.175+0000 7fbd5343c700 1 --2- 172.21.15.28:0/340217091 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd4c102d50 0x7fbd4c109600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.175+0000 7fbd5343c700 1 --2- 172.21.15.28:0/340217091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd4c1011b0 0x7fbd4c1015d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.175+0000 7fbd5343c700 1 --2- 172.21.15.28:0/340217091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd4c1023b0 0x7fbd4c102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.175+0000 7fbd5343c700 1 -- 172.21.15.28:0/340217091 >> 172.21.15.28:0/340217091 conn(0x7fbd4c0fc750 msgr2=0x7fbd4c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.175+0000 7fbd5343c700 1 -- 172.21.15.28:0/340217091 shutdown_connections 2024-04-25T01:28:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.175+0000 7fbd5343c700 1 -- 172.21.15.28:0/340217091 wait complete. 2024-04-25T01:28:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.176+0000 7fbd5343c700 1 Processor -- start 2024-04-25T01:28:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.176+0000 7fbd5343c700 1 -- start start 2024-04-25T01:28:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.176+0000 7fbd5343c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd4c1011b0 0x7fbd4c06b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.176+0000 7fbd5343c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd4c1023b0 0x7fbd4c19d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.176+0000 7fbd5343c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd4c102d50 0x7fbd4c19f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.176+0000 7fbd5343c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c10cdf0 con 0x7fbd4c1023b0 2024-04-25T01:28:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.176+0000 7fbd5343c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c10cc70 con 0x7fbd4c1011b0 2024-04-25T01:28:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.176+0000 7fbd5343c700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c10cf70 con 0x7fbd4c102d50 2024-04-25T01:28:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.176+0000 7fbd509d7700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd4c1023b0 0x7fbd4c19d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.176+0000 7fbd509d7700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd4c1023b0 0x7fbd4c19d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58300/0 (socket says 172.21.15.28:58300) 2024-04-25T01:28:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.176+0000 7fbd509d7700 1 -- 172.21.15.28:0/1346343625 learned_addr learned my addr 172.21.15.28:0/1346343625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:28:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.176+0000 7fbd511d8700 1 --2- 172.21.15.28:0/1346343625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd4c1011b0 0x7fbd4c06b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.176+0000 7fbd519d9700 1 --2- 172.21.15.28:0/1346343625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd4c102d50 0x7fbd4c19f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.177+0000 7fbd509d7700 1 -- 172.21.15.28:0/1346343625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd4c102d50 msgr2=0x7fbd4c19f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.177+0000 7fbd509d7700 1 --2- 172.21.15.28:0/1346343625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd4c102d50 0x7fbd4c19f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.177+0000 7fbd509d7700 1 -- 172.21.15.28:0/1346343625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd4c1011b0 msgr2=0x7fbd4c06b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.177+0000 7fbd509d7700 1 --2- 172.21.15.28:0/1346343625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd4c1011b0 0x7fbd4c06b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.177+0000 7fbd509d7700 1 -- 172.21.15.28:0/1346343625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd4c1a0150 con 0x7fbd4c1023b0 2024-04-25T01:28:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.177+0000 7fbd511d8700 1 --2- 172.21.15.28:0/1346343625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd4c1011b0 0x7fbd4c06b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:28:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.177+0000 7fbd509d7700 1 --2- 172.21.15.28:0/1346343625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd4c1023b0 0x7fbd4c19d240 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c00d950 tx=0x7fbd3c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.177+0000 7fbd427fc700 1 -- 172.21.15.28:0/1346343625 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c016070 con 0x7fbd4c1023b0 2024-04-25T01:28:24.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.177+0000 7fbd427fc700 1 -- 172.21.15.28:0/1346343625 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd3c00fd70 con 0x7fbd4c1023b0 2024-04-25T01:28:24.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.177+0000 7fbd427fc700 1 -- 172.21.15.28:0/1346343625 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c00e610 con 0x7fbd4c1023b0 2024-04-25T01:28:24.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.177+0000 7fbd5343c700 1 -- 172.21.15.28:0/1346343625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd4c1a0440 con 0x7fbd4c1023b0 2024-04-25T01:28:24.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.177+0000 7fbd5343c700 1 -- 172.21.15.28:0/1346343625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd4c1a0a00 con 0x7fbd4c1023b0 2024-04-25T01:28:24.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.178+0000 7fbd5343c700 1 -- 172.21.15.28:0/1346343625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd4c068f20 con 0x7fbd4c1023b0 2024-04-25T01:28:24.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.180+0000 7fbd427fc700 1 -- 172.21.15.28:0/1346343625 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbd3c019070 con 0x7fbd4c1023b0 2024-04-25T01:28:24.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.180+0000 7fbd427fc700 1 --2- 172.21.15.28:0/1346343625 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbd380a2c20 0x7fbd380a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:24.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.180+0000 7fbd511d8700 1 --2- 172.21.15.28:0/1346343625 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbd380a2c20 0x7fbd380a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:24.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.180+0000 7fbd427fc700 1 -- 172.21.15.28:0/1346343625 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5829+0+0 (secure 0 0 0) 0x7fbd3c0ce120 con 0x7fbd4c1023b0 2024-04-25T01:28:24.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.181+0000 7fbd511d8700 1 --2- 172.21.15.28:0/1346343625 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbd380a2c20 0x7fbd380a50e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbd4800b580 tx=0x7fbd48005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:24.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.184+0000 7fbd427fc700 1 -- 172.21.15.28:0/1346343625 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbd3c09ba30 con 0x7fbd4c1023b0 2024-04-25T01:28:24.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:24.404+0000 7fbd5343c700 1 -- 172.21.15.28:0/1346343625 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fbd4c062670 con 0x7fbd380a2c20 2024-04-25T01:28:24.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:24 smithi130 ceph-mon[32702]: osdmap e64: 9 total, 9 up, 9 in 2024-04-25T01:28:24.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:24 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:28:24.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:24 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:28:24.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:24 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:24.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:24 smithi032 ceph-mon[32461]: osdmap e64: 9 total, 9 up, 9 in 2024-04-25T01:28:24.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:24 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:28:24.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:24 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:28:24.987 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:24 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:25.019 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:24 smithi028 ceph-mon[29390]: osdmap e64: 9 total, 9 up, 9 in 2024-04-25T01:28:25.019 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:24 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:28:25.020 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:24 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:28:25.020 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:24 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:25.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:25 smithi130 ceph-mon[32702]: pgmap v206: 1 pgs: 1 peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:25.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:25 smithi130 ceph-mon[32702]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:28:25.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:25 smithi130 ceph-mon[32702]: osdmap e65: 9 total, 9 up, 9 in 2024-04-25T01:28:25.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:25 smithi032 ceph-mon[32461]: pgmap v206: 1 pgs: 1 peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:25.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:25 smithi032 ceph-mon[32461]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:28:25.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:25 smithi032 ceph-mon[32461]: osdmap e65: 9 total, 9 up, 9 in 2024-04-25T01:28:26.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:25 smithi028 ceph-mon[29390]: pgmap v206: 1 pgs: 1 peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:26.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:25 smithi028 ceph-mon[29390]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:28:26.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:25 smithi028 ceph-mon[29390]: osdmap e65: 9 total, 9 up, 9 in 2024-04-25T01:28:27.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:27 smithi130 ceph-mon[32702]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-04-25T01:28:27.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:27 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1930079769' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18835e9d-8405-4586-89b2-75d3e565fad9"}]: dispatch 2024-04-25T01:28:27.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:27 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1930079769' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18835e9d-8405-4586-89b2-75d3e565fad9"}]': finished 2024-04-25T01:28:27.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:27 smithi130 ceph-mon[32702]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T01:28:27.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:27 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:28:27.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:27 smithi032 ceph-mon[32461]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-04-25T01:28:27.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:27 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1930079769' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18835e9d-8405-4586-89b2-75d3e565fad9"}]: dispatch 2024-04-25T01:28:27.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:27 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1930079769' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18835e9d-8405-4586-89b2-75d3e565fad9"}]': finished 2024-04-25T01:28:27.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:27 smithi032 ceph-mon[32461]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T01:28:27.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:27 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:28:28.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:27 smithi028 ceph-mon[29390]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-04-25T01:28:28.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:27 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1930079769' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18835e9d-8405-4586-89b2-75d3e565fad9"}]: dispatch 2024-04-25T01:28:28.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:27 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1930079769' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18835e9d-8405-4586-89b2-75d3e565fad9"}]': finished 2024-04-25T01:28:28.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:27 smithi028 ceph-mon[29390]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T01:28:28.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:27 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:28:28.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:28 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3112814846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:28:28.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:28 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3112814846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:28:29.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:28 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3112814846' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:28:29.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:29 smithi130 ceph-mon[32702]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-25T01:28:29.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:29 smithi032 ceph-mon[32461]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-25T01:28:30.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:29 smithi028 ceph-mon[29390]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-25T01:28:31.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:31 smithi130 ceph-mon[32702]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-25T01:28:31.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:31 smithi032 ceph-mon[32461]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-25T01:28:32.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:31 smithi028 ceph-mon[29390]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-25T01:28:33.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:33 smithi028 ceph-mon[29390]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:28:33.959 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:33 smithi130 ceph-mon[32702]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:28:33.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:33 smithi032 ceph-mon[32461]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T01:28:35.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:35 smithi130 ceph-mon[32702]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-25T01:28:35.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:35 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T01:28:35.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:35 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:35.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:35 smithi032 ceph-mon[32461]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-25T01:28:35.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:35 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T01:28:35.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:35 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:35.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:35 smithi028 ceph-mon[29390]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-25T01:28:35.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T01:28:35.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:36.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:36 smithi130 ceph-mon[32702]: Deploying daemon osd.9 on smithi028 2024-04-25T01:28:36.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:36 smithi032 ceph-mon[32461]: Deploying daemon osd.9 on smithi028 2024-04-25T01:28:37.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:36 smithi028 ceph-mon[29390]: Deploying daemon osd.9 on smithi028 2024-04-25T01:28:37.959 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:37 smithi130 ceph-mon[32702]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:37.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:37 smithi032 ceph-mon[32461]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:37.996 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:37 smithi028 ceph-mon[29390]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:39.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:39 smithi130 ceph-mon[32702]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:39.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:39 smithi032 ceph-mon[32461]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:40.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:39 smithi028 ceph-mon[29390]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:42.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:41 smithi032 ceph-mon[32461]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:42.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:41 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:42.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:41 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:42.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:41 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:28:42.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:41 smithi028 ceph-mon[29390]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:42.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:41 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:42.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:41 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:42.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:41 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:28:42.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:41 smithi130 ceph-mon[32702]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:42.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:41 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:42.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:41 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:42.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:41 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:28:44.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:43 smithi028 ceph-mon[29390]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:44.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:43 smithi032 ceph-mon[32461]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:44.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:43 smithi130 ceph-mon[32702]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:45.048 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 9 on host 'smithi028' 2024-04-25T01:28:45.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:45.069+0000 7fbd427fc700 1 -- 172.21.15.28:0/1346343625 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbd4c062670 con 0x7fbd380a2c20 2024-04-25T01:28:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:45.071+0000 7fbd5343c700 1 -- 172.21.15.28:0/1346343625 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbd380a2c20 msgr2=0x7fbd380a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:45.071+0000 7fbd5343c700 1 --2- 172.21.15.28:0/1346343625 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbd380a2c20 0x7fbd380a50e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbd4800b580 tx=0x7fbd48005fb0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:45.072+0000 7fbd5343c700 1 -- 172.21.15.28:0/1346343625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd4c1023b0 msgr2=0x7fbd4c19d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:45.072+0000 7fbd5343c700 1 --2- 172.21.15.28:0/1346343625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd4c1023b0 0x7fbd4c19d240 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c00d950 tx=0x7fbd3c00dc60 comp rx=0 tx=0).stop 2024-04-25T01:28:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:45.072+0000 7fbd5343c700 1 -- 172.21.15.28:0/1346343625 shutdown_connections 2024-04-25T01:28:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:45.072+0000 7fbd5343c700 1 --2- 172.21.15.28:0/1346343625 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbd380a2c20 0x7fbd380a50e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:45.072+0000 7fbd5343c700 1 --2- 172.21.15.28:0/1346343625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbd4c102d50 0x7fbd4c19f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:45.072+0000 7fbd5343c700 1 --2- 172.21.15.28:0/1346343625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbd4c1011b0 0x7fbd4c06b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:45.072+0000 7fbd5343c700 1 --2- 172.21.15.28:0/1346343625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd4c1023b0 0x7fbd4c19d240 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:45.072+0000 7fbd5343c700 1 -- 172.21.15.28:0/1346343625 >> 172.21.15.28:0/1346343625 conn(0x7fbd4c0fc750 msgr2=0x7fbd4c1056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:45.072+0000 7fbd5343c700 1 -- 172.21.15.28:0/1346343625 shutdown_connections 2024-04-25T01:28:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:28:45.072+0000 7fbd5343c700 1 -- 172.21.15.28:0/1346343625 wait complete. 2024-04-25T01:28:45.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:45 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:45.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:45 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:45.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:45 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:45.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:45 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:45.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:45 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:45.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:45 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:28:45.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:45 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:45.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:45 smithi130 ceph-mon[32702]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:45.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:45 smithi130 ceph-mon[32702]: from='osd.9 [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T01:28:45.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:45 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:45.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:45 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:45.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:45 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:28:45.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:45 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:45.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:45 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:45.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:45 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:45.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:45 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:28:45.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:45 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:45.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:45 smithi032 ceph-mon[32461]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:45.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:45 smithi032 ceph-mon[32461]: from='osd.9 [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T01:28:45.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:45 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:45.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:45 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:45.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:45 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:28:45.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:45 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:45.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:45 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:28:45.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:45 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:45.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:45 smithi028 ceph-mon[29390]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:45.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:45 smithi028 ceph-mon[29390]: from='osd.9 [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T01:28:45.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:45 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:45.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:45 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:45.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:45 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:28:45.757 DEBUG:teuthology.orchestra.run.smithi028:osd.9> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.9.service 2024-04-25T01:28:45.759 INFO:tasks.cephadm:Deploying osd.10 on smithi032 with /dev/vg_nvme/lv_1... 2024-04-25T01:28:45.759 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T01:28:46.017 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:28:46.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:46 smithi028 ceph-mon[29390]: from='osd.9 [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T01:28:46.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:46 smithi028 ceph-mon[29390]: osdmap e67: 10 total, 9 up, 10 in 2024-04-25T01:28:46.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:46 smithi028 ceph-mon[29390]: from='osd.9 [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-25T01:28:46.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:46 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:28:46.304 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:28:46 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:28:46.200+0000 7fcca0181700 -1 osd.9 0 waiting for initial osdmap 2024-04-25T01:28:46.304 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:28:46 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:28:46.214+0000 7fcc9bf74700 -1 osd.9 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:28:46.424 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:46 smithi032 ceph-mon[32461]: from='osd.9 [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T01:28:46.424 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:46 smithi032 ceph-mon[32461]: osdmap e67: 10 total, 9 up, 10 in 2024-04-25T01:28:46.424 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:46 smithi032 ceph-mon[32461]: from='osd.9 [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-25T01:28:46.424 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:46 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:28:46.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:46 smithi130 ceph-mon[32702]: from='osd.9 [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T01:28:46.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:46 smithi130 ceph-mon[32702]: osdmap e67: 10 total, 9 up, 10 in 2024-04-25T01:28:46.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:46 smithi130 ceph-mon[32702]: from='osd.9 [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-25T01:28:46.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:46 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:28:47.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:47 smithi028 ceph-mon[29390]: from='osd.9 [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-25T01:28:47.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:47 smithi028 ceph-mon[29390]: osdmap e68: 10 total, 9 up, 10 in 2024-04-25T01:28:47.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:47 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:28:47.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:47 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:28:47.223 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:47 smithi028 ceph-mon[29390]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:47.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:47 smithi130 ceph-mon[32702]: from='osd.9 [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-25T01:28:47.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:47 smithi130 ceph-mon[32702]: osdmap e68: 10 total, 9 up, 10 in 2024-04-25T01:28:47.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:47 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:28:47.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:47 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:28:47.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:47 smithi130 ceph-mon[32702]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:47.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:47 smithi032 ceph-mon[32461]: from='osd.9 [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-25T01:28:47.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:47 smithi032 ceph-mon[32461]: osdmap e68: 10 total, 9 up, 10 in 2024-04-25T01:28:47.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:47 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:28:47.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:47 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:28:47.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:47 smithi032 ceph-mon[32461]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:28:48.354 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T01:28:48.354 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T01:28:48.354 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-04-25T01:28:48.354 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-04-25T01:28:48.354 INFO:teuthology.orchestra.run.smithi032.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00819472 s, 1.3 GB/s 2024-04-25T01:28:48.355 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-04-25T01:28:48.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:48 smithi130 ceph-mon[32702]: purged_snaps scrub starts 2024-04-25T01:28:48.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:48 smithi130 ceph-mon[32702]: purged_snaps scrub ok 2024-04-25T01:28:48.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:48 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:28:48.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:48 smithi130 ceph-mon[32702]: osd.9 [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206] boot 2024-04-25T01:28:48.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:48 smithi130 ceph-mon[32702]: osdmap e69: 10 total, 10 up, 10 in 2024-04-25T01:28:48.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:48 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:28:48.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:48 smithi032 ceph-mon[32461]: purged_snaps scrub starts 2024-04-25T01:28:48.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:48 smithi032 ceph-mon[32461]: purged_snaps scrub ok 2024-04-25T01:28:48.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:48 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:28:48.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:48 smithi032 ceph-mon[32461]: osd.9 [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206] boot 2024-04-25T01:28:48.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:48 smithi032 ceph-mon[32461]: osdmap e69: 10 total, 10 up, 10 in 2024-04-25T01:28:48.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:48 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:28:48.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:48 smithi028 ceph-mon[29390]: purged_snaps scrub starts 2024-04-25T01:28:48.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:48 smithi028 ceph-mon[29390]: purged_snaps scrub ok 2024-04-25T01:28:48.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:48 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:28:48.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:48 smithi028 ceph-mon[29390]: osd.9 [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206] boot 2024-04-25T01:28:48.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:48 smithi028 ceph-mon[29390]: osdmap e69: 10 total, 10 up, 10 in 2024-04-25T01:28:48.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:48 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:28:49.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_1 2024-04-25T01:28:49.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:49 smithi130 ceph-mon[32702]: osdmap e70: 10 total, 10 up, 10 in 2024-04-25T01:28:49.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:49 smithi130 ceph-mon[32702]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:28:49.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:49 smithi130 ceph-mon[32702]: Detected new or changed devices on smithi028 2024-04-25T01:28:49.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:49 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:49.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:49 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:49.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:49 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:49.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:49 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:49.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:49 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:49.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:49 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:49.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:49 smithi130 ceph-mon[32702]: Adjusting osd_memory_target on smithi028 to 4273M 2024-04-25T01:28:49.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:49 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:49.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:49 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:49.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:49 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:28:49.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:49 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:49.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:49 smithi032 ceph-mon[32461]: osdmap e70: 10 total, 10 up, 10 in 2024-04-25T01:28:49.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:49 smithi032 ceph-mon[32461]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:28:49.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:49 smithi032 ceph-mon[32461]: Detected new or changed devices on smithi028 2024-04-25T01:28:49.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:49 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:49.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:49 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:49.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:49 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:49.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:49 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:49.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:49 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:49.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:49 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:49.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:49 smithi032 ceph-mon[32461]: Adjusting osd_memory_target on smithi028 to 4273M 2024-04-25T01:28:49.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:49 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:49.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:49 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:49.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:49 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:28:49.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:49 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:28:49.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:49 smithi028 ceph-mon[29390]: osdmap e70: 10 total, 10 up, 10 in 2024-04-25T01:28:49.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:49 smithi028 ceph-mon[29390]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:28:49.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:49 smithi028 ceph-mon[29390]: Detected new or changed devices on smithi028 2024-04-25T01:28:49.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:49 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:49.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:49 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:49.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:49 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:49.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:49 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:49.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:49 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:49.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:49 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:28:49.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:49 smithi028 ceph-mon[29390]: Adjusting osd_memory_target on smithi028 to 4273M 2024-04-25T01:28:49.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:49 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:49.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:49 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:49.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:49 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:28:49.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:49 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:28:51.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.104+0000 7f63c3099700 1 -- 172.21.15.32:0/872953560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63bc103490 msgr2=0x7f63bc1038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:51.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.104+0000 7f63c3099700 1 --2- 172.21.15.32:0/872953560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63bc103490 0x7f63bc1038f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f63a80099c0 tx=0x7f63a8009cd0 comp rx=0 tx=0).stop 2024-04-25T01:28:51.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.104+0000 7f63c3099700 1 -- 172.21.15.32:0/872953560 shutdown_connections 2024-04-25T01:28:51.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.104+0000 7f63c3099700 1 --2- 172.21.15.32:0/872953560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63bc103e30 0x7f63bc10a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:51.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.104+0000 7f63c3099700 1 --2- 172.21.15.32:0/872953560 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63bc102290 0x7f63bc1026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:51.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.104+0000 7f63c3099700 1 --2- 172.21.15.32:0/872953560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63bc103490 0x7f63bc1038f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:51.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.104+0000 7f63c3099700 1 -- 172.21.15.32:0/872953560 >> 172.21.15.32:0/872953560 conn(0x7f63bc0fd830 msgr2=0x7f63bc0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:28:51.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.104+0000 7f63c3099700 1 -- 172.21.15.32:0/872953560 shutdown_connections 2024-04-25T01:28:51.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.104+0000 7f63c3099700 1 -- 172.21.15.32:0/872953560 wait complete. 2024-04-25T01:28:51.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.105+0000 7f63c3099700 1 Processor -- start 2024-04-25T01:28:51.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.105+0000 7f63c3099700 1 -- start start 2024-04-25T01:28:51.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.105+0000 7f63c3099700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63bc102290 0x7f63bc075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:51.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.105+0000 7f63c3099700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63bc103490 0x7f63bc074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:51.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.105+0000 7f63c3099700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63bc103e30 0x7f63bc074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:51.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.105+0000 7f63c3099700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f63bc10de50 con 0x7f63bc103e30 2024-04-25T01:28:51.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.105+0000 7f63c3099700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f63bc10dcd0 con 0x7f63bc102290 2024-04-25T01:28:51.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.105+0000 7f63c3099700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f63bc10dfd0 con 0x7f63bc103490 2024-04-25T01:28:51.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.105+0000 7f63c0e35700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63bc102290 0x7f63bc075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:51.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.105+0000 7f63c0e35700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63bc102290 0x7f63bc075fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58994/0 (socket says 172.21.15.32:58994) 2024-04-25T01:28:51.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.105+0000 7f63c0e35700 1 -- 172.21.15.32:0/3598890573 learned_addr learned my addr 172.21.15.32:0/3598890573 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-25T01:28:51.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.105+0000 7f63bbfff700 1 --2- 172.21.15.32:0/3598890573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63bc103490 0x7f63bc074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:51.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.105+0000 7f63c1636700 1 --2- 172.21.15.32:0/3598890573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63bc103e30 0x7f63bc074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:51.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.105+0000 7f63c0e35700 1 -- 172.21.15.32:0/3598890573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63bc103490 msgr2=0x7f63bc074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:51.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.105+0000 7f63c0e35700 1 --2- 172.21.15.32:0/3598890573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63bc103490 0x7f63bc074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:51.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.105+0000 7f63c0e35700 1 -- 172.21.15.32:0/3598890573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63bc103e30 msgr2=0x7f63bc074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:28:51.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.105+0000 7f63c0e35700 1 --2- 172.21.15.32:0/3598890573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63bc103e30 0x7f63bc074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:28:51.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.105+0000 7f63c0e35700 1 -- 172.21.15.32:0/3598890573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63bc075410 con 0x7f63bc102290 2024-04-25T01:28:51.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.105+0000 7f63c0e35700 1 --2- 172.21.15.32:0/3598890573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63bc102290 0x7f63bc075fd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f63b000d980 tx=0x7f63b000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:51.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.106+0000 7f63b9ffb700 1 -- 172.21.15.32:0/3598890573 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b0016070 con 0x7f63bc102290 2024-04-25T01:28:51.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.106+0000 7f63b9ffb700 1 -- 172.21.15.32:0/3598890573 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63b000fd70 con 0x7f63bc102290 2024-04-25T01:28:51.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.106+0000 7f63c3099700 1 -- 172.21.15.32:0/3598890573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63bc1a2650 con 0x7f63bc102290 2024-04-25T01:28:51.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.106+0000 7f63b9ffb700 1 -- 172.21.15.32:0/3598890573 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b0019360 con 0x7f63bc102290 2024-04-25T01:28:51.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.106+0000 7f63c3099700 1 -- 172.21.15.32:0/3598890573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63bc1a2b60 con 0x7f63bc102290 2024-04-25T01:28:51.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.107+0000 7f63c3099700 1 -- 172.21.15.32:0/3598890573 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63bc068f20 con 0x7f63bc102290 2024-04-25T01:28:51.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.108+0000 7f63b9ffb700 1 -- 172.21.15.32:0/3598890573 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f63b0019070 con 0x7f63bc102290 2024-04-25T01:28:51.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.108+0000 7f63b9ffb700 1 --2- 172.21.15.32:0/3598890573 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f63ac0a2c20 0x7f63ac0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:28:51.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.109+0000 7f63b9ffb700 1 -- 172.21.15.32:0/3598890573 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6261+0+0 (secure 0 0 0) 0x7f63b00cdfc0 con 0x7f63bc102290 2024-04-25T01:28:51.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.109+0000 7f63bbfff700 1 --2- 172.21.15.32:0/3598890573 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f63ac0a2c20 0x7f63ac0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:28:51.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.109+0000 7f63bbfff700 1 --2- 172.21.15.32:0/3598890573 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f63ac0a2c20 0x7f63ac0a50e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f63a8000c00 tx=0x7f63a8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:28:51.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.113+0000 7f63b9ffb700 1 -- 172.21.15.32:0/3598890573 <== mon.2 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f63b009b680 con 0x7f63bc102290 2024-04-25T01:28:51.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:28:51.334+0000 7f63c3099700 1 -- 172.21.15.32:0/3598890573 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f63bc062670 con 0x7f63ac0a2c20 2024-04-25T01:28:51.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:51 smithi130 ceph-mon[32702]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:28:51.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:51 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:28:51.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:51 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:28:51.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:51 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:51.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:51 smithi032 ceph-mon[32461]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:28:51.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:51 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:28:51.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:51 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:28:51.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:51 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:51.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:51 smithi028 ceph-mon[29390]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:28:51.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:51 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:28:51.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:51 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:28:51.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:51 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:28:52.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:52 smithi130 ceph-mon[32702]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:28:52.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:52 smithi032 ceph-mon[32461]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:28:52.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:52 smithi028 ceph-mon[29390]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:28:53.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:53 smithi130 ceph-mon[32702]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:28:53.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:53 smithi032 ceph-mon[32461]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:28:53.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:53 smithi028 ceph-mon[29390]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:28:54.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:54 smithi130 ceph-mon[32702]: from='client.? 172.21.15.32:0/1246723296' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22b3f803-fccb-43d3-941e-b7cfd9b11461"}]: dispatch 2024-04-25T01:28:54.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:54 smithi130 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22b3f803-fccb-43d3-941e-b7cfd9b11461"}]: dispatch 2024-04-25T01:28:54.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:54 smithi130 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22b3f803-fccb-43d3-941e-b7cfd9b11461"}]': finished 2024-04-25T01:28:54.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:54 smithi130 ceph-mon[32702]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T01:28:54.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:54 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:28:54.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:54 smithi032 ceph-mon[32461]: from='client.? 172.21.15.32:0/1246723296' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22b3f803-fccb-43d3-941e-b7cfd9b11461"}]: dispatch 2024-04-25T01:28:54.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:54 smithi032 ceph-mon[32461]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22b3f803-fccb-43d3-941e-b7cfd9b11461"}]: dispatch 2024-04-25T01:28:54.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:54 smithi032 ceph-mon[32461]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22b3f803-fccb-43d3-941e-b7cfd9b11461"}]': finished 2024-04-25T01:28:54.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:54 smithi032 ceph-mon[32461]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T01:28:54.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:54 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:28:54.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:54 smithi028 ceph-mon[29390]: from='client.? 172.21.15.32:0/1246723296' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22b3f803-fccb-43d3-941e-b7cfd9b11461"}]: dispatch 2024-04-25T01:28:54.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:54 smithi028 ceph-mon[29390]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22b3f803-fccb-43d3-941e-b7cfd9b11461"}]: dispatch 2024-04-25T01:28:54.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:54 smithi028 ceph-mon[29390]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22b3f803-fccb-43d3-941e-b7cfd9b11461"}]': finished 2024-04-25T01:28:54.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:54 smithi028 ceph-mon[29390]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T01:28:54.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:54 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:28:55.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:55 smithi130 ceph-mon[32702]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:28:55.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:55 smithi130 ceph-mon[32702]: from='client.? 172.21.15.32:0/1416363217' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:28:55.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:55 smithi032 ceph-mon[32461]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:28:55.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:55 smithi032 ceph-mon[32461]: from='client.? 172.21.15.32:0/1416363217' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:28:55.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:55 smithi028 ceph-mon[29390]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:28:55.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:55 smithi028 ceph-mon[29390]: from='client.? 172.21.15.32:0/1416363217' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:28:57.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:57 smithi130 ceph-mon[32702]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:28:57.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:57 smithi032 ceph-mon[32461]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:28:57.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:57 smithi028 ceph-mon[29390]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:28:59.446 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:28:59 smithi032 ceph-mon[32461]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:28:59.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:28:59 smithi130 ceph-mon[32702]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:28:59.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:28:59 smithi028 ceph-mon[29390]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:01.558 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:01 smithi032 ceph-mon[32461]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:01.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:01 smithi130 ceph-mon[32702]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:01.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:01 smithi028 ceph-mon[29390]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:02.587 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:02 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T01:29:02.587 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:02 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:02.587 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:02 smithi032 ceph-mon[32461]: Deploying daemon osd.10 on smithi032 2024-04-25T01:29:02.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:02 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T01:29:02.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:02 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:02.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:02 smithi130 ceph-mon[32702]: Deploying daemon osd.10 on smithi032 2024-04-25T01:29:02.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:02 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T01:29:02.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:02 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:02.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:02 smithi028 ceph-mon[29390]: Deploying daemon osd.10 on smithi032 2024-04-25T01:29:03.617 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:03 smithi032 ceph-mon[32461]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:03.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:03 smithi130 ceph-mon[32702]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:03.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:03 smithi028 ceph-mon[29390]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:05.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:05 smithi032 ceph-mon[32461]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:05.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:05 smithi130 ceph-mon[32702]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:05.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:05 smithi028 ceph-mon[29390]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:07.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:07 smithi130 ceph-mon[32702]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:07.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:07 smithi032 ceph-mon[32461]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:07.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:07 smithi028 ceph-mon[29390]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:08.959 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:08 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:08.959 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:08 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:08.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:08 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:08.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:08 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:08.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:08 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:08.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:08 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:09.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:08 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:09.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:08 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:09.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:08 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:09.959 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:09 smithi130 ceph-mon[32702]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:09.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:09 smithi032 ceph-mon[32461]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:10.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:09 smithi028 ceph-mon[29390]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:11.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:29:11.154+0000 7f63b9ffb700 1 -- 172.21.15.32:0/3598890573 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f63bc062670 con 0x7f63ac0a2c20 2024-04-25T01:29:11.133 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 10 on host 'smithi032' 2024-04-25T01:29:11.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:29:11.156+0000 7f63c3099700 1 -- 172.21.15.32:0/3598890573 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f63ac0a2c20 msgr2=0x7f63ac0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:11.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:29:11.156+0000 7f63c3099700 1 --2- 172.21.15.32:0/3598890573 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f63ac0a2c20 0x7f63ac0a50e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f63a8000c00 tx=0x7f63a8005fb0 comp rx=0 tx=0).stop 2024-04-25T01:29:11.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:29:11.156+0000 7f63c3099700 1 -- 172.21.15.32:0/3598890573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63bc102290 msgr2=0x7f63bc075fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:11.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:29:11.156+0000 7f63c3099700 1 --2- 172.21.15.32:0/3598890573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63bc102290 0x7f63bc075fd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f63b000d980 tx=0x7f63b000dc90 comp rx=0 tx=0).stop 2024-04-25T01:29:11.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:29:11.156+0000 7f63c3099700 1 -- 172.21.15.32:0/3598890573 shutdown_connections 2024-04-25T01:29:11.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:29:11.156+0000 7f63c3099700 1 --2- 172.21.15.32:0/3598890573 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f63ac0a2c20 0x7f63ac0a50e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:11.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:29:11.156+0000 7f63c3099700 1 --2- 172.21.15.32:0/3598890573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63bc103e30 0x7f63bc074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:11.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:29:11.156+0000 7f63c3099700 1 --2- 172.21.15.32:0/3598890573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63bc102290 0x7f63bc075fd0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:11.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:29:11.156+0000 7f63c3099700 1 --2- 172.21.15.32:0/3598890573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63bc103490 0x7f63bc074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:11.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:29:11.156+0000 7f63c3099700 1 -- 172.21.15.32:0/3598890573 >> 172.21.15.32:0/3598890573 conn(0x7f63bc0fd830 msgr2=0x7f63bc1067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:11.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:29:11.156+0000 7f63c3099700 1 -- 172.21.15.32:0/3598890573 shutdown_connections 2024-04-25T01:29:11.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-25T01:29:11.156+0000 7f63c3099700 1 -- 172.21.15.32:0/3598890573 wait complete. 2024-04-25T01:29:11.837 DEBUG:teuthology.orchestra.run.smithi032:osd.10> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.10.service 2024-04-25T01:29:11.839 INFO:tasks.cephadm:Deploying osd.11 on smithi130 with /dev/vg_nvme/lv_1... 2024-04-25T01:29:11.840 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T01:29:12.089 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:29:12.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:12 smithi130 ceph-mon[32702]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:12.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:12 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:12.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:12 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:12.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:12 smithi130 ceph-mon[32702]: from='osd.10 [v2:172.21.15.32:6824/13541811,v1:172.21.15.32:6825/13541811]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:29:12.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:12 smithi130 ceph-mon[32702]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:29:12.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:12 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:12.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:12 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:12.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:12 smithi032 ceph-mon[32461]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:12.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:12 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:12.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:12 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:12.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:12 smithi032 ceph-mon[32461]: from='osd.10 [v2:172.21.15.32:6824/13541811,v1:172.21.15.32:6825/13541811]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:29:12.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:12 smithi032 ceph-mon[32461]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:29:12.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:12 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:12.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:12 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:12.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:12 smithi028 ceph-mon[29390]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:12.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:12 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:12.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:12 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:12.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:12 smithi028 ceph-mon[29390]: from='osd.10 [v2:172.21.15.32:6824/13541811,v1:172.21.15.32:6825/13541811]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:29:12.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:12 smithi028 ceph-mon[29390]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:29:12.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:12 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:12.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:12 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:13.208 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:13 smithi032 ceph-mon[32461]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T01:29:13.208 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:13 smithi032 ceph-mon[32461]: osdmap e72: 11 total, 10 up, 11 in 2024-04-25T01:29:13.208 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:29:13.208 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:13 smithi032 ceph-mon[32461]: from='osd.10 [v2:172.21.15.32:6824/13541811,v1:172.21.15.32:6825/13541811]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:29:13.209 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:13 smithi032 ceph-mon[32461]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:29:13.209 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:13 smithi032 ceph-mon[32461]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:13.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:13 smithi130 ceph-mon[32702]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T01:29:13.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:13 smithi130 ceph-mon[32702]: osdmap e72: 11 total, 10 up, 11 in 2024-04-25T01:29:13.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:29:13.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:13 smithi130 ceph-mon[32702]: from='osd.10 [v2:172.21.15.32:6824/13541811,v1:172.21.15.32:6825/13541811]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:29:13.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:13 smithi130 ceph-mon[32702]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:29:13.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:13 smithi130 ceph-mon[32702]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:13.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:13 smithi028 ceph-mon[29390]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T01:29:13.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:13 smithi028 ceph-mon[29390]: osdmap e72: 11 total, 10 up, 11 in 2024-04-25T01:29:13.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:29:13.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:13 smithi028 ceph-mon[29390]: from='osd.10 [v2:172.21.15.32:6824/13541811,v1:172.21.15.32:6825/13541811]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:29:13.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:13 smithi028 ceph-mon[29390]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-25T01:29:13.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:13 smithi028 ceph-mon[29390]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:29:13.945 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:29:13 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:29:13.681+0000 7fa981ae1700 -1 osd.10 0 waiting for initial osdmap 2024-04-25T01:29:13.945 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:29:13 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:29:13.694+0000 7fa9798cc700 -1 osd.10 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:29:14.207 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:14 smithi032 ceph-mon[32461]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-25T01:29:14.207 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:14 smithi032 ceph-mon[32461]: osdmap e73: 11 total, 10 up, 11 in 2024-04-25T01:29:14.207 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:14 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:29:14.207 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:14 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:29:14.207 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:14 smithi032 ceph-mon[32461]: from='osd.10 ' entity='osd.10' 2024-04-25T01:29:14.207 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:14 smithi032 ceph-mon[32461]: osd.10 [v2:172.21.15.32:6824/13541811,v1:172.21.15.32:6825/13541811] boot 2024-04-25T01:29:14.207 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:14 smithi032 ceph-mon[32461]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T01:29:14.208 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:14 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:29:14.290 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T01:29:14.290 INFO:teuthology.orchestra.run.smithi130.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T01:29:14.291 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10+0 records in 2024-04-25T01:29:14.291 INFO:teuthology.orchestra.run.smithi130.stderr:10+0 records out 2024-04-25T01:29:14.291 INFO:teuthology.orchestra.run.smithi130.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128076 s, 819 MB/s 2024-04-25T01:29:14.291 INFO:teuthology.orchestra.run.smithi130.stderr:--> Zapping successful for: 2024-04-25T01:29:14.401 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:14 smithi130 ceph-mon[32702]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-25T01:29:14.401 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:14 smithi130 ceph-mon[32702]: osdmap e73: 11 total, 10 up, 11 in 2024-04-25T01:29:14.401 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:14 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:29:14.402 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:14 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:29:14.402 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:14 smithi130 ceph-mon[32702]: from='osd.10 ' entity='osd.10' 2024-04-25T01:29:14.402 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:14 smithi130 ceph-mon[32702]: osd.10 [v2:172.21.15.32:6824/13541811,v1:172.21.15.32:6825/13541811] boot 2024-04-25T01:29:14.402 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:14 smithi130 ceph-mon[32702]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T01:29:14.402 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:14 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:29:14.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:14 smithi028 ceph-mon[29390]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-25T01:29:14.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:14 smithi028 ceph-mon[29390]: osdmap e73: 11 total, 10 up, 11 in 2024-04-25T01:29:14.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:14 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:29:14.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:14 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:29:14.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:14 smithi028 ceph-mon[29390]: from='osd.10 ' entity='osd.10' 2024-04-25T01:29:14.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:14 smithi028 ceph-mon[29390]: osd.10 [v2:172.21.15.32:6824/13541811,v1:172.21.15.32:6825/13541811] boot 2024-04-25T01:29:14.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:14 smithi028 ceph-mon[29390]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T01:29:14.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:14 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:29:14.994 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi130:vg_nvme/lv_1 2024-04-25T01:29:15.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:15 smithi032 ceph-mon[32461]: purged_snaps scrub starts 2024-04-25T01:29:15.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:15 smithi032 ceph-mon[32461]: purged_snaps scrub ok 2024-04-25T01:29:15.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:15 smithi032 ceph-mon[32461]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 727 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:15.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:15 smithi032 ceph-mon[32461]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T01:29:15.249 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:29:15.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:15 smithi130 ceph-mon[32702]: purged_snaps scrub starts 2024-04-25T01:29:15.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:15 smithi130 ceph-mon[32702]: purged_snaps scrub ok 2024-04-25T01:29:15.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:15 smithi130 ceph-mon[32702]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 727 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:15.371 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:15 smithi130 ceph-mon[32702]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T01:29:15.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:15 smithi028 ceph-mon[29390]: purged_snaps scrub starts 2024-04-25T01:29:15.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:15 smithi028 ceph-mon[29390]: purged_snaps scrub ok 2024-04-25T01:29:15.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:15 smithi028 ceph-mon[29390]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 727 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:15.464 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:15 smithi028 ceph-mon[29390]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T01:29:16.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:16 smithi130 ceph-mon[32702]: Detected new or changed devices on smithi032 2024-04-25T01:29:16.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:16.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:16.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:16.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:16.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:16.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:16.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:16 smithi130 ceph-mon[32702]: Adjusting osd_memory_target on smithi032 to 4273M 2024-04-25T01:29:16.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:16.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:16.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:16.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:16.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:16.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:16 smithi130 ceph-mon[32702]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T01:29:16.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:16.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:16.711 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:16.943 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.964+0000 7f17bd2c8700 1 -- 172.21.15.130:0/2542933095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17b8102d20 msgr2=0x7f17b81095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:16.943 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.964+0000 7f17bd2c8700 1 --2- 172.21.15.130:0/2542933095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17b8102d20 0x7f17b81095d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f17a8009990 tx=0x7f17a8009ca0 comp rx=0 tx=0).stop 2024-04-25T01:29:16.943 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.964+0000 7f17bd2c8700 1 -- 172.21.15.130:0/2542933095 shutdown_connections 2024-04-25T01:29:16.943 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.964+0000 7f17bd2c8700 1 --2- 172.21.15.130:0/2542933095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17b8102d20 0x7f17b81095d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:16.943 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.964+0000 7f17bd2c8700 1 --2- 172.21.15.130:0/2542933095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17b8101180 0x7f17b81015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:16.943 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.964+0000 7f17bd2c8700 1 --2- 172.21.15.130:0/2542933095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b8102380 0x7f17b81027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:16.943 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.964+0000 7f17bd2c8700 1 -- 172.21.15.130:0/2542933095 >> 172.21.15.130:0/2542933095 conn(0x7f17b80fc720 msgr2=0x7f17b80feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:16.943 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.964+0000 7f17bd2c8700 1 -- 172.21.15.130:0/2542933095 shutdown_connections 2024-04-25T01:29:16.943 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.964+0000 7f17bd2c8700 1 -- 172.21.15.130:0/2542933095 wait complete. 2024-04-25T01:29:16.943 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.964+0000 7f17bd2c8700 1 Processor -- start 2024-04-25T01:29:16.944 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.965+0000 7f17bd2c8700 1 -- start start 2024-04-25T01:29:16.946 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.965+0000 7f17bd2c8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17b8101180 0x7f17b806b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:16.946 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.965+0000 7f17bd2c8700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17b8102380 0x7f17b819d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:16.946 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.965+0000 7f17bd2c8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b8102d20 0x7f17b819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:16.946 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.965+0000 7f17bd2c8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17b810cdd0 con 0x7f17b8101180 2024-04-25T01:29:16.946 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.965+0000 7f17bd2c8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f17b810cc50 con 0x7f17b8102d20 2024-04-25T01:29:16.946 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.965+0000 7f17bd2c8700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f17b810cf50 con 0x7f17b8102380 2024-04-25T01:29:16.946 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.965+0000 7f17b67fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17b8102380 0x7f17b819d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:16.946 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.965+0000 7f17b67fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17b8102380 0x7f17b819d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:48378/0 (socket says 172.21.15.130:48378) 2024-04-25T01:29:16.946 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.965+0000 7f17b67fc700 1 -- 172.21.15.130:0/2172898732 learned_addr learned my addr 172.21.15.130:0/2172898732 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-25T01:29:16.946 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.965+0000 7f17b6ffd700 1 --2- 172.21.15.130:0/2172898732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17b8101180 0x7f17b806b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:16.947 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.965+0000 7f17b77fe700 1 --2- 172.21.15.130:0/2172898732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b8102d20 0x7f17b819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:16.947 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.965+0000 7f17b67fc700 1 -- 172.21.15.130:0/2172898732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b8102d20 msgr2=0x7f17b819f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:16.947 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.965+0000 7f17b67fc700 1 --2- 172.21.15.130:0/2172898732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b8102d20 0x7f17b819f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:16.947 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.965+0000 7f17b67fc700 1 -- 172.21.15.130:0/2172898732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17b8101180 msgr2=0x7f17b806b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:16.947 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.965+0000 7f17b67fc700 1 --2- 172.21.15.130:0/2172898732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17b8101180 0x7f17b806b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:16.947 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.965+0000 7f17b67fc700 1 -- 172.21.15.130:0/2172898732 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b81a00b0 con 0x7f17b8102380 2024-04-25T01:29:16.947 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.965+0000 7f17b6ffd700 1 --2- 172.21.15.130:0/2172898732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17b8101180 0x7f17b806b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:16.947 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.965+0000 7f17b67fc700 1 --2- 172.21.15.130:0/2172898732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17b8102380 0x7f17b819d190 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f17ac00ba50 tx=0x7f17ac00be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:16.947 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.966+0000 7f179ffff700 1 -- 172.21.15.130:0/2172898732 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ac019070 con 0x7f17b8102380 2024-04-25T01:29:16.947 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.966+0000 7f17bd2c8700 1 -- 172.21.15.130:0/2172898732 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17b81a03a0 con 0x7f17b8102380 2024-04-25T01:29:16.947 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.966+0000 7f179ffff700 1 -- 172.21.15.130:0/2172898732 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17ac014440 con 0x7f17b8102380 2024-04-25T01:29:16.947 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.966+0000 7f179ffff700 1 -- 172.21.15.130:0/2172898732 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ac0075d0 con 0x7f17b8102380 2024-04-25T01:29:16.947 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.966+0000 7f17bd2c8700 1 -- 172.21.15.130:0/2172898732 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17b81a0960 con 0x7f17b8102380 2024-04-25T01:29:16.947 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.967+0000 7f17bd2c8700 1 -- 172.21.15.130:0/2172898732 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17b8068f20 con 0x7f17b8102380 2024-04-25T01:29:16.948 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.968+0000 7f179ffff700 1 -- 172.21.15.130:0/2172898732 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f17ac01c070 con 0x7f17b8102380 2024-04-25T01:29:16.949 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.968+0000 7f179ffff700 1 --2- 172.21.15.130:0/2172898732 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f17a00a2c20 0x7f17a00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:16.949 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.969+0000 7f179ffff700 1 -- 172.21.15.130:0/2172898732 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 6694+0+0 (secure 0 0 0) 0x7f17ac0cdea0 con 0x7f17b8102380 2024-04-25T01:29:16.949 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.969+0000 7f17b6ffd700 1 --2- 172.21.15.130:0/2172898732 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f17a00a2c20 0x7f17a00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:16.949 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.969+0000 7f17b6ffd700 1 --2- 172.21.15.130:0/2172898732 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f17a00a2c20 0x7f17a00a50e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f17b81021e0 tx=0x7f17b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:16.951 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:16.973+0000 7f179ffff700 1 -- 172.21.15.130:0/2172898732 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f17ac09b450 con 0x7f17b8102380 2024-04-25T01:29:16.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:16 smithi032 ceph-mon[32461]: Detected new or changed devices on smithi032 2024-04-25T01:29:16.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:16.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:16.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:16.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:16.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:16.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:16.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:16 smithi032 ceph-mon[32461]: Adjusting osd_memory_target on smithi032 to 4273M 2024-04-25T01:29:16.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:16.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:16.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:16.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:16.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:16.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:16 smithi032 ceph-mon[32461]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T01:29:16.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:16.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:16.986 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:17.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:16 smithi028 ceph-mon[29390]: Detected new or changed devices on smithi032 2024-04-25T01:29:17.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:17.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:17.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:17.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:17.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:17.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:17.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:16 smithi028 ceph-mon[29390]: Adjusting osd_memory_target on smithi032 to 4273M 2024-04-25T01:29:17.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:17.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:17.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:17.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:17.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:17.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:16 smithi028 ceph-mon[29390]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T01:29:17.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:17.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:17.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:17.158 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:17.179+0000 7f17bd2c8700 1 -- 172.21.15.130:0/2172898732 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f17b8062670 con 0x7f17a00a2c20 2024-04-25T01:29:17.959 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:17 smithi130 ceph-mon[32702]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 728 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:17.959 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:17 smithi130 ceph-mon[32702]: osdmap e77: 11 total, 11 up, 11 in 2024-04-25T01:29:17.959 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:17 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:29:17.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:17 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:29:17.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:17 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:17.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:17 smithi032 ceph-mon[32461]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 728 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:17.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:17 smithi032 ceph-mon[32461]: osdmap e77: 11 total, 11 up, 11 in 2024-04-25T01:29:17.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:17 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:29:17.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:17 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:29:17.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:17 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:18.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:17 smithi028 ceph-mon[29390]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 728 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:18.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:17 smithi028 ceph-mon[29390]: osdmap e77: 11 total, 11 up, 11 in 2024-04-25T01:29:18.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:17 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:29:18.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:17 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:29:18.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:17 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:18.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:18 smithi130 ceph-mon[32702]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:29:18.834 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:18 smithi130 ceph-mon[32702]: osdmap e78: 11 total, 11 up, 11 in 2024-04-25T01:29:18.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:18 smithi032 ceph-mon[32461]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:29:18.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:18 smithi032 ceph-mon[32461]: osdmap e78: 11 total, 11 up, 11 in 2024-04-25T01:29:19.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:18 smithi028 ceph-mon[29390]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi130:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:29:19.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:18 smithi028 ceph-mon[29390]: osdmap e78: 11 total, 11 up, 11 in 2024-04-25T01:29:19.959 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:19 smithi130 ceph-mon[32702]: pgmap v247: 1 pgs: 1 remapped+peering; 577 KiB data, 732 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:19.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:19 smithi032 ceph-mon[32461]: pgmap v247: 1 pgs: 1 remapped+peering; 577 KiB data, 732 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:20.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:19 smithi028 ceph-mon[29390]: pgmap v247: 1 pgs: 1 remapped+peering; 577 KiB data, 732 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:20.959 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:20 smithi130 ceph-mon[32702]: from='client.? 172.21.15.130:0/1731045037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c129f89-41b2-4ba9-8e22-0863133613c2"}]: dispatch 2024-04-25T01:29:20.959 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:20 smithi130 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c129f89-41b2-4ba9-8e22-0863133613c2"}]: dispatch 2024-04-25T01:29:20.959 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:20 smithi130 ceph-mon[32702]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c129f89-41b2-4ba9-8e22-0863133613c2"}]': finished 2024-04-25T01:29:20.959 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:20 smithi130 ceph-mon[32702]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T01:29:20.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:29:20.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:20 smithi130 ceph-mon[32702]: from='client.? 172.21.15.130:0/3784302850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:29:20.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:20 smithi032 ceph-mon[32461]: from='client.? 172.21.15.130:0/1731045037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c129f89-41b2-4ba9-8e22-0863133613c2"}]: dispatch 2024-04-25T01:29:20.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:20 smithi032 ceph-mon[32461]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c129f89-41b2-4ba9-8e22-0863133613c2"}]: dispatch 2024-04-25T01:29:20.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:20 smithi032 ceph-mon[32461]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c129f89-41b2-4ba9-8e22-0863133613c2"}]': finished 2024-04-25T01:29:20.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:20 smithi032 ceph-mon[32461]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T01:29:20.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:20 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:29:20.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:20 smithi032 ceph-mon[32461]: from='client.? 172.21.15.130:0/3784302850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:29:21.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:20 smithi028 ceph-mon[29390]: from='client.? 172.21.15.130:0/1731045037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c129f89-41b2-4ba9-8e22-0863133613c2"}]: dispatch 2024-04-25T01:29:21.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:20 smithi028 ceph-mon[29390]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c129f89-41b2-4ba9-8e22-0863133613c2"}]: dispatch 2024-04-25T01:29:21.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:20 smithi028 ceph-mon[29390]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c129f89-41b2-4ba9-8e22-0863133613c2"}]': finished 2024-04-25T01:29:21.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:20 smithi028 ceph-mon[29390]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T01:29:21.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:29:21.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:20 smithi028 ceph-mon[29390]: from='client.? 172.21.15.130:0/3784302850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:29:22.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:22 smithi028 ceph-mon[29390]: pgmap v249: 1 pgs: 1 remapped+peering; 577 KiB data, 338 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:22.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:22 smithi130 ceph-mon[32702]: pgmap v249: 1 pgs: 1 remapped+peering; 577 KiB data, 338 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:22.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:22 smithi032 ceph-mon[32461]: pgmap v249: 1 pgs: 1 remapped+peering; 577 KiB data, 338 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:24.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:24 smithi028 ceph-mon[29390]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:29:24.386 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:24 smithi130 ceph-mon[32702]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:29:24.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:24 smithi032 ceph-mon[32461]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:29:26.290 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:26 smithi130 ceph-mon[32702]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-25T01:29:26.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:26 smithi028 ceph-mon[29390]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-25T01:29:26.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:26 smithi032 ceph-mon[32461]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-25T01:29:28.282 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:28 smithi130 ceph-mon[32702]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-25T01:29:28.282 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:28 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T01:29:28.282 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:28 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:28.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:28 smithi028 ceph-mon[29390]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-25T01:29:28.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:28 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T01:29:28.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:28 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:28.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:28 smithi032 ceph-mon[32461]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-25T01:29:28.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:28 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T01:29:28.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:28 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:29.261 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:29 smithi130 ceph-mon[32702]: Deploying daemon osd.11 on smithi130 2024-04-25T01:29:29.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:29 smithi028 ceph-mon[29390]: Deploying daemon osd.11 on smithi130 2024-04-25T01:29:29.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:29 smithi032 ceph-mon[32461]: Deploying daemon osd.11 on smithi130 2024-04-25T01:29:30.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:30 smithi130 ceph-mon[32702]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:29:30.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:30 smithi028 ceph-mon[29390]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:29:30.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:30 smithi032 ceph-mon[32461]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:29:32.287 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:32 smithi032 ceph-mon[32461]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-25T01:29:32.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:32 smithi028 ceph-mon[29390]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-25T01:29:32.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:32 smithi130 ceph-mon[32702]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-25T01:29:34.126 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:34 smithi130 ceph-mon[32702]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T01:29:34.126 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:34 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:34.126 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:34 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:34.126 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:34 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:34.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:34 smithi028 ceph-mon[29390]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T01:29:34.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:34 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:34.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:34 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:34.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:34 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:34.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:34 smithi032 ceph-mon[32461]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T01:29:34.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:34 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:34.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:34 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:34.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:34 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:36.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:36 smithi130 ceph-mon[32702]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:36.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:36 smithi028 ceph-mon[29390]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:36.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:36 smithi032 ceph-mon[32461]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:37.281 INFO:teuthology.orchestra.run.smithi130.stdout:Created osd(s) 11 on host 'smithi130' 2024-04-25T01:29:37.282 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:37.302+0000 7f179ffff700 1 -- 172.21.15.130:0/2172898732 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f17b8062670 con 0x7f17a00a2c20 2024-04-25T01:29:37.283 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:37.305+0000 7f17bd2c8700 1 -- 172.21.15.130:0/2172898732 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f17a00a2c20 msgr2=0x7f17a00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:37.284 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:37.305+0000 7f17bd2c8700 1 --2- 172.21.15.130:0/2172898732 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f17a00a2c20 0x7f17a00a50e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f17b81021e0 tx=0x7f17b0005f90 comp rx=0 tx=0).stop 2024-04-25T01:29:37.284 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:37.305+0000 7f17bd2c8700 1 -- 172.21.15.130:0/2172898732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17b8102380 msgr2=0x7f17b819d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:37.284 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:37.305+0000 7f17bd2c8700 1 --2- 172.21.15.130:0/2172898732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17b8102380 0x7f17b819d190 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f17ac00ba50 tx=0x7f17ac00be10 comp rx=0 tx=0).stop 2024-04-25T01:29:37.284 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:37.305+0000 7f17bd2c8700 1 -- 172.21.15.130:0/2172898732 shutdown_connections 2024-04-25T01:29:37.284 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:37.305+0000 7f17bd2c8700 1 --2- 172.21.15.130:0/2172898732 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f17a00a2c20 0x7f17a00a50e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:37.284 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:37.305+0000 7f17bd2c8700 1 --2- 172.21.15.130:0/2172898732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17b8102d20 0x7f17b819f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:37.284 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:37.305+0000 7f17bd2c8700 1 --2- 172.21.15.130:0/2172898732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17b8101180 0x7f17b806b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:37.284 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:37.305+0000 7f17bd2c8700 1 --2- 172.21.15.130:0/2172898732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17b8102380 0x7f17b819d190 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:37.284 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:37.305+0000 7f17bd2c8700 1 -- 172.21.15.130:0/2172898732 >> 172.21.15.130:0/2172898732 conn(0x7f17b80fc720 msgr2=0x7f17b8105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:37.284 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:37.305+0000 7f17bd2c8700 1 -- 172.21.15.130:0/2172898732 shutdown_connections 2024-04-25T01:29:37.284 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:37.305+0000 7f17bd2c8700 1 -- 172.21.15.130:0/2172898732 wait complete. 2024-04-25T01:29:37.996 DEBUG:teuthology.orchestra.run.smithi130:osd.11> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.11.service 2024-04-25T01:29:37.998 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-25T01:29:37.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T01:29:38.262 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:29:38.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:38 smithi028 ceph-mon[29390]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:38.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:38 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:38.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:38 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:38.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:38 smithi028 ceph-mon[29390]: from='osd.11 [v2:172.21.15.130:6824/3995209547,v1:172.21.15.130:6825/3995209547]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:29:38.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:38 smithi028 ceph-mon[29390]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:29:38.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:38 smithi130 ceph-mon[32702]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:38.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:38 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:38.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:38 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:38.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:38 smithi130 ceph-mon[32702]: from='osd.11 [v2:172.21.15.130:6824/3995209547,v1:172.21.15.130:6825/3995209547]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:29:38.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:38 smithi130 ceph-mon[32702]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:29:38.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:38 smithi032 ceph-mon[32461]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:38.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:38 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:38.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:38 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:38.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:38 smithi032 ceph-mon[32461]: from='osd.11 [v2:172.21.15.130:6824/3995209547,v1:172.21.15.130:6825/3995209547]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:29:38.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:38 smithi032 ceph-mon[32461]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:29:39.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:39 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:39.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:39 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:39.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:39 smithi130 ceph-mon[32702]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T01:29:39.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:39 smithi130 ceph-mon[32702]: osdmap e80: 12 total, 11 up, 12 in 2024-04-25T01:29:39.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:39 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:29:39.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:39 smithi130 ceph-mon[32702]: from='osd.11 [v2:172.21.15.130:6824/3995209547,v1:172.21.15.130:6825/3995209547]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:29:39.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:39 smithi130 ceph-mon[32702]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:29:39.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:39 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:39.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:39 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:39.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:39 smithi028 ceph-mon[29390]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T01:29:39.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:39 smithi028 ceph-mon[29390]: osdmap e80: 12 total, 11 up, 12 in 2024-04-25T01:29:39.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:39 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:29:39.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:39 smithi028 ceph-mon[29390]: from='osd.11 [v2:172.21.15.130:6824/3995209547,v1:172.21.15.130:6825/3995209547]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:29:39.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:39 smithi028 ceph-mon[29390]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:29:39.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:39 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:39.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:39 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:39.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:39 smithi032 ceph-mon[32461]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T01:29:39.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:39 smithi032 ceph-mon[32461]: osdmap e80: 12 total, 11 up, 12 in 2024-04-25T01:29:39.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:39 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:29:39.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:39 smithi032 ceph-mon[32461]: from='osd.11 [v2:172.21.15.130:6824/3995209547,v1:172.21.15.130:6825/3995209547]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:29:39.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:39 smithi032 ceph-mon[32461]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]: dispatch 2024-04-25T01:29:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.032+0000 7fa977ffe700 1 -- 172.21.15.28:0/613717285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9701022b0 msgr2=0x7fa9701026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.032+0000 7fa977ffe700 1 --2- 172.21.15.28:0/613717285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9701022b0 0x7fa9701026d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa96c0099c0 tx=0x7fa96c009cd0 comp rx=0 tx=0).stop 2024-04-25T01:29:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.032+0000 7fa977ffe700 1 -- 172.21.15.28:0/613717285 shutdown_connections 2024-04-25T01:29:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.032+0000 7fa977ffe700 1 --2- 172.21.15.28:0/613717285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa970103e50 0x7fa97010a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.032+0000 7fa977ffe700 1 --2- 172.21.15.28:0/613717285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9701022b0 0x7fa9701026d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.032+0000 7fa977ffe700 1 --2- 172.21.15.28:0/613717285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9701034b0 0x7fa970103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.032+0000 7fa977ffe700 1 -- 172.21.15.28:0/613717285 >> 172.21.15.28:0/613717285 conn(0x7fa9700fd850 msgr2=0x7fa9700ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.032+0000 7fa977ffe700 1 -- 172.21.15.28:0/613717285 shutdown_connections 2024-04-25T01:29:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.032+0000 7fa977ffe700 1 -- 172.21.15.28:0/613717285 wait complete. 2024-04-25T01:29:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.032+0000 7fa977ffe700 1 Processor -- start 2024-04-25T01:29:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.032+0000 7fa977ffe700 1 -- start start 2024-04-25T01:29:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.033+0000 7fa977ffe700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9701022b0 0x7fa97019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.033+0000 7fa977ffe700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9701034b0 0x7fa97019ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.033+0000 7fa977ffe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa970103e50 0x7fa9701a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.033+0000 7fa977ffe700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa97010dee0 con 0x7fa970103e50 2024-04-25T01:29:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.033+0000 7fa977ffe700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa97010dd60 con 0x7fa9701022b0 2024-04-25T01:29:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.033+0000 7fa977ffe700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa97010e060 con 0x7fa9701034b0 2024-04-25T01:29:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.033+0000 7fa97659b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa970103e50 0x7fa9701a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.033+0000 7fa97659b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa970103e50 0x7fa9701a1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60122/0 (socket says 172.21.15.28:60122) 2024-04-25T01:29:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.033+0000 7fa97659b700 1 -- 172.21.15.28:0/3622035664 learned_addr learned my addr 172.21.15.28:0/3622035664 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:29:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.033+0000 7fa975599700 1 --2- 172.21.15.28:0/3622035664 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9701034b0 0x7fa97019ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.033+0000 7fa975d9a700 1 --2- 172.21.15.28:0/3622035664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9701022b0 0x7fa97019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.033+0000 7fa97659b700 1 -- 172.21.15.28:0/3622035664 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9701034b0 msgr2=0x7fa97019ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.033+0000 7fa97659b700 1 --2- 172.21.15.28:0/3622035664 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9701034b0 0x7fa97019ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.033+0000 7fa97659b700 1 -- 172.21.15.28:0/3622035664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9701022b0 msgr2=0x7fa97019a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.033+0000 7fa97659b700 1 --2- 172.21.15.28:0/3622035664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9701022b0 0x7fa97019a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.033+0000 7fa97659b700 1 -- 172.21.15.28:0/3622035664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9701a2390 con 0x7fa970103e50 2024-04-25T01:29:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.033+0000 7fa975599700 1 --2- 172.21.15.28:0/3622035664 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9701034b0 0x7fa97019ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:29:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.033+0000 7fa97659b700 1 --2- 172.21.15.28:0/3622035664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa970103e50 0x7fa9701a1cf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa96800add0 tx=0x7fa96800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.033+0000 7fa966ffd700 1 -- 172.21.15.28:0/3622035664 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa968017020 con 0x7fa970103e50 2024-04-25T01:29:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.034+0000 7fa966ffd700 1 -- 172.21.15.28:0/3622035664 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa96800ae00 con 0x7fa970103e50 2024-04-25T01:29:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.034+0000 7fa977ffe700 1 -- 172.21.15.28:0/3622035664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9701a2680 con 0x7fa970103e50 2024-04-25T01:29:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.034+0000 7fa966ffd700 1 -- 172.21.15.28:0/3622035664 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9680053e0 con 0x7fa970103e50 2024-04-25T01:29:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.034+0000 7fa977ffe700 1 -- 172.21.15.28:0/3622035664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9701a2bc0 con 0x7fa970103e50 2024-04-25T01:29:40.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.035+0000 7fa977ffe700 1 -- 172.21.15.28:0/3622035664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa970068f20 con 0x7fa970103e50 2024-04-25T01:29:40.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.036+0000 7fa966ffd700 1 -- 172.21.15.28:0/3622035664 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa96800c890 con 0x7fa970103e50 2024-04-25T01:29:40.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.036+0000 7fa966ffd700 1 --2- 172.21.15.28:0/3622035664 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa95c0a2ca0 0x7fa95c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:40.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.037+0000 7fa975d9a700 1 --2- 172.21.15.28:0/3622035664 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa95c0a2ca0 0x7fa95c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:40.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.037+0000 7fa966ffd700 1 -- 172.21.15.28:0/3622035664 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 6847+0+0 (secure 0 0 0) 0x7fa9680ce1b0 con 0x7fa970103e50 2024-04-25T01:29:40.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.037+0000 7fa975d9a700 1 --2- 172.21.15.28:0/3622035664 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa95c0a2ca0 0x7fa95c0a5160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa96c00b580 tx=0x7fa96c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.041+0000 7fa966ffd700 1 -- 172.21.15.28:0/3622035664 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa968021020 con 0x7fa970103e50 2024-04-25T01:29:40.028 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:29:39 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:29:39.959+0000 7f5fb282c700 -1 osd.11 0 waiting for initial osdmap 2024-04-25T01:29:40.028 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:29:39 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:29:39.971+0000 7f5faa617700 -1 osd.11 81 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:29:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.250+0000 7fa977ffe700 1 -- 172.21.15.28:0/3622035664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fa9701a3000 con 0x7fa970103e50 2024-04-25T01:29:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.250+0000 7fa966ffd700 1 -- 172.21.15.28:0/3622035664 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7fa96809b760 con 0x7fa970103e50 2024-04-25T01:29:40.228 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:29:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.252+0000 7fa977ffe700 1 -- 172.21.15.28:0/3622035664 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa95c0a2ca0 msgr2=0x7fa95c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.252+0000 7fa977ffe700 1 --2- 172.21.15.28:0/3622035664 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa95c0a2ca0 0x7fa95c0a5160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa96c00b580 tx=0x7fa96c005fb0 comp rx=0 tx=0).stop 2024-04-25T01:29:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.252+0000 7fa977ffe700 1 -- 172.21.15.28:0/3622035664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa970103e50 msgr2=0x7fa9701a1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.252+0000 7fa977ffe700 1 --2- 172.21.15.28:0/3622035664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa970103e50 0x7fa9701a1cf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa96800add0 tx=0x7fa96800c330 comp rx=0 tx=0).stop 2024-04-25T01:29:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.252+0000 7fa977ffe700 1 -- 172.21.15.28:0/3622035664 shutdown_connections 2024-04-25T01:29:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.252+0000 7fa977ffe700 1 --2- 172.21.15.28:0/3622035664 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa95c0a2ca0 0x7fa95c0a5160 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.252+0000 7fa977ffe700 1 --2- 172.21.15.28:0/3622035664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa970103e50 0x7fa9701a1cf0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.252+0000 7fa977ffe700 1 --2- 172.21.15.28:0/3622035664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9701022b0 0x7fa97019a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.252+0000 7fa977ffe700 1 --2- 172.21.15.28:0/3622035664 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa9701034b0 0x7fa97019ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.252+0000 7fa977ffe700 1 -- 172.21.15.28:0/3622035664 >> 172.21.15.28:0/3622035664 conn(0x7fa9700fd850 msgr2=0x7fa970074460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.252+0000 7fa977ffe700 1 -- 172.21.15.28:0/3622035664 shutdown_connections 2024-04-25T01:29:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:40.252+0000 7fa977ffe700 1 -- 172.21.15.28:0/3622035664 wait complete. 2024-04-25T01:29:40.345 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:40 smithi028 ceph-mon[29390]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:40.345 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:40 smithi028 ceph-mon[29390]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-25T01:29:40.345 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:40 smithi028 ceph-mon[29390]: osdmap e81: 12 total, 11 up, 12 in 2024-04-25T01:29:40.345 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:40 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:29:40.345 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:40 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:29:40.345 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:40 smithi028 ceph-mon[29390]: from='osd.11 ' entity='osd.11' 2024-04-25T01:29:40.356 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:40 smithi130 ceph-mon[32702]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:40.357 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:40 smithi130 ceph-mon[32702]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-25T01:29:40.357 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:40 smithi130 ceph-mon[32702]: osdmap e81: 12 total, 11 up, 12 in 2024-04-25T01:29:40.357 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:40 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:29:40.357 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:40 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:29:40.357 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:40 smithi130 ceph-mon[32702]: from='osd.11 ' entity='osd.11' 2024-04-25T01:29:40.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:40 smithi032 ceph-mon[32461]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:40.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:40 smithi032 ceph-mon[32461]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi130", "root=default"]}]': finished 2024-04-25T01:29:40.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:40 smithi032 ceph-mon[32461]: osdmap e81: 12 total, 11 up, 12 in 2024-04-25T01:29:40.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:40 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:29:40.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:40 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:29:40.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:40 smithi032 ceph-mon[32461]: from='osd.11 ' entity='osd.11' 2024-04-25T01:29:40.882 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"num_osds":12,"num_up_osds":11,"osd_up_since":1714008554,"num_in_osds":12,"osd_in_since":1714008560,"num_remapped_pgs":0} 2024-04-25T01:29:41.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:41 smithi130 ceph-mon[32702]: purged_snaps scrub starts 2024-04-25T01:29:41.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:41 smithi130 ceph-mon[32702]: purged_snaps scrub ok 2024-04-25T01:29:41.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:41 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3622035664' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:29:41.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:41 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:29:41.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:41 smithi130 ceph-mon[32702]: osd.11 [v2:172.21.15.130:6824/3995209547,v1:172.21.15.130:6825/3995209547] boot 2024-04-25T01:29:41.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:41 smithi130 ceph-mon[32702]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T01:29:41.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:41 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:29:41.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:41 smithi032 ceph-mon[32461]: purged_snaps scrub starts 2024-04-25T01:29:41.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:41 smithi032 ceph-mon[32461]: purged_snaps scrub ok 2024-04-25T01:29:41.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:41 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3622035664' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:29:41.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:41 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:29:41.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:41 smithi032 ceph-mon[32461]: osd.11 [v2:172.21.15.130:6824/3995209547,v1:172.21.15.130:6825/3995209547] boot 2024-04-25T01:29:41.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:41 smithi032 ceph-mon[32461]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T01:29:41.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:41 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:29:41.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:41 smithi028 ceph-mon[29390]: purged_snaps scrub starts 2024-04-25T01:29:41.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:41 smithi028 ceph-mon[29390]: purged_snaps scrub ok 2024-04-25T01:29:41.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:41 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3622035664' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:29:41.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:41 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:29:41.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:41 smithi028 ceph-mon[29390]: osd.11 [v2:172.21.15.130:6824/3995209547,v1:172.21.15.130:6825/3995209547] boot 2024-04-25T01:29:41.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:41 smithi028 ceph-mon[29390]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T01:29:41.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:41 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:29:41.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T01:29:42.131 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:29:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:42 smithi028 ceph-mon[29390]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:42 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:42 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:42 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:42 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:42 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:42 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:42.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:42 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:42.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:42 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:42.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:42 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:42.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:42 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:42.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:42 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:42.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:42 smithi130 ceph-mon[32702]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:42.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:42 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:42.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:42 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:42.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:42 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:42.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:42 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:42.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:42 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:42.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:42 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:42.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:42 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:42.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:42 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:42.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:42 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:42.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:42 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:42.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:42 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:42.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:42 smithi032 ceph-mon[32461]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:29:42.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:42 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:42.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:42 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:42.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:42 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:42.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:42 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:42.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:42 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:42.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:42 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:29:42.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:42 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:42.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:42 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:42.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:42 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:42.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:42 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:42.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:42 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:43.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:43 smithi028 ceph-mon[29390]: Detected new or changed devices on smithi130 2024-04-25T01:29:43.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:43 smithi028 ceph-mon[29390]: Adjusting osd_memory_target on smithi130 to 4272M 2024-04-25T01:29:43.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:43 smithi028 ceph-mon[29390]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T01:29:43.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:43 smithi028 ceph-mon[29390]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:43.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:43 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:43.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:43 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:43.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:43 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:43.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:43 smithi130 ceph-mon[32702]: Detected new or changed devices on smithi130 2024-04-25T01:29:43.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:43 smithi130 ceph-mon[32702]: Adjusting osd_memory_target on smithi130 to 4272M 2024-04-25T01:29:43.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:43 smithi130 ceph-mon[32702]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T01:29:43.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:43 smithi130 ceph-mon[32702]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:43.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:43 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:43.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:43 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:43.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:43 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:43.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:43 smithi032 ceph-mon[32461]: Detected new or changed devices on smithi130 2024-04-25T01:29:43.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:43 smithi032 ceph-mon[32461]: Adjusting osd_memory_target on smithi130 to 4272M 2024-04-25T01:29:43.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:43 smithi032 ceph-mon[32461]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T01:29:43.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:43 smithi032 ceph-mon[32461]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:43.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:43 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:43.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:43 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:43.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:43 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.936+0000 7fe99eb99700 1 -- 172.21.15.28:0/1547863406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe998101180 msgr2=0x7fe9981015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.936+0000 7fe99eb99700 1 --2- 172.21.15.28:0/1547863406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe998101180 0x7fe9981015a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe98c0099c0 tx=0x7fe98c009cd0 comp rx=0 tx=0).stop 2024-04-25T01:29:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.936+0000 7fe99eb99700 1 -- 172.21.15.28:0/1547863406 shutdown_connections 2024-04-25T01:29:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.936+0000 7fe99eb99700 1 --2- 172.21.15.28:0/1547863406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe998102d20 0x7fe9981095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.936+0000 7fe99eb99700 1 --2- 172.21.15.28:0/1547863406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe998101180 0x7fe9981015a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.936+0000 7fe99eb99700 1 --2- 172.21.15.28:0/1547863406 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe998102380 0x7fe9981027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.936+0000 7fe99eb99700 1 -- 172.21.15.28:0/1547863406 >> 172.21.15.28:0/1547863406 conn(0x7fe9980fc720 msgr2=0x7fe9980feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.936+0000 7fe99eb99700 1 -- 172.21.15.28:0/1547863406 shutdown_connections 2024-04-25T01:29:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.936+0000 7fe99eb99700 1 -- 172.21.15.28:0/1547863406 wait complete. 2024-04-25T01:29:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe99eb99700 1 Processor -- start 2024-04-25T01:29:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe99eb99700 1 -- start start 2024-04-25T01:29:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe99eb99700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe998101180 0x7fe998105110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe99eb99700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe998102380 0x7fe998103760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe99eb99700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe998102d20 0x7fe998103ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe99eb99700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe99810ce40 con 0x7fe998102d20 2024-04-25T01:29:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe99eb99700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe99810ccc0 con 0x7fe998101180 2024-04-25T01:29:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe99eb99700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe99810cfc0 con 0x7fe998102380 2024-04-25T01:29:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe99d136700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe998102d20 0x7fe998103ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe99d136700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe998102d20 0x7fe998103ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60140/0 (socket says 172.21.15.28:60140) 2024-04-25T01:29:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe99d136700 1 -- 172.21.15.28:0/104106378 learned_addr learned my addr 172.21.15.28:0/104106378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:29:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe99c935700 1 --2- 172.21.15.28:0/104106378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe998101180 0x7fe998105110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe997fff700 1 --2- 172.21.15.28:0/104106378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe998102380 0x7fe998103760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe99d136700 1 -- 172.21.15.28:0/104106378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe998102380 msgr2=0x7fe998103760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe99d136700 1 --2- 172.21.15.28:0/104106378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe998102380 0x7fe998103760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe99d136700 1 -- 172.21.15.28:0/104106378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe998101180 msgr2=0x7fe998105110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe99d136700 1 --2- 172.21.15.28:0/104106378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe998101180 0x7fe998105110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe99d136700 1 -- 172.21.15.28:0/104106378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe998104550 con 0x7fe998102d20 2024-04-25T01:29:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe997fff700 1 --2- 172.21.15.28:0/104106378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe998102380 0x7fe998103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe99c935700 1 --2- 172.21.15.28:0/104106378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe998101180 0x7fe998105110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:29:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.937+0000 7fe99d136700 1 --2- 172.21.15.28:0/104106378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe998102d20 0x7fe998103ca0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe99000b970 tx=0x7fe99000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.938+0000 7fe995ffb700 1 -- 172.21.15.28:0/104106378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe990019070 con 0x7fe998102d20 2024-04-25T01:29:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.938+0000 7fe99eb99700 1 -- 172.21.15.28:0/104106378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9981a27b0 con 0x7fe998102d20 2024-04-25T01:29:43.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.938+0000 7fe995ffb700 1 -- 172.21.15.28:0/104106378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe990014440 con 0x7fe998102d20 2024-04-25T01:29:43.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.938+0000 7fe995ffb700 1 -- 172.21.15.28:0/104106378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9900075d0 con 0x7fe998102d20 2024-04-25T01:29:43.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.938+0000 7fe99eb99700 1 -- 172.21.15.28:0/104106378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9981a2d40 con 0x7fe998102d20 2024-04-25T01:29:43.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.939+0000 7fe99eb99700 1 -- 172.21.15.28:0/104106378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe998068f20 con 0x7fe998102d20 2024-04-25T01:29:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.940+0000 7fe995ffb700 1 -- 172.21.15.28:0/104106378 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe99001c070 con 0x7fe998102d20 2024-04-25T01:29:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.941+0000 7fe995ffb700 1 --2- 172.21.15.28:0/104106378 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe9880a2c20 0x7fe9880a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.941+0000 7fe99c935700 1 --2- 172.21.15.28:0/104106378 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe9880a2c20 0x7fe9880a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.941+0000 7fe995ffb700 1 -- 172.21.15.28:0/104106378 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fe9900ce4d0 con 0x7fe998102d20 2024-04-25T01:29:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.941+0000 7fe99c935700 1 --2- 172.21.15.28:0/104106378 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe9880a2c20 0x7fe9880a50e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe98c003890 tx=0x7fe98c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:43.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:43.944+0000 7fe995ffb700 1 -- 172.21.15.28:0/104106378 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe99009b8d0 con 0x7fe998102d20 2024-04-25T01:29:44.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:44.152+0000 7fe99eb99700 1 -- 172.21.15.28:0/104106378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fe9981a3160 con 0x7fe998102d20 2024-04-25T01:29:44.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:44.152+0000 7fe995ffb700 1 -- 172.21.15.28:0/104106378 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v83) ==== 74+0+133 (secure 0 0 0) 0x7fe99009b1d0 con 0x7fe998102d20 2024-04-25T01:29:44.131 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:29:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:44.154+0000 7fe99eb99700 1 -- 172.21.15.28:0/104106378 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe9880a2c20 msgr2=0x7fe9880a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:44.154+0000 7fe99eb99700 1 --2- 172.21.15.28:0/104106378 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe9880a2c20 0x7fe9880a50e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe98c003890 tx=0x7fe98c005fd0 comp rx=0 tx=0).stop 2024-04-25T01:29:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:44.154+0000 7fe99eb99700 1 -- 172.21.15.28:0/104106378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe998102d20 msgr2=0x7fe998103ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:44.154+0000 7fe99eb99700 1 --2- 172.21.15.28:0/104106378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe998102d20 0x7fe998103ca0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe99000b970 tx=0x7fe99000bd30 comp rx=0 tx=0).stop 2024-04-25T01:29:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:44.154+0000 7fe99eb99700 1 -- 172.21.15.28:0/104106378 shutdown_connections 2024-04-25T01:29:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:44.154+0000 7fe99eb99700 1 --2- 172.21.15.28:0/104106378 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe9880a2c20 0x7fe9880a50e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:44.154+0000 7fe99eb99700 1 --2- 172.21.15.28:0/104106378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe998102d20 0x7fe998103ca0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:44.154+0000 7fe99eb99700 1 --2- 172.21.15.28:0/104106378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe998101180 0x7fe998105110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:44.154+0000 7fe99eb99700 1 --2- 172.21.15.28:0/104106378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe998102380 0x7fe998103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:44.154+0000 7fe99eb99700 1 -- 172.21.15.28:0/104106378 >> 172.21.15.28:0/104106378 conn(0x7fe9980fc720 msgr2=0x7fe99806a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:44.154+0000 7fe99eb99700 1 -- 172.21.15.28:0/104106378 shutdown_connections 2024-04-25T01:29:44.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:44.154+0000 7fe99eb99700 1 -- 172.21.15.28:0/104106378 wait complete. 2024-04-25T01:29:44.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:44 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/104106378' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:29:44.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:44 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/104106378' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:29:44.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:44 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/104106378' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:29:44.847 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":83,"num_osds":12,"num_up_osds":12,"osd_up_since":1714008580,"num_in_osds":12,"osd_in_since":1714008560,"num_remapped_pgs":0} 2024-04-25T01:29:44.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T01:29:45.095 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:29:45.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:45 smithi130 ceph-mon[32702]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:45.461 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:45 smithi028 ceph-mon[29390]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:45.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:45 smithi032 ceph-mon[32461]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.847+0000 7f320a720700 1 -- 172.21.15.28:0/1271419865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3204107570 msgr2=0x7f320410f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.847+0000 7f320a720700 1 --2- 172.21.15.28:0/1271419865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3204107570 0x7f320410f140 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f31fc00b0a0 tx=0x7f31fc00b3b0 comp rx=0 tx=0).stop 2024-04-25T01:29:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.848+0000 7f320a720700 1 -- 172.21.15.28:0/1271419865 shutdown_connections 2024-04-25T01:29:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.848+0000 7f320a720700 1 --2- 172.21.15.28:0/1271419865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3204107570 0x7f320410f140 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.848+0000 7f320a720700 1 --2- 172.21.15.28:0/1271419865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f32041024e0 0x7f32041028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.848+0000 7f320a720700 1 --2- 172.21.15.28:0/1271419865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3204102e90 0x7f3204106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.848+0000 7f320a720700 1 -- 172.21.15.28:0/1271419865 >> 172.21.15.28:0/1271419865 conn(0x7f3204079390 msgr2=0x7f32040797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.848+0000 7f320a720700 1 -- 172.21.15.28:0/1271419865 shutdown_connections 2024-04-25T01:29:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.848+0000 7f320a720700 1 -- 172.21.15.28:0/1271419865 wait complete. 2024-04-25T01:29:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.848+0000 7f320a720700 1 Processor -- start 2024-04-25T01:29:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.848+0000 7f320a720700 1 -- start start 2024-04-25T01:29:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.849+0000 7f320a720700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32041024e0 0x7f320419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.849+0000 7f320a720700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3204102e90 0x7f320419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.849+0000 7f320a720700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3204107570 0x7f32041988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:46.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.849+0000 7f320a720700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3204112630 con 0x7f32041024e0 2024-04-25T01:29:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.849+0000 7f320a720700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f32041124b0 con 0x7f3204102e90 2024-04-25T01:29:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.849+0000 7f320a720700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f32041127b0 con 0x7f3204107570 2024-04-25T01:29:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.849+0000 7f3203fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32041024e0 0x7f320419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.849+0000 7f3203fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32041024e0 0x7f320419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34502/0 (socket says 172.21.15.28:34502) 2024-04-25T01:29:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.849+0000 7f3203fff700 1 -- 172.21.15.28:0/1869169170 learned_addr learned my addr 172.21.15.28:0/1869169170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:29:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.849+0000 7f3208cbd700 1 --2- 172.21.15.28:0/1869169170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3204107570 0x7f32041988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.849+0000 7f32037fe700 1 --2- 172.21.15.28:0/1869169170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3204102e90 0x7f320419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.849+0000 7f3203fff700 1 -- 172.21.15.28:0/1869169170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3204107570 msgr2=0x7f32041988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.849+0000 7f3203fff700 1 --2- 172.21.15.28:0/1869169170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3204107570 0x7f32041988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.849+0000 7f3203fff700 1 -- 172.21.15.28:0/1869169170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3204102e90 msgr2=0x7f320419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.849+0000 7f3203fff700 1 --2- 172.21.15.28:0/1869169170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3204102e90 0x7f320419ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:46.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.849+0000 7f3203fff700 1 -- 172.21.15.28:0/1869169170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3204199010 con 0x7f32041024e0 2024-04-25T01:29:46.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.849+0000 7f3203fff700 1 --2- 172.21.15.28:0/1869169170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32041024e0 0x7f320419e6a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f31f400b530 tx=0x7f31f400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:46.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.850+0000 7f32017fa700 1 -- 172.21.15.28:0/1869169170 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f4015020 con 0x7f32041024e0 2024-04-25T01:29:46.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.850+0000 7f32017fa700 1 -- 172.21.15.28:0/1869169170 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31f4011d10 con 0x7f32041024e0 2024-04-25T01:29:46.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.850+0000 7f320a720700 1 -- 172.21.15.28:0/1869169170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3204199300 con 0x7f32041024e0 2024-04-25T01:29:46.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.850+0000 7f32017fa700 1 -- 172.21.15.28:0/1869169170 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f400f2e0 con 0x7f32041024e0 2024-04-25T01:29:46.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.850+0000 7f320a720700 1 -- 172.21.15.28:0/1869169170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32041a5570 con 0x7f32041024e0 2024-04-25T01:29:46.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.851+0000 7f320a720700 1 -- 172.21.15.28:0/1869169170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3204068f20 con 0x7f32041024e0 2024-04-25T01:29:46.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.852+0000 7f32017fa700 1 -- 172.21.15.28:0/1869169170 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f31f4018020 con 0x7f32041024e0 2024-04-25T01:29:46.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.852+0000 7f32017fa700 1 --2- 172.21.15.28:0/1869169170 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f31ec0a2ca0 0x7f31ec0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:46.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.853+0000 7f32037fe700 1 --2- 172.21.15.28:0/1869169170 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f31ec0a2ca0 0x7f31ec0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:46.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.853+0000 7f32017fa700 1 -- 172.21.15.28:0/1869169170 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f31f40ce960 con 0x7f32041024e0 2024-04-25T01:29:46.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.853+0000 7f32037fe700 1 --2- 172.21.15.28:0/1869169170 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f31ec0a2ca0 0x7f31ec0a5160 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f31f8006fd0 tx=0x7f31f8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:46.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:46.856+0000 7f32017fa700 1 -- 172.21.15.28:0/1869169170 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f31f401f020 con 0x7f32041024e0 2024-04-25T01:29:47.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:47.066+0000 7f320a720700 1 -- 172.21.15.28:0/1869169170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3204199d90 con 0x7f32041024e0 2024-04-25T01:29:47.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:47.068+0000 7f32017fa700 1 -- 172.21.15.28:0/1869169170 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18421 (secure 0 0 0) 0x7f31f409bd60 con 0x7f32041024e0 2024-04-25T01:29:47.047 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:29:47.047 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":83,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:29:42.122155+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:29:47.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:47.071+0000 7f320a720700 1 -- 172.21.15.28:0/1869169170 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f31ec0a2ca0 msgr2=0x7f31ec0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:47.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:47.071+0000 7f320a720700 1 --2- 172.21.15.28:0/1869169170 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f31ec0a2ca0 0x7f31ec0a5160 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f31f8006fd0 tx=0x7f31f8008040 comp rx=0 tx=0).stop 2024-04-25T01:29:47.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:47.071+0000 7f320a720700 1 -- 172.21.15.28:0/1869169170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32041024e0 msgr2=0x7f320419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:47.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:47.071+0000 7f320a720700 1 --2- 172.21.15.28:0/1869169170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32041024e0 0x7f320419e6a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f31f400b530 tx=0x7f31f400b840 comp rx=0 tx=0).stop 2024-04-25T01:29:47.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:47.071+0000 7f320a720700 1 -- 172.21.15.28:0/1869169170 shutdown_connections 2024-04-25T01:29:47.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:47.071+0000 7f320a720700 1 --2- 172.21.15.28:0/1869169170 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f31ec0a2ca0 0x7f31ec0a5160 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:47.071+0000 7f320a720700 1 --2- 172.21.15.28:0/1869169170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3204107570 0x7f32041988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:47.071+0000 7f320a720700 1 --2- 172.21.15.28:0/1869169170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32041024e0 0x7f320419e6a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:47.071+0000 7f320a720700 1 --2- 172.21.15.28:0/1869169170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3204102e90 0x7f320419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:47.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:47.071+0000 7f320a720700 1 -- 172.21.15.28:0/1869169170 >> 172.21.15.28:0/1869169170 conn(0x7f3204079390 msgr2=0x7f32041002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:47.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:47.071+0000 7f320a720700 1 -- 172.21.15.28:0/1869169170 shutdown_connections 2024-04-25T01:29:47.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:47.071+0000 7f320a720700 1 -- 172.21.15.28:0/1869169170 wait complete. 2024-04-25T01:29:47.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:47 smithi130 ceph-mon[32702]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:47.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:47 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1869169170' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:29:47.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:47 smithi032 ceph-mon[32461]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:47.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:47 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1869169170' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:29:47.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:47 smithi028 ceph-mon[29390]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:47.797 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:47 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1869169170' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:29:47.832 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:25:48.767414+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:29:47.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-25T01:29:48.083 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:29:49.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:49 smithi028 ceph-mon[29390]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:49.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:49 smithi130 ceph-mon[32702]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:49.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:49 smithi032 ceph-mon[32461]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.907+0000 7f8f10ac5700 1 -- 172.21.15.28:0/828619553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f0c06b5f0 msgr2=0x7f8f0c106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.907+0000 7f8f10ac5700 1 --2- 172.21.15.28:0/828619553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f0c06b5f0 0x7f8f0c106eb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8ef40099c0 tx=0x7f8ef4009cd0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.907+0000 7f8f10ac5700 1 -- 172.21.15.28:0/828619553 shutdown_connections 2024-04-25T01:29:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.907+0000 7f8f10ac5700 1 --2- 172.21.15.28:0/828619553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f0c107590 0x7f8f0c10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.907+0000 7f8f10ac5700 1 --2- 172.21.15.28:0/828619553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f0c06ac40 0x7f8f0c06b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.907+0000 7f8f10ac5700 1 --2- 172.21.15.28:0/828619553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f0c06b5f0 0x7f8f0c106eb0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.907+0000 7f8f10ac5700 1 -- 172.21.15.28:0/828619553 >> 172.21.15.28:0/828619553 conn(0x7f8f0c0781d0 msgr2=0x7f8f0c0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:49.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.907+0000 7f8f10ac5700 1 -- 172.21.15.28:0/828619553 shutdown_connections 2024-04-25T01:29:49.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.907+0000 7f8f10ac5700 1 -- 172.21.15.28:0/828619553 wait complete. 2024-04-25T01:29:49.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.908+0000 7f8f10ac5700 1 Processor -- start 2024-04-25T01:29:49.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.908+0000 7f8f10ac5700 1 -- start start 2024-04-25T01:29:49.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.908+0000 7f8f10ac5700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f0c06ac40 0x7f8f0c19e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.908+0000 7f8f10ac5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f0c06b5f0 0x7f8f0c19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.908+0000 7f8f10ac5700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f0c107590 0x7f8f0c1988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.908+0000 7f8f10ac5700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c1125c0 con 0x7f8f0c06ac40 2024-04-25T01:29:49.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.908+0000 7f8f10ac5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c112440 con 0x7f8f0c06b5f0 2024-04-25T01:29:49.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.908+0000 7f8f10ac5700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c112740 con 0x7f8f0c107590 2024-04-25T01:29:49.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.908+0000 7f8f0a59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f0c06ac40 0x7f8f0c19e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.908+0000 7f8f0a59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f0c06ac40 0x7f8f0c19e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34518/0 (socket says 172.21.15.28:34518) 2024-04-25T01:29:49.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.908+0000 7f8f0a59c700 1 -- 172.21.15.28:0/1692297601 learned_addr learned my addr 172.21.15.28:0/1692297601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:29:49.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.908+0000 7f8f03fff700 1 --2- 172.21.15.28:0/1692297601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f0c06b5f0 0x7f8f0c19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.908+0000 7f8f0ad9d700 1 --2- 172.21.15.28:0/1692297601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f0c107590 0x7f8f0c1988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.908+0000 7f8f0a59c700 1 -- 172.21.15.28:0/1692297601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f0c107590 msgr2=0x7f8f0c1988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.909+0000 7f8f0a59c700 1 --2- 172.21.15.28:0/1692297601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f0c107590 0x7f8f0c1988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.909+0000 7f8f0a59c700 1 -- 172.21.15.28:0/1692297601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f0c06b5f0 msgr2=0x7f8f0c19ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.909+0000 7f8f0a59c700 1 --2- 172.21.15.28:0/1692297601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f0c06b5f0 0x7f8f0c19ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.909+0000 7f8f0a59c700 1 -- 172.21.15.28:0/1692297601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f0c199020 con 0x7f8f0c06ac40 2024-04-25T01:29:49.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.909+0000 7f8f0ad9d700 1 --2- 172.21.15.28:0/1692297601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f0c107590 0x7f8f0c1988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:49.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.909+0000 7f8f0a59c700 1 --2- 172.21.15.28:0/1692297601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f0c06ac40 0x7f8f0c19e700 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8efc00d980 tx=0x7f8efc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:49.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.909+0000 7f8f037fe700 1 -- 172.21.15.28:0/1692297601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8efc016070 con 0x7f8f0c06ac40 2024-04-25T01:29:49.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.909+0000 7f8f10ac5700 1 -- 172.21.15.28:0/1692297601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f0c199310 con 0x7f8f0c06ac40 2024-04-25T01:29:49.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.909+0000 7f8f10ac5700 1 -- 172.21.15.28:0/1692297601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f0c1a55d0 con 0x7f8f0c06ac40 2024-04-25T01:29:49.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.909+0000 7f8f037fe700 1 -- 172.21.15.28:0/1692297601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8efc004d10 con 0x7f8f0c06ac40 2024-04-25T01:29:49.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.909+0000 7f8f037fe700 1 -- 172.21.15.28:0/1692297601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8efc0055f0 con 0x7f8f0c06ac40 2024-04-25T01:29:49.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.910+0000 7f8f10ac5700 1 -- 172.21.15.28:0/1692297601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8eec005330 con 0x7f8f0c06ac40 2024-04-25T01:29:49.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.912+0000 7f8f037fe700 1 -- 172.21.15.28:0/1692297601 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8efc004340 con 0x7f8f0c06ac40 2024-04-25T01:29:49.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.913+0000 7f8f037fe700 1 --2- 172.21.15.28:0/1692297601 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8ef80a2ca0 0x7f8ef80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:49.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.913+0000 7f8f03fff700 1 --2- 172.21.15.28:0/1692297601 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8ef80a2ca0 0x7f8ef80a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:49.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.913+0000 7f8f037fe700 1 -- 172.21.15.28:0/1692297601 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f8efc0ceef0 con 0x7f8f0c06ac40 2024-04-25T01:29:49.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.913+0000 7f8f03fff700 1 --2- 172.21.15.28:0/1692297601 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8ef80a2ca0 0x7f8ef80a5160 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4000c00 tx=0x7f8ef4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:49.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:49.916+0000 7f8f037fe700 1 -- 172.21.15.28:0/1692297601 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8efc01f020 con 0x7f8f0c06ac40 2024-04-25T01:29:50.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:50.120+0000 7f8f10ac5700 1 -- 172.21.15.28:0/1692297601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8eec0058f0 con 0x7f8f0c06ac40 2024-04-25T01:29:50.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:50.121+0000 7f8f037fe700 1 -- 172.21.15.28:0/1692297601 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f8efc09c290 con 0x7f8f0c06ac40 2024-04-25T01:29:50.099 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T01:29:50.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:50.123+0000 7f8f10ac5700 1 -- 172.21.15.28:0/1692297601 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8ef80a2ca0 msgr2=0x7f8ef80a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:50.123+0000 7f8f10ac5700 1 --2- 172.21.15.28:0/1692297601 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8ef80a2ca0 0x7f8ef80a5160 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4000c00 tx=0x7f8ef4005fb0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:50.123+0000 7f8f10ac5700 1 -- 172.21.15.28:0/1692297601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f0c06ac40 msgr2=0x7f8f0c19e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:50.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:50.123+0000 7f8f10ac5700 1 --2- 172.21.15.28:0/1692297601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f0c06ac40 0x7f8f0c19e700 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8efc00d980 tx=0x7f8efc00dc90 comp rx=0 tx=0).stop 2024-04-25T01:29:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:50.123+0000 7f8f10ac5700 1 -- 172.21.15.28:0/1692297601 shutdown_connections 2024-04-25T01:29:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:50.123+0000 7f8f10ac5700 1 --2- 172.21.15.28:0/1692297601 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8ef80a2ca0 0x7f8ef80a5160 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:50.123+0000 7f8f10ac5700 1 --2- 172.21.15.28:0/1692297601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8f0c107590 0x7f8f0c1988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:50.123+0000 7f8f10ac5700 1 --2- 172.21.15.28:0/1692297601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f0c06ac40 0x7f8f0c19e700 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:50.123+0000 7f8f10ac5700 1 --2- 172.21.15.28:0/1692297601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f0c06b5f0 0x7f8f0c19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:50.123+0000 7f8f10ac5700 1 -- 172.21.15.28:0/1692297601 >> 172.21.15.28:0/1692297601 conn(0x7f8f0c0781d0 msgr2=0x7f8f0c0ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:50.123+0000 7f8f10ac5700 1 -- 172.21.15.28:0/1692297601 shutdown_connections 2024-04-25T01:29:50.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:50.123+0000 7f8f10ac5700 1 -- 172.21.15.28:0/1692297601 wait complete. 2024-04-25T01:29:50.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:50 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1692297601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:29:50.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:50 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1692297601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:29:50.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:50 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1692297601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:29:50.745 INFO:tasks.cephadm:Adding mds.a on smithi028 2024-04-25T01:29:50.745 INFO:tasks.cephadm:Adding mds.d on smithi028 2024-04-25T01:29:50.745 INFO:tasks.cephadm:Adding mds.g on smithi028 2024-04-25T01:29:50.745 INFO:tasks.cephadm:Adding mds.j on smithi028 2024-04-25T01:29:50.745 INFO:tasks.cephadm:Adding mds.b on smithi032 2024-04-25T01:29:50.745 INFO:tasks.cephadm:Adding mds.e on smithi032 2024-04-25T01:29:50.745 INFO:tasks.cephadm:Adding mds.h on smithi032 2024-04-25T01:29:50.745 INFO:tasks.cephadm:Adding mds.k on smithi032 2024-04-25T01:29:50.745 INFO:tasks.cephadm:Adding mds.c on smithi130 2024-04-25T01:29:50.745 INFO:tasks.cephadm:Adding mds.f on smithi130 2024-04-25T01:29:50.745 INFO:tasks.cephadm:Adding mds.i on smithi130 2024-04-25T01:29:50.746 INFO:tasks.cephadm:Adding mds.l on smithi130 2024-04-25T01:29:50.746 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi032=b;smithi032=e;smithi032=h;smithi032=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l' 2024-04-25T01:29:50.992 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:29:51.440 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:51 smithi130 ceph-mon[32702]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:51.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:51 smithi032 ceph-mon[32461]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:51.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:51 smithi028 ceph-mon[29390]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:52.658 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.677+0000 7fd0ab6e4700 1 -- 172.21.15.130:0/712392419 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd0a4102d10 msgr2=0x7fd0a4105100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:52.658 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.677+0000 7fd0ab6e4700 1 --2- 172.21.15.130:0/712392419 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd0a4102d10 0x7fd0a4105100 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0009990 tx=0x7fd0a0009ca0 comp rx=0 tx=0).stop 2024-04-25T01:29:52.658 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.677+0000 7fd0ab6e4700 1 -- 172.21.15.130:0/712392419 shutdown_connections 2024-04-25T01:29:52.658 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.677+0000 7fd0ab6e4700 1 --2- 172.21.15.130:0/712392419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0a4107f70 0x7fd0a410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:52.658 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.677+0000 7fd0ab6e4700 1 --2- 172.21.15.130:0/712392419 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd0a4102d10 0x7fd0a4105100 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:52.658 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.678+0000 7fd0ab6e4700 1 --2- 172.21.15.130:0/712392419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a4105640 0x7fd0a4107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:52.658 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.678+0000 7fd0ab6e4700 1 -- 172.21.15.130:0/712392419 >> 172.21.15.130:0/712392419 conn(0x7fd0a40fc720 msgr2=0x7fd0a40feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:52.658 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.678+0000 7fd0ab6e4700 1 -- 172.21.15.130:0/712392419 shutdown_connections 2024-04-25T01:29:52.658 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.678+0000 7fd0ab6e4700 1 -- 172.21.15.130:0/712392419 wait complete. 2024-04-25T01:29:52.658 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.678+0000 7fd0ab6e4700 1 Processor -- start 2024-04-25T01:29:52.659 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.678+0000 7fd0ab6e4700 1 -- start start 2024-04-25T01:29:52.659 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.678+0000 7fd0ab6e4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0a4102d10 0x7fd0a4102670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:52.659 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.678+0000 7fd0ab6e4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a4105640 0x7fd0a4100cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:52.659 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.678+0000 7fd0ab6e4700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd0a4107f70 0x7fd0a4101200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:52.659 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.678+0000 7fd0ab6e4700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a410ded0 con 0x7fd0a4105640 2024-04-25T01:29:52.659 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.679+0000 7fd0ab6e4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a410dd50 con 0x7fd0a4102d10 2024-04-25T01:29:52.659 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.679+0000 7fd0ab6e4700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a410e050 con 0x7fd0a4107f70 2024-04-25T01:29:52.659 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.679+0000 7fd0a9c81700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd0a4107f70 0x7fd0a4101200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:52.659 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.679+0000 7fd0a9c81700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd0a4107f70 0x7fd0a4101200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.130:49162/0 (socket says 172.21.15.130:49162) 2024-04-25T01:29:52.659 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.679+0000 7fd0a9c81700 1 -- 172.21.15.130:0/3718885243 learned_addr learned my addr 172.21.15.130:0/3718885243 (peer_addr_for_me v2:172.21.15.130:0/0) 2024-04-25T01:29:52.659 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.679+0000 7fd0a8c7f700 1 --2- 172.21.15.130:0/3718885243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a4105640 0x7fd0a4100cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:52.659 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.679+0000 7fd0a9480700 1 --2- 172.21.15.130:0/3718885243 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0a4102d10 0x7fd0a4102670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:52.659 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.679+0000 7fd0a9c81700 1 -- 172.21.15.130:0/3718885243 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0a4102d10 msgr2=0x7fd0a4102670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:52.660 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.679+0000 7fd0a9c81700 1 --2- 172.21.15.130:0/3718885243 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0a4102d10 0x7fd0a4102670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:52.660 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.679+0000 7fd0a9c81700 1 -- 172.21.15.130:0/3718885243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a4105640 msgr2=0x7fd0a4100cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:52.660 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.679+0000 7fd0a9c81700 1 --2- 172.21.15.130:0/3718885243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a4105640 0x7fd0a4100cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:52.660 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.679+0000 7fd0a9c81700 1 -- 172.21.15.130:0/3718885243 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a4101ab0 con 0x7fd0a4107f70 2024-04-25T01:29:52.660 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.679+0000 7fd0a9c81700 1 --2- 172.21.15.130:0/3718885243 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd0a4107f70 0x7fd0a4101200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd09c00b9e0 tx=0x7fd09c00bcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:52.660 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.679+0000 7fd09a7fc700 1 -- 172.21.15.130:0/3718885243 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09c019070 con 0x7fd0a4107f70 2024-04-25T01:29:52.660 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.679+0000 7fd0ab6e4700 1 -- 172.21.15.130:0/3718885243 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0a41a2640 con 0x7fd0a4107f70 2024-04-25T01:29:52.660 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.679+0000 7fd09a7fc700 1 -- 172.21.15.130:0/3718885243 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd09c00ce10 con 0x7fd0a4107f70 2024-04-25T01:29:52.660 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.679+0000 7fd09a7fc700 1 -- 172.21.15.130:0/3718885243 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09c012360 con 0x7fd0a4107f70 2024-04-25T01:29:52.660 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.679+0000 7fd0ab6e4700 1 -- 172.21.15.130:0/3718885243 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0a41a2bd0 con 0x7fd0a4107f70 2024-04-25T01:29:52.661 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.680+0000 7fd0ab6e4700 1 -- 172.21.15.130:0/3718885243 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0a4068f20 con 0x7fd0a4107f70 2024-04-25T01:29:52.661 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.681+0000 7fd09a7fc700 1 -- 172.21.15.130:0/3718885243 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd09c01c070 con 0x7fd0a4107f70 2024-04-25T01:29:52.661 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.682+0000 7fd09a7fc700 1 --2- 172.21.15.130:0/3718885243 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd0900a2c20 0x7fd0900a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:52.661 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.682+0000 7fd09a7fc700 1 -- 172.21.15.130:0/3718885243 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fd09c0cdf30 con 0x7fd0a4107f70 2024-04-25T01:29:52.661 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.682+0000 7fd0a9480700 1 --2- 172.21.15.130:0/3718885243 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd0900a2c20 0x7fd0900a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:52.662 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.682+0000 7fd0a9480700 1 --2- 172.21.15.130:0/3718885243 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd0900a2c20 0x7fd0900a50e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd0a000b5c0 tx=0x7fd0a0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:52.664 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.686+0000 7fd09a7fc700 1 -- 172.21.15.130:0/3718885243 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd09c09b330 con 0x7fd0a4107f70 2024-04-25T01:29:52.876 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.897+0000 7fd0ab6e4700 1 -- 172.21.15.130:0/3718885243 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi032=b;smithi032=e;smithi032=h;smithi032=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l", "target": ["mon-mgr", ""]}) -- 0x7fd0a4062770 con 0x7fd0900a2c20 2024-04-25T01:29:52.884 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.906+0000 7fd09a7fc700 1 -- 172.21.15.130:0/3718885243 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fd0a4062770 con 0x7fd0900a2c20 2024-04-25T01:29:52.884 INFO:teuthology.orchestra.run.smithi130.stdout:Scheduled mds.all update... 2024-04-25T01:29:52.886 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.908+0000 7fd0ab6e4700 1 -- 172.21.15.130:0/3718885243 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd0900a2c20 msgr2=0x7fd0900a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:52.886 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.908+0000 7fd0ab6e4700 1 --2- 172.21.15.130:0/3718885243 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd0900a2c20 0x7fd0900a50e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd0a000b5c0 tx=0x7fd0a0005fb0 comp rx=0 tx=0).stop 2024-04-25T01:29:52.886 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.908+0000 7fd0ab6e4700 1 -- 172.21.15.130:0/3718885243 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd0a4107f70 msgr2=0x7fd0a4101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:52.886 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.908+0000 7fd0ab6e4700 1 --2- 172.21.15.130:0/3718885243 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd0a4107f70 0x7fd0a4101200 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd09c00b9e0 tx=0x7fd09c00bcf0 comp rx=0 tx=0).stop 2024-04-25T01:29:52.886 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.908+0000 7fd0ab6e4700 1 -- 172.21.15.130:0/3718885243 shutdown_connections 2024-04-25T01:29:52.887 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.908+0000 7fd0ab6e4700 1 --2- 172.21.15.130:0/3718885243 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd0900a2c20 0x7fd0900a50e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:52.887 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.908+0000 7fd0ab6e4700 1 --2- 172.21.15.130:0/3718885243 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd0a4107f70 0x7fd0a4101200 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:52.887 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.908+0000 7fd0ab6e4700 1 --2- 172.21.15.130:0/3718885243 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0a4102d10 0x7fd0a4102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:52.887 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.908+0000 7fd0ab6e4700 1 --2- 172.21.15.130:0/3718885243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a4105640 0x7fd0a4100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:52.887 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.908+0000 7fd0ab6e4700 1 -- 172.21.15.130:0/3718885243 >> 172.21.15.130:0/3718885243 conn(0x7fd0a40fc720 msgr2=0x7fd0a4103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:52.887 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.908+0000 7fd0ab6e4700 1 -- 172.21.15.130:0/3718885243 shutdown_connections 2024-04-25T01:29:52.887 INFO:teuthology.orchestra.run.smithi130.stderr:2024-04-25T01:29:52.908+0000 7fd0ab6e4700 1 -- 172.21.15.130:0/3718885243 wait complete. 2024-04-25T01:29:53.616 DEBUG:teuthology.orchestra.run.smithi028:mds.a> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.a.service 2024-04-25T01:29:53.620 DEBUG:teuthology.orchestra.run.smithi028:mds.d> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.d.service 2024-04-25T01:29:53.663 DEBUG:teuthology.orchestra.run.smithi028:mds.g> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.g.service 2024-04-25T01:29:53.706 DEBUG:teuthology.orchestra.run.smithi028:mds.j> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.j.service 2024-04-25T01:29:53.749 DEBUG:teuthology.orchestra.run.smithi032:mds.b> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.b.service 2024-04-25T01:29:53.752 DEBUG:teuthology.orchestra.run.smithi032:mds.e> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.e.service 2024-04-25T01:29:53.795 DEBUG:teuthology.orchestra.run.smithi032:mds.h> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.h.service 2024-04-25T01:29:53.838 DEBUG:teuthology.orchestra.run.smithi032:mds.k> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.k.service 2024-04-25T01:29:53.881 DEBUG:teuthology.orchestra.run.smithi130:mds.c> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.c.service 2024-04-25T01:29:53.884 DEBUG:teuthology.orchestra.run.smithi130:mds.f> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.f.service 2024-04-25T01:29:53.927 DEBUG:teuthology.orchestra.run.smithi130:mds.i> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.i.service 2024-04-25T01:29:53.971 DEBUG:teuthology.orchestra.run.smithi130:mds.l> sudo journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.l.service 2024-04-25T01:29:54.014 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-25T01:29:54.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:29:54.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:53 smithi028 ceph-mon[29390]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:54.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:53 smithi028 ceph-mon[29390]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi032=b;smithi032=e;smithi032=h;smithi032=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:29:54.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:53 smithi028 ceph-mon[29390]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi032=b;smithi032=e;smithi032=h;smithi032=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l;count:12 2024-04-25T01:29:54.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:54.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:54.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:54.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:54.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:54.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:29:54.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:29:54.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:53 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:54.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:53 smithi028 ceph-mon[29390]: Deploying daemon mds.l on smithi130 2024-04-25T01:29:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.160+0000 7f6dc579e640 1 -- 172.21.15.28:0/2209235909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc01015b0 msgr2=0x7f6dc01019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.160+0000 7f6dc579e640 1 --2- 172.21.15.28:0/2209235909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc01015b0 0x7f6dc01019d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6db0009900 tx=0x7f6db002a410 comp rx=0 tx=0).stop 2024-04-25T01:29:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.160+0000 7f6dc579e640 1 -- 172.21.15.28:0/2209235909 shutdown_connections 2024-04-25T01:29:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.160+0000 7f6dc579e640 1 --2- 172.21.15.28:0/2209235909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6dc0103160 0x7f6dc01099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.160+0000 7f6dc579e640 1 --2- 172.21.15.28:0/2209235909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6dc01027a0 0x7f6dc0102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.160+0000 7f6dc579e640 1 --2- 172.21.15.28:0/2209235909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc01015b0 0x7f6dc01019d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.160+0000 7f6dc579e640 1 -- 172.21.15.28:0/2209235909 >> 172.21.15.28:0/2209235909 conn(0x7f6dc00fcb20 msgr2=0x7f6dc00fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.160+0000 7f6dc579e640 1 -- 172.21.15.28:0/2209235909 shutdown_connections 2024-04-25T01:29:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.160+0000 7f6dc579e640 1 -- 172.21.15.28:0/2209235909 wait complete. 2024-04-25T01:29:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.160+0000 7f6dc579e640 1 Processor -- start 2024-04-25T01:29:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.161+0000 7f6dc579e640 1 -- start start 2024-04-25T01:29:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.161+0000 7f6dc579e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6dc01015b0 0x7f6dc01978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.161+0000 7f6dc579e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc01027a0 0x7f6dc0197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.161+0000 7f6dc579e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6dc0103160 0x7f6dc019ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.161+0000 7f6dc579e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc010d220 con 0x7f6dc01027a0 2024-04-25T01:29:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.161+0000 7f6dc579e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc010d0a0 con 0x7f6dc0103160 2024-04-25T01:29:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.161+0000 7f6dc579e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc010d3a0 con 0x7f6dc01015b0 2024-04-25T01:29:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.161+0000 7f6dbf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc01027a0 0x7f6dc0197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.161+0000 7f6dbf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc01027a0 0x7f6dc0197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34532/0 (socket says 172.21.15.28:34532) 2024-04-25T01:29:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.161+0000 7f6dbf7fe640 1 -- 172.21.15.28:0/2772723887 learned_addr learned my addr 172.21.15.28:0/2772723887 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:29:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.161+0000 7f6dbffff640 1 --2- 172.21.15.28:0/2772723887 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6dc01015b0 0x7f6dc01978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.161+0000 7f6dc4f9d640 1 --2- 172.21.15.28:0/2772723887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6dc0103160 0x7f6dc019ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.161+0000 7f6dbf7fe640 1 -- 172.21.15.28:0/2772723887 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6dc01015b0 msgr2=0x7f6dc01978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.161+0000 7f6dbf7fe640 1 --2- 172.21.15.28:0/2772723887 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6dc01015b0 0x7f6dc01978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.161+0000 7f6dbf7fe640 1 -- 172.21.15.28:0/2772723887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6dc0103160 msgr2=0x7f6dc019ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.161+0000 7f6dbf7fe640 1 --2- 172.21.15.28:0/2772723887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6dc0103160 0x7f6dc019ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.161+0000 7f6dbf7fe640 1 -- 172.21.15.28:0/2772723887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc019f4e0 con 0x7f6dc01027a0 2024-04-25T01:29:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.161+0000 7f6dbffff640 1 --2- 172.21.15.28:0/2772723887 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6dc01015b0 0x7f6dc01978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:29:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.162+0000 7f6dbf7fe640 1 --2- 172.21.15.28:0/2772723887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc01027a0 0x7f6dc0197e00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6db400e9f0 tx=0x7f6db400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.162+0000 7f6dbd7fa640 1 -- 172.21.15.28:0/2772723887 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db400cd80 con 0x7f6dc01027a0 2024-04-25T01:29:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.162+0000 7f6dc579e640 1 -- 172.21.15.28:0/2772723887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dc019f7d0 con 0x7f6dc01027a0 2024-04-25T01:29:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.162+0000 7f6dbd7fa640 1 -- 172.21.15.28:0/2772723887 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6db4010910 con 0x7f6dc01027a0 2024-04-25T01:29:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.162+0000 7f6dbd7fa640 1 -- 172.21.15.28:0/2772723887 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db40188f0 con 0x7f6dc01027a0 2024-04-25T01:29:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.162+0000 7f6dc579e640 1 -- 172.21.15.28:0/2772723887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dc019fd70 con 0x7f6dc01027a0 2024-04-25T01:29:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.163+0000 7f6dc579e640 1 -- 172.21.15.28:0/2772723887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dc01088d0 con 0x7f6dc01027a0 2024-04-25T01:29:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.164+0000 7f6dbd7fa640 1 -- 172.21.15.28:0/2772723887 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6db4010ac0 con 0x7f6dc01027a0 2024-04-25T01:29:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.165+0000 7f6dbd7fa640 1 --2- 172.21.15.28:0/2772723887 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6da80a2b90 0x7f6da80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.165+0000 7f6dbd7fa640 1 -- 172.21.15.28:0/2772723887 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f6db40cf3e0 con 0x7f6dc01027a0 2024-04-25T01:29:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.165+0000 7f6dbffff640 1 --2- 172.21.15.28:0/2772723887 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6da80a2b90 0x7f6da80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.165+0000 7f6dbffff640 1 --2- 172.21.15.28:0/2772723887 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6da80a2b90 0x7f6da80a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6db0005ab0 tx=0x7f6db0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:54.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.168+0000 7f6dbd7fa640 1 -- 172.21.15.28:0/2772723887 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6db409c7e0 con 0x7f6dc01027a0 2024-04-25T01:29:54.166 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:53 smithi130 ceph-mon[32702]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:54.166 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:53 smithi130 ceph-mon[32702]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi032=b;smithi032=e;smithi032=h;smithi032=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:29:54.167 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:53 smithi130 ceph-mon[32702]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi032=b;smithi032=e;smithi032=h;smithi032=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l;count:12 2024-04-25T01:29:54.167 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:54.167 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:54.167 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:54.167 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:54.167 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:54.167 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:29:54.167 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:29:54.167 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:53 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:54.167 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:53 smithi130 ceph-mon[32702]: Deploying daemon mds.l on smithi130 2024-04-25T01:29:54.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:53 smithi032 ceph-mon[32461]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:54.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:53 smithi032 ceph-mon[32461]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi032=b;smithi032=e;smithi032=h;smithi032=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:29:54.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:53 smithi032 ceph-mon[32461]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi032=b;smithi032=e;smithi032=h;smithi032=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l;count:12 2024-04-25T01:29:54.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:54.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:29:54.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:54.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:29:54.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:54.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:29:54.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:29:54.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:53 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:54.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:53 smithi032 ceph-mon[32461]: Deploying daemon mds.l on smithi130 2024-04-25T01:29:54.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.355+0000 7f6dc579e640 1 -- 172.21.15.28:0/2772723887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6dc002d370 con 0x7f6dc01027a0 2024-04-25T01:29:54.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.356+0000 7f6dbd7fa640 1 -- 172.21.15.28:0/2772723887 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18421 (secure 0 0 0) 0x7f6db409c0e0 con 0x7f6dc01027a0 2024-04-25T01:29:54.335 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:29:54.335 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":83,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:29:42.122155+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:29:54.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.358+0000 7f6dc579e640 1 -- 172.21.15.28:0/2772723887 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6da80a2b90 msgr2=0x7f6da80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:54.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.358+0000 7f6dc579e640 1 --2- 172.21.15.28:0/2772723887 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6da80a2b90 0x7f6da80a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6db0005ab0 tx=0x7f6db0000bc0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.358+0000 7f6dc579e640 1 -- 172.21.15.28:0/2772723887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc01027a0 msgr2=0x7f6dc0197e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:54.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.358+0000 7f6dc579e640 1 --2- 172.21.15.28:0/2772723887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc01027a0 0x7f6dc0197e00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6db400e9f0 tx=0x7f6db400eeb0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.358+0000 7f6dc579e640 1 -- 172.21.15.28:0/2772723887 shutdown_connections 2024-04-25T01:29:54.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.358+0000 7f6dc579e640 1 --2- 172.21.15.28:0/2772723887 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6da80a2b90 0x7f6da80a5050 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.358+0000 7f6dc579e640 1 --2- 172.21.15.28:0/2772723887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6dc0103160 0x7f6dc019ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.358+0000 7f6dc579e640 1 --2- 172.21.15.28:0/2772723887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dc01027a0 0x7f6dc0197e00 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.358+0000 7f6dc579e640 1 --2- 172.21.15.28:0/2772723887 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6dc01015b0 0x7f6dc01978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.358+0000 7f6dc579e640 1 -- 172.21.15.28:0/2772723887 >> 172.21.15.28:0/2772723887 conn(0x7f6dc00fcb20 msgr2=0x7f6dc0105aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:54.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.358+0000 7f6dc579e640 1 -- 172.21.15.28:0/2772723887 shutdown_connections 2024-04-25T01:29:54.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.358+0000 7f6dc579e640 1 -- 172.21.15.28:0/2772723887 wait complete. 2024-04-25T01:29:54.348 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:25:48.767414+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:29:54.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:29:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.474+0000 7ff138412640 1 -- 172.21.15.28:0/125841370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff130102230 msgr2=0x7ff130108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.474+0000 7ff138412640 1 --2- 172.21.15.28:0/125841370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff130102230 0x7ff130108ac0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff128009ca0 tx=0x7ff12802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.474+0000 7ff138412640 1 -- 172.21.15.28:0/125841370 shutdown_connections 2024-04-25T01:29:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.474+0000 7ff138412640 1 --2- 172.21.15.28:0/125841370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff130102230 0x7ff130108ac0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.474+0000 7ff138412640 1 --2- 172.21.15.28:0/125841370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff130101870 0x7ff130101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.474+0000 7ff138412640 1 --2- 172.21.15.28:0/125841370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff130100680 0x7ff130100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.474+0000 7ff138412640 1 -- 172.21.15.28:0/125841370 >> 172.21.15.28:0/125841370 conn(0x7ff1300fbbf0 msgr2=0x7ff1300fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.474+0000 7ff138412640 1 -- 172.21.15.28:0/125841370 shutdown_connections 2024-04-25T01:29:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.474+0000 7ff138412640 1 -- 172.21.15.28:0/125841370 wait complete. 2024-04-25T01:29:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.475+0000 7ff138412640 1 Processor -- start 2024-04-25T01:29:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.475+0000 7ff138412640 1 -- start start 2024-04-25T01:29:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.475+0000 7ff138412640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff130100680 0x7ff130069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.475+0000 7ff138412640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff130101870 0x7ff13019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.475+0000 7ff138412640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff130102230 0x7ff1301a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.475+0000 7ff138412640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff13010c300 con 0x7ff130102230 2024-04-25T01:29:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.475+0000 7ff138412640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff13010c180 con 0x7ff130101870 2024-04-25T01:29:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.475+0000 7ff138412640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff13010c480 con 0x7ff130100680 2024-04-25T01:29:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.475+0000 7ff1369af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff130102230 0x7ff1301a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.475+0000 7ff1369af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff130102230 0x7ff1301a1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34544/0 (socket says 172.21.15.28:34544) 2024-04-25T01:29:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.475+0000 7ff1369af640 1 -- 172.21.15.28:0/790834835 learned_addr learned my addr 172.21.15.28:0/790834835 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:29:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.475+0000 7ff1361ae640 1 --2- 172.21.15.28:0/790834835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff130100680 0x7ff130069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.475+0000 7ff1359ad640 1 --2- 172.21.15.28:0/790834835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff130101870 0x7ff13019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.476+0000 7ff1369af640 1 -- 172.21.15.28:0/790834835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff130100680 msgr2=0x7ff130069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.476+0000 7ff1369af640 1 --2- 172.21.15.28:0/790834835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff130100680 0x7ff130069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.476+0000 7ff1369af640 1 -- 172.21.15.28:0/790834835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff130101870 msgr2=0x7ff13019ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.476+0000 7ff1369af640 1 --2- 172.21.15.28:0/790834835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff130101870 0x7ff13019ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.476+0000 7ff1369af640 1 -- 172.21.15.28:0/790834835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1301a1910 con 0x7ff130102230 2024-04-25T01:29:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.476+0000 7ff1369af640 1 --2- 172.21.15.28:0/790834835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff130102230 0x7ff1301a1120 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff128031ae0 tx=0x7ff128031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.476+0000 7ff1277fe640 1 -- 172.21.15.28:0/790834835 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff128003c30 con 0x7ff130102230 2024-04-25T01:29:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.476+0000 7ff1277fe640 1 -- 172.21.15.28:0/790834835 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff128003dd0 con 0x7ff130102230 2024-04-25T01:29:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.476+0000 7ff138412640 1 -- 172.21.15.28:0/790834835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1301a1b40 con 0x7ff130102230 2024-04-25T01:29:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.476+0000 7ff1277fe640 1 -- 172.21.15.28:0/790834835 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12802fcb0 con 0x7ff130102230 2024-04-25T01:29:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.476+0000 7ff138412640 1 -- 172.21.15.28:0/790834835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1301a20e0 con 0x7ff130102230 2024-04-25T01:29:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.477+0000 7ff138412640 1 -- 172.21.15.28:0/790834835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff130107a70 con 0x7ff130102230 2024-04-25T01:29:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.478+0000 7ff1277fe640 1 -- 172.21.15.28:0/790834835 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff128004050 con 0x7ff130102230 2024-04-25T01:29:54.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.479+0000 7ff1277fe640 1 --2- 172.21.15.28:0/790834835 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff11c0a2c40 0x7ff11c0a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:54.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.479+0000 7ff1361ae640 1 --2- 172.21.15.28:0/790834835 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff11c0a2c40 0x7ff11c0a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:54.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.479+0000 7ff1277fe640 1 -- 172.21.15.28:0/790834835 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7ff1280f0010 con 0x7ff130102230 2024-04-25T01:29:54.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.479+0000 7ff1361ae640 1 --2- 172.21.15.28:0/790834835 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff11c0a2c40 0x7ff11c0a5100 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff1301016d0 tx=0x7ff12c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:54.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.482+0000 7ff1277fe640 1 -- 172.21.15.28:0/790834835 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff1280bd370 con 0x7ff130102230 2024-04-25T01:29:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.667+0000 7ff138412640 1 -- 172.21.15.28:0/790834835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff13002d170 con 0x7ff130102230 2024-04-25T01:29:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.667+0000 7ff1277fe640 1 -- 172.21.15.28:0/790834835 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7ff1280bcc70 con 0x7ff130102230 2024-04-25T01:29:54.646 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T01:29:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.669+0000 7ff138412640 1 -- 172.21.15.28:0/790834835 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff11c0a2c40 msgr2=0x7ff11c0a5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.669+0000 7ff138412640 1 --2- 172.21.15.28:0/790834835 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff11c0a2c40 0x7ff11c0a5100 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff1301016d0 tx=0x7ff12c00b390 comp rx=0 tx=0).stop 2024-04-25T01:29:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.669+0000 7ff138412640 1 -- 172.21.15.28:0/790834835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff130102230 msgr2=0x7ff1301a1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.669+0000 7ff138412640 1 --2- 172.21.15.28:0/790834835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff130102230 0x7ff1301a1120 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff128031ae0 tx=0x7ff128031ee0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.669+0000 7ff138412640 1 -- 172.21.15.28:0/790834835 shutdown_connections 2024-04-25T01:29:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.669+0000 7ff138412640 1 --2- 172.21.15.28:0/790834835 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff11c0a2c40 0x7ff11c0a5100 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.669+0000 7ff138412640 1 --2- 172.21.15.28:0/790834835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff130102230 0x7ff1301a1120 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.669+0000 7ff138412640 1 --2- 172.21.15.28:0/790834835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff130101870 0x7ff13019ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.669+0000 7ff138412640 1 --2- 172.21.15.28:0/790834835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff130100680 0x7ff130069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.669+0000 7ff138412640 1 -- 172.21.15.28:0/790834835 >> 172.21.15.28:0/790834835 conn(0x7ff1300fbbf0 msgr2=0x7ff130104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:54.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.669+0000 7ff138412640 1 -- 172.21.15.28:0/790834835 shutdown_connections 2024-04-25T01:29:54.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.669+0000 7ff138412640 1 -- 172.21.15.28:0/790834835 wait complete. 2024-04-25T01:29:54.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:29:54.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.782+0000 7fd4d15ed640 1 -- 172.21.15.28:0/4027726850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4cc101790 msgr2=0x7fd4cc101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.782+0000 7fd4d15ed640 1 --2- 172.21.15.28:0/4027726850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4cc101790 0x7fd4cc101bb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc009900 tx=0x7fd4bc02a410 comp rx=0 tx=0).stop 2024-04-25T01:29:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.782+0000 7fd4d15ed640 1 -- 172.21.15.28:0/4027726850 shutdown_connections 2024-04-25T01:29:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.782+0000 7fd4d15ed640 1 --2- 172.21.15.28:0/4027726850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd4cc103340 0x7fd4cc109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.782+0000 7fd4d15ed640 1 --2- 172.21.15.28:0/4027726850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4cc102980 0x7fd4cc102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.782+0000 7fd4d15ed640 1 --2- 172.21.15.28:0/4027726850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4cc101790 0x7fd4cc101bb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.782+0000 7fd4d15ed640 1 -- 172.21.15.28:0/4027726850 >> 172.21.15.28:0/4027726850 conn(0x7fd4cc0fcd20 msgr2=0x7fd4cc0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.782+0000 7fd4d15ed640 1 -- 172.21.15.28:0/4027726850 shutdown_connections 2024-04-25T01:29:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.782+0000 7fd4d15ed640 1 -- 172.21.15.28:0/4027726850 wait complete. 2024-04-25T01:29:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.782+0000 7fd4d15ed640 1 Processor -- start 2024-04-25T01:29:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.783+0000 7fd4d15ed640 1 -- start start 2024-04-25T01:29:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.783+0000 7fd4d15ed640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd4cc101790 0x7fd4cc197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.783+0000 7fd4d15ed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4cc102980 0x7fd4cc198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.783+0000 7fd4d15ed640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4cc103340 0x7fd4cc19f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.783+0000 7fd4d15ed640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd4cc10d480 con 0x7fd4cc102980 2024-04-25T01:29:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.783+0000 7fd4d15ed640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd4cc10d300 con 0x7fd4cc103340 2024-04-25T01:29:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.783+0000 7fd4d15ed640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd4cc10d600 con 0x7fd4cc101790 2024-04-25T01:29:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.783+0000 7fd4ca7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4cc102980 0x7fd4cc198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.783+0000 7fd4ca7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4cc102980 0x7fd4cc198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34558/0 (socket says 172.21.15.28:34558) 2024-04-25T01:29:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.783+0000 7fd4ca7fc640 1 -- 172.21.15.28:0/1876377653 learned_addr learned my addr 172.21.15.28:0/1876377653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:29:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.783+0000 7fd4cb7fe640 1 --2- 172.21.15.28:0/1876377653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4cc103340 0x7fd4cc19f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.783+0000 7fd4caffd640 1 --2- 172.21.15.28:0/1876377653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd4cc101790 0x7fd4cc197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.783+0000 7fd4ca7fc640 1 -- 172.21.15.28:0/1876377653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd4cc101790 msgr2=0x7fd4cc197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.783+0000 7fd4ca7fc640 1 --2- 172.21.15.28:0/1876377653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd4cc101790 0x7fd4cc197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.783+0000 7fd4ca7fc640 1 -- 172.21.15.28:0/1876377653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4cc103340 msgr2=0x7fd4cc19f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.783+0000 7fd4ca7fc640 1 --2- 172.21.15.28:0/1876377653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4cc103340 0x7fd4cc19f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.783+0000 7fd4ca7fc640 1 -- 172.21.15.28:0/1876377653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4cc19f770 con 0x7fd4cc102980 2024-04-25T01:29:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.783+0000 7fd4ca7fc640 1 --2- 172.21.15.28:0/1876377653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4cc102980 0x7fd4cc198090 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd4c000b4a0 tx=0x7fd4c000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.783+0000 7fd4b3fff640 1 -- 172.21.15.28:0/1876377653 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0014070 con 0x7fd4cc102980 2024-04-25T01:29:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.784+0000 7fd4b3fff640 1 -- 172.21.15.28:0/1876377653 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4c0010af0 con 0x7fd4cc102980 2024-04-25T01:29:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.784+0000 7fd4d15ed640 1 -- 172.21.15.28:0/1876377653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4cc19fa60 con 0x7fd4cc102980 2024-04-25T01:29:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.784+0000 7fd4b3fff640 1 -- 172.21.15.28:0/1876377653 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0017d10 con 0x7fd4cc102980 2024-04-25T01:29:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.784+0000 7fd4d15ed640 1 -- 172.21.15.28:0/1876377653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4cc19ffb0 con 0x7fd4cc102980 2024-04-25T01:29:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.785+0000 7fd4d15ed640 1 -- 172.21.15.28:0/1876377653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4cc108b30 con 0x7fd4cc102980 2024-04-25T01:29:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.786+0000 7fd4b3fff640 1 -- 172.21.15.28:0/1876377653 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd4c0017070 con 0x7fd4cc102980 2024-04-25T01:29:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.786+0000 7fd4b3fff640 1 --2- 172.21.15.28:0/1876377653 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd4b40a2b90 0x7fd4b40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.787+0000 7fd4caffd640 1 --2- 172.21.15.28:0/1876377653 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd4b40a2b90 0x7fd4b40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.787+0000 7fd4b3fff640 1 -- 172.21.15.28:0/1876377653 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fd4c00cdf70 con 0x7fd4cc102980 2024-04-25T01:29:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.787+0000 7fd4caffd640 1 --2- 172.21.15.28:0/1876377653 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd4b40a2b90 0x7fd4b40a5050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc006010 tx=0x7fd4bc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:54.791+0000 7fd4b3fff640 1 -- 172.21.15.28:0/1876377653 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd4c009b370 con 0x7fd4cc102980 2024-04-25T01:29:54.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:55.014+0000 7fd4d15ed640 1 -- 172.21.15.28:0/1876377653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4cc02d380 con 0x7fd4cc102980 2024-04-25T01:29:54.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:55.014+0000 7fd4b3fff640 1 -- 172.21.15.28:0/1876377653 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fd4c009ac70 con 0x7fd4cc102980 2024-04-25T01:29:54.993 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:29:54.993 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-25T01:29:54.993 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 1 2024-04-25T01:29:54.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:55.016+0000 7fd4d15ed640 1 -- 172.21.15.28:0/1876377653 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd4b40a2b90 msgr2=0x7fd4b40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:55.016+0000 7fd4d15ed640 1 --2- 172.21.15.28:0/1876377653 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd4b40a2b90 0x7fd4b40a5050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc006010 tx=0x7fd4bc000bc0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:55.016+0000 7fd4d15ed640 1 -- 172.21.15.28:0/1876377653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4cc102980 msgr2=0x7fd4cc198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:55.016+0000 7fd4d15ed640 1 --2- 172.21.15.28:0/1876377653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4cc102980 0x7fd4cc198090 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd4c000b4a0 tx=0x7fd4c000b960 comp rx=0 tx=0).stop 2024-04-25T01:29:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:55.016+0000 7fd4d15ed640 1 -- 172.21.15.28:0/1876377653 shutdown_connections 2024-04-25T01:29:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:55.016+0000 7fd4d15ed640 1 --2- 172.21.15.28:0/1876377653 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd4b40a2b90 0x7fd4b40a5050 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:55.016+0000 7fd4d15ed640 1 --2- 172.21.15.28:0/1876377653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd4cc103340 0x7fd4cc19f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:55.016+0000 7fd4d15ed640 1 --2- 172.21.15.28:0/1876377653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4cc102980 0x7fd4cc198090 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:55.016+0000 7fd4d15ed640 1 --2- 172.21.15.28:0/1876377653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd4cc101790 0x7fd4cc197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:55.016+0000 7fd4d15ed640 1 -- 172.21.15.28:0/1876377653 >> 172.21.15.28:0/1876377653 conn(0x7fd4cc0fcd20 msgr2=0x7fd4cc105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:55.016+0000 7fd4d15ed640 1 -- 172.21.15.28:0/1876377653 shutdown_connections 2024-04-25T01:29:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:55.016+0000 7fd4d15ed640 1 -- 172.21.15.28:0/1876377653 wait complete. 2024-04-25T01:29:55.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:54 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2772723887' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:29:55.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:54 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/790834835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:29:55.148 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:54 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2772723887' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:29:55.148 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:54 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/790834835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:29:55.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:54 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2772723887' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:29:55.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:54 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/790834835' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:29:55.459 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 25 01:29:55 smithi130 systemd[1]: Starting Ceph mds.l for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:29:55.827 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 25 01:29:55 smithi130 podman[62120]: 2024-04-25 01:29:55.465491377 +0000 UTC m=+0.018168333 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:29:55.827 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 25 01:29:55 smithi130 podman[62120]: 2024-04-25 01:29:55.596786444 +0000 UTC m=+0.149463442 container create fac764c82e12f5bb0ab6f4fc228da4e249d3375c6712607d92fb1b72b1e1201e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-l, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0) 2024-04-25T01:29:55.827 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 25 01:29:55 smithi130 podman[62120]: 2024-04-25 01:29:55.8495387 +0000 UTC m=+0.402215656 container init fac764c82e12f5bb0ab6f4fc228da4e249d3375c6712607d92fb1b72b1e1201e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-l, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-25T01:29:56.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:55 smithi130 ceph-mon[32702]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:56.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:55 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1876377653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:29:56.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:55 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.130:6832/1080169582,v1:172.21.15.130:6833/1080169582] up:boot 2024-04-25T01:29:56.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:55 smithi130 ceph-mon[32702]: fsmap 1 up:standby 2024-04-25T01:29:56.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:55 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:29:56.210 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 25 01:29:55 smithi130 podman[62120]: 2024-04-25 01:29:55.852902455 +0000 UTC m=+0.405579418 container start fac764c82e12f5bb0ab6f4fc228da4e249d3375c6712607d92fb1b72b1e1201e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T01:29:56.210 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 25 01:29:55 smithi130 ceph-mds[62139]: -- 172.21.15.130:0/3521828512 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ae58dc5a40 con 0x55ae58d98c00 2024-04-25T01:29:56.210 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 25 01:29:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-l[62135]: starting mds.l at 2024-04-25T01:29:56.210 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 25 01:29:55 smithi130 bash[62120]: fac764c82e12f5bb0ab6f4fc228da4e249d3375c6712607d92fb1b72b1e1201e 2024-04-25T01:29:56.210 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 25 01:29:55 smithi130 systemd[1]: Started Ceph mds.l for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:29:56.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:55 smithi032 ceph-mon[32461]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:56.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:55 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1876377653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:29:56.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:55 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.130:6832/1080169582,v1:172.21.15.130:6833/1080169582] up:boot 2024-04-25T01:29:56.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:55 smithi032 ceph-mon[32461]: fsmap 1 up:standby 2024-04-25T01:29:56.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:55 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:29:56.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:55 smithi028 ceph-mon[29390]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:56.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:55 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1876377653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:29:56.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:55 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.130:6832/1080169582,v1:172.21.15.130:6833/1080169582] up:boot 2024-04-25T01:29:56.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:55 smithi028 ceph-mon[29390]: fsmap 1 up:standby 2024-04-25T01:29:56.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:55 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:29:57.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:29:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.143+0000 7f9b1884c640 1 -- 172.21.15.28:0/375436033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b14107540 msgr2=0x7f9b14109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.143+0000 7f9b1884c640 1 --2- 172.21.15.28:0/375436033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b14107540 0x7f9b14109930 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c009ca0 tx=0x7f9b0c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:29:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.143+0000 7f9b1884c640 1 -- 172.21.15.28:0/375436033 shutdown_connections 2024-04-25T01:29:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.143+0000 7f9b1884c640 1 --2- 172.21.15.28:0/375436033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b14107540 0x7f9b14109930 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.143+0000 7f9b1884c640 1 --2- 172.21.15.28:0/375436033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9b14104c10 0x7f9b14107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.143+0000 7f9b1884c640 1 --2- 172.21.15.28:0/375436033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b141022e0 0x7f9b141046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.143+0000 7f9b1884c640 1 -- 172.21.15.28:0/375436033 >> 172.21.15.28:0/375436033 conn(0x7f9b140fbc30 msgr2=0x7f9b140fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.143+0000 7f9b1884c640 1 -- 172.21.15.28:0/375436033 shutdown_connections 2024-04-25T01:29:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.143+0000 7f9b1884c640 1 -- 172.21.15.28:0/375436033 wait complete. 2024-04-25T01:29:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.144+0000 7f9b1884c640 1 Processor -- start 2024-04-25T01:29:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.144+0000 7f9b1884c640 1 -- start start 2024-04-25T01:29:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.144+0000 7f9b1884c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b141022e0 0x7f9b141979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:57.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.144+0000 7f9b1884c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9b14104c10 0x7f9b14197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.144+0000 7f9b1884c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b14107540 0x7f9b1419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.144+0000 7f9b1884c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1410d450 con 0x7f9b14107540 2024-04-25T01:29:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.144+0000 7f9b1884c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1410d2d0 con 0x7f9b141022e0 2024-04-25T01:29:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.144+0000 7f9b1884c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1410d5d0 con 0x7f9b14104c10 2024-04-25T01:29:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.144+0000 7f9b12d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b14107540 0x7f9b1419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.144+0000 7f9b12d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b14107540 0x7f9b1419ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36710/0 (socket says 172.21.15.28:36710) 2024-04-25T01:29:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.144+0000 7f9b12d9d640 1 -- 172.21.15.28:0/739830233 learned_addr learned my addr 172.21.15.28:0/739830233 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:29:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.144+0000 7f9b0bfff640 1 --2- 172.21.15.28:0/739830233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9b14104c10 0x7f9b14197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.144+0000 7f9b1259c640 1 --2- 172.21.15.28:0/739830233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b141022e0 0x7f9b141979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.144+0000 7f9b12d9d640 1 -- 172.21.15.28:0/739830233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9b14104c10 msgr2=0x7f9b14197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.144+0000 7f9b12d9d640 1 --2- 172.21.15.28:0/739830233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9b14104c10 0x7f9b14197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.144+0000 7f9b12d9d640 1 -- 172.21.15.28:0/739830233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b141022e0 msgr2=0x7f9b141979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.144+0000 7f9b12d9d640 1 --2- 172.21.15.28:0/739830233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b141022e0 0x7f9b141979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.144+0000 7f9b12d9d640 1 -- 172.21.15.28:0/739830233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b1419f600 con 0x7f9b14107540 2024-04-25T01:29:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.145+0000 7f9b12d9d640 1 --2- 172.21.15.28:0/739830233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b14107540 0x7f9b1419ef90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c0098d0 tx=0x7f9b0c02f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.145+0000 7f9b0b7fe640 1 -- 172.21.15.28:0/739830233 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c02fc30 con 0x7f9b14107540 2024-04-25T01:29:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.145+0000 7f9b1884c640 1 -- 172.21.15.28:0/739830233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b1419f890 con 0x7f9b14107540 2024-04-25T01:29:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.145+0000 7f9b0b7fe640 1 -- 172.21.15.28:0/739830233 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b0c003e50 con 0x7f9b14107540 2024-04-25T01:29:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.145+0000 7f9b0b7fe640 1 -- 172.21.15.28:0/739830233 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c004810 con 0x7f9b14107540 2024-04-25T01:29:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.145+0000 7f9b1884c640 1 -- 172.21.15.28:0/739830233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b1419fe30 con 0x7f9b14107540 2024-04-25T01:29:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.146+0000 7f9b1884c640 1 -- 172.21.15.28:0/739830233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b140fd720 con 0x7f9b14107540 2024-04-25T01:29:57.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.147+0000 7f9b0b7fe640 1 -- 172.21.15.28:0/739830233 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9b0c003970 con 0x7f9b14107540 2024-04-25T01:29:57.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.148+0000 7f9b0b7fe640 1 --2- 172.21.15.28:0/739830233 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9b000a2b90 0x7f9b000a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:57.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.148+0000 7f9b1259c640 1 --2- 172.21.15.28:0/739830233 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9b000a2b90 0x7f9b000a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:57.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.148+0000 7f9b0b7fe640 1 -- 172.21.15.28:0/739830233 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f9b0c0eff90 con 0x7f9b14107540 2024-04-25T01:29:57.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.148+0000 7f9b1259c640 1 --2- 172.21.15.28:0/739830233 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9b000a2b90 0x7f9b000a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9b0400ab40 tx=0x7f9b0400a250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:57.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.153+0000 7f9b0b7fe640 1 -- 172.21.15.28:0/739830233 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9b0c0bd410 con 0x7f9b14107540 2024-04-25T01:29:57.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:56 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:57.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:56 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:57.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:56 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:57.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:56 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:29:57.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:56 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:29:57.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:56 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:57.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:56 smithi130 ceph-mon[32702]: Deploying daemon mds.k on smithi032 2024-04-25T01:29:57.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:56 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:57.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:56 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:57.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:56 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:57.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:56 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:29:57.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:56 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:29:57.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:56 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:57.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:56 smithi032 ceph-mon[32461]: Deploying daemon mds.k on smithi032 2024-04-25T01:29:57.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:56 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:57.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:56 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:57.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:56 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:29:57.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:56 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:29:57.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:56 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:29:57.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:56 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:29:57.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:56 smithi028 ceph-mon[29390]: Deploying daemon mds.k on smithi032 2024-04-25T01:29:57.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.390+0000 7f9b1884c640 1 -- 172.21.15.28:0/739830233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b14062340 con 0x7f9b14107540 2024-04-25T01:29:57.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.390+0000 7f9b0b7fe640 1 -- 172.21.15.28:0/739830233 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f9b0c0bcd10 con 0x7f9b14107540 2024-04-25T01:29:57.369 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:29:57.369 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:29:57.369 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 2 2024-04-25T01:29:57.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.392+0000 7f9b1884c640 1 -- 172.21.15.28:0/739830233 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9b000a2b90 msgr2=0x7f9b000a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:57.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.392+0000 7f9b1884c640 1 --2- 172.21.15.28:0/739830233 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9b000a2b90 0x7f9b000a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9b0400ab40 tx=0x7f9b0400a250 comp rx=0 tx=0).stop 2024-04-25T01:29:57.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.392+0000 7f9b1884c640 1 -- 172.21.15.28:0/739830233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b14107540 msgr2=0x7f9b1419ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:57.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.392+0000 7f9b1884c640 1 --2- 172.21.15.28:0/739830233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b14107540 0x7f9b1419ef90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c0098d0 tx=0x7f9b0c02f840 comp rx=0 tx=0).stop 2024-04-25T01:29:57.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.392+0000 7f9b1884c640 1 -- 172.21.15.28:0/739830233 shutdown_connections 2024-04-25T01:29:57.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.392+0000 7f9b1884c640 1 --2- 172.21.15.28:0/739830233 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9b000a2b90 0x7f9b000a5050 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:57.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.392+0000 7f9b1884c640 1 --2- 172.21.15.28:0/739830233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b14107540 0x7f9b1419ef90 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:57.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.392+0000 7f9b1884c640 1 --2- 172.21.15.28:0/739830233 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9b14104c10 0x7f9b14197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:57.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.392+0000 7f9b1884c640 1 --2- 172.21.15.28:0/739830233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9b141022e0 0x7f9b141979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:57.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.392+0000 7f9b1884c640 1 -- 172.21.15.28:0/739830233 >> 172.21.15.28:0/739830233 conn(0x7f9b140fbc30 msgr2=0x7f9b14102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:57.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.392+0000 7f9b1884c640 1 -- 172.21.15.28:0/739830233 shutdown_connections 2024-04-25T01:29:57.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:57.392+0000 7f9b1884c640 1 -- 172.21.15.28:0/739830233 wait complete. 2024-04-25T01:29:58.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:57 smithi032 ceph-mon[32461]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:58.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:57 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/739830233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:29:58.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:57 smithi028 ceph-mon[29390]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:58.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:57 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/739830233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:29:58.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:57 smithi130 ceph-mon[32702]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:29:58.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:57 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/739830233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:29:58.484 INFO:journalctl@ceph.mds.k.smithi032.stdout:Apr 25 01:29:58 smithi032 podman[62200]: 2024-04-25 01:29:58.378406588 +0000 UTC m=+0.018459122 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:29:58.796 INFO:journalctl@ceph.mds.k.smithi032.stdout:Apr 25 01:29:58 smithi032 podman[62200]: 2024-04-25 01:29:58.507932924 +0000 UTC m=+0.147985429 container create 68adf427738ec563503f3ee9a580eb66e97e00032bf55239d5fc1b42da854137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-25T01:29:58.797 INFO:journalctl@ceph.mds.k.smithi032.stdout:Apr 25 01:29:58 smithi032 podman[62200]: 2024-04-25 01:29:58.727603597 +0000 UTC m=+0.367656104 container init 68adf427738ec563503f3ee9a580eb66e97e00032bf55239d5fc1b42da854137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-25T01:29:58.797 INFO:journalctl@ceph.mds.k.smithi032.stdout:Apr 25 01:29:58 smithi032 podman[62200]: 2024-04-25 01:29:58.7303066 +0000 UTC m=+0.370359118 container start 68adf427738ec563503f3ee9a580eb66e97e00032bf55239d5fc1b42da854137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-k, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-25T01:29:58.797 INFO:journalctl@ceph.mds.k.smithi032.stdout:Apr 25 01:29:58 smithi032 ceph-mds[62219]: -- 172.21.15.32:0/868587041 <== mon.2 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b874c9fa40 con 0x55b874c72c00 2024-04-25T01:29:58.797 INFO:journalctl@ceph.mds.k.smithi032.stdout:Apr 25 01:29:58 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-k[62215]: starting mds.k at 2024-04-25T01:29:58.797 INFO:journalctl@ceph.mds.k.smithi032.stdout:Apr 25 01:29:58 smithi032 bash[62200]: 68adf427738ec563503f3ee9a580eb66e97e00032bf55239d5fc1b42da854137 2024-04-25T01:29:59.234 INFO:journalctl@ceph.mds.k.smithi032.stdout:Apr 25 01:29:58 smithi032 systemd[1]: Started Ceph mds.k for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:29:59.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:29:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.515+0000 7fb7cf948640 1 -- 172.21.15.28:0/2390642374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c81074e0 msgr2=0x7fb7c81098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.515+0000 7fb7cf948640 1 --2- 172.21.15.28:0/2390642374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c81074e0 0x7fb7c81098d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0009ca0 tx=0x7fb7c002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:29:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.515+0000 7fb7cf948640 1 -- 172.21.15.28:0/2390642374 shutdown_connections 2024-04-25T01:29:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.515+0000 7fb7cf948640 1 --2- 172.21.15.28:0/2390642374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c81074e0 0x7fb7c81098d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.515+0000 7fb7cf948640 1 --2- 172.21.15.28:0/2390642374 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8104bb0 0x7fb7c8106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.515+0000 7fb7cf948640 1 --2- 172.21.15.28:0/2390642374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c8102280 0x7fb7c8104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.515+0000 7fb7cf948640 1 -- 172.21.15.28:0/2390642374 >> 172.21.15.28:0/2390642374 conn(0x7fb7c80fbc30 msgr2=0x7fb7c80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.515+0000 7fb7cf948640 1 -- 172.21.15.28:0/2390642374 shutdown_connections 2024-04-25T01:29:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.515+0000 7fb7cf948640 1 -- 172.21.15.28:0/2390642374 wait complete. 2024-04-25T01:29:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.516+0000 7fb7cf948640 1 Processor -- start 2024-04-25T01:29:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.516+0000 7fb7cf948640 1 -- start start 2024-04-25T01:29:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.516+0000 7fb7cf948640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c8102280 0x7fb7c81979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.516+0000 7fb7cf948640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8104bb0 0x7fb7c8197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.516+0000 7fb7cf948640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c81074e0 0x7fb7c819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.516+0000 7fb7cf948640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c810d420 con 0x7fb7c81074e0 2024-04-25T01:29:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.516+0000 7fb7cf948640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c810d2a0 con 0x7fb7c8102280 2024-04-25T01:29:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.516+0000 7fb7cf948640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c810d5a0 con 0x7fb7c8104bb0 2024-04-25T01:29:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.516+0000 7fb7cdee5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c81074e0 0x7fb7c819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.516+0000 7fb7cdee5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c81074e0 0x7fb7c819ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36730/0 (socket says 172.21.15.28:36730) 2024-04-25T01:29:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.516+0000 7fb7cdee5640 1 -- 172.21.15.28:0/3236427897 learned_addr learned my addr 172.21.15.28:0/3236427897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:29:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.516+0000 7fb7cd6e4640 1 --2- 172.21.15.28:0/3236427897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c8102280 0x7fb7c81979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.516+0000 7fb7ccee3640 1 --2- 172.21.15.28:0/3236427897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8104bb0 0x7fb7c8197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.516+0000 7fb7cdee5640 1 -- 172.21.15.28:0/3236427897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8104bb0 msgr2=0x7fb7c8197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.516+0000 7fb7cdee5640 1 --2- 172.21.15.28:0/3236427897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8104bb0 0x7fb7c8197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.516+0000 7fb7cdee5640 1 -- 172.21.15.28:0/3236427897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c8102280 msgr2=0x7fb7c81979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.516+0000 7fb7cdee5640 1 --2- 172.21.15.28:0/3236427897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c8102280 0x7fb7c81979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.516+0000 7fb7cdee5640 1 -- 172.21.15.28:0/3236427897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c819f5d0 con 0x7fb7c81074e0 2024-04-25T01:29:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.516+0000 7fb7ccee3640 1 --2- 172.21.15.28:0/3236427897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8104bb0 0x7fb7c8197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:29:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.517+0000 7fb7cdee5640 1 --2- 172.21.15.28:0/3236427897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c81074e0 0x7fb7c819ef60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0005e30 tx=0x7fb7c0031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.517+0000 7fb7be7fc640 1 -- 172.21.15.28:0/3236427897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0003c30 con 0x7fb7c81074e0 2024-04-25T01:29:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.517+0000 7fb7cf948640 1 -- 172.21.15.28:0/3236427897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7c819f860 con 0x7fb7c81074e0 2024-04-25T01:29:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.517+0000 7fb7be7fc640 1 -- 172.21.15.28:0/3236427897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7c0003dd0 con 0x7fb7c81074e0 2024-04-25T01:29:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.517+0000 7fb7be7fc640 1 -- 172.21.15.28:0/3236427897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c002fde0 con 0x7fb7c81074e0 2024-04-25T01:29:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.517+0000 7fb7cf948640 1 -- 172.21.15.28:0/3236427897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7c819fdb0 con 0x7fb7c81074e0 2024-04-25T01:29:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.518+0000 7fb7cf948640 1 -- 172.21.15.28:0/3236427897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7c80fd720 con 0x7fb7c81074e0 2024-04-25T01:29:59.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.519+0000 7fb7be7fc640 1 -- 172.21.15.28:0/3236427897 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb7c0004050 con 0x7fb7c81074e0 2024-04-25T01:29:59.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.519+0000 7fb7be7fc640 1 --2- 172.21.15.28:0/3236427897 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb7b40a2b90 0x7fb7b40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:29:59.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.519+0000 7fb7cd6e4640 1 --2- 172.21.15.28:0/3236427897 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb7b40a2b90 0x7fb7b40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:29:59.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.519+0000 7fb7be7fc640 1 -- 172.21.15.28:0/3236427897 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fb7c00effc0 con 0x7fb7c81074e0 2024-04-25T01:29:59.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.520+0000 7fb7cd6e4640 1 --2- 172.21.15.28:0/3236427897 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb7b40a2b90 0x7fb7b40a5050 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb7c400bbe0 tx=0x7fb7c400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:29:59.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.524+0000 7fb7be7fc640 1 -- 172.21.15.28:0/3236427897 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb7c00bd3c0 con 0x7fb7c81074e0 2024-04-25T01:29:59.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.749+0000 7fb7cf948640 1 -- 172.21.15.28:0/3236427897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7c802d170 con 0x7fb7c81074e0 2024-04-25T01:29:59.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.749+0000 7fb7be7fc640 1 -- 172.21.15.28:0/3236427897 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7fb7c00bccc0 con 0x7fb7c81074e0 2024-04-25T01:29:59.728 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:29:59.728 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:29:59.728 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 3 2024-04-25T01:29:59.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.751+0000 7fb7cf948640 1 -- 172.21.15.28:0/3236427897 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb7b40a2b90 msgr2=0x7fb7b40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:59.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.751+0000 7fb7cf948640 1 --2- 172.21.15.28:0/3236427897 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb7b40a2b90 0x7fb7b40a5050 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb7c400bbe0 tx=0x7fb7c400b390 comp rx=0 tx=0).stop 2024-04-25T01:29:59.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.751+0000 7fb7cf948640 1 -- 172.21.15.28:0/3236427897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c81074e0 msgr2=0x7fb7c819ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:29:59.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.751+0000 7fb7cf948640 1 --2- 172.21.15.28:0/3236427897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c81074e0 0x7fb7c819ef60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0005e30 tx=0x7fb7c0031ee0 comp rx=0 tx=0).stop 2024-04-25T01:29:59.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.751+0000 7fb7cf948640 1 -- 172.21.15.28:0/3236427897 shutdown_connections 2024-04-25T01:29:59.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.751+0000 7fb7cf948640 1 --2- 172.21.15.28:0/3236427897 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb7b40a2b90 0x7fb7b40a5050 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:59.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.751+0000 7fb7cf948640 1 --2- 172.21.15.28:0/3236427897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c81074e0 0x7fb7c819ef60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:59.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.751+0000 7fb7cf948640 1 --2- 172.21.15.28:0/3236427897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8104bb0 0x7fb7c8197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:59.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.751+0000 7fb7cf948640 1 --2- 172.21.15.28:0/3236427897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c8102280 0x7fb7c81979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:29:59.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.751+0000 7fb7cf948640 1 -- 172.21.15.28:0/3236427897 >> 172.21.15.28:0/3236427897 conn(0x7fb7c80fbc30 msgr2=0x7fb7c8102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:29:59.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.751+0000 7fb7cf948640 1 -- 172.21.15.28:0/3236427897 shutdown_connections 2024-04-25T01:29:59.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:29:59.751+0000 7fb7cf948640 1 -- 172.21.15.28:0/3236427897 wait complete. 2024-04-25T01:30:00.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:59 smithi028 ceph-mon[29390]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:00.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:59 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:00.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:59 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:00.010 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:59 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:00.010 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:59 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:00.010 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:59 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:00.010 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:59 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:00.010 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:59 smithi028 ceph-mon[29390]: Deploying daemon mds.j on smithi028 2024-04-25T01:30:00.010 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:59 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.32:6832/983600975,v1:172.21.15.32:6833/983600975] up:boot 2024-04-25T01:30:00.010 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:59 smithi028 ceph-mon[29390]: fsmap 2 up:standby 2024-04-25T01:30:00.010 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:59 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:30:00.010 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:29:59 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3236427897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:00.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:59 smithi130 ceph-mon[32702]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:00.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:59 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:00.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:59 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:00.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:59 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:00.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:59 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:00.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:59 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:00.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:59 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:00.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:59 smithi130 ceph-mon[32702]: Deploying daemon mds.j on smithi028 2024-04-25T01:30:00.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:59 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.32:6832/983600975,v1:172.21.15.32:6833/983600975] up:boot 2024-04-25T01:30:00.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:59 smithi130 ceph-mon[32702]: fsmap 2 up:standby 2024-04-25T01:30:00.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:59 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:30:00.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:29:59 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3236427897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:00.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:59 smithi032 ceph-mon[32461]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:00.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:59 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:00.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:59 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:00.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:59 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:00.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:59 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:00.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:59 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:00.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:59 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:00.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:59 smithi032 ceph-mon[32461]: Deploying daemon mds.j on smithi028 2024-04-25T01:30:00.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:59 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.32:6832/983600975,v1:172.21.15.32:6833/983600975] up:boot 2024-04-25T01:30:00.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:59 smithi032 ceph-mon[32461]: fsmap 2 up:standby 2024-04-25T01:30:00.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:59 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:30:00.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:29:59 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3236427897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:00.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:00 smithi032 ceph-mon[32461]: overall HEALTH_OK 2024-04-25T01:30:01.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:00 smithi028 ceph-mon[29390]: overall HEALTH_OK 2024-04-25T01:30:01.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:00 smithi130 ceph-mon[32702]: overall HEALTH_OK 2024-04-25T01:30:01.542 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 25 01:30:01 smithi028 systemd[1]: Starting Ceph mds.j for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:30:01.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:30:01.803 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 25 01:30:01 smithi028 podman[67900]: 2024-04-25 01:30:01.564258609 +0000 UTC m=+0.110374451 container create a369550f8aceb873d2f0ac2eee67efdc98b7943506460e0259f02cdfc4ac8ba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-25T01:30:01.803 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 25 01:30:01 smithi028 podman[67900]: 2024-04-25 01:30:01.472631612 +0000 UTC m=+0.018747468 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:30:01.803 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 25 01:30:01 smithi028 podman[67900]: 2024-04-25 01:30:01.825892358 +0000 UTC m=+0.372008227 container init a369550f8aceb873d2f0ac2eee67efdc98b7943506460e0259f02cdfc4ac8ba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-25T01:30:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.872+0000 7fd6c63b7640 1 -- 172.21.15.28:0/3452187672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c0102230 msgr2=0x7fd6c0108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.872+0000 7fd6c63b7640 1 --2- 172.21.15.28:0/3452187672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c0102230 0x7fd6c0108ac0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8009ca0 tx=0x7fd6b802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:30:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.872+0000 7fd6c63b7640 1 -- 172.21.15.28:0/3452187672 shutdown_connections 2024-04-25T01:30:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.872+0000 7fd6c63b7640 1 --2- 172.21.15.28:0/3452187672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c0102230 0x7fd6c0108ac0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.872+0000 7fd6c63b7640 1 --2- 172.21.15.28:0/3452187672 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6c0101870 0x7fd6c0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.872+0000 7fd6c63b7640 1 --2- 172.21.15.28:0/3452187672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6c0100680 0x7fd6c0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.872+0000 7fd6c63b7640 1 -- 172.21.15.28:0/3452187672 >> 172.21.15.28:0/3452187672 conn(0x7fd6c00fbbf0 msgr2=0x7fd6c00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.872+0000 7fd6c63b7640 1 -- 172.21.15.28:0/3452187672 shutdown_connections 2024-04-25T01:30:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.872+0000 7fd6c63b7640 1 -- 172.21.15.28:0/3452187672 wait complete. 2024-04-25T01:30:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.872+0000 7fd6c63b7640 1 Processor -- start 2024-04-25T01:30:01.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6c63b7640 1 -- start start 2024-04-25T01:30:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6c63b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c0100680 0x7fd6c0196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6c63b7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6c0101870 0x7fd6c0196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6c63b7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6c0102230 0x7fd6c019def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6c63b7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c010c300 con 0x7fd6c0100680 2024-04-25T01:30:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6c63b7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c010c180 con 0x7fd6c0101870 2024-04-25T01:30:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6c63b7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c010c480 con 0x7fd6c0102230 2024-04-25T01:30:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c0100680 0x7fd6c0196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c0100680 0x7fd6c0196940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36752/0 (socket says 172.21.15.28:36752) 2024-04-25T01:30:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6bffff640 1 -- 172.21.15.28:0/4098942461 learned_addr learned my addr 172.21.15.28:0/4098942461 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6bf7fe640 1 --2- 172.21.15.28:0/4098942461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6c0101870 0x7fd6c0196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6bffff640 1 -- 172.21.15.28:0/4098942461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6c0102230 msgr2=0x7fd6c019def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6c4954640 1 --2- 172.21.15.28:0/4098942461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6c0102230 0x7fd6c019def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6bffff640 1 --2- 172.21.15.28:0/4098942461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6c0102230 0x7fd6c019def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6bffff640 1 -- 172.21.15.28:0/4098942461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6c0101870 msgr2=0x7fd6c0196e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6bffff640 1 --2- 172.21.15.28:0/4098942461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6c0101870 0x7fd6c0196e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6bffff640 1 -- 172.21.15.28:0/4098942461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c019e560 con 0x7fd6c0100680 2024-04-25T01:30:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6c4954640 1 --2- 172.21.15.28:0/4098942461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6c0102230 0x7fd6c019def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6bf7fe640 1 --2- 172.21.15.28:0/4098942461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6c0101870 0x7fd6c0196e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.873+0000 7fd6bffff640 1 --2- 172.21.15.28:0/4098942461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c0100680 0x7fd6c0196940 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000e9c0 tx=0x7fd6b000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.874+0000 7fd6bd7fa640 1 -- 172.21.15.28:0/4098942461 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b000cd80 con 0x7fd6c0100680 2024-04-25T01:30:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.874+0000 7fd6bd7fa640 1 -- 172.21.15.28:0/4098942461 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6b00035f0 con 0x7fd6c0100680 2024-04-25T01:30:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.874+0000 7fd6c63b7640 1 -- 172.21.15.28:0/4098942461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6c019e850 con 0x7fd6c0100680 2024-04-25T01:30:01.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.874+0000 7fd6bd7fa640 1 -- 172.21.15.28:0/4098942461 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0010690 con 0x7fd6c0100680 2024-04-25T01:30:01.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.874+0000 7fd6c63b7640 1 -- 172.21.15.28:0/4098942461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6c019eda0 con 0x7fd6c0100680 2024-04-25T01:30:01.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.875+0000 7fd6c63b7640 1 -- 172.21.15.28:0/4098942461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6c01079a0 con 0x7fd6c0100680 2024-04-25T01:30:01.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.876+0000 7fd6bd7fa640 1 -- 172.21.15.28:0/4098942461 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd6b00037a0 con 0x7fd6c0100680 2024-04-25T01:30:01.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.877+0000 7fd6bd7fa640 1 --2- 172.21.15.28:0/4098942461 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd6a80a2b90 0x7fd6a80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:01.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.877+0000 7fd6bf7fe640 1 --2- 172.21.15.28:0/4098942461 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd6a80a2b90 0x7fd6a80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:01.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.877+0000 7fd6bd7fa640 1 -- 172.21.15.28:0/4098942461 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fd6b00cf030 con 0x7fd6c0100680 2024-04-25T01:30:01.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.877+0000 7fd6bf7fe640 1 --2- 172.21.15.28:0/4098942461 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd6a80a2b90 0x7fd6a80a5050 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0197e50 tx=0x7fd6b400a250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:01.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:01.884+0000 7fd6bd7fa640 1 -- 172.21.15.28:0/4098942461 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd6b009c430 con 0x7fd6c0100680 2024-04-25T01:30:02.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:01 smithi028 ceph-mon[29390]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:02.062 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 25 01:30:01 smithi028 podman[67900]: 2024-04-25 01:30:01.829648657 +0000 UTC m=+0.375764501 container start a369550f8aceb873d2f0ac2eee67efdc98b7943506460e0259f02cdfc4ac8ba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-j, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-25T01:30:02.062 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 25 01:30:01 smithi028 ceph-mds[67939]: -- 172.21.15.28:0/3627230004 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562609abfa40 con 0x562609a92c00 2024-04-25T01:30:02.062 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 25 01:30:01 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-j[67915]: starting mds.j at 2024-04-25T01:30:02.062 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 25 01:30:01 smithi028 bash[67900]: a369550f8aceb873d2f0ac2eee67efdc98b7943506460e0259f02cdfc4ac8ba0 2024-04-25T01:30:02.062 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 25 01:30:01 smithi028 systemd[1]: Started Ceph mds.j for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:30:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:02.113+0000 7fd6c63b7640 1 -- 172.21.15.28:0/4098942461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6c002d170 con 0x7fd6c0100680 2024-04-25T01:30:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:02.114+0000 7fd6bd7fa640 1 -- 172.21.15.28:0/4098942461 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7fd6b009bd30 con 0x7fd6c0100680 2024-04-25T01:30:02.092 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:02.092 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:30:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 4 2024-04-25T01:30:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:02.116+0000 7fd6c63b7640 1 -- 172.21.15.28:0/4098942461 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd6a80a2b90 msgr2=0x7fd6a80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:02.116+0000 7fd6c63b7640 1 --2- 172.21.15.28:0/4098942461 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd6a80a2b90 0x7fd6a80a5050 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0197e50 tx=0x7fd6b400a250 comp rx=0 tx=0).stop 2024-04-25T01:30:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:02.116+0000 7fd6c63b7640 1 -- 172.21.15.28:0/4098942461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c0100680 msgr2=0x7fd6c0196940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:02.116+0000 7fd6c63b7640 1 --2- 172.21.15.28:0/4098942461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c0100680 0x7fd6c0196940 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000e9c0 tx=0x7fd6b000ee80 comp rx=0 tx=0).stop 2024-04-25T01:30:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:02.117+0000 7fd6c63b7640 1 -- 172.21.15.28:0/4098942461 shutdown_connections 2024-04-25T01:30:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:02.117+0000 7fd6c63b7640 1 --2- 172.21.15.28:0/4098942461 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd6a80a2b90 0x7fd6a80a5050 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:02.117+0000 7fd6c63b7640 1 --2- 172.21.15.28:0/4098942461 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd6c0102230 0x7fd6c019def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:02.117+0000 7fd6c63b7640 1 --2- 172.21.15.28:0/4098942461 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd6c0101870 0x7fd6c0196e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:02.117+0000 7fd6c63b7640 1 --2- 172.21.15.28:0/4098942461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6c0100680 0x7fd6c0196940 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:02.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:02.117+0000 7fd6c63b7640 1 -- 172.21.15.28:0/4098942461 >> 172.21.15.28:0/4098942461 conn(0x7fd6c00fbbf0 msgr2=0x7fd6c0104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:02.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:02.117+0000 7fd6c63b7640 1 -- 172.21.15.28:0/4098942461 shutdown_connections 2024-04-25T01:30:02.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:02.117+0000 7fd6c63b7640 1 -- 172.21.15.28:0/4098942461 wait complete. 2024-04-25T01:30:02.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:01 smithi130 ceph-mon[32702]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:02.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:01 smithi032 ceph-mon[32461]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:02.898 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:02 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.28:6834/2774244884,v1:172.21.15.28:6835/2774244884] up:boot 2024-04-25T01:30:02.899 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:02 smithi028 ceph-mon[29390]: fsmap 3 up:standby 2024-04-25T01:30:02.899 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:02 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:30:02.899 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:02 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:02.899 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:02 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:02.899 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:02 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:02.899 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:02 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:02.899 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:02 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:02.899 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:02 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:02.899 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:02 smithi028 ceph-mon[29390]: Deploying daemon mds.d on smithi028 2024-04-25T01:30:02.899 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:02 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/4098942461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:03.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:02 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.28:6834/2774244884,v1:172.21.15.28:6835/2774244884] up:boot 2024-04-25T01:30:03.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:02 smithi130 ceph-mon[32702]: fsmap 3 up:standby 2024-04-25T01:30:03.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:02 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:30:03.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:02 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:03.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:02 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:03.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:02 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:03.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:02 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:03.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:02 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:03.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:02 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:03.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:02 smithi130 ceph-mon[32702]: Deploying daemon mds.d on smithi028 2024-04-25T01:30:03.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:02 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/4098942461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:03.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:02 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.28:6834/2774244884,v1:172.21.15.28:6835/2774244884] up:boot 2024-04-25T01:30:03.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:02 smithi032 ceph-mon[32461]: fsmap 3 up:standby 2024-04-25T01:30:03.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:02 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:30:03.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:02 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:03.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:02 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:03.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:02 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:03.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:02 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:03.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:02 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:03.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:02 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:03.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:02 smithi032 ceph-mon[32461]: Deploying daemon mds.d on smithi028 2024-04-25T01:30:03.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:02 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/4098942461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:04.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:30:04.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:04 smithi032 ceph-mon[32461]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:04.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.303+0000 7f663259e640 1 -- 172.21.15.28:0/3411553564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f662c1016b0 msgr2=0x7f662c101b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.303+0000 7f663259e640 1 --2- 172.21.15.28:0/3411553564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f662c1016b0 0x7f662c101b30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f661c009900 tx=0x7f661c02a410 comp rx=0 tx=0).stop 2024-04-25T01:30:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.303+0000 7f663259e640 1 -- 172.21.15.28:0/3411553564 shutdown_connections 2024-04-25T01:30:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.303+0000 7f663259e640 1 --2- 172.21.15.28:0/3411553564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f662c102070 0x7f662c108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.303+0000 7f663259e640 1 --2- 172.21.15.28:0/3411553564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f662c1016b0 0x7f662c101b30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.303+0000 7f663259e640 1 --2- 172.21.15.28:0/3411553564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f662c1004c0 0x7f662c1008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.303+0000 7f663259e640 1 -- 172.21.15.28:0/3411553564 >> 172.21.15.28:0/3411553564 conn(0x7f662c0fba10 msgr2=0x7f662c0fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.303+0000 7f663259e640 1 -- 172.21.15.28:0/3411553564 shutdown_connections 2024-04-25T01:30:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.303+0000 7f663259e640 1 -- 172.21.15.28:0/3411553564 wait complete. 2024-04-25T01:30:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.303+0000 7f663259e640 1 Processor -- start 2024-04-25T01:30:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.304+0000 7f663259e640 1 -- start start 2024-04-25T01:30:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.304+0000 7f663259e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f662c1004c0 0x7f662c1978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.304+0000 7f663259e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f662c1016b0 0x7f662c197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.304+0000 7f663259e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f662c102070 0x7f662c19ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.304+0000 7f663259e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f662c10c150 con 0x7f662c102070 2024-04-25T01:30:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.304+0000 7f663259e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f662c10bfd0 con 0x7f662c1004c0 2024-04-25T01:30:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.304+0000 7f663259e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f662c10c2d0 con 0x7f662c1016b0 2024-04-25T01:30:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.304+0000 7f6631d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f662c102070 0x7f662c19ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.304+0000 7f6631d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f662c102070 0x7f662c19ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36764/0 (socket says 172.21.15.28:36764) 2024-04-25T01:30:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.304+0000 7f6631d9d640 1 -- 172.21.15.28:0/2821325677 learned_addr learned my addr 172.21.15.28:0/2821325677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.304+0000 7f663159c640 1 --2- 172.21.15.28:0/2821325677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f662c1004c0 0x7f662c1978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.304+0000 7f6630d9b640 1 --2- 172.21.15.28:0/2821325677 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f662c1016b0 0x7f662c197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.304+0000 7f6631d9d640 1 -- 172.21.15.28:0/2821325677 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f662c1016b0 msgr2=0x7f662c197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.304+0000 7f6631d9d640 1 --2- 172.21.15.28:0/2821325677 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f662c1016b0 0x7f662c197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.304+0000 7f6631d9d640 1 -- 172.21.15.28:0/2821325677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f662c1004c0 msgr2=0x7f662c1978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.304+0000 7f6631d9d640 1 --2- 172.21.15.28:0/2821325677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f662c1004c0 0x7f662c1978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.304+0000 7f6631d9d640 1 -- 172.21.15.28:0/2821325677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f662c19f4e0 con 0x7f662c102070 2024-04-25T01:30:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.304+0000 7f6631d9d640 1 --2- 172.21.15.28:0/2821325677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f662c102070 0x7f662c19ee70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f662400eb10 tx=0x7f662400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:04.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.304+0000 7f66227fc640 1 -- 172.21.15.28:0/2821325677 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6624004b20 con 0x7f662c102070 2024-04-25T01:30:04.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.305+0000 7f66227fc640 1 -- 172.21.15.28:0/2821325677 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6624004cc0 con 0x7f662c102070 2024-04-25T01:30:04.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.305+0000 7f663259e640 1 -- 172.21.15.28:0/2821325677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f662c19f7d0 con 0x7f662c102070 2024-04-25T01:30:04.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.305+0000 7f66227fc640 1 -- 172.21.15.28:0/2821325677 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6624010760 con 0x7f662c102070 2024-04-25T01:30:04.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.305+0000 7f663259e640 1 -- 172.21.15.28:0/2821325677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f662c19fcf0 con 0x7f662c102070 2024-04-25T01:30:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.306+0000 7f663259e640 1 -- 172.21.15.28:0/2821325677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f662c19ffb0 con 0x7f662c102070 2024-04-25T01:30:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.307+0000 7f66227fc640 1 -- 172.21.15.28:0/2821325677 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6624005020 con 0x7f662c102070 2024-04-25T01:30:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.308+0000 7f66227fc640 1 --2- 172.21.15.28:0/2821325677 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f66180a2b90 0x7f66180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.308+0000 7f66227fc640 1 -- 172.21.15.28:0/2821325677 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f66240cf330 con 0x7f662c102070 2024-04-25T01:30:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.308+0000 7f663159c640 1 --2- 172.21.15.28:0/2821325677 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f66180a2b90 0x7f66180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.308+0000 7f663159c640 1 --2- 172.21.15.28:0/2821325677 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f66180a2b90 0x7f66180a5050 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6628005fd0 tx=0x7f662800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:04.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.313+0000 7f66227fc640 1 -- 172.21.15.28:0/2821325677 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f662409c690 con 0x7f662c102070 2024-04-25T01:30:04.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:04 smithi028 ceph-mon[29390]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:04.313 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 25 01:30:04 smithi028 systemd[1]: Starting Ceph mds.d for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:30:04.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:04 smithi130 ceph-mon[32702]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:04.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.540+0000 7f663259e640 1 -- 172.21.15.28:0/2821325677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f662c1a0180 con 0x7f662c102070 2024-04-25T01:30:04.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.541+0000 7f66227fc640 1 -- 172.21.15.28:0/2821325677 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7f662409bf90 con 0x7f662c102070 2024-04-25T01:30:04.519 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:04.520 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:30:04.520 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 4 2024-04-25T01:30:04.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.543+0000 7f663259e640 1 -- 172.21.15.28:0/2821325677 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f66180a2b90 msgr2=0x7f66180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:04.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.543+0000 7f663259e640 1 --2- 172.21.15.28:0/2821325677 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f66180a2b90 0x7f66180a5050 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6628005fd0 tx=0x7f662800b410 comp rx=0 tx=0).stop 2024-04-25T01:30:04.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.543+0000 7f663259e640 1 -- 172.21.15.28:0/2821325677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f662c102070 msgr2=0x7f662c19ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:04.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.543+0000 7f663259e640 1 --2- 172.21.15.28:0/2821325677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f662c102070 0x7f662c19ee70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f662400eb10 tx=0x7f662400c2d0 comp rx=0 tx=0).stop 2024-04-25T01:30:04.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.543+0000 7f663259e640 1 -- 172.21.15.28:0/2821325677 shutdown_connections 2024-04-25T01:30:04.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.543+0000 7f663259e640 1 --2- 172.21.15.28:0/2821325677 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f66180a2b90 0x7f66180a5050 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:04.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.543+0000 7f663259e640 1 --2- 172.21.15.28:0/2821325677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f662c102070 0x7f662c19ee70 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:04.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.543+0000 7f663259e640 1 --2- 172.21.15.28:0/2821325677 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f662c1016b0 0x7f662c197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:04.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.543+0000 7f663259e640 1 --2- 172.21.15.28:0/2821325677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f662c1004c0 0x7f662c1978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:04.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.543+0000 7f663259e640 1 -- 172.21.15.28:0/2821325677 >> 172.21.15.28:0/2821325677 conn(0x7f662c0fba10 msgr2=0x7f662c0fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:04.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.543+0000 7f663259e640 1 -- 172.21.15.28:0/2821325677 shutdown_connections 2024-04-25T01:30:04.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:04.543+0000 7f663259e640 1 -- 172.21.15.28:0/2821325677 wait complete. 2024-04-25T01:30:04.710 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 25 01:30:04 smithi028 podman[68284]: 2024-04-25 01:30:04.362821274 +0000 UTC m=+0.019330660 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:30:04.710 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 25 01:30:04 smithi028 podman[68284]: 2024-04-25 01:30:04.477292598 +0000 UTC m=+0.133801963 container create 7e9e51fb5540b0dbb5bc6eee5131b8584c69fa42c089632cfb00d34d3680571a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:30:04.710 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 25 01:30:04 smithi028 podman[68284]: 2024-04-25 01:30:04.705599551 +0000 UTC m=+0.362108927 container init 7e9e51fb5540b0dbb5bc6eee5131b8584c69fa42c089632cfb00d34d3680571a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-25T01:30:04.711 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 25 01:30:04 smithi028 podman[68284]: 2024-04-25 01:30:04.70845228 +0000 UTC m=+0.364961659 container start 7e9e51fb5540b0dbb5bc6eee5131b8584c69fa42c089632cfb00d34d3680571a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T01:30:04.711 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 25 01:30:04 smithi028 ceph-mds[68305]: -- 172.21.15.28:0/1826787319 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563fc5e9da40 con 0x563fc5e71000 2024-04-25T01:30:04.985 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 25 01:30:04 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-d[68301]: starting mds.d at 2024-04-25T01:30:04.986 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 25 01:30:04 smithi028 bash[68284]: 7e9e51fb5540b0dbb5bc6eee5131b8584c69fa42c089632cfb00d34d3680571a 2024-04-25T01:30:04.986 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 25 01:30:04 smithi028 systemd[1]: Started Ceph mds.d for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:30:05.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:05 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2821325677' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:05.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:05 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:05.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:05 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:05.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:05 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:05.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:05 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:05.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:05 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:05.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:05 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:05.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:05 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2821325677' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:05.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:05 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:05.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:05 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:05.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:05 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:05.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:05 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:05.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:05 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:05.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:05 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:05.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:05 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2821325677' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:05.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:05 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:05.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:05 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:05.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:05 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:05.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:05 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:05.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:05 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:05.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:05 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:06.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:06 smithi032 ceph-mon[32461]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:06.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:06 smithi032 ceph-mon[32461]: Deploying daemon mds.e on smithi032 2024-04-25T01:30:06.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:06 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.28:6836/2622429747,v1:172.21.15.28:6837/2622429747] up:boot 2024-04-25T01:30:06.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:06 smithi032 ceph-mon[32461]: fsmap 4 up:standby 2024-04-25T01:30:06.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:06 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:30:06.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:06 smithi028 ceph-mon[29390]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:06.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:06 smithi028 ceph-mon[29390]: Deploying daemon mds.e on smithi032 2024-04-25T01:30:06.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:06 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.28:6836/2622429747,v1:172.21.15.28:6837/2622429747] up:boot 2024-04-25T01:30:06.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:06 smithi028 ceph-mon[29390]: fsmap 4 up:standby 2024-04-25T01:30:06.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:06 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:30:06.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:06 smithi130 ceph-mon[32702]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:06.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:06 smithi130 ceph-mon[32702]: Deploying daemon mds.e on smithi032 2024-04-25T01:30:06.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:06 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.28:6836/2622429747,v1:172.21.15.28:6837/2622429747] up:boot 2024-04-25T01:30:06.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:06 smithi130 ceph-mon[32702]: fsmap 4 up:standby 2024-04-25T01:30:06.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:06 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:30:06.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:30:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.671+0000 7f4e08589640 1 -- 172.21.15.28:0/2187953291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e001006a0 msgr2=0x7f4e00100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.671+0000 7f4e08589640 1 --2- 172.21.15.28:0/2187953291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e001006a0 0x7f4e00100ac0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc009ca0 tx=0x7f4dfc02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:30:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.671+0000 7f4e08589640 1 -- 172.21.15.28:0/2187953291 shutdown_connections 2024-04-25T01:30:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.671+0000 7f4e08589640 1 --2- 172.21.15.28:0/2187953291 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e00102250 0x7f4e00108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.671+0000 7f4e08589640 1 --2- 172.21.15.28:0/2187953291 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e00101890 0x7f4e00101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.671+0000 7f4e08589640 1 --2- 172.21.15.28:0/2187953291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e001006a0 0x7f4e00100ac0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.671+0000 7f4e08589640 1 -- 172.21.15.28:0/2187953291 >> 172.21.15.28:0/2187953291 conn(0x7f4e000fbc30 msgr2=0x7f4e000fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.671+0000 7f4e08589640 1 -- 172.21.15.28:0/2187953291 shutdown_connections 2024-04-25T01:30:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.671+0000 7f4e08589640 1 -- 172.21.15.28:0/2187953291 wait complete. 2024-04-25T01:30:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.672+0000 7f4e08589640 1 Processor -- start 2024-04-25T01:30:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.672+0000 7f4e08589640 1 -- start start 2024-04-25T01:30:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.672+0000 7f4e08589640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e001006a0 0x7f4e0006a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.672+0000 7f4e08589640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e00101890 0x7f4e0019c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.672+0000 7f4e08589640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e00102250 0x7f4e0019edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.672+0000 7f4e08589640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0010c2f0 con 0x7f4e001006a0 2024-04-25T01:30:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.672+0000 7f4e08589640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0010c170 con 0x7f4e00102250 2024-04-25T01:30:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.672+0000 7f4e08589640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0010c470 con 0x7f4e00101890 2024-04-25T01:30:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.672+0000 7f4e06325640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e001006a0 0x7f4e0006a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.672+0000 7f4e06325640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e001006a0 0x7f4e0006a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40898/0 (socket says 172.21.15.28:40898) 2024-04-25T01:30:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.672+0000 7f4e06325640 1 -- 172.21.15.28:0/272563627 learned_addr learned my addr 172.21.15.28:0/272563627 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.672+0000 7f4e06b26640 1 --2- 172.21.15.28:0/272563627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e00102250 0x7f4e0019edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.672+0000 7f4e05b24640 1 --2- 172.21.15.28:0/272563627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e00101890 0x7f4e0019c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.672+0000 7f4e06325640 1 -- 172.21.15.28:0/272563627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e00101890 msgr2=0x7f4e0019c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.672+0000 7f4e06325640 1 --2- 172.21.15.28:0/272563627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e00101890 0x7f4e0019c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.673+0000 7f4e06325640 1 -- 172.21.15.28:0/272563627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e00102250 msgr2=0x7f4e0019edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.673+0000 7f4e06325640 1 --2- 172.21.15.28:0/272563627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e00102250 0x7f4e0019edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.673+0000 7f4e06325640 1 -- 172.21.15.28:0/272563627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0019f670 con 0x7f4e001006a0 2024-04-25T01:30:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.673+0000 7f4e05b24640 1 --2- 172.21.15.28:0/272563627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e00101890 0x7f4e0019c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.673+0000 7f4e06325640 1 --2- 172.21.15.28:0/272563627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e001006a0 0x7f4e0006a930 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc009bf0 tx=0x7f4dfc031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.673+0000 7f4df77fe640 1 -- 172.21.15.28:0/272563627 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc02fae0 con 0x7f4e001006a0 2024-04-25T01:30:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.673+0000 7f4df77fe640 1 -- 172.21.15.28:0/272563627 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dfc02fc80 con 0x7f4e001006a0 2024-04-25T01:30:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.673+0000 7f4e08589640 1 -- 172.21.15.28:0/272563627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e0019f900 con 0x7f4e001006a0 2024-04-25T01:30:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.673+0000 7f4df77fe640 1 -- 172.21.15.28:0/272563627 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc0467e0 con 0x7f4e001006a0 2024-04-25T01:30:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.673+0000 7f4e08589640 1 -- 172.21.15.28:0/272563627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e0019fea0 con 0x7f4e001006a0 2024-04-25T01:30:06.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.674+0000 7f4e08589640 1 -- 172.21.15.28:0/272563627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e001079c0 con 0x7f4e001006a0 2024-04-25T01:30:06.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.675+0000 7f4df77fe640 1 -- 172.21.15.28:0/272563627 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4dfc030b30 con 0x7f4e001006a0 2024-04-25T01:30:06.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.676+0000 7f4df77fe640 1 --2- 172.21.15.28:0/272563627 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4dec0a2c10 0x7f4dec0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:06.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.676+0000 7f4e05b24640 1 --2- 172.21.15.28:0/272563627 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4dec0a2c10 0x7f4dec0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:06.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.676+0000 7f4df77fe640 1 -- 172.21.15.28:0/272563627 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f4dfc0f00b0 con 0x7f4e001006a0 2024-04-25T01:30:06.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.676+0000 7f4e05b24640 1 --2- 172.21.15.28:0/272563627 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4dec0a2c10 0x7f4dec0a50d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4e0006a6a0 tx=0x7f4df0005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:06.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.680+0000 7f4df77fe640 1 -- 172.21.15.28:0/272563627 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4dfc0bd410 con 0x7f4e001006a0 2024-04-25T01:30:06.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.906+0000 7f4e08589640 1 -- 172.21.15.28:0/272563627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e00062340 con 0x7f4e001006a0 2024-04-25T01:30:06.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.906+0000 7f4df77fe640 1 -- 172.21.15.28:0/272563627 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7f4dfc0bcd10 con 0x7f4e001006a0 2024-04-25T01:30:06.885 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:06.885 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:30:06.885 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 5 2024-04-25T01:30:06.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.909+0000 7f4e08589640 1 -- 172.21.15.28:0/272563627 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4dec0a2c10 msgr2=0x7f4dec0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:06.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.909+0000 7f4e08589640 1 --2- 172.21.15.28:0/272563627 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4dec0a2c10 0x7f4dec0a50d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4e0006a6a0 tx=0x7f4df0005d90 comp rx=0 tx=0).stop 2024-04-25T01:30:06.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.909+0000 7f4e08589640 1 -- 172.21.15.28:0/272563627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e001006a0 msgr2=0x7f4e0006a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:06.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.909+0000 7f4e08589640 1 --2- 172.21.15.28:0/272563627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e001006a0 0x7f4e0006a930 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc009bf0 tx=0x7f4dfc031ee0 comp rx=0 tx=0).stop 2024-04-25T01:30:06.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.909+0000 7f4e08589640 1 -- 172.21.15.28:0/272563627 shutdown_connections 2024-04-25T01:30:06.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.909+0000 7f4e08589640 1 --2- 172.21.15.28:0/272563627 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4dec0a2c10 0x7f4dec0a50d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:06.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.909+0000 7f4e08589640 1 --2- 172.21.15.28:0/272563627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e00102250 0x7f4e0019edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:06.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.909+0000 7f4e08589640 1 --2- 172.21.15.28:0/272563627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e00101890 0x7f4e0019c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:06.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.909+0000 7f4e08589640 1 --2- 172.21.15.28:0/272563627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e001006a0 0x7f4e0006a930 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:06.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.909+0000 7f4e08589640 1 -- 172.21.15.28:0/272563627 >> 172.21.15.28:0/272563627 conn(0x7f4e000fbc30 msgr2=0x7f4e00104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:06.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.909+0000 7f4e08589640 1 -- 172.21.15.28:0/272563627 shutdown_connections 2024-04-25T01:30:06.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:06.909+0000 7f4e08589640 1 -- 172.21.15.28:0/272563627 wait complete. 2024-04-25T01:30:07.208 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:07 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/272563627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:07.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:07 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/272563627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:07.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:07 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/272563627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:07.484 INFO:journalctl@ceph.mds.e.smithi032.stdout:Apr 25 01:30:07 smithi032 podman[62525]: 2024-04-25 01:30:07.464971501 +0000 UTC m=+0.128120159 container create f70b176c5ce2c5bc2a63b08b81bea240ae0b10e4dc8fb0f838cc1a170092909e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-e, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:30:07.984 INFO:journalctl@ceph.mds.e.smithi032.stdout:Apr 25 01:30:07 smithi032 podman[62525]: 2024-04-25 01:30:07.718114458 +0000 UTC m=+0.381263119 container init f70b176c5ce2c5bc2a63b08b81bea240ae0b10e4dc8fb0f838cc1a170092909e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=) 2024-04-25T01:30:07.985 INFO:journalctl@ceph.mds.e.smithi032.stdout:Apr 25 01:30:07 smithi032 podman[62525]: 2024-04-25 01:30:07.721138972 +0000 UTC m=+0.384287631 container start f70b176c5ce2c5bc2a63b08b81bea240ae0b10e4dc8fb0f838cc1a170092909e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-e, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-25T01:30:07.985 INFO:journalctl@ceph.mds.e.smithi032.stdout:Apr 25 01:30:07 smithi032 ceph-mds[62544]: -- 172.21.15.32:0/1759700725 <== mon.2 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d075621a40 con 0x55d0755f4c00 2024-04-25T01:30:07.985 INFO:journalctl@ceph.mds.e.smithi032.stdout:Apr 25 01:30:07 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-e[62540]: starting mds.e at 2024-04-25T01:30:07.985 INFO:journalctl@ceph.mds.e.smithi032.stdout:Apr 25 01:30:07 smithi032 bash[62525]: f70b176c5ce2c5bc2a63b08b81bea240ae0b10e4dc8fb0f838cc1a170092909e 2024-04-25T01:30:07.985 INFO:journalctl@ceph.mds.e.smithi032.stdout:Apr 25 01:30:07 smithi032 systemd[1]: Started Ceph mds.e for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:30:08.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:08 smithi028 ceph-mon[29390]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:08.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:08 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:08.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:08 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:08.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:08 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:08.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:08 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:08.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:08 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:08.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:08 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:08.347 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:08 smithi130 ceph-mon[32702]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:08.347 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:08 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:08.347 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:08 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:08.347 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:08 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:08.347 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:08 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:08.348 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:08 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:08.348 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:08 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:08.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:08 smithi032 ceph-mon[32461]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:08.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:08 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:08.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:08 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:08.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:08 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:08.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:08 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:08.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:08 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:08.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:08 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:08.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:30:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.037+0000 7fb12be72640 1 -- 172.21.15.28:0/1410325479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1241012f0 msgr2=0x7fb124101770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.037+0000 7fb12be72640 1 --2- 172.21.15.28:0/1410325479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1241012f0 0x7fb124101770 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009ca0 tx=0x7fb11c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:30:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.037+0000 7fb12be72640 1 -- 172.21.15.28:0/1410325479 shutdown_connections 2024-04-25T01:30:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.037+0000 7fb12be72640 1 --2- 172.21.15.28:0/1410325479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1241012f0 0x7fb124101770 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.037+0000 7fb12be72640 1 --2- 172.21.15.28:0/1410325479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb124100950 0x7fb124100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.037+0000 7fb12be72640 1 --2- 172.21.15.28:0/1410325479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1240ffff0 0x7fb124100410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.037+0000 7fb12be72640 1 -- 172.21.15.28:0/1410325479 >> 172.21.15.28:0/1410325479 conn(0x7fb1240fbbf0 msgr2=0x7fb1240fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.037+0000 7fb12be72640 1 -- 172.21.15.28:0/1410325479 shutdown_connections 2024-04-25T01:30:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.037+0000 7fb12be72640 1 -- 172.21.15.28:0/1410325479 wait complete. 2024-04-25T01:30:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.037+0000 7fb12be72640 1 Processor -- start 2024-04-25T01:30:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.037+0000 7fb12be72640 1 -- start start 2024-04-25T01:30:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.038+0000 7fb12be72640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1240ffff0 0x7fb124199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.038+0000 7fb12be72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb124100950 0x7fb12419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.038+0000 7fb12be72640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1241012f0 0x7fb1241a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.038+0000 7fb12be72640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb12410f620 con 0x7fb124100950 2024-04-25T01:30:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.038+0000 7fb12be72640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb12410f4a0 con 0x7fb1241012f0 2024-04-25T01:30:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.038+0000 7fb12be72640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb12410f7a0 con 0x7fb1240ffff0 2024-04-25T01:30:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.038+0000 7fb12940d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb124100950 0x7fb12419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.038+0000 7fb12940d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb124100950 0x7fb12419a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40926/0 (socket says 172.21.15.28:40926) 2024-04-25T01:30:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.038+0000 7fb12940d640 1 -- 172.21.15.28:0/3490905912 learned_addr learned my addr 172.21.15.28:0/3490905912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.038+0000 7fb129c0e640 1 --2- 172.21.15.28:0/3490905912 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1240ffff0 0x7fb124199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.038+0000 7fb12a40f640 1 --2- 172.21.15.28:0/3490905912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1241012f0 0x7fb1241a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.038+0000 7fb12940d640 1 -- 172.21.15.28:0/3490905912 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1240ffff0 msgr2=0x7fb124199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.038+0000 7fb12940d640 1 --2- 172.21.15.28:0/3490905912 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1240ffff0 0x7fb124199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.038+0000 7fb12940d640 1 -- 172.21.15.28:0/3490905912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1241012f0 msgr2=0x7fb1241a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.038+0000 7fb12940d640 1 --2- 172.21.15.28:0/3490905912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1241012f0 0x7fb1241a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.038+0000 7fb12940d640 1 -- 172.21.15.28:0/3490905912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1241a1830 con 0x7fb124100950 2024-04-25T01:30:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.038+0000 7fb12940d640 1 --2- 172.21.15.28:0/3490905912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb124100950 0x7fb12419a160 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb11400d950 tx=0x7fb11400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.038+0000 7fb11affd640 1 -- 172.21.15.28:0/3490905912 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb114017070 con 0x7fb124100950 2024-04-25T01:30:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.039+0000 7fb11affd640 1 -- 172.21.15.28:0/3490905912 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb114009c30 con 0x7fb124100950 2024-04-25T01:30:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.039+0000 7fb12be72640 1 -- 172.21.15.28:0/3490905912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1241a1b20 con 0x7fb124100950 2024-04-25T01:30:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.039+0000 7fb11affd640 1 -- 172.21.15.28:0/3490905912 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11400f6b0 con 0x7fb124100950 2024-04-25T01:30:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.039+0000 7fb12be72640 1 -- 172.21.15.28:0/3490905912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1241a2070 con 0x7fb124100950 2024-04-25T01:30:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.040+0000 7fb12be72640 1 -- 172.21.15.28:0/3490905912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb12410ae50 con 0x7fb124100950 2024-04-25T01:30:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.041+0000 7fb11affd640 1 -- 172.21.15.28:0/3490905912 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb1140109d0 con 0x7fb124100950 2024-04-25T01:30:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.041+0000 7fb11affd640 1 --2- 172.21.15.28:0/3490905912 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb1100a2b90 0x7fb1100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.042+0000 7fb11affd640 1 -- 172.21.15.28:0/3490905912 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fb1140ce730 con 0x7fb124100950 2024-04-25T01:30:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.042+0000 7fb129c0e640 1 --2- 172.21.15.28:0/3490905912 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb1100a2b90 0x7fb1100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.042+0000 7fb129c0e640 1 --2- 172.21.15.28:0/3490905912 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb1100a2b90 0x7fb1100a5050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb12419b0f0 tx=0x7fb12000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:09.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.045+0000 7fb11affd640 1 -- 172.21.15.28:0/3490905912 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb114009de0 con 0x7fb124100950 2024-04-25T01:30:09.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.271+0000 7fb12be72640 1 -- 172.21.15.28:0/3490905912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb12402d290 con 0x7fb124100950 2024-04-25T01:30:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.272+0000 7fb11affd640 1 -- 172.21.15.28:0/3490905912 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4690 (secure 0 0 0) 0x7fb11409b4e0 con 0x7fb124100950 2024-04-25T01:30:09.250 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:09.253 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-25T01:30:09.254 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 6 2024-04-25T01:30:09.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.274+0000 7fb12be72640 1 -- 172.21.15.28:0/3490905912 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb1100a2b90 msgr2=0x7fb1100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:09.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.274+0000 7fb12be72640 1 --2- 172.21.15.28:0/3490905912 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb1100a2b90 0x7fb1100a5050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb12419b0f0 tx=0x7fb12000b410 comp rx=0 tx=0).stop 2024-04-25T01:30:09.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.274+0000 7fb12be72640 1 -- 172.21.15.28:0/3490905912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb124100950 msgr2=0x7fb12419a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:09.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.274+0000 7fb12be72640 1 --2- 172.21.15.28:0/3490905912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb124100950 0x7fb12419a160 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb11400d950 tx=0x7fb11400de10 comp rx=0 tx=0).stop 2024-04-25T01:30:09.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.274+0000 7fb12be72640 1 -- 172.21.15.28:0/3490905912 shutdown_connections 2024-04-25T01:30:09.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.274+0000 7fb12be72640 1 --2- 172.21.15.28:0/3490905912 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb1100a2b90 0x7fb1100a5050 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:09.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.274+0000 7fb12be72640 1 --2- 172.21.15.28:0/3490905912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1241012f0 0x7fb1241a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:09.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.274+0000 7fb12be72640 1 --2- 172.21.15.28:0/3490905912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb124100950 0x7fb12419a160 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:09.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.274+0000 7fb12be72640 1 --2- 172.21.15.28:0/3490905912 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1240ffff0 0x7fb124199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:09.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.274+0000 7fb12be72640 1 -- 172.21.15.28:0/3490905912 >> 172.21.15.28:0/3490905912 conn(0x7fb1240fbbf0 msgr2=0x7fb1240fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:09.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.274+0000 7fb12be72640 1 -- 172.21.15.28:0/3490905912 shutdown_connections 2024-04-25T01:30:09.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:09.274+0000 7fb12be72640 1 -- 172.21.15.28:0/3490905912 wait complete. 2024-04-25T01:30:09.262 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:09 smithi028 ceph-mon[29390]: Deploying daemon mds.f on smithi130 2024-04-25T01:30:09.262 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:09 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.32:6834/356243399,v1:172.21.15.32:6835/356243399] up:boot 2024-04-25T01:30:09.262 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:09 smithi028 ceph-mon[29390]: fsmap 5 up:standby 2024-04-25T01:30:09.262 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:09 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:30:09.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:09 smithi130 ceph-mon[32702]: Deploying daemon mds.f on smithi130 2024-04-25T01:30:09.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:09 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.32:6834/356243399,v1:172.21.15.32:6835/356243399] up:boot 2024-04-25T01:30:09.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:09 smithi130 ceph-mon[32702]: fsmap 5 up:standby 2024-04-25T01:30:09.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:09 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:30:09.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:09 smithi032 ceph-mon[32461]: Deploying daemon mds.f on smithi130 2024-04-25T01:30:09.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:09 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.32:6834/356243399,v1:172.21.15.32:6835/356243399] up:boot 2024-04-25T01:30:09.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:09 smithi032 ceph-mon[32461]: fsmap 5 up:standby 2024-04-25T01:30:09.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:09 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:30:10.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:10 smithi130 ceph-mon[32702]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:10.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:10 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3490905912' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:10.210 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:30:10 smithi130 systemd[1]: Starting Ceph mds.f for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:30:10.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:10 smithi028 ceph-mon[29390]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:10.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:10 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3490905912' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:10.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:10 smithi032 ceph-mon[32461]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:10.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:10 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3490905912' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:10.599 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:30:10 smithi130 podman[62435]: 2024-04-25 01:30:10.216310186 +0000 UTC m=+0.018456095 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:30:10.599 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:30:10 smithi130 podman[62435]: 2024-04-25 01:30:10.353057727 +0000 UTC m=+0.155203618 container create e9caaadfedf84525b5430331ea28aa186e8ce3d30e182da31e2eac3378fee612 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-25T01:30:10.600 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:30:10 smithi130 podman[62435]: 2024-04-25 01:30:10.622368111 +0000 UTC m=+0.424514002 container init e9caaadfedf84525b5430331ea28aa186e8ce3d30e182da31e2eac3378fee612 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-25T01:30:10.959 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:30:10 smithi130 podman[62435]: 2024-04-25 01:30:10.625105163 +0000 UTC m=+0.427251063 container start e9caaadfedf84525b5430331ea28aa186e8ce3d30e182da31e2eac3378fee612 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD) 2024-04-25T01:30:10.959 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:30:10 smithi130 ceph-mds[62455]: -- 172.21.15.130:0/3446619571 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5596803d1a40 con 0x5596803a5000 2024-04-25T01:30:10.959 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:30:10 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-f[62451]: starting mds.f at 2024-04-25T01:30:10.959 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:30:10 smithi130 bash[62435]: e9caaadfedf84525b5430331ea28aa186e8ce3d30e182da31e2eac3378fee612 2024-04-25T01:30:10.959 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:30:10 smithi130 systemd[1]: Started Ceph mds.f for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:30:11.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:30:11.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.402+0000 7fc7a2ed4640 1 -- 172.21.15.28:0/1282996982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc79c102230 msgr2=0x7fc79c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:11.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.402+0000 7fc7a2ed4640 1 --2- 172.21.15.28:0/1282996982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc79c102230 0x7fc79c108ac0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc794009ca0 tx=0x7fc79402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:30:11.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.402+0000 7fc7a2ed4640 1 -- 172.21.15.28:0/1282996982 shutdown_connections 2024-04-25T01:30:11.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.402+0000 7fc7a2ed4640 1 --2- 172.21.15.28:0/1282996982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc79c102230 0x7fc79c108ac0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.402+0000 7fc7a2ed4640 1 --2- 172.21.15.28:0/1282996982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc79c101870 0x7fc79c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.402+0000 7fc7a2ed4640 1 --2- 172.21.15.28:0/1282996982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc79c100680 0x7fc79c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.402+0000 7fc7a2ed4640 1 -- 172.21.15.28:0/1282996982 >> 172.21.15.28:0/1282996982 conn(0x7fc79c0fbbf0 msgr2=0x7fc79c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.402+0000 7fc7a2ed4640 1 -- 172.21.15.28:0/1282996982 shutdown_connections 2024-04-25T01:30:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.402+0000 7fc7a2ed4640 1 -- 172.21.15.28:0/1282996982 wait complete. 2024-04-25T01:30:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.402+0000 7fc7a2ed4640 1 Processor -- start 2024-04-25T01:30:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.402+0000 7fc7a2ed4640 1 -- start start 2024-04-25T01:30:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc7a2ed4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc79c100680 0x7fc79c197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc7a2ed4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc79c101870 0x7fc79c197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc7a2ed4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc79c102230 0x7fc79c19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc7a2ed4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc79c10c2e0 con 0x7fc79c101870 2024-04-25T01:30:11.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc7a2ed4640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc79c10c160 con 0x7fc79c102230 2024-04-25T01:30:11.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc7a2ed4640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc79c10c460 con 0x7fc79c100680 2024-04-25T01:30:11.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc79bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc79c101870 0x7fc79c197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:11.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc79bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc79c101870 0x7fc79c197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40946/0 (socket says 172.21.15.28:40946) 2024-04-25T01:30:11.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc79bfff640 1 -- 172.21.15.28:0/1487238265 learned_addr learned my addr 172.21.15.28:0/1487238265 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:11.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc7a0c70640 1 --2- 172.21.15.28:0/1487238265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc79c100680 0x7fc79c197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:11.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc7a1471640 1 --2- 172.21.15.28:0/1487238265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc79c102230 0x7fc79c19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:11.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc79bfff640 1 -- 172.21.15.28:0/1487238265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc79c100680 msgr2=0x7fc79c197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:11.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc79bfff640 1 --2- 172.21.15.28:0/1487238265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc79c100680 0x7fc79c197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:11.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc79bfff640 1 -- 172.21.15.28:0/1487238265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc79c102230 msgr2=0x7fc79c19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:11.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc79bfff640 1 --2- 172.21.15.28:0/1487238265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc79c102230 0x7fc79c19f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:11.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc79bfff640 1 -- 172.21.15.28:0/1487238265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc79c19f680 con 0x7fc79c101870 2024-04-25T01:30:11.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc79bfff640 1 --2- 172.21.15.28:0/1487238265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc79c101870 0x7fc79c197fa0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc79000e9f0 tx=0x7fc79000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:11.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc799ffb640 1 -- 172.21.15.28:0/1487238265 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79000cd80 con 0x7fc79c101870 2024-04-25T01:30:11.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc799ffb640 1 -- 172.21.15.28:0/1487238265 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc790010910 con 0x7fc79c101870 2024-04-25T01:30:11.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc7a2ed4640 1 -- 172.21.15.28:0/1487238265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc79c19f970 con 0x7fc79c101870 2024-04-25T01:30:11.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.403+0000 7fc799ffb640 1 -- 172.21.15.28:0/1487238265 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc790018990 con 0x7fc79c101870 2024-04-25T01:30:11.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.404+0000 7fc7a2ed4640 1 -- 172.21.15.28:0/1487238265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc79c19ff10 con 0x7fc79c101870 2024-04-25T01:30:11.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.405+0000 7fc7a2ed4640 1 -- 172.21.15.28:0/1487238265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc79c1a01a0 con 0x7fc79c101870 2024-04-25T01:30:11.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.406+0000 7fc799ffb640 1 -- 172.21.15.28:0/1487238265 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc790010ac0 con 0x7fc79c101870 2024-04-25T01:30:11.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.407+0000 7fc799ffb640 1 --2- 172.21.15.28:0/1487238265 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc7840a2c10 0x7fc7840a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:11.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.407+0000 7fc7a0c70640 1 --2- 172.21.15.28:0/1487238265 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc7840a2c10 0x7fc7840a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:11.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.407+0000 7fc799ffb640 1 -- 172.21.15.28:0/1487238265 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fc7900cee20 con 0x7fc79c101870 2024-04-25T01:30:11.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.407+0000 7fc7a0c70640 1 --2- 172.21.15.28:0/1487238265 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc7840a2c10 0x7fc7840a50d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc78c005fd0 tx=0x7fc78c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:11.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.411+0000 7fc799ffb640 1 -- 172.21.15.28:0/1487238265 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc79009c220 con 0x7fc79c101870 2024-04-25T01:30:11.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.634+0000 7fc7a2ed4640 1 -- 172.21.15.28:0/1487238265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc79c1a0370 con 0x7fc79c101870 2024-04-25T01:30:11.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.635+0000 7fc799ffb640 1 -- 172.21.15.28:0/1487238265 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5515 (secure 0 0 0) 0x7fc79009bb20 con 0x7fc79c101870 2024-04-25T01:30:11.612 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:11.612 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-25T01:30:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 7 2024-04-25T01:30:11.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.637+0000 7fc7a2ed4640 1 -- 172.21.15.28:0/1487238265 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc7840a2c10 msgr2=0x7fc7840a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:11.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.637+0000 7fc7a2ed4640 1 --2- 172.21.15.28:0/1487238265 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc7840a2c10 0x7fc7840a50d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc78c005fd0 tx=0x7fc78c00b410 comp rx=0 tx=0).stop 2024-04-25T01:30:11.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.637+0000 7fc7a2ed4640 1 -- 172.21.15.28:0/1487238265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc79c101870 msgr2=0x7fc79c197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.637+0000 7fc7a2ed4640 1 --2- 172.21.15.28:0/1487238265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc79c101870 0x7fc79c197fa0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc79000e9f0 tx=0x7fc79000eeb0 comp rx=0 tx=0).stop 2024-04-25T01:30:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.637+0000 7fc7a2ed4640 1 -- 172.21.15.28:0/1487238265 shutdown_connections 2024-04-25T01:30:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.637+0000 7fc7a2ed4640 1 --2- 172.21.15.28:0/1487238265 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc7840a2c10 0x7fc7840a50d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.637+0000 7fc7a2ed4640 1 --2- 172.21.15.28:0/1487238265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc79c102230 0x7fc79c19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.637+0000 7fc7a2ed4640 1 --2- 172.21.15.28:0/1487238265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc79c101870 0x7fc79c197fa0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.637+0000 7fc7a2ed4640 1 --2- 172.21.15.28:0/1487238265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc79c100680 0x7fc79c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.637+0000 7fc7a2ed4640 1 -- 172.21.15.28:0/1487238265 >> 172.21.15.28:0/1487238265 conn(0x7fc79c0fbbf0 msgr2=0x7fc79c0688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.637+0000 7fc7a2ed4640 1 -- 172.21.15.28:0/1487238265 shutdown_connections 2024-04-25T01:30:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:11.637+0000 7fc7a2ed4640 1 -- 172.21.15.28:0/1487238265 wait complete. 2024-04-25T01:30:11.959 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:11 smithi130 ceph-mon[32702]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:11.959 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:11 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:11.959 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:11 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:11.959 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:11 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:11.959 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:11 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:11.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:11 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:11.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:11 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:11.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:11 smithi130 ceph-mon[32702]: Deploying daemon mds.c on smithi130 2024-04-25T01:30:11.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:11 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] up:boot 2024-04-25T01:30:11.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:11 smithi130 ceph-mon[32702]: fsmap 6 up:standby 2024-04-25T01:30:11.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:11 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:30:11.960 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:11 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1487238265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:11.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:11 smithi032 ceph-mon[32461]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:11.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:11 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:11.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:11 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:11.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:11 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:11.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:11 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:11.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:11 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:11.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:11 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:11.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:11 smithi032 ceph-mon[32461]: Deploying daemon mds.c on smithi130 2024-04-25T01:30:11.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:11 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] up:boot 2024-04-25T01:30:11.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:11 smithi032 ceph-mon[32461]: fsmap 6 up:standby 2024-04-25T01:30:11.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:11 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:30:11.985 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:11 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1487238265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:12.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:11 smithi028 ceph-mon[29390]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:12.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:11 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:12.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:11 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:12.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:11 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:12.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:11 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:12.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:11 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:12.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:11 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:12.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:11 smithi028 ceph-mon[29390]: Deploying daemon mds.c on smithi130 2024-04-25T01:30:12.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:11 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] up:boot 2024-04-25T01:30:12.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:11 smithi028 ceph-mon[29390]: fsmap 6 up:standby 2024-04-25T01:30:12.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:11 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:30:12.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:11 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1487238265' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:12.959 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 25 01:30:12 smithi130 systemd[1]: Starting Ceph mds.c for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:30:13.297 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 25 01:30:13 smithi130 podman[62746]: 2024-04-25 01:30:12.927694769 +0000 UTC m=+0.018134434 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:30:13.297 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 25 01:30:13 smithi130 podman[62746]: 2024-04-25 01:30:13.05905264 +0000 UTC m=+0.149492351 container create 9e1ca5001ff5d59dd0eef29d2e8a6c8431c4a1af183aaf472c06aec7b1157049 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-25T01:30:13.298 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 25 01:30:13 smithi130 podman[62746]: 2024-04-25 01:30:13.320157129 +0000 UTC m=+0.410596797 container init 9e1ca5001ff5d59dd0eef29d2e8a6c8431c4a1af183aaf472c06aec7b1157049 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0) 2024-04-25T01:30:13.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:30:13.709 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 25 01:30:13 smithi130 podman[62746]: 2024-04-25 01:30:13.322922689 +0000 UTC m=+0.413362364 container start 9e1ca5001ff5d59dd0eef29d2e8a6c8431c4a1af183aaf472c06aec7b1157049 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9) 2024-04-25T01:30:13.709 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 25 01:30:13 smithi130 ceph-mds[62766]: -- 172.21.15.130:0/3121935370 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556bb5029a40 con 0x556bb4ffcc00 2024-04-25T01:30:13.709 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 25 01:30:13 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-c[62762]: starting mds.c at 2024-04-25T01:30:13.709 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 25 01:30:13 smithi130 bash[62746]: 9e1ca5001ff5d59dd0eef29d2e8a6c8431c4a1af183aaf472c06aec7b1157049 2024-04-25T01:30:13.710 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 25 01:30:13 smithi130 systemd[1]: Started Ceph mds.c for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:30:13.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.764+0000 7f04ae390640 1 -- 172.21.15.28:0/2976436298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a8104bb0 msgr2=0x7f04a8106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:13.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.764+0000 7f04ae390640 1 --2- 172.21.15.28:0/2976436298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a8104bb0 0x7f04a8106fa0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f049c009900 tx=0x7f049c02a410 comp rx=0 tx=0).stop 2024-04-25T01:30:13.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.764+0000 7f04ae390640 1 -- 172.21.15.28:0/2976436298 shutdown_connections 2024-04-25T01:30:13.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.764+0000 7f04ae390640 1 --2- 172.21.15.28:0/2976436298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04a81074e0 0x7f04a81098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.764+0000 7f04ae390640 1 --2- 172.21.15.28:0/2976436298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a8104bb0 0x7f04a8106fa0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.764+0000 7f04ae390640 1 --2- 172.21.15.28:0/2976436298 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f04a8102280 0x7f04a8104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.764+0000 7f04ae390640 1 -- 172.21.15.28:0/2976436298 >> 172.21.15.28:0/2976436298 conn(0x7f04a80fbc30 msgr2=0x7f04a80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:13.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.764+0000 7f04ae390640 1 -- 172.21.15.28:0/2976436298 shutdown_connections 2024-04-25T01:30:13.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.764+0000 7f04ae390640 1 -- 172.21.15.28:0/2976436298 wait complete. 2024-04-25T01:30:13.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.765+0000 7f04ae390640 1 Processor -- start 2024-04-25T01:30:13.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.765+0000 7f04ae390640 1 -- start start 2024-04-25T01:30:13.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.765+0000 7f04ae390640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a8102280 0x7f04a8197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.765+0000 7f04ae390640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f04a8104bb0 0x7f04a8198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.765+0000 7f04ae390640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04a81074e0 0x7f04a819f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.765+0000 7f04ae390640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04a810d410 con 0x7f04a8102280 2024-04-25T01:30:13.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.765+0000 7f04ae390640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f04a810d290 con 0x7f04a81074e0 2024-04-25T01:30:13.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.765+0000 7f04ae390640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f04a810d590 con 0x7f04a8104bb0 2024-04-25T01:30:13.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.765+0000 7f04a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a8102280 0x7f04a8197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.765+0000 7f04a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a8102280 0x7f04a8197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40954/0 (socket says 172.21.15.28:40954) 2024-04-25T01:30:13.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.765+0000 7f04a7fff640 1 -- 172.21.15.28:0/1833085128 learned_addr learned my addr 172.21.15.28:0/1833085128 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:13.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.765+0000 7f04ac92d640 1 --2- 172.21.15.28:0/1833085128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04a81074e0 0x7f04a819f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.765+0000 7f04a77fe640 1 --2- 172.21.15.28:0/1833085128 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f04a8104bb0 0x7f04a8198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.766+0000 7f04a7fff640 1 -- 172.21.15.28:0/1833085128 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f04a8104bb0 msgr2=0x7f04a8198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:13.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.766+0000 7f04a7fff640 1 --2- 172.21.15.28:0/1833085128 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f04a8104bb0 0x7f04a8198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.766+0000 7f04a7fff640 1 -- 172.21.15.28:0/1833085128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04a81074e0 msgr2=0x7f04a819f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:13.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.766+0000 7f04a7fff640 1 --2- 172.21.15.28:0/1833085128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04a81074e0 0x7f04a819f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.766+0000 7f04a7fff640 1 -- 172.21.15.28:0/1833085128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a819f710 con 0x7f04a8102280 2024-04-25T01:30:13.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.766+0000 7f04a7fff640 1 --2- 172.21.15.28:0/1833085128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a8102280 0x7f04a8197af0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f049800e9f0 tx=0x7f049800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:13.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.766+0000 7f04a57fa640 1 -- 172.21.15.28:0/1833085128 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049800cd80 con 0x7f04a8102280 2024-04-25T01:30:13.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.766+0000 7f04a57fa640 1 -- 172.21.15.28:0/1833085128 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0498005500 con 0x7f04a8102280 2024-04-25T01:30:13.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.766+0000 7f04ae390640 1 -- 172.21.15.28:0/1833085128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04a819fa00 con 0x7f04a8102280 2024-04-25T01:30:13.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.766+0000 7f04a57fa640 1 -- 172.21.15.28:0/1833085128 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0498010690 con 0x7f04a8102280 2024-04-25T01:30:13.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.766+0000 7f04ae390640 1 -- 172.21.15.28:0/1833085128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04a819fed0 con 0x7f04a8102280 2024-04-25T01:30:13.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.767+0000 7f04ae390640 1 -- 172.21.15.28:0/1833085128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04a80fd720 con 0x7f04a8102280 2024-04-25T01:30:13.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.768+0000 7f04a57fa640 1 -- 172.21.15.28:0/1833085128 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f04980046c0 con 0x7f04a8102280 2024-04-25T01:30:13.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.769+0000 7f04a57fa640 1 --2- 172.21.15.28:0/1833085128 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f04900a2b90 0x7f04900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:13.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.769+0000 7f04a77fe640 1 --2- 172.21.15.28:0/1833085128 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f04900a2b90 0x7f04900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:13.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.769+0000 7f04a57fa640 1 -- 172.21.15.28:0/1833085128 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f04980cf060 con 0x7f04a8102280 2024-04-25T01:30:13.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.769+0000 7f04a77fe640 1 --2- 172.21.15.28:0/1833085128 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f04900a2b90 0x7f04900a5050 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f049c005ab0 tx=0x7f049c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:13.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:13.773+0000 7f04a57fa640 1 -- 172.21.15.28:0/1833085128 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f049809c3c0 con 0x7f04a8102280 2024-04-25T01:30:13.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:14.000+0000 7f04ae390640 1 -- 172.21.15.28:0/1833085128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04a802d170 con 0x7f04a8102280 2024-04-25T01:30:13.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:14.000+0000 7f04a57fa640 1 -- 172.21.15.28:0/1833085128 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6340 (secure 0 0 0) 0x7f049809bcc0 con 0x7f04a8102280 2024-04-25T01:30:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:13.978 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-25T01:30:13.978 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-04-25T01:30:13.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:14.002+0000 7f04ae390640 1 -- 172.21.15.28:0/1833085128 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f04900a2b90 msgr2=0x7f04900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:13.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:14.002+0000 7f04ae390640 1 --2- 172.21.15.28:0/1833085128 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f04900a2b90 0x7f04900a5050 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f049c005ab0 tx=0x7f049c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:14.002+0000 7f04ae390640 1 -- 172.21.15.28:0/1833085128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a8102280 msgr2=0x7f04a8197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:13.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:14.002+0000 7f04ae390640 1 --2- 172.21.15.28:0/1833085128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a8102280 0x7f04a8197af0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f049800e9f0 tx=0x7f049800edf0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:14.002+0000 7f04ae390640 1 -- 172.21.15.28:0/1833085128 shutdown_connections 2024-04-25T01:30:13.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:14.002+0000 7f04ae390640 1 --2- 172.21.15.28:0/1833085128 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f04900a2b90 0x7f04900a5050 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:14.002+0000 7f04ae390640 1 --2- 172.21.15.28:0/1833085128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f04a81074e0 0x7f04a819f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:14.002+0000 7f04ae390640 1 --2- 172.21.15.28:0/1833085128 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f04a8104bb0 0x7f04a8198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:14.002+0000 7f04ae390640 1 --2- 172.21.15.28:0/1833085128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a8102280 0x7f04a8197af0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:13.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:14.002+0000 7f04ae390640 1 -- 172.21.15.28:0/1833085128 >> 172.21.15.28:0/1833085128 conn(0x7f04a80fbc30 msgr2=0x7f04a8102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:13.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:14.002+0000 7f04ae390640 1 -- 172.21.15.28:0/1833085128 shutdown_connections 2024-04-25T01:30:13.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:14.002+0000 7f04ae390640 1 -- 172.21.15.28:0/1833085128 wait complete. 2024-04-25T01:30:13.990 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:13 smithi028 ceph-mon[29390]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:13.990 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:13.990 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:13.990 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:13.990 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:13.990 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:13.990 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:14.161 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:13 smithi032 ceph-mon[32461]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:14.162 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:14.162 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:14.162 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:14.162 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:14.162 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:14.162 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:14.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:13 smithi130 ceph-mon[32702]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:14.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:14.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:14.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:14.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:14.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:14.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:15.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:14 smithi028 ceph-mon[29390]: Deploying daemon mds.h on smithi032 2024-04-25T01:30:15.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:14 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] up:boot 2024-04-25T01:30:15.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:14 smithi028 ceph-mon[29390]: fsmap 7 up:standby 2024-04-25T01:30:15.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:14 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:30:15.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:14 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1833085128' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:15.183 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:14 smithi032 ceph-mon[32461]: Deploying daemon mds.h on smithi032 2024-04-25T01:30:15.183 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:14 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] up:boot 2024-04-25T01:30:15.184 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:14 smithi032 ceph-mon[32461]: fsmap 7 up:standby 2024-04-25T01:30:15.184 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:14 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:30:15.184 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:14 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1833085128' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:15.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:14 smithi130 ceph-mon[32702]: Deploying daemon mds.h on smithi032 2024-04-25T01:30:15.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:14 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] up:boot 2024-04-25T01:30:15.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:14 smithi130 ceph-mon[32702]: fsmap 7 up:standby 2024-04-25T01:30:15.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:14 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:30:15.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:14 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1833085128' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:15.984 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:15 smithi032 ceph-mon[32461]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:15.985 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:30:15 smithi032 systemd[1]: Starting Ceph mds.h for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:30:15.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:30:16.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:15 smithi028 ceph-mon[29390]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:16.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.119+0000 7f3726a5d640 1 -- 172.21.15.28:0/2513682142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37201017b0 msgr2=0x7f3720101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:16.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.119+0000 7f3726a5d640 1 --2- 172.21.15.28:0/2513682142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37201017b0 0x7f3720101bd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3710009900 tx=0x7f371002a410 comp rx=0 tx=0).stop 2024-04-25T01:30:16.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.119+0000 7f3726a5d640 1 -- 172.21.15.28:0/2513682142 shutdown_connections 2024-04-25T01:30:16.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.119+0000 7f3726a5d640 1 --2- 172.21.15.28:0/2513682142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3720103360 0x7f3720109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:16.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.119+0000 7f3726a5d640 1 --2- 172.21.15.28:0/2513682142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37201029a0 0x7f3720102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:16.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.119+0000 7f3726a5d640 1 --2- 172.21.15.28:0/2513682142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37201017b0 0x7f3720101bd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:16.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.119+0000 7f3726a5d640 1 -- 172.21.15.28:0/2513682142 >> 172.21.15.28:0/2513682142 conn(0x7f37200fcd20 msgr2=0x7f37200ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:16.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.119+0000 7f3726a5d640 1 -- 172.21.15.28:0/2513682142 shutdown_connections 2024-04-25T01:30:16.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.119+0000 7f3726a5d640 1 -- 172.21.15.28:0/2513682142 wait complete. 2024-04-25T01:30:16.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.120+0000 7f3726a5d640 1 Processor -- start 2024-04-25T01:30:16.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.120+0000 7f3726a5d640 1 -- start start 2024-04-25T01:30:16.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.120+0000 7f3726a5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37201017b0 0x7f3720197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:16.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.120+0000 7f3726a5d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37201029a0 0x7f3720197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:16.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.120+0000 7f3726a5d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3720103360 0x7f372019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:16.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.120+0000 7f3726a5d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f372010d430 con 0x7f37201017b0 2024-04-25T01:30:16.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.120+0000 7f3726a5d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f372010d2b0 con 0x7f3720103360 2024-04-25T01:30:16.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.120+0000 7f3726a5d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f372010d5b0 con 0x7f37201029a0 2024-04-25T01:30:16.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.120+0000 7f371ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37201017b0 0x7f3720197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:16.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.120+0000 7f371ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37201017b0 0x7f3720197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40982/0 (socket says 172.21.15.28:40982) 2024-04-25T01:30:16.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.121+0000 7f371ffff640 1 -- 172.21.15.28:0/1004445602 learned_addr learned my addr 172.21.15.28:0/1004445602 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:16.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.121+0000 7f371f7fe640 1 --2- 172.21.15.28:0/1004445602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37201029a0 0x7f3720197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:16.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.121+0000 7f3724ffa640 1 --2- 172.21.15.28:0/1004445602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3720103360 0x7f372019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:16.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.121+0000 7f371ffff640 1 -- 172.21.15.28:0/1004445602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37201029a0 msgr2=0x7f3720197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:16.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.121+0000 7f371ffff640 1 --2- 172.21.15.28:0/1004445602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37201029a0 0x7f3720197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:16.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.121+0000 7f371ffff640 1 -- 172.21.15.28:0/1004445602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3720103360 msgr2=0x7f372019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:16.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.121+0000 7f371ffff640 1 --2- 172.21.15.28:0/1004445602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3720103360 0x7f372019f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:16.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.121+0000 7f371ffff640 1 -- 172.21.15.28:0/1004445602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f372019f6c0 con 0x7f37201017b0 2024-04-25T01:30:16.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.121+0000 7f3724ffa640 1 --2- 172.21.15.28:0/1004445602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3720103360 0x7f372019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:16.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.121+0000 7f371f7fe640 1 --2- 172.21.15.28:0/1004445602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37201029a0 0x7f3720197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:16.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.121+0000 7f371ffff640 1 --2- 172.21.15.28:0/1004445602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37201017b0 0x7f3720197aa0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3710031610 tx=0x7f37100043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:16.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.121+0000 7f371d7fa640 1 -- 172.21.15.28:0/1004445602 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371003f030 con 0x7f37201017b0 2024-04-25T01:30:16.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.121+0000 7f3726a5d640 1 -- 172.21.15.28:0/1004445602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f372019f950 con 0x7f37201017b0 2024-04-25T01:30:16.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.121+0000 7f371d7fa640 1 -- 172.21.15.28:0/1004445602 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3710003ce0 con 0x7f37201017b0 2024-04-25T01:30:16.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.121+0000 7f371d7fa640 1 -- 172.21.15.28:0/1004445602 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3710035070 con 0x7f37201017b0 2024-04-25T01:30:16.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.121+0000 7f3726a5d640 1 -- 172.21.15.28:0/1004445602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f372019fe90 con 0x7f37201017b0 2024-04-25T01:30:16.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.122+0000 7f3726a5d640 1 -- 172.21.15.28:0/1004445602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3720108ad0 con 0x7f37201017b0 2024-04-25T01:30:16.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.123+0000 7f371d7fa640 1 -- 172.21.15.28:0/1004445602 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3710042030 con 0x7f37201017b0 2024-04-25T01:30:16.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.124+0000 7f371d7fa640 1 --2- 172.21.15.28:0/1004445602 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f37080a2b90 0x7f37080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:16.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.124+0000 7f371d7fa640 1 -- 172.21.15.28:0/1004445602 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f37100ef4f0 con 0x7f37201017b0 2024-04-25T01:30:16.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.124+0000 7f371f7fe640 1 --2- 172.21.15.28:0/1004445602 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f37080a2b90 0x7f37080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:16.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.125+0000 7f371f7fe640 1 --2- 172.21.15.28:0/1004445602 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f37080a2b90 0x7f37080a5050 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f37140058c0 tx=0x7f3714005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:16.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.132+0000 7f371d7fa640 1 -- 172.21.15.28:0/1004445602 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f37100bc8f0 con 0x7f37201017b0 2024-04-25T01:30:16.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:15 smithi130 ceph-mon[32702]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:16.289 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:30:16 smithi032 podman[62837]: 2024-04-25 01:30:15.963017414 +0000 UTC m=+0.018693050 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:30:16.289 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:30:16 smithi032 podman[62837]: 2024-04-25 01:30:16.07231906 +0000 UTC m=+0.127994709 container create 4a375432e3bf098820e6c6480f9b03c0c450324344e507bb4ed2b799c96e0c78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-25T01:30:16.289 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:30:16 smithi032 podman[62837]: 2024-04-25 01:30:16.308570684 +0000 UTC m=+0.364246288 container init 4a375432e3bf098820e6c6480f9b03c0c450324344e507bb4ed2b799c96e0c78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:30:16.289 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:30:16 smithi032 podman[62837]: 2024-04-25 01:30:16.311404871 +0000 UTC m=+0.367080477 container start 4a375432e3bf098820e6c6480f9b03c0c450324344e507bb4ed2b799c96e0c78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:30:16.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.356+0000 7f3726a5d640 1 -- 172.21.15.28:0/1004445602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f372002d400 con 0x7f37201017b0 2024-04-25T01:30:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.357+0000 7f371d7fa640 1 -- 172.21.15.28:0/1004445602 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6340 (secure 0 0 0) 0x7f37100bc1f0 con 0x7f37201017b0 2024-04-25T01:30:16.334 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:16.334 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-25T01:30:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-04-25T01:30:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.359+0000 7f3726a5d640 1 -- 172.21.15.28:0/1004445602 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f37080a2b90 msgr2=0x7f37080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.359+0000 7f3726a5d640 1 --2- 172.21.15.28:0/1004445602 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f37080a2b90 0x7f37080a5050 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f37140058c0 tx=0x7f3714005850 comp rx=0 tx=0).stop 2024-04-25T01:30:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.359+0000 7f3726a5d640 1 -- 172.21.15.28:0/1004445602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37201017b0 msgr2=0x7f3720197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.359+0000 7f3726a5d640 1 --2- 172.21.15.28:0/1004445602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37201017b0 0x7f3720197aa0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3710031610 tx=0x7f37100043b0 comp rx=0 tx=0).stop 2024-04-25T01:30:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.359+0000 7f3726a5d640 1 -- 172.21.15.28:0/1004445602 shutdown_connections 2024-04-25T01:30:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.359+0000 7f3726a5d640 1 --2- 172.21.15.28:0/1004445602 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f37080a2b90 0x7f37080a5050 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.359+0000 7f3726a5d640 1 --2- 172.21.15.28:0/1004445602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3720103360 0x7f372019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.359+0000 7f3726a5d640 1 --2- 172.21.15.28:0/1004445602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37201029a0 0x7f3720197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.359+0000 7f3726a5d640 1 --2- 172.21.15.28:0/1004445602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37201017b0 0x7f3720197aa0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.359+0000 7f3726a5d640 1 -- 172.21.15.28:0/1004445602 >> 172.21.15.28:0/1004445602 conn(0x7f37200fcd20 msgr2=0x7f3720105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.359+0000 7f3726a5d640 1 -- 172.21.15.28:0/1004445602 shutdown_connections 2024-04-25T01:30:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:16.359+0000 7f3726a5d640 1 -- 172.21.15.28:0/1004445602 wait complete. 2024-04-25T01:30:16.734 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:30:16 smithi032 ceph-mds[62856]: -- 172.21.15.32:0/3115027427 <== mon.2 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561ac5283a40 con 0x561ac5257000 2024-04-25T01:30:16.734 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:30:16 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-h[62852]: starting mds.h at 2024-04-25T01:30:16.735 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:30:16 smithi032 bash[62837]: 4a375432e3bf098820e6c6480f9b03c0c450324344e507bb4ed2b799c96e0c78 2024-04-25T01:30:16.735 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:30:16 smithi032 systemd[1]: Started Ceph mds.h for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:30:17.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:16 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1004445602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:17.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:17.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:17.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:17.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:17.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:17.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:16 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:17.155 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:16 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1004445602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:17.155 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:17.155 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:17.155 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:17.155 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:17.155 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:17.155 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:16 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:17.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:16 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1004445602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:17.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:17.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:17.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:17.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:17.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:17.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:16 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:18.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:17 smithi028 ceph-mon[29390]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:18.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:17 smithi028 ceph-mon[29390]: Deploying daemon mds.i on smithi130 2024-04-25T01:30:18.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:17 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] up:boot 2024-04-25T01:30:18.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:17 smithi028 ceph-mon[29390]: fsmap 8 up:standby 2024-04-25T01:30:18.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:17 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:30:18.180 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:17 smithi130 ceph-mon[32702]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:18.180 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:17 smithi130 ceph-mon[32702]: Deploying daemon mds.i on smithi130 2024-04-25T01:30:18.180 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:17 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] up:boot 2024-04-25T01:30:18.181 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:17 smithi130 ceph-mon[32702]: fsmap 8 up:standby 2024-04-25T01:30:18.181 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:17 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:30:18.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:17 smithi032 ceph-mon[32461]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:18.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:17 smithi032 ceph-mon[32461]: Deploying daemon mds.i on smithi130 2024-04-25T01:30:18.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:17 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] up:boot 2024-04-25T01:30:18.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:17 smithi032 ceph-mon[32461]: fsmap 8 up:standby 2024-04-25T01:30:18.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:17 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:30:18.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:30:18.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.486+0000 7fc43959e640 1 -- 172.21.15.28:0/1765330267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4340ffc40 msgr2=0x7fc434100060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:18.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.486+0000 7fc43959e640 1 --2- 172.21.15.28:0/1765330267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4340ffc40 0x7fc434100060 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc424009900 tx=0x7fc42402a410 comp rx=0 tx=0).stop 2024-04-25T01:30:18.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.486+0000 7fc43959e640 1 -- 172.21.15.28:0/1765330267 shutdown_connections 2024-04-25T01:30:18.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.486+0000 7fc43959e640 1 --2- 172.21.15.28:0/1765330267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc4341017f0 0x7fc434108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:18.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.486+0000 7fc43959e640 1 --2- 172.21.15.28:0/1765330267 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc434100e30 0x7fc4341012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:18.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.486+0000 7fc43959e640 1 --2- 172.21.15.28:0/1765330267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4340ffc40 0x7fc434100060 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:18.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.486+0000 7fc43959e640 1 -- 172.21.15.28:0/1765330267 >> 172.21.15.28:0/1765330267 conn(0x7fc4340fb1d0 msgr2=0x7fc4340fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:18.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.486+0000 7fc43959e640 1 -- 172.21.15.28:0/1765330267 shutdown_connections 2024-04-25T01:30:18.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.487+0000 7fc43959e640 1 -- 172.21.15.28:0/1765330267 wait complete. 2024-04-25T01:30:18.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.487+0000 7fc43959e640 1 Processor -- start 2024-04-25T01:30:18.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.487+0000 7fc43959e640 1 -- start start 2024-04-25T01:30:18.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.488+0000 7fc43959e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4340ffc40 0x7fc434197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:18.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.488+0000 7fc43959e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc434100e30 0x7fc434197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:18.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.488+0000 7fc43959e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4341017f0 0x7fc43419eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.488+0000 7fc43959e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc43410c1d0 con 0x7fc4341017f0 2024-04-25T01:30:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.488+0000 7fc43959e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc43410c050 con 0x7fc434100e30 2024-04-25T01:30:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.488+0000 7fc43959e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc43410c350 con 0x7fc4340ffc40 2024-04-25T01:30:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.488+0000 7fc438d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4341017f0 0x7fc43419eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.488+0000 7fc433fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4340ffc40 0x7fc434197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.488+0000 7fc438d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4341017f0 0x7fc43419eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45122/0 (socket says 172.21.15.28:45122) 2024-04-25T01:30:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.488+0000 7fc438d9d640 1 -- 172.21.15.28:0/1906840453 learned_addr learned my addr 172.21.15.28:0/1906840453 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.488+0000 7fc4337fe640 1 --2- 172.21.15.28:0/1906840453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc434100e30 0x7fc434197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.488+0000 7fc438d9d640 1 -- 172.21.15.28:0/1906840453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4340ffc40 msgr2=0x7fc434197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.488+0000 7fc438d9d640 1 --2- 172.21.15.28:0/1906840453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4340ffc40 0x7fc434197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.488+0000 7fc438d9d640 1 -- 172.21.15.28:0/1906840453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc434100e30 msgr2=0x7fc434197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.488+0000 7fc438d9d640 1 --2- 172.21.15.28:0/1906840453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc434100e30 0x7fc434197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:18.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.488+0000 7fc438d9d640 1 -- 172.21.15.28:0/1906840453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc434102050 con 0x7fc4341017f0 2024-04-25T01:30:18.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.488+0000 7fc433fff640 1 --2- 172.21.15.28:0/1906840453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4340ffc40 0x7fc434197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:18.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.488+0000 7fc4337fe640 1 --2- 172.21.15.28:0/1906840453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc434100e30 0x7fc434197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:18.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.488+0000 7fc438d9d640 1 --2- 172.21.15.28:0/1906840453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4341017f0 0x7fc43419eeb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc42c00cce0 tx=0x7fc42c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:18.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.489+0000 7fc4317fa640 1 -- 172.21.15.28:0/1906840453 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c00f450 con 0x7fc4341017f0 2024-04-25T01:30:18.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.489+0000 7fc4317fa640 1 -- 172.21.15.28:0/1906840453 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc42c00fad0 con 0x7fc4341017f0 2024-04-25T01:30:18.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.489+0000 7fc43959e640 1 -- 172.21.15.28:0/1906840453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc434102340 con 0x7fc4341017f0 2024-04-25T01:30:18.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.489+0000 7fc4317fa640 1 -- 172.21.15.28:0/1906840453 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c018c10 con 0x7fc4341017f0 2024-04-25T01:30:18.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.489+0000 7fc43959e640 1 -- 172.21.15.28:0/1906840453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc434102840 con 0x7fc4341017f0 2024-04-25T01:30:18.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.490+0000 7fc43959e640 1 -- 172.21.15.28:0/1906840453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc434102b00 con 0x7fc4341017f0 2024-04-25T01:30:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.491+0000 7fc4317fa640 1 -- 172.21.15.28:0/1906840453 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc42c00f5f0 con 0x7fc4341017f0 2024-04-25T01:30:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.491+0000 7fc4317fa640 1 --2- 172.21.15.28:0/1906840453 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc4200a2b90 0x7fc4200a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.492+0000 7fc433fff640 1 --2- 172.21.15.28:0/1906840453 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc4200a2b90 0x7fc4200a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.492+0000 7fc4317fa640 1 -- 172.21.15.28:0/1906840453 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fc42c0cf290 con 0x7fc4341017f0 2024-04-25T01:30:18.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.492+0000 7fc433fff640 1 --2- 172.21.15.28:0/1906840453 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc4200a2b90 0x7fc4200a5050 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc42402aa00 tx=0x7fc4240315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:18.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.496+0000 7fc4317fa640 1 -- 172.21.15.28:0/1906840453 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc42c014030 con 0x7fc4341017f0 2024-04-25T01:30:18.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.720+0000 7fc43959e640 1 -- 172.21.15.28:0/1906840453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4341032d0 con 0x7fc4341017f0 2024-04-25T01:30:18.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.721+0000 7fc4317fa640 1 -- 172.21.15.28:0/1906840453 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7165 (secure 0 0 0) 0x7fc42c09c690 con 0x7fc4341017f0 2024-04-25T01:30:18.699 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:18.699 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-25T01:30:18.699 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-04-25T01:30:18.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.723+0000 7fc43959e640 1 -- 172.21.15.28:0/1906840453 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc4200a2b90 msgr2=0x7fc4200a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:18.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.723+0000 7fc43959e640 1 --2- 172.21.15.28:0/1906840453 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc4200a2b90 0x7fc4200a5050 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc42402aa00 tx=0x7fc4240315b0 comp rx=0 tx=0).stop 2024-04-25T01:30:18.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.723+0000 7fc43959e640 1 -- 172.21.15.28:0/1906840453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4341017f0 msgr2=0x7fc43419eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:18.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.723+0000 7fc43959e640 1 --2- 172.21.15.28:0/1906840453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4341017f0 0x7fc43419eeb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc42c00cce0 tx=0x7fc42c00a620 comp rx=0 tx=0).stop 2024-04-25T01:30:18.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.723+0000 7fc43959e640 1 -- 172.21.15.28:0/1906840453 shutdown_connections 2024-04-25T01:30:18.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.723+0000 7fc43959e640 1 --2- 172.21.15.28:0/1906840453 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc4200a2b90 0x7fc4200a5050 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:18.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.723+0000 7fc43959e640 1 --2- 172.21.15.28:0/1906840453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4341017f0 0x7fc43419eeb0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:18.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.723+0000 7fc43959e640 1 --2- 172.21.15.28:0/1906840453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc434100e30 0x7fc434197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:18.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.723+0000 7fc43959e640 1 --2- 172.21.15.28:0/1906840453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc4340ffc40 0x7fc434197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:18.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.723+0000 7fc43959e640 1 -- 172.21.15.28:0/1906840453 >> 172.21.15.28:0/1906840453 conn(0x7fc4340fb1d0 msgr2=0x7fc4340fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:18.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.723+0000 7fc43959e640 1 -- 172.21.15.28:0/1906840453 shutdown_connections 2024-04-25T01:30:18.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:18.723+0000 7fc43959e640 1 -- 172.21.15.28:0/1906840453 wait complete. 2024-04-25T01:30:18.773 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 25 01:30:18 smithi130 systemd[1]: Starting Ceph mds.i for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:30:19.031 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:18 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1906840453' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:19.032 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 25 01:30:18 smithi130 podman[63053]: 2024-04-25 01:30:18.788182358 +0000 UTC m=+0.018510054 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:30:19.032 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 25 01:30:18 smithi130 podman[63053]: 2024-04-25 01:30:18.924706308 +0000 UTC m=+0.155034018 container create af61a3080c0dd2c8f8c6181b5c29ce650cb2a204d2c26e3af90fc322c364b2d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-i, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-25T01:30:19.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:18 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1906840453' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:19.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:18 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1906840453' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:19.459 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 25 01:30:19 smithi130 podman[63053]: 2024-04-25 01:30:19.186245783 +0000 UTC m=+0.416573451 container init af61a3080c0dd2c8f8c6181b5c29ce650cb2a204d2c26e3af90fc322c364b2d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-i, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T01:30:19.459 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 25 01:30:19 smithi130 podman[63053]: 2024-04-25 01:30:19.188823025 +0000 UTC m=+0.419150693 container start af61a3080c0dd2c8f8c6181b5c29ce650cb2a204d2c26e3af90fc322c364b2d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD) 2024-04-25T01:30:19.459 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 25 01:30:19 smithi130 ceph-mds[63072]: -- 172.21.15.130:0/1025950377 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f1b7847a40 con 0x55f1b781a800 2024-04-25T01:30:19.460 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 25 01:30:19 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-i[63068]: starting mds.i at 2024-04-25T01:30:19.460 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 25 01:30:19 smithi130 bash[63053]: af61a3080c0dd2c8f8c6181b5c29ce650cb2a204d2c26e3af90fc322c364b2d9 2024-04-25T01:30:19.460 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 25 01:30:19 smithi130 systemd[1]: Started Ceph mds.i for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:30:20.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:19 smithi028 ceph-mon[29390]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:20.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:19 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:20.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:19 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:20.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:19 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:20.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:19 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:20.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:19 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:20.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:19 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:20.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:19 smithi130 ceph-mon[32702]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:20.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:19 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:20.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:19 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:20.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:19 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:20.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:19 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:20.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:19 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:20.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:19 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:20.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:19 smithi032 ceph-mon[32461]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:20.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:19 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:20.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:19 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:20.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:19 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:20.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:19 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:20.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:19 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:20.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:19 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:20.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:30:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.850+0000 7f1672415640 1 -- 172.21.15.28:0/431835088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f166c1006a0 msgr2=0x7f166c100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.850+0000 7f1672415640 1 --2- 172.21.15.28:0/431835088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f166c1006a0 0x7f166c100ac0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f165c009900 tx=0x7f165c02a410 comp rx=0 tx=0).stop 2024-04-25T01:30:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.850+0000 7f1672415640 1 -- 172.21.15.28:0/431835088 shutdown_connections 2024-04-25T01:30:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.850+0000 7f1672415640 1 --2- 172.21.15.28:0/431835088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f166c102250 0x7f166c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.850+0000 7f1672415640 1 --2- 172.21.15.28:0/431835088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166c101890 0x7f166c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.850+0000 7f1672415640 1 --2- 172.21.15.28:0/431835088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f166c1006a0 0x7f166c100ac0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.850+0000 7f1672415640 1 -- 172.21.15.28:0/431835088 >> 172.21.15.28:0/431835088 conn(0x7f166c0fbc30 msgr2=0x7f166c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.850+0000 7f1672415640 1 -- 172.21.15.28:0/431835088 shutdown_connections 2024-04-25T01:30:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.850+0000 7f1672415640 1 -- 172.21.15.28:0/431835088 wait complete. 2024-04-25T01:30:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.850+0000 7f1672415640 1 Processor -- start 2024-04-25T01:30:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.851+0000 7f1672415640 1 -- start start 2024-04-25T01:30:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.851+0000 7f1672415640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f166c1006a0 0x7f166c06a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:20.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.851+0000 7f1672415640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f166c101890 0x7f166c19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:20.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.851+0000 7f1672415640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166c102250 0x7f166c19ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:20.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.851+0000 7f1672415640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f166c10c370 con 0x7f166c101890 2024-04-25T01:30:20.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.851+0000 7f1672415640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f166c10c1f0 con 0x7f166c102250 2024-04-25T01:30:20.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.851+0000 7f1672415640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f166c10c4f0 con 0x7f166c1006a0 2024-04-25T01:30:20.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.851+0000 7f166b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f166c101890 0x7f166c19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:20.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.851+0000 7f166b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f166c101890 0x7f166c19c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45130/0 (socket says 172.21.15.28:45130) 2024-04-25T01:30:20.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.851+0000 7f166b7fe640 1 -- 172.21.15.28:0/2774870757 learned_addr learned my addr 172.21.15.28:0/2774870757 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:20.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.851+0000 7f166bfff640 1 --2- 172.21.15.28:0/2774870757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f166c1006a0 0x7f166c06a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:20.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.851+0000 7f16709b2640 1 --2- 172.21.15.28:0/2774870757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166c102250 0x7f166c19ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:20.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.851+0000 7f166b7fe640 1 -- 172.21.15.28:0/2774870757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f166c1006a0 msgr2=0x7f166c06a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:20.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.851+0000 7f166b7fe640 1 --2- 172.21.15.28:0/2774870757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f166c1006a0 0x7f166c06a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:20.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.851+0000 7f166b7fe640 1 -- 172.21.15.28:0/2774870757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166c102250 msgr2=0x7f166c19ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:20.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.851+0000 7f166b7fe640 1 --2- 172.21.15.28:0/2774870757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166c102250 0x7f166c19ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:20.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.851+0000 7f166b7fe640 1 -- 172.21.15.28:0/2774870757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f166c19f710 con 0x7f166c101890 2024-04-25T01:30:20.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.852+0000 7f166b7fe640 1 --2- 172.21.15.28:0/2774870757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f166c101890 0x7f166c19c7e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f166000d950 tx=0x7f166000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:20.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.852+0000 7f16697fa640 1 -- 172.21.15.28:0/2774870757 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660017070 con 0x7f166c101890 2024-04-25T01:30:20.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.852+0000 7f16697fa640 1 -- 172.21.15.28:0/2774870757 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1660009c30 con 0x7f166c101890 2024-04-25T01:30:20.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.852+0000 7f1672415640 1 -- 172.21.15.28:0/2774870757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f166c19f9a0 con 0x7f166c101890 2024-04-25T01:30:20.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.852+0000 7f16697fa640 1 -- 172.21.15.28:0/2774870757 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166000f6b0 con 0x7f166c101890 2024-04-25T01:30:20.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.852+0000 7f1672415640 1 -- 172.21.15.28:0/2774870757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f166c19fef0 con 0x7f166c101890 2024-04-25T01:30:20.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.853+0000 7f1672415640 1 -- 172.21.15.28:0/2774870757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f166c1079c0 con 0x7f166c101890 2024-04-25T01:30:20.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.854+0000 7f16697fa640 1 -- 172.21.15.28:0/2774870757 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f16600109d0 con 0x7f166c101890 2024-04-25T01:30:20.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.854+0000 7f16697fa640 1 --2- 172.21.15.28:0/2774870757 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f16540a2b90 0x7f16540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:20.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.855+0000 7f16697fa640 1 -- 172.21.15.28:0/2774870757 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f16600ce7f0 con 0x7f166c101890 2024-04-25T01:30:20.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.855+0000 7f166bfff640 1 --2- 172.21.15.28:0/2774870757 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f16540a2b90 0x7f16540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:20.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.855+0000 7f166bfff640 1 --2- 172.21.15.28:0/2774870757 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f16540a2b90 0x7f16540a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f165c02aa00 tx=0x7f165c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:20.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:20.859+0000 7f16697fa640 1 -- 172.21.15.28:0/2774870757 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1660009de0 con 0x7f166c101890 2024-04-25T01:30:21.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:20 smithi028 ceph-mon[29390]: Deploying daemon mds.g on smithi028 2024-04-25T01:30:21.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:20 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] up:boot 2024-04-25T01:30:21.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:20 smithi028 ceph-mon[29390]: fsmap 9 up:standby 2024-04-25T01:30:21.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:30:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:21.086+0000 7f1672415640 1 -- 172.21.15.28:0/2774870757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f166c02d3c0 con 0x7f166c101890 2024-04-25T01:30:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:21.087+0000 7f16697fa640 1 -- 172.21.15.28:0/2774870757 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7992 (secure 0 0 0) 0x7f166009bbf0 con 0x7f166c101890 2024-04-25T01:30:21.065 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:21.066 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:30:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 10 2024-04-25T01:30:21.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:21.089+0000 7f1672415640 1 -- 172.21.15.28:0/2774870757 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f16540a2b90 msgr2=0x7f16540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:21.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:21.089+0000 7f1672415640 1 --2- 172.21.15.28:0/2774870757 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f16540a2b90 0x7f16540a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f165c02aa00 tx=0x7f165c0315b0 comp rx=0 tx=0).stop 2024-04-25T01:30:21.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:21.089+0000 7f1672415640 1 -- 172.21.15.28:0/2774870757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f166c101890 msgr2=0x7f166c19c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:21.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:21.089+0000 7f1672415640 1 --2- 172.21.15.28:0/2774870757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f166c101890 0x7f166c19c7e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f166000d950 tx=0x7f166000de10 comp rx=0 tx=0).stop 2024-04-25T01:30:21.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:21.089+0000 7f1672415640 1 -- 172.21.15.28:0/2774870757 shutdown_connections 2024-04-25T01:30:21.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:21.089+0000 7f1672415640 1 --2- 172.21.15.28:0/2774870757 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f16540a2b90 0x7f16540a5050 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:21.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:21.089+0000 7f1672415640 1 --2- 172.21.15.28:0/2774870757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f166c102250 0x7f166c19ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:21.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:21.089+0000 7f1672415640 1 --2- 172.21.15.28:0/2774870757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f166c101890 0x7f166c19c7e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:21.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:21.089+0000 7f1672415640 1 --2- 172.21.15.28:0/2774870757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f166c1006a0 0x7f166c06a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:21.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:21.089+0000 7f1672415640 1 -- 172.21.15.28:0/2774870757 >> 172.21.15.28:0/2774870757 conn(0x7f166c0fbc30 msgr2=0x7f166c104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:21.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:21.089+0000 7f1672415640 1 -- 172.21.15.28:0/2774870757 shutdown_connections 2024-04-25T01:30:21.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:21.089+0000 7f1672415640 1 -- 172.21.15.28:0/2774870757 wait complete. 2024-04-25T01:30:21.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:20 smithi130 ceph-mon[32702]: Deploying daemon mds.g on smithi028 2024-04-25T01:30:21.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:20 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] up:boot 2024-04-25T01:30:21.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:20 smithi130 ceph-mon[32702]: fsmap 9 up:standby 2024-04-25T01:30:21.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:30:21.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:20 smithi032 ceph-mon[32461]: Deploying daemon mds.g on smithi028 2024-04-25T01:30:21.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:20 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] up:boot 2024-04-25T01:30:21.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:20 smithi032 ceph-mon[32461]: fsmap 9 up:standby 2024-04-25T01:30:21.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:20 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:30:21.731 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 25 01:30:21 smithi028 systemd[1]: Starting Ceph mds.g for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:30:22.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:21 smithi028 ceph-mon[29390]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:22.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:21 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2774870757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:22.005 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 25 01:30:21 smithi028 podman[68894]: 2024-04-25 01:30:21.657129638 +0000 UTC m=+0.018509612 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:30:22.005 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 25 01:30:21 smithi028 podman[68894]: 2024-04-25 01:30:21.774821259 +0000 UTC m=+0.136201229 container create d92727996a48949f3f3d170c487efcbe2b81ed316195e203a43ae91e58078a91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9) 2024-04-25T01:30:22.005 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 25 01:30:22 smithi028 podman[68894]: 2024-04-25 01:30:22.027424527 +0000 UTC m=+0.388804491 container init d92727996a48949f3f3d170c487efcbe2b81ed316195e203a43ae91e58078a91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-25T01:30:22.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:21 smithi130 ceph-mon[32702]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:22.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:21 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2774870757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:22.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:21 smithi032 ceph-mon[32461]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:22.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:21 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2774870757' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:22.312 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 25 01:30:22 smithi028 podman[68894]: 2024-04-25 01:30:22.030318766 +0000 UTC m=+0.391698737 container start d92727996a48949f3f3d170c487efcbe2b81ed316195e203a43ae91e58078a91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-25T01:30:22.312 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 25 01:30:22 smithi028 ceph-mds[68913]: -- 172.21.15.28:0/2428731945 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562d4d7dfa40 con 0x562d4d7b2c00 2024-04-25T01:30:22.312 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 25 01:30:22 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-g[68909]: starting mds.g at 2024-04-25T01:30:22.312 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 25 01:30:22 smithi028 bash[68894]: d92727996a48949f3f3d170c487efcbe2b81ed316195e203a43ae91e58078a91 2024-04-25T01:30:22.312 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 25 01:30:22 smithi028 systemd[1]: Started Ceph mds.g for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:30:23.080 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:30:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.213+0000 7fefc8112640 1 -- 172.21.15.28:0/2549190367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0107460 msgr2=0x7fefc01098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.213+0000 7fefc8112640 1 --2- 172.21.15.28:0/2549190367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0107460 0x7fefc01098b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fefbc009900 tx=0x7fefbc02a410 comp rx=0 tx=0).stop 2024-04-25T01:30:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.213+0000 7fefc8112640 1 -- 172.21.15.28:0/2549190367 shutdown_connections 2024-04-25T01:30:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.213+0000 7fefc8112640 1 --2- 172.21.15.28:0/2549190367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0107460 0x7fefc01098b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.213+0000 7fefc8112640 1 --2- 172.21.15.28:0/2549190367 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefc0104b30 0x7fefc0106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.213+0000 7fefc8112640 1 --2- 172.21.15.28:0/2549190367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefc0102200 0x7fefc01045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.213+0000 7fefc8112640 1 -- 172.21.15.28:0/2549190367 >> 172.21.15.28:0/2549190367 conn(0x7fefc00fbbf0 msgr2=0x7fefc00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.213+0000 7fefc8112640 1 -- 172.21.15.28:0/2549190367 shutdown_connections 2024-04-25T01:30:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.213+0000 7fefc8112640 1 -- 172.21.15.28:0/2549190367 wait complete. 2024-04-25T01:30:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.214+0000 7fefc8112640 1 Processor -- start 2024-04-25T01:30:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.214+0000 7fefc8112640 1 -- start start 2024-04-25T01:30:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.214+0000 7fefc8112640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefc0102200 0x7fefc0197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.214+0000 7fefc8112640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefc0104b30 0x7fefc0197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.214+0000 7fefc8112640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0107460 0x7fefc019f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.214+0000 7fefc8112640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fefc010d470 con 0x7fefc0107460 2024-04-25T01:30:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.214+0000 7fefc8112640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fefc010d2f0 con 0x7fefc0102200 2024-04-25T01:30:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.214+0000 7fefc8112640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fefc010d5f0 con 0x7fefc0104b30 2024-04-25T01:30:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.214+0000 7fefc66af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0107460 0x7fefc019f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.214+0000 7fefc66af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0107460 0x7fefc019f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45150/0 (socket says 172.21.15.28:45150) 2024-04-25T01:30:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.214+0000 7fefc66af640 1 -- 172.21.15.28:0/11945309 learned_addr learned my addr 172.21.15.28:0/11945309 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.214+0000 7fefc56ad640 1 --2- 172.21.15.28:0/11945309 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefc0104b30 0x7fefc0197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.214+0000 7fefc5eae640 1 --2- 172.21.15.28:0/11945309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefc0102200 0x7fefc0197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.214+0000 7fefc66af640 1 -- 172.21.15.28:0/11945309 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefc0104b30 msgr2=0x7fefc0197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.214+0000 7fefc66af640 1 --2- 172.21.15.28:0/11945309 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefc0104b30 0x7fefc0197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.214+0000 7fefc66af640 1 -- 172.21.15.28:0/11945309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefc0102200 msgr2=0x7fefc0197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.214+0000 7fefc66af640 1 --2- 172.21.15.28:0/11945309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefc0102200 0x7fefc0197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:23.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.214+0000 7fefc66af640 1 -- 172.21.15.28:0/11945309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefc019f670 con 0x7fefc0107460 2024-04-25T01:30:23.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.214+0000 7fefc5eae640 1 --2- 172.21.15.28:0/11945309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefc0102200 0x7fefc0197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:23.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.215+0000 7fefc66af640 1 --2- 172.21.15.28:0/11945309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0107460 0x7fefc019f000 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fefbc02a930 tx=0x7fefbc032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:23.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.215+0000 7fefb6ffd640 1 -- 172.21.15.28:0/11945309 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefbc03f030 con 0x7fefc0107460 2024-04-25T01:30:23.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.215+0000 7fefc8112640 1 -- 172.21.15.28:0/11945309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefc019f900 con 0x7fefc0107460 2024-04-25T01:30:23.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.215+0000 7fefb6ffd640 1 -- 172.21.15.28:0/11945309 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefbc032e50 con 0x7fefc0107460 2024-04-25T01:30:23.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.215+0000 7fefb6ffd640 1 -- 172.21.15.28:0/11945309 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefbc035070 con 0x7fefc0107460 2024-04-25T01:30:23.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.215+0000 7fefc8112640 1 -- 172.21.15.28:0/11945309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefc019fdf0 con 0x7fefc0107460 2024-04-25T01:30:23.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.216+0000 7fefc8112640 1 -- 172.21.15.28:0/11945309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefc00fd6e0 con 0x7fefc0107460 2024-04-25T01:30:23.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.217+0000 7fefb6ffd640 1 -- 172.21.15.28:0/11945309 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fefbc042030 con 0x7fefc0107460 2024-04-25T01:30:23.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.217+0000 7fefb6ffd640 1 --2- 172.21.15.28:0/11945309 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fefac0a2b90 0x7fefac0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:23.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.217+0000 7fefc5eae640 1 --2- 172.21.15.28:0/11945309 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fefac0a2b90 0x7fefac0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:23.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.217+0000 7fefb6ffd640 1 -- 172.21.15.28:0/11945309 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fefbc0ef120 con 0x7fefc0107460 2024-04-25T01:30:23.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.218+0000 7fefc5eae640 1 --2- 172.21.15.28:0/11945309 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fefac0a2b90 0x7fefac0a5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fefb0005d00 tx=0x7fefb0005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:23.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.222+0000 7fefb6ffd640 1 -- 172.21.15.28:0/11945309 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fefbc0bc520 con 0x7fefc0107460 2024-04-25T01:30:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.443+0000 7fefc8112640 1 -- 172.21.15.28:0/11945309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefc002d370 con 0x7fefc0107460 2024-04-25T01:30:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.444+0000 7fefb6ffd640 1 -- 172.21.15.28:0/11945309 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8818 (secure 0 0 0) 0x7fefbc0bbe20 con 0x7fefc0107460 2024-04-25T01:30:23.422 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:23.422 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:30:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 11 2024-04-25T01:30:23.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.446+0000 7fefc8112640 1 -- 172.21.15.28:0/11945309 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fefac0a2b90 msgr2=0x7fefac0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:23.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.446+0000 7fefc8112640 1 --2- 172.21.15.28:0/11945309 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fefac0a2b90 0x7fefac0a5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fefb0005d00 tx=0x7fefb0005c70 comp rx=0 tx=0).stop 2024-04-25T01:30:23.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.446+0000 7fefc8112640 1 -- 172.21.15.28:0/11945309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0107460 msgr2=0x7fefc019f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:23.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.446+0000 7fefc8112640 1 --2- 172.21.15.28:0/11945309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0107460 0x7fefc019f000 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fefbc02a930 tx=0x7fefbc032c60 comp rx=0 tx=0).stop 2024-04-25T01:30:23.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.446+0000 7fefc8112640 1 -- 172.21.15.28:0/11945309 shutdown_connections 2024-04-25T01:30:23.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.446+0000 7fefc8112640 1 --2- 172.21.15.28:0/11945309 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fefac0a2b90 0x7fefac0a5050 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:23.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.446+0000 7fefc8112640 1 --2- 172.21.15.28:0/11945309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc0107460 0x7fefc019f000 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:23.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.446+0000 7fefc8112640 1 --2- 172.21.15.28:0/11945309 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fefc0104b30 0x7fefc0197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:23.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.446+0000 7fefc8112640 1 --2- 172.21.15.28:0/11945309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fefc0102200 0x7fefc0197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.446+0000 7fefc8112640 1 -- 172.21.15.28:0/11945309 >> 172.21.15.28:0/11945309 conn(0x7fefc00fbbf0 msgr2=0x7fefc0102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.446+0000 7fefc8112640 1 -- 172.21.15.28:0/11945309 shutdown_connections 2024-04-25T01:30:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:23.446+0000 7fefc8112640 1 -- 172.21.15.28:0/11945309 wait complete. 2024-04-25T01:30:23.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:23.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:23.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:23.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:23.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:23.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:23.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:23 smithi028 ceph-mon[29390]: Deploying daemon mds.a on smithi028 2024-04-25T01:30:23.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:23 smithi028 ceph-mon[29390]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:23.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:23 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.28:6838/3206225061,v1:172.21.15.28:6839/3206225061] up:boot 2024-04-25T01:30:23.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:23 smithi028 ceph-mon[29390]: fsmap 10 up:standby 2024-04-25T01:30:23.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:23 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:30:23.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:23.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:23.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:23.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:23.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:23.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:23.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:23 smithi130 ceph-mon[32702]: Deploying daemon mds.a on smithi028 2024-04-25T01:30:23.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:23 smithi130 ceph-mon[32702]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:23.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:23 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.28:6838/3206225061,v1:172.21.15.28:6839/3206225061] up:boot 2024-04-25T01:30:23.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:23 smithi130 ceph-mon[32702]: fsmap 10 up:standby 2024-04-25T01:30:23.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:23 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:30:23.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:23.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:23.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:23.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:23.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:23.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:23.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:23 smithi032 ceph-mon[32461]: Deploying daemon mds.a on smithi028 2024-04-25T01:30:23.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:23 smithi032 ceph-mon[32461]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:23.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:23 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.28:6838/3206225061,v1:172.21.15.28:6839/3206225061] up:boot 2024-04-25T01:30:23.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:23 smithi032 ceph-mon[32461]: fsmap 10 up:standby 2024-04-25T01:30:23.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:23 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:30:24.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:24 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/11945309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:24.562 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:30:24 smithi028 systemd[1]: Starting Ceph mds.a for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:30:24.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:24 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/11945309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:24.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:24 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/11945309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:24.935 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:30:24 smithi028 podman[69241]: 2024-04-25 01:30:24.538783149 +0000 UTC m=+0.018484892 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:30:24.935 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:30:24 smithi028 podman[69241]: 2024-04-25 01:30:24.672024914 +0000 UTC m=+0.151726630 container create 739a81cfd2eb67a95fa51dd3de5787ef8439fcde5bb1117181135bc865997d10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T01:30:25.312 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:30:24 smithi028 podman[69241]: 2024-04-25 01:30:24.95785362 +0000 UTC m=+0.437555341 container init 739a81cfd2eb67a95fa51dd3de5787ef8439fcde5bb1117181135bc865997d10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9) 2024-04-25T01:30:25.312 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:30:24 smithi028 podman[69241]: 2024-04-25 01:30:24.960533919 +0000 UTC m=+0.440235642 container start 739a81cfd2eb67a95fa51dd3de5787ef8439fcde5bb1117181135bc865997d10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-25T01:30:25.312 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:30:24 smithi028 ceph-mds[69261]: -- 172.21.15.28:0/2529165405 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b7d11eda40 con 0x55b7d11c0800 2024-04-25T01:30:25.312 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:30:24 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-a[69257]: starting mds.a at 2024-04-25T01:30:25.312 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:30:25 smithi028 bash[69241]: 739a81cfd2eb67a95fa51dd3de5787ef8439fcde5bb1117181135bc865997d10 2024-04-25T01:30:25.312 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:30:25 smithi028 systemd[1]: Started Ceph mds.a for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:30:25.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:30:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.567+0000 7f1a6de38640 1 -- 172.21.15.28:0/4179869840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a681017b0 msgr2=0x7f1a68101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.567+0000 7f1a6de38640 1 --2- 172.21.15.28:0/4179869840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a681017b0 0x7f1a68101bd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1a58009900 tx=0x7f1a5802a410 comp rx=0 tx=0).stop 2024-04-25T01:30:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.567+0000 7f1a6de38640 1 -- 172.21.15.28:0/4179869840 shutdown_connections 2024-04-25T01:30:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.567+0000 7f1a6de38640 1 --2- 172.21.15.28:0/4179869840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a68103360 0x7f1a68109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.567+0000 7f1a6de38640 1 --2- 172.21.15.28:0/4179869840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a681029a0 0x7f1a68102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.567+0000 7f1a6de38640 1 --2- 172.21.15.28:0/4179869840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a681017b0 0x7f1a68101bd0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.567+0000 7f1a6de38640 1 -- 172.21.15.28:0/4179869840 >> 172.21.15.28:0/4179869840 conn(0x7f1a680fcd40 msgr2=0x7f1a680ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.567+0000 7f1a6de38640 1 -- 172.21.15.28:0/4179869840 shutdown_connections 2024-04-25T01:30:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.567+0000 7f1a6de38640 1 -- 172.21.15.28:0/4179869840 wait complete. 2024-04-25T01:30:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a6de38640 1 Processor -- start 2024-04-25T01:30:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a6de38640 1 -- start start 2024-04-25T01:30:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a6de38640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a681017b0 0x7f1a68197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a6de38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a681029a0 0x7f1a68197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a6de38640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a68103360 0x7f1a6819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a6de38640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6810d400 con 0x7f1a681029a0 2024-04-25T01:30:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a6de38640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6810d280 con 0x7f1a681017b0 2024-04-25T01:30:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a6de38640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6810d580 con 0x7f1a68103360 2024-04-25T01:30:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a66ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a681029a0 0x7f1a68197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a66ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a681029a0 0x7f1a68197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45184/0 (socket says 172.21.15.28:45184) 2024-04-25T01:30:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a66ffd640 1 -- 172.21.15.28:0/207912558 learned_addr learned my addr 172.21.15.28:0/207912558 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a677fe640 1 --2- 172.21.15.28:0/207912558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a681017b0 0x7f1a68197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a67fff640 1 --2- 172.21.15.28:0/207912558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a68103360 0x7f1a6819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a66ffd640 1 -- 172.21.15.28:0/207912558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a68103360 msgr2=0x7f1a6819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a66ffd640 1 --2- 172.21.15.28:0/207912558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a68103360 0x7f1a6819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a66ffd640 1 -- 172.21.15.28:0/207912558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a681017b0 msgr2=0x7f1a68197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a66ffd640 1 --2- 172.21.15.28:0/207912558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a681017b0 0x7f1a68197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a66ffd640 1 -- 172.21.15.28:0/207912558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a6819f6d0 con 0x7f1a681029a0 2024-04-25T01:30:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a67fff640 1 --2- 172.21.15.28:0/207912558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a68103360 0x7f1a6819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.568+0000 7f1a677fe640 1 --2- 172.21.15.28:0/207912558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a681017b0 0x7f1a68197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.569+0000 7f1a66ffd640 1 --2- 172.21.15.28:0/207912558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a681029a0 0x7f1a68197ff0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c00e9f0 tx=0x7f1a5c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.569+0000 7f1a64ff9640 1 -- 172.21.15.28:0/207912558 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c00cd80 con 0x7f1a681029a0 2024-04-25T01:30:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.569+0000 7f1a64ff9640 1 -- 172.21.15.28:0/207912558 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a5c010910 con 0x7f1a681029a0 2024-04-25T01:30:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.569+0000 7f1a6de38640 1 -- 172.21.15.28:0/207912558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a6819f9c0 con 0x7f1a681029a0 2024-04-25T01:30:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.569+0000 7f1a64ff9640 1 -- 172.21.15.28:0/207912558 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c018990 con 0x7f1a681029a0 2024-04-25T01:30:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.569+0000 7f1a6de38640 1 -- 172.21.15.28:0/207912558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a6819ff10 con 0x7f1a681029a0 2024-04-25T01:30:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.570+0000 7f1a6de38640 1 -- 172.21.15.28:0/207912558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a68108a70 con 0x7f1a681029a0 2024-04-25T01:30:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.571+0000 7f1a64ff9640 1 -- 172.21.15.28:0/207912558 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1a5c010ac0 con 0x7f1a681029a0 2024-04-25T01:30:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.572+0000 7f1a64ff9640 1 --2- 172.21.15.28:0/207912558 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1a500a2b90 0x7f1a500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.572+0000 7f1a677fe640 1 --2- 172.21.15.28:0/207912558 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1a500a2b90 0x7f1a500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.572+0000 7f1a64ff9640 1 -- 172.21.15.28:0/207912558 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f1a5c0cf3a0 con 0x7f1a681029a0 2024-04-25T01:30:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.572+0000 7f1a677fe640 1 --2- 172.21.15.28:0/207912558 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1a500a2b90 0x7f1a500a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1a58009230 tx=0x7f1a58031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.576+0000 7f1a64ff9640 1 -- 172.21.15.28:0/207912558 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1a5c09c7a0 con 0x7f1a681029a0 2024-04-25T01:30:25.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.803+0000 7f1a6de38640 1 -- 172.21.15.28:0/207912558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a6802d370 con 0x7f1a681029a0 2024-04-25T01:30:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.804+0000 7f1a64ff9640 1 -- 172.21.15.28:0/207912558 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9644 (secure 0 0 0) 0x7f1a5c09c0a0 con 0x7f1a681029a0 2024-04-25T01:30:25.783 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:25.783 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:30:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 12 2024-04-25T01:30:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.806+0000 7f1a6de38640 1 -- 172.21.15.28:0/207912558 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1a500a2b90 msgr2=0x7f1a500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.806+0000 7f1a6de38640 1 --2- 172.21.15.28:0/207912558 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1a500a2b90 0x7f1a500a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1a58009230 tx=0x7f1a58031400 comp rx=0 tx=0).stop 2024-04-25T01:30:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.806+0000 7f1a6de38640 1 -- 172.21.15.28:0/207912558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a681029a0 msgr2=0x7f1a68197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.806+0000 7f1a6de38640 1 --2- 172.21.15.28:0/207912558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a681029a0 0x7f1a68197ff0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c00e9f0 tx=0x7f1a5c00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:30:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.806+0000 7f1a6de38640 1 -- 172.21.15.28:0/207912558 shutdown_connections 2024-04-25T01:30:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.806+0000 7f1a6de38640 1 --2- 172.21.15.28:0/207912558 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1a500a2b90 0x7f1a500a5050 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.806+0000 7f1a6de38640 1 --2- 172.21.15.28:0/207912558 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a68103360 0x7f1a6819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.806+0000 7f1a6de38640 1 --2- 172.21.15.28:0/207912558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a681029a0 0x7f1a68197ff0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.806+0000 7f1a6de38640 1 --2- 172.21.15.28:0/207912558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a681017b0 0x7f1a68197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.806+0000 7f1a6de38640 1 -- 172.21.15.28:0/207912558 >> 172.21.15.28:0/207912558 conn(0x7f1a680fcd40 msgr2=0x7f1a68105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.806+0000 7f1a6de38640 1 -- 172.21.15.28:0/207912558 shutdown_connections 2024-04-25T01:30:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:25.806+0000 7f1a6de38640 1 -- 172.21.15.28:0/207912558 wait complete. 2024-04-25T01:30:26.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:26 smithi130 ceph-mon[32702]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:26.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:26 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:26.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:26 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:26.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:26 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:26.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:26 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:26.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:26 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:26.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:26 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:26.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:26 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.28:6840/3666045393,v1:172.21.15.28:6841/3666045393] up:boot 2024-04-25T01:30:26.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:26 smithi130 ceph-mon[32702]: fsmap 11 up:standby 2024-04-25T01:30:26.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:26 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:30:26.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:26 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/207912558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:26.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:26 smithi032 ceph-mon[32461]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:26.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:26 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:26.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:26 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:26.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:26 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:26.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:26 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:26.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:26 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:26.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:26 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:26.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:26 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.28:6840/3666045393,v1:172.21.15.28:6841/3666045393] up:boot 2024-04-25T01:30:26.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:26 smithi032 ceph-mon[32461]: fsmap 11 up:standby 2024-04-25T01:30:26.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:26 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:30:26.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:26 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/207912558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:26.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:26 smithi028 ceph-mon[29390]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:26.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:26 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:26.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:26 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:26.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:26 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:26.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:26 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:30:26.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:26 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:30:26.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:26 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:26.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:26 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.28:6840/3666045393,v1:172.21.15.28:6841/3666045393] up:boot 2024-04-25T01:30:26.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:26 smithi028 ceph-mon[29390]: fsmap 11 up:standby 2024-04-25T01:30:26.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:26 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:30:26.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:26 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/207912558' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:27.394 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:27 smithi032 ceph-mon[32461]: Deploying daemon mds.b on smithi032 2024-04-25T01:30:27.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:27 smithi130 ceph-mon[32702]: Deploying daemon mds.b on smithi032 2024-04-25T01:30:27.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:27 smithi028 ceph-mon[29390]: Deploying daemon mds.b on smithi032 2024-04-25T01:30:27.734 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:30:27 smithi032 systemd[1]: Starting Ceph mds.b for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:30:27.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:30:27.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.933+0000 7f68eef9e640 1 -- 172.21.15.28:0/1248358956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e8100680 msgr2=0x7f68e8100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:27.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.933+0000 7f68eef9e640 1 --2- 172.21.15.28:0/1248358956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e8100680 0x7f68e8100aa0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f68d8009900 tx=0x7f68d802a410 comp rx=0 tx=0).stop 2024-04-25T01:30:27.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.934+0000 7f68eef9e640 1 -- 172.21.15.28:0/1248358956 shutdown_connections 2024-04-25T01:30:27.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.934+0000 7f68eef9e640 1 --2- 172.21.15.28:0/1248358956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68e8102230 0x7f68e8108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:27.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.934+0000 7f68eef9e640 1 --2- 172.21.15.28:0/1248358956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68e8101870 0x7f68e8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:27.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.934+0000 7f68eef9e640 1 --2- 172.21.15.28:0/1248358956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e8100680 0x7f68e8100aa0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:27.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.934+0000 7f68eef9e640 1 -- 172.21.15.28:0/1248358956 >> 172.21.15.28:0/1248358956 conn(0x7f68e80fbbf0 msgr2=0x7f68e80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:27.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.934+0000 7f68eef9e640 1 -- 172.21.15.28:0/1248358956 shutdown_connections 2024-04-25T01:30:27.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.934+0000 7f68eef9e640 1 -- 172.21.15.28:0/1248358956 wait complete. 2024-04-25T01:30:27.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.934+0000 7f68eef9e640 1 Processor -- start 2024-04-25T01:30:27.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.934+0000 7f68eef9e640 1 -- start start 2024-04-25T01:30:27.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.934+0000 7f68eef9e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68e8100680 0x7f68e8104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:27.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.934+0000 7f68eef9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e8101870 0x7f68e8102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:27.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.934+0000 7f68eef9e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68e8102230 0x7f68e8103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:27.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.934+0000 7f68eef9e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f68e810c2d0 con 0x7f68e8101870 2024-04-25T01:30:27.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.934+0000 7f68eef9e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f68e810c150 con 0x7f68e8102230 2024-04-25T01:30:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.934+0000 7f68eef9e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f68e810c450 con 0x7f68e8100680 2024-04-25T01:30:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.934+0000 7f68e7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e8101870 0x7f68e8102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.934+0000 7f68e7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e8101870 0x7f68e8102c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55548/0 (socket says 172.21.15.28:55548) 2024-04-25T01:30:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.935+0000 7f68e7fff640 1 -- 172.21.15.28:0/2975142536 learned_addr learned my addr 172.21.15.28:0/2975142536 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.935+0000 7f68ed53b640 1 --2- 172.21.15.28:0/2975142536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68e8102230 0x7f68e8103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.935+0000 7f68ecd3a640 1 --2- 172.21.15.28:0/2975142536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68e8100680 0x7f68e8104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.935+0000 7f68e7fff640 1 -- 172.21.15.28:0/2975142536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68e8100680 msgr2=0x7f68e8104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.935+0000 7f68e7fff640 1 --2- 172.21.15.28:0/2975142536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68e8100680 0x7f68e8104610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.935+0000 7f68e7fff640 1 -- 172.21.15.28:0/2975142536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68e8102230 msgr2=0x7f68e8103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.935+0000 7f68e7fff640 1 --2- 172.21.15.28:0/2975142536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68e8102230 0x7f68e8103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.935+0000 7f68e7fff640 1 -- 172.21.15.28:0/2975142536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68e8103a20 con 0x7f68e8101870 2024-04-25T01:30:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.935+0000 7f68e7fff640 1 --2- 172.21.15.28:0/2975142536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e8101870 0x7f68e8102c50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f68dc00d950 tx=0x7f68dc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:27.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.935+0000 7f68e5ffb640 1 -- 172.21.15.28:0/2975142536 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68dc017030 con 0x7f68e8101870 2024-04-25T01:30:27.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.935+0000 7f68e5ffb640 1 -- 172.21.15.28:0/2975142536 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68dc004020 con 0x7f68e8101870 2024-04-25T01:30:27.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.935+0000 7f68eef9e640 1 -- 172.21.15.28:0/2975142536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68e81a1c10 con 0x7f68e8101870 2024-04-25T01:30:27.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.935+0000 7f68e5ffb640 1 -- 172.21.15.28:0/2975142536 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68dc0047a0 con 0x7f68e8101870 2024-04-25T01:30:27.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.935+0000 7f68eef9e640 1 -- 172.21.15.28:0/2975142536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68e81a2160 con 0x7f68e8101870 2024-04-25T01:30:27.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.936+0000 7f68eef9e640 1 -- 172.21.15.28:0/2975142536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68e81a2390 con 0x7f68e8101870 2024-04-25T01:30:27.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.937+0000 7f68e5ffb640 1 -- 172.21.15.28:0/2975142536 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f68dc0041d0 con 0x7f68e8101870 2024-04-25T01:30:27.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.938+0000 7f68e5ffb640 1 --2- 172.21.15.28:0/2975142536 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f68d00a2b90 0x7f68d00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:27.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.938+0000 7f68ecd3a640 1 --2- 172.21.15.28:0/2975142536 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f68d00a2b90 0x7f68d00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:27.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.938+0000 7f68e5ffb640 1 -- 172.21.15.28:0/2975142536 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f68dc0ce700 con 0x7f68e8101870 2024-04-25T01:30:27.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.939+0000 7f68ecd3a640 1 --2- 172.21.15.28:0/2975142536 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f68d00a2b90 0x7f68d00a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f68d8006010 tx=0x7f68d8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:27.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:27.943+0000 7f68e5ffb640 1 -- 172.21.15.28:0/2975142536 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f68dc09ba60 con 0x7f68e8101870 2024-04-25T01:30:28.087 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:30:27 smithi032 podman[63144]: 2024-04-25 01:30:27.741388332 +0000 UTC m=+0.018871288 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:30:28.087 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:30:27 smithi032 podman[63144]: 2024-04-25 01:30:27.838358281 +0000 UTC m=+0.115841228 container create f3b2a1d13e2fead614dc5b171c74383f7c1ef20d1f1f0b438a3f7d78a2d184e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-b, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-25T01:30:28.087 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:30:28 smithi032 podman[63144]: 2024-04-25 01:30:28.082964733 +0000 UTC m=+0.360447678 container init f3b2a1d13e2fead614dc5b171c74383f7c1ef20d1f1f0b438a3f7d78a2d184e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=squid-d2c45d9, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-25T01:30:28.087 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:30:28 smithi032 podman[63144]: 2024-04-25 01:30:28.085653905 +0000 UTC m=+0.363136849 container start f3b2a1d13e2fead614dc5b171c74383f7c1ef20d1f1f0b438a3f7d78a2d184e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True) 2024-04-25T01:30:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:28.168+0000 7f68eef9e640 1 -- 172.21.15.28:0/2975142536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68e81a2720 con 0x7f68e8101870 2024-04-25T01:30:28.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:28.169+0000 7f68e5ffb640 1 -- 172.21.15.28:0/2975142536 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9644 (secure 0 0 0) 0x7f68dc09b360 con 0x7f68e8101870 2024-04-25T01:30:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:28.147 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:30:28.148 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 12 2024-04-25T01:30:28.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:28.171+0000 7f68eef9e640 1 -- 172.21.15.28:0/2975142536 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f68d00a2b90 msgr2=0x7f68d00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:28.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:28.171+0000 7f68eef9e640 1 --2- 172.21.15.28:0/2975142536 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f68d00a2b90 0x7f68d00a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f68d8006010 tx=0x7f68d8000bc0 comp rx=0 tx=0).stop 2024-04-25T01:30:28.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:28.171+0000 7f68eef9e640 1 -- 172.21.15.28:0/2975142536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e8101870 msgr2=0x7f68e8102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:28.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:28.171+0000 7f68eef9e640 1 --2- 172.21.15.28:0/2975142536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e8101870 0x7f68e8102c50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f68dc00d950 tx=0x7f68dc00de10 comp rx=0 tx=0).stop 2024-04-25T01:30:28.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:28.171+0000 7f68eef9e640 1 -- 172.21.15.28:0/2975142536 shutdown_connections 2024-04-25T01:30:28.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:28.171+0000 7f68eef9e640 1 --2- 172.21.15.28:0/2975142536 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f68d00a2b90 0x7f68d00a5050 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:28.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:28.171+0000 7f68eef9e640 1 --2- 172.21.15.28:0/2975142536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68e8102230 0x7f68e8103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:28.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:28.171+0000 7f68eef9e640 1 --2- 172.21.15.28:0/2975142536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68e8101870 0x7f68e8102c50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:28.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:28.171+0000 7f68eef9e640 1 --2- 172.21.15.28:0/2975142536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f68e8100680 0x7f68e8104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:28.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:28.171+0000 7f68eef9e640 1 -- 172.21.15.28:0/2975142536 >> 172.21.15.28:0/2975142536 conn(0x7f68e80fbbf0 msgr2=0x7f68e80fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:28.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:28.171+0000 7f68eef9e640 1 -- 172.21.15.28:0/2975142536 shutdown_connections 2024-04-25T01:30:28.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:28.171+0000 7f68eef9e640 1 -- 172.21.15.28:0/2975142536 wait complete. 2024-04-25T01:30:28.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:28 smithi130 ceph-mon[32702]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:28.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:28 smithi032 ceph-mon[32461]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:28.484 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:30:28 smithi032 ceph-mds[63164]: -- 172.21.15.32:0/1500843334 <== mon.2 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5583daf5ba40 con 0x5583daf2ec00 2024-04-25T01:30:28.484 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:30:28 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-b[63160]: starting mds.b at 2024-04-25T01:30:28.484 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:30:28 smithi032 bash[63144]: f3b2a1d13e2fead614dc5b171c74383f7c1ef20d1f1f0b438a3f7d78a2d184e6 2024-04-25T01:30:28.484 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:30:28 smithi032 systemd[1]: Started Ceph mds.b for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:30:28.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:28 smithi028 ceph-mon[29390]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:29.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:29 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2975142536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:29.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:29 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:29.372 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:29 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:29.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:29 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:29.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:29 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:29.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:29 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:29.373 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:29 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:30:29.406 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:29 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2975142536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:29.406 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:29 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:29.406 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:29 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:29.407 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:29 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:29.407 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:29 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:29.407 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:29 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:29.407 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:29 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:30:29.440 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:29 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2975142536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:29.440 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:29 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:29.440 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:29 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:29.440 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:29 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:29.440 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:29 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:29.440 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:29 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:29.441 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:29 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:30:30.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:30:30.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.290+0000 7f8adc7e7640 1 -- 172.21.15.28:0/3822400293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ad4102230 msgr2=0x7f8ad4108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:30.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.290+0000 7f8adc7e7640 1 --2- 172.21.15.28:0/3822400293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ad4102230 0x7f8ad4108ac0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8ac4009900 tx=0x7f8ac402a410 comp rx=0 tx=0).stop 2024-04-25T01:30:30.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.290+0000 7f8adc7e7640 1 -- 172.21.15.28:0/3822400293 shutdown_connections 2024-04-25T01:30:30.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.290+0000 7f8adc7e7640 1 --2- 172.21.15.28:0/3822400293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ad4102230 0x7f8ad4108ac0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.290+0000 7f8adc7e7640 1 --2- 172.21.15.28:0/3822400293 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ad4101870 0x7f8ad4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.290+0000 7f8adc7e7640 1 --2- 172.21.15.28:0/3822400293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad4100680 0x7f8ad4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.290+0000 7f8adc7e7640 1 -- 172.21.15.28:0/3822400293 >> 172.21.15.28:0/3822400293 conn(0x7f8ad40fbbf0 msgr2=0x7f8ad40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:30.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.290+0000 7f8adc7e7640 1 -- 172.21.15.28:0/3822400293 shutdown_connections 2024-04-25T01:30:30.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.290+0000 7f8adc7e7640 1 -- 172.21.15.28:0/3822400293 wait complete. 2024-04-25T01:30:30.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.290+0000 7f8adc7e7640 1 Processor -- start 2024-04-25T01:30:30.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.291+0000 7f8adc7e7640 1 -- start start 2024-04-25T01:30:30.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.291+0000 7f8adc7e7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad4100680 0x7f8ad40694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:30.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.291+0000 7f8adc7e7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ad4101870 0x7f8ad419c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:30.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.291+0000 7f8adc7e7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ad4102230 0x7f8ad419eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:30.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.291+0000 7f8adc7e7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad410c300 con 0x7f8ad4102230 2024-04-25T01:30:30.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.291+0000 7f8adc7e7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad410c180 con 0x7f8ad4100680 2024-04-25T01:30:30.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.291+0000 7f8adc7e7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad410c480 con 0x7f8ad4101870 2024-04-25T01:30:30.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.291+0000 7f8adad84640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ad4102230 0x7f8ad419eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:30.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.291+0000 7f8adad84640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ad4102230 0x7f8ad419eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55560/0 (socket says 172.21.15.28:55560) 2024-04-25T01:30:30.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.291+0000 7f8adad84640 1 -- 172.21.15.28:0/1420382526 learned_addr learned my addr 172.21.15.28:0/1420382526 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:30.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.291+0000 7f8ada583640 1 --2- 172.21.15.28:0/1420382526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad4100680 0x7f8ad40694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:30.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.291+0000 7f8ad9d82640 1 --2- 172.21.15.28:0/1420382526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ad4101870 0x7f8ad419c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:30.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.291+0000 7f8ada583640 1 -- 172.21.15.28:0/1420382526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ad4101870 msgr2=0x7f8ad419c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:30.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.291+0000 7f8ada583640 1 --2- 172.21.15.28:0/1420382526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ad4101870 0x7f8ad419c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.291+0000 7f8ada583640 1 -- 172.21.15.28:0/1420382526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ad4102230 msgr2=0x7f8ad419eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:30.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.291+0000 7f8ada583640 1 --2- 172.21.15.28:0/1420382526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ad4102230 0x7f8ad419eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.291+0000 7f8ada583640 1 -- 172.21.15.28:0/1420382526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad419f660 con 0x7f8ad4100680 2024-04-25T01:30:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.291+0000 7f8adad84640 1 --2- 172.21.15.28:0/1420382526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ad4102230 0x7f8ad419eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.291+0000 7f8ada583640 1 --2- 172.21.15.28:0/1420382526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad4100680 0x7f8ad40694a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8acc005fd0 tx=0x7f8acc00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.292+0000 7f8ad37fe640 1 -- 172.21.15.28:0/1420382526 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8acc010630 con 0x7f8ad4100680 2024-04-25T01:30:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.292+0000 7f8ad37fe640 1 -- 172.21.15.28:0/1420382526 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8acc010cb0 con 0x7f8ad4100680 2024-04-25T01:30:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.292+0000 7f8adc7e7640 1 -- 172.21.15.28:0/1420382526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ad419f950 con 0x7f8ad4100680 2024-04-25T01:30:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.292+0000 7f8ad37fe640 1 -- 172.21.15.28:0/1420382526 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8acc018d90 con 0x7f8ad4100680 2024-04-25T01:30:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.292+0000 7f8adc7e7640 1 -- 172.21.15.28:0/1420382526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ad419fef0 con 0x7f8ad4100680 2024-04-25T01:30:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.293+0000 7f8adc7e7640 1 -- 172.21.15.28:0/1420382526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ad41079a0 con 0x7f8ad4100680 2024-04-25T01:30:30.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.294+0000 7f8ad37fe640 1 -- 172.21.15.28:0/1420382526 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8acc01f020 con 0x7f8ad4100680 2024-04-25T01:30:30.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.294+0000 7f8ad37fe640 1 --2- 172.21.15.28:0/1420382526 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8abc0a2c10 0x7f8abc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:30.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.294+0000 7f8ad37fe640 1 -- 172.21.15.28:0/1420382526 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f8acc0cf0a0 con 0x7f8ad4100680 2024-04-25T01:30:30.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.294+0000 7f8ad9d82640 1 --2- 172.21.15.28:0/1420382526 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8abc0a2c10 0x7f8abc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:30.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.295+0000 7f8ad9d82640 1 --2- 172.21.15.28:0/1420382526 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8abc0a2c10 0x7f8abc0a50d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4069210 tx=0x7f8ac8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:30.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.299+0000 7f8ad37fe640 1 -- 172.21.15.28:0/1420382526 <== mon.2 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8acc053740 con 0x7f8ad4100680 2024-04-25T01:30:30.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:30 smithi130 ceph-mon[32702]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:30.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:30 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] up:boot 2024-04-25T01:30:30.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:30 smithi130 ceph-mon[32702]: fsmap 12 up:standby 2024-04-25T01:30:30.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:30 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:30:30.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:30 smithi032 ceph-mon[32461]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:30.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:30 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] up:boot 2024-04-25T01:30:30.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:30 smithi032 ceph-mon[32461]: fsmap 12 up:standby 2024-04-25T01:30:30.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:30 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:30:30.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.524+0000 7f8adc7e7640 1 -- 172.21.15.28:0/1420382526 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ad402d400 con 0x7f8ad4100680 2024-04-25T01:30:30.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.525+0000 7f8ad37fe640 1 -- 172.21.15.28:0/1420382526 <== mon.2 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10470 (secure 0 0 0) 0x7f8acc09c480 con 0x7f8ad4100680 2024-04-25T01:30:30.503 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:30.504 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24512,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-25T01:30:30.504 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 13 2024-04-25T01:30:30.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.527+0000 7f8adc7e7640 1 -- 172.21.15.28:0/1420382526 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8abc0a2c10 msgr2=0x7f8abc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.527+0000 7f8adc7e7640 1 --2- 172.21.15.28:0/1420382526 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8abc0a2c10 0x7f8abc0a50d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4069210 tx=0x7f8ac8005f90 comp rx=0 tx=0).stop 2024-04-25T01:30:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.527+0000 7f8adc7e7640 1 -- 172.21.15.28:0/1420382526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad4100680 msgr2=0x7f8ad40694a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.527+0000 7f8adc7e7640 1 --2- 172.21.15.28:0/1420382526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad4100680 0x7f8ad40694a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8acc005fd0 tx=0x7f8acc00c6a0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.527+0000 7f8adc7e7640 1 -- 172.21.15.28:0/1420382526 shutdown_connections 2024-04-25T01:30:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.527+0000 7f8adc7e7640 1 --2- 172.21.15.28:0/1420382526 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8abc0a2c10 0x7f8abc0a50d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.527+0000 7f8adc7e7640 1 --2- 172.21.15.28:0/1420382526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ad4102230 0x7f8ad419eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.527+0000 7f8adc7e7640 1 --2- 172.21.15.28:0/1420382526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ad4101870 0x7f8ad419c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.527+0000 7f8adc7e7640 1 --2- 172.21.15.28:0/1420382526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ad4100680 0x7f8ad40694a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.527+0000 7f8adc7e7640 1 -- 172.21.15.28:0/1420382526 >> 172.21.15.28:0/1420382526 conn(0x7f8ad40fbbf0 msgr2=0x7f8ad4104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.527+0000 7f8adc7e7640 1 -- 172.21.15.28:0/1420382526 shutdown_connections 2024-04-25T01:30:30.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.527+0000 7f8adc7e7640 1 -- 172.21.15.28:0/1420382526 wait complete. 2024-04-25T01:30:30.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:30 smithi028 ceph-mon[29390]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:30.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:30 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] up:boot 2024-04-25T01:30:30.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:30 smithi028 ceph-mon[29390]: fsmap 12 up:standby 2024-04-25T01:30:30.515 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:30 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:30:30.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:30:30.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.644+0000 7f45316fc640 1 -- 172.21.15.28:0/2126347452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f452c101870 msgr2=0x7f452c101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:30.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.644+0000 7f45316fc640 1 --2- 172.21.15.28:0/2126347452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f452c101870 0x7f452c101cf0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f451c009900 tx=0x7f451c02a410 comp rx=0 tx=0).stop 2024-04-25T01:30:30.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.644+0000 7f45316fc640 1 -- 172.21.15.28:0/2126347452 shutdown_connections 2024-04-25T01:30:30.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.644+0000 7f45316fc640 1 --2- 172.21.15.28:0/2126347452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c102230 0x7f452c108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.644+0000 7f45316fc640 1 --2- 172.21.15.28:0/2126347452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f452c101870 0x7f452c101cf0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.644+0000 7f45316fc640 1 --2- 172.21.15.28:0/2126347452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f452c100680 0x7f452c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.644+0000 7f45316fc640 1 -- 172.21.15.28:0/2126347452 >> 172.21.15.28:0/2126347452 conn(0x7f452c0fbbf0 msgr2=0x7f452c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:30.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.644+0000 7f45316fc640 1 -- 172.21.15.28:0/2126347452 shutdown_connections 2024-04-25T01:30:30.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.644+0000 7f45316fc640 1 -- 172.21.15.28:0/2126347452 wait complete. 2024-04-25T01:30:30.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.645+0000 7f45316fc640 1 Processor -- start 2024-04-25T01:30:30.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.645+0000 7f45316fc640 1 -- start start 2024-04-25T01:30:30.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.645+0000 7f45316fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f452c100680 0x7f452c104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:30.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.645+0000 7f45316fc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f452c101870 0x7f452c102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:30.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.645+0000 7f45316fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c102230 0x7f452c103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:30.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.645+0000 7f45316fc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f452c10c300 con 0x7f452c100680 2024-04-25T01:30:30.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.645+0000 7f45316fc640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f452c10c180 con 0x7f452c102230 2024-04-25T01:30:30.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.645+0000 7f45316fc640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f452c10c480 con 0x7f452c101870 2024-04-25T01:30:30.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.645+0000 7f452affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f452c100680 0x7f452c104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:30.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.645+0000 7f452affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f452c100680 0x7f452c104610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55584/0 (socket says 172.21.15.28:55584) 2024-04-25T01:30:30.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.645+0000 7f452affd640 1 -- 172.21.15.28:0/2257165174 learned_addr learned my addr 172.21.15.28:0/2257165174 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:30.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.645+0000 7f452b7fe640 1 --2- 172.21.15.28:0/2257165174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c102230 0x7f452c103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:30.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.645+0000 7f452a7fc640 1 --2- 172.21.15.28:0/2257165174 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f452c101870 0x7f452c102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:30.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.646+0000 7f452affd640 1 -- 172.21.15.28:0/2257165174 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f452c101870 msgr2=0x7f452c102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:30.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.646+0000 7f452affd640 1 --2- 172.21.15.28:0/2257165174 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f452c101870 0x7f452c102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.646+0000 7f452affd640 1 -- 172.21.15.28:0/2257165174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c102230 msgr2=0x7f452c103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:30.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.646+0000 7f452affd640 1 --2- 172.21.15.28:0/2257165174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c102230 0x7f452c103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.646+0000 7f452affd640 1 -- 172.21.15.28:0/2257165174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452c103a20 con 0x7f452c100680 2024-04-25T01:30:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.646+0000 7f452affd640 1 --2- 172.21.15.28:0/2257165174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f452c100680 0x7f452c104610 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f452000d950 tx=0x7f452000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.646+0000 7f4513fff640 1 -- 172.21.15.28:0/2257165174 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4520017070 con 0x7f452c100680 2024-04-25T01:30:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.646+0000 7f4513fff640 1 -- 172.21.15.28:0/2257165174 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4520009c30 con 0x7f452c100680 2024-04-25T01:30:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.646+0000 7f45316fc640 1 -- 172.21.15.28:0/2257165174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f452c1a1bc0 con 0x7f452c100680 2024-04-25T01:30:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.646+0000 7f4513fff640 1 -- 172.21.15.28:0/2257165174 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f452000f600 con 0x7f452c100680 2024-04-25T01:30:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.646+0000 7f45316fc640 1 -- 172.21.15.28:0/2257165174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f452c1a2090 con 0x7f452c100680 2024-04-25T01:30:30.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.647+0000 7f45316fc640 1 -- 172.21.15.28:0/2257165174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f452c068560 con 0x7f452c100680 2024-04-25T01:30:30.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.648+0000 7f4513fff640 1 -- 172.21.15.28:0/2257165174 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f452001a070 con 0x7f452c100680 2024-04-25T01:30:30.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.649+0000 7f4513fff640 1 --2- 172.21.15.28:0/2257165174 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f45140a2b90 0x7f45140a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:30.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.649+0000 7f452a7fc640 1 --2- 172.21.15.28:0/2257165174 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f45140a2b90 0x7f45140a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:30.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.649+0000 7f4513fff640 1 -- 172.21.15.28:0/2257165174 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f45200ce630 con 0x7f452c100680 2024-04-25T01:30:30.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.650+0000 7f452a7fc640 1 --2- 172.21.15.28:0/2257165174 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f45140a2b90 0x7f45140a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f451c0314e0 tx=0x7f451c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.654+0000 7f4513fff640 1 -- 172.21.15.28:0/2257165174 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4520009de0 con 0x7f452c100680 2024-04-25T01:30:30.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.859+0000 7f45316fc640 1 -- 172.21.15.28:0/2257165174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f452c1a29c0 con 0x7f452c100680 2024-04-25T01:30:30.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.860+0000 7f4513fff640 1 -- 172.21.15.28:0/2257165174 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18421 (secure 0 0 0) 0x7f452009b340 con 0x7f452c100680 2024-04-25T01:30:30.838 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:30.839 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":83,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:29:42.122155+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:30:30.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.862+0000 7f45316fc640 1 -- 172.21.15.28:0/2257165174 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f45140a2b90 msgr2=0x7f45140a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:30.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.862+0000 7f45316fc640 1 --2- 172.21.15.28:0/2257165174 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f45140a2b90 0x7f45140a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f451c0314e0 tx=0x7f451c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.862+0000 7f45316fc640 1 -- 172.21.15.28:0/2257165174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f452c100680 msgr2=0x7f452c104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:30.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.862+0000 7f45316fc640 1 --2- 172.21.15.28:0/2257165174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f452c100680 0x7f452c104610 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f452000d950 tx=0x7f452000de10 comp rx=0 tx=0).stop 2024-04-25T01:30:30.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.862+0000 7f45316fc640 1 -- 172.21.15.28:0/2257165174 shutdown_connections 2024-04-25T01:30:30.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.862+0000 7f45316fc640 1 --2- 172.21.15.28:0/2257165174 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f45140a2b90 0x7f45140a5050 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.862+0000 7f45316fc640 1 --2- 172.21.15.28:0/2257165174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f452c102230 0x7f452c103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.862+0000 7f45316fc640 1 --2- 172.21.15.28:0/2257165174 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f452c101870 0x7f452c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.862+0000 7f45316fc640 1 --2- 172.21.15.28:0/2257165174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f452c100680 0x7f452c104610 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.862+0000 7f45316fc640 1 -- 172.21.15.28:0/2257165174 >> 172.21.15.28:0/2257165174 conn(0x7f452c0fbbf0 msgr2=0x7f452c0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:30.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.862+0000 7f45316fc640 1 -- 172.21.15.28:0/2257165174 shutdown_connections 2024-04-25T01:30:30.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.862+0000 7f45316fc640 1 -- 172.21.15.28:0/2257165174 wait complete. 2024-04-25T01:30:30.851 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:25:48.767414+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:30:30.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:30:30.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.984+0000 7ff10886d640 1 -- 172.21.15.28:0/2745940318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1041008f0 msgr2=0x7ff104100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.984+0000 7ff10886d640 1 --2- 172.21.15.28:0/2745940318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1041008f0 0x7ff104100d70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec009900 tx=0x7ff0ec02a410 comp rx=0 tx=0).stop 2024-04-25T01:30:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.984+0000 7ff10886d640 1 -- 172.21.15.28:0/2745940318 shutdown_connections 2024-04-25T01:30:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.984+0000 7ff10886d640 1 --2- 172.21.15.28:0/2745940318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1041012b0 0x7ff104108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.984+0000 7ff10886d640 1 --2- 172.21.15.28:0/2745940318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1041008f0 0x7ff104100d70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.984+0000 7ff10886d640 1 --2- 172.21.15.28:0/2745940318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff104101ca0 0x7ff1041020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.984+0000 7ff10886d640 1 -- 172.21.15.28:0/2745940318 >> 172.21.15.28:0/2745940318 conn(0x7ff1040fb380 msgr2=0x7ff1040fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.984+0000 7ff10886d640 1 -- 172.21.15.28:0/2745940318 shutdown_connections 2024-04-25T01:30:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.984+0000 7ff10886d640 1 -- 172.21.15.28:0/2745940318 wait complete. 2024-04-25T01:30:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.985+0000 7ff10886d640 1 Processor -- start 2024-04-25T01:30:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.985+0000 7ff10886d640 1 -- start start 2024-04-25T01:30:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.985+0000 7ff10886d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1041008f0 0x7ff104197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.985+0000 7ff10886d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1041012b0 0x7ff104197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.985+0000 7ff10886d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff104101ca0 0x7ff10419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.985+0000 7ff10886d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff10410c320 con 0x7ff104101ca0 2024-04-25T01:30:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.985+0000 7ff10886d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff10410c1a0 con 0x7ff1041008f0 2024-04-25T01:30:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.985+0000 7ff10886d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff10410c4a0 con 0x7ff1041012b0 2024-04-25T01:30:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.985+0000 7ff102d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff104101ca0 0x7ff10419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.985+0000 7ff102d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff104101ca0 0x7ff10419f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55600/0 (socket says 172.21.15.28:55600) 2024-04-25T01:30:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.985+0000 7ff102d9d640 1 -- 172.21.15.28:0/674534838 learned_addr learned my addr 172.21.15.28:0/674534838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.985+0000 7ff101d9b640 1 --2- 172.21.15.28:0/674534838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1041012b0 0x7ff104197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.985+0000 7ff10259c640 1 --2- 172.21.15.28:0/674534838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1041008f0 0x7ff104197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.985+0000 7ff102d9d640 1 -- 172.21.15.28:0/674534838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1041012b0 msgr2=0x7ff104197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.985+0000 7ff102d9d640 1 --2- 172.21.15.28:0/674534838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1041012b0 0x7ff104197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.985+0000 7ff102d9d640 1 -- 172.21.15.28:0/674534838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1041008f0 msgr2=0x7ff104197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.985+0000 7ff102d9d640 1 --2- 172.21.15.28:0/674534838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1041008f0 0x7ff104197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.985+0000 7ff102d9d640 1 -- 172.21.15.28:0/674534838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10419f6d0 con 0x7ff104101ca0 2024-04-25T01:30:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.986+0000 7ff10259c640 1 --2- 172.21.15.28:0/674534838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1041008f0 0x7ff104197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.986+0000 7ff102d9d640 1 --2- 172.21.15.28:0/674534838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff104101ca0 0x7ff10419f060 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc009fd0 tx=0x7ff0fc00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.986+0000 7ff0fb7fe640 1 -- 172.21.15.28:0/674534838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc017070 con 0x7ff104101ca0 2024-04-25T01:30:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.986+0000 7ff10886d640 1 -- 172.21.15.28:0/674534838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff10419f9c0 con 0x7ff104101ca0 2024-04-25T01:30:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.986+0000 7ff0fb7fe640 1 -- 172.21.15.28:0/674534838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0fc003e50 con 0x7ff104101ca0 2024-04-25T01:30:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.986+0000 7ff0fb7fe640 1 -- 172.21.15.28:0/674534838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc0106c0 con 0x7ff104101ca0 2024-04-25T01:30:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.986+0000 7ff10886d640 1 -- 172.21.15.28:0/674534838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff10419fee0 con 0x7ff104101ca0 2024-04-25T01:30:30.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.987+0000 7ff10886d640 1 -- 172.21.15.28:0/674534838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1040689b0 con 0x7ff104101ca0 2024-04-25T01:30:30.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.988+0000 7ff0fb7fe640 1 -- 172.21.15.28:0/674534838 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff0fc003970 con 0x7ff104101ca0 2024-04-25T01:30:30.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.989+0000 7ff0fb7fe640 1 --2- 172.21.15.28:0/674534838 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff0f00a2b90 0x7ff0f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:30.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.989+0000 7ff10259c640 1 --2- 172.21.15.28:0/674534838 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff0f00a2b90 0x7ff0f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:30.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.989+0000 7ff0fb7fe640 1 -- 172.21.15.28:0/674534838 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7ff0fc0ce8f0 con 0x7ff104101ca0 2024-04-25T01:30:30.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.989+0000 7ff10259c640 1 --2- 172.21.15.28:0/674534838 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff0f00a2b90 0x7ff0f00a5050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff104069ba0 tx=0x7ff0f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:30.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:30.994+0000 7ff0fb7fe640 1 -- 172.21.15.28:0/674534838 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff0fc09b6a0 con 0x7ff104101ca0 2024-04-25T01:30:31.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.181+0000 7ff10886d640 1 -- 172.21.15.28:0/674534838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff1041a02e0 con 0x7ff104101ca0 2024-04-25T01:30:31.160 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.160 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.160 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.160 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.160 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.160 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.161 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:31 smithi130 ceph-mon[32702]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:31.161 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.161 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:31 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1420382526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:31.161 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:31.161 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:30:31.161 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.161 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:31 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2257165174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:31.161 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:31 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:30:31.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.182+0000 7ff0fb7fe640 1 -- 172.21.15.28:0/674534838 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7ff0fc09f9a0 con 0x7ff104101ca0 2024-04-25T01:30:31.161 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T01:30:31.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.184+0000 7ff10886d640 1 -- 172.21.15.28:0/674534838 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff0f00a2b90 msgr2=0x7ff0f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:31.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.184+0000 7ff10886d640 1 --2- 172.21.15.28:0/674534838 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff0f00a2b90 0x7ff0f00a5050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff104069ba0 tx=0x7ff0f4005f90 comp rx=0 tx=0).stop 2024-04-25T01:30:31.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.184+0000 7ff10886d640 1 -- 172.21.15.28:0/674534838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff104101ca0 msgr2=0x7ff10419f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.184+0000 7ff10886d640 1 --2- 172.21.15.28:0/674534838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff104101ca0 0x7ff10419f060 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc009fd0 tx=0x7ff0fc00b260 comp rx=0 tx=0).stop 2024-04-25T01:30:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.184+0000 7ff10886d640 1 -- 172.21.15.28:0/674534838 shutdown_connections 2024-04-25T01:30:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.184+0000 7ff10886d640 1 --2- 172.21.15.28:0/674534838 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff0f00a2b90 0x7ff0f00a5050 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.184+0000 7ff10886d640 1 --2- 172.21.15.28:0/674534838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff104101ca0 0x7ff10419f060 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.184+0000 7ff10886d640 1 --2- 172.21.15.28:0/674534838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1041012b0 0x7ff104197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.184+0000 7ff10886d640 1 --2- 172.21.15.28:0/674534838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1041008f0 0x7ff104197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.184+0000 7ff10886d640 1 -- 172.21.15.28:0/674534838 >> 172.21.15.28:0/674534838 conn(0x7ff1040fb380 msgr2=0x7ff1040ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.184+0000 7ff10886d640 1 -- 172.21.15.28:0/674534838 shutdown_connections 2024-04-25T01:30:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.184+0000 7ff10886d640 1 -- 172.21.15.28:0/674534838 wait complete. 2024-04-25T01:30:31.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T01:30:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.298+0000 7f7b1d75a640 1 -- 172.21.15.28:0/670971273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b18101890 msgr2=0x7f7b18101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.298+0000 7f7b1d75a640 1 --2- 172.21.15.28:0/670971273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b18101890 0x7f7b18101d10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c009900 tx=0x7f7b0c02a410 comp rx=0 tx=0).stop 2024-04-25T01:30:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.298+0000 7f7b1d75a640 1 -- 172.21.15.28:0/670971273 shutdown_connections 2024-04-25T01:30:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.298+0000 7f7b1d75a640 1 --2- 172.21.15.28:0/670971273 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b18102250 0x7f7b18108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.298+0000 7f7b1d75a640 1 --2- 172.21.15.28:0/670971273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b18101890 0x7f7b18101d10 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.298+0000 7f7b1d75a640 1 --2- 172.21.15.28:0/670971273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b181006a0 0x7f7b18100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.298+0000 7f7b1d75a640 1 -- 172.21.15.28:0/670971273 >> 172.21.15.28:0/670971273 conn(0x7f7b180fbc30 msgr2=0x7f7b180fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.298+0000 7f7b1d75a640 1 -- 172.21.15.28:0/670971273 shutdown_connections 2024-04-25T01:30:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.298+0000 7f7b1d75a640 1 -- 172.21.15.28:0/670971273 wait complete. 2024-04-25T01:30:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.298+0000 7f7b1d75a640 1 Processor -- start 2024-04-25T01:30:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.299+0000 7f7b1d75a640 1 -- start start 2024-04-25T01:30:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.299+0000 7f7b1d75a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b181006a0 0x7f7b1806a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.299+0000 7f7b1d75a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b18101890 0x7f7b1819c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.299+0000 7f7b1d75a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b18102250 0x7f7b1819ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.299+0000 7f7b1d75a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1810c320 con 0x7f7b18102250 2024-04-25T01:30:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.299+0000 7f7b1d75a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1810c1a0 con 0x7f7b181006a0 2024-04-25T01:30:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.299+0000 7f7b1d75a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1810c4a0 con 0x7f7b18101890 2024-04-25T01:30:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.299+0000 7f7b177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b18102250 0x7f7b1819ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.299+0000 7f7b177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b18102250 0x7f7b1819ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55614/0 (socket says 172.21.15.28:55614) 2024-04-25T01:30:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.299+0000 7f7b177fe640 1 -- 172.21.15.28:0/3403110406 learned_addr learned my addr 172.21.15.28:0/3403110406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.299+0000 7f7b167fc640 1 --2- 172.21.15.28:0/3403110406 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b18101890 0x7f7b1819c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.299+0000 7f7b16ffd640 1 --2- 172.21.15.28:0/3403110406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b181006a0 0x7f7b1806a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.299+0000 7f7b16ffd640 1 -- 172.21.15.28:0/3403110406 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b18101890 msgr2=0x7f7b1819c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.299+0000 7f7b16ffd640 1 --2- 172.21.15.28:0/3403110406 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b18101890 0x7f7b1819c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.299+0000 7f7b16ffd640 1 -- 172.21.15.28:0/3403110406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b18102250 msgr2=0x7f7b1819ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.299+0000 7f7b16ffd640 1 --2- 172.21.15.28:0/3403110406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b18102250 0x7f7b1819ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.299+0000 7f7b16ffd640 1 -- 172.21.15.28:0/3403110406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b1819f710 con 0x7f7b181006a0 2024-04-25T01:30:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.299+0000 7f7b177fe640 1 --2- 172.21.15.28:0/3403110406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b18102250 0x7f7b1819ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.300+0000 7f7b16ffd640 1 --2- 172.21.15.28:0/3403110406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b181006a0 0x7f7b1806a910 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7b0800e9c0 tx=0x7f7b0800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.300+0000 7f7afffff640 1 -- 172.21.15.28:0/3403110406 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0800ccc0 con 0x7f7b181006a0 2024-04-25T01:30:31.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.300+0000 7f7afffff640 1 -- 172.21.15.28:0/3403110406 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b080035f0 con 0x7f7b181006a0 2024-04-25T01:30:31.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.300+0000 7f7b1d75a640 1 -- 172.21.15.28:0/3403110406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b1819f9a0 con 0x7f7b181006a0 2024-04-25T01:30:31.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.300+0000 7f7afffff640 1 -- 172.21.15.28:0/3403110406 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0800ce60 con 0x7f7b181006a0 2024-04-25T01:30:31.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.300+0000 7f7b1d75a640 1 -- 172.21.15.28:0/3403110406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b1819fec0 con 0x7f7b181006a0 2024-04-25T01:30:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.301+0000 7f7b1d75a640 1 -- 172.21.15.28:0/3403110406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b18107a40 con 0x7f7b181006a0 2024-04-25T01:30:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.302+0000 7f7afffff640 1 -- 172.21.15.28:0/3403110406 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7b080037a0 con 0x7f7b181006a0 2024-04-25T01:30:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.303+0000 7f7afffff640 1 --2- 172.21.15.28:0/3403110406 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f7b000a2b90 0x7f7b000a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.303+0000 7f7b167fc640 1 --2- 172.21.15.28:0/3403110406 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f7b000a2b90 0x7f7b000a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.303+0000 7f7afffff640 1 -- 172.21.15.28:0/3403110406 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f7b080cef90 con 0x7f7b181006a0 2024-04-25T01:30:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.303+0000 7f7b167fc640 1 --2- 172.21.15.28:0/3403110406 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f7b000a2b90 0x7f7b000a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c0314e0 tx=0x7f7b0c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.306+0000 7f7afffff640 1 -- 172.21.15.28:0/3403110406 <== mon.2 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7b0809c2f0 con 0x7f7b181006a0 2024-04-25T01:30:31.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:31 smithi032 ceph-mon[32461]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:31.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:31 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1420382526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:31.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:31.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:30:31.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:31 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2257165174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:31.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:31 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:30:31.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.517+0000 7f7b1d75a640 1 -- 172.21.15.28:0/3403110406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f7b1802d330 con 0x7f7b181006a0 2024-04-25T01:30:31.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.518+0000 7f7afffff640 1 -- 172.21.15.28:0/3403110406 <== mon.2 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v83) ==== 84+0+65 (secure 0 0 0) 0x7f7b0809bbf0 con 0x7f7b181006a0 2024-04-25T01:30:31.496 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:31.496 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-04-25T01:30:31.496 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:30:31.496 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-04-25T01:30:31.496 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-04-25T01:30:31.496 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:30:31.496 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-04-25T01:30:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.520+0000 7f7b1d75a640 1 -- 172.21.15.28:0/3403110406 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f7b000a2b90 msgr2=0x7f7b000a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.520+0000 7f7b1d75a640 1 --2- 172.21.15.28:0/3403110406 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f7b000a2b90 0x7f7b000a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c0314e0 tx=0x7f7b0c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:30:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.520+0000 7f7b1d75a640 1 -- 172.21.15.28:0/3403110406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b181006a0 msgr2=0x7f7b1806a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.520+0000 7f7b1d75a640 1 --2- 172.21.15.28:0/3403110406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b181006a0 0x7f7b1806a910 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7b0800e9c0 tx=0x7f7b0800ee80 comp rx=0 tx=0).stop 2024-04-25T01:30:31.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.520+0000 7f7b1d75a640 1 -- 172.21.15.28:0/3403110406 shutdown_connections 2024-04-25T01:30:31.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.520+0000 7f7b1d75a640 1 --2- 172.21.15.28:0/3403110406 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f7b000a2b90 0x7f7b000a5050 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:31.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.520+0000 7f7b1d75a640 1 --2- 172.21.15.28:0/3403110406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b18102250 0x7f7b1819ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:31.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.520+0000 7f7b1d75a640 1 --2- 172.21.15.28:0/3403110406 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b18101890 0x7f7b1819c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:31.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.520+0000 7f7b1d75a640 1 --2- 172.21.15.28:0/3403110406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b181006a0 0x7f7b1806a910 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:31.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.520+0000 7f7b1d75a640 1 -- 172.21.15.28:0/3403110406 >> 172.21.15.28:0/3403110406 conn(0x7f7b180fbc30 msgr2=0x7f7b18104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:31.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.520+0000 7f7b1d75a640 1 -- 172.21.15.28:0/3403110406 shutdown_connections 2024-04-25T01:30:31.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.520+0000 7f7b1d75a640 1 -- 172.21.15.28:0/3403110406 wait complete. 2024-04-25T01:30:31.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:31 smithi028 ceph-mon[29390]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:31.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:31 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1420382526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:31.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:31.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:30:31.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:31.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:31 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2257165174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:31.511 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:31 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:30:31.512 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T01:30:31.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T01:30:31.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.649+0000 7f690be58640 1 -- 172.21.15.28:0/309003724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904101790 msgr2=0x7f6904101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:31.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.649+0000 7f690be58640 1 --2- 172.21.15.28:0/309003724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904101790 0x7f6904101bb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f68f4009900 tx=0x7f68f402a410 comp rx=0 tx=0).stop 2024-04-25T01:30:31.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.649+0000 7f690be58640 1 -- 172.21.15.28:0/309003724 shutdown_connections 2024-04-25T01:30:31.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.649+0000 7f690be58640 1 --2- 172.21.15.28:0/309003724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6904103340 0x7f6904109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:31.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.649+0000 7f690be58640 1 --2- 172.21.15.28:0/309003724 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6904102980 0x7f6904102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:31.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.649+0000 7f690be58640 1 --2- 172.21.15.28:0/309003724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904101790 0x7f6904101bb0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:31.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.649+0000 7f690be58640 1 -- 172.21.15.28:0/309003724 >> 172.21.15.28:0/309003724 conn(0x7f69040fcd20 msgr2=0x7f69040ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:31.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.649+0000 7f690be58640 1 -- 172.21.15.28:0/309003724 shutdown_connections 2024-04-25T01:30:31.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.649+0000 7f690be58640 1 -- 172.21.15.28:0/309003724 wait complete. 2024-04-25T01:30:31.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.650+0000 7f690be58640 1 Processor -- start 2024-04-25T01:30:31.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.650+0000 7f690be58640 1 -- start start 2024-04-25T01:30:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.650+0000 7f690be58640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904101790 0x7f69040754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.650+0000 7f690be58640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6904102980 0x7f6904073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.650+0000 7f690be58640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6904103340 0x7f6904074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.650+0000 7f690be58640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f690410d430 con 0x7f6904101790 2024-04-25T01:30:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.650+0000 7f690be58640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f690410d2b0 con 0x7f6904102980 2024-04-25T01:30:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.650+0000 7f690be58640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f690410d5b0 con 0x7f6904103340 2024-04-25T01:30:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.650+0000 7f6909bf4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904101790 0x7f69040754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.650+0000 7f6909bf4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904101790 0x7f69040754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55640/0 (socket says 172.21.15.28:55640) 2024-04-25T01:30:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.650+0000 7f6909bf4640 1 -- 172.21.15.28:0/4137795314 learned_addr learned my addr 172.21.15.28:0/4137795314 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.651+0000 7f690a3f5640 1 --2- 172.21.15.28:0/4137795314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6904103340 0x7f6904074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.651+0000 7f6909bf4640 1 -- 172.21.15.28:0/4137795314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6904103340 msgr2=0x7f6904074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.651+0000 7f69093f3640 1 --2- 172.21.15.28:0/4137795314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6904102980 0x7f6904073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.651+0000 7f6909bf4640 1 --2- 172.21.15.28:0/4137795314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6904103340 0x7f6904074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.651+0000 7f6909bf4640 1 -- 172.21.15.28:0/4137795314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6904102980 msgr2=0x7f6904073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.651+0000 7f6909bf4640 1 --2- 172.21.15.28:0/4137795314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6904102980 0x7f6904073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.651+0000 7f6909bf4640 1 -- 172.21.15.28:0/4137795314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6904074900 con 0x7f6904101790 2024-04-25T01:30:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.651+0000 7f690a3f5640 1 --2- 172.21.15.28:0/4137795314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6904103340 0x7f6904074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.651+0000 7f6909bf4640 1 --2- 172.21.15.28:0/4137795314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904101790 0x7f69040754f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f68f4031440 tx=0x7f68f4032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.651+0000 7f68faffd640 1 -- 172.21.15.28:0/4137795314 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f403f030 con 0x7f6904101790 2024-04-25T01:30:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.651+0000 7f690be58640 1 -- 172.21.15.28:0/4137795314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69041a1b90 con 0x7f6904101790 2024-04-25T01:30:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.651+0000 7f68faffd640 1 -- 172.21.15.28:0/4137795314 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68f4003ce0 con 0x7f6904101790 2024-04-25T01:30:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.651+0000 7f68faffd640 1 -- 172.21.15.28:0/4137795314 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f4035070 con 0x7f6904101790 2024-04-25T01:30:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.651+0000 7f690be58640 1 -- 172.21.15.28:0/4137795314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69041a2130 con 0x7f6904101790 2024-04-25T01:30:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.652+0000 7f690be58640 1 -- 172.21.15.28:0/4137795314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6904108ab0 con 0x7f6904101790 2024-04-25T01:30:31.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.653+0000 7f68faffd640 1 -- 172.21.15.28:0/4137795314 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f68f4042030 con 0x7f6904101790 2024-04-25T01:30:31.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.654+0000 7f68faffd640 1 --2- 172.21.15.28:0/4137795314 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f68f00a2b90 0x7f68f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:31.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.654+0000 7f69093f3640 1 --2- 172.21.15.28:0/4137795314 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f68f00a2b90 0x7f68f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:31.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.654+0000 7f68faffd640 1 -- 172.21.15.28:0/4137795314 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f68f40ef180 con 0x7f6904101790 2024-04-25T01:30:31.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.654+0000 7f69093f3640 1 --2- 172.21.15.28:0/4137795314 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f68f00a2b90 0x7f68f00a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6904075260 tx=0x7f6900005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.658+0000 7f68faffd640 1 -- 172.21.15.28:0/4137795314 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f68f40bc560 con 0x7f6904101790 2024-04-25T01:30:31.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:31.843+0000 7f690be58640 1 -- 172.21.15.28:0/4137795314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f690402d170 con 0x7f6904101790 2024-04-25T01:30:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.326+0000 7f68faffd640 1 -- 172.21.15.28:0/4137795314 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v84) ==== 138+0+0 (secure 0 0 0) 0x7f68f40bbe60 con 0x7f6904101790 2024-04-25T01:30:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-04-25T01:30:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.329+0000 7f690be58640 1 -- 172.21.15.28:0/4137795314 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f68f00a2b90 msgr2=0x7f68f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.329+0000 7f690be58640 1 --2- 172.21.15.28:0/4137795314 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f68f00a2b90 0x7f68f00a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6904075260 tx=0x7f6900005f90 comp rx=0 tx=0).stop 2024-04-25T01:30:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.329+0000 7f690be58640 1 -- 172.21.15.28:0/4137795314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904101790 msgr2=0x7f69040754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.329+0000 7f690be58640 1 --2- 172.21.15.28:0/4137795314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904101790 0x7f69040754f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f68f4031440 tx=0x7f68f4032ce0 comp rx=0 tx=0).stop 2024-04-25T01:30:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.329+0000 7f690be58640 1 -- 172.21.15.28:0/4137795314 shutdown_connections 2024-04-25T01:30:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.329+0000 7f690be58640 1 --2- 172.21.15.28:0/4137795314 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f68f00a2b90 0x7f68f00a5050 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.329+0000 7f690be58640 1 --2- 172.21.15.28:0/4137795314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6904103340 0x7f6904074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:32.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.329+0000 7f690be58640 1 --2- 172.21.15.28:0/4137795314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6904102980 0x7f6904073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.329+0000 7f690be58640 1 --2- 172.21.15.28:0/4137795314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6904101790 0x7f69040754f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.329+0000 7f690be58640 1 -- 172.21.15.28:0/4137795314 >> 172.21.15.28:0/4137795314 conn(0x7f69040fcd20 msgr2=0x7f6904105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.329+0000 7f690be58640 1 -- 172.21.15.28:0/4137795314 shutdown_connections 2024-04-25T01:30:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.329+0000 7f690be58640 1 -- 172.21.15.28:0/4137795314 wait complete. 2024-04-25T01:30:32.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-25T01:30:32.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.466+0000 7fa4566f5640 1 -- 172.21.15.28:0/3700439154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450101790 msgr2=0x7fa450101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:32.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.466+0000 7fa4566f5640 1 --2- 172.21.15.28:0/3700439154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450101790 0x7fa450101bb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa444009900 tx=0x7fa44402a410 comp rx=0 tx=0).stop 2024-04-25T01:30:32.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.467+0000 7fa4566f5640 1 -- 172.21.15.28:0/3700439154 shutdown_connections 2024-04-25T01:30:32.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.467+0000 7fa4566f5640 1 --2- 172.21.15.28:0/3700439154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa450103340 0x7fa450109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:32.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.467+0000 7fa4566f5640 1 --2- 172.21.15.28:0/3700439154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa450102980 0x7fa450102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:32.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.467+0000 7fa4566f5640 1 --2- 172.21.15.28:0/3700439154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450101790 0x7fa450101bb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:32.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.467+0000 7fa4566f5640 1 -- 172.21.15.28:0/3700439154 >> 172.21.15.28:0/3700439154 conn(0x7fa4500fcd20 msgr2=0x7fa4500ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:32.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.467+0000 7fa4566f5640 1 -- 172.21.15.28:0/3700439154 shutdown_connections 2024-04-25T01:30:32.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.467+0000 7fa4566f5640 1 -- 172.21.15.28:0/3700439154 wait complete. 2024-04-25T01:30:32.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.467+0000 7fa4566f5640 1 Processor -- start 2024-04-25T01:30:32.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.467+0000 7fa4566f5640 1 -- start start 2024-04-25T01:30:32.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.468+0000 7fa4566f5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa450101790 0x7fa450199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:32.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.468+0000 7fa4566f5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa450102980 0x7fa45019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:32.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.468+0000 7fa4566f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450103340 0x7fa4501a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:32.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.468+0000 7fa4566f5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa45010d3e0 con 0x7fa450103340 2024-04-25T01:30:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.468+0000 7fa4566f5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa45010d260 con 0x7fa450102980 2024-04-25T01:30:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.468+0000 7fa4566f5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa45010d560 con 0x7fa450101790 2024-04-25T01:30:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.468+0000 7fa454c92640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450103340 0x7fa4501a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.468+0000 7fa454c92640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450103340 0x7fa4501a12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55652/0 (socket says 172.21.15.28:55652) 2024-04-25T01:30:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.468+0000 7fa454c92640 1 -- 172.21.15.28:0/378909753 learned_addr learned my addr 172.21.15.28:0/378909753 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.468+0000 7fa44f7fe640 1 --2- 172.21.15.28:0/378909753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa450102980 0x7fa45019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.468+0000 7fa44ffff640 1 --2- 172.21.15.28:0/378909753 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa450101790 0x7fa450199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.468+0000 7fa44f7fe640 1 -- 172.21.15.28:0/378909753 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa450101790 msgr2=0x7fa450199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.468+0000 7fa44f7fe640 1 --2- 172.21.15.28:0/378909753 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa450101790 0x7fa450199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.468+0000 7fa44f7fe640 1 -- 172.21.15.28:0/378909753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450103340 msgr2=0x7fa4501a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.468+0000 7fa44f7fe640 1 --2- 172.21.15.28:0/378909753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450103340 0x7fa4501a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.468+0000 7fa44f7fe640 1 -- 172.21.15.28:0/378909753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4501a1940 con 0x7fa450102980 2024-04-25T01:30:32.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.468+0000 7fa454c92640 1 --2- 172.21.15.28:0/378909753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450103340 0x7fa4501a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:32.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.468+0000 7fa44ffff640 1 --2- 172.21.15.28:0/378909753 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa450101790 0x7fa450199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:32.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.468+0000 7fa44f7fe640 1 --2- 172.21.15.28:0/378909753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa450102980 0x7fa45019a270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa44000e9f0 tx=0x7fa44000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:32.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.469+0000 7fa44d7fa640 1 -- 172.21.15.28:0/378909753 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44000ccc0 con 0x7fa450102980 2024-04-25T01:30:32.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.469+0000 7fa4566f5640 1 -- 172.21.15.28:0/378909753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4501a1c30 con 0x7fa450102980 2024-04-25T01:30:32.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.469+0000 7fa44d7fa640 1 -- 172.21.15.28:0/378909753 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa440010910 con 0x7fa450102980 2024-04-25T01:30:32.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.469+0000 7fa44d7fa640 1 -- 172.21.15.28:0/378909753 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa440018960 con 0x7fa450102980 2024-04-25T01:30:32.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.469+0000 7fa4566f5640 1 -- 172.21.15.28:0/378909753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4501a2150 con 0x7fa450102980 2024-04-25T01:30:32.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.470+0000 7fa4566f5640 1 -- 172.21.15.28:0/378909753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4501a2320 con 0x7fa450102980 2024-04-25T01:30:32.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.471+0000 7fa44d7fa640 1 -- 172.21.15.28:0/378909753 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa440010430 con 0x7fa450102980 2024-04-25T01:30:32.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.472+0000 7fa44d7fa640 1 --2- 172.21.15.28:0/378909753 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa4380a2b90 0x7fa4380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:32.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.472+0000 7fa44d7fa640 1 -- 172.21.15.28:0/378909753 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7497+0+0 (secure 0 0 0) 0x7fa4400cf1a0 con 0x7fa450102980 2024-04-25T01:30:32.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.472+0000 7fa44ffff640 1 --2- 172.21.15.28:0/378909753 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa4380a2b90 0x7fa4380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:32.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.473+0000 7fa44ffff640 1 --2- 172.21.15.28:0/378909753 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa4380a2b90 0x7fa4380a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa444005ab0 tx=0x7fa444000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:32.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.477+0000 7fa44d7fa640 1 -- 172.21.15.28:0/378909753 <== mon.2 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa44009bde0 con 0x7fa450102980 2024-04-25T01:30:32.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:32 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/674534838' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:30:32.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:32 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3403110406' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T01:30:32.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:32 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/674534838' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:30:32.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:32 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3403110406' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T01:30:32.645 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:32 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/674534838' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:30:32.646 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:32 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3403110406' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T01:30:32.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:32.686+0000 7fa4566f5640 1 -- 172.21.15.28:0/378909753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fa4501a2590 con 0x7fa450102980 2024-04-25T01:30:33.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.332+0000 7fa44d7fa640 1 -- 172.21.15.28:0/378909753 <== mon.2 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v85) ==== 171+0+0 (secure 0 0 0) 0x7fa4400a00e0 con 0x7fa450102980 2024-04-25T01:30:33.311 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-04-25T01:30:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.336+0000 7fa4566f5640 1 -- 172.21.15.28:0/378909753 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa4380a2b90 msgr2=0x7fa4380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.336+0000 7fa4566f5640 1 --2- 172.21.15.28:0/378909753 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa4380a2b90 0x7fa4380a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa444005ab0 tx=0x7fa444000bc0 comp rx=0 tx=0).stop 2024-04-25T01:30:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.336+0000 7fa4566f5640 1 -- 172.21.15.28:0/378909753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa450102980 msgr2=0x7fa45019a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.336+0000 7fa4566f5640 1 --2- 172.21.15.28:0/378909753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa450102980 0x7fa45019a270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa44000e9f0 tx=0x7fa44000eeb0 comp rx=0 tx=0).stop 2024-04-25T01:30:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.336+0000 7fa4566f5640 1 -- 172.21.15.28:0/378909753 shutdown_connections 2024-04-25T01:30:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.336+0000 7fa4566f5640 1 --2- 172.21.15.28:0/378909753 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa4380a2b90 0x7fa4380a5050 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.336+0000 7fa4566f5640 1 --2- 172.21.15.28:0/378909753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450103340 0x7fa4501a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.336+0000 7fa4566f5640 1 --2- 172.21.15.28:0/378909753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa450102980 0x7fa45019a270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.336+0000 7fa4566f5640 1 --2- 172.21.15.28:0/378909753 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa450101790 0x7fa450199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.336+0000 7fa4566f5640 1 -- 172.21.15.28:0/378909753 >> 172.21.15.28:0/378909753 conn(0x7fa4500fcd20 msgr2=0x7fa4501081a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.337+0000 7fa4566f5640 1 -- 172.21.15.28:0/378909753 shutdown_connections 2024-04-25T01:30:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.337+0000 7fa4566f5640 1 -- 172.21.15.28:0/378909753 wait complete. 2024-04-25T01:30:33.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:33 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/4137795314' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T01:30:33.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:33 smithi028 ceph-mon[29390]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:33.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:33 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/4137795314' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T01:30:33.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:33 smithi028 ceph-mon[29390]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T01:30:33.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:33 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/378909753' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-25T01:30:33.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:33 smithi028 ceph-mon[29390]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-25T01:30:33.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:33.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:30:33.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T01:30:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.461+0000 7fb2ca363640 1 -- 172.21.15.28:0/1675656780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c4102250 msgr2=0x7fb2c4108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.461+0000 7fb2ca363640 1 --2- 172.21.15.28:0/1675656780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c4102250 0x7fb2c4108ae0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc009ca0 tx=0x7fb2bc02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:30:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.461+0000 7fb2ca363640 1 -- 172.21.15.28:0/1675656780 shutdown_connections 2024-04-25T01:30:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.461+0000 7fb2ca363640 1 --2- 172.21.15.28:0/1675656780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c4102250 0x7fb2c4108ae0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.461+0000 7fb2ca363640 1 --2- 172.21.15.28:0/1675656780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb2c4101890 0x7fb2c4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.461+0000 7fb2ca363640 1 --2- 172.21.15.28:0/1675656780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2c41006a0 0x7fb2c4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.461+0000 7fb2ca363640 1 -- 172.21.15.28:0/1675656780 >> 172.21.15.28:0/1675656780 conn(0x7fb2c40fbc30 msgr2=0x7fb2c40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.461+0000 7fb2ca363640 1 -- 172.21.15.28:0/1675656780 shutdown_connections 2024-04-25T01:30:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.462+0000 7fb2ca363640 1 -- 172.21.15.28:0/1675656780 wait complete. 2024-04-25T01:30:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.462+0000 7fb2ca363640 1 Processor -- start 2024-04-25T01:30:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.462+0000 7fb2ca363640 1 -- start start 2024-04-25T01:30:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.462+0000 7fb2ca363640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2c41006a0 0x7fb2c4197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.462+0000 7fb2ca363640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c4101890 0x7fb2c4197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.462+0000 7fb2ca363640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb2c4102250 0x7fb2c419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.462+0000 7fb2ca363640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c410c300 con 0x7fb2c4101890 2024-04-25T01:30:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.462+0000 7fb2ca363640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c410c180 con 0x7fb2c41006a0 2024-04-25T01:30:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.462+0000 7fb2ca363640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c410c480 con 0x7fb2c4102250 2024-04-25T01:30:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.463+0000 7fb2c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c4101890 0x7fb2c4197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.463+0000 7fb2c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c4101890 0x7fb2c4197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55666/0 (socket says 172.21.15.28:55666) 2024-04-25T01:30:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.463+0000 7fb2c37fe640 1 -- 172.21.15.28:0/3450061279 learned_addr learned my addr 172.21.15.28:0/3450061279 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.463+0000 7fb2c3fff640 1 --2- 172.21.15.28:0/3450061279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2c41006a0 0x7fb2c4197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.463+0000 7fb2c8900640 1 --2- 172.21.15.28:0/3450061279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb2c4102250 0x7fb2c419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.463+0000 7fb2c37fe640 1 -- 172.21.15.28:0/3450061279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb2c4102250 msgr2=0x7fb2c419efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.463+0000 7fb2c37fe640 1 --2- 172.21.15.28:0/3450061279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb2c4102250 0x7fb2c419efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.463+0000 7fb2c37fe640 1 -- 172.21.15.28:0/3450061279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2c41006a0 msgr2=0x7fb2c4197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.463+0000 7fb2c37fe640 1 --2- 172.21.15.28:0/3450061279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2c41006a0 0x7fb2c4197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.463+0000 7fb2c37fe640 1 -- 172.21.15.28:0/3450061279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c419f650 con 0x7fb2c4101890 2024-04-25T01:30:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.463+0000 7fb2c8900640 1 --2- 172.21.15.28:0/3450061279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb2c4102250 0x7fb2c419efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.463+0000 7fb2c37fe640 1 --2- 172.21.15.28:0/3450061279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c4101890 0x7fb2c4197f70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb2b400e9f0 tx=0x7fb2b400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.463+0000 7fb2c17fa640 1 -- 172.21.15.28:0/3450061279 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b400cd80 con 0x7fb2c4101890 2024-04-25T01:30:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.463+0000 7fb2ca363640 1 -- 172.21.15.28:0/3450061279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c419f940 con 0x7fb2c4101890 2024-04-25T01:30:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.463+0000 7fb2c17fa640 1 -- 172.21.15.28:0/3450061279 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2b4010910 con 0x7fb2c4101890 2024-04-25T01:30:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.463+0000 7fb2c17fa640 1 -- 172.21.15.28:0/3450061279 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b40188f0 con 0x7fb2c4101890 2024-04-25T01:30:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.463+0000 7fb2ca363640 1 -- 172.21.15.28:0/3450061279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c419fe90 con 0x7fb2c4101890 2024-04-25T01:30:33.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.464+0000 7fb2ca363640 1 -- 172.21.15.28:0/3450061279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2c41a0120 con 0x7fb2c4101890 2024-04-25T01:30:33.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.465+0000 7fb2c17fa640 1 -- 172.21.15.28:0/3450061279 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb2b4010ac0 con 0x7fb2c4101890 2024-04-25T01:30:33.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.466+0000 7fb2c17fa640 1 --2- 172.21.15.28:0/3450061279 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb2b00a2b90 0x7fb2b00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:33.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.466+0000 7fb2c3fff640 1 --2- 172.21.15.28:0/3450061279 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb2b00a2b90 0x7fb2b00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:33.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.466+0000 7fb2c17fa640 1 -- 172.21.15.28:0/3450061279 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7879+0+0 (secure 0 0 0) 0x7fb2b40cf370 con 0x7fb2c4101890 2024-04-25T01:30:33.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.466+0000 7fb2c3fff640 1 --2- 172.21.15.28:0/3450061279 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb2b00a2b90 0x7fb2b00a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb2c41016f0 tx=0x7fb2ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:33.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.471+0000 7fb2c17fa640 1 -- 172.21.15.28:0/3450061279 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb2b409c3e0 con 0x7fb2c4101890 2024-04-25T01:30:33.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:33.694+0000 7fb2ca363640 1 -- 172.21.15.28:0/3450061279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb2c41a0520 con 0x7fb2c4101890 2024-04-25T01:30:33.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:33 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/4137795314' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T01:30:33.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:33 smithi130 ceph-mon[32702]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:33.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:33 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/4137795314' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T01:30:33.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:33 smithi130 ceph-mon[32702]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T01:30:33.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:33 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/378909753' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-25T01:30:33.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:33 smithi130 ceph-mon[32702]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-25T01:30:33.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:33.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:30:33.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:33 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/4137795314' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T01:30:33.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:33 smithi032 ceph-mon[32461]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:33.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:33 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/4137795314' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T01:30:33.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:33 smithi032 ceph-mon[32461]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T01:30:33.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:33 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/378909753' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-25T01:30:33.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:33 smithi032 ceph-mon[32461]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-25T01:30:33.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:33.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:30:33.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:30:33.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:30:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.330+0000 7fb2c17fa640 1 -- 172.21.15.28:0/3450061279 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fb2b409bce0 con 0x7fb2c4101890 2024-04-25T01:30:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T01:30:34.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.335+0000 7fb2ca363640 1 -- 172.21.15.28:0/3450061279 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb2b00a2b90 msgr2=0x7fb2b00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:34.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.335+0000 7fb2ca363640 1 --2- 172.21.15.28:0/3450061279 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb2b00a2b90 0x7fb2b00a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb2c41016f0 tx=0x7fb2ac005f90 comp rx=0 tx=0).stop 2024-04-25T01:30:34.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.335+0000 7fb2ca363640 1 -- 172.21.15.28:0/3450061279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c4101890 msgr2=0x7fb2c4197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:34.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.335+0000 7fb2ca363640 1 --2- 172.21.15.28:0/3450061279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c4101890 0x7fb2c4197f70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb2b400e9f0 tx=0x7fb2b400eeb0 comp rx=0 tx=0).stop 2024-04-25T01:30:34.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.335+0000 7fb2ca363640 1 -- 172.21.15.28:0/3450061279 shutdown_connections 2024-04-25T01:30:34.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.335+0000 7fb2ca363640 1 --2- 172.21.15.28:0/3450061279 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb2b00a2b90 0x7fb2b00a5050 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:34.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.335+0000 7fb2ca363640 1 --2- 172.21.15.28:0/3450061279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb2c4102250 0x7fb2c419efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:34.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.335+0000 7fb2ca363640 1 --2- 172.21.15.28:0/3450061279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c4101890 0x7fb2c4197f70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:34.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.335+0000 7fb2ca363640 1 --2- 172.21.15.28:0/3450061279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb2c41006a0 0x7fb2c4197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.335+0000 7fb2ca363640 1 -- 172.21.15.28:0/3450061279 >> 172.21.15.28:0/3450061279 conn(0x7fb2c40fbc30 msgr2=0x7fb2c40fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.335+0000 7fb2ca363640 1 -- 172.21.15.28:0/3450061279 shutdown_connections 2024-04-25T01:30:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.335+0000 7fb2ca363640 1 -- 172.21.15.28:0/3450061279 wait complete. 2024-04-25T01:30:34.336 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-25T01:30:34.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-25T01:30:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.475+0000 7f635444d640 1 -- 172.21.15.28:0/228298208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c100680 msgr2=0x7f634c100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.475+0000 7f635444d640 1 --2- 172.21.15.28:0/228298208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c100680 0x7f634c100aa0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6348009900 tx=0x7f634802a410 comp rx=0 tx=0).stop 2024-04-25T01:30:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.475+0000 7f635444d640 1 -- 172.21.15.28:0/228298208 shutdown_connections 2024-04-25T01:30:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.475+0000 7f635444d640 1 --2- 172.21.15.28:0/228298208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f634c102230 0x7f634c108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.475+0000 7f635444d640 1 --2- 172.21.15.28:0/228298208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f634c101870 0x7f634c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.475+0000 7f635444d640 1 --2- 172.21.15.28:0/228298208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c100680 0x7f634c100aa0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.475+0000 7f635444d640 1 -- 172.21.15.28:0/228298208 >> 172.21.15.28:0/228298208 conn(0x7f634c0fbbf0 msgr2=0x7f634c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.475+0000 7f635444d640 1 -- 172.21.15.28:0/228298208 shutdown_connections 2024-04-25T01:30:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.475+0000 7f635444d640 1 -- 172.21.15.28:0/228298208 wait complete. 2024-04-25T01:30:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.476+0000 7f635444d640 1 Processor -- start 2024-04-25T01:30:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.476+0000 7f635444d640 1 -- start start 2024-04-25T01:30:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.476+0000 7f635444d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f634c100680 0x7f634c197cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.476+0000 7f635444d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c101870 0x7f634c198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.476+0000 7f635444d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f634c102230 0x7f634c19f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.476+0000 7f635444d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f634c10c3d0 con 0x7f634c101870 2024-04-25T01:30:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.476+0000 7f635444d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f634c10c250 con 0x7f634c100680 2024-04-25T01:30:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.476+0000 7f635444d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f634c10c550 con 0x7f634c102230 2024-04-25T01:30:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.476+0000 7f63519e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c101870 0x7f634c198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.476+0000 7f63519e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c101870 0x7f634c198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55686/0 (socket says 172.21.15.28:55686) 2024-04-25T01:30:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.476+0000 7f63519e8640 1 -- 172.21.15.28:0/3223755355 learned_addr learned my addr 172.21.15.28:0/3223755355 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.476+0000 7f63529ea640 1 --2- 172.21.15.28:0/3223755355 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f634c102230 0x7f634c19f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.476+0000 7f63521e9640 1 --2- 172.21.15.28:0/3223755355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f634c100680 0x7f634c197cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.476+0000 7f63519e8640 1 -- 172.21.15.28:0/3223755355 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f634c102230 msgr2=0x7f634c19f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.476+0000 7f63519e8640 1 --2- 172.21.15.28:0/3223755355 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f634c102230 0x7f634c19f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.476+0000 7f63519e8640 1 -- 172.21.15.28:0/3223755355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f634c100680 msgr2=0x7f634c197cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.476+0000 7f63519e8640 1 --2- 172.21.15.28:0/3223755355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f634c100680 0x7f634c197cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.476+0000 7f63519e8640 1 -- 172.21.15.28:0/3223755355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f634c19f8b0 con 0x7f634c101870 2024-04-25T01:30:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.476+0000 7f63519e8640 1 --2- 172.21.15.28:0/3223755355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c101870 0x7f634c198200 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f633c00b4a0 tx=0x7f633c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.477+0000 7f63437fe640 1 -- 172.21.15.28:0/3223755355 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c014070 con 0x7f634c101870 2024-04-25T01:30:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.477+0000 7f635444d640 1 -- 172.21.15.28:0/3223755355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f634c19fba0 con 0x7f634c101870 2024-04-25T01:30:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.477+0000 7f63437fe640 1 -- 172.21.15.28:0/3223755355 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f633c010af0 con 0x7f634c101870 2024-04-25T01:30:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.477+0000 7f63437fe640 1 -- 172.21.15.28:0/3223755355 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c017d10 con 0x7f634c101870 2024-04-25T01:30:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.477+0000 7f635444d640 1 -- 172.21.15.28:0/3223755355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f634c1a00f0 con 0x7f634c101870 2024-04-25T01:30:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.478+0000 7f635444d640 1 -- 172.21.15.28:0/3223755355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f634c1a0380 con 0x7f634c101870 2024-04-25T01:30:34.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.479+0000 7f63437fe640 1 -- 172.21.15.28:0/3223755355 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f633c017070 con 0x7f634c101870 2024-04-25T01:30:34.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.479+0000 7f63437fe640 1 --2- 172.21.15.28:0/3223755355 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f63380a2b90 0x7f63380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:34.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.479+0000 7f63521e9640 1 --2- 172.21.15.28:0/3223755355 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f63380a2b90 0x7f63380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:34.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.479+0000 7f63437fe640 1 -- 172.21.15.28:0/3223755355 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f633c0ce700 con 0x7f634c101870 2024-04-25T01:30:34.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.480+0000 7f63521e9640 1 --2- 172.21.15.28:0/3223755355 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f63380a2b90 0x7f63380a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f63480314e0 tx=0x7f6348000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.483+0000 7f63437fe640 1 -- 172.21.15.28:0/3223755355 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f633c09b7b0 con 0x7f634c101870 2024-04-25T01:30:34.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:34.677+0000 7f635444d640 1 -- 172.21.15.28:0/3223755355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f634c1a0550 con 0x7f634c101870 2024-04-25T01:30:34.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:34 smithi130 ceph-mon[32702]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-25T01:30:34.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:34 smithi130 ceph-mon[32702]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T01:30:34.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:34 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3450061279' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T01:30:34.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:34 smithi130 ceph-mon[32702]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T01:30:34.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:34 smithi032 ceph-mon[32461]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-25T01:30:34.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:34 smithi032 ceph-mon[32461]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T01:30:34.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:34 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3450061279' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T01:30:34.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:34 smithi032 ceph-mon[32461]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T01:30:34.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:34 smithi028 ceph-mon[29390]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-25T01:30:34.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:34 smithi028 ceph-mon[29390]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T01:30:34.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:34 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3450061279' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T01:30:34.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:34 smithi028 ceph-mon[29390]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T01:30:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.349+0000 7f63437fe640 1 -- 172.21.15.28:0/3223755355 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v87) ==== 157+0+0 (secure 0 0 0) 0x7f633c09b0b0 con 0x7f634c101870 2024-04-25T01:30:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.353+0000 7f635444d640 1 -- 172.21.15.28:0/3223755355 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f63380a2b90 msgr2=0x7f63380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.353+0000 7f635444d640 1 --2- 172.21.15.28:0/3223755355 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f63380a2b90 0x7f63380a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f63480314e0 tx=0x7f6348000bc0 comp rx=0 tx=0).stop 2024-04-25T01:30:35.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.353+0000 7f635444d640 1 -- 172.21.15.28:0/3223755355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c101870 msgr2=0x7f634c198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:35.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.353+0000 7f635444d640 1 --2- 172.21.15.28:0/3223755355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c101870 0x7f634c198200 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f633c00b4a0 tx=0x7f633c00b960 comp rx=0 tx=0).stop 2024-04-25T01:30:35.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.353+0000 7f635444d640 1 -- 172.21.15.28:0/3223755355 shutdown_connections 2024-04-25T01:30:35.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.353+0000 7f635444d640 1 --2- 172.21.15.28:0/3223755355 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f63380a2b90 0x7f63380a5050 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:35.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.353+0000 7f635444d640 1 --2- 172.21.15.28:0/3223755355 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f634c102230 0x7f634c19f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:35.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.353+0000 7f635444d640 1 --2- 172.21.15.28:0/3223755355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c101870 0x7f634c198200 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:35.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.353+0000 7f635444d640 1 --2- 172.21.15.28:0/3223755355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f634c100680 0x7f634c197cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:35.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.353+0000 7f635444d640 1 -- 172.21.15.28:0/3223755355 >> 172.21.15.28:0/3223755355 conn(0x7f634c0fbbf0 msgr2=0x7f634c0688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:35.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.353+0000 7f635444d640 1 -- 172.21.15.28:0/3223755355 shutdown_connections 2024-04-25T01:30:35.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.353+0000 7f635444d640 1 -- 172.21.15.28:0/3223755355 wait complete. 2024-04-25T01:30:35.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T01:30:35.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.482+0000 7f9520e7d640 1 -- 172.21.15.28:0/357329705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c104b90 msgr2=0x7f951c106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:35.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.482+0000 7f9520e7d640 1 --2- 172.21.15.28:0/357329705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c104b90 0x7f951c106f80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9504009900 tx=0x7f950402a410 comp rx=0 tx=0).stop 2024-04-25T01:30:35.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.483+0000 7f9520e7d640 1 -- 172.21.15.28:0/357329705 shutdown_connections 2024-04-25T01:30:35.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.483+0000 7f9520e7d640 1 --2- 172.21.15.28:0/357329705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f951c1074c0 0x7f951c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:35.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.483+0000 7f9520e7d640 1 --2- 172.21.15.28:0/357329705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c104b90 0x7f951c106f80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:35.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.483+0000 7f9520e7d640 1 --2- 172.21.15.28:0/357329705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c102260 0x7f951c104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:35.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.483+0000 7f9520e7d640 1 -- 172.21.15.28:0/357329705 >> 172.21.15.28:0/357329705 conn(0x7f951c0fbbf0 msgr2=0x7f951c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:35.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.483+0000 7f9520e7d640 1 -- 172.21.15.28:0/357329705 shutdown_connections 2024-04-25T01:30:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.483+0000 7f9520e7d640 1 -- 172.21.15.28:0/357329705 wait complete. 2024-04-25T01:30:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.483+0000 7f9520e7d640 1 Processor -- start 2024-04-25T01:30:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.483+0000 7f9520e7d640 1 -- start start 2024-04-25T01:30:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f9520e7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c102260 0x7f951c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f9520e7d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c104b90 0x7f951c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f9520e7d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f951c1074c0 0x7f951c19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f9520e7d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f951c10d450 con 0x7f951c102260 2024-04-25T01:30:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f9520e7d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f951c10d2d0 con 0x7f951c104b90 2024-04-25T01:30:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f9520e7d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f951c10d5d0 con 0x7f951c1074c0 2024-04-25T01:30:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f951a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c102260 0x7f951c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f951a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c102260 0x7f951c197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55696/0 (socket says 172.21.15.28:55696) 2024-04-25T01:30:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f951a59c640 1 -- 172.21.15.28:0/2835382696 learned_addr learned my addr 172.21.15.28:0/2835382696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f951ad9d640 1 --2- 172.21.15.28:0/2835382696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f951c1074c0 0x7f951c19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f9519d9b640 1 --2- 172.21.15.28:0/2835382696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c104b90 0x7f951c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:35.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f951a59c640 1 -- 172.21.15.28:0/2835382696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f951c1074c0 msgr2=0x7f951c19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:35.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f951a59c640 1 --2- 172.21.15.28:0/2835382696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f951c1074c0 0x7f951c19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:35.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f951a59c640 1 -- 172.21.15.28:0/2835382696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c104b90 msgr2=0x7f951c197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:35.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f951a59c640 1 --2- 172.21.15.28:0/2835382696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c104b90 0x7f951c197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:35.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f951a59c640 1 -- 172.21.15.28:0/2835382696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f951c19f6c0 con 0x7f951c102260 2024-04-25T01:30:35.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f951ad9d640 1 --2- 172.21.15.28:0/2835382696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f951c1074c0 0x7f951c19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:35.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f951a59c640 1 --2- 172.21.15.28:0/2835382696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c102260 0x7f951c197aa0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f951400c890 tx=0x7f951400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:35.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f95137fe640 1 -- 172.21.15.28:0/2835382696 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9514004220 con 0x7f951c102260 2024-04-25T01:30:35.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f9520e7d640 1 -- 172.21.15.28:0/2835382696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f951c19f9b0 con 0x7f951c102260 2024-04-25T01:30:35.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f95137fe640 1 -- 172.21.15.28:0/2835382696 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95140043c0 con 0x7f951c102260 2024-04-25T01:30:35.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f95137fe640 1 -- 172.21.15.28:0/2835382696 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9514004a00 con 0x7f951c102260 2024-04-25T01:30:35.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.484+0000 7f9520e7d640 1 -- 172.21.15.28:0/2835382696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f951c19fed0 con 0x7f951c102260 2024-04-25T01:30:35.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.485+0000 7f9520e7d640 1 -- 172.21.15.28:0/2835382696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f951c0fd6e0 con 0x7f951c102260 2024-04-25T01:30:35.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.487+0000 7f95137fe640 1 -- 172.21.15.28:0/2835382696 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f95140079b0 con 0x7f951c102260 2024-04-25T01:30:35.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.487+0000 7f95137fe640 1 --2- 172.21.15.28:0/2835382696 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f95080a2b90 0x7f95080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:35.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.487+0000 7f9519d9b640 1 --2- 172.21.15.28:0/2835382696 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f95080a2b90 0x7f95080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:35.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.487+0000 7f95137fe640 1 -- 172.21.15.28:0/2835382696 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8277+0+0 (secure 0 0 0) 0x7f95140cf1f0 con 0x7f951c102260 2024-04-25T01:30:35.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.488+0000 7f9519d9b640 1 --2- 172.21.15.28:0/2835382696 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f95080a2b90 0x7f95080a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f950402aa00 tx=0x7f9504031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:35.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.491+0000 7f95137fe640 1 -- 172.21.15.28:0/2835382696 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9514014030 con 0x7f951c102260 2024-04-25T01:30:35.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:35.676+0000 7f9520e7d640 1 -- 172.21.15.28:0/2835382696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f951c062340 con 0x7f951c102260 2024-04-25T01:30:35.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:35 smithi130 ceph-mon[32702]: pgmap v292: 97 pgs: 37 creating+peering, 12 active+clean, 48 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:35.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:35 smithi130 ceph-mon[32702]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:30:35.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:35 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3450061279' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T01:30:35.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:35 smithi130 ceph-mon[32702]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T01:30:35.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:35 smithi130 ceph-mon[32702]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T01:30:35.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:35 smithi130 ceph-mon[32702]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T01:30:35.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:35 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3223755355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-25T01:30:35.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:35 smithi032 ceph-mon[32461]: pgmap v292: 97 pgs: 37 creating+peering, 12 active+clean, 48 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:35.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:35 smithi032 ceph-mon[32461]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:30:35.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:35 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3450061279' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T01:30:35.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:35 smithi032 ceph-mon[32461]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T01:30:35.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:35 smithi032 ceph-mon[32461]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T01:30:35.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:35 smithi032 ceph-mon[32461]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T01:30:35.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:35 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3223755355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-25T01:30:35.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:35 smithi028 ceph-mon[29390]: pgmap v292: 97 pgs: 37 creating+peering, 12 active+clean, 48 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:30:35.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:35 smithi028 ceph-mon[29390]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:30:35.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:35 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3450061279' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T01:30:35.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:35 smithi028 ceph-mon[29390]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T01:30:35.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:35 smithi028 ceph-mon[29390]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T01:30:35.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:35 smithi028 ceph-mon[29390]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T01:30:35.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:35 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3223755355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-25T01:30:37.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:36 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] up:active 2024-04-25T01:30:37.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:36 smithi028 ceph-mon[29390]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:30:37.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:36 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3223755355' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-25T01:30:37.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:36 smithi028 ceph-mon[29390]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T01:30:37.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:36 smithi028 ceph-mon[29390]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:30:37.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:36 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2835382696' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T01:30:37.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:36 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] up:active 2024-04-25T01:30:37.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:36 smithi130 ceph-mon[32702]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:30:37.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:36 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3223755355' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-25T01:30:37.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:36 smithi130 ceph-mon[32702]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T01:30:37.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:36 smithi130 ceph-mon[32702]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:30:37.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:36 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2835382696' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T01:30:37.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:36 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] up:active 2024-04-25T01:30:37.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:36 smithi032 ceph-mon[32461]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:30:37.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:36 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3223755355' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-25T01:30:37.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:36 smithi032 ceph-mon[32461]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T01:30:37.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:36 smithi032 ceph-mon[32461]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:30:37.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:36 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2835382696' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T01:30:37.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.353+0000 7f95137fe640 1 -- 172.21.15.28:0/2835382696 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v89) ==== 228+0+0 (secure 0 0 0) 0x7f951409c120 con 0x7f951c102260 2024-04-25T01:30:37.331 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data_ec' created 2024-04-25T01:30:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.356+0000 7f9520e7d640 1 -- 172.21.15.28:0/2835382696 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f95080a2b90 msgr2=0x7f95080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.356+0000 7f9520e7d640 1 --2- 172.21.15.28:0/2835382696 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f95080a2b90 0x7f95080a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f950402aa00 tx=0x7f9504031400 comp rx=0 tx=0).stop 2024-04-25T01:30:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.356+0000 7f9520e7d640 1 -- 172.21.15.28:0/2835382696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c102260 msgr2=0x7f951c197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.356+0000 7f9520e7d640 1 --2- 172.21.15.28:0/2835382696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c102260 0x7f951c197aa0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f951400c890 tx=0x7f951400cd50 comp rx=0 tx=0).stop 2024-04-25T01:30:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.356+0000 7f9520e7d640 1 -- 172.21.15.28:0/2835382696 shutdown_connections 2024-04-25T01:30:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.356+0000 7f9520e7d640 1 --2- 172.21.15.28:0/2835382696 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f95080a2b90 0x7f95080a5050 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.356+0000 7f9520e7d640 1 --2- 172.21.15.28:0/2835382696 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f951c1074c0 0x7f951c19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.356+0000 7f9520e7d640 1 --2- 172.21.15.28:0/2835382696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f951c104b90 0x7f951c197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.356+0000 7f9520e7d640 1 --2- 172.21.15.28:0/2835382696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f951c102260 0x7f951c197aa0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.356+0000 7f9520e7d640 1 -- 172.21.15.28:0/2835382696 >> 172.21.15.28:0/2835382696 conn(0x7f951c0fbbf0 msgr2=0x7f951c102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:37.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.357+0000 7f9520e7d640 1 -- 172.21.15.28:0/2835382696 shutdown_connections 2024-04-25T01:30:37.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.357+0000 7f9520e7d640 1 -- 172.21.15.28:0/2835382696 wait complete. 2024-04-25T01:30:37.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-25T01:30:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.487+0000 7fafc04f5640 1 -- 172.21.15.28:0/1249187522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafb8102250 msgr2=0x7fafb8108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.487+0000 7fafc04f5640 1 --2- 172.21.15.28:0/1249187522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafb8102250 0x7fafb8108ae0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009ca0 tx=0x7fafb002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:30:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.487+0000 7fafc04f5640 1 -- 172.21.15.28:0/1249187522 shutdown_connections 2024-04-25T01:30:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.487+0000 7fafc04f5640 1 --2- 172.21.15.28:0/1249187522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafb8102250 0x7fafb8108ae0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.487+0000 7fafc04f5640 1 --2- 172.21.15.28:0/1249187522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb8101890 0x7fafb8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.487+0000 7fafc04f5640 1 --2- 172.21.15.28:0/1249187522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafb81006a0 0x7fafb8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.487+0000 7fafc04f5640 1 -- 172.21.15.28:0/1249187522 >> 172.21.15.28:0/1249187522 conn(0x7fafb80fbc30 msgr2=0x7fafb80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.487+0000 7fafc04f5640 1 -- 172.21.15.28:0/1249187522 shutdown_connections 2024-04-25T01:30:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.487+0000 7fafc04f5640 1 -- 172.21.15.28:0/1249187522 wait complete. 2024-04-25T01:30:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.488+0000 7fafc04f5640 1 Processor -- start 2024-04-25T01:30:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.488+0000 7fafc04f5640 1 -- start start 2024-04-25T01:30:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.488+0000 7fafc04f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafb81006a0 0x7fafb8197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.488+0000 7fafc04f5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb8101890 0x7fafb8198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.488+0000 7fafc04f5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafb8102250 0x7fafb819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.488+0000 7fafc04f5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fafb810d440 con 0x7fafb81006a0 2024-04-25T01:30:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.488+0000 7fafc04f5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fafb810d2c0 con 0x7fafb8101890 2024-04-25T01:30:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.488+0000 7fafc04f5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fafb810d5c0 con 0x7fafb8102250 2024-04-25T01:30:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.488+0000 7fafbe291640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafb81006a0 0x7fafb8197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.488+0000 7fafbe291640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafb81006a0 0x7fafb8197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37844/0 (socket says 172.21.15.28:37844) 2024-04-25T01:30:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.488+0000 7fafbe291640 1 -- 172.21.15.28:0/3377833132 learned_addr learned my addr 172.21.15.28:0/3377833132 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.488+0000 7fafbda90640 1 --2- 172.21.15.28:0/3377833132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb8101890 0x7fafb8198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.488+0000 7fafbea92640 1 --2- 172.21.15.28:0/3377833132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafb8102250 0x7fafb819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.488+0000 7fafbe291640 1 -- 172.21.15.28:0/3377833132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafb8102250 msgr2=0x7fafb819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.488+0000 7fafbe291640 1 --2- 172.21.15.28:0/3377833132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafb8102250 0x7fafb819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.488+0000 7fafbe291640 1 -- 172.21.15.28:0/3377833132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb8101890 msgr2=0x7fafb8198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.488+0000 7fafbe291640 1 --2- 172.21.15.28:0/3377833132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb8101890 0x7fafb8198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.488+0000 7fafbe291640 1 -- 172.21.15.28:0/3377833132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafb819f6e0 con 0x7fafb81006a0 2024-04-25T01:30:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.488+0000 7fafbea92640 1 --2- 172.21.15.28:0/3377833132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafb8102250 0x7fafb819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.489+0000 7fafbe291640 1 --2- 172.21.15.28:0/3377833132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafb81006a0 0x7fafb8197ac0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fafb400d950 tx=0x7fafb400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.489+0000 7fafaf7fe640 1 -- 172.21.15.28:0/3377833132 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb4017070 con 0x7fafb81006a0 2024-04-25T01:30:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.489+0000 7fafaf7fe640 1 -- 172.21.15.28:0/3377833132 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafb4009c30 con 0x7fafb81006a0 2024-04-25T01:30:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.489+0000 7fafc04f5640 1 -- 172.21.15.28:0/3377833132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafb819f9d0 con 0x7fafb81006a0 2024-04-25T01:30:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.489+0000 7fafaf7fe640 1 -- 172.21.15.28:0/3377833132 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb400f600 con 0x7fafb81006a0 2024-04-25T01:30:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.489+0000 7fafc04f5640 1 -- 172.21.15.28:0/3377833132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafb819fed0 con 0x7fafb81006a0 2024-04-25T01:30:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.490+0000 7fafc04f5640 1 -- 172.21.15.28:0/3377833132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafb8107960 con 0x7fafb81006a0 2024-04-25T01:30:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.491+0000 7fafaf7fe640 1 -- 172.21.15.28:0/3377833132 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fafb401a070 con 0x7fafb81006a0 2024-04-25T01:30:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.492+0000 7fafaf7fe640 1 --2- 172.21.15.28:0/3377833132 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fafa40a2b90 0x7fafa40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.492+0000 7fafbda90640 1 --2- 172.21.15.28:0/3377833132 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fafa40a2b90 0x7fafa40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.492+0000 7fafaf7fe640 1 -- 172.21.15.28:0/3377833132 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8770+0+0 (secure 0 0 0) 0x7fafb40ced40 con 0x7fafb81006a0 2024-04-25T01:30:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.492+0000 7fafbda90640 1 --2- 172.21.15.28:0/3377833132 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fafa40a2b90 0x7fafa40a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fafb8198fd0 tx=0x7fafa800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.495+0000 7fafaf7fe640 1 -- 172.21.15.28:0/3377833132 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fafb4009de0 con 0x7fafb81006a0 2024-04-25T01:30:37.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:37.679+0000 7fafc04f5640 1 -- 172.21.15.28:0/3377833132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fafb802d170 con 0x7fafb81006a0 2024-04-25T01:30:38.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:37 smithi028 ceph-mon[29390]: pgmap v295: 97 pgs: 39 creating+peering, 52 active+clean, 6 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2024-04-25T01:30:38.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:37 smithi028 ceph-mon[29390]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T01:30:38.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:37 smithi028 ceph-mon[29390]: Cluster is now healthy 2024-04-25T01:30:38.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:37 smithi028 ceph-mon[29390]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T01:30:38.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:37 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2835382696' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T01:30:38.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:37 smithi028 ceph-mon[29390]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T01:30:38.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:37 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3377833132' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-25T01:30:38.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:37 smithi130 ceph-mon[32702]: pgmap v295: 97 pgs: 39 creating+peering, 52 active+clean, 6 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2024-04-25T01:30:38.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:37 smithi130 ceph-mon[32702]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T01:30:38.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:37 smithi130 ceph-mon[32702]: Cluster is now healthy 2024-04-25T01:30:38.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:37 smithi130 ceph-mon[32702]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T01:30:38.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:37 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2835382696' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T01:30:38.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:37 smithi130 ceph-mon[32702]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T01:30:38.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:37 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3377833132' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-25T01:30:38.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:37 smithi032 ceph-mon[32461]: pgmap v295: 97 pgs: 39 creating+peering, 52 active+clean, 6 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2024-04-25T01:30:38.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:37 smithi032 ceph-mon[32461]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T01:30:38.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:37 smithi032 ceph-mon[32461]: Cluster is now healthy 2024-04-25T01:30:38.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:37 smithi032 ceph-mon[32461]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T01:30:38.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:37 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2835382696' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T01:30:38.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:37 smithi032 ceph-mon[32461]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T01:30:38.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:37 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3377833132' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-25T01:30:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.355+0000 7fafaf7fe640 1 -- 172.21.15.28:0/3377833132 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v90) ==== 169+0+0 (secure 0 0 0) 0x7fafb409ba20 con 0x7fafb81006a0 2024-04-25T01:30:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 4 allow_ec_overwrites to true 2024-04-25T01:30:38.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.359+0000 7fafc04f5640 1 -- 172.21.15.28:0/3377833132 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fafa40a2b90 msgr2=0x7fafa40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:38.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.359+0000 7fafc04f5640 1 --2- 172.21.15.28:0/3377833132 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fafa40a2b90 0x7fafa40a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fafb8198fd0 tx=0x7fafa800b390 comp rx=0 tx=0).stop 2024-04-25T01:30:38.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.359+0000 7fafc04f5640 1 -- 172.21.15.28:0/3377833132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafb81006a0 msgr2=0x7fafb8197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:38.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.359+0000 7fafc04f5640 1 --2- 172.21.15.28:0/3377833132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafb81006a0 0x7fafb8197ac0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fafb400d950 tx=0x7fafb400de10 comp rx=0 tx=0).stop 2024-04-25T01:30:38.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.359+0000 7fafc04f5640 1 -- 172.21.15.28:0/3377833132 shutdown_connections 2024-04-25T01:30:38.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.359+0000 7fafc04f5640 1 --2- 172.21.15.28:0/3377833132 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fafa40a2b90 0x7fafa40a5050 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:38.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.359+0000 7fafc04f5640 1 --2- 172.21.15.28:0/3377833132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fafb8102250 0x7fafb819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:38.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.359+0000 7fafc04f5640 1 --2- 172.21.15.28:0/3377833132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafb8101890 0x7fafb8198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:38.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.359+0000 7fafc04f5640 1 --2- 172.21.15.28:0/3377833132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafb81006a0 0x7fafb8197ac0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:38.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.359+0000 7fafc04f5640 1 -- 172.21.15.28:0/3377833132 >> 172.21.15.28:0/3377833132 conn(0x7fafb80fbc30 msgr2=0x7fafb8104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:38.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.359+0000 7fafc04f5640 1 -- 172.21.15.28:0/3377833132 shutdown_connections 2024-04-25T01:30:38.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.359+0000 7fafc04f5640 1 -- 172.21.15.28:0/3377833132 wait complete. 2024-04-25T01:30:38.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-25T01:30:38.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.493+0000 7f2e837a9640 1 -- 172.21.15.28:0/3835193716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e7c101890 msgr2=0x7f2e7c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:38.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.493+0000 7f2e837a9640 1 --2- 172.21.15.28:0/3835193716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e7c101890 0x7f2e7c101d10 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c009900 tx=0x7f2e6c02a410 comp rx=0 tx=0).stop 2024-04-25T01:30:38.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.493+0000 7f2e837a9640 1 -- 172.21.15.28:0/3835193716 shutdown_connections 2024-04-25T01:30:38.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.493+0000 7f2e837a9640 1 --2- 172.21.15.28:0/3835193716 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e7c102250 0x7f2e7c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:38.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.493+0000 7f2e837a9640 1 --2- 172.21.15.28:0/3835193716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e7c101890 0x7f2e7c101d10 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:38.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.493+0000 7f2e837a9640 1 --2- 172.21.15.28:0/3835193716 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e7c1006a0 0x7f2e7c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:38.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.493+0000 7f2e837a9640 1 -- 172.21.15.28:0/3835193716 >> 172.21.15.28:0/3835193716 conn(0x7f2e7c0fbc30 msgr2=0x7f2e7c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:38.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.493+0000 7f2e837a9640 1 -- 172.21.15.28:0/3835193716 shutdown_connections 2024-04-25T01:30:38.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.493+0000 7f2e837a9640 1 -- 172.21.15.28:0/3835193716 wait complete. 2024-04-25T01:30:38.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.493+0000 7f2e837a9640 1 Processor -- start 2024-04-25T01:30:38.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.493+0000 7f2e837a9640 1 -- start start 2024-04-25T01:30:38.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e837a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e7c1006a0 0x7f2e7c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:38.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e837a9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e7c101890 0x7f2e7c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:38.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e837a9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e7c102250 0x7f2e7c19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:38.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e837a9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c10c340 con 0x7f2e7c1006a0 2024-04-25T01:30:38.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e837a9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c10c1c0 con 0x7f2e7c101890 2024-04-25T01:30:38.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e837a9640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c10c4c0 con 0x7f2e7c102250 2024-04-25T01:30:38.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e81545640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e7c1006a0 0x7f2e7c197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:38.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e81545640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e7c1006a0 0x7f2e7c197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37862/0 (socket says 172.21.15.28:37862) 2024-04-25T01:30:38.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e81545640 1 -- 172.21.15.28:0/363589393 learned_addr learned my addr 172.21.15.28:0/363589393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:38.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e80d44640 1 --2- 172.21.15.28:0/363589393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e7c101890 0x7f2e7c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:38.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e81d46640 1 --2- 172.21.15.28:0/363589393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e7c102250 0x7f2e7c19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:38.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e81545640 1 -- 172.21.15.28:0/363589393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e7c102250 msgr2=0x7f2e7c19f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:38.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e81545640 1 --2- 172.21.15.28:0/363589393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e7c102250 0x7f2e7c19f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:38.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e81545640 1 -- 172.21.15.28:0/363589393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e7c101890 msgr2=0x7f2e7c198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:38.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e81545640 1 --2- 172.21.15.28:0/363589393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e7c101890 0x7f2e7c198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:38.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e81545640 1 -- 172.21.15.28:0/363589393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e7c19f720 con 0x7f2e7c1006a0 2024-04-25T01:30:38.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e80d44640 1 --2- 172.21.15.28:0/363589393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e7c101890 0x7f2e7c198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:38.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e81545640 1 --2- 172.21.15.28:0/363589393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e7c1006a0 0x7f2e7c197b00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400e9c0 tx=0x7f2e7400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:38.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e727fc640 1 -- 172.21.15.28:0/363589393 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7400cd80 con 0x7f2e7c1006a0 2024-04-25T01:30:38.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e727fc640 1 -- 172.21.15.28:0/363589393 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e740035f0 con 0x7f2e7c1006a0 2024-04-25T01:30:38.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e837a9640 1 -- 172.21.15.28:0/363589393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e7c19fa10 con 0x7f2e7c1006a0 2024-04-25T01:30:38.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e727fc640 1 -- 172.21.15.28:0/363589393 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e74010690 con 0x7f2e7c1006a0 2024-04-25T01:30:38.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.494+0000 7f2e837a9640 1 -- 172.21.15.28:0/363589393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e7c19fee0 con 0x7f2e7c1006a0 2024-04-25T01:30:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.496+0000 7f2e837a9640 1 -- 172.21.15.28:0/363589393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e7c1a0170 con 0x7f2e7c1006a0 2024-04-25T01:30:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.497+0000 7f2e727fc640 1 -- 172.21.15.28:0/363589393 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2e740037a0 con 0x7f2e7c1006a0 2024-04-25T01:30:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.497+0000 7f2e727fc640 1 --2- 172.21.15.28:0/363589393 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f2e680a2b90 0x7f2e680a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.498+0000 7f2e80d44640 1 --2- 172.21.15.28:0/363589393 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f2e680a2b90 0x7f2e680a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.498+0000 7f2e727fc640 1 -- 172.21.15.28:0/363589393 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8770+0+0 (secure 0 0 0) 0x7f2e740cf750 con 0x7f2e7c1006a0 2024-04-25T01:30:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.498+0000 7f2e80d44640 1 --2- 172.21.15.28:0/363589393 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f2e680a2b90 0x7f2e680a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c000c00 tx=0x7f2e6c0095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:38.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.502+0000 7f2e727fc640 1 -- 172.21.15.28:0/363589393 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2e7409c490 con 0x7f2e7c1006a0 2024-04-25T01:30:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:38.728+0000 7f2e837a9640 1 -- 172.21.15.28:0/363589393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f2e7c1a0900 con 0x7f2e7c1006a0 2024-04-25T01:30:39.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.361+0000 7f2e727fc640 1 -- 172.21.15.28:0/363589393 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f2e7409bd90 con 0x7f2e7c1006a0 2024-04-25T01:30:39.339 INFO:teuthology.orchestra.run.smithi028.stderr:added data pool 4 to fsmap 2024-04-25T01:30:39.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.365+0000 7f2e837a9640 1 -- 172.21.15.28:0/363589393 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f2e680a2b90 msgr2=0x7f2e680a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:39.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.365+0000 7f2e837a9640 1 --2- 172.21.15.28:0/363589393 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f2e680a2b90 0x7f2e680a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c000c00 tx=0x7f2e6c0095f0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.365+0000 7f2e837a9640 1 -- 172.21.15.28:0/363589393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e7c1006a0 msgr2=0x7f2e7c197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:39.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.365+0000 7f2e837a9640 1 --2- 172.21.15.28:0/363589393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e7c1006a0 0x7f2e7c197b00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400e9c0 tx=0x7f2e7400ee80 comp rx=0 tx=0).stop 2024-04-25T01:30:39.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.365+0000 7f2e837a9640 1 -- 172.21.15.28:0/363589393 shutdown_connections 2024-04-25T01:30:39.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.365+0000 7f2e837a9640 1 --2- 172.21.15.28:0/363589393 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f2e680a2b90 0x7f2e680a5050 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.365+0000 7f2e837a9640 1 --2- 172.21.15.28:0/363589393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2e7c102250 0x7f2e7c19f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.365+0000 7f2e837a9640 1 --2- 172.21.15.28:0/363589393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2e7c101890 0x7f2e7c198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.365+0000 7f2e837a9640 1 --2- 172.21.15.28:0/363589393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e7c1006a0 0x7f2e7c197b00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.365+0000 7f2e837a9640 1 -- 172.21.15.28:0/363589393 >> 172.21.15.28:0/363589393 conn(0x7f2e7c0fbc30 msgr2=0x7f2e7c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:39.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.365+0000 7f2e837a9640 1 -- 172.21.15.28:0/363589393 shutdown_connections 2024-04-25T01:30:39.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.365+0000 7f2e837a9640 1 -- 172.21.15.28:0/363589393 wait complete. 2024-04-25T01:30:39.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:30:39.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.491+0000 7ff41b0f5640 1 -- 172.21.15.28:0/1982080640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff414104c10 msgr2=0x7ff414107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.491+0000 7ff41b0f5640 1 --2- 172.21.15.28:0/1982080640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff414104c10 0x7ff414107000 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff408009900 tx=0x7ff40802a410 comp rx=0 tx=0).stop 2024-04-25T01:30:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.491+0000 7ff41b0f5640 1 -- 172.21.15.28:0/1982080640 shutdown_connections 2024-04-25T01:30:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.491+0000 7ff41b0f5640 1 --2- 172.21.15.28:0/1982080640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff414107540 0x7ff414109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.491+0000 7ff41b0f5640 1 --2- 172.21.15.28:0/1982080640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff414104c10 0x7ff414107000 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.491+0000 7ff41b0f5640 1 --2- 172.21.15.28:0/1982080640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4141022e0 0x7ff4141046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.491+0000 7ff41b0f5640 1 -- 172.21.15.28:0/1982080640 >> 172.21.15.28:0/1982080640 conn(0x7ff4140fbc30 msgr2=0x7ff4140fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.491+0000 7ff41b0f5640 1 -- 172.21.15.28:0/1982080640 shutdown_connections 2024-04-25T01:30:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.491+0000 7ff41b0f5640 1 -- 172.21.15.28:0/1982080640 wait complete. 2024-04-25T01:30:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.492+0000 7ff41b0f5640 1 Processor -- start 2024-04-25T01:30:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.492+0000 7ff41b0f5640 1 -- start start 2024-04-25T01:30:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.492+0000 7ff41b0f5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4141022e0 0x7ff414101c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.492+0000 7ff41b0f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff414104c10 0x7ff414100290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.492+0000 7ff41b0f5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff414107540 0x7ff4141007d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.492+0000 7ff41b0f5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff41410d400 con 0x7ff414104c10 2024-04-25T01:30:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.492+0000 7ff41b0f5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff41410d280 con 0x7ff414107540 2024-04-25T01:30:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.492+0000 7ff41b0f5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff41410d580 con 0x7ff4141022e0 2024-04-25T01:30:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.492+0000 7ff413fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff414104c10 0x7ff414100290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.492+0000 7ff413fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff414104c10 0x7ff414100290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37890/0 (socket says 172.21.15.28:37890) 2024-04-25T01:30:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.492+0000 7ff413fff640 1 -- 172.21.15.28:0/627235147 learned_addr learned my addr 172.21.15.28:0/627235147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.492+0000 7ff418e91640 1 --2- 172.21.15.28:0/627235147 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4141022e0 0x7ff414101c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.493+0000 7ff419692640 1 --2- 172.21.15.28:0/627235147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff414107540 0x7ff4141007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.493+0000 7ff413fff640 1 -- 172.21.15.28:0/627235147 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4141022e0 msgr2=0x7ff414101c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.493+0000 7ff413fff640 1 --2- 172.21.15.28:0/627235147 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4141022e0 0x7ff414101c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.493+0000 7ff413fff640 1 -- 172.21.15.28:0/627235147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff414107540 msgr2=0x7ff4141007d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.493+0000 7ff413fff640 1 --2- 172.21.15.28:0/627235147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff414107540 0x7ff4141007d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.493+0000 7ff413fff640 1 -- 172.21.15.28:0/627235147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff414101060 con 0x7ff414104c10 2024-04-25T01:30:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.493+0000 7ff419692640 1 --2- 172.21.15.28:0/627235147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff414107540 0x7ff4141007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.493+0000 7ff413fff640 1 --2- 172.21.15.28:0/627235147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff414104c10 0x7ff414100290 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff408031610 tx=0x7ff4080043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.493+0000 7ff411ffb640 1 -- 172.21.15.28:0/627235147 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff40803f030 con 0x7ff414104c10 2024-04-25T01:30:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.493+0000 7ff411ffb640 1 -- 172.21.15.28:0/627235147 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff408003ce0 con 0x7ff414104c10 2024-04-25T01:30:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.493+0000 7ff411ffb640 1 -- 172.21.15.28:0/627235147 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff408035070 con 0x7ff414104c10 2024-04-25T01:30:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.493+0000 7ff41b0f5640 1 -- 172.21.15.28:0/627235147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4141a1bc0 con 0x7ff414104c10 2024-04-25T01:30:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.493+0000 7ff41b0f5640 1 -- 172.21.15.28:0/627235147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4141a2110 con 0x7ff414104c10 2024-04-25T01:30:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.494+0000 7ff41b0f5640 1 -- 172.21.15.28:0/627235147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4140fd720 con 0x7ff414104c10 2024-04-25T01:30:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.495+0000 7ff411ffb640 1 -- 172.21.15.28:0/627235147 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff408042030 con 0x7ff414104c10 2024-04-25T01:30:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.496+0000 7ff411ffb640 1 --2- 172.21.15.28:0/627235147 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff3fc0a2b90 0x7ff3fc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.496+0000 7ff418e91640 1 --2- 172.21.15.28:0/627235147 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff3fc0a2b90 0x7ff3fc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.496+0000 7ff411ffb640 1 -- 172.21.15.28:0/627235147 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7ff4080ef850 con 0x7ff414104c10 2024-04-25T01:30:39.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.497+0000 7ff418e91640 1 --2- 172.21.15.28:0/627235147 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff3fc0a2b90 0x7ff3fc0a5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff40400bbe0 tx=0x7ff40400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:39.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.500+0000 7ff411ffb640 1 -- 172.21.15.28:0/627235147 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff4080bc570 con 0x7ff414104c10 2024-04-25T01:30:39.484 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:30:39 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-b[63160]: 2024-04-25T01:30:39.123+0000 7f7824fdf700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T01:30:39.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:39 smithi032 ceph-mon[32461]: pgmap v298: 129 pgs: 41 creating+peering, 64 active+clean, 24 unknown; 577 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 7 op/s 2024-04-25T01:30:39.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:39 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3377833132' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-25T01:30:39.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:39 smithi032 ceph-mon[32461]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T01:30:39.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:39 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/363589393' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-25T01:30:39.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:39 smithi032 ceph-mon[32461]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:30:39.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:39 smithi032 ceph-mon[32461]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T01:30:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.727+0000 7ff41b0f5640 1 -- 172.21.15.28:0/627235147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff414062370 con 0x7ff414104c10 2024-04-25T01:30:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.727+0000 7ff411ffb640 1 -- 172.21.15.28:0/627235147 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11788 (secure 0 0 0) 0x7ff4080bbe70 con 0x7ff414104c10 2024-04-25T01:30:39.706 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:39.706 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:30:38.730500+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24512,"qdb_cluster":[24512]},"id":1}]} 2024-04-25T01:30:39.706 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2024-04-25T01:30:39.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.730+0000 7ff41b0f5640 1 -- 172.21.15.28:0/627235147 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff3fc0a2b90 msgr2=0x7ff3fc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:39.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.730+0000 7ff41b0f5640 1 --2- 172.21.15.28:0/627235147 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff3fc0a2b90 0x7ff3fc0a5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff40400bbe0 tx=0x7ff40400b390 comp rx=0 tx=0).stop 2024-04-25T01:30:39.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.730+0000 7ff41b0f5640 1 -- 172.21.15.28:0/627235147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff414104c10 msgr2=0x7ff414100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:39.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.730+0000 7ff41b0f5640 1 --2- 172.21.15.28:0/627235147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff414104c10 0x7ff414100290 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff408031610 tx=0x7ff4080043b0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.730+0000 7ff41b0f5640 1 -- 172.21.15.28:0/627235147 shutdown_connections 2024-04-25T01:30:39.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.730+0000 7ff41b0f5640 1 --2- 172.21.15.28:0/627235147 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff3fc0a2b90 0x7ff3fc0a5050 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.730+0000 7ff41b0f5640 1 --2- 172.21.15.28:0/627235147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff414107540 0x7ff4141007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.730+0000 7ff41b0f5640 1 --2- 172.21.15.28:0/627235147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff414104c10 0x7ff414100290 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.730+0000 7ff41b0f5640 1 --2- 172.21.15.28:0/627235147 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4141022e0 0x7ff414101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.730+0000 7ff41b0f5640 1 -- 172.21.15.28:0/627235147 >> 172.21.15.28:0/627235147 conn(0x7ff4140fbc30 msgr2=0x7ff414102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:39.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.730+0000 7ff41b0f5640 1 -- 172.21.15.28:0/627235147 shutdown_connections 2024-04-25T01:30:39.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.730+0000 7ff41b0f5640 1 -- 172.21.15.28:0/627235147 wait complete. 2024-04-25T01:30:39.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:39 smithi130 ceph-mon[32702]: pgmap v298: 129 pgs: 41 creating+peering, 64 active+clean, 24 unknown; 577 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 7 op/s 2024-04-25T01:30:39.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:39 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3377833132' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-25T01:30:39.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:39 smithi130 ceph-mon[32702]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T01:30:39.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:39 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/363589393' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-25T01:30:39.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:39 smithi130 ceph-mon[32702]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:30:39.710 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:39 smithi130 ceph-mon[32702]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T01:30:39.718 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:39 smithi028 ceph-mon[29390]: pgmap v298: 129 pgs: 41 creating+peering, 64 active+clean, 24 unknown; 577 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 7 op/s 2024-04-25T01:30:39.718 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:39 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3377833132' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-25T01:30:39.718 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:39 smithi028 ceph-mon[29390]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T01:30:39.718 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:39 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/363589393' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-25T01:30:39.719 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:39 smithi028 ceph-mon[29390]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:30:39.719 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:39 smithi028 ceph-mon[29390]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T01:30:39.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:30:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.843+0000 7f0911f12640 1 -- 172.21.15.28:0/380943499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f090c102280 msgr2=0x7f090c104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.843+0000 7f0911f12640 1 --2- 172.21.15.28:0/380943499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f090c102280 0x7f090c104670 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f08fc009900 tx=0x7f08fc02a410 comp rx=0 tx=0).stop 2024-04-25T01:30:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.843+0000 7f0911f12640 1 -- 172.21.15.28:0/380943499 shutdown_connections 2024-04-25T01:30:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.843+0000 7f0911f12640 1 --2- 172.21.15.28:0/380943499 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f090c1074e0 0x7f090c1098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.843+0000 7f0911f12640 1 --2- 172.21.15.28:0/380943499 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f090c104bb0 0x7f090c106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.843+0000 7f0911f12640 1 --2- 172.21.15.28:0/380943499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f090c102280 0x7f090c104670 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.843+0000 7f0911f12640 1 -- 172.21.15.28:0/380943499 >> 172.21.15.28:0/380943499 conn(0x7f090c0fbc30 msgr2=0x7f090c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.843+0000 7f0911f12640 1 -- 172.21.15.28:0/380943499 shutdown_connections 2024-04-25T01:30:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.843+0000 7f0911f12640 1 -- 172.21.15.28:0/380943499 wait complete. 2024-04-25T01:30:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.843+0000 7f0911f12640 1 Processor -- start 2024-04-25T01:30:39.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.844+0000 7f0911f12640 1 -- start start 2024-04-25T01:30:39.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.844+0000 7f0911f12640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f090c102280 0x7f090c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.844+0000 7f0911f12640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f090c104bb0 0x7f090c198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.844+0000 7f0911f12640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f090c1074e0 0x7f090c19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.844+0000 7f0911f12640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f090c10d490 con 0x7f090c1074e0 2024-04-25T01:30:39.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.844+0000 7f0911f12640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f090c10d310 con 0x7f090c102280 2024-04-25T01:30:39.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.844+0000 7f0911f12640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f090c10d610 con 0x7f090c104bb0 2024-04-25T01:30:39.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.844+0000 7f090bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f090c1074e0 0x7f090c19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.844+0000 7f090bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f090c1074e0 0x7f090c19f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37910/0 (socket says 172.21.15.28:37910) 2024-04-25T01:30:39.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.844+0000 7f090bfff640 1 -- 172.21.15.28:0/3696944235 learned_addr learned my addr 172.21.15.28:0/3696944235 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.844+0000 7f090b7fe640 1 --2- 172.21.15.28:0/3696944235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f090c102280 0x7f090c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.844+0000 7f090affd640 1 --2- 172.21.15.28:0/3696944235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f090c104bb0 0x7f090c198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.844+0000 7f090bfff640 1 -- 172.21.15.28:0/3696944235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f090c104bb0 msgr2=0x7f090c198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.844+0000 7f090bfff640 1 --2- 172.21.15.28:0/3696944235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f090c104bb0 0x7f090c198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.844+0000 7f090bfff640 1 -- 172.21.15.28:0/3696944235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f090c102280 msgr2=0x7f090c197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.844+0000 7f090bfff640 1 --2- 172.21.15.28:0/3696944235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f090c102280 0x7f090c197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.844+0000 7f090bfff640 1 -- 172.21.15.28:0/3696944235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f090c19f6f0 con 0x7f090c1074e0 2024-04-25T01:30:39.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.844+0000 7f090affd640 1 --2- 172.21.15.28:0/3696944235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f090c104bb0 0x7f090c198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:39.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.844+0000 7f090bfff640 1 --2- 172.21.15.28:0/3696944235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f090c1074e0 0x7f090c19f080 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f090400cce0 tx=0x7f090400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:39.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.845+0000 7f0908ff9640 1 -- 172.21.15.28:0/3696944235 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090400f450 con 0x7f090c1074e0 2024-04-25T01:30:39.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.845+0000 7f0911f12640 1 -- 172.21.15.28:0/3696944235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f090c19f9e0 con 0x7f090c1074e0 2024-04-25T01:30:39.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.845+0000 7f0908ff9640 1 -- 172.21.15.28:0/3696944235 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f090400fad0 con 0x7f090c1074e0 2024-04-25T01:30:39.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.845+0000 7f0908ff9640 1 -- 172.21.15.28:0/3696944235 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0904018c10 con 0x7f090c1074e0 2024-04-25T01:30:39.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.845+0000 7f0911f12640 1 -- 172.21.15.28:0/3696944235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f090c19ff60 con 0x7f090c1074e0 2024-04-25T01:30:39.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.846+0000 7f0911f12640 1 -- 172.21.15.28:0/3696944235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f090c0fd720 con 0x7f090c1074e0 2024-04-25T01:30:39.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.847+0000 7f0908ff9640 1 -- 172.21.15.28:0/3696944235 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f090400f5f0 con 0x7f090c1074e0 2024-04-25T01:30:39.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.847+0000 7f0908ff9640 1 --2- 172.21.15.28:0/3696944235 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f08f40a2b90 0x7f08f40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:39.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.847+0000 7f0908ff9640 1 -- 172.21.15.28:0/3696944235 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f09040cf970 con 0x7f090c1074e0 2024-04-25T01:30:39.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.847+0000 7f090b7fe640 1 --2- 172.21.15.28:0/3696944235 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f08f40a2b90 0x7f08f40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:39.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.848+0000 7f090b7fe640 1 --2- 172.21.15.28:0/3696944235 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f08f40a2b90 0x7f08f40a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f08fc0314e0 tx=0x7f08fc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:39.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:39.852+0000 7f0908ff9640 1 -- 172.21.15.28:0/3696944235 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0904014030 con 0x7f090c1074e0 2024-04-25T01:30:40.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.037+0000 7f0911f12640 1 -- 172.21.15.28:0/3696944235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f090c062340 con 0x7f090c1074e0 2024-04-25T01:30:40.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.039+0000 7f0908ff9640 1 -- 172.21.15.28:0/3696944235 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24093 (secure 0 0 0) 0x7f090409c690 con 0x7f090c1074e0 2024-04-25T01:30:40.017 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:40.018 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:30:38.730329+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:30:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.041+0000 7f0911f12640 1 -- 172.21.15.28:0/3696944235 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f08f40a2b90 msgr2=0x7f08f40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.041+0000 7f0911f12640 1 --2- 172.21.15.28:0/3696944235 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f08f40a2b90 0x7f08f40a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f08fc0314e0 tx=0x7f08fc000bc0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.041+0000 7f0911f12640 1 -- 172.21.15.28:0/3696944235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f090c1074e0 msgr2=0x7f090c19f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.041+0000 7f0911f12640 1 --2- 172.21.15.28:0/3696944235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f090c1074e0 0x7f090c19f080 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f090400cce0 tx=0x7f090400a620 comp rx=0 tx=0).stop 2024-04-25T01:30:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.041+0000 7f0911f12640 1 -- 172.21.15.28:0/3696944235 shutdown_connections 2024-04-25T01:30:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.041+0000 7f0911f12640 1 --2- 172.21.15.28:0/3696944235 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f08f40a2b90 0x7f08f40a5050 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.041+0000 7f0911f12640 1 --2- 172.21.15.28:0/3696944235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f090c1074e0 0x7f090c19f080 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.041+0000 7f0911f12640 1 --2- 172.21.15.28:0/3696944235 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f090c104bb0 0x7f090c198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.041+0000 7f0911f12640 1 --2- 172.21.15.28:0/3696944235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f090c102280 0x7f090c197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.041+0000 7f0911f12640 1 -- 172.21.15.28:0/3696944235 >> 172.21.15.28:0/3696944235 conn(0x7f090c0fbc30 msgr2=0x7f090c102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.041+0000 7f0911f12640 1 -- 172.21.15.28:0/3696944235 shutdown_connections 2024-04-25T01:30:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.041+0000 7f0911f12640 1 -- 172.21.15.28:0/3696944235 wait complete. 2024-04-25T01:30:40.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:30:40.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.154+0000 7f716f0a5640 1 -- 172.21.15.28:0/980451803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7168100680 msgr2=0x7f7168100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:40.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.154+0000 7f716f0a5640 1 --2- 172.21.15.28:0/980451803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7168100680 0x7f7168100aa0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7160009ca0 tx=0x7f716002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.155+0000 7f716f0a5640 1 -- 172.21.15.28:0/980451803 shutdown_connections 2024-04-25T01:30:40.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.155+0000 7f716f0a5640 1 --2- 172.21.15.28:0/980451803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7168102230 0x7f7168108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.155+0000 7f716f0a5640 1 --2- 172.21.15.28:0/980451803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7168101870 0x7f7168101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.155+0000 7f716f0a5640 1 --2- 172.21.15.28:0/980451803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7168100680 0x7f7168100aa0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.155+0000 7f716f0a5640 1 -- 172.21.15.28:0/980451803 >> 172.21.15.28:0/980451803 conn(0x7f71680fbbf0 msgr2=0x7f71680fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:40.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.155+0000 7f716f0a5640 1 -- 172.21.15.28:0/980451803 shutdown_connections 2024-04-25T01:30:40.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.155+0000 7f716f0a5640 1 -- 172.21.15.28:0/980451803 wait complete. 2024-04-25T01:30:40.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.155+0000 7f716f0a5640 1 Processor -- start 2024-04-25T01:30:40.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.155+0000 7f716f0a5640 1 -- start start 2024-04-25T01:30:40.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.155+0000 7f716f0a5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7168100680 0x7f7168197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:40.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.155+0000 7f716f0a5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7168101870 0x7f7168197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:40.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.155+0000 7f716f0a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7168102230 0x7f716819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:40.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.155+0000 7f716f0a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f716810c2d0 con 0x7f7168102230 2024-04-25T01:30:40.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.155+0000 7f716f0a5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f716810c150 con 0x7f7168101870 2024-04-25T01:30:40.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.155+0000 7f716f0a5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f716810c450 con 0x7f7168100680 2024-04-25T01:30:40.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.156+0000 7f716d642640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7168102230 0x7f716819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:40.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.156+0000 7f716d642640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7168102230 0x7f716819f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37924/0 (socket says 172.21.15.28:37924) 2024-04-25T01:30:40.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.156+0000 7f716d642640 1 -- 172.21.15.28:0/3703313296 learned_addr learned my addr 172.21.15.28:0/3703313296 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:40.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.156+0000 7f716ce41640 1 --2- 172.21.15.28:0/3703313296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7168100680 0x7f7168197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:40.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.156+0000 7f7167fff640 1 --2- 172.21.15.28:0/3703313296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7168101870 0x7f7168197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:40.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.156+0000 7f716d642640 1 -- 172.21.15.28:0/3703313296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7168100680 msgr2=0x7f7168197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:40.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.156+0000 7f716d642640 1 --2- 172.21.15.28:0/3703313296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7168100680 0x7f7168197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.156+0000 7f716d642640 1 -- 172.21.15.28:0/3703313296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7168101870 msgr2=0x7f7168197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:40.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.156+0000 7f716d642640 1 --2- 172.21.15.28:0/3703313296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7168101870 0x7f7168197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.156+0000 7f716d642640 1 -- 172.21.15.28:0/3703313296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716819f680 con 0x7f7168102230 2024-04-25T01:30:40.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.156+0000 7f716d642640 1 --2- 172.21.15.28:0/3703313296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7168102230 0x7f716819f010 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f715800c970 tx=0x7f715800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:40.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.156+0000 7f7165ffb640 1 -- 172.21.15.28:0/3703313296 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7158007bb0 con 0x7f7168102230 2024-04-25T01:30:40.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.156+0000 7f716f0a5640 1 -- 172.21.15.28:0/3703313296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f716819f970 con 0x7f7168102230 2024-04-25T01:30:40.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.156+0000 7f7165ffb640 1 -- 172.21.15.28:0/3703313296 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7158007d50 con 0x7f7168102230 2024-04-25T01:30:40.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.156+0000 7f7165ffb640 1 -- 172.21.15.28:0/3703313296 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7158018800 con 0x7f7168102230 2024-04-25T01:30:40.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.156+0000 7f716f0a5640 1 -- 172.21.15.28:0/3703313296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f716819fe70 con 0x7f7168102230 2024-04-25T01:30:40.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.157+0000 7f716f0a5640 1 -- 172.21.15.28:0/3703313296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71681a0100 con 0x7f7168102230 2024-04-25T01:30:40.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.158+0000 7f7165ffb640 1 -- 172.21.15.28:0/3703313296 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f715800fa50 con 0x7f7168102230 2024-04-25T01:30:40.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.159+0000 7f7165ffb640 1 --2- 172.21.15.28:0/3703313296 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f71540a2b90 0x7f71540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:40.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.159+0000 7f7165ffb640 1 -- 172.21.15.28:0/3703313296 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f71580cf320 con 0x7f7168102230 2024-04-25T01:30:40.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.159+0000 7f716ce41640 1 --2- 172.21.15.28:0/3703313296 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f71540a2b90 0x7f71540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:40.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.159+0000 7f716ce41640 1 --2- 172.21.15.28:0/3703313296 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f71540a2b90 0x7f71540a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7160006010 tx=0x7f7160000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:40.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.163+0000 7f7165ffb640 1 -- 172.21.15.28:0/3703313296 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7158014030 con 0x7f7168102230 2024-04-25T01:30:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.351+0000 7f716f0a5640 1 -- 172.21.15.28:0/3703313296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f71681a0370 con 0x7f7168102230 2024-04-25T01:30:40.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.352+0000 7f7165ffb640 1 -- 172.21.15.28:0/3703313296 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24093 (secure 0 0 0) 0x7f715809bfe0 con 0x7f7168102230 2024-04-25T01:30:40.331 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:40.331 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:30:38.730329+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:30:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.354+0000 7f716f0a5640 1 -- 172.21.15.28:0/3703313296 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f71540a2b90 msgr2=0x7f71540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.354+0000 7f716f0a5640 1 --2- 172.21.15.28:0/3703313296 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f71540a2b90 0x7f71540a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7160006010 tx=0x7f7160000f40 comp rx=0 tx=0).stop 2024-04-25T01:30:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.354+0000 7f716f0a5640 1 -- 172.21.15.28:0/3703313296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7168102230 msgr2=0x7f716819f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.354+0000 7f716f0a5640 1 --2- 172.21.15.28:0/3703313296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7168102230 0x7f716819f010 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f715800c970 tx=0x7f715800ce30 comp rx=0 tx=0).stop 2024-04-25T01:30:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.354+0000 7f716f0a5640 1 -- 172.21.15.28:0/3703313296 shutdown_connections 2024-04-25T01:30:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.354+0000 7f716f0a5640 1 --2- 172.21.15.28:0/3703313296 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f71540a2b90 0x7f71540a5050 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.354+0000 7f716f0a5640 1 --2- 172.21.15.28:0/3703313296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7168102230 0x7f716819f010 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.354+0000 7f716f0a5640 1 --2- 172.21.15.28:0/3703313296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7168101870 0x7f7168197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.354+0000 7f716f0a5640 1 --2- 172.21.15.28:0/3703313296 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7168100680 0x7f7168197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.354+0000 7f716f0a5640 1 -- 172.21.15.28:0/3703313296 >> 172.21.15.28:0/3703313296 conn(0x7f71680fbbf0 msgr2=0x7f71680fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.354+0000 7f716f0a5640 1 -- 172.21.15.28:0/3703313296 shutdown_connections 2024-04-25T01:30:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.355+0000 7f716f0a5640 1 -- 172.21.15.28:0/3703313296 wait complete. 2024-04-25T01:30:40.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:30:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.483+0000 7f361d2da640 1 -- 172.21.15.28:0/1189686090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36181029a0 msgr2=0x7f3618102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.483+0000 7f361d2da640 1 --2- 172.21.15.28:0/1189686090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36181029a0 0x7f3618102e20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f360c009900 tx=0x7f360c02a410 comp rx=0 tx=0).stop 2024-04-25T01:30:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.483+0000 7f361d2da640 1 -- 172.21.15.28:0/1189686090 shutdown_connections 2024-04-25T01:30:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.483+0000 7f361d2da640 1 --2- 172.21.15.28:0/1189686090 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3618103360 0x7f3618109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.483+0000 7f361d2da640 1 --2- 172.21.15.28:0/1189686090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36181029a0 0x7f3618102e20 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.483+0000 7f361d2da640 1 --2- 172.21.15.28:0/1189686090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36181017b0 0x7f3618101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.483+0000 7f361d2da640 1 -- 172.21.15.28:0/1189686090 >> 172.21.15.28:0/1189686090 conn(0x7f36180fcd40 msgr2=0x7f36180ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.483+0000 7f361d2da640 1 -- 172.21.15.28:0/1189686090 shutdown_connections 2024-04-25T01:30:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.483+0000 7f361d2da640 1 -- 172.21.15.28:0/1189686090 wait complete. 2024-04-25T01:30:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.484+0000 7f361d2da640 1 Processor -- start 2024-04-25T01:30:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.484+0000 7f361d2da640 1 -- start start 2024-04-25T01:30:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.484+0000 7f361d2da640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36181017b0 0x7f36180754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.484+0000 7f361d2da640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36181029a0 0x7f3618073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.484+0000 7f361d2da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3618103360 0x7f3618074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.484+0000 7f361d2da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f361810d520 con 0x7f3618103360 2024-04-25T01:30:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.484+0000 7f361d2da640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f361810d3a0 con 0x7f36181029a0 2024-04-25T01:30:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.484+0000 7f361d2da640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f361810d6a0 con 0x7f36181017b0 2024-04-25T01:30:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.484+0000 7f36177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3618103360 0x7f3618074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.484+0000 7f36177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3618103360 0x7f3618074070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37946/0 (socket says 172.21.15.28:37946) 2024-04-25T01:30:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.484+0000 7f36177fe640 1 -- 172.21.15.28:0/3269608519 learned_addr learned my addr 172.21.15.28:0/3269608519 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.484+0000 7f3616ffd640 1 --2- 172.21.15.28:0/3269608519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36181017b0 0x7f36180754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.484+0000 7f36177fe640 1 -- 172.21.15.28:0/3269608519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36181017b0 msgr2=0x7f36180754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.484+0000 7f36167fc640 1 --2- 172.21.15.28:0/3269608519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36181029a0 0x7f3618073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.484+0000 7f36177fe640 1 --2- 172.21.15.28:0/3269608519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36181017b0 0x7f36180754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.484+0000 7f36177fe640 1 -- 172.21.15.28:0/3269608519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36181029a0 msgr2=0x7f3618073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.484+0000 7f36177fe640 1 --2- 172.21.15.28:0/3269608519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36181029a0 0x7f3618073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.484+0000 7f36177fe640 1 -- 172.21.15.28:0/3269608519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3618074900 con 0x7f3618103360 2024-04-25T01:30:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.485+0000 7f36177fe640 1 --2- 172.21.15.28:0/3269608519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3618103360 0x7f3618074070 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f361000b970 tx=0x7f361000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.485+0000 7f35fffff640 1 -- 172.21.15.28:0/3269608519 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3610019070 con 0x7f3618103360 2024-04-25T01:30:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.485+0000 7f35fffff640 1 -- 172.21.15.28:0/3269608519 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f361000cb40 con 0x7f3618103360 2024-04-25T01:30:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.485+0000 7f361d2da640 1 -- 172.21.15.28:0/3269608519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36181a1d10 con 0x7f3618103360 2024-04-25T01:30:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.485+0000 7f35fffff640 1 -- 172.21.15.28:0/3269608519 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3610007650 con 0x7f3618103360 2024-04-25T01:30:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.485+0000 7f361d2da640 1 -- 172.21.15.28:0/3269608519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36181a2230 con 0x7f3618103360 2024-04-25T01:30:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.486+0000 7f361d2da640 1 -- 172.21.15.28:0/3269608519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3618108b70 con 0x7f3618103360 2024-04-25T01:30:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.487+0000 7f35fffff640 1 -- 172.21.15.28:0/3269608519 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f361001c070 con 0x7f3618103360 2024-04-25T01:30:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.487+0000 7f35fffff640 1 --2- 172.21.15.28:0/3269608519 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f36000a2b90 0x7f36000a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.488+0000 7f3616ffd640 1 --2- 172.21.15.28:0/3269608519 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f36000a2b90 0x7f36000a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.488+0000 7f35fffff640 1 -- 172.21.15.28:0/3269608519 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f36100cee00 con 0x7f3618103360 2024-04-25T01:30:40.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.488+0000 7f3616ffd640 1 --2- 172.21.15.28:0/3269608519 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f36000a2b90 0x7f36000a5050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3618102800 tx=0x7f3608008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:40.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.491+0000 7f35fffff640 1 -- 172.21.15.28:0/3269608519 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f361009bac0 con 0x7f3618103360 2024-04-25T01:30:40.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.713+0000 7f361d2da640 1 -- 172.21.15.28:0/3269608519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f361802d170 con 0x7f3618103360 2024-04-25T01:30:40.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.713+0000 7f35fffff640 1 -- 172.21.15.28:0/3269608519 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11788 (secure 0 0 0) 0x7f361009b3c0 con 0x7f3618103360 2024-04-25T01:30:40.692 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:40.692 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14643,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14655,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24401,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:30:38.730500+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24512,"qdb_cluster":[24512]},"id":1}]} 2024-04-25T01:30:40.692 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2024-04-25T01:30:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.715+0000 7f361d2da640 1 -- 172.21.15.28:0/3269608519 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f36000a2b90 msgr2=0x7f36000a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.715+0000 7f361d2da640 1 --2- 172.21.15.28:0/3269608519 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f36000a2b90 0x7f36000a5050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3618102800 tx=0x7f3608008040 comp rx=0 tx=0).stop 2024-04-25T01:30:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.715+0000 7f361d2da640 1 -- 172.21.15.28:0/3269608519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3618103360 msgr2=0x7f3618074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.715+0000 7f361d2da640 1 --2- 172.21.15.28:0/3269608519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3618103360 0x7f3618074070 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f361000b970 tx=0x7f361000be30 comp rx=0 tx=0).stop 2024-04-25T01:30:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.715+0000 7f361d2da640 1 -- 172.21.15.28:0/3269608519 shutdown_connections 2024-04-25T01:30:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.715+0000 7f361d2da640 1 --2- 172.21.15.28:0/3269608519 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f36000a2b90 0x7f36000a5050 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.715+0000 7f361d2da640 1 --2- 172.21.15.28:0/3269608519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3618103360 0x7f3618074070 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.715+0000 7f361d2da640 1 --2- 172.21.15.28:0/3269608519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36181029a0 0x7f3618073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.715+0000 7f361d2da640 1 --2- 172.21.15.28:0/3269608519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f36181017b0 0x7f36180754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.715+0000 7f361d2da640 1 -- 172.21.15.28:0/3269608519 >> 172.21.15.28:0/3269608519 conn(0x7f36180fcd40 msgr2=0x7f3618105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:40.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.716+0000 7f361d2da640 1 -- 172.21.15.28:0/3269608519 shutdown_connections 2024-04-25T01:30:40.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.716+0000 7f361d2da640 1 -- 172.21.15.28:0/3269608519 wait complete. 2024-04-25T01:30:40.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:40 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/363589393' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-25T01:30:40.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:40 smithi028 ceph-mon[29390]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:30:40.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:40 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/627235147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:40.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:40 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3696944235' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:40.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:40 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3703313296' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:40.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:30:40.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:40 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/363589393' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-25T01:30:40.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:40 smithi130 ceph-mon[32702]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:30:40.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:40 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/627235147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:40.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:40 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3696944235' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:40.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:40 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3703313296' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:40.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:40 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/363589393' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-25T01:30:40.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:40 smithi032 ceph-mon[32461]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:30:40.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:40 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/627235147' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:40.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:40 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3696944235' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:40.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:40 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3703313296' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.828+0000 7fd5662da640 1 -- 172.21.15.28:0/859855005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5601017b0 msgr2=0x7fd560101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.828+0000 7fd5662da640 1 --2- 172.21.15.28:0/859855005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5601017b0 0x7fd560101bd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd550009900 tx=0x7fd55002a410 comp rx=0 tx=0).stop 2024-04-25T01:30:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.828+0000 7fd5662da640 1 -- 172.21.15.28:0/859855005 shutdown_connections 2024-04-25T01:30:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.828+0000 7fd5662da640 1 --2- 172.21.15.28:0/859855005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd560103360 0x7fd560109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.828+0000 7fd5662da640 1 --2- 172.21.15.28:0/859855005 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd5601029a0 0x7fd560102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.828+0000 7fd5662da640 1 --2- 172.21.15.28:0/859855005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5601017b0 0x7fd560101bd0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.828+0000 7fd5662da640 1 -- 172.21.15.28:0/859855005 >> 172.21.15.28:0/859855005 conn(0x7fd5600fcd40 msgr2=0x7fd5600ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.828+0000 7fd5662da640 1 -- 172.21.15.28:0/859855005 shutdown_connections 2024-04-25T01:30:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.828+0000 7fd5662da640 1 -- 172.21.15.28:0/859855005 wait complete. 2024-04-25T01:30:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.829+0000 7fd5662da640 1 Processor -- start 2024-04-25T01:30:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.829+0000 7fd5662da640 1 -- start start 2024-04-25T01:30:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.829+0000 7fd5662da640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd5601017b0 0x7fd560199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.829+0000 7fd5662da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5601029a0 0x7fd56019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.829+0000 7fd5662da640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd560103360 0x7fd5601a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.829+0000 7fd5662da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd56010d400 con 0x7fd5601029a0 2024-04-25T01:30:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.829+0000 7fd5662da640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd56010d280 con 0x7fd560103360 2024-04-25T01:30:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.829+0000 7fd5662da640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd56010d580 con 0x7fd5601017b0 2024-04-25T01:30:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.829+0000 7fd55f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5601029a0 0x7fd56019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.829+0000 7fd55f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5601029a0 0x7fd56019a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37964/0 (socket says 172.21.15.28:37964) 2024-04-25T01:30:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.830+0000 7fd55f7fe640 1 -- 172.21.15.28:0/2989884644 learned_addr learned my addr 172.21.15.28:0/2989884644 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.830+0000 7fd55ffff640 1 --2- 172.21.15.28:0/2989884644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd5601017b0 0x7fd560199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:40.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.830+0000 7fd564877640 1 --2- 172.21.15.28:0/2989884644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd560103360 0x7fd5601a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:40.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.830+0000 7fd55f7fe640 1 -- 172.21.15.28:0/2989884644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd5601017b0 msgr2=0x7fd560199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:40.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.830+0000 7fd55f7fe640 1 --2- 172.21.15.28:0/2989884644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd5601017b0 0x7fd560199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.830+0000 7fd55f7fe640 1 -- 172.21.15.28:0/2989884644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd560103360 msgr2=0x7fd5601a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:40.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.830+0000 7fd55f7fe640 1 --2- 172.21.15.28:0/2989884644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd560103360 0x7fd5601a1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:40.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.830+0000 7fd55f7fe640 1 -- 172.21.15.28:0/2989884644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5601a18c0 con 0x7fd5601029a0 2024-04-25T01:30:40.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.830+0000 7fd55f7fe640 1 --2- 172.21.15.28:0/2989884644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5601029a0 0x7fd56019a260 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd55400d950 tx=0x7fd55400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:40.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.830+0000 7fd55d7fa640 1 -- 172.21.15.28:0/2989884644 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd554017070 con 0x7fd5601029a0 2024-04-25T01:30:40.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.830+0000 7fd5662da640 1 -- 172.21.15.28:0/2989884644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5601a1bb0 con 0x7fd5601029a0 2024-04-25T01:30:40.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.830+0000 7fd55d7fa640 1 -- 172.21.15.28:0/2989884644 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd554009c30 con 0x7fd5601029a0 2024-04-25T01:30:40.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.830+0000 7fd55d7fa640 1 -- 172.21.15.28:0/2989884644 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55400f6b0 con 0x7fd5601029a0 2024-04-25T01:30:40.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.830+0000 7fd5662da640 1 -- 172.21.15.28:0/2989884644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5601a20d0 con 0x7fd5601029a0 2024-04-25T01:30:40.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.831+0000 7fd5662da640 1 -- 172.21.15.28:0/2989884644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5601a22a0 con 0x7fd5601029a0 2024-04-25T01:30:40.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.832+0000 7fd55d7fa640 1 -- 172.21.15.28:0/2989884644 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd5540109d0 con 0x7fd5601029a0 2024-04-25T01:30:40.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.833+0000 7fd55d7fa640 1 --2- 172.21.15.28:0/2989884644 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd5480a2b90 0x7fd5480a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.833+0000 7fd55ffff640 1 --2- 172.21.15.28:0/2989884644 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd5480a2b90 0x7fd5480a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.833+0000 7fd55d7fa640 1 -- 172.21.15.28:0/2989884644 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fd5540cee80 con 0x7fd5601029a0 2024-04-25T01:30:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.833+0000 7fd55ffff640 1 --2- 172.21.15.28:0/2989884644 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd5480a2b90 0x7fd5480a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd550005ab0 tx=0x7fd550000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:40.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:40.838+0000 7fd55d7fa640 1 -- 172.21.15.28:0/2989884644 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd554009de0 con 0x7fd5601029a0 2024-04-25T01:30:40.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.021+0000 7fd5662da640 1 -- 172.21.15.28:0/2989884644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd5601a2630 con 0x7fd5601029a0 2024-04-25T01:30:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.023+0000 7fd55d7fa640 1 -- 172.21.15.28:0/2989884644 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24093 (secure 0 0 0) 0x7fd55409b4f0 con 0x7fd5601029a0 2024-04-25T01:30:41.002 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:41.002 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:30:38.730329+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:30:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.025+0000 7fd5662da640 1 -- 172.21.15.28:0/2989884644 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd5480a2b90 msgr2=0x7fd5480a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.025+0000 7fd5662da640 1 --2- 172.21.15.28:0/2989884644 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd5480a2b90 0x7fd5480a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd550005ab0 tx=0x7fd550000bc0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.025+0000 7fd5662da640 1 -- 172.21.15.28:0/2989884644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5601029a0 msgr2=0x7fd56019a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.025+0000 7fd5662da640 1 --2- 172.21.15.28:0/2989884644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5601029a0 0x7fd56019a260 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd55400d950 tx=0x7fd55400de10 comp rx=0 tx=0).stop 2024-04-25T01:30:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.025+0000 7fd5662da640 1 -- 172.21.15.28:0/2989884644 shutdown_connections 2024-04-25T01:30:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.025+0000 7fd5662da640 1 --2- 172.21.15.28:0/2989884644 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd5480a2b90 0x7fd5480a5050 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.025+0000 7fd5662da640 1 --2- 172.21.15.28:0/2989884644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd560103360 0x7fd5601a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.025+0000 7fd5662da640 1 --2- 172.21.15.28:0/2989884644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5601029a0 0x7fd56019a260 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.025+0000 7fd5662da640 1 --2- 172.21.15.28:0/2989884644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd5601017b0 0x7fd560199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.025+0000 7fd5662da640 1 -- 172.21.15.28:0/2989884644 >> 172.21.15.28:0/2989884644 conn(0x7fd5600fcd40 msgr2=0x7fd560073970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.025+0000 7fd5662da640 1 -- 172.21.15.28:0/2989884644 shutdown_connections 2024-04-25T01:30:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.025+0000 7fd5662da640 1 -- 172.21.15.28:0/2989884644 wait complete. 2024-04-25T01:30:41.015 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T01:30:33.696870+0000', 'modified': '2024-04-25T01:30:38.730500+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24512}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24512': {'gid': 24512, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6839/2835949714', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 2835949714}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 2835949714}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24512, 'qdb_cluster': [24512]} 2024-04-25T01:30:41.015 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T01:30:41.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T01:30:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.139+0000 7f9aa8a76640 1 -- 172.21.15.28:0/1861928131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa4100680 msgr2=0x7f9aa4100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.139+0000 7f9aa8a76640 1 --2- 172.21.15.28:0/1861928131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa4100680 0x7f9aa4100aa0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009ca0 tx=0x7f9a9402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.139+0000 7f9aa8a76640 1 -- 172.21.15.28:0/1861928131 shutdown_connections 2024-04-25T01:30:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.139+0000 7f9aa8a76640 1 --2- 172.21.15.28:0/1861928131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa4102230 0x7f9aa4108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.139+0000 7f9aa8a76640 1 --2- 172.21.15.28:0/1861928131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9aa4101870 0x7f9aa4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.139+0000 7f9aa8a76640 1 --2- 172.21.15.28:0/1861928131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa4100680 0x7f9aa4100aa0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.139+0000 7f9aa8a76640 1 -- 172.21.15.28:0/1861928131 >> 172.21.15.28:0/1861928131 conn(0x7f9aa40fbbf0 msgr2=0x7f9aa40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.139+0000 7f9aa8a76640 1 -- 172.21.15.28:0/1861928131 shutdown_connections 2024-04-25T01:30:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.139+0000 7f9aa8a76640 1 -- 172.21.15.28:0/1861928131 wait complete. 2024-04-25T01:30:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.140+0000 7f9aa8a76640 1 Processor -- start 2024-04-25T01:30:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.140+0000 7f9aa8a76640 1 -- start start 2024-04-25T01:30:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.140+0000 7f9aa8a76640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa4100680 0x7f9aa4199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.140+0000 7f9aa8a76640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9aa4101870 0x7f9aa419a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.140+0000 7f9aa8a76640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa4102230 0x7f9aa41a1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.140+0000 7f9aa8a76640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa410c3d0 con 0x7f9aa4102230 2024-04-25T01:30:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.140+0000 7f9aa8a76640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa410c250 con 0x7f9aa4100680 2024-04-25T01:30:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.140+0000 7f9aa8a76640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa410c550 con 0x7f9aa4101870 2024-04-25T01:30:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.140+0000 7f9aa2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa4102230 0x7f9aa41a1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.140+0000 7f9aa2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa4102230 0x7f9aa41a1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37992/0 (socket says 172.21.15.28:37992) 2024-04-25T01:30:41.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.140+0000 7f9aa2d9d640 1 -- 172.21.15.28:0/3329081601 learned_addr learned my addr 172.21.15.28:0/3329081601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:41.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.140+0000 7f9aa259c640 1 --2- 172.21.15.28:0/3329081601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa4100680 0x7f9aa4199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:41.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.141+0000 7f9aa1d9b640 1 --2- 172.21.15.28:0/3329081601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9aa4101870 0x7f9aa419a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:41.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.141+0000 7f9aa2d9d640 1 -- 172.21.15.28:0/3329081601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9aa4101870 msgr2=0x7f9aa419a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:41.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.141+0000 7f9aa2d9d640 1 --2- 172.21.15.28:0/3329081601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9aa4101870 0x7f9aa419a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.141+0000 7f9aa2d9d640 1 -- 172.21.15.28:0/3329081601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa4100680 msgr2=0x7f9aa4199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:41.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.141+0000 7f9aa2d9d640 1 --2- 172.21.15.28:0/3329081601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa4100680 0x7f9aa4199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.141+0000 7f9aa2d9d640 1 -- 172.21.15.28:0/3329081601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa41a1a90 con 0x7f9aa4102230 2024-04-25T01:30:41.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.141+0000 7f9aa2d9d640 1 --2- 172.21.15.28:0/3329081601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa4102230 0x7f9aa41a1420 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c009fd0 tx=0x7f9a9c00b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:41.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.141+0000 7f9a937fe640 1 -- 172.21.15.28:0/3329081601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c015020 con 0x7f9aa4102230 2024-04-25T01:30:41.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.141+0000 7f9a937fe640 1 -- 172.21.15.28:0/3329081601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a9c011d10 con 0x7f9aa4102230 2024-04-25T01:30:41.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.141+0000 7f9aa8a76640 1 -- 172.21.15.28:0/3329081601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa41a1d80 con 0x7f9aa4102230 2024-04-25T01:30:41.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.141+0000 7f9a937fe640 1 -- 172.21.15.28:0/3329081601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c00f2e0 con 0x7f9aa4102230 2024-04-25T01:30:41.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.141+0000 7f9aa8a76640 1 -- 172.21.15.28:0/3329081601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa41a22f0 con 0x7f9aa4102230 2024-04-25T01:30:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.143+0000 7f9a937fe640 1 -- 172.21.15.28:0/3329081601 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9a9c018020 con 0x7f9aa4102230 2024-04-25T01:30:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.144+0000 7f9aab784740 1 Processor -- start 2024-04-25T01:30:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.144+0000 7f9aab784740 1 -- start start 2024-04-25T01:30:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.144+0000 7f9aab784740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74e7440 0x55f0d74e7860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.144+0000 7f9aab784740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7e30 0x55f0d74ec930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.144+0000 7f9aab784740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f0d74ecfa0 0x55f0d74ef430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.144+0000 7f9aab784740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f0d744af00 con 0x55f0d74e7e30 2024-04-25T01:30:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.144+0000 7f9aab784740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f0d73f3750 con 0x55f0d74ecfa0 2024-04-25T01:30:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.144+0000 7f9aab784740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55f0d7446ec0 con 0x55f0d74e7440 2024-04-25T01:30:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.144+0000 7f9a937fe640 1 --2- 172.21.15.28:0/3329081601 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9a8c0a2c10 0x7f9a8c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.144+0000 7f9aa259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7e30 0x55f0d74ec930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.145+0000 7f9aa259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7e30 0x55f0d74ec930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38002/0 (socket says 172.21.15.28:38002) 2024-04-25T01:30:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.145+0000 7f9aa1d9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74e7440 0x55f0d74e7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.145+0000 7f9aa259c640 1 -- 172.21.15.28:0/2953200111 learned_addr learned my addr 172.21.15.28:0/2953200111 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.145+0000 7f9aa1d9b640 1 --2- 172.21.15.28:0/2953200111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f0d74ecfa0 0x55f0d74ef430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.145+0000 7f9aa2d9d640 1 --2- 172.21.15.28:0/3329081601 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9a8c0a2c10 0x7f9a8c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.145+0000 7f9a937fe640 1 -- 172.21.15.28:0/3329081601 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f9a9c0ceab0 con 0x7f9aa4102230 2024-04-25T01:30:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.145+0000 7f9aa1d9b640 1 -- 172.21.15.28:0/2953200111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f0d74ecfa0 msgr2=0x55f0d74ef430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.145+0000 7f9aa1d9b640 1 --2- 172.21.15.28:0/2953200111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f0d74ecfa0 0x55f0d74ef430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.145+0000 7f9aa1d9b640 1 -- 172.21.15.28:0/2953200111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7e30 msgr2=0x55f0d74ec930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.145+0000 7f9aa1d9b640 1 --2- 172.21.15.28:0/2953200111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7e30 0x55f0d74ec930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.145+0000 7f9aa1d9b640 1 -- 172.21.15.28:0/2953200111 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0d715f2d0 con 0x55f0d74e7440 2024-04-25T01:30:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.145+0000 7f9aa259c640 1 --2- 172.21.15.28:0/2953200111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7e30 0x55f0d74ec930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.145+0000 7f9aa2d9d640 1 --2- 172.21.15.28:0/3329081601 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9a8c0a2c10 0x7f9a8c0a50d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c097a40 tx=0x7f9a9c065c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.145+0000 7f9aa1d9b640 1 --2- 172.21.15.28:0/2953200111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74e7440 0x55f0d74e7860 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009f90 tx=0x7f9a98009470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bea9fa55983f2fa0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.145+0000 7f9a90ff9640 1 -- 172.21.15.28:0/2953200111 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a98017e60 con 0x55f0d74e7440 2024-04-25T01:30:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.146+0000 7f9a90ff9640 1 -- 172.21.15.28:0/2953200111 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a9801d070 con 0x55f0d74e7440 2024-04-25T01:30:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.146+0000 7f9a90ff9640 1 -- 172.21.15.28:0/2953200111 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a98018a20 con 0x55f0d74e7440 2024-04-25T01:30:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.146+0000 7f9aab784740 1 -- 172.21.15.28:0/2953200111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74e7440 msgr2=0x55f0d74e7860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.146+0000 7f9aab784740 1 --2- 172.21.15.28:0/2953200111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74e7440 0x55f0d74e7860 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009f90 tx=0x7f9a98009470 comp rx=0 tx=0).stop 2024-04-25T01:30:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.146+0000 7f9aab784740 1 -- 172.21.15.28:0/2953200111 shutdown_connections 2024-04-25T01:30:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.146+0000 7f9aab784740 1 --2- 172.21.15.28:0/2953200111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f0d74ecfa0 0x55f0d74ef430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.146+0000 7f9aab784740 1 --2- 172.21.15.28:0/2953200111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7e30 0x55f0d74ec930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.146+0000 7f9aab784740 1 --2- 172.21.15.28:0/2953200111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74e7440 0x55f0d74e7860 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.146+0000 7f9aab784740 1 -- 172.21.15.28:0/2953200111 >> 172.21.15.28:0/2953200111 conn(0x55f0d74e2ab0 msgr2=0x55f0d74e2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.146+0000 7f9aab784740 1 -- 172.21.15.28:0/2953200111 shutdown_connections 2024-04-25T01:30:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.146+0000 7f9aab784740 1 -- 172.21.15.28:0/2953200111 wait complete. 2024-04-25T01:30:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.147+0000 7f9aab784740 1 -- start start 2024-04-25T01:30:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.147+0000 7f9aab784740 1 Processor -- start 2024-04-25T01:30:41.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.147+0000 7f9aab784740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7440 0x55f0d74f1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:41.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.147+0000 7f9aab784740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74e7e30 0x55f0d74f1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:41.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.147+0000 7f9aab784740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f0d74ecfa0 0x55f0d74f8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:41.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.147+0000 7f9aab784740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f0d74efef0 con 0x55f0d74e7440 2024-04-25T01:30:41.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.147+0000 7f9aab784740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f0d7406cf0 con 0x55f0d74ecfa0 2024-04-25T01:30:41.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.147+0000 7f9aab784740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55f0d74f0070 con 0x55f0d74e7e30 2024-04-25T01:30:41.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.147+0000 7f9aab784740 10 client.0 _ll_register_callbacks cb 0x55f0d74066e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:41.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.147+0000 7f9aa1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7440 0x55f0d74f1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:41.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.147+0000 7f9aa1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7440 0x55f0d74f1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38018/0 (socket says 172.21.15.28:38018) 2024-04-25T01:30:41.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.147+0000 7f9aa1d9b640 1 -- 172.21.15.28:0/3961959067 learned_addr learned my addr 172.21.15.28:0/3961959067 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:41.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.147+0000 7f9aa259c640 1 --2- 172.21.15.28:0/3961959067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74e7e30 0x55f0d74f1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:41.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.147+0000 7f9aa1d9b640 1 -- 172.21.15.28:0/3961959067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74e7e30 msgr2=0x55f0d74f1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:41.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.147+0000 7f9aa1d9b640 1 --2- 172.21.15.28:0/3961959067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74e7e30 0x55f0d74f1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.147+0000 7f9aa1d9b640 1 -- 172.21.15.28:0/3961959067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f0d74ecfa0 msgr2=0x55f0d74f8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:41.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.147+0000 7f9aa1d9b640 1 --2- 172.21.15.28:0/3961959067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f0d74ecfa0 0x55f0d74f8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.147+0000 7f9aa1d9b640 1 -- 172.21.15.28:0/3961959067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0d72296b0 con 0x55f0d74e7440 2024-04-25T01:30:41.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.148+0000 7f9aa1d9b640 1 --2- 172.21.15.28:0/3961959067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7440 0x55f0d74f1830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9a98006b90 tx=0x7f9a98009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d79eec92a85276 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:41.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.148+0000 7f9a89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:30:41.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.148+0000 7f9a89ffb640 1 -- 172.21.15.28:0/3961959067 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a98026020 con 0x55f0d74e7440 2024-04-25T01:30:41.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.148+0000 7f9a89ffb640 1 -- 172.21.15.28:0/3961959067 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f9a980187a0 con 0x55f0d74e7440 2024-04-25T01:30:41.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.148+0000 7f9aab784740 1 -- 172.21.15.28:0/3961959067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a9801d070 con 0x55f0d74e7440 2024-04-25T01:30:41.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.148+0000 7f9a89ffb640 1 -- 172.21.15.28:0/3961959067 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f9a98031050 con 0x55f0d74e7440 2024-04-25T01:30:41.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.148+0000 7f9aab784740 10 client.14781 fetch_fsmap learned FSMap version 17 2024-04-25T01:30:41.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.148+0000 7f9aab784740 1 -- 172.21.15.28:0/3961959067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55f0d74fbd90 con 0x55f0d74e7440 2024-04-25T01:30:41.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.148+0000 7f9a89ffb640 1 -- 172.21.15.28:0/3961959067 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a9801d070 con 0x55f0d74e7440 2024-04-25T01:30:41.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.148+0000 7f9a89ffb640 1 -- 172.21.15.28:0/3961959067 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f9a98022710 con 0x55f0d74e7440 2024-04-25T01:30:41.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.148+0000 7f9aab784740 10 client.14781 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T01:30:41.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.148+0000 7f9aab784740 10 client.14781 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:30:41.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.148+0000 7f9aab784740 20 client.14781 populate_metadata read hostname 'smithi028' 2024-04-25T01:30:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.148+0000 7f9aab784740 1 --2- 172.21.15.28:0/3961959067 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f0d74fdf50 0x55f0d74ff3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.148+0000 7f9aab784740 4 client.14781 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:30:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.148+0000 7f9aab784740 1 -- 172.21.15.28:0/3961959067 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0d7458440 con 0x55f0d74fdf50 2024-04-25T01:30:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.149+0000 7f9aa259c640 1 --2- 172.21.15.28:0/3961959067 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f0d74fdf50 0x55f0d74ff3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.149+0000 7f9a89ffb640 10 client.14781 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:30:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.149+0000 7f9aa259c640 1 --2- 172.21.15.28:0/3961959067 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f0d74fdf50 0x55f0d74ff3c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddff7c8d6899dd9b server_cookie=8f52c190c6fb8579 in_seq=0 out_seq=0 2024-04-25T01:30:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.150+0000 7f9a89ffb640 1 -- 172.21.15.28:0/3961959067 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f0d7458440 con 0x55f0d74fdf50 2024-04-25T01:30:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.150+0000 7f9a89ffb640 10 client.14781 handle_command_reply: tid=0 2024-04-25T01:30:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.150+0000 7f9aab784740 1 client.14781 shutdown 2024-04-25T01:30:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.150+0000 7f9aab784740 1 -- 172.21.15.28:0/3961959067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7440 msgr2=0x55f0d74f1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.150+0000 7f9aab784740 1 --2- 172.21.15.28:0/3961959067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7440 0x55f0d74f1830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9a98006b90 tx=0x7f9a98009a50 comp rx=0 tx=0).stop 2024-04-25T01:30:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.150+0000 7f9aab784740 1 -- 172.21.15.28:0/3961959067 shutdown_connections 2024-04-25T01:30:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.150+0000 7f9aab784740 1 --2- 172.21.15.28:0/3961959067 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f0d74fdf50 0x55f0d74ff3c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.150+0000 7f9aab784740 1 --2- 172.21.15.28:0/3961959067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f0d74ecfa0 0x55f0d74f8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.150+0000 7f9aab784740 1 --2- 172.21.15.28:0/3961959067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74e7e30 0x55f0d74f1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.150+0000 7f9a89ffb640 0 client.14781 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:30:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.150+0000 7f9aab784740 1 --2- 172.21.15.28:0/3961959067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7440 0x55f0d74f1830 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.150+0000 7f9aab784740 1 -- 172.21.15.28:0/3961959067 >> 172.21.15.28:0/3961959067 conn(0x55f0d74e2ab0 msgr2=0x55f0d74e1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.150+0000 7f9aab784740 1 -- 172.21.15.28:0/3961959067 shutdown_connections 2024-04-25T01:30:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.150+0000 7f9aab784740 1 -- 172.21.15.28:0/3961959067 wait complete. 2024-04-25T01:30:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.150+0000 7f9aab784740 20 client.14781 trim_cache size 0 max 16384 2024-04-25T01:30:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.168+0000 7f9aab784740 1 Processor -- start 2024-04-25T01:30:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.168+0000 7f9aab784740 1 -- start start 2024-04-25T01:30:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.168+0000 7f9aab784740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f0d74e7440 0x55f0d74f9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.168+0000 7f9aab784740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74e7e30 0x55f0d74e88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.168+0000 7f9aab784740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74ecfa0 0x55f0d74e2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.168+0000 7f9aab784740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f0d74fd700 con 0x55f0d74ecfa0 2024-04-25T01:30:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.168+0000 7f9aab784740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f0d74fddd0 con 0x55f0d74e7440 2024-04-25T01:30:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.168+0000 7f9aab784740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55f0d74fe4a0 con 0x55f0d74e7e30 2024-04-25T01:30:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.168+0000 7f9aa1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74ecfa0 0x55f0d74e2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:41.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.168+0000 7f9aa1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74ecfa0 0x55f0d74e2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38026/0 (socket says 172.21.15.28:38026) 2024-04-25T01:30:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.168+0000 7f9aa1d9b640 1 -- 172.21.15.28:0/2265232286 learned_addr learned my addr 172.21.15.28:0/2265232286 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.169+0000 7f9aa1d9b640 1 --2- 172.21.15.28:0/2265232286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f0d74e7440 0x55f0d74f9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.169+0000 7f9aa1d9b640 1 -- 172.21.15.28:0/2265232286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74e7e30 msgr2=0x55f0d74e88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.169+0000 7f9aa1d9b640 1 --2- 172.21.15.28:0/2265232286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74e7e30 0x55f0d74e88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.169+0000 7f9aa1d9b640 1 -- 172.21.15.28:0/2265232286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f0d74e7440 msgr2=0x55f0d74f9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.169+0000 7f9aa1d9b640 1 --2- 172.21.15.28:0/2265232286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f0d74e7440 0x55f0d74f9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.169+0000 7f9aa1d9b640 1 -- 172.21.15.28:0/2265232286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0d74f2820 con 0x55f0d74ecfa0 2024-04-25T01:30:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.169+0000 7f9aa1d9b640 1 --2- 172.21.15.28:0/2265232286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74ecfa0 0x55f0d74e2e90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009840 tx=0x7f9a98017500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffbdeb7a1c699960 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.169+0000 7f9a8bfff640 1 -- 172.21.15.28:0/2265232286 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a98026070 con 0x55f0d74ecfa0 2024-04-25T01:30:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.169+0000 7f9a8bfff640 1 -- 172.21.15.28:0/2265232286 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a9801d040 con 0x55f0d74ecfa0 2024-04-25T01:30:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.169+0000 7f9a8bfff640 1 -- 172.21.15.28:0/2265232286 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a98018b70 con 0x55f0d74ecfa0 2024-04-25T01:30:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.169+0000 7f9aab784740 1 -- 172.21.15.28:0/2265232286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74ecfa0 msgr2=0x55f0d74e2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:41.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.169+0000 7f9aab784740 1 --2- 172.21.15.28:0/2265232286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74ecfa0 0x55f0d74e2e90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9a98009840 tx=0x7f9a98017500 comp rx=0 tx=0).stop 2024-04-25T01:30:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aab784740 1 -- 172.21.15.28:0/2265232286 shutdown_connections 2024-04-25T01:30:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aab784740 1 --2- 172.21.15.28:0/2265232286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74ecfa0 0x55f0d74e2e90 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aab784740 1 --2- 172.21.15.28:0/2265232286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74e7e30 0x55f0d74e88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aab784740 1 --2- 172.21.15.28:0/2265232286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f0d74e7440 0x55f0d74f9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aab784740 1 -- 172.21.15.28:0/2265232286 >> 172.21.15.28:0/2265232286 conn(0x55f0d74e2ab0 msgr2=0x55f0d74ec890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aab784740 1 -- 172.21.15.28:0/2265232286 shutdown_connections 2024-04-25T01:30:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aab784740 1 -- 172.21.15.28:0/2265232286 wait complete. 2024-04-25T01:30:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aab784740 1 -- start start 2024-04-25T01:30:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aab784740 1 Processor -- start 2024-04-25T01:30:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aab784740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f0d74e7440 0x55f0d74eb250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aab784740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7e30 0x55f0d74eb790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aab784740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74ecfa0 0x55f0d74f7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aab784740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f0d74ff240 con 0x55f0d74e7e30 2024-04-25T01:30:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aab784740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f0d74feb70 con 0x55f0d74e7440 2024-04-25T01:30:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aab784740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55f0d74ff910 con 0x55f0d74ecfa0 2024-04-25T01:30:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aab784740 10 client.0 _ll_register_callbacks cb 0x7f9a98017600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:30:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aa259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7e30 0x55f0d74eb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aa259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7e30 0x55f0d74eb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38036/0 (socket says 172.21.15.28:38036) 2024-04-25T01:30:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aa259c640 1 -- 172.21.15.28:0/2239408173 learned_addr learned my addr 172.21.15.28:0/2239408173 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aa1d9b640 1 --2- 172.21.15.28:0/2239408173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f0d74e7440 0x55f0d74eb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aa259c640 1 -- 172.21.15.28:0/2239408173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74ecfa0 msgr2=0x55f0d74f7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aa1d9b640 1 --2- 172.21.15.28:0/2239408173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74ecfa0 0x55f0d74f7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aa259c640 1 --2- 172.21.15.28:0/2239408173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74ecfa0 0x55f0d74f7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aa259c640 1 -- 172.21.15.28:0/2239408173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f0d74e7440 msgr2=0x55f0d74eb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:30:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aa259c640 1 --2- 172.21.15.28:0/2239408173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f0d74e7440 0x55f0d74eb250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.170+0000 7f9aa259c640 1 -- 172.21.15.28:0/2239408173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0d74f7820 con 0x55f0d74e7e30 2024-04-25T01:30:41.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.171+0000 7f9aa1d9b640 1 --2- 172.21.15.28:0/2239408173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74ecfa0 0x55f0d74f7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:41.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.171+0000 7f9aa259c640 1 --2- 172.21.15.28:0/2239408173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7e30 0x55f0d74eb790 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009bf0 tx=0x7f9a9402f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b8da1d570d8fdd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:41.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.171+0000 7f9a89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:30:41.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.171+0000 7f9a89ffb640 1 -- 172.21.15.28:0/2239408173 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a94050070 con 0x55f0d74e7e30 2024-04-25T01:30:41.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.171+0000 7f9aab784740 1 -- 172.21.15.28:0/2239408173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a9801d040 con 0x55f0d74e7e30 2024-04-25T01:30:41.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.171+0000 7f9a89ffb640 1 -- 172.21.15.28:0/2239408173 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f9a94031860 con 0x55f0d74e7e30 2024-04-25T01:30:41.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.171+0000 7f9a89ffb640 1 -- 172.21.15.28:0/2239408173 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f9a9405b050 con 0x55f0d74e7e30 2024-04-25T01:30:41.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.171+0000 7f9aab784740 10 client.14787 fetch_fsmap learned FSMap version 17 2024-04-25T01:30:41.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.171+0000 7f9aab784740 1 -- 172.21.15.28:0/2239408173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55f0d7564270 con 0x55f0d74e7e30 2024-04-25T01:30:41.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.171+0000 7f9a89ffb640 1 -- 172.21.15.28:0/2239408173 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a9801d040 con 0x55f0d74e7e30 2024-04-25T01:30:41.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.171+0000 7f9a89ffb640 1 -- 172.21.15.28:0/2239408173 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f9a9402fa50 con 0x55f0d74e7e30 2024-04-25T01:30:41.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.171+0000 7f9aab784740 10 client.14787 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T01:30:41.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.171+0000 7f9aab784740 10 client.14787 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:30:41.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.171+0000 7f9aab784740 20 client.14787 populate_metadata read hostname 'smithi028' 2024-04-25T01:30:41.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.171+0000 7f9aab784740 1 --2- 172.21.15.28:0/2239408173 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f0d7568e80 0x55f0d7567b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:41.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.171+0000 7f9aab784740 4 client.14787 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:30:41.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.171+0000 7f9aab784740 1 -- 172.21.15.28:0/2239408173 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "status"}) -- 0x55f0d7452790 con 0x55f0d7568e80 2024-04-25T01:30:41.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.172+0000 7f9aa1d9b640 1 --2- 172.21.15.28:0/2239408173 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f0d7568e80 0x55f0d7567b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:41.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.172+0000 7f9a89ffb640 10 client.14787 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:30:41.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.172+0000 7f9aa1d9b640 1 --2- 172.21.15.28:0/2239408173 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f0d7568e80 0x55f0d7567b90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bf40aff362cad27 server_cookie=94a917e7001e81ce in_seq=0 out_seq=0 2024-04-25T01:30:41.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.172+0000 7f9a89ffb640 1 -- 172.21.15.28:0/2239408173 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55f0d7452790 con 0x55f0d7568e80 2024-04-25T01:30:41.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.172+0000 7f9a89ffb640 10 client.14787 handle_command_reply: tid=0 2024-04-25T01:30:41.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.172+0000 7f9aab784740 1 client.14787 shutdown 2024-04-25T01:30:41.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.172+0000 7f9aab784740 1 -- 172.21.15.28:0/2239408173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7e30 msgr2=0x55f0d74eb790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:41.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.172+0000 7f9aab784740 1 --2- 172.21.15.28:0/2239408173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7e30 0x55f0d74eb790 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009bf0 tx=0x7f9a9402f310 comp rx=0 tx=0).stop 2024-04-25T01:30:41.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.172+0000 7f9aab784740 1 -- 172.21.15.28:0/2239408173 shutdown_connections 2024-04-25T01:30:41.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.172+0000 7f9aab784740 1 --2- 172.21.15.28:0/2239408173 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f0d7568e80 0x55f0d7567b90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.172+0000 7f9aab784740 1 --2- 172.21.15.28:0/2239408173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f0d74ecfa0 0x55f0d74f7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.172+0000 7f9a89ffb640 0 client.14787 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:30:41.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.172+0000 7f9aab784740 1 --2- 172.21.15.28:0/2239408173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f0d74e7e30 0x55f0d74eb790 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.172+0000 7f9aab784740 1 --2- 172.21.15.28:0/2239408173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f0d74e7440 0x55f0d74eb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.172+0000 7f9aab784740 1 -- 172.21.15.28:0/2239408173 >> 172.21.15.28:0/2239408173 conn(0x55f0d74e2ab0 msgr2=0x55f0d74dd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:41.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.173+0000 7f9aab784740 1 -- 172.21.15.28:0/2239408173 shutdown_connections 2024-04-25T01:30:41.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.173+0000 7f9aab784740 1 -- 172.21.15.28:0/2239408173 wait complete. 2024-04-25T01:30:41.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.173+0000 7f9aab784740 20 client.14787 trim_cache size 0 max 16384 2024-04-25T01:30:41.159 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:30:41.159 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "0e88f60a-02a2-11ef-bc93-c7b262605968", 2024-04-25T01:30:41.159 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-04-25T01:30:41.160 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24512, 2024-04-25T01:30:41.160 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-25T01:30:41.160 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-25T01:30:41.160 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-25T01:30:41.160 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.8401292659999999, 2024-04-25T01:30:41.160 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 17, 2024-04-25T01:30:41.160 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 91, 2024-04-25T01:30:41.160 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 91, 2024-04-25T01:30:41.160 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 13.061467345000001 2024-04-25T01:30:41.160 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:30:41.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.173+0000 7f9aa8a76640 1 -- 172.21.15.28:0/3329081601 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9a8c0a2c10 msgr2=0x7f9a8c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:41.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.173+0000 7f9aa8a76640 1 --2- 172.21.15.28:0/3329081601 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9a8c0a2c10 0x7f9a8c0a50d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c097a40 tx=0x7f9a9c065c40 comp rx=0 tx=0).stop 2024-04-25T01:30:41.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.173+0000 7f9aa8a76640 1 -- 172.21.15.28:0/3329081601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa4102230 msgr2=0x7f9aa41a1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:41.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.173+0000 7f9aa8a76640 1 --2- 172.21.15.28:0/3329081601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa4102230 0x7f9aa41a1420 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c009fd0 tx=0x7f9a9c00b820 comp rx=0 tx=0).stop 2024-04-25T01:30:41.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.173+0000 7f9aa8a76640 1 -- 172.21.15.28:0/3329081601 shutdown_connections 2024-04-25T01:30:41.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.173+0000 7f9aa8a76640 1 --2- 172.21.15.28:0/3329081601 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9a8c0a2c10 0x7f9a8c0a50d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.173+0000 7f9aa8a76640 1 --2- 172.21.15.28:0/3329081601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9aa4102230 0x7f9aa41a1420 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.173+0000 7f9aa8a76640 1 --2- 172.21.15.28:0/3329081601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9aa4101870 0x7f9aa419a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.173+0000 7f9aa8a76640 1 --2- 172.21.15.28:0/3329081601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9aa4100680 0x7f9aa4199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:41.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.173+0000 7f9aa8a76640 1 -- 172.21.15.28:0/3329081601 >> 172.21.15.28:0/3329081601 conn(0x7f9aa40fbbf0 msgr2=0x7f9aa4068920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:41.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.173+0000 7f9aa8a76640 1 -- 172.21.15.28:0/3329081601 shutdown_connections 2024-04-25T01:30:41.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:41.173+0000 7f9aa8a76640 1 -- 172.21.15.28:0/3329081601 wait complete. 2024-04-25T01:30:41.163 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T01:30:41.163 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:30:41.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-25T01:30:41.407 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:30:41.625 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:41 smithi130 ceph-mon[32702]: pgmap v301: 129 pgs: 8 creating+peering, 116 active+clean, 5 unknown; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2024-04-25T01:30:41.626 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:41 smithi130 ceph-mon[32702]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:30:41.626 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:41 smithi130 ceph-mon[32702]: Cluster is now healthy 2024-04-25T01:30:41.626 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:41 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3269608519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:41.626 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:41 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2989884644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:41.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:41 smithi028 ceph-mon[29390]: pgmap v301: 129 pgs: 8 creating+peering, 116 active+clean, 5 unknown; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2024-04-25T01:30:41.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:41 smithi028 ceph-mon[29390]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:30:41.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:41 smithi028 ceph-mon[29390]: Cluster is now healthy 2024-04-25T01:30:41.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:41 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3269608519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:41.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:41 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2989884644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:41.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:41 smithi032 ceph-mon[32461]: pgmap v301: 129 pgs: 8 creating+peering, 116 active+clean, 5 unknown; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2024-04-25T01:30:41.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:41 smithi032 ceph-mon[32461]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:30:41.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:41 smithi032 ceph-mon[32461]: Cluster is now healthy 2024-04-25T01:30:41.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:41 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3269608519' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:30:41.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:41 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2989884644' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:43.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.186+0000 7f9da49f8700 1 -- 172.21.15.28:0/620837250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da0074d80 msgr2=0x7f9da010e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:43.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.186+0000 7f9da49f8700 1 --2- 172.21.15.28:0/620837250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da0074d80 0x7f9da010e6f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9d94009990 tx=0x7f9d94009ca0 comp rx=0 tx=0).stop 2024-04-25T01:30:43.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.186+0000 7f9da49f8700 1 -- 172.21.15.28:0/620837250 shutdown_connections 2024-04-25T01:30:43.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.186+0000 7f9da49f8700 1 --2- 172.21.15.28:0/620837250 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9da010ec30 0x7f9da0111020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:43.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.186+0000 7f9da49f8700 1 --2- 172.21.15.28:0/620837250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da0074460 0x7f9da0074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:43.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.186+0000 7f9da49f8700 1 --2- 172.21.15.28:0/620837250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da0074d80 0x7f9da010e6f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:43.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.186+0000 7f9da49f8700 1 -- 172.21.15.28:0/620837250 >> 172.21.15.28:0/620837250 conn(0x7f9da0079390 msgr2=0x7f9da00797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:43.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.186+0000 7f9da49f8700 1 -- 172.21.15.28:0/620837250 shutdown_connections 2024-04-25T01:30:43.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.186+0000 7f9da49f8700 1 -- 172.21.15.28:0/620837250 wait complete. 2024-04-25T01:30:43.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.187+0000 7f9da49f8700 1 Processor -- start 2024-04-25T01:30:43.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.187+0000 7f9da49f8700 1 -- start start 2024-04-25T01:30:43.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.187+0000 7f9da49f8700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9da0074460 0x7f9da01a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:43.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.187+0000 7f9da49f8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da0074d80 0x7f9da01a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:43.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.187+0000 7f9da49f8700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da010ec30 0x7f9da019a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:43.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.187+0000 7f9da49f8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9da0114720 con 0x7f9da0074d80 2024-04-25T01:30:43.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.187+0000 7f9da49f8700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9da01145a0 con 0x7f9da010ec30 2024-04-25T01:30:43.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.187+0000 7f9da49f8700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9da01148a0 con 0x7f9da0074460 2024-04-25T01:30:43.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.187+0000 7f9d9dd9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da0074d80 0x7f9da01a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:43.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.187+0000 7f9d9dd9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da0074d80 0x7f9da01a0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38068/0 (socket says 172.21.15.28:38068) 2024-04-25T01:30:43.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.187+0000 7f9d9dd9b700 1 -- 172.21.15.28:0/1633966996 learned_addr learned my addr 172.21.15.28:0/1633966996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:43.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.187+0000 7f9d9ed9d700 1 --2- 172.21.15.28:0/1633966996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da010ec30 0x7f9da019a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:43.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.187+0000 7f9d9e59c700 1 --2- 172.21.15.28:0/1633966996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9da0074460 0x7f9da01a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:43.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.188+0000 7f9d9dd9b700 1 -- 172.21.15.28:0/1633966996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9da0074460 msgr2=0x7f9da01a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:43.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.188+0000 7f9d9dd9b700 1 --2- 172.21.15.28:0/1633966996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9da0074460 0x7f9da01a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:43.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.188+0000 7f9d9dd9b700 1 -- 172.21.15.28:0/1633966996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da010ec30 msgr2=0x7f9da019a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:43.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.188+0000 7f9d9dd9b700 1 --2- 172.21.15.28:0/1633966996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da010ec30 0x7f9da019a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:43.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.188+0000 7f9d9dd9b700 1 -- 172.21.15.28:0/1633966996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9da019b140 con 0x7f9da0074d80 2024-04-25T01:30:43.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.188+0000 7f9d9e59c700 1 --2- 172.21.15.28:0/1633966996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9da0074460 0x7f9da01a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:43.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.188+0000 7f9d9dd9b700 1 --2- 172.21.15.28:0/1633966996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da0074d80 0x7f9da01a0d60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9d940038b0 tx=0x7f9d94009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:43.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.188+0000 7f9d8f7fe700 1 -- 172.21.15.28:0/1633966996 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d94027070 con 0x7f9da0074d80 2024-04-25T01:30:43.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.188+0000 7f9d8f7fe700 1 -- 172.21.15.28:0/1633966996 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d94004050 con 0x7f9da0074d80 2024-04-25T01:30:43.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.188+0000 7f9da49f8700 1 -- 172.21.15.28:0/1633966996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9da019b3d0 con 0x7f9da0074d80 2024-04-25T01:30:43.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.188+0000 7f9d8f7fe700 1 -- 172.21.15.28:0/1633966996 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d94022cd0 con 0x7f9da0074d80 2024-04-25T01:30:43.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.188+0000 7f9da49f8700 1 -- 172.21.15.28:0/1633966996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9da01a76f0 con 0x7f9da0074d80 2024-04-25T01:30:43.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.189+0000 7f9da49f8700 1 -- 172.21.15.28:0/1633966996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9da0068f20 con 0x7f9da0074d80 2024-04-25T01:30:43.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.190+0000 7f9d8f7fe700 1 -- 172.21.15.28:0/1633966996 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9d9402a070 con 0x7f9da0074d80 2024-04-25T01:30:43.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.191+0000 7f9d8f7fe700 1 --2- 172.21.15.28:0/1633966996 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9d880a2c20 0x7f9d880a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:43.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.191+0000 7f9d9e59c700 1 --2- 172.21.15.28:0/1633966996 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9d880a2c20 0x7f9d880a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:43.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.191+0000 7f9d8f7fe700 1 -- 172.21.15.28:0/1633966996 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f9d940d03c0 con 0x7f9da0074d80 2024-04-25T01:30:43.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.191+0000 7f9d9e59c700 1 --2- 172.21.15.28:0/1633966996 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9d880a2c20 0x7f9d880a50e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9d90005fd0 tx=0x7f9d90005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:43.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.195+0000 7f9d8f7fe700 1 -- 172.21.15.28:0/1633966996 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9d94013070 con 0x7f9da0074d80 2024-04-25T01:30:43.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.451+0000 7f9da49f8700 1 -- 172.21.15.28:0/1633966996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9da002db30 con 0x7f9da0074d80 2024-04-25T01:30:43.430 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:43 smithi028 ceph-mon[29390]: pgmap v302: 129 pgs: 8 creating+peering, 121 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-04-25T01:30:43.430 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T01:30:43.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.452+0000 7f9d8f7fe700 1 -- 172.21.15.28:0/1633966996 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f9d9409d170 con 0x7f9da0074d80 2024-04-25T01:30:43.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.454+0000 7f9da49f8700 1 -- 172.21.15.28:0/1633966996 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9d880a2c20 msgr2=0x7f9d880a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.454+0000 7f9da49f8700 1 --2- 172.21.15.28:0/1633966996 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9d880a2c20 0x7f9d880a50e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9d90005fd0 tx=0x7f9d90005f00 comp rx=0 tx=0).stop 2024-04-25T01:30:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.454+0000 7f9da49f8700 1 -- 172.21.15.28:0/1633966996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da0074d80 msgr2=0x7f9da01a0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.454+0000 7f9da49f8700 1 --2- 172.21.15.28:0/1633966996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da0074d80 0x7f9da01a0d60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9d940038b0 tx=0x7f9d94009f50 comp rx=0 tx=0).stop 2024-04-25T01:30:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.454+0000 7f9da49f8700 1 -- 172.21.15.28:0/1633966996 shutdown_connections 2024-04-25T01:30:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.454+0000 7f9da49f8700 1 --2- 172.21.15.28:0/1633966996 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9d880a2c20 0x7f9d880a50e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.454+0000 7f9da49f8700 1 --2- 172.21.15.28:0/1633966996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9da010ec30 0x7f9da019a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.454+0000 7f9da49f8700 1 --2- 172.21.15.28:0/1633966996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9da0074460 0x7f9da01a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.454+0000 7f9da49f8700 1 --2- 172.21.15.28:0/1633966996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da0074d80 0x7f9da01a0d60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.454+0000 7f9da49f8700 1 -- 172.21.15.28:0/1633966996 >> 172.21.15.28:0/1633966996 conn(0x7f9da0079390 msgr2=0x7f9da00ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.455+0000 7f9da49f8700 1 -- 172.21.15.28:0/1633966996 shutdown_connections 2024-04-25T01:30:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:43.455+0000 7f9da49f8700 1 -- 172.21.15.28:0/1633966996 wait complete. 2024-04-25T01:30:43.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:43 smithi130 ceph-mon[32702]: pgmap v302: 129 pgs: 8 creating+peering, 121 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-04-25T01:30:43.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:43 smithi032 ceph-mon[32461]: pgmap v302: 129 pgs: 8 creating+peering, 121 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-04-25T01:30:44.081 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T01:30:44.081 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:30:44.081 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-04-25T01:30:44.081 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-25T01:30:44.081 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:30:44.081 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T01:30:44.081 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T01:30:44.081 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:30:44.081 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T01:30:44.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:30:44.082 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-04-25T01:30:44.099 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:30:44.099 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:30:44.099 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:30:44.099 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:30:44.099 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-04-25T01:30:44.099 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:30:44.099 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:30:44.099 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:30:44.099 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-04-25T01:30:44.099 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:30:44.099 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:30:44.099 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-04-25T01:30:44.099 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:30:44.100 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-04-25T01:30:44.100 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-04-25T01:30:44.100 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-25T01:30:44.100 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42171sec preferred_lft 42171sec 2024-04-25T01:30:44.100 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-04-25T01:30:44.100 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:30:44.100 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:30:44.100 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-04-25T01:30:44.100 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-04-25T01:30:44.101 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T01:30:44.101 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:30:44.101 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T01:30:44.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-04-25T01:30:44.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-04-25T01:30:44.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-04-25T01:30:44.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T01:30:44.101 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T01:30:44.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:30:44.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:30:44.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:30:44.289 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:30:44.314 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-04-25T01:30:44.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:30:44.316 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-04-25T01:30:44.384 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-04-25T01:30:44.385 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:30:44.385 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-25T01:30:44.385 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-25T01:30:44.385 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:30:44.385 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:30:44.385 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T01:30:44.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-25T01:30:44.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-25T01:30:44.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-25T01:30:44.386 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T01:30:44.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:30:44.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:44 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1633966996' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:30:44.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:30:44.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:30:44.678 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-25T01:30:44.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:30:44.694 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-04-25T01:30:44.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:44 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1633966996' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:30:44.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:44 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1633966996' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:30:44.749 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:30:44.749 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T01:30:44.749 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:30:44.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-25T01:30:44.750 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T01:30:44.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:30:44.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:30:44.868 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-25T01:30:44.868 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:30:44.868 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T01:30:44.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-25T01:30:44.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T01:30:44.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-25T01:30:44.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-25T01:30:44.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-25T01:30:44.868 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T01:30:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:30:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:30:44.992 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:30:44.992 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T01:30:44.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-04-25T01:30:44.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T01:30:44.992 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T01:30:45.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:30:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:30:45.069 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-25T01:30:45.069 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:30:45.069 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:30:45.084 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T01:30:45.084 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:30:45.084 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:30:45.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-04-25T01:30:45.205 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T01:30:45.262 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-04-25T01:30:45.262 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-04-25T01:30:45.263 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-04-25T01:30:45.263 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-04-25T01:30:45.263 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-04-25T01:30:45.263 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-04-25T01:30:45.263 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-04-25T01:30:45.263 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-04-25T01:30:45.263 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-04-25T01:30:45.263 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-04-25T01:30:45.263 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-04-25T01:30:45.263 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/selinux 2024-04-25T01:30:45.263 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:30:45.263 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-04-25T01:30:45.263 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-04-25T01:30:45.264 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-04-25T01:30:45.264 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-04-25T01:30:45.264 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-04-25T01:30:45.264 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T01:30:45.264 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-04-25T01:30:45.264 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T01:30:45.264 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T01:30:45.264 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T01:30:45.264 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T01:30:45.264 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-04-25T01:30:45.264 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-04-25T01:30:45.264 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay 2024-04-25T01:30:45.264 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/4f6227525fd9c6b053cda8aa69f2942dbeaa2d007154d2768dab00cd0a473c17/merged 2024-04-25T01:30:45.264 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/7acd5873b1db1ccca1db212f6adc750453a00a247d0f07dfddf092f9946e2263/merged 2024-04-25T01:30:45.265 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/0 2024-04-25T01:30:45.265 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:30:45.265 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/5ff2be39d0e50065ce5f004816cad4ef606b39ee412f853c7c5a4d937f9ec338/merged 2024-04-25T01:30:45.265 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/8bd31929cd30e3a5fb7a73e0863b6efa767910b19bb238c2bb43caaa678b835f/merged 2024-04-25T01:30:45.265 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/50d095b810f62a32e95391256a5a11e13a5eccafc2414e9a2cbbd0baf1e9e046/merged 2024-04-25T01:30:45.265 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/71f56e4f30637cd823b52b90d259fe74467628e4a29ea51c54e617c0f8a6d015/merged 2024-04-25T01:30:45.265 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/a675b0d55ee2d38d59f12757ade1c3122bc7b638aada19110c3583d4448c7934/merged 2024-04-25T01:30:45.265 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/dfc8f380b40e51c51394b772c7e04ac152a42c294af8d24f3d3fe24edf00a51b/merged 2024-04-25T01:30:45.265 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/5f098d1e60927687cc846e814fcf3302f6c8702b00ef53ddab09e6097dd4b094/merged 2024-04-25T01:30:45.265 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/000c28cab02fb1181d471eee0f1a779cfc12cffe4bc1776d4a7001273b1952c7/merged 2024-04-25T01:30:45.265 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-04-25T01:30:45.265 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:30:45.265 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:30:45.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:30:45.266 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-04-25T01:30:45.319 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T01:30:45.319 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-25T01:30:45.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-04-25T01:30:45.387 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T01:30:45.444 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-04-25T01:30:45.444 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-04-25T01:30:45.444 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-04-25T01:30:45.444 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-04-25T01:30:45.444 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-04-25T01:30:45.444 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-04-25T01:30:45.444 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-04-25T01:30:45.444 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-04-25T01:30:45.444 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-04-25T01:30:45.444 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-04-25T01:30:45.445 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-04-25T01:30:45.445 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/selinux 2024-04-25T01:30:45.445 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:30:45.445 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-04-25T01:30:45.445 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-04-25T01:30:45.445 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-04-25T01:30:45.445 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-04-25T01:30:45.445 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-04-25T01:30:45.445 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T01:30:45.445 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-04-25T01:30:45.445 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T01:30:45.445 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T01:30:45.445 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T01:30:45.445 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T01:30:45.446 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-04-25T01:30:45.446 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-04-25T01:30:45.446 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay 2024-04-25T01:30:45.446 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/4f6227525fd9c6b053cda8aa69f2942dbeaa2d007154d2768dab00cd0a473c17/merged 2024-04-25T01:30:45.446 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/7acd5873b1db1ccca1db212f6adc750453a00a247d0f07dfddf092f9946e2263/merged 2024-04-25T01:30:45.446 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/0 2024-04-25T01:30:45.446 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T01:30:45.446 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/5ff2be39d0e50065ce5f004816cad4ef606b39ee412f853c7c5a4d937f9ec338/merged 2024-04-25T01:30:45.446 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/8bd31929cd30e3a5fb7a73e0863b6efa767910b19bb238c2bb43caaa678b835f/merged 2024-04-25T01:30:45.446 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/50d095b810f62a32e95391256a5a11e13a5eccafc2414e9a2cbbd0baf1e9e046/merged 2024-04-25T01:30:45.446 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/71f56e4f30637cd823b52b90d259fe74467628e4a29ea51c54e617c0f8a6d015/merged 2024-04-25T01:30:45.446 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/a675b0d55ee2d38d59f12757ade1c3122bc7b638aada19110c3583d4448c7934/merged 2024-04-25T01:30:45.447 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/dfc8f380b40e51c51394b772c7e04ac152a42c294af8d24f3d3fe24edf00a51b/merged 2024-04-25T01:30:45.447 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/5f098d1e60927687cc846e814fcf3302f6c8702b00ef53ddab09e6097dd4b094/merged 2024-04-25T01:30:45.447 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/000c28cab02fb1181d471eee0f1a779cfc12cffe4bc1776d4a7001273b1952c7/merged 2024-04-25T01:30:45.447 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-04-25T01:30:45.447 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:30:45.447 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:30:45.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:30:45.447 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-04-25T01:30:45.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:2024-04-25T01:30:45.479+0000 7f0bc20ac480 -1 init, newargv = 0x56324ccc90e0 newargc=15 2024-04-25T01:30:45.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[71516]: starting ceph client 2024-04-25T01:30:45.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[71516]: starting fuse 2024-04-25T01:30:45.474 INFO:teuthology.orchestra.run.smithi028.stdout:115 2024-04-25T01:30:45.475 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-04-25T01:30:45.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo stdin-killer -- python3 -c ' 2024-04-25T01:30:45.475 DEBUG:teuthology.orchestra.run.smithi028:> import glob 2024-04-25T01:30:45.475 DEBUG:teuthology.orchestra.run.smithi028:> import re 2024-04-25T01:30:45.475 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-04-25T01:30:45.475 DEBUG:teuthology.orchestra.run.smithi028:> import subprocess 2024-04-25T01:30:45.475 DEBUG:teuthology.orchestra.run.smithi028:> 2024-04-25T01:30:45.475 DEBUG:teuthology.orchestra.run.smithi028:> def _find_admin_socket(client_name): 2024-04-25T01:30:45.475 DEBUG:teuthology.orchestra.run.smithi028:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-25T01:30:45.475 DEBUG:teuthology.orchestra.run.smithi028:> files = glob.glob(asok_path) 2024-04-25T01:30:45.475 DEBUG:teuthology.orchestra.run.smithi028:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-25T01:30:45.475 DEBUG:teuthology.orchestra.run.smithi028:> 2024-04-25T01:30:45.476 DEBUG:teuthology.orchestra.run.smithi028:> # Given a non-glob path, it better be there 2024-04-25T01:30:45.476 DEBUG:teuthology.orchestra.run.smithi028:> if "*" not in asok_path: 2024-04-25T01:30:45.476 DEBUG:teuthology.orchestra.run.smithi028:> assert(len(files) == 1) 2024-04-25T01:30:45.476 DEBUG:teuthology.orchestra.run.smithi028:> return files[0] 2024-04-25T01:30:45.476 DEBUG:teuthology.orchestra.run.smithi028:> 2024-04-25T01:30:45.476 DEBUG:teuthology.orchestra.run.smithi028:> for f in files: 2024-04-25T01:30:45.476 DEBUG:teuthology.orchestra.run.smithi028:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T01:30:45.476 DEBUG:teuthology.orchestra.run.smithi028:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T01:30:45.476 DEBUG:teuthology.orchestra.run.smithi028:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T01:30:45.476 DEBUG:teuthology.orchestra.run.smithi028:> contents = proc_f.read() 2024-04-25T01:30:45.476 DEBUG:teuthology.orchestra.run.smithi028:> if mountpoint in contents: 2024-04-25T01:30:45.476 DEBUG:teuthology.orchestra.run.smithi028:> return f 2024-04-25T01:30:45.476 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T01:30:45.476 DEBUG:teuthology.orchestra.run.smithi028:> 2024-04-25T01:30:45.476 DEBUG:teuthology.orchestra.run.smithi028:> print(_find_admin_socket("client.admin")) 2024-04-25T01:30:45.477 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-04-25T01:30:45.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:45 smithi028 ceph-mon[29390]: pgmap v303: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 3 op/s 2024-04-25T01:30:45.581 INFO:teuthology.orchestra.run.smithi028.stdout:/var/run/ceph/ceph-client.admin.71516.asok 2024-04-25T01:30:45.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:30:45.594 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.71516.asok 2024-04-25T01:30:45.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:30:45.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.71516.asok status 2024-04-25T01:30:45.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:45 smithi130 ceph-mon[32702]: pgmap v303: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 3 op/s 2024-04-25T01:30:45.733 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:30:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "metadata": { 2024-04-25T01:30:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_sha1": "d2c45d977142e024a59b68a8f1dce8f0b9840551", 2024-04-25T01:30:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:30:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "entity_id": "admin", 2024-04-25T01:30:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": "smithi028", 2024-04-25T01:30:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-25T01:30:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "pid": "71516", 2024-04-25T01:30:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "root": "/" 2024-04-25T01:30:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:30:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_count": 0, 2024-04-25T01:30:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_pinned_count": 0, 2024-04-25T01:30:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 14799, 2024-04-25T01:30:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "inst": { 2024-04-25T01:30:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "name": { 2024-04-25T01:30:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client", 2024-04-25T01:30:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 14799 2024-04-25T01:30:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:30:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-04-25T01:30:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-04-25T01:30:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-04-25T01:30:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 751748844 2024-04-25T01:30:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:30:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:30:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-04-25T01:30:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-04-25T01:30:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-04-25T01:30:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 751748844 2024-04-25T01:30:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:30:45.735 INFO:teuthology.orchestra.run.smithi028.stdout: "inst_str": "client.14799 192.168.0.1:0/751748844", 2024-04-25T01:30:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "addr_str": "192.168.0.1:0/751748844", 2024-04-25T01:30:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "inode_count": 1, 2024-04-25T01:30:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_epoch": 17, 2024-04-25T01:30:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch": 91, 2024-04-25T01:30:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch_barrier": 0, 2024-04-25T01:30:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklisted": false, 2024-04-25T01:30:45.736 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs" 2024-04-25T01:30:45.736 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:30:45.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:45 smithi032 ceph-mon[32461]: pgmap v303: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 3 op/s 2024-04-25T01:30:45.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:30:45.743 DEBUG:teuthology.orchestra.run.smithi028:> stat --file-system '--printf=%T 2024-04-25T01:30:45.743 DEBUG:teuthology.orchestra.run.smithi028:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:30:45.759 INFO:teuthology.orchestra.run.smithi028.stdout:fuseblk 2024-04-25T01:30:45.759 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:30:45.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:30:45.760 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:30:45.827 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-25T01:30:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:30:45.942 INFO:teuthology.orchestra.run.smithi028.stdout:# file: . 2024-04-25T01:30:45.942 INFO:teuthology.orchestra.run.smithi028.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-25T01:30:45.942 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:45.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:30:45.949 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:30:45.949 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:30:45.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:30:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.130+0000 7f8c08c61640 1 -- 172.21.15.28:0/1536144568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c04100700 msgr2=0x7f8c04100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.130+0000 7f8c08c61640 1 --2- 172.21.15.28:0/1536144568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c04100700 0x7f8c04100b20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8009900 tx=0x7f8bf802a410 comp rx=0 tx=0).stop 2024-04-25T01:30:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.130+0000 7f8c08c61640 1 -- 172.21.15.28:0/1536144568 shutdown_connections 2024-04-25T01:30:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.130+0000 7f8c08c61640 1 --2- 172.21.15.28:0/1536144568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c041022b0 0x7f8c04108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.130+0000 7f8c08c61640 1 --2- 172.21.15.28:0/1536144568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c041018f0 0x7f8c04101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.130+0000 7f8c08c61640 1 --2- 172.21.15.28:0/1536144568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c04100700 0x7f8c04100b20 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.130+0000 7f8c08c61640 1 -- 172.21.15.28:0/1536144568 >> 172.21.15.28:0/1536144568 conn(0x7f8c040fbc90 msgr2=0x7f8c040fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.130+0000 7f8c08c61640 1 -- 172.21.15.28:0/1536144568 shutdown_connections 2024-04-25T01:30:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.130+0000 7f8c08c61640 1 -- 172.21.15.28:0/1536144568 wait complete. 2024-04-25T01:30:46.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.131+0000 7f8c08c61640 1 Processor -- start 2024-04-25T01:30:46.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.131+0000 7f8c08c61640 1 -- start start 2024-04-25T01:30:46.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.131+0000 7f8c08c61640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c04100700 0x7f8c041969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:46.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.131+0000 7f8c08c61640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c041018f0 0x7f8c04196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:46.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.131+0000 7f8c08c61640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c041022b0 0x7f8c0419df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:46.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.131+0000 7f8c08c61640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8c0410c2e0 con 0x7f8c041018f0 2024-04-25T01:30:46.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.131+0000 7f8c08c61640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8c0410c160 con 0x7f8c04100700 2024-04-25T01:30:46.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.131+0000 7f8c08c61640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8c0410c460 con 0x7f8c041022b0 2024-04-25T01:30:46.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.131+0000 7f8c01d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c041018f0 0x7f8c04196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:46.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.131+0000 7f8c01d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c041018f0 0x7f8c04196f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38086/0 (socket says 172.21.15.28:38086) 2024-04-25T01:30:46.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.131+0000 7f8c01d9b640 1 -- 172.21.15.28:0/3355371691 learned_addr learned my addr 172.21.15.28:0/3355371691 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:46.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.131+0000 7f8c0259c640 1 --2- 172.21.15.28:0/3355371691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c04100700 0x7f8c041969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:46.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.131+0000 7f8c02d9d640 1 --2- 172.21.15.28:0/3355371691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c041022b0 0x7f8c0419df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:46.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.131+0000 7f8c01d9b640 1 -- 172.21.15.28:0/3355371691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c041022b0 msgr2=0x7f8c0419df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:46.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.131+0000 7f8c01d9b640 1 --2- 172.21.15.28:0/3355371691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c041022b0 0x7f8c0419df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.131+0000 7f8c01d9b640 1 -- 172.21.15.28:0/3355371691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c04100700 msgr2=0x7f8c041969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:46.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.131+0000 7f8c01d9b640 1 --2- 172.21.15.28:0/3355371691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c04100700 0x7f8c041969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.131+0000 7f8c01d9b640 1 -- 172.21.15.28:0/3355371691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c0419e600 con 0x7f8c041018f0 2024-04-25T01:30:46.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.131+0000 7f8c02d9d640 1 --2- 172.21.15.28:0/3355371691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c041022b0 0x7f8c0419df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:46.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.132+0000 7f8c01d9b640 1 --2- 172.21.15.28:0/3355371691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c041018f0 0x7f8c04196f20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8bf400d950 tx=0x7f8bf400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:46.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.132+0000 7f8bf37fe640 1 -- 172.21.15.28:0/3355371691 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf4017070 con 0x7f8c041018f0 2024-04-25T01:30:46.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.132+0000 7f8c08c61640 1 -- 172.21.15.28:0/3355371691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c0419e8f0 con 0x7f8c041018f0 2024-04-25T01:30:46.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.132+0000 7f8bf37fe640 1 -- 172.21.15.28:0/3355371691 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bf4009c30 con 0x7f8c041018f0 2024-04-25T01:30:46.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.132+0000 7f8bf37fe640 1 -- 172.21.15.28:0/3355371691 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf400f6b0 con 0x7f8c041018f0 2024-04-25T01:30:46.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.132+0000 7f8c08c61640 1 -- 172.21.15.28:0/3355371691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c0419ee40 con 0x7f8c041018f0 2024-04-25T01:30:46.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.133+0000 7f8c08c61640 1 -- 172.21.15.28:0/3355371691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c04107af0 con 0x7f8c041018f0 2024-04-25T01:30:46.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.134+0000 7f8bf37fe640 1 -- 172.21.15.28:0/3355371691 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8bf40109d0 con 0x7f8c041018f0 2024-04-25T01:30:46.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.135+0000 7f8bf37fe640 1 --2- 172.21.15.28:0/3355371691 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8bec0a2b90 0x7f8bec0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:46.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.135+0000 7f8c0259c640 1 --2- 172.21.15.28:0/3355371691 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8bec0a2b90 0x7f8bec0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:46.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.135+0000 7f8bf37fe640 1 -- 172.21.15.28:0/3355371691 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f8bf40ceed0 con 0x7f8c041018f0 2024-04-25T01:30:46.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.135+0000 7f8c0259c640 1 --2- 172.21.15.28:0/3355371691 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8bec0a2b90 0x7f8bec0a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8005ab0 tx=0x7f8bf8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:46.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.139+0000 7f8bf37fe640 1 -- 172.21.15.28:0/3355371691 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8bf4009de0 con 0x7f8c041018f0 2024-04-25T01:30:46.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.327+0000 7f8c08c61640 1 -- 172.21.15.28:0/3355371691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8c04062340 con 0x7f8c041018f0 2024-04-25T01:30:46.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.329+0000 7f8bf37fe640 1 -- 172.21.15.28:0/3355371691 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24093 (secure 0 0 0) 0x7f8bf409bbf0 con 0x7f8c041018f0 2024-04-25T01:30:46.307 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:46.307 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:30:38.730329+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:30:46.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.331+0000 7f8c08c61640 1 -- 172.21.15.28:0/3355371691 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8bec0a2b90 msgr2=0x7f8bec0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.331+0000 7f8c08c61640 1 --2- 172.21.15.28:0/3355371691 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8bec0a2b90 0x7f8bec0a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8005ab0 tx=0x7f8bf8000bc0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.331+0000 7f8c08c61640 1 -- 172.21.15.28:0/3355371691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c041018f0 msgr2=0x7f8c04196f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.331+0000 7f8c08c61640 1 --2- 172.21.15.28:0/3355371691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c041018f0 0x7f8c04196f20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8bf400d950 tx=0x7f8bf400de10 comp rx=0 tx=0).stop 2024-04-25T01:30:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.331+0000 7f8c08c61640 1 -- 172.21.15.28:0/3355371691 shutdown_connections 2024-04-25T01:30:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.331+0000 7f8c08c61640 1 --2- 172.21.15.28:0/3355371691 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8bec0a2b90 0x7f8bec0a5050 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.331+0000 7f8c08c61640 1 --2- 172.21.15.28:0/3355371691 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c041022b0 0x7f8c0419df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.331+0000 7f8c08c61640 1 --2- 172.21.15.28:0/3355371691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c041018f0 0x7f8c04196f20 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.331+0000 7f8c08c61640 1 --2- 172.21.15.28:0/3355371691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c04100700 0x7f8c041969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.331+0000 7f8c08c61640 1 -- 172.21.15.28:0/3355371691 >> 172.21.15.28:0/3355371691 conn(0x7f8c040fbc90 msgr2=0x7f8c04104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.331+0000 7f8c08c61640 1 -- 172.21.15.28:0/3355371691 shutdown_connections 2024-04-25T01:30:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.331+0000 7f8c08c61640 1 -- 172.21.15.28:0/3355371691 wait complete. 2024-04-25T01:30:46.321 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:25:48.767414+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:30:32.282430+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:30:33.127049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:30:36.796132+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:30:46.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:30:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.451+0000 7ff50dcee640 1 -- 172.21.15.28:0/120779231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff508102260 msgr2=0x7ff508104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.451+0000 7ff50dcee640 1 --2- 172.21.15.28:0/120779231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff508102260 0x7ff508104650 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009900 tx=0x7ff4f802a410 comp rx=0 tx=0).stop 2024-04-25T01:30:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.452+0000 7ff50dcee640 1 -- 172.21.15.28:0/120779231 shutdown_connections 2024-04-25T01:30:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.452+0000 7ff50dcee640 1 --2- 172.21.15.28:0/120779231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5081074c0 0x7ff5081098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.452+0000 7ff50dcee640 1 --2- 172.21.15.28:0/120779231 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff508104b90 0x7ff508106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.452+0000 7ff50dcee640 1 --2- 172.21.15.28:0/120779231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff508102260 0x7ff508104650 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.452+0000 7ff50dcee640 1 -- 172.21.15.28:0/120779231 >> 172.21.15.28:0/120779231 conn(0x7ff5080fbbf0 msgr2=0x7ff5080fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.452+0000 7ff50dcee640 1 -- 172.21.15.28:0/120779231 shutdown_connections 2024-04-25T01:30:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.452+0000 7ff50dcee640 1 -- 172.21.15.28:0/120779231 wait complete. 2024-04-25T01:30:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.452+0000 7ff50dcee640 1 Processor -- start 2024-04-25T01:30:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.452+0000 7ff50dcee640 1 -- start start 2024-04-25T01:30:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff50dcee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff508102260 0x7ff508197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff50dcee640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff508104b90 0x7ff508198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff50dcee640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5081074c0 0x7ff50819f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff50dcee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff50810d470 con 0x7ff508102260 2024-04-25T01:30:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff50dcee640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff50810d2f0 con 0x7ff508104b90 2024-04-25T01:30:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff50dcee640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff50810d5f0 con 0x7ff5081074c0 2024-04-25T01:30:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff5077fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff508102260 0x7ff508197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff5077fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff508102260 0x7ff508197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38104/0 (socket says 172.21.15.28:38104) 2024-04-25T01:30:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff5077fe640 1 -- 172.21.15.28:0/3049734976 learned_addr learned my addr 172.21.15.28:0/3049734976 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff506ffd640 1 --2- 172.21.15.28:0/3049734976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff508104b90 0x7ff508198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff5077fe640 1 -- 172.21.15.28:0/3049734976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5081074c0 msgr2=0x7ff50819f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff5077fe640 1 --2- 172.21.15.28:0/3049734976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5081074c0 0x7ff50819f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff5077fe640 1 -- 172.21.15.28:0/3049734976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff508104b90 msgr2=0x7ff508198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff5077fe640 1 --2- 172.21.15.28:0/3049734976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff508104b90 0x7ff508198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff5077fe640 1 -- 172.21.15.28:0/3049734976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff50819f760 con 0x7ff508102260 2024-04-25T01:30:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff5077fe640 1 --2- 172.21.15.28:0/3049734976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff508102260 0x7ff508197b40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8031440 tx=0x7ff4f8032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff504ff9640 1 -- 172.21.15.28:0/3049734976 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f803f030 con 0x7ff508102260 2024-04-25T01:30:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff50dcee640 1 -- 172.21.15.28:0/3049734976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff50819f9f0 con 0x7ff508102260 2024-04-25T01:30:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff504ff9640 1 -- 172.21.15.28:0/3049734976 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4f8003ce0 con 0x7ff508102260 2024-04-25T01:30:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.453+0000 7ff504ff9640 1 -- 172.21.15.28:0/3049734976 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8035070 con 0x7ff508102260 2024-04-25T01:30:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.454+0000 7ff50dcee640 1 -- 172.21.15.28:0/3049734976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff50819fee0 con 0x7ff508102260 2024-04-25T01:30:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.454+0000 7ff50dcee640 1 -- 172.21.15.28:0/3049734976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5080fd6e0 con 0x7ff508102260 2024-04-25T01:30:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.456+0000 7ff504ff9640 1 -- 172.21.15.28:0/3049734976 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff4f8042030 con 0x7ff508102260 2024-04-25T01:30:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.457+0000 7ff504ff9640 1 --2- 172.21.15.28:0/3049734976 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4f40a2b90 0x7ff4f40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.457+0000 7ff506ffd640 1 --2- 172.21.15.28:0/3049734976 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4f40a2b90 0x7ff4f40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.457+0000 7ff504ff9640 1 -- 172.21.15.28:0/3049734976 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7ff4f80ef870 con 0x7ff508102260 2024-04-25T01:30:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.457+0000 7ff506ffd640 1 --2- 172.21.15.28:0/3049734976 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4f40a2b90 0x7ff4f40a5050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff508199050 tx=0x7ff4f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:46.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.461+0000 7ff504ff9640 1 -- 172.21.15.28:0/3049734976 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff4f80bc590 con 0x7ff508102260 2024-04-25T01:30:46.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.660+0000 7ff50dcee640 1 -- 172.21.15.28:0/3049734976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff508062340 con 0x7ff508102260 2024-04-25T01:30:46.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.660+0000 7ff504ff9640 1 -- 172.21.15.28:0/3049734976 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7ff4f80bbe90 con 0x7ff508102260 2024-04-25T01:30:46.638 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T01:30:46.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.662+0000 7ff50dcee640 1 -- 172.21.15.28:0/3049734976 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4f40a2b90 msgr2=0x7ff4f40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:46.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.662+0000 7ff50dcee640 1 --2- 172.21.15.28:0/3049734976 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4f40a2b90 0x7ff4f40a5050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff508199050 tx=0x7ff4f0005f90 comp rx=0 tx=0).stop 2024-04-25T01:30:46.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.662+0000 7ff50dcee640 1 -- 172.21.15.28:0/3049734976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff508102260 msgr2=0x7ff508197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:46.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.662+0000 7ff50dcee640 1 --2- 172.21.15.28:0/3049734976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff508102260 0x7ff508197b40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8031440 tx=0x7ff4f8032ce0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.662+0000 7ff50dcee640 1 -- 172.21.15.28:0/3049734976 shutdown_connections 2024-04-25T01:30:46.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.662+0000 7ff50dcee640 1 --2- 172.21.15.28:0/3049734976 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4f40a2b90 0x7ff4f40a5050 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.662+0000 7ff50dcee640 1 --2- 172.21.15.28:0/3049734976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5081074c0 0x7ff50819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.662+0000 7ff50dcee640 1 --2- 172.21.15.28:0/3049734976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff508104b90 0x7ff508198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.662+0000 7ff50dcee640 1 --2- 172.21.15.28:0/3049734976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff508102260 0x7ff508197b40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.662+0000 7ff50dcee640 1 -- 172.21.15.28:0/3049734976 >> 172.21.15.28:0/3049734976 conn(0x7ff5080fbbf0 msgr2=0x7ff508102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:46.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.663+0000 7ff50dcee640 1 -- 172.21.15.28:0/3049734976 shutdown_connections 2024-04-25T01:30:46.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.663+0000 7ff50dcee640 1 -- 172.21.15.28:0/3049734976 wait complete. 2024-04-25T01:30:46.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:46 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3355371691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:46.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:30:46.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:46 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3355371691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:46.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:46 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3355371691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:46.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.779+0000 7fdac7f9e640 1 -- 172.21.15.28:0/2699547702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdac01006a0 msgr2=0x7fdac0100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:46.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.779+0000 7fdac7f9e640 1 --2- 172.21.15.28:0/2699547702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdac01006a0 0x7fdac0100ac0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdabc009900 tx=0x7fdabc02a410 comp rx=0 tx=0).stop 2024-04-25T01:30:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.779+0000 7fdac7f9e640 1 -- 172.21.15.28:0/2699547702 shutdown_connections 2024-04-25T01:30:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.779+0000 7fdac7f9e640 1 --2- 172.21.15.28:0/2699547702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdac0102250 0x7fdac0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.779+0000 7fdac7f9e640 1 --2- 172.21.15.28:0/2699547702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac0101890 0x7fdac0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.779+0000 7fdac7f9e640 1 --2- 172.21.15.28:0/2699547702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdac01006a0 0x7fdac0100ac0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.779+0000 7fdac7f9e640 1 -- 172.21.15.28:0/2699547702 >> 172.21.15.28:0/2699547702 conn(0x7fdac00fbc30 msgr2=0x7fdac00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.779+0000 7fdac7f9e640 1 -- 172.21.15.28:0/2699547702 shutdown_connections 2024-04-25T01:30:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.779+0000 7fdac7f9e640 1 -- 172.21.15.28:0/2699547702 wait complete. 2024-04-25T01:30:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.779+0000 7fdac7f9e640 1 Processor -- start 2024-04-25T01:30:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.779+0000 7fdac7f9e640 1 -- start start 2024-04-25T01:30:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.779+0000 7fdac7f9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdac01006a0 0x7fdac0197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.779+0000 7fdac7f9e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdac0101890 0x7fdac0198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:46.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.779+0000 7fdac7f9e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac0102250 0x7fdac019f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdac7f9e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdac010c370 con 0x7fdac01006a0 2024-04-25T01:30:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdac7f9e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdac010c1f0 con 0x7fdac0102250 2024-04-25T01:30:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdac7f9e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fdac010c4f0 con 0x7fdac0101890 2024-04-25T01:30:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdac5d3a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdac01006a0 0x7fdac0197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdac5d3a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdac01006a0 0x7fdac0197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60368/0 (socket says 172.21.15.28:60368) 2024-04-25T01:30:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdac5d3a640 1 -- 172.21.15.28:0/1228569030 learned_addr learned my addr 172.21.15.28:0/1228569030 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdac653b640 1 --2- 172.21.15.28:0/1228569030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac0102250 0x7fdac019f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdac5539640 1 --2- 172.21.15.28:0/1228569030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdac0101890 0x7fdac0198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdac5d3a640 1 -- 172.21.15.28:0/1228569030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdac0101890 msgr2=0x7fdac0198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdac5d3a640 1 --2- 172.21.15.28:0/1228569030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdac0101890 0x7fdac0198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdac5d3a640 1 -- 172.21.15.28:0/1228569030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac0102250 msgr2=0x7fdac019f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdac5d3a640 1 --2- 172.21.15.28:0/1228569030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac0102250 0x7fdac019f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdac5d3a640 1 -- 172.21.15.28:0/1228569030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac019f760 con 0x7fdac01006a0 2024-04-25T01:30:46.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdac653b640 1 --2- 172.21.15.28:0/1228569030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac0102250 0x7fdac019f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:46.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdac5539640 1 --2- 172.21.15.28:0/1228569030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdac0101890 0x7fdac0198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:46.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdac5d3a640 1 --2- 172.21.15.28:0/1228569030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdac01006a0 0x7fdac0197b40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdabc031440 tx=0x7fdabc032e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:46.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdab6ffd640 1 -- 172.21.15.28:0/1228569030 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc03f030 con 0x7fdac01006a0 2024-04-25T01:30:46.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdab6ffd640 1 -- 172.21.15.28:0/1228569030 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdabc003ce0 con 0x7fdac01006a0 2024-04-25T01:30:46.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdac7f9e640 1 -- 172.21.15.28:0/1228569030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdac019f9f0 con 0x7fdac01006a0 2024-04-25T01:30:46.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdab6ffd640 1 -- 172.21.15.28:0/1228569030 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc035070 con 0x7fdac01006a0 2024-04-25T01:30:46.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.780+0000 7fdac7f9e640 1 -- 172.21.15.28:0/1228569030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdac019fee0 con 0x7fdac01006a0 2024-04-25T01:30:46.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.781+0000 7fdac7f9e640 1 -- 172.21.15.28:0/1228569030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdac01a00b0 con 0x7fdac01006a0 2024-04-25T01:30:46.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.782+0000 7fdab6ffd640 1 -- 172.21.15.28:0/1228569030 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fdabc042030 con 0x7fdac01006a0 2024-04-25T01:30:46.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.783+0000 7fdab6ffd640 1 --2- 172.21.15.28:0/1228569030 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fdaac0a2b90 0x7fdaac0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:46.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.783+0000 7fdac5539640 1 --2- 172.21.15.28:0/1228569030 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fdaac0a2b90 0x7fdaac0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:46.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.783+0000 7fdab6ffd640 1 -- 172.21.15.28:0/1228569030 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fdabc0ef680 con 0x7fdac01006a0 2024-04-25T01:30:46.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.783+0000 7fdac5539640 1 --2- 172.21.15.28:0/1228569030 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fdaac0a2b90 0x7fdaac0a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdab0005d00 tx=0x7fdab0005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:46.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.787+0000 7fdab6ffd640 1 -- 172.21.15.28:0/1228569030 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdabc0bc340 con 0x7fdac01006a0 2024-04-25T01:30:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.970+0000 7fdac7f9e640 1 -- 172.21.15.28:0/1228569030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdac01a0440 con 0x7fdac01006a0 2024-04-25T01:30:46.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.971+0000 7fdab6ffd640 1 -- 172.21.15.28:0/1228569030 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fdabc0bbc40 con 0x7fdac01006a0 2024-04-25T01:30:46.948 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:30:46.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.973+0000 7fdac7f9e640 1 -- 172.21.15.28:0/1228569030 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fdaac0a2b90 msgr2=0x7fdaac0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:46.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.973+0000 7fdac7f9e640 1 --2- 172.21.15.28:0/1228569030 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fdaac0a2b90 0x7fdaac0a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdab0005d00 tx=0x7fdab0005ed0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.973+0000 7fdac7f9e640 1 -- 172.21.15.28:0/1228569030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdac01006a0 msgr2=0x7fdac0197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:46.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.973+0000 7fdac7f9e640 1 --2- 172.21.15.28:0/1228569030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdac01006a0 0x7fdac0197b40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdabc031440 tx=0x7fdabc032e20 comp rx=0 tx=0).stop 2024-04-25T01:30:46.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.973+0000 7fdac7f9e640 1 -- 172.21.15.28:0/1228569030 shutdown_connections 2024-04-25T01:30:46.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.973+0000 7fdac7f9e640 1 --2- 172.21.15.28:0/1228569030 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fdaac0a2b90 0x7fdaac0a5050 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.973+0000 7fdac7f9e640 1 --2- 172.21.15.28:0/1228569030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac0102250 0x7fdac019f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.973+0000 7fdac7f9e640 1 --2- 172.21.15.28:0/1228569030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdac0101890 0x7fdac0198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.973+0000 7fdac7f9e640 1 --2- 172.21.15.28:0/1228569030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdac01006a0 0x7fdac0197b40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:46.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.973+0000 7fdac7f9e640 1 -- 172.21.15.28:0/1228569030 >> 172.21.15.28:0/1228569030 conn(0x7fdac00fbc30 msgr2=0x7fdac00fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:46.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.973+0000 7fdac7f9e640 1 -- 172.21.15.28:0/1228569030 shutdown_connections 2024-04-25T01:30:46.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:46.973+0000 7fdac7f9e640 1 -- 172.21.15.28:0/1228569030 wait complete. 2024-04-25T01:30:46.961 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:30:47.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.088+0000 7f911e909640 1 -- 172.21.15.28:0/2058185909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9118101890 msgr2=0x7f9118101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:47.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.088+0000 7f911e909640 1 --2- 172.21.15.28:0/2058185909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9118101890 0x7f9118101d10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f910c009900 tx=0x7f910c02a410 comp rx=0 tx=0).stop 2024-04-25T01:30:47.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.088+0000 7f911e909640 1 -- 172.21.15.28:0/2058185909 shutdown_connections 2024-04-25T01:30:47.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.088+0000 7f911e909640 1 --2- 172.21.15.28:0/2058185909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9118102250 0x7f9118108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.088+0000 7f911e909640 1 --2- 172.21.15.28:0/2058185909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9118101890 0x7f9118101d10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.088+0000 7f911e909640 1 --2- 172.21.15.28:0/2058185909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f91181006a0 0x7f9118100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.088+0000 7f911e909640 1 -- 172.21.15.28:0/2058185909 >> 172.21.15.28:0/2058185909 conn(0x7f91180fbc30 msgr2=0x7f91180fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.088+0000 7f911e909640 1 -- 172.21.15.28:0/2058185909 shutdown_connections 2024-04-25T01:30:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.088+0000 7f911e909640 1 -- 172.21.15.28:0/2058185909 wait complete. 2024-04-25T01:30:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.088+0000 7f911e909640 1 Processor -- start 2024-04-25T01:30:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.088+0000 7f911e909640 1 -- start start 2024-04-25T01:30:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f911e909640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91181006a0 0x7f9118197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f911e909640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9118101890 0x7f9118198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f911e909640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9118102250 0x7f911819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f911e909640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f911810d440 con 0x7f91181006a0 2024-04-25T01:30:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f911e909640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f911810d2c0 con 0x7f9118102250 2024-04-25T01:30:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f911e909640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f911810d5c0 con 0x7f9118101890 2024-04-25T01:30:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f9117fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91181006a0 0x7f9118197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f9117fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91181006a0 0x7f9118197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60382/0 (socket says 172.21.15.28:60382) 2024-04-25T01:30:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f9117fff640 1 -- 172.21.15.28:0/1054225367 learned_addr learned my addr 172.21.15.28:0/1054225367 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f91177fe640 1 --2- 172.21.15.28:0/1054225367 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9118101890 0x7f9118198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f911cea6640 1 --2- 172.21.15.28:0/1054225367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9118102250 0x7f911819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f9117fff640 1 -- 172.21.15.28:0/1054225367 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9118101890 msgr2=0x7f9118198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f9117fff640 1 --2- 172.21.15.28:0/1054225367 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9118101890 0x7f9118198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f9117fff640 1 -- 172.21.15.28:0/1054225367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9118102250 msgr2=0x7f911819f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f9117fff640 1 --2- 172.21.15.28:0/1054225367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9118102250 0x7f911819f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f9117fff640 1 -- 172.21.15.28:0/1054225367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f911819f6f0 con 0x7f91181006a0 2024-04-25T01:30:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f91177fe640 1 --2- 172.21.15.28:0/1054225367 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9118101890 0x7f9118198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f9117fff640 1 --2- 172.21.15.28:0/1054225367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91181006a0 0x7f9118197ad0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f910800d950 tx=0x7f910800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f91157fa640 1 -- 172.21.15.28:0/1054225367 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9108017070 con 0x7f91181006a0 2024-04-25T01:30:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f911e909640 1 -- 172.21.15.28:0/1054225367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f911819f9e0 con 0x7f91181006a0 2024-04-25T01:30:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f91157fa640 1 -- 172.21.15.28:0/1054225367 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9108009c30 con 0x7f91181006a0 2024-04-25T01:30:47.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.089+0000 7f91157fa640 1 -- 172.21.15.28:0/1054225367 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f910800f600 con 0x7f91181006a0 2024-04-25T01:30:47.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.090+0000 7f911e909640 1 -- 172.21.15.28:0/1054225367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f911819fee0 con 0x7f91181006a0 2024-04-25T01:30:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.091+0000 7f911e909640 1 -- 172.21.15.28:0/1054225367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9118107a40 con 0x7f91181006a0 2024-04-25T01:30:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.092+0000 7f91157fa640 1 -- 172.21.15.28:0/1054225367 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f910801a070 con 0x7f91181006a0 2024-04-25T01:30:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.092+0000 7f91157fa640 1 --2- 172.21.15.28:0/1054225367 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f91000a2b90 0x7f91000a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.092+0000 7f91157fa640 1 -- 172.21.15.28:0/1054225367 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f91080cedc0 con 0x7f91181006a0 2024-04-25T01:30:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.093+0000 7f91177fe640 1 --2- 172.21.15.28:0/1054225367 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f91000a2b90 0x7f91000a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.093+0000 7f91177fe640 1 --2- 172.21.15.28:0/1054225367 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f91000a2b90 0x7f91000a5050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f910c000c00 tx=0x7f910c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:47.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.097+0000 7f91157fa640 1 -- 172.21.15.28:0/1054225367 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9108009de0 con 0x7f91181006a0 2024-04-25T01:30:47.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.279+0000 7f911e909640 1 -- 172.21.15.28:0/1054225367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f911802d170 con 0x7f91181006a0 2024-04-25T01:30:47.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.279+0000 7f91157fa640 1 -- 172.21.15.28:0/1054225367 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f911802d170 con 0x7f91181006a0 2024-04-25T01:30:47.257 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T01:30:47.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.282+0000 7f911e909640 1 -- 172.21.15.28:0/1054225367 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f91000a2b90 msgr2=0x7f91000a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:47.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.282+0000 7f911e909640 1 --2- 172.21.15.28:0/1054225367 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f91000a2b90 0x7f91000a5050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f910c000c00 tx=0x7f910c031400 comp rx=0 tx=0).stop 2024-04-25T01:30:47.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.282+0000 7f911e909640 1 -- 172.21.15.28:0/1054225367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91181006a0 msgr2=0x7f9118197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:47.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.282+0000 7f911e909640 1 --2- 172.21.15.28:0/1054225367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91181006a0 0x7f9118197ad0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f910800d950 tx=0x7f910800de10 comp rx=0 tx=0).stop 2024-04-25T01:30:47.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.282+0000 7f911e909640 1 -- 172.21.15.28:0/1054225367 shutdown_connections 2024-04-25T01:30:47.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.282+0000 7f911e909640 1 --2- 172.21.15.28:0/1054225367 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f91000a2b90 0x7f91000a5050 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.282+0000 7f911e909640 1 --2- 172.21.15.28:0/1054225367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9118102250 0x7f911819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.282+0000 7f911e909640 1 --2- 172.21.15.28:0/1054225367 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9118101890 0x7f9118198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.282+0000 7f911e909640 1 --2- 172.21.15.28:0/1054225367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91181006a0 0x7f9118197ad0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.282+0000 7f911e909640 1 -- 172.21.15.28:0/1054225367 >> 172.21.15.28:0/1054225367 conn(0x7f91180fbc30 msgr2=0x7f9118104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.282+0000 7f911e909640 1 -- 172.21.15.28:0/1054225367 shutdown_connections 2024-04-25T01:30:47.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.282+0000 7f911e909640 1 -- 172.21.15.28:0/1054225367 wait complete. 2024-04-25T01:30:47.270 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:30:47.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.397+0000 7f2498316640 1 -- 172.21.15.28:0/1162600989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2490107500 msgr2=0x7f24901098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:47.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.397+0000 7f2498316640 1 --2- 172.21.15.28:0/1162600989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2490107500 0x7f24901098f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2488009ca0 tx=0x7f248802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.397+0000 7f2498316640 1 -- 172.21.15.28:0/1162600989 shutdown_connections 2024-04-25T01:30:47.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.397+0000 7f2498316640 1 --2- 172.21.15.28:0/1162600989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2490107500 0x7f24901098f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.397+0000 7f2498316640 1 --2- 172.21.15.28:0/1162600989 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2490104bd0 0x7f2490106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.397+0000 7f2498316640 1 --2- 172.21.15.28:0/1162600989 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24901022a0 0x7f2490104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.397+0000 7f2498316640 1 -- 172.21.15.28:0/1162600989 >> 172.21.15.28:0/1162600989 conn(0x7f24900fbc30 msgr2=0x7f24900fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.397+0000 7f2498316640 1 -- 172.21.15.28:0/1162600989 shutdown_connections 2024-04-25T01:30:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.397+0000 7f2498316640 1 -- 172.21.15.28:0/1162600989 wait complete. 2024-04-25T01:30:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.398+0000 7f2498316640 1 Processor -- start 2024-04-25T01:30:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.398+0000 7f2498316640 1 -- start start 2024-04-25T01:30:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.398+0000 7f2498316640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24901022a0 0x7f2490199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.398+0000 7f2498316640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2490104bd0 0x7f249019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.398+0000 7f2498316640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2490107500 0x7f24901a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.398+0000 7f2498316640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f249010d4e0 con 0x7f24901022a0 2024-04-25T01:30:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.398+0000 7f2498316640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f249010d360 con 0x7f2490104bd0 2024-04-25T01:30:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.398+0000 7f2498316640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f249010d660 con 0x7f2490107500 2024-04-25T01:30:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.398+0000 7f24960b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24901022a0 0x7f2490199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.398+0000 7f24960b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24901022a0 0x7f2490199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60406/0 (socket says 172.21.15.28:60406) 2024-04-25T01:30:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.398+0000 7f24960b2640 1 -- 172.21.15.28:0/2934318743 learned_addr learned my addr 172.21.15.28:0/2934318743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.398+0000 7f24958b1640 1 --2- 172.21.15.28:0/2934318743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2490104bd0 0x7f249019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.398+0000 7f24968b3640 1 --2- 172.21.15.28:0/2934318743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2490107500 0x7f24901a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.398+0000 7f24960b2640 1 -- 172.21.15.28:0/2934318743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2490107500 msgr2=0x7f24901a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.398+0000 7f24960b2640 1 --2- 172.21.15.28:0/2934318743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2490107500 0x7f24901a12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.398+0000 7f24960b2640 1 -- 172.21.15.28:0/2934318743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2490104bd0 msgr2=0x7f249019a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.398+0000 7f24960b2640 1 --2- 172.21.15.28:0/2934318743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2490104bd0 0x7f249019a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.398+0000 7f24960b2640 1 -- 172.21.15.28:0/2934318743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24901a1960 con 0x7f24901022a0 2024-04-25T01:30:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.398+0000 7f24958b1640 1 --2- 172.21.15.28:0/2934318743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2490104bd0 0x7f249019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.399+0000 7f24960b2640 1 --2- 172.21.15.28:0/2934318743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24901022a0 0x7f2490199d50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f248c00b4a0 tx=0x7f248c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.399+0000 7f24877fe640 1 -- 172.21.15.28:0/2934318743 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c014070 con 0x7f24901022a0 2024-04-25T01:30:47.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.399+0000 7f24877fe640 1 -- 172.21.15.28:0/2934318743 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f248c010af0 con 0x7f24901022a0 2024-04-25T01:30:47.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.399+0000 7f2498316640 1 -- 172.21.15.28:0/2934318743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24901a1c50 con 0x7f24901022a0 2024-04-25T01:30:47.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.399+0000 7f24877fe640 1 -- 172.21.15.28:0/2934318743 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c017d10 con 0x7f24901022a0 2024-04-25T01:30:47.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.399+0000 7f2498316640 1 -- 172.21.15.28:0/2934318743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24901a21a0 con 0x7f24901022a0 2024-04-25T01:30:47.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.400+0000 7f2498316640 1 -- 172.21.15.28:0/2934318743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2490108ff0 con 0x7f24901022a0 2024-04-25T01:30:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.401+0000 7f24877fe640 1 -- 172.21.15.28:0/2934318743 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f248c017070 con 0x7f24901022a0 2024-04-25T01:30:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.401+0000 7f24877fe640 1 --2- 172.21.15.28:0/2934318743 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f247c0a2b90 0x7f247c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.402+0000 7f24958b1640 1 --2- 172.21.15.28:0/2934318743 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f247c0a2b90 0x7f247c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.402+0000 7f24877fe640 1 -- 172.21.15.28:0/2934318743 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f248c0ce7f0 con 0x7f24901022a0 2024-04-25T01:30:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.402+0000 7f24958b1640 1 --2- 172.21.15.28:0/2934318743 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f247c0a2b90 0x7f247c0a5050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f249006a820 tx=0x7f2480005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:47.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.406+0000 7f24877fe640 1 -- 172.21.15.28:0/2934318743 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f248c09b4b0 con 0x7f24901022a0 2024-04-25T01:30:47.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:47 smithi028 ceph-mon[29390]: pgmap v304: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s wr, 1 op/s 2024-04-25T01:30:47.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:47 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3049734976' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:30:47.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:47 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1228569030' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:30:47.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:47 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1054225367' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:30:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.590+0000 7f2498316640 1 -- 172.21.15.28:0/2934318743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f2490062440 con 0x7f24901022a0 2024-04-25T01:30:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.590+0000 7f24877fe640 1 -- 172.21.15.28:0/2934318743 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f248c09adb0 con 0x7f24901022a0 2024-04-25T01:30:47.567 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:30:47.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.592+0000 7f2498316640 1 -- 172.21.15.28:0/2934318743 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f247c0a2b90 msgr2=0x7f247c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:47.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.592+0000 7f2498316640 1 --2- 172.21.15.28:0/2934318743 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f247c0a2b90 0x7f247c0a5050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f249006a820 tx=0x7f2480005e50 comp rx=0 tx=0).stop 2024-04-25T01:30:47.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.592+0000 7f2498316640 1 -- 172.21.15.28:0/2934318743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24901022a0 msgr2=0x7f2490199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:47.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.592+0000 7f2498316640 1 --2- 172.21.15.28:0/2934318743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24901022a0 0x7f2490199d50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f248c00b4a0 tx=0x7f248c00b960 comp rx=0 tx=0).stop 2024-04-25T01:30:47.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.592+0000 7f2498316640 1 -- 172.21.15.28:0/2934318743 shutdown_connections 2024-04-25T01:30:47.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.592+0000 7f2498316640 1 --2- 172.21.15.28:0/2934318743 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f247c0a2b90 0x7f247c0a5050 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.592+0000 7f2498316640 1 --2- 172.21.15.28:0/2934318743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2490107500 0x7f24901a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.592+0000 7f2498316640 1 --2- 172.21.15.28:0/2934318743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2490104bd0 0x7f249019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.592+0000 7f2498316640 1 --2- 172.21.15.28:0/2934318743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24901022a0 0x7f2490199d50 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.592+0000 7f2498316640 1 -- 172.21.15.28:0/2934318743 >> 172.21.15.28:0/2934318743 conn(0x7f24900fbc30 msgr2=0x7f2490101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:47.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.592+0000 7f2498316640 1 -- 172.21.15.28:0/2934318743 shutdown_connections 2024-04-25T01:30:47.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.592+0000 7f2498316640 1 -- 172.21.15.28:0/2934318743 wait complete. 2024-04-25T01:30:47.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:30:47.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.706+0000 7fcc32130640 1 -- 172.21.15.28:0/2043876949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc2c1074e0 msgr2=0x7fcc2c1098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:47.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.706+0000 7fcc32130640 1 --2- 172.21.15.28:0/2043876949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc2c1074e0 0x7fcc2c1098d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcc24009ca0 tx=0x7fcc2402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.706+0000 7fcc32130640 1 -- 172.21.15.28:0/2043876949 shutdown_connections 2024-04-25T01:30:47.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.706+0000 7fcc32130640 1 --2- 172.21.15.28:0/2043876949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc2c1074e0 0x7fcc2c1098d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.706+0000 7fcc32130640 1 --2- 172.21.15.28:0/2043876949 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc2c104bb0 0x7fcc2c106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.706+0000 7fcc32130640 1 --2- 172.21.15.28:0/2043876949 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc2c102280 0x7fcc2c104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.706+0000 7fcc32130640 1 -- 172.21.15.28:0/2043876949 >> 172.21.15.28:0/2043876949 conn(0x7fcc2c0fbc30 msgr2=0x7fcc2c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:47.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.706+0000 7fcc32130640 1 -- 172.21.15.28:0/2043876949 shutdown_connections 2024-04-25T01:30:47.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.706+0000 7fcc32130640 1 -- 172.21.15.28:0/2043876949 wait complete. 2024-04-25T01:30:47.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.706+0000 7fcc32130640 1 Processor -- start 2024-04-25T01:30:47.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.707+0000 7fcc32130640 1 -- start start 2024-04-25T01:30:47.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.707+0000 7fcc32130640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc2c102280 0x7fcc2c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:47.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.707+0000 7fcc32130640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc2c104bb0 0x7fcc2c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:47.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.707+0000 7fcc32130640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc2c1074e0 0x7fcc2c19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:47.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.707+0000 7fcc32130640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c10d440 con 0x7fcc2c1074e0 2024-04-25T01:30:47.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.707+0000 7fcc32130640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c10d2c0 con 0x7fcc2c102280 2024-04-25T01:30:47.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.707+0000 7fcc32130640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c10d5c0 con 0x7fcc2c104bb0 2024-04-25T01:30:47.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.707+0000 7fcc2bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc2c1074e0 0x7fcc2c19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:47.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.707+0000 7fcc2bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc2c1074e0 0x7fcc2c19f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60426/0 (socket says 172.21.15.28:60426) 2024-04-25T01:30:47.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.707+0000 7fcc2bfff640 1 -- 172.21.15.28:0/1584058078 learned_addr learned my addr 172.21.15.28:0/1584058078 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:47.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.707+0000 7fcc2b7fe640 1 --2- 172.21.15.28:0/1584058078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc2c102280 0x7fcc2c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:47.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.707+0000 7fcc2affd640 1 --2- 172.21.15.28:0/1584058078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc2c104bb0 0x7fcc2c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:47.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.707+0000 7fcc2bfff640 1 -- 172.21.15.28:0/1584058078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc2c104bb0 msgr2=0x7fcc2c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:47.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.707+0000 7fcc2bfff640 1 --2- 172.21.15.28:0/1584058078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc2c104bb0 0x7fcc2c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.707+0000 7fcc2bfff640 1 -- 172.21.15.28:0/1584058078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc2c102280 msgr2=0x7fcc2c197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:47.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.707+0000 7fcc2bfff640 1 --2- 172.21.15.28:0/1584058078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc2c102280 0x7fcc2c197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.707+0000 7fcc2bfff640 1 -- 172.21.15.28:0/1584058078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2c19f6a0 con 0x7fcc2c1074e0 2024-04-25T01:30:47.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.707+0000 7fcc2affd640 1 --2- 172.21.15.28:0/1584058078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc2c104bb0 0x7fcc2c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:47.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.707+0000 7fcc2bfff640 1 --2- 172.21.15.28:0/1584058078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc2c1074e0 0x7fcc2c19f030 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcc24031ae0 tx=0x7fcc24031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:47.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.708+0000 7fcc28ff9640 1 -- 172.21.15.28:0/1584058078 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24003c30 con 0x7fcc2c1074e0 2024-04-25T01:30:47.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.708+0000 7fcc28ff9640 1 -- 172.21.15.28:0/1584058078 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc24003dd0 con 0x7fcc2c1074e0 2024-04-25T01:30:47.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.708+0000 7fcc32130640 1 -- 172.21.15.28:0/1584058078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc2c19f930 con 0x7fcc2c1074e0 2024-04-25T01:30:47.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.708+0000 7fcc28ff9640 1 -- 172.21.15.28:0/1584058078 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2402fcb0 con 0x7fcc2c1074e0 2024-04-25T01:30:47.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.708+0000 7fcc32130640 1 -- 172.21.15.28:0/1584058078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc2c19fed0 con 0x7fcc2c1074e0 2024-04-25T01:30:47.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.709+0000 7fcc32130640 1 -- 172.21.15.28:0/1584058078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc2c0fd720 con 0x7fcc2c1074e0 2024-04-25T01:30:47.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.710+0000 7fcc28ff9640 1 -- 172.21.15.28:0/1584058078 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fcc24004050 con 0x7fcc2c1074e0 2024-04-25T01:30:47.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.710+0000 7fcc28ff9640 1 --2- 172.21.15.28:0/1584058078 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fcc140a2c10 0x7fcc140a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:47.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.711+0000 7fcc2b7fe640 1 --2- 172.21.15.28:0/1584058078 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fcc140a2c10 0x7fcc140a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:47.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.711+0000 7fcc28ff9640 1 -- 172.21.15.28:0/1584058078 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fcc240f0700 con 0x7fcc2c1074e0 2024-04-25T01:30:47.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.711+0000 7fcc2b7fe640 1 --2- 172.21.15.28:0/1584058078 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fcc140a2c10 0x7fcc140a50d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c009bd0 tx=0x7fcc1c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:47.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.714+0000 7fcc28ff9640 1 -- 172.21.15.28:0/1584058078 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcc240bd420 con 0x7fcc2c1074e0 2024-04-25T01:30:47.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:47 smithi130 ceph-mon[32702]: pgmap v304: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s wr, 1 op/s 2024-04-25T01:30:47.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:47 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3049734976' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:30:47.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:47 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1228569030' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:30:47.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:47 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1054225367' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:30:47.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:47 smithi032 ceph-mon[32461]: pgmap v304: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s wr, 1 op/s 2024-04-25T01:30:47.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:47 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3049734976' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:30:47.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:47 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1228569030' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:30:47.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:47 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1054225367' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:30:47.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.914+0000 7fcc32130640 1 -- 172.21.15.28:0/1584058078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcc2c02d170 con 0x7fcc2c1074e0 2024-04-25T01:30:47.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.914+0000 7fcc28ff9640 1 -- 172.21.15.28:0/1584058078 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) ==== 81+0+591 (secure 0 0 0) 0x7fcc240bcd20 con 0x7fcc2c1074e0 2024-04-25T01:30:47.891 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/2407469300 2024-04-26T01:23:20.244799+0000 2024-04-25T01:30:47.891 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/4112717790 2024-04-26T01:23:20.244799+0000 2024-04-25T01:30:47.892 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/2407469300 2024-04-26T01:23:20.244799+0000 2024-04-25T01:30:47.892 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2222781993 2024-04-26T01:23:20.244799+0000 2024-04-25T01:30:47.892 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/3973667708 2024-04-26T01:22:53.256620+0000 2024-04-25T01:30:47.892 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2339219943 2024-04-26T01:22:53.256620+0000 2024-04-25T01:30:47.892 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/3973667708 2024-04-26T01:22:53.256620+0000 2024-04-25T01:30:47.892 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/747955335 2024-04-26T01:22:53.256620+0000 2024-04-25T01:30:47.892 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/4131336597 2024-04-26T01:22:53.256620+0000 2024-04-25T01:30:47.892 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3232269852 2024-04-26T01:23:20.244799+0000 2024-04-25T01:30:47.892 INFO:teuthology.orchestra.run.smithi028.stderr:listed 10 entries 2024-04-25T01:30:47.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.916+0000 7fcc32130640 1 -- 172.21.15.28:0/1584058078 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fcc140a2c10 msgr2=0x7fcc140a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:47.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.916+0000 7fcc32130640 1 --2- 172.21.15.28:0/1584058078 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fcc140a2c10 0x7fcc140a50d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c009bd0 tx=0x7fcc1c005f90 comp rx=0 tx=0).stop 2024-04-25T01:30:47.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.916+0000 7fcc32130640 1 -- 172.21.15.28:0/1584058078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc2c1074e0 msgr2=0x7fcc2c19f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:47.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.916+0000 7fcc32130640 1 --2- 172.21.15.28:0/1584058078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc2c1074e0 0x7fcc2c19f030 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcc24031ae0 tx=0x7fcc24031ee0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.916+0000 7fcc32130640 1 -- 172.21.15.28:0/1584058078 shutdown_connections 2024-04-25T01:30:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.916+0000 7fcc32130640 1 --2- 172.21.15.28:0/1584058078 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fcc140a2c10 0x7fcc140a50d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.916+0000 7fcc32130640 1 --2- 172.21.15.28:0/1584058078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc2c1074e0 0x7fcc2c19f030 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.916+0000 7fcc32130640 1 --2- 172.21.15.28:0/1584058078 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc2c104bb0 0x7fcc2c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.916+0000 7fcc32130640 1 --2- 172.21.15.28:0/1584058078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc2c102280 0x7fcc2c197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.916+0000 7fcc32130640 1 -- 172.21.15.28:0/1584058078 >> 172.21.15.28:0/1584058078 conn(0x7fcc2c0fbc30 msgr2=0x7fcc2c102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.916+0000 7fcc32130640 1 -- 172.21.15.28:0/1584058078 shutdown_connections 2024-04-25T01:30:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:47.916+0000 7fcc32130640 1 -- 172.21.15.28:0/1584058078 wait complete. 2024-04-25T01:30:47.904 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:30:47.904 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:30:47.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:30:48.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.085+0000 7f229aa6b640 1 -- 172.21.15.28:0/68737210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2294102250 msgr2=0x7f2294108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:48.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.085+0000 7f229aa6b640 1 --2- 172.21.15.28:0/68737210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2294102250 0x7f2294108ae0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f228c009ca0 tx=0x7f228c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.085+0000 7f229aa6b640 1 -- 172.21.15.28:0/68737210 shutdown_connections 2024-04-25T01:30:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.085+0000 7f229aa6b640 1 --2- 172.21.15.28:0/68737210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2294102250 0x7f2294108ae0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.085+0000 7f229aa6b640 1 --2- 172.21.15.28:0/68737210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2294101890 0x7f2294101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.085+0000 7f229aa6b640 1 --2- 172.21.15.28:0/68737210 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f22941006a0 0x7f2294100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.085+0000 7f229aa6b640 1 -- 172.21.15.28:0/68737210 >> 172.21.15.28:0/68737210 conn(0x7f22940fbc30 msgr2=0x7f22940fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.085+0000 7f229aa6b640 1 -- 172.21.15.28:0/68737210 shutdown_connections 2024-04-25T01:30:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.085+0000 7f229aa6b640 1 -- 172.21.15.28:0/68737210 wait complete. 2024-04-25T01:30:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.086+0000 7f229aa6b640 1 Processor -- start 2024-04-25T01:30:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.086+0000 7f229aa6b640 1 -- start start 2024-04-25T01:30:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.086+0000 7f229aa6b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22941006a0 0x7f2294197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.086+0000 7f229aa6b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2294101890 0x7f2294197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.086+0000 7f229aa6b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2294102250 0x7f229419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.086+0000 7f229aa6b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f229410c300 con 0x7f2294101890 2024-04-25T01:30:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.086+0000 7f229aa6b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f229410c180 con 0x7f22941006a0 2024-04-25T01:30:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.086+0000 7f229aa6b640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f229410c480 con 0x7f2294102250 2024-04-25T01:30:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.086+0000 7f2293fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2294101890 0x7f2294197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.086+0000 7f2293fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2294101890 0x7f2294197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60444/0 (socket says 172.21.15.28:60444) 2024-04-25T01:30:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.086+0000 7f2293fff640 1 -- 172.21.15.28:0/1950752166 learned_addr learned my addr 172.21.15.28:0/1950752166 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.086+0000 7f2298807640 1 --2- 172.21.15.28:0/1950752166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22941006a0 0x7f2294197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.086+0000 7f2299008640 1 --2- 172.21.15.28:0/1950752166 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2294102250 0x7f229419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.086+0000 7f2293fff640 1 -- 172.21.15.28:0/1950752166 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2294102250 msgr2=0x7f229419eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.086+0000 7f2293fff640 1 --2- 172.21.15.28:0/1950752166 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2294102250 0x7f229419eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.086+0000 7f2293fff640 1 -- 172.21.15.28:0/1950752166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22941006a0 msgr2=0x7f2294197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.086+0000 7f2293fff640 1 --2- 172.21.15.28:0/1950752166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22941006a0 0x7f2294197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.086+0000 7f2293fff640 1 -- 172.21.15.28:0/1950752166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f229419f660 con 0x7f2294101890 2024-04-25T01:30:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.086+0000 7f2293fff640 1 --2- 172.21.15.28:0/1950752166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2294101890 0x7f2294197f80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f228000e9f0 tx=0x7f228000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.087+0000 7f2291ffb640 1 -- 172.21.15.28:0/1950752166 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228000cd80 con 0x7f2294101890 2024-04-25T01:30:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.087+0000 7f229aa6b640 1 -- 172.21.15.28:0/1950752166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f229419f950 con 0x7f2294101890 2024-04-25T01:30:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.087+0000 7f2291ffb640 1 -- 172.21.15.28:0/1950752166 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2280010910 con 0x7f2294101890 2024-04-25T01:30:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.087+0000 7f2291ffb640 1 -- 172.21.15.28:0/1950752166 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22800188f0 con 0x7f2294101890 2024-04-25T01:30:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.087+0000 7f229aa6b640 1 -- 172.21.15.28:0/1950752166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f229419fea0 con 0x7f2294101890 2024-04-25T01:30:48.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.088+0000 7f229aa6b640 1 -- 172.21.15.28:0/1950752166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22941a0130 con 0x7f2294101890 2024-04-25T01:30:48.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.089+0000 7f2291ffb640 1 -- 172.21.15.28:0/1950752166 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2280010ac0 con 0x7f2294101890 2024-04-25T01:30:48.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.089+0000 7f2291ffb640 1 --2- 172.21.15.28:0/1950752166 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f22840a2b90 0x7f22840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:48.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.090+0000 7f2298807640 1 --2- 172.21.15.28:0/1950752166 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f22840a2b90 0x7f22840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:48.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.090+0000 7f2291ffb640 1 -- 172.21.15.28:0/1950752166 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f22800cf720 con 0x7f2294101890 2024-04-25T01:30:48.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.090+0000 7f2298807640 1 --2- 172.21.15.28:0/1950752166 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f22840a2b90 0x7f22840a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f22941016f0 tx=0x7f228800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.094+0000 7f2291ffb640 1 -- 172.21.15.28:0/1950752166 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f228009c3e0 con 0x7f2294101890 2024-04-25T01:30:48.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.278+0000 7f229aa6b640 1 -- 172.21.15.28:0/1950752166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22941a0820 con 0x7f2294101890 2024-04-25T01:30:48.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.280+0000 7f2291ffb640 1 -- 172.21.15.28:0/1950752166 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24093 (secure 0 0 0) 0x7f228009bce0 con 0x7f2294101890 2024-04-25T01:30:48.257 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:48.258 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:30:38.730329+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:30:48.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.282+0000 7f229aa6b640 1 -- 172.21.15.28:0/1950752166 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f22840a2b90 msgr2=0x7f22840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:48.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.282+0000 7f229aa6b640 1 --2- 172.21.15.28:0/1950752166 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f22840a2b90 0x7f22840a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f22941016f0 tx=0x7f228800b390 comp rx=0 tx=0).stop 2024-04-25T01:30:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.282+0000 7f229aa6b640 1 -- 172.21.15.28:0/1950752166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2294101890 msgr2=0x7f2294197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.282+0000 7f229aa6b640 1 --2- 172.21.15.28:0/1950752166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2294101890 0x7f2294197f80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f228000e9f0 tx=0x7f228000eeb0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.282+0000 7f229aa6b640 1 -- 172.21.15.28:0/1950752166 shutdown_connections 2024-04-25T01:30:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.282+0000 7f229aa6b640 1 --2- 172.21.15.28:0/1950752166 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f22840a2b90 0x7f22840a5050 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.282+0000 7f229aa6b640 1 --2- 172.21.15.28:0/1950752166 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2294102250 0x7f229419eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.282+0000 7f229aa6b640 1 --2- 172.21.15.28:0/1950752166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2294101890 0x7f2294197f80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.282+0000 7f229aa6b640 1 --2- 172.21.15.28:0/1950752166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22941006a0 0x7f2294197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.282+0000 7f229aa6b640 1 -- 172.21.15.28:0/1950752166 >> 172.21.15.28:0/1950752166 conn(0x7f22940fbc30 msgr2=0x7f22940fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.282+0000 7f229aa6b640 1 -- 172.21.15.28:0/1950752166 shutdown_connections 2024-04-25T01:30:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.282+0000 7f229aa6b640 1 -- 172.21.15.28:0/1950752166 wait complete. 2024-04-25T01:30:48.270 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:25:48.767414+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:30:32.282430+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:30:33.127049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:30:36.796132+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:30:48.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:30:48.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.400+0000 7f240bd63640 1 -- 172.21.15.28:0/811694991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24041006a0 msgr2=0x7f2404100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:48.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.400+0000 7f240bd63640 1 --2- 172.21.15.28:0/811694991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24041006a0 0x7f2404100ac0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2400009900 tx=0x7f240002a410 comp rx=0 tx=0).stop 2024-04-25T01:30:48.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.400+0000 7f240bd63640 1 -- 172.21.15.28:0/811694991 shutdown_connections 2024-04-25T01:30:48.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.400+0000 7f240bd63640 1 --2- 172.21.15.28:0/811694991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2404102250 0x7f2404108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.400+0000 7f240bd63640 1 --2- 172.21.15.28:0/811694991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2404101890 0x7f2404101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.400+0000 7f240bd63640 1 --2- 172.21.15.28:0/811694991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24041006a0 0x7f2404100ac0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.400+0000 7f240bd63640 1 -- 172.21.15.28:0/811694991 >> 172.21.15.28:0/811694991 conn(0x7f24040fbc10 msgr2=0x7f24040fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.400+0000 7f240bd63640 1 -- 172.21.15.28:0/811694991 shutdown_connections 2024-04-25T01:30:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.400+0000 7f240bd63640 1 -- 172.21.15.28:0/811694991 wait complete. 2024-04-25T01:30:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.400+0000 7f240bd63640 1 Processor -- start 2024-04-25T01:30:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.400+0000 7f240bd63640 1 -- start start 2024-04-25T01:30:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.400+0000 7f240bd63640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24041006a0 0x7f2404197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.400+0000 7f240bd63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2404101890 0x7f2404198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.400+0000 7f240bd63640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2404102250 0x7f240419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.400+0000 7f240bd63640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f240410c370 con 0x7f2404101890 2024-04-25T01:30:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.400+0000 7f240bd63640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f240410c1f0 con 0x7f2404102250 2024-04-25T01:30:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.400+0000 7f240bd63640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f240410c4f0 con 0x7f24041006a0 2024-04-25T01:30:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.401+0000 7f24092fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2404101890 0x7f2404198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.401+0000 7f24092fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2404101890 0x7f2404198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60462/0 (socket says 172.21.15.28:60462) 2024-04-25T01:30:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.401+0000 7f24092fe640 1 -- 172.21.15.28:0/3575598217 learned_addr learned my addr 172.21.15.28:0/3575598217 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.401+0000 7f2409aff640 1 --2- 172.21.15.28:0/3575598217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24041006a0 0x7f2404197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.401+0000 7f240a300640 1 --2- 172.21.15.28:0/3575598217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2404102250 0x7f240419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.401+0000 7f24092fe640 1 -- 172.21.15.28:0/3575598217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24041006a0 msgr2=0x7f2404197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.401+0000 7f24092fe640 1 --2- 172.21.15.28:0/3575598217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24041006a0 0x7f2404197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.401+0000 7f24092fe640 1 -- 172.21.15.28:0/3575598217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2404102250 msgr2=0x7f240419f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.401+0000 7f24092fe640 1 --2- 172.21.15.28:0/3575598217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2404102250 0x7f240419f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.401+0000 7f24092fe640 1 -- 172.21.15.28:0/3575598217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f240419f720 con 0x7f2404101890 2024-04-25T01:30:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.401+0000 7f2409aff640 1 --2- 172.21.15.28:0/3575598217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24041006a0 0x7f2404197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.401+0000 7f240a300640 1 --2- 172.21.15.28:0/3575598217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2404102250 0x7f240419f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.401+0000 7f24092fe640 1 --2- 172.21.15.28:0/3575598217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2404101890 0x7f2404198040 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f23f400c920 tx=0x7f23f400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.401+0000 7f23faffd640 1 -- 172.21.15.28:0/3575598217 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f4007c70 con 0x7f2404101890 2024-04-25T01:30:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.401+0000 7f23faffd640 1 -- 172.21.15.28:0/3575598217 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23f4007e10 con 0x7f2404101890 2024-04-25T01:30:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.401+0000 7f240bd63640 1 -- 172.21.15.28:0/3575598217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f240419fa10 con 0x7f2404101890 2024-04-25T01:30:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.401+0000 7f23faffd640 1 -- 172.21.15.28:0/3575598217 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f4018890 con 0x7f2404101890 2024-04-25T01:30:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.401+0000 7f240bd63640 1 -- 172.21.15.28:0/3575598217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f240419ffe0 con 0x7f2404101890 2024-04-25T01:30:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.402+0000 7f240bd63640 1 -- 172.21.15.28:0/3575598217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24041a0270 con 0x7f2404101890 2024-04-25T01:30:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.404+0000 7f23faffd640 1 -- 172.21.15.28:0/3575598217 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f23f4018a70 con 0x7f2404101890 2024-04-25T01:30:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.404+0000 7f23faffd640 1 --2- 172.21.15.28:0/3575598217 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f23f00a2b90 0x7f23f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.404+0000 7f2409aff640 1 --2- 172.21.15.28:0/3575598217 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f23f00a2b90 0x7f23f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.404+0000 7f23faffd640 1 -- 172.21.15.28:0/3575598217 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f23f40cf480 con 0x7f2404101890 2024-04-25T01:30:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.405+0000 7f2409aff640 1 --2- 172.21.15.28:0/3575598217 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f23f00a2b90 0x7f23f00a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2400006010 tx=0x7f2400000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.409+0000 7f23faffd640 1 -- 172.21.15.28:0/3575598217 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f23f4014030 con 0x7f2404101890 2024-04-25T01:30:48.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:48 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2934318743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:30:48.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:48 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1584058078' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:30:48.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:48 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1950752166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:48.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.594+0000 7f240bd63640 1 -- 172.21.15.28:0/3575598217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f24041a0440 con 0x7f2404101890 2024-04-25T01:30:48.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.594+0000 7f23faffd640 1 -- 172.21.15.28:0/3575598217 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f23f400de20 con 0x7f2404101890 2024-04-25T01:30:48.572 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T01:30:48.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.596+0000 7f240bd63640 1 -- 172.21.15.28:0/3575598217 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f23f00a2b90 msgr2=0x7f23f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:48.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.596+0000 7f240bd63640 1 --2- 172.21.15.28:0/3575598217 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f23f00a2b90 0x7f23f00a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2400006010 tx=0x7f2400000bc0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.596+0000 7f240bd63640 1 -- 172.21.15.28:0/3575598217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2404101890 msgr2=0x7f2404198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:48.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.596+0000 7f240bd63640 1 --2- 172.21.15.28:0/3575598217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2404101890 0x7f2404198040 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f23f400c920 tx=0x7f23f400cde0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.596+0000 7f240bd63640 1 -- 172.21.15.28:0/3575598217 shutdown_connections 2024-04-25T01:30:48.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.597+0000 7f240bd63640 1 --2- 172.21.15.28:0/3575598217 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f23f00a2b90 0x7f23f00a5050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.597+0000 7f240bd63640 1 --2- 172.21.15.28:0/3575598217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2404102250 0x7f240419f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.597+0000 7f240bd63640 1 --2- 172.21.15.28:0/3575598217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2404101890 0x7f2404198040 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.597+0000 7f240bd63640 1 --2- 172.21.15.28:0/3575598217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f24041006a0 0x7f2404197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.597+0000 7f240bd63640 1 -- 172.21.15.28:0/3575598217 >> 172.21.15.28:0/3575598217 conn(0x7f24040fbc10 msgr2=0x7f24040fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:48.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.597+0000 7f240bd63640 1 -- 172.21.15.28:0/3575598217 shutdown_connections 2024-04-25T01:30:48.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.597+0000 7f240bd63640 1 -- 172.21.15.28:0/3575598217 wait complete. 2024-04-25T01:30:48.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:30:48.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.713+0000 7f85f753b640 1 -- 172.21.15.28:0/3571765528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f0102250 msgr2=0x7f85f0108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:48.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.713+0000 7f85f753b640 1 --2- 172.21.15.28:0/3571765528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f0102250 0x7f85f0108ae0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009ca0 tx=0x7f85e802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.713+0000 7f85f753b640 1 -- 172.21.15.28:0/3571765528 shutdown_connections 2024-04-25T01:30:48.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.713+0000 7f85f753b640 1 --2- 172.21.15.28:0/3571765528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f0102250 0x7f85f0108ae0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.713+0000 7f85f753b640 1 --2- 172.21.15.28:0/3571765528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85f0101890 0x7f85f0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.713+0000 7f85f753b640 1 --2- 172.21.15.28:0/3571765528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85f01006a0 0x7f85f0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.713+0000 7f85f753b640 1 -- 172.21.15.28:0/3571765528 >> 172.21.15.28:0/3571765528 conn(0x7f85f00fbc30 msgr2=0x7f85f00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:48.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.713+0000 7f85f753b640 1 -- 172.21.15.28:0/3571765528 shutdown_connections 2024-04-25T01:30:48.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.713+0000 7f85f753b640 1 -- 172.21.15.28:0/3571765528 wait complete. 2024-04-25T01:30:48.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.713+0000 7f85f753b640 1 Processor -- start 2024-04-25T01:30:48.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.713+0000 7f85f753b640 1 -- start start 2024-04-25T01:30:48.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85f753b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85f01006a0 0x7f85f010b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:48.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85f753b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85f0101890 0x7f85f0109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:48.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85f753b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f0102250 0x7f85f0109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:48.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85f753b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85f010d440 con 0x7f85f0102250 2024-04-25T01:30:48.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85f753b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f85f010d2c0 con 0x7f85f01006a0 2024-04-25T01:30:48.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85f753b640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f85f010d5c0 con 0x7f85f0101890 2024-04-25T01:30:48.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85f5ad8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f0102250 0x7f85f0109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:48.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85f5ad8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f0102250 0x7f85f0109ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60480/0 (socket says 172.21.15.28:60480) 2024-04-25T01:30:48.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85f5ad8640 1 -- 172.21.15.28:0/2792406744 learned_addr learned my addr 172.21.15.28:0/2792406744 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:48.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85f52d7640 1 --2- 172.21.15.28:0/2792406744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85f01006a0 0x7f85f010b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:48.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85f4ad6640 1 --2- 172.21.15.28:0/2792406744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85f0101890 0x7f85f0109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:48.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85f5ad8640 1 -- 172.21.15.28:0/2792406744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85f0101890 msgr2=0x7f85f0109660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:48.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85f5ad8640 1 --2- 172.21.15.28:0/2792406744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85f0101890 0x7f85f0109660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85f5ad8640 1 -- 172.21.15.28:0/2792406744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85f01006a0 msgr2=0x7f85f010b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:48.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85f5ad8640 1 --2- 172.21.15.28:0/2792406744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85f01006a0 0x7f85f010b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85f5ad8640 1 -- 172.21.15.28:0/2792406744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f010a430 con 0x7f85f0102250 2024-04-25T01:30:48.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85f4ad6640 1 --2- 172.21.15.28:0/2792406744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85f0101890 0x7f85f0109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:48.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85f5ad8640 1 --2- 172.21.15.28:0/2792406744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f0102250 0x7f85f0109ba0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f85e8005e30 tx=0x7f85e8031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:48.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85e67fc640 1 -- 172.21.15.28:0/2792406744 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e8003c30 con 0x7f85f0102250 2024-04-25T01:30:48.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.714+0000 7f85e67fc640 1 -- 172.21.15.28:0/2792406744 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85e8003dd0 con 0x7f85f0102250 2024-04-25T01:30:48.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.715+0000 7f85f753b640 1 -- 172.21.15.28:0/2792406744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85f01a1af0 con 0x7f85f0102250 2024-04-25T01:30:48.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.715+0000 7f85e67fc640 1 -- 172.21.15.28:0/2792406744 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e802fde0 con 0x7f85f0102250 2024-04-25T01:30:48.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.715+0000 7f85f753b640 1 -- 172.21.15.28:0/2792406744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85f01a2090 con 0x7f85f0102250 2024-04-25T01:30:48.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.716+0000 7f85f753b640 1 -- 172.21.15.28:0/2792406744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85f0107960 con 0x7f85f0102250 2024-04-25T01:30:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.717+0000 7f85e67fc640 1 -- 172.21.15.28:0/2792406744 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f85e8004050 con 0x7f85f0102250 2024-04-25T01:30:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.717+0000 7f85e67fc640 1 --2- 172.21.15.28:0/2792406744 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f85dc0a2c10 0x7f85dc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.717+0000 7f85f52d7640 1 --2- 172.21.15.28:0/2792406744 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f85dc0a2c10 0x7f85dc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.717+0000 7f85e67fc640 1 -- 172.21.15.28:0/2792406744 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f85e80f06a0 con 0x7f85f0102250 2024-04-25T01:30:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.718+0000 7f85f52d7640 1 --2- 172.21.15.28:0/2792406744 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f85dc0a2c10 0x7f85dc0a50d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f85f01016f0 tx=0x7f85ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:48.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.722+0000 7f85e67fc640 1 -- 172.21.15.28:0/2792406744 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f85e80bd3c0 con 0x7f85f0102250 2024-04-25T01:30:48.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:48 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2934318743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:30:48.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:48 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1584058078' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:30:48.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:48 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1950752166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:48.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:48 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2934318743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:30:48.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:48 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1584058078' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:30:48.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:48 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1950752166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:48.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.907+0000 7f85f753b640 1 -- 172.21.15.28:0/2792406744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f85f0062340 con 0x7f85f0102250 2024-04-25T01:30:48.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.907+0000 7f85e67fc640 1 -- 172.21.15.28:0/2792406744 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f85e80bccc0 con 0x7f85f0102250 2024-04-25T01:30:48.884 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:30:48.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.909+0000 7f85f753b640 1 -- 172.21.15.28:0/2792406744 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f85dc0a2c10 msgr2=0x7f85dc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:48.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.909+0000 7f85f753b640 1 --2- 172.21.15.28:0/2792406744 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f85dc0a2c10 0x7f85dc0a50d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f85f01016f0 tx=0x7f85ec005f90 comp rx=0 tx=0).stop 2024-04-25T01:30:48.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.909+0000 7f85f753b640 1 -- 172.21.15.28:0/2792406744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f0102250 msgr2=0x7f85f0109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:48.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.909+0000 7f85f753b640 1 --2- 172.21.15.28:0/2792406744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f0102250 0x7f85f0109ba0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f85e8005e30 tx=0x7f85e8031ee0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.909+0000 7f85f753b640 1 -- 172.21.15.28:0/2792406744 shutdown_connections 2024-04-25T01:30:48.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.909+0000 7f85f753b640 1 --2- 172.21.15.28:0/2792406744 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f85dc0a2c10 0x7f85dc0a50d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.909+0000 7f85f753b640 1 --2- 172.21.15.28:0/2792406744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f0102250 0x7f85f0109ba0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.909+0000 7f85f753b640 1 --2- 172.21.15.28:0/2792406744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85f0101890 0x7f85f0109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.909+0000 7f85f753b640 1 --2- 172.21.15.28:0/2792406744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85f01006a0 0x7f85f010b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:48.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.909+0000 7f85f753b640 1 -- 172.21.15.28:0/2792406744 >> 172.21.15.28:0/2792406744 conn(0x7f85f00fbc30 msgr2=0x7f85f0104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:48.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.909+0000 7f85f753b640 1 -- 172.21.15.28:0/2792406744 shutdown_connections 2024-04-25T01:30:48.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:48.909+0000 7f85f753b640 1 -- 172.21.15.28:0/2792406744 wait complete. 2024-04-25T01:30:48.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:30:49.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.025+0000 7fbc25da2640 1 -- 172.21.15.28:0/2052780517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc200ffa70 msgr2=0x7fbc200ffe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:49.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.025+0000 7fbc25da2640 1 --2- 172.21.15.28:0/2052780517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc200ffa70 0x7fbc200ffe90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbc10009930 tx=0x7fbc1002a420 comp rx=0 tx=0).stop 2024-04-25T01:30:49.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.025+0000 7fbc25da2640 1 -- 172.21.15.28:0/2052780517 shutdown_connections 2024-04-25T01:30:49.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.025+0000 7fbc25da2640 1 --2- 172.21.15.28:0/2052780517 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc201034a0 0x7fbc20109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.025+0000 7fbc25da2640 1 --2- 172.21.15.28:0/2052780517 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc20102ae0 0x7fbc20102f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.025+0000 7fbc25da2640 1 --2- 172.21.15.28:0/2052780517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc200ffa70 0x7fbc200ffe90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.025+0000 7fbc25da2640 1 -- 172.21.15.28:0/2052780517 >> 172.21.15.28:0/2052780517 conn(0x7fbc200fb380 msgr2=0x7fbc200fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:49.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.025+0000 7fbc25da2640 1 -- 172.21.15.28:0/2052780517 shutdown_connections 2024-04-25T01:30:49.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.025+0000 7fbc25da2640 1 -- 172.21.15.28:0/2052780517 wait complete. 2024-04-25T01:30:49.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.026+0000 7fbc25da2640 1 Processor -- start 2024-04-25T01:30:49.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.026+0000 7fbc25da2640 1 -- start start 2024-04-25T01:30:49.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.026+0000 7fbc25da2640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc200ffa70 0x7fbc20197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:49.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.026+0000 7fbc25da2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc20102ae0 0x7fbc20198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:49.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.026+0000 7fbc25da2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc201034a0 0x7fbc2019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.026+0000 7fbc25da2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2010d420 con 0x7fbc201034a0 2024-04-25T01:30:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.026+0000 7fbc25da2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2010d2a0 con 0x7fbc20102ae0 2024-04-25T01:30:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.026+0000 7fbc25da2640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2010d5a0 con 0x7fbc200ffa70 2024-04-25T01:30:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.026+0000 7fbc1ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc201034a0 0x7fbc2019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.026+0000 7fbc1ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc201034a0 0x7fbc2019f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60500/0 (socket says 172.21.15.28:60500) 2024-04-25T01:30:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.026+0000 7fbc1ffff640 1 -- 172.21.15.28:0/3027608244 learned_addr learned my addr 172.21.15.28:0/3027608244 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.026+0000 7fbc1effd640 1 --2- 172.21.15.28:0/3027608244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc20102ae0 0x7fbc20198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.026+0000 7fbc1f7fe640 1 --2- 172.21.15.28:0/3027608244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc200ffa70 0x7fbc20197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.027+0000 7fbc1ffff640 1 -- 172.21.15.28:0/3027608244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc200ffa70 msgr2=0x7fbc20197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.027+0000 7fbc1ffff640 1 --2- 172.21.15.28:0/3027608244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc200ffa70 0x7fbc20197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.027+0000 7fbc1ffff640 1 -- 172.21.15.28:0/3027608244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc20102ae0 msgr2=0x7fbc20198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.027+0000 7fbc1ffff640 1 --2- 172.21.15.28:0/3027608244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc20102ae0 0x7fbc20198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.027+0000 7fbc1ffff640 1 -- 172.21.15.28:0/3027608244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2019f6f0 con 0x7fbc201034a0 2024-04-25T01:30:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.027+0000 7fbc1ffff640 1 --2- 172.21.15.28:0/3027608244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc201034a0 0x7fbc2019f080 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbc1800eaa0 tx=0x7fbc1800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.027+0000 7fbc1cff9640 1 -- 172.21.15.28:0/3027608244 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1800cdb0 con 0x7fbc201034a0 2024-04-25T01:30:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.027+0000 7fbc1cff9640 1 -- 172.21.15.28:0/3027608244 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc18005020 con 0x7fbc201034a0 2024-04-25T01:30:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.027+0000 7fbc25da2640 1 -- 172.21.15.28:0/3027608244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc2019f9e0 con 0x7fbc201034a0 2024-04-25T01:30:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.027+0000 7fbc1cff9640 1 -- 172.21.15.28:0/3027608244 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc180116b0 con 0x7fbc201034a0 2024-04-25T01:30:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.027+0000 7fbc25da2640 1 -- 172.21.15.28:0/3027608244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc2019ff30 con 0x7fbc201034a0 2024-04-25T01:30:49.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.028+0000 7fbc25da2640 1 -- 172.21.15.28:0/3027608244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc20108c10 con 0x7fbc201034a0 2024-04-25T01:30:49.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.029+0000 7fbc1cff9640 1 -- 172.21.15.28:0/3027608244 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbc18005560 con 0x7fbc201034a0 2024-04-25T01:30:49.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.030+0000 7fbc1cff9640 1 --2- 172.21.15.28:0/3027608244 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbc0c0a2b90 0x7fbc0c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:49.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.030+0000 7fbc1f7fe640 1 --2- 172.21.15.28:0/3027608244 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbc0c0a2b90 0x7fbc0c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:49.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.030+0000 7fbc1cff9640 1 -- 172.21.15.28:0/3027608244 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fbc180cf4a0 con 0x7fbc201034a0 2024-04-25T01:30:49.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.030+0000 7fbc1f7fe640 1 --2- 172.21.15.28:0/3027608244 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbc0c0a2b90 0x7fbc0c0a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbc10031440 tx=0x7fbc10000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:49.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.034+0000 7fbc1cff9640 1 -- 172.21.15.28:0/3027608244 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbc18016030 con 0x7fbc201034a0 2024-04-25T01:30:49.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.218+0000 7fbc25da2640 1 -- 172.21.15.28:0/3027608244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbc20062340 con 0x7fbc201034a0 2024-04-25T01:30:49.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.219+0000 7fbc1cff9640 1 -- 172.21.15.28:0/3027608244 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fbc1809bb10 con 0x7fbc201034a0 2024-04-25T01:30:49.196 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T01:30:49.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.221+0000 7fbc25da2640 1 -- 172.21.15.28:0/3027608244 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbc0c0a2b90 msgr2=0x7fbc0c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:49.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.221+0000 7fbc25da2640 1 --2- 172.21.15.28:0/3027608244 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbc0c0a2b90 0x7fbc0c0a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbc10031440 tx=0x7fbc10000bc0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.221+0000 7fbc25da2640 1 -- 172.21.15.28:0/3027608244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc201034a0 msgr2=0x7fbc2019f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:49.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.221+0000 7fbc25da2640 1 --2- 172.21.15.28:0/3027608244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc201034a0 0x7fbc2019f080 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbc1800eaa0 tx=0x7fbc1800ef60 comp rx=0 tx=0).stop 2024-04-25T01:30:49.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.221+0000 7fbc25da2640 1 -- 172.21.15.28:0/3027608244 shutdown_connections 2024-04-25T01:30:49.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.221+0000 7fbc25da2640 1 --2- 172.21.15.28:0/3027608244 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbc0c0a2b90 0x7fbc0c0a5050 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.221+0000 7fbc25da2640 1 --2- 172.21.15.28:0/3027608244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc201034a0 0x7fbc2019f080 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.221+0000 7fbc25da2640 1 --2- 172.21.15.28:0/3027608244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc20102ae0 0x7fbc20198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.221+0000 7fbc25da2640 1 --2- 172.21.15.28:0/3027608244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc200ffa70 0x7fbc20197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.221+0000 7fbc25da2640 1 -- 172.21.15.28:0/3027608244 >> 172.21.15.28:0/3027608244 conn(0x7fbc200fb380 msgr2=0x7fbc20105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.221+0000 7fbc25da2640 1 -- 172.21.15.28:0/3027608244 shutdown_connections 2024-04-25T01:30:49.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.221+0000 7fbc25da2640 1 -- 172.21.15.28:0/3027608244 wait complete. 2024-04-25T01:30:49.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:30:49.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.336+0000 7fbe67088640 1 -- 172.21.15.28:0/3481902042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe60102250 msgr2=0x7fbe60108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:49.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.336+0000 7fbe67088640 1 --2- 172.21.15.28:0/3481902042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe60102250 0x7fbe60108ae0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbe58009ca0 tx=0x7fbe5802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.336+0000 7fbe67088640 1 -- 172.21.15.28:0/3481902042 shutdown_connections 2024-04-25T01:30:49.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.336+0000 7fbe67088640 1 --2- 172.21.15.28:0/3481902042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe60102250 0x7fbe60108ae0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.336+0000 7fbe67088640 1 --2- 172.21.15.28:0/3481902042 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe60101890 0x7fbe60101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.336+0000 7fbe67088640 1 --2- 172.21.15.28:0/3481902042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe601006a0 0x7fbe60100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.336+0000 7fbe67088640 1 -- 172.21.15.28:0/3481902042 >> 172.21.15.28:0/3481902042 conn(0x7fbe600fbc30 msgr2=0x7fbe600fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.336+0000 7fbe67088640 1 -- 172.21.15.28:0/3481902042 shutdown_connections 2024-04-25T01:30:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.336+0000 7fbe67088640 1 -- 172.21.15.28:0/3481902042 wait complete. 2024-04-25T01:30:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.337+0000 7fbe67088640 1 Processor -- start 2024-04-25T01:30:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.337+0000 7fbe67088640 1 -- start start 2024-04-25T01:30:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.337+0000 7fbe67088640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe601006a0 0x7fbe60197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.337+0000 7fbe67088640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe60101890 0x7fbe60197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.337+0000 7fbe67088640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe60102250 0x7fbe6019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.337+0000 7fbe67088640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6010c300 con 0x7fbe60101890 2024-04-25T01:30:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.337+0000 7fbe67088640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6010c180 con 0x7fbe60102250 2024-04-25T01:30:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.337+0000 7fbe67088640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6010c480 con 0x7fbe601006a0 2024-04-25T01:30:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.337+0000 7fbe5ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe60101890 0x7fbe60197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.337+0000 7fbe5ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe60101890 0x7fbe60197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60512/0 (socket says 172.21.15.28:60512) 2024-04-25T01:30:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.337+0000 7fbe5ffff640 1 -- 172.21.15.28:0/2737038418 learned_addr learned my addr 172.21.15.28:0/2737038418 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.337+0000 7fbe64e24640 1 --2- 172.21.15.28:0/2737038418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe601006a0 0x7fbe60197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.337+0000 7fbe65625640 1 --2- 172.21.15.28:0/2737038418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe60102250 0x7fbe6019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.337+0000 7fbe5ffff640 1 -- 172.21.15.28:0/2737038418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe601006a0 msgr2=0x7fbe60197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.337+0000 7fbe5ffff640 1 --2- 172.21.15.28:0/2737038418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe601006a0 0x7fbe60197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.337+0000 7fbe5ffff640 1 -- 172.21.15.28:0/2737038418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe60102250 msgr2=0x7fbe6019efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.337+0000 7fbe5ffff640 1 --2- 172.21.15.28:0/2737038418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe60102250 0x7fbe6019efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.337+0000 7fbe5ffff640 1 -- 172.21.15.28:0/2737038418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe6019f610 con 0x7fbe60101890 2024-04-25T01:30:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.337+0000 7fbe5ffff640 1 --2- 172.21.15.28:0/2737038418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe60101890 0x7fbe60197f80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c00c920 tx=0x7fbe4c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.338+0000 7fbe5dffb640 1 -- 172.21.15.28:0/2737038418 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c007c70 con 0x7fbe60101890 2024-04-25T01:30:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.338+0000 7fbe5dffb640 1 -- 172.21.15.28:0/2737038418 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe4c007e10 con 0x7fbe60101890 2024-04-25T01:30:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.338+0000 7fbe67088640 1 -- 172.21.15.28:0/2737038418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe6019f900 con 0x7fbe60101890 2024-04-25T01:30:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.338+0000 7fbe5dffb640 1 -- 172.21.15.28:0/2737038418 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c018890 con 0x7fbe60101890 2024-04-25T01:30:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.338+0000 7fbe67088640 1 -- 172.21.15.28:0/2737038418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe6019fe50 con 0x7fbe60101890 2024-04-25T01:30:49.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.339+0000 7fbe67088640 1 -- 172.21.15.28:0/2737038418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe601a00e0 con 0x7fbe60101890 2024-04-25T01:30:49.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.340+0000 7fbe5dffb640 1 -- 172.21.15.28:0/2737038418 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbe4c018a70 con 0x7fbe60101890 2024-04-25T01:30:49.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.341+0000 7fbe5dffb640 1 --2- 172.21.15.28:0/2737038418 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbe500a2b90 0x7fbe500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:49.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.341+0000 7fbe64e24640 1 --2- 172.21.15.28:0/2737038418 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbe500a2b90 0x7fbe500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:49.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.341+0000 7fbe5dffb640 1 -- 172.21.15.28:0/2737038418 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fbe4c0cf320 con 0x7fbe60101890 2024-04-25T01:30:49.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.341+0000 7fbe64e24640 1 --2- 172.21.15.28:0/2737038418 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbe500a2b90 0x7fbe500a5050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbe601016f0 tx=0x7fbe5400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:49.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.346+0000 7fbe5dffb640 1 -- 172.21.15.28:0/2737038418 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbe4c014030 con 0x7fbe60101890 2024-04-25T01:30:49.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.533+0000 7fbe67088640 1 -- 172.21.15.28:0/2737038418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fbe601a02b0 con 0x7fbe60101890 2024-04-25T01:30:49.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.534+0000 7fbe5dffb640 1 -- 172.21.15.28:0/2737038418 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7fbe4c09c040 con 0x7fbe60101890 2024-04-25T01:30:49.511 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:30:49.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.535+0000 7fbe67088640 1 -- 172.21.15.28:0/2737038418 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbe500a2b90 msgr2=0x7fbe500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:49.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.535+0000 7fbe67088640 1 --2- 172.21.15.28:0/2737038418 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbe500a2b90 0x7fbe500a5050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbe601016f0 tx=0x7fbe5400b390 comp rx=0 tx=0).stop 2024-04-25T01:30:49.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.536+0000 7fbe67088640 1 -- 172.21.15.28:0/2737038418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe60101890 msgr2=0x7fbe60197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:49.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.536+0000 7fbe67088640 1 --2- 172.21.15.28:0/2737038418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe60101890 0x7fbe60197f80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c00c920 tx=0x7fbe4c00cde0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.536+0000 7fbe67088640 1 -- 172.21.15.28:0/2737038418 shutdown_connections 2024-04-25T01:30:49.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.536+0000 7fbe67088640 1 --2- 172.21.15.28:0/2737038418 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbe500a2b90 0x7fbe500a5050 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.536+0000 7fbe67088640 1 --2- 172.21.15.28:0/2737038418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe60102250 0x7fbe6019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.536+0000 7fbe67088640 1 --2- 172.21.15.28:0/2737038418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe60101890 0x7fbe60197f80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.536+0000 7fbe67088640 1 --2- 172.21.15.28:0/2737038418 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe601006a0 0x7fbe60197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.536+0000 7fbe67088640 1 -- 172.21.15.28:0/2737038418 >> 172.21.15.28:0/2737038418 conn(0x7fbe600fbc30 msgr2=0x7fbe600fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.536+0000 7fbe67088640 1 -- 172.21.15.28:0/2737038418 shutdown_connections 2024-04-25T01:30:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.536+0000 7fbe67088640 1 -- 172.21.15.28:0/2737038418 wait complete. 2024-04-25T01:30:49.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:49 smithi028 ceph-mon[29390]: pgmap v305: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 513 B/s wr, 1 op/s 2024-04-25T01:30:49.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:49 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3575598217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:30:49.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:49 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2792406744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:30:49.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:49 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3027608244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:30:49.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:30:49.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.649+0000 7f9e0f615640 1 -- 172.21.15.28:0/1973829240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e08103350 msgr2=0x7f9e08109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:49.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.649+0000 7f9e0f615640 1 --2- 172.21.15.28:0/1973829240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e08103350 0x7f9e08109be0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009ca0 tx=0x7f9e0002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.649+0000 7f9e0f615640 1 -- 172.21.15.28:0/1973829240 shutdown_connections 2024-04-25T01:30:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.649+0000 7f9e0f615640 1 --2- 172.21.15.28:0/1973829240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e08103350 0x7f9e08109be0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.649+0000 7f9e0f615640 1 --2- 172.21.15.28:0/1973829240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e08102990 0x7f9e08102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.649+0000 7f9e0f615640 1 --2- 172.21.15.28:0/1973829240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e081017a0 0x7f9e08101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.649+0000 7f9e0f615640 1 -- 172.21.15.28:0/1973829240 >> 172.21.15.28:0/1973829240 conn(0x7f9e080fcd10 msgr2=0x7f9e080ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.650+0000 7f9e0f615640 1 -- 172.21.15.28:0/1973829240 shutdown_connections 2024-04-25T01:30:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.650+0000 7f9e0f615640 1 -- 172.21.15.28:0/1973829240 wait complete. 2024-04-25T01:30:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.650+0000 7f9e0f615640 1 Processor -- start 2024-04-25T01:30:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.650+0000 7f9e0f615640 1 -- start start 2024-04-25T01:30:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.650+0000 7f9e0f615640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e081017a0 0x7f9e08197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.650+0000 7f9e0f615640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e08102990 0x7f9e08198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:49.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.650+0000 7f9e0f615640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e08103350 0x7f9e0819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.650+0000 7f9e0f615640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0810d420 con 0x7f9e081017a0 2024-04-25T01:30:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.650+0000 7f9e0f615640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0810d2a0 con 0x7f9e08103350 2024-04-25T01:30:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.650+0000 7f9e0f615640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0810d5a0 con 0x7f9e08102990 2024-04-25T01:30:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.650+0000 7f9e0d3b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e081017a0 0x7f9e08197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.650+0000 7f9e0d3b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e081017a0 0x7f9e08197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60526/0 (socket says 172.21.15.28:60526) 2024-04-25T01:30:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.650+0000 7f9e0d3b1640 1 -- 172.21.15.28:0/1248464543 learned_addr learned my addr 172.21.15.28:0/1248464543 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.651+0000 7f9e0cbb0640 1 --2- 172.21.15.28:0/1248464543 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e08102990 0x7f9e08198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.651+0000 7f9e0dbb2640 1 --2- 172.21.15.28:0/1248464543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e08103350 0x7f9e0819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.651+0000 7f9e0d3b1640 1 -- 172.21.15.28:0/1248464543 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e08102990 msgr2=0x7f9e08198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.651+0000 7f9e0d3b1640 1 --2- 172.21.15.28:0/1248464543 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e08102990 0x7f9e08198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.651+0000 7f9e0d3b1640 1 -- 172.21.15.28:0/1248464543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e08103350 msgr2=0x7f9e0819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.651+0000 7f9e0d3b1640 1 --2- 172.21.15.28:0/1248464543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e08103350 0x7f9e0819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.651+0000 7f9e0d3b1640 1 -- 172.21.15.28:0/1248464543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e0819f6e0 con 0x7f9e081017a0 2024-04-25T01:30:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.651+0000 7f9e0dbb2640 1 --2- 172.21.15.28:0/1248464543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e08103350 0x7f9e0819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.651+0000 7f9e0d3b1640 1 --2- 172.21.15.28:0/1248464543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e081017a0 0x7f9e08197ac0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400d950 tx=0x7f9e0400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.651+0000 7f9dfe7fc640 1 -- 172.21.15.28:0/1248464543 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e04017070 con 0x7f9e081017a0 2024-04-25T01:30:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.651+0000 7f9dfe7fc640 1 -- 172.21.15.28:0/1248464543 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e04009c30 con 0x7f9e081017a0 2024-04-25T01:30:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.651+0000 7f9e0f615640 1 -- 172.21.15.28:0/1248464543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e0819f9d0 con 0x7f9e081017a0 2024-04-25T01:30:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.651+0000 7f9dfe7fc640 1 -- 172.21.15.28:0/1248464543 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0400f600 con 0x7f9e081017a0 2024-04-25T01:30:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.651+0000 7f9e0f615640 1 -- 172.21.15.28:0/1248464543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e0819fed0 con 0x7f9e081017a0 2024-04-25T01:30:49.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.652+0000 7f9e0f615640 1 -- 172.21.15.28:0/1248464543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e08108a60 con 0x7f9e081017a0 2024-04-25T01:30:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.653+0000 7f9dfe7fc640 1 -- 172.21.15.28:0/1248464543 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9e0401a070 con 0x7f9e081017a0 2024-04-25T01:30:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.654+0000 7f9dfe7fc640 1 --2- 172.21.15.28:0/1248464543 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9df40a2b90 0x7f9df40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.654+0000 7f9e0cbb0640 1 --2- 172.21.15.28:0/1248464543 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9df40a2b90 0x7f9df40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.654+0000 7f9dfe7fc640 1 -- 172.21.15.28:0/1248464543 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f9e040ced10 con 0x7f9e081017a0 2024-04-25T01:30:49.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.654+0000 7f9e0cbb0640 1 --2- 172.21.15.28:0/1248464543 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9df40a2b90 0x7f9df40a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9e08198fd0 tx=0x7f9df8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:49.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.658+0000 7f9dfe7fc640 1 -- 172.21.15.28:0/1248464543 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9e04009de0 con 0x7f9e081017a0 2024-04-25T01:30:49.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:49 smithi130 ceph-mon[32702]: pgmap v305: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 513 B/s wr, 1 op/s 2024-04-25T01:30:49.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:49 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3575598217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:30:49.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:49 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2792406744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:30:49.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:49 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3027608244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:30:49.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:49 smithi032 ceph-mon[32461]: pgmap v305: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 513 B/s wr, 1 op/s 2024-04-25T01:30:49.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:49 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3575598217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:30:49.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:49 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2792406744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:30:49.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:49 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3027608244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:30:49.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.860+0000 7f9e0f615640 1 -- 172.21.15.28:0/1248464543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9e0802d170 con 0x7f9e081017a0 2024-04-25T01:30:49.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.860+0000 7f9dfe7fc640 1 -- 172.21.15.28:0/1248464543 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) ==== 81+0+591 (secure 0 0 0) 0x7f9e0409ba30 con 0x7f9e081017a0 2024-04-25T01:30:49.838 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/2407469300 2024-04-26T01:23:20.244799+0000 2024-04-25T01:30:49.838 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/4112717790 2024-04-26T01:23:20.244799+0000 2024-04-25T01:30:49.838 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/2407469300 2024-04-26T01:23:20.244799+0000 2024-04-25T01:30:49.838 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2222781993 2024-04-26T01:23:20.244799+0000 2024-04-25T01:30:49.838 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/3973667708 2024-04-26T01:22:53.256620+0000 2024-04-25T01:30:49.838 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2339219943 2024-04-26T01:22:53.256620+0000 2024-04-25T01:30:49.838 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/3973667708 2024-04-26T01:22:53.256620+0000 2024-04-25T01:30:49.838 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/747955335 2024-04-26T01:22:53.256620+0000 2024-04-25T01:30:49.838 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/4131336597 2024-04-26T01:22:53.256620+0000 2024-04-25T01:30:49.838 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3232269852 2024-04-26T01:23:20.244799+0000 2024-04-25T01:30:49.839 INFO:teuthology.orchestra.run.smithi028.stderr:listed 10 entries 2024-04-25T01:30:49.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.863+0000 7f9e0f615640 1 -- 172.21.15.28:0/1248464543 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9df40a2b90 msgr2=0x7f9df40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:49.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.863+0000 7f9e0f615640 1 --2- 172.21.15.28:0/1248464543 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9df40a2b90 0x7f9df40a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9e08198fd0 tx=0x7f9df8005f90 comp rx=0 tx=0).stop 2024-04-25T01:30:49.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.863+0000 7f9e0f615640 1 -- 172.21.15.28:0/1248464543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e081017a0 msgr2=0x7f9e08197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:49.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.863+0000 7f9e0f615640 1 --2- 172.21.15.28:0/1248464543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e081017a0 0x7f9e08197ac0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400d950 tx=0x7f9e0400de10 comp rx=0 tx=0).stop 2024-04-25T01:30:49.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.863+0000 7f9e0f615640 1 -- 172.21.15.28:0/1248464543 shutdown_connections 2024-04-25T01:30:49.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.863+0000 7f9e0f615640 1 --2- 172.21.15.28:0/1248464543 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9df40a2b90 0x7f9df40a5050 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.863+0000 7f9e0f615640 1 --2- 172.21.15.28:0/1248464543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e08103350 0x7f9e0819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.863+0000 7f9e0f615640 1 --2- 172.21.15.28:0/1248464543 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9e08102990 0x7f9e08198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.863+0000 7f9e0f615640 1 --2- 172.21.15.28:0/1248464543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e081017a0 0x7f9e08197ac0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:49.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.863+0000 7f9e0f615640 1 -- 172.21.15.28:0/1248464543 >> 172.21.15.28:0/1248464543 conn(0x7f9e080fcd10 msgr2=0x7f9e08105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:49.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.863+0000 7f9e0f615640 1 -- 172.21.15.28:0/1248464543 shutdown_connections 2024-04-25T01:30:49.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:49.863+0000 7f9e0f615640 1 -- 172.21.15.28:0/1248464543 wait complete. 2024-04-25T01:30:49.851 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi028.front.sepia.ceph.com... 2024-04-25T01:30:49.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:30:49.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-25T01:30:49.873 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[71516]: fuse finished with error 0 and tester_r 0 2024-04-25T01:30:49.875 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T01:30:50.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:50 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2737038418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:30:50.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:50 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1248464543' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:30:50.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:50 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2737038418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:30:50.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:50 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1248464543' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:30:50.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:50 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2737038418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:30:50.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:50 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1248464543' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:30:51.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:51 smithi130 ceph-mon[32702]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 532 B/s wr, 0 op/s 2024-04-25T01:30:51.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:51 smithi032 ceph-mon[32461]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 532 B/s wr, 0 op/s 2024-04-25T01:30:51.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:51 smithi028 ceph-mon[29390]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 532 B/s wr, 0 op/s 2024-04-25T01:30:53.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:53 smithi130 ceph-mon[32702]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-25T01:30:53.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:53 smithi032 ceph-mon[32461]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-25T01:30:53.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:53 smithi028 ceph-mon[29390]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-25T01:30:55.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:55 smithi130 ceph-mon[32702]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T01:30:55.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:55 smithi032 ceph-mon[32461]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T01:30:55.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:55 smithi028 ceph-mon[29390]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T01:30:55.977 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T01:30:55.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:30:55.977 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-25T01:30:56.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:30:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.172+0000 7f9893e46640 1 -- 172.21.15.28:0/2480732818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f988c105330 msgr2=0x7f988c1057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.172+0000 7f9893e46640 1 --2- 172.21.15.28:0/2480732818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f988c105330 0x7f988c1057b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9884009ca0 tx=0x7f988402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.172+0000 7f9893e46640 1 -- 172.21.15.28:0/2480732818 shutdown_connections 2024-04-25T01:30:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.172+0000 7f9893e46640 1 --2- 172.21.15.28:0/2480732818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f988c105330 0x7f988c1057b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.172+0000 7f9893e46640 1 --2- 172.21.15.28:0/2480732818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f988c102980 0x7f988c104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.172+0000 7f9893e46640 1 --2- 172.21.15.28:0/2480732818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f988c100010 0x7f988c102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.172+0000 7f9893e46640 1 -- 172.21.15.28:0/2480732818 >> 172.21.15.28:0/2480732818 conn(0x7f988c0fbc30 msgr2=0x7f988c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.172+0000 7f9893e46640 1 -- 172.21.15.28:0/2480732818 shutdown_connections 2024-04-25T01:30:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.173+0000 7f9893e46640 1 -- 172.21.15.28:0/2480732818 wait complete. 2024-04-25T01:30:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.173+0000 7f9893e46640 1 Processor -- start 2024-04-25T01:30:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.173+0000 7f9893e46640 1 -- start start 2024-04-25T01:30:56.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.173+0000 7f9893e46640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f988c100010 0x7f988c19e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.173+0000 7f9893e46640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f988c102980 0x7f988c19e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.173+0000 7f9893e46640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f988c105330 0x7f988c1a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.173+0000 7f9893e46640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f988c113ae0 con 0x7f988c100010 2024-04-25T01:30:56.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.173+0000 7f9893e46640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f988c113960 con 0x7f988c102980 2024-04-25T01:30:56.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.173+0000 7f9893e46640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f988c113c60 con 0x7f988c105330 2024-04-25T01:30:56.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.173+0000 7f9891be2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f988c100010 0x7f988c19e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.173+0000 7f9891be2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f988c100010 0x7f988c19e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60532/0 (socket says 172.21.15.28:60532) 2024-04-25T01:30:56.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.173+0000 7f9891be2640 1 -- 172.21.15.28:0/3631503925 learned_addr learned my addr 172.21.15.28:0/3631503925 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:56.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.174+0000 7f98913e1640 1 --2- 172.21.15.28:0/3631503925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f988c102980 0x7f988c19e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.174+0000 7f98923e3640 1 --2- 172.21.15.28:0/3631503925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f988c105330 0x7f988c1a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.174+0000 7f9891be2640 1 -- 172.21.15.28:0/3631503925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f988c105330 msgr2=0x7f988c1a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.174+0000 7f9891be2640 1 --2- 172.21.15.28:0/3631503925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f988c105330 0x7f988c1a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.174+0000 7f9891be2640 1 -- 172.21.15.28:0/3631503925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f988c102980 msgr2=0x7f988c19e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.174+0000 7f9891be2640 1 --2- 172.21.15.28:0/3631503925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f988c102980 0x7f988c19e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.174+0000 7f9891be2640 1 -- 172.21.15.28:0/3631503925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988c1a5da0 con 0x7f988c100010 2024-04-25T01:30:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.174+0000 7f98913e1640 1 --2- 172.21.15.28:0/3631503925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f988c102980 0x7f988c19e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.174+0000 7f9891be2640 1 --2- 172.21.15.28:0/3631503925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f988c100010 0x7f988c19e180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f988800b4a0 tx=0x7f988800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.174+0000 7f9882ffd640 1 -- 172.21.15.28:0/3631503925 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9888014070 con 0x7f988c100010 2024-04-25T01:30:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.174+0000 7f9893e46640 1 -- 172.21.15.28:0/3631503925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f988c1a6090 con 0x7f988c100010 2024-04-25T01:30:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.174+0000 7f9882ffd640 1 -- 172.21.15.28:0/3631503925 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9888010af0 con 0x7f988c100010 2024-04-25T01:30:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.174+0000 7f9882ffd640 1 -- 172.21.15.28:0/3631503925 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9888017d10 con 0x7f988c100010 2024-04-25T01:30:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.174+0000 7f9893e46640 1 -- 172.21.15.28:0/3631503925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f988c1a6660 con 0x7f988c100010 2024-04-25T01:30:56.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.175+0000 7f9893e46640 1 -- 172.21.15.28:0/3631503925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f988c0fd720 con 0x7f988c100010 2024-04-25T01:30:56.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.176+0000 7f9882ffd640 1 -- 172.21.15.28:0/3631503925 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9888017070 con 0x7f988c100010 2024-04-25T01:30:56.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.177+0000 7f9882ffd640 1 --2- 172.21.15.28:0/3631503925 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f98780a2b90 0x7f98780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.177+0000 7f98913e1640 1 --2- 172.21.15.28:0/3631503925 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f98780a2b90 0x7f98780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.177+0000 7f9882ffd640 1 -- 172.21.15.28:0/3631503925 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f98880ce760 con 0x7f988c100010 2024-04-25T01:30:56.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.177+0000 7f98913e1640 1 --2- 172.21.15.28:0/3631503925 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f98780a2b90 0x7f98780a5050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f988c19f690 tx=0x7f987c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:56.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.182+0000 7f9882ffd640 1 -- 172.21.15.28:0/3631503925 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f988809b420 con 0x7f988c100010 2024-04-25T01:30:56.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.368+0000 7f9893e46640 1 -- 172.21.15.28:0/3631503925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f988c062340 con 0x7f988c100010 2024-04-25T01:30:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.369+0000 7f9882ffd640 1 -- 172.21.15.28:0/3631503925 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24093 (secure 0 0 0) 0x7f988809ad20 con 0x7f988c100010 2024-04-25T01:30:56.347 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:56.348 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:30:38.730329+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:30:56.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.371+0000 7f9893e46640 1 -- 172.21.15.28:0/3631503925 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f98780a2b90 msgr2=0x7f98780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.371+0000 7f9893e46640 1 --2- 172.21.15.28:0/3631503925 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f98780a2b90 0x7f98780a5050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f988c19f690 tx=0x7f987c008040 comp rx=0 tx=0).stop 2024-04-25T01:30:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.371+0000 7f9893e46640 1 -- 172.21.15.28:0/3631503925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f988c100010 msgr2=0x7f988c19e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.371+0000 7f9893e46640 1 --2- 172.21.15.28:0/3631503925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f988c100010 0x7f988c19e180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f988800b4a0 tx=0x7f988800b960 comp rx=0 tx=0).stop 2024-04-25T01:30:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.371+0000 7f9893e46640 1 -- 172.21.15.28:0/3631503925 shutdown_connections 2024-04-25T01:30:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.371+0000 7f9893e46640 1 --2- 172.21.15.28:0/3631503925 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f98780a2b90 0x7f98780a5050 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.371+0000 7f9893e46640 1 --2- 172.21.15.28:0/3631503925 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f988c105330 0x7f988c1a5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.371+0000 7f9893e46640 1 --2- 172.21.15.28:0/3631503925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f988c102980 0x7f988c19e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.371+0000 7f9893e46640 1 --2- 172.21.15.28:0/3631503925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f988c100010 0x7f988c19e180 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.371+0000 7f9893e46640 1 -- 172.21.15.28:0/3631503925 >> 172.21.15.28:0/3631503925 conn(0x7f988c0fbc30 msgr2=0x7f988c100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.371+0000 7f9893e46640 1 -- 172.21.15.28:0/3631503925 shutdown_connections 2024-04-25T01:30:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.371+0000 7f9893e46640 1 -- 172.21.15.28:0/3631503925 wait complete. 2024-04-25T01:30:56.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:30:56.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.486+0000 7f861c109640 1 -- 172.21.15.28:0/3412996983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8614100f70 msgr2=0x7f86141013f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.486+0000 7f861c109640 1 --2- 172.21.15.28:0/3412996983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8614100f70 0x7f86141013f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8604009900 tx=0x7f860402a410 comp rx=0 tx=0).stop 2024-04-25T01:30:56.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.486+0000 7f861c109640 1 -- 172.21.15.28:0/3412996983 shutdown_connections 2024-04-25T01:30:56.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.486+0000 7f861c109640 1 --2- 172.21.15.28:0/3412996983 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8614101930 0x7f8614108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.486+0000 7f861c109640 1 --2- 172.21.15.28:0/3412996983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8614100f70 0x7f86141013f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.486+0000 7f861c109640 1 --2- 172.21.15.28:0/3412996983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86140ffe20 0x7f8614100240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.486+0000 7f861c109640 1 -- 172.21.15.28:0/3412996983 >> 172.21.15.28:0/3412996983 conn(0x7f86140fb3b0 msgr2=0x7f86140fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.486+0000 7f861c109640 1 -- 172.21.15.28:0/3412996983 shutdown_connections 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.486+0000 7f861c109640 1 -- 172.21.15.28:0/3412996983 wait complete. 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.487+0000 7f861c109640 1 Processor -- start 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.487+0000 7f861c109640 1 -- start start 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.487+0000 7f861c109640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86140ffe20 0x7f8614196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.487+0000 7f861c109640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8614100f70 0x7f8614196f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.487+0000 7f861c109640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8614101930 0x7f861419dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.487+0000 7f861c109640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f861410baa0 con 0x7f8614100f70 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.487+0000 7f861c109640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f861410b920 con 0x7f86140ffe20 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.487+0000 7f861c109640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f861410bc20 con 0x7f8614101930 2024-04-25T01:30:56.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.487+0000 7f86196a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8614100f70 0x7f8614196f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.487+0000 7f86196a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8614100f70 0x7f8614196f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60548/0 (socket says 172.21.15.28:60548) 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.487+0000 7f86196a4640 1 -- 172.21.15.28:0/428654100 learned_addr learned my addr 172.21.15.28:0/428654100 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.487+0000 7f8619ea5640 1 --2- 172.21.15.28:0/428654100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86140ffe20 0x7f8614196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.487+0000 7f861a6a6640 1 --2- 172.21.15.28:0/428654100 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8614101930 0x7f861419dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.487+0000 7f86196a4640 1 -- 172.21.15.28:0/428654100 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8614101930 msgr2=0x7f861419dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.487+0000 7f86196a4640 1 --2- 172.21.15.28:0/428654100 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8614101930 0x7f861419dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.487+0000 7f86196a4640 1 -- 172.21.15.28:0/428654100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86140ffe20 msgr2=0x7f8614196a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.487+0000 7f86196a4640 1 --2- 172.21.15.28:0/428654100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86140ffe20 0x7f8614196a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.487+0000 7f86196a4640 1 -- 172.21.15.28:0/428654100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f861419e610 con 0x7f8614100f70 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.487+0000 7f86196a4640 1 --2- 172.21.15.28:0/428654100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8614100f70 0x7f8614196f80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8604031610 tx=0x7f86040043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.488+0000 7f860affd640 1 -- 172.21.15.28:0/428654100 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860403f030 con 0x7f8614100f70 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.488+0000 7f860affd640 1 -- 172.21.15.28:0/428654100 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8604003ce0 con 0x7f8614100f70 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.488+0000 7f861c109640 1 -- 172.21.15.28:0/428654100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f861419e8a0 con 0x7f8614100f70 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.488+0000 7f860affd640 1 -- 172.21.15.28:0/428654100 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604035070 con 0x7f8614100f70 2024-04-25T01:30:56.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.488+0000 7f861c109640 1 -- 172.21.15.28:0/428654100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f861419ede0 con 0x7f8614100f70 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.489+0000 7f861c109640 1 -- 172.21.15.28:0/428654100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86141078f0 con 0x7f8614100f70 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.490+0000 7f860affd640 1 -- 172.21.15.28:0/428654100 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8604042030 con 0x7f8614100f70 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.490+0000 7f860affd640 1 --2- 172.21.15.28:0/428654100 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f86000a2c10 0x7f86000a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.490+0000 7f8619ea5640 1 --2- 172.21.15.28:0/428654100 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f86000a2c10 0x7f86000a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.490+0000 7f860affd640 1 -- 172.21.15.28:0/428654100 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f86040ef920 con 0x7f8614100f70 2024-04-25T01:30:56.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.491+0000 7f8619ea5640 1 --2- 172.21.15.28:0/428654100 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f86000a2c10 0x7f86000a50d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f86100058c0 tx=0x7f861000a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:56.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.495+0000 7f860affd640 1 -- 172.21.15.28:0/428654100 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f86040bc640 con 0x7f8614100f70 2024-04-25T01:30:56.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.705+0000 7f861c109640 1 -- 172.21.15.28:0/428654100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8614062340 con 0x7f8614100f70 2024-04-25T01:30:56.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.707+0000 7f860affd640 1 -- 172.21.15.28:0/428654100 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24093 (secure 0 0 0) 0x7f86040bbf40 con 0x7f8614100f70 2024-04-25T01:30:56.685 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:30:56.685 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:30:38.730329+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:30:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.709+0000 7f861c109640 1 -- 172.21.15.28:0/428654100 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f86000a2c10 msgr2=0x7f86000a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.709+0000 7f861c109640 1 --2- 172.21.15.28:0/428654100 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f86000a2c10 0x7f86000a50d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f86100058c0 tx=0x7f861000a4c0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.709+0000 7f861c109640 1 -- 172.21.15.28:0/428654100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8614100f70 msgr2=0x7f8614196f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.709+0000 7f861c109640 1 --2- 172.21.15.28:0/428654100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8614100f70 0x7f8614196f80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8604031610 tx=0x7f86040043b0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.709+0000 7f861c109640 1 -- 172.21.15.28:0/428654100 shutdown_connections 2024-04-25T01:30:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.709+0000 7f861c109640 1 --2- 172.21.15.28:0/428654100 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f86000a2c10 0x7f86000a50d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.709+0000 7f861c109640 1 --2- 172.21.15.28:0/428654100 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8614101930 0x7f861419dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.709+0000 7f861c109640 1 --2- 172.21.15.28:0/428654100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8614100f70 0x7f8614196f80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.709+0000 7f861c109640 1 --2- 172.21.15.28:0/428654100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86140ffe20 0x7f8614196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.709+0000 7f861c109640 1 -- 172.21.15.28:0/428654100 >> 172.21.15.28:0/428654100 conn(0x7f86140fb3b0 msgr2=0x7f8614104ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:56.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.709+0000 7f861c109640 1 -- 172.21.15.28:0/428654100 shutdown_connections 2024-04-25T01:30:56.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.709+0000 7f861c109640 1 -- 172.21.15.28:0/428654100 wait complete. 2024-04-25T01:30:56.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:56 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3631503925' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:56.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T01:30:56.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:56 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3631503925' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:56.733 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:56 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3631503925' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.825+0000 7f5b838bd640 1 -- 172.21.15.28:0/3908748761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b7c102ad0 msgr2=0x7f5b7c102f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.825+0000 7f5b838bd640 1 --2- 172.21.15.28:0/3908748761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b7c102ad0 0x7f5b7c102f50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c009900 tx=0x7f5b6c02a400 comp rx=0 tx=0).stop 2024-04-25T01:30:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.825+0000 7f5b838bd640 1 -- 172.21.15.28:0/3908748761 shutdown_connections 2024-04-25T01:30:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.825+0000 7f5b838bd640 1 --2- 172.21.15.28:0/3908748761 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b7c103490 0x7f5b7c109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.825+0000 7f5b838bd640 1 --2- 172.21.15.28:0/3908748761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b7c102ad0 0x7f5b7c102f50 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.825+0000 7f5b838bd640 1 --2- 172.21.15.28:0/3908748761 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b7c0ffa60 0x7f5b7c0ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.825+0000 7f5b838bd640 1 -- 172.21.15.28:0/3908748761 >> 172.21.15.28:0/3908748761 conn(0x7f5b7c0fb370 msgr2=0x7f5b7c0fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.825+0000 7f5b838bd640 1 -- 172.21.15.28:0/3908748761 shutdown_connections 2024-04-25T01:30:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.825+0000 7f5b838bd640 1 -- 172.21.15.28:0/3908748761 wait complete. 2024-04-25T01:30:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b838bd640 1 Processor -- start 2024-04-25T01:30:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b838bd640 1 -- start start 2024-04-25T01:30:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b838bd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b7c0ffa60 0x7f5b7c197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b838bd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b7c102ad0 0x7f5b7c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b838bd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b7c103490 0x7f5b7c19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b838bd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c10d490 con 0x7f5b7c102ad0 2024-04-25T01:30:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b838bd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c10d310 con 0x7f5b7c0ffa60 2024-04-25T01:30:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b838bd640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7c10d610 con 0x7f5b7c103490 2024-04-25T01:30:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b80e58640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b7c102ad0 0x7f5b7c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b80e58640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b7c102ad0 0x7f5b7c197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51910/0 (socket says 172.21.15.28:51910) 2024-04-25T01:30:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b80e58640 1 -- 172.21.15.28:0/1386425485 learned_addr learned my addr 172.21.15.28:0/1386425485 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b81e5a640 1 --2- 172.21.15.28:0/1386425485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b7c103490 0x7f5b7c19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b81659640 1 --2- 172.21.15.28:0/1386425485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b7c0ffa60 0x7f5b7c197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b80e58640 1 -- 172.21.15.28:0/1386425485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b7c103490 msgr2=0x7f5b7c19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b80e58640 1 --2- 172.21.15.28:0/1386425485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b7c103490 0x7f5b7c19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b80e58640 1 -- 172.21.15.28:0/1386425485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b7c0ffa60 msgr2=0x7f5b7c197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b80e58640 1 --2- 172.21.15.28:0/1386425485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b7c0ffa60 0x7f5b7c197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b80e58640 1 -- 172.21.15.28:0/1386425485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b7c19f6d0 con 0x7f5b7c102ad0 2024-04-25T01:30:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b81e5a640 1 --2- 172.21.15.28:0/1386425485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b7c103490 0x7f5b7c19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.826+0000 7f5b80e58640 1 --2- 172.21.15.28:0/1386425485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b7c102ad0 0x7f5b7c197ff0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c031440 tx=0x7f5b6c032e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.827+0000 7f5b727fc640 1 -- 172.21.15.28:0/1386425485 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c03f030 con 0x7f5b7c102ad0 2024-04-25T01:30:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.827+0000 7f5b727fc640 1 -- 172.21.15.28:0/1386425485 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b6c003ce0 con 0x7f5b7c102ad0 2024-04-25T01:30:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.827+0000 7f5b727fc640 1 -- 172.21.15.28:0/1386425485 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c035070 con 0x7f5b7c102ad0 2024-04-25T01:30:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.827+0000 7f5b838bd640 1 -- 172.21.15.28:0/1386425485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b7c19f960 con 0x7f5b7c102ad0 2024-04-25T01:30:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.827+0000 7f5b838bd640 1 -- 172.21.15.28:0/1386425485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b7c19fe50 con 0x7f5b7c102ad0 2024-04-25T01:30:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.828+0000 7f5b838bd640 1 -- 172.21.15.28:0/1386425485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b7c108c00 con 0x7f5b7c102ad0 2024-04-25T01:30:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.829+0000 7f5b727fc640 1 -- 172.21.15.28:0/1386425485 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5b6c042030 con 0x7f5b7c102ad0 2024-04-25T01:30:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.829+0000 7f5b727fc640 1 --2- 172.21.15.28:0/1386425485 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5b680a2b90 0x7f5b680a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.830+0000 7f5b727fc640 1 -- 172.21.15.28:0/1386425485 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f5b6c0efd60 con 0x7f5b7c102ad0 2024-04-25T01:30:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.830+0000 7f5b81659640 1 --2- 172.21.15.28:0/1386425485 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5b680a2b90 0x7f5b680a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.830+0000 7f5b81659640 1 --2- 172.21.15.28:0/1386425485 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5b680a2b90 0x7f5b680a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5b78005fd0 tx=0x7f5b78005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:56.833+0000 7f5b727fc640 1 -- 172.21.15.28:0/1386425485 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5b6c0bca80 con 0x7f5b7c102ad0 2024-04-25T01:30:57.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.057+0000 7f5b838bd640 1 -- 172.21.15.28:0/1386425485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f5b7c02d170 con 0x7f5b7c102ad0 2024-04-25T01:30:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.407+0000 7f5b727fc640 1 -- 172.21.15.28:0/1386425485 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f5b6c0bc380 con 0x7f5b7c102ad0 2024-04-25T01:30:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.410+0000 7f5b838bd640 1 -- 172.21.15.28:0/1386425485 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5b680a2b90 msgr2=0x7f5b680a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.410+0000 7f5b838bd640 1 --2- 172.21.15.28:0/1386425485 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5b680a2b90 0x7f5b680a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5b78005fd0 tx=0x7f5b78005850 comp rx=0 tx=0).stop 2024-04-25T01:30:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.410+0000 7f5b838bd640 1 -- 172.21.15.28:0/1386425485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b7c102ad0 msgr2=0x7f5b7c197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.410+0000 7f5b838bd640 1 --2- 172.21.15.28:0/1386425485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b7c102ad0 0x7f5b7c197ff0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c031440 tx=0x7f5b6c032e20 comp rx=0 tx=0).stop 2024-04-25T01:30:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.410+0000 7f5b838bd640 1 -- 172.21.15.28:0/1386425485 shutdown_connections 2024-04-25T01:30:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.410+0000 7f5b838bd640 1 --2- 172.21.15.28:0/1386425485 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5b680a2b90 0x7f5b680a5050 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.410+0000 7f5b838bd640 1 --2- 172.21.15.28:0/1386425485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5b7c103490 0x7f5b7c19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.410+0000 7f5b838bd640 1 --2- 172.21.15.28:0/1386425485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b7c102ad0 0x7f5b7c197ff0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.410+0000 7f5b838bd640 1 --2- 172.21.15.28:0/1386425485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b7c0ffa60 0x7f5b7c197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.410+0000 7f5b838bd640 1 -- 172.21.15.28:0/1386425485 >> 172.21.15.28:0/1386425485 conn(0x7f5b7c0fb370 msgr2=0x7f5b7c105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.410+0000 7f5b838bd640 1 -- 172.21.15.28:0/1386425485 shutdown_connections 2024-04-25T01:30:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.410+0000 7f5b838bd640 1 -- 172.21.15.28:0/1386425485 wait complete. 2024-04-25T01:30:57.401 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-25T01:30:57.401 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-25T01:30:57.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.528+0000 7f905275c640 1 -- 172.21.15.28:0/964455223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f904c101890 msgr2=0x7f904c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:57.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.528+0000 7f905275c640 1 --2- 172.21.15.28:0/964455223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f904c101890 0x7f904c101d10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9040009900 tx=0x7f904002a410 comp rx=0 tx=0).stop 2024-04-25T01:30:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.528+0000 7f905275c640 1 -- 172.21.15.28:0/964455223 shutdown_connections 2024-04-25T01:30:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.528+0000 7f905275c640 1 --2- 172.21.15.28:0/964455223 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f904c102250 0x7f904c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.528+0000 7f905275c640 1 --2- 172.21.15.28:0/964455223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f904c101890 0x7f904c101d10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.528+0000 7f905275c640 1 --2- 172.21.15.28:0/964455223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f904c1006a0 0x7f904c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.528+0000 7f905275c640 1 -- 172.21.15.28:0/964455223 >> 172.21.15.28:0/964455223 conn(0x7f904c0fbc30 msgr2=0x7f904c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.528+0000 7f905275c640 1 -- 172.21.15.28:0/964455223 shutdown_connections 2024-04-25T01:30:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.528+0000 7f905275c640 1 -- 172.21.15.28:0/964455223 wait complete. 2024-04-25T01:30:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.529+0000 7f905275c640 1 Processor -- start 2024-04-25T01:30:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.529+0000 7f905275c640 1 -- start start 2024-04-25T01:30:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.529+0000 7f905275c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f904c1006a0 0x7f904c197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.529+0000 7f905275c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f904c101890 0x7f904c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.529+0000 7f905275c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f904c102250 0x7f904c19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.529+0000 7f905275c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f904c10c320 con 0x7f904c102250 2024-04-25T01:30:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.529+0000 7f905275c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f904c10c1a0 con 0x7f904c101890 2024-04-25T01:30:57.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.529+0000 7f905275c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f904c10c4a0 con 0x7f904c1006a0 2024-04-25T01:30:57.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.529+0000 7f9050cf9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f904c102250 0x7f904c19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:57.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.529+0000 7f9050cf9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f904c102250 0x7f904c19f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51926/0 (socket says 172.21.15.28:51926) 2024-04-25T01:30:57.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.529+0000 7f9050cf9640 1 -- 172.21.15.28:0/3639401597 learned_addr learned my addr 172.21.15.28:0/3639401597 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:57.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.529+0000 7f904b7fe640 1 --2- 172.21.15.28:0/3639401597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f904c101890 0x7f904c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:57.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.529+0000 7f904bfff640 1 --2- 172.21.15.28:0/3639401597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f904c1006a0 0x7f904c197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:57.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.530+0000 7f9050cf9640 1 -- 172.21.15.28:0/3639401597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f904c1006a0 msgr2=0x7f904c197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:57.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.530+0000 7f9050cf9640 1 --2- 172.21.15.28:0/3639401597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f904c1006a0 0x7f904c197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:57.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.530+0000 7f9050cf9640 1 -- 172.21.15.28:0/3639401597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f904c101890 msgr2=0x7f904c1980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:57.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.530+0000 7f9050cf9640 1 --2- 172.21.15.28:0/3639401597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f904c101890 0x7f904c1980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:57.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.530+0000 7f9050cf9640 1 -- 172.21.15.28:0/3639401597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f904c19f720 con 0x7f904c102250 2024-04-25T01:30:57.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.530+0000 7f904bfff640 1 --2- 172.21.15.28:0/3639401597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f904c1006a0 0x7f904c197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:30:57.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.530+0000 7f9050cf9640 1 --2- 172.21.15.28:0/3639401597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f904c102250 0x7f904c19f0b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f904400eb10 tx=0x7f904400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:57.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.530+0000 7f90497fa640 1 -- 172.21.15.28:0/3639401597 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9044004b20 con 0x7f904c102250 2024-04-25T01:30:57.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.530+0000 7f905275c640 1 -- 172.21.15.28:0/3639401597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f904c19fa10 con 0x7f904c102250 2024-04-25T01:30:57.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.530+0000 7f90497fa640 1 -- 172.21.15.28:0/3639401597 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9044004cc0 con 0x7f904c102250 2024-04-25T01:30:57.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.530+0000 7f90497fa640 1 -- 172.21.15.28:0/3639401597 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9044010760 con 0x7f904c102250 2024-04-25T01:30:57.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.530+0000 7f905275c640 1 -- 172.21.15.28:0/3639401597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f904c19ff30 con 0x7f904c102250 2024-04-25T01:30:57.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.531+0000 7f905275c640 1 -- 172.21.15.28:0/3639401597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f904c1a01c0 con 0x7f904c102250 2024-04-25T01:30:57.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.532+0000 7f90497fa640 1 -- 172.21.15.28:0/3639401597 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9044005000 con 0x7f904c102250 2024-04-25T01:30:57.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.533+0000 7f90497fa640 1 --2- 172.21.15.28:0/3639401597 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f90340a2b90 0x7f90340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:57.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.533+0000 7f904bfff640 1 --2- 172.21.15.28:0/3639401597 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f90340a2b90 0x7f90340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:57.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.533+0000 7f90497fa640 1 -- 172.21.15.28:0/3639401597 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f90440cfd50 con 0x7f904c102250 2024-04-25T01:30:57.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.533+0000 7f904bfff640 1 --2- 172.21.15.28:0/3639401597 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f90340a2b90 0x7f90340a5050 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f903c0058c0 tx=0x7f903c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:57.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.536+0000 7f90497fa640 1 -- 172.21.15.28:0/3639401597 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f904409ca70 con 0x7f904c102250 2024-04-25T01:30:57.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:57 smithi130 ceph-mon[32702]: pgmap v309: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T01:30:57.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:57 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/428654100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:57.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:57 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1386425485' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T01:30:57.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:57 smithi032 ceph-mon[32461]: pgmap v309: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T01:30:57.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:57 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/428654100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:57.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:57 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1386425485' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T01:30:57.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:57.758+0000 7f905275c640 1 -- 172.21.15.28:0/3639401597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f904c1a03e0 con 0x7f904c102250 2024-04-25T01:30:57.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:57 smithi028 ceph-mon[29390]: pgmap v309: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T01:30:57.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:57 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/428654100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:30:57.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:57 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1386425485' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T01:30:58.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.418+0000 7f90497fa640 1 -- 172.21.15.28:0/3639401597 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f904409c370 con 0x7f904c102250 2024-04-25T01:30:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.422+0000 7f905275c640 1 -- 172.21.15.28:0/3639401597 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f90340a2b90 msgr2=0x7f90340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.422+0000 7f905275c640 1 --2- 172.21.15.28:0/3639401597 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f90340a2b90 0x7f90340a5050 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f903c0058c0 tx=0x7f903c005850 comp rx=0 tx=0).stop 2024-04-25T01:30:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.422+0000 7f905275c640 1 -- 172.21.15.28:0/3639401597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f904c102250 msgr2=0x7f904c19f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.422+0000 7f905275c640 1 --2- 172.21.15.28:0/3639401597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f904c102250 0x7f904c19f0b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f904400eb10 tx=0x7f904400c2d0 comp rx=0 tx=0).stop 2024-04-25T01:30:58.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.422+0000 7f905275c640 1 -- 172.21.15.28:0/3639401597 shutdown_connections 2024-04-25T01:30:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.422+0000 7f905275c640 1 --2- 172.21.15.28:0/3639401597 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f90340a2b90 0x7f90340a5050 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.422+0000 7f905275c640 1 --2- 172.21.15.28:0/3639401597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f904c102250 0x7f904c19f0b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.422+0000 7f905275c640 1 --2- 172.21.15.28:0/3639401597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f904c101890 0x7f904c1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.422+0000 7f905275c640 1 --2- 172.21.15.28:0/3639401597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f904c1006a0 0x7f904c197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.422+0000 7f905275c640 1 -- 172.21.15.28:0/3639401597 >> 172.21.15.28:0/3639401597 conn(0x7f904c0fbc30 msgr2=0x7f904c0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.422+0000 7f905275c640 1 -- 172.21.15.28:0/3639401597 shutdown_connections 2024-04-25T01:30:58.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.422+0000 7f905275c640 1 -- 172.21.15.28:0/3639401597 wait complete. 2024-04-25T01:30:58.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-25T01:30:58.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.551+0000 7f4d5bfc1640 1 -- 172.21.15.28:0/3422933699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d541006a0 msgr2=0x7f4d54100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:58.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.551+0000 7f4d5bfc1640 1 --2- 172.21.15.28:0/3422933699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d541006a0 0x7f4d54100ac0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009900 tx=0x7f4d5002a410 comp rx=0 tx=0).stop 2024-04-25T01:30:58.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.551+0000 7f4d5bfc1640 1 -- 172.21.15.28:0/3422933699 shutdown_connections 2024-04-25T01:30:58.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.551+0000 7f4d5bfc1640 1 --2- 172.21.15.28:0/3422933699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4d54102250 0x7f4d54108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:58.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.551+0000 7f4d5bfc1640 1 --2- 172.21.15.28:0/3422933699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d54101890 0x7f4d54101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:58.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.551+0000 7f4d5bfc1640 1 --2- 172.21.15.28:0/3422933699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d541006a0 0x7f4d54100ac0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:58.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.551+0000 7f4d5bfc1640 1 -- 172.21.15.28:0/3422933699 >> 172.21.15.28:0/3422933699 conn(0x7f4d540fbc30 msgr2=0x7f4d540fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:58.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.551+0000 7f4d5bfc1640 1 -- 172.21.15.28:0/3422933699 shutdown_connections 2024-04-25T01:30:58.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.551+0000 7f4d5bfc1640 1 -- 172.21.15.28:0/3422933699 wait complete. 2024-04-25T01:30:58.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.552+0000 7f4d5bfc1640 1 Processor -- start 2024-04-25T01:30:58.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.552+0000 7f4d5bfc1640 1 -- start start 2024-04-25T01:30:58.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.552+0000 7f4d5bfc1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d541006a0 0x7f4d5406a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:58.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.552+0000 7f4d5bfc1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4d54101890 0x7f4d5419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:58.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.552+0000 7f4d5bfc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54102250 0x7f4d5419ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:58.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.552+0000 7f4d5bfc1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5410c3c0 con 0x7f4d54102250 2024-04-25T01:30:58.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.552+0000 7f4d5bfc1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5410c240 con 0x7f4d541006a0 2024-04-25T01:30:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.552+0000 7f4d5bfc1640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5410c540 con 0x7f4d54101890 2024-04-25T01:30:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.552+0000 7f4d5a55e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54102250 0x7f4d5419ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.552+0000 7f4d5a55e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54102250 0x7f4d5419ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51950/0 (socket says 172.21.15.28:51950) 2024-04-25T01:30:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.552+0000 7f4d5a55e640 1 -- 172.21.15.28:0/3092997734 learned_addr learned my addr 172.21.15.28:0/3092997734 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.552+0000 7f4d59d5d640 1 --2- 172.21.15.28:0/3092997734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d541006a0 0x7f4d5406a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.552+0000 7f4d5955c640 1 --2- 172.21.15.28:0/3092997734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4d54101890 0x7f4d5419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.552+0000 7f4d5a55e640 1 -- 172.21.15.28:0/3092997734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4d54101890 msgr2=0x7f4d5419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.552+0000 7f4d5a55e640 1 --2- 172.21.15.28:0/3092997734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4d54101890 0x7f4d5419c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.552+0000 7f4d5a55e640 1 -- 172.21.15.28:0/3092997734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d541006a0 msgr2=0x7f4d5406a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.552+0000 7f4d5a55e640 1 --2- 172.21.15.28:0/3092997734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d541006a0 0x7f4d5406a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.552+0000 7f4d5a55e640 1 -- 172.21.15.28:0/3092997734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d5419f750 con 0x7f4d54102250 2024-04-25T01:30:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.552+0000 7f4d5955c640 1 --2- 172.21.15.28:0/3092997734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4d54101890 0x7f4d5419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:30:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.553+0000 7f4d5a55e640 1 --2- 172.21.15.28:0/3092997734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54102250 0x7f4d5419ee40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00b970 tx=0x7f4d4c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.553+0000 7f4d4affd640 1 -- 172.21.15.28:0/3092997734 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c019070 con 0x7f4d54102250 2024-04-25T01:30:58.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.553+0000 7f4d4affd640 1 -- 172.21.15.28:0/3092997734 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d4c00cb40 con 0x7f4d54102250 2024-04-25T01:30:58.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.553+0000 7f4d5bfc1640 1 -- 172.21.15.28:0/3092997734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d5419f9e0 con 0x7f4d54102250 2024-04-25T01:30:58.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.553+0000 7f4d4affd640 1 -- 172.21.15.28:0/3092997734 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c007650 con 0x7f4d54102250 2024-04-25T01:30:58.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.553+0000 7f4d5bfc1640 1 -- 172.21.15.28:0/3092997734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d5419ffb0 con 0x7f4d54102250 2024-04-25T01:30:58.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.554+0000 7f4d5bfc1640 1 -- 172.21.15.28:0/3092997734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d54107960 con 0x7f4d54102250 2024-04-25T01:30:58.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.555+0000 7f4d4affd640 1 -- 172.21.15.28:0/3092997734 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4d4c01c070 con 0x7f4d54102250 2024-04-25T01:30:58.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.555+0000 7f4d4affd640 1 --2- 172.21.15.28:0/3092997734 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4d400a2b90 0x7f4d400a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:58.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.556+0000 7f4d59d5d640 1 --2- 172.21.15.28:0/3092997734 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4d400a2b90 0x7f4d400a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:58.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.556+0000 7f4d4affd640 1 -- 172.21.15.28:0/3092997734 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f4d4c0ce870 con 0x7f4d54102250 2024-04-25T01:30:58.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.556+0000 7f4d59d5d640 1 --2- 172.21.15.28:0/3092997734 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4d400a2b90 0x7f4d400a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4d5002aa00 tx=0x7f4d500315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:58.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.560+0000 7f4d4affd640 1 -- 172.21.15.28:0/3092997734 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4d4c09aee0 con 0x7f4d54102250 2024-04-25T01:30:58.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:58 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1386425485' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T01:30:58.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:58 smithi130 ceph-mon[32702]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:30:58.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:58 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3639401597' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-25T01:30:58.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:58 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1386425485' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T01:30:58.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:58 smithi032 ceph-mon[32461]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:30:58.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:58 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3639401597' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-25T01:30:58.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:58.780+0000 7f4d5bfc1640 1 -- 172.21.15.28:0/3092997734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f4d54062340 con 0x7f4d54102250 2024-04-25T01:30:58.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:58 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1386425485' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T01:30:58.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:58 smithi028 ceph-mon[29390]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:30:58.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:58 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3639401597' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-25T01:30:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.426+0000 7f4d4affd640 1 -- 172.21.15.28:0/3092997734 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7f4d4c09f1e0 con 0x7f4d54102250 2024-04-25T01:30:59.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.429+0000 7f4d5bfc1640 1 -- 172.21.15.28:0/3092997734 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4d400a2b90 msgr2=0x7f4d400a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:59.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.429+0000 7f4d5bfc1640 1 --2- 172.21.15.28:0/3092997734 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4d400a2b90 0x7f4d400a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4d5002aa00 tx=0x7f4d500315b0 comp rx=0 tx=0).stop 2024-04-25T01:30:59.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.429+0000 7f4d5bfc1640 1 -- 172.21.15.28:0/3092997734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54102250 msgr2=0x7f4d5419ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:59.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.429+0000 7f4d5bfc1640 1 --2- 172.21.15.28:0/3092997734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54102250 0x7f4d5419ee40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00b970 tx=0x7f4d4c00be30 comp rx=0 tx=0).stop 2024-04-25T01:30:59.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.429+0000 7f4d5bfc1640 1 -- 172.21.15.28:0/3092997734 shutdown_connections 2024-04-25T01:30:59.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.429+0000 7f4d5bfc1640 1 --2- 172.21.15.28:0/3092997734 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4d400a2b90 0x7f4d400a5050 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:59.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.429+0000 7f4d5bfc1640 1 --2- 172.21.15.28:0/3092997734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54102250 0x7f4d5419ee40 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:59.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.429+0000 7f4d5bfc1640 1 --2- 172.21.15.28:0/3092997734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4d54101890 0x7f4d5419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:59.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.429+0000 7f4d5bfc1640 1 --2- 172.21.15.28:0/3092997734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d541006a0 0x7f4d5406a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:59.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.429+0000 7f4d5bfc1640 1 -- 172.21.15.28:0/3092997734 >> 172.21.15.28:0/3092997734 conn(0x7f4d540fbc30 msgr2=0x7f4d54104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:59.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.429+0000 7f4d5bfc1640 1 -- 172.21.15.28:0/3092997734 shutdown_connections 2024-04-25T01:30:59.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.429+0000 7f4d5bfc1640 1 -- 172.21.15.28:0/3092997734 wait complete. 2024-04-25T01:30:59.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-25T01:30:59.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.558+0000 7fa951e2c640 1 -- 172.21.15.28:0/1614343609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c101310 msgr2=0x7fa94c101790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:59.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.558+0000 7fa951e2c640 1 --2- 172.21.15.28:0/1614343609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c101310 0x7fa94c101790 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa944009ca0 tx=0x7fa94402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:30:59.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.558+0000 7fa951e2c640 1 -- 172.21.15.28:0/1614343609 shutdown_connections 2024-04-25T01:30:59.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.558+0000 7fa951e2c640 1 --2- 172.21.15.28:0/1614343609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c101310 0x7fa94c101790 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:59.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.558+0000 7fa951e2c640 1 --2- 172.21.15.28:0/1614343609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa94c100970 0x7fa94c100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:59.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.558+0000 7fa951e2c640 1 --2- 172.21.15.28:0/1614343609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c100010 0x7fa94c100430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:59.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.558+0000 7fa951e2c640 1 -- 172.21.15.28:0/1614343609 >> 172.21.15.28:0/1614343609 conn(0x7fa94c0fbc10 msgr2=0x7fa94c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:30:59.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.558+0000 7fa951e2c640 1 -- 172.21.15.28:0/1614343609 shutdown_connections 2024-04-25T01:30:59.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.558+0000 7fa951e2c640 1 -- 172.21.15.28:0/1614343609 wait complete. 2024-04-25T01:30:59.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.559+0000 7fa951e2c640 1 Processor -- start 2024-04-25T01:30:59.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.559+0000 7fa951e2c640 1 -- start start 2024-04-25T01:30:59.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.559+0000 7fa951e2c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa94c100010 0x7fa94c199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:59.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.559+0000 7fa951e2c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c100970 0x7fa94c19a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:59.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.559+0000 7fa951e2c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c101310 0x7fa94c1a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:59.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.559+0000 7fa951e2c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c10f610 con 0x7fa94c101310 2024-04-25T01:30:59.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.559+0000 7fa951e2c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c10f490 con 0x7fa94c100970 2024-04-25T01:30:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.559+0000 7fa951e2c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c10f790 con 0x7fa94c100010 2024-04-25T01:30:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.559+0000 7fa94bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c101310 0x7fa94c1a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.559+0000 7fa94bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c101310 0x7fa94c1a11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51962/0 (socket says 172.21.15.28:51962) 2024-04-25T01:30:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.559+0000 7fa94bfff640 1 -- 172.21.15.28:0/1026192058 learned_addr learned my addr 172.21.15.28:0/1026192058 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:30:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.559+0000 7fa94affd640 1 --2- 172.21.15.28:0/1026192058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c100970 0x7fa94c19a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.560+0000 7fa94b7fe640 1 --2- 172.21.15.28:0/1026192058 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa94c100010 0x7fa94c199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.560+0000 7fa94bfff640 1 -- 172.21.15.28:0/1026192058 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa94c100010 msgr2=0x7fa94c199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.560+0000 7fa94bfff640 1 --2- 172.21.15.28:0/1026192058 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa94c100010 0x7fa94c199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.560+0000 7fa94bfff640 1 -- 172.21.15.28:0/1026192058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c100970 msgr2=0x7fa94c19a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:30:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.560+0000 7fa94bfff640 1 --2- 172.21.15.28:0/1026192058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c100970 0x7fa94c19a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:30:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.560+0000 7fa94bfff640 1 -- 172.21.15.28:0/1026192058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c1a1890 con 0x7fa94c101310 2024-04-25T01:30:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.560+0000 7fa94affd640 1 --2- 172.21.15.28:0/1026192058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c100970 0x7fa94c19a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:30:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.560+0000 7fa94bfff640 1 --2- 172.21.15.28:0/1026192058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c101310 0x7fa94c1a11f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa944031ae0 tx=0x7fa944031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.560+0000 7fa948ff9640 1 -- 172.21.15.28:0/1026192058 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa944003c30 con 0x7fa94c101310 2024-04-25T01:30:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.560+0000 7fa951e2c640 1 -- 172.21.15.28:0/1026192058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa94c1a1b20 con 0x7fa94c101310 2024-04-25T01:30:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.560+0000 7fa948ff9640 1 -- 172.21.15.28:0/1026192058 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa944003dd0 con 0x7fa94c101310 2024-04-25T01:30:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.560+0000 7fa948ff9640 1 -- 172.21.15.28:0/1026192058 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94402fcb0 con 0x7fa94c101310 2024-04-25T01:30:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.560+0000 7fa951e2c640 1 -- 172.21.15.28:0/1026192058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa94c1a20c0 con 0x7fa94c101310 2024-04-25T01:30:59.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.561+0000 7fa951e2c640 1 -- 172.21.15.28:0/1026192058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa94c10af00 con 0x7fa94c101310 2024-04-25T01:30:59.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.562+0000 7fa948ff9640 1 -- 172.21.15.28:0/1026192058 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa944004050 con 0x7fa94c101310 2024-04-25T01:30:59.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.563+0000 7fa948ff9640 1 --2- 172.21.15.28:0/1026192058 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa9340a2b90 0x7fa9340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:30:59.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.563+0000 7fa94b7fe640 1 --2- 172.21.15.28:0/1026192058 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa9340a2b90 0x7fa9340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:30:59.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.563+0000 7fa948ff9640 1 -- 172.21.15.28:0/1026192058 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fa9440f0870 con 0x7fa94c101310 2024-04-25T01:30:59.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.563+0000 7fa94b7fe640 1 --2- 172.21.15.28:0/1026192058 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa9340a2b90 0x7fa9340a5050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa94c19b150 tx=0x7fa93c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:30:59.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.566+0000 7fa948ff9640 1 -- 172.21.15.28:0/1026192058 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa9440bd530 con 0x7fa94c101310 2024-04-25T01:30:59.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:59 smithi130 ceph-mon[32702]: pgmap v310: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T01:30:59.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:59 smithi130 ceph-mon[32702]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T01:30:59.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:59 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3639401597' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-25T01:30:59.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:59 smithi130 ceph-mon[32702]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T01:30:59.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:59 smithi130 ceph-mon[32702]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:30:59.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:59 smithi130 ceph-mon[32702]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T01:30:59.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:59 smithi130 ceph-mon[32702]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T01:30:59.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:30:59 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3092997734' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T01:30:59.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:59 smithi032 ceph-mon[32461]: pgmap v310: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T01:30:59.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:59 smithi032 ceph-mon[32461]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T01:30:59.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:59 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3639401597' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-25T01:30:59.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:59 smithi032 ceph-mon[32461]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T01:30:59.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:59 smithi032 ceph-mon[32461]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:30:59.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:59 smithi032 ceph-mon[32461]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T01:30:59.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:59 smithi032 ceph-mon[32461]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T01:30:59.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:30:59 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3092997734' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T01:30:59.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:30:59.790+0000 7fa951e2c640 1 -- 172.21.15.28:0/1026192058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fa94c02d170 con 0x7fa94c101310 2024-04-25T01:30:59.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:59 smithi028 ceph-mon[29390]: pgmap v310: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T01:30:59.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:59 smithi028 ceph-mon[29390]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T01:30:59.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:59 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3639401597' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-25T01:30:59.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:59 smithi028 ceph-mon[29390]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T01:30:59.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:59 smithi028 ceph-mon[29390]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:30:59.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:59 smithi028 ceph-mon[29390]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T01:30:59.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:59 smithi028 ceph-mon[29390]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T01:30:59.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:30:59 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3092997734' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T01:31:00.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.435+0000 7fa948ff9640 1 -- 172.21.15.28:0/1026192058 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7fa9440bce30 con 0x7fa94c101310 2024-04-25T01:31:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.438+0000 7fa951e2c640 1 -- 172.21.15.28:0/1026192058 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa9340a2b90 msgr2=0x7fa9340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.438+0000 7fa951e2c640 1 --2- 172.21.15.28:0/1026192058 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa9340a2b90 0x7fa9340a5050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa94c19b150 tx=0x7fa93c005f90 comp rx=0 tx=0).stop 2024-04-25T01:31:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.438+0000 7fa951e2c640 1 -- 172.21.15.28:0/1026192058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c101310 msgr2=0x7fa94c1a11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.438+0000 7fa951e2c640 1 --2- 172.21.15.28:0/1026192058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c101310 0x7fa94c1a11f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa944031ae0 tx=0x7fa944031ee0 comp rx=0 tx=0).stop 2024-04-25T01:31:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.438+0000 7fa951e2c640 1 -- 172.21.15.28:0/1026192058 shutdown_connections 2024-04-25T01:31:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.438+0000 7fa951e2c640 1 --2- 172.21.15.28:0/1026192058 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa9340a2b90 0x7fa9340a5050 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.438+0000 7fa951e2c640 1 --2- 172.21.15.28:0/1026192058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c101310 0x7fa94c1a11f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.438+0000 7fa951e2c640 1 --2- 172.21.15.28:0/1026192058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa94c100970 0x7fa94c19a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.438+0000 7fa951e2c640 1 --2- 172.21.15.28:0/1026192058 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa94c100010 0x7fa94c199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.438+0000 7fa951e2c640 1 -- 172.21.15.28:0/1026192058 >> 172.21.15.28:0/1026192058 conn(0x7fa94c0fbc10 msgr2=0x7fa94c0fd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.439+0000 7fa951e2c640 1 -- 172.21.15.28:0/1026192058 shutdown_connections 2024-04-25T01:31:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.439+0000 7fa951e2c640 1 -- 172.21.15.28:0/1026192058 wait complete. 2024-04-25T01:31:00.434 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-04-25T01:31:00.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-25T01:31:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.568+0000 7f6d859ee640 1 -- 172.21.15.28:0/4224238479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d801006a0 msgr2=0x7f6d80100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.568+0000 7f6d859ee640 1 --2- 172.21.15.28:0/4224238479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d801006a0 0x7f6d80100ac0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6d70009900 tx=0x7f6d7002a410 comp rx=0 tx=0).stop 2024-04-25T01:31:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.568+0000 7f6d859ee640 1 -- 172.21.15.28:0/4224238479 shutdown_connections 2024-04-25T01:31:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.568+0000 7f6d859ee640 1 --2- 172.21.15.28:0/4224238479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d80102250 0x7f6d80108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.568+0000 7f6d859ee640 1 --2- 172.21.15.28:0/4224238479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d80101890 0x7f6d80101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.568+0000 7f6d859ee640 1 --2- 172.21.15.28:0/4224238479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d801006a0 0x7f6d80100ac0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.568+0000 7f6d859ee640 1 -- 172.21.15.28:0/4224238479 >> 172.21.15.28:0/4224238479 conn(0x7f6d800fbc30 msgr2=0x7f6d800fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:00.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.568+0000 7f6d859ee640 1 -- 172.21.15.28:0/4224238479 shutdown_connections 2024-04-25T01:31:00.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.568+0000 7f6d859ee640 1 -- 172.21.15.28:0/4224238479 wait complete. 2024-04-25T01:31:00.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.568+0000 7f6d859ee640 1 Processor -- start 2024-04-25T01:31:00.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d859ee640 1 -- start start 2024-04-25T01:31:00.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d859ee640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d801006a0 0x7f6d801969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:00.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d859ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d80101890 0x7f6d80196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:00.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d859ee640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d80102250 0x7f6d8019df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:00.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d859ee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8010c320 con 0x7f6d80101890 2024-04-25T01:31:00.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d859ee640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8010c1a0 con 0x7f6d80102250 2024-04-25T01:31:00.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d859ee640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8010c4a0 con 0x7f6d801006a0 2024-04-25T01:31:00.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d7e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d80101890 0x7f6d80196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:00.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d7e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d80101890 0x7f6d80196f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51970/0 (socket says 172.21.15.28:51970) 2024-04-25T01:31:00.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d7e7fc640 1 -- 172.21.15.28:0/1065033046 learned_addr learned my addr 172.21.15.28:0/1065033046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:00.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d7f7fe640 1 --2- 172.21.15.28:0/1065033046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d80102250 0x7f6d8019df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:00.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d7effd640 1 --2- 172.21.15.28:0/1065033046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d801006a0 0x7f6d801969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:00.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d7e7fc640 1 -- 172.21.15.28:0/1065033046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d801006a0 msgr2=0x7f6d801969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:00.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d7e7fc640 1 --2- 172.21.15.28:0/1065033046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d801006a0 0x7f6d801969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:00.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d7e7fc640 1 -- 172.21.15.28:0/1065033046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d80102250 msgr2=0x7f6d8019df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:00.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d7e7fc640 1 --2- 172.21.15.28:0/1065033046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d80102250 0x7f6d8019df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:00.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d7e7fc640 1 -- 172.21.15.28:0/1065033046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d8019e5e0 con 0x7f6d80101890 2024-04-25T01:31:00.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d7e7fc640 1 --2- 172.21.15.28:0/1065033046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d80101890 0x7f6d80196f00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6d6800e9f0 tx=0x7f6d6800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:00.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d849ec640 1 -- 172.21.15.28:0/1065033046 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6800cd80 con 0x7f6d80101890 2024-04-25T01:31:00.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d859ee640 1 -- 172.21.15.28:0/1065033046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d8019e8d0 con 0x7f6d80101890 2024-04-25T01:31:00.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.569+0000 7f6d849ec640 1 -- 172.21.15.28:0/1065033046 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d68010910 con 0x7f6d80101890 2024-04-25T01:31:00.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.570+0000 7f6d849ec640 1 -- 172.21.15.28:0/1065033046 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d68018990 con 0x7f6d80101890 2024-04-25T01:31:00.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.570+0000 7f6d859ee640 1 -- 172.21.15.28:0/1065033046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d8019ee20 con 0x7f6d80101890 2024-04-25T01:31:00.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.571+0000 7f6d859ee640 1 -- 172.21.15.28:0/1065033046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d801079c0 con 0x7f6d80101890 2024-04-25T01:31:00.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.572+0000 7f6d849ec640 1 -- 172.21.15.28:0/1065033046 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6d68010ac0 con 0x7f6d80101890 2024-04-25T01:31:00.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.572+0000 7f6d849ec640 1 --2- 172.21.15.28:0/1065033046 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6d6c0a2b90 0x7f6d6c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:00.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.573+0000 7f6d7effd640 1 --2- 172.21.15.28:0/1065033046 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6d6c0a2b90 0x7f6d6c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:00.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.573+0000 7f6d849ec640 1 -- 172.21.15.28:0/1065033046 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f6d680cf620 con 0x7f6d80101890 2024-04-25T01:31:00.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.573+0000 7f6d7effd640 1 --2- 172.21.15.28:0/1065033046 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6d6c0a2b90 0x7f6d6c0a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6d70006010 tx=0x7f6d70000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:00.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.577+0000 7f6d849ec640 1 -- 172.21.15.28:0/1065033046 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6d6809c340 con 0x7f6d80101890 2024-04-25T01:31:00.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:00 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3092997734' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T01:31:00.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:00 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] up:active 2024-04-25T01:31:00.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:00 smithi130 ceph-mon[32702]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T01:31:00.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:00 smithi130 ceph-mon[32702]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T01:31:00.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:00 smithi130 ceph-mon[32702]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-25T01:31:00.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:00 smithi130 ceph-mon[32702]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-25T01:31:00.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:00 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1026192058' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T01:31:00.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:00 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3092997734' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T01:31:00.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:00 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] up:active 2024-04-25T01:31:00.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:00 smithi032 ceph-mon[32461]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T01:31:00.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:00 smithi032 ceph-mon[32461]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T01:31:00.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:00 smithi032 ceph-mon[32461]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-25T01:31:00.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:00 smithi032 ceph-mon[32461]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-25T01:31:00.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:00 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1026192058' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T01:31:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:00.790+0000 7f6d859ee640 1 -- 172.21.15.28:0/1065033046 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f6d80061b80 con 0x7f6d6c0a2b90 2024-04-25T01:31:00.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:00 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3092997734' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T01:31:00.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:00 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] up:active 2024-04-25T01:31:00.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:00 smithi028 ceph-mon[29390]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T01:31:00.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:00 smithi028 ceph-mon[29390]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T01:31:00.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:00 smithi028 ceph-mon[29390]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-25T01:31:00.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:00 smithi028 ceph-mon[29390]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-25T01:31:00.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:00 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1026192058' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T01:31:01.062 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:31:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:31:00.795+0000 7f8fa322e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:31:01.062 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:31:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:31:00.796+0000 7f8fa322e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:31:01.062 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:31:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:31:00.796+0000 7f8fa322e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:31:01.062 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:31:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:31:00.796+0000 7f8fa322e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:31:01.062 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:31:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:31:00.796+0000 7f8fa322e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:31:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.473+0000 7f6d849ec640 1 -- 172.21.15.28:0/1065033046 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6d80061b80 con 0x7f6d6c0a2b90 2024-04-25T01:31:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.476+0000 7f6d859ee640 1 -- 172.21.15.28:0/1065033046 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6d6c0a2b90 msgr2=0x7f6d6c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.476+0000 7f6d859ee640 1 --2- 172.21.15.28:0/1065033046 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6d6c0a2b90 0x7f6d6c0a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6d70006010 tx=0x7f6d70000bc0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.476+0000 7f6d859ee640 1 -- 172.21.15.28:0/1065033046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d80101890 msgr2=0x7f6d80196f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.476+0000 7f6d859ee640 1 --2- 172.21.15.28:0/1065033046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d80101890 0x7f6d80196f00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f6d6800e9f0 tx=0x7f6d6800eeb0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.476+0000 7f6d859ee640 1 -- 172.21.15.28:0/1065033046 shutdown_connections 2024-04-25T01:31:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.476+0000 7f6d859ee640 1 --2- 172.21.15.28:0/1065033046 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6d6c0a2b90 0x7f6d6c0a5050 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.476+0000 7f6d859ee640 1 --2- 172.21.15.28:0/1065033046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d80102250 0x7f6d8019df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.476+0000 7f6d859ee640 1 --2- 172.21.15.28:0/1065033046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d80101890 0x7f6d80196f00 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.476+0000 7f6d859ee640 1 --2- 172.21.15.28:0/1065033046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d801006a0 0x7f6d801969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.476+0000 7f6d859ee640 1 -- 172.21.15.28:0/1065033046 >> 172.21.15.28:0/1065033046 conn(0x7f6d800fbc30 msgr2=0x7f6d80104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:01.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.476+0000 7f6d859ee640 1 -- 172.21.15.28:0/1065033046 shutdown_connections 2024-04-25T01:31:01.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.476+0000 7f6d859ee640 1 -- 172.21.15.28:0/1065033046 wait complete. 2024-04-25T01:31:01.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-25T01:31:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.604+0000 7fe90fe38640 1 -- 172.21.15.28:0/3219385817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9081006a0 msgr2=0x7fe908100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.604+0000 7fe90fe38640 1 --2- 172.21.15.28:0/3219385817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9081006a0 0x7fe908100ac0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8009900 tx=0x7fe8f802a410 comp rx=0 tx=0).stop 2024-04-25T01:31:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.604+0000 7fe90fe38640 1 -- 172.21.15.28:0/3219385817 shutdown_connections 2024-04-25T01:31:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.604+0000 7fe90fe38640 1 --2- 172.21.15.28:0/3219385817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe908102250 0x7fe908108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.604+0000 7fe90fe38640 1 --2- 172.21.15.28:0/3219385817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe908101890 0x7fe908101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.604+0000 7fe90fe38640 1 --2- 172.21.15.28:0/3219385817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9081006a0 0x7fe908100ac0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.604+0000 7fe90fe38640 1 -- 172.21.15.28:0/3219385817 >> 172.21.15.28:0/3219385817 conn(0x7fe9080fbc30 msgr2=0x7fe9080fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.604+0000 7fe90fe38640 1 -- 172.21.15.28:0/3219385817 shutdown_connections 2024-04-25T01:31:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.604+0000 7fe90fe38640 1 -- 172.21.15.28:0/3219385817 wait complete. 2024-04-25T01:31:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90fe38640 1 Processor -- start 2024-04-25T01:31:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90fe38640 1 -- start start 2024-04-25T01:31:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90fe38640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9081006a0 0x7fe908197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90fe38640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe908101890 0x7fe908198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90fe38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe908102250 0x7fe90819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90fe38640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe90810c320 con 0x7fe908102250 2024-04-25T01:31:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90fe38640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe90810c1a0 con 0x7fe908101890 2024-04-25T01:31:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90fe38640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe90810c4a0 con 0x7fe9081006a0 2024-04-25T01:31:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90e3d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe908102250 0x7fe90819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90e3d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe908102250 0x7fe90819f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52000/0 (socket says 172.21.15.28:52000) 2024-04-25T01:31:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90e3d5640 1 -- 172.21.15.28:0/3129187370 learned_addr learned my addr 172.21.15.28:0/3129187370 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90d3d3640 1 --2- 172.21.15.28:0/3129187370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe908101890 0x7fe908198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90dbd4640 1 --2- 172.21.15.28:0/3129187370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9081006a0 0x7fe908197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90e3d5640 1 -- 172.21.15.28:0/3129187370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9081006a0 msgr2=0x7fe908197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90e3d5640 1 --2- 172.21.15.28:0/3129187370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9081006a0 0x7fe908197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90e3d5640 1 -- 172.21.15.28:0/3129187370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe908101890 msgr2=0x7fe908198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90e3d5640 1 --2- 172.21.15.28:0/3129187370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe908101890 0x7fe908198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90e3d5640 1 -- 172.21.15.28:0/3129187370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe90819f740 con 0x7fe908102250 2024-04-25T01:31:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90dbd4640 1 --2- 172.21.15.28:0/3129187370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9081006a0 0x7fe908197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe90e3d5640 1 --2- 172.21.15.28:0/3129187370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe908102250 0x7fe90819f0d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe90000eb10 tx=0x7fe90000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.605+0000 7fe8feffd640 1 -- 172.21.15.28:0/3129187370 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe900004b20 con 0x7fe908102250 2024-04-25T01:31:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.606+0000 7fe8feffd640 1 -- 172.21.15.28:0/3129187370 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe900004cc0 con 0x7fe908102250 2024-04-25T01:31:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.606+0000 7fe90fe38640 1 -- 172.21.15.28:0/3129187370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe90819fa30 con 0x7fe908102250 2024-04-25T01:31:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.606+0000 7fe8feffd640 1 -- 172.21.15.28:0/3129187370 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe900010760 con 0x7fe908102250 2024-04-25T01:31:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.606+0000 7fe90fe38640 1 -- 172.21.15.28:0/3129187370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe90819ff80 con 0x7fe908102250 2024-04-25T01:31:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.607+0000 7fe90fe38640 1 -- 172.21.15.28:0/3129187370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9081a0210 con 0x7fe908102250 2024-04-25T01:31:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.608+0000 7fe8feffd640 1 -- 172.21.15.28:0/3129187370 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe900005000 con 0x7fe908102250 2024-04-25T01:31:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.608+0000 7fe8feffd640 1 --2- 172.21.15.28:0/3129187370 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe8f40a2c10 0x7fe8f40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.608+0000 7fe90dbd4640 1 --2- 172.21.15.28:0/3129187370 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe8f40a2c10 0x7fe8f40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.609+0000 7fe8feffd640 1 -- 172.21.15.28:0/3129187370 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fe9000cfa50 con 0x7fe908102250 2024-04-25T01:31:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.609+0000 7fe90dbd4640 1 --2- 172.21.15.28:0/3129187370 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe8f40a2c10 0x7fe8f40a50d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8000c00 tx=0x7fe8f8031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:01.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.612+0000 7fe8feffd640 1 -- 172.21.15.28:0/3129187370 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe900053740 con 0x7fe908102250 2024-04-25T01:31:01.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:01 smithi130 ceph-mon[32702]: pgmap v311: 129 pgs: 129 active+clean; 583 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 3 op/s 2024-04-25T01:31:01.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:01 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1026192058' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T01:31:01.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:01 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] up:active 2024-04-25T01:31:01.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:01 smithi130 ceph-mon[32702]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-25T01:31:01.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:01 smithi130 ceph-mon[32702]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-25T01:31:01.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:01 smithi130 ceph-mon[32702]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-04-25T01:31:01.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:01 smithi130 ceph-mon[32702]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-04-25T01:31:01.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:01 smithi130 ceph-mon[32702]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:31:01.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:01 smithi032 ceph-mon[32461]: pgmap v311: 129 pgs: 129 active+clean; 583 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 3 op/s 2024-04-25T01:31:01.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:01 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1026192058' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T01:31:01.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:01 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] up:active 2024-04-25T01:31:01.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:01 smithi032 ceph-mon[32461]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-25T01:31:01.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:01 smithi032 ceph-mon[32461]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-25T01:31:01.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:01 smithi032 ceph-mon[32461]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-04-25T01:31:01.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:01 smithi032 ceph-mon[32461]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-04-25T01:31:01.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:01 smithi032 ceph-mon[32461]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:31:01.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.827+0000 7fe90fe38640 1 -- 172.21.15.28:0/3129187370 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fe90806a450 con 0x7fe8f40a2c10 2024-04-25T01:31:01.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:01 smithi028 ceph-mon[29390]: pgmap v311: 129 pgs: 129 active+clean; 583 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 3 op/s 2024-04-25T01:31:01.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:01 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1026192058' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T01:31:01.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:01 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] up:active 2024-04-25T01:31:01.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:01 smithi028 ceph-mon[29390]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-25T01:31:01.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:01 smithi028 ceph-mon[29390]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-25T01:31:01.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:01 smithi028 ceph-mon[29390]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-04-25T01:31:01.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:01 smithi028 ceph-mon[29390]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-04-25T01:31:01.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:01 smithi028 ceph-mon[29390]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:31:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.848+0000 7fe8feffd640 1 -- 172.21.15.28:0/3129187370 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe90806a450 con 0x7fe8f40a2c10 2024-04-25T01:31:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.850+0000 7fe90fe38640 1 -- 172.21.15.28:0/3129187370 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe8f40a2c10 msgr2=0x7fe8f40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.850+0000 7fe90fe38640 1 --2- 172.21.15.28:0/3129187370 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe8f40a2c10 0x7fe8f40a50d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8000c00 tx=0x7fe8f8031400 comp rx=0 tx=0).stop 2024-04-25T01:31:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.850+0000 7fe90fe38640 1 -- 172.21.15.28:0/3129187370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe908102250 msgr2=0x7fe90819f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.850+0000 7fe90fe38640 1 --2- 172.21.15.28:0/3129187370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe908102250 0x7fe90819f0d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe90000eb10 tx=0x7fe90000c2d0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.850+0000 7fe90fe38640 1 -- 172.21.15.28:0/3129187370 shutdown_connections 2024-04-25T01:31:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.850+0000 7fe90fe38640 1 --2- 172.21.15.28:0/3129187370 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe8f40a2c10 0x7fe8f40a50d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.850+0000 7fe90fe38640 1 --2- 172.21.15.28:0/3129187370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe908102250 0x7fe90819f0d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.850+0000 7fe90fe38640 1 --2- 172.21.15.28:0/3129187370 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe908101890 0x7fe908198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.850+0000 7fe90fe38640 1 --2- 172.21.15.28:0/3129187370 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9081006a0 0x7fe908197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.850+0000 7fe90fe38640 1 -- 172.21.15.28:0/3129187370 >> 172.21.15.28:0/3129187370 conn(0x7fe9080fbc30 msgr2=0x7fe908069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.851+0000 7fe90fe38640 1 -- 172.21.15.28:0/3129187370 shutdown_connections 2024-04-25T01:31:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.851+0000 7fe90fe38640 1 -- 172.21.15.28:0/3129187370 wait complete. 2024-04-25T01:31:01.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-25T01:31:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.969+0000 7fd00df34640 1 -- 172.21.15.28:0/4198512886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd008101790 msgr2=0x7fd008101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.969+0000 7fd00df34640 1 --2- 172.21.15.28:0/4198512886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd008101790 0x7fd008101bb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009900 tx=0x7fcff802a410 comp rx=0 tx=0).stop 2024-04-25T01:31:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.969+0000 7fd00df34640 1 -- 172.21.15.28:0/4198512886 shutdown_connections 2024-04-25T01:31:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.969+0000 7fd00df34640 1 --2- 172.21.15.28:0/4198512886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd008103340 0x7fd008109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.969+0000 7fd00df34640 1 --2- 172.21.15.28:0/4198512886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd008102980 0x7fd008102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.969+0000 7fd00df34640 1 --2- 172.21.15.28:0/4198512886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd008101790 0x7fd008101bb0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.969+0000 7fd00df34640 1 -- 172.21.15.28:0/4198512886 >> 172.21.15.28:0/4198512886 conn(0x7fd0080fcd20 msgr2=0x7fd0080ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.969+0000 7fd00df34640 1 -- 172.21.15.28:0/4198512886 shutdown_connections 2024-04-25T01:31:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.969+0000 7fd00df34640 1 -- 172.21.15.28:0/4198512886 wait complete. 2024-04-25T01:31:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.969+0000 7fd00df34640 1 Processor -- start 2024-04-25T01:31:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.969+0000 7fd00df34640 1 -- start start 2024-04-25T01:31:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd00df34640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd008101790 0x7fd008197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd00df34640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd008102980 0x7fd008198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd00df34640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd008103340 0x7fd00819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd00df34640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd00810d3e0 con 0x7fd008103340 2024-04-25T01:31:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd00df34640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd00810d260 con 0x7fd008101790 2024-04-25T01:31:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd00df34640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd00810d560 con 0x7fd008102980 2024-04-25T01:31:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd007fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd008103340 0x7fd00819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd006ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd008102980 0x7fd008198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd0077fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd008101790 0x7fd008197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd007fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd008103340 0x7fd00819f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52018/0 (socket says 172.21.15.28:52018) 2024-04-25T01:31:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd007fff640 1 -- 172.21.15.28:0/1930083029 learned_addr learned my addr 172.21.15.28:0/1930083029 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd007fff640 1 -- 172.21.15.28:0/1930083029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd008102980 msgr2=0x7fd008198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:01.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd007fff640 1 --2- 172.21.15.28:0/1930083029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd008102980 0x7fd008198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd007fff640 1 -- 172.21.15.28:0/1930083029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd008101790 msgr2=0x7fd008197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:01.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd007fff640 1 --2- 172.21.15.28:0/1930083029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd008101790 0x7fd008197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:01.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd007fff640 1 -- 172.21.15.28:0/1930083029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd00819f700 con 0x7fd008103340 2024-04-25T01:31:01.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd0077fe640 1 --2- 172.21.15.28:0/1930083029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd008101790 0x7fd008197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:31:01.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd007fff640 1 --2- 172.21.15.28:0/1930083029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd008103340 0x7fd00819f090 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd00000eb10 tx=0x7fd00000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:01.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd004ff9640 1 -- 172.21.15.28:0/1930083029 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd000004b20 con 0x7fd008103340 2024-04-25T01:31:01.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd004ff9640 1 -- 172.21.15.28:0/1930083029 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd000004cc0 con 0x7fd008103340 2024-04-25T01:31:01.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd00df34640 1 -- 172.21.15.28:0/1930083029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd00819f9f0 con 0x7fd008103340 2024-04-25T01:31:01.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.970+0000 7fd004ff9640 1 -- 172.21.15.28:0/1930083029 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd000010760 con 0x7fd008103340 2024-04-25T01:31:01.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.971+0000 7fd00df34640 1 -- 172.21.15.28:0/1930083029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd00819fef0 con 0x7fd008103340 2024-04-25T01:31:01.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.972+0000 7fd00df34640 1 -- 172.21.15.28:0/1930083029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd008108b80 con 0x7fd008103340 2024-04-25T01:31:01.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.973+0000 7fd004ff9640 1 -- 172.21.15.28:0/1930083029 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd000005020 con 0x7fd008103340 2024-04-25T01:31:01.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.973+0000 7fd004ff9640 1 --2- 172.21.15.28:0/1930083029 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fcff00a2b90 0x7fcff00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:01.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.973+0000 7fd0077fe640 1 --2- 172.21.15.28:0/1930083029 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fcff00a2b90 0x7fcff00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:01.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.973+0000 7fd004ff9640 1 -- 172.21.15.28:0/1930083029 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fd0000cf830 con 0x7fd008103340 2024-04-25T01:31:01.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.974+0000 7fd0077fe640 1 --2- 172.21.15.28:0/1930083029 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fcff00a2b90 0x7fcff00a5050 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcff8005ab0 tx=0x7fcff8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:01.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:01.978+0000 7fd004ff9640 1 -- 172.21.15.28:0/1930083029 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd00009c550 con 0x7fd008103340 2024-04-25T01:31:02.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.195+0000 7fd00df34640 1 -- 172.21.15.28:0/1930083029 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fd008061b80 con 0x7fcff00a2b90 2024-04-25T01:31:02.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.200+0000 7fd004ff9640 1 -- 172.21.15.28:0/1930083029 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fd008061b80 con 0x7fcff00a2b90 2024-04-25T01:31:02.178 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_0/08ee3e9b-ba58-4daf-98e8-f459e17913b1 2024-04-25T01:31:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.202+0000 7fd00df34640 1 -- 172.21.15.28:0/1930083029 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fcff00a2b90 msgr2=0x7fcff00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.202+0000 7fd00df34640 1 --2- 172.21.15.28:0/1930083029 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fcff00a2b90 0x7fcff00a5050 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcff8005ab0 tx=0x7fcff8000bc0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.202+0000 7fd00df34640 1 -- 172.21.15.28:0/1930083029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd008103340 msgr2=0x7fd00819f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.202+0000 7fd00df34640 1 --2- 172.21.15.28:0/1930083029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd008103340 0x7fd00819f090 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd00000eb10 tx=0x7fd00000c2d0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.202+0000 7fd00df34640 1 -- 172.21.15.28:0/1930083029 shutdown_connections 2024-04-25T01:31:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.202+0000 7fd00df34640 1 --2- 172.21.15.28:0/1930083029 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fcff00a2b90 0x7fcff00a5050 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.202+0000 7fd00df34640 1 --2- 172.21.15.28:0/1930083029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd008103340 0x7fd00819f090 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.202+0000 7fd00df34640 1 --2- 172.21.15.28:0/1930083029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd008102980 0x7fd008198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.202+0000 7fd00df34640 1 --2- 172.21.15.28:0/1930083029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd008101790 0x7fd008197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.202+0000 7fd00df34640 1 -- 172.21.15.28:0/1930083029 >> 172.21.15.28:0/1930083029 conn(0x7fd0080fcd20 msgr2=0x7fd008105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:02.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.202+0000 7fd00df34640 1 -- 172.21.15.28:0/1930083029 shutdown_connections 2024-04-25T01:31:02.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.202+0000 7fd00df34640 1 -- 172.21.15.28:0/1930083029 wait complete. 2024-04-25T01:31:02.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-25T01:31:02.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.320+0000 7fe7d4aa5640 1 -- 172.21.15.28:0/762829068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d0102250 msgr2=0x7fe7d0108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:02.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.320+0000 7fe7d4aa5640 1 --2- 172.21.15.28:0/762829068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d0102250 0x7fe7d0108ae0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8009ca0 tx=0x7fe7c802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.320+0000 7fe7d4aa5640 1 -- 172.21.15.28:0/762829068 shutdown_connections 2024-04-25T01:31:02.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.320+0000 7fe7d4aa5640 1 --2- 172.21.15.28:0/762829068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d0102250 0x7fe7d0108ae0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.320+0000 7fe7d4aa5640 1 --2- 172.21.15.28:0/762829068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7d0101890 0x7fe7d0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.320+0000 7fe7d4aa5640 1 --2- 172.21.15.28:0/762829068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7d01006a0 0x7fe7d0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.320+0000 7fe7d4aa5640 1 -- 172.21.15.28:0/762829068 >> 172.21.15.28:0/762829068 conn(0x7fe7d00fbc30 msgr2=0x7fe7d00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:02.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.320+0000 7fe7d4aa5640 1 -- 172.21.15.28:0/762829068 shutdown_connections 2024-04-25T01:31:02.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.320+0000 7fe7d4aa5640 1 -- 172.21.15.28:0/762829068 wait complete. 2024-04-25T01:31:02.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7d4aa5640 1 Processor -- start 2024-04-25T01:31:02.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7d4aa5640 1 -- start start 2024-04-25T01:31:02.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7d4aa5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7d01006a0 0x7fe7d0197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:02.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7d4aa5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7d0101890 0x7fe7d0197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:02.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7d4aa5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d0102250 0x7fe7d019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:02.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7d4aa5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d010c350 con 0x7fe7d0102250 2024-04-25T01:31:02.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7d4aa5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d010c1d0 con 0x7fe7d01006a0 2024-04-25T01:31:02.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7d4aa5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d010c4d0 con 0x7fe7d0101890 2024-04-25T01:31:02.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7ced9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d0102250 0x7fe7d019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:02.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7cdd9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7d0101890 0x7fe7d0197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:02.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7ced9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d0102250 0x7fe7d019f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52042/0 (socket says 172.21.15.28:52042) 2024-04-25T01:31:02.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7ced9d640 1 -- 172.21.15.28:0/1896244650 learned_addr learned my addr 172.21.15.28:0/1896244650 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:02.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7ce59c640 1 --2- 172.21.15.28:0/1896244650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7d01006a0 0x7fe7d0197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:02.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7ced9d640 1 -- 172.21.15.28:0/1896244650 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7d0101890 msgr2=0x7fe7d0197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:02.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7ced9d640 1 --2- 172.21.15.28:0/1896244650 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7d0101890 0x7fe7d0197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7ced9d640 1 -- 172.21.15.28:0/1896244650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7d01006a0 msgr2=0x7fe7d0197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:02.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7ced9d640 1 --2- 172.21.15.28:0/1896244650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7d01006a0 0x7fe7d0197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7ced9d640 1 -- 172.21.15.28:0/1896244650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d019f680 con 0x7fe7d0102250 2024-04-25T01:31:02.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7ce59c640 1 --2- 172.21.15.28:0/1896244650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7d01006a0 0x7fe7d0197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:31:02.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.321+0000 7fe7ced9d640 1 --2- 172.21.15.28:0/1896244650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d0102250 0x7fe7d019f010 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8009430 tx=0x7fe7c80319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:02.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.322+0000 7fe7bf7fe640 1 -- 172.21.15.28:0/1896244650 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8003b70 con 0x7fe7d0102250 2024-04-25T01:31:02.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.322+0000 7fe7bf7fe640 1 -- 172.21.15.28:0/1896244650 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7c8003d10 con 0x7fe7d0102250 2024-04-25T01:31:02.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.322+0000 7fe7d4aa5640 1 -- 172.21.15.28:0/1896244650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7d019f910 con 0x7fe7d0102250 2024-04-25T01:31:02.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.322+0000 7fe7bf7fe640 1 -- 172.21.15.28:0/1896244650 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c802fc50 con 0x7fe7d0102250 2024-04-25T01:31:02.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.322+0000 7fe7d4aa5640 1 -- 172.21.15.28:0/1896244650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7d019feb0 con 0x7fe7d0102250 2024-04-25T01:31:02.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.323+0000 7fe7d4aa5640 1 -- 172.21.15.28:0/1896244650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7d01a00e0 con 0x7fe7d0102250 2024-04-25T01:31:02.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.324+0000 7fe7bf7fe640 1 -- 172.21.15.28:0/1896244650 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe7c8004040 con 0x7fe7d0102250 2024-04-25T01:31:02.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.324+0000 7fe7bf7fe640 1 --2- 172.21.15.28:0/1896244650 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe7b80a2c10 0x7fe7b80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:02.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.324+0000 7fe7bf7fe640 1 -- 172.21.15.28:0/1896244650 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fe7c80f0670 con 0x7fe7d0102250 2024-04-25T01:31:02.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.324+0000 7fe7ce59c640 1 --2- 172.21.15.28:0/1896244650 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe7b80a2c10 0x7fe7b80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:02.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.325+0000 7fe7ce59c640 1 --2- 172.21.15.28:0/1896244650 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe7b80a2c10 0x7fe7b80a50d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0005fd0 tx=0x7fe7c000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:02.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.329+0000 7fe7bf7fe640 1 -- 172.21.15.28:0/1896244650 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe7c80bd390 con 0x7fe7d0102250 2024-04-25T01:31:02.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.543+0000 7fe7d4aa5640 1 -- 172.21.15.28:0/1896244650 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fe7d00fdd90 con 0x7fe7b80a2c10 2024-04-25T01:31:02.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.571+0000 7fe7bf7fe640 1 -- 172.21.15.28:0/1896244650 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe7d00fdd90 con 0x7fe7b80a2c10 2024-04-25T01:31:02.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.573+0000 7fe7d4aa5640 1 -- 172.21.15.28:0/1896244650 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe7b80a2c10 msgr2=0x7fe7b80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:02.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.573+0000 7fe7d4aa5640 1 --2- 172.21.15.28:0/1896244650 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe7b80a2c10 0x7fe7b80a50d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0005fd0 tx=0x7fe7c000b410 comp rx=0 tx=0).stop 2024-04-25T01:31:02.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.573+0000 7fe7d4aa5640 1 -- 172.21.15.28:0/1896244650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d0102250 msgr2=0x7fe7d019f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.573+0000 7fe7d4aa5640 1 --2- 172.21.15.28:0/1896244650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d0102250 0x7fe7d019f010 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8009430 tx=0x7fe7c80319f0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.573+0000 7fe7d4aa5640 1 -- 172.21.15.28:0/1896244650 shutdown_connections 2024-04-25T01:31:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.573+0000 7fe7d4aa5640 1 --2- 172.21.15.28:0/1896244650 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe7b80a2c10 0x7fe7b80a50d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.573+0000 7fe7d4aa5640 1 --2- 172.21.15.28:0/1896244650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d0102250 0x7fe7d019f010 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.573+0000 7fe7d4aa5640 1 --2- 172.21.15.28:0/1896244650 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7d0101890 0x7fe7d0197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.573+0000 7fe7d4aa5640 1 --2- 172.21.15.28:0/1896244650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7d01006a0 0x7fe7d0197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.573+0000 7fe7d4aa5640 1 -- 172.21.15.28:0/1896244650 >> 172.21.15.28:0/1896244650 conn(0x7fe7d00fbc30 msgr2=0x7fe7d00fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.573+0000 7fe7d4aa5640 1 -- 172.21.15.28:0/1896244650 shutdown_connections 2024-04-25T01:31:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.573+0000 7fe7d4aa5640 1 -- 172.21.15.28:0/1896244650 wait complete. 2024-04-25T01:31:02.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:02 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] up:active 2024-04-25T01:31:02.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:02 smithi028 ceph-mon[29390]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-25T01:31:02.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:02 smithi028 ceph-mon[29390]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T01:31:02.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:02 smithi028 ceph-mon[29390]: Cluster is now healthy 2024-04-25T01:31:02.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:02 smithi028 ceph-mon[29390]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2024-04-25T01:31:02.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:02 smithi028 ceph-mon[29390]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-25T01:31:02.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:02 smithi028 ceph-mon[29390]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-04-25T01:31:02.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:02 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] up:active 2024-04-25T01:31:02.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:02 smithi028 ceph-mon[29390]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:31:02.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:02 smithi028 ceph-mon[29390]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:31:02.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-25T01:31:02.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.691+0000 7f990bd77640 1 -- 172.21.15.28:0/2148411117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904101890 msgr2=0x7f9904101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:02.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.691+0000 7f990bd77640 1 --2- 172.21.15.28:0/2148411117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904101890 0x7f9904101d10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f98f4009900 tx=0x7f98f402a410 comp rx=0 tx=0).stop 2024-04-25T01:31:02.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.691+0000 7f990bd77640 1 -- 172.21.15.28:0/2148411117 shutdown_connections 2024-04-25T01:31:02.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.691+0000 7f990bd77640 1 --2- 172.21.15.28:0/2148411117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9904102250 0x7f9904108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.691+0000 7f990bd77640 1 --2- 172.21.15.28:0/2148411117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904101890 0x7f9904101d10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.691+0000 7f990bd77640 1 --2- 172.21.15.28:0/2148411117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99041006a0 0x7f9904100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.691+0000 7f990bd77640 1 -- 172.21.15.28:0/2148411117 >> 172.21.15.28:0/2148411117 conn(0x7f99040fbc30 msgr2=0x7f99040fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:02.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.692+0000 7f990bd77640 1 -- 172.21.15.28:0/2148411117 shutdown_connections 2024-04-25T01:31:02.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.692+0000 7f990bd77640 1 -- 172.21.15.28:0/2148411117 wait complete. 2024-04-25T01:31:02.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.692+0000 7f990bd77640 1 Processor -- start 2024-04-25T01:31:02.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.692+0000 7f990bd77640 1 -- start start 2024-04-25T01:31:02.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.692+0000 7f990bd77640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f99041006a0 0x7f9904197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.692+0000 7f990bd77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904101890 0x7f9904197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.692+0000 7f990bd77640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9904102250 0x7f990419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.692+0000 7f990bd77640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f990410c320 con 0x7f9904101890 2024-04-25T01:31:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.692+0000 7f990bd77640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f990410c1a0 con 0x7f9904102250 2024-04-25T01:31:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.692+0000 7f990bd77640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f990410c4a0 con 0x7f99041006a0 2024-04-25T01:31:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.692+0000 7f9909312640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904101890 0x7f9904197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.692+0000 7f9909312640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904101890 0x7f9904197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52060/0 (socket says 172.21.15.28:52060) 2024-04-25T01:31:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.693+0000 7f9909312640 1 -- 172.21.15.28:0/2921151 learned_addr learned my addr 172.21.15.28:0/2921151 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.692+0000 7f9909b13640 1 --2- 172.21.15.28:0/2921151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f99041006a0 0x7f9904197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.693+0000 7f990a314640 1 --2- 172.21.15.28:0/2921151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9904102250 0x7f990419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.693+0000 7f9909312640 1 -- 172.21.15.28:0/2921151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f99041006a0 msgr2=0x7f9904197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.693+0000 7f9909312640 1 --2- 172.21.15.28:0/2921151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f99041006a0 0x7f9904197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.693+0000 7f9909312640 1 -- 172.21.15.28:0/2921151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9904102250 msgr2=0x7f990419f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:02.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.693+0000 7f9909312640 1 --2- 172.21.15.28:0/2921151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9904102250 0x7f990419f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.693+0000 7f9909312640 1 -- 172.21.15.28:0/2921151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990419f690 con 0x7f9904101890 2024-04-25T01:31:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.693+0000 7f9909312640 1 --2- 172.21.15.28:0/2921151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904101890 0x7f9904197fb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f98f4009230 tx=0x7f98f4031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.693+0000 7f98faffd640 1 -- 172.21.15.28:0/2921151 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f403f030 con 0x7f9904101890 2024-04-25T01:31:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.693+0000 7f990bd77640 1 -- 172.21.15.28:0/2921151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f990419f920 con 0x7f9904101890 2024-04-25T01:31:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.693+0000 7f98faffd640 1 -- 172.21.15.28:0/2921151 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98f4003ad0 con 0x7f9904101890 2024-04-25T01:31:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.693+0000 7f98faffd640 1 -- 172.21.15.28:0/2921151 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f4035070 con 0x7f9904101890 2024-04-25T01:31:02.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.693+0000 7f990bd77640 1 -- 172.21.15.28:0/2921151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f990419fe10 con 0x7f9904101890 2024-04-25T01:31:02.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.694+0000 7f990bd77640 1 -- 172.21.15.28:0/2921151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99040689b0 con 0x7f9904101890 2024-04-25T01:31:02.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.695+0000 7f98faffd640 1 -- 172.21.15.28:0/2921151 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f98f4042030 con 0x7f9904101890 2024-04-25T01:31:02.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.696+0000 7f98faffd640 1 --2- 172.21.15.28:0/2921151 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f98f00a2b90 0x7f98f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:02.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.696+0000 7f9909b13640 1 --2- 172.21.15.28:0/2921151 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f98f00a2b90 0x7f98f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:02.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.696+0000 7f98faffd640 1 -- 172.21.15.28:0/2921151 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f98f40efbb0 con 0x7f9904101890 2024-04-25T01:31:02.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.696+0000 7f9909b13640 1 --2- 172.21.15.28:0/2921151 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f98f00a2b90 0x7f98f00a5050 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f99041016f0 tx=0x7f9900005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:02.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.699+0000 7f98faffd640 1 -- 172.21.15.28:0/2921151 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f98f40bc280 con 0x7f9904101890 2024-04-25T01:31:02.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:02 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] up:active 2024-04-25T01:31:02.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:02 smithi130 ceph-mon[32702]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-25T01:31:02.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:02 smithi130 ceph-mon[32702]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T01:31:02.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:02 smithi130 ceph-mon[32702]: Cluster is now healthy 2024-04-25T01:31:02.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:02 smithi130 ceph-mon[32702]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2024-04-25T01:31:02.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:02 smithi130 ceph-mon[32702]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-25T01:31:02.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:02 smithi130 ceph-mon[32702]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-04-25T01:31:02.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:02 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] up:active 2024-04-25T01:31:02.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:02 smithi130 ceph-mon[32702]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:31:02.709 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:02 smithi130 ceph-mon[32702]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:31:02.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:02 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] up:active 2024-04-25T01:31:02.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:02 smithi032 ceph-mon[32461]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-25T01:31:02.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:02 smithi032 ceph-mon[32461]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T01:31:02.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:02 smithi032 ceph-mon[32461]: Cluster is now healthy 2024-04-25T01:31:02.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:02 smithi032 ceph-mon[32461]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2024-04-25T01:31:02.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:02 smithi032 ceph-mon[32461]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-25T01:31:02.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:02 smithi032 ceph-mon[32461]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-04-25T01:31:02.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:02 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] up:active 2024-04-25T01:31:02.734 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:02 smithi032 ceph-mon[32461]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:31:02.735 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:02 smithi032 ceph-mon[32461]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:31:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.818+0000 7f98faffd640 1 -- 172.21.15.28:0/2921151 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f98f403b460 con 0x7f9904101890 2024-04-25T01:31:02.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.920+0000 7f990bd77640 1 -- 172.21.15.28:0/2921151 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f990406a450 con 0x7f98f00a2b90 2024-04-25T01:31:02.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.925+0000 7f98faffd640 1 -- 172.21.15.28:0/2921151 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f990406a450 con 0x7f98f00a2b90 2024-04-25T01:31:02.903 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_1/bc0d796f-3fd7-48fb-b5d6-fa28c50c6182 2024-04-25T01:31:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.927+0000 7f990bd77640 1 -- 172.21.15.28:0/2921151 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f98f00a2b90 msgr2=0x7f98f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.927+0000 7f990bd77640 1 --2- 172.21.15.28:0/2921151 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f98f00a2b90 0x7f98f00a5050 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f99041016f0 tx=0x7f9900005f90 comp rx=0 tx=0).stop 2024-04-25T01:31:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.927+0000 7f990bd77640 1 -- 172.21.15.28:0/2921151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904101890 msgr2=0x7f9904197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.927+0000 7f990bd77640 1 --2- 172.21.15.28:0/2921151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904101890 0x7f9904197fb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f98f4009230 tx=0x7f98f4031b60 comp rx=0 tx=0).stop 2024-04-25T01:31:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.928+0000 7f990bd77640 1 -- 172.21.15.28:0/2921151 shutdown_connections 2024-04-25T01:31:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.928+0000 7f990bd77640 1 --2- 172.21.15.28:0/2921151 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f98f00a2b90 0x7f98f00a5050 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.928+0000 7f990bd77640 1 --2- 172.21.15.28:0/2921151 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9904102250 0x7f990419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.928+0000 7f990bd77640 1 --2- 172.21.15.28:0/2921151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904101890 0x7f9904197fb0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.928+0000 7f990bd77640 1 --2- 172.21.15.28:0/2921151 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f99041006a0 0x7f9904197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.928+0000 7f990bd77640 1 -- 172.21.15.28:0/2921151 >> 172.21.15.28:0/2921151 conn(0x7f99040fbc30 msgr2=0x7f9904069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.928+0000 7f990bd77640 1 -- 172.21.15.28:0/2921151 shutdown_connections 2024-04-25T01:31:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:02.928+0000 7f990bd77640 1 -- 172.21.15.28:0/2921151 wait complete. 2024-04-25T01:31:02.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:31:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.041+0000 7fcc19d3b640 1 -- 172.21.15.28:0/1512850964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc141006a0 msgr2=0x7fcc14100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.041+0000 7fcc19d3b640 1 --2- 172.21.15.28:0/1512850964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc141006a0 0x7fcc14100ac0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcc04009900 tx=0x7fcc0402a410 comp rx=0 tx=0).stop 2024-04-25T01:31:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.041+0000 7fcc19d3b640 1 -- 172.21.15.28:0/1512850964 shutdown_connections 2024-04-25T01:31:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.041+0000 7fcc19d3b640 1 --2- 172.21.15.28:0/1512850964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc14102250 0x7fcc14108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.041+0000 7fcc19d3b640 1 --2- 172.21.15.28:0/1512850964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc14101890 0x7fcc14101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.041+0000 7fcc19d3b640 1 --2- 172.21.15.28:0/1512850964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc141006a0 0x7fcc14100ac0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.041+0000 7fcc19d3b640 1 -- 172.21.15.28:0/1512850964 >> 172.21.15.28:0/1512850964 conn(0x7fcc140fbc30 msgr2=0x7fcc140fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.041+0000 7fcc19d3b640 1 -- 172.21.15.28:0/1512850964 shutdown_connections 2024-04-25T01:31:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.041+0000 7fcc19d3b640 1 -- 172.21.15.28:0/1512850964 wait complete. 2024-04-25T01:31:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.041+0000 7fcc19d3b640 1 Processor -- start 2024-04-25T01:31:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.042+0000 7fcc19d3b640 1 -- start start 2024-04-25T01:31:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.042+0000 7fcc19d3b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc141006a0 0x7fcc14197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.042+0000 7fcc19d3b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc14101890 0x7fcc14198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.042+0000 7fcc19d3b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc14102250 0x7fcc1419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.042+0000 7fcc19d3b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1410c2f0 con 0x7fcc141006a0 2024-04-25T01:31:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.042+0000 7fcc19d3b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1410c170 con 0x7fcc14102250 2024-04-25T01:31:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.042+0000 7fcc19d3b640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1410c470 con 0x7fcc14101890 2024-04-25T01:31:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.042+0000 7fcc137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc141006a0 0x7fcc14197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.042+0000 7fcc137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc141006a0 0x7fcc14197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52082/0 (socket says 172.21.15.28:52082) 2024-04-25T01:31:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.042+0000 7fcc137fe640 1 -- 172.21.15.28:0/1676418052 learned_addr learned my addr 172.21.15.28:0/1676418052 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.042+0000 7fcc13fff640 1 --2- 172.21.15.28:0/1676418052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc14102250 0x7fcc1419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.042+0000 7fcc12ffd640 1 --2- 172.21.15.28:0/1676418052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc14101890 0x7fcc14198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.042+0000 7fcc137fe640 1 -- 172.21.15.28:0/1676418052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc14101890 msgr2=0x7fcc14198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.042+0000 7fcc137fe640 1 --2- 172.21.15.28:0/1676418052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc14101890 0x7fcc14198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.042+0000 7fcc137fe640 1 -- 172.21.15.28:0/1676418052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc14102250 msgr2=0x7fcc1419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.042+0000 7fcc137fe640 1 --2- 172.21.15.28:0/1676418052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc14102250 0x7fcc1419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.042+0000 7fcc137fe640 1 -- 172.21.15.28:0/1676418052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1419f6f0 con 0x7fcc141006a0 2024-04-25T01:31:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.043+0000 7fcc137fe640 1 --2- 172.21.15.28:0/1676418052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc141006a0 0x7fcc14197b40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fcc04031610 tx=0x7fcc040043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.043+0000 7fcc10ff9640 1 -- 172.21.15.28:0/1676418052 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0403f030 con 0x7fcc141006a0 2024-04-25T01:31:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.043+0000 7fcc19d3b640 1 -- 172.21.15.28:0/1676418052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc1419f980 con 0x7fcc141006a0 2024-04-25T01:31:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.043+0000 7fcc10ff9640 1 -- 172.21.15.28:0/1676418052 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc04003ce0 con 0x7fcc141006a0 2024-04-25T01:31:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.043+0000 7fcc19d3b640 1 -- 172.21.15.28:0/1676418052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc1419fe70 con 0x7fcc141006a0 2024-04-25T01:31:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.043+0000 7fcc10ff9640 1 -- 172.21.15.28:0/1676418052 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc04049060 con 0x7fcc141006a0 2024-04-25T01:31:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.044+0000 7fcc19d3b640 1 -- 172.21.15.28:0/1676418052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc141a00a0 con 0x7fcc141006a0 2024-04-25T01:31:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.045+0000 7fcc10ff9640 1 -- 172.21.15.28:0/1676418052 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fcc04042030 con 0x7fcc141006a0 2024-04-25T01:31:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.046+0000 7fcc10ff9640 1 --2- 172.21.15.28:0/1676418052 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fcbfc0a2c20 0x7fcbfc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.046+0000 7fcc12ffd640 1 --2- 172.21.15.28:0/1676418052 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fcbfc0a2c20 0x7fcbfc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.046+0000 7fcc10ff9640 1 -- 172.21.15.28:0/1676418052 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fcc040f0080 con 0x7fcc141006a0 2024-04-25T01:31:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.046+0000 7fcc12ffd640 1 --2- 172.21.15.28:0/1676418052 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fcbfc0a2c20 0x7fcbfc0a50e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcc08005e10 tx=0x7fcc08005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:03.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.050+0000 7fcc10ff9640 1 -- 172.21.15.28:0/1676418052 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcc040bcbe0 con 0x7fcc141006a0 2024-04-25T01:31:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.272+0000 7fcc19d3b640 1 -- 172.21.15.28:0/1676418052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc141a0270 con 0x7fcc141006a0 2024-04-25T01:31:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.273+0000 7fcc10ff9640 1 -- 172.21.15.28:0/1676418052 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11923 (secure 0 0 0) 0x7fcc040bc4e0 con 0x7fcc141006a0 2024-04-25T01:31:03.251 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:31:03.251 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:31:01.802718+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24454,24442,24512,24478,24473]},"id":1}]} 2024-04-25T01:31:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-04-25T01:31:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.275+0000 7fcc19d3b640 1 -- 172.21.15.28:0/1676418052 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fcbfc0a2c20 msgr2=0x7fcbfc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.275+0000 7fcc19d3b640 1 --2- 172.21.15.28:0/1676418052 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fcbfc0a2c20 0x7fcbfc0a50e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcc08005e10 tx=0x7fcc08005da0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.275+0000 7fcc19d3b640 1 -- 172.21.15.28:0/1676418052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc141006a0 msgr2=0x7fcc14197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.275+0000 7fcc19d3b640 1 --2- 172.21.15.28:0/1676418052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc141006a0 0x7fcc14197b40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fcc04031610 tx=0x7fcc040043b0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.275+0000 7fcc19d3b640 1 -- 172.21.15.28:0/1676418052 shutdown_connections 2024-04-25T01:31:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.275+0000 7fcc19d3b640 1 --2- 172.21.15.28:0/1676418052 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fcbfc0a2c20 0x7fcbfc0a50e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.275+0000 7fcc19d3b640 1 --2- 172.21.15.28:0/1676418052 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc14102250 0x7fcc1419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.275+0000 7fcc19d3b640 1 --2- 172.21.15.28:0/1676418052 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc14101890 0x7fcc14198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.275+0000 7fcc19d3b640 1 --2- 172.21.15.28:0/1676418052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc141006a0 0x7fcc14197b40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.275+0000 7fcc19d3b640 1 -- 172.21.15.28:0/1676418052 >> 172.21.15.28:0/1676418052 conn(0x7fcc140fbc30 msgr2=0x7fcc140fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.275+0000 7fcc19d3b640 1 -- 172.21.15.28:0/1676418052 shutdown_connections 2024-04-25T01:31:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.275+0000 7fcc19d3b640 1 -- 172.21.15.28:0/1676418052 wait complete. 2024-04-25T01:31:03.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:31:03.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.389+0000 7fbae4264640 1 -- 172.21.15.28:0/2145110663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbadc107480 msgr2=0x7fbadc1098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:03.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.389+0000 7fbae4264640 1 --2- 172.21.15.28:0/2145110663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbadc107480 0x7fbadc1098d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbad4009900 tx=0x7fbad402a410 comp rx=0 tx=0).stop 2024-04-25T01:31:03.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.389+0000 7fbae4264640 1 -- 172.21.15.28:0/2145110663 shutdown_connections 2024-04-25T01:31:03.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.389+0000 7fbae4264640 1 --2- 172.21.15.28:0/2145110663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbadc107480 0x7fbadc1098d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.389+0000 7fbae4264640 1 --2- 172.21.15.28:0/2145110663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbadc104b50 0x7fbadc106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.389+0000 7fbae4264640 1 --2- 172.21.15.28:0/2145110663 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbadc102220 0x7fbadc104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.389+0000 7fbae4264640 1 -- 172.21.15.28:0/2145110663 >> 172.21.15.28:0/2145110663 conn(0x7fbadc0fbc30 msgr2=0x7fbadc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:03.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.389+0000 7fbae4264640 1 -- 172.21.15.28:0/2145110663 shutdown_connections 2024-04-25T01:31:03.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.389+0000 7fbae4264640 1 -- 172.21.15.28:0/2145110663 wait complete. 2024-04-25T01:31:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.389+0000 7fbae4264640 1 Processor -- start 2024-04-25T01:31:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbae4264640 1 -- start start 2024-04-25T01:31:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbae4264640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbadc102220 0x7fbadc197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbae4264640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbadc104b50 0x7fbadc197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbae4264640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbadc107480 0x7fbadc19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbae4264640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc10d420 con 0x7fbadc102220 2024-04-25T01:31:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbae4264640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc10d2a0 con 0x7fbadc107480 2024-04-25T01:31:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbae4264640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc10d5a0 con 0x7fbadc104b50 2024-04-25T01:31:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbae2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbadc102220 0x7fbadc197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbae2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbadc102220 0x7fbadc197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52094/0 (socket says 172.21.15.28:52094) 2024-04-25T01:31:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbae2000640 1 -- 172.21.15.28:0/2992089776 learned_addr learned my addr 172.21.15.28:0/2992089776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbae2801640 1 --2- 172.21.15.28:0/2992089776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbadc107480 0x7fbadc19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbae2000640 1 -- 172.21.15.28:0/2992089776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbadc104b50 msgr2=0x7fbadc197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbae2000640 1 --2- 172.21.15.28:0/2992089776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbadc104b50 0x7fbadc197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbae2000640 1 -- 172.21.15.28:0/2992089776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbadc107480 msgr2=0x7fbadc19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:03.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbae2000640 1 --2- 172.21.15.28:0/2992089776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbadc107480 0x7fbadc19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbae2000640 1 -- 172.21.15.28:0/2992089776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbadc19f660 con 0x7fbadc102220 2024-04-25T01:31:03.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbae2801640 1 --2- 172.21.15.28:0/2992089776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbadc107480 0x7fbadc19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:03.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbae2000640 1 --2- 172.21.15.28:0/2992089776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbadc102220 0x7fbadc197a40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbad800ead0 tx=0x7fbad800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:03.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbad2ffd640 1 -- 172.21.15.28:0/2992089776 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad800ce40 con 0x7fbadc102220 2024-04-25T01:31:03.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbae4264640 1 -- 172.21.15.28:0/2992089776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbadc19f950 con 0x7fbadc102220 2024-04-25T01:31:03.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbad2ffd640 1 -- 172.21.15.28:0/2992089776 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbad8003e50 con 0x7fbadc102220 2024-04-25T01:31:03.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.390+0000 7fbad2ffd640 1 -- 172.21.15.28:0/2992089776 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad8011760 con 0x7fbadc102220 2024-04-25T01:31:03.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.391+0000 7fbae4264640 1 -- 172.21.15.28:0/2992089776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbadc19fe50 con 0x7fbadc102220 2024-04-25T01:31:03.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.392+0000 7fbae4264640 1 -- 172.21.15.28:0/2992089776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbadc0fd720 con 0x7fbadc102220 2024-04-25T01:31:03.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.393+0000 7fbad2ffd640 1 -- 172.21.15.28:0/2992089776 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fbad8004020 con 0x7fbadc102220 2024-04-25T01:31:03.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.393+0000 7fbad2ffd640 1 --2- 172.21.15.28:0/2992089776 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbac80a2c20 0x7fbac80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:03.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.393+0000 7fbae17ff640 1 --2- 172.21.15.28:0/2992089776 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbac80a2c20 0x7fbac80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:03.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.393+0000 7fbad2ffd640 1 -- 172.21.15.28:0/2992089776 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fbad80cf750 con 0x7fbadc102220 2024-04-25T01:31:03.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.394+0000 7fbae17ff640 1 --2- 172.21.15.28:0/2992089776 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbac80a2c20 0x7fbac80a50e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbacc0058c0 tx=0x7fbacc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:03.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.398+0000 7fbad2ffd640 1 -- 172.21.15.28:0/2992089776 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbad8016030 con 0x7fbadc102220 2024-04-25T01:31:03.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.584+0000 7fbae4264640 1 -- 172.21.15.28:0/2992089776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbadc02d420 con 0x7fbadc102220 2024-04-25T01:31:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.586+0000 7fbad2ffd640 1 -- 172.21.15.28:0/2992089776 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24093 (secure 0 0 0) 0x7fbad809c2b0 con 0x7fbadc102220 2024-04-25T01:31:03.564 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:31:03.564 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:30:38.730329+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:31:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.588+0000 7fbae4264640 1 -- 172.21.15.28:0/2992089776 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbac80a2c20 msgr2=0x7fbac80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.588+0000 7fbae4264640 1 --2- 172.21.15.28:0/2992089776 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbac80a2c20 0x7fbac80a50e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbacc0058c0 tx=0x7fbacc005850 comp rx=0 tx=0).stop 2024-04-25T01:31:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.588+0000 7fbae4264640 1 -- 172.21.15.28:0/2992089776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbadc102220 msgr2=0x7fbadc197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.588+0000 7fbae4264640 1 --2- 172.21.15.28:0/2992089776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbadc102220 0x7fbadc197a40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fbad800ead0 tx=0x7fbad800ef90 comp rx=0 tx=0).stop 2024-04-25T01:31:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.588+0000 7fbae4264640 1 -- 172.21.15.28:0/2992089776 shutdown_connections 2024-04-25T01:31:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.588+0000 7fbae4264640 1 --2- 172.21.15.28:0/2992089776 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbac80a2c20 0x7fbac80a50e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.588+0000 7fbae4264640 1 --2- 172.21.15.28:0/2992089776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbadc107480 0x7fbadc19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.588+0000 7fbae4264640 1 --2- 172.21.15.28:0/2992089776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbadc104b50 0x7fbadc197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.588+0000 7fbae4264640 1 --2- 172.21.15.28:0/2992089776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbadc102220 0x7fbadc197a40 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.588+0000 7fbae4264640 1 -- 172.21.15.28:0/2992089776 >> 172.21.15.28:0/2992089776 conn(0x7fbadc0fbc30 msgr2=0x7fbadc102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.588+0000 7fbae4264640 1 -- 172.21.15.28:0/2992089776 shutdown_connections 2024-04-25T01:31:03.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.588+0000 7fbae4264640 1 -- 172.21.15.28:0/2992089776 wait complete. 2024-04-25T01:31:03.577 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T01:31:03.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:31:03.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.701+0000 7fc27d82a640 1 -- 172.21.15.28:0/2752382239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc278103360 msgr2=0x7fc278109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.701+0000 7fc27d82a640 1 --2- 172.21.15.28:0/2752382239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc278103360 0x7fc278109bf0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc270009ca0 tx=0x7fc27002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.701+0000 7fc27d82a640 1 -- 172.21.15.28:0/2752382239 shutdown_connections 2024-04-25T01:31:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.701+0000 7fc27d82a640 1 --2- 172.21.15.28:0/2752382239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc278103360 0x7fc278109bf0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.701+0000 7fc27d82a640 1 --2- 172.21.15.28:0/2752382239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc2781029a0 0x7fc278102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.701+0000 7fc27d82a640 1 --2- 172.21.15.28:0/2752382239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2781017b0 0x7fc278101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.701+0000 7fc27d82a640 1 -- 172.21.15.28:0/2752382239 >> 172.21.15.28:0/2752382239 conn(0x7fc2780fcd40 msgr2=0x7fc2780ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.701+0000 7fc27d82a640 1 -- 172.21.15.28:0/2752382239 shutdown_connections 2024-04-25T01:31:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.701+0000 7fc27d82a640 1 -- 172.21.15.28:0/2752382239 wait complete. 2024-04-25T01:31:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.701+0000 7fc27d82a640 1 Processor -- start 2024-04-25T01:31:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.701+0000 7fc27d82a640 1 -- start start 2024-04-25T01:31:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.701+0000 7fc27d82a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2781017b0 0x7fc278197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc27d82a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2781029a0 0x7fc278197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc27d82a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc278103360 0x7fc27819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc27d82a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc27810d430 con 0x7fc2781029a0 2024-04-25T01:31:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc27d82a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc27810d2b0 con 0x7fc2781017b0 2024-04-25T01:31:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc27d82a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc27810d5b0 con 0x7fc278103360 2024-04-25T01:31:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc2767fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2781029a0 0x7fc278197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc2767fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2781029a0 0x7fc278197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52112/0 (socket says 172.21.15.28:52112) 2024-04-25T01:31:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc2767fc640 1 -- 172.21.15.28:0/2655719976 learned_addr learned my addr 172.21.15.28:0/2655719976 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc276ffd640 1 --2- 172.21.15.28:0/2655719976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2781017b0 0x7fc278197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc2777fe640 1 --2- 172.21.15.28:0/2655719976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc278103360 0x7fc27819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc2767fc640 1 -- 172.21.15.28:0/2655719976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc278103360 msgr2=0x7fc27819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc2767fc640 1 --2- 172.21.15.28:0/2655719976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc278103360 0x7fc27819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc2767fc640 1 -- 172.21.15.28:0/2655719976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2781017b0 msgr2=0x7fc278197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc2767fc640 1 --2- 172.21.15.28:0/2655719976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2781017b0 0x7fc278197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc2767fc640 1 -- 172.21.15.28:0/2655719976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc27819f690 con 0x7fc2781029a0 2024-04-25T01:31:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc2777fe640 1 --2- 172.21.15.28:0/2655719976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc278103360 0x7fc27819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc2767fc640 1 --2- 172.21.15.28:0/2655719976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2781029a0 0x7fc278197fb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc26c00d770 tx=0x7fc26c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc27c828640 1 -- 172.21.15.28:0/2655719976 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26c017070 con 0x7fc2781029a0 2024-04-25T01:31:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc27c828640 1 -- 172.21.15.28:0/2655719976 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc26c005500 con 0x7fc2781029a0 2024-04-25T01:31:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc27c828640 1 -- 172.21.15.28:0/2655719976 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26c004e00 con 0x7fc2781029a0 2024-04-25T01:31:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc27d82a640 1 -- 172.21.15.28:0/2655719976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc27819f980 con 0x7fc2781029a0 2024-04-25T01:31:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.702+0000 7fc27d82a640 1 -- 172.21.15.28:0/2655719976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc27819fe80 con 0x7fc2781029a0 2024-04-25T01:31:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.703+0000 7fc27d82a640 1 -- 172.21.15.28:0/2655719976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc278108a70 con 0x7fc2781029a0 2024-04-25T01:31:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.704+0000 7fc27c828640 1 -- 172.21.15.28:0/2655719976 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fc26c004fa0 con 0x7fc2781029a0 2024-04-25T01:31:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.705+0000 7fc27c828640 1 --2- 172.21.15.28:0/2655719976 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc2600a2c20 0x7fc2600a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.705+0000 7fc27c828640 1 -- 172.21.15.28:0/2655719976 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fc26c0cede0 con 0x7fc2781029a0 2024-04-25T01:31:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.705+0000 7fc276ffd640 1 --2- 172.21.15.28:0/2655719976 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc2600a2c20 0x7fc2600a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.706+0000 7fc276ffd640 1 --2- 172.21.15.28:0/2655719976 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc2600a2c20 0x7fc2600a50e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc268005fd0 tx=0x7fc26800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.709+0000 7fc27c828640 1 -- 172.21.15.28:0/2655719976 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc26c09b9a0 con 0x7fc2781029a0 2024-04-25T01:31:03.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.892+0000 7fc27d82a640 1 -- 172.21.15.28:0/2655719976 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc278061b80 con 0x7fc2600a2c20 2024-04-25T01:31:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.903+0000 7fc27c828640 1 -- 172.21.15.28:0/2655719976 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340391 (secure 0 0 0) 0x7fc278061b80 con 0x7fc2600a2c20 2024-04-25T01:31:03.885 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:31:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T01:31:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.906+0000 7fc27d82a640 1 -- 172.21.15.28:0/2655719976 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc2600a2c20 msgr2=0x7fc2600a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.906+0000 7fc27d82a640 1 --2- 172.21.15.28:0/2655719976 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc2600a2c20 0x7fc2600a50e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc268005fd0 tx=0x7fc26800b410 comp rx=0 tx=0).stop 2024-04-25T01:31:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.906+0000 7fc27d82a640 1 -- 172.21.15.28:0/2655719976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2781029a0 msgr2=0x7fc278197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.906+0000 7fc27d82a640 1 --2- 172.21.15.28:0/2655719976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2781029a0 0x7fc278197fb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc26c00d770 tx=0x7fc26c00dc30 comp rx=0 tx=0).stop 2024-04-25T01:31:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.906+0000 7fc27d82a640 1 -- 172.21.15.28:0/2655719976 shutdown_connections 2024-04-25T01:31:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.906+0000 7fc27d82a640 1 --2- 172.21.15.28:0/2655719976 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc2600a2c20 0x7fc2600a50e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.906+0000 7fc27d82a640 1 --2- 172.21.15.28:0/2655719976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc278103360 0x7fc27819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.906+0000 7fc27d82a640 1 --2- 172.21.15.28:0/2655719976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2781029a0 0x7fc278197fb0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.906+0000 7fc27d82a640 1 --2- 172.21.15.28:0/2655719976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2781017b0 0x7fc278197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.906+0000 7fc27d82a640 1 -- 172.21.15.28:0/2655719976 >> 172.21.15.28:0/2655719976 conn(0x7fc2780fcd40 msgr2=0x7fc278105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.906+0000 7fc27d82a640 1 -- 172.21.15.28:0/2655719976 shutdown_connections 2024-04-25T01:31:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:03.906+0000 7fc27d82a640 1 -- 172.21.15.28:0/2655719976 wait complete. 2024-04-25T01:31:03.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:03 smithi028 ceph-mon[29390]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:31:03.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:03 smithi028 ceph-mon[29390]: pgmap v312: 129 pgs: 129 active+clean; 583 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 5 op/s 2024-04-25T01:31:03.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:03 smithi028 ceph-mon[29390]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:31:03.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:03 smithi028 ceph-mon[29390]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T01:31:03.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:03 smithi028 ceph-mon[29390]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:31:03.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:03 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1676418052' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:31:03.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:03 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2992089776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:31:03.896 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2024-04-25T01:31:02.335182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596616,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":290,"num_write_kb":2978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":379488,"kb_used_data":13652,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124562336,"statfs":{"total":1151940427776,"available":1151551832064,"internally_reserved":0,"allocated":13979648,"data_stored":7613062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1749,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003882"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.954172+0000","last_change":"2024-04-25T01:30:34.358339+0000","last_active":"2024-04-25T01:30:38.954172+0000","last_peered":"2024-04-25T01:30:38.954172+0000","last_clean":"2024-04-25T01:30:38.954172+0000","last_became_active":"2024-04-25T01:30:34.357884+0000","last_became_peered":"2024-04-25T01:30:34.357884+0000","last_unstale":"2024-04-25T01:30:38.954172+0000","last_undegraded":"2024-04-25T01:30:38.954172+0000","last_fullsized":"2024-04-25T01:30:38.954172+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:56.409080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114521+0000","last_change":"2024-04-25T01:30:34.359933+0000","last_active":"2024-04-25T01:30:39.114521+0000","last_peered":"2024-04-25T01:30:39.114521+0000","last_clean":"2024-04-25T01:30:39.114521+0000","last_became_active":"2024-04-25T01:30:34.359576+0000","last_became_peered":"2024-04-25T01:30:34.359576+0000","last_unstale":"2024-04-25T01:30:39.114521+0000","last_undegraded":"2024-04-25T01:30:39.114521+0000","last_fullsized":"2024-04-25T01:30:39.114521+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:48:47.764121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114072+0000","last_change":"2024-04-25T01:30:34.357772+0000","last_active":"2024-04-25T01:30:39.114072+0000","last_peered":"2024-04-25T01:30:39.114072+0000","last_clean":"2024-04-25T01:30:39.114072+0000","last_became_active":"2024-04-25T01:30:34.357434+0000","last_became_peered":"2024-04-25T01:30:34.357434+0000","last_unstale":"2024-04-25T01:30:39.114072+0000","last_undegraded":"2024-04-25T01:30:39.114072+0000","last_fullsized":"2024-04-25T01:30:39.114072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:19.926180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804914+0000","last_change":"2024-04-25T01:30:34.351250+0000","last_active":"2024-04-25T01:30:38.804914+0000","last_peered":"2024-04-25T01:30:38.804914+0000","last_clean":"2024-04-25T01:30:38.804914+0000","last_became_active":"2024-04-25T01:30:34.349976+0000","last_became_peered":"2024-04-25T01:30:34.349976+0000","last_unstale":"2024-04-25T01:30:38.804914+0000","last_undegraded":"2024-04-25T01:30:38.804914+0000","last_fullsized":"2024-04-25T01:30:38.804914+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:19:51.605307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114119+0000","last_change":"2024-04-25T01:30:34.367882+0000","last_active":"2024-04-25T01:30:39.114119+0000","last_peered":"2024-04-25T01:30:39.114119+0000","last_clean":"2024-04-25T01:30:39.114119+0000","last_became_active":"2024-04-25T01:30:34.366797+0000","last_became_peered":"2024-04-25T01:30:34.366797+0000","last_unstale":"2024-04-25T01:30:39.114119+0000","last_undegraded":"2024-04-25T01:30:39.114119+0000","last_fullsized":"2024-04-25T01:30:39.114119+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:35:23.439061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.116169+0000","last_change":"2024-04-25T01:30:34.352506+0000","last_active":"2024-04-25T01:30:39.116169+0000","last_peered":"2024-04-25T01:30:39.116169+0000","last_clean":"2024-04-25T01:30:39.116169+0000","last_became_active":"2024-04-25T01:30:34.352100+0000","last_became_peered":"2024-04-25T01:30:34.352100+0000","last_unstale":"2024-04-25T01:30:39.116169+0000","last_undegraded":"2024-04-25T01:30:39.116169+0000","last_fullsized":"2024-04-25T01:30:39.116169+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:38.113959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114501+0000","last_change":"2024-04-25T01:30:34.355352+0000","last_active":"2024-04-25T01:30:39.114501+0000","last_peered":"2024-04-25T01:30:39.114501+0000","last_clean":"2024-04-25T01:30:39.114501+0000","last_became_active":"2024-04-25T01:30:34.355008+0000","last_became_peered":"2024-04-25T01:30:34.355008+0000","last_unstale":"2024-04-25T01:30:39.114501+0000","last_undegraded":"2024-04-25T01:30:39.114501+0000","last_fullsized":"2024-04-25T01:30:39.114501+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:23.582742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114692+0000","last_change":"2024-04-25T01:30:34.360644+0000","last_active":"2024-04-25T01:30:39.114692+0000","last_peered":"2024-04-25T01:30:39.114692+0000","last_clean":"2024-04-25T01:30:39.114692+0000","last_became_active":"2024-04-25T01:30:34.357841+0000","last_became_peered":"2024-04-25T01:30:34.357841+0000","last_unstale":"2024-04-25T01:30:39.114692+0000","last_undegraded":"2024-04-25T01:30:39.114692+0000","last_fullsized":"2024-04-25T01:30:39.114692+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:26:21.606219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115818+0000","last_change":"2024-04-25T01:30:34.358417+0000","last_active":"2024-04-25T01:30:39.115818+0000","last_peered":"2024-04-25T01:30:39.115818+0000","last_clean":"2024-04-25T01:30:39.115818+0000","last_became_active":"2024-04-25T01:30:34.358039+0000","last_became_peered":"2024-04-25T01:30:34.358039+0000","last_unstale":"2024-04-25T01:30:39.115818+0000","last_undegraded":"2024-04-25T01:30:39.115818+0000","last_fullsized":"2024-04-25T01:30:39.115818+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:42.600693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.116709+0000","last_change":"2024-04-25T01:30:34.357803+0000","last_active":"2024-04-25T01:30:39.116709+0000","last_peered":"2024-04-25T01:30:39.116709+0000","last_clean":"2024-04-25T01:30:39.116709+0000","last_became_active":"2024-04-25T01:30:34.357391+0000","last_became_peered":"2024-04-25T01:30:34.357391+0000","last_unstale":"2024-04-25T01:30:39.116709+0000","last_undegraded":"2024-04-25T01:30:39.116709+0000","last_fullsized":"2024-04-25T01:30:39.116709+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:57.857303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115350+0000","last_change":"2024-04-25T01:30:34.363340+0000","last_active":"2024-04-25T01:30:39.115350+0000","last_peered":"2024-04-25T01:30:39.115350+0000","last_clean":"2024-04-25T01:30:39.115350+0000","last_became_active":"2024-04-25T01:30:34.362980+0000","last_became_peered":"2024-04-25T01:30:34.362980+0000","last_unstale":"2024-04-25T01:30:39.115350+0000","last_undegraded":"2024-04-25T01:30:39.115350+0000","last_fullsized":"2024-04-25T01:30:39.115350+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:46.291459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804860+0000","last_change":"2024-04-25T01:30:34.358898+0000","last_active":"2024-04-25T01:30:38.804860+0000","last_peered":"2024-04-25T01:30:38.804860+0000","last_clean":"2024-04-25T01:30:38.804860+0000","last_became_active":"2024-04-25T01:30:34.358495+0000","last_became_peered":"2024-04-25T01:30:34.358495+0000","last_unstale":"2024-04-25T01:30:38.804860+0000","last_undegraded":"2024-04-25T01:30:38.804860+0000","last_fullsized":"2024-04-25T01:30:38.804860+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:34:21.860010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.454881+0000","last_change":"2024-04-25T01:30:34.363708+0000","last_active":"2024-04-25T01:30:39.454881+0000","last_peered":"2024-04-25T01:30:39.454881+0000","last_clean":"2024-04-25T01:30:39.454881+0000","last_became_active":"2024-04-25T01:30:34.363207+0000","last_became_peered":"2024-04-25T01:30:34.363207+0000","last_unstale":"2024-04-25T01:30:39.454881+0000","last_undegraded":"2024-04-25T01:30:39.454881+0000","last_fullsized":"2024-04-25T01:30:39.454881+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:53.442987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114013+0000","last_change":"2024-04-25T01:30:34.371031+0000","last_active":"2024-04-25T01:30:39.114013+0000","last_peered":"2024-04-25T01:30:39.114013+0000","last_clean":"2024-04-25T01:30:39.114013+0000","last_became_active":"2024-04-25T01:30:34.370687+0000","last_became_peered":"2024-04-25T01:30:34.370687+0000","last_unstale":"2024-04-25T01:30:39.114013+0000","last_undegraded":"2024-04-25T01:30:39.114013+0000","last_fullsized":"2024-04-25T01:30:39.114013+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:03:20.700960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114529+0000","last_change":"2024-04-25T01:30:34.366212+0000","last_active":"2024-04-25T01:30:39.114529+0000","last_peered":"2024-04-25T01:30:39.114529+0000","last_clean":"2024-04-25T01:30:39.114529+0000","last_became_active":"2024-04-25T01:30:34.365849+0000","last_became_peered":"2024-04-25T01:30:34.365849+0000","last_unstale":"2024-04-25T01:30:39.114529+0000","last_undegraded":"2024-04-25T01:30:39.114529+0000","last_fullsized":"2024-04-25T01:30:39.114529+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:24:02.397615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114257+0000","last_change":"2024-04-25T01:30:34.367380+0000","last_active":"2024-04-25T01:30:39.114257+0000","last_peered":"2024-04-25T01:30:39.114257+0000","last_clean":"2024-04-25T01:30:39.114257+0000","last_became_active":"2024-04-25T01:30:34.366872+0000","last_became_peered":"2024-04-25T01:30:34.366872+0000","last_unstale":"2024-04-25T01:30:39.114257+0000","last_undegraded":"2024-04-25T01:30:39.114257+0000","last_fullsized":"2024-04-25T01:30:39.114257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:18.767108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.954033+0000","last_change":"2024-04-25T01:30:34.359752+0000","last_active":"2024-04-25T01:30:38.954033+0000","last_peered":"2024-04-25T01:30:38.954033+0000","last_clean":"2024-04-25T01:30:38.954033+0000","last_became_active":"2024-04-25T01:30:34.358403+0000","last_became_peered":"2024-04-25T01:30:34.358403+0000","last_unstale":"2024-04-25T01:30:38.954033+0000","last_undegraded":"2024-04-25T01:30:38.954033+0000","last_fullsized":"2024-04-25T01:30:38.954033+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:14.833126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.803536+0000","last_change":"2024-04-25T01:30:34.361373+0000","last_active":"2024-04-25T01:30:38.803536+0000","last_peered":"2024-04-25T01:30:38.803536+0000","last_clean":"2024-04-25T01:30:38.803536+0000","last_became_active":"2024-04-25T01:30:34.361046+0000","last_became_peered":"2024-04-25T01:30:34.361046+0000","last_unstale":"2024-04-25T01:30:38.803536+0000","last_undegraded":"2024-04-25T01:30:38.803536+0000","last_fullsized":"2024-04-25T01:30:38.803536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:21.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.805495+0000","last_change":"2024-04-25T01:30:34.367975+0000","last_active":"2024-04-25T01:30:38.805495+0000","last_peered":"2024-04-25T01:30:38.805495+0000","last_clean":"2024-04-25T01:30:38.805495+0000","last_became_active":"2024-04-25T01:30:34.366383+0000","last_became_peered":"2024-04-25T01:30:34.366383+0000","last_unstale":"2024-04-25T01:30:38.805495+0000","last_undegraded":"2024-04-25T01:30:38.805495+0000","last_fullsized":"2024-04-25T01:30:38.805495+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:53:40.279067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804884+0000","last_change":"2024-04-25T01:30:38.382168+0000","last_active":"2024-04-25T01:30:38.804884+0000","last_peered":"2024-04-25T01:30:38.804884+0000","last_clean":"2024-04-25T01:30:38.804884+0000","last_became_active":"2024-04-25T01:30:38.381740+0000","last_became_peered":"2024-04-25T01:30:38.381740+0000","last_unstale":"2024-04-25T01:30:38.804884+0000","last_undegraded":"2024-04-25T01:30:38.804884+0000","last_fullsized":"2024-04-25T01:30:38.804884+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:27.425991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.803674+0000","last_change":"2024-04-25T01:30:33.345547+0000","last_active":"2024-04-25T01:30:38.803674+0000","last_peered":"2024-04-25T01:30:38.803674+0000","last_clean":"2024-04-25T01:30:38.803674+0000","last_became_active":"2024-04-25T01:30:33.345229+0000","last_became_peered":"2024-04-25T01:30:33.345229+0000","last_unstale":"2024-04-25T01:30:38.803674+0000","last_undegraded":"2024-04-25T01:30:38.803674+0000","last_fullsized":"2024-04-25T01:30:38.803674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:32:46.637720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115066+0000","last_change":"2024-04-25T01:30:34.354703+0000","last_active":"2024-04-25T01:30:39.115066+0000","last_peered":"2024-04-25T01:30:39.115066+0000","last_clean":"2024-04-25T01:30:39.115066+0000","last_became_active":"2024-04-25T01:30:34.354362+0000","last_became_peered":"2024-04-25T01:30:34.354362+0000","last_unstale":"2024-04-25T01:30:39.115066+0000","last_undegraded":"2024-04-25T01:30:39.115066+0000","last_fullsized":"2024-04-25T01:30:39.115066+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:51.358599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804546+0000","last_change":"2024-04-25T01:30:38.380912+0000","last_active":"2024-04-25T01:30:38.804546+0000","last_peered":"2024-04-25T01:30:38.804546+0000","last_clean":"2024-04-25T01:30:38.804546+0000","last_became_active":"2024-04-25T01:30:38.380439+0000","last_became_peered":"2024-04-25T01:30:38.380439+0000","last_unstale":"2024-04-25T01:30:38.804546+0000","last_undegraded":"2024-04-25T01:30:38.804546+0000","last_fullsized":"2024-04-25T01:30:38.804546+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:22:00.527268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115074+0000","last_change":"2024-04-25T01:30:33.345620+0000","last_active":"2024-04-25T01:30:39.115074+0000","last_peered":"2024-04-25T01:30:39.115074+0000","last_clean":"2024-04-25T01:30:39.115074+0000","last_became_active":"2024-04-25T01:30:33.345315+0000","last_became_peered":"2024-04-25T01:30:33.345315+0000","last_unstale":"2024-04-25T01:30:39.115074+0000","last_undegraded":"2024-04-25T01:30:39.115074+0000","last_fullsized":"2024-04-25T01:30:39.115074+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:48:00.083728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114067+0000","last_change":"2024-04-25T01:30:34.356589+0000","last_active":"2024-04-25T01:30:39.114067+0000","last_peered":"2024-04-25T01:30:39.114067+0000","last_clean":"2024-04-25T01:30:39.114067+0000","last_became_active":"2024-04-25T01:30:34.356114+0000","last_became_peered":"2024-04-25T01:30:34.356114+0000","last_unstale":"2024-04-25T01:30:39.114067+0000","last_undegraded":"2024-04-25T01:30:39.114067+0000","last_fullsized":"2024-04-25T01:30:39.114067+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:39.039517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804557+0000","last_change":"2024-04-25T01:30:38.374656+0000","last_active":"2024-04-25T01:30:38.804557+0000","last_peered":"2024-04-25T01:30:38.804557+0000","last_clean":"2024-04-25T01:30:38.804557+0000","last_became_active":"2024-04-25T01:30:38.374189+0000","last_became_peered":"2024-04-25T01:30:38.374189+0000","last_unstale":"2024-04-25T01:30:38.804557+0000","last_undegraded":"2024-04-25T01:30:38.804557+0000","last_fullsized":"2024-04-25T01:30:38.804557+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:47:24.261071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:31:00.451120+0000","last_change":"2024-04-25T01:30:33.345575+0000","last_active":"2024-04-25T01:31:00.451120+0000","last_peered":"2024-04-25T01:31:00.451120+0000","last_clean":"2024-04-25T01:31:00.451120+0000","last_became_active":"2024-04-25T01:30:33.345231+0000","last_became_peered":"2024-04-25T01:30:33.345231+0000","last_unstale":"2024-04-25T01:31:00.451120+0000","last_undegraded":"2024-04-25T01:31:00.451120+0000","last_fullsized":"2024-04-25T01:31:00.451120+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:04:21.090788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115206+0000","last_change":"2024-04-25T01:30:34.371317+0000","last_active":"2024-04-25T01:30:39.115206+0000","last_peered":"2024-04-25T01:30:39.115206+0000","last_clean":"2024-04-25T01:30:39.115206+0000","last_became_active":"2024-04-25T01:30:34.370926+0000","last_became_peered":"2024-04-25T01:30:34.370926+0000","last_unstale":"2024-04-25T01:30:39.115206+0000","last_undegraded":"2024-04-25T01:30:39.115206+0000","last_fullsized":"2024-04-25T01:30:39.115206+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:58:39.898471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804491+0000","last_change":"2024-04-25T01:30:38.380982+0000","last_active":"2024-04-25T01:30:38.804491+0000","last_peered":"2024-04-25T01:30:38.804491+0000","last_clean":"2024-04-25T01:30:38.804491+0000","last_became_active":"2024-04-25T01:30:38.380510+0000","last_became_peered":"2024-04-25T01:30:38.380510+0000","last_unstale":"2024-04-25T01:30:38.804491+0000","last_undegraded":"2024-04-25T01:30:38.804491+0000","last_fullsized":"2024-04-25T01:30:38.804491+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:00.510697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114578+0000","last_change":"2024-04-25T01:30:33.351561+0000","last_active":"2024-04-25T01:30:39.114578+0000","last_peered":"2024-04-25T01:30:39.114578+0000","last_clean":"2024-04-25T01:30:39.114578+0000","last_became_active":"2024-04-25T01:30:33.351230+0000","last_became_peered":"2024-04-25T01:30:33.351230+0000","last_unstale":"2024-04-25T01:30:39.114578+0000","last_undegraded":"2024-04-25T01:30:39.114578+0000","last_fullsized":"2024-04-25T01:30:39.114578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:56.594457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115362+0000","last_change":"2024-04-25T01:30:34.358821+0000","last_active":"2024-04-25T01:30:39.115362+0000","last_peered":"2024-04-25T01:30:39.115362+0000","last_clean":"2024-04-25T01:30:39.115362+0000","last_became_active":"2024-04-25T01:30:34.358303+0000","last_became_peered":"2024-04-25T01:30:34.358303+0000","last_unstale":"2024-04-25T01:30:39.115362+0000","last_undegraded":"2024-04-25T01:30:39.115362+0000","last_fullsized":"2024-04-25T01:30:39.115362+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:26:23.352101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.805384+0000","last_change":"2024-04-25T01:30:38.372281+0000","last_active":"2024-04-25T01:30:38.805384+0000","last_peered":"2024-04-25T01:30:38.805384+0000","last_clean":"2024-04-25T01:30:38.805384+0000","last_became_active":"2024-04-25T01:30:38.371876+0000","last_became_peered":"2024-04-25T01:30:38.371876+0000","last_unstale":"2024-04-25T01:30:38.805384+0000","last_undegraded":"2024-04-25T01:30:38.805384+0000","last_fullsized":"2024-04-25T01:30:38.805384+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:54.254011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.442985+0000","last_change":"2024-04-25T01:30:33.347262+0000","last_active":"2024-04-25T01:30:58.442985+0000","last_peered":"2024-04-25T01:30:58.442985+0000","last_clean":"2024-04-25T01:30:58.442985+0000","last_became_active":"2024-04-25T01:30:33.346900+0000","last_became_peered":"2024-04-25T01:30:33.346900+0000","last_unstale":"2024-04-25T01:30:58.442985+0000","last_undegraded":"2024-04-25T01:30:58.442985+0000","last_fullsized":"2024-04-25T01:30:58.442985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:29:30.960456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114160+0000","last_change":"2024-04-25T01:30:34.364870+0000","last_active":"2024-04-25T01:30:39.114160+0000","last_peered":"2024-04-25T01:30:39.114160+0000","last_clean":"2024-04-25T01:30:39.114160+0000","last_became_active":"2024-04-25T01:30:34.364459+0000","last_became_peered":"2024-04-25T01:30:34.364459+0000","last_unstale":"2024-04-25T01:30:39.114160+0000","last_undegraded":"2024-04-25T01:30:39.114160+0000","last_fullsized":"2024-04-25T01:30:39.114160+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:42:45.311668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.805090+0000","last_change":"2024-04-25T01:30:38.382175+0000","last_active":"2024-04-25T01:30:38.805090+0000","last_peered":"2024-04-25T01:30:38.805090+0000","last_clean":"2024-04-25T01:30:38.805090+0000","last_became_active":"2024-04-25T01:30:38.381771+0000","last_became_peered":"2024-04-25T01:30:38.381771+0000","last_unstale":"2024-04-25T01:30:38.805090+0000","last_undegraded":"2024-04-25T01:30:38.805090+0000","last_fullsized":"2024-04-25T01:30:38.805090+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:25:31.041566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.116119+0000","last_change":"2024-04-25T01:30:33.343605+0000","last_active":"2024-04-25T01:30:39.116119+0000","last_peered":"2024-04-25T01:30:39.116119+0000","last_clean":"2024-04-25T01:30:39.116119+0000","last_became_active":"2024-04-25T01:30:33.343254+0000","last_became_peered":"2024-04-25T01:30:33.343254+0000","last_unstale":"2024-04-25T01:30:39.116119+0000","last_undegraded":"2024-04-25T01:30:39.116119+0000","last_fullsized":"2024-04-25T01:30:39.116119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:18:16.156285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115323+0000","last_change":"2024-04-25T01:30:34.354146+0000","last_active":"2024-04-25T01:30:39.115323+0000","last_peered":"2024-04-25T01:30:39.115323+0000","last_clean":"2024-04-25T01:30:39.115323+0000","last_became_active":"2024-04-25T01:30:34.353770+0000","last_became_peered":"2024-04-25T01:30:34.353770+0000","last_unstale":"2024-04-25T01:30:39.115323+0000","last_undegraded":"2024-04-25T01:30:39.115323+0000","last_fullsized":"2024-04-25T01:30:39.115323+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:06:28.725828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.954044+0000","last_change":"2024-04-25T01:30:38.383984+0000","last_active":"2024-04-25T01:30:38.954044+0000","last_peered":"2024-04-25T01:30:38.954044+0000","last_clean":"2024-04-25T01:30:38.954044+0000","last_became_active":"2024-04-25T01:30:38.383534+0000","last_became_peered":"2024-04-25T01:30:38.383534+0000","last_unstale":"2024-04-25T01:30:38.954044+0000","last_undegraded":"2024-04-25T01:30:38.954044+0000","last_fullsized":"2024-04-25T01:30:38.954044+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:07.924839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:59.443307+0000","last_change":"2024-04-25T01:30:33.351621+0000","last_active":"2024-04-25T01:30:59.443307+0000","last_peered":"2024-04-25T01:30:59.443307+0000","last_clean":"2024-04-25T01:30:59.443307+0000","last_became_active":"2024-04-25T01:30:33.351289+0000","last_became_peered":"2024-04-25T01:30:33.351289+0000","last_unstale":"2024-04-25T01:30:59.443307+0000","last_undegraded":"2024-04-25T01:30:59.443307+0000","last_fullsized":"2024-04-25T01:30:59.443307+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:46.044826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114864+0000","last_change":"2024-04-25T01:30:34.350685+0000","last_active":"2024-04-25T01:30:39.114864+0000","last_peered":"2024-04-25T01:30:39.114864+0000","last_clean":"2024-04-25T01:30:39.114864+0000","last_became_active":"2024-04-25T01:30:34.350308+0000","last_became_peered":"2024-04-25T01:30:34.350308+0000","last_unstale":"2024-04-25T01:30:39.114864+0000","last_undegraded":"2024-04-25T01:30:39.114864+0000","last_fullsized":"2024-04-25T01:30:39.114864+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:11.690381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114526+0000","last_change":"2024-04-25T01:30:38.377403+0000","last_active":"2024-04-25T01:30:39.114526+0000","last_peered":"2024-04-25T01:30:39.114526+0000","last_clean":"2024-04-25T01:30:39.114526+0000","last_became_active":"2024-04-25T01:30:38.376966+0000","last_became_peered":"2024-04-25T01:30:38.376966+0000","last_unstale":"2024-04-25T01:30:39.114526+0000","last_undegraded":"2024-04-25T01:30:39.114526+0000","last_fullsized":"2024-04-25T01:30:39.114526+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:31:10.229443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114716+0000","last_change":"2024-04-25T01:30:33.348455+0000","last_active":"2024-04-25T01:30:39.114716+0000","last_peered":"2024-04-25T01:30:39.114716+0000","last_clean":"2024-04-25T01:30:39.114716+0000","last_became_active":"2024-04-25T01:30:33.348142+0000","last_became_peered":"2024-04-25T01:30:33.348142+0000","last_unstale":"2024-04-25T01:30:39.114716+0000","last_undegraded":"2024-04-25T01:30:39.114716+0000","last_fullsized":"2024-04-25T01:30:39.114716+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:43:07.695580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.803468+0000","last_change":"2024-04-25T01:30:34.359949+0000","last_active":"2024-04-25T01:30:38.803468+0000","last_peered":"2024-04-25T01:30:38.803468+0000","last_clean":"2024-04-25T01:30:38.803468+0000","last_became_active":"2024-04-25T01:30:34.359609+0000","last_became_peered":"2024-04-25T01:30:34.359609+0000","last_unstale":"2024-04-25T01:30:38.803468+0000","last_undegraded":"2024-04-25T01:30:38.803468+0000","last_fullsized":"2024-04-25T01:30:38.803468+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:49.668800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114268+0000","last_change":"2024-04-25T01:30:38.380260+0000","last_active":"2024-04-25T01:30:39.114268+0000","last_peered":"2024-04-25T01:30:39.114268+0000","last_clean":"2024-04-25T01:30:39.114268+0000","last_became_active":"2024-04-25T01:30:38.379624+0000","last_became_peered":"2024-04-25T01:30:38.379624+0000","last_unstale":"2024-04-25T01:30:39.114268+0000","last_undegraded":"2024-04-25T01:30:39.114268+0000","last_fullsized":"2024-04-25T01:30:39.114268+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:26:35.235335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.437389+0000","last_change":"2024-04-25T01:30:33.349490+0000","last_active":"2024-04-25T01:30:58.437389+0000","last_peered":"2024-04-25T01:30:58.437389+0000","last_clean":"2024-04-25T01:30:58.437389+0000","last_became_active":"2024-04-25T01:30:33.349136+0000","last_became_peered":"2024-04-25T01:30:33.349136+0000","last_unstale":"2024-04-25T01:30:58.437389+0000","last_undegraded":"2024-04-25T01:30:58.437389+0000","last_fullsized":"2024-04-25T01:30:58.437389+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:16:23.552073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115747+0000","last_change":"2024-04-25T01:30:34.372701+0000","last_active":"2024-04-25T01:30:39.115747+0000","last_peered":"2024-04-25T01:30:39.115747+0000","last_clean":"2024-04-25T01:30:39.115747+0000","last_became_active":"2024-04-25T01:30:34.372321+0000","last_became_peered":"2024-04-25T01:30:34.372321+0000","last_unstale":"2024-04-25T01:30:39.115747+0000","last_undegraded":"2024-04-25T01:30:39.115747+0000","last_fullsized":"2024-04-25T01:30:39.115747+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:02.025015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.803500+0000","last_change":"2024-04-25T01:30:38.378036+0000","last_active":"2024-04-25T01:30:38.803500+0000","last_peered":"2024-04-25T01:30:38.803500+0000","last_clean":"2024-04-25T01:30:38.803500+0000","last_became_active":"2024-04-25T01:30:38.377602+0000","last_became_peered":"2024-04-25T01:30:38.377602+0000","last_unstale":"2024-04-25T01:30:38.803500+0000","last_undegraded":"2024-04-25T01:30:38.803500+0000","last_fullsized":"2024-04-25T01:30:38.803500+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:45.210518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.435196+0000","last_change":"2024-04-25T01:30:33.352928+0000","last_active":"2024-04-25T01:30:58.435196+0000","last_peered":"2024-04-25T01:30:58.435196+0000","last_clean":"2024-04-25T01:30:58.435196+0000","last_became_active":"2024-04-25T01:30:33.352567+0000","last_became_peered":"2024-04-25T01:30:33.352567+0000","last_unstale":"2024-04-25T01:30:58.435196+0000","last_undegraded":"2024-04-25T01:30:58.435196+0000","last_fullsized":"2024-04-25T01:30:58.435196+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:47:21.941179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114253+0000","last_change":"2024-04-25T01:30:34.366947+0000","last_active":"2024-04-25T01:30:39.114253+0000","last_peered":"2024-04-25T01:30:39.114253+0000","last_clean":"2024-04-25T01:30:39.114253+0000","last_became_active":"2024-04-25T01:30:34.366337+0000","last_became_peered":"2024-04-25T01:30:34.366337+0000","last_unstale":"2024-04-25T01:30:39.114253+0000","last_undegraded":"2024-04-25T01:30:39.114253+0000","last_fullsized":"2024-04-25T01:30:39.114253+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:48:43.528380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804886+0000","last_change":"2024-04-25T01:30:38.372369+0000","last_active":"2024-04-25T01:30:38.804886+0000","last_peered":"2024-04-25T01:30:38.804886+0000","last_clean":"2024-04-25T01:30:38.804886+0000","last_became_active":"2024-04-25T01:30:38.371920+0000","last_became_peered":"2024-04-25T01:30:38.371920+0000","last_unstale":"2024-04-25T01:30:38.804886+0000","last_undegraded":"2024-04-25T01:30:38.804886+0000","last_fullsized":"2024-04-25T01:30:38.804886+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:57.299089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115324+0000","last_change":"2024-04-25T01:30:33.346003+0000","last_active":"2024-04-25T01:30:39.115324+0000","last_peered":"2024-04-25T01:30:39.115324+0000","last_clean":"2024-04-25T01:30:39.115324+0000","last_became_active":"2024-04-25T01:30:33.345254+0000","last_became_peered":"2024-04-25T01:30:33.345254+0000","last_unstale":"2024-04-25T01:30:39.115324+0000","last_undegraded":"2024-04-25T01:30:39.115324+0000","last_fullsized":"2024-04-25T01:30:39.115324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:18:37.957047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115423+0000","last_change":"2024-04-25T01:30:34.366108+0000","last_active":"2024-04-25T01:30:39.115423+0000","last_peered":"2024-04-25T01:30:39.115423+0000","last_clean":"2024-04-25T01:30:39.115423+0000","last_became_active":"2024-04-25T01:30:34.365702+0000","last_became_peered":"2024-04-25T01:30:34.365702+0000","last_unstale":"2024-04-25T01:30:39.115423+0000","last_undegraded":"2024-04-25T01:30:39.115423+0000","last_fullsized":"2024-04-25T01:30:39.115423+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:24:06.802424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.803621+0000","last_change":"2024-04-25T01:30:38.376342+0000","last_active":"2024-04-25T01:30:38.803621+0000","last_peered":"2024-04-25T01:30:38.803621+0000","last_clean":"2024-04-25T01:30:38.803621+0000","last_became_active":"2024-04-25T01:30:38.375905+0000","last_became_peered":"2024-04-25T01:30:38.375905+0000","last_unstale":"2024-04-25T01:30:38.803621+0000","last_undegraded":"2024-04-25T01:30:38.803621+0000","last_fullsized":"2024-04-25T01:30:38.803621+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:31.615356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'12","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.437393+0000","last_change":"2024-04-25T01:30:33.350770+0000","last_active":"2024-04-25T01:30:58.437393+0000","last_peered":"2024-04-25T01:30:58.437393+0000","last_clean":"2024-04-25T01:30:58.437393+0000","last_became_active":"2024-04-25T01:30:33.350414+0000","last_became_peered":"2024-04-25T01:30:33.350414+0000","last_unstale":"2024-04-25T01:30:58.437393+0000","last_undegraded":"2024-04-25T01:30:58.437393+0000","last_fullsized":"2024-04-25T01:30:58.437393+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:01.771182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114585+0000","last_change":"2024-04-25T01:30:34.362532+0000","last_active":"2024-04-25T01:30:39.114585+0000","last_peered":"2024-04-25T01:30:39.114585+0000","last_clean":"2024-04-25T01:30:39.114585+0000","last_became_active":"2024-04-25T01:30:34.362137+0000","last_became_peered":"2024-04-25T01:30:34.362137+0000","last_unstale":"2024-04-25T01:30:39.114585+0000","last_undegraded":"2024-04-25T01:30:39.114585+0000","last_fullsized":"2024-04-25T01:30:39.114585+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:28.105250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115694+0000","last_change":"2024-04-25T01:30:38.374591+0000","last_active":"2024-04-25T01:30:39.115694+0000","last_peered":"2024-04-25T01:30:39.115694+0000","last_clean":"2024-04-25T01:30:39.115694+0000","last_became_active":"2024-04-25T01:30:38.373237+0000","last_became_peered":"2024-04-25T01:30:38.373237+0000","last_unstale":"2024-04-25T01:30:39.115694+0000","last_undegraded":"2024-04-25T01:30:39.115694+0000","last_fullsized":"2024-04-25T01:30:39.115694+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:21:25.937405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.805156+0000","last_change":"2024-04-25T01:30:33.344341+0000","last_active":"2024-04-25T01:30:38.805156+0000","last_peered":"2024-04-25T01:30:38.805156+0000","last_clean":"2024-04-25T01:30:38.805156+0000","last_became_active":"2024-04-25T01:30:33.343046+0000","last_became_peered":"2024-04-25T01:30:33.343046+0000","last_unstale":"2024-04-25T01:30:38.805156+0000","last_undegraded":"2024-04-25T01:30:38.805156+0000","last_fullsized":"2024-04-25T01:30:38.805156+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:52.291970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.454783+0000","last_change":"2024-04-25T01:30:34.354444+0000","last_active":"2024-04-25T01:30:39.454783+0000","last_peered":"2024-04-25T01:30:39.454783+0000","last_clean":"2024-04-25T01:30:39.454783+0000","last_became_active":"2024-04-25T01:30:34.354070+0000","last_became_peered":"2024-04-25T01:30:34.354070+0000","last_unstale":"2024-04-25T01:30:39.454783+0000","last_undegraded":"2024-04-25T01:30:39.454783+0000","last_fullsized":"2024-04-25T01:30:39.454783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:39.893995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.953846+0000","last_change":"2024-04-25T01:30:38.373570+0000","last_active":"2024-04-25T01:30:38.953846+0000","last_peered":"2024-04-25T01:30:38.953846+0000","last_clean":"2024-04-25T01:30:38.953846+0000","last_became_active":"2024-04-25T01:30:38.373144+0000","last_became_peered":"2024-04-25T01:30:38.373144+0000","last_unstale":"2024-04-25T01:30:38.953846+0000","last_undegraded":"2024-04-25T01:30:38.953846+0000","last_fullsized":"2024-04-25T01:30:38.953846+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:53:32.933998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.439139+0000","last_change":"2024-04-25T01:30:33.346145+0000","last_active":"2024-04-25T01:30:58.439139+0000","last_peered":"2024-04-25T01:30:58.439139+0000","last_clean":"2024-04-25T01:30:58.439139+0000","last_became_active":"2024-04-25T01:30:33.344294+0000","last_became_peered":"2024-04-25T01:30:33.344294+0000","last_unstale":"2024-04-25T01:30:58.439139+0000","last_undegraded":"2024-04-25T01:30:58.439139+0000","last_fullsized":"2024-04-25T01:30:58.439139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:37:17.147148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114821+0000","last_change":"2024-04-25T01:30:34.360769+0000","last_active":"2024-04-25T01:30:39.114821+0000","last_peered":"2024-04-25T01:30:39.114821+0000","last_clean":"2024-04-25T01:30:39.114821+0000","last_became_active":"2024-04-25T01:30:34.360076+0000","last_became_peered":"2024-04-25T01:30:34.360076+0000","last_unstale":"2024-04-25T01:30:39.114821+0000","last_undegraded":"2024-04-25T01:30:39.114821+0000","last_fullsized":"2024-04-25T01:30:39.114821+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:56:21.515511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.803481+0000","last_change":"2024-04-25T01:30:38.379198+0000","last_active":"2024-04-25T01:30:38.803481+0000","last_peered":"2024-04-25T01:30:38.803481+0000","last_clean":"2024-04-25T01:30:38.803481+0000","last_became_active":"2024-04-25T01:30:38.378798+0000","last_became_peered":"2024-04-25T01:30:38.378798+0000","last_unstale":"2024-04-25T01:30:38.803481+0000","last_undegraded":"2024-04-25T01:30:38.803481+0000","last_fullsized":"2024-04-25T01:30:38.803481+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:40.192239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804472+0000","last_change":"2024-04-25T01:30:33.343232+0000","last_active":"2024-04-25T01:30:38.804472+0000","last_peered":"2024-04-25T01:30:38.804472+0000","last_clean":"2024-04-25T01:30:38.804472+0000","last_became_active":"2024-04-25T01:30:33.342876+0000","last_became_peered":"2024-04-25T01:30:33.342876+0000","last_unstale":"2024-04-25T01:30:38.804472+0000","last_undegraded":"2024-04-25T01:30:38.804472+0000","last_fullsized":"2024-04-25T01:30:38.804472+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:45.032564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114383+0000","last_change":"2024-04-25T01:30:34.358210+0000","last_active":"2024-04-25T01:30:39.114383+0000","last_peered":"2024-04-25T01:30:39.114383+0000","last_clean":"2024-04-25T01:30:39.114383+0000","last_became_active":"2024-04-25T01:30:34.357862+0000","last_became_peered":"2024-04-25T01:30:34.357862+0000","last_unstale":"2024-04-25T01:30:39.114383+0000","last_undegraded":"2024-04-25T01:30:39.114383+0000","last_fullsized":"2024-04-25T01:30:39.114383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:43.751361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.455266+0000","last_change":"2024-04-25T01:30:38.380413+0000","last_active":"2024-04-25T01:30:39.455266+0000","last_peered":"2024-04-25T01:30:39.455266+0000","last_clean":"2024-04-25T01:30:39.455266+0000","last_became_active":"2024-04-25T01:30:38.379612+0000","last_became_peered":"2024-04-25T01:30:38.379612+0000","last_unstale":"2024-04-25T01:30:39.455266+0000","last_undegraded":"2024-04-25T01:30:39.455266+0000","last_fullsized":"2024-04-25T01:30:39.455266+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:45.328642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'16","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:59.444283+0000","last_change":"2024-04-25T01:30:33.344938+0000","last_active":"2024-04-25T01:30:59.444283+0000","last_peered":"2024-04-25T01:30:59.444283+0000","last_clean":"2024-04-25T01:30:59.444283+0000","last_became_active":"2024-04-25T01:30:33.344582+0000","last_became_peered":"2024-04-25T01:30:33.344582+0000","last_unstale":"2024-04-25T01:30:59.444283+0000","last_undegraded":"2024-04-25T01:30:59.444283+0000","last_fullsized":"2024-04-25T01:30:59.444283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:43:33.777319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114724+0000","last_change":"2024-04-25T01:30:34.354115+0000","last_active":"2024-04-25T01:30:39.114724+0000","last_peered":"2024-04-25T01:30:39.114724+0000","last_clean":"2024-04-25T01:30:39.114724+0000","last_became_active":"2024-04-25T01:30:34.353722+0000","last_became_peered":"2024-04-25T01:30:34.353722+0000","last_unstale":"2024-04-25T01:30:39.114724+0000","last_undegraded":"2024-04-25T01:30:39.114724+0000","last_fullsized":"2024-04-25T01:30:39.114724+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:22.003521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114604+0000","last_change":"2024-04-25T01:30:38.380914+0000","last_active":"2024-04-25T01:30:39.114604+0000","last_peered":"2024-04-25T01:30:39.114604+0000","last_clean":"2024-04-25T01:30:39.114604+0000","last_became_active":"2024-04-25T01:30:38.380477+0000","last_became_peered":"2024-04-25T01:30:38.380477+0000","last_unstale":"2024-04-25T01:30:39.114604+0000","last_undegraded":"2024-04-25T01:30:39.114604+0000","last_fullsized":"2024-04-25T01:30:39.114604+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:23:27.882266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114449+0000","last_change":"2024-04-25T01:30:33.345772+0000","last_active":"2024-04-25T01:30:39.114449+0000","last_peered":"2024-04-25T01:30:39.114449+0000","last_clean":"2024-04-25T01:30:39.114449+0000","last_became_active":"2024-04-25T01:30:33.345455+0000","last_became_peered":"2024-04-25T01:30:33.345455+0000","last_unstale":"2024-04-25T01:30:39.114449+0000","last_undegraded":"2024-04-25T01:30:39.114449+0000","last_fullsized":"2024-04-25T01:30:39.114449+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:14:57.465161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.953974+0000","last_change":"2024-04-25T01:30:34.358665+0000","last_active":"2024-04-25T01:30:38.953974+0000","last_peered":"2024-04-25T01:30:38.953974+0000","last_clean":"2024-04-25T01:30:38.953974+0000","last_became_active":"2024-04-25T01:30:34.358284+0000","last_became_peered":"2024-04-25T01:30:34.358284+0000","last_unstale":"2024-04-25T01:30:38.953974+0000","last_undegraded":"2024-04-25T01:30:38.953974+0000","last_fullsized":"2024-04-25T01:30:38.953974+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:12:22.832265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114254+0000","last_change":"2024-04-25T01:30:38.450027+0000","last_active":"2024-04-25T01:30:39.114254+0000","last_peered":"2024-04-25T01:30:39.114254+0000","last_clean":"2024-04-25T01:30:39.114254+0000","last_became_active":"2024-04-25T01:30:38.449564+0000","last_became_peered":"2024-04-25T01:30:38.449564+0000","last_unstale":"2024-04-25T01:30:39.114254+0000","last_undegraded":"2024-04-25T01:30:39.114254+0000","last_fullsized":"2024-04-25T01:30:39.114254+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:34:26.542351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.803395+0000","last_change":"2024-04-25T01:30:33.349757+0000","last_active":"2024-04-25T01:30:38.803395+0000","last_peered":"2024-04-25T01:30:38.803395+0000","last_clean":"2024-04-25T01:30:38.803395+0000","last_became_active":"2024-04-25T01:30:33.349439+0000","last_became_peered":"2024-04-25T01:30:33.349439+0000","last_unstale":"2024-04-25T01:30:38.803395+0000","last_undegraded":"2024-04-25T01:30:38.803395+0000","last_fullsized":"2024-04-25T01:30:38.803395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:35.514160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.116544+0000","last_change":"2024-04-25T01:30:34.367310+0000","last_active":"2024-04-25T01:30:39.116544+0000","last_peered":"2024-04-25T01:30:39.116544+0000","last_clean":"2024-04-25T01:30:39.116544+0000","last_became_active":"2024-04-25T01:30:34.366950+0000","last_became_peered":"2024-04-25T01:30:34.366950+0000","last_unstale":"2024-04-25T01:30:39.116544+0000","last_undegraded":"2024-04-25T01:30:39.116544+0000","last_fullsized":"2024-04-25T01:30:39.116544+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:26:17.700512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.455026+0000","last_change":"2024-04-25T01:30:38.372259+0000","last_active":"2024-04-25T01:30:39.455026+0000","last_peered":"2024-04-25T01:30:39.455026+0000","last_clean":"2024-04-25T01:30:39.455026+0000","last_became_active":"2024-04-25T01:30:38.371847+0000","last_became_peered":"2024-04-25T01:30:38.371847+0000","last_unstale":"2024-04-25T01:30:39.455026+0000","last_undegraded":"2024-04-25T01:30:39.455026+0000","last_fullsized":"2024-04-25T01:30:39.455026+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:55.047739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114803+0000","last_change":"2024-04-25T01:30:33.352078+0000","last_active":"2024-04-25T01:30:39.114803+0000","last_peered":"2024-04-25T01:30:39.114803+0000","last_clean":"2024-04-25T01:30:39.114803+0000","last_became_active":"2024-04-25T01:30:33.351725+0000","last_became_peered":"2024-04-25T01:30:33.351725+0000","last_unstale":"2024-04-25T01:30:39.114803+0000","last_undegraded":"2024-04-25T01:30:39.114803+0000","last_fullsized":"2024-04-25T01:30:39.114803+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:55:55.236040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.116297+0000","last_change":"2024-04-25T01:30:34.360324+0000","last_active":"2024-04-25T01:30:39.116297+0000","last_peered":"2024-04-25T01:30:39.116297+0000","last_clean":"2024-04-25T01:30:39.116297+0000","last_became_active":"2024-04-25T01:30:34.359114+0000","last_became_peered":"2024-04-25T01:30:34.359114+0000","last_unstale":"2024-04-25T01:30:39.116297+0000","last_undegraded":"2024-04-25T01:30:39.116297+0000","last_fullsized":"2024-04-25T01:30:39.116297+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:51.927640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.455117+0000","last_change":"2024-04-25T01:30:38.450650+0000","last_active":"2024-04-25T01:30:39.455117+0000","last_peered":"2024-04-25T01:30:39.455117+0000","last_clean":"2024-04-25T01:30:39.455117+0000","last_became_active":"2024-04-25T01:30:38.450149+0000","last_became_peered":"2024-04-25T01:30:38.450149+0000","last_unstale":"2024-04-25T01:30:39.455117+0000","last_undegraded":"2024-04-25T01:30:39.455117+0000","last_fullsized":"2024-04-25T01:30:39.455117+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:34.630803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114887+0000","last_change":"2024-04-25T01:30:34.352961+0000","last_active":"2024-04-25T01:30:39.114887+0000","last_peered":"2024-04-25T01:30:39.114887+0000","last_clean":"2024-04-25T01:30:39.114887+0000","last_became_active":"2024-04-25T01:30:34.352657+0000","last_became_peered":"2024-04-25T01:30:34.352657+0000","last_unstale":"2024-04-25T01:30:39.114887+0000","last_undegraded":"2024-04-25T01:30:39.114887+0000","last_fullsized":"2024-04-25T01:30:39.114887+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:20.934228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:59.444315+0000","last_change":"2024-04-25T01:30:33.344562+0000","last_active":"2024-04-25T01:30:59.444315+0000","last_peered":"2024-04-25T01:30:59.444315+0000","last_clean":"2024-04-25T01:30:59.444315+0000","last_became_active":"2024-04-25T01:30:33.344212+0000","last_became_peered":"2024-04-25T01:30:33.344212+0000","last_unstale":"2024-04-25T01:30:59.444315+0000","last_undegraded":"2024-04-25T01:30:59.444315+0000","last_fullsized":"2024-04-25T01:30:59.444315+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:20:51.110618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115132+0000","last_change":"2024-04-25T01:30:38.384316+0000","last_active":"2024-04-25T01:30:39.115132+0000","last_peered":"2024-04-25T01:30:39.115132+0000","last_clean":"2024-04-25T01:30:39.115132+0000","last_became_active":"2024-04-25T01:30:38.383775+0000","last_became_peered":"2024-04-25T01:30:38.383775+0000","last_unstale":"2024-04-25T01:30:39.115132+0000","last_undegraded":"2024-04-25T01:30:39.115132+0000","last_fullsized":"2024-04-25T01:30:39.115132+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:54.165691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114224+0000","last_change":"2024-04-25T01:30:34.355077+0000","last_active":"2024-04-25T01:30:39.114224+0000","last_peered":"2024-04-25T01:30:39.114224+0000","last_clean":"2024-04-25T01:30:39.114224+0000","last_became_active":"2024-04-25T01:30:34.352742+0000","last_became_peered":"2024-04-25T01:30:34.352742+0000","last_unstale":"2024-04-25T01:30:39.114224+0000","last_undegraded":"2024-04-25T01:30:39.114224+0000","last_fullsized":"2024-04-25T01:30:39.114224+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:48:31.495699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.438784+0000","last_change":"2024-04-25T01:30:33.345283+0000","last_active":"2024-04-25T01:30:58.438784+0000","last_peered":"2024-04-25T01:30:58.438784+0000","last_clean":"2024-04-25T01:30:58.438784+0000","last_became_active":"2024-04-25T01:30:33.344903+0000","last_became_peered":"2024-04-25T01:30:33.344903+0000","last_unstale":"2024-04-25T01:30:58.438784+0000","last_undegraded":"2024-04-25T01:30:58.438784+0000","last_fullsized":"2024-04-25T01:30:58.438784+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:57:01.982850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":359,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114627+0000","last_change":"2024-04-25T01:29:17.689639+0000","last_active":"2024-04-25T01:30:39.114627+0000","last_peered":"2024-04-25T01:30:39.114627+0000","last_clean":"2024-04-25T01:30:39.114627+0000","last_became_active":"2024-04-25T01:29:17.689046+0000","last_became_peered":"2024-04-25T01:29:17.689046+0000","last_unstale":"2024-04-25T01:30:39.114627+0000","last_undegraded":"2024-04-25T01:30:39.114627+0000","last_fullsized":"2024-04-25T01:30:39.114627+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:48.767945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:48.767945+0000","last_clean_scrub_stamp":"2024-04-25T01:25:48.767945+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:04:06.092632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115580+0000","last_change":"2024-04-25T01:30:38.450938+0000","last_active":"2024-04-25T01:30:39.115580+0000","last_peered":"2024-04-25T01:30:39.115580+0000","last_clean":"2024-04-25T01:30:39.115580+0000","last_became_active":"2024-04-25T01:30:38.450488+0000","last_became_peered":"2024-04-25T01:30:38.450488+0000","last_unstale":"2024-04-25T01:30:39.115580+0000","last_undegraded":"2024-04-25T01:30:39.115580+0000","last_fullsized":"2024-04-25T01:30:39.115580+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:44.778133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.454492+0000","last_change":"2024-04-25T01:30:34.352841+0000","last_active":"2024-04-25T01:30:39.454492+0000","last_peered":"2024-04-25T01:30:39.454492+0000","last_clean":"2024-04-25T01:30:39.454492+0000","last_became_active":"2024-04-25T01:30:34.352460+0000","last_became_peered":"2024-04-25T01:30:34.352460+0000","last_unstale":"2024-04-25T01:30:39.454492+0000","last_undegraded":"2024-04-25T01:30:39.454492+0000","last_fullsized":"2024-04-25T01:30:39.454492+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:18:46.579947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:59.442330+0000","last_change":"2024-04-25T01:30:33.348198+0000","last_active":"2024-04-25T01:30:59.442330+0000","last_peered":"2024-04-25T01:30:59.442330+0000","last_clean":"2024-04-25T01:30:59.442330+0000","last_became_active":"2024-04-25T01:30:33.347876+0000","last_became_peered":"2024-04-25T01:30:33.347876+0000","last_unstale":"2024-04-25T01:30:59.442330+0000","last_undegraded":"2024-04-25T01:30:59.442330+0000","last_fullsized":"2024-04-25T01:30:59.442330+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:48:33.116571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.953972+0000","last_change":"2024-04-25T01:30:38.372331+0000","last_active":"2024-04-25T01:30:38.953972+0000","last_peered":"2024-04-25T01:30:38.953972+0000","last_clean":"2024-04-25T01:30:38.953972+0000","last_became_active":"2024-04-25T01:30:38.371923+0000","last_became_peered":"2024-04-25T01:30:38.371923+0000","last_unstale":"2024-04-25T01:30:38.953972+0000","last_undegraded":"2024-04-25T01:30:38.953972+0000","last_fullsized":"2024-04-25T01:30:38.953972+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:06.416967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:31:00.451341+0000","last_change":"2024-04-25T01:30:33.346606+0000","last_active":"2024-04-25T01:31:00.451341+0000","last_peered":"2024-04-25T01:31:00.451341+0000","last_clean":"2024-04-25T01:31:00.451341+0000","last_became_active":"2024-04-25T01:30:33.346297+0000","last_became_peered":"2024-04-25T01:30:33.346297+0000","last_unstale":"2024-04-25T01:31:00.451341+0000","last_undegraded":"2024-04-25T01:31:00.451341+0000","last_fullsized":"2024-04-25T01:31:00.451341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:07:16.069285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.455021+0000","last_change":"2024-04-25T01:30:34.349136+0000","last_active":"2024-04-25T01:30:39.455021+0000","last_peered":"2024-04-25T01:30:39.455021+0000","last_clean":"2024-04-25T01:30:39.455021+0000","last_became_active":"2024-04-25T01:30:34.348810+0000","last_became_peered":"2024-04-25T01:30:34.348810+0000","last_unstale":"2024-04-25T01:30:39.455021+0000","last_undegraded":"2024-04-25T01:30:39.455021+0000","last_fullsized":"2024-04-25T01:30:39.455021+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:39:34.593986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114836+0000","last_change":"2024-04-25T01:30:38.450428+0000","last_active":"2024-04-25T01:30:39.114836+0000","last_peered":"2024-04-25T01:30:39.114836+0000","last_clean":"2024-04-25T01:30:39.114836+0000","last_became_active":"2024-04-25T01:30:38.449971+0000","last_became_peered":"2024-04-25T01:30:38.449971+0000","last_unstale":"2024-04-25T01:30:39.114836+0000","last_undegraded":"2024-04-25T01:30:39.114836+0000","last_fullsized":"2024-04-25T01:30:39.114836+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:52:37.046462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.803587+0000","last_change":"2024-04-25T01:30:33.348547+0000","last_active":"2024-04-25T01:30:38.803587+0000","last_peered":"2024-04-25T01:30:38.803587+0000","last_clean":"2024-04-25T01:30:38.803587+0000","last_became_active":"2024-04-25T01:30:33.348181+0000","last_became_peered":"2024-04-25T01:30:33.348181+0000","last_unstale":"2024-04-25T01:30:38.803587+0000","last_undegraded":"2024-04-25T01:30:38.803587+0000","last_fullsized":"2024-04-25T01:30:38.803587+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:11:58.696829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115147+0000","last_change":"2024-04-25T01:30:34.351987+0000","last_active":"2024-04-25T01:30:39.115147+0000","last_peered":"2024-04-25T01:30:39.115147+0000","last_clean":"2024-04-25T01:30:39.115147+0000","last_became_active":"2024-04-25T01:30:34.351571+0000","last_became_peered":"2024-04-25T01:30:34.351571+0000","last_unstale":"2024-04-25T01:30:39.115147+0000","last_undegraded":"2024-04-25T01:30:39.115147+0000","last_fullsized":"2024-04-25T01:30:39.115147+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:23.585865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804768+0000","last_change":"2024-04-25T01:30:38.372566+0000","last_active":"2024-04-25T01:30:38.804768+0000","last_peered":"2024-04-25T01:30:38.804768+0000","last_clean":"2024-04-25T01:30:38.804768+0000","last_became_active":"2024-04-25T01:30:38.372102+0000","last_became_peered":"2024-04-25T01:30:38.372102+0000","last_unstale":"2024-04-25T01:30:38.804768+0000","last_undegraded":"2024-04-25T01:30:38.804768+0000","last_fullsized":"2024-04-25T01:30:38.804768+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:22:16.791842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.440012+0000","last_change":"2024-04-25T01:30:33.344065+0000","last_active":"2024-04-25T01:30:58.440012+0000","last_peered":"2024-04-25T01:30:58.440012+0000","last_clean":"2024-04-25T01:30:58.440012+0000","last_became_active":"2024-04-25T01:30:33.343700+0000","last_became_peered":"2024-04-25T01:30:33.343700+0000","last_unstale":"2024-04-25T01:30:58.440012+0000","last_undegraded":"2024-04-25T01:30:58.440012+0000","last_fullsized":"2024-04-25T01:30:58.440012+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:54:08.265744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114656+0000","last_change":"2024-04-25T01:30:34.354644+0000","last_active":"2024-04-25T01:30:39.114656+0000","last_peered":"2024-04-25T01:30:39.114656+0000","last_clean":"2024-04-25T01:30:39.114656+0000","last_became_active":"2024-04-25T01:30:34.354330+0000","last_became_peered":"2024-04-25T01:30:34.354330+0000","last_unstale":"2024-04-25T01:30:39.114656+0000","last_undegraded":"2024-04-25T01:30:39.114656+0000","last_fullsized":"2024-04-25T01:30:39.114656+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:31:50.399983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.805165+0000","last_change":"2024-04-25T01:30:38.450743+0000","last_active":"2024-04-25T01:30:38.805165+0000","last_peered":"2024-04-25T01:30:38.805165+0000","last_clean":"2024-04-25T01:30:38.805165+0000","last_became_active":"2024-04-25T01:30:38.450303+0000","last_became_peered":"2024-04-25T01:30:38.450303+0000","last_unstale":"2024-04-25T01:30:38.805165+0000","last_undegraded":"2024-04-25T01:30:38.805165+0000","last_fullsized":"2024-04-25T01:30:38.805165+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:37.919250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.438441+0000","last_change":"2024-04-25T01:30:33.350709+0000","last_active":"2024-04-25T01:30:58.438441+0000","last_peered":"2024-04-25T01:30:58.438441+0000","last_clean":"2024-04-25T01:30:58.438441+0000","last_became_active":"2024-04-25T01:30:33.350387+0000","last_became_peered":"2024-04-25T01:30:33.350387+0000","last_unstale":"2024-04-25T01:30:58.438441+0000","last_undegraded":"2024-04-25T01:30:58.438441+0000","last_fullsized":"2024-04-25T01:30:58.438441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:30:01.252121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114002+0000","last_change":"2024-04-25T01:30:34.362040+0000","last_active":"2024-04-25T01:30:39.114002+0000","last_peered":"2024-04-25T01:30:39.114002+0000","last_clean":"2024-04-25T01:30:39.114002+0000","last_became_active":"2024-04-25T01:30:34.361734+0000","last_became_peered":"2024-04-25T01:30:34.361734+0000","last_unstale":"2024-04-25T01:30:39.114002+0000","last_undegraded":"2024-04-25T01:30:39.114002+0000","last_fullsized":"2024-04-25T01:30:39.114002+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:11:28.409764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.803675+0000","last_change":"2024-04-25T01:30:38.383202+0000","last_active":"2024-04-25T01:30:38.803675+0000","last_peered":"2024-04-25T01:30:38.803675+0000","last_clean":"2024-04-25T01:30:38.803675+0000","last_became_active":"2024-04-25T01:30:38.382071+0000","last_became_peered":"2024-04-25T01:30:38.382071+0000","last_unstale":"2024-04-25T01:30:38.803675+0000","last_undegraded":"2024-04-25T01:30:38.803675+0000","last_fullsized":"2024-04-25T01:30:38.803675+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:06:49.283136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.438358+0000","last_change":"2024-04-25T01:30:33.350207+0000","last_active":"2024-04-25T01:30:58.438358+0000","last_peered":"2024-04-25T01:30:58.438358+0000","last_clean":"2024-04-25T01:30:58.438358+0000","last_became_active":"2024-04-25T01:30:33.349892+0000","last_became_peered":"2024-04-25T01:30:33.349892+0000","last_unstale":"2024-04-25T01:30:58.438358+0000","last_undegraded":"2024-04-25T01:30:58.438358+0000","last_fullsized":"2024-04-25T01:30:58.438358+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:26:35.186486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.953587+0000","last_change":"2024-04-25T01:30:34.355196+0000","last_active":"2024-04-25T01:30:38.953587+0000","last_peered":"2024-04-25T01:30:38.953587+0000","last_clean":"2024-04-25T01:30:38.953587+0000","last_became_active":"2024-04-25T01:30:34.354794+0000","last_became_peered":"2024-04-25T01:30:34.354794+0000","last_unstale":"2024-04-25T01:30:38.953587+0000","last_undegraded":"2024-04-25T01:30:38.953587+0000","last_fullsized":"2024-04-25T01:30:38.953587+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:43:37.495974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804757+0000","last_change":"2024-04-25T01:30:38.384159+0000","last_active":"2024-04-25T01:30:38.804757+0000","last_peered":"2024-04-25T01:30:38.804757+0000","last_clean":"2024-04-25T01:30:38.804757+0000","last_became_active":"2024-04-25T01:30:38.383676+0000","last_became_peered":"2024-04-25T01:30:38.383676+0000","last_unstale":"2024-04-25T01:30:38.804757+0000","last_undegraded":"2024-04-25T01:30:38.804757+0000","last_fullsized":"2024-04-25T01:30:38.804757+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:28.491522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804837+0000","last_change":"2024-04-25T01:30:33.341379+0000","last_active":"2024-04-25T01:30:38.804837+0000","last_peered":"2024-04-25T01:30:38.804837+0000","last_clean":"2024-04-25T01:30:38.804837+0000","last_became_active":"2024-04-25T01:30:33.340999+0000","last_became_peered":"2024-04-25T01:30:33.340999+0000","last_unstale":"2024-04-25T01:30:38.804837+0000","last_undegraded":"2024-04-25T01:30:38.804837+0000","last_fullsized":"2024-04-25T01:30:38.804837+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:57:02.059834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114849+0000","last_change":"2024-04-25T01:30:34.358110+0000","last_active":"2024-04-25T01:30:39.114849+0000","last_peered":"2024-04-25T01:30:39.114849+0000","last_clean":"2024-04-25T01:30:39.114849+0000","last_became_active":"2024-04-25T01:30:34.357555+0000","last_became_peered":"2024-04-25T01:30:34.357555+0000","last_unstale":"2024-04-25T01:30:39.114849+0000","last_undegraded":"2024-04-25T01:30:39.114849+0000","last_fullsized":"2024-04-25T01:30:39.114849+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:11.636320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.805212+0000","last_change":"2024-04-25T01:30:38.450680+0000","last_active":"2024-04-25T01:30:38.805212+0000","last_peered":"2024-04-25T01:30:38.805212+0000","last_clean":"2024-04-25T01:30:38.805212+0000","last_became_active":"2024-04-25T01:30:38.450168+0000","last_became_peered":"2024-04-25T01:30:38.450168+0000","last_unstale":"2024-04-25T01:30:38.805212+0000","last_undegraded":"2024-04-25T01:30:38.805212+0000","last_fullsized":"2024-04-25T01:30:38.805212+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:39:24.236371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.437842+0000","last_change":"2024-04-25T01:30:33.343848+0000","last_active":"2024-04-25T01:30:58.437842+0000","last_peered":"2024-04-25T01:30:58.437842+0000","last_clean":"2024-04-25T01:30:58.437842+0000","last_became_active":"2024-04-25T01:30:33.343498+0000","last_became_peered":"2024-04-25T01:30:33.343498+0000","last_unstale":"2024-04-25T01:30:58.437842+0000","last_undegraded":"2024-04-25T01:30:58.437842+0000","last_fullsized":"2024-04-25T01:30:58.437842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:58:09.184571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114944+0000","last_change":"2024-04-25T01:30:34.361994+0000","last_active":"2024-04-25T01:30:39.114944+0000","last_peered":"2024-04-25T01:30:39.114944+0000","last_clean":"2024-04-25T01:30:39.114944+0000","last_became_active":"2024-04-25T01:30:34.361532+0000","last_became_peered":"2024-04-25T01:30:34.361532+0000","last_unstale":"2024-04-25T01:30:39.114944+0000","last_undegraded":"2024-04-25T01:30:39.114944+0000","last_fullsized":"2024-04-25T01:30:39.114944+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:30.667672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114812+0000","last_change":"2024-04-25T01:30:38.369041+0000","last_active":"2024-04-25T01:30:39.114812+0000","last_peered":"2024-04-25T01:30:39.114812+0000","last_clean":"2024-04-25T01:30:39.114812+0000","last_became_active":"2024-04-25T01:30:38.368586+0000","last_became_peered":"2024-04-25T01:30:38.368586+0000","last_unstale":"2024-04-25T01:30:39.114812+0000","last_undegraded":"2024-04-25T01:30:39.114812+0000","last_fullsized":"2024-04-25T01:30:39.114812+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:58:53.743012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.805258+0000","last_change":"2024-04-25T01:30:33.345501+0000","last_active":"2024-04-25T01:30:38.805258+0000","last_peered":"2024-04-25T01:30:38.805258+0000","last_clean":"2024-04-25T01:30:38.805258+0000","last_became_active":"2024-04-25T01:30:33.345179+0000","last_became_peered":"2024-04-25T01:30:33.345179+0000","last_unstale":"2024-04-25T01:30:38.805258+0000","last_undegraded":"2024-04-25T01:30:38.805258+0000","last_fullsized":"2024-04-25T01:30:38.805258+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:29:41.275480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115455+0000","last_change":"2024-04-25T01:30:34.361631+0000","last_active":"2024-04-25T01:30:39.115455+0000","last_peered":"2024-04-25T01:30:39.115455+0000","last_clean":"2024-04-25T01:30:39.115455+0000","last_became_active":"2024-04-25T01:30:34.360266+0000","last_became_peered":"2024-04-25T01:30:34.360266+0000","last_unstale":"2024-04-25T01:30:39.115455+0000","last_undegraded":"2024-04-25T01:30:39.115455+0000","last_fullsized":"2024-04-25T01:30:39.115455+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:03.094166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.113965+0000","last_change":"2024-04-25T01:30:38.380779+0000","last_active":"2024-04-25T01:30:39.113965+0000","last_peered":"2024-04-25T01:30:39.113965+0000","last_clean":"2024-04-25T01:30:39.113965+0000","last_became_active":"2024-04-25T01:30:38.379959+0000","last_became_peered":"2024-04-25T01:30:38.379959+0000","last_unstale":"2024-04-25T01:30:39.113965+0000","last_undegraded":"2024-04-25T01:30:39.113965+0000","last_fullsized":"2024-04-25T01:30:39.113965+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:48:01.816311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115462+0000","last_change":"2024-04-25T01:30:33.345730+0000","last_active":"2024-04-25T01:30:39.115462+0000","last_peered":"2024-04-25T01:30:39.115462+0000","last_clean":"2024-04-25T01:30:39.115462+0000","last_became_active":"2024-04-25T01:30:33.344832+0000","last_became_peered":"2024-04-25T01:30:33.344832+0000","last_unstale":"2024-04-25T01:30:39.115462+0000","last_undegraded":"2024-04-25T01:30:39.115462+0000","last_fullsized":"2024-04-25T01:30:39.115462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:47.018811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114584+0000","last_change":"2024-04-25T01:30:34.355324+0000","last_active":"2024-04-25T01:30:39.114584+0000","last_peered":"2024-04-25T01:30:39.114584+0000","last_clean":"2024-04-25T01:30:39.114584+0000","last_became_active":"2024-04-25T01:30:34.354954+0000","last_became_peered":"2024-04-25T01:30:34.354954+0000","last_unstale":"2024-04-25T01:30:39.114584+0000","last_undegraded":"2024-04-25T01:30:39.114584+0000","last_fullsized":"2024-04-25T01:30:39.114584+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:36:41.932391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114343+0000","last_change":"2024-04-25T01:30:38.374386+0000","last_active":"2024-04-25T01:30:39.114343+0000","last_peered":"2024-04-25T01:30:39.114343+0000","last_clean":"2024-04-25T01:30:39.114343+0000","last_became_active":"2024-04-25T01:30:38.373454+0000","last_became_peered":"2024-04-25T01:30:38.373454+0000","last_unstale":"2024-04-25T01:30:39.114343+0000","last_undegraded":"2024-04-25T01:30:39.114343+0000","last_fullsized":"2024-04-25T01:30:39.114343+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:54.456218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.455119+0000","last_change":"2024-04-25T01:30:33.339821+0000","last_active":"2024-04-25T01:30:39.455119+0000","last_peered":"2024-04-25T01:30:39.455119+0000","last_clean":"2024-04-25T01:30:39.455119+0000","last_became_active":"2024-04-25T01:30:33.339500+0000","last_became_peered":"2024-04-25T01:30:33.339500+0000","last_unstale":"2024-04-25T01:30:39.455119+0000","last_undegraded":"2024-04-25T01:30:39.455119+0000","last_fullsized":"2024-04-25T01:30:39.455119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:44:25.880828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114676+0000","last_change":"2024-04-25T01:30:34.362736+0000","last_active":"2024-04-25T01:30:39.114676+0000","last_peered":"2024-04-25T01:30:39.114676+0000","last_clean":"2024-04-25T01:30:39.114676+0000","last_became_active":"2024-04-25T01:30:34.362371+0000","last_became_peered":"2024-04-25T01:30:34.362371+0000","last_unstale":"2024-04-25T01:30:39.114676+0000","last_undegraded":"2024-04-25T01:30:39.114676+0000","last_fullsized":"2024-04-25T01:30:39.114676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:57:36.088231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.953698+0000","last_change":"2024-04-25T01:30:34.359217+0000","last_active":"2024-04-25T01:30:38.953698+0000","last_peered":"2024-04-25T01:30:38.953698+0000","last_clean":"2024-04-25T01:30:38.953698+0000","last_became_active":"2024-04-25T01:30:34.357095+0000","last_became_peered":"2024-04-25T01:30:34.357095+0000","last_unstale":"2024-04-25T01:30:38.953698+0000","last_undegraded":"2024-04-25T01:30:38.953698+0000","last_fullsized":"2024-04-25T01:30:38.953698+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:58.603484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.805053+0000","last_change":"2024-04-25T01:30:34.359513+0000","last_active":"2024-04-25T01:30:38.805053+0000","last_peered":"2024-04-25T01:30:38.805053+0000","last_clean":"2024-04-25T01:30:38.805053+0000","last_became_active":"2024-04-25T01:30:34.359089+0000","last_became_peered":"2024-04-25T01:30:34.359089+0000","last_unstale":"2024-04-25T01:30:38.805053+0000","last_undegraded":"2024-04-25T01:30:38.805053+0000","last_fullsized":"2024-04-25T01:30:38.805053+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:36.159703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804757+0000","last_change":"2024-04-25T01:30:34.368739+0000","last_active":"2024-04-25T01:30:38.804757+0000","last_peered":"2024-04-25T01:30:38.804757+0000","last_clean":"2024-04-25T01:30:38.804757+0000","last_became_active":"2024-04-25T01:30:34.368392+0000","last_became_peered":"2024-04-25T01:30:34.368392+0000","last_unstale":"2024-04-25T01:30:38.804757+0000","last_undegraded":"2024-04-25T01:30:38.804757+0000","last_fullsized":"2024-04-25T01:30:38.804757+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:39:32.241889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114249+0000","last_change":"2024-04-25T01:30:34.360136+0000","last_active":"2024-04-25T01:30:39.114249+0000","last_peered":"2024-04-25T01:30:39.114249+0000","last_clean":"2024-04-25T01:30:39.114249+0000","last_became_active":"2024-04-25T01:30:34.359744+0000","last_became_peered":"2024-04-25T01:30:34.359744+0000","last_unstale":"2024-04-25T01:30:39.114249+0000","last_undegraded":"2024-04-25T01:30:39.114249+0000","last_fullsized":"2024-04-25T01:30:39.114249+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:03:35.517440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.455094+0000","last_change":"2024-04-25T01:30:34.360979+0000","last_active":"2024-04-25T01:30:39.455094+0000","last_peered":"2024-04-25T01:30:39.455094+0000","last_clean":"2024-04-25T01:30:39.455094+0000","last_became_active":"2024-04-25T01:30:34.360630+0000","last_became_peered":"2024-04-25T01:30:34.360630+0000","last_unstale":"2024-04-25T01:30:39.455094+0000","last_undegraded":"2024-04-25T01:30:39.455094+0000","last_fullsized":"2024-04-25T01:30:39.455094+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:05:19.284577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.953993+0000","last_change":"2024-04-25T01:30:34.362621+0000","last_active":"2024-04-25T01:30:38.953993+0000","last_peered":"2024-04-25T01:30:38.953993+0000","last_clean":"2024-04-25T01:30:38.953993+0000","last_became_active":"2024-04-25T01:30:34.362290+0000","last_became_peered":"2024-04-25T01:30:34.362290+0000","last_unstale":"2024-04-25T01:30:38.953993+0000","last_undegraded":"2024-04-25T01:30:38.953993+0000","last_fullsized":"2024-04-25T01:30:38.953993+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:28:07.750906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.805533+0000","last_change":"2024-04-25T01:30:34.351339+0000","last_active":"2024-04-25T01:30:38.805533+0000","last_peered":"2024-04-25T01:30:38.805533+0000","last_clean":"2024-04-25T01:30:38.805533+0000","last_became_active":"2024-04-25T01:30:34.350655+0000","last_became_peered":"2024-04-25T01:30:34.350655+0000","last_unstale":"2024-04-25T01:30:38.805533+0000","last_undegraded":"2024-04-25T01:30:38.805533+0000","last_fullsized":"2024-04-25T01:30:38.805533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:05.322547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115268+0000","last_change":"2024-04-25T01:30:34.353876+0000","last_active":"2024-04-25T01:30:39.115268+0000","last_peered":"2024-04-25T01:30:39.115268+0000","last_clean":"2024-04-25T01:30:39.115268+0000","last_became_active":"2024-04-25T01:30:34.352942+0000","last_became_peered":"2024-04-25T01:30:34.352942+0000","last_unstale":"2024-04-25T01:30:39.115268+0000","last_undegraded":"2024-04-25T01:30:39.115268+0000","last_fullsized":"2024-04-25T01:30:39.115268+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:47.260171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114774+0000","last_change":"2024-04-25T01:30:34.352039+0000","last_active":"2024-04-25T01:30:39.114774+0000","last_peered":"2024-04-25T01:30:39.114774+0000","last_clean":"2024-04-25T01:30:39.114774+0000","last_became_active":"2024-04-25T01:30:34.351612+0000","last_became_peered":"2024-04-25T01:30:34.351612+0000","last_unstale":"2024-04-25T01:30:39.114774+0000","last_undegraded":"2024-04-25T01:30:39.114774+0000","last_fullsized":"2024-04-25T01:30:39.114774+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:03.607711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114518+0000","last_change":"2024-04-25T01:30:34.353692+0000","last_active":"2024-04-25T01:30:39.114518+0000","last_peered":"2024-04-25T01:30:39.114518+0000","last_clean":"2024-04-25T01:30:39.114518+0000","last_became_active":"2024-04-25T01:30:34.353035+0000","last_became_peered":"2024-04-25T01:30:34.353035+0000","last_unstale":"2024-04-25T01:30:39.114518+0000","last_undegraded":"2024-04-25T01:30:39.114518+0000","last_fullsized":"2024-04-25T01:30:39.114518+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:15:54.540315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114965+0000","last_change":"2024-04-25T01:30:34.370797+0000","last_active":"2024-04-25T01:30:39.114965+0000","last_peered":"2024-04-25T01:30:39.114965+0000","last_clean":"2024-04-25T01:30:39.114965+0000","last_became_active":"2024-04-25T01:30:34.367533+0000","last_became_peered":"2024-04-25T01:30:34.367533+0000","last_unstale":"2024-04-25T01:30:39.114965+0000","last_undegraded":"2024-04-25T01:30:39.114965+0000","last_fullsized":"2024-04-25T01:30:39.114965+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:27:15.478808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.454766+0000","last_change":"2024-04-25T01:30:34.364843+0000","last_active":"2024-04-25T01:30:39.454766+0000","last_peered":"2024-04-25T01:30:39.454766+0000","last_clean":"2024-04-25T01:30:39.454766+0000","last_became_active":"2024-04-25T01:30:34.364515+0000","last_became_peered":"2024-04-25T01:30:34.364515+0000","last_unstale":"2024-04-25T01:30:39.454766+0000","last_undegraded":"2024-04-25T01:30:39.454766+0000","last_fullsized":"2024-04-25T01:30:39.454766+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:50:03.517814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.953848+0000","last_change":"2024-04-25T01:30:34.363715+0000","last_active":"2024-04-25T01:30:38.953848+0000","last_peered":"2024-04-25T01:30:38.953848+0000","last_clean":"2024-04-25T01:30:38.953848+0000","last_became_active":"2024-04-25T01:30:34.363383+0000","last_became_peered":"2024-04-25T01:30:34.363383+0000","last_unstale":"2024-04-25T01:30:38.953848+0000","last_undegraded":"2024-04-25T01:30:38.953848+0000","last_fullsized":"2024-04-25T01:30:38.953848+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:50:18.322550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6248,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":19968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318290,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":486531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579927,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36932,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708220,"statfs":{"total":95995035648,"available":95957217280,"internally_reserved":0,"allocated":1634304,"data_stored":1080505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743453,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1007616,"data_stored":485921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005090,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":486077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266727,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1622016,"data_stored":1078171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430253,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":485339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":489181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738443,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1007616,"data_stored":485429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":489215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1011712,"data_stored":485443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888118,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":485361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691890,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32752,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712400,"statfs":{"total":95995035648,"available":95961497600,"internally_reserved":0,"allocated":1609728,"data_stored":1075889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:31:03.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:31:04.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.059+0000 7f6357cb3640 1 -- 172.21.15.28:0/1161613774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6350102280 msgr2=0x7f6350104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:04.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.059+0000 7f6357cb3640 1 --2- 172.21.15.28:0/1161613774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6350102280 0x7f6350104670 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f634c009900 tx=0x7f634c02a410 comp rx=0 tx=0).stop 2024-04-25T01:31:04.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.059+0000 7f6357cb3640 1 -- 172.21.15.28:0/1161613774 shutdown_connections 2024-04-25T01:31:04.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.059+0000 7f6357cb3640 1 --2- 172.21.15.28:0/1161613774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63501074e0 0x7f63501098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:04.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.059+0000 7f6357cb3640 1 --2- 172.21.15.28:0/1161613774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6350104bb0 0x7f6350106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.059+0000 7f6357cb3640 1 --2- 172.21.15.28:0/1161613774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6350102280 0x7f6350104670 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.059+0000 7f6357cb3640 1 -- 172.21.15.28:0/1161613774 >> 172.21.15.28:0/1161613774 conn(0x7f63500fbc30 msgr2=0x7f63500fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.059+0000 7f6357cb3640 1 -- 172.21.15.28:0/1161613774 shutdown_connections 2024-04-25T01:31:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.059+0000 7f6357cb3640 1 -- 172.21.15.28:0/1161613774 wait complete. 2024-04-25T01:31:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.060+0000 7f6357cb3640 1 Processor -- start 2024-04-25T01:31:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.060+0000 7f6357cb3640 1 -- start start 2024-04-25T01:31:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.060+0000 7f6357cb3640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6350102280 0x7f6350197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.060+0000 7f6357cb3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6350104bb0 0x7f6350197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.060+0000 7f6357cb3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63501074e0 0x7f635019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.060+0000 7f6357cb3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f635010d440 con 0x7f6350104bb0 2024-04-25T01:31:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.060+0000 7f6357cb3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f635010d2c0 con 0x7f63501074e0 2024-04-25T01:31:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.060+0000 7f6357cb3640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f635010d5c0 con 0x7f6350102280 2024-04-25T01:31:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.060+0000 7f635524e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6350104bb0 0x7f6350197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:04.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.060+0000 7f635524e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6350104bb0 0x7f6350197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52130/0 (socket says 172.21.15.28:52130) 2024-04-25T01:31:04.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.060+0000 7f635524e640 1 -- 172.21.15.28:0/3131563462 learned_addr learned my addr 172.21.15.28:0/3131563462 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:04.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.060+0000 7f6356250640 1 --2- 172.21.15.28:0/3131563462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63501074e0 0x7f635019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:04.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.060+0000 7f6355a4f640 1 --2- 172.21.15.28:0/3131563462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6350102280 0x7f6350197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:04.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.061+0000 7f635524e640 1 -- 172.21.15.28:0/3131563462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6350102280 msgr2=0x7f6350197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:04.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.061+0000 7f635524e640 1 --2- 172.21.15.28:0/3131563462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6350102280 0x7f6350197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:04.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.061+0000 7f635524e640 1 -- 172.21.15.28:0/3131563462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63501074e0 msgr2=0x7f635019f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:04.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.061+0000 7f635524e640 1 --2- 172.21.15.28:0/3131563462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63501074e0 0x7f635019f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:04.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.061+0000 7f635524e640 1 -- 172.21.15.28:0/3131563462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f635019f6a0 con 0x7f6350104bb0 2024-04-25T01:31:04.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.061+0000 7f635524e640 1 --2- 172.21.15.28:0/3131563462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6350104bb0 0x7f6350197fc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f634000c970 tx=0x7f634000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:04.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.061+0000 7f6346ffd640 1 -- 172.21.15.28:0/3131563462 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6340014070 con 0x7f6350104bb0 2024-04-25T01:31:04.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.061+0000 7f6357cb3640 1 -- 172.21.15.28:0/3131563462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f635019f990 con 0x7f6350104bb0 2024-04-25T01:31:04.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.061+0000 7f6346ffd640 1 -- 172.21.15.28:0/3131563462 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6340004440 con 0x7f6350104bb0 2024-04-25T01:31:04.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.061+0000 7f6346ffd640 1 -- 172.21.15.28:0/3131563462 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634000f450 con 0x7f6350104bb0 2024-04-25T01:31:04.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.061+0000 7f6357cb3640 1 -- 172.21.15.28:0/3131563462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f635019fee0 con 0x7f6350104bb0 2024-04-25T01:31:04.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.062+0000 7f6357cb3640 1 -- 172.21.15.28:0/3131563462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63500fd720 con 0x7f6350104bb0 2024-04-25T01:31:04.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.063+0000 7f6346ffd640 1 -- 172.21.15.28:0/3131563462 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6340017070 con 0x7f6350104bb0 2024-04-25T01:31:04.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.064+0000 7f6346ffd640 1 --2- 172.21.15.28:0/3131563462 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f633c0a2c20 0x7f633c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:04.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.064+0000 7f6355a4f640 1 --2- 172.21.15.28:0/3131563462 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f633c0a2c20 0x7f633c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:04.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.064+0000 7f6346ffd640 1 -- 172.21.15.28:0/3131563462 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f63400cf1b0 con 0x7f6350104bb0 2024-04-25T01:31:04.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.064+0000 7f6355a4f640 1 --2- 172.21.15.28:0/3131563462 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f633c0a2c20 0x7f633c0a50e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f634c0314e0 tx=0x7f634c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:04.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.068+0000 7f6346ffd640 1 -- 172.21.15.28:0/3131563462 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f634001e020 con 0x7f6350104bb0 2024-04-25T01:31:04.102 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:03 smithi032 ceph-mon[32461]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:31:04.103 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:03 smithi032 ceph-mon[32461]: pgmap v312: 129 pgs: 129 active+clean; 583 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 5 op/s 2024-04-25T01:31:04.103 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:03 smithi032 ceph-mon[32461]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:31:04.103 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:03 smithi032 ceph-mon[32461]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T01:31:04.103 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:03 smithi032 ceph-mon[32461]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:31:04.103 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:03 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1676418052' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:31:04.103 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:03 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2992089776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:31:04.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:03 smithi130 ceph-mon[32702]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:31:04.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:03 smithi130 ceph-mon[32702]: pgmap v312: 129 pgs: 129 active+clean; 583 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 5 op/s 2024-04-25T01:31:04.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:03 smithi130 ceph-mon[32702]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:31:04.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:03 smithi130 ceph-mon[32702]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T01:31:04.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:03 smithi130 ceph-mon[32702]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:31:04.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:03 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1676418052' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:31:04.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:03 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2992089776' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:31:04.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.249+0000 7f6357cb3640 1 -- 172.21.15.28:0/3131563462 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6350061b80 con 0x7f633c0a2c20 2024-04-25T01:31:04.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.258+0000 7f6346ffd640 1 -- 172.21.15.28:0/3131563462 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340391 (secure 0 0 0) 0x7f6350061b80 con 0x7f633c0a2c20 2024-04-25T01:31:04.240 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:31:04.242 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T01:31:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.262+0000 7f6357cb3640 1 -- 172.21.15.28:0/3131563462 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f633c0a2c20 msgr2=0x7f633c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.262+0000 7f6357cb3640 1 --2- 172.21.15.28:0/3131563462 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f633c0a2c20 0x7f633c0a50e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f634c0314e0 tx=0x7f634c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:31:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.262+0000 7f6357cb3640 1 -- 172.21.15.28:0/3131563462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6350104bb0 msgr2=0x7f6350197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.262+0000 7f6357cb3640 1 --2- 172.21.15.28:0/3131563462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6350104bb0 0x7f6350197fc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f634000c970 tx=0x7f634000ce30 comp rx=0 tx=0).stop 2024-04-25T01:31:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.262+0000 7f6357cb3640 1 -- 172.21.15.28:0/3131563462 shutdown_connections 2024-04-25T01:31:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.262+0000 7f6357cb3640 1 --2- 172.21.15.28:0/3131563462 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f633c0a2c20 0x7f633c0a50e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:04.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.262+0000 7f6357cb3640 1 --2- 172.21.15.28:0/3131563462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63501074e0 0x7f635019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:04.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.262+0000 7f6357cb3640 1 --2- 172.21.15.28:0/3131563462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6350104bb0 0x7f6350197fc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:04.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.262+0000 7f6357cb3640 1 --2- 172.21.15.28:0/3131563462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6350102280 0x7f6350197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:04.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.262+0000 7f6357cb3640 1 -- 172.21.15.28:0/3131563462 >> 172.21.15.28:0/3131563462 conn(0x7f63500fbc30 msgr2=0x7f6350102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:04.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.262+0000 7f6357cb3640 1 -- 172.21.15.28:0/3131563462 shutdown_connections 2024-04-25T01:31:04.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:04.262+0000 7f6357cb3640 1 -- 172.21.15.28:0/3131563462 wait complete. 2024-04-25T01:31:04.251 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2024-04-25T01:31:02.335182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596616,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":290,"num_write_kb":2978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":233,"ondisk_log_size":233,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":379488,"kb_used_data":13652,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124562336,"statfs":{"total":1151940427776,"available":1151551832064,"internally_reserved":0,"allocated":13979648,"data_stored":7613062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1749,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003882"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.954172+0000","last_change":"2024-04-25T01:30:34.358339+0000","last_active":"2024-04-25T01:30:38.954172+0000","last_peered":"2024-04-25T01:30:38.954172+0000","last_clean":"2024-04-25T01:30:38.954172+0000","last_became_active":"2024-04-25T01:30:34.357884+0000","last_became_peered":"2024-04-25T01:30:34.357884+0000","last_unstale":"2024-04-25T01:30:38.954172+0000","last_undegraded":"2024-04-25T01:30:38.954172+0000","last_fullsized":"2024-04-25T01:30:38.954172+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:56.409080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114521+0000","last_change":"2024-04-25T01:30:34.359933+0000","last_active":"2024-04-25T01:30:39.114521+0000","last_peered":"2024-04-25T01:30:39.114521+0000","last_clean":"2024-04-25T01:30:39.114521+0000","last_became_active":"2024-04-25T01:30:34.359576+0000","last_became_peered":"2024-04-25T01:30:34.359576+0000","last_unstale":"2024-04-25T01:30:39.114521+0000","last_undegraded":"2024-04-25T01:30:39.114521+0000","last_fullsized":"2024-04-25T01:30:39.114521+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:48:47.764121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114072+0000","last_change":"2024-04-25T01:30:34.357772+0000","last_active":"2024-04-25T01:30:39.114072+0000","last_peered":"2024-04-25T01:30:39.114072+0000","last_clean":"2024-04-25T01:30:39.114072+0000","last_became_active":"2024-04-25T01:30:34.357434+0000","last_became_peered":"2024-04-25T01:30:34.357434+0000","last_unstale":"2024-04-25T01:30:39.114072+0000","last_undegraded":"2024-04-25T01:30:39.114072+0000","last_fullsized":"2024-04-25T01:30:39.114072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:19.926180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804914+0000","last_change":"2024-04-25T01:30:34.351250+0000","last_active":"2024-04-25T01:30:38.804914+0000","last_peered":"2024-04-25T01:30:38.804914+0000","last_clean":"2024-04-25T01:30:38.804914+0000","last_became_active":"2024-04-25T01:30:34.349976+0000","last_became_peered":"2024-04-25T01:30:34.349976+0000","last_unstale":"2024-04-25T01:30:38.804914+0000","last_undegraded":"2024-04-25T01:30:38.804914+0000","last_fullsized":"2024-04-25T01:30:38.804914+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:19:51.605307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114119+0000","last_change":"2024-04-25T01:30:34.367882+0000","last_active":"2024-04-25T01:30:39.114119+0000","last_peered":"2024-04-25T01:30:39.114119+0000","last_clean":"2024-04-25T01:30:39.114119+0000","last_became_active":"2024-04-25T01:30:34.366797+0000","last_became_peered":"2024-04-25T01:30:34.366797+0000","last_unstale":"2024-04-25T01:30:39.114119+0000","last_undegraded":"2024-04-25T01:30:39.114119+0000","last_fullsized":"2024-04-25T01:30:39.114119+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:35:23.439061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.116169+0000","last_change":"2024-04-25T01:30:34.352506+0000","last_active":"2024-04-25T01:30:39.116169+0000","last_peered":"2024-04-25T01:30:39.116169+0000","last_clean":"2024-04-25T01:30:39.116169+0000","last_became_active":"2024-04-25T01:30:34.352100+0000","last_became_peered":"2024-04-25T01:30:34.352100+0000","last_unstale":"2024-04-25T01:30:39.116169+0000","last_undegraded":"2024-04-25T01:30:39.116169+0000","last_fullsized":"2024-04-25T01:30:39.116169+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:38.113959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114501+0000","last_change":"2024-04-25T01:30:34.355352+0000","last_active":"2024-04-25T01:30:39.114501+0000","last_peered":"2024-04-25T01:30:39.114501+0000","last_clean":"2024-04-25T01:30:39.114501+0000","last_became_active":"2024-04-25T01:30:34.355008+0000","last_became_peered":"2024-04-25T01:30:34.355008+0000","last_unstale":"2024-04-25T01:30:39.114501+0000","last_undegraded":"2024-04-25T01:30:39.114501+0000","last_fullsized":"2024-04-25T01:30:39.114501+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:23.582742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114692+0000","last_change":"2024-04-25T01:30:34.360644+0000","last_active":"2024-04-25T01:30:39.114692+0000","last_peered":"2024-04-25T01:30:39.114692+0000","last_clean":"2024-04-25T01:30:39.114692+0000","last_became_active":"2024-04-25T01:30:34.357841+0000","last_became_peered":"2024-04-25T01:30:34.357841+0000","last_unstale":"2024-04-25T01:30:39.114692+0000","last_undegraded":"2024-04-25T01:30:39.114692+0000","last_fullsized":"2024-04-25T01:30:39.114692+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:26:21.606219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115818+0000","last_change":"2024-04-25T01:30:34.358417+0000","last_active":"2024-04-25T01:30:39.115818+0000","last_peered":"2024-04-25T01:30:39.115818+0000","last_clean":"2024-04-25T01:30:39.115818+0000","last_became_active":"2024-04-25T01:30:34.358039+0000","last_became_peered":"2024-04-25T01:30:34.358039+0000","last_unstale":"2024-04-25T01:30:39.115818+0000","last_undegraded":"2024-04-25T01:30:39.115818+0000","last_fullsized":"2024-04-25T01:30:39.115818+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:42.600693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.116709+0000","last_change":"2024-04-25T01:30:34.357803+0000","last_active":"2024-04-25T01:30:39.116709+0000","last_peered":"2024-04-25T01:30:39.116709+0000","last_clean":"2024-04-25T01:30:39.116709+0000","last_became_active":"2024-04-25T01:30:34.357391+0000","last_became_peered":"2024-04-25T01:30:34.357391+0000","last_unstale":"2024-04-25T01:30:39.116709+0000","last_undegraded":"2024-04-25T01:30:39.116709+0000","last_fullsized":"2024-04-25T01:30:39.116709+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:57.857303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115350+0000","last_change":"2024-04-25T01:30:34.363340+0000","last_active":"2024-04-25T01:30:39.115350+0000","last_peered":"2024-04-25T01:30:39.115350+0000","last_clean":"2024-04-25T01:30:39.115350+0000","last_became_active":"2024-04-25T01:30:34.362980+0000","last_became_peered":"2024-04-25T01:30:34.362980+0000","last_unstale":"2024-04-25T01:30:39.115350+0000","last_undegraded":"2024-04-25T01:30:39.115350+0000","last_fullsized":"2024-04-25T01:30:39.115350+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:46.291459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804860+0000","last_change":"2024-04-25T01:30:34.358898+0000","last_active":"2024-04-25T01:30:38.804860+0000","last_peered":"2024-04-25T01:30:38.804860+0000","last_clean":"2024-04-25T01:30:38.804860+0000","last_became_active":"2024-04-25T01:30:34.358495+0000","last_became_peered":"2024-04-25T01:30:34.358495+0000","last_unstale":"2024-04-25T01:30:38.804860+0000","last_undegraded":"2024-04-25T01:30:38.804860+0000","last_fullsized":"2024-04-25T01:30:38.804860+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:34:21.860010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.454881+0000","last_change":"2024-04-25T01:30:34.363708+0000","last_active":"2024-04-25T01:30:39.454881+0000","last_peered":"2024-04-25T01:30:39.454881+0000","last_clean":"2024-04-25T01:30:39.454881+0000","last_became_active":"2024-04-25T01:30:34.363207+0000","last_became_peered":"2024-04-25T01:30:34.363207+0000","last_unstale":"2024-04-25T01:30:39.454881+0000","last_undegraded":"2024-04-25T01:30:39.454881+0000","last_fullsized":"2024-04-25T01:30:39.454881+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:53.442987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114013+0000","last_change":"2024-04-25T01:30:34.371031+0000","last_active":"2024-04-25T01:30:39.114013+0000","last_peered":"2024-04-25T01:30:39.114013+0000","last_clean":"2024-04-25T01:30:39.114013+0000","last_became_active":"2024-04-25T01:30:34.370687+0000","last_became_peered":"2024-04-25T01:30:34.370687+0000","last_unstale":"2024-04-25T01:30:39.114013+0000","last_undegraded":"2024-04-25T01:30:39.114013+0000","last_fullsized":"2024-04-25T01:30:39.114013+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:03:20.700960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114529+0000","last_change":"2024-04-25T01:30:34.366212+0000","last_active":"2024-04-25T01:30:39.114529+0000","last_peered":"2024-04-25T01:30:39.114529+0000","last_clean":"2024-04-25T01:30:39.114529+0000","last_became_active":"2024-04-25T01:30:34.365849+0000","last_became_peered":"2024-04-25T01:30:34.365849+0000","last_unstale":"2024-04-25T01:30:39.114529+0000","last_undegraded":"2024-04-25T01:30:39.114529+0000","last_fullsized":"2024-04-25T01:30:39.114529+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:24:02.397615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114257+0000","last_change":"2024-04-25T01:30:34.367380+0000","last_active":"2024-04-25T01:30:39.114257+0000","last_peered":"2024-04-25T01:30:39.114257+0000","last_clean":"2024-04-25T01:30:39.114257+0000","last_became_active":"2024-04-25T01:30:34.366872+0000","last_became_peered":"2024-04-25T01:30:34.366872+0000","last_unstale":"2024-04-25T01:30:39.114257+0000","last_undegraded":"2024-04-25T01:30:39.114257+0000","last_fullsized":"2024-04-25T01:30:39.114257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:18.767108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.954033+0000","last_change":"2024-04-25T01:30:34.359752+0000","last_active":"2024-04-25T01:30:38.954033+0000","last_peered":"2024-04-25T01:30:38.954033+0000","last_clean":"2024-04-25T01:30:38.954033+0000","last_became_active":"2024-04-25T01:30:34.358403+0000","last_became_peered":"2024-04-25T01:30:34.358403+0000","last_unstale":"2024-04-25T01:30:38.954033+0000","last_undegraded":"2024-04-25T01:30:38.954033+0000","last_fullsized":"2024-04-25T01:30:38.954033+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:14.833126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.803536+0000","last_change":"2024-04-25T01:30:34.361373+0000","last_active":"2024-04-25T01:30:38.803536+0000","last_peered":"2024-04-25T01:30:38.803536+0000","last_clean":"2024-04-25T01:30:38.803536+0000","last_became_active":"2024-04-25T01:30:34.361046+0000","last_became_peered":"2024-04-25T01:30:34.361046+0000","last_unstale":"2024-04-25T01:30:38.803536+0000","last_undegraded":"2024-04-25T01:30:38.803536+0000","last_fullsized":"2024-04-25T01:30:38.803536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:21.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.805495+0000","last_change":"2024-04-25T01:30:34.367975+0000","last_active":"2024-04-25T01:30:38.805495+0000","last_peered":"2024-04-25T01:30:38.805495+0000","last_clean":"2024-04-25T01:30:38.805495+0000","last_became_active":"2024-04-25T01:30:34.366383+0000","last_became_peered":"2024-04-25T01:30:34.366383+0000","last_unstale":"2024-04-25T01:30:38.805495+0000","last_undegraded":"2024-04-25T01:30:38.805495+0000","last_fullsized":"2024-04-25T01:30:38.805495+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:53:40.279067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804884+0000","last_change":"2024-04-25T01:30:38.382168+0000","last_active":"2024-04-25T01:30:38.804884+0000","last_peered":"2024-04-25T01:30:38.804884+0000","last_clean":"2024-04-25T01:30:38.804884+0000","last_became_active":"2024-04-25T01:30:38.381740+0000","last_became_peered":"2024-04-25T01:30:38.381740+0000","last_unstale":"2024-04-25T01:30:38.804884+0000","last_undegraded":"2024-04-25T01:30:38.804884+0000","last_fullsized":"2024-04-25T01:30:38.804884+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:27.425991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.803674+0000","last_change":"2024-04-25T01:30:33.345547+0000","last_active":"2024-04-25T01:30:38.803674+0000","last_peered":"2024-04-25T01:30:38.803674+0000","last_clean":"2024-04-25T01:30:38.803674+0000","last_became_active":"2024-04-25T01:30:33.345229+0000","last_became_peered":"2024-04-25T01:30:33.345229+0000","last_unstale":"2024-04-25T01:30:38.803674+0000","last_undegraded":"2024-04-25T01:30:38.803674+0000","last_fullsized":"2024-04-25T01:30:38.803674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:32:46.637720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115066+0000","last_change":"2024-04-25T01:30:34.354703+0000","last_active":"2024-04-25T01:30:39.115066+0000","last_peered":"2024-04-25T01:30:39.115066+0000","last_clean":"2024-04-25T01:30:39.115066+0000","last_became_active":"2024-04-25T01:30:34.354362+0000","last_became_peered":"2024-04-25T01:30:34.354362+0000","last_unstale":"2024-04-25T01:30:39.115066+0000","last_undegraded":"2024-04-25T01:30:39.115066+0000","last_fullsized":"2024-04-25T01:30:39.115066+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:51.358599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804546+0000","last_change":"2024-04-25T01:30:38.380912+0000","last_active":"2024-04-25T01:30:38.804546+0000","last_peered":"2024-04-25T01:30:38.804546+0000","last_clean":"2024-04-25T01:30:38.804546+0000","last_became_active":"2024-04-25T01:30:38.380439+0000","last_became_peered":"2024-04-25T01:30:38.380439+0000","last_unstale":"2024-04-25T01:30:38.804546+0000","last_undegraded":"2024-04-25T01:30:38.804546+0000","last_fullsized":"2024-04-25T01:30:38.804546+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:22:00.527268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115074+0000","last_change":"2024-04-25T01:30:33.345620+0000","last_active":"2024-04-25T01:30:39.115074+0000","last_peered":"2024-04-25T01:30:39.115074+0000","last_clean":"2024-04-25T01:30:39.115074+0000","last_became_active":"2024-04-25T01:30:33.345315+0000","last_became_peered":"2024-04-25T01:30:33.345315+0000","last_unstale":"2024-04-25T01:30:39.115074+0000","last_undegraded":"2024-04-25T01:30:39.115074+0000","last_fullsized":"2024-04-25T01:30:39.115074+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:48:00.083728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114067+0000","last_change":"2024-04-25T01:30:34.356589+0000","last_active":"2024-04-25T01:30:39.114067+0000","last_peered":"2024-04-25T01:30:39.114067+0000","last_clean":"2024-04-25T01:30:39.114067+0000","last_became_active":"2024-04-25T01:30:34.356114+0000","last_became_peered":"2024-04-25T01:30:34.356114+0000","last_unstale":"2024-04-25T01:30:39.114067+0000","last_undegraded":"2024-04-25T01:30:39.114067+0000","last_fullsized":"2024-04-25T01:30:39.114067+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:39.039517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804557+0000","last_change":"2024-04-25T01:30:38.374656+0000","last_active":"2024-04-25T01:30:38.804557+0000","last_peered":"2024-04-25T01:30:38.804557+0000","last_clean":"2024-04-25T01:30:38.804557+0000","last_became_active":"2024-04-25T01:30:38.374189+0000","last_became_peered":"2024-04-25T01:30:38.374189+0000","last_unstale":"2024-04-25T01:30:38.804557+0000","last_undegraded":"2024-04-25T01:30:38.804557+0000","last_fullsized":"2024-04-25T01:30:38.804557+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:47:24.261071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:31:00.451120+0000","last_change":"2024-04-25T01:30:33.345575+0000","last_active":"2024-04-25T01:31:00.451120+0000","last_peered":"2024-04-25T01:31:00.451120+0000","last_clean":"2024-04-25T01:31:00.451120+0000","last_became_active":"2024-04-25T01:30:33.345231+0000","last_became_peered":"2024-04-25T01:30:33.345231+0000","last_unstale":"2024-04-25T01:31:00.451120+0000","last_undegraded":"2024-04-25T01:31:00.451120+0000","last_fullsized":"2024-04-25T01:31:00.451120+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:04:21.090788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115206+0000","last_change":"2024-04-25T01:30:34.371317+0000","last_active":"2024-04-25T01:30:39.115206+0000","last_peered":"2024-04-25T01:30:39.115206+0000","last_clean":"2024-04-25T01:30:39.115206+0000","last_became_active":"2024-04-25T01:30:34.370926+0000","last_became_peered":"2024-04-25T01:30:34.370926+0000","last_unstale":"2024-04-25T01:30:39.115206+0000","last_undegraded":"2024-04-25T01:30:39.115206+0000","last_fullsized":"2024-04-25T01:30:39.115206+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:58:39.898471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804491+0000","last_change":"2024-04-25T01:30:38.380982+0000","last_active":"2024-04-25T01:30:38.804491+0000","last_peered":"2024-04-25T01:30:38.804491+0000","last_clean":"2024-04-25T01:30:38.804491+0000","last_became_active":"2024-04-25T01:30:38.380510+0000","last_became_peered":"2024-04-25T01:30:38.380510+0000","last_unstale":"2024-04-25T01:30:38.804491+0000","last_undegraded":"2024-04-25T01:30:38.804491+0000","last_fullsized":"2024-04-25T01:30:38.804491+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:00.510697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114578+0000","last_change":"2024-04-25T01:30:33.351561+0000","last_active":"2024-04-25T01:30:39.114578+0000","last_peered":"2024-04-25T01:30:39.114578+0000","last_clean":"2024-04-25T01:30:39.114578+0000","last_became_active":"2024-04-25T01:30:33.351230+0000","last_became_peered":"2024-04-25T01:30:33.351230+0000","last_unstale":"2024-04-25T01:30:39.114578+0000","last_undegraded":"2024-04-25T01:30:39.114578+0000","last_fullsized":"2024-04-25T01:30:39.114578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:56.594457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115362+0000","last_change":"2024-04-25T01:30:34.358821+0000","last_active":"2024-04-25T01:30:39.115362+0000","last_peered":"2024-04-25T01:30:39.115362+0000","last_clean":"2024-04-25T01:30:39.115362+0000","last_became_active":"2024-04-25T01:30:34.358303+0000","last_became_peered":"2024-04-25T01:30:34.358303+0000","last_unstale":"2024-04-25T01:30:39.115362+0000","last_undegraded":"2024-04-25T01:30:39.115362+0000","last_fullsized":"2024-04-25T01:30:39.115362+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:26:23.352101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.805384+0000","last_change":"2024-04-25T01:30:38.372281+0000","last_active":"2024-04-25T01:30:38.805384+0000","last_peered":"2024-04-25T01:30:38.805384+0000","last_clean":"2024-04-25T01:30:38.805384+0000","last_became_active":"2024-04-25T01:30:38.371876+0000","last_became_peered":"2024-04-25T01:30:38.371876+0000","last_unstale":"2024-04-25T01:30:38.805384+0000","last_undegraded":"2024-04-25T01:30:38.805384+0000","last_fullsized":"2024-04-25T01:30:38.805384+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:54.254011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.442985+0000","last_change":"2024-04-25T01:30:33.347262+0000","last_active":"2024-04-25T01:30:58.442985+0000","last_peered":"2024-04-25T01:30:58.442985+0000","last_clean":"2024-04-25T01:30:58.442985+0000","last_became_active":"2024-04-25T01:30:33.346900+0000","last_became_peered":"2024-04-25T01:30:33.346900+0000","last_unstale":"2024-04-25T01:30:58.442985+0000","last_undegraded":"2024-04-25T01:30:58.442985+0000","last_fullsized":"2024-04-25T01:30:58.442985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:29:30.960456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114160+0000","last_change":"2024-04-25T01:30:34.364870+0000","last_active":"2024-04-25T01:30:39.114160+0000","last_peered":"2024-04-25T01:30:39.114160+0000","last_clean":"2024-04-25T01:30:39.114160+0000","last_became_active":"2024-04-25T01:30:34.364459+0000","last_became_peered":"2024-04-25T01:30:34.364459+0000","last_unstale":"2024-04-25T01:30:39.114160+0000","last_undegraded":"2024-04-25T01:30:39.114160+0000","last_fullsized":"2024-04-25T01:30:39.114160+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:42:45.311668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.805090+0000","last_change":"2024-04-25T01:30:38.382175+0000","last_active":"2024-04-25T01:30:38.805090+0000","last_peered":"2024-04-25T01:30:38.805090+0000","last_clean":"2024-04-25T01:30:38.805090+0000","last_became_active":"2024-04-25T01:30:38.381771+0000","last_became_peered":"2024-04-25T01:30:38.381771+0000","last_unstale":"2024-04-25T01:30:38.805090+0000","last_undegraded":"2024-04-25T01:30:38.805090+0000","last_fullsized":"2024-04-25T01:30:38.805090+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:25:31.041566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.116119+0000","last_change":"2024-04-25T01:30:33.343605+0000","last_active":"2024-04-25T01:30:39.116119+0000","last_peered":"2024-04-25T01:30:39.116119+0000","last_clean":"2024-04-25T01:30:39.116119+0000","last_became_active":"2024-04-25T01:30:33.343254+0000","last_became_peered":"2024-04-25T01:30:33.343254+0000","last_unstale":"2024-04-25T01:30:39.116119+0000","last_undegraded":"2024-04-25T01:30:39.116119+0000","last_fullsized":"2024-04-25T01:30:39.116119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:18:16.156285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115323+0000","last_change":"2024-04-25T01:30:34.354146+0000","last_active":"2024-04-25T01:30:39.115323+0000","last_peered":"2024-04-25T01:30:39.115323+0000","last_clean":"2024-04-25T01:30:39.115323+0000","last_became_active":"2024-04-25T01:30:34.353770+0000","last_became_peered":"2024-04-25T01:30:34.353770+0000","last_unstale":"2024-04-25T01:30:39.115323+0000","last_undegraded":"2024-04-25T01:30:39.115323+0000","last_fullsized":"2024-04-25T01:30:39.115323+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:06:28.725828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.954044+0000","last_change":"2024-04-25T01:30:38.383984+0000","last_active":"2024-04-25T01:30:38.954044+0000","last_peered":"2024-04-25T01:30:38.954044+0000","last_clean":"2024-04-25T01:30:38.954044+0000","last_became_active":"2024-04-25T01:30:38.383534+0000","last_became_peered":"2024-04-25T01:30:38.383534+0000","last_unstale":"2024-04-25T01:30:38.954044+0000","last_undegraded":"2024-04-25T01:30:38.954044+0000","last_fullsized":"2024-04-25T01:30:38.954044+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:50:07.924839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:59.443307+0000","last_change":"2024-04-25T01:30:33.351621+0000","last_active":"2024-04-25T01:30:59.443307+0000","last_peered":"2024-04-25T01:30:59.443307+0000","last_clean":"2024-04-25T01:30:59.443307+0000","last_became_active":"2024-04-25T01:30:33.351289+0000","last_became_peered":"2024-04-25T01:30:33.351289+0000","last_unstale":"2024-04-25T01:30:59.443307+0000","last_undegraded":"2024-04-25T01:30:59.443307+0000","last_fullsized":"2024-04-25T01:30:59.443307+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:46.044826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114864+0000","last_change":"2024-04-25T01:30:34.350685+0000","last_active":"2024-04-25T01:30:39.114864+0000","last_peered":"2024-04-25T01:30:39.114864+0000","last_clean":"2024-04-25T01:30:39.114864+0000","last_became_active":"2024-04-25T01:30:34.350308+0000","last_became_peered":"2024-04-25T01:30:34.350308+0000","last_unstale":"2024-04-25T01:30:39.114864+0000","last_undegraded":"2024-04-25T01:30:39.114864+0000","last_fullsized":"2024-04-25T01:30:39.114864+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:11.690381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114526+0000","last_change":"2024-04-25T01:30:38.377403+0000","last_active":"2024-04-25T01:30:39.114526+0000","last_peered":"2024-04-25T01:30:39.114526+0000","last_clean":"2024-04-25T01:30:39.114526+0000","last_became_active":"2024-04-25T01:30:38.376966+0000","last_became_peered":"2024-04-25T01:30:38.376966+0000","last_unstale":"2024-04-25T01:30:39.114526+0000","last_undegraded":"2024-04-25T01:30:39.114526+0000","last_fullsized":"2024-04-25T01:30:39.114526+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:31:10.229443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114716+0000","last_change":"2024-04-25T01:30:33.348455+0000","last_active":"2024-04-25T01:30:39.114716+0000","last_peered":"2024-04-25T01:30:39.114716+0000","last_clean":"2024-04-25T01:30:39.114716+0000","last_became_active":"2024-04-25T01:30:33.348142+0000","last_became_peered":"2024-04-25T01:30:33.348142+0000","last_unstale":"2024-04-25T01:30:39.114716+0000","last_undegraded":"2024-04-25T01:30:39.114716+0000","last_fullsized":"2024-04-25T01:30:39.114716+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:43:07.695580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.803468+0000","last_change":"2024-04-25T01:30:34.359949+0000","last_active":"2024-04-25T01:30:38.803468+0000","last_peered":"2024-04-25T01:30:38.803468+0000","last_clean":"2024-04-25T01:30:38.803468+0000","last_became_active":"2024-04-25T01:30:34.359609+0000","last_became_peered":"2024-04-25T01:30:34.359609+0000","last_unstale":"2024-04-25T01:30:38.803468+0000","last_undegraded":"2024-04-25T01:30:38.803468+0000","last_fullsized":"2024-04-25T01:30:38.803468+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:49.668800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114268+0000","last_change":"2024-04-25T01:30:38.380260+0000","last_active":"2024-04-25T01:30:39.114268+0000","last_peered":"2024-04-25T01:30:39.114268+0000","last_clean":"2024-04-25T01:30:39.114268+0000","last_became_active":"2024-04-25T01:30:38.379624+0000","last_became_peered":"2024-04-25T01:30:38.379624+0000","last_unstale":"2024-04-25T01:30:39.114268+0000","last_undegraded":"2024-04-25T01:30:39.114268+0000","last_fullsized":"2024-04-25T01:30:39.114268+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:26:35.235335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.437389+0000","last_change":"2024-04-25T01:30:33.349490+0000","last_active":"2024-04-25T01:30:58.437389+0000","last_peered":"2024-04-25T01:30:58.437389+0000","last_clean":"2024-04-25T01:30:58.437389+0000","last_became_active":"2024-04-25T01:30:33.349136+0000","last_became_peered":"2024-04-25T01:30:33.349136+0000","last_unstale":"2024-04-25T01:30:58.437389+0000","last_undegraded":"2024-04-25T01:30:58.437389+0000","last_fullsized":"2024-04-25T01:30:58.437389+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:16:23.552073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115747+0000","last_change":"2024-04-25T01:30:34.372701+0000","last_active":"2024-04-25T01:30:39.115747+0000","last_peered":"2024-04-25T01:30:39.115747+0000","last_clean":"2024-04-25T01:30:39.115747+0000","last_became_active":"2024-04-25T01:30:34.372321+0000","last_became_peered":"2024-04-25T01:30:34.372321+0000","last_unstale":"2024-04-25T01:30:39.115747+0000","last_undegraded":"2024-04-25T01:30:39.115747+0000","last_fullsized":"2024-04-25T01:30:39.115747+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:02.025015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.803500+0000","last_change":"2024-04-25T01:30:38.378036+0000","last_active":"2024-04-25T01:30:38.803500+0000","last_peered":"2024-04-25T01:30:38.803500+0000","last_clean":"2024-04-25T01:30:38.803500+0000","last_became_active":"2024-04-25T01:30:38.377602+0000","last_became_peered":"2024-04-25T01:30:38.377602+0000","last_unstale":"2024-04-25T01:30:38.803500+0000","last_undegraded":"2024-04-25T01:30:38.803500+0000","last_fullsized":"2024-04-25T01:30:38.803500+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:45.210518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.435196+0000","last_change":"2024-04-25T01:30:33.352928+0000","last_active":"2024-04-25T01:30:58.435196+0000","last_peered":"2024-04-25T01:30:58.435196+0000","last_clean":"2024-04-25T01:30:58.435196+0000","last_became_active":"2024-04-25T01:30:33.352567+0000","last_became_peered":"2024-04-25T01:30:33.352567+0000","last_unstale":"2024-04-25T01:30:58.435196+0000","last_undegraded":"2024-04-25T01:30:58.435196+0000","last_fullsized":"2024-04-25T01:30:58.435196+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:47:21.941179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114253+0000","last_change":"2024-04-25T01:30:34.366947+0000","last_active":"2024-04-25T01:30:39.114253+0000","last_peered":"2024-04-25T01:30:39.114253+0000","last_clean":"2024-04-25T01:30:39.114253+0000","last_became_active":"2024-04-25T01:30:34.366337+0000","last_became_peered":"2024-04-25T01:30:34.366337+0000","last_unstale":"2024-04-25T01:30:39.114253+0000","last_undegraded":"2024-04-25T01:30:39.114253+0000","last_fullsized":"2024-04-25T01:30:39.114253+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:48:43.528380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804886+0000","last_change":"2024-04-25T01:30:38.372369+0000","last_active":"2024-04-25T01:30:38.804886+0000","last_peered":"2024-04-25T01:30:38.804886+0000","last_clean":"2024-04-25T01:30:38.804886+0000","last_became_active":"2024-04-25T01:30:38.371920+0000","last_became_peered":"2024-04-25T01:30:38.371920+0000","last_unstale":"2024-04-25T01:30:38.804886+0000","last_undegraded":"2024-04-25T01:30:38.804886+0000","last_fullsized":"2024-04-25T01:30:38.804886+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:57.299089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115324+0000","last_change":"2024-04-25T01:30:33.346003+0000","last_active":"2024-04-25T01:30:39.115324+0000","last_peered":"2024-04-25T01:30:39.115324+0000","last_clean":"2024-04-25T01:30:39.115324+0000","last_became_active":"2024-04-25T01:30:33.345254+0000","last_became_peered":"2024-04-25T01:30:33.345254+0000","last_unstale":"2024-04-25T01:30:39.115324+0000","last_undegraded":"2024-04-25T01:30:39.115324+0000","last_fullsized":"2024-04-25T01:30:39.115324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:18:37.957047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115423+0000","last_change":"2024-04-25T01:30:34.366108+0000","last_active":"2024-04-25T01:30:39.115423+0000","last_peered":"2024-04-25T01:30:39.115423+0000","last_clean":"2024-04-25T01:30:39.115423+0000","last_became_active":"2024-04-25T01:30:34.365702+0000","last_became_peered":"2024-04-25T01:30:34.365702+0000","last_unstale":"2024-04-25T01:30:39.115423+0000","last_undegraded":"2024-04-25T01:30:39.115423+0000","last_fullsized":"2024-04-25T01:30:39.115423+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:24:06.802424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.803621+0000","last_change":"2024-04-25T01:30:38.376342+0000","last_active":"2024-04-25T01:30:38.803621+0000","last_peered":"2024-04-25T01:30:38.803621+0000","last_clean":"2024-04-25T01:30:38.803621+0000","last_became_active":"2024-04-25T01:30:38.375905+0000","last_became_peered":"2024-04-25T01:30:38.375905+0000","last_unstale":"2024-04-25T01:30:38.803621+0000","last_undegraded":"2024-04-25T01:30:38.803621+0000","last_fullsized":"2024-04-25T01:30:38.803621+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:31.615356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'12","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.437393+0000","last_change":"2024-04-25T01:30:33.350770+0000","last_active":"2024-04-25T01:30:58.437393+0000","last_peered":"2024-04-25T01:30:58.437393+0000","last_clean":"2024-04-25T01:30:58.437393+0000","last_became_active":"2024-04-25T01:30:33.350414+0000","last_became_peered":"2024-04-25T01:30:33.350414+0000","last_unstale":"2024-04-25T01:30:58.437393+0000","last_undegraded":"2024-04-25T01:30:58.437393+0000","last_fullsized":"2024-04-25T01:30:58.437393+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:01.771182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114585+0000","last_change":"2024-04-25T01:30:34.362532+0000","last_active":"2024-04-25T01:30:39.114585+0000","last_peered":"2024-04-25T01:30:39.114585+0000","last_clean":"2024-04-25T01:30:39.114585+0000","last_became_active":"2024-04-25T01:30:34.362137+0000","last_became_peered":"2024-04-25T01:30:34.362137+0000","last_unstale":"2024-04-25T01:30:39.114585+0000","last_undegraded":"2024-04-25T01:30:39.114585+0000","last_fullsized":"2024-04-25T01:30:39.114585+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:28.105250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115694+0000","last_change":"2024-04-25T01:30:38.374591+0000","last_active":"2024-04-25T01:30:39.115694+0000","last_peered":"2024-04-25T01:30:39.115694+0000","last_clean":"2024-04-25T01:30:39.115694+0000","last_became_active":"2024-04-25T01:30:38.373237+0000","last_became_peered":"2024-04-25T01:30:38.373237+0000","last_unstale":"2024-04-25T01:30:39.115694+0000","last_undegraded":"2024-04-25T01:30:39.115694+0000","last_fullsized":"2024-04-25T01:30:39.115694+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:21:25.937405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.805156+0000","last_change":"2024-04-25T01:30:33.344341+0000","last_active":"2024-04-25T01:30:38.805156+0000","last_peered":"2024-04-25T01:30:38.805156+0000","last_clean":"2024-04-25T01:30:38.805156+0000","last_became_active":"2024-04-25T01:30:33.343046+0000","last_became_peered":"2024-04-25T01:30:33.343046+0000","last_unstale":"2024-04-25T01:30:38.805156+0000","last_undegraded":"2024-04-25T01:30:38.805156+0000","last_fullsized":"2024-04-25T01:30:38.805156+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:52.291970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.454783+0000","last_change":"2024-04-25T01:30:34.354444+0000","last_active":"2024-04-25T01:30:39.454783+0000","last_peered":"2024-04-25T01:30:39.454783+0000","last_clean":"2024-04-25T01:30:39.454783+0000","last_became_active":"2024-04-25T01:30:34.354070+0000","last_became_peered":"2024-04-25T01:30:34.354070+0000","last_unstale":"2024-04-25T01:30:39.454783+0000","last_undegraded":"2024-04-25T01:30:39.454783+0000","last_fullsized":"2024-04-25T01:30:39.454783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:39.893995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.953846+0000","last_change":"2024-04-25T01:30:38.373570+0000","last_active":"2024-04-25T01:30:38.953846+0000","last_peered":"2024-04-25T01:30:38.953846+0000","last_clean":"2024-04-25T01:30:38.953846+0000","last_became_active":"2024-04-25T01:30:38.373144+0000","last_became_peered":"2024-04-25T01:30:38.373144+0000","last_unstale":"2024-04-25T01:30:38.953846+0000","last_undegraded":"2024-04-25T01:30:38.953846+0000","last_fullsized":"2024-04-25T01:30:38.953846+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:53:32.933998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.439139+0000","last_change":"2024-04-25T01:30:33.346145+0000","last_active":"2024-04-25T01:30:58.439139+0000","last_peered":"2024-04-25T01:30:58.439139+0000","last_clean":"2024-04-25T01:30:58.439139+0000","last_became_active":"2024-04-25T01:30:33.344294+0000","last_became_peered":"2024-04-25T01:30:33.344294+0000","last_unstale":"2024-04-25T01:30:58.439139+0000","last_undegraded":"2024-04-25T01:30:58.439139+0000","last_fullsized":"2024-04-25T01:30:58.439139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:37:17.147148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114821+0000","last_change":"2024-04-25T01:30:34.360769+0000","last_active":"2024-04-25T01:30:39.114821+0000","last_peered":"2024-04-25T01:30:39.114821+0000","last_clean":"2024-04-25T01:30:39.114821+0000","last_became_active":"2024-04-25T01:30:34.360076+0000","last_became_peered":"2024-04-25T01:30:34.360076+0000","last_unstale":"2024-04-25T01:30:39.114821+0000","last_undegraded":"2024-04-25T01:30:39.114821+0000","last_fullsized":"2024-04-25T01:30:39.114821+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:56:21.515511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.803481+0000","last_change":"2024-04-25T01:30:38.379198+0000","last_active":"2024-04-25T01:30:38.803481+0000","last_peered":"2024-04-25T01:30:38.803481+0000","last_clean":"2024-04-25T01:30:38.803481+0000","last_became_active":"2024-04-25T01:30:38.378798+0000","last_became_peered":"2024-04-25T01:30:38.378798+0000","last_unstale":"2024-04-25T01:30:38.803481+0000","last_undegraded":"2024-04-25T01:30:38.803481+0000","last_fullsized":"2024-04-25T01:30:38.803481+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:40.192239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804472+0000","last_change":"2024-04-25T01:30:33.343232+0000","last_active":"2024-04-25T01:30:38.804472+0000","last_peered":"2024-04-25T01:30:38.804472+0000","last_clean":"2024-04-25T01:30:38.804472+0000","last_became_active":"2024-04-25T01:30:33.342876+0000","last_became_peered":"2024-04-25T01:30:33.342876+0000","last_unstale":"2024-04-25T01:30:38.804472+0000","last_undegraded":"2024-04-25T01:30:38.804472+0000","last_fullsized":"2024-04-25T01:30:38.804472+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:45.032564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114383+0000","last_change":"2024-04-25T01:30:34.358210+0000","last_active":"2024-04-25T01:30:39.114383+0000","last_peered":"2024-04-25T01:30:39.114383+0000","last_clean":"2024-04-25T01:30:39.114383+0000","last_became_active":"2024-04-25T01:30:34.357862+0000","last_became_peered":"2024-04-25T01:30:34.357862+0000","last_unstale":"2024-04-25T01:30:39.114383+0000","last_undegraded":"2024-04-25T01:30:39.114383+0000","last_fullsized":"2024-04-25T01:30:39.114383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:43.751361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.455266+0000","last_change":"2024-04-25T01:30:38.380413+0000","last_active":"2024-04-25T01:30:39.455266+0000","last_peered":"2024-04-25T01:30:39.455266+0000","last_clean":"2024-04-25T01:30:39.455266+0000","last_became_active":"2024-04-25T01:30:38.379612+0000","last_became_peered":"2024-04-25T01:30:38.379612+0000","last_unstale":"2024-04-25T01:30:39.455266+0000","last_undegraded":"2024-04-25T01:30:39.455266+0000","last_fullsized":"2024-04-25T01:30:39.455266+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:45.328642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'16","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:59.444283+0000","last_change":"2024-04-25T01:30:33.344938+0000","last_active":"2024-04-25T01:30:59.444283+0000","last_peered":"2024-04-25T01:30:59.444283+0000","last_clean":"2024-04-25T01:30:59.444283+0000","last_became_active":"2024-04-25T01:30:33.344582+0000","last_became_peered":"2024-04-25T01:30:33.344582+0000","last_unstale":"2024-04-25T01:30:59.444283+0000","last_undegraded":"2024-04-25T01:30:59.444283+0000","last_fullsized":"2024-04-25T01:30:59.444283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:43:33.777319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114724+0000","last_change":"2024-04-25T01:30:34.354115+0000","last_active":"2024-04-25T01:30:39.114724+0000","last_peered":"2024-04-25T01:30:39.114724+0000","last_clean":"2024-04-25T01:30:39.114724+0000","last_became_active":"2024-04-25T01:30:34.353722+0000","last_became_peered":"2024-04-25T01:30:34.353722+0000","last_unstale":"2024-04-25T01:30:39.114724+0000","last_undegraded":"2024-04-25T01:30:39.114724+0000","last_fullsized":"2024-04-25T01:30:39.114724+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:22.003521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114604+0000","last_change":"2024-04-25T01:30:38.380914+0000","last_active":"2024-04-25T01:30:39.114604+0000","last_peered":"2024-04-25T01:30:39.114604+0000","last_clean":"2024-04-25T01:30:39.114604+0000","last_became_active":"2024-04-25T01:30:38.380477+0000","last_became_peered":"2024-04-25T01:30:38.380477+0000","last_unstale":"2024-04-25T01:30:39.114604+0000","last_undegraded":"2024-04-25T01:30:39.114604+0000","last_fullsized":"2024-04-25T01:30:39.114604+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:23:27.882266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114449+0000","last_change":"2024-04-25T01:30:33.345772+0000","last_active":"2024-04-25T01:30:39.114449+0000","last_peered":"2024-04-25T01:30:39.114449+0000","last_clean":"2024-04-25T01:30:39.114449+0000","last_became_active":"2024-04-25T01:30:33.345455+0000","last_became_peered":"2024-04-25T01:30:33.345455+0000","last_unstale":"2024-04-25T01:30:39.114449+0000","last_undegraded":"2024-04-25T01:30:39.114449+0000","last_fullsized":"2024-04-25T01:30:39.114449+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:14:57.465161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.953974+0000","last_change":"2024-04-25T01:30:34.358665+0000","last_active":"2024-04-25T01:30:38.953974+0000","last_peered":"2024-04-25T01:30:38.953974+0000","last_clean":"2024-04-25T01:30:38.953974+0000","last_became_active":"2024-04-25T01:30:34.358284+0000","last_became_peered":"2024-04-25T01:30:34.358284+0000","last_unstale":"2024-04-25T01:30:38.953974+0000","last_undegraded":"2024-04-25T01:30:38.953974+0000","last_fullsized":"2024-04-25T01:30:38.953974+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:12:22.832265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114254+0000","last_change":"2024-04-25T01:30:38.450027+0000","last_active":"2024-04-25T01:30:39.114254+0000","last_peered":"2024-04-25T01:30:39.114254+0000","last_clean":"2024-04-25T01:30:39.114254+0000","last_became_active":"2024-04-25T01:30:38.449564+0000","last_became_peered":"2024-04-25T01:30:38.449564+0000","last_unstale":"2024-04-25T01:30:39.114254+0000","last_undegraded":"2024-04-25T01:30:39.114254+0000","last_fullsized":"2024-04-25T01:30:39.114254+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:34:26.542351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.803395+0000","last_change":"2024-04-25T01:30:33.349757+0000","last_active":"2024-04-25T01:30:38.803395+0000","last_peered":"2024-04-25T01:30:38.803395+0000","last_clean":"2024-04-25T01:30:38.803395+0000","last_became_active":"2024-04-25T01:30:33.349439+0000","last_became_peered":"2024-04-25T01:30:33.349439+0000","last_unstale":"2024-04-25T01:30:38.803395+0000","last_undegraded":"2024-04-25T01:30:38.803395+0000","last_fullsized":"2024-04-25T01:30:38.803395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:35.514160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.116544+0000","last_change":"2024-04-25T01:30:34.367310+0000","last_active":"2024-04-25T01:30:39.116544+0000","last_peered":"2024-04-25T01:30:39.116544+0000","last_clean":"2024-04-25T01:30:39.116544+0000","last_became_active":"2024-04-25T01:30:34.366950+0000","last_became_peered":"2024-04-25T01:30:34.366950+0000","last_unstale":"2024-04-25T01:30:39.116544+0000","last_undegraded":"2024-04-25T01:30:39.116544+0000","last_fullsized":"2024-04-25T01:30:39.116544+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:26:17.700512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.455026+0000","last_change":"2024-04-25T01:30:38.372259+0000","last_active":"2024-04-25T01:30:39.455026+0000","last_peered":"2024-04-25T01:30:39.455026+0000","last_clean":"2024-04-25T01:30:39.455026+0000","last_became_active":"2024-04-25T01:30:38.371847+0000","last_became_peered":"2024-04-25T01:30:38.371847+0000","last_unstale":"2024-04-25T01:30:39.455026+0000","last_undegraded":"2024-04-25T01:30:39.455026+0000","last_fullsized":"2024-04-25T01:30:39.455026+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:55.047739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114803+0000","last_change":"2024-04-25T01:30:33.352078+0000","last_active":"2024-04-25T01:30:39.114803+0000","last_peered":"2024-04-25T01:30:39.114803+0000","last_clean":"2024-04-25T01:30:39.114803+0000","last_became_active":"2024-04-25T01:30:33.351725+0000","last_became_peered":"2024-04-25T01:30:33.351725+0000","last_unstale":"2024-04-25T01:30:39.114803+0000","last_undegraded":"2024-04-25T01:30:39.114803+0000","last_fullsized":"2024-04-25T01:30:39.114803+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:55:55.236040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.116297+0000","last_change":"2024-04-25T01:30:34.360324+0000","last_active":"2024-04-25T01:30:39.116297+0000","last_peered":"2024-04-25T01:30:39.116297+0000","last_clean":"2024-04-25T01:30:39.116297+0000","last_became_active":"2024-04-25T01:30:34.359114+0000","last_became_peered":"2024-04-25T01:30:34.359114+0000","last_unstale":"2024-04-25T01:30:39.116297+0000","last_undegraded":"2024-04-25T01:30:39.116297+0000","last_fullsized":"2024-04-25T01:30:39.116297+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:51.927640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.455117+0000","last_change":"2024-04-25T01:30:38.450650+0000","last_active":"2024-04-25T01:30:39.455117+0000","last_peered":"2024-04-25T01:30:39.455117+0000","last_clean":"2024-04-25T01:30:39.455117+0000","last_became_active":"2024-04-25T01:30:38.450149+0000","last_became_peered":"2024-04-25T01:30:38.450149+0000","last_unstale":"2024-04-25T01:30:39.455117+0000","last_undegraded":"2024-04-25T01:30:39.455117+0000","last_fullsized":"2024-04-25T01:30:39.455117+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:34.630803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114887+0000","last_change":"2024-04-25T01:30:34.352961+0000","last_active":"2024-04-25T01:30:39.114887+0000","last_peered":"2024-04-25T01:30:39.114887+0000","last_clean":"2024-04-25T01:30:39.114887+0000","last_became_active":"2024-04-25T01:30:34.352657+0000","last_became_peered":"2024-04-25T01:30:34.352657+0000","last_unstale":"2024-04-25T01:30:39.114887+0000","last_undegraded":"2024-04-25T01:30:39.114887+0000","last_fullsized":"2024-04-25T01:30:39.114887+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:20.934228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:59.444315+0000","last_change":"2024-04-25T01:30:33.344562+0000","last_active":"2024-04-25T01:30:59.444315+0000","last_peered":"2024-04-25T01:30:59.444315+0000","last_clean":"2024-04-25T01:30:59.444315+0000","last_became_active":"2024-04-25T01:30:33.344212+0000","last_became_peered":"2024-04-25T01:30:33.344212+0000","last_unstale":"2024-04-25T01:30:59.444315+0000","last_undegraded":"2024-04-25T01:30:59.444315+0000","last_fullsized":"2024-04-25T01:30:59.444315+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:20:51.110618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115132+0000","last_change":"2024-04-25T01:30:38.384316+0000","last_active":"2024-04-25T01:30:39.115132+0000","last_peered":"2024-04-25T01:30:39.115132+0000","last_clean":"2024-04-25T01:30:39.115132+0000","last_became_active":"2024-04-25T01:30:38.383775+0000","last_became_peered":"2024-04-25T01:30:38.383775+0000","last_unstale":"2024-04-25T01:30:39.115132+0000","last_undegraded":"2024-04-25T01:30:39.115132+0000","last_fullsized":"2024-04-25T01:30:39.115132+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:54.165691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114224+0000","last_change":"2024-04-25T01:30:34.355077+0000","last_active":"2024-04-25T01:30:39.114224+0000","last_peered":"2024-04-25T01:30:39.114224+0000","last_clean":"2024-04-25T01:30:39.114224+0000","last_became_active":"2024-04-25T01:30:34.352742+0000","last_became_peered":"2024-04-25T01:30:34.352742+0000","last_unstale":"2024-04-25T01:30:39.114224+0000","last_undegraded":"2024-04-25T01:30:39.114224+0000","last_fullsized":"2024-04-25T01:30:39.114224+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:48:31.495699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.438784+0000","last_change":"2024-04-25T01:30:33.345283+0000","last_active":"2024-04-25T01:30:58.438784+0000","last_peered":"2024-04-25T01:30:58.438784+0000","last_clean":"2024-04-25T01:30:58.438784+0000","last_became_active":"2024-04-25T01:30:33.344903+0000","last_became_peered":"2024-04-25T01:30:33.344903+0000","last_unstale":"2024-04-25T01:30:58.438784+0000","last_undegraded":"2024-04-25T01:30:58.438784+0000","last_fullsized":"2024-04-25T01:30:58.438784+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:57:01.982850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":359,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114627+0000","last_change":"2024-04-25T01:29:17.689639+0000","last_active":"2024-04-25T01:30:39.114627+0000","last_peered":"2024-04-25T01:30:39.114627+0000","last_clean":"2024-04-25T01:30:39.114627+0000","last_became_active":"2024-04-25T01:29:17.689046+0000","last_became_peered":"2024-04-25T01:29:17.689046+0000","last_unstale":"2024-04-25T01:30:39.114627+0000","last_undegraded":"2024-04-25T01:30:39.114627+0000","last_fullsized":"2024-04-25T01:30:39.114627+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:48.767945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:48.767945+0000","last_clean_scrub_stamp":"2024-04-25T01:25:48.767945+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:04:06.092632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115580+0000","last_change":"2024-04-25T01:30:38.450938+0000","last_active":"2024-04-25T01:30:39.115580+0000","last_peered":"2024-04-25T01:30:39.115580+0000","last_clean":"2024-04-25T01:30:39.115580+0000","last_became_active":"2024-04-25T01:30:38.450488+0000","last_became_peered":"2024-04-25T01:30:38.450488+0000","last_unstale":"2024-04-25T01:30:39.115580+0000","last_undegraded":"2024-04-25T01:30:39.115580+0000","last_fullsized":"2024-04-25T01:30:39.115580+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:44.778133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.454492+0000","last_change":"2024-04-25T01:30:34.352841+0000","last_active":"2024-04-25T01:30:39.454492+0000","last_peered":"2024-04-25T01:30:39.454492+0000","last_clean":"2024-04-25T01:30:39.454492+0000","last_became_active":"2024-04-25T01:30:34.352460+0000","last_became_peered":"2024-04-25T01:30:34.352460+0000","last_unstale":"2024-04-25T01:30:39.454492+0000","last_undegraded":"2024-04-25T01:30:39.454492+0000","last_fullsized":"2024-04-25T01:30:39.454492+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:18:46.579947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:59.442330+0000","last_change":"2024-04-25T01:30:33.348198+0000","last_active":"2024-04-25T01:30:59.442330+0000","last_peered":"2024-04-25T01:30:59.442330+0000","last_clean":"2024-04-25T01:30:59.442330+0000","last_became_active":"2024-04-25T01:30:33.347876+0000","last_became_peered":"2024-04-25T01:30:33.347876+0000","last_unstale":"2024-04-25T01:30:59.442330+0000","last_undegraded":"2024-04-25T01:30:59.442330+0000","last_fullsized":"2024-04-25T01:30:59.442330+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:48:33.116571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.953972+0000","last_change":"2024-04-25T01:30:38.372331+0000","last_active":"2024-04-25T01:30:38.953972+0000","last_peered":"2024-04-25T01:30:38.953972+0000","last_clean":"2024-04-25T01:30:38.953972+0000","last_became_active":"2024-04-25T01:30:38.371923+0000","last_became_peered":"2024-04-25T01:30:38.371923+0000","last_unstale":"2024-04-25T01:30:38.953972+0000","last_undegraded":"2024-04-25T01:30:38.953972+0000","last_fullsized":"2024-04-25T01:30:38.953972+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:06.416967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:31:00.451341+0000","last_change":"2024-04-25T01:30:33.346606+0000","last_active":"2024-04-25T01:31:00.451341+0000","last_peered":"2024-04-25T01:31:00.451341+0000","last_clean":"2024-04-25T01:31:00.451341+0000","last_became_active":"2024-04-25T01:30:33.346297+0000","last_became_peered":"2024-04-25T01:30:33.346297+0000","last_unstale":"2024-04-25T01:31:00.451341+0000","last_undegraded":"2024-04-25T01:31:00.451341+0000","last_fullsized":"2024-04-25T01:31:00.451341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:07:16.069285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.455021+0000","last_change":"2024-04-25T01:30:34.349136+0000","last_active":"2024-04-25T01:30:39.455021+0000","last_peered":"2024-04-25T01:30:39.455021+0000","last_clean":"2024-04-25T01:30:39.455021+0000","last_became_active":"2024-04-25T01:30:34.348810+0000","last_became_peered":"2024-04-25T01:30:34.348810+0000","last_unstale":"2024-04-25T01:30:39.455021+0000","last_undegraded":"2024-04-25T01:30:39.455021+0000","last_fullsized":"2024-04-25T01:30:39.455021+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:39:34.593986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114836+0000","last_change":"2024-04-25T01:30:38.450428+0000","last_active":"2024-04-25T01:30:39.114836+0000","last_peered":"2024-04-25T01:30:39.114836+0000","last_clean":"2024-04-25T01:30:39.114836+0000","last_became_active":"2024-04-25T01:30:38.449971+0000","last_became_peered":"2024-04-25T01:30:38.449971+0000","last_unstale":"2024-04-25T01:30:39.114836+0000","last_undegraded":"2024-04-25T01:30:39.114836+0000","last_fullsized":"2024-04-25T01:30:39.114836+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:52:37.046462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.803587+0000","last_change":"2024-04-25T01:30:33.348547+0000","last_active":"2024-04-25T01:30:38.803587+0000","last_peered":"2024-04-25T01:30:38.803587+0000","last_clean":"2024-04-25T01:30:38.803587+0000","last_became_active":"2024-04-25T01:30:33.348181+0000","last_became_peered":"2024-04-25T01:30:33.348181+0000","last_unstale":"2024-04-25T01:30:38.803587+0000","last_undegraded":"2024-04-25T01:30:38.803587+0000","last_fullsized":"2024-04-25T01:30:38.803587+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:11:58.696829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115147+0000","last_change":"2024-04-25T01:30:34.351987+0000","last_active":"2024-04-25T01:30:39.115147+0000","last_peered":"2024-04-25T01:30:39.115147+0000","last_clean":"2024-04-25T01:30:39.115147+0000","last_became_active":"2024-04-25T01:30:34.351571+0000","last_became_peered":"2024-04-25T01:30:34.351571+0000","last_unstale":"2024-04-25T01:30:39.115147+0000","last_undegraded":"2024-04-25T01:30:39.115147+0000","last_fullsized":"2024-04-25T01:30:39.115147+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:23.585865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804768+0000","last_change":"2024-04-25T01:30:38.372566+0000","last_active":"2024-04-25T01:30:38.804768+0000","last_peered":"2024-04-25T01:30:38.804768+0000","last_clean":"2024-04-25T01:30:38.804768+0000","last_became_active":"2024-04-25T01:30:38.372102+0000","last_became_peered":"2024-04-25T01:30:38.372102+0000","last_unstale":"2024-04-25T01:30:38.804768+0000","last_undegraded":"2024-04-25T01:30:38.804768+0000","last_fullsized":"2024-04-25T01:30:38.804768+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:22:16.791842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.440012+0000","last_change":"2024-04-25T01:30:33.344065+0000","last_active":"2024-04-25T01:30:58.440012+0000","last_peered":"2024-04-25T01:30:58.440012+0000","last_clean":"2024-04-25T01:30:58.440012+0000","last_became_active":"2024-04-25T01:30:33.343700+0000","last_became_peered":"2024-04-25T01:30:33.343700+0000","last_unstale":"2024-04-25T01:30:58.440012+0000","last_undegraded":"2024-04-25T01:30:58.440012+0000","last_fullsized":"2024-04-25T01:30:58.440012+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:54:08.265744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114656+0000","last_change":"2024-04-25T01:30:34.354644+0000","last_active":"2024-04-25T01:30:39.114656+0000","last_peered":"2024-04-25T01:30:39.114656+0000","last_clean":"2024-04-25T01:30:39.114656+0000","last_became_active":"2024-04-25T01:30:34.354330+0000","last_became_peered":"2024-04-25T01:30:34.354330+0000","last_unstale":"2024-04-25T01:30:39.114656+0000","last_undegraded":"2024-04-25T01:30:39.114656+0000","last_fullsized":"2024-04-25T01:30:39.114656+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:31:50.399983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.805165+0000","last_change":"2024-04-25T01:30:38.450743+0000","last_active":"2024-04-25T01:30:38.805165+0000","last_peered":"2024-04-25T01:30:38.805165+0000","last_clean":"2024-04-25T01:30:38.805165+0000","last_became_active":"2024-04-25T01:30:38.450303+0000","last_became_peered":"2024-04-25T01:30:38.450303+0000","last_unstale":"2024-04-25T01:30:38.805165+0000","last_undegraded":"2024-04-25T01:30:38.805165+0000","last_fullsized":"2024-04-25T01:30:38.805165+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:37.919250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.438441+0000","last_change":"2024-04-25T01:30:33.350709+0000","last_active":"2024-04-25T01:30:58.438441+0000","last_peered":"2024-04-25T01:30:58.438441+0000","last_clean":"2024-04-25T01:30:58.438441+0000","last_became_active":"2024-04-25T01:30:33.350387+0000","last_became_peered":"2024-04-25T01:30:33.350387+0000","last_unstale":"2024-04-25T01:30:58.438441+0000","last_undegraded":"2024-04-25T01:30:58.438441+0000","last_fullsized":"2024-04-25T01:30:58.438441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:30:01.252121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114002+0000","last_change":"2024-04-25T01:30:34.362040+0000","last_active":"2024-04-25T01:30:39.114002+0000","last_peered":"2024-04-25T01:30:39.114002+0000","last_clean":"2024-04-25T01:30:39.114002+0000","last_became_active":"2024-04-25T01:30:34.361734+0000","last_became_peered":"2024-04-25T01:30:34.361734+0000","last_unstale":"2024-04-25T01:30:39.114002+0000","last_undegraded":"2024-04-25T01:30:39.114002+0000","last_fullsized":"2024-04-25T01:30:39.114002+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:11:28.409764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.803675+0000","last_change":"2024-04-25T01:30:38.383202+0000","last_active":"2024-04-25T01:30:38.803675+0000","last_peered":"2024-04-25T01:30:38.803675+0000","last_clean":"2024-04-25T01:30:38.803675+0000","last_became_active":"2024-04-25T01:30:38.382071+0000","last_became_peered":"2024-04-25T01:30:38.382071+0000","last_unstale":"2024-04-25T01:30:38.803675+0000","last_undegraded":"2024-04-25T01:30:38.803675+0000","last_fullsized":"2024-04-25T01:30:38.803675+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:06:49.283136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.438358+0000","last_change":"2024-04-25T01:30:33.350207+0000","last_active":"2024-04-25T01:30:58.438358+0000","last_peered":"2024-04-25T01:30:58.438358+0000","last_clean":"2024-04-25T01:30:58.438358+0000","last_became_active":"2024-04-25T01:30:33.349892+0000","last_became_peered":"2024-04-25T01:30:33.349892+0000","last_unstale":"2024-04-25T01:30:58.438358+0000","last_undegraded":"2024-04-25T01:30:58.438358+0000","last_fullsized":"2024-04-25T01:30:58.438358+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:26:35.186486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.953587+0000","last_change":"2024-04-25T01:30:34.355196+0000","last_active":"2024-04-25T01:30:38.953587+0000","last_peered":"2024-04-25T01:30:38.953587+0000","last_clean":"2024-04-25T01:30:38.953587+0000","last_became_active":"2024-04-25T01:30:34.354794+0000","last_became_peered":"2024-04-25T01:30:34.354794+0000","last_unstale":"2024-04-25T01:30:38.953587+0000","last_undegraded":"2024-04-25T01:30:38.953587+0000","last_fullsized":"2024-04-25T01:30:38.953587+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:43:37.495974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804757+0000","last_change":"2024-04-25T01:30:38.384159+0000","last_active":"2024-04-25T01:30:38.804757+0000","last_peered":"2024-04-25T01:30:38.804757+0000","last_clean":"2024-04-25T01:30:38.804757+0000","last_became_active":"2024-04-25T01:30:38.383676+0000","last_became_peered":"2024-04-25T01:30:38.383676+0000","last_unstale":"2024-04-25T01:30:38.804757+0000","last_undegraded":"2024-04-25T01:30:38.804757+0000","last_fullsized":"2024-04-25T01:30:38.804757+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:28.491522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804837+0000","last_change":"2024-04-25T01:30:33.341379+0000","last_active":"2024-04-25T01:30:38.804837+0000","last_peered":"2024-04-25T01:30:38.804837+0000","last_clean":"2024-04-25T01:30:38.804837+0000","last_became_active":"2024-04-25T01:30:33.340999+0000","last_became_peered":"2024-04-25T01:30:33.340999+0000","last_unstale":"2024-04-25T01:30:38.804837+0000","last_undegraded":"2024-04-25T01:30:38.804837+0000","last_fullsized":"2024-04-25T01:30:38.804837+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:57:02.059834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114849+0000","last_change":"2024-04-25T01:30:34.358110+0000","last_active":"2024-04-25T01:30:39.114849+0000","last_peered":"2024-04-25T01:30:39.114849+0000","last_clean":"2024-04-25T01:30:39.114849+0000","last_became_active":"2024-04-25T01:30:34.357555+0000","last_became_peered":"2024-04-25T01:30:34.357555+0000","last_unstale":"2024-04-25T01:30:39.114849+0000","last_undegraded":"2024-04-25T01:30:39.114849+0000","last_fullsized":"2024-04-25T01:30:39.114849+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:11.636320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.805212+0000","last_change":"2024-04-25T01:30:38.450680+0000","last_active":"2024-04-25T01:30:38.805212+0000","last_peered":"2024-04-25T01:30:38.805212+0000","last_clean":"2024-04-25T01:30:38.805212+0000","last_became_active":"2024-04-25T01:30:38.450168+0000","last_became_peered":"2024-04-25T01:30:38.450168+0000","last_unstale":"2024-04-25T01:30:38.805212+0000","last_undegraded":"2024-04-25T01:30:38.805212+0000","last_fullsized":"2024-04-25T01:30:38.805212+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:39:24.236371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:58.437842+0000","last_change":"2024-04-25T01:30:33.343848+0000","last_active":"2024-04-25T01:30:58.437842+0000","last_peered":"2024-04-25T01:30:58.437842+0000","last_clean":"2024-04-25T01:30:58.437842+0000","last_became_active":"2024-04-25T01:30:33.343498+0000","last_became_peered":"2024-04-25T01:30:33.343498+0000","last_unstale":"2024-04-25T01:30:58.437842+0000","last_undegraded":"2024-04-25T01:30:58.437842+0000","last_fullsized":"2024-04-25T01:30:58.437842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:58:09.184571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114944+0000","last_change":"2024-04-25T01:30:34.361994+0000","last_active":"2024-04-25T01:30:39.114944+0000","last_peered":"2024-04-25T01:30:39.114944+0000","last_clean":"2024-04-25T01:30:39.114944+0000","last_became_active":"2024-04-25T01:30:34.361532+0000","last_became_peered":"2024-04-25T01:30:34.361532+0000","last_unstale":"2024-04-25T01:30:39.114944+0000","last_undegraded":"2024-04-25T01:30:39.114944+0000","last_fullsized":"2024-04-25T01:30:39.114944+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:30.667672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114812+0000","last_change":"2024-04-25T01:30:38.369041+0000","last_active":"2024-04-25T01:30:39.114812+0000","last_peered":"2024-04-25T01:30:39.114812+0000","last_clean":"2024-04-25T01:30:39.114812+0000","last_became_active":"2024-04-25T01:30:38.368586+0000","last_became_peered":"2024-04-25T01:30:38.368586+0000","last_unstale":"2024-04-25T01:30:39.114812+0000","last_undegraded":"2024-04-25T01:30:39.114812+0000","last_fullsized":"2024-04-25T01:30:39.114812+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:58:53.743012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.805258+0000","last_change":"2024-04-25T01:30:33.345501+0000","last_active":"2024-04-25T01:30:38.805258+0000","last_peered":"2024-04-25T01:30:38.805258+0000","last_clean":"2024-04-25T01:30:38.805258+0000","last_became_active":"2024-04-25T01:30:33.345179+0000","last_became_peered":"2024-04-25T01:30:33.345179+0000","last_unstale":"2024-04-25T01:30:38.805258+0000","last_undegraded":"2024-04-25T01:30:38.805258+0000","last_fullsized":"2024-04-25T01:30:38.805258+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:29:41.275480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115455+0000","last_change":"2024-04-25T01:30:34.361631+0000","last_active":"2024-04-25T01:30:39.115455+0000","last_peered":"2024-04-25T01:30:39.115455+0000","last_clean":"2024-04-25T01:30:39.115455+0000","last_became_active":"2024-04-25T01:30:34.360266+0000","last_became_peered":"2024-04-25T01:30:34.360266+0000","last_unstale":"2024-04-25T01:30:39.115455+0000","last_undegraded":"2024-04-25T01:30:39.115455+0000","last_fullsized":"2024-04-25T01:30:39.115455+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:03.094166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.113965+0000","last_change":"2024-04-25T01:30:38.380779+0000","last_active":"2024-04-25T01:30:39.113965+0000","last_peered":"2024-04-25T01:30:39.113965+0000","last_clean":"2024-04-25T01:30:39.113965+0000","last_became_active":"2024-04-25T01:30:38.379959+0000","last_became_peered":"2024-04-25T01:30:38.379959+0000","last_unstale":"2024-04-25T01:30:39.113965+0000","last_undegraded":"2024-04-25T01:30:39.113965+0000","last_fullsized":"2024-04-25T01:30:39.113965+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:48:01.816311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115462+0000","last_change":"2024-04-25T01:30:33.345730+0000","last_active":"2024-04-25T01:30:39.115462+0000","last_peered":"2024-04-25T01:30:39.115462+0000","last_clean":"2024-04-25T01:30:39.115462+0000","last_became_active":"2024-04-25T01:30:33.344832+0000","last_became_peered":"2024-04-25T01:30:33.344832+0000","last_unstale":"2024-04-25T01:30:39.115462+0000","last_undegraded":"2024-04-25T01:30:39.115462+0000","last_fullsized":"2024-04-25T01:30:39.115462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:47.018811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114584+0000","last_change":"2024-04-25T01:30:34.355324+0000","last_active":"2024-04-25T01:30:39.114584+0000","last_peered":"2024-04-25T01:30:39.114584+0000","last_clean":"2024-04-25T01:30:39.114584+0000","last_became_active":"2024-04-25T01:30:34.354954+0000","last_became_peered":"2024-04-25T01:30:34.354954+0000","last_unstale":"2024-04-25T01:30:39.114584+0000","last_undegraded":"2024-04-25T01:30:39.114584+0000","last_fullsized":"2024-04-25T01:30:39.114584+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:36:41.932391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114343+0000","last_change":"2024-04-25T01:30:38.374386+0000","last_active":"2024-04-25T01:30:39.114343+0000","last_peered":"2024-04-25T01:30:39.114343+0000","last_clean":"2024-04-25T01:30:39.114343+0000","last_became_active":"2024-04-25T01:30:38.373454+0000","last_became_peered":"2024-04-25T01:30:38.373454+0000","last_unstale":"2024-04-25T01:30:39.114343+0000","last_undegraded":"2024-04-25T01:30:39.114343+0000","last_fullsized":"2024-04-25T01:30:39.114343+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:54.456218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.455119+0000","last_change":"2024-04-25T01:30:33.339821+0000","last_active":"2024-04-25T01:30:39.455119+0000","last_peered":"2024-04-25T01:30:39.455119+0000","last_clean":"2024-04-25T01:30:39.455119+0000","last_became_active":"2024-04-25T01:30:33.339500+0000","last_became_peered":"2024-04-25T01:30:33.339500+0000","last_unstale":"2024-04-25T01:30:39.455119+0000","last_undegraded":"2024-04-25T01:30:39.455119+0000","last_fullsized":"2024-04-25T01:30:39.455119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:44:25.880828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114676+0000","last_change":"2024-04-25T01:30:34.362736+0000","last_active":"2024-04-25T01:30:39.114676+0000","last_peered":"2024-04-25T01:30:39.114676+0000","last_clean":"2024-04-25T01:30:39.114676+0000","last_became_active":"2024-04-25T01:30:34.362371+0000","last_became_peered":"2024-04-25T01:30:34.362371+0000","last_unstale":"2024-04-25T01:30:39.114676+0000","last_undegraded":"2024-04-25T01:30:39.114676+0000","last_fullsized":"2024-04-25T01:30:39.114676+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:57:36.088231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.953698+0000","last_change":"2024-04-25T01:30:34.359217+0000","last_active":"2024-04-25T01:30:38.953698+0000","last_peered":"2024-04-25T01:30:38.953698+0000","last_clean":"2024-04-25T01:30:38.953698+0000","last_became_active":"2024-04-25T01:30:34.357095+0000","last_became_peered":"2024-04-25T01:30:34.357095+0000","last_unstale":"2024-04-25T01:30:38.953698+0000","last_undegraded":"2024-04-25T01:30:38.953698+0000","last_fullsized":"2024-04-25T01:30:38.953698+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:58.603484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.805053+0000","last_change":"2024-04-25T01:30:34.359513+0000","last_active":"2024-04-25T01:30:38.805053+0000","last_peered":"2024-04-25T01:30:38.805053+0000","last_clean":"2024-04-25T01:30:38.805053+0000","last_became_active":"2024-04-25T01:30:34.359089+0000","last_became_peered":"2024-04-25T01:30:34.359089+0000","last_unstale":"2024-04-25T01:30:38.805053+0000","last_undegraded":"2024-04-25T01:30:38.805053+0000","last_fullsized":"2024-04-25T01:30:38.805053+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:36.159703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.804757+0000","last_change":"2024-04-25T01:30:34.368739+0000","last_active":"2024-04-25T01:30:38.804757+0000","last_peered":"2024-04-25T01:30:38.804757+0000","last_clean":"2024-04-25T01:30:38.804757+0000","last_became_active":"2024-04-25T01:30:34.368392+0000","last_became_peered":"2024-04-25T01:30:34.368392+0000","last_unstale":"2024-04-25T01:30:38.804757+0000","last_undegraded":"2024-04-25T01:30:38.804757+0000","last_fullsized":"2024-04-25T01:30:38.804757+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:39:32.241889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114249+0000","last_change":"2024-04-25T01:30:34.360136+0000","last_active":"2024-04-25T01:30:39.114249+0000","last_peered":"2024-04-25T01:30:39.114249+0000","last_clean":"2024-04-25T01:30:39.114249+0000","last_became_active":"2024-04-25T01:30:34.359744+0000","last_became_peered":"2024-04-25T01:30:34.359744+0000","last_unstale":"2024-04-25T01:30:39.114249+0000","last_undegraded":"2024-04-25T01:30:39.114249+0000","last_fullsized":"2024-04-25T01:30:39.114249+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:03:35.517440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.455094+0000","last_change":"2024-04-25T01:30:34.360979+0000","last_active":"2024-04-25T01:30:39.455094+0000","last_peered":"2024-04-25T01:30:39.455094+0000","last_clean":"2024-04-25T01:30:39.455094+0000","last_became_active":"2024-04-25T01:30:34.360630+0000","last_became_peered":"2024-04-25T01:30:34.360630+0000","last_unstale":"2024-04-25T01:30:39.455094+0000","last_undegraded":"2024-04-25T01:30:39.455094+0000","last_fullsized":"2024-04-25T01:30:39.455094+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:05:19.284577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.953993+0000","last_change":"2024-04-25T01:30:34.362621+0000","last_active":"2024-04-25T01:30:38.953993+0000","last_peered":"2024-04-25T01:30:38.953993+0000","last_clean":"2024-04-25T01:30:38.953993+0000","last_became_active":"2024-04-25T01:30:34.362290+0000","last_became_peered":"2024-04-25T01:30:34.362290+0000","last_unstale":"2024-04-25T01:30:38.953993+0000","last_undegraded":"2024-04-25T01:30:38.953993+0000","last_fullsized":"2024-04-25T01:30:38.953993+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:28:07.750906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.805533+0000","last_change":"2024-04-25T01:30:34.351339+0000","last_active":"2024-04-25T01:30:38.805533+0000","last_peered":"2024-04-25T01:30:38.805533+0000","last_clean":"2024-04-25T01:30:38.805533+0000","last_became_active":"2024-04-25T01:30:34.350655+0000","last_became_peered":"2024-04-25T01:30:34.350655+0000","last_unstale":"2024-04-25T01:30:38.805533+0000","last_undegraded":"2024-04-25T01:30:38.805533+0000","last_fullsized":"2024-04-25T01:30:38.805533+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:05.322547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.115268+0000","last_change":"2024-04-25T01:30:34.353876+0000","last_active":"2024-04-25T01:30:39.115268+0000","last_peered":"2024-04-25T01:30:39.115268+0000","last_clean":"2024-04-25T01:30:39.115268+0000","last_became_active":"2024-04-25T01:30:34.352942+0000","last_became_peered":"2024-04-25T01:30:34.352942+0000","last_unstale":"2024-04-25T01:30:39.115268+0000","last_undegraded":"2024-04-25T01:30:39.115268+0000","last_fullsized":"2024-04-25T01:30:39.115268+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:47.260171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114774+0000","last_change":"2024-04-25T01:30:34.352039+0000","last_active":"2024-04-25T01:30:39.114774+0000","last_peered":"2024-04-25T01:30:39.114774+0000","last_clean":"2024-04-25T01:30:39.114774+0000","last_became_active":"2024-04-25T01:30:34.351612+0000","last_became_peered":"2024-04-25T01:30:34.351612+0000","last_unstale":"2024-04-25T01:30:39.114774+0000","last_undegraded":"2024-04-25T01:30:39.114774+0000","last_fullsized":"2024-04-25T01:30:39.114774+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:03.607711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114518+0000","last_change":"2024-04-25T01:30:34.353692+0000","last_active":"2024-04-25T01:30:39.114518+0000","last_peered":"2024-04-25T01:30:39.114518+0000","last_clean":"2024-04-25T01:30:39.114518+0000","last_became_active":"2024-04-25T01:30:34.353035+0000","last_became_peered":"2024-04-25T01:30:34.353035+0000","last_unstale":"2024-04-25T01:30:39.114518+0000","last_undegraded":"2024-04-25T01:30:39.114518+0000","last_fullsized":"2024-04-25T01:30:39.114518+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:15:54.540315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.114965+0000","last_change":"2024-04-25T01:30:34.370797+0000","last_active":"2024-04-25T01:30:39.114965+0000","last_peered":"2024-04-25T01:30:39.114965+0000","last_clean":"2024-04-25T01:30:39.114965+0000","last_became_active":"2024-04-25T01:30:34.367533+0000","last_became_peered":"2024-04-25T01:30:34.367533+0000","last_unstale":"2024-04-25T01:30:39.114965+0000","last_undegraded":"2024-04-25T01:30:39.114965+0000","last_fullsized":"2024-04-25T01:30:39.114965+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:27:15.478808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:39.454766+0000","last_change":"2024-04-25T01:30:34.364843+0000","last_active":"2024-04-25T01:30:39.454766+0000","last_peered":"2024-04-25T01:30:39.454766+0000","last_clean":"2024-04-25T01:30:39.454766+0000","last_became_active":"2024-04-25T01:30:34.364515+0000","last_became_peered":"2024-04-25T01:30:34.364515+0000","last_unstale":"2024-04-25T01:30:39.454766+0000","last_undegraded":"2024-04-25T01:30:39.454766+0000","last_fullsized":"2024-04-25T01:30:39.454766+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:50:03.517814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-25T01:30:38.953848+0000","last_change":"2024-04-25T01:30:34.363715+0000","last_active":"2024-04-25T01:30:38.953848+0000","last_peered":"2024-04-25T01:30:38.953848+0000","last_clean":"2024-04-25T01:30:38.953848+0000","last_became_active":"2024-04-25T01:30:34.363383+0000","last_became_peered":"2024-04-25T01:30:34.363383+0000","last_unstale":"2024-04-25T01:30:38.953848+0000","last_undegraded":"2024-04-25T01:30:38.953848+0000","last_fullsized":"2024-04-25T01:30:38.953848+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:50:18.322550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6248,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":19968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318290,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":486531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579927,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36932,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708220,"statfs":{"total":95995035648,"available":95957217280,"internally_reserved":0,"allocated":1634304,"data_stored":1080505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743453,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1007616,"data_stored":485921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005090,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":486077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266727,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1622016,"data_stored":1078171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430253,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":485339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":489181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738443,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1007616,"data_stored":485429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":489215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1011712,"data_stored":485443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888118,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":485361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691890,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32752,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712400,"statfs":{"total":95995035648,"available":95961497600,"internally_reserved":0,"allocated":1609728,"data_stored":1075889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:31:04.257 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T01:31:04.257 INFO:tasks.cephadm:Setting up client nodes... 2024-04-25T01:31:04.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-25T01:31:04.484 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:31:04 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-b[63160]: 2024-04-25T01:31:04.125+0000 7f7824fdf700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T01:31:04.484 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:31:04 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-b[63160]: 2024-04-25T01:31:04.125+0000 7f7824fdf700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-25T01:31:04.484 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:31:04 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-b[63160]: 2024-04-25T01:31:04.125+0000 7f7824fdf700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T01:31:04.540 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:04.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:04 smithi028 ceph-mon[29390]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:31:05.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:04 smithi130 ceph-mon[32702]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:31:05.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:04 smithi032 ceph-mon[32461]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:31:06.052 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:05 smithi028 ceph-mon[29390]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:31:06.052 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:05 smithi028 ceph-mon[29390]: pgmap v313: 129 pgs: 129 active+clean; 627 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.5 KiB/s wr, 16 op/s 2024-04-25T01:31:06.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:05 smithi130 ceph-mon[32702]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:31:06.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:05 smithi130 ceph-mon[32702]: pgmap v313: 129 pgs: 129 active+clean; 627 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.5 KiB/s wr, 16 op/s 2024-04-25T01:31:06.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:05 smithi032 ceph-mon[32461]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:31:06.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:05 smithi032 ceph-mon[32461]: pgmap v313: 129 pgs: 129 active+clean; 627 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.5 KiB/s wr, 16 op/s 2024-04-25T01:31:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.478+0000 7fa315865700 1 -- 172.21.15.28:0/3326580563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa310108040 msgr2=0x7fa31010a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.478+0000 7fa315865700 1 --2- 172.21.15.28:0/3326580563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa310108040 0x7fa31010a430 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa30800b0a0 tx=0x7fa30800b3b0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.478+0000 7fa315865700 1 -- 172.21.15.28:0/3326580563 shutdown_connections 2024-04-25T01:31:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.478+0000 7fa315865700 1 --2- 172.21.15.28:0/3326580563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa310108040 0x7fa31010a430 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.478+0000 7fa315865700 1 --2- 172.21.15.28:0/3326580563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa310102de0 0x7fa3101051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.478+0000 7fa315865700 1 --2- 172.21.15.28:0/3326580563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa310105710 0x7fa310107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.478+0000 7fa315865700 1 -- 172.21.15.28:0/3326580563 >> 172.21.15.28:0/3326580563 conn(0x7fa3100fc750 msgr2=0x7fa3100feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.478+0000 7fa315865700 1 -- 172.21.15.28:0/3326580563 shutdown_connections 2024-04-25T01:31:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.478+0000 7fa315865700 1 -- 172.21.15.28:0/3326580563 wait complete. 2024-04-25T01:31:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.479+0000 7fa315865700 1 Processor -- start 2024-04-25T01:31:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.479+0000 7fa315865700 1 -- start start 2024-04-25T01:31:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.479+0000 7fa315865700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa310102de0 0x7fa310102740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.479+0000 7fa315865700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa310105710 0x7fa310100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.479+0000 7fa315865700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa310108040 0x7fa3101012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.479+0000 7fa315865700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa31010dec0 con 0x7fa310102de0 2024-04-25T01:31:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.479+0000 7fa315865700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa31010dd40 con 0x7fa310105710 2024-04-25T01:31:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.479+0000 7fa315865700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa31010e040 con 0x7fa310108040 2024-04-25T01:31:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.479+0000 7fa30effd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa310102de0 0x7fa310102740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.479+0000 7fa30effd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa310102de0 0x7fa310102740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52146/0 (socket says 172.21.15.28:52146) 2024-04-25T01:31:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.479+0000 7fa30effd700 1 -- 172.21.15.28:0/2342751631 learned_addr learned my addr 172.21.15.28:0/2342751631 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.479+0000 7fa30e7fc700 1 --2- 172.21.15.28:0/2342751631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa310105710 0x7fa310100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.479+0000 7fa30f7fe700 1 --2- 172.21.15.28:0/2342751631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa310108040 0x7fa3101012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.479+0000 7fa30effd700 1 -- 172.21.15.28:0/2342751631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa310108040 msgr2=0x7fa3101012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.479+0000 7fa30effd700 1 --2- 172.21.15.28:0/2342751631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa310108040 0x7fa3101012d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.479+0000 7fa30effd700 1 -- 172.21.15.28:0/2342751631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa310105710 msgr2=0x7fa310100d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.479+0000 7fa30effd700 1 --2- 172.21.15.28:0/2342751631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa310105710 0x7fa310100d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.479+0000 7fa30effd700 1 -- 172.21.15.28:0/2342751631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa310101b80 con 0x7fa310102de0 2024-04-25T01:31:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.480+0000 7fa30f7fe700 1 --2- 172.21.15.28:0/2342751631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa310108040 0x7fa3101012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.480+0000 7fa30effd700 1 --2- 172.21.15.28:0/2342751631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa310102de0 0x7fa310102740 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa2f800d950 tx=0x7fa2f800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.480+0000 7fa314863700 1 -- 172.21.15.28:0/2342751631 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8016070 con 0x7fa310102de0 2024-04-25T01:31:06.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.480+0000 7fa315865700 1 -- 172.21.15.28:0/2342751631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3101a24e0 con 0x7fa310102de0 2024-04-25T01:31:06.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.480+0000 7fa314863700 1 -- 172.21.15.28:0/2342751631 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2f8009e20 con 0x7fa310102de0 2024-04-25T01:31:06.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.480+0000 7fa314863700 1 -- 172.21.15.28:0/2342751631 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f800f620 con 0x7fa310102de0 2024-04-25T01:31:06.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.480+0000 7fa315865700 1 -- 172.21.15.28:0/2342751631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3101a29d0 con 0x7fa310102de0 2024-04-25T01:31:06.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.481+0000 7fa315865700 1 -- 172.21.15.28:0/2342751631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3100507a0 con 0x7fa310102de0 2024-04-25T01:31:06.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.482+0000 7fa314863700 1 -- 172.21.15.28:0/2342751631 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa2f8019070 con 0x7fa310102de0 2024-04-25T01:31:06.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.483+0000 7fa314863700 1 --2- 172.21.15.28:0/2342751631 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa2fc0a2d60 0x7fa2fc0a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:06.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.483+0000 7fa30e7fc700 1 --2- 172.21.15.28:0/2342751631 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa2fc0a2d60 0x7fa2fc0a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:06.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.483+0000 7fa314863700 1 -- 172.21.15.28:0/2342751631 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fa2f80cefe0 con 0x7fa310102de0 2024-04-25T01:31:06.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.484+0000 7fa30e7fc700 1 --2- 172.21.15.28:0/2342751631 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa2fc0a2d60 0x7fa2fc0a5220 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa300009c60 tx=0x7fa300009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:06.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.487+0000 7fa314863700 1 -- 172.21.15.28:0/2342751631 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa2f809bba0 con 0x7fa310102de0 2024-04-25T01:31:06.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.790+0000 7fa315865700 1 -- 172.21.15.28:0/2342751631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fa310062e40 con 0x7fa310102de0 2024-04-25T01:31:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.792+0000 7fa314863700 1 -- 172.21.15.28:0/2342751631 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fa2f809b4a0 con 0x7fa310102de0 2024-04-25T01:31:06.770 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-04-25T01:31:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBasilm1vw1LxAAkZMkn4A8S/S+A15HMVhzEw== 2024-04-25T01:31:06.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.794+0000 7fa315865700 1 -- 172.21.15.28:0/2342751631 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa2fc0a2d60 msgr2=0x7fa2fc0a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.794+0000 7fa315865700 1 --2- 172.21.15.28:0/2342751631 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa2fc0a2d60 0x7fa2fc0a5220 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa300009c60 tx=0x7fa300009260 comp rx=0 tx=0).stop 2024-04-25T01:31:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.794+0000 7fa315865700 1 -- 172.21.15.28:0/2342751631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa310102de0 msgr2=0x7fa310102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.794+0000 7fa315865700 1 --2- 172.21.15.28:0/2342751631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa310102de0 0x7fa310102740 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa2f800d950 tx=0x7fa2f800dc60 comp rx=0 tx=0).stop 2024-04-25T01:31:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.794+0000 7fa315865700 1 -- 172.21.15.28:0/2342751631 shutdown_connections 2024-04-25T01:31:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.794+0000 7fa315865700 1 --2- 172.21.15.28:0/2342751631 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa2fc0a2d60 0x7fa2fc0a5220 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.794+0000 7fa315865700 1 --2- 172.21.15.28:0/2342751631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa310108040 0x7fa3101012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.794+0000 7fa315865700 1 --2- 172.21.15.28:0/2342751631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa310102de0 0x7fa310102740 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.794+0000 7fa315865700 1 --2- 172.21.15.28:0/2342751631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa310105710 0x7fa310100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.794+0000 7fa315865700 1 -- 172.21.15.28:0/2342751631 >> 172.21.15.28:0/2342751631 conn(0x7fa3100fc750 msgr2=0x7fa310103720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.795+0000 7fa315865700 1 -- 172.21.15.28:0/2342751631 shutdown_connections 2024-04-25T01:31:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:06.795+0000 7fa315865700 1 -- 172.21.15.28:0/2342751631 wait complete. 2024-04-25T01:31:07.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:06 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2342751631' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T01:31:07.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:06 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2342751631' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T01:31:07.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:06 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2342751631' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T01:31:07.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:06 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2342751631' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T01:31:07.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:06 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2342751631' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T01:31:07.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:06 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2342751631' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T01:31:07.543 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:31:07.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-25T01:31:07.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T01:31:07.582 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T01:31:07.582 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-25T01:31:07.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-25T01:31:07.871 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:08.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:07 smithi028 ceph-mon[29390]: pgmap v314: 129 pgs: 129 active+clean; 628 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 8.2 KiB/s wr, 19 op/s 2024-04-25T01:31:08.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:07 smithi130 ceph-mon[32702]: pgmap v314: 129 pgs: 129 active+clean; 628 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 8.2 KiB/s wr, 19 op/s 2024-04-25T01:31:08.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:07 smithi032 ceph-mon[32461]: pgmap v314: 129 pgs: 129 active+clean; 628 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 8.2 KiB/s wr, 19 op/s 2024-04-25T01:31:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.724+0000 7effc4c95700 1 -- 172.21.15.28:0/1382748479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effc0104e00 msgr2=0x7effc0111310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.724+0000 7effc4c95700 1 --2- 172.21.15.28:0/1382748479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effc0104e00 0x7effc0111310 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7effb800b0a0 tx=0x7effb800b3b0 comp rx=0 tx=0).stop 2024-04-25T01:31:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.724+0000 7effc4c95700 1 -- 172.21.15.28:0/1382748479 shutdown_connections 2024-04-25T01:31:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.724+0000 7effc4c95700 1 --2- 172.21.15.28:0/1382748479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effc0104e00 0x7effc0111310 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.724+0000 7effc4c95700 1 --2- 172.21.15.28:0/1382748479 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7effc010a4d0 0x7effc010a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.724+0000 7effc4c95700 1 --2- 172.21.15.28:0/1382748479 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effc0104440 0x7effc01048c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.724+0000 7effc4c95700 1 -- 172.21.15.28:0/1382748479 >> 172.21.15.28:0/1382748479 conn(0x7effc00fff30 msgr2=0x7effc0102350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:09.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.724+0000 7effc4c95700 1 -- 172.21.15.28:0/1382748479 shutdown_connections 2024-04-25T01:31:09.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.724+0000 7effc4c95700 1 -- 172.21.15.28:0/1382748479 wait complete. 2024-04-25T01:31:09.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.725+0000 7effc4c95700 1 Processor -- start 2024-04-25T01:31:09.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.725+0000 7effc4c95700 1 -- start start 2024-04-25T01:31:09.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.726+0000 7effc4c95700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7effc0104440 0x7effc01a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:09.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.726+0000 7effc4c95700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effc0104e00 0x7effc01a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:09.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.726+0000 7effc4c95700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effc010a4d0 0x7effc019a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:09.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.726+0000 7effc4c95700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7effc01147b0 con 0x7effc0104e00 2024-04-25T01:31:09.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.726+0000 7effc4c95700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7effc0114630 con 0x7effc010a4d0 2024-04-25T01:31:09.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.726+0000 7effc4c95700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7effc0114930 con 0x7effc0104440 2024-04-25T01:31:09.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.726+0000 7effbdd9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effc0104e00 0x7effc01a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:09.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.726+0000 7effbdd9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effc0104e00 0x7effc01a0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58338/0 (socket says 172.21.15.28:58338) 2024-04-25T01:31:09.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.726+0000 7effbdd9b700 1 -- 172.21.15.28:0/3030126545 learned_addr learned my addr 172.21.15.28:0/3030126545 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:09.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.726+0000 7effbe59c700 1 --2- 172.21.15.28:0/3030126545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7effc0104440 0x7effc01a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:09.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.726+0000 7effbed9d700 1 --2- 172.21.15.28:0/3030126545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effc010a4d0 0x7effc019a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:09.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.726+0000 7effbdd9b700 1 -- 172.21.15.28:0/3030126545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7effc0104440 msgr2=0x7effc01a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:09.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.726+0000 7effbdd9b700 1 --2- 172.21.15.28:0/3030126545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7effc0104440 0x7effc01a0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:09.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.726+0000 7effbdd9b700 1 -- 172.21.15.28:0/3030126545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effc010a4d0 msgr2=0x7effc019a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:09.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.726+0000 7effbdd9b700 1 --2- 172.21.15.28:0/3030126545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effc010a4d0 0x7effc019a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:09.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.726+0000 7effbdd9b700 1 -- 172.21.15.28:0/3030126545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc019b150 con 0x7effc0104e00 2024-04-25T01:31:09.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.726+0000 7effbed9d700 1 --2- 172.21.15.28:0/3030126545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effc010a4d0 0x7effc019a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:09.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.726+0000 7effbdd9b700 1 --2- 172.21.15.28:0/3030126545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effc0104e00 0x7effc01a0d70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7effb400d950 tx=0x7effb400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:09.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.726+0000 7effaf7fe700 1 -- 172.21.15.28:0/3030126545 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb4016070 con 0x7effc0104e00 2024-04-25T01:31:09.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.727+0000 7effaf7fe700 1 -- 172.21.15.28:0/3030126545 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effb4009b40 con 0x7effc0104e00 2024-04-25T01:31:09.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.727+0000 7effc4c95700 1 -- 172.21.15.28:0/3030126545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effc019b440 con 0x7effc0104e00 2024-04-25T01:31:09.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.727+0000 7effaf7fe700 1 -- 172.21.15.28:0/3030126545 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb400f620 con 0x7effc0104e00 2024-04-25T01:31:09.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.727+0000 7effc4c95700 1 -- 172.21.15.28:0/3030126545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effc01a7700 con 0x7effc0104e00 2024-04-25T01:31:09.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.728+0000 7effc4c95700 1 -- 172.21.15.28:0/3030126545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effc0068f20 con 0x7effc0104e00 2024-04-25T01:31:09.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.729+0000 7effaf7fe700 1 -- 172.21.15.28:0/3030126545 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7effb4019070 con 0x7effc0104e00 2024-04-25T01:31:09.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.729+0000 7effaf7fe700 1 --2- 172.21.15.28:0/3030126545 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7effa80a2cb0 0x7effa80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:09.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.730+0000 7effbe59c700 1 --2- 172.21.15.28:0/3030126545 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7effa80a2cb0 0x7effa80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:09.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.730+0000 7effaf7fe700 1 -- 172.21.15.28:0/3030126545 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7effb40cf470 con 0x7effc0104e00 2024-04-25T01:31:09.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.730+0000 7effbe59c700 1 --2- 172.21.15.28:0/3030126545 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7effa80a2cb0 0x7effa80a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7effb0006fd0 tx=0x7effb0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:09.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:09.734+0000 7effaf7fe700 1 -- 172.21.15.28:0/3030126545 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7effb409c030 con 0x7effc0104e00 2024-04-25T01:31:09.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:10.004+0000 7effc4c95700 1 -- 172.21.15.28:0/3030126545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7effc01a79f0 con 0x7effc0104e00 2024-04-25T01:31:09.983 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:09 smithi028 ceph-mon[29390]: pgmap v315: 129 pgs: 129 active+clean; 628 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 8.3 KiB/s wr, 20 op/s 2024-04-25T01:31:09.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:10.009+0000 7effaf7fe700 1 -- 172.21.15.28:0/3030126545 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263200 (secure 0 0 0) 0x7effb409b930 con 0x7effc0104e00 2024-04-25T01:31:09.991 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:31:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:10.012+0000 7effc4c95700 1 -- 172.21.15.28:0/3030126545 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7effa80a2cb0 msgr2=0x7effa80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:10.012+0000 7effc4c95700 1 --2- 172.21.15.28:0/3030126545 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7effa80a2cb0 0x7effa80a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7effb0006fd0 tx=0x7effb0008040 comp rx=0 tx=0).stop 2024-04-25T01:31:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:10.012+0000 7effc4c95700 1 -- 172.21.15.28:0/3030126545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effc0104e00 msgr2=0x7effc01a0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:10.012+0000 7effc4c95700 1 --2- 172.21.15.28:0/3030126545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effc0104e00 0x7effc01a0d70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7effb400d950 tx=0x7effb400dd10 comp rx=0 tx=0).stop 2024-04-25T01:31:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:10.012+0000 7effc4c95700 1 -- 172.21.15.28:0/3030126545 shutdown_connections 2024-04-25T01:31:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:10.012+0000 7effc4c95700 1 --2- 172.21.15.28:0/3030126545 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7effa80a2cb0 0x7effa80a5170 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:09.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:10.012+0000 7effc4c95700 1 --2- 172.21.15.28:0/3030126545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effc010a4d0 0x7effc019a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:10.012+0000 7effc4c95700 1 --2- 172.21.15.28:0/3030126545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7effc0104440 0x7effc01a0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:10.012+0000 7effc4c95700 1 --2- 172.21.15.28:0/3030126545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effc0104e00 0x7effc01a0d70 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:10.012+0000 7effc4c95700 1 -- 172.21.15.28:0/3030126545 >> 172.21.15.28:0/3030126545 conn(0x7effc00fff30 msgr2=0x7effc0100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:10.012+0000 7effc4c95700 1 -- 172.21.15.28:0/3030126545 shutdown_connections 2024-04-25T01:31:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:10.012+0000 7effc4c95700 1 -- 172.21.15.28:0/3030126545 wait complete. 2024-04-25T01:31:10.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:09 smithi130 ceph-mon[32702]: pgmap v315: 129 pgs: 129 active+clean; 628 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 8.3 KiB/s wr, 20 op/s 2024-04-25T01:31:10.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:09 smithi032 ceph-mon[32461]: pgmap v315: 129 pgs: 129 active+clean; 628 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 8.3 KiB/s wr, 20 op/s 2024-04-25T01:31:10.762 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1149553567},{"type":"v1","addr":"172.21.15.28:6801","nonce":1149553567}]},"active_addr":"172.21.15.28:6801/1149553567","active_change":"2024-04-25T01:23:20.244907+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.28:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3630315328}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":500268183}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3499593842}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3812371257}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3042020065}]}]} 2024-04-25T01:31:10.767 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-25T01:31:10.767 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-25T01:31:10.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T01:31:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:11.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:10 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3030126545' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T01:31:11.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:10 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3030126545' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T01:31:11.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:10 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3030126545' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T01:31:12.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:11 smithi028 ceph-mon[29390]: pgmap v316: 129 pgs: 129 active+clean; 636 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.3 KiB/s wr, 24 op/s 2024-04-25T01:31:12.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:11 smithi130 ceph-mon[32702]: pgmap v316: 129 pgs: 129 active+clean; 636 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.3 KiB/s wr, 24 op/s 2024-04-25T01:31:12.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:11 smithi032 ceph-mon[32461]: pgmap v316: 129 pgs: 129 active+clean; 636 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.3 KiB/s wr, 24 op/s 2024-04-25T01:31:12.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.784+0000 7fd9bff8f700 1 -- 172.21.15.28:0/2638048529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b81015b0 msgr2=0x7fd9b810d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:12.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.784+0000 7fd9bff8f700 1 --2- 172.21.15.28:0/2638048529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b81015b0 0x7fd9b810d7c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd9b000b0a0 tx=0x7fd9b000b3b0 comp rx=0 tx=0).stop 2024-04-25T01:31:12.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.784+0000 7fd9bff8f700 1 -- 172.21.15.28:0/2638048529 shutdown_connections 2024-04-25T01:31:12.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.784+0000 7fd9bff8f700 1 --2- 172.21.15.28:0/2638048529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b81015b0 0x7fd9b810d7c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:12.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.784+0000 7fd9bff8f700 1 --2- 172.21.15.28:0/2638048529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9b81069a0 0x7fd9b8106d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:12.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.784+0000 7fd9bff8f700 1 --2- 172.21.15.28:0/2638048529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9b8100bf0 0x7fd9b8101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:12.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.784+0000 7fd9bff8f700 1 -- 172.21.15.28:0/2638048529 >> 172.21.15.28:0/2638048529 conn(0x7fd9b8076450 msgr2=0x7fd9b8076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:12.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.784+0000 7fd9bff8f700 1 -- 172.21.15.28:0/2638048529 shutdown_connections 2024-04-25T01:31:12.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.784+0000 7fd9bff8f700 1 -- 172.21.15.28:0/2638048529 wait complete. 2024-04-25T01:31:12.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.785+0000 7fd9bff8f700 1 Processor -- start 2024-04-25T01:31:12.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.785+0000 7fd9bff8f700 1 -- start start 2024-04-25T01:31:12.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.785+0000 7fd9bff8f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b8100bf0 0x7fd9b8196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:12.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.785+0000 7fd9bff8f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9b81015b0 0x7fd9b8196bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:12.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.785+0000 7fd9bff8f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9b81069a0 0x7fd9b819af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:12.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.785+0000 7fd9bff8f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b81103b0 con 0x7fd9b8100bf0 2024-04-25T01:31:12.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.785+0000 7fd9bff8f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b8110230 con 0x7fd9b81069a0 2024-04-25T01:31:12.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.785+0000 7fd9bff8f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b8110530 con 0x7fd9b81015b0 2024-04-25T01:31:12.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.785+0000 7fd9bdd2b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b8100bf0 0x7fd9b8196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:12.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.785+0000 7fd9bdd2b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b8100bf0 0x7fd9b8196680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58354/0 (socket says 172.21.15.28:58354) 2024-04-25T01:31:12.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.785+0000 7fd9bdd2b700 1 -- 172.21.15.28:0/831287978 learned_addr learned my addr 172.21.15.28:0/831287978 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:12.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.786+0000 7fd9be52c700 1 --2- 172.21.15.28:0/831287978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9b81069a0 0x7fd9b819af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:12.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.786+0000 7fd9bd52a700 1 --2- 172.21.15.28:0/831287978 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9b81015b0 0x7fd9b8196bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:12.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.786+0000 7fd9bdd2b700 1 -- 172.21.15.28:0/831287978 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9b81015b0 msgr2=0x7fd9b8196bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:12.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.786+0000 7fd9bdd2b700 1 --2- 172.21.15.28:0/831287978 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9b81015b0 0x7fd9b8196bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:12.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.786+0000 7fd9bdd2b700 1 -- 172.21.15.28:0/831287978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9b81069a0 msgr2=0x7fd9b819af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.786+0000 7fd9bdd2b700 1 --2- 172.21.15.28:0/831287978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9b81069a0 0x7fd9b819af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.786+0000 7fd9bdd2b700 1 -- 172.21.15.28:0/831287978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b819b690 con 0x7fd9b8100bf0 2024-04-25T01:31:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.786+0000 7fd9be52c700 1 --2- 172.21.15.28:0/831287978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9b81069a0 0x7fd9b819af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.786+0000 7fd9bdd2b700 1 --2- 172.21.15.28:0/831287978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b8100bf0 0x7fd9b8196680 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd9b400c9b0 tx=0x7fd9b400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.786+0000 7fd9aeffd700 1 -- 172.21.15.28:0/831287978 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4007b00 con 0x7fd9b8100bf0 2024-04-25T01:31:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.786+0000 7fd9bff8f700 1 -- 172.21.15.28:0/831287978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9b819b980 con 0x7fd9b8100bf0 2024-04-25T01:31:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.786+0000 7fd9aeffd700 1 -- 172.21.15.28:0/831287978 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9b4007ca0 con 0x7fd9b8100bf0 2024-04-25T01:31:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.786+0000 7fd9aeffd700 1 -- 172.21.15.28:0/831287978 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4018780 con 0x7fd9b8100bf0 2024-04-25T01:31:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.786+0000 7fd9bff8f700 1 -- 172.21.15.28:0/831287978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9b81a3200 con 0x7fd9b8100bf0 2024-04-25T01:31:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.787+0000 7fd9bff8f700 1 -- 172.21.15.28:0/831287978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9b8068f20 con 0x7fd9b8100bf0 2024-04-25T01:31:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.788+0000 7fd9aeffd700 1 -- 172.21.15.28:0/831287978 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd9b400f450 con 0x7fd9b8100bf0 2024-04-25T01:31:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.789+0000 7fd9aeffd700 1 --2- 172.21.15.28:0/831287978 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd9a40a2cb0 0x7fd9a40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.789+0000 7fd9bd52a700 1 --2- 172.21.15.28:0/831287978 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd9a40a2cb0 0x7fd9a40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.789+0000 7fd9aeffd700 1 -- 172.21.15.28:0/831287978 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fd9b40cf4b0 con 0x7fd9b8100bf0 2024-04-25T01:31:12.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.789+0000 7fd9bd52a700 1 --2- 172.21.15.28:0/831287978 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd9a40a2cb0 0x7fd9a40a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8076870 tx=0x7fd9a800a370 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:12.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:12.793+0000 7fd9aeffd700 1 -- 172.21.15.28:0/831287978 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd9b4014030 con 0x7fd9b8100bf0 2024-04-25T01:31:12.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:13.000+0000 7fd9bff8f700 1 -- 172.21.15.28:0/831287978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd9b81a34f0 con 0x7fd9b8100bf0 2024-04-25T01:31:12.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:13.002+0000 7fd9aeffd700 1 -- 172.21.15.28:0/831287978 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24093 (secure 0 0 0) 0x7fd9b409c070 con 0x7fd9b8100bf0 2024-04-25T01:31:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:31:12.980 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:30:38.730329+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:31:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:13.004+0000 7fd9bff8f700 1 -- 172.21.15.28:0/831287978 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd9a40a2cb0 msgr2=0x7fd9a40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:13.004+0000 7fd9bff8f700 1 --2- 172.21.15.28:0/831287978 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd9a40a2cb0 0x7fd9a40a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8076870 tx=0x7fd9a800a370 comp rx=0 tx=0).stop 2024-04-25T01:31:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:13.004+0000 7fd9bff8f700 1 -- 172.21.15.28:0/831287978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b8100bf0 msgr2=0x7fd9b8196680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:13.004+0000 7fd9bff8f700 1 --2- 172.21.15.28:0/831287978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b8100bf0 0x7fd9b8196680 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd9b400c9b0 tx=0x7fd9b400cd70 comp rx=0 tx=0).stop 2024-04-25T01:31:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:13.004+0000 7fd9bff8f700 1 -- 172.21.15.28:0/831287978 shutdown_connections 2024-04-25T01:31:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:13.004+0000 7fd9bff8f700 1 --2- 172.21.15.28:0/831287978 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd9a40a2cb0 0x7fd9a40a5170 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:13.004+0000 7fd9bff8f700 1 --2- 172.21.15.28:0/831287978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9b81069a0 0x7fd9b819af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:13.004+0000 7fd9bff8f700 1 --2- 172.21.15.28:0/831287978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9b8100bf0 0x7fd9b8196680 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:13.004+0000 7fd9bff8f700 1 --2- 172.21.15.28:0/831287978 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9b81015b0 0x7fd9b8196bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:13.004+0000 7fd9bff8f700 1 -- 172.21.15.28:0/831287978 >> 172.21.15.28:0/831287978 conn(0x7fd9b8076450 msgr2=0x7fd9b80fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:13.004+0000 7fd9bff8f700 1 -- 172.21.15.28:0/831287978 shutdown_connections 2024-04-25T01:31:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:13.004+0000 7fd9bff8f700 1 -- 172.21.15.28:0/831287978 wait complete. 2024-04-25T01:31:13.827 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-25T01:31:13.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T01:31:14.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:13 smithi028 ceph-mon[29390]: pgmap v317: 129 pgs: 129 active+clean; 636 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.3 KiB/s wr, 22 op/s 2024-04-25T01:31:14.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:13 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/831287978' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:31:14.081 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:14.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:13 smithi130 ceph-mon[32702]: pgmap v317: 129 pgs: 129 active+clean; 636 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.3 KiB/s wr, 22 op/s 2024-04-25T01:31:14.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:13 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/831287978' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:31:14.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:13 smithi032 ceph-mon[32461]: pgmap v317: 129 pgs: 129 active+clean; 636 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.3 KiB/s wr, 22 op/s 2024-04-25T01:31:14.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:13 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/831287978' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:31:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.856+0000 7f4b0a291700 1 -- 172.21.15.28:0/2998460259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04100610 msgr2=0x7f4b041009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.856+0000 7f4b0a291700 1 --2- 172.21.15.28:0/2998460259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04100610 0x7f4b041009f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4af40099c0 tx=0x7f4af4009cd0 comp rx=0 tx=0).stop 2024-04-25T01:31:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.856+0000 7f4b0a291700 1 -- 172.21.15.28:0/2998460259 shutdown_connections 2024-04-25T01:31:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.856+0000 7f4b0a291700 1 --2- 172.21.15.28:0/2998460259 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b041086f0 0x7f4b0410f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.856+0000 7f4b0a291700 1 --2- 172.21.15.28:0/2998460259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04100610 0x7f4b041009f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.856+0000 7f4b0a291700 1 --2- 172.21.15.28:0/2998460259 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b04100fc0 0x7f4b04108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.856+0000 7f4b0a291700 1 -- 172.21.15.28:0/2998460259 >> 172.21.15.28:0/2998460259 conn(0x7f4b04077330 msgr2=0x7f4b040fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.856+0000 7f4b0a291700 1 -- 172.21.15.28:0/2998460259 shutdown_connections 2024-04-25T01:31:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.856+0000 7f4b0a291700 1 -- 172.21.15.28:0/2998460259 wait complete. 2024-04-25T01:31:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.857+0000 7f4b0a291700 1 Processor -- start 2024-04-25T01:31:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.857+0000 7f4b0a291700 1 -- start start 2024-04-25T01:31:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.857+0000 7f4b0a291700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b04100610 0x7f4b0419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.857+0000 7f4b0a291700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04100fc0 0x7f4b0419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.857+0000 7f4b0a291700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b041086f0 0x7f4b04198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.857+0000 7f4b0a291700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b041125b0 con 0x7f4b04100fc0 2024-04-25T01:31:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.857+0000 7f4b0a291700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4b04112430 con 0x7f4b04100610 2024-04-25T01:31:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.857+0000 7f4b0a291700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4b04112730 con 0x7f4b041086f0 2024-04-25T01:31:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.857+0000 7f4b037fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04100fc0 0x7f4b0419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.857+0000 7f4b037fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04100fc0 0x7f4b0419eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58374/0 (socket says 172.21.15.28:58374) 2024-04-25T01:31:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.857+0000 7f4b037fe700 1 -- 172.21.15.28:0/3916756381 learned_addr learned my addr 172.21.15.28:0/3916756381 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.858+0000 7f4b03fff700 1 --2- 172.21.15.28:0/3916756381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b04100610 0x7f4b0419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.858+0000 7f4b0882e700 1 --2- 172.21.15.28:0/3916756381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b041086f0 0x7f4b04198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.858+0000 7f4b037fe700 1 -- 172.21.15.28:0/3916756381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b041086f0 msgr2=0x7f4b04198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.858+0000 7f4b037fe700 1 --2- 172.21.15.28:0/3916756381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b041086f0 0x7f4b04198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.858+0000 7f4b037fe700 1 -- 172.21.15.28:0/3916756381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b04100610 msgr2=0x7f4b0419e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.858+0000 7f4b037fe700 1 --2- 172.21.15.28:0/3916756381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b04100610 0x7f4b0419e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.858+0000 7f4b037fe700 1 -- 172.21.15.28:0/3916756381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b04198f70 con 0x7f4b04100fc0 2024-04-25T01:31:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.858+0000 7f4b037fe700 1 --2- 172.21.15.28:0/3916756381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04100fc0 0x7f4b0419eb50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4af800c9e0 tx=0x7f4af800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.858+0000 7f4b017fa700 1 -- 172.21.15.28:0/3916756381 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af8007aa0 con 0x7f4b04100fc0 2024-04-25T01:31:15.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.858+0000 7f4b0a291700 1 -- 172.21.15.28:0/3916756381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b04199260 con 0x7f4b04100fc0 2024-04-25T01:31:15.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.858+0000 7f4b017fa700 1 -- 172.21.15.28:0/3916756381 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4af8007c40 con 0x7f4b04100fc0 2024-04-25T01:31:15.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.858+0000 7f4b017fa700 1 -- 172.21.15.28:0/3916756381 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af8018640 con 0x7f4b04100fc0 2024-04-25T01:31:15.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.858+0000 7f4b0a291700 1 -- 172.21.15.28:0/3916756381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b041a5470 con 0x7f4b04100fc0 2024-04-25T01:31:15.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.859+0000 7f4b0a291700 1 -- 172.21.15.28:0/3916756381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b04101740 con 0x7f4b04100fc0 2024-04-25T01:31:15.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.861+0000 7f4b017fa700 1 -- 172.21.15.28:0/3916756381 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f4af800f450 con 0x7f4b04100fc0 2024-04-25T01:31:15.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.861+0000 7f4b017fa700 1 --2- 172.21.15.28:0/3916756381 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4aec0a2cb0 0x7f4aec0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:15.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.861+0000 7f4b03fff700 1 --2- 172.21.15.28:0/3916756381 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4aec0a2cb0 0x7f4aec0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:15.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.861+0000 7f4b017fa700 1 -- 172.21.15.28:0/3916756381 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f4af80cf390 con 0x7f4b04100fc0 2024-04-25T01:31:15.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.862+0000 7f4b03fff700 1 --2- 172.21.15.28:0/3916756381 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4aec0a2cb0 0x7f4aec0a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009fd0 tx=0x7f4af4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:15.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:15.866+0000 7f4b017fa700 1 -- 172.21.15.28:0/3916756381 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4af8014030 con 0x7f4b04100fc0 2024-04-25T01:31:16.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:16.074+0000 7f4b0a291700 1 -- 172.21.15.28:0/3916756381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4b04199cc0 con 0x7f4b04100fc0 2024-04-25T01:31:16.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:16.076+0000 7f4b017fa700 1 -- 172.21.15.28:0/3916756381 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24093 (secure 0 0 0) 0x7f4af809bf50 con 0x7f4b04100fc0 2024-04-25T01:31:16.054 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:31:16.054 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:30:38.730329+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:31:16.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:16.078+0000 7f4b0a291700 1 -- 172.21.15.28:0/3916756381 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4aec0a2cb0 msgr2=0x7f4aec0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:16.078+0000 7f4b0a291700 1 --2- 172.21.15.28:0/3916756381 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4aec0a2cb0 0x7f4aec0a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009fd0 tx=0x7f4af4005fb0 comp rx=0 tx=0).stop 2024-04-25T01:31:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:16.078+0000 7f4b0a291700 1 -- 172.21.15.28:0/3916756381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04100fc0 msgr2=0x7f4b0419eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:16.078+0000 7f4b0a291700 1 --2- 172.21.15.28:0/3916756381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04100fc0 0x7f4b0419eb50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4af800c9e0 tx=0x7f4af800ccf0 comp rx=0 tx=0).stop 2024-04-25T01:31:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:16.078+0000 7f4b0a291700 1 -- 172.21.15.28:0/3916756381 shutdown_connections 2024-04-25T01:31:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:16.078+0000 7f4b0a291700 1 --2- 172.21.15.28:0/3916756381 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4aec0a2cb0 0x7f4aec0a5170 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:16.078+0000 7f4b0a291700 1 --2- 172.21.15.28:0/3916756381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4b041086f0 0x7f4b04198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:16.078+0000 7f4b0a291700 1 --2- 172.21.15.28:0/3916756381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4b04100610 0x7f4b0419e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:16.078+0000 7f4b0a291700 1 --2- 172.21.15.28:0/3916756381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04100fc0 0x7f4b0419eb50 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:16.078+0000 7f4b0a291700 1 -- 172.21.15.28:0/3916756381 >> 172.21.15.28:0/3916756381 conn(0x7f4b04077330 msgr2=0x7f4b040ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:16.078+0000 7f4b0a291700 1 -- 172.21.15.28:0/3916756381 shutdown_connections 2024-04-25T01:31:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:16.078+0000 7f4b0a291700 1 -- 172.21.15.28:0/3916756381 wait complete. 2024-04-25T01:31:16.067 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:15 smithi028 ceph-mon[29390]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.0 KiB/s wr, 23 op/s 2024-04-25T01:31:16.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:15 smithi130 ceph-mon[32702]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.0 KiB/s wr, 23 op/s 2024-04-25T01:31:16.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:15 smithi032 ceph-mon[32461]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.0 KiB/s wr, 23 op/s 2024-04-25T01:31:16.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-25T01:31:16.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-25T01:31:16.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-25T01:31:16.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-25T01:31:16.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-25T01:31:16.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-25T01:31:16.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-25T01:31:16.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-25T01:31:16.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-25T01:31:16.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-25T01:31:16.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-25T01:31:16.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-25T01:31:17.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:16 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3916756381' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:31:17.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:16 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3916756381' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:31:17.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:16 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3916756381' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:31:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:18.009 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:17 smithi028 ceph-mon[29390]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-04-25T01:31:18.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:17 smithi130 ceph-mon[32702]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-04-25T01:31:18.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:17 smithi032 ceph-mon[32461]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-04-25T01:31:19.066 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:19.067 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:19.798 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:20.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:19 smithi028 ceph-mon[29390]: pgmap v320: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-04-25T01:31:20.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:19 smithi130 ceph-mon[32702]: pgmap v320: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-04-25T01:31:20.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:19 smithi032 ceph-mon[32461]: pgmap v320: 129 pgs: 129 active+clean; 641 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-04-25T01:31:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:20.929 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:20.930 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:20.930 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:20.932 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:20.932 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:21.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-25T01:31:21.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-25T01:31:21.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-25T01:31:21.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-04-25T01:31:21.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-04-25T01:31:21.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-04-25T01:31:21.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [5, 8]}]: dispatch 2024-04-25T01:31:21.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-25T01:31:21.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-04-25T01:31:21.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:20 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 10]}]: dispatch 2024-04-25T01:31:21.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-25T01:31:21.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-25T01:31:21.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-25T01:31:21.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-04-25T01:31:21.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-04-25T01:31:21.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-04-25T01:31:21.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [5, 8]}]: dispatch 2024-04-25T01:31:21.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-25T01:31:21.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-04-25T01:31:21.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:20 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 10]}]: dispatch 2024-04-25T01:31:21.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:20 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-25T01:31:21.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:20 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-25T01:31:21.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:20 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-25T01:31:21.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:20 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-04-25T01:31:21.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:20 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]: dispatch 2024-04-25T01:31:21.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:20 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-04-25T01:31:21.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:20 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [5, 8]}]: dispatch 2024-04-25T01:31:21.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:20 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-25T01:31:21.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:20 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-04-25T01:31:21.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:20 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 10]}]: dispatch 2024-04-25T01:31:22.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:21 smithi028 ceph-mon[29390]: pgmap v321: 129 pgs: 129 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-04-25T01:31:22.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:21 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-25T01:31:22.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:21 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-25T01:31:22.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:21 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-25T01:31:22.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:21 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-04-25T01:31:22.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:21 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-04-25T01:31:22.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:21 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-04-25T01:31:22.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:21 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [5, 8]}]': finished 2024-04-25T01:31:22.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:21 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-25T01:31:22.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:21 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-04-25T01:31:22.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:21 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 10]}]': finished 2024-04-25T01:31:22.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:21 smithi028 ceph-mon[29390]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T01:31:22.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:21 smithi130 ceph-mon[32702]: pgmap v321: 129 pgs: 129 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-04-25T01:31:22.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:21 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-25T01:31:22.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:21 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-25T01:31:22.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:21 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-25T01:31:22.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:21 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-04-25T01:31:22.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:21 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-04-25T01:31:22.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:21 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-04-25T01:31:22.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:21 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [5, 8]}]': finished 2024-04-25T01:31:22.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:21 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-25T01:31:22.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:21 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-04-25T01:31:22.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:21 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 10]}]': finished 2024-04-25T01:31:22.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:21 smithi130 ceph-mon[32702]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T01:31:22.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:21 smithi032 ceph-mon[32461]: pgmap v321: 129 pgs: 129 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-04-25T01:31:22.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:21 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-25T01:31:22.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:21 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-25T01:31:22.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:21 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-25T01:31:22.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:21 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-04-25T01:31:22.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:21 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [5, 9]}]': finished 2024-04-25T01:31:22.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:21 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-04-25T01:31:22.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:21 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [5, 8]}]': finished 2024-04-25T01:31:22.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:21 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-25T01:31:22.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:21 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-04-25T01:31:22.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:21 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 10]}]': finished 2024-04-25T01:31:22.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:21 smithi032 ceph-mon[32461]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T01:31:23.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:22 smithi028 ceph-mon[29390]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T01:31:23.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:22 smithi130 ceph-mon[32702]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T01:31:23.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:22 smithi032 ceph-mon[32461]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T01:31:23.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.806+0000 7f493b8be700 1 -- 172.21.15.28:0/3321351270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49341064b0 msgr2=0x7f493410e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:23.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.806+0000 7f493b8be700 1 --2- 172.21.15.28:0/3321351270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49341064b0 0x7f493410e0c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f492c00b0d0 tx=0x7f492c00b3e0 comp rx=0 tx=0).stop 2024-04-25T01:31:23.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.806+0000 7f493b8be700 1 -- 172.21.15.28:0/3321351270 shutdown_connections 2024-04-25T01:31:23.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.806+0000 7f493b8be700 1 --2- 172.21.15.28:0/3321351270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49341064b0 0x7f493410e0c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:23.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.806+0000 7f493b8be700 1 --2- 172.21.15.28:0/3321351270 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4934101b90 0x7f4934101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:23.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.806+0000 7f493b8be700 1 --2- 172.21.15.28:0/3321351270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4934102540 0x7f4934105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:23.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.806+0000 7f493b8be700 1 -- 172.21.15.28:0/3321351270 >> 172.21.15.28:0/3321351270 conn(0x7f49340777a0 msgr2=0x7f49340feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:23.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.806+0000 7f493b8be700 1 -- 172.21.15.28:0/3321351270 shutdown_connections 2024-04-25T01:31:23.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.806+0000 7f493b8be700 1 -- 172.21.15.28:0/3321351270 wait complete. 2024-04-25T01:31:23.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.807+0000 7f493b8be700 1 Processor -- start 2024-04-25T01:31:23.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.807+0000 7f493b8be700 1 -- start start 2024-04-25T01:31:23.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.807+0000 7f493b8be700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4934101b90 0x7f493419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:23.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.807+0000 7f493b8be700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4934102540 0x7f493419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:23.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.807+0000 7f493b8be700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49341064b0 0x7f4934198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:23.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.807+0000 7f493b8be700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f49341114f0 con 0x7f4934101b90 2024-04-25T01:31:23.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.807+0000 7f493b8be700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4934111370 con 0x7f4934102540 2024-04-25T01:31:23.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.807+0000 7f493b8be700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4934111670 con 0x7f49341064b0 2024-04-25T01:31:23.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.807+0000 7f493965a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4934101b90 0x7f493419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:23.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.807+0000 7f493965a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4934101b90 0x7f493419e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52756/0 (socket says 172.21.15.28:52756) 2024-04-25T01:31:23.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.807+0000 7f493965a700 1 -- 172.21.15.28:0/337483422 learned_addr learned my addr 172.21.15.28:0/337483422 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.807+0000 7f4939e5b700 1 --2- 172.21.15.28:0/337483422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49341064b0 0x7f4934198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.807+0000 7f493965a700 1 -- 172.21.15.28:0/337483422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49341064b0 msgr2=0x7f4934198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.807+0000 7f4938e59700 1 --2- 172.21.15.28:0/337483422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4934102540 0x7f493419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.807+0000 7f493965a700 1 --2- 172.21.15.28:0/337483422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f49341064b0 0x7f4934198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.807+0000 7f493965a700 1 -- 172.21.15.28:0/337483422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4934102540 msgr2=0x7f493419eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.807+0000 7f493965a700 1 --2- 172.21.15.28:0/337483422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4934102540 0x7f493419eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.807+0000 7f493965a700 1 -- 172.21.15.28:0/337483422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4934199000 con 0x7f4934101b90 2024-04-25T01:31:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.807+0000 7f493965a700 1 --2- 172.21.15.28:0/337483422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4934101b90 0x7f493419e650 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f493000d980 tx=0x7f493000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.808+0000 7f492a7fc700 1 -- 172.21.15.28:0/337483422 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4930017070 con 0x7f4934101b90 2024-04-25T01:31:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.808+0000 7f493b8be700 1 -- 172.21.15.28:0/337483422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49341992f0 con 0x7f4934101b90 2024-04-25T01:31:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.808+0000 7f492a7fc700 1 -- 172.21.15.28:0/337483422 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4930005020 con 0x7f4934101b90 2024-04-25T01:31:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.808+0000 7f492a7fc700 1 -- 172.21.15.28:0/337483422 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4930009bc0 con 0x7f4934101b90 2024-04-25T01:31:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.808+0000 7f493b8be700 1 -- 172.21.15.28:0/337483422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49341a54b0 con 0x7f4934101b90 2024-04-25T01:31:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.808+0000 7f493b8be700 1 -- 172.21.15.28:0/337483422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4934062940 con 0x7f4934101b90 2024-04-25T01:31:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.810+0000 7f492a7fc700 1 -- 172.21.15.28:0/337483422 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f49300051d0 con 0x7f4934101b90 2024-04-25T01:31:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.810+0000 7f492a7fc700 1 --2- 172.21.15.28:0/337483422 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f49200a2cb0 0x7f49200a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.811+0000 7f4938e59700 1 --2- 172.21.15.28:0/337483422 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f49200a2cb0 0x7f49200a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.811+0000 7f492a7fc700 1 -- 172.21.15.28:0/337483422 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f49300cf9f0 con 0x7f4934101b90 2024-04-25T01:31:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.811+0000 7f492a7fc700 1 --2- 172.21.15.28:0/337483422 >> [v2:172.21.15.130:6800/3497329111,v1:172.21.15.130:6801/3497329111] conn(0x7f49200ade10 0x7f49200b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.811+0000 7f492a7fc700 1 -- 172.21.15.28:0/337483422 --> [v2:172.21.15.130:6800/3497329111,v1:172.21.15.130:6801/3497329111] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f49300044e0 con 0x7f49200ade10 2024-04-25T01:31:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.811+0000 7f492a7fc700 1 -- 172.21.15.28:0/337483422 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f49300cfe40 con 0x7f4934101b90 2024-04-25T01:31:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.811+0000 7f4938e59700 1 --2- 172.21.15.28:0/337483422 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f49200a2cb0 0x7f49200a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4924005fd0 tx=0x7f4924005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.811+0000 7f4939e5b700 1 --2- 172.21.15.28:0/337483422 >> [v2:172.21.15.130:6800/3497329111,v1:172.21.15.130:6801/3497329111] conn(0x7f49200ade10 0x7f49200b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.812+0000 7f4939e5b700 1 --2- 172.21.15.28:0/337483422 >> [v2:172.21.15.130:6800/3497329111,v1:172.21.15.130:6801/3497329111] conn(0x7f49200ade10 0x7f49200b02b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.813+0000 7f492a7fc700 1 -- 172.21.15.28:0/337483422 <== osd.2 v2:172.21.15.130:6800/3497329111 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f49300044e0 con 0x7f49200ade10 2024-04-25T01:31:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.829+0000 7f493b8be700 1 -- 172.21.15.28:0/337483422 --> [v2:172.21.15.130:6800/3497329111,v1:172.21.15.130:6801/3497329111] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4934062dc0 con 0x7f49200ade10 2024-04-25T01:31:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.830+0000 7f492a7fc700 1 -- 172.21.15.28:0/337483422 <== osd.2 v2:172.21.15.130:6800/3497329111 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4934062dc0 con 0x7f49200ade10 2024-04-25T01:31:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.830+0000 7f493b8be700 1 -- 172.21.15.28:0/337483422 >> [v2:172.21.15.130:6800/3497329111,v1:172.21.15.130:6801/3497329111] conn(0x7f49200ade10 msgr2=0x7f49200b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.830+0000 7f493b8be700 1 --2- 172.21.15.28:0/337483422 >> [v2:172.21.15.130:6800/3497329111,v1:172.21.15.130:6801/3497329111] conn(0x7f49200ade10 0x7f49200b02b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.831+0000 7f493b8be700 1 -- 172.21.15.28:0/337483422 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f49200a2cb0 msgr2=0x7f49200a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.831+0000 7f493b8be700 1 --2- 172.21.15.28:0/337483422 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f49200a2cb0 0x7f49200a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4924005fd0 tx=0x7f4924005ee0 comp rx=0 tx=0).stop 2024-04-25T01:31:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.831+0000 7f493b8be700 1 -- 172.21.15.28:0/337483422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4934101b90 msgr2=0x7f493419e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.831+0000 7f493b8be700 1 --2- 172.21.15.28:0/337483422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4934101b90 0x7f493419e650 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f493000d980 tx=0x7f493000dc90 comp rx=0 tx=0).stop 2024-04-25T01:31:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.831+0000 7f4939e5b700 1 -- 172.21.15.28:0/337483422 reap_dead start 2024-04-25T01:31:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.831+0000 7f493b8be700 1 -- 172.21.15.28:0/337483422 shutdown_connections 2024-04-25T01:31:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.831+0000 7f493b8be700 1 -- 172.21.15.28:0/337483422 >> 172.21.15.28:0/337483422 conn(0x7f49340777a0 msgr2=0x7f4934100a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.831+0000 7f493b8be700 1 -- 172.21.15.28:0/337483422 shutdown_connections 2024-04-25T01:31:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:23.831+0000 7f493b8be700 1 -- 172.21.15.28:0/337483422 wait complete. 2024-04-25T01:31:23.982 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:23 smithi028 ceph-mon[29390]: pgmap v324: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-25T01:31:24.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:23 smithi130 ceph-mon[32702]: pgmap v324: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-25T01:31:24.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:23 smithi032 ceph-mon[32461]: pgmap v324: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-25T01:31:24.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.609+0000 7fb7fdc86700 1 -- 172.21.15.28:0/383764923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f81024e0 msgr2=0x7fb7f81028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:24.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.609+0000 7fb7fdc86700 1 --2- 172.21.15.28:0/383764923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f81024e0 0x7fb7f81028c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec009990 tx=0x7fb7ec009ca0 comp rx=0 tx=0).stop 2024-04-25T01:31:24.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.609+0000 7fb7fdc86700 1 -- 172.21.15.28:0/383764923 shutdown_connections 2024-04-25T01:31:24.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.609+0000 7fb7fdc86700 1 --2- 172.21.15.28:0/383764923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7f8107570 0x7fb7f810f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:24.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.609+0000 7fb7fdc86700 1 --2- 172.21.15.28:0/383764923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f81024e0 0x7fb7f81028c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:24.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.609+0000 7fb7fdc86700 1 --2- 172.21.15.28:0/383764923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7f8102e90 0x7fb7f8106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:24.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.609+0000 7fb7fdc86700 1 -- 172.21.15.28:0/383764923 >> 172.21.15.28:0/383764923 conn(0x7fb7f8079390 msgr2=0x7fb7f80797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:24.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.609+0000 7fb7fdc86700 1 -- 172.21.15.28:0/383764923 shutdown_connections 2024-04-25T01:31:24.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.609+0000 7fb7fdc86700 1 -- 172.21.15.28:0/383764923 wait complete. 2024-04-25T01:31:24.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.610+0000 7fb7fdc86700 1 Processor -- start 2024-04-25T01:31:24.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.610+0000 7fb7fdc86700 1 -- start start 2024-04-25T01:31:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.610+0000 7fb7fdc86700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7f81024e0 0x7fb7f819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.610+0000 7fb7fdc86700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7f8102e90 0x7fb7f819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.610+0000 7fb7fdc86700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f8107570 0x7fb7f8198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.610+0000 7fb7fdc86700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f8112530 con 0x7fb7f8107570 2024-04-25T01:31:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.610+0000 7fb7fdc86700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f81123b0 con 0x7fb7f81024e0 2024-04-25T01:31:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.610+0000 7fb7fdc86700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f81126b0 con 0x7fb7f8102e90 2024-04-25T01:31:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.610+0000 7fb7f7fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f8107570 0x7fb7f8198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.610+0000 7fb7f7fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f8107570 0x7fb7f8198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52776/0 (socket says 172.21.15.28:52776) 2024-04-25T01:31:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.610+0000 7fb7f7fff700 1 -- 172.21.15.28:0/1013767338 learned_addr learned my addr 172.21.15.28:0/1013767338 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.610+0000 7fb7f77fe700 1 --2- 172.21.15.28:0/1013767338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7f81024e0 0x7fb7f819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:24.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.610+0000 7fb7f6ffd700 1 --2- 172.21.15.28:0/1013767338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7f8102e90 0x7fb7f819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:24.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.611+0000 7fb7f7fff700 1 -- 172.21.15.28:0/1013767338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7f8102e90 msgr2=0x7fb7f819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:24.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.611+0000 7fb7f7fff700 1 --2- 172.21.15.28:0/1013767338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7f8102e90 0x7fb7f819eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:24.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.611+0000 7fb7f7fff700 1 -- 172.21.15.28:0/1013767338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7f81024e0 msgr2=0x7fb7f819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:24.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.611+0000 7fb7f7fff700 1 --2- 172.21.15.28:0/1013767338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7f81024e0 0x7fb7f819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:24.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.611+0000 7fb7f7fff700 1 -- 172.21.15.28:0/1013767338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f8199050 con 0x7fb7f8107570 2024-04-25T01:31:24.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.611+0000 7fb7f7fff700 1 --2- 172.21.15.28:0/1013767338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f8107570 0x7fb7f8198770 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0007d70 tx=0x7fb7f000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.611+0000 7fb7f4ff9700 1 -- 172.21.15.28:0/1013767338 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f0016020 con 0x7fb7f8107570 2024-04-25T01:31:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.611+0000 7fb7fdc86700 1 -- 172.21.15.28:0/1013767338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7f8199340 con 0x7fb7f8107570 2024-04-25T01:31:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.611+0000 7fb7f4ff9700 1 -- 172.21.15.28:0/1013767338 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7f000ac90 con 0x7fb7f8107570 2024-04-25T01:31:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.611+0000 7fb7f4ff9700 1 -- 172.21.15.28:0/1013767338 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f00115c0 con 0x7fb7f8107570 2024-04-25T01:31:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.611+0000 7fb7fdc86700 1 -- 172.21.15.28:0/1013767338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7f81a55d0 con 0x7fb7f8107570 2024-04-25T01:31:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.612+0000 7fb7fdc86700 1 -- 172.21.15.28:0/1013767338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb7f8062980 con 0x7fb7f8107570 2024-04-25T01:31:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.613+0000 7fb7f4ff9700 1 -- 172.21.15.28:0/1013767338 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb7f0019020 con 0x7fb7f8107570 2024-04-25T01:31:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.614+0000 7fb7f4ff9700 1 --2- 172.21.15.28:0/1013767338 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb7e00a2cb0 0x7fb7e00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.614+0000 7fb7f77fe700 1 --2- 172.21.15.28:0/1013767338 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb7e00a2cb0 0x7fb7e00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.614+0000 7fb7f4ff9700 1 -- 172.21.15.28:0/1013767338 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fb7f00cf920 con 0x7fb7f8107570 2024-04-25T01:31:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.614+0000 7fb7f4ff9700 1 --2- 172.21.15.28:0/1013767338 >> [v2:172.21.15.130:6824/3995209547,v1:172.21.15.130:6825/3995209547] conn(0x7fb7e00ade10 0x7fb7e00b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.614+0000 7fb7f4ff9700 1 -- 172.21.15.28:0/1013767338 --> [v2:172.21.15.130:6824/3995209547,v1:172.21.15.130:6825/3995209547] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7f0022e70 con 0x7fb7e00ade10 2024-04-25T01:31:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.614+0000 7fb7f4ff9700 1 -- 172.21.15.28:0/1013767338 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fb7f00cfcd0 con 0x7fb7f8107570 2024-04-25T01:31:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.614+0000 7fb7f77fe700 1 --2- 172.21.15.28:0/1013767338 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb7e00a2cb0 0x7fb7e00a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec00b5c0 tx=0x7fb7ec005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.614+0000 7fb7f6ffd700 1 --2- 172.21.15.28:0/1013767338 >> [v2:172.21.15.130:6824/3995209547,v1:172.21.15.130:6825/3995209547] conn(0x7fb7e00ade10 0x7fb7e00b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.615+0000 7fb7f6ffd700 1 --2- 172.21.15.28:0/1013767338 >> [v2:172.21.15.130:6824/3995209547,v1:172.21.15.130:6825/3995209547] conn(0x7fb7e00ade10 0x7fb7e00b02b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.616+0000 7fb7f4ff9700 1 -- 172.21.15.28:0/1013767338 <== osd.11 v2:172.21.15.130:6824/3995209547 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb7f0022e70 con 0x7fb7e00ade10 2024-04-25T01:31:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.632+0000 7fb7fdc86700 1 -- 172.21.15.28:0/1013767338 --> [v2:172.21.15.130:6824/3995209547,v1:172.21.15.130:6825/3995209547] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb7f8062e00 con 0x7fb7e00ade10 2024-04-25T01:31:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.632+0000 7fb7f4ff9700 1 -- 172.21.15.28:0/1013767338 <== osd.11 v2:172.21.15.130:6824/3995209547 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb7f8062e00 con 0x7fb7e00ade10 2024-04-25T01:31:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.633+0000 7fb7fdc86700 1 -- 172.21.15.28:0/1013767338 >> [v2:172.21.15.130:6824/3995209547,v1:172.21.15.130:6825/3995209547] conn(0x7fb7e00ade10 msgr2=0x7fb7e00b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.633+0000 7fb7fdc86700 1 --2- 172.21.15.28:0/1013767338 >> [v2:172.21.15.130:6824/3995209547,v1:172.21.15.130:6825/3995209547] conn(0x7fb7e00ade10 0x7fb7e00b02b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.633+0000 7fb7fdc86700 1 -- 172.21.15.28:0/1013767338 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb7e00a2cb0 msgr2=0x7fb7e00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.633+0000 7fb7fdc86700 1 --2- 172.21.15.28:0/1013767338 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb7e00a2cb0 0x7fb7e00a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec00b5c0 tx=0x7fb7ec005960 comp rx=0 tx=0).stop 2024-04-25T01:31:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.633+0000 7fb7fdc86700 1 -- 172.21.15.28:0/1013767338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f8107570 msgr2=0x7fb7f8198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.633+0000 7fb7fdc86700 1 --2- 172.21.15.28:0/1013767338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f8107570 0x7fb7f8198770 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0007d70 tx=0x7fb7f000a340 comp rx=0 tx=0).stop 2024-04-25T01:31:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.633+0000 7fb7f7fff700 1 -- 172.21.15.28:0/1013767338 reap_dead start 2024-04-25T01:31:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.633+0000 7fb7fdc86700 1 -- 172.21.15.28:0/1013767338 shutdown_connections 2024-04-25T01:31:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.633+0000 7fb7fdc86700 1 -- 172.21.15.28:0/1013767338 >> 172.21.15.28:0/1013767338 conn(0x7fb7f8079390 msgr2=0x7fb7f81002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.633+0000 7fb7fdc86700 1 -- 172.21.15.28:0/1013767338 shutdown_connections 2024-04-25T01:31:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:24.633+0000 7fb7fdc86700 1 -- 172.21.15.28:0/1013767338 wait complete. 2024-04-25T01:31:25.878 INFO:teuthology.orchestra.run.smithi028.stdout:352187318295 2024-04-25T01:31:25.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-25T01:31:26.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:25 smithi028 ceph-mon[29390]: pgmap v325: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-25T01:31:26.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:25 smithi130 ceph-mon[32702]: pgmap v325: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-25T01:31:26.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:25 smithi032 ceph-mon[32461]: pgmap v325: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-25T01:31:26.464 INFO:teuthology.orchestra.run.smithi028.stdout:77309411399 2024-04-25T01:31:26.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-25T01:31:28.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:27 smithi028 ceph-mon[29390]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s; 8 B/s, 1 keys/s, 0 objects/s recovering 2024-04-25T01:31:28.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:27 smithi130 ceph-mon[32702]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s; 8 B/s, 1 keys/s, 0 objects/s recovering 2024-04-25T01:31:28.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:27 smithi032 ceph-mon[32461]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s; 8 B/s, 1 keys/s, 0 objects/s recovering 2024-04-25T01:31:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.432+0000 7f41d7ff0700 1 -- 172.21.15.28:0/912035847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41d0107570 msgr2=0x7f41d010f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.432+0000 7f41d7ff0700 1 --2- 172.21.15.28:0/912035847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41d0107570 0x7f41d010f140 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f41c0009990 tx=0x7f41c0009ca0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.432+0000 7f41d7ff0700 1 -- 172.21.15.28:0/912035847 shutdown_connections 2024-04-25T01:31:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.432+0000 7f41d7ff0700 1 --2- 172.21.15.28:0/912035847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41d0107570 0x7f41d010f140 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.432+0000 7f41d7ff0700 1 --2- 172.21.15.28:0/912035847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41d01024e0 0x7f41d01028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.432+0000 7f41d7ff0700 1 --2- 172.21.15.28:0/912035847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41d0102e90 0x7f41d0106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.432+0000 7f41d7ff0700 1 -- 172.21.15.28:0/912035847 >> 172.21.15.28:0/912035847 conn(0x7f41d0079390 msgr2=0x7f41d00797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.432+0000 7f41d7ff0700 1 -- 172.21.15.28:0/912035847 shutdown_connections 2024-04-25T01:31:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.432+0000 7f41d7ff0700 1 -- 172.21.15.28:0/912035847 wait complete. 2024-04-25T01:31:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.433+0000 7f41d7ff0700 1 Processor -- start 2024-04-25T01:31:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.433+0000 7f41d7ff0700 1 -- start start 2024-04-25T01:31:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.433+0000 7f41d7ff0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41d01024e0 0x7f41d01a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.433+0000 7f41d7ff0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41d0102e90 0x7f41d01a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.433+0000 7f41d7ff0700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41d0107570 0x7f41d019aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.433+0000 7f41d7ff0700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41d01125b0 con 0x7f41d0102e90 2024-04-25T01:31:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.433+0000 7f41d7ff0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f41d0112430 con 0x7f41d01024e0 2024-04-25T01:31:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.433+0000 7f41d7ff0700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f41d0112730 con 0x7f41d0107570 2024-04-25T01:31:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.433+0000 7f41d558b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41d0102e90 0x7f41d01a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.433+0000 7f41d558b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41d0102e90 0x7f41d01a0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47758/0 (socket says 172.21.15.28:47758) 2024-04-25T01:31:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.433+0000 7f41d558b700 1 -- 172.21.15.28:0/1057041057 learned_addr learned my addr 172.21.15.28:0/1057041057 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.433+0000 7f41d658d700 1 --2- 172.21.15.28:0/1057041057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41d0107570 0x7f41d019aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.433+0000 7f41d5d8c700 1 --2- 172.21.15.28:0/1057041057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41d01024e0 0x7f41d01a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.433+0000 7f41d558b700 1 -- 172.21.15.28:0/1057041057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41d0107570 msgr2=0x7f41d019aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.433+0000 7f41d558b700 1 --2- 172.21.15.28:0/1057041057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41d0107570 0x7f41d019aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.433+0000 7f41d558b700 1 -- 172.21.15.28:0/1057041057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41d01024e0 msgr2=0x7f41d01a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.433+0000 7f41d558b700 1 --2- 172.21.15.28:0/1057041057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41d01024e0 0x7f41d01a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.433+0000 7f41d558b700 1 -- 172.21.15.28:0/1057041057 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41d019b170 con 0x7f41d0102e90 2024-04-25T01:31:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.433+0000 7f41d5d8c700 1 --2- 172.21.15.28:0/1057041057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41d01024e0 0x7f41d01a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.434+0000 7f41d558b700 1 --2- 172.21.15.28:0/1057041057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41d0102e90 0x7f41d01a0d90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f41c400c8f0 tx=0x7f41c400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.434+0000 7f41ceffd700 1 -- 172.21.15.28:0/1057041057 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c4007940 con 0x7f41d0102e90 2024-04-25T01:31:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.434+0000 7f41d7ff0700 1 -- 172.21.15.28:0/1057041057 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41d019b460 con 0x7f41d0102e90 2024-04-25T01:31:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.434+0000 7f41ceffd700 1 -- 172.21.15.28:0/1057041057 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41c400f450 con 0x7f41d0102e90 2024-04-25T01:31:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.434+0000 7f41ceffd700 1 -- 172.21.15.28:0/1057041057 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c4018660 con 0x7f41d0102e90 2024-04-25T01:31:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.434+0000 7f41d7ff0700 1 -- 172.21.15.28:0/1057041057 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41d01a7720 con 0x7f41d0102e90 2024-04-25T01:31:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.435+0000 7f41d7ff0700 1 -- 172.21.15.28:0/1057041057 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f41d010e7f0 con 0x7f41d0102e90 2024-04-25T01:31:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.436+0000 7f41ceffd700 1 -- 172.21.15.28:0/1057041057 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f41c400f600 con 0x7f41d0102e90 2024-04-25T01:31:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.436+0000 7f41ceffd700 1 --2- 172.21.15.28:0/1057041057 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f41b80a2cb0 0x7f41b80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.437+0000 7f41d5d8c700 1 --2- 172.21.15.28:0/1057041057 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f41b80a2cb0 0x7f41b80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.437+0000 7f41ceffd700 1 -- 172.21.15.28:0/1057041057 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f41c40cf6b0 con 0x7f41d0102e90 2024-04-25T01:31:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.437+0000 7f41ceffd700 1 --2- 172.21.15.28:0/1057041057 >> [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523] conn(0x7f41b80ade10 0x7f41b80b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.437+0000 7f41ceffd700 1 -- 172.21.15.28:0/1057041057 --> [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f41c4024ea0 con 0x7f41b80ade10 2024-04-25T01:31:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.437+0000 7f41ceffd700 1 -- 172.21.15.28:0/1057041057 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f41c40cfb00 con 0x7f41d0102e90 2024-04-25T01:31:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.437+0000 7f41d658d700 1 --2- 172.21.15.28:0/1057041057 >> [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523] conn(0x7f41b80ade10 0x7f41b80b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.437+0000 7f41d5d8c700 1 --2- 172.21.15.28:0/1057041057 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f41b80a2cb0 0x7f41b80a5170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f41d019bb50 tx=0x7f41c8009570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.437+0000 7f41d658d700 1 --2- 172.21.15.28:0/1057041057 >> [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523] conn(0x7f41b80ade10 0x7f41b80b02b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:28.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.439+0000 7f41ceffd700 1 -- 172.21.15.28:0/1057041057 <== osd.6 v2:172.21.15.28:6818/2330487523 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f41c4024ea0 con 0x7f41b80ade10 2024-04-25T01:31:28.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.456+0000 7f41d7ff0700 1 -- 172.21.15.28:0/1057041057 --> [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f41d0103970 con 0x7f41b80ade10 2024-04-25T01:31:28.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.456+0000 7f41ceffd700 1 -- 172.21.15.28:0/1057041057 <== osd.6 v2:172.21.15.28:6818/2330487523 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f41d0103970 con 0x7f41b80ade10 2024-04-25T01:31:28.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.456+0000 7f41d7ff0700 1 -- 172.21.15.28:0/1057041057 >> [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523] conn(0x7f41b80ade10 msgr2=0x7f41b80b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.456+0000 7f41d7ff0700 1 --2- 172.21.15.28:0/1057041057 >> [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523] conn(0x7f41b80ade10 0x7f41b80b02b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:28.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.457+0000 7f41d7ff0700 1 -- 172.21.15.28:0/1057041057 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f41b80a2cb0 msgr2=0x7f41b80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.457+0000 7f41d7ff0700 1 --2- 172.21.15.28:0/1057041057 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f41b80a2cb0 0x7f41b80a5170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f41d019bb50 tx=0x7f41c8009570 comp rx=0 tx=0).stop 2024-04-25T01:31:28.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.457+0000 7f41d7ff0700 1 -- 172.21.15.28:0/1057041057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41d0102e90 msgr2=0x7f41d01a0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:28.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.457+0000 7f41d7ff0700 1 --2- 172.21.15.28:0/1057041057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41d0102e90 0x7f41d01a0d90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f41c400c8f0 tx=0x7f41c400cc00 comp rx=0 tx=0).stop 2024-04-25T01:31:28.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.457+0000 7f41d658d700 1 -- 172.21.15.28:0/1057041057 reap_dead start 2024-04-25T01:31:28.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.457+0000 7f41d7ff0700 1 -- 172.21.15.28:0/1057041057 shutdown_connections 2024-04-25T01:31:28.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.457+0000 7f41d7ff0700 1 -- 172.21.15.28:0/1057041057 >> 172.21.15.28:0/1057041057 conn(0x7f41d0079390 msgr2=0x7f41d01003f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:28.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.457+0000 7f41d7ff0700 1 -- 172.21.15.28:0/1057041057 shutdown_connections 2024-04-25T01:31:28.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.457+0000 7f41d7ff0700 1 -- 172.21.15.28:0/1057041057 wait complete. 2024-04-25T01:31:30.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:29 smithi028 ceph-mon[29390]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s; 8 B/s, 1 keys/s, 0 objects/s recovering 2024-04-25T01:31:30.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:29 smithi130 ceph-mon[32702]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s; 8 B/s, 1 keys/s, 0 objects/s recovering 2024-04-25T01:31:30.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:29 smithi032 ceph-mon[32461]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s; 8 B/s, 1 keys/s, 0 objects/s recovering 2024-04-25T01:31:30.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.649+0000 7f4fb0319700 1 -- 172.21.15.28:0/3720127573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fa8101130 msgr2=0x7f4fa81015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:30.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.649+0000 7f4fb0319700 1 --2- 172.21.15.28:0/3720127573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fa8101130 0x7f4fa81015b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009990 tx=0x7f4f98009ca0 comp rx=0 tx=0).stop 2024-04-25T01:31:30.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.649+0000 7f4fb0319700 1 -- 172.21.15.28:0/3720127573 shutdown_connections 2024-04-25T01:31:30.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.649+0000 7f4fb0319700 1 --2- 172.21.15.28:0/3720127573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4fa8101af0 0x7f4fa810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:30.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.649+0000 7f4fb0319700 1 --2- 172.21.15.28:0/3720127573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fa81071a0 0x7f4fa8107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:30.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.649+0000 7f4fb0319700 1 --2- 172.21.15.28:0/3720127573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fa8101130 0x7f4fa81015b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:30.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.649+0000 7f4fb0319700 1 -- 172.21.15.28:0/3720127573 >> 172.21.15.28:0/3720127573 conn(0x7f4fa80780e0 msgr2=0x7f4fa80784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:30.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.649+0000 7f4fb0319700 1 -- 172.21.15.28:0/3720127573 shutdown_connections 2024-04-25T01:31:30.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.649+0000 7f4fb0319700 1 -- 172.21.15.28:0/3720127573 wait complete. 2024-04-25T01:31:30.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.650+0000 7f4fb0319700 1 Processor -- start 2024-04-25T01:31:30.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.650+0000 7f4fb0319700 1 -- start start 2024-04-25T01:31:30.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.650+0000 7f4fb0319700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fa8101130 0x7f4fa819e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:30.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.650+0000 7f4fb0319700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4fa8101af0 0x7f4fa819ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:30.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.650+0000 7f4fb0319700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fa81071a0 0x7f4fa8198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:30.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.650+0000 7f4fb0319700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa8111510 con 0x7f4fa81071a0 2024-04-25T01:31:30.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.650+0000 7f4fb0319700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa8111390 con 0x7f4fa8101130 2024-04-25T01:31:30.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.650+0000 7f4fb0319700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa8111690 con 0x7f4fa8101af0 2024-04-25T01:31:30.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.650+0000 7f4fae8b6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fa81071a0 0x7f4fa8198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:30.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.650+0000 7f4fae8b6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fa81071a0 0x7f4fa8198900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47766/0 (socket says 172.21.15.28:47766) 2024-04-25T01:31:30.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.650+0000 7f4fae8b6700 1 -- 172.21.15.28:0/3632349331 learned_addr learned my addr 172.21.15.28:0/3632349331 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:30.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.650+0000 7f4fae0b5700 1 --2- 172.21.15.28:0/3632349331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fa8101130 0x7f4fa819e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:30.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.650+0000 7f4fad8b4700 1 --2- 172.21.15.28:0/3632349331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4fa8101af0 0x7f4fa819ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:30.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.650+0000 7f4fae8b6700 1 -- 172.21.15.28:0/3632349331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4fa8101af0 msgr2=0x7f4fa819ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:30.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.650+0000 7f4fae8b6700 1 --2- 172.21.15.28:0/3632349331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4fa8101af0 0x7f4fa819ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:30.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.650+0000 7f4fae8b6700 1 -- 172.21.15.28:0/3632349331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fa8101130 msgr2=0x7f4fa819e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:30.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.650+0000 7f4fae8b6700 1 --2- 172.21.15.28:0/3632349331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fa8101130 0x7f4fa819e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:30.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.650+0000 7f4fae8b6700 1 -- 172.21.15.28:0/3632349331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fa8199060 con 0x7f4fa81071a0 2024-04-25T01:31:30.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.651+0000 7f4fae8b6700 1 --2- 172.21.15.28:0/3632349331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fa81071a0 0x7f4fa8198900 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4fa000b970 tx=0x7f4fa000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:30.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.651+0000 7f4f9f7fe700 1 -- 172.21.15.28:0/3632349331 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa000d860 con 0x7f4fa81071a0 2024-04-25T01:31:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.651+0000 7f4f9f7fe700 1 -- 172.21.15.28:0/3632349331 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fa000e430 con 0x7f4fa81071a0 2024-04-25T01:31:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.651+0000 7f4fb0319700 1 -- 172.21.15.28:0/3632349331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fa8199350 con 0x7f4fa81071a0 2024-04-25T01:31:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.651+0000 7f4f9f7fe700 1 -- 172.21.15.28:0/3632349331 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa0017660 con 0x7f4fa81071a0 2024-04-25T01:31:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.651+0000 7f4fb0319700 1 -- 172.21.15.28:0/3632349331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fa81a5610 con 0x7f4fa81071a0 2024-04-25T01:31:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.652+0000 7f4fb0319700 1 -- 172.21.15.28:0/3632349331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4fa81052a0 con 0x7f4fa81071a0 2024-04-25T01:31:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.653+0000 7f4f9f7fe700 1 -- 172.21.15.28:0/3632349331 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f4fa000e5e0 con 0x7f4fa81071a0 2024-04-25T01:31:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.654+0000 7f4f9f7fe700 1 --2- 172.21.15.28:0/3632349331 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4f940a2d30 0x7f4f940a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.654+0000 7f4fae0b5700 1 --2- 172.21.15.28:0/3632349331 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4f940a2d30 0x7f4f940a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.654+0000 7f4f9f7fe700 1 -- 172.21.15.28:0/3632349331 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f4fa00cf490 con 0x7f4fa81071a0 2024-04-25T01:31:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.654+0000 7f4f9f7fe700 1 --2- 172.21.15.28:0/3632349331 >> [v2:172.21.15.32:6816/1373301181,v1:172.21.15.32:6817/1373301181] conn(0x7f4f940ade90 0x7f4f940b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.654+0000 7f4f9f7fe700 1 -- 172.21.15.28:0/3632349331 --> [v2:172.21.15.32:6816/1373301181,v1:172.21.15.32:6817/1373301181] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4fa0011ec0 con 0x7f4f940ade90 2024-04-25T01:31:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.654+0000 7f4f9f7fe700 1 -- 172.21.15.28:0/3632349331 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f4fa00cf8e0 con 0x7f4fa81071a0 2024-04-25T01:31:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.654+0000 7f4fae0b5700 1 --2- 172.21.15.28:0/3632349331 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4f940a2d30 0x7f4f940a51f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8199a40 tx=0x7f4fa4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.654+0000 7f4fad8b4700 1 --2- 172.21.15.28:0/3632349331 >> [v2:172.21.15.32:6816/1373301181,v1:172.21.15.32:6817/1373301181] conn(0x7f4f940ade90 0x7f4f940b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.655+0000 7f4fad8b4700 1 --2- 172.21.15.28:0/3632349331 >> [v2:172.21.15.32:6816/1373301181,v1:172.21.15.32:6817/1373301181] conn(0x7f4f940ade90 0x7f4f940b0330 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.656+0000 7f4f9f7fe700 1 -- 172.21.15.28:0/3632349331 <== osd.7 v2:172.21.15.32:6816/1373301181 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4fa0011ec0 con 0x7f4f940ade90 2024-04-25T01:31:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.672+0000 7f4fb0319700 1 -- 172.21.15.28:0/3632349331 --> [v2:172.21.15.32:6816/1373301181,v1:172.21.15.32:6817/1373301181] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4fa8102850 con 0x7f4f940ade90 2024-04-25T01:31:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.673+0000 7f4f9f7fe700 1 -- 172.21.15.28:0/3632349331 <== osd.7 v2:172.21.15.32:6816/1373301181 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4fa8102850 con 0x7f4f940ade90 2024-04-25T01:31:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.673+0000 7f4fb0319700 1 -- 172.21.15.28:0/3632349331 >> [v2:172.21.15.32:6816/1373301181,v1:172.21.15.32:6817/1373301181] conn(0x7f4f940ade90 msgr2=0x7f4f940b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.673+0000 7f4fb0319700 1 --2- 172.21.15.28:0/3632349331 >> [v2:172.21.15.32:6816/1373301181,v1:172.21.15.32:6817/1373301181] conn(0x7f4f940ade90 0x7f4f940b0330 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.673+0000 7f4fb0319700 1 -- 172.21.15.28:0/3632349331 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4f940a2d30 msgr2=0x7f4f940a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.673+0000 7f4fb0319700 1 --2- 172.21.15.28:0/3632349331 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f4f940a2d30 0x7f4f940a51f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8199a40 tx=0x7f4fa4009190 comp rx=0 tx=0).stop 2024-04-25T01:31:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.673+0000 7f4fb0319700 1 -- 172.21.15.28:0/3632349331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fa81071a0 msgr2=0x7f4fa8198900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.673+0000 7f4fb0319700 1 --2- 172.21.15.28:0/3632349331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fa81071a0 0x7f4fa8198900 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4fa000b970 tx=0x7f4fa000bc80 comp rx=0 tx=0).stop 2024-04-25T01:31:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.673+0000 7f4fae8b6700 1 -- 172.21.15.28:0/3632349331 reap_dead start 2024-04-25T01:31:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.673+0000 7f4fb0319700 1 -- 172.21.15.28:0/3632349331 shutdown_connections 2024-04-25T01:31:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.673+0000 7f4fb0319700 1 -- 172.21.15.28:0/3632349331 >> 172.21.15.28:0/3632349331 conn(0x7f4fa80780e0 msgr2=0x7f4fa8100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:30.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.673+0000 7f4fb0319700 1 -- 172.21.15.28:0/3632349331 shutdown_connections 2024-04-25T01:31:30.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:28.673+0000 7f4fb0319700 1 -- 172.21.15.28:0/3632349331 wait complete. 2024-04-25T01:31:31.235 INFO:teuthology.orchestra.run.smithi028.stdout:227633266733 2024-04-25T01:31:31.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-25T01:31:32.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:31 smithi130 ceph-mon[32702]: pgmap v328: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 323 B/s wr, 11 op/s; 7 B/s, 1 keys/s, 0 objects/s recovering 2024-04-25T01:31:32.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:31 smithi032 ceph-mon[32461]: pgmap v328: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 323 B/s wr, 11 op/s; 7 B/s, 1 keys/s, 0 objects/s recovering 2024-04-25T01:31:32.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:31 smithi028 ceph-mon[29390]: pgmap v328: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 323 B/s wr, 11 op/s; 7 B/s, 1 keys/s, 0 objects/s recovering 2024-04-25T01:31:33.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.252+0000 7fd8370fc700 1 -- 172.21.15.28:0/1203153816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd830108410 msgr2=0x7fd8301087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:33.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.252+0000 7fd8370fc700 1 --2- 172.21.15.28:0/1203153816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd830108410 0x7fd8301087f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd824009990 tx=0x7fd824009ca0 comp rx=0 tx=0).stop 2024-04-25T01:31:33.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.252+0000 7fd8370fc700 1 -- 172.21.15.28:0/1203153816 shutdown_connections 2024-04-25T01:31:33.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.252+0000 7fd8370fc700 1 --2- 172.21.15.28:0/1203153816 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd83006b2c0 0x7fd83010f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:33.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.252+0000 7fd8370fc700 1 --2- 172.21.15.28:0/1203153816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd830108410 0x7fd8301087f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:33.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.252+0000 7fd8370fc700 1 --2- 172.21.15.28:0/1203153816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83006a900 0x7fd83006ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:33.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.252+0000 7fd8370fc700 1 -- 172.21.15.28:0/1203153816 >> 172.21.15.28:0/1203153816 conn(0x7fd8300780e0 msgr2=0x7fd8300784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:33.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.252+0000 7fd8370fc700 1 -- 172.21.15.28:0/1203153816 shutdown_connections 2024-04-25T01:31:33.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.252+0000 7fd8370fc700 1 -- 172.21.15.28:0/1203153816 wait complete. 2024-04-25T01:31:33.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.252+0000 7fd8370fc700 1 Processor -- start 2024-04-25T01:31:33.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.252+0000 7fd8370fc700 1 -- start start 2024-04-25T01:31:33.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.253+0000 7fd8370fc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd83006a900 0x7fd8301015e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:33.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.253+0000 7fd8370fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83006b2c0 0x7fd830101b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:33.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.253+0000 7fd8370fc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd830108410 0x7fd830102060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:33.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.253+0000 7fd8370fc700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd830112500 con 0x7fd83006a900 2024-04-25T01:31:33.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.253+0000 7fd8370fc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd830112380 con 0x7fd83006b2c0 2024-04-25T01:31:33.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.253+0000 7fd8370fc700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd830112680 con 0x7fd830108410 2024-04-25T01:31:33.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.253+0000 7fd834e98700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd83006a900 0x7fd8301015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:33.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.253+0000 7fd834e98700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd83006a900 0x7fd8301015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47778/0 (socket says 172.21.15.28:47778) 2024-04-25T01:31:33.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.253+0000 7fd834e98700 1 -- 172.21.15.28:0/23272531 learned_addr learned my addr 172.21.15.28:0/23272531 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:33.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.253+0000 7fd82ffff700 1 --2- 172.21.15.28:0/23272531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83006b2c0 0x7fd830101b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:33.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.253+0000 7fd835699700 1 --2- 172.21.15.28:0/23272531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd830108410 0x7fd830102060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:33.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.253+0000 7fd834e98700 1 -- 172.21.15.28:0/23272531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd830108410 msgr2=0x7fd830102060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:33.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.253+0000 7fd834e98700 1 --2- 172.21.15.28:0/23272531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd830108410 0x7fd830102060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:33.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.253+0000 7fd834e98700 1 -- 172.21.15.28:0/23272531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83006b2c0 msgr2=0x7fd830101b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:33.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.253+0000 7fd834e98700 1 --2- 172.21.15.28:0/23272531 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd83006b2c0 0x7fd830101b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:33.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.253+0000 7fd834e98700 1 -- 172.21.15.28:0/23272531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8301a6ea0 con 0x7fd83006a900 2024-04-25T01:31:33.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.253+0000 7fd835699700 1 --2- 172.21.15.28:0/23272531 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd830108410 0x7fd830102060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:33.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.253+0000 7fd834e98700 1 --2- 172.21.15.28:0/23272531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd83006a900 0x7fd8301015e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd824000c00 tx=0x7fd8240047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:33.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.254+0000 7fd82dffb700 1 -- 172.21.15.28:0/23272531 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd824027070 con 0x7fd83006a900 2024-04-25T01:31:33.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.254+0000 7fd8370fc700 1 -- 172.21.15.28:0/23272531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8301a7130 con 0x7fd83006a900 2024-04-25T01:31:33.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.254+0000 7fd82dffb700 1 -- 172.21.15.28:0/23272531 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd824003780 con 0x7fd83006a900 2024-04-25T01:31:33.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.254+0000 7fd82dffb700 1 -- 172.21.15.28:0/23272531 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd824022dd0 con 0x7fd83006a900 2024-04-25T01:31:33.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.254+0000 7fd8370fc700 1 -- 172.21.15.28:0/23272531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8301a75e0 con 0x7fd83006a900 2024-04-25T01:31:33.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.255+0000 7fd8370fc700 1 -- 172.21.15.28:0/23272531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd8300629c0 con 0x7fd83006a900 2024-04-25T01:31:33.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.256+0000 7fd82dffb700 1 -- 172.21.15.28:0/23272531 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fd82402a070 con 0x7fd83006a900 2024-04-25T01:31:33.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.257+0000 7fd82dffb700 1 --2- 172.21.15.28:0/23272531 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd8200a2cb0 0x7fd8200a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.257+0000 7fd82ffff700 1 --2- 172.21.15.28:0/23272531 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd8200a2cb0 0x7fd8200a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.257+0000 7fd82dffb700 1 -- 172.21.15.28:0/23272531 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fd8240d07d0 con 0x7fd83006a900 2024-04-25T01:31:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.257+0000 7fd82dffb700 1 --2- 172.21.15.28:0/23272531 >> [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772] conn(0x7fd8200ade10 0x7fd8200b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.257+0000 7fd82dffb700 1 -- 172.21.15.28:0/23272531 --> [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd824016e70 con 0x7fd8200ade10 2024-04-25T01:31:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.257+0000 7fd82dffb700 1 -- 172.21.15.28:0/23272531 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fd8240d0c20 con 0x7fd83006a900 2024-04-25T01:31:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.257+0000 7fd835699700 1 --2- 172.21.15.28:0/23272531 >> [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772] conn(0x7fd8200ade10 0x7fd8200b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.257+0000 7fd82ffff700 1 --2- 172.21.15.28:0/23272531 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd8200a2cb0 0x7fd8200a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd8300786e0 tx=0x7fd81c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.257+0000 7fd835699700 1 --2- 172.21.15.28:0/23272531 >> [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772] conn(0x7fd8200ade10 0x7fd8200b02b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:33.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.258+0000 7fd82dffb700 1 -- 172.21.15.28:0/23272531 <== osd.0 v2:172.21.15.28:6802/2903822772 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd824016e70 con 0x7fd8200ade10 2024-04-25T01:31:33.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.275+0000 7fd8370fc700 1 -- 172.21.15.28:0/23272531 --> [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd830062e40 con 0x7fd8200ade10 2024-04-25T01:31:33.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.275+0000 7fd82dffb700 1 -- 172.21.15.28:0/23272531 <== osd.0 v2:172.21.15.28:6802/2903822772 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd830062e40 con 0x7fd8200ade10 2024-04-25T01:31:33.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.275+0000 7fd8370fc700 1 -- 172.21.15.28:0/23272531 >> [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772] conn(0x7fd8200ade10 msgr2=0x7fd8200b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:33.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.275+0000 7fd8370fc700 1 --2- 172.21.15.28:0/23272531 >> [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772] conn(0x7fd8200ade10 0x7fd8200b02b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:33.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.275+0000 7fd8370fc700 1 -- 172.21.15.28:0/23272531 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd8200a2cb0 msgr2=0x7fd8200a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:33.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.275+0000 7fd8370fc700 1 --2- 172.21.15.28:0/23272531 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd8200a2cb0 0x7fd8200a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd8300786e0 tx=0x7fd81c00b410 comp rx=0 tx=0).stop 2024-04-25T01:31:33.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.275+0000 7fd8370fc700 1 -- 172.21.15.28:0/23272531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd83006a900 msgr2=0x7fd8301015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:33.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.275+0000 7fd8370fc700 1 --2- 172.21.15.28:0/23272531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd83006a900 0x7fd8301015e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd824000c00 tx=0x7fd8240047b0 comp rx=0 tx=0).stop 2024-04-25T01:31:33.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.275+0000 7fd835699700 1 -- 172.21.15.28:0/23272531 reap_dead start 2024-04-25T01:31:33.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.275+0000 7fd8370fc700 1 -- 172.21.15.28:0/23272531 shutdown_connections 2024-04-25T01:31:33.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.275+0000 7fd8370fc700 1 -- 172.21.15.28:0/23272531 >> 172.21.15.28:0/23272531 conn(0x7fd8300780e0 msgr2=0x7fd830100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:33.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.276+0000 7fd8370fc700 1 -- 172.21.15.28:0/23272531 shutdown_connections 2024-04-25T01:31:33.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.276+0000 7fd8370fc700 1 -- 172.21.15.28:0/23272531 wait complete. 2024-04-25T01:31:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.707+0000 7f8c2952a700 1 -- 172.21.15.28:0/3661034070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c24107190 msgr2=0x7f8c24107570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.707+0000 7f8c2952a700 1 --2- 172.21.15.28:0/3661034070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c24107190 0x7f8c24107570 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8c14009990 tx=0x7f8c14009ca0 comp rx=0 tx=0).stop 2024-04-25T01:31:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.707+0000 7f8c2952a700 1 -- 172.21.15.28:0/3661034070 shutdown_connections 2024-04-25T01:31:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.707+0000 7f8c2952a700 1 --2- 172.21.15.28:0/3661034070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c24101ae0 0x7f8c2410dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.707+0000 7f8c2952a700 1 --2- 172.21.15.28:0/3661034070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c24107190 0x7f8c24107570 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.707+0000 7f8c2952a700 1 --2- 172.21.15.28:0/3661034070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c24101120 0x7f8c241015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.707+0000 7f8c2952a700 1 -- 172.21.15.28:0/3661034070 >> 172.21.15.28:0/3661034070 conn(0x7f8c240780a0 msgr2=0x7f8c240784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:33.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.707+0000 7f8c2952a700 1 -- 172.21.15.28:0/3661034070 shutdown_connections 2024-04-25T01:31:33.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.707+0000 7f8c2952a700 1 -- 172.21.15.28:0/3661034070 wait complete. 2024-04-25T01:31:33.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.708+0000 7f8c2952a700 1 Processor -- start 2024-04-25T01:31:33.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.708+0000 7f8c2952a700 1 -- start start 2024-04-25T01:31:33.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.708+0000 7f8c2952a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c24101120 0x7f8c24102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.708+0000 7f8c2952a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c24101ae0 0x7f8c24103340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.708+0000 7f8c2952a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c24107190 0x7f8c24103880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.708+0000 7f8c2952a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8c24111400 con 0x7f8c24107190 2024-04-25T01:31:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.708+0000 7f8c2952a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8c24111280 con 0x7f8c24101ae0 2024-04-25T01:31:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.708+0000 7f8c2952a700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8c24111580 con 0x7f8c24101120 2024-04-25T01:31:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.708+0000 7f8c227fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c24101ae0 0x7f8c24103340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.708+0000 7f8c237fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c24107190 0x7f8c24103880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.708+0000 7f8c22ffd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c24101120 0x7f8c24102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.708+0000 7f8c227fc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c24101ae0 0x7f8c24103340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:47864/0 (socket says 172.21.15.28:47864) 2024-04-25T01:31:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.708+0000 7f8c227fc700 1 -- 172.21.15.28:0/444041080 learned_addr learned my addr 172.21.15.28:0/444041080 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.709+0000 7f8c227fc700 1 -- 172.21.15.28:0/444041080 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c24101120 msgr2=0x7f8c24102e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.709+0000 7f8c227fc700 1 --2- 172.21.15.28:0/444041080 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8c24101120 0x7f8c24102e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.709+0000 7f8c227fc700 1 -- 172.21.15.28:0/444041080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c24107190 msgr2=0x7f8c24103880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:33.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.709+0000 7f8c227fc700 1 --2- 172.21.15.28:0/444041080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c24107190 0x7f8c24103880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:33.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.709+0000 7f8c227fc700 1 -- 172.21.15.28:0/444041080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c241a6f10 con 0x7f8c24101ae0 2024-04-25T01:31:33.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.709+0000 7f8c237fe700 1 --2- 172.21.15.28:0/444041080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c24107190 0x7f8c24103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:31:33.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.709+0000 7f8c227fc700 1 --2- 172.21.15.28:0/444041080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c24101ae0 0x7f8c24103340 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c00e9a0 tx=0x7f8c0c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:33.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.709+0000 7f8c1bfff700 1 -- 172.21.15.28:0/444041080 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c00ca10 con 0x7f8c24101ae0 2024-04-25T01:31:33.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.709+0000 7f8c2952a700 1 -- 172.21.15.28:0/444041080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c241a7200 con 0x7f8c24101ae0 2024-04-25T01:31:33.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.709+0000 7f8c1bfff700 1 -- 172.21.15.28:0/444041080 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c0c00cbb0 con 0x7f8c24101ae0 2024-04-25T01:31:33.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.709+0000 7f8c1bfff700 1 -- 172.21.15.28:0/444041080 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c018680 con 0x7f8c24101ae0 2024-04-25T01:31:33.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.709+0000 7f8c2952a700 1 -- 172.21.15.28:0/444041080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c241a7790 con 0x7f8c24101ae0 2024-04-25T01:31:33.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.710+0000 7f8c2952a700 1 -- 172.21.15.28:0/444041080 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8c24105210 con 0x7f8c24101ae0 2024-04-25T01:31:33.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.711+0000 7f8c1bfff700 1 -- 172.21.15.28:0/444041080 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f8c0c010430 con 0x7f8c24101ae0 2024-04-25T01:31:33.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.712+0000 7f8c1bfff700 1 --2- 172.21.15.28:0/444041080 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8c100a2cb0 0x7f8c100a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:33.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.712+0000 7f8c22ffd700 1 --2- 172.21.15.28:0/444041080 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8c100a2cb0 0x7f8c100a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:33.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.712+0000 7f8c1bfff700 1 -- 172.21.15.28:0/444041080 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f8c0c0cffe0 con 0x7f8c24101ae0 2024-04-25T01:31:33.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.712+0000 7f8c1bfff700 1 --2- 172.21.15.28:0/444041080 >> [v2:172.21.15.32:6800/2516196299,v1:172.21.15.32:6801/2516196299] conn(0x7f8c100ade10 0x7f8c100b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:33.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.712+0000 7f8c1bfff700 1 -- 172.21.15.28:0/444041080 --> [v2:172.21.15.32:6800/2516196299,v1:172.21.15.32:6801/2516196299] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8c0c01bec0 con 0x7f8c100ade10 2024-04-25T01:31:33.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.712+0000 7f8c1bfff700 1 -- 172.21.15.28:0/444041080 <== mon.2 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f8c0c0d03f0 con 0x7f8c24101ae0 2024-04-25T01:31:33.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.712+0000 7f8c237fe700 1 --2- 172.21.15.28:0/444041080 >> [v2:172.21.15.32:6800/2516196299,v1:172.21.15.32:6801/2516196299] conn(0x7f8c100ade10 0x7f8c100b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:33.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.713+0000 7f8c22ffd700 1 --2- 172.21.15.28:0/444041080 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8c100a2cb0 0x7f8c100a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8c1400b2e0 tx=0x7f8c14005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:33.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.713+0000 7f8c237fe700 1 --2- 172.21.15.28:0/444041080 >> [v2:172.21.15.32:6800/2516196299,v1:172.21.15.32:6801/2516196299] conn(0x7f8c100ade10 0x7f8c100b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:33.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.714+0000 7f8c1bfff700 1 -- 172.21.15.28:0/444041080 <== osd.1 v2:172.21.15.32:6800/2516196299 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8c0c01bec0 con 0x7f8c100ade10 2024-04-25T01:31:33.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.730+0000 7f8c2952a700 1 -- 172.21.15.28:0/444041080 --> [v2:172.21.15.32:6800/2516196299,v1:172.21.15.32:6801/2516196299] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8c24102840 con 0x7f8c100ade10 2024-04-25T01:31:33.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.731+0000 7f8c1bfff700 1 -- 172.21.15.28:0/444041080 <== osd.1 v2:172.21.15.32:6800/2516196299 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8c24102840 con 0x7f8c100ade10 2024-04-25T01:31:33.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.731+0000 7f8c2952a700 1 -- 172.21.15.28:0/444041080 >> [v2:172.21.15.32:6800/2516196299,v1:172.21.15.32:6801/2516196299] conn(0x7f8c100ade10 msgr2=0x7f8c100b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:33.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.731+0000 7f8c2952a700 1 --2- 172.21.15.28:0/444041080 >> [v2:172.21.15.32:6800/2516196299,v1:172.21.15.32:6801/2516196299] conn(0x7f8c100ade10 0x7f8c100b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:33.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.731+0000 7f8c2952a700 1 -- 172.21.15.28:0/444041080 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8c100a2cb0 msgr2=0x7f8c100a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:33.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.731+0000 7f8c2952a700 1 --2- 172.21.15.28:0/444041080 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8c100a2cb0 0x7f8c100a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8c1400b2e0 tx=0x7f8c14005960 comp rx=0 tx=0).stop 2024-04-25T01:31:33.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.731+0000 7f8c2952a700 1 -- 172.21.15.28:0/444041080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c24101ae0 msgr2=0x7f8c24103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:33.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.731+0000 7f8c2952a700 1 --2- 172.21.15.28:0/444041080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8c24101ae0 0x7f8c24103340 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c00e9a0 tx=0x7f8c0c00ecb0 comp rx=0 tx=0).stop 2024-04-25T01:31:33.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.731+0000 7f8c237fe700 1 -- 172.21.15.28:0/444041080 reap_dead start 2024-04-25T01:31:33.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.731+0000 7f8c2952a700 1 -- 172.21.15.28:0/444041080 shutdown_connections 2024-04-25T01:31:33.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.731+0000 7f8c2952a700 1 -- 172.21.15.28:0/444041080 >> 172.21.15.28:0/444041080 conn(0x7f8c240780a0 msgr2=0x7f8c24100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:33.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.731+0000 7f8c2952a700 1 -- 172.21.15.28:0/444041080 shutdown_connections 2024-04-25T01:31:33.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:33.731+0000 7f8c2952a700 1 -- 172.21.15.28:0/444041080 wait complete. 2024-04-25T01:31:34.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:33 smithi028 ceph-mon[29390]: pgmap v329: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 293 B/s wr, 9 op/s; 6 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T01:31:34.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:33 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:31:34.125 INFO:teuthology.orchestra.run.smithi028.stdout:206158430259 2024-04-25T01:31:34.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-25T01:31:34.132 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:33 smithi032 ceph-mon[32461]: pgmap v329: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 293 B/s wr, 9 op/s; 6 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T01:31:34.132 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:33 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:31:34.170 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:33 smithi130 ceph-mon[32702]: pgmap v329: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 293 B/s wr, 9 op/s; 6 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T01:31:34.170 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:33 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:31:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.451+0000 7f14257af700 1 -- 172.21.15.28:0/2357781545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1420107570 msgr2=0x7f142010f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.451+0000 7f14257af700 1 --2- 172.21.15.28:0/2357781545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1420107570 0x7f142010f140 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f141800b0a0 tx=0x7f141800b3b0 comp rx=0 tx=0).stop 2024-04-25T01:31:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.451+0000 7f14257af700 1 -- 172.21.15.28:0/2357781545 shutdown_connections 2024-04-25T01:31:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.451+0000 7f14257af700 1 --2- 172.21.15.28:0/2357781545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1420107570 0x7f142010f140 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.451+0000 7f14257af700 1 --2- 172.21.15.28:0/2357781545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14201024e0 0x7f14201028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.451+0000 7f14257af700 1 --2- 172.21.15.28:0/2357781545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1420102e90 0x7f1420106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.451+0000 7f14257af700 1 -- 172.21.15.28:0/2357781545 >> 172.21.15.28:0/2357781545 conn(0x7f1420079390 msgr2=0x7f14200797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.451+0000 7f14257af700 1 -- 172.21.15.28:0/2357781545 shutdown_connections 2024-04-25T01:31:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.451+0000 7f14257af700 1 -- 172.21.15.28:0/2357781545 wait complete. 2024-04-25T01:31:34.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.452+0000 7f14257af700 1 Processor -- start 2024-04-25T01:31:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.452+0000 7f14257af700 1 -- start start 2024-04-25T01:31:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.452+0000 7f14257af700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14201024e0 0x7f142019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.452+0000 7f14257af700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1420102e90 0x7f142019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.452+0000 7f14257af700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1420107570 0x7f1420198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.452+0000 7f14257af700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1420112550 con 0x7f1420107570 2024-04-25T01:31:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.452+0000 7f14257af700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f14201123d0 con 0x7f14201024e0 2024-04-25T01:31:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.452+0000 7f14257af700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f14201126d0 con 0x7f1420102e90 2024-04-25T01:31:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.453+0000 7f141effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14201024e0 0x7f142019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.453+0000 7f141effd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14201024e0 0x7f142019e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:47880/0 (socket says 172.21.15.28:47880) 2024-04-25T01:31:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.453+0000 7f141effd700 1 -- 172.21.15.28:0/1425005066 learned_addr learned my addr 172.21.15.28:0/1425005066 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.453+0000 7f141f7fe700 1 --2- 172.21.15.28:0/1425005066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1420107570 0x7f1420198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.453+0000 7f141e7fc700 1 --2- 172.21.15.28:0/1425005066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1420102e90 0x7f142019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.453+0000 7f141e7fc700 1 -- 172.21.15.28:0/1425005066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14201024e0 msgr2=0x7f142019e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.453+0000 7f141e7fc700 1 --2- 172.21.15.28:0/1425005066 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14201024e0 0x7f142019e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.453+0000 7f141e7fc700 1 -- 172.21.15.28:0/1425005066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1420107570 msgr2=0x7f1420198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.453+0000 7f141e7fc700 1 --2- 172.21.15.28:0/1425005066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1420107570 0x7f1420198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.453+0000 7f141e7fc700 1 -- 172.21.15.28:0/1425005066 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1420198ff0 con 0x7f1420102e90 2024-04-25T01:31:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.453+0000 7f141e7fc700 1 --2- 172.21.15.28:0/1425005066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1420102e90 0x7f142019eb80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f140800e9c0 tx=0x7f140800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.454+0000 7f1417fff700 1 -- 172.21.15.28:0/1425005066 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140800c9c0 con 0x7f1420102e90 2024-04-25T01:31:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.454+0000 7f1417fff700 1 -- 172.21.15.28:0/1425005066 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f140800cb60 con 0x7f1420102e90 2024-04-25T01:31:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.454+0000 7f1417fff700 1 -- 172.21.15.28:0/1425005066 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1408018680 con 0x7f1420102e90 2024-04-25T01:31:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.454+0000 7f14257af700 1 -- 172.21.15.28:0/1425005066 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14201992e0 con 0x7f1420102e90 2024-04-25T01:31:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.454+0000 7f14257af700 1 -- 172.21.15.28:0/1425005066 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14201a54a0 con 0x7f1420102e90 2024-04-25T01:31:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.455+0000 7f14257af700 1 -- 172.21.15.28:0/1425005066 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f14200629c0 con 0x7f1420102e90 2024-04-25T01:31:34.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.457+0000 7f1417fff700 1 -- 172.21.15.28:0/1425005066 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1408010430 con 0x7f1420102e90 2024-04-25T01:31:34.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.457+0000 7f1417fff700 1 --2- 172.21.15.28:0/1425005066 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f140c0a2d30 0x7f140c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:34.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.457+0000 7f141effd700 1 --2- 172.21.15.28:0/1425005066 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f140c0a2d30 0x7f140c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:34.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.457+0000 7f1417fff700 1 -- 172.21.15.28:0/1425005066 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f14080d0200 con 0x7f1420102e90 2024-04-25T01:31:34.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.458+0000 7f1417fff700 1 --2- 172.21.15.28:0/1425005066 >> [v2:172.21.15.130:6808/4253601662,v1:172.21.15.130:6809/4253601662] conn(0x7f140c0ade90 0x7f140c0b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.458+0000 7f1417fff700 1 -- 172.21.15.28:0/1425005066 --> [v2:172.21.15.130:6808/4253601662,v1:172.21.15.130:6809/4253601662] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f140801ae50 con 0x7f140c0ade90 2024-04-25T01:31:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.458+0000 7f1417fff700 1 -- 172.21.15.28:0/1425005066 <== mon.1 v2:172.21.15.130:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f14080d05b0 con 0x7f1420102e90 2024-04-25T01:31:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.458+0000 7f141f7fe700 1 --2- 172.21.15.28:0/1425005066 >> [v2:172.21.15.130:6808/4253601662,v1:172.21.15.130:6809/4253601662] conn(0x7f140c0ade90 0x7f140c0b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.458+0000 7f141effd700 1 --2- 172.21.15.28:0/1425005066 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f140c0a2d30 0x7f140c0a51f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f141000bd30 tx=0x7f141000b480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.459+0000 7f141f7fe700 1 --2- 172.21.15.28:0/1425005066 >> [v2:172.21.15.130:6808/4253601662,v1:172.21.15.130:6809/4253601662] conn(0x7f140c0ade90 0x7f140c0b0330 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.460+0000 7f1417fff700 1 -- 172.21.15.28:0/1425005066 <== osd.5 v2:172.21.15.130:6808/4253601662 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f140801ae50 con 0x7f140c0ade90 2024-04-25T01:31:34.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.477+0000 7f14257af700 1 -- 172.21.15.28:0/1425005066 --> [v2:172.21.15.130:6808/4253601662,v1:172.21.15.130:6809/4253601662] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1420062dc0 con 0x7f140c0ade90 2024-04-25T01:31:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.477+0000 7f1417fff700 1 -- 172.21.15.28:0/1425005066 <== osd.5 v2:172.21.15.130:6808/4253601662 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1420062dc0 con 0x7f140c0ade90 2024-04-25T01:31:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.478+0000 7f14257af700 1 -- 172.21.15.28:0/1425005066 >> [v2:172.21.15.130:6808/4253601662,v1:172.21.15.130:6809/4253601662] conn(0x7f140c0ade90 msgr2=0x7f140c0b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.478+0000 7f14257af700 1 --2- 172.21.15.28:0/1425005066 >> [v2:172.21.15.130:6808/4253601662,v1:172.21.15.130:6809/4253601662] conn(0x7f140c0ade90 0x7f140c0b0330 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.478+0000 7f14257af700 1 -- 172.21.15.28:0/1425005066 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f140c0a2d30 msgr2=0x7f140c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.478+0000 7f14257af700 1 --2- 172.21.15.28:0/1425005066 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f140c0a2d30 0x7f140c0a51f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f141000bd30 tx=0x7f141000b480 comp rx=0 tx=0).stop 2024-04-25T01:31:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.478+0000 7f14257af700 1 -- 172.21.15.28:0/1425005066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1420102e90 msgr2=0x7f142019eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.478+0000 7f14257af700 1 --2- 172.21.15.28:0/1425005066 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1420102e90 0x7f142019eb80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f140800e9c0 tx=0x7f140800ecd0 comp rx=0 tx=0).stop 2024-04-25T01:31:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.478+0000 7f141f7fe700 1 -- 172.21.15.28:0/1425005066 reap_dead start 2024-04-25T01:31:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.478+0000 7f14257af700 1 -- 172.21.15.28:0/1425005066 shutdown_connections 2024-04-25T01:31:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.478+0000 7f14257af700 1 -- 172.21.15.28:0/1425005066 >> 172.21.15.28:0/1425005066 conn(0x7f1420079390 msgr2=0x7f14201002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.478+0000 7f14257af700 1 -- 172.21.15.28:0/1425005066 shutdown_connections 2024-04-25T01:31:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:34.478+0000 7f14257af700 1 -- 172.21.15.28:0/1425005066 wait complete. 2024-04-25T01:31:35.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.996+0000 7f11fdddd700 1 -- 172.21.15.28:0/3005201533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f8102540 msgr2=0x7f11f8102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:35.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.996+0000 7f11fdddd700 1 --2- 172.21.15.28:0/3005201533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f8102540 0x7f11f8102920 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f11e8009990 tx=0x7f11e8009ca0 comp rx=0 tx=0).stop 2024-04-25T01:31:35.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.996+0000 7f11fdddd700 1 -- 172.21.15.28:0/3005201533 shutdown_connections 2024-04-25T01:31:35.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.996+0000 7f11fdddd700 1 --2- 172.21.15.28:0/3005201533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11f81075a0 0x7f11f810f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:35.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.996+0000 7f11fdddd700 1 --2- 172.21.15.28:0/3005201533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f8102540 0x7f11f8102920 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:35.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.996+0000 7f11fdddd700 1 --2- 172.21.15.28:0/3005201533 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11f8102ef0 0x7f11f8106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:35.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.996+0000 7f11fdddd700 1 -- 172.21.15.28:0/3005201533 >> 172.21.15.28:0/3005201533 conn(0x7f11f8079310 msgr2=0x7f11f8079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:35.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.996+0000 7f11fdddd700 1 -- 172.21.15.28:0/3005201533 shutdown_connections 2024-04-25T01:31:35.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.996+0000 7f11fdddd700 1 -- 172.21.15.28:0/3005201533 wait complete. 2024-04-25T01:31:35.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.997+0000 7f11fdddd700 1 Processor -- start 2024-04-25T01:31:35.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.997+0000 7f11fdddd700 1 -- start start 2024-04-25T01:31:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.997+0000 7f11fdddd700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11f8102540 0x7f11f8074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.997+0000 7f11fdddd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11f8102ef0 0x7f11f8074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.997+0000 7f11fdddd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f81075a0 0x7f11f8074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.997+0000 7f11fdddd700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11f8112560 con 0x7f11f81075a0 2024-04-25T01:31:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.997+0000 7f11fdddd700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f11f81123e0 con 0x7f11f8102ef0 2024-04-25T01:31:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.997+0000 7f11fdddd700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f11f81126e0 con 0x7f11f8102540 2024-04-25T01:31:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.997+0000 7f11f7fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f81075a0 0x7f11f8074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.997+0000 7f11eeffd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11f8102ef0 0x7f11f8074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.997+0000 7f11f7fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f81075a0 0x7f11f8074fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47836/0 (socket says 172.21.15.28:47836) 2024-04-25T01:31:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.997+0000 7f11f7fff700 1 -- 172.21.15.28:0/2870448662 learned_addr learned my addr 172.21.15.28:0/2870448662 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.998+0000 7f11f77fe700 1 --2- 172.21.15.28:0/2870448662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11f8102540 0x7f11f8074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.998+0000 7f11eeffd700 1 -- 172.21.15.28:0/2870448662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11f8102540 msgr2=0x7f11f8074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.998+0000 7f11eeffd700 1 --2- 172.21.15.28:0/2870448662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11f8102540 0x7f11f8074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.998+0000 7f11eeffd700 1 -- 172.21.15.28:0/2870448662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f81075a0 msgr2=0x7f11f8074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:35.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.998+0000 7f11eeffd700 1 --2- 172.21.15.28:0/2870448662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f81075a0 0x7f11f8074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:35.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.998+0000 7f11eeffd700 1 -- 172.21.15.28:0/2870448662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f81a6e80 con 0x7f11f8102ef0 2024-04-25T01:31:35.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.998+0000 7f11f7fff700 1 --2- 172.21.15.28:0/2870448662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11f81075a0 0x7f11f8074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:31:35.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.998+0000 7f11eeffd700 1 --2- 172.21.15.28:0/2870448662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11f8102ef0 0x7f11f8074a70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f11e000c940 tx=0x7f11e000cc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:35.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.998+0000 7f11f57fa700 1 -- 172.21.15.28:0/2870448662 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e0014070 con 0x7f11f8102ef0 2024-04-25T01:31:35.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.998+0000 7f11f57fa700 1 -- 172.21.15.28:0/2870448662 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11e0004440 con 0x7f11f8102ef0 2024-04-25T01:31:35.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.998+0000 7f11fdddd700 1 -- 172.21.15.28:0/2870448662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11f81a7170 con 0x7f11f8102ef0 2024-04-25T01:31:35.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.998+0000 7f11f57fa700 1 -- 172.21.15.28:0/2870448662 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e0007bc0 con 0x7f11f8102ef0 2024-04-25T01:31:35.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.998+0000 7f11fdddd700 1 -- 172.21.15.28:0/2870448662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11f81a76b0 con 0x7f11f8102ef0 2024-04-25T01:31:35.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:35.999+0000 7f11fdddd700 1 -- 172.21.15.28:0/2870448662 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f11f8062940 con 0x7f11f8102ef0 2024-04-25T01:31:35.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.000+0000 7f11f57fa700 1 -- 172.21.15.28:0/2870448662 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f11e0017070 con 0x7f11f8102ef0 2024-04-25T01:31:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.001+0000 7f11f57fa700 1 --2- 172.21.15.28:0/2870448662 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f11e40a2cb0 0x7f11e40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.001+0000 7f11f77fe700 1 --2- 172.21.15.28:0/2870448662 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f11e40a2cb0 0x7f11e40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.001+0000 7f11f57fa700 1 -- 172.21.15.28:0/2870448662 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f11e00cf590 con 0x7f11f8102ef0 2024-04-25T01:31:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.001+0000 7f11f57fa700 1 --2- 172.21.15.28:0/2870448662 >> [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927] conn(0x7f11e40ade10 0x7f11e40b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.001+0000 7f11f57fa700 1 -- 172.21.15.28:0/2870448662 --> [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f11e40b0980 con 0x7f11e40ade10 2024-04-25T01:31:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.001+0000 7f11f57fa700 1 -- 172.21.15.28:0/2870448662 <== mon.2 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f11e00cf940 con 0x7f11f8102ef0 2024-04-25T01:31:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.001+0000 7f11f7fff700 1 --2- 172.21.15.28:0/2870448662 >> [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927] conn(0x7f11e40ade10 0x7f11e40b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.001+0000 7f11f77fe700 1 --2- 172.21.15.28:0/2870448662 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f11e40a2cb0 0x7f11e40a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f11e80098e0 tx=0x7f11e8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.002+0000 7f11f7fff700 1 --2- 172.21.15.28:0/2870448662 >> [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927] conn(0x7f11e40ade10 0x7f11e40b02b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.002+0000 7f11f57fa700 1 -- 172.21.15.28:0/2870448662 <== osd.3 v2:172.21.15.28:6810/4005405927 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f11e40b0980 con 0x7f11e40ade10 2024-04-25T01:31:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.019+0000 7f11fdddd700 1 -- 172.21.15.28:0/2870448662 --> [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f11f8062d40 con 0x7f11e40ade10 2024-04-25T01:31:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.019+0000 7f11f57fa700 1 -- 172.21.15.28:0/2870448662 <== osd.3 v2:172.21.15.28:6810/4005405927 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f11f8062d40 con 0x7f11e40ade10 2024-04-25T01:31:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.020+0000 7f11fdddd700 1 -- 172.21.15.28:0/2870448662 >> [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927] conn(0x7f11e40ade10 msgr2=0x7f11e40b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.020+0000 7f11fdddd700 1 --2- 172.21.15.28:0/2870448662 >> [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927] conn(0x7f11e40ade10 0x7f11e40b02b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.020+0000 7f11fdddd700 1 -- 172.21.15.28:0/2870448662 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f11e40a2cb0 msgr2=0x7f11e40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.020+0000 7f11fdddd700 1 --2- 172.21.15.28:0/2870448662 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f11e40a2cb0 0x7f11e40a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f11e80098e0 tx=0x7f11e8005960 comp rx=0 tx=0).stop 2024-04-25T01:31:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.020+0000 7f11fdddd700 1 -- 172.21.15.28:0/2870448662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11f8102ef0 msgr2=0x7f11f8074a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.020+0000 7f11fdddd700 1 --2- 172.21.15.28:0/2870448662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11f8102ef0 0x7f11f8074a70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f11e000c940 tx=0x7f11e000cc50 comp rx=0 tx=0).stop 2024-04-25T01:31:35.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.020+0000 7f11f7fff700 1 -- 172.21.15.28:0/2870448662 reap_dead start 2024-04-25T01:31:35.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.020+0000 7f11fdddd700 1 -- 172.21.15.28:0/2870448662 shutdown_connections 2024-04-25T01:31:35.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.020+0000 7f11fdddd700 1 -- 172.21.15.28:0/2870448662 >> 172.21.15.28:0/2870448662 conn(0x7f11f8079310 msgr2=0x7f11f81003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:35.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.020+0000 7f11fdddd700 1 -- 172.21.15.28:0/2870448662 shutdown_connections 2024-04-25T01:31:35.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:36.020+0000 7f11fdddd700 1 -- 172.21.15.28:0/2870448662 wait complete. 2024-04-25T01:31:36.011 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:35 smithi028 ceph-mon[29390]: pgmap v330: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s; 5 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T01:31:36.011 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:31:36.011 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:31:36.011 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:31:36.011 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:35 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:31:36.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:35 smithi032 ceph-mon[32461]: pgmap v330: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s; 5 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T01:31:36.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:35 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:31:36.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:35 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:31:36.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:35 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:31:36.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:35 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:31:36.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:35 smithi130 ceph-mon[32702]: pgmap v330: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s; 5 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T01:31:36.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:35 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:31:36.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:35 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:31:36.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:35 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:31:36.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:35 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:31:37.793 INFO:teuthology.orchestra.run.smithi028.stdout:34359738450 2024-04-25T01:31:37.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-25T01:31:37.799 INFO:teuthology.orchestra.run.smithi028.stdout:55834574926 2024-04-25T01:31:37.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-25T01:31:38.027 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:37 smithi028 ceph-mon[29390]: pgmap v331: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s; 5 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T01:31:38.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:37 smithi032 ceph-mon[32461]: pgmap v331: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s; 5 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T01:31:38.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:37 smithi130 ceph-mon[32702]: pgmap v331: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s; 5 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T01:31:38.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.494+0000 7f5f67cea700 1 -- 172.21.15.28:0/2114099465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f60101120 msgr2=0x7f5f601015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:38.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.494+0000 7f5f67cea700 1 --2- 172.21.15.28:0/2114099465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f60101120 0x7f5f601015a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009990 tx=0x7f5f54009ca0 comp rx=0 tx=0).stop 2024-04-25T01:31:38.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.495+0000 7f5f67cea700 1 -- 172.21.15.28:0/2114099465 shutdown_connections 2024-04-25T01:31:38.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.495+0000 7f5f67cea700 1 --2- 172.21.15.28:0/2114099465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f60101ae0 0x7f5f6010dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:38.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.495+0000 7f5f67cea700 1 --2- 172.21.15.28:0/2114099465 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f60107190 0x7f5f60107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:38.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.495+0000 7f5f67cea700 1 --2- 172.21.15.28:0/2114099465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f60101120 0x7f5f601015a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:38.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.495+0000 7f5f67cea700 1 -- 172.21.15.28:0/2114099465 >> 172.21.15.28:0/2114099465 conn(0x7f5f600780a0 msgr2=0x7f5f600784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:38.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.495+0000 7f5f67cea700 1 -- 172.21.15.28:0/2114099465 shutdown_connections 2024-04-25T01:31:38.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.495+0000 7f5f67cea700 1 -- 172.21.15.28:0/2114099465 wait complete. 2024-04-25T01:31:38.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.495+0000 7f5f67cea700 1 Processor -- start 2024-04-25T01:31:38.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.495+0000 7f5f67cea700 1 -- start start 2024-04-25T01:31:38.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f67cea700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f60101120 0x7f5f601997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:38.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f67cea700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f60101ae0 0x7f5f60199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:38.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f67cea700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f60107190 0x7f5f601a4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:38.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f67cea700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5f60111420 con 0x7f5f60101ae0 2024-04-25T01:31:38.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f67cea700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5f601112a0 con 0x7f5f60107190 2024-04-25T01:31:38.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f67cea700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f5f601115a0 con 0x7f5f60101120 2024-04-25T01:31:38.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f65285700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f60101ae0 0x7f5f60199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:38.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f65285700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f60101ae0 0x7f5f60199ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56166/0 (socket says 172.21.15.28:56166) 2024-04-25T01:31:38.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f65285700 1 -- 172.21.15.28:0/1820248269 learned_addr learned my addr 172.21.15.28:0/1820248269 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:38.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f65a86700 1 --2- 172.21.15.28:0/1820248269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f60101120 0x7f5f601997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:38.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f66287700 1 --2- 172.21.15.28:0/1820248269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f60107190 0x7f5f601a4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:38.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f65285700 1 -- 172.21.15.28:0/1820248269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f60101120 msgr2=0x7f5f601997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:38.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f65285700 1 --2- 172.21.15.28:0/1820248269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f60101120 0x7f5f601997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f65285700 1 -- 172.21.15.28:0/1820248269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f60107190 msgr2=0x7f5f601a4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f65285700 1 --2- 172.21.15.28:0/1820248269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f60107190 0x7f5f601a4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f65285700 1 -- 172.21.15.28:0/1820248269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f601a4d60 con 0x7f5f60101ae0 2024-04-25T01:31:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f66287700 1 --2- 172.21.15.28:0/1820248269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f60107190 0x7f5f601a4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f65285700 1 --2- 172.21.15.28:0/1820248269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f60101ae0 0x7f5f60199ce0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5f54000c00 tx=0x7f5f540047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f52ffd700 1 -- 172.21.15.28:0/1820248269 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f54027070 con 0x7f5f60101ae0 2024-04-25T01:31:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f67cea700 1 -- 172.21.15.28:0/1820248269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f601a4f30 con 0x7f5f60101ae0 2024-04-25T01:31:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f52ffd700 1 -- 172.21.15.28:0/1820248269 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f54003780 con 0x7f5f60101ae0 2024-04-25T01:31:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.496+0000 7f5f52ffd700 1 -- 172.21.15.28:0/1820248269 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f54022dd0 con 0x7f5f60101ae0 2024-04-25T01:31:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.497+0000 7f5f67cea700 1 -- 172.21.15.28:0/1820248269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f601a5320 con 0x7f5f60101ae0 2024-04-25T01:31:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.497+0000 7f5f67cea700 1 -- 172.21.15.28:0/1820248269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5f600628c0 con 0x7f5f60101ae0 2024-04-25T01:31:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.499+0000 7f5f52ffd700 1 -- 172.21.15.28:0/1820248269 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5f5402a070 con 0x7f5f60101ae0 2024-04-25T01:31:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.500+0000 7f5f52ffd700 1 --2- 172.21.15.28:0/1820248269 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5f4c0a2cb0 0x7f5f4c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.500+0000 7f5f65a86700 1 --2- 172.21.15.28:0/1820248269 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5f4c0a2cb0 0x7f5f4c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.500+0000 7f5f52ffd700 1 -- 172.21.15.28:0/1820248269 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f5f540d0810 con 0x7f5f60101ae0 2024-04-25T01:31:38.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.500+0000 7f5f52ffd700 1 --2- 172.21.15.28:0/1820248269 >> [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206] conn(0x7f5f4c0ade10 0x7f5f4c0b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:38.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.500+0000 7f5f52ffd700 1 -- 172.21.15.28:0/1820248269 --> [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5f54016e70 con 0x7f5f4c0ade10 2024-04-25T01:31:38.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.500+0000 7f5f52ffd700 1 -- 172.21.15.28:0/1820248269 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f5f540d0c60 con 0x7f5f60101ae0 2024-04-25T01:31:38.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.500+0000 7f5f66287700 1 --2- 172.21.15.28:0/1820248269 >> [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206] conn(0x7f5f4c0ade10 0x7f5f4c0b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:38.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.500+0000 7f5f65a86700 1 --2- 172.21.15.28:0/1820248269 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5f4c0a2cb0 0x7f5f4c0a5170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c0045b0 tx=0x7f5f5c00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:38.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.501+0000 7f5f66287700 1 --2- 172.21.15.28:0/1820248269 >> [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206] conn(0x7f5f4c0ade10 0x7f5f4c0b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:38.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.501+0000 7f5f52ffd700 1 -- 172.21.15.28:0/1820248269 <== osd.9 v2:172.21.15.28:6826/2946375206 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5f54016e70 con 0x7f5f4c0ade10 2024-04-25T01:31:38.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.525+0000 7f5f67cea700 1 -- 172.21.15.28:0/1820248269 --> [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5f60062d40 con 0x7f5f4c0ade10 2024-04-25T01:31:38.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.525+0000 7f5f52ffd700 1 -- 172.21.15.28:0/1820248269 <== osd.9 v2:172.21.15.28:6826/2946375206 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5f60062d40 con 0x7f5f4c0ade10 2024-04-25T01:31:38.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.526+0000 7f5f67cea700 1 -- 172.21.15.28:0/1820248269 >> [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206] conn(0x7f5f4c0ade10 msgr2=0x7f5f4c0b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:38.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.526+0000 7f5f67cea700 1 --2- 172.21.15.28:0/1820248269 >> [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206] conn(0x7f5f4c0ade10 0x7f5f4c0b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:38.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.526+0000 7f5f67cea700 1 -- 172.21.15.28:0/1820248269 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5f4c0a2cb0 msgr2=0x7f5f4c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:38.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.526+0000 7f5f67cea700 1 --2- 172.21.15.28:0/1820248269 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5f4c0a2cb0 0x7f5f4c0a5170 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c0045b0 tx=0x7f5f5c00b3c0 comp rx=0 tx=0).stop 2024-04-25T01:31:38.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.526+0000 7f5f67cea700 1 -- 172.21.15.28:0/1820248269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f60101ae0 msgr2=0x7f5f60199ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:38.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.526+0000 7f5f67cea700 1 --2- 172.21.15.28:0/1820248269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f60101ae0 0x7f5f60199ce0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5f54000c00 tx=0x7f5f540047b0 comp rx=0 tx=0).stop 2024-04-25T01:31:38.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.526+0000 7f5f66287700 1 -- 172.21.15.28:0/1820248269 reap_dead start 2024-04-25T01:31:38.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.526+0000 7f5f67cea700 1 -- 172.21.15.28:0/1820248269 shutdown_connections 2024-04-25T01:31:38.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.526+0000 7f5f67cea700 1 -- 172.21.15.28:0/1820248269 >> 172.21.15.28:0/1820248269 conn(0x7f5f600780a0 msgr2=0x7f5f600ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:38.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.526+0000 7f5f67cea700 1 -- 172.21.15.28:0/1820248269 shutdown_connections 2024-04-25T01:31:38.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.526+0000 7f5f67cea700 1 -- 172.21.15.28:0/1820248269 wait complete. 2024-04-25T01:31:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.694+0000 7f039990d700 1 -- 172.21.15.28:0/3866838069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0394107190 msgr2=0x7f0394107570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.694+0000 7f039990d700 1 --2- 172.21.15.28:0/3866838069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0394107190 0x7f0394107570 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0384009990 tx=0x7f0384009ca0 comp rx=0 tx=0).stop 2024-04-25T01:31:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.695+0000 7f039990d700 1 -- 172.21.15.28:0/3866838069 shutdown_connections 2024-04-25T01:31:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.695+0000 7f039990d700 1 --2- 172.21.15.28:0/3866838069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0394101ae0 0x7f039410dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.695+0000 7f039990d700 1 --2- 172.21.15.28:0/3866838069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0394107190 0x7f0394107570 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.695+0000 7f039990d700 1 --2- 172.21.15.28:0/3866838069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0394101120 0x7f03941015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.695+0000 7f039990d700 1 -- 172.21.15.28:0/3866838069 >> 172.21.15.28:0/3866838069 conn(0x7f03940780a0 msgr2=0x7f03940784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.695+0000 7f039990d700 1 -- 172.21.15.28:0/3866838069 shutdown_connections 2024-04-25T01:31:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.695+0000 7f039990d700 1 -- 172.21.15.28:0/3866838069 wait complete. 2024-04-25T01:31:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.695+0000 7f039990d700 1 Processor -- start 2024-04-25T01:31:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.695+0000 7f039990d700 1 -- start start 2024-04-25T01:31:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.695+0000 7f039990d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0394101120 0x7f0394102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.695+0000 7f039990d700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0394101ae0 0x7f0394103340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.695+0000 7f039990d700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0394107190 0x7f0394103880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f039990d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0394111480 con 0x7f0394101120 2024-04-25T01:31:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f039990d700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0394111300 con 0x7f0394107190 2024-04-25T01:31:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f039990d700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0394111600 con 0x7f0394101ae0 2024-04-25T01:31:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f0392ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0394101120 0x7f0394102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f0392ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0394101120 0x7f0394102e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56176/0 (socket says 172.21.15.28:56176) 2024-04-25T01:31:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f0392ffd700 1 -- 172.21.15.28:0/1180747554 learned_addr learned my addr 172.21.15.28:0/1180747554 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f038bfff700 1 --2- 172.21.15.28:0/1180747554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0394101ae0 0x7f0394103340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f03937fe700 1 --2- 172.21.15.28:0/1180747554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0394107190 0x7f0394103880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f0392ffd700 1 -- 172.21.15.28:0/1180747554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0394101ae0 msgr2=0x7f0394103340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f0392ffd700 1 --2- 172.21.15.28:0/1180747554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0394101ae0 0x7f0394103340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f0392ffd700 1 -- 172.21.15.28:0/1180747554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0394107190 msgr2=0x7f0394103880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f0392ffd700 1 --2- 172.21.15.28:0/1180747554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0394107190 0x7f0394103880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f0392ffd700 1 -- 172.21.15.28:0/1180747554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03941a6f50 con 0x7f0394101120 2024-04-25T01:31:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f038bfff700 1 --2- 172.21.15.28:0/1180747554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0394101ae0 0x7f0394103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f03937fe700 1 --2- 172.21.15.28:0/1180747554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0394107190 0x7f0394103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f0392ffd700 1 --2- 172.21.15.28:0/1180747554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0394101120 0x7f0394102e00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f03840038b0 tx=0x7f0384005590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f039890b700 1 -- 172.21.15.28:0/1180747554 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0384027070 con 0x7f0394101120 2024-04-25T01:31:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f039990d700 1 -- 172.21.15.28:0/1180747554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03941a71e0 con 0x7f0394101120 2024-04-25T01:31:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f039890b700 1 -- 172.21.15.28:0/1180747554 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03840040d0 con 0x7f0394101120 2024-04-25T01:31:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f039890b700 1 -- 172.21.15.28:0/1180747554 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0384022dd0 con 0x7f0394101120 2024-04-25T01:31:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.696+0000 7f039990d700 1 -- 172.21.15.28:0/1180747554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03941a7710 con 0x7f0394101120 2024-04-25T01:31:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.697+0000 7f039990d700 1 -- 172.21.15.28:0/1180747554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f039406aa20 con 0x7f0394101120 2024-04-25T01:31:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.698+0000 7f039890b700 1 -- 172.21.15.28:0/1180747554 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f038402a070 con 0x7f0394101120 2024-04-25T01:31:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.699+0000 7f039890b700 1 --2- 172.21.15.28:0/1180747554 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f03800a2cb0 0x7f03800a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.699+0000 7f038bfff700 1 --2- 172.21.15.28:0/1180747554 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f03800a2cb0 0x7f03800a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.699+0000 7f039890b700 1 -- 172.21.15.28:0/1180747554 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f03840d1060 con 0x7f0394101120 2024-04-25T01:31:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.699+0000 7f039890b700 1 --2- 172.21.15.28:0/1180747554 >> [v2:172.21.15.32:6808/3679896058,v1:172.21.15.32:6809/3679896058] conn(0x7f03800ade10 0x7f03800b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.699+0000 7f039890b700 1 -- 172.21.15.28:0/1180747554 --> [v2:172.21.15.32:6808/3679896058,v1:172.21.15.32:6809/3679896058] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f038400b0b0 con 0x7f03800ade10 2024-04-25T01:31:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.700+0000 7f039890b700 1 -- 172.21.15.28:0/1180747554 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f03840d1410 con 0x7f0394101120 2024-04-25T01:31:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.700+0000 7f038bfff700 1 --2- 172.21.15.28:0/1180747554 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f03800a2cb0 0x7f03800a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f037c005950 tx=0x7f037c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.700+0000 7f03937fe700 1 --2- 172.21.15.28:0/1180747554 >> [v2:172.21.15.32:6808/3679896058,v1:172.21.15.32:6809/3679896058] conn(0x7f03800ade10 0x7f03800b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.700+0000 7f03937fe700 1 --2- 172.21.15.28:0/1180747554 >> [v2:172.21.15.32:6808/3679896058,v1:172.21.15.32:6809/3679896058] conn(0x7f03800ade10 0x7f03800b02b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.701+0000 7f039890b700 1 -- 172.21.15.28:0/1180747554 <== osd.4 v2:172.21.15.32:6808/3679896058 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f038400b0b0 con 0x7f03800ade10 2024-04-25T01:31:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.718+0000 7f039990d700 1 -- 172.21.15.28:0/1180747554 --> [v2:172.21.15.32:6808/3679896058,v1:172.21.15.32:6809/3679896058] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0394102840 con 0x7f03800ade10 2024-04-25T01:31:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.718+0000 7f039890b700 1 -- 172.21.15.28:0/1180747554 <== osd.4 v2:172.21.15.32:6808/3679896058 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0394102840 con 0x7f03800ade10 2024-04-25T01:31:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.719+0000 7f039990d700 1 -- 172.21.15.28:0/1180747554 >> [v2:172.21.15.32:6808/3679896058,v1:172.21.15.32:6809/3679896058] conn(0x7f03800ade10 msgr2=0x7f03800b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.719+0000 7f039990d700 1 --2- 172.21.15.28:0/1180747554 >> [v2:172.21.15.32:6808/3679896058,v1:172.21.15.32:6809/3679896058] conn(0x7f03800ade10 0x7f03800b02b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.719+0000 7f039990d700 1 -- 172.21.15.28:0/1180747554 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f03800a2cb0 msgr2=0x7f03800a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.719+0000 7f039990d700 1 --2- 172.21.15.28:0/1180747554 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f03800a2cb0 0x7f03800a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f037c005950 tx=0x7f037c0058e0 comp rx=0 tx=0).stop 2024-04-25T01:31:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.719+0000 7f039990d700 1 -- 172.21.15.28:0/1180747554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0394101120 msgr2=0x7f0394102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.719+0000 7f039990d700 1 --2- 172.21.15.28:0/1180747554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0394101120 0x7f0394102e00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f03840038b0 tx=0x7f0384005590 comp rx=0 tx=0).stop 2024-04-25T01:31:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.719+0000 7f03937fe700 1 -- 172.21.15.28:0/1180747554 reap_dead start 2024-04-25T01:31:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.719+0000 7f039990d700 1 -- 172.21.15.28:0/1180747554 shutdown_connections 2024-04-25T01:31:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.719+0000 7f039990d700 1 -- 172.21.15.28:0/1180747554 >> 172.21.15.28:0/1180747554 conn(0x7f03940780a0 msgr2=0x7f0394100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.719+0000 7f039990d700 1 -- 172.21.15.28:0/1180747554 shutdown_connections 2024-04-25T01:31:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:38.719+0000 7f039990d700 1 -- 172.21.15.28:0/1180747554 wait complete. 2024-04-25T01:31:40.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:39 smithi028 ceph-mon[29390]: pgmap v332: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T01:31:40.221 INFO:teuthology.orchestra.run.smithi028.stdout:146028888126 2024-04-25T01:31:40.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-25T01:31:40.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:39 smithi032 ceph-mon[32461]: pgmap v332: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T01:31:40.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:39 smithi130 ceph-mon[32702]: pgmap v332: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T01:31:40.987 INFO:teuthology.orchestra.run.smithi028.stdout:111669149764 2024-04-25T01:31:40.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-25T01:31:41.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.540+0000 7fbfabfcc700 1 -- 172.21.15.28:0/3659808554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa4107420 msgr2=0x7fbfa4107800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:41.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.540+0000 7fbfabfcc700 1 --2- 172.21.15.28:0/3659808554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa4107420 0x7fbfa4107800 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbfa000b0d0 tx=0x7fbfa000b3e0 comp rx=0 tx=0).stop 2024-04-25T01:31:41.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.540+0000 7fbfabfcc700 1 -- 172.21.15.28:0/3659808554 shutdown_connections 2024-04-25T01:31:41.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.540+0000 7fbfabfcc700 1 --2- 172.21.15.28:0/3659808554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfa41026c0 0x7fbfa410e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:41.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.540+0000 7fbfabfcc700 1 --2- 172.21.15.28:0/3659808554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa4107420 0x7fbfa4107800 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:41.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.540+0000 7fbfabfcc700 1 --2- 172.21.15.28:0/3659808554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbfa4101d00 0x7fbfa4102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:41.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.540+0000 7fbfabfcc700 1 -- 172.21.15.28:0/3659808554 >> 172.21.15.28:0/3659808554 conn(0x7fbfa4076450 msgr2=0x7fbfa4076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:41.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.540+0000 7fbfabfcc700 1 -- 172.21.15.28:0/3659808554 shutdown_connections 2024-04-25T01:31:41.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.540+0000 7fbfabfcc700 1 -- 172.21.15.28:0/3659808554 wait complete. 2024-04-25T01:31:41.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.540+0000 7fbfabfcc700 1 Processor -- start 2024-04-25T01:31:41.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.540+0000 7fbfabfcc700 1 -- start start 2024-04-25T01:31:41.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.541+0000 7fbfabfcc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa4101d00 0x7fbfa41a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:41.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.541+0000 7fbfabfcc700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfa41026c0 0x7fbfa41a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:41.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.541+0000 7fbfabfcc700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbfa4107420 0x7fbfa419a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:41.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.541+0000 7fbfabfcc700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa4111570 con 0x7fbfa4101d00 2024-04-25T01:31:41.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.541+0000 7fbfabfcc700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa41113f0 con 0x7fbfa41026c0 2024-04-25T01:31:41.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.541+0000 7fbfabfcc700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa41116f0 con 0x7fbfa4107420 2024-04-25T01:31:41.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.541+0000 7fbfa9d68700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa4101d00 0x7fbfa41a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:41.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.541+0000 7fbfa9d68700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa4101d00 0x7fbfa41a0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56188/0 (socket says 172.21.15.28:56188) 2024-04-25T01:31:41.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.541+0000 7fbfa9d68700 1 -- 172.21.15.28:0/2900160981 learned_addr learned my addr 172.21.15.28:0/2900160981 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:41.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.541+0000 7fbfa9567700 1 --2- 172.21.15.28:0/2900160981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfa41026c0 0x7fbfa41a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:41.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.541+0000 7fbfaa569700 1 --2- 172.21.15.28:0/2900160981 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbfa4107420 0x7fbfa419a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:41.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.541+0000 7fbfa9d68700 1 -- 172.21.15.28:0/2900160981 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbfa4107420 msgr2=0x7fbfa419a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:41.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.541+0000 7fbfa9d68700 1 --2- 172.21.15.28:0/2900160981 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbfa4107420 0x7fbfa419a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:41.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.541+0000 7fbfa9d68700 1 -- 172.21.15.28:0/2900160981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfa41026c0 msgr2=0x7fbfa41a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:41.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.541+0000 7fbfa9d68700 1 --2- 172.21.15.28:0/2900160981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbfa41026c0 0x7fbfa41a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:41.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.541+0000 7fbfa9d68700 1 -- 172.21.15.28:0/2900160981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa419b120 con 0x7fbfa4101d00 2024-04-25T01:31:41.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.541+0000 7fbfa9d68700 1 --2- 172.21.15.28:0/2900160981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa4101d00 0x7fbfa41a0800 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fbfa000b820 tx=0x7fbfa0004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:41.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.541+0000 7fbf9affd700 1 -- 172.21.15.28:0/2900160981 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0021020 con 0x7fbfa4101d00 2024-04-25T01:31:41.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.542+0000 7fbfabfcc700 1 -- 172.21.15.28:0/2900160981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa419b3b0 con 0x7fbfa4101d00 2024-04-25T01:31:41.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.542+0000 7fbf9affd700 1 -- 172.21.15.28:0/2900160981 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfa0007c80 con 0x7fbfa4101d00 2024-04-25T01:31:41.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.542+0000 7fbf9affd700 1 -- 172.21.15.28:0/2900160981 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0005470 con 0x7fbfa4101d00 2024-04-25T01:31:41.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.542+0000 7fbfabfcc700 1 -- 172.21.15.28:0/2900160981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa41a76d0 con 0x7fbfa4101d00 2024-04-25T01:31:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.542+0000 7fbfabfcc700 1 -- 172.21.15.28:0/2900160981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbfa410ce50 con 0x7fbfa4101d00 2024-04-25T01:31:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.544+0000 7fbf9affd700 1 -- 172.21.15.28:0/2900160981 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fbfa0024020 con 0x7fbfa4101d00 2024-04-25T01:31:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.544+0000 7fbf9affd700 1 --2- 172.21.15.28:0/2900160981 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbf900a2d30 0x7fbf900a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.544+0000 7fbfa9567700 1 --2- 172.21.15.28:0/2900160981 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbf900a2d30 0x7fbf900a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.544+0000 7fbf9affd700 1 -- 172.21.15.28:0/2900160981 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fbfa000f020 con 0x7fbfa4101d00 2024-04-25T01:31:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.545+0000 7fbf9affd700 1 --2- 172.21.15.28:0/2900160981 >> [v2:172.21.15.130:6816/2911117678,v1:172.21.15.130:6817/2911117678] conn(0x7fbf900ade90 0x7fbf900b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.545+0000 7fbf9affd700 1 -- 172.21.15.28:0/2900160981 --> [v2:172.21.15.130:6816/2911117678,v1:172.21.15.130:6817/2911117678] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbfa002fe50 con 0x7fbf900ade90 2024-04-25T01:31:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.545+0000 7fbf9affd700 1 -- 172.21.15.28:0/2900160981 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fbfa00d1220 con 0x7fbfa4101d00 2024-04-25T01:31:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.545+0000 7fbfa9567700 1 --2- 172.21.15.28:0/2900160981 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbf900a2d30 0x7fbf900a51f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4076870 tx=0x7fbf9400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.545+0000 7fbfaa569700 1 --2- 172.21.15.28:0/2900160981 >> [v2:172.21.15.130:6816/2911117678,v1:172.21.15.130:6817/2911117678] conn(0x7fbf900ade90 0x7fbf900b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.545+0000 7fbfaa569700 1 --2- 172.21.15.28:0/2900160981 >> [v2:172.21.15.130:6816/2911117678,v1:172.21.15.130:6817/2911117678] conn(0x7fbf900ade90 0x7fbf900b0330 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.546+0000 7fbf9affd700 1 -- 172.21.15.28:0/2900160981 <== osd.8 v2:172.21.15.130:6816/2911117678 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbfa002fe50 con 0x7fbf900ade90 2024-04-25T01:31:41.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.562+0000 7fbfabfcc700 1 -- 172.21.15.28:0/2900160981 --> [v2:172.21.15.130:6816/2911117678,v1:172.21.15.130:6817/2911117678] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbfa406a520 con 0x7fbf900ade90 2024-04-25T01:31:41.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.563+0000 7fbf9affd700 1 -- 172.21.15.28:0/2900160981 <== osd.8 v2:172.21.15.130:6816/2911117678 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbfa406a520 con 0x7fbf900ade90 2024-04-25T01:31:41.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.563+0000 7fbfabfcc700 1 -- 172.21.15.28:0/2900160981 >> [v2:172.21.15.130:6816/2911117678,v1:172.21.15.130:6817/2911117678] conn(0x7fbf900ade90 msgr2=0x7fbf900b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.563+0000 7fbfabfcc700 1 --2- 172.21.15.28:0/2900160981 >> [v2:172.21.15.130:6816/2911117678,v1:172.21.15.130:6817/2911117678] conn(0x7fbf900ade90 0x7fbf900b0330 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.563+0000 7fbfabfcc700 1 -- 172.21.15.28:0/2900160981 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbf900a2d30 msgr2=0x7fbf900a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.563+0000 7fbfabfcc700 1 --2- 172.21.15.28:0/2900160981 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbf900a2d30 0x7fbf900a51f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4076870 tx=0x7fbf9400a380 comp rx=0 tx=0).stop 2024-04-25T01:31:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.563+0000 7fbfabfcc700 1 -- 172.21.15.28:0/2900160981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa4101d00 msgr2=0x7fbfa41a0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.563+0000 7fbfabfcc700 1 --2- 172.21.15.28:0/2900160981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa4101d00 0x7fbfa41a0800 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fbfa000b820 tx=0x7fbfa0004e20 comp rx=0 tx=0).stop 2024-04-25T01:31:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.563+0000 7fbfaa569700 1 -- 172.21.15.28:0/2900160981 reap_dead start 2024-04-25T01:31:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.563+0000 7fbfabfcc700 1 -- 172.21.15.28:0/2900160981 shutdown_connections 2024-04-25T01:31:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.563+0000 7fbfabfcc700 1 -- 172.21.15.28:0/2900160981 >> 172.21.15.28:0/2900160981 conn(0x7fbfa4076450 msgr2=0x7fbfa410c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.563+0000 7fbfabfcc700 1 -- 172.21.15.28:0/2900160981 shutdown_connections 2024-04-25T01:31:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:41.563+0000 7fbfabfcc700 1 -- 172.21.15.28:0/2900160981 wait complete. 2024-04-25T01:31:41.786 INFO:teuthology.orchestra.run.smithi028.stdout:171798691898 2024-04-25T01:31:41.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-25T01:31:42.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:41 smithi028 ceph-mon[29390]: pgmap v333: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T01:31:42.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:41 smithi032 ceph-mon[32461]: pgmap v333: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T01:31:42.428 INFO:teuthology.orchestra.run.smithi028.stdout:261993005099 2024-04-25T01:31:42.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-25T01:31:42.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:41 smithi130 ceph-mon[32702]: pgmap v333: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T01:31:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.871+0000 7fb43c689700 1 -- 172.21.15.28:0/2467189207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb434101130 msgr2=0x7fb4341015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.871+0000 7fb43c689700 1 --2- 172.21.15.28:0/2467189207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb434101130 0x7fb4341015b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb424009990 tx=0x7fb424009ca0 comp rx=0 tx=0).stop 2024-04-25T01:31:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.871+0000 7fb43c689700 1 -- 172.21.15.28:0/2467189207 shutdown_connections 2024-04-25T01:31:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.871+0000 7fb43c689700 1 --2- 172.21.15.28:0/2467189207 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb434101af0 0x7fb43410dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.871+0000 7fb43c689700 1 --2- 172.21.15.28:0/2467189207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb4341071a0 0x7fb434107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.871+0000 7fb43c689700 1 --2- 172.21.15.28:0/2467189207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb434101130 0x7fb4341015b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.871+0000 7fb43c689700 1 -- 172.21.15.28:0/2467189207 >> 172.21.15.28:0/2467189207 conn(0x7fb4340780e0 msgr2=0x7fb4340784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.871+0000 7fb43c689700 1 -- 172.21.15.28:0/2467189207 shutdown_connections 2024-04-25T01:31:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.871+0000 7fb43c689700 1 -- 172.21.15.28:0/2467189207 wait complete. 2024-04-25T01:31:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.872+0000 7fb43c689700 1 Processor -- start 2024-04-25T01:31:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.872+0000 7fb43c689700 1 -- start start 2024-04-25T01:31:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.872+0000 7fb43c689700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb434101130 0x7fb434102e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.872+0000 7fb43c689700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb434101af0 0x7fb434103350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.872+0000 7fb43c689700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb4341071a0 0x7fb434103890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.872+0000 7fb43c689700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb434111460 con 0x7fb434101130 2024-04-25T01:31:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.872+0000 7fb43c689700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb4341112e0 con 0x7fb434101af0 2024-04-25T01:31:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.872+0000 7fb43c689700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb4341115e0 con 0x7fb4341071a0 2024-04-25T01:31:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.872+0000 7fb43a425700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb434101130 0x7fb434102e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.872+0000 7fb43a425700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb434101130 0x7fb434102e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56208/0 (socket says 172.21.15.28:56208) 2024-04-25T01:31:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.872+0000 7fb43a425700 1 -- 172.21.15.28:0/820566976 learned_addr learned my addr 172.21.15.28:0/820566976 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.872+0000 7fb439c24700 1 --2- 172.21.15.28:0/820566976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb434101af0 0x7fb434103350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.872+0000 7fb43a425700 1 -- 172.21.15.28:0/820566976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb4341071a0 msgr2=0x7fb434103890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.872+0000 7fb43ac26700 1 --2- 172.21.15.28:0/820566976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb4341071a0 0x7fb434103890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.872+0000 7fb43a425700 1 --2- 172.21.15.28:0/820566976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb4341071a0 0x7fb434103890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.872+0000 7fb43a425700 1 -- 172.21.15.28:0/820566976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb434101af0 msgr2=0x7fb434103350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.872+0000 7fb43a425700 1 --2- 172.21.15.28:0/820566976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb434101af0 0x7fb434103350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.872+0000 7fb43a425700 1 -- 172.21.15.28:0/820566976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4341a6f30 con 0x7fb434101130 2024-04-25T01:31:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.873+0000 7fb43a425700 1 --2- 172.21.15.28:0/820566976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb434101130 0x7fb434102e10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb43000d950 tx=0x7fb43000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.873+0000 7fb42b7fe700 1 -- 172.21.15.28:0/820566976 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb430016070 con 0x7fb434101130 2024-04-25T01:31:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.873+0000 7fb42b7fe700 1 -- 172.21.15.28:0/820566976 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb430009e20 con 0x7fb434101130 2024-04-25T01:31:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.873+0000 7fb43c689700 1 -- 172.21.15.28:0/820566976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4341a7220 con 0x7fb434101130 2024-04-25T01:31:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.873+0000 7fb42b7fe700 1 -- 172.21.15.28:0/820566976 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb43000f620 con 0x7fb434101130 2024-04-25T01:31:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.873+0000 7fb43c689700 1 -- 172.21.15.28:0/820566976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4341a77b0 con 0x7fb434101130 2024-04-25T01:31:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.874+0000 7fb43c689700 1 -- 172.21.15.28:0/820566976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb434105200 con 0x7fb434101130 2024-04-25T01:31:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.875+0000 7fb42b7fe700 1 -- 172.21.15.28:0/820566976 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb430019070 con 0x7fb434101130 2024-04-25T01:31:42.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.876+0000 7fb42b7fe700 1 --2- 172.21.15.28:0/820566976 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb4200a2d30 0x7fb4200a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:42.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.876+0000 7fb439c24700 1 --2- 172.21.15.28:0/820566976 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb4200a2d30 0x7fb4200a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:42.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.876+0000 7fb42b7fe700 1 -- 172.21.15.28:0/820566976 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fb4300cf510 con 0x7fb434101130 2024-04-25T01:31:42.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.877+0000 7fb42b7fe700 1 --2- 172.21.15.28:0/820566976 >> [v2:172.21.15.32:6824/13541811,v1:172.21.15.32:6825/13541811] conn(0x7fb4200ade90 0x7fb4200b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:42.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.877+0000 7fb42b7fe700 1 -- 172.21.15.28:0/820566976 --> [v2:172.21.15.32:6824/13541811,v1:172.21.15.32:6825/13541811] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb430010e50 con 0x7fb4200ade90 2024-04-25T01:31:42.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.877+0000 7fb42b7fe700 1 -- 172.21.15.28:0/820566976 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fb4300cf950 con 0x7fb434101130 2024-04-25T01:31:42.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.877+0000 7fb439c24700 1 --2- 172.21.15.28:0/820566976 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb4200a2d30 0x7fb4200a51f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb4340786e0 tx=0x7fb424005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:42.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.877+0000 7fb43ac26700 1 --2- 172.21.15.28:0/820566976 >> [v2:172.21.15.32:6824/13541811,v1:172.21.15.32:6825/13541811] conn(0x7fb4200ade90 0x7fb4200b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:42.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.877+0000 7fb43ac26700 1 --2- 172.21.15.28:0/820566976 >> [v2:172.21.15.32:6824/13541811,v1:172.21.15.32:6825/13541811] conn(0x7fb4200ade90 0x7fb4200b0330 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:42.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.878+0000 7fb42b7fe700 1 -- 172.21.15.28:0/820566976 <== osd.10 v2:172.21.15.32:6824/13541811 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb430010e50 con 0x7fb4200ade90 2024-04-25T01:31:42.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.895+0000 7fb43c689700 1 -- 172.21.15.28:0/820566976 --> [v2:172.21.15.32:6824/13541811,v1:172.21.15.32:6825/13541811] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb434102850 con 0x7fb4200ade90 2024-04-25T01:31:42.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.895+0000 7fb42b7fe700 1 -- 172.21.15.28:0/820566976 <== osd.10 v2:172.21.15.32:6824/13541811 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb434102850 con 0x7fb4200ade90 2024-04-25T01:31:42.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.896+0000 7fb43c689700 1 -- 172.21.15.28:0/820566976 >> [v2:172.21.15.32:6824/13541811,v1:172.21.15.32:6825/13541811] conn(0x7fb4200ade90 msgr2=0x7fb4200b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:42.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.896+0000 7fb43c689700 1 --2- 172.21.15.28:0/820566976 >> [v2:172.21.15.32:6824/13541811,v1:172.21.15.32:6825/13541811] conn(0x7fb4200ade90 0x7fb4200b0330 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:42.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.896+0000 7fb43c689700 1 -- 172.21.15.28:0/820566976 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb4200a2d30 msgr2=0x7fb4200a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:42.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.896+0000 7fb43c689700 1 --2- 172.21.15.28:0/820566976 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb4200a2d30 0x7fb4200a51f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb4340786e0 tx=0x7fb424005fd0 comp rx=0 tx=0).stop 2024-04-25T01:31:42.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.896+0000 7fb43c689700 1 -- 172.21.15.28:0/820566976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb434101130 msgr2=0x7fb434102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:42.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.896+0000 7fb43c689700 1 --2- 172.21.15.28:0/820566976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb434101130 0x7fb434102e10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb43000d950 tx=0x7fb43000dc60 comp rx=0 tx=0).stop 2024-04-25T01:31:42.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.896+0000 7fb43ac26700 1 -- 172.21.15.28:0/820566976 reap_dead start 2024-04-25T01:31:42.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.896+0000 7fb43c689700 1 -- 172.21.15.28:0/820566976 shutdown_connections 2024-04-25T01:31:42.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.896+0000 7fb43c689700 1 -- 172.21.15.28:0/820566976 >> 172.21.15.28:0/820566976 conn(0x7fb4340780e0 msgr2=0x7fb434100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:42.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.896+0000 7fb43c689700 1 -- 172.21.15.28:0/820566976 shutdown_connections 2024-04-25T01:31:42.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:42.896+0000 7fb43c689700 1 -- 172.21.15.28:0/820566976 wait complete. 2024-04-25T01:31:43.586 INFO:teuthology.orchestra.run.smithi028.stdout:317827579936 2024-04-25T01:31:43.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-25T01:31:44.057 INFO:teuthology.orchestra.run.smithi028.stdout:296352743462 2024-04-25T01:31:44.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-25T01:31:44.063 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:44.064 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:44.071 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:44.072 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:44.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:43 smithi032 ceph-mon[32461]: pgmap v334: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:31:44.237 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:43 smithi028 ceph-mon[29390]: pgmap v334: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:31:44.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:43 smithi130 ceph-mon[32702]: pgmap v334: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:31:46.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:46 smithi032 ceph-mon[32461]: pgmap v335: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:31:46.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:46 smithi028 ceph-mon[29390]: pgmap v335: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:31:46.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:46 smithi130 ceph-mon[32702]: pgmap v335: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:31:46.796 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:47.635 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:47.636 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:47.646 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:47.647 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:47.647 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:31:48.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:48 smithi032 ceph-mon[32461]: pgmap v336: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:31:48.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:48 smithi028 ceph-mon[29390]: pgmap v336: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:31:48.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:48 smithi130 ceph-mon[32702]: pgmap v336: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:31:49.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.715+0000 7f439a77b700 1 -- 172.21.15.28:0/1041840715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43941064c0 msgr2=0x7f439410e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:49.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.715+0000 7f439a77b700 1 --2- 172.21.15.28:0/1041840715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43941064c0 0x7f439410e060 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f438c00b0d0 tx=0x7f438c00b3e0 comp rx=0 tx=0).stop 2024-04-25T01:31:49.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.715+0000 7f439a77b700 1 -- 172.21.15.28:0/1041840715 shutdown_connections 2024-04-25T01:31:49.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.715+0000 7f439a77b700 1 --2- 172.21.15.28:0/1041840715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43941064c0 0x7f439410e060 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:49.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.715+0000 7f439a77b700 1 --2- 172.21.15.28:0/1041840715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4394101770 0x7f4394101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:49.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.715+0000 7f439a77b700 1 --2- 172.21.15.28:0/1041840715 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4394102120 0x7f4394105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:49.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.715+0000 7f439a77b700 1 -- 172.21.15.28:0/1041840715 >> 172.21.15.28:0/1041840715 conn(0x7f43940780e0 msgr2=0x7f43940784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:49.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.715+0000 7f439a77b700 1 -- 172.21.15.28:0/1041840715 shutdown_connections 2024-04-25T01:31:49.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.715+0000 7f439a77b700 1 -- 172.21.15.28:0/1041840715 wait complete. 2024-04-25T01:31:49.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.716+0000 7f439a77b700 1 Processor -- start 2024-04-25T01:31:49.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.716+0000 7f439a77b700 1 -- start start 2024-04-25T01:31:49.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.716+0000 7f439a77b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4394101770 0x7f43941997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:49.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.716+0000 7f439a77b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4394102120 0x7f4394199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:49.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.716+0000 7f439a77b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43941064c0 0x7f43941a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:49.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.717+0000 7f439a77b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4394111480 con 0x7f4394101770 2024-04-25T01:31:49.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.717+0000 7f439a77b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4394111300 con 0x7f43941064c0 2024-04-25T01:31:49.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.717+0000 7f439a77b700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4394111600 con 0x7f4394102120 2024-04-25T01:31:49.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.717+0000 7f4393fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4394101770 0x7f43941997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:49.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.717+0000 7f4393fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4394101770 0x7f43941997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38644/0 (socket says 172.21.15.28:38644) 2024-04-25T01:31:49.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.717+0000 7f4393fff700 1 -- 172.21.15.28:0/1861144245 learned_addr learned my addr 172.21.15.28:0/1861144245 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:49.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.717+0000 7f43937fe700 1 --2- 172.21.15.28:0/1861144245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4394102120 0x7f4394199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:49.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.717+0000 7f4398d18700 1 --2- 172.21.15.28:0/1861144245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43941064c0 0x7f43941a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:49.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.717+0000 7f4393fff700 1 -- 172.21.15.28:0/1861144245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4394102120 msgr2=0x7f4394199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:49.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.717+0000 7f4393fff700 1 --2- 172.21.15.28:0/1861144245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4394102120 0x7f4394199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:49.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.717+0000 7f4393fff700 1 -- 172.21.15.28:0/1861144245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43941064c0 msgr2=0x7f43941a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:49.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.717+0000 7f4393fff700 1 --2- 172.21.15.28:0/1861144245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43941064c0 0x7f43941a4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:49.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.717+0000 7f4393fff700 1 -- 172.21.15.28:0/1861144245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43941a4da0 con 0x7f4394101770 2024-04-25T01:31:49.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.717+0000 7f4393fff700 1 --2- 172.21.15.28:0/1861144245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4394101770 0x7f43941997e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f438400d950 tx=0x7f438400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:49.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.717+0000 7f43917fa700 1 -- 172.21.15.28:0/1861144245 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4384016070 con 0x7f4394101770 2024-04-25T01:31:49.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.717+0000 7f439a77b700 1 -- 172.21.15.28:0/1861144245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43941a4fd0 con 0x7f4394101770 2024-04-25T01:31:49.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.717+0000 7f43917fa700 1 -- 172.21.15.28:0/1861144245 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f438400fd70 con 0x7f4394101770 2024-04-25T01:31:49.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.717+0000 7f43917fa700 1 -- 172.21.15.28:0/1861144245 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438400e610 con 0x7f4394101770 2024-04-25T01:31:49.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.718+0000 7f439a77b700 1 -- 172.21.15.28:0/1861144245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43941a54d0 con 0x7f4394101770 2024-04-25T01:31:49.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.718+0000 7f439a77b700 1 -- 172.21.15.28:0/1861144245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4394102870 con 0x7f4394101770 2024-04-25T01:31:49.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.720+0000 7f43917fa700 1 -- 172.21.15.28:0/1861144245 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f4384019070 con 0x7f4394101770 2024-04-25T01:31:49.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.720+0000 7f43917fa700 1 --2- 172.21.15.28:0/1861144245 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f437c0a2cb0 0x7f437c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:49.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.721+0000 7f43937fe700 1 --2- 172.21.15.28:0/1861144245 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f437c0a2cb0 0x7f437c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:49.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.721+0000 7f43917fa700 1 -- 172.21.15.28:0/1861144245 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f43840ceff0 con 0x7f4394101770 2024-04-25T01:31:49.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.721+0000 7f43937fe700 1 --2- 172.21.15.28:0/1861144245 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f437c0a2cb0 0x7f437c0a5170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4394107910 tx=0x7f438800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:49.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.724+0000 7f43917fa700 1 -- 172.21.15.28:0/1861144245 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f438409ba90 con 0x7f4394101770 2024-04-25T01:31:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.932+0000 7f439a77b700 1 -- 172.21.15.28:0/1861144245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f439402db30 con 0x7f4394101770 2024-04-25T01:31:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.932+0000 7f43917fa700 1 -- 172.21.15.28:0/1861144245 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f438409b390 con 0x7f4394101770 2024-04-25T01:31:49.911 INFO:teuthology.orchestra.run.smithi028.stdout:352187318300 2024-04-25T01:31:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.934+0000 7f439a77b700 1 -- 172.21.15.28:0/1861144245 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f437c0a2cb0 msgr2=0x7f437c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:49.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.934+0000 7f439a77b700 1 --2- 172.21.15.28:0/1861144245 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f437c0a2cb0 0x7f437c0a5170 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4394107910 tx=0x7f438800b410 comp rx=0 tx=0).stop 2024-04-25T01:31:49.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.934+0000 7f439a77b700 1 -- 172.21.15.28:0/1861144245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4394101770 msgr2=0x7f43941997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:49.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.934+0000 7f439a77b700 1 --2- 172.21.15.28:0/1861144245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4394101770 0x7f43941997e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f438400d950 tx=0x7f438400dc60 comp rx=0 tx=0).stop 2024-04-25T01:31:49.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.935+0000 7f439a77b700 1 -- 172.21.15.28:0/1861144245 shutdown_connections 2024-04-25T01:31:49.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.935+0000 7f439a77b700 1 --2- 172.21.15.28:0/1861144245 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f437c0a2cb0 0x7f437c0a5170 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:49.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.935+0000 7f439a77b700 1 --2- 172.21.15.28:0/1861144245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43941064c0 0x7f43941a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:49.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.935+0000 7f439a77b700 1 --2- 172.21.15.28:0/1861144245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4394101770 0x7f43941997e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:49.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.935+0000 7f439a77b700 1 --2- 172.21.15.28:0/1861144245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4394102120 0x7f4394199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:49.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.935+0000 7f439a77b700 1 -- 172.21.15.28:0/1861144245 >> 172.21.15.28:0/1861144245 conn(0x7f43940780e0 msgr2=0x7f43940ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:49.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.935+0000 7f439a77b700 1 -- 172.21.15.28:0/1861144245 shutdown_connections 2024-04-25T01:31:49.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:49.935+0000 7f439a77b700 1 -- 172.21.15.28:0/1861144245 wait complete. 2024-04-25T01:31:50.210 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:50 smithi028 ceph-mon[29390]: pgmap v337: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:31:50.210 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:50 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1861144245' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T01:31:50.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:50 smithi032 ceph-mon[32461]: pgmap v337: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:31:50.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:50 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1861144245' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T01:31:50.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:50 smithi130 ceph-mon[32702]: pgmap v337: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:31:50.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:50 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1861144245' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T01:31:51.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.057+0000 7fa64304a700 1 -- 172.21.15.28:0/1675221354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa63c06b300 msgr2=0x7fa63c10f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:51.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.057+0000 7fa64304a700 1 --2- 172.21.15.28:0/1675221354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa63c06b300 0x7fa63c10f0f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa63400b0a0 tx=0x7fa63400b3b0 comp rx=0 tx=0).stop 2024-04-25T01:31:51.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.057+0000 7fa64304a700 1 -- 172.21.15.28:0/1675221354 shutdown_connections 2024-04-25T01:31:51.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.057+0000 7fa64304a700 1 --2- 172.21.15.28:0/1675221354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa63c06b300 0x7fa63c10f0f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:51.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.057+0000 7fa64304a700 1 --2- 172.21.15.28:0/1675221354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa63c1082b0 0x7fa63c108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:51.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.057+0000 7fa64304a700 1 --2- 172.21.15.28:0/1675221354 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa63c06a960 0x7fa63c06adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:51.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.057+0000 7fa64304a700 1 -- 172.21.15.28:0/1675221354 >> 172.21.15.28:0/1675221354 conn(0x7fa63c0780e0 msgr2=0x7fa63c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:51.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.057+0000 7fa64304a700 1 -- 172.21.15.28:0/1675221354 shutdown_connections 2024-04-25T01:31:51.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.057+0000 7fa64304a700 1 -- 172.21.15.28:0/1675221354 wait complete. 2024-04-25T01:31:51.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa64304a700 1 Processor -- start 2024-04-25T01:31:51.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa64304a700 1 -- start start 2024-04-25T01:31:51.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa64304a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa63c06a960 0x7fa63c1a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:51.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa64304a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa63c06b300 0x7fa63c1a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:51.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa64304a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa63c1082b0 0x7fa63c19ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:51.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa64304a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa63c112500 con 0x7fa63c06a960 2024-04-25T01:31:51.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa64304a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa63c112380 con 0x7fa63c1082b0 2024-04-25T01:31:51.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa64304a700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa63c112680 con 0x7fa63c06b300 2024-04-25T01:31:51.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa640de6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa63c06a960 0x7fa63c1a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:51.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa640de6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa63c06a960 0x7fa63c1a0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38652/0 (socket says 172.21.15.28:38652) 2024-04-25T01:31:51.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa640de6700 1 -- 172.21.15.28:0/866591013 learned_addr learned my addr 172.21.15.28:0/866591013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:51.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa63bfff700 1 --2- 172.21.15.28:0/866591013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa63c06b300 0x7fa63c1a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:51.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa6415e7700 1 --2- 172.21.15.28:0/866591013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa63c1082b0 0x7fa63c19ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:51.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa640de6700 1 -- 172.21.15.28:0/866591013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa63c06b300 msgr2=0x7fa63c1a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:51.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa640de6700 1 --2- 172.21.15.28:0/866591013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa63c06b300 0x7fa63c1a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:51.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa640de6700 1 -- 172.21.15.28:0/866591013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa63c1082b0 msgr2=0x7fa63c19ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:51.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa640de6700 1 --2- 172.21.15.28:0/866591013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa63c1082b0 0x7fa63c19ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:51.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa640de6700 1 -- 172.21.15.28:0/866591013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63c19b400 con 0x7fa63c06a960 2024-04-25T01:31:51.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa6415e7700 1 --2- 172.21.15.28:0/866591013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa63c1082b0 0x7fa63c19ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:31:51.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.058+0000 7fa63bfff700 1 --2- 172.21.15.28:0/866591013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa63c06b300 0x7fa63c1a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:51.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.059+0000 7fa640de6700 1 --2- 172.21.15.28:0/866591013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa63c06a960 0x7fa63c1a0a50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa63000d950 tx=0x7fa63000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:51.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.059+0000 7fa639ffb700 1 -- 172.21.15.28:0/866591013 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa630016070 con 0x7fa63c06a960 2024-04-25T01:31:51.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.059+0000 7fa639ffb700 1 -- 172.21.15.28:0/866591013 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa630009b40 con 0x7fa63c06a960 2024-04-25T01:31:51.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.059+0000 7fa64304a700 1 -- 172.21.15.28:0/866591013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa63c19b6f0 con 0x7fa63c06a960 2024-04-25T01:31:51.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.059+0000 7fa639ffb700 1 -- 172.21.15.28:0/866591013 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63000f620 con 0x7fa63c06a960 2024-04-25T01:31:51.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.059+0000 7fa64304a700 1 -- 172.21.15.28:0/866591013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa63c1a79d0 con 0x7fa63c06a960 2024-04-25T01:31:51.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.060+0000 7fa64304a700 1 -- 172.21.15.28:0/866591013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa63c103960 con 0x7fa63c06a960 2024-04-25T01:31:51.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.061+0000 7fa639ffb700 1 -- 172.21.15.28:0/866591013 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa630019070 con 0x7fa63c06a960 2024-04-25T01:31:51.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.062+0000 7fa639ffb700 1 --2- 172.21.15.28:0/866591013 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa62c0a2cb0 0x7fa62c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:51.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.062+0000 7fa639ffb700 1 -- 172.21.15.28:0/866591013 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fa6300cf4f0 con 0x7fa63c06a960 2024-04-25T01:31:51.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.062+0000 7fa63bfff700 1 --2- 172.21.15.28:0/866591013 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa62c0a2cb0 0x7fa62c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:51.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.062+0000 7fa63bfff700 1 --2- 172.21.15.28:0/866591013 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa62c0a2cb0 0x7fa62c0a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa63c19c270 tx=0x7fa628009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:51.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.066+0000 7fa639ffb700 1 -- 172.21.15.28:0/866591013 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa63009bf90 con 0x7fa63c06a960 2024-04-25T01:31:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.283+0000 7fa64304a700 1 -- 172.21.15.28:0/866591013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fa63c19bfe0 con 0x7fa63c06a960 2024-04-25T01:31:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.283+0000 7fa639ffb700 1 -- 172.21.15.28:0/866591013 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa630084e20 con 0x7fa63c06a960 2024-04-25T01:31:51.261 INFO:teuthology.orchestra.run.smithi028.stdout:77309411404 2024-04-25T01:31:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.285+0000 7fa64304a700 1 -- 172.21.15.28:0/866591013 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa62c0a2cb0 msgr2=0x7fa62c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.285+0000 7fa64304a700 1 --2- 172.21.15.28:0/866591013 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa62c0a2cb0 0x7fa62c0a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa63c19c270 tx=0x7fa628009260 comp rx=0 tx=0).stop 2024-04-25T01:31:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.285+0000 7fa64304a700 1 -- 172.21.15.28:0/866591013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa63c06a960 msgr2=0x7fa63c1a0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.285+0000 7fa64304a700 1 --2- 172.21.15.28:0/866591013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa63c06a960 0x7fa63c1a0a50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa63000d950 tx=0x7fa63000dd10 comp rx=0 tx=0).stop 2024-04-25T01:31:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.285+0000 7fa64304a700 1 -- 172.21.15.28:0/866591013 shutdown_connections 2024-04-25T01:31:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.285+0000 7fa64304a700 1 --2- 172.21.15.28:0/866591013 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa62c0a2cb0 0x7fa62c0a5170 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.285+0000 7fa64304a700 1 --2- 172.21.15.28:0/866591013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa63c1082b0 0x7fa63c19ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.285+0000 7fa64304a700 1 --2- 172.21.15.28:0/866591013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa63c06a960 0x7fa63c1a0a50 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.285+0000 7fa64304a700 1 --2- 172.21.15.28:0/866591013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa63c06b300 0x7fa63c1a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.285+0000 7fa64304a700 1 -- 172.21.15.28:0/866591013 >> 172.21.15.28:0/866591013 conn(0x7fa63c0780e0 msgr2=0x7fa63c10d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.285+0000 7fa64304a700 1 -- 172.21.15.28:0/866591013 shutdown_connections 2024-04-25T01:31:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:51.285+0000 7fa64304a700 1 -- 172.21.15.28:0/866591013 wait complete. 2024-04-25T01:31:52.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:52 smithi028 ceph-mon[29390]: pgmap v338: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:31:52.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:52 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/866591013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T01:31:52.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:52 smithi130 ceph-mon[32702]: pgmap v338: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:31:52.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:52 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/866591013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T01:31:52.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:52 smithi032 ceph-mon[32461]: pgmap v338: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:31:52.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:52 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/866591013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T01:31:53.184 INFO:tasks.cephadm.ceph_manager.ceph:need seq 352187318295 got 352187318300 for osd.11 2024-04-25T01:31:53.184 DEBUG:teuthology.parallel:result is None 2024-04-25T01:31:54.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:54 smithi028 ceph-mon[29390]: pgmap v339: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:31:54.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:54 smithi130 ceph-mon[32702]: pgmap v339: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:31:54.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:54 smithi032 ceph-mon[32461]: pgmap v339: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:31:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.929+0000 7fad0c90b700 1 -- 172.21.15.28:0/2449453214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad081075e0 msgr2=0x7fad0810f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.929+0000 7fad0c90b700 1 --2- 172.21.15.28:0/2449453214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad081075e0 0x7fad0810f1b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fad0000b0d0 tx=0x7fad0000b3e0 comp rx=0 tx=0).stop 2024-04-25T01:31:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.930+0000 7fad0c90b700 1 -- 172.21.15.28:0/2449453214 shutdown_connections 2024-04-25T01:31:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.930+0000 7fad0c90b700 1 --2- 172.21.15.28:0/2449453214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad081075e0 0x7fad0810f1b0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.930+0000 7fad0c90b700 1 --2- 172.21.15.28:0/2449453214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad08102510 0x7fad081028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.930+0000 7fad0c90b700 1 --2- 172.21.15.28:0/2449453214 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad08102ec0 0x7fad08106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.930+0000 7fad0c90b700 1 -- 172.21.15.28:0/2449453214 >> 172.21.15.28:0/2449453214 conn(0x7fad080793e0 msgr2=0x7fad080797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.930+0000 7fad0c90b700 1 -- 172.21.15.28:0/2449453214 shutdown_connections 2024-04-25T01:31:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.930+0000 7fad0c90b700 1 -- 172.21.15.28:0/2449453214 wait complete. 2024-04-25T01:31:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.931+0000 7fad0c90b700 1 Processor -- start 2024-04-25T01:31:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.931+0000 7fad0c90b700 1 -- start start 2024-04-25T01:31:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.931+0000 7fad0c90b700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad08102510 0x7fad0819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.931+0000 7fad0c90b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad08102ec0 0x7fad0819ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.931+0000 7fad0c90b700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad081075e0 0x7fad081988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.931+0000 7fad0c90b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad08112620 con 0x7fad08102ec0 2024-04-25T01:31:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.931+0000 7fad0c90b700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fad081124a0 con 0x7fad08102510 2024-04-25T01:31:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.931+0000 7fad0c90b700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fad081127a0 con 0x7fad081075e0 2024-04-25T01:31:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.931+0000 7fad05d9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad08102ec0 0x7fad0819ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.931+0000 7fad05d9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad08102ec0 0x7fad0819ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38670/0 (socket says 172.21.15.28:38670) 2024-04-25T01:31:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.931+0000 7fad05d9b700 1 -- 172.21.15.28:0/1453615400 learned_addr learned my addr 172.21.15.28:0/1453615400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.931+0000 7fad0659c700 1 --2- 172.21.15.28:0/1453615400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad08102510 0x7fad0819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.931+0000 7fad06d9d700 1 --2- 172.21.15.28:0/1453615400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad081075e0 0x7fad081988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.931+0000 7fad05d9b700 1 -- 172.21.15.28:0/1453615400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad081075e0 msgr2=0x7fad081988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.931+0000 7fad05d9b700 1 --2- 172.21.15.28:0/1453615400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad081075e0 0x7fad081988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.932+0000 7fad05d9b700 1 -- 172.21.15.28:0/1453615400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad08102510 msgr2=0x7fad0819e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.932+0000 7fad05d9b700 1 --2- 172.21.15.28:0/1453615400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad08102510 0x7fad0819e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.932+0000 7fad05d9b700 1 -- 172.21.15.28:0/1453615400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad08199020 con 0x7fad08102ec0 2024-04-25T01:31:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.932+0000 7fad05d9b700 1 --2- 172.21.15.28:0/1453615400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad08102ec0 0x7fad0819ebf0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7facfc00c8f0 tx=0x7facfc00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.932+0000 7facf77fe700 1 -- 172.21.15.28:0/1453615400 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc007940 con 0x7fad08102ec0 2024-04-25T01:31:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.932+0000 7fad0c90b700 1 -- 172.21.15.28:0/1453615400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad08199310 con 0x7fad08102ec0 2024-04-25T01:31:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.932+0000 7fad0c90b700 1 -- 172.21.15.28:0/1453615400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad081a5580 con 0x7fad08102ec0 2024-04-25T01:31:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.932+0000 7facf77fe700 1 -- 172.21.15.28:0/1453615400 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facfc00f450 con 0x7fad08102ec0 2024-04-25T01:31:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.932+0000 7facf77fe700 1 -- 172.21.15.28:0/1453615400 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc018660 con 0x7fad08102ec0 2024-04-25T01:31:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.933+0000 7fad0c90b700 1 -- 172.21.15.28:0/1453615400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad08068f20 con 0x7fad08102ec0 2024-04-25T01:31:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.934+0000 7facf77fe700 1 -- 172.21.15.28:0/1453615400 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7facfc00f600 con 0x7fad08102ec0 2024-04-25T01:31:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.935+0000 7facf77fe700 1 --2- 172.21.15.28:0/1453615400 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7facf00a2cb0 0x7facf00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:54.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.935+0000 7facf77fe700 1 -- 172.21.15.28:0/1453615400 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7facfc0cf380 con 0x7fad08102ec0 2024-04-25T01:31:54.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.935+0000 7fad0659c700 1 --2- 172.21.15.28:0/1453615400 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7facf00a2cb0 0x7facf00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:54.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.936+0000 7fad0659c700 1 --2- 172.21.15.28:0/1453615400 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7facf00a2cb0 0x7facf00a5170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7facf8005950 tx=0x7facf800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:54.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:53.940+0000 7facf77fe700 1 -- 172.21.15.28:0/1453615400 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7facfc014030 con 0x7fad08102ec0 2024-04-25T01:31:54.934 INFO:teuthology.orchestra.run.smithi028.stdout:55834574929 2024-04-25T01:31:54.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:54.153+0000 7fad0c90b700 1 -- 172.21.15.28:0/1453615400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fad08199da0 con 0x7fad08102ec0 2024-04-25T01:31:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:54.154+0000 7facf77fe700 1 -- 172.21.15.28:0/1453615400 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7facfc014030 con 0x7fad08102ec0 2024-04-25T01:31:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:54.156+0000 7fad0c90b700 1 -- 172.21.15.28:0/1453615400 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7facf00a2cb0 msgr2=0x7facf00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:54.156+0000 7fad0c90b700 1 --2- 172.21.15.28:0/1453615400 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7facf00a2cb0 0x7facf00a5170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7facf8005950 tx=0x7facf800a300 comp rx=0 tx=0).stop 2024-04-25T01:31:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:54.156+0000 7fad0c90b700 1 -- 172.21.15.28:0/1453615400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad08102ec0 msgr2=0x7fad0819ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:54.156+0000 7fad0c90b700 1 --2- 172.21.15.28:0/1453615400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad08102ec0 0x7fad0819ebf0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7facfc00c8f0 tx=0x7facfc00cc00 comp rx=0 tx=0).stop 2024-04-25T01:31:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:54.156+0000 7fad0c90b700 1 -- 172.21.15.28:0/1453615400 shutdown_connections 2024-04-25T01:31:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:54.156+0000 7fad0c90b700 1 --2- 172.21.15.28:0/1453615400 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7facf00a2cb0 0x7facf00a5170 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:54.156+0000 7fad0c90b700 1 --2- 172.21.15.28:0/1453615400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fad081075e0 0x7fad081988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:54.156+0000 7fad0c90b700 1 --2- 172.21.15.28:0/1453615400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad08102510 0x7fad0819e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:54.156+0000 7fad0c90b700 1 --2- 172.21.15.28:0/1453615400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad08102ec0 0x7fad0819ebf0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:54.156+0000 7fad0c90b700 1 -- 172.21.15.28:0/1453615400 >> 172.21.15.28:0/1453615400 conn(0x7fad080793e0 msgr2=0x7fad08100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:54.156+0000 7fad0c90b700 1 -- 172.21.15.28:0/1453615400 shutdown_connections 2024-04-25T01:31:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:54.156+0000 7fad0c90b700 1 -- 172.21.15.28:0/1453615400 wait complete. 2024-04-25T01:31:55.225 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:55 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1453615400' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T01:31:55.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:55 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1453615400' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T01:31:55.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:55 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1453615400' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T01:31:56.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:56 smithi028 ceph-mon[29390]: pgmap v340: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:31:56.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:56 smithi130 ceph-mon[32702]: pgmap v340: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:31:56.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:56 smithi032 ceph-mon[32461]: pgmap v340: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:31:57.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.790+0000 7f345eb87700 1 -- 172.21.15.28:0/2785393150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3458102e90 msgr2=0x7f3458106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:57.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.790+0000 7f345eb87700 1 --2- 172.21.15.28:0/2785393150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3458102e90 0x7f3458106e90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3444009990 tx=0x7f3444009ca0 comp rx=0 tx=0).stop 2024-04-25T01:31:57.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.790+0000 7f345eb87700 1 -- 172.21.15.28:0/2785393150 shutdown_connections 2024-04-25T01:31:57.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.790+0000 7f345eb87700 1 --2- 172.21.15.28:0/2785393150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3458107570 0x7f345810f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:57.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.790+0000 7f345eb87700 1 --2- 172.21.15.28:0/2785393150 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f34581024e0 0x7f34581028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:57.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.790+0000 7f345eb87700 1 --2- 172.21.15.28:0/2785393150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3458102e90 0x7f3458106e90 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:57.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.790+0000 7f345eb87700 1 -- 172.21.15.28:0/2785393150 >> 172.21.15.28:0/2785393150 conn(0x7f3458079390 msgr2=0x7f34580797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:57.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.790+0000 7f345eb87700 1 -- 172.21.15.28:0/2785393150 shutdown_connections 2024-04-25T01:31:57.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.790+0000 7f345eb87700 1 -- 172.21.15.28:0/2785393150 wait complete. 2024-04-25T01:31:57.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.790+0000 7f345eb87700 1 Processor -- start 2024-04-25T01:31:57.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.790+0000 7f345eb87700 1 -- start start 2024-04-25T01:31:57.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f345eb87700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34581024e0 0x7f3458074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:57.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f345eb87700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3458102e90 0x7f3458074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:57.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f345eb87700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3458107570 0x7f3458074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:57.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f345eb87700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3458112510 con 0x7f34581024e0 2024-04-25T01:31:57.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f345eb87700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3458112390 con 0x7f3458102e90 2024-04-25T01:31:57.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f345eb87700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f3458112690 con 0x7f3458107570 2024-04-25T01:31:57.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f345c923700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34581024e0 0x7f3458074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:57.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f345c923700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34581024e0 0x7f3458074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35204/0 (socket says 172.21.15.28:35204) 2024-04-25T01:31:57.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f345c923700 1 -- 172.21.15.28:0/502360790 learned_addr learned my addr 172.21.15.28:0/502360790 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:57.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f3457fff700 1 --2- 172.21.15.28:0/502360790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3458102e90 0x7f3458074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:57.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f345d124700 1 --2- 172.21.15.28:0/502360790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3458107570 0x7f3458074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:57.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f345c923700 1 -- 172.21.15.28:0/502360790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3458107570 msgr2=0x7f3458074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:57.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f345c923700 1 --2- 172.21.15.28:0/502360790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3458107570 0x7f3458074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:57.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f345c923700 1 -- 172.21.15.28:0/502360790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3458102e90 msgr2=0x7f3458074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:57.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f345c923700 1 --2- 172.21.15.28:0/502360790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3458102e90 0x7f3458074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:57.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f345c923700 1 -- 172.21.15.28:0/502360790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34581a6ef0 con 0x7f34581024e0 2024-04-25T01:31:57.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f345d124700 1 --2- 172.21.15.28:0/502360790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3458107570 0x7f3458074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:57.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f345c923700 1 --2- 172.21.15.28:0/502360790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34581024e0 0x7f3458074530 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f344c00c9b0 tx=0x7f344c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:57.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f3455ffb700 1 -- 172.21.15.28:0/502360790 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344c007b00 con 0x7f34581024e0 2024-04-25T01:31:57.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f3455ffb700 1 -- 172.21.15.28:0/502360790 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f344c007ca0 con 0x7f34581024e0 2024-04-25T01:31:57.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f345eb87700 1 -- 172.21.15.28:0/502360790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34581a71e0 con 0x7f34581024e0 2024-04-25T01:31:57.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.791+0000 7f3455ffb700 1 -- 172.21.15.28:0/502360790 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344c018730 con 0x7f34581024e0 2024-04-25T01:31:57.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.792+0000 7f345eb87700 1 -- 172.21.15.28:0/502360790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34581a7740 con 0x7f34581024e0 2024-04-25T01:31:57.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.793+0000 7f345eb87700 1 -- 172.21.15.28:0/502360790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3458068f20 con 0x7f34581024e0 2024-04-25T01:31:57.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.794+0000 7f3455ffb700 1 -- 172.21.15.28:0/502360790 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f344c0188d0 con 0x7f34581024e0 2024-04-25T01:31:57.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.794+0000 7f3455ffb700 1 --2- 172.21.15.28:0/502360790 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f34480a2d30 0x7f34480a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:57.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.794+0000 7f3455ffb700 1 -- 172.21.15.28:0/502360790 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f344c0cf770 con 0x7f34581024e0 2024-04-25T01:31:57.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.794+0000 7f3457fff700 1 --2- 172.21.15.28:0/502360790 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f34480a2d30 0x7f34480a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:57.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.795+0000 7f3457fff700 1 --2- 172.21.15.28:0/502360790 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f34480a2d30 0x7f34480a51f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f344400b5c0 tx=0x7f3444005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:57.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:57.798+0000 7f3455ffb700 1 -- 172.21.15.28:0/502360790 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f344c014030 con 0x7f34581024e0 2024-04-25T01:31:57.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.014+0000 7f345eb87700 1 -- 172.21.15.28:0/502360790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f345802db30 con 0x7f34581024e0 2024-04-25T01:31:57.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.014+0000 7f3455ffb700 1 -- 172.21.15.28:0/502360790 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f344c09c2c0 con 0x7f34581024e0 2024-04-25T01:31:57.991 INFO:teuthology.orchestra.run.smithi028.stdout:34359738455 2024-04-25T01:31:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.017+0000 7f345eb87700 1 -- 172.21.15.28:0/502360790 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f34480a2d30 msgr2=0x7f34480a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.017+0000 7f345eb87700 1 --2- 172.21.15.28:0/502360790 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f34480a2d30 0x7f34480a51f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f344400b5c0 tx=0x7f3444005960 comp rx=0 tx=0).stop 2024-04-25T01:31:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.017+0000 7f345eb87700 1 -- 172.21.15.28:0/502360790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34581024e0 msgr2=0x7f3458074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.017+0000 7f345eb87700 1 --2- 172.21.15.28:0/502360790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34581024e0 0x7f3458074530 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f344c00c9b0 tx=0x7f344c00ccc0 comp rx=0 tx=0).stop 2024-04-25T01:31:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.017+0000 7f345eb87700 1 -- 172.21.15.28:0/502360790 shutdown_connections 2024-04-25T01:31:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.017+0000 7f345eb87700 1 --2- 172.21.15.28:0/502360790 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f34480a2d30 0x7f34480a51f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.017+0000 7f345eb87700 1 --2- 172.21.15.28:0/502360790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3458107570 0x7f3458074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.017+0000 7f345eb87700 1 --2- 172.21.15.28:0/502360790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34581024e0 0x7f3458074530 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.017+0000 7f345eb87700 1 --2- 172.21.15.28:0/502360790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3458102e90 0x7f3458074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:57.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.017+0000 7f345eb87700 1 -- 172.21.15.28:0/502360790 >> 172.21.15.28:0/502360790 conn(0x7f3458079390 msgr2=0x7f3458100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:57.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.017+0000 7f345eb87700 1 -- 172.21.15.28:0/502360790 shutdown_connections 2024-04-25T01:31:57.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.017+0000 7f345eb87700 1 -- 172.21.15.28:0/502360790 wait complete. 2024-04-25T01:31:58.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:58 smithi028 ceph-mon[29390]: pgmap v341: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:31:58.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:58 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/502360790' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T01:31:58.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:58 smithi130 ceph-mon[32702]: pgmap v341: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:31:58.461 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:58 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/502360790' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T01:31:58.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:58 smithi032 ceph-mon[32461]: pgmap v341: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:31:58.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:58 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/502360790' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T01:31:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.774+0000 7f299c8a0700 1 -- 172.21.15.28:0/3018612858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998101af0 msgr2=0x7f299810dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.774+0000 7f299c8a0700 1 --2- 172.21.15.28:0/3018612858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998101af0 0x7f299810dfe0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f299000b0a0 tx=0x7f299000b3b0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.774+0000 7f299c8a0700 1 -- 172.21.15.28:0/3018612858 shutdown_connections 2024-04-25T01:31:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.774+0000 7f299c8a0700 1 --2- 172.21.15.28:0/3018612858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998101af0 0x7f299810dfe0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.774+0000 7f299c8a0700 1 --2- 172.21.15.28:0/3018612858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29981071a0 0x7f2998107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.774+0000 7f299c8a0700 1 --2- 172.21.15.28:0/3018612858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2998101130 0x7f29981015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.774+0000 7f299c8a0700 1 -- 172.21.15.28:0/3018612858 >> 172.21.15.28:0/3018612858 conn(0x7f29980780e0 msgr2=0x7f29980784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.774+0000 7f299c8a0700 1 -- 172.21.15.28:0/3018612858 shutdown_connections 2024-04-25T01:31:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.774+0000 7f299c8a0700 1 -- 172.21.15.28:0/3018612858 wait complete. 2024-04-25T01:31:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.775+0000 7f299c8a0700 1 Processor -- start 2024-04-25T01:31:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.775+0000 7f299c8a0700 1 -- start start 2024-04-25T01:31:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.775+0000 7f299c8a0700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2998101130 0x7f299819e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.775+0000 7f299c8a0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998101af0 0x7f299819ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.775+0000 7f299c8a0700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29981071a0 0x7f2998198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.775+0000 7f299c8a0700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f29981114e0 con 0x7f2998101af0 2024-04-25T01:31:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.775+0000 7f299c8a0700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2998111360 con 0x7f29981071a0 2024-04-25T01:31:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.775+0000 7f299c8a0700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2998111660 con 0x7f2998101130 2024-04-25T01:31:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.776+0000 7f2995d9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998101af0 0x7f299819ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.776+0000 7f2995d9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998101af0 0x7f299819ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35220/0 (socket says 172.21.15.28:35220) 2024-04-25T01:31:58.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.776+0000 7f2995d9b700 1 -- 172.21.15.28:0/3950299489 learned_addr learned my addr 172.21.15.28:0/3950299489 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:58.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.776+0000 7f2996d9d700 1 --2- 172.21.15.28:0/3950299489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29981071a0 0x7f2998198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.776+0000 7f2995d9b700 1 -- 172.21.15.28:0/3950299489 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2998101130 msgr2=0x7f299819e6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T01:31:58.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.776+0000 7f2995d9b700 1 --2- 172.21.15.28:0/3950299489 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2998101130 0x7f299819e6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.776+0000 7f2995d9b700 1 -- 172.21.15.28:0/3950299489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29981071a0 msgr2=0x7f2998198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:58.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.776+0000 7f2995d9b700 1 --2- 172.21.15.28:0/3950299489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29981071a0 0x7f2998198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.776+0000 7f2995d9b700 1 -- 172.21.15.28:0/3950299489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2998199030 con 0x7f2998101af0 2024-04-25T01:31:58.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.776+0000 7f2995d9b700 1 --2- 172.21.15.28:0/3950299489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998101af0 0x7f299819ec10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f298c00b530 tx=0x7f298c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:58.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.776+0000 7f29877fe700 1 -- 172.21.15.28:0/3950299489 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c015020 con 0x7f2998101af0 2024-04-25T01:31:58.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.776+0000 7f29877fe700 1 -- 172.21.15.28:0/3950299489 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f298c011d10 con 0x7f2998101af0 2024-04-25T01:31:58.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.776+0000 7f299c8a0700 1 -- 172.21.15.28:0/3950299489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2998199320 con 0x7f2998101af0 2024-04-25T01:31:58.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.776+0000 7f29877fe700 1 -- 172.21.15.28:0/3950299489 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c00f2e0 con 0x7f2998101af0 2024-04-25T01:31:58.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.776+0000 7f299c8a0700 1 -- 172.21.15.28:0/3950299489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29981a5530 con 0x7f2998101af0 2024-04-25T01:31:58.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.777+0000 7f299c8a0700 1 -- 172.21.15.28:0/3950299489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2998102850 con 0x7f2998101af0 2024-04-25T01:31:58.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.778+0000 7f29877fe700 1 -- 172.21.15.28:0/3950299489 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f298c018020 con 0x7f2998101af0 2024-04-25T01:31:58.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.779+0000 7f29877fe700 1 --2- 172.21.15.28:0/3950299489 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f29800a2cb0 0x7f29800a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:58.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.779+0000 7f299659c700 1 --2- 172.21.15.28:0/3950299489 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f29800a2cb0 0x7f29800a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:58.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.779+0000 7f29877fe700 1 -- 172.21.15.28:0/3950299489 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f298c0cec80 con 0x7f2998101af0 2024-04-25T01:31:58.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.780+0000 7f299659c700 1 --2- 172.21.15.28:0/3950299489 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f29800a2cb0 0x7f29800a5170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f298800a910 tx=0x7f2988005c10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:58.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.784+0000 7f29877fe700 1 -- 172.21.15.28:0/3950299489 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f298c01f020 con 0x7f2998101af0 2024-04-25T01:31:58.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.994+0000 7f299c8a0700 1 -- 172.21.15.28:0/3950299489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f29981a57c0 con 0x7f2998101af0 2024-04-25T01:31:58.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.995+0000 7f29877fe700 1 -- 172.21.15.28:0/3950299489 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f298c09b830 con 0x7f2998101af0 2024-04-25T01:31:58.972 INFO:teuthology.orchestra.run.smithi028.stdout:206158430265 2024-04-25T01:31:58.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.997+0000 7f299c8a0700 1 -- 172.21.15.28:0/3950299489 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f29800a2cb0 msgr2=0x7f29800a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:58.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.997+0000 7f299c8a0700 1 --2- 172.21.15.28:0/3950299489 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f29800a2cb0 0x7f29800a5170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f298800a910 tx=0x7f2988005c10 comp rx=0 tx=0).stop 2024-04-25T01:31:58.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.997+0000 7f299c8a0700 1 -- 172.21.15.28:0/3950299489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998101af0 msgr2=0x7f299819ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:58.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.997+0000 7f299c8a0700 1 --2- 172.21.15.28:0/3950299489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998101af0 0x7f299819ec10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f298c00b530 tx=0x7f298c00b840 comp rx=0 tx=0).stop 2024-04-25T01:31:58.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.997+0000 7f299c8a0700 1 -- 172.21.15.28:0/3950299489 shutdown_connections 2024-04-25T01:31:58.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.997+0000 7f299c8a0700 1 --2- 172.21.15.28:0/3950299489 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f29800a2cb0 0x7f29800a5170 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.997+0000 7f299c8a0700 1 --2- 172.21.15.28:0/3950299489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29981071a0 0x7f2998198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.997+0000 7f299c8a0700 1 --2- 172.21.15.28:0/3950299489 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2998101130 0x7f299819e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.997+0000 7f299c8a0700 1 --2- 172.21.15.28:0/3950299489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2998101af0 0x7f299819ec10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:58.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.997+0000 7f299c8a0700 1 -- 172.21.15.28:0/3950299489 >> 172.21.15.28:0/3950299489 conn(0x7f29980780e0 msgr2=0x7f2998100360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:58.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.997+0000 7f299c8a0700 1 -- 172.21.15.28:0/3950299489 shutdown_connections 2024-04-25T01:31:58.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:58.997+0000 7f299c8a0700 1 -- 172.21.15.28:0/3950299489 wait complete. 2024-04-25T01:31:59.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:31:59 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3950299489' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T01:31:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.318+0000 7f6b65e2a700 1 -- 172.21.15.28:0/4038679338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6010a4c0 msgr2=0x7f6b6010a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.318+0000 7f6b65e2a700 1 --2- 172.21.15.28:0/4038679338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6010a4c0 0x7f6b6010a8a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6b50009990 tx=0x7f6b50009ca0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.318+0000 7f6b65e2a700 1 -- 172.21.15.28:0/4038679338 shutdown_connections 2024-04-25T01:31:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.318+0000 7f6b65e2a700 1 --2- 172.21.15.28:0/4038679338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b60104df0 0x7f6b60111300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.318+0000 7f6b65e2a700 1 --2- 172.21.15.28:0/4038679338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6010a4c0 0x7f6b6010a8a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.318+0000 7f6b65e2a700 1 --2- 172.21.15.28:0/4038679338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b60104430 0x7f6b601048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.318+0000 7f6b65e2a700 1 -- 172.21.15.28:0/4038679338 >> 172.21.15.28:0/4038679338 conn(0x7f6b600fff40 msgr2=0x7f6b60102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.318+0000 7f6b65e2a700 1 -- 172.21.15.28:0/4038679338 shutdown_connections 2024-04-25T01:31:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.318+0000 7f6b65e2a700 1 -- 172.21.15.28:0/4038679338 wait complete. 2024-04-25T01:31:59.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.319+0000 7f6b65e2a700 1 Processor -- start 2024-04-25T01:31:59.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.319+0000 7f6b65e2a700 1 -- start start 2024-04-25T01:31:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.319+0000 7f6b65e2a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b60104430 0x7f6b6019ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.319+0000 7f6b65e2a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b60104df0 0x7f6b6019b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.319+0000 7f6b65e2a700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b6010a4c0 0x7f6b6019f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.319+0000 7f6b65e2a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b60114820 con 0x7f6b60104430 2024-04-25T01:31:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.319+0000 7f6b65e2a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6b601146a0 con 0x7f6b60104df0 2024-04-25T01:31:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.319+0000 7f6b65e2a700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6b601149a0 con 0x7f6b6010a4c0 2024-04-25T01:31:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.319+0000 7f6b5f7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b60104430 0x7f6b6019ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.319+0000 7f6b5f7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b60104430 0x7f6b6019ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35234/0 (socket says 172.21.15.28:35234) 2024-04-25T01:31:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.319+0000 7f6b5f7fe700 1 -- 172.21.15.28:0/2978712137 learned_addr learned my addr 172.21.15.28:0/2978712137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.319+0000 7f6b56ffd700 1 --2- 172.21.15.28:0/2978712137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b60104df0 0x7f6b6019b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.320+0000 7f6b5f7fe700 1 -- 172.21.15.28:0/2978712137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b6010a4c0 msgr2=0x7f6b6019f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.320+0000 7f6b5f7fe700 1 --2- 172.21.15.28:0/2978712137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b6010a4c0 0x7f6b6019f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.320+0000 7f6b5f7fe700 1 -- 172.21.15.28:0/2978712137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b60104df0 msgr2=0x7f6b6019b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.320+0000 7f6b5f7fe700 1 --2- 172.21.15.28:0/2978712137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b60104df0 0x7f6b6019b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.320+0000 7f6b5f7fe700 1 -- 172.21.15.28:0/2978712137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b6019fd20 con 0x7f6b60104430 2024-04-25T01:31:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.320+0000 7f6b56ffd700 1 --2- 172.21.15.28:0/2978712137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b60104df0 0x7f6b6019b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:31:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.320+0000 7f6b5f7fe700 1 --2- 172.21.15.28:0/2978712137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b60104430 0x7f6b6019ad10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6b5000bff0 tx=0x7f6b50005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.320+0000 7f6b5d7fa700 1 -- 172.21.15.28:0/2978712137 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b50027070 con 0x7f6b60104430 2024-04-25T01:31:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.320+0000 7f6b5d7fa700 1 -- 172.21.15.28:0/2978712137 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b50004150 con 0x7f6b60104430 2024-04-25T01:31:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.320+0000 7f6b65e2a700 1 -- 172.21.15.28:0/2978712137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b6019ffb0 con 0x7f6b60104430 2024-04-25T01:31:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.320+0000 7f6b5d7fa700 1 -- 172.21.15.28:0/2978712137 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5000f460 con 0x7f6b60104430 2024-04-25T01:31:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.320+0000 7f6b65e2a700 1 -- 172.21.15.28:0/2978712137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b601a7890 con 0x7f6b60104430 2024-04-25T01:31:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.321+0000 7f6b65e2a700 1 -- 172.21.15.28:0/2978712137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b60068f20 con 0x7f6b60104430 2024-04-25T01:31:59.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.323+0000 7f6b5d7fa700 1 -- 172.21.15.28:0/2978712137 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6b5002a070 con 0x7f6b60104430 2024-04-25T01:31:59.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.323+0000 7f6b5d7fa700 1 --2- 172.21.15.28:0/2978712137 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6b4c0a2cb0 0x7f6b4c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.324+0000 7f6b5d7fa700 1 -- 172.21.15.28:0/2978712137 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f6b500d0bc0 con 0x7f6b60104430 2024-04-25T01:31:59.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.324+0000 7f6b56ffd700 1 --2- 172.21.15.28:0/2978712137 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6b4c0a2cb0 0x7f6b4c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.324+0000 7f6b56ffd700 1 --2- 172.21.15.28:0/2978712137 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6b4c0a2cb0 0x7f6b4c0a5170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6b60102590 tx=0x7f6b48008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.327+0000 7f6b5d7fa700 1 -- 172.21.15.28:0/2978712137 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6b50031020 con 0x7f6b60104430 2024-04-25T01:31:59.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:31:59 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3950299489' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T01:31:59.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:31:59 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3950299489' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T01:31:59.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.535+0000 7f6b65e2a700 1 -- 172.21.15.28:0/2978712137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f6b6002db30 con 0x7f6b60104430 2024-04-25T01:31:59.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.536+0000 7f6b5d7fa700 1 -- 172.21.15.28:0/2978712137 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6b50013070 con 0x7f6b60104430 2024-04-25T01:31:59.513 INFO:teuthology.orchestra.run.smithi028.stdout:227633266739 2024-04-25T01:31:59.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.538+0000 7f6b65e2a700 1 -- 172.21.15.28:0/2978712137 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6b4c0a2cb0 msgr2=0x7f6b4c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.538+0000 7f6b65e2a700 1 --2- 172.21.15.28:0/2978712137 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6b4c0a2cb0 0x7f6b4c0a5170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6b60102590 tx=0x7f6b48008040 comp rx=0 tx=0).stop 2024-04-25T01:31:59.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.538+0000 7f6b65e2a700 1 -- 172.21.15.28:0/2978712137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b60104430 msgr2=0x7f6b6019ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.538+0000 7f6b65e2a700 1 --2- 172.21.15.28:0/2978712137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b60104430 0x7f6b6019ad10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6b5000bff0 tx=0x7f6b50005640 comp rx=0 tx=0).stop 2024-04-25T01:31:59.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.538+0000 7f6b65e2a700 1 -- 172.21.15.28:0/2978712137 shutdown_connections 2024-04-25T01:31:59.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.538+0000 7f6b65e2a700 1 --2- 172.21.15.28:0/2978712137 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6b4c0a2cb0 0x7f6b4c0a5170 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.538+0000 7f6b65e2a700 1 --2- 172.21.15.28:0/2978712137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b6010a4c0 0x7f6b6019f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.538+0000 7f6b65e2a700 1 --2- 172.21.15.28:0/2978712137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b60104430 0x7f6b6019ad10 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.538+0000 7f6b65e2a700 1 --2- 172.21.15.28:0/2978712137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b60104df0 0x7f6b6019b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.538+0000 7f6b65e2a700 1 -- 172.21.15.28:0/2978712137 >> 172.21.15.28:0/2978712137 conn(0x7f6b600fff40 msgr2=0x7f6b60100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.538+0000 7f6b65e2a700 1 -- 172.21.15.28:0/2978712137 shutdown_connections 2024-04-25T01:31:59.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.538+0000 7f6b65e2a700 1 -- 172.21.15.28:0/2978712137 wait complete. 2024-04-25T01:31:59.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.545+0000 7f13d4ba6700 1 -- 172.21.15.28:0/3527589937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13d0069f40 msgr2=0x7f13d006a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.545+0000 7f13d4ba6700 1 --2- 172.21.15.28:0/3527589937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13d0069f40 0x7f13d006a3a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f13c4009990 tx=0x7f13c4009ca0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.545+0000 7f13d4ba6700 1 -- 172.21.15.28:0/3527589937 shutdown_connections 2024-04-25T01:31:59.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.545+0000 7f13d4ba6700 1 --2- 172.21.15.28:0/3527589937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13d006a8e0 0x7f13d010f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.545+0000 7f13d4ba6700 1 --2- 172.21.15.28:0/3527589937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13d01082b0 0x7f13d0108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.545+0000 7f13d4ba6700 1 --2- 172.21.15.28:0/3527589937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13d0069f40 0x7f13d006a3a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.545+0000 7f13d4ba6700 1 -- 172.21.15.28:0/3527589937 >> 172.21.15.28:0/3527589937 conn(0x7f13d00780e0 msgr2=0x7f13d00784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.545+0000 7f13d4ba6700 1 -- 172.21.15.28:0/3527589937 shutdown_connections 2024-04-25T01:31:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.545+0000 7f13d4ba6700 1 -- 172.21.15.28:0/3527589937 wait complete. 2024-04-25T01:31:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.546+0000 7f13d4ba6700 1 Processor -- start 2024-04-25T01:31:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.546+0000 7f13d4ba6700 1 -- start start 2024-04-25T01:31:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.546+0000 7f13d4ba6700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13d0069f40 0x7f13d019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.546+0000 7f13d4ba6700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13d006a8e0 0x7f13d019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.546+0000 7f13d4ba6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13d01082b0 0x7f13d01987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.546+0000 7f13d4ba6700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13d0112510 con 0x7f13d01082b0 2024-04-25T01:31:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.546+0000 7f13d4ba6700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f13d0112390 con 0x7f13d0069f40 2024-04-25T01:31:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.546+0000 7f13d4ba6700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f13d0112690 con 0x7f13d006a8e0 2024-04-25T01:31:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.546+0000 7f13ced9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13d01082b0 0x7f13d01987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.546+0000 7f13ced9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13d01082b0 0x7f13d01987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35254/0 (socket says 172.21.15.28:35254) 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.546+0000 7f13ced9d700 1 -- 172.21.15.28:0/1352065829 learned_addr learned my addr 172.21.15.28:0/1352065829 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.546+0000 7f13cdd9b700 1 --2- 172.21.15.28:0/1352065829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13d006a8e0 0x7f13d019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.546+0000 7f13ce59c700 1 --2- 172.21.15.28:0/1352065829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13d0069f40 0x7f13d019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.547+0000 7f13ced9d700 1 -- 172.21.15.28:0/1352065829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13d006a8e0 msgr2=0x7f13d019ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.547+0000 7f13ced9d700 1 --2- 172.21.15.28:0/1352065829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13d006a8e0 0x7f13d019ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.547+0000 7f13ced9d700 1 -- 172.21.15.28:0/1352065829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13d0069f40 msgr2=0x7f13d019e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.547+0000 7f13ced9d700 1 --2- 172.21.15.28:0/1352065829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13d0069f40 0x7f13d019e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.547+0000 7f13ced9d700 1 -- 172.21.15.28:0/1352065829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d0198fd0 con 0x7f13d01082b0 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.547+0000 7f13ced9d700 1 --2- 172.21.15.28:0/1352065829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13d01082b0 0x7f13d01987f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f13c800ef30 tx=0x7f13c800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.547+0000 7f13bf7fe700 1 -- 172.21.15.28:0/1352065829 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c80092e0 con 0x7f13d01082b0 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.547+0000 7f13bf7fe700 1 -- 172.21.15.28:0/1352065829 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13c8004bb0 con 0x7f13d01082b0 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.547+0000 7f13d4ba6700 1 -- 172.21.15.28:0/1352065829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d0199260 con 0x7f13d01082b0 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.547+0000 7f13bf7fe700 1 -- 172.21.15.28:0/1352065829 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c8004280 con 0x7f13d01082b0 2024-04-25T01:31:59.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.547+0000 7f13d4ba6700 1 -- 172.21.15.28:0/1352065829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d01a54d0 con 0x7f13d01082b0 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.548+0000 7f13d4ba6700 1 -- 172.21.15.28:0/1352065829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13b0005330 con 0x7f13d01082b0 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.550+0000 7f13bf7fe700 1 -- 172.21.15.28:0/1352065829 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f13c801b030 con 0x7f13d01082b0 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.551+0000 7f13bf7fe700 1 --2- 172.21.15.28:0/1352065829 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f13b80a2cb0 0x7f13b80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.551+0000 7f13bf7fe700 1 -- 172.21.15.28:0/1352065829 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f13c80cf8d0 con 0x7f13d01082b0 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.551+0000 7f13ce59c700 1 --2- 172.21.15.28:0/1352065829 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f13b80a2cb0 0x7f13b80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.551+0000 7f13ce59c700 1 --2- 172.21.15.28:0/1352065829 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f13b80a2cb0 0x7f13b80a5170 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f13c0007b60 tx=0x7f13c00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:31:59.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.555+0000 7f13bf7fe700 1 -- 172.21.15.28:0/1352065829 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f13c8022020 con 0x7f13d01082b0 2024-04-25T01:31:59.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.768+0000 7f13d4ba6700 1 -- 172.21.15.28:0/1352065829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f13b00058f0 con 0x7f13d01082b0 2024-04-25T01:31:59.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.768+0000 7f13bf7fe700 1 -- 172.21.15.28:0/1352065829 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f13c8010040 con 0x7f13d01082b0 2024-04-25T01:31:59.745 INFO:teuthology.orchestra.run.smithi028.stdout:111669149768 2024-04-25T01:31:59.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.770+0000 7f13d4ba6700 1 -- 172.21.15.28:0/1352065829 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f13b80a2cb0 msgr2=0x7f13b80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.770+0000 7f13d4ba6700 1 --2- 172.21.15.28:0/1352065829 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f13b80a2cb0 0x7f13b80a5170 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f13c0007b60 tx=0x7f13c00073d0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.770+0000 7f13d4ba6700 1 -- 172.21.15.28:0/1352065829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13d01082b0 msgr2=0x7f13d01987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:31:59.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.770+0000 7f13d4ba6700 1 --2- 172.21.15.28:0/1352065829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13d01082b0 0x7f13d01987f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f13c800ef30 tx=0x7f13c800c3a0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.770+0000 7f13d4ba6700 1 -- 172.21.15.28:0/1352065829 shutdown_connections 2024-04-25T01:31:59.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.770+0000 7f13d4ba6700 1 --2- 172.21.15.28:0/1352065829 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f13b80a2cb0 0x7f13b80a5170 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.770+0000 7f13d4ba6700 1 --2- 172.21.15.28:0/1352065829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13d01082b0 0x7f13d01987f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.770+0000 7f13d4ba6700 1 --2- 172.21.15.28:0/1352065829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f13d0069f40 0x7f13d019e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.770+0000 7f13d4ba6700 1 --2- 172.21.15.28:0/1352065829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f13d006a8e0 0x7f13d019ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:31:59.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.770+0000 7f13d4ba6700 1 -- 172.21.15.28:0/1352065829 >> 172.21.15.28:0/1352065829 conn(0x7f13d00780e0 msgr2=0x7f13d00ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:31:59.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.770+0000 7f13d4ba6700 1 -- 172.21.15.28:0/1352065829 shutdown_connections 2024-04-25T01:31:59.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:31:59.770+0000 7f13d4ba6700 1 -- 172.21.15.28:0/1352065829 wait complete. 2024-04-25T01:32:00.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:00 smithi028 ceph-mon[29390]: pgmap v342: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:00.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:00 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2978712137' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T01:32:00.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:00 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1352065829' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T01:32:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.297+0000 7fa151e88700 1 -- 172.21.15.28:0/112639913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c101130 msgr2=0x7fa14c1015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.297+0000 7fa151e88700 1 --2- 172.21.15.28:0/112639913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c101130 0x7fa14c1015b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa140009990 tx=0x7fa140009ca0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.297+0000 7fa151e88700 1 -- 172.21.15.28:0/112639913 shutdown_connections 2024-04-25T01:32:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.297+0000 7fa151e88700 1 --2- 172.21.15.28:0/112639913 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa14c101af0 0x7fa14c10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.297+0000 7fa151e88700 1 --2- 172.21.15.28:0/112639913 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa14c1071a0 0x7fa14c107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.297+0000 7fa151e88700 1 --2- 172.21.15.28:0/112639913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c101130 0x7fa14c1015b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.297+0000 7fa151e88700 1 -- 172.21.15.28:0/112639913 >> 172.21.15.28:0/112639913 conn(0x7fa14c0780e0 msgr2=0x7fa14c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.297+0000 7fa151e88700 1 -- 172.21.15.28:0/112639913 shutdown_connections 2024-04-25T01:32:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.297+0000 7fa151e88700 1 -- 172.21.15.28:0/112639913 wait complete. 2024-04-25T01:32:00.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.298+0000 7fa151e88700 1 Processor -- start 2024-04-25T01:32:00.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.298+0000 7fa151e88700 1 -- start start 2024-04-25T01:32:00.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.298+0000 7fa151e88700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c101130 0x7fa14c198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.298+0000 7fa151e88700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa14c101af0 0x7fa14c199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.298+0000 7fa151e88700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa14c1071a0 0x7fa14c19d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.298+0000 7fa151e88700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa14c1114b0 con 0x7fa14c101130 2024-04-25T01:32:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.298+0000 7fa151e88700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa14c111330 con 0x7fa14c1071a0 2024-04-25T01:32:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.298+0000 7fa151e88700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa14c111630 con 0x7fa14c101af0 2024-04-25T01:32:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.298+0000 7fa14b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c101130 0x7fa14c198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.298+0000 7fa14b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c101130 0x7fa14c198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35274/0 (socket says 172.21.15.28:35274) 2024-04-25T01:32:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.298+0000 7fa14b7fe700 1 -- 172.21.15.28:0/4244198431 learned_addr learned my addr 172.21.15.28:0/4244198431 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.298+0000 7fa14bfff700 1 --2- 172.21.15.28:0/4244198431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa14c1071a0 0x7fa14c19d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.298+0000 7fa14affd700 1 --2- 172.21.15.28:0/4244198431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa14c101af0 0x7fa14c199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.299+0000 7fa14b7fe700 1 -- 172.21.15.28:0/4244198431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa14c101af0 msgr2=0x7fa14c199040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.299+0000 7fa14b7fe700 1 --2- 172.21.15.28:0/4244198431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa14c101af0 0x7fa14c199040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.299+0000 7fa14b7fe700 1 -- 172.21.15.28:0/4244198431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa14c1071a0 msgr2=0x7fa14c19d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.299+0000 7fa14b7fe700 1 --2- 172.21.15.28:0/4244198431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa14c1071a0 0x7fa14c19d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.299+0000 7fa14b7fe700 1 -- 172.21.15.28:0/4244198431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa14c19db00 con 0x7fa14c101130 2024-04-25T01:32:00.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.299+0000 7fa14b7fe700 1 --2- 172.21.15.28:0/4244198431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c101130 0x7fa14c198b00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa13c00d980 tx=0x7fa13c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:00.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.299+0000 7fa148ff9700 1 -- 172.21.15.28:0/4244198431 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c017070 con 0x7fa14c101130 2024-04-25T01:32:00.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.299+0000 7fa151e88700 1 -- 172.21.15.28:0/4244198431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa14c19ddf0 con 0x7fa14c101130 2024-04-25T01:32:00.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.299+0000 7fa148ff9700 1 -- 172.21.15.28:0/4244198431 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa13c005020 con 0x7fa14c101130 2024-04-25T01:32:00.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.299+0000 7fa148ff9700 1 -- 172.21.15.28:0/4244198431 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c00fbf0 con 0x7fa14c101130 2024-04-25T01:32:00.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.299+0000 7fa151e88700 1 -- 172.21.15.28:0/4244198431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa14c102d40 con 0x7fa14c101130 2024-04-25T01:32:00.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.300+0000 7fa151e88700 1 -- 172.21.15.28:0/4244198431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa14c102850 con 0x7fa14c101130 2024-04-25T01:32:00.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.301+0000 7fa148ff9700 1 -- 172.21.15.28:0/4244198431 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa13c0051d0 con 0x7fa14c101130 2024-04-25T01:32:00.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.302+0000 7fa148ff9700 1 --2- 172.21.15.28:0/4244198431 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa1340a2cb0 0x7fa1340a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:00.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.302+0000 7fa148ff9700 1 -- 172.21.15.28:0/4244198431 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fa13c0cf740 con 0x7fa14c101130 2024-04-25T01:32:00.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.302+0000 7fa14affd700 1 --2- 172.21.15.28:0/4244198431 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa1340a2cb0 0x7fa1340a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:00.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.302+0000 7fa14affd700 1 --2- 172.21.15.28:0/4244198431 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa1340a2cb0 0x7fa1340a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa1400098e0 tx=0x7fa140005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.305+0000 7fa148ff9700 1 -- 172.21.15.28:0/4244198431 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa13c020020 con 0x7fa14c101130 2024-04-25T01:32:00.359 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:00 smithi130 ceph-mon[32702]: pgmap v342: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:00.359 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:00 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2978712137' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T01:32:00.359 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:00 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1352065829' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T01:32:00.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:00 smithi032 ceph-mon[32461]: pgmap v342: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:00.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:00 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2978712137' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T01:32:00.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:00 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1352065829' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T01:32:00.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.513+0000 7fa151e88700 1 -- 172.21.15.28:0/4244198431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fa14c068f20 con 0x7fa14c101130 2024-04-25T01:32:00.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.513+0000 7fa148ff9700 1 -- 172.21.15.28:0/4244198431 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa13c09c1e0 con 0x7fa14c101130 2024-04-25T01:32:00.490 INFO:teuthology.orchestra.run.smithi028.stdout:171798691902 2024-04-25T01:32:00.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.515+0000 7fa151e88700 1 -- 172.21.15.28:0/4244198431 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa1340a2cb0 msgr2=0x7fa1340a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:00.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.515+0000 7fa151e88700 1 --2- 172.21.15.28:0/4244198431 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa1340a2cb0 0x7fa1340a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa1400098e0 tx=0x7fa140005960 comp rx=0 tx=0).stop 2024-04-25T01:32:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.515+0000 7fa151e88700 1 -- 172.21.15.28:0/4244198431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c101130 msgr2=0x7fa14c198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.515+0000 7fa151e88700 1 --2- 172.21.15.28:0/4244198431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c101130 0x7fa14c198b00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa13c00d980 tx=0x7fa13c00dc90 comp rx=0 tx=0).stop 2024-04-25T01:32:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.516+0000 7fa151e88700 1 -- 172.21.15.28:0/4244198431 shutdown_connections 2024-04-25T01:32:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.516+0000 7fa151e88700 1 --2- 172.21.15.28:0/4244198431 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa1340a2cb0 0x7fa1340a5170 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.516+0000 7fa151e88700 1 --2- 172.21.15.28:0/4244198431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa14c1071a0 0x7fa14c19d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.516+0000 7fa151e88700 1 --2- 172.21.15.28:0/4244198431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa14c101130 0x7fa14c198b00 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.516+0000 7fa151e88700 1 --2- 172.21.15.28:0/4244198431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa14c101af0 0x7fa14c199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.516+0000 7fa151e88700 1 -- 172.21.15.28:0/4244198431 >> 172.21.15.28:0/4244198431 conn(0x7fa14c0780e0 msgr2=0x7fa14c100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.516+0000 7fa151e88700 1 -- 172.21.15.28:0/4244198431 shutdown_connections 2024-04-25T01:32:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:00.516+0000 7fa151e88700 1 -- 172.21.15.28:0/4244198431 wait complete. 2024-04-25T01:32:00.763 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738450 got 34359738455 for osd.0 2024-04-25T01:32:00.763 DEBUG:teuthology.parallel:result is None 2024-04-25T01:32:01.015 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411399 got 77309411404 for osd.2 2024-04-25T01:32:01.015 DEBUG:teuthology.parallel:result is None 2024-04-25T01:32:01.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:01 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/4244198431' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T01:32:01.378 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:01 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/4244198431' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T01:32:01.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:01 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/4244198431' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T01:32:01.717 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574926 got 55834574929 for osd.1 2024-04-25T01:32:01.718 DEBUG:teuthology.parallel:result is None 2024-04-25T01:32:02.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:02 smithi028 ceph-mon[29390]: pgmap v343: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:02.385 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:02 smithi032 ceph-mon[32461]: pgmap v343: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:02.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:02 smithi130 ceph-mon[32702]: pgmap v343: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:02.696 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430259 got 206158430265 for osd.6 2024-04-25T01:32:02.696 DEBUG:teuthology.parallel:result is None 2024-04-25T01:32:04.196 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691898 got 171798691902 for osd.5 2024-04-25T01:32:04.196 DEBUG:teuthology.parallel:result is None 2024-04-25T01:32:04.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:04 smithi028 ceph-mon[29390]: pgmap v344: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:32:04.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:04 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3245369124' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T01:32:04.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.563+0000 7f67de456700 1 -- 172.21.15.28:0/2045814035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d8105dd0 msgr2=0x7f67d8106230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:04.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.563+0000 7f67de456700 1 --2- 172.21.15.28:0/2045814035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d8105dd0 0x7f67d8106230 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f67cc009990 tx=0x7f67cc009ca0 comp rx=0 tx=0).stop 2024-04-25T01:32:04.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.563+0000 7f67de456700 1 -- 172.21.15.28:0/2045814035 shutdown_connections 2024-04-25T01:32:04.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.563+0000 7f67de456700 1 --2- 172.21.15.28:0/2045814035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67d8106770 0x7f67d810e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:04.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.563+0000 7f67de456700 1 --2- 172.21.15.28:0/2045814035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67d81016d0 0x7f67d8101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:04.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.563+0000 7f67de456700 1 --2- 172.21.15.28:0/2045814035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d8105dd0 0x7f67d8106230 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:04.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.563+0000 7f67de456700 1 -- 172.21.15.28:0/2045814035 >> 172.21.15.28:0/2045814035 conn(0x7f67d8077ff0 msgr2=0x7f67d8078400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:04.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.563+0000 7f67de456700 1 -- 172.21.15.28:0/2045814035 shutdown_connections 2024-04-25T01:32:04.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.563+0000 7f67de456700 1 -- 172.21.15.28:0/2045814035 wait complete. 2024-04-25T01:32:04.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.563+0000 7f67de456700 1 Processor -- start 2024-04-25T01:32:04.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.563+0000 7f67de456700 1 -- start start 2024-04-25T01:32:04.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.564+0000 7f67de456700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d81016d0 0x7f67d819e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:04.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.564+0000 7f67de456700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67d8105dd0 0x7f67d819ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:04.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.564+0000 7f67de456700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67d8106770 0x7f67d81988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:04.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.564+0000 7f67de456700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f67d81114b0 con 0x7f67d81016d0 2024-04-25T01:32:04.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.564+0000 7f67de456700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67d8111330 con 0x7f67d8105dd0 2024-04-25T01:32:04.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.564+0000 7f67de456700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f67d8111630 con 0x7f67d8106770 2024-04-25T01:32:04.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.564+0000 7f67d7fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d81016d0 0x7f67d819e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:04.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.564+0000 7f67d7fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d81016d0 0x7f67d819e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35296/0 (socket says 172.21.15.28:35296) 2024-04-25T01:32:04.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.564+0000 7f67d7fff700 1 -- 172.21.15.28:0/3245369124 learned_addr learned my addr 172.21.15.28:0/3245369124 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:04.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.564+0000 7f67d77fe700 1 --2- 172.21.15.28:0/3245369124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67d8105dd0 0x7f67d819ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:04.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.564+0000 7f67dc9f3700 1 --2- 172.21.15.28:0/3245369124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67d8106770 0x7f67d81988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:04.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.564+0000 7f67d7fff700 1 -- 172.21.15.28:0/3245369124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67d8106770 msgr2=0x7f67d81988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:04.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.564+0000 7f67d7fff700 1 --2- 172.21.15.28:0/3245369124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67d8106770 0x7f67d81988a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:04.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.564+0000 7f67d7fff700 1 -- 172.21.15.28:0/3245369124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67d8105dd0 msgr2=0x7f67d819ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:04.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.564+0000 7f67d7fff700 1 --2- 172.21.15.28:0/3245369124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67d8105dd0 0x7f67d819ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:04.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.564+0000 7f67d7fff700 1 -- 172.21.15.28:0/3245369124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d8199000 con 0x7f67d81016d0 2024-04-25T01:32:04.457 INFO:teuthology.orchestra.run.smithi028.stdout:261993005103 2024-04-25T01:32:04.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.564+0000 7f67d7fff700 1 --2- 172.21.15.28:0/3245369124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d81016d0 0x7f67d819e6e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f67c800b530 tx=0x7f67c800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:04.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.564+0000 7f67d57fa700 1 -- 172.21.15.28:0/3245369124 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c8015020 con 0x7f67d81016d0 2024-04-25T01:32:04.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.564+0000 7f67de456700 1 -- 172.21.15.28:0/3245369124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67d81992f0 con 0x7f67d81016d0 2024-04-25T01:32:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.565+0000 7f67d57fa700 1 -- 172.21.15.28:0/3245369124 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67c8011d10 con 0x7f67d81016d0 2024-04-25T01:32:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.565+0000 7f67d57fa700 1 -- 172.21.15.28:0/3245369124 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c800f2e0 con 0x7f67d81016d0 2024-04-25T01:32:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.565+0000 7f67de456700 1 -- 172.21.15.28:0/3245369124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67d81a55b0 con 0x7f67d81016d0 2024-04-25T01:32:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.565+0000 7f67de456700 1 -- 172.21.15.28:0/3245369124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67d8068f20 con 0x7f67d81016d0 2024-04-25T01:32:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.567+0000 7f67d57fa700 1 -- 172.21.15.28:0/3245369124 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f67c8018020 con 0x7f67d81016d0 2024-04-25T01:32:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.567+0000 7f67d57fa700 1 --2- 172.21.15.28:0/3245369124 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67c00a2d30 0x7f67c00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.568+0000 7f67d77fe700 1 --2- 172.21.15.28:0/3245369124 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67c00a2d30 0x7f67c00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.568+0000 7f67d57fa700 1 -- 172.21.15.28:0/3245369124 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f67c80cf1c0 con 0x7f67d81016d0 2024-04-25T01:32:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.568+0000 7f67d77fe700 1 --2- 172.21.15.28:0/3245369124 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67c00a2d30 0x7f67c00a51f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f67cc00b5c0 tx=0x7f67cc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.571+0000 7f67d57fa700 1 -- 172.21.15.28:0/3245369124 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f67c801f020 con 0x7f67d81016d0 2024-04-25T01:32:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.786+0000 7f67de456700 1 -- 172.21.15.28:0/3245369124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f67d8199dd0 con 0x7f67d81016d0 2024-04-25T01:32:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.786+0000 7f67d57fa700 1 -- 172.21.15.28:0/3245369124 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f67c80cce20 con 0x7f67d81016d0 2024-04-25T01:32:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.789+0000 7f67de456700 1 -- 172.21.15.28:0/3245369124 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67c00a2d30 msgr2=0x7f67c00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.789+0000 7f67de456700 1 --2- 172.21.15.28:0/3245369124 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67c00a2d30 0x7f67c00a51f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f67cc00b5c0 tx=0x7f67cc005960 comp rx=0 tx=0).stop 2024-04-25T01:32:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.789+0000 7f67de456700 1 -- 172.21.15.28:0/3245369124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d81016d0 msgr2=0x7f67d819e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.789+0000 7f67de456700 1 --2- 172.21.15.28:0/3245369124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d81016d0 0x7f67d819e6e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f67c800b530 tx=0x7f67c800b840 comp rx=0 tx=0).stop 2024-04-25T01:32:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.789+0000 7f67de456700 1 -- 172.21.15.28:0/3245369124 shutdown_connections 2024-04-25T01:32:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.789+0000 7f67de456700 1 --2- 172.21.15.28:0/3245369124 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67c00a2d30 0x7f67c00a51f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.789+0000 7f67de456700 1 --2- 172.21.15.28:0/3245369124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67d8106770 0x7f67d81988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.789+0000 7f67de456700 1 --2- 172.21.15.28:0/3245369124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d81016d0 0x7f67d819e6e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.789+0000 7f67de456700 1 --2- 172.21.15.28:0/3245369124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67d8105dd0 0x7f67d819ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.789+0000 7f67de456700 1 -- 172.21.15.28:0/3245369124 >> 172.21.15.28:0/3245369124 conn(0x7f67d8077ff0 msgr2=0x7f67d806acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.789+0000 7f67de456700 1 -- 172.21.15.28:0/3245369124 shutdown_connections 2024-04-25T01:32:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:03.789+0000 7f67de456700 1 -- 172.21.15.28:0/3245369124 wait complete. 2024-04-25T01:32:04.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:04 smithi130 ceph-mon[32702]: pgmap v344: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:32:04.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:04 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3245369124' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T01:32:04.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:04 smithi032 ceph-mon[32461]: pgmap v344: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:32:04.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:04 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3245369124' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T01:32:05.203 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266733 got 227633266739 for osd.7 2024-04-25T01:32:05.203 DEBUG:teuthology.parallel:result is None 2024-04-25T01:32:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.417+0000 7f3981e3c700 1 -- 172.21.15.28:0/1770011602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f397c074d80 msgr2=0x7f397c10e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.417+0000 7f3981e3c700 1 --2- 172.21.15.28:0/1770011602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f397c074d80 0x7f397c10e6f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3970009990 tx=0x7f3970009ca0 comp rx=0 tx=0).stop 2024-04-25T01:32:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.417+0000 7f3981e3c700 1 -- 172.21.15.28:0/1770011602 shutdown_connections 2024-04-25T01:32:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.417+0000 7f3981e3c700 1 --2- 172.21.15.28:0/1770011602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f397c10ec30 0x7f397c111020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.417+0000 7f3981e3c700 1 --2- 172.21.15.28:0/1770011602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f397c074460 0x7f397c074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.417+0000 7f3981e3c700 1 --2- 172.21.15.28:0/1770011602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f397c074d80 0x7f397c10e6f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.417+0000 7f3981e3c700 1 -- 172.21.15.28:0/1770011602 >> 172.21.15.28:0/1770011602 conn(0x7f397c079390 msgr2=0x7f397c0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.417+0000 7f3981e3c700 1 -- 172.21.15.28:0/1770011602 shutdown_connections 2024-04-25T01:32:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.417+0000 7f3981e3c700 1 -- 172.21.15.28:0/1770011602 wait complete. 2024-04-25T01:32:05.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.417+0000 7f3981e3c700 1 Processor -- start 2024-04-25T01:32:05.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.418+0000 7f3981e3c700 1 -- start start 2024-04-25T01:32:05.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.418+0000 7f3981e3c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f397c074460 0x7f397c1a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:05.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.418+0000 7f3981e3c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f397c074d80 0x7f397c1a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:05.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.418+0000 7f3981e3c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f397c10ec30 0x7f397c19aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:05.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.418+0000 7f3981e3c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f397c1147a0 con 0x7f397c10ec30 2024-04-25T01:32:05.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.418+0000 7f3981e3c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f397c114620 con 0x7f397c074460 2024-04-25T01:32:05.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.418+0000 7f3981e3c700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f397c114920 con 0x7f397c074d80 2024-04-25T01:32:05.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.418+0000 7f397bfff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f397c10ec30 0x7f397c19aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:05.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.418+0000 7f397bfff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f397c10ec30 0x7f397c19aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35318/0 (socket says 172.21.15.28:35318) 2024-04-25T01:32:05.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.418+0000 7f397bfff700 1 -- 172.21.15.28:0/1039547635 learned_addr learned my addr 172.21.15.28:0/1039547635 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:05.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.418+0000 7f397affd700 1 --2- 172.21.15.28:0/1039547635 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f397c074d80 0x7f397c1a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:05.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.418+0000 7f397b7fe700 1 --2- 172.21.15.28:0/1039547635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f397c074460 0x7f397c1a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:05.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.418+0000 7f397bfff700 1 -- 172.21.15.28:0/1039547635 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f397c074d80 msgr2=0x7f397c1a0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:05.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.418+0000 7f397bfff700 1 --2- 172.21.15.28:0/1039547635 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f397c074d80 0x7f397c1a0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:05.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.418+0000 7f397bfff700 1 -- 172.21.15.28:0/1039547635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f397c074460 msgr2=0x7f397c1a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:05.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.418+0000 7f397bfff700 1 --2- 172.21.15.28:0/1039547635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f397c074460 0x7f397c1a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:05.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.418+0000 7f397bfff700 1 -- 172.21.15.28:0/1039547635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f397c19b1e0 con 0x7f397c10ec30 2024-04-25T01:32:05.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.418+0000 7f397b7fe700 1 --2- 172.21.15.28:0/1039547635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f397c074460 0x7f397c1a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:05.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.419+0000 7f397bfff700 1 --2- 172.21.15.28:0/1039547635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f397c10ec30 0x7f397c19aa80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3974007ee0 tx=0x7f397400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:05.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.419+0000 7f3978ff9700 1 -- 172.21.15.28:0/1039547635 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397400dbd0 con 0x7f397c10ec30 2024-04-25T01:32:05.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.419+0000 7f3981e3c700 1 -- 172.21.15.28:0/1039547635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f397c19b4d0 con 0x7f397c10ec30 2024-04-25T01:32:05.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.419+0000 7f3978ff9700 1 -- 172.21.15.28:0/1039547635 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f397400dd70 con 0x7f397c10ec30 2024-04-25T01:32:05.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.419+0000 7f3978ff9700 1 -- 172.21.15.28:0/1039547635 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3974005a60 con 0x7f397c10ec30 2024-04-25T01:32:05.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.419+0000 7f3981e3c700 1 -- 172.21.15.28:0/1039547635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f397c1a7790 con 0x7f397c10ec30 2024-04-25T01:32:05.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.420+0000 7f3981e3c700 1 -- 172.21.15.28:0/1039547635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f397c068f20 con 0x7f397c10ec30 2024-04-25T01:32:05.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.421+0000 7f3978ff9700 1 -- 172.21.15.28:0/1039547635 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f397401a020 con 0x7f397c10ec30 2024-04-25T01:32:05.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.422+0000 7f3978ff9700 1 --2- 172.21.15.28:0/1039547635 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f39640a2cb0 0x7f39640a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:05.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.422+0000 7f3978ff9700 1 -- 172.21.15.28:0/1039547635 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f39740cecb0 con 0x7f397c10ec30 2024-04-25T01:32:05.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.422+0000 7f397b7fe700 1 --2- 172.21.15.28:0/1039547635 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f39640a2cb0 0x7f39640a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:05.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.422+0000 7f397b7fe700 1 --2- 172.21.15.28:0/1039547635 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f39640a2cb0 0x7f39640a5170 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f397c19bbc0 tx=0x7f396c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:05.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.426+0000 7f3978ff9700 1 -- 172.21.15.28:0/1039547635 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3974010070 con 0x7f397c10ec30 2024-04-25T01:32:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.634+0000 7f3981e3c700 1 -- 172.21.15.28:0/1039547635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f397c02db30 con 0x7f397c10ec30 2024-04-25T01:32:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.635+0000 7f3978ff9700 1 -- 172.21.15.28:0/1039547635 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f397409b6f0 con 0x7f397c10ec30 2024-04-25T01:32:05.613 INFO:teuthology.orchestra.run.smithi028.stdout:146028888132 2024-04-25T01:32:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.637+0000 7f3981e3c700 1 -- 172.21.15.28:0/1039547635 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f39640a2cb0 msgr2=0x7f39640a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.637+0000 7f3981e3c700 1 --2- 172.21.15.28:0/1039547635 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f39640a2cb0 0x7f39640a5170 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f397c19bbc0 tx=0x7f396c008040 comp rx=0 tx=0).stop 2024-04-25T01:32:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.637+0000 7f3981e3c700 1 -- 172.21.15.28:0/1039547635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f397c10ec30 msgr2=0x7f397c19aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.637+0000 7f3981e3c700 1 --2- 172.21.15.28:0/1039547635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f397c10ec30 0x7f397c19aa80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3974007ee0 tx=0x7f397400d3b0 comp rx=0 tx=0).stop 2024-04-25T01:32:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.637+0000 7f3981e3c700 1 -- 172.21.15.28:0/1039547635 shutdown_connections 2024-04-25T01:32:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.637+0000 7f3981e3c700 1 --2- 172.21.15.28:0/1039547635 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f39640a2cb0 0x7f39640a5170 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.637+0000 7f3981e3c700 1 --2- 172.21.15.28:0/1039547635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f397c10ec30 0x7f397c19aa80 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.637+0000 7f3981e3c700 1 --2- 172.21.15.28:0/1039547635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f397c074460 0x7f397c1a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.637+0000 7f3981e3c700 1 --2- 172.21.15.28:0/1039547635 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f397c074d80 0x7f397c1a0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.637+0000 7f3981e3c700 1 -- 172.21.15.28:0/1039547635 >> 172.21.15.28:0/1039547635 conn(0x7f397c079390 msgr2=0x7f397c0ff400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.637+0000 7f3981e3c700 1 -- 172.21.15.28:0/1039547635 shutdown_connections 2024-04-25T01:32:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:05.637+0000 7f3981e3c700 1 -- 172.21.15.28:0/1039547635 wait complete. 2024-04-25T01:32:05.825 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149764 got 111669149768 for osd.3 2024-04-25T01:32:05.825 DEBUG:teuthology.parallel:result is None 2024-04-25T01:32:06.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:06 smithi028 ceph-mon[29390]: pgmap v345: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:06.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:06 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1039547635' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T01:32:06.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:06 smithi130 ceph-mon[32702]: pgmap v345: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:06.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:06 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1039547635' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T01:32:06.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:06 smithi032 ceph-mon[32461]: pgmap v345: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:06.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:06 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1039547635' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T01:32:06.628 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:32:06.629 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:32:06.659 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888126 got 146028888132 for osd.4 2024-04-25T01:32:06.659 DEBUG:teuthology.parallel:result is None 2024-04-25T01:32:08.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:08 smithi028 ceph-mon[29390]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:08.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:08 smithi130 ceph-mon[32702]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:08.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:08 smithi032 ceph-mon[32461]: pgmap v346: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:09.450 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005099 got 261993005103 for osd.8 2024-04-25T01:32:09.450 DEBUG:teuthology.parallel:result is None 2024-04-25T01:32:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.946+0000 7f86940e9700 1 -- 172.21.15.28:0/200732168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f868c1022c0 msgr2=0x7f868c102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.946+0000 7f86940e9700 1 --2- 172.21.15.28:0/200732168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f868c1022c0 0x7f868c102740 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f867c009990 tx=0x7f867c009ca0 comp rx=0 tx=0).stop 2024-04-25T01:32:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.946+0000 7f86940e9700 1 -- 172.21.15.28:0/200732168 shutdown_connections 2024-04-25T01:32:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.946+0000 7f86940e9700 1 --2- 172.21.15.28:0/200732168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f868c102c80 0x7f868c10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.946+0000 7f86940e9700 1 --2- 172.21.15.28:0/200732168 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f868c108360 0x7f868c108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.946+0000 7f86940e9700 1 --2- 172.21.15.28:0/200732168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f868c1022c0 0x7f868c102740 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.946+0000 7f86940e9700 1 -- 172.21.15.28:0/200732168 >> 172.21.15.28:0/200732168 conn(0x7f868c079420 msgr2=0x7f868c079830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.946+0000 7f86940e9700 1 -- 172.21.15.28:0/200732168 shutdown_connections 2024-04-25T01:32:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.946+0000 7f86940e9700 1 -- 172.21.15.28:0/200732168 wait complete. 2024-04-25T01:32:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.946+0000 7f86940e9700 1 Processor -- start 2024-04-25T01:32:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.946+0000 7f86940e9700 1 -- start start 2024-04-25T01:32:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.947+0000 7f86940e9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f868c1022c0 0x7f868c1a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.947+0000 7f86940e9700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f868c102c80 0x7f868c1a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.947+0000 7f86940e9700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f868c108360 0x7f868c19aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.947+0000 7f86940e9700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f868c112580 con 0x7f868c1022c0 2024-04-25T01:32:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.947+0000 7f86940e9700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f868c112400 con 0x7f868c102c80 2024-04-25T01:32:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.947+0000 7f86940e9700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f868c112700 con 0x7f868c108360 2024-04-25T01:32:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.947+0000 7f8691e85700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f868c1022c0 0x7f868c1a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.947+0000 7f8691e85700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f868c1022c0 0x7f868c1a08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48406/0 (socket says 172.21.15.28:48406) 2024-04-25T01:32:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.947+0000 7f8691e85700 1 -- 172.21.15.28:0/3695695278 learned_addr learned my addr 172.21.15.28:0/3695695278 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.947+0000 7f8692686700 1 --2- 172.21.15.28:0/3695695278 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f868c108360 0x7f868c19aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.947+0000 7f8691684700 1 --2- 172.21.15.28:0/3695695278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f868c102c80 0x7f868c1a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.947+0000 7f8691e85700 1 -- 172.21.15.28:0/3695695278 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f868c108360 msgr2=0x7f868c19aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.947+0000 7f8691e85700 1 --2- 172.21.15.28:0/3695695278 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f868c108360 0x7f868c19aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.947+0000 7f8691e85700 1 -- 172.21.15.28:0/3695695278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f868c102c80 msgr2=0x7f868c1a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.947+0000 7f8691e85700 1 --2- 172.21.15.28:0/3695695278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f868c102c80 0x7f868c1a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.947+0000 7f8691e85700 1 -- 172.21.15.28:0/3695695278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f868c19b240 con 0x7f868c1022c0 2024-04-25T01:32:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.947+0000 7f8691684700 1 --2- 172.21.15.28:0/3695695278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f868c102c80 0x7f868c1a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.948+0000 7f8691e85700 1 --2- 172.21.15.28:0/3695695278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f868c1022c0 0x7f868c1a08d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f868800d950 tx=0x7f868800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.948+0000 7f8682ffd700 1 -- 172.21.15.28:0/3695695278 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8688016070 con 0x7f868c1022c0 2024-04-25T01:32:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.948+0000 7f86940e9700 1 -- 172.21.15.28:0/3695695278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f868c19b530 con 0x7f868c1022c0 2024-04-25T01:32:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.948+0000 7f8682ffd700 1 -- 172.21.15.28:0/3695695278 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8688009e20 con 0x7f868c1022c0 2024-04-25T01:32:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.948+0000 7f8682ffd700 1 -- 172.21.15.28:0/3695695278 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f868800f620 con 0x7f868c1022c0 2024-04-25T01:32:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.948+0000 7f86940e9700 1 -- 172.21.15.28:0/3695695278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f868c1a77a0 con 0x7f868c1022c0 2024-04-25T01:32:09.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.949+0000 7f86940e9700 1 -- 172.21.15.28:0/3695695278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f868c068f20 con 0x7f868c1022c0 2024-04-25T01:32:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.950+0000 7f8682ffd700 1 -- 172.21.15.28:0/3695695278 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f8688019070 con 0x7f868c1022c0 2024-04-25T01:32:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.951+0000 7f8682ffd700 1 --2- 172.21.15.28:0/3695695278 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f86780a2cb0 0x7f86780a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.951+0000 7f8691684700 1 --2- 172.21.15.28:0/3695695278 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f86780a2cb0 0x7f86780a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.951+0000 7f8682ffd700 1 -- 172.21.15.28:0/3695695278 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f86880cf410 con 0x7f868c1022c0 2024-04-25T01:32:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.952+0000 7f8691684700 1 --2- 172.21.15.28:0/3695695278 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f86780a2cb0 0x7f86780a5170 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f867c006010 tx=0x7f867c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:09.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:09.957+0000 7f8682ffd700 1 -- 172.21.15.28:0/3695695278 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f868809beb0 con 0x7f868c1022c0 2024-04-25T01:32:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:10.171+0000 7f86940e9700 1 -- 172.21.15.28:0/3695695278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f868c02db30 con 0x7f868c1022c0 2024-04-25T01:32:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:10.172+0000 7f8682ffd700 1 -- 172.21.15.28:0/3695695278 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f868809b7b0 con 0x7f868c1022c0 2024-04-25T01:32:10.150 INFO:teuthology.orchestra.run.smithi028.stdout:296352743468 2024-04-25T01:32:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:10.174+0000 7f86940e9700 1 -- 172.21.15.28:0/3695695278 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f86780a2cb0 msgr2=0x7f86780a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:10.174+0000 7f86940e9700 1 --2- 172.21.15.28:0/3695695278 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f86780a2cb0 0x7f86780a5170 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f867c006010 tx=0x7f867c00b540 comp rx=0 tx=0).stop 2024-04-25T01:32:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:10.174+0000 7f86940e9700 1 -- 172.21.15.28:0/3695695278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f868c1022c0 msgr2=0x7f868c1a08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:10.174+0000 7f86940e9700 1 --2- 172.21.15.28:0/3695695278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f868c1022c0 0x7f868c1a08d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f868800d950 tx=0x7f868800dc60 comp rx=0 tx=0).stop 2024-04-25T01:32:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:10.174+0000 7f86940e9700 1 -- 172.21.15.28:0/3695695278 shutdown_connections 2024-04-25T01:32:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:10.174+0000 7f86940e9700 1 --2- 172.21.15.28:0/3695695278 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f86780a2cb0 0x7f86780a5170 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:10.174+0000 7f86940e9700 1 --2- 172.21.15.28:0/3695695278 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f868c108360 0x7f868c19aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:10.174+0000 7f86940e9700 1 --2- 172.21.15.28:0/3695695278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f868c1022c0 0x7f868c1a08d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:10.174+0000 7f86940e9700 1 --2- 172.21.15.28:0/3695695278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f868c102c80 0x7f868c1a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:10.174+0000 7f86940e9700 1 -- 172.21.15.28:0/3695695278 >> 172.21.15.28:0/3695695278 conn(0x7f868c079420 msgr2=0x7f868c074850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:10.174+0000 7f86940e9700 1 -- 172.21.15.28:0/3695695278 shutdown_connections 2024-04-25T01:32:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:10.174+0000 7f86940e9700 1 -- 172.21.15.28:0/3695695278 wait complete. 2024-04-25T01:32:10.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:10 smithi028 ceph-mon[29390]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:10.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:10 smithi130 ceph-mon[32702]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:10.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:10 smithi032 ceph-mon[32461]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:10.870 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743462 got 296352743468 for osd.9 2024-04-25T01:32:10.870 DEBUG:teuthology.parallel:result is None 2024-04-25T01:32:11.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:11 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3695695278' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T01:32:11.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.141+0000 7f9ee4af2700 1 -- 172.21.15.28:0/728957522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee006b280 msgr2=0x7f9ee010e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:11.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.141+0000 7f9ee4af2700 1 --2- 172.21.15.28:0/728957522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee006b280 0x7f9ee010e7d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009990 tx=0x7f9ec8009ca0 comp rx=0 tx=0).stop 2024-04-25T01:32:11.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.141+0000 7f9ee4af2700 1 -- 172.21.15.28:0/728957522 shutdown_connections 2024-04-25T01:32:11.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.141+0000 7f9ee4af2700 1 --2- 172.21.15.28:0/728957522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ee010ed10 0x7f9ee0111100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:11.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.141+0000 7f9ee4af2700 1 --2- 172.21.15.28:0/728957522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ee006a960 0x7f9ee006ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:11.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.141+0000 7f9ee4af2700 1 --2- 172.21.15.28:0/728957522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee006b280 0x7f9ee010e7d0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:11.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.141+0000 7f9ee4af2700 1 -- 172.21.15.28:0/728957522 >> 172.21.15.28:0/728957522 conn(0x7f9ee00780e0 msgr2=0x7f9ee00784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:11.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.141+0000 7f9ee4af2700 1 -- 172.21.15.28:0/728957522 shutdown_connections 2024-04-25T01:32:11.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.141+0000 7f9ee4af2700 1 -- 172.21.15.28:0/728957522 wait complete. 2024-04-25T01:32:11.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.142+0000 7f9ee4af2700 1 Processor -- start 2024-04-25T01:32:11.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.142+0000 7f9ee4af2700 1 -- start start 2024-04-25T01:32:11.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.142+0000 7f9ee4af2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ee006a960 0x7f9ee01a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:11.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.142+0000 7f9ee4af2700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee006b280 0x7f9ee01a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:11.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.142+0000 7f9ee4af2700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ee010ed10 0x7f9ee019aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:11.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.142+0000 7f9ee4af2700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee0114740 con 0x7f9ee006b280 2024-04-25T01:32:11.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.142+0000 7f9ee4af2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee01145c0 con 0x7f9ee006a960 2024-04-25T01:32:11.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.142+0000 7f9ee4af2700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee01148c0 con 0x7f9ee010ed10 2024-04-25T01:32:11.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.142+0000 7f9eddd9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee006b280 0x7f9ee01a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:11.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.142+0000 7f9ede59c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ee006a960 0x7f9ee01a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:11.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.142+0000 7f9eddd9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee006b280 0x7f9ee01a0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48414/0 (socket says 172.21.15.28:48414) 2024-04-25T01:32:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.142+0000 7f9eddd9b700 1 -- 172.21.15.28:0/3648206455 learned_addr learned my addr 172.21.15.28:0/3648206455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.143+0000 7f9eded9d700 1 --2- 172.21.15.28:0/3648206455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ee010ed10 0x7f9ee019aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.143+0000 7f9ede59c700 1 -- 172.21.15.28:0/3648206455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ee010ed10 msgr2=0x7f9ee019aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.143+0000 7f9ede59c700 1 --2- 172.21.15.28:0/3648206455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ee010ed10 0x7f9ee019aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.143+0000 7f9ede59c700 1 -- 172.21.15.28:0/3648206455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee006b280 msgr2=0x7f9ee01a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.143+0000 7f9ede59c700 1 --2- 172.21.15.28:0/3648206455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee006b280 0x7f9ee01a0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.143+0000 7f9ede59c700 1 -- 172.21.15.28:0/3648206455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee019b1e0 con 0x7f9ee006a960 2024-04-25T01:32:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.143+0000 7f9eddd9b700 1 --2- 172.21.15.28:0/3648206455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee006b280 0x7f9ee01a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.143+0000 7f9ede59c700 1 --2- 172.21.15.28:0/3648206455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ee006a960 0x7f9ee01a0870 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9ed000d950 tx=0x7f9ed000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.143+0000 7f9ed77fe700 1 -- 172.21.15.28:0/3648206455 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0016070 con 0x7f9ee006a960 2024-04-25T01:32:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.143+0000 7f9ed77fe700 1 -- 172.21.15.28:0/3648206455 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ed000fd70 con 0x7f9ee006a960 2024-04-25T01:32:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.143+0000 7f9ed77fe700 1 -- 172.21.15.28:0/3648206455 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed000e3b0 con 0x7f9ee006a960 2024-04-25T01:32:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.143+0000 7f9ee4af2700 1 -- 172.21.15.28:0/3648206455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ee019b4d0 con 0x7f9ee006a960 2024-04-25T01:32:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.143+0000 7f9ee4af2700 1 -- 172.21.15.28:0/3648206455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ee01a7740 con 0x7f9ee006a960 2024-04-25T01:32:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.144+0000 7f9ee4af2700 1 -- 172.21.15.28:0/3648206455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ee0068f20 con 0x7f9ee006a960 2024-04-25T01:32:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.146+0000 7f9ed77fe700 1 -- 172.21.15.28:0/3648206455 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f9ed0019070 con 0x7f9ee006a960 2024-04-25T01:32:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.147+0000 7f9ed77fe700 1 --2- 172.21.15.28:0/3648206455 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9ecc0a2cb0 0x7f9ecc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.147+0000 7f9ed77fe700 1 -- 172.21.15.28:0/3648206455 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f9ed00cf180 con 0x7f9ee006a960 2024-04-25T01:32:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.147+0000 7f9eddd9b700 1 --2- 172.21.15.28:0/3648206455 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9ecc0a2cb0 0x7f9ecc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.147+0000 7f9eddd9b700 1 --2- 172.21.15.28:0/3648206455 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9ecc0a2cb0 0x7f9ecc0a5170 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0101070 tx=0x7f9ec8005c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.150+0000 7f9ed77fe700 1 -- 172.21.15.28:0/3648206455 <== mon.2 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9ed0010460 con 0x7f9ee006a960 2024-04-25T01:32:11.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.370+0000 7f9ee4af2700 1 -- 172.21.15.28:0/3648206455 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f9ee019bf60 con 0x7f9ee006a960 2024-04-25T01:32:11.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.370+0000 7f9ed77fe700 1 -- 172.21.15.28:0/3648206455 <== mon.2 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f9ed009bc20 con 0x7f9ee006a960 2024-04-25T01:32:11.348 INFO:teuthology.orchestra.run.smithi028.stdout:317827579942 2024-04-25T01:32:11.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.372+0000 7f9ee4af2700 1 -- 172.21.15.28:0/3648206455 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9ecc0a2cb0 msgr2=0x7f9ecc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:11.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.372+0000 7f9ee4af2700 1 --2- 172.21.15.28:0/3648206455 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9ecc0a2cb0 0x7f9ecc0a5170 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0101070 tx=0x7f9ec8005c00 comp rx=0 tx=0).stop 2024-04-25T01:32:11.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.372+0000 7f9ee4af2700 1 -- 172.21.15.28:0/3648206455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ee006a960 msgr2=0x7f9ee01a0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:11.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.372+0000 7f9ee4af2700 1 --2- 172.21.15.28:0/3648206455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ee006a960 0x7f9ee01a0870 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9ed000d950 tx=0x7f9ed000dc60 comp rx=0 tx=0).stop 2024-04-25T01:32:11.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.373+0000 7f9ee4af2700 1 -- 172.21.15.28:0/3648206455 shutdown_connections 2024-04-25T01:32:11.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.373+0000 7f9ee4af2700 1 --2- 172.21.15.28:0/3648206455 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9ecc0a2cb0 0x7f9ecc0a5170 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:11.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.373+0000 7f9ee4af2700 1 --2- 172.21.15.28:0/3648206455 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9ee010ed10 0x7f9ee019aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:11.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.373+0000 7f9ee4af2700 1 --2- 172.21.15.28:0/3648206455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9ee006a960 0x7f9ee01a0870 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:11.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.373+0000 7f9ee4af2700 1 --2- 172.21.15.28:0/3648206455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ee006b280 0x7f9ee01a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:11.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.373+0000 7f9ee4af2700 1 -- 172.21.15.28:0/3648206455 >> 172.21.15.28:0/3648206455 conn(0x7f9ee00780e0 msgr2=0x7f9ee0110160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:11.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.373+0000 7f9ee4af2700 1 -- 172.21.15.28:0/3648206455 shutdown_connections 2024-04-25T01:32:11.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:11.373+0000 7f9ee4af2700 1 -- 172.21.15.28:0/3648206455 wait complete. 2024-04-25T01:32:11.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:11 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3695695278' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T01:32:11.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:11 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3695695278' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T01:32:12.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:12 smithi028 ceph-mon[29390]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:12.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:12 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:32:12.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:12 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:32:12.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:12 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3648206455' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T01:32:12.142 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579936 got 317827579942 for osd.10 2024-04-25T01:32:12.142 DEBUG:teuthology.parallel:result is None 2024-04-25T01:32:12.143 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T01:32:12.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T01:32:12.398 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:32:12.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:12 smithi130 ceph-mon[32702]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:12.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:12 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:32:12.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:12 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:32:12.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:12 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3648206455' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T01:32:12.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:12 smithi032 ceph-mon[32461]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:12.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:12 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:32:12.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:12 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:32:12.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:12 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3648206455' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T01:32:13.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:32:13.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:32:13.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:13 smithi028 ceph-mon[29390]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:32:13.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:32:13.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:32:13.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:13 smithi130 ceph-mon[32702]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:32:13.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:32:13.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:32:13.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:13 smithi032 ceph-mon[32461]: from='mgr.14152 172.21.15.28:0/2082289020' entity='mgr.x' 2024-04-25T01:32:14.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:14 smithi028 ceph-mon[29390]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:32:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.214+0000 7f8743740700 1 -- 172.21.15.28:0/3711969873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f873c1071a0 msgr2=0x7f873c107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.214+0000 7f8743740700 1 --2- 172.21.15.28:0/3711969873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f873c1071a0 0x7f873c107580 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8738009990 tx=0x7f8738009ca0 comp rx=0 tx=0).stop 2024-04-25T01:32:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.214+0000 7f8743740700 1 -- 172.21.15.28:0/3711969873 shutdown_connections 2024-04-25T01:32:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.214+0000 7f8743740700 1 --2- 172.21.15.28:0/3711969873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f873c101af0 0x7f873c10dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.214+0000 7f8743740700 1 --2- 172.21.15.28:0/3711969873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f873c1071a0 0x7f873c107580 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.214+0000 7f8743740700 1 --2- 172.21.15.28:0/3711969873 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f873c101130 0x7f873c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.214+0000 7f8743740700 1 -- 172.21.15.28:0/3711969873 >> 172.21.15.28:0/3711969873 conn(0x7f873c0780e0 msgr2=0x7f873c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.214+0000 7f8743740700 1 -- 172.21.15.28:0/3711969873 shutdown_connections 2024-04-25T01:32:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.214+0000 7f8743740700 1 -- 172.21.15.28:0/3711969873 wait complete. 2024-04-25T01:32:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.215+0000 7f8743740700 1 Processor -- start 2024-04-25T01:32:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.215+0000 7f8743740700 1 -- start start 2024-04-25T01:32:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.215+0000 7f8743740700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f873c101130 0x7f873c10ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.215+0000 7f8743740700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f873c101af0 0x7f873c10efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.215+0000 7f8743740700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f873c1071a0 0x7f873c10f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.215+0000 7f8743740700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f873c112590 con 0x7f873c101130 2024-04-25T01:32:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.215+0000 7f8743740700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f873c112410 con 0x7f873c1071a0 2024-04-25T01:32:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.215+0000 7f8743740700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f873c112710 con 0x7f873c101af0 2024-04-25T01:32:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.215+0000 7f87414dc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f873c101130 0x7f873c10ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.215+0000 7f87414dc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f873c101130 0x7f873c10ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48420/0 (socket says 172.21.15.28:48420) 2024-04-25T01:32:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.215+0000 7f87414dc700 1 -- 172.21.15.28:0/1239493889 learned_addr learned my addr 172.21.15.28:0/1239493889 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.215+0000 7f8740cdb700 1 --2- 172.21.15.28:0/1239493889 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f873c101af0 0x7f873c10efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.215+0000 7f8741cdd700 1 --2- 172.21.15.28:0/1239493889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f873c1071a0 0x7f873c10f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.215+0000 7f87414dc700 1 -- 172.21.15.28:0/1239493889 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f873c101af0 msgr2=0x7f873c10efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.215+0000 7f87414dc700 1 --2- 172.21.15.28:0/1239493889 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f873c101af0 0x7f873c10efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.215+0000 7f87414dc700 1 -- 172.21.15.28:0/1239493889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f873c1071a0 msgr2=0x7f873c10f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.215+0000 7f87414dc700 1 --2- 172.21.15.28:0/1239493889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f873c1071a0 0x7f873c10f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:14.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.215+0000 7f87414dc700 1 -- 172.21.15.28:0/1239493889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f873c1a6ff0 con 0x7f873c101130 2024-04-25T01:32:14.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.215+0000 7f87414dc700 1 --2- 172.21.15.28:0/1239493889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f873c101130 0x7f873c10ea70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f873800b5c0 tx=0x7f87380047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:14.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.216+0000 7f87327fc700 1 -- 172.21.15.28:0/1239493889 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8738027070 con 0x7f873c101130 2024-04-25T01:32:14.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.216+0000 7f8743740700 1 -- 172.21.15.28:0/1239493889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f873c1a7280 con 0x7f873c101130 2024-04-25T01:32:14.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.216+0000 7f87327fc700 1 -- 172.21.15.28:0/1239493889 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87380040b0 con 0x7f873c101130 2024-04-25T01:32:14.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.216+0000 7f87327fc700 1 -- 172.21.15.28:0/1239493889 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873800f460 con 0x7f873c101130 2024-04-25T01:32:14.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.216+0000 7f8743740700 1 -- 172.21.15.28:0/1239493889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f873c1a7730 con 0x7f873c101130 2024-04-25T01:32:14.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.217+0000 7f8743740700 1 -- 172.21.15.28:0/1239493889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f873c068f20 con 0x7f873c101130 2024-04-25T01:32:14.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.218+0000 7f87327fc700 1 -- 172.21.15.28:0/1239493889 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f873802a070 con 0x7f873c101130 2024-04-25T01:32:14.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.219+0000 7f87327fc700 1 --2- 172.21.15.28:0/1239493889 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f87280a2cb0 0x7f87280a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:14.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.219+0000 7f8740cdb700 1 --2- 172.21.15.28:0/1239493889 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f87280a2cb0 0x7f87280a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:14.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.219+0000 7f87327fc700 1 -- 172.21.15.28:0/1239493889 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f87380d0a40 con 0x7f873c101130 2024-04-25T01:32:14.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.219+0000 7f8740cdb700 1 --2- 172.21.15.28:0/1239493889 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f87280a2cb0 0x7f87280a5170 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f872c006fd0 tx=0x7f872c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:14.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.223+0000 7f87327fc700 1 -- 172.21.15.28:0/1239493889 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8738031070 con 0x7f873c101130 2024-04-25T01:32:14.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.429+0000 7f8743740700 1 -- 172.21.15.28:0/1239493889 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f873c062670 con 0x7f87280a2cb0 2024-04-25T01:32:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.439+0000 7f87327fc700 1 -- 172.21.15.28:0/1239493889 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340420 (secure 0 0 0) 0x7f873c062670 con 0x7f87280a2cb0 2024-04-25T01:32:14.422 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:32:14.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.443+0000 7f8743740700 1 -- 172.21.15.28:0/1239493889 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f87280a2cb0 msgr2=0x7f87280a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:14.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.443+0000 7f8743740700 1 --2- 172.21.15.28:0/1239493889 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f87280a2cb0 0x7f87280a5170 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f872c006fd0 tx=0x7f872c008040 comp rx=0 tx=0).stop 2024-04-25T01:32:14.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.443+0000 7f8743740700 1 -- 172.21.15.28:0/1239493889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f873c101130 msgr2=0x7f873c10ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:14.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.443+0000 7f8743740700 1 --2- 172.21.15.28:0/1239493889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f873c101130 0x7f873c10ea70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f873800b5c0 tx=0x7f87380047b0 comp rx=0 tx=0).stop 2024-04-25T01:32:14.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.443+0000 7f8743740700 1 -- 172.21.15.28:0/1239493889 shutdown_connections 2024-04-25T01:32:14.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.443+0000 7f8743740700 1 --2- 172.21.15.28:0/1239493889 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f87280a2cb0 0x7f87280a5170 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:14.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.443+0000 7f8743740700 1 --2- 172.21.15.28:0/1239493889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f873c1071a0 0x7f873c10f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:14.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.444+0000 7f8743740700 1 --2- 172.21.15.28:0/1239493889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f873c101130 0x7f873c10ea70 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:14.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.444+0000 7f8743740700 1 --2- 172.21.15.28:0/1239493889 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f873c101af0 0x7f873c10efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:14.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.444+0000 7f8743740700 1 -- 172.21.15.28:0/1239493889 >> 172.21.15.28:0/1239493889 conn(0x7f873c0780e0 msgr2=0x7f873c0ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:14.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.444+0000 7f8743740700 1 -- 172.21.15.28:0/1239493889 shutdown_connections 2024-04-25T01:32:14.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:14.444+0000 7f8743740700 1 -- 172.21.15.28:0/1239493889 wait complete. 2024-04-25T01:32:14.426 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T01:32:14.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:14 smithi130 ceph-mon[32702]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:32:14.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:14 smithi032 ceph-mon[32461]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:32:15.250 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-04-25T01:32:14.354892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660139,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":892,"num_read_kb":574,"num_write":424,"num_write_kb":3081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1180804,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":350,"ondisk_log_size":350,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380324,"kb_used_data":14480,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124561500,"statfs":{"total":1151940427776,"available":1151550976000,"internally_reserved":0,"allocated":14827520,"data_stored":8063519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":70,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003427"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.870244+0000","last_change":"2024-04-25T01:30:34.358339+0000","last_active":"2024-04-25T01:31:21.870244+0000","last_peered":"2024-04-25T01:31:21.870244+0000","last_clean":"2024-04-25T01:31:21.870244+0000","last_became_active":"2024-04-25T01:30:34.357884+0000","last_became_peered":"2024-04-25T01:30:34.357884+0000","last_unstale":"2024-04-25T01:31:21.870244+0000","last_undegraded":"2024-04-25T01:31:21.870244+0000","last_fullsized":"2024-04-25T01:31:21.870244+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:56.409080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869592+0000","last_change":"2024-04-25T01:30:34.359933+0000","last_active":"2024-04-25T01:31:21.869592+0000","last_peered":"2024-04-25T01:31:21.869592+0000","last_clean":"2024-04-25T01:31:21.869592+0000","last_became_active":"2024-04-25T01:30:34.359576+0000","last_became_peered":"2024-04-25T01:30:34.359576+0000","last_unstale":"2024-04-25T01:31:21.869592+0000","last_undegraded":"2024-04-25T01:31:21.869592+0000","last_fullsized":"2024-04-25T01:31:21.869592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:48:47.764121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869363+0000","last_change":"2024-04-25T01:30:34.357772+0000","last_active":"2024-04-25T01:31:21.869363+0000","last_peered":"2024-04-25T01:31:21.869363+0000","last_clean":"2024-04-25T01:31:21.869363+0000","last_became_active":"2024-04-25T01:30:34.357434+0000","last_became_peered":"2024-04-25T01:30:34.357434+0000","last_unstale":"2024-04-25T01:31:21.869363+0000","last_undegraded":"2024-04-25T01:31:21.869363+0000","last_fullsized":"2024-04-25T01:31:21.869363+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:19.926180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867798+0000","last_change":"2024-04-25T01:30:34.351250+0000","last_active":"2024-04-25T01:31:21.867798+0000","last_peered":"2024-04-25T01:31:21.867798+0000","last_clean":"2024-04-25T01:31:21.867798+0000","last_became_active":"2024-04-25T01:30:34.349976+0000","last_became_peered":"2024-04-25T01:30:34.349976+0000","last_unstale":"2024-04-25T01:31:21.867798+0000","last_undegraded":"2024-04-25T01:31:21.867798+0000","last_fullsized":"2024-04-25T01:31:21.867798+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:19:51.605307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.874477+0000","last_change":"2024-04-25T01:30:34.367882+0000","last_active":"2024-04-25T01:31:21.874477+0000","last_peered":"2024-04-25T01:31:21.874477+0000","last_clean":"2024-04-25T01:31:21.874477+0000","last_became_active":"2024-04-25T01:30:34.366797+0000","last_became_peered":"2024-04-25T01:30:34.366797+0000","last_unstale":"2024-04-25T01:31:21.874477+0000","last_undegraded":"2024-04-25T01:31:21.874477+0000","last_fullsized":"2024-04-25T01:31:21.874477+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:35:23.439061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872254+0000","last_change":"2024-04-25T01:30:34.352506+0000","last_active":"2024-04-25T01:31:21.872254+0000","last_peered":"2024-04-25T01:31:21.872254+0000","last_clean":"2024-04-25T01:31:21.872254+0000","last_became_active":"2024-04-25T01:30:34.352100+0000","last_became_peered":"2024-04-25T01:30:34.352100+0000","last_unstale":"2024-04-25T01:31:21.872254+0000","last_undegraded":"2024-04-25T01:31:21.872254+0000","last_fullsized":"2024-04-25T01:31:21.872254+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:38.113959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.048442+0000","last_change":"2024-04-25T01:30:34.355352+0000","last_active":"2024-04-25T01:31:22.048442+0000","last_peered":"2024-04-25T01:31:22.048442+0000","last_clean":"2024-04-25T01:31:22.048442+0000","last_became_active":"2024-04-25T01:30:34.355008+0000","last_became_peered":"2024-04-25T01:30:34.355008+0000","last_unstale":"2024-04-25T01:31:22.048442+0000","last_undegraded":"2024-04-25T01:31:22.048442+0000","last_fullsized":"2024-04-25T01:31:22.048442+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:23.582742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872403+0000","last_change":"2024-04-25T01:30:34.360644+0000","last_active":"2024-04-25T01:31:21.872403+0000","last_peered":"2024-04-25T01:31:21.872403+0000","last_clean":"2024-04-25T01:31:21.872403+0000","last_became_active":"2024-04-25T01:30:34.357841+0000","last_became_peered":"2024-04-25T01:30:34.357841+0000","last_unstale":"2024-04-25T01:31:21.872403+0000","last_undegraded":"2024-04-25T01:31:21.872403+0000","last_fullsized":"2024-04-25T01:31:21.872403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:26:21.606219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049329+0000","last_change":"2024-04-25T01:30:34.358417+0000","last_active":"2024-04-25T01:31:22.049329+0000","last_peered":"2024-04-25T01:31:22.049329+0000","last_clean":"2024-04-25T01:31:22.049329+0000","last_became_active":"2024-04-25T01:30:34.358039+0000","last_became_peered":"2024-04-25T01:30:34.358039+0000","last_unstale":"2024-04-25T01:31:22.049329+0000","last_undegraded":"2024-04-25T01:31:22.049329+0000","last_fullsized":"2024-04-25T01:31:22.049329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:42.600693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872775+0000","last_change":"2024-04-25T01:30:34.357803+0000","last_active":"2024-04-25T01:31:21.872775+0000","last_peered":"2024-04-25T01:31:21.872775+0000","last_clean":"2024-04-25T01:31:21.872775+0000","last_became_active":"2024-04-25T01:30:34.357391+0000","last_became_peered":"2024-04-25T01:30:34.357391+0000","last_unstale":"2024-04-25T01:31:21.872775+0000","last_undegraded":"2024-04-25T01:31:21.872775+0000","last_fullsized":"2024-04-25T01:31:21.872775+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:57.857303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049654+0000","last_change":"2024-04-25T01:30:34.363340+0000","last_active":"2024-04-25T01:31:22.049654+0000","last_peered":"2024-04-25T01:31:22.049654+0000","last_clean":"2024-04-25T01:31:22.049654+0000","last_became_active":"2024-04-25T01:30:34.362980+0000","last_became_peered":"2024-04-25T01:30:34.362980+0000","last_unstale":"2024-04-25T01:31:22.049654+0000","last_undegraded":"2024-04-25T01:31:22.049654+0000","last_fullsized":"2024-04-25T01:31:22.049654+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:46.291459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867712+0000","last_change":"2024-04-25T01:30:34.358898+0000","last_active":"2024-04-25T01:31:21.867712+0000","last_peered":"2024-04-25T01:31:21.867712+0000","last_clean":"2024-04-25T01:31:21.867712+0000","last_became_active":"2024-04-25T01:30:34.358495+0000","last_became_peered":"2024-04-25T01:30:34.358495+0000","last_unstale":"2024-04-25T01:31:21.867712+0000","last_undegraded":"2024-04-25T01:31:21.867712+0000","last_fullsized":"2024-04-25T01:31:21.867712+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:34:21.860010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866836+0000","last_change":"2024-04-25T01:30:34.363708+0000","last_active":"2024-04-25T01:31:21.866836+0000","last_peered":"2024-04-25T01:31:21.866836+0000","last_clean":"2024-04-25T01:31:21.866836+0000","last_became_active":"2024-04-25T01:30:34.363207+0000","last_became_peered":"2024-04-25T01:30:34.363207+0000","last_unstale":"2024-04-25T01:31:21.866836+0000","last_undegraded":"2024-04-25T01:31:21.866836+0000","last_fullsized":"2024-04-25T01:31:21.866836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:53.442987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869362+0000","last_change":"2024-04-25T01:30:34.371031+0000","last_active":"2024-04-25T01:31:21.869362+0000","last_peered":"2024-04-25T01:31:21.869362+0000","last_clean":"2024-04-25T01:31:21.869362+0000","last_became_active":"2024-04-25T01:30:34.370687+0000","last_became_peered":"2024-04-25T01:30:34.370687+0000","last_unstale":"2024-04-25T01:31:21.869362+0000","last_undegraded":"2024-04-25T01:31:21.869362+0000","last_fullsized":"2024-04-25T01:31:21.869362+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:03:20.700960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869153+0000","last_change":"2024-04-25T01:30:34.366212+0000","last_active":"2024-04-25T01:31:21.869153+0000","last_peered":"2024-04-25T01:31:21.869153+0000","last_clean":"2024-04-25T01:31:21.869153+0000","last_became_active":"2024-04-25T01:30:34.365849+0000","last_became_peered":"2024-04-25T01:30:34.365849+0000","last_unstale":"2024-04-25T01:31:21.869153+0000","last_undegraded":"2024-04-25T01:31:21.869153+0000","last_fullsized":"2024-04-25T01:31:21.869153+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:24:02.397615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049096+0000","last_change":"2024-04-25T01:30:34.367380+0000","last_active":"2024-04-25T01:31:22.049096+0000","last_peered":"2024-04-25T01:31:22.049096+0000","last_clean":"2024-04-25T01:31:22.049096+0000","last_became_active":"2024-04-25T01:30:34.366872+0000","last_became_peered":"2024-04-25T01:30:34.366872+0000","last_unstale":"2024-04-25T01:31:22.049096+0000","last_undegraded":"2024-04-25T01:31:22.049096+0000","last_fullsized":"2024-04-25T01:31:22.049096+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:18.767108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.870145+0000","last_change":"2024-04-25T01:30:34.359752+0000","last_active":"2024-04-25T01:31:21.870145+0000","last_peered":"2024-04-25T01:31:21.870145+0000","last_clean":"2024-04-25T01:31:21.870145+0000","last_became_active":"2024-04-25T01:30:34.358403+0000","last_became_peered":"2024-04-25T01:30:34.358403+0000","last_unstale":"2024-04-25T01:31:21.870145+0000","last_undegraded":"2024-04-25T01:31:21.870145+0000","last_fullsized":"2024-04-25T01:31:21.870145+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:14.833126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.871052+0000","last_change":"2024-04-25T01:30:34.361373+0000","last_active":"2024-04-25T01:31:21.871052+0000","last_peered":"2024-04-25T01:31:21.871052+0000","last_clean":"2024-04-25T01:31:21.871052+0000","last_became_active":"2024-04-25T01:30:34.361046+0000","last_became_peered":"2024-04-25T01:30:34.361046+0000","last_unstale":"2024-04-25T01:31:21.871052+0000","last_undegraded":"2024-04-25T01:31:21.871052+0000","last_fullsized":"2024-04-25T01:31:21.871052+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:21.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873421+0000","last_change":"2024-04-25T01:30:34.367975+0000","last_active":"2024-04-25T01:31:21.873421+0000","last_peered":"2024-04-25T01:31:21.873421+0000","last_clean":"2024-04-25T01:31:21.873421+0000","last_became_active":"2024-04-25T01:30:34.366383+0000","last_became_peered":"2024-04-25T01:30:34.366383+0000","last_unstale":"2024-04-25T01:31:21.873421+0000","last_undegraded":"2024-04-25T01:31:21.873421+0000","last_fullsized":"2024-04-25T01:31:21.873421+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:53:40.279067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873118+0000","last_change":"2024-04-25T01:30:38.382168+0000","last_active":"2024-04-25T01:31:21.873118+0000","last_peered":"2024-04-25T01:31:21.873118+0000","last_clean":"2024-04-25T01:31:21.873118+0000","last_became_active":"2024-04-25T01:30:38.381740+0000","last_became_peered":"2024-04-25T01:30:38.381740+0000","last_unstale":"2024-04-25T01:31:21.873118+0000","last_undegraded":"2024-04-25T01:31:21.873118+0000","last_fullsized":"2024-04-25T01:31:21.873118+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:27.425991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'13","reported_seq":113,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:32:11.905400+0000","last_change":"2024-04-25T01:30:33.345547+0000","last_active":"2024-04-25T01:32:11.905400+0000","last_peered":"2024-04-25T01:32:11.905400+0000","last_clean":"2024-04-25T01:32:11.905400+0000","last_became_active":"2024-04-25T01:30:33.345229+0000","last_became_peered":"2024-04-25T01:30:33.345229+0000","last_unstale":"2024-04-25T01:32:11.905400+0000","last_undegraded":"2024-04-25T01:32:11.905400+0000","last_fullsized":"2024-04-25T01:32:11.905400+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:32:46.637720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873194+0000","last_change":"2024-04-25T01:30:34.354703+0000","last_active":"2024-04-25T01:31:21.873194+0000","last_peered":"2024-04-25T01:31:21.873194+0000","last_clean":"2024-04-25T01:31:21.873194+0000","last_became_active":"2024-04-25T01:30:34.354362+0000","last_became_peered":"2024-04-25T01:30:34.354362+0000","last_unstale":"2024-04-25T01:31:21.873194+0000","last_undegraded":"2024-04-25T01:31:21.873194+0000","last_fullsized":"2024-04-25T01:31:21.873194+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:51.358599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867199+0000","last_change":"2024-04-25T01:30:38.380912+0000","last_active":"2024-04-25T01:31:21.867199+0000","last_peered":"2024-04-25T01:31:21.867199+0000","last_clean":"2024-04-25T01:31:21.867199+0000","last_became_active":"2024-04-25T01:30:38.380439+0000","last_became_peered":"2024-04-25T01:30:38.380439+0000","last_unstale":"2024-04-25T01:31:21.867199+0000","last_undegraded":"2024-04-25T01:31:21.867199+0000","last_fullsized":"2024-04-25T01:31:21.867199+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:22:00.527268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873300+0000","last_change":"2024-04-25T01:30:33.345620+0000","last_active":"2024-04-25T01:31:21.873300+0000","last_peered":"2024-04-25T01:31:21.873300+0000","last_clean":"2024-04-25T01:31:21.873300+0000","last_became_active":"2024-04-25T01:30:33.345315+0000","last_became_peered":"2024-04-25T01:30:33.345315+0000","last_unstale":"2024-04-25T01:31:21.873300+0000","last_undegraded":"2024-04-25T01:31:21.873300+0000","last_fullsized":"2024-04-25T01:31:21.873300+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:48:00.083728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869679+0000","last_change":"2024-04-25T01:30:34.356589+0000","last_active":"2024-04-25T01:31:21.869679+0000","last_peered":"2024-04-25T01:31:21.869679+0000","last_clean":"2024-04-25T01:31:21.869679+0000","last_became_active":"2024-04-25T01:30:34.356114+0000","last_became_peered":"2024-04-25T01:30:34.356114+0000","last_unstale":"2024-04-25T01:31:21.869679+0000","last_undegraded":"2024-04-25T01:31:21.869679+0000","last_fullsized":"2024-04-25T01:31:21.869679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:39.039517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.876578+0000","last_change":"2024-04-25T01:31:21.876578+0000","last_active":"2024-04-25T01:31:21.876578+0000","last_peered":"2024-04-25T01:31:21.876578+0000","last_clean":"2024-04-25T01:31:21.876578+0000","last_became_active":"2024-04-25T01:31:21.876126+0000","last_became_peered":"2024-04-25T01:31:21.876126+0000","last_unstale":"2024-04-25T01:31:21.876578+0000","last_undegraded":"2024-04-25T01:31:21.876578+0000","last_fullsized":"2024-04-25T01:31:21.876578+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:55.909353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'12","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.874895+0000","last_change":"2024-04-25T01:30:33.345575+0000","last_active":"2024-04-25T01:31:21.874895+0000","last_peered":"2024-04-25T01:31:21.874895+0000","last_clean":"2024-04-25T01:31:21.874895+0000","last_became_active":"2024-04-25T01:30:33.345231+0000","last_became_peered":"2024-04-25T01:30:33.345231+0000","last_unstale":"2024-04-25T01:31:21.874895+0000","last_undegraded":"2024-04-25T01:31:21.874895+0000","last_fullsized":"2024-04-25T01:31:21.874895+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:04:21.090788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049253+0000","last_change":"2024-04-25T01:30:34.371317+0000","last_active":"2024-04-25T01:31:22.049253+0000","last_peered":"2024-04-25T01:31:22.049253+0000","last_clean":"2024-04-25T01:31:22.049253+0000","last_became_active":"2024-04-25T01:30:34.370926+0000","last_became_peered":"2024-04-25T01:30:34.370926+0000","last_unstale":"2024-04-25T01:31:22.049253+0000","last_undegraded":"2024-04-25T01:31:22.049253+0000","last_fullsized":"2024-04-25T01:31:22.049253+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:58:39.898471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867100+0000","last_change":"2024-04-25T01:30:38.380982+0000","last_active":"2024-04-25T01:31:21.867100+0000","last_peered":"2024-04-25T01:31:21.867100+0000","last_clean":"2024-04-25T01:31:21.867100+0000","last_became_active":"2024-04-25T01:30:38.380510+0000","last_became_peered":"2024-04-25T01:30:38.380510+0000","last_unstale":"2024-04-25T01:31:21.867100+0000","last_undegraded":"2024-04-25T01:31:21.867100+0000","last_fullsized":"2024-04-25T01:31:21.867100+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:00.510697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.874833+0000","last_change":"2024-04-25T01:30:33.351561+0000","last_active":"2024-04-25T01:31:21.874833+0000","last_peered":"2024-04-25T01:31:21.874833+0000","last_clean":"2024-04-25T01:31:21.874833+0000","last_became_active":"2024-04-25T01:30:33.351230+0000","last_became_peered":"2024-04-25T01:30:33.351230+0000","last_unstale":"2024-04-25T01:31:21.874833+0000","last_undegraded":"2024-04-25T01:31:21.874833+0000","last_fullsized":"2024-04-25T01:31:21.874833+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:56.594457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867798+0000","last_change":"2024-04-25T01:30:34.358821+0000","last_active":"2024-04-25T01:31:21.867798+0000","last_peered":"2024-04-25T01:31:21.867798+0000","last_clean":"2024-04-25T01:31:21.867798+0000","last_became_active":"2024-04-25T01:30:34.358303+0000","last_became_peered":"2024-04-25T01:30:34.358303+0000","last_unstale":"2024-04-25T01:31:21.867798+0000","last_undegraded":"2024-04-25T01:31:21.867798+0000","last_fullsized":"2024-04-25T01:31:21.867798+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:26:23.352101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873266+0000","last_change":"2024-04-25T01:30:38.372281+0000","last_active":"2024-04-25T01:31:21.873266+0000","last_peered":"2024-04-25T01:31:21.873266+0000","last_clean":"2024-04-25T01:31:21.873266+0000","last_became_active":"2024-04-25T01:30:38.371876+0000","last_became_peered":"2024-04-25T01:30:38.371876+0000","last_unstale":"2024-04-25T01:31:21.873266+0000","last_undegraded":"2024-04-25T01:31:21.873266+0000","last_fullsized":"2024-04-25T01:31:21.873266+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:54.254011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'6","reported_seq":107,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:32:11.913467+0000","last_change":"2024-04-25T01:30:33.347262+0000","last_active":"2024-04-25T01:32:11.913467+0000","last_peered":"2024-04-25T01:32:11.913467+0000","last_clean":"2024-04-25T01:32:11.913467+0000","last_became_active":"2024-04-25T01:30:33.346900+0000","last_became_peered":"2024-04-25T01:30:33.346900+0000","last_unstale":"2024-04-25T01:32:11.913467+0000","last_undegraded":"2024-04-25T01:32:11.913467+0000","last_fullsized":"2024-04-25T01:32:11.913467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:29:30.960456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.874102+0000","last_change":"2024-04-25T01:30:34.364870+0000","last_active":"2024-04-25T01:31:21.874102+0000","last_peered":"2024-04-25T01:31:21.874102+0000","last_clean":"2024-04-25T01:31:21.874102+0000","last_became_active":"2024-04-25T01:30:34.364459+0000","last_became_peered":"2024-04-25T01:30:34.364459+0000","last_unstale":"2024-04-25T01:31:21.874102+0000","last_undegraded":"2024-04-25T01:31:21.874102+0000","last_fullsized":"2024-04-25T01:31:21.874102+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:42:45.311668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872894+0000","last_change":"2024-04-25T01:30:38.382175+0000","last_active":"2024-04-25T01:31:21.872894+0000","last_peered":"2024-04-25T01:31:21.872894+0000","last_clean":"2024-04-25T01:31:21.872894+0000","last_became_active":"2024-04-25T01:30:38.381771+0000","last_became_peered":"2024-04-25T01:30:38.381771+0000","last_unstale":"2024-04-25T01:31:21.872894+0000","last_undegraded":"2024-04-25T01:31:21.872894+0000","last_fullsized":"2024-04-25T01:31:21.872894+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:25:31.041566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872207+0000","last_change":"2024-04-25T01:30:33.343605+0000","last_active":"2024-04-25T01:31:21.872207+0000","last_peered":"2024-04-25T01:31:21.872207+0000","last_clean":"2024-04-25T01:31:21.872207+0000","last_became_active":"2024-04-25T01:30:33.343254+0000","last_became_peered":"2024-04-25T01:30:33.343254+0000","last_unstale":"2024-04-25T01:31:21.872207+0000","last_undegraded":"2024-04-25T01:31:21.872207+0000","last_fullsized":"2024-04-25T01:31:21.872207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:18:16.156285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.874640+0000","last_change":"2024-04-25T01:31:21.874202+0000","last_active":"2024-04-25T01:31:21.874640+0000","last_peered":"2024-04-25T01:31:21.874640+0000","last_clean":"2024-04-25T01:31:21.874640+0000","last_became_active":"2024-04-25T01:31:21.873839+0000","last_became_peered":"2024-04-25T01:31:21.873839+0000","last_unstale":"2024-04-25T01:31:21.874640+0000","last_undegraded":"2024-04-25T01:31:21.874640+0000","last_fullsized":"2024-04-25T01:31:21.874640+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:39.774281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.875259+0000","last_change":"2024-04-25T01:31:21.875259+0000","last_active":"2024-04-25T01:31:21.875259+0000","last_peered":"2024-04-25T01:31:21.875259+0000","last_clean":"2024-04-25T01:31:21.875259+0000","last_became_active":"2024-04-25T01:31:21.874800+0000","last_became_peered":"2024-04-25T01:31:21.874800+0000","last_unstale":"2024-04-25T01:31:21.875259+0000","last_undegraded":"2024-04-25T01:31:21.875259+0000","last_fullsized":"2024-04-25T01:31:21.875259+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:44.828620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,10,0],"acting":[11,4,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866568+0000","last_change":"2024-04-25T01:30:33.351621+0000","last_active":"2024-04-25T01:31:21.866568+0000","last_peered":"2024-04-25T01:31:21.866568+0000","last_clean":"2024-04-25T01:31:21.866568+0000","last_became_active":"2024-04-25T01:30:33.351289+0000","last_became_peered":"2024-04-25T01:30:33.351289+0000","last_unstale":"2024-04-25T01:31:21.866568+0000","last_undegraded":"2024-04-25T01:31:21.866568+0000","last_fullsized":"2024-04-25T01:31:21.866568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:46.044826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872529+0000","last_change":"2024-04-25T01:30:34.350685+0000","last_active":"2024-04-25T01:31:21.872529+0000","last_peered":"2024-04-25T01:31:21.872529+0000","last_clean":"2024-04-25T01:31:21.872529+0000","last_became_active":"2024-04-25T01:30:34.350308+0000","last_became_peered":"2024-04-25T01:30:34.350308+0000","last_unstale":"2024-04-25T01:31:21.872529+0000","last_undegraded":"2024-04-25T01:31:21.872529+0000","last_fullsized":"2024-04-25T01:31:21.872529+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:11.690381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869484+0000","last_change":"2024-04-25T01:30:38.377403+0000","last_active":"2024-04-25T01:31:21.869484+0000","last_peered":"2024-04-25T01:31:21.869484+0000","last_clean":"2024-04-25T01:31:21.869484+0000","last_became_active":"2024-04-25T01:30:38.376966+0000","last_became_peered":"2024-04-25T01:30:38.376966+0000","last_unstale":"2024-04-25T01:31:21.869484+0000","last_undegraded":"2024-04-25T01:31:21.869484+0000","last_fullsized":"2024-04-25T01:31:21.869484+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:31:10.229443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867282+0000","last_change":"2024-04-25T01:30:33.348455+0000","last_active":"2024-04-25T01:31:21.867282+0000","last_peered":"2024-04-25T01:31:21.867282+0000","last_clean":"2024-04-25T01:31:21.867282+0000","last_became_active":"2024-04-25T01:30:33.348142+0000","last_became_peered":"2024-04-25T01:30:33.348142+0000","last_unstale":"2024-04-25T01:31:21.867282+0000","last_undegraded":"2024-04-25T01:31:21.867282+0000","last_fullsized":"2024-04-25T01:31:21.867282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:43:07.695580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.870724+0000","last_change":"2024-04-25T01:30:34.359949+0000","last_active":"2024-04-25T01:31:21.870724+0000","last_peered":"2024-04-25T01:31:21.870724+0000","last_clean":"2024-04-25T01:31:21.870724+0000","last_became_active":"2024-04-25T01:30:34.359609+0000","last_became_peered":"2024-04-25T01:30:34.359609+0000","last_unstale":"2024-04-25T01:31:21.870724+0000","last_undegraded":"2024-04-25T01:31:21.870724+0000","last_fullsized":"2024-04-25T01:31:21.870724+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:49.668800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872956+0000","last_change":"2024-04-25T01:30:38.380260+0000","last_active":"2024-04-25T01:31:21.872956+0000","last_peered":"2024-04-25T01:31:21.872956+0000","last_clean":"2024-04-25T01:31:21.872956+0000","last_became_active":"2024-04-25T01:30:38.379624+0000","last_became_peered":"2024-04-25T01:30:38.379624+0000","last_unstale":"2024-04-25T01:31:21.872956+0000","last_undegraded":"2024-04-25T01:31:21.872956+0000","last_fullsized":"2024-04-25T01:31:21.872956+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:26:35.235335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":177,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:32:13.915484+0000","last_change":"2024-04-25T01:30:33.349490+0000","last_active":"2024-04-25T01:32:13.915484+0000","last_peered":"2024-04-25T01:32:13.915484+0000","last_clean":"2024-04-25T01:32:13.915484+0000","last_became_active":"2024-04-25T01:30:33.349136+0000","last_became_peered":"2024-04-25T01:30:33.349136+0000","last_unstale":"2024-04-25T01:32:13.915484+0000","last_undegraded":"2024-04-25T01:32:13.915484+0000","last_fullsized":"2024-04-25T01:32:13.915484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:16:23.552073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":146,"num_read_kb":146,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049084+0000","last_change":"2024-04-25T01:30:34.372701+0000","last_active":"2024-04-25T01:31:22.049084+0000","last_peered":"2024-04-25T01:31:22.049084+0000","last_clean":"2024-04-25T01:31:22.049084+0000","last_became_active":"2024-04-25T01:30:34.372321+0000","last_became_peered":"2024-04-25T01:30:34.372321+0000","last_unstale":"2024-04-25T01:31:22.049084+0000","last_undegraded":"2024-04-25T01:31:22.049084+0000","last_fullsized":"2024-04-25T01:31:22.049084+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:02.025015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869936+0000","last_change":"2024-04-25T01:30:38.378036+0000","last_active":"2024-04-25T01:31:21.869936+0000","last_peered":"2024-04-25T01:31:21.869936+0000","last_clean":"2024-04-25T01:31:21.869936+0000","last_became_active":"2024-04-25T01:30:38.377602+0000","last_became_peered":"2024-04-25T01:30:38.377602+0000","last_unstale":"2024-04-25T01:31:21.869936+0000","last_undegraded":"2024-04-25T01:31:21.869936+0000","last_fullsized":"2024-04-25T01:31:21.869936+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:45.210518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.048871+0000","last_change":"2024-04-25T01:30:33.352928+0000","last_active":"2024-04-25T01:31:22.048871+0000","last_peered":"2024-04-25T01:31:22.048871+0000","last_clean":"2024-04-25T01:31:22.048871+0000","last_became_active":"2024-04-25T01:30:33.352567+0000","last_became_peered":"2024-04-25T01:30:33.352567+0000","last_unstale":"2024-04-25T01:31:22.048871+0000","last_undegraded":"2024-04-25T01:31:22.048871+0000","last_fullsized":"2024-04-25T01:31:22.048871+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:47:21.941179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869787+0000","last_change":"2024-04-25T01:30:34.366947+0000","last_active":"2024-04-25T01:31:21.869787+0000","last_peered":"2024-04-25T01:31:21.869787+0000","last_clean":"2024-04-25T01:31:21.869787+0000","last_became_active":"2024-04-25T01:30:34.366337+0000","last_became_peered":"2024-04-25T01:30:34.366337+0000","last_unstale":"2024-04-25T01:31:21.869787+0000","last_undegraded":"2024-04-25T01:31:21.869787+0000","last_fullsized":"2024-04-25T01:31:21.869787+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:48:43.528380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.050799+0000","last_change":"2024-04-25T01:31:22.050799+0000","last_active":"2024-04-25T01:31:22.050799+0000","last_peered":"2024-04-25T01:31:22.050799+0000","last_clean":"2024-04-25T01:31:22.050799+0000","last_became_active":"2024-04-25T01:31:22.050130+0000","last_became_peered":"2024-04-25T01:31:22.050130+0000","last_unstale":"2024-04-25T01:31:22.050799+0000","last_undegraded":"2024-04-25T01:31:22.050799+0000","last_fullsized":"2024-04-25T01:31:22.050799+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:40:43.549713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867754+0000","last_change":"2024-04-25T01:30:33.346003+0000","last_active":"2024-04-25T01:31:21.867754+0000","last_peered":"2024-04-25T01:31:21.867754+0000","last_clean":"2024-04-25T01:31:21.867754+0000","last_became_active":"2024-04-25T01:30:33.345254+0000","last_became_peered":"2024-04-25T01:30:33.345254+0000","last_unstale":"2024-04-25T01:31:21.867754+0000","last_undegraded":"2024-04-25T01:31:21.867754+0000","last_fullsized":"2024-04-25T01:31:21.867754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:18:37.957047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049419+0000","last_change":"2024-04-25T01:30:34.366108+0000","last_active":"2024-04-25T01:31:22.049419+0000","last_peered":"2024-04-25T01:31:22.049419+0000","last_clean":"2024-04-25T01:31:22.049419+0000","last_became_active":"2024-04-25T01:30:34.365702+0000","last_became_peered":"2024-04-25T01:30:34.365702+0000","last_unstale":"2024-04-25T01:31:22.049419+0000","last_undegraded":"2024-04-25T01:31:22.049419+0000","last_fullsized":"2024-04-25T01:31:22.049419+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:24:06.802424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.871178+0000","last_change":"2024-04-25T01:30:38.376342+0000","last_active":"2024-04-25T01:31:21.871178+0000","last_peered":"2024-04-25T01:31:21.871178+0000","last_clean":"2024-04-25T01:31:21.871178+0000","last_became_active":"2024-04-25T01:30:38.375905+0000","last_became_peered":"2024-04-25T01:30:38.375905+0000","last_unstale":"2024-04-25T01:31:21.871178+0000","last_undegraded":"2024-04-25T01:31:21.871178+0000","last_fullsized":"2024-04-25T01:31:21.871178+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:31.615356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'36","reported_seq":217,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:32:13.915521+0000","last_change":"2024-04-25T01:30:33.350770+0000","last_active":"2024-04-25T01:32:13.915521+0000","last_peered":"2024-04-25T01:32:13.915521+0000","last_clean":"2024-04-25T01:32:13.915521+0000","last_became_active":"2024-04-25T01:30:33.350414+0000","last_became_peered":"2024-04-25T01:30:33.350414+0000","last_unstale":"2024-04-25T01:32:13.915521+0000","last_undegraded":"2024-04-25T01:32:13.915521+0000","last_fullsized":"2024-04-25T01:32:13.915521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:01.771182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62333,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":154,"num_read_kb":139,"num_write":38,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.874658+0000","last_change":"2024-04-25T01:30:34.362532+0000","last_active":"2024-04-25T01:31:21.874658+0000","last_peered":"2024-04-25T01:31:21.874658+0000","last_clean":"2024-04-25T01:31:21.874658+0000","last_became_active":"2024-04-25T01:30:34.362137+0000","last_became_peered":"2024-04-25T01:30:34.362137+0000","last_unstale":"2024-04-25T01:31:21.874658+0000","last_undegraded":"2024-04-25T01:31:21.874658+0000","last_fullsized":"2024-04-25T01:31:21.874658+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:28.105250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.868022+0000","last_change":"2024-04-25T01:30:38.374591+0000","last_active":"2024-04-25T01:31:21.868022+0000","last_peered":"2024-04-25T01:31:21.868022+0000","last_clean":"2024-04-25T01:31:21.868022+0000","last_became_active":"2024-04-25T01:30:38.373237+0000","last_became_peered":"2024-04-25T01:30:38.373237+0000","last_unstale":"2024-04-25T01:31:21.868022+0000","last_undegraded":"2024-04-25T01:31:21.868022+0000","last_fullsized":"2024-04-25T01:31:21.868022+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:21:25.937405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'9","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.885341+0000","last_change":"2024-04-25T01:31:21.885341+0000","last_active":"2024-04-25T01:31:21.885341+0000","last_peered":"2024-04-25T01:31:21.885341+0000","last_clean":"2024-04-25T01:31:21.885341+0000","last_became_active":"2024-04-25T01:31:21.874312+0000","last_became_peered":"2024-04-25T01:31:21.874312+0000","last_unstale":"2024-04-25T01:31:21.885341+0000","last_undegraded":"2024-04-25T01:31:21.885341+0000","last_fullsized":"2024-04-25T01:31:21.885341+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:23:55.388089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866527+0000","last_change":"2024-04-25T01:30:34.354444+0000","last_active":"2024-04-25T01:31:21.866527+0000","last_peered":"2024-04-25T01:31:21.866527+0000","last_clean":"2024-04-25T01:31:21.866527+0000","last_became_active":"2024-04-25T01:30:34.354070+0000","last_became_peered":"2024-04-25T01:30:34.354070+0000","last_unstale":"2024-04-25T01:31:21.866527+0000","last_undegraded":"2024-04-25T01:31:21.866527+0000","last_fullsized":"2024-04-25T01:31:21.866527+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:39.893995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869979+0000","last_change":"2024-04-25T01:30:38.373570+0000","last_active":"2024-04-25T01:31:21.869979+0000","last_peered":"2024-04-25T01:31:21.869979+0000","last_clean":"2024-04-25T01:31:21.869979+0000","last_became_active":"2024-04-25T01:30:38.373144+0000","last_became_peered":"2024-04-25T01:30:38.373144+0000","last_unstale":"2024-04-25T01:31:21.869979+0000","last_undegraded":"2024-04-25T01:31:21.869979+0000","last_fullsized":"2024-04-25T01:31:21.869979+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:53:32.933998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867431+0000","last_change":"2024-04-25T01:30:33.346145+0000","last_active":"2024-04-25T01:31:21.867431+0000","last_peered":"2024-04-25T01:31:21.867431+0000","last_clean":"2024-04-25T01:31:21.867431+0000","last_became_active":"2024-04-25T01:30:33.344294+0000","last_became_peered":"2024-04-25T01:30:33.344294+0000","last_unstale":"2024-04-25T01:31:21.867431+0000","last_undegraded":"2024-04-25T01:31:21.867431+0000","last_fullsized":"2024-04-25T01:31:21.867431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:37:17.147148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.048716+0000","last_change":"2024-04-25T01:30:34.360769+0000","last_active":"2024-04-25T01:31:22.048716+0000","last_peered":"2024-04-25T01:31:22.048716+0000","last_clean":"2024-04-25T01:31:22.048716+0000","last_became_active":"2024-04-25T01:30:34.360076+0000","last_became_peered":"2024-04-25T01:30:34.360076+0000","last_unstale":"2024-04-25T01:31:22.048716+0000","last_undegraded":"2024-04-25T01:31:22.048716+0000","last_fullsized":"2024-04-25T01:31:22.048716+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:56:21.515511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.870901+0000","last_change":"2024-04-25T01:30:38.379198+0000","last_active":"2024-04-25T01:31:21.870901+0000","last_peered":"2024-04-25T01:31:21.870901+0000","last_clean":"2024-04-25T01:31:21.870901+0000","last_became_active":"2024-04-25T01:30:38.378798+0000","last_became_peered":"2024-04-25T01:30:38.378798+0000","last_unstale":"2024-04-25T01:31:21.870901+0000","last_undegraded":"2024-04-25T01:31:21.870901+0000","last_fullsized":"2024-04-25T01:31:21.870901+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:40.192239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":107,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:32:12.907572+0000","last_change":"2024-04-25T01:30:33.343232+0000","last_active":"2024-04-25T01:32:12.907572+0000","last_peered":"2024-04-25T01:32:12.907572+0000","last_clean":"2024-04-25T01:32:12.907572+0000","last_became_active":"2024-04-25T01:30:33.342876+0000","last_became_peered":"2024-04-25T01:30:33.342876+0000","last_unstale":"2024-04-25T01:32:12.907572+0000","last_undegraded":"2024-04-25T01:32:12.907572+0000","last_fullsized":"2024-04-25T01:32:12.907572+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:45.032564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.048465+0000","last_change":"2024-04-25T01:30:34.358210+0000","last_active":"2024-04-25T01:31:22.048465+0000","last_peered":"2024-04-25T01:31:22.048465+0000","last_clean":"2024-04-25T01:31:22.048465+0000","last_became_active":"2024-04-25T01:30:34.357862+0000","last_became_peered":"2024-04-25T01:30:34.357862+0000","last_unstale":"2024-04-25T01:31:22.048465+0000","last_undegraded":"2024-04-25T01:31:22.048465+0000","last_fullsized":"2024-04-25T01:31:22.048465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:43.751361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866946+0000","last_change":"2024-04-25T01:30:38.380413+0000","last_active":"2024-04-25T01:31:21.866946+0000","last_peered":"2024-04-25T01:31:21.866946+0000","last_clean":"2024-04-25T01:31:21.866946+0000","last_became_active":"2024-04-25T01:30:38.379612+0000","last_became_peered":"2024-04-25T01:30:38.379612+0000","last_unstale":"2024-04-25T01:31:21.866946+0000","last_undegraded":"2024-04-25T01:31:21.866946+0000","last_fullsized":"2024-04-25T01:31:21.866946+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:45.328642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'17","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.874013+0000","last_change":"2024-04-25T01:30:33.344938+0000","last_active":"2024-04-25T01:31:21.874013+0000","last_peered":"2024-04-25T01:31:21.874013+0000","last_clean":"2024-04-25T01:31:21.874013+0000","last_became_active":"2024-04-25T01:30:33.344582+0000","last_became_peered":"2024-04-25T01:30:33.344582+0000","last_unstale":"2024-04-25T01:31:21.874013+0000","last_undegraded":"2024-04-25T01:31:21.874013+0000","last_fullsized":"2024-04-25T01:31:21.874013+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:43:33.777319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867324+0000","last_change":"2024-04-25T01:30:34.354115+0000","last_active":"2024-04-25T01:31:21.867324+0000","last_peered":"2024-04-25T01:31:21.867324+0000","last_clean":"2024-04-25T01:31:21.867324+0000","last_became_active":"2024-04-25T01:30:34.353722+0000","last_became_peered":"2024-04-25T01:30:34.353722+0000","last_unstale":"2024-04-25T01:31:21.867324+0000","last_undegraded":"2024-04-25T01:31:21.867324+0000","last_fullsized":"2024-04-25T01:31:21.867324+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:22.003521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867270+0000","last_change":"2024-04-25T01:30:38.380914+0000","last_active":"2024-04-25T01:31:21.867270+0000","last_peered":"2024-04-25T01:31:21.867270+0000","last_clean":"2024-04-25T01:31:21.867270+0000","last_became_active":"2024-04-25T01:30:38.380477+0000","last_became_peered":"2024-04-25T01:30:38.380477+0000","last_unstale":"2024-04-25T01:31:21.867270+0000","last_undegraded":"2024-04-25T01:31:21.867270+0000","last_fullsized":"2024-04-25T01:31:21.867270+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:23:27.882266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.875670+0000","last_change":"2024-04-25T01:30:33.345772+0000","last_active":"2024-04-25T01:31:21.875670+0000","last_peered":"2024-04-25T01:31:21.875670+0000","last_clean":"2024-04-25T01:31:21.875670+0000","last_became_active":"2024-04-25T01:30:33.345455+0000","last_became_peered":"2024-04-25T01:30:33.345455+0000","last_unstale":"2024-04-25T01:31:21.875670+0000","last_undegraded":"2024-04-25T01:31:21.875670+0000","last_fullsized":"2024-04-25T01:31:21.875670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:14:57.465161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869815+0000","last_change":"2024-04-25T01:30:34.358665+0000","last_active":"2024-04-25T01:31:21.869815+0000","last_peered":"2024-04-25T01:31:21.869815+0000","last_clean":"2024-04-25T01:31:21.869815+0000","last_became_active":"2024-04-25T01:30:34.358284+0000","last_became_peered":"2024-04-25T01:30:34.358284+0000","last_unstale":"2024-04-25T01:31:21.869815+0000","last_undegraded":"2024-04-25T01:31:21.869815+0000","last_fullsized":"2024-04-25T01:31:21.869815+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:12:22.832265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873969+0000","last_change":"2024-04-25T01:30:38.450027+0000","last_active":"2024-04-25T01:31:21.873969+0000","last_peered":"2024-04-25T01:31:21.873969+0000","last_clean":"2024-04-25T01:31:21.873969+0000","last_became_active":"2024-04-25T01:30:38.449564+0000","last_became_peered":"2024-04-25T01:30:38.449564+0000","last_unstale":"2024-04-25T01:31:21.873969+0000","last_undegraded":"2024-04-25T01:31:21.873969+0000","last_fullsized":"2024-04-25T01:31:21.873969+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:34:26.542351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"91'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.870455+0000","last_change":"2024-04-25T01:30:33.349757+0000","last_active":"2024-04-25T01:31:21.870455+0000","last_peered":"2024-04-25T01:31:21.870455+0000","last_clean":"2024-04-25T01:31:21.870455+0000","last_became_active":"2024-04-25T01:30:33.349439+0000","last_became_peered":"2024-04-25T01:30:33.349439+0000","last_unstale":"2024-04-25T01:31:21.870455+0000","last_undegraded":"2024-04-25T01:31:21.870455+0000","last_fullsized":"2024-04-25T01:31:21.870455+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:35.514160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872423+0000","last_change":"2024-04-25T01:30:34.367310+0000","last_active":"2024-04-25T01:31:21.872423+0000","last_peered":"2024-04-25T01:31:21.872423+0000","last_clean":"2024-04-25T01:31:21.872423+0000","last_became_active":"2024-04-25T01:30:34.366950+0000","last_became_peered":"2024-04-25T01:30:34.366950+0000","last_unstale":"2024-04-25T01:31:21.872423+0000","last_undegraded":"2024-04-25T01:31:21.872423+0000","last_fullsized":"2024-04-25T01:31:21.872423+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:26:17.700512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866608+0000","last_change":"2024-04-25T01:30:38.372259+0000","last_active":"2024-04-25T01:31:21.866608+0000","last_peered":"2024-04-25T01:31:21.866608+0000","last_clean":"2024-04-25T01:31:21.866608+0000","last_became_active":"2024-04-25T01:30:38.371847+0000","last_became_peered":"2024-04-25T01:30:38.371847+0000","last_unstale":"2024-04-25T01:31:21.866608+0000","last_undegraded":"2024-04-25T01:31:21.866608+0000","last_fullsized":"2024-04-25T01:31:21.866608+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:55.047739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'5","reported_seq":103,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:32:09.903350+0000","last_change":"2024-04-25T01:30:33.352078+0000","last_active":"2024-04-25T01:32:09.903350+0000","last_peered":"2024-04-25T01:32:09.903350+0000","last_clean":"2024-04-25T01:32:09.903350+0000","last_became_active":"2024-04-25T01:30:33.351725+0000","last_became_peered":"2024-04-25T01:30:33.351725+0000","last_unstale":"2024-04-25T01:32:09.903350+0000","last_undegraded":"2024-04-25T01:32:09.903350+0000","last_fullsized":"2024-04-25T01:32:09.903350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:55:55.236040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872450+0000","last_change":"2024-04-25T01:30:34.360324+0000","last_active":"2024-04-25T01:31:21.872450+0000","last_peered":"2024-04-25T01:31:21.872450+0000","last_clean":"2024-04-25T01:31:21.872450+0000","last_became_active":"2024-04-25T01:30:34.359114+0000","last_became_peered":"2024-04-25T01:30:34.359114+0000","last_unstale":"2024-04-25T01:31:21.872450+0000","last_undegraded":"2024-04-25T01:31:21.872450+0000","last_fullsized":"2024-04-25T01:31:21.872450+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:51.927640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866716+0000","last_change":"2024-04-25T01:30:38.450650+0000","last_active":"2024-04-25T01:31:21.866716+0000","last_peered":"2024-04-25T01:31:21.866716+0000","last_clean":"2024-04-25T01:31:21.866716+0000","last_became_active":"2024-04-25T01:30:38.450149+0000","last_became_peered":"2024-04-25T01:30:38.450149+0000","last_unstale":"2024-04-25T01:31:21.866716+0000","last_undegraded":"2024-04-25T01:31:21.866716+0000","last_fullsized":"2024-04-25T01:31:21.866716+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:34.630803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867436+0000","last_change":"2024-04-25T01:30:34.352961+0000","last_active":"2024-04-25T01:31:21.867436+0000","last_peered":"2024-04-25T01:31:21.867436+0000","last_clean":"2024-04-25T01:31:21.867436+0000","last_became_active":"2024-04-25T01:30:34.352657+0000","last_became_peered":"2024-04-25T01:30:34.352657+0000","last_unstale":"2024-04-25T01:31:21.867436+0000","last_undegraded":"2024-04-25T01:31:21.867436+0000","last_fullsized":"2024-04-25T01:31:21.867436+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:20.934228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":101,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:32:07.899693+0000","last_change":"2024-04-25T01:30:33.344562+0000","last_active":"2024-04-25T01:32:07.899693+0000","last_peered":"2024-04-25T01:32:07.899693+0000","last_clean":"2024-04-25T01:32:07.899693+0000","last_became_active":"2024-04-25T01:30:33.344212+0000","last_became_peered":"2024-04-25T01:30:33.344212+0000","last_unstale":"2024-04-25T01:32:07.899693+0000","last_undegraded":"2024-04-25T01:32:07.899693+0000","last_fullsized":"2024-04-25T01:32:07.899693+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:20:51.110618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049258+0000","last_change":"2024-04-25T01:30:38.384316+0000","last_active":"2024-04-25T01:31:22.049258+0000","last_peered":"2024-04-25T01:31:22.049258+0000","last_clean":"2024-04-25T01:31:22.049258+0000","last_became_active":"2024-04-25T01:30:38.383775+0000","last_became_peered":"2024-04-25T01:30:38.383775+0000","last_unstale":"2024-04-25T01:31:22.049258+0000","last_undegraded":"2024-04-25T01:31:22.049258+0000","last_fullsized":"2024-04-25T01:31:22.049258+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:54.165691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.050351+0000","last_change":"2024-04-25T01:31:22.050014+0000","last_active":"2024-04-25T01:31:22.050351+0000","last_peered":"2024-04-25T01:31:22.050351+0000","last_clean":"2024-04-25T01:31:22.050351+0000","last_became_active":"2024-04-25T01:31:22.049718+0000","last_became_peered":"2024-04-25T01:31:22.049718+0000","last_unstale":"2024-04-25T01:31:22.050351+0000","last_undegraded":"2024-04-25T01:31:22.050351+0000","last_fullsized":"2024-04-25T01:31:22.050351+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:40:22.233372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867550+0000","last_change":"2024-04-25T01:30:33.345283+0000","last_active":"2024-04-25T01:31:21.867550+0000","last_peered":"2024-04-25T01:31:21.867550+0000","last_clean":"2024-04-25T01:31:21.867550+0000","last_became_active":"2024-04-25T01:30:33.344903+0000","last_became_peered":"2024-04-25T01:30:33.344903+0000","last_unstale":"2024-04-25T01:31:21.867550+0000","last_undegraded":"2024-04-25T01:31:21.867550+0000","last_fullsized":"2024-04-25T01:31:21.867550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:57:01.982850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":364,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.048240+0000","last_change":"2024-04-25T01:29:17.689639+0000","last_active":"2024-04-25T01:31:22.048240+0000","last_peered":"2024-04-25T01:31:22.048240+0000","last_clean":"2024-04-25T01:31:22.048240+0000","last_became_active":"2024-04-25T01:29:17.689046+0000","last_became_peered":"2024-04-25T01:29:17.689046+0000","last_unstale":"2024-04-25T01:31:22.048240+0000","last_undegraded":"2024-04-25T01:31:22.048240+0000","last_fullsized":"2024-04-25T01:31:22.048240+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:48.767945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:48.767945+0000","last_clean_scrub_stamp":"2024-04-25T01:25:48.767945+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:04:06.092632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.868030+0000","last_change":"2024-04-25T01:30:38.450938+0000","last_active":"2024-04-25T01:31:21.868030+0000","last_peered":"2024-04-25T01:31:21.868030+0000","last_clean":"2024-04-25T01:31:21.868030+0000","last_became_active":"2024-04-25T01:30:38.450488+0000","last_became_peered":"2024-04-25T01:30:38.450488+0000","last_unstale":"2024-04-25T01:31:21.868030+0000","last_undegraded":"2024-04-25T01:31:21.868030+0000","last_fullsized":"2024-04-25T01:31:21.868030+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:44.778133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.875719+0000","last_change":"2024-04-25T01:31:21.875719+0000","last_active":"2024-04-25T01:31:21.875719+0000","last_peered":"2024-04-25T01:31:21.875719+0000","last_clean":"2024-04-25T01:31:21.875719+0000","last_became_active":"2024-04-25T01:31:21.875283+0000","last_became_peered":"2024-04-25T01:31:21.875283+0000","last_unstale":"2024-04-25T01:31:21.875719+0000","last_undegraded":"2024-04-25T01:31:21.875719+0000","last_fullsized":"2024-04-25T01:31:21.875719+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:12:39.097019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.048316+0000","last_change":"2024-04-25T01:30:33.348198+0000","last_active":"2024-04-25T01:31:22.048316+0000","last_peered":"2024-04-25T01:31:22.048316+0000","last_clean":"2024-04-25T01:31:22.048316+0000","last_became_active":"2024-04-25T01:30:33.347876+0000","last_became_peered":"2024-04-25T01:30:33.347876+0000","last_unstale":"2024-04-25T01:31:22.048316+0000","last_undegraded":"2024-04-25T01:31:22.048316+0000","last_fullsized":"2024-04-25T01:31:22.048316+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:48:33.116571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869567+0000","last_change":"2024-04-25T01:30:38.372331+0000","last_active":"2024-04-25T01:31:21.869567+0000","last_peered":"2024-04-25T01:31:21.869567+0000","last_clean":"2024-04-25T01:31:21.869567+0000","last_became_active":"2024-04-25T01:30:38.371923+0000","last_became_peered":"2024-04-25T01:30:38.371923+0000","last_unstale":"2024-04-25T01:31:21.869567+0000","last_undegraded":"2024-04-25T01:31:21.869567+0000","last_fullsized":"2024-04-25T01:31:21.869567+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:06.416967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.874700+0000","last_change":"2024-04-25T01:30:33.346606+0000","last_active":"2024-04-25T01:31:21.874700+0000","last_peered":"2024-04-25T01:31:21.874700+0000","last_clean":"2024-04-25T01:31:21.874700+0000","last_became_active":"2024-04-25T01:30:33.346297+0000","last_became_peered":"2024-04-25T01:30:33.346297+0000","last_unstale":"2024-04-25T01:31:21.874700+0000","last_undegraded":"2024-04-25T01:31:21.874700+0000","last_fullsized":"2024-04-25T01:31:21.874700+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:07:16.069285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866732+0000","last_change":"2024-04-25T01:30:34.349136+0000","last_active":"2024-04-25T01:31:21.866732+0000","last_peered":"2024-04-25T01:31:21.866732+0000","last_clean":"2024-04-25T01:31:21.866732+0000","last_became_active":"2024-04-25T01:30:34.348810+0000","last_became_peered":"2024-04-25T01:30:34.348810+0000","last_unstale":"2024-04-25T01:31:21.866732+0000","last_undegraded":"2024-04-25T01:31:21.866732+0000","last_fullsized":"2024-04-25T01:31:21.866732+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:39:34.593986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872418+0000","last_change":"2024-04-25T01:30:38.450428+0000","last_active":"2024-04-25T01:31:21.872418+0000","last_peered":"2024-04-25T01:31:21.872418+0000","last_clean":"2024-04-25T01:31:21.872418+0000","last_became_active":"2024-04-25T01:30:38.449971+0000","last_became_peered":"2024-04-25T01:30:38.449971+0000","last_unstale":"2024-04-25T01:31:21.872418+0000","last_undegraded":"2024-04-25T01:31:21.872418+0000","last_fullsized":"2024-04-25T01:31:21.872418+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:52:37.046462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'11","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.870939+0000","last_change":"2024-04-25T01:30:33.348547+0000","last_active":"2024-04-25T01:31:21.870939+0000","last_peered":"2024-04-25T01:31:21.870939+0000","last_clean":"2024-04-25T01:31:21.870939+0000","last_became_active":"2024-04-25T01:30:33.348181+0000","last_became_peered":"2024-04-25T01:30:33.348181+0000","last_unstale":"2024-04-25T01:31:21.870939+0000","last_undegraded":"2024-04-25T01:31:21.870939+0000","last_fullsized":"2024-04-25T01:31:21.870939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:11:58.696829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873436+0000","last_change":"2024-04-25T01:30:34.351987+0000","last_active":"2024-04-25T01:31:21.873436+0000","last_peered":"2024-04-25T01:31:21.873436+0000","last_clean":"2024-04-25T01:31:21.873436+0000","last_became_active":"2024-04-25T01:30:34.351571+0000","last_became_peered":"2024-04-25T01:30:34.351571+0000","last_unstale":"2024-04-25T01:31:21.873436+0000","last_undegraded":"2024-04-25T01:31:21.873436+0000","last_fullsized":"2024-04-25T01:31:21.873436+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:23.585865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872904+0000","last_change":"2024-04-25T01:30:38.372566+0000","last_active":"2024-04-25T01:31:21.872904+0000","last_peered":"2024-04-25T01:31:21.872904+0000","last_clean":"2024-04-25T01:31:21.872904+0000","last_became_active":"2024-04-25T01:30:38.372102+0000","last_became_peered":"2024-04-25T01:30:38.372102+0000","last_unstale":"2024-04-25T01:31:21.872904+0000","last_undegraded":"2024-04-25T01:31:21.872904+0000","last_fullsized":"2024-04-25T01:31:21.872904+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:22:16.791842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867899+0000","last_change":"2024-04-25T01:30:33.344065+0000","last_active":"2024-04-25T01:31:21.867899+0000","last_peered":"2024-04-25T01:31:21.867899+0000","last_clean":"2024-04-25T01:31:21.867899+0000","last_became_active":"2024-04-25T01:30:33.343700+0000","last_became_peered":"2024-04-25T01:30:33.343700+0000","last_unstale":"2024-04-25T01:31:21.867899+0000","last_undegraded":"2024-04-25T01:31:21.867899+0000","last_fullsized":"2024-04-25T01:31:21.867899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:54:08.265744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869647+0000","last_change":"2024-04-25T01:30:34.354644+0000","last_active":"2024-04-25T01:31:21.869647+0000","last_peered":"2024-04-25T01:31:21.869647+0000","last_clean":"2024-04-25T01:31:21.869647+0000","last_became_active":"2024-04-25T01:30:34.354330+0000","last_became_peered":"2024-04-25T01:30:34.354330+0000","last_unstale":"2024-04-25T01:31:21.869647+0000","last_undegraded":"2024-04-25T01:31:21.869647+0000","last_fullsized":"2024-04-25T01:31:21.869647+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:31:50.399983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867522+0000","last_change":"2024-04-25T01:30:38.450743+0000","last_active":"2024-04-25T01:31:21.867522+0000","last_peered":"2024-04-25T01:31:21.867522+0000","last_clean":"2024-04-25T01:31:21.867522+0000","last_became_active":"2024-04-25T01:30:38.450303+0000","last_became_peered":"2024-04-25T01:30:38.450303+0000","last_unstale":"2024-04-25T01:31:21.867522+0000","last_undegraded":"2024-04-25T01:31:21.867522+0000","last_fullsized":"2024-04-25T01:31:21.867522+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:37.919250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049267+0000","last_change":"2024-04-25T01:30:33.350709+0000","last_active":"2024-04-25T01:31:22.049267+0000","last_peered":"2024-04-25T01:31:22.049267+0000","last_clean":"2024-04-25T01:31:22.049267+0000","last_became_active":"2024-04-25T01:30:33.350387+0000","last_became_peered":"2024-04-25T01:30:33.350387+0000","last_unstale":"2024-04-25T01:31:22.049267+0000","last_undegraded":"2024-04-25T01:31:22.049267+0000","last_fullsized":"2024-04-25T01:31:22.049267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:30:01.252121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873782+0000","last_change":"2024-04-25T01:30:34.362040+0000","last_active":"2024-04-25T01:31:21.873782+0000","last_peered":"2024-04-25T01:31:21.873782+0000","last_clean":"2024-04-25T01:31:21.873782+0000","last_became_active":"2024-04-25T01:30:34.361734+0000","last_became_peered":"2024-04-25T01:30:34.361734+0000","last_unstale":"2024-04-25T01:31:21.873782+0000","last_undegraded":"2024-04-25T01:31:21.873782+0000","last_fullsized":"2024-04-25T01:31:21.873782+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:11:28.409764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.877148+0000","last_change":"2024-04-25T01:31:21.877148+0000","last_active":"2024-04-25T01:31:21.877148+0000","last_peered":"2024-04-25T01:31:21.877148+0000","last_clean":"2024-04-25T01:31:21.877148+0000","last_became_active":"2024-04-25T01:31:21.876665+0000","last_became_peered":"2024-04-25T01:31:21.876665+0000","last_unstale":"2024-04-25T01:31:21.877148+0000","last_undegraded":"2024-04-25T01:31:21.877148+0000","last_fullsized":"2024-04-25T01:31:21.877148+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:32:38.425434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,8],"acting":[9,3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873084+0000","last_change":"2024-04-25T01:30:33.350207+0000","last_active":"2024-04-25T01:31:21.873084+0000","last_peered":"2024-04-25T01:31:21.873084+0000","last_clean":"2024-04-25T01:31:21.873084+0000","last_became_active":"2024-04-25T01:30:33.349892+0000","last_became_peered":"2024-04-25T01:30:33.349892+0000","last_unstale":"2024-04-25T01:31:21.873084+0000","last_undegraded":"2024-04-25T01:31:21.873084+0000","last_fullsized":"2024-04-25T01:31:21.873084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:26:35.186486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869326+0000","last_change":"2024-04-25T01:30:34.355196+0000","last_active":"2024-04-25T01:31:21.869326+0000","last_peered":"2024-04-25T01:31:21.869326+0000","last_clean":"2024-04-25T01:31:21.869326+0000","last_became_active":"2024-04-25T01:30:34.354794+0000","last_became_peered":"2024-04-25T01:30:34.354794+0000","last_unstale":"2024-04-25T01:31:21.869326+0000","last_undegraded":"2024-04-25T01:31:21.869326+0000","last_fullsized":"2024-04-25T01:31:21.869326+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:43:37.495974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867406+0000","last_change":"2024-04-25T01:30:38.384159+0000","last_active":"2024-04-25T01:31:21.867406+0000","last_peered":"2024-04-25T01:31:21.867406+0000","last_clean":"2024-04-25T01:31:21.867406+0000","last_became_active":"2024-04-25T01:30:38.383676+0000","last_became_peered":"2024-04-25T01:30:38.383676+0000","last_unstale":"2024-04-25T01:31:21.867406+0000","last_undegraded":"2024-04-25T01:31:21.867406+0000","last_fullsized":"2024-04-25T01:31:21.867406+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:28.491522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.889311+0000","last_change":"2024-04-25T01:31:21.889311+0000","last_active":"2024-04-25T01:31:21.889311+0000","last_peered":"2024-04-25T01:31:21.889311+0000","last_clean":"2024-04-25T01:31:21.889311+0000","last_became_active":"2024-04-25T01:31:21.877027+0000","last_became_peered":"2024-04-25T01:31:21.877027+0000","last_unstale":"2024-04-25T01:31:21.889311+0000","last_undegraded":"2024-04-25T01:31:21.889311+0000","last_fullsized":"2024-04-25T01:31:21.889311+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:02.206671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872996+0000","last_change":"2024-04-25T01:30:34.358110+0000","last_active":"2024-04-25T01:31:21.872996+0000","last_peered":"2024-04-25T01:31:21.872996+0000","last_clean":"2024-04-25T01:31:21.872996+0000","last_became_active":"2024-04-25T01:30:34.357555+0000","last_became_peered":"2024-04-25T01:30:34.357555+0000","last_unstale":"2024-04-25T01:31:21.872996+0000","last_undegraded":"2024-04-25T01:31:21.872996+0000","last_fullsized":"2024-04-25T01:31:21.872996+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:11.636320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867320+0000","last_change":"2024-04-25T01:30:38.450680+0000","last_active":"2024-04-25T01:31:21.867320+0000","last_peered":"2024-04-25T01:31:21.867320+0000","last_clean":"2024-04-25T01:31:21.867320+0000","last_became_active":"2024-04-25T01:30:38.450168+0000","last_became_peered":"2024-04-25T01:30:38.450168+0000","last_unstale":"2024-04-25T01:31:21.867320+0000","last_undegraded":"2024-04-25T01:31:21.867320+0000","last_fullsized":"2024-04-25T01:31:21.867320+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:39:24.236371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.874429+0000","last_change":"2024-04-25T01:30:33.343848+0000","last_active":"2024-04-25T01:31:21.874429+0000","last_peered":"2024-04-25T01:31:21.874429+0000","last_clean":"2024-04-25T01:31:21.874429+0000","last_became_active":"2024-04-25T01:30:33.343498+0000","last_became_peered":"2024-04-25T01:30:33.343498+0000","last_unstale":"2024-04-25T01:31:21.874429+0000","last_undegraded":"2024-04-25T01:31:21.874429+0000","last_fullsized":"2024-04-25T01:31:21.874429+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:58:09.184571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049053+0000","last_change":"2024-04-25T01:30:34.361994+0000","last_active":"2024-04-25T01:31:22.049053+0000","last_peered":"2024-04-25T01:31:22.049053+0000","last_clean":"2024-04-25T01:31:22.049053+0000","last_became_active":"2024-04-25T01:30:34.361532+0000","last_became_peered":"2024-04-25T01:30:34.361532+0000","last_unstale":"2024-04-25T01:31:22.049053+0000","last_undegraded":"2024-04-25T01:31:22.049053+0000","last_fullsized":"2024-04-25T01:31:22.049053+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:30.667672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.048447+0000","last_change":"2024-04-25T01:30:38.369041+0000","last_active":"2024-04-25T01:31:22.048447+0000","last_peered":"2024-04-25T01:31:22.048447+0000","last_clean":"2024-04-25T01:31:22.048447+0000","last_became_active":"2024-04-25T01:30:38.368586+0000","last_became_peered":"2024-04-25T01:30:38.368586+0000","last_unstale":"2024-04-25T01:31:22.048447+0000","last_undegraded":"2024-04-25T01:31:22.048447+0000","last_fullsized":"2024-04-25T01:31:22.048447+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:58:53.743012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'6","reported_seq":106,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:32:10.911932+0000","last_change":"2024-04-25T01:30:33.345501+0000","last_active":"2024-04-25T01:32:10.911932+0000","last_peered":"2024-04-25T01:32:10.911932+0000","last_clean":"2024-04-25T01:32:10.911932+0000","last_became_active":"2024-04-25T01:30:33.345179+0000","last_became_peered":"2024-04-25T01:30:33.345179+0000","last_unstale":"2024-04-25T01:32:10.911932+0000","last_undegraded":"2024-04-25T01:32:10.911932+0000","last_fullsized":"2024-04-25T01:32:10.911932+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:29:41.275480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049572+0000","last_change":"2024-04-25T01:30:34.361631+0000","last_active":"2024-04-25T01:31:22.049572+0000","last_peered":"2024-04-25T01:31:22.049572+0000","last_clean":"2024-04-25T01:31:22.049572+0000","last_became_active":"2024-04-25T01:30:34.360266+0000","last_became_peered":"2024-04-25T01:30:34.360266+0000","last_unstale":"2024-04-25T01:31:22.049572+0000","last_undegraded":"2024-04-25T01:31:22.049572+0000","last_fullsized":"2024-04-25T01:31:22.049572+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:03.094166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869056+0000","last_change":"2024-04-25T01:30:38.380779+0000","last_active":"2024-04-25T01:31:21.869056+0000","last_peered":"2024-04-25T01:31:21.869056+0000","last_clean":"2024-04-25T01:31:21.869056+0000","last_became_active":"2024-04-25T01:30:38.379959+0000","last_became_peered":"2024-04-25T01:30:38.379959+0000","last_unstale":"2024-04-25T01:31:21.869056+0000","last_undegraded":"2024-04-25T01:31:21.869056+0000","last_fullsized":"2024-04-25T01:31:21.869056+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:48:01.816311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"91'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867900+0000","last_change":"2024-04-25T01:30:33.345730+0000","last_active":"2024-04-25T01:31:21.867900+0000","last_peered":"2024-04-25T01:31:21.867900+0000","last_clean":"2024-04-25T01:31:21.867900+0000","last_became_active":"2024-04-25T01:30:33.344832+0000","last_became_peered":"2024-04-25T01:30:33.344832+0000","last_unstale":"2024-04-25T01:31:21.867900+0000","last_undegraded":"2024-04-25T01:31:21.867900+0000","last_fullsized":"2024-04-25T01:31:21.867900+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:47.018811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049042+0000","last_change":"2024-04-25T01:30:34.355324+0000","last_active":"2024-04-25T01:31:22.049042+0000","last_peered":"2024-04-25T01:31:22.049042+0000","last_clean":"2024-04-25T01:31:22.049042+0000","last_became_active":"2024-04-25T01:30:34.354954+0000","last_became_peered":"2024-04-25T01:30:34.354954+0000","last_unstale":"2024-04-25T01:31:22.049042+0000","last_undegraded":"2024-04-25T01:31:22.049042+0000","last_fullsized":"2024-04-25T01:31:22.049042+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:36:41.932391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867087+0000","last_change":"2024-04-25T01:30:38.374386+0000","last_active":"2024-04-25T01:31:21.867087+0000","last_peered":"2024-04-25T01:31:21.867087+0000","last_clean":"2024-04-25T01:31:21.867087+0000","last_became_active":"2024-04-25T01:30:38.373454+0000","last_became_peered":"2024-04-25T01:30:38.373454+0000","last_unstale":"2024-04-25T01:31:21.867087+0000","last_undegraded":"2024-04-25T01:31:21.867087+0000","last_fullsized":"2024-04-25T01:31:21.867087+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:54.456218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"91'5","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866862+0000","last_change":"2024-04-25T01:30:33.339821+0000","last_active":"2024-04-25T01:31:21.866862+0000","last_peered":"2024-04-25T01:31:21.866862+0000","last_clean":"2024-04-25T01:31:21.866862+0000","last_became_active":"2024-04-25T01:30:33.339500+0000","last_became_peered":"2024-04-25T01:30:33.339500+0000","last_unstale":"2024-04-25T01:31:21.866862+0000","last_undegraded":"2024-04-25T01:31:21.866862+0000","last_fullsized":"2024-04-25T01:31:21.866862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:44:25.880828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049617+0000","last_change":"2024-04-25T01:30:34.362736+0000","last_active":"2024-04-25T01:31:22.049617+0000","last_peered":"2024-04-25T01:31:22.049617+0000","last_clean":"2024-04-25T01:31:22.049617+0000","last_became_active":"2024-04-25T01:30:34.362371+0000","last_became_peered":"2024-04-25T01:30:34.362371+0000","last_unstale":"2024-04-25T01:31:22.049617+0000","last_undegraded":"2024-04-25T01:31:22.049617+0000","last_fullsized":"2024-04-25T01:31:22.049617+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:57:36.088231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869403+0000","last_change":"2024-04-25T01:30:34.359217+0000","last_active":"2024-04-25T01:31:21.869403+0000","last_peered":"2024-04-25T01:31:21.869403+0000","last_clean":"2024-04-25T01:31:21.869403+0000","last_became_active":"2024-04-25T01:30:34.357095+0000","last_became_peered":"2024-04-25T01:30:34.357095+0000","last_unstale":"2024-04-25T01:31:21.869403+0000","last_undegraded":"2024-04-25T01:31:21.869403+0000","last_fullsized":"2024-04-25T01:31:21.869403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:58.603484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873268+0000","last_change":"2024-04-25T01:30:34.359513+0000","last_active":"2024-04-25T01:31:21.873268+0000","last_peered":"2024-04-25T01:31:21.873268+0000","last_clean":"2024-04-25T01:31:21.873268+0000","last_became_active":"2024-04-25T01:30:34.359089+0000","last_became_peered":"2024-04-25T01:30:34.359089+0000","last_unstale":"2024-04-25T01:31:21.873268+0000","last_undegraded":"2024-04-25T01:31:21.873268+0000","last_fullsized":"2024-04-25T01:31:21.873268+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:36.159703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872696+0000","last_change":"2024-04-25T01:30:34.368739+0000","last_active":"2024-04-25T01:31:21.872696+0000","last_peered":"2024-04-25T01:31:21.872696+0000","last_clean":"2024-04-25T01:31:21.872696+0000","last_became_active":"2024-04-25T01:30:34.368392+0000","last_became_peered":"2024-04-25T01:30:34.368392+0000","last_unstale":"2024-04-25T01:31:21.872696+0000","last_undegraded":"2024-04-25T01:31:21.872696+0000","last_fullsized":"2024-04-25T01:31:21.872696+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:39:32.241889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.877568+0000","last_change":"2024-04-25T01:31:21.877568+0000","last_active":"2024-04-25T01:31:21.877568+0000","last_peered":"2024-04-25T01:31:21.877568+0000","last_clean":"2024-04-25T01:31:21.877568+0000","last_became_active":"2024-04-25T01:31:21.877071+0000","last_became_peered":"2024-04-25T01:31:21.877071+0000","last_unstale":"2024-04-25T01:31:21.877568+0000","last_undegraded":"2024-04-25T01:31:21.877568+0000","last_fullsized":"2024-04-25T01:31:21.877568+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:02:24.789974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866839+0000","last_change":"2024-04-25T01:30:34.360979+0000","last_active":"2024-04-25T01:31:21.866839+0000","last_peered":"2024-04-25T01:31:21.866839+0000","last_clean":"2024-04-25T01:31:21.866839+0000","last_became_active":"2024-04-25T01:30:34.360630+0000","last_became_peered":"2024-04-25T01:30:34.360630+0000","last_unstale":"2024-04-25T01:31:21.866839+0000","last_undegraded":"2024-04-25T01:31:21.866839+0000","last_fullsized":"2024-04-25T01:31:21.866839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:05:19.284577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869923+0000","last_change":"2024-04-25T01:30:34.362621+0000","last_active":"2024-04-25T01:31:21.869923+0000","last_peered":"2024-04-25T01:31:21.869923+0000","last_clean":"2024-04-25T01:31:21.869923+0000","last_became_active":"2024-04-25T01:30:34.362290+0000","last_became_peered":"2024-04-25T01:30:34.362290+0000","last_unstale":"2024-04-25T01:31:21.869923+0000","last_undegraded":"2024-04-25T01:31:21.869923+0000","last_fullsized":"2024-04-25T01:31:21.869923+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:28:07.750906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873423+0000","last_change":"2024-04-25T01:30:34.351339+0000","last_active":"2024-04-25T01:31:21.873423+0000","last_peered":"2024-04-25T01:31:21.873423+0000","last_clean":"2024-04-25T01:31:21.873423+0000","last_became_active":"2024-04-25T01:30:34.350655+0000","last_became_peered":"2024-04-25T01:30:34.350655+0000","last_unstale":"2024-04-25T01:31:21.873423+0000","last_undegraded":"2024-04-25T01:31:21.873423+0000","last_fullsized":"2024-04-25T01:31:21.873423+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:05.322547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873518+0000","last_change":"2024-04-25T01:30:34.353876+0000","last_active":"2024-04-25T01:31:21.873518+0000","last_peered":"2024-04-25T01:31:21.873518+0000","last_clean":"2024-04-25T01:31:21.873518+0000","last_became_active":"2024-04-25T01:30:34.352942+0000","last_became_peered":"2024-04-25T01:30:34.352942+0000","last_unstale":"2024-04-25T01:31:21.873518+0000","last_undegraded":"2024-04-25T01:31:21.873518+0000","last_fullsized":"2024-04-25T01:31:21.873518+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:47.260171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867312+0000","last_change":"2024-04-25T01:30:34.352039+0000","last_active":"2024-04-25T01:31:21.867312+0000","last_peered":"2024-04-25T01:31:21.867312+0000","last_clean":"2024-04-25T01:31:21.867312+0000","last_became_active":"2024-04-25T01:30:34.351612+0000","last_became_peered":"2024-04-25T01:30:34.351612+0000","last_unstale":"2024-04-25T01:31:21.867312+0000","last_undegraded":"2024-04-25T01:31:21.867312+0000","last_fullsized":"2024-04-25T01:31:21.867312+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:03.607711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873300+0000","last_change":"2024-04-25T01:30:34.353692+0000","last_active":"2024-04-25T01:31:21.873300+0000","last_peered":"2024-04-25T01:31:21.873300+0000","last_clean":"2024-04-25T01:31:21.873300+0000","last_became_active":"2024-04-25T01:30:34.353035+0000","last_became_peered":"2024-04-25T01:30:34.353035+0000","last_unstale":"2024-04-25T01:31:21.873300+0000","last_undegraded":"2024-04-25T01:31:21.873300+0000","last_fullsized":"2024-04-25T01:31:21.873300+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:15:54.540315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.048744+0000","last_change":"2024-04-25T01:30:34.370797+0000","last_active":"2024-04-25T01:31:22.048744+0000","last_peered":"2024-04-25T01:31:22.048744+0000","last_clean":"2024-04-25T01:31:22.048744+0000","last_became_active":"2024-04-25T01:30:34.367533+0000","last_became_peered":"2024-04-25T01:30:34.367533+0000","last_unstale":"2024-04-25T01:31:22.048744+0000","last_undegraded":"2024-04-25T01:31:22.048744+0000","last_fullsized":"2024-04-25T01:31:22.048744+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:27:15.478808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866640+0000","last_change":"2024-04-25T01:30:34.364843+0000","last_active":"2024-04-25T01:31:21.866640+0000","last_peered":"2024-04-25T01:31:21.866640+0000","last_clean":"2024-04-25T01:31:21.866640+0000","last_became_active":"2024-04-25T01:30:34.364515+0000","last_became_peered":"2024-04-25T01:30:34.364515+0000","last_unstale":"2024-04-25T01:31:21.866640+0000","last_undegraded":"2024-04-25T01:31:21.866640+0000","last_fullsized":"2024-04-25T01:31:21.866640+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:50:03.517814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869814+0000","last_change":"2024-04-25T01:30:34.363715+0000","last_active":"2024-04-25T01:31:21.869814+0000","last_peered":"2024-04-25T01:31:21.869814+0000","last_clean":"2024-04-25T01:31:21.869814+0000","last_became_active":"2024-04-25T01:30:34.363383+0000","last_became_peered":"2024-04-25T01:30:34.363383+0000","last_unstale":"2024-04-25T01:31:21.869814+0000","last_undegraded":"2024-04-25T01:31:21.869814+0000","last_fullsized":"2024-04-25T01:31:21.869814+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:50:18.322550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69537,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":756,"num_read_kb":455,"num_write":241,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":208611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318305,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1060864,"data_stored":510000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579943,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37044,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708108,"statfs":{"total":95995035648,"available":95957102592,"internally_reserved":0,"allocated":1748992,"data_stored":1158690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743469,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":513567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005105,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1073152,"data_stored":509849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266742,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28656,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716496,"statfs":{"total":95995035648,"available":95965691904,"internally_reserved":0,"allocated":1671168,"data_stored":1106346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430268,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":504975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1122304,"data_stored":568338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36396,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708756,"statfs":{"total":95995035648,"available":95957766144,"internally_reserved":0,"allocated":1077248,"data_stored":506822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36444,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708708,"statfs":{"total":95995035648,"available":95957716992,"internally_reserved":0,"allocated":1134592,"data_stored":569272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1064960,"data_stored":504683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888134,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1060864,"data_stored":509049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691906,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32820,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712332,"statfs":{"total":95995035648,"available":95961427968,"internally_reserved":0,"allocated":1679360,"data_stored":1101928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":65019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":64069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:32:15.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T01:32:15.512 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:32:16.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:16 smithi028 ceph-mon[29390]: pgmap v350: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:16.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:16 smithi028 ceph-mon[29390]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:32:16.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:16 smithi130 ceph-mon[32702]: pgmap v350: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:16.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:16 smithi130 ceph-mon[32702]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:32:16.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:16 smithi032 ceph-mon[32461]: pgmap v350: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:16.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:16 smithi032 ceph-mon[32461]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:32:17.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.467+0000 7ff13eda3700 1 -- 172.21.15.28:0/374355625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1381022e0 msgr2=0x7ff138102760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:17.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.467+0000 7ff13eda3700 1 --2- 172.21.15.28:0/374355625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1381022e0 0x7ff138102760 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff1240099c0 tx=0x7ff124009cd0 comp rx=0 tx=0).stop 2024-04-25T01:32:17.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.467+0000 7ff13eda3700 1 -- 172.21.15.28:0/374355625 shutdown_connections 2024-04-25T01:32:17.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.467+0000 7ff13eda3700 1 --2- 172.21.15.28:0/374355625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff138102ca0 0x7ff13810f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:17.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.467+0000 7ff13eda3700 1 --2- 172.21.15.28:0/374355625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1381082f0 0x7ff1381086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:17.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.467+0000 7ff13eda3700 1 --2- 172.21.15.28:0/374355625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1381022e0 0x7ff138102760 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:17.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.467+0000 7ff13eda3700 1 -- 172.21.15.28:0/374355625 >> 172.21.15.28:0/374355625 conn(0x7ff138079390 msgr2=0x7ff1380797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:17.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.467+0000 7ff13eda3700 1 -- 172.21.15.28:0/374355625 shutdown_connections 2024-04-25T01:32:17.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.467+0000 7ff13eda3700 1 -- 172.21.15.28:0/374355625 wait complete. 2024-04-25T01:32:17.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.468+0000 7ff13eda3700 1 Processor -- start 2024-04-25T01:32:17.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.468+0000 7ff13eda3700 1 -- start start 2024-04-25T01:32:17.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.468+0000 7ff13eda3700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1381022e0 0x7ff13819e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:17.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.468+0000 7ff13eda3700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff138102ca0 0x7ff13819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.468+0000 7ff13eda3700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1381082f0 0x7ff1381987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.468+0000 7ff13eda3700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1381125f0 con 0x7ff1381022e0 2024-04-25T01:32:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.468+0000 7ff13eda3700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff138112470 con 0x7ff1381082f0 2024-04-25T01:32:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.468+0000 7ff13eda3700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff138112770 con 0x7ff138102ca0 2024-04-25T01:32:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.468+0000 7ff13cb3f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1381022e0 0x7ff13819e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.468+0000 7ff13cb3f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1381022e0 0x7ff13819e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45182/0 (socket says 172.21.15.28:45182) 2024-04-25T01:32:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.468+0000 7ff13cb3f700 1 -- 172.21.15.28:0/1762609162 learned_addr learned my addr 172.21.15.28:0/1762609162 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.468+0000 7ff13d340700 1 --2- 172.21.15.28:0/1762609162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1381082f0 0x7ff1381987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.468+0000 7ff137fff700 1 --2- 172.21.15.28:0/1762609162 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff138102ca0 0x7ff13819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.468+0000 7ff13cb3f700 1 -- 172.21.15.28:0/1762609162 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff138102ca0 msgr2=0x7ff13819eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.469+0000 7ff13cb3f700 1 --2- 172.21.15.28:0/1762609162 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff138102ca0 0x7ff13819eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.469+0000 7ff13cb3f700 1 -- 172.21.15.28:0/1762609162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1381082f0 msgr2=0x7ff1381987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:17.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.469+0000 7ff13cb3f700 1 --2- 172.21.15.28:0/1762609162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1381082f0 0x7ff1381987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:17.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.469+0000 7ff13cb3f700 1 -- 172.21.15.28:0/1762609162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff138198f90 con 0x7ff1381022e0 2024-04-25T01:32:17.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.469+0000 7ff13d340700 1 --2- 172.21.15.28:0/1762609162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1381082f0 0x7ff1381987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:17.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.469+0000 7ff13cb3f700 1 --2- 172.21.15.28:0/1762609162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1381022e0 0x7ff13819e630 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00b530 tx=0x7ff12c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:17.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.469+0000 7ff135ffb700 1 -- 172.21.15.28:0/1762609162 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c015020 con 0x7ff1381022e0 2024-04-25T01:32:17.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.469+0000 7ff13eda3700 1 -- 172.21.15.28:0/1762609162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff138199280 con 0x7ff1381022e0 2024-04-25T01:32:17.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.469+0000 7ff135ffb700 1 -- 172.21.15.28:0/1762609162 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff12c011d10 con 0x7ff1381022e0 2024-04-25T01:32:17.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.469+0000 7ff135ffb700 1 -- 172.21.15.28:0/1762609162 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c010e50 con 0x7ff1381022e0 2024-04-25T01:32:17.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.469+0000 7ff13eda3700 1 -- 172.21.15.28:0/1762609162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1381a5490 con 0x7ff1381022e0 2024-04-25T01:32:17.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.470+0000 7ff13eda3700 1 -- 172.21.15.28:0/1762609162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1381039a0 con 0x7ff1381022e0 2024-04-25T01:32:17.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.471+0000 7ff135ffb700 1 -- 172.21.15.28:0/1762609162 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff12c018020 con 0x7ff1381022e0 2024-04-25T01:32:17.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.472+0000 7ff135ffb700 1 --2- 172.21.15.28:0/1762609162 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1280a2e50 0x7ff1280a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:17.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.472+0000 7ff135ffb700 1 -- 172.21.15.28:0/1762609162 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7ff12c0cecb0 con 0x7ff1381022e0 2024-04-25T01:32:17.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.472+0000 7ff137fff700 1 --2- 172.21.15.28:0/1762609162 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1280a2e50 0x7ff1280a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:17.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.472+0000 7ff137fff700 1 --2- 172.21.15.28:0/1762609162 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1280a2e50 0x7ff1280a5310 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff138079840 tx=0x7ff124005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:17.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.477+0000 7ff135ffb700 1 -- 172.21.15.28:0/1762609162 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff12c01f020 con 0x7ff1381022e0 2024-04-25T01:32:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.679+0000 7ff13eda3700 1 -- 172.21.15.28:0/1762609162 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff138101300 con 0x7ff1280a2e50 2024-04-25T01:32:17.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.689+0000 7ff135ffb700 1 -- 172.21.15.28:0/1762609162 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340420 (secure 0 0 0) 0x7ff138101300 con 0x7ff1280a2e50 2024-04-25T01:32:17.671 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:32:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.692+0000 7ff13eda3700 1 -- 172.21.15.28:0/1762609162 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1280a2e50 msgr2=0x7ff1280a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.692+0000 7ff13eda3700 1 --2- 172.21.15.28:0/1762609162 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1280a2e50 0x7ff1280a5310 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff138079840 tx=0x7ff124005fd0 comp rx=0 tx=0).stop 2024-04-25T01:32:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.692+0000 7ff13eda3700 1 -- 172.21.15.28:0/1762609162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1381022e0 msgr2=0x7ff13819e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.692+0000 7ff13eda3700 1 --2- 172.21.15.28:0/1762609162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1381022e0 0x7ff13819e630 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00b530 tx=0x7ff12c00b8f0 comp rx=0 tx=0).stop 2024-04-25T01:32:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.692+0000 7ff13eda3700 1 -- 172.21.15.28:0/1762609162 shutdown_connections 2024-04-25T01:32:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.692+0000 7ff13eda3700 1 --2- 172.21.15.28:0/1762609162 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1280a2e50 0x7ff1280a5310 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.692+0000 7ff13eda3700 1 --2- 172.21.15.28:0/1762609162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1381082f0 0x7ff1381987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.692+0000 7ff13eda3700 1 --2- 172.21.15.28:0/1762609162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1381022e0 0x7ff13819e630 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.692+0000 7ff13eda3700 1 --2- 172.21.15.28:0/1762609162 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff138102ca0 0x7ff13819eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.692+0000 7ff13eda3700 1 -- 172.21.15.28:0/1762609162 >> 172.21.15.28:0/1762609162 conn(0x7ff138079390 msgr2=0x7ff138100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.692+0000 7ff13eda3700 1 -- 172.21.15.28:0/1762609162 shutdown_connections 2024-04-25T01:32:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:17.692+0000 7ff13eda3700 1 -- 172.21.15.28:0/1762609162 wait complete. 2024-04-25T01:32:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T01:32:18.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:18 smithi028 ceph-mon[29390]: pgmap v351: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:18.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:18 smithi130 ceph-mon[32702]: pgmap v351: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:18.467 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-04-25T01:32:16.355426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660139,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":902,"num_read_kb":579,"num_write":424,"num_write_kb":3081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1180804,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":350,"ondisk_log_size":350,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380324,"kb_used_data":14480,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124561500,"statfs":{"total":1151940427776,"available":1151550976000,"internally_reserved":0,"allocated":14827520,"data_stored":8063519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":374453562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":110,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003482"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.870244+0000","last_change":"2024-04-25T01:30:34.358339+0000","last_active":"2024-04-25T01:31:21.870244+0000","last_peered":"2024-04-25T01:31:21.870244+0000","last_clean":"2024-04-25T01:31:21.870244+0000","last_became_active":"2024-04-25T01:30:34.357884+0000","last_became_peered":"2024-04-25T01:30:34.357884+0000","last_unstale":"2024-04-25T01:31:21.870244+0000","last_undegraded":"2024-04-25T01:31:21.870244+0000","last_fullsized":"2024-04-25T01:31:21.870244+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:56.409080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869592+0000","last_change":"2024-04-25T01:30:34.359933+0000","last_active":"2024-04-25T01:31:21.869592+0000","last_peered":"2024-04-25T01:31:21.869592+0000","last_clean":"2024-04-25T01:31:21.869592+0000","last_became_active":"2024-04-25T01:30:34.359576+0000","last_became_peered":"2024-04-25T01:30:34.359576+0000","last_unstale":"2024-04-25T01:31:21.869592+0000","last_undegraded":"2024-04-25T01:31:21.869592+0000","last_fullsized":"2024-04-25T01:31:21.869592+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:48:47.764121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869363+0000","last_change":"2024-04-25T01:30:34.357772+0000","last_active":"2024-04-25T01:31:21.869363+0000","last_peered":"2024-04-25T01:31:21.869363+0000","last_clean":"2024-04-25T01:31:21.869363+0000","last_became_active":"2024-04-25T01:30:34.357434+0000","last_became_peered":"2024-04-25T01:30:34.357434+0000","last_unstale":"2024-04-25T01:31:21.869363+0000","last_undegraded":"2024-04-25T01:31:21.869363+0000","last_fullsized":"2024-04-25T01:31:21.869363+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:19.926180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867798+0000","last_change":"2024-04-25T01:30:34.351250+0000","last_active":"2024-04-25T01:31:21.867798+0000","last_peered":"2024-04-25T01:31:21.867798+0000","last_clean":"2024-04-25T01:31:21.867798+0000","last_became_active":"2024-04-25T01:30:34.349976+0000","last_became_peered":"2024-04-25T01:30:34.349976+0000","last_unstale":"2024-04-25T01:31:21.867798+0000","last_undegraded":"2024-04-25T01:31:21.867798+0000","last_fullsized":"2024-04-25T01:31:21.867798+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:19:51.605307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.874477+0000","last_change":"2024-04-25T01:30:34.367882+0000","last_active":"2024-04-25T01:31:21.874477+0000","last_peered":"2024-04-25T01:31:21.874477+0000","last_clean":"2024-04-25T01:31:21.874477+0000","last_became_active":"2024-04-25T01:30:34.366797+0000","last_became_peered":"2024-04-25T01:30:34.366797+0000","last_unstale":"2024-04-25T01:31:21.874477+0000","last_undegraded":"2024-04-25T01:31:21.874477+0000","last_fullsized":"2024-04-25T01:31:21.874477+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:35:23.439061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872254+0000","last_change":"2024-04-25T01:30:34.352506+0000","last_active":"2024-04-25T01:31:21.872254+0000","last_peered":"2024-04-25T01:31:21.872254+0000","last_clean":"2024-04-25T01:31:21.872254+0000","last_became_active":"2024-04-25T01:30:34.352100+0000","last_became_peered":"2024-04-25T01:30:34.352100+0000","last_unstale":"2024-04-25T01:31:21.872254+0000","last_undegraded":"2024-04-25T01:31:21.872254+0000","last_fullsized":"2024-04-25T01:31:21.872254+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:38.113959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.048442+0000","last_change":"2024-04-25T01:30:34.355352+0000","last_active":"2024-04-25T01:31:22.048442+0000","last_peered":"2024-04-25T01:31:22.048442+0000","last_clean":"2024-04-25T01:31:22.048442+0000","last_became_active":"2024-04-25T01:30:34.355008+0000","last_became_peered":"2024-04-25T01:30:34.355008+0000","last_unstale":"2024-04-25T01:31:22.048442+0000","last_undegraded":"2024-04-25T01:31:22.048442+0000","last_fullsized":"2024-04-25T01:31:22.048442+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:23.582742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872403+0000","last_change":"2024-04-25T01:30:34.360644+0000","last_active":"2024-04-25T01:31:21.872403+0000","last_peered":"2024-04-25T01:31:21.872403+0000","last_clean":"2024-04-25T01:31:21.872403+0000","last_became_active":"2024-04-25T01:30:34.357841+0000","last_became_peered":"2024-04-25T01:30:34.357841+0000","last_unstale":"2024-04-25T01:31:21.872403+0000","last_undegraded":"2024-04-25T01:31:21.872403+0000","last_fullsized":"2024-04-25T01:31:21.872403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:26:21.606219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049329+0000","last_change":"2024-04-25T01:30:34.358417+0000","last_active":"2024-04-25T01:31:22.049329+0000","last_peered":"2024-04-25T01:31:22.049329+0000","last_clean":"2024-04-25T01:31:22.049329+0000","last_became_active":"2024-04-25T01:30:34.358039+0000","last_became_peered":"2024-04-25T01:30:34.358039+0000","last_unstale":"2024-04-25T01:31:22.049329+0000","last_undegraded":"2024-04-25T01:31:22.049329+0000","last_fullsized":"2024-04-25T01:31:22.049329+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:42.600693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872775+0000","last_change":"2024-04-25T01:30:34.357803+0000","last_active":"2024-04-25T01:31:21.872775+0000","last_peered":"2024-04-25T01:31:21.872775+0000","last_clean":"2024-04-25T01:31:21.872775+0000","last_became_active":"2024-04-25T01:30:34.357391+0000","last_became_peered":"2024-04-25T01:30:34.357391+0000","last_unstale":"2024-04-25T01:31:21.872775+0000","last_undegraded":"2024-04-25T01:31:21.872775+0000","last_fullsized":"2024-04-25T01:31:21.872775+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:57.857303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049654+0000","last_change":"2024-04-25T01:30:34.363340+0000","last_active":"2024-04-25T01:31:22.049654+0000","last_peered":"2024-04-25T01:31:22.049654+0000","last_clean":"2024-04-25T01:31:22.049654+0000","last_became_active":"2024-04-25T01:30:34.362980+0000","last_became_peered":"2024-04-25T01:30:34.362980+0000","last_unstale":"2024-04-25T01:31:22.049654+0000","last_undegraded":"2024-04-25T01:31:22.049654+0000","last_fullsized":"2024-04-25T01:31:22.049654+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:46.291459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867712+0000","last_change":"2024-04-25T01:30:34.358898+0000","last_active":"2024-04-25T01:31:21.867712+0000","last_peered":"2024-04-25T01:31:21.867712+0000","last_clean":"2024-04-25T01:31:21.867712+0000","last_became_active":"2024-04-25T01:30:34.358495+0000","last_became_peered":"2024-04-25T01:30:34.358495+0000","last_unstale":"2024-04-25T01:31:21.867712+0000","last_undegraded":"2024-04-25T01:31:21.867712+0000","last_fullsized":"2024-04-25T01:31:21.867712+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:34:21.860010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866836+0000","last_change":"2024-04-25T01:30:34.363708+0000","last_active":"2024-04-25T01:31:21.866836+0000","last_peered":"2024-04-25T01:31:21.866836+0000","last_clean":"2024-04-25T01:31:21.866836+0000","last_became_active":"2024-04-25T01:30:34.363207+0000","last_became_peered":"2024-04-25T01:30:34.363207+0000","last_unstale":"2024-04-25T01:31:21.866836+0000","last_undegraded":"2024-04-25T01:31:21.866836+0000","last_fullsized":"2024-04-25T01:31:21.866836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:53.442987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869362+0000","last_change":"2024-04-25T01:30:34.371031+0000","last_active":"2024-04-25T01:31:21.869362+0000","last_peered":"2024-04-25T01:31:21.869362+0000","last_clean":"2024-04-25T01:31:21.869362+0000","last_became_active":"2024-04-25T01:30:34.370687+0000","last_became_peered":"2024-04-25T01:30:34.370687+0000","last_unstale":"2024-04-25T01:31:21.869362+0000","last_undegraded":"2024-04-25T01:31:21.869362+0000","last_fullsized":"2024-04-25T01:31:21.869362+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:03:20.700960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869153+0000","last_change":"2024-04-25T01:30:34.366212+0000","last_active":"2024-04-25T01:31:21.869153+0000","last_peered":"2024-04-25T01:31:21.869153+0000","last_clean":"2024-04-25T01:31:21.869153+0000","last_became_active":"2024-04-25T01:30:34.365849+0000","last_became_peered":"2024-04-25T01:30:34.365849+0000","last_unstale":"2024-04-25T01:31:21.869153+0000","last_undegraded":"2024-04-25T01:31:21.869153+0000","last_fullsized":"2024-04-25T01:31:21.869153+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:24:02.397615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049096+0000","last_change":"2024-04-25T01:30:34.367380+0000","last_active":"2024-04-25T01:31:22.049096+0000","last_peered":"2024-04-25T01:31:22.049096+0000","last_clean":"2024-04-25T01:31:22.049096+0000","last_became_active":"2024-04-25T01:30:34.366872+0000","last_became_peered":"2024-04-25T01:30:34.366872+0000","last_unstale":"2024-04-25T01:31:22.049096+0000","last_undegraded":"2024-04-25T01:31:22.049096+0000","last_fullsized":"2024-04-25T01:31:22.049096+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:18.767108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.870145+0000","last_change":"2024-04-25T01:30:34.359752+0000","last_active":"2024-04-25T01:31:21.870145+0000","last_peered":"2024-04-25T01:31:21.870145+0000","last_clean":"2024-04-25T01:31:21.870145+0000","last_became_active":"2024-04-25T01:30:34.358403+0000","last_became_peered":"2024-04-25T01:30:34.358403+0000","last_unstale":"2024-04-25T01:31:21.870145+0000","last_undegraded":"2024-04-25T01:31:21.870145+0000","last_fullsized":"2024-04-25T01:31:21.870145+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:14.833126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.871052+0000","last_change":"2024-04-25T01:30:34.361373+0000","last_active":"2024-04-25T01:31:21.871052+0000","last_peered":"2024-04-25T01:31:21.871052+0000","last_clean":"2024-04-25T01:31:21.871052+0000","last_became_active":"2024-04-25T01:30:34.361046+0000","last_became_peered":"2024-04-25T01:30:34.361046+0000","last_unstale":"2024-04-25T01:31:21.871052+0000","last_undegraded":"2024-04-25T01:31:21.871052+0000","last_fullsized":"2024-04-25T01:31:21.871052+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:21.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873421+0000","last_change":"2024-04-25T01:30:34.367975+0000","last_active":"2024-04-25T01:31:21.873421+0000","last_peered":"2024-04-25T01:31:21.873421+0000","last_clean":"2024-04-25T01:31:21.873421+0000","last_became_active":"2024-04-25T01:30:34.366383+0000","last_became_peered":"2024-04-25T01:30:34.366383+0000","last_unstale":"2024-04-25T01:31:21.873421+0000","last_undegraded":"2024-04-25T01:31:21.873421+0000","last_fullsized":"2024-04-25T01:31:21.873421+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:53:40.279067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873118+0000","last_change":"2024-04-25T01:30:38.382168+0000","last_active":"2024-04-25T01:31:21.873118+0000","last_peered":"2024-04-25T01:31:21.873118+0000","last_clean":"2024-04-25T01:31:21.873118+0000","last_became_active":"2024-04-25T01:30:38.381740+0000","last_became_peered":"2024-04-25T01:30:38.381740+0000","last_unstale":"2024-04-25T01:31:21.873118+0000","last_undegraded":"2024-04-25T01:31:21.873118+0000","last_fullsized":"2024-04-25T01:31:21.873118+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:27.425991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'13","reported_seq":113,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:32:11.905400+0000","last_change":"2024-04-25T01:30:33.345547+0000","last_active":"2024-04-25T01:32:11.905400+0000","last_peered":"2024-04-25T01:32:11.905400+0000","last_clean":"2024-04-25T01:32:11.905400+0000","last_became_active":"2024-04-25T01:30:33.345229+0000","last_became_peered":"2024-04-25T01:30:33.345229+0000","last_unstale":"2024-04-25T01:32:11.905400+0000","last_undegraded":"2024-04-25T01:32:11.905400+0000","last_fullsized":"2024-04-25T01:32:11.905400+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:32:46.637720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873194+0000","last_change":"2024-04-25T01:30:34.354703+0000","last_active":"2024-04-25T01:31:21.873194+0000","last_peered":"2024-04-25T01:31:21.873194+0000","last_clean":"2024-04-25T01:31:21.873194+0000","last_became_active":"2024-04-25T01:30:34.354362+0000","last_became_peered":"2024-04-25T01:30:34.354362+0000","last_unstale":"2024-04-25T01:31:21.873194+0000","last_undegraded":"2024-04-25T01:31:21.873194+0000","last_fullsized":"2024-04-25T01:31:21.873194+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:51.358599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867199+0000","last_change":"2024-04-25T01:30:38.380912+0000","last_active":"2024-04-25T01:31:21.867199+0000","last_peered":"2024-04-25T01:31:21.867199+0000","last_clean":"2024-04-25T01:31:21.867199+0000","last_became_active":"2024-04-25T01:30:38.380439+0000","last_became_peered":"2024-04-25T01:30:38.380439+0000","last_unstale":"2024-04-25T01:31:21.867199+0000","last_undegraded":"2024-04-25T01:31:21.867199+0000","last_fullsized":"2024-04-25T01:31:21.867199+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:22:00.527268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873300+0000","last_change":"2024-04-25T01:30:33.345620+0000","last_active":"2024-04-25T01:31:21.873300+0000","last_peered":"2024-04-25T01:31:21.873300+0000","last_clean":"2024-04-25T01:31:21.873300+0000","last_became_active":"2024-04-25T01:30:33.345315+0000","last_became_peered":"2024-04-25T01:30:33.345315+0000","last_unstale":"2024-04-25T01:31:21.873300+0000","last_undegraded":"2024-04-25T01:31:21.873300+0000","last_fullsized":"2024-04-25T01:31:21.873300+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:48:00.083728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869679+0000","last_change":"2024-04-25T01:30:34.356589+0000","last_active":"2024-04-25T01:31:21.869679+0000","last_peered":"2024-04-25T01:31:21.869679+0000","last_clean":"2024-04-25T01:31:21.869679+0000","last_became_active":"2024-04-25T01:30:34.356114+0000","last_became_peered":"2024-04-25T01:30:34.356114+0000","last_unstale":"2024-04-25T01:31:21.869679+0000","last_undegraded":"2024-04-25T01:31:21.869679+0000","last_fullsized":"2024-04-25T01:31:21.869679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:39.039517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.876578+0000","last_change":"2024-04-25T01:31:21.876578+0000","last_active":"2024-04-25T01:31:21.876578+0000","last_peered":"2024-04-25T01:31:21.876578+0000","last_clean":"2024-04-25T01:31:21.876578+0000","last_became_active":"2024-04-25T01:31:21.876126+0000","last_became_peered":"2024-04-25T01:31:21.876126+0000","last_unstale":"2024-04-25T01:31:21.876578+0000","last_undegraded":"2024-04-25T01:31:21.876578+0000","last_fullsized":"2024-04-25T01:31:21.876578+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:55.909353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'12","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.874895+0000","last_change":"2024-04-25T01:30:33.345575+0000","last_active":"2024-04-25T01:31:21.874895+0000","last_peered":"2024-04-25T01:31:21.874895+0000","last_clean":"2024-04-25T01:31:21.874895+0000","last_became_active":"2024-04-25T01:30:33.345231+0000","last_became_peered":"2024-04-25T01:30:33.345231+0000","last_unstale":"2024-04-25T01:31:21.874895+0000","last_undegraded":"2024-04-25T01:31:21.874895+0000","last_fullsized":"2024-04-25T01:31:21.874895+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:04:21.090788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049253+0000","last_change":"2024-04-25T01:30:34.371317+0000","last_active":"2024-04-25T01:31:22.049253+0000","last_peered":"2024-04-25T01:31:22.049253+0000","last_clean":"2024-04-25T01:31:22.049253+0000","last_became_active":"2024-04-25T01:30:34.370926+0000","last_became_peered":"2024-04-25T01:30:34.370926+0000","last_unstale":"2024-04-25T01:31:22.049253+0000","last_undegraded":"2024-04-25T01:31:22.049253+0000","last_fullsized":"2024-04-25T01:31:22.049253+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:58:39.898471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867100+0000","last_change":"2024-04-25T01:30:38.380982+0000","last_active":"2024-04-25T01:31:21.867100+0000","last_peered":"2024-04-25T01:31:21.867100+0000","last_clean":"2024-04-25T01:31:21.867100+0000","last_became_active":"2024-04-25T01:30:38.380510+0000","last_became_peered":"2024-04-25T01:30:38.380510+0000","last_unstale":"2024-04-25T01:31:21.867100+0000","last_undegraded":"2024-04-25T01:31:21.867100+0000","last_fullsized":"2024-04-25T01:31:21.867100+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:00.510697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.874833+0000","last_change":"2024-04-25T01:30:33.351561+0000","last_active":"2024-04-25T01:31:21.874833+0000","last_peered":"2024-04-25T01:31:21.874833+0000","last_clean":"2024-04-25T01:31:21.874833+0000","last_became_active":"2024-04-25T01:30:33.351230+0000","last_became_peered":"2024-04-25T01:30:33.351230+0000","last_unstale":"2024-04-25T01:31:21.874833+0000","last_undegraded":"2024-04-25T01:31:21.874833+0000","last_fullsized":"2024-04-25T01:31:21.874833+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:56.594457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867798+0000","last_change":"2024-04-25T01:30:34.358821+0000","last_active":"2024-04-25T01:31:21.867798+0000","last_peered":"2024-04-25T01:31:21.867798+0000","last_clean":"2024-04-25T01:31:21.867798+0000","last_became_active":"2024-04-25T01:30:34.358303+0000","last_became_peered":"2024-04-25T01:30:34.358303+0000","last_unstale":"2024-04-25T01:31:21.867798+0000","last_undegraded":"2024-04-25T01:31:21.867798+0000","last_fullsized":"2024-04-25T01:31:21.867798+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:26:23.352101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873266+0000","last_change":"2024-04-25T01:30:38.372281+0000","last_active":"2024-04-25T01:31:21.873266+0000","last_peered":"2024-04-25T01:31:21.873266+0000","last_clean":"2024-04-25T01:31:21.873266+0000","last_became_active":"2024-04-25T01:30:38.371876+0000","last_became_peered":"2024-04-25T01:30:38.371876+0000","last_unstale":"2024-04-25T01:31:21.873266+0000","last_undegraded":"2024-04-25T01:31:21.873266+0000","last_fullsized":"2024-04-25T01:31:21.873266+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:54.254011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'6","reported_seq":107,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:32:11.913467+0000","last_change":"2024-04-25T01:30:33.347262+0000","last_active":"2024-04-25T01:32:11.913467+0000","last_peered":"2024-04-25T01:32:11.913467+0000","last_clean":"2024-04-25T01:32:11.913467+0000","last_became_active":"2024-04-25T01:30:33.346900+0000","last_became_peered":"2024-04-25T01:30:33.346900+0000","last_unstale":"2024-04-25T01:32:11.913467+0000","last_undegraded":"2024-04-25T01:32:11.913467+0000","last_fullsized":"2024-04-25T01:32:11.913467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:29:30.960456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.874102+0000","last_change":"2024-04-25T01:30:34.364870+0000","last_active":"2024-04-25T01:31:21.874102+0000","last_peered":"2024-04-25T01:31:21.874102+0000","last_clean":"2024-04-25T01:31:21.874102+0000","last_became_active":"2024-04-25T01:30:34.364459+0000","last_became_peered":"2024-04-25T01:30:34.364459+0000","last_unstale":"2024-04-25T01:31:21.874102+0000","last_undegraded":"2024-04-25T01:31:21.874102+0000","last_fullsized":"2024-04-25T01:31:21.874102+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:42:45.311668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872894+0000","last_change":"2024-04-25T01:30:38.382175+0000","last_active":"2024-04-25T01:31:21.872894+0000","last_peered":"2024-04-25T01:31:21.872894+0000","last_clean":"2024-04-25T01:31:21.872894+0000","last_became_active":"2024-04-25T01:30:38.381771+0000","last_became_peered":"2024-04-25T01:30:38.381771+0000","last_unstale":"2024-04-25T01:31:21.872894+0000","last_undegraded":"2024-04-25T01:31:21.872894+0000","last_fullsized":"2024-04-25T01:31:21.872894+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:25:31.041566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872207+0000","last_change":"2024-04-25T01:30:33.343605+0000","last_active":"2024-04-25T01:31:21.872207+0000","last_peered":"2024-04-25T01:31:21.872207+0000","last_clean":"2024-04-25T01:31:21.872207+0000","last_became_active":"2024-04-25T01:30:33.343254+0000","last_became_peered":"2024-04-25T01:30:33.343254+0000","last_unstale":"2024-04-25T01:31:21.872207+0000","last_undegraded":"2024-04-25T01:31:21.872207+0000","last_fullsized":"2024-04-25T01:31:21.872207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:18:16.156285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.874640+0000","last_change":"2024-04-25T01:31:21.874202+0000","last_active":"2024-04-25T01:31:21.874640+0000","last_peered":"2024-04-25T01:31:21.874640+0000","last_clean":"2024-04-25T01:31:21.874640+0000","last_became_active":"2024-04-25T01:31:21.873839+0000","last_became_peered":"2024-04-25T01:31:21.873839+0000","last_unstale":"2024-04-25T01:31:21.874640+0000","last_undegraded":"2024-04-25T01:31:21.874640+0000","last_fullsized":"2024-04-25T01:31:21.874640+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:39.774281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.875259+0000","last_change":"2024-04-25T01:31:21.875259+0000","last_active":"2024-04-25T01:31:21.875259+0000","last_peered":"2024-04-25T01:31:21.875259+0000","last_clean":"2024-04-25T01:31:21.875259+0000","last_became_active":"2024-04-25T01:31:21.874800+0000","last_became_peered":"2024-04-25T01:31:21.874800+0000","last_unstale":"2024-04-25T01:31:21.875259+0000","last_undegraded":"2024-04-25T01:31:21.875259+0000","last_fullsized":"2024-04-25T01:31:21.875259+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:44.828620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,10,0],"acting":[11,4,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866568+0000","last_change":"2024-04-25T01:30:33.351621+0000","last_active":"2024-04-25T01:31:21.866568+0000","last_peered":"2024-04-25T01:31:21.866568+0000","last_clean":"2024-04-25T01:31:21.866568+0000","last_became_active":"2024-04-25T01:30:33.351289+0000","last_became_peered":"2024-04-25T01:30:33.351289+0000","last_unstale":"2024-04-25T01:31:21.866568+0000","last_undegraded":"2024-04-25T01:31:21.866568+0000","last_fullsized":"2024-04-25T01:31:21.866568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:46.044826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872529+0000","last_change":"2024-04-25T01:30:34.350685+0000","last_active":"2024-04-25T01:31:21.872529+0000","last_peered":"2024-04-25T01:31:21.872529+0000","last_clean":"2024-04-25T01:31:21.872529+0000","last_became_active":"2024-04-25T01:30:34.350308+0000","last_became_peered":"2024-04-25T01:30:34.350308+0000","last_unstale":"2024-04-25T01:31:21.872529+0000","last_undegraded":"2024-04-25T01:31:21.872529+0000","last_fullsized":"2024-04-25T01:31:21.872529+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:11.690381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869484+0000","last_change":"2024-04-25T01:30:38.377403+0000","last_active":"2024-04-25T01:31:21.869484+0000","last_peered":"2024-04-25T01:31:21.869484+0000","last_clean":"2024-04-25T01:31:21.869484+0000","last_became_active":"2024-04-25T01:30:38.376966+0000","last_became_peered":"2024-04-25T01:30:38.376966+0000","last_unstale":"2024-04-25T01:31:21.869484+0000","last_undegraded":"2024-04-25T01:31:21.869484+0000","last_fullsized":"2024-04-25T01:31:21.869484+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:31:10.229443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867282+0000","last_change":"2024-04-25T01:30:33.348455+0000","last_active":"2024-04-25T01:31:21.867282+0000","last_peered":"2024-04-25T01:31:21.867282+0000","last_clean":"2024-04-25T01:31:21.867282+0000","last_became_active":"2024-04-25T01:30:33.348142+0000","last_became_peered":"2024-04-25T01:30:33.348142+0000","last_unstale":"2024-04-25T01:31:21.867282+0000","last_undegraded":"2024-04-25T01:31:21.867282+0000","last_fullsized":"2024-04-25T01:31:21.867282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:43:07.695580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.870724+0000","last_change":"2024-04-25T01:30:34.359949+0000","last_active":"2024-04-25T01:31:21.870724+0000","last_peered":"2024-04-25T01:31:21.870724+0000","last_clean":"2024-04-25T01:31:21.870724+0000","last_became_active":"2024-04-25T01:30:34.359609+0000","last_became_peered":"2024-04-25T01:30:34.359609+0000","last_unstale":"2024-04-25T01:31:21.870724+0000","last_undegraded":"2024-04-25T01:31:21.870724+0000","last_fullsized":"2024-04-25T01:31:21.870724+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:49.668800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872956+0000","last_change":"2024-04-25T01:30:38.380260+0000","last_active":"2024-04-25T01:31:21.872956+0000","last_peered":"2024-04-25T01:31:21.872956+0000","last_clean":"2024-04-25T01:31:21.872956+0000","last_became_active":"2024-04-25T01:30:38.379624+0000","last_became_peered":"2024-04-25T01:30:38.379624+0000","last_unstale":"2024-04-25T01:31:21.872956+0000","last_undegraded":"2024-04-25T01:31:21.872956+0000","last_fullsized":"2024-04-25T01:31:21.872956+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:26:35.235335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":177,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:32:13.915484+0000","last_change":"2024-04-25T01:30:33.349490+0000","last_active":"2024-04-25T01:32:13.915484+0000","last_peered":"2024-04-25T01:32:13.915484+0000","last_clean":"2024-04-25T01:32:13.915484+0000","last_became_active":"2024-04-25T01:30:33.349136+0000","last_became_peered":"2024-04-25T01:30:33.349136+0000","last_unstale":"2024-04-25T01:32:13.915484+0000","last_undegraded":"2024-04-25T01:32:13.915484+0000","last_fullsized":"2024-04-25T01:32:13.915484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:16:23.552073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":146,"num_read_kb":146,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049084+0000","last_change":"2024-04-25T01:30:34.372701+0000","last_active":"2024-04-25T01:31:22.049084+0000","last_peered":"2024-04-25T01:31:22.049084+0000","last_clean":"2024-04-25T01:31:22.049084+0000","last_became_active":"2024-04-25T01:30:34.372321+0000","last_became_peered":"2024-04-25T01:30:34.372321+0000","last_unstale":"2024-04-25T01:31:22.049084+0000","last_undegraded":"2024-04-25T01:31:22.049084+0000","last_fullsized":"2024-04-25T01:31:22.049084+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:02.025015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869936+0000","last_change":"2024-04-25T01:30:38.378036+0000","last_active":"2024-04-25T01:31:21.869936+0000","last_peered":"2024-04-25T01:31:21.869936+0000","last_clean":"2024-04-25T01:31:21.869936+0000","last_became_active":"2024-04-25T01:30:38.377602+0000","last_became_peered":"2024-04-25T01:30:38.377602+0000","last_unstale":"2024-04-25T01:31:21.869936+0000","last_undegraded":"2024-04-25T01:31:21.869936+0000","last_fullsized":"2024-04-25T01:31:21.869936+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:45.210518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.048871+0000","last_change":"2024-04-25T01:30:33.352928+0000","last_active":"2024-04-25T01:31:22.048871+0000","last_peered":"2024-04-25T01:31:22.048871+0000","last_clean":"2024-04-25T01:31:22.048871+0000","last_became_active":"2024-04-25T01:30:33.352567+0000","last_became_peered":"2024-04-25T01:30:33.352567+0000","last_unstale":"2024-04-25T01:31:22.048871+0000","last_undegraded":"2024-04-25T01:31:22.048871+0000","last_fullsized":"2024-04-25T01:31:22.048871+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:47:21.941179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869787+0000","last_change":"2024-04-25T01:30:34.366947+0000","last_active":"2024-04-25T01:31:21.869787+0000","last_peered":"2024-04-25T01:31:21.869787+0000","last_clean":"2024-04-25T01:31:21.869787+0000","last_became_active":"2024-04-25T01:30:34.366337+0000","last_became_peered":"2024-04-25T01:30:34.366337+0000","last_unstale":"2024-04-25T01:31:21.869787+0000","last_undegraded":"2024-04-25T01:31:21.869787+0000","last_fullsized":"2024-04-25T01:31:21.869787+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:48:43.528380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.050799+0000","last_change":"2024-04-25T01:31:22.050799+0000","last_active":"2024-04-25T01:31:22.050799+0000","last_peered":"2024-04-25T01:31:22.050799+0000","last_clean":"2024-04-25T01:31:22.050799+0000","last_became_active":"2024-04-25T01:31:22.050130+0000","last_became_peered":"2024-04-25T01:31:22.050130+0000","last_unstale":"2024-04-25T01:31:22.050799+0000","last_undegraded":"2024-04-25T01:31:22.050799+0000","last_fullsized":"2024-04-25T01:31:22.050799+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:40:43.549713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867754+0000","last_change":"2024-04-25T01:30:33.346003+0000","last_active":"2024-04-25T01:31:21.867754+0000","last_peered":"2024-04-25T01:31:21.867754+0000","last_clean":"2024-04-25T01:31:21.867754+0000","last_became_active":"2024-04-25T01:30:33.345254+0000","last_became_peered":"2024-04-25T01:30:33.345254+0000","last_unstale":"2024-04-25T01:31:21.867754+0000","last_undegraded":"2024-04-25T01:31:21.867754+0000","last_fullsized":"2024-04-25T01:31:21.867754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:18:37.957047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049419+0000","last_change":"2024-04-25T01:30:34.366108+0000","last_active":"2024-04-25T01:31:22.049419+0000","last_peered":"2024-04-25T01:31:22.049419+0000","last_clean":"2024-04-25T01:31:22.049419+0000","last_became_active":"2024-04-25T01:30:34.365702+0000","last_became_peered":"2024-04-25T01:30:34.365702+0000","last_unstale":"2024-04-25T01:31:22.049419+0000","last_undegraded":"2024-04-25T01:31:22.049419+0000","last_fullsized":"2024-04-25T01:31:22.049419+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:24:06.802424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.871178+0000","last_change":"2024-04-25T01:30:38.376342+0000","last_active":"2024-04-25T01:31:21.871178+0000","last_peered":"2024-04-25T01:31:21.871178+0000","last_clean":"2024-04-25T01:31:21.871178+0000","last_became_active":"2024-04-25T01:30:38.375905+0000","last_became_peered":"2024-04-25T01:30:38.375905+0000","last_unstale":"2024-04-25T01:31:21.871178+0000","last_undegraded":"2024-04-25T01:31:21.871178+0000","last_fullsized":"2024-04-25T01:31:21.871178+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:31.615356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'36","reported_seq":217,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:32:13.915521+0000","last_change":"2024-04-25T01:30:33.350770+0000","last_active":"2024-04-25T01:32:13.915521+0000","last_peered":"2024-04-25T01:32:13.915521+0000","last_clean":"2024-04-25T01:32:13.915521+0000","last_became_active":"2024-04-25T01:30:33.350414+0000","last_became_peered":"2024-04-25T01:30:33.350414+0000","last_unstale":"2024-04-25T01:32:13.915521+0000","last_undegraded":"2024-04-25T01:32:13.915521+0000","last_fullsized":"2024-04-25T01:32:13.915521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:01.771182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62333,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":154,"num_read_kb":139,"num_write":38,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.874658+0000","last_change":"2024-04-25T01:30:34.362532+0000","last_active":"2024-04-25T01:31:21.874658+0000","last_peered":"2024-04-25T01:31:21.874658+0000","last_clean":"2024-04-25T01:31:21.874658+0000","last_became_active":"2024-04-25T01:30:34.362137+0000","last_became_peered":"2024-04-25T01:30:34.362137+0000","last_unstale":"2024-04-25T01:31:21.874658+0000","last_undegraded":"2024-04-25T01:31:21.874658+0000","last_fullsized":"2024-04-25T01:31:21.874658+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:28.105250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.868022+0000","last_change":"2024-04-25T01:30:38.374591+0000","last_active":"2024-04-25T01:31:21.868022+0000","last_peered":"2024-04-25T01:31:21.868022+0000","last_clean":"2024-04-25T01:31:21.868022+0000","last_became_active":"2024-04-25T01:30:38.373237+0000","last_became_peered":"2024-04-25T01:30:38.373237+0000","last_unstale":"2024-04-25T01:31:21.868022+0000","last_undegraded":"2024-04-25T01:31:21.868022+0000","last_fullsized":"2024-04-25T01:31:21.868022+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:21:25.937405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'9","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.885341+0000","last_change":"2024-04-25T01:31:21.885341+0000","last_active":"2024-04-25T01:31:21.885341+0000","last_peered":"2024-04-25T01:31:21.885341+0000","last_clean":"2024-04-25T01:31:21.885341+0000","last_became_active":"2024-04-25T01:31:21.874312+0000","last_became_peered":"2024-04-25T01:31:21.874312+0000","last_unstale":"2024-04-25T01:31:21.885341+0000","last_undegraded":"2024-04-25T01:31:21.885341+0000","last_fullsized":"2024-04-25T01:31:21.885341+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:23:55.388089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866527+0000","last_change":"2024-04-25T01:30:34.354444+0000","last_active":"2024-04-25T01:31:21.866527+0000","last_peered":"2024-04-25T01:31:21.866527+0000","last_clean":"2024-04-25T01:31:21.866527+0000","last_became_active":"2024-04-25T01:30:34.354070+0000","last_became_peered":"2024-04-25T01:30:34.354070+0000","last_unstale":"2024-04-25T01:31:21.866527+0000","last_undegraded":"2024-04-25T01:31:21.866527+0000","last_fullsized":"2024-04-25T01:31:21.866527+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:39.893995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869979+0000","last_change":"2024-04-25T01:30:38.373570+0000","last_active":"2024-04-25T01:31:21.869979+0000","last_peered":"2024-04-25T01:31:21.869979+0000","last_clean":"2024-04-25T01:31:21.869979+0000","last_became_active":"2024-04-25T01:30:38.373144+0000","last_became_peered":"2024-04-25T01:30:38.373144+0000","last_unstale":"2024-04-25T01:31:21.869979+0000","last_undegraded":"2024-04-25T01:31:21.869979+0000","last_fullsized":"2024-04-25T01:31:21.869979+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:53:32.933998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867431+0000","last_change":"2024-04-25T01:30:33.346145+0000","last_active":"2024-04-25T01:31:21.867431+0000","last_peered":"2024-04-25T01:31:21.867431+0000","last_clean":"2024-04-25T01:31:21.867431+0000","last_became_active":"2024-04-25T01:30:33.344294+0000","last_became_peered":"2024-04-25T01:30:33.344294+0000","last_unstale":"2024-04-25T01:31:21.867431+0000","last_undegraded":"2024-04-25T01:31:21.867431+0000","last_fullsized":"2024-04-25T01:31:21.867431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:37:17.147148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.048716+0000","last_change":"2024-04-25T01:30:34.360769+0000","last_active":"2024-04-25T01:31:22.048716+0000","last_peered":"2024-04-25T01:31:22.048716+0000","last_clean":"2024-04-25T01:31:22.048716+0000","last_became_active":"2024-04-25T01:30:34.360076+0000","last_became_peered":"2024-04-25T01:30:34.360076+0000","last_unstale":"2024-04-25T01:31:22.048716+0000","last_undegraded":"2024-04-25T01:31:22.048716+0000","last_fullsized":"2024-04-25T01:31:22.048716+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:56:21.515511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.870901+0000","last_change":"2024-04-25T01:30:38.379198+0000","last_active":"2024-04-25T01:31:21.870901+0000","last_peered":"2024-04-25T01:31:21.870901+0000","last_clean":"2024-04-25T01:31:21.870901+0000","last_became_active":"2024-04-25T01:30:38.378798+0000","last_became_peered":"2024-04-25T01:30:38.378798+0000","last_unstale":"2024-04-25T01:31:21.870901+0000","last_undegraded":"2024-04-25T01:31:21.870901+0000","last_fullsized":"2024-04-25T01:31:21.870901+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:40.192239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":107,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:32:12.907572+0000","last_change":"2024-04-25T01:30:33.343232+0000","last_active":"2024-04-25T01:32:12.907572+0000","last_peered":"2024-04-25T01:32:12.907572+0000","last_clean":"2024-04-25T01:32:12.907572+0000","last_became_active":"2024-04-25T01:30:33.342876+0000","last_became_peered":"2024-04-25T01:30:33.342876+0000","last_unstale":"2024-04-25T01:32:12.907572+0000","last_undegraded":"2024-04-25T01:32:12.907572+0000","last_fullsized":"2024-04-25T01:32:12.907572+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:45.032564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.048465+0000","last_change":"2024-04-25T01:30:34.358210+0000","last_active":"2024-04-25T01:31:22.048465+0000","last_peered":"2024-04-25T01:31:22.048465+0000","last_clean":"2024-04-25T01:31:22.048465+0000","last_became_active":"2024-04-25T01:30:34.357862+0000","last_became_peered":"2024-04-25T01:30:34.357862+0000","last_unstale":"2024-04-25T01:31:22.048465+0000","last_undegraded":"2024-04-25T01:31:22.048465+0000","last_fullsized":"2024-04-25T01:31:22.048465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:43.751361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866946+0000","last_change":"2024-04-25T01:30:38.380413+0000","last_active":"2024-04-25T01:31:21.866946+0000","last_peered":"2024-04-25T01:31:21.866946+0000","last_clean":"2024-04-25T01:31:21.866946+0000","last_became_active":"2024-04-25T01:30:38.379612+0000","last_became_peered":"2024-04-25T01:30:38.379612+0000","last_unstale":"2024-04-25T01:31:21.866946+0000","last_undegraded":"2024-04-25T01:31:21.866946+0000","last_fullsized":"2024-04-25T01:31:21.866946+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:45.328642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'17","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.874013+0000","last_change":"2024-04-25T01:30:33.344938+0000","last_active":"2024-04-25T01:31:21.874013+0000","last_peered":"2024-04-25T01:31:21.874013+0000","last_clean":"2024-04-25T01:31:21.874013+0000","last_became_active":"2024-04-25T01:30:33.344582+0000","last_became_peered":"2024-04-25T01:30:33.344582+0000","last_unstale":"2024-04-25T01:31:21.874013+0000","last_undegraded":"2024-04-25T01:31:21.874013+0000","last_fullsized":"2024-04-25T01:31:21.874013+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:43:33.777319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867324+0000","last_change":"2024-04-25T01:30:34.354115+0000","last_active":"2024-04-25T01:31:21.867324+0000","last_peered":"2024-04-25T01:31:21.867324+0000","last_clean":"2024-04-25T01:31:21.867324+0000","last_became_active":"2024-04-25T01:30:34.353722+0000","last_became_peered":"2024-04-25T01:30:34.353722+0000","last_unstale":"2024-04-25T01:31:21.867324+0000","last_undegraded":"2024-04-25T01:31:21.867324+0000","last_fullsized":"2024-04-25T01:31:21.867324+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:22.003521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867270+0000","last_change":"2024-04-25T01:30:38.380914+0000","last_active":"2024-04-25T01:31:21.867270+0000","last_peered":"2024-04-25T01:31:21.867270+0000","last_clean":"2024-04-25T01:31:21.867270+0000","last_became_active":"2024-04-25T01:30:38.380477+0000","last_became_peered":"2024-04-25T01:30:38.380477+0000","last_unstale":"2024-04-25T01:31:21.867270+0000","last_undegraded":"2024-04-25T01:31:21.867270+0000","last_fullsized":"2024-04-25T01:31:21.867270+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:23:27.882266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.875670+0000","last_change":"2024-04-25T01:30:33.345772+0000","last_active":"2024-04-25T01:31:21.875670+0000","last_peered":"2024-04-25T01:31:21.875670+0000","last_clean":"2024-04-25T01:31:21.875670+0000","last_became_active":"2024-04-25T01:30:33.345455+0000","last_became_peered":"2024-04-25T01:30:33.345455+0000","last_unstale":"2024-04-25T01:31:21.875670+0000","last_undegraded":"2024-04-25T01:31:21.875670+0000","last_fullsized":"2024-04-25T01:31:21.875670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:14:57.465161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869815+0000","last_change":"2024-04-25T01:30:34.358665+0000","last_active":"2024-04-25T01:31:21.869815+0000","last_peered":"2024-04-25T01:31:21.869815+0000","last_clean":"2024-04-25T01:31:21.869815+0000","last_became_active":"2024-04-25T01:30:34.358284+0000","last_became_peered":"2024-04-25T01:30:34.358284+0000","last_unstale":"2024-04-25T01:31:21.869815+0000","last_undegraded":"2024-04-25T01:31:21.869815+0000","last_fullsized":"2024-04-25T01:31:21.869815+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:12:22.832265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873969+0000","last_change":"2024-04-25T01:30:38.450027+0000","last_active":"2024-04-25T01:31:21.873969+0000","last_peered":"2024-04-25T01:31:21.873969+0000","last_clean":"2024-04-25T01:31:21.873969+0000","last_became_active":"2024-04-25T01:30:38.449564+0000","last_became_peered":"2024-04-25T01:30:38.449564+0000","last_unstale":"2024-04-25T01:31:21.873969+0000","last_undegraded":"2024-04-25T01:31:21.873969+0000","last_fullsized":"2024-04-25T01:31:21.873969+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:34:26.542351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"91'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.870455+0000","last_change":"2024-04-25T01:30:33.349757+0000","last_active":"2024-04-25T01:31:21.870455+0000","last_peered":"2024-04-25T01:31:21.870455+0000","last_clean":"2024-04-25T01:31:21.870455+0000","last_became_active":"2024-04-25T01:30:33.349439+0000","last_became_peered":"2024-04-25T01:30:33.349439+0000","last_unstale":"2024-04-25T01:31:21.870455+0000","last_undegraded":"2024-04-25T01:31:21.870455+0000","last_fullsized":"2024-04-25T01:31:21.870455+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:35.514160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872423+0000","last_change":"2024-04-25T01:30:34.367310+0000","last_active":"2024-04-25T01:31:21.872423+0000","last_peered":"2024-04-25T01:31:21.872423+0000","last_clean":"2024-04-25T01:31:21.872423+0000","last_became_active":"2024-04-25T01:30:34.366950+0000","last_became_peered":"2024-04-25T01:30:34.366950+0000","last_unstale":"2024-04-25T01:31:21.872423+0000","last_undegraded":"2024-04-25T01:31:21.872423+0000","last_fullsized":"2024-04-25T01:31:21.872423+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:26:17.700512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866608+0000","last_change":"2024-04-25T01:30:38.372259+0000","last_active":"2024-04-25T01:31:21.866608+0000","last_peered":"2024-04-25T01:31:21.866608+0000","last_clean":"2024-04-25T01:31:21.866608+0000","last_became_active":"2024-04-25T01:30:38.371847+0000","last_became_peered":"2024-04-25T01:30:38.371847+0000","last_unstale":"2024-04-25T01:31:21.866608+0000","last_undegraded":"2024-04-25T01:31:21.866608+0000","last_fullsized":"2024-04-25T01:31:21.866608+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:55.047739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'5","reported_seq":108,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:32:14.910062+0000","last_change":"2024-04-25T01:30:33.352078+0000","last_active":"2024-04-25T01:32:14.910062+0000","last_peered":"2024-04-25T01:32:14.910062+0000","last_clean":"2024-04-25T01:32:14.910062+0000","last_became_active":"2024-04-25T01:30:33.351725+0000","last_became_peered":"2024-04-25T01:30:33.351725+0000","last_unstale":"2024-04-25T01:32:14.910062+0000","last_undegraded":"2024-04-25T01:32:14.910062+0000","last_fullsized":"2024-04-25T01:32:14.910062+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:55:55.236040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872450+0000","last_change":"2024-04-25T01:30:34.360324+0000","last_active":"2024-04-25T01:31:21.872450+0000","last_peered":"2024-04-25T01:31:21.872450+0000","last_clean":"2024-04-25T01:31:21.872450+0000","last_became_active":"2024-04-25T01:30:34.359114+0000","last_became_peered":"2024-04-25T01:30:34.359114+0000","last_unstale":"2024-04-25T01:31:21.872450+0000","last_undegraded":"2024-04-25T01:31:21.872450+0000","last_fullsized":"2024-04-25T01:31:21.872450+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:51.927640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866716+0000","last_change":"2024-04-25T01:30:38.450650+0000","last_active":"2024-04-25T01:31:21.866716+0000","last_peered":"2024-04-25T01:31:21.866716+0000","last_clean":"2024-04-25T01:31:21.866716+0000","last_became_active":"2024-04-25T01:30:38.450149+0000","last_became_peered":"2024-04-25T01:30:38.450149+0000","last_unstale":"2024-04-25T01:31:21.866716+0000","last_undegraded":"2024-04-25T01:31:21.866716+0000","last_fullsized":"2024-04-25T01:31:21.866716+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:34.630803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867436+0000","last_change":"2024-04-25T01:30:34.352961+0000","last_active":"2024-04-25T01:31:21.867436+0000","last_peered":"2024-04-25T01:31:21.867436+0000","last_clean":"2024-04-25T01:31:21.867436+0000","last_became_active":"2024-04-25T01:30:34.352657+0000","last_became_peered":"2024-04-25T01:30:34.352657+0000","last_unstale":"2024-04-25T01:31:21.867436+0000","last_undegraded":"2024-04-25T01:31:21.867436+0000","last_fullsized":"2024-04-25T01:31:21.867436+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:20.934228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":106,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:32:12.907019+0000","last_change":"2024-04-25T01:30:33.344562+0000","last_active":"2024-04-25T01:32:12.907019+0000","last_peered":"2024-04-25T01:32:12.907019+0000","last_clean":"2024-04-25T01:32:12.907019+0000","last_became_active":"2024-04-25T01:30:33.344212+0000","last_became_peered":"2024-04-25T01:30:33.344212+0000","last_unstale":"2024-04-25T01:32:12.907019+0000","last_undegraded":"2024-04-25T01:32:12.907019+0000","last_fullsized":"2024-04-25T01:32:12.907019+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:20:51.110618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049258+0000","last_change":"2024-04-25T01:30:38.384316+0000","last_active":"2024-04-25T01:31:22.049258+0000","last_peered":"2024-04-25T01:31:22.049258+0000","last_clean":"2024-04-25T01:31:22.049258+0000","last_became_active":"2024-04-25T01:30:38.383775+0000","last_became_peered":"2024-04-25T01:30:38.383775+0000","last_unstale":"2024-04-25T01:31:22.049258+0000","last_undegraded":"2024-04-25T01:31:22.049258+0000","last_fullsized":"2024-04-25T01:31:22.049258+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:54.165691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.050351+0000","last_change":"2024-04-25T01:31:22.050014+0000","last_active":"2024-04-25T01:31:22.050351+0000","last_peered":"2024-04-25T01:31:22.050351+0000","last_clean":"2024-04-25T01:31:22.050351+0000","last_became_active":"2024-04-25T01:31:22.049718+0000","last_became_peered":"2024-04-25T01:31:22.049718+0000","last_unstale":"2024-04-25T01:31:22.050351+0000","last_undegraded":"2024-04-25T01:31:22.050351+0000","last_fullsized":"2024-04-25T01:31:22.050351+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:40:22.233372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867550+0000","last_change":"2024-04-25T01:30:33.345283+0000","last_active":"2024-04-25T01:31:21.867550+0000","last_peered":"2024-04-25T01:31:21.867550+0000","last_clean":"2024-04-25T01:31:21.867550+0000","last_became_active":"2024-04-25T01:30:33.344903+0000","last_became_peered":"2024-04-25T01:30:33.344903+0000","last_unstale":"2024-04-25T01:31:21.867550+0000","last_undegraded":"2024-04-25T01:31:21.867550+0000","last_fullsized":"2024-04-25T01:31:21.867550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:57:01.982850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":364,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.048240+0000","last_change":"2024-04-25T01:29:17.689639+0000","last_active":"2024-04-25T01:31:22.048240+0000","last_peered":"2024-04-25T01:31:22.048240+0000","last_clean":"2024-04-25T01:31:22.048240+0000","last_became_active":"2024-04-25T01:29:17.689046+0000","last_became_peered":"2024-04-25T01:29:17.689046+0000","last_unstale":"2024-04-25T01:31:22.048240+0000","last_undegraded":"2024-04-25T01:31:22.048240+0000","last_fullsized":"2024-04-25T01:31:22.048240+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:48.767945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:48.767945+0000","last_clean_scrub_stamp":"2024-04-25T01:25:48.767945+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:04:06.092632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.868030+0000","last_change":"2024-04-25T01:30:38.450938+0000","last_active":"2024-04-25T01:31:21.868030+0000","last_peered":"2024-04-25T01:31:21.868030+0000","last_clean":"2024-04-25T01:31:21.868030+0000","last_became_active":"2024-04-25T01:30:38.450488+0000","last_became_peered":"2024-04-25T01:30:38.450488+0000","last_unstale":"2024-04-25T01:31:21.868030+0000","last_undegraded":"2024-04-25T01:31:21.868030+0000","last_fullsized":"2024-04-25T01:31:21.868030+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:44.778133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.875719+0000","last_change":"2024-04-25T01:31:21.875719+0000","last_active":"2024-04-25T01:31:21.875719+0000","last_peered":"2024-04-25T01:31:21.875719+0000","last_clean":"2024-04-25T01:31:21.875719+0000","last_became_active":"2024-04-25T01:31:21.875283+0000","last_became_peered":"2024-04-25T01:31:21.875283+0000","last_unstale":"2024-04-25T01:31:21.875719+0000","last_undegraded":"2024-04-25T01:31:21.875719+0000","last_fullsized":"2024-04-25T01:31:21.875719+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:12:39.097019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.048316+0000","last_change":"2024-04-25T01:30:33.348198+0000","last_active":"2024-04-25T01:31:22.048316+0000","last_peered":"2024-04-25T01:31:22.048316+0000","last_clean":"2024-04-25T01:31:22.048316+0000","last_became_active":"2024-04-25T01:30:33.347876+0000","last_became_peered":"2024-04-25T01:30:33.347876+0000","last_unstale":"2024-04-25T01:31:22.048316+0000","last_undegraded":"2024-04-25T01:31:22.048316+0000","last_fullsized":"2024-04-25T01:31:22.048316+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:48:33.116571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869567+0000","last_change":"2024-04-25T01:30:38.372331+0000","last_active":"2024-04-25T01:31:21.869567+0000","last_peered":"2024-04-25T01:31:21.869567+0000","last_clean":"2024-04-25T01:31:21.869567+0000","last_became_active":"2024-04-25T01:30:38.371923+0000","last_became_peered":"2024-04-25T01:30:38.371923+0000","last_unstale":"2024-04-25T01:31:21.869567+0000","last_undegraded":"2024-04-25T01:31:21.869567+0000","last_fullsized":"2024-04-25T01:31:21.869567+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:06.416967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.874700+0000","last_change":"2024-04-25T01:30:33.346606+0000","last_active":"2024-04-25T01:31:21.874700+0000","last_peered":"2024-04-25T01:31:21.874700+0000","last_clean":"2024-04-25T01:31:21.874700+0000","last_became_active":"2024-04-25T01:30:33.346297+0000","last_became_peered":"2024-04-25T01:30:33.346297+0000","last_unstale":"2024-04-25T01:31:21.874700+0000","last_undegraded":"2024-04-25T01:31:21.874700+0000","last_fullsized":"2024-04-25T01:31:21.874700+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:07:16.069285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866732+0000","last_change":"2024-04-25T01:30:34.349136+0000","last_active":"2024-04-25T01:31:21.866732+0000","last_peered":"2024-04-25T01:31:21.866732+0000","last_clean":"2024-04-25T01:31:21.866732+0000","last_became_active":"2024-04-25T01:30:34.348810+0000","last_became_peered":"2024-04-25T01:30:34.348810+0000","last_unstale":"2024-04-25T01:31:21.866732+0000","last_undegraded":"2024-04-25T01:31:21.866732+0000","last_fullsized":"2024-04-25T01:31:21.866732+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:39:34.593986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872418+0000","last_change":"2024-04-25T01:30:38.450428+0000","last_active":"2024-04-25T01:31:21.872418+0000","last_peered":"2024-04-25T01:31:21.872418+0000","last_clean":"2024-04-25T01:31:21.872418+0000","last_became_active":"2024-04-25T01:30:38.449971+0000","last_became_peered":"2024-04-25T01:30:38.449971+0000","last_unstale":"2024-04-25T01:31:21.872418+0000","last_undegraded":"2024-04-25T01:31:21.872418+0000","last_fullsized":"2024-04-25T01:31:21.872418+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:52:37.046462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'11","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.870939+0000","last_change":"2024-04-25T01:30:33.348547+0000","last_active":"2024-04-25T01:31:21.870939+0000","last_peered":"2024-04-25T01:31:21.870939+0000","last_clean":"2024-04-25T01:31:21.870939+0000","last_became_active":"2024-04-25T01:30:33.348181+0000","last_became_peered":"2024-04-25T01:30:33.348181+0000","last_unstale":"2024-04-25T01:31:21.870939+0000","last_undegraded":"2024-04-25T01:31:21.870939+0000","last_fullsized":"2024-04-25T01:31:21.870939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:11:58.696829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873436+0000","last_change":"2024-04-25T01:30:34.351987+0000","last_active":"2024-04-25T01:31:21.873436+0000","last_peered":"2024-04-25T01:31:21.873436+0000","last_clean":"2024-04-25T01:31:21.873436+0000","last_became_active":"2024-04-25T01:30:34.351571+0000","last_became_peered":"2024-04-25T01:30:34.351571+0000","last_unstale":"2024-04-25T01:31:21.873436+0000","last_undegraded":"2024-04-25T01:31:21.873436+0000","last_fullsized":"2024-04-25T01:31:21.873436+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:23.585865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872904+0000","last_change":"2024-04-25T01:30:38.372566+0000","last_active":"2024-04-25T01:31:21.872904+0000","last_peered":"2024-04-25T01:31:21.872904+0000","last_clean":"2024-04-25T01:31:21.872904+0000","last_became_active":"2024-04-25T01:30:38.372102+0000","last_became_peered":"2024-04-25T01:30:38.372102+0000","last_unstale":"2024-04-25T01:31:21.872904+0000","last_undegraded":"2024-04-25T01:31:21.872904+0000","last_fullsized":"2024-04-25T01:31:21.872904+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:22:16.791842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867899+0000","last_change":"2024-04-25T01:30:33.344065+0000","last_active":"2024-04-25T01:31:21.867899+0000","last_peered":"2024-04-25T01:31:21.867899+0000","last_clean":"2024-04-25T01:31:21.867899+0000","last_became_active":"2024-04-25T01:30:33.343700+0000","last_became_peered":"2024-04-25T01:30:33.343700+0000","last_unstale":"2024-04-25T01:31:21.867899+0000","last_undegraded":"2024-04-25T01:31:21.867899+0000","last_fullsized":"2024-04-25T01:31:21.867899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:54:08.265744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869647+0000","last_change":"2024-04-25T01:30:34.354644+0000","last_active":"2024-04-25T01:31:21.869647+0000","last_peered":"2024-04-25T01:31:21.869647+0000","last_clean":"2024-04-25T01:31:21.869647+0000","last_became_active":"2024-04-25T01:30:34.354330+0000","last_became_peered":"2024-04-25T01:30:34.354330+0000","last_unstale":"2024-04-25T01:31:21.869647+0000","last_undegraded":"2024-04-25T01:31:21.869647+0000","last_fullsized":"2024-04-25T01:31:21.869647+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:31:50.399983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867522+0000","last_change":"2024-04-25T01:30:38.450743+0000","last_active":"2024-04-25T01:31:21.867522+0000","last_peered":"2024-04-25T01:31:21.867522+0000","last_clean":"2024-04-25T01:31:21.867522+0000","last_became_active":"2024-04-25T01:30:38.450303+0000","last_became_peered":"2024-04-25T01:30:38.450303+0000","last_unstale":"2024-04-25T01:31:21.867522+0000","last_undegraded":"2024-04-25T01:31:21.867522+0000","last_fullsized":"2024-04-25T01:31:21.867522+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:37.919250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049267+0000","last_change":"2024-04-25T01:30:33.350709+0000","last_active":"2024-04-25T01:31:22.049267+0000","last_peered":"2024-04-25T01:31:22.049267+0000","last_clean":"2024-04-25T01:31:22.049267+0000","last_became_active":"2024-04-25T01:30:33.350387+0000","last_became_peered":"2024-04-25T01:30:33.350387+0000","last_unstale":"2024-04-25T01:31:22.049267+0000","last_undegraded":"2024-04-25T01:31:22.049267+0000","last_fullsized":"2024-04-25T01:31:22.049267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:30:01.252121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873782+0000","last_change":"2024-04-25T01:30:34.362040+0000","last_active":"2024-04-25T01:31:21.873782+0000","last_peered":"2024-04-25T01:31:21.873782+0000","last_clean":"2024-04-25T01:31:21.873782+0000","last_became_active":"2024-04-25T01:30:34.361734+0000","last_became_peered":"2024-04-25T01:30:34.361734+0000","last_unstale":"2024-04-25T01:31:21.873782+0000","last_undegraded":"2024-04-25T01:31:21.873782+0000","last_fullsized":"2024-04-25T01:31:21.873782+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:11:28.409764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.877148+0000","last_change":"2024-04-25T01:31:21.877148+0000","last_active":"2024-04-25T01:31:21.877148+0000","last_peered":"2024-04-25T01:31:21.877148+0000","last_clean":"2024-04-25T01:31:21.877148+0000","last_became_active":"2024-04-25T01:31:21.876665+0000","last_became_peered":"2024-04-25T01:31:21.876665+0000","last_unstale":"2024-04-25T01:31:21.877148+0000","last_undegraded":"2024-04-25T01:31:21.877148+0000","last_fullsized":"2024-04-25T01:31:21.877148+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:32:38.425434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,8],"acting":[9,3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'7","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873084+0000","last_change":"2024-04-25T01:30:33.350207+0000","last_active":"2024-04-25T01:31:21.873084+0000","last_peered":"2024-04-25T01:31:21.873084+0000","last_clean":"2024-04-25T01:31:21.873084+0000","last_became_active":"2024-04-25T01:30:33.349892+0000","last_became_peered":"2024-04-25T01:30:33.349892+0000","last_unstale":"2024-04-25T01:31:21.873084+0000","last_undegraded":"2024-04-25T01:31:21.873084+0000","last_fullsized":"2024-04-25T01:31:21.873084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:26:35.186486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869326+0000","last_change":"2024-04-25T01:30:34.355196+0000","last_active":"2024-04-25T01:31:21.869326+0000","last_peered":"2024-04-25T01:31:21.869326+0000","last_clean":"2024-04-25T01:31:21.869326+0000","last_became_active":"2024-04-25T01:30:34.354794+0000","last_became_peered":"2024-04-25T01:30:34.354794+0000","last_unstale":"2024-04-25T01:31:21.869326+0000","last_undegraded":"2024-04-25T01:31:21.869326+0000","last_fullsized":"2024-04-25T01:31:21.869326+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:43:37.495974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867406+0000","last_change":"2024-04-25T01:30:38.384159+0000","last_active":"2024-04-25T01:31:21.867406+0000","last_peered":"2024-04-25T01:31:21.867406+0000","last_clean":"2024-04-25T01:31:21.867406+0000","last_became_active":"2024-04-25T01:30:38.383676+0000","last_became_peered":"2024-04-25T01:30:38.383676+0000","last_unstale":"2024-04-25T01:31:21.867406+0000","last_undegraded":"2024-04-25T01:31:21.867406+0000","last_fullsized":"2024-04-25T01:31:21.867406+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:28.491522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.889311+0000","last_change":"2024-04-25T01:31:21.889311+0000","last_active":"2024-04-25T01:31:21.889311+0000","last_peered":"2024-04-25T01:31:21.889311+0000","last_clean":"2024-04-25T01:31:21.889311+0000","last_became_active":"2024-04-25T01:31:21.877027+0000","last_became_peered":"2024-04-25T01:31:21.877027+0000","last_unstale":"2024-04-25T01:31:21.889311+0000","last_undegraded":"2024-04-25T01:31:21.889311+0000","last_fullsized":"2024-04-25T01:31:21.889311+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:02.206671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872996+0000","last_change":"2024-04-25T01:30:34.358110+0000","last_active":"2024-04-25T01:31:21.872996+0000","last_peered":"2024-04-25T01:31:21.872996+0000","last_clean":"2024-04-25T01:31:21.872996+0000","last_became_active":"2024-04-25T01:30:34.357555+0000","last_became_peered":"2024-04-25T01:30:34.357555+0000","last_unstale":"2024-04-25T01:31:21.872996+0000","last_undegraded":"2024-04-25T01:31:21.872996+0000","last_fullsized":"2024-04-25T01:31:21.872996+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:11.636320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867320+0000","last_change":"2024-04-25T01:30:38.450680+0000","last_active":"2024-04-25T01:31:21.867320+0000","last_peered":"2024-04-25T01:31:21.867320+0000","last_clean":"2024-04-25T01:31:21.867320+0000","last_became_active":"2024-04-25T01:30:38.450168+0000","last_became_peered":"2024-04-25T01:30:38.450168+0000","last_unstale":"2024-04-25T01:31:21.867320+0000","last_undegraded":"2024-04-25T01:31:21.867320+0000","last_fullsized":"2024-04-25T01:31:21.867320+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:39:24.236371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.874429+0000","last_change":"2024-04-25T01:30:33.343848+0000","last_active":"2024-04-25T01:31:21.874429+0000","last_peered":"2024-04-25T01:31:21.874429+0000","last_clean":"2024-04-25T01:31:21.874429+0000","last_became_active":"2024-04-25T01:30:33.343498+0000","last_became_peered":"2024-04-25T01:30:33.343498+0000","last_unstale":"2024-04-25T01:31:21.874429+0000","last_undegraded":"2024-04-25T01:31:21.874429+0000","last_fullsized":"2024-04-25T01:31:21.874429+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:58:09.184571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049053+0000","last_change":"2024-04-25T01:30:34.361994+0000","last_active":"2024-04-25T01:31:22.049053+0000","last_peered":"2024-04-25T01:31:22.049053+0000","last_clean":"2024-04-25T01:31:22.049053+0000","last_became_active":"2024-04-25T01:30:34.361532+0000","last_became_peered":"2024-04-25T01:30:34.361532+0000","last_unstale":"2024-04-25T01:31:22.049053+0000","last_undegraded":"2024-04-25T01:31:22.049053+0000","last_fullsized":"2024-04-25T01:31:22.049053+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:30.667672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.048447+0000","last_change":"2024-04-25T01:30:38.369041+0000","last_active":"2024-04-25T01:31:22.048447+0000","last_peered":"2024-04-25T01:31:22.048447+0000","last_clean":"2024-04-25T01:31:22.048447+0000","last_became_active":"2024-04-25T01:30:38.368586+0000","last_became_peered":"2024-04-25T01:30:38.368586+0000","last_unstale":"2024-04-25T01:31:22.048447+0000","last_undegraded":"2024-04-25T01:31:22.048447+0000","last_fullsized":"2024-04-25T01:31:22.048447+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:58:53.743012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'6","reported_seq":106,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:32:10.911932+0000","last_change":"2024-04-25T01:30:33.345501+0000","last_active":"2024-04-25T01:32:10.911932+0000","last_peered":"2024-04-25T01:32:10.911932+0000","last_clean":"2024-04-25T01:32:10.911932+0000","last_became_active":"2024-04-25T01:30:33.345179+0000","last_became_peered":"2024-04-25T01:30:33.345179+0000","last_unstale":"2024-04-25T01:32:10.911932+0000","last_undegraded":"2024-04-25T01:32:10.911932+0000","last_fullsized":"2024-04-25T01:32:10.911932+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:29:41.275480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049572+0000","last_change":"2024-04-25T01:30:34.361631+0000","last_active":"2024-04-25T01:31:22.049572+0000","last_peered":"2024-04-25T01:31:22.049572+0000","last_clean":"2024-04-25T01:31:22.049572+0000","last_became_active":"2024-04-25T01:30:34.360266+0000","last_became_peered":"2024-04-25T01:30:34.360266+0000","last_unstale":"2024-04-25T01:31:22.049572+0000","last_undegraded":"2024-04-25T01:31:22.049572+0000","last_fullsized":"2024-04-25T01:31:22.049572+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:03.094166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869056+0000","last_change":"2024-04-25T01:30:38.380779+0000","last_active":"2024-04-25T01:31:21.869056+0000","last_peered":"2024-04-25T01:31:21.869056+0000","last_clean":"2024-04-25T01:31:21.869056+0000","last_became_active":"2024-04-25T01:30:38.379959+0000","last_became_peered":"2024-04-25T01:30:38.379959+0000","last_unstale":"2024-04-25T01:31:21.869056+0000","last_undegraded":"2024-04-25T01:31:21.869056+0000","last_fullsized":"2024-04-25T01:31:21.869056+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:48:01.816311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"91'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867900+0000","last_change":"2024-04-25T01:30:33.345730+0000","last_active":"2024-04-25T01:31:21.867900+0000","last_peered":"2024-04-25T01:31:21.867900+0000","last_clean":"2024-04-25T01:31:21.867900+0000","last_became_active":"2024-04-25T01:30:33.344832+0000","last_became_peered":"2024-04-25T01:30:33.344832+0000","last_unstale":"2024-04-25T01:31:21.867900+0000","last_undegraded":"2024-04-25T01:31:21.867900+0000","last_fullsized":"2024-04-25T01:31:21.867900+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:47.018811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049042+0000","last_change":"2024-04-25T01:30:34.355324+0000","last_active":"2024-04-25T01:31:22.049042+0000","last_peered":"2024-04-25T01:31:22.049042+0000","last_clean":"2024-04-25T01:31:22.049042+0000","last_became_active":"2024-04-25T01:30:34.354954+0000","last_became_peered":"2024-04-25T01:30:34.354954+0000","last_unstale":"2024-04-25T01:31:22.049042+0000","last_undegraded":"2024-04-25T01:31:22.049042+0000","last_fullsized":"2024-04-25T01:31:22.049042+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:36:41.932391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867087+0000","last_change":"2024-04-25T01:30:38.374386+0000","last_active":"2024-04-25T01:31:21.867087+0000","last_peered":"2024-04-25T01:31:21.867087+0000","last_clean":"2024-04-25T01:31:21.867087+0000","last_became_active":"2024-04-25T01:30:38.373454+0000","last_became_peered":"2024-04-25T01:30:38.373454+0000","last_unstale":"2024-04-25T01:31:21.867087+0000","last_undegraded":"2024-04-25T01:31:21.867087+0000","last_fullsized":"2024-04-25T01:31:21.867087+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:54.456218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"91'5","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866862+0000","last_change":"2024-04-25T01:30:33.339821+0000","last_active":"2024-04-25T01:31:21.866862+0000","last_peered":"2024-04-25T01:31:21.866862+0000","last_clean":"2024-04-25T01:31:21.866862+0000","last_became_active":"2024-04-25T01:30:33.339500+0000","last_became_peered":"2024-04-25T01:30:33.339500+0000","last_unstale":"2024-04-25T01:31:21.866862+0000","last_undegraded":"2024-04-25T01:31:21.866862+0000","last_fullsized":"2024-04-25T01:31:21.866862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:44:25.880828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.049617+0000","last_change":"2024-04-25T01:30:34.362736+0000","last_active":"2024-04-25T01:31:22.049617+0000","last_peered":"2024-04-25T01:31:22.049617+0000","last_clean":"2024-04-25T01:31:22.049617+0000","last_became_active":"2024-04-25T01:30:34.362371+0000","last_became_peered":"2024-04-25T01:30:34.362371+0000","last_unstale":"2024-04-25T01:31:22.049617+0000","last_undegraded":"2024-04-25T01:31:22.049617+0000","last_fullsized":"2024-04-25T01:31:22.049617+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:57:36.088231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869403+0000","last_change":"2024-04-25T01:30:34.359217+0000","last_active":"2024-04-25T01:31:21.869403+0000","last_peered":"2024-04-25T01:31:21.869403+0000","last_clean":"2024-04-25T01:31:21.869403+0000","last_became_active":"2024-04-25T01:30:34.357095+0000","last_became_peered":"2024-04-25T01:30:34.357095+0000","last_unstale":"2024-04-25T01:31:21.869403+0000","last_undegraded":"2024-04-25T01:31:21.869403+0000","last_fullsized":"2024-04-25T01:31:21.869403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:58.603484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873268+0000","last_change":"2024-04-25T01:30:34.359513+0000","last_active":"2024-04-25T01:31:21.873268+0000","last_peered":"2024-04-25T01:31:21.873268+0000","last_clean":"2024-04-25T01:31:21.873268+0000","last_became_active":"2024-04-25T01:30:34.359089+0000","last_became_peered":"2024-04-25T01:30:34.359089+0000","last_unstale":"2024-04-25T01:31:21.873268+0000","last_undegraded":"2024-04-25T01:31:21.873268+0000","last_fullsized":"2024-04-25T01:31:21.873268+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:36.159703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.872696+0000","last_change":"2024-04-25T01:30:34.368739+0000","last_active":"2024-04-25T01:31:21.872696+0000","last_peered":"2024-04-25T01:31:21.872696+0000","last_clean":"2024-04-25T01:31:21.872696+0000","last_became_active":"2024-04-25T01:30:34.368392+0000","last_became_peered":"2024-04-25T01:30:34.368392+0000","last_unstale":"2024-04-25T01:31:21.872696+0000","last_undegraded":"2024-04-25T01:31:21.872696+0000","last_fullsized":"2024-04-25T01:31:21.872696+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:39:32.241889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.877568+0000","last_change":"2024-04-25T01:31:21.877568+0000","last_active":"2024-04-25T01:31:21.877568+0000","last_peered":"2024-04-25T01:31:21.877568+0000","last_clean":"2024-04-25T01:31:21.877568+0000","last_became_active":"2024-04-25T01:31:21.877071+0000","last_became_peered":"2024-04-25T01:31:21.877071+0000","last_unstale":"2024-04-25T01:31:21.877568+0000","last_undegraded":"2024-04-25T01:31:21.877568+0000","last_fullsized":"2024-04-25T01:31:21.877568+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:02:24.789974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866839+0000","last_change":"2024-04-25T01:30:34.360979+0000","last_active":"2024-04-25T01:31:21.866839+0000","last_peered":"2024-04-25T01:31:21.866839+0000","last_clean":"2024-04-25T01:31:21.866839+0000","last_became_active":"2024-04-25T01:30:34.360630+0000","last_became_peered":"2024-04-25T01:30:34.360630+0000","last_unstale":"2024-04-25T01:31:21.866839+0000","last_undegraded":"2024-04-25T01:31:21.866839+0000","last_fullsized":"2024-04-25T01:31:21.866839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:05:19.284577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869923+0000","last_change":"2024-04-25T01:30:34.362621+0000","last_active":"2024-04-25T01:31:21.869923+0000","last_peered":"2024-04-25T01:31:21.869923+0000","last_clean":"2024-04-25T01:31:21.869923+0000","last_became_active":"2024-04-25T01:30:34.362290+0000","last_became_peered":"2024-04-25T01:30:34.362290+0000","last_unstale":"2024-04-25T01:31:21.869923+0000","last_undegraded":"2024-04-25T01:31:21.869923+0000","last_fullsized":"2024-04-25T01:31:21.869923+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:28:07.750906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873423+0000","last_change":"2024-04-25T01:30:34.351339+0000","last_active":"2024-04-25T01:31:21.873423+0000","last_peered":"2024-04-25T01:31:21.873423+0000","last_clean":"2024-04-25T01:31:21.873423+0000","last_became_active":"2024-04-25T01:30:34.350655+0000","last_became_peered":"2024-04-25T01:30:34.350655+0000","last_unstale":"2024-04-25T01:31:21.873423+0000","last_undegraded":"2024-04-25T01:31:21.873423+0000","last_fullsized":"2024-04-25T01:31:21.873423+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:05.322547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873518+0000","last_change":"2024-04-25T01:30:34.353876+0000","last_active":"2024-04-25T01:31:21.873518+0000","last_peered":"2024-04-25T01:31:21.873518+0000","last_clean":"2024-04-25T01:31:21.873518+0000","last_became_active":"2024-04-25T01:30:34.352942+0000","last_became_peered":"2024-04-25T01:30:34.352942+0000","last_unstale":"2024-04-25T01:31:21.873518+0000","last_undegraded":"2024-04-25T01:31:21.873518+0000","last_fullsized":"2024-04-25T01:31:21.873518+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:47.260171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.867312+0000","last_change":"2024-04-25T01:30:34.352039+0000","last_active":"2024-04-25T01:31:21.867312+0000","last_peered":"2024-04-25T01:31:21.867312+0000","last_clean":"2024-04-25T01:31:21.867312+0000","last_became_active":"2024-04-25T01:30:34.351612+0000","last_became_peered":"2024-04-25T01:30:34.351612+0000","last_unstale":"2024-04-25T01:31:21.867312+0000","last_undegraded":"2024-04-25T01:31:21.867312+0000","last_fullsized":"2024-04-25T01:31:21.867312+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:03.607711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.873300+0000","last_change":"2024-04-25T01:30:34.353692+0000","last_active":"2024-04-25T01:31:21.873300+0000","last_peered":"2024-04-25T01:31:21.873300+0000","last_clean":"2024-04-25T01:31:21.873300+0000","last_became_active":"2024-04-25T01:30:34.353035+0000","last_became_peered":"2024-04-25T01:30:34.353035+0000","last_unstale":"2024-04-25T01:31:21.873300+0000","last_undegraded":"2024-04-25T01:31:21.873300+0000","last_fullsized":"2024-04-25T01:31:21.873300+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:15:54.540315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:22.048744+0000","last_change":"2024-04-25T01:30:34.370797+0000","last_active":"2024-04-25T01:31:22.048744+0000","last_peered":"2024-04-25T01:31:22.048744+0000","last_clean":"2024-04-25T01:31:22.048744+0000","last_became_active":"2024-04-25T01:30:34.367533+0000","last_became_peered":"2024-04-25T01:30:34.367533+0000","last_unstale":"2024-04-25T01:31:22.048744+0000","last_undegraded":"2024-04-25T01:31:22.048744+0000","last_fullsized":"2024-04-25T01:31:22.048744+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:27:15.478808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.866640+0000","last_change":"2024-04-25T01:30:34.364843+0000","last_active":"2024-04-25T01:31:21.866640+0000","last_peered":"2024-04-25T01:31:21.866640+0000","last_clean":"2024-04-25T01:31:21.866640+0000","last_became_active":"2024-04-25T01:30:34.364515+0000","last_became_peered":"2024-04-25T01:30:34.364515+0000","last_unstale":"2024-04-25T01:31:21.866640+0000","last_undegraded":"2024-04-25T01:31:21.866640+0000","last_fullsized":"2024-04-25T01:31:21.866640+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:50:03.517814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-25T01:31:21.869814+0000","last_change":"2024-04-25T01:30:34.363715+0000","last_active":"2024-04-25T01:31:21.869814+0000","last_peered":"2024-04-25T01:31:21.869814+0000","last_clean":"2024-04-25T01:31:21.869814+0000","last_became_active":"2024-04-25T01:30:34.363383+0000","last_became_peered":"2024-04-25T01:30:34.363383+0000","last_unstale":"2024-04-25T01:31:21.869814+0000","last_undegraded":"2024-04-25T01:31:21.869814+0000","last_fullsized":"2024-04-25T01:31:21.869814+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:50:18.322550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69537,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":766,"num_read_kb":460,"num_write":241,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":208611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318306,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1060864,"data_stored":510000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579943,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37044,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708108,"statfs":{"total":95995035648,"available":95957102592,"internally_reserved":0,"allocated":1748992,"data_stored":1158690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743469,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":513567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005106,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1073152,"data_stored":509849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266743,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28656,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716496,"statfs":{"total":95995035648,"available":95965691904,"internally_reserved":0,"allocated":1671168,"data_stored":1106346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430269,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":504975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1122304,"data_stored":568338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36396,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708756,"statfs":{"total":95995035648,"available":95957766144,"internally_reserved":0,"allocated":1077248,"data_stored":506822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36444,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708708,"statfs":{"total":95995035648,"available":95957716992,"internally_reserved":0,"allocated":1134592,"data_stored":569272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1064960,"data_stored":504683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888134,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1060864,"data_stored":509049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691906,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32820,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712332,"statfs":{"total":95995035648,"available":95961427968,"internally_reserved":0,"allocated":1679360,"data_stored":1101928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":65019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":64069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:32:18.472 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T01:32:18.472 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T01:32:18.472 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-25T01:32:18.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-25T01:32:18.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:18 smithi032 ceph-mon[32461]: pgmap v351: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:18.724 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:32:19.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:19 smithi028 ceph-mon[29390]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:32:19.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:19 smithi130 ceph-mon[32702]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:32:19.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:19 smithi032 ceph-mon[32461]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:32:20.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:20 smithi028 ceph-mon[29390]: pgmap v352: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:20.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:20 smithi130 ceph-mon[32702]: pgmap v352: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:20.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:20 smithi032 ceph-mon[32461]: pgmap v352: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.623+0000 7f673e98c700 1 -- 172.21.15.28:0/1593457238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6738102510 msgr2=0x7f67381028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.623+0000 7f673e98c700 1 --2- 172.21.15.28:0/1593457238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6738102510 0x7f67381028f0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f67280099c0 tx=0x7f6728009cd0 comp rx=0 tx=0).stop 2024-04-25T01:32:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.624+0000 7f673e98c700 1 -- 172.21.15.28:0/1593457238 shutdown_connections 2024-04-25T01:32:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.624+0000 7f673e98c700 1 --2- 172.21.15.28:0/1593457238 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67381075e0 0x7f673810f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.624+0000 7f673e98c700 1 --2- 172.21.15.28:0/1593457238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6738102510 0x7f67381028f0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.624+0000 7f673e98c700 1 --2- 172.21.15.28:0/1593457238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6738102ec0 0x7f6738106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.624+0000 7f673e98c700 1 -- 172.21.15.28:0/1593457238 >> 172.21.15.28:0/1593457238 conn(0x7f67380793e0 msgr2=0x7f67380797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.624+0000 7f673e98c700 1 -- 172.21.15.28:0/1593457238 shutdown_connections 2024-04-25T01:32:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.624+0000 7f673e98c700 1 -- 172.21.15.28:0/1593457238 wait complete. 2024-04-25T01:32:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.624+0000 7f673e98c700 1 Processor -- start 2024-04-25T01:32:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.624+0000 7f673e98c700 1 -- start start 2024-04-25T01:32:20.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.624+0000 7f673e98c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6738102510 0x7f673819e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:20.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.624+0000 7f673e98c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6738102ec0 0x7f673819ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:20.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.625+0000 7f673e98c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67381075e0 0x7f6738198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:20.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.625+0000 7f673e98c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6738112640 con 0x7f67381075e0 2024-04-25T01:32:20.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.625+0000 7f673e98c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67381124c0 con 0x7f6738102ec0 2024-04-25T01:32:20.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.625+0000 7f673e98c700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f67381127c0 con 0x7f6738102510 2024-04-25T01:32:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.625+0000 7f673cf29700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67381075e0 0x7f6738198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.625+0000 7f673cf29700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67381075e0 0x7f6738198800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45196/0 (socket says 172.21.15.28:45196) 2024-04-25T01:32:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.625+0000 7f673cf29700 1 -- 172.21.15.28:0/3927275793 learned_addr learned my addr 172.21.15.28:0/3927275793 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.625+0000 7f6737fff700 1 --2- 172.21.15.28:0/3927275793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6738102510 0x7f673819e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.625+0000 7f672f7fe700 1 --2- 172.21.15.28:0/3927275793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6738102ec0 0x7f673819ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.625+0000 7f673cf29700 1 -- 172.21.15.28:0/3927275793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6738102510 msgr2=0x7f673819e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.625+0000 7f673cf29700 1 --2- 172.21.15.28:0/3927275793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6738102510 0x7f673819e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.625+0000 7f673cf29700 1 -- 172.21.15.28:0/3927275793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6738102ec0 msgr2=0x7f673819ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.625+0000 7f673cf29700 1 --2- 172.21.15.28:0/3927275793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6738102ec0 0x7f673819ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.625+0000 7f673cf29700 1 -- 172.21.15.28:0/3927275793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6738198fe0 con 0x7f67381075e0 2024-04-25T01:32:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.625+0000 7f672f7fe700 1 --2- 172.21.15.28:0/3927275793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6738102ec0 0x7f673819ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.625+0000 7f673cf29700 1 --2- 172.21.15.28:0/3927275793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67381075e0 0x7f6738198800 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f673000b970 tx=0x7f673000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.625+0000 7f6735ffb700 1 -- 172.21.15.28:0/3927275793 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6730019070 con 0x7f67381075e0 2024-04-25T01:32:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.625+0000 7f6735ffb700 1 -- 172.21.15.28:0/3927275793 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f673000ce10 con 0x7f67381075e0 2024-04-25T01:32:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.625+0000 7f6735ffb700 1 -- 172.21.15.28:0/3927275793 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6730007370 con 0x7f67381075e0 2024-04-25T01:32:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.626+0000 7f673e98c700 1 -- 172.21.15.28:0/3927275793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67381992d0 con 0x7f67381075e0 2024-04-25T01:32:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.626+0000 7f673e98c700 1 -- 172.21.15.28:0/3927275793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67381a54e0 con 0x7f67381075e0 2024-04-25T01:32:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.627+0000 7f673e98c700 1 -- 172.21.15.28:0/3927275793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67381039e0 con 0x7f67381075e0 2024-04-25T01:32:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.628+0000 7f6735ffb700 1 -- 172.21.15.28:0/3927275793 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f673001c070 con 0x7f67381075e0 2024-04-25T01:32:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.629+0000 7f6735ffb700 1 --2- 172.21.15.28:0/3927275793 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67240a2cb0 0x7f67240a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.629+0000 7f6737fff700 1 --2- 172.21.15.28:0/3927275793 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67240a2cb0 0x7f67240a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.629+0000 7f6735ffb700 1 -- 172.21.15.28:0/3927275793 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f67300cec80 con 0x7f67381075e0 2024-04-25T01:32:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.630+0000 7f6737fff700 1 --2- 172.21.15.28:0/3927275793 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67240a2cb0 0x7f67240a5170 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6728000c00 tx=0x7f6728005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:20.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.633+0000 7f6735ffb700 1 -- 172.21.15.28:0/3927275793 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f673009b7d0 con 0x7f67381075e0 2024-04-25T01:32:20.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.887+0000 7f673e98c700 1 -- 172.21.15.28:0/3927275793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f6738199d30 con 0x7f67381075e0 2024-04-25T01:32:20.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.887+0000 7f6735ffb700 1 -- 172.21.15.28:0/3927275793 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f673009b0d0 con 0x7f67381075e0 2024-04-25T01:32:20.865 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:32:20.865 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T01:32:20.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.889+0000 7f673e98c700 1 -- 172.21.15.28:0/3927275793 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67240a2cb0 msgr2=0x7f67240a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:20.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.889+0000 7f673e98c700 1 --2- 172.21.15.28:0/3927275793 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67240a2cb0 0x7f67240a5170 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6728000c00 tx=0x7f6728005fb0 comp rx=0 tx=0).stop 2024-04-25T01:32:20.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.889+0000 7f673e98c700 1 -- 172.21.15.28:0/3927275793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67381075e0 msgr2=0x7f6738198800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:20.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.889+0000 7f673e98c700 1 --2- 172.21.15.28:0/3927275793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67381075e0 0x7f6738198800 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f673000b970 tx=0x7f673000bc80 comp rx=0 tx=0).stop 2024-04-25T01:32:20.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.889+0000 7f673e98c700 1 -- 172.21.15.28:0/3927275793 shutdown_connections 2024-04-25T01:32:20.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.889+0000 7f673e98c700 1 --2- 172.21.15.28:0/3927275793 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67240a2cb0 0x7f67240a5170 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:20.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.889+0000 7f673e98c700 1 --2- 172.21.15.28:0/3927275793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67381075e0 0x7f6738198800 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:20.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.889+0000 7f673e98c700 1 --2- 172.21.15.28:0/3927275793 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6738102510 0x7f673819e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:20.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.889+0000 7f673e98c700 1 --2- 172.21.15.28:0/3927275793 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6738102ec0 0x7f673819ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:20.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.889+0000 7f673e98c700 1 -- 172.21.15.28:0/3927275793 >> 172.21.15.28:0/3927275793 conn(0x7f67380793e0 msgr2=0x7f67381002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:20.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.890+0000 7f673e98c700 1 -- 172.21.15.28:0/3927275793 shutdown_connections 2024-04-25T01:32:20.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:20.890+0000 7f673e98c700 1 -- 172.21.15.28:0/3927275793 wait complete. 2024-04-25T01:32:21.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:21 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3927275793' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T01:32:21.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:21 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3927275793' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T01:32:21.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:21 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3927275793' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T01:32:21.627 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-25T01:32:21.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:32:21.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.756+0000 7ff3cc4d4640 1 -- 172.21.15.28:0/1951491033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c4103360 msgr2=0x7ff3c4109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:21.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.756+0000 7ff3cc4d4640 1 --2- 172.21.15.28:0/1951491033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c4103360 0x7ff3c4109bf0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009ca0 tx=0x7ff3bc02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.756+0000 7ff3cc4d4640 1 -- 172.21.15.28:0/1951491033 shutdown_connections 2024-04-25T01:32:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.756+0000 7ff3cc4d4640 1 --2- 172.21.15.28:0/1951491033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c4103360 0x7ff3c4109bf0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.756+0000 7ff3cc4d4640 1 --2- 172.21.15.28:0/1951491033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3c41029a0 0x7ff3c4102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.756+0000 7ff3cc4d4640 1 --2- 172.21.15.28:0/1951491033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3c41017b0 0x7ff3c4101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.756+0000 7ff3cc4d4640 1 -- 172.21.15.28:0/1951491033 >> 172.21.15.28:0/1951491033 conn(0x7ff3c40fcd40 msgr2=0x7ff3c40ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.756+0000 7ff3cc4d4640 1 -- 172.21.15.28:0/1951491033 shutdown_connections 2024-04-25T01:32:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.756+0000 7ff3cc4d4640 1 -- 172.21.15.28:0/1951491033 wait complete. 2024-04-25T01:32:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.757+0000 7ff3cc4d4640 1 Processor -- start 2024-04-25T01:32:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.757+0000 7ff3cc4d4640 1 -- start start 2024-04-25T01:32:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.757+0000 7ff3cc4d4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3c41017b0 0x7ff3c4197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.757+0000 7ff3cc4d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c41029a0 0x7ff3c4197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.757+0000 7ff3cc4d4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3c4103360 0x7ff3c419eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:21.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.757+0000 7ff3cc4d4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c410d410 con 0x7ff3c41029a0 2024-04-25T01:32:21.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.757+0000 7ff3cc4d4640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c410d290 con 0x7ff3c41017b0 2024-04-25T01:32:21.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.757+0000 7ff3cc4d4640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c410d590 con 0x7ff3c4103360 2024-04-25T01:32:21.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.757+0000 7ff3c9a6f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c41029a0 0x7ff3c4197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:21.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.757+0000 7ff3c9a6f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c41029a0 0x7ff3c4197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45222/0 (socket says 172.21.15.28:45222) 2024-04-25T01:32:21.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.757+0000 7ff3c9a6f640 1 -- 172.21.15.28:0/466114098 learned_addr learned my addr 172.21.15.28:0/466114098 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:21.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.757+0000 7ff3ca270640 1 --2- 172.21.15.28:0/466114098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3c41017b0 0x7ff3c4197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:21.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.757+0000 7ff3caa71640 1 --2- 172.21.15.28:0/466114098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3c4103360 0x7ff3c419eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:21.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.758+0000 7ff3c9a6f640 1 -- 172.21.15.28:0/466114098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3c4103360 msgr2=0x7ff3c419eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:21.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.758+0000 7ff3c9a6f640 1 --2- 172.21.15.28:0/466114098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3c4103360 0x7ff3c419eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:21.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.758+0000 7ff3c9a6f640 1 -- 172.21.15.28:0/466114098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3c41017b0 msgr2=0x7ff3c4197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:21.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.758+0000 7ff3c9a6f640 1 --2- 172.21.15.28:0/466114098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3c41017b0 0x7ff3c4197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:21.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.758+0000 7ff3c9a6f640 1 -- 172.21.15.28:0/466114098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3c419f520 con 0x7ff3c41029a0 2024-04-25T01:32:21.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.758+0000 7ff3c9a6f640 1 --2- 172.21.15.28:0/466114098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c41029a0 0x7ff3c4197e40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff3b400c920 tx=0x7ff3b400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:21.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.758+0000 7ff3bb7fe640 1 -- 172.21.15.28:0/466114098 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4007c70 con 0x7ff3c41029a0 2024-04-25T01:32:21.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.758+0000 7ff3bb7fe640 1 -- 172.21.15.28:0/466114098 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3b4007e10 con 0x7ff3c41029a0 2024-04-25T01:32:21.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.758+0000 7ff3bb7fe640 1 -- 172.21.15.28:0/466114098 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4018890 con 0x7ff3c41029a0 2024-04-25T01:32:21.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.758+0000 7ff3cc4d4640 1 -- 172.21.15.28:0/466114098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3c419f810 con 0x7ff3c41029a0 2024-04-25T01:32:21.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.758+0000 7ff3cc4d4640 1 -- 172.21.15.28:0/466114098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3c419fd60 con 0x7ff3c41029a0 2024-04-25T01:32:21.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.759+0000 7ff3cc4d4640 1 -- 172.21.15.28:0/466114098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3c4108a00 con 0x7ff3c41029a0 2024-04-25T01:32:21.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.760+0000 7ff3bb7fe640 1 -- 172.21.15.28:0/466114098 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff3b4018a70 con 0x7ff3c41029a0 2024-04-25T01:32:21.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.761+0000 7ff3bb7fe640 1 --2- 172.21.15.28:0/466114098 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff3b00a2b90 0x7ff3b00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:21.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.761+0000 7ff3ca270640 1 --2- 172.21.15.28:0/466114098 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff3b00a2b90 0x7ff3b00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:21.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.761+0000 7ff3bb7fe640 1 -- 172.21.15.28:0/466114098 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7ff3b40cf4a0 con 0x7ff3c41029a0 2024-04-25T01:32:21.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.761+0000 7ff3ca270640 1 --2- 172.21.15.28:0/466114098 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff3b00a2b90 0x7ff3b00a5050 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4102800 tx=0x7ff3c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:21.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.765+0000 7ff3bb7fe640 1 -- 172.21.15.28:0/466114098 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff3b4014030 con 0x7ff3c41029a0 2024-04-25T01:32:21.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.952+0000 7ff3cc4d4640 1 -- 172.21.15.28:0/466114098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff3c4062340 con 0x7ff3c41029a0 2024-04-25T01:32:21.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.953+0000 7ff3bb7fe640 1 -- 172.21.15.28:0/466114098 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24564 (secure 0 0 0) 0x7ff3b409c040 con 0x7ff3c41029a0 2024-04-25T01:32:21.931 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:32:21.931 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:31:21.857121+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:32:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.955+0000 7ff3cc4d4640 1 -- 172.21.15.28:0/466114098 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff3b00a2b90 msgr2=0x7ff3b00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.955+0000 7ff3cc4d4640 1 --2- 172.21.15.28:0/466114098 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff3b00a2b90 0x7ff3b00a5050 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4102800 tx=0x7ff3c0005f90 comp rx=0 tx=0).stop 2024-04-25T01:32:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.955+0000 7ff3cc4d4640 1 -- 172.21.15.28:0/466114098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c41029a0 msgr2=0x7ff3c4197e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.955+0000 7ff3cc4d4640 1 --2- 172.21.15.28:0/466114098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c41029a0 0x7ff3c4197e40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff3b400c920 tx=0x7ff3b400cde0 comp rx=0 tx=0).stop 2024-04-25T01:32:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.955+0000 7ff3cc4d4640 1 -- 172.21.15.28:0/466114098 shutdown_connections 2024-04-25T01:32:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.955+0000 7ff3cc4d4640 1 --2- 172.21.15.28:0/466114098 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff3b00a2b90 0x7ff3b00a5050 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.955+0000 7ff3cc4d4640 1 --2- 172.21.15.28:0/466114098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3c4103360 0x7ff3c419eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.955+0000 7ff3cc4d4640 1 --2- 172.21.15.28:0/466114098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3c41029a0 0x7ff3c4197e40 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.955+0000 7ff3cc4d4640 1 --2- 172.21.15.28:0/466114098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff3c41017b0 0x7ff3c4197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.955+0000 7ff3cc4d4640 1 -- 172.21.15.28:0/466114098 >> 172.21.15.28:0/466114098 conn(0x7ff3c40fcd40 msgr2=0x7ff3c4105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.956+0000 7ff3cc4d4640 1 -- 172.21.15.28:0/466114098 shutdown_connections 2024-04-25T01:32:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:21.956+0000 7ff3cc4d4640 1 -- 172.21.15.28:0/466114098 wait complete. 2024-04-25T01:32:21.945 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:25:48.767414+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:30:32.282430+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:30:33.127049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:30:36.796132+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:32:21.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:32:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.072+0000 7f1d16761640 1 -- 172.21.15.28:0/2450260796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d10102230 msgr2=0x7f1d10108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.072+0000 7f1d16761640 1 --2- 172.21.15.28:0/2450260796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d10102230 0x7f1d10108ac0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1d08009ca0 tx=0x7f1d0802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.072+0000 7f1d16761640 1 -- 172.21.15.28:0/2450260796 shutdown_connections 2024-04-25T01:32:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.072+0000 7f1d16761640 1 --2- 172.21.15.28:0/2450260796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d10102230 0x7f1d10108ac0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.072+0000 7f1d16761640 1 --2- 172.21.15.28:0/2450260796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d10101870 0x7f1d10101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.072+0000 7f1d16761640 1 --2- 172.21.15.28:0/2450260796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d10100680 0x7f1d10100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.072+0000 7f1d16761640 1 -- 172.21.15.28:0/2450260796 >> 172.21.15.28:0/2450260796 conn(0x7f1d100fbbf0 msgr2=0x7f1d100fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.072+0000 7f1d16761640 1 -- 172.21.15.28:0/2450260796 shutdown_connections 2024-04-25T01:32:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.072+0000 7f1d16761640 1 -- 172.21.15.28:0/2450260796 wait complete. 2024-04-25T01:32:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.073+0000 7f1d16761640 1 Processor -- start 2024-04-25T01:32:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.073+0000 7f1d16761640 1 -- start start 2024-04-25T01:32:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.073+0000 7f1d16761640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d10100680 0x7f1d10069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.073+0000 7f1d16761640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d10101870 0x7f1d1019c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.073+0000 7f1d16761640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d10102230 0x7f1d1019edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.073+0000 7f1d16761640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1010c300 con 0x7f1d10101870 2024-04-25T01:32:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.073+0000 7f1d16761640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1010c180 con 0x7f1d10100680 2024-04-25T01:32:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.073+0000 7f1d16761640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1010c480 con 0x7f1d10102230 2024-04-25T01:32:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.073+0000 7f1d0f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d10101870 0x7f1d1019c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:22.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.074+0000 7f1d0f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d10101870 0x7f1d1019c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45244/0 (socket says 172.21.15.28:45244) 2024-04-25T01:32:22.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.074+0000 7f1d0f7fe640 1 -- 172.21.15.28:0/4057367025 learned_addr learned my addr 172.21.15.28:0/4057367025 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:22.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.074+0000 7f1d14cfe640 1 --2- 172.21.15.28:0/4057367025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d10102230 0x7f1d1019edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:22.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.074+0000 7f1d0ffff640 1 --2- 172.21.15.28:0/4057367025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d10100680 0x7f1d10069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:22.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.074+0000 7f1d0f7fe640 1 -- 172.21.15.28:0/4057367025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d10102230 msgr2=0x7f1d1019edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:22.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.074+0000 7f1d0f7fe640 1 --2- 172.21.15.28:0/4057367025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d10102230 0x7f1d1019edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.074+0000 7f1d0f7fe640 1 -- 172.21.15.28:0/4057367025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d10100680 msgr2=0x7f1d10069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:22.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.074+0000 7f1d0f7fe640 1 --2- 172.21.15.28:0/4057367025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d10100680 0x7f1d10069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.074+0000 7f1d0f7fe640 1 -- 172.21.15.28:0/4057367025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d1019f700 con 0x7f1d10101870 2024-04-25T01:32:22.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.074+0000 7f1d0ffff640 1 --2- 172.21.15.28:0/4057367025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d10100680 0x7f1d10069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:22.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.074+0000 7f1d0f7fe640 1 --2- 172.21.15.28:0/4057367025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d10101870 0x7f1d1019c7d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000d770 tx=0x7f1d0000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:22.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.074+0000 7f1d0d7fa640 1 -- 172.21.15.28:0/4057367025 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d00017070 con 0x7f1d10101870 2024-04-25T01:32:22.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.074+0000 7f1d0d7fa640 1 -- 172.21.15.28:0/4057367025 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d00005500 con 0x7f1d10101870 2024-04-25T01:32:22.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.074+0000 7f1d16761640 1 -- 172.21.15.28:0/4057367025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d1019f990 con 0x7f1d10101870 2024-04-25T01:32:22.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.074+0000 7f1d0d7fa640 1 -- 172.21.15.28:0/4057367025 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d00004e00 con 0x7f1d10101870 2024-04-25T01:32:22.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.074+0000 7f1d16761640 1 -- 172.21.15.28:0/4057367025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d1019fe90 con 0x7f1d10101870 2024-04-25T01:32:22.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.075+0000 7f1d16761640 1 -- 172.21.15.28:0/4057367025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d10107a20 con 0x7f1d10101870 2024-04-25T01:32:22.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.076+0000 7f1d0d7fa640 1 -- 172.21.15.28:0/4057367025 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1d00004fa0 con 0x7f1d10101870 2024-04-25T01:32:22.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.077+0000 7f1d0d7fa640 1 --2- 172.21.15.28:0/4057367025 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1cfc0a2b90 0x7f1cfc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:22.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.077+0000 7f1d0ffff640 1 --2- 172.21.15.28:0/4057367025 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1cfc0a2b90 0x7f1cfc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:22.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.077+0000 7f1d0d7fa640 1 -- 172.21.15.28:0/4057367025 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f1d000cee20 con 0x7f1d10101870 2024-04-25T01:32:22.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.078+0000 7f1d0ffff640 1 --2- 172.21.15.28:0/4057367025 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1cfc0a2b90 0x7f1cfc0a5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80058c0 tx=0x7f1cf8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:22.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.082+0000 7f1d0d7fa640 1 -- 172.21.15.28:0/4057367025 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1d0009ba20 con 0x7f1d10101870 2024-04-25T01:32:22.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.264+0000 7f1d16761640 1 -- 172.21.15.28:0/4057367025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1d10062340 con 0x7f1d10101870 2024-04-25T01:32:22.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.265+0000 7f1d0d7fa640 1 -- 172.21.15.28:0/4057367025 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f1d0009b320 con 0x7f1d10101870 2024-04-25T01:32:22.243 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T01:32:22.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.267+0000 7f1d16761640 1 -- 172.21.15.28:0/4057367025 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1cfc0a2b90 msgr2=0x7f1cfc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:22.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.267+0000 7f1d16761640 1 --2- 172.21.15.28:0/4057367025 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1cfc0a2b90 0x7f1cfc0a5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80058c0 tx=0x7f1cf8005850 comp rx=0 tx=0).stop 2024-04-25T01:32:22.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.267+0000 7f1d16761640 1 -- 172.21.15.28:0/4057367025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d10101870 msgr2=0x7f1d1019c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:22.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.267+0000 7f1d16761640 1 --2- 172.21.15.28:0/4057367025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d10101870 0x7f1d1019c7d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000d770 tx=0x7f1d0000dc30 comp rx=0 tx=0).stop 2024-04-25T01:32:22.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.267+0000 7f1d16761640 1 -- 172.21.15.28:0/4057367025 shutdown_connections 2024-04-25T01:32:22.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.267+0000 7f1d16761640 1 --2- 172.21.15.28:0/4057367025 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1cfc0a2b90 0x7f1cfc0a5050 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.267+0000 7f1d16761640 1 --2- 172.21.15.28:0/4057367025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1d10102230 0x7f1d1019edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.267+0000 7f1d16761640 1 --2- 172.21.15.28:0/4057367025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d10101870 0x7f1d1019c7d0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.267+0000 7f1d16761640 1 --2- 172.21.15.28:0/4057367025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1d10100680 0x7f1d10069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.267+0000 7f1d16761640 1 -- 172.21.15.28:0/4057367025 >> 172.21.15.28:0/4057367025 conn(0x7f1d100fbbf0 msgr2=0x7f1d10104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:22.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.267+0000 7f1d16761640 1 -- 172.21.15.28:0/4057367025 shutdown_connections 2024-04-25T01:32:22.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.267+0000 7f1d16761640 1 -- 172.21.15.28:0/4057367025 wait complete. 2024-04-25T01:32:22.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:22 smithi028 ceph-mon[29390]: pgmap v353: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:22.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:22 smithi028 ceph-mon[29390]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-25T01:32:22.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:22 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/466114098' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:22.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:32:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.385+0000 7f8ba1ecd640 1 -- 172.21.15.28:0/3582612894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c102230 msgr2=0x7f8b9c108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.385+0000 7f8ba1ecd640 1 --2- 172.21.15.28:0/3582612894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c102230 0x7f8b9c108ac0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009be0 tx=0x7f8b9402a6c0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.385+0000 7f8ba1ecd640 1 -- 172.21.15.28:0/3582612894 shutdown_connections 2024-04-25T01:32:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.385+0000 7f8ba1ecd640 1 --2- 172.21.15.28:0/3582612894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c102230 0x7f8b9c108ac0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.385+0000 7f8ba1ecd640 1 --2- 172.21.15.28:0/3582612894 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b9c101870 0x7f8b9c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.385+0000 7f8ba1ecd640 1 --2- 172.21.15.28:0/3582612894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b9c100680 0x7f8b9c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.385+0000 7f8ba1ecd640 1 -- 172.21.15.28:0/3582612894 >> 172.21.15.28:0/3582612894 conn(0x7f8b9c0fbbf0 msgr2=0x7f8b9c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:22.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.385+0000 7f8ba1ecd640 1 -- 172.21.15.28:0/3582612894 shutdown_connections 2024-04-25T01:32:22.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.385+0000 7f8ba1ecd640 1 -- 172.21.15.28:0/3582612894 wait complete. 2024-04-25T01:32:22.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.385+0000 7f8ba1ecd640 1 Processor -- start 2024-04-25T01:32:22.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.386+0000 7f8ba1ecd640 1 -- start start 2024-04-25T01:32:22.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.386+0000 7f8ba1ecd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b9c100680 0x7f8b9c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:22.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.386+0000 7f8ba1ecd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b9c101870 0x7f8b9c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:22.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.386+0000 7f8ba1ecd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c102230 0x7f8b9c19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:22.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.386+0000 7f8ba1ecd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c10c2c0 con 0x7f8b9c102230 2024-04-25T01:32:22.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.386+0000 7f8ba1ecd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c10c140 con 0x7f8b9c100680 2024-04-25T01:32:22.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.386+0000 7f8ba1ecd640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c10c440 con 0x7f8b9c101870 2024-04-25T01:32:22.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.386+0000 7f8b9bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c102230 0x7f8b9c19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:22.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.386+0000 7f8b9bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c102230 0x7f8b9c19f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45252/0 (socket says 172.21.15.28:45252) 2024-04-25T01:32:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.386+0000 7f8b9bfff640 1 -- 172.21.15.28:0/56066172 learned_addr learned my addr 172.21.15.28:0/56066172 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.386+0000 7f8b9b7fe640 1 --2- 172.21.15.28:0/56066172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b9c100680 0x7f8b9c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.386+0000 7f8b9affd640 1 --2- 172.21.15.28:0/56066172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b9c101870 0x7f8b9c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.386+0000 7f8b9bfff640 1 -- 172.21.15.28:0/56066172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b9c101870 msgr2=0x7f8b9c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.386+0000 7f8b9bfff640 1 --2- 172.21.15.28:0/56066172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b9c101870 0x7f8b9c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.386+0000 7f8b9bfff640 1 -- 172.21.15.28:0/56066172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b9c100680 msgr2=0x7f8b9c197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.386+0000 7f8b9bfff640 1 --2- 172.21.15.28:0/56066172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b9c100680 0x7f8b9c197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.386+0000 7f8b9bfff640 1 -- 172.21.15.28:0/56066172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b9c19f6a0 con 0x7f8b9c102230 2024-04-25T01:32:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.386+0000 7f8b9bfff640 1 --2- 172.21.15.28:0/56066172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c102230 0x7f8b9c19f030 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8b940305b0 tx=0x7f8b940305e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.387+0000 7f8b98ff9640 1 -- 172.21.15.28:0/56066172 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94041020 con 0x7f8b9c102230 2024-04-25T01:32:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.387+0000 7f8b98ff9640 1 -- 172.21.15.28:0/56066172 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b94037040 con 0x7f8b9c102230 2024-04-25T01:32:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.387+0000 7f8ba1ecd640 1 -- 172.21.15.28:0/56066172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b9c19f930 con 0x7f8b9c102230 2024-04-25T01:32:22.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.387+0000 7f8b98ff9640 1 -- 172.21.15.28:0/56066172 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94036070 con 0x7f8b9c102230 2024-04-25T01:32:22.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.387+0000 7f8ba1ecd640 1 -- 172.21.15.28:0/56066172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b9c19fe20 con 0x7f8b9c102230 2024-04-25T01:32:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.388+0000 7f8ba1ecd640 1 -- 172.21.15.28:0/56066172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b9c1a0050 con 0x7f8b9c102230 2024-04-25T01:32:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.389+0000 7f8b98ff9640 1 -- 172.21.15.28:0/56066172 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8b94044020 con 0x7f8b9c102230 2024-04-25T01:32:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.389+0000 7f8b98ff9640 1 --2- 172.21.15.28:0/56066172 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8b880a2b90 0x7f8b880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.389+0000 7f8b9b7fe640 1 --2- 172.21.15.28:0/56066172 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8b880a2b90 0x7f8b880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.389+0000 7f8b98ff9640 1 -- 172.21.15.28:0/56066172 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f8b940eff80 con 0x7f8b9c102230 2024-04-25T01:32:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.390+0000 7f8b9b7fe640 1 --2- 172.21.15.28:0/56066172 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8b880a2b90 0x7f8b880a5050 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c009530 tx=0x7f8b8c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:22.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.394+0000 7f8b98ff9640 1 -- 172.21.15.28:0/56066172 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8b9404b020 con 0x7f8b9c102230 2024-04-25T01:32:22.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:22 smithi130 ceph-mon[32702]: pgmap v353: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:22.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:22 smithi130 ceph-mon[32702]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-25T01:32:22.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:22 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/466114098' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:22.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:22 smithi032 ceph-mon[32461]: pgmap v353: 129 pgs: 129 active+clean; 645 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:22.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:22 smithi032 ceph-mon[32461]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-25T01:32:22.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:22 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/466114098' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:22.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.579+0000 7f8ba1ecd640 1 -- 172.21.15.28:0/56066172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8b9c1a0220 con 0x7f8b9c102230 2024-04-25T01:32:22.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.579+0000 7f8b98ff9640 1 -- 172.21.15.28:0/56066172 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f8b940bcb80 con 0x7f8b9c102230 2024-04-25T01:32:22.557 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:32:22.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.581+0000 7f8ba1ecd640 1 -- 172.21.15.28:0/56066172 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8b880a2b90 msgr2=0x7f8b880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:22.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.581+0000 7f8ba1ecd640 1 --2- 172.21.15.28:0/56066172 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8b880a2b90 0x7f8b880a5050 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c009530 tx=0x7f8b8c009260 comp rx=0 tx=0).stop 2024-04-25T01:32:22.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.582+0000 7f8ba1ecd640 1 -- 172.21.15.28:0/56066172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c102230 msgr2=0x7f8b9c19f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:22.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.582+0000 7f8ba1ecd640 1 --2- 172.21.15.28:0/56066172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c102230 0x7f8b9c19f030 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8b940305b0 tx=0x7f8b940305e0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.582+0000 7f8ba1ecd640 1 -- 172.21.15.28:0/56066172 shutdown_connections 2024-04-25T01:32:22.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.582+0000 7f8ba1ecd640 1 --2- 172.21.15.28:0/56066172 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8b880a2b90 0x7f8b880a5050 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.582+0000 7f8ba1ecd640 1 --2- 172.21.15.28:0/56066172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b9c102230 0x7f8b9c19f030 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.582+0000 7f8ba1ecd640 1 --2- 172.21.15.28:0/56066172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8b9c101870 0x7f8b9c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.582+0000 7f8ba1ecd640 1 --2- 172.21.15.28:0/56066172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b9c100680 0x7f8b9c197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.582+0000 7f8ba1ecd640 1 -- 172.21.15.28:0/56066172 >> 172.21.15.28:0/56066172 conn(0x7f8b9c0fbbf0 msgr2=0x7f8b9c0688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:22.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.582+0000 7f8ba1ecd640 1 -- 172.21.15.28:0/56066172 shutdown_connections 2024-04-25T01:32:22.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.582+0000 7f8ba1ecd640 1 -- 172.21.15.28:0/56066172 wait complete. 2024-04-25T01:32:22.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:32:22.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.698+0000 7fd9d2472640 1 -- 172.21.15.28:0/4088516141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc1022e0 msgr2=0x7fd9cc1046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:22.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.698+0000 7fd9d2472640 1 --2- 172.21.15.28:0/4088516141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc1022e0 0x7fd9cc1046d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc009900 tx=0x7fd9bc02a410 comp rx=0 tx=0).stop 2024-04-25T01:32:22.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.698+0000 7fd9d2472640 1 -- 172.21.15.28:0/4088516141 shutdown_connections 2024-04-25T01:32:22.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.698+0000 7fd9d2472640 1 --2- 172.21.15.28:0/4088516141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9cc107540 0x7fd9cc109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.698+0000 7fd9d2472640 1 --2- 172.21.15.28:0/4088516141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9cc104c10 0x7fd9cc107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.698+0000 7fd9d2472640 1 --2- 172.21.15.28:0/4088516141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc1022e0 0x7fd9cc1046d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.698+0000 7fd9d2472640 1 -- 172.21.15.28:0/4088516141 >> 172.21.15.28:0/4088516141 conn(0x7fd9cc0fbc30 msgr2=0x7fd9cc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:22.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.698+0000 7fd9d2472640 1 -- 172.21.15.28:0/4088516141 shutdown_connections 2024-04-25T01:32:22.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.698+0000 7fd9d2472640 1 -- 172.21.15.28:0/4088516141 wait complete. 2024-04-25T01:32:22.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.699+0000 7fd9d2472640 1 Processor -- start 2024-04-25T01:32:22.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.699+0000 7fd9d2472640 1 -- start start 2024-04-25T01:32:22.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.699+0000 7fd9d2472640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc1022e0 0x7fd9cc197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:22.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.699+0000 7fd9d2472640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9cc104c10 0x7fd9cc198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:22.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.699+0000 7fd9d2472640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9cc107540 0x7fd9cc19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:22.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.699+0000 7fd9d2472640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc10d450 con 0x7fd9cc1022e0 2024-04-25T01:32:22.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.699+0000 7fd9d2472640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc10d2d0 con 0x7fd9cc104c10 2024-04-25T01:32:22.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.699+0000 7fd9d2472640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc10d5d0 con 0x7fd9cc107540 2024-04-25T01:32:22.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.699+0000 7fd9cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc1022e0 0x7fd9cc197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:22.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.699+0000 7fd9cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc1022e0 0x7fd9cc197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45260/0 (socket says 172.21.15.28:45260) 2024-04-25T01:32:22.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.699+0000 7fd9cbfff640 1 -- 172.21.15.28:0/1997618818 learned_addr learned my addr 172.21.15.28:0/1997618818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:22.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.699+0000 7fd9cb7fe640 1 --2- 172.21.15.28:0/1997618818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9cc104c10 0x7fd9cc198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:22.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.699+0000 7fd9d0a0f640 1 --2- 172.21.15.28:0/1997618818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9cc107540 0x7fd9cc19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:22.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.699+0000 7fd9cbfff640 1 -- 172.21.15.28:0/1997618818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9cc107540 msgr2=0x7fd9cc19f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:22.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.699+0000 7fd9cbfff640 1 --2- 172.21.15.28:0/1997618818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9cc107540 0x7fd9cc19f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.699+0000 7fd9cbfff640 1 -- 172.21.15.28:0/1997618818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9cc104c10 msgr2=0x7fd9cc198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:22.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.699+0000 7fd9cbfff640 1 --2- 172.21.15.28:0/1997618818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9cc104c10 0x7fd9cc198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.699+0000 7fd9cbfff640 1 -- 172.21.15.28:0/1997618818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9cc19f6e0 con 0x7fd9cc1022e0 2024-04-25T01:32:22.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.700+0000 7fd9cbfff640 1 --2- 172.21.15.28:0/1997618818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc1022e0 0x7fd9cc197ac0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc009230 tx=0x7fd9bc004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:22.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.700+0000 7fd9c97fa640 1 -- 172.21.15.28:0/1997618818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc03f030 con 0x7fd9cc1022e0 2024-04-25T01:32:22.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.700+0000 7fd9d2472640 1 -- 172.21.15.28:0/1997618818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9cc19f970 con 0x7fd9cc1022e0 2024-04-25T01:32:22.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.700+0000 7fd9c97fa640 1 -- 172.21.15.28:0/1997618818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9bc003ce0 con 0x7fd9cc1022e0 2024-04-25T01:32:22.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.700+0000 7fd9c97fa640 1 -- 172.21.15.28:0/1997618818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc035070 con 0x7fd9cc1022e0 2024-04-25T01:32:22.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.700+0000 7fd9d2472640 1 -- 172.21.15.28:0/1997618818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9cc19feb0 con 0x7fd9cc1022e0 2024-04-25T01:32:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.701+0000 7fd9d2472640 1 -- 172.21.15.28:0/1997618818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9cc0fd720 con 0x7fd9cc1022e0 2024-04-25T01:32:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.702+0000 7fd9c97fa640 1 -- 172.21.15.28:0/1997618818 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd9bc042030 con 0x7fd9cc1022e0 2024-04-25T01:32:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.703+0000 7fd9c97fa640 1 --2- 172.21.15.28:0/1997618818 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd9b40a2b90 0x7fd9b40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.703+0000 7fd9c97fa640 1 -- 172.21.15.28:0/1997618818 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fd9bc0efdf0 con 0x7fd9cc1022e0 2024-04-25T01:32:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.703+0000 7fd9cb7fe640 1 --2- 172.21.15.28:0/1997618818 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd9b40a2b90 0x7fd9b40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.703+0000 7fd9cb7fe640 1 --2- 172.21.15.28:0/1997618818 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd9b40a2b90 0x7fd9b40a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc198fd0 tx=0x7fd9c000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:22.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.707+0000 7fd9c97fa640 1 -- 172.21.15.28:0/1997618818 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd9bc0bc990 con 0x7fd9cc1022e0 2024-04-25T01:32:22.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.892+0000 7fd9d2472640 1 -- 172.21.15.28:0/1997618818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd9cc062340 con 0x7fd9cc1022e0 2024-04-25T01:32:22.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.893+0000 7fd9c97fa640 1 -- 172.21.15.28:0/1997618818 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fd9bc0bc290 con 0x7fd9cc1022e0 2024-04-25T01:32:22.870 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T01:32:22.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.894+0000 7fd9d2472640 1 -- 172.21.15.28:0/1997618818 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd9b40a2b90 msgr2=0x7fd9b40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:22.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.894+0000 7fd9d2472640 1 --2- 172.21.15.28:0/1997618818 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd9b40a2b90 0x7fd9b40a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc198fd0 tx=0x7fd9c000b390 comp rx=0 tx=0).stop 2024-04-25T01:32:22.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.894+0000 7fd9d2472640 1 -- 172.21.15.28:0/1997618818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc1022e0 msgr2=0x7fd9cc197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:22.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.894+0000 7fd9d2472640 1 --2- 172.21.15.28:0/1997618818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc1022e0 0x7fd9cc197ac0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc009230 tx=0x7fd9bc004360 comp rx=0 tx=0).stop 2024-04-25T01:32:22.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.895+0000 7fd9d2472640 1 -- 172.21.15.28:0/1997618818 shutdown_connections 2024-04-25T01:32:22.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.895+0000 7fd9d2472640 1 --2- 172.21.15.28:0/1997618818 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fd9b40a2b90 0x7fd9b40a5050 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.895+0000 7fd9d2472640 1 --2- 172.21.15.28:0/1997618818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd9cc107540 0x7fd9cc19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.895+0000 7fd9d2472640 1 --2- 172.21.15.28:0/1997618818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd9cc104c10 0x7fd9cc198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.895+0000 7fd9d2472640 1 --2- 172.21.15.28:0/1997618818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9cc1022e0 0x7fd9cc197ac0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.895+0000 7fd9d2472640 1 -- 172.21.15.28:0/1997618818 >> 172.21.15.28:0/1997618818 conn(0x7fd9cc0fbc30 msgr2=0x7fd9cc102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:22.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.895+0000 7fd9d2472640 1 -- 172.21.15.28:0/1997618818 shutdown_connections 2024-04-25T01:32:22.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:22.895+0000 7fd9d2472640 1 -- 172.21.15.28:0/1997618818 wait complete. 2024-04-25T01:32:22.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:32:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.011+0000 7f67ee54f640 1 -- 172.21.15.28:0/3427703927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e8101870 msgr2=0x7f67e8101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.011+0000 7f67ee54f640 1 --2- 172.21.15.28:0/3427703927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e8101870 0x7f67e8101cf0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f67dc009900 tx=0x7f67dc02a410 comp rx=0 tx=0).stop 2024-04-25T01:32:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.011+0000 7f67ee54f640 1 -- 172.21.15.28:0/3427703927 shutdown_connections 2024-04-25T01:32:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.011+0000 7f67ee54f640 1 --2- 172.21.15.28:0/3427703927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67e8102230 0x7f67e8108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.011+0000 7f67ee54f640 1 --2- 172.21.15.28:0/3427703927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e8101870 0x7f67e8101cf0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.011+0000 7f67ee54f640 1 --2- 172.21.15.28:0/3427703927 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67e8100680 0x7f67e8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.011+0000 7f67ee54f640 1 -- 172.21.15.28:0/3427703927 >> 172.21.15.28:0/3427703927 conn(0x7f67e80fbbf0 msgr2=0x7f67e80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.011+0000 7f67ee54f640 1 -- 172.21.15.28:0/3427703927 shutdown_connections 2024-04-25T01:32:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.011+0000 7f67ee54f640 1 -- 172.21.15.28:0/3427703927 wait complete. 2024-04-25T01:32:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.012+0000 7f67ee54f640 1 Processor -- start 2024-04-25T01:32:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.012+0000 7f67ee54f640 1 -- start start 2024-04-25T01:32:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.012+0000 7f67ee54f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67e8100680 0x7f67e8197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.012+0000 7f67ee54f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e8101870 0x7f67e81980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.012+0000 7f67ee54f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67e8102230 0x7f67e819f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.012+0000 7f67ee54f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f67e810d480 con 0x7f67e8101870 2024-04-25T01:32:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.012+0000 7f67ee54f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67e810d300 con 0x7f67e8100680 2024-04-25T01:32:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.012+0000 7f67ee54f640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f67e810d600 con 0x7f67e8102230 2024-04-25T01:32:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.012+0000 7f67e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e8101870 0x7f67e81980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.012+0000 7f67e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e8101870 0x7f67e81980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45278/0 (socket says 172.21.15.28:45278) 2024-04-25T01:32:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.013+0000 7f67e77fe640 1 -- 172.21.15.28:0/3899692487 learned_addr learned my addr 172.21.15.28:0/3899692487 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.013+0000 7f67ecaec640 1 --2- 172.21.15.28:0/3899692487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67e8102230 0x7f67e819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.013+0000 7f67e7fff640 1 --2- 172.21.15.28:0/3899692487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67e8100680 0x7f67e8197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.013+0000 7f67e77fe640 1 -- 172.21.15.28:0/3899692487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67e8102230 msgr2=0x7f67e819f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.013+0000 7f67e77fe640 1 --2- 172.21.15.28:0/3899692487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67e8102230 0x7f67e819f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.013+0000 7f67e77fe640 1 -- 172.21.15.28:0/3899692487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67e8100680 msgr2=0x7f67e8197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.013+0000 7f67e77fe640 1 --2- 172.21.15.28:0/3899692487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67e8100680 0x7f67e8197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.013+0000 7f67e77fe640 1 -- 172.21.15.28:0/3899692487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e819f780 con 0x7f67e8101870 2024-04-25T01:32:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.013+0000 7f67e7fff640 1 --2- 172.21.15.28:0/3899692487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67e8100680 0x7f67e8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.013+0000 7f67e77fe640 1 --2- 172.21.15.28:0/3899692487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e8101870 0x7f67e81980a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f67dc009230 tx=0x7f67dc004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.013+0000 7f67e57fa640 1 -- 172.21.15.28:0/3899692487 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc03f030 con 0x7f67e8101870 2024-04-25T01:32:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.013+0000 7f67ee54f640 1 -- 172.21.15.28:0/3899692487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67e819fa10 con 0x7f67e8101870 2024-04-25T01:32:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.013+0000 7f67e57fa640 1 -- 172.21.15.28:0/3899692487 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67dc003ce0 con 0x7f67e8101870 2024-04-25T01:32:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.013+0000 7f67e57fa640 1 -- 172.21.15.28:0/3899692487 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc035070 con 0x7f67e8101870 2024-04-25T01:32:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.013+0000 7f67ee54f640 1 -- 172.21.15.28:0/3899692487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67e819ffd0 con 0x7f67e8101870 2024-04-25T01:32:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.014+0000 7f67ee54f640 1 -- 172.21.15.28:0/3899692487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67e81079c0 con 0x7f67e8101870 2024-04-25T01:32:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.015+0000 7f67e57fa640 1 -- 172.21.15.28:0/3899692487 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f67dc042030 con 0x7f67e8101870 2024-04-25T01:32:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.016+0000 7f67e57fa640 1 --2- 172.21.15.28:0/3899692487 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67d00a2b90 0x7f67d00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.016+0000 7f67e7fff640 1 --2- 172.21.15.28:0/3899692487 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67d00a2b90 0x7f67d00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.016+0000 7f67e57fa640 1 -- 172.21.15.28:0/3899692487 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f67dc0efdc0 con 0x7f67e8101870 2024-04-25T01:32:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.016+0000 7f67e7fff640 1 --2- 172.21.15.28:0/3899692487 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67d00a2b90 0x7f67d00a5050 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f67d8006fd0 tx=0x7f67d8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:22.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.020+0000 7f67e57fa640 1 -- 172.21.15.28:0/3899692487 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f67dc0bc9c0 con 0x7f67e8101870 2024-04-25T01:32:23.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.206+0000 7f67ee54f640 1 -- 172.21.15.28:0/3899692487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f67e8062340 con 0x7f67e8101870 2024-04-25T01:32:23.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.206+0000 7f67e57fa640 1 -- 172.21.15.28:0/3899692487 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f67dc0bc2c0 con 0x7f67e8101870 2024-04-25T01:32:23.184 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:32:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.208+0000 7f67ee54f640 1 -- 172.21.15.28:0/3899692487 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67d00a2b90 msgr2=0x7f67d00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.208+0000 7f67ee54f640 1 --2- 172.21.15.28:0/3899692487 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67d00a2b90 0x7f67d00a5050 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f67d8006fd0 tx=0x7f67d8008040 comp rx=0 tx=0).stop 2024-04-25T01:32:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.208+0000 7f67ee54f640 1 -- 172.21.15.28:0/3899692487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e8101870 msgr2=0x7f67e81980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.208+0000 7f67ee54f640 1 --2- 172.21.15.28:0/3899692487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e8101870 0x7f67e81980a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f67dc009230 tx=0x7f67dc004360 comp rx=0 tx=0).stop 2024-04-25T01:32:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.208+0000 7f67ee54f640 1 -- 172.21.15.28:0/3899692487 shutdown_connections 2024-04-25T01:32:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.208+0000 7f67ee54f640 1 --2- 172.21.15.28:0/3899692487 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67d00a2b90 0x7f67d00a5050 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.208+0000 7f67ee54f640 1 --2- 172.21.15.28:0/3899692487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67e8102230 0x7f67e819f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.208+0000 7f67ee54f640 1 --2- 172.21.15.28:0/3899692487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e8101870 0x7f67e81980a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.208+0000 7f67ee54f640 1 --2- 172.21.15.28:0/3899692487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67e8100680 0x7f67e8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.208+0000 7f67ee54f640 1 -- 172.21.15.28:0/3899692487 >> 172.21.15.28:0/3899692487 conn(0x7f67e80fbbf0 msgr2=0x7f67e8104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.208+0000 7f67ee54f640 1 -- 172.21.15.28:0/3899692487 shutdown_connections 2024-04-25T01:32:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.208+0000 7f67ee54f640 1 -- 172.21.15.28:0/3899692487 wait complete. 2024-04-25T01:32:23.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:23 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/4057367025' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:32:23.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:23 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/56066172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:32:23.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:23 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1997618818' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:32:23.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:32:23.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.321+0000 7f2ca4dc1640 1 -- 172.21.15.28:0/3955123816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca01006a0 msgr2=0x7f2ca0100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:23.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.321+0000 7f2ca4dc1640 1 --2- 172.21.15.28:0/3955123816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca01006a0 0x7f2ca0100ac0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f2c90009ca0 tx=0x7f2c9002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.321+0000 7f2ca4dc1640 1 -- 172.21.15.28:0/3955123816 shutdown_connections 2024-04-25T01:32:23.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.321+0000 7f2ca4dc1640 1 --2- 172.21.15.28:0/3955123816 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ca0102250 0x7f2ca0108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.321+0000 7f2ca4dc1640 1 --2- 172.21.15.28:0/3955123816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca0101890 0x7f2ca0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.321+0000 7f2ca4dc1640 1 --2- 172.21.15.28:0/3955123816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca01006a0 0x7f2ca0100ac0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.321+0000 7f2ca4dc1640 1 -- 172.21.15.28:0/3955123816 >> 172.21.15.28:0/3955123816 conn(0x7f2ca00fbc30 msgr2=0x7f2ca00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.321+0000 7f2ca4dc1640 1 -- 172.21.15.28:0/3955123816 shutdown_connections 2024-04-25T01:32:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.321+0000 7f2ca4dc1640 1 -- 172.21.15.28:0/3955123816 wait complete. 2024-04-25T01:32:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.322+0000 7f2ca4dc1640 1 Processor -- start 2024-04-25T01:32:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.322+0000 7f2ca4dc1640 1 -- start start 2024-04-25T01:32:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.322+0000 7f2ca4dc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca01006a0 0x7f2ca0197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.322+0000 7f2ca4dc1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca0101890 0x7f2ca0198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.322+0000 7f2ca4dc1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ca0102250 0x7f2ca019f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.322+0000 7f2ca4dc1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca010c2f0 con 0x7f2ca01006a0 2024-04-25T01:32:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.322+0000 7f2ca4dc1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca010c170 con 0x7f2ca0101890 2024-04-25T01:32:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.322+0000 7f2ca4dc1640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca010c470 con 0x7f2ca0102250 2024-04-25T01:32:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.322+0000 7f2c9e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca01006a0 0x7f2ca0197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:23.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.322+0000 7f2c9e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca01006a0 0x7f2ca0197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45292/0 (socket says 172.21.15.28:45292) 2024-04-25T01:32:23.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.322+0000 7f2c9e59c640 1 -- 172.21.15.28:0/3748914429 learned_addr learned my addr 172.21.15.28:0/3748914429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:23.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.323+0000 7f2c9ed9d640 1 --2- 172.21.15.28:0/3748914429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ca0102250 0x7f2ca019f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:23.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.323+0000 7f2c9dd9b640 1 --2- 172.21.15.28:0/3748914429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca0101890 0x7f2ca0198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:23.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.323+0000 7f2c9e59c640 1 -- 172.21.15.28:0/3748914429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ca0102250 msgr2=0x7f2ca019f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:23.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.323+0000 7f2c9e59c640 1 --2- 172.21.15.28:0/3748914429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ca0102250 0x7f2ca019f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.323+0000 7f2c9e59c640 1 -- 172.21.15.28:0/3748914429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca0101890 msgr2=0x7f2ca0198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:23.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.323+0000 7f2c9e59c640 1 --2- 172.21.15.28:0/3748914429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca0101890 0x7f2ca0198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.323+0000 7f2c9e59c640 1 -- 172.21.15.28:0/3748914429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca019f760 con 0x7f2ca01006a0 2024-04-25T01:32:23.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.323+0000 7f2c9e59c640 1 --2- 172.21.15.28:0/3748914429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca01006a0 0x7f2ca0197b40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2c90004af0 tx=0x7f2c90031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:23.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.323+0000 7f2c977fe640 1 -- 172.21.15.28:0/3748914429 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9002fb60 con 0x7f2ca01006a0 2024-04-25T01:32:23.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.323+0000 7f2ca4dc1640 1 -- 172.21.15.28:0/3748914429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ca019f9f0 con 0x7f2ca01006a0 2024-04-25T01:32:23.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.323+0000 7f2c977fe640 1 -- 172.21.15.28:0/3748914429 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c9002fd00 con 0x7f2ca01006a0 2024-04-25T01:32:23.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.323+0000 7f2ca4dc1640 1 -- 172.21.15.28:0/3748914429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ca019ff90 con 0x7f2ca01006a0 2024-04-25T01:32:23.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.323+0000 7f2c977fe640 1 -- 172.21.15.28:0/3748914429 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c90046920 con 0x7f2ca01006a0 2024-04-25T01:32:23.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.324+0000 7f2ca4dc1640 1 -- 172.21.15.28:0/3748914429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ca01a0220 con 0x7f2ca01006a0 2024-04-25T01:32:23.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.325+0000 7f2c977fe640 1 -- 172.21.15.28:0/3748914429 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2c90030c30 con 0x7f2ca01006a0 2024-04-25T01:32:23.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.326+0000 7f2c977fe640 1 --2- 172.21.15.28:0/3748914429 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f2c8c0a2b90 0x7f2c8c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:23.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.326+0000 7f2c977fe640 1 -- 172.21.15.28:0/3748914429 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f2c900f0920 con 0x7f2ca01006a0 2024-04-25T01:32:23.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.326+0000 7f2c9dd9b640 1 --2- 172.21.15.28:0/3748914429 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f2c8c0a2b90 0x7f2c8c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:23.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.326+0000 7f2c9dd9b640 1 --2- 172.21.15.28:0/3748914429 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f2c8c0a2b90 0x7f2c8c0a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0199050 tx=0x7f2c88005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:23.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.330+0000 7f2c977fe640 1 -- 172.21.15.28:0/3748914429 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2c900bd520 con 0x7f2ca01006a0 2024-04-25T01:32:23.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:23 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/4057367025' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:32:23.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:23 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/56066172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:32:23.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:23 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1997618818' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:32:23.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:23 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/4057367025' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:32:23.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:23 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/56066172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:32:23.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:23 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1997618818' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:32:23.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.559+0000 7f2ca4dc1640 1 -- 172.21.15.28:0/3748914429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ca01a03f0 con 0x7f2ca01006a0 2024-04-25T01:32:23.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.559+0000 7f2c977fe640 1 -- 172.21.15.28:0/3748914429 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11923 (secure 0 0 0) 0x7f2c900bce20 con 0x7f2ca01006a0 2024-04-25T01:32:23.537 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:32:23.537 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:31:01.802718+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24454,24442,24512,24478,24473]},"id":1}]} 2024-04-25T01:32:23.537 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-04-25T01:32:23.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.562+0000 7f2ca4dc1640 1 -- 172.21.15.28:0/3748914429 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f2c8c0a2b90 msgr2=0x7f2c8c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:23.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.562+0000 7f2ca4dc1640 1 --2- 172.21.15.28:0/3748914429 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f2c8c0a2b90 0x7f2c8c0a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0199050 tx=0x7f2c88005c60 comp rx=0 tx=0).stop 2024-04-25T01:32:23.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.562+0000 7f2ca4dc1640 1 -- 172.21.15.28:0/3748914429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca01006a0 msgr2=0x7f2ca0197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:23.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.562+0000 7f2ca4dc1640 1 --2- 172.21.15.28:0/3748914429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca01006a0 0x7f2ca0197b40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2c90004af0 tx=0x7f2c90031ee0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.562+0000 7f2ca4dc1640 1 -- 172.21.15.28:0/3748914429 shutdown_connections 2024-04-25T01:32:23.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.562+0000 7f2ca4dc1640 1 --2- 172.21.15.28:0/3748914429 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f2c8c0a2b90 0x7f2c8c0a5050 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.562+0000 7f2ca4dc1640 1 --2- 172.21.15.28:0/3748914429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ca0102250 0x7f2ca019f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.562+0000 7f2ca4dc1640 1 --2- 172.21.15.28:0/3748914429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ca0101890 0x7f2ca0198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.562+0000 7f2ca4dc1640 1 --2- 172.21.15.28:0/3748914429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca01006a0 0x7f2ca0197b40 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.562+0000 7f2ca4dc1640 1 -- 172.21.15.28:0/3748914429 >> 172.21.15.28:0/3748914429 conn(0x7f2ca00fbc30 msgr2=0x7f2ca00fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:23.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.562+0000 7f2ca4dc1640 1 -- 172.21.15.28:0/3748914429 shutdown_connections 2024-04-25T01:32:23.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.562+0000 7f2ca4dc1640 1 -- 172.21.15.28:0/3748914429 wait complete. 2024-04-25T01:32:23.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:32:23.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.675+0000 7f90707d5640 1 -- 172.21.15.28:0/2320546029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9068103360 msgr2=0x7f9068109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:23.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.675+0000 7f90707d5640 1 --2- 172.21.15.28:0/2320546029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9068103360 0x7f9068109bf0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9060009ca0 tx=0x7f906002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.675+0000 7f90707d5640 1 -- 172.21.15.28:0/2320546029 shutdown_connections 2024-04-25T01:32:23.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.675+0000 7f90707d5640 1 --2- 172.21.15.28:0/2320546029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9068103360 0x7f9068109bf0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.675+0000 7f90707d5640 1 --2- 172.21.15.28:0/2320546029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90681029a0 0x7f9068102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.675+0000 7f90707d5640 1 --2- 172.21.15.28:0/2320546029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f90681017b0 0x7f9068101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.675+0000 7f90707d5640 1 -- 172.21.15.28:0/2320546029 >> 172.21.15.28:0/2320546029 conn(0x7f90680fcd40 msgr2=0x7f90680ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.675+0000 7f90707d5640 1 -- 172.21.15.28:0/2320546029 shutdown_connections 2024-04-25T01:32:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.675+0000 7f90707d5640 1 -- 172.21.15.28:0/2320546029 wait complete. 2024-04-25T01:32:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.675+0000 7f90707d5640 1 Processor -- start 2024-04-25T01:32:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.675+0000 7f90707d5640 1 -- start start 2024-04-25T01:32:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.676+0000 7f90707d5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90681017b0 0x7f9068197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.676+0000 7f90707d5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f90681029a0 0x7f9068197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.676+0000 7f90707d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9068103360 0x7f906819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.676+0000 7f90707d5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f906810d480 con 0x7f9068103360 2024-04-25T01:32:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.676+0000 7f90707d5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f906810d300 con 0x7f90681017b0 2024-04-25T01:32:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.676+0000 7f90707d5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f906810d600 con 0x7f90681029a0 2024-04-25T01:32:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.676+0000 7f906ed72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9068103360 0x7f906819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.676+0000 7f906ed72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9068103360 0x7f906819f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45308/0 (socket says 172.21.15.28:45308) 2024-04-25T01:32:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.676+0000 7f906ed72640 1 -- 172.21.15.28:0/3482601432 learned_addr learned my addr 172.21.15.28:0/3482601432 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.676+0000 7f906e571640 1 --2- 172.21.15.28:0/3482601432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90681017b0 0x7f9068197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.676+0000 7f906ed72640 1 -- 172.21.15.28:0/3482601432 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f90681029a0 msgr2=0x7f9068197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.676+0000 7f906dd70640 1 --2- 172.21.15.28:0/3482601432 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f90681029a0 0x7f9068197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.676+0000 7f906ed72640 1 --2- 172.21.15.28:0/3482601432 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f90681029a0 0x7f9068197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.676+0000 7f906ed72640 1 -- 172.21.15.28:0/3482601432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90681017b0 msgr2=0x7f9068197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.676+0000 7f906ed72640 1 --2- 172.21.15.28:0/3482601432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90681017b0 0x7f9068197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.676+0000 7f906ed72640 1 -- 172.21.15.28:0/3482601432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f906819f680 con 0x7f9068103360 2024-04-25T01:32:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.676+0000 7f906ed72640 1 --2- 172.21.15.28:0/3482601432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9068103360 0x7f906819f010 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f90600319f0 tx=0x7f906002f450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.676+0000 7f905f7fe640 1 -- 172.21.15.28:0/3482601432 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9060003c70 con 0x7f9068103360 2024-04-25T01:32:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.676+0000 7f905f7fe640 1 -- 172.21.15.28:0/3482601432 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9060003e10 con 0x7f9068103360 2024-04-25T01:32:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.677+0000 7f905f7fe640 1 -- 172.21.15.28:0/3482601432 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906002fd30 con 0x7f9068103360 2024-04-25T01:32:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.677+0000 7f90707d5640 1 -- 172.21.15.28:0/3482601432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f906819f910 con 0x7f9068103360 2024-04-25T01:32:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.677+0000 7f90707d5640 1 -- 172.21.15.28:0/3482601432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f906819fe60 con 0x7f9068103360 2024-04-25T01:32:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.678+0000 7f90707d5640 1 -- 172.21.15.28:0/3482601432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9068108a00 con 0x7f9068103360 2024-04-25T01:32:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.679+0000 7f905f7fe640 1 -- 172.21.15.28:0/3482601432 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9060004040 con 0x7f9068103360 2024-04-25T01:32:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.679+0000 7f905f7fe640 1 --2- 172.21.15.28:0/3482601432 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f90540a2b90 0x7f90540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.680+0000 7f906e571640 1 --2- 172.21.15.28:0/3482601432 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f90540a2b90 0x7f90540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.680+0000 7f905f7fe640 1 -- 172.21.15.28:0/3482601432 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f90600f0b30 con 0x7f9068103360 2024-04-25T01:32:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.680+0000 7f906e571640 1 --2- 172.21.15.28:0/3482601432 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f90540a2b90 0x7f90540a5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9064005fd0 tx=0x7f906400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.684+0000 7f905f7fe640 1 -- 172.21.15.28:0/3482601432 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f90600bd730 con 0x7f9068103360 2024-04-25T01:32:23.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.869+0000 7f90707d5640 1 -- 172.21.15.28:0/3482601432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f906802d390 con 0x7f9068103360 2024-04-25T01:32:23.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.870+0000 7f905f7fe640 1 -- 172.21.15.28:0/3482601432 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24564 (secure 0 0 0) 0x7f90600bd030 con 0x7f9068103360 2024-04-25T01:32:23.848 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:32:23.848 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:31:21.857121+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:32:23.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.873+0000 7f90707d5640 1 -- 172.21.15.28:0/3482601432 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f90540a2b90 msgr2=0x7f90540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:23.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.873+0000 7f90707d5640 1 --2- 172.21.15.28:0/3482601432 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f90540a2b90 0x7f90540a5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9064005fd0 tx=0x7f906400b390 comp rx=0 tx=0).stop 2024-04-25T01:32:23.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.873+0000 7f90707d5640 1 -- 172.21.15.28:0/3482601432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9068103360 msgr2=0x7f906819f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:23.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.873+0000 7f90707d5640 1 --2- 172.21.15.28:0/3482601432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9068103360 0x7f906819f010 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f90600319f0 tx=0x7f906002f450 comp rx=0 tx=0).stop 2024-04-25T01:32:23.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.873+0000 7f90707d5640 1 -- 172.21.15.28:0/3482601432 shutdown_connections 2024-04-25T01:32:23.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.873+0000 7f90707d5640 1 --2- 172.21.15.28:0/3482601432 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f90540a2b90 0x7f90540a5050 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.873+0000 7f90707d5640 1 --2- 172.21.15.28:0/3482601432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9068103360 0x7f906819f010 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.873+0000 7f90707d5640 1 --2- 172.21.15.28:0/3482601432 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f90681029a0 0x7f9068197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.873+0000 7f90707d5640 1 --2- 172.21.15.28:0/3482601432 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f90681017b0 0x7f9068197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.873+0000 7f90707d5640 1 -- 172.21.15.28:0/3482601432 >> 172.21.15.28:0/3482601432 conn(0x7f90680fcd40 msgr2=0x7f9068105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:23.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.873+0000 7f90707d5640 1 -- 172.21.15.28:0/3482601432 shutdown_connections 2024-04-25T01:32:23.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.873+0000 7f90707d5640 1 -- 172.21.15.28:0/3482601432 wait complete. 2024-04-25T01:32:23.862 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:25:48.767414+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:30:32.282430+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:30:33.127049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:30:36.796132+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:32:23.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:32:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.990+0000 7f83d57ee640 1 -- 172.21.15.28:0/380245082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d0100680 msgr2=0x7f83d0100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.990+0000 7f83d57ee640 1 --2- 172.21.15.28:0/380245082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d0100680 0x7f83d0100aa0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f83c0009900 tx=0x7f83c002a410 comp rx=0 tx=0).stop 2024-04-25T01:32:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.990+0000 7f83d57ee640 1 -- 172.21.15.28:0/380245082 shutdown_connections 2024-04-25T01:32:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.990+0000 7f83d57ee640 1 --2- 172.21.15.28:0/380245082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83d0102230 0x7f83d0108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.990+0000 7f83d57ee640 1 --2- 172.21.15.28:0/380245082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83d0101870 0x7f83d0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.990+0000 7f83d57ee640 1 --2- 172.21.15.28:0/380245082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d0100680 0x7f83d0100aa0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.990+0000 7f83d57ee640 1 -- 172.21.15.28:0/380245082 >> 172.21.15.28:0/380245082 conn(0x7f83d00fbbf0 msgr2=0x7f83d00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.990+0000 7f83d57ee640 1 -- 172.21.15.28:0/380245082 shutdown_connections 2024-04-25T01:32:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.990+0000 7f83d57ee640 1 -- 172.21.15.28:0/380245082 wait complete. 2024-04-25T01:32:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.991+0000 7f83d57ee640 1 Processor -- start 2024-04-25T01:32:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.991+0000 7f83d57ee640 1 -- start start 2024-04-25T01:32:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.991+0000 7f83d57ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d0100680 0x7f83d0197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.991+0000 7f83d57ee640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83d0101870 0x7f83d0198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.991+0000 7f83d57ee640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83d0102230 0x7f83d019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.991+0000 7f83d57ee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83d010c300 con 0x7f83d0100680 2024-04-25T01:32:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.991+0000 7f83d57ee640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f83d010c180 con 0x7f83d0102230 2024-04-25T01:32:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.991+0000 7f83d57ee640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f83d010c480 con 0x7f83d0101870 2024-04-25T01:32:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.991+0000 7f83ceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d0100680 0x7f83d0197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.991+0000 7f83ceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d0100680 0x7f83d0197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45338/0 (socket says 172.21.15.28:45338) 2024-04-25T01:32:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.991+0000 7f83ceffd640 1 -- 172.21.15.28:0/2886081701 learned_addr learned my addr 172.21.15.28:0/2886081701 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.991+0000 7f83cf7fe640 1 --2- 172.21.15.28:0/2886081701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83d0102230 0x7f83d019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.991+0000 7f83c7fff640 1 --2- 172.21.15.28:0/2886081701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83d0101870 0x7f83d0198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.991+0000 7f83ceffd640 1 -- 172.21.15.28:0/2886081701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83d0101870 msgr2=0x7f83d0198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.991+0000 7f83ceffd640 1 --2- 172.21.15.28:0/2886081701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83d0101870 0x7f83d0198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.991+0000 7f83ceffd640 1 -- 172.21.15.28:0/2886081701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83d0102230 msgr2=0x7f83d019f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.991+0000 7f83ceffd640 1 --2- 172.21.15.28:0/2886081701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83d0102230 0x7f83d019f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.991+0000 7f83ceffd640 1 -- 172.21.15.28:0/2886081701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d019f720 con 0x7f83d0100680 2024-04-25T01:32:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.992+0000 7f83c7fff640 1 --2- 172.21.15.28:0/2886081701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83d0101870 0x7f83d0198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.992+0000 7f83cf7fe640 1 --2- 172.21.15.28:0/2886081701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83d0102230 0x7f83d019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.992+0000 7f83ceffd640 1 --2- 172.21.15.28:0/2886081701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d0100680 0x7f83d0197b00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f83c0031610 tx=0x7f83c00043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.992+0000 7f83ccff9640 1 -- 172.21.15.28:0/2886081701 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c003f030 con 0x7f83d0100680 2024-04-25T01:32:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.992+0000 7f83ccff9640 1 -- 172.21.15.28:0/2886081701 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83c0003ce0 con 0x7f83d0100680 2024-04-25T01:32:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.992+0000 7f83d57ee640 1 -- 172.21.15.28:0/2886081701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d019f9b0 con 0x7f83d0100680 2024-04-25T01:32:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.992+0000 7f83ccff9640 1 -- 172.21.15.28:0/2886081701 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c0035070 con 0x7f83d0100680 2024-04-25T01:32:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.992+0000 7f83d57ee640 1 -- 172.21.15.28:0/2886081701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83d019fea0 con 0x7f83d0100680 2024-04-25T01:32:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.993+0000 7f83d57ee640 1 -- 172.21.15.28:0/2886081701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83d01a0070 con 0x7f83d0100680 2024-04-25T01:32:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.994+0000 7f83ccff9640 1 -- 172.21.15.28:0/2886081701 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f83c0042030 con 0x7f83d0100680 2024-04-25T01:32:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.995+0000 7f83ccff9640 1 --2- 172.21.15.28:0/2886081701 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f83bc0a2b90 0x7f83bc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.995+0000 7f83c7fff640 1 --2- 172.21.15.28:0/2886081701 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f83bc0a2b90 0x7f83bc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.995+0000 7f83ccff9640 1 -- 172.21.15.28:0/2886081701 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f83c00efa50 con 0x7f83d0100680 2024-04-25T01:32:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.995+0000 7f83c7fff640 1 --2- 172.21.15.28:0/2886081701 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f83bc0a2b90 0x7f83bc0a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f83d0199010 tx=0x7f83b8005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:23.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:23.999+0000 7f83ccff9640 1 -- 172.21.15.28:0/2886081701 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f83c00bc5f0 con 0x7f83d0100680 2024-04-25T01:32:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.188+0000 7f83d57ee640 1 -- 172.21.15.28:0/2886081701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f83d01a0400 con 0x7f83d0100680 2024-04-25T01:32:24.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.188+0000 7f83ccff9640 1 -- 172.21.15.28:0/2886081701 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f83c00bbef0 con 0x7f83d0100680 2024-04-25T01:32:24.166 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T01:32:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.190+0000 7f83d57ee640 1 -- 172.21.15.28:0/2886081701 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f83bc0a2b90 msgr2=0x7f83bc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.190+0000 7f83d57ee640 1 --2- 172.21.15.28:0/2886081701 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f83bc0a2b90 0x7f83bc0a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f83d0199010 tx=0x7f83b8005d30 comp rx=0 tx=0).stop 2024-04-25T01:32:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.190+0000 7f83d57ee640 1 -- 172.21.15.28:0/2886081701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d0100680 msgr2=0x7f83d0197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:24.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.190+0000 7f83d57ee640 1 --2- 172.21.15.28:0/2886081701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d0100680 0x7f83d0197b00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f83c0031610 tx=0x7f83c00043b0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.191+0000 7f83d57ee640 1 -- 172.21.15.28:0/2886081701 shutdown_connections 2024-04-25T01:32:24.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.191+0000 7f83d57ee640 1 --2- 172.21.15.28:0/2886081701 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f83bc0a2b90 0x7f83bc0a5050 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.191+0000 7f83d57ee640 1 --2- 172.21.15.28:0/2886081701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f83d0102230 0x7f83d019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.191+0000 7f83d57ee640 1 --2- 172.21.15.28:0/2886081701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f83d0101870 0x7f83d0198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.191+0000 7f83d57ee640 1 --2- 172.21.15.28:0/2886081701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83d0100680 0x7f83d0197b00 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.191+0000 7f83d57ee640 1 -- 172.21.15.28:0/2886081701 >> 172.21.15.28:0/2886081701 conn(0x7f83d00fbbf0 msgr2=0x7f83d00fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:24.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.191+0000 7f83d57ee640 1 -- 172.21.15.28:0/2886081701 shutdown_connections 2024-04-25T01:32:24.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.191+0000 7f83d57ee640 1 -- 172.21.15.28:0/2886081701 wait complete. 2024-04-25T01:32:24.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:24 smithi028 ceph-mon[29390]: pgmap v354: 129 pgs: 129 active+clean; 645 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:32:24.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:24 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3899692487' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:32:24.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:24 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3748914429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:24.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:24 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3482601432' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:24.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:32:24.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.307+0000 7f5868716640 1 -- 172.21.15.28:0/1787932666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5860102240 msgr2=0x7f5860108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.307+0000 7f5868716640 1 --2- 172.21.15.28:0/1787932666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5860102240 0x7f5860108ad0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5858009ca0 tx=0x7f585802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.307+0000 7f5868716640 1 -- 172.21.15.28:0/1787932666 shutdown_connections 2024-04-25T01:32:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.307+0000 7f5868716640 1 --2- 172.21.15.28:0/1787932666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5860102240 0x7f5860108ad0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.307+0000 7f5868716640 1 --2- 172.21.15.28:0/1787932666 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5860101880 0x7f5860101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.307+0000 7f5868716640 1 --2- 172.21.15.28:0/1787932666 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5860100690 0x7f5860100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.307+0000 7f5868716640 1 -- 172.21.15.28:0/1787932666 >> 172.21.15.28:0/1787932666 conn(0x7f58600fbc00 msgr2=0x7f58600fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.307+0000 7f5868716640 1 -- 172.21.15.28:0/1787932666 shutdown_connections 2024-04-25T01:32:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.307+0000 7f5868716640 1 -- 172.21.15.28:0/1787932666 wait complete. 2024-04-25T01:32:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.307+0000 7f5868716640 1 Processor -- start 2024-04-25T01:32:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.307+0000 7f5868716640 1 -- start start 2024-04-25T01:32:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f5868716640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5860100690 0x7f586006a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f5868716640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5860101880 0x7f586019c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f5868716640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5860102240 0x7f586019ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f5868716640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f586010c310 con 0x7f5860100690 2024-04-25T01:32:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f5868716640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f586010c190 con 0x7f5860102240 2024-04-25T01:32:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f5868716640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f586010c490 con 0x7f5860101880 2024-04-25T01:32:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f58664b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5860100690 0x7f586006a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f58664b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5860100690 0x7f586006a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45362/0 (socket says 172.21.15.28:45362) 2024-04-25T01:32:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f58664b2640 1 -- 172.21.15.28:0/3900092904 learned_addr learned my addr 172.21.15.28:0/3900092904 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f5866cb3640 1 --2- 172.21.15.28:0/3900092904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5860102240 0x7f586019ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f5865cb1640 1 --2- 172.21.15.28:0/3900092904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5860101880 0x7f586019c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f58664b2640 1 -- 172.21.15.28:0/3900092904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5860101880 msgr2=0x7f586019c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f58664b2640 1 --2- 172.21.15.28:0/3900092904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5860101880 0x7f586019c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f58664b2640 1 -- 172.21.15.28:0/3900092904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5860102240 msgr2=0x7f586019ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f58664b2640 1 --2- 172.21.15.28:0/3900092904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5860102240 0x7f586019ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f58664b2640 1 -- 172.21.15.28:0/3900092904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f586019f6c0 con 0x7f5860100690 2024-04-25T01:32:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f5865cb1640 1 --2- 172.21.15.28:0/3900092904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5860101880 0x7f586019c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f58664b2640 1 --2- 172.21.15.28:0/3900092904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5860100690 0x7f586006a530 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f585c00d950 tx=0x7f585c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f58577fe640 1 -- 172.21.15.28:0/3900092904 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c017020 con 0x7f5860100690 2024-04-25T01:32:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f58577fe640 1 -- 172.21.15.28:0/3900092904 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f585c009c30 con 0x7f5860100690 2024-04-25T01:32:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f58577fe640 1 -- 172.21.15.28:0/3900092904 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c00f740 con 0x7f5860100690 2024-04-25T01:32:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.308+0000 7f5868716640 1 -- 172.21.15.28:0/3900092904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f586019f9b0 con 0x7f5860100690 2024-04-25T01:32:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.309+0000 7f5868716640 1 -- 172.21.15.28:0/3900092904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f586019ff50 con 0x7f5860100690 2024-04-25T01:32:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.310+0000 7f5868716640 1 -- 172.21.15.28:0/3900092904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5860107a50 con 0x7f5860100690 2024-04-25T01:32:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.311+0000 7f58577fe640 1 -- 172.21.15.28:0/3900092904 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f585c01a020 con 0x7f5860100690 2024-04-25T01:32:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.311+0000 7f58577fe640 1 --2- 172.21.15.28:0/3900092904 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f584c0a2c10 0x7f584c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.311+0000 7f5865cb1640 1 --2- 172.21.15.28:0/3900092904 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f584c0a2c10 0x7f584c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.311+0000 7f58577fe640 1 -- 172.21.15.28:0/3900092904 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f585c0cf160 con 0x7f5860100690 2024-04-25T01:32:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.312+0000 7f5865cb1640 1 --2- 172.21.15.28:0/3900092904 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f584c0a2c10 0x7f584c0a50d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f586006a2a0 tx=0x7f5850005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:24.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.315+0000 7f58577fe640 1 -- 172.21.15.28:0/3900092904 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f585c021070 con 0x7f5860100690 2024-04-25T01:32:24.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:24 smithi130 ceph-mon[32702]: pgmap v354: 129 pgs: 129 active+clean; 645 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:32:24.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:24 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3899692487' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:32:24.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:24 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3748914429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:24.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:24 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3482601432' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:24.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.499+0000 7f5868716640 1 -- 172.21.15.28:0/3900092904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f586002d170 con 0x7f5860100690 2024-04-25T01:32:24.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.499+0000 7f58577fe640 1 -- 172.21.15.28:0/3900092904 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f585c009de0 con 0x7f5860100690 2024-04-25T01:32:24.477 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:32:24.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.501+0000 7f5868716640 1 -- 172.21.15.28:0/3900092904 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f584c0a2c10 msgr2=0x7f584c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:24.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.501+0000 7f5868716640 1 --2- 172.21.15.28:0/3900092904 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f584c0a2c10 0x7f584c0a50d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f586006a2a0 tx=0x7f5850005e50 comp rx=0 tx=0).stop 2024-04-25T01:32:24.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.501+0000 7f5868716640 1 -- 172.21.15.28:0/3900092904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5860100690 msgr2=0x7f586006a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:24.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.501+0000 7f5868716640 1 --2- 172.21.15.28:0/3900092904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5860100690 0x7f586006a530 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f585c00d950 tx=0x7f585c00de10 comp rx=0 tx=0).stop 2024-04-25T01:32:24.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.501+0000 7f5868716640 1 -- 172.21.15.28:0/3900092904 shutdown_connections 2024-04-25T01:32:24.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.501+0000 7f5868716640 1 --2- 172.21.15.28:0/3900092904 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f584c0a2c10 0x7f584c0a50d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.501+0000 7f5868716640 1 --2- 172.21.15.28:0/3900092904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5860102240 0x7f586019ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.501+0000 7f5868716640 1 --2- 172.21.15.28:0/3900092904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5860101880 0x7f586019c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.501+0000 7f5868716640 1 --2- 172.21.15.28:0/3900092904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5860100690 0x7f586006a530 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.501+0000 7f5868716640 1 -- 172.21.15.28:0/3900092904 >> 172.21.15.28:0/3900092904 conn(0x7f58600fbc00 msgr2=0x7f5860104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:24.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.501+0000 7f5868716640 1 -- 172.21.15.28:0/3900092904 shutdown_connections 2024-04-25T01:32:24.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.501+0000 7f5868716640 1 -- 172.21.15.28:0/3900092904 wait complete. 2024-04-25T01:32:24.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:24 smithi032 ceph-mon[32461]: pgmap v354: 129 pgs: 129 active+clean; 645 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:32:24.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:24 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3899692487' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:32:24.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:24 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3748914429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:24.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:24 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3482601432' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:24.490 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:32:24.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.617+0000 7fdf066a5640 1 -- 172.21.15.28:0/4294953185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf00102250 msgr2=0x7fdf00108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.617+0000 7fdf066a5640 1 --2- 172.21.15.28:0/4294953185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf00102250 0x7fdf00108ae0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fdef8009ca0 tx=0x7fdef802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.617+0000 7fdf066a5640 1 -- 172.21.15.28:0/4294953185 shutdown_connections 2024-04-25T01:32:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.617+0000 7fdf066a5640 1 --2- 172.21.15.28:0/4294953185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf00102250 0x7fdf00108ae0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.617+0000 7fdf066a5640 1 --2- 172.21.15.28:0/4294953185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf00101890 0x7fdf00101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.617+0000 7fdf066a5640 1 --2- 172.21.15.28:0/4294953185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf001006a0 0x7fdf00100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.617+0000 7fdf066a5640 1 -- 172.21.15.28:0/4294953185 >> 172.21.15.28:0/4294953185 conn(0x7fdf000fbc30 msgr2=0x7fdf000fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.617+0000 7fdf066a5640 1 -- 172.21.15.28:0/4294953185 shutdown_connections 2024-04-25T01:32:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.617+0000 7fdf066a5640 1 -- 172.21.15.28:0/4294953185 wait complete. 2024-04-25T01:32:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.617+0000 7fdf066a5640 1 Processor -- start 2024-04-25T01:32:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.617+0000 7fdf066a5640 1 -- start start 2024-04-25T01:32:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.618+0000 7fdf066a5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf001006a0 0x7fdf00196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.618+0000 7fdf066a5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf00101890 0x7fdf00196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.618+0000 7fdf066a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf00102250 0x7fdf0019df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.618+0000 7fdf066a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0010c320 con 0x7fdf00102250 2024-04-25T01:32:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.618+0000 7fdf066a5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0010c1a0 con 0x7fdf001006a0 2024-04-25T01:32:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.618+0000 7fdf066a5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0010c4a0 con 0x7fdf00101890 2024-04-25T01:32:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.618+0000 7fdf04c42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf00102250 0x7fdf0019df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.618+0000 7fdf04c42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf00102250 0x7fdf0019df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45378/0 (socket says 172.21.15.28:45378) 2024-04-25T01:32:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.618+0000 7fdf04c42640 1 -- 172.21.15.28:0/1879632095 learned_addr learned my addr 172.21.15.28:0/1879632095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.618+0000 7fdeff7fe640 1 --2- 172.21.15.28:0/1879632095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf00101890 0x7fdf00196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.618+0000 7fdefffff640 1 --2- 172.21.15.28:0/1879632095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf001006a0 0x7fdf00196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.618+0000 7fdf04c42640 1 -- 172.21.15.28:0/1879632095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf00101890 msgr2=0x7fdf00196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.618+0000 7fdf04c42640 1 --2- 172.21.15.28:0/1879632095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf00101890 0x7fdf00196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.618+0000 7fdf04c42640 1 -- 172.21.15.28:0/1879632095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf001006a0 msgr2=0x7fdf00196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.618+0000 7fdf04c42640 1 --2- 172.21.15.28:0/1879632095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf001006a0 0x7fdf00196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.618+0000 7fdf04c42640 1 -- 172.21.15.28:0/1879632095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf0019e580 con 0x7fdf00102250 2024-04-25T01:32:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.618+0000 7fdf04c42640 1 --2- 172.21.15.28:0/1879632095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf00102250 0x7fdf0019df10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fdef8005e30 tx=0x7fdef8031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.619+0000 7fdefd7fa640 1 -- 172.21.15.28:0/1879632095 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef8003c30 con 0x7fdf00102250 2024-04-25T01:32:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.619+0000 7fdefd7fa640 1 -- 172.21.15.28:0/1879632095 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdef8003dd0 con 0x7fdf00102250 2024-04-25T01:32:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.619+0000 7fdefd7fa640 1 -- 172.21.15.28:0/1879632095 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef802fde0 con 0x7fdf00102250 2024-04-25T01:32:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.619+0000 7fdf066a5640 1 -- 172.21.15.28:0/1879632095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf0019e810 con 0x7fdf00102250 2024-04-25T01:32:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.619+0000 7fdf066a5640 1 -- 172.21.15.28:0/1879632095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf0019ed60 con 0x7fdf00102250 2024-04-25T01:32:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.620+0000 7fdf066a5640 1 -- 172.21.15.28:0/1879632095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf001079c0 con 0x7fdf00102250 2024-04-25T01:32:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.621+0000 7fdefd7fa640 1 -- 172.21.15.28:0/1879632095 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fdef8004050 con 0x7fdf00102250 2024-04-25T01:32:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.621+0000 7fdefd7fa640 1 --2- 172.21.15.28:0/1879632095 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fdee80a2b90 0x7fdee80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.622+0000 7fdefd7fa640 1 -- 172.21.15.28:0/1879632095 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fdef80f0990 con 0x7fdf00102250 2024-04-25T01:32:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.622+0000 7fdefffff640 1 --2- 172.21.15.28:0/1879632095 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fdee80a2b90 0x7fdee80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.622+0000 7fdefffff640 1 --2- 172.21.15.28:0/1879632095 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fdee80a2b90 0x7fdee80a5050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdef0005db0 tx=0x7fdef0005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:24.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.625+0000 7fdefd7fa640 1 -- 172.21.15.28:0/1879632095 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdef80bd590 con 0x7fdf00102250 2024-04-25T01:32:24.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.809+0000 7fdf066a5640 1 -- 172.21.15.28:0/1879632095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdf0002d300 con 0x7fdf00102250 2024-04-25T01:32:24.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.810+0000 7fdefd7fa640 1 -- 172.21.15.28:0/1879632095 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fdef80bce90 con 0x7fdf00102250 2024-04-25T01:32:24.787 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T01:32:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.811+0000 7fdf066a5640 1 -- 172.21.15.28:0/1879632095 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fdee80a2b90 msgr2=0x7fdee80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.811+0000 7fdf066a5640 1 --2- 172.21.15.28:0/1879632095 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fdee80a2b90 0x7fdee80a5050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdef0005db0 tx=0x7fdef0005d40 comp rx=0 tx=0).stop 2024-04-25T01:32:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.811+0000 7fdf066a5640 1 -- 172.21.15.28:0/1879632095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf00102250 msgr2=0x7fdf0019df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.811+0000 7fdf066a5640 1 --2- 172.21.15.28:0/1879632095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf00102250 0x7fdf0019df10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fdef8005e30 tx=0x7fdef8031ee0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.812+0000 7fdf066a5640 1 -- 172.21.15.28:0/1879632095 shutdown_connections 2024-04-25T01:32:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.812+0000 7fdf066a5640 1 --2- 172.21.15.28:0/1879632095 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fdee80a2b90 0x7fdee80a5050 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.812+0000 7fdf066a5640 1 --2- 172.21.15.28:0/1879632095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf00102250 0x7fdf0019df10 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.812+0000 7fdf066a5640 1 --2- 172.21.15.28:0/1879632095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdf00101890 0x7fdf00196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.812+0000 7fdf066a5640 1 --2- 172.21.15.28:0/1879632095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdf001006a0 0x7fdf00196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.812+0000 7fdf066a5640 1 -- 172.21.15.28:0/1879632095 >> 172.21.15.28:0/1879632095 conn(0x7fdf000fbc30 msgr2=0x7fdf00104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.812+0000 7fdf066a5640 1 -- 172.21.15.28:0/1879632095 shutdown_connections 2024-04-25T01:32:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.812+0000 7fdf066a5640 1 -- 172.21.15.28:0/1879632095 wait complete. 2024-04-25T01:32:24.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:32:24.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.927+0000 7fc0d08a7640 1 -- 172.21.15.28:0/3121868222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0cc1017a0 msgr2=0x7fc0cc101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:24.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.927+0000 7fc0d08a7640 1 --2- 172.21.15.28:0/3121868222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0cc1017a0 0x7fc0cc101bc0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc009900 tx=0x7fc0bc02a410 comp rx=0 tx=0).stop 2024-04-25T01:32:24.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.927+0000 7fc0d08a7640 1 -- 172.21.15.28:0/3121868222 shutdown_connections 2024-04-25T01:32:24.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.928+0000 7fc0d08a7640 1 --2- 172.21.15.28:0/3121868222 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc0cc103350 0x7fc0cc109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.928+0000 7fc0d08a7640 1 --2- 172.21.15.28:0/3121868222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0cc102990 0x7fc0cc102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.928+0000 7fc0d08a7640 1 --2- 172.21.15.28:0/3121868222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0cc1017a0 0x7fc0cc101bc0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.928+0000 7fc0d08a7640 1 -- 172.21.15.28:0/3121868222 >> 172.21.15.28:0/3121868222 conn(0x7fc0cc0fcd10 msgr2=0x7fc0cc0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:24.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.928+0000 7fc0d08a7640 1 -- 172.21.15.28:0/3121868222 shutdown_connections 2024-04-25T01:32:24.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.928+0000 7fc0d08a7640 1 -- 172.21.15.28:0/3121868222 wait complete. 2024-04-25T01:32:24.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.928+0000 7fc0d08a7640 1 Processor -- start 2024-04-25T01:32:24.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.928+0000 7fc0d08a7640 1 -- start start 2024-04-25T01:32:24.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.928+0000 7fc0d08a7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0cc1017a0 0x7fc0cc197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:24.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.928+0000 7fc0d08a7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0cc102990 0x7fc0cc197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:24.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.928+0000 7fc0d08a7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc0cc103350 0x7fc0cc19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:24.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0d08a7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0cc10d470 con 0x7fc0cc1017a0 2024-04-25T01:32:24.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0d08a7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc0cc10d2f0 con 0x7fc0cc102990 2024-04-25T01:32:24.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0d08a7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc0cc10d5f0 con 0x7fc0cc103350 2024-04-25T01:32:24.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0ca59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0cc1017a0 0x7fc0cc197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:24.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0ca59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0cc1017a0 0x7fc0cc197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45404/0 (socket says 172.21.15.28:45404) 2024-04-25T01:32:24.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0ca59c640 1 -- 172.21.15.28:0/3580889582 learned_addr learned my addr 172.21.15.28:0/3580889582 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:24.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0cad9d640 1 --2- 172.21.15.28:0/3580889582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc0cc103350 0x7fc0cc19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:24.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0c9d9b640 1 --2- 172.21.15.28:0/3580889582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0cc102990 0x7fc0cc197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:24.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0ca59c640 1 -- 172.21.15.28:0/3580889582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc0cc103350 msgr2=0x7fc0cc19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:24.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0ca59c640 1 --2- 172.21.15.28:0/3580889582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc0cc103350 0x7fc0cc19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0ca59c640 1 -- 172.21.15.28:0/3580889582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0cc102990 msgr2=0x7fc0cc197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:24.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0ca59c640 1 --2- 172.21.15.28:0/3580889582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0cc102990 0x7fc0cc197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:24.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0ca59c640 1 -- 172.21.15.28:0/3580889582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0cc19f6d0 con 0x7fc0cc1017a0 2024-04-25T01:32:24.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0c9d9b640 1 --2- 172.21.15.28:0/3580889582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0cc102990 0x7fc0cc197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:24.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0ca59c640 1 --2- 172.21.15.28:0/3580889582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0cc1017a0 0x7fc0cc197ab0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc031440 tx=0x7fc0bc032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:24.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0bb7fe640 1 -- 172.21.15.28:0/3580889582 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc03f030 con 0x7fc0cc1017a0 2024-04-25T01:32:24.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0d08a7640 1 -- 172.21.15.28:0/3580889582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0cc19f960 con 0x7fc0cc1017a0 2024-04-25T01:32:24.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0bb7fe640 1 -- 172.21.15.28:0/3580889582 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0bc003ce0 con 0x7fc0cc1017a0 2024-04-25T01:32:24.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0bb7fe640 1 -- 172.21.15.28:0/3580889582 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc035070 con 0x7fc0cc1017a0 2024-04-25T01:32:24.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.929+0000 7fc0d08a7640 1 -- 172.21.15.28:0/3580889582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0cc19fe50 con 0x7fc0cc1017a0 2024-04-25T01:32:24.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.931+0000 7fc0d08a7640 1 -- 172.21.15.28:0/3580889582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0cc108ae0 con 0x7fc0cc1017a0 2024-04-25T01:32:24.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.931+0000 7fc0bb7fe640 1 -- 172.21.15.28:0/3580889582 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc0bc042030 con 0x7fc0cc1017a0 2024-04-25T01:32:24.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.932+0000 7fc0bb7fe640 1 --2- 172.21.15.28:0/3580889582 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc0b40a2b90 0x7fc0b40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:24.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.932+0000 7fc0c9d9b640 1 --2- 172.21.15.28:0/3580889582 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc0b40a2b90 0x7fc0b40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:24.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.933+0000 7fc0bb7fe640 1 -- 172.21.15.28:0/3580889582 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fc0bc0efa10 con 0x7fc0cc1017a0 2024-04-25T01:32:24.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.933+0000 7fc0c9d9b640 1 --2- 172.21.15.28:0/3580889582 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc0b40a2b90 0x7fc0b40a5050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc198fc0 tx=0x7fc0c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:24.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:24.937+0000 7fc0bb7fe640 1 -- 172.21.15.28:0/3580889582 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc0bc0bc610 con 0x7fc0cc1017a0 2024-04-25T01:32:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.123+0000 7fc0d08a7640 1 -- 172.21.15.28:0/3580889582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc0cc062340 con 0x7fc0cc1017a0 2024-04-25T01:32:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.123+0000 7fc0bb7fe640 1 -- 172.21.15.28:0/3580889582 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fc0bc0bbf10 con 0x7fc0cc1017a0 2024-04-25T01:32:25.101 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:32:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.125+0000 7fc0d08a7640 1 -- 172.21.15.28:0/3580889582 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc0b40a2b90 msgr2=0x7fc0b40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.125+0000 7fc0d08a7640 1 --2- 172.21.15.28:0/3580889582 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc0b40a2b90 0x7fc0b40a5050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc198fc0 tx=0x7fc0c0005f90 comp rx=0 tx=0).stop 2024-04-25T01:32:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.125+0000 7fc0d08a7640 1 -- 172.21.15.28:0/3580889582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0cc1017a0 msgr2=0x7fc0cc197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.125+0000 7fc0d08a7640 1 --2- 172.21.15.28:0/3580889582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0cc1017a0 0x7fc0cc197ab0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc031440 tx=0x7fc0bc032ce0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.125+0000 7fc0d08a7640 1 -- 172.21.15.28:0/3580889582 shutdown_connections 2024-04-25T01:32:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.125+0000 7fc0d08a7640 1 --2- 172.21.15.28:0/3580889582 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fc0b40a2b90 0x7fc0b40a5050 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.125+0000 7fc0d08a7640 1 --2- 172.21.15.28:0/3580889582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc0cc103350 0x7fc0cc19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.125+0000 7fc0d08a7640 1 --2- 172.21.15.28:0/3580889582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0cc102990 0x7fc0cc197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.125+0000 7fc0d08a7640 1 --2- 172.21.15.28:0/3580889582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0cc1017a0 0x7fc0cc197ab0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.125+0000 7fc0d08a7640 1 -- 172.21.15.28:0/3580889582 >> 172.21.15.28:0/3580889582 conn(0x7fc0cc0fcd10 msgr2=0x7fc0cc105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.125+0000 7fc0d08a7640 1 -- 172.21.15.28:0/3580889582 shutdown_connections 2024-04-25T01:32:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.125+0000 7fc0d08a7640 1 -- 172.21.15.28:0/3580889582 wait complete. 2024-04-25T01:32:25.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:32:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.239+0000 7f749a9d5640 1 -- 172.21.15.28:0/214800041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74941029a0 msgr2=0x7f7494102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.239+0000 7f749a9d5640 1 --2- 172.21.15.28:0/214800041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74941029a0 0x7f7494102e20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7488009900 tx=0x7f748802a410 comp rx=0 tx=0).stop 2024-04-25T01:32:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.239+0000 7f749a9d5640 1 -- 172.21.15.28:0/214800041 shutdown_connections 2024-04-25T01:32:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.239+0000 7f749a9d5640 1 --2- 172.21.15.28:0/214800041 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7494103360 0x7f7494109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.239+0000 7f749a9d5640 1 --2- 172.21.15.28:0/214800041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74941029a0 0x7f7494102e20 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.239+0000 7f749a9d5640 1 --2- 172.21.15.28:0/214800041 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74941017b0 0x7f7494101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.239+0000 7f749a9d5640 1 -- 172.21.15.28:0/214800041 >> 172.21.15.28:0/214800041 conn(0x7f74940fcd40 msgr2=0x7f74940ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.239+0000 7f749a9d5640 1 -- 172.21.15.28:0/214800041 shutdown_connections 2024-04-25T01:32:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.239+0000 7f749a9d5640 1 -- 172.21.15.28:0/214800041 wait complete. 2024-04-25T01:32:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.240+0000 7f749a9d5640 1 Processor -- start 2024-04-25T01:32:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.240+0000 7f749a9d5640 1 -- start start 2024-04-25T01:32:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.240+0000 7f749a9d5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74941017b0 0x7f7494197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.240+0000 7f749a9d5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f74941029a0 0x7f7494197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.240+0000 7f749a9d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7494103360 0x7f749419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.240+0000 7f749a9d5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f749410d410 con 0x7f7494103360 2024-04-25T01:32:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.240+0000 7f749a9d5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f749410d290 con 0x7f74941017b0 2024-04-25T01:32:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.240+0000 7f749a9d5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f749410d590 con 0x7f74941029a0 2024-04-25T01:32:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.240+0000 7f7498f72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7494103360 0x7f749419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.240+0000 7f7498f72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7494103360 0x7f749419f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45430/0 (socket says 172.21.15.28:45430) 2024-04-25T01:32:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.240+0000 7f7498f72640 1 -- 172.21.15.28:0/4094872170 learned_addr learned my addr 172.21.15.28:0/4094872170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.240+0000 7f7493fff640 1 --2- 172.21.15.28:0/4094872170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74941017b0 0x7f7494197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.240+0000 7f74937fe640 1 --2- 172.21.15.28:0/4094872170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f74941029a0 0x7f7494197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.240+0000 7f7498f72640 1 -- 172.21.15.28:0/4094872170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f74941029a0 msgr2=0x7f7494197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.240+0000 7f7498f72640 1 --2- 172.21.15.28:0/4094872170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f74941029a0 0x7f7494197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.240+0000 7f7498f72640 1 -- 172.21.15.28:0/4094872170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74941017b0 msgr2=0x7f7494197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.240+0000 7f7498f72640 1 --2- 172.21.15.28:0/4094872170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74941017b0 0x7f7494197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.240+0000 7f7498f72640 1 -- 172.21.15.28:0/4094872170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f749419f6a0 con 0x7f7494103360 2024-04-25T01:32:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.241+0000 7f7493fff640 1 --2- 172.21.15.28:0/4094872170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74941017b0 0x7f7494197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.241+0000 7f7498f72640 1 --2- 172.21.15.28:0/4094872170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7494103360 0x7f749419f030 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f748c00eaa0 tx=0x7f748c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.241+0000 7f74917fa640 1 -- 172.21.15.28:0/4094872170 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f748c00cdb0 con 0x7f7494103360 2024-04-25T01:32:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.241+0000 7f74917fa640 1 -- 172.21.15.28:0/4094872170 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f748c005020 con 0x7f7494103360 2024-04-25T01:32:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.241+0000 7f749a9d5640 1 -- 172.21.15.28:0/4094872170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f749419f990 con 0x7f7494103360 2024-04-25T01:32:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.241+0000 7f74917fa640 1 -- 172.21.15.28:0/4094872170 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f748c0116b0 con 0x7f7494103360 2024-04-25T01:32:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.241+0000 7f749a9d5640 1 -- 172.21.15.28:0/4094872170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f749419fe60 con 0x7f7494103360 2024-04-25T01:32:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.242+0000 7f749a9d5640 1 -- 172.21.15.28:0/4094872170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7494108ad0 con 0x7f7494103360 2024-04-25T01:32:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.243+0000 7f74917fa640 1 -- 172.21.15.28:0/4094872170 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f748c005560 con 0x7f7494103360 2024-04-25T01:32:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.243+0000 7f74917fa640 1 --2- 172.21.15.28:0/4094872170 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f747c0a2b90 0x7f747c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.244+0000 7f7493fff640 1 --2- 172.21.15.28:0/4094872170 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f747c0a2b90 0x7f747c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.244+0000 7f74917fa640 1 -- 172.21.15.28:0/4094872170 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f748c0cf680 con 0x7f7494103360 2024-04-25T01:32:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.244+0000 7f7493fff640 1 --2- 172.21.15.28:0/4094872170 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f747c0a2b90 0x7f747c0a5050 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7494102800 tx=0x7f7484005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:25.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.248+0000 7f74917fa640 1 -- 172.21.15.28:0/4094872170 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f748c016030 con 0x7f7494103360 2024-04-25T01:32:25.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.474+0000 7f749a9d5640 1 -- 172.21.15.28:0/4094872170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f749402d170 con 0x7f7494103360 2024-04-25T01:32:25.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.475+0000 7f74917fa640 1 -- 172.21.15.28:0/4094872170 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11923 (secure 0 0 0) 0x7f748c09c220 con 0x7f7494103360 2024-04-25T01:32:25.453 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:32:25.453 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:31:01.802718+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24454,24442,24512,24478,24473]},"id":1}]} 2024-04-25T01:32:25.453 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-04-25T01:32:25.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.477+0000 7f749a9d5640 1 -- 172.21.15.28:0/4094872170 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f747c0a2b90 msgr2=0x7f747c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.477+0000 7f749a9d5640 1 --2- 172.21.15.28:0/4094872170 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f747c0a2b90 0x7f747c0a5050 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7494102800 tx=0x7f7484005f90 comp rx=0 tx=0).stop 2024-04-25T01:32:25.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.477+0000 7f749a9d5640 1 -- 172.21.15.28:0/4094872170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7494103360 msgr2=0x7f749419f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.477+0000 7f749a9d5640 1 --2- 172.21.15.28:0/4094872170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7494103360 0x7f749419f030 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f748c00eaa0 tx=0x7f748c00ef60 comp rx=0 tx=0).stop 2024-04-25T01:32:25.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.477+0000 7f749a9d5640 1 -- 172.21.15.28:0/4094872170 shutdown_connections 2024-04-25T01:32:25.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.477+0000 7f749a9d5640 1 --2- 172.21.15.28:0/4094872170 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f747c0a2b90 0x7f747c0a5050 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.477+0000 7f749a9d5640 1 --2- 172.21.15.28:0/4094872170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7494103360 0x7f749419f030 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.477+0000 7f749a9d5640 1 --2- 172.21.15.28:0/4094872170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f74941029a0 0x7f7494197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.477+0000 7f749a9d5640 1 --2- 172.21.15.28:0/4094872170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74941017b0 0x7f7494197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.477+0000 7f749a9d5640 1 -- 172.21.15.28:0/4094872170 >> 172.21.15.28:0/4094872170 conn(0x7f74940fcd40 msgr2=0x7f7494105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:25.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.477+0000 7f749a9d5640 1 -- 172.21.15.28:0/4094872170 shutdown_connections 2024-04-25T01:32:25.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.477+0000 7f749a9d5640 1 -- 172.21.15.28:0/4094872170 wait complete. 2024-04-25T01:32:25.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:25 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2886081701' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:32:25.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:25 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3900092904' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:32:25.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:25 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1879632095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:32:25.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:25 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2886081701' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:32:25.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:25 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3900092904' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:32:25.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:25 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1879632095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:32:25.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:32:25.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:25 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2886081701' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:32:25.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:25 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3900092904' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:32:25.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:25 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1879632095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:32:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.589+0000 7fe9dda64640 1 -- 172.21.15.28:0/3215783465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9d81006a0 msgr2=0x7fe9d8100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.589+0000 7fe9dda64640 1 --2- 172.21.15.28:0/3215783465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9d81006a0 0x7fe9d8100ac0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8009900 tx=0x7fe9c802a410 comp rx=0 tx=0).stop 2024-04-25T01:32:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.589+0000 7fe9dda64640 1 -- 172.21.15.28:0/3215783465 shutdown_connections 2024-04-25T01:32:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.590+0000 7fe9dda64640 1 --2- 172.21.15.28:0/3215783465 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9d8102250 0x7fe9d8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.590+0000 7fe9dda64640 1 --2- 172.21.15.28:0/3215783465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9d8101890 0x7fe9d8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.590+0000 7fe9dda64640 1 --2- 172.21.15.28:0/3215783465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9d81006a0 0x7fe9d8100ac0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.590+0000 7fe9dda64640 1 -- 172.21.15.28:0/3215783465 >> 172.21.15.28:0/3215783465 conn(0x7fe9d80fbc30 msgr2=0x7fe9d80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.590+0000 7fe9dda64640 1 -- 172.21.15.28:0/3215783465 shutdown_connections 2024-04-25T01:32:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.590+0000 7fe9dda64640 1 -- 172.21.15.28:0/3215783465 wait complete. 2024-04-25T01:32:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.590+0000 7fe9dda64640 1 Processor -- start 2024-04-25T01:32:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.590+0000 7fe9dda64640 1 -- start start 2024-04-25T01:32:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.590+0000 7fe9dda64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9d81006a0 0x7fe9d806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.590+0000 7fe9dda64640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9d8101890 0x7fe9d819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.590+0000 7fe9dda64640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9d8102250 0x7fe9d819edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9dda64640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d810c320 con 0x7fe9d81006a0 2024-04-25T01:32:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9dda64640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d810c1a0 con 0x7fe9d8102250 2024-04-25T01:32:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9dda64640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d810c4a0 con 0x7fe9d8101890 2024-04-25T01:32:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9d77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9d81006a0 0x7fe9d806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9d77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9d81006a0 0x7fe9d806a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45444/0 (socket says 172.21.15.28:45444) 2024-04-25T01:32:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9d77fe640 1 -- 172.21.15.28:0/3489085706 learned_addr learned my addr 172.21.15.28:0/3489085706 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9d6ffd640 1 --2- 172.21.15.28:0/3489085706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9d8101890 0x7fe9d819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9d77fe640 1 -- 172.21.15.28:0/3489085706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9d8101890 msgr2=0x7fe9d819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9d7fff640 1 --2- 172.21.15.28:0/3489085706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9d8102250 0x7fe9d819edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9d77fe640 1 --2- 172.21.15.28:0/3489085706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9d8101890 0x7fe9d819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9d77fe640 1 -- 172.21.15.28:0/3489085706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9d8102250 msgr2=0x7fe9d819edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9d77fe640 1 --2- 172.21.15.28:0/3489085706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9d8102250 0x7fe9d819edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9d77fe640 1 -- 172.21.15.28:0/3489085706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9d819f6b0 con 0x7fe9d81006a0 2024-04-25T01:32:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9d6ffd640 1 --2- 172.21.15.28:0/3489085706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9d8101890 0x7fe9d819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9d77fe640 1 --2- 172.21.15.28:0/3489085706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9d81006a0 0x7fe9d806a8c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8031610 tx=0x7fe9c80043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9d4ff9640 1 -- 172.21.15.28:0/3489085706 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c803f030 con 0x7fe9d81006a0 2024-04-25T01:32:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9d4ff9640 1 -- 172.21.15.28:0/3489085706 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9c8003ce0 con 0x7fe9d81006a0 2024-04-25T01:32:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9d4ff9640 1 -- 172.21.15.28:0/3489085706 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8035070 con 0x7fe9d81006a0 2024-04-25T01:32:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9dda64640 1 -- 172.21.15.28:0/3489085706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9d819f940 con 0x7fe9d81006a0 2024-04-25T01:32:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.591+0000 7fe9dda64640 1 -- 172.21.15.28:0/3489085706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9d819fde0 con 0x7fe9d81006a0 2024-04-25T01:32:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.592+0000 7fe9dda64640 1 -- 172.21.15.28:0/3489085706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9d8107a40 con 0x7fe9d81006a0 2024-04-25T01:32:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.594+0000 7fe9d4ff9640 1 -- 172.21.15.28:0/3489085706 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe9c8042030 con 0x7fe9d81006a0 2024-04-25T01:32:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.595+0000 7fe9d4ff9640 1 --2- 172.21.15.28:0/3489085706 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe9c00a2c10 0x7fe9c00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.595+0000 7fe9d6ffd640 1 --2- 172.21.15.28:0/3489085706 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe9c00a2c10 0x7fe9c00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.595+0000 7fe9d4ff9640 1 -- 172.21.15.28:0/3489085706 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fe9c80efd00 con 0x7fe9d81006a0 2024-04-25T01:32:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.595+0000 7fe9d6ffd640 1 --2- 172.21.15.28:0/3489085706 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe9c00a2c10 0x7fe9c00a50d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc005fd0 tx=0x7fe9cc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.598+0000 7fe9d4ff9640 1 -- 172.21.15.28:0/3489085706 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe9c80bc8a0 con 0x7fe9d81006a0 2024-04-25T01:32:25.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.783+0000 7fe9dda64640 1 -- 172.21.15.28:0/3489085706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe9d80623e0 con 0x7fe9d81006a0 2024-04-25T01:32:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.785+0000 7fe9d4ff9640 1 -- 172.21.15.28:0/3489085706 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24564 (secure 0 0 0) 0x7fe9c80bc1a0 con 0x7fe9d81006a0 2024-04-25T01:32:25.763 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:32:25.763 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:31:21.857121+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:32:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.787+0000 7fe9dda64640 1 -- 172.21.15.28:0/3489085706 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe9c00a2c10 msgr2=0x7fe9c00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.787+0000 7fe9dda64640 1 --2- 172.21.15.28:0/3489085706 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe9c00a2c10 0x7fe9c00a50d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc005fd0 tx=0x7fe9cc005850 comp rx=0 tx=0).stop 2024-04-25T01:32:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.787+0000 7fe9dda64640 1 -- 172.21.15.28:0/3489085706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9d81006a0 msgr2=0x7fe9d806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.787+0000 7fe9dda64640 1 --2- 172.21.15.28:0/3489085706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9d81006a0 0x7fe9d806a8c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8031610 tx=0x7fe9c80043b0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.787+0000 7fe9dda64640 1 -- 172.21.15.28:0/3489085706 shutdown_connections 2024-04-25T01:32:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.787+0000 7fe9dda64640 1 --2- 172.21.15.28:0/3489085706 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe9c00a2c10 0x7fe9c00a50d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.787+0000 7fe9dda64640 1 --2- 172.21.15.28:0/3489085706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9d8102250 0x7fe9d819edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.787+0000 7fe9dda64640 1 --2- 172.21.15.28:0/3489085706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9d8101890 0x7fe9d819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.787+0000 7fe9dda64640 1 --2- 172.21.15.28:0/3489085706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9d81006a0 0x7fe9d806a8c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.787+0000 7fe9dda64640 1 -- 172.21.15.28:0/3489085706 >> 172.21.15.28:0/3489085706 conn(0x7fe9d80fbc30 msgr2=0x7fe9d8104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.787+0000 7fe9dda64640 1 -- 172.21.15.28:0/3489085706 shutdown_connections 2024-04-25T01:32:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.787+0000 7fe9dda64640 1 -- 172.21.15.28:0/3489085706 wait complete. 2024-04-25T01:32:25.776 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:32:25.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.901+0000 7f23631dc640 1 -- 172.21.15.28:0/1855346768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f235c1006a0 msgr2=0x7f235c100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.901+0000 7f23631dc640 1 --2- 172.21.15.28:0/1855346768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f235c1006a0 0x7f235c100ac0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2350009900 tx=0x7f235002a410 comp rx=0 tx=0).stop 2024-04-25T01:32:25.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.901+0000 7f23631dc640 1 -- 172.21.15.28:0/1855346768 shutdown_connections 2024-04-25T01:32:25.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.901+0000 7f23631dc640 1 --2- 172.21.15.28:0/1855346768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f235c102250 0x7f235c108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.901+0000 7f23631dc640 1 --2- 172.21.15.28:0/1855346768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f235c101890 0x7f235c101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.901+0000 7f23631dc640 1 --2- 172.21.15.28:0/1855346768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f235c1006a0 0x7f235c100ac0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.901+0000 7f23631dc640 1 -- 172.21.15.28:0/1855346768 >> 172.21.15.28:0/1855346768 conn(0x7f235c0fbc30 msgr2=0x7f235c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:25.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.901+0000 7f23631dc640 1 -- 172.21.15.28:0/1855346768 shutdown_connections 2024-04-25T01:32:25.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.901+0000 7f23631dc640 1 -- 172.21.15.28:0/1855346768 wait complete. 2024-04-25T01:32:25.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.902+0000 7f23631dc640 1 Processor -- start 2024-04-25T01:32:25.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.902+0000 7f23631dc640 1 -- start start 2024-04-25T01:32:25.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.902+0000 7f23631dc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f235c1006a0 0x7f235c197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.902+0000 7f23631dc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f235c101890 0x7f235c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.902+0000 7f23631dc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f235c102250 0x7f235c19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.902+0000 7f23631dc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f235c10c2f0 con 0x7f235c102250 2024-04-25T01:32:25.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.902+0000 7f23631dc640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f235c10c170 con 0x7f235c101890 2024-04-25T01:32:25.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.902+0000 7f23631dc640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f235c10c470 con 0x7f235c1006a0 2024-04-25T01:32:25.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.902+0000 7f2361779640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f235c102250 0x7f235c19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.902+0000 7f2361779640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f235c102250 0x7f235c19f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45460/0 (socket says 172.21.15.28:45460) 2024-04-25T01:32:25.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.902+0000 7f2361779640 1 -- 172.21.15.28:0/3950680693 learned_addr learned my addr 172.21.15.28:0/3950680693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:25.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.902+0000 7f235bfff640 1 --2- 172.21.15.28:0/3950680693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f235c101890 0x7f235c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.902+0000 7f2360f78640 1 --2- 172.21.15.28:0/3950680693 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f235c1006a0 0x7f235c197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.902+0000 7f2361779640 1 -- 172.21.15.28:0/3950680693 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f235c1006a0 msgr2=0x7f235c197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.902+0000 7f2361779640 1 --2- 172.21.15.28:0/3950680693 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f235c1006a0 0x7f235c197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.902+0000 7f2361779640 1 -- 172.21.15.28:0/3950680693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f235c101890 msgr2=0x7f235c198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:25.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.902+0000 7f2361779640 1 --2- 172.21.15.28:0/3950680693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f235c101890 0x7f235c198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:25.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.902+0000 7f2361779640 1 -- 172.21.15.28:0/3950680693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f235c19f760 con 0x7f235c102250 2024-04-25T01:32:25.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.902+0000 7f2361779640 1 --2- 172.21.15.28:0/3950680693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f235c102250 0x7f235c19f0f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f235400eb10 tx=0x7f235400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:25.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.903+0000 7f2359ffb640 1 -- 172.21.15.28:0/3950680693 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2354004b20 con 0x7f235c102250 2024-04-25T01:32:25.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.903+0000 7f2359ffb640 1 -- 172.21.15.28:0/3950680693 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2354004cc0 con 0x7f235c102250 2024-04-25T01:32:25.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.903+0000 7f23631dc640 1 -- 172.21.15.28:0/3950680693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f235c19fa50 con 0x7f235c102250 2024-04-25T01:32:25.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.903+0000 7f2359ffb640 1 -- 172.21.15.28:0/3950680693 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2354010760 con 0x7f235c102250 2024-04-25T01:32:25.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.903+0000 7f23631dc640 1 -- 172.21.15.28:0/3950680693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f235c19ff50 con 0x7f235c102250 2024-04-25T01:32:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.904+0000 7f23631dc640 1 -- 172.21.15.28:0/3950680693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f235c1a0240 con 0x7f235c102250 2024-04-25T01:32:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.905+0000 7f2359ffb640 1 -- 172.21.15.28:0/3950680693 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2354005020 con 0x7f235c102250 2024-04-25T01:32:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.906+0000 7f2359ffb640 1 --2- 172.21.15.28:0/3950680693 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f234c0a2b90 0x7f234c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.906+0000 7f2360f78640 1 --2- 172.21.15.28:0/3950680693 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f234c0a2b90 0x7f234c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.906+0000 7f2359ffb640 1 -- 172.21.15.28:0/3950680693 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f23540cfa60 con 0x7f235c102250 2024-04-25T01:32:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.906+0000 7f2360f78640 1 --2- 172.21.15.28:0/3950680693 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f234c0a2b90 0x7f234c0a5050 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2350006010 tx=0x7f2350000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:25.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:25.910+0000 7f2359ffb640 1 -- 172.21.15.28:0/3950680693 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f235409c600 con 0x7f235c102250 2024-04-25T01:32:26.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.135+0000 7f23631dc640 1 -- 172.21.15.28:0/3950680693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f235c1a0410 con 0x7f235c102250 2024-04-25T01:32:26.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.135+0000 7f2359ffb640 1 -- 172.21.15.28:0/3950680693 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11923 (secure 0 0 0) 0x7f235409bf00 con 0x7f235c102250 2024-04-25T01:32:26.113 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:32:26.113 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:31:01.802718+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24454,24442,24512,24478,24473]},"id":1}]} 2024-04-25T01:32:26.114 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-04-25T01:32:26.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.138+0000 7f23631dc640 1 -- 172.21.15.28:0/3950680693 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f234c0a2b90 msgr2=0x7f234c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.138+0000 7f23631dc640 1 --2- 172.21.15.28:0/3950680693 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f234c0a2b90 0x7f234c0a5050 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2350006010 tx=0x7f2350000bc0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.138+0000 7f23631dc640 1 -- 172.21.15.28:0/3950680693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f235c102250 msgr2=0x7f235c19f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.138+0000 7f23631dc640 1 --2- 172.21.15.28:0/3950680693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f235c102250 0x7f235c19f0f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f235400eb10 tx=0x7f235400c2d0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.138+0000 7f23631dc640 1 -- 172.21.15.28:0/3950680693 shutdown_connections 2024-04-25T01:32:26.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.138+0000 7f23631dc640 1 --2- 172.21.15.28:0/3950680693 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f234c0a2b90 0x7f234c0a5050 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.138+0000 7f23631dc640 1 --2- 172.21.15.28:0/3950680693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f235c102250 0x7f235c19f0f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.138+0000 7f23631dc640 1 --2- 172.21.15.28:0/3950680693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f235c101890 0x7f235c198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.138+0000 7f23631dc640 1 --2- 172.21.15.28:0/3950680693 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f235c1006a0 0x7f235c197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.138+0000 7f23631dc640 1 -- 172.21.15.28:0/3950680693 >> 172.21.15.28:0/3950680693 conn(0x7f235c0fbc30 msgr2=0x7f235c0fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.138+0000 7f23631dc640 1 -- 172.21.15.28:0/3950680693 shutdown_connections 2024-04-25T01:32:26.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.138+0000 7f23631dc640 1 -- 172.21.15.28:0/3950680693 wait complete. 2024-04-25T01:32:26.127 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T01:30:33.696870+0000', 'modified': '2024-04-25T01:31:01.802718+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24512, 'mds_1': 24478, 'mds_2': 24473, 'mds_3': 24454, 'mds_4': 24442}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14643': {'gid': 14643, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6839/3206225061', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 3206225061}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 3206225061}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14655': {'gid': 14655, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6841/3666045393', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6840', 'nonce': 3666045393}, {'type': 'v1', 'addr': '172.21.15.28:6841', 'nonce': 3666045393}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24401': {'gid': 24401, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.32:6833/983600975', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 983600975}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 983600975}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24412': {'gid': 24412, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.130:6833/1080169582', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6832', 'nonce': 1080169582}, {'type': 'v1', 'addr': '172.21.15.130:6833', 'nonce': 1080169582}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24440': {'gid': 24440, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.32:6835/356243399', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 356243399}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 356243399}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.130:6835/348635896', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6834', 'nonce': 348635896}, {'type': 'v1', 'addr': '172.21.15.130:6835', 'nonce': 348635896}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.130:6837/211524954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6836', 'nonce': 211524954}, {'type': 'v1', 'addr': '172.21.15.130:6837', 'nonce': 211524954}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.32:6837/2771586449', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 2771586449}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 2771586449}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.130:6839/463478317', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6838', 'nonce': 463478317}, {'type': 'v1', 'addr': '172.21.15.130:6839', 'nonce': 463478317}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24512': {'gid': 24512, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6839/2835949714', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 2835949714}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 2835949714}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24512, 'qdb_cluster': [24454, 24442, 24512, 24478, 24473]} 2024-04-25T01:32:26.127 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-25T01:32:26.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-25T01:32:26.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.253+0000 7fb6b5341640 1 -- 172.21.15.28:0/680872973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0102230 msgr2=0x7fb6b0108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.253+0000 7fb6b5341640 1 --2- 172.21.15.28:0/680872973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0102230 0x7fb6b0108ac0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0009900 tx=0x7fb6a002a410 comp rx=0 tx=0).stop 2024-04-25T01:32:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.253+0000 7fb6b5341640 1 -- 172.21.15.28:0/680872973 shutdown_connections 2024-04-25T01:32:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.253+0000 7fb6b5341640 1 --2- 172.21.15.28:0/680872973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0102230 0x7fb6b0108ac0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.253+0000 7fb6b5341640 1 --2- 172.21.15.28:0/680872973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6b0101870 0x7fb6b0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.253+0000 7fb6b5341640 1 --2- 172.21.15.28:0/680872973 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6b0100680 0x7fb6b0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.253+0000 7fb6b5341640 1 -- 172.21.15.28:0/680872973 >> 172.21.15.28:0/680872973 conn(0x7fb6b00fbbf0 msgr2=0x7fb6b00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.253+0000 7fb6b5341640 1 -- 172.21.15.28:0/680872973 shutdown_connections 2024-04-25T01:32:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.253+0000 7fb6b5341640 1 -- 172.21.15.28:0/680872973 wait complete. 2024-04-25T01:32:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.253+0000 7fb6b5341640 1 Processor -- start 2024-04-25T01:32:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.253+0000 7fb6b5341640 1 -- start start 2024-04-25T01:32:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6b5341640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0100680 0x7fb6b0197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6b5341640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6b0101870 0x7fb6b0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6b5341640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6b0102230 0x7fb6b019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6b5341640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b010c300 con 0x7fb6b0100680 2024-04-25T01:32:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6b5341640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b010c180 con 0x7fb6b0101870 2024-04-25T01:32:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6b5341640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b010c480 con 0x7fb6b0102230 2024-04-25T01:32:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6aeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0100680 0x7fb6b0197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6aeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0100680 0x7fb6b0197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45482/0 (socket says 172.21.15.28:45482) 2024-04-25T01:32:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6aeffd640 1 -- 172.21.15.28:0/820231044 learned_addr learned my addr 172.21.15.28:0/820231044 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6ae7fc640 1 --2- 172.21.15.28:0/820231044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6b0101870 0x7fb6b0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6aeffd640 1 -- 172.21.15.28:0/820231044 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6b0102230 msgr2=0x7fb6b019f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6af7fe640 1 --2- 172.21.15.28:0/820231044 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6b0102230 0x7fb6b019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6aeffd640 1 --2- 172.21.15.28:0/820231044 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6b0102230 0x7fb6b019f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6aeffd640 1 -- 172.21.15.28:0/820231044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6b0101870 msgr2=0x7fb6b0197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6aeffd640 1 --2- 172.21.15.28:0/820231044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6b0101870 0x7fb6b0197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6aeffd640 1 -- 172.21.15.28:0/820231044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6b019f6b0 con 0x7fb6b0100680 2024-04-25T01:32:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6aeffd640 1 --2- 172.21.15.28:0/820231044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0100680 0x7fb6b0197a90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb6a800eb70 tx=0x7fb6a800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6a7fff640 1 -- 172.21.15.28:0/820231044 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8003b70 con 0x7fb6b0100680 2024-04-25T01:32:26.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6b5341640 1 -- 172.21.15.28:0/820231044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6b019f9a0 con 0x7fb6b0100680 2024-04-25T01:32:26.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6a7fff640 1 -- 172.21.15.28:0/820231044 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6a800ee70 con 0x7fb6b0100680 2024-04-25T01:32:26.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6a7fff640 1 -- 172.21.15.28:0/820231044 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8010830 con 0x7fb6b0100680 2024-04-25T01:32:26.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.254+0000 7fb6b5341640 1 -- 172.21.15.28:0/820231044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6b019fef0 con 0x7fb6b0100680 2024-04-25T01:32:26.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.257+0000 7fb6a7fff640 1 -- 172.21.15.28:0/820231044 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb6a8004020 con 0x7fb6b0100680 2024-04-25T01:32:26.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.257+0000 7fb6b8045740 1 Processor -- start 2024-04-25T01:32:26.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.257+0000 7fb6b8045740 1 -- start start 2024-04-25T01:32:26.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.257+0000 7fb6b8045740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9e8440 0x55b88c9e8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.257+0000 7fb6b8045740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9e8e30 0x55b88c9ed930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.257+0000 7fb6b8045740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9edfa0 0x55b88c9f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.257+0000 7fb6b8045740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b88c94bf00 con 0x55b88c9e8e30 2024-04-25T01:32:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.257+0000 7fb6b8045740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b88c8f4750 con 0x55b88c9edfa0 2024-04-25T01:32:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.257+0000 7fb6b8045740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b88c947ec0 con 0x55b88c9e8440 2024-04-25T01:32:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.257+0000 7fb6aeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9e8e30 0x55b88c9ed930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.257+0000 7fb6a7fff640 1 --2- 172.21.15.28:0/820231044 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb69c0a2b90 0x7fb69c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.257+0000 7fb6aeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9e8e30 0x55b88c9ed930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45490/0 (socket says 172.21.15.28:45490) 2024-04-25T01:32:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.257+0000 7fb6aeffd640 1 -- 172.21.15.28:0/3792371906 learned_addr learned my addr 172.21.15.28:0/3792371906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.258+0000 7fb6af7fe640 1 --2- 172.21.15.28:0/3792371906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9e8440 0x55b88c9e8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.258+0000 7fb6ae7fc640 1 --2- 172.21.15.28:0/3792371906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9edfa0 0x55b88c9f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.258+0000 7fb6af7fe640 1 --2- 172.21.15.28:0/820231044 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb69c0a2b90 0x7fb69c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.258+0000 7fb6a7fff640 1 -- 172.21.15.28:0/820231044 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fb6a80cfb70 con 0x7fb6b0100680 2024-04-25T01:32:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.258+0000 7fb6aeffd640 1 -- 172.21.15.28:0/3792371906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9e8440 msgr2=0x55b88c9e8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.258+0000 7fb6aeffd640 1 --2- 172.21.15.28:0/3792371906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9e8440 0x55b88c9e8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.258+0000 7fb6aeffd640 1 -- 172.21.15.28:0/3792371906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9edfa0 msgr2=0x55b88c9f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.258+0000 7fb6aeffd640 1 --2- 172.21.15.28:0/3792371906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9edfa0 0x55b88c9f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.258+0000 7fb6aeffd640 1 -- 172.21.15.28:0/3792371906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b88c6602d0 con 0x55b88c9e8e30 2024-04-25T01:32:26.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.258+0000 7fb6aeffd640 1 --2- 172.21.15.28:0/3792371906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9e8e30 0x55b88c9ed930 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80cd310 tx=0x7fb6a800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5446ccfe978ab40 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.258+0000 7fb6af7fe640 1 --2- 172.21.15.28:0/820231044 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb69c0a2b90 0x7fb69c0a5050 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb6a00314e0 tx=0x7fb6a0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.258+0000 7fb6a57fa640 1 -- 172.21.15.28:0/3792371906 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a809b9d0 con 0x55b88c9e8e30 2024-04-25T01:32:26.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.258+0000 7fb6a57fa640 1 -- 172.21.15.28:0/3792371906 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6a809c770 con 0x55b88c9e8e30 2024-04-25T01:32:26.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.258+0000 7fb6a57fa640 1 -- 172.21.15.28:0/3792371906 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8016d90 con 0x55b88c9e8e30 2024-04-25T01:32:26.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.259+0000 7fb6b8045740 1 -- 172.21.15.28:0/3792371906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9e8e30 msgr2=0x55b88c9ed930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.259+0000 7fb6b8045740 1 --2- 172.21.15.28:0/3792371906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9e8e30 0x55b88c9ed930 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80cd310 tx=0x7fb6a800cf90 comp rx=0 tx=0).stop 2024-04-25T01:32:26.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.259+0000 7fb6b8045740 1 -- 172.21.15.28:0/3792371906 shutdown_connections 2024-04-25T01:32:26.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.259+0000 7fb6b8045740 1 --2- 172.21.15.28:0/3792371906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9edfa0 0x55b88c9f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.259+0000 7fb6b8045740 1 --2- 172.21.15.28:0/3792371906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9e8e30 0x55b88c9ed930 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.259+0000 7fb6b8045740 1 --2- 172.21.15.28:0/3792371906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9e8440 0x55b88c9e8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.259+0000 7fb6b8045740 1 -- 172.21.15.28:0/3792371906 >> 172.21.15.28:0/3792371906 conn(0x55b88c9e3ab0 msgr2=0x55b88c9e3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.259+0000 7fb6b8045740 1 -- 172.21.15.28:0/3792371906 shutdown_connections 2024-04-25T01:32:26.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.259+0000 7fb6b8045740 1 -- 172.21.15.28:0/3792371906 wait complete. 2024-04-25T01:32:26.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.259+0000 7fb6b8045740 1 -- start start 2024-04-25T01:32:26.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.259+0000 7fb6b8045740 1 Processor -- start 2024-04-25T01:32:26.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6b8045740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9e8440 0x55b88c9f2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6b8045740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9e8e30 0x55b88c9f2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6b8045740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9edfa0 0x55b88c9f9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6b8045740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b88c9f0ef0 con 0x55b88c9e8e30 2024-04-25T01:32:26.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6b8045740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b88c907cf0 con 0x55b88c9e8440 2024-04-25T01:32:26.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6b8045740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b88c9f1070 con 0x55b88c9edfa0 2024-04-25T01:32:26.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6b8045740 10 client.0 _ll_register_callbacks cb 0x55b88c9076e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:26.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6ae7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9e8e30 0x55b88c9f2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6ae7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9e8e30 0x55b88c9f2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45492/0 (socket says 172.21.15.28:45492) 2024-04-25T01:32:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6ae7fc640 1 -- 172.21.15.28:0/3733836453 learned_addr learned my addr 172.21.15.28:0/3733836453 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6ae7fc640 1 -- 172.21.15.28:0/3733836453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9edfa0 msgr2=0x55b88c9f9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6aeffd640 1 --2- 172.21.15.28:0/3733836453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9e8440 0x55b88c9f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6af7fe640 1 --2- 172.21.15.28:0/3733836453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9edfa0 0x55b88c9f9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6ae7fc640 1 --2- 172.21.15.28:0/3733836453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9edfa0 0x55b88c9f9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6ae7fc640 1 -- 172.21.15.28:0/3733836453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9e8440 msgr2=0x55b88c9f2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6ae7fc640 1 --2- 172.21.15.28:0/3733836453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9e8440 0x55b88c9f2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6ae7fc640 1 -- 172.21.15.28:0/3733836453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b88c72a6b0 con 0x55b88c9e8e30 2024-04-25T01:32:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6aeffd640 1 --2- 172.21.15.28:0/3733836453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9e8440 0x55b88c9f2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6ae7fc640 1 --2- 172.21.15.28:0/3733836453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9e8e30 0x55b88c9f2e60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb698013710 tx=0x7fb698013bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5caa913e11d3582 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6967fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6967fc640 1 -- 172.21.15.28:0/3733836453 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb698011d70 con 0x55b88c9e8e30 2024-04-25T01:32:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6967fc640 1 -- 172.21.15.28:0/3733836453 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fb698004030 con 0x55b88c9e8e30 2024-04-25T01:32:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.260+0000 7fb6b8045740 1 -- 172.21.15.28:0/3733836453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6a809c770 con 0x55b88c9e8e30 2024-04-25T01:32:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.261+0000 7fb6967fc640 1 -- 172.21.15.28:0/3733836453 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb698004230 con 0x55b88c9e8e30 2024-04-25T01:32:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.261+0000 7fb6b8045740 10 client.15249 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.261+0000 7fb6b8045740 1 -- 172.21.15.28:0/3733836453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b88c9fcee0 con 0x55b88c9e8e30 2024-04-25T01:32:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.261+0000 7fb6967fc640 1 -- 172.21.15.28:0/3733836453 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6a809c770 con 0x55b88c9e8e30 2024-04-25T01:32:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.261+0000 7fb6967fc640 1 -- 172.21.15.28:0/3733836453 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fb69801a070 con 0x55b88c9e8e30 2024-04-25T01:32:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.261+0000 7fb6b8045740 10 client.15249 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.261+0000 7fb6b8045740 10 client.15249 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T01:32:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.261+0000 7fb6b8045740 20 client.15249 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.261+0000 7fb6b8045740 1 --2- 172.21.15.28:0/3733836453 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55b88c9ff5f0 0x55b88ca00a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.261+0000 7fb6b8045740 4 client.15249 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.261+0000 7fb6b8045740 1 -- 172.21.15.28:0/3733836453 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b88c959440 con 0x55b88c9ff5f0 2024-04-25T01:32:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.261+0000 7fb6aeffd640 1 --2- 172.21.15.28:0/3733836453 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55b88c9ff5f0 0x55b88ca00a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.262+0000 7fb6aeffd640 1 --2- 172.21.15.28:0/3733836453 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55b88c9ff5f0 0x55b88ca00a60 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6c47eef81c18853 server_cookie=7824a187f9410ccb in_seq=0 out_seq=0 2024-04-25T01:32:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.262+0000 7fb6967fc640 10 client.15249 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:32:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.262+0000 7fb6967fc640 1 -- 172.21.15.28:0/3733836453 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b88c959440 con 0x55b88c9ff5f0 2024-04-25T01:32:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.262+0000 7fb6967fc640 10 client.15249 handle_command_reply: tid=0 2024-04-25T01:32:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.263+0000 7fb6b8045740 1 client.15249 shutdown 2024-04-25T01:32:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.263+0000 7fb6b8045740 1 -- 172.21.15.28:0/3733836453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9e8e30 msgr2=0x55b88c9f2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.263+0000 7fb6b8045740 1 --2- 172.21.15.28:0/3733836453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9e8e30 0x55b88c9f2e60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb698013710 tx=0x7fb698013bd0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.263+0000 7fb6b8045740 1 -- 172.21.15.28:0/3733836453 shutdown_connections 2024-04-25T01:32:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.263+0000 7fb6b8045740 1 --2- 172.21.15.28:0/3733836453 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55b88c9ff5f0 0x55b88ca00a60 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.263+0000 7fb6b8045740 1 --2- 172.21.15.28:0/3733836453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9edfa0 0x55b88c9f9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.263+0000 7fb6b8045740 1 --2- 172.21.15.28:0/3733836453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9e8e30 0x55b88c9f2e60 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.263+0000 7fb6967fc640 0 client.15249 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:32:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.263+0000 7fb6b8045740 1 --2- 172.21.15.28:0/3733836453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9e8440 0x55b88c9f2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.263+0000 7fb6b8045740 1 -- 172.21.15.28:0/3733836453 >> 172.21.15.28:0/3733836453 conn(0x55b88c9e3ab0 msgr2=0x55b88c9e2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.263+0000 7fb6b8045740 1 -- 172.21.15.28:0/3733836453 shutdown_connections 2024-04-25T01:32:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.263+0000 7fb6b8045740 1 -- 172.21.15.28:0/3733836453 wait complete. 2024-04-25T01:32:26.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.263+0000 7fb6b8045740 20 client.15249 trim_cache size 0 max 16384 2024-04-25T01:32:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.281+0000 7fb6b8045740 1 Processor -- start 2024-04-25T01:32:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.281+0000 7fb6b8045740 1 -- start start 2024-04-25T01:32:26.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6b8045740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9e8440 0x55b88c9f2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6b8045740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9e8e30 0x55b88c9f9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6b8045740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9edfa0 0x55b88c9f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6b8045740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b88c9feda0 con 0x55b88c9edfa0 2024-04-25T01:32:26.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6b8045740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b88c9ff470 con 0x55b88c9e8e30 2024-04-25T01:32:26.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6b8045740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b88c9ffb40 con 0x55b88c9e8440 2024-04-25T01:32:26.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6af7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9edfa0 0x55b88c9f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6af7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9edfa0 0x55b88c9f2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45502/0 (socket says 172.21.15.28:45502) 2024-04-25T01:32:26.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6af7fe640 1 -- 172.21.15.28:0/3776447084 learned_addr learned my addr 172.21.15.28:0/3776447084 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6aeffd640 1 --2- 172.21.15.28:0/3776447084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9e8440 0x55b88c9f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6af7fe640 1 -- 172.21.15.28:0/3776447084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9e8440 msgr2=0x55b88c9f2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6af7fe640 1 --2- 172.21.15.28:0/3776447084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9e8440 0x55b88c9f2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6ae7fc640 1 --2- 172.21.15.28:0/3776447084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9e8e30 0x55b88c9f9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6af7fe640 1 -- 172.21.15.28:0/3776447084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9e8e30 msgr2=0x55b88c9f9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6af7fe640 1 --2- 172.21.15.28:0/3776447084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9e8e30 0x55b88c9f9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6af7fe640 1 -- 172.21.15.28:0/3776447084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b88c9f39e0 con 0x55b88c9edfa0 2024-04-25T01:32:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6ae7fc640 1 --2- 172.21.15.28:0/3776447084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9e8e30 0x55b88c9f9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6af7fe640 1 --2- 172.21.15.28:0/3776447084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9edfa0 0x55b88c9f2a70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb6a002a930 tx=0x7fb6a0031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913b7f3d6d51a0cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6a4ff9640 1 -- 172.21.15.28:0/3776447084 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a0035040 con 0x55b88c9edfa0 2024-04-25T01:32:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6a4ff9640 1 -- 172.21.15.28:0/3776447084 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6a0051070 con 0x55b88c9edfa0 2024-04-25T01:32:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.282+0000 7fb6a4ff9640 1 -- 172.21.15.28:0/3776447084 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a004c6e0 con 0x55b88c9edfa0 2024-04-25T01:32:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.283+0000 7fb6b8045740 1 -- 172.21.15.28:0/3776447084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9edfa0 msgr2=0x55b88c9f2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.283+0000 7fb6b8045740 1 --2- 172.21.15.28:0/3776447084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9edfa0 0x55b88c9f2a70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb6a002a930 tx=0x7fb6a0031400 comp rx=0 tx=0).stop 2024-04-25T01:32:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.283+0000 7fb6b8045740 1 -- 172.21.15.28:0/3776447084 shutdown_connections 2024-04-25T01:32:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.283+0000 7fb6b8045740 1 --2- 172.21.15.28:0/3776447084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9edfa0 0x55b88c9f2a70 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.283+0000 7fb6b8045740 1 --2- 172.21.15.28:0/3776447084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9e8e30 0x55b88c9f9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.283+0000 7fb6b8045740 1 --2- 172.21.15.28:0/3776447084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9e8440 0x55b88c9f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.283+0000 7fb6b8045740 1 -- 172.21.15.28:0/3776447084 >> 172.21.15.28:0/3776447084 conn(0x55b88c9e3ab0 msgr2=0x55b88c9de690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.283+0000 7fb6b8045740 1 -- 172.21.15.28:0/3776447084 shutdown_connections 2024-04-25T01:32:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.283+0000 7fb6b8045740 1 -- 172.21.15.28:0/3776447084 wait complete. 2024-04-25T01:32:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.283+0000 7fb6b8045740 1 -- start start 2024-04-25T01:32:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.283+0000 7fb6b8045740 1 Processor -- start 2024-04-25T01:32:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6b8045740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9e8440 0x55b88c9f60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6b8045740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9e8e30 0x55b88c9f65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6b8045740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9edfa0 0x55b88c9f9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6b8045740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b88ca008e0 con 0x55b88c9edfa0 2024-04-25T01:32:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6b8045740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b88ca00210 con 0x55b88c9e8440 2024-04-25T01:32:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6b8045740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b88ca00fb0 con 0x55b88c9e8e30 2024-04-25T01:32:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6b8045740 10 client.0 _ll_register_callbacks cb 0x7fb698009d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6af7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9edfa0 0x55b88c9f9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6af7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9edfa0 0x55b88c9f9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45510/0 (socket says 172.21.15.28:45510) 2024-04-25T01:32:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6af7fe640 1 -- 172.21.15.28:0/2824977274 learned_addr learned my addr 172.21.15.28:0/2824977274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6ae7fc640 1 --2- 172.21.15.28:0/2824977274 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9e8e30 0x55b88c9f65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6aeffd640 1 --2- 172.21.15.28:0/2824977274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9e8440 0x55b88c9f60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6af7fe640 1 -- 172.21.15.28:0/2824977274 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9e8e30 msgr2=0x55b88c9f65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6af7fe640 1 --2- 172.21.15.28:0/2824977274 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9e8e30 0x55b88c9f65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6af7fe640 1 -- 172.21.15.28:0/2824977274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9e8440 msgr2=0x55b88c9f60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6af7fe640 1 --2- 172.21.15.28:0/2824977274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9e8440 0x55b88c9f60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6af7fe640 1 -- 172.21.15.28:0/2824977274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b88c9f9bf0 con 0x55b88c9edfa0 2024-04-25T01:32:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6aeffd640 1 --2- 172.21.15.28:0/2824977274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9e8440 0x55b88c9f60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:26.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6af7fe640 1 --2- 172.21.15.28:0/2824977274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9edfa0 0x55b88c9f9600 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0004050 tx=0x7fb6a0004080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e3e64c2eb9ab31b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6967fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6967fc640 1 -- 172.21.15.28:0/2824977274 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a0035030 con 0x55b88c9edfa0 2024-04-25T01:32:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6967fc640 1 -- 172.21.15.28:0/2824977274 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fb6a004c480 con 0x55b88c9edfa0 2024-04-25T01:32:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.284+0000 7fb6b8045740 1 -- 172.21.15.28:0/2824977274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6a0051070 con 0x55b88c9edfa0 2024-04-25T01:32:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.285+0000 7fb6967fc640 1 -- 172.21.15.28:0/2824977274 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb6a0068050 con 0x55b88c9edfa0 2024-04-25T01:32:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.285+0000 7fb6b8045740 10 client.15255 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.285+0000 7fb6b8045740 1 -- 172.21.15.28:0/2824977274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b88ca66030 con 0x55b88c9edfa0 2024-04-25T01:32:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.285+0000 7fb6967fc640 1 -- 172.21.15.28:0/2824977274 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6a0051070 con 0x55b88c9edfa0 2024-04-25T01:32:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.285+0000 7fb6967fc640 1 -- 172.21.15.28:0/2824977274 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fb6a003a030 con 0x55b88c9edfa0 2024-04-25T01:32:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.285+0000 7fb6b8045740 10 client.15255 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.285+0000 7fb6b8045740 10 client.15255 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T01:32:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.285+0000 7fb6b8045740 20 client.15255 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.285+0000 7fb6b8045740 1 --2- 172.21.15.28:0/2824977274 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55b88ca6a800 0x55b88ca69510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.285+0000 7fb6b8045740 4 client.15255 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:32:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.285+0000 7fb6b8045740 1 -- 172.21.15.28:0/2824977274 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "status"}) -- 0x55b88c953790 con 0x55b88ca6a800 2024-04-25T01:32:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.285+0000 7fb6aeffd640 1 --2- 172.21.15.28:0/2824977274 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55b88ca6a800 0x55b88ca69510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.286+0000 7fb6967fc640 10 client.15255 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:32:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.286+0000 7fb6aeffd640 1 --2- 172.21.15.28:0/2824977274 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55b88ca6a800 0x55b88ca69510 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=50b483793376455b server_cookie=f85f5c2397ba9daa in_seq=0 out_seq=0 2024-04-25T01:32:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.286+0000 7fb6967fc640 1 -- 172.21.15.28:0/2824977274 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b88c953790 con 0x55b88ca6a800 2024-04-25T01:32:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.286+0000 7fb6967fc640 10 client.15255 handle_command_reply: tid=0 2024-04-25T01:32:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.286+0000 7fb6b8045740 1 client.15255 shutdown 2024-04-25T01:32:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.286+0000 7fb6b8045740 1 -- 172.21.15.28:0/2824977274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9edfa0 msgr2=0x55b88c9f9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.286+0000 7fb6b8045740 1 --2- 172.21.15.28:0/2824977274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9edfa0 0x55b88c9f9600 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0004050 tx=0x7fb6a0004080 comp rx=0 tx=0).stop 2024-04-25T01:32:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.286+0000 7fb6b8045740 1 -- 172.21.15.28:0/2824977274 shutdown_connections 2024-04-25T01:32:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.286+0000 7fb6b8045740 1 --2- 172.21.15.28:0/2824977274 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55b88ca6a800 0x55b88ca69510 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.286+0000 7fb6b8045740 1 --2- 172.21.15.28:0/2824977274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b88c9edfa0 0x55b88c9f9600 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.286+0000 7fb6b8045740 1 --2- 172.21.15.28:0/2824977274 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b88c9e8e30 0x55b88c9f65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.286+0000 7fb6b8045740 1 --2- 172.21.15.28:0/2824977274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b88c9e8440 0x55b88c9f60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.286+0000 7fb6967fc640 0 client.15255 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:32:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.286+0000 7fb6b8045740 1 -- 172.21.15.28:0/2824977274 >> 172.21.15.28:0/2824977274 conn(0x55b88c9e3ab0 msgr2=0x55b88c9f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.286+0000 7fb6b8045740 1 -- 172.21.15.28:0/2824977274 shutdown_connections 2024-04-25T01:32:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.286+0000 7fb6b8045740 1 -- 172.21.15.28:0/2824977274 wait complete. 2024-04-25T01:32:26.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.286+0000 7fb6b8045740 20 client.15255 trim_cache size 0 max 16384 2024-04-25T01:32:26.271 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:32:26.271 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "0e88f60a-02a2-11ef-bc93-c7b262605968", 2024-04-25T01:32:26.271 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 4, 2024-04-25T01:32:26.271 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24442, 2024-04-25T01:32:26.271 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-25T01:32:26.272 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-25T01:32:26.272 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-25T01:32:26.272 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 84.837572926000007, 2024-04-25T01:32:26.272 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-04-25T01:32:26.272 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-04-25T01:32:26.272 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 91, 2024-04-25T01:32:26.272 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 135.636461191 2024-04-25T01:32:26.272 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:32:26.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.287+0000 7fb6b5341640 1 -- 172.21.15.28:0/820231044 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb69c0a2b90 msgr2=0x7fb69c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.287+0000 7fb6b5341640 1 --2- 172.21.15.28:0/820231044 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb69c0a2b90 0x7fb69c0a5050 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb6a00314e0 tx=0x7fb6a0005fd0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.287+0000 7fb6b5341640 1 -- 172.21.15.28:0/820231044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0100680 msgr2=0x7fb6b0197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.287+0000 7fb6b5341640 1 --2- 172.21.15.28:0/820231044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0100680 0x7fb6b0197a90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb6a800eb70 tx=0x7fb6a800c6a0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.287+0000 7fb6b5341640 1 -- 172.21.15.28:0/820231044 shutdown_connections 2024-04-25T01:32:26.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.287+0000 7fb6b5341640 1 --2- 172.21.15.28:0/820231044 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb69c0a2b90 0x7fb69c0a5050 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.287+0000 7fb6b5341640 1 --2- 172.21.15.28:0/820231044 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb6b0102230 0x7fb6b019f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.287+0000 7fb6b5341640 1 --2- 172.21.15.28:0/820231044 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6b0101870 0x7fb6b0197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.287+0000 7fb6b5341640 1 --2- 172.21.15.28:0/820231044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b0100680 0x7fb6b0197a90 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.287+0000 7fb6b5341640 1 -- 172.21.15.28:0/820231044 >> 172.21.15.28:0/820231044 conn(0x7fb6b00fbbf0 msgr2=0x7fb6b00fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.287+0000 7fb6b5341640 1 -- 172.21.15.28:0/820231044 shutdown_connections 2024-04-25T01:32:26.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.287+0000 7fb6b5341640 1 -- 172.21.15.28:0/820231044 wait complete. 2024-04-25T01:32:26.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T01:32:26.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.403+0000 7f17a80c1640 1 -- 172.21.15.28:0/1940237902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17a0102230 msgr2=0x7f17a0108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.403+0000 7f17a80c1640 1 --2- 172.21.15.28:0/1940237902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17a0102230 0x7f17a0108ac0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1798009ca0 tx=0x7f179802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.403+0000 7f17a80c1640 1 -- 172.21.15.28:0/1940237902 shutdown_connections 2024-04-25T01:32:26.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.403+0000 7f17a80c1640 1 --2- 172.21.15.28:0/1940237902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17a0102230 0x7f17a0108ac0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.403+0000 7f17a80c1640 1 --2- 172.21.15.28:0/1940237902 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17a0101870 0x7f17a0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.403+0000 7f17a80c1640 1 --2- 172.21.15.28:0/1940237902 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17a0100680 0x7f17a0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.403+0000 7f17a80c1640 1 -- 172.21.15.28:0/1940237902 >> 172.21.15.28:0/1940237902 conn(0x7f17a00fbbf0 msgr2=0x7f17a00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.403+0000 7f17a80c1640 1 -- 172.21.15.28:0/1940237902 shutdown_connections 2024-04-25T01:32:26.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.403+0000 7f17a80c1640 1 -- 172.21.15.28:0/1940237902 wait complete. 2024-04-25T01:32:26.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.404+0000 7f17a80c1640 1 Processor -- start 2024-04-25T01:32:26.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.404+0000 7f17a80c1640 1 -- start start 2024-04-25T01:32:26.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.404+0000 7f17a80c1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17a0100680 0x7f17a0196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.404+0000 7f17a80c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17a0101870 0x7f17a0196eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.404+0000 7f17a80c1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17a0102230 0x7f17a019df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.404+0000 7f17a80c1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17a010c300 con 0x7f17a0101870 2024-04-25T01:32:26.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.404+0000 7f17a80c1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f17a010c180 con 0x7f17a0100680 2024-04-25T01:32:26.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.404+0000 7f17a80c1640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f17a010c480 con 0x7f17a0102230 2024-04-25T01:32:26.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.404+0000 7f17a565c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17a0101870 0x7f17a0196eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.404+0000 7f17a565c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17a0101870 0x7f17a0196eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45540/0 (socket says 172.21.15.28:45540) 2024-04-25T01:32:26.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.404+0000 7f17a565c640 1 -- 172.21.15.28:0/3788304533 learned_addr learned my addr 172.21.15.28:0/3788304533 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.404+0000 7f17a5e5d640 1 --2- 172.21.15.28:0/3788304533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17a0100680 0x7f17a0196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.404+0000 7f17a665e640 1 --2- 172.21.15.28:0/3788304533 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17a0102230 0x7f17a019df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.405+0000 7f17a565c640 1 -- 172.21.15.28:0/3788304533 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17a0102230 msgr2=0x7f17a019df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.405+0000 7f17a565c640 1 --2- 172.21.15.28:0/3788304533 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17a0102230 0x7f17a019df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.405+0000 7f17a565c640 1 -- 172.21.15.28:0/3788304533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17a0100680 msgr2=0x7f17a0196970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.405+0000 7f17a565c640 1 --2- 172.21.15.28:0/3788304533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17a0100680 0x7f17a0196970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.405+0000 7f17a565c640 1 -- 172.21.15.28:0/3788304533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a019e590 con 0x7f17a0101870 2024-04-25T01:32:26.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.405+0000 7f17a565c640 1 --2- 172.21.15.28:0/3788304533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17a0101870 0x7f17a0196eb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f179000d770 tx=0x7f179000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.405+0000 7f1796ffd640 1 -- 172.21.15.28:0/3788304533 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1790017070 con 0x7f17a0101870 2024-04-25T01:32:26.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.405+0000 7f1796ffd640 1 -- 172.21.15.28:0/3788304533 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1790005500 con 0x7f17a0101870 2024-04-25T01:32:26.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.405+0000 7f17a80c1640 1 -- 172.21.15.28:0/3788304533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17a019e880 con 0x7f17a0101870 2024-04-25T01:32:26.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.405+0000 7f1796ffd640 1 -- 172.21.15.28:0/3788304533 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1790004e00 con 0x7f17a0101870 2024-04-25T01:32:26.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.405+0000 7f17a80c1640 1 -- 172.21.15.28:0/3788304533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17a019edd0 con 0x7f17a0101870 2024-04-25T01:32:26.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.407+0000 7f1796ffd640 1 -- 172.21.15.28:0/3788304533 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1790004fa0 con 0x7f17a0101870 2024-04-25T01:32:26.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.408+0000 7f17aadcd740 1 Processor -- start 2024-04-25T01:32:26.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.408+0000 7f17aadcd740 1 -- start start 2024-04-25T01:32:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.408+0000 7f17aadcd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32443440 0x55fa32443860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.408+0000 7f17aadcd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32443e30 0x55fa32448930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.408+0000 7f17aadcd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32448fa0 0x55fa3244b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.408+0000 7f17aadcd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fa323a6f00 con 0x55fa32448fa0 2024-04-25T01:32:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.408+0000 7f17aadcd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fa3234f750 con 0x55fa32443e30 2024-04-25T01:32:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.408+0000 7f17aadcd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55fa323a2ec0 con 0x55fa32443440 2024-04-25T01:32:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.408+0000 7f17a565c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32448fa0 0x55fa3244b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.408+0000 7f1796ffd640 1 --2- 172.21.15.28:0/3788304533 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f178c0a2c10 0x7f178c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.408+0000 7f17a565c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32448fa0 0x55fa3244b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45552/0 (socket says 172.21.15.28:45552) 2024-04-25T01:32:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.408+0000 7f17a565c640 1 -- 172.21.15.28:0/4006296796 learned_addr learned my addr 172.21.15.28:0/4006296796 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.408+0000 7f17a5e5d640 1 --2- 172.21.15.28:0/4006296796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32443e30 0x55fa32448930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.408+0000 7f17a665e640 1 --2- 172.21.15.28:0/4006296796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32443440 0x55fa32443860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.408+0000 7f17a665e640 1 --2- 172.21.15.28:0/3788304533 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f178c0a2c10 0x7f178c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.408+0000 7f1796ffd640 1 -- 172.21.15.28:0/3788304533 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f17900cecc0 con 0x7f17a0101870 2024-04-25T01:32:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.409+0000 7f17a5e5d640 1 -- 172.21.15.28:0/4006296796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32443440 msgr2=0x55fa32443860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.409+0000 7f17a5e5d640 1 --2- 172.21.15.28:0/4006296796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32443440 0x55fa32443860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.409+0000 7f17a5e5d640 1 -- 172.21.15.28:0/4006296796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32448fa0 msgr2=0x55fa3244b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.409+0000 7f17a5e5d640 1 --2- 172.21.15.28:0/4006296796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32448fa0 0x55fa3244b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.409+0000 7f17a5e5d640 1 -- 172.21.15.28:0/4006296796 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa320bb2d0 con 0x55fa32443e30 2024-04-25T01:32:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.409+0000 7f17a565c640 1 --2- 172.21.15.28:0/4006296796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32448fa0 0x55fa3244b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.409+0000 7f17a665e640 1 --2- 172.21.15.28:0/3788304533 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f178c0a2c10 0x7f178c0a50d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x55fa32444690 tx=0x7f1798005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.409+0000 7f17a5e5d640 1 --2- 172.21.15.28:0/4006296796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32443e30 0x55fa32448930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f179c00bbc0 tx=0x7f179c0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cf8c4182c43a524 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.409+0000 7f178bfff640 1 -- 172.21.15.28:0/4006296796 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c004030 con 0x55fa32443e30 2024-04-25T01:32:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.409+0000 7f178bfff640 1 -- 172.21.15.28:0/4006296796 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f179c005500 con 0x55fa32443e30 2024-04-25T01:32:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.409+0000 7f178bfff640 1 -- 172.21.15.28:0/4006296796 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c004c00 con 0x55fa32443e30 2024-04-25T01:32:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.410+0000 7f17aadcd740 1 -- 172.21.15.28:0/4006296796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32443e30 msgr2=0x55fa32448930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.410+0000 7f17aadcd740 1 --2- 172.21.15.28:0/4006296796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32443e30 0x55fa32448930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f179c00bbc0 tx=0x7f179c0107a0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.410+0000 7f17aadcd740 1 -- 172.21.15.28:0/4006296796 shutdown_connections 2024-04-25T01:32:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.410+0000 7f17aadcd740 1 --2- 172.21.15.28:0/4006296796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32448fa0 0x55fa3244b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.410+0000 7f17aadcd740 1 --2- 172.21.15.28:0/4006296796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32443e30 0x55fa32448930 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.410+0000 7f17aadcd740 1 --2- 172.21.15.28:0/4006296796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32443440 0x55fa32443860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.410+0000 7f17aadcd740 1 -- 172.21.15.28:0/4006296796 >> 172.21.15.28:0/4006296796 conn(0x55fa3243eab0 msgr2=0x55fa3243eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.410+0000 7f17aadcd740 1 -- 172.21.15.28:0/4006296796 shutdown_connections 2024-04-25T01:32:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.410+0000 7f17aadcd740 1 -- 172.21.15.28:0/4006296796 wait complete. 2024-04-25T01:32:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.410+0000 7f17aadcd740 1 -- start start 2024-04-25T01:32:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.410+0000 7f17aadcd740 1 Processor -- start 2024-04-25T01:32:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.411+0000 7f17aadcd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32443440 0x55fa3244fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.411+0000 7f17aadcd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32443e30 0x55fa3244ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.411+0000 7f17aadcd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32448fa0 0x55fa32456fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.411+0000 7f17aadcd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fa3244e160 con 0x55fa32443e30 2024-04-25T01:32:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.411+0000 7f17aadcd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fa32362cf0 con 0x55fa32448fa0 2024-04-25T01:32:26.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.411+0000 7f17aadcd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55fa3244e2e0 con 0x55fa32443440 2024-04-25T01:32:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.411+0000 7f17aadcd740 10 client.0 _ll_register_callbacks cb 0x55fa323626e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.411+0000 7f17a565c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32443e30 0x55fa3244ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.411+0000 7f17a565c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32443e30 0x55fa3244ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45564/0 (socket says 172.21.15.28:45564) 2024-04-25T01:32:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.411+0000 7f17a565c640 1 -- 172.21.15.28:0/2327213725 learned_addr learned my addr 172.21.15.28:0/2327213725 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.411+0000 7f17a665e640 1 --2- 172.21.15.28:0/2327213725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32448fa0 0x55fa32456fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.411+0000 7f17a5e5d640 1 --2- 172.21.15.28:0/2327213725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32443440 0x55fa3244fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.411+0000 7f17a565c640 1 -- 172.21.15.28:0/2327213725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32443440 msgr2=0x55fa3244fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.411+0000 7f17a565c640 1 --2- 172.21.15.28:0/2327213725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32443440 0x55fa3244fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.411+0000 7f17a565c640 1 -- 172.21.15.28:0/2327213725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32448fa0 msgr2=0x55fa32456fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.411+0000 7f17a565c640 1 --2- 172.21.15.28:0/2327213725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32448fa0 0x55fa32456fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.411+0000 7f17a565c640 1 -- 172.21.15.28:0/2327213725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa321856b0 con 0x55fa32443e30 2024-04-25T01:32:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.411+0000 7f17a565c640 1 --2- 172.21.15.28:0/2327213725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32443e30 0x55fa3244ff80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1790021000 tx=0x7f17900048a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3d063d39d55e1dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.411+0000 7f17897fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.412+0000 7f17897fa640 1 -- 172.21.15.28:0/2327213725 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17900d0030 con 0x55fa32443e30 2024-04-25T01:32:26.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.412+0000 7f17897fa640 1 -- 172.21.15.28:0/2327213725 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f1790099570 con 0x55fa32443e30 2024-04-25T01:32:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.412+0000 7f17aadcd740 1 -- 172.21.15.28:0/2327213725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f179c005500 con 0x55fa32443e30 2024-04-25T01:32:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.412+0000 7f17897fa640 1 -- 172.21.15.28:0/2327213725 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1790020040 con 0x55fa32443e30 2024-04-25T01:32:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.412+0000 7f17aadcd740 10 client.15267 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.412+0000 7f17aadcd740 1 -- 172.21.15.28:0/2327213725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fa3245a060 con 0x55fa32443e30 2024-04-25T01:32:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.412+0000 7f17897fa640 1 -- 172.21.15.28:0/2327213725 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f179c005500 con 0x55fa32443e30 2024-04-25T01:32:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.412+0000 7f17897fa640 1 -- 172.21.15.28:0/2327213725 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f17900bb0b0 con 0x55fa32443e30 2024-04-25T01:32:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.412+0000 7f17aadcd740 10 client.15267 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.412+0000 7f17aadcd740 10 client.15267 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T01:32:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.412+0000 7f17aadcd740 20 client.15267 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.412+0000 7f17aadcd740 1 --2- 172.21.15.28:0/2327213725 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55fa3245c700 0x55fa3245db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.412+0000 7f17aadcd740 4 client.15267 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.412+0000 7f17aadcd740 1 -- 172.21.15.28:0/2327213725 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa323b4440 con 0x55fa3245c700 2024-04-25T01:32:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.413+0000 7f17a5e5d640 1 --2- 172.21.15.28:0/2327213725 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55fa3245c700 0x55fa3245db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.413+0000 7f17897fa640 10 client.15267 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:32:26.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.413+0000 7f17a5e5d640 1 --2- 172.21.15.28:0/2327213725 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55fa3245c700 0x55fa3245db70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d940de6bfe94fe8e server_cookie=5b639f11e533958b in_seq=0 out_seq=0 2024-04-25T01:32:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.414+0000 7f17897fa640 1 -- 172.21.15.28:0/2327213725 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fa323b4440 con 0x55fa3245c700 2024-04-25T01:32:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.414+0000 7f17897fa640 10 client.15267 handle_command_reply: tid=0 2024-04-25T01:32:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.414+0000 7f17aadcd740 1 client.15267 shutdown 2024-04-25T01:32:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.414+0000 7f17aadcd740 1 -- 172.21.15.28:0/2327213725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32443e30 msgr2=0x55fa3244ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.414+0000 7f17aadcd740 1 --2- 172.21.15.28:0/2327213725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32443e30 0x55fa3244ff80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1790021000 tx=0x7f17900048a0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.414+0000 7f17aadcd740 1 -- 172.21.15.28:0/2327213725 shutdown_connections 2024-04-25T01:32:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.414+0000 7f17aadcd740 1 --2- 172.21.15.28:0/2327213725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32448fa0 0x55fa32456fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.414+0000 7f17aadcd740 1 --2- 172.21.15.28:0/2327213725 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55fa3245c700 0x55fa3245db70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.414+0000 7f17aadcd740 1 --2- 172.21.15.28:0/2327213725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32443e30 0x55fa3244ff80 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.414+0000 7f17aadcd740 1 --2- 172.21.15.28:0/2327213725 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32443440 0x55fa3244fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.414+0000 7f17aadcd740 1 -- 172.21.15.28:0/2327213725 >> 172.21.15.28:0/2327213725 conn(0x55fa3243eab0 msgr2=0x55fa3243dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.414+0000 7f17897fa640 0 client.15267 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:32:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.414+0000 7f17aadcd740 1 -- 172.21.15.28:0/2327213725 shutdown_connections 2024-04-25T01:32:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.414+0000 7f17aadcd740 1 -- 172.21.15.28:0/2327213725 wait complete. 2024-04-25T01:32:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.414+0000 7f17aadcd740 20 client.15267 trim_cache size 0 max 16384 2024-04-25T01:32:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.432+0000 7f17aadcd740 1 Processor -- start 2024-04-25T01:32:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.432+0000 7f17aadcd740 1 -- start start 2024-04-25T01:32:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f17aadcd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32443440 0x55fa3244b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f17aadcd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32443e30 0x55fa3244fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f17aadcd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32448fa0 0x55fa3244ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f17aadcd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fa3245beb0 con 0x55fa32448fa0 2024-04-25T01:32:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f17aadcd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fa3245c580 con 0x55fa32443440 2024-04-25T01:32:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f17aadcd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55fa3245cc50 con 0x55fa32443e30 2024-04-25T01:32:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f17a665e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32448fa0 0x55fa3244ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f17a665e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32448fa0 0x55fa3244ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45580/0 (socket says 172.21.15.28:45580) 2024-04-25T01:32:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f17a665e640 1 -- 172.21.15.28:0/4260860555 learned_addr learned my addr 172.21.15.28:0/4260860555 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f17a5e5d640 1 --2- 172.21.15.28:0/4260860555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32443440 0x55fa3244b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f17a565c640 1 --2- 172.21.15.28:0/4260860555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32443e30 0x55fa3244fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f17a665e640 1 -- 172.21.15.28:0/4260860555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32443e30 msgr2=0x55fa3244fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f17a665e640 1 --2- 172.21.15.28:0/4260860555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32443e30 0x55fa3244fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f17a665e640 1 -- 172.21.15.28:0/4260860555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32443440 msgr2=0x55fa3244b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f17a665e640 1 --2- 172.21.15.28:0/4260860555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32443440 0x55fa3244b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f17a665e640 1 -- 172.21.15.28:0/4260860555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa3243f780 con 0x55fa32448fa0 2024-04-25T01:32:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f17a5e5d640 1 --2- 172.21.15.28:0/4260860555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32443440 0x55fa3244b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f17a565c640 1 --2- 172.21.15.28:0/4260860555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32443e30 0x55fa3244fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f17a665e640 1 --2- 172.21.15.28:0/4260860555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32448fa0 0x55fa3244ff80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1798009ca0 tx=0x7f1798040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a500e35ba69c4164 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f178b7fe640 1 -- 172.21.15.28:0/4260860555 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1798059070 con 0x55fa32448fa0 2024-04-25T01:32:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f178b7fe640 1 -- 172.21.15.28:0/4260860555 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1798036070 con 0x55fa32448fa0 2024-04-25T01:32:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.433+0000 7f178b7fe640 1 -- 172.21.15.28:0/4260860555 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1798041050 con 0x55fa32448fa0 2024-04-25T01:32:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.434+0000 7f17aadcd740 1 -- 172.21.15.28:0/4260860555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32448fa0 msgr2=0x55fa3244ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.434+0000 7f17aadcd740 1 --2- 172.21.15.28:0/4260860555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32448fa0 0x55fa3244ff80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1798009ca0 tx=0x7f1798040000 comp rx=0 tx=0).stop 2024-04-25T01:32:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.434+0000 7f17aadcd740 1 -- 172.21.15.28:0/4260860555 shutdown_connections 2024-04-25T01:32:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.434+0000 7f17aadcd740 1 --2- 172.21.15.28:0/4260860555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32448fa0 0x55fa3244ff80 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.434+0000 7f17aadcd740 1 --2- 172.21.15.28:0/4260860555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32443e30 0x55fa3244fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.434+0000 7f17aadcd740 1 --2- 172.21.15.28:0/4260860555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32443440 0x55fa3244b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.434+0000 7f17aadcd740 1 -- 172.21.15.28:0/4260860555 >> 172.21.15.28:0/4260860555 conn(0x55fa3243eab0 msgr2=0x55fa32439690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.434+0000 7f17aadcd740 1 -- 172.21.15.28:0/4260860555 shutdown_connections 2024-04-25T01:32:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.434+0000 7f17aadcd740 1 -- 172.21.15.28:0/4260860555 wait complete. 2024-04-25T01:32:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.434+0000 7f17aadcd740 1 -- start start 2024-04-25T01:32:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.434+0000 7f17aadcd740 1 Processor -- start 2024-04-25T01:32:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17aadcd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32443440 0x55fa324472b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17aadcd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32443e30 0x55fa324477f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17aadcd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32448fa0 0x55fa3244c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17aadcd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fa3245d9f0 con 0x55fa32443e30 2024-04-25T01:32:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17aadcd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fa3245d320 con 0x55fa32443440 2024-04-25T01:32:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17aadcd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55fa3245e0c0 con 0x55fa32448fa0 2024-04-25T01:32:26.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17aadcd740 10 client.0 _ll_register_callbacks cb 0x7f179009add0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:26.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17a565c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32443e30 0x55fa324477f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17a565c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32443e30 0x55fa324477f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45596/0 (socket says 172.21.15.28:45596) 2024-04-25T01:32:26.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17a565c640 1 -- 172.21.15.28:0/428736184 learned_addr learned my addr 172.21.15.28:0/428736184 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17a5e5d640 1 --2- 172.21.15.28:0/428736184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32443440 0x55fa324472b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17a665e640 1 --2- 172.21.15.28:0/428736184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32448fa0 0x55fa3244c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17a565c640 1 -- 172.21.15.28:0/428736184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32448fa0 msgr2=0x55fa3244c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17a565c640 1 --2- 172.21.15.28:0/428736184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32448fa0 0x55fa3244c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17a565c640 1 -- 172.21.15.28:0/428736184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32443440 msgr2=0x55fa324472b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17a565c640 1 --2- 172.21.15.28:0/428736184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32443440 0x55fa324472b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17a565c640 1 -- 172.21.15.28:0/428736184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa3244c740 con 0x55fa32443e30 2024-04-25T01:32:26.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17a665e640 1 --2- 172.21.15.28:0/428736184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32448fa0 0x55fa3244c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:26.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17a565c640 1 --2- 172.21.15.28:0/428736184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32443e30 0x55fa324477f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f17900976d0 tx=0x7f17900cc420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=384d6fdee6a23083 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17897fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17897fa640 1 -- 172.21.15.28:0/428736184 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17900ac040 con 0x55fa32443e30 2024-04-25T01:32:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17897fa640 1 -- 172.21.15.28:0/428736184 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f17900b9e00 con 0x55fa32443e30 2024-04-25T01:32:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.435+0000 7f17aadcd740 1 -- 172.21.15.28:0/428736184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1798036070 con 0x55fa32443e30 2024-04-25T01:32:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.436+0000 7f17897fa640 1 -- 172.21.15.28:0/428736184 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f179009f570 con 0x55fa32443e30 2024-04-25T01:32:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.436+0000 7f17aadcd740 10 client.15273 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.436+0000 7f17aadcd740 1 -- 172.21.15.28:0/428736184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fa3244d950 con 0x55fa32443e30 2024-04-25T01:32:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.436+0000 7f17897fa640 1 -- 172.21.15.28:0/428736184 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1798036070 con 0x55fa32443e30 2024-04-25T01:32:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.436+0000 7f17897fa640 1 -- 172.21.15.28:0/428736184 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f179009a6c0 con 0x55fa32443e30 2024-04-25T01:32:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.436+0000 7f17aadcd740 10 client.15273 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.436+0000 7f17aadcd740 10 client.15273 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T01:32:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.436+0000 7f17aadcd740 20 client.15273 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.436+0000 7f17aadcd740 1 --2- 172.21.15.28:0/428736184 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55fa324c8e20 0x55fa324c7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.436+0000 7f17aadcd740 4 client.15273 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:32:26.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.436+0000 7f17aadcd740 1 -- 172.21.15.28:0/428736184 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "status"}) -- 0x55fa323ae790 con 0x55fa324c8e20 2024-04-25T01:32:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.436+0000 7f17a5e5d640 1 --2- 172.21.15.28:0/428736184 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55fa324c8e20 0x55fa324c7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.437+0000 7f17a5e5d640 1 --2- 172.21.15.28:0/428736184 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55fa324c8e20 0x55fa324c7b30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5ac00193fc509477 server_cookie=a2ad23149f02557b in_seq=0 out_seq=0 2024-04-25T01:32:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.437+0000 7f17897fa640 10 client.15273 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:32:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.437+0000 7f17897fa640 1 -- 172.21.15.28:0/428736184 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55fa323ae790 con 0x55fa324c8e20 2024-04-25T01:32:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.437+0000 7f17897fa640 10 client.15273 handle_command_reply: tid=0 2024-04-25T01:32:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.437+0000 7f17aadcd740 1 client.15273 shutdown 2024-04-25T01:32:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.437+0000 7f17aadcd740 1 -- 172.21.15.28:0/428736184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32443e30 msgr2=0x55fa324477f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.437+0000 7f17aadcd740 1 --2- 172.21.15.28:0/428736184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32443e30 0x55fa324477f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f17900976d0 tx=0x7f17900cc420 comp rx=0 tx=0).stop 2024-04-25T01:32:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.437+0000 7f17aadcd740 1 -- 172.21.15.28:0/428736184 shutdown_connections 2024-04-25T01:32:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.437+0000 7f17aadcd740 1 --2- 172.21.15.28:0/428736184 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fa32448fa0 0x55fa3244c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.437+0000 7f17aadcd740 1 --2- 172.21.15.28:0/428736184 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55fa324c8e20 0x55fa324c7b30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.437+0000 7f17897fa640 0 client.15273 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:32:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.437+0000 7f17aadcd740 1 --2- 172.21.15.28:0/428736184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa32443e30 0x55fa324477f0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.437+0000 7f17aadcd740 1 --2- 172.21.15.28:0/428736184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fa32443440 0x55fa324472b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.437+0000 7f17aadcd740 1 -- 172.21.15.28:0/428736184 >> 172.21.15.28:0/428736184 conn(0x55fa3243eab0 msgr2=0x55fa3244fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.437+0000 7f17aadcd740 1 -- 172.21.15.28:0/428736184 shutdown_connections 2024-04-25T01:32:26.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.437+0000 7f17aadcd740 1 -- 172.21.15.28:0/428736184 wait complete. 2024-04-25T01:32:26.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.437+0000 7f17aadcd740 20 client.15273 trim_cache size 0 max 16384 2024-04-25T01:32:26.422 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:32:26.422 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "0e88f60a-02a2-11ef-bc93-c7b262605968", 2024-04-25T01:32:26.422 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 3, 2024-04-25T01:32:26.422 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24454, 2024-04-25T01:32:26.422 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-25T01:32:26.422 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-25T01:32:26.422 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-25T01:32:26.423 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 85.996256078000002, 2024-04-25T01:32:26.423 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-04-25T01:32:26.423 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-04-25T01:32:26.423 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 91, 2024-04-25T01:32:26.423 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 133.08984204999999 2024-04-25T01:32:26.423 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:32:26.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.438+0000 7f17a80c1640 1 -- 172.21.15.28:0/3788304533 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f178c0a2c10 msgr2=0x7f178c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.438+0000 7f17a80c1640 1 --2- 172.21.15.28:0/3788304533 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f178c0a2c10 0x7f178c0a50d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x55fa32444690 tx=0x7f1798005d80 comp rx=0 tx=0).stop 2024-04-25T01:32:26.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.438+0000 7f17a80c1640 1 -- 172.21.15.28:0/3788304533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17a0101870 msgr2=0x7f17a0196eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.438+0000 7f17a80c1640 1 --2- 172.21.15.28:0/3788304533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17a0101870 0x7f17a0196eb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f179000d770 tx=0x7f179000dc30 comp rx=0 tx=0).stop 2024-04-25T01:32:26.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.438+0000 7f17a80c1640 1 -- 172.21.15.28:0/3788304533 shutdown_connections 2024-04-25T01:32:26.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.438+0000 7f17a80c1640 1 --2- 172.21.15.28:0/3788304533 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f178c0a2c10 0x7f178c0a50d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.438+0000 7f17a80c1640 1 --2- 172.21.15.28:0/3788304533 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f17a0102230 0x7f17a019df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.438+0000 7f17a80c1640 1 --2- 172.21.15.28:0/3788304533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17a0101870 0x7f17a0196eb0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.438+0000 7f17a80c1640 1 --2- 172.21.15.28:0/3788304533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17a0100680 0x7f17a0196970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.438+0000 7f17a80c1640 1 -- 172.21.15.28:0/3788304533 >> 172.21.15.28:0/3788304533 conn(0x7f17a00fbbf0 msgr2=0x7f17a0104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.438+0000 7f17a80c1640 1 -- 172.21.15.28:0/3788304533 shutdown_connections 2024-04-25T01:32:26.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.438+0000 7f17a80c1640 1 -- 172.21.15.28:0/3788304533 wait complete. 2024-04-25T01:32:26.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:26 smithi028 ceph-mon[29390]: pgmap v355: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-25T01:32:26.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:26 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3580889582' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:32:26.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:26 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/4094872170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:26.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:26 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3489085706' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:26.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-25T01:32:26.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:26 smithi130 ceph-mon[32702]: pgmap v355: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-25T01:32:26.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:26 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3580889582' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:32:26.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:26 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/4094872170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:26.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:26 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3489085706' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:26.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:26 smithi032 ceph-mon[32461]: pgmap v355: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-25T01:32:26.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:26 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3580889582' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:32:26.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:26 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/4094872170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:26.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:26 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3489085706' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.552+0000 7f579c3a9640 1 -- 172.21.15.28:0/2114389955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794101890 msgr2=0x7f5794101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.552+0000 7f579c3a9640 1 --2- 172.21.15.28:0/2114389955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794101890 0x7f5794101d10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5790009900 tx=0x7f579002a410 comp rx=0 tx=0).stop 2024-04-25T01:32:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.552+0000 7f579c3a9640 1 -- 172.21.15.28:0/2114389955 shutdown_connections 2024-04-25T01:32:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.552+0000 7f579c3a9640 1 --2- 172.21.15.28:0/2114389955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5794102250 0x7f5794108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.552+0000 7f579c3a9640 1 --2- 172.21.15.28:0/2114389955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794101890 0x7f5794101d10 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.552+0000 7f579c3a9640 1 --2- 172.21.15.28:0/2114389955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57941006a0 0x7f5794100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.552+0000 7f579c3a9640 1 -- 172.21.15.28:0/2114389955 >> 172.21.15.28:0/2114389955 conn(0x7f57940fbc30 msgr2=0x7f57940fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.552+0000 7f579c3a9640 1 -- 172.21.15.28:0/2114389955 shutdown_connections 2024-04-25T01:32:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.552+0000 7f579c3a9640 1 -- 172.21.15.28:0/2114389955 wait complete. 2024-04-25T01:32:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.553+0000 7f579c3a9640 1 Processor -- start 2024-04-25T01:32:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.553+0000 7f579c3a9640 1 -- start start 2024-04-25T01:32:26.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.553+0000 7f579c3a9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57941006a0 0x7f579406a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.553+0000 7f579c3a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794101890 0x7f579419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.553+0000 7f579c3a9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5794102250 0x7f579419ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.553+0000 7f579c3a9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f579410c380 con 0x7f5794101890 2024-04-25T01:32:26.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.553+0000 7f579c3a9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f579410c200 con 0x7f57941006a0 2024-04-25T01:32:26.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.553+0000 7f579c3a9640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f579410c500 con 0x7f5794102250 2024-04-25T01:32:26.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.553+0000 7f5799944640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794101890 0x7f579419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.553+0000 7f5799944640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794101890 0x7f579419c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45618/0 (socket says 172.21.15.28:45618) 2024-04-25T01:32:26.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.553+0000 7f5799944640 1 -- 172.21.15.28:0/1321222776 learned_addr learned my addr 172.21.15.28:0/1321222776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.553+0000 7f579a145640 1 --2- 172.21.15.28:0/1321222776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57941006a0 0x7f579406a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.553+0000 7f579a946640 1 --2- 172.21.15.28:0/1321222776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5794102250 0x7f579419ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.553+0000 7f5799944640 1 -- 172.21.15.28:0/1321222776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5794102250 msgr2=0x7f579419ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.553+0000 7f5799944640 1 --2- 172.21.15.28:0/1321222776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5794102250 0x7f579419ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.553+0000 7f5799944640 1 -- 172.21.15.28:0/1321222776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57941006a0 msgr2=0x7f579406a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.553+0000 7f5799944640 1 --2- 172.21.15.28:0/1321222776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57941006a0 0x7f579406a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.553+0000 7f5799944640 1 -- 172.21.15.28:0/1321222776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f579419f7a0 con 0x7f5794101890 2024-04-25T01:32:26.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.554+0000 7f5799944640 1 --2- 172.21.15.28:0/1321222776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794101890 0x7f579419c870 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5790009230 tx=0x7f5790031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.554+0000 7f57877fe640 1 -- 172.21.15.28:0/1321222776 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579003f030 con 0x7f5794101890 2024-04-25T01:32:26.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.554+0000 7f579c3a9640 1 -- 172.21.15.28:0/1321222776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f579419f9d0 con 0x7f5794101890 2024-04-25T01:32:26.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.554+0000 7f57877fe640 1 -- 172.21.15.28:0/1321222776 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5790003ad0 con 0x7f5794101890 2024-04-25T01:32:26.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.554+0000 7f57877fe640 1 -- 172.21.15.28:0/1321222776 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5790035070 con 0x7f5794101890 2024-04-25T01:32:26.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.554+0000 7f579c3a9640 1 -- 172.21.15.28:0/1321222776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f579419ff40 con 0x7f5794101890 2024-04-25T01:32:26.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.556+0000 7f57877fe640 1 -- 172.21.15.28:0/1321222776 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5790042030 con 0x7f5794101890 2024-04-25T01:32:26.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.556+0000 7f579f198740 1 Processor -- start 2024-04-25T01:32:26.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f579f198740 1 -- start start 2024-04-25T01:32:26.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f579f198740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79440 0x557151a79860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f579f198740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a79e30 0x557151a7e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f579f198740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a7efa0 0x557151a81430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f579f198740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5571519dcf00 con 0x557151a79440 2024-04-25T01:32:26.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f579f198740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557151985750 con 0x557151a7efa0 2024-04-25T01:32:26.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f579f198740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5571519d8ec0 con 0x557151a79e30 2024-04-25T01:32:26.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f57877fe640 1 --2- 172.21.15.28:0/1321222776 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f57800a2b90 0x7f57800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f579a946640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79440 0x557151a79860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f579a946640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79440 0x557151a79860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45628/0 (socket says 172.21.15.28:45628) 2024-04-25T01:32:26.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f579a946640 1 -- 172.21.15.28:0/777422984 learned_addr learned my addr 172.21.15.28:0/777422984 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f579a145640 1 --2- 172.21.15.28:0/777422984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a79e30 0x557151a7e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f57877fe640 1 -- 172.21.15.28:0/1321222776 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f57900ef930 con 0x7f5794101890 2024-04-25T01:32:26.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f5799944640 1 --2- 172.21.15.28:0/777422984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a7efa0 0x557151a81430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f579a946640 1 --2- 172.21.15.28:0/1321222776 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f57800a2b90 0x7f57800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f579a946640 1 -- 172.21.15.28:0/777422984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a79e30 msgr2=0x557151a7e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f579a946640 1 --2- 172.21.15.28:0/777422984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a79e30 0x557151a7e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f579a946640 1 -- 172.21.15.28:0/777422984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a7efa0 msgr2=0x557151a81430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f579a946640 1 --2- 172.21.15.28:0/777422984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a7efa0 0x557151a81430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.557+0000 7f579a946640 1 -- 172.21.15.28:0/777422984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571516f12d0 con 0x557151a79440 2024-04-25T01:32:26.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.558+0000 7f579a946640 1 --2- 172.21.15.28:0/777422984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79440 0x557151a79860 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f578c00cb20 tx=0x7f578c0072b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=159448d71e236aef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.558+0000 7f5784bfe640 1 -- 172.21.15.28:0/777422984 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f578c015070 con 0x557151a79440 2024-04-25T01:32:26.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.558+0000 7f5784bfe640 1 -- 172.21.15.28:0/777422984 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f578c016040 con 0x557151a79440 2024-04-25T01:32:26.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.558+0000 7f5784bfe640 1 -- 172.21.15.28:0/777422984 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f578c01e020 con 0x557151a79440 2024-04-25T01:32:26.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.558+0000 7f579f198740 1 -- 172.21.15.28:0/777422984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79440 msgr2=0x557151a79860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.558+0000 7f579f198740 1 --2- 172.21.15.28:0/777422984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79440 0x557151a79860 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f578c00cb20 tx=0x7f578c0072b0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.558+0000 7f579a946640 1 --2- 172.21.15.28:0/1321222776 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f57800a2b90 0x7f57800a5050 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f578c01f000 tx=0x7f578c01b000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.559+0000 7f579f198740 1 -- 172.21.15.28:0/777422984 shutdown_connections 2024-04-25T01:32:26.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.559+0000 7f579f198740 1 --2- 172.21.15.28:0/777422984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a7efa0 0x557151a81430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.559+0000 7f579f198740 1 --2- 172.21.15.28:0/777422984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a79e30 0x557151a7e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.559+0000 7f579f198740 1 --2- 172.21.15.28:0/777422984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79440 0x557151a79860 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.559+0000 7f579f198740 1 -- 172.21.15.28:0/777422984 >> 172.21.15.28:0/777422984 conn(0x557151a74ab0 msgr2=0x557151a74eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.559+0000 7f579f198740 1 -- 172.21.15.28:0/777422984 shutdown_connections 2024-04-25T01:32:26.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.559+0000 7f579f198740 1 -- 172.21.15.28:0/777422984 wait complete. 2024-04-25T01:32:26.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.559+0000 7f579f198740 1 -- start start 2024-04-25T01:32:26.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.559+0000 7f579f198740 1 Processor -- start 2024-04-25T01:32:26.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.559+0000 7f579f198740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a79440 0x557151a83890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.559+0000 7f579f198740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79e30 0x557151a83dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.559+0000 7f579f198740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a7efa0 0x557151a8ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.559+0000 7f579f198740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557151a81f50 con 0x557151a79e30 2024-04-25T01:32:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.559+0000 7f579f198740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557151998cf0 con 0x557151a79440 2024-04-25T01:32:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.559+0000 7f579f198740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x557151a820d0 con 0x557151a7efa0 2024-04-25T01:32:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.559+0000 7f579f198740 10 client.0 _ll_register_callbacks cb 0x5571519986e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.559+0000 7f5799944640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79e30 0x557151a83dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.559+0000 7f5799944640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79e30 0x557151a83dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45644/0 (socket says 172.21.15.28:45644) 2024-04-25T01:32:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.559+0000 7f5799944640 1 -- 172.21.15.28:0/3808588511 learned_addr learned my addr 172.21.15.28:0/3808588511 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.560+0000 7f579a946640 1 --2- 172.21.15.28:0/3808588511 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a7efa0 0x557151a8ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.560+0000 7f579a145640 1 --2- 172.21.15.28:0/3808588511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a79440 0x557151a83890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.560+0000 7f5799944640 1 -- 172.21.15.28:0/3808588511 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a7efa0 msgr2=0x557151a8ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.560+0000 7f5799944640 1 --2- 172.21.15.28:0/3808588511 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a7efa0 0x557151a8ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.560+0000 7f5799944640 1 -- 172.21.15.28:0/3808588511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a79440 msgr2=0x557151a83890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.560+0000 7f5799944640 1 --2- 172.21.15.28:0/3808588511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a79440 0x557151a83890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.560+0000 7f5799944640 1 -- 172.21.15.28:0/3808588511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571517bb6b0 con 0x557151a79e30 2024-04-25T01:32:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.560+0000 7f579a145640 1 --2- 172.21.15.28:0/3808588511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a79440 0x557151a83890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.560+0000 7f5799944640 1 --2- 172.21.15.28:0/3808588511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79e30 0x557151a83dd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f57900b8290 tx=0x7f57900bc530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=773d07c2b29ec9ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.560+0000 7f577dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.560+0000 7f577dffb640 1 -- 172.21.15.28:0/3808588511 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5790049040 con 0x557151a79e30 2024-04-25T01:32:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.560+0000 7f577dffb640 1 -- 172.21.15.28:0/3808588511 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f57900b84a0 con 0x557151a79e30 2024-04-25T01:32:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.560+0000 7f579f198740 1 -- 172.21.15.28:0/3808588511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f578c016040 con 0x557151a79e30 2024-04-25T01:32:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.560+0000 7f577dffb640 1 -- 172.21.15.28:0/3808588511 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f57900c01e0 con 0x557151a79e30 2024-04-25T01:32:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.560+0000 7f579f198740 10 client.15285 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.560+0000 7f579f198740 1 -- 172.21.15.28:0/3808588511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557151a8ddf0 con 0x557151a79e30 2024-04-25T01:32:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.560+0000 7f577dffb640 1 -- 172.21.15.28:0/3808588511 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f578c016040 con 0x557151a79e30 2024-04-25T01:32:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.560+0000 7f577dffb640 1 -- 172.21.15.28:0/3808588511 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f57900bc090 con 0x557151a79e30 2024-04-25T01:32:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.561+0000 7f579f198740 10 client.15285 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.561+0000 7f579f198740 10 client.15285 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T01:32:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.561+0000 7f579f198740 20 client.15285 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.561+0000 7f579f198740 1 --2- 172.21.15.28:0/3808588511 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x557151a90580 0x557151a919f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.561+0000 7f579f198740 4 client.15285 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.561+0000 7f579f198740 1 -- 172.21.15.28:0/3808588511 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5571519ea440 con 0x557151a90580 2024-04-25T01:32:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.561+0000 7f579a145640 1 --2- 172.21.15.28:0/3808588511 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x557151a90580 0x557151a919f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.561+0000 7f577dffb640 10 client.15285 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:32:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.561+0000 7f579a145640 1 --2- 172.21.15.28:0/3808588511 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x557151a90580 0x557151a919f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9361938f88b978b3 server_cookie=e54541ff36ce4e05 in_seq=0 out_seq=0 2024-04-25T01:32:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.562+0000 7f577dffb640 1 -- 172.21.15.28:0/3808588511 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5571519ea440 con 0x557151a90580 2024-04-25T01:32:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.562+0000 7f577dffb640 10 client.15285 handle_command_reply: tid=0 2024-04-25T01:32:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.562+0000 7f579f198740 1 client.15285 shutdown 2024-04-25T01:32:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.562+0000 7f579f198740 1 -- 172.21.15.28:0/3808588511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79e30 msgr2=0x557151a83dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.562+0000 7f579f198740 1 --2- 172.21.15.28:0/3808588511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79e30 0x557151a83dd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f57900b8290 tx=0x7f57900bc530 comp rx=0 tx=0).stop 2024-04-25T01:32:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.562+0000 7f579f198740 1 -- 172.21.15.28:0/3808588511 shutdown_connections 2024-04-25T01:32:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.562+0000 7f579f198740 1 --2- 172.21.15.28:0/3808588511 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x557151a90580 0x557151a919f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.562+0000 7f579f198740 1 --2- 172.21.15.28:0/3808588511 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a7efa0 0x557151a8ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.562+0000 7f577dffb640 0 client.15285 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:32:26.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.562+0000 7f579f198740 1 --2- 172.21.15.28:0/3808588511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79e30 0x557151a83dd0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.562+0000 7f579f198740 1 --2- 172.21.15.28:0/3808588511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a79440 0x557151a83890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.562+0000 7f579f198740 1 -- 172.21.15.28:0/3808588511 >> 172.21.15.28:0/3808588511 conn(0x557151a74ab0 msgr2=0x557151a73ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.562+0000 7f579f198740 1 -- 172.21.15.28:0/3808588511 shutdown_connections 2024-04-25T01:32:26.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.562+0000 7f579f198740 1 -- 172.21.15.28:0/3808588511 wait complete. 2024-04-25T01:32:26.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.563+0000 7f579f198740 20 client.15285 trim_cache size 0 max 16384 2024-04-25T01:32:26.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.581+0000 7f579f198740 1 Processor -- start 2024-04-25T01:32:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.581+0000 7f579f198740 1 -- start start 2024-04-25T01:32:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.581+0000 7f579f198740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79440 0x557151a7a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.581+0000 7f579f198740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a79e30 0x557151a8add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.581+0000 7f579f198740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a7efa0 0x557151a6f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.581+0000 7f579f198740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557151a8fd30 con 0x557151a79440 2024-04-25T01:32:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.581+0000 7f579f198740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557151a90400 con 0x557151a7efa0 2024-04-25T01:32:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.581+0000 7f579f198740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x557151a90ad0 con 0x557151a79e30 2024-04-25T01:32:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.582+0000 7f579a145640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79440 0x557151a7a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.582+0000 7f579a145640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79440 0x557151a7a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45650/0 (socket says 172.21.15.28:45650) 2024-04-25T01:32:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.582+0000 7f579a145640 1 -- 172.21.15.28:0/1493877424 learned_addr learned my addr 172.21.15.28:0/1493877424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.582+0000 7f579a946640 1 --2- 172.21.15.28:0/1493877424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a7efa0 0x557151a6f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.582+0000 7f5799944640 1 --2- 172.21.15.28:0/1493877424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a79e30 0x557151a8add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.582+0000 7f579a145640 1 -- 172.21.15.28:0/1493877424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a79e30 msgr2=0x557151a8add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.582+0000 7f579a145640 1 --2- 172.21.15.28:0/1493877424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a79e30 0x557151a8add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.582+0000 7f579a145640 1 -- 172.21.15.28:0/1493877424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a7efa0 msgr2=0x557151a6f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.582+0000 7f579a145640 1 --2- 172.21.15.28:0/1493877424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a7efa0 0x557151a6f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.582+0000 7f579a145640 1 -- 172.21.15.28:0/1493877424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557151a7a6c0 con 0x557151a79440 2024-04-25T01:32:26.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.582+0000 7f579a145640 1 --2- 172.21.15.28:0/1493877424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79440 0x557151a7a8d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x557151a7a8a0 tx=0x7f5788012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=718cde11685ce247 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.582+0000 7f577ffff640 1 -- 172.21.15.28:0/1493877424 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5788004050 con 0x557151a79440 2024-04-25T01:32:26.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.582+0000 7f577ffff640 1 -- 172.21.15.28:0/1493877424 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57880041f0 con 0x557151a79440 2024-04-25T01:32:26.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.582+0000 7f577ffff640 1 -- 172.21.15.28:0/1493877424 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5788003cc0 con 0x557151a79440 2024-04-25T01:32:26.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f579f198740 1 -- 172.21.15.28:0/1493877424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79440 msgr2=0x557151a7a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f579f198740 1 --2- 172.21.15.28:0/1493877424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79440 0x557151a7a8d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x557151a7a8a0 tx=0x7f5788012820 comp rx=0 tx=0).stop 2024-04-25T01:32:26.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f579f198740 1 -- 172.21.15.28:0/1493877424 shutdown_connections 2024-04-25T01:32:26.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f579f198740 1 --2- 172.21.15.28:0/1493877424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a7efa0 0x557151a6f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f579f198740 1 --2- 172.21.15.28:0/1493877424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a79e30 0x557151a8add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f579f198740 1 --2- 172.21.15.28:0/1493877424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79440 0x557151a7a8d0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f579f198740 1 -- 172.21.15.28:0/1493877424 >> 172.21.15.28:0/1493877424 conn(0x557151a74ab0 msgr2=0x557151a74e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f579f198740 1 -- 172.21.15.28:0/1493877424 shutdown_connections 2024-04-25T01:32:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f579f198740 1 -- 172.21.15.28:0/1493877424 wait complete. 2024-04-25T01:32:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f579f198740 1 -- start start 2024-04-25T01:32:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f579f198740 1 Processor -- start 2024-04-25T01:32:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f579f198740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a79440 0x557151a89010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f579f198740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79e30 0x557151a89550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f579f198740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a7efa0 0x557151a89a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f579f198740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557151a91870 con 0x557151a79e30 2024-04-25T01:32:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f579f198740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557151a911a0 con 0x557151a7efa0 2024-04-25T01:32:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f579f198740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x557151a91f40 con 0x557151a79440 2024-04-25T01:32:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f579f198740 10 client.0 _ll_register_callbacks cb 0x7f57900bb970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f5799944640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79e30 0x557151a89550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f5799944640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79e30 0x557151a89550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45660/0 (socket says 172.21.15.28:45660) 2024-04-25T01:32:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.583+0000 7f5799944640 1 -- 172.21.15.28:0/3801466155 learned_addr learned my addr 172.21.15.28:0/3801466155 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.584+0000 7f579a946640 1 --2- 172.21.15.28:0/3801466155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a7efa0 0x557151a89a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.584+0000 7f579a145640 1 --2- 172.21.15.28:0/3801466155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a79440 0x557151a89010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.584+0000 7f5799944640 1 -- 172.21.15.28:0/3801466155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a79440 msgr2=0x557151a89010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.584+0000 7f5799944640 1 --2- 172.21.15.28:0/3801466155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a79440 0x557151a89010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.584+0000 7f5799944640 1 -- 172.21.15.28:0/3801466155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a7efa0 msgr2=0x557151a89a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.584+0000 7f5799944640 1 --2- 172.21.15.28:0/3801466155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a7efa0 0x557151a89a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.584+0000 7f5799944640 1 -- 172.21.15.28:0/3801466155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557151a8a220 con 0x557151a79e30 2024-04-25T01:32:26.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.584+0000 7f579a145640 1 --2- 172.21.15.28:0/3801466155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a79440 0x557151a89010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:26.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.584+0000 7f5799944640 1 --2- 172.21.15.28:0/3801466155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79e30 0x557151a89550 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f57900bbdc0 tx=0x7f57900b80d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1171852ca1f697e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.584+0000 7f577dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:26.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.584+0000 7f577dffb640 1 -- 172.21.15.28:0/3801466155 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5790049070 con 0x557151a79e30 2024-04-25T01:32:26.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.584+0000 7f577dffb640 1 -- 172.21.15.28:0/3801466155 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f5790003c80 con 0x557151a79e30 2024-04-25T01:32:26.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.584+0000 7f579f198740 1 -- 172.21.15.28:0/3801466155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57880041f0 con 0x557151a79e30 2024-04-25T01:32:26.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.584+0000 7f577dffb640 1 -- 172.21.15.28:0/3801466155 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f57900c4aa0 con 0x557151a79e30 2024-04-25T01:32:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.584+0000 7f579f198740 10 client.15291 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.584+0000 7f579f198740 1 -- 172.21.15.28:0/3801466155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557151af66b0 con 0x557151a79e30 2024-04-25T01:32:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.584+0000 7f577dffb640 1 -- 172.21.15.28:0/3801466155 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57880041f0 con 0x557151a79e30 2024-04-25T01:32:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.584+0000 7f577dffb640 1 -- 172.21.15.28:0/3801466155 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f5790031cd0 con 0x557151a79e30 2024-04-25T01:32:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.585+0000 7f579f198740 10 client.15291 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.585+0000 7f579f198740 10 client.15291 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T01:32:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.585+0000 7f579f198740 20 client.15291 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.585+0000 7f579f198740 1 --2- 172.21.15.28:0/3801466155 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x557151afb120 0x557151af9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.585+0000 7f579f198740 4 client.15291 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:32:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.585+0000 7f579f198740 1 -- 172.21.15.28:0/3801466155 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "status"}) -- 0x5571519e4790 con 0x557151afb120 2024-04-25T01:32:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.585+0000 7f579a145640 1 --2- 172.21.15.28:0/3801466155 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x557151afb120 0x557151af9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.585+0000 7f577dffb640 10 client.15291 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:32:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.585+0000 7f579a145640 1 --2- 172.21.15.28:0/3801466155 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x557151afb120 0x557151af9e30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3bb6463d88a286d3 server_cookie=ae846bee3584bcff in_seq=0 out_seq=0 2024-04-25T01:32:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.585+0000 7f577dffb640 1 -- 172.21.15.28:0/3801466155 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5571519e4790 con 0x557151afb120 2024-04-25T01:32:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.585+0000 7f577dffb640 10 client.15291 handle_command_reply: tid=0 2024-04-25T01:32:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.585+0000 7f579f198740 1 client.15291 shutdown 2024-04-25T01:32:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579f198740 1 -- 172.21.15.28:0/3801466155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79e30 msgr2=0x557151a89550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579f198740 1 --2- 172.21.15.28:0/3801466155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79e30 0x557151a89550 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f57900bbdc0 tx=0x7f57900b80d0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579f198740 1 -- 172.21.15.28:0/3801466155 shutdown_connections 2024-04-25T01:32:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579f198740 1 --2- 172.21.15.28:0/3801466155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557151a7efa0 0x557151a89a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579f198740 1 --2- 172.21.15.28:0/3801466155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557151a79e30 0x557151a89550 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579f198740 1 --2- 172.21.15.28:0/3801466155 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x557151afb120 0x557151af9e30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579f198740 1 --2- 172.21.15.28:0/3801466155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557151a79440 0x557151a89010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f577dffb640 0 client.15291 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:32:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579f198740 1 -- 172.21.15.28:0/3801466155 >> 172.21.15.28:0/3801466155 conn(0x557151a74ab0 msgr2=0x557151a6f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579f198740 1 -- 172.21.15.28:0/3801466155 shutdown_connections 2024-04-25T01:32:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579f198740 1 -- 172.21.15.28:0/3801466155 wait complete. 2024-04-25T01:32:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579f198740 20 client.15291 trim_cache size 0 max 16384 2024-04-25T01:32:26.571 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:32:26.571 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "0e88f60a-02a2-11ef-bc93-c7b262605968", 2024-04-25T01:32:26.571 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-04-25T01:32:26.571 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24473, 2024-04-25T01:32:26.571 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-25T01:32:26.571 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-25T01:32:26.571 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-25T01:32:26.571 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 87.153293038000001, 2024-04-25T01:32:26.571 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-04-25T01:32:26.571 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-04-25T01:32:26.571 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 91, 2024-04-25T01:32:26.572 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 130.24890459900001 2024-04-25T01:32:26.572 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:32:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579c3a9640 1 -- 172.21.15.28:0/1321222776 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f57800a2b90 msgr2=0x7f57800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579c3a9640 1 --2- 172.21.15.28:0/1321222776 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f57800a2b90 0x7f57800a5050 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f578c01f000 tx=0x7f578c01b000 comp rx=0 tx=0).stop 2024-04-25T01:32:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579c3a9640 1 -- 172.21.15.28:0/1321222776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794101890 msgr2=0x7f579419c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579c3a9640 1 --2- 172.21.15.28:0/1321222776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794101890 0x7f579419c870 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5790009230 tx=0x7f5790031b60 comp rx=0 tx=0).stop 2024-04-25T01:32:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579c3a9640 1 -- 172.21.15.28:0/1321222776 shutdown_connections 2024-04-25T01:32:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579c3a9640 1 --2- 172.21.15.28:0/1321222776 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f57800a2b90 0x7f57800a5050 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579c3a9640 1 --2- 172.21.15.28:0/1321222776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5794102250 0x7f579419ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579c3a9640 1 --2- 172.21.15.28:0/1321222776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5794101890 0x7f579419c870 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579c3a9640 1 --2- 172.21.15.28:0/1321222776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57941006a0 0x7f579406a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579c3a9640 1 -- 172.21.15.28:0/1321222776 >> 172.21.15.28:0/1321222776 conn(0x7f57940fbc30 msgr2=0x7f5794104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579c3a9640 1 -- 172.21.15.28:0/1321222776 shutdown_connections 2024-04-25T01:32:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.586+0000 7f579c3a9640 1 -- 172.21.15.28:0/1321222776 wait complete. 2024-04-25T01:32:26.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T01:32:26.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.700+0000 7f751e42c640 1 -- 172.21.15.28:0/957360306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7518102250 msgr2=0x7f7518108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.700+0000 7f751e42c640 1 --2- 172.21.15.28:0/957360306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7518102250 0x7f7518108ae0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f7510009ca0 tx=0x7f751002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.700+0000 7f751e42c640 1 -- 172.21.15.28:0/957360306 shutdown_connections 2024-04-25T01:32:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.700+0000 7f751e42c640 1 --2- 172.21.15.28:0/957360306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7518102250 0x7f7518108ae0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.700+0000 7f751e42c640 1 --2- 172.21.15.28:0/957360306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7518101890 0x7f7518101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.700+0000 7f751e42c640 1 --2- 172.21.15.28:0/957360306 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75181006a0 0x7f7518100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.700+0000 7f751e42c640 1 -- 172.21.15.28:0/957360306 >> 172.21.15.28:0/957360306 conn(0x7f75180fbc10 msgr2=0x7f75180fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.700+0000 7f751e42c640 1 -- 172.21.15.28:0/957360306 shutdown_connections 2024-04-25T01:32:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.700+0000 7f751e42c640 1 -- 172.21.15.28:0/957360306 wait complete. 2024-04-25T01:32:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.700+0000 7f751e42c640 1 Processor -- start 2024-04-25T01:32:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.700+0000 7f751e42c640 1 -- start start 2024-04-25T01:32:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f751e42c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75181006a0 0x7f75181979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f751e42c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7518101890 0x7f7518197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f751e42c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7518102250 0x7f751819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f751e42c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f751810d400 con 0x7f75181006a0 2024-04-25T01:32:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f751e42c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f751810d280 con 0x7f7518102250 2024-04-25T01:32:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f751e42c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f751810d580 con 0x7f7518101890 2024-04-25T01:32:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f7517fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75181006a0 0x7f75181979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f7517fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75181006a0 0x7f75181979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60654/0 (socket says 172.21.15.28:60654) 2024-04-25T01:32:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f7517fff640 1 -- 172.21.15.28:0/4210474466 learned_addr learned my addr 172.21.15.28:0/4210474466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f75177fe640 1 --2- 172.21.15.28:0/4210474466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7518101890 0x7f7518197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f751c9c9640 1 --2- 172.21.15.28:0/4210474466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7518102250 0x7f751819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f7517fff640 1 -- 172.21.15.28:0/4210474466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7518101890 msgr2=0x7f7518197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f7517fff640 1 --2- 172.21.15.28:0/4210474466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7518101890 0x7f7518197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f7517fff640 1 -- 172.21.15.28:0/4210474466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7518102250 msgr2=0x7f751819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f7517fff640 1 --2- 172.21.15.28:0/4210474466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7518102250 0x7f751819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f7517fff640 1 -- 172.21.15.28:0/4210474466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f751819f600 con 0x7f75181006a0 2024-04-25T01:32:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f751c9c9640 1 --2- 172.21.15.28:0/4210474466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7518102250 0x7f751819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f7517fff640 1 --2- 172.21.15.28:0/4210474466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75181006a0 0x7f75181979e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f750800c970 tx=0x7f750800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f75157fa640 1 -- 172.21.15.28:0/4210474466 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7508007c10 con 0x7f75181006a0 2024-04-25T01:32:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f751e42c640 1 -- 172.21.15.28:0/4210474466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f751819f8f0 con 0x7f75181006a0 2024-04-25T01:32:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.701+0000 7f75157fa640 1 -- 172.21.15.28:0/4210474466 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7508007db0 con 0x7f75181006a0 2024-04-25T01:32:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.702+0000 7f75157fa640 1 -- 172.21.15.28:0/4210474466 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f750800f8a0 con 0x7f75181006a0 2024-04-25T01:32:26.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.702+0000 7f751e42c640 1 -- 172.21.15.28:0/4210474466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f751819fe40 con 0x7f75181006a0 2024-04-25T01:32:26.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.704+0000 7f75157fa640 1 -- 172.21.15.28:0/4210474466 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f750800fa60 con 0x7f75181006a0 2024-04-25T01:32:26.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.704+0000 7f752120f740 1 Processor -- start 2024-04-25T01:32:26.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.704+0000 7f752120f740 1 -- start start 2024-04-25T01:32:26.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.704+0000 7f752120f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ec8440 0x5587c8ec8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.704+0000 7f752120f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8e30 0x5587c8ecd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.704+0000 7f752120f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 0x5587c8ed0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.704+0000 7f752120f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5587c8e2bf00 con 0x5587c8ecdfa0 2024-04-25T01:32:26.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.704+0000 7f752120f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5587c8dd4750 con 0x5587c8ec8e30 2024-04-25T01:32:26.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.704+0000 7f752120f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5587c8e27ec0 con 0x5587c8ec8440 2024-04-25T01:32:26.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.704+0000 7f75157fa640 1 --2- 172.21.15.28:0/4210474466 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f75040a2b90 0x7f75040a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.704+0000 7f75177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 0x5587c8ed0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.704+0000 7f751c9c9640 1 --2- 172.21.15.28:0/4210474466 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f75040a2b90 0x7f75040a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.704+0000 7f75177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 0x5587c8ed0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60666/0 (socket says 172.21.15.28:60666) 2024-04-25T01:32:26.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.704+0000 7f75177fe640 1 -- 172.21.15.28:0/3364527648 learned_addr learned my addr 172.21.15.28:0/3364527648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.705+0000 7f7517fff640 1 --2- 172.21.15.28:0/3364527648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8e30 0x5587c8ecd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.705+0000 7f751c9c9640 1 --2- 172.21.15.28:0/3364527648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ec8440 0x5587c8ec8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.705+0000 7f75157fa640 1 -- 172.21.15.28:0/4210474466 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f75080cf7a0 con 0x7f75181006a0 2024-04-25T01:32:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.705+0000 7f75177fe640 1 -- 172.21.15.28:0/3364527648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ec8440 msgr2=0x5587c8ec8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.705+0000 7f75177fe640 1 --2- 172.21.15.28:0/3364527648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ec8440 0x5587c8ec8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.705+0000 7f75177fe640 1 -- 172.21.15.28:0/3364527648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8e30 msgr2=0x5587c8ecd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.705+0000 7f75177fe640 1 --2- 172.21.15.28:0/3364527648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8e30 0x5587c8ecd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.705+0000 7f75177fe640 1 -- 172.21.15.28:0/3364527648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587c8b402d0 con 0x5587c8ecdfa0 2024-04-25T01:32:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.705+0000 7f75177fe640 1 --2- 172.21.15.28:0/3364527648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 0x5587c8ed0430 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f750000bc80 tx=0x7f750000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4b6c34f22c99e93 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.705+0000 7f751c9c9640 1 --2- 172.21.15.28:0/4210474466 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f75040a2b90 0x7f75040a5050 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x5587c8ec9690 tx=0x7f7510009380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.705+0000 7f750e7fc640 1 -- 172.21.15.28:0/3364527648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7500015070 con 0x5587c8ecdfa0 2024-04-25T01:32:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.705+0000 7f750e7fc640 1 -- 172.21.15.28:0/3364527648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f750001a940 con 0x5587c8ecdfa0 2024-04-25T01:32:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.705+0000 7f750e7fc640 1 -- 172.21.15.28:0/3364527648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7500011a10 con 0x5587c8ecdfa0 2024-04-25T01:32:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.706+0000 7f752120f740 1 -- 172.21.15.28:0/3364527648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 msgr2=0x5587c8ed0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.706+0000 7f752120f740 1 --2- 172.21.15.28:0/3364527648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 0x5587c8ed0430 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f750000bc80 tx=0x7f750000f730 comp rx=0 tx=0).stop 2024-04-25T01:32:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.706+0000 7f752120f740 1 -- 172.21.15.28:0/3364527648 shutdown_connections 2024-04-25T01:32:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.706+0000 7f752120f740 1 --2- 172.21.15.28:0/3364527648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 0x5587c8ed0430 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.706+0000 7f752120f740 1 --2- 172.21.15.28:0/3364527648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8e30 0x5587c8ecd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.706+0000 7f752120f740 1 --2- 172.21.15.28:0/3364527648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ec8440 0x5587c8ec8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.706+0000 7f752120f740 1 -- 172.21.15.28:0/3364527648 >> 172.21.15.28:0/3364527648 conn(0x5587c8ec3ab0 msgr2=0x5587c8ec3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.706+0000 7f752120f740 1 -- 172.21.15.28:0/3364527648 shutdown_connections 2024-04-25T01:32:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.706+0000 7f752120f740 1 -- 172.21.15.28:0/3364527648 wait complete. 2024-04-25T01:32:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.706+0000 7f752120f740 1 -- start start 2024-04-25T01:32:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.706+0000 7f752120f740 1 Processor -- start 2024-04-25T01:32:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f752120f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ec8440 0x5587c8ed27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f752120f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8e30 0x5587c8ed2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f752120f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 0x5587c8ed9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f752120f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5587c8ed0f50 con 0x5587c8ecdfa0 2024-04-25T01:32:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f752120f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5587c8de7cf0 con 0x5587c8ec8e30 2024-04-25T01:32:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f752120f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5587c8ed10d0 con 0x5587c8ec8440 2024-04-25T01:32:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f752120f740 10 client.0 _ll_register_callbacks cb 0x5587c8de76e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f751c9c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 0x5587c8ed9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f751c9c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 0x5587c8ed9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60672/0 (socket says 172.21.15.28:60672) 2024-04-25T01:32:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f751c9c9640 1 -- 172.21.15.28:0/2554386280 learned_addr learned my addr 172.21.15.28:0/2554386280 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f75177fe640 1 --2- 172.21.15.28:0/2554386280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8e30 0x5587c8ed2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f7517fff640 1 --2- 172.21.15.28:0/2554386280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ec8440 0x5587c8ed27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f751c9c9640 1 -- 172.21.15.28:0/2554386280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ec8440 msgr2=0x5587c8ed27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f751c9c9640 1 --2- 172.21.15.28:0/2554386280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ec8440 0x5587c8ed27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f751c9c9640 1 -- 172.21.15.28:0/2554386280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8e30 msgr2=0x5587c8ed2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f751c9c9640 1 --2- 172.21.15.28:0/2554386280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8e30 0x5587c8ed2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f751c9c9640 1 -- 172.21.15.28:0/2554386280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587c8c0a6b0 con 0x5587c8ecdfa0 2024-04-25T01:32:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f7517fff640 1 --2- 172.21.15.28:0/2554386280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ec8440 0x5587c8ed27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f751c9c9640 1 --2- 172.21.15.28:0/2554386280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 0x5587c8ed9d60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f751002f8c0 tx=0x7f751002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=492bc3ae8bca2d98 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f74f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f74f77fe640 1 -- 172.21.15.28:0/2554386280 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7510050070 con 0x5587c8ecdfa0 2024-04-25T01:32:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f74f77fe640 1 -- 172.21.15.28:0/2554386280 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f75100307d0 con 0x5587c8ecdfa0 2024-04-25T01:32:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.707+0000 7f752120f740 1 -- 172.21.15.28:0/2554386280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f750001a940 con 0x5587c8ecdfa0 2024-04-25T01:32:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.708+0000 7f74f77fe640 1 -- 172.21.15.28:0/2554386280 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7510044070 con 0x5587c8ecdfa0 2024-04-25T01:32:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.708+0000 7f752120f740 10 client.15303 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.708+0000 7f752120f740 1 -- 172.21.15.28:0/2554386280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5587c8edcdb0 con 0x5587c8ecdfa0 2024-04-25T01:32:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.708+0000 7f74f77fe640 1 -- 172.21.15.28:0/2554386280 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f750001a940 con 0x5587c8ecdfa0 2024-04-25T01:32:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.708+0000 7f74f77fe640 1 -- 172.21.15.28:0/2554386280 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f7510047050 con 0x5587c8ecdfa0 2024-04-25T01:32:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.708+0000 7f752120f740 10 client.15303 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.708+0000 7f752120f740 10 client.15303 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:32:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.708+0000 7f752120f740 20 client.15303 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.708+0000 7f752120f740 1 --2- 172.21.15.28:0/2554386280 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5587c8edf510 0x5587c8ee0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.708+0000 7f752120f740 4 client.15303 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.708+0000 7f752120f740 1 -- 172.21.15.28:0/2554386280 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587c8e39440 con 0x5587c8edf510 2024-04-25T01:32:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.708+0000 7f7517fff640 1 --2- 172.21.15.28:0/2554386280 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5587c8edf510 0x5587c8ee0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.709+0000 7f7517fff640 1 --2- 172.21.15.28:0/2554386280 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5587c8edf510 0x5587c8ee0980 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c683c3a3cf3693e0 server_cookie=c87d53de962877f5 in_seq=0 out_seq=0 2024-04-25T01:32:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.709+0000 7f74f77fe640 10 client.15303 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:32:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.710+0000 7f74f77fe640 1 -- 172.21.15.28:0/2554386280 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5587c8e39440 con 0x5587c8edf510 2024-04-25T01:32:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.710+0000 7f74f77fe640 10 client.15303 handle_command_reply: tid=0 2024-04-25T01:32:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.710+0000 7f752120f740 1 client.15303 shutdown 2024-04-25T01:32:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.710+0000 7f752120f740 1 -- 172.21.15.28:0/2554386280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 msgr2=0x5587c8ed9d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.710+0000 7f752120f740 1 --2- 172.21.15.28:0/2554386280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 0x5587c8ed9d60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f751002f8c0 tx=0x7f751002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.710+0000 7f752120f740 1 -- 172.21.15.28:0/2554386280 shutdown_connections 2024-04-25T01:32:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.710+0000 7f752120f740 1 --2- 172.21.15.28:0/2554386280 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5587c8edf510 0x5587c8ee0980 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.710+0000 7f752120f740 1 --2- 172.21.15.28:0/2554386280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 0x5587c8ed9d60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.710+0000 7f752120f740 1 --2- 172.21.15.28:0/2554386280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8e30 0x5587c8ed2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.710+0000 7f74f77fe640 0 client.15303 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:32:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.710+0000 7f752120f740 1 --2- 172.21.15.28:0/2554386280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ec8440 0x5587c8ed27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.710+0000 7f752120f740 1 -- 172.21.15.28:0/2554386280 >> 172.21.15.28:0/2554386280 conn(0x5587c8ec3ab0 msgr2=0x5587c8ec2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.710+0000 7f752120f740 1 -- 172.21.15.28:0/2554386280 shutdown_connections 2024-04-25T01:32:26.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.710+0000 7f752120f740 1 -- 172.21.15.28:0/2554386280 wait complete. 2024-04-25T01:32:26.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.710+0000 7f752120f740 20 client.15303 trim_cache size 0 max 16384 2024-04-25T01:32:26.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.728+0000 7f752120f740 1 Processor -- start 2024-04-25T01:32:26.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.728+0000 7f752120f740 1 -- start start 2024-04-25T01:32:26.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.728+0000 7f752120f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8440 0x5587c8ec96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.728+0000 7f752120f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ec8e30 0x5587c8ed1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.728+0000 7f752120f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 0x5587c8ed2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.728+0000 7f752120f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5587c8edecc0 con 0x5587c8ecdfa0 2024-04-25T01:32:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.728+0000 7f752120f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5587c8edf390 con 0x5587c8ec8440 2024-04-25T01:32:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.728+0000 7f752120f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5587c8edfa60 con 0x5587c8ec8e30 2024-04-25T01:32:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.728+0000 7f751c9c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 0x5587c8ed2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.729+0000 7f751c9c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 0x5587c8ed2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60680/0 (socket says 172.21.15.28:60680) 2024-04-25T01:32:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.729+0000 7f751c9c9640 1 -- 172.21.15.28:0/3222255535 learned_addr learned my addr 172.21.15.28:0/3222255535 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.729+0000 7f75177fe640 1 --2- 172.21.15.28:0/3222255535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ec8e30 0x5587c8ed1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.729+0000 7f7517fff640 1 --2- 172.21.15.28:0/3222255535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8440 0x5587c8ec96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.729+0000 7f751c9c9640 1 -- 172.21.15.28:0/3222255535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ec8e30 msgr2=0x5587c8ed1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.729+0000 7f751c9c9640 1 --2- 172.21.15.28:0/3222255535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ec8e30 0x5587c8ed1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.729+0000 7f751c9c9640 1 -- 172.21.15.28:0/3222255535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8440 msgr2=0x5587c8ec96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.729+0000 7f751c9c9640 1 --2- 172.21.15.28:0/3222255535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8440 0x5587c8ec96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.729+0000 7f751c9c9640 1 -- 172.21.15.28:0/3222255535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587c8ec9210 con 0x5587c8ecdfa0 2024-04-25T01:32:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.729+0000 7f751c9c9640 1 --2- 172.21.15.28:0/3222255535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 0x5587c8ed2010 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f751004aa50 tx=0x7f7510031cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de2f10864f130ec6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.729+0000 7f750dffb640 1 -- 172.21.15.28:0/3222255535 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7510050040 con 0x5587c8ecdfa0 2024-04-25T01:32:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.729+0000 7f750dffb640 1 -- 172.21.15.28:0/3222255535 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7510036040 con 0x5587c8ecdfa0 2024-04-25T01:32:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.729+0000 7f750dffb640 1 -- 172.21.15.28:0/3222255535 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7510003d60 con 0x5587c8ecdfa0 2024-04-25T01:32:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.730+0000 7f752120f740 1 -- 172.21.15.28:0/3222255535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 msgr2=0x5587c8ed2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.730+0000 7f752120f740 1 --2- 172.21.15.28:0/3222255535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 0x5587c8ed2010 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f751004aa50 tx=0x7f7510031cb0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.730+0000 7f752120f740 1 -- 172.21.15.28:0/3222255535 shutdown_connections 2024-04-25T01:32:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.730+0000 7f752120f740 1 --2- 172.21.15.28:0/3222255535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ecdfa0 0x5587c8ed2010 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.730+0000 7f752120f740 1 --2- 172.21.15.28:0/3222255535 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ec8e30 0x5587c8ed1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.730+0000 7f752120f740 1 --2- 172.21.15.28:0/3222255535 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8440 0x5587c8ec96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.730+0000 7f752120f740 1 -- 172.21.15.28:0/3222255535 >> 172.21.15.28:0/3222255535 conn(0x5587c8ec3ab0 msgr2=0x5587c8eca1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.730+0000 7f752120f740 1 -- 172.21.15.28:0/3222255535 shutdown_connections 2024-04-25T01:32:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.730+0000 7f752120f740 1 -- 172.21.15.28:0/3222255535 wait complete. 2024-04-25T01:32:26.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.730+0000 7f752120f740 1 -- start start 2024-04-25T01:32:26.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.730+0000 7f752120f740 1 Processor -- start 2024-04-25T01:32:26.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.730+0000 7f752120f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8440 0x5587c8f436a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f752120f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ec8e30 0x5587c8f45bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f752120f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ecdfa0 0x5587c8f48280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f752120f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5587c8ee0800 con 0x5587c8ec8e30 2024-04-25T01:32:26.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f752120f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5587c8ee0130 con 0x5587c8ec8440 2024-04-25T01:32:26.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f752120f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5587c8ee0ed0 con 0x5587c8ecdfa0 2024-04-25T01:32:26.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f752120f740 10 client.0 _ll_register_callbacks cb 0x7f7510031830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:26.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f75177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ec8e30 0x5587c8f45bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f75177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ec8e30 0x5587c8f45bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60696/0 (socket says 172.21.15.28:60696) 2024-04-25T01:32:26.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f75177fe640 1 -- 172.21.15.28:0/825330226 learned_addr learned my addr 172.21.15.28:0/825330226 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f7517fff640 1 --2- 172.21.15.28:0/825330226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8440 0x5587c8f436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f751c9c9640 1 --2- 172.21.15.28:0/825330226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ecdfa0 0x5587c8f48280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f75177fe640 1 -- 172.21.15.28:0/825330226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ecdfa0 msgr2=0x5587c8f48280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f75177fe640 1 --2- 172.21.15.28:0/825330226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ecdfa0 0x5587c8f48280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f75177fe640 1 -- 172.21.15.28:0/825330226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8440 msgr2=0x5587c8f436a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f75177fe640 1 --2- 172.21.15.28:0/825330226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8440 0x5587c8f436a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f75177fe640 1 -- 172.21.15.28:0/825330226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587c8f48ab0 con 0x5587c8ec8e30 2024-04-25T01:32:26.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f75177fe640 1 --2- 172.21.15.28:0/825330226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ec8e30 0x5587c8f45bf0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f750000fc30 tx=0x7f7500004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5144754285cc46ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f74f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:26.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f74f77fe640 1 -- 172.21.15.28:0/825330226 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7500029060 con 0x5587c8ec8e30 2024-04-25T01:32:26.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f74f77fe640 1 -- 172.21.15.28:0/825330226 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f7500005040 con 0x5587c8ec8e30 2024-04-25T01:32:26.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.731+0000 7f752120f740 1 -- 172.21.15.28:0/825330226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7510036040 con 0x5587c8ec8e30 2024-04-25T01:32:26.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.732+0000 7f74f77fe640 1 -- 172.21.15.28:0/825330226 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7500030050 con 0x5587c8ec8e30 2024-04-25T01:32:26.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.732+0000 7f752120f740 10 client.15309 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:26.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.732+0000 7f752120f740 1 -- 172.21.15.28:0/825330226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5587c8f49e20 con 0x5587c8ec8e30 2024-04-25T01:32:26.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.732+0000 7f74f77fe640 1 -- 172.21.15.28:0/825330226 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7510036040 con 0x5587c8ec8e30 2024-04-25T01:32:26.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.732+0000 7f74f77fe640 1 -- 172.21.15.28:0/825330226 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f75000035f0 con 0x5587c8ec8e30 2024-04-25T01:32:26.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.732+0000 7f752120f740 10 client.15309 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:26.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.732+0000 7f752120f740 10 client.15309 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:32:26.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.732+0000 7f752120f740 20 client.15309 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:26.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.732+0000 7f752120f740 1 --2- 172.21.15.28:0/825330226 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5587c8f4e6a0 0x5587c8f4aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.732+0000 7f752120f740 4 client.15309 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:32:26.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.732+0000 7f752120f740 1 -- 172.21.15.28:0/825330226 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "status"}) -- 0x5587c8e33790 con 0x5587c8f4e6a0 2024-04-25T01:32:26.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.732+0000 7f7517fff640 1 --2- 172.21.15.28:0/825330226 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5587c8f4e6a0 0x5587c8f4aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.733+0000 7f74f77fe640 10 client.15309 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:32:26.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.733+0000 7f7517fff640 1 --2- 172.21.15.28:0/825330226 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5587c8f4e6a0 0x5587c8f4aae0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=baf8cce75cc87e65 server_cookie=45b726ab5fbbfc89 in_seq=0 out_seq=0 2024-04-25T01:32:26.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.733+0000 7f74f77fe640 1 -- 172.21.15.28:0/825330226 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5587c8e33790 con 0x5587c8f4e6a0 2024-04-25T01:32:26.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.733+0000 7f74f77fe640 10 client.15309 handle_command_reply: tid=0 2024-04-25T01:32:26.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.733+0000 7f752120f740 1 client.15309 shutdown 2024-04-25T01:32:26.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.733+0000 7f752120f740 1 -- 172.21.15.28:0/825330226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ec8e30 msgr2=0x5587c8f45bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.733+0000 7f752120f740 1 --2- 172.21.15.28:0/825330226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ec8e30 0x5587c8f45bf0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f750000fc30 tx=0x7f7500004270 comp rx=0 tx=0).stop 2024-04-25T01:32:26.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.733+0000 7f752120f740 1 -- 172.21.15.28:0/825330226 shutdown_connections 2024-04-25T01:32:26.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.733+0000 7f752120f740 1 --2- 172.21.15.28:0/825330226 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5587c8f4e6a0 0x5587c8f4aae0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.733+0000 7f752120f740 1 --2- 172.21.15.28:0/825330226 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5587c8ecdfa0 0x5587c8f48280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.733+0000 7f752120f740 1 --2- 172.21.15.28:0/825330226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5587c8ec8e30 0x5587c8f45bf0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.733+0000 7f74f77fe640 0 client.15309 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:32:26.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.733+0000 7f752120f740 1 --2- 172.21.15.28:0/825330226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5587c8ec8440 0x5587c8f436a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.733+0000 7f752120f740 1 -- 172.21.15.28:0/825330226 >> 172.21.15.28:0/825330226 conn(0x5587c8ec3ab0 msgr2=0x5587c8ed1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.733+0000 7f752120f740 1 -- 172.21.15.28:0/825330226 shutdown_connections 2024-04-25T01:32:26.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.733+0000 7f752120f740 1 -- 172.21.15.28:0/825330226 wait complete. 2024-04-25T01:32:26.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.733+0000 7f752120f740 20 client.15309 trim_cache size 0 max 16384 2024-04-25T01:32:26.718 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:32:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "0e88f60a-02a2-11ef-bc93-c7b262605968", 2024-04-25T01:32:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-04-25T01:32:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24478, 2024-04-25T01:32:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-25T01:32:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-25T01:32:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-25T01:32:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 88.306537582999994, 2024-04-25T01:32:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-04-25T01:32:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-04-25T01:32:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 91, 2024-04-25T01:32:26.718 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 127.519985134 2024-04-25T01:32:26.718 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:32:26.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.734+0000 7f751e42c640 1 -- 172.21.15.28:0/4210474466 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f75040a2b90 msgr2=0x7f75040a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.734+0000 7f751e42c640 1 --2- 172.21.15.28:0/4210474466 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f75040a2b90 0x7f75040a5050 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x5587c8ec9690 tx=0x7f7510009380 comp rx=0 tx=0).stop 2024-04-25T01:32:26.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.734+0000 7f751e42c640 1 -- 172.21.15.28:0/4210474466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75181006a0 msgr2=0x7f75181979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.734+0000 7f751e42c640 1 --2- 172.21.15.28:0/4210474466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75181006a0 0x7f75181979e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f750800c970 tx=0x7f750800ce30 comp rx=0 tx=0).stop 2024-04-25T01:32:26.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.734+0000 7f751e42c640 1 -- 172.21.15.28:0/4210474466 shutdown_connections 2024-04-25T01:32:26.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.734+0000 7f751e42c640 1 --2- 172.21.15.28:0/4210474466 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f75040a2b90 0x7f75040a5050 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.734+0000 7f751e42c640 1 --2- 172.21.15.28:0/4210474466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7518102250 0x7f751819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.734+0000 7f751e42c640 1 --2- 172.21.15.28:0/4210474466 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7518101890 0x7f7518197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.734+0000 7f751e42c640 1 --2- 172.21.15.28:0/4210474466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75181006a0 0x7f75181979e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.734+0000 7f751e42c640 1 -- 172.21.15.28:0/4210474466 >> 172.21.15.28:0/4210474466 conn(0x7f75180fbc10 msgr2=0x7f7518104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.734+0000 7f751e42c640 1 -- 172.21.15.28:0/4210474466 shutdown_connections 2024-04-25T01:32:26.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.734+0000 7f751e42c640 1 -- 172.21.15.28:0/4210474466 wait complete. 2024-04-25T01:32:26.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T01:32:26.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.848+0000 7f47557b6640 1 -- 172.21.15.28:0/4209598277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4750104b30 msgr2=0x7f4750106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.848+0000 7f47557b6640 1 --2- 172.21.15.28:0/4209598277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4750104b30 0x7f4750106f20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4744009900 tx=0x7f474402a410 comp rx=0 tx=0).stop 2024-04-25T01:32:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.848+0000 7f47557b6640 1 -- 172.21.15.28:0/4209598277 shutdown_connections 2024-04-25T01:32:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.848+0000 7f47557b6640 1 --2- 172.21.15.28:0/4209598277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4750107460 0x7f47501098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.848+0000 7f47557b6640 1 --2- 172.21.15.28:0/4209598277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4750104b30 0x7f4750106f20 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.848+0000 7f47557b6640 1 --2- 172.21.15.28:0/4209598277 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4750102200 0x7f47501045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.848+0000 7f47557b6640 1 -- 172.21.15.28:0/4209598277 >> 172.21.15.28:0/4209598277 conn(0x7f47500fbbf0 msgr2=0x7f47500fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.848+0000 7f47557b6640 1 -- 172.21.15.28:0/4209598277 shutdown_connections 2024-04-25T01:32:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.849+0000 7f47557b6640 1 -- 172.21.15.28:0/4209598277 wait complete. 2024-04-25T01:32:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.849+0000 7f47557b6640 1 Processor -- start 2024-04-25T01:32:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.849+0000 7f47557b6640 1 -- start start 2024-04-25T01:32:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.849+0000 7f47557b6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4750102200 0x7f4750197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.849+0000 7f47557b6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4750104b30 0x7f4750198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.849+0000 7f47557b6640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4750107460 0x7f475019f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.849+0000 7f47557b6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f475010d410 con 0x7f4750104b30 2024-04-25T01:32:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.849+0000 7f47557b6640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f475010d290 con 0x7f4750102200 2024-04-25T01:32:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.849+0000 7f47557b6640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f475010d590 con 0x7f4750107460 2024-04-25T01:32:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.849+0000 7f474e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4750104b30 0x7f4750198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.850+0000 7f474e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4750104b30 0x7f4750198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60708/0 (socket says 172.21.15.28:60708) 2024-04-25T01:32:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.850+0000 7f474e7fc640 1 -- 172.21.15.28:0/295119592 learned_addr learned my addr 172.21.15.28:0/295119592 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.850+0000 7f474f7fe640 1 --2- 172.21.15.28:0/295119592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4750107460 0x7f475019f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.850+0000 7f474effd640 1 --2- 172.21.15.28:0/295119592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4750102200 0x7f4750197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.850+0000 7f474e7fc640 1 -- 172.21.15.28:0/295119592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4750107460 msgr2=0x7f475019f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.850+0000 7f474e7fc640 1 --2- 172.21.15.28:0/295119592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4750107460 0x7f475019f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.850+0000 7f474e7fc640 1 -- 172.21.15.28:0/295119592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4750102200 msgr2=0x7f4750197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.850+0000 7f474e7fc640 1 --2- 172.21.15.28:0/295119592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4750102200 0x7f4750197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.850+0000 7f474e7fc640 1 -- 172.21.15.28:0/295119592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f475019f760 con 0x7f4750104b30 2024-04-25T01:32:26.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.850+0000 7f474e7fc640 1 --2- 172.21.15.28:0/295119592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4750104b30 0x7f4750198080 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4744031610 tx=0x7f4744004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.850+0000 7f4737fff640 1 -- 172.21.15.28:0/295119592 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474403f030 con 0x7f4750104b30 2024-04-25T01:32:26.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.850+0000 7f4737fff640 1 -- 172.21.15.28:0/295119592 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4744003ad0 con 0x7f4750104b30 2024-04-25T01:32:26.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.850+0000 7f47557b6640 1 -- 172.21.15.28:0/295119592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f475019f9f0 con 0x7f4750104b30 2024-04-25T01:32:26.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.850+0000 7f4737fff640 1 -- 172.21.15.28:0/295119592 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4744035070 con 0x7f4750104b30 2024-04-25T01:32:26.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.850+0000 7f47557b6640 1 -- 172.21.15.28:0/295119592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f475019fee0 con 0x7f4750104b30 2024-04-25T01:32:26.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.853+0000 7f4737fff640 1 -- 172.21.15.28:0/295119592 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4744042030 con 0x7f4750104b30 2024-04-25T01:32:26.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.853+0000 7f475859c740 1 Processor -- start 2024-04-25T01:32:26.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.853+0000 7f475859c740 1 -- start start 2024-04-25T01:32:26.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.853+0000 7f475859c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9440 0x563f605f9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.853+0000 7f475859c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f605f9e30 0x563f605fe930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.853+0000 7f475859c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605fefa0 0x563f60601430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.853+0000 7f475859c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563f6055cf00 con 0x563f605f9440 2024-04-25T01:32:26.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.853+0000 7f475859c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563f60505750 con 0x563f605fefa0 2024-04-25T01:32:26.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.853+0000 7f475859c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x563f60558ec0 con 0x563f605f9e30 2024-04-25T01:32:26.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.853+0000 7f474f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9440 0x563f605f9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.853+0000 7f4737fff640 1 --2- 172.21.15.28:0/295119592 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f47380a2b90 0x7f47380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.853+0000 7f474f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9440 0x563f605f9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60710/0 (socket says 172.21.15.28:60710) 2024-04-25T01:32:26.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.853+0000 7f474f7fe640 1 -- 172.21.15.28:0/1731676218 learned_addr learned my addr 172.21.15.28:0/1731676218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.853+0000 7f474e7fc640 1 --2- 172.21.15.28:0/1731676218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605fefa0 0x563f60601430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.853+0000 7f474effd640 1 --2- 172.21.15.28:0/1731676218 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f605f9e30 0x563f605fe930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.853+0000 7f474f7fe640 1 --2- 172.21.15.28:0/295119592 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f47380a2b90 0x7f47380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.854+0000 7f4737fff640 1 -- 172.21.15.28:0/295119592 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f47440efb20 con 0x7f4750104b30 2024-04-25T01:32:26.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.854+0000 7f474f7fe640 1 -- 172.21.15.28:0/1731676218 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f605f9e30 msgr2=0x563f605fe930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.854+0000 7f474f7fe640 1 --2- 172.21.15.28:0/1731676218 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f605f9e30 0x563f605fe930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.854+0000 7f474f7fe640 1 -- 172.21.15.28:0/1731676218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605fefa0 msgr2=0x563f60601430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.854+0000 7f474f7fe640 1 --2- 172.21.15.28:0/1731676218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605fefa0 0x563f60601430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.854+0000 7f474f7fe640 1 -- 172.21.15.28:0/1731676218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f602712d0 con 0x563f605f9440 2024-04-25T01:32:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.854+0000 7f474f7fe640 1 --2- 172.21.15.28:0/1731676218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9440 0x563f605f9860 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4748013b20 tx=0x7f474800f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=375fc5a683d411ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.854+0000 7f47357fa640 1 -- 172.21.15.28:0/1731676218 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4748022020 con 0x563f605f9440 2024-04-25T01:32:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.854+0000 7f47357fa640 1 -- 172.21.15.28:0/1731676218 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f474801e400 con 0x563f605f9440 2024-04-25T01:32:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.854+0000 7f47357fa640 1 -- 172.21.15.28:0/1731676218 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4748013560 con 0x563f605f9440 2024-04-25T01:32:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.854+0000 7f474f7fe640 1 --2- 172.21.15.28:0/295119592 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f47380a2b90 0x7f47380a5050 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4748023000 tx=0x7f4748021000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f475859c740 1 -- 172.21.15.28:0/1731676218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9440 msgr2=0x563f605f9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f475859c740 1 --2- 172.21.15.28:0/1731676218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9440 0x563f605f9860 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4748013b20 tx=0x7f474800f3c0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f475859c740 1 -- 172.21.15.28:0/1731676218 shutdown_connections 2024-04-25T01:32:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f475859c740 1 --2- 172.21.15.28:0/1731676218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605fefa0 0x563f60601430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f475859c740 1 --2- 172.21.15.28:0/1731676218 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f605f9e30 0x563f605fe930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f475859c740 1 --2- 172.21.15.28:0/1731676218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9440 0x563f605f9860 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f475859c740 1 -- 172.21.15.28:0/1731676218 >> 172.21.15.28:0/1731676218 conn(0x563f605f4ab0 msgr2=0x563f605f4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f475859c740 1 -- 172.21.15.28:0/1731676218 shutdown_connections 2024-04-25T01:32:26.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f475859c740 1 -- 172.21.15.28:0/1731676218 wait complete. 2024-04-25T01:32:26.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f475859c740 1 -- start start 2024-04-25T01:32:26.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f475859c740 1 Processor -- start 2024-04-25T01:32:26.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f475859c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f605f9440 0x563f60603840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f475859c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9e30 0x563f60603d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f475859c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605fefa0 0x563f6060adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f475859c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563f60601f50 con 0x563f605f9e30 2024-04-25T01:32:26.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f475859c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563f60518cf0 con 0x563f605fefa0 2024-04-25T01:32:26.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f475859c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x563f606020d0 con 0x563f605f9440 2024-04-25T01:32:26.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f475859c740 10 client.0 _ll_register_callbacks cb 0x563f605186e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:26.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f474e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9e30 0x563f60603d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.855+0000 7f474e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9e30 0x563f60603d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60720/0 (socket says 172.21.15.28:60720) 2024-04-25T01:32:26.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.856+0000 7f474e7fc640 1 -- 172.21.15.28:0/2374547120 learned_addr learned my addr 172.21.15.28:0/2374547120 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.856+0000 7f474effd640 1 --2- 172.21.15.28:0/2374547120 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f605f9440 0x563f60603840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.856+0000 7f474f7fe640 1 --2- 172.21.15.28:0/2374547120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605fefa0 0x563f6060adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.856+0000 7f474e7fc640 1 -- 172.21.15.28:0/2374547120 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f605f9440 msgr2=0x563f60603840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.856+0000 7f474e7fc640 1 --2- 172.21.15.28:0/2374547120 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f605f9440 0x563f60603840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.856+0000 7f474e7fc640 1 -- 172.21.15.28:0/2374547120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605fefa0 msgr2=0x563f6060adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.856+0000 7f474e7fc640 1 --2- 172.21.15.28:0/2374547120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605fefa0 0x563f6060adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.856+0000 7f474e7fc640 1 -- 172.21.15.28:0/2374547120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f6033b6b0 con 0x563f605f9e30 2024-04-25T01:32:26.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.856+0000 7f474e7fc640 1 --2- 172.21.15.28:0/2374547120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9e30 0x563f60603d80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f47440b89f0 tx=0x7f474408afa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=973e43921495a4e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.856+0000 7f4732ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:26.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.856+0000 7f4732ff5640 1 -- 172.21.15.28:0/2374547120 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4744049040 con 0x563f605f9e30 2024-04-25T01:32:26.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.856+0000 7f4732ff5640 1 -- 172.21.15.28:0/2374547120 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f474403b460 con 0x563f605f9e30 2024-04-25T01:32:26.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.856+0000 7f475859c740 1 -- 172.21.15.28:0/2374547120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f474801e400 con 0x563f605f9e30 2024-04-25T01:32:26.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.856+0000 7f4732ff5640 1 -- 172.21.15.28:0/2374547120 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4744087100 con 0x563f605f9e30 2024-04-25T01:32:26.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.856+0000 7f475859c740 10 client.15321 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:26.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.856+0000 7f475859c740 1 -- 172.21.15.28:0/2374547120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563f6060dda0 con 0x563f605f9e30 2024-04-25T01:32:26.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.856+0000 7f4732ff5640 1 -- 172.21.15.28:0/2374547120 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f474801e400 con 0x563f605f9e30 2024-04-25T01:32:26.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.857+0000 7f4732ff5640 1 -- 172.21.15.28:0/2374547120 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f47440d8080 con 0x563f605f9e30 2024-04-25T01:32:26.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.857+0000 7f475859c740 10 client.15321 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:26.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.857+0000 7f475859c740 10 client.15321 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:32:26.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.857+0000 7f475859c740 20 client.15321 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:26.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.857+0000 7f475859c740 1 --2- 172.21.15.28:0/2374547120 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563f60610420 0x563f60611890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.857+0000 7f475859c740 4 client.15321 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:26.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.857+0000 7f475859c740 1 -- 172.21.15.28:0/2374547120 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f6056a440 con 0x563f60610420 2024-04-25T01:32:26.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.857+0000 7f474effd640 1 --2- 172.21.15.28:0/2374547120 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563f60610420 0x563f60611890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.857+0000 7f4732ff5640 10 client.15321 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:32:26.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.857+0000 7f474effd640 1 --2- 172.21.15.28:0/2374547120 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563f60610420 0x563f60611890 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d6349b2cd6a468a server_cookie=ca5b8ccdd7877d9b in_seq=0 out_seq=0 2024-04-25T01:32:26.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.858+0000 7f4732ff5640 1 -- 172.21.15.28:0/2374547120 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563f6056a440 con 0x563f60610420 2024-04-25T01:32:26.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.858+0000 7f4732ff5640 10 client.15321 handle_command_reply: tid=0 2024-04-25T01:32:26.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.858+0000 7f475859c740 1 client.15321 shutdown 2024-04-25T01:32:26.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.858+0000 7f475859c740 1 -- 172.21.15.28:0/2374547120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9e30 msgr2=0x563f60603d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.858+0000 7f475859c740 1 --2- 172.21.15.28:0/2374547120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9e30 0x563f60603d80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f47440b89f0 tx=0x7f474408afa0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.858+0000 7f475859c740 1 -- 172.21.15.28:0/2374547120 shutdown_connections 2024-04-25T01:32:26.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.858+0000 7f475859c740 1 --2- 172.21.15.28:0/2374547120 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563f60610420 0x563f60611890 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.858+0000 7f475859c740 1 --2- 172.21.15.28:0/2374547120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605fefa0 0x563f6060adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.858+0000 7f475859c740 1 --2- 172.21.15.28:0/2374547120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9e30 0x563f60603d80 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.858+0000 7f475859c740 1 --2- 172.21.15.28:0/2374547120 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f605f9440 0x563f60603840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.858+0000 7f4732ff5640 0 client.15321 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:32:26.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.858+0000 7f475859c740 1 -- 172.21.15.28:0/2374547120 >> 172.21.15.28:0/2374547120 conn(0x563f605f4ab0 msgr2=0x563f605f3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.858+0000 7f475859c740 1 -- 172.21.15.28:0/2374547120 shutdown_connections 2024-04-25T01:32:26.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.858+0000 7f475859c740 1 -- 172.21.15.28:0/2374547120 wait complete. 2024-04-25T01:32:26.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.858+0000 7f475859c740 20 client.15321 trim_cache size 0 max 16384 2024-04-25T01:32:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.876+0000 7f475859c740 1 Processor -- start 2024-04-25T01:32:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.876+0000 7f475859c740 1 -- start start 2024-04-25T01:32:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f475859c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9440 0x563f60601390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f475859c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605f9e30 0x563f6060ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f475859c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f605fefa0 0x563f605ef690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f475859c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563f6060fbd0 con 0x563f605f9440 2024-04-25T01:32:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f475859c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563f606102a0 con 0x563f605f9e30 2024-04-25T01:32:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f475859c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x563f60610970 con 0x563f605fefa0 2024-04-25T01:32:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f474effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9440 0x563f60601390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f474effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9440 0x563f60601390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60730/0 (socket says 172.21.15.28:60730) 2024-04-25T01:32:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f474effd640 1 -- 172.21.15.28:0/2935573790 learned_addr learned my addr 172.21.15.28:0/2935573790 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f474e7fc640 1 --2- 172.21.15.28:0/2935573790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605f9e30 0x563f6060ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f474effd640 1 -- 172.21.15.28:0/2935573790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f605fefa0 msgr2=0x563f605ef690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f474effd640 1 --2- 172.21.15.28:0/2935573790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f605fefa0 0x563f605ef690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f474effd640 1 -- 172.21.15.28:0/2935573790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605f9e30 msgr2=0x563f6060ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f474effd640 1 --2- 172.21.15.28:0/2935573790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605f9e30 0x563f6060ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f474effd640 1 -- 172.21.15.28:0/2935573790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f60610e20 con 0x563f605f9440 2024-04-25T01:32:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f474e7fc640 1 --2- 172.21.15.28:0/2935573790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605f9e30 0x563f6060ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f474effd640 1 --2- 172.21.15.28:0/2935573790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9440 0x563f60601390 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x563f605fa230 tx=0x7f47400138f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=349f3b09313da686 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f4734ff9640 1 -- 172.21.15.28:0/2935573790 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474001d430 con 0x563f605f9440 2024-04-25T01:32:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f4734ff9640 1 -- 172.21.15.28:0/2935573790 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4740005020 con 0x563f605f9440 2024-04-25T01:32:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.877+0000 7f4734ff9640 1 -- 172.21.15.28:0/2935573790 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4740004ca0 con 0x563f605f9440 2024-04-25T01:32:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.878+0000 7f475859c740 1 -- 172.21.15.28:0/2935573790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9440 msgr2=0x563f60601390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.878+0000 7f475859c740 1 --2- 172.21.15.28:0/2935573790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9440 0x563f60601390 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x563f605fa230 tx=0x7f47400138f0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.878+0000 7f475859c740 1 -- 172.21.15.28:0/2935573790 shutdown_connections 2024-04-25T01:32:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.878+0000 7f475859c740 1 --2- 172.21.15.28:0/2935573790 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f605fefa0 0x563f605ef690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.878+0000 7f475859c740 1 --2- 172.21.15.28:0/2935573790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605f9e30 0x563f6060ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.878+0000 7f475859c740 1 --2- 172.21.15.28:0/2935573790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605f9440 0x563f60601390 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.878+0000 7f475859c740 1 -- 172.21.15.28:0/2935573790 >> 172.21.15.28:0/2935573790 conn(0x563f605f4ab0 msgr2=0x563f605f4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.878+0000 7f475859c740 1 -- 172.21.15.28:0/2935573790 shutdown_connections 2024-04-25T01:32:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.878+0000 7f475859c740 1 -- 172.21.15.28:0/2935573790 wait complete. 2024-04-25T01:32:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.878+0000 7f475859c740 1 -- start start 2024-04-25T01:32:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.878+0000 7f475859c740 1 Processor -- start 2024-04-25T01:32:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.878+0000 7f475859c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f605f9440 0x563f60606c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.878+0000 7f475859c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605f9e30 0x563f60607190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.878+0000 7f475859c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605fefa0 0x563f6060a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.878+0000 7f475859c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563f60611710 con 0x563f605fefa0 2024-04-25T01:32:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.878+0000 7f475859c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563f60611040 con 0x563f605f9e30 2024-04-25T01:32:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.878+0000 7f475859c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x563f60611de0 con 0x563f605f9440 2024-04-25T01:32:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.878+0000 7f475859c740 10 client.0 _ll_register_callbacks cb 0x7f4744009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.879+0000 7f474f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605fefa0 0x563f6060a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.879+0000 7f474f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605fefa0 0x563f6060a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60744/0 (socket says 172.21.15.28:60744) 2024-04-25T01:32:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.879+0000 7f474f7fe640 1 -- 172.21.15.28:0/1807687958 learned_addr learned my addr 172.21.15.28:0/1807687958 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.879+0000 7f474e7fc640 1 --2- 172.21.15.28:0/1807687958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605f9e30 0x563f60607190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.879+0000 7f474effd640 1 --2- 172.21.15.28:0/1807687958 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f605f9440 0x563f60606c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.879+0000 7f474f7fe640 1 -- 172.21.15.28:0/1807687958 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f605f9440 msgr2=0x563f60606c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.879+0000 7f474f7fe640 1 --2- 172.21.15.28:0/1807687958 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f605f9440 0x563f60606c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.879+0000 7f474f7fe640 1 -- 172.21.15.28:0/1807687958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605f9e30 msgr2=0x563f60607190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.879+0000 7f474f7fe640 1 --2- 172.21.15.28:0/1807687958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605f9e30 0x563f60607190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.879+0000 7f474f7fe640 1 -- 172.21.15.28:0/1807687958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f6060a730 con 0x563f605fefa0 2024-04-25T01:32:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.879+0000 7f474f7fe640 1 --2- 172.21.15.28:0/1807687958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605fefa0 0x563f6060a140 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4748011000 tx=0x7f4748005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf68865bd0ffdd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.879+0000 7f4732ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.879+0000 7f4732ff5640 1 -- 172.21.15.28:0/1807687958 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4748038060 con 0x563f605fefa0 2024-04-25T01:32:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.879+0000 7f4732ff5640 1 -- 172.21.15.28:0/1807687958 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f4748012070 con 0x563f605fefa0 2024-04-25T01:32:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.879+0000 7f475859c740 1 -- 172.21.15.28:0/1807687958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4740005020 con 0x563f605fefa0 2024-04-25T01:32:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.879+0000 7f4732ff5640 1 -- 172.21.15.28:0/1807687958 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f474803f050 con 0x563f605fefa0 2024-04-25T01:32:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.879+0000 7f475859c740 10 client.15327 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.879+0000 7f475859c740 1 -- 172.21.15.28:0/1807687958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563f60676b30 con 0x563f605fefa0 2024-04-25T01:32:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.880+0000 7f4732ff5640 1 -- 172.21.15.28:0/1807687958 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4740005020 con 0x563f605fefa0 2024-04-25T01:32:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.880+0000 7f4732ff5640 1 -- 172.21.15.28:0/1807687958 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f4748034520 con 0x563f605fefa0 2024-04-25T01:32:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.880+0000 7f475859c740 10 client.15327 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.880+0000 7f475859c740 10 client.15327 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:32:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.880+0000 7f475859c740 20 client.15327 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.880+0000 7f475859c740 1 --2- 172.21.15.28:0/1807687958 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563f6067b590 0x563f6067a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.880+0000 7f475859c740 4 client.15327 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:32:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.880+0000 7f475859c740 1 -- 172.21.15.28:0/1807687958 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "status"}) -- 0x563f60564790 con 0x563f6067b590 2024-04-25T01:32:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.880+0000 7f474effd640 1 --2- 172.21.15.28:0/1807687958 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563f6067b590 0x563f6067a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.881+0000 7f4732ff5640 10 client.15327 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:32:26.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.881+0000 7f474effd640 1 --2- 172.21.15.28:0/1807687958 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563f6067b590 0x563f6067a2a0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a2d978d45bd4f09 server_cookie=2df42c9ebb6ab3c0 in_seq=0 out_seq=0 2024-04-25T01:32:26.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.881+0000 7f4732ff5640 1 -- 172.21.15.28:0/1807687958 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x563f60564790 con 0x563f6067b590 2024-04-25T01:32:26.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.881+0000 7f4732ff5640 10 client.15327 handle_command_reply: tid=0 2024-04-25T01:32:26.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.881+0000 7f475859c740 1 client.15327 shutdown 2024-04-25T01:32:26.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.881+0000 7f475859c740 1 -- 172.21.15.28:0/1807687958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605fefa0 msgr2=0x563f6060a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.881+0000 7f475859c740 1 --2- 172.21.15.28:0/1807687958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605fefa0 0x563f6060a140 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4748011000 tx=0x7f4748005bd0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.881+0000 7f475859c740 1 -- 172.21.15.28:0/1807687958 shutdown_connections 2024-04-25T01:32:26.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.881+0000 7f475859c740 1 --2- 172.21.15.28:0/1807687958 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563f6067b590 0x563f6067a2a0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.881+0000 7f475859c740 1 --2- 172.21.15.28:0/1807687958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f605fefa0 0x563f6060a140 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.881+0000 7f475859c740 1 --2- 172.21.15.28:0/1807687958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563f605f9e30 0x563f60607190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.881+0000 7f475859c740 1 --2- 172.21.15.28:0/1807687958 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563f605f9440 0x563f60606c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.881+0000 7f4732ff5640 0 client.15327 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:32:26.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.881+0000 7f475859c740 1 -- 172.21.15.28:0/1807687958 >> 172.21.15.28:0/1807687958 conn(0x563f605f4ab0 msgr2=0x563f605ef690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.881+0000 7f475859c740 1 -- 172.21.15.28:0/1807687958 shutdown_connections 2024-04-25T01:32:26.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.881+0000 7f475859c740 1 -- 172.21.15.28:0/1807687958 wait complete. 2024-04-25T01:32:26.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.881+0000 7f475859c740 20 client.15327 trim_cache size 0 max 16384 2024-04-25T01:32:26.867 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:32:26.867 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "0e88f60a-02a2-11ef-bc93-c7b262605968", 2024-04-25T01:32:26.867 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-04-25T01:32:26.867 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24512, 2024-04-25T01:32:26.867 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-25T01:32:26.867 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-25T01:32:26.867 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-25T01:32:26.867 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 112.54864956900001, 2024-04-25T01:32:26.868 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-04-25T01:32:26.868 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-04-25T01:32:26.868 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 91, 2024-04-25T01:32:26.868 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 118.76998842499999 2024-04-25T01:32:26.868 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:32:26.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.882+0000 7f47557b6640 1 -- 172.21.15.28:0/295119592 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f47380a2b90 msgr2=0x7f47380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.882+0000 7f47557b6640 1 --2- 172.21.15.28:0/295119592 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f47380a2b90 0x7f47380a5050 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4748023000 tx=0x7f4748021000 comp rx=0 tx=0).stop 2024-04-25T01:32:26.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.882+0000 7f47557b6640 1 -- 172.21.15.28:0/295119592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4750104b30 msgr2=0x7f4750198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:26.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.882+0000 7f47557b6640 1 --2- 172.21.15.28:0/295119592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4750104b30 0x7f4750198080 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4744031610 tx=0x7f4744004270 comp rx=0 tx=0).stop 2024-04-25T01:32:26.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.882+0000 7f47557b6640 1 -- 172.21.15.28:0/295119592 shutdown_connections 2024-04-25T01:32:26.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.882+0000 7f47557b6640 1 --2- 172.21.15.28:0/295119592 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f47380a2b90 0x7f47380a5050 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.882+0000 7f47557b6640 1 --2- 172.21.15.28:0/295119592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4750107460 0x7f475019f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.882+0000 7f47557b6640 1 --2- 172.21.15.28:0/295119592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4750104b30 0x7f4750198080 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.882+0000 7f47557b6640 1 --2- 172.21.15.28:0/295119592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4750102200 0x7f4750197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:26.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.882+0000 7f47557b6640 1 -- 172.21.15.28:0/295119592 >> 172.21.15.28:0/295119592 conn(0x7f47500fbbf0 msgr2=0x7f4750102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:26.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.882+0000 7f47557b6640 1 -- 172.21.15.28:0/295119592 shutdown_connections 2024-04-25T01:32:26.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:26.882+0000 7f47557b6640 1 -- 172.21.15.28:0/295119592 wait complete. 2024-04-25T01:32:26.871 INFO:tasks.cephadm:Setup complete, yielding 2024-04-25T01:32:26.871 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T01:32:26.880 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T01:32:26.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-25T01:32:27.136 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:32:27.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:27 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3950680693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:27.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:27 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3950680693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:27.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:27 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3950680693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:28.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:28 smithi130 ceph-mon[32702]: pgmap v356: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:32:28.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:28 smithi032 ceph-mon[32461]: pgmap v356: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:32:28.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:28 smithi028 ceph-mon[29390]: pgmap v356: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:32:29.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.129+0000 7f85f31c7700 1 -- 172.21.15.28:0/3246172641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ec107fa0 msgr2=0x7f85ec10a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:29.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.129+0000 7f85f31c7700 1 --2- 172.21.15.28:0/3246172641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ec107fa0 0x7f85ec10a410 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f85e400b0a0 tx=0x7f85e400b3b0 comp rx=0 tx=0).stop 2024-04-25T01:32:29.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.129+0000 7f85f31c7700 1 -- 172.21.15.28:0/3246172641 shutdown_connections 2024-04-25T01:32:29.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.129+0000 7f85f31c7700 1 --2- 172.21.15.28:0/3246172641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ec107fa0 0x7f85ec10a410 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:29.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.129+0000 7f85f31c7700 1 --2- 172.21.15.28:0/3246172641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102d40 0x7f85ec105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:29.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.129+0000 7f85f31c7700 1 --2- 172.21.15.28:0/3246172641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85ec105670 0x7f85ec107a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:29.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.129+0000 7f85f31c7700 1 -- 172.21.15.28:0/3246172641 >> 172.21.15.28:0/3246172641 conn(0x7f85ec0fc750 msgr2=0x7f85ec0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:29.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.129+0000 7f85f31c7700 1 -- 172.21.15.28:0/3246172641 shutdown_connections 2024-04-25T01:32:29.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.129+0000 7f85f31c7700 1 -- 172.21.15.28:0/3246172641 wait complete. 2024-04-25T01:32:29.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.130+0000 7f85f31c7700 1 Processor -- start 2024-04-25T01:32:29.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.130+0000 7f85f31c7700 1 -- start start 2024-04-25T01:32:29.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.130+0000 7f85f31c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102d40 0x7f85ec1026a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:29.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.130+0000 7f85f31c7700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85ec105670 0x7f85ec100cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:29.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.130+0000 7f85f31c7700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ec107fa0 0x7f85ec101230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:29.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.130+0000 7f85f31c7700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85ec10dec0 con 0x7f85ec107fa0 2024-04-25T01:32:29.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.130+0000 7f85f31c7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f85ec10dd40 con 0x7f85ec102d40 2024-04-25T01:32:29.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.130+0000 7f85f31c7700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f85ec10e040 con 0x7f85ec105670 2024-04-25T01:32:29.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.130+0000 7f85f1764700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ec107fa0 0x7f85ec101230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:29.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.130+0000 7f85f1764700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ec107fa0 0x7f85ec101230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60756/0 (socket says 172.21.15.28:60756) 2024-04-25T01:32:29.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.130+0000 7f85f1764700 1 -- 172.21.15.28:0/1653293968 learned_addr learned my addr 172.21.15.28:0/1653293968 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:29.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.130+0000 7f85ebfff700 1 --2- 172.21.15.28:0/1653293968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85ec105670 0x7f85ec100cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.130+0000 7f85f0f63700 1 --2- 172.21.15.28:0/1653293968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102d40 0x7f85ec1026a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.130+0000 7f85f1764700 1 -- 172.21.15.28:0/1653293968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85ec105670 msgr2=0x7f85ec100cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.130+0000 7f85f1764700 1 --2- 172.21.15.28:0/1653293968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85ec105670 0x7f85ec100cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.130+0000 7f85f1764700 1 -- 172.21.15.28:0/1653293968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102d40 msgr2=0x7f85ec1026a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.130+0000 7f85f1764700 1 --2- 172.21.15.28:0/1653293968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102d40 0x7f85ec1026a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.130+0000 7f85f1764700 1 -- 172.21.15.28:0/1653293968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ec101ae0 con 0x7f85ec107fa0 2024-04-25T01:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.130+0000 7f85f1764700 1 --2- 172.21.15.28:0/1653293968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ec107fa0 0x7f85ec101230 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f85e4007f00 tx=0x7f85e4004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.131+0000 7f85e9ffb700 1 -- 172.21.15.28:0/1653293968 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e4021070 con 0x7f85ec107fa0 2024-04-25T01:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.131+0000 7f85e9ffb700 1 -- 172.21.15.28:0/1653293968 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85e401d930 con 0x7f85ec107fa0 2024-04-25T01:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.131+0000 7f85f31c7700 1 -- 172.21.15.28:0/1653293968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85ec1a2540 con 0x7f85ec107fa0 2024-04-25T01:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.131+0000 7f85e9ffb700 1 -- 172.21.15.28:0/1653293968 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e40055a0 con 0x7f85ec107fa0 2024-04-25T01:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.131+0000 7f85f31c7700 1 -- 172.21.15.28:0/1653293968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85ec1a29a0 con 0x7f85ec107fa0 2024-04-25T01:32:29.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.132+0000 7f85f31c7700 1 -- 172.21.15.28:0/1653293968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85ec068f20 con 0x7f85ec107fa0 2024-04-25T01:32:29.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.133+0000 7f85e9ffb700 1 -- 172.21.15.28:0/1653293968 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f85e4024070 con 0x7f85ec107fa0 2024-04-25T01:32:29.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.134+0000 7f85e9ffb700 1 --2- 172.21.15.28:0/1653293968 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f85d40a2c20 0x7f85d40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:29.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.134+0000 7f85f0f63700 1 --2- 172.21.15.28:0/1653293968 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f85d40a2c20 0x7f85d40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:29.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.134+0000 7f85e9ffb700 1 -- 172.21.15.28:0/1653293968 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f85e400f020 con 0x7f85ec107fa0 2024-04-25T01:32:29.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.135+0000 7f85f0f63700 1 --2- 172.21.15.28:0/1653293968 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f85d40a2c20 0x7f85d40a50e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f85dc005950 tx=0x7f85dc005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:29.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.138+0000 7f85e9ffb700 1 -- 172.21.15.28:0/1653293968 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f85e402b020 con 0x7f85ec107fa0 2024-04-25T01:32:29.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.364+0000 7f85f31c7700 1 -- 172.21.15.28:0/1653293968 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f85ec062670 con 0x7f85d40a2c20 2024-04-25T01:32:29.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.365+0000 7f85e9ffb700 1 -- 172.21.15.28:0/1653293968 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f85ec062670 con 0x7f85d40a2c20 2024-04-25T01:32:29.342 INFO:teuthology.orchestra.run.smithi028.stdout:Backend: cephadm 2024-04-25T01:32:29.342 INFO:teuthology.orchestra.run.smithi028.stdout:Available: Yes 2024-04-25T01:32:29.342 INFO:teuthology.orchestra.run.smithi028.stdout:Paused: No 2024-04-25T01:32:29.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.367+0000 7f85f31c7700 1 -- 172.21.15.28:0/1653293968 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f85d40a2c20 msgr2=0x7f85d40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:29.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.367+0000 7f85f31c7700 1 --2- 172.21.15.28:0/1653293968 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f85d40a2c20 0x7f85d40a50e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f85dc005950 tx=0x7f85dc005cf0 comp rx=0 tx=0).stop 2024-04-25T01:32:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.367+0000 7f85f31c7700 1 -- 172.21.15.28:0/1653293968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ec107fa0 msgr2=0x7f85ec101230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.367+0000 7f85f31c7700 1 --2- 172.21.15.28:0/1653293968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ec107fa0 0x7f85ec101230 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f85e4007f00 tx=0x7f85e4004e40 comp rx=0 tx=0).stop 2024-04-25T01:32:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.367+0000 7f85f31c7700 1 -- 172.21.15.28:0/1653293968 shutdown_connections 2024-04-25T01:32:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.367+0000 7f85f31c7700 1 --2- 172.21.15.28:0/1653293968 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f85d40a2c20 0x7f85d40a50e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.367+0000 7f85f31c7700 1 --2- 172.21.15.28:0/1653293968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ec107fa0 0x7f85ec101230 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.367+0000 7f85f31c7700 1 --2- 172.21.15.28:0/1653293968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85ec102d40 0x7f85ec1026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.367+0000 7f85f31c7700 1 --2- 172.21.15.28:0/1653293968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f85ec105670 0x7f85ec100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.367+0000 7f85f31c7700 1 -- 172.21.15.28:0/1653293968 >> 172.21.15.28:0/1653293968 conn(0x7f85ec0fc750 msgr2=0x7f85ec103680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.367+0000 7f85f31c7700 1 -- 172.21.15.28:0/1653293968 shutdown_connections 2024-04-25T01:32:29.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:29.367+0000 7f85f31c7700 1 -- 172.21.15.28:0/1653293968 wait complete. 2024-04-25T01:32:30.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-25T01:32:30.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:30 smithi028 ceph-mon[29390]: pgmap v357: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:32:30.334 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:32:30.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:30 smithi130 ceph-mon[32702]: pgmap v357: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:32:30.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:30 smithi032 ceph-mon[32461]: pgmap v357: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T01:32:31.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:31 smithi130 ceph-mon[32702]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:32:31.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:31 smithi032 ceph-mon[32461]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:32:31.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:31 smithi028 ceph-mon[29390]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:32:32.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.144+0000 7ff874af4700 1 -- 172.21.15.28:0/840749594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff870100960 msgr2=0x7ff870107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.144+0000 7ff874af4700 1 --2- 172.21.15.28:0/840749594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff870100960 0x7ff870107970 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff864009990 tx=0x7ff864009ca0 comp rx=0 tx=0).stop 2024-04-25T01:32:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.145+0000 7ff874af4700 1 -- 172.21.15.28:0/840749594 shutdown_connections 2024-04-25T01:32:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.145+0000 7ff874af4700 1 --2- 172.21.15.28:0/840749594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff870107f40 0x7ff87010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.145+0000 7ff874af4700 1 --2- 172.21.15.28:0/840749594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff870100000 0x7ff870100420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.145+0000 7ff874af4700 1 --2- 172.21.15.28:0/840749594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff870100960 0x7ff870107970 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.145+0000 7ff874af4700 1 -- 172.21.15.28:0/840749594 >> 172.21.15.28:0/840749594 conn(0x7ff8700fba80 msgr2=0x7ff8700fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.145+0000 7ff874af4700 1 -- 172.21.15.28:0/840749594 shutdown_connections 2024-04-25T01:32:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.145+0000 7ff874af4700 1 -- 172.21.15.28:0/840749594 wait complete. 2024-04-25T01:32:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.145+0000 7ff874af4700 1 Processor -- start 2024-04-25T01:32:32.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.145+0000 7ff874af4700 1 -- start start 2024-04-25T01:32:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.146+0000 7ff874af4700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff870100000 0x7ff8701032b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.146+0000 7ff874af4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff870100960 0x7ff870101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.146+0000 7ff874af4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff870107f40 0x7ff870101e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.146+0000 7ff874af4700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff87010de60 con 0x7ff870107f40 2024-04-25T01:32:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.146+0000 7ff874af4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff87010dce0 con 0x7ff870100960 2024-04-25T01:32:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.146+0000 7ff874af4700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff87010dfe0 con 0x7ff870100000 2024-04-25T01:32:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.146+0000 7ff86ed9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff870107f40 0x7ff870101e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.146+0000 7ff86ed9d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff870107f40 0x7ff870101e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60776/0 (socket says 172.21.15.28:60776) 2024-04-25T01:32:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.146+0000 7ff86ed9d700 1 -- 172.21.15.28:0/2274755297 learned_addr learned my addr 172.21.15.28:0/2274755297 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.146+0000 7ff86e59c700 1 --2- 172.21.15.28:0/2274755297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff870100000 0x7ff8701032b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.146+0000 7ff86dd9b700 1 --2- 172.21.15.28:0/2274755297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff870100960 0x7ff870101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:32.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.146+0000 7ff86ed9d700 1 -- 172.21.15.28:0/2274755297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff870100000 msgr2=0x7ff8701032b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.146+0000 7ff86ed9d700 1 --2- 172.21.15.28:0/2274755297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff870100000 0x7ff8701032b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.146+0000 7ff86ed9d700 1 -- 172.21.15.28:0/2274755297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff870100960 msgr2=0x7ff870101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.146+0000 7ff86ed9d700 1 --2- 172.21.15.28:0/2274755297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff870100960 0x7ff870101900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.146+0000 7ff86ed9d700 1 -- 172.21.15.28:0/2274755297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8701026f0 con 0x7ff870107f40 2024-04-25T01:32:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.146+0000 7ff86ed9d700 1 --2- 172.21.15.28:0/2274755297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff870107f40 0x7ff870101e40 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff86800ada0 tx=0x7ff86800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.147+0000 7ff85f7fe700 1 -- 172.21.15.28:0/2274755297 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff868017020 con 0x7ff870107f40 2024-04-25T01:32:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.147+0000 7ff85f7fe700 1 -- 172.21.15.28:0/2274755297 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff86800a0d0 con 0x7ff870107f40 2024-04-25T01:32:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.147+0000 7ff874af4700 1 -- 172.21.15.28:0/2274755297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8701a2680 con 0x7ff870107f40 2024-04-25T01:32:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.147+0000 7ff85f7fe700 1 -- 172.21.15.28:0/2274755297 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff868005480 con 0x7ff870107f40 2024-04-25T01:32:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.147+0000 7ff874af4700 1 -- 172.21.15.28:0/2274755297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8701a2b90 con 0x7ff870107f40 2024-04-25T01:32:32.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.148+0000 7ff874af4700 1 -- 172.21.15.28:0/2274755297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff870068f20 con 0x7ff870107f40 2024-04-25T01:32:32.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.149+0000 7ff85f7fe700 1 -- 172.21.15.28:0/2274755297 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff868005620 con 0x7ff870107f40 2024-04-25T01:32:32.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.149+0000 7ff85f7fe700 1 --2- 172.21.15.28:0/2274755297 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff8580a2ca0 0x7ff8580a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:32.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.149+0000 7ff86e59c700 1 --2- 172.21.15.28:0/2274755297 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff8580a2ca0 0x7ff8580a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:32.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.149+0000 7ff85f7fe700 1 -- 172.21.15.28:0/2274755297 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7ff8680ceac0 con 0x7ff870107f40 2024-04-25T01:32:32.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.150+0000 7ff86e59c700 1 --2- 172.21.15.28:0/2274755297 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff8580a2ca0 0x7ff8580a5160 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff860005950 tx=0x7ff86000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:32.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.154+0000 7ff85f7fe700 1 -- 172.21.15.28:0/2274755297 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff868021020 con 0x7ff870107f40 2024-04-25T01:32:32.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:32 smithi028 ceph-mon[29390]: pgmap v358: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-25T01:32:32.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.372+0000 7ff874af4700 1 -- 172.21.15.28:0/2274755297 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7ff870062670 con 0x7ff8580a2ca0 2024-04-25T01:32:32.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.386+0000 7ff85f7fe700 1 -- 172.21.15.28:0/2274755297 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7ff870062670 con 0x7ff8580a2ca0 2024-04-25T01:32:32.363 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-25T01:32:32.363 INFO:teuthology.orchestra.run.smithi028.stdout:mds.a smithi028 running (2m) 21s ago 2m 18.7M - 19.0.0-2445-gd2c45d97 f68e37b807ef 739a81cfd2eb 2024-04-25T01:32:32.364 INFO:teuthology.orchestra.run.smithi028.stdout:mds.b smithi032 running (2m) 57s ago 2m 26.1M - 19.0.0-2445-gd2c45d97 f68e37b807ef f3b2a1d13e2f 2024-04-25T01:32:32.364 INFO:teuthology.orchestra.run.smithi028.stdout:mds.c smithi130 running (2m) 57s ago 2m 19.6M - 19.0.0-2445-gd2c45d97 f68e37b807ef 9e1ca5001ff5 2024-04-25T01:32:32.364 INFO:teuthology.orchestra.run.smithi028.stdout:mds.d smithi028 running (2m) 21s ago 2m 15.4M - 19.0.0-2445-gd2c45d97 f68e37b807ef 7e9e51fb5540 2024-04-25T01:32:32.364 INFO:teuthology.orchestra.run.smithi028.stdout:mds.e smithi032 running (2m) 57s ago 2m 19.5M - 19.0.0-2445-gd2c45d97 f68e37b807ef f70b176c5ce2 2024-04-25T01:32:32.364 INFO:teuthology.orchestra.run.smithi028.stdout:mds.f smithi130 running (2m) 57s ago 2m 19.1M - 19.0.0-2445-gd2c45d97 f68e37b807ef e9caaadfedf8 2024-04-25T01:32:32.364 INFO:teuthology.orchestra.run.smithi028.stdout:mds.g smithi028 running (2m) 21s ago 2m 18.6M - 19.0.0-2445-gd2c45d97 f68e37b807ef d92727996a48 2024-04-25T01:32:32.364 INFO:teuthology.orchestra.run.smithi028.stdout:mds.h smithi032 running (2m) 57s ago 2m 18.5M - 19.0.0-2445-gd2c45d97 f68e37b807ef 4a375432e3bf 2024-04-25T01:32:32.364 INFO:teuthology.orchestra.run.smithi028.stdout:mds.i smithi130 running (2m) 57s ago 2m 17.5M - 19.0.0-2445-gd2c45d97 f68e37b807ef af61a3080c0d 2024-04-25T01:32:32.364 INFO:teuthology.orchestra.run.smithi028.stdout:mds.j smithi028 running (2m) 21s ago 2m 15.4M - 19.0.0-2445-gd2c45d97 f68e37b807ef a369550f8ace 2024-04-25T01:32:32.364 INFO:teuthology.orchestra.run.smithi028.stdout:mds.k smithi032 running (2m) 57s ago 2m 17.4M - 19.0.0-2445-gd2c45d97 f68e37b807ef 68adf427738e 2024-04-25T01:32:32.364 INFO:teuthology.orchestra.run.smithi028.stdout:mds.l smithi130 running (2m) 57s ago 2m 18.3M - 19.0.0-2445-gd2c45d97 f68e37b807ef fac764c82e12 2024-04-25T01:32:32.364 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi028 *:9283,8765 running (9m) 21s ago 9m 514M - 19.0.0-2445-gd2c45d97 f68e37b807ef a137f351b2ca 2024-04-25T01:32:32.364 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi032 *:8443,8765 running (7m) 57s ago 7m 451M - 19.0.0-2445-gd2c45d97 f68e37b807ef 4db81abec7f9 2024-04-25T01:32:32.364 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.z smithi130 *:8443,8765 running (8m) 57s ago 7m 454M - 19.0.0-2445-gd2c45d97 f68e37b807ef 0e6b2e060ce6 2024-04-25T01:32:32.365 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (10m) 21s ago 10m 50.8M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef 8b32243128b3 2024-04-25T01:32:32.365 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi032 running (8m) 57s ago 8m 40.7M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef 9ebdfa9017eb 2024-04-25T01:32:32.365 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi130 running (8m) 57s ago 8m 42.8M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef 3cdb8931f7c8 2024-04-25T01:32:32.365 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (7m) 21s ago 7m 52.9M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef fff12e04c30a 2024-04-25T01:32:32.365 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi032 running (7m) 57s ago 7m 47.2M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef b0bf2dcffea2 2024-04-25T01:32:32.365 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi130 running (6m) 57s ago 6m 54.7M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 6bd6c9047a9c 2024-04-25T01:32:32.365 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (6m) 21s ago 6m 54.7M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef 7e189f64276b 2024-04-25T01:32:32.365 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi032 running (6m) 57s ago 6m 41.8M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef 1ac9e92d035a 2024-04-25T01:32:32.365 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi130 running (5m) 57s ago 5m 55.4M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 1d62fdc627f4 2024-04-25T01:32:32.365 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi028 running (5m) 21s ago 5m 43.8M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef 9917501fa82b 2024-04-25T01:32:32.365 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi032 running (4m) 57s ago 4m 44.8M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef 5446ee14100d 2024-04-25T01:32:32.365 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 smithi130 running (4m) 57s ago 4m 43.6M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 33660ba81c17 2024-04-25T01:32:32.365 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 smithi028 running (3m) 21s ago 3m 45.0M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef 7223c0482776 2024-04-25T01:32:32.365 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 smithi032 running (3m) 57s ago 3m 50.1M 4273M 19.0.0-2445-gd2c45d97 f68e37b807ef baf79060daa2 2024-04-25T01:32:32.365 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 smithi130 running (2m) 57s ago 2m 50.5M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 13318442a760 2024-04-25T01:32:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.389+0000 7ff874af4700 1 -- 172.21.15.28:0/2274755297 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff8580a2ca0 msgr2=0x7ff8580a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.389+0000 7ff874af4700 1 --2- 172.21.15.28:0/2274755297 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff8580a2ca0 0x7ff8580a5160 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff860005950 tx=0x7ff86000b410 comp rx=0 tx=0).stop 2024-04-25T01:32:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.389+0000 7ff874af4700 1 -- 172.21.15.28:0/2274755297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff870107f40 msgr2=0x7ff870101e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.389+0000 7ff874af4700 1 --2- 172.21.15.28:0/2274755297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff870107f40 0x7ff870101e40 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff86800ada0 tx=0x7ff86800c330 comp rx=0 tx=0).stop 2024-04-25T01:32:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.389+0000 7ff874af4700 1 -- 172.21.15.28:0/2274755297 shutdown_connections 2024-04-25T01:32:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.389+0000 7ff874af4700 1 --2- 172.21.15.28:0/2274755297 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff8580a2ca0 0x7ff8580a5160 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.389+0000 7ff874af4700 1 --2- 172.21.15.28:0/2274755297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff870107f40 0x7ff870101e40 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.389+0000 7ff874af4700 1 --2- 172.21.15.28:0/2274755297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff870100000 0x7ff8701032b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.389+0000 7ff874af4700 1 --2- 172.21.15.28:0/2274755297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff870100960 0x7ff870101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.389+0000 7ff874af4700 1 -- 172.21.15.28:0/2274755297 >> 172.21.15.28:0/2274755297 conn(0x7ff8700fba80 msgr2=0x7ff8700fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.389+0000 7ff874af4700 1 -- 172.21.15.28:0/2274755297 shutdown_connections 2024-04-25T01:32:32.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:32.389+0000 7ff874af4700 1 -- 172.21.15.28:0/2274755297 wait complete. 2024-04-25T01:32:32.384 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:32 smithi032 ceph-mon[32461]: pgmap v358: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-25T01:32:32.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:32 smithi130 ceph-mon[32702]: pgmap v358: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-25T01:32:33.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-25T01:32:33.352 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:32:34.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:34 smithi130 ceph-mon[32702]: pgmap v359: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-04-25T01:32:34.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:34 smithi130 ceph-mon[32702]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:32:34.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:34 smithi028 ceph-mon[29390]: pgmap v359: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-04-25T01:32:34.476 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:34 smithi028 ceph-mon[29390]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:32:34.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:34 smithi032 ceph-mon[32461]: pgmap v359: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-04-25T01:32:34.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:34 smithi032 ceph-mon[32461]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:32:35.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.094+0000 7f9879ce4700 1 -- 172.21.15.28:0/3429188176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9874103e60 msgr2=0x7f987410a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:35.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.094+0000 7f9879ce4700 1 --2- 172.21.15.28:0/3429188176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9874103e60 0x7f987410a710 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f986c00b0d0 tx=0x7f986c00b3e0 comp rx=0 tx=0).stop 2024-04-25T01:32:35.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.094+0000 7f9879ce4700 1 -- 172.21.15.28:0/3429188176 shutdown_connections 2024-04-25T01:32:35.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.094+0000 7f9879ce4700 1 --2- 172.21.15.28:0/3429188176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9874103e60 0x7f987410a710 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:35.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.094+0000 7f9879ce4700 1 --2- 172.21.15.28:0/3429188176 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98741022c0 0x7f98741026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:35.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.094+0000 7f9879ce4700 1 --2- 172.21.15.28:0/3429188176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98741034c0 0x7f9874103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:35.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.094+0000 7f9879ce4700 1 -- 172.21.15.28:0/3429188176 >> 172.21.15.28:0/3429188176 conn(0x7f98740fd880 msgr2=0x7f98740ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:35.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.094+0000 7f9879ce4700 1 -- 172.21.15.28:0/3429188176 shutdown_connections 2024-04-25T01:32:35.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.094+0000 7f9879ce4700 1 -- 172.21.15.28:0/3429188176 wait complete. 2024-04-25T01:32:35.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.094+0000 7f9879ce4700 1 Processor -- start 2024-04-25T01:32:35.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.095+0000 7f9879ce4700 1 -- start start 2024-04-25T01:32:35.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.095+0000 7f9879ce4700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98741022c0 0x7f98741983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:35.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.095+0000 7f9879ce4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98741034c0 0x7f98741988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:35.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.095+0000 7f9879ce4700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9874103e60 0x7f987419f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:35.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.095+0000 7f9879ce4700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f987410de80 con 0x7f98741034c0 2024-04-25T01:32:35.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.095+0000 7f9879ce4700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f987410dd00 con 0x7f98741022c0 2024-04-25T01:32:35.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.095+0000 7f9879ce4700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f987410e000 con 0x7f9874103e60 2024-04-25T01:32:35.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.095+0000 7f9872ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98741034c0 0x7f98741988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:35.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.095+0000 7f9872ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98741034c0 0x7f98741988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60790/0 (socket says 172.21.15.28:60790) 2024-04-25T01:32:35.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.095+0000 7f9872ffd700 1 -- 172.21.15.28:0/33079410 learned_addr learned my addr 172.21.15.28:0/33079410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.095+0000 7f9873fff700 1 --2- 172.21.15.28:0/33079410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9874103e60 0x7f987419f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.095+0000 7f98737fe700 1 --2- 172.21.15.28:0/33079410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98741022c0 0x7f98741983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.095+0000 7f9872ffd700 1 -- 172.21.15.28:0/33079410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9874103e60 msgr2=0x7f987419f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.095+0000 7f9872ffd700 1 --2- 172.21.15.28:0/33079410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9874103e60 0x7f987419f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.095+0000 7f9872ffd700 1 -- 172.21.15.28:0/33079410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98741022c0 msgr2=0x7f98741983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.095+0000 7f9872ffd700 1 --2- 172.21.15.28:0/33079410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98741022c0 0x7f98741983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.095+0000 7f9872ffd700 1 -- 172.21.15.28:0/33079410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98741a0000 con 0x7f98741034c0 2024-04-25T01:32:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.095+0000 7f98737fe700 1 --2- 172.21.15.28:0/33079410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98741022c0 0x7f98741983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.095+0000 7f9873fff700 1 --2- 172.21.15.28:0/33079410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9874103e60 0x7f987419f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.096+0000 7f9872ffd700 1 --2- 172.21.15.28:0/33079410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98741034c0 0x7f98741988e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f986800d980 tx=0x7f986800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.096+0000 7f9870ff9700 1 -- 172.21.15.28:0/33079410 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9868016070 con 0x7f98741034c0 2024-04-25T01:32:35.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.096+0000 7f9870ff9700 1 -- 172.21.15.28:0/33079410 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f986800de70 con 0x7f98741034c0 2024-04-25T01:32:35.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.096+0000 7f9879ce4700 1 -- 172.21.15.28:0/33079410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98741a02f0 con 0x7f98741034c0 2024-04-25T01:32:35.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.096+0000 7f9870ff9700 1 -- 172.21.15.28:0/33079410 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986800f3c0 con 0x7f98741034c0 2024-04-25T01:32:35.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.096+0000 7f9879ce4700 1 -- 172.21.15.28:0/33079410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98741a0880 con 0x7f98741034c0 2024-04-25T01:32:35.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.097+0000 7f9879ce4700 1 -- 172.21.15.28:0/33079410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9874068f20 con 0x7f98741034c0 2024-04-25T01:32:35.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.098+0000 7f9870ff9700 1 -- 172.21.15.28:0/33079410 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9868019070 con 0x7f98741034c0 2024-04-25T01:32:35.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.099+0000 7f9870ff9700 1 --2- 172.21.15.28:0/33079410 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f985c0a2ca0 0x7f985c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:35.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.099+0000 7f9870ff9700 1 -- 172.21.15.28:0/33079410 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f98680cee00 con 0x7f98741034c0 2024-04-25T01:32:35.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.099+0000 7f98737fe700 1 --2- 172.21.15.28:0/33079410 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f985c0a2ca0 0x7f985c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:35.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.099+0000 7f98737fe700 1 --2- 172.21.15.28:0/33079410 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f985c0a2ca0 0x7f985c0a5160 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9874103320 tx=0x7f9864009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:35.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.102+0000 7f9870ff9700 1 -- 172.21.15.28:0/33079410 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f986809ba00 con 0x7f98741034c0 2024-04-25T01:32:35.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.319+0000 7f9879ce4700 1 -- 172.21.15.28:0/33079410 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f9874062670 con 0x7f985c0a2ca0 2024-04-25T01:32:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.325+0000 7f9870ff9700 1 -- 172.21.15.28:0/33079410 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f9874062670 con 0x7f985c0a2ca0 2024-04-25T01:32:35.302 INFO:teuthology.orchestra.run.smithi028.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-25T01:32:35.302 INFO:teuthology.orchestra.run.smithi028.stdout:mds.all 12/12 60s ago 2m smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi032=b;smithi032=e;smithi032=h;smithi032=k;smithi130=c;smithi130=f;smithi130=i;smithi130=l;count:12 2024-04-25T01:32:35.302 INFO:teuthology.orchestra.run.smithi028.stdout:mgr 3/3 60s ago 8m smithi028=x;smithi032=y;smithi130=z;count:3 2024-04-25T01:32:35.302 INFO:teuthology.orchestra.run.smithi028.stdout:mon 3/3 60s ago 8m smithi028:172.21.15.28=a;smithi032:172.21.15.32=b;smithi130:172.21.15.130=c;count:3 2024-04-25T01:32:35.302 INFO:teuthology.orchestra.run.smithi028.stdout:osd 12 60s ago - 2024-04-25T01:32:35.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.327+0000 7f9879ce4700 1 -- 172.21.15.28:0/33079410 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f985c0a2ca0 msgr2=0x7f985c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:35.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.327+0000 7f9879ce4700 1 --2- 172.21.15.28:0/33079410 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f985c0a2ca0 0x7f985c0a5160 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9874103320 tx=0x7f9864009260 comp rx=0 tx=0).stop 2024-04-25T01:32:35.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.327+0000 7f9879ce4700 1 -- 172.21.15.28:0/33079410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98741034c0 msgr2=0x7f98741988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:35.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.327+0000 7f9879ce4700 1 --2- 172.21.15.28:0/33079410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98741034c0 0x7f98741988e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f986800d980 tx=0x7f986800dc90 comp rx=0 tx=0).stop 2024-04-25T01:32:35.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.327+0000 7f9879ce4700 1 -- 172.21.15.28:0/33079410 shutdown_connections 2024-04-25T01:32:35.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.327+0000 7f9879ce4700 1 --2- 172.21.15.28:0/33079410 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f985c0a2ca0 0x7f985c0a5160 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:35.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.327+0000 7f9879ce4700 1 --2- 172.21.15.28:0/33079410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9874103e60 0x7f987419f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:35.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.327+0000 7f9879ce4700 1 --2- 172.21.15.28:0/33079410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98741022c0 0x7f98741983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:35.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.327+0000 7f9879ce4700 1 --2- 172.21.15.28:0/33079410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98741034c0 0x7f98741988e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:35.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.327+0000 7f9879ce4700 1 -- 172.21.15.28:0/33079410 >> 172.21.15.28:0/33079410 conn(0x7f98740fd880 msgr2=0x7f98741066f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:35.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.328+0000 7f9879ce4700 1 -- 172.21.15.28:0/33079410 shutdown_connections 2024-04-25T01:32:35.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:35.328+0000 7f9879ce4700 1 -- 172.21.15.28:0/33079410 wait complete. 2024-04-25T01:32:35.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-25T01:32:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:32:36.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:36 smithi028 ceph-mon[29390]: pgmap v360: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-25T01:32:36.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:36 smithi130 ceph-mon[32702]: pgmap v360: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-25T01:32:36.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:36 smithi032 ceph-mon[32461]: pgmap v360: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-25T01:32:37.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:37 smithi028 ceph-mon[29390]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:32:37.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:37 smithi130 ceph-mon[32702]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:32:37.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:37 smithi032 ceph-mon[32461]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:32:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.039+0000 7f507706c700 1 -- 172.21.15.28:0/366328774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50701034b0 msgr2=0x7f5070103910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:38.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.039+0000 7f507706c700 1 --2- 172.21.15.28:0/366328774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50701034b0 0x7f5070103910 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f505c0099c0 tx=0x7f505c009cd0 comp rx=0 tx=0).stop 2024-04-25T01:32:38.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.039+0000 7f507706c700 1 -- 172.21.15.28:0/366328774 shutdown_connections 2024-04-25T01:32:38.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.039+0000 7f507706c700 1 --2- 172.21.15.28:0/366328774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5070103e50 0x7f507010a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:38.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.039+0000 7f507706c700 1 --2- 172.21.15.28:0/366328774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f50701022b0 0x7f50701026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:38.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.039+0000 7f507706c700 1 --2- 172.21.15.28:0/366328774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50701034b0 0x7f5070103910 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:38.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.039+0000 7f507706c700 1 -- 172.21.15.28:0/366328774 >> 172.21.15.28:0/366328774 conn(0x7f50700fd850 msgr2=0x7f50700ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:38.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.039+0000 7f507706c700 1 -- 172.21.15.28:0/366328774 shutdown_connections 2024-04-25T01:32:38.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.039+0000 7f507706c700 1 -- 172.21.15.28:0/366328774 wait complete. 2024-04-25T01:32:38.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.040+0000 7f507706c700 1 Processor -- start 2024-04-25T01:32:38.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.040+0000 7f507706c700 1 -- start start 2024-04-25T01:32:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.040+0000 7f507706c700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50701022b0 0x7f5070075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.040+0000 7f507706c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50701034b0 0x7f5070074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.040+0000 7f507706c700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5070103e50 0x7f5070074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.040+0000 7f507706c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f507010dea0 con 0x7f50701034b0 2024-04-25T01:32:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.040+0000 7f507706c700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f507010dd20 con 0x7f50701022b0 2024-04-25T01:32:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.040+0000 7f507706c700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f507010e020 con 0x7f5070103e50 2024-04-25T01:32:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.040+0000 7f506ffff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50701034b0 0x7f5070074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.040+0000 7f506ffff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50701034b0 0x7f5070074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38898/0 (socket says 172.21.15.28:38898) 2024-04-25T01:32:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.040+0000 7f506ffff700 1 -- 172.21.15.28:0/525555174 learned_addr learned my addr 172.21.15.28:0/525555174 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.040+0000 7f5074e08700 1 --2- 172.21.15.28:0/525555174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50701022b0 0x7f5070075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.040+0000 7f5075609700 1 --2- 172.21.15.28:0/525555174 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5070103e50 0x7f5070074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.041+0000 7f506ffff700 1 -- 172.21.15.28:0/525555174 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5070103e50 msgr2=0x7f5070074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.041+0000 7f506ffff700 1 --2- 172.21.15.28:0/525555174 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5070103e50 0x7f5070074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:38.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.041+0000 7f506ffff700 1 -- 172.21.15.28:0/525555174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50701022b0 msgr2=0x7f5070075fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:38.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.041+0000 7f506ffff700 1 --2- 172.21.15.28:0/525555174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50701022b0 0x7f5070075fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:38.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.041+0000 7f506ffff700 1 -- 172.21.15.28:0/525555174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5070075410 con 0x7f50701034b0 2024-04-25T01:32:38.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.041+0000 7f5074e08700 1 --2- 172.21.15.28:0/525555174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50701022b0 0x7f5070075fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:38.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.041+0000 7f506ffff700 1 --2- 172.21.15.28:0/525555174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50701034b0 0x7f5070074620 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f505c006010 tx=0x7f505c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:38.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.041+0000 7f506dffb700 1 -- 172.21.15.28:0/525555174 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505c027070 con 0x7f50701034b0 2024-04-25T01:32:38.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.041+0000 7f507706c700 1 -- 172.21.15.28:0/525555174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50701a2600 con 0x7f50701034b0 2024-04-25T01:32:38.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.041+0000 7f506dffb700 1 -- 172.21.15.28:0/525555174 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f505c004150 con 0x7f50701034b0 2024-04-25T01:32:38.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.041+0000 7f506dffb700 1 -- 172.21.15.28:0/525555174 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505c00f460 con 0x7f50701034b0 2024-04-25T01:32:38.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.041+0000 7f507706c700 1 -- 172.21.15.28:0/525555174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50701a2a50 con 0x7f50701034b0 2024-04-25T01:32:38.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.042+0000 7f507706c700 1 -- 172.21.15.28:0/525555174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5070068f20 con 0x7f50701034b0 2024-04-25T01:32:38.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.043+0000 7f506dffb700 1 -- 172.21.15.28:0/525555174 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f505c02a070 con 0x7f50701034b0 2024-04-25T01:32:38.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.044+0000 7f506dffb700 1 --2- 172.21.15.28:0/525555174 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f50600a2c20 0x7f50600a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:38.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.044+0000 7f5074e08700 1 --2- 172.21.15.28:0/525555174 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f50600a2c20 0x7f50600a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:38.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.044+0000 7f506dffb700 1 -- 172.21.15.28:0/525555174 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f505c0d0270 con 0x7f50701034b0 2024-04-25T01:32:38.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.045+0000 7f5074e08700 1 --2- 172.21.15.28:0/525555174 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f50600a2c20 0x7f50600a50e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5064005950 tx=0x7f50640058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:38.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.050+0000 7f506dffb700 1 -- 172.21.15.28:0/525555174 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f505c013070 con 0x7f50701034b0 2024-04-25T01:32:38.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.263+0000 7f507706c700 1 -- 172.21.15.28:0/525555174 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f5070062670 con 0x7f50600a2c20 2024-04-25T01:32:38.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.264+0000 7f506dffb700 1 -- 172.21.15.28:0/525555174 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f5070062670 con 0x7f50600a2c20 2024-04-25T01:32:38.241 INFO:teuthology.orchestra.run.smithi028.stdout:HOST ADDR LABELS STATUS 2024-04-25T01:32:38.241 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 172.21.15.28 2024-04-25T01:32:38.241 INFO:teuthology.orchestra.run.smithi028.stdout:smithi032 172.21.15.32 2024-04-25T01:32:38.241 INFO:teuthology.orchestra.run.smithi028.stdout:smithi130 172.21.15.130 2024-04-25T01:32:38.241 INFO:teuthology.orchestra.run.smithi028.stdout:3 hosts in cluster 2024-04-25T01:32:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.266+0000 7f507706c700 1 -- 172.21.15.28:0/525555174 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f50600a2c20 msgr2=0x7f50600a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.267+0000 7f507706c700 1 --2- 172.21.15.28:0/525555174 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f50600a2c20 0x7f50600a50e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5064005950 tx=0x7f50640058e0 comp rx=0 tx=0).stop 2024-04-25T01:32:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.267+0000 7f507706c700 1 -- 172.21.15.28:0/525555174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50701034b0 msgr2=0x7f5070074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.267+0000 7f507706c700 1 --2- 172.21.15.28:0/525555174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50701034b0 0x7f5070074620 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f505c006010 tx=0x7f505c0047b0 comp rx=0 tx=0).stop 2024-04-25T01:32:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.267+0000 7f507706c700 1 -- 172.21.15.28:0/525555174 shutdown_connections 2024-04-25T01:32:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.267+0000 7f507706c700 1 --2- 172.21.15.28:0/525555174 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f50600a2c20 0x7f50600a50e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.267+0000 7f507706c700 1 --2- 172.21.15.28:0/525555174 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5070103e50 0x7f5070074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.267+0000 7f507706c700 1 --2- 172.21.15.28:0/525555174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f50701022b0 0x7f5070075fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.267+0000 7f507706c700 1 --2- 172.21.15.28:0/525555174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50701034b0 0x7f5070074620 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.267+0000 7f507706c700 1 -- 172.21.15.28:0/525555174 >> 172.21.15.28:0/525555174 conn(0x7f50700fd850 msgr2=0x7f5070106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.267+0000 7f507706c700 1 -- 172.21.15.28:0/525555174 shutdown_connections 2024-04-25T01:32:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:38.267+0000 7f507706c700 1 -- 172.21.15.28:0/525555174 wait complete. 2024-04-25T01:32:38.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:38 smithi028 ceph-mon[29390]: pgmap v361: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:38.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:38 smithi130 ceph-mon[32702]: pgmap v361: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:38.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:38 smithi032 ceph-mon[32461]: pgmap v361: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:38.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-25T01:32:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:32:40.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:40 smithi028 ceph-mon[29390]: from='client.15351 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:32:40.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:40 smithi028 ceph-mon[29390]: pgmap v362: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:40.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:40 smithi130 ceph-mon[32702]: from='client.15351 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:32:40.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:40 smithi130 ceph-mon[32702]: pgmap v362: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:40.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:40 smithi032 ceph-mon[32461]: from='client.15351 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:32:40.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:40 smithi032 ceph-mon[32461]: pgmap v362: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:40.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.853+0000 7f87cba6f700 1 -- 172.21.15.28:0/100203850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87c4101190 msgr2=0x7f87c41015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:40.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.853+0000 7f87cba6f700 1 --2- 172.21.15.28:0/100203850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87c4101190 0x7f87c41015b0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f87b40099c0 tx=0x7f87b4009cd0 comp rx=0 tx=0).stop 2024-04-25T01:32:40.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.853+0000 7f87cba6f700 1 -- 172.21.15.28:0/100203850 shutdown_connections 2024-04-25T01:32:40.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.853+0000 7f87cba6f700 1 --2- 172.21.15.28:0/100203850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87c4102d30 0x7f87c41095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:40.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.853+0000 7f87cba6f700 1 --2- 172.21.15.28:0/100203850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87c4101190 0x7f87c41015b0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:40.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.853+0000 7f87cba6f700 1 --2- 172.21.15.28:0/100203850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87c4102390 0x7f87c41027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.853+0000 7f87cba6f700 1 -- 172.21.15.28:0/100203850 >> 172.21.15.28:0/100203850 conn(0x7f87c40fc750 msgr2=0x7f87c40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.853+0000 7f87cba6f700 1 -- 172.21.15.28:0/100203850 shutdown_connections 2024-04-25T01:32:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.853+0000 7f87cba6f700 1 -- 172.21.15.28:0/100203850 wait complete. 2024-04-25T01:32:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.854+0000 7f87cba6f700 1 Processor -- start 2024-04-25T01:32:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.854+0000 7f87cba6f700 1 -- start start 2024-04-25T01:32:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.854+0000 7f87cba6f700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87c4101190 0x7f87c410bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.854+0000 7f87cba6f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87c4102390 0x7f87c410a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.854+0000 7f87cba6f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87c4102d30 0x7f87c410a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.854+0000 7f87cba6f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f87c410def0 con 0x7f87c4102390 2024-04-25T01:32:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.854+0000 7f87cba6f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f87c410dd70 con 0x7f87c4102d30 2024-04-25T01:32:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.854+0000 7f87cba6f700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f87c410e070 con 0x7f87c4101190 2024-04-25T01:32:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.854+0000 7f87c900a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87c4102390 0x7f87c410a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.854+0000 7f87c900a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87c4102390 0x7f87c410a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38924/0 (socket says 172.21.15.28:38924) 2024-04-25T01:32:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.854+0000 7f87c900a700 1 -- 172.21.15.28:0/3358631969 learned_addr learned my addr 172.21.15.28:0/3358631969 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.854+0000 7f87c980b700 1 --2- 172.21.15.28:0/3358631969 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87c4101190 0x7f87c410bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.854+0000 7f87ca00c700 1 --2- 172.21.15.28:0/3358631969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87c4102d30 0x7f87c410a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.855+0000 7f87c900a700 1 -- 172.21.15.28:0/3358631969 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87c4101190 msgr2=0x7f87c410bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.855+0000 7f87c900a700 1 --2- 172.21.15.28:0/3358631969 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87c4101190 0x7f87c410bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.855+0000 7f87c900a700 1 -- 172.21.15.28:0/3358631969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87c4102d30 msgr2=0x7f87c410a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.855+0000 7f87c900a700 1 --2- 172.21.15.28:0/3358631969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87c4102d30 0x7f87c410a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.855+0000 7f87c900a700 1 -- 172.21.15.28:0/3358631969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87c410afa0 con 0x7f87c4102390 2024-04-25T01:32:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.855+0000 7f87c900a700 1 --2- 172.21.15.28:0/3358631969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87c4102390 0x7f87c410a1b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f87c000d980 tx=0x7f87c000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.855+0000 7f87baffd700 1 -- 172.21.15.28:0/3358631969 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c0017070 con 0x7f87c4102390 2024-04-25T01:32:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.855+0000 7f87baffd700 1 -- 172.21.15.28:0/3358631969 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87c0005020 con 0x7f87c4102390 2024-04-25T01:32:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.855+0000 7f87baffd700 1 -- 172.21.15.28:0/3358631969 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c000fbf0 con 0x7f87c4102390 2024-04-25T01:32:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.855+0000 7f87cba6f700 1 -- 172.21.15.28:0/3358631969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87c41a26c0 con 0x7f87c4102390 2024-04-25T01:32:40.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.855+0000 7f87cba6f700 1 -- 172.21.15.28:0/3358631969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87c41a2c00 con 0x7f87c4102390 2024-04-25T01:32:40.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.856+0000 7f87cba6f700 1 -- 172.21.15.28:0/3358631969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87c4068f20 con 0x7f87c4102390 2024-04-25T01:32:40.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.857+0000 7f87baffd700 1 -- 172.21.15.28:0/3358631969 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f87c00051d0 con 0x7f87c4102390 2024-04-25T01:32:40.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.858+0000 7f87baffd700 1 --2- 172.21.15.28:0/3358631969 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f87b00a2c20 0x7f87b00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:40.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.858+0000 7f87c980b700 1 --2- 172.21.15.28:0/3358631969 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f87b00a2c20 0x7f87b00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:40.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.858+0000 7f87baffd700 1 -- 172.21.15.28:0/3358631969 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f87c00cf610 con 0x7f87c4102390 2024-04-25T01:32:40.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.859+0000 7f87c980b700 1 --2- 172.21.15.28:0/3358631969 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f87b00a2c20 0x7f87b00a50e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f87b400b580 tx=0x7f87b4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:40.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:40.862+0000 7f87baffd700 1 -- 172.21.15.28:0/3358631969 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f87c0020020 con 0x7f87c4102390 2024-04-25T01:32:41.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:41.079+0000 7f87cba6f700 1 -- 172.21.15.28:0/3358631969 --> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f87c4062670 con 0x7f87b00a2c20 2024-04-25T01:32:41.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:41.081+0000 7f87baffd700 1 -- 172.21.15.28:0/3358631969 <== mgr.14152 v2:172.21.15.28:6800/1149553567 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f87c4062670 con 0x7f87b00a2c20 2024-04-25T01:32:41.058 INFO:teuthology.orchestra.run.smithi028.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-25T01:32:41.058 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747401F4480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-25T01:32:41.058 INFO:teuthology.orchestra.run.smithi028.stdout:smithi032 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747402DN480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-25T01:32:41.059 INFO:teuthology.orchestra.run.smithi028.stdout:smithi130 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500Q1400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-25T01:32:41.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:41.083+0000 7f87cba6f700 1 -- 172.21.15.28:0/3358631969 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f87b00a2c20 msgr2=0x7f87b00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:41.083+0000 7f87cba6f700 1 --2- 172.21.15.28:0/3358631969 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f87b00a2c20 0x7f87b00a50e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f87b400b580 tx=0x7f87b4005fb0 comp rx=0 tx=0).stop 2024-04-25T01:32:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:41.083+0000 7f87cba6f700 1 -- 172.21.15.28:0/3358631969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87c4102390 msgr2=0x7f87c410a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:41.083+0000 7f87cba6f700 1 --2- 172.21.15.28:0/3358631969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87c4102390 0x7f87c410a1b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f87c000d980 tx=0x7f87c000dc90 comp rx=0 tx=0).stop 2024-04-25T01:32:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:41.083+0000 7f87cba6f700 1 -- 172.21.15.28:0/3358631969 shutdown_connections 2024-04-25T01:32:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:41.083+0000 7f87cba6f700 1 --2- 172.21.15.28:0/3358631969 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f87b00a2c20 0x7f87b00a50e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:41.083+0000 7f87cba6f700 1 --2- 172.21.15.28:0/3358631969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87c4102d30 0x7f87c410a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:41.083+0000 7f87cba6f700 1 --2- 172.21.15.28:0/3358631969 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f87c4101190 0x7f87c410bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:41.083+0000 7f87cba6f700 1 --2- 172.21.15.28:0/3358631969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87c4102390 0x7f87c410a1b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:41.083+0000 7f87cba6f700 1 -- 172.21.15.28:0/3358631969 >> 172.21.15.28:0/3358631969 conn(0x7f87c40fc750 msgr2=0x7f87c4105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:41.083+0000 7f87cba6f700 1 -- 172.21.15.28:0/3358631969 shutdown_connections 2024-04-25T01:32:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:41.083+0000 7f87cba6f700 1 -- 172.21.15.28:0/3358631969 wait complete. 2024-04-25T01:32:41.734 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T01:32:41.742 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T01:32:41.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-25T01:32:42.003 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:32:42.358 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:42 smithi028 ceph-mon[29390]: pgmap v363: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:42.359 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:42 smithi028 ceph-mon[29390]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:32:42.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:42 smithi130 ceph-mon[32702]: pgmap v363: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:42.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:42 smithi130 ceph-mon[32702]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:32:42.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:42 smithi032 ceph-mon[32461]: pgmap v363: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:42.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:42 smithi032 ceph-mon[32461]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:32:43.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.762+0000 7f8db9733700 1 -- 172.21.15.28:0/304911111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db4102d50 msgr2=0x7f8db4109600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:43.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.762+0000 7f8db9733700 1 --2- 172.21.15.28:0/304911111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db4102d50 0x7f8db4109600 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8dac00b0d0 tx=0x7f8dac00b3e0 comp rx=0 tx=0).stop 2024-04-25T01:32:43.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.762+0000 7f8db9733700 1 -- 172.21.15.28:0/304911111 shutdown_connections 2024-04-25T01:32:43.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.762+0000 7f8db9733700 1 --2- 172.21.15.28:0/304911111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db4102d50 0x7f8db4109600 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:43.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.762+0000 7f8db9733700 1 --2- 172.21.15.28:0/304911111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8db41011b0 0x7f8db41015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:43.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.762+0000 7f8db9733700 1 --2- 172.21.15.28:0/304911111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8db41023b0 0x7f8db4102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:43.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.762+0000 7f8db9733700 1 -- 172.21.15.28:0/304911111 >> 172.21.15.28:0/304911111 conn(0x7f8db40fc750 msgr2=0x7f8db40feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:43.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.762+0000 7f8db9733700 1 -- 172.21.15.28:0/304911111 shutdown_connections 2024-04-25T01:32:43.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.762+0000 7f8db9733700 1 -- 172.21.15.28:0/304911111 wait complete. 2024-04-25T01:32:43.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.763+0000 7f8db9733700 1 Processor -- start 2024-04-25T01:32:43.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.763+0000 7f8db9733700 1 -- start start 2024-04-25T01:32:43.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.763+0000 7f8db9733700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db41011b0 0x7f8db4198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:43.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.763+0000 7f8db9733700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8db41023b0 0x7f8db4198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:43.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.763+0000 7f8db9733700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8db4102d50 0x7f8db419f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:43.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.763+0000 7f8db9733700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8db410cda0 con 0x7f8db41011b0 2024-04-25T01:32:43.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.763+0000 7f8db9733700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8db410cc20 con 0x7f8db41023b0 2024-04-25T01:32:43.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.763+0000 7f8db9733700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8db410cf20 con 0x7f8db4102d50 2024-04-25T01:32:43.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.764+0000 7f8db2ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db41011b0 0x7f8db4198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:43.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.764+0000 7f8db2ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db41011b0 0x7f8db4198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38934/0 (socket says 172.21.15.28:38934) 2024-04-25T01:32:43.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.764+0000 7f8db2ffd700 1 -- 172.21.15.28:0/3676378884 learned_addr learned my addr 172.21.15.28:0/3676378884 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:43.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.764+0000 7f8db27fc700 1 --2- 172.21.15.28:0/3676378884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8db41023b0 0x7f8db4198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:43.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.764+0000 7f8db37fe700 1 --2- 172.21.15.28:0/3676378884 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8db4102d50 0x7f8db419f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:43.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.764+0000 7f8db2ffd700 1 -- 172.21.15.28:0/3676378884 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8db4102d50 msgr2=0x7f8db419f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:43.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.764+0000 7f8db2ffd700 1 --2- 172.21.15.28:0/3676378884 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8db4102d50 0x7f8db419f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:43.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.764+0000 7f8db2ffd700 1 -- 172.21.15.28:0/3676378884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8db41023b0 msgr2=0x7f8db4198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:43.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.764+0000 7f8db2ffd700 1 --2- 172.21.15.28:0/3676378884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8db41023b0 0x7f8db4198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:43.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.764+0000 7f8db2ffd700 1 -- 172.21.15.28:0/3676378884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db41a0060 con 0x7f8db41011b0 2024-04-25T01:32:43.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.764+0000 7f8db2ffd700 1 --2- 172.21.15.28:0/3676378884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db41011b0 0x7f8db4198450 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8da400c920 tx=0x7f8da400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:43.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.764+0000 7f8d9bfff700 1 -- 172.21.15.28:0/3676378884 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da4004220 con 0x7f8db41011b0 2024-04-25T01:32:43.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.764+0000 7f8db9733700 1 -- 172.21.15.28:0/3676378884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8db41a0350 con 0x7f8db41011b0 2024-04-25T01:32:43.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.764+0000 7f8d9bfff700 1 -- 172.21.15.28:0/3676378884 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8da40043c0 con 0x7f8db41011b0 2024-04-25T01:32:43.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.764+0000 7f8d9bfff700 1 -- 172.21.15.28:0/3676378884 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da4003cd0 con 0x7f8db41011b0 2024-04-25T01:32:43.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.765+0000 7f8db9733700 1 -- 172.21.15.28:0/3676378884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8db41a0890 con 0x7f8db41011b0 2024-04-25T01:32:43.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.765+0000 7f8db9733700 1 -- 172.21.15.28:0/3676378884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8db4068f20 con 0x7f8db41011b0 2024-04-25T01:32:43.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.767+0000 7f8d9bfff700 1 -- 172.21.15.28:0/3676378884 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8da40076d0 con 0x7f8db41011b0 2024-04-25T01:32:43.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.767+0000 7f8d9bfff700 1 --2- 172.21.15.28:0/3676378884 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8d9c0a2c20 0x7f8d9c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:43.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.767+0000 7f8db27fc700 1 --2- 172.21.15.28:0/3676378884 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8d9c0a2c20 0x7f8d9c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:43.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.767+0000 7f8d9bfff700 1 -- 172.21.15.28:0/3676378884 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f8da40ceef0 con 0x7f8db41011b0 2024-04-25T01:32:43.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.768+0000 7f8db27fc700 1 --2- 172.21.15.28:0/3676378884 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8d9c0a2c20 0x7f8d9c0a50e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8da8005950 tx=0x7f8da800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:43.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:43.772+0000 7f8d9bfff700 1 -- 172.21.15.28:0/3676378884 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8da4014030 con 0x7f8db41011b0 2024-04-25T01:32:44.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:44.022+0000 7f8db9733700 1 -- 172.21.15.28:0/3676378884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f8db41a1290 con 0x7f8db41011b0 2024-04-25T01:32:44.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:44.023+0000 7f8d9bfff700 1 -- 172.21.15.28:0/3676378884 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) ==== 76+0+3086 (secure 0 0 0) 0x7f8da409baf0 con 0x7f8db41011b0 2024-04-25T01:32:44.001 INFO:teuthology.orchestra.run.smithi028.stdout:e27 2024-04-25T01:32:44.002 INFO:teuthology.orchestra.run.smithi028.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T01:32:44.002 INFO:teuthology.orchestra.run.smithi028.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:32:44.002 INFO:teuthology.orchestra.run.smithi028.stdout:legacy client fscid: 1 2024-04-25T01:32:44.002 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:32:44.003 INFO:teuthology.orchestra.run.smithi028.stdout:Filesystem 'cephfs' (1) 2024-04-25T01:32:44.003 INFO:teuthology.orchestra.run.smithi028.stdout:fs_name cephfs 2024-04-25T01:32:44.003 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 27 2024-04-25T01:32:44.003 INFO:teuthology.orchestra.run.smithi028.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-25T01:32:44.003 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-04-25T01:30:33.696870+0000 2024-04-25T01:32:44.003 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-04-25T01:31:01.802718+0000 2024-04-25T01:32:44.004 INFO:teuthology.orchestra.run.smithi028.stdout:tableserver 0 2024-04-25T01:32:44.004 INFO:teuthology.orchestra.run.smithi028.stdout:root 0 2024-04-25T01:32:44.004 INFO:teuthology.orchestra.run.smithi028.stdout:session_timeout 300 2024-04-25T01:32:44.004 INFO:teuthology.orchestra.run.smithi028.stdout:session_autoclose 300 2024-04-25T01:32:44.005 INFO:teuthology.orchestra.run.smithi028.stdout:max_file_size 1099511627776 2024-04-25T01:32:44.005 INFO:teuthology.orchestra.run.smithi028.stdout:max_xattr_size 65536 2024-04-25T01:32:44.005 INFO:teuthology.orchestra.run.smithi028.stdout:required_client_features {} 2024-04-25T01:32:44.005 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure 0 2024-04-25T01:32:44.005 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure_osd_epoch 0 2024-04-25T01:32:44.005 INFO:teuthology.orchestra.run.smithi028.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:32:44.006 INFO:teuthology.orchestra.run.smithi028.stdout:max_mds 5 2024-04-25T01:32:44.006 INFO:teuthology.orchestra.run.smithi028.stdout:in 0,1,2,3,4 2024-04-25T01:32:44.006 INFO:teuthology.orchestra.run.smithi028.stdout:up {0=24512,1=24478,2=24473,3=24454,4=24442} 2024-04-25T01:32:44.007 INFO:teuthology.orchestra.run.smithi028.stdout:failed 2024-04-25T01:32:44.007 INFO:teuthology.orchestra.run.smithi028.stdout:damaged 2024-04-25T01:32:44.007 INFO:teuthology.orchestra.run.smithi028.stdout:stopped 2024-04-25T01:32:44.007 INFO:teuthology.orchestra.run.smithi028.stdout:data_pools [3,4] 2024-04-25T01:32:44.008 INFO:teuthology.orchestra.run.smithi028.stdout:metadata_pool 2 2024-04-25T01:32:44.008 INFO:teuthology.orchestra.run.smithi028.stdout:inline_data disabled 2024-04-25T01:32:44.008 INFO:teuthology.orchestra.run.smithi028.stdout:balancer 2024-04-25T01:32:44.008 INFO:teuthology.orchestra.run.smithi028.stdout:bal_rank_mask -1 2024-04-25T01:32:44.008 INFO:teuthology.orchestra.run.smithi028.stdout:standby_count_wanted 0 2024-04-25T01:32:44.008 INFO:teuthology.orchestra.run.smithi028.stdout:qdb_cluster leader: 24512 members: 24454,24442,24512,24478,24473 2024-04-25T01:32:44.008 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b{0:24512} state up:active seq 3 addr [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:32:44.008 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.e{0:24440} state up:standby-replay seq 1 addr [v2:172.21.15.32:6834/356243399,v1:172.21.15.32:6835/356243399] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:32:44.008 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.i{1:24478} state up:active seq 11 addr [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:32:44.008 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.l{1:24412} state up:standby-replay seq 1 addr [v2:172.21.15.130:6832/1080169582,v1:172.21.15.130:6833/1080169582] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:32:44.009 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.h{2:24473} state up:active seq 12 addr [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:32:44.009 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.k{2:24401} state up:standby-replay seq 1 addr [v2:172.21.15.32:6832/983600975,v1:172.21.15.32:6833/983600975] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:32:44.009 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c{3:24454} state up:active seq 13 addr [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:32:44.009 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a{3:14655} state up:standby-replay seq 1 addr [v2:172.21.15.28:6840/3666045393,v1:172.21.15.28:6841/3666045393] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:32:44.009 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.f{4:24442} state up:active seq 14 addr [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:32:44.009 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.g{4:14643} state up:standby-replay seq 1 addr [v2:172.21.15.28:6838/3206225061,v1:172.21.15.28:6839/3206225061] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:32:44.009 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:32:44.009 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:32:44.010 INFO:teuthology.orchestra.run.smithi028.stdout:Standby daemons: 2024-04-25T01:32:44.010 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:32:44.010 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.j{-1:14565} state up:standby seq 1 addr [v2:172.21.15.28:6834/2774244884,v1:172.21.15.28:6835/2774244884] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:32:44.010 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d{-1:14583} state up:standby seq 1 addr [v2:172.21.15.28:6836/2622429747,v1:172.21.15.28:6837/2622429747] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:32:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:44.025+0000 7f8db9733700 1 -- 172.21.15.28:0/3676378884 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8d9c0a2c20 msgr2=0x7f8d9c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:44.025+0000 7f8db9733700 1 --2- 172.21.15.28:0/3676378884 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8d9c0a2c20 0x7f8d9c0a50e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8da8005950 tx=0x7f8da800b410 comp rx=0 tx=0).stop 2024-04-25T01:32:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:44.025+0000 7f8db9733700 1 -- 172.21.15.28:0/3676378884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db41011b0 msgr2=0x7f8db4198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:44.025+0000 7f8db9733700 1 --2- 172.21.15.28:0/3676378884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db41011b0 0x7f8db4198450 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8da400c920 tx=0x7f8da400cc30 comp rx=0 tx=0).stop 2024-04-25T01:32:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:44.025+0000 7f8db9733700 1 -- 172.21.15.28:0/3676378884 shutdown_connections 2024-04-25T01:32:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:44.025+0000 7f8db9733700 1 --2- 172.21.15.28:0/3676378884 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f8d9c0a2c20 0x7f8d9c0a50e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:44.025+0000 7f8db9733700 1 --2- 172.21.15.28:0/3676378884 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8db4102d50 0x7f8db419f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:44.025+0000 7f8db9733700 1 --2- 172.21.15.28:0/3676378884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db41011b0 0x7f8db4198450 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:44.025+0000 7f8db9733700 1 --2- 172.21.15.28:0/3676378884 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8db41023b0 0x7f8db4198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:44.025+0000 7f8db9733700 1 -- 172.21.15.28:0/3676378884 >> 172.21.15.28:0/3676378884 conn(0x7f8db40fc750 msgr2=0x7f8db406a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:44.025+0000 7f8db9733700 1 -- 172.21.15.28:0/3676378884 shutdown_connections 2024-04-25T01:32:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:44.025+0000 7f8db9733700 1 -- 172.21.15.28:0/3676378884 wait complete. 2024-04-25T01:32:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-04-25T01:32:44.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:44 smithi028 ceph-mon[29390]: pgmap v364: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:32:44.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:44 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3676378884' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T01:32:44.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:44 smithi130 ceph-mon[32702]: pgmap v364: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:32:44.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:44 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3676378884' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T01:32:44.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:44 smithi032 ceph-mon[32461]: pgmap v364: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:32:44.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:44 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3676378884' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T01:32:44.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-25T01:32:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:32:46.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:46 smithi028 ceph-mon[29390]: pgmap v365: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:46.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:46 smithi130 ceph-mon[32702]: pgmap v365: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:46.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:46 smithi032 ceph-mon[32461]: pgmap v365: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.833+0000 7fbe0aa84700 1 -- 172.21.15.28:0/569665074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe04102d30 msgr2=0x7fbe041095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.833+0000 7fbe0aa84700 1 --2- 172.21.15.28:0/569665074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe04102d30 0x7fbe041095e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fbdf40099c0 tx=0x7fbdf4009cd0 comp rx=0 tx=0).stop 2024-04-25T01:32:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.833+0000 7fbe0aa84700 1 -- 172.21.15.28:0/569665074 shutdown_connections 2024-04-25T01:32:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.833+0000 7fbe0aa84700 1 --2- 172.21.15.28:0/569665074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe04102d30 0x7fbe041095e0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.833+0000 7fbe0aa84700 1 --2- 172.21.15.28:0/569665074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe04101190 0x7fbe041015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.833+0000 7fbe0aa84700 1 --2- 172.21.15.28:0/569665074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe04102390 0x7fbe041027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.833+0000 7fbe0aa84700 1 -- 172.21.15.28:0/569665074 >> 172.21.15.28:0/569665074 conn(0x7fbe040fc730 msgr2=0x7fbe040feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.833+0000 7fbe0aa84700 1 -- 172.21.15.28:0/569665074 shutdown_connections 2024-04-25T01:32:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.833+0000 7fbe0aa84700 1 -- 172.21.15.28:0/569665074 wait complete. 2024-04-25T01:32:46.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.834+0000 7fbe0aa84700 1 Processor -- start 2024-04-25T01:32:46.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.834+0000 7fbe0aa84700 1 -- start start 2024-04-25T01:32:46.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.834+0000 7fbe0aa84700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe04101190 0x7fbe04198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:46.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.834+0000 7fbe0aa84700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe04102390 0x7fbe041989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:46.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.834+0000 7fbe0aa84700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe04102d30 0x7fbe0419fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:46.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.834+0000 7fbe0aa84700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0410cd80 con 0x7fbe04101190 2024-04-25T01:32:46.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.834+0000 7fbe0aa84700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0410cc00 con 0x7fbe04102390 2024-04-25T01:32:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.834+0000 7fbe0aa84700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0410cf00 con 0x7fbe04102d30 2024-04-25T01:32:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.834+0000 7fbe08820700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe04101190 0x7fbe04198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.834+0000 7fbe08820700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe04101190 0x7fbe04198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40852/0 (socket says 172.21.15.28:40852) 2024-04-25T01:32:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.834+0000 7fbe08820700 1 -- 172.21.15.28:0/58888567 learned_addr learned my addr 172.21.15.28:0/58888567 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.834+0000 7fbe03fff700 1 --2- 172.21.15.28:0/58888567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe04102390 0x7fbe041989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.834+0000 7fbe09021700 1 --2- 172.21.15.28:0/58888567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe04102d30 0x7fbe0419fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.835+0000 7fbe08820700 1 -- 172.21.15.28:0/58888567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe04102d30 msgr2=0x7fbe0419fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.835+0000 7fbe08820700 1 --2- 172.21.15.28:0/58888567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe04102d30 0x7fbe0419fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.835+0000 7fbe08820700 1 -- 172.21.15.28:0/58888567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe04102390 msgr2=0x7fbe041989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.835+0000 7fbe08820700 1 --2- 172.21.15.28:0/58888567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe04102390 0x7fbe041989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.835+0000 7fbe08820700 1 -- 172.21.15.28:0/58888567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe041a00c0 con 0x7fbe04101190 2024-04-25T01:32:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.835+0000 7fbe09021700 1 --2- 172.21.15.28:0/58888567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe04102d30 0x7fbe0419fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.835+0000 7fbe08820700 1 --2- 172.21.15.28:0/58888567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe04101190 0x7fbe04198460 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc009f20 tx=0x7fbdfc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.835+0000 7fbe01ffb700 1 -- 172.21.15.28:0/58888567 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc003a00 con 0x7fbe04101190 2024-04-25T01:32:46.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.835+0000 7fbe0aa84700 1 -- 172.21.15.28:0/58888567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe041a03b0 con 0x7fbe04101190 2024-04-25T01:32:46.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.835+0000 7fbe01ffb700 1 -- 172.21.15.28:0/58888567 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdfc003ba0 con 0x7fbe04101190 2024-04-25T01:32:46.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.835+0000 7fbe01ffb700 1 -- 172.21.15.28:0/58888567 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc007500 con 0x7fbe04101190 2024-04-25T01:32:46.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.835+0000 7fbe0aa84700 1 -- 172.21.15.28:0/58888567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe041a08f0 con 0x7fbe04101190 2024-04-25T01:32:46.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.836+0000 7fbe0aa84700 1 -- 172.21.15.28:0/58888567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe04068f20 con 0x7fbe04101190 2024-04-25T01:32:46.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.837+0000 7fbe01ffb700 1 -- 172.21.15.28:0/58888567 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fbdfc012050 con 0x7fbe04101190 2024-04-25T01:32:46.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.838+0000 7fbe01ffb700 1 --2- 172.21.15.28:0/58888567 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbdec0a2c20 0x7fbdec0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:46.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.838+0000 7fbe03fff700 1 --2- 172.21.15.28:0/58888567 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbdec0a2c20 0x7fbdec0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:46.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.838+0000 7fbe01ffb700 1 -- 172.21.15.28:0/58888567 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fbdfc0cf200 con 0x7fbe04101190 2024-04-25T01:32:46.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.839+0000 7fbe03fff700 1 --2- 172.21.15.28:0/58888567 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbdec0a2c20 0x7fbdec0a50e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbdf800ac50 tx=0x7fbdf800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:46.843+0000 7fbe01ffb700 1 -- 172.21.15.28:0/58888567 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbdfc010040 con 0x7fbe04101190 2024-04-25T01:32:47.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.051+0000 7fbe0aa84700 1 -- 172.21.15.28:0/58888567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fbe041a1290 con 0x7fbe04101190 2024-04-25T01:32:47.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.053+0000 7fbe01ffb700 1 -- 172.21.15.28:0/58888567 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5591 (secure 0 0 0) 0x7fbdfc09be00 con 0x7fbe04101190 2024-04-25T01:32:47.030 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 93 2024-04-25T01:32:47.030 INFO:teuthology.orchestra.run.smithi028.stdout:fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:32:47.031 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-04-25T01:22:26.299667+0000 2024-04-25T01:32:47.031 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-04-25T01:31:21.857121+0000 2024-04-25T01:32:47.031 INFO:teuthology.orchestra.run.smithi028.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-25T01:32:47.031 INFO:teuthology.orchestra.run.smithi028.stdout:crush_version 27 2024-04-25T01:32:47.031 INFO:teuthology.orchestra.run.smithi028.stdout:full_ratio 0.95 2024-04-25T01:32:47.031 INFO:teuthology.orchestra.run.smithi028.stdout:backfillfull_ratio 0.9 2024-04-25T01:32:47.032 INFO:teuthology.orchestra.run.smithi028.stdout:nearfull_ratio 0.85 2024-04-25T01:32:47.032 INFO:teuthology.orchestra.run.smithi028.stdout:require_min_compat_client luminous 2024-04-25T01:32:47.032 INFO:teuthology.orchestra.run.smithi028.stdout:min_compat_client luminous 2024-04-25T01:32:47.032 INFO:teuthology.orchestra.run.smithi028.stdout:require_osd_release squid 2024-04-25T01:32:47.032 INFO:teuthology.orchestra.run.smithi028.stdout:stretch_mode_enabled false 2024-04-25T01:32:47.032 INFO:teuthology.orchestra.run.smithi028.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-25T01:32:47.033 INFO:teuthology.orchestra.run.smithi028.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-04-25T01:32:47.033 INFO:teuthology.orchestra.run.smithi028.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 87 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-25T01:32:47.033 INFO:teuthology.orchestra.run.smithi028.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 91 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-25T01:32:47.033 INFO:teuthology.orchestra.run.smithi028.stdout:max_osd 12 2024-04-25T01:32:47.033 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 up in weight 1 up_from 8 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6802/2903822772,v1:172.21.15.28:6803/2903822772] [v2:172.21.15.28:6804/2903822772,v1:172.21.15.28:6805/2903822772] exists,up a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b 2024-04-25T01:32:47.034 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 up in weight 1 up_from 13 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6800/2516196299,v1:172.21.15.32:6801/2516196299] [v2:172.21.15.32:6802/2516196299,v1:172.21.15.32:6803/2516196299] exists,up 0a457e4c-75ec-4b92-9aaa-f66036d4b492 2024-04-25T01:32:47.034 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 up in weight 1 up_from 18 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6800/3497329111,v1:172.21.15.130:6801/3497329111] [v2:172.21.15.130:6802/3497329111,v1:172.21.15.130:6803/3497329111] exists,up 2b6e267e-fe68-4971-8f49-3e51b82d1e40 2024-04-25T01:32:47.034 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 up in weight 1 up_from 26 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6810/4005405927,v1:172.21.15.28:6811/4005405927] [v2:172.21.15.28:6812/4005405927,v1:172.21.15.28:6813/4005405927] exists,up fd309ba8-012b-4058-bcc0-57f61f41c810 2024-04-25T01:32:47.034 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 up in weight 1 up_from 34 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6808/3679896058,v1:172.21.15.32:6809/3679896058] [v2:172.21.15.32:6810/3679896058,v1:172.21.15.32:6811/3679896058] exists,up 80f71a46-76da-45eb-b59b-6af112d2e0aa 2024-04-25T01:32:47.034 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 up in weight 1 up_from 40 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6808/4253601662,v1:172.21.15.130:6809/4253601662] [v2:172.21.15.130:6810/4253601662,v1:172.21.15.130:6811/4253601662] exists,up 0394f4a0-297e-470a-b491-6da6616a7ed8 2024-04-25T01:32:47.034 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 up in weight 1 up_from 48 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6818/2330487523,v1:172.21.15.28:6819/2330487523] [v2:172.21.15.28:6820/2330487523,v1:172.21.15.28:6821/2330487523] exists,up 84cb9069-9d19-4d0e-8db4-cb0d41d7e48a 2024-04-25T01:32:47.035 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 up in weight 1 up_from 53 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6816/1373301181,v1:172.21.15.32:6817/1373301181] [v2:172.21.15.32:6818/1373301181,v1:172.21.15.32:6819/1373301181] exists,up ea0a98a6-80dc-4c7d-b335-b22f9326de2d 2024-04-25T01:32:47.035 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 up in weight 1 up_from 61 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6816/2911117678,v1:172.21.15.130:6817/2911117678] [v2:172.21.15.130:6818/2911117678,v1:172.21.15.130:6819/2911117678] exists,up d5beef33-36c9-4759-a168-c104ee1b5f75 2024-04-25T01:32:47.035 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 up in weight 1 up_from 69 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6826/2946375206,v1:172.21.15.28:6827/2946375206] [v2:172.21.15.28:6828/2946375206,v1:172.21.15.28:6829/2946375206] exists,up 18835e9d-8405-4586-89b2-75d3e565fad9 2024-04-25T01:32:47.035 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 up in weight 1 up_from 74 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6824/13541811,v1:172.21.15.32:6825/13541811] [v2:172.21.15.32:6826/13541811,v1:172.21.15.32:6827/13541811] exists,up 22b3f803-fccb-43d3-941e-b7cfd9b11461 2024-04-25T01:32:47.035 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 up in weight 1 up_from 82 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.130:6824/3995209547,v1:172.21.15.130:6825/3995209547] [v2:172.21.15.130:6826/3995209547,v1:172.21.15.130:6827/3995209547] exists,up 3c129f89-41b2-4ba9-8e22-0863133613c2 2024-04-25T01:32:47.035 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.a [2,11] 2024-04-25T01:32:47.036 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.13 [2,11] 2024-04-25T01:32:47.036 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.1 [4,9] 2024-04-25T01:32:47.036 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.2 [3,9] 2024-04-25T01:32:47.036 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.1b [5,9] 2024-04-25T01:32:47.036 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.23 [4,8] 2024-04-25T01:32:47.036 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.f [5,8] 2024-04-25T01:32:47.037 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.13 [9,8] 2024-04-25T01:32:47.037 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.1b [9,8] 2024-04-25T01:32:47.037 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.1f [9,10] 2024-04-25T01:32:47.037 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/2407469300 expires 2024-04-26T01:23:20.244799+0000 2024-04-25T01:32:47.037 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/4112717790 expires 2024-04-26T01:23:20.244799+0000 2024-04-25T01:32:47.037 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/2407469300 expires 2024-04-26T01:23:20.244799+0000 2024-04-25T01:32:47.038 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2222781993 expires 2024-04-26T01:23:20.244799+0000 2024-04-25T01:32:47.038 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/3973667708 expires 2024-04-26T01:22:53.256620+0000 2024-04-25T01:32:47.038 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2339219943 expires 2024-04-26T01:22:53.256620+0000 2024-04-25T01:32:47.039 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/3973667708 expires 2024-04-26T01:22:53.256620+0000 2024-04-25T01:32:47.039 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/747955335 expires 2024-04-26T01:22:53.256620+0000 2024-04-25T01:32:47.039 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/4131336597 expires 2024-04-26T01:22:53.256620+0000 2024-04-25T01:32:47.039 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/3232269852 expires 2024-04-26T01:23:20.244799+0000 2024-04-25T01:32:47.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.055+0000 7fbe0aa84700 1 -- 172.21.15.28:0/58888567 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbdec0a2c20 msgr2=0x7fbdec0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:47.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.055+0000 7fbe0aa84700 1 --2- 172.21.15.28:0/58888567 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbdec0a2c20 0x7fbdec0a50e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbdf800ac50 tx=0x7fbdf800a380 comp rx=0 tx=0).stop 2024-04-25T01:32:47.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.055+0000 7fbe0aa84700 1 -- 172.21.15.28:0/58888567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe04101190 msgr2=0x7fbe04198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:47.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.055+0000 7fbe0aa84700 1 --2- 172.21.15.28:0/58888567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe04101190 0x7fbe04198460 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc009f20 tx=0x7fbdfc00c3a0 comp rx=0 tx=0).stop 2024-04-25T01:32:47.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.055+0000 7fbe0aa84700 1 -- 172.21.15.28:0/58888567 shutdown_connections 2024-04-25T01:32:47.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.055+0000 7fbe0aa84700 1 --2- 172.21.15.28:0/58888567 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fbdec0a2c20 0x7fbdec0a50e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:47.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.055+0000 7fbe0aa84700 1 --2- 172.21.15.28:0/58888567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbe04102d30 0x7fbe0419fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:47.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.055+0000 7fbe0aa84700 1 --2- 172.21.15.28:0/58888567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe04101190 0x7fbe04198460 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:47.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.055+0000 7fbe0aa84700 1 --2- 172.21.15.28:0/58888567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe04102390 0x7fbe041989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:47.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.055+0000 7fbe0aa84700 1 -- 172.21.15.28:0/58888567 >> 172.21.15.28:0/58888567 conn(0x7fbe040fc730 msgr2=0x7fbe040fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:47.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.055+0000 7fbe0aa84700 1 -- 172.21.15.28:0/58888567 shutdown_connections 2024-04-25T01:32:47.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.055+0000 7fbe0aa84700 1 -- 172.21.15.28:0/58888567 wait complete. 2024-04-25T01:32:47.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:47 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/58888567' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T01:32:47.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:47 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/58888567' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T01:32:47.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:47 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/58888567' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T01:32:47.750 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-25T01:32:47.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:32:47.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.899+0000 7f30e67d5640 1 -- 172.21.15.28:0/2231774882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e0107480 msgr2=0x7f30e01098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:47.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.899+0000 7f30e67d5640 1 --2- 172.21.15.28:0/2231774882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e0107480 0x7f30e01098d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f30d8009ca0 tx=0x7f30d802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:47.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.899+0000 7f30e67d5640 1 -- 172.21.15.28:0/2231774882 shutdown_connections 2024-04-25T01:32:47.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.899+0000 7f30e67d5640 1 --2- 172.21.15.28:0/2231774882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e0107480 0x7f30e01098d0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:47.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.899+0000 7f30e67d5640 1 --2- 172.21.15.28:0/2231774882 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30e0104b50 0x7f30e0106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:47.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.899+0000 7f30e67d5640 1 --2- 172.21.15.28:0/2231774882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e0102220 0x7f30e0104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:47.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.899+0000 7f30e67d5640 1 -- 172.21.15.28:0/2231774882 >> 172.21.15.28:0/2231774882 conn(0x7f30e00fbc30 msgr2=0x7f30e00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:47.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.899+0000 7f30e67d5640 1 -- 172.21.15.28:0/2231774882 shutdown_connections 2024-04-25T01:32:47.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.899+0000 7f30e67d5640 1 -- 172.21.15.28:0/2231774882 wait complete. 2024-04-25T01:32:47.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.900+0000 7f30e67d5640 1 Processor -- start 2024-04-25T01:32:47.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.900+0000 7f30e67d5640 1 -- start start 2024-04-25T01:32:47.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.900+0000 7f30e67d5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30e0102220 0x7f30e0197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:47.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.900+0000 7f30e67d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e0104b50 0x7f30e0197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:47.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.900+0000 7f30e67d5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e0107480 0x7f30e019eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:47.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.900+0000 7f30e67d5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f30e010d470 con 0x7f30e0104b50 2024-04-25T01:32:47.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.900+0000 7f30e67d5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f30e010d2f0 con 0x7f30e0107480 2024-04-25T01:32:47.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.900+0000 7f30e67d5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f30e010d5f0 con 0x7f30e0102220 2024-04-25T01:32:47.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.900+0000 7f30df7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e0104b50 0x7f30e0197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.900+0000 7f30df7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e0104b50 0x7f30e0197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40858/0 (socket says 172.21.15.28:40858) 2024-04-25T01:32:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.900+0000 7f30df7fe640 1 -- 172.21.15.28:0/433615074 learned_addr learned my addr 172.21.15.28:0/433615074 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.900+0000 7f30e4d72640 1 --2- 172.21.15.28:0/433615074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e0107480 0x7f30e019eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.900+0000 7f30dffff640 1 --2- 172.21.15.28:0/433615074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30e0102220 0x7f30e0197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.900+0000 7f30df7fe640 1 -- 172.21.15.28:0/433615074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30e0102220 msgr2=0x7f30e0197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.900+0000 7f30df7fe640 1 --2- 172.21.15.28:0/433615074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30e0102220 0x7f30e0197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.901+0000 7f30df7fe640 1 -- 172.21.15.28:0/433615074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e0107480 msgr2=0x7f30e019eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.901+0000 7f30df7fe640 1 --2- 172.21.15.28:0/433615074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e0107480 0x7f30e019eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.901+0000 7f30df7fe640 1 -- 172.21.15.28:0/433615074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e019f540 con 0x7f30e0104b50 2024-04-25T01:32:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.901+0000 7f30e4d72640 1 --2- 172.21.15.28:0/433615074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e0107480 0x7f30e019eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.901+0000 7f30dffff640 1 --2- 172.21.15.28:0/433615074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30e0102220 0x7f30e0197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.901+0000 7f30df7fe640 1 --2- 172.21.15.28:0/433615074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e0104b50 0x7f30e0197e60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f30d400c970 tx=0x7f30d400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.901+0000 7f30dd7fa640 1 -- 172.21.15.28:0/433615074 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d4014070 con 0x7f30e0104b50 2024-04-25T01:32:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.901+0000 7f30e67d5640 1 -- 172.21.15.28:0/433615074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30e019f830 con 0x7f30e0104b50 2024-04-25T01:32:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.901+0000 7f30dd7fa640 1 -- 172.21.15.28:0/433615074 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30d4004440 con 0x7f30e0104b50 2024-04-25T01:32:47.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.901+0000 7f30dd7fa640 1 -- 172.21.15.28:0/433615074 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d400f450 con 0x7f30e0104b50 2024-04-25T01:32:47.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.901+0000 7f30e67d5640 1 -- 172.21.15.28:0/433615074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30e019fd80 con 0x7f30e0104b50 2024-04-25T01:32:47.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.902+0000 7f30e67d5640 1 -- 172.21.15.28:0/433615074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30e00fd720 con 0x7f30e0104b50 2024-04-25T01:32:47.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.903+0000 7f30dd7fa640 1 -- 172.21.15.28:0/433615074 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f30d4017070 con 0x7f30e0104b50 2024-04-25T01:32:47.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.904+0000 7f30dd7fa640 1 --2- 172.21.15.28:0/433615074 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f30c80a2b90 0x7f30c80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:47.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.904+0000 7f30dd7fa640 1 -- 172.21.15.28:0/433615074 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f30d40cf2b0 con 0x7f30e0104b50 2024-04-25T01:32:47.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.904+0000 7f30dffff640 1 --2- 172.21.15.28:0/433615074 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f30c80a2b90 0x7f30c80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:47.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.904+0000 7f30dffff640 1 --2- 172.21.15.28:0/433615074 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f30c80a2b90 0x7f30c80a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f30d000baa0 tx=0x7f30d0005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:47.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:47.908+0000 7f30dd7fa640 1 -- 172.21.15.28:0/433615074 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f30d401e020 con 0x7f30e0104b50 2024-04-25T01:32:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.092+0000 7f30e67d5640 1 -- 172.21.15.28:0/433615074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f30e002d170 con 0x7f30e0104b50 2024-04-25T01:32:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.094+0000 7f30dd7fa640 1 -- 172.21.15.28:0/433615074 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24564 (secure 0 0 0) 0x7f30d409beb0 con 0x7f30e0104b50 2024-04-25T01:32:48.072 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:32:48.072 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:31:21.857121+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:32:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.096+0000 7f30e67d5640 1 -- 172.21.15.28:0/433615074 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f30c80a2b90 msgr2=0x7f30c80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.096+0000 7f30e67d5640 1 --2- 172.21.15.28:0/433615074 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f30c80a2b90 0x7f30c80a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f30d000baa0 tx=0x7f30d0005d90 comp rx=0 tx=0).stop 2024-04-25T01:32:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.096+0000 7f30e67d5640 1 -- 172.21.15.28:0/433615074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e0104b50 msgr2=0x7f30e0197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.096+0000 7f30e67d5640 1 --2- 172.21.15.28:0/433615074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e0104b50 0x7f30e0197e60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f30d400c970 tx=0x7f30d400ce30 comp rx=0 tx=0).stop 2024-04-25T01:32:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.096+0000 7f30e67d5640 1 -- 172.21.15.28:0/433615074 shutdown_connections 2024-04-25T01:32:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.096+0000 7f30e67d5640 1 --2- 172.21.15.28:0/433615074 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f30c80a2b90 0x7f30c80a5050 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.096+0000 7f30e67d5640 1 --2- 172.21.15.28:0/433615074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30e0107480 0x7f30e019eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.096+0000 7f30e67d5640 1 --2- 172.21.15.28:0/433615074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e0104b50 0x7f30e0197e60 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.096+0000 7f30e67d5640 1 --2- 172.21.15.28:0/433615074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f30e0102220 0x7f30e0197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.096+0000 7f30e67d5640 1 -- 172.21.15.28:0/433615074 >> 172.21.15.28:0/433615074 conn(0x7f30e00fbc30 msgr2=0x7f30e0102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.096+0000 7f30e67d5640 1 -- 172.21.15.28:0/433615074 shutdown_connections 2024-04-25T01:32:48.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.096+0000 7f30e67d5640 1 -- 172.21.15.28:0/433615074 wait complete. 2024-04-25T01:32:48.085 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:25:48.767414+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:30:32.282430+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:30:33.127049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:30:36.796132+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:32:48.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:32:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.213+0000 7f72dccbd640 1 -- 172.21.15.28:0/2835658779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d8102230 msgr2=0x7f72d8108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.213+0000 7f72dccbd640 1 --2- 172.21.15.28:0/2835658779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d8102230 0x7f72d8108ac0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009ca0 tx=0x7f72d002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.213+0000 7f72dccbd640 1 -- 172.21.15.28:0/2835658779 shutdown_connections 2024-04-25T01:32:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.213+0000 7f72dccbd640 1 --2- 172.21.15.28:0/2835658779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d8102230 0x7f72d8108ac0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.213+0000 7f72dccbd640 1 --2- 172.21.15.28:0/2835658779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72d8101870 0x7f72d8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.213+0000 7f72dccbd640 1 --2- 172.21.15.28:0/2835658779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72d8100680 0x7f72d8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.213+0000 7f72dccbd640 1 -- 172.21.15.28:0/2835658779 >> 172.21.15.28:0/2835658779 conn(0x7f72d80fbbf0 msgr2=0x7f72d80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.213+0000 7f72dccbd640 1 -- 172.21.15.28:0/2835658779 shutdown_connections 2024-04-25T01:32:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.213+0000 7f72dccbd640 1 -- 172.21.15.28:0/2835658779 wait complete. 2024-04-25T01:32:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.213+0000 7f72dccbd640 1 Processor -- start 2024-04-25T01:32:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72dccbd640 1 -- start start 2024-04-25T01:32:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72dccbd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72d8100680 0x7f72d8069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72dccbd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d8101870 0x7f72d819c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72dccbd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72d8102230 0x7f72d819ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72dccbd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f72d810c350 con 0x7f72d8101870 2024-04-25T01:32:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72dccbd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f72d810c1d0 con 0x7f72d8100680 2024-04-25T01:32:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72dccbd640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f72d810c4d0 con 0x7f72d8102230 2024-04-25T01:32:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72d5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d8101870 0x7f72d819c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72d5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d8101870 0x7f72d819c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40878/0 (socket says 172.21.15.28:40878) 2024-04-25T01:32:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72d5d9b640 1 -- 172.21.15.28:0/15869248 learned_addr learned my addr 172.21.15.28:0/15869248 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72d659c640 1 --2- 172.21.15.28:0/15869248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72d8100680 0x7f72d8069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72d6d9d640 1 --2- 172.21.15.28:0/15869248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72d8102230 0x7f72d819ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72d5d9b640 1 -- 172.21.15.28:0/15869248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72d8102230 msgr2=0x7f72d819ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72d5d9b640 1 --2- 172.21.15.28:0/15869248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72d8102230 0x7f72d819ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72d5d9b640 1 -- 172.21.15.28:0/15869248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72d8100680 msgr2=0x7f72d8069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72d5d9b640 1 --2- 172.21.15.28:0/15869248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72d8100680 0x7f72d8069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72d5d9b640 1 -- 172.21.15.28:0/15869248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d819f720 con 0x7f72d8101870 2024-04-25T01:32:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72d659c640 1 --2- 172.21.15.28:0/15869248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72d8100680 0x7f72d8069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72d6d9d640 1 --2- 172.21.15.28:0/15869248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72d8102230 0x7f72d819ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72d5d9b640 1 --2- 172.21.15.28:0/15869248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d8101870 0x7f72d819c7f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f72cc00d950 tx=0x7f72cc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.214+0000 7f72c77fe640 1 -- 172.21.15.28:0/15869248 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc017030 con 0x7f72d8101870 2024-04-25T01:32:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.215+0000 7f72c77fe640 1 -- 172.21.15.28:0/15869248 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72cc004020 con 0x7f72d8101870 2024-04-25T01:32:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.215+0000 7f72dccbd640 1 -- 172.21.15.28:0/15869248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d819f9b0 con 0x7f72d8101870 2024-04-25T01:32:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.215+0000 7f72c77fe640 1 -- 172.21.15.28:0/15869248 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc0047a0 con 0x7f72d8101870 2024-04-25T01:32:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.215+0000 7f72dccbd640 1 -- 172.21.15.28:0/15869248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d819ff00 con 0x7f72d8101870 2024-04-25T01:32:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.216+0000 7f72dccbd640 1 -- 172.21.15.28:0/15869248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72d8107940 con 0x7f72d8101870 2024-04-25T01:32:48.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.217+0000 7f72c77fe640 1 -- 172.21.15.28:0/15869248 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f72cc0041d0 con 0x7f72d8101870 2024-04-25T01:32:48.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.217+0000 7f72c77fe640 1 --2- 172.21.15.28:0/15869248 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f72c00a2b90 0x7f72c00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:48.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.217+0000 7f72d659c640 1 --2- 172.21.15.28:0/15869248 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f72c00a2b90 0x7f72c00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:48.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.217+0000 7f72c77fe640 1 -- 172.21.15.28:0/15869248 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f72cc0cee20 con 0x7f72d8101870 2024-04-25T01:32:48.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.218+0000 7f72d659c640 1 --2- 172.21.15.28:0/15869248 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f72c00a2b90 0x7f72c00a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f72d81016d0 tx=0x7f72c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:48.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.222+0000 7f72c77fe640 1 -- 172.21.15.28:0/15869248 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f72cc09ba20 con 0x7f72d8101870 2024-04-25T01:32:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.406+0000 7f72dccbd640 1 -- 172.21.15.28:0/15869248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f72d8062370 con 0x7f72d8101870 2024-04-25T01:32:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.406+0000 7f72c77fe640 1 -- 172.21.15.28:0/15869248 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f72cc09b320 con 0x7f72d8101870 2024-04-25T01:32:48.383 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T01:32:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.408+0000 7f72dccbd640 1 -- 172.21.15.28:0/15869248 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f72c00a2b90 msgr2=0x7f72c00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.408+0000 7f72dccbd640 1 --2- 172.21.15.28:0/15869248 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f72c00a2b90 0x7f72c00a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f72d81016d0 tx=0x7f72c8005f90 comp rx=0 tx=0).stop 2024-04-25T01:32:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.408+0000 7f72dccbd640 1 -- 172.21.15.28:0/15869248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d8101870 msgr2=0x7f72d819c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.408+0000 7f72dccbd640 1 --2- 172.21.15.28:0/15869248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d8101870 0x7f72d819c7f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f72cc00d950 tx=0x7f72cc00de10 comp rx=0 tx=0).stop 2024-04-25T01:32:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.408+0000 7f72dccbd640 1 -- 172.21.15.28:0/15869248 shutdown_connections 2024-04-25T01:32:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.408+0000 7f72dccbd640 1 --2- 172.21.15.28:0/15869248 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f72c00a2b90 0x7f72c00a5050 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.408+0000 7f72dccbd640 1 --2- 172.21.15.28:0/15869248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72d8102230 0x7f72d819ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.408+0000 7f72dccbd640 1 --2- 172.21.15.28:0/15869248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d8101870 0x7f72d819c7f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.408+0000 7f72dccbd640 1 --2- 172.21.15.28:0/15869248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72d8100680 0x7f72d8069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.408+0000 7f72dccbd640 1 -- 172.21.15.28:0/15869248 >> 172.21.15.28:0/15869248 conn(0x7f72d80fbbf0 msgr2=0x7f72d8104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.408+0000 7f72dccbd640 1 -- 172.21.15.28:0/15869248 shutdown_connections 2024-04-25T01:32:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.408+0000 7f72dccbd640 1 -- 172.21.15.28:0/15869248 wait complete. 2024-04-25T01:32:48.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:48 smithi028 ceph-mon[29390]: pgmap v366: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:48.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:48 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/433615074' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:48.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:32:48.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:48 smithi130 ceph-mon[32702]: pgmap v366: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:48.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:48 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/433615074' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:48.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:48 smithi032 ceph-mon[32461]: pgmap v366: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:48.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:48 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/433615074' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:48.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.524+0000 7f7a15f16640 1 -- 172.21.15.28:0/2483362887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a10101890 msgr2=0x7f7a10101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:48.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.524+0000 7f7a15f16640 1 --2- 172.21.15.28:0/2483362887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a10101890 0x7f7a10101d10 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7a04009900 tx=0x7f7a0402a410 comp rx=0 tx=0).stop 2024-04-25T01:32:48.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.525+0000 7f7a15f16640 1 -- 172.21.15.28:0/2483362887 shutdown_connections 2024-04-25T01:32:48.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.525+0000 7f7a15f16640 1 --2- 172.21.15.28:0/2483362887 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a10102250 0x7f7a10108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.525+0000 7f7a15f16640 1 --2- 172.21.15.28:0/2483362887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a10101890 0x7f7a10101d10 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.525+0000 7f7a15f16640 1 --2- 172.21.15.28:0/2483362887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a101006a0 0x7f7a10100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.525+0000 7f7a15f16640 1 -- 172.21.15.28:0/2483362887 >> 172.21.15.28:0/2483362887 conn(0x7f7a100fbc30 msgr2=0x7f7a100fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:48.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.525+0000 7f7a15f16640 1 -- 172.21.15.28:0/2483362887 shutdown_connections 2024-04-25T01:32:48.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.525+0000 7f7a15f16640 1 -- 172.21.15.28:0/2483362887 wait complete. 2024-04-25T01:32:48.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.525+0000 7f7a15f16640 1 Processor -- start 2024-04-25T01:32:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.525+0000 7f7a15f16640 1 -- start start 2024-04-25T01:32:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.525+0000 7f7a15f16640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a101006a0 0x7f7a1006a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.525+0000 7f7a15f16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a10101890 0x7f7a1019c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a15f16640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a10102250 0x7f7a1019edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a15f16640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1010c300 con 0x7f7a10101890 2024-04-25T01:32:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a15f16640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1010c180 con 0x7f7a10102250 2024-04-25T01:32:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a15f16640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1010c480 con 0x7f7a101006a0 2024-04-25T01:32:48.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a0effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a10101890 0x7f7a1019c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:48.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a0effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a10101890 0x7f7a1019c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40902/0 (socket says 172.21.15.28:40902) 2024-04-25T01:32:48.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a0effd640 1 -- 172.21.15.28:0/3440903480 learned_addr learned my addr 172.21.15.28:0/3440903480 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:48.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a0f7fe640 1 --2- 172.21.15.28:0/3440903480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a101006a0 0x7f7a1006a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:48.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a0ffff640 1 --2- 172.21.15.28:0/3440903480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a10102250 0x7f7a1019edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:48.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a0effd640 1 -- 172.21.15.28:0/3440903480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a101006a0 msgr2=0x7f7a1006a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:48.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a0effd640 1 --2- 172.21.15.28:0/3440903480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a101006a0 0x7f7a1006a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a0effd640 1 -- 172.21.15.28:0/3440903480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a10102250 msgr2=0x7f7a1019edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:48.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a0effd640 1 --2- 172.21.15.28:0/3440903480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a10102250 0x7f7a1019edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a0effd640 1 -- 172.21.15.28:0/3440903480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a1019f6d0 con 0x7f7a10101890 2024-04-25T01:32:48.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a0ffff640 1 --2- 172.21.15.28:0/3440903480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a10102250 0x7f7a1019edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:48.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a0effd640 1 --2- 172.21.15.28:0/3440903480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a10101890 0x7f7a1019c7a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7a04009230 tx=0x7f7a04031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:48.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a0cff9640 1 -- 172.21.15.28:0/3440903480 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0403f030 con 0x7f7a10101890 2024-04-25T01:32:48.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a0cff9640 1 -- 172.21.15.28:0/3440903480 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a04003ad0 con 0x7f7a10101890 2024-04-25T01:32:48.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a15f16640 1 -- 172.21.15.28:0/3440903480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a1019f900 con 0x7f7a10101890 2024-04-25T01:32:48.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a0cff9640 1 -- 172.21.15.28:0/3440903480 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a04035070 con 0x7f7a10101890 2024-04-25T01:32:48.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.526+0000 7f7a15f16640 1 -- 172.21.15.28:0/3440903480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a1019fdf0 con 0x7f7a10101890 2024-04-25T01:32:48.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.527+0000 7f7a15f16640 1 -- 172.21.15.28:0/3440903480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a101079c0 con 0x7f7a10101890 2024-04-25T01:32:48.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.528+0000 7f7a0cff9640 1 -- 172.21.15.28:0/3440903480 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7a04042030 con 0x7f7a10101890 2024-04-25T01:32:48.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.529+0000 7f7a0cff9640 1 --2- 172.21.15.28:0/3440903480 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f79f80a2b90 0x7f79f80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:48.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.529+0000 7f7a0f7fe640 1 --2- 172.21.15.28:0/3440903480 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f79f80a2b90 0x7f79f80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:48.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.529+0000 7f7a0cff9640 1 -- 172.21.15.28:0/3440903480 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f7a040ef9e0 con 0x7f7a10101890 2024-04-25T01:32:48.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.530+0000 7f7a0f7fe640 1 --2- 172.21.15.28:0/3440903480 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f79f80a2b90 0x7f79f80a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7a101016f0 tx=0x7f7a00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:48.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.533+0000 7f7a0cff9640 1 -- 172.21.15.28:0/3440903480 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7a040bc5e0 con 0x7f7a10101890 2024-04-25T01:32:48.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.716+0000 7f7a15f16640 1 -- 172.21.15.28:0/3440903480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7a10062340 con 0x7f7a10101890 2024-04-25T01:32:48.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.716+0000 7f7a0cff9640 1 -- 172.21.15.28:0/3440903480 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f7a040bbee0 con 0x7f7a10101890 2024-04-25T01:32:48.694 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:32:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.718+0000 7f7a15f16640 1 -- 172.21.15.28:0/3440903480 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f79f80a2b90 msgr2=0x7f79f80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.718+0000 7f7a15f16640 1 --2- 172.21.15.28:0/3440903480 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f79f80a2b90 0x7f79f80a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7a101016f0 tx=0x7f7a00005f90 comp rx=0 tx=0).stop 2024-04-25T01:32:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.718+0000 7f7a15f16640 1 -- 172.21.15.28:0/3440903480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a10101890 msgr2=0x7f7a1019c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.718+0000 7f7a15f16640 1 --2- 172.21.15.28:0/3440903480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a10101890 0x7f7a1019c7a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7a04009230 tx=0x7f7a04031b60 comp rx=0 tx=0).stop 2024-04-25T01:32:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.718+0000 7f7a15f16640 1 -- 172.21.15.28:0/3440903480 shutdown_connections 2024-04-25T01:32:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.718+0000 7f7a15f16640 1 --2- 172.21.15.28:0/3440903480 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f79f80a2b90 0x7f79f80a5050 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.718+0000 7f7a15f16640 1 --2- 172.21.15.28:0/3440903480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a10102250 0x7f7a1019edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.718+0000 7f7a15f16640 1 --2- 172.21.15.28:0/3440903480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a10101890 0x7f7a1019c7a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.718+0000 7f7a15f16640 1 --2- 172.21.15.28:0/3440903480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a101006a0 0x7f7a1006a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.718+0000 7f7a15f16640 1 -- 172.21.15.28:0/3440903480 >> 172.21.15.28:0/3440903480 conn(0x7f7a100fbc30 msgr2=0x7f7a10104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.718+0000 7f7a15f16640 1 -- 172.21.15.28:0/3440903480 shutdown_connections 2024-04-25T01:32:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.718+0000 7f7a15f16640 1 -- 172.21.15.28:0/3440903480 wait complete. 2024-04-25T01:32:48.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:32:48.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.834+0000 7f1c10121640 1 -- 172.21.15.28:0/3759033338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c08101890 msgr2=0x7f1c08101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.834+0000 7f1c10121640 1 --2- 172.21.15.28:0/3759033338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c08101890 0x7f1c08101d10 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8009900 tx=0x7f1bf802a410 comp rx=0 tx=0).stop 2024-04-25T01:32:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.834+0000 7f1c10121640 1 -- 172.21.15.28:0/3759033338 shutdown_connections 2024-04-25T01:32:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.834+0000 7f1c10121640 1 --2- 172.21.15.28:0/3759033338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c08102250 0x7f1c08108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.834+0000 7f1c10121640 1 --2- 172.21.15.28:0/3759033338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c08101890 0x7f1c08101d10 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.834+0000 7f1c10121640 1 --2- 172.21.15.28:0/3759033338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c081006a0 0x7f1c08100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.834+0000 7f1c10121640 1 -- 172.21.15.28:0/3759033338 >> 172.21.15.28:0/3759033338 conn(0x7f1c080fbc30 msgr2=0x7f1c080fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.834+0000 7f1c10121640 1 -- 172.21.15.28:0/3759033338 shutdown_connections 2024-04-25T01:32:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.834+0000 7f1c10121640 1 -- 172.21.15.28:0/3759033338 wait complete. 2024-04-25T01:32:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.835+0000 7f1c10121640 1 Processor -- start 2024-04-25T01:32:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.835+0000 7f1c10121640 1 -- start start 2024-04-25T01:32:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.835+0000 7f1c10121640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c081006a0 0x7f1c0806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.835+0000 7f1c10121640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c08101890 0x7f1c0819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.835+0000 7f1c10121640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c08102250 0x7f1c0819ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.835+0000 7f1c10121640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0810c320 con 0x7f1c081006a0 2024-04-25T01:32:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.835+0000 7f1c10121640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0810c1a0 con 0x7f1c08101890 2024-04-25T01:32:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.835+0000 7f1c10121640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0810c4a0 con 0x7f1c08102250 2024-04-25T01:32:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.835+0000 7f1c0debd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c081006a0 0x7f1c0806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.835+0000 7f1c0debd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c081006a0 0x7f1c0806a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40932/0 (socket says 172.21.15.28:40932) 2024-04-25T01:32:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.835+0000 7f1c0debd640 1 -- 172.21.15.28:0/2651165787 learned_addr learned my addr 172.21.15.28:0/2651165787 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.835+0000 7f1c0d6bc640 1 --2- 172.21.15.28:0/2651165787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c08101890 0x7f1c0819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.835+0000 7f1c0e6be640 1 --2- 172.21.15.28:0/2651165787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c08102250 0x7f1c0819ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.835+0000 7f1c0debd640 1 -- 172.21.15.28:0/2651165787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c08102250 msgr2=0x7f1c0819ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.835+0000 7f1c0debd640 1 --2- 172.21.15.28:0/2651165787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c08102250 0x7f1c0819ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.835+0000 7f1c0debd640 1 -- 172.21.15.28:0/2651165787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c08101890 msgr2=0x7f1c0819c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.835+0000 7f1c0debd640 1 --2- 172.21.15.28:0/2651165787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c08101890 0x7f1c0819c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.835+0000 7f1c0debd640 1 -- 172.21.15.28:0/2651165787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0819f6c0 con 0x7f1c081006a0 2024-04-25T01:32:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.836+0000 7f1c0debd640 1 --2- 172.21.15.28:0/2651165787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c081006a0 0x7f1c0806a8d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400d770 tx=0x7f1c0400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.836+0000 7f1bfeffd640 1 -- 172.21.15.28:0/2651165787 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c04017070 con 0x7f1c081006a0 2024-04-25T01:32:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.836+0000 7f1c10121640 1 -- 172.21.15.28:0/2651165787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c0819f9b0 con 0x7f1c081006a0 2024-04-25T01:32:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.836+0000 7f1bfeffd640 1 -- 172.21.15.28:0/2651165787 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c04009c30 con 0x7f1c081006a0 2024-04-25T01:32:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.836+0000 7f1bfeffd640 1 -- 172.21.15.28:0/2651165787 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c04003dd0 con 0x7f1c081006a0 2024-04-25T01:32:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.836+0000 7f1c10121640 1 -- 172.21.15.28:0/2651165787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c0819fed0 con 0x7f1c081006a0 2024-04-25T01:32:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.837+0000 7f1c10121640 1 -- 172.21.15.28:0/2651165787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c08107a40 con 0x7f1c081006a0 2024-04-25T01:32:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.838+0000 7f1bfeffd640 1 -- 172.21.15.28:0/2651165787 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1c04003f90 con 0x7f1c081006a0 2024-04-25T01:32:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.839+0000 7f1bfeffd640 1 --2- 172.21.15.28:0/2651165787 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1bf40a2b90 0x7f1bf40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.839+0000 7f1c0d6bc640 1 --2- 172.21.15.28:0/2651165787 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1bf40a2b90 0x7f1bf40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.839+0000 7f1bfeffd640 1 -- 172.21.15.28:0/2651165787 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f1c040cf100 con 0x7f1c081006a0 2024-04-25T01:32:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.839+0000 7f1c0d6bc640 1 --2- 172.21.15.28:0/2651165787 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1bf40a2b90 0x7f1bf40a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1bf80314e0 tx=0x7f1bf8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:48.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:48.843+0000 7f1bfeffd640 1 -- 172.21.15.28:0/2651165787 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1c04009de0 con 0x7f1c081006a0 2024-04-25T01:32:49.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.027+0000 7f1c10121640 1 -- 172.21.15.28:0/2651165787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1c0802d320 con 0x7f1c081006a0 2024-04-25T01:32:49.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.027+0000 7f1bfeffd640 1 -- 172.21.15.28:0/2651165787 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f1c0409b650 con 0x7f1c081006a0 2024-04-25T01:32:49.005 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T01:32:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.029+0000 7f1c10121640 1 -- 172.21.15.28:0/2651165787 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1bf40a2b90 msgr2=0x7f1bf40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.029+0000 7f1c10121640 1 --2- 172.21.15.28:0/2651165787 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1bf40a2b90 0x7f1bf40a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1bf80314e0 tx=0x7f1bf8000bc0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.029+0000 7f1c10121640 1 -- 172.21.15.28:0/2651165787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c081006a0 msgr2=0x7f1c0806a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.029+0000 7f1c10121640 1 --2- 172.21.15.28:0/2651165787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c081006a0 0x7f1c0806a8d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400d770 tx=0x7f1c0400dc30 comp rx=0 tx=0).stop 2024-04-25T01:32:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.029+0000 7f1c10121640 1 -- 172.21.15.28:0/2651165787 shutdown_connections 2024-04-25T01:32:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.029+0000 7f1c10121640 1 --2- 172.21.15.28:0/2651165787 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1bf40a2b90 0x7f1bf40a5050 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.029+0000 7f1c10121640 1 --2- 172.21.15.28:0/2651165787 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c08102250 0x7f1c0819ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.029+0000 7f1c10121640 1 --2- 172.21.15.28:0/2651165787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c08101890 0x7f1c0819c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.029+0000 7f1c10121640 1 --2- 172.21.15.28:0/2651165787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c081006a0 0x7f1c0806a8d0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.029+0000 7f1c10121640 1 -- 172.21.15.28:0/2651165787 >> 172.21.15.28:0/2651165787 conn(0x7f1c080fbc30 msgr2=0x7f1c08104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:49.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.029+0000 7f1c10121640 1 -- 172.21.15.28:0/2651165787 shutdown_connections 2024-04-25T01:32:49.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.029+0000 7f1c10121640 1 -- 172.21.15.28:0/2651165787 wait complete. 2024-04-25T01:32:49.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:32:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.147+0000 7f5ec7763640 1 -- 172.21.15.28:0/494804737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec0102230 msgr2=0x7f5ec0108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.147+0000 7f5ec7763640 1 --2- 172.21.15.28:0/494804737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec0102230 0x7f5ec0108ac0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009ca0 tx=0x7f5eb802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.147+0000 7f5ec7763640 1 -- 172.21.15.28:0/494804737 shutdown_connections 2024-04-25T01:32:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.147+0000 7f5ec7763640 1 --2- 172.21.15.28:0/494804737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec0102230 0x7f5ec0108ac0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.147+0000 7f5ec7763640 1 --2- 172.21.15.28:0/494804737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ec0101870 0x7f5ec0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.147+0000 7f5ec7763640 1 --2- 172.21.15.28:0/494804737 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5ec0100680 0x7f5ec0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.147+0000 7f5ec7763640 1 -- 172.21.15.28:0/494804737 >> 172.21.15.28:0/494804737 conn(0x7f5ec00fbbf0 msgr2=0x7f5ec00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.147+0000 7f5ec7763640 1 -- 172.21.15.28:0/494804737 shutdown_connections 2024-04-25T01:32:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.147+0000 7f5ec7763640 1 -- 172.21.15.28:0/494804737 wait complete. 2024-04-25T01:32:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.148+0000 7f5ec7763640 1 Processor -- start 2024-04-25T01:32:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.148+0000 7f5ec7763640 1 -- start start 2024-04-25T01:32:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.148+0000 7f5ec7763640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ec0100680 0x7f5ec0197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.148+0000 7f5ec7763640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5ec0101870 0x7f5ec0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.148+0000 7f5ec7763640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec0102230 0x7f5ec019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.148+0000 7f5ec7763640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec010c330 con 0x7f5ec0102230 2024-04-25T01:32:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.148+0000 7f5ec7763640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec010c1b0 con 0x7f5ec0100680 2024-04-25T01:32:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.148+0000 7f5ec7763640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec010c4b0 con 0x7f5ec0101870 2024-04-25T01:32:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.148+0000 7f5ec5d00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec0102230 0x7f5ec019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.148+0000 7f5ec5d00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec0102230 0x7f5ec019f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40942/0 (socket says 172.21.15.28:40942) 2024-04-25T01:32:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.148+0000 7f5ec5d00640 1 -- 172.21.15.28:0/4180049155 learned_addr learned my addr 172.21.15.28:0/4180049155 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.148+0000 7f5ec4cfe640 1 --2- 172.21.15.28:0/4180049155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5ec0101870 0x7f5ec0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.148+0000 7f5ec54ff640 1 --2- 172.21.15.28:0/4180049155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ec0100680 0x7f5ec0197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.148+0000 7f5ec5d00640 1 -- 172.21.15.28:0/4180049155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5ec0101870 msgr2=0x7f5ec0197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.148+0000 7f5ec5d00640 1 --2- 172.21.15.28:0/4180049155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5ec0101870 0x7f5ec0197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.148+0000 7f5ec5d00640 1 -- 172.21.15.28:0/4180049155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ec0100680 msgr2=0x7f5ec0197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.148+0000 7f5ec5d00640 1 --2- 172.21.15.28:0/4180049155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ec0100680 0x7f5ec0197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.148+0000 7f5ec5d00640 1 -- 172.21.15.28:0/4180049155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ec019f6b0 con 0x7f5ec0102230 2024-04-25T01:32:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.149+0000 7f5ec5d00640 1 --2- 172.21.15.28:0/4180049155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec0102230 0x7f5ec019f040 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009430 tx=0x7f5eb80319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.149+0000 7f5eb67fc640 1 -- 172.21.15.28:0/4180049155 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb8003b70 con 0x7f5ec0102230 2024-04-25T01:32:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.149+0000 7f5eb67fc640 1 -- 172.21.15.28:0/4180049155 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5eb8003d10 con 0x7f5ec0102230 2024-04-25T01:32:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.149+0000 7f5ec7763640 1 -- 172.21.15.28:0/4180049155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ec019f940 con 0x7f5ec0102230 2024-04-25T01:32:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.149+0000 7f5eb67fc640 1 -- 172.21.15.28:0/4180049155 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb802fc50 con 0x7f5ec0102230 2024-04-25T01:32:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.149+0000 7f5ec7763640 1 -- 172.21.15.28:0/4180049155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ec019fee0 con 0x7f5ec0102230 2024-04-25T01:32:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.150+0000 7f5ec7763640 1 -- 172.21.15.28:0/4180049155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ec01a0110 con 0x7f5ec0102230 2024-04-25T01:32:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.151+0000 7f5eb67fc640 1 -- 172.21.15.28:0/4180049155 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5eb8004040 con 0x7f5ec0102230 2024-04-25T01:32:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.152+0000 7f5eb67fc640 1 --2- 172.21.15.28:0/4180049155 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5eac0a2c10 0x7f5eac0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.152+0000 7f5ec54ff640 1 --2- 172.21.15.28:0/4180049155 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5eac0a2c10 0x7f5eac0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.152+0000 7f5eb67fc640 1 -- 172.21.15.28:0/4180049155 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f5eb80f07f0 con 0x7f5ec0102230 2024-04-25T01:32:49.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.152+0000 7f5ec54ff640 1 --2- 172.21.15.28:0/4180049155 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5eac0a2c10 0x7f5eac0a50d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5ec01016d0 tx=0x7f5ebc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:49.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.156+0000 7f5eb67fc640 1 -- 172.21.15.28:0/4180049155 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5eb80bd390 con 0x7f5ec0102230 2024-04-25T01:32:49.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:49 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/15869248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:32:49.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:49 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3440903480' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:32:49.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:49 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2651165787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:32:49.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.341+0000 7f5ec7763640 1 -- 172.21.15.28:0/4180049155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f5ec01a02e0 con 0x7f5ec0102230 2024-04-25T01:32:49.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.342+0000 7f5eb67fc640 1 -- 172.21.15.28:0/4180049155 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f5eb80bcc90 con 0x7f5ec0102230 2024-04-25T01:32:49.319 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:32:49.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.344+0000 7f5ec7763640 1 -- 172.21.15.28:0/4180049155 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5eac0a2c10 msgr2=0x7f5eac0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:49.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.344+0000 7f5ec7763640 1 --2- 172.21.15.28:0/4180049155 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5eac0a2c10 0x7f5eac0a50d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5ec01016d0 tx=0x7f5ebc005f90 comp rx=0 tx=0).stop 2024-04-25T01:32:49.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.344+0000 7f5ec7763640 1 -- 172.21.15.28:0/4180049155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec0102230 msgr2=0x7f5ec019f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:49.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.344+0000 7f5ec7763640 1 --2- 172.21.15.28:0/4180049155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec0102230 0x7f5ec019f040 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8009430 tx=0x7f5eb80319f0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.344+0000 7f5ec7763640 1 -- 172.21.15.28:0/4180049155 shutdown_connections 2024-04-25T01:32:49.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.344+0000 7f5ec7763640 1 --2- 172.21.15.28:0/4180049155 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5eac0a2c10 0x7f5eac0a50d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.344+0000 7f5ec7763640 1 --2- 172.21.15.28:0/4180049155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ec0102230 0x7f5ec019f040 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.344+0000 7f5ec7763640 1 --2- 172.21.15.28:0/4180049155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5ec0101870 0x7f5ec0197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.344+0000 7f5ec7763640 1 --2- 172.21.15.28:0/4180049155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5ec0100680 0x7f5ec0197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.344+0000 7f5ec7763640 1 -- 172.21.15.28:0/4180049155 >> 172.21.15.28:0/4180049155 conn(0x7f5ec00fbbf0 msgr2=0x7f5ec00fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:49.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.344+0000 7f5ec7763640 1 -- 172.21.15.28:0/4180049155 shutdown_connections 2024-04-25T01:32:49.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.344+0000 7f5ec7763640 1 -- 172.21.15.28:0/4180049155 wait complete. 2024-04-25T01:32:49.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:32:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.458+0000 7faf31510640 1 -- 172.21.15.28:0/3464052187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf2c1029a0 msgr2=0x7faf2c102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.458+0000 7faf31510640 1 --2- 172.21.15.28:0/3464052187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf2c1029a0 0x7faf2c102e20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7faf20009900 tx=0x7faf2002a410 comp rx=0 tx=0).stop 2024-04-25T01:32:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.458+0000 7faf31510640 1 -- 172.21.15.28:0/3464052187 shutdown_connections 2024-04-25T01:32:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.458+0000 7faf31510640 1 --2- 172.21.15.28:0/3464052187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf2c103360 0x7faf2c109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.458+0000 7faf31510640 1 --2- 172.21.15.28:0/3464052187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf2c1029a0 0x7faf2c102e20 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.458+0000 7faf31510640 1 --2- 172.21.15.28:0/3464052187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c1017b0 0x7faf2c101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.458+0000 7faf31510640 1 -- 172.21.15.28:0/3464052187 >> 172.21.15.28:0/3464052187 conn(0x7faf2c0fcd40 msgr2=0x7faf2c0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.458+0000 7faf31510640 1 -- 172.21.15.28:0/3464052187 shutdown_connections 2024-04-25T01:32:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.458+0000 7faf31510640 1 -- 172.21.15.28:0/3464052187 wait complete. 2024-04-25T01:32:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.458+0000 7faf31510640 1 Processor -- start 2024-04-25T01:32:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.458+0000 7faf31510640 1 -- start start 2024-04-25T01:32:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf31510640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c1017b0 0x7faf2c197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf31510640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf2c1029a0 0x7faf2c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf31510640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf2c103360 0x7faf2c19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf31510640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c10d410 con 0x7faf2c103360 2024-04-25T01:32:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf31510640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c10d290 con 0x7faf2c1017b0 2024-04-25T01:32:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf31510640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c10d590 con 0x7faf2c1029a0 2024-04-25T01:32:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf2b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf2c103360 0x7faf2c19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf2b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf2c103360 0x7faf2c19f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40964/0 (socket says 172.21.15.28:40964) 2024-04-25T01:32:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf2b7fe640 1 -- 172.21.15.28:0/1270727332 learned_addr learned my addr 172.21.15.28:0/1270727332 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf2a7fc640 1 --2- 172.21.15.28:0/1270727332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf2c1029a0 0x7faf2c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf2affd640 1 --2- 172.21.15.28:0/1270727332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c1017b0 0x7faf2c197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf2b7fe640 1 -- 172.21.15.28:0/1270727332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf2c1029a0 msgr2=0x7faf2c197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf2b7fe640 1 --2- 172.21.15.28:0/1270727332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf2c1029a0 0x7faf2c197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf2b7fe640 1 -- 172.21.15.28:0/1270727332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c1017b0 msgr2=0x7faf2c197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf2b7fe640 1 --2- 172.21.15.28:0/1270727332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c1017b0 0x7faf2c197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf2b7fe640 1 -- 172.21.15.28:0/1270727332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf2c19f6d0 con 0x7faf2c103360 2024-04-25T01:32:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf2b7fe640 1 --2- 172.21.15.28:0/1270727332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf2c103360 0x7faf2c19f060 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7faf2400eaa0 tx=0x7faf2400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf13fff640 1 -- 172.21.15.28:0/1270727332 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf2400cdb0 con 0x7faf2c103360 2024-04-25T01:32:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf13fff640 1 -- 172.21.15.28:0/1270727332 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf24005020 con 0x7faf2c103360 2024-04-25T01:32:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf31510640 1 -- 172.21.15.28:0/1270727332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf2c19f9c0 con 0x7faf2c103360 2024-04-25T01:32:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf13fff640 1 -- 172.21.15.28:0/1270727332 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf240116b0 con 0x7faf2c103360 2024-04-25T01:32:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.459+0000 7faf31510640 1 -- 172.21.15.28:0/1270727332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf2c19fe90 con 0x7faf2c103360 2024-04-25T01:32:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.461+0000 7faf31510640 1 -- 172.21.15.28:0/1270727332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf2c108ad0 con 0x7faf2c103360 2024-04-25T01:32:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.462+0000 7faf13fff640 1 -- 172.21.15.28:0/1270727332 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7faf24005560 con 0x7faf2c103360 2024-04-25T01:32:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.462+0000 7faf13fff640 1 --2- 172.21.15.28:0/1270727332 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7faf140a2b90 0x7faf140a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.463+0000 7faf2affd640 1 --2- 172.21.15.28:0/1270727332 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7faf140a2b90 0x7faf140a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.463+0000 7faf13fff640 1 -- 172.21.15.28:0/1270727332 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7faf240cf570 con 0x7faf2c103360 2024-04-25T01:32:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.463+0000 7faf2affd640 1 --2- 172.21.15.28:0/1270727332 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7faf140a2b90 0x7faf140a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7faf2c102800 tx=0x7faf1c00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.467+0000 7faf13fff640 1 -- 172.21.15.28:0/1270727332 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faf24016030 con 0x7faf2c103360 2024-04-25T01:32:49.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:49 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/15869248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:32:49.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:49 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3440903480' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:32:49.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:49 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2651165787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:32:49.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:49 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/15869248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:32:49.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:49 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3440903480' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:32:49.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:49 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2651165787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:32:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.693+0000 7faf31510640 1 -- 172.21.15.28:0/1270727332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf2c02d250 con 0x7faf2c103360 2024-04-25T01:32:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.693+0000 7faf13fff640 1 -- 172.21.15.28:0/1270727332 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11923 (secure 0 0 0) 0x7faf2409bb20 con 0x7faf2c103360 2024-04-25T01:32:49.671 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:32:49.671 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:31:01.802718+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24454,24442,24512,24478,24473]},"id":1}]} 2024-04-25T01:32:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-04-25T01:32:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.695+0000 7faf31510640 1 -- 172.21.15.28:0/1270727332 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7faf140a2b90 msgr2=0x7faf140a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.696+0000 7faf31510640 1 --2- 172.21.15.28:0/1270727332 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7faf140a2b90 0x7faf140a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7faf2c102800 tx=0x7faf1c00a2d0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.696+0000 7faf31510640 1 -- 172.21.15.28:0/1270727332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf2c103360 msgr2=0x7faf2c19f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.696+0000 7faf31510640 1 --2- 172.21.15.28:0/1270727332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf2c103360 0x7faf2c19f060 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7faf2400eaa0 tx=0x7faf2400ef60 comp rx=0 tx=0).stop 2024-04-25T01:32:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.696+0000 7faf31510640 1 -- 172.21.15.28:0/1270727332 shutdown_connections 2024-04-25T01:32:49.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.696+0000 7faf31510640 1 --2- 172.21.15.28:0/1270727332 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7faf140a2b90 0x7faf140a5050 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.696+0000 7faf31510640 1 --2- 172.21.15.28:0/1270727332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf2c103360 0x7faf2c19f060 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.696+0000 7faf31510640 1 --2- 172.21.15.28:0/1270727332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf2c1029a0 0x7faf2c197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.696+0000 7faf31510640 1 --2- 172.21.15.28:0/1270727332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf2c1017b0 0x7faf2c197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.696+0000 7faf31510640 1 -- 172.21.15.28:0/1270727332 >> 172.21.15.28:0/1270727332 conn(0x7faf2c0fcd40 msgr2=0x7faf2c105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:49.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.696+0000 7faf31510640 1 -- 172.21.15.28:0/1270727332 shutdown_connections 2024-04-25T01:32:49.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.696+0000 7faf31510640 1 -- 172.21.15.28:0/1270727332 wait complete. 2024-04-25T01:32:49.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:32:49.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.812+0000 7f5aa6bd5640 1 -- 172.21.15.28:0/4208291396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aa0100680 msgr2=0x7f5aa0100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:49.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.812+0000 7f5aa6bd5640 1 --2- 172.21.15.28:0/4208291396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aa0100680 0x7f5aa0100aa0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f5a94009900 tx=0x7f5a9402a410 comp rx=0 tx=0).stop 2024-04-25T01:32:49.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.812+0000 7f5aa6bd5640 1 -- 172.21.15.28:0/4208291396 shutdown_connections 2024-04-25T01:32:49.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.812+0000 7f5aa6bd5640 1 --2- 172.21.15.28:0/4208291396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aa0102230 0x7f5aa0108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.812+0000 7f5aa6bd5640 1 --2- 172.21.15.28:0/4208291396 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5aa0101870 0x7f5aa0101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.812+0000 7f5aa6bd5640 1 --2- 172.21.15.28:0/4208291396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aa0100680 0x7f5aa0100aa0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.812+0000 7f5aa6bd5640 1 -- 172.21.15.28:0/4208291396 >> 172.21.15.28:0/4208291396 conn(0x7f5aa00fbbf0 msgr2=0x7f5aa00fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:49.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.812+0000 7f5aa6bd5640 1 -- 172.21.15.28:0/4208291396 shutdown_connections 2024-04-25T01:32:49.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.812+0000 7f5aa6bd5640 1 -- 172.21.15.28:0/4208291396 wait complete. 2024-04-25T01:32:49.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.813+0000 7f5aa6bd5640 1 Processor -- start 2024-04-25T01:32:49.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.813+0000 7f5aa6bd5640 1 -- start start 2024-04-25T01:32:49.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.813+0000 7f5aa6bd5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aa0100680 0x7f5aa0069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:49.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.813+0000 7f5aa6bd5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5aa0101870 0x7f5aa019c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:49.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.813+0000 7f5aa6bd5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aa0102230 0x7f5aa019edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:49.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.813+0000 7f5aa6bd5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa010c2d0 con 0x7f5aa0100680 2024-04-25T01:32:49.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.813+0000 7f5aa6bd5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa010c150 con 0x7f5aa0102230 2024-04-25T01:32:49.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.813+0000 7f5aa6bd5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa010c450 con 0x7f5aa0101870 2024-04-25T01:32:49.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.813+0000 7f5aa4971640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aa0100680 0x7f5aa0069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:49.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.813+0000 7f5aa4971640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aa0100680 0x7f5aa0069440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40988/0 (socket says 172.21.15.28:40988) 2024-04-25T01:32:49.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.813+0000 7f5aa4971640 1 -- 172.21.15.28:0/3980377315 learned_addr learned my addr 172.21.15.28:0/3980377315 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:49.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.813+0000 7f5aa5172640 1 --2- 172.21.15.28:0/3980377315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aa0102230 0x7f5aa019edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:49.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.813+0000 7f5a9ffff640 1 --2- 172.21.15.28:0/3980377315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5aa0101870 0x7f5aa019c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:49.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.813+0000 7f5aa4971640 1 -- 172.21.15.28:0/3980377315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5aa0101870 msgr2=0x7f5aa019c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:49.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.813+0000 7f5aa4971640 1 --2- 172.21.15.28:0/3980377315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5aa0101870 0x7f5aa019c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.813+0000 7f5aa4971640 1 -- 172.21.15.28:0/3980377315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aa0102230 msgr2=0x7f5aa019edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.813+0000 7f5aa4971640 1 --2- 172.21.15.28:0/3980377315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aa0102230 0x7f5aa019edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.813+0000 7f5aa4971640 1 -- 172.21.15.28:0/3980377315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa019f6c0 con 0x7f5aa0100680 2024-04-25T01:32:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.813+0000 7f5aa5172640 1 --2- 172.21.15.28:0/3980377315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aa0102230 0x7f5aa019edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.814+0000 7f5aa4971640 1 --2- 172.21.15.28:0/3980377315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aa0100680 0x7f5aa0069440 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5a94031610 tx=0x7f5a940043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.814+0000 7f5a9dffb640 1 -- 172.21.15.28:0/3980377315 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9403f030 con 0x7f5aa0100680 2024-04-25T01:32:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.814+0000 7f5aa6bd5640 1 -- 172.21.15.28:0/3980377315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aa019f8f0 con 0x7f5aa0100680 2024-04-25T01:32:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.814+0000 7f5a9dffb640 1 -- 172.21.15.28:0/3980377315 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a94003ce0 con 0x7f5aa0100680 2024-04-25T01:32:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.814+0000 7f5a9dffb640 1 -- 172.21.15.28:0/3980377315 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a94035070 con 0x7f5aa0100680 2024-04-25T01:32:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.814+0000 7f5aa6bd5640 1 -- 172.21.15.28:0/3980377315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aa019fde0 con 0x7f5aa0100680 2024-04-25T01:32:49.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.815+0000 7f5aa6bd5640 1 -- 172.21.15.28:0/3980377315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aa01079a0 con 0x7f5aa0100680 2024-04-25T01:32:49.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.816+0000 7f5a9dffb640 1 -- 172.21.15.28:0/3980377315 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5a94042030 con 0x7f5aa0100680 2024-04-25T01:32:49.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.817+0000 7f5a9dffb640 1 --2- 172.21.15.28:0/3980377315 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5a900a2b90 0x7f5a900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:49.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.817+0000 7f5a9ffff640 1 --2- 172.21.15.28:0/3980377315 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5a900a2b90 0x7f5a900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:49.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.817+0000 7f5a9dffb640 1 -- 172.21.15.28:0/3980377315 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f5a940efd50 con 0x7f5aa0100680 2024-04-25T01:32:49.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.817+0000 7f5a9ffff640 1 --2- 172.21.15.28:0/3980377315 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5a900a2b90 0x7f5a900a5050 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c005e10 tx=0x7f5a8c005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:49.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:49.820+0000 7f5a9dffb640 1 -- 172.21.15.28:0/3980377315 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5a940bc8f0 con 0x7f5aa0100680 2024-04-25T01:32:49.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.009+0000 7f5aa6bd5640 1 -- 172.21.15.28:0/3980377315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5aa002d3c0 con 0x7f5aa0100680 2024-04-25T01:32:49.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.011+0000 7f5a9dffb640 1 -- 172.21.15.28:0/3980377315 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24564 (secure 0 0 0) 0x7f5a940bc1f0 con 0x7f5aa0100680 2024-04-25T01:32:49.989 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:32:49.989 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:31:21.857121+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:32:49.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.013+0000 7f5aa6bd5640 1 -- 172.21.15.28:0/3980377315 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5a900a2b90 msgr2=0x7f5a900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:49.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.013+0000 7f5aa6bd5640 1 --2- 172.21.15.28:0/3980377315 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5a900a2b90 0x7f5a900a5050 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c005e10 tx=0x7f5a8c005da0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.013+0000 7f5aa6bd5640 1 -- 172.21.15.28:0/3980377315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aa0100680 msgr2=0x7f5aa0069440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:49.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.013+0000 7f5aa6bd5640 1 --2- 172.21.15.28:0/3980377315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aa0100680 0x7f5aa0069440 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5a94031610 tx=0x7f5a940043b0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.013+0000 7f5aa6bd5640 1 -- 172.21.15.28:0/3980377315 shutdown_connections 2024-04-25T01:32:49.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.013+0000 7f5aa6bd5640 1 --2- 172.21.15.28:0/3980377315 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f5a900a2b90 0x7f5a900a5050 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.013+0000 7f5aa6bd5640 1 --2- 172.21.15.28:0/3980377315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5aa0102230 0x7f5aa019edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.013+0000 7f5aa6bd5640 1 --2- 172.21.15.28:0/3980377315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5aa0101870 0x7f5aa019c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.013+0000 7f5aa6bd5640 1 --2- 172.21.15.28:0/3980377315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5aa0100680 0x7f5aa0069440 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:49.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.013+0000 7f5aa6bd5640 1 -- 172.21.15.28:0/3980377315 >> 172.21.15.28:0/3980377315 conn(0x7f5aa00fbbf0 msgr2=0x7f5aa0104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:49.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.013+0000 7f5aa6bd5640 1 -- 172.21.15.28:0/3980377315 shutdown_connections 2024-04-25T01:32:49.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.013+0000 7f5aa6bd5640 1 -- 172.21.15.28:0/3980377315 wait complete. 2024-04-25T01:32:50.002 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:25:48.767414+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:30:32.282430+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:30:33.127049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:30:36.796132+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:32:50.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:32:50.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.129+0000 7ff75f16a640 1 -- 172.21.15.28:0/1151835368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff758103350 msgr2=0x7ff758109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:50.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.129+0000 7ff75f16a640 1 --2- 172.21.15.28:0/1151835368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff758103350 0x7ff758109be0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff750009ca0 tx=0x7ff75002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.129+0000 7ff75f16a640 1 -- 172.21.15.28:0/1151835368 shutdown_connections 2024-04-25T01:32:50.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.129+0000 7ff75f16a640 1 --2- 172.21.15.28:0/1151835368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff758103350 0x7ff758109be0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.129+0000 7ff75f16a640 1 --2- 172.21.15.28:0/1151835368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff758102990 0x7ff758102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.129+0000 7ff75f16a640 1 --2- 172.21.15.28:0/1151835368 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff7581017a0 0x7ff758101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.129+0000 7ff75f16a640 1 -- 172.21.15.28:0/1151835368 >> 172.21.15.28:0/1151835368 conn(0x7ff7580fcd10 msgr2=0x7ff7580ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:50.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.130+0000 7ff75f16a640 1 -- 172.21.15.28:0/1151835368 shutdown_connections 2024-04-25T01:32:50.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.130+0000 7ff75f16a640 1 -- 172.21.15.28:0/1151835368 wait complete. 2024-04-25T01:32:50.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.130+0000 7ff75f16a640 1 Processor -- start 2024-04-25T01:32:50.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.130+0000 7ff75f16a640 1 -- start start 2024-04-25T01:32:50.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.130+0000 7ff75f16a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7581017a0 0x7ff758197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:50.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.130+0000 7ff75f16a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff758102990 0x7ff758197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:50.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.130+0000 7ff75f16a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff758103350 0x7ff75819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:50.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.130+0000 7ff75f16a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff75810d400 con 0x7ff758102990 2024-04-25T01:32:50.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.130+0000 7ff75f16a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff75810d280 con 0x7ff7581017a0 2024-04-25T01:32:50.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.130+0000 7ff75f16a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff75810d580 con 0x7ff758103350 2024-04-25T01:32:50.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.130+0000 7ff757fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff758102990 0x7ff758197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:50.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.130+0000 7ff757fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff758102990 0x7ff758197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41004/0 (socket says 172.21.15.28:41004) 2024-04-25T01:32:50.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.130+0000 7ff757fff640 1 -- 172.21.15.28:0/2236853588 learned_addr learned my addr 172.21.15.28:0/2236853588 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:50.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.131+0000 7ff75cf06640 1 --2- 172.21.15.28:0/2236853588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7581017a0 0x7ff758197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:50.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.131+0000 7ff75d707640 1 --2- 172.21.15.28:0/2236853588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff758103350 0x7ff75819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:50.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.131+0000 7ff757fff640 1 -- 172.21.15.28:0/2236853588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff758103350 msgr2=0x7ff75819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:50.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.131+0000 7ff757fff640 1 --2- 172.21.15.28:0/2236853588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff758103350 0x7ff75819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.131+0000 7ff757fff640 1 -- 172.21.15.28:0/2236853588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7581017a0 msgr2=0x7ff758197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:50.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.131+0000 7ff757fff640 1 --2- 172.21.15.28:0/2236853588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7581017a0 0x7ff758197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.131+0000 7ff757fff640 1 -- 172.21.15.28:0/2236853588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75819f660 con 0x7ff758102990 2024-04-25T01:32:50.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.131+0000 7ff757fff640 1 --2- 172.21.15.28:0/2236853588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff758102990 0x7ff758197f80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00d950 tx=0x7ff74c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:50.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.131+0000 7ff755ffb640 1 -- 172.21.15.28:0/2236853588 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c017070 con 0x7ff758102990 2024-04-25T01:32:50.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.131+0000 7ff75f16a640 1 -- 172.21.15.28:0/2236853588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff75819f950 con 0x7ff758102990 2024-04-25T01:32:50.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.131+0000 7ff755ffb640 1 -- 172.21.15.28:0/2236853588 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff74c009c30 con 0x7ff758102990 2024-04-25T01:32:50.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.131+0000 7ff755ffb640 1 -- 172.21.15.28:0/2236853588 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c00f6b0 con 0x7ff758102990 2024-04-25T01:32:50.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.131+0000 7ff75f16a640 1 -- 172.21.15.28:0/2236853588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff75819fea0 con 0x7ff758102990 2024-04-25T01:32:50.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.132+0000 7ff75f16a640 1 -- 172.21.15.28:0/2236853588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff758108ac0 con 0x7ff758102990 2024-04-25T01:32:50.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.133+0000 7ff755ffb640 1 -- 172.21.15.28:0/2236853588 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff74c0109d0 con 0x7ff758102990 2024-04-25T01:32:50.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.134+0000 7ff755ffb640 1 --2- 172.21.15.28:0/2236853588 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff7400a2b90 0x7ff7400a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:50.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.134+0000 7ff75cf06640 1 --2- 172.21.15.28:0/2236853588 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff7400a2b90 0x7ff7400a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:50.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.134+0000 7ff755ffb640 1 -- 172.21.15.28:0/2236853588 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7ff74c0cf0b0 con 0x7ff758102990 2024-04-25T01:32:50.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.134+0000 7ff75cf06640 1 --2- 172.21.15.28:0/2236853588 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff7400a2b90 0x7ff7400a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff748005d00 tx=0x7ff748005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:50.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.138+0000 7ff755ffb640 1 -- 172.21.15.28:0/2236853588 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff74c009de0 con 0x7ff758102990 2024-04-25T01:32:50.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.323+0000 7ff75f16a640 1 -- 172.21.15.28:0/2236853588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff75802d170 con 0x7ff758102990 2024-04-25T01:32:50.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.323+0000 7ff755ffb640 1 -- 172.21.15.28:0/2236853588 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7ff74c09bc50 con 0x7ff758102990 2024-04-25T01:32:50.301 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T01:32:50.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.325+0000 7ff75f16a640 1 -- 172.21.15.28:0/2236853588 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff7400a2b90 msgr2=0x7ff7400a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:50.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.325+0000 7ff75f16a640 1 --2- 172.21.15.28:0/2236853588 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff7400a2b90 0x7ff7400a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff748005d00 tx=0x7ff748005c50 comp rx=0 tx=0).stop 2024-04-25T01:32:50.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.325+0000 7ff75f16a640 1 -- 172.21.15.28:0/2236853588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff758102990 msgr2=0x7ff758197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:50.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.325+0000 7ff75f16a640 1 --2- 172.21.15.28:0/2236853588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff758102990 0x7ff758197f80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00d950 tx=0x7ff74c00de10 comp rx=0 tx=0).stop 2024-04-25T01:32:50.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.325+0000 7ff75f16a640 1 -- 172.21.15.28:0/2236853588 shutdown_connections 2024-04-25T01:32:50.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.325+0000 7ff75f16a640 1 --2- 172.21.15.28:0/2236853588 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff7400a2b90 0x7ff7400a5050 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.325+0000 7ff75f16a640 1 --2- 172.21.15.28:0/2236853588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff758103350 0x7ff75819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.325+0000 7ff75f16a640 1 --2- 172.21.15.28:0/2236853588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff758102990 0x7ff758197f80 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.325+0000 7ff75f16a640 1 --2- 172.21.15.28:0/2236853588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7581017a0 0x7ff758197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.325+0000 7ff75f16a640 1 -- 172.21.15.28:0/2236853588 >> 172.21.15.28:0/2236853588 conn(0x7ff7580fcd10 msgr2=0x7ff758105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:50.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.326+0000 7ff75f16a640 1 -- 172.21.15.28:0/2236853588 shutdown_connections 2024-04-25T01:32:50.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.326+0000 7ff75f16a640 1 -- 172.21.15.28:0/2236853588 wait complete. 2024-04-25T01:32:50.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:50 smithi028 ceph-mon[29390]: pgmap v367: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:50.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:50 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/4180049155' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:32:50.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:50 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1270727332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:50.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:50 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3980377315' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:50.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:32:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.444+0000 7f45e67b7640 1 -- 172.21.15.28:0/3795231223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e0102250 msgr2=0x7f45e0108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.444+0000 7f45e67b7640 1 --2- 172.21.15.28:0/3795231223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e0102250 0x7f45e0108ae0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f45d8009ca0 tx=0x7f45d802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.444+0000 7f45e67b7640 1 -- 172.21.15.28:0/3795231223 shutdown_connections 2024-04-25T01:32:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.444+0000 7f45e67b7640 1 --2- 172.21.15.28:0/3795231223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e0102250 0x7f45e0108ae0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.444+0000 7f45e67b7640 1 --2- 172.21.15.28:0/3795231223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45e0101890 0x7f45e0101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.444+0000 7f45e67b7640 1 --2- 172.21.15.28:0/3795231223 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45e01006a0 0x7f45e0100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.444+0000 7f45e67b7640 1 -- 172.21.15.28:0/3795231223 >> 172.21.15.28:0/3795231223 conn(0x7f45e00fbc30 msgr2=0x7f45e00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.444+0000 7f45e67b7640 1 -- 172.21.15.28:0/3795231223 shutdown_connections 2024-04-25T01:32:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.444+0000 7f45e67b7640 1 -- 172.21.15.28:0/3795231223 wait complete. 2024-04-25T01:32:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.445+0000 7f45e67b7640 1 Processor -- start 2024-04-25T01:32:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.445+0000 7f45e67b7640 1 -- start start 2024-04-25T01:32:50.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.445+0000 7f45e67b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e01006a0 0x7f45e0197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:50.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.445+0000 7f45e67b7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45e0101890 0x7f45e0198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:50.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.445+0000 7f45e67b7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45e0102250 0x7f45e019f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:50.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.445+0000 7f45e67b7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45e010c460 con 0x7f45e01006a0 2024-04-25T01:32:50.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.445+0000 7f45e67b7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f45e010c2e0 con 0x7f45e0101890 2024-04-25T01:32:50.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.445+0000 7f45e67b7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f45e010c5e0 con 0x7f45e0102250 2024-04-25T01:32:50.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.445+0000 7f45dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e01006a0 0x7f45e0197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:50.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.445+0000 7f45dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e01006a0 0x7f45e0197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41026/0 (socket says 172.21.15.28:41026) 2024-04-25T01:32:50.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.445+0000 7f45dffff640 1 -- 172.21.15.28:0/3801749895 learned_addr learned my addr 172.21.15.28:0/3801749895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:50.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.445+0000 7f45df7fe640 1 --2- 172.21.15.28:0/3801749895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45e0101890 0x7f45e0198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:50.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.446+0000 7f45e4d54640 1 --2- 172.21.15.28:0/3801749895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45e0102250 0x7f45e019f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:50.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.446+0000 7f45dffff640 1 -- 172.21.15.28:0/3801749895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45e0102250 msgr2=0x7f45e019f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:50.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.446+0000 7f45dffff640 1 --2- 172.21.15.28:0/3801749895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45e0102250 0x7f45e019f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.446+0000 7f45dffff640 1 -- 172.21.15.28:0/3801749895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45e0101890 msgr2=0x7f45e0198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.446+0000 7f45dffff640 1 --2- 172.21.15.28:0/3801749895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45e0101890 0x7f45e0198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.446+0000 7f45dffff640 1 -- 172.21.15.28:0/3801749895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45e019f7f0 con 0x7f45e01006a0 2024-04-25T01:32:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.446+0000 7f45dffff640 1 --2- 172.21.15.28:0/3801749895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e01006a0 0x7f45e0197bd0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f45d000b450 tx=0x7f45d000b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.446+0000 7f45dd7fa640 1 -- 172.21.15.28:0/3801749895 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d0015020 con 0x7f45e01006a0 2024-04-25T01:32:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.446+0000 7f45dd7fa640 1 -- 172.21.15.28:0/3801749895 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45d0011d10 con 0x7f45e01006a0 2024-04-25T01:32:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.446+0000 7f45dd7fa640 1 -- 172.21.15.28:0/3801749895 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d000f2e0 con 0x7f45e01006a0 2024-04-25T01:32:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.446+0000 7f45e67b7640 1 -- 172.21.15.28:0/3801749895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45e019fae0 con 0x7f45e01006a0 2024-04-25T01:32:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.446+0000 7f45e67b7640 1 -- 172.21.15.28:0/3801749895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45e01a0030 con 0x7f45e01006a0 2024-04-25T01:32:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.447+0000 7f45e67b7640 1 -- 172.21.15.28:0/3801749895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45e01a0260 con 0x7f45e01006a0 2024-04-25T01:32:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.448+0000 7f45dd7fa640 1 -- 172.21.15.28:0/3801749895 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f45d0018020 con 0x7f45e01006a0 2024-04-25T01:32:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.449+0000 7f45dd7fa640 1 --2- 172.21.15.28:0/3801749895 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f45cc0a2c10 0x7f45cc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.449+0000 7f45df7fe640 1 --2- 172.21.15.28:0/3801749895 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f45cc0a2c10 0x7f45cc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.449+0000 7f45dd7fa640 1 -- 172.21.15.28:0/3801749895 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f45d00cf150 con 0x7f45e01006a0 2024-04-25T01:32:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.449+0000 7f45df7fe640 1 --2- 172.21.15.28:0/3801749895 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f45cc0a2c10 0x7f45cc0a50d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f45e01990e0 tx=0x7f45c8009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:50.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.452+0000 7f45dd7fa640 1 -- 172.21.15.28:0/3801749895 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f45d001f020 con 0x7f45e01006a0 2024-04-25T01:32:50.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:50 smithi130 ceph-mon[32702]: pgmap v367: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:50.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:50 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/4180049155' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:32:50.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:50 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1270727332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:50.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:50 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3980377315' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:50.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:50 smithi032 ceph-mon[32461]: pgmap v367: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-25T01:32:50.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:50 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/4180049155' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:32:50.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:50 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1270727332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:50.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:50 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3980377315' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:50.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.637+0000 7f45e67b7640 1 -- 172.21.15.28:0/3801749895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f45e01a0430 con 0x7f45e01006a0 2024-04-25T01:32:50.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.637+0000 7f45dd7fa640 1 -- 172.21.15.28:0/3801749895 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f45d009bcf0 con 0x7f45e01006a0 2024-04-25T01:32:50.615 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:32:50.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.639+0000 7f45e67b7640 1 -- 172.21.15.28:0/3801749895 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f45cc0a2c10 msgr2=0x7f45cc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:50.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.639+0000 7f45e67b7640 1 --2- 172.21.15.28:0/3801749895 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f45cc0a2c10 0x7f45cc0a50d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f45e01990e0 tx=0x7f45c8009110 comp rx=0 tx=0).stop 2024-04-25T01:32:50.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.639+0000 7f45e67b7640 1 -- 172.21.15.28:0/3801749895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e01006a0 msgr2=0x7f45e0197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:50.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.639+0000 7f45e67b7640 1 --2- 172.21.15.28:0/3801749895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e01006a0 0x7f45e0197bd0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f45d000b450 tx=0x7f45d000b850 comp rx=0 tx=0).stop 2024-04-25T01:32:50.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.640+0000 7f45e67b7640 1 -- 172.21.15.28:0/3801749895 shutdown_connections 2024-04-25T01:32:50.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.640+0000 7f45e67b7640 1 --2- 172.21.15.28:0/3801749895 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f45cc0a2c10 0x7f45cc0a50d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.640+0000 7f45e67b7640 1 --2- 172.21.15.28:0/3801749895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45e0102250 0x7f45e019f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.640+0000 7f45e67b7640 1 --2- 172.21.15.28:0/3801749895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45e0101890 0x7f45e0198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.640+0000 7f45e67b7640 1 --2- 172.21.15.28:0/3801749895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e01006a0 0x7f45e0197bd0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.640+0000 7f45e67b7640 1 -- 172.21.15.28:0/3801749895 >> 172.21.15.28:0/3801749895 conn(0x7f45e00fbc30 msgr2=0x7f45e00fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:50.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.640+0000 7f45e67b7640 1 -- 172.21.15.28:0/3801749895 shutdown_connections 2024-04-25T01:32:50.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.640+0000 7f45e67b7640 1 -- 172.21.15.28:0/3801749895 wait complete. 2024-04-25T01:32:50.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:32:50.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.756+0000 7f1e5bbf9640 1 -- 172.21.15.28:0/801156140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e54103350 msgr2=0x7f1e54109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.756+0000 7f1e5bbf9640 1 --2- 172.21.15.28:0/801156140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e54103350 0x7f1e54109be0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c009ca0 tx=0x7f1e4c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.756+0000 7f1e5bbf9640 1 -- 172.21.15.28:0/801156140 shutdown_connections 2024-04-25T01:32:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.756+0000 7f1e5bbf9640 1 --2- 172.21.15.28:0/801156140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e54103350 0x7f1e54109be0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.756+0000 7f1e5bbf9640 1 --2- 172.21.15.28:0/801156140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e54102990 0x7f1e54102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.756+0000 7f1e5bbf9640 1 --2- 172.21.15.28:0/801156140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e541017a0 0x7f1e54101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.756+0000 7f1e5bbf9640 1 -- 172.21.15.28:0/801156140 >> 172.21.15.28:0/801156140 conn(0x7f1e540fcd10 msgr2=0x7f1e540ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.756+0000 7f1e5bbf9640 1 -- 172.21.15.28:0/801156140 shutdown_connections 2024-04-25T01:32:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.756+0000 7f1e5bbf9640 1 -- 172.21.15.28:0/801156140 wait complete. 2024-04-25T01:32:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.756+0000 7f1e5bbf9640 1 Processor -- start 2024-04-25T01:32:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e5bbf9640 1 -- start start 2024-04-25T01:32:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e5bbf9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e541017a0 0x7f1e54197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e5bbf9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e54102990 0x7f1e54197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e5bbf9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e54103350 0x7f1e5419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e5bbf9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5410d4d0 con 0x7f1e541017a0 2024-04-25T01:32:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e5bbf9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5410d350 con 0x7f1e54102990 2024-04-25T01:32:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e5bbf9640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5410d650 con 0x7f1e54103350 2024-04-25T01:32:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e59995640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e541017a0 0x7f1e54197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e59995640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e541017a0 0x7f1e54197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41038/0 (socket says 172.21.15.28:41038) 2024-04-25T01:32:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e59995640 1 -- 172.21.15.28:0/1631222602 learned_addr learned my addr 172.21.15.28:0/1631222602 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e59194640 1 --2- 172.21.15.28:0/1631222602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e54102990 0x7f1e54197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e5a196640 1 --2- 172.21.15.28:0/1631222602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e54103350 0x7f1e5419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e59995640 1 -- 172.21.15.28:0/1631222602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e54103350 msgr2=0x7f1e5419f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e59995640 1 --2- 172.21.15.28:0/1631222602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e54103350 0x7f1e5419f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e59995640 1 -- 172.21.15.28:0/1631222602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e54102990 msgr2=0x7f1e54197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e59995640 1 --2- 172.21.15.28:0/1631222602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e54102990 0x7f1e54197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e59995640 1 -- 172.21.15.28:0/1631222602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5419f6d0 con 0x7f1e541017a0 2024-04-25T01:32:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e5a196640 1 --2- 172.21.15.28:0/1631222602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e54103350 0x7f1e5419f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:50.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e59995640 1 --2- 172.21.15.28:0/1631222602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e541017a0 0x7f1e54197ab0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1e5000d770 tx=0x7f1e5000db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:50.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.757+0000 7f1e4affd640 1 -- 172.21.15.28:0/1631222602 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e50017070 con 0x7f1e541017a0 2024-04-25T01:32:50.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.758+0000 7f1e4affd640 1 -- 172.21.15.28:0/1631222602 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e50009d00 con 0x7f1e541017a0 2024-04-25T01:32:50.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.758+0000 7f1e5bbf9640 1 -- 172.21.15.28:0/1631222602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e5419f9c0 con 0x7f1e541017a0 2024-04-25T01:32:50.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.758+0000 7f1e4affd640 1 -- 172.21.15.28:0/1631222602 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e50003d30 con 0x7f1e541017a0 2024-04-25T01:32:50.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.758+0000 7f1e5bbf9640 1 -- 172.21.15.28:0/1631222602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e5419ffe0 con 0x7f1e541017a0 2024-04-25T01:32:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.759+0000 7f1e5bbf9640 1 -- 172.21.15.28:0/1631222602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e54108a60 con 0x7f1e541017a0 2024-04-25T01:32:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.760+0000 7f1e4affd640 1 -- 172.21.15.28:0/1631222602 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1e50009820 con 0x7f1e541017a0 2024-04-25T01:32:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.760+0000 7f1e4affd640 1 --2- 172.21.15.28:0/1631222602 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1e400a2c10 0x7f1e400a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.761+0000 7f1e59194640 1 --2- 172.21.15.28:0/1631222602 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1e400a2c10 0x7f1e400a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.761+0000 7f1e4affd640 1 -- 172.21.15.28:0/1631222602 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f1e500cecc0 con 0x7f1e541017a0 2024-04-25T01:32:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.761+0000 7f1e59194640 1 --2- 172.21.15.28:0/1631222602 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1e400a2c10 0x7f1e400a50d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1e54198fc0 tx=0x7f1e44009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:50.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.765+0000 7f1e4affd640 1 -- 172.21.15.28:0/1631222602 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1e5009b8c0 con 0x7f1e541017a0 2024-04-25T01:32:50.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.951+0000 7f1e5bbf9640 1 -- 172.21.15.28:0/1631222602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1e54062340 con 0x7f1e541017a0 2024-04-25T01:32:50.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.951+0000 7f1e4affd640 1 -- 172.21.15.28:0/1631222602 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f1e5009b1c0 con 0x7f1e541017a0 2024-04-25T01:32:50.929 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T01:32:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.953+0000 7f1e5bbf9640 1 -- 172.21.15.28:0/1631222602 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1e400a2c10 msgr2=0x7f1e400a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.953+0000 7f1e5bbf9640 1 --2- 172.21.15.28:0/1631222602 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1e400a2c10 0x7f1e400a50d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1e54198fc0 tx=0x7f1e44009110 comp rx=0 tx=0).stop 2024-04-25T01:32:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.953+0000 7f1e5bbf9640 1 -- 172.21.15.28:0/1631222602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e541017a0 msgr2=0x7f1e54197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.953+0000 7f1e5bbf9640 1 --2- 172.21.15.28:0/1631222602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e541017a0 0x7f1e54197ab0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1e5000d770 tx=0x7f1e5000db70 comp rx=0 tx=0).stop 2024-04-25T01:32:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.953+0000 7f1e5bbf9640 1 -- 172.21.15.28:0/1631222602 shutdown_connections 2024-04-25T01:32:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.953+0000 7f1e5bbf9640 1 --2- 172.21.15.28:0/1631222602 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1e400a2c10 0x7f1e400a50d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.953+0000 7f1e5bbf9640 1 --2- 172.21.15.28:0/1631222602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e54103350 0x7f1e5419f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.953+0000 7f1e5bbf9640 1 --2- 172.21.15.28:0/1631222602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e54102990 0x7f1e54197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.953+0000 7f1e5bbf9640 1 --2- 172.21.15.28:0/1631222602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e541017a0 0x7f1e54197ab0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.953+0000 7f1e5bbf9640 1 -- 172.21.15.28:0/1631222602 >> 172.21.15.28:0/1631222602 conn(0x7f1e540fcd10 msgr2=0x7f1e54105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.953+0000 7f1e5bbf9640 1 -- 172.21.15.28:0/1631222602 shutdown_connections 2024-04-25T01:32:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:50.953+0000 7f1e5bbf9640 1 -- 172.21.15.28:0/1631222602 wait complete. 2024-04-25T01:32:50.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:32:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.069+0000 7fdfb7d09640 1 -- 172.21.15.28:0/3109801497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfb0102220 msgr2=0x7fdfb0104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.069+0000 7fdfb7d09640 1 --2- 172.21.15.28:0/3109801497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfb0102220 0x7fdfb0104610 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fdfac009900 tx=0x7fdfac02a410 comp rx=0 tx=0).stop 2024-04-25T01:32:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.069+0000 7fdfb7d09640 1 -- 172.21.15.28:0/3109801497 shutdown_connections 2024-04-25T01:32:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.069+0000 7fdfb7d09640 1 --2- 172.21.15.28:0/3109801497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb0107480 0x7fdfb01098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.069+0000 7fdfb7d09640 1 --2- 172.21.15.28:0/3109801497 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdfb0104b50 0x7fdfb0106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.069+0000 7fdfb7d09640 1 --2- 172.21.15.28:0/3109801497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfb0102220 0x7fdfb0104610 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.069+0000 7fdfb7d09640 1 -- 172.21.15.28:0/3109801497 >> 172.21.15.28:0/3109801497 conn(0x7fdfb00fbc30 msgr2=0x7fdfb00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.069+0000 7fdfb7d09640 1 -- 172.21.15.28:0/3109801497 shutdown_connections 2024-04-25T01:32:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.069+0000 7fdfb7d09640 1 -- 172.21.15.28:0/3109801497 wait complete. 2024-04-25T01:32:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.069+0000 7fdfb7d09640 1 Processor -- start 2024-04-25T01:32:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb7d09640 1 -- start start 2024-04-25T01:32:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb7d09640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb0102220 0x7fdfb0197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb7d09640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfb0104b50 0x7fdfb0198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb7d09640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdfb0107480 0x7fdfb019f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb7d09640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb010d560 con 0x7fdfb0104b50 2024-04-25T01:32:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb7d09640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb010d3e0 con 0x7fdfb0102220 2024-04-25T01:32:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb7d09640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb010d6e0 con 0x7fdfb0107480 2024-04-25T01:32:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb52a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfb0104b50 0x7fdfb0198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb52a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfb0104b50 0x7fdfb0198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41058/0 (socket says 172.21.15.28:41058) 2024-04-25T01:32:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb52a4640 1 -- 172.21.15.28:0/840490266 learned_addr learned my addr 172.21.15.28:0/840490266 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb5aa5640 1 --2- 172.21.15.28:0/840490266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb0102220 0x7fdfb0197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb62a6640 1 --2- 172.21.15.28:0/840490266 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdfb0107480 0x7fdfb019f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb52a4640 1 -- 172.21.15.28:0/840490266 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdfb0107480 msgr2=0x7fdfb019f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb52a4640 1 --2- 172.21.15.28:0/840490266 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdfb0107480 0x7fdfb019f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb52a4640 1 -- 172.21.15.28:0/840490266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb0102220 msgr2=0x7fdfb0197c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb52a4640 1 --2- 172.21.15.28:0/840490266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb0102220 0x7fdfb0197c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb52a4640 1 -- 172.21.15.28:0/840490266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb019f860 con 0x7fdfb0104b50 2024-04-25T01:32:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb5aa5640 1 --2- 172.21.15.28:0/840490266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb0102220 0x7fdfb0197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb62a6640 1 --2- 172.21.15.28:0/840490266 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdfb0107480 0x7fdfb019f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.070+0000 7fdfb52a4640 1 --2- 172.21.15.28:0/840490266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfb0104b50 0x7fdfb0198190 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000b450 tx=0x7fdfa000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.071+0000 7fdfa6ffd640 1 -- 172.21.15.28:0/840490266 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0015020 con 0x7fdfb0104b50 2024-04-25T01:32:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.071+0000 7fdfb7d09640 1 -- 172.21.15.28:0/840490266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfb019fb50 con 0x7fdfb0104b50 2024-04-25T01:32:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.071+0000 7fdfa6ffd640 1 -- 172.21.15.28:0/840490266 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfa0011d70 con 0x7fdfb0104b50 2024-04-25T01:32:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.071+0000 7fdfa6ffd640 1 -- 172.21.15.28:0/840490266 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa000f340 con 0x7fdfb0104b50 2024-04-25T01:32:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.071+0000 7fdfb7d09640 1 -- 172.21.15.28:0/840490266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfb01a00a0 con 0x7fdfb0104b50 2024-04-25T01:32:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.072+0000 7fdfb7d09640 1 -- 172.21.15.28:0/840490266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfb00fd720 con 0x7fdfb0104b50 2024-04-25T01:32:51.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.073+0000 7fdfa6ffd640 1 -- 172.21.15.28:0/840490266 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fdfa0018020 con 0x7fdfb0104b50 2024-04-25T01:32:51.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.073+0000 7fdfa6ffd640 1 --2- 172.21.15.28:0/840490266 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fdf9c0a2b90 0x7fdf9c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:51.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.074+0000 7fdfb5aa5640 1 --2- 172.21.15.28:0/840490266 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fdf9c0a2b90 0x7fdf9c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:51.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.074+0000 7fdfa6ffd640 1 -- 172.21.15.28:0/840490266 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fdfa00cf500 con 0x7fdfb0104b50 2024-04-25T01:32:51.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.074+0000 7fdfb5aa5640 1 --2- 172.21.15.28:0/840490266 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fdf9c0a2b90 0x7fdf9c0a5050 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdfac006010 tx=0x7fdfac000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:51.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.077+0000 7fdfa6ffd640 1 -- 172.21.15.28:0/840490266 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdfa001f020 con 0x7fdfb0104b50 2024-04-25T01:32:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.265+0000 7fdfb7d09640 1 -- 172.21.15.28:0/840490266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fdfb002d3b0 con 0x7fdfb0104b50 2024-04-25T01:32:51.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.265+0000 7fdfa6ffd640 1 -- 172.21.15.28:0/840490266 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fdfa000d040 con 0x7fdfb0104b50 2024-04-25T01:32:51.243 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:32:51.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.267+0000 7fdfb7d09640 1 -- 172.21.15.28:0/840490266 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fdf9c0a2b90 msgr2=0x7fdf9c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:51.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.267+0000 7fdfb7d09640 1 --2- 172.21.15.28:0/840490266 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fdf9c0a2b90 0x7fdf9c0a5050 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdfac006010 tx=0x7fdfac000bc0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.267+0000 7fdfb7d09640 1 -- 172.21.15.28:0/840490266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfb0104b50 msgr2=0x7fdfb0198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:51.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.267+0000 7fdfb7d09640 1 --2- 172.21.15.28:0/840490266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfb0104b50 0x7fdfb0198190 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000b450 tx=0x7fdfa000b910 comp rx=0 tx=0).stop 2024-04-25T01:32:51.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.268+0000 7fdfb7d09640 1 -- 172.21.15.28:0/840490266 shutdown_connections 2024-04-25T01:32:51.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.268+0000 7fdfb7d09640 1 --2- 172.21.15.28:0/840490266 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fdf9c0a2b90 0x7fdf9c0a5050 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.268+0000 7fdfb7d09640 1 --2- 172.21.15.28:0/840490266 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdfb0107480 0x7fdfb019f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.268+0000 7fdfb7d09640 1 --2- 172.21.15.28:0/840490266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfb0104b50 0x7fdfb0198190 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.268+0000 7fdfb7d09640 1 --2- 172.21.15.28:0/840490266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb0102220 0x7fdfb0197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.268+0000 7fdfb7d09640 1 -- 172.21.15.28:0/840490266 >> 172.21.15.28:0/840490266 conn(0x7fdfb00fbc30 msgr2=0x7fdfb0102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.268+0000 7fdfb7d09640 1 -- 172.21.15.28:0/840490266 shutdown_connections 2024-04-25T01:32:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.268+0000 7fdfb7d09640 1 -- 172.21.15.28:0/840490266 wait complete. 2024-04-25T01:32:51.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:51 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2236853588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:32:51.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:51 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3801749895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:32:51.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:51 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1631222602' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:32:51.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:32:51.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.380+0000 7fa0923ed640 1 -- 172.21.15.28:0/1946536092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c107520 msgr2=0x7fa08c109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.380+0000 7fa0923ed640 1 --2- 172.21.15.28:0/1946536092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c107520 0x7fa08c109910 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa084009ca0 tx=0x7fa08402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.380+0000 7fa0923ed640 1 -- 172.21.15.28:0/1946536092 shutdown_connections 2024-04-25T01:32:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.380+0000 7fa0923ed640 1 --2- 172.21.15.28:0/1946536092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c107520 0x7fa08c109910 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.380+0000 7fa0923ed640 1 --2- 172.21.15.28:0/1946536092 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa08c104bf0 0x7fa08c106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.380+0000 7fa0923ed640 1 --2- 172.21.15.28:0/1946536092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa08c1022c0 0x7fa08c1046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.380+0000 7fa0923ed640 1 -- 172.21.15.28:0/1946536092 >> 172.21.15.28:0/1946536092 conn(0x7fa08c0fbc30 msgr2=0x7fa08c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.380+0000 7fa0923ed640 1 -- 172.21.15.28:0/1946536092 shutdown_connections 2024-04-25T01:32:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.380+0000 7fa0923ed640 1 -- 172.21.15.28:0/1946536092 wait complete. 2024-04-25T01:32:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.381+0000 7fa0923ed640 1 Processor -- start 2024-04-25T01:32:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.381+0000 7fa0923ed640 1 -- start start 2024-04-25T01:32:51.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.381+0000 7fa0923ed640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa08c1022c0 0x7fa08c101c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.381+0000 7fa0923ed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c104bf0 0x7fa08c100270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.381+0000 7fa0923ed640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa08c107520 0x7fa08c1007b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.381+0000 7fa0923ed640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa08c10d490 con 0x7fa08c104bf0 2024-04-25T01:32:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.381+0000 7fa0923ed640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa08c10d310 con 0x7fa08c1022c0 2024-04-25T01:32:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.381+0000 7fa0923ed640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa08c10d610 con 0x7fa08c107520 2024-04-25T01:32:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.382+0000 7fa08b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c104bf0 0x7fa08c100270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.382+0000 7fa08b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c104bf0 0x7fa08c100270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41074/0 (socket says 172.21.15.28:41074) 2024-04-25T01:32:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.382+0000 7fa08b7fe640 1 -- 172.21.15.28:0/1392132689 learned_addr learned my addr 172.21.15.28:0/1392132689 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.382+0000 7fa08bfff640 1 --2- 172.21.15.28:0/1392132689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa08c1022c0 0x7fa08c101c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.382+0000 7fa09098a640 1 --2- 172.21.15.28:0/1392132689 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa08c107520 0x7fa08c1007b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.382+0000 7fa08b7fe640 1 -- 172.21.15.28:0/1392132689 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa08c107520 msgr2=0x7fa08c1007b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.382+0000 7fa08b7fe640 1 --2- 172.21.15.28:0/1392132689 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa08c107520 0x7fa08c1007b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.382+0000 7fa08b7fe640 1 -- 172.21.15.28:0/1392132689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa08c1022c0 msgr2=0x7fa08c101c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.382+0000 7fa08b7fe640 1 --2- 172.21.15.28:0/1392132689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa08c1022c0 0x7fa08c101c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.382+0000 7fa08b7fe640 1 -- 172.21.15.28:0/1392132689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa08c101040 con 0x7fa08c104bf0 2024-04-25T01:32:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.382+0000 7fa08b7fe640 1 --2- 172.21.15.28:0/1392132689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c104bf0 0x7fa08c100270 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009d10 tx=0x7fa07c013f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.382+0000 7fa0897fa640 1 -- 172.21.15.28:0/1392132689 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c014df0 con 0x7fa08c104bf0 2024-04-25T01:32:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.383+0000 7fa0923ed640 1 -- 172.21.15.28:0/1392132689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa08c197720 con 0x7fa08c104bf0 2024-04-25T01:32:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.383+0000 7fa0897fa640 1 -- 172.21.15.28:0/1392132689 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa07c018070 con 0x7fa08c104bf0 2024-04-25T01:32:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.383+0000 7fa0897fa640 1 -- 172.21.15.28:0/1392132689 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c01c670 con 0x7fa08c104bf0 2024-04-25T01:32:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.383+0000 7fa0923ed640 1 -- 172.21.15.28:0/1392132689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa08c197cf0 con 0x7fa08c104bf0 2024-04-25T01:32:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.384+0000 7fa0923ed640 1 -- 172.21.15.28:0/1392132689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa08c0fd720 con 0x7fa08c104bf0 2024-04-25T01:32:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.385+0000 7fa0897fa640 1 -- 172.21.15.28:0/1392132689 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa07c01a030 con 0x7fa08c104bf0 2024-04-25T01:32:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.386+0000 7fa0897fa640 1 --2- 172.21.15.28:0/1392132689 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa0780a2b90 0x7fa0780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.386+0000 7fa08bfff640 1 --2- 172.21.15.28:0/1392132689 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa0780a2b90 0x7fa0780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.386+0000 7fa0897fa640 1 -- 172.21.15.28:0/1392132689 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fa07c0d65c0 con 0x7fa08c104bf0 2024-04-25T01:32:51.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.386+0000 7fa08bfff640 1 --2- 172.21.15.28:0/1392132689 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa0780a2b90 0x7fa0780a5050 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa0740058c0 tx=0x7fa07400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:51.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.390+0000 7fa0897fa640 1 -- 172.21.15.28:0/1392132689 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa07c0a2c00 con 0x7fa08c104bf0 2024-04-25T01:32:51.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:51 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2236853588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:32:51.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:51 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3801749895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:32:51.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:51 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1631222602' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:32:51.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:51 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2236853588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:32:51.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:51 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3801749895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:32:51.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:51 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1631222602' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:32:51.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.612+0000 7fa0923ed640 1 -- 172.21.15.28:0/1392132689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa08c02d170 con 0x7fa08c104bf0 2024-04-25T01:32:51.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.613+0000 7fa0897fa640 1 -- 172.21.15.28:0/1392132689 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11923 (secure 0 0 0) 0x7fa07c0a6f00 con 0x7fa08c104bf0 2024-04-25T01:32:51.590 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:32:51.590 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:31:01.802718+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24454,24442,24512,24478,24473]},"id":1}]} 2024-04-25T01:32:51.591 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-04-25T01:32:51.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.615+0000 7fa0923ed640 1 -- 172.21.15.28:0/1392132689 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa0780a2b90 msgr2=0x7fa0780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:51.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.615+0000 7fa0923ed640 1 --2- 172.21.15.28:0/1392132689 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa0780a2b90 0x7fa0780a5050 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa0740058c0 tx=0x7fa07400a2d0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.615+0000 7fa0923ed640 1 -- 172.21.15.28:0/1392132689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c104bf0 msgr2=0x7fa08c100270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:51.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.615+0000 7fa0923ed640 1 --2- 172.21.15.28:0/1392132689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c104bf0 0x7fa08c100270 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009d10 tx=0x7fa07c013f70 comp rx=0 tx=0).stop 2024-04-25T01:32:51.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.615+0000 7fa0923ed640 1 -- 172.21.15.28:0/1392132689 shutdown_connections 2024-04-25T01:32:51.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.615+0000 7fa0923ed640 1 --2- 172.21.15.28:0/1392132689 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa0780a2b90 0x7fa0780a5050 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.615+0000 7fa0923ed640 1 --2- 172.21.15.28:0/1392132689 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa08c107520 0x7fa08c1007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.615+0000 7fa0923ed640 1 --2- 172.21.15.28:0/1392132689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c104bf0 0x7fa08c100270 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.615+0000 7fa0923ed640 1 --2- 172.21.15.28:0/1392132689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa08c1022c0 0x7fa08c101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.615+0000 7fa0923ed640 1 -- 172.21.15.28:0/1392132689 >> 172.21.15.28:0/1392132689 conn(0x7fa08c0fbc30 msgr2=0x7fa08c102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:51.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.615+0000 7fa0923ed640 1 -- 172.21.15.28:0/1392132689 shutdown_connections 2024-04-25T01:32:51.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.615+0000 7fa0923ed640 1 -- 172.21.15.28:0/1392132689 wait complete. 2024-04-25T01:32:51.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:32:51.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.728+0000 7f9c8dcba640 1 -- 172.21.15.28:0/1941729661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c88103360 msgr2=0x7f9c88109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:51.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.728+0000 7f9c8dcba640 1 --2- 172.21.15.28:0/1941729661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c88103360 0x7f9c88109bf0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f9c80009ca0 tx=0x7f9c8002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.728+0000 7f9c8dcba640 1 -- 172.21.15.28:0/1941729661 shutdown_connections 2024-04-25T01:32:51.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.728+0000 7f9c8dcba640 1 --2- 172.21.15.28:0/1941729661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c88103360 0x7f9c88109bf0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.728+0000 7f9c8dcba640 1 --2- 172.21.15.28:0/1941729661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c881029a0 0x7f9c88102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.728+0000 7f9c8dcba640 1 --2- 172.21.15.28:0/1941729661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c881017b0 0x7f9c88101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.728+0000 7f9c8dcba640 1 -- 172.21.15.28:0/1941729661 >> 172.21.15.28:0/1941729661 conn(0x7f9c880fcd40 msgr2=0x7f9c880ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:51.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.728+0000 7f9c8dcba640 1 -- 172.21.15.28:0/1941729661 shutdown_connections 2024-04-25T01:32:51.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.728+0000 7f9c8dcba640 1 -- 172.21.15.28:0/1941729661 wait complete. 2024-04-25T01:32:51.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.728+0000 7f9c8dcba640 1 Processor -- start 2024-04-25T01:32:51.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.728+0000 7f9c8dcba640 1 -- start start 2024-04-25T01:32:51.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c8dcba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c881017b0 0x7f9c88197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:51.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c8dcba640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c881029a0 0x7f9c88197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:51.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c8dcba640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c88103360 0x7f9c8819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c8dcba640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8810d410 con 0x7f9c881017b0 2024-04-25T01:32:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c8dcba640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8810d290 con 0x7f9c881029a0 2024-04-25T01:32:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c8dcba640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8810d590 con 0x7f9c88103360 2024-04-25T01:32:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c881017b0 0x7f9c88197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c881017b0 0x7f9c88197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41096/0 (socket says 172.21.15.28:41096) 2024-04-25T01:32:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c877fe640 1 -- 172.21.15.28:0/1822658888 learned_addr learned my addr 172.21.15.28:0/1822658888 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c87fff640 1 --2- 172.21.15.28:0/1822658888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c88103360 0x7f9c8819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c877fe640 1 -- 172.21.15.28:0/1822658888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c88103360 msgr2=0x7f9c8819efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c877fe640 1 --2- 172.21.15.28:0/1822658888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c88103360 0x7f9c8819efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c877fe640 1 -- 172.21.15.28:0/1822658888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c881029a0 msgr2=0x7f9c88197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c877fe640 1 --2- 172.21.15.28:0/1822658888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c881029a0 0x7f9c88197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c877fe640 1 -- 172.21.15.28:0/1822658888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c8819f640 con 0x7f9c881017b0 2024-04-25T01:32:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c87fff640 1 --2- 172.21.15.28:0/1822658888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c88103360 0x7f9c8819efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:51.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c877fe640 1 --2- 172.21.15.28:0/1822658888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c881017b0 0x7f9c88197a20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9c7800d770 tx=0x7f9c7800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c84ff9640 1 -- 172.21.15.28:0/1822658888 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c78017070 con 0x7f9c881017b0 2024-04-25T01:32:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c8dcba640 1 -- 172.21.15.28:0/1822658888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c8819f930 con 0x7f9c881017b0 2024-04-25T01:32:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c84ff9640 1 -- 172.21.15.28:0/1822658888 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c78009c30 con 0x7f9c881017b0 2024-04-25T01:32:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c84ff9640 1 -- 172.21.15.28:0/1822658888 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c78003dd0 con 0x7f9c881017b0 2024-04-25T01:32:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.729+0000 7f9c8dcba640 1 -- 172.21.15.28:0/1822658888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c8819fe80 con 0x7f9c881017b0 2024-04-25T01:32:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.730+0000 7f9c8dcba640 1 -- 172.21.15.28:0/1822658888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c88108a00 con 0x7f9c881017b0 2024-04-25T01:32:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.731+0000 7f9c84ff9640 1 -- 172.21.15.28:0/1822658888 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9c78003f70 con 0x7f9c881017b0 2024-04-25T01:32:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.732+0000 7f9c84ff9640 1 --2- 172.21.15.28:0/1822658888 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9c700a2b90 0x7f9c700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.732+0000 7f9c86ffd640 1 --2- 172.21.15.28:0/1822658888 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9c700a2b90 0x7f9c700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:51.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.732+0000 7f9c84ff9640 1 -- 172.21.15.28:0/1822658888 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f9c780cee70 con 0x7f9c881017b0 2024-04-25T01:32:51.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.733+0000 7f9c86ffd640 1 --2- 172.21.15.28:0/1822658888 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9c700a2b90 0x7f9c700a5050 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c005fd0 tx=0x7f9c7c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:51.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.737+0000 7f9c84ff9640 1 -- 172.21.15.28:0/1822658888 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9c7809ba10 con 0x7f9c881017b0 2024-04-25T01:32:51.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.923+0000 7f9c8dcba640 1 -- 172.21.15.28:0/1822658888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9c8802d170 con 0x7f9c881017b0 2024-04-25T01:32:51.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.924+0000 7f9c84ff9640 1 -- 172.21.15.28:0/1822658888 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24564 (secure 0 0 0) 0x7f9c78009de0 con 0x7f9c881017b0 2024-04-25T01:32:51.902 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:32:51.902 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:31:21.857121+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:32:51.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.926+0000 7f9c8dcba640 1 -- 172.21.15.28:0/1822658888 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9c700a2b90 msgr2=0x7f9c700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:51.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.926+0000 7f9c8dcba640 1 --2- 172.21.15.28:0/1822658888 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9c700a2b90 0x7f9c700a5050 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c005fd0 tx=0x7f9c7c005850 comp rx=0 tx=0).stop 2024-04-25T01:32:51.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.926+0000 7f9c8dcba640 1 -- 172.21.15.28:0/1822658888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c881017b0 msgr2=0x7f9c88197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:51.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.926+0000 7f9c8dcba640 1 --2- 172.21.15.28:0/1822658888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c881017b0 0x7f9c88197a20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9c7800d770 tx=0x7f9c7800dc30 comp rx=0 tx=0).stop 2024-04-25T01:32:51.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.926+0000 7f9c8dcba640 1 -- 172.21.15.28:0/1822658888 shutdown_connections 2024-04-25T01:32:51.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.926+0000 7f9c8dcba640 1 --2- 172.21.15.28:0/1822658888 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9c700a2b90 0x7f9c700a5050 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.926+0000 7f9c8dcba640 1 --2- 172.21.15.28:0/1822658888 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c88103360 0x7f9c8819efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.926+0000 7f9c8dcba640 1 --2- 172.21.15.28:0/1822658888 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c881029a0 0x7f9c88197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.926+0000 7f9c8dcba640 1 --2- 172.21.15.28:0/1822658888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c881017b0 0x7f9c88197a20 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:51.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.926+0000 7f9c8dcba640 1 -- 172.21.15.28:0/1822658888 >> 172.21.15.28:0/1822658888 conn(0x7f9c880fcd40 msgr2=0x7f9c88105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:51.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.926+0000 7f9c8dcba640 1 -- 172.21.15.28:0/1822658888 shutdown_connections 2024-04-25T01:32:51.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:51.927+0000 7f9c8dcba640 1 -- 172.21.15.28:0/1822658888 wait complete. 2024-04-25T01:32:51.916 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T01:30:33.696870+0000', 'modified': '2024-04-25T01:31:01.802718+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24512, 'mds_1': 24478, 'mds_2': 24473, 'mds_3': 24454, 'mds_4': 24442}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14643': {'gid': 14643, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6839/3206225061', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 3206225061}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 3206225061}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14655': {'gid': 14655, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6841/3666045393', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6840', 'nonce': 3666045393}, {'type': 'v1', 'addr': '172.21.15.28:6841', 'nonce': 3666045393}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24401': {'gid': 24401, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.32:6833/983600975', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6832', 'nonce': 983600975}, {'type': 'v1', 'addr': '172.21.15.32:6833', 'nonce': 983600975}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24412': {'gid': 24412, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.130:6833/1080169582', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6832', 'nonce': 1080169582}, {'type': 'v1', 'addr': '172.21.15.130:6833', 'nonce': 1080169582}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24440': {'gid': 24440, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.32:6835/356243399', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6834', 'nonce': 356243399}, {'type': 'v1', 'addr': '172.21.15.32:6835', 'nonce': 356243399}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.130:6835/348635896', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6834', 'nonce': 348635896}, {'type': 'v1', 'addr': '172.21.15.130:6835', 'nonce': 348635896}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.130:6837/211524954', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6836', 'nonce': 211524954}, {'type': 'v1', 'addr': '172.21.15.130:6837', 'nonce': 211524954}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.32:6837/2771586449', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 2771586449}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 2771586449}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.130:6839/463478317', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.130:6838', 'nonce': 463478317}, {'type': 'v1', 'addr': '172.21.15.130:6839', 'nonce': 463478317}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24512': {'gid': 24512, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.32:6839/2835949714', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 2835949714}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 2835949714}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24512, 'qdb_cluster': [24454, 24442, 24512, 24478, 24473]} 2024-04-25T01:32:51.916 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-25T01:32:51.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-25T01:32:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.042+0000 7f9a61c12640 1 -- 172.21.15.28:0/3516536045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c104c10 msgr2=0x7f9a5c107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.042+0000 7f9a61c12640 1 --2- 172.21.15.28:0/3516536045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c104c10 0x7f9a5c107000 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9a50009900 tx=0x7f9a5002a410 comp rx=0 tx=0).stop 2024-04-25T01:32:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.042+0000 7f9a61c12640 1 -- 172.21.15.28:0/3516536045 shutdown_connections 2024-04-25T01:32:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.042+0000 7f9a61c12640 1 --2- 172.21.15.28:0/3516536045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a5c107540 0x7f9a5c109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.042+0000 7f9a61c12640 1 --2- 172.21.15.28:0/3516536045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c104c10 0x7f9a5c107000 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.042+0000 7f9a61c12640 1 --2- 172.21.15.28:0/3516536045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a5c1022e0 0x7f9a5c1046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.042+0000 7f9a61c12640 1 -- 172.21.15.28:0/3516536045 >> 172.21.15.28:0/3516536045 conn(0x7f9a5c0fbc30 msgr2=0x7f9a5c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.042+0000 7f9a61c12640 1 -- 172.21.15.28:0/3516536045 shutdown_connections 2024-04-25T01:32:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.042+0000 7f9a61c12640 1 -- 172.21.15.28:0/3516536045 wait complete. 2024-04-25T01:32:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.043+0000 7f9a61c12640 1 Processor -- start 2024-04-25T01:32:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.043+0000 7f9a61c12640 1 -- start start 2024-04-25T01:32:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.043+0000 7f9a61c12640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a5c1022e0 0x7f9a5c197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.043+0000 7f9a61c12640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c104c10 0x7f9a5c198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.043+0000 7f9a61c12640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a5c107540 0x7f9a5c19f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.043+0000 7f9a61c12640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c10d420 con 0x7f9a5c104c10 2024-04-25T01:32:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.043+0000 7f9a61c12640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c10d2a0 con 0x7f9a5c107540 2024-04-25T01:32:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.043+0000 7f9a61c12640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c10d5a0 con 0x7f9a5c1022e0 2024-04-25T01:32:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.043+0000 7f9a5affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c104c10 0x7f9a5c198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.043+0000 7f9a5affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c104c10 0x7f9a5c198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41108/0 (socket says 172.21.15.28:41108) 2024-04-25T01:32:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.043+0000 7f9a5affd640 1 -- 172.21.15.28:0/4254195863 learned_addr learned my addr 172.21.15.28:0/4254195863 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.043+0000 7f9a5b7fe640 1 --2- 172.21.15.28:0/4254195863 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a5c1022e0 0x7f9a5c197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.043+0000 7f9a5bfff640 1 --2- 172.21.15.28:0/4254195863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a5c107540 0x7f9a5c19f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.043+0000 7f9a5affd640 1 -- 172.21.15.28:0/4254195863 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a5c1022e0 msgr2=0x7f9a5c197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.043+0000 7f9a5affd640 1 --2- 172.21.15.28:0/4254195863 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a5c1022e0 0x7f9a5c197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.043+0000 7f9a5affd640 1 -- 172.21.15.28:0/4254195863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a5c107540 msgr2=0x7f9a5c19f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.043+0000 7f9a5affd640 1 --2- 172.21.15.28:0/4254195863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a5c107540 0x7f9a5c19f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.043+0000 7f9a5affd640 1 -- 172.21.15.28:0/4254195863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a5c19f770 con 0x7f9a5c104c10 2024-04-25T01:32:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.043+0000 7f9a5bfff640 1 --2- 172.21.15.28:0/4254195863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a5c107540 0x7f9a5c19f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.044+0000 7f9a5affd640 1 --2- 172.21.15.28:0/4254195863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c104c10 0x7f9a5c198090 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9a50031610 tx=0x7f9a500043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.044+0000 7f9a58ff9640 1 -- 172.21.15.28:0/4254195863 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5003f030 con 0x7f9a5c104c10 2024-04-25T01:32:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.044+0000 7f9a58ff9640 1 -- 172.21.15.28:0/4254195863 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a50003ce0 con 0x7f9a5c104c10 2024-04-25T01:32:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.044+0000 7f9a61c12640 1 -- 172.21.15.28:0/4254195863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a5c19fa00 con 0x7f9a5c104c10 2024-04-25T01:32:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.044+0000 7f9a58ff9640 1 -- 172.21.15.28:0/4254195863 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a50035070 con 0x7f9a5c104c10 2024-04-25T01:32:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.044+0000 7f9a61c12640 1 -- 172.21.15.28:0/4254195863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a5c19fef0 con 0x7f9a5c104c10 2024-04-25T01:32:52.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.046+0000 7f9a58ff9640 1 -- 172.21.15.28:0/4254195863 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9a50042030 con 0x7f9a5c104c10 2024-04-25T01:32:52.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.046+0000 7f9a649f7740 1 Processor -- start 2024-04-25T01:32:52.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.046+0000 7f9a649f7740 1 -- start start 2024-04-25T01:32:52.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.046+0000 7f9a649f7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba52440 0x56032ba52860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.046+0000 7f9a649f7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 0x56032ba57930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.047+0000 7f9a649f7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba57fa0 0x56032ba5a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.047+0000 7f9a649f7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56032b9b5f00 con 0x56032ba52440 2024-04-25T01:32:52.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.047+0000 7f9a649f7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56032b95e750 con 0x56032ba52e30 2024-04-25T01:32:52.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.047+0000 7f9a649f7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56032b9b1ec0 con 0x56032ba57fa0 2024-04-25T01:32:52.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.047+0000 7f9a5bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba52440 0x56032ba52860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.047+0000 7f9a5bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba52440 0x56032ba52860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41118/0 (socket says 172.21.15.28:41118) 2024-04-25T01:32:52.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.047+0000 7f9a5bfff640 1 -- 172.21.15.28:0/103687853 learned_addr learned my addr 172.21.15.28:0/103687853 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.047+0000 7f9a58ff9640 1 --2- 172.21.15.28:0/4254195863 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9a440a2b90 0x7f9a440a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.047+0000 7f9a5affd640 1 --2- 172.21.15.28:0/103687853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba57fa0 0x56032ba5a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.047+0000 7f9a5b7fe640 1 --2- 172.21.15.28:0/103687853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 0x56032ba57930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.047+0000 7f9a5bfff640 1 -- 172.21.15.28:0/103687853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba57fa0 msgr2=0x56032ba5a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.047+0000 7f9a5bfff640 1 --2- 172.21.15.28:0/103687853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba57fa0 0x56032ba5a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.047+0000 7f9a5bfff640 1 -- 172.21.15.28:0/103687853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 msgr2=0x56032ba57930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.047+0000 7f9a5bfff640 1 --2- 172.21.15.28:0/103687853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 0x56032ba57930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.047+0000 7f9a5bfff640 1 -- 172.21.15.28:0/103687853 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56032b6ca2d0 con 0x56032ba52440 2024-04-25T01:32:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.047+0000 7f9a58ff9640 1 -- 172.21.15.28:0/4254195863 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f9a500ef970 con 0x7f9a5c104c10 2024-04-25T01:32:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.047+0000 7f9a5bfff640 1 --2- 172.21.15.28:0/4254195863 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9a440a2b90 0x7f9a440a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.047+0000 7f9a5bfff640 1 --2- 172.21.15.28:0/103687853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba52440 0x56032ba52860 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400f530 tx=0x7f9a5400f9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb8825a760a4f70 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.047+0000 7f9a41ffb640 1 -- 172.21.15.28:0/103687853 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5401c400 con 0x56032ba52440 2024-04-25T01:32:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.048+0000 7f9a41ffb640 1 -- 172.21.15.28:0/103687853 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a540124b0 con 0x56032ba52440 2024-04-25T01:32:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.048+0000 7f9a41ffb640 1 -- 172.21.15.28:0/103687853 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5400d7a0 con 0x56032ba52440 2024-04-25T01:32:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.048+0000 7f9a5bfff640 1 --2- 172.21.15.28:0/4254195863 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9a440a2b90 0x7f9a440a5050 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9a54020000 tx=0x7f9a5401d000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.048+0000 7f9a649f7740 1 -- 172.21.15.28:0/103687853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba52440 msgr2=0x56032ba52860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.048+0000 7f9a649f7740 1 --2- 172.21.15.28:0/103687853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba52440 0x56032ba52860 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400f530 tx=0x7f9a5400f9f0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.048+0000 7f9a649f7740 1 -- 172.21.15.28:0/103687853 shutdown_connections 2024-04-25T01:32:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.048+0000 7f9a649f7740 1 --2- 172.21.15.28:0/103687853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba57fa0 0x56032ba5a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.048+0000 7f9a649f7740 1 --2- 172.21.15.28:0/103687853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 0x56032ba57930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.048+0000 7f9a649f7740 1 --2- 172.21.15.28:0/103687853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba52440 0x56032ba52860 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.048+0000 7f9a649f7740 1 -- 172.21.15.28:0/103687853 >> 172.21.15.28:0/103687853 conn(0x56032ba4dab0 msgr2=0x56032ba4deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.048+0000 7f9a649f7740 1 -- 172.21.15.28:0/103687853 shutdown_connections 2024-04-25T01:32:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.048+0000 7f9a649f7740 1 -- 172.21.15.28:0/103687853 wait complete. 2024-04-25T01:32:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.049+0000 7f9a649f7740 1 -- start start 2024-04-25T01:32:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.049+0000 7f9a649f7740 1 Processor -- start 2024-04-25T01:32:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.049+0000 7f9a649f7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba52440 0x56032ba60c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.049+0000 7f9a649f7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 0x56032ba611b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.049+0000 7f9a649f7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba57fa0 0x56032ba681f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.049+0000 7f9a649f7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56032ba5af50 con 0x56032ba57fa0 2024-04-25T01:32:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.049+0000 7f9a649f7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56032b971cf0 con 0x56032ba52e30 2024-04-25T01:32:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.049+0000 7f9a649f7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56032ba5b0d0 con 0x56032ba52440 2024-04-25T01:32:52.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.049+0000 7f9a649f7740 10 client.0 _ll_register_callbacks cb 0x56032b9716e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:52.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.049+0000 7f9a5bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba57fa0 0x56032ba681f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.049+0000 7f9a5bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba57fa0 0x56032ba681f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41134/0 (socket says 172.21.15.28:41134) 2024-04-25T01:32:52.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.049+0000 7f9a5bfff640 1 -- 172.21.15.28:0/2137394208 learned_addr learned my addr 172.21.15.28:0/2137394208 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.049+0000 7f9a5affd640 1 --2- 172.21.15.28:0/2137394208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 0x56032ba611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.049+0000 7f9a5b7fe640 1 --2- 172.21.15.28:0/2137394208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba52440 0x56032ba60c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.049+0000 7f9a5bfff640 1 -- 172.21.15.28:0/2137394208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba52440 msgr2=0x56032ba60c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.049+0000 7f9a5bfff640 1 --2- 172.21.15.28:0/2137394208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba52440 0x56032ba60c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.049+0000 7f9a5bfff640 1 -- 172.21.15.28:0/2137394208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 msgr2=0x56032ba611b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.050+0000 7f9a5bfff640 1 --2- 172.21.15.28:0/2137394208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 0x56032ba611b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.050+0000 7f9a5bfff640 1 -- 172.21.15.28:0/2137394208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56032b7946b0 con 0x56032ba57fa0 2024-04-25T01:32:52.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.050+0000 7f9a5affd640 1 --2- 172.21.15.28:0/2137394208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 0x56032ba611b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:52.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.050+0000 7f9a5bfff640 1 --2- 172.21.15.28:0/2137394208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba57fa0 0x56032ba681f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9a54010010 tx=0x7f9a5400d790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cba891f621772c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.050+0000 7f9a3affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:52.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.050+0000 7f9a3affd640 1 -- 172.21.15.28:0/2137394208 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5400dd50 con 0x56032ba57fa0 2024-04-25T01:32:52.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.050+0000 7f9a3affd640 1 -- 172.21.15.28:0/2137394208 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f9a540049e0 con 0x56032ba57fa0 2024-04-25T01:32:52.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.050+0000 7f9a649f7740 1 -- 172.21.15.28:0/2137394208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a540124b0 con 0x56032ba57fa0 2024-04-25T01:32:52.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.050+0000 7f9a3affd640 1 -- 172.21.15.28:0/2137394208 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9a54039050 con 0x56032ba57fa0 2024-04-25T01:32:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.050+0000 7f9a649f7740 10 client.15459 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.050+0000 7f9a649f7740 1 -- 172.21.15.28:0/2137394208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56032ba6b1d0 con 0x56032ba57fa0 2024-04-25T01:32:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.050+0000 7f9a3affd640 1 -- 172.21.15.28:0/2137394208 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a540124b0 con 0x56032ba57fa0 2024-04-25T01:32:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.050+0000 7f9a3affd640 1 -- 172.21.15.28:0/2137394208 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f9a54026480 con 0x56032ba57fa0 2024-04-25T01:32:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.050+0000 7f9a649f7740 10 client.15459 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.050+0000 7f9a649f7740 10 client.15459 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T01:32:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.050+0000 7f9a649f7740 20 client.15459 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:52.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.051+0000 7f9a649f7740 1 --2- 172.21.15.28:0/2137394208 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x56032ba6d680 0x56032ba6eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.051+0000 7f9a649f7740 4 client.15459 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:52.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.051+0000 7f9a649f7740 1 -- 172.21.15.28:0/2137394208 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56032b9c3440 con 0x56032ba6d680 2024-04-25T01:32:52.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.051+0000 7f9a5b7fe640 1 --2- 172.21.15.28:0/2137394208 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x56032ba6d680 0x56032ba6eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.051+0000 7f9a3affd640 10 client.15459 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:32:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.051+0000 7f9a5b7fe640 1 --2- 172.21.15.28:0/2137394208 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x56032ba6d680 0x56032ba6eaf0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b7879ebf7372ab4d server_cookie=d7bb97955acebf7e in_seq=0 out_seq=0 2024-04-25T01:32:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.052+0000 7f9a3affd640 1 -- 172.21.15.28:0/2137394208 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56032b9c3440 con 0x56032ba6d680 2024-04-25T01:32:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.052+0000 7f9a3affd640 10 client.15459 handle_command_reply: tid=0 2024-04-25T01:32:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.052+0000 7f9a649f7740 1 client.15459 shutdown 2024-04-25T01:32:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.052+0000 7f9a649f7740 1 -- 172.21.15.28:0/2137394208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba57fa0 msgr2=0x56032ba681f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.052+0000 7f9a649f7740 1 --2- 172.21.15.28:0/2137394208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba57fa0 0x56032ba681f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9a54010010 tx=0x7f9a5400d790 comp rx=0 tx=0).stop 2024-04-25T01:32:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.052+0000 7f9a649f7740 1 -- 172.21.15.28:0/2137394208 shutdown_connections 2024-04-25T01:32:52.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.052+0000 7f9a649f7740 1 --2- 172.21.15.28:0/2137394208 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x56032ba6d680 0x56032ba6eaf0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.052+0000 7f9a649f7740 1 --2- 172.21.15.28:0/2137394208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba57fa0 0x56032ba681f0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.052+0000 7f9a649f7740 1 --2- 172.21.15.28:0/2137394208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 0x56032ba611b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.052+0000 7f9a3affd640 0 client.15459 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:32:52.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.052+0000 7f9a649f7740 1 --2- 172.21.15.28:0/2137394208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba52440 0x56032ba60c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.052+0000 7f9a649f7740 1 -- 172.21.15.28:0/2137394208 >> 172.21.15.28:0/2137394208 conn(0x56032ba4dab0 msgr2=0x56032ba4cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.052+0000 7f9a649f7740 1 -- 172.21.15.28:0/2137394208 shutdown_connections 2024-04-25T01:32:52.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.052+0000 7f9a649f7740 1 -- 172.21.15.28:0/2137394208 wait complete. 2024-04-25T01:32:52.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.053+0000 7f9a649f7740 20 client.15459 trim_cache size 0 max 16384 2024-04-25T01:32:52.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.070+0000 7f9a649f7740 1 Processor -- start 2024-04-25T01:32:52.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.070+0000 7f9a649f7740 1 -- start start 2024-04-25T01:32:52.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a649f7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba52440 0x56032ba5a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a649f7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 0x56032ba68110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a649f7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba57fa0 0x56032ba48690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a649f7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56032ba6ce30 con 0x56032ba57fa0 2024-04-25T01:32:52.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a649f7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56032ba6d500 con 0x56032ba52e30 2024-04-25T01:32:52.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a649f7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56032ba6dbd0 con 0x56032ba52440 2024-04-25T01:32:52.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a5bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba57fa0 0x56032ba48690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a5bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba57fa0 0x56032ba48690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41140/0 (socket says 172.21.15.28:41140) 2024-04-25T01:32:52.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a5bfff640 1 -- 172.21.15.28:0/1845886823 learned_addr learned my addr 172.21.15.28:0/1845886823 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a5affd640 1 --2- 172.21.15.28:0/1845886823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 0x56032ba68110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a5bfff640 1 -- 172.21.15.28:0/1845886823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba52440 msgr2=0x56032ba5a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a5bfff640 1 --2- 172.21.15.28:0/1845886823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba52440 0x56032ba5a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a5bfff640 1 -- 172.21.15.28:0/1845886823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 msgr2=0x56032ba68110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a5bfff640 1 --2- 172.21.15.28:0/1845886823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 0x56032ba68110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a5bfff640 1 -- 172.21.15.28:0/1845886823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56032ba57470 con 0x56032ba57fa0 2024-04-25T01:32:52.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a5affd640 1 --2- 172.21.15.28:0/1845886823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 0x56032ba68110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:52.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a5bfff640 1 --2- 172.21.15.28:0/1845886823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba57fa0 0x56032ba48690 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9a54009bb0 tx=0x7f9a54009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=190f01946c25cdc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a417fa640 1 -- 172.21.15.28:0/1845886823 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5400d800 con 0x56032ba57fa0 2024-04-25T01:32:52.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a417fa640 1 -- 172.21.15.28:0/1845886823 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a5402a040 con 0x56032ba57fa0 2024-04-25T01:32:52.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.071+0000 7f9a417fa640 1 -- 172.21.15.28:0/1845886823 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a54026480 con 0x56032ba57fa0 2024-04-25T01:32:52.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.072+0000 7f9a649f7740 1 -- 172.21.15.28:0/1845886823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba57fa0 msgr2=0x56032ba48690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.072+0000 7f9a649f7740 1 --2- 172.21.15.28:0/1845886823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba57fa0 0x56032ba48690 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9a54009bb0 tx=0x7f9a54009a50 comp rx=0 tx=0).stop 2024-04-25T01:32:52.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.072+0000 7f9a649f7740 1 -- 172.21.15.28:0/1845886823 shutdown_connections 2024-04-25T01:32:52.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.072+0000 7f9a649f7740 1 --2- 172.21.15.28:0/1845886823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba57fa0 0x56032ba48690 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.072+0000 7f9a649f7740 1 --2- 172.21.15.28:0/1845886823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 0x56032ba68110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.072+0000 7f9a649f7740 1 --2- 172.21.15.28:0/1845886823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba52440 0x56032ba5a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.072+0000 7f9a649f7740 1 -- 172.21.15.28:0/1845886823 >> 172.21.15.28:0/1845886823 conn(0x56032ba4dab0 msgr2=0x56032ba4de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.072+0000 7f9a649f7740 1 -- 172.21.15.28:0/1845886823 shutdown_connections 2024-04-25T01:32:52.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.072+0000 7f9a649f7740 1 -- 172.21.15.28:0/1845886823 wait complete. 2024-04-25T01:32:52.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.072+0000 7f9a649f7740 1 -- start start 2024-04-25T01:32:52.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.072+0000 7f9a649f7740 1 Processor -- start 2024-04-25T01:32:52.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a649f7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba52440 0x56032ba56260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a649f7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 0x56032ba567a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a649f7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba57fa0 0x56032bad1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a649f7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56032ba6e970 con 0x56032ba52440 2024-04-25T01:32:52.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a649f7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56032ba6e2a0 con 0x56032ba52e30 2024-04-25T01:32:52.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a649f7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56032ba6f040 con 0x56032ba57fa0 2024-04-25T01:32:52.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a649f7740 10 client.0 _ll_register_callbacks cb 0x7f9a5401cc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:52.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a5b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba52440 0x56032ba56260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a5b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba52440 0x56032ba56260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41144/0 (socket says 172.21.15.28:41144) 2024-04-25T01:32:52.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a5b7fe640 1 -- 172.21.15.28:0/47842147 learned_addr learned my addr 172.21.15.28:0/47842147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a5bfff640 1 --2- 172.21.15.28:0/47842147 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba57fa0 0x56032bad1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a5affd640 1 --2- 172.21.15.28:0/47842147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 0x56032ba567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a5b7fe640 1 -- 172.21.15.28:0/47842147 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba57fa0 msgr2=0x56032bad1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a5b7fe640 1 --2- 172.21.15.28:0/47842147 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba57fa0 0x56032bad1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a5b7fe640 1 -- 172.21.15.28:0/47842147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 msgr2=0x56032ba567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a5b7fe640 1 --2- 172.21.15.28:0/47842147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 0x56032ba567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a5b7fe640 1 -- 172.21.15.28:0/47842147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56032bad1b60 con 0x56032ba52440 2024-04-25T01:32:52.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a5affd640 1 --2- 172.21.15.28:0/47842147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 0x56032ba567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:52.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a5bfff640 1 --2- 172.21.15.28:0/47842147 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba57fa0 0x56032bad1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:52.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a5b7fe640 1 --2- 172.21.15.28:0/47842147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba52440 0x56032ba56260 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c005fd0 tx=0x7f9a4c00bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61eb0281103868f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a3affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:52.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a3affd640 1 -- 172.21.15.28:0/47842147 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c013c00 con 0x56032ba52440 2024-04-25T01:32:52.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a3affd640 1 -- 172.21.15.28:0/47842147 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f9a4c013da0 con 0x56032ba52440 2024-04-25T01:32:52.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.073+0000 7f9a649f7740 1 -- 172.21.15.28:0/47842147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a5402a040 con 0x56032ba52440 2024-04-25T01:32:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.074+0000 7f9a3affd640 1 -- 172.21.15.28:0/47842147 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9a4c02b050 con 0x56032ba52440 2024-04-25T01:32:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.074+0000 7f9a649f7740 10 client.15465 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.074+0000 7f9a649f7740 1 -- 172.21.15.28:0/47842147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56032bad2ea0 con 0x56032ba52440 2024-04-25T01:32:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.074+0000 7f9a3affd640 1 -- 172.21.15.28:0/47842147 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a5402a040 con 0x56032ba52440 2024-04-25T01:32:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.074+0000 7f9a3affd640 1 -- 172.21.15.28:0/47842147 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f9a4c003bd0 con 0x56032ba52440 2024-04-25T01:32:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.074+0000 7f9a649f7740 10 client.15465 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.074+0000 7f9a649f7740 10 client.15465 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-25T01:32:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.074+0000 7f9a649f7740 20 client.15465 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.074+0000 7f9a649f7740 1 --2- 172.21.15.28:0/47842147 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x56032bad7b20 0x56032bad6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.074+0000 7f9a649f7740 4 client.15465 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:32:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.074+0000 7f9a649f7740 1 -- 172.21.15.28:0/47842147 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "status"}) -- 0x56032b9bd790 con 0x56032bad7b20 2024-04-25T01:32:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.074+0000 7f9a5affd640 1 --2- 172.21.15.28:0/47842147 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x56032bad7b20 0x56032bad6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.075+0000 7f9a5affd640 1 --2- 172.21.15.28:0/47842147 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x56032bad7b20 0x56032bad6830 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a0ac3abd13da63a2 server_cookie=5a53132d1ed536e7 in_seq=0 out_seq=0 2024-04-25T01:32:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.075+0000 7f9a3affd640 10 client.15465 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:32:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.075+0000 7f9a3affd640 1 -- 172.21.15.28:0/47842147 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56032b9bd790 con 0x56032bad7b20 2024-04-25T01:32:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.075+0000 7f9a3affd640 10 client.15465 handle_command_reply: tid=0 2024-04-25T01:32:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.075+0000 7f9a649f7740 1 client.15465 shutdown 2024-04-25T01:32:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.075+0000 7f9a649f7740 1 -- 172.21.15.28:0/47842147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba52440 msgr2=0x56032ba56260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.075+0000 7f9a649f7740 1 --2- 172.21.15.28:0/47842147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba52440 0x56032ba56260 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c005fd0 tx=0x7f9a4c00bde0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.075+0000 7f9a649f7740 1 -- 172.21.15.28:0/47842147 shutdown_connections 2024-04-25T01:32:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.075+0000 7f9a649f7740 1 --2- 172.21.15.28:0/47842147 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x56032bad7b20 0x56032bad6830 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.075+0000 7f9a649f7740 1 --2- 172.21.15.28:0/47842147 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ba57fa0 0x56032bad1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.075+0000 7f9a649f7740 1 --2- 172.21.15.28:0/47842147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ba52e30 0x56032ba567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.075+0000 7f9a649f7740 1 --2- 172.21.15.28:0/47842147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ba52440 0x56032ba56260 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.075+0000 7f9a3affd640 0 client.15465 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:32:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.075+0000 7f9a649f7740 1 -- 172.21.15.28:0/47842147 >> 172.21.15.28:0/47842147 conn(0x56032ba4dab0 msgr2=0x56032ba48690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.075+0000 7f9a649f7740 1 -- 172.21.15.28:0/47842147 shutdown_connections 2024-04-25T01:32:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.075+0000 7f9a649f7740 1 -- 172.21.15.28:0/47842147 wait complete. 2024-04-25T01:32:52.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.075+0000 7f9a649f7740 20 client.15465 trim_cache size 0 max 16384 2024-04-25T01:32:52.062 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:32:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "0e88f60a-02a2-11ef-bc93-c7b262605968", 2024-04-25T01:32:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 4, 2024-04-25T01:32:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24442, 2024-04-25T01:32:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-25T01:32:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-25T01:32:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-25T01:32:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 110.626425254, 2024-04-25T01:32:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-04-25T01:32:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-04-25T01:32:52.063 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 91, 2024-04-25T01:32:52.063 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 161.42531293900001 2024-04-25T01:32:52.063 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:32:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.076+0000 7f9a61c12640 1 -- 172.21.15.28:0/4254195863 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9a440a2b90 msgr2=0x7f9a440a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.076+0000 7f9a61c12640 1 --2- 172.21.15.28:0/4254195863 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9a440a2b90 0x7f9a440a5050 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9a54020000 tx=0x7f9a5401d000 comp rx=0 tx=0).stop 2024-04-25T01:32:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.076+0000 7f9a61c12640 1 -- 172.21.15.28:0/4254195863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c104c10 msgr2=0x7f9a5c198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.076+0000 7f9a61c12640 1 --2- 172.21.15.28:0/4254195863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c104c10 0x7f9a5c198090 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9a50031610 tx=0x7f9a500043b0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.076+0000 7f9a61c12640 1 -- 172.21.15.28:0/4254195863 shutdown_connections 2024-04-25T01:32:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.076+0000 7f9a61c12640 1 --2- 172.21.15.28:0/4254195863 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f9a440a2b90 0x7f9a440a5050 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.076+0000 7f9a61c12640 1 --2- 172.21.15.28:0/4254195863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a5c107540 0x7f9a5c19f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.076+0000 7f9a61c12640 1 --2- 172.21.15.28:0/4254195863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a5c104c10 0x7f9a5c198090 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.076+0000 7f9a61c12640 1 --2- 172.21.15.28:0/4254195863 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a5c1022e0 0x7f9a5c197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.076+0000 7f9a61c12640 1 -- 172.21.15.28:0/4254195863 >> 172.21.15.28:0/4254195863 conn(0x7f9a5c0fbc30 msgr2=0x7f9a5c102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.076+0000 7f9a61c12640 1 -- 172.21.15.28:0/4254195863 shutdown_connections 2024-04-25T01:32:52.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.076+0000 7f9a61c12640 1 -- 172.21.15.28:0/4254195863 wait complete. 2024-04-25T01:32:52.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T01:32:52.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.191+0000 7fce8937e640 1 -- 172.21.15.28:0/912090029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce84102980 msgr2=0x7fce84102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.191+0000 7fce8937e640 1 --2- 172.21.15.28:0/912090029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce84102980 0x7fce84102e00 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fce6c009900 tx=0x7fce6c02a410 comp rx=0 tx=0).stop 2024-04-25T01:32:52.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.191+0000 7fce8937e640 1 -- 172.21.15.28:0/912090029 shutdown_connections 2024-04-25T01:32:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.191+0000 7fce8937e640 1 --2- 172.21.15.28:0/912090029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce84103340 0x7fce84109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.191+0000 7fce8937e640 1 --2- 172.21.15.28:0/912090029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce84102980 0x7fce84102e00 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.191+0000 7fce8937e640 1 --2- 172.21.15.28:0/912090029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce84101790 0x7fce84101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.191+0000 7fce8937e640 1 -- 172.21.15.28:0/912090029 >> 172.21.15.28:0/912090029 conn(0x7fce840fcd20 msgr2=0x7fce840ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.192+0000 7fce8937e640 1 -- 172.21.15.28:0/912090029 shutdown_connections 2024-04-25T01:32:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.192+0000 7fce8937e640 1 -- 172.21.15.28:0/912090029 wait complete. 2024-04-25T01:32:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.192+0000 7fce8937e640 1 Processor -- start 2024-04-25T01:32:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.192+0000 7fce8937e640 1 -- start start 2024-04-25T01:32:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.192+0000 7fce8937e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce84101790 0x7fce84197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.192+0000 7fce8937e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce84102980 0x7fce84198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.192+0000 7fce8937e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce84103340 0x7fce8419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.192+0000 7fce8937e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fce8410d410 con 0x7fce84101790 2024-04-25T01:32:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.192+0000 7fce8937e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fce8410d290 con 0x7fce84103340 2024-04-25T01:32:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.192+0000 7fce8937e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fce8410d590 con 0x7fce84102980 2024-04-25T01:32:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.192+0000 7fce82ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce84101790 0x7fce84197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.192+0000 7fce82ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce84101790 0x7fce84197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41162/0 (socket says 172.21.15.28:41162) 2024-04-25T01:32:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.192+0000 7fce82ffd640 1 -- 172.21.15.28:0/1502788164 learned_addr learned my addr 172.21.15.28:0/1502788164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.192+0000 7fce837fe640 1 --2- 172.21.15.28:0/1502788164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce84103340 0x7fce8419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.193+0000 7fce7bfff640 1 --2- 172.21.15.28:0/1502788164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce84102980 0x7fce84198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.193+0000 7fce82ffd640 1 -- 172.21.15.28:0/1502788164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce84102980 msgr2=0x7fce84198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.193+0000 7fce82ffd640 1 --2- 172.21.15.28:0/1502788164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce84102980 0x7fce84198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.193+0000 7fce82ffd640 1 -- 172.21.15.28:0/1502788164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce84103340 msgr2=0x7fce8419f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.193+0000 7fce82ffd640 1 --2- 172.21.15.28:0/1502788164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce84103340 0x7fce8419f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.193+0000 7fce82ffd640 1 -- 172.21.15.28:0/1502788164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce8419f6e0 con 0x7fce84101790 2024-04-25T01:32:52.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.193+0000 7fce82ffd640 1 --2- 172.21.15.28:0/1502788164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce84101790 0x7fce84197ac0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fce7400c970 tx=0x7fce7400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.193+0000 7fce80ff9640 1 -- 172.21.15.28:0/1502788164 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce74004220 con 0x7fce84101790 2024-04-25T01:32:52.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.193+0000 7fce80ff9640 1 -- 172.21.15.28:0/1502788164 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce740043c0 con 0x7fce84101790 2024-04-25T01:32:52.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.193+0000 7fce8937e640 1 -- 172.21.15.28:0/1502788164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce8419f9d0 con 0x7fce84101790 2024-04-25T01:32:52.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.193+0000 7fce80ff9640 1 -- 172.21.15.28:0/1502788164 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce7400f450 con 0x7fce84101790 2024-04-25T01:32:52.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.193+0000 7fce8937e640 1 -- 172.21.15.28:0/1502788164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce84073970 con 0x7fce84101790 2024-04-25T01:32:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.195+0000 7fce80ff9640 1 -- 172.21.15.28:0/1502788164 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fce740079b0 con 0x7fce84101790 2024-04-25T01:32:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce8c16d740 1 Processor -- start 2024-04-25T01:32:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce8c16d740 1 -- start start 2024-04-25T01:32:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce8c16d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 0x5644b5433860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce8c16d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5433e30 0x5644b5438930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce8c16d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5438fa0 0x5644b543b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce8c16d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5644b5396f00 con 0x5644b5438fa0 2024-04-25T01:32:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce8c16d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5644b533f750 con 0x5644b5433e30 2024-04-25T01:32:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce8c16d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5644b5392ec0 con 0x5644b5433440 2024-04-25T01:32:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce80ff9640 1 --2- 172.21.15.28:0/1502788164 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fce700a7020 0x7fce700a94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce7bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5438fa0 0x5644b543b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce7bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5438fa0 0x5644b543b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41176/0 (socket says 172.21.15.28:41176) 2024-04-25T01:32:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce7bfff640 1 -- 172.21.15.28:0/602445956 learned_addr learned my addr 172.21.15.28:0/602445956 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce837fe640 1 --2- 172.21.15.28:0/1502788164 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fce700a7020 0x7fce700a94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce80ff9640 1 -- 172.21.15.28:0/1502788164 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fce740cf140 con 0x7fce84101790 2024-04-25T01:32:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce82ffd640 1 --2- 172.21.15.28:0/602445956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5433e30 0x5644b5438930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce837fe640 1 --2- 172.21.15.28:0/602445956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 0x5644b5433860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce7bfff640 1 -- 172.21.15.28:0/602445956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 msgr2=0x5644b5433860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce7bfff640 1 --2- 172.21.15.28:0/602445956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 0x5644b5433860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce7bfff640 1 -- 172.21.15.28:0/602445956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5433e30 msgr2=0x5644b5438930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce7bfff640 1 --2- 172.21.15.28:0/602445956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5433e30 0x5644b5438930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.196+0000 7fce7bfff640 1 -- 172.21.15.28:0/602445956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644b50ab2d0 con 0x5644b5438fa0 2024-04-25T01:32:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.197+0000 7fce82ffd640 1 --2- 172.21.15.28:0/602445956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5433e30 0x5644b5438930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.197+0000 7fce837fe640 1 --2- 172.21.15.28:0/1502788164 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fce700a7020 0x7fce700a94e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fce7c009fd0 tx=0x7fce7c012af0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.197+0000 7fce7bfff640 1 --2- 172.21.15.28:0/602445956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5438fa0 0x5644b543b430 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fce6c02a930 tx=0x7fce6c032d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8f5320d8f8abf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.197+0000 7fce797fa640 1 -- 172.21.15.28:0/602445956 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c03f070 con 0x5644b5438fa0 2024-04-25T01:32:52.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.197+0000 7fce797fa640 1 -- 172.21.15.28:0/602445956 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce6c0318b0 con 0x5644b5438fa0 2024-04-25T01:32:52.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.197+0000 7fce797fa640 1 -- 172.21.15.28:0/602445956 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c035070 con 0x5644b5438fa0 2024-04-25T01:32:52.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce8c16d740 1 -- 172.21.15.28:0/602445956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5438fa0 msgr2=0x5644b543b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce8c16d740 1 --2- 172.21.15.28:0/602445956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5438fa0 0x5644b543b430 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fce6c02a930 tx=0x7fce6c032d60 comp rx=0 tx=0).stop 2024-04-25T01:32:52.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce8c16d740 1 -- 172.21.15.28:0/602445956 shutdown_connections 2024-04-25T01:32:52.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce8c16d740 1 --2- 172.21.15.28:0/602445956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5438fa0 0x5644b543b430 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce8c16d740 1 --2- 172.21.15.28:0/602445956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5433e30 0x5644b5438930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce8c16d740 1 --2- 172.21.15.28:0/602445956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 0x5644b5433860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce8c16d740 1 -- 172.21.15.28:0/602445956 >> 172.21.15.28:0/602445956 conn(0x5644b542eab0 msgr2=0x5644b542eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce8c16d740 1 -- 172.21.15.28:0/602445956 shutdown_connections 2024-04-25T01:32:52.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce8c16d740 1 -- 172.21.15.28:0/602445956 wait complete. 2024-04-25T01:32:52.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce8c16d740 1 -- start start 2024-04-25T01:32:52.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce8c16d740 1 Processor -- start 2024-04-25T01:32:52.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce8c16d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 0x5644b543d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce8c16d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 0x5644b543dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce8c16d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5438fa0 0x5644b5444ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce8c16d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5644b543bef0 con 0x5644b5433e30 2024-04-25T01:32:52.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce8c16d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5644b5352cf0 con 0x5644b5438fa0 2024-04-25T01:32:52.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce8c16d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5644b543c070 con 0x5644b5433440 2024-04-25T01:32:52.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce8c16d740 10 client.0 _ll_register_callbacks cb 0x5644b53526e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:52.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce7bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 0x5644b543dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce7bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 0x5644b543dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41190/0 (socket says 172.21.15.28:41190) 2024-04-25T01:32:52.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.198+0000 7fce7bfff640 1 -- 172.21.15.28:0/236223851 learned_addr learned my addr 172.21.15.28:0/236223851 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.199+0000 7fce82ffd640 1 --2- 172.21.15.28:0/236223851 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 0x5644b543d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.199+0000 7fce7bfff640 1 -- 172.21.15.28:0/236223851 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 msgr2=0x5644b543d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.199+0000 7fce837fe640 1 --2- 172.21.15.28:0/236223851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5438fa0 0x5644b5444ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.199+0000 7fce7bfff640 1 --2- 172.21.15.28:0/236223851 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 0x5644b543d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.199+0000 7fce7bfff640 1 -- 172.21.15.28:0/236223851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5438fa0 msgr2=0x5644b5444ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.199+0000 7fce7bfff640 1 --2- 172.21.15.28:0/236223851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5438fa0 0x5644b5444ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.199+0000 7fce7bfff640 1 -- 172.21.15.28:0/236223851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644b51756b0 con 0x5644b5433e30 2024-04-25T01:32:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.199+0000 7fce7bfff640 1 --2- 172.21.15.28:0/236223851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 0x5644b543dca0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fce6c0314e0 tx=0x7fce6c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d27ef93bd2b7a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.199+0000 7fce6a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.199+0000 7fce6a7fc640 1 -- 172.21.15.28:0/236223851 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c04a070 con 0x5644b5433e30 2024-04-25T01:32:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.199+0000 7fce6a7fc640 1 -- 172.21.15.28:0/236223851 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fce6c003c20 con 0x5644b5433e30 2024-04-25T01:32:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.199+0000 7fce8c16d740 1 -- 172.21.15.28:0/236223851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce6c0318b0 con 0x5644b5433e30 2024-04-25T01:32:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.199+0000 7fce6a7fc640 1 -- 172.21.15.28:0/236223851 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fce6c056050 con 0x5644b5433e30 2024-04-25T01:32:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.199+0000 7fce8c16d740 10 client.15477 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.199+0000 7fce8c16d740 1 -- 172.21.15.28:0/236223851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5644b5447cc0 con 0x5644b5433e30 2024-04-25T01:32:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.199+0000 7fce6a7fc640 1 -- 172.21.15.28:0/236223851 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce6c0318b0 con 0x5644b5433e30 2024-04-25T01:32:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.200+0000 7fce6a7fc640 1 -- 172.21.15.28:0/236223851 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fce6c03bce0 con 0x5644b5433e30 2024-04-25T01:32:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.200+0000 7fce8c16d740 10 client.15477 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.200+0000 7fce8c16d740 10 client.15477 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T01:32:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.200+0000 7fce8c16d740 20 client.15477 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:52.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.200+0000 7fce8c16d740 1 --2- 172.21.15.28:0/236223851 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5644b5449f20 0x5644b544b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.200+0000 7fce8c16d740 4 client.15477 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.200+0000 7fce8c16d740 1 -- 172.21.15.28:0/236223851 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644b53a4440 con 0x5644b5449f20 2024-04-25T01:32:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.200+0000 7fce82ffd640 1 --2- 172.21.15.28:0/236223851 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5644b5449f20 0x5644b544b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.200+0000 7fce6a7fc640 10 client.15477 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:32:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.200+0000 7fce82ffd640 1 --2- 172.21.15.28:0/236223851 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5644b5449f20 0x5644b544b390 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=48082c3f010257b2 server_cookie=79c5ff4904dfeb41 in_seq=0 out_seq=0 2024-04-25T01:32:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.201+0000 7fce6a7fc640 1 -- 172.21.15.28:0/236223851 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5644b53a4440 con 0x5644b5449f20 2024-04-25T01:32:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.201+0000 7fce6a7fc640 10 client.15477 handle_command_reply: tid=0 2024-04-25T01:32:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.201+0000 7fce8c16d740 1 client.15477 shutdown 2024-04-25T01:32:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.201+0000 7fce8c16d740 1 -- 172.21.15.28:0/236223851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 msgr2=0x5644b543dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.201+0000 7fce8c16d740 1 --2- 172.21.15.28:0/236223851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 0x5644b543dca0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fce6c0314e0 tx=0x7fce6c003820 comp rx=0 tx=0).stop 2024-04-25T01:32:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.201+0000 7fce8c16d740 1 -- 172.21.15.28:0/236223851 shutdown_connections 2024-04-25T01:32:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.201+0000 7fce8c16d740 1 --2- 172.21.15.28:0/236223851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5438fa0 0x5644b5444ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.201+0000 7fce8c16d740 1 --2- 172.21.15.28:0/236223851 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5644b5449f20 0x5644b544b390 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.201+0000 7fce8c16d740 1 --2- 172.21.15.28:0/236223851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 0x5644b543dca0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.201+0000 7fce8c16d740 1 --2- 172.21.15.28:0/236223851 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 0x5644b543d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.201+0000 7fce6a7fc640 0 client.15477 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:32:52.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.201+0000 7fce8c16d740 1 -- 172.21.15.28:0/236223851 >> 172.21.15.28:0/236223851 conn(0x5644b542eab0 msgr2=0x5644b542dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.201+0000 7fce8c16d740 1 -- 172.21.15.28:0/236223851 shutdown_connections 2024-04-25T01:32:52.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.201+0000 7fce8c16d740 1 -- 172.21.15.28:0/236223851 wait complete. 2024-04-25T01:32:52.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.201+0000 7fce8c16d740 20 client.15477 trim_cache size 0 max 16384 2024-04-25T01:32:52.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.219+0000 7fce8c16d740 1 Processor -- start 2024-04-25T01:32:52.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.219+0000 7fce8c16d740 1 -- start start 2024-04-25T01:32:52.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.220+0000 7fce8c16d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 0x5644b542d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.220+0000 7fce8c16d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 0x5644b5444ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.220+0000 7fce8c16d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5438fa0 0x5644b542d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.220+0000 7fce8c16d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5644b54496d0 con 0x5644b5433e30 2024-04-25T01:32:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.220+0000 7fce8c16d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5644b5449da0 con 0x5644b5438fa0 2024-04-25T01:32:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.220+0000 7fce8c16d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5644b544a470 con 0x5644b5433440 2024-04-25T01:32:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.220+0000 7fce7bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 0x5644b5444ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.220+0000 7fce7bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 0x5644b5444ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41206/0 (socket says 172.21.15.28:41206) 2024-04-25T01:32:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.220+0000 7fce7bfff640 1 -- 172.21.15.28:0/2720818642 learned_addr learned my addr 172.21.15.28:0/2720818642 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.220+0000 7fce837fe640 1 --2- 172.21.15.28:0/2720818642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5438fa0 0x5644b542d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.220+0000 7fce82ffd640 1 --2- 172.21.15.28:0/2720818642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 0x5644b542d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.220+0000 7fce7bfff640 1 -- 172.21.15.28:0/2720818642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 msgr2=0x5644b542d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.220+0000 7fce7bfff640 1 --2- 172.21.15.28:0/2720818642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 0x5644b542d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.220+0000 7fce7bfff640 1 -- 172.21.15.28:0/2720818642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5438fa0 msgr2=0x5644b542d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.220+0000 7fce7bfff640 1 --2- 172.21.15.28:0/2720818642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5438fa0 0x5644b542d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.220+0000 7fce7bfff640 1 -- 172.21.15.28:0/2720818642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644b542f780 con 0x5644b5433e30 2024-04-25T01:32:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.220+0000 7fce82ffd640 1 --2- 172.21.15.28:0/2720818642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 0x5644b542d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.220+0000 7fce7bfff640 1 --2- 172.21.15.28:0/2720818642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 0x5644b5444ca0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fce6c006010 tx=0x7fce6c031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aff408aebd02686 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.221+0000 7fce78ff9640 1 -- 172.21.15.28:0/2720818642 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c04a070 con 0x5644b5433e30 2024-04-25T01:32:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.221+0000 7fce78ff9640 1 -- 172.21.15.28:0/2720818642 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce6c0438f0 con 0x5644b5433e30 2024-04-25T01:32:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.221+0000 7fce78ff9640 1 -- 172.21.15.28:0/2720818642 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c03f030 con 0x5644b5433e30 2024-04-25T01:32:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.221+0000 7fce8c16d740 1 -- 172.21.15.28:0/2720818642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 msgr2=0x5644b5444ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.221+0000 7fce8c16d740 1 --2- 172.21.15.28:0/2720818642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 0x5644b5444ca0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fce6c006010 tx=0x7fce6c031400 comp rx=0 tx=0).stop 2024-04-25T01:32:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.221+0000 7fce8c16d740 1 -- 172.21.15.28:0/2720818642 shutdown_connections 2024-04-25T01:32:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.221+0000 7fce8c16d740 1 --2- 172.21.15.28:0/2720818642 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5438fa0 0x5644b542d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.221+0000 7fce8c16d740 1 --2- 172.21.15.28:0/2720818642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 0x5644b5444ca0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.221+0000 7fce8c16d740 1 --2- 172.21.15.28:0/2720818642 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 0x5644b542d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.221+0000 7fce8c16d740 1 -- 172.21.15.28:0/2720818642 >> 172.21.15.28:0/2720818642 conn(0x5644b542eab0 msgr2=0x5644b54346c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.221+0000 7fce8c16d740 1 -- 172.21.15.28:0/2720818642 shutdown_connections 2024-04-25T01:32:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.221+0000 7fce8c16d740 1 -- 172.21.15.28:0/2720818642 wait complete. 2024-04-25T01:32:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce8c16d740 1 -- start start 2024-04-25T01:32:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce8c16d740 1 Processor -- start 2024-04-25T01:32:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce8c16d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 0x5644b5440e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce8c16d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 0x5644b5441370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce8c16d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5438fa0 0x5644b5444030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce8c16d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5644b544b210 con 0x5644b5433e30 2024-04-25T01:32:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce8c16d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5644b544ab40 con 0x5644b5438fa0 2024-04-25T01:32:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce8c16d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5644b544b8e0 con 0x5644b5433440 2024-04-25T01:32:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce8c16d740 10 client.0 _ll_register_callbacks cb 0x7fce6c004500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce7bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 0x5644b5441370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce7bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 0x5644b5441370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41214/0 (socket says 172.21.15.28:41214) 2024-04-25T01:32:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce7bfff640 1 -- 172.21.15.28:0/2861168452 learned_addr learned my addr 172.21.15.28:0/2861168452 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce837fe640 1 --2- 172.21.15.28:0/2861168452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5438fa0 0x5644b5444030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce82ffd640 1 --2- 172.21.15.28:0/2861168452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 0x5644b5440e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce7bfff640 1 -- 172.21.15.28:0/2861168452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 msgr2=0x5644b5440e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce7bfff640 1 --2- 172.21.15.28:0/2861168452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 0x5644b5440e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce7bfff640 1 -- 172.21.15.28:0/2861168452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5438fa0 msgr2=0x5644b5444030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce7bfff640 1 --2- 172.21.15.28:0/2861168452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5438fa0 0x5644b5444030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce7bfff640 1 -- 172.21.15.28:0/2861168452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644b5444620 con 0x5644b5433e30 2024-04-25T01:32:52.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce837fe640 1 --2- 172.21.15.28:0/2861168452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5438fa0 0x5644b5444030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:52.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.222+0000 7fce82ffd640 1 --2- 172.21.15.28:0/2861168452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 0x5644b5440e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:52.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.223+0000 7fce7bfff640 1 --2- 172.21.15.28:0/2861168452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 0x5644b5441370 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fce6c0435a0 tx=0x7fce6c043670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf8f8030ce16f2d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.223+0000 7fce6a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:52.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.223+0000 7fce6a7fc640 1 -- 172.21.15.28:0/2861168452 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c04a030 con 0x5644b5433e30 2024-04-25T01:32:52.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.223+0000 7fce6a7fc640 1 -- 172.21.15.28:0/2861168452 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fce6c003e20 con 0x5644b5433e30 2024-04-25T01:32:52.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.223+0000 7fce8c16d740 1 -- 172.21.15.28:0/2861168452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce6c0438f0 con 0x5644b5433e30 2024-04-25T01:32:52.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.223+0000 7fce6a7fc640 1 -- 172.21.15.28:0/2861168452 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fce6c068050 con 0x5644b5433e30 2024-04-25T01:32:52.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.223+0000 7fce8c16d740 10 client.15483 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:52.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.223+0000 7fce8c16d740 1 -- 172.21.15.28:0/2861168452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5644b54b04e0 con 0x5644b5433e30 2024-04-25T01:32:52.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.223+0000 7fce6a7fc640 1 -- 172.21.15.28:0/2861168452 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce6c0438f0 con 0x5644b5433e30 2024-04-25T01:32:52.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.223+0000 7fce6a7fc640 1 -- 172.21.15.28:0/2861168452 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fce6c0046c0 con 0x5644b5433e30 2024-04-25T01:32:52.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.223+0000 7fce8c16d740 10 client.15483 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:52.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.223+0000 7fce8c16d740 10 client.15483 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T01:32:52.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.223+0000 7fce8c16d740 20 client.15483 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:52.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.223+0000 7fce8c16d740 1 --2- 172.21.15.28:0/2861168452 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5644b54b4dc0 0x5644b54b3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.223+0000 7fce8c16d740 4 client.15483 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:32:52.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.223+0000 7fce8c16d740 1 -- 172.21.15.28:0/2861168452 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "status"}) -- 0x5644b539e790 con 0x5644b54b4dc0 2024-04-25T01:32:52.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.224+0000 7fce82ffd640 1 --2- 172.21.15.28:0/2861168452 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5644b54b4dc0 0x5644b54b3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.224+0000 7fce6a7fc640 10 client.15483 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:32:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.224+0000 7fce82ffd640 1 --2- 172.21.15.28:0/2861168452 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5644b54b4dc0 0x5644b54b3ad0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=58dad3e1a9b1545 server_cookie=eb57ddf4fdfe4b5c in_seq=0 out_seq=0 2024-04-25T01:32:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.224+0000 7fce6a7fc640 1 -- 172.21.15.28:0/2861168452 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5644b539e790 con 0x5644b54b4dc0 2024-04-25T01:32:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.224+0000 7fce6a7fc640 10 client.15483 handle_command_reply: tid=0 2024-04-25T01:32:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.224+0000 7fce8c16d740 1 client.15483 shutdown 2024-04-25T01:32:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.224+0000 7fce8c16d740 1 -- 172.21.15.28:0/2861168452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 msgr2=0x5644b5441370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.224+0000 7fce8c16d740 1 --2- 172.21.15.28:0/2861168452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 0x5644b5441370 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fce6c0435a0 tx=0x7fce6c043670 comp rx=0 tx=0).stop 2024-04-25T01:32:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.224+0000 7fce8c16d740 1 -- 172.21.15.28:0/2861168452 shutdown_connections 2024-04-25T01:32:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.224+0000 7fce8c16d740 1 --2- 172.21.15.28:0/2861168452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644b5438fa0 0x5644b5444030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.224+0000 7fce8c16d740 1 --2- 172.21.15.28:0/2861168452 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5644b54b4dc0 0x5644b54b3ad0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.224+0000 7fce8c16d740 1 --2- 172.21.15.28:0/2861168452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644b5433e30 0x5644b5441370 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.224+0000 7fce6a7fc640 0 client.15483 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:32:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.224+0000 7fce8c16d740 1 --2- 172.21.15.28:0/2861168452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644b5433440 0x5644b5440e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.224+0000 7fce8c16d740 1 -- 172.21.15.28:0/2861168452 >> 172.21.15.28:0/2861168452 conn(0x5644b542eab0 msgr2=0x5644b5431990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.224+0000 7fce8c16d740 1 -- 172.21.15.28:0/2861168452 shutdown_connections 2024-04-25T01:32:52.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.224+0000 7fce8c16d740 1 -- 172.21.15.28:0/2861168452 wait complete. 2024-04-25T01:32:52.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.224+0000 7fce8c16d740 20 client.15483 trim_cache size 0 max 16384 2024-04-25T01:32:52.211 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:32:52.211 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "0e88f60a-02a2-11ef-bc93-c7b262605968", 2024-04-25T01:32:52.211 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 3, 2024-04-25T01:32:52.211 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24454, 2024-04-25T01:32:52.211 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-25T01:32:52.211 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-25T01:32:52.211 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-25T01:32:52.211 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 111.783554385, 2024-04-25T01:32:52.212 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-04-25T01:32:52.212 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-04-25T01:32:52.212 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 91, 2024-04-25T01:32:52.212 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 158.87713946900001 2024-04-25T01:32:52.212 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:32:52.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.225+0000 7fce8937e640 1 -- 172.21.15.28:0/1502788164 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fce700a7020 msgr2=0x7fce700a94e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.225+0000 7fce8937e640 1 --2- 172.21.15.28:0/1502788164 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fce700a7020 0x7fce700a94e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fce7c009fd0 tx=0x7fce7c012af0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.225+0000 7fce8937e640 1 -- 172.21.15.28:0/1502788164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce84101790 msgr2=0x7fce84197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.225+0000 7fce8937e640 1 --2- 172.21.15.28:0/1502788164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce84101790 0x7fce84197ac0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fce7400c970 tx=0x7fce7400ce30 comp rx=0 tx=0).stop 2024-04-25T01:32:52.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.225+0000 7fce8937e640 1 -- 172.21.15.28:0/1502788164 shutdown_connections 2024-04-25T01:32:52.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.225+0000 7fce8937e640 1 --2- 172.21.15.28:0/1502788164 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fce700a7020 0x7fce700a94e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.225+0000 7fce8937e640 1 --2- 172.21.15.28:0/1502788164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce84103340 0x7fce8419f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.225+0000 7fce8937e640 1 --2- 172.21.15.28:0/1502788164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fce84102980 0x7fce84198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.225+0000 7fce8937e640 1 --2- 172.21.15.28:0/1502788164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce84101790 0x7fce84197ac0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.225+0000 7fce8937e640 1 -- 172.21.15.28:0/1502788164 >> 172.21.15.28:0/1502788164 conn(0x7fce840fcd20 msgr2=0x7fce84105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.225+0000 7fce8937e640 1 -- 172.21.15.28:0/1502788164 shutdown_connections 2024-04-25T01:32:52.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.225+0000 7fce8937e640 1 -- 172.21.15.28:0/1502788164 wait complete. 2024-04-25T01:32:52.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:52 smithi028 ceph-mon[29390]: pgmap v368: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:52.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:52 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/840490266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:32:52.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:52 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1392132689' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:52.215 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:52 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1822658888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:52.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-25T01:32:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.340+0000 7fa8ed82a640 1 -- 172.21.15.28:0/362556204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e8101890 msgr2=0x7fa8e8101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.340+0000 7fa8ed82a640 1 --2- 172.21.15.28:0/362556204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e8101890 0x7fa8e8101d10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0009900 tx=0x7fa8d002a410 comp rx=0 tx=0).stop 2024-04-25T01:32:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.340+0000 7fa8ed82a640 1 -- 172.21.15.28:0/362556204 shutdown_connections 2024-04-25T01:32:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.340+0000 7fa8ed82a640 1 --2- 172.21.15.28:0/362556204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e8102250 0x7fa8e8108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.340+0000 7fa8ed82a640 1 --2- 172.21.15.28:0/362556204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e8101890 0x7fa8e8101d10 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.340+0000 7fa8ed82a640 1 --2- 172.21.15.28:0/362556204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8e81006a0 0x7fa8e8100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.340+0000 7fa8ed82a640 1 -- 172.21.15.28:0/362556204 >> 172.21.15.28:0/362556204 conn(0x7fa8e80fbc30 msgr2=0x7fa8e80fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.341+0000 7fa8ed82a640 1 -- 172.21.15.28:0/362556204 shutdown_connections 2024-04-25T01:32:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.341+0000 7fa8ed82a640 1 -- 172.21.15.28:0/362556204 wait complete. 2024-04-25T01:32:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.341+0000 7fa8ed82a640 1 Processor -- start 2024-04-25T01:32:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.341+0000 7fa8ed82a640 1 -- start start 2024-04-25T01:32:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.341+0000 7fa8ed82a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e81006a0 0x7fa8e8197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.341+0000 7fa8ed82a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8e8101890 0x7fa8e8198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.341+0000 7fa8ed82a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e8102250 0x7fa8e819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.341+0000 7fa8ed82a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e810c320 con 0x7fa8e81006a0 2024-04-25T01:32:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.341+0000 7fa8ed82a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e810c1a0 con 0x7fa8e8102250 2024-04-25T01:32:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.341+0000 7fa8ed82a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e810c4a0 con 0x7fa8e8101890 2024-04-25T01:32:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.341+0000 7fa8e6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e81006a0 0x7fa8e8197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.341+0000 7fa8e6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e81006a0 0x7fa8e8197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41232/0 (socket says 172.21.15.28:41232) 2024-04-25T01:32:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.341+0000 7fa8e6ffd640 1 -- 172.21.15.28:0/1200876266 learned_addr learned my addr 172.21.15.28:0/1200876266 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.341+0000 7fa8e77fe640 1 --2- 172.21.15.28:0/1200876266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e8102250 0x7fa8e819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.342+0000 7fa8dffff640 1 --2- 172.21.15.28:0/1200876266 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8e8101890 0x7fa8e8198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.342+0000 7fa8e6ffd640 1 -- 172.21.15.28:0/1200876266 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8e8101890 msgr2=0x7fa8e8198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.342+0000 7fa8e6ffd640 1 --2- 172.21.15.28:0/1200876266 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8e8101890 0x7fa8e8198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.342+0000 7fa8e6ffd640 1 -- 172.21.15.28:0/1200876266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e8102250 msgr2=0x7fa8e819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.342+0000 7fa8e6ffd640 1 --2- 172.21.15.28:0/1200876266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e8102250 0x7fa8e819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.342+0000 7fa8e6ffd640 1 -- 172.21.15.28:0/1200876266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e819f6e0 con 0x7fa8e81006a0 2024-04-25T01:32:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.342+0000 7fa8e6ffd640 1 --2- 172.21.15.28:0/1200876266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e81006a0 0x7fa8e8197b10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa8d800d950 tx=0x7fa8d800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.342+0000 7fa8ec828640 1 -- 172.21.15.28:0/1200876266 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8017070 con 0x7fa8e81006a0 2024-04-25T01:32:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.342+0000 7fa8ec828640 1 -- 172.21.15.28:0/1200876266 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8d8009c30 con 0x7fa8e81006a0 2024-04-25T01:32:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.342+0000 7fa8ed82a640 1 -- 172.21.15.28:0/1200876266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8e819f9d0 con 0x7fa8e81006a0 2024-04-25T01:32:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.342+0000 7fa8ec828640 1 -- 172.21.15.28:0/1200876266 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d800f600 con 0x7fa8e81006a0 2024-04-25T01:32:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.342+0000 7fa8ed82a640 1 -- 172.21.15.28:0/1200876266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8e819fea0 con 0x7fa8e81006a0 2024-04-25T01:32:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.344+0000 7fa8ec828640 1 -- 172.21.15.28:0/1200876266 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa8d801a070 con 0x7fa8e81006a0 2024-04-25T01:32:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.345+0000 7fa8f0614740 1 Processor -- start 2024-04-25T01:32:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.345+0000 7fa8f0614740 1 -- start start 2024-04-25T01:32:52.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.345+0000 7fa8f0614740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670f440 0x55ecb670f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.345+0000 7fa8f0614740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb670fe30 0x55ecb6714930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.345+0000 7fa8f0614740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb6714fa0 0x55ecb6717430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.345+0000 7fa8f0614740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ecb6672f00 con 0x55ecb670f440 2024-04-25T01:32:52.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.345+0000 7fa8f0614740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ecb661b750 con 0x55ecb670fe30 2024-04-25T01:32:52.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.345+0000 7fa8f0614740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ecb666eec0 con 0x55ecb6714fa0 2024-04-25T01:32:52.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.345+0000 7fa8e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670f440 0x55ecb670f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.345+0000 7fa8e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670f440 0x55ecb670f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41244/0 (socket says 172.21.15.28:41244) 2024-04-25T01:32:52.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.345+0000 7fa8e77fe640 1 -- 172.21.15.28:0/604948690 learned_addr learned my addr 172.21.15.28:0/604948690 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.345+0000 7fa8ec828640 1 --2- 172.21.15.28:0/1200876266 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa8d40a2b90 0x7fa8d40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.345+0000 7fa8e6ffd640 1 --2- 172.21.15.28:0/604948690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb670fe30 0x55ecb6714930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.345+0000 7fa8dffff640 1 --2- 172.21.15.28:0/604948690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb6714fa0 0x55ecb6717430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.345+0000 7fa8e77fe640 1 --2- 172.21.15.28:0/1200876266 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa8d40a2b90 0x7fa8d40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.345+0000 7fa8e77fe640 1 -- 172.21.15.28:0/604948690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb6714fa0 msgr2=0x55ecb6717430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.346+0000 7fa8e77fe640 1 --2- 172.21.15.28:0/604948690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb6714fa0 0x55ecb6717430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.346+0000 7fa8ec828640 1 -- 172.21.15.28:0/1200876266 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fa8d80cef50 con 0x7fa8e81006a0 2024-04-25T01:32:52.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.346+0000 7fa8e77fe640 1 -- 172.21.15.28:0/604948690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb670fe30 msgr2=0x55ecb6714930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.346+0000 7fa8e77fe640 1 --2- 172.21.15.28:0/604948690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb670fe30 0x55ecb6714930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.346+0000 7fa8e77fe640 1 -- 172.21.15.28:0/604948690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecb63872d0 con 0x55ecb670f440 2024-04-25T01:32:52.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.346+0000 7fa8dffff640 1 --2- 172.21.15.28:0/604948690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb6714fa0 0x55ecb6717430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:52.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.346+0000 7fa8e77fe640 1 --2- 172.21.15.28:0/604948690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670f440 0x55ecb670f860 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00149c0 tx=0x7fa8e0014e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2395131291daa2a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.346+0000 7fa8ddffb640 1 -- 172.21.15.28:0/604948690 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e001eb40 con 0x55ecb670f440 2024-04-25T01:32:52.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.346+0000 7fa8ddffb640 1 -- 172.21.15.28:0/604948690 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8e0023070 con 0x55ecb670f440 2024-04-25T01:32:52.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.346+0000 7fa8ddffb640 1 -- 172.21.15.28:0/604948690 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0013b20 con 0x55ecb670f440 2024-04-25T01:32:52.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.346+0000 7fa8e77fe640 1 --2- 172.21.15.28:0/1200876266 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa8d40a2b90 0x7fa8d40a5050 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0022000 tx=0x7fa8e0020000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.347+0000 7fa8f0614740 1 -- 172.21.15.28:0/604948690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670f440 msgr2=0x55ecb670f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.347+0000 7fa8f0614740 1 --2- 172.21.15.28:0/604948690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670f440 0x55ecb670f860 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00149c0 tx=0x7fa8e0014e80 comp rx=0 tx=0).stop 2024-04-25T01:32:52.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.347+0000 7fa8f0614740 1 -- 172.21.15.28:0/604948690 shutdown_connections 2024-04-25T01:32:52.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.347+0000 7fa8f0614740 1 --2- 172.21.15.28:0/604948690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb6714fa0 0x55ecb6717430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.347+0000 7fa8f0614740 1 --2- 172.21.15.28:0/604948690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb670fe30 0x55ecb6714930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.347+0000 7fa8f0614740 1 --2- 172.21.15.28:0/604948690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670f440 0x55ecb670f860 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.347+0000 7fa8f0614740 1 -- 172.21.15.28:0/604948690 >> 172.21.15.28:0/604948690 conn(0x55ecb670aab0 msgr2=0x55ecb670aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.347+0000 7fa8f0614740 1 -- 172.21.15.28:0/604948690 shutdown_connections 2024-04-25T01:32:52.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.347+0000 7fa8f0614740 1 -- 172.21.15.28:0/604948690 wait complete. 2024-04-25T01:32:52.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.347+0000 7fa8f0614740 1 -- start start 2024-04-25T01:32:52.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.347+0000 7fa8f0614740 1 Processor -- start 2024-04-25T01:32:52.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.347+0000 7fa8f0614740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb670f440 0x55ecb6719880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.347+0000 7fa8f0614740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 0x55ecb6719dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.347+0000 7fa8f0614740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb6714fa0 0x55ecb6720e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.347+0000 7fa8f0614740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ecb6717f50 con 0x55ecb670fe30 2024-04-25T01:32:52.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.347+0000 7fa8f0614740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ecb662ecf0 con 0x55ecb6714fa0 2024-04-25T01:32:52.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.347+0000 7fa8f0614740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ecb67180d0 con 0x55ecb670f440 2024-04-25T01:32:52.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.347+0000 7fa8f0614740 10 client.0 _ll_register_callbacks cb 0x55ecb662e6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:52.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.348+0000 7fa8dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 0x55ecb6719dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.348+0000 7fa8dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 0x55ecb6719dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41252/0 (socket says 172.21.15.28:41252) 2024-04-25T01:32:52.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.348+0000 7fa8dffff640 1 -- 172.21.15.28:0/564487676 learned_addr learned my addr 172.21.15.28:0/564487676 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.348+0000 7fa8e6ffd640 1 --2- 172.21.15.28:0/564487676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb670f440 0x55ecb6719880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.348+0000 7fa8e77fe640 1 --2- 172.21.15.28:0/564487676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb6714fa0 0x55ecb6720e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.348+0000 7fa8dffff640 1 -- 172.21.15.28:0/564487676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb670f440 msgr2=0x55ecb6719880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.348+0000 7fa8dffff640 1 --2- 172.21.15.28:0/564487676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb670f440 0x55ecb6719880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.348+0000 7fa8dffff640 1 -- 172.21.15.28:0/564487676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb6714fa0 msgr2=0x55ecb6720e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.348+0000 7fa8dffff640 1 --2- 172.21.15.28:0/564487676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb6714fa0 0x55ecb6720e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.348+0000 7fa8dffff640 1 -- 172.21.15.28:0/564487676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecb64516b0 con 0x55ecb670fe30 2024-04-25T01:32:52.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.348+0000 7fa8e77fe640 1 --2- 172.21.15.28:0/564487676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb6714fa0 0x55ecb6720e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:52.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.348+0000 7fa8dffff640 1 --2- 172.21.15.28:0/564487676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 0x55ecb6719dc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0009900 tx=0x7fa8d0031960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffad9b403ea4ba8b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.348+0000 7fa8c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:52.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.348+0000 7fa8c6ffd640 1 -- 172.21.15.28:0/564487676 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0035070 con 0x55ecb670fe30 2024-04-25T01:32:52.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.348+0000 7fa8f0614740 1 -- 172.21.15.28:0/564487676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8e0023070 con 0x55ecb670fe30 2024-04-25T01:32:52.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.348+0000 7fa8c6ffd640 1 -- 172.21.15.28:0/564487676 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fa8d0004240 con 0x55ecb670fe30 2024-04-25T01:32:52.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.348+0000 7fa8c6ffd640 1 -- 172.21.15.28:0/564487676 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa8d004d050 con 0x55ecb670fe30 2024-04-25T01:32:52.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.348+0000 7fa8f0614740 10 client.15495 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:52.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.348+0000 7fa8f0614740 1 -- 172.21.15.28:0/564487676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ecb6723e50 con 0x55ecb670fe30 2024-04-25T01:32:52.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.349+0000 7fa8c6ffd640 1 -- 172.21.15.28:0/564487676 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8e0023070 con 0x55ecb670fe30 2024-04-25T01:32:52.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.349+0000 7fa8c6ffd640 1 -- 172.21.15.28:0/564487676 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fa8d0004ac0 con 0x55ecb670fe30 2024-04-25T01:32:52.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.349+0000 7fa8f0614740 10 client.15495 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:52.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.349+0000 7fa8f0614740 10 client.15495 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T01:32:52.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.349+0000 7fa8f0614740 20 client.15495 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:52.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.349+0000 7fa8f0614740 1 --2- 172.21.15.28:0/564487676 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ecb6726500 0x55ecb6727970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.349+0000 7fa8f0614740 4 client.15495 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:52.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.349+0000 7fa8f0614740 1 -- 172.21.15.28:0/564487676 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ecb6680440 con 0x55ecb6726500 2024-04-25T01:32:52.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.349+0000 7fa8e6ffd640 1 --2- 172.21.15.28:0/564487676 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ecb6726500 0x55ecb6727970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.349+0000 7fa8e6ffd640 1 --2- 172.21.15.28:0/564487676 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ecb6726500 0x55ecb6727970 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=825ebd2a2a317d69 server_cookie=56b4f11e27b1addb in_seq=0 out_seq=0 2024-04-25T01:32:52.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.349+0000 7fa8c6ffd640 10 client.15495 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:32:52.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.350+0000 7fa8c6ffd640 1 -- 172.21.15.28:0/564487676 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ecb6680440 con 0x55ecb6726500 2024-04-25T01:32:52.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.350+0000 7fa8c6ffd640 10 client.15495 handle_command_reply: tid=0 2024-04-25T01:32:52.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.350+0000 7fa8f0614740 1 client.15495 shutdown 2024-04-25T01:32:52.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.350+0000 7fa8f0614740 1 -- 172.21.15.28:0/564487676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 msgr2=0x55ecb6719dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.350+0000 7fa8f0614740 1 --2- 172.21.15.28:0/564487676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 0x55ecb6719dc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0009900 tx=0x7fa8d0031960 comp rx=0 tx=0).stop 2024-04-25T01:32:52.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.350+0000 7fa8f0614740 1 -- 172.21.15.28:0/564487676 shutdown_connections 2024-04-25T01:32:52.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.350+0000 7fa8f0614740 1 --2- 172.21.15.28:0/564487676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb6714fa0 0x55ecb6720e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.350+0000 7fa8f0614740 1 --2- 172.21.15.28:0/564487676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 0x55ecb6719dc0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.350+0000 7fa8f0614740 1 --2- 172.21.15.28:0/564487676 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ecb6726500 0x55ecb6727970 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.350+0000 7fa8f0614740 1 --2- 172.21.15.28:0/564487676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb670f440 0x55ecb6719880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.350+0000 7fa8f0614740 1 -- 172.21.15.28:0/564487676 >> 172.21.15.28:0/564487676 conn(0x55ecb670aab0 msgr2=0x55ecb6709ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.350+0000 7fa8c6ffd640 0 client.15495 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:32:52.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.351+0000 7fa8f0614740 1 -- 172.21.15.28:0/564487676 shutdown_connections 2024-04-25T01:32:52.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.351+0000 7fa8f0614740 1 -- 172.21.15.28:0/564487676 wait complete. 2024-04-25T01:32:52.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.351+0000 7fa8f0614740 20 client.15495 trim_cache size 0 max 16384 2024-04-25T01:32:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.369+0000 7fa8f0614740 1 Processor -- start 2024-04-25T01:32:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.369+0000 7fa8f0614740 1 -- start start 2024-04-25T01:32:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.369+0000 7fa8f0614740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb670f440 0x55ecb6717390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.369+0000 7fa8f0614740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 0x55ecb6720dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.369+0000 7fa8f0614740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb6714fa0 0x55ecb67108d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.369+0000 7fa8f0614740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ecb6725cb0 con 0x55ecb670fe30 2024-04-25T01:32:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.369+0000 7fa8f0614740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ecb6726380 con 0x55ecb670f440 2024-04-25T01:32:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.369+0000 7fa8f0614740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ecb6726a50 con 0x55ecb6714fa0 2024-04-25T01:32:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.369+0000 7fa8dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 0x55ecb6720dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.369+0000 7fa8dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 0x55ecb6720dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41258/0 (socket says 172.21.15.28:41258) 2024-04-25T01:32:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.369+0000 7fa8dffff640 1 -- 172.21.15.28:0/1595692671 learned_addr learned my addr 172.21.15.28:0/1595692671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.369+0000 7fa8e6ffd640 1 --2- 172.21.15.28:0/1595692671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb670f440 0x55ecb6717390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.369+0000 7fa8e77fe640 1 --2- 172.21.15.28:0/1595692671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb6714fa0 0x55ecb67108d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.369+0000 7fa8dffff640 1 -- 172.21.15.28:0/1595692671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb6714fa0 msgr2=0x55ecb67108d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.369+0000 7fa8dffff640 1 --2- 172.21.15.28:0/1595692671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb6714fa0 0x55ecb67108d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.370+0000 7fa8dffff640 1 -- 172.21.15.28:0/1595692671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb670f440 msgr2=0x55ecb6717390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.370+0000 7fa8dffff640 1 --2- 172.21.15.28:0/1595692671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb670f440 0x55ecb6717390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.370+0000 7fa8dffff640 1 -- 172.21.15.28:0/1595692671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecb67275d0 con 0x55ecb670fe30 2024-04-25T01:32:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.370+0000 7fa8e6ffd640 1 --2- 172.21.15.28:0/1595692671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb670f440 0x55ecb6717390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.370+0000 7fa8dffff640 1 --2- 172.21.15.28:0/1595692671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 0x55ecb6720dc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0000c00 tx=0x7fa8d0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=197a95cd9817d1b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.370+0000 7fa8dd7fa640 1 -- 172.21.15.28:0/1595692671 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0056070 con 0x55ecb670fe30 2024-04-25T01:32:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.370+0000 7fa8dd7fa640 1 -- 172.21.15.28:0/1595692671 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8d00046c0 con 0x55ecb670fe30 2024-04-25T01:32:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.370+0000 7fa8dd7fa640 1 -- 172.21.15.28:0/1595692671 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0035070 con 0x55ecb670fe30 2024-04-25T01:32:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.370+0000 7fa8f0614740 1 -- 172.21.15.28:0/1595692671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 msgr2=0x55ecb6720dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.370+0000 7fa8f0614740 1 --2- 172.21.15.28:0/1595692671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 0x55ecb6720dc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0000c00 tx=0x7fa8d0001e70 comp rx=0 tx=0).stop 2024-04-25T01:32:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.370+0000 7fa8f0614740 1 -- 172.21.15.28:0/1595692671 shutdown_connections 2024-04-25T01:32:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.370+0000 7fa8f0614740 1 --2- 172.21.15.28:0/1595692671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb6714fa0 0x55ecb67108d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.371+0000 7fa8f0614740 1 --2- 172.21.15.28:0/1595692671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 0x55ecb6720dc0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.371+0000 7fa8f0614740 1 --2- 172.21.15.28:0/1595692671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb670f440 0x55ecb6717390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.371+0000 7fa8f0614740 1 -- 172.21.15.28:0/1595692671 >> 172.21.15.28:0/1595692671 conn(0x55ecb670aab0 msgr2=0x55ecb670ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.371+0000 7fa8f0614740 1 -- 172.21.15.28:0/1595692671 shutdown_connections 2024-04-25T01:32:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.371+0000 7fa8f0614740 1 -- 172.21.15.28:0/1595692671 wait complete. 2024-04-25T01:32:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.371+0000 7fa8f0614740 1 -- start start 2024-04-25T01:32:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.371+0000 7fa8f0614740 1 Processor -- start 2024-04-25T01:32:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.371+0000 7fa8f0614740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb670f440 0x55ecb671ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.371+0000 7fa8f0614740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 0x55ecb6788490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.371+0000 7fa8f0614740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb6714fa0 0x55ecb678a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.371+0000 7fa8f0614740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ecb67277f0 con 0x55ecb670fe30 2024-04-25T01:32:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.371+0000 7fa8f0614740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ecb6727120 con 0x55ecb6714fa0 2024-04-25T01:32:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.371+0000 7fa8f0614740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ecb6727ec0 con 0x55ecb670f440 2024-04-25T01:32:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.371+0000 7fa8f0614740 10 client.0 _ll_register_callbacks cb 0x7fa8d0005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.371+0000 7fa8dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 0x55ecb6788490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.371+0000 7fa8dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 0x55ecb6788490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41262/0 (socket says 172.21.15.28:41262) 2024-04-25T01:32:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.371+0000 7fa8dffff640 1 -- 172.21.15.28:0/85046434 learned_addr learned my addr 172.21.15.28:0/85046434 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.371+0000 7fa8e6ffd640 1 --2- 172.21.15.28:0/85046434 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb670f440 0x55ecb671ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.372+0000 7fa8e77fe640 1 --2- 172.21.15.28:0/85046434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb6714fa0 0x55ecb678a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.372+0000 7fa8dffff640 1 -- 172.21.15.28:0/85046434 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb670f440 msgr2=0x55ecb671ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.372+0000 7fa8dffff640 1 --2- 172.21.15.28:0/85046434 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb670f440 0x55ecb671ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.372+0000 7fa8dffff640 1 -- 172.21.15.28:0/85046434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb6714fa0 msgr2=0x55ecb678a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.372+0000 7fa8dffff640 1 --2- 172.21.15.28:0/85046434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb6714fa0 0x55ecb678a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.372+0000 7fa8dffff640 1 -- 172.21.15.28:0/85046434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecb678af20 con 0x55ecb670fe30 2024-04-25T01:32:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.372+0000 7fa8e77fe640 1 --2- 172.21.15.28:0/85046434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb6714fa0 0x55ecb678a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.372+0000 7fa8dffff640 1 --2- 172.21.15.28:0/85046434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 0x55ecb6788490 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa8d00058f0 tx=0x7fa8d0031b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bbea2ff92847c1d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.372+0000 7fa8c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.372+0000 7fa8c6ffd640 1 -- 172.21.15.28:0/85046434 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0035070 con 0x55ecb670fe30 2024-04-25T01:32:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.372+0000 7fa8c6ffd640 1 -- 172.21.15.28:0/85046434 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fa8d0004a80 con 0x55ecb670fe30 2024-04-25T01:32:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.372+0000 7fa8f0614740 1 -- 172.21.15.28:0/85046434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8d00046c0 con 0x55ecb670fe30 2024-04-25T01:32:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.372+0000 7fa8c6ffd640 1 -- 172.21.15.28:0/85046434 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa8d0065050 con 0x55ecb670fe30 2024-04-25T01:32:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.372+0000 7fa8f0614740 10 client.15501 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.372+0000 7fa8f0614740 1 -- 172.21.15.28:0/85046434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ecb678c150 con 0x55ecb670fe30 2024-04-25T01:32:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.372+0000 7fa8c6ffd640 1 -- 172.21.15.28:0/85046434 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8d00046c0 con 0x55ecb670fe30 2024-04-25T01:32:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.373+0000 7fa8c6ffd640 1 -- 172.21.15.28:0/85046434 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fa8d00446d0 con 0x55ecb670fe30 2024-04-25T01:32:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.373+0000 7fa8f0614740 10 client.15501 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.373+0000 7fa8f0614740 10 client.15501 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T01:32:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.373+0000 7fa8f0614740 20 client.15501 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.373+0000 7fa8f0614740 1 --2- 172.21.15.28:0/85046434 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ecb67909a0 0x55ecb678f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.373+0000 7fa8f0614740 4 client.15501 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:32:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.373+0000 7fa8f0614740 1 -- 172.21.15.28:0/85046434 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "status"}) -- 0x55ecb667a790 con 0x55ecb67909a0 2024-04-25T01:32:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.373+0000 7fa8e6ffd640 1 --2- 172.21.15.28:0/85046434 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ecb67909a0 0x55ecb678f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.373+0000 7fa8e6ffd640 1 --2- 172.21.15.28:0/85046434 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ecb67909a0 0x55ecb678f6b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=49dabb69c7227b59 server_cookie=2121db2b66f100b6 in_seq=0 out_seq=0 2024-04-25T01:32:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.373+0000 7fa8c6ffd640 10 client.15501 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:32:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.374+0000 7fa8c6ffd640 1 -- 172.21.15.28:0/85046434 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ecb667a790 con 0x55ecb67909a0 2024-04-25T01:32:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.374+0000 7fa8c6ffd640 10 client.15501 handle_command_reply: tid=0 2024-04-25T01:32:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.374+0000 7fa8f0614740 1 client.15501 shutdown 2024-04-25T01:32:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.374+0000 7fa8f0614740 1 -- 172.21.15.28:0/85046434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 msgr2=0x55ecb6788490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.374+0000 7fa8f0614740 1 --2- 172.21.15.28:0/85046434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 0x55ecb6788490 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa8d00058f0 tx=0x7fa8d0031b90 comp rx=0 tx=0).stop 2024-04-25T01:32:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.374+0000 7fa8f0614740 1 -- 172.21.15.28:0/85046434 shutdown_connections 2024-04-25T01:32:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.374+0000 7fa8f0614740 1 --2- 172.21.15.28:0/85046434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ecb6714fa0 0x55ecb678a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.374+0000 7fa8f0614740 1 --2- 172.21.15.28:0/85046434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ecb670fe30 0x55ecb6788490 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.374+0000 7fa8f0614740 1 --2- 172.21.15.28:0/85046434 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ecb67909a0 0x55ecb678f6b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.374+0000 7fa8f0614740 1 --2- 172.21.15.28:0/85046434 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ecb670f440 0x55ecb671ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.374+0000 7fa8c6ffd640 0 client.15501 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:32:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.374+0000 7fa8f0614740 1 -- 172.21.15.28:0/85046434 >> 172.21.15.28:0/85046434 conn(0x55ecb670aab0 msgr2=0x55ecb6705690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.374+0000 7fa8f0614740 1 -- 172.21.15.28:0/85046434 shutdown_connections 2024-04-25T01:32:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.374+0000 7fa8f0614740 1 -- 172.21.15.28:0/85046434 wait complete. 2024-04-25T01:32:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.374+0000 7fa8f0614740 20 client.15501 trim_cache size 0 max 16384 2024-04-25T01:32:52.359 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:32:52.359 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "0e88f60a-02a2-11ef-bc93-c7b262605968", 2024-04-25T01:32:52.359 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-04-25T01:32:52.359 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24473, 2024-04-25T01:32:52.359 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-25T01:32:52.359 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-25T01:32:52.359 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-25T01:32:52.360 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 112.94141426500001, 2024-04-25T01:32:52.360 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-04-25T01:32:52.360 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-04-25T01:32:52.360 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 91, 2024-04-25T01:32:52.360 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 156.03702554899999 2024-04-25T01:32:52.360 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:32:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.375+0000 7fa8ed82a640 1 -- 172.21.15.28:0/1200876266 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa8d40a2b90 msgr2=0x7fa8d40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.375+0000 7fa8ed82a640 1 --2- 172.21.15.28:0/1200876266 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa8d40a2b90 0x7fa8d40a5050 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0022000 tx=0x7fa8e0020000 comp rx=0 tx=0).stop 2024-04-25T01:32:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.375+0000 7fa8ed82a640 1 -- 172.21.15.28:0/1200876266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e81006a0 msgr2=0x7fa8e8197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.375+0000 7fa8ed82a640 1 --2- 172.21.15.28:0/1200876266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e81006a0 0x7fa8e8197b10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa8d800d950 tx=0x7fa8d800de10 comp rx=0 tx=0).stop 2024-04-25T01:32:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.375+0000 7fa8ed82a640 1 -- 172.21.15.28:0/1200876266 shutdown_connections 2024-04-25T01:32:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.375+0000 7fa8ed82a640 1 --2- 172.21.15.28:0/1200876266 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fa8d40a2b90 0x7fa8d40a5050 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.375+0000 7fa8ed82a640 1 --2- 172.21.15.28:0/1200876266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e8102250 0x7fa8e819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.375+0000 7fa8ed82a640 1 --2- 172.21.15.28:0/1200876266 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa8e8101890 0x7fa8e8198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.375+0000 7fa8ed82a640 1 --2- 172.21.15.28:0/1200876266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e81006a0 0x7fa8e8197b10 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.375+0000 7fa8ed82a640 1 -- 172.21.15.28:0/1200876266 >> 172.21.15.28:0/1200876266 conn(0x7fa8e80fbc30 msgr2=0x7fa8e80fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.375+0000 7fa8ed82a640 1 -- 172.21.15.28:0/1200876266 shutdown_connections 2024-04-25T01:32:52.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.375+0000 7fa8ed82a640 1 -- 172.21.15.28:0/1200876266 wait complete. 2024-04-25T01:32:52.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T01:32:52.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:52 smithi130 ceph-mon[32702]: pgmap v368: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:52.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:52 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/840490266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:32:52.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:52 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1392132689' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:52.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:52 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1822658888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.490+0000 7f1a2a75a640 1 -- 172.21.15.28:0/2434088006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a24102990 msgr2=0x7f1a24102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.490+0000 7f1a2a75a640 1 --2- 172.21.15.28:0/2434088006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a24102990 0x7f1a24102e10 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c009900 tx=0x7f1a0c02a410 comp rx=0 tx=0).stop 2024-04-25T01:32:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.490+0000 7f1a2a75a640 1 -- 172.21.15.28:0/2434088006 shutdown_connections 2024-04-25T01:32:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.490+0000 7f1a2a75a640 1 --2- 172.21.15.28:0/2434088006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a24103350 0x7f1a24109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.490+0000 7f1a2a75a640 1 --2- 172.21.15.28:0/2434088006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a24102990 0x7f1a24102e10 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.490+0000 7f1a2a75a640 1 --2- 172.21.15.28:0/2434088006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a241017a0 0x7f1a24101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.490+0000 7f1a2a75a640 1 -- 172.21.15.28:0/2434088006 >> 172.21.15.28:0/2434088006 conn(0x7f1a240fcd10 msgr2=0x7f1a240ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.490+0000 7f1a2a75a640 1 -- 172.21.15.28:0/2434088006 shutdown_connections 2024-04-25T01:32:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.490+0000 7f1a2a75a640 1 -- 172.21.15.28:0/2434088006 wait complete. 2024-04-25T01:32:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.491+0000 7f1a2a75a640 1 Processor -- start 2024-04-25T01:32:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.491+0000 7f1a2a75a640 1 -- start start 2024-04-25T01:32:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.491+0000 7f1a2a75a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a241017a0 0x7f1a241979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.491+0000 7f1a2a75a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a24102990 0x7f1a24197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.491+0000 7f1a2a75a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a24103350 0x7f1a2419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.491+0000 7f1a2a75a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2410d420 con 0x7f1a24102990 2024-04-25T01:32:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.491+0000 7f1a2a75a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2410d2a0 con 0x7f1a24103350 2024-04-25T01:32:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.491+0000 7f1a2a75a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2410d5a0 con 0x7f1a241017a0 2024-04-25T01:32:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.491+0000 7f1a237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a24102990 0x7f1a24197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.491+0000 7f1a237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a24102990 0x7f1a24197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41270/0 (socket says 172.21.15.28:41270) 2024-04-25T01:32:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.491+0000 7f1a237fe640 1 -- 172.21.15.28:0/1102701344 learned_addr learned my addr 172.21.15.28:0/1102701344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.491+0000 7f1a23fff640 1 --2- 172.21.15.28:0/1102701344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a241017a0 0x7f1a241979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.491+0000 7f1a28cf7640 1 --2- 172.21.15.28:0/1102701344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a24103350 0x7f1a2419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.492+0000 7f1a237fe640 1 -- 172.21.15.28:0/1102701344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a241017a0 msgr2=0x7f1a241979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.492+0000 7f1a237fe640 1 --2- 172.21.15.28:0/1102701344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a241017a0 0x7f1a241979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.492+0000 7f1a237fe640 1 -- 172.21.15.28:0/1102701344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a24103350 msgr2=0x7f1a2419ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.492+0000 7f1a237fe640 1 --2- 172.21.15.28:0/1102701344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a24103350 0x7f1a2419ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.492+0000 7f1a237fe640 1 -- 172.21.15.28:0/1102701344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a2419f600 con 0x7f1a24102990 2024-04-25T01:32:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.492+0000 7f1a23fff640 1 --2- 172.21.15.28:0/1102701344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a241017a0 0x7f1a241979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.492+0000 7f1a28cf7640 1 --2- 172.21.15.28:0/1102701344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a24103350 0x7f1a2419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:32:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.492+0000 7f1a237fe640 1 --2- 172.21.15.28:0/1102701344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a24102990 0x7f1a24197f20 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c009230 tx=0x7f1a0c004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.492+0000 7f1a217fa640 1 -- 172.21.15.28:0/1102701344 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c03f030 con 0x7f1a24102990 2024-04-25T01:32:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.492+0000 7f1a217fa640 1 -- 172.21.15.28:0/1102701344 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a0c003ce0 con 0x7f1a24102990 2024-04-25T01:32:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.492+0000 7f1a217fa640 1 -- 172.21.15.28:0/1102701344 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c035070 con 0x7f1a24102990 2024-04-25T01:32:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.492+0000 7f1a2a75a640 1 -- 172.21.15.28:0/1102701344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a2419f890 con 0x7f1a24102990 2024-04-25T01:32:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.492+0000 7f1a2a75a640 1 -- 172.21.15.28:0/1102701344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a2419fd80 con 0x7f1a24102990 2024-04-25T01:32:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.494+0000 7f1a217fa640 1 -- 172.21.15.28:0/1102701344 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1a0c042030 con 0x7f1a24102990 2024-04-25T01:32:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.494+0000 7f1a2d461740 1 Processor -- start 2024-04-25T01:32:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.494+0000 7f1a2d461740 1 -- start start 2024-04-25T01:32:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.494+0000 7f1a2d461740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1dc440 0x55b45c1dc860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.494+0000 7f1a2d461740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dce30 0x55b45c1e1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.494+0000 7f1a2d461740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 0x55b45c1e4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a2d461740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b45c13ff00 con 0x55b45c1e1fa0 2024-04-25T01:32:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a2d461740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b45c0e8750 con 0x55b45c1dc440 2024-04-25T01:32:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a2d461740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b45c13bec0 con 0x55b45c1dce30 2024-04-25T01:32:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 0x55b45c1e4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a217fa640 1 --2- 172.21.15.28:0/1102701344 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1a100a2b90 0x7f1a100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 0x55b45c1e4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41280/0 (socket says 172.21.15.28:41280) 2024-04-25T01:32:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a237fe640 1 -- 172.21.15.28:0/1978317910 learned_addr learned my addr 172.21.15.28:0/1978317910 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a23fff640 1 --2- 172.21.15.28:0/1978317910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dce30 0x55b45c1e1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a28cf7640 1 --2- 172.21.15.28:0/1978317910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1dc440 0x55b45c1dc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a28cf7640 1 --2- 172.21.15.28:0/1102701344 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1a100a2b90 0x7f1a100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a217fa640 1 -- 172.21.15.28:0/1102701344 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f1a0c0efe40 con 0x7f1a24102990 2024-04-25T01:32:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a237fe640 1 -- 172.21.15.28:0/1978317910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dce30 msgr2=0x55b45c1e1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a237fe640 1 --2- 172.21.15.28:0/1978317910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dce30 0x55b45c1e1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a237fe640 1 -- 172.21.15.28:0/1978317910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1dc440 msgr2=0x55b45c1dc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a237fe640 1 --2- 172.21.15.28:0/1978317910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1dc440 0x55b45c1dc860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a237fe640 1 -- 172.21.15.28:0/1978317910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b45be542d0 con 0x55b45c1e1fa0 2024-04-25T01:32:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a237fe640 1 --2- 172.21.15.28:0/1978317910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 0x55b45c1e4430 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c0ed5e0 tx=0x7f1a0c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e6ea8e06c681343 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a28cf7640 1 --2- 172.21.15.28:0/1102701344 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1a100a2b90 0x7f1a100a5050 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x55b45c1dd690 tx=0x7f1a1c0094a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a1a7fc640 1 -- 172.21.15.28:0/1978317910 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c049070 con 0x55b45c1e1fa0 2024-04-25T01:32:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.495+0000 7f1a1a7fc640 1 -- 172.21.15.28:0/1978317910 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a0c0bca40 con 0x55b45c1e1fa0 2024-04-25T01:32:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.496+0000 7f1a1a7fc640 1 -- 172.21.15.28:0/1978317910 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c0ba6d0 con 0x55b45c1e1fa0 2024-04-25T01:32:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.496+0000 7f1a2d461740 1 -- 172.21.15.28:0/1978317910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 msgr2=0x55b45c1e4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.496+0000 7f1a2d461740 1 --2- 172.21.15.28:0/1978317910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 0x55b45c1e4430 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c0ed5e0 tx=0x7f1a0c009260 comp rx=0 tx=0).stop 2024-04-25T01:32:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.496+0000 7f1a2d461740 1 -- 172.21.15.28:0/1978317910 shutdown_connections 2024-04-25T01:32:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.496+0000 7f1a2d461740 1 --2- 172.21.15.28:0/1978317910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 0x55b45c1e4430 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.496+0000 7f1a2d461740 1 --2- 172.21.15.28:0/1978317910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dce30 0x55b45c1e1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.496+0000 7f1a2d461740 1 --2- 172.21.15.28:0/1978317910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1dc440 0x55b45c1dc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.496+0000 7f1a2d461740 1 -- 172.21.15.28:0/1978317910 >> 172.21.15.28:0/1978317910 conn(0x55b45c1d7ab0 msgr2=0x55b45c1d7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.496+0000 7f1a2d461740 1 -- 172.21.15.28:0/1978317910 shutdown_connections 2024-04-25T01:32:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.496+0000 7f1a2d461740 1 -- 172.21.15.28:0/1978317910 wait complete. 2024-04-25T01:32:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.497+0000 7f1a2d461740 1 -- start start 2024-04-25T01:32:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.497+0000 7f1a2d461740 1 Processor -- start 2024-04-25T01:32:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.497+0000 7f1a2d461740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1dc440 0x55b45c1e6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.497+0000 7f1a2d461740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dce30 0x55b45c1e6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.497+0000 7f1a2d461740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1e1fa0 0x55b45c1eda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.497+0000 7f1a2d461740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b45c13ff00 con 0x55b45c1dc440 2024-04-25T01:32:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.497+0000 7f1a2d461740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b45c0fbcf0 con 0x55b45c1e1fa0 2024-04-25T01:32:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.497+0000 7f1a2d461740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b45c1e4d10 con 0x55b45c1dce30 2024-04-25T01:32:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.497+0000 7f1a2d461740 10 client.0 _ll_register_callbacks cb 0x55b45c0fb6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.497+0000 7f1a23fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1dc440 0x55b45c1e6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.497+0000 7f1a23fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1dc440 0x55b45c1e6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41294/0 (socket says 172.21.15.28:41294) 2024-04-25T01:32:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.497+0000 7f1a23fff640 1 -- 172.21.15.28:0/1026703779 learned_addr learned my addr 172.21.15.28:0/1026703779 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.497+0000 7f1a237fe640 1 --2- 172.21.15.28:0/1026703779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dce30 0x55b45c1e6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.497+0000 7f1a28cf7640 1 --2- 172.21.15.28:0/1026703779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1e1fa0 0x55b45c1eda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.497+0000 7f1a23fff640 1 -- 172.21.15.28:0/1026703779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dce30 msgr2=0x55b45c1e6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.497+0000 7f1a23fff640 1 --2- 172.21.15.28:0/1026703779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dce30 0x55b45c1e6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.497+0000 7f1a23fff640 1 -- 172.21.15.28:0/1026703779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1e1fa0 msgr2=0x55b45c1eda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.497+0000 7f1a23fff640 1 --2- 172.21.15.28:0/1026703779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1e1fa0 0x55b45c1eda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.497+0000 7f1a23fff640 1 -- 172.21.15.28:0/1026703779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b45bf1e6b0 con 0x55b45c1dc440 2024-04-25T01:32:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.498+0000 7f1a23fff640 1 --2- 172.21.15.28:0/1026703779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1dc440 0x55b45c1e6580 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f1a14011830 tx=0x7f1a14011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aac844d9d9295f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.498+0000 7f1a037fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.498+0000 7f1a037fe640 1 -- 172.21.15.28:0/1026703779 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a1400f940 con 0x55b45c1dc440 2024-04-25T01:32:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.498+0000 7f1a2d461740 1 -- 172.21.15.28:0/1026703779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a0c0bca40 con 0x55b45c1dc440 2024-04-25T01:32:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.498+0000 7f1a037fe640 1 -- 172.21.15.28:0/1026703779 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f1a1400fae0 con 0x55b45c1dc440 2024-04-25T01:32:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.498+0000 7f1a037fe640 1 -- 172.21.15.28:0/1026703779 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1a14022050 con 0x55b45c1dc440 2024-04-25T01:32:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.498+0000 7f1a2d461740 10 client.15513 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.498+0000 7f1a2d461740 1 -- 172.21.15.28:0/1026703779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b45c1f0b40 con 0x55b45c1dc440 2024-04-25T01:32:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.498+0000 7f1a037fe640 1 -- 172.21.15.28:0/1026703779 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a0c0bca40 con 0x55b45c1dc440 2024-04-25T01:32:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.498+0000 7f1a037fe640 1 -- 172.21.15.28:0/1026703779 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f1a14018030 con 0x55b45c1dc440 2024-04-25T01:32:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.498+0000 7f1a2d461740 10 client.15513 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.498+0000 7f1a2d461740 10 client.15513 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:32:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.498+0000 7f1a2d461740 20 client.15513 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.498+0000 7f1a2d461740 1 --2- 172.21.15.28:0/1026703779 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55b45c1f3230 0x55b45c1f46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.499+0000 7f1a2d461740 4 client.15513 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.499+0000 7f1a2d461740 1 -- 172.21.15.28:0/1026703779 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b45c14d440 con 0x55b45c1f3230 2024-04-25T01:32:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.499+0000 7f1a237fe640 1 --2- 172.21.15.28:0/1026703779 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55b45c1f3230 0x55b45c1f46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.499+0000 7f1a037fe640 10 client.15513 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:32:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.499+0000 7f1a237fe640 1 --2- 172.21.15.28:0/1026703779 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55b45c1f3230 0x55b45c1f46a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=91a9460ab9c84123 server_cookie=60946ba1ddfc6c7e in_seq=0 out_seq=0 2024-04-25T01:32:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.500+0000 7f1a037fe640 1 -- 172.21.15.28:0/1026703779 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b45c14d440 con 0x55b45c1f3230 2024-04-25T01:32:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.500+0000 7f1a037fe640 10 client.15513 handle_command_reply: tid=0 2024-04-25T01:32:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.500+0000 7f1a2d461740 1 client.15513 shutdown 2024-04-25T01:32:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.500+0000 7f1a2d461740 1 -- 172.21.15.28:0/1026703779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1dc440 msgr2=0x55b45c1e6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.500+0000 7f1a2d461740 1 --2- 172.21.15.28:0/1026703779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1dc440 0x55b45c1e6580 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f1a14011830 tx=0x7f1a14011cf0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.500+0000 7f1a2d461740 1 -- 172.21.15.28:0/1026703779 shutdown_connections 2024-04-25T01:32:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.500+0000 7f1a2d461740 1 --2- 172.21.15.28:0/1026703779 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55b45c1f3230 0x55b45c1f46a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.500+0000 7f1a2d461740 1 --2- 172.21.15.28:0/1026703779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1e1fa0 0x55b45c1eda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.500+0000 7f1a2d461740 1 --2- 172.21.15.28:0/1026703779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dce30 0x55b45c1e6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.500+0000 7f1a037fe640 0 client.15513 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:32:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.500+0000 7f1a2d461740 1 --2- 172.21.15.28:0/1026703779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1dc440 0x55b45c1e6580 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.500+0000 7f1a2d461740 1 -- 172.21.15.28:0/1026703779 >> 172.21.15.28:0/1026703779 conn(0x55b45c1d7ab0 msgr2=0x55b45c1d6ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.500+0000 7f1a2d461740 1 -- 172.21.15.28:0/1026703779 shutdown_connections 2024-04-25T01:32:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.500+0000 7f1a2d461740 1 -- 172.21.15.28:0/1026703779 wait complete. 2024-04-25T01:32:52.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.500+0000 7f1a2d461740 20 client.15513 trim_cache size 0 max 16384 2024-04-25T01:32:52.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:52 smithi032 ceph-mon[32461]: pgmap v368: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:32:52.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:52 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/840490266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:32:52.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:52 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1392132689' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:32:52.486 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:52 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1822658888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:52.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.518+0000 7f1a2d461740 1 Processor -- start 2024-04-25T01:32:52.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.519+0000 7f1a2d461740 1 -- start start 2024-04-25T01:32:52.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.519+0000 7f1a2d461740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dc440 0x55b45c1d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.519+0000 7f1a2d461740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1dce30 0x55b45c1d6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.519+0000 7f1a2d461740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 0x55b45c1d6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.519+0000 7f1a2d461740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b45c1f29e0 con 0x55b45c1e1fa0 2024-04-25T01:32:52.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.519+0000 7f1a2d461740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b45c1f30b0 con 0x55b45c1dce30 2024-04-25T01:32:52.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.519+0000 7f1a2d461740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b45c1f3780 con 0x55b45c1dc440 2024-04-25T01:32:52.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.519+0000 7f1a28cf7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 0x55b45c1d6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.519+0000 7f1a28cf7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 0x55b45c1d6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41302/0 (socket says 172.21.15.28:41302) 2024-04-25T01:32:52.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.519+0000 7f1a28cf7640 1 -- 172.21.15.28:0/995009119 learned_addr learned my addr 172.21.15.28:0/995009119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.519+0000 7f1a23fff640 1 --2- 172.21.15.28:0/995009119 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dc440 0x55b45c1d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.519+0000 7f1a237fe640 1 --2- 172.21.15.28:0/995009119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1dce30 0x55b45c1d6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.519+0000 7f1a28cf7640 1 -- 172.21.15.28:0/995009119 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dc440 msgr2=0x55b45c1d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.519+0000 7f1a28cf7640 1 --2- 172.21.15.28:0/995009119 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dc440 0x55b45c1d2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.519+0000 7f1a28cf7640 1 -- 172.21.15.28:0/995009119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1dce30 msgr2=0x55b45c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.519+0000 7f1a28cf7640 1 --2- 172.21.15.28:0/995009119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1dce30 0x55b45c1d6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.519+0000 7f1a28cf7640 1 -- 172.21.15.28:0/995009119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b45c1d78e0 con 0x55b45c1e1fa0 2024-04-25T01:32:52.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.519+0000 7f1a28cf7640 1 --2- 172.21.15.28:0/995009119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 0x55b45c1d6790 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c012a40 tx=0x7f1a1c009770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d6d6586dc95caa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.519+0000 7f1a19ffb640 1 -- 172.21.15.28:0/995009119 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c004b20 con 0x55b45c1e1fa0 2024-04-25T01:32:52.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.520+0000 7f1a19ffb640 1 -- 172.21.15.28:0/995009119 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a1c026070 con 0x55b45c1e1fa0 2024-04-25T01:32:52.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.520+0000 7f1a19ffb640 1 -- 172.21.15.28:0/995009119 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c0056a0 con 0x55b45c1e1fa0 2024-04-25T01:32:52.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.520+0000 7f1a2d461740 1 -- 172.21.15.28:0/995009119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 msgr2=0x55b45c1d6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.520+0000 7f1a2d461740 1 --2- 172.21.15.28:0/995009119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 0x55b45c1d6790 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c012a40 tx=0x7f1a1c009770 comp rx=0 tx=0).stop 2024-04-25T01:32:52.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.520+0000 7f1a2d461740 1 -- 172.21.15.28:0/995009119 shutdown_connections 2024-04-25T01:32:52.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.520+0000 7f1a2d461740 1 --2- 172.21.15.28:0/995009119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 0x55b45c1d6790 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.520+0000 7f1a2d461740 1 --2- 172.21.15.28:0/995009119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1dce30 0x55b45c1d6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.520+0000 7f1a2d461740 1 --2- 172.21.15.28:0/995009119 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dc440 0x55b45c1d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.520+0000 7f1a2d461740 1 -- 172.21.15.28:0/995009119 >> 172.21.15.28:0/995009119 conn(0x55b45c1d7ab0 msgr2=0x55b45c1d7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.520+0000 7f1a2d461740 1 -- 172.21.15.28:0/995009119 shutdown_connections 2024-04-25T01:32:52.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.520+0000 7f1a2d461740 1 -- 172.21.15.28:0/995009119 wait complete. 2024-04-25T01:32:52.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.520+0000 7f1a2d461740 1 -- start start 2024-04-25T01:32:52.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a2d461740 1 Processor -- start 2024-04-25T01:32:52.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a2d461740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1dc440 0x55b45c2571d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a2d461740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dce30 0x55b45c259720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a2d461740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 0x55b45c25bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a2d461740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b45c1f4520 con 0x55b45c1e1fa0 2024-04-25T01:32:52.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a2d461740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b45c1f3e50 con 0x55b45c1dc440 2024-04-25T01:32:52.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a2d461740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b45c1f4bf0 con 0x55b45c1dce30 2024-04-25T01:32:52.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a2d461740 10 client.0 _ll_register_callbacks cb 0x7f1a14011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:52.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a28cf7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 0x55b45c25bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a28cf7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 0x55b45c25bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41304/0 (socket says 172.21.15.28:41304) 2024-04-25T01:32:52.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a28cf7640 1 -- 172.21.15.28:0/843827806 learned_addr learned my addr 172.21.15.28:0/843827806 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a237fe640 1 --2- 172.21.15.28:0/843827806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dce30 0x55b45c259720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a23fff640 1 --2- 172.21.15.28:0/843827806 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1dc440 0x55b45c2571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a28cf7640 1 -- 172.21.15.28:0/843827806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dce30 msgr2=0x55b45c259720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a28cf7640 1 --2- 172.21.15.28:0/843827806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dce30 0x55b45c259720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a28cf7640 1 -- 172.21.15.28:0/843827806 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1dc440 msgr2=0x55b45c2571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a28cf7640 1 --2- 172.21.15.28:0/843827806 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1dc440 0x55b45c2571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a28cf7640 1 -- 172.21.15.28:0/843827806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b45c25c590 con 0x55b45c1e1fa0 2024-04-25T01:32:52.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a28cf7640 1 --2- 172.21.15.28:0/843827806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 0x55b45c25bd10 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c009a20 tx=0x7f1a1c009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25c645d7617059 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.521+0000 7f1a037fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:52.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.522+0000 7f1a037fe640 1 -- 172.21.15.28:0/843827806 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c0109d0 con 0x55b45c1e1fa0 2024-04-25T01:32:52.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.522+0000 7f1a037fe640 1 -- 172.21.15.28:0/843827806 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f1a1c010b70 con 0x55b45c1e1fa0 2024-04-25T01:32:52.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.522+0000 7f1a2d461740 1 -- 172.21.15.28:0/843827806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a1c026070 con 0x55b45c1e1fa0 2024-04-25T01:32:52.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.522+0000 7f1a037fe640 1 -- 172.21.15.28:0/843827806 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1a1c010d70 con 0x55b45c1e1fa0 2024-04-25T01:32:52.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.522+0000 7f1a2d461740 10 client.15519 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:52.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.522+0000 7f1a2d461740 1 -- 172.21.15.28:0/843827806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b45c25d830 con 0x55b45c1e1fa0 2024-04-25T01:32:52.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.522+0000 7f1a037fe640 1 -- 172.21.15.28:0/843827806 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a1c026070 con 0x55b45c1e1fa0 2024-04-25T01:32:52.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.522+0000 7f1a037fe640 1 -- 172.21.15.28:0/843827806 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f1a1c031070 con 0x55b45c1e1fa0 2024-04-25T01:32:52.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.522+0000 7f1a2d461740 10 client.15519 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:52.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.522+0000 7f1a2d461740 10 client.15519 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:32:52.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.522+0000 7f1a2d461740 20 client.15519 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:52.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.522+0000 7f1a2d461740 1 --2- 172.21.15.28:0/843827806 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55b45c2622b0 0x55b45c260fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.522+0000 7f1a2d461740 4 client.15519 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:32:52.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.522+0000 7f1a2d461740 1 -- 172.21.15.28:0/843827806 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "status"}) -- 0x55b45c147790 con 0x55b45c2622b0 2024-04-25T01:32:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.522+0000 7f1a23fff640 1 --2- 172.21.15.28:0/843827806 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55b45c2622b0 0x55b45c260fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.523+0000 7f1a037fe640 10 client.15519 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:32:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.523+0000 7f1a23fff640 1 --2- 172.21.15.28:0/843827806 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55b45c2622b0 0x55b45c260fc0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fa99bdc09c8e6ffb server_cookie=5157555a33ed140c in_seq=0 out_seq=0 2024-04-25T01:32:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.523+0000 7f1a037fe640 1 -- 172.21.15.28:0/843827806 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55b45c147790 con 0x55b45c2622b0 2024-04-25T01:32:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.523+0000 7f1a037fe640 10 client.15519 handle_command_reply: tid=0 2024-04-25T01:32:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.523+0000 7f1a2d461740 1 client.15519 shutdown 2024-04-25T01:32:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.523+0000 7f1a2d461740 1 -- 172.21.15.28:0/843827806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 msgr2=0x55b45c25bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.523+0000 7f1a2d461740 1 --2- 172.21.15.28:0/843827806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 0x55b45c25bd10 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c009a20 tx=0x7f1a1c009a50 comp rx=0 tx=0).stop 2024-04-25T01:32:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.523+0000 7f1a2d461740 1 -- 172.21.15.28:0/843827806 shutdown_connections 2024-04-25T01:32:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.523+0000 7f1a2d461740 1 --2- 172.21.15.28:0/843827806 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55b45c2622b0 0x55b45c260fc0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.523+0000 7f1a2d461740 1 --2- 172.21.15.28:0/843827806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b45c1e1fa0 0x55b45c25bd10 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.523+0000 7f1a2d461740 1 --2- 172.21.15.28:0/843827806 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b45c1dce30 0x55b45c259720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.523+0000 7f1a037fe640 0 client.15519 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:32:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.523+0000 7f1a2d461740 1 --2- 172.21.15.28:0/843827806 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b45c1dc440 0x55b45c2571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.523+0000 7f1a2d461740 1 -- 172.21.15.28:0/843827806 >> 172.21.15.28:0/843827806 conn(0x55b45c1d7ab0 msgr2=0x55b45c1d2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.523+0000 7f1a2d461740 1 -- 172.21.15.28:0/843827806 shutdown_connections 2024-04-25T01:32:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.523+0000 7f1a2d461740 1 -- 172.21.15.28:0/843827806 wait complete. 2024-04-25T01:32:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.524+0000 7f1a2d461740 20 client.15519 trim_cache size 0 max 16384 2024-04-25T01:32:52.509 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:32:52.509 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "0e88f60a-02a2-11ef-bc93-c7b262605968", 2024-04-25T01:32:52.509 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-04-25T01:32:52.510 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24478, 2024-04-25T01:32:52.510 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-25T01:32:52.510 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-25T01:32:52.510 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-25T01:32:52.510 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 114.09667534, 2024-04-25T01:32:52.510 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-04-25T01:32:52.510 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-04-25T01:32:52.510 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 91, 2024-04-25T01:32:52.510 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 153.31012249899999 2024-04-25T01:32:52.510 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:32:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.524+0000 7f1a2a75a640 1 -- 172.21.15.28:0/1102701344 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1a100a2b90 msgr2=0x7f1a100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.524+0000 7f1a2a75a640 1 --2- 172.21.15.28:0/1102701344 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1a100a2b90 0x7f1a100a5050 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x55b45c1dd690 tx=0x7f1a1c0094a0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.524+0000 7f1a2a75a640 1 -- 172.21.15.28:0/1102701344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a24102990 msgr2=0x7f1a24197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.524+0000 7f1a2a75a640 1 --2- 172.21.15.28:0/1102701344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a24102990 0x7f1a24197f20 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c009230 tx=0x7f1a0c004360 comp rx=0 tx=0).stop 2024-04-25T01:32:52.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.524+0000 7f1a2a75a640 1 -- 172.21.15.28:0/1102701344 shutdown_connections 2024-04-25T01:32:52.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.524+0000 7f1a2a75a640 1 --2- 172.21.15.28:0/1102701344 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1a100a2b90 0x7f1a100a5050 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.524+0000 7f1a2a75a640 1 --2- 172.21.15.28:0/1102701344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a24103350 0x7f1a2419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.524+0000 7f1a2a75a640 1 --2- 172.21.15.28:0/1102701344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a24102990 0x7f1a24197f20 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.524+0000 7f1a2a75a640 1 --2- 172.21.15.28:0/1102701344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a241017a0 0x7f1a241979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.524+0000 7f1a2a75a640 1 -- 172.21.15.28:0/1102701344 >> 172.21.15.28:0/1102701344 conn(0x7f1a240fcd10 msgr2=0x7f1a24105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.524+0000 7f1a2a75a640 1 -- 172.21.15.28:0/1102701344 shutdown_connections 2024-04-25T01:32:52.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.524+0000 7f1a2a75a640 1 -- 172.21.15.28:0/1102701344 wait complete. 2024-04-25T01:32:52.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T01:32:52.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.638+0000 7f2773509640 1 -- 172.21.15.28:0/1421659996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f276c102200 msgr2=0x7f276c1045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.638+0000 7f2773509640 1 --2- 172.21.15.28:0/1421659996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f276c102200 0x7f276c1045f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2768009900 tx=0x7f276802a410 comp rx=0 tx=0).stop 2024-04-25T01:32:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.639+0000 7f2773509640 1 -- 172.21.15.28:0/1421659996 shutdown_connections 2024-04-25T01:32:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.639+0000 7f2773509640 1 --2- 172.21.15.28:0/1421659996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f276c107460 0x7f276c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.639+0000 7f2773509640 1 --2- 172.21.15.28:0/1421659996 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f276c104b30 0x7f276c106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.639+0000 7f2773509640 1 --2- 172.21.15.28:0/1421659996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f276c102200 0x7f276c1045f0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.639+0000 7f2773509640 1 -- 172.21.15.28:0/1421659996 >> 172.21.15.28:0/1421659996 conn(0x7f276c0fbbf0 msgr2=0x7f276c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.639+0000 7f2773509640 1 -- 172.21.15.28:0/1421659996 shutdown_connections 2024-04-25T01:32:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.639+0000 7f2773509640 1 -- 172.21.15.28:0/1421659996 wait complete. 2024-04-25T01:32:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.639+0000 7f2773509640 1 Processor -- start 2024-04-25T01:32:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.639+0000 7f2773509640 1 -- start start 2024-04-25T01:32:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.639+0000 7f2773509640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f276c102200 0x7f276c197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.639+0000 7f2773509640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f276c104b30 0x7f276c1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.639+0000 7f2773509640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f276c107460 0x7f276c19f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.639+0000 7f2773509640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f276c10d4a0 con 0x7f276c104b30 2024-04-25T01:32:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.639+0000 7f2773509640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f276c10d320 con 0x7f276c102200 2024-04-25T01:32:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.639+0000 7f2773509640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f276c10d620 con 0x7f276c107460 2024-04-25T01:32:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.639+0000 7f2770aa4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f276c104b30 0x7f276c1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.640+0000 7f2770aa4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f276c104b30 0x7f276c1980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41314/0 (socket says 172.21.15.28:41314) 2024-04-25T01:32:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.640+0000 7f2770aa4640 1 -- 172.21.15.28:0/4284852258 learned_addr learned my addr 172.21.15.28:0/4284852258 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.640+0000 7f27712a5640 1 --2- 172.21.15.28:0/4284852258 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f276c102200 0x7f276c197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.640+0000 7f2771aa6640 1 --2- 172.21.15.28:0/4284852258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f276c107460 0x7f276c19f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.640+0000 7f2770aa4640 1 -- 172.21.15.28:0/4284852258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f276c107460 msgr2=0x7f276c19f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.640+0000 7f2770aa4640 1 --2- 172.21.15.28:0/4284852258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f276c107460 0x7f276c19f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.640+0000 7f2770aa4640 1 -- 172.21.15.28:0/4284852258 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f276c102200 msgr2=0x7f276c197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.640+0000 7f2770aa4640 1 --2- 172.21.15.28:0/4284852258 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f276c102200 0x7f276c197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.640+0000 7f2770aa4640 1 -- 172.21.15.28:0/4284852258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f276c19f7c0 con 0x7f276c104b30 2024-04-25T01:32:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.640+0000 7f2771aa6640 1 --2- 172.21.15.28:0/4284852258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f276c107460 0x7f276c19f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.640+0000 7f2770aa4640 1 --2- 172.21.15.28:0/4284852258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f276c104b30 0x7f276c1980e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f275c00d740 tx=0x7f275c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.640+0000 7f27627fc640 1 -- 172.21.15.28:0/4284852258 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275c017070 con 0x7f276c104b30 2024-04-25T01:32:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.640+0000 7f27627fc640 1 -- 172.21.15.28:0/4284852258 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f275c00de50 con 0x7f276c104b30 2024-04-25T01:32:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.640+0000 7f2773509640 1 -- 172.21.15.28:0/4284852258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f276c19fab0 con 0x7f276c104b30 2024-04-25T01:32:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.640+0000 7f27627fc640 1 -- 172.21.15.28:0/4284852258 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275c004e00 con 0x7f276c104b30 2024-04-25T01:32:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.640+0000 7f2773509640 1 -- 172.21.15.28:0/4284852258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f276c1a0030 con 0x7f276c104b30 2024-04-25T01:32:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.642+0000 7f27627fc640 1 -- 172.21.15.28:0/4284852258 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f275c004fa0 con 0x7f276c104b30 2024-04-25T01:32:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f277620c740 1 Processor -- start 2024-04-25T01:32:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f277620c740 1 -- start start 2024-04-25T01:32:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f277620c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 0x55c016700860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f277620c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016700e30 0x55c016705930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f277620c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016705fa0 0x55c016708430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f277620c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c016663f00 con 0x55c016700440 2024-04-25T01:32:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f277620c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c01660c750 con 0x55c016705fa0 2024-04-25T01:32:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f277620c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c01665fec0 con 0x55c016700e30 2024-04-25T01:32:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f2771aa6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 0x55c016700860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f2771aa6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 0x55c016700860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41330/0 (socket says 172.21.15.28:41330) 2024-04-25T01:32:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f2771aa6640 1 -- 172.21.15.28:0/974136648 learned_addr learned my addr 172.21.15.28:0/974136648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f27627fc640 1 --2- 172.21.15.28:0/4284852258 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f27580a2b90 0x7f27580a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f2770aa4640 1 --2- 172.21.15.28:0/974136648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016705fa0 0x55c016708430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f27712a5640 1 --2- 172.21.15.28:0/974136648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016700e30 0x55c016705930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f2771aa6640 1 --2- 172.21.15.28:0/4284852258 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f27580a2b90 0x7f27580a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f27627fc640 1 -- 172.21.15.28:0/4284852258 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f275c0cee80 con 0x7f276c104b30 2024-04-25T01:32:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f2771aa6640 1 -- 172.21.15.28:0/974136648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016700e30 msgr2=0x55c016705930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f2771aa6640 1 --2- 172.21.15.28:0/974136648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016700e30 0x55c016705930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f2771aa6640 1 -- 172.21.15.28:0/974136648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016705fa0 msgr2=0x55c016708430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f2771aa6640 1 --2- 172.21.15.28:0/974136648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016705fa0 0x55c016708430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.643+0000 7f2771aa6640 1 -- 172.21.15.28:0/974136648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0163782d0 con 0x55c016700440 2024-04-25T01:32:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.644+0000 7f2771aa6640 1 --2- 172.21.15.28:0/974136648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 0x55c016700860 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2764010a30 tx=0x7f2764010ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47bce354e4ef85c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.644+0000 7f27577fe640 1 -- 172.21.15.28:0/974136648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2764020070 con 0x55c016700440 2024-04-25T01:32:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.644+0000 7f27577fe640 1 -- 172.21.15.28:0/974136648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f276401bbd0 con 0x55c016700440 2024-04-25T01:32:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.644+0000 7f27577fe640 1 -- 172.21.15.28:0/974136648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27640235e0 con 0x55c016700440 2024-04-25T01:32:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.645+0000 7f277620c740 1 -- 172.21.15.28:0/974136648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 msgr2=0x55c016700860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.645+0000 7f277620c740 1 --2- 172.21.15.28:0/974136648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 0x55c016700860 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2764010a30 tx=0x7f2764010ef0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.645+0000 7f2771aa6640 1 --2- 172.21.15.28:0/4284852258 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f27580a2b90 0x7f27580a5050 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f276401d000 tx=0x7f2764019000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.645+0000 7f277620c740 1 -- 172.21.15.28:0/974136648 shutdown_connections 2024-04-25T01:32:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.645+0000 7f277620c740 1 --2- 172.21.15.28:0/974136648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016705fa0 0x55c016708430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.645+0000 7f277620c740 1 --2- 172.21.15.28:0/974136648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016700e30 0x55c016705930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.645+0000 7f277620c740 1 --2- 172.21.15.28:0/974136648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 0x55c016700860 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.645+0000 7f277620c740 1 -- 172.21.15.28:0/974136648 >> 172.21.15.28:0/974136648 conn(0x55c0166fbab0 msgr2=0x55c0166fbeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.645+0000 7f277620c740 1 -- 172.21.15.28:0/974136648 shutdown_connections 2024-04-25T01:32:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.645+0000 7f277620c740 1 -- 172.21.15.28:0/974136648 wait complete. 2024-04-25T01:32:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.645+0000 7f277620c740 1 -- start start 2024-04-25T01:32:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.645+0000 7f277620c740 1 Processor -- start 2024-04-25T01:32:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.645+0000 7f277620c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 0x55c01670a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.645+0000 7f277620c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016700e30 0x55c01670ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.645+0000 7f277620c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016705fa0 0x55c016711d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.645+0000 7f277620c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c016708f50 con 0x55c016700440 2024-04-25T01:32:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.645+0000 7f277620c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c01661fcf0 con 0x55c016705fa0 2024-04-25T01:32:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.645+0000 7f277620c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c0167090d0 con 0x55c016700e30 2024-04-25T01:32:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.645+0000 7f277620c740 10 client.0 _ll_register_callbacks cb 0x55c01661f6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.646+0000 7f27712a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 0x55c01670a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.646+0000 7f27712a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 0x55c01670a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41340/0 (socket says 172.21.15.28:41340) 2024-04-25T01:32:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.646+0000 7f27712a5640 1 -- 172.21.15.28:0/2711883209 learned_addr learned my addr 172.21.15.28:0/2711883209 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.646+0000 7f2771aa6640 1 --2- 172.21.15.28:0/2711883209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016705fa0 0x55c016711d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.646+0000 7f2770aa4640 1 --2- 172.21.15.28:0/2711883209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016700e30 0x55c01670ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.646+0000 7f27712a5640 1 -- 172.21.15.28:0/2711883209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016700e30 msgr2=0x55c01670ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.646+0000 7f27712a5640 1 --2- 172.21.15.28:0/2711883209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016700e30 0x55c01670ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.646+0000 7f27712a5640 1 -- 172.21.15.28:0/2711883209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016705fa0 msgr2=0x55c016711d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.646+0000 7f27712a5640 1 --2- 172.21.15.28:0/2711883209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016705fa0 0x55c016711d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.646+0000 7f27712a5640 1 -- 172.21.15.28:0/2711883209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0164426b0 con 0x55c016700440 2024-04-25T01:32:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.646+0000 7f27712a5640 1 --2- 172.21.15.28:0/2711883209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 0x55c01670a7f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f276802a930 tx=0x7f2768031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b64a428e3188f0de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.646+0000 7f2754ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.646+0000 7f2754ff9640 1 -- 172.21.15.28:0/2711883209 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2768035040 con 0x55c016700440 2024-04-25T01:32:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.646+0000 7f2754ff9640 1 -- 172.21.15.28:0/2711883209 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f2768004400 con 0x55c016700440 2024-04-25T01:32:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.646+0000 7f277620c740 1 -- 172.21.15.28:0/2711883209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f276401bbd0 con 0x55c016700440 2024-04-25T01:32:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.646+0000 7f2754ff9640 1 -- 172.21.15.28:0/2711883209 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f276804e050 con 0x55c016700440 2024-04-25T01:32:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.646+0000 7f277620c740 10 client.15531 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.646+0000 7f277620c740 1 -- 172.21.15.28:0/2711883209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c016714dc0 con 0x55c016700440 2024-04-25T01:32:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.647+0000 7f2754ff9640 1 -- 172.21.15.28:0/2711883209 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f276401bbd0 con 0x55c016700440 2024-04-25T01:32:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.647+0000 7f2754ff9640 1 -- 172.21.15.28:0/2711883209 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f2768004bf0 con 0x55c016700440 2024-04-25T01:32:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.647+0000 7f277620c740 10 client.15531 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.647+0000 7f277620c740 10 client.15531 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:32:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.647+0000 7f277620c740 20 client.15531 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.647+0000 7f277620c740 1 --2- 172.21.15.28:0/2711883209 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c016717440 0x55c0167188b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.647+0000 7f277620c740 4 client.15531 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:32:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.647+0000 7f277620c740 1 -- 172.21.15.28:0/2711883209 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c016671440 con 0x55c016717440 2024-04-25T01:32:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.647+0000 7f2770aa4640 1 --2- 172.21.15.28:0/2711883209 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c016717440 0x55c0167188b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.647+0000 7f2770aa4640 1 --2- 172.21.15.28:0/2711883209 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c016717440 0x55c0167188b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83885916cb029677 server_cookie=9ab28fcc1735a0d7 in_seq=0 out_seq=0 2024-04-25T01:32:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.647+0000 7f2754ff9640 10 client.15531 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:32:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.648+0000 7f2754ff9640 1 -- 172.21.15.28:0/2711883209 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c016671440 con 0x55c016717440 2024-04-25T01:32:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.648+0000 7f2754ff9640 10 client.15531 handle_command_reply: tid=0 2024-04-25T01:32:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.648+0000 7f277620c740 1 client.15531 shutdown 2024-04-25T01:32:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.648+0000 7f277620c740 1 -- 172.21.15.28:0/2711883209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 msgr2=0x55c01670a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.648+0000 7f277620c740 1 --2- 172.21.15.28:0/2711883209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 0x55c01670a7f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f276802a930 tx=0x7f2768031a90 comp rx=0 tx=0).stop 2024-04-25T01:32:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.648+0000 7f277620c740 1 -- 172.21.15.28:0/2711883209 shutdown_connections 2024-04-25T01:32:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.648+0000 7f277620c740 1 --2- 172.21.15.28:0/2711883209 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c016717440 0x55c0167188b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.648+0000 7f277620c740 1 --2- 172.21.15.28:0/2711883209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016705fa0 0x55c016711d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.648+0000 7f277620c740 1 --2- 172.21.15.28:0/2711883209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016700e30 0x55c01670ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.648+0000 7f277620c740 1 --2- 172.21.15.28:0/2711883209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 0x55c01670a7f0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.648+0000 7f277620c740 1 -- 172.21.15.28:0/2711883209 >> 172.21.15.28:0/2711883209 conn(0x55c0166fbab0 msgr2=0x55c0166faae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.648+0000 7f2754ff9640 0 client.15531 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:32:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.648+0000 7f277620c740 1 -- 172.21.15.28:0/2711883209 shutdown_connections 2024-04-25T01:32:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.648+0000 7f277620c740 1 -- 172.21.15.28:0/2711883209 wait complete. 2024-04-25T01:32:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.649+0000 7f277620c740 20 client.15531 trim_cache size 0 max 16384 2024-04-25T01:32:52.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.667+0000 7f277620c740 1 Processor -- start 2024-04-25T01:32:52.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.667+0000 7f277620c740 1 -- start start 2024-04-25T01:32:52.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.667+0000 7f277620c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016700440 0x55c016705860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.667+0000 7f277620c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016700e30 0x55c016711c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.667+0000 7f277620c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016705fa0 0x55c0166fa250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.667+0000 7f277620c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c016716bf0 con 0x55c016705fa0 2024-04-25T01:32:52.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.667+0000 7f277620c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c0167172c0 con 0x55c016700440 2024-04-25T01:32:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.667+0000 7f277620c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c016717990 con 0x55c016700e30 2024-04-25T01:32:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.667+0000 7f2771aa6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016705fa0 0x55c0166fa250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.667+0000 7f2771aa6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016705fa0 0x55c0166fa250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41354/0 (socket says 172.21.15.28:41354) 2024-04-25T01:32:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.667+0000 7f2771aa6640 1 -- 172.21.15.28:0/2964676312 learned_addr learned my addr 172.21.15.28:0/2964676312 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.667+0000 7f2770aa4640 1 --2- 172.21.15.28:0/2964676312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016700e30 0x55c016711c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.667+0000 7f27712a5640 1 --2- 172.21.15.28:0/2964676312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016700440 0x55c016705860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.667+0000 7f2771aa6640 1 -- 172.21.15.28:0/2964676312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016700e30 msgr2=0x55c016711c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.667+0000 7f2771aa6640 1 --2- 172.21.15.28:0/2964676312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016700e30 0x55c016711c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.667+0000 7f2771aa6640 1 -- 172.21.15.28:0/2964676312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016700440 msgr2=0x55c016705860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.667+0000 7f2771aa6640 1 --2- 172.21.15.28:0/2964676312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016700440 0x55c016705860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.667+0000 7f2771aa6640 1 -- 172.21.15.28:0/2964676312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c016701210 con 0x55c016705fa0 2024-04-25T01:32:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.667+0000 7f2771aa6640 1 --2- 172.21.15.28:0/2964676312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016705fa0 0x55c0166fa250 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f276400c000 tx=0x7f276400d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b1f9ca7644f592b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.668+0000 7f2756ffd640 1 -- 172.21.15.28:0/2964676312 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2764031020 con 0x55c016705fa0 2024-04-25T01:32:52.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.668+0000 7f2756ffd640 1 -- 172.21.15.28:0/2964676312 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27640103c0 con 0x55c016705fa0 2024-04-25T01:32:52.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.668+0000 7f2756ffd640 1 -- 172.21.15.28:0/2964676312 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2764020040 con 0x55c016705fa0 2024-04-25T01:32:52.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.668+0000 7f277620c740 1 -- 172.21.15.28:0/2964676312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016705fa0 msgr2=0x55c0166fa250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.668+0000 7f277620c740 1 --2- 172.21.15.28:0/2964676312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016705fa0 0x55c0166fa250 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f276400c000 tx=0x7f276400d000 comp rx=0 tx=0).stop 2024-04-25T01:32:52.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.668+0000 7f277620c740 1 -- 172.21.15.28:0/2964676312 shutdown_connections 2024-04-25T01:32:52.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.668+0000 7f277620c740 1 --2- 172.21.15.28:0/2964676312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016705fa0 0x55c0166fa250 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.668+0000 7f277620c740 1 --2- 172.21.15.28:0/2964676312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016700e30 0x55c016711c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.668+0000 7f277620c740 1 --2- 172.21.15.28:0/2964676312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016700440 0x55c016705860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.668+0000 7f277620c740 1 -- 172.21.15.28:0/2964676312 >> 172.21.15.28:0/2964676312 conn(0x55c0166fbab0 msgr2=0x55c0166f6690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.668+0000 7f277620c740 1 -- 172.21.15.28:0/2964676312 shutdown_connections 2024-04-25T01:32:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.668+0000 7f277620c740 1 -- 172.21.15.28:0/2964676312 wait complete. 2024-04-25T01:32:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.668+0000 7f277620c740 1 -- start start 2024-04-25T01:32:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.669+0000 7f277620c740 1 Processor -- start 2024-04-25T01:32:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.669+0000 7f277620c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 0x55c01670dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.669+0000 7f277620c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016700e30 0x55c01670e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.669+0000 7f277620c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016705fa0 0x55c016710cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.669+0000 7f277620c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c016718730 con 0x55c016700440 2024-04-25T01:32:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.669+0000 7f277620c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c016718060 con 0x55c016700e30 2024-04-25T01:32:52.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.669+0000 7f277620c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c016718e00 con 0x55c016705fa0 2024-04-25T01:32:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.669+0000 7f277620c740 10 client.0 _ll_register_callbacks cb 0x7f2768009620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:32:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.669+0000 7f27712a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 0x55c01670dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.669+0000 7f27712a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 0x55c01670dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41362/0 (socket says 172.21.15.28:41362) 2024-04-25T01:32:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.669+0000 7f27712a5640 1 -- 172.21.15.28:0/3052717182 learned_addr learned my addr 172.21.15.28:0/3052717182 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.669+0000 7f2771aa6640 1 --2- 172.21.15.28:0/3052717182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016705fa0 0x55c016710cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.669+0000 7f2770aa4640 1 --2- 172.21.15.28:0/3052717182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016700e30 0x55c01670e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.669+0000 7f27712a5640 1 -- 172.21.15.28:0/3052717182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016705fa0 msgr2=0x55c016710cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.669+0000 7f27712a5640 1 --2- 172.21.15.28:0/3052717182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016705fa0 0x55c016710cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.669+0000 7f27712a5640 1 -- 172.21.15.28:0/3052717182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016700e30 msgr2=0x55c01670e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:32:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.669+0000 7f27712a5640 1 --2- 172.21.15.28:0/3052717182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016700e30 0x55c01670e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.669+0000 7f27712a5640 1 -- 172.21.15.28:0/3052717182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c016711260 con 0x55c016700440 2024-04-25T01:32:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.669+0000 7f2770aa4640 1 --2- 172.21.15.28:0/3052717182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016700e30 0x55c01670e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:32:52.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.670+0000 7f27712a5640 1 --2- 172.21.15.28:0/3052717182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 0x55c01670dec0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f276802a450 tx=0x7f2768005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40dc50b77d393e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.670+0000 7f2754ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:32:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.670+0000 7f2754ff9640 1 -- 172.21.15.28:0/3052717182 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2768035070 con 0x55c016700440 2024-04-25T01:32:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.670+0000 7f2754ff9640 1 -- 172.21.15.28:0/3052717182 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f276804b040 con 0x55c016700440 2024-04-25T01:32:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.670+0000 7f277620c740 1 -- 172.21.15.28:0/3052717182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c01666b790 con 0x55c016700440 2024-04-25T01:32:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.670+0000 7f2754ff9640 1 -- 172.21.15.28:0/3052717182 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f276805f050 con 0x55c016700440 2024-04-25T01:32:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.670+0000 7f277620c740 10 client.15537 fetch_fsmap learned FSMap version 27 2024-04-25T01:32:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.670+0000 7f277620c740 1 -- 172.21.15.28:0/3052717182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c01677d7a0 con 0x55c016700440 2024-04-25T01:32:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.670+0000 7f2754ff9640 1 -- 172.21.15.28:0/3052717182 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c01666b790 con 0x55c016700440 2024-04-25T01:32:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.670+0000 7f2754ff9640 1 -- 172.21.15.28:0/3052717182 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f27680046c0 con 0x55c016700440 2024-04-25T01:32:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.670+0000 7f277620c740 10 client.15537 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:32:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.670+0000 7f277620c740 10 client.15537 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:32:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.670+0000 7f277620c740 20 client.15537 populate_metadata read hostname 'smithi028' 2024-04-25T01:32:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.670+0000 7f277620c740 1 --2- 172.21.15.28:0/3052717182 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c016782320 0x55c01677e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.670+0000 7f277620c740 4 client.15537 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:32:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.670+0000 7f277620c740 1 -- 172.21.15.28:0/3052717182 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "status"}) -- 0x55c0165e9b80 con 0x55c016782320 2024-04-25T01:32:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.671+0000 7f2770aa4640 1 --2- 172.21.15.28:0/3052717182 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c016782320 0x55c01677e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.671+0000 7f2770aa4640 1 --2- 172.21.15.28:0/3052717182 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c016782320 0x55c01677e760 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af6a006672db5240 server_cookie=9c8f8f57b7dbe393 in_seq=0 out_seq=0 2024-04-25T01:32:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.671+0000 7f2754ff9640 10 client.15537 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:32:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.671+0000 7f2754ff9640 1 -- 172.21.15.28:0/3052717182 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55c0165e9b80 con 0x55c016782320 2024-04-25T01:32:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.671+0000 7f2754ff9640 10 client.15537 handle_command_reply: tid=0 2024-04-25T01:32:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.671+0000 7f277620c740 1 client.15537 shutdown 2024-04-25T01:32:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.671+0000 7f277620c740 1 -- 172.21.15.28:0/3052717182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 msgr2=0x55c01670dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.671+0000 7f277620c740 1 --2- 172.21.15.28:0/3052717182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 0x55c01670dec0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f276802a450 tx=0x7f2768005dc0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.671+0000 7f277620c740 1 -- 172.21.15.28:0/3052717182 shutdown_connections 2024-04-25T01:32:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.671+0000 7f277620c740 1 --2- 172.21.15.28:0/3052717182 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c016782320 0x55c01677e760 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.671+0000 7f277620c740 1 --2- 172.21.15.28:0/3052717182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c016705fa0 0x55c016710cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.671+0000 7f277620c740 1 --2- 172.21.15.28:0/3052717182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c016700e30 0x55c01670e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.671+0000 7f277620c740 1 --2- 172.21.15.28:0/3052717182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c016700440 0x55c01670dec0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.671+0000 7f277620c740 1 -- 172.21.15.28:0/3052717182 >> 172.21.15.28:0/3052717182 conn(0x55c0166fbab0 msgr2=0x55c016702260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.671+0000 7f2754ff9640 0 client.15537 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:32:52.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.671+0000 7f277620c740 1 -- 172.21.15.28:0/3052717182 shutdown_connections 2024-04-25T01:32:52.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.671+0000 7f277620c740 1 -- 172.21.15.28:0/3052717182 wait complete. 2024-04-25T01:32:52.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.672+0000 7f277620c740 20 client.15537 trim_cache size 0 max 16384 2024-04-25T01:32:52.657 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:32:52.657 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "0e88f60a-02a2-11ef-bc93-c7b262605968", 2024-04-25T01:32:52.657 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-04-25T01:32:52.657 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24512, 2024-04-25T01:32:52.657 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-25T01:32:52.657 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-25T01:32:52.658 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-25T01:32:52.658 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 138.33898358499999, 2024-04-25T01:32:52.658 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 27, 2024-04-25T01:32:52.658 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-04-25T01:32:52.658 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 91, 2024-04-25T01:32:52.658 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 144.5603223 2024-04-25T01:32:52.658 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:32:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.672+0000 7f2773509640 1 -- 172.21.15.28:0/4284852258 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f27580a2b90 msgr2=0x7f27580a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.672+0000 7f2773509640 1 --2- 172.21.15.28:0/4284852258 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f27580a2b90 0x7f27580a5050 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f276401d000 tx=0x7f2764019000 comp rx=0 tx=0).stop 2024-04-25T01:32:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.672+0000 7f2773509640 1 -- 172.21.15.28:0/4284852258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f276c104b30 msgr2=0x7f276c1980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.672+0000 7f2773509640 1 --2- 172.21.15.28:0/4284852258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f276c104b30 0x7f276c1980e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f275c00d740 tx=0x7f275c00dc00 comp rx=0 tx=0).stop 2024-04-25T01:32:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.672+0000 7f2773509640 1 -- 172.21.15.28:0/4284852258 shutdown_connections 2024-04-25T01:32:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.672+0000 7f2773509640 1 --2- 172.21.15.28:0/4284852258 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f27580a2b90 0x7f27580a5050 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.672+0000 7f2773509640 1 --2- 172.21.15.28:0/4284852258 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f276c107460 0x7f276c19f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.672+0000 7f2773509640 1 --2- 172.21.15.28:0/4284852258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f276c104b30 0x7f276c1980e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.672+0000 7f2773509640 1 --2- 172.21.15.28:0/4284852258 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f276c102200 0x7f276c197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.672+0000 7f2773509640 1 -- 172.21.15.28:0/4284852258 >> 172.21.15.28:0/4284852258 conn(0x7f276c0fbbf0 msgr2=0x7f276c102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.672+0000 7f2773509640 1 -- 172.21.15.28:0/4284852258 shutdown_connections 2024-04-25T01:32:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.672+0000 7f2773509640 1 -- 172.21.15.28:0/4284852258 wait complete. 2024-04-25T01:32:52.661 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T01:32:52.671 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T01:32:52.671 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T01:32:52.671 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T01:32:52.671 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-25T01:32:52.672 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T01:32:52.672 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T01:32:52.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-25T01:32:52.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.797+0000 7fb8e5a76640 1 -- 172.21.15.28:0/3192091880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8e0102990 msgr2=0x7fb8e0102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.797+0000 7fb8e5a76640 1 --2- 172.21.15.28:0/3192091880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8e0102990 0x7fb8e0102e10 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4009900 tx=0x7fb8d402a410 comp rx=0 tx=0).stop 2024-04-25T01:32:52.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.797+0000 7fb8e5a76640 1 -- 172.21.15.28:0/3192091880 shutdown_connections 2024-04-25T01:32:52.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.797+0000 7fb8e5a76640 1 --2- 172.21.15.28:0/3192091880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e0103350 0x7fb8e0109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.797+0000 7fb8e5a76640 1 --2- 172.21.15.28:0/3192091880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8e0102990 0x7fb8e0102e10 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.797+0000 7fb8e5a76640 1 --2- 172.21.15.28:0/3192091880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb8e01017a0 0x7fb8e0101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.797+0000 7fb8e5a76640 1 -- 172.21.15.28:0/3192091880 >> 172.21.15.28:0/3192091880 conn(0x7fb8e00fcd10 msgr2=0x7fb8e00ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:52.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.797+0000 7fb8e5a76640 1 -- 172.21.15.28:0/3192091880 shutdown_connections 2024-04-25T01:32:52.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.797+0000 7fb8e5a76640 1 -- 172.21.15.28:0/3192091880 wait complete. 2024-04-25T01:32:52.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.797+0000 7fb8e5a76640 1 Processor -- start 2024-04-25T01:32:52.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.798+0000 7fb8e5a76640 1 -- start start 2024-04-25T01:32:52.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.798+0000 7fb8e5a76640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb8e01017a0 0x7fb8e0197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.798+0000 7fb8e5a76640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e0102990 0x7fb8e0197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.798+0000 7fb8e5a76640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8e0103350 0x7fb8e019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.798+0000 7fb8e5a76640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e010d470 con 0x7fb8e0103350 2024-04-25T01:32:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.798+0000 7fb8e5a76640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e010d2f0 con 0x7fb8e0102990 2024-04-25T01:32:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.798+0000 7fb8e5a76640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e010d5f0 con 0x7fb8e01017a0 2024-04-25T01:32:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.798+0000 7fb8dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8e0103350 0x7fb8e019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.798+0000 7fb8dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8e0103350 0x7fb8e019efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41392/0 (socket says 172.21.15.28:41392) 2024-04-25T01:32:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.798+0000 7fb8dffff640 1 -- 172.21.15.28:0/2521816003 learned_addr learned my addr 172.21.15.28:0/2521816003 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.798+0000 7fb8df7fe640 1 --2- 172.21.15.28:0/2521816003 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb8e01017a0 0x7fb8e0197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.798+0000 7fb8deffd640 1 --2- 172.21.15.28:0/2521816003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e0102990 0x7fb8e0197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.798+0000 7fb8dffff640 1 -- 172.21.15.28:0/2521816003 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb8e01017a0 msgr2=0x7fb8e0197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.798+0000 7fb8dffff640 1 --2- 172.21.15.28:0/2521816003 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb8e01017a0 0x7fb8e0197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.798+0000 7fb8dffff640 1 -- 172.21.15.28:0/2521816003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e0102990 msgr2=0x7fb8e0197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.798+0000 7fb8dffff640 1 --2- 172.21.15.28:0/2521816003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e0102990 0x7fb8e0197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.798+0000 7fb8dffff640 1 -- 172.21.15.28:0/2521816003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e019f620 con 0x7fb8e0103350 2024-04-25T01:32:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.798+0000 7fb8dffff640 1 --2- 172.21.15.28:0/2521816003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8e0103350 0x7fb8e019efb0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800eaa0 tx=0x7fb8d800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.799+0000 7fb8dcff9640 1 -- 172.21.15.28:0/2521816003 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d800cdb0 con 0x7fb8e0103350 2024-04-25T01:32:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.799+0000 7fb8dcff9640 1 -- 172.21.15.28:0/2521816003 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8d8005020 con 0x7fb8e0103350 2024-04-25T01:32:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.799+0000 7fb8e5a76640 1 -- 172.21.15.28:0/2521816003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8e019f910 con 0x7fb8e0103350 2024-04-25T01:32:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.799+0000 7fb8dcff9640 1 -- 172.21.15.28:0/2521816003 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d80116b0 con 0x7fb8e0103350 2024-04-25T01:32:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.799+0000 7fb8e5a76640 1 -- 172.21.15.28:0/2521816003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8e019fe10 con 0x7fb8e0103350 2024-04-25T01:32:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.800+0000 7fb8e5a76640 1 -- 172.21.15.28:0/2521816003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8e01089f0 con 0x7fb8e0103350 2024-04-25T01:32:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.801+0000 7fb8dcff9640 1 -- 172.21.15.28:0/2521816003 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb8d8005560 con 0x7fb8e0103350 2024-04-25T01:32:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.801+0000 7fb8dcff9640 1 --2- 172.21.15.28:0/2521816003 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb8c80a2c30 0x7fb8c80a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.802+0000 7fb8dcff9640 1 -- 172.21.15.28:0/2521816003 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fb8d80cf5f0 con 0x7fb8e0103350 2024-04-25T01:32:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.802+0000 7fb8df7fe640 1 --2- 172.21.15.28:0/2521816003 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb8c80a2c30 0x7fb8c80a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.802+0000 7fb8df7fe640 1 --2- 172.21.15.28:0/2521816003 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb8c80a2c30 0x7fb8c80a50f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb8d00058c0 tx=0x7fb8d000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:52.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:52.806+0000 7fb8dcff9640 1 -- 172.21.15.28:0/2521816003 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb8d8016030 con 0x7fb8e0103350 2024-04-25T01:32:53.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.042+0000 7fb8e5a76640 1 -- 172.21.15.28:0/2521816003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fb8e0062340 con 0x7fb8e0103350 2024-04-25T01:32:53.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.162+0000 7fb8dcff9640 1 -- 172.21.15.28:0/2521816003 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fb8d809c1f0 con 0x7fb8e0103350 2024-04-25T01:32:53.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.164+0000 7fb8e5a76640 1 -- 172.21.15.28:0/2521816003 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb8c80a2c30 msgr2=0x7fb8c80a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:53.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.164+0000 7fb8e5a76640 1 --2- 172.21.15.28:0/2521816003 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb8c80a2c30 0x7fb8c80a50f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb8d00058c0 tx=0x7fb8d000a2d0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.164+0000 7fb8e5a76640 1 -- 172.21.15.28:0/2521816003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8e0103350 msgr2=0x7fb8e019efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:53.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.164+0000 7fb8e5a76640 1 --2- 172.21.15.28:0/2521816003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8e0103350 0x7fb8e019efb0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800eaa0 tx=0x7fb8d800ef60 comp rx=0 tx=0).stop 2024-04-25T01:32:53.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.164+0000 7fb8e5a76640 1 -- 172.21.15.28:0/2521816003 shutdown_connections 2024-04-25T01:32:53.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.164+0000 7fb8e5a76640 1 --2- 172.21.15.28:0/2521816003 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fb8c80a2c30 0x7fb8c80a50f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.164+0000 7fb8e5a76640 1 --2- 172.21.15.28:0/2521816003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8e0103350 0x7fb8e019efb0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.164+0000 7fb8e5a76640 1 --2- 172.21.15.28:0/2521816003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8e0102990 0x7fb8e0197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.164+0000 7fb8e5a76640 1 --2- 172.21.15.28:0/2521816003 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb8e01017a0 0x7fb8e0197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.164+0000 7fb8e5a76640 1 -- 172.21.15.28:0/2521816003 >> 172.21.15.28:0/2521816003 conn(0x7fb8e00fcd10 msgr2=0x7fb8e0105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:53.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.164+0000 7fb8e5a76640 1 -- 172.21.15.28:0/2521816003 shutdown_connections 2024-04-25T01:32:53.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.164+0000 7fb8e5a76640 1 -- 172.21.15.28:0/2521816003 wait complete. 2024-04-25T01:32:53.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-25T01:32:53.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.291+0000 7f1bd32f5640 1 -- 172.21.15.28:0/804612875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bcc102250 msgr2=0x7f1bcc108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:53.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.291+0000 7f1bd32f5640 1 --2- 172.21.15.28:0/804612875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bcc102250 0x7f1bcc108ae0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4009ca0 tx=0x7f1bc402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.291+0000 7f1bd32f5640 1 -- 172.21.15.28:0/804612875 shutdown_connections 2024-04-25T01:32:53.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.291+0000 7f1bd32f5640 1 --2- 172.21.15.28:0/804612875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bcc102250 0x7f1bcc108ae0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.291+0000 7f1bd32f5640 1 --2- 172.21.15.28:0/804612875 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1bcc101890 0x7f1bcc101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.291+0000 7f1bd32f5640 1 --2- 172.21.15.28:0/804612875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bcc1006a0 0x7f1bcc100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.291+0000 7f1bd32f5640 1 -- 172.21.15.28:0/804612875 >> 172.21.15.28:0/804612875 conn(0x7f1bcc0fbc10 msgr2=0x7f1bcc0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:53.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.291+0000 7f1bd32f5640 1 -- 172.21.15.28:0/804612875 shutdown_connections 2024-04-25T01:32:53.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.291+0000 7f1bd32f5640 1 -- 172.21.15.28:0/804612875 wait complete. 2024-04-25T01:32:53.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.292+0000 7f1bd32f5640 1 Processor -- start 2024-04-25T01:32:53.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.292+0000 7f1bd32f5640 1 -- start start 2024-04-25T01:32:53.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.292+0000 7f1bd32f5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bcc1006a0 0x7f1bcc104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:53.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.292+0000 7f1bd32f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bcc101890 0x7f1bcc102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:53.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.292+0000 7f1bd32f5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1bcc102250 0x7f1bcc1031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:53.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.292+0000 7f1bd32f5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1bcc10c300 con 0x7f1bcc101890 2024-04-25T01:32:53.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.292+0000 7f1bd32f5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1bcc10c180 con 0x7f1bcc1006a0 2024-04-25T01:32:53.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.292+0000 7f1bd32f5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1bcc10c480 con 0x7f1bcc102250 2024-04-25T01:32:53.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.292+0000 7f1bd0890640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bcc101890 0x7f1bcc102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:53.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.292+0000 7f1bd1091640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bcc1006a0 0x7f1bcc104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:53.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.292+0000 7f1bd0890640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bcc101890 0x7f1bcc102c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41446/0 (socket says 172.21.15.28:41446) 2024-04-25T01:32:53.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.292+0000 7f1bd0890640 1 -- 172.21.15.28:0/2936490070 learned_addr learned my addr 172.21.15.28:0/2936490070 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:53.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.292+0000 7f1bd1892640 1 --2- 172.21.15.28:0/2936490070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1bcc102250 0x7f1bcc1031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:53.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.293+0000 7f1bd0890640 1 -- 172.21.15.28:0/2936490070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1bcc102250 msgr2=0x7f1bcc1031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:53.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.293+0000 7f1bd0890640 1 --2- 172.21.15.28:0/2936490070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1bcc102250 0x7f1bcc1031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.293+0000 7f1bd0890640 1 -- 172.21.15.28:0/2936490070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bcc1006a0 msgr2=0x7f1bcc104630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:53.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.293+0000 7f1bd0890640 1 --2- 172.21.15.28:0/2936490070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bcc1006a0 0x7f1bcc104630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.293+0000 7f1bd0890640 1 -- 172.21.15.28:0/2936490070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bcc103a40 con 0x7f1bcc101890 2024-04-25T01:32:53.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.293+0000 7f1bd1091640 1 --2- 172.21.15.28:0/2936490070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bcc1006a0 0x7f1bcc104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:53.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.293+0000 7f1bd0890640 1 --2- 172.21.15.28:0/2936490070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bcc101890 0x7f1bcc102c70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00e9f0 tx=0x7f1bbc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:53.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.293+0000 7f1bc27fc640 1 -- 172.21.15.28:0/2936490070 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc00cd80 con 0x7f1bcc101890 2024-04-25T01:32:53.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.293+0000 7f1bd32f5640 1 -- 172.21.15.28:0/2936490070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bcc1a1af0 con 0x7f1bcc101890 2024-04-25T01:32:53.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.293+0000 7f1bc27fc640 1 -- 172.21.15.28:0/2936490070 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bbc010910 con 0x7f1bcc101890 2024-04-25T01:32:53.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.293+0000 7f1bc27fc640 1 -- 172.21.15.28:0/2936490070 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc018950 con 0x7f1bcc101890 2024-04-25T01:32:53.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.293+0000 7f1bd32f5640 1 -- 172.21.15.28:0/2936490070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bcc1a2040 con 0x7f1bcc101890 2024-04-25T01:32:53.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.294+0000 7f1bd32f5640 1 -- 172.21.15.28:0/2936490070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bcc1a2210 con 0x7f1bcc101890 2024-04-25T01:32:53.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.295+0000 7f1bc27fc640 1 -- 172.21.15.28:0/2936490070 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f1bbc010ac0 con 0x7f1bcc101890 2024-04-25T01:32:53.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.296+0000 7f1bc27fc640 1 --2- 172.21.15.28:0/2936490070 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1bb80a2c80 0x7f1bb80a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:53.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.296+0000 7f1bd1091640 1 -- 172.21.15.28:0/2936490070 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1bb80a2c80 msgr2=0x7f1bb80a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1149553567 2024-04-25T01:32:53.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.296+0000 7f1bc27fc640 1 -- 172.21.15.28:0/2936490070 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f1bbc0cf810 con 0x7f1bcc101890 2024-04-25T01:32:53.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.296+0000 7f1bd1091640 1 --2- 172.21.15.28:0/2936490070 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1bb80a2c80 0x7f1bb80a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:32:53.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.301+0000 7f1bc27fc640 1 -- 172.21.15.28:0/2936490070 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1bbc09bd60 con 0x7f1bcc101890 2024-04-25T01:32:53.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:53 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2521816003' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T01:32:53.458 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:53 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: ignoring --setuser ceph since I am not root 2024-04-25T01:32:53.459 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:53 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: ignoring --setgroup ceph since I am not root 2024-04-25T01:32:53.459 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:53 smithi130 ceph-mgr[34572]: -- 172.21.15.130:0/3115522135 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d2f24da5a0 con 0x55d2f24aec00 2024-04-25T01:32:53.459 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:53 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:53.345+0000 7f7ca34ae200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:32:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.489+0000 7f1bd32f5640 1 -- 172.21.15.28:0/2936490070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f1bcc1a23e0 con 0x7f1bcc101890 2024-04-25T01:32:53.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.493+0000 7f1bc27fc640 1 -- 172.21.15.28:0/2936490070 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) ==== 138+0+0 (secure 0 0 0) 0x7f1bbc0a0060 con 0x7f1bcc101890 2024-04-25T01:32:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.495+0000 7f1bd32f5640 1 -- 172.21.15.28:0/2936490070 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1bb80a2c80 msgr2=0x7f1bb80a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:32:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.495+0000 7f1bd32f5640 1 --2- 172.21.15.28:0/2936490070 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1bb80a2c80 0x7f1bb80a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.495+0000 7f1bd32f5640 1 -- 172.21.15.28:0/2936490070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bcc101890 msgr2=0x7f1bcc102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.495+0000 7f1bd32f5640 1 --2- 172.21.15.28:0/2936490070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bcc101890 0x7f1bcc102c70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00e9f0 tx=0x7f1bbc00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.495+0000 7f1bd32f5640 1 -- 172.21.15.28:0/2936490070 shutdown_connections 2024-04-25T01:32:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.495+0000 7f1bd32f5640 1 --2- 172.21.15.28:0/2936490070 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f1bb80a2c80 0x7f1bb80a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.495+0000 7f1bd32f5640 1 --2- 172.21.15.28:0/2936490070 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1bcc102250 0x7f1bcc1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.495+0000 7f1bd32f5640 1 --2- 172.21.15.28:0/2936490070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bcc101890 0x7f1bcc102c70 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.495+0000 7f1bd32f5640 1 --2- 172.21.15.28:0/2936490070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1bcc1006a0 0x7f1bcc104630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.495+0000 7f1bd32f5640 1 -- 172.21.15.28:0/2936490070 >> 172.21.15.28:0/2936490070 conn(0x7f1bcc0fbc10 msgr2=0x7f1bcc0fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.495+0000 7f1bd32f5640 1 -- 172.21.15.28:0/2936490070 shutdown_connections 2024-04-25T01:32:53.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.495+0000 7f1bd32f5640 1 -- 172.21.15.28:0/2936490070 wait complete. 2024-04-25T01:32:53.483 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:53 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: ignoring --setuser ceph since I am not root 2024-04-25T01:32:53.483 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:53 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: ignoring --setgroup ceph since I am not root 2024-04-25T01:32:53.483 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:53 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:53.354+0000 7faee5fab200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:32:53.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:53 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2521816003' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T01:32:53.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-25T01:32:53.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:53 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2521816003' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T01:32:53.485 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:53 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: ignoring --setuser ceph since I am not root 2024-04-25T01:32:53.485 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:53 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: ignoring --setgroup ceph since I am not root 2024-04-25T01:32:53.485 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:53 smithi032 ceph-mgr[33857]: -- 172.21.15.32:0/1606835792 <== mon.2 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e165eee5a0 con 0x55e165ec3000 2024-04-25T01:32:53.485 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:53 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:53.348+0000 7fa50c434200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:32:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.612+0000 7f57b3653640 1 -- 172.21.15.28:0/1841460351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57ac104bf0 msgr2=0x7f57ac106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.612+0000 7f57b3653640 1 --2- 172.21.15.28:0/1841460351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57ac104bf0 0x7f57ac106fe0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f57a8009900 tx=0x7f57a802a410 comp rx=0 tx=0).stop 2024-04-25T01:32:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.612+0000 7f57b3653640 1 -- 172.21.15.28:0/1841460351 shutdown_connections 2024-04-25T01:32:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.612+0000 7f57b3653640 1 --2- 172.21.15.28:0/1841460351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ac107520 0x7f57ac109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.612+0000 7f57b3653640 1 --2- 172.21.15.28:0/1841460351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57ac104bf0 0x7f57ac106fe0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.612+0000 7f57b3653640 1 --2- 172.21.15.28:0/1841460351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57ac1022c0 0x7f57ac1046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.612+0000 7f57b3653640 1 -- 172.21.15.28:0/1841460351 >> 172.21.15.28:0/1841460351 conn(0x7f57ac0fbc10 msgr2=0x7f57ac0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.612+0000 7f57b3653640 1 -- 172.21.15.28:0/1841460351 shutdown_connections 2024-04-25T01:32:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.612+0000 7f57b3653640 1 -- 172.21.15.28:0/1841460351 wait complete. 2024-04-25T01:32:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.613+0000 7f57b3653640 1 Processor -- start 2024-04-25T01:32:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.613+0000 7f57b3653640 1 -- start start 2024-04-25T01:32:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.613+0000 7f57b3653640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ac1022c0 0x7f57ac197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.613+0000 7f57b3653640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57ac104bf0 0x7f57ac197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.613+0000 7f57b3653640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57ac107520 0x7f57ac19f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.613+0000 7f57b3653640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac10d410 con 0x7f57ac104bf0 2024-04-25T01:32:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.613+0000 7f57b3653640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac10d290 con 0x7f57ac1022c0 2024-04-25T01:32:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.613+0000 7f57b3653640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac10d590 con 0x7f57ac107520 2024-04-25T01:32:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.614+0000 7f57b0bee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57ac104bf0 0x7f57ac197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.614+0000 7f57b0bee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57ac104bf0 0x7f57ac197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41458/0 (socket says 172.21.15.28:41458) 2024-04-25T01:32:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.614+0000 7f57b0bee640 1 -- 172.21.15.28:0/1778896617 learned_addr learned my addr 172.21.15.28:0/1778896617 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.614+0000 7f57b13ef640 1 --2- 172.21.15.28:0/1778896617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ac1022c0 0x7f57ac197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.614+0000 7f57b1bf0640 1 --2- 172.21.15.28:0/1778896617 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57ac107520 0x7f57ac19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.614+0000 7f57b0bee640 1 -- 172.21.15.28:0/1778896617 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57ac107520 msgr2=0x7f57ac19f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.614+0000 7f57b0bee640 1 --2- 172.21.15.28:0/1778896617 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57ac107520 0x7f57ac19f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.614+0000 7f57b0bee640 1 -- 172.21.15.28:0/1778896617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ac1022c0 msgr2=0x7f57ac197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.614+0000 7f57b0bee640 1 --2- 172.21.15.28:0/1778896617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ac1022c0 0x7f57ac197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.614+0000 7f57b0bee640 1 -- 172.21.15.28:0/1778896617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57ac19f670 con 0x7f57ac104bf0 2024-04-25T01:32:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.614+0000 7f57b13ef640 1 --2- 172.21.15.28:0/1778896617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ac1022c0 0x7f57ac197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.614+0000 7f57b0bee640 1 --2- 172.21.15.28:0/1778896617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57ac104bf0 0x7f57ac197f90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f57a8031610 tx=0x7f57a8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.614+0000 7f57a27fc640 1 -- 172.21.15.28:0/1778896617 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a803f030 con 0x7f57ac104bf0 2024-04-25T01:32:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.614+0000 7f57a27fc640 1 -- 172.21.15.28:0/1778896617 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57a8003ad0 con 0x7f57ac104bf0 2024-04-25T01:32:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.614+0000 7f57a27fc640 1 -- 172.21.15.28:0/1778896617 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a8035070 con 0x7f57ac104bf0 2024-04-25T01:32:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.614+0000 7f57b3653640 1 -- 172.21.15.28:0/1778896617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57ac19f900 con 0x7f57ac104bf0 2024-04-25T01:32:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.614+0000 7f57b3653640 1 -- 172.21.15.28:0/1778896617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57ac19fdf0 con 0x7f57ac104bf0 2024-04-25T01:32:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.615+0000 7f57b3653640 1 -- 172.21.15.28:0/1778896617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57ac0fd700 con 0x7f57ac104bf0 2024-04-25T01:32:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.616+0000 7f57a27fc640 1 -- 172.21.15.28:0/1778896617 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f57a8042030 con 0x7f57ac104bf0 2024-04-25T01:32:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.617+0000 7f57a27fc640 1 --2- 172.21.15.28:0/1778896617 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f57980a2c80 0x7f57980a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.617+0000 7f57b13ef640 1 -- 172.21.15.28:0/1778896617 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f57980a2c80 msgr2=0x7f57980a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1149553567 2024-04-25T01:32:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.617+0000 7f57a27fc640 1 -- 172.21.15.28:0/1778896617 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f57a80ef9f0 con 0x7f57ac104bf0 2024-04-25T01:32:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.617+0000 7f57b13ef640 1 --2- 172.21.15.28:0/1778896617 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f57980a2c80 0x7f57980a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:32:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.621+0000 7f57a27fc640 1 -- 172.21.15.28:0/1778896617 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f57a80bbf40 con 0x7f57ac104bf0 2024-04-25T01:32:53.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.807+0000 7f57b3653640 1 -- 172.21.15.28:0/1778896617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f57ac0623b0 con 0x7f57ac104bf0 2024-04-25T01:32:53.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.811+0000 7f57a27fc640 1 -- 172.21.15.28:0/1778896617 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7f57a80c0240 con 0x7f57ac104bf0 2024-04-25T01:32:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.813+0000 7f57b3653640 1 -- 172.21.15.28:0/1778896617 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f57980a2c80 msgr2=0x7f57980a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:32:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.813+0000 7f57b3653640 1 --2- 172.21.15.28:0/1778896617 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f57980a2c80 0x7f57980a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.813+0000 7f57b3653640 1 -- 172.21.15.28:0/1778896617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57ac104bf0 msgr2=0x7f57ac197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.813+0000 7f57b3653640 1 --2- 172.21.15.28:0/1778896617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57ac104bf0 0x7f57ac197f90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f57a8031610 tx=0x7f57a8004270 comp rx=0 tx=0).stop 2024-04-25T01:32:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.813+0000 7f57b3653640 1 -- 172.21.15.28:0/1778896617 shutdown_connections 2024-04-25T01:32:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.813+0000 7f57b3653640 1 --2- 172.21.15.28:0/1778896617 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f57980a2c80 0x7f57980a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.813+0000 7f57b3653640 1 --2- 172.21.15.28:0/1778896617 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f57ac107520 0x7f57ac19f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.813+0000 7f57b3653640 1 --2- 172.21.15.28:0/1778896617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57ac104bf0 0x7f57ac197f90 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.813+0000 7f57b3653640 1 --2- 172.21.15.28:0/1778896617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f57ac1022c0 0x7f57ac197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.813+0000 7f57b3653640 1 -- 172.21.15.28:0/1778896617 >> 172.21.15.28:0/1778896617 conn(0x7f57ac0fbc10 msgr2=0x7f57ac102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.813+0000 7f57b3653640 1 -- 172.21.15.28:0/1778896617 shutdown_connections 2024-04-25T01:32:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:53.813+0000 7f57b3653640 1 -- 172.21.15.28:0/1778896617 wait complete. 2024-04-25T01:32:53.801 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:53 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:53.548+0000 7faee5fab200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:32:53.801 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:53 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:53.660+0000 7faee5fab200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:32:53.802 INFO:teuthology.run_tasks:Running task kclient... 2024-04-25T01:32:53.817 INFO:tasks.kclient:Mounting kernel clients... 2024-04-25T01:32:53.817 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-25T01:32:53.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:32:53.817 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-25T01:32:53.833 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-25T01:32:53.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:32:53.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T01:32:53.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:32:53.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-04-25T01:32:53.912 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:53 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:53.529+0000 7f7ca34ae200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:32:53.912 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:53 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:53.630+0000 7f7ca34ae200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:32:53.944 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:53 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:53.540+0000 7fa50c434200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:32:53.944 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:53 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:53.645+0000 7fa50c434200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:32:53.944 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:53 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:53.966+0000 7fa50c434200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:32:54.014 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-25T01:32:54.014 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T01:32:54.014 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T01:32:54.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-25T01:32:54.061 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:53 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:53.992+0000 7faee5fab200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:32:54.208 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:53 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:53.934+0000 7f7ca34ae200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:32:54.208 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:54 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:54.020+0000 7f7ca34ae200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:32:54.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:54 smithi130 ceph-mon[32702]: pgmap v369: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:32:54.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:54 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2521816003' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T01:32:54.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:54 smithi130 ceph-mon[32702]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-25T01:32:54.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:54 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2936490070' entity='client.admin' 2024-04-25T01:32:54.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:54 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1778896617' entity='client.admin' 2024-04-25T01:32:54.233 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:54 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:54.054+0000 7fa50c434200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:32:54.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:54 smithi032 ceph-mon[32461]: pgmap v369: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:32:54.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:54 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2521816003' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T01:32:54.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:54 smithi032 ceph-mon[32461]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-25T01:32:54.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:54 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2936490070' entity='client.admin' 2024-04-25T01:32:54.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:54 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1778896617' entity='client.admin' 2024-04-25T01:32:54.272 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:32:54.396 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:54 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:54.086+0000 7faee5fab200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:32:54.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:54 smithi028 ceph-mon[29390]: pgmap v369: 129 pgs: 129 active+clean; 645 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:32:54.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:54 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2521816003' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T01:32:54.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:54 smithi028 ceph-mon[29390]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-25T01:32:54.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:54 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2936490070' entity='client.admin' 2024-04-25T01:32:54.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:54 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1778896617' entity='client.admin' 2024-04-25T01:32:55.435 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:55 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:55.362+0000 7faee5fab200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:32:55.458 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:55.189+0000 7f7ca34ae200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:32:55.458 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:55.357+0000 7f7ca34ae200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:32:55.627 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:55.296+0000 7fa50c434200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:32:55.627 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:55.473+0000 7fa50c434200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:32:55.627 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:55.649+0000 7fa50c434200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:32:55.703 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:55 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:55.546+0000 7faee5fab200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:32:55.786 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:55.527+0000 7f7ca34ae200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:32:55.973 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:55 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:55.726+0000 7faee5fab200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:32:55.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.008+0000 7f277e034700 1 -- 172.21.15.28:0/332988171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778101130 msgr2=0x7f27781015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:55.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.008+0000 7f277e034700 1 --2- 172.21.15.28:0/332988171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778101130 0x7f27781015b0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2768009990 tx=0x7f2768009ca0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.008+0000 7f277e034700 1 -- 172.21.15.28:0/332988171 shutdown_connections 2024-04-25T01:32:55.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.008+0000 7f277e034700 1 --2- 172.21.15.28:0/332988171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2778101af0 0x7f277810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.008+0000 7f277e034700 1 --2- 172.21.15.28:0/332988171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f27781071a0 0x7f2778107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.008+0000 7f277e034700 1 --2- 172.21.15.28:0/332988171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778101130 0x7f27781015b0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.008+0000 7f277e034700 1 -- 172.21.15.28:0/332988171 >> 172.21.15.28:0/332988171 conn(0x7f27780780e0 msgr2=0x7f27780784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:55.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.008+0000 7f277e034700 1 -- 172.21.15.28:0/332988171 shutdown_connections 2024-04-25T01:32:55.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.008+0000 7f277e034700 1 -- 172.21.15.28:0/332988171 wait complete. 2024-04-25T01:32:55.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.009+0000 7f277e034700 1 Processor -- start 2024-04-25T01:32:55.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.009+0000 7f277e034700 1 -- start start 2024-04-25T01:32:55.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.009+0000 7f277e034700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778101130 0x7f277819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.009+0000 7f277e034700 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2778101af0 0x7f277819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.009+0000 7f277e034700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27781071a0 0x7f27781987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.009+0000 7f277e034700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2778111480 con 0x7f2778101130 2024-04-25T01:32:55.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.009+0000 7f277e034700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2778111300 con 0x7f27781071a0 2024-04-25T01:32:55.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.009+0000 7f277e034700 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2778111600 con 0x7f2778101af0 2024-04-25T01:32:55.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.009+0000 7f27777fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778101130 0x7f277819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.010+0000 7f27777fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778101130 0x7f277819e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41470/0 (socket says 172.21.15.28:41470) 2024-04-25T01:32:55.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.010+0000 7f27777fe700 1 -- 172.21.15.28:0/3654926744 learned_addr learned my addr 172.21.15.28:0/3654926744 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:55.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.010+0000 7f2777fff700 1 --2- 172.21.15.28:0/3654926744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27781071a0 0x7f27781987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.010+0000 7f2776ffd700 1 --2- 172.21.15.28:0/3654926744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2778101af0 0x7f277819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:55.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.010+0000 7f27777fe700 1 -- 172.21.15.28:0/3654926744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2778101af0 msgr2=0x7f277819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:55.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.010+0000 7f27777fe700 1 --2- 172.21.15.28:0/3654926744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2778101af0 0x7f277819eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.010+0000 7f27777fe700 1 -- 172.21.15.28:0/3654926744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27781071a0 msgr2=0x7f27781987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:55.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.010+0000 7f27777fe700 1 --2- 172.21.15.28:0/3654926744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27781071a0 0x7f27781987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:55.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.010+0000 7f27777fe700 1 -- 172.21.15.28:0/3654926744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2778198fb0 con 0x7f2778101130 2024-04-25T01:32:55.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.010+0000 7f2777fff700 1 --2- 172.21.15.28:0/3654926744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27781071a0 0x7f27781987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:55.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.010+0000 7f27777fe700 1 --2- 172.21.15.28:0/3654926744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778101130 0x7f277819e650 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f276000e9c0 tx=0x7f276000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:55.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.010+0000 7f2774ff9700 1 -- 172.21.15.28:0/3654926744 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276000cbd0 con 0x7f2778101130 2024-04-25T01:32:55.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.010+0000 7f277e034700 1 -- 172.21.15.28:0/3654926744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27781992a0 con 0x7f2778101130 2024-04-25T01:32:55.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.010+0000 7f2774ff9700 1 -- 172.21.15.28:0/3654926744 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f276000ee60 con 0x7f2778101130 2024-04-25T01:32:55.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.010+0000 7f2774ff9700 1 -- 172.21.15.28:0/3654926744 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2760005630 con 0x7f2778101130 2024-04-25T01:32:55.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.010+0000 7f277e034700 1 -- 172.21.15.28:0/3654926744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27781a54b0 con 0x7f2778101130 2024-04-25T01:32:55.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.011+0000 7f277e034700 1 -- 172.21.15.28:0/3654926744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2778102850 con 0x7f2778101130 2024-04-25T01:32:55.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.013+0000 7f2774ff9700 1 -- 172.21.15.28:0/3654926744 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f2760004750 con 0x7f2778101130 2024-04-25T01:32:55.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.014+0000 7f2774ff9700 1 --2- 172.21.15.28:0/3654926744 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f27640a2cf0 0x7f27640a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:55.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.014+0000 7f2776ffd700 1 -- 172.21.15.28:0/3654926744 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f27640a2cf0 msgr2=0x7f27640a51b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1149553567 2024-04-25T01:32:55.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.014+0000 7f2774ff9700 1 -- 172.21.15.28:0/3654926744 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f27600cf750 con 0x7f2778101130 2024-04-25T01:32:55.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.014+0000 7f2776ffd700 1 --2- 172.21.15.28:0/3654926744 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f27640a2cf0 0x7f27640a51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:32:55.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.017+0000 7f2774ff9700 1 -- 172.21.15.28:0/3654926744 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2760053740 con 0x7f2778101130 2024-04-25T01:32:56.081 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:55.809+0000 7f7ca34ae200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:32:56.081 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:55.901+0000 7f7ca34ae200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:32:56.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.215+0000 7f2776ffd700 1 -- 172.21.15.28:0/3654926744 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f27640a2cf0 msgr2=0x7f27640a51b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1149553567 2024-04-25T01:32:56.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.215+0000 7f2776ffd700 1 --2- 172.21.15.28:0/3654926744 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f27640a2cf0 0x7f27640a51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:32:56.232 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:55.947+0000 7fa50c434200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:32:56.232 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:56 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:56.043+0000 7fa50c434200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:32:56.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.273+0000 7f277e034700 1 -- 172.21.15.28:0/3654926744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f27781a5740 con 0x7f2778101130 2024-04-25T01:32:56.251 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:56 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:56.038+0000 7faee5fab200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:32:56.251 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:56 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:56.138+0000 7faee5fab200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:32:56.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.274+0000 7f2774ff9700 1 -- 172.21.15.28:0/3654926744 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+88 (secure 0 0 0) 0x7f276009c350 con 0x7f2778101130 2024-04-25T01:32:56.251 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T01:32:56.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.276+0000 7f277e034700 1 -- 172.21.15.28:0/3654926744 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f27640a2cf0 msgr2=0x7f27640a51b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:32:56.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.276+0000 7f277e034700 1 --2- 172.21.15.28:0/3654926744 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f27640a2cf0 0x7f27640a51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:56.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.276+0000 7f277e034700 1 -- 172.21.15.28:0/3654926744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778101130 msgr2=0x7f277819e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:56.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.276+0000 7f277e034700 1 --2- 172.21.15.28:0/3654926744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778101130 0x7f277819e650 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f276000e9c0 tx=0x7f276000ecd0 comp rx=0 tx=0).stop 2024-04-25T01:32:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.276+0000 7f277e034700 1 -- 172.21.15.28:0/3654926744 shutdown_connections 2024-04-25T01:32:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.276+0000 7f277e034700 1 --2- 172.21.15.28:0/3654926744 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f27640a2cf0 0x7f27640a51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.276+0000 7f277e034700 1 --2- 172.21.15.28:0/3654926744 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f27781071a0 0x7f27781987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.276+0000 7f277e034700 1 --2- 172.21.15.28:0/3654926744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2778101130 0x7f277819e650 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.276+0000 7f277e034700 1 --2- 172.21.15.28:0/3654926744 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2778101af0 0x7f277819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.276+0000 7f277e034700 1 -- 172.21.15.28:0/3654926744 >> 172.21.15.28:0/3654926744 conn(0x7f27780780e0 msgr2=0x7f2778100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.276+0000 7f277e034700 1 -- 172.21.15.28:0/3654926744 shutdown_connections 2024-04-25T01:32:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:56.276+0000 7f277e034700 1 -- 172.21.15.28:0/3654926744 wait complete. 2024-04-25T01:32:56.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:56 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3654926744' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:32:56.458 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:56 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:56.103+0000 7f7ca34ae200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:32:56.458 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:56 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:56.294+0000 7f7ca34ae200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:32:56.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:56 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3654926744' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:32:56.483 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:56 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:56.254+0000 7fa50c434200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:32:56.483 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:56 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:56.454+0000 7fa50c434200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:32:56.504 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:56 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:56.360+0000 7faee5fab200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:32:56.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:56 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3654926744' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:32:56.811 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:56 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:56.569+0000 7faee5fab200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:32:56.931 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T01:32:56.931 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:32:56.932 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-04-25T01:32:56.932 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T01:32:56.932 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T01:32:56.932 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T01:32:56.932 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T01:32:56.932 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:32:56.932 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:32:56.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:32:56.932 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-04-25T01:32:56.949 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:32:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:32:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:32:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:32:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-04-25T01:32:56.949 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:32:56.949 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:32:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:32:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-04-25T01:32:56.950 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:32:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:32:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-04-25T01:32:56.950 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:32:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-04-25T01:32:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-04-25T01:32:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-25T01:32:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42038sec preferred_lft 42038sec 2024-04-25T01:32:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-04-25T01:32:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:32:56.950 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:32:56.950 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-04-25T01:32:56.951 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-04-25T01:32:56.951 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T01:32:56.951 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:32:56.951 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T01:32:56.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-04-25T01:32:56.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-04-25T01:32:56.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-04-25T01:32:56.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T01:32:56.951 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T01:32:57.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:32:57.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:32:57.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:32:57.096 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:32:57.121 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-04-25T01:32:57.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:32:57.123 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-04-25T01:32:57.180 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-04-25T01:32:57.180 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:32:57.180 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-25T01:32:57.180 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-25T01:32:57.180 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:32:57.181 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:32:57.181 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T01:32:57.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-25T01:32:57.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-25T01:32:57.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-25T01:32:57.181 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T01:32:57.208 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:56 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:56.887+0000 7f7ca34ae200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:32:57.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:32:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:32:57.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:32:57.311 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-25T01:32:57.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:32:57.327 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-04-25T01:32:57.383 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:32:57.383 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T01:32:57.383 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:32:57.383 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-25T01:32:57.383 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T01:32:57.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:32:57.481 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:57 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:57.213+0000 7faee5fab200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:32:57.483 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:57 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:57.079+0000 7fa50c434200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:32:57.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:32:57.503 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-25T01:32:57.503 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:32:57.503 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T01:32:57.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-25T01:32:57.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T01:32:57.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-25T01:32:57.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-25T01:32:57.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-25T01:32:57.503 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T01:32:57.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:32:57.574 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:57 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:57.301+0000 7f7ca34ae200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:32:57.574 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:57 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:57.397+0000 7f7ca34ae200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:32:57.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:32:57.622 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:32:57.622 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T01:32:57.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-04-25T01:32:57.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T01:32:57.622 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T01:32:57.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:32:57.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:32:57.701 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:32:57.701 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T01:32:57.716 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T01:32:57.716 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:32:57.716 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:32:57.769 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:57 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:57.687+0000 7faee5fab200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:32:57.769 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:57 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:57.791+0000 7faee5fab200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:32:57.771 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-25T01:32:57.771 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-25T01:32:57.771 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/bc0d796f-3fd7-48fb-b5d6-fa28c50c6182 2024-04-25T01:32:57.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:32:57.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/bc0d796f-3fd7-48fb-b5d6-fa28c50c6182 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-04-25T01:32:57.803 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:57 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:57.518+0000 7fa50c434200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:32:57.803 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:57 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:57.618+0000 7fa50c434200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:32:57.856 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-04-25T01:32:57.856 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-04-25T01:32:57.856 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-04-25T01:32:57.958 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:57 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:57.596+0000 7f7ca34ae200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:32:58.016 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-04-25T01:32:58.016 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-04-25T01:32:58.016 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-04-25T01:32:58.016 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.32:3300,172.21.15.130:3300" 2024-04-25T01:32:58.016 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:3300,172.21.15.32:3300,172.21.15.130:3300:/volumes/qa/sv_1/bc0d796f-3fd7-48fb-b5d6-fa28c50c6182 2024-04-25T01:32:58.016 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync,key=0,fsid=0e88f60a-02a2-11ef-bc93-c7b262605968" will pass to kernel 2024-04-25T01:32:58.017 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T01:32:58.017 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-04-25T01:32:58.017 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T01:32:58.017 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T01:32:58.017 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T01:32:58.019 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T01:32:58.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:32:58.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T01:32:58.062 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:58.031+0000 7faee5fab200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:32:58.089 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:32:58.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:32:58.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:32:58.189 INFO:teuthology.orchestra.run.smithi028.stdout:client25178 2024-04-25T01:32:58.189 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:32:58.189 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:32:58.207 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:32:58.233 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:57 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:57.825+0000 7fa50c434200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:32:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:32:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:32:58.305 INFO:teuthology.orchestra.run.smithi028.stdout:client25178 2024-04-25T01:32:58.305 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:32:58.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:32:58.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:32:58.366 INFO:teuthology.orchestra.run.smithi028.stdout:client25178 2024-04-25T01:32:58.366 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968.client25178/status') 2024-04-25T01:32:58.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:32:58.426 INFO:teuthology.orchestra.run.smithi028.stdout:instance: client.25178 (3)172.21.15.28:0/3076411517 2024-04-25T01:32:58.426 INFO:teuthology.orchestra.run.smithi028.stdout:blocklisted: false 2024-04-25T01:32:58.426 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records in 2024-04-25T01:32:58.426 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records out 2024-04-25T01:32:58.426 INFO:teuthology.orchestra.run.smithi028.stderr:71 bytes copied, 2.6023e-05 s, 2.7 MB/s 2024-04-25T01:32:58.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:32:58.434 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T01:32:58.443 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T01:32:58.443 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T01:32:58.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-25T01:32:58.444 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:58 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:58.127+0000 7f7ca34ae200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:32:58.444 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:58 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:58.219+0000 7f7ca34ae200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:32:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.570+0000 7fe404cca640 1 -- 172.21.15.28:0/574943216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe400101890 msgr2=0x7fe400101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.570+0000 7fe404cca640 1 --2- 172.21.15.28:0/574943216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe400101890 0x7fe400101d10 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8009900 tx=0x7fe3e802a410 comp rx=0 tx=0).stop 2024-04-25T01:32:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.570+0000 7fe404cca640 1 -- 172.21.15.28:0/574943216 shutdown_connections 2024-04-25T01:32:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.570+0000 7fe404cca640 1 --2- 172.21.15.28:0/574943216 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe400102250 0x7fe400108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.570+0000 7fe404cca640 1 --2- 172.21.15.28:0/574943216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe400101890 0x7fe400101d10 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.570+0000 7fe404cca640 1 --2- 172.21.15.28:0/574943216 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4001006a0 0x7fe400100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.570+0000 7fe404cca640 1 -- 172.21.15.28:0/574943216 >> 172.21.15.28:0/574943216 conn(0x7fe4000fbc10 msgr2=0x7fe4000fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.570+0000 7fe404cca640 1 -- 172.21.15.28:0/574943216 shutdown_connections 2024-04-25T01:32:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.570+0000 7fe404cca640 1 -- 172.21.15.28:0/574943216 wait complete. 2024-04-25T01:32:58.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.571+0000 7fe404cca640 1 Processor -- start 2024-04-25T01:32:58.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.571+0000 7fe404cca640 1 -- start start 2024-04-25T01:32:58.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.571+0000 7fe404cca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4001006a0 0x7fe400199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:58.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.571+0000 7fe404cca640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe400101890 0x7fe40019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:58.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.571+0000 7fe404cca640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe400102250 0x7fe4001a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:58.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.571+0000 7fe404cca640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe40010c370 con 0x7fe4001006a0 2024-04-25T01:32:58.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.571+0000 7fe404cca640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe40010c1f0 con 0x7fe400102250 2024-04-25T01:32:58.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.571+0000 7fe404cca640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe40010c4f0 con 0x7fe400101890 2024-04-25T01:32:58.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.571+0000 7fe3fe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4001006a0 0x7fe400199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:58.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.571+0000 7fe3fe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4001006a0 0x7fe400199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54716/0 (socket says 172.21.15.28:54716) 2024-04-25T01:32:58.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.571+0000 7fe3fe59c640 1 -- 172.21.15.28:0/1700063202 learned_addr learned my addr 172.21.15.28:0/1700063202 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:58.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.571+0000 7fe3fed9d640 1 --2- 172.21.15.28:0/1700063202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe400102250 0x7fe4001a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.571+0000 7fe3fdd9b640 1 --2- 172.21.15.28:0/1700063202 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe400101890 0x7fe40019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.572+0000 7fe3fed9d640 1 -- 172.21.15.28:0/1700063202 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe400101890 msgr2=0x7fe40019a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.572+0000 7fe3fed9d640 1 --2- 172.21.15.28:0/1700063202 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe400101890 0x7fe40019a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.572+0000 7fe3fed9d640 1 -- 172.21.15.28:0/1700063202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4001006a0 msgr2=0x7fe400199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.572+0000 7fe3fed9d640 1 --2- 172.21.15.28:0/1700063202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4001006a0 0x7fe400199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.572+0000 7fe3fed9d640 1 -- 172.21.15.28:0/1700063202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4001a1810 con 0x7fe400102250 2024-04-25T01:32:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.572+0000 7fe3fe59c640 1 --2- 172.21.15.28:0/1700063202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4001006a0 0x7fe400199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.572+0000 7fe3fed9d640 1 --2- 172.21.15.28:0/1700063202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe400102250 0x7fe4001a11a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009fd0 tx=0x7fe3f800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.572+0000 7fe3f77fe640 1 -- 172.21.15.28:0/1700063202 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f800ce20 con 0x7fe400102250 2024-04-25T01:32:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.572+0000 7fe404cca640 1 -- 172.21.15.28:0/1700063202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4001a1b00 con 0x7fe400102250 2024-04-25T01:32:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.572+0000 7fe3f77fe640 1 -- 172.21.15.28:0/1700063202 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3f8005020 con 0x7fe400102250 2024-04-25T01:32:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.572+0000 7fe3f77fe640 1 -- 172.21.15.28:0/1700063202 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8010720 con 0x7fe400102250 2024-04-25T01:32:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.572+0000 7fe404cca640 1 -- 172.21.15.28:0/1700063202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4001a2070 con 0x7fe400102250 2024-04-25T01:32:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.573+0000 7fe404cca640 1 -- 172.21.15.28:0/1700063202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4001a2300 con 0x7fe400102250 2024-04-25T01:32:58.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.575+0000 7fe3f77fe640 1 -- 172.21.15.28:0/1700063202 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7fe3f8005540 con 0x7fe400102250 2024-04-25T01:32:58.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.576+0000 7fe3f77fe640 1 --2- 172.21.15.28:0/1700063202 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe3ec0a2c80 0x7fe3ec0a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:58.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.576+0000 7fe3fe59c640 1 -- 172.21.15.28:0/1700063202 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe3ec0a2c80 msgr2=0x7fe3ec0a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1149553567 2024-04-25T01:32:58.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.576+0000 7fe3f77fe640 1 -- 172.21.15.28:0/1700063202 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fe3f80cfcc0 con 0x7fe400102250 2024-04-25T01:32:58.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.576+0000 7fe3fe59c640 1 --2- 172.21.15.28:0/1700063202 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe3ec0a2c80 0x7fe3ec0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:32:58.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.583+0000 7fe3f77fe640 1 -- 172.21.15.28:0/1700063202 <== mon.2 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe3f809c950 con 0x7fe400102250 2024-04-25T01:32:58.680 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:58.602+0000 7faee5fab200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:32:58.703 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:58 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:58.376+0000 7fa50c434200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:32:58.703 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:58 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:58.471+0000 7fa50c434200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:32:58.708 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:58 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:58.464+0000 7f7ca34ae200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:32:58.708 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:58 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:58.597+0000 7f7ca34ae200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:32:58.708 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:58 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:58.690+0000 7f7ca34ae200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:32:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.777+0000 7fe3fe59c640 1 -- 172.21.15.28:0/1700063202 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe3ec0a2c80 msgr2=0x7fe3ec0a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1149553567 2024-04-25T01:32:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.777+0000 7fe3fe59c640 1 --2- 172.21.15.28:0/1700063202 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe3ec0a2c80 0x7fe3ec0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:32:58.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.807+0000 7fe404cca640 1 -- 172.21.15.28:0/1700063202 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7fe4001a26f0 con 0x7fe400102250 2024-04-25T01:32:58.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.868+0000 7fe3f77fe640 1 -- 172.21.15.28:0/1700063202 <== mon.2 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v28) ==== 117+0+0 (secure 0 0 0) 0x7fe3f809c250 con 0x7fe400102250 2024-04-25T01:32:58.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.871+0000 7fe404cca640 1 -- 172.21.15.28:0/1700063202 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe3ec0a2c80 msgr2=0x7fe3ec0a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:32:58.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.871+0000 7fe404cca640 1 --2- 172.21.15.28:0/1700063202 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe3ec0a2c80 0x7fe3ec0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:58.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.871+0000 7fe404cca640 1 -- 172.21.15.28:0/1700063202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe400102250 msgr2=0x7fe4001a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:58.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.871+0000 7fe404cca640 1 --2- 172.21.15.28:0/1700063202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe400102250 0x7fe4001a11a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009fd0 tx=0x7fe3f800ef10 comp rx=0 tx=0).stop 2024-04-25T01:32:58.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.871+0000 7fe404cca640 1 -- 172.21.15.28:0/1700063202 shutdown_connections 2024-04-25T01:32:58.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.871+0000 7fe404cca640 1 --2- 172.21.15.28:0/1700063202 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7fe3ec0a2c80 0x7fe3ec0a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:58.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.871+0000 7fe404cca640 1 --2- 172.21.15.28:0/1700063202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe400102250 0x7fe4001a11a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:58.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.871+0000 7fe404cca640 1 --2- 172.21.15.28:0/1700063202 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe400101890 0x7fe40019a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:58.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.871+0000 7fe404cca640 1 --2- 172.21.15.28:0/1700063202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4001006a0 0x7fe400199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:58.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.871+0000 7fe404cca640 1 -- 172.21.15.28:0/1700063202 >> 172.21.15.28:0/1700063202 conn(0x7fe4000fbc10 msgr2=0x7fe4000fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:58.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.871+0000 7fe404cca640 1 -- 172.21.15.28:0/1700063202 shutdown_connections 2024-04-25T01:32:58.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:58.871+0000 7fe404cca640 1 -- 172.21.15.28:0/1700063202 wait complete. 2024-04-25T01:32:58.860 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-25T01:32:58.871 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-25T01:32:58.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:32:58.940 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:58.702+0000 7faee5fab200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:32:58.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:58 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1700063202' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T01:32:58.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:58 smithi028 ceph-mon[29390]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T01:32:58.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:58 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1700063202' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T01:32:58.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:58 smithi032 ceph-mon[32461]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T01:32:58.983 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:58 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:58.725+0000 7fa50c434200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:32:58.984 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:58 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:58.866+0000 7fa50c434200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:32:58.984 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:58 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:58.964+0000 7fa50c434200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:32:59.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.046+0000 7ff17431d640 1 -- 172.21.15.28:0/2129227644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff16c104c10 msgr2=0x7ff16c107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:59.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.046+0000 7ff17431d640 1 --2- 172.21.15.28:0/2129227644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff16c104c10 0x7ff16c107000 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff168009900 tx=0x7ff16802a410 comp rx=0 tx=0).stop 2024-04-25T01:32:59.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.046+0000 7ff17431d640 1 -- 172.21.15.28:0/2129227644 shutdown_connections 2024-04-25T01:32:59.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.046+0000 7ff17431d640 1 --2- 172.21.15.28:0/2129227644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff16c107540 0x7ff16c109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.046+0000 7ff17431d640 1 --2- 172.21.15.28:0/2129227644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff16c104c10 0x7ff16c107000 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.046+0000 7ff17431d640 1 --2- 172.21.15.28:0/2129227644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff16c1022e0 0x7ff16c1046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.046+0000 7ff17431d640 1 -- 172.21.15.28:0/2129227644 >> 172.21.15.28:0/2129227644 conn(0x7ff16c0fbc30 msgr2=0x7ff16c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.046+0000 7ff17431d640 1 -- 172.21.15.28:0/2129227644 shutdown_connections 2024-04-25T01:32:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.046+0000 7ff17431d640 1 -- 172.21.15.28:0/2129227644 wait complete. 2024-04-25T01:32:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.046+0000 7ff17431d640 1 Processor -- start 2024-04-25T01:32:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.046+0000 7ff17431d640 1 -- start start 2024-04-25T01:32:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.046+0000 7ff17431d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff16c1022e0 0x7ff16c197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.046+0000 7ff17431d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff16c104c10 0x7ff16c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.046+0000 7ff17431d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff16c107540 0x7ff16c19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.046+0000 7ff17431d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff16c10d480 con 0x7ff16c107540 2024-04-25T01:32:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.046+0000 7ff17431d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff16c10d300 con 0x7ff16c104c10 2024-04-25T01:32:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.046+0000 7ff17431d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff16c10d600 con 0x7ff16c1022e0 2024-04-25T01:32:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.047+0000 7ff1728ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff16c107540 0x7ff16c19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.047+0000 7ff1728ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff16c107540 0x7ff16c19f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54730/0 (socket says 172.21.15.28:54730) 2024-04-25T01:32:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.047+0000 7ff1728ba640 1 -- 172.21.15.28:0/1512634964 learned_addr learned my addr 172.21.15.28:0/1512634964 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.047+0000 7ff1718b8640 1 --2- 172.21.15.28:0/1512634964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff16c104c10 0x7ff16c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.047+0000 7ff1720b9640 1 --2- 172.21.15.28:0/1512634964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff16c1022e0 0x7ff16c197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.047+0000 7ff1728ba640 1 -- 172.21.15.28:0/1512634964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff16c1022e0 msgr2=0x7ff16c197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.047+0000 7ff1728ba640 1 --2- 172.21.15.28:0/1512634964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff16c1022e0 0x7ff16c197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.047+0000 7ff1728ba640 1 -- 172.21.15.28:0/1512634964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff16c104c10 msgr2=0x7ff16c198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.047+0000 7ff1728ba640 1 --2- 172.21.15.28:0/1512634964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff16c104c10 0x7ff16c198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.047+0000 7ff1728ba640 1 -- 172.21.15.28:0/1512634964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff16c19f710 con 0x7ff16c107540 2024-04-25T01:32:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.047+0000 7ff1718b8640 1 --2- 172.21.15.28:0/1512634964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff16c104c10 0x7ff16c198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.047+0000 7ff1728ba640 1 --2- 172.21.15.28:0/1512634964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff16c107540 0x7ff16c19f0a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff16400eb10 tx=0x7ff16400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.047+0000 7ff15f7fe640 1 -- 172.21.15.28:0/1512634964 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff164004b20 con 0x7ff16c107540 2024-04-25T01:32:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.047+0000 7ff15f7fe640 1 -- 172.21.15.28:0/1512634964 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff164004cc0 con 0x7ff16c107540 2024-04-25T01:32:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.047+0000 7ff15f7fe640 1 -- 172.21.15.28:0/1512634964 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff164010760 con 0x7ff16c107540 2024-04-25T01:32:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.047+0000 7ff17431d640 1 -- 172.21.15.28:0/1512634964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff16c19fa00 con 0x7ff16c107540 2024-04-25T01:32:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.047+0000 7ff17431d640 1 -- 172.21.15.28:0/1512634964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff16c19ff20 con 0x7ff16c107540 2024-04-25T01:32:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.049+0000 7ff17431d640 1 -- 172.21.15.28:0/1512634964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff16c0fd720 con 0x7ff16c107540 2024-04-25T01:32:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.049+0000 7ff15f7fe640 1 -- 172.21.15.28:0/1512634964 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7ff164005020 con 0x7ff16c107540 2024-04-25T01:32:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.050+0000 7ff15f7fe640 1 --2- 172.21.15.28:0/1512634964 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1580a2be0 0x7ff1580a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.050+0000 7ff1720b9640 1 -- 172.21.15.28:0/1512634964 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1580a2be0 msgr2=0x7ff1580a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1149553567 2024-04-25T01:32:59.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.050+0000 7ff15f7fe640 1 -- 172.21.15.28:0/1512634964 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7ff1640cfaa0 con 0x7ff16c107540 2024-04-25T01:32:59.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.050+0000 7ff1720b9640 1 --2- 172.21.15.28:0/1512634964 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1580a2be0 0x7ff1580a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:32:59.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.054+0000 7ff15f7fe640 1 -- 172.21.15.28:0/1512634964 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff16409c6a0 con 0x7ff16c107540 2024-04-25T01:32:59.207 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:58.967+0000 7faee5fab200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:32:59.207 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:59 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:59.128+0000 7faee5fab200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:32:59.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:58 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1700063202' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T01:32:59.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:58 smithi130 ceph-mon[32702]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T01:32:59.208 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:58 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:58.806+0000 7f7ca34ae200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:32:59.208 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:58 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:58.899+0000 7f7ca34ae200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:32:59.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.251+0000 7ff1720b9640 1 -- 172.21.15.28:0/1512634964 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1580a2be0 msgr2=0x7ff1580a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1149553567 2024-04-25T01:32:59.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.251+0000 7ff1720b9640 1 --2- 172.21.15.28:0/1512634964 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1580a2be0 0x7ff1580a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:32:59.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.260+0000 7ff17431d640 1 -- 172.21.15.28:0/1512634964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff16c062340 con 0x7ff16c107540 2024-04-25T01:32:59.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.262+0000 7ff15f7fe640 1 -- 172.21.15.28:0/1512634964 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24564 (secure 0 0 0) 0x7ff16409bfa0 con 0x7ff16c107540 2024-04-25T01:32:59.239 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:32:59.239 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:31:21.857121+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:32:59.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.264+0000 7ff17431d640 1 -- 172.21.15.28:0/1512634964 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1580a2be0 msgr2=0x7ff1580a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:32:59.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.264+0000 7ff17431d640 1 --2- 172.21.15.28:0/1512634964 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1580a2be0 0x7ff1580a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.264+0000 7ff17431d640 1 -- 172.21.15.28:0/1512634964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff16c107540 msgr2=0x7ff16c19f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:59.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.264+0000 7ff17431d640 1 --2- 172.21.15.28:0/1512634964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff16c107540 0x7ff16c19f0a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff16400eb10 tx=0x7ff16400c2d0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.264+0000 7ff17431d640 1 -- 172.21.15.28:0/1512634964 shutdown_connections 2024-04-25T01:32:59.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.264+0000 7ff17431d640 1 --2- 172.21.15.28:0/1512634964 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff1580a2be0 0x7ff1580a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.264+0000 7ff17431d640 1 --2- 172.21.15.28:0/1512634964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff16c107540 0x7ff16c19f0a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.264+0000 7ff17431d640 1 --2- 172.21.15.28:0/1512634964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff16c104c10 0x7ff16c198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.264+0000 7ff17431d640 1 --2- 172.21.15.28:0/1512634964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff16c1022e0 0x7ff16c197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.264+0000 7ff17431d640 1 -- 172.21.15.28:0/1512634964 >> 172.21.15.28:0/1512634964 conn(0x7ff16c0fbc30 msgr2=0x7ff16c102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:59.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.264+0000 7ff17431d640 1 -- 172.21.15.28:0/1512634964 shutdown_connections 2024-04-25T01:32:59.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.264+0000 7ff17431d640 1 -- 172.21.15.28:0/1512634964 wait complete. 2024-04-25T01:32:59.253 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:25:48.767414+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:30:32.282430+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:30:33.127049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:30:36.796132+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:32:59.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:32:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.422+0000 7f6eba89a640 1 -- 172.21.15.28:0/3356925777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eb4102250 msgr2=0x7f6eb4108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.422+0000 7f6eba89a640 1 --2- 172.21.15.28:0/3356925777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eb4102250 0x7f6eb4108ae0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6eac009ca0 tx=0x7f6eac02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.422+0000 7f6eba89a640 1 -- 172.21.15.28:0/3356925777 shutdown_connections 2024-04-25T01:32:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.422+0000 7f6eba89a640 1 --2- 172.21.15.28:0/3356925777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eb4102250 0x7f6eb4108ae0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.422+0000 7f6eba89a640 1 --2- 172.21.15.28:0/3356925777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6eb4101890 0x7f6eb4101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.422+0000 7f6eba89a640 1 --2- 172.21.15.28:0/3356925777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6eb41006a0 0x7f6eb4100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.422+0000 7f6eba89a640 1 -- 172.21.15.28:0/3356925777 >> 172.21.15.28:0/3356925777 conn(0x7f6eb40fbc30 msgr2=0x7f6eb40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.422+0000 7f6eba89a640 1 -- 172.21.15.28:0/3356925777 shutdown_connections 2024-04-25T01:32:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.422+0000 7f6eba89a640 1 -- 172.21.15.28:0/3356925777 wait complete. 2024-04-25T01:32:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.423+0000 7f6eba89a640 1 Processor -- start 2024-04-25T01:32:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.423+0000 7f6eba89a640 1 -- start start 2024-04-25T01:32:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.423+0000 7f6eba89a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6eb41006a0 0x7f6eb4197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.423+0000 7f6eba89a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6eb4101890 0x7f6eb4198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.423+0000 7f6eba89a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eb4102250 0x7f6eb419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.423+0000 7f6eba89a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb410c320 con 0x7f6eb4102250 2024-04-25T01:32:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.423+0000 7f6eba89a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb410c1a0 con 0x7f6eb41006a0 2024-04-25T01:32:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.423+0000 7f6eba89a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb410c4a0 con 0x7f6eb4101890 2024-04-25T01:32:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.423+0000 7f6eb8e37640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eb4102250 0x7f6eb419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.424+0000 7f6eb8e37640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eb4102250 0x7f6eb419f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54748/0 (socket says 172.21.15.28:54748) 2024-04-25T01:32:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.424+0000 7f6eb37fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6eb4101890 0x7f6eb4198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.424+0000 7f6eb8e37640 1 -- 172.21.15.28:0/55104449 learned_addr learned my addr 172.21.15.28:0/55104449 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.424+0000 7f6eb3fff640 1 --2- 172.21.15.28:0/55104449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6eb41006a0 0x7f6eb4197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.424+0000 7f6eb8e37640 1 -- 172.21.15.28:0/55104449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6eb4101890 msgr2=0x7f6eb4198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.424+0000 7f6eb8e37640 1 --2- 172.21.15.28:0/55104449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6eb4101890 0x7f6eb4198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.424+0000 7f6eb8e37640 1 -- 172.21.15.28:0/55104449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6eb41006a0 msgr2=0x7f6eb4197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.424+0000 7f6eb8e37640 1 --2- 172.21.15.28:0/55104449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6eb41006a0 0x7f6eb4197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.424+0000 7f6eb8e37640 1 -- 172.21.15.28:0/55104449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb419f6d0 con 0x7f6eb4102250 2024-04-25T01:32:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.424+0000 7f6eb37fe640 1 --2- 172.21.15.28:0/55104449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6eb4101890 0x7f6eb4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.424+0000 7f6eb3fff640 1 --2- 172.21.15.28:0/55104449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6eb41006a0 0x7f6eb4197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.424+0000 7f6eb8e37640 1 --2- 172.21.15.28:0/55104449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eb4102250 0x7f6eb419f060 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6eac005e30 tx=0x7f6eac004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.424+0000 7f6eb17fa640 1 -- 172.21.15.28:0/55104449 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eac003b70 con 0x7f6eb4102250 2024-04-25T01:32:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.424+0000 7f6eb17fa640 1 -- 172.21.15.28:0/55104449 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6eac003d10 con 0x7f6eb4102250 2024-04-25T01:32:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.424+0000 7f6eba89a640 1 -- 172.21.15.28:0/55104449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eb419f960 con 0x7f6eb4102250 2024-04-25T01:32:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.424+0000 7f6eb17fa640 1 -- 172.21.15.28:0/55104449 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eac02fad0 con 0x7f6eb4102250 2024-04-25T01:32:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.424+0000 7f6eba89a640 1 -- 172.21.15.28:0/55104449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eb419feb0 con 0x7f6eb4102250 2024-04-25T01:32:59.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.425+0000 7f6eba89a640 1 -- 172.21.15.28:0/55104449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eb41a0080 con 0x7f6eb4102250 2024-04-25T01:32:59.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.427+0000 7f6eb17fa640 1 -- 172.21.15.28:0/55104449 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f6eac0045c0 con 0x7f6eb4102250 2024-04-25T01:32:59.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.427+0000 7f6eb17fa640 1 --2- 172.21.15.28:0/55104449 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6e9c0a2be0 0x7f6e9c0a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:59.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.427+0000 7f6eb3fff640 1 -- 172.21.15.28:0/55104449 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6e9c0a2be0 msgr2=0x7f6e9c0a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1149553567 2024-04-25T01:32:59.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.427+0000 7f6eb17fa640 1 -- 172.21.15.28:0/55104449 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f6eac0f05d0 con 0x7f6eb4102250 2024-04-25T01:32:59.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.427+0000 7f6eb3fff640 1 --2- 172.21.15.28:0/55104449 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6e9c0a2be0 0x7f6e9c0a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:32:59.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.431+0000 7f6eb17fa640 1 -- 172.21.15.28:0/55104449 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6eac0bd1d0 con 0x7f6eb4102250 2024-04-25T01:32:59.483 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:59 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:59.086+0000 7fa50c434200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:32:59.483 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:32:59 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:32:59.180+0000 7fa50c434200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:32:59.561 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:59 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:59.229+0000 7faee5fab200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:32:59.561 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:59 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:59.355+0000 7faee5fab200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:32:59.561 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:32:59 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:32:59.475+0000 7faee5fab200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:32:59.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.628+0000 7f6eb3fff640 1 -- 172.21.15.28:0/55104449 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6e9c0a2be0 msgr2=0x7f6e9c0a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1149553567 2024-04-25T01:32:59.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.628+0000 7f6eb3fff640 1 --2- 172.21.15.28:0/55104449 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6e9c0a2be0 0x7f6e9c0a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:32:59.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.648+0000 7f6eba89a640 1 -- 172.21.15.28:0/55104449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6eb41a0250 con 0x7f6eb4102250 2024-04-25T01:32:59.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.648+0000 7f6eb17fa640 1 -- 172.21.15.28:0/55104449 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f6eac0bcad0 con 0x7f6eb4102250 2024-04-25T01:32:59.626 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T01:32:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.650+0000 7f6eba89a640 1 -- 172.21.15.28:0/55104449 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6e9c0a2be0 msgr2=0x7f6e9c0a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:32:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.650+0000 7f6eba89a640 1 --2- 172.21.15.28:0/55104449 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6e9c0a2be0 0x7f6e9c0a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.650+0000 7f6eba89a640 1 -- 172.21.15.28:0/55104449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eb4102250 msgr2=0x7f6eb419f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.650+0000 7f6eba89a640 1 --2- 172.21.15.28:0/55104449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eb4102250 0x7f6eb419f060 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6eac005e30 tx=0x7f6eac004ab0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.650+0000 7f6eba89a640 1 -- 172.21.15.28:0/55104449 shutdown_connections 2024-04-25T01:32:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.650+0000 7f6eba89a640 1 --2- 172.21.15.28:0/55104449 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f6e9c0a2be0 0x7f6e9c0a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.650+0000 7f6eba89a640 1 --2- 172.21.15.28:0/55104449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6eb4102250 0x7f6eb419f060 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.650+0000 7f6eba89a640 1 --2- 172.21.15.28:0/55104449 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6eb4101890 0x7f6eb4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.650+0000 7f6eba89a640 1 --2- 172.21.15.28:0/55104449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6eb41006a0 0x7f6eb4197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.650+0000 7f6eba89a640 1 -- 172.21.15.28:0/55104449 >> 172.21.15.28:0/55104449 conn(0x7f6eb40fbc30 msgr2=0x7f6eb40fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.651+0000 7f6eba89a640 1 -- 172.21.15.28:0/55104449 shutdown_connections 2024-04-25T01:32:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.651+0000 7f6eba89a640 1 -- 172.21.15.28:0/55104449 wait complete. 2024-04-25T01:32:59.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:32:59.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.769+0000 7f9165bb6640 1 -- 172.21.15.28:0/4283332871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91601017a0 msgr2=0x7f9160101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:59.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.769+0000 7f9165bb6640 1 --2- 172.21.15.28:0/4283332871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91601017a0 0x7f9160101bc0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9148009900 tx=0x7f914802a410 comp rx=0 tx=0).stop 2024-04-25T01:32:59.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.769+0000 7f9165bb6640 1 -- 172.21.15.28:0/4283332871 shutdown_connections 2024-04-25T01:32:59.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.769+0000 7f9165bb6640 1 --2- 172.21.15.28:0/4283332871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9160103350 0x7f9160109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.769+0000 7f9165bb6640 1 --2- 172.21.15.28:0/4283332871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9160102990 0x7f9160102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.769+0000 7f9165bb6640 1 --2- 172.21.15.28:0/4283332871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91601017a0 0x7f9160101bc0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.769+0000 7f9165bb6640 1 -- 172.21.15.28:0/4283332871 >> 172.21.15.28:0/4283332871 conn(0x7f91600fcd10 msgr2=0x7f91600ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:59.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.769+0000 7f9165bb6640 1 -- 172.21.15.28:0/4283332871 shutdown_connections 2024-04-25T01:32:59.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.769+0000 7f9165bb6640 1 -- 172.21.15.28:0/4283332871 wait complete. 2024-04-25T01:32:59.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.770+0000 7f9165bb6640 1 Processor -- start 2024-04-25T01:32:59.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.770+0000 7f9165bb6640 1 -- start start 2024-04-25T01:32:59.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.770+0000 7f9165bb6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91601017a0 0x7f9160197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:59.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.770+0000 7f9165bb6640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9160102990 0x7f9160198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:59.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.770+0000 7f9165bb6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9160103350 0x7f916019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:59.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.770+0000 7f9165bb6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f916010d410 con 0x7f9160103350 2024-04-25T01:32:59.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.770+0000 7f9165bb6640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f916010d290 con 0x7f91601017a0 2024-04-25T01:32:59.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.770+0000 7f9165bb6640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f916010d590 con 0x7f9160102990 2024-04-25T01:32:59.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.770+0000 7f915ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9160103350 0x7f916019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:59.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.770+0000 7f915ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9160103350 0x7f916019f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54766/0 (socket says 172.21.15.28:54766) 2024-04-25T01:32:59.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.770+0000 7f915ffff640 1 -- 172.21.15.28:0/2133026426 learned_addr learned my addr 172.21.15.28:0/2133026426 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:32:59.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.770+0000 7f915effd640 1 --2- 172.21.15.28:0/2133026426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9160102990 0x7f9160198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:59.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.770+0000 7f915f7fe640 1 --2- 172.21.15.28:0/2133026426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91601017a0 0x7f9160197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:32:59.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.771+0000 7f915ffff640 1 -- 172.21.15.28:0/2133026426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9160102990 msgr2=0x7f9160198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:59.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.771+0000 7f915ffff640 1 --2- 172.21.15.28:0/2133026426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9160102990 0x7f9160198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.771+0000 7f915ffff640 1 -- 172.21.15.28:0/2133026426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91601017a0 msgr2=0x7f9160197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:59.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.771+0000 7f915ffff640 1 --2- 172.21.15.28:0/2133026426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91601017a0 0x7f9160197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.771+0000 7f915ffff640 1 -- 172.21.15.28:0/2133026426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f916019f700 con 0x7f9160103350 2024-04-25T01:32:59.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.771+0000 7f915effd640 1 --2- 172.21.15.28:0/2133026426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9160102990 0x7f9160198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:59.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.771+0000 7f915f7fe640 1 --2- 172.21.15.28:0/2133026426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91601017a0 0x7f9160197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:32:59.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.771+0000 7f915ffff640 1 --2- 172.21.15.28:0/2133026426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9160103350 0x7f916019f090 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f915800eb10 tx=0x7f915800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:32:59.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.771+0000 7f915cff9640 1 -- 172.21.15.28:0/2133026426 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9158004b20 con 0x7f9160103350 2024-04-25T01:32:59.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.771+0000 7f9165bb6640 1 -- 172.21.15.28:0/2133026426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f916019f9f0 con 0x7f9160103350 2024-04-25T01:32:59.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.771+0000 7f915cff9640 1 -- 172.21.15.28:0/2133026426 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9158004cc0 con 0x7f9160103350 2024-04-25T01:32:59.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.771+0000 7f915cff9640 1 -- 172.21.15.28:0/2133026426 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9158010760 con 0x7f9160103350 2024-04-25T01:32:59.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.771+0000 7f9165bb6640 1 -- 172.21.15.28:0/2133026426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f916019ff40 con 0x7f9160103350 2024-04-25T01:32:59.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.772+0000 7f9165bb6640 1 -- 172.21.15.28:0/2133026426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9160108b40 con 0x7f9160103350 2024-04-25T01:32:59.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.774+0000 7f915cff9640 1 -- 172.21.15.28:0/2133026426 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f9158005000 con 0x7f9160103350 2024-04-25T01:32:59.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.774+0000 7f915cff9640 1 --2- 172.21.15.28:0/2133026426 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f914c0a2be0 0x7f914c0a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:32:59.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.774+0000 7f915f7fe640 1 -- 172.21.15.28:0/2133026426 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f914c0a2be0 msgr2=0x7f914c0a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1149553567 2024-04-25T01:32:59.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.774+0000 7f915cff9640 1 -- 172.21.15.28:0/2133026426 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f91580cfba0 con 0x7f9160103350 2024-04-25T01:32:59.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.775+0000 7f915f7fe640 1 --2- 172.21.15.28:0/2133026426 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f914c0a2be0 0x7f914c0a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:32:59.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.777+0000 7f915cff9640 1 -- 172.21.15.28:0/2133026426 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9158053740 con 0x7f9160103350 2024-04-25T01:32:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.962+0000 7f9165bb6640 1 -- 172.21.15.28:0/2133026426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f916002d3c0 con 0x7f9160103350 2024-04-25T01:32:59.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.962+0000 7f915cff9640 1 -- 172.21.15.28:0/2133026426 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f915809c7a0 con 0x7f9160103350 2024-04-25T01:32:59.940 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:32:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.964+0000 7f9165bb6640 1 -- 172.21.15.28:0/2133026426 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f914c0a2be0 msgr2=0x7f914c0a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:32:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.964+0000 7f9165bb6640 1 --2- 172.21.15.28:0/2133026426 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f914c0a2be0 0x7f914c0a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.964+0000 7f9165bb6640 1 -- 172.21.15.28:0/2133026426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9160103350 msgr2=0x7f916019f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:32:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.965+0000 7f9165bb6640 1 --2- 172.21.15.28:0/2133026426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9160103350 0x7f916019f090 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f915800eb10 tx=0x7f915800c2d0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.965+0000 7f9165bb6640 1 -- 172.21.15.28:0/2133026426 shutdown_connections 2024-04-25T01:32:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.965+0000 7f9165bb6640 1 --2- 172.21.15.28:0/2133026426 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f914c0a2be0 0x7f914c0a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.965+0000 7f9165bb6640 1 --2- 172.21.15.28:0/2133026426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9160103350 0x7f916019f090 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.965+0000 7f9165bb6640 1 --2- 172.21.15.28:0/2133026426 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9160102990 0x7f9160198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.965+0000 7f9165bb6640 1 --2- 172.21.15.28:0/2133026426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91601017a0 0x7f9160197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:32:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.965+0000 7f9165bb6640 1 -- 172.21.15.28:0/2133026426 >> 172.21.15.28:0/2133026426 conn(0x7f91600fcd10 msgr2=0x7f9160105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:32:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.965+0000 7f9165bb6640 1 -- 172.21.15.28:0/2133026426 shutdown_connections 2024-04-25T01:32:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:32:59.965+0000 7f9165bb6640 1 -- 172.21.15.28:0/2133026426 wait complete. 2024-04-25T01:32:59.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:59 smithi028 ceph-mon[29390]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T01:32:59.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:59 smithi028 ceph-mon[29390]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:32:59.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:59 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1512634964' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:59.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:32:59 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/55104449' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:32:59.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:32:59.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:59 smithi130 ceph-mon[32702]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T01:32:59.961 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:59 smithi130 ceph-mon[32702]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:32:59.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:59 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1512634964' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:32:59.962 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:32:59 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/55104449' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:32:59.962 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:59 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:59.707+0000 7f7ca34ae200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:32:59.963 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:32:59 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z[34568]: 2024-04-25T01:32:59.921+0000 7f7ca34ae200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:33:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.083+0000 7f6749709640 1 -- 172.21.15.28:0/3129842163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6744103340 msgr2=0x7f6744109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.083+0000 7f6749709640 1 --2- 172.21.15.28:0/3129842163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6744103340 0x7f6744109bd0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f673c009ca0 tx=0x7f673c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.083+0000 7f6749709640 1 -- 172.21.15.28:0/3129842163 shutdown_connections 2024-04-25T01:33:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.083+0000 7f6749709640 1 --2- 172.21.15.28:0/3129842163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6744103340 0x7f6744109bd0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.083+0000 7f6749709640 1 --2- 172.21.15.28:0/3129842163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6744102980 0x7f6744102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.083+0000 7f6749709640 1 --2- 172.21.15.28:0/3129842163 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6744101790 0x7f6744101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.083+0000 7f6749709640 1 -- 172.21.15.28:0/3129842163 >> 172.21.15.28:0/3129842163 conn(0x7f67440fcd20 msgr2=0x7f67440ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.083+0000 7f6749709640 1 -- 172.21.15.28:0/3129842163 shutdown_connections 2024-04-25T01:33:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.083+0000 7f6749709640 1 -- 172.21.15.28:0/3129842163 wait complete. 2024-04-25T01:33:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.083+0000 7f6749709640 1 Processor -- start 2024-04-25T01:33:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.083+0000 7f6749709640 1 -- start start 2024-04-25T01:33:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.084+0000 7f6749709640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6744101790 0x7f67440754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.084+0000 7f6749709640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6744102980 0x7f6744073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.084+0000 7f6749709640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6744103340 0x7f6744074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.084+0000 7f6749709640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f674410d410 con 0x7f6744101790 2024-04-25T01:33:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.084+0000 7f6749709640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f674410d290 con 0x7f6744103340 2024-04-25T01:33:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.084+0000 7f6749709640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f674410d590 con 0x7f6744102980 2024-04-25T01:33:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.084+0000 7f6742ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6744101790 0x7f67440754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.084+0000 7f6742ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6744101790 0x7f67440754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54792/0 (socket says 172.21.15.28:54792) 2024-04-25T01:33:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.084+0000 7f6742ffd640 1 -- 172.21.15.28:0/1559929410 learned_addr learned my addr 172.21.15.28:0/1559929410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.084+0000 7f67437fe640 1 --2- 172.21.15.28:0/1559929410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6744103340 0x7f6744074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.084+0000 7f67427fc640 1 --2- 172.21.15.28:0/1559929410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6744102980 0x7f6744073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.084+0000 7f67437fe640 1 -- 172.21.15.28:0/1559929410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6744102980 msgr2=0x7f6744073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.084+0000 7f67437fe640 1 --2- 172.21.15.28:0/1559929410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6744102980 0x7f6744073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.084+0000 7f67437fe640 1 -- 172.21.15.28:0/1559929410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6744101790 msgr2=0x7f67440754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.084+0000 7f67437fe640 1 --2- 172.21.15.28:0/1559929410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6744101790 0x7f67440754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.084+0000 7f67437fe640 1 -- 172.21.15.28:0/1559929410 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6744074900 con 0x7f6744103340 2024-04-25T01:33:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.084+0000 7f6742ffd640 1 --2- 172.21.15.28:0/1559929410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6744101790 0x7f67440754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.084+0000 7f67437fe640 1 --2- 172.21.15.28:0/1559929410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6744103340 0x7f6744074070 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f673c031ae0 tx=0x7f673c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.085+0000 7f673bfff640 1 -- 172.21.15.28:0/1559929410 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673c003b70 con 0x7f6744103340 2024-04-25T01:33:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.085+0000 7f6749709640 1 -- 172.21.15.28:0/1559929410 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67441a1b80 con 0x7f6744103340 2024-04-25T01:33:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.085+0000 7f673bfff640 1 -- 172.21.15.28:0/1559929410 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f673c003d10 con 0x7f6744103340 2024-04-25T01:33:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.085+0000 7f673bfff640 1 -- 172.21.15.28:0/1559929410 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673c02fc40 con 0x7f6744103340 2024-04-25T01:33:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.085+0000 7f6749709640 1 -- 172.21.15.28:0/1559929410 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67441a20f0 con 0x7f6744103340 2024-04-25T01:33:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.086+0000 7f6749709640 1 -- 172.21.15.28:0/1559929410 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6744108ab0 con 0x7f6744103340 2024-04-25T01:33:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.087+0000 7f673bfff640 1 -- 172.21.15.28:0/1559929410 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7f673c004050 con 0x7f6744103340 2024-04-25T01:33:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.087+0000 7f673bfff640 1 --2- 172.21.15.28:0/1559929410 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67300a2be0 0x7f67300a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.088+0000 7f6742ffd640 1 -- 172.21.15.28:0/1559929410 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67300a2be0 msgr2=0x7f67300a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1149553567 2024-04-25T01:33:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.088+0000 7f673bfff640 1 -- 172.21.15.28:0/1559929410 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f673c0f0750 con 0x7f6744103340 2024-04-25T01:33:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.088+0000 7f6742ffd640 1 --2- 172.21.15.28:0/1559929410 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67300a2be0 0x7f67300a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:33:00.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.094+0000 7f673bfff640 1 -- 172.21.15.28:0/1559929410 <== mon.2 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f673c0bd410 con 0x7f6744103340 2024-04-25T01:33:00.230 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:33:00.028+0000 7fa50c434200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:33:00.230 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:59 smithi032 ceph-mon[32461]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T01:33:00.230 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:59 smithi032 ceph-mon[32461]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T01:33:00.231 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:59 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1512634964' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:33:00.231 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:32:59 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/55104449' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:33:00.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.278+0000 7f6749709640 1 -- 172.21.15.28:0/1559929410 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f674402d170 con 0x7f6744103340 2024-04-25T01:33:00.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.278+0000 7f673bfff640 1 -- 172.21.15.28:0/1559929410 <== mon.2 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f673c0bcd10 con 0x7f6744103340 2024-04-25T01:33:00.256 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T01:33:00.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.280+0000 7f6749709640 1 -- 172.21.15.28:0/1559929410 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67300a2be0 msgr2=0x7f67300a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:33:00.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.280+0000 7f6749709640 1 --2- 172.21.15.28:0/1559929410 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67300a2be0 0x7f67300a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.280+0000 7f6749709640 1 -- 172.21.15.28:0/1559929410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6744103340 msgr2=0x7f6744074070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:00.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.280+0000 7f6749709640 1 --2- 172.21.15.28:0/1559929410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6744103340 0x7f6744074070 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f673c031ae0 tx=0x7f673c004ab0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.281+0000 7f6749709640 1 -- 172.21.15.28:0/1559929410 shutdown_connections 2024-04-25T01:33:00.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.281+0000 7f6749709640 1 --2- 172.21.15.28:0/1559929410 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7f67300a2be0 0x7f67300a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.281+0000 7f6749709640 1 --2- 172.21.15.28:0/1559929410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6744103340 0x7f6744074070 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.281+0000 7f6749709640 1 --2- 172.21.15.28:0/1559929410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6744102980 0x7f6744073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.281+0000 7f6749709640 1 --2- 172.21.15.28:0/1559929410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6744101790 0x7f67440754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.281+0000 7f6749709640 1 -- 172.21.15.28:0/1559929410 >> 172.21.15.28:0/1559929410 conn(0x7f67440fcd20 msgr2=0x7f6744105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:00.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.281+0000 7f6749709640 1 -- 172.21.15.28:0/1559929410 shutdown_connections 2024-04-25T01:33:00.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.281+0000 7f6749709640 1 -- 172.21.15.28:0/1559929410 wait complete. 2024-04-25T01:33:00.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:33:00.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.399+0000 7ff4ccd7e640 1 -- 172.21.15.28:0/506984560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4c8100680 msgr2=0x7ff4c8100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:00.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.399+0000 7ff4ccd7e640 1 --2- 172.21.15.28:0/506984560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4c8100680 0x7ff4c8100aa0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8009900 tx=0x7ff4b802a410 comp rx=0 tx=0).stop 2024-04-25T01:33:00.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.399+0000 7ff4ccd7e640 1 -- 172.21.15.28:0/506984560 shutdown_connections 2024-04-25T01:33:00.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.399+0000 7ff4ccd7e640 1 --2- 172.21.15.28:0/506984560 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4c8102230 0x7ff4c8108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.399+0000 7ff4ccd7e640 1 --2- 172.21.15.28:0/506984560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4c8101870 0x7ff4c8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.399+0000 7ff4ccd7e640 1 --2- 172.21.15.28:0/506984560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4c8100680 0x7ff4c8100aa0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.399+0000 7ff4ccd7e640 1 -- 172.21.15.28:0/506984560 >> 172.21.15.28:0/506984560 conn(0x7ff4c80fbbf0 msgr2=0x7ff4c80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.399+0000 7ff4ccd7e640 1 -- 172.21.15.28:0/506984560 shutdown_connections 2024-04-25T01:33:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.399+0000 7ff4ccd7e640 1 -- 172.21.15.28:0/506984560 wait complete. 2024-04-25T01:33:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.399+0000 7ff4ccd7e640 1 Processor -- start 2024-04-25T01:33:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.400+0000 7ff4ccd7e640 1 -- start start 2024-04-25T01:33:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.400+0000 7ff4ccd7e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4c8100680 0x7ff4c8197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.400+0000 7ff4ccd7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4c8101870 0x7ff4c8198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.400+0000 7ff4ccd7e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4c8102230 0x7ff4c819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.400+0000 7ff4ccd7e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c810c300 con 0x7ff4c8101870 2024-04-25T01:33:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.400+0000 7ff4ccd7e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c810c180 con 0x7ff4c8100680 2024-04-25T01:33:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.400+0000 7ff4ccd7e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c810c480 con 0x7ff4c8102230 2024-04-25T01:33:00.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.400+0000 7ff4bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4c8101870 0x7ff4c8198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.400+0000 7ff4bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4c8101870 0x7ff4c8198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54810/0 (socket says 172.21.15.28:54810) 2024-04-25T01:33:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.400+0000 7ff4bffff640 1 -- 172.21.15.28:0/3210592446 learned_addr learned my addr 172.21.15.28:0/3210592446 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.400+0000 7ff4c659c640 1 --2- 172.21.15.28:0/3210592446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4c8100680 0x7ff4c8197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.400+0000 7ff4c6d9d640 1 --2- 172.21.15.28:0/3210592446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4c8102230 0x7ff4c819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.400+0000 7ff4bffff640 1 -- 172.21.15.28:0/3210592446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4c8102230 msgr2=0x7ff4c819f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.400+0000 7ff4bffff640 1 --2- 172.21.15.28:0/3210592446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4c8102230 0x7ff4c819f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.400+0000 7ff4bffff640 1 -- 172.21.15.28:0/3210592446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4c8100680 msgr2=0x7ff4c8197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.400+0000 7ff4bffff640 1 --2- 172.21.15.28:0/3210592446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4c8100680 0x7ff4c8197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.400+0000 7ff4bffff640 1 -- 172.21.15.28:0/3210592446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c819f6f0 con 0x7ff4c8101870 2024-04-25T01:33:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.400+0000 7ff4bffff640 1 --2- 172.21.15.28:0/3210592446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4c8101870 0x7ff4c8198010 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff4b000d950 tx=0x7ff4b000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.400+0000 7ff4bf7fe640 1 -- 172.21.15.28:0/3210592446 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b0017070 con 0x7ff4c8101870 2024-04-25T01:33:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.401+0000 7ff4bf7fe640 1 -- 172.21.15.28:0/3210592446 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4b0009c30 con 0x7ff4c8101870 2024-04-25T01:33:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.401+0000 7ff4ccd7e640 1 -- 172.21.15.28:0/3210592446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4c819f9e0 con 0x7ff4c8101870 2024-04-25T01:33:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.401+0000 7ff4bf7fe640 1 -- 172.21.15.28:0/3210592446 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b000f6b0 con 0x7ff4c8101870 2024-04-25T01:33:00.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.401+0000 7ff4ccd7e640 1 -- 172.21.15.28:0/3210592446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4c819ff30 con 0x7ff4c8101870 2024-04-25T01:33:00.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.402+0000 7ff4ccd7e640 1 -- 172.21.15.28:0/3210592446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4c81a0160 con 0x7ff4c8101870 2024-04-25T01:33:00.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.403+0000 7ff4bf7fe640 1 -- 172.21.15.28:0/3210592446 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137850+0+0 (secure 0 0 0) 0x7ff4b00109d0 con 0x7ff4c8101870 2024-04-25T01:33:00.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.404+0000 7ff4bf7fe640 1 --2- 172.21.15.28:0/3210592446 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4b40a2be0 0x7ff4b40a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:00.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.404+0000 7ff4c659c640 1 -- 172.21.15.28:0/3210592446 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4b40a2be0 msgr2=0x7ff4b40a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1149553567 2024-04-25T01:33:00.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.404+0000 7ff4bf7fe640 1 -- 172.21.15.28:0/3210592446 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7ff4b00cf1a0 con 0x7ff4c8101870 2024-04-25T01:33:00.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.404+0000 7ff4c659c640 1 --2- 172.21.15.28:0/3210592446 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4b40a2be0 0x7ff4b40a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:33:00.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.408+0000 7ff4bf7fe640 1 -- 172.21.15.28:0/3210592446 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff4b0009de0 con 0x7ff4c8101870 2024-04-25T01:33:00.483 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y[33853]: 2024-04-25T01:33:00.253+0000 7fa50c434200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:33:00.561 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:33:00.356+0000 7faee5fab200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:33:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.605+0000 7ff4c659c640 1 -- 172.21.15.28:0/3210592446 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4b40a2be0 msgr2=0x7ff4b40a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/1149553567 2024-04-25T01:33:00.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.605+0000 7ff4c659c640 1 --2- 172.21.15.28:0/3210592446 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4b40a2be0 0x7ff4b40a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:33:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.627+0000 7ff4bf7fe640 1 -- 172.21.15.28:0/3210592446 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 20) ==== 137518+0+0 (secure 0 0 0) 0x7ff4b0097cb0 con 0x7ff4c8101870 2024-04-25T01:33:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.627+0000 7ff4bf7fe640 1 -- 172.21.15.28:0/3210592446 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4b40a2be0 msgr2=0x7ff4b40a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:33:00.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.627+0000 7ff4bf7fe640 1 --2- 172.21.15.28:0/3210592446 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4b40a2be0 0x7ff4b40a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.631+0000 7ff4ccd7e640 1 -- 172.21.15.28:0/3210592446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff4c81a0330 con 0x7ff4c8101870 2024-04-25T01:33:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.632+0000 7ff4bf7fe640 1 -- 172.21.15.28:0/3210592446 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7ff4b0097020 con 0x7ff4c8101870 2024-04-25T01:33:00.610 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:33:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.634+0000 7ff4ccd7e640 1 -- 172.21.15.28:0/3210592446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4c8101870 msgr2=0x7ff4c8198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.634+0000 7ff4ccd7e640 1 --2- 172.21.15.28:0/3210592446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4c8101870 0x7ff4c8198010 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff4b000d950 tx=0x7ff4b000de10 comp rx=0 tx=0).stop 2024-04-25T01:33:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.634+0000 7ff4ccd7e640 1 -- 172.21.15.28:0/3210592446 shutdown_connections 2024-04-25T01:33:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.634+0000 7ff4ccd7e640 1 --2- 172.21.15.28:0/3210592446 >> [v2:172.21.15.28:6800/1149553567,v1:172.21.15.28:6801/1149553567] conn(0x7ff4b40a2be0 0x7ff4b40a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.634+0000 7ff4ccd7e640 1 --2- 172.21.15.28:0/3210592446 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4c8102230 0x7ff4c819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.634+0000 7ff4ccd7e640 1 --2- 172.21.15.28:0/3210592446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4c8101870 0x7ff4c8198010 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.634+0000 7ff4ccd7e640 1 --2- 172.21.15.28:0/3210592446 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4c8100680 0x7ff4c8197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.634+0000 7ff4ccd7e640 1 -- 172.21.15.28:0/3210592446 >> 172.21.15.28:0/3210592446 conn(0x7ff4c80fbbf0 msgr2=0x7ff4c80688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.634+0000 7ff4ccd7e640 1 -- 172.21.15.28:0/3210592446 shutdown_connections 2024-04-25T01:33:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.635+0000 7ff4ccd7e640 1 -- 172.21.15.28:0/3210592446 wait complete. 2024-04-25T01:33:00.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:33:00.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.769+0000 7f7c999e7640 1 -- 172.21.15.28:0/1752070336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c94102250 msgr2=0x7f7c94108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.769+0000 7f7c999e7640 1 --2- 172.21.15.28:0/1752070336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c94102250 0x7f7c94108ae0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c009be0 tx=0x7f7c8c02a6c0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.769+0000 7f7c999e7640 1 -- 172.21.15.28:0/1752070336 shutdown_connections 2024-04-25T01:33:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.769+0000 7f7c999e7640 1 --2- 172.21.15.28:0/1752070336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c94102250 0x7f7c94108ae0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.769+0000 7f7c999e7640 1 --2- 172.21.15.28:0/1752070336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c94101890 0x7f7c94101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.769+0000 7f7c999e7640 1 --2- 172.21.15.28:0/1752070336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c941006a0 0x7f7c94100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.769+0000 7f7c999e7640 1 -- 172.21.15.28:0/1752070336 >> 172.21.15.28:0/1752070336 conn(0x7f7c940fbc30 msgr2=0x7f7c940fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.769+0000 7f7c999e7640 1 -- 172.21.15.28:0/1752070336 shutdown_connections 2024-04-25T01:33:00.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.769+0000 7f7c999e7640 1 -- 172.21.15.28:0/1752070336 wait complete. 2024-04-25T01:33:00.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.770+0000 7f7c999e7640 1 Processor -- start 2024-04-25T01:33:00.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.770+0000 7f7c999e7640 1 -- start start 2024-04-25T01:33:00.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.771+0000 7f7c999e7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c941006a0 0x7f7c94197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.771+0000 7f7c999e7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c94101890 0x7f7c94197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.771+0000 7f7c999e7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c94102250 0x7f7c9419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.771+0000 7f7c999e7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c9410d3a0 con 0x7f7c941006a0 2024-04-25T01:33:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.771+0000 7f7c999e7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7c9410d220 con 0x7f7c94102250 2024-04-25T01:33:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.771+0000 7f7c999e7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7c9410d520 con 0x7f7c94101890 2024-04-25T01:33:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.771+0000 7f7c937fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c94102250 0x7f7c9419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.771+0000 7f7c937fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c94102250 0x7f7c9419eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:37756/0 (socket says 172.21.15.28:37756) 2024-04-25T01:33:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.771+0000 7f7c937fe640 1 -- 172.21.15.28:0/1722478351 learned_addr learned my addr 172.21.15.28:0/1722478351 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.771+0000 7f7c927fc640 1 --2- 172.21.15.28:0/1722478351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c94101890 0x7f7c94197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:00.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.771+0000 7f7c92ffd640 1 --2- 172.21.15.28:0/1722478351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c941006a0 0x7f7c94197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:00.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.772+0000 7f7c937fe640 1 -- 172.21.15.28:0/1722478351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c94101890 msgr2=0x7f7c94197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:00.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.772+0000 7f7c937fe640 1 --2- 172.21.15.28:0/1722478351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c94101890 0x7f7c94197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.772+0000 7f7c937fe640 1 -- 172.21.15.28:0/1722478351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c941006a0 msgr2=0x7f7c94197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:00.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.772+0000 7f7c937fe640 1 --2- 172.21.15.28:0/1722478351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c941006a0 0x7f7c94197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.772+0000 7f7c937fe640 1 -- 172.21.15.28:0/1722478351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c9419f660 con 0x7f7c94102250 2024-04-25T01:33:00.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.772+0000 7f7c937fe640 1 --2- 172.21.15.28:0/1722478351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c94102250 0x7f7c9419eff0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c0305b0 tx=0x7f7c8c0305e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:00.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.772+0000 7f7c989e5640 1 -- 172.21.15.28:0/1722478351 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c041020 con 0x7f7c94102250 2024-04-25T01:33:00.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.772+0000 7f7c989e5640 1 -- 172.21.15.28:0/1722478351 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c8c037040 con 0x7f7c94102250 2024-04-25T01:33:00.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.772+0000 7f7c989e5640 1 -- 172.21.15.28:0/1722478351 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c036070 con 0x7f7c94102250 2024-04-25T01:33:00.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.772+0000 7f7c999e7640 1 -- 172.21.15.28:0/1722478351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c941091a0 con 0x7f7c94102250 2024-04-25T01:33:00.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.773+0000 7f7c999e7640 1 -- 172.21.15.28:0/1722478351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c94109640 con 0x7f7c94102250 2024-04-25T01:33:00.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.774+0000 7f7c999e7640 1 -- 172.21.15.28:0/1722478351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c941079c0 con 0x7f7c94102250 2024-04-25T01:33:00.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.775+0000 7f7c989e5640 1 -- 172.21.15.28:0/1722478351 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 20) ==== 137518+0+0 (secure 0 0 0) 0x7f7c8c044020 con 0x7f7c94102250 2024-04-25T01:33:00.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.776+0000 7f7c989e5640 1 -- 172.21.15.28:0/1722478351 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f7c8c0efa80 con 0x7f7c94102250 2024-04-25T01:33:00.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:00.780+0000 7f7c989e5640 1 -- 172.21.15.28:0/1722478351 <== mon.2 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7c8c04b020 con 0x7f7c94102250 2024-04-25T01:33:00.845 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:33:00.614+0000 7faee5fab200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:33:00.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.013+0000 7f7c999e7640 1 -- 172.21.15.28:0/1722478351 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c9410aa20 con 0x7f7c94102250 2024-04-25T01:33:00.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.015+0000 7f7c989e5640 1 -- 172.21.15.28:0/1722478351 <== mon.2 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24941 (secure 0 0 0) 0x7f7c8c004670 con 0x7f7c94102250 2024-04-25T01:33:00.993 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:33:00.993 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:33:00.617205+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/1149553567":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/500268183":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/3499593842":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/3630315328":"2024-04-26T01:33:00.617174+0000","172.21.15.28:6800/1149553567":"2024-04-26T01:33:00.617174+0000","172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3812371257":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:33:00.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.018+0000 7f7c999e7640 1 -- 172.21.15.28:0/1722478351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c94102250 msgr2=0x7f7c9419eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:00.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.018+0000 7f7c999e7640 1 --2- 172.21.15.28:0/1722478351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c94102250 0x7f7c9419eff0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c0305b0 tx=0x7f7c8c0305e0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.019+0000 7f7c999e7640 1 -- 172.21.15.28:0/1722478351 shutdown_connections 2024-04-25T01:33:00.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.019+0000 7f7c999e7640 1 --2- 172.21.15.28:0/1722478351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7c94102250 0x7f7c9419eff0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.019+0000 7f7c999e7640 1 --2- 172.21.15.28:0/1722478351 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7c94101890 0x7f7c94197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.019+0000 7f7c999e7640 1 --2- 172.21.15.28:0/1722478351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c941006a0 0x7f7c94197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:00.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.019+0000 7f7c999e7640 1 -- 172.21.15.28:0/1722478351 >> 172.21.15.28:0/1722478351 conn(0x7f7c940fbc30 msgr2=0x7f7c94104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:00.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.019+0000 7f7c999e7640 1 -- 172.21.15.28:0/1722478351 shutdown_connections 2024-04-25T01:33:00.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.019+0000 7f7c999e7640 1 -- 172.21.15.28:0/1722478351 wait complete. 2024-04-25T01:33:01.007 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:25:48.767414+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:30:32.282430+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:30:33.127049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:30:36.796132+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:33:01.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:33:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.141+0000 7efe2fce7640 1 -- 172.21.15.28:0/2471252922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe28104b90 msgr2=0x7efe28106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.141+0000 7efe2fce7640 1 --2- 172.21.15.28:0/2471252922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe28104b90 0x7efe28106f80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7efe18009900 tx=0x7efe1802a410 comp rx=0 tx=0).stop 2024-04-25T01:33:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.141+0000 7efe2fce7640 1 -- 172.21.15.28:0/2471252922 shutdown_connections 2024-04-25T01:33:01.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.141+0000 7efe2fce7640 1 --2- 172.21.15.28:0/2471252922 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efe281074c0 0x7efe281098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.141+0000 7efe2fce7640 1 --2- 172.21.15.28:0/2471252922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe28104b90 0x7efe28106f80 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.141+0000 7efe2fce7640 1 --2- 172.21.15.28:0/2471252922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe28102260 0x7efe28104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.141+0000 7efe2fce7640 1 -- 172.21.15.28:0/2471252922 >> 172.21.15.28:0/2471252922 conn(0x7efe280fbbf0 msgr2=0x7efe280fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:01.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.141+0000 7efe2fce7640 1 -- 172.21.15.28:0/2471252922 shutdown_connections 2024-04-25T01:33:01.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.141+0000 7efe2fce7640 1 -- 172.21.15.28:0/2471252922 wait complete. 2024-04-25T01:33:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.142+0000 7efe2fce7640 1 Processor -- start 2024-04-25T01:33:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.142+0000 7efe2fce7640 1 -- start start 2024-04-25T01:33:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.142+0000 7efe2fce7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efe28102260 0x7efe28197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.142+0000 7efe2fce7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe281074c0 0x7efe28198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.142+0000 7efe2fce7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe2819d280 0x7efe2819f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.142+0000 7efe2fce7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efe2810d400 con 0x7efe2819d280 2024-04-25T01:33:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.142+0000 7efe2fce7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efe2810d280 con 0x7efe281074c0 2024-04-25T01:33:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.142+0000 7efe2fce7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7efe2810d580 con 0x7efe28102260 2024-04-25T01:33:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.142+0000 7efe2e284640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe2819d280 0x7efe2819f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.142+0000 7efe2d282640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe281074c0 0x7efe28198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.142+0000 7efe2d282640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe281074c0 0x7efe28198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:37816/0 (socket says 172.21.15.28:37816) 2024-04-25T01:33:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.142+0000 7efe2e284640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe2819d280 0x7efe2819f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54942/0 (socket says 172.21.15.28:54942) 2024-04-25T01:33:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.142+0000 7efe2d282640 1 -- 172.21.15.28:0/185566967 learned_addr learned my addr 172.21.15.28:0/185566967 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.142+0000 7efe2da83640 1 --2- 172.21.15.28:0/185566967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efe28102260 0x7efe28197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.142+0000 7efe2d282640 1 -- 172.21.15.28:0/185566967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efe28102260 msgr2=0x7efe28197d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.143+0000 7efe2d282640 1 --2- 172.21.15.28:0/185566967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efe28102260 0x7efe28197d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.143+0000 7efe2d282640 1 -- 172.21.15.28:0/185566967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe2819d280 msgr2=0x7efe2819f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.143+0000 7efe2d282640 1 --2- 172.21.15.28:0/185566967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe2819d280 0x7efe2819f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.143+0000 7efe2d282640 1 -- 172.21.15.28:0/185566967 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe28100050 con 0x7efe281074c0 2024-04-25T01:33:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.143+0000 7efe2e284640 1 --2- 172.21.15.28:0/185566967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe2819d280 0x7efe2819f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.143+0000 7efe2d282640 1 --2- 172.21.15.28:0/185566967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe281074c0 0x7efe28198260 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efe18009230 tx=0x7efe180042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.143+0000 7efe1effd640 1 -- 172.21.15.28:0/185566967 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe1803f030 con 0x7efe281074c0 2024-04-25T01:33:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.143+0000 7efe2fce7640 1 -- 172.21.15.28:0/185566967 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe281002e0 con 0x7efe281074c0 2024-04-25T01:33:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.143+0000 7efe1effd640 1 -- 172.21.15.28:0/185566967 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe18031a20 con 0x7efe281074c0 2024-04-25T01:33:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.143+0000 7efe1effd640 1 -- 172.21.15.28:0/185566967 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe18035070 con 0x7efe281074c0 2024-04-25T01:33:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.143+0000 7efe2fce7640 1 -- 172.21.15.28:0/185566967 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe28100780 con 0x7efe281074c0 2024-04-25T01:33:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.144+0000 7efe2fce7640 1 -- 172.21.15.28:0/185566967 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe2802d170 con 0x7efe281074c0 2024-04-25T01:33:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.145+0000 7efe1effd640 1 -- 172.21.15.28:0/185566967 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 20) ==== 137518+0+0 (secure 0 0 0) 0x7efe18042030 con 0x7efe281074c0 2024-04-25T01:33:01.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.146+0000 7efe1effd640 1 -- 172.21.15.28:0/185566967 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7efe180ef580 con 0x7efe281074c0 2024-04-25T01:33:01.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.150+0000 7efe1effd640 1 -- 172.21.15.28:0/185566967 <== mon.2 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efe18049030 con 0x7efe281074c0 2024-04-25T01:33:01.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: Standby manager daemon z restarted 2024-04-25T01:33:01.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: Standby manager daemon z started 2024-04-25T01:33:01.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.? 172.21.15.130:0/1288514923' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:33:01.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.? 172.21.15.130:0/1288514923' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:33:01.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.? 172.21.15.130:0/1288514923' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:33:01.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.? 172.21.15.130:0/1288514923' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:33:01.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2133026426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:33:01.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: Standby manager daemon y restarted 2024-04-25T01:33:01.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: Standby manager daemon y started 2024-04-25T01:33:01.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.? 172.21.15.32:0/3894033142' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:33:01.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.? 172.21.15.32:0/3894033142' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:33:01.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.? 172.21.15.32:0/3894033142' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:33:01.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.? 172.21.15.32:0/3894033142' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:33:01.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1559929410' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:33:01.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: Active manager daemon x restarted 2024-04-25T01:33:01.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: Activating manager daemon x 2024-04-25T01:33:01.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T01:33:01.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: mgrmap e20: x(active, starting, since 0.0123883s), standbys: z, y 2024-04-25T01:33:01.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:33:01.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:33:01.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:33:01.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:33:01.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:33:01.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3210592446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:33:01.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:33:01.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:33:01.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:33:01.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:33:01.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:33:01.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:33:01.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:33:01.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:33:01.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:33:01.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:33:01.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:33:01.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:33:01.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:33:01.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:33:01.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:33:01.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:33:01.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:33:01.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:33:01.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:33:01.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:33:01.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:33:01.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:33:01.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:33:01.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:33:01.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:33:01.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:33:01.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:33:01.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:33:01.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: Manager daemon x is now available 2024-04-25T01:33:01.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:33:01.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:33:01.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:00 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:33:01.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: Standby manager daemon z restarted 2024-04-25T01:33:01.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: Standby manager daemon z started 2024-04-25T01:33:01.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.? 172.21.15.130:0/1288514923' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:33:01.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.? 172.21.15.130:0/1288514923' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:33:01.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.? 172.21.15.130:0/1288514923' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:33:01.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.? 172.21.15.130:0/1288514923' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:33:01.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2133026426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:33:01.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: Standby manager daemon y restarted 2024-04-25T01:33:01.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: Standby manager daemon y started 2024-04-25T01:33:01.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.? 172.21.15.32:0/3894033142' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:33:01.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.? 172.21.15.32:0/3894033142' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:33:01.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.? 172.21.15.32:0/3894033142' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:33:01.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.? 172.21.15.32:0/3894033142' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:33:01.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1559929410' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:33:01.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: Active manager daemon x restarted 2024-04-25T01:33:01.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: Activating manager daemon x 2024-04-25T01:33:01.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T01:33:01.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: mgrmap e20: x(active, starting, since 0.0123883s), standbys: z, y 2024-04-25T01:33:01.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:33:01.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:33:01.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:33:01.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:33:01.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:33:01.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3210592446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:33:01.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:33:01.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:33:01.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:33:01.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:33:01.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:33:01.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:33:01.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:33:01.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:33:01.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:33:01.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:33:01.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:33:01.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:33:01.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:33:01.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:33:01.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:33:01.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:33:01.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:33:01.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:33:01.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:33:01.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:33:01.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:33:01.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:33:01.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:33:01.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:33:01.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:33:01.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:33:01.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:33:01.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:33:01.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: Manager daemon x is now available 2024-04-25T01:33:01.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:33:01.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:33:01.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:00 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:33:01.311 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:33:00.893+0000 7fae4492e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:33:01.311 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:33:00.893+0000 7fae4492e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:33:01.311 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:33:00.893+0000 7fae4492e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:33:01.311 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:33:00.893+0000 7fae4492e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:33:01.311 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:33:00.893+0000 7fae4492e700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:33:01.311 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:33:00.894+0000 7fae47133700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:33:01.311 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:33:00.894+0000 7fae47133700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:33:01.311 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:33:00.894+0000 7fae47133700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:33:01.312 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:33:00.894+0000 7fae47133700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:33:01.312 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:33:00.894+0000 7fae47133700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:33:01.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: Standby manager daemon z restarted 2024-04-25T01:33:01.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: Standby manager daemon z started 2024-04-25T01:33:01.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.? 172.21.15.130:0/1288514923' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:33:01.312 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.? 172.21.15.130:0/1288514923' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:33:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.? 172.21.15.130:0/1288514923' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:33:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.? 172.21.15.130:0/1288514923' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:33:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2133026426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:33:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: Standby manager daemon y restarted 2024-04-25T01:33:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: Standby manager daemon y started 2024-04-25T01:33:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.? 172.21.15.32:0/3894033142' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:33:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.? 172.21.15.32:0/3894033142' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:33:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.? 172.21.15.32:0/3894033142' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:33:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.? 172.21.15.32:0/3894033142' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:33:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1559929410' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:33:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: Active manager daemon x restarted 2024-04-25T01:33:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: Activating manager daemon x 2024-04-25T01:33:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T01:33:01.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: mgrmap e20: x(active, starting, since 0.0123883s), standbys: z, y 2024-04-25T01:33:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:33:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:33:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:33:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:33:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:33:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3210592446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:33:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:33:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:33:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:33:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:33:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:33:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:33:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:33:01.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:33:01.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:33:01.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:33:01.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:33:01.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:33:01.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:33:01.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:33:01.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:33:01.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:33:01.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:33:01.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:33:01.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:33:01.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:33:01.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:33:01.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:33:01.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:33:01.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:33:01.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:33:01.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:33:01.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:33:01.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:33:01.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: Manager daemon x is now available 2024-04-25T01:33:01.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:33:01.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:33:01.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:00 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:33:01.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.337+0000 7efe2fce7640 1 -- 172.21.15.28:0/185566967 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efe280fd250 con 0x7efe281074c0 2024-04-25T01:33:01.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.337+0000 7efe1effd640 1 -- 172.21.15.28:0/185566967 <== mon.2 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7efe180bbdd0 con 0x7efe281074c0 2024-04-25T01:33:01.318 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T01:33:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.340+0000 7efe2fce7640 1 -- 172.21.15.28:0/185566967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe281074c0 msgr2=0x7efe28198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.340+0000 7efe2fce7640 1 --2- 172.21.15.28:0/185566967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe281074c0 0x7efe28198260 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efe18009230 tx=0x7efe180042f0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.340+0000 7efe2fce7640 1 -- 172.21.15.28:0/185566967 shutdown_connections 2024-04-25T01:33:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.340+0000 7efe2fce7640 1 --2- 172.21.15.28:0/185566967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe2819d280 0x7efe2819f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.340+0000 7efe2fce7640 1 --2- 172.21.15.28:0/185566967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe281074c0 0x7efe28198260 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.340+0000 7efe2fce7640 1 --2- 172.21.15.28:0/185566967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efe28102260 0x7efe28197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.340+0000 7efe2fce7640 1 -- 172.21.15.28:0/185566967 >> 172.21.15.28:0/185566967 conn(0x7efe280fbbf0 msgr2=0x7efe28106a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.340+0000 7efe2fce7640 1 -- 172.21.15.28:0/185566967 shutdown_connections 2024-04-25T01:33:01.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.340+0000 7efe2fce7640 1 -- 172.21.15.28:0/185566967 wait complete. 2024-04-25T01:33:01.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:33:01.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.462+0000 7f661de9e640 1 -- 172.21.15.28:0/3766759714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6618100680 msgr2=0x7f6618100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:01.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.462+0000 7f661de9e640 1 --2- 172.21.15.28:0/3766759714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6618100680 0x7f6618100aa0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f6608009900 tx=0x7f660802a410 comp rx=0 tx=0).stop 2024-04-25T01:33:01.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.462+0000 7f661de9e640 1 -- 172.21.15.28:0/3766759714 shutdown_connections 2024-04-25T01:33:01.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.462+0000 7f661de9e640 1 --2- 172.21.15.28:0/3766759714 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6618102230 0x7f6618108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.462+0000 7f661de9e640 1 --2- 172.21.15.28:0/3766759714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6618101870 0x7f6618101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.462+0000 7f661de9e640 1 --2- 172.21.15.28:0/3766759714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6618100680 0x7f6618100aa0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.462+0000 7f661de9e640 1 -- 172.21.15.28:0/3766759714 >> 172.21.15.28:0/3766759714 conn(0x7f66180fbbf0 msgr2=0x7f66180fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.462+0000 7f661de9e640 1 -- 172.21.15.28:0/3766759714 shutdown_connections 2024-04-25T01:33:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.462+0000 7f661de9e640 1 -- 172.21.15.28:0/3766759714 wait complete. 2024-04-25T01:33:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.463+0000 7f661de9e640 1 Processor -- start 2024-04-25T01:33:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.463+0000 7f661de9e640 1 -- start start 2024-04-25T01:33:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.463+0000 7f661de9e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6618100680 0x7f6618104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.463+0000 7f661de9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6618101870 0x7f6618102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.463+0000 7f661de9e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6618102230 0x7f6618103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.463+0000 7f661de9e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f661810c300 con 0x7f6618101870 2024-04-25T01:33:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.463+0000 7f661de9e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f661810c180 con 0x7f6618102230 2024-04-25T01:33:01.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.463+0000 7f661de9e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f661810c480 con 0x7f6618100680 2024-04-25T01:33:01.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.463+0000 7f6616ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6618101870 0x7f6618102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:01.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.463+0000 7f6616ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6618101870 0x7f6618102c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54964/0 (socket says 172.21.15.28:54964) 2024-04-25T01:33:01.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.463+0000 7f6616ffd640 1 -- 172.21.15.28:0/560444590 learned_addr learned my addr 172.21.15.28:0/560444590 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:01.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.463+0000 7f6617fff640 1 --2- 172.21.15.28:0/560444590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6618102230 0x7f6618103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:01.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.463+0000 7f66177fe640 1 --2- 172.21.15.28:0/560444590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6618100680 0x7f6618104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:01.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.464+0000 7f6616ffd640 1 -- 172.21.15.28:0/560444590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6618100680 msgr2=0x7f6618104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:01.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.464+0000 7f6616ffd640 1 --2- 172.21.15.28:0/560444590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6618100680 0x7f6618104610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.464+0000 7f6616ffd640 1 -- 172.21.15.28:0/560444590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6618102230 msgr2=0x7f6618103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:01.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.464+0000 7f6616ffd640 1 --2- 172.21.15.28:0/560444590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6618102230 0x7f6618103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.464+0000 7f6616ffd640 1 -- 172.21.15.28:0/560444590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6618103a20 con 0x7f6618101870 2024-04-25T01:33:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.464+0000 7f6617fff640 1 --2- 172.21.15.28:0/560444590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6618102230 0x7f6618103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.464+0000 7f6616ffd640 1 --2- 172.21.15.28:0/560444590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6618101870 0x7f6618102c50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f660c00e9f0 tx=0x7f660c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.464+0000 7f6614ff9640 1 -- 172.21.15.28:0/560444590 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c00cd80 con 0x7f6618101870 2024-04-25T01:33:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.464+0000 7f6614ff9640 1 -- 172.21.15.28:0/560444590 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f660c010910 con 0x7f6618101870 2024-04-25T01:33:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.464+0000 7f661de9e640 1 -- 172.21.15.28:0/560444590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66181a1bf0 con 0x7f6618101870 2024-04-25T01:33:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.464+0000 7f6614ff9640 1 -- 172.21.15.28:0/560444590 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c018990 con 0x7f6618101870 2024-04-25T01:33:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.464+0000 7f661de9e640 1 -- 172.21.15.28:0/560444590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66181a2140 con 0x7f6618101870 2024-04-25T01:33:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.465+0000 7f661de9e640 1 -- 172.21.15.28:0/560444590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66181a2370 con 0x7f6618101870 2024-04-25T01:33:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.466+0000 7f6614ff9640 1 -- 172.21.15.28:0/560444590 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137518+0+0 (secure 0 0 0) 0x7f660c010ac0 con 0x7f6618101870 2024-04-25T01:33:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.467+0000 7f6614ff9640 1 -- 172.21.15.28:0/560444590 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f660c0cf210 con 0x7f6618101870 2024-04-25T01:33:01.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.471+0000 7f6614ff9640 1 -- 172.21.15.28:0/560444590 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f660c09bac0 con 0x7f6618101870 2024-04-25T01:33:01.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.639+0000 7f6614ff9640 1 -- 172.21.15.28:0/560444590 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7f660c097810 con 0x7f6618101870 2024-04-25T01:33:01.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.640+0000 7f6614ff9640 1 --2- 172.21.15.28:0/560444590 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f66000aaea0 0x7f66000ad290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:01.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.640+0000 7f66177fe640 1 --2- 172.21.15.28:0/560444590 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f66000aaea0 0x7f66000ad290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:01.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.641+0000 7f66177fe640 1 --2- 172.21.15.28:0/560444590 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f66000aaea0 0x7f66000ad290 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6608005ab0 tx=0x7f6608000bc0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:01.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.677+0000 7f661de9e640 1 -- 172.21.15.28:0/560444590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f66181a2540 con 0x7f6618101870 2024-04-25T01:33:01.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.677+0000 7f6614ff9640 1 -- 172.21.15.28:0/560444590 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f660c091640 con 0x7f6618101870 2024-04-25T01:33:01.655 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:33:01.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.679+0000 7f661de9e640 1 -- 172.21.15.28:0/560444590 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f66000aaea0 msgr2=0x7f66000ad290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:01.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.679+0000 7f661de9e640 1 --2- 172.21.15.28:0/560444590 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f66000aaea0 0x7f66000ad290 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6608005ab0 tx=0x7f6608000bc0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.679+0000 7f661de9e640 1 -- 172.21.15.28:0/560444590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6618101870 msgr2=0x7f6618102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.679+0000 7f661de9e640 1 --2- 172.21.15.28:0/560444590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6618101870 0x7f6618102c50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f660c00e9f0 tx=0x7f660c00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.679+0000 7f661de9e640 1 -- 172.21.15.28:0/560444590 shutdown_connections 2024-04-25T01:33:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.680+0000 7f661de9e640 1 --2- 172.21.15.28:0/560444590 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f66000aaea0 0x7f66000ad290 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.680+0000 7f661de9e640 1 --2- 172.21.15.28:0/560444590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6618102230 0x7f6618103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.680+0000 7f661de9e640 1 --2- 172.21.15.28:0/560444590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6618101870 0x7f6618102c50 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.680+0000 7f661de9e640 1 --2- 172.21.15.28:0/560444590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6618100680 0x7f6618104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.680+0000 7f661de9e640 1 -- 172.21.15.28:0/560444590 >> 172.21.15.28:0/560444590 conn(0x7f66180fbbf0 msgr2=0x7f66180fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.680+0000 7f661de9e640 1 -- 172.21.15.28:0/560444590 shutdown_connections 2024-04-25T01:33:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.680+0000 7f661de9e640 1 -- 172.21.15.28:0/560444590 wait complete. 2024-04-25T01:33:01.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:33:01.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.804+0000 7ff38c2e1640 1 -- 172.21.15.28:0/1535090334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3841074d0 msgr2=0x7ff3841098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:01.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.804+0000 7ff38c2e1640 1 --2- 172.21.15.28:0/1535090334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3841074d0 0x7ff3841098c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009900 tx=0x7ff37c02a410 comp rx=0 tx=0).stop 2024-04-25T01:33:01.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.804+0000 7ff38c2e1640 1 -- 172.21.15.28:0/1535090334 shutdown_connections 2024-04-25T01:33:01.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.804+0000 7ff38c2e1640 1 --2- 172.21.15.28:0/1535090334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3841074d0 0x7ff3841098c0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.804+0000 7ff38c2e1640 1 --2- 172.21.15.28:0/1535090334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff384104ba0 0x7ff384106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.804+0000 7ff38c2e1640 1 --2- 172.21.15.28:0/1535090334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff384102270 0x7ff384104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.804+0000 7ff38c2e1640 1 -- 172.21.15.28:0/1535090334 >> 172.21.15.28:0/1535090334 conn(0x7ff3840fbc00 msgr2=0x7ff3840fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.804+0000 7ff38c2e1640 1 -- 172.21.15.28:0/1535090334 shutdown_connections 2024-04-25T01:33:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.804+0000 7ff38c2e1640 1 -- 172.21.15.28:0/1535090334 wait complete. 2024-04-25T01:33:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.804+0000 7ff38c2e1640 1 Processor -- start 2024-04-25T01:33:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.804+0000 7ff38c2e1640 1 -- start start 2024-04-25T01:33:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.804+0000 7ff38c2e1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff384102270 0x7ff384197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.804+0000 7ff38c2e1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff384104ba0 0x7ff384197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.804+0000 7ff38c2e1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3841074d0 0x7ff38419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.804+0000 7ff38c2e1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff38410d430 con 0x7ff384104ba0 2024-04-25T01:33:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.804+0000 7ff38c2e1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff38410d2b0 con 0x7ff384102270 2024-04-25T01:33:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.804+0000 7ff38c2e1640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff38410d5b0 con 0x7ff3841074d0 2024-04-25T01:33:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.805+0000 7ff38987c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff384104ba0 0x7ff384197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:01.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.805+0000 7ff38987c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff384104ba0 0x7ff384197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54994/0 (socket says 172.21.15.28:54994) 2024-04-25T01:33:01.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.805+0000 7ff38987c640 1 -- 172.21.15.28:0/3955829508 learned_addr learned my addr 172.21.15.28:0/3955829508 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:01.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.805+0000 7ff38a87e640 1 --2- 172.21.15.28:0/3955829508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3841074d0 0x7ff38419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:01.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.805+0000 7ff38a07d640 1 --2- 172.21.15.28:0/3955829508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff384102270 0x7ff384197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:01.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.805+0000 7ff38987c640 1 -- 172.21.15.28:0/3955829508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3841074d0 msgr2=0x7ff38419f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:01.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.805+0000 7ff38987c640 1 --2- 172.21.15.28:0/3955829508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3841074d0 0x7ff38419f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.805+0000 7ff38987c640 1 -- 172.21.15.28:0/3955829508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff384102270 msgr2=0x7ff384197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:01.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.805+0000 7ff38987c640 1 --2- 172.21.15.28:0/3955829508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff384102270 0x7ff384197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.805+0000 7ff38987c640 1 -- 172.21.15.28:0/3955829508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff38419f6d0 con 0x7ff384104ba0 2024-04-25T01:33:01.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.805+0000 7ff38987c640 1 --2- 172.21.15.28:0/3955829508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff384104ba0 0x7ff384197ff0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff37400e9f0 tx=0x7ff37400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:01.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.805+0000 7ff37b7fe640 1 -- 172.21.15.28:0/3955829508 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37400ccc0 con 0x7ff384104ba0 2024-04-25T01:33:01.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.805+0000 7ff37b7fe640 1 -- 172.21.15.28:0/3955829508 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff374005500 con 0x7ff384104ba0 2024-04-25T01:33:01.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.805+0000 7ff38c2e1640 1 -- 172.21.15.28:0/3955829508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff38419f9c0 con 0x7ff384104ba0 2024-04-25T01:33:01.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.805+0000 7ff37b7fe640 1 -- 172.21.15.28:0/3955829508 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff374010690 con 0x7ff384104ba0 2024-04-25T01:33:01.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.805+0000 7ff38c2e1640 1 -- 172.21.15.28:0/3955829508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff38419fec0 con 0x7ff384104ba0 2024-04-25T01:33:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.806+0000 7ff38c2e1640 1 -- 172.21.15.28:0/3955829508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3840fd6f0 con 0x7ff384104ba0 2024-04-25T01:33:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.807+0000 7ff37b7fe640 1 -- 172.21.15.28:0/3955829508 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7ff3740046c0 con 0x7ff384104ba0 2024-04-25T01:33:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.808+0000 7ff37b7fe640 1 --2- 172.21.15.28:0/3955829508 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff3700a2990 0x7ff3700a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.808+0000 7ff38a07d640 1 --2- 172.21.15.28:0/3955829508 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff3700a2990 0x7ff3700a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.808+0000 7ff37b7fe640 1 -- 172.21.15.28:0/3955829508 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff3740cf410 con 0x7ff384104ba0 2024-04-25T01:33:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.809+0000 7ff38a07d640 1 --2- 172.21.15.28:0/3955829508 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff3700a2990 0x7ff3700a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff380007350 tx=0x7ff3800072e0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:01.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:01.813+0000 7ff37b7fe640 1 -- 172.21.15.28:0/3955829508 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff37409c2d0 con 0x7ff384104ba0 2024-04-25T01:33:01.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:01 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1722478351' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:33:01.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:01 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/185566967' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:33:01.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:01 smithi028 ceph-mon[29390]: mgrmap e21: x(active, since 1.02233s), standbys: z, y 2024-04-25T01:33:01.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:01 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/560444590' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:33:01.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.002+0000 7ff38c2e1640 1 -- 172.21.15.28:0/3955829508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff384062340 con 0x7ff384104ba0 2024-04-25T01:33:01.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.003+0000 7ff37b7fe640 1 -- 172.21.15.28:0/3955829508 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7ff374086030 con 0x7ff384104ba0 2024-04-25T01:33:01.980 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T01:33:01.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.005+0000 7ff38c2e1640 1 -- 172.21.15.28:0/3955829508 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff3700a2990 msgr2=0x7ff3700a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:01.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.005+0000 7ff38c2e1640 1 --2- 172.21.15.28:0/3955829508 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff3700a2990 0x7ff3700a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff380007350 tx=0x7ff3800072e0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.005+0000 7ff38c2e1640 1 -- 172.21.15.28:0/3955829508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff384104ba0 msgr2=0x7ff384197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:01.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.005+0000 7ff38c2e1640 1 --2- 172.21.15.28:0/3955829508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff384104ba0 0x7ff384197ff0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff37400e9f0 tx=0x7ff37400eeb0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.005+0000 7ff38c2e1640 1 -- 172.21.15.28:0/3955829508 shutdown_connections 2024-04-25T01:33:01.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.005+0000 7ff38c2e1640 1 --2- 172.21.15.28:0/3955829508 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff3700a2990 0x7ff3700a4e50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.005+0000 7ff38c2e1640 1 --2- 172.21.15.28:0/3955829508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff3841074d0 0x7ff38419f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.005+0000 7ff38c2e1640 1 --2- 172.21.15.28:0/3955829508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff384104ba0 0x7ff384197ff0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.005+0000 7ff38c2e1640 1 --2- 172.21.15.28:0/3955829508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff384102270 0x7ff384197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:01.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.005+0000 7ff38c2e1640 1 -- 172.21.15.28:0/3955829508 >> 172.21.15.28:0/3955829508 conn(0x7ff3840fbc00 msgr2=0x7ff384102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:01.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.005+0000 7ff38c2e1640 1 -- 172.21.15.28:0/3955829508 shutdown_connections 2024-04-25T01:33:01.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.005+0000 7ff38c2e1640 1 -- 172.21.15.28:0/3955829508 wait complete. 2024-04-25T01:33:01.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:33:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.129+0000 7f5a6211b640 1 -- 172.21.15.28:0/1909541625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5c104b30 msgr2=0x7f5a5c106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.129+0000 7f5a6211b640 1 --2- 172.21.15.28:0/1909541625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5c104b30 0x7f5a5c106f20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5a50009900 tx=0x7f5a5002a410 comp rx=0 tx=0).stop 2024-04-25T01:33:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.129+0000 7f5a6211b640 1 -- 172.21.15.28:0/1909541625 shutdown_connections 2024-04-25T01:33:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.129+0000 7f5a6211b640 1 --2- 172.21.15.28:0/1909541625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a5c107460 0x7f5a5c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.129+0000 7f5a6211b640 1 --2- 172.21.15.28:0/1909541625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5c104b30 0x7f5a5c106f20 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.129+0000 7f5a6211b640 1 --2- 172.21.15.28:0/1909541625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a5c102200 0x7f5a5c1045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.129+0000 7f5a6211b640 1 -- 172.21.15.28:0/1909541625 >> 172.21.15.28:0/1909541625 conn(0x7f5a5c0fbbf0 msgr2=0x7f5a5c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.129+0000 7f5a6211b640 1 -- 172.21.15.28:0/1909541625 shutdown_connections 2024-04-25T01:33:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.129+0000 7f5a6211b640 1 -- 172.21.15.28:0/1909541625 wait complete. 2024-04-25T01:33:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.129+0000 7f5a6211b640 1 Processor -- start 2024-04-25T01:33:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.129+0000 7f5a6211b640 1 -- start start 2024-04-25T01:33:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.129+0000 7f5a6211b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a5c102200 0x7f5a5c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a6211b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a5c104b30 0x7f5a5c198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a6211b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5c107460 0x7f5a5c19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a6211b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5c10d450 con 0x7f5a5c107460 2024-04-25T01:33:02.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a6211b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5c10d2d0 con 0x7f5a5c104b30 2024-04-25T01:33:02.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a6211b640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5c10d5d0 con 0x7f5a5c102200 2024-04-25T01:33:02.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a5bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5c107460 0x7f5a5c19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:02.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a5bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5c107460 0x7f5a5c19f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55014/0 (socket says 172.21.15.28:55014) 2024-04-25T01:33:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a5bfff640 1 -- 172.21.15.28:0/53527557 learned_addr learned my addr 172.21.15.28:0/53527557 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a5b7fe640 1 --2- 172.21.15.28:0/53527557 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a5c102200 0x7f5a5c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a5affd640 1 --2- 172.21.15.28:0/53527557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a5c104b30 0x7f5a5c198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a5bfff640 1 -- 172.21.15.28:0/53527557 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a5c102200 msgr2=0x7f5a5c197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a5bfff640 1 --2- 172.21.15.28:0/53527557 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a5c102200 0x7f5a5c197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a5bfff640 1 -- 172.21.15.28:0/53527557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a5c104b30 msgr2=0x7f5a5c198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a5bfff640 1 --2- 172.21.15.28:0/53527557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a5c104b30 0x7f5a5c198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a5bfff640 1 -- 172.21.15.28:0/53527557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a5c19f6f0 con 0x7f5a5c107460 2024-04-25T01:33:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a5bfff640 1 --2- 172.21.15.28:0/53527557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5c107460 0x7f5a5c19f080 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5a5400eaa0 tx=0x7f5a5400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a58ff9640 1 -- 172.21.15.28:0/53527557 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5400cdb0 con 0x7f5a5c107460 2024-04-25T01:33:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a58ff9640 1 -- 172.21.15.28:0/53527557 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a54005020 con 0x7f5a5c107460 2024-04-25T01:33:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a6211b640 1 -- 172.21.15.28:0/53527557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a5c19f9e0 con 0x7f5a5c107460 2024-04-25T01:33:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a58ff9640 1 -- 172.21.15.28:0/53527557 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a540116b0 con 0x7f5a5c107460 2024-04-25T01:33:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.130+0000 7f5a6211b640 1 -- 172.21.15.28:0/53527557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a5c19ff00 con 0x7f5a5c107460 2024-04-25T01:33:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.131+0000 7f5a6211b640 1 -- 172.21.15.28:0/53527557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a5c0fd6e0 con 0x7f5a5c107460 2024-04-25T01:33:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.132+0000 7f5a58ff9640 1 -- 172.21.15.28:0/53527557 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7f5a54005560 con 0x7f5a5c107460 2024-04-25T01:33:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.133+0000 7f5a58ff9640 1 --2- 172.21.15.28:0/53527557 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f5a440a2990 0x7f5a440a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.133+0000 7f5a5b7fe640 1 --2- 172.21.15.28:0/53527557 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f5a440a2990 0x7f5a440a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.133+0000 7f5a58ff9640 1 -- 172.21.15.28:0/53527557 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f5a540cf310 con 0x7f5a5c107460 2024-04-25T01:33:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.134+0000 7f5a5b7fe640 1 --2- 172.21.15.28:0/53527557 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f5a440a2990 0x7f5a440a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c009bd0 tx=0x7f5a4c005f90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.138+0000 7f5a58ff9640 1 -- 172.21.15.28:0/53527557 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5a54016030 con 0x7f5a5c107460 2024-04-25T01:33:02.134 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:01 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1722478351' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:33:02.134 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:01 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/185566967' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:33:02.134 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:01 smithi130 ceph-mon[32702]: mgrmap e21: x(active, since 1.02233s), standbys: z, y 2024-04-25T01:33:02.134 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:01 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/560444590' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:33:02.137 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:01 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1722478351' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:33:02.137 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:01 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/185566967' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:33:02.137 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:01 smithi032 ceph-mon[32461]: mgrmap e21: x(active, since 1.02233s), standbys: z, y 2024-04-25T01:33:02.137 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:01 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/560444590' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:33:02.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.324+0000 7f5a6211b640 1 -- 172.21.15.28:0/53527557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f5a5c062340 con 0x7f5a5c107460 2024-04-25T01:33:02.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.324+0000 7f5a58ff9640 1 -- 172.21.15.28:0/53527557 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f5a5409c100 con 0x7f5a5c107460 2024-04-25T01:33:02.302 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:33:02.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.326+0000 7f5a6211b640 1 -- 172.21.15.28:0/53527557 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f5a440a2990 msgr2=0x7f5a440a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:02.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.326+0000 7f5a6211b640 1 --2- 172.21.15.28:0/53527557 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f5a440a2990 0x7f5a440a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c009bd0 tx=0x7f5a4c005f90 comp rx=0 tx=0).stop 2024-04-25T01:33:02.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.326+0000 7f5a6211b640 1 -- 172.21.15.28:0/53527557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5c107460 msgr2=0x7f5a5c19f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:02.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.326+0000 7f5a6211b640 1 --2- 172.21.15.28:0/53527557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5c107460 0x7f5a5c19f080 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5a5400eaa0 tx=0x7f5a5400ef60 comp rx=0 tx=0).stop 2024-04-25T01:33:02.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.326+0000 7f5a6211b640 1 -- 172.21.15.28:0/53527557 shutdown_connections 2024-04-25T01:33:02.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.326+0000 7f5a6211b640 1 --2- 172.21.15.28:0/53527557 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f5a440a2990 0x7f5a440a4e50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.326+0000 7f5a6211b640 1 --2- 172.21.15.28:0/53527557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5c107460 0x7f5a5c19f080 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.326+0000 7f5a6211b640 1 --2- 172.21.15.28:0/53527557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5a5c104b30 0x7f5a5c198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.326+0000 7f5a6211b640 1 --2- 172.21.15.28:0/53527557 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5a5c102200 0x7f5a5c197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.326+0000 7f5a6211b640 1 -- 172.21.15.28:0/53527557 >> 172.21.15.28:0/53527557 conn(0x7f5a5c0fbbf0 msgr2=0x7f5a5c102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:02.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.326+0000 7f5a6211b640 1 -- 172.21.15.28:0/53527557 shutdown_connections 2024-04-25T01:33:02.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.326+0000 7f5a6211b640 1 -- 172.21.15.28:0/53527557 wait complete. 2024-04-25T01:33:02.318 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-25T01:33:02.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:33:02.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.447+0000 7eff186b3640 1 -- 172.21.15.28:0/2811575365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff10100680 msgr2=0x7eff10100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:02.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.447+0000 7eff186b3640 1 --2- 172.21.15.28:0/2811575365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff10100680 0x7eff10100aa0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7eff0c009900 tx=0x7eff0c02a410 comp rx=0 tx=0).stop 2024-04-25T01:33:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.447+0000 7eff186b3640 1 -- 172.21.15.28:0/2811575365 shutdown_connections 2024-04-25T01:33:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.447+0000 7eff186b3640 1 --2- 172.21.15.28:0/2811575365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff10102230 0x7eff10108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.447+0000 7eff186b3640 1 --2- 172.21.15.28:0/2811575365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff10101870 0x7eff10101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.447+0000 7eff186b3640 1 --2- 172.21.15.28:0/2811575365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff10100680 0x7eff10100aa0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.447+0000 7eff186b3640 1 -- 172.21.15.28:0/2811575365 >> 172.21.15.28:0/2811575365 conn(0x7eff100fbbf0 msgr2=0x7eff100fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.447+0000 7eff186b3640 1 -- 172.21.15.28:0/2811575365 shutdown_connections 2024-04-25T01:33:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.447+0000 7eff186b3640 1 -- 172.21.15.28:0/2811575365 wait complete. 2024-04-25T01:33:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.448+0000 7eff186b3640 1 Processor -- start 2024-04-25T01:33:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.448+0000 7eff186b3640 1 -- start start 2024-04-25T01:33:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.448+0000 7eff186b3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff10100680 0x7eff100694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.448+0000 7eff186b3640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff10101870 0x7eff1019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.448+0000 7eff186b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff10102230 0x7eff1019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.448+0000 7eff186b3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff1010c300 con 0x7eff10102230 2024-04-25T01:33:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.448+0000 7eff186b3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7eff1010c180 con 0x7eff10100680 2024-04-25T01:33:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.448+0000 7eff186b3640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7eff1010c480 con 0x7eff10101870 2024-04-25T01:33:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.448+0000 7eff16c50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff10102230 0x7eff1019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.448+0000 7eff16c50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff10102230 0x7eff1019edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55038/0 (socket says 172.21.15.28:55038) 2024-04-25T01:33:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.448+0000 7eff16c50640 1 -- 172.21.15.28:0/2174995530 learned_addr learned my addr 172.21.15.28:0/2174995530 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.448+0000 7eff15c4e640 1 --2- 172.21.15.28:0/2174995530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff10101870 0x7eff1019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.448+0000 7eff1644f640 1 --2- 172.21.15.28:0/2174995530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff10100680 0x7eff100694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.448+0000 7eff16c50640 1 -- 172.21.15.28:0/2174995530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff10101870 msgr2=0x7eff1019c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.448+0000 7eff16c50640 1 --2- 172.21.15.28:0/2174995530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff10101870 0x7eff1019c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.448+0000 7eff16c50640 1 -- 172.21.15.28:0/2174995530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff10100680 msgr2=0x7eff100694a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.448+0000 7eff16c50640 1 --2- 172.21.15.28:0/2174995530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff10100680 0x7eff100694a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.448+0000 7eff16c50640 1 -- 172.21.15.28:0/2174995530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff1019f690 con 0x7eff10102230 2024-04-25T01:33:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.448+0000 7eff16c50640 1 --2- 172.21.15.28:0/2174995530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff10102230 0x7eff1019edd0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7eff0800eb10 tx=0x7eff0800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.449+0000 7eff077fe640 1 -- 172.21.15.28:0/2174995530 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff08004b20 con 0x7eff10102230 2024-04-25T01:33:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.449+0000 7eff077fe640 1 -- 172.21.15.28:0/2174995530 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff08004cc0 con 0x7eff10102230 2024-04-25T01:33:02.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.449+0000 7eff186b3640 1 -- 172.21.15.28:0/2174995530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff1019f980 con 0x7eff10102230 2024-04-25T01:33:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.449+0000 7eff077fe640 1 -- 172.21.15.28:0/2174995530 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff08010760 con 0x7eff10102230 2024-04-25T01:33:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.449+0000 7eff186b3640 1 -- 172.21.15.28:0/2174995530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff1019fed0 con 0x7eff10102230 2024-04-25T01:33:02.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.450+0000 7eff186b3640 1 -- 172.21.15.28:0/2174995530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff101079a0 con 0x7eff10102230 2024-04-25T01:33:02.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.451+0000 7eff077fe640 1 -- 172.21.15.28:0/2174995530 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7eff08005020 con 0x7eff10102230 2024-04-25T01:33:02.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.452+0000 7eff077fe640 1 --2- 172.21.15.28:0/2174995530 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7efefc0a2990 0x7efefc0a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:02.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.452+0000 7eff1644f640 1 --2- 172.21.15.28:0/2174995530 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7efefc0a2990 0x7efefc0a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:02.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.452+0000 7eff077fe640 1 -- 172.21.15.28:0/2174995530 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7eff080cf780 con 0x7eff10102230 2024-04-25T01:33:02.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.452+0000 7eff1644f640 1 --2- 172.21.15.28:0/2174995530 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7efefc0a2990 0x7efefc0a4e50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7eff0c005ab0 tx=0x7eff0c000bc0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:02.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.457+0000 7eff077fe640 1 -- 172.21.15.28:0/2174995530 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7eff0809c530 con 0x7eff10102230 2024-04-25T01:33:02.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.706+0000 7eff186b3640 1 -- 172.21.15.28:0/2174995530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff10062340 con 0x7eff10102230 2024-04-25T01:33:02.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.707+0000 7eff077fe640 1 -- 172.21.15.28:0/2174995530 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7eff0809be30 con 0x7eff10102230 2024-04-25T01:33:02.685 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:33:02.685 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:33:02.685 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:33:02.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.710+0000 7eff186b3640 1 -- 172.21.15.28:0/2174995530 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7efefc0a2990 msgr2=0x7efefc0a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:02.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.710+0000 7eff186b3640 1 --2- 172.21.15.28:0/2174995530 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7efefc0a2990 0x7efefc0a4e50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7eff0c005ab0 tx=0x7eff0c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.710+0000 7eff186b3640 1 -- 172.21.15.28:0/2174995530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff10102230 msgr2=0x7eff1019edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:02.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.710+0000 7eff186b3640 1 --2- 172.21.15.28:0/2174995530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff10102230 0x7eff1019edd0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7eff0800eb10 tx=0x7eff0800c2d0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.710+0000 7eff186b3640 1 -- 172.21.15.28:0/2174995530 shutdown_connections 2024-04-25T01:33:02.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.710+0000 7eff186b3640 1 --2- 172.21.15.28:0/2174995530 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7efefc0a2990 0x7efefc0a4e50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.710+0000 7eff186b3640 1 --2- 172.21.15.28:0/2174995530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff10102230 0x7eff1019edd0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.710+0000 7eff186b3640 1 --2- 172.21.15.28:0/2174995530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff10101870 0x7eff1019c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.710+0000 7eff186b3640 1 --2- 172.21.15.28:0/2174995530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff10100680 0x7eff100694a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.710+0000 7eff186b3640 1 -- 172.21.15.28:0/2174995530 >> 172.21.15.28:0/2174995530 conn(0x7eff100fbbf0 msgr2=0x7eff10104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.710+0000 7eff186b3640 1 -- 172.21.15.28:0/2174995530 shutdown_connections 2024-04-25T01:33:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.710+0000 7eff186b3640 1 -- 172.21.15.28:0/2174995530 wait complete. 2024-04-25T01:33:02.700 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-25T01:33:02.700 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-25T01:33:02.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:33:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.837+0000 7f97faa41640 1 -- 172.21.15.28:0/570290651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97f4102200 msgr2=0x7f97f41045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.837+0000 7f97faa41640 1 --2- 172.21.15.28:0/570290651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97f4102200 0x7f97f41045f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009900 tx=0x7f97e402a410 comp rx=0 tx=0).stop 2024-04-25T01:33:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.837+0000 7f97faa41640 1 -- 172.21.15.28:0/570290651 shutdown_connections 2024-04-25T01:33:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.838+0000 7f97faa41640 1 --2- 172.21.15.28:0/570290651 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97f4107460 0x7f97f41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.838+0000 7f97faa41640 1 --2- 172.21.15.28:0/570290651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97f4104b30 0x7f97f4106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.838+0000 7f97faa41640 1 --2- 172.21.15.28:0/570290651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97f4102200 0x7f97f41045f0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.838+0000 7f97faa41640 1 -- 172.21.15.28:0/570290651 >> 172.21.15.28:0/570290651 conn(0x7f97f40fbbf0 msgr2=0x7f97f40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.838+0000 7f97faa41640 1 -- 172.21.15.28:0/570290651 shutdown_connections 2024-04-25T01:33:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.838+0000 7f97faa41640 1 -- 172.21.15.28:0/570290651 wait complete. 2024-04-25T01:33:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.838+0000 7f97faa41640 1 Processor -- start 2024-04-25T01:33:02.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.838+0000 7f97faa41640 1 -- start start 2024-04-25T01:33:02.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.838+0000 7f97faa41640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97f4102200 0x7f97f4197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:02.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.838+0000 7f97faa41640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97f4104b30 0x7f97f4197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:02.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.838+0000 7f97faa41640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97f4107460 0x7f97f419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:02.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.838+0000 7f97faa41640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97f410d400 con 0x7f97f4102200 2024-04-25T01:33:02.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.838+0000 7f97faa41640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f97f410d280 con 0x7f97f4104b30 2024-04-25T01:33:02.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.839+0000 7f97faa41640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f97f410d580 con 0x7f97f4107460 2024-04-25T01:33:02.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.839+0000 7f97f3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97f4102200 0x7f97f4197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:02.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.839+0000 7f97f3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97f4102200 0x7f97f4197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55084/0 (socket says 172.21.15.28:55084) 2024-04-25T01:33:02.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.839+0000 7f97f3fff640 1 -- 172.21.15.28:0/1202673137 learned_addr learned my addr 172.21.15.28:0/1202673137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:02.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.839+0000 7f97f37fe640 1 --2- 172.21.15.28:0/1202673137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97f4104b30 0x7f97f4197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:02.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.839+0000 7f97f8fde640 1 --2- 172.21.15.28:0/1202673137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97f4107460 0x7f97f419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:02.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.839+0000 7f97f37fe640 1 -- 172.21.15.28:0/1202673137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97f4107460 msgr2=0x7f97f419f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:02.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.839+0000 7f97f37fe640 1 --2- 172.21.15.28:0/1202673137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97f4107460 0x7f97f419f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.839+0000 7f97f37fe640 1 -- 172.21.15.28:0/1202673137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97f4102200 msgr2=0x7f97f4197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:02.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.839+0000 7f97f37fe640 1 --2- 172.21.15.28:0/1202673137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97f4102200 0x7f97f4197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:02.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.839+0000 7f97f37fe640 1 -- 172.21.15.28:0/1202673137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f419f6d0 con 0x7f97f4104b30 2024-04-25T01:33:02.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.839+0000 7f97f37fe640 1 --2- 172.21.15.28:0/1202673137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97f4104b30 0x7f97f4197ff0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f97e800e9c0 tx=0x7f97e800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:02.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.839+0000 7f97f17fa640 1 -- 172.21.15.28:0/1202673137 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e800ccc0 con 0x7f97f4104b30 2024-04-25T01:33:02.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.839+0000 7f97faa41640 1 -- 172.21.15.28:0/1202673137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97f419f9c0 con 0x7f97f4104b30 2024-04-25T01:33:02.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.840+0000 7f97f17fa640 1 -- 172.21.15.28:0/1202673137 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97e8010910 con 0x7f97f4104b30 2024-04-25T01:33:02.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.840+0000 7f97faa41640 1 -- 172.21.15.28:0/1202673137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97f419ff60 con 0x7f97f4104b30 2024-04-25T01:33:02.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.840+0000 7f97f17fa640 1 -- 172.21.15.28:0/1202673137 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e8018880 con 0x7f97f4104b30 2024-04-25T01:33:02.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.840+0000 7f97faa41640 1 -- 172.21.15.28:0/1202673137 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97f40fd6e0 con 0x7f97f4104b30 2024-04-25T01:33:02.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.842+0000 7f97f17fa640 1 -- 172.21.15.28:0/1202673137 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 21) ==== 137645+0+0 (secure 0 0 0) 0x7f97e8010ac0 con 0x7f97f4104b30 2024-04-25T01:33:02.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.843+0000 7f97f17fa640 1 --2- 172.21.15.28:0/1202673137 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97dc0a2990 0x7f97dc0a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:02.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.843+0000 7f97f17fa640 1 -- 172.21.15.28:0/1202673137 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f97e80cf920 con 0x7f97f4104b30 2024-04-25T01:33:02.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.843+0000 7f97f3fff640 1 --2- 172.21.15.28:0/1202673137 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97dc0a2990 0x7f97dc0a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.843+0000 7f97f3fff640 1 --2- 172.21.15.28:0/1202673137 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97dc0a2990 0x7f97dc0a4e50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f97e4031440 tx=0x7f97e40315b0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:02.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.846+0000 7f97f17fa640 1 -- 172.21.15.28:0/1202673137 <== mon.2 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f97e809c080 con 0x7f97f4104b30 2024-04-25T01:33:02.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:02.884+0000 7f97f17fa640 1 -- 172.21.15.28:0/1202673137 <== mon.2 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7f97e80986f0 con 0x7f97f4104b30 2024-04-25T01:33:03.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.042+0000 7f97faa41640 1 -- 172.21.15.28:0/1202673137 --> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f97f4061b80 con 0x7f97dc0a2990 2024-04-25T01:33:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.055+0000 7f97f17fa640 1 -- 172.21.15.28:0/1202673137 <== mgr.15561 v2:172.21.15.28:6800/712762371 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339435 (secure 0 0 0) 0x7f97f4061b80 con 0x7f97dc0a2990 2024-04-25T01:33:03.037 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:33:03.039 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T01:33:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.059+0000 7f97faa41640 1 -- 172.21.15.28:0/1202673137 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97dc0a2990 msgr2=0x7f97dc0a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.059+0000 7f97faa41640 1 --2- 172.21.15.28:0/1202673137 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97dc0a2990 0x7f97dc0a4e50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f97e4031440 tx=0x7f97e40315b0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.059+0000 7f97faa41640 1 -- 172.21.15.28:0/1202673137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97f4104b30 msgr2=0x7f97f4197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.059+0000 7f97faa41640 1 --2- 172.21.15.28:0/1202673137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97f4104b30 0x7f97f4197ff0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f97e800e9c0 tx=0x7f97e800ee80 comp rx=0 tx=0).stop 2024-04-25T01:33:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.059+0000 7f97faa41640 1 -- 172.21.15.28:0/1202673137 shutdown_connections 2024-04-25T01:33:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.059+0000 7f97faa41640 1 --2- 172.21.15.28:0/1202673137 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97dc0a2990 0x7f97dc0a4e50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.059+0000 7f97faa41640 1 --2- 172.21.15.28:0/1202673137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97f4107460 0x7f97f419f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.059+0000 7f97faa41640 1 --2- 172.21.15.28:0/1202673137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97f4104b30 0x7f97f4197ff0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.059+0000 7f97faa41640 1 --2- 172.21.15.28:0/1202673137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97f4102200 0x7f97f4197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.059+0000 7f97faa41640 1 -- 172.21.15.28:0/1202673137 >> 172.21.15.28:0/1202673137 conn(0x7f97f40fbbf0 msgr2=0x7f97f4102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.059+0000 7f97faa41640 1 -- 172.21.15.28:0/1202673137 shutdown_connections 2024-04-25T01:33:03.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.059+0000 7f97faa41640 1 -- 172.21.15.28:0/1202673137 wait complete. 2024-04-25T01:33:03.046 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:02 smithi028 ceph-mon[29390]: [25/Apr/2024:01:33:01] ENGINE Bus STARTING 2024-04-25T01:33:03.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:02 smithi028 ceph-mon[29390]: [25/Apr/2024:01:33:01] ENGINE Serving on https://172.21.15.28:7150 2024-04-25T01:33:03.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:02 smithi028 ceph-mon[29390]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:33:03.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:02 smithi028 ceph-mon[29390]: [25/Apr/2024:01:33:01] ENGINE Serving on http://172.21.15.28:8765 2024-04-25T01:33:03.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:02 smithi028 ceph-mon[29390]: [25/Apr/2024:01:33:01] ENGINE Bus STARTED 2024-04-25T01:33:03.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:02 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3955829508' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:33:03.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:02 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/53527557' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:33:03.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:02 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2174995530' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:33:03.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:02 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:03.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:02 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:03.048 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-04-25T01:33:02.635937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1191980,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1473,"num_read_kb":905,"num_write":556,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1180804,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":942,"ondisk_log_size":942,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":389848,"kb_used_data":15700,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124551976,"statfs":{"total":1151940427776,"available":1151541223424,"internally_reserved":0,"allocated":16076800,"data_stored":9241723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":382973242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.986097"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641539+0000","last_change":"2024-04-25T01:30:34.358339+0000","last_active":"2024-04-25T01:33:00.641539+0000","last_peered":"2024-04-25T01:33:00.641539+0000","last_clean":"2024-04-25T01:33:00.641539+0000","last_became_active":"2024-04-25T01:30:34.357884+0000","last_became_peered":"2024-04-25T01:30:34.357884+0000","last_unstale":"2024-04-25T01:33:00.641539+0000","last_undegraded":"2024-04-25T01:33:00.641539+0000","last_fullsized":"2024-04-25T01:33:00.641539+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:56.409080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643075+0000","last_change":"2024-04-25T01:30:34.359933+0000","last_active":"2024-04-25T01:33:00.643075+0000","last_peered":"2024-04-25T01:33:00.643075+0000","last_clean":"2024-04-25T01:33:00.643075+0000","last_became_active":"2024-04-25T01:30:34.359576+0000","last_became_peered":"2024-04-25T01:30:34.359576+0000","last_unstale":"2024-04-25T01:33:00.643075+0000","last_undegraded":"2024-04-25T01:33:00.643075+0000","last_fullsized":"2024-04-25T01:33:00.643075+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:48:47.764121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642789+0000","last_change":"2024-04-25T01:30:34.357772+0000","last_active":"2024-04-25T01:33:00.642789+0000","last_peered":"2024-04-25T01:33:00.642789+0000","last_clean":"2024-04-25T01:33:00.642789+0000","last_became_active":"2024-04-25T01:30:34.357434+0000","last_became_peered":"2024-04-25T01:30:34.357434+0000","last_unstale":"2024-04-25T01:33:00.642789+0000","last_undegraded":"2024-04-25T01:33:00.642789+0000","last_fullsized":"2024-04-25T01:33:00.642789+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:19.926180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641967+0000","last_change":"2024-04-25T01:30:34.351250+0000","last_active":"2024-04-25T01:33:00.641967+0000","last_peered":"2024-04-25T01:33:00.641967+0000","last_clean":"2024-04-25T01:33:00.641967+0000","last_became_active":"2024-04-25T01:30:34.349976+0000","last_became_peered":"2024-04-25T01:30:34.349976+0000","last_unstale":"2024-04-25T01:33:00.641967+0000","last_undegraded":"2024-04-25T01:33:00.641967+0000","last_fullsized":"2024-04-25T01:33:00.641967+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:19:51.605307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730612+0000","last_change":"2024-04-25T01:30:34.367882+0000","last_active":"2024-04-25T01:33:00.730612+0000","last_peered":"2024-04-25T01:33:00.730612+0000","last_clean":"2024-04-25T01:33:00.730612+0000","last_became_active":"2024-04-25T01:30:34.366797+0000","last_became_peered":"2024-04-25T01:30:34.366797+0000","last_unstale":"2024-04-25T01:33:00.730612+0000","last_undegraded":"2024-04-25T01:33:00.730612+0000","last_fullsized":"2024-04-25T01:33:00.730612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:35:23.439061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.623521+0000","last_change":"2024-04-25T01:30:34.352506+0000","last_active":"2024-04-25T01:33:00.623521+0000","last_peered":"2024-04-25T01:33:00.623521+0000","last_clean":"2024-04-25T01:33:00.623521+0000","last_became_active":"2024-04-25T01:30:34.352100+0000","last_became_peered":"2024-04-25T01:30:34.352100+0000","last_unstale":"2024-04-25T01:33:00.623521+0000","last_undegraded":"2024-04-25T01:33:00.623521+0000","last_fullsized":"2024-04-25T01:33:00.623521+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:38.113959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641641+0000","last_change":"2024-04-25T01:30:34.355352+0000","last_active":"2024-04-25T01:33:00.641641+0000","last_peered":"2024-04-25T01:33:00.641641+0000","last_clean":"2024-04-25T01:33:00.641641+0000","last_became_active":"2024-04-25T01:30:34.355008+0000","last_became_peered":"2024-04-25T01:30:34.355008+0000","last_unstale":"2024-04-25T01:33:00.641641+0000","last_undegraded":"2024-04-25T01:33:00.641641+0000","last_fullsized":"2024-04-25T01:33:00.641641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:23.582742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622074+0000","last_change":"2024-04-25T01:30:34.360644+0000","last_active":"2024-04-25T01:33:00.622074+0000","last_peered":"2024-04-25T01:33:00.622074+0000","last_clean":"2024-04-25T01:33:00.622074+0000","last_became_active":"2024-04-25T01:30:34.357841+0000","last_became_peered":"2024-04-25T01:30:34.357841+0000","last_unstale":"2024-04-25T01:33:00.622074+0000","last_undegraded":"2024-04-25T01:33:00.622074+0000","last_fullsized":"2024-04-25T01:33:00.622074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:26:21.606219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642196+0000","last_change":"2024-04-25T01:30:34.358417+0000","last_active":"2024-04-25T01:33:00.642196+0000","last_peered":"2024-04-25T01:33:00.642196+0000","last_clean":"2024-04-25T01:33:00.642196+0000","last_became_active":"2024-04-25T01:30:34.358039+0000","last_became_peered":"2024-04-25T01:30:34.358039+0000","last_unstale":"2024-04-25T01:33:00.642196+0000","last_undegraded":"2024-04-25T01:33:00.642196+0000","last_fullsized":"2024-04-25T01:33:00.642196+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:42.600693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.624086+0000","last_change":"2024-04-25T01:30:34.357803+0000","last_active":"2024-04-25T01:33:00.624086+0000","last_peered":"2024-04-25T01:33:00.624086+0000","last_clean":"2024-04-25T01:33:00.624086+0000","last_became_active":"2024-04-25T01:30:34.357391+0000","last_became_peered":"2024-04-25T01:30:34.357391+0000","last_unstale":"2024-04-25T01:33:00.624086+0000","last_undegraded":"2024-04-25T01:33:00.624086+0000","last_fullsized":"2024-04-25T01:33:00.624086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:57.857303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642471+0000","last_change":"2024-04-25T01:30:34.363340+0000","last_active":"2024-04-25T01:33:00.642471+0000","last_peered":"2024-04-25T01:33:00.642471+0000","last_clean":"2024-04-25T01:33:00.642471+0000","last_became_active":"2024-04-25T01:30:34.362980+0000","last_became_peered":"2024-04-25T01:30:34.362980+0000","last_unstale":"2024-04-25T01:33:00.642471+0000","last_undegraded":"2024-04-25T01:33:00.642471+0000","last_fullsized":"2024-04-25T01:33:00.642471+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:46.291459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641988+0000","last_change":"2024-04-25T01:30:34.358898+0000","last_active":"2024-04-25T01:33:00.641988+0000","last_peered":"2024-04-25T01:33:00.641988+0000","last_clean":"2024-04-25T01:33:00.641988+0000","last_became_active":"2024-04-25T01:30:34.358495+0000","last_became_peered":"2024-04-25T01:30:34.358495+0000","last_unstale":"2024-04-25T01:33:00.641988+0000","last_undegraded":"2024-04-25T01:33:00.641988+0000","last_fullsized":"2024-04-25T01:33:00.641988+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:34:21.860010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643535+0000","last_change":"2024-04-25T01:30:34.363708+0000","last_active":"2024-04-25T01:33:00.643535+0000","last_peered":"2024-04-25T01:33:00.643535+0000","last_clean":"2024-04-25T01:33:00.643535+0000","last_became_active":"2024-04-25T01:30:34.363207+0000","last_became_peered":"2024-04-25T01:30:34.363207+0000","last_unstale":"2024-04-25T01:33:00.643535+0000","last_undegraded":"2024-04-25T01:33:00.643535+0000","last_fullsized":"2024-04-25T01:33:00.643535+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:53.442987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642070+0000","last_change":"2024-04-25T01:30:34.371031+0000","last_active":"2024-04-25T01:33:00.642070+0000","last_peered":"2024-04-25T01:33:00.642070+0000","last_clean":"2024-04-25T01:33:00.642070+0000","last_became_active":"2024-04-25T01:30:34.370687+0000","last_became_peered":"2024-04-25T01:30:34.370687+0000","last_unstale":"2024-04-25T01:33:00.642070+0000","last_undegraded":"2024-04-25T01:33:00.642070+0000","last_fullsized":"2024-04-25T01:33:00.642070+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:03:20.700960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642487+0000","last_change":"2024-04-25T01:30:34.366212+0000","last_active":"2024-04-25T01:33:00.642487+0000","last_peered":"2024-04-25T01:33:00.642487+0000","last_clean":"2024-04-25T01:33:00.642487+0000","last_became_active":"2024-04-25T01:30:34.365849+0000","last_became_peered":"2024-04-25T01:30:34.365849+0000","last_unstale":"2024-04-25T01:33:00.642487+0000","last_undegraded":"2024-04-25T01:33:00.642487+0000","last_fullsized":"2024-04-25T01:33:00.642487+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:24:02.397615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640584+0000","last_change":"2024-04-25T01:30:34.367380+0000","last_active":"2024-04-25T01:33:00.640584+0000","last_peered":"2024-04-25T01:33:00.640584+0000","last_clean":"2024-04-25T01:33:00.640584+0000","last_became_active":"2024-04-25T01:30:34.366872+0000","last_became_peered":"2024-04-25T01:30:34.366872+0000","last_unstale":"2024-04-25T01:33:00.640584+0000","last_undegraded":"2024-04-25T01:33:00.640584+0000","last_fullsized":"2024-04-25T01:33:00.640584+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:18.767108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641445+0000","last_change":"2024-04-25T01:30:34.359752+0000","last_active":"2024-04-25T01:33:00.641445+0000","last_peered":"2024-04-25T01:33:00.641445+0000","last_clean":"2024-04-25T01:33:00.641445+0000","last_became_active":"2024-04-25T01:30:34.358403+0000","last_became_peered":"2024-04-25T01:30:34.358403+0000","last_unstale":"2024-04-25T01:33:00.641445+0000","last_undegraded":"2024-04-25T01:33:00.641445+0000","last_fullsized":"2024-04-25T01:33:00.641445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:14.833126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641209+0000","last_change":"2024-04-25T01:30:34.361373+0000","last_active":"2024-04-25T01:33:00.641209+0000","last_peered":"2024-04-25T01:33:00.641209+0000","last_clean":"2024-04-25T01:33:00.641209+0000","last_became_active":"2024-04-25T01:30:34.361046+0000","last_became_peered":"2024-04-25T01:30:34.361046+0000","last_unstale":"2024-04-25T01:33:00.641209+0000","last_undegraded":"2024-04-25T01:33:00.641209+0000","last_fullsized":"2024-04-25T01:33:00.641209+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:21.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.626292+0000","last_change":"2024-04-25T01:30:34.367975+0000","last_active":"2024-04-25T01:33:00.626292+0000","last_peered":"2024-04-25T01:33:00.626292+0000","last_clean":"2024-04-25T01:33:00.626292+0000","last_became_active":"2024-04-25T01:30:34.366383+0000","last_became_peered":"2024-04-25T01:30:34.366383+0000","last_unstale":"2024-04-25T01:33:00.626292+0000","last_undegraded":"2024-04-25T01:33:00.626292+0000","last_fullsized":"2024-04-25T01:33:00.626292+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:53:40.279067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.625811+0000","last_change":"2024-04-25T01:30:38.382168+0000","last_active":"2024-04-25T01:33:00.625811+0000","last_peered":"2024-04-25T01:33:00.625811+0000","last_clean":"2024-04-25T01:33:00.625811+0000","last_became_active":"2024-04-25T01:30:38.381740+0000","last_became_peered":"2024-04-25T01:30:38.381740+0000","last_unstale":"2024-04-25T01:33:00.625811+0000","last_undegraded":"2024-04-25T01:33:00.625811+0000","last_fullsized":"2024-04-25T01:33:00.625811+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:27.425991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'13","reported_seq":164,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641584+0000","last_change":"2024-04-25T01:30:33.345547+0000","last_active":"2024-04-25T01:33:00.641584+0000","last_peered":"2024-04-25T01:33:00.641584+0000","last_clean":"2024-04-25T01:33:00.641584+0000","last_became_active":"2024-04-25T01:30:33.345229+0000","last_became_peered":"2024-04-25T01:30:33.345229+0000","last_unstale":"2024-04-25T01:33:00.641584+0000","last_undegraded":"2024-04-25T01:33:00.641584+0000","last_fullsized":"2024-04-25T01:33:00.641584+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:32:46.637720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622343+0000","last_change":"2024-04-25T01:30:34.354703+0000","last_active":"2024-04-25T01:33:00.622343+0000","last_peered":"2024-04-25T01:33:00.622343+0000","last_clean":"2024-04-25T01:33:00.622343+0000","last_became_active":"2024-04-25T01:30:34.354362+0000","last_became_peered":"2024-04-25T01:30:34.354362+0000","last_unstale":"2024-04-25T01:33:00.622343+0000","last_undegraded":"2024-04-25T01:33:00.622343+0000","last_fullsized":"2024-04-25T01:33:00.622343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:51.358599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640543+0000","last_change":"2024-04-25T01:30:38.380912+0000","last_active":"2024-04-25T01:33:00.640543+0000","last_peered":"2024-04-25T01:33:00.640543+0000","last_clean":"2024-04-25T01:33:00.640543+0000","last_became_active":"2024-04-25T01:30:38.380439+0000","last_became_peered":"2024-04-25T01:30:38.380439+0000","last_unstale":"2024-04-25T01:33:00.640543+0000","last_undegraded":"2024-04-25T01:33:00.640543+0000","last_fullsized":"2024-04-25T01:33:00.640543+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:22:00.527268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622488+0000","last_change":"2024-04-25T01:30:33.345620+0000","last_active":"2024-04-25T01:33:00.622488+0000","last_peered":"2024-04-25T01:33:00.622488+0000","last_clean":"2024-04-25T01:33:00.622488+0000","last_became_active":"2024-04-25T01:30:33.345315+0000","last_became_peered":"2024-04-25T01:30:33.345315+0000","last_unstale":"2024-04-25T01:33:00.622488+0000","last_undegraded":"2024-04-25T01:33:00.622488+0000","last_fullsized":"2024-04-25T01:33:00.622488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:48:00.083728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643224+0000","last_change":"2024-04-25T01:30:34.356589+0000","last_active":"2024-04-25T01:33:00.643224+0000","last_peered":"2024-04-25T01:33:00.643224+0000","last_clean":"2024-04-25T01:33:00.643224+0000","last_became_active":"2024-04-25T01:30:34.356114+0000","last_became_peered":"2024-04-25T01:30:34.356114+0000","last_unstale":"2024-04-25T01:33:00.643224+0000","last_undegraded":"2024-04-25T01:33:00.643224+0000","last_fullsized":"2024-04-25T01:33:00.643224+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:39.039517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640748+0000","last_change":"2024-04-25T01:31:21.876578+0000","last_active":"2024-04-25T01:33:00.640748+0000","last_peered":"2024-04-25T01:33:00.640748+0000","last_clean":"2024-04-25T01:33:00.640748+0000","last_became_active":"2024-04-25T01:31:21.876126+0000","last_became_peered":"2024-04-25T01:31:21.876126+0000","last_unstale":"2024-04-25T01:33:00.640748+0000","last_undegraded":"2024-04-25T01:33:00.640748+0000","last_fullsized":"2024-04-25T01:33:00.640748+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:55.909353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'12","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730820+0000","last_change":"2024-04-25T01:30:33.345575+0000","last_active":"2024-04-25T01:33:00.730820+0000","last_peered":"2024-04-25T01:33:00.730820+0000","last_clean":"2024-04-25T01:33:00.730820+0000","last_became_active":"2024-04-25T01:30:33.345231+0000","last_became_peered":"2024-04-25T01:30:33.345231+0000","last_unstale":"2024-04-25T01:33:00.730820+0000","last_undegraded":"2024-04-25T01:33:00.730820+0000","last_fullsized":"2024-04-25T01:33:00.730820+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:04:21.090788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641074+0000","last_change":"2024-04-25T01:30:34.371317+0000","last_active":"2024-04-25T01:33:00.641074+0000","last_peered":"2024-04-25T01:33:00.641074+0000","last_clean":"2024-04-25T01:33:00.641074+0000","last_became_active":"2024-04-25T01:30:34.370926+0000","last_became_peered":"2024-04-25T01:30:34.370926+0000","last_unstale":"2024-04-25T01:33:00.641074+0000","last_undegraded":"2024-04-25T01:33:00.641074+0000","last_fullsized":"2024-04-25T01:33:00.641074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:58:39.898471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641235+0000","last_change":"2024-04-25T01:30:38.380982+0000","last_active":"2024-04-25T01:33:00.641235+0000","last_peered":"2024-04-25T01:33:00.641235+0000","last_clean":"2024-04-25T01:33:00.641235+0000","last_became_active":"2024-04-25T01:30:38.380510+0000","last_became_peered":"2024-04-25T01:30:38.380510+0000","last_unstale":"2024-04-25T01:33:00.641235+0000","last_undegraded":"2024-04-25T01:33:00.641235+0000","last_fullsized":"2024-04-25T01:33:00.641235+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:00.510697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"91'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730845+0000","last_change":"2024-04-25T01:30:33.351561+0000","last_active":"2024-04-25T01:33:00.730845+0000","last_peered":"2024-04-25T01:33:00.730845+0000","last_clean":"2024-04-25T01:33:00.730845+0000","last_became_active":"2024-04-25T01:30:33.351230+0000","last_became_peered":"2024-04-25T01:30:33.351230+0000","last_unstale":"2024-04-25T01:33:00.730845+0000","last_undegraded":"2024-04-25T01:33:00.730845+0000","last_fullsized":"2024-04-25T01:33:00.730845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:56.594457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641855+0000","last_change":"2024-04-25T01:30:34.358821+0000","last_active":"2024-04-25T01:33:00.641855+0000","last_peered":"2024-04-25T01:33:00.641855+0000","last_clean":"2024-04-25T01:33:00.641855+0000","last_became_active":"2024-04-25T01:30:34.358303+0000","last_became_peered":"2024-04-25T01:30:34.358303+0000","last_unstale":"2024-04-25T01:33:00.641855+0000","last_undegraded":"2024-04-25T01:33:00.641855+0000","last_fullsized":"2024-04-25T01:33:00.641855+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:26:23.352101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.626107+0000","last_change":"2024-04-25T01:30:38.372281+0000","last_active":"2024-04-25T01:33:00.626107+0000","last_peered":"2024-04-25T01:33:00.626107+0000","last_clean":"2024-04-25T01:33:00.626107+0000","last_became_active":"2024-04-25T01:30:38.371876+0000","last_became_peered":"2024-04-25T01:30:38.371876+0000","last_unstale":"2024-04-25T01:33:00.626107+0000","last_undegraded":"2024-04-25T01:33:00.626107+0000","last_fullsized":"2024-04-25T01:33:00.626107+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:54.254011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'7","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.980108+0000","last_change":"2024-04-25T01:30:33.347262+0000","last_active":"2024-04-25T01:33:00.980108+0000","last_peered":"2024-04-25T01:33:00.980108+0000","last_clean":"2024-04-25T01:33:00.980108+0000","last_became_active":"2024-04-25T01:30:33.346900+0000","last_became_peered":"2024-04-25T01:30:33.346900+0000","last_unstale":"2024-04-25T01:33:00.980108+0000","last_undegraded":"2024-04-25T01:33:00.980108+0000","last_fullsized":"2024-04-25T01:33:00.980108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:29:30.960456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730404+0000","last_change":"2024-04-25T01:30:34.364870+0000","last_active":"2024-04-25T01:33:00.730404+0000","last_peered":"2024-04-25T01:33:00.730404+0000","last_clean":"2024-04-25T01:33:00.730404+0000","last_became_active":"2024-04-25T01:30:34.364459+0000","last_became_peered":"2024-04-25T01:30:34.364459+0000","last_unstale":"2024-04-25T01:33:00.730404+0000","last_undegraded":"2024-04-25T01:33:00.730404+0000","last_fullsized":"2024-04-25T01:33:00.730404+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:42:45.311668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.626138+0000","last_change":"2024-04-25T01:30:38.382175+0000","last_active":"2024-04-25T01:33:00.626138+0000","last_peered":"2024-04-25T01:33:00.626138+0000","last_clean":"2024-04-25T01:33:00.626138+0000","last_became_active":"2024-04-25T01:30:38.381771+0000","last_became_peered":"2024-04-25T01:30:38.381771+0000","last_unstale":"2024-04-25T01:33:00.626138+0000","last_undegraded":"2024-04-25T01:33:00.626138+0000","last_fullsized":"2024-04-25T01:33:00.626138+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:25:31.041566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.623581+0000","last_change":"2024-04-25T01:30:33.343605+0000","last_active":"2024-04-25T01:33:00.623581+0000","last_peered":"2024-04-25T01:33:00.623581+0000","last_clean":"2024-04-25T01:33:00.623581+0000","last_became_active":"2024-04-25T01:30:33.343254+0000","last_became_peered":"2024-04-25T01:30:33.343254+0000","last_unstale":"2024-04-25T01:33:00.623581+0000","last_undegraded":"2024-04-25T01:33:00.623581+0000","last_fullsized":"2024-04-25T01:33:00.623581+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:18:16.156285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640694+0000","last_change":"2024-04-25T01:31:21.874202+0000","last_active":"2024-04-25T01:33:00.640694+0000","last_peered":"2024-04-25T01:33:00.640694+0000","last_clean":"2024-04-25T01:33:00.640694+0000","last_became_active":"2024-04-25T01:31:21.873839+0000","last_became_peered":"2024-04-25T01:31:21.873839+0000","last_unstale":"2024-04-25T01:33:00.640694+0000","last_undegraded":"2024-04-25T01:33:00.640694+0000","last_fullsized":"2024-04-25T01:33:00.640694+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:39.774281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641353+0000","last_change":"2024-04-25T01:31:21.875259+0000","last_active":"2024-04-25T01:33:00.641353+0000","last_peered":"2024-04-25T01:33:00.641353+0000","last_clean":"2024-04-25T01:33:00.641353+0000","last_became_active":"2024-04-25T01:31:21.874800+0000","last_became_peered":"2024-04-25T01:31:21.874800+0000","last_unstale":"2024-04-25T01:33:00.641353+0000","last_undegraded":"2024-04-25T01:33:00.641353+0000","last_fullsized":"2024-04-25T01:33:00.641353+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:44.828620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,10,0],"acting":[11,4,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'3","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642169+0000","last_change":"2024-04-25T01:30:33.351621+0000","last_active":"2024-04-25T01:33:00.642169+0000","last_peered":"2024-04-25T01:33:00.642169+0000","last_clean":"2024-04-25T01:33:00.642169+0000","last_became_active":"2024-04-25T01:30:33.351289+0000","last_became_peered":"2024-04-25T01:30:33.351289+0000","last_unstale":"2024-04-25T01:33:00.642169+0000","last_undegraded":"2024-04-25T01:33:00.642169+0000","last_fullsized":"2024-04-25T01:33:00.642169+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:46.044826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622260+0000","last_change":"2024-04-25T01:30:34.350685+0000","last_active":"2024-04-25T01:33:00.622260+0000","last_peered":"2024-04-25T01:33:00.622260+0000","last_clean":"2024-04-25T01:33:00.622260+0000","last_became_active":"2024-04-25T01:30:34.350308+0000","last_became_peered":"2024-04-25T01:30:34.350308+0000","last_unstale":"2024-04-25T01:33:00.622260+0000","last_undegraded":"2024-04-25T01:33:00.622260+0000","last_fullsized":"2024-04-25T01:33:00.622260+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:11.690381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643016+0000","last_change":"2024-04-25T01:30:38.377403+0000","last_active":"2024-04-25T01:33:00.643016+0000","last_peered":"2024-04-25T01:33:00.643016+0000","last_clean":"2024-04-25T01:33:00.643016+0000","last_became_active":"2024-04-25T01:30:38.376966+0000","last_became_peered":"2024-04-25T01:30:38.376966+0000","last_unstale":"2024-04-25T01:33:00.643016+0000","last_undegraded":"2024-04-25T01:33:00.643016+0000","last_fullsized":"2024-04-25T01:33:00.643016+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:31:10.229443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'3","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641337+0000","last_change":"2024-04-25T01:30:33.348455+0000","last_active":"2024-04-25T01:33:00.641337+0000","last_peered":"2024-04-25T01:33:00.641337+0000","last_clean":"2024-04-25T01:33:00.641337+0000","last_became_active":"2024-04-25T01:30:33.348142+0000","last_became_peered":"2024-04-25T01:30:33.348142+0000","last_unstale":"2024-04-25T01:33:00.641337+0000","last_undegraded":"2024-04-25T01:33:00.641337+0000","last_fullsized":"2024-04-25T01:33:00.641337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:43:07.695580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640378+0000","last_change":"2024-04-25T01:30:34.359949+0000","last_active":"2024-04-25T01:33:00.640378+0000","last_peered":"2024-04-25T01:33:00.640378+0000","last_clean":"2024-04-25T01:33:00.640378+0000","last_became_active":"2024-04-25T01:30:34.359609+0000","last_became_peered":"2024-04-25T01:30:34.359609+0000","last_unstale":"2024-04-25T01:33:00.640378+0000","last_undegraded":"2024-04-25T01:33:00.640378+0000","last_fullsized":"2024-04-25T01:33:00.640378+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:49.668800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622175+0000","last_change":"2024-04-25T01:30:38.380260+0000","last_active":"2024-04-25T01:33:00.622175+0000","last_peered":"2024-04-25T01:33:00.622175+0000","last_clean":"2024-04-25T01:33:00.622175+0000","last_became_active":"2024-04-25T01:30:38.379624+0000","last_became_peered":"2024-04-25T01:30:38.379624+0000","last_unstale":"2024-04-25T01:33:00.622175+0000","last_undegraded":"2024-04-25T01:33:00.622175+0000","last_fullsized":"2024-04-25T01:33:00.622175+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:26:35.235335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'6","reported_seq":276,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.979614+0000","last_change":"2024-04-25T01:30:33.349490+0000","last_active":"2024-04-25T01:33:00.979614+0000","last_peered":"2024-04-25T01:33:00.979614+0000","last_clean":"2024-04-25T01:33:00.979614+0000","last_became_active":"2024-04-25T01:30:33.349136+0000","last_became_peered":"2024-04-25T01:30:33.349136+0000","last_unstale":"2024-04-25T01:33:00.979614+0000","last_undegraded":"2024-04-25T01:33:00.979614+0000","last_fullsized":"2024-04-25T01:33:00.979614+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:16:23.552073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":240,"num_read_kb":240,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641812+0000","last_change":"2024-04-25T01:30:34.372701+0000","last_active":"2024-04-25T01:33:00.641812+0000","last_peered":"2024-04-25T01:33:00.641812+0000","last_clean":"2024-04-25T01:33:00.641812+0000","last_became_active":"2024-04-25T01:30:34.372321+0000","last_became_peered":"2024-04-25T01:30:34.372321+0000","last_unstale":"2024-04-25T01:33:00.641812+0000","last_undegraded":"2024-04-25T01:33:00.641812+0000","last_fullsized":"2024-04-25T01:33:00.641812+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:02.025015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640169+0000","last_change":"2024-04-25T01:30:38.378036+0000","last_active":"2024-04-25T01:33:00.640169+0000","last_peered":"2024-04-25T01:33:00.640169+0000","last_clean":"2024-04-25T01:33:00.640169+0000","last_became_active":"2024-04-25T01:30:38.377602+0000","last_became_peered":"2024-04-25T01:30:38.377602+0000","last_unstale":"2024-04-25T01:33:00.640169+0000","last_undegraded":"2024-04-25T01:33:00.640169+0000","last_fullsized":"2024-04-25T01:33:00.640169+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:45.210518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641468+0000","last_change":"2024-04-25T01:30:33.352928+0000","last_active":"2024-04-25T01:33:00.641468+0000","last_peered":"2024-04-25T01:33:00.641468+0000","last_clean":"2024-04-25T01:33:00.641468+0000","last_became_active":"2024-04-25T01:30:33.352567+0000","last_became_peered":"2024-04-25T01:30:33.352567+0000","last_unstale":"2024-04-25T01:33:00.641468+0000","last_undegraded":"2024-04-25T01:33:00.641468+0000","last_fullsized":"2024-04-25T01:33:00.641468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:47:21.941179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643449+0000","last_change":"2024-04-25T01:30:34.366947+0000","last_active":"2024-04-25T01:33:00.643449+0000","last_peered":"2024-04-25T01:33:00.643449+0000","last_clean":"2024-04-25T01:33:00.643449+0000","last_became_active":"2024-04-25T01:30:34.366337+0000","last_became_peered":"2024-04-25T01:30:34.366337+0000","last_unstale":"2024-04-25T01:33:00.643449+0000","last_undegraded":"2024-04-25T01:33:00.643449+0000","last_fullsized":"2024-04-25T01:33:00.643449+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:48:43.528380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641593+0000","last_change":"2024-04-25T01:31:22.050799+0000","last_active":"2024-04-25T01:33:00.641593+0000","last_peered":"2024-04-25T01:33:00.641593+0000","last_clean":"2024-04-25T01:33:00.641593+0000","last_became_active":"2024-04-25T01:31:22.050130+0000","last_became_peered":"2024-04-25T01:31:22.050130+0000","last_unstale":"2024-04-25T01:33:00.641593+0000","last_undegraded":"2024-04-25T01:33:00.641593+0000","last_fullsized":"2024-04-25T01:33:00.641593+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:40:43.549713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641872+0000","last_change":"2024-04-25T01:30:33.346003+0000","last_active":"2024-04-25T01:33:00.641872+0000","last_peered":"2024-04-25T01:33:00.641872+0000","last_clean":"2024-04-25T01:33:00.641872+0000","last_became_active":"2024-04-25T01:30:33.345254+0000","last_became_peered":"2024-04-25T01:30:33.345254+0000","last_unstale":"2024-04-25T01:33:00.641872+0000","last_undegraded":"2024-04-25T01:33:00.641872+0000","last_fullsized":"2024-04-25T01:33:00.641872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:18:37.957047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641299+0000","last_change":"2024-04-25T01:30:34.366108+0000","last_active":"2024-04-25T01:33:00.641299+0000","last_peered":"2024-04-25T01:33:00.641299+0000","last_clean":"2024-04-25T01:33:00.641299+0000","last_became_active":"2024-04-25T01:30:34.365702+0000","last_became_peered":"2024-04-25T01:30:34.365702+0000","last_unstale":"2024-04-25T01:33:00.641299+0000","last_undegraded":"2024-04-25T01:33:00.641299+0000","last_fullsized":"2024-04-25T01:33:00.641299+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:24:06.802424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640332+0000","last_change":"2024-04-25T01:30:38.376342+0000","last_active":"2024-04-25T01:33:00.640332+0000","last_peered":"2024-04-25T01:33:00.640332+0000","last_clean":"2024-04-25T01:33:00.640332+0000","last_became_active":"2024-04-25T01:30:38.375905+0000","last_became_peered":"2024-04-25T01:30:38.375905+0000","last_unstale":"2024-04-25T01:33:00.640332+0000","last_undegraded":"2024-04-25T01:33:00.640332+0000","last_fullsized":"2024-04-25T01:33:00.640332+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:31.615356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'44","reported_seq":325,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.980291+0000","last_change":"2024-04-25T01:30:33.350770+0000","last_active":"2024-04-25T01:33:00.980291+0000","last_peered":"2024-04-25T01:33:00.980291+0000","last_clean":"2024-04-25T01:33:00.980291+0000","last_became_active":"2024-04-25T01:30:33.350414+0000","last_became_peered":"2024-04-25T01:30:33.350414+0000","last_unstale":"2024-04-25T01:33:00.980291+0000","last_undegraded":"2024-04-25T01:33:00.980291+0000","last_fullsized":"2024-04-25T01:33:00.980291+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:01.771182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67986,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":251,"num_read_kb":193,"num_write":46,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730709+0000","last_change":"2024-04-25T01:30:34.362532+0000","last_active":"2024-04-25T01:33:00.730709+0000","last_peered":"2024-04-25T01:33:00.730709+0000","last_clean":"2024-04-25T01:33:00.730709+0000","last_became_active":"2024-04-25T01:30:34.362137+0000","last_became_peered":"2024-04-25T01:30:34.362137+0000","last_unstale":"2024-04-25T01:33:00.730709+0000","last_undegraded":"2024-04-25T01:33:00.730709+0000","last_fullsized":"2024-04-25T01:33:00.730709+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:28.105250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642169+0000","last_change":"2024-04-25T01:30:38.374591+0000","last_active":"2024-04-25T01:33:00.642169+0000","last_peered":"2024-04-25T01:33:00.642169+0000","last_clean":"2024-04-25T01:33:00.642169+0000","last_became_active":"2024-04-25T01:30:38.373237+0000","last_became_peered":"2024-04-25T01:30:38.373237+0000","last_unstale":"2024-04-25T01:33:00.642169+0000","last_undegraded":"2024-04-25T01:33:00.642169+0000","last_fullsized":"2024-04-25T01:33:00.642169+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:21:25.937405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'9","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640904+0000","last_change":"2024-04-25T01:31:21.885341+0000","last_active":"2024-04-25T01:33:00.640904+0000","last_peered":"2024-04-25T01:33:00.640904+0000","last_clean":"2024-04-25T01:33:00.640904+0000","last_became_active":"2024-04-25T01:31:21.874312+0000","last_became_peered":"2024-04-25T01:31:21.874312+0000","last_unstale":"2024-04-25T01:33:00.640904+0000","last_undegraded":"2024-04-25T01:33:00.640904+0000","last_fullsized":"2024-04-25T01:33:00.640904+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:23:55.388089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642553+0000","last_change":"2024-04-25T01:30:34.354444+0000","last_active":"2024-04-25T01:33:00.642553+0000","last_peered":"2024-04-25T01:33:00.642553+0000","last_clean":"2024-04-25T01:33:00.642553+0000","last_became_active":"2024-04-25T01:30:34.354070+0000","last_became_peered":"2024-04-25T01:30:34.354070+0000","last_unstale":"2024-04-25T01:33:00.642553+0000","last_undegraded":"2024-04-25T01:33:00.642553+0000","last_fullsized":"2024-04-25T01:33:00.642553+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:39.893995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640956+0000","last_change":"2024-04-25T01:30:38.373570+0000","last_active":"2024-04-25T01:33:00.640956+0000","last_peered":"2024-04-25T01:33:00.640956+0000","last_clean":"2024-04-25T01:33:00.640956+0000","last_became_active":"2024-04-25T01:30:38.373144+0000","last_became_peered":"2024-04-25T01:30:38.373144+0000","last_unstale":"2024-04-25T01:33:00.640956+0000","last_undegraded":"2024-04-25T01:33:00.640956+0000","last_fullsized":"2024-04-25T01:33:00.640956+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:53:32.933998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641434+0000","last_change":"2024-04-25T01:30:33.346145+0000","last_active":"2024-04-25T01:33:00.641434+0000","last_peered":"2024-04-25T01:33:00.641434+0000","last_clean":"2024-04-25T01:33:00.641434+0000","last_became_active":"2024-04-25T01:30:33.344294+0000","last_became_peered":"2024-04-25T01:30:33.344294+0000","last_unstale":"2024-04-25T01:33:00.641434+0000","last_undegraded":"2024-04-25T01:33:00.641434+0000","last_fullsized":"2024-04-25T01:33:00.641434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:37:17.147148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641769+0000","last_change":"2024-04-25T01:30:34.360769+0000","last_active":"2024-04-25T01:33:00.641769+0000","last_peered":"2024-04-25T01:33:00.641769+0000","last_clean":"2024-04-25T01:33:00.641769+0000","last_became_active":"2024-04-25T01:30:34.360076+0000","last_became_peered":"2024-04-25T01:30:34.360076+0000","last_unstale":"2024-04-25T01:33:00.641769+0000","last_undegraded":"2024-04-25T01:33:00.641769+0000","last_fullsized":"2024-04-25T01:33:00.641769+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:56:21.515511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640926+0000","last_change":"2024-04-25T01:30:38.379198+0000","last_active":"2024-04-25T01:33:00.640926+0000","last_peered":"2024-04-25T01:33:00.640926+0000","last_clean":"2024-04-25T01:33:00.640926+0000","last_became_active":"2024-04-25T01:30:38.378798+0000","last_became_peered":"2024-04-25T01:30:38.378798+0000","last_unstale":"2024-04-25T01:33:00.640926+0000","last_undegraded":"2024-04-25T01:33:00.640926+0000","last_fullsized":"2024-04-25T01:33:00.640926+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:40.192239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'8","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.972284+0000","last_change":"2024-04-25T01:30:33.343232+0000","last_active":"2024-04-25T01:33:00.972284+0000","last_peered":"2024-04-25T01:33:00.972284+0000","last_clean":"2024-04-25T01:33:00.972284+0000","last_became_active":"2024-04-25T01:30:33.342876+0000","last_became_peered":"2024-04-25T01:30:33.342876+0000","last_unstale":"2024-04-25T01:33:00.972284+0000","last_undegraded":"2024-04-25T01:33:00.972284+0000","last_fullsized":"2024-04-25T01:33:00.972284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:45.032564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1678,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":125,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641248+0000","last_change":"2024-04-25T01:30:34.358210+0000","last_active":"2024-04-25T01:33:00.641248+0000","last_peered":"2024-04-25T01:33:00.641248+0000","last_clean":"2024-04-25T01:33:00.641248+0000","last_became_active":"2024-04-25T01:30:34.357862+0000","last_became_peered":"2024-04-25T01:30:34.357862+0000","last_unstale":"2024-04-25T01:33:00.641248+0000","last_undegraded":"2024-04-25T01:33:00.641248+0000","last_fullsized":"2024-04-25T01:33:00.641248+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:43.751361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643538+0000","last_change":"2024-04-25T01:30:38.380413+0000","last_active":"2024-04-25T01:33:00.643538+0000","last_peered":"2024-04-25T01:33:00.643538+0000","last_clean":"2024-04-25T01:33:00.643538+0000","last_became_active":"2024-04-25T01:30:38.379612+0000","last_became_peered":"2024-04-25T01:30:38.379612+0000","last_unstale":"2024-04-25T01:33:00.643538+0000","last_undegraded":"2024-04-25T01:33:00.643538+0000","last_fullsized":"2024-04-25T01:33:00.643538+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:45.328642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'17","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730348+0000","last_change":"2024-04-25T01:30:33.344938+0000","last_active":"2024-04-25T01:33:00.730348+0000","last_peered":"2024-04-25T01:33:00.730348+0000","last_clean":"2024-04-25T01:33:00.730348+0000","last_became_active":"2024-04-25T01:30:33.344582+0000","last_became_peered":"2024-04-25T01:30:33.344582+0000","last_unstale":"2024-04-25T01:33:00.730348+0000","last_undegraded":"2024-04-25T01:33:00.730348+0000","last_fullsized":"2024-04-25T01:33:00.730348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:43:33.777319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641031+0000","last_change":"2024-04-25T01:30:34.354115+0000","last_active":"2024-04-25T01:33:00.641031+0000","last_peered":"2024-04-25T01:33:00.641031+0000","last_clean":"2024-04-25T01:33:00.641031+0000","last_became_active":"2024-04-25T01:30:34.353722+0000","last_became_peered":"2024-04-25T01:30:34.353722+0000","last_unstale":"2024-04-25T01:33:00.641031+0000","last_undegraded":"2024-04-25T01:33:00.641031+0000","last_fullsized":"2024-04-25T01:33:00.641031+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:22.003521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641136+0000","last_change":"2024-04-25T01:30:38.380914+0000","last_active":"2024-04-25T01:33:00.641136+0000","last_peered":"2024-04-25T01:33:00.641136+0000","last_clean":"2024-04-25T01:33:00.641136+0000","last_became_active":"2024-04-25T01:30:38.380477+0000","last_became_peered":"2024-04-25T01:30:38.380477+0000","last_unstale":"2024-04-25T01:33:00.641136+0000","last_undegraded":"2024-04-25T01:33:00.641136+0000","last_fullsized":"2024-04-25T01:33:00.641136+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:23:27.882266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730890+0000","last_change":"2024-04-25T01:30:33.345772+0000","last_active":"2024-04-25T01:33:00.730890+0000","last_peered":"2024-04-25T01:33:00.730890+0000","last_clean":"2024-04-25T01:33:00.730890+0000","last_became_active":"2024-04-25T01:30:33.345455+0000","last_became_peered":"2024-04-25T01:30:33.345455+0000","last_unstale":"2024-04-25T01:33:00.730890+0000","last_undegraded":"2024-04-25T01:33:00.730890+0000","last_fullsized":"2024-04-25T01:33:00.730890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:14:57.465161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641257+0000","last_change":"2024-04-25T01:30:34.358665+0000","last_active":"2024-04-25T01:33:00.641257+0000","last_peered":"2024-04-25T01:33:00.641257+0000","last_clean":"2024-04-25T01:33:00.641257+0000","last_became_active":"2024-04-25T01:30:34.358284+0000","last_became_peered":"2024-04-25T01:30:34.358284+0000","last_unstale":"2024-04-25T01:33:00.641257+0000","last_undegraded":"2024-04-25T01:33:00.641257+0000","last_fullsized":"2024-04-25T01:33:00.641257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:12:22.832265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730394+0000","last_change":"2024-04-25T01:30:38.450027+0000","last_active":"2024-04-25T01:33:00.730394+0000","last_peered":"2024-04-25T01:33:00.730394+0000","last_clean":"2024-04-25T01:33:00.730394+0000","last_became_active":"2024-04-25T01:30:38.449564+0000","last_became_peered":"2024-04-25T01:30:38.449564+0000","last_unstale":"2024-04-25T01:33:00.730394+0000","last_undegraded":"2024-04-25T01:33:00.730394+0000","last_fullsized":"2024-04-25T01:33:00.730394+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:34:26.542351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"91'1","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641180+0000","last_change":"2024-04-25T01:30:33.349757+0000","last_active":"2024-04-25T01:33:00.641180+0000","last_peered":"2024-04-25T01:33:00.641180+0000","last_clean":"2024-04-25T01:33:00.641180+0000","last_became_active":"2024-04-25T01:30:33.349439+0000","last_became_peered":"2024-04-25T01:30:33.349439+0000","last_unstale":"2024-04-25T01:33:00.641180+0000","last_undegraded":"2024-04-25T01:33:00.641180+0000","last_fullsized":"2024-04-25T01:33:00.641180+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:35.514160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.623885+0000","last_change":"2024-04-25T01:30:34.367310+0000","last_active":"2024-04-25T01:33:00.623885+0000","last_peered":"2024-04-25T01:33:00.623885+0000","last_clean":"2024-04-25T01:33:00.623885+0000","last_became_active":"2024-04-25T01:30:34.366950+0000","last_became_peered":"2024-04-25T01:30:34.366950+0000","last_unstale":"2024-04-25T01:33:00.623885+0000","last_undegraded":"2024-04-25T01:33:00.623885+0000","last_fullsized":"2024-04-25T01:33:00.623885+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:26:17.700512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642919+0000","last_change":"2024-04-25T01:30:38.372259+0000","last_active":"2024-04-25T01:33:00.642919+0000","last_peered":"2024-04-25T01:33:00.642919+0000","last_clean":"2024-04-25T01:33:00.642919+0000","last_became_active":"2024-04-25T01:30:38.371847+0000","last_became_peered":"2024-04-25T01:30:38.371847+0000","last_unstale":"2024-04-25T01:33:00.642919+0000","last_undegraded":"2024-04-25T01:33:00.642919+0000","last_fullsized":"2024-04-25T01:33:00.642919+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:55.047739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'5","reported_seq":156,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642256+0000","last_change":"2024-04-25T01:30:33.352078+0000","last_active":"2024-04-25T01:33:00.642256+0000","last_peered":"2024-04-25T01:33:00.642256+0000","last_clean":"2024-04-25T01:33:00.642256+0000","last_became_active":"2024-04-25T01:30:33.351725+0000","last_became_peered":"2024-04-25T01:30:33.351725+0000","last_unstale":"2024-04-25T01:33:00.642256+0000","last_undegraded":"2024-04-25T01:33:00.642256+0000","last_fullsized":"2024-04-25T01:33:00.642256+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:55:55.236040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.623632+0000","last_change":"2024-04-25T01:30:34.360324+0000","last_active":"2024-04-25T01:33:00.623632+0000","last_peered":"2024-04-25T01:33:00.623632+0000","last_clean":"2024-04-25T01:33:00.623632+0000","last_became_active":"2024-04-25T01:30:34.359114+0000","last_became_peered":"2024-04-25T01:30:34.359114+0000","last_unstale":"2024-04-25T01:33:00.623632+0000","last_undegraded":"2024-04-25T01:33:00.623632+0000","last_fullsized":"2024-04-25T01:33:00.623632+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:51.927640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643419+0000","last_change":"2024-04-25T01:30:38.450650+0000","last_active":"2024-04-25T01:33:00.643419+0000","last_peered":"2024-04-25T01:33:00.643419+0000","last_clean":"2024-04-25T01:33:00.643419+0000","last_became_active":"2024-04-25T01:30:38.450149+0000","last_became_peered":"2024-04-25T01:30:38.450149+0000","last_unstale":"2024-04-25T01:33:00.643419+0000","last_undegraded":"2024-04-25T01:33:00.643419+0000","last_fullsized":"2024-04-25T01:33:00.643419+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:34.630803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641415+0000","last_change":"2024-04-25T01:30:34.352961+0000","last_active":"2024-04-25T01:33:00.641415+0000","last_peered":"2024-04-25T01:33:00.641415+0000","last_clean":"2024-04-25T01:33:00.641415+0000","last_became_active":"2024-04-25T01:30:34.352657+0000","last_became_peered":"2024-04-25T01:30:34.352657+0000","last_unstale":"2024-04-25T01:33:00.641415+0000","last_undegraded":"2024-04-25T01:33:00.641415+0000","last_fullsized":"2024-04-25T01:33:00.641415+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:20.934228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":157,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641398+0000","last_change":"2024-04-25T01:30:33.344562+0000","last_active":"2024-04-25T01:33:00.641398+0000","last_peered":"2024-04-25T01:33:00.641398+0000","last_clean":"2024-04-25T01:33:00.641398+0000","last_became_active":"2024-04-25T01:30:33.344212+0000","last_became_peered":"2024-04-25T01:30:33.344212+0000","last_unstale":"2024-04-25T01:33:00.641398+0000","last_undegraded":"2024-04-25T01:33:00.641398+0000","last_fullsized":"2024-04-25T01:33:00.641398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:20:51.110618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641120+0000","last_change":"2024-04-25T01:30:38.384316+0000","last_active":"2024-04-25T01:33:00.641120+0000","last_peered":"2024-04-25T01:33:00.641120+0000","last_clean":"2024-04-25T01:33:00.641120+0000","last_became_active":"2024-04-25T01:30:38.383775+0000","last_became_peered":"2024-04-25T01:30:38.383775+0000","last_unstale":"2024-04-25T01:33:00.641120+0000","last_undegraded":"2024-04-25T01:33:00.641120+0000","last_fullsized":"2024-04-25T01:33:00.641120+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:54.165691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640600+0000","last_change":"2024-04-25T01:31:22.050014+0000","last_active":"2024-04-25T01:33:00.640600+0000","last_peered":"2024-04-25T01:33:00.640600+0000","last_clean":"2024-04-25T01:33:00.640600+0000","last_became_active":"2024-04-25T01:31:22.049718+0000","last_became_peered":"2024-04-25T01:31:22.049718+0000","last_unstale":"2024-04-25T01:33:00.640600+0000","last_undegraded":"2024-04-25T01:33:00.640600+0000","last_fullsized":"2024-04-25T01:33:00.640600+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:40:22.233372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"91'6","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640866+0000","last_change":"2024-04-25T01:30:33.345283+0000","last_active":"2024-04-25T01:33:00.640866+0000","last_peered":"2024-04-25T01:33:00.640866+0000","last_clean":"2024-04-25T01:33:00.640866+0000","last_became_active":"2024-04-25T01:30:33.344903+0000","last_became_peered":"2024-04-25T01:30:33.344903+0000","last_unstale":"2024-04-25T01:33:00.640866+0000","last_undegraded":"2024-04-25T01:33:00.640866+0000","last_fullsized":"2024-04-25T01:33:00.640866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:57:01.982850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":366,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640929+0000","last_change":"2024-04-25T01:29:17.689639+0000","last_active":"2024-04-25T01:33:00.640929+0000","last_peered":"2024-04-25T01:33:00.640929+0000","last_clean":"2024-04-25T01:33:00.640929+0000","last_became_active":"2024-04-25T01:29:17.689046+0000","last_became_peered":"2024-04-25T01:29:17.689046+0000","last_unstale":"2024-04-25T01:33:00.640929+0000","last_undegraded":"2024-04-25T01:33:00.640929+0000","last_fullsized":"2024-04-25T01:33:00.640929+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:48.767945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:48.767945+0000","last_clean_scrub_stamp":"2024-04-25T01:25:48.767945+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:04:06.092632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641924+0000","last_change":"2024-04-25T01:30:38.450938+0000","last_active":"2024-04-25T01:33:00.641924+0000","last_peered":"2024-04-25T01:33:00.641924+0000","last_clean":"2024-04-25T01:33:00.641924+0000","last_became_active":"2024-04-25T01:30:38.450488+0000","last_became_peered":"2024-04-25T01:30:38.450488+0000","last_unstale":"2024-04-25T01:33:00.641924+0000","last_undegraded":"2024-04-25T01:33:00.641924+0000","last_fullsized":"2024-04-25T01:33:00.641924+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:44.778133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641178+0000","last_change":"2024-04-25T01:31:21.875719+0000","last_active":"2024-04-25T01:33:00.641178+0000","last_peered":"2024-04-25T01:33:00.641178+0000","last_clean":"2024-04-25T01:33:00.641178+0000","last_became_active":"2024-04-25T01:31:21.875283+0000","last_became_peered":"2024-04-25T01:31:21.875283+0000","last_unstale":"2024-04-25T01:33:00.641178+0000","last_undegraded":"2024-04-25T01:33:00.641178+0000","last_fullsized":"2024-04-25T01:33:00.641178+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:12:39.097019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641047+0000","last_change":"2024-04-25T01:30:33.348198+0000","last_active":"2024-04-25T01:33:00.641047+0000","last_peered":"2024-04-25T01:33:00.641047+0000","last_clean":"2024-04-25T01:33:00.641047+0000","last_became_active":"2024-04-25T01:30:33.347876+0000","last_became_peered":"2024-04-25T01:30:33.347876+0000","last_unstale":"2024-04-25T01:33:00.641047+0000","last_undegraded":"2024-04-25T01:33:00.641047+0000","last_fullsized":"2024-04-25T01:33:00.641047+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:48:33.116571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641704+0000","last_change":"2024-04-25T01:30:38.372331+0000","last_active":"2024-04-25T01:33:00.641704+0000","last_peered":"2024-04-25T01:33:00.641704+0000","last_clean":"2024-04-25T01:33:00.641704+0000","last_became_active":"2024-04-25T01:30:38.371923+0000","last_became_peered":"2024-04-25T01:30:38.371923+0000","last_unstale":"2024-04-25T01:33:00.641704+0000","last_undegraded":"2024-04-25T01:33:00.641704+0000","last_fullsized":"2024-04-25T01:33:00.641704+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:06.416967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730656+0000","last_change":"2024-04-25T01:30:33.346606+0000","last_active":"2024-04-25T01:33:00.730656+0000","last_peered":"2024-04-25T01:33:00.730656+0000","last_clean":"2024-04-25T01:33:00.730656+0000","last_became_active":"2024-04-25T01:30:33.346297+0000","last_became_peered":"2024-04-25T01:30:33.346297+0000","last_unstale":"2024-04-25T01:33:00.730656+0000","last_undegraded":"2024-04-25T01:33:00.730656+0000","last_fullsized":"2024-04-25T01:33:00.730656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:07:16.069285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643021+0000","last_change":"2024-04-25T01:30:34.349136+0000","last_active":"2024-04-25T01:33:00.643021+0000","last_peered":"2024-04-25T01:33:00.643021+0000","last_clean":"2024-04-25T01:33:00.643021+0000","last_became_active":"2024-04-25T01:30:34.348810+0000","last_became_peered":"2024-04-25T01:30:34.348810+0000","last_unstale":"2024-04-25T01:33:00.643021+0000","last_undegraded":"2024-04-25T01:33:00.643021+0000","last_fullsized":"2024-04-25T01:33:00.643021+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:39:34.593986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.621999+0000","last_change":"2024-04-25T01:30:38.450428+0000","last_active":"2024-04-25T01:33:00.621999+0000","last_peered":"2024-04-25T01:33:00.621999+0000","last_clean":"2024-04-25T01:33:00.621999+0000","last_became_active":"2024-04-25T01:30:38.449971+0000","last_became_peered":"2024-04-25T01:30:38.449971+0000","last_unstale":"2024-04-25T01:33:00.621999+0000","last_undegraded":"2024-04-25T01:33:00.621999+0000","last_fullsized":"2024-04-25T01:33:00.621999+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:52:37.046462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'47","reported_seq":100,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.847942+0000","last_change":"2024-04-25T01:30:33.348547+0000","last_active":"2024-04-25T01:33:00.847942+0000","last_peered":"2024-04-25T01:33:00.847942+0000","last_clean":"2024-04-25T01:33:00.847942+0000","last_became_active":"2024-04-25T01:30:33.348181+0000","last_became_peered":"2024-04-25T01:30:33.348181+0000","last_unstale":"2024-04-25T01:33:00.847942+0000","last_undegraded":"2024-04-25T01:33:00.847942+0000","last_fullsized":"2024-04-25T01:33:00.847942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:11:58.696829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":82,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622499+0000","last_change":"2024-04-25T01:30:34.351987+0000","last_active":"2024-04-25T01:33:00.622499+0000","last_peered":"2024-04-25T01:33:00.622499+0000","last_clean":"2024-04-25T01:33:00.622499+0000","last_became_active":"2024-04-25T01:30:34.351571+0000","last_became_peered":"2024-04-25T01:30:34.351571+0000","last_unstale":"2024-04-25T01:33:00.622499+0000","last_undegraded":"2024-04-25T01:33:00.622499+0000","last_fullsized":"2024-04-25T01:33:00.622499+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:23.585865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.625691+0000","last_change":"2024-04-25T01:30:38.372566+0000","last_active":"2024-04-25T01:33:00.625691+0000","last_peered":"2024-04-25T01:33:00.625691+0000","last_clean":"2024-04-25T01:33:00.625691+0000","last_became_active":"2024-04-25T01:30:38.372102+0000","last_became_peered":"2024-04-25T01:30:38.372102+0000","last_unstale":"2024-04-25T01:33:00.625691+0000","last_undegraded":"2024-04-25T01:33:00.625691+0000","last_fullsized":"2024-04-25T01:33:00.625691+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:22:16.791842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642090+0000","last_change":"2024-04-25T01:30:33.344065+0000","last_active":"2024-04-25T01:33:00.642090+0000","last_peered":"2024-04-25T01:33:00.642090+0000","last_clean":"2024-04-25T01:33:00.642090+0000","last_became_active":"2024-04-25T01:30:33.343700+0000","last_became_peered":"2024-04-25T01:30:33.343700+0000","last_unstale":"2024-04-25T01:33:00.642090+0000","last_undegraded":"2024-04-25T01:33:00.642090+0000","last_fullsized":"2024-04-25T01:33:00.642090+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:54:08.265744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643124+0000","last_change":"2024-04-25T01:30:34.354644+0000","last_active":"2024-04-25T01:33:00.643124+0000","last_peered":"2024-04-25T01:33:00.643124+0000","last_clean":"2024-04-25T01:33:00.643124+0000","last_became_active":"2024-04-25T01:30:34.354330+0000","last_became_peered":"2024-04-25T01:30:34.354330+0000","last_unstale":"2024-04-25T01:33:00.643124+0000","last_undegraded":"2024-04-25T01:33:00.643124+0000","last_fullsized":"2024-04-25T01:33:00.643124+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:31:50.399983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641601+0000","last_change":"2024-04-25T01:30:38.450743+0000","last_active":"2024-04-25T01:33:00.641601+0000","last_peered":"2024-04-25T01:33:00.641601+0000","last_clean":"2024-04-25T01:33:00.641601+0000","last_became_active":"2024-04-25T01:30:38.450303+0000","last_became_peered":"2024-04-25T01:30:38.450303+0000","last_unstale":"2024-04-25T01:33:00.641601+0000","last_undegraded":"2024-04-25T01:33:00.641601+0000","last_fullsized":"2024-04-25T01:33:00.641601+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:37.919250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640527+0000","last_change":"2024-04-25T01:30:33.350709+0000","last_active":"2024-04-25T01:33:00.640527+0000","last_peered":"2024-04-25T01:33:00.640527+0000","last_clean":"2024-04-25T01:33:00.640527+0000","last_became_active":"2024-04-25T01:30:33.350387+0000","last_became_peered":"2024-04-25T01:30:33.350387+0000","last_unstale":"2024-04-25T01:33:00.640527+0000","last_undegraded":"2024-04-25T01:33:00.640527+0000","last_fullsized":"2024-04-25T01:33:00.640527+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:30:01.252121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730287+0000","last_change":"2024-04-25T01:30:34.362040+0000","last_active":"2024-04-25T01:33:00.730287+0000","last_peered":"2024-04-25T01:33:00.730287+0000","last_clean":"2024-04-25T01:33:00.730287+0000","last_became_active":"2024-04-25T01:30:34.361734+0000","last_became_peered":"2024-04-25T01:30:34.361734+0000","last_unstale":"2024-04-25T01:33:00.730287+0000","last_undegraded":"2024-04-25T01:33:00.730287+0000","last_fullsized":"2024-04-25T01:33:00.730287+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:11:28.409764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641032+0000","last_change":"2024-04-25T01:31:21.877148+0000","last_active":"2024-04-25T01:33:00.641032+0000","last_peered":"2024-04-25T01:33:00.641032+0000","last_clean":"2024-04-25T01:33:00.641032+0000","last_became_active":"2024-04-25T01:31:21.876665+0000","last_became_peered":"2024-04-25T01:31:21.876665+0000","last_unstale":"2024-04-25T01:33:00.641032+0000","last_undegraded":"2024-04-25T01:33:00.641032+0000","last_fullsized":"2024-04-25T01:33:00.641032+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:32:38.425434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,8],"acting":[9,3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622123+0000","last_change":"2024-04-25T01:30:33.350207+0000","last_active":"2024-04-25T01:33:00.622123+0000","last_peered":"2024-04-25T01:33:00.622123+0000","last_clean":"2024-04-25T01:33:00.622123+0000","last_became_active":"2024-04-25T01:30:33.349892+0000","last_became_peered":"2024-04-25T01:30:33.349892+0000","last_unstale":"2024-04-25T01:33:00.622123+0000","last_undegraded":"2024-04-25T01:33:00.622123+0000","last_fullsized":"2024-04-25T01:33:00.622123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:26:35.186486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640590+0000","last_change":"2024-04-25T01:30:34.355196+0000","last_active":"2024-04-25T01:33:00.640590+0000","last_peered":"2024-04-25T01:33:00.640590+0000","last_clean":"2024-04-25T01:33:00.640590+0000","last_became_active":"2024-04-25T01:30:34.354794+0000","last_became_peered":"2024-04-25T01:30:34.354794+0000","last_unstale":"2024-04-25T01:33:00.640590+0000","last_undegraded":"2024-04-25T01:33:00.640590+0000","last_fullsized":"2024-04-25T01:33:00.640590+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:43:37.495974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641714+0000","last_change":"2024-04-25T01:30:38.384159+0000","last_active":"2024-04-25T01:33:00.641714+0000","last_peered":"2024-04-25T01:33:00.641714+0000","last_clean":"2024-04-25T01:33:00.641714+0000","last_became_active":"2024-04-25T01:30:38.383676+0000","last_became_peered":"2024-04-25T01:30:38.383676+0000","last_unstale":"2024-04-25T01:33:00.641714+0000","last_undegraded":"2024-04-25T01:33:00.641714+0000","last_fullsized":"2024-04-25T01:33:00.641714+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:28.491522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":19,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640644+0000","last_change":"2024-04-25T01:31:21.889311+0000","last_active":"2024-04-25T01:33:00.640644+0000","last_peered":"2024-04-25T01:33:00.640644+0000","last_clean":"2024-04-25T01:33:00.640644+0000","last_became_active":"2024-04-25T01:31:21.877027+0000","last_became_peered":"2024-04-25T01:31:21.877027+0000","last_unstale":"2024-04-25T01:33:00.640644+0000","last_undegraded":"2024-04-25T01:33:00.640644+0000","last_fullsized":"2024-04-25T01:33:00.640644+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:02.206671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622555+0000","last_change":"2024-04-25T01:30:34.358110+0000","last_active":"2024-04-25T01:33:00.622555+0000","last_peered":"2024-04-25T01:33:00.622555+0000","last_clean":"2024-04-25T01:33:00.622555+0000","last_became_active":"2024-04-25T01:30:34.357555+0000","last_became_peered":"2024-04-25T01:30:34.357555+0000","last_unstale":"2024-04-25T01:33:00.622555+0000","last_undegraded":"2024-04-25T01:33:00.622555+0000","last_fullsized":"2024-04-25T01:33:00.622555+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:11.636320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641877+0000","last_change":"2024-04-25T01:30:38.450680+0000","last_active":"2024-04-25T01:33:00.641877+0000","last_peered":"2024-04-25T01:33:00.641877+0000","last_clean":"2024-04-25T01:33:00.641877+0000","last_became_active":"2024-04-25T01:30:38.450168+0000","last_became_peered":"2024-04-25T01:30:38.450168+0000","last_unstale":"2024-04-25T01:33:00.641877+0000","last_undegraded":"2024-04-25T01:33:00.641877+0000","last_fullsized":"2024-04-25T01:33:00.641877+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:39:24.236371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730493+0000","last_change":"2024-04-25T01:30:33.343848+0000","last_active":"2024-04-25T01:33:00.730493+0000","last_peered":"2024-04-25T01:33:00.730493+0000","last_clean":"2024-04-25T01:33:00.730493+0000","last_became_active":"2024-04-25T01:30:33.343498+0000","last_became_peered":"2024-04-25T01:30:33.343498+0000","last_unstale":"2024-04-25T01:33:00.730493+0000","last_undegraded":"2024-04-25T01:33:00.730493+0000","last_fullsized":"2024-04-25T01:33:00.730493+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:58:09.184571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642308+0000","last_change":"2024-04-25T01:30:34.361994+0000","last_active":"2024-04-25T01:33:00.642308+0000","last_peered":"2024-04-25T01:33:00.642308+0000","last_clean":"2024-04-25T01:33:00.642308+0000","last_became_active":"2024-04-25T01:30:34.361532+0000","last_became_peered":"2024-04-25T01:30:34.361532+0000","last_unstale":"2024-04-25T01:33:00.642308+0000","last_undegraded":"2024-04-25T01:33:00.642308+0000","last_fullsized":"2024-04-25T01:33:00.642308+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:30.667672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640976+0000","last_change":"2024-04-25T01:30:38.369041+0000","last_active":"2024-04-25T01:33:00.640976+0000","last_peered":"2024-04-25T01:33:00.640976+0000","last_clean":"2024-04-25T01:33:00.640976+0000","last_became_active":"2024-04-25T01:30:38.368586+0000","last_became_peered":"2024-04-25T01:30:38.368586+0000","last_unstale":"2024-04-25T01:33:00.640976+0000","last_undegraded":"2024-04-25T01:33:00.640976+0000","last_fullsized":"2024-04-25T01:33:00.640976+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:58:53.743012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'7","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.626422+0000","last_change":"2024-04-25T01:30:33.345501+0000","last_active":"2024-04-25T01:33:00.626422+0000","last_peered":"2024-04-25T01:33:00.626422+0000","last_clean":"2024-04-25T01:33:00.626422+0000","last_became_active":"2024-04-25T01:30:33.345179+0000","last_became_peered":"2024-04-25T01:30:33.345179+0000","last_unstale":"2024-04-25T01:33:00.626422+0000","last_undegraded":"2024-04-25T01:33:00.626422+0000","last_fullsized":"2024-04-25T01:33:00.626422+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:29:41.275480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642445+0000","last_change":"2024-04-25T01:30:34.361631+0000","last_active":"2024-04-25T01:33:00.642445+0000","last_peered":"2024-04-25T01:33:00.642445+0000","last_clean":"2024-04-25T01:33:00.642445+0000","last_became_active":"2024-04-25T01:30:34.360266+0000","last_became_peered":"2024-04-25T01:30:34.360266+0000","last_unstale":"2024-04-25T01:33:00.642445+0000","last_undegraded":"2024-04-25T01:33:00.642445+0000","last_fullsized":"2024-04-25T01:33:00.642445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:03.094166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642246+0000","last_change":"2024-04-25T01:30:38.380779+0000","last_active":"2024-04-25T01:33:00.642246+0000","last_peered":"2024-04-25T01:33:00.642246+0000","last_clean":"2024-04-25T01:33:00.642246+0000","last_became_active":"2024-04-25T01:30:38.379959+0000","last_became_peered":"2024-04-25T01:30:38.379959+0000","last_unstale":"2024-04-25T01:33:00.642246+0000","last_undegraded":"2024-04-25T01:33:00.642246+0000","last_fullsized":"2024-04-25T01:33:00.642246+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:48:01.816311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'41","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.845488+0000","last_change":"2024-04-25T01:30:33.345730+0000","last_active":"2024-04-25T01:33:00.845488+0000","last_peered":"2024-04-25T01:33:00.845488+0000","last_clean":"2024-04-25T01:33:00.845488+0000","last_became_active":"2024-04-25T01:30:33.344832+0000","last_became_peered":"2024-04-25T01:30:33.344832+0000","last_unstale":"2024-04-25T01:33:00.845488+0000","last_undegraded":"2024-04-25T01:33:00.845488+0000","last_fullsized":"2024-04-25T01:33:00.845488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:47.018811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642105+0000","last_change":"2024-04-25T01:30:34.355324+0000","last_active":"2024-04-25T01:33:00.642105+0000","last_peered":"2024-04-25T01:33:00.642105+0000","last_clean":"2024-04-25T01:33:00.642105+0000","last_became_active":"2024-04-25T01:30:34.354954+0000","last_became_peered":"2024-04-25T01:30:34.354954+0000","last_unstale":"2024-04-25T01:33:00.642105+0000","last_undegraded":"2024-04-25T01:33:00.642105+0000","last_fullsized":"2024-04-25T01:33:00.642105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:36:41.932391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640928+0000","last_change":"2024-04-25T01:30:38.374386+0000","last_active":"2024-04-25T01:33:00.640928+0000","last_peered":"2024-04-25T01:33:00.640928+0000","last_clean":"2024-04-25T01:33:00.640928+0000","last_became_active":"2024-04-25T01:30:38.373454+0000","last_became_peered":"2024-04-25T01:30:38.373454+0000","last_unstale":"2024-04-25T01:33:00.640928+0000","last_undegraded":"2024-04-25T01:33:00.640928+0000","last_fullsized":"2024-04-25T01:33:00.640928+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:54.456218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"91'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643371+0000","last_change":"2024-04-25T01:30:33.339821+0000","last_active":"2024-04-25T01:33:00.643371+0000","last_peered":"2024-04-25T01:33:00.643371+0000","last_clean":"2024-04-25T01:33:00.643371+0000","last_became_active":"2024-04-25T01:30:33.339500+0000","last_became_peered":"2024-04-25T01:30:33.339500+0000","last_unstale":"2024-04-25T01:33:00.643371+0000","last_undegraded":"2024-04-25T01:33:00.643371+0000","last_fullsized":"2024-04-25T01:33:00.643371+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:44:25.880828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642426+0000","last_change":"2024-04-25T01:30:34.362736+0000","last_active":"2024-04-25T01:33:00.642426+0000","last_peered":"2024-04-25T01:33:00.642426+0000","last_clean":"2024-04-25T01:33:00.642426+0000","last_became_active":"2024-04-25T01:30:34.362371+0000","last_became_peered":"2024-04-25T01:30:34.362371+0000","last_unstale":"2024-04-25T01:33:00.642426+0000","last_undegraded":"2024-04-25T01:33:00.642426+0000","last_fullsized":"2024-04-25T01:33:00.642426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:57:36.088231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640673+0000","last_change":"2024-04-25T01:30:34.359217+0000","last_active":"2024-04-25T01:33:00.640673+0000","last_peered":"2024-04-25T01:33:00.640673+0000","last_clean":"2024-04-25T01:33:00.640673+0000","last_became_active":"2024-04-25T01:30:34.357095+0000","last_became_peered":"2024-04-25T01:30:34.357095+0000","last_unstale":"2024-04-25T01:33:00.640673+0000","last_undegraded":"2024-04-25T01:33:00.640673+0000","last_fullsized":"2024-04-25T01:33:00.640673+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:58.603484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.625896+0000","last_change":"2024-04-25T01:30:34.359513+0000","last_active":"2024-04-25T01:33:00.625896+0000","last_peered":"2024-04-25T01:33:00.625896+0000","last_clean":"2024-04-25T01:33:00.625896+0000","last_became_active":"2024-04-25T01:30:34.359089+0000","last_became_peered":"2024-04-25T01:30:34.359089+0000","last_unstale":"2024-04-25T01:33:00.625896+0000","last_undegraded":"2024-04-25T01:33:00.625896+0000","last_fullsized":"2024-04-25T01:33:00.625896+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:36.159703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.625752+0000","last_change":"2024-04-25T01:30:34.368739+0000","last_active":"2024-04-25T01:33:00.625752+0000","last_peered":"2024-04-25T01:33:00.625752+0000","last_clean":"2024-04-25T01:33:00.625752+0000","last_became_active":"2024-04-25T01:30:34.368392+0000","last_became_peered":"2024-04-25T01:30:34.368392+0000","last_unstale":"2024-04-25T01:33:00.625752+0000","last_undegraded":"2024-04-25T01:33:00.625752+0000","last_fullsized":"2024-04-25T01:33:00.625752+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:39:32.241889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642467+0000","last_change":"2024-04-25T01:31:21.877568+0000","last_active":"2024-04-25T01:33:00.642467+0000","last_peered":"2024-04-25T01:33:00.642467+0000","last_clean":"2024-04-25T01:33:00.642467+0000","last_became_active":"2024-04-25T01:31:21.877071+0000","last_became_peered":"2024-04-25T01:31:21.877071+0000","last_unstale":"2024-04-25T01:33:00.642467+0000","last_undegraded":"2024-04-25T01:33:00.642467+0000","last_fullsized":"2024-04-25T01:33:00.642467+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:02:24.789974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643046+0000","last_change":"2024-04-25T01:30:34.360979+0000","last_active":"2024-04-25T01:33:00.643046+0000","last_peered":"2024-04-25T01:33:00.643046+0000","last_clean":"2024-04-25T01:33:00.643046+0000","last_became_active":"2024-04-25T01:30:34.360630+0000","last_became_peered":"2024-04-25T01:30:34.360630+0000","last_unstale":"2024-04-25T01:33:00.643046+0000","last_undegraded":"2024-04-25T01:33:00.643046+0000","last_fullsized":"2024-04-25T01:33:00.643046+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:05:19.284577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641137+0000","last_change":"2024-04-25T01:30:34.362621+0000","last_active":"2024-04-25T01:33:00.641137+0000","last_peered":"2024-04-25T01:33:00.641137+0000","last_clean":"2024-04-25T01:33:00.641137+0000","last_became_active":"2024-04-25T01:30:34.362290+0000","last_became_peered":"2024-04-25T01:30:34.362290+0000","last_unstale":"2024-04-25T01:33:00.641137+0000","last_undegraded":"2024-04-25T01:33:00.641137+0000","last_fullsized":"2024-04-25T01:33:00.641137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:28:07.750906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.626403+0000","last_change":"2024-04-25T01:30:34.351339+0000","last_active":"2024-04-25T01:33:00.626403+0000","last_peered":"2024-04-25T01:33:00.626403+0000","last_clean":"2024-04-25T01:33:00.626403+0000","last_became_active":"2024-04-25T01:30:34.350655+0000","last_became_peered":"2024-04-25T01:30:34.350655+0000","last_unstale":"2024-04-25T01:33:00.626403+0000","last_undegraded":"2024-04-25T01:33:00.626403+0000","last_fullsized":"2024-04-25T01:33:00.626403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:05.322547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622618+0000","last_change":"2024-04-25T01:30:34.353876+0000","last_active":"2024-04-25T01:33:00.622618+0000","last_peered":"2024-04-25T01:33:00.622618+0000","last_clean":"2024-04-25T01:33:00.622618+0000","last_became_active":"2024-04-25T01:30:34.352942+0000","last_became_peered":"2024-04-25T01:30:34.352942+0000","last_unstale":"2024-04-25T01:33:00.622618+0000","last_undegraded":"2024-04-25T01:33:00.622618+0000","last_fullsized":"2024-04-25T01:33:00.622618+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:47.260171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641358+0000","last_change":"2024-04-25T01:30:34.352039+0000","last_active":"2024-04-25T01:33:00.641358+0000","last_peered":"2024-04-25T01:33:00.641358+0000","last_clean":"2024-04-25T01:33:00.641358+0000","last_became_active":"2024-04-25T01:30:34.351612+0000","last_became_peered":"2024-04-25T01:30:34.351612+0000","last_unstale":"2024-04-25T01:33:00.641358+0000","last_undegraded":"2024-04-25T01:33:00.641358+0000","last_fullsized":"2024-04-25T01:33:00.641358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:03.607711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622510+0000","last_change":"2024-04-25T01:30:34.353692+0000","last_active":"2024-04-25T01:33:00.622510+0000","last_peered":"2024-04-25T01:33:00.622510+0000","last_clean":"2024-04-25T01:33:00.622510+0000","last_became_active":"2024-04-25T01:30:34.353035+0000","last_became_peered":"2024-04-25T01:30:34.353035+0000","last_unstale":"2024-04-25T01:33:00.622510+0000","last_undegraded":"2024-04-25T01:33:00.622510+0000","last_fullsized":"2024-04-25T01:33:00.622510+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:15:54.540315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641104+0000","last_change":"2024-04-25T01:30:34.370797+0000","last_active":"2024-04-25T01:33:00.641104+0000","last_peered":"2024-04-25T01:33:00.641104+0000","last_clean":"2024-04-25T01:33:00.641104+0000","last_became_active":"2024-04-25T01:30:34.367533+0000","last_became_peered":"2024-04-25T01:30:34.367533+0000","last_unstale":"2024-04-25T01:33:00.641104+0000","last_undegraded":"2024-04-25T01:33:00.641104+0000","last_fullsized":"2024-04-25T01:33:00.641104+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:27:15.478808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643248+0000","last_change":"2024-04-25T01:30:34.364843+0000","last_active":"2024-04-25T01:33:00.643248+0000","last_peered":"2024-04-25T01:33:00.643248+0000","last_clean":"2024-04-25T01:33:00.643248+0000","last_became_active":"2024-04-25T01:30:34.364515+0000","last_became_peered":"2024-04-25T01:30:34.364515+0000","last_unstale":"2024-04-25T01:33:00.643248+0000","last_undegraded":"2024-04-25T01:33:00.643248+0000","last_fullsized":"2024-04-25T01:33:00.643248+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:50:03.517814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641005+0000","last_change":"2024-04-25T01:30:34.363715+0000","last_active":"2024-04-25T01:33:00.641005+0000","last_peered":"2024-04-25T01:33:00.641005+0000","last_clean":"2024-04-25T01:33:00.641005+0000","last_became_active":"2024-04-25T01:30:34.363383+0000","last_became_peered":"2024-04-25T01:30:34.363383+0000","last_unstale":"2024-04-25T01:33:00.641005+0000","last_undegraded":"2024-04-25T01:33:00.641005+0000","last_fullsized":"2024-04-25T01:33:00.641005+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:50:18.322550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":601378,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1337,"num_read_kb":786,"num_write":373,"num_write_kb":1470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1804134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":303,"ondisk_log_size":303,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318314,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1077248,"data_stored":519949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579951,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37196,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707956,"statfs":{"total":95995035648,"available":95956946944,"internally_reserved":0,"allocated":1904640,"data_stored":1306841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743477,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1490944,"data_stored":916591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005114,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":509849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266751,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28668,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716484,"statfs":{"total":95995035648,"available":95965679616,"internally_reserved":0,"allocated":1687552,"data_stored":1116994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430277,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1069056,"data_stored":514783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1142784,"data_stored":583799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1093632,"data_stored":516771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36468,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708684,"statfs":{"total":95995035648,"available":95957692416,"internally_reserved":0,"allocated":1155072,"data_stored":585015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36908,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708244,"statfs":{"total":95995035648,"available":95957241856,"internally_reserved":0,"allocated":1609728,"data_stored":1039307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888142,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":519947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691914,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36996,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708156,"statfs":{"total":95995035648,"available":95957151744,"internally_reserved":0,"allocated":1695744,"data_stored":1111877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":70954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":134952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":394716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":202412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:33:03.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:33:03.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.214+0000 7ff1082a5640 1 -- 172.21.15.28:0/537111217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100100680 msgr2=0x7ff100100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.214+0000 7ff1082a5640 1 --2- 172.21.15.28:0/537111217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100100680 0x7ff100100aa0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0009900 tx=0x7ff0f002a410 comp rx=0 tx=0).stop 2024-04-25T01:33:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.214+0000 7ff1082a5640 1 -- 172.21.15.28:0/537111217 shutdown_connections 2024-04-25T01:33:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.214+0000 7ff1082a5640 1 --2- 172.21.15.28:0/537111217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff100102230 0x7ff100108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.214+0000 7ff1082a5640 1 --2- 172.21.15.28:0/537111217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff100101870 0x7ff100101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.214+0000 7ff1082a5640 1 --2- 172.21.15.28:0/537111217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100100680 0x7ff100100aa0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.214+0000 7ff1082a5640 1 -- 172.21.15.28:0/537111217 >> 172.21.15.28:0/537111217 conn(0x7ff1000fbbf0 msgr2=0x7ff1000fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.214+0000 7ff1082a5640 1 -- 172.21.15.28:0/537111217 shutdown_connections 2024-04-25T01:33:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.214+0000 7ff1082a5640 1 -- 172.21.15.28:0/537111217 wait complete. 2024-04-25T01:33:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.215+0000 7ff1082a5640 1 Processor -- start 2024-04-25T01:33:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.215+0000 7ff1082a5640 1 -- start start 2024-04-25T01:33:03.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.215+0000 7ff1082a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100100680 0x7ff100197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.215+0000 7ff1082a5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff100101870 0x7ff100197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.215+0000 7ff1082a5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff100102230 0x7ff10019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.215+0000 7ff1082a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff10010c300 con 0x7ff100100680 2024-04-25T01:33:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.215+0000 7ff1082a5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff10010c180 con 0x7ff100101870 2024-04-25T01:33:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.215+0000 7ff1082a5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff10010c480 con 0x7ff100102230 2024-04-25T01:33:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.215+0000 7ff106041640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100100680 0x7ff100197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.215+0000 7ff106041640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100100680 0x7ff100197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55102/0 (socket says 172.21.15.28:55102) 2024-04-25T01:33:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.215+0000 7ff106842640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff100102230 0x7ff10019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.215+0000 7ff106041640 1 -- 172.21.15.28:0/867021896 learned_addr learned my addr 172.21.15.28:0/867021896 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.215+0000 7ff105840640 1 --2- 172.21.15.28:0/867021896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff100101870 0x7ff100197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.215+0000 7ff106041640 1 -- 172.21.15.28:0/867021896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff100102230 msgr2=0x7ff10019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.215+0000 7ff106041640 1 --2- 172.21.15.28:0/867021896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff100102230 0x7ff10019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.215+0000 7ff106041640 1 -- 172.21.15.28:0/867021896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff100101870 msgr2=0x7ff100197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.215+0000 7ff106041640 1 --2- 172.21.15.28:0/867021896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff100101870 0x7ff100197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.216+0000 7ff106041640 1 -- 172.21.15.28:0/867021896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10019f6d0 con 0x7ff100100680 2024-04-25T01:33:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.216+0000 7ff105840640 1 --2- 172.21.15.28:0/867021896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff100101870 0x7ff100197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.216+0000 7ff106041640 1 --2- 172.21.15.28:0/867021896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100100680 0x7ff100197ab0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0031610 tx=0x7ff0f0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.216+0000 7ff0f77fe640 1 -- 172.21.15.28:0/867021896 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f003f030 con 0x7ff100100680 2024-04-25T01:33:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.216+0000 7ff0f77fe640 1 -- 172.21.15.28:0/867021896 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0f0003ad0 con 0x7ff100100680 2024-04-25T01:33:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.216+0000 7ff1082a5640 1 -- 172.21.15.28:0/867021896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff10019f960 con 0x7ff100100680 2024-04-25T01:33:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.216+0000 7ff0f77fe640 1 -- 172.21.15.28:0/867021896 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0035070 con 0x7ff100100680 2024-04-25T01:33:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.216+0000 7ff1082a5640 1 -- 172.21.15.28:0/867021896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff10019fe50 con 0x7ff100100680 2024-04-25T01:33:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.217+0000 7ff1082a5640 1 -- 172.21.15.28:0/867021896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1001a0020 con 0x7ff100100680 2024-04-25T01:33:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.219+0000 7ff0f77fe640 1 -- 172.21.15.28:0/867021896 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7ff0f0042030 con 0x7ff100100680 2024-04-25T01:33:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.219+0000 7ff0f77fe640 1 --2- 172.21.15.28:0/867021896 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff0ec0a2d00 0x7ff0ec0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.220+0000 7ff105840640 1 --2- 172.21.15.28:0/867021896 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff0ec0a2d00 0x7ff0ec0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.220+0000 7ff0f77fe640 1 -- 172.21.15.28:0/867021896 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff0f00effe0 con 0x7ff100100680 2024-04-25T01:33:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.220+0000 7ff105840640 1 --2- 172.21.15.28:0/867021896 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff0ec0a2d00 0x7ff0ec0a51c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff100198fc0 tx=0x7ff0fc005f90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.223+0000 7ff0f77fe640 1 -- 172.21.15.28:0/867021896 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff0f00bc270 con 0x7ff100100680 2024-04-25T01:33:03.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:02 smithi130 ceph-mon[32702]: [25/Apr/2024:01:33:01] ENGINE Bus STARTING 2024-04-25T01:33:03.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:02 smithi130 ceph-mon[32702]: [25/Apr/2024:01:33:01] ENGINE Serving on https://172.21.15.28:7150 2024-04-25T01:33:03.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:02 smithi130 ceph-mon[32702]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:33:03.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:02 smithi130 ceph-mon[32702]: [25/Apr/2024:01:33:01] ENGINE Serving on http://172.21.15.28:8765 2024-04-25T01:33:03.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:02 smithi130 ceph-mon[32702]: [25/Apr/2024:01:33:01] ENGINE Bus STARTED 2024-04-25T01:33:03.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:02 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3955829508' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:33:03.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:02 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/53527557' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:33:03.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:02 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2174995530' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:33:03.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:02 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:03.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:02 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:03.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:02 smithi032 ceph-mon[32461]: [25/Apr/2024:01:33:01] ENGINE Bus STARTING 2024-04-25T01:33:03.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:02 smithi032 ceph-mon[32461]: [25/Apr/2024:01:33:01] ENGINE Serving on https://172.21.15.28:7150 2024-04-25T01:33:03.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:02 smithi032 ceph-mon[32461]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:33:03.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:02 smithi032 ceph-mon[32461]: [25/Apr/2024:01:33:01] ENGINE Serving on http://172.21.15.28:8765 2024-04-25T01:33:03.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:02 smithi032 ceph-mon[32461]: [25/Apr/2024:01:33:01] ENGINE Bus STARTED 2024-04-25T01:33:03.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:02 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3955829508' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:33:03.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:02 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/53527557' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:33:03.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:02 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2174995530' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:33:03.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:02 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:03.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:02 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:03.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.407+0000 7ff1082a5640 1 -- 172.21.15.28:0/867021896 --> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff1001a01f0 con 0x7ff0ec0a2d00 2024-04-25T01:33:03.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.416+0000 7ff0f77fe640 1 -- 172.21.15.28:0/867021896 <== mgr.15561 v2:172.21.15.28:6800/712762371 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339435 (secure 0 0 0) 0x7ff1001a01f0 con 0x7ff0ec0a2d00 2024-04-25T01:33:03.398 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:33:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T01:33:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.420+0000 7ff1082a5640 1 -- 172.21.15.28:0/867021896 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff0ec0a2d00 msgr2=0x7ff0ec0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.420+0000 7ff1082a5640 1 --2- 172.21.15.28:0/867021896 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff0ec0a2d00 0x7ff0ec0a51c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff100198fc0 tx=0x7ff0fc005f90 comp rx=0 tx=0).stop 2024-04-25T01:33:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.420+0000 7ff1082a5640 1 -- 172.21.15.28:0/867021896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100100680 msgr2=0x7ff100197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.420+0000 7ff1082a5640 1 --2- 172.21.15.28:0/867021896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100100680 0x7ff100197ab0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0031610 tx=0x7ff0f0004270 comp rx=0 tx=0).stop 2024-04-25T01:33:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.420+0000 7ff1082a5640 1 -- 172.21.15.28:0/867021896 shutdown_connections 2024-04-25T01:33:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.420+0000 7ff1082a5640 1 --2- 172.21.15.28:0/867021896 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff0ec0a2d00 0x7ff0ec0a51c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.420+0000 7ff1082a5640 1 --2- 172.21.15.28:0/867021896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff100102230 0x7ff10019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.420+0000 7ff1082a5640 1 --2- 172.21.15.28:0/867021896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff100101870 0x7ff100197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.420+0000 7ff1082a5640 1 --2- 172.21.15.28:0/867021896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100100680 0x7ff100197ab0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.420+0000 7ff1082a5640 1 -- 172.21.15.28:0/867021896 >> 172.21.15.28:0/867021896 conn(0x7ff1000fbbf0 msgr2=0x7ff1000fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.420+0000 7ff1082a5640 1 -- 172.21.15.28:0/867021896 shutdown_connections 2024-04-25T01:33:03.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.420+0000 7ff1082a5640 1 -- 172.21.15.28:0/867021896 wait complete. 2024-04-25T01:33:03.409 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-04-25T01:33:02.635937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1191980,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1473,"num_read_kb":905,"num_write":556,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1180804,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":942,"ondisk_log_size":942,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":389848,"kb_used_data":15700,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124551976,"statfs":{"total":1151940427776,"available":1151541223424,"internally_reserved":0,"allocated":16076800,"data_stored":9241723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":382973242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.986097"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641539+0000","last_change":"2024-04-25T01:30:34.358339+0000","last_active":"2024-04-25T01:33:00.641539+0000","last_peered":"2024-04-25T01:33:00.641539+0000","last_clean":"2024-04-25T01:33:00.641539+0000","last_became_active":"2024-04-25T01:30:34.357884+0000","last_became_peered":"2024-04-25T01:30:34.357884+0000","last_unstale":"2024-04-25T01:33:00.641539+0000","last_undegraded":"2024-04-25T01:33:00.641539+0000","last_fullsized":"2024-04-25T01:33:00.641539+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:56.409080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643075+0000","last_change":"2024-04-25T01:30:34.359933+0000","last_active":"2024-04-25T01:33:00.643075+0000","last_peered":"2024-04-25T01:33:00.643075+0000","last_clean":"2024-04-25T01:33:00.643075+0000","last_became_active":"2024-04-25T01:30:34.359576+0000","last_became_peered":"2024-04-25T01:30:34.359576+0000","last_unstale":"2024-04-25T01:33:00.643075+0000","last_undegraded":"2024-04-25T01:33:00.643075+0000","last_fullsized":"2024-04-25T01:33:00.643075+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:48:47.764121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642789+0000","last_change":"2024-04-25T01:30:34.357772+0000","last_active":"2024-04-25T01:33:00.642789+0000","last_peered":"2024-04-25T01:33:00.642789+0000","last_clean":"2024-04-25T01:33:00.642789+0000","last_became_active":"2024-04-25T01:30:34.357434+0000","last_became_peered":"2024-04-25T01:30:34.357434+0000","last_unstale":"2024-04-25T01:33:00.642789+0000","last_undegraded":"2024-04-25T01:33:00.642789+0000","last_fullsized":"2024-04-25T01:33:00.642789+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:19.926180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641967+0000","last_change":"2024-04-25T01:30:34.351250+0000","last_active":"2024-04-25T01:33:00.641967+0000","last_peered":"2024-04-25T01:33:00.641967+0000","last_clean":"2024-04-25T01:33:00.641967+0000","last_became_active":"2024-04-25T01:30:34.349976+0000","last_became_peered":"2024-04-25T01:30:34.349976+0000","last_unstale":"2024-04-25T01:33:00.641967+0000","last_undegraded":"2024-04-25T01:33:00.641967+0000","last_fullsized":"2024-04-25T01:33:00.641967+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:19:51.605307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730612+0000","last_change":"2024-04-25T01:30:34.367882+0000","last_active":"2024-04-25T01:33:00.730612+0000","last_peered":"2024-04-25T01:33:00.730612+0000","last_clean":"2024-04-25T01:33:00.730612+0000","last_became_active":"2024-04-25T01:30:34.366797+0000","last_became_peered":"2024-04-25T01:30:34.366797+0000","last_unstale":"2024-04-25T01:33:00.730612+0000","last_undegraded":"2024-04-25T01:33:00.730612+0000","last_fullsized":"2024-04-25T01:33:00.730612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:35:23.439061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.623521+0000","last_change":"2024-04-25T01:30:34.352506+0000","last_active":"2024-04-25T01:33:00.623521+0000","last_peered":"2024-04-25T01:33:00.623521+0000","last_clean":"2024-04-25T01:33:00.623521+0000","last_became_active":"2024-04-25T01:30:34.352100+0000","last_became_peered":"2024-04-25T01:30:34.352100+0000","last_unstale":"2024-04-25T01:33:00.623521+0000","last_undegraded":"2024-04-25T01:33:00.623521+0000","last_fullsized":"2024-04-25T01:33:00.623521+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:38.113959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641641+0000","last_change":"2024-04-25T01:30:34.355352+0000","last_active":"2024-04-25T01:33:00.641641+0000","last_peered":"2024-04-25T01:33:00.641641+0000","last_clean":"2024-04-25T01:33:00.641641+0000","last_became_active":"2024-04-25T01:30:34.355008+0000","last_became_peered":"2024-04-25T01:30:34.355008+0000","last_unstale":"2024-04-25T01:33:00.641641+0000","last_undegraded":"2024-04-25T01:33:00.641641+0000","last_fullsized":"2024-04-25T01:33:00.641641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:23.582742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622074+0000","last_change":"2024-04-25T01:30:34.360644+0000","last_active":"2024-04-25T01:33:00.622074+0000","last_peered":"2024-04-25T01:33:00.622074+0000","last_clean":"2024-04-25T01:33:00.622074+0000","last_became_active":"2024-04-25T01:30:34.357841+0000","last_became_peered":"2024-04-25T01:30:34.357841+0000","last_unstale":"2024-04-25T01:33:00.622074+0000","last_undegraded":"2024-04-25T01:33:00.622074+0000","last_fullsized":"2024-04-25T01:33:00.622074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:26:21.606219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642196+0000","last_change":"2024-04-25T01:30:34.358417+0000","last_active":"2024-04-25T01:33:00.642196+0000","last_peered":"2024-04-25T01:33:00.642196+0000","last_clean":"2024-04-25T01:33:00.642196+0000","last_became_active":"2024-04-25T01:30:34.358039+0000","last_became_peered":"2024-04-25T01:30:34.358039+0000","last_unstale":"2024-04-25T01:33:00.642196+0000","last_undegraded":"2024-04-25T01:33:00.642196+0000","last_fullsized":"2024-04-25T01:33:00.642196+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:42.600693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.624086+0000","last_change":"2024-04-25T01:30:34.357803+0000","last_active":"2024-04-25T01:33:00.624086+0000","last_peered":"2024-04-25T01:33:00.624086+0000","last_clean":"2024-04-25T01:33:00.624086+0000","last_became_active":"2024-04-25T01:30:34.357391+0000","last_became_peered":"2024-04-25T01:30:34.357391+0000","last_unstale":"2024-04-25T01:33:00.624086+0000","last_undegraded":"2024-04-25T01:33:00.624086+0000","last_fullsized":"2024-04-25T01:33:00.624086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:57.857303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642471+0000","last_change":"2024-04-25T01:30:34.363340+0000","last_active":"2024-04-25T01:33:00.642471+0000","last_peered":"2024-04-25T01:33:00.642471+0000","last_clean":"2024-04-25T01:33:00.642471+0000","last_became_active":"2024-04-25T01:30:34.362980+0000","last_became_peered":"2024-04-25T01:30:34.362980+0000","last_unstale":"2024-04-25T01:33:00.642471+0000","last_undegraded":"2024-04-25T01:33:00.642471+0000","last_fullsized":"2024-04-25T01:33:00.642471+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:46.291459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641988+0000","last_change":"2024-04-25T01:30:34.358898+0000","last_active":"2024-04-25T01:33:00.641988+0000","last_peered":"2024-04-25T01:33:00.641988+0000","last_clean":"2024-04-25T01:33:00.641988+0000","last_became_active":"2024-04-25T01:30:34.358495+0000","last_became_peered":"2024-04-25T01:30:34.358495+0000","last_unstale":"2024-04-25T01:33:00.641988+0000","last_undegraded":"2024-04-25T01:33:00.641988+0000","last_fullsized":"2024-04-25T01:33:00.641988+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:34:21.860010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643535+0000","last_change":"2024-04-25T01:30:34.363708+0000","last_active":"2024-04-25T01:33:00.643535+0000","last_peered":"2024-04-25T01:33:00.643535+0000","last_clean":"2024-04-25T01:33:00.643535+0000","last_became_active":"2024-04-25T01:30:34.363207+0000","last_became_peered":"2024-04-25T01:30:34.363207+0000","last_unstale":"2024-04-25T01:33:00.643535+0000","last_undegraded":"2024-04-25T01:33:00.643535+0000","last_fullsized":"2024-04-25T01:33:00.643535+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:53.442987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642070+0000","last_change":"2024-04-25T01:30:34.371031+0000","last_active":"2024-04-25T01:33:00.642070+0000","last_peered":"2024-04-25T01:33:00.642070+0000","last_clean":"2024-04-25T01:33:00.642070+0000","last_became_active":"2024-04-25T01:30:34.370687+0000","last_became_peered":"2024-04-25T01:30:34.370687+0000","last_unstale":"2024-04-25T01:33:00.642070+0000","last_undegraded":"2024-04-25T01:33:00.642070+0000","last_fullsized":"2024-04-25T01:33:00.642070+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:03:20.700960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642487+0000","last_change":"2024-04-25T01:30:34.366212+0000","last_active":"2024-04-25T01:33:00.642487+0000","last_peered":"2024-04-25T01:33:00.642487+0000","last_clean":"2024-04-25T01:33:00.642487+0000","last_became_active":"2024-04-25T01:30:34.365849+0000","last_became_peered":"2024-04-25T01:30:34.365849+0000","last_unstale":"2024-04-25T01:33:00.642487+0000","last_undegraded":"2024-04-25T01:33:00.642487+0000","last_fullsized":"2024-04-25T01:33:00.642487+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:24:02.397615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640584+0000","last_change":"2024-04-25T01:30:34.367380+0000","last_active":"2024-04-25T01:33:00.640584+0000","last_peered":"2024-04-25T01:33:00.640584+0000","last_clean":"2024-04-25T01:33:00.640584+0000","last_became_active":"2024-04-25T01:30:34.366872+0000","last_became_peered":"2024-04-25T01:30:34.366872+0000","last_unstale":"2024-04-25T01:33:00.640584+0000","last_undegraded":"2024-04-25T01:33:00.640584+0000","last_fullsized":"2024-04-25T01:33:00.640584+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:18.767108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641445+0000","last_change":"2024-04-25T01:30:34.359752+0000","last_active":"2024-04-25T01:33:00.641445+0000","last_peered":"2024-04-25T01:33:00.641445+0000","last_clean":"2024-04-25T01:33:00.641445+0000","last_became_active":"2024-04-25T01:30:34.358403+0000","last_became_peered":"2024-04-25T01:30:34.358403+0000","last_unstale":"2024-04-25T01:33:00.641445+0000","last_undegraded":"2024-04-25T01:33:00.641445+0000","last_fullsized":"2024-04-25T01:33:00.641445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:14.833126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641209+0000","last_change":"2024-04-25T01:30:34.361373+0000","last_active":"2024-04-25T01:33:00.641209+0000","last_peered":"2024-04-25T01:33:00.641209+0000","last_clean":"2024-04-25T01:33:00.641209+0000","last_became_active":"2024-04-25T01:30:34.361046+0000","last_became_peered":"2024-04-25T01:30:34.361046+0000","last_unstale":"2024-04-25T01:33:00.641209+0000","last_undegraded":"2024-04-25T01:33:00.641209+0000","last_fullsized":"2024-04-25T01:33:00.641209+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:21.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.626292+0000","last_change":"2024-04-25T01:30:34.367975+0000","last_active":"2024-04-25T01:33:00.626292+0000","last_peered":"2024-04-25T01:33:00.626292+0000","last_clean":"2024-04-25T01:33:00.626292+0000","last_became_active":"2024-04-25T01:30:34.366383+0000","last_became_peered":"2024-04-25T01:30:34.366383+0000","last_unstale":"2024-04-25T01:33:00.626292+0000","last_undegraded":"2024-04-25T01:33:00.626292+0000","last_fullsized":"2024-04-25T01:33:00.626292+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:53:40.279067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.625811+0000","last_change":"2024-04-25T01:30:38.382168+0000","last_active":"2024-04-25T01:33:00.625811+0000","last_peered":"2024-04-25T01:33:00.625811+0000","last_clean":"2024-04-25T01:33:00.625811+0000","last_became_active":"2024-04-25T01:30:38.381740+0000","last_became_peered":"2024-04-25T01:30:38.381740+0000","last_unstale":"2024-04-25T01:33:00.625811+0000","last_undegraded":"2024-04-25T01:33:00.625811+0000","last_fullsized":"2024-04-25T01:33:00.625811+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:27.425991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'13","reported_seq":164,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641584+0000","last_change":"2024-04-25T01:30:33.345547+0000","last_active":"2024-04-25T01:33:00.641584+0000","last_peered":"2024-04-25T01:33:00.641584+0000","last_clean":"2024-04-25T01:33:00.641584+0000","last_became_active":"2024-04-25T01:30:33.345229+0000","last_became_peered":"2024-04-25T01:30:33.345229+0000","last_unstale":"2024-04-25T01:33:00.641584+0000","last_undegraded":"2024-04-25T01:33:00.641584+0000","last_fullsized":"2024-04-25T01:33:00.641584+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:32:46.637720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622343+0000","last_change":"2024-04-25T01:30:34.354703+0000","last_active":"2024-04-25T01:33:00.622343+0000","last_peered":"2024-04-25T01:33:00.622343+0000","last_clean":"2024-04-25T01:33:00.622343+0000","last_became_active":"2024-04-25T01:30:34.354362+0000","last_became_peered":"2024-04-25T01:30:34.354362+0000","last_unstale":"2024-04-25T01:33:00.622343+0000","last_undegraded":"2024-04-25T01:33:00.622343+0000","last_fullsized":"2024-04-25T01:33:00.622343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:51.358599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640543+0000","last_change":"2024-04-25T01:30:38.380912+0000","last_active":"2024-04-25T01:33:00.640543+0000","last_peered":"2024-04-25T01:33:00.640543+0000","last_clean":"2024-04-25T01:33:00.640543+0000","last_became_active":"2024-04-25T01:30:38.380439+0000","last_became_peered":"2024-04-25T01:30:38.380439+0000","last_unstale":"2024-04-25T01:33:00.640543+0000","last_undegraded":"2024-04-25T01:33:00.640543+0000","last_fullsized":"2024-04-25T01:33:00.640543+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:22:00.527268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622488+0000","last_change":"2024-04-25T01:30:33.345620+0000","last_active":"2024-04-25T01:33:00.622488+0000","last_peered":"2024-04-25T01:33:00.622488+0000","last_clean":"2024-04-25T01:33:00.622488+0000","last_became_active":"2024-04-25T01:30:33.345315+0000","last_became_peered":"2024-04-25T01:30:33.345315+0000","last_unstale":"2024-04-25T01:33:00.622488+0000","last_undegraded":"2024-04-25T01:33:00.622488+0000","last_fullsized":"2024-04-25T01:33:00.622488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:48:00.083728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643224+0000","last_change":"2024-04-25T01:30:34.356589+0000","last_active":"2024-04-25T01:33:00.643224+0000","last_peered":"2024-04-25T01:33:00.643224+0000","last_clean":"2024-04-25T01:33:00.643224+0000","last_became_active":"2024-04-25T01:30:34.356114+0000","last_became_peered":"2024-04-25T01:30:34.356114+0000","last_unstale":"2024-04-25T01:33:00.643224+0000","last_undegraded":"2024-04-25T01:33:00.643224+0000","last_fullsized":"2024-04-25T01:33:00.643224+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:39.039517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640748+0000","last_change":"2024-04-25T01:31:21.876578+0000","last_active":"2024-04-25T01:33:00.640748+0000","last_peered":"2024-04-25T01:33:00.640748+0000","last_clean":"2024-04-25T01:33:00.640748+0000","last_became_active":"2024-04-25T01:31:21.876126+0000","last_became_peered":"2024-04-25T01:31:21.876126+0000","last_unstale":"2024-04-25T01:33:00.640748+0000","last_undegraded":"2024-04-25T01:33:00.640748+0000","last_fullsized":"2024-04-25T01:33:00.640748+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:55.909353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'12","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730820+0000","last_change":"2024-04-25T01:30:33.345575+0000","last_active":"2024-04-25T01:33:00.730820+0000","last_peered":"2024-04-25T01:33:00.730820+0000","last_clean":"2024-04-25T01:33:00.730820+0000","last_became_active":"2024-04-25T01:30:33.345231+0000","last_became_peered":"2024-04-25T01:30:33.345231+0000","last_unstale":"2024-04-25T01:33:00.730820+0000","last_undegraded":"2024-04-25T01:33:00.730820+0000","last_fullsized":"2024-04-25T01:33:00.730820+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:04:21.090788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641074+0000","last_change":"2024-04-25T01:30:34.371317+0000","last_active":"2024-04-25T01:33:00.641074+0000","last_peered":"2024-04-25T01:33:00.641074+0000","last_clean":"2024-04-25T01:33:00.641074+0000","last_became_active":"2024-04-25T01:30:34.370926+0000","last_became_peered":"2024-04-25T01:30:34.370926+0000","last_unstale":"2024-04-25T01:33:00.641074+0000","last_undegraded":"2024-04-25T01:33:00.641074+0000","last_fullsized":"2024-04-25T01:33:00.641074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:58:39.898471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641235+0000","last_change":"2024-04-25T01:30:38.380982+0000","last_active":"2024-04-25T01:33:00.641235+0000","last_peered":"2024-04-25T01:33:00.641235+0000","last_clean":"2024-04-25T01:33:00.641235+0000","last_became_active":"2024-04-25T01:30:38.380510+0000","last_became_peered":"2024-04-25T01:30:38.380510+0000","last_unstale":"2024-04-25T01:33:00.641235+0000","last_undegraded":"2024-04-25T01:33:00.641235+0000","last_fullsized":"2024-04-25T01:33:00.641235+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:00.510697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"91'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730845+0000","last_change":"2024-04-25T01:30:33.351561+0000","last_active":"2024-04-25T01:33:00.730845+0000","last_peered":"2024-04-25T01:33:00.730845+0000","last_clean":"2024-04-25T01:33:00.730845+0000","last_became_active":"2024-04-25T01:30:33.351230+0000","last_became_peered":"2024-04-25T01:30:33.351230+0000","last_unstale":"2024-04-25T01:33:00.730845+0000","last_undegraded":"2024-04-25T01:33:00.730845+0000","last_fullsized":"2024-04-25T01:33:00.730845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:56.594457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641855+0000","last_change":"2024-04-25T01:30:34.358821+0000","last_active":"2024-04-25T01:33:00.641855+0000","last_peered":"2024-04-25T01:33:00.641855+0000","last_clean":"2024-04-25T01:33:00.641855+0000","last_became_active":"2024-04-25T01:30:34.358303+0000","last_became_peered":"2024-04-25T01:30:34.358303+0000","last_unstale":"2024-04-25T01:33:00.641855+0000","last_undegraded":"2024-04-25T01:33:00.641855+0000","last_fullsized":"2024-04-25T01:33:00.641855+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:26:23.352101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.626107+0000","last_change":"2024-04-25T01:30:38.372281+0000","last_active":"2024-04-25T01:33:00.626107+0000","last_peered":"2024-04-25T01:33:00.626107+0000","last_clean":"2024-04-25T01:33:00.626107+0000","last_became_active":"2024-04-25T01:30:38.371876+0000","last_became_peered":"2024-04-25T01:30:38.371876+0000","last_unstale":"2024-04-25T01:33:00.626107+0000","last_undegraded":"2024-04-25T01:33:00.626107+0000","last_fullsized":"2024-04-25T01:33:00.626107+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:54.254011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'7","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.980108+0000","last_change":"2024-04-25T01:30:33.347262+0000","last_active":"2024-04-25T01:33:00.980108+0000","last_peered":"2024-04-25T01:33:00.980108+0000","last_clean":"2024-04-25T01:33:00.980108+0000","last_became_active":"2024-04-25T01:30:33.346900+0000","last_became_peered":"2024-04-25T01:30:33.346900+0000","last_unstale":"2024-04-25T01:33:00.980108+0000","last_undegraded":"2024-04-25T01:33:00.980108+0000","last_fullsized":"2024-04-25T01:33:00.980108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:29:30.960456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730404+0000","last_change":"2024-04-25T01:30:34.364870+0000","last_active":"2024-04-25T01:33:00.730404+0000","last_peered":"2024-04-25T01:33:00.730404+0000","last_clean":"2024-04-25T01:33:00.730404+0000","last_became_active":"2024-04-25T01:30:34.364459+0000","last_became_peered":"2024-04-25T01:30:34.364459+0000","last_unstale":"2024-04-25T01:33:00.730404+0000","last_undegraded":"2024-04-25T01:33:00.730404+0000","last_fullsized":"2024-04-25T01:33:00.730404+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:42:45.311668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.626138+0000","last_change":"2024-04-25T01:30:38.382175+0000","last_active":"2024-04-25T01:33:00.626138+0000","last_peered":"2024-04-25T01:33:00.626138+0000","last_clean":"2024-04-25T01:33:00.626138+0000","last_became_active":"2024-04-25T01:30:38.381771+0000","last_became_peered":"2024-04-25T01:30:38.381771+0000","last_unstale":"2024-04-25T01:33:00.626138+0000","last_undegraded":"2024-04-25T01:33:00.626138+0000","last_fullsized":"2024-04-25T01:33:00.626138+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:25:31.041566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.623581+0000","last_change":"2024-04-25T01:30:33.343605+0000","last_active":"2024-04-25T01:33:00.623581+0000","last_peered":"2024-04-25T01:33:00.623581+0000","last_clean":"2024-04-25T01:33:00.623581+0000","last_became_active":"2024-04-25T01:30:33.343254+0000","last_became_peered":"2024-04-25T01:30:33.343254+0000","last_unstale":"2024-04-25T01:33:00.623581+0000","last_undegraded":"2024-04-25T01:33:00.623581+0000","last_fullsized":"2024-04-25T01:33:00.623581+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:18:16.156285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640694+0000","last_change":"2024-04-25T01:31:21.874202+0000","last_active":"2024-04-25T01:33:00.640694+0000","last_peered":"2024-04-25T01:33:00.640694+0000","last_clean":"2024-04-25T01:33:00.640694+0000","last_became_active":"2024-04-25T01:31:21.873839+0000","last_became_peered":"2024-04-25T01:31:21.873839+0000","last_unstale":"2024-04-25T01:33:00.640694+0000","last_undegraded":"2024-04-25T01:33:00.640694+0000","last_fullsized":"2024-04-25T01:33:00.640694+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:39.774281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641353+0000","last_change":"2024-04-25T01:31:21.875259+0000","last_active":"2024-04-25T01:33:00.641353+0000","last_peered":"2024-04-25T01:33:00.641353+0000","last_clean":"2024-04-25T01:33:00.641353+0000","last_became_active":"2024-04-25T01:31:21.874800+0000","last_became_peered":"2024-04-25T01:31:21.874800+0000","last_unstale":"2024-04-25T01:33:00.641353+0000","last_undegraded":"2024-04-25T01:33:00.641353+0000","last_fullsized":"2024-04-25T01:33:00.641353+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:44.828620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,10,0],"acting":[11,4,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'3","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642169+0000","last_change":"2024-04-25T01:30:33.351621+0000","last_active":"2024-04-25T01:33:00.642169+0000","last_peered":"2024-04-25T01:33:00.642169+0000","last_clean":"2024-04-25T01:33:00.642169+0000","last_became_active":"2024-04-25T01:30:33.351289+0000","last_became_peered":"2024-04-25T01:30:33.351289+0000","last_unstale":"2024-04-25T01:33:00.642169+0000","last_undegraded":"2024-04-25T01:33:00.642169+0000","last_fullsized":"2024-04-25T01:33:00.642169+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:46.044826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622260+0000","last_change":"2024-04-25T01:30:34.350685+0000","last_active":"2024-04-25T01:33:00.622260+0000","last_peered":"2024-04-25T01:33:00.622260+0000","last_clean":"2024-04-25T01:33:00.622260+0000","last_became_active":"2024-04-25T01:30:34.350308+0000","last_became_peered":"2024-04-25T01:30:34.350308+0000","last_unstale":"2024-04-25T01:33:00.622260+0000","last_undegraded":"2024-04-25T01:33:00.622260+0000","last_fullsized":"2024-04-25T01:33:00.622260+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:11.690381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643016+0000","last_change":"2024-04-25T01:30:38.377403+0000","last_active":"2024-04-25T01:33:00.643016+0000","last_peered":"2024-04-25T01:33:00.643016+0000","last_clean":"2024-04-25T01:33:00.643016+0000","last_became_active":"2024-04-25T01:30:38.376966+0000","last_became_peered":"2024-04-25T01:30:38.376966+0000","last_unstale":"2024-04-25T01:33:00.643016+0000","last_undegraded":"2024-04-25T01:33:00.643016+0000","last_fullsized":"2024-04-25T01:33:00.643016+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:31:10.229443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'3","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641337+0000","last_change":"2024-04-25T01:30:33.348455+0000","last_active":"2024-04-25T01:33:00.641337+0000","last_peered":"2024-04-25T01:33:00.641337+0000","last_clean":"2024-04-25T01:33:00.641337+0000","last_became_active":"2024-04-25T01:30:33.348142+0000","last_became_peered":"2024-04-25T01:30:33.348142+0000","last_unstale":"2024-04-25T01:33:00.641337+0000","last_undegraded":"2024-04-25T01:33:00.641337+0000","last_fullsized":"2024-04-25T01:33:00.641337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:43:07.695580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640378+0000","last_change":"2024-04-25T01:30:34.359949+0000","last_active":"2024-04-25T01:33:00.640378+0000","last_peered":"2024-04-25T01:33:00.640378+0000","last_clean":"2024-04-25T01:33:00.640378+0000","last_became_active":"2024-04-25T01:30:34.359609+0000","last_became_peered":"2024-04-25T01:30:34.359609+0000","last_unstale":"2024-04-25T01:33:00.640378+0000","last_undegraded":"2024-04-25T01:33:00.640378+0000","last_fullsized":"2024-04-25T01:33:00.640378+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:49.668800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622175+0000","last_change":"2024-04-25T01:30:38.380260+0000","last_active":"2024-04-25T01:33:00.622175+0000","last_peered":"2024-04-25T01:33:00.622175+0000","last_clean":"2024-04-25T01:33:00.622175+0000","last_became_active":"2024-04-25T01:30:38.379624+0000","last_became_peered":"2024-04-25T01:30:38.379624+0000","last_unstale":"2024-04-25T01:33:00.622175+0000","last_undegraded":"2024-04-25T01:33:00.622175+0000","last_fullsized":"2024-04-25T01:33:00.622175+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:26:35.235335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'6","reported_seq":276,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.979614+0000","last_change":"2024-04-25T01:30:33.349490+0000","last_active":"2024-04-25T01:33:00.979614+0000","last_peered":"2024-04-25T01:33:00.979614+0000","last_clean":"2024-04-25T01:33:00.979614+0000","last_became_active":"2024-04-25T01:30:33.349136+0000","last_became_peered":"2024-04-25T01:30:33.349136+0000","last_unstale":"2024-04-25T01:33:00.979614+0000","last_undegraded":"2024-04-25T01:33:00.979614+0000","last_fullsized":"2024-04-25T01:33:00.979614+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:16:23.552073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":240,"num_read_kb":240,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641812+0000","last_change":"2024-04-25T01:30:34.372701+0000","last_active":"2024-04-25T01:33:00.641812+0000","last_peered":"2024-04-25T01:33:00.641812+0000","last_clean":"2024-04-25T01:33:00.641812+0000","last_became_active":"2024-04-25T01:30:34.372321+0000","last_became_peered":"2024-04-25T01:30:34.372321+0000","last_unstale":"2024-04-25T01:33:00.641812+0000","last_undegraded":"2024-04-25T01:33:00.641812+0000","last_fullsized":"2024-04-25T01:33:00.641812+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:02.025015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640169+0000","last_change":"2024-04-25T01:30:38.378036+0000","last_active":"2024-04-25T01:33:00.640169+0000","last_peered":"2024-04-25T01:33:00.640169+0000","last_clean":"2024-04-25T01:33:00.640169+0000","last_became_active":"2024-04-25T01:30:38.377602+0000","last_became_peered":"2024-04-25T01:30:38.377602+0000","last_unstale":"2024-04-25T01:33:00.640169+0000","last_undegraded":"2024-04-25T01:33:00.640169+0000","last_fullsized":"2024-04-25T01:33:00.640169+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:45.210518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641468+0000","last_change":"2024-04-25T01:30:33.352928+0000","last_active":"2024-04-25T01:33:00.641468+0000","last_peered":"2024-04-25T01:33:00.641468+0000","last_clean":"2024-04-25T01:33:00.641468+0000","last_became_active":"2024-04-25T01:30:33.352567+0000","last_became_peered":"2024-04-25T01:30:33.352567+0000","last_unstale":"2024-04-25T01:33:00.641468+0000","last_undegraded":"2024-04-25T01:33:00.641468+0000","last_fullsized":"2024-04-25T01:33:00.641468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:47:21.941179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643449+0000","last_change":"2024-04-25T01:30:34.366947+0000","last_active":"2024-04-25T01:33:00.643449+0000","last_peered":"2024-04-25T01:33:00.643449+0000","last_clean":"2024-04-25T01:33:00.643449+0000","last_became_active":"2024-04-25T01:30:34.366337+0000","last_became_peered":"2024-04-25T01:30:34.366337+0000","last_unstale":"2024-04-25T01:33:00.643449+0000","last_undegraded":"2024-04-25T01:33:00.643449+0000","last_fullsized":"2024-04-25T01:33:00.643449+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:48:43.528380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641593+0000","last_change":"2024-04-25T01:31:22.050799+0000","last_active":"2024-04-25T01:33:00.641593+0000","last_peered":"2024-04-25T01:33:00.641593+0000","last_clean":"2024-04-25T01:33:00.641593+0000","last_became_active":"2024-04-25T01:31:22.050130+0000","last_became_peered":"2024-04-25T01:31:22.050130+0000","last_unstale":"2024-04-25T01:33:00.641593+0000","last_undegraded":"2024-04-25T01:33:00.641593+0000","last_fullsized":"2024-04-25T01:33:00.641593+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:40:43.549713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641872+0000","last_change":"2024-04-25T01:30:33.346003+0000","last_active":"2024-04-25T01:33:00.641872+0000","last_peered":"2024-04-25T01:33:00.641872+0000","last_clean":"2024-04-25T01:33:00.641872+0000","last_became_active":"2024-04-25T01:30:33.345254+0000","last_became_peered":"2024-04-25T01:30:33.345254+0000","last_unstale":"2024-04-25T01:33:00.641872+0000","last_undegraded":"2024-04-25T01:33:00.641872+0000","last_fullsized":"2024-04-25T01:33:00.641872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:18:37.957047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641299+0000","last_change":"2024-04-25T01:30:34.366108+0000","last_active":"2024-04-25T01:33:00.641299+0000","last_peered":"2024-04-25T01:33:00.641299+0000","last_clean":"2024-04-25T01:33:00.641299+0000","last_became_active":"2024-04-25T01:30:34.365702+0000","last_became_peered":"2024-04-25T01:30:34.365702+0000","last_unstale":"2024-04-25T01:33:00.641299+0000","last_undegraded":"2024-04-25T01:33:00.641299+0000","last_fullsized":"2024-04-25T01:33:00.641299+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:24:06.802424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640332+0000","last_change":"2024-04-25T01:30:38.376342+0000","last_active":"2024-04-25T01:33:00.640332+0000","last_peered":"2024-04-25T01:33:00.640332+0000","last_clean":"2024-04-25T01:33:00.640332+0000","last_became_active":"2024-04-25T01:30:38.375905+0000","last_became_peered":"2024-04-25T01:30:38.375905+0000","last_unstale":"2024-04-25T01:33:00.640332+0000","last_undegraded":"2024-04-25T01:33:00.640332+0000","last_fullsized":"2024-04-25T01:33:00.640332+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:31.615356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'44","reported_seq":325,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.980291+0000","last_change":"2024-04-25T01:30:33.350770+0000","last_active":"2024-04-25T01:33:00.980291+0000","last_peered":"2024-04-25T01:33:00.980291+0000","last_clean":"2024-04-25T01:33:00.980291+0000","last_became_active":"2024-04-25T01:30:33.350414+0000","last_became_peered":"2024-04-25T01:30:33.350414+0000","last_unstale":"2024-04-25T01:33:00.980291+0000","last_undegraded":"2024-04-25T01:33:00.980291+0000","last_fullsized":"2024-04-25T01:33:00.980291+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:01.771182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67986,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":251,"num_read_kb":193,"num_write":46,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730709+0000","last_change":"2024-04-25T01:30:34.362532+0000","last_active":"2024-04-25T01:33:00.730709+0000","last_peered":"2024-04-25T01:33:00.730709+0000","last_clean":"2024-04-25T01:33:00.730709+0000","last_became_active":"2024-04-25T01:30:34.362137+0000","last_became_peered":"2024-04-25T01:30:34.362137+0000","last_unstale":"2024-04-25T01:33:00.730709+0000","last_undegraded":"2024-04-25T01:33:00.730709+0000","last_fullsized":"2024-04-25T01:33:00.730709+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:28.105250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642169+0000","last_change":"2024-04-25T01:30:38.374591+0000","last_active":"2024-04-25T01:33:00.642169+0000","last_peered":"2024-04-25T01:33:00.642169+0000","last_clean":"2024-04-25T01:33:00.642169+0000","last_became_active":"2024-04-25T01:30:38.373237+0000","last_became_peered":"2024-04-25T01:30:38.373237+0000","last_unstale":"2024-04-25T01:33:00.642169+0000","last_undegraded":"2024-04-25T01:33:00.642169+0000","last_fullsized":"2024-04-25T01:33:00.642169+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:21:25.937405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'9","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640904+0000","last_change":"2024-04-25T01:31:21.885341+0000","last_active":"2024-04-25T01:33:00.640904+0000","last_peered":"2024-04-25T01:33:00.640904+0000","last_clean":"2024-04-25T01:33:00.640904+0000","last_became_active":"2024-04-25T01:31:21.874312+0000","last_became_peered":"2024-04-25T01:31:21.874312+0000","last_unstale":"2024-04-25T01:33:00.640904+0000","last_undegraded":"2024-04-25T01:33:00.640904+0000","last_fullsized":"2024-04-25T01:33:00.640904+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:23:55.388089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642553+0000","last_change":"2024-04-25T01:30:34.354444+0000","last_active":"2024-04-25T01:33:00.642553+0000","last_peered":"2024-04-25T01:33:00.642553+0000","last_clean":"2024-04-25T01:33:00.642553+0000","last_became_active":"2024-04-25T01:30:34.354070+0000","last_became_peered":"2024-04-25T01:30:34.354070+0000","last_unstale":"2024-04-25T01:33:00.642553+0000","last_undegraded":"2024-04-25T01:33:00.642553+0000","last_fullsized":"2024-04-25T01:33:00.642553+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:39.893995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640956+0000","last_change":"2024-04-25T01:30:38.373570+0000","last_active":"2024-04-25T01:33:00.640956+0000","last_peered":"2024-04-25T01:33:00.640956+0000","last_clean":"2024-04-25T01:33:00.640956+0000","last_became_active":"2024-04-25T01:30:38.373144+0000","last_became_peered":"2024-04-25T01:30:38.373144+0000","last_unstale":"2024-04-25T01:33:00.640956+0000","last_undegraded":"2024-04-25T01:33:00.640956+0000","last_fullsized":"2024-04-25T01:33:00.640956+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:53:32.933998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641434+0000","last_change":"2024-04-25T01:30:33.346145+0000","last_active":"2024-04-25T01:33:00.641434+0000","last_peered":"2024-04-25T01:33:00.641434+0000","last_clean":"2024-04-25T01:33:00.641434+0000","last_became_active":"2024-04-25T01:30:33.344294+0000","last_became_peered":"2024-04-25T01:30:33.344294+0000","last_unstale":"2024-04-25T01:33:00.641434+0000","last_undegraded":"2024-04-25T01:33:00.641434+0000","last_fullsized":"2024-04-25T01:33:00.641434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:37:17.147148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641769+0000","last_change":"2024-04-25T01:30:34.360769+0000","last_active":"2024-04-25T01:33:00.641769+0000","last_peered":"2024-04-25T01:33:00.641769+0000","last_clean":"2024-04-25T01:33:00.641769+0000","last_became_active":"2024-04-25T01:30:34.360076+0000","last_became_peered":"2024-04-25T01:30:34.360076+0000","last_unstale":"2024-04-25T01:33:00.641769+0000","last_undegraded":"2024-04-25T01:33:00.641769+0000","last_fullsized":"2024-04-25T01:33:00.641769+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:56:21.515511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640926+0000","last_change":"2024-04-25T01:30:38.379198+0000","last_active":"2024-04-25T01:33:00.640926+0000","last_peered":"2024-04-25T01:33:00.640926+0000","last_clean":"2024-04-25T01:33:00.640926+0000","last_became_active":"2024-04-25T01:30:38.378798+0000","last_became_peered":"2024-04-25T01:30:38.378798+0000","last_unstale":"2024-04-25T01:33:00.640926+0000","last_undegraded":"2024-04-25T01:33:00.640926+0000","last_fullsized":"2024-04-25T01:33:00.640926+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:40.192239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'8","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.972284+0000","last_change":"2024-04-25T01:30:33.343232+0000","last_active":"2024-04-25T01:33:00.972284+0000","last_peered":"2024-04-25T01:33:00.972284+0000","last_clean":"2024-04-25T01:33:00.972284+0000","last_became_active":"2024-04-25T01:30:33.342876+0000","last_became_peered":"2024-04-25T01:30:33.342876+0000","last_unstale":"2024-04-25T01:33:00.972284+0000","last_undegraded":"2024-04-25T01:33:00.972284+0000","last_fullsized":"2024-04-25T01:33:00.972284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:45.032564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1678,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":125,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641248+0000","last_change":"2024-04-25T01:30:34.358210+0000","last_active":"2024-04-25T01:33:00.641248+0000","last_peered":"2024-04-25T01:33:00.641248+0000","last_clean":"2024-04-25T01:33:00.641248+0000","last_became_active":"2024-04-25T01:30:34.357862+0000","last_became_peered":"2024-04-25T01:30:34.357862+0000","last_unstale":"2024-04-25T01:33:00.641248+0000","last_undegraded":"2024-04-25T01:33:00.641248+0000","last_fullsized":"2024-04-25T01:33:00.641248+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:43.751361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643538+0000","last_change":"2024-04-25T01:30:38.380413+0000","last_active":"2024-04-25T01:33:00.643538+0000","last_peered":"2024-04-25T01:33:00.643538+0000","last_clean":"2024-04-25T01:33:00.643538+0000","last_became_active":"2024-04-25T01:30:38.379612+0000","last_became_peered":"2024-04-25T01:30:38.379612+0000","last_unstale":"2024-04-25T01:33:00.643538+0000","last_undegraded":"2024-04-25T01:33:00.643538+0000","last_fullsized":"2024-04-25T01:33:00.643538+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:45.328642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'17","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730348+0000","last_change":"2024-04-25T01:30:33.344938+0000","last_active":"2024-04-25T01:33:00.730348+0000","last_peered":"2024-04-25T01:33:00.730348+0000","last_clean":"2024-04-25T01:33:00.730348+0000","last_became_active":"2024-04-25T01:30:33.344582+0000","last_became_peered":"2024-04-25T01:30:33.344582+0000","last_unstale":"2024-04-25T01:33:00.730348+0000","last_undegraded":"2024-04-25T01:33:00.730348+0000","last_fullsized":"2024-04-25T01:33:00.730348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:43:33.777319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641031+0000","last_change":"2024-04-25T01:30:34.354115+0000","last_active":"2024-04-25T01:33:00.641031+0000","last_peered":"2024-04-25T01:33:00.641031+0000","last_clean":"2024-04-25T01:33:00.641031+0000","last_became_active":"2024-04-25T01:30:34.353722+0000","last_became_peered":"2024-04-25T01:30:34.353722+0000","last_unstale":"2024-04-25T01:33:00.641031+0000","last_undegraded":"2024-04-25T01:33:00.641031+0000","last_fullsized":"2024-04-25T01:33:00.641031+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:22.003521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641136+0000","last_change":"2024-04-25T01:30:38.380914+0000","last_active":"2024-04-25T01:33:00.641136+0000","last_peered":"2024-04-25T01:33:00.641136+0000","last_clean":"2024-04-25T01:33:00.641136+0000","last_became_active":"2024-04-25T01:30:38.380477+0000","last_became_peered":"2024-04-25T01:30:38.380477+0000","last_unstale":"2024-04-25T01:33:00.641136+0000","last_undegraded":"2024-04-25T01:33:00.641136+0000","last_fullsized":"2024-04-25T01:33:00.641136+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:23:27.882266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730890+0000","last_change":"2024-04-25T01:30:33.345772+0000","last_active":"2024-04-25T01:33:00.730890+0000","last_peered":"2024-04-25T01:33:00.730890+0000","last_clean":"2024-04-25T01:33:00.730890+0000","last_became_active":"2024-04-25T01:30:33.345455+0000","last_became_peered":"2024-04-25T01:30:33.345455+0000","last_unstale":"2024-04-25T01:33:00.730890+0000","last_undegraded":"2024-04-25T01:33:00.730890+0000","last_fullsized":"2024-04-25T01:33:00.730890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:14:57.465161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641257+0000","last_change":"2024-04-25T01:30:34.358665+0000","last_active":"2024-04-25T01:33:00.641257+0000","last_peered":"2024-04-25T01:33:00.641257+0000","last_clean":"2024-04-25T01:33:00.641257+0000","last_became_active":"2024-04-25T01:30:34.358284+0000","last_became_peered":"2024-04-25T01:30:34.358284+0000","last_unstale":"2024-04-25T01:33:00.641257+0000","last_undegraded":"2024-04-25T01:33:00.641257+0000","last_fullsized":"2024-04-25T01:33:00.641257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:12:22.832265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730394+0000","last_change":"2024-04-25T01:30:38.450027+0000","last_active":"2024-04-25T01:33:00.730394+0000","last_peered":"2024-04-25T01:33:00.730394+0000","last_clean":"2024-04-25T01:33:00.730394+0000","last_became_active":"2024-04-25T01:30:38.449564+0000","last_became_peered":"2024-04-25T01:30:38.449564+0000","last_unstale":"2024-04-25T01:33:00.730394+0000","last_undegraded":"2024-04-25T01:33:00.730394+0000","last_fullsized":"2024-04-25T01:33:00.730394+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:34:26.542351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"91'1","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641180+0000","last_change":"2024-04-25T01:30:33.349757+0000","last_active":"2024-04-25T01:33:00.641180+0000","last_peered":"2024-04-25T01:33:00.641180+0000","last_clean":"2024-04-25T01:33:00.641180+0000","last_became_active":"2024-04-25T01:30:33.349439+0000","last_became_peered":"2024-04-25T01:30:33.349439+0000","last_unstale":"2024-04-25T01:33:00.641180+0000","last_undegraded":"2024-04-25T01:33:00.641180+0000","last_fullsized":"2024-04-25T01:33:00.641180+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:35.514160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.623885+0000","last_change":"2024-04-25T01:30:34.367310+0000","last_active":"2024-04-25T01:33:00.623885+0000","last_peered":"2024-04-25T01:33:00.623885+0000","last_clean":"2024-04-25T01:33:00.623885+0000","last_became_active":"2024-04-25T01:30:34.366950+0000","last_became_peered":"2024-04-25T01:30:34.366950+0000","last_unstale":"2024-04-25T01:33:00.623885+0000","last_undegraded":"2024-04-25T01:33:00.623885+0000","last_fullsized":"2024-04-25T01:33:00.623885+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:26:17.700512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642919+0000","last_change":"2024-04-25T01:30:38.372259+0000","last_active":"2024-04-25T01:33:00.642919+0000","last_peered":"2024-04-25T01:33:00.642919+0000","last_clean":"2024-04-25T01:33:00.642919+0000","last_became_active":"2024-04-25T01:30:38.371847+0000","last_became_peered":"2024-04-25T01:30:38.371847+0000","last_unstale":"2024-04-25T01:33:00.642919+0000","last_undegraded":"2024-04-25T01:33:00.642919+0000","last_fullsized":"2024-04-25T01:33:00.642919+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:55.047739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'5","reported_seq":156,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642256+0000","last_change":"2024-04-25T01:30:33.352078+0000","last_active":"2024-04-25T01:33:00.642256+0000","last_peered":"2024-04-25T01:33:00.642256+0000","last_clean":"2024-04-25T01:33:00.642256+0000","last_became_active":"2024-04-25T01:30:33.351725+0000","last_became_peered":"2024-04-25T01:30:33.351725+0000","last_unstale":"2024-04-25T01:33:00.642256+0000","last_undegraded":"2024-04-25T01:33:00.642256+0000","last_fullsized":"2024-04-25T01:33:00.642256+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:55:55.236040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.623632+0000","last_change":"2024-04-25T01:30:34.360324+0000","last_active":"2024-04-25T01:33:00.623632+0000","last_peered":"2024-04-25T01:33:00.623632+0000","last_clean":"2024-04-25T01:33:00.623632+0000","last_became_active":"2024-04-25T01:30:34.359114+0000","last_became_peered":"2024-04-25T01:30:34.359114+0000","last_unstale":"2024-04-25T01:33:00.623632+0000","last_undegraded":"2024-04-25T01:33:00.623632+0000","last_fullsized":"2024-04-25T01:33:00.623632+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:51.927640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643419+0000","last_change":"2024-04-25T01:30:38.450650+0000","last_active":"2024-04-25T01:33:00.643419+0000","last_peered":"2024-04-25T01:33:00.643419+0000","last_clean":"2024-04-25T01:33:00.643419+0000","last_became_active":"2024-04-25T01:30:38.450149+0000","last_became_peered":"2024-04-25T01:30:38.450149+0000","last_unstale":"2024-04-25T01:33:00.643419+0000","last_undegraded":"2024-04-25T01:33:00.643419+0000","last_fullsized":"2024-04-25T01:33:00.643419+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:34.630803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641415+0000","last_change":"2024-04-25T01:30:34.352961+0000","last_active":"2024-04-25T01:33:00.641415+0000","last_peered":"2024-04-25T01:33:00.641415+0000","last_clean":"2024-04-25T01:33:00.641415+0000","last_became_active":"2024-04-25T01:30:34.352657+0000","last_became_peered":"2024-04-25T01:30:34.352657+0000","last_unstale":"2024-04-25T01:33:00.641415+0000","last_undegraded":"2024-04-25T01:33:00.641415+0000","last_fullsized":"2024-04-25T01:33:00.641415+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:20.934228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":157,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641398+0000","last_change":"2024-04-25T01:30:33.344562+0000","last_active":"2024-04-25T01:33:00.641398+0000","last_peered":"2024-04-25T01:33:00.641398+0000","last_clean":"2024-04-25T01:33:00.641398+0000","last_became_active":"2024-04-25T01:30:33.344212+0000","last_became_peered":"2024-04-25T01:30:33.344212+0000","last_unstale":"2024-04-25T01:33:00.641398+0000","last_undegraded":"2024-04-25T01:33:00.641398+0000","last_fullsized":"2024-04-25T01:33:00.641398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:20:51.110618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641120+0000","last_change":"2024-04-25T01:30:38.384316+0000","last_active":"2024-04-25T01:33:00.641120+0000","last_peered":"2024-04-25T01:33:00.641120+0000","last_clean":"2024-04-25T01:33:00.641120+0000","last_became_active":"2024-04-25T01:30:38.383775+0000","last_became_peered":"2024-04-25T01:30:38.383775+0000","last_unstale":"2024-04-25T01:33:00.641120+0000","last_undegraded":"2024-04-25T01:33:00.641120+0000","last_fullsized":"2024-04-25T01:33:00.641120+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:54.165691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640600+0000","last_change":"2024-04-25T01:31:22.050014+0000","last_active":"2024-04-25T01:33:00.640600+0000","last_peered":"2024-04-25T01:33:00.640600+0000","last_clean":"2024-04-25T01:33:00.640600+0000","last_became_active":"2024-04-25T01:31:22.049718+0000","last_became_peered":"2024-04-25T01:31:22.049718+0000","last_unstale":"2024-04-25T01:33:00.640600+0000","last_undegraded":"2024-04-25T01:33:00.640600+0000","last_fullsized":"2024-04-25T01:33:00.640600+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:40:22.233372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"91'6","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640866+0000","last_change":"2024-04-25T01:30:33.345283+0000","last_active":"2024-04-25T01:33:00.640866+0000","last_peered":"2024-04-25T01:33:00.640866+0000","last_clean":"2024-04-25T01:33:00.640866+0000","last_became_active":"2024-04-25T01:30:33.344903+0000","last_became_peered":"2024-04-25T01:30:33.344903+0000","last_unstale":"2024-04-25T01:33:00.640866+0000","last_undegraded":"2024-04-25T01:33:00.640866+0000","last_fullsized":"2024-04-25T01:33:00.640866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:57:01.982850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":366,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640929+0000","last_change":"2024-04-25T01:29:17.689639+0000","last_active":"2024-04-25T01:33:00.640929+0000","last_peered":"2024-04-25T01:33:00.640929+0000","last_clean":"2024-04-25T01:33:00.640929+0000","last_became_active":"2024-04-25T01:29:17.689046+0000","last_became_peered":"2024-04-25T01:29:17.689046+0000","last_unstale":"2024-04-25T01:33:00.640929+0000","last_undegraded":"2024-04-25T01:33:00.640929+0000","last_fullsized":"2024-04-25T01:33:00.640929+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:48.767945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:48.767945+0000","last_clean_scrub_stamp":"2024-04-25T01:25:48.767945+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:04:06.092632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641924+0000","last_change":"2024-04-25T01:30:38.450938+0000","last_active":"2024-04-25T01:33:00.641924+0000","last_peered":"2024-04-25T01:33:00.641924+0000","last_clean":"2024-04-25T01:33:00.641924+0000","last_became_active":"2024-04-25T01:30:38.450488+0000","last_became_peered":"2024-04-25T01:30:38.450488+0000","last_unstale":"2024-04-25T01:33:00.641924+0000","last_undegraded":"2024-04-25T01:33:00.641924+0000","last_fullsized":"2024-04-25T01:33:00.641924+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:44.778133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641178+0000","last_change":"2024-04-25T01:31:21.875719+0000","last_active":"2024-04-25T01:33:00.641178+0000","last_peered":"2024-04-25T01:33:00.641178+0000","last_clean":"2024-04-25T01:33:00.641178+0000","last_became_active":"2024-04-25T01:31:21.875283+0000","last_became_peered":"2024-04-25T01:31:21.875283+0000","last_unstale":"2024-04-25T01:33:00.641178+0000","last_undegraded":"2024-04-25T01:33:00.641178+0000","last_fullsized":"2024-04-25T01:33:00.641178+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:12:39.097019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641047+0000","last_change":"2024-04-25T01:30:33.348198+0000","last_active":"2024-04-25T01:33:00.641047+0000","last_peered":"2024-04-25T01:33:00.641047+0000","last_clean":"2024-04-25T01:33:00.641047+0000","last_became_active":"2024-04-25T01:30:33.347876+0000","last_became_peered":"2024-04-25T01:30:33.347876+0000","last_unstale":"2024-04-25T01:33:00.641047+0000","last_undegraded":"2024-04-25T01:33:00.641047+0000","last_fullsized":"2024-04-25T01:33:00.641047+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:48:33.116571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641704+0000","last_change":"2024-04-25T01:30:38.372331+0000","last_active":"2024-04-25T01:33:00.641704+0000","last_peered":"2024-04-25T01:33:00.641704+0000","last_clean":"2024-04-25T01:33:00.641704+0000","last_became_active":"2024-04-25T01:30:38.371923+0000","last_became_peered":"2024-04-25T01:30:38.371923+0000","last_unstale":"2024-04-25T01:33:00.641704+0000","last_undegraded":"2024-04-25T01:33:00.641704+0000","last_fullsized":"2024-04-25T01:33:00.641704+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:06.416967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730656+0000","last_change":"2024-04-25T01:30:33.346606+0000","last_active":"2024-04-25T01:33:00.730656+0000","last_peered":"2024-04-25T01:33:00.730656+0000","last_clean":"2024-04-25T01:33:00.730656+0000","last_became_active":"2024-04-25T01:30:33.346297+0000","last_became_peered":"2024-04-25T01:30:33.346297+0000","last_unstale":"2024-04-25T01:33:00.730656+0000","last_undegraded":"2024-04-25T01:33:00.730656+0000","last_fullsized":"2024-04-25T01:33:00.730656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:07:16.069285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643021+0000","last_change":"2024-04-25T01:30:34.349136+0000","last_active":"2024-04-25T01:33:00.643021+0000","last_peered":"2024-04-25T01:33:00.643021+0000","last_clean":"2024-04-25T01:33:00.643021+0000","last_became_active":"2024-04-25T01:30:34.348810+0000","last_became_peered":"2024-04-25T01:30:34.348810+0000","last_unstale":"2024-04-25T01:33:00.643021+0000","last_undegraded":"2024-04-25T01:33:00.643021+0000","last_fullsized":"2024-04-25T01:33:00.643021+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:39:34.593986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.621999+0000","last_change":"2024-04-25T01:30:38.450428+0000","last_active":"2024-04-25T01:33:00.621999+0000","last_peered":"2024-04-25T01:33:00.621999+0000","last_clean":"2024-04-25T01:33:00.621999+0000","last_became_active":"2024-04-25T01:30:38.449971+0000","last_became_peered":"2024-04-25T01:30:38.449971+0000","last_unstale":"2024-04-25T01:33:00.621999+0000","last_undegraded":"2024-04-25T01:33:00.621999+0000","last_fullsized":"2024-04-25T01:33:00.621999+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:52:37.046462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'47","reported_seq":100,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.847942+0000","last_change":"2024-04-25T01:30:33.348547+0000","last_active":"2024-04-25T01:33:00.847942+0000","last_peered":"2024-04-25T01:33:00.847942+0000","last_clean":"2024-04-25T01:33:00.847942+0000","last_became_active":"2024-04-25T01:30:33.348181+0000","last_became_peered":"2024-04-25T01:30:33.348181+0000","last_unstale":"2024-04-25T01:33:00.847942+0000","last_undegraded":"2024-04-25T01:33:00.847942+0000","last_fullsized":"2024-04-25T01:33:00.847942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:11:58.696829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":82,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622499+0000","last_change":"2024-04-25T01:30:34.351987+0000","last_active":"2024-04-25T01:33:00.622499+0000","last_peered":"2024-04-25T01:33:00.622499+0000","last_clean":"2024-04-25T01:33:00.622499+0000","last_became_active":"2024-04-25T01:30:34.351571+0000","last_became_peered":"2024-04-25T01:30:34.351571+0000","last_unstale":"2024-04-25T01:33:00.622499+0000","last_undegraded":"2024-04-25T01:33:00.622499+0000","last_fullsized":"2024-04-25T01:33:00.622499+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:23.585865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.625691+0000","last_change":"2024-04-25T01:30:38.372566+0000","last_active":"2024-04-25T01:33:00.625691+0000","last_peered":"2024-04-25T01:33:00.625691+0000","last_clean":"2024-04-25T01:33:00.625691+0000","last_became_active":"2024-04-25T01:30:38.372102+0000","last_became_peered":"2024-04-25T01:30:38.372102+0000","last_unstale":"2024-04-25T01:33:00.625691+0000","last_undegraded":"2024-04-25T01:33:00.625691+0000","last_fullsized":"2024-04-25T01:33:00.625691+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:22:16.791842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642090+0000","last_change":"2024-04-25T01:30:33.344065+0000","last_active":"2024-04-25T01:33:00.642090+0000","last_peered":"2024-04-25T01:33:00.642090+0000","last_clean":"2024-04-25T01:33:00.642090+0000","last_became_active":"2024-04-25T01:30:33.343700+0000","last_became_peered":"2024-04-25T01:30:33.343700+0000","last_unstale":"2024-04-25T01:33:00.642090+0000","last_undegraded":"2024-04-25T01:33:00.642090+0000","last_fullsized":"2024-04-25T01:33:00.642090+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:54:08.265744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643124+0000","last_change":"2024-04-25T01:30:34.354644+0000","last_active":"2024-04-25T01:33:00.643124+0000","last_peered":"2024-04-25T01:33:00.643124+0000","last_clean":"2024-04-25T01:33:00.643124+0000","last_became_active":"2024-04-25T01:30:34.354330+0000","last_became_peered":"2024-04-25T01:30:34.354330+0000","last_unstale":"2024-04-25T01:33:00.643124+0000","last_undegraded":"2024-04-25T01:33:00.643124+0000","last_fullsized":"2024-04-25T01:33:00.643124+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:31:50.399983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641601+0000","last_change":"2024-04-25T01:30:38.450743+0000","last_active":"2024-04-25T01:33:00.641601+0000","last_peered":"2024-04-25T01:33:00.641601+0000","last_clean":"2024-04-25T01:33:00.641601+0000","last_became_active":"2024-04-25T01:30:38.450303+0000","last_became_peered":"2024-04-25T01:30:38.450303+0000","last_unstale":"2024-04-25T01:33:00.641601+0000","last_undegraded":"2024-04-25T01:33:00.641601+0000","last_fullsized":"2024-04-25T01:33:00.641601+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:37.919250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640527+0000","last_change":"2024-04-25T01:30:33.350709+0000","last_active":"2024-04-25T01:33:00.640527+0000","last_peered":"2024-04-25T01:33:00.640527+0000","last_clean":"2024-04-25T01:33:00.640527+0000","last_became_active":"2024-04-25T01:30:33.350387+0000","last_became_peered":"2024-04-25T01:30:33.350387+0000","last_unstale":"2024-04-25T01:33:00.640527+0000","last_undegraded":"2024-04-25T01:33:00.640527+0000","last_fullsized":"2024-04-25T01:33:00.640527+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:30:01.252121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730287+0000","last_change":"2024-04-25T01:30:34.362040+0000","last_active":"2024-04-25T01:33:00.730287+0000","last_peered":"2024-04-25T01:33:00.730287+0000","last_clean":"2024-04-25T01:33:00.730287+0000","last_became_active":"2024-04-25T01:30:34.361734+0000","last_became_peered":"2024-04-25T01:30:34.361734+0000","last_unstale":"2024-04-25T01:33:00.730287+0000","last_undegraded":"2024-04-25T01:33:00.730287+0000","last_fullsized":"2024-04-25T01:33:00.730287+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:11:28.409764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641032+0000","last_change":"2024-04-25T01:31:21.877148+0000","last_active":"2024-04-25T01:33:00.641032+0000","last_peered":"2024-04-25T01:33:00.641032+0000","last_clean":"2024-04-25T01:33:00.641032+0000","last_became_active":"2024-04-25T01:31:21.876665+0000","last_became_peered":"2024-04-25T01:31:21.876665+0000","last_unstale":"2024-04-25T01:33:00.641032+0000","last_undegraded":"2024-04-25T01:33:00.641032+0000","last_fullsized":"2024-04-25T01:33:00.641032+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:32:38.425434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,8],"acting":[9,3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622123+0000","last_change":"2024-04-25T01:30:33.350207+0000","last_active":"2024-04-25T01:33:00.622123+0000","last_peered":"2024-04-25T01:33:00.622123+0000","last_clean":"2024-04-25T01:33:00.622123+0000","last_became_active":"2024-04-25T01:30:33.349892+0000","last_became_peered":"2024-04-25T01:30:33.349892+0000","last_unstale":"2024-04-25T01:33:00.622123+0000","last_undegraded":"2024-04-25T01:33:00.622123+0000","last_fullsized":"2024-04-25T01:33:00.622123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:26:35.186486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640590+0000","last_change":"2024-04-25T01:30:34.355196+0000","last_active":"2024-04-25T01:33:00.640590+0000","last_peered":"2024-04-25T01:33:00.640590+0000","last_clean":"2024-04-25T01:33:00.640590+0000","last_became_active":"2024-04-25T01:30:34.354794+0000","last_became_peered":"2024-04-25T01:30:34.354794+0000","last_unstale":"2024-04-25T01:33:00.640590+0000","last_undegraded":"2024-04-25T01:33:00.640590+0000","last_fullsized":"2024-04-25T01:33:00.640590+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:43:37.495974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641714+0000","last_change":"2024-04-25T01:30:38.384159+0000","last_active":"2024-04-25T01:33:00.641714+0000","last_peered":"2024-04-25T01:33:00.641714+0000","last_clean":"2024-04-25T01:33:00.641714+0000","last_became_active":"2024-04-25T01:30:38.383676+0000","last_became_peered":"2024-04-25T01:30:38.383676+0000","last_unstale":"2024-04-25T01:33:00.641714+0000","last_undegraded":"2024-04-25T01:33:00.641714+0000","last_fullsized":"2024-04-25T01:33:00.641714+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:28.491522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":19,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640644+0000","last_change":"2024-04-25T01:31:21.889311+0000","last_active":"2024-04-25T01:33:00.640644+0000","last_peered":"2024-04-25T01:33:00.640644+0000","last_clean":"2024-04-25T01:33:00.640644+0000","last_became_active":"2024-04-25T01:31:21.877027+0000","last_became_peered":"2024-04-25T01:31:21.877027+0000","last_unstale":"2024-04-25T01:33:00.640644+0000","last_undegraded":"2024-04-25T01:33:00.640644+0000","last_fullsized":"2024-04-25T01:33:00.640644+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:02.206671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622555+0000","last_change":"2024-04-25T01:30:34.358110+0000","last_active":"2024-04-25T01:33:00.622555+0000","last_peered":"2024-04-25T01:33:00.622555+0000","last_clean":"2024-04-25T01:33:00.622555+0000","last_became_active":"2024-04-25T01:30:34.357555+0000","last_became_peered":"2024-04-25T01:30:34.357555+0000","last_unstale":"2024-04-25T01:33:00.622555+0000","last_undegraded":"2024-04-25T01:33:00.622555+0000","last_fullsized":"2024-04-25T01:33:00.622555+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:11.636320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641877+0000","last_change":"2024-04-25T01:30:38.450680+0000","last_active":"2024-04-25T01:33:00.641877+0000","last_peered":"2024-04-25T01:33:00.641877+0000","last_clean":"2024-04-25T01:33:00.641877+0000","last_became_active":"2024-04-25T01:30:38.450168+0000","last_became_peered":"2024-04-25T01:30:38.450168+0000","last_unstale":"2024-04-25T01:33:00.641877+0000","last_undegraded":"2024-04-25T01:33:00.641877+0000","last_fullsized":"2024-04-25T01:33:00.641877+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:39:24.236371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730493+0000","last_change":"2024-04-25T01:30:33.343848+0000","last_active":"2024-04-25T01:33:00.730493+0000","last_peered":"2024-04-25T01:33:00.730493+0000","last_clean":"2024-04-25T01:33:00.730493+0000","last_became_active":"2024-04-25T01:30:33.343498+0000","last_became_peered":"2024-04-25T01:30:33.343498+0000","last_unstale":"2024-04-25T01:33:00.730493+0000","last_undegraded":"2024-04-25T01:33:00.730493+0000","last_fullsized":"2024-04-25T01:33:00.730493+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:58:09.184571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642308+0000","last_change":"2024-04-25T01:30:34.361994+0000","last_active":"2024-04-25T01:33:00.642308+0000","last_peered":"2024-04-25T01:33:00.642308+0000","last_clean":"2024-04-25T01:33:00.642308+0000","last_became_active":"2024-04-25T01:30:34.361532+0000","last_became_peered":"2024-04-25T01:30:34.361532+0000","last_unstale":"2024-04-25T01:33:00.642308+0000","last_undegraded":"2024-04-25T01:33:00.642308+0000","last_fullsized":"2024-04-25T01:33:00.642308+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:30.667672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640976+0000","last_change":"2024-04-25T01:30:38.369041+0000","last_active":"2024-04-25T01:33:00.640976+0000","last_peered":"2024-04-25T01:33:00.640976+0000","last_clean":"2024-04-25T01:33:00.640976+0000","last_became_active":"2024-04-25T01:30:38.368586+0000","last_became_peered":"2024-04-25T01:30:38.368586+0000","last_unstale":"2024-04-25T01:33:00.640976+0000","last_undegraded":"2024-04-25T01:33:00.640976+0000","last_fullsized":"2024-04-25T01:33:00.640976+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:58:53.743012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'7","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.626422+0000","last_change":"2024-04-25T01:30:33.345501+0000","last_active":"2024-04-25T01:33:00.626422+0000","last_peered":"2024-04-25T01:33:00.626422+0000","last_clean":"2024-04-25T01:33:00.626422+0000","last_became_active":"2024-04-25T01:30:33.345179+0000","last_became_peered":"2024-04-25T01:30:33.345179+0000","last_unstale":"2024-04-25T01:33:00.626422+0000","last_undegraded":"2024-04-25T01:33:00.626422+0000","last_fullsized":"2024-04-25T01:33:00.626422+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:29:41.275480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642445+0000","last_change":"2024-04-25T01:30:34.361631+0000","last_active":"2024-04-25T01:33:00.642445+0000","last_peered":"2024-04-25T01:33:00.642445+0000","last_clean":"2024-04-25T01:33:00.642445+0000","last_became_active":"2024-04-25T01:30:34.360266+0000","last_became_peered":"2024-04-25T01:30:34.360266+0000","last_unstale":"2024-04-25T01:33:00.642445+0000","last_undegraded":"2024-04-25T01:33:00.642445+0000","last_fullsized":"2024-04-25T01:33:00.642445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:03.094166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642246+0000","last_change":"2024-04-25T01:30:38.380779+0000","last_active":"2024-04-25T01:33:00.642246+0000","last_peered":"2024-04-25T01:33:00.642246+0000","last_clean":"2024-04-25T01:33:00.642246+0000","last_became_active":"2024-04-25T01:30:38.379959+0000","last_became_peered":"2024-04-25T01:30:38.379959+0000","last_unstale":"2024-04-25T01:33:00.642246+0000","last_undegraded":"2024-04-25T01:33:00.642246+0000","last_fullsized":"2024-04-25T01:33:00.642246+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:48:01.816311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'41","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.845488+0000","last_change":"2024-04-25T01:30:33.345730+0000","last_active":"2024-04-25T01:33:00.845488+0000","last_peered":"2024-04-25T01:33:00.845488+0000","last_clean":"2024-04-25T01:33:00.845488+0000","last_became_active":"2024-04-25T01:30:33.344832+0000","last_became_peered":"2024-04-25T01:30:33.344832+0000","last_unstale":"2024-04-25T01:33:00.845488+0000","last_undegraded":"2024-04-25T01:33:00.845488+0000","last_fullsized":"2024-04-25T01:33:00.845488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:47.018811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642105+0000","last_change":"2024-04-25T01:30:34.355324+0000","last_active":"2024-04-25T01:33:00.642105+0000","last_peered":"2024-04-25T01:33:00.642105+0000","last_clean":"2024-04-25T01:33:00.642105+0000","last_became_active":"2024-04-25T01:30:34.354954+0000","last_became_peered":"2024-04-25T01:30:34.354954+0000","last_unstale":"2024-04-25T01:33:00.642105+0000","last_undegraded":"2024-04-25T01:33:00.642105+0000","last_fullsized":"2024-04-25T01:33:00.642105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:36:41.932391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640928+0000","last_change":"2024-04-25T01:30:38.374386+0000","last_active":"2024-04-25T01:33:00.640928+0000","last_peered":"2024-04-25T01:33:00.640928+0000","last_clean":"2024-04-25T01:33:00.640928+0000","last_became_active":"2024-04-25T01:30:38.373454+0000","last_became_peered":"2024-04-25T01:30:38.373454+0000","last_unstale":"2024-04-25T01:33:00.640928+0000","last_undegraded":"2024-04-25T01:33:00.640928+0000","last_fullsized":"2024-04-25T01:33:00.640928+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:54.456218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"91'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643371+0000","last_change":"2024-04-25T01:30:33.339821+0000","last_active":"2024-04-25T01:33:00.643371+0000","last_peered":"2024-04-25T01:33:00.643371+0000","last_clean":"2024-04-25T01:33:00.643371+0000","last_became_active":"2024-04-25T01:30:33.339500+0000","last_became_peered":"2024-04-25T01:30:33.339500+0000","last_unstale":"2024-04-25T01:33:00.643371+0000","last_undegraded":"2024-04-25T01:33:00.643371+0000","last_fullsized":"2024-04-25T01:33:00.643371+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:44:25.880828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642426+0000","last_change":"2024-04-25T01:30:34.362736+0000","last_active":"2024-04-25T01:33:00.642426+0000","last_peered":"2024-04-25T01:33:00.642426+0000","last_clean":"2024-04-25T01:33:00.642426+0000","last_became_active":"2024-04-25T01:30:34.362371+0000","last_became_peered":"2024-04-25T01:30:34.362371+0000","last_unstale":"2024-04-25T01:33:00.642426+0000","last_undegraded":"2024-04-25T01:33:00.642426+0000","last_fullsized":"2024-04-25T01:33:00.642426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:57:36.088231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640673+0000","last_change":"2024-04-25T01:30:34.359217+0000","last_active":"2024-04-25T01:33:00.640673+0000","last_peered":"2024-04-25T01:33:00.640673+0000","last_clean":"2024-04-25T01:33:00.640673+0000","last_became_active":"2024-04-25T01:30:34.357095+0000","last_became_peered":"2024-04-25T01:30:34.357095+0000","last_unstale":"2024-04-25T01:33:00.640673+0000","last_undegraded":"2024-04-25T01:33:00.640673+0000","last_fullsized":"2024-04-25T01:33:00.640673+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:58.603484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.625896+0000","last_change":"2024-04-25T01:30:34.359513+0000","last_active":"2024-04-25T01:33:00.625896+0000","last_peered":"2024-04-25T01:33:00.625896+0000","last_clean":"2024-04-25T01:33:00.625896+0000","last_became_active":"2024-04-25T01:30:34.359089+0000","last_became_peered":"2024-04-25T01:30:34.359089+0000","last_unstale":"2024-04-25T01:33:00.625896+0000","last_undegraded":"2024-04-25T01:33:00.625896+0000","last_fullsized":"2024-04-25T01:33:00.625896+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:36.159703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.625752+0000","last_change":"2024-04-25T01:30:34.368739+0000","last_active":"2024-04-25T01:33:00.625752+0000","last_peered":"2024-04-25T01:33:00.625752+0000","last_clean":"2024-04-25T01:33:00.625752+0000","last_became_active":"2024-04-25T01:30:34.368392+0000","last_became_peered":"2024-04-25T01:30:34.368392+0000","last_unstale":"2024-04-25T01:33:00.625752+0000","last_undegraded":"2024-04-25T01:33:00.625752+0000","last_fullsized":"2024-04-25T01:33:00.625752+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:39:32.241889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642467+0000","last_change":"2024-04-25T01:31:21.877568+0000","last_active":"2024-04-25T01:33:00.642467+0000","last_peered":"2024-04-25T01:33:00.642467+0000","last_clean":"2024-04-25T01:33:00.642467+0000","last_became_active":"2024-04-25T01:31:21.877071+0000","last_became_peered":"2024-04-25T01:31:21.877071+0000","last_unstale":"2024-04-25T01:33:00.642467+0000","last_undegraded":"2024-04-25T01:33:00.642467+0000","last_fullsized":"2024-04-25T01:33:00.642467+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:02:24.789974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643046+0000","last_change":"2024-04-25T01:30:34.360979+0000","last_active":"2024-04-25T01:33:00.643046+0000","last_peered":"2024-04-25T01:33:00.643046+0000","last_clean":"2024-04-25T01:33:00.643046+0000","last_became_active":"2024-04-25T01:30:34.360630+0000","last_became_peered":"2024-04-25T01:30:34.360630+0000","last_unstale":"2024-04-25T01:33:00.643046+0000","last_undegraded":"2024-04-25T01:33:00.643046+0000","last_fullsized":"2024-04-25T01:33:00.643046+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:05:19.284577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641137+0000","last_change":"2024-04-25T01:30:34.362621+0000","last_active":"2024-04-25T01:33:00.641137+0000","last_peered":"2024-04-25T01:33:00.641137+0000","last_clean":"2024-04-25T01:33:00.641137+0000","last_became_active":"2024-04-25T01:30:34.362290+0000","last_became_peered":"2024-04-25T01:30:34.362290+0000","last_unstale":"2024-04-25T01:33:00.641137+0000","last_undegraded":"2024-04-25T01:33:00.641137+0000","last_fullsized":"2024-04-25T01:33:00.641137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:28:07.750906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.626403+0000","last_change":"2024-04-25T01:30:34.351339+0000","last_active":"2024-04-25T01:33:00.626403+0000","last_peered":"2024-04-25T01:33:00.626403+0000","last_clean":"2024-04-25T01:33:00.626403+0000","last_became_active":"2024-04-25T01:30:34.350655+0000","last_became_peered":"2024-04-25T01:30:34.350655+0000","last_unstale":"2024-04-25T01:33:00.626403+0000","last_undegraded":"2024-04-25T01:33:00.626403+0000","last_fullsized":"2024-04-25T01:33:00.626403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:05.322547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622618+0000","last_change":"2024-04-25T01:30:34.353876+0000","last_active":"2024-04-25T01:33:00.622618+0000","last_peered":"2024-04-25T01:33:00.622618+0000","last_clean":"2024-04-25T01:33:00.622618+0000","last_became_active":"2024-04-25T01:30:34.352942+0000","last_became_peered":"2024-04-25T01:30:34.352942+0000","last_unstale":"2024-04-25T01:33:00.622618+0000","last_undegraded":"2024-04-25T01:33:00.622618+0000","last_fullsized":"2024-04-25T01:33:00.622618+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:47.260171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641358+0000","last_change":"2024-04-25T01:30:34.352039+0000","last_active":"2024-04-25T01:33:00.641358+0000","last_peered":"2024-04-25T01:33:00.641358+0000","last_clean":"2024-04-25T01:33:00.641358+0000","last_became_active":"2024-04-25T01:30:34.351612+0000","last_became_peered":"2024-04-25T01:30:34.351612+0000","last_unstale":"2024-04-25T01:33:00.641358+0000","last_undegraded":"2024-04-25T01:33:00.641358+0000","last_fullsized":"2024-04-25T01:33:00.641358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:03.607711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622510+0000","last_change":"2024-04-25T01:30:34.353692+0000","last_active":"2024-04-25T01:33:00.622510+0000","last_peered":"2024-04-25T01:33:00.622510+0000","last_clean":"2024-04-25T01:33:00.622510+0000","last_became_active":"2024-04-25T01:30:34.353035+0000","last_became_peered":"2024-04-25T01:30:34.353035+0000","last_unstale":"2024-04-25T01:33:00.622510+0000","last_undegraded":"2024-04-25T01:33:00.622510+0000","last_fullsized":"2024-04-25T01:33:00.622510+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:15:54.540315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641104+0000","last_change":"2024-04-25T01:30:34.370797+0000","last_active":"2024-04-25T01:33:00.641104+0000","last_peered":"2024-04-25T01:33:00.641104+0000","last_clean":"2024-04-25T01:33:00.641104+0000","last_became_active":"2024-04-25T01:30:34.367533+0000","last_became_peered":"2024-04-25T01:30:34.367533+0000","last_unstale":"2024-04-25T01:33:00.641104+0000","last_undegraded":"2024-04-25T01:33:00.641104+0000","last_fullsized":"2024-04-25T01:33:00.641104+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:27:15.478808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643248+0000","last_change":"2024-04-25T01:30:34.364843+0000","last_active":"2024-04-25T01:33:00.643248+0000","last_peered":"2024-04-25T01:33:00.643248+0000","last_clean":"2024-04-25T01:33:00.643248+0000","last_became_active":"2024-04-25T01:30:34.364515+0000","last_became_peered":"2024-04-25T01:30:34.364515+0000","last_unstale":"2024-04-25T01:33:00.643248+0000","last_undegraded":"2024-04-25T01:33:00.643248+0000","last_fullsized":"2024-04-25T01:33:00.643248+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:50:03.517814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641005+0000","last_change":"2024-04-25T01:30:34.363715+0000","last_active":"2024-04-25T01:33:00.641005+0000","last_peered":"2024-04-25T01:33:00.641005+0000","last_clean":"2024-04-25T01:33:00.641005+0000","last_became_active":"2024-04-25T01:30:34.363383+0000","last_became_peered":"2024-04-25T01:30:34.363383+0000","last_unstale":"2024-04-25T01:33:00.641005+0000","last_undegraded":"2024-04-25T01:33:00.641005+0000","last_fullsized":"2024-04-25T01:33:00.641005+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:50:18.322550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":601378,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1337,"num_read_kb":786,"num_write":373,"num_write_kb":1470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1804134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":303,"ondisk_log_size":303,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318314,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1077248,"data_stored":519949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579951,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37196,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707956,"statfs":{"total":95995035648,"available":95956946944,"internally_reserved":0,"allocated":1904640,"data_stored":1306841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743477,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1490944,"data_stored":916591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005114,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":509849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266751,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28668,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716484,"statfs":{"total":95995035648,"available":95965679616,"internally_reserved":0,"allocated":1687552,"data_stored":1116994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430277,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1069056,"data_stored":514783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1142784,"data_stored":583799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1093632,"data_stored":516771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36468,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708684,"statfs":{"total":95995035648,"available":95957692416,"internally_reserved":0,"allocated":1155072,"data_stored":585015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36908,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708244,"statfs":{"total":95995035648,"available":95957241856,"internally_reserved":0,"allocated":1609728,"data_stored":1039307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888142,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":519947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691914,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36996,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708156,"statfs":{"total":95995035648,"available":95957151744,"internally_reserved":0,"allocated":1695744,"data_stored":1111877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":70954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":134952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":394716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":202412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:33:03.415 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-25T01:33:03.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:33:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.576+0000 7f09e46c1640 1 -- 172.21.15.28:0/4130375486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09dc101870 msgr2=0x7f09dc101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.576+0000 7f09e46c1640 1 --2- 172.21.15.28:0/4130375486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09dc101870 0x7f09dc101cf0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f09cc009900 tx=0x7f09cc02a410 comp rx=0 tx=0).stop 2024-04-25T01:33:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.577+0000 7f09e46c1640 1 -- 172.21.15.28:0/4130375486 shutdown_connections 2024-04-25T01:33:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.577+0000 7f09e46c1640 1 --2- 172.21.15.28:0/4130375486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09dc102230 0x7f09dc108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.577+0000 7f09e46c1640 1 --2- 172.21.15.28:0/4130375486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09dc101870 0x7f09dc101cf0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.577+0000 7f09e46c1640 1 --2- 172.21.15.28:0/4130375486 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09dc100680 0x7f09dc100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.577+0000 7f09e46c1640 1 -- 172.21.15.28:0/4130375486 >> 172.21.15.28:0/4130375486 conn(0x7f09dc0fbbf0 msgr2=0x7f09dc0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.577+0000 7f09e46c1640 1 -- 172.21.15.28:0/4130375486 shutdown_connections 2024-04-25T01:33:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.577+0000 7f09e46c1640 1 -- 172.21.15.28:0/4130375486 wait complete. 2024-04-25T01:33:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.577+0000 7f09e46c1640 1 Processor -- start 2024-04-25T01:33:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.577+0000 7f09e46c1640 1 -- start start 2024-04-25T01:33:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.577+0000 7f09e46c1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09dc100680 0x7f09dc104610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.577+0000 7f09e46c1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09dc101870 0x7f09dc102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.577+0000 7f09e46c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09dc102230 0x7f09dc103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.577+0000 7f09e46c1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc10c2e0 con 0x7f09dc102230 2024-04-25T01:33:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.578+0000 7f09e46c1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc10c160 con 0x7f09dc100680 2024-04-25T01:33:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.578+0000 7f09e46c1640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc10c460 con 0x7f09dc101870 2024-04-25T01:33:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.578+0000 7f09e2c5e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09dc102230 0x7f09dc103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.578+0000 7f09e1c5c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09dc101870 0x7f09dc102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.578+0000 7f09e2c5e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09dc102230 0x7f09dc103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55122/0 (socket says 172.21.15.28:55122) 2024-04-25T01:33:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.578+0000 7f09e245d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09dc100680 0x7f09dc104610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.578+0000 7f09e1c5c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09dc101870 0x7f09dc102c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.28:38146/0 (socket says 172.21.15.28:38146) 2024-04-25T01:33:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.578+0000 7f09e2c5e640 1 -- 172.21.15.28:0/624329798 learned_addr learned my addr 172.21.15.28:0/624329798 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.578+0000 7f09e245d640 1 -- 172.21.15.28:0/624329798 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09dc101870 msgr2=0x7f09dc102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.578+0000 7f09e245d640 1 --2- 172.21.15.28:0/624329798 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09dc101870 0x7f09dc102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.578+0000 7f09e245d640 1 -- 172.21.15.28:0/624329798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09dc102230 msgr2=0x7f09dc103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.578+0000 7f09e245d640 1 --2- 172.21.15.28:0/624329798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09dc102230 0x7f09dc103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.578+0000 7f09e245d640 1 -- 172.21.15.28:0/624329798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09dc103a20 con 0x7f09dc100680 2024-04-25T01:33:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.578+0000 7f09e2c5e640 1 --2- 172.21.15.28:0/624329798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09dc102230 0x7f09dc103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.578+0000 7f09e245d640 1 --2- 172.21.15.28:0/624329798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09dc100680 0x7f09dc104610 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f09dc1016d0 tx=0x7f09d800dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.578+0000 7f09d37fe640 1 -- 172.21.15.28:0/624329798 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d8017070 con 0x7f09dc100680 2024-04-25T01:33:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.578+0000 7f09e46c1640 1 -- 172.21.15.28:0/624329798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09dc1a1ba0 con 0x7f09dc100680 2024-04-25T01:33:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.578+0000 7f09d37fe640 1 -- 172.21.15.28:0/624329798 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09d8009c30 con 0x7f09dc100680 2024-04-25T01:33:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.578+0000 7f09d37fe640 1 -- 172.21.15.28:0/624329798 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d800f620 con 0x7f09dc100680 2024-04-25T01:33:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.579+0000 7f09e46c1640 1 -- 172.21.15.28:0/624329798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09dc1a20c0 con 0x7f09dc100680 2024-04-25T01:33:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.579+0000 7f09e46c1640 1 -- 172.21.15.28:0/624329798 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09dc1a2290 con 0x7f09dc100680 2024-04-25T01:33:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.581+0000 7f09d37fe640 1 -- 172.21.15.28:0/624329798 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7f09d801a070 con 0x7f09dc100680 2024-04-25T01:33:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.581+0000 7f09d37fe640 1 --2- 172.21.15.28:0/624329798 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f09c80a2d00 0x7f09c80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.581+0000 7f09e1c5c640 1 --2- 172.21.15.28:0/624329798 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f09c80a2d00 0x7f09c80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.581+0000 7f09d37fe640 1 -- 172.21.15.28:0/624329798 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f09d80cf170 con 0x7f09dc100680 2024-04-25T01:33:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.582+0000 7f09e1c5c640 1 --2- 172.21.15.28:0/624329798 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f09c80a2d00 0x7f09c80a51c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f09cc0314e0 tx=0x7f09cc000bc0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.585+0000 7f09d37fe640 1 -- 172.21.15.28:0/624329798 <== mon.2 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f09d809b3e0 con 0x7f09dc100680 2024-04-25T01:33:03.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.770+0000 7f09e46c1640 1 -- 172.21.15.28:0/624329798 --> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f09dc0fdde0 con 0x7f09c80a2d00 2024-04-25T01:33:03.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.780+0000 7f09d37fe640 1 -- 172.21.15.28:0/624329798 <== mgr.15561 v2:172.21.15.28:6800/712762371 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339435 (secure 0 0 0) 0x7f09dc0fdde0 con 0x7f09c80a2d00 2024-04-25T01:33:03.761 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:33:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T01:33:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.783+0000 7f09e46c1640 1 -- 172.21.15.28:0/624329798 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f09c80a2d00 msgr2=0x7f09c80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.783+0000 7f09e46c1640 1 --2- 172.21.15.28:0/624329798 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f09c80a2d00 0x7f09c80a51c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f09cc0314e0 tx=0x7f09cc000bc0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.783+0000 7f09e46c1640 1 -- 172.21.15.28:0/624329798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09dc100680 msgr2=0x7f09dc104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.783+0000 7f09e46c1640 1 --2- 172.21.15.28:0/624329798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09dc100680 0x7f09dc104610 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f09dc1016d0 tx=0x7f09d800dde0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.783+0000 7f09e46c1640 1 -- 172.21.15.28:0/624329798 shutdown_connections 2024-04-25T01:33:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.783+0000 7f09e46c1640 1 --2- 172.21.15.28:0/624329798 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f09c80a2d00 0x7f09c80a51c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.783+0000 7f09e46c1640 1 --2- 172.21.15.28:0/624329798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09dc102230 0x7f09dc103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.783+0000 7f09e46c1640 1 --2- 172.21.15.28:0/624329798 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f09dc101870 0x7f09dc102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.783+0000 7f09e46c1640 1 --2- 172.21.15.28:0/624329798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f09dc100680 0x7f09dc104610 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.783+0000 7f09e46c1640 1 -- 172.21.15.28:0/624329798 >> 172.21.15.28:0/624329798 conn(0x7f09dc0fbbf0 msgr2=0x7f09dc0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.784+0000 7f09e46c1640 1 -- 172.21.15.28:0/624329798 shutdown_connections 2024-04-25T01:33:03.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.784+0000 7f09e46c1640 1 -- 172.21.15.28:0/624329798 wait complete. 2024-04-25T01:33:03.777 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-04-25T01:33:02.635937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1191980,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1473,"num_read_kb":905,"num_write":556,"num_write_kb":4410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1180804,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":942,"ondisk_log_size":942,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":389848,"kb_used_data":15700,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124551976,"statfs":{"total":1151940427776,"available":1151541223424,"internally_reserved":0,"allocated":16076800,"data_stored":9241723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":382973242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.986097"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641539+0000","last_change":"2024-04-25T01:30:34.358339+0000","last_active":"2024-04-25T01:33:00.641539+0000","last_peered":"2024-04-25T01:33:00.641539+0000","last_clean":"2024-04-25T01:33:00.641539+0000","last_became_active":"2024-04-25T01:30:34.357884+0000","last_became_peered":"2024-04-25T01:30:34.357884+0000","last_unstale":"2024-04-25T01:33:00.641539+0000","last_undegraded":"2024-04-25T01:33:00.641539+0000","last_fullsized":"2024-04-25T01:33:00.641539+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:56.409080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643075+0000","last_change":"2024-04-25T01:30:34.359933+0000","last_active":"2024-04-25T01:33:00.643075+0000","last_peered":"2024-04-25T01:33:00.643075+0000","last_clean":"2024-04-25T01:33:00.643075+0000","last_became_active":"2024-04-25T01:30:34.359576+0000","last_became_peered":"2024-04-25T01:30:34.359576+0000","last_unstale":"2024-04-25T01:33:00.643075+0000","last_undegraded":"2024-04-25T01:33:00.643075+0000","last_fullsized":"2024-04-25T01:33:00.643075+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:48:47.764121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642789+0000","last_change":"2024-04-25T01:30:34.357772+0000","last_active":"2024-04-25T01:33:00.642789+0000","last_peered":"2024-04-25T01:33:00.642789+0000","last_clean":"2024-04-25T01:33:00.642789+0000","last_became_active":"2024-04-25T01:30:34.357434+0000","last_became_peered":"2024-04-25T01:30:34.357434+0000","last_unstale":"2024-04-25T01:33:00.642789+0000","last_undegraded":"2024-04-25T01:33:00.642789+0000","last_fullsized":"2024-04-25T01:33:00.642789+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:19.926180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641967+0000","last_change":"2024-04-25T01:30:34.351250+0000","last_active":"2024-04-25T01:33:00.641967+0000","last_peered":"2024-04-25T01:33:00.641967+0000","last_clean":"2024-04-25T01:33:00.641967+0000","last_became_active":"2024-04-25T01:30:34.349976+0000","last_became_peered":"2024-04-25T01:30:34.349976+0000","last_unstale":"2024-04-25T01:33:00.641967+0000","last_undegraded":"2024-04-25T01:33:00.641967+0000","last_fullsized":"2024-04-25T01:33:00.641967+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:19:51.605307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730612+0000","last_change":"2024-04-25T01:30:34.367882+0000","last_active":"2024-04-25T01:33:00.730612+0000","last_peered":"2024-04-25T01:33:00.730612+0000","last_clean":"2024-04-25T01:33:00.730612+0000","last_became_active":"2024-04-25T01:30:34.366797+0000","last_became_peered":"2024-04-25T01:30:34.366797+0000","last_unstale":"2024-04-25T01:33:00.730612+0000","last_undegraded":"2024-04-25T01:33:00.730612+0000","last_fullsized":"2024-04-25T01:33:00.730612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:35:23.439061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.623521+0000","last_change":"2024-04-25T01:30:34.352506+0000","last_active":"2024-04-25T01:33:00.623521+0000","last_peered":"2024-04-25T01:33:00.623521+0000","last_clean":"2024-04-25T01:33:00.623521+0000","last_became_active":"2024-04-25T01:30:34.352100+0000","last_became_peered":"2024-04-25T01:30:34.352100+0000","last_unstale":"2024-04-25T01:33:00.623521+0000","last_undegraded":"2024-04-25T01:33:00.623521+0000","last_fullsized":"2024-04-25T01:33:00.623521+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:38.113959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641641+0000","last_change":"2024-04-25T01:30:34.355352+0000","last_active":"2024-04-25T01:33:00.641641+0000","last_peered":"2024-04-25T01:33:00.641641+0000","last_clean":"2024-04-25T01:33:00.641641+0000","last_became_active":"2024-04-25T01:30:34.355008+0000","last_became_peered":"2024-04-25T01:30:34.355008+0000","last_unstale":"2024-04-25T01:33:00.641641+0000","last_undegraded":"2024-04-25T01:33:00.641641+0000","last_fullsized":"2024-04-25T01:33:00.641641+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:23.582742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622074+0000","last_change":"2024-04-25T01:30:34.360644+0000","last_active":"2024-04-25T01:33:00.622074+0000","last_peered":"2024-04-25T01:33:00.622074+0000","last_clean":"2024-04-25T01:33:00.622074+0000","last_became_active":"2024-04-25T01:30:34.357841+0000","last_became_peered":"2024-04-25T01:30:34.357841+0000","last_unstale":"2024-04-25T01:33:00.622074+0000","last_undegraded":"2024-04-25T01:33:00.622074+0000","last_fullsized":"2024-04-25T01:33:00.622074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:26:21.606219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642196+0000","last_change":"2024-04-25T01:30:34.358417+0000","last_active":"2024-04-25T01:33:00.642196+0000","last_peered":"2024-04-25T01:33:00.642196+0000","last_clean":"2024-04-25T01:33:00.642196+0000","last_became_active":"2024-04-25T01:30:34.358039+0000","last_became_peered":"2024-04-25T01:30:34.358039+0000","last_unstale":"2024-04-25T01:33:00.642196+0000","last_undegraded":"2024-04-25T01:33:00.642196+0000","last_fullsized":"2024-04-25T01:33:00.642196+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:42.600693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.624086+0000","last_change":"2024-04-25T01:30:34.357803+0000","last_active":"2024-04-25T01:33:00.624086+0000","last_peered":"2024-04-25T01:33:00.624086+0000","last_clean":"2024-04-25T01:33:00.624086+0000","last_became_active":"2024-04-25T01:30:34.357391+0000","last_became_peered":"2024-04-25T01:30:34.357391+0000","last_unstale":"2024-04-25T01:33:00.624086+0000","last_undegraded":"2024-04-25T01:33:00.624086+0000","last_fullsized":"2024-04-25T01:33:00.624086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:57.857303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642471+0000","last_change":"2024-04-25T01:30:34.363340+0000","last_active":"2024-04-25T01:33:00.642471+0000","last_peered":"2024-04-25T01:33:00.642471+0000","last_clean":"2024-04-25T01:33:00.642471+0000","last_became_active":"2024-04-25T01:30:34.362980+0000","last_became_peered":"2024-04-25T01:30:34.362980+0000","last_unstale":"2024-04-25T01:33:00.642471+0000","last_undegraded":"2024-04-25T01:33:00.642471+0000","last_fullsized":"2024-04-25T01:33:00.642471+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:46.291459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641988+0000","last_change":"2024-04-25T01:30:34.358898+0000","last_active":"2024-04-25T01:33:00.641988+0000","last_peered":"2024-04-25T01:33:00.641988+0000","last_clean":"2024-04-25T01:33:00.641988+0000","last_became_active":"2024-04-25T01:30:34.358495+0000","last_became_peered":"2024-04-25T01:30:34.358495+0000","last_unstale":"2024-04-25T01:33:00.641988+0000","last_undegraded":"2024-04-25T01:33:00.641988+0000","last_fullsized":"2024-04-25T01:33:00.641988+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:34:21.860010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643535+0000","last_change":"2024-04-25T01:30:34.363708+0000","last_active":"2024-04-25T01:33:00.643535+0000","last_peered":"2024-04-25T01:33:00.643535+0000","last_clean":"2024-04-25T01:33:00.643535+0000","last_became_active":"2024-04-25T01:30:34.363207+0000","last_became_peered":"2024-04-25T01:30:34.363207+0000","last_unstale":"2024-04-25T01:33:00.643535+0000","last_undegraded":"2024-04-25T01:33:00.643535+0000","last_fullsized":"2024-04-25T01:33:00.643535+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:53.442987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642070+0000","last_change":"2024-04-25T01:30:34.371031+0000","last_active":"2024-04-25T01:33:00.642070+0000","last_peered":"2024-04-25T01:33:00.642070+0000","last_clean":"2024-04-25T01:33:00.642070+0000","last_became_active":"2024-04-25T01:30:34.370687+0000","last_became_peered":"2024-04-25T01:30:34.370687+0000","last_unstale":"2024-04-25T01:33:00.642070+0000","last_undegraded":"2024-04-25T01:33:00.642070+0000","last_fullsized":"2024-04-25T01:33:00.642070+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:03:20.700960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642487+0000","last_change":"2024-04-25T01:30:34.366212+0000","last_active":"2024-04-25T01:33:00.642487+0000","last_peered":"2024-04-25T01:33:00.642487+0000","last_clean":"2024-04-25T01:33:00.642487+0000","last_became_active":"2024-04-25T01:30:34.365849+0000","last_became_peered":"2024-04-25T01:30:34.365849+0000","last_unstale":"2024-04-25T01:33:00.642487+0000","last_undegraded":"2024-04-25T01:33:00.642487+0000","last_fullsized":"2024-04-25T01:33:00.642487+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:24:02.397615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640584+0000","last_change":"2024-04-25T01:30:34.367380+0000","last_active":"2024-04-25T01:33:00.640584+0000","last_peered":"2024-04-25T01:33:00.640584+0000","last_clean":"2024-04-25T01:33:00.640584+0000","last_became_active":"2024-04-25T01:30:34.366872+0000","last_became_peered":"2024-04-25T01:30:34.366872+0000","last_unstale":"2024-04-25T01:33:00.640584+0000","last_undegraded":"2024-04-25T01:33:00.640584+0000","last_fullsized":"2024-04-25T01:33:00.640584+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:15:18.767108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641445+0000","last_change":"2024-04-25T01:30:34.359752+0000","last_active":"2024-04-25T01:33:00.641445+0000","last_peered":"2024-04-25T01:33:00.641445+0000","last_clean":"2024-04-25T01:33:00.641445+0000","last_became_active":"2024-04-25T01:30:34.358403+0000","last_became_peered":"2024-04-25T01:30:34.358403+0000","last_unstale":"2024-04-25T01:33:00.641445+0000","last_undegraded":"2024-04-25T01:33:00.641445+0000","last_fullsized":"2024-04-25T01:33:00.641445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:14.833126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641209+0000","last_change":"2024-04-25T01:30:34.361373+0000","last_active":"2024-04-25T01:33:00.641209+0000","last_peered":"2024-04-25T01:33:00.641209+0000","last_clean":"2024-04-25T01:33:00.641209+0000","last_became_active":"2024-04-25T01:30:34.361046+0000","last_became_peered":"2024-04-25T01:30:34.361046+0000","last_unstale":"2024-04-25T01:33:00.641209+0000","last_undegraded":"2024-04-25T01:33:00.641209+0000","last_fullsized":"2024-04-25T01:33:00.641209+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:21.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.626292+0000","last_change":"2024-04-25T01:30:34.367975+0000","last_active":"2024-04-25T01:33:00.626292+0000","last_peered":"2024-04-25T01:33:00.626292+0000","last_clean":"2024-04-25T01:33:00.626292+0000","last_became_active":"2024-04-25T01:30:34.366383+0000","last_became_peered":"2024-04-25T01:30:34.366383+0000","last_unstale":"2024-04-25T01:33:00.626292+0000","last_undegraded":"2024-04-25T01:33:00.626292+0000","last_fullsized":"2024-04-25T01:33:00.626292+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:53:40.279067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.625811+0000","last_change":"2024-04-25T01:30:38.382168+0000","last_active":"2024-04-25T01:33:00.625811+0000","last_peered":"2024-04-25T01:33:00.625811+0000","last_clean":"2024-04-25T01:33:00.625811+0000","last_became_active":"2024-04-25T01:30:38.381740+0000","last_became_peered":"2024-04-25T01:30:38.381740+0000","last_unstale":"2024-04-25T01:33:00.625811+0000","last_undegraded":"2024-04-25T01:33:00.625811+0000","last_fullsized":"2024-04-25T01:33:00.625811+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:27.425991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'13","reported_seq":164,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641584+0000","last_change":"2024-04-25T01:30:33.345547+0000","last_active":"2024-04-25T01:33:00.641584+0000","last_peered":"2024-04-25T01:33:00.641584+0000","last_clean":"2024-04-25T01:33:00.641584+0000","last_became_active":"2024-04-25T01:30:33.345229+0000","last_became_peered":"2024-04-25T01:30:33.345229+0000","last_unstale":"2024-04-25T01:33:00.641584+0000","last_undegraded":"2024-04-25T01:33:00.641584+0000","last_fullsized":"2024-04-25T01:33:00.641584+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:32:46.637720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622343+0000","last_change":"2024-04-25T01:30:34.354703+0000","last_active":"2024-04-25T01:33:00.622343+0000","last_peered":"2024-04-25T01:33:00.622343+0000","last_clean":"2024-04-25T01:33:00.622343+0000","last_became_active":"2024-04-25T01:30:34.354362+0000","last_became_peered":"2024-04-25T01:30:34.354362+0000","last_unstale":"2024-04-25T01:33:00.622343+0000","last_undegraded":"2024-04-25T01:33:00.622343+0000","last_fullsized":"2024-04-25T01:33:00.622343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:45:51.358599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640543+0000","last_change":"2024-04-25T01:30:38.380912+0000","last_active":"2024-04-25T01:33:00.640543+0000","last_peered":"2024-04-25T01:33:00.640543+0000","last_clean":"2024-04-25T01:33:00.640543+0000","last_became_active":"2024-04-25T01:30:38.380439+0000","last_became_peered":"2024-04-25T01:30:38.380439+0000","last_unstale":"2024-04-25T01:33:00.640543+0000","last_undegraded":"2024-04-25T01:33:00.640543+0000","last_fullsized":"2024-04-25T01:33:00.640543+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:22:00.527268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622488+0000","last_change":"2024-04-25T01:30:33.345620+0000","last_active":"2024-04-25T01:33:00.622488+0000","last_peered":"2024-04-25T01:33:00.622488+0000","last_clean":"2024-04-25T01:33:00.622488+0000","last_became_active":"2024-04-25T01:30:33.345315+0000","last_became_peered":"2024-04-25T01:30:33.345315+0000","last_unstale":"2024-04-25T01:33:00.622488+0000","last_undegraded":"2024-04-25T01:33:00.622488+0000","last_fullsized":"2024-04-25T01:33:00.622488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:48:00.083728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643224+0000","last_change":"2024-04-25T01:30:34.356589+0000","last_active":"2024-04-25T01:33:00.643224+0000","last_peered":"2024-04-25T01:33:00.643224+0000","last_clean":"2024-04-25T01:33:00.643224+0000","last_became_active":"2024-04-25T01:30:34.356114+0000","last_became_peered":"2024-04-25T01:30:34.356114+0000","last_unstale":"2024-04-25T01:33:00.643224+0000","last_undegraded":"2024-04-25T01:33:00.643224+0000","last_fullsized":"2024-04-25T01:33:00.643224+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:23:39.039517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640748+0000","last_change":"2024-04-25T01:31:21.876578+0000","last_active":"2024-04-25T01:33:00.640748+0000","last_peered":"2024-04-25T01:33:00.640748+0000","last_clean":"2024-04-25T01:33:00.640748+0000","last_became_active":"2024-04-25T01:31:21.876126+0000","last_became_peered":"2024-04-25T01:31:21.876126+0000","last_unstale":"2024-04-25T01:33:00.640748+0000","last_undegraded":"2024-04-25T01:33:00.640748+0000","last_fullsized":"2024-04-25T01:33:00.640748+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:36:55.909353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'12","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730820+0000","last_change":"2024-04-25T01:30:33.345575+0000","last_active":"2024-04-25T01:33:00.730820+0000","last_peered":"2024-04-25T01:33:00.730820+0000","last_clean":"2024-04-25T01:33:00.730820+0000","last_became_active":"2024-04-25T01:30:33.345231+0000","last_became_peered":"2024-04-25T01:30:33.345231+0000","last_unstale":"2024-04-25T01:33:00.730820+0000","last_undegraded":"2024-04-25T01:33:00.730820+0000","last_fullsized":"2024-04-25T01:33:00.730820+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:04:21.090788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641074+0000","last_change":"2024-04-25T01:30:34.371317+0000","last_active":"2024-04-25T01:33:00.641074+0000","last_peered":"2024-04-25T01:33:00.641074+0000","last_clean":"2024-04-25T01:33:00.641074+0000","last_became_active":"2024-04-25T01:30:34.370926+0000","last_became_peered":"2024-04-25T01:30:34.370926+0000","last_unstale":"2024-04-25T01:33:00.641074+0000","last_undegraded":"2024-04-25T01:33:00.641074+0000","last_fullsized":"2024-04-25T01:33:00.641074+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:58:39.898471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641235+0000","last_change":"2024-04-25T01:30:38.380982+0000","last_active":"2024-04-25T01:33:00.641235+0000","last_peered":"2024-04-25T01:33:00.641235+0000","last_clean":"2024-04-25T01:33:00.641235+0000","last_became_active":"2024-04-25T01:30:38.380510+0000","last_became_peered":"2024-04-25T01:30:38.380510+0000","last_unstale":"2024-04-25T01:33:00.641235+0000","last_undegraded":"2024-04-25T01:33:00.641235+0000","last_fullsized":"2024-04-25T01:33:00.641235+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:00.510697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"91'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730845+0000","last_change":"2024-04-25T01:30:33.351561+0000","last_active":"2024-04-25T01:33:00.730845+0000","last_peered":"2024-04-25T01:33:00.730845+0000","last_clean":"2024-04-25T01:33:00.730845+0000","last_became_active":"2024-04-25T01:30:33.351230+0000","last_became_peered":"2024-04-25T01:30:33.351230+0000","last_unstale":"2024-04-25T01:33:00.730845+0000","last_undegraded":"2024-04-25T01:33:00.730845+0000","last_fullsized":"2024-04-25T01:33:00.730845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:56.594457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641855+0000","last_change":"2024-04-25T01:30:34.358821+0000","last_active":"2024-04-25T01:33:00.641855+0000","last_peered":"2024-04-25T01:33:00.641855+0000","last_clean":"2024-04-25T01:33:00.641855+0000","last_became_active":"2024-04-25T01:30:34.358303+0000","last_became_peered":"2024-04-25T01:30:34.358303+0000","last_unstale":"2024-04-25T01:33:00.641855+0000","last_undegraded":"2024-04-25T01:33:00.641855+0000","last_fullsized":"2024-04-25T01:33:00.641855+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:26:23.352101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.626107+0000","last_change":"2024-04-25T01:30:38.372281+0000","last_active":"2024-04-25T01:33:00.626107+0000","last_peered":"2024-04-25T01:33:00.626107+0000","last_clean":"2024-04-25T01:33:00.626107+0000","last_became_active":"2024-04-25T01:30:38.371876+0000","last_became_peered":"2024-04-25T01:30:38.371876+0000","last_unstale":"2024-04-25T01:33:00.626107+0000","last_undegraded":"2024-04-25T01:33:00.626107+0000","last_fullsized":"2024-04-25T01:33:00.626107+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:19:54.254011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'7","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.980108+0000","last_change":"2024-04-25T01:30:33.347262+0000","last_active":"2024-04-25T01:33:00.980108+0000","last_peered":"2024-04-25T01:33:00.980108+0000","last_clean":"2024-04-25T01:33:00.980108+0000","last_became_active":"2024-04-25T01:30:33.346900+0000","last_became_peered":"2024-04-25T01:30:33.346900+0000","last_unstale":"2024-04-25T01:33:00.980108+0000","last_undegraded":"2024-04-25T01:33:00.980108+0000","last_fullsized":"2024-04-25T01:33:00.980108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:29:30.960456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730404+0000","last_change":"2024-04-25T01:30:34.364870+0000","last_active":"2024-04-25T01:33:00.730404+0000","last_peered":"2024-04-25T01:33:00.730404+0000","last_clean":"2024-04-25T01:33:00.730404+0000","last_became_active":"2024-04-25T01:30:34.364459+0000","last_became_peered":"2024-04-25T01:30:34.364459+0000","last_unstale":"2024-04-25T01:33:00.730404+0000","last_undegraded":"2024-04-25T01:33:00.730404+0000","last_fullsized":"2024-04-25T01:33:00.730404+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:42:45.311668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.626138+0000","last_change":"2024-04-25T01:30:38.382175+0000","last_active":"2024-04-25T01:33:00.626138+0000","last_peered":"2024-04-25T01:33:00.626138+0000","last_clean":"2024-04-25T01:33:00.626138+0000","last_became_active":"2024-04-25T01:30:38.381771+0000","last_became_peered":"2024-04-25T01:30:38.381771+0000","last_unstale":"2024-04-25T01:33:00.626138+0000","last_undegraded":"2024-04-25T01:33:00.626138+0000","last_fullsized":"2024-04-25T01:33:00.626138+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:25:31.041566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.623581+0000","last_change":"2024-04-25T01:30:33.343605+0000","last_active":"2024-04-25T01:33:00.623581+0000","last_peered":"2024-04-25T01:33:00.623581+0000","last_clean":"2024-04-25T01:33:00.623581+0000","last_became_active":"2024-04-25T01:30:33.343254+0000","last_became_peered":"2024-04-25T01:30:33.343254+0000","last_unstale":"2024-04-25T01:33:00.623581+0000","last_undegraded":"2024-04-25T01:33:00.623581+0000","last_fullsized":"2024-04-25T01:33:00.623581+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:18:16.156285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640694+0000","last_change":"2024-04-25T01:31:21.874202+0000","last_active":"2024-04-25T01:33:00.640694+0000","last_peered":"2024-04-25T01:33:00.640694+0000","last_clean":"2024-04-25T01:33:00.640694+0000","last_became_active":"2024-04-25T01:31:21.873839+0000","last_became_peered":"2024-04-25T01:31:21.873839+0000","last_unstale":"2024-04-25T01:33:00.640694+0000","last_undegraded":"2024-04-25T01:33:00.640694+0000","last_fullsized":"2024-04-25T01:33:00.640694+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:39.774281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641353+0000","last_change":"2024-04-25T01:31:21.875259+0000","last_active":"2024-04-25T01:33:00.641353+0000","last_peered":"2024-04-25T01:33:00.641353+0000","last_clean":"2024-04-25T01:33:00.641353+0000","last_became_active":"2024-04-25T01:31:21.874800+0000","last_became_peered":"2024-04-25T01:31:21.874800+0000","last_unstale":"2024-04-25T01:33:00.641353+0000","last_undegraded":"2024-04-25T01:33:00.641353+0000","last_fullsized":"2024-04-25T01:33:00.641353+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:44.828620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,10,0],"acting":[11,4,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'3","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642169+0000","last_change":"2024-04-25T01:30:33.351621+0000","last_active":"2024-04-25T01:33:00.642169+0000","last_peered":"2024-04-25T01:33:00.642169+0000","last_clean":"2024-04-25T01:33:00.642169+0000","last_became_active":"2024-04-25T01:30:33.351289+0000","last_became_peered":"2024-04-25T01:30:33.351289+0000","last_unstale":"2024-04-25T01:33:00.642169+0000","last_undegraded":"2024-04-25T01:33:00.642169+0000","last_fullsized":"2024-04-25T01:33:00.642169+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:46.044826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622260+0000","last_change":"2024-04-25T01:30:34.350685+0000","last_active":"2024-04-25T01:33:00.622260+0000","last_peered":"2024-04-25T01:33:00.622260+0000","last_clean":"2024-04-25T01:33:00.622260+0000","last_became_active":"2024-04-25T01:30:34.350308+0000","last_became_peered":"2024-04-25T01:30:34.350308+0000","last_unstale":"2024-04-25T01:33:00.622260+0000","last_undegraded":"2024-04-25T01:33:00.622260+0000","last_fullsized":"2024-04-25T01:33:00.622260+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:07:11.690381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643016+0000","last_change":"2024-04-25T01:30:38.377403+0000","last_active":"2024-04-25T01:33:00.643016+0000","last_peered":"2024-04-25T01:33:00.643016+0000","last_clean":"2024-04-25T01:33:00.643016+0000","last_became_active":"2024-04-25T01:30:38.376966+0000","last_became_peered":"2024-04-25T01:30:38.376966+0000","last_unstale":"2024-04-25T01:33:00.643016+0000","last_undegraded":"2024-04-25T01:33:00.643016+0000","last_fullsized":"2024-04-25T01:33:00.643016+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:31:10.229443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'3","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641337+0000","last_change":"2024-04-25T01:30:33.348455+0000","last_active":"2024-04-25T01:33:00.641337+0000","last_peered":"2024-04-25T01:33:00.641337+0000","last_clean":"2024-04-25T01:33:00.641337+0000","last_became_active":"2024-04-25T01:30:33.348142+0000","last_became_peered":"2024-04-25T01:30:33.348142+0000","last_unstale":"2024-04-25T01:33:00.641337+0000","last_undegraded":"2024-04-25T01:33:00.641337+0000","last_fullsized":"2024-04-25T01:33:00.641337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:43:07.695580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640378+0000","last_change":"2024-04-25T01:30:34.359949+0000","last_active":"2024-04-25T01:33:00.640378+0000","last_peered":"2024-04-25T01:33:00.640378+0000","last_clean":"2024-04-25T01:33:00.640378+0000","last_became_active":"2024-04-25T01:30:34.359609+0000","last_became_peered":"2024-04-25T01:30:34.359609+0000","last_unstale":"2024-04-25T01:33:00.640378+0000","last_undegraded":"2024-04-25T01:33:00.640378+0000","last_fullsized":"2024-04-25T01:33:00.640378+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:45:49.668800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622175+0000","last_change":"2024-04-25T01:30:38.380260+0000","last_active":"2024-04-25T01:33:00.622175+0000","last_peered":"2024-04-25T01:33:00.622175+0000","last_clean":"2024-04-25T01:33:00.622175+0000","last_became_active":"2024-04-25T01:30:38.379624+0000","last_became_peered":"2024-04-25T01:30:38.379624+0000","last_unstale":"2024-04-25T01:33:00.622175+0000","last_undegraded":"2024-04-25T01:33:00.622175+0000","last_fullsized":"2024-04-25T01:33:00.622175+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:26:35.235335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'6","reported_seq":276,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.979614+0000","last_change":"2024-04-25T01:30:33.349490+0000","last_active":"2024-04-25T01:33:00.979614+0000","last_peered":"2024-04-25T01:33:00.979614+0000","last_clean":"2024-04-25T01:33:00.979614+0000","last_became_active":"2024-04-25T01:30:33.349136+0000","last_became_peered":"2024-04-25T01:30:33.349136+0000","last_unstale":"2024-04-25T01:33:00.979614+0000","last_undegraded":"2024-04-25T01:33:00.979614+0000","last_fullsized":"2024-04-25T01:33:00.979614+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:16:23.552073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":240,"num_read_kb":240,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641812+0000","last_change":"2024-04-25T01:30:34.372701+0000","last_active":"2024-04-25T01:33:00.641812+0000","last_peered":"2024-04-25T01:33:00.641812+0000","last_clean":"2024-04-25T01:33:00.641812+0000","last_became_active":"2024-04-25T01:30:34.372321+0000","last_became_peered":"2024-04-25T01:30:34.372321+0000","last_unstale":"2024-04-25T01:33:00.641812+0000","last_undegraded":"2024-04-25T01:33:00.641812+0000","last_fullsized":"2024-04-25T01:33:00.641812+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:02.025015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640169+0000","last_change":"2024-04-25T01:30:38.378036+0000","last_active":"2024-04-25T01:33:00.640169+0000","last_peered":"2024-04-25T01:33:00.640169+0000","last_clean":"2024-04-25T01:33:00.640169+0000","last_became_active":"2024-04-25T01:30:38.377602+0000","last_became_peered":"2024-04-25T01:30:38.377602+0000","last_unstale":"2024-04-25T01:33:00.640169+0000","last_undegraded":"2024-04-25T01:33:00.640169+0000","last_fullsized":"2024-04-25T01:33:00.640169+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:15:45.210518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641468+0000","last_change":"2024-04-25T01:30:33.352928+0000","last_active":"2024-04-25T01:33:00.641468+0000","last_peered":"2024-04-25T01:33:00.641468+0000","last_clean":"2024-04-25T01:33:00.641468+0000","last_became_active":"2024-04-25T01:30:33.352567+0000","last_became_peered":"2024-04-25T01:30:33.352567+0000","last_unstale":"2024-04-25T01:33:00.641468+0000","last_undegraded":"2024-04-25T01:33:00.641468+0000","last_fullsized":"2024-04-25T01:33:00.641468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:47:21.941179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643449+0000","last_change":"2024-04-25T01:30:34.366947+0000","last_active":"2024-04-25T01:33:00.643449+0000","last_peered":"2024-04-25T01:33:00.643449+0000","last_clean":"2024-04-25T01:33:00.643449+0000","last_became_active":"2024-04-25T01:30:34.366337+0000","last_became_peered":"2024-04-25T01:30:34.366337+0000","last_unstale":"2024-04-25T01:33:00.643449+0000","last_undegraded":"2024-04-25T01:33:00.643449+0000","last_fullsized":"2024-04-25T01:33:00.643449+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:48:43.528380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641593+0000","last_change":"2024-04-25T01:31:22.050799+0000","last_active":"2024-04-25T01:33:00.641593+0000","last_peered":"2024-04-25T01:33:00.641593+0000","last_clean":"2024-04-25T01:33:00.641593+0000","last_became_active":"2024-04-25T01:31:22.050130+0000","last_became_peered":"2024-04-25T01:31:22.050130+0000","last_unstale":"2024-04-25T01:33:00.641593+0000","last_undegraded":"2024-04-25T01:33:00.641593+0000","last_fullsized":"2024-04-25T01:33:00.641593+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:40:43.549713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641872+0000","last_change":"2024-04-25T01:30:33.346003+0000","last_active":"2024-04-25T01:33:00.641872+0000","last_peered":"2024-04-25T01:33:00.641872+0000","last_clean":"2024-04-25T01:33:00.641872+0000","last_became_active":"2024-04-25T01:30:33.345254+0000","last_became_peered":"2024-04-25T01:30:33.345254+0000","last_unstale":"2024-04-25T01:33:00.641872+0000","last_undegraded":"2024-04-25T01:33:00.641872+0000","last_fullsized":"2024-04-25T01:33:00.641872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:18:37.957047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641299+0000","last_change":"2024-04-25T01:30:34.366108+0000","last_active":"2024-04-25T01:33:00.641299+0000","last_peered":"2024-04-25T01:33:00.641299+0000","last_clean":"2024-04-25T01:33:00.641299+0000","last_became_active":"2024-04-25T01:30:34.365702+0000","last_became_peered":"2024-04-25T01:30:34.365702+0000","last_unstale":"2024-04-25T01:33:00.641299+0000","last_undegraded":"2024-04-25T01:33:00.641299+0000","last_fullsized":"2024-04-25T01:33:00.641299+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:24:06.802424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640332+0000","last_change":"2024-04-25T01:30:38.376342+0000","last_active":"2024-04-25T01:33:00.640332+0000","last_peered":"2024-04-25T01:33:00.640332+0000","last_clean":"2024-04-25T01:33:00.640332+0000","last_became_active":"2024-04-25T01:30:38.375905+0000","last_became_peered":"2024-04-25T01:30:38.375905+0000","last_unstale":"2024-04-25T01:33:00.640332+0000","last_undegraded":"2024-04-25T01:33:00.640332+0000","last_fullsized":"2024-04-25T01:33:00.640332+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:31.615356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'44","reported_seq":325,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.980291+0000","last_change":"2024-04-25T01:30:33.350770+0000","last_active":"2024-04-25T01:33:00.980291+0000","last_peered":"2024-04-25T01:33:00.980291+0000","last_clean":"2024-04-25T01:33:00.980291+0000","last_became_active":"2024-04-25T01:30:33.350414+0000","last_became_peered":"2024-04-25T01:30:33.350414+0000","last_unstale":"2024-04-25T01:33:00.980291+0000","last_undegraded":"2024-04-25T01:33:00.980291+0000","last_fullsized":"2024-04-25T01:33:00.980291+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:01.771182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67986,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":251,"num_read_kb":193,"num_write":46,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730709+0000","last_change":"2024-04-25T01:30:34.362532+0000","last_active":"2024-04-25T01:33:00.730709+0000","last_peered":"2024-04-25T01:33:00.730709+0000","last_clean":"2024-04-25T01:33:00.730709+0000","last_became_active":"2024-04-25T01:30:34.362137+0000","last_became_peered":"2024-04-25T01:30:34.362137+0000","last_unstale":"2024-04-25T01:33:00.730709+0000","last_undegraded":"2024-04-25T01:33:00.730709+0000","last_fullsized":"2024-04-25T01:33:00.730709+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:53:28.105250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642169+0000","last_change":"2024-04-25T01:30:38.374591+0000","last_active":"2024-04-25T01:33:00.642169+0000","last_peered":"2024-04-25T01:33:00.642169+0000","last_clean":"2024-04-25T01:33:00.642169+0000","last_became_active":"2024-04-25T01:30:38.373237+0000","last_became_peered":"2024-04-25T01:30:38.373237+0000","last_unstale":"2024-04-25T01:33:00.642169+0000","last_undegraded":"2024-04-25T01:33:00.642169+0000","last_fullsized":"2024-04-25T01:33:00.642169+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:21:25.937405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'9","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640904+0000","last_change":"2024-04-25T01:31:21.885341+0000","last_active":"2024-04-25T01:33:00.640904+0000","last_peered":"2024-04-25T01:33:00.640904+0000","last_clean":"2024-04-25T01:33:00.640904+0000","last_became_active":"2024-04-25T01:31:21.874312+0000","last_became_peered":"2024-04-25T01:31:21.874312+0000","last_unstale":"2024-04-25T01:33:00.640904+0000","last_undegraded":"2024-04-25T01:33:00.640904+0000","last_fullsized":"2024-04-25T01:33:00.640904+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:23:55.388089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642553+0000","last_change":"2024-04-25T01:30:34.354444+0000","last_active":"2024-04-25T01:33:00.642553+0000","last_peered":"2024-04-25T01:33:00.642553+0000","last_clean":"2024-04-25T01:33:00.642553+0000","last_became_active":"2024-04-25T01:30:34.354070+0000","last_became_peered":"2024-04-25T01:30:34.354070+0000","last_unstale":"2024-04-25T01:33:00.642553+0000","last_undegraded":"2024-04-25T01:33:00.642553+0000","last_fullsized":"2024-04-25T01:33:00.642553+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:15:39.893995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640956+0000","last_change":"2024-04-25T01:30:38.373570+0000","last_active":"2024-04-25T01:33:00.640956+0000","last_peered":"2024-04-25T01:33:00.640956+0000","last_clean":"2024-04-25T01:33:00.640956+0000","last_became_active":"2024-04-25T01:30:38.373144+0000","last_became_peered":"2024-04-25T01:30:38.373144+0000","last_unstale":"2024-04-25T01:33:00.640956+0000","last_undegraded":"2024-04-25T01:33:00.640956+0000","last_fullsized":"2024-04-25T01:33:00.640956+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:53:32.933998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641434+0000","last_change":"2024-04-25T01:30:33.346145+0000","last_active":"2024-04-25T01:33:00.641434+0000","last_peered":"2024-04-25T01:33:00.641434+0000","last_clean":"2024-04-25T01:33:00.641434+0000","last_became_active":"2024-04-25T01:30:33.344294+0000","last_became_peered":"2024-04-25T01:30:33.344294+0000","last_unstale":"2024-04-25T01:33:00.641434+0000","last_undegraded":"2024-04-25T01:33:00.641434+0000","last_fullsized":"2024-04-25T01:33:00.641434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:37:17.147148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641769+0000","last_change":"2024-04-25T01:30:34.360769+0000","last_active":"2024-04-25T01:33:00.641769+0000","last_peered":"2024-04-25T01:33:00.641769+0000","last_clean":"2024-04-25T01:33:00.641769+0000","last_became_active":"2024-04-25T01:30:34.360076+0000","last_became_peered":"2024-04-25T01:30:34.360076+0000","last_unstale":"2024-04-25T01:33:00.641769+0000","last_undegraded":"2024-04-25T01:33:00.641769+0000","last_fullsized":"2024-04-25T01:33:00.641769+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:56:21.515511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640926+0000","last_change":"2024-04-25T01:30:38.379198+0000","last_active":"2024-04-25T01:33:00.640926+0000","last_peered":"2024-04-25T01:33:00.640926+0000","last_clean":"2024-04-25T01:33:00.640926+0000","last_became_active":"2024-04-25T01:30:38.378798+0000","last_became_peered":"2024-04-25T01:30:38.378798+0000","last_unstale":"2024-04-25T01:33:00.640926+0000","last_undegraded":"2024-04-25T01:33:00.640926+0000","last_fullsized":"2024-04-25T01:33:00.640926+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:14:40.192239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'8","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.972284+0000","last_change":"2024-04-25T01:30:33.343232+0000","last_active":"2024-04-25T01:33:00.972284+0000","last_peered":"2024-04-25T01:33:00.972284+0000","last_clean":"2024-04-25T01:33:00.972284+0000","last_became_active":"2024-04-25T01:30:33.342876+0000","last_became_peered":"2024-04-25T01:30:33.342876+0000","last_unstale":"2024-04-25T01:33:00.972284+0000","last_undegraded":"2024-04-25T01:33:00.972284+0000","last_fullsized":"2024-04-25T01:33:00.972284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:45.032564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1678,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":125,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641248+0000","last_change":"2024-04-25T01:30:34.358210+0000","last_active":"2024-04-25T01:33:00.641248+0000","last_peered":"2024-04-25T01:33:00.641248+0000","last_clean":"2024-04-25T01:33:00.641248+0000","last_became_active":"2024-04-25T01:30:34.357862+0000","last_became_peered":"2024-04-25T01:30:34.357862+0000","last_unstale":"2024-04-25T01:33:00.641248+0000","last_undegraded":"2024-04-25T01:33:00.641248+0000","last_fullsized":"2024-04-25T01:33:00.641248+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:43.751361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643538+0000","last_change":"2024-04-25T01:30:38.380413+0000","last_active":"2024-04-25T01:33:00.643538+0000","last_peered":"2024-04-25T01:33:00.643538+0000","last_clean":"2024-04-25T01:33:00.643538+0000","last_became_active":"2024-04-25T01:30:38.379612+0000","last_became_peered":"2024-04-25T01:30:38.379612+0000","last_unstale":"2024-04-25T01:33:00.643538+0000","last_undegraded":"2024-04-25T01:33:00.643538+0000","last_fullsized":"2024-04-25T01:33:00.643538+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:45.328642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'17","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730348+0000","last_change":"2024-04-25T01:30:33.344938+0000","last_active":"2024-04-25T01:33:00.730348+0000","last_peered":"2024-04-25T01:33:00.730348+0000","last_clean":"2024-04-25T01:33:00.730348+0000","last_became_active":"2024-04-25T01:30:33.344582+0000","last_became_peered":"2024-04-25T01:30:33.344582+0000","last_unstale":"2024-04-25T01:33:00.730348+0000","last_undegraded":"2024-04-25T01:33:00.730348+0000","last_fullsized":"2024-04-25T01:33:00.730348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:43:33.777319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641031+0000","last_change":"2024-04-25T01:30:34.354115+0000","last_active":"2024-04-25T01:33:00.641031+0000","last_peered":"2024-04-25T01:33:00.641031+0000","last_clean":"2024-04-25T01:33:00.641031+0000","last_became_active":"2024-04-25T01:30:34.353722+0000","last_became_peered":"2024-04-25T01:30:34.353722+0000","last_unstale":"2024-04-25T01:33:00.641031+0000","last_undegraded":"2024-04-25T01:33:00.641031+0000","last_fullsized":"2024-04-25T01:33:00.641031+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:22.003521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641136+0000","last_change":"2024-04-25T01:30:38.380914+0000","last_active":"2024-04-25T01:33:00.641136+0000","last_peered":"2024-04-25T01:33:00.641136+0000","last_clean":"2024-04-25T01:33:00.641136+0000","last_became_active":"2024-04-25T01:30:38.380477+0000","last_became_peered":"2024-04-25T01:30:38.380477+0000","last_unstale":"2024-04-25T01:33:00.641136+0000","last_undegraded":"2024-04-25T01:33:00.641136+0000","last_fullsized":"2024-04-25T01:33:00.641136+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:23:27.882266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"94'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730890+0000","last_change":"2024-04-25T01:30:33.345772+0000","last_active":"2024-04-25T01:33:00.730890+0000","last_peered":"2024-04-25T01:33:00.730890+0000","last_clean":"2024-04-25T01:33:00.730890+0000","last_became_active":"2024-04-25T01:30:33.345455+0000","last_became_peered":"2024-04-25T01:30:33.345455+0000","last_unstale":"2024-04-25T01:33:00.730890+0000","last_undegraded":"2024-04-25T01:33:00.730890+0000","last_fullsized":"2024-04-25T01:33:00.730890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:14:57.465161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641257+0000","last_change":"2024-04-25T01:30:34.358665+0000","last_active":"2024-04-25T01:33:00.641257+0000","last_peered":"2024-04-25T01:33:00.641257+0000","last_clean":"2024-04-25T01:33:00.641257+0000","last_became_active":"2024-04-25T01:30:34.358284+0000","last_became_peered":"2024-04-25T01:30:34.358284+0000","last_unstale":"2024-04-25T01:33:00.641257+0000","last_undegraded":"2024-04-25T01:33:00.641257+0000","last_fullsized":"2024-04-25T01:33:00.641257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:12:22.832265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730394+0000","last_change":"2024-04-25T01:30:38.450027+0000","last_active":"2024-04-25T01:33:00.730394+0000","last_peered":"2024-04-25T01:33:00.730394+0000","last_clean":"2024-04-25T01:33:00.730394+0000","last_became_active":"2024-04-25T01:30:38.449564+0000","last_became_peered":"2024-04-25T01:30:38.449564+0000","last_unstale":"2024-04-25T01:33:00.730394+0000","last_undegraded":"2024-04-25T01:33:00.730394+0000","last_fullsized":"2024-04-25T01:33:00.730394+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:34:26.542351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"91'1","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641180+0000","last_change":"2024-04-25T01:30:33.349757+0000","last_active":"2024-04-25T01:33:00.641180+0000","last_peered":"2024-04-25T01:33:00.641180+0000","last_clean":"2024-04-25T01:33:00.641180+0000","last_became_active":"2024-04-25T01:30:33.349439+0000","last_became_peered":"2024-04-25T01:30:33.349439+0000","last_unstale":"2024-04-25T01:33:00.641180+0000","last_undegraded":"2024-04-25T01:33:00.641180+0000","last_fullsized":"2024-04-25T01:33:00.641180+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:17:35.514160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.623885+0000","last_change":"2024-04-25T01:30:34.367310+0000","last_active":"2024-04-25T01:33:00.623885+0000","last_peered":"2024-04-25T01:33:00.623885+0000","last_clean":"2024-04-25T01:33:00.623885+0000","last_became_active":"2024-04-25T01:30:34.366950+0000","last_became_peered":"2024-04-25T01:30:34.366950+0000","last_unstale":"2024-04-25T01:33:00.623885+0000","last_undegraded":"2024-04-25T01:33:00.623885+0000","last_fullsized":"2024-04-25T01:33:00.623885+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:26:17.700512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642919+0000","last_change":"2024-04-25T01:30:38.372259+0000","last_active":"2024-04-25T01:33:00.642919+0000","last_peered":"2024-04-25T01:33:00.642919+0000","last_clean":"2024-04-25T01:33:00.642919+0000","last_became_active":"2024-04-25T01:30:38.371847+0000","last_became_peered":"2024-04-25T01:30:38.371847+0000","last_unstale":"2024-04-25T01:33:00.642919+0000","last_undegraded":"2024-04-25T01:33:00.642919+0000","last_fullsized":"2024-04-25T01:33:00.642919+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:55.047739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'5","reported_seq":156,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642256+0000","last_change":"2024-04-25T01:30:33.352078+0000","last_active":"2024-04-25T01:33:00.642256+0000","last_peered":"2024-04-25T01:33:00.642256+0000","last_clean":"2024-04-25T01:33:00.642256+0000","last_became_active":"2024-04-25T01:30:33.351725+0000","last_became_peered":"2024-04-25T01:30:33.351725+0000","last_unstale":"2024-04-25T01:33:00.642256+0000","last_undegraded":"2024-04-25T01:33:00.642256+0000","last_fullsized":"2024-04-25T01:33:00.642256+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:55:55.236040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.623632+0000","last_change":"2024-04-25T01:30:34.360324+0000","last_active":"2024-04-25T01:33:00.623632+0000","last_peered":"2024-04-25T01:33:00.623632+0000","last_clean":"2024-04-25T01:33:00.623632+0000","last_became_active":"2024-04-25T01:30:34.359114+0000","last_became_peered":"2024-04-25T01:30:34.359114+0000","last_unstale":"2024-04-25T01:33:00.623632+0000","last_undegraded":"2024-04-25T01:33:00.623632+0000","last_fullsized":"2024-04-25T01:33:00.623632+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:51.927640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643419+0000","last_change":"2024-04-25T01:30:38.450650+0000","last_active":"2024-04-25T01:33:00.643419+0000","last_peered":"2024-04-25T01:33:00.643419+0000","last_clean":"2024-04-25T01:33:00.643419+0000","last_became_active":"2024-04-25T01:30:38.450149+0000","last_became_peered":"2024-04-25T01:30:38.450149+0000","last_unstale":"2024-04-25T01:33:00.643419+0000","last_undegraded":"2024-04-25T01:33:00.643419+0000","last_fullsized":"2024-04-25T01:33:00.643419+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:34.630803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641415+0000","last_change":"2024-04-25T01:30:34.352961+0000","last_active":"2024-04-25T01:33:00.641415+0000","last_peered":"2024-04-25T01:33:00.641415+0000","last_clean":"2024-04-25T01:33:00.641415+0000","last_became_active":"2024-04-25T01:30:34.352657+0000","last_became_peered":"2024-04-25T01:30:34.352657+0000","last_unstale":"2024-04-25T01:33:00.641415+0000","last_undegraded":"2024-04-25T01:33:00.641415+0000","last_fullsized":"2024-04-25T01:33:00.641415+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:20.934228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'6","reported_seq":157,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641398+0000","last_change":"2024-04-25T01:30:33.344562+0000","last_active":"2024-04-25T01:33:00.641398+0000","last_peered":"2024-04-25T01:33:00.641398+0000","last_clean":"2024-04-25T01:33:00.641398+0000","last_became_active":"2024-04-25T01:30:33.344212+0000","last_became_peered":"2024-04-25T01:30:33.344212+0000","last_unstale":"2024-04-25T01:33:00.641398+0000","last_undegraded":"2024-04-25T01:33:00.641398+0000","last_fullsized":"2024-04-25T01:33:00.641398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:20:51.110618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641120+0000","last_change":"2024-04-25T01:30:38.384316+0000","last_active":"2024-04-25T01:33:00.641120+0000","last_peered":"2024-04-25T01:33:00.641120+0000","last_clean":"2024-04-25T01:33:00.641120+0000","last_became_active":"2024-04-25T01:30:38.383775+0000","last_became_peered":"2024-04-25T01:30:38.383775+0000","last_unstale":"2024-04-25T01:33:00.641120+0000","last_undegraded":"2024-04-25T01:33:00.641120+0000","last_fullsized":"2024-04-25T01:33:00.641120+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:54.165691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640600+0000","last_change":"2024-04-25T01:31:22.050014+0000","last_active":"2024-04-25T01:33:00.640600+0000","last_peered":"2024-04-25T01:33:00.640600+0000","last_clean":"2024-04-25T01:33:00.640600+0000","last_became_active":"2024-04-25T01:31:22.049718+0000","last_became_peered":"2024-04-25T01:31:22.049718+0000","last_unstale":"2024-04-25T01:33:00.640600+0000","last_undegraded":"2024-04-25T01:33:00.640600+0000","last_fullsized":"2024-04-25T01:33:00.640600+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:40:22.233372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"91'6","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640866+0000","last_change":"2024-04-25T01:30:33.345283+0000","last_active":"2024-04-25T01:33:00.640866+0000","last_peered":"2024-04-25T01:33:00.640866+0000","last_clean":"2024-04-25T01:33:00.640866+0000","last_became_active":"2024-04-25T01:30:33.344903+0000","last_became_peered":"2024-04-25T01:30:33.344903+0000","last_unstale":"2024-04-25T01:33:00.640866+0000","last_undegraded":"2024-04-25T01:33:00.640866+0000","last_fullsized":"2024-04-25T01:33:00.640866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:57:01.982850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":366,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640929+0000","last_change":"2024-04-25T01:29:17.689639+0000","last_active":"2024-04-25T01:33:00.640929+0000","last_peered":"2024-04-25T01:33:00.640929+0000","last_clean":"2024-04-25T01:33:00.640929+0000","last_became_active":"2024-04-25T01:29:17.689046+0000","last_became_peered":"2024-04-25T01:29:17.689046+0000","last_unstale":"2024-04-25T01:33:00.640929+0000","last_undegraded":"2024-04-25T01:33:00.640929+0000","last_fullsized":"2024-04-25T01:33:00.640929+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:25:48.767945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:25:48.767945+0000","last_clean_scrub_stamp":"2024-04-25T01:25:48.767945+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:04:06.092632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641924+0000","last_change":"2024-04-25T01:30:38.450938+0000","last_active":"2024-04-25T01:33:00.641924+0000","last_peered":"2024-04-25T01:33:00.641924+0000","last_clean":"2024-04-25T01:33:00.641924+0000","last_became_active":"2024-04-25T01:30:38.450488+0000","last_became_peered":"2024-04-25T01:30:38.450488+0000","last_unstale":"2024-04-25T01:33:00.641924+0000","last_undegraded":"2024-04-25T01:33:00.641924+0000","last_fullsized":"2024-04-25T01:33:00.641924+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:46:44.778133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641178+0000","last_change":"2024-04-25T01:31:21.875719+0000","last_active":"2024-04-25T01:33:00.641178+0000","last_peered":"2024-04-25T01:33:00.641178+0000","last_clean":"2024-04-25T01:33:00.641178+0000","last_became_active":"2024-04-25T01:31:21.875283+0000","last_became_peered":"2024-04-25T01:31:21.875283+0000","last_unstale":"2024-04-25T01:33:00.641178+0000","last_undegraded":"2024-04-25T01:33:00.641178+0000","last_fullsized":"2024-04-25T01:33:00.641178+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:12:39.097019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641047+0000","last_change":"2024-04-25T01:30:33.348198+0000","last_active":"2024-04-25T01:33:00.641047+0000","last_peered":"2024-04-25T01:33:00.641047+0000","last_clean":"2024-04-25T01:33:00.641047+0000","last_became_active":"2024-04-25T01:30:33.347876+0000","last_became_peered":"2024-04-25T01:30:33.347876+0000","last_unstale":"2024-04-25T01:33:00.641047+0000","last_undegraded":"2024-04-25T01:33:00.641047+0000","last_fullsized":"2024-04-25T01:33:00.641047+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:48:33.116571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641704+0000","last_change":"2024-04-25T01:30:38.372331+0000","last_active":"2024-04-25T01:33:00.641704+0000","last_peered":"2024-04-25T01:33:00.641704+0000","last_clean":"2024-04-25T01:33:00.641704+0000","last_became_active":"2024-04-25T01:30:38.371923+0000","last_became_peered":"2024-04-25T01:30:38.371923+0000","last_unstale":"2024-04-25T01:33:00.641704+0000","last_undegraded":"2024-04-25T01:33:00.641704+0000","last_fullsized":"2024-04-25T01:33:00.641704+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:11:06.416967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730656+0000","last_change":"2024-04-25T01:30:33.346606+0000","last_active":"2024-04-25T01:33:00.730656+0000","last_peered":"2024-04-25T01:33:00.730656+0000","last_clean":"2024-04-25T01:33:00.730656+0000","last_became_active":"2024-04-25T01:30:33.346297+0000","last_became_peered":"2024-04-25T01:30:33.346297+0000","last_unstale":"2024-04-25T01:33:00.730656+0000","last_undegraded":"2024-04-25T01:33:00.730656+0000","last_fullsized":"2024-04-25T01:33:00.730656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:07:16.069285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643021+0000","last_change":"2024-04-25T01:30:34.349136+0000","last_active":"2024-04-25T01:33:00.643021+0000","last_peered":"2024-04-25T01:33:00.643021+0000","last_clean":"2024-04-25T01:33:00.643021+0000","last_became_active":"2024-04-25T01:30:34.348810+0000","last_became_peered":"2024-04-25T01:30:34.348810+0000","last_unstale":"2024-04-25T01:33:00.643021+0000","last_undegraded":"2024-04-25T01:33:00.643021+0000","last_fullsized":"2024-04-25T01:33:00.643021+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:39:34.593986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.621999+0000","last_change":"2024-04-25T01:30:38.450428+0000","last_active":"2024-04-25T01:33:00.621999+0000","last_peered":"2024-04-25T01:33:00.621999+0000","last_clean":"2024-04-25T01:33:00.621999+0000","last_became_active":"2024-04-25T01:30:38.449971+0000","last_became_peered":"2024-04-25T01:30:38.449971+0000","last_unstale":"2024-04-25T01:33:00.621999+0000","last_undegraded":"2024-04-25T01:33:00.621999+0000","last_fullsized":"2024-04-25T01:33:00.621999+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:52:37.046462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'47","reported_seq":100,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.847942+0000","last_change":"2024-04-25T01:30:33.348547+0000","last_active":"2024-04-25T01:33:00.847942+0000","last_peered":"2024-04-25T01:33:00.847942+0000","last_clean":"2024-04-25T01:33:00.847942+0000","last_became_active":"2024-04-25T01:30:33.348181+0000","last_became_peered":"2024-04-25T01:30:33.348181+0000","last_unstale":"2024-04-25T01:33:00.847942+0000","last_undegraded":"2024-04-25T01:33:00.847942+0000","last_fullsized":"2024-04-25T01:33:00.847942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:11:58.696829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":82,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622499+0000","last_change":"2024-04-25T01:30:34.351987+0000","last_active":"2024-04-25T01:33:00.622499+0000","last_peered":"2024-04-25T01:33:00.622499+0000","last_clean":"2024-04-25T01:33:00.622499+0000","last_became_active":"2024-04-25T01:30:34.351571+0000","last_became_peered":"2024-04-25T01:30:34.351571+0000","last_unstale":"2024-04-25T01:33:00.622499+0000","last_undegraded":"2024-04-25T01:33:00.622499+0000","last_fullsized":"2024-04-25T01:33:00.622499+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:23.585865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.625691+0000","last_change":"2024-04-25T01:30:38.372566+0000","last_active":"2024-04-25T01:33:00.625691+0000","last_peered":"2024-04-25T01:33:00.625691+0000","last_clean":"2024-04-25T01:33:00.625691+0000","last_became_active":"2024-04-25T01:30:38.372102+0000","last_became_peered":"2024-04-25T01:30:38.372102+0000","last_unstale":"2024-04-25T01:33:00.625691+0000","last_undegraded":"2024-04-25T01:33:00.625691+0000","last_fullsized":"2024-04-25T01:33:00.625691+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:22:16.791842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642090+0000","last_change":"2024-04-25T01:30:33.344065+0000","last_active":"2024-04-25T01:33:00.642090+0000","last_peered":"2024-04-25T01:33:00.642090+0000","last_clean":"2024-04-25T01:33:00.642090+0000","last_became_active":"2024-04-25T01:30:33.343700+0000","last_became_peered":"2024-04-25T01:30:33.343700+0000","last_unstale":"2024-04-25T01:33:00.642090+0000","last_undegraded":"2024-04-25T01:33:00.642090+0000","last_fullsized":"2024-04-25T01:33:00.642090+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:54:08.265744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643124+0000","last_change":"2024-04-25T01:30:34.354644+0000","last_active":"2024-04-25T01:33:00.643124+0000","last_peered":"2024-04-25T01:33:00.643124+0000","last_clean":"2024-04-25T01:33:00.643124+0000","last_became_active":"2024-04-25T01:30:34.354330+0000","last_became_peered":"2024-04-25T01:30:34.354330+0000","last_unstale":"2024-04-25T01:33:00.643124+0000","last_undegraded":"2024-04-25T01:33:00.643124+0000","last_fullsized":"2024-04-25T01:33:00.643124+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:31:50.399983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641601+0000","last_change":"2024-04-25T01:30:38.450743+0000","last_active":"2024-04-25T01:33:00.641601+0000","last_peered":"2024-04-25T01:33:00.641601+0000","last_clean":"2024-04-25T01:33:00.641601+0000","last_became_active":"2024-04-25T01:30:38.450303+0000","last_became_peered":"2024-04-25T01:30:38.450303+0000","last_unstale":"2024-04-25T01:33:00.641601+0000","last_undegraded":"2024-04-25T01:33:00.641601+0000","last_fullsized":"2024-04-25T01:33:00.641601+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:37.919250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640527+0000","last_change":"2024-04-25T01:30:33.350709+0000","last_active":"2024-04-25T01:33:00.640527+0000","last_peered":"2024-04-25T01:33:00.640527+0000","last_clean":"2024-04-25T01:33:00.640527+0000","last_became_active":"2024-04-25T01:30:33.350387+0000","last_became_peered":"2024-04-25T01:30:33.350387+0000","last_unstale":"2024-04-25T01:33:00.640527+0000","last_undegraded":"2024-04-25T01:33:00.640527+0000","last_fullsized":"2024-04-25T01:33:00.640527+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:30:01.252121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730287+0000","last_change":"2024-04-25T01:30:34.362040+0000","last_active":"2024-04-25T01:33:00.730287+0000","last_peered":"2024-04-25T01:33:00.730287+0000","last_clean":"2024-04-25T01:33:00.730287+0000","last_became_active":"2024-04-25T01:30:34.361734+0000","last_became_peered":"2024-04-25T01:30:34.361734+0000","last_unstale":"2024-04-25T01:33:00.730287+0000","last_undegraded":"2024-04-25T01:33:00.730287+0000","last_fullsized":"2024-04-25T01:33:00.730287+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:11:28.409764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641032+0000","last_change":"2024-04-25T01:31:21.877148+0000","last_active":"2024-04-25T01:33:00.641032+0000","last_peered":"2024-04-25T01:33:00.641032+0000","last_clean":"2024-04-25T01:33:00.641032+0000","last_became_active":"2024-04-25T01:31:21.876665+0000","last_became_peered":"2024-04-25T01:31:21.876665+0000","last_unstale":"2024-04-25T01:33:00.641032+0000","last_undegraded":"2024-04-25T01:33:00.641032+0000","last_fullsized":"2024-04-25T01:33:00.641032+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:32:38.425434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,8],"acting":[9,3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622123+0000","last_change":"2024-04-25T01:30:33.350207+0000","last_active":"2024-04-25T01:33:00.622123+0000","last_peered":"2024-04-25T01:33:00.622123+0000","last_clean":"2024-04-25T01:33:00.622123+0000","last_became_active":"2024-04-25T01:30:33.349892+0000","last_became_peered":"2024-04-25T01:30:33.349892+0000","last_unstale":"2024-04-25T01:33:00.622123+0000","last_undegraded":"2024-04-25T01:33:00.622123+0000","last_fullsized":"2024-04-25T01:33:00.622123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:26:35.186486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640590+0000","last_change":"2024-04-25T01:30:34.355196+0000","last_active":"2024-04-25T01:33:00.640590+0000","last_peered":"2024-04-25T01:33:00.640590+0000","last_clean":"2024-04-25T01:33:00.640590+0000","last_became_active":"2024-04-25T01:30:34.354794+0000","last_became_peered":"2024-04-25T01:30:34.354794+0000","last_unstale":"2024-04-25T01:33:00.640590+0000","last_undegraded":"2024-04-25T01:33:00.640590+0000","last_fullsized":"2024-04-25T01:33:00.640590+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:43:37.495974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641714+0000","last_change":"2024-04-25T01:30:38.384159+0000","last_active":"2024-04-25T01:33:00.641714+0000","last_peered":"2024-04-25T01:33:00.641714+0000","last_clean":"2024-04-25T01:33:00.641714+0000","last_became_active":"2024-04-25T01:30:38.383676+0000","last_became_peered":"2024-04-25T01:30:38.383676+0000","last_unstale":"2024-04-25T01:33:00.641714+0000","last_undegraded":"2024-04-25T01:33:00.641714+0000","last_fullsized":"2024-04-25T01:33:00.641714+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:22:28.491522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":19,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640644+0000","last_change":"2024-04-25T01:31:21.889311+0000","last_active":"2024-04-25T01:33:00.640644+0000","last_peered":"2024-04-25T01:33:00.640644+0000","last_clean":"2024-04-25T01:33:00.640644+0000","last_became_active":"2024-04-25T01:31:21.877027+0000","last_became_peered":"2024-04-25T01:31:21.877027+0000","last_unstale":"2024-04-25T01:33:00.640644+0000","last_undegraded":"2024-04-25T01:33:00.640644+0000","last_fullsized":"2024-04-25T01:33:00.640644+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:02.206671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622555+0000","last_change":"2024-04-25T01:30:34.358110+0000","last_active":"2024-04-25T01:33:00.622555+0000","last_peered":"2024-04-25T01:33:00.622555+0000","last_clean":"2024-04-25T01:33:00.622555+0000","last_became_active":"2024-04-25T01:30:34.357555+0000","last_became_peered":"2024-04-25T01:30:34.357555+0000","last_unstale":"2024-04-25T01:33:00.622555+0000","last_undegraded":"2024-04-25T01:33:00.622555+0000","last_fullsized":"2024-04-25T01:33:00.622555+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:30:11.636320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641877+0000","last_change":"2024-04-25T01:30:38.450680+0000","last_active":"2024-04-25T01:33:00.641877+0000","last_peered":"2024-04-25T01:33:00.641877+0000","last_clean":"2024-04-25T01:33:00.641877+0000","last_became_active":"2024-04-25T01:30:38.450168+0000","last_became_peered":"2024-04-25T01:30:38.450168+0000","last_unstale":"2024-04-25T01:33:00.641877+0000","last_undegraded":"2024-04-25T01:33:00.641877+0000","last_fullsized":"2024-04-25T01:33:00.641877+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:39:24.236371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.730493+0000","last_change":"2024-04-25T01:30:33.343848+0000","last_active":"2024-04-25T01:33:00.730493+0000","last_peered":"2024-04-25T01:33:00.730493+0000","last_clean":"2024-04-25T01:33:00.730493+0000","last_became_active":"2024-04-25T01:30:33.343498+0000","last_became_peered":"2024-04-25T01:30:33.343498+0000","last_unstale":"2024-04-25T01:33:00.730493+0000","last_undegraded":"2024-04-25T01:33:00.730493+0000","last_fullsized":"2024-04-25T01:33:00.730493+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:58:09.184571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642308+0000","last_change":"2024-04-25T01:30:34.361994+0000","last_active":"2024-04-25T01:33:00.642308+0000","last_peered":"2024-04-25T01:33:00.642308+0000","last_clean":"2024-04-25T01:33:00.642308+0000","last_became_active":"2024-04-25T01:30:34.361532+0000","last_became_peered":"2024-04-25T01:30:34.361532+0000","last_unstale":"2024-04-25T01:33:00.642308+0000","last_undegraded":"2024-04-25T01:33:00.642308+0000","last_fullsized":"2024-04-25T01:33:00.642308+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:30.667672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640976+0000","last_change":"2024-04-25T01:30:38.369041+0000","last_active":"2024-04-25T01:33:00.640976+0000","last_peered":"2024-04-25T01:33:00.640976+0000","last_clean":"2024-04-25T01:33:00.640976+0000","last_became_active":"2024-04-25T01:30:38.368586+0000","last_became_peered":"2024-04-25T01:30:38.368586+0000","last_unstale":"2024-04-25T01:33:00.640976+0000","last_undegraded":"2024-04-25T01:33:00.640976+0000","last_fullsized":"2024-04-25T01:33:00.640976+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:58:53.743012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'7","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.626422+0000","last_change":"2024-04-25T01:30:33.345501+0000","last_active":"2024-04-25T01:33:00.626422+0000","last_peered":"2024-04-25T01:33:00.626422+0000","last_clean":"2024-04-25T01:33:00.626422+0000","last_became_active":"2024-04-25T01:30:33.345179+0000","last_became_peered":"2024-04-25T01:30:33.345179+0000","last_unstale":"2024-04-25T01:33:00.626422+0000","last_undegraded":"2024-04-25T01:33:00.626422+0000","last_fullsized":"2024-04-25T01:33:00.626422+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:29:41.275480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642445+0000","last_change":"2024-04-25T01:30:34.361631+0000","last_active":"2024-04-25T01:33:00.642445+0000","last_peered":"2024-04-25T01:33:00.642445+0000","last_clean":"2024-04-25T01:33:00.642445+0000","last_became_active":"2024-04-25T01:30:34.360266+0000","last_became_peered":"2024-04-25T01:30:34.360266+0000","last_unstale":"2024-04-25T01:33:00.642445+0000","last_undegraded":"2024-04-25T01:33:00.642445+0000","last_fullsized":"2024-04-25T01:33:00.642445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:25:03.094166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642246+0000","last_change":"2024-04-25T01:30:38.380779+0000","last_active":"2024-04-25T01:33:00.642246+0000","last_peered":"2024-04-25T01:33:00.642246+0000","last_clean":"2024-04-25T01:33:00.642246+0000","last_became_active":"2024-04-25T01:30:38.379959+0000","last_became_peered":"2024-04-25T01:30:38.379959+0000","last_unstale":"2024-04-25T01:33:00.642246+0000","last_undegraded":"2024-04-25T01:33:00.642246+0000","last_fullsized":"2024-04-25T01:33:00.642246+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:48:01.816311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'41","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.845488+0000","last_change":"2024-04-25T01:30:33.345730+0000","last_active":"2024-04-25T01:33:00.845488+0000","last_peered":"2024-04-25T01:33:00.845488+0000","last_clean":"2024-04-25T01:33:00.845488+0000","last_became_active":"2024-04-25T01:30:33.344832+0000","last_became_peered":"2024-04-25T01:30:33.344832+0000","last_unstale":"2024-04-25T01:33:00.845488+0000","last_undegraded":"2024-04-25T01:33:00.845488+0000","last_fullsized":"2024-04-25T01:33:00.845488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:10:47.018811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642105+0000","last_change":"2024-04-25T01:30:34.355324+0000","last_active":"2024-04-25T01:33:00.642105+0000","last_peered":"2024-04-25T01:33:00.642105+0000","last_clean":"2024-04-25T01:33:00.642105+0000","last_became_active":"2024-04-25T01:30:34.354954+0000","last_became_peered":"2024-04-25T01:30:34.354954+0000","last_unstale":"2024-04-25T01:33:00.642105+0000","last_undegraded":"2024-04-25T01:33:00.642105+0000","last_fullsized":"2024-04-25T01:33:00.642105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:36:41.932391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640928+0000","last_change":"2024-04-25T01:30:38.374386+0000","last_active":"2024-04-25T01:33:00.640928+0000","last_peered":"2024-04-25T01:33:00.640928+0000","last_clean":"2024-04-25T01:33:00.640928+0000","last_became_active":"2024-04-25T01:30:38.373454+0000","last_became_peered":"2024-04-25T01:30:38.373454+0000","last_unstale":"2024-04-25T01:33:00.640928+0000","last_undegraded":"2024-04-25T01:33:00.640928+0000","last_fullsized":"2024-04-25T01:33:00.640928+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:37.352147+0000","last_clean_scrub_stamp":"2024-04-25T01:30:37.352147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:54.456218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"91'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643371+0000","last_change":"2024-04-25T01:30:33.339821+0000","last_active":"2024-04-25T01:33:00.643371+0000","last_peered":"2024-04-25T01:33:00.643371+0000","last_clean":"2024-04-25T01:33:00.643371+0000","last_became_active":"2024-04-25T01:30:33.339500+0000","last_became_peered":"2024-04-25T01:30:33.339500+0000","last_unstale":"2024-04-25T01:33:00.643371+0000","last_undegraded":"2024-04-25T01:33:00.643371+0000","last_fullsized":"2024-04-25T01:33:00.643371+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:32.326042+0000","last_clean_scrub_stamp":"2024-04-25T01:30:32.326042+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:44:25.880828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642426+0000","last_change":"2024-04-25T01:30:34.362736+0000","last_active":"2024-04-25T01:33:00.642426+0000","last_peered":"2024-04-25T01:33:00.642426+0000","last_clean":"2024-04-25T01:33:00.642426+0000","last_became_active":"2024-04-25T01:30:34.362371+0000","last_became_peered":"2024-04-25T01:30:34.362371+0000","last_unstale":"2024-04-25T01:33:00.642426+0000","last_undegraded":"2024-04-25T01:33:00.642426+0000","last_fullsized":"2024-04-25T01:33:00.642426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:57:36.088231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.640673+0000","last_change":"2024-04-25T01:30:34.359217+0000","last_active":"2024-04-25T01:33:00.640673+0000","last_peered":"2024-04-25T01:33:00.640673+0000","last_clean":"2024-04-25T01:33:00.640673+0000","last_became_active":"2024-04-25T01:30:34.357095+0000","last_became_peered":"2024-04-25T01:30:34.357095+0000","last_unstale":"2024-04-25T01:33:00.640673+0000","last_undegraded":"2024-04-25T01:33:00.640673+0000","last_fullsized":"2024-04-25T01:33:00.640673+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:58.603484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.625896+0000","last_change":"2024-04-25T01:30:34.359513+0000","last_active":"2024-04-25T01:33:00.625896+0000","last_peered":"2024-04-25T01:33:00.625896+0000","last_clean":"2024-04-25T01:33:00.625896+0000","last_became_active":"2024-04-25T01:30:34.359089+0000","last_became_peered":"2024-04-25T01:30:34.359089+0000","last_unstale":"2024-04-25T01:33:00.625896+0000","last_undegraded":"2024-04-25T01:33:00.625896+0000","last_fullsized":"2024-04-25T01:33:00.625896+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:36.159703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.625752+0000","last_change":"2024-04-25T01:30:34.368739+0000","last_active":"2024-04-25T01:33:00.625752+0000","last_peered":"2024-04-25T01:33:00.625752+0000","last_clean":"2024-04-25T01:33:00.625752+0000","last_became_active":"2024-04-25T01:30:34.368392+0000","last_became_peered":"2024-04-25T01:30:34.368392+0000","last_unstale":"2024-04-25T01:33:00.625752+0000","last_undegraded":"2024-04-25T01:33:00.625752+0000","last_fullsized":"2024-04-25T01:33:00.625752+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:39:32.241889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.642467+0000","last_change":"2024-04-25T01:31:21.877568+0000","last_active":"2024-04-25T01:33:00.642467+0000","last_peered":"2024-04-25T01:33:00.642467+0000","last_clean":"2024-04-25T01:33:00.642467+0000","last_became_active":"2024-04-25T01:31:21.877071+0000","last_became_peered":"2024-04-25T01:31:21.877071+0000","last_unstale":"2024-04-25T01:33:00.642467+0000","last_undegraded":"2024-04-25T01:33:00.642467+0000","last_fullsized":"2024-04-25T01:33:00.642467+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:02:24.789974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643046+0000","last_change":"2024-04-25T01:30:34.360979+0000","last_active":"2024-04-25T01:33:00.643046+0000","last_peered":"2024-04-25T01:33:00.643046+0000","last_clean":"2024-04-25T01:33:00.643046+0000","last_became_active":"2024-04-25T01:30:34.360630+0000","last_became_peered":"2024-04-25T01:30:34.360630+0000","last_unstale":"2024-04-25T01:33:00.643046+0000","last_undegraded":"2024-04-25T01:33:00.643046+0000","last_fullsized":"2024-04-25T01:33:00.643046+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:05:19.284577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641137+0000","last_change":"2024-04-25T01:30:34.362621+0000","last_active":"2024-04-25T01:33:00.641137+0000","last_peered":"2024-04-25T01:33:00.641137+0000","last_clean":"2024-04-25T01:33:00.641137+0000","last_became_active":"2024-04-25T01:30:34.362290+0000","last_became_peered":"2024-04-25T01:30:34.362290+0000","last_unstale":"2024-04-25T01:33:00.641137+0000","last_undegraded":"2024-04-25T01:33:00.641137+0000","last_fullsized":"2024-04-25T01:33:00.641137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:28:07.750906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.626403+0000","last_change":"2024-04-25T01:30:34.351339+0000","last_active":"2024-04-25T01:33:00.626403+0000","last_peered":"2024-04-25T01:33:00.626403+0000","last_clean":"2024-04-25T01:33:00.626403+0000","last_became_active":"2024-04-25T01:30:34.350655+0000","last_became_peered":"2024-04-25T01:30:34.350655+0000","last_unstale":"2024-04-25T01:33:00.626403+0000","last_undegraded":"2024-04-25T01:33:00.626403+0000","last_fullsized":"2024-04-25T01:33:00.626403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:05.322547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622618+0000","last_change":"2024-04-25T01:30:34.353876+0000","last_active":"2024-04-25T01:33:00.622618+0000","last_peered":"2024-04-25T01:33:00.622618+0000","last_clean":"2024-04-25T01:33:00.622618+0000","last_became_active":"2024-04-25T01:30:34.352942+0000","last_became_peered":"2024-04-25T01:30:34.352942+0000","last_unstale":"2024-04-25T01:33:00.622618+0000","last_undegraded":"2024-04-25T01:33:00.622618+0000","last_fullsized":"2024-04-25T01:33:00.622618+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:47.260171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641358+0000","last_change":"2024-04-25T01:30:34.352039+0000","last_active":"2024-04-25T01:33:00.641358+0000","last_peered":"2024-04-25T01:33:00.641358+0000","last_clean":"2024-04-25T01:33:00.641358+0000","last_became_active":"2024-04-25T01:30:34.351612+0000","last_became_peered":"2024-04-25T01:30:34.351612+0000","last_unstale":"2024-04-25T01:33:00.641358+0000","last_undegraded":"2024-04-25T01:33:00.641358+0000","last_fullsized":"2024-04-25T01:33:00.641358+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:03.607711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.622510+0000","last_change":"2024-04-25T01:30:34.353692+0000","last_active":"2024-04-25T01:33:00.622510+0000","last_peered":"2024-04-25T01:33:00.622510+0000","last_clean":"2024-04-25T01:33:00.622510+0000","last_became_active":"2024-04-25T01:30:34.353035+0000","last_became_peered":"2024-04-25T01:30:34.353035+0000","last_unstale":"2024-04-25T01:33:00.622510+0000","last_undegraded":"2024-04-25T01:33:00.622510+0000","last_fullsized":"2024-04-25T01:33:00.622510+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:15:54.540315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641104+0000","last_change":"2024-04-25T01:30:34.370797+0000","last_active":"2024-04-25T01:33:00.641104+0000","last_peered":"2024-04-25T01:33:00.641104+0000","last_clean":"2024-04-25T01:33:00.641104+0000","last_became_active":"2024-04-25T01:30:34.367533+0000","last_became_peered":"2024-04-25T01:30:34.367533+0000","last_unstale":"2024-04-25T01:33:00.641104+0000","last_undegraded":"2024-04-25T01:33:00.641104+0000","last_fullsized":"2024-04-25T01:33:00.641104+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:27:15.478808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.643248+0000","last_change":"2024-04-25T01:30:34.364843+0000","last_active":"2024-04-25T01:33:00.643248+0000","last_peered":"2024-04-25T01:33:00.643248+0000","last_clean":"2024-04-25T01:33:00.643248+0000","last_became_active":"2024-04-25T01:30:34.364515+0000","last_became_peered":"2024-04-25T01:30:34.364515+0000","last_unstale":"2024-04-25T01:33:00.643248+0000","last_undegraded":"2024-04-25T01:33:00.643248+0000","last_fullsized":"2024-04-25T01:33:00.643248+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:50:03.517814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T01:33:00.641005+0000","last_change":"2024-04-25T01:30:34.363715+0000","last_active":"2024-04-25T01:33:00.641005+0000","last_peered":"2024-04-25T01:33:00.641005+0000","last_clean":"2024-04-25T01:33:00.641005+0000","last_became_active":"2024-04-25T01:30:34.363383+0000","last_became_peered":"2024-04-25T01:30:34.363383+0000","last_unstale":"2024-04-25T01:33:00.641005+0000","last_undegraded":"2024-04-25T01:33:00.641005+0000","last_fullsized":"2024-04-25T01:33:00.641005+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:30:33.327549+0000","last_clean_scrub_stamp":"2024-04-25T01:30:33.327549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:50:18.322550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":601378,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1337,"num_read_kb":786,"num_write":373,"num_write_kb":1470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":1804134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":303,"ondisk_log_size":303,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318314,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1077248,"data_stored":519949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579951,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37196,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707956,"statfs":{"total":95995035648,"available":95956946944,"internally_reserved":0,"allocated":1904640,"data_stored":1306841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743477,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1490944,"data_stored":916591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005114,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":509849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266751,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28668,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716484,"statfs":{"total":95995035648,"available":95965679616,"internally_reserved":0,"allocated":1687552,"data_stored":1116994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430277,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1069056,"data_stored":514783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1142784,"data_stored":583799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1093632,"data_stored":516771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36468,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708684,"statfs":{"total":95995035648,"available":95957692416,"internally_reserved":0,"allocated":1155072,"data_stored":585015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36908,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708244,"statfs":{"total":95995035648,"available":95957241856,"internally_reserved":0,"allocated":1609728,"data_stored":1039307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888142,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":519947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691914,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36996,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708156,"statfs":{"total":95995035648,"available":95957151744,"internally_reserved":0,"allocated":1695744,"data_stored":1111877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":70954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":525246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":134952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":394716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":394338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":202412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:33:03.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:33:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.946+0000 7f67fe89a640 1 -- 172.21.15.28:0/257071631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f8100680 msgr2=0x7f67f8100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.946+0000 7f67fe89a640 1 --2- 172.21.15.28:0/257071631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f8100680 0x7f67f8100aa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f67e8009860 tx=0x7f67e802a340 comp rx=0 tx=0).stop 2024-04-25T01:33:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.946+0000 7f67fe89a640 1 -- 172.21.15.28:0/257071631 shutdown_connections 2024-04-25T01:33:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.946+0000 7f67fe89a640 1 --2- 172.21.15.28:0/257071631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f8102230 0x7f67f8108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.946+0000 7f67fe89a640 1 --2- 172.21.15.28:0/257071631 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67f8101870 0x7f67f8101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.946+0000 7f67fe89a640 1 --2- 172.21.15.28:0/257071631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f8100680 0x7f67f8100aa0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.946+0000 7f67fe89a640 1 -- 172.21.15.28:0/257071631 >> 172.21.15.28:0/257071631 conn(0x7f67f80fbbf0 msgr2=0x7f67f80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.946+0000 7f67fe89a640 1 -- 172.21.15.28:0/257071631 shutdown_connections 2024-04-25T01:33:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.946+0000 7f67fe89a640 1 -- 172.21.15.28:0/257071631 wait complete. 2024-04-25T01:33:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67fe89a640 1 Processor -- start 2024-04-25T01:33:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67fe89a640 1 -- start start 2024-04-25T01:33:03.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67fe89a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f8100680 0x7f67f81969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67fe89a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67f8101870 0x7f67f8196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67fe89a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f8102230 0x7f67f819df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67fe89a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f67f810c2d0 con 0x7f67f8102230 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67fe89a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67f810c150 con 0x7f67f8100680 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67fe89a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f67f810c450 con 0x7f67f8101870 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67fce37640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f8102230 0x7f67f819df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67f7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f8100680 0x7f67f81969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67fce37640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f8102230 0x7f67f819df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55138/0 (socket says 172.21.15.28:55138) 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67f7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f8100680 0x7f67f81969d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:37996/0 (socket says 172.21.15.28:37996) 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67fce37640 1 -- 172.21.15.28:0/1698511569 learned_addr learned my addr 172.21.15.28:0/1698511569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67f77fe640 1 --2- 172.21.15.28:0/1698511569 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67f8101870 0x7f67f8196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67f7fff640 1 -- 172.21.15.28:0/1698511569 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67f8101870 msgr2=0x7f67f8196f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67f7fff640 1 --2- 172.21.15.28:0/1698511569 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67f8101870 0x7f67f8196f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67f7fff640 1 -- 172.21.15.28:0/1698511569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f8102230 msgr2=0x7f67f819df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67f7fff640 1 --2- 172.21.15.28:0/1698511569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f8102230 0x7f67f819df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67f7fff640 1 -- 172.21.15.28:0/1698511569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f819e5f0 con 0x7f67f8100680 2024-04-25T01:33:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67f77fe640 1 --2- 172.21.15.28:0/1698511569 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67f8101870 0x7f67f8196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.947+0000 7f67fce37640 1 --2- 172.21.15.28:0/1698511569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f8102230 0x7f67f819df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.948+0000 7f67f7fff640 1 --2- 172.21.15.28:0/1698511569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f8100680 0x7f67f81969d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f67e8035040 tx=0x7f67e8033d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.948+0000 7f67f57fa640 1 -- 172.21.15.28:0/1698511569 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e8037040 con 0x7f67f8100680 2024-04-25T01:33:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.948+0000 7f67f57fa640 1 -- 172.21.15.28:0/1698511569 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67e8004040 con 0x7f67f8100680 2024-04-25T01:33:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.948+0000 7f67fe89a640 1 -- 172.21.15.28:0/1698511569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67f819e880 con 0x7f67f8100680 2024-04-25T01:33:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.948+0000 7f67f57fa640 1 -- 172.21.15.28:0/1698511569 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e8005650 con 0x7f67f8100680 2024-04-25T01:33:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.948+0000 7f67fe89a640 1 -- 172.21.15.28:0/1698511569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67f819ed70 con 0x7f67f8100680 2024-04-25T01:33:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.949+0000 7f67fe89a640 1 -- 172.21.15.28:0/1698511569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f8107a20 con 0x7f67f8100680 2024-04-25T01:33:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.950+0000 7f67f57fa640 1 -- 172.21.15.28:0/1698511569 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7f67e8042020 con 0x7f67f8100680 2024-04-25T01:33:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.951+0000 7f67f57fa640 1 --2- 172.21.15.28:0/1698511569 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f67e00a2d80 0x7f67e00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.951+0000 7f67f57fa640 1 -- 172.21.15.28:0/1698511569 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f67e80ef200 con 0x7f67f8100680 2024-04-25T01:33:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.951+0000 7f67f77fe640 1 --2- 172.21.15.28:0/1698511569 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f67e00a2d80 0x7f67e00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.951+0000 7f67f77fe640 1 --2- 172.21.15.28:0/1698511569 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f67e00a2d80 0x7f67e00a5240 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f67f8197ee0 tx=0x7f67ec005f90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:03.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:03.954+0000 7f67f57fa640 1 -- 172.21.15.28:0/1698511569 <== mon.2 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f67e8049020 con 0x7f67f8100680 2024-04-25T01:33:04.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:03 smithi028 ceph-mon[29390]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:33:04.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:03 smithi028 ceph-mon[29390]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-04-25T01:33:04.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:03 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:04.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:03 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:04.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:03 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:04.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:03 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:04.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:03 smithi028 ceph-mon[29390]: from='client.25286 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:33:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.146+0000 7f67fe89a640 1 -- 172.21.15.28:0/1698511569 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f67f8062340 con 0x7f67f8100680 2024-04-25T01:33:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.148+0000 7f67f57fa640 1 -- 172.21.15.28:0/1698511569 <== mon.2 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24941 (secure 0 0 0) 0x7f67e80bbb90 con 0x7f67f8100680 2024-04-25T01:33:04.125 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:33:04.126 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:33:00.617205+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/1149553567":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/500268183":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/3499593842":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/3630315328":"2024-04-26T01:33:00.617174+0000","172.21.15.28:6800/1149553567":"2024-04-26T01:33:00.617174+0000","172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3812371257":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:33:04.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.150+0000 7f67fe89a640 1 -- 172.21.15.28:0/1698511569 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f67e00a2d80 msgr2=0x7f67e00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.150+0000 7f67fe89a640 1 --2- 172.21.15.28:0/1698511569 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f67e00a2d80 0x7f67e00a5240 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f67f8197ee0 tx=0x7f67ec005f90 comp rx=0 tx=0).stop 2024-04-25T01:33:04.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.150+0000 7f67fe89a640 1 -- 172.21.15.28:0/1698511569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f8100680 msgr2=0x7f67f81969d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.150+0000 7f67fe89a640 1 --2- 172.21.15.28:0/1698511569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f8100680 0x7f67f81969d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f67e8035040 tx=0x7f67e8033d90 comp rx=0 tx=0).stop 2024-04-25T01:33:04.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.150+0000 7f67fe89a640 1 -- 172.21.15.28:0/1698511569 shutdown_connections 2024-04-25T01:33:04.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.150+0000 7f67fe89a640 1 --2- 172.21.15.28:0/1698511569 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f67e00a2d80 0x7f67e00a5240 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.150+0000 7f67fe89a640 1 --2- 172.21.15.28:0/1698511569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f8102230 0x7f67f819df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.150+0000 7f67fe89a640 1 --2- 172.21.15.28:0/1698511569 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f67f8101870 0x7f67f8196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.150+0000 7f67fe89a640 1 --2- 172.21.15.28:0/1698511569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f8100680 0x7f67f81969d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.150+0000 7f67fe89a640 1 -- 172.21.15.28:0/1698511569 >> 172.21.15.28:0/1698511569 conn(0x7f67f80fbbf0 msgr2=0x7f67f8104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:04.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.150+0000 7f67fe89a640 1 -- 172.21.15.28:0/1698511569 shutdown_connections 2024-04-25T01:33:04.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.150+0000 7f67fe89a640 1 -- 172.21.15.28:0/1698511569 wait complete. 2024-04-25T01:33:04.139 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:25:48.767414+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:30:32.282430+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:30:33.127049+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:30:36.796132+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:33:04.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:33:04.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:03 smithi130 ceph-mon[32702]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:33:04.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:03 smithi130 ceph-mon[32702]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-04-25T01:33:04.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:03 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:04.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:03 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:04.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:03 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:04.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:03 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:04.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:03 smithi130 ceph-mon[32702]: from='client.25286 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:33:04.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:03 smithi032 ceph-mon[32461]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:33:04.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:03 smithi032 ceph-mon[32461]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-04-25T01:33:04.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:03 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:04.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:03 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:04.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:03 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:04.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:03 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:04.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:03 smithi032 ceph-mon[32461]: from='client.25286 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:33:04.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.272+0000 7f9a2c0f5640 1 -- 172.21.15.28:0/3088137975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24102200 msgr2=0x7f9a241045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.272+0000 7f9a2c0f5640 1 --2- 172.21.15.28:0/3088137975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24102200 0x7f9a241045f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9a20009900 tx=0x7f9a2002a410 comp rx=0 tx=0).stop 2024-04-25T01:33:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.272+0000 7f9a2c0f5640 1 -- 172.21.15.28:0/3088137975 shutdown_connections 2024-04-25T01:33:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.272+0000 7f9a2c0f5640 1 --2- 172.21.15.28:0/3088137975 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a24107460 0x7f9a241098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.272+0000 7f9a2c0f5640 1 --2- 172.21.15.28:0/3088137975 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a24104b30 0x7f9a24106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.272+0000 7f9a2c0f5640 1 --2- 172.21.15.28:0/3088137975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24102200 0x7f9a241045f0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.272+0000 7f9a2c0f5640 1 -- 172.21.15.28:0/3088137975 >> 172.21.15.28:0/3088137975 conn(0x7f9a240fbbf0 msgr2=0x7f9a240fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.272+0000 7f9a2c0f5640 1 -- 172.21.15.28:0/3088137975 shutdown_connections 2024-04-25T01:33:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.272+0000 7f9a2c0f5640 1 -- 172.21.15.28:0/3088137975 wait complete. 2024-04-25T01:33:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.273+0000 7f9a2c0f5640 1 Processor -- start 2024-04-25T01:33:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.273+0000 7f9a2c0f5640 1 -- start start 2024-04-25T01:33:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.273+0000 7f9a2c0f5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a24102200 0x7f9a24197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.273+0000 7f9a2c0f5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a24104b30 0x7f9a24198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.273+0000 7f9a2c0f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24107460 0x7f9a2419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.273+0000 7f9a2c0f5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2410d470 con 0x7f9a24107460 2024-04-25T01:33:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.273+0000 7f9a2c0f5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2410d2f0 con 0x7f9a24102200 2024-04-25T01:33:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.273+0000 7f9a2c0f5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2410d5f0 con 0x7f9a24104b30 2024-04-25T01:33:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.273+0000 7f9a2a692640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24107460 0x7f9a2419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.273+0000 7f9a29e91640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a24102200 0x7f9a24197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.273+0000 7f9a2a692640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24107460 0x7f9a2419f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55144/0 (socket says 172.21.15.28:55144) 2024-04-25T01:33:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.273+0000 7f9a2a692640 1 -- 172.21.15.28:0/3302292741 learned_addr learned my addr 172.21.15.28:0/3302292741 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.273+0000 7f9a29e91640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a24102200 0x7f9a24197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:38026/0 (socket says 172.21.15.28:38026) 2024-04-25T01:33:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.273+0000 7f9a29690640 1 --2- 172.21.15.28:0/3302292741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a24104b30 0x7f9a24198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.274+0000 7f9a2a692640 1 -- 172.21.15.28:0/3302292741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a24104b30 msgr2=0x7f9a24198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.274+0000 7f9a2a692640 1 --2- 172.21.15.28:0/3302292741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a24104b30 0x7f9a24198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.274+0000 7f9a2a692640 1 -- 172.21.15.28:0/3302292741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a24102200 msgr2=0x7f9a24197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.274+0000 7f9a2a692640 1 --2- 172.21.15.28:0/3302292741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a24102200 0x7f9a24197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.274+0000 7f9a2a692640 1 -- 172.21.15.28:0/3302292741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a2419f710 con 0x7f9a24107460 2024-04-25T01:33:04.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.274+0000 7f9a2a692640 1 --2- 172.21.15.28:0/3302292741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24107460 0x7f9a2419f0a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00eb10 tx=0x7f9a1c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.274+0000 7f9a1affd640 1 -- 172.21.15.28:0/3302292741 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c004b20 con 0x7f9a24107460 2024-04-25T01:33:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.274+0000 7f9a2c0f5640 1 -- 172.21.15.28:0/3302292741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a2419fa00 con 0x7f9a24107460 2024-04-25T01:33:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.274+0000 7f9a1affd640 1 -- 172.21.15.28:0/3302292741 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a1c004cc0 con 0x7f9a24107460 2024-04-25T01:33:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.274+0000 7f9a1affd640 1 -- 172.21.15.28:0/3302292741 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c010760 con 0x7f9a24107460 2024-04-25T01:33:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.274+0000 7f9a2c0f5640 1 -- 172.21.15.28:0/3302292741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a2419ffa0 con 0x7f9a24107460 2024-04-25T01:33:04.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.275+0000 7f9a2c0f5640 1 -- 172.21.15.28:0/3302292741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a240fd6e0 con 0x7f9a24107460 2024-04-25T01:33:04.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.276+0000 7f9a1affd640 1 -- 172.21.15.28:0/3302292741 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7f9a1c005000 con 0x7f9a24107460 2024-04-25T01:33:04.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.277+0000 7f9a1affd640 1 --2- 172.21.15.28:0/3302292741 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9a100a2d00 0x7f9a100a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.277+0000 7f9a29e91640 1 --2- 172.21.15.28:0/3302292741 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9a100a2d00 0x7f9a100a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.277+0000 7f9a1affd640 1 -- 172.21.15.28:0/3302292741 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f9a1c0d01b0 con 0x7f9a24107460 2024-04-25T01:33:04.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.278+0000 7f9a29e91640 1 --2- 172.21.15.28:0/3302292741 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9a100a2d00 0x7f9a100a51c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9a20031440 tx=0x7f9a200315b0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:04.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.283+0000 7f9a1affd640 1 -- 172.21.15.28:0/3302292741 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9a1c09ca90 con 0x7f9a24107460 2024-04-25T01:33:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.482+0000 7f9a2c0f5640 1 -- 172.21.15.28:0/3302292741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9a2402d3c0 con 0x7f9a24107460 2024-04-25T01:33:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.482+0000 7f9a1affd640 1 -- 172.21.15.28:0/3302292741 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f9a1c09c390 con 0x7f9a24107460 2024-04-25T01:33:04.460 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T01:33:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.484+0000 7f9a2c0f5640 1 -- 172.21.15.28:0/3302292741 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9a100a2d00 msgr2=0x7f9a100a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.484+0000 7f9a2c0f5640 1 --2- 172.21.15.28:0/3302292741 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9a100a2d00 0x7f9a100a51c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9a20031440 tx=0x7f9a200315b0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.484+0000 7f9a2c0f5640 1 -- 172.21.15.28:0/3302292741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24107460 msgr2=0x7f9a2419f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.484+0000 7f9a2c0f5640 1 --2- 172.21.15.28:0/3302292741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24107460 0x7f9a2419f0a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00eb10 tx=0x7f9a1c00c2d0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.484+0000 7f9a2c0f5640 1 -- 172.21.15.28:0/3302292741 shutdown_connections 2024-04-25T01:33:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.484+0000 7f9a2c0f5640 1 --2- 172.21.15.28:0/3302292741 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9a100a2d00 0x7f9a100a51c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.484+0000 7f9a2c0f5640 1 --2- 172.21.15.28:0/3302292741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a24107460 0x7f9a2419f0a0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.484+0000 7f9a2c0f5640 1 --2- 172.21.15.28:0/3302292741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a24104b30 0x7f9a24198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.484+0000 7f9a2c0f5640 1 --2- 172.21.15.28:0/3302292741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a24102200 0x7f9a24197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.484+0000 7f9a2c0f5640 1 -- 172.21.15.28:0/3302292741 >> 172.21.15.28:0/3302292741 conn(0x7f9a240fbbf0 msgr2=0x7f9a24102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.484+0000 7f9a2c0f5640 1 -- 172.21.15.28:0/3302292741 shutdown_connections 2024-04-25T01:33:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.484+0000 7f9a2c0f5640 1 -- 172.21.15.28:0/3302292741 wait complete. 2024-04-25T01:33:04.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:33:04.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.629+0000 7f2001658640 1 -- 172.21.15.28:0/1876808851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ffc0ffe00 msgr2=0x7f1ffc100220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.629+0000 7f2001658640 1 --2- 172.21.15.28:0/1876808851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ffc0ffe00 0x7f1ffc100220 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1fec009900 tx=0x7f1fec02a410 comp rx=0 tx=0).stop 2024-04-25T01:33:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.629+0000 7f2001658640 1 -- 172.21.15.28:0/1876808851 shutdown_connections 2024-04-25T01:33:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.629+0000 7f2001658640 1 --2- 172.21.15.28:0/1876808851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ffc1019b0 0x7f1ffc108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.629+0000 7f2001658640 1 --2- 172.21.15.28:0/1876808851 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ffc100ff0 0x7f1ffc101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.629+0000 7f2001658640 1 --2- 172.21.15.28:0/1876808851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ffc0ffe00 0x7f1ffc100220 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.629+0000 7f2001658640 1 -- 172.21.15.28:0/1876808851 >> 172.21.15.28:0/1876808851 conn(0x7f1ffc0fb370 msgr2=0x7f1ffc0fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.629+0000 7f2001658640 1 -- 172.21.15.28:0/1876808851 shutdown_connections 2024-04-25T01:33:04.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.629+0000 7f2001658640 1 -- 172.21.15.28:0/1876808851 wait complete. 2024-04-25T01:33:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.630+0000 7f2001658640 1 Processor -- start 2024-04-25T01:33:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.630+0000 7f2001658640 1 -- start start 2024-04-25T01:33:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.630+0000 7f2001658640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ffc0ffe00 0x7f1ffc197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.630+0000 7f2001658640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ffc100ff0 0x7f1ffc198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.630+0000 7f2001658640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ffc1019b0 0x7f1ffc19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.630+0000 7f2001658640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc10d410 con 0x7f1ffc100ff0 2024-04-25T01:33:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.630+0000 7f2001658640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc10d290 con 0x7f1ffc0ffe00 2024-04-25T01:33:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.630+0000 7f2001658640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc10d590 con 0x7f1ffc1019b0 2024-04-25T01:33:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.630+0000 7f1ffa7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ffc100ff0 0x7f1ffc198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.631+0000 7f1ffb7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ffc1019b0 0x7f1ffc19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.631+0000 7f1ffa7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ffc100ff0 0x7f1ffc198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55172/0 (socket says 172.21.15.28:55172) 2024-04-25T01:33:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.631+0000 7f1ffaffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ffc0ffe00 0x7f1ffc197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.631+0000 7f1ffa7fc640 1 -- 172.21.15.28:0/1627571869 learned_addr learned my addr 172.21.15.28:0/1627571869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.631+0000 7f1ffa7fc640 1 -- 172.21.15.28:0/1627571869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ffc1019b0 msgr2=0x7f1ffc19f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.631+0000 7f1ffa7fc640 1 --2- 172.21.15.28:0/1627571869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ffc1019b0 0x7f1ffc19f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.631+0000 7f1ffa7fc640 1 -- 172.21.15.28:0/1627571869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ffc0ffe00 msgr2=0x7f1ffc197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.631+0000 7f1ffa7fc640 1 --2- 172.21.15.28:0/1627571869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ffc0ffe00 0x7f1ffc197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.631+0000 7f1ffa7fc640 1 -- 172.21.15.28:0/1627571869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ffc19f6f0 con 0x7f1ffc100ff0 2024-04-25T01:33:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.631+0000 7f1ffb7fe640 1 --2- 172.21.15.28:0/1627571869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ffc1019b0 0x7f1ffc19f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.631+0000 7f1ffaffd640 1 --2- 172.21.15.28:0/1627571869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ffc0ffe00 0x7f1ffc197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.631+0000 7f1ffa7fc640 1 --2- 172.21.15.28:0/1627571869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ffc100ff0 0x7f1ffc198010 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1ff000d950 tx=0x7f1ff000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.631+0000 7f1fe3fff640 1 -- 172.21.15.28:0/1627571869 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0017030 con 0x7f1ffc100ff0 2024-04-25T01:33:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.631+0000 7f1fe3fff640 1 -- 172.21.15.28:0/1627571869 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ff0004020 con 0x7f1ffc100ff0 2024-04-25T01:33:04.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.631+0000 7f2001658640 1 -- 172.21.15.28:0/1627571869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ffc19f9e0 con 0x7f1ffc100ff0 2024-04-25T01:33:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.631+0000 7f1fe3fff640 1 -- 172.21.15.28:0/1627571869 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff00047a0 con 0x7f1ffc100ff0 2024-04-25T01:33:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.631+0000 7f2001658640 1 -- 172.21.15.28:0/1627571869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ffc19ff30 con 0x7f1ffc100ff0 2024-04-25T01:33:04.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.633+0000 7f2001658640 1 -- 172.21.15.28:0/1627571869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ffc1071a0 con 0x7f1ffc100ff0 2024-04-25T01:33:04.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.634+0000 7f1fe3fff640 1 -- 172.21.15.28:0/1627571869 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7f1ff00041d0 con 0x7f1ffc100ff0 2024-04-25T01:33:04.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.635+0000 7f1fe3fff640 1 --2- 172.21.15.28:0/1627571869 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1fe40a2d00 0x7f1fe40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.635+0000 7f1ffaffd640 1 --2- 172.21.15.28:0/1627571869 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1fe40a2d00 0x7f1fe40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.635+0000 7f1fe3fff640 1 -- 172.21.15.28:0/1627571869 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1ff00cf150 con 0x7f1ffc100ff0 2024-04-25T01:33:04.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.635+0000 7f1ffaffd640 1 --2- 172.21.15.28:0/1627571869 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1fe40a2d00 0x7f1fe40a51c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1fec005ab0 tx=0x7f1fec000bc0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:04.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.639+0000 7f1fe3fff640 1 -- 172.21.15.28:0/1627571869 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1ff009ba30 con 0x7f1ffc100ff0 2024-04-25T01:33:04.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.829+0000 7f2001658640 1 -- 172.21.15.28:0/1627571869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1ffc02d380 con 0x7f1ffc100ff0 2024-04-25T01:33:04.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.830+0000 7f1fe3fff640 1 -- 172.21.15.28:0/1627571869 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f1ff009b330 con 0x7f1ffc100ff0 2024-04-25T01:33:04.807 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:33:04.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.832+0000 7f2001658640 1 -- 172.21.15.28:0/1627571869 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1fe40a2d00 msgr2=0x7f1fe40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.832+0000 7f2001658640 1 --2- 172.21.15.28:0/1627571869 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1fe40a2d00 0x7f1fe40a51c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1fec005ab0 tx=0x7f1fec000bc0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.832+0000 7f2001658640 1 -- 172.21.15.28:0/1627571869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ffc100ff0 msgr2=0x7f1ffc198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.832+0000 7f2001658640 1 --2- 172.21.15.28:0/1627571869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ffc100ff0 0x7f1ffc198010 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1ff000d950 tx=0x7f1ff000de10 comp rx=0 tx=0).stop 2024-04-25T01:33:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.832+0000 7f2001658640 1 -- 172.21.15.28:0/1627571869 shutdown_connections 2024-04-25T01:33:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.832+0000 7f2001658640 1 --2- 172.21.15.28:0/1627571869 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1fe40a2d00 0x7f1fe40a51c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.832+0000 7f2001658640 1 --2- 172.21.15.28:0/1627571869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ffc1019b0 0x7f1ffc19f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.832+0000 7f2001658640 1 --2- 172.21.15.28:0/1627571869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ffc100ff0 0x7f1ffc198010 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.832+0000 7f2001658640 1 --2- 172.21.15.28:0/1627571869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ffc0ffe00 0x7f1ffc197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.832+0000 7f2001658640 1 -- 172.21.15.28:0/1627571869 >> 172.21.15.28:0/1627571869 conn(0x7f1ffc0fb370 msgr2=0x7f1ffc104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.832+0000 7f2001658640 1 -- 172.21.15.28:0/1627571869 shutdown_connections 2024-04-25T01:33:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.832+0000 7f2001658640 1 -- 172.21.15.28:0/1627571869 wait complete. 2024-04-25T01:33:04.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:33:04.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.954+0000 7f23d0a2a640 1 -- 172.21.15.28:0/2065906405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc101790 msgr2=0x7f23cc101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.954+0000 7f23d0a2a640 1 --2- 172.21.15.28:0/2065906405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc101790 0x7f23cc101bb0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009900 tx=0x7f23bc02a410 comp rx=0 tx=0).stop 2024-04-25T01:33:04.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.954+0000 7f23d0a2a640 1 -- 172.21.15.28:0/2065906405 shutdown_connections 2024-04-25T01:33:04.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.954+0000 7f23d0a2a640 1 --2- 172.21.15.28:0/2065906405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23cc103340 0x7f23cc109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.954+0000 7f23d0a2a640 1 --2- 172.21.15.28:0/2065906405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23cc102980 0x7f23cc102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.954+0000 7f23d0a2a640 1 --2- 172.21.15.28:0/2065906405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc101790 0x7f23cc101bb0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.954+0000 7f23d0a2a640 1 -- 172.21.15.28:0/2065906405 >> 172.21.15.28:0/2065906405 conn(0x7f23cc0fcd20 msgr2=0x7f23cc0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.954+0000 7f23d0a2a640 1 -- 172.21.15.28:0/2065906405 shutdown_connections 2024-04-25T01:33:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.954+0000 7f23d0a2a640 1 -- 172.21.15.28:0/2065906405 wait complete. 2024-04-25T01:33:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.955+0000 7f23d0a2a640 1 Processor -- start 2024-04-25T01:33:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.955+0000 7f23d0a2a640 1 -- start start 2024-04-25T01:33:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.955+0000 7f23d0a2a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23cc101790 0x7f23cc0754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.955+0000 7f23d0a2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc102980 0x7f23cc073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.955+0000 7f23d0a2a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23cc103340 0x7f23cc074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.955+0000 7f23d0a2a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f23cc10d400 con 0x7f23cc102980 2024-04-25T01:33:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.955+0000 7f23d0a2a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f23cc10d280 con 0x7f23cc103340 2024-04-25T01:33:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.955+0000 7f23d0a2a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f23cc10d580 con 0x7f23cc101790 2024-04-25T01:33:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.955+0000 7f23c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc102980 0x7f23cc073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.955+0000 7f23c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc102980 0x7f23cc073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55180/0 (socket says 172.21.15.28:55180) 2024-04-25T01:33:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.955+0000 7f23ca59c640 1 --2- 172.21.15.28:0/2617255544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23cc101790 0x7f23cc0754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.955+0000 7f23cad9d640 1 --2- 172.21.15.28:0/2617255544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23cc103340 0x7f23cc074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.955+0000 7f23c3fff640 1 -- 172.21.15.28:0/2617255544 learned_addr learned my addr 172.21.15.28:0/2617255544 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.956+0000 7f23c3fff640 1 -- 172.21.15.28:0/2617255544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23cc101790 msgr2=0x7f23cc0754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.956+0000 7f23c3fff640 1 --2- 172.21.15.28:0/2617255544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23cc101790 0x7f23cc0754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.956+0000 7f23c3fff640 1 -- 172.21.15.28:0/2617255544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23cc103340 msgr2=0x7f23cc074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.956+0000 7f23c3fff640 1 --2- 172.21.15.28:0/2617255544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23cc103340 0x7f23cc074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.956+0000 7f23c3fff640 1 -- 172.21.15.28:0/2617255544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23cc074900 con 0x7f23cc102980 2024-04-25T01:33:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.956+0000 7f23c3fff640 1 --2- 172.21.15.28:0/2617255544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc102980 0x7f23cc073b30 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f23b400d770 tx=0x7f23b400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:04.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.960+0000 7f23c37fe640 1 -- 172.21.15.28:0/2617255544 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b4017070 con 0x7f23cc102980 2024-04-25T01:33:04.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.960+0000 7f23c37fe640 1 -- 172.21.15.28:0/2617255544 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23b4005500 con 0x7f23cc102980 2024-04-25T01:33:04.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.960+0000 7f23c37fe640 1 -- 172.21.15.28:0/2617255544 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b4004e00 con 0x7f23cc102980 2024-04-25T01:33:04.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.960+0000 7f23d0a2a640 1 -- 172.21.15.28:0/2617255544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23cc1a1bb0 con 0x7f23cc102980 2024-04-25T01:33:04.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.960+0000 7f23d0a2a640 1 -- 172.21.15.28:0/2617255544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23cc1a2100 con 0x7f23cc102980 2024-04-25T01:33:04.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.961+0000 7f23d0a2a640 1 -- 172.21.15.28:0/2617255544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23cc108b50 con 0x7f23cc102980 2024-04-25T01:33:04.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.962+0000 7f23c37fe640 1 -- 172.21.15.28:0/2617255544 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 22) ==== 137953+0+0 (secure 0 0 0) 0x7f23b4005000 con 0x7f23cc102980 2024-04-25T01:33:04.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.963+0000 7f23c37fe640 1 --2- 172.21.15.28:0/2617255544 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23b80a2d80 0x7f23b80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:04.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.963+0000 7f23c37fe640 1 -- 172.21.15.28:0/2617255544 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f23b40cf0d0 con 0x7f23cc102980 2024-04-25T01:33:04.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.963+0000 7f23ca59c640 1 --2- 172.21.15.28:0/2617255544 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23b80a2d80 0x7f23b80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:04.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.964+0000 7f23ca59c640 1 --2- 172.21.15.28:0/2617255544 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23b80a2d80 0x7f23b80a5240 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f23bc005ab0 tx=0x7f23bc000bc0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:04.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:04.968+0000 7f23c37fe640 1 -- 172.21.15.28:0/2617255544 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f23b409b360 con 0x7f23cc102980 2024-04-25T01:33:05.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.196+0000 7f23c37fe640 1 -- 172.21.15.28:0/2617255544 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f23b40cc760 con 0x7f23cc102980 2024-04-25T01:33:05.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.201+0000 7f23d0a2a640 1 -- 172.21.15.28:0/2617255544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f23cc02d320 con 0x7f23cc102980 2024-04-25T01:33:05.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.202+0000 7f23c37fe640 1 -- 172.21.15.28:0/2617255544 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f23b409ba60 con 0x7f23cc102980 2024-04-25T01:33:05.179 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T01:33:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.204+0000 7f23d0a2a640 1 -- 172.21.15.28:0/2617255544 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23b80a2d80 msgr2=0x7f23b80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.204+0000 7f23d0a2a640 1 --2- 172.21.15.28:0/2617255544 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23b80a2d80 0x7f23b80a5240 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f23bc005ab0 tx=0x7f23bc000bc0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.204+0000 7f23d0a2a640 1 -- 172.21.15.28:0/2617255544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc102980 msgr2=0x7f23cc073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.204+0000 7f23d0a2a640 1 --2- 172.21.15.28:0/2617255544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc102980 0x7f23cc073b30 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f23b400d770 tx=0x7f23b400dc30 comp rx=0 tx=0).stop 2024-04-25T01:33:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.204+0000 7f23d0a2a640 1 -- 172.21.15.28:0/2617255544 shutdown_connections 2024-04-25T01:33:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.204+0000 7f23d0a2a640 1 --2- 172.21.15.28:0/2617255544 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23b80a2d80 0x7f23b80a5240 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.204+0000 7f23d0a2a640 1 --2- 172.21.15.28:0/2617255544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23cc103340 0x7f23cc074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.204+0000 7f23d0a2a640 1 --2- 172.21.15.28:0/2617255544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc102980 0x7f23cc073b30 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.204+0000 7f23d0a2a640 1 --2- 172.21.15.28:0/2617255544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23cc101790 0x7f23cc0754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.204+0000 7f23d0a2a640 1 -- 172.21.15.28:0/2617255544 >> 172.21.15.28:0/2617255544 conn(0x7f23cc0fcd20 msgr2=0x7f23cc105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:05.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.204+0000 7f23d0a2a640 1 -- 172.21.15.28:0/2617255544 shutdown_connections 2024-04-25T01:33:05.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.204+0000 7f23d0a2a640 1 -- 172.21.15.28:0/2617255544 wait complete. 2024-04-25T01:33:05.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='client.15705 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:33:05.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='client.25298 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:33:05.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:05.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:05.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: Adjusting osd_memory_target on smithi130 to 176.2M 2024-04-25T01:33:05.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: Unable to set osd_memory_target on smithi130 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-25T01:33:05.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:05.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:05.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1698511569' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:33:05.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-04-25T01:33:05.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: Unable to set osd_memory_target on smithi028 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-25T01:33:05.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:05.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:05.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: Adjusting osd_memory_target on smithi032 to 177.2M 2024-04-25T01:33:05.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: Unable to set osd_memory_target on smithi032 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-25T01:33:05.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:33:05.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:33:05.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-25T01:33:05.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-25T01:33:05.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-25T01:33:05.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3302292741' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:33:05.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:04 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1627571869' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:33:05.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:33:05.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='client.15705 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:33:05.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='client.25298 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:33:05.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:05.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:05.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: Adjusting osd_memory_target on smithi130 to 176.2M 2024-04-25T01:33:05.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: Unable to set osd_memory_target on smithi130 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-25T01:33:05.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:05.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:05.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1698511569' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:33:05.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-04-25T01:33:05.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: Unable to set osd_memory_target on smithi028 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-25T01:33:05.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:05.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:05.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: Adjusting osd_memory_target on smithi032 to 177.2M 2024-04-25T01:33:05.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: Unable to set osd_memory_target on smithi032 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-25T01:33:05.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:33:05.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:33:05.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-25T01:33:05.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-25T01:33:05.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-25T01:33:05.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3302292741' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:33:05.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:04 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1627571869' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:33:05.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='client.15705 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:33:05.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='client.25298 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:33:05.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:05.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:05.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: Adjusting osd_memory_target on smithi130 to 176.2M 2024-04-25T01:33:05.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: Unable to set osd_memory_target on smithi130 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-25T01:33:05.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:05.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:05.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1698511569' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:33:05.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-04-25T01:33:05.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: Unable to set osd_memory_target on smithi028 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-25T01:33:05.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:05.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:05.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:33:05.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: Adjusting osd_memory_target on smithi032 to 177.2M 2024-04-25T01:33:05.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: Unable to set osd_memory_target on smithi032 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-25T01:33:05.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:33:05.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:33:05.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-25T01:33:05.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-25T01:33:05.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: Updating smithi130:/etc/ceph/ceph.conf 2024-04-25T01:33:05.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3302292741' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:33:05.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:04 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1627571869' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:33:05.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.332+0000 7f0e3da2a640 1 -- 172.21.15.28:0/3173968443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e38100e60 msgr2=0x7f0e3810dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.332+0000 7f0e3da2a640 1 --2- 172.21.15.28:0/3173968443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e38100e60 0x7f0e3810dbb0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c009900 tx=0x7f0e2c02a410 comp rx=0 tx=0).stop 2024-04-25T01:33:05.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.332+0000 7f0e3da2a640 1 -- 172.21.15.28:0/3173968443 shutdown_connections 2024-04-25T01:33:05.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.332+0000 7f0e3da2a640 1 --2- 172.21.15.28:0/3173968443 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e3810e0f0 0x7f0e381104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.332+0000 7f0e3da2a640 1 --2- 172.21.15.28:0/3173968443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e38100e60 0x7f0e3810dbb0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.332+0000 7f0e3da2a640 1 --2- 172.21.15.28:0/3173968443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e38100540 0x7f0e38100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.332+0000 7f0e3da2a640 1 -- 172.21.15.28:0/3173968443 >> 172.21.15.28:0/3173968443 conn(0x7f0e38077740 msgr2=0x7f0e38077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:05.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.332+0000 7f0e3da2a640 1 -- 172.21.15.28:0/3173968443 shutdown_connections 2024-04-25T01:33:05.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.332+0000 7f0e3da2a640 1 -- 172.21.15.28:0/3173968443 wait complete. 2024-04-25T01:33:05.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.332+0000 7f0e3da2a640 1 Processor -- start 2024-04-25T01:33:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.333+0000 7f0e3da2a640 1 -- start start 2024-04-25T01:33:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.333+0000 7f0e3da2a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e38100540 0x7f0e3819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.333+0000 7f0e3da2a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e38100e60 0x7f0e3819a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.333+0000 7f0e3da2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3810e0f0 0x7f0e3819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.333+0000 7f0e3da2a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e38113d40 con 0x7f0e3810e0f0 2024-04-25T01:33:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.333+0000 7f0e3da2a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0e38113bc0 con 0x7f0e38100e60 2024-04-25T01:33:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.333+0000 7f0e3da2a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0e38113ec0 con 0x7f0e38100540 2024-04-25T01:33:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.333+0000 7f0e377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3810e0f0 0x7f0e3819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.333+0000 7f0e377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3810e0f0 0x7f0e3819ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55184/0 (socket says 172.21.15.28:55184) 2024-04-25T01:33:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.333+0000 7f0e377fe640 1 -- 172.21.15.28:0/1976109733 learned_addr learned my addr 172.21.15.28:0/1976109733 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.333+0000 7f0e36ffd640 1 --2- 172.21.15.28:0/1976109733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e38100540 0x7f0e3819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.333+0000 7f0e367fc640 1 --2- 172.21.15.28:0/1976109733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e38100e60 0x7f0e3819a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.333+0000 7f0e377fe640 1 -- 172.21.15.28:0/1976109733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e38100540 msgr2=0x7f0e3819a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.333+0000 7f0e377fe640 1 --2- 172.21.15.28:0/1976109733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e38100540 0x7f0e3819a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.333+0000 7f0e377fe640 1 -- 172.21.15.28:0/1976109733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e38100e60 msgr2=0x7f0e3819a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.333+0000 7f0e377fe640 1 --2- 172.21.15.28:0/1976109733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e38100e60 0x7f0e3819a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.333+0000 7f0e377fe640 1 -- 172.21.15.28:0/1976109733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e3819f200 con 0x7f0e3810e0f0 2024-04-25T01:33:05.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.333+0000 7f0e36ffd640 1 --2- 172.21.15.28:0/1976109733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e38100540 0x7f0e3819a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:05.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.334+0000 7f0e377fe640 1 --2- 172.21.15.28:0/1976109733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3810e0f0 0x7f0e3819ead0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0e3000eb40 tx=0x7f0e3000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:05.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.334+0000 7f0e3ca28640 1 -- 172.21.15.28:0/1976109733 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e30004b20 con 0x7f0e3810e0f0 2024-04-25T01:33:05.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.334+0000 7f0e3da2a640 1 -- 172.21.15.28:0/1976109733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e3819f4f0 con 0x7f0e3810e0f0 2024-04-25T01:33:05.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.334+0000 7f0e3ca28640 1 -- 172.21.15.28:0/1976109733 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e30004cc0 con 0x7f0e3810e0f0 2024-04-25T01:33:05.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.334+0000 7f0e3ca28640 1 -- 172.21.15.28:0/1976109733 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e30010850 con 0x7f0e3810e0f0 2024-04-25T01:33:05.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.334+0000 7f0e3da2a640 1 -- 172.21.15.28:0/1976109733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e381a6d80 con 0x7f0e3810e0f0 2024-04-25T01:33:05.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.335+0000 7f0e3da2a640 1 -- 172.21.15.28:0/1976109733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e38068560 con 0x7f0e3810e0f0 2024-04-25T01:33:05.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.336+0000 7f0e3ca28640 1 -- 172.21.15.28:0/1976109733 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f0e30005020 con 0x7f0e3810e0f0 2024-04-25T01:33:05.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.337+0000 7f0e3ca28640 1 --2- 172.21.15.28:0/1976109733 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0e200a2d90 0x7f0e200a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.337+0000 7f0e3ca28640 1 -- 172.21.15.28:0/1976109733 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f0e300d0090 con 0x7f0e3810e0f0 2024-04-25T01:33:05.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.337+0000 7f0e36ffd640 1 --2- 172.21.15.28:0/1976109733 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0e200a2d90 0x7f0e200a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.338+0000 7f0e36ffd640 1 --2- 172.21.15.28:0/1976109733 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0e200a2d90 0x7f0e200a5250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009fc0 tx=0x7f0e28005f90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.342+0000 7f0e3ca28640 1 -- 172.21.15.28:0/1976109733 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0e3009c800 con 0x7f0e3810e0f0 2024-04-25T01:33:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.529+0000 7f0e3da2a640 1 -- 172.21.15.28:0/1976109733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0e3802d170 con 0x7f0e3810e0f0 2024-04-25T01:33:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.530+0000 7f0e3ca28640 1 -- 172.21.15.28:0/1976109733 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f0e3009c100 con 0x7f0e3810e0f0 2024-04-25T01:33:05.507 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:33:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.532+0000 7f0e3da2a640 1 -- 172.21.15.28:0/1976109733 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0e200a2d90 msgr2=0x7f0e200a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.532+0000 7f0e3da2a640 1 --2- 172.21.15.28:0/1976109733 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0e200a2d90 0x7f0e200a5250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009fc0 tx=0x7f0e28005f90 comp rx=0 tx=0).stop 2024-04-25T01:33:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.532+0000 7f0e3da2a640 1 -- 172.21.15.28:0/1976109733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3810e0f0 msgr2=0x7f0e3819ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.532+0000 7f0e3da2a640 1 --2- 172.21.15.28:0/1976109733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3810e0f0 0x7f0e3819ead0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0e3000eb40 tx=0x7f0e3000c2d0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.532+0000 7f0e3da2a640 1 -- 172.21.15.28:0/1976109733 shutdown_connections 2024-04-25T01:33:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.532+0000 7f0e3da2a640 1 --2- 172.21.15.28:0/1976109733 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0e200a2d90 0x7f0e200a5250 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.532+0000 7f0e3da2a640 1 --2- 172.21.15.28:0/1976109733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3810e0f0 0x7f0e3819ead0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.532+0000 7f0e3da2a640 1 --2- 172.21.15.28:0/1976109733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e38100e60 0x7f0e3819a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.532+0000 7f0e3da2a640 1 --2- 172.21.15.28:0/1976109733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0e38100540 0x7f0e3819a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.532+0000 7f0e3da2a640 1 -- 172.21.15.28:0/1976109733 >> 172.21.15.28:0/1976109733 conn(0x7f0e38077740 msgr2=0x7f0e3810f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.532+0000 7f0e3da2a640 1 -- 172.21.15.28:0/1976109733 shutdown_connections 2024-04-25T01:33:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.532+0000 7f0e3da2a640 1 -- 172.21.15.28:0/1976109733 wait complete. 2024-04-25T01:33:05.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:33:05.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.651+0000 7f0905942640 1 -- 172.21.15.28:0/1176857676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0900101050 msgr2=0x7f090010d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.651+0000 7f0905942640 1 --2- 172.21.15.28:0/1176857676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0900101050 0x7f090010d510 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f08f8009cd0 tx=0x7f08f802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.651+0000 7f0905942640 1 -- 172.21.15.28:0/1176857676 shutdown_connections 2024-04-25T01:33:05.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.651+0000 7f0905942640 1 --2- 172.21.15.28:0/1176857676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0900101050 0x7f090010d510 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.651+0000 7f0905942640 1 --2- 172.21.15.28:0/1176857676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0900100690 0x7f0900100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.651+0000 7f0905942640 1 --2- 172.21.15.28:0/1176857676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0900106650 0x7f0900106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.651+0000 7f0905942640 1 -- 172.21.15.28:0/1176857676 >> 172.21.15.28:0/1176857676 conn(0x7f09000777a0 msgr2=0x7f0900077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:05.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.651+0000 7f0905942640 1 -- 172.21.15.28:0/1176857676 shutdown_connections 2024-04-25T01:33:05.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.651+0000 7f0905942640 1 -- 172.21.15.28:0/1176857676 wait complete. 2024-04-25T01:33:05.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.651+0000 7f0905942640 1 Processor -- start 2024-04-25T01:33:05.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.651+0000 7f0905942640 1 -- start start 2024-04-25T01:33:05.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f0905942640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0900100690 0x7f0900198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f0905942640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0900101050 0x7f0900199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f0905942640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0900106650 0x7f09001a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f0905942640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0900110ab0 con 0x7f0900101050 2024-04-25T01:33:05.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f0905942640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0900110930 con 0x7f0900106650 2024-04-25T01:33:05.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f0905942640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0900110c30 con 0x7f0900100690 2024-04-25T01:33:05.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f08fe7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0900101050 0x7f0900199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f08fe7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0900101050 0x7f0900199390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55204/0 (socket says 172.21.15.28:55204) 2024-04-25T01:33:05.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f08fe7fc640 1 -- 172.21.15.28:0/3593217540 learned_addr learned my addr 172.21.15.28:0/3593217540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f08ff7fe640 1 --2- 172.21.15.28:0/3593217540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0900106650 0x7f09001a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f08feffd640 1 --2- 172.21.15.28:0/3593217540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0900100690 0x7f0900198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f08fe7fc640 1 -- 172.21.15.28:0/3593217540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0900100690 msgr2=0x7f0900198e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f08fe7fc640 1 --2- 172.21.15.28:0/3593217540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0900100690 0x7f0900198e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f08fe7fc640 1 -- 172.21.15.28:0/3593217540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0900106650 msgr2=0x7f09001a3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f08fe7fc640 1 --2- 172.21.15.28:0/3593217540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0900106650 0x7f09001a3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f08fe7fc640 1 -- 172.21.15.28:0/3593217540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09001a4410 con 0x7f0900101050 2024-04-25T01:33:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f08fe7fc640 1 --2- 172.21.15.28:0/3593217540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0900101050 0x7f0900199390 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f08f400d950 tx=0x7f08f400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f0904940640 1 -- 172.21.15.28:0/3593217540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f4017070 con 0x7f0900101050 2024-04-25T01:33:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f0905942640 1 -- 172.21.15.28:0/3593217540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09001a45e0 con 0x7f0900101050 2024-04-25T01:33:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.652+0000 7f0904940640 1 -- 172.21.15.28:0/3593217540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08f4009c30 con 0x7f0900101050 2024-04-25T01:33:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.653+0000 7f0905942640 1 -- 172.21.15.28:0/3593217540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09001a4b80 con 0x7f0900101050 2024-04-25T01:33:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.653+0000 7f0904940640 1 -- 172.21.15.28:0/3593217540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f4020060 con 0x7f0900101050 2024-04-25T01:33:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.653+0000 7f0905942640 1 -- 172.21.15.28:0/3593217540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0900101d80 con 0x7f0900101050 2024-04-25T01:33:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.655+0000 7f0904940640 1 -- 172.21.15.28:0/3593217540 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f08f40109d0 con 0x7f0900101050 2024-04-25T01:33:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.655+0000 7f0904940640 1 --2- 172.21.15.28:0/3593217540 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f08e80a2d90 0x7f08e80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.655+0000 7f08feffd640 1 --2- 172.21.15.28:0/3593217540 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f08e80a2d90 0x7f08e80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.656+0000 7f0904940640 1 -- 172.21.15.28:0/3593217540 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f08f40cf910 con 0x7f0900101050 2024-04-25T01:33:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.656+0000 7f08feffd640 1 --2- 172.21.15.28:0/3593217540 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f08e80a2d90 0x7f08e80a5250 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f08f0005fd0 tx=0x7f08f000b510 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:05.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.660+0000 7f0904940640 1 -- 172.21.15.28:0/3593217540 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f08f4009de0 con 0x7f0900101050 2024-04-25T01:33:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.882+0000 7f0905942640 1 -- 172.21.15.28:0/3593217540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0900108bb0 con 0x7f0900101050 2024-04-25T01:33:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.883+0000 7f0904940640 1 -- 172.21.15.28:0/3593217540 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7f08f409c080 con 0x7f0900101050 2024-04-25T01:33:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:33:05.860 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:33:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:33:05.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.885+0000 7f0905942640 1 -- 172.21.15.28:0/3593217540 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f08e80a2d90 msgr2=0x7f08e80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.885+0000 7f0905942640 1 --2- 172.21.15.28:0/3593217540 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f08e80a2d90 0x7f08e80a5250 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f08f0005fd0 tx=0x7f08f000b510 comp rx=0 tx=0).stop 2024-04-25T01:33:05.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.885+0000 7f0905942640 1 -- 172.21.15.28:0/3593217540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0900101050 msgr2=0x7f0900199390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.885+0000 7f0905942640 1 --2- 172.21.15.28:0/3593217540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0900101050 0x7f0900199390 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f08f400d950 tx=0x7f08f400de10 comp rx=0 tx=0).stop 2024-04-25T01:33:05.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.885+0000 7f0905942640 1 -- 172.21.15.28:0/3593217540 shutdown_connections 2024-04-25T01:33:05.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.885+0000 7f0905942640 1 --2- 172.21.15.28:0/3593217540 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f08e80a2d90 0x7f08e80a5250 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.885+0000 7f0905942640 1 --2- 172.21.15.28:0/3593217540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0900106650 0x7f09001a3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.885+0000 7f0905942640 1 --2- 172.21.15.28:0/3593217540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0900101050 0x7f0900199390 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.885+0000 7f0905942640 1 --2- 172.21.15.28:0/3593217540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0900100690 0x7f0900198e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.885+0000 7f0905942640 1 -- 172.21.15.28:0/3593217540 >> 172.21.15.28:0/3593217540 conn(0x7f09000777a0 msgr2=0x7f0900103170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:05.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.885+0000 7f0905942640 1 -- 172.21.15.28:0/3593217540 shutdown_connections 2024-04-25T01:33:05.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.885+0000 7f0905942640 1 -- 172.21.15.28:0/3593217540 wait complete. 2024-04-25T01:33:05.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:33:05.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.999+0000 7f2792fd5640 1 -- 172.21.15.28:0/4177514903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f278c106650 msgr2=0x7f278c106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.999+0000 7f2792fd5640 1 --2- 172.21.15.28:0/4177514903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f278c106650 0x7f278c106a30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f277c009930 tx=0x7f277c02a420 comp rx=0 tx=0).stop 2024-04-25T01:33:05.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.999+0000 7f2792fd5640 1 -- 172.21.15.28:0/4177514903 shutdown_connections 2024-04-25T01:33:05.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.999+0000 7f2792fd5640 1 --2- 172.21.15.28:0/4177514903 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f278c101050 0x7f278c10d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.999+0000 7f2792fd5640 1 --2- 172.21.15.28:0/4177514903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f278c100690 0x7f278c100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.999+0000 7f2792fd5640 1 --2- 172.21.15.28:0/4177514903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f278c106650 0x7f278c106a30 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.999+0000 7f2792fd5640 1 -- 172.21.15.28:0/4177514903 >> 172.21.15.28:0/4177514903 conn(0x7f278c0777a0 msgr2=0x7f278c077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.999+0000 7f2792fd5640 1 -- 172.21.15.28:0/4177514903 shutdown_connections 2024-04-25T01:33:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.999+0000 7f2792fd5640 1 -- 172.21.15.28:0/4177514903 wait complete. 2024-04-25T01:33:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.999+0000 7f2792fd5640 1 Processor -- start 2024-04-25T01:33:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.999+0000 7f2792fd5640 1 -- start start 2024-04-25T01:33:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.999+0000 7f2792fd5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f278c100690 0x7f278c198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.999+0000 7f2792fd5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f278c101050 0x7f278c198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.999+0000 7f2792fd5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f278c106650 0x7f278c19c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.999+0000 7f2792fd5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f278c111b60 con 0x7f278c106650 2024-04-25T01:33:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.999+0000 7f2792fd5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f278c1119e0 con 0x7f278c100690 2024-04-25T01:33:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:05.999+0000 7f2792fd5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f278c111ce0 con 0x7f278c101050 2024-04-25T01:33:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.000+0000 7f2791572640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f278c106650 0x7f278c19c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.000+0000 7f2791572640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f278c106650 0x7f278c19c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55212/0 (socket says 172.21.15.28:55212) 2024-04-25T01:33:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.000+0000 7f2791572640 1 -- 172.21.15.28:0/4270071796 learned_addr learned my addr 172.21.15.28:0/4270071796 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.000+0000 7f278bfff640 1 --2- 172.21.15.28:0/4270071796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f278c101050 0x7f278c198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.000+0000 7f2790d71640 1 --2- 172.21.15.28:0/4270071796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f278c100690 0x7f278c198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.000+0000 7f2791572640 1 -- 172.21.15.28:0/4270071796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f278c101050 msgr2=0x7f278c198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.000+0000 7f2791572640 1 --2- 172.21.15.28:0/4270071796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f278c101050 0x7f278c198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.000+0000 7f2791572640 1 -- 172.21.15.28:0/4270071796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f278c100690 msgr2=0x7f278c198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.000+0000 7f2791572640 1 --2- 172.21.15.28:0/4270071796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f278c100690 0x7f278c198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.000+0000 7f2791572640 1 -- 172.21.15.28:0/4270071796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278c19d120 con 0x7f278c106650 2024-04-25T01:33:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.000+0000 7f2791572640 1 --2- 172.21.15.28:0/4270071796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f278c106650 0x7f278c19c9f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f278400cab0 tx=0x7f278400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.000+0000 7f2789ffb640 1 -- 172.21.15.28:0/4270071796 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278400adc0 con 0x7f278c106650 2024-04-25T01:33:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.000+0000 7f2789ffb640 1 -- 172.21.15.28:0/4270071796 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f278400f930 con 0x7f278c106650 2024-04-25T01:33:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.000+0000 7f2789ffb640 1 -- 172.21.15.28:0/4270071796 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2784018ae0 con 0x7f278c106650 2024-04-25T01:33:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.000+0000 7f2792fd5640 1 -- 172.21.15.28:0/4270071796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f278c19d410 con 0x7f278c106650 2024-04-25T01:33:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.000+0000 7f2792fd5640 1 -- 172.21.15.28:0/4270071796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f278c1a4ca0 con 0x7f278c106650 2024-04-25T01:33:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.002+0000 7f2792fd5640 1 -- 172.21.15.28:0/4270071796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f278c0689b0 con 0x7f278c106650 2024-04-25T01:33:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.003+0000 7f2789ffb640 1 -- 172.21.15.28:0/4270071796 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f278400fae0 con 0x7f278c106650 2024-04-25T01:33:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.003+0000 7f2789ffb640 1 --2- 172.21.15.28:0/4270071796 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f27740a2e10 0x7f27740a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.004+0000 7f2790d71640 1 --2- 172.21.15.28:0/4270071796 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f27740a2e10 0x7f27740a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.004+0000 7f2789ffb640 1 -- 172.21.15.28:0/4270071796 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f27840cfa70 con 0x7f278c106650 2024-04-25T01:33:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.004+0000 7f2790d71640 1 --2- 172.21.15.28:0/4270071796 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f27740a2e10 0x7f27740a52d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f277c000c00 tx=0x7f277c005a20 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.008+0000 7f2789ffb640 1 -- 172.21.15.28:0/4270071796 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2784014030 con 0x7f278c106650 2024-04-25T01:33:06.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.198+0000 7f2792fd5640 1 -- 172.21.15.28:0/4270071796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f278c108c10 con 0x7f278c106650 2024-04-25T01:33:06.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.199+0000 7f2789ffb640 1 -- 172.21.15.28:0/4270071796 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24941 (secure 0 0 0) 0x7f278409c1e0 con 0x7f278c106650 2024-04-25T01:33:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:33:06.177 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:33:00.617205+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/1149553567":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/500268183":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/3499593842":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/3630315328":"2024-04-26T01:33:00.617174+0000","172.21.15.28:6800/1149553567":"2024-04-26T01:33:00.617174+0000","172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3812371257":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:33:06.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.202+0000 7f2792fd5640 1 -- 172.21.15.28:0/4270071796 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f27740a2e10 msgr2=0x7f27740a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.202+0000 7f2792fd5640 1 --2- 172.21.15.28:0/4270071796 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f27740a2e10 0x7f27740a52d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f277c000c00 tx=0x7f277c005a20 comp rx=0 tx=0).stop 2024-04-25T01:33:06.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.202+0000 7f2792fd5640 1 -- 172.21.15.28:0/4270071796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f278c106650 msgr2=0x7f278c19c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.202+0000 7f2792fd5640 1 --2- 172.21.15.28:0/4270071796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f278c106650 0x7f278c19c9f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f278400cab0 tx=0x7f278400cf70 comp rx=0 tx=0).stop 2024-04-25T01:33:06.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.202+0000 7f2792fd5640 1 -- 172.21.15.28:0/4270071796 shutdown_connections 2024-04-25T01:33:06.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.202+0000 7f2792fd5640 1 --2- 172.21.15.28:0/4270071796 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f27740a2e10 0x7f27740a52d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.202+0000 7f2792fd5640 1 --2- 172.21.15.28:0/4270071796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f278c106650 0x7f278c19c9f0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.202+0000 7f2792fd5640 1 --2- 172.21.15.28:0/4270071796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f278c101050 0x7f278c198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.202+0000 7f2792fd5640 1 --2- 172.21.15.28:0/4270071796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f278c100690 0x7f278c198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.202+0000 7f2792fd5640 1 -- 172.21.15.28:0/4270071796 >> 172.21.15.28:0/4270071796 conn(0x7f278c0777a0 msgr2=0x7f278c103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:06.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.202+0000 7f2792fd5640 1 -- 172.21.15.28:0/4270071796 shutdown_connections 2024-04-25T01:33:06.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.202+0000 7f2792fd5640 1 -- 172.21.15.28:0/4270071796 wait complete. 2024-04-25T01:33:06.191 DEBUG:tasks.fwd_scrub:Yielding 2024-04-25T01:33:06.191 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T01:33:06.200 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T01:33:06.200 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T01:33:06.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-25T01:33:06.201 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-25T01:33:06.201 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:33:06.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:33:06.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:33:06.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.329+0000 7f6ba1116640 1 -- 172.21.15.28:0/2739221488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b9c1032f0 msgr2=0x7f6b9c10b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.329+0000 7f6ba1116640 1 --2- 172.21.15.28:0/2739221488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b9c1032f0 0x7f6b9c10b7d0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6b90009900 tx=0x7f6b9002a510 comp rx=0 tx=0).stop 2024-04-25T01:33:06.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.329+0000 7f6ba1116640 1 -- 172.21.15.28:0/2739221488 shutdown_connections 2024-04-25T01:33:06.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.329+0000 7f6ba1116640 1 --2- 172.21.15.28:0/2739221488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b9c10bd10 0x7f6b9c10e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.329+0000 7f6ba1116640 1 --2- 172.21.15.28:0/2739221488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b9c1032f0 0x7f6b9c10b7d0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.329+0000 7f6ba1116640 1 --2- 172.21.15.28:0/2739221488 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b9c1029d0 0x7f6b9c102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.329+0000 7f6ba1116640 1 -- 172.21.15.28:0/2739221488 >> 172.21.15.28:0/2739221488 conn(0x7f6b9c0777c0 msgr2=0x7f6b9c077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:06.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.329+0000 7f6ba1116640 1 -- 172.21.15.28:0/2739221488 shutdown_connections 2024-04-25T01:33:06.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.329+0000 7f6ba1116640 1 -- 172.21.15.28:0/2739221488 wait complete. 2024-04-25T01:33:06.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.330+0000 7f6ba1116640 1 Processor -- start 2024-04-25T01:33:06.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.330+0000 7f6ba1116640 1 -- start start 2024-04-25T01:33:06.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.330+0000 7f285dd09640 1 -- 172.21.15.28:0/771625318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2858100610 msgr2=0x7f2858100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.330+0000 7f285dd09640 1 --2- 172.21.15.28:0/771625318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2858100610 0x7f2858100a90 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f284c009930 tx=0x7f284c02a420 comp rx=0 tx=0).stop 2024-04-25T01:33:06.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.330+0000 7f285dd09640 1 -- 172.21.15.28:0/771625318 shutdown_connections 2024-04-25T01:33:06.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.330+0000 7f285dd09640 1 --2- 172.21.15.28:0/771625318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2858100fd0 0x7f285810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.330+0000 7f285dd09640 1 --2- 172.21.15.28:0/771625318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2858100610 0x7f2858100a90 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.330+0000 7f285dd09640 1 --2- 172.21.15.28:0/771625318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28581066b0 0x7f2858106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f285dd09640 1 -- 172.21.15.28:0/771625318 >> 172.21.15.28:0/771625318 conn(0x7f2858077740 msgr2=0x7f2858077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:06.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f6ba1116640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b9c1029d0 0x7f6b9c197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f285dd09640 1 -- 172.21.15.28:0/771625318 shutdown_connections 2024-04-25T01:33:06.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f285dd09640 1 -- 172.21.15.28:0/771625318 wait complete. 2024-04-25T01:33:06.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f6ba1116640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b9c1032f0 0x7f6b9c1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f6ba1116640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b9c10bd10 0x7f6b9c19c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f6ba1116640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9c111b10 con 0x7f6b9c1029d0 2024-04-25T01:33:06.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f6ba1116640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9c111990 con 0x7f6b9c1032f0 2024-04-25T01:33:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f6ba1116640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9c111c90 con 0x7f6b9c10bd10 2024-04-25T01:33:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f6b9ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b9c1029d0 0x7f6b9c197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f6b9ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b9c1029d0 0x7f6b9c197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55240/0 (socket says 172.21.15.28:55240) 2024-04-25T01:33:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f6b9ad9d640 1 -- 172.21.15.28:0/1582657603 learned_addr learned my addr 172.21.15.28:0/1582657603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f6b9b59e640 1 --2- 172.21.15.28:0/1582657603 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b9c10bd10 0x7f6b9c19c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f6b9a59c640 1 --2- 172.21.15.28:0/1582657603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b9c1032f0 0x7f6b9c1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f6b9ad9d640 1 -- 172.21.15.28:0/1582657603 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b9c10bd10 msgr2=0x7f6b9c19c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f6b9ad9d640 1 --2- 172.21.15.28:0/1582657603 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b9c10bd10 0x7f6b9c19c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f6b9ad9d640 1 -- 172.21.15.28:0/1582657603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b9c1032f0 msgr2=0x7f6b9c1983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f6b9ad9d640 1 --2- 172.21.15.28:0/1582657603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b9c1032f0 0x7f6b9c1983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f6b9ad9d640 1 -- 172.21.15.28:0/1582657603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9c19cea0 con 0x7f6b9c1029d0 2024-04-25T01:33:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f285dd09640 1 Processor -- start 2024-04-25T01:33:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f6b9ad9d640 1 --2- 172.21.15.28:0/1582657603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b9c1029d0 0x7f6b9c197eb0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400c920 tx=0x7f6b9400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f285dd09640 1 -- start start 2024-04-25T01:33:06.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f6b83fff640 1 -- 172.21.15.28:0/1582657603 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b94007c70 con 0x7f6b9c1029d0 2024-04-25T01:33:06.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.331+0000 7f6b83fff640 1 -- 172.21.15.28:0/1582657603 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b94007e10 con 0x7f6b9c1029d0 2024-04-25T01:33:06.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.332+0000 7f6ba1116640 1 -- 172.21.15.28:0/1582657603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b9c19d190 con 0x7f6b9c1029d0 2024-04-25T01:33:06.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.332+0000 7f6b83fff640 1 -- 172.21.15.28:0/1582657603 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b94018890 con 0x7f6b9c1029d0 2024-04-25T01:33:06.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.332+0000 7f6ba1116640 1 -- 172.21.15.28:0/1582657603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b9c1a4a20 con 0x7f6b9c1029d0 2024-04-25T01:33:06.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.332+0000 7f285dd09640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2858100610 0x7f2858198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.332+0000 7f285dd09640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2858100fd0 0x7f2858198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.332+0000 7f285dd09640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28581066b0 0x7f285819c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.332+0000 7f285dd09640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2858110a40 con 0x7f2858100fd0 2024-04-25T01:33:06.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.332+0000 7f285dd09640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f28581108c0 con 0x7f28581066b0 2024-04-25T01:33:06.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.332+0000 7f285dd09640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2858110bc0 con 0x7f2858100610 2024-04-25T01:33:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.332+0000 7f2856ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2858100fd0 0x7f2858198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.332+0000 7f2856ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2858100fd0 0x7f2858198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55252/0 (socket says 172.21.15.28:55252) 2024-04-25T01:33:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.332+0000 7f2856ffd640 1 -- 172.21.15.28:0/1047119553 learned_addr learned my addr 172.21.15.28:0/1047119553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.332+0000 7f28577fe640 1 --2- 172.21.15.28:0/1047119553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2858100610 0x7f2858198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.332+0000 7f2857fff640 1 --2- 172.21.15.28:0/1047119553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28581066b0 0x7f285819c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.332+0000 7f2856ffd640 1 -- 172.21.15.28:0/1047119553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2858100610 msgr2=0x7f2858198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.332+0000 7f2856ffd640 1 --2- 172.21.15.28:0/1047119553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2858100610 0x7f2858198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.332+0000 7f2856ffd640 1 -- 172.21.15.28:0/1047119553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28581066b0 msgr2=0x7f285819c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.332+0000 7f2856ffd640 1 --2- 172.21.15.28:0/1047119553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28581066b0 0x7f285819c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.332+0000 7f2856ffd640 1 -- 172.21.15.28:0/1047119553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285819cfe0 con 0x7f2858100fd0 2024-04-25T01:33:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.333+0000 7f2856ffd640 1 --2- 172.21.15.28:0/1047119553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2858100fd0 0x7f2858198580 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f284c02a940 tx=0x7f284c003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.333+0000 7f2854ff9640 1 -- 172.21.15.28:0/1047119553 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c03f070 con 0x7f2858100fd0 2024-04-25T01:33:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.333+0000 7f285dd09640 1 -- 172.21.15.28:0/1047119553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f285819d270 con 0x7f2858100fd0 2024-04-25T01:33:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.333+0000 7f285dd09640 1 -- 172.21.15.28:0/1047119553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28581a4b60 con 0x7f2858100fd0 2024-04-25T01:33:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.333+0000 7f2854ff9640 1 -- 172.21.15.28:0/1047119553 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f284c031c90 con 0x7f2858100fd0 2024-04-25T01:33:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.333+0000 7f2854ff9640 1 -- 172.21.15.28:0/1047119553 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c035030 con 0x7f2858100fd0 2024-04-25T01:33:06.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.333+0000 7f6ba1116640 1 -- 172.21.15.28:0/1582657603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b9c04f500 con 0x7f6b9c1029d0 2024-04-25T01:33:06.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.334+0000 7f6b83fff640 1 -- 172.21.15.28:0/1582657603 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f6b94018a70 con 0x7f6b9c1029d0 2024-04-25T01:33:06.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.335+0000 7f6b83fff640 1 --2- 172.21.15.28:0/1582657603 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6b840a2e40 0x7f6b840a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.335+0000 7f6b83fff640 1 -- 172.21.15.28:0/1582657603 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f6b940cfa80 con 0x7f6b9c1029d0 2024-04-25T01:33:06.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.335+0000 7f6b9a59c640 1 --2- 172.21.15.28:0/1582657603 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6b840a2e40 0x7f6b840a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.335+0000 7f2854ff9640 1 -- 172.21.15.28:0/1047119553 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f284c042070 con 0x7f2858100fd0 2024-04-25T01:33:06.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.335+0000 7f6b9a59c640 1 --2- 172.21.15.28:0/1582657603 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6b840a2e40 0x7f6b840a5300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c10d910 tx=0x7f6b90031490 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:06.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2860a0f740 1 Processor -- start 2024-04-25T01:33:06.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2860a0f740 1 -- start start 2024-04-25T01:33:06.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2860a0f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d40980 0x560722d3ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2860a0f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d3f290 0x560722d3f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2854ff9640 1 --2- 172.21.15.28:0/1047119553 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f28400a2e10 0x7f28400a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2860a0f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d4e420 0x560722d50810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2860a0f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560722ca32c0 con 0x560722d3f290 2024-04-25T01:33:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2860a0f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560722c4b910 con 0x560722d40980 2024-04-25T01:33:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2860a0f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x560722c9f280 con 0x560722d4e420 2024-04-25T01:33:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2856ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d3f290 0x560722d3f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2856ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d3f290 0x560722d3f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55260/0 (socket says 172.21.15.28:55260) 2024-04-25T01:33:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2856ffd640 1 -- 172.21.15.28:0/4117801700 learned_addr learned my addr 172.21.15.28:0/4117801700 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f28577fe640 1 --2- 172.21.15.28:0/1047119553 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f28400a2e10 0x7f28400a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2854ff9640 1 -- 172.21.15.28:0/1047119553 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f284c0efa00 con 0x7f2858100fd0 2024-04-25T01:33:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2857fff640 1 --2- 172.21.15.28:0/4117801700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d4e420 0x560722d50810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2857fff640 1 --2- 172.21.15.28:0/4117801700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d40980 0x560722d3ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2856ffd640 1 -- 172.21.15.28:0/4117801700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d4e420 msgr2=0x560722d50810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2856ffd640 1 --2- 172.21.15.28:0/4117801700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d4e420 0x560722d50810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2856ffd640 1 -- 172.21.15.28:0/4117801700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d40980 msgr2=0x560722d3ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2856ffd640 1 --2- 172.21.15.28:0/4117801700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d40980 0x560722d3ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2856ffd640 1 -- 172.21.15.28:0/4117801700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560722c5cfb0 con 0x560722d3f290 2024-04-25T01:33:06.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.336+0000 7f2857fff640 1 --2- 172.21.15.28:0/4117801700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d40980 0x560722d3ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:06.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.337+0000 7f2856ffd640 1 --2- 172.21.15.28:0/4117801700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d3f290 0x560722d3f710 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f284c0ed410 tx=0x7f284c02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20d1b041c9cf1c65 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:06.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.337+0000 7f28577fe640 1 --2- 172.21.15.28:0/1047119553 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f28400a2e10 0x7f28400a52d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2858101750 tx=0x7f2848005f90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:06.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.339+0000 7f283dffb640 1 -- 172.21.15.28:0/4117801700 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c049040 con 0x560722d3f290 2024-04-25T01:33:06.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.340+0000 7f283dffb640 1 -- 172.21.15.28:0/4117801700 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f284c08df80 con 0x560722d3f290 2024-04-25T01:33:06.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.340+0000 7f283dffb640 1 -- 172.21.15.28:0/4117801700 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c0bfe20 con 0x560722d3f290 2024-04-25T01:33:06.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.340+0000 7f6b83fff640 1 -- 172.21.15.28:0/1582657603 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6b94014030 con 0x7f6b9c1029d0 2024-04-25T01:33:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.340+0000 7f2860a0f740 1 -- 172.21.15.28:0/4117801700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d3f290 msgr2=0x560722d3f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.340+0000 7f2860a0f740 1 --2- 172.21.15.28:0/4117801700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d3f290 0x560722d3f710 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f284c0ed410 tx=0x7f284c02a420 comp rx=0 tx=0).stop 2024-04-25T01:33:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.340+0000 7f2860a0f740 1 -- 172.21.15.28:0/4117801700 shutdown_connections 2024-04-25T01:33:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.340+0000 7f2860a0f740 1 --2- 172.21.15.28:0/4117801700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d4e420 0x560722d50810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.340+0000 7f2860a0f740 1 --2- 172.21.15.28:0/4117801700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d3f290 0x560722d3f710 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.340+0000 7f2860a0f740 1 --2- 172.21.15.28:0/4117801700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d40980 0x560722d3ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.340+0000 7f2860a0f740 1 -- 172.21.15.28:0/4117801700 >> 172.21.15.28:0/4117801700 conn(0x560722d07950 msgr2=0x560722d3b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.340+0000 7f2860a0f740 1 -- 172.21.15.28:0/4117801700 shutdown_connections 2024-04-25T01:33:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.340+0000 7f2860a0f740 1 -- 172.21.15.28:0/4117801700 wait complete. 2024-04-25T01:33:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.341+0000 7f2860a0f740 1 -- start start 2024-04-25T01:33:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.341+0000 7f2860a0f740 1 Processor -- start 2024-04-25T01:33:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.341+0000 7f2860a0f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d3f290 0x560722d4b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.341+0000 7f2860a0f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d40980 0x560722d4b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.341+0000 7f2860a0f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d4e420 0x560722d4be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.341+0000 7f2860a0f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560722d51070 con 0x560722d40980 2024-04-25T01:33:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.341+0000 7f2860a0f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560722c5ee90 con 0x560722d4e420 2024-04-25T01:33:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.341+0000 7f2860a0f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x560722d511f0 con 0x560722d3f290 2024-04-25T01:33:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f2860a0f740 10 client.0 _ll_register_callbacks cb 0x560722c9a970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f2856ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d40980 0x560722d4b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f2856ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d40980 0x560722d4b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55274/0 (socket says 172.21.15.28:55274) 2024-04-25T01:33:06.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f2856ffd640 1 -- 172.21.15.28:0/4143947063 learned_addr learned my addr 172.21.15.28:0/4143947063 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f28577fe640 1 --2- 172.21.15.28:0/4143947063 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d3f290 0x560722d4b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f2857fff640 1 --2- 172.21.15.28:0/4143947063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d4e420 0x560722d4be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f2856ffd640 1 -- 172.21.15.28:0/4143947063 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d3f290 msgr2=0x560722d4b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f2856ffd640 1 --2- 172.21.15.28:0/4143947063 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d3f290 0x560722d4b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f2856ffd640 1 -- 172.21.15.28:0/4143947063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d4e420 msgr2=0x560722d4be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f2856ffd640 1 --2- 172.21.15.28:0/4143947063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d4e420 0x560722d4be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f2856ffd640 1 -- 172.21.15.28:0/4143947063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560722a7f7e0 con 0x560722d40980 2024-04-25T01:33:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f2857fff640 1 --2- 172.21.15.28:0/4143947063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d4e420 0x560722d4be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f2856ffd640 1 --2- 172.21.15.28:0/4143947063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d40980 0x560722d4b910 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f284c0efdb0 tx=0x7f284c0ba030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a8f56dfa6ee2a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f2836ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f2836ffd640 1 -- 172.21.15.28:0/4143947063 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c049050 con 0x560722d40980 2024-04-25T01:33:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f2836ffd640 1 -- 172.21.15.28:0/4143947063 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f284c0ed000 con 0x560722d40980 2024-04-25T01:33:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f2860a0f740 1 -- 172.21.15.28:0/4143947063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f284c08df80 con 0x560722d40980 2024-04-25T01:33:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f2836ffd640 1 -- 172.21.15.28:0/4143947063 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f284c08dd90 con 0x560722d40980 2024-04-25T01:33:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f2860a0f740 10 client.15771 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.342+0000 7f2860a0f740 1 -- 172.21.15.28:0/4143947063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560722adff70 con 0x560722d40980 2024-04-25T01:33:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.343+0000 7f2836ffd640 1 -- 172.21.15.28:0/4143947063 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f284c08df80 con 0x560722d40980 2024-04-25T01:33:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.343+0000 7f2836ffd640 1 -- 172.21.15.28:0/4143947063 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f284c04c060 con 0x560722d40980 2024-04-25T01:33:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.343+0000 7f2860a0f740 10 client.15771 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.343+0000 7f2860a0f740 10 client.15771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.343+0000 7f2860a0f740 20 client.15771 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.343+0000 7f2860a0f740 1 --2- 172.21.15.28:0/4143947063 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x560722d5e180 0x560722d60570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.343+0000 7f2860a0f740 4 client.15771 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.343+0000 7f2860a0f740 1 -- 172.21.15.28:0/4143947063 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560722c775a0 con 0x560722d5e180 2024-04-25T01:33:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.343+0000 7f28577fe640 1 --2- 172.21.15.28:0/4143947063 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x560722d5e180 0x560722d60570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.344+0000 7f28577fe640 1 --2- 172.21.15.28:0/4143947063 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x560722d5e180 0x560722d60570 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c27a435647cfd0d5 server_cookie=58962f0ff5802c17 in_seq=0 out_seq=0 2024-04-25T01:33:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.344+0000 7f2836ffd640 10 client.15771 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.344+0000 7f2836ffd640 1 -- 172.21.15.28:0/4143947063 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560722c775a0 con 0x560722d5e180 2024-04-25T01:33:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.344+0000 7f2836ffd640 10 client.15771 handle_command_reply: tid=0 2024-04-25T01:33:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.344+0000 7f2860a0f740 1 client.15771 shutdown 2024-04-25T01:33:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.344+0000 7f2860a0f740 1 -- 172.21.15.28:0/4143947063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d40980 msgr2=0x560722d4b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.344+0000 7f2860a0f740 1 --2- 172.21.15.28:0/4143947063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d40980 0x560722d4b910 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f284c0efdb0 tx=0x7f284c0ba030 comp rx=0 tx=0).stop 2024-04-25T01:33:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.345+0000 7f2860a0f740 1 -- 172.21.15.28:0/4143947063 shutdown_connections 2024-04-25T01:33:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.345+0000 7f2860a0f740 1 --2- 172.21.15.28:0/4143947063 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x560722d5e180 0x560722d60570 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.345+0000 7f2860a0f740 1 --2- 172.21.15.28:0/4143947063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d4e420 0x560722d4be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.345+0000 7f2836ffd640 0 client.15771 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.345+0000 7f2860a0f740 1 --2- 172.21.15.28:0/4143947063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d40980 0x560722d4b910 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.345+0000 7f2860a0f740 1 --2- 172.21.15.28:0/4143947063 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d3f290 0x560722d4b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.345+0000 7f2860a0f740 1 -- 172.21.15.28:0/4143947063 >> 172.21.15.28:0/4143947063 conn(0x560722d07950 msgr2=0x560722d415f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.345+0000 7f2860a0f740 1 -- 172.21.15.28:0/4143947063 shutdown_connections 2024-04-25T01:33:06.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.345+0000 7f2860a0f740 1 -- 172.21.15.28:0/4143947063 wait complete. 2024-04-25T01:33:06.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.345+0000 7f2860a0f740 20 client.15771 trim_cache size 0 max 16384 2024-04-25T01:33:06.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.383+0000 7f2860a0f740 1 Processor -- start 2024-04-25T01:33:06.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.383+0000 7f2860a0f740 1 -- start start 2024-04-25T01:33:06.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.384+0000 7f2860a0f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d3f290 0x560722d3a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.384+0000 7f2860a0f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d40980 0x560722d4bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.384+0000 7f2860a0f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d4e420 0x560722d41470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.384+0000 7f2860a0f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f284400fd10 con 0x560722d40980 2024-04-25T01:33:06.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.384+0000 7f2860a0f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2844011850 con 0x560722d3f290 2024-04-25T01:33:06.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.384+0000 7f2860a0f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f28440103e0 con 0x560722d4e420 2024-04-25T01:33:06.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.384+0000 7f2856ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d40980 0x560722d4bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.384+0000 7f2856ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d40980 0x560722d4bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55282/0 (socket says 172.21.15.28:55282) 2024-04-25T01:33:06.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.384+0000 7f2856ffd640 1 -- 172.21.15.28:0/3023741597 learned_addr learned my addr 172.21.15.28:0/3023741597 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:06.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.384+0000 7f28577fe640 1 --2- 172.21.15.28:0/3023741597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d3f290 0x560722d3a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.384+0000 7f2856ffd640 1 -- 172.21.15.28:0/3023741597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d4e420 msgr2=0x560722d41470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:06.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.384+0000 7f2857fff640 1 --2- 172.21.15.28:0/3023741597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d4e420 0x560722d41470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.384+0000 7f2856ffd640 1 --2- 172.21.15.28:0/3023741597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d4e420 0x560722d41470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.384+0000 7f2856ffd640 1 -- 172.21.15.28:0/3023741597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d3f290 msgr2=0x560722d3a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:06.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.384+0000 7f2856ffd640 1 --2- 172.21.15.28:0/3023741597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d3f290 0x560722d3a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.384+0000 7f2856ffd640 1 -- 172.21.15.28:0/3023741597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560722977170 con 0x560722d40980 2024-04-25T01:33:06.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.384+0000 7f28577fe640 1 --2- 172.21.15.28:0/3023741597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d3f290 0x560722d3a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:06.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.384+0000 7f2856ffd640 1 --2- 172.21.15.28:0/3023741597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d40980 0x560722d4bc70 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f284c0c4820 tx=0x7f284c031490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e131158ab639d2a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.384+0000 7f283d7fa640 1 -- 172.21.15.28:0/3023741597 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c049070 con 0x560722d40980 2024-04-25T01:33:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.385+0000 7f283d7fa640 1 -- 172.21.15.28:0/3023741597 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f284c0ed000 con 0x560722d40980 2024-04-25T01:33:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.385+0000 7f283d7fa640 1 -- 172.21.15.28:0/3023741597 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c08e190 con 0x560722d40980 2024-04-25T01:33:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.385+0000 7f2860a0f740 1 -- 172.21.15.28:0/3023741597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d40980 msgr2=0x560722d4bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.385+0000 7f2860a0f740 1 --2- 172.21.15.28:0/3023741597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d40980 0x560722d4bc70 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f284c0c4820 tx=0x7f284c031490 comp rx=0 tx=0).stop 2024-04-25T01:33:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.385+0000 7f2860a0f740 1 -- 172.21.15.28:0/3023741597 shutdown_connections 2024-04-25T01:33:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.385+0000 7f2860a0f740 1 --2- 172.21.15.28:0/3023741597 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d4e420 0x560722d41470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.385+0000 7f2860a0f740 1 --2- 172.21.15.28:0/3023741597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d40980 0x560722d4bc70 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.385+0000 7f2860a0f740 1 --2- 172.21.15.28:0/3023741597 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d3f290 0x560722d3a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.385+0000 7f2860a0f740 1 -- 172.21.15.28:0/3023741597 >> 172.21.15.28:0/3023741597 conn(0x560722d07950 msgr2=0x560722d40d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.385+0000 7f2860a0f740 1 -- 172.21.15.28:0/3023741597 shutdown_connections 2024-04-25T01:33:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.385+0000 7f2860a0f740 1 -- 172.21.15.28:0/3023741597 wait complete. 2024-04-25T01:33:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.386+0000 7f2860a0f740 1 -- start start 2024-04-25T01:33:06.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.386+0000 7f2860a0f740 1 Processor -- start 2024-04-25T01:33:06.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.386+0000 7f2860a0f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d3f290 0x560722d56650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.386+0000 7f2860a0f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d40980 0x560722d52a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.386+0000 7f2860a0f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d4e420 0x560722d52f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.386+0000 7f2860a0f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2844011180 con 0x560722d4e420 2024-04-25T01:33:06.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.386+0000 7f2860a0f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2844010ab0 con 0x560722d3f290 2024-04-25T01:33:06.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.386+0000 7f2860a0f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f284400dcf0 con 0x560722d40980 2024-04-25T01:33:06.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.386+0000 7f2860a0f740 10 client.0 _ll_register_callbacks cb 0x7f2850009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:06.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.387+0000 7f2857fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d4e420 0x560722d52f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.387+0000 7f2857fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d4e420 0x560722d52f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55286/0 (socket says 172.21.15.28:55286) 2024-04-25T01:33:06.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.387+0000 7f2857fff640 1 -- 172.21.15.28:0/3368210961 learned_addr learned my addr 172.21.15.28:0/3368210961 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:06.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.387+0000 7f28577fe640 1 --2- 172.21.15.28:0/3368210961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d3f290 0x560722d56650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.387+0000 7f2856ffd640 1 --2- 172.21.15.28:0/3368210961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d40980 0x560722d52a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.387+0000 7f2857fff640 1 -- 172.21.15.28:0/3368210961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d40980 msgr2=0x560722d52a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:06.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.387+0000 7f2857fff640 1 --2- 172.21.15.28:0/3368210961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d40980 0x560722d52a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.387+0000 7f2857fff640 1 -- 172.21.15.28:0/3368210961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d3f290 msgr2=0x560722d56650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:06.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.387+0000 7f2857fff640 1 --2- 172.21.15.28:0/3368210961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d3f290 0x560722d56650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.387+0000 7f2857fff640 1 -- 172.21.15.28:0/3368210961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560722d535a0 con 0x560722d4e420 2024-04-25T01:33:06.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.387+0000 7f2857fff640 1 --2- 172.21.15.28:0/3368210961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d4e420 0x560722d52f60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2850010b10 tx=0x7f2850009e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=514d27d37bb017a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:06.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.387+0000 7f2836ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:06.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.387+0000 7f2836ffd640 1 -- 172.21.15.28:0/3368210961 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850021070 con 0x560722d4e420 2024-04-25T01:33:06.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.387+0000 7f2836ffd640 1 -- 172.21.15.28:0/3368210961 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f285000fce0 con 0x560722d4e420 2024-04-25T01:33:06.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.387+0000 7f2860a0f740 1 -- 172.21.15.28:0/3368210961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f284c0ed000 con 0x560722d4e420 2024-04-25T01:33:06.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.387+0000 7f2836ffd640 1 -- 172.21.15.28:0/3368210961 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f285002c050 con 0x560722d4e420 2024-04-25T01:33:06.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.387+0000 7f2860a0f740 10 client.15777 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:06.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.387+0000 7f2860a0f740 1 -- 172.21.15.28:0/3368210961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560722dc5560 con 0x560722d4e420 2024-04-25T01:33:06.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.388+0000 7f2836ffd640 1 -- 172.21.15.28:0/3368210961 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f284c0ed000 con 0x560722d4e420 2024-04-25T01:33:06.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.388+0000 7f2836ffd640 1 -- 172.21.15.28:0/3368210961 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f285000e4d0 con 0x560722d4e420 2024-04-25T01:33:06.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.388+0000 7f2860a0f740 10 client.15777 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:06.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.388+0000 7f2860a0f740 10 client.15777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:06.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.388+0000 7f2860a0f740 20 client.15777 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:06.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.388+0000 7f2860a0f740 1 --2- 172.21.15.28:0/3368210961 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x560722dc5d10 0x560722dc8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.388+0000 7f2860a0f740 4 client.15777 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:33:06.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.388+0000 7f2860a0f740 1 -- 172.21.15.28:0/3368210961 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560722cb0800 con 0x560722dc5d10 2024-04-25T01:33:06.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.388+0000 7f28577fe640 1 --2- 172.21.15.28:0/3368210961 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x560722dc5d10 0x560722dc8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.388+0000 7f2836ffd640 10 client.15777 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:06.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.389+0000 7f28577fe640 1 --2- 172.21.15.28:0/3368210961 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x560722dc5d10 0x560722dc8130 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf9745f1ca87dfab server_cookie=e7bbbfe00f78d696 in_seq=0 out_seq=0 2024-04-25T01:33:06.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.390+0000 7f2836ffd640 1 -- 172.21.15.28:0/3368210961 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560722cb0800 con 0x560722dc5d10 2024-04-25T01:33:06.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.390+0000 7f2836ffd640 10 client.15777 handle_command_reply: tid=0 2024-04-25T01:33:06.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.390+0000 7f2860a0f740 1 client.15777 shutdown 2024-04-25T01:33:06.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.390+0000 7f2860a0f740 1 -- 172.21.15.28:0/3368210961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d4e420 msgr2=0x560722d52f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.390+0000 7f2860a0f740 1 --2- 172.21.15.28:0/3368210961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d4e420 0x560722d52f60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2850010b10 tx=0x7f2850009e50 comp rx=0 tx=0).stop 2024-04-25T01:33:06.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.390+0000 7f2860a0f740 1 -- 172.21.15.28:0/3368210961 shutdown_connections 2024-04-25T01:33:06.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.390+0000 7f2860a0f740 1 --2- 172.21.15.28:0/3368210961 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x560722dc5d10 0x560722dc8130 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.390+0000 7f2860a0f740 1 --2- 172.21.15.28:0/3368210961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560722d4e420 0x560722d52f60 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.390+0000 7f2860a0f740 1 --2- 172.21.15.28:0/3368210961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560722d40980 0x560722d52a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.390+0000 7f2860a0f740 1 --2- 172.21.15.28:0/3368210961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560722d3f290 0x560722d56650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.390+0000 7f2860a0f740 1 -- 172.21.15.28:0/3368210961 >> 172.21.15.28:0/3368210961 conn(0x560722d07950 msgr2=0x560722d415f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:06.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.390+0000 7f2836ffd640 0 client.15777 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:06.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.390+0000 7f2860a0f740 1 -- 172.21.15.28:0/3368210961 shutdown_connections 2024-04-25T01:33:06.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.390+0000 7f2860a0f740 1 -- 172.21.15.28:0/3368210961 wait complete. 2024-04-25T01:33:06.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.390+0000 7f2860a0f740 20 client.15777 trim_cache size 0 max 16384 2024-04-25T01:33:06.373 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:33:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:33:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "a99d3ad2-ea93-436e-a506-707174e63fb3", 2024-04-25T01:33:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:33:06.374 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:33:06.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.391+0000 7f285dd09640 1 -- 172.21.15.28:0/1047119553 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f28400a2e10 msgr2=0x7f28400a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.391+0000 7f285dd09640 1 --2- 172.21.15.28:0/1047119553 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f28400a2e10 0x7f28400a52d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2858101750 tx=0x7f2848005f90 comp rx=0 tx=0).stop 2024-04-25T01:33:06.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.391+0000 7f285dd09640 1 -- 172.21.15.28:0/1047119553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2858100fd0 msgr2=0x7f2858198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.391+0000 7f285dd09640 1 --2- 172.21.15.28:0/1047119553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2858100fd0 0x7f2858198580 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f284c02a940 tx=0x7f284c003b20 comp rx=0 tx=0).stop 2024-04-25T01:33:06.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.391+0000 7f285dd09640 1 -- 172.21.15.28:0/1047119553 shutdown_connections 2024-04-25T01:33:06.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.391+0000 7f285dd09640 1 --2- 172.21.15.28:0/1047119553 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f28400a2e10 0x7f28400a52d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.391+0000 7f285dd09640 1 --2- 172.21.15.28:0/1047119553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28581066b0 0x7f285819c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.391+0000 7f285dd09640 1 --2- 172.21.15.28:0/1047119553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2858100fd0 0x7f2858198580 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.391+0000 7f285dd09640 1 --2- 172.21.15.28:0/1047119553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2858100610 0x7f2858198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.391+0000 7f285dd09640 1 -- 172.21.15.28:0/1047119553 >> 172.21.15.28:0/1047119553 conn(0x7f2858077740 msgr2=0x7f28580ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:06.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.391+0000 7f285dd09640 1 -- 172.21.15.28:0/1047119553 shutdown_connections 2024-04-25T01:33:06.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.391+0000 7f285dd09640 1 -- 172.21.15.28:0/1047119553 wait complete. 2024-04-25T01:33:06.383 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:33:06.383 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:33:06.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: Updating smithi028:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:33:06.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: Updating smithi130:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:33:06.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: Updating smithi032:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:33:06.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:33:06.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:33:06.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:33:06.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:33:06.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: Updating smithi130:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:33:06.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: Updating smithi032:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:33:06.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: Updating smithi028:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:33:06.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-04-25T01:33:06.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2617255544' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:33:06.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.459 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:33:06.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:33:06.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:33:06.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1976109733' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:33:06.460 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:06 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3593217540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:33:06.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: Updating smithi028:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:33:06.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: Updating smithi130:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:33:06.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: Updating smithi032:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:33:06.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:33:06.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:33:06.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:33:06.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:33:06.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: Updating smithi130:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:33:06.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: Updating smithi032:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:33:06.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: Updating smithi028:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:33:06.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-04-25T01:33:06.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2617255544' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:33:06.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:33:06.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:33:06.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:33:06.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1976109733' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:33:06.485 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:06 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3593217540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:33:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.510+0000 7ff17e316640 1 -- 172.21.15.28:0/3615065141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff178100fd0 msgr2=0x7ff17810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.510+0000 7ff17e316640 1 --2- 172.21.15.28:0/3615065141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff178100fd0 0x7ff17810d4e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff170009cd0 tx=0x7ff17002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.510+0000 7ff17e316640 1 -- 172.21.15.28:0/3615065141 shutdown_connections 2024-04-25T01:33:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.510+0000 7ff17e316640 1 --2- 172.21.15.28:0/3615065141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff178100fd0 0x7ff17810d4e0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.510+0000 7ff17e316640 1 --2- 172.21.15.28:0/3615065141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff178100610 0x7ff178100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.510+0000 7ff17e316640 1 --2- 172.21.15.28:0/3615065141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff1781066b0 0x7ff178106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.510+0000 7ff17e316640 1 -- 172.21.15.28:0/3615065141 >> 172.21.15.28:0/3615065141 conn(0x7ff178077740 msgr2=0x7ff178077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.510+0000 7ff17e316640 1 -- 172.21.15.28:0/3615065141 shutdown_connections 2024-04-25T01:33:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.510+0000 7ff17e316640 1 -- 172.21.15.28:0/3615065141 wait complete. 2024-04-25T01:33:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.510+0000 7ff17e316640 1 Processor -- start 2024-04-25T01:33:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.510+0000 7ff17e316640 1 -- start start 2024-04-25T01:33:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.511+0000 7ff17e316640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff178100610 0x7ff1781022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.511+0000 7ff17e316640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff178100fd0 0x7ff1781027f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.511+0000 7ff17e316640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1781066b0 0x7ff178102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.511+0000 7ff17e316640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1781109f0 con 0x7ff1781066b0 2024-04-25T01:33:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.511+0000 7ff17e316640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff178110870 con 0x7ff178100fd0 2024-04-25T01:33:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.511+0000 7ff17e316640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff178110b70 con 0x7ff178100610 2024-04-25T01:33:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.511+0000 7ff17c8b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1781066b0 0x7ff178102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.511+0000 7ff17c8b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1781066b0 0x7ff178102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55310/0 (socket says 172.21.15.28:55310) 2024-04-25T01:33:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.511+0000 7ff17c8b3640 1 -- 172.21.15.28:0/1189865622 learned_addr learned my addr 172.21.15.28:0/1189865622 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.511+0000 7ff1777fe640 1 --2- 172.21.15.28:0/1189865622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff178100fd0 0x7ff1781027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.511+0000 7ff177fff640 1 --2- 172.21.15.28:0/1189865622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff178100610 0x7ff1781022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.511+0000 7ff17c8b3640 1 -- 172.21.15.28:0/1189865622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff178100610 msgr2=0x7ff1781022b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.511+0000 7ff17c8b3640 1 --2- 172.21.15.28:0/1189865622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff178100610 0x7ff1781022b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.511+0000 7ff17c8b3640 1 -- 172.21.15.28:0/1189865622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff178100fd0 msgr2=0x7ff1781027f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.511+0000 7ff17c8b3640 1 --2- 172.21.15.28:0/1189865622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff178100fd0 0x7ff1781027f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.511+0000 7ff17c8b3640 1 -- 172.21.15.28:0/1189865622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1781a6460 con 0x7ff1781066b0 2024-04-25T01:33:06.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.511+0000 7ff17c8b3640 1 --2- 172.21.15.28:0/1189865622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1781066b0 0x7ff178102d30 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff170005cc0 tx=0x7ff170032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:06.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.512+0000 7ff1757fa640 1 -- 172.21.15.28:0/1189865622 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff170004040 con 0x7ff1781066b0 2024-04-25T01:33:06.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.512+0000 7ff1757fa640 1 -- 172.21.15.28:0/1189865622 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1700046c0 con 0x7ff1781066b0 2024-04-25T01:33:06.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.512+0000 7ff17e316640 1 -- 172.21.15.28:0/1189865622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1781a66f0 con 0x7ff1781066b0 2024-04-25T01:33:06.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.512+0000 7ff1757fa640 1 -- 172.21.15.28:0/1189865622 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff170030e00 con 0x7ff1781066b0 2024-04-25T01:33:06.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.512+0000 7ff17e316640 1 -- 172.21.15.28:0/1189865622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1781a6c40 con 0x7ff1781066b0 2024-04-25T01:33:06.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.514+0000 7ff18101c740 1 Processor -- start 2024-04-25T01:33:06.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.514+0000 7ff1757fa640 1 -- 172.21.15.28:0/1189865622 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7ff1700305a0 con 0x7ff1781066b0 2024-04-25T01:33:06.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.514+0000 7ff18101c740 1 -- start start 2024-04-25T01:33:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.514+0000 7ff18101c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384aba630 0x55b384abaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.514+0000 7ff18101c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384abaf50 0x55b384ac50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.514+0000 7ff18101c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384ac5620 0x55b384ac7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.514+0000 7ff18101c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b384a1f6b0 con 0x55b384abaf50 2024-04-25T01:33:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.514+0000 7ff18101c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b3849c7880 con 0x55b384ac5620 2024-04-25T01:33:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.514+0000 7ff18101c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b384a1b670 con 0x55b384aba630 2024-04-25T01:33:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.515+0000 7ff177fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384abaf50 0x55b384ac50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.515+0000 7ff177fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384abaf50 0x55b384ac50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55318/0 (socket says 172.21.15.28:55318) 2024-04-25T01:33:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.515+0000 7ff177fff640 1 -- 172.21.15.28:0/3227390647 learned_addr learned my addr 172.21.15.28:0/3227390647 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.515+0000 7ff1777fe640 1 --2- 172.21.15.28:0/3227390647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384aba630 0x55b384abaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.515+0000 7ff1777fe640 1 --2- 172.21.15.28:0/3227390647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384ac5620 0x55b384ac7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.515+0000 7ff1757fa640 1 --2- 172.21.15.28:0/1189865622 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff1640a2d90 0x7ff1640a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.515+0000 7ff177fff640 1 -- 172.21.15.28:0/3227390647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384aba630 msgr2=0x55b384abaa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.515+0000 7ff177fff640 1 --2- 172.21.15.28:0/3227390647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384aba630 0x55b384abaa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.515+0000 7ff177fff640 1 -- 172.21.15.28:0/3227390647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384ac5620 msgr2=0x55b384ac7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.515+0000 7ff177fff640 1 --2- 172.21.15.28:0/3227390647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384ac5620 0x55b384ac7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.515+0000 7ff177fff640 1 -- 172.21.15.28:0/3227390647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3847fd7d0 con 0x55b384abaf50 2024-04-25T01:33:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.515+0000 7ff1777fe640 1 --2- 172.21.15.28:0/3227390647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384aba630 0x55b384abaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.515+0000 7ff17c8b3640 1 --2- 172.21.15.28:0/1189865622 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff1640a2d90 0x7ff1640a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.515+0000 7ff177fff640 1 --2- 172.21.15.28:0/3227390647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384abaf50 0x55b384ac50e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff168004590 tx=0x7ff16800bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb5775f548a83b5b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.515+0000 7ff1757fa640 1 -- 172.21.15.28:0/1189865622 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff17002e030 con 0x7ff1781066b0 2024-04-25T01:33:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.515+0000 7ff16e7fc640 1 -- 172.21.15.28:0/3227390647 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff168003bd0 con 0x55b384abaf50 2024-04-25T01:33:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.515+0000 7ff16e7fc640 1 -- 172.21.15.28:0/3227390647 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff168015070 con 0x55b384abaf50 2024-04-25T01:33:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.515+0000 7ff16e7fc640 1 -- 172.21.15.28:0/3227390647 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1680116a0 con 0x55b384abaf50 2024-04-25T01:33:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.516+0000 7ff17c8b3640 1 --2- 172.21.15.28:0/1189865622 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff1640a2d90 0x7ff1640a5250 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff1700ee480 tx=0x7ff1700c0040 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.516+0000 7ff18101c740 1 -- 172.21.15.28:0/3227390647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384abaf50 msgr2=0x55b384ac50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.516+0000 7ff18101c740 1 --2- 172.21.15.28:0/3227390647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384abaf50 0x55b384ac50e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff168004590 tx=0x7ff16800bb20 comp rx=0 tx=0).stop 2024-04-25T01:33:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.516+0000 7ff18101c740 1 -- 172.21.15.28:0/3227390647 shutdown_connections 2024-04-25T01:33:06.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.516+0000 7ff18101c740 1 --2- 172.21.15.28:0/3227390647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384ac5620 0x55b384ac7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.516+0000 7ff18101c740 1 --2- 172.21.15.28:0/3227390647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384abaf50 0x55b384ac50e0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.516+0000 7ff18101c740 1 --2- 172.21.15.28:0/3227390647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384aba630 0x55b384abaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.516+0000 7ff18101c740 1 -- 172.21.15.28:0/3227390647 >> 172.21.15.28:0/3227390647 conn(0x55b384ab6950 msgr2=0x55b384ab6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:06.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.516+0000 7ff18101c740 1 -- 172.21.15.28:0/3227390647 shutdown_connections 2024-04-25T01:33:06.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.516+0000 7ff18101c740 1 -- 172.21.15.28:0/3227390647 wait complete. 2024-04-25T01:33:06.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.517+0000 7ff18101c740 1 -- start start 2024-04-25T01:33:06.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.517+0000 7ff18101c740 1 Processor -- start 2024-04-25T01:33:06.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.517+0000 7ff18101c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384aba630 0x55b384ac2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.517+0000 7ff18101c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384abaf50 0x55b384ac2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.517+0000 7ff18101c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384ac5620 0x55b384ac2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.517+0000 7ff18101c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b384ac8410 con 0x55b384aba630 2024-04-25T01:33:06.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.517+0000 7ff18101c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b3849daf50 con 0x55b384ac5620 2024-04-25T01:33:06.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.517+0000 7ff18101c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b384ac8590 con 0x55b384abaf50 2024-04-25T01:33:06.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.517+0000 7ff18101c740 10 client.0 _ll_register_callbacks cb 0x55b384a16000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:06.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.517+0000 7ff1777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384aba630 0x55b384ac2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.517+0000 7ff1777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384aba630 0x55b384ac2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55326/0 (socket says 172.21.15.28:55326) 2024-04-25T01:33:06.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.517+0000 7ff1777fe640 1 -- 172.21.15.28:0/1402844680 learned_addr learned my addr 172.21.15.28:0/1402844680 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:06.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.517+0000 7ff1777fe640 1 --2- 172.21.15.28:0/1402844680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384ac5620 0x55b384ac2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.517+0000 7ff177fff640 1 --2- 172.21.15.28:0/1402844680 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384abaf50 0x55b384ac2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.518+0000 7ff1777fe640 1 -- 172.21.15.28:0/1402844680 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384abaf50 msgr2=0x55b384ac2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:06.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.518+0000 7ff1777fe640 1 --2- 172.21.15.28:0/1402844680 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384abaf50 0x55b384ac2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.518+0000 7ff1777fe640 1 -- 172.21.15.28:0/1402844680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384ac5620 msgr2=0x55b384ac2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.518+0000 7ff1777fe640 1 --2- 172.21.15.28:0/1402844680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384ac5620 0x55b384ac2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.518+0000 7ff1777fe640 1 -- 172.21.15.28:0/1402844680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3847333f0 con 0x55b384aba630 2024-04-25T01:33:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.518+0000 7ff1777fe640 1 --2- 172.21.15.28:0/1402844680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384aba630 0x55b384ac2570 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff160011f10 tx=0x7ff160009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da3302872d0f4d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.518+0000 7ff1577fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.518+0000 7ff1577fe640 1 -- 172.21.15.28:0/1402844680 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff160019040 con 0x55b384aba630 2024-04-25T01:33:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.518+0000 7ff1577fe640 1 -- 172.21.15.28:0/1402844680 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff160005220 con 0x55b384aba630 2024-04-25T01:33:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.518+0000 7ff18101c740 1 -- 172.21.15.28:0/1402844680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff168015070 con 0x55b384aba630 2024-04-25T01:33:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.518+0000 7ff1577fe640 1 -- 172.21.15.28:0/1402844680 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff16002b050 con 0x55b384aba630 2024-04-25T01:33:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.518+0000 7ff18101c740 10 client.15789 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.518+0000 7ff18101c740 1 -- 172.21.15.28:0/1402844680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b3846f30f0 con 0x55b384aba630 2024-04-25T01:33:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.518+0000 7ff1577fe640 1 -- 172.21.15.28:0/1402844680 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff168015070 con 0x55b384aba630 2024-04-25T01:33:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.518+0000 7ff1577fe640 1 -- 172.21.15.28:0/1402844680 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff16001c030 con 0x55b384aba630 2024-04-25T01:33:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.519+0000 7ff18101c740 10 client.15789 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.519+0000 7ff18101c740 10 client.15789 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.519+0000 7ff18101c740 20 client.15789 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.519+0000 7ff18101c740 1 --2- 172.21.15.28:0/1402844680 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b384ad5910 0x55b384ad7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.519+0000 7ff18101c740 4 client.15789 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.519+0000 7ff18101c740 1 -- 172.21.15.28:0/1402844680 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b384a2cbf0 con 0x55b384ad5910 2024-04-25T01:33:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.519+0000 7ff177fff640 1 --2- 172.21.15.28:0/1402844680 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b384ad5910 0x55b384ad7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.519+0000 7ff177fff640 1 --2- 172.21.15.28:0/1402844680 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b384ad5910 0x55b384ad7d40 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b43238d7c99dbc77 server_cookie=7f3bb2ccd3c26103 in_seq=0 out_seq=0 2024-04-25T01:33:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.519+0000 7ff1577fe640 10 client.15789 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.520+0000 7ff1577fe640 1 -- 172.21.15.28:0/1402844680 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b384a2cbf0 con 0x55b384ad5910 2024-04-25T01:33:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.520+0000 7ff1577fe640 10 client.15789 handle_command_reply: tid=0 2024-04-25T01:33:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.520+0000 7ff18101c740 1 client.15789 shutdown 2024-04-25T01:33:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.520+0000 7ff18101c740 1 -- 172.21.15.28:0/1402844680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384aba630 msgr2=0x55b384ac2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.520+0000 7ff18101c740 1 --2- 172.21.15.28:0/1402844680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384aba630 0x55b384ac2570 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff160011f10 tx=0x7ff160009b40 comp rx=0 tx=0).stop 2024-04-25T01:33:06.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.520+0000 7ff18101c740 1 -- 172.21.15.28:0/1402844680 shutdown_connections 2024-04-25T01:33:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.520+0000 7ff18101c740 1 --2- 172.21.15.28:0/1402844680 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b384ad5910 0x55b384ad7d40 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.520+0000 7ff18101c740 1 --2- 172.21.15.28:0/1402844680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384ac5620 0x55b384ac2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.520+0000 7ff18101c740 1 --2- 172.21.15.28:0/1402844680 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384abaf50 0x55b384ac2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.520+0000 7ff1577fe640 0 client.15789 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.520+0000 7ff18101c740 1 --2- 172.21.15.28:0/1402844680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384aba630 0x55b384ac2570 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.520+0000 7ff18101c740 1 -- 172.21.15.28:0/1402844680 >> 172.21.15.28:0/1402844680 conn(0x55b384ab6950 msgr2=0x55b384abc750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.520+0000 7ff18101c740 1 -- 172.21.15.28:0/1402844680 shutdown_connections 2024-04-25T01:33:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.520+0000 7ff18101c740 1 -- 172.21.15.28:0/1402844680 wait complete. 2024-04-25T01:33:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.520+0000 7ff18101c740 20 client.15789 trim_cache size 0 max 16384 2024-04-25T01:33:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.538+0000 7ff18101c740 1 Processor -- start 2024-04-25T01:33:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.539+0000 7ff18101c740 1 -- start start 2024-04-25T01:33:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.539+0000 7ff18101c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384aba630 0x55b384ad12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.539+0000 7ff18101c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384abaf50 0x55b384abcea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.539+0000 7ff18101c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384ac5620 0x55b384ac2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.539+0000 7ff18101c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1781b0bc0 con 0x55b384ac5620 2024-04-25T01:33:06.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.539+0000 7ff18101c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff1781b2700 con 0x55b384abaf50 2024-04-25T01:33:06.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.539+0000 7ff18101c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff1781b1290 con 0x55b384aba630 2024-04-25T01:33:06.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.539+0000 7ff1777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384ac5620 0x55b384ac2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.539+0000 7ff1777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384ac5620 0x55b384ac2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55336/0 (socket says 172.21.15.28:55336) 2024-04-25T01:33:06.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.539+0000 7ff1777fe640 1 -- 172.21.15.28:0/2478566187 learned_addr learned my addr 172.21.15.28:0/2478566187 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:06.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.539+0000 7ff177fff640 1 --2- 172.21.15.28:0/2478566187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384abaf50 0x55b384abcea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.539+0000 7ff1777fe640 1 --2- 172.21.15.28:0/2478566187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384aba630 0x55b384ad12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.539+0000 7ff1777fe640 1 -- 172.21.15.28:0/2478566187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384aba630 msgr2=0x55b384ad12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:06.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.539+0000 7ff1777fe640 1 --2- 172.21.15.28:0/2478566187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384aba630 0x55b384ad12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.539+0000 7ff1777fe640 1 -- 172.21.15.28:0/2478566187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384abaf50 msgr2=0x55b384abcea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.539+0000 7ff1777fe640 1 --2- 172.21.15.28:0/2478566187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384abaf50 0x55b384abcea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.539+0000 7ff1777fe640 1 -- 172.21.15.28:0/2478566187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b384aa8090 con 0x55b384ac5620 2024-04-25T01:33:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.539+0000 7ff1777fe640 1 --2- 172.21.15.28:0/2478566187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384ac5620 0x55b384ac2c40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff1600204f0 tx=0x7ff160005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf3dd11069dcb1a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.540+0000 7ff16dffb640 1 -- 172.21.15.28:0/2478566187 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff160020770 con 0x55b384ac5620 2024-04-25T01:33:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.540+0000 7ff16dffb640 1 -- 172.21.15.28:0/2478566187 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff160011d10 con 0x55b384ac5620 2024-04-25T01:33:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.540+0000 7ff16dffb640 1 -- 172.21.15.28:0/2478566187 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff160004920 con 0x55b384ac5620 2024-04-25T01:33:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.540+0000 7ff18101c740 1 -- 172.21.15.28:0/2478566187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384ac5620 msgr2=0x55b384ac2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.540+0000 7ff18101c740 1 --2- 172.21.15.28:0/2478566187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384ac5620 0x55b384ac2c40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff1600204f0 tx=0x7ff160005f90 comp rx=0 tx=0).stop 2024-04-25T01:33:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.540+0000 7ff18101c740 1 -- 172.21.15.28:0/2478566187 shutdown_connections 2024-04-25T01:33:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.540+0000 7ff18101c740 1 --2- 172.21.15.28:0/2478566187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384ac5620 0x55b384ac2c40 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.540+0000 7ff18101c740 1 --2- 172.21.15.28:0/2478566187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384abaf50 0x55b384abcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.540+0000 7ff18101c740 1 --2- 172.21.15.28:0/2478566187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384aba630 0x55b384ad12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.540+0000 7ff18101c740 1 -- 172.21.15.28:0/2478566187 >> 172.21.15.28:0/2478566187 conn(0x55b384ab6950 msgr2=0x55b384ac2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.540+0000 7ff18101c740 1 -- 172.21.15.28:0/2478566187 shutdown_connections 2024-04-25T01:33:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.540+0000 7ff18101c740 1 -- 172.21.15.28:0/2478566187 wait complete. 2024-04-25T01:33:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.541+0000 7ff18101c740 1 -- start start 2024-04-25T01:33:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.541+0000 7ff18101c740 1 Processor -- start 2024-04-25T01:33:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.541+0000 7ff18101c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384aba630 0x55b384acd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.541+0000 7ff18101c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384abaf50 0x55b384acdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.541+0000 7ff18101c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384ac5620 0x55b384abe960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.541+0000 7ff18101c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1781b2030 con 0x55b384aba630 2024-04-25T01:33:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.541+0000 7ff18101c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff1781b1960 con 0x55b384ac5620 2024-04-25T01:33:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.541+0000 7ff18101c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff1781aeba0 con 0x55b384abaf50 2024-04-25T01:33:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.541+0000 7ff18101c740 10 client.0 _ll_register_callbacks cb 0x7ff160009cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.541+0000 7ff1777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384aba630 0x55b384acd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.541+0000 7ff1777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384aba630 0x55b384acd7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55340/0 (socket says 172.21.15.28:55340) 2024-04-25T01:33:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.541+0000 7ff1777fe640 1 -- 172.21.15.28:0/44265482 learned_addr learned my addr 172.21.15.28:0/44265482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.541+0000 7ff1777fe640 1 --2- 172.21.15.28:0/44265482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384ac5620 0x55b384abe960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.541+0000 7ff177fff640 1 --2- 172.21.15.28:0/44265482 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384abaf50 0x55b384acdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.542+0000 7ff1777fe640 1 -- 172.21.15.28:0/44265482 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384abaf50 msgr2=0x55b384acdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.542+0000 7ff1777fe640 1 --2- 172.21.15.28:0/44265482 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384abaf50 0x55b384acdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.542+0000 7ff1777fe640 1 -- 172.21.15.28:0/44265482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384ac5620 msgr2=0x55b384abe960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.542+0000 7ff1777fe640 1 --2- 172.21.15.28:0/44265482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384ac5620 0x55b384abe960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.542+0000 7ff1777fe640 1 -- 172.21.15.28:0/44265482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b384abf160 con 0x55b384aba630 2024-04-25T01:33:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.542+0000 7ff1777fe640 1 --2- 172.21.15.28:0/44265482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384aba630 0x55b384acd7e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff160009c20 tx=0x7ff160004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41eeda4dc8c20d21 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.542+0000 7ff1577fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.542+0000 7ff1577fe640 1 -- 172.21.15.28:0/44265482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1600209f0 con 0x55b384aba630 2024-04-25T01:33:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.542+0000 7ff1577fe640 1 -- 172.21.15.28:0/44265482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff160020b90 con 0x55b384aba630 2024-04-25T01:33:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.542+0000 7ff18101c740 1 -- 172.21.15.28:0/44265482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff160011d10 con 0x55b384aba630 2024-04-25T01:33:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.542+0000 7ff1577fe640 1 -- 172.21.15.28:0/44265482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff160020d90 con 0x55b384aba630 2024-04-25T01:33:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.542+0000 7ff18101c740 10 client.15795 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.542+0000 7ff18101c740 1 -- 172.21.15.28:0/44265482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b384b3b930 con 0x55b384aba630 2024-04-25T01:33:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.542+0000 7ff1577fe640 1 -- 172.21.15.28:0/44265482 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff160011d10 con 0x55b384aba630 2024-04-25T01:33:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.543+0000 7ff1577fe640 1 -- 172.21.15.28:0/44265482 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff16001c070 con 0x55b384aba630 2024-04-25T01:33:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.543+0000 7ff18101c740 10 client.15795 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.543+0000 7ff18101c740 10 client.15795 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.543+0000 7ff18101c740 20 client.15795 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.543+0000 7ff18101c740 1 --2- 172.21.15.28:0/44265482 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b384b3bf90 0x55b384b3e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.543+0000 7ff18101c740 4 client.15795 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:33:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.543+0000 7ff18101c740 1 -- 172.21.15.28:0/44265482 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b384a26f40 con 0x55b384b3bf90 2024-04-25T01:33:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.543+0000 7ff177fff640 1 --2- 172.21.15.28:0/44265482 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b384b3bf90 0x55b384b3e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.543+0000 7ff177fff640 1 --2- 172.21.15.28:0/44265482 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b384b3bf90 0x55b384b3e3b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=984466a5844ec412 server_cookie=7835d525cba3d289 in_seq=0 out_seq=0 2024-04-25T01:33:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.543+0000 7ff1577fe640 10 client.15795 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.543+0000 7ff1577fe640 1 -- 172.21.15.28:0/44265482 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b384a26f40 con 0x55b384b3bf90 2024-04-25T01:33:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.543+0000 7ff1577fe640 10 client.15795 handle_command_reply: tid=0 2024-04-25T01:33:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.543+0000 7ff18101c740 1 client.15795 shutdown 2024-04-25T01:33:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff18101c740 1 -- 172.21.15.28:0/44265482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384aba630 msgr2=0x55b384acd7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff18101c740 1 --2- 172.21.15.28:0/44265482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384aba630 0x55b384acd7e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff160009c20 tx=0x7ff160004470 comp rx=0 tx=0).stop 2024-04-25T01:33:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff18101c740 1 -- 172.21.15.28:0/44265482 shutdown_connections 2024-04-25T01:33:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff18101c740 1 --2- 172.21.15.28:0/44265482 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b384b3bf90 0x55b384b3e3b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff18101c740 1 --2- 172.21.15.28:0/44265482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b384ac5620 0x55b384abe960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff18101c740 1 --2- 172.21.15.28:0/44265482 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b384abaf50 0x55b384acdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff1577fe640 0 client.15795 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:06.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff18101c740 1 --2- 172.21.15.28:0/44265482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b384aba630 0x55b384acd7e0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff18101c740 1 -- 172.21.15.28:0/44265482 >> 172.21.15.28:0/44265482 conn(0x55b384ab6950 msgr2=0x55b384ab5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:06.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff18101c740 1 -- 172.21.15.28:0/44265482 shutdown_connections 2024-04-25T01:33:06.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff18101c740 1 -- 172.21.15.28:0/44265482 wait complete. 2024-04-25T01:33:06.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff18101c740 20 client.15795 trim_cache size 0 max 16384 2024-04-25T01:33:06.531 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:33:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:33:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-25T01:33:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "a99d3ad2-ea93-436e-a506-707174e63fb3": { 2024-04-25T01:33:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-25T01:33:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "a99d3ad2-ea93-436e-a506-707174e63fb3", 2024-04-25T01:33:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-25T01:33:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:33:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:33:06.532 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:33:06.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff17e316640 1 -- 172.21.15.28:0/1189865622 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff1640a2d90 msgr2=0x7ff1640a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff17e316640 1 --2- 172.21.15.28:0/1189865622 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff1640a2d90 0x7ff1640a5250 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff1700ee480 tx=0x7ff1700c0040 comp rx=0 tx=0).stop 2024-04-25T01:33:06.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff17e316640 1 -- 172.21.15.28:0/1189865622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1781066b0 msgr2=0x7ff178102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff17e316640 1 --2- 172.21.15.28:0/1189865622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1781066b0 0x7ff178102d30 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff170005cc0 tx=0x7ff170032f40 comp rx=0 tx=0).stop 2024-04-25T01:33:06.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff17e316640 1 -- 172.21.15.28:0/1189865622 shutdown_connections 2024-04-25T01:33:06.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff17e316640 1 --2- 172.21.15.28:0/1189865622 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff1640a2d90 0x7ff1640a5250 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff17e316640 1 --2- 172.21.15.28:0/1189865622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1781066b0 0x7ff178102d30 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff17e316640 1 --2- 172.21.15.28:0/1189865622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff178100fd0 0x7ff1781027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff17e316640 1 --2- 172.21.15.28:0/1189865622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff178100610 0x7ff1781022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.544+0000 7ff17e316640 1 -- 172.21.15.28:0/1189865622 >> 172.21.15.28:0/1189865622 conn(0x7ff178077740 msgr2=0x7ff1780ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:06.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.545+0000 7ff17e316640 1 -- 172.21.15.28:0/1189865622 shutdown_connections 2024-04-25T01:33:06.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.545+0000 7ff17e316640 1 -- 172.21.15.28:0/1189865622 wait complete. 2024-04-25T01:33:06.534 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: Updating smithi028:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:33:06.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: Updating smithi130:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:33:06.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: Updating smithi032:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:33:06.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:33:06.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: Updating smithi130:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:33:06.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:33:06.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:33:06.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: Updating smithi130:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:33:06.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: Updating smithi032:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:33:06.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: Updating smithi028:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:33:06.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: mgrmap e23: x(active, since 4s), standbys: z, y 2024-04-25T01:33:06.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2617255544' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:33:06.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:33:06.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:33:06.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:33:06.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:33:06.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1976109733' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:33:06.536 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:06 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3593217540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:33:06.537 INFO:tasks.cephfs.filesystem:scrub status for tag:a99d3ad2-ea93-436e-a506-707174e63fb3 - {'path': '/', 'tag': 'a99d3ad2-ea93-436e-a506-707174e63fb3', 'options': 'recursive,force'} 2024-04-25T01:33:06.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.583+0000 7f6ba1116640 1 -- 172.21.15.28:0/1582657603 --> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f6b9c0fdca0 con 0x7f6b840a2e40 2024-04-25T01:33:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.655+0000 7f6b83fff640 1 -- 172.21.15.28:0/1582657603 <== mgr.15561 v2:172.21.15.28:6800/712762371 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6b9c0fdca0 con 0x7f6b840a2e40 2024-04-25T01:33:06.633 INFO:teuthology.orchestra.run.smithi028.stdout:Schedule set for path / 2024-04-25T01:33:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.657+0000 7f6ba1116640 1 -- 172.21.15.28:0/1582657603 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6b840a2e40 msgr2=0x7f6b840a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.657+0000 7f6ba1116640 1 --2- 172.21.15.28:0/1582657603 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6b840a2e40 0x7f6b840a5300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c10d910 tx=0x7f6b90031490 comp rx=0 tx=0).stop 2024-04-25T01:33:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.657+0000 7f6ba1116640 1 -- 172.21.15.28:0/1582657603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b9c1029d0 msgr2=0x7f6b9c197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.657+0000 7f6ba1116640 1 --2- 172.21.15.28:0/1582657603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b9c1029d0 0x7f6b9c197eb0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400c920 tx=0x7f6b9400cde0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.657+0000 7f6ba1116640 1 -- 172.21.15.28:0/1582657603 shutdown_connections 2024-04-25T01:33:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.657+0000 7f6ba1116640 1 --2- 172.21.15.28:0/1582657603 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6b840a2e40 0x7f6b840a5300 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.657+0000 7f6ba1116640 1 --2- 172.21.15.28:0/1582657603 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b9c10bd10 0x7f6b9c19c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.657+0000 7f6ba1116640 1 --2- 172.21.15.28:0/1582657603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b9c1032f0 0x7f6b9c1983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.657+0000 7f6ba1116640 1 --2- 172.21.15.28:0/1582657603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b9c1029d0 0x7f6b9c197eb0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.657+0000 7f6ba1116640 1 -- 172.21.15.28:0/1582657603 >> 172.21.15.28:0/1582657603 conn(0x7f6b9c0777c0 msgr2=0x7f6b9c0fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.658+0000 7f6ba1116640 1 -- 172.21.15.28:0/1582657603 shutdown_connections 2024-04-25T01:33:06.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.658+0000 7f6ba1116640 1 -- 172.21.15.28:0/1582657603 wait complete. 2024-04-25T01:33:06.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-25T01:33:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.814+0000 7f4373914640 1 -- 172.21.15.28:0/2036074409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f436c10bd10 msgr2=0x7f436c10e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.814+0000 7f4373914640 1 --2- 172.21.15.28:0/2036074409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f436c10bd10 0x7f436c10e1b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f4364009ca0 tx=0x7f436402a9e0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.814+0000 7f4373914640 1 -- 172.21.15.28:0/2036074409 shutdown_connections 2024-04-25T01:33:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.814+0000 7f4373914640 1 --2- 172.21.15.28:0/2036074409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f436c10bd10 0x7f436c10e1b0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.814+0000 7f4373914640 1 --2- 172.21.15.28:0/2036074409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f436c1032f0 0x7f436c10b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.814+0000 7f4373914640 1 --2- 172.21.15.28:0/2036074409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f436c1029d0 0x7f436c102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.815+0000 7f4373914640 1 -- 172.21.15.28:0/2036074409 >> 172.21.15.28:0/2036074409 conn(0x7f436c0777c0 msgr2=0x7f436c077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.815+0000 7f4373914640 1 -- 172.21.15.28:0/2036074409 shutdown_connections 2024-04-25T01:33:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.815+0000 7f4373914640 1 -- 172.21.15.28:0/2036074409 wait complete. 2024-04-25T01:33:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.815+0000 7f4373914640 1 Processor -- start 2024-04-25T01:33:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.815+0000 7f4373914640 1 -- start start 2024-04-25T01:33:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.815+0000 7f4373914640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f436c1029d0 0x7f436c100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.815+0000 7f4373914640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f436c1032f0 0x7f436c100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.815+0000 7f4373914640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f436c10bd10 0x7f436c1010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.815+0000 7f4373914640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f436c111b10 con 0x7f436c1032f0 2024-04-25T01:33:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.815+0000 7f4373914640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f436c111990 con 0x7f436c10bd10 2024-04-25T01:33:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.815+0000 7f4373914640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f436c111c90 con 0x7f436c1029d0 2024-04-25T01:33:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.815+0000 7f4370eaf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f436c1032f0 0x7f436c100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.816+0000 7f4370eaf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f436c1032f0 0x7f436c100b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39244/0 (socket says 172.21.15.28:39244) 2024-04-25T01:33:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.816+0000 7f4370eaf640 1 -- 172.21.15.28:0/3202815168 learned_addr learned my addr 172.21.15.28:0/3202815168 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.816+0000 7f4371eb1640 1 --2- 172.21.15.28:0/3202815168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f436c10bd10 0x7f436c1010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.816+0000 7f43716b0640 1 --2- 172.21.15.28:0/3202815168 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f436c1029d0 0x7f436c100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.816+0000 7f4370eaf640 1 -- 172.21.15.28:0/3202815168 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f436c1029d0 msgr2=0x7f436c100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.816+0000 7f4370eaf640 1 --2- 172.21.15.28:0/3202815168 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f436c1029d0 0x7f436c100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.816+0000 7f4370eaf640 1 -- 172.21.15.28:0/3202815168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f436c10bd10 msgr2=0x7f436c1010a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.816+0000 7f4370eaf640 1 --2- 172.21.15.28:0/3202815168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f436c10bd10 0x7f436c1010a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.816+0000 7f4370eaf640 1 -- 172.21.15.28:0/3202815168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436c1a64b0 con 0x7f436c1032f0 2024-04-25T01:33:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.816+0000 7f4370eaf640 1 --2- 172.21.15.28:0/3202815168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f436c1032f0 0x7f436c100b60 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f435c00d950 tx=0x7f435c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.816+0000 7f43627fc640 1 -- 172.21.15.28:0/3202815168 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435c017030 con 0x7f436c1032f0 2024-04-25T01:33:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.816+0000 7f4373914640 1 -- 172.21.15.28:0/3202815168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f436c1a67a0 con 0x7f436c1032f0 2024-04-25T01:33:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.816+0000 7f43627fc640 1 -- 172.21.15.28:0/3202815168 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f435c004020 con 0x7f436c1032f0 2024-04-25T01:33:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.816+0000 7f43627fc640 1 -- 172.21.15.28:0/3202815168 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435c0047a0 con 0x7f436c1032f0 2024-04-25T01:33:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.816+0000 7f4373914640 1 -- 172.21.15.28:0/3202815168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f436c1a6d40 con 0x7f436c1032f0 2024-04-25T01:33:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.817+0000 7f4373914640 1 -- 172.21.15.28:0/3202815168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f436c04f500 con 0x7f436c1032f0 2024-04-25T01:33:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.818+0000 7f43627fc640 1 -- 172.21.15.28:0/3202815168 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f435c0041d0 con 0x7f436c1032f0 2024-04-25T01:33:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.819+0000 7f43627fc640 1 --2- 172.21.15.28:0/3202815168 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f43580a2e40 0x7f43580a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.819+0000 7f43716b0640 1 --2- 172.21.15.28:0/3202815168 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f43580a2e40 0x7f43580a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.819+0000 7f43627fc640 1 -- 172.21.15.28:0/3202815168 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f435c0cf270 con 0x7f436c1032f0 2024-04-25T01:33:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.819+0000 7f43716b0640 1 --2- 172.21.15.28:0/3202815168 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f43580a2e40 0x7f43580a5300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f43680058c0 tx=0x7f436800a2d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:06.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:06.823+0000 7f43627fc640 1 -- 172.21.15.28:0/3202815168 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f435c09b980 con 0x7f436c1032f0 2024-04-25T01:33:07.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.031+0000 7f4373914640 1 -- 172.21.15.28:0/3202815168 --> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f436c061b80 con 0x7f43580a2e40 2024-04-25T01:33:07.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.094+0000 7f43627fc640 1 -- 172.21.15.28:0/3202815168 <== mgr.15561 v2:172.21.15.28:6800/712762371 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f436c061b80 con 0x7f43580a2e40 2024-04-25T01:33:07.071 INFO:teuthology.orchestra.run.smithi028.stdout:Retention added to path / 2024-04-25T01:33:07.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.096+0000 7f4373914640 1 -- 172.21.15.28:0/3202815168 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f43580a2e40 msgr2=0x7f43580a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:07.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.096+0000 7f4373914640 1 --2- 172.21.15.28:0/3202815168 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f43580a2e40 0x7f43580a5300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f43680058c0 tx=0x7f436800a2d0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.096+0000 7f4373914640 1 -- 172.21.15.28:0/3202815168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f436c1032f0 msgr2=0x7f436c100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:07.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.096+0000 7f4373914640 1 --2- 172.21.15.28:0/3202815168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f436c1032f0 0x7f436c100b60 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f435c00d950 tx=0x7f435c00de10 comp rx=0 tx=0).stop 2024-04-25T01:33:07.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.096+0000 7f4373914640 1 -- 172.21.15.28:0/3202815168 shutdown_connections 2024-04-25T01:33:07.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.096+0000 7f4373914640 1 --2- 172.21.15.28:0/3202815168 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f43580a2e40 0x7f43580a5300 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.096+0000 7f4373914640 1 --2- 172.21.15.28:0/3202815168 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f436c10bd10 0x7f436c1010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.096+0000 7f4373914640 1 --2- 172.21.15.28:0/3202815168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f436c1032f0 0x7f436c100b60 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.096+0000 7f4373914640 1 --2- 172.21.15.28:0/3202815168 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f436c1029d0 0x7f436c100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.096+0000 7f4373914640 1 -- 172.21.15.28:0/3202815168 >> 172.21.15.28:0/3202815168 conn(0x7f436c0777c0 msgr2=0x7f436c0fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:07.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.096+0000 7f4373914640 1 -- 172.21.15.28:0/3202815168 shutdown_connections 2024-04-25T01:33:07.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.096+0000 7f4373914640 1 -- 172.21.15.28:0/3202815168 wait complete. 2024-04-25T01:33:07.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-25T01:33:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.211+0000 7f382db12640 1 -- 172.21.15.28:0/4266370545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3828101020 msgr2=0x7f382810d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.211+0000 7f382db12640 1 --2- 172.21.15.28:0/4266370545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3828101020 0x7f382810d530 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3820009cd0 tx=0x7f382002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.211+0000 7f382db12640 1 -- 172.21.15.28:0/4266370545 shutdown_connections 2024-04-25T01:33:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.211+0000 7f382db12640 1 --2- 172.21.15.28:0/4266370545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3828101020 0x7f382810d530 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.211+0000 7f382db12640 1 --2- 172.21.15.28:0/4266370545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3828100660 0x7f3828100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.211+0000 7f382db12640 1 --2- 172.21.15.28:0/4266370545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3828106670 0x7f3828106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.211+0000 7f382db12640 1 -- 172.21.15.28:0/4266370545 >> 172.21.15.28:0/4266370545 conn(0x7f38280769c0 msgr2=0x7f3828076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.211+0000 7f382db12640 1 -- 172.21.15.28:0/4266370545 shutdown_connections 2024-04-25T01:33:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.211+0000 7f382db12640 1 -- 172.21.15.28:0/4266370545 wait complete. 2024-04-25T01:33:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.211+0000 7f382db12640 1 Processor -- start 2024-04-25T01:33:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.211+0000 7f382db12640 1 -- start start 2024-04-25T01:33:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.212+0000 7f382db12640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3828100660 0x7f382819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.212+0000 7f382db12640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3828101020 0x7f382819a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.212+0000 7f382db12640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3828106670 0x7f382819e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.212+0000 7f382db12640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3828110a80 con 0x7f3828100660 2024-04-25T01:33:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.212+0000 7f382db12640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3828110900 con 0x7f3828101020 2024-04-25T01:33:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.212+0000 7f382db12640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f3828110c00 con 0x7f3828106670 2024-04-25T01:33:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.212+0000 7f38277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3828100660 0x7f382819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.212+0000 7f38277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3828100660 0x7f382819a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39272/0 (socket says 172.21.15.28:39272) 2024-04-25T01:33:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.212+0000 7f38277fe640 1 -- 172.21.15.28:0/2557353760 learned_addr learned my addr 172.21.15.28:0/2557353760 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.212+0000 7f3826ffd640 1 --2- 172.21.15.28:0/2557353760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3828101020 0x7f382819a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.212+0000 7f38277fe640 1 -- 172.21.15.28:0/2557353760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3828106670 msgr2=0x7f382819e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.212+0000 7f38277fe640 1 --2- 172.21.15.28:0/2557353760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3828106670 0x7f382819e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.212+0000 7f38277fe640 1 -- 172.21.15.28:0/2557353760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3828101020 msgr2=0x7f382819a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.212+0000 7f38277fe640 1 --2- 172.21.15.28:0/2557353760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3828101020 0x7f382819a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.212+0000 7f38277fe640 1 -- 172.21.15.28:0/2557353760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f382819f0b0 con 0x7f3828100660 2024-04-25T01:33:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.212+0000 7f38277fe640 1 --2- 172.21.15.28:0/2557353760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3828100660 0x7f382819a0c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f381800b4a0 tx=0x7f381800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.212+0000 7f3824ff9640 1 -- 172.21.15.28:0/2557353760 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3818015020 con 0x7f3828100660 2024-04-25T01:33:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.212+0000 7f382db12640 1 -- 172.21.15.28:0/2557353760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f382819f3a0 con 0x7f3828100660 2024-04-25T01:33:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.213+0000 7f3824ff9640 1 -- 172.21.15.28:0/2557353760 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3818011d10 con 0x7f3828100660 2024-04-25T01:33:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.213+0000 7f3824ff9640 1 -- 172.21.15.28:0/2557353760 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f381800f2e0 con 0x7f3828100660 2024-04-25T01:33:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.213+0000 7f382db12640 1 -- 172.21.15.28:0/2557353760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38281a6c30 con 0x7f3828100660 2024-04-25T01:33:07.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.213+0000 7f382db12640 1 -- 172.21.15.28:0/2557353760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f382804f500 con 0x7f3828100660 2024-04-25T01:33:07.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.215+0000 7f3824ff9640 1 -- 172.21.15.28:0/2557353760 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f3818018020 con 0x7f3828100660 2024-04-25T01:33:07.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.215+0000 7f3824ff9640 1 --2- 172.21.15.28:0/2557353760 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f38100a2e40 0x7f38100a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:07.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.215+0000 7f3826ffd640 1 --2- 172.21.15.28:0/2557353760 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f38100a2e40 0x7f38100a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:07.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.215+0000 7f3824ff9640 1 -- 172.21.15.28:0/2557353760 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f38180cf110 con 0x7f3828100660 2024-04-25T01:33:07.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.216+0000 7f3826ffd640 1 --2- 172.21.15.28:0/2557353760 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f38100a2e40 0x7f38100a5300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f38280772c0 tx=0x7f381c00a2d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:07.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.220+0000 7f3824ff9640 1 -- 172.21.15.28:0/2557353760 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f381801f020 con 0x7f3828100660 2024-04-25T01:33:07.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.424+0000 7f382db12640 1 -- 172.21.15.28:0/2557353760 --> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f3828102870 con 0x7f38100a2e40 2024-04-25T01:33:07.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.432+0000 7f3824ff9640 1 -- 172.21.15.28:0/2557353760 <== mgr.15561 v2:172.21.15.28:6800/712762371 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f3828102870 con 0x7f38100a2e40 2024-04-25T01:33:07.409 INFO:teuthology.orchestra.run.smithi028.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-25T00:00:00", "created": "2024-04-25T01:33:06", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-25T01:33:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.434+0000 7f382db12640 1 -- 172.21.15.28:0/2557353760 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f38100a2e40 msgr2=0x7f38100a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.434+0000 7f382db12640 1 --2- 172.21.15.28:0/2557353760 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f38100a2e40 0x7f38100a5300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f38280772c0 tx=0x7f381c00a2d0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.434+0000 7f382db12640 1 -- 172.21.15.28:0/2557353760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3828100660 msgr2=0x7f382819a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.434+0000 7f382db12640 1 --2- 172.21.15.28:0/2557353760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3828100660 0x7f382819a0c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f381800b4a0 tx=0x7f381800b8a0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.434+0000 7f382db12640 1 -- 172.21.15.28:0/2557353760 shutdown_connections 2024-04-25T01:33:07.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.434+0000 7f382db12640 1 --2- 172.21.15.28:0/2557353760 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f38100a2e40 0x7f38100a5300 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.434+0000 7f382db12640 1 --2- 172.21.15.28:0/2557353760 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3828106670 0x7f382819e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.434+0000 7f382db12640 1 --2- 172.21.15.28:0/2557353760 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3828101020 0x7f382819a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.434+0000 7f382db12640 1 --2- 172.21.15.28:0/2557353760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3828100660 0x7f382819a0c0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.434+0000 7f382db12640 1 -- 172.21.15.28:0/2557353760 >> 172.21.15.28:0/2557353760 conn(0x7f38280769c0 msgr2=0x7f3828102160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:07.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.434+0000 7f382db12640 1 -- 172.21.15.28:0/2557353760 shutdown_connections 2024-04-25T01:33:07.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.434+0000 7f382db12640 1 -- 172.21.15.28:0/2557353760 wait complete. 2024-04-25T01:33:07.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:07 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/4270071796' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:33:07.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:07 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:33:07.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:07 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:07.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:07 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:33:07.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:07 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:07.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-25T01:33:07.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:07 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/4270071796' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:33:07.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:07 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:33:07.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:07 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:07.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:07 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:33:07.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:07 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:07.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:07 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/4270071796' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:33:07.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:07 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:33:07.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:07 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:07.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:07 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:33:07.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:07 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:07.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.548+0000 7fe587b9b640 1 -- 172.21.15.28:0/3662551340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe580102850 msgr2=0x7fe580102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.548+0000 7fe587b9b640 1 --2- 172.21.15.28:0/3662551340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe580102850 0x7fe580102c30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe57c009900 tx=0x7fe57c02a480 comp rx=0 tx=0).stop 2024-04-25T01:33:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.548+0000 7fe587b9b640 1 -- 172.21.15.28:0/3662551340 shutdown_connections 2024-04-25T01:33:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.548+0000 7fe587b9b640 1 --2- 172.21.15.28:0/3662551340 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe58010bd70 0x7fe58010e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.548+0000 7fe587b9b640 1 --2- 172.21.15.28:0/3662551340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe580103170 0x7fe58010b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.548+0000 7fe587b9b640 1 --2- 172.21.15.28:0/3662551340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe580102850 0x7fe580102c30 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.548+0000 7fe587b9b640 1 -- 172.21.15.28:0/3662551340 >> 172.21.15.28:0/3662551340 conn(0x7fe5800777c0 msgr2=0x7fe580077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.549+0000 7fe587b9b640 1 -- 172.21.15.28:0/3662551340 shutdown_connections 2024-04-25T01:33:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.549+0000 7fe587b9b640 1 -- 172.21.15.28:0/3662551340 wait complete. 2024-04-25T01:33:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.549+0000 7fe587b9b640 1 Processor -- start 2024-04-25T01:33:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.549+0000 7fe587b9b640 1 -- start start 2024-04-25T01:33:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.549+0000 7fe587b9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe580102850 0x7fe580100680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.549+0000 7fe587b9b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe580103170 0x7fe580100bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.549+0000 7fe587b9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58010bd70 0x7fe580101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.549+0000 7fe587b9b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe580111b10 con 0x7fe58010bd70 2024-04-25T01:33:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.549+0000 7fe587b9b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe580111990 con 0x7fe580102850 2024-04-25T01:33:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.549+0000 7fe587b9b640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe580111c90 con 0x7fe580103170 2024-04-25T01:33:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.550+0000 7fe586138640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58010bd70 0x7fe580101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.550+0000 7fe586138640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58010bd70 0x7fe580101100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39290/0 (socket says 172.21.15.28:39290) 2024-04-25T01:33:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.550+0000 7fe586138640 1 -- 172.21.15.28:0/2157989451 learned_addr learned my addr 172.21.15.28:0/2157989451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.550+0000 7fe585937640 1 --2- 172.21.15.28:0/2157989451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe580102850 0x7fe580100680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.550+0000 7fe586138640 1 -- 172.21.15.28:0/2157989451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe580103170 msgr2=0x7fe580100bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.550+0000 7fe586138640 1 --2- 172.21.15.28:0/2157989451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe580103170 0x7fe580100bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.550+0000 7fe586138640 1 -- 172.21.15.28:0/2157989451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe580102850 msgr2=0x7fe580100680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.550+0000 7fe586138640 1 --2- 172.21.15.28:0/2157989451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe580102850 0x7fe580100680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.550+0000 7fe586138640 1 -- 172.21.15.28:0/2157989451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5801a6550 con 0x7fe58010bd70 2024-04-25T01:33:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.550+0000 7fe586138640 1 --2- 172.21.15.28:0/2157989451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58010bd70 0x7fe580101100 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe57800eaa0 tx=0x7fe57800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.550+0000 7fe576ffd640 1 -- 172.21.15.28:0/2157989451 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57800cdb0 con 0x7fe58010bd70 2024-04-25T01:33:07.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.550+0000 7fe587b9b640 1 -- 172.21.15.28:0/2157989451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5801a6840 con 0x7fe58010bd70 2024-04-25T01:33:07.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.550+0000 7fe576ffd640 1 -- 172.21.15.28:0/2157989451 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe578005020 con 0x7fe58010bd70 2024-04-25T01:33:07.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.550+0000 7fe576ffd640 1 -- 172.21.15.28:0/2157989451 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5780116b0 con 0x7fe58010bd70 2024-04-25T01:33:07.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.550+0000 7fe587b9b640 1 -- 172.21.15.28:0/2157989451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5801a6d90 con 0x7fe58010bd70 2024-04-25T01:33:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.551+0000 7fe587b9b640 1 -- 172.21.15.28:0/2157989451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe58004f500 con 0x7fe58010bd70 2024-04-25T01:33:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.552+0000 7fe576ffd640 1 -- 172.21.15.28:0/2157989451 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7fe578005560 con 0x7fe58010bd70 2024-04-25T01:33:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.553+0000 7fe576ffd640 1 --2- 172.21.15.28:0/2157989451 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe56c0a2e40 0x7fe56c0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.553+0000 7fe585937640 1 --2- 172.21.15.28:0/2157989451 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe56c0a2e40 0x7fe56c0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.553+0000 7fe576ffd640 1 -- 172.21.15.28:0/2157989451 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fe5780cfb70 con 0x7fe58010bd70 2024-04-25T01:33:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.553+0000 7fe585937640 1 --2- 172.21.15.28:0/2157989451 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe56c0a2e40 0x7fe56c0a5300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe57c009230 tx=0x7fe57c0058d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:07.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.557+0000 7fe576ffd640 1 -- 172.21.15.28:0/2157989451 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe578016030 con 0x7fe58010bd70 2024-04-25T01:33:07.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.764+0000 7fe587b9b640 1 -- 172.21.15.28:0/2157989451 --> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fe580106f60 con 0x7fe56c0a2e40 2024-04-25T01:33:07.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.770+0000 7fe576ffd640 1 -- 172.21.15.28:0/2157989451 <== mgr.15561 v2:172.21.15.28:6800/712762371 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fe57c031630 con 0x7fe56c0a2e40 2024-04-25T01:33:07.747 INFO:teuthology.orchestra.run.smithi028.stdout:/ 1m 6m3h 2024-04-25T01:33:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.772+0000 7fe587b9b640 1 -- 172.21.15.28:0/2157989451 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe56c0a2e40 msgr2=0x7fe56c0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.772+0000 7fe587b9b640 1 --2- 172.21.15.28:0/2157989451 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe56c0a2e40 0x7fe56c0a5300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe57c009230 tx=0x7fe57c0058d0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.772+0000 7fe587b9b640 1 -- 172.21.15.28:0/2157989451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58010bd70 msgr2=0x7fe580101100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.772+0000 7fe587b9b640 1 --2- 172.21.15.28:0/2157989451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58010bd70 0x7fe580101100 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe57800eaa0 tx=0x7fe57800ef60 comp rx=0 tx=0).stop 2024-04-25T01:33:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.772+0000 7fe587b9b640 1 -- 172.21.15.28:0/2157989451 shutdown_connections 2024-04-25T01:33:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.772+0000 7fe587b9b640 1 --2- 172.21.15.28:0/2157989451 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe56c0a2e40 0x7fe56c0a5300 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.772+0000 7fe587b9b640 1 --2- 172.21.15.28:0/2157989451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58010bd70 0x7fe580101100 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.772+0000 7fe587b9b640 1 --2- 172.21.15.28:0/2157989451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe580103170 0x7fe580100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.772+0000 7fe587b9b640 1 --2- 172.21.15.28:0/2157989451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe580102850 0x7fe580100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:07.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.772+0000 7fe587b9b640 1 -- 172.21.15.28:0/2157989451 >> 172.21.15.28:0/2157989451 conn(0x7fe5800777c0 msgr2=0x7fe5800fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:07.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.772+0000 7fe587b9b640 1 -- 172.21.15.28:0/2157989451 shutdown_connections 2024-04-25T01:33:07.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:07.772+0000 7fe587b9b640 1 -- 172.21.15.28:0/2157989451 wait complete. 2024-04-25T01:33:07.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-25T01:33:07.786 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-25T01:33:07.797 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T01:33:07.845 INFO:tasks.workunit:Pulling workunits from ref d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:33:07.845 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T01:33:07.845 INFO:tasks.workunit:timeout=3h 2024-04-25T01:33:07.845 INFO:tasks.workunit:cleanup=True 2024-04-25T01:33:07.846 DEBUG:teuthology.orchestra.run.smithi028:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-25T01:33:07.861 INFO:teuthology.orchestra.run.smithi028.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-25T01:33:07.861 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-25T01:33:07.861 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-25T01:33:07.861 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:33:07.861 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-25T01:33:07.861 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-25 01:31:02.549539353 +0000 2024-04-25T01:33:07.862 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-25 01:31:02.549539353 +0000 2024-04-25T01:33:07.862 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-25 01:32:58.106517517 +0000 2024-04-25T01:33:07.862 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-25 01:31:02.549539353 +0000 2024-04-25T01:33:07.862 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-25T01:33:07.862 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-25T01:33:07.934 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:33:07.991 INFO:tasks.workunit.client.0.smithi028.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-25T01:33:08.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:08 smithi130 ceph-mon[32702]: from='client.15762 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:33:08.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:08 smithi130 ceph-mon[32702]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:33:08.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:08 smithi130 ceph-mon[32702]: from='client.15801 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:33:08.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:08 smithi032 ceph-mon[32461]: from='client.15762 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:33:08.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:08 smithi032 ceph-mon[32461]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:33:08.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:08 smithi032 ceph-mon[32461]: from='client.15801 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:33:08.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:08 smithi028 ceph-mon[29390]: from='client.15762 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:33:08.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:08 smithi028 ceph-mon[29390]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:33:08.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:08 smithi028 ceph-mon[29390]: from='client.15801 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:33:09.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:09 smithi130 ceph-mon[32702]: from='client.15807 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:33:09.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:09 smithi130 ceph-mon[32702]: from='client.15813 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:33:09.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:09 smithi032 ceph-mon[32461]: from='client.15807 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:33:09.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:09 smithi032 ceph-mon[32461]: from='client.15813 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:33:09.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:09 smithi028 ceph-mon[29390]: from='client.15807 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:33:09.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:09 smithi028 ceph-mon[29390]: from='client.15813 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:33:10.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:10 smithi130 ceph-mon[32702]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 93 KiB/s wr, 41 op/s 2024-04-25T01:33:10.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:10 smithi032 ceph-mon[32461]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 93 KiB/s wr, 41 op/s 2024-04-25T01:33:10.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:10 smithi028 ceph-mon[29390]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 93 KiB/s wr, 41 op/s 2024-04-25T01:33:12.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:12 smithi130 ceph-mon[32702]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 73 KiB/s wr, 32 op/s 2024-04-25T01:33:12.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:12 smithi032 ceph-mon[32461]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 73 KiB/s wr, 32 op/s 2024-04-25T01:33:12.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:12 smithi028 ceph-mon[29390]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 73 KiB/s wr, 32 op/s 2024-04-25T01:33:14.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:14 smithi130 ceph-mon[32702]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-04-25T01:33:14.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:14 smithi032 ceph-mon[32461]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-04-25T01:33:14.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:14 smithi028 ceph-mon[29390]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-04-25T01:33:16.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:16 smithi130 ceph-mon[32702]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-25T01:33:16.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:16 smithi032 ceph-mon[32461]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-25T01:33:16.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:16 smithi028 ceph-mon[29390]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-04-25T01:33:18.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:18 smithi130 ceph-mon[32702]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-04-25T01:33:18.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:18 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:33:18.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:18 smithi032 ceph-mon[32461]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-04-25T01:33:18.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:18 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:33:18.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:18 smithi028 ceph-mon[29390]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-04-25T01:33:18.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:18 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:33:20.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:20 smithi130 ceph-mon[32702]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-04-25T01:33:20.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:20 smithi032 ceph-mon[32461]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-04-25T01:33:20.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:20 smithi028 ceph-mon[29390]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 55 op/s 2024-04-25T01:33:22.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:22 smithi130 ceph-mon[32702]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 23 KiB/s wr, 31 op/s 2024-04-25T01:33:22.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:22 smithi032 ceph-mon[32461]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 23 KiB/s wr, 31 op/s 2024-04-25T01:33:22.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:22 smithi028 ceph-mon[29390]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 23 KiB/s wr, 31 op/s 2024-04-25T01:33:24.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:24 smithi130 ceph-mon[32702]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 35 op/s 2024-04-25T01:33:24.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:24 smithi032 ceph-mon[32461]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 35 op/s 2024-04-25T01:33:24.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:24 smithi028 ceph-mon[29390]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 KiB/s wr, 35 op/s 2024-04-25T01:33:26.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:26 smithi130 ceph-mon[32702]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:33:26.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:26 smithi032 ceph-mon[32461]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:33:26.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:26 smithi028 ceph-mon[29390]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:33:28.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:28 smithi130 ceph-mon[32702]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:33:28.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:28 smithi032 ceph-mon[32461]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:33:28.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:28 smithi028 ceph-mon[29390]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:33:30.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:30 smithi130 ceph-mon[32702]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:33:30.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:30 smithi032 ceph-mon[32461]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:33:30.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:30 smithi028 ceph-mon[29390]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-04-25T01:33:32.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:32 smithi032 ceph-mon[32461]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:33:32.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:32 smithi028 ceph-mon[29390]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:33:32.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:32 smithi130 ceph-mon[32702]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:33:33.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:33 smithi032 ceph-mon[32461]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:33:33.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:33 smithi028 ceph-mon[29390]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:33:33.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:33 smithi130 ceph-mon[32702]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:33:35.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:35 smithi130 ceph-mon[32702]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:33:35.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:35 smithi032 ceph-mon[32461]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:33:36.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:35 smithi028 ceph-mon[29390]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:33:36.538 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:33:36.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:33:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.681+0000 7ff4d7b9e640 1 -- 172.21.15.28:0/1812692429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d01020d0 msgr2=0x7ff4d010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.681+0000 7ff4d7b9e640 1 --2- 172.21.15.28:0/1812692429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d01020d0 0x7ff4d010e5e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8009cd0 tx=0x7ff4c802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.681+0000 7ff4d7b9e640 1 -- 172.21.15.28:0/1812692429 shutdown_connections 2024-04-25T01:33:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.681+0000 7ff4d7b9e640 1 --2- 172.21.15.28:0/1812692429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d01020d0 0x7ff4d010e5e0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.681+0000 7ff4d7b9e640 1 --2- 172.21.15.28:0/1812692429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4d0101710 0x7ff4d0101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.681+0000 7ff4d7b9e640 1 --2- 172.21.15.28:0/1812692429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4d0107720 0x7ff4d0107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.681+0000 7ff4d7b9e640 1 -- 172.21.15.28:0/1812692429 >> 172.21.15.28:0/1812692429 conn(0x7ff4d00fd180 msgr2=0x7ff4d00ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.681+0000 7ff4d7b9e640 1 -- 172.21.15.28:0/1812692429 shutdown_connections 2024-04-25T01:33:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.681+0000 7ff4d7b9e640 1 -- 172.21.15.28:0/1812692429 wait complete. 2024-04-25T01:33:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.682+0000 7ff4d7b9e640 1 Processor -- start 2024-04-25T01:33:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.682+0000 7ff4d7b9e640 1 -- start start 2024-04-25T01:33:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.682+0000 7ff4d7b9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0101710 0x7ff4d0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.682+0000 7ff4d7b9e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4d01020d0 0x7ff4d0198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.682+0000 7ff4d7b9e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4d0107720 0x7ff4d019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.682+0000 7ff4d7b9e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d0111b30 con 0x7ff4d0101710 2024-04-25T01:33:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.682+0000 7ff4d7b9e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d01119b0 con 0x7ff4d01020d0 2024-04-25T01:33:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.682+0000 7ff4d7b9e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d0111cb0 con 0x7ff4d0107720 2024-04-25T01:33:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.682+0000 7ff4d593a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0101710 0x7ff4d0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.682+0000 7ff4d593a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0101710 0x7ff4d0197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39976/0 (socket says 172.21.15.28:39976) 2024-04-25T01:33:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.682+0000 7ff4d593a640 1 -- 172.21.15.28:0/2885856709 learned_addr learned my addr 172.21.15.28:0/2885856709 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.682+0000 7ff4d5139640 1 --2- 172.21.15.28:0/2885856709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4d01020d0 0x7ff4d0198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.682+0000 7ff4d613b640 1 --2- 172.21.15.28:0/2885856709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4d0107720 0x7ff4d019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.682+0000 7ff4d593a640 1 -- 172.21.15.28:0/2885856709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4d0107720 msgr2=0x7ff4d019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.682+0000 7ff4d593a640 1 --2- 172.21.15.28:0/2885856709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4d0107720 0x7ff4d019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.682+0000 7ff4d593a640 1 -- 172.21.15.28:0/2885856709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4d01020d0 msgr2=0x7ff4d0198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.682+0000 7ff4d593a640 1 --2- 172.21.15.28:0/2885856709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4d01020d0 0x7ff4d0198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.683+0000 7ff4d593a640 1 -- 172.21.15.28:0/2885856709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d019cfc0 con 0x7ff4d0101710 2024-04-25T01:33:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.683+0000 7ff4d613b640 1 --2- 172.21.15.28:0/2885856709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4d0107720 0x7ff4d019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.683+0000 7ff4d593a640 1 --2- 172.21.15.28:0/2885856709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0101710 0x7ff4d0197fd0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc00b4a0 tx=0x7ff4cc00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.683+0000 7ff4c6ffd640 1 -- 172.21.15.28:0/2885856709 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc015020 con 0x7ff4d0101710 2024-04-25T01:33:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.683+0000 7ff4d7b9e640 1 -- 172.21.15.28:0/2885856709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4d019d2b0 con 0x7ff4d0101710 2024-04-25T01:33:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.683+0000 7ff4c6ffd640 1 -- 172.21.15.28:0/2885856709 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4cc011d10 con 0x7ff4d0101710 2024-04-25T01:33:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.683+0000 7ff4c6ffd640 1 -- 172.21.15.28:0/2885856709 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc00f2e0 con 0x7ff4d0101710 2024-04-25T01:33:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.683+0000 7ff4d7b9e640 1 -- 172.21.15.28:0/2885856709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4d01a4b40 con 0x7ff4d0101710 2024-04-25T01:33:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.685+0000 7ff4c6ffd640 1 -- 172.21.15.28:0/2885856709 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7ff4cc018020 con 0x7ff4d0101710 2024-04-25T01:33:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.686+0000 7ff4da987740 1 Processor -- start 2024-04-25T01:33:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.686+0000 7ff4da987740 1 -- start start 2024-04-25T01:33:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.686+0000 7ff4da987740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 0x55cde2b86a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.686+0000 7ff4da987740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b86f50 0x55cde2b910e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.686+0000 7ff4da987740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b91620 0x55cde2b93a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.687+0000 7ff4da987740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cde2aeb6b0 con 0x55cde2b86630 2024-04-25T01:33:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.687+0000 7ff4da987740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cde2a93880 con 0x55cde2b91620 2024-04-25T01:33:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.687+0000 7ff4da987740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55cde2ae7670 con 0x55cde2b86f50 2024-04-25T01:33:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.687+0000 7ff4c6ffd640 1 --2- 172.21.15.28:0/2885856709 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4bc0a2d90 0x7ff4bc0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.687+0000 7ff4d613b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 0x55cde2b86a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.687+0000 7ff4d613b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 0x55cde2b86a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39988/0 (socket says 172.21.15.28:39988) 2024-04-25T01:33:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.687+0000 7ff4d613b640 1 -- 172.21.15.28:0/1712071851 learned_addr learned my addr 172.21.15.28:0/1712071851 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.687+0000 7ff4d593a640 1 --2- 172.21.15.28:0/1712071851 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b86f50 0x55cde2b910e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.687+0000 7ff4d613b640 1 --2- 172.21.15.28:0/2885856709 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4bc0a2d90 0x7ff4bc0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.687+0000 7ff4d5139640 1 --2- 172.21.15.28:0/1712071851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b91620 0x55cde2b93a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.687+0000 7ff4c6ffd640 1 -- 172.21.15.28:0/2885856709 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff4cc0cf1c0 con 0x7ff4d0101710 2024-04-25T01:33:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.687+0000 7ff4d613b640 1 -- 172.21.15.28:0/1712071851 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b86f50 msgr2=0x55cde2b910e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.687+0000 7ff4d613b640 1 --2- 172.21.15.28:0/1712071851 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b86f50 0x55cde2b910e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.687+0000 7ff4d613b640 1 -- 172.21.15.28:0/1712071851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b91620 msgr2=0x55cde2b93a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.687+0000 7ff4d613b640 1 --2- 172.21.15.28:0/1712071851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b91620 0x55cde2b93a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.687+0000 7ff4d613b640 1 -- 172.21.15.28:0/1712071851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cde28c97d0 con 0x55cde2b86630 2024-04-25T01:33:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.687+0000 7ff4d613b640 1 --2- 172.21.15.28:0/1712071851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 0x55cde2b86a10 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8031eb0 tx=0x7ff4c8003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c9da0bbced1ff88 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.688+0000 7ff4d613b640 1 --2- 172.21.15.28:0/2885856709 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4bc0a2d90 0x7ff4bc0a5250 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff4c80039e0 tx=0x7ff4c8047000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.688+0000 7ff4bbfff640 1 -- 172.21.15.28:0/1712071851 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8003e50 con 0x55cde2b86630 2024-04-25T01:33:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.688+0000 7ff4bbfff640 1 -- 172.21.15.28:0/1712071851 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4c8036040 con 0x55cde2b86630 2024-04-25T01:33:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.688+0000 7ff4bbfff640 1 -- 172.21.15.28:0/1712071851 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8049630 con 0x55cde2b86630 2024-04-25T01:33:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.688+0000 7ff4da987740 1 -- 172.21.15.28:0/1712071851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 msgr2=0x55cde2b86a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.688+0000 7ff4da987740 1 --2- 172.21.15.28:0/1712071851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 0x55cde2b86a10 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8031eb0 tx=0x7ff4c8003970 comp rx=0 tx=0).stop 2024-04-25T01:33:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.688+0000 7ff4da987740 1 -- 172.21.15.28:0/1712071851 shutdown_connections 2024-04-25T01:33:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.688+0000 7ff4da987740 1 --2- 172.21.15.28:0/1712071851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b91620 0x55cde2b93a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.688+0000 7ff4da987740 1 --2- 172.21.15.28:0/1712071851 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b86f50 0x55cde2b910e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.688+0000 7ff4da987740 1 --2- 172.21.15.28:0/1712071851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 0x55cde2b86a10 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.688+0000 7ff4da987740 1 -- 172.21.15.28:0/1712071851 >> 172.21.15.28:0/1712071851 conn(0x55cde2b82950 msgr2=0x55cde2b82d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.689+0000 7ff4da987740 1 -- 172.21.15.28:0/1712071851 shutdown_connections 2024-04-25T01:33:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.689+0000 7ff4da987740 1 -- 172.21.15.28:0/1712071851 wait complete. 2024-04-25T01:33:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.689+0000 7ff4da987740 1 -- start start 2024-04-25T01:33:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.689+0000 7ff4da987740 1 Processor -- start 2024-04-25T01:33:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.689+0000 7ff4da987740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b86630 0x55cde2b8e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.689+0000 7ff4da987740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b86f50 0x55cde2b8eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.689+0000 7ff4da987740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b91620 0x55cde2b8f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.689+0000 7ff4da987740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cde2b943b0 con 0x55cde2b91620 2024-04-25T01:33:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.689+0000 7ff4da987740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cde2aa6f50 con 0x55cde2b86630 2024-04-25T01:33:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.689+0000 7ff4da987740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55cde2b94530 con 0x55cde2b86f50 2024-04-25T01:33:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.689+0000 7ff4da987740 10 client.0 _ll_register_callbacks cb 0x55cde2ae2000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.689+0000 7ff4d613b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b91620 0x55cde2b8f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.689+0000 7ff4d613b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b91620 0x55cde2b8f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39998/0 (socket says 172.21.15.28:39998) 2024-04-25T01:33:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.690+0000 7ff4d613b640 1 -- 172.21.15.28:0/2757891656 learned_addr learned my addr 172.21.15.28:0/2757891656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.690+0000 7ff4d593a640 1 --2- 172.21.15.28:0/2757891656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b86630 0x55cde2b8e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.690+0000 7ff4d5139640 1 --2- 172.21.15.28:0/2757891656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b86f50 0x55cde2b8eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.690+0000 7ff4d613b640 1 -- 172.21.15.28:0/2757891656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b86f50 msgr2=0x55cde2b8eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.690+0000 7ff4d613b640 1 --2- 172.21.15.28:0/2757891656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b86f50 0x55cde2b8eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.690+0000 7ff4d613b640 1 -- 172.21.15.28:0/2757891656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b86630 msgr2=0x55cde2b8e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.690+0000 7ff4d613b640 1 --2- 172.21.15.28:0/2757891656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b86630 0x55cde2b8e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.690+0000 7ff4d613b640 1 -- 172.21.15.28:0/2757891656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cde27ff3f0 con 0x55cde2b91620 2024-04-25T01:33:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.690+0000 7ff4d613b640 1 --2- 172.21.15.28:0/2757891656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b91620 0x55cde2b8f080 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff4c802a920 tx=0x7ff4c8009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1052a79b03e2c92 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.690+0000 7ff4b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.690+0000 7ff4b97fa640 1 -- 172.21.15.28:0/2757891656 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c80307a0 con 0x55cde2b91620 2024-04-25T01:33:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.690+0000 7ff4b97fa640 1 -- 172.21.15.28:0/2757891656 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff4c8030940 con 0x55cde2b91620 2024-04-25T01:33:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.690+0000 7ff4da987740 1 -- 172.21.15.28:0/2757891656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4c8036040 con 0x55cde2b91620 2024-04-25T01:33:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.690+0000 7ff4b97fa640 1 -- 172.21.15.28:0/2757891656 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff4c803e430 con 0x55cde2b91620 2024-04-25T01:33:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.690+0000 7ff4da987740 10 client.15825 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.690+0000 7ff4da987740 1 -- 172.21.15.28:0/2757891656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cde27bf0f0 con 0x55cde2b91620 2024-04-25T01:33:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.691+0000 7ff4b97fa640 1 -- 172.21.15.28:0/2757891656 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4c8036040 con 0x55cde2b91620 2024-04-25T01:33:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.691+0000 7ff4b97fa640 1 -- 172.21.15.28:0/2757891656 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff4c8048070 con 0x55cde2b91620 2024-04-25T01:33:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.691+0000 7ff4da987740 10 client.15825 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.691+0000 7ff4da987740 10 client.15825 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.691+0000 7ff4da987740 20 client.15825 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.691+0000 7ff4da987740 1 --2- 172.21.15.28:0/2757891656 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cde2ba1780 0x55cde2ba3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.691+0000 7ff4da987740 4 client.15825 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.691+0000 7ff4da987740 1 -- 172.21.15.28:0/2757891656 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cde2af8bf0 con 0x55cde2ba1780 2024-04-25T01:33:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.691+0000 7ff4d593a640 1 --2- 172.21.15.28:0/2757891656 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cde2ba1780 0x55cde2ba3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.692+0000 7ff4d593a640 1 --2- 172.21.15.28:0/2757891656 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cde2ba1780 0x55cde2ba3b70 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21e6f89afdfde12 server_cookie=bba14478c2aa83a1 in_seq=0 out_seq=0 2024-04-25T01:33:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.692+0000 7ff4b97fa640 10 client.15825 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.692+0000 7ff4b97fa640 1 -- 172.21.15.28:0/2757891656 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cde2af8bf0 con 0x55cde2ba1780 2024-04-25T01:33:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.692+0000 7ff4b97fa640 10 client.15825 handle_command_reply: tid=0 2024-04-25T01:33:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.692+0000 7ff4da987740 1 client.15825 shutdown 2024-04-25T01:33:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.692+0000 7ff4da987740 1 -- 172.21.15.28:0/2757891656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b91620 msgr2=0x55cde2b8f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.692+0000 7ff4da987740 1 --2- 172.21.15.28:0/2757891656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b91620 0x55cde2b8f080 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff4c802a920 tx=0x7ff4c8009c20 comp rx=0 tx=0).stop 2024-04-25T01:33:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.693+0000 7ff4da987740 1 -- 172.21.15.28:0/2757891656 shutdown_connections 2024-04-25T01:33:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.693+0000 7ff4da987740 1 --2- 172.21.15.28:0/2757891656 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cde2ba1780 0x55cde2ba3b70 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.693+0000 7ff4da987740 1 --2- 172.21.15.28:0/2757891656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b91620 0x55cde2b8f080 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.693+0000 7ff4da987740 1 --2- 172.21.15.28:0/2757891656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b86f50 0x55cde2b8eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.693+0000 7ff4b97fa640 0 client.15825 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.693+0000 7ff4da987740 1 --2- 172.21.15.28:0/2757891656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b86630 0x55cde2b8e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.693+0000 7ff4da987740 1 -- 172.21.15.28:0/2757891656 >> 172.21.15.28:0/2757891656 conn(0x55cde2b82950 msgr2=0x55cde2b88750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.693+0000 7ff4da987740 1 -- 172.21.15.28:0/2757891656 shutdown_connections 2024-04-25T01:33:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.693+0000 7ff4da987740 1 -- 172.21.15.28:0/2757891656 wait complete. 2024-04-25T01:33:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.693+0000 7ff4da987740 20 client.15825 trim_cache size 0 max 16384 2024-04-25T01:33:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.711+0000 7ff4da987740 1 Processor -- start 2024-04-25T01:33:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.711+0000 7ff4da987740 1 -- start start 2024-04-25T01:33:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.711+0000 7ff4da987740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b82950 0x55cde2b9d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.711+0000 7ff4da987740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 0x55cde2b88d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.711+0000 7ff4da987740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b86f50 0x55cde2ba9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.711+0000 7ff4da987740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d01aeaa0 con 0x55cde2b86630 2024-04-25T01:33:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.711+0000 7ff4da987740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d01b05e0 con 0x55cde2b86f50 2024-04-25T01:33:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.711+0000 7ff4da987740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d01af170 con 0x55cde2b82950 2024-04-25T01:33:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.711+0000 7ff4d5139640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 0x55cde2b88d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.711+0000 7ff4d5139640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 0x55cde2b88d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40012/0 (socket says 172.21.15.28:40012) 2024-04-25T01:33:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.711+0000 7ff4d5139640 1 -- 172.21.15.28:0/2193947663 learned_addr learned my addr 172.21.15.28:0/2193947663 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.711+0000 7ff4d613b640 1 --2- 172.21.15.28:0/2193947663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b86f50 0x55cde2ba9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.711+0000 7ff4d593a640 1 --2- 172.21.15.28:0/2193947663 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b82950 0x55cde2b9d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.711+0000 7ff4d5139640 1 -- 172.21.15.28:0/2193947663 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b82950 msgr2=0x55cde2b9d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.711+0000 7ff4d5139640 1 --2- 172.21.15.28:0/2193947663 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b82950 0x55cde2b9d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.711+0000 7ff4d5139640 1 -- 172.21.15.28:0/2193947663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b86f50 msgr2=0x55cde2ba9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.711+0000 7ff4d5139640 1 --2- 172.21.15.28:0/2193947663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b86f50 0x55cde2ba9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.711+0000 7ff4d5139640 1 -- 172.21.15.28:0/2193947663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cde2b878c0 con 0x55cde2b86630 2024-04-25T01:33:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.712+0000 7ff4d5139640 1 --2- 172.21.15.28:0/2193947663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 0x55cde2b88d90 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000acb0 tx=0x7ff4c0010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5745ce0ff2cc35b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.712+0000 7ff4bb7fe640 1 -- 172.21.15.28:0/2193947663 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0020070 con 0x55cde2b86630 2024-04-25T01:33:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.712+0000 7ff4bb7fe640 1 -- 172.21.15.28:0/2193947663 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4c001bae0 con 0x55cde2b86630 2024-04-25T01:33:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.712+0000 7ff4bb7fe640 1 -- 172.21.15.28:0/2193947663 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c000fd60 con 0x55cde2b86630 2024-04-25T01:33:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.712+0000 7ff4da987740 1 -- 172.21.15.28:0/2193947663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 msgr2=0x55cde2b88d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.712+0000 7ff4da987740 1 --2- 172.21.15.28:0/2193947663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 0x55cde2b88d90 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000acb0 tx=0x7ff4c0010800 comp rx=0 tx=0).stop 2024-04-25T01:33:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.713+0000 7ff4da987740 1 -- 172.21.15.28:0/2193947663 shutdown_connections 2024-04-25T01:33:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.713+0000 7ff4da987740 1 --2- 172.21.15.28:0/2193947663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b86f50 0x55cde2ba9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.713+0000 7ff4da987740 1 --2- 172.21.15.28:0/2193947663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 0x55cde2b88d90 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.713+0000 7ff4da987740 1 --2- 172.21.15.28:0/2193947663 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b82950 0x55cde2b9d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.713+0000 7ff4da987740 1 -- 172.21.15.28:0/2193947663 >> 172.21.15.28:0/2193947663 conn(0x55cde2b8d240 msgr2=0x55cde2b81af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.713+0000 7ff4da987740 1 -- 172.21.15.28:0/2193947663 shutdown_connections 2024-04-25T01:33:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.713+0000 7ff4da987740 1 -- 172.21.15.28:0/2193947663 wait complete. 2024-04-25T01:33:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.713+0000 7ff4da987740 1 -- start start 2024-04-25T01:33:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.713+0000 7ff4da987740 1 Processor -- start 2024-04-25T01:33:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.713+0000 7ff4da987740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b82950 0x55cde2b8b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.713+0000 7ff4da987740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 0x55cde2b8baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.713+0000 7ff4da987740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b86f50 0x55cde2c069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.713+0000 7ff4da987740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d01aff10 con 0x55cde2b86630 2024-04-25T01:33:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.713+0000 7ff4da987740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d01af840 con 0x55cde2b82950 2024-04-25T01:33:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.713+0000 7ff4da987740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d01aca80 con 0x55cde2b86f50 2024-04-25T01:33:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.713+0000 7ff4da987740 10 client.0 _ll_register_callbacks cb 0x7ff4c000a630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.714+0000 7ff4d5139640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 0x55cde2b8baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.714+0000 7ff4d5139640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 0x55cde2b8baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40028/0 (socket says 172.21.15.28:40028) 2024-04-25T01:33:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.714+0000 7ff4d5139640 1 -- 172.21.15.28:0/2351139190 learned_addr learned my addr 172.21.15.28:0/2351139190 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.714+0000 7ff4d593a640 1 --2- 172.21.15.28:0/2351139190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b82950 0x55cde2b8b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.714+0000 7ff4d613b640 1 --2- 172.21.15.28:0/2351139190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b86f50 0x55cde2c069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.714+0000 7ff4d5139640 1 -- 172.21.15.28:0/2351139190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b86f50 msgr2=0x55cde2c069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.714+0000 7ff4d5139640 1 --2- 172.21.15.28:0/2351139190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b86f50 0x55cde2c069e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.714+0000 7ff4d5139640 1 -- 172.21.15.28:0/2351139190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b82950 msgr2=0x55cde2b8b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.714+0000 7ff4d5139640 1 --2- 172.21.15.28:0/2351139190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b82950 0x55cde2b8b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.714+0000 7ff4d5139640 1 -- 172.21.15.28:0/2351139190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cde2c06f20 con 0x55cde2b86630 2024-04-25T01:33:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.714+0000 7ff4d5139640 1 --2- 172.21.15.28:0/2351139190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 0x55cde2b8baa0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0010d00 tx=0x7ff4c000aa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c016f4574832a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.714+0000 7ff4b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.714+0000 7ff4b97fa640 1 -- 172.21.15.28:0/2351139190 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0029070 con 0x55cde2b86630 2024-04-25T01:33:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.714+0000 7ff4b97fa640 1 -- 172.21.15.28:0/2351139190 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff4c000fbe0 con 0x55cde2b86630 2024-04-25T01:33:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.714+0000 7ff4da987740 1 -- 172.21.15.28:0/2351139190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4c001bae0 con 0x55cde2b86630 2024-04-25T01:33:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.714+0000 7ff4b97fa640 1 -- 172.21.15.28:0/2351139190 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff4c0035050 con 0x55cde2b86630 2024-04-25T01:33:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.714+0000 7ff4da987740 10 client.15831 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.714+0000 7ff4da987740 1 -- 172.21.15.28:0/2351139190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cde2c07f90 con 0x55cde2b86630 2024-04-25T01:33:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.715+0000 7ff4b97fa640 1 -- 172.21.15.28:0/2351139190 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4c001bae0 con 0x55cde2b86630 2024-04-25T01:33:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.715+0000 7ff4b97fa640 1 -- 172.21.15.28:0/2351139190 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff4c00046c0 con 0x55cde2b86630 2024-04-25T01:33:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.715+0000 7ff4da987740 10 client.15831 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.715+0000 7ff4da987740 10 client.15831 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.715+0000 7ff4da987740 20 client.15831 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:36.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.715+0000 7ff4da987740 1 --2- 172.21.15.28:0/2351139190 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cde2b91620 0x55cde2c0a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.715+0000 7ff4da987740 4 client.15831 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:33:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.715+0000 7ff4da987740 1 -- 172.21.15.28:0/2351139190 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cde2af2f40 con 0x55cde2b91620 2024-04-25T01:33:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.715+0000 7ff4d593a640 1 --2- 172.21.15.28:0/2351139190 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cde2b91620 0x55cde2c0a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.715+0000 7ff4d593a640 1 --2- 172.21.15.28:0/2351139190 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cde2b91620 0x55cde2c0a610 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48756d4008daa56c server_cookie=48d83b4034d431d1 in_seq=0 out_seq=0 2024-04-25T01:33:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.715+0000 7ff4b97fa640 10 client.15831 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.716+0000 7ff4b97fa640 1 -- 172.21.15.28:0/2351139190 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55cde2af2f40 con 0x55cde2b91620 2024-04-25T01:33:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.716+0000 7ff4b97fa640 10 client.15831 handle_command_reply: tid=0 2024-04-25T01:33:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.716+0000 7ff4da987740 1 client.15831 shutdown 2024-04-25T01:33:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.716+0000 7ff4da987740 1 -- 172.21.15.28:0/2351139190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 msgr2=0x55cde2b8baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.716+0000 7ff4da987740 1 --2- 172.21.15.28:0/2351139190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 0x55cde2b8baa0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0010d00 tx=0x7ff4c000aa10 comp rx=0 tx=0).stop 2024-04-25T01:33:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.716+0000 7ff4da987740 1 -- 172.21.15.28:0/2351139190 shutdown_connections 2024-04-25T01:33:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.716+0000 7ff4da987740 1 --2- 172.21.15.28:0/2351139190 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cde2b91620 0x55cde2c0a610 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.716+0000 7ff4da987740 1 --2- 172.21.15.28:0/2351139190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cde2b86f50 0x55cde2c069e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.716+0000 7ff4b97fa640 0 client.15831 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.716+0000 7ff4da987740 1 --2- 172.21.15.28:0/2351139190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cde2b86630 0x55cde2b8baa0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.716+0000 7ff4da987740 1 --2- 172.21.15.28:0/2351139190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cde2b82950 0x55cde2b8b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.716+0000 7ff4da987740 1 -- 172.21.15.28:0/2351139190 >> 172.21.15.28:0/2351139190 conn(0x55cde2b8d240 msgr2=0x55cde2b8e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.716+0000 7ff4da987740 1 -- 172.21.15.28:0/2351139190 shutdown_connections 2024-04-25T01:33:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.716+0000 7ff4da987740 1 -- 172.21.15.28:0/2351139190 wait complete. 2024-04-25T01:33:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.716+0000 7ff4da987740 20 client.15831 trim_cache size 0 max 16384 2024-04-25T01:33:36.704 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:33:36.704 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:33:36.705 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:33:36.705 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:33:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.717+0000 7ff4d7b9e640 1 -- 172.21.15.28:0/2885856709 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4bc0a2d90 msgr2=0x7ff4bc0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.717+0000 7ff4d7b9e640 1 --2- 172.21.15.28:0/2885856709 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4bc0a2d90 0x7ff4bc0a5250 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff4c80039e0 tx=0x7ff4c8047000 comp rx=0 tx=0).stop 2024-04-25T01:33:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.717+0000 7ff4d7b9e640 1 -- 172.21.15.28:0/2885856709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0101710 msgr2=0x7ff4d0197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.717+0000 7ff4d7b9e640 1 --2- 172.21.15.28:0/2885856709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0101710 0x7ff4d0197fd0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc00b4a0 tx=0x7ff4cc00b8a0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.717+0000 7ff4d7b9e640 1 -- 172.21.15.28:0/2885856709 shutdown_connections 2024-04-25T01:33:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.717+0000 7ff4d7b9e640 1 --2- 172.21.15.28:0/2885856709 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4bc0a2d90 0x7ff4bc0a5250 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.717+0000 7ff4d7b9e640 1 --2- 172.21.15.28:0/2885856709 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4d0107720 0x7ff4d019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.717+0000 7ff4d7b9e640 1 --2- 172.21.15.28:0/2885856709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4d01020d0 0x7ff4d0198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.717+0000 7ff4d7b9e640 1 --2- 172.21.15.28:0/2885856709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0101710 0x7ff4d0197fd0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.717+0000 7ff4d7b9e640 1 -- 172.21.15.28:0/2885856709 >> 172.21.15.28:0/2885856709 conn(0x7ff4d00fd180 msgr2=0x7ff4d00fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.717+0000 7ff4d7b9e640 1 -- 172.21.15.28:0/2885856709 shutdown_connections 2024-04-25T01:33:36.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.717+0000 7ff4d7b9e640 1 -- 172.21.15.28:0/2885856709 wait complete. 2024-04-25T01:33:36.707 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:33:36.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:33:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.832+0000 7fca28e38640 1 -- 172.21.15.28:0/4269299967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca241032f0 msgr2=0x7fca2410b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.832+0000 7fca28e38640 1 --2- 172.21.15.28:0/4269299967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca241032f0 0x7fca2410b7d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009900 tx=0x7fca0c02a510 comp rx=0 tx=0).stop 2024-04-25T01:33:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.832+0000 7fca28e38640 1 -- 172.21.15.28:0/4269299967 shutdown_connections 2024-04-25T01:33:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.832+0000 7fca28e38640 1 --2- 172.21.15.28:0/4269299967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca2410bd10 0x7fca2410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.832+0000 7fca28e38640 1 --2- 172.21.15.28:0/4269299967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca241032f0 0x7fca2410b7d0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.832+0000 7fca28e38640 1 --2- 172.21.15.28:0/4269299967 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fca241029d0 0x7fca24102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.832+0000 7fca28e38640 1 -- 172.21.15.28:0/4269299967 >> 172.21.15.28:0/4269299967 conn(0x7fca24077740 msgr2=0x7fca24077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.832+0000 7fca28e38640 1 -- 172.21.15.28:0/4269299967 shutdown_connections 2024-04-25T01:33:36.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.832+0000 7fca28e38640 1 -- 172.21.15.28:0/4269299967 wait complete. 2024-04-25T01:33:36.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.833+0000 7fca28e38640 1 Processor -- start 2024-04-25T01:33:36.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.833+0000 7fca28e38640 1 -- start start 2024-04-25T01:33:36.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.833+0000 7fca28e38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca241029d0 0x7fca24197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.833+0000 7fca28e38640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca241032f0 0x7fca24198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.833+0000 7fca28e38640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fca2410bd10 0x7fca2419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.833+0000 7fca28e38640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fca24111b10 con 0x7fca241029d0 2024-04-25T01:33:36.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.833+0000 7fca28e38640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fca24111990 con 0x7fca241032f0 2024-04-25T01:33:36.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.833+0000 7fca28e38640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fca24111c90 con 0x7fca2410bd10 2024-04-25T01:33:36.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.833+0000 7fca2259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca241029d0 0x7fca24197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.833+0000 7fca2259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca241029d0 0x7fca24197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40036/0 (socket says 172.21.15.28:40036) 2024-04-25T01:33:36.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.833+0000 7fca2259c640 1 -- 172.21.15.28:0/238256142 learned_addr learned my addr 172.21.15.28:0/238256142 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:36.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.834+0000 7fca22d9d640 1 --2- 172.21.15.28:0/238256142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fca2410bd10 0x7fca2419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.834+0000 7fca1bfff640 1 --2- 172.21.15.28:0/238256142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca241032f0 0x7fca24198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.834+0000 7fca2259c640 1 -- 172.21.15.28:0/238256142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fca2410bd10 msgr2=0x7fca2419c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.834+0000 7fca2259c640 1 --2- 172.21.15.28:0/238256142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fca2410bd10 0x7fca2419c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.834+0000 7fca2259c640 1 -- 172.21.15.28:0/238256142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca241032f0 msgr2=0x7fca24198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.834+0000 7fca2259c640 1 --2- 172.21.15.28:0/238256142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca241032f0 0x7fca24198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.834+0000 7fca2259c640 1 -- 172.21.15.28:0/238256142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca2419cf70 con 0x7fca241029d0 2024-04-25T01:33:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.834+0000 7fca2259c640 1 --2- 172.21.15.28:0/238256142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca241029d0 0x7fca24197fd0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fca1400d770 tx=0x7fca1400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.834+0000 7fca1b7fe640 1 -- 172.21.15.28:0/238256142 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca14017070 con 0x7fca241029d0 2024-04-25T01:33:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.834+0000 7fca1b7fe640 1 -- 172.21.15.28:0/238256142 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca14005500 con 0x7fca241029d0 2024-04-25T01:33:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.834+0000 7fca1b7fe640 1 -- 172.21.15.28:0/238256142 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca14004e00 con 0x7fca241029d0 2024-04-25T01:33:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.834+0000 7fca28e38640 1 -- 172.21.15.28:0/238256142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca2419d260 con 0x7fca241029d0 2024-04-25T01:33:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.834+0000 7fca28e38640 1 -- 172.21.15.28:0/238256142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca241a4af0 con 0x7fca241029d0 2024-04-25T01:33:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.835+0000 7fca28e38640 1 -- 172.21.15.28:0/238256142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca24068560 con 0x7fca241029d0 2024-04-25T01:33:36.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.836+0000 7fca1b7fe640 1 -- 172.21.15.28:0/238256142 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7fca14004fa0 con 0x7fca241029d0 2024-04-25T01:33:36.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.837+0000 7fca1b7fe640 1 --2- 172.21.15.28:0/238256142 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fca100a2d90 0x7fca100a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:36.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.837+0000 7fca1b7fe640 1 -- 172.21.15.28:0/238256142 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fca140cf300 con 0x7fca241029d0 2024-04-25T01:33:36.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.837+0000 7fca1bfff640 1 --2- 172.21.15.28:0/238256142 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fca100a2d90 0x7fca100a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:36.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.837+0000 7fca1bfff640 1 --2- 172.21.15.28:0/238256142 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fca100a2d90 0x7fca100a5250 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009230 tx=0x7fca0c0058d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:36.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:36.841+0000 7fca1b7fe640 1 -- 172.21.15.28:0/238256142 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fca1409b420 con 0x7fca241029d0 2024-04-25T01:33:37.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.065+0000 7fca28e38640 1 -- 172.21.15.28:0/238256142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca24106e90 con 0x7fca241029d0 2024-04-25T01:33:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.066+0000 7fca1b7fe640 1 -- 172.21.15.28:0/238256142 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7fca1409f720 con 0x7fca241029d0 2024-04-25T01:33:37.043 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:33:37.044 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:33:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:33:37.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.068+0000 7fca28e38640 1 -- 172.21.15.28:0/238256142 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fca100a2d90 msgr2=0x7fca100a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.068+0000 7fca28e38640 1 --2- 172.21.15.28:0/238256142 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fca100a2d90 0x7fca100a5250 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009230 tx=0x7fca0c0058d0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.068+0000 7fca28e38640 1 -- 172.21.15.28:0/238256142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca241029d0 msgr2=0x7fca24197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.068+0000 7fca28e38640 1 --2- 172.21.15.28:0/238256142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca241029d0 0x7fca24197fd0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fca1400d770 tx=0x7fca1400dc30 comp rx=0 tx=0).stop 2024-04-25T01:33:37.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.068+0000 7fca28e38640 1 -- 172.21.15.28:0/238256142 shutdown_connections 2024-04-25T01:33:37.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.068+0000 7fca28e38640 1 --2- 172.21.15.28:0/238256142 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fca100a2d90 0x7fca100a5250 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.068+0000 7fca28e38640 1 --2- 172.21.15.28:0/238256142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fca2410bd10 0x7fca2419c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.068+0000 7fca28e38640 1 --2- 172.21.15.28:0/238256142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca241032f0 0x7fca24198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.068+0000 7fca28e38640 1 --2- 172.21.15.28:0/238256142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca241029d0 0x7fca24197fd0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.068+0000 7fca28e38640 1 -- 172.21.15.28:0/238256142 >> 172.21.15.28:0/238256142 conn(0x7fca24077740 msgr2=0x7fca240fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.068+0000 7fca28e38640 1 -- 172.21.15.28:0/238256142 shutdown_connections 2024-04-25T01:33:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.068+0000 7fca28e38640 1 -- 172.21.15.28:0/238256142 wait complete. 2024-04-25T01:33:37.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:33:37.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.184+0000 7ff5259a9640 1 -- 172.21.15.28:0/1002820011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff52010bd00 msgr2=0x7ff52010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.184+0000 7ff5259a9640 1 --2- 172.21.15.28:0/1002820011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff52010bd00 0x7ff52010e1a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff518009ca0 tx=0x7ff51802a9e0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.184+0000 7ff5259a9640 1 -- 172.21.15.28:0/1002820011 shutdown_connections 2024-04-25T01:33:37.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.184+0000 7ff5259a9640 1 --2- 172.21.15.28:0/1002820011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff52010bd00 0x7ff52010e1a0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.184+0000 7ff5259a9640 1 --2- 172.21.15.28:0/1002820011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff5201032e0 0x7ff52010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.184+0000 7ff5259a9640 1 --2- 172.21.15.28:0/1002820011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5201029c0 0x7ff520102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.184+0000 7ff5259a9640 1 -- 172.21.15.28:0/1002820011 >> 172.21.15.28:0/1002820011 conn(0x7ff520077740 msgr2=0x7ff520077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.184+0000 7ff5259a9640 1 -- 172.21.15.28:0/1002820011 shutdown_connections 2024-04-25T01:33:37.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.184+0000 7ff5259a9640 1 -- 172.21.15.28:0/1002820011 wait complete. 2024-04-25T01:33:37.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.185+0000 7ff5259a9640 1 Processor -- start 2024-04-25T01:33:37.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.185+0000 7ff5259a9640 1 -- start start 2024-04-25T01:33:37.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.185+0000 7ff5259a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5201029c0 0x7ff52019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.185+0000 7ff5259a9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5201032e0 0x7ff52019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.185+0000 7ff5259a9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff52010bd00 0x7ff52019ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.185+0000 7ff5259a9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff520111b00 con 0x7ff5201029c0 2024-04-25T01:33:37.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.185+0000 7ff5259a9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff520111980 con 0x7ff52010bd00 2024-04-25T01:33:37.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.185+0000 7ff5259a9640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff520111c80 con 0x7ff5201032e0 2024-04-25T01:33:37.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.185+0000 7ff51effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5201029c0 0x7ff52019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.186+0000 7ff51effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5201029c0 0x7ff52019a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40066/0 (socket says 172.21.15.28:40066) 2024-04-25T01:33:37.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.186+0000 7ff51effd640 1 -- 172.21.15.28:0/533576249 learned_addr learned my addr 172.21.15.28:0/533576249 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.186+0000 7ff51e7fc640 1 --2- 172.21.15.28:0/533576249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5201032e0 0x7ff52019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.186+0000 7ff51f7fe640 1 --2- 172.21.15.28:0/533576249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff52010bd00 0x7ff52019ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.186+0000 7ff51effd640 1 -- 172.21.15.28:0/533576249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5201032e0 msgr2=0x7ff52019a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.186+0000 7ff51effd640 1 --2- 172.21.15.28:0/533576249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5201032e0 0x7ff52019a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.186+0000 7ff51effd640 1 -- 172.21.15.28:0/533576249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff52010bd00 msgr2=0x7ff52019ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.186+0000 7ff51effd640 1 --2- 172.21.15.28:0/533576249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff52010bd00 0x7ff52019ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.186+0000 7ff51effd640 1 -- 172.21.15.28:0/533576249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52019f190 con 0x7ff5201029c0 2024-04-25T01:33:37.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.186+0000 7ff51f7fe640 1 --2- 172.21.15.28:0/533576249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff52010bd00 0x7ff52019ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:37.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.186+0000 7ff51effd640 1 --2- 172.21.15.28:0/533576249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5201029c0 0x7ff52019a1a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff51000c890 tx=0x7ff51000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.186+0000 7ff5249a7640 1 -- 172.21.15.28:0/533576249 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff510004220 con 0x7ff5201029c0 2024-04-25T01:33:37.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.186+0000 7ff5249a7640 1 -- 172.21.15.28:0/533576249 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5100043c0 con 0x7ff5201029c0 2024-04-25T01:33:37.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.186+0000 7ff5259a9640 1 -- 172.21.15.28:0/533576249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff52019f480 con 0x7ff5201029c0 2024-04-25T01:33:37.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.186+0000 7ff5249a7640 1 -- 172.21.15.28:0/533576249 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff510004a00 con 0x7ff5201029c0 2024-04-25T01:33:37.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.186+0000 7ff5259a9640 1 -- 172.21.15.28:0/533576249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5201a6d10 con 0x7ff5201029c0 2024-04-25T01:33:37.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.188+0000 7ff5249a7640 1 -- 172.21.15.28:0/533576249 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7ff5100079b0 con 0x7ff5201029c0 2024-04-25T01:33:37.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff528799740 1 Processor -- start 2024-04-25T01:33:37.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff528799740 1 -- start start 2024-04-25T01:33:37.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff528799740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 0x5622f5441960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff528799740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f544c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff528799740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 0x5622f544e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff528799740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5622f53a6670 con 0x5622f5441ea0 2024-04-25T01:33:37.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff528799740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5622f534e970 con 0x5622f5441580 2024-04-25T01:33:37.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff528799740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5622f53a2630 con 0x5622f544c570 2024-04-25T01:33:37.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff51effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f544c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff51effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f544c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40078/0 (socket says 172.21.15.28:40078) 2024-04-25T01:33:37.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff51effd640 1 -- 172.21.15.28:0/1992495031 learned_addr learned my addr 172.21.15.28:0/1992495031 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff5249a7640 1 --2- 172.21.15.28:0/533576249 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff5080a2d90 0x7ff5080a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff51f7fe640 1 --2- 172.21.15.28:0/1992495031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 0x5622f5441960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff51e7fc640 1 --2- 172.21.15.28:0/1992495031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 0x5622f544e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff51f7fe640 1 --2- 172.21.15.28:0/533576249 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff5080a2d90 0x7ff5080a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff5249a7640 1 -- 172.21.15.28:0/533576249 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff5100cf630 con 0x7ff5201029c0 2024-04-25T01:33:37.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff51effd640 1 -- 172.21.15.28:0/1992495031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 msgr2=0x5622f544e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff51effd640 1 --2- 172.21.15.28:0/1992495031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 0x5622f544e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff51effd640 1 -- 172.21.15.28:0/1992495031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 msgr2=0x5622f5441960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff51effd640 1 --2- 172.21.15.28:0/1992495031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 0x5622f5441960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.189+0000 7ff51effd640 1 -- 172.21.15.28:0/1992495031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622f5372400 con 0x5622f5441ea0 2024-04-25T01:33:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.190+0000 7ff51effd640 1 --2- 172.21.15.28:0/1992495031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f544c030 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff51009b3a0 tx=0x7ff51009b5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b8125f469de9f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.190+0000 7ff51f7fe640 1 --2- 172.21.15.28:0/533576249 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff5080a2d90 0x7ff5080a5250 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff518009430 tx=0x7ff5180095c0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.190+0000 7ff505ffb640 1 -- 172.21.15.28:0/1992495031 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51001d050 con 0x5622f5441ea0 2024-04-25T01:33:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.190+0000 7ff505ffb640 1 -- 172.21.15.28:0/1992495031 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff51009bda0 con 0x5622f5441ea0 2024-04-25T01:33:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.190+0000 7ff505ffb640 1 -- 172.21.15.28:0/1992495031 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5100cf2d0 con 0x5622f5441ea0 2024-04-25T01:33:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.190+0000 7ff528799740 1 -- 172.21.15.28:0/1992495031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 msgr2=0x5622f544c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.190+0000 7ff528799740 1 --2- 172.21.15.28:0/1992495031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f544c030 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff51009b3a0 tx=0x7ff51009b5a0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.191+0000 7ff528799740 1 -- 172.21.15.28:0/1992495031 shutdown_connections 2024-04-25T01:33:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.191+0000 7ff528799740 1 --2- 172.21.15.28:0/1992495031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 0x5622f544e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.191+0000 7ff528799740 1 --2- 172.21.15.28:0/1992495031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f544c030 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.191+0000 7ff528799740 1 --2- 172.21.15.28:0/1992495031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 0x5622f5441960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.191+0000 7ff528799740 1 -- 172.21.15.28:0/1992495031 >> 172.21.15.28:0/1992495031 conn(0x5622f543d8a0 msgr2=0x5622f543dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.191+0000 7ff528799740 1 -- 172.21.15.28:0/1992495031 shutdown_connections 2024-04-25T01:33:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.191+0000 7ff528799740 1 -- 172.21.15.28:0/1992495031 wait complete. 2024-04-25T01:33:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.191+0000 7ff528799740 1 -- start start 2024-04-25T01:33:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.191+0000 7ff528799740 1 Processor -- start 2024-04-25T01:33:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.191+0000 7ff528799740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 0x5622f544bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.191+0000 7ff528799740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f54491d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.191+0000 7ff528799740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 0x5622f5449730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.191+0000 7ff528799740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5622f544f300 con 0x5622f5441ea0 2024-04-25T01:33:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.191+0000 7ff528799740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5622f5362050 con 0x5622f5441580 2024-04-25T01:33:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.191+0000 7ff528799740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5622f544f480 con 0x5622f544c570 2024-04-25T01:33:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.191+0000 7ff528799740 10 client.0 _ll_register_callbacks cb 0x5622f50b9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.191+0000 7ff51e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f54491d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.192+0000 7ff51e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f54491d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40080/0 (socket says 172.21.15.28:40080) 2024-04-25T01:33:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.192+0000 7ff51e7fc640 1 -- 172.21.15.28:0/2775157063 learned_addr learned my addr 172.21.15.28:0/2775157063 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.192+0000 7ff51f7fe640 1 --2- 172.21.15.28:0/2775157063 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 0x5622f5449730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.192+0000 7ff51effd640 1 --2- 172.21.15.28:0/2775157063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 0x5622f544bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.192+0000 7ff51e7fc640 1 -- 172.21.15.28:0/2775157063 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 msgr2=0x5622f5449730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.192+0000 7ff51e7fc640 1 --2- 172.21.15.28:0/2775157063 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 0x5622f5449730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.192+0000 7ff51e7fc640 1 -- 172.21.15.28:0/2775157063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 msgr2=0x5622f544bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.192+0000 7ff51e7fc640 1 --2- 172.21.15.28:0/2775157063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 0x5622f544bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.192+0000 7ff51e7fc640 1 -- 172.21.15.28:0/2775157063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622f5333dc0 con 0x5622f5441ea0 2024-04-25T01:33:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.192+0000 7ff51e7fc640 1 --2- 172.21.15.28:0/2775157063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f54491d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7ff51400be30 tx=0x7ff514012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edbf3833130c1dbf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.192+0000 7ff4feffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.192+0000 7ff4feffd640 1 -- 172.21.15.28:0/2775157063 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff514004220 con 0x5622f5441ea0 2024-04-25T01:33:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.192+0000 7ff4feffd640 1 -- 172.21.15.28:0/2775157063 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff5140043c0 con 0x5622f5441ea0 2024-04-25T01:33:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.192+0000 7ff528799740 1 -- 172.21.15.28:0/2775157063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff51009bda0 con 0x5622f5441ea0 2024-04-25T01:33:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.192+0000 7ff4feffd640 1 -- 172.21.15.28:0/2775157063 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff514005020 con 0x5622f5441ea0 2024-04-25T01:33:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.192+0000 7ff528799740 10 client.15849 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.192+0000 7ff528799740 1 -- 172.21.15.28:0/2775157063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5622f50ba3f0 con 0x5622f5441ea0 2024-04-25T01:33:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.193+0000 7ff4feffd640 1 -- 172.21.15.28:0/2775157063 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff51009bda0 con 0x5622f5441ea0 2024-04-25T01:33:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.193+0000 7ff4feffd640 1 -- 172.21.15.28:0/2775157063 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff514019070 con 0x5622f5441ea0 2024-04-25T01:33:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.193+0000 7ff528799740 10 client.15849 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.193+0000 7ff528799740 10 client.15849 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:33:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.193+0000 7ff528799740 20 client.15849 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.193+0000 7ff528799740 1 --2- 172.21.15.28:0/2775157063 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5622f545c6e0 0x5622f545eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.193+0000 7ff528799740 4 client.15849 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.193+0000 7ff528799740 1 -- 172.21.15.28:0/2775157063 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622f53b3bb0 con 0x5622f545c6e0 2024-04-25T01:33:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.193+0000 7ff51effd640 1 --2- 172.21.15.28:0/2775157063 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5622f545c6e0 0x5622f545eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.193+0000 7ff4feffd640 10 client.15849 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:33:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.193+0000 7ff51effd640 1 --2- 172.21.15.28:0/2775157063 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5622f545c6e0 0x5622f545eb20 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8e701ddb8bb25199 server_cookie=c5531ff213636cd9 in_seq=0 out_seq=0 2024-04-25T01:33:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.194+0000 7ff4feffd640 1 -- 172.21.15.28:0/2775157063 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5622f53b3bb0 con 0x5622f545c6e0 2024-04-25T01:33:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.194+0000 7ff4feffd640 10 client.15849 handle_command_reply: tid=0 2024-04-25T01:33:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.194+0000 7ff528799740 1 client.15849 shutdown 2024-04-25T01:33:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.194+0000 7ff528799740 1 -- 172.21.15.28:0/2775157063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 msgr2=0x5622f54491d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.194+0000 7ff528799740 1 --2- 172.21.15.28:0/2775157063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f54491d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7ff51400be30 tx=0x7ff514012820 comp rx=0 tx=0).stop 2024-04-25T01:33:37.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.194+0000 7ff528799740 1 -- 172.21.15.28:0/2775157063 shutdown_connections 2024-04-25T01:33:37.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.194+0000 7ff528799740 1 --2- 172.21.15.28:0/2775157063 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5622f545c6e0 0x5622f545eb20 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.194+0000 7ff528799740 1 --2- 172.21.15.28:0/2775157063 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 0x5622f5449730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.194+0000 7ff4feffd640 0 client.15849 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:33:37.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.194+0000 7ff528799740 1 --2- 172.21.15.28:0/2775157063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f54491d0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.194+0000 7ff528799740 1 --2- 172.21.15.28:0/2775157063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 0x5622f544bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.194+0000 7ff528799740 1 -- 172.21.15.28:0/2775157063 >> 172.21.15.28:0/2775157063 conn(0x5622f543d8a0 msgr2=0x5622f543cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.195+0000 7ff528799740 1 -- 172.21.15.28:0/2775157063 shutdown_connections 2024-04-25T01:33:37.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.195+0000 7ff528799740 1 -- 172.21.15.28:0/2775157063 wait complete. 2024-04-25T01:33:37.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.195+0000 7ff528799740 20 client.15849 trim_cache size 0 max 16384 2024-04-25T01:33:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.208+0000 7ff528799740 1 Processor -- start 2024-04-25T01:33:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.208+0000 7ff528799740 1 -- start start 2024-04-25T01:33:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.208+0000 7ff528799740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 0x5622f543d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.208+0000 7ff528799740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f5438ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.208+0000 7ff528799740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 0x5622f544ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.208+0000 7ff528799740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff5201b0d30 con 0x5622f5441ea0 2024-04-25T01:33:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.208+0000 7ff528799740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff5201b2870 con 0x5622f5441580 2024-04-25T01:33:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.208+0000 7ff528799740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff5201b1400 con 0x5622f544c570 2024-04-25T01:33:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.208+0000 7ff51e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f5438ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.208+0000 7ff51e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f5438ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40088/0 (socket says 172.21.15.28:40088) 2024-04-25T01:33:37.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.208+0000 7ff51e7fc640 1 -- 172.21.15.28:0/3618712994 learned_addr learned my addr 172.21.15.28:0/3618712994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.208+0000 7ff51effd640 1 --2- 172.21.15.28:0/3618712994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 0x5622f543d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.208+0000 7ff51e7fc640 1 -- 172.21.15.28:0/3618712994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 msgr2=0x5622f544ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.208+0000 7ff51e7fc640 1 --2- 172.21.15.28:0/3618712994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 0x5622f544ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.208+0000 7ff51e7fc640 1 -- 172.21.15.28:0/3618712994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 msgr2=0x5622f543d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.208+0000 7ff51e7fc640 1 --2- 172.21.15.28:0/3618712994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 0x5622f543d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.208+0000 7ff51e7fc640 1 -- 172.21.15.28:0/3618712994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622f5441cd0 con 0x5622f5441ea0 2024-04-25T01:33:37.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.209+0000 7ff51e7fc640 1 --2- 172.21.15.28:0/3618712994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f5438ad0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff514010980 tx=0x7ff51400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a598be290abd2c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.209+0000 7ff5057fa640 1 -- 172.21.15.28:0/3618712994 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff514010a00 con 0x5622f5441ea0 2024-04-25T01:33:37.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.209+0000 7ff5057fa640 1 -- 172.21.15.28:0/3618712994 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff514010ba0 con 0x5622f5441ea0 2024-04-25T01:33:37.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.209+0000 7ff5057fa640 1 -- 172.21.15.28:0/3618712994 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff514027450 con 0x5622f5441ea0 2024-04-25T01:33:37.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.209+0000 7ff528799740 1 -- 172.21.15.28:0/3618712994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 msgr2=0x5622f5438ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.209+0000 7ff528799740 1 --2- 172.21.15.28:0/3618712994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f5438ad0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff514010980 tx=0x7ff51400b790 comp rx=0 tx=0).stop 2024-04-25T01:33:37.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.209+0000 7ff528799740 1 -- 172.21.15.28:0/3618712994 shutdown_connections 2024-04-25T01:33:37.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.209+0000 7ff528799740 1 --2- 172.21.15.28:0/3618712994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 0x5622f544ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.209+0000 7ff528799740 1 --2- 172.21.15.28:0/3618712994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f5438ad0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.209+0000 7ff528799740 1 --2- 172.21.15.28:0/3618712994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 0x5622f543d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.209+0000 7ff528799740 1 -- 172.21.15.28:0/3618712994 >> 172.21.15.28:0/3618712994 conn(0x5622f543d8a0 msgr2=0x5622f5443560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.209+0000 7ff528799740 1 -- 172.21.15.28:0/3618712994 shutdown_connections 2024-04-25T01:33:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.209+0000 7ff528799740 1 -- 172.21.15.28:0/3618712994 wait complete. 2024-04-25T01:33:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.210+0000 7ff528799740 1 -- start start 2024-04-25T01:33:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.210+0000 7ff528799740 1 Processor -- start 2024-04-25T01:33:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.210+0000 7ff528799740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 0x5622f5454960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.210+0000 7ff528799740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f5445a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.210+0000 7ff528799740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 0x5622f5445f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.210+0000 7ff528799740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff5201b21a0 con 0x5622f5441ea0 2024-04-25T01:33:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.210+0000 7ff528799740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff5201b1ad0 con 0x5622f5441580 2024-04-25T01:33:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.210+0000 7ff528799740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff5201aed10 con 0x5622f544c570 2024-04-25T01:33:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.210+0000 7ff528799740 10 client.0 _ll_register_callbacks cb 0x7ff51400ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.210+0000 7ff51e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f5445a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.210+0000 7ff51e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f5445a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40098/0 (socket says 172.21.15.28:40098) 2024-04-25T01:33:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.210+0000 7ff51e7fc640 1 -- 172.21.15.28:0/1493487860 learned_addr learned my addr 172.21.15.28:0/1493487860 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.211+0000 7ff51f7fe640 1 --2- 172.21.15.28:0/1493487860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 0x5622f5445f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.211+0000 7ff51effd640 1 --2- 172.21.15.28:0/1493487860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 0x5622f5454960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.211+0000 7ff51e7fc640 1 -- 172.21.15.28:0/1493487860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 msgr2=0x5622f5445f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.211+0000 7ff51e7fc640 1 --2- 172.21.15.28:0/1493487860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 0x5622f5445f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.211+0000 7ff51e7fc640 1 -- 172.21.15.28:0/1493487860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 msgr2=0x5622f5454960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.211+0000 7ff51e7fc640 1 --2- 172.21.15.28:0/1493487860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 0x5622f5454960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.211+0000 7ff51e7fc640 1 -- 172.21.15.28:0/1493487860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622f51847d0 con 0x5622f5441ea0 2024-04-25T01:33:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.211+0000 7ff51f7fe640 1 --2- 172.21.15.28:0/1493487860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 0x5622f5445f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.211+0000 7ff51e7fc640 1 --2- 172.21.15.28:0/1493487860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f5445a40 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff514010710 tx=0x7ff51400b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f622e075005945 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.211+0000 7ff4feffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.211+0000 7ff4feffd640 1 -- 172.21.15.28:0/1493487860 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5140109b0 con 0x5622f5441ea0 2024-04-25T01:33:37.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.211+0000 7ff4feffd640 1 -- 172.21.15.28:0/1493487860 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff514004020 con 0x5622f5441ea0 2024-04-25T01:33:37.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.211+0000 7ff528799740 1 -- 172.21.15.28:0/1493487860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff514010ba0 con 0x5622f5441ea0 2024-04-25T01:33:37.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.211+0000 7ff4feffd640 1 -- 172.21.15.28:0/1493487860 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff514004220 con 0x5622f5441ea0 2024-04-25T01:33:37.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.211+0000 7ff528799740 10 client.15855 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:37.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.211+0000 7ff528799740 1 -- 172.21.15.28:0/1493487860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5622f54c28f0 con 0x5622f5441ea0 2024-04-25T01:33:37.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.211+0000 7ff4feffd640 1 -- 172.21.15.28:0/1493487860 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff514010ba0 con 0x5622f5441ea0 2024-04-25T01:33:37.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.211+0000 7ff4feffd640 1 -- 172.21.15.28:0/1493487860 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff514019040 con 0x5622f5441ea0 2024-04-25T01:33:37.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.212+0000 7ff528799740 10 client.15855 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:37.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.212+0000 7ff528799740 10 client.15855 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:33:37.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.212+0000 7ff528799740 20 client.15855 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:37.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.212+0000 7ff528799740 1 --2- 172.21.15.28:0/1493487860 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5622f54c2f50 0x5622f54c5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.212+0000 7ff528799740 4 client.15855 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:33:37.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.212+0000 7ff528799740 1 -- 172.21.15.28:0/1493487860 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5622f53adf00 con 0x5622f54c2f50 2024-04-25T01:33:37.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.212+0000 7ff51effd640 1 --2- 172.21.15.28:0/1493487860 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5622f54c2f50 0x5622f54c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.212+0000 7ff51effd640 1 --2- 172.21.15.28:0/1493487860 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5622f54c2f50 0x5622f54c5340 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=39db773c0774e9fd server_cookie=ab9a8ba23023000a in_seq=0 out_seq=0 2024-04-25T01:33:37.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.212+0000 7ff4feffd640 10 client.15855 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:33:37.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff4feffd640 1 -- 172.21.15.28:0/1493487860 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5622f53adf00 con 0x5622f54c2f50 2024-04-25T01:33:37.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff4feffd640 10 client.15855 handle_command_reply: tid=0 2024-04-25T01:33:37.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff528799740 1 client.15855 shutdown 2024-04-25T01:33:37.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff528799740 1 -- 172.21.15.28:0/1493487860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 msgr2=0x5622f5445a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff528799740 1 --2- 172.21.15.28:0/1493487860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f5445a40 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff514010710 tx=0x7ff51400b9b0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff528799740 1 -- 172.21.15.28:0/1493487860 shutdown_connections 2024-04-25T01:33:37.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff528799740 1 --2- 172.21.15.28:0/1493487860 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5622f54c2f50 0x5622f54c5340 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff528799740 1 --2- 172.21.15.28:0/1493487860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5622f544c570 0x5622f5445f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff4feffd640 0 client.15855 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:33:37.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff528799740 1 --2- 172.21.15.28:0/1493487860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622f5441ea0 0x5622f5445a40 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff528799740 1 --2- 172.21.15.28:0/1493487860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5622f5441580 0x5622f5454960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff528799740 1 -- 172.21.15.28:0/1493487860 >> 172.21.15.28:0/1493487860 conn(0x5622f543d8a0 msgr2=0x5622f5443560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff528799740 1 -- 172.21.15.28:0/1493487860 shutdown_connections 2024-04-25T01:33:37.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff528799740 1 -- 172.21.15.28:0/1493487860 wait complete. 2024-04-25T01:33:37.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff528799740 20 client.15855 trim_cache size 0 max 16384 2024-04-25T01:33:37.199 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:33:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff5259a9640 1 -- 172.21.15.28:0/533576249 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff5080a2d90 msgr2=0x7ff5080a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff5259a9640 1 --2- 172.21.15.28:0/533576249 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff5080a2d90 0x7ff5080a5250 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff518009430 tx=0x7ff5180095c0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff5259a9640 1 -- 172.21.15.28:0/533576249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5201029c0 msgr2=0x7ff52019a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff5259a9640 1 --2- 172.21.15.28:0/533576249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5201029c0 0x7ff52019a1a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff51000c890 tx=0x7ff51000cd50 comp rx=0 tx=0).stop 2024-04-25T01:33:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff5259a9640 1 -- 172.21.15.28:0/533576249 shutdown_connections 2024-04-25T01:33:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff5259a9640 1 --2- 172.21.15.28:0/533576249 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff5080a2d90 0x7ff5080a5250 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.213+0000 7ff5259a9640 1 --2- 172.21.15.28:0/533576249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff52010bd00 0x7ff52019ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.214+0000 7ff5259a9640 1 --2- 172.21.15.28:0/533576249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff5201032e0 0x7ff52019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.214+0000 7ff5259a9640 1 --2- 172.21.15.28:0/533576249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5201029c0 0x7ff52019a1a0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.214+0000 7ff5259a9640 1 -- 172.21.15.28:0/533576249 >> 172.21.15.28:0/533576249 conn(0x7ff520077740 msgr2=0x7ff520107bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.214+0000 7ff5259a9640 1 -- 172.21.15.28:0/533576249 shutdown_connections 2024-04-25T01:33:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.214+0000 7ff5259a9640 1 -- 172.21.15.28:0/533576249 wait complete. 2024-04-25T01:33:37.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:33:37.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.328+0000 7f1f6f329640 1 -- 172.21.15.28:0/1751258458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f68069dc0 msgr2=0x7f1f6806a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.328+0000 7f1f6f329640 1 --2- 172.21.15.28:0/1751258458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f68069dc0 0x7f1f6806a240 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009930 tx=0x7f1f5802a420 comp rx=0 tx=0).stop 2024-04-25T01:33:37.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.329+0000 7f1f6f329640 1 -- 172.21.15.28:0/1751258458 shutdown_connections 2024-04-25T01:33:37.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.329+0000 7f1f6f329640 1 --2- 172.21.15.28:0/1751258458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f6806a780 0x7f1f6810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.329+0000 7f1f6f329640 1 --2- 172.21.15.28:0/1751258458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f68069dc0 0x7f1f6806a240 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.329+0000 7f1f6f329640 1 --2- 172.21.15.28:0/1751258458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f68107730 0x7f1f68107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.329+0000 7f1f6f329640 1 -- 172.21.15.28:0/1751258458 >> 172.21.15.28:0/1751258458 conn(0x7f1f680776a0 msgr2=0x7f1f68077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.329+0000 7f1f6f329640 1 -- 172.21.15.28:0/1751258458 shutdown_connections 2024-04-25T01:33:37.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.329+0000 7f1f6f329640 1 -- 172.21.15.28:0/1751258458 wait complete. 2024-04-25T01:33:37.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.329+0000 7f1f6f329640 1 Processor -- start 2024-04-25T01:33:37.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.329+0000 7f1f6f329640 1 -- start start 2024-04-25T01:33:37.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f6f329640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f68069dc0 0x7f1f68100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f6f329640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f6806a780 0x7f1f68100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f6f329640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f68107730 0x7f1f681010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f6f329640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f68111b40 con 0x7f1f68069dc0 2024-04-25T01:33:37.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f6f329640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1f681119c0 con 0x7f1f6806a780 2024-04-25T01:33:37.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f6f329640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1f68111cc0 con 0x7f1f68107730 2024-04-25T01:33:37.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f6d0c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f68069dc0 0x7f1f68100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f6d0c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f68069dc0 0x7f1f68100670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40120/0 (socket says 172.21.15.28:40120) 2024-04-25T01:33:37.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f6d0c5640 1 -- 172.21.15.28:0/1868815721 learned_addr learned my addr 172.21.15.28:0/1868815721 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f6d8c6640 1 --2- 172.21.15.28:0/1868815721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f68107730 0x7f1f681010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f6c8c4640 1 --2- 172.21.15.28:0/1868815721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f6806a780 0x7f1f68100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f6d0c5640 1 -- 172.21.15.28:0/1868815721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f68107730 msgr2=0x7f1f681010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f6d0c5640 1 --2- 172.21.15.28:0/1868815721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f68107730 0x7f1f681010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f6d0c5640 1 -- 172.21.15.28:0/1868815721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f6806a780 msgr2=0x7f1f68100bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f6d0c5640 1 --2- 172.21.15.28:0/1868815721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f6806a780 0x7f1f68100bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f6d0c5640 1 -- 172.21.15.28:0/1868815721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f681a6500 con 0x7f1f68069dc0 2024-04-25T01:33:37.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f6d0c5640 1 --2- 172.21.15.28:0/1868815721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f68069dc0 0x7f1f68100670 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1f6400d770 tx=0x7f1f6400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f5e7fc640 1 -- 172.21.15.28:0/1868815721 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f64017070 con 0x7f1f68069dc0 2024-04-25T01:33:37.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f5e7fc640 1 -- 172.21.15.28:0/1868815721 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f64005500 con 0x7f1f68069dc0 2024-04-25T01:33:37.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f6f329640 1 -- 172.21.15.28:0/1868815721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f681a67f0 con 0x7f1f68069dc0 2024-04-25T01:33:37.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f5e7fc640 1 -- 172.21.15.28:0/1868815721 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f64004e00 con 0x7f1f68069dc0 2024-04-25T01:33:37.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.330+0000 7f1f6f329640 1 -- 172.21.15.28:0/1868815721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f681a6d10 con 0x7f1f68069dc0 2024-04-25T01:33:37.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.332+0000 7f1f5e7fc640 1 -- 172.21.15.28:0/1868815721 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f1f64004fa0 con 0x7f1f68069dc0 2024-04-25T01:33:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.333+0000 7f1f7203a740 1 Processor -- start 2024-04-25T01:33:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.333+0000 7f1f7203a740 1 -- start start 2024-04-25T01:33:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.333+0000 7f1f7203a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55832611d580 0x55832611d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.333+0000 7f1f7203a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 0x558326128030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.333+0000 7f1f7203a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 0x55832612a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.333+0000 7f1f7203a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558326082670 con 0x558326128570 2024-04-25T01:33:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.333+0000 7f1f7203a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55832602a970 con 0x55832611d580 2024-04-25T01:33:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.333+0000 7f1f7203a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55832607e630 con 0x55832611dea0 2024-04-25T01:33:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.333+0000 7f1f5e7fc640 1 --2- 172.21.15.28:0/1868815721 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1f540a2e10 0x7f1f540a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.333+0000 7f1f6d8c6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55832611d580 0x55832611d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.333+0000 7f1f6d8c6640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55832611d580 0x55832611d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:58996/0 (socket says 172.21.15.28:58996) 2024-04-25T01:33:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.333+0000 7f1f6c8c4640 1 --2- 172.21.15.28:0/528487747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 0x55832612a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.333+0000 7f1f6d8c6640 1 -- 172.21.15.28:0/528487747 learned_addr learned my addr 172.21.15.28:0/528487747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.333+0000 7f1f6d0c5640 1 --2- 172.21.15.28:0/528487747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 0x558326128030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.334+0000 7f1f6d8c6640 1 --2- 172.21.15.28:0/1868815721 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1f540a2e10 0x7f1f540a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.334+0000 7f1f6c8c4640 1 -- 172.21.15.28:0/528487747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 msgr2=0x558326128030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.334+0000 7f1f6c8c4640 1 --2- 172.21.15.28:0/528487747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 0x558326128030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.334+0000 7f1f5e7fc640 1 -- 172.21.15.28:0/1868815721 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1f640cf380 con 0x7f1f68069dc0 2024-04-25T01:33:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.334+0000 7f1f6c8c4640 1 -- 172.21.15.28:0/528487747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55832611d580 msgr2=0x55832611d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.334+0000 7f1f6c8c4640 1 --2- 172.21.15.28:0/528487747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55832611d580 0x55832611d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.334+0000 7f1f6c8c4640 1 -- 172.21.15.28:0/528487747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55832604e400 con 0x558326128570 2024-04-25T01:33:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.334+0000 7f1f6c8c4640 1 --2- 172.21.15.28:0/528487747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 0x55832612a960 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009900 tx=0x7f1f58005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=735bda3d593c4cda server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.334+0000 7f1f6d8c6640 1 --2- 172.21.15.28:0/1868815721 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1f540a2e10 0x7f1f540a52d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009fd0 tx=0x7f1f60004ab0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.334+0000 7f1f537fe640 1 -- 172.21.15.28:0/528487747 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5803f070 con 0x558326128570 2024-04-25T01:33:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.334+0000 7f1f537fe640 1 -- 172.21.15.28:0/528487747 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f58004030 con 0x558326128570 2024-04-25T01:33:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.334+0000 7f1f537fe640 1 -- 172.21.15.28:0/528487747 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f58035070 con 0x558326128570 2024-04-25T01:33:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.335+0000 7f1f7203a740 1 -- 172.21.15.28:0/528487747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 msgr2=0x55832612a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.335+0000 7f1f7203a740 1 --2- 172.21.15.28:0/528487747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 0x55832612a960 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009900 tx=0x7f1f58005e80 comp rx=0 tx=0).stop 2024-04-25T01:33:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.335+0000 7f1f7203a740 1 -- 172.21.15.28:0/528487747 shutdown_connections 2024-04-25T01:33:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.335+0000 7f1f7203a740 1 --2- 172.21.15.28:0/528487747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 0x55832612a960 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.335+0000 7f1f7203a740 1 --2- 172.21.15.28:0/528487747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 0x558326128030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.335+0000 7f1f7203a740 1 --2- 172.21.15.28:0/528487747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55832611d580 0x55832611d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.335+0000 7f1f7203a740 1 -- 172.21.15.28:0/528487747 >> 172.21.15.28:0/528487747 conn(0x5583261198a0 msgr2=0x558326119c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.335+0000 7f1f7203a740 1 -- 172.21.15.28:0/528487747 shutdown_connections 2024-04-25T01:33:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.335+0000 7f1f7203a740 1 -- 172.21.15.28:0/528487747 wait complete. 2024-04-25T01:33:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.335+0000 7f1f7203a740 1 -- start start 2024-04-25T01:33:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.335+0000 7f1f7203a740 1 Processor -- start 2024-04-25T01:33:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f7203a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55832611d580 0x558326127f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f7203a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 0x5583261251f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f7203a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 0x558326125770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f7203a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55832612b300 con 0x558326128570 2024-04-25T01:33:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f7203a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55832603e050 con 0x55832611d580 2024-04-25T01:33:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f7203a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55832612b480 con 0x55832611dea0 2024-04-25T01:33:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f7203a740 10 client.0 _ll_register_callbacks cb 0x558325d95be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f6d8c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 0x558326125770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f6d8c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 0x558326125770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40126/0 (socket says 172.21.15.28:40126) 2024-04-25T01:33:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f6d8c6640 1 -- 172.21.15.28:0/2816500480 learned_addr learned my addr 172.21.15.28:0/2816500480 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f6d0c5640 1 --2- 172.21.15.28:0/2816500480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55832611d580 0x558326127f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f6c8c4640 1 --2- 172.21.15.28:0/2816500480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 0x5583261251f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f6d8c6640 1 -- 172.21.15.28:0/2816500480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 msgr2=0x5583261251f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f6d8c6640 1 --2- 172.21.15.28:0/2816500480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 0x5583261251f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f6d8c6640 1 -- 172.21.15.28:0/2816500480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55832611d580 msgr2=0x558326127f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f6d8c6640 1 --2- 172.21.15.28:0/2816500480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55832611d580 0x558326127f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f6d8c6640 1 -- 172.21.15.28:0/2816500480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55832600fdc0 con 0x558326128570 2024-04-25T01:33:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f6d0c5640 1 --2- 172.21.15.28:0/2816500480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55832611d580 0x558326127f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f6d8c6640 1 --2- 172.21.15.28:0/2816500480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 0x558326125770 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009340 tx=0x7f1f60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcefc7f750a64309 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.336+0000 7f1f50ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:37.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.337+0000 7f1f50ff9640 1 -- 172.21.15.28:0/2816500480 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f60025070 con 0x558326128570 2024-04-25T01:33:37.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.337+0000 7f1f50ff9640 1 -- 172.21.15.28:0/2816500480 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1f600045c0 con 0x558326128570 2024-04-25T01:33:37.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.337+0000 7f1f7203a740 1 -- 172.21.15.28:0/2816500480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f58004030 con 0x558326128570 2024-04-25T01:33:37.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.337+0000 7f1f50ff9640 1 -- 172.21.15.28:0/2816500480 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1f60031050 con 0x558326128570 2024-04-25T01:33:37.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.337+0000 7f1f7203a740 10 client.15867 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:37.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.337+0000 7f1f7203a740 1 -- 172.21.15.28:0/2816500480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558325d963f0 con 0x558326128570 2024-04-25T01:33:37.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.337+0000 7f1f50ff9640 1 -- 172.21.15.28:0/2816500480 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f58004030 con 0x558326128570 2024-04-25T01:33:37.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.337+0000 7f1f50ff9640 1 -- 172.21.15.28:0/2816500480 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1f60020040 con 0x558326128570 2024-04-25T01:33:37.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.337+0000 7f1f7203a740 10 client.15867 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:37.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.337+0000 7f1f7203a740 10 client.15867 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:33:37.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.337+0000 7f1f7203a740 20 client.15867 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:37.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.337+0000 7f1f7203a740 1 --2- 172.21.15.28:0/2816500480 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x558326138570 0x55832613aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.337+0000 7f1f7203a740 4 client.15867 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:37.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.337+0000 7f1f7203a740 1 -- 172.21.15.28:0/2816500480 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55832608fbb0 con 0x558326138570 2024-04-25T01:33:37.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.338+0000 7f1f6d0c5640 1 --2- 172.21.15.28:0/2816500480 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x558326138570 0x55832613aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.338+0000 7f1f6d0c5640 1 --2- 172.21.15.28:0/2816500480 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x558326138570 0x55832613aa10 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9eee73a4761d099e server_cookie=9676c80346b20165 in_seq=0 out_seq=0 2024-04-25T01:33:37.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.338+0000 7f1f50ff9640 10 client.15867 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:33:37.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.339+0000 7f1f50ff9640 1 -- 172.21.15.28:0/2816500480 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55832608fbb0 con 0x558326138570 2024-04-25T01:33:37.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.339+0000 7f1f50ff9640 10 client.15867 handle_command_reply: tid=0 2024-04-25T01:33:37.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.339+0000 7f1f7203a740 1 client.15867 shutdown 2024-04-25T01:33:37.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.339+0000 7f1f7203a740 1 -- 172.21.15.28:0/2816500480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 msgr2=0x558326125770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.339+0000 7f1f7203a740 1 --2- 172.21.15.28:0/2816500480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 0x558326125770 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009340 tx=0x7f1f60004a40 comp rx=0 tx=0).stop 2024-04-25T01:33:37.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.339+0000 7f1f7203a740 1 -- 172.21.15.28:0/2816500480 shutdown_connections 2024-04-25T01:33:37.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.339+0000 7f1f7203a740 1 --2- 172.21.15.28:0/2816500480 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x558326138570 0x55832613aa10 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.339+0000 7f1f7203a740 1 --2- 172.21.15.28:0/2816500480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 0x558326125770 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.339+0000 7f1f7203a740 1 --2- 172.21.15.28:0/2816500480 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 0x5583261251f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.339+0000 7f1f7203a740 1 --2- 172.21.15.28:0/2816500480 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55832611d580 0x558326127f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.339+0000 7f1f50ff9640 0 client.15867 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:33:37.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.339+0000 7f1f7203a740 1 -- 172.21.15.28:0/2816500480 >> 172.21.15.28:0/2816500480 conn(0x5583261198a0 msgr2=0x558326118b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.339+0000 7f1f7203a740 1 -- 172.21.15.28:0/2816500480 shutdown_connections 2024-04-25T01:33:37.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.339+0000 7f1f7203a740 1 -- 172.21.15.28:0/2816500480 wait complete. 2024-04-25T01:33:37.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.339+0000 7f1f7203a740 20 client.15867 trim_cache size 0 max 16384 2024-04-25T01:33:37.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.354+0000 7f1f7203a740 1 Processor -- start 2024-04-25T01:33:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.354+0000 7f1f7203a740 1 -- start start 2024-04-25T01:33:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.354+0000 7f1f7203a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55832611d580 0x5583261341c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.354+0000 7f1f7203a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 0x558326127a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.354+0000 7f1f7203a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 0x5583261402e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.354+0000 7f1f7203a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f681b0b70 con 0x558326128570 2024-04-25T01:33:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.354+0000 7f1f7203a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1f681b26b0 con 0x55832611d580 2024-04-25T01:33:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.354+0000 7f1f7203a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1f681b1240 con 0x55832611dea0 2024-04-25T01:33:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.354+0000 7f1f6d8c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 0x5583261402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.354+0000 7f1f6d0c5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55832611d580 0x5583261341c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.354+0000 7f1f6d8c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 0x5583261402e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40130/0 (socket says 172.21.15.28:40130) 2024-04-25T01:33:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.354+0000 7f1f6d8c6640 1 -- 172.21.15.28:0/2880220526 learned_addr learned my addr 172.21.15.28:0/2880220526 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.354+0000 7f1f6c8c4640 1 --2- 172.21.15.28:0/2880220526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 0x558326127a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.355+0000 7f1f6d8c6640 1 -- 172.21.15.28:0/2880220526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 msgr2=0x558326127a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.355+0000 7f1f6d8c6640 1 --2- 172.21.15.28:0/2880220526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 0x558326127a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.355+0000 7f1f6d8c6640 1 -- 172.21.15.28:0/2880220526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55832611d580 msgr2=0x5583261341c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.355+0000 7f1f6d8c6640 1 --2- 172.21.15.28:0/2880220526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55832611d580 0x5583261341c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.355+0000 7f1f6d8c6640 1 -- 172.21.15.28:0/2880220526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55832611dcd0 con 0x558326128570 2024-04-25T01:33:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.355+0000 7f1f6d8c6640 1 --2- 172.21.15.28:0/2880220526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 0x5583261402e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1f60011c50 tx=0x7f1f60009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66e3f964e68677fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.355+0000 7f1f52ffd640 1 -- 172.21.15.28:0/2880220526 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6003c070 con 0x558326128570 2024-04-25T01:33:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.355+0000 7f1f52ffd640 1 -- 172.21.15.28:0/2880220526 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f60021400 con 0x558326128570 2024-04-25T01:33:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.355+0000 7f1f52ffd640 1 -- 172.21.15.28:0/2880220526 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f60025050 con 0x558326128570 2024-04-25T01:33:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.356+0000 7f1f7203a740 1 -- 172.21.15.28:0/2880220526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 msgr2=0x5583261402e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.356+0000 7f1f7203a740 1 --2- 172.21.15.28:0/2880220526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 0x5583261402e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1f60011c50 tx=0x7f1f60009790 comp rx=0 tx=0).stop 2024-04-25T01:33:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.356+0000 7f1f7203a740 1 -- 172.21.15.28:0/2880220526 shutdown_connections 2024-04-25T01:33:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.356+0000 7f1f7203a740 1 --2- 172.21.15.28:0/2880220526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558326128570 0x5583261402e0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.356+0000 7f1f7203a740 1 --2- 172.21.15.28:0/2880220526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 0x558326127a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.356+0000 7f1f7203a740 1 --2- 172.21.15.28:0/2880220526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55832611d580 0x5583261341c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.356+0000 7f1f7203a740 1 -- 172.21.15.28:0/2880220526 >> 172.21.15.28:0/2880220526 conn(0x5583261198a0 msgr2=0x558326120860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.356+0000 7f1f7203a740 1 -- 172.21.15.28:0/2880220526 shutdown_connections 2024-04-25T01:33:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.356+0000 7f1f7203a740 1 -- 172.21.15.28:0/2880220526 wait complete. 2024-04-25T01:33:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.356+0000 7f1f7203a740 1 -- start start 2024-04-25T01:33:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.356+0000 7f1f7203a740 1 Processor -- start 2024-04-25T01:33:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.356+0000 7f1f7203a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55832611d580 0x55832612c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.356+0000 7f1f7203a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 0x55832612c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.356+0000 7f1f7203a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558326128570 0x55832612caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.356+0000 7f1f7203a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f681b1fe0 con 0x55832611d580 2024-04-25T01:33:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.356+0000 7f1f7203a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1f681b1910 con 0x558326128570 2024-04-25T01:33:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.356+0000 7f1f7203a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1f681aeb50 con 0x55832611dea0 2024-04-25T01:33:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f7203a740 10 client.0 _ll_register_callbacks cb 0x7f1f600148e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f6d0c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55832611d580 0x55832612c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f6d0c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55832611d580 0x55832612c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40140/0 (socket says 172.21.15.28:40140) 2024-04-25T01:33:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f6d0c5640 1 -- 172.21.15.28:0/2115110033 learned_addr learned my addr 172.21.15.28:0/2115110033 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f6d8c6640 1 --2- 172.21.15.28:0/2115110033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558326128570 0x55832612caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f6c8c4640 1 --2- 172.21.15.28:0/2115110033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 0x55832612c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f6d0c5640 1 -- 172.21.15.28:0/2115110033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 msgr2=0x55832612c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f6d0c5640 1 --2- 172.21.15.28:0/2115110033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 0x55832612c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f6d0c5640 1 -- 172.21.15.28:0/2115110033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558326128570 msgr2=0x55832612caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f6d0c5640 1 --2- 172.21.15.28:0/2115110033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558326128570 0x55832612caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f6d0c5640 1 -- 172.21.15.28:0/2115110033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558325e607d0 con 0x55832611d580 2024-04-25T01:33:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f6c8c4640 1 --2- 172.21.15.28:0/2115110033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 0x55832612c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f6d0c5640 1 --2- 172.21.15.28:0/2115110033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55832611d580 0x55832612c070 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f1f6409b380 tx=0x7f1f6409f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84124607872067af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f50ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f50ff9640 1 -- 172.21.15.28:0/2115110033 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f640b4060 con 0x55832611d580 2024-04-25T01:33:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f50ff9640 1 -- 172.21.15.28:0/2115110033 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1f640996f0 con 0x55832611d580 2024-04-25T01:33:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f7203a740 1 -- 172.21.15.28:0/2115110033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f60021400 con 0x55832611d580 2024-04-25T01:33:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f50ff9640 1 -- 172.21.15.28:0/2115110033 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1f64020040 con 0x55832611d580 2024-04-25T01:33:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f7203a740 10 client.15873 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.357+0000 7f1f7203a740 1 -- 172.21.15.28:0/2115110033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55832619ebe0 con 0x55832611d580 2024-04-25T01:33:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.358+0000 7f1f50ff9640 1 -- 172.21.15.28:0/2115110033 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f60021400 con 0x55832611d580 2024-04-25T01:33:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.358+0000 7f1f50ff9640 1 -- 172.21.15.28:0/2115110033 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1f64097a60 con 0x55832611d580 2024-04-25T01:33:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.358+0000 7f1f7203a740 10 client.15873 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.358+0000 7f1f7203a740 10 client.15873 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:33:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.358+0000 7f1f7203a740 20 client.15873 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.358+0000 7f1f7203a740 1 --2- 172.21.15.28:0/2115110033 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55832619f290 0x5583261a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.358+0000 7f1f7203a740 4 client.15873 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:33:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.358+0000 7f1f7203a740 1 -- 172.21.15.28:0/2115110033 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558326089f00 con 0x55832619f290 2024-04-25T01:33:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.358+0000 7f1f6c8c4640 1 --2- 172.21.15.28:0/2115110033 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55832619f290 0x5583261a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.359+0000 7f1f50ff9640 10 client.15873 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:33:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.359+0000 7f1f6c8c4640 1 --2- 172.21.15.28:0/2115110033 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55832619f290 0x5583261a16b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e5f473868fcd03a server_cookie=27c640f4728c9046 in_seq=0 out_seq=0 2024-04-25T01:33:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.359+0000 7f1f50ff9640 1 -- 172.21.15.28:0/2115110033 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558326089f00 con 0x55832619f290 2024-04-25T01:33:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.359+0000 7f1f50ff9640 10 client.15873 handle_command_reply: tid=0 2024-04-25T01:33:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.359+0000 7f1f7203a740 1 client.15873 shutdown 2024-04-25T01:33:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.359+0000 7f1f7203a740 1 -- 172.21.15.28:0/2115110033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55832611d580 msgr2=0x55832612c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.359+0000 7f1f7203a740 1 --2- 172.21.15.28:0/2115110033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55832611d580 0x55832612c070 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f1f6409b380 tx=0x7f1f6409f040 comp rx=0 tx=0).stop 2024-04-25T01:33:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.359+0000 7f1f7203a740 1 -- 172.21.15.28:0/2115110033 shutdown_connections 2024-04-25T01:33:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.359+0000 7f1f7203a740 1 --2- 172.21.15.28:0/2115110033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558326128570 0x55832612caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.359+0000 7f1f7203a740 1 --2- 172.21.15.28:0/2115110033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55832611dea0 0x55832612c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.359+0000 7f1f7203a740 1 --2- 172.21.15.28:0/2115110033 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55832619f290 0x5583261a16b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.359+0000 7f1f7203a740 1 --2- 172.21.15.28:0/2115110033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55832611d580 0x55832612c070 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.359+0000 7f1f50ff9640 0 client.15873 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:33:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.359+0000 7f1f7203a740 1 -- 172.21.15.28:0/2115110033 >> 172.21.15.28:0/2115110033 conn(0x5583261198a0 msgr2=0x5583261331b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.359+0000 7f1f7203a740 1 -- 172.21.15.28:0/2115110033 shutdown_connections 2024-04-25T01:33:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.359+0000 7f1f7203a740 1 -- 172.21.15.28:0/2115110033 wait complete. 2024-04-25T01:33:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.359+0000 7f1f7203a740 20 client.15873 trim_cache size 0 max 16384 2024-04-25T01:33:37.344 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:33:37.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.360+0000 7f1f6f329640 1 -- 172.21.15.28:0/1868815721 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1f540a2e10 msgr2=0x7f1f540a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.360+0000 7f1f6f329640 1 --2- 172.21.15.28:0/1868815721 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1f540a2e10 0x7f1f540a52d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009fd0 tx=0x7f1f60004ab0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.360+0000 7f1f6f329640 1 -- 172.21.15.28:0/1868815721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f68069dc0 msgr2=0x7f1f68100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.360+0000 7f1f6f329640 1 --2- 172.21.15.28:0/1868815721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f68069dc0 0x7f1f68100670 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1f6400d770 tx=0x7f1f6400dc30 comp rx=0 tx=0).stop 2024-04-25T01:33:37.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.360+0000 7f1f6f329640 1 -- 172.21.15.28:0/1868815721 shutdown_connections 2024-04-25T01:33:37.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.360+0000 7f1f6f329640 1 --2- 172.21.15.28:0/1868815721 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1f540a2e10 0x7f1f540a52d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.360+0000 7f1f6f329640 1 --2- 172.21.15.28:0/1868815721 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1f68107730 0x7f1f681010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.360+0000 7f1f6f329640 1 --2- 172.21.15.28:0/1868815721 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1f6806a780 0x7f1f68100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.360+0000 7f1f6f329640 1 --2- 172.21.15.28:0/1868815721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f68069dc0 0x7f1f68100670 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.360+0000 7f1f6f329640 1 -- 172.21.15.28:0/1868815721 >> 172.21.15.28:0/1868815721 conn(0x7f1f680776a0 msgr2=0x7f1f680fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.360+0000 7f1f6f329640 1 -- 172.21.15.28:0/1868815721 shutdown_connections 2024-04-25T01:33:37.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.360+0000 7f1f6f329640 1 -- 172.21.15.28:0/1868815721 wait complete. 2024-04-25T01:33:37.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:33:37.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.474+0000 7f47aed14640 1 -- 172.21.15.28:0/1503032219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a81066b0 msgr2=0x7f47a8106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.474+0000 7f47aed14640 1 --2- 172.21.15.28:0/1503032219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a81066b0 0x7f47a8106a90 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f47a0009930 tx=0x7f47a002a420 comp rx=0 tx=0).stop 2024-04-25T01:33:37.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.474+0000 7f47aed14640 1 -- 172.21.15.28:0/1503032219 shutdown_connections 2024-04-25T01:33:37.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.474+0000 7f47aed14640 1 --2- 172.21.15.28:0/1503032219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f47a8100fd0 0x7f47a810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.474+0000 7f47aed14640 1 --2- 172.21.15.28:0/1503032219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47a8100610 0x7f47a8100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.474+0000 7f47aed14640 1 --2- 172.21.15.28:0/1503032219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a81066b0 0x7f47a8106a90 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.474+0000 7f47aed14640 1 -- 172.21.15.28:0/1503032219 >> 172.21.15.28:0/1503032219 conn(0x7f47a8077740 msgr2=0x7f47a8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.475+0000 7f47aed14640 1 -- 172.21.15.28:0/1503032219 shutdown_connections 2024-04-25T01:33:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.475+0000 7f47aed14640 1 -- 172.21.15.28:0/1503032219 wait complete. 2024-04-25T01:33:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.475+0000 7f47aed14640 1 Processor -- start 2024-04-25T01:33:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.475+0000 7f47aed14640 1 -- start start 2024-04-25T01:33:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.475+0000 7f47aed14640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47a8100610 0x7f47a8198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.475+0000 7f47aed14640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f47a8100fd0 0x7f47a8199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.475+0000 7f47aed14640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a81066b0 0x7f47a81a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.475+0000 7f47aed14640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47a8110a30 con 0x7f47a81066b0 2024-04-25T01:33:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.475+0000 7f47aed14640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f47a81108b0 con 0x7f47a8100610 2024-04-25T01:33:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.475+0000 7f47aed14640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f47a8110bb0 con 0x7f47a8100fd0 2024-04-25T01:33:37.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.476+0000 7f47ad2b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a81066b0 0x7f47a81a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.476+0000 7f47ad2b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a81066b0 0x7f47a81a3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40148/0 (socket says 172.21.15.28:40148) 2024-04-25T01:33:37.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.476+0000 7f47ad2b1640 1 -- 172.21.15.28:0/1568608285 learned_addr learned my addr 172.21.15.28:0/1568608285 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.476+0000 7f47a7fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f47a8100fd0 0x7f47a8199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.476+0000 7f47acab0640 1 --2- 172.21.15.28:0/1568608285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47a8100610 0x7f47a8198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.476+0000 7f47ad2b1640 1 -- 172.21.15.28:0/1568608285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f47a8100fd0 msgr2=0x7f47a8199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.476+0000 7f47ad2b1640 1 --2- 172.21.15.28:0/1568608285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f47a8100fd0 0x7f47a8199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.476+0000 7f47ad2b1640 1 -- 172.21.15.28:0/1568608285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47a8100610 msgr2=0x7f47a8198e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.476+0000 7f47ad2b1640 1 --2- 172.21.15.28:0/1568608285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47a8100610 0x7f47a8198e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.476+0000 7f47ad2b1640 1 -- 172.21.15.28:0/1568608285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a81a43f0 con 0x7f47a81066b0 2024-04-25T01:33:37.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.476+0000 7f47ad2b1640 1 --2- 172.21.15.28:0/1568608285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a81066b0 0x7f47a81a3eb0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f479000d750 tx=0x7f479000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.476+0000 7f47a5ffb640 1 -- 172.21.15.28:0/1568608285 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4790017070 con 0x7f47a81066b0 2024-04-25T01:33:37.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.476+0000 7f47a5ffb640 1 -- 172.21.15.28:0/1568608285 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4790005500 con 0x7f47a81066b0 2024-04-25T01:33:37.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.476+0000 7f47a5ffb640 1 -- 172.21.15.28:0/1568608285 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4790004d90 con 0x7f47a81066b0 2024-04-25T01:33:37.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.476+0000 7f47aed14640 1 -- 172.21.15.28:0/1568608285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47a81a45c0 con 0x7f47a81066b0 2024-04-25T01:33:37.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.476+0000 7f47aed14640 1 -- 172.21.15.28:0/1568608285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47a81a4a80 con 0x7f47a81066b0 2024-04-25T01:33:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.479+0000 7f47a5ffb640 1 -- 172.21.15.28:0/1568608285 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f4790004f30 con 0x7f47a81066b0 2024-04-25T01:33:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.479+0000 7f47b1a24740 1 Processor -- start 2024-04-25T01:33:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.479+0000 7f47b1a24740 1 -- start start 2024-04-25T01:33:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.479+0000 7f47a5ffb640 1 --2- 172.21.15.28:0/1568608285 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f47940a2e10 0x7f47940a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47a7fff640 1 --2- 172.21.15.28:0/1568608285 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f47940a2e10 0x7f47940a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47b1a24740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122ef580 0x55ec122ef960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47b1a24740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122efea0 0x55ec122fa030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47b1a24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122fa570 0x55ec122fc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47b1a24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ec12254670 con 0x55ec122fa570 2024-04-25T01:33:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47b1a24740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ec121fc970 con 0x55ec122efea0 2024-04-25T01:33:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47b1a24740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ec12250630 con 0x55ec122ef580 2024-04-25T01:33:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47a5ffb640 1 -- 172.21.15.28:0/1568608285 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f47900cf130 con 0x7f47a81066b0 2024-04-25T01:33:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47ad2b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122fa570 0x55ec122fc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47ad2b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122fa570 0x55ec122fc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40154/0 (socket says 172.21.15.28:40154) 2024-04-25T01:33:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47ad2b1640 1 -- 172.21.15.28:0/3107316832 learned_addr learned my addr 172.21.15.28:0/3107316832 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47acab0640 1 --2- 172.21.15.28:0/3107316832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122ef580 0x55ec122ef960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47a7fff640 1 --2- 172.21.15.28:0/3107316832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122efea0 0x55ec122fa030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47a7fff640 1 --2- 172.21.15.28:0/1568608285 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f47940a2e10 0x7f47940a52d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f47a80693d0 tx=0x7f4798003d40 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47ad2b1640 1 -- 172.21.15.28:0/3107316832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122ef580 msgr2=0x55ec122ef960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47ad2b1640 1 --2- 172.21.15.28:0/3107316832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122ef580 0x55ec122ef960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47ad2b1640 1 -- 172.21.15.28:0/3107316832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122efea0 msgr2=0x55ec122fa030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47ad2b1640 1 --2- 172.21.15.28:0/3107316832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122efea0 0x55ec122fa030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47ad2b1640 1 -- 172.21.15.28:0/3107316832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec12220400 con 0x55ec122fa570 2024-04-25T01:33:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47a7fff640 1 --2- 172.21.15.28:0/3107316832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122efea0 0x55ec122fa030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.480+0000 7f47ad2b1640 1 --2- 172.21.15.28:0/3107316832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122fa570 0x55ec122fc960 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f479009af50 tx=0x7f4790000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3927a825d0af69c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.481+0000 7f479effd640 1 -- 172.21.15.28:0/3107316832 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4790020070 con 0x55ec122fa570 2024-04-25T01:33:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.481+0000 7f479effd640 1 -- 172.21.15.28:0/3107316832 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f479009f550 con 0x55ec122fa570 2024-04-25T01:33:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.481+0000 7f479effd640 1 -- 172.21.15.28:0/3107316832 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47900bb0b0 con 0x55ec122fa570 2024-04-25T01:33:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.481+0000 7f47b1a24740 1 -- 172.21.15.28:0/3107316832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122fa570 msgr2=0x55ec122fc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.481+0000 7f47b1a24740 1 --2- 172.21.15.28:0/3107316832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122fa570 0x55ec122fc960 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f479009af50 tx=0x7f4790000bc0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.481+0000 7f47b1a24740 1 -- 172.21.15.28:0/3107316832 shutdown_connections 2024-04-25T01:33:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.481+0000 7f47b1a24740 1 --2- 172.21.15.28:0/3107316832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122fa570 0x55ec122fc960 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.481+0000 7f47b1a24740 1 --2- 172.21.15.28:0/3107316832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122efea0 0x55ec122fa030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.481+0000 7f47b1a24740 1 --2- 172.21.15.28:0/3107316832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122ef580 0x55ec122ef960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.481+0000 7f47b1a24740 1 -- 172.21.15.28:0/3107316832 >> 172.21.15.28:0/3107316832 conn(0x55ec122eb8a0 msgr2=0x55ec122ebc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.481+0000 7f47b1a24740 1 -- 172.21.15.28:0/3107316832 shutdown_connections 2024-04-25T01:33:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.481+0000 7f47b1a24740 1 -- 172.21.15.28:0/3107316832 wait complete. 2024-04-25T01:33:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.482+0000 7f47b1a24740 1 -- start start 2024-04-25T01:33:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.482+0000 7f47b1a24740 1 Processor -- start 2024-04-25T01:33:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.482+0000 7f47b1a24740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122ef580 0x55ec122f9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.482+0000 7f47b1a24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122efea0 0x55ec122f7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.482+0000 7f47b1a24740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122fa570 0x55ec122f7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.482+0000 7f47b1a24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ec122fd300 con 0x55ec122efea0 2024-04-25T01:33:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.482+0000 7f47b1a24740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ec12210050 con 0x55ec122fa570 2024-04-25T01:33:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.482+0000 7f47b1a24740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ec122fd480 con 0x55ec122ef580 2024-04-25T01:33:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.482+0000 7f47b1a24740 10 client.0 _ll_register_callbacks cb 0x55ec11f67be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.482+0000 7f47a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122efea0 0x55ec122f7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.482+0000 7f47a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122efea0 0x55ec122f7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40168/0 (socket says 172.21.15.28:40168) 2024-04-25T01:33:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.482+0000 7f47a7fff640 1 -- 172.21.15.28:0/964986193 learned_addr learned my addr 172.21.15.28:0/964986193 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.483+0000 7f47ad2b1640 1 --2- 172.21.15.28:0/964986193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122fa570 0x55ec122f7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.483+0000 7f47acab0640 1 --2- 172.21.15.28:0/964986193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122ef580 0x55ec122f9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.483+0000 7f47a7fff640 1 -- 172.21.15.28:0/964986193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122ef580 msgr2=0x55ec122f9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.483+0000 7f47a7fff640 1 --2- 172.21.15.28:0/964986193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122ef580 0x55ec122f9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.483+0000 7f47a7fff640 1 -- 172.21.15.28:0/964986193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122fa570 msgr2=0x55ec122f7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.483+0000 7f47a7fff640 1 --2- 172.21.15.28:0/964986193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122fa570 0x55ec122f7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.483+0000 7f47a7fff640 1 -- 172.21.15.28:0/964986193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec121e1dc0 con 0x55ec122efea0 2024-04-25T01:33:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.483+0000 7f47a7fff640 1 --2- 172.21.15.28:0/964986193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122efea0 0x55ec122f7280 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f4798013c80 tx=0x7f4798004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f18b7e95b41ce19d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.483+0000 7f4787fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.483+0000 7f4787fff640 1 -- 172.21.15.28:0/964986193 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4798011960 con 0x55ec122efea0 2024-04-25T01:33:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.483+0000 7f4787fff640 1 -- 172.21.15.28:0/964986193 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f4798011b00 con 0x55ec122efea0 2024-04-25T01:33:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.483+0000 7f47b1a24740 1 -- 172.21.15.28:0/964986193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f479009f550 con 0x55ec122efea0 2024-04-25T01:33:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.483+0000 7f4787fff640 1 -- 172.21.15.28:0/964986193 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4798011d00 con 0x55ec122efea0 2024-04-25T01:33:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.483+0000 7f47b1a24740 10 client.15885 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.483+0000 7f47b1a24740 1 -- 172.21.15.28:0/964986193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ec11f683f0 con 0x55ec122efea0 2024-04-25T01:33:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.483+0000 7f4787fff640 1 -- 172.21.15.28:0/964986193 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f479009f550 con 0x55ec122efea0 2024-04-25T01:33:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.484+0000 7f4787fff640 1 -- 172.21.15.28:0/964986193 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f479800f040 con 0x55ec122efea0 2024-04-25T01:33:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.484+0000 7f47b1a24740 10 client.15885 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.484+0000 7f47b1a24740 10 client.15885 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:33:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.484+0000 7f47b1a24740 20 client.15885 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.484+0000 7f47b1a24740 1 --2- 172.21.15.28:0/964986193 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ec1230a720 0x55ec1230cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.484+0000 7f47b1a24740 4 client.15885 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.484+0000 7f47b1a24740 1 -- 172.21.15.28:0/964986193 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec12261bb0 con 0x55ec1230a720 2024-04-25T01:33:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.484+0000 7f47acab0640 1 --2- 172.21.15.28:0/964986193 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ec1230a720 0x55ec1230cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.485+0000 7f47acab0640 1 --2- 172.21.15.28:0/964986193 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ec1230a720 0x55ec1230cba0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=475d14114230604 server_cookie=525440d8e98db55 in_seq=0 out_seq=0 2024-04-25T01:33:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.485+0000 7f4787fff640 10 client.15885 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:33:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.485+0000 7f4787fff640 1 -- 172.21.15.28:0/964986193 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ec12261bb0 con 0x55ec1230a720 2024-04-25T01:33:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.485+0000 7f4787fff640 10 client.15885 handle_command_reply: tid=0 2024-04-25T01:33:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.485+0000 7f47b1a24740 1 client.15885 shutdown 2024-04-25T01:33:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.485+0000 7f47b1a24740 1 -- 172.21.15.28:0/964986193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122efea0 msgr2=0x55ec122f7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.485+0000 7f47b1a24740 1 --2- 172.21.15.28:0/964986193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122efea0 0x55ec122f7280 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f4798013c80 tx=0x7f4798004ab0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.485+0000 7f47b1a24740 1 -- 172.21.15.28:0/964986193 shutdown_connections 2024-04-25T01:33:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.485+0000 7f47b1a24740 1 --2- 172.21.15.28:0/964986193 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122fa570 0x55ec122f7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.485+0000 7f47b1a24740 1 --2- 172.21.15.28:0/964986193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122efea0 0x55ec122f7280 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.485+0000 7f47b1a24740 1 --2- 172.21.15.28:0/964986193 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ec1230a720 0x55ec1230cba0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.485+0000 7f47b1a24740 1 --2- 172.21.15.28:0/964986193 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122ef580 0x55ec122f9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.485+0000 7f47b1a24740 1 -- 172.21.15.28:0/964986193 >> 172.21.15.28:0/964986193 conn(0x55ec122eb8a0 msgr2=0x55ec122eab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.485+0000 7f4787fff640 0 client.15885 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:33:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.486+0000 7f47b1a24740 1 -- 172.21.15.28:0/964986193 shutdown_connections 2024-04-25T01:33:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.486+0000 7f47b1a24740 1 -- 172.21.15.28:0/964986193 wait complete. 2024-04-25T01:33:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.486+0000 7f47b1a24740 20 client.15885 trim_cache size 0 max 16384 2024-04-25T01:33:37.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.499+0000 7f47b1a24740 1 Processor -- start 2024-04-25T01:33:37.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.499+0000 7f47b1a24740 1 -- start start 2024-04-25T01:33:37.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.499+0000 7f47b1a24740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122ef580 0x55ec122eb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.499+0000 7f47b1a24740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122efea0 0x55ec123061a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.499+0000 7f47b1a24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122fa570 0x55ec123124b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.499+0000 7f47b1a24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47a81aead0 con 0x55ec122fa570 2024-04-25T01:33:37.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.499+0000 7f47b1a24740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f47a81b0610 con 0x55ec122efea0 2024-04-25T01:33:37.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.499+0000 7f47b1a24740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f47a81af1a0 con 0x55ec122ef580 2024-04-25T01:33:37.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.499+0000 7f47ad2b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122fa570 0x55ec123124b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.499+0000 7f47ad2b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122fa570 0x55ec123124b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40180/0 (socket says 172.21.15.28:40180) 2024-04-25T01:33:37.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.499+0000 7f47ad2b1640 1 -- 172.21.15.28:0/3408474528 learned_addr learned my addr 172.21.15.28:0/3408474528 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.499+0000 7f47a7fff640 1 --2- 172.21.15.28:0/3408474528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122efea0 0x55ec123061a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.499+0000 7f47acab0640 1 --2- 172.21.15.28:0/3408474528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122ef580 0x55ec122eb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.499+0000 7f47ad2b1640 1 -- 172.21.15.28:0/3408474528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122ef580 msgr2=0x55ec122eb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.499+0000 7f47ad2b1640 1 --2- 172.21.15.28:0/3408474528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122ef580 0x55ec122eb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.500+0000 7f47ad2b1640 1 -- 172.21.15.28:0/3408474528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122efea0 msgr2=0x55ec123061a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.500+0000 7f47ad2b1640 1 --2- 172.21.15.28:0/3408474528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122efea0 0x55ec123061a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.500+0000 7f47ad2b1640 1 -- 172.21.15.28:0/3408474528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec120327d0 con 0x55ec122fa570 2024-04-25T01:33:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.500+0000 7f47ad2b1640 1 --2- 172.21.15.28:0/3408474528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122fa570 0x55ec123124b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f47900976b0 tx=0x7f4790005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49ccd67626f5b268 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.500+0000 7f479e7fc640 1 -- 172.21.15.28:0/3408474528 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4790020040 con 0x55ec122fa570 2024-04-25T01:33:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.500+0000 7f479e7fc640 1 -- 172.21.15.28:0/3408474528 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4790009c60 con 0x55ec122fa570 2024-04-25T01:33:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.500+0000 7f479e7fc640 1 -- 172.21.15.28:0/3408474528 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4790097a20 con 0x55ec122fa570 2024-04-25T01:33:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.500+0000 7f47b1a24740 1 -- 172.21.15.28:0/3408474528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122fa570 msgr2=0x55ec123124b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.500+0000 7f47b1a24740 1 --2- 172.21.15.28:0/3408474528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122fa570 0x55ec123124b0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f47900976b0 tx=0x7f4790005bb0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.500+0000 7f47b1a24740 1 -- 172.21.15.28:0/3408474528 shutdown_connections 2024-04-25T01:33:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.500+0000 7f47b1a24740 1 --2- 172.21.15.28:0/3408474528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122fa570 0x55ec123124b0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.500+0000 7f47b1a24740 1 --2- 172.21.15.28:0/3408474528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122efea0 0x55ec123061a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.500+0000 7f47b1a24740 1 --2- 172.21.15.28:0/3408474528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122ef580 0x55ec122eb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.500+0000 7f47b1a24740 1 -- 172.21.15.28:0/3408474528 >> 172.21.15.28:0/3408474528 conn(0x55ec122eb8a0 msgr2=0x55ec12305a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.501+0000 7f47b1a24740 1 -- 172.21.15.28:0/3408474528 shutdown_connections 2024-04-25T01:33:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.501+0000 7f47b1a24740 1 -- 172.21.15.28:0/3408474528 wait complete. 2024-04-25T01:33:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.501+0000 7f47b1a24740 1 -- start start 2024-04-25T01:33:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.501+0000 7f47b1a24740 1 Processor -- start 2024-04-25T01:33:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.501+0000 7f47b1a24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122ef580 0x55ec12302a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.501+0000 7f47b1a24740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122efea0 0x55ec122ffe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.501+0000 7f47b1a24740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122fa570 0x55ec12300350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.501+0000 7f47b1a24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47a81aff40 con 0x55ec122ef580 2024-04-25T01:33:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.501+0000 7f47b1a24740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f47a81af870 con 0x55ec122fa570 2024-04-25T01:33:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.501+0000 7f47b1a24740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f47a81acab0 con 0x55ec122efea0 2024-04-25T01:33:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.501+0000 7f47b1a24740 10 client.0 _ll_register_callbacks cb 0x7f47980138d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.501+0000 7f47acab0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122ef580 0x55ec12302a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.501+0000 7f47acab0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122ef580 0x55ec12302a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40194/0 (socket says 172.21.15.28:40194) 2024-04-25T01:33:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.501+0000 7f47acab0640 1 -- 172.21.15.28:0/1409152662 learned_addr learned my addr 172.21.15.28:0/1409152662 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.501+0000 7f47a7fff640 1 --2- 172.21.15.28:0/1409152662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122efea0 0x55ec122ffe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.502+0000 7f47acab0640 1 -- 172.21.15.28:0/1409152662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122efea0 msgr2=0x55ec122ffe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.502+0000 7f47acab0640 1 --2- 172.21.15.28:0/1409152662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122efea0 0x55ec122ffe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.502+0000 7f47acab0640 1 -- 172.21.15.28:0/1409152662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122fa570 msgr2=0x55ec12300350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.502+0000 7f47acab0640 1 --2- 172.21.15.28:0/1409152662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122fa570 0x55ec12300350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.502+0000 7f47acab0640 1 -- 172.21.15.28:0/1409152662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec12300b50 con 0x55ec122ef580 2024-04-25T01:33:37.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.502+0000 7f47acab0640 1 --2- 172.21.15.28:0/1409152662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122ef580 0x55ec12302a30 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f47a0009900 tx=0x7f47a00319e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c04510eeef22fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.502+0000 7f4787fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:37.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.502+0000 7f4787fff640 1 -- 172.21.15.28:0/1409152662 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a004a070 con 0x55ec122ef580 2024-04-25T01:33:37.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.502+0000 7f4787fff640 1 -- 172.21.15.28:0/1409152662 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f47a0004040 con 0x55ec122ef580 2024-04-25T01:33:37.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.502+0000 7f47b1a24740 1 -- 172.21.15.28:0/1409152662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4790009c60 con 0x55ec122ef580 2024-04-25T01:33:37.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.502+0000 7f4787fff640 1 -- 172.21.15.28:0/1409152662 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f47a0056050 con 0x55ec122ef580 2024-04-25T01:33:37.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.502+0000 7f47b1a24740 10 client.15891 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:37.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.502+0000 7f47b1a24740 1 -- 172.21.15.28:0/1409152662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ec12370ce0 con 0x55ec122ef580 2024-04-25T01:33:37.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.502+0000 7f4787fff640 1 -- 172.21.15.28:0/1409152662 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4790009c60 con 0x55ec122ef580 2024-04-25T01:33:37.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.503+0000 7f4787fff640 1 -- 172.21.15.28:0/1409152662 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f47a00463e0 con 0x55ec122ef580 2024-04-25T01:33:37.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.503+0000 7f47b1a24740 10 client.15891 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:37.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.503+0000 7f47b1a24740 10 client.15891 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:33:37.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.503+0000 7f47b1a24740 20 client.15891 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:37.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.503+0000 7f47b1a24740 1 --2- 172.21.15.28:0/1409152662 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ec12371410 0x55ec12373830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.503+0000 7f47b1a24740 4 client.15891 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:33:37.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.503+0000 7f47b1a24740 1 -- 172.21.15.28:0/1409152662 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ec1225bf00 con 0x55ec12371410 2024-04-25T01:33:37.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.503+0000 7f47a7fff640 1 --2- 172.21.15.28:0/1409152662 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ec12371410 0x55ec12373830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.503+0000 7f47a7fff640 1 --2- 172.21.15.28:0/1409152662 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ec12371410 0x55ec12373830 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=36f638a2e22883cb server_cookie=50356d20064a441f in_seq=0 out_seq=0 2024-04-25T01:33:37.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.503+0000 7f4787fff640 10 client.15891 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:33:37.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.504+0000 7f4787fff640 1 -- 172.21.15.28:0/1409152662 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ec1225bf00 con 0x55ec12371410 2024-04-25T01:33:37.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.504+0000 7f4787fff640 10 client.15891 handle_command_reply: tid=0 2024-04-25T01:33:37.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.504+0000 7f47b1a24740 1 client.15891 shutdown 2024-04-25T01:33:37.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.504+0000 7f47b1a24740 1 -- 172.21.15.28:0/1409152662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122ef580 msgr2=0x55ec12302a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.504+0000 7f47b1a24740 1 --2- 172.21.15.28:0/1409152662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122ef580 0x55ec12302a30 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f47a0009900 tx=0x7f47a00319e0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.504+0000 7f47b1a24740 1 -- 172.21.15.28:0/1409152662 shutdown_connections 2024-04-25T01:33:37.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.504+0000 7f47b1a24740 1 --2- 172.21.15.28:0/1409152662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ec122fa570 0x55ec12300350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.504+0000 7f47b1a24740 1 --2- 172.21.15.28:0/1409152662 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ec12371410 0x55ec12373830 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.504+0000 7f47b1a24740 1 --2- 172.21.15.28:0/1409152662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ec122efea0 0x55ec122ffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.504+0000 7f47b1a24740 1 --2- 172.21.15.28:0/1409152662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec122ef580 0x55ec12302a30 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.504+0000 7f4787fff640 0 client.15891 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:33:37.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.504+0000 7f47b1a24740 1 -- 172.21.15.28:0/1409152662 >> 172.21.15.28:0/1409152662 conn(0x55ec122eb8a0 msgr2=0x55ec122ea2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.504+0000 7f47b1a24740 1 -- 172.21.15.28:0/1409152662 shutdown_connections 2024-04-25T01:33:37.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.504+0000 7f47b1a24740 1 -- 172.21.15.28:0/1409152662 wait complete. 2024-04-25T01:33:37.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.504+0000 7f47b1a24740 20 client.15891 trim_cache size 0 max 16384 2024-04-25T01:33:37.488 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:33:37.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.504+0000 7f47aed14640 1 -- 172.21.15.28:0/1568608285 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f47940a2e10 msgr2=0x7f47940a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.504+0000 7f47aed14640 1 --2- 172.21.15.28:0/1568608285 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f47940a2e10 0x7f47940a52d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f47a80693d0 tx=0x7f4798003d40 comp rx=0 tx=0).stop 2024-04-25T01:33:37.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.504+0000 7f47aed14640 1 -- 172.21.15.28:0/1568608285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a81066b0 msgr2=0x7f47a81a3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.504+0000 7f47aed14640 1 --2- 172.21.15.28:0/1568608285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a81066b0 0x7f47a81a3eb0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f479000d750 tx=0x7f479000dc10 comp rx=0 tx=0).stop 2024-04-25T01:33:37.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.505+0000 7f47aed14640 1 -- 172.21.15.28:0/1568608285 shutdown_connections 2024-04-25T01:33:37.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.505+0000 7f47aed14640 1 --2- 172.21.15.28:0/1568608285 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f47940a2e10 0x7f47940a52d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.505+0000 7f47aed14640 1 --2- 172.21.15.28:0/1568608285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a81066b0 0x7f47a81a3eb0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.505+0000 7f47aed14640 1 --2- 172.21.15.28:0/1568608285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f47a8100fd0 0x7f47a8199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.505+0000 7f47aed14640 1 --2- 172.21.15.28:0/1568608285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f47a8100610 0x7f47a8198e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.505+0000 7f47aed14640 1 -- 172.21.15.28:0/1568608285 >> 172.21.15.28:0/1568608285 conn(0x7f47a8077740 msgr2=0x7f47a80ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.505+0000 7f47aed14640 1 -- 172.21.15.28:0/1568608285 shutdown_connections 2024-04-25T01:33:37.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.505+0000 7f47aed14640 1 -- 172.21.15.28:0/1568608285 wait complete. 2024-04-25T01:33:37.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:33:37.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.624+0000 7ff8892bd640 1 -- 172.21.15.28:0/3246744800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff884106630 msgr2=0x7ff884106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.624+0000 7ff8892bd640 1 --2- 172.21.15.28:0/3246744800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff884106630 0x7ff884106a10 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff874009cd0 tx=0x7ff87402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.624+0000 7ff8892bd640 1 -- 172.21.15.28:0/3246744800 shutdown_connections 2024-04-25T01:33:37.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.624+0000 7ff8892bd640 1 --2- 172.21.15.28:0/3246744800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff884100fe0 0x7ff88410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.624+0000 7ff8892bd640 1 --2- 172.21.15.28:0/3246744800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff884100620 0x7ff884100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.624+0000 7ff8892bd640 1 --2- 172.21.15.28:0/3246744800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff884106630 0x7ff884106a10 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.624+0000 7ff8892bd640 1 -- 172.21.15.28:0/3246744800 >> 172.21.15.28:0/3246744800 conn(0x7ff8840776a0 msgr2=0x7ff884077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.624+0000 7ff8892bd640 1 -- 172.21.15.28:0/3246744800 shutdown_connections 2024-04-25T01:33:37.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.624+0000 7ff8892bd640 1 -- 172.21.15.28:0/3246744800 wait complete. 2024-04-25T01:33:37.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.625+0000 7ff8892bd640 1 Processor -- start 2024-04-25T01:33:37.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.625+0000 7ff8892bd640 1 -- start start 2024-04-25T01:33:37.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.625+0000 7ff8892bd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff884100620 0x7ff884198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.625+0000 7ff8892bd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff884100fe0 0x7ff884199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.625+0000 7ff8892bd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff884106630 0x7ff884199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.625+0000 7ff8892bd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff884110a90 con 0x7ff884100fe0 2024-04-25T01:33:37.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.625+0000 7ff8892bd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff884110910 con 0x7ff884106630 2024-04-25T01:33:37.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.625+0000 7ff8892bd640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff884110c10 con 0x7ff884100620 2024-04-25T01:33:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.625+0000 7ff8827fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff884100fe0 0x7ff884199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.625+0000 7ff8827fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff884100fe0 0x7ff884199490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40218/0 (socket says 172.21.15.28:40218) 2024-04-25T01:33:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.625+0000 7ff8827fc640 1 -- 172.21.15.28:0/2909773523 learned_addr learned my addr 172.21.15.28:0/2909773523 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.625+0000 7ff8837fe640 1 --2- 172.21.15.28:0/2909773523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff884106630 0x7ff884199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.625+0000 7ff882ffd640 1 --2- 172.21.15.28:0/2909773523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff884100620 0x7ff884198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.625+0000 7ff8827fc640 1 -- 172.21.15.28:0/2909773523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff884100620 msgr2=0x7ff884198f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.625+0000 7ff8827fc640 1 --2- 172.21.15.28:0/2909773523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff884100620 0x7ff884198f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.625+0000 7ff8827fc640 1 -- 172.21.15.28:0/2909773523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff884106630 msgr2=0x7ff884199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.625+0000 7ff8827fc640 1 --2- 172.21.15.28:0/2909773523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff884106630 0x7ff884199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.625+0000 7ff8827fc640 1 -- 172.21.15.28:0/2909773523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8841a4350 con 0x7ff884100fe0 2024-04-25T01:33:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.626+0000 7ff8827fc640 1 --2- 172.21.15.28:0/2909773523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff884100fe0 0x7ff884199490 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ff87800d950 tx=0x7ff87800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.626+0000 7ff86bfff640 1 -- 172.21.15.28:0/2909773523 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff878017070 con 0x7ff884100fe0 2024-04-25T01:33:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.626+0000 7ff86bfff640 1 -- 172.21.15.28:0/2909773523 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff878009c30 con 0x7ff884100fe0 2024-04-25T01:33:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.626+0000 7ff8892bd640 1 -- 172.21.15.28:0/2909773523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8841a4640 con 0x7ff884100fe0 2024-04-25T01:33:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.626+0000 7ff86bfff640 1 -- 172.21.15.28:0/2909773523 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87800f6b0 con 0x7ff884100fe0 2024-04-25T01:33:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.626+0000 7ff8892bd640 1 -- 172.21.15.28:0/2909773523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8841a4be0 con 0x7ff884100fe0 2024-04-25T01:33:37.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.629+0000 7ff86bfff640 1 -- 172.21.15.28:0/2909773523 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7ff8780109d0 con 0x7ff884100fe0 2024-04-25T01:33:37.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.629+0000 7ff88bfd1740 1 Processor -- start 2024-04-25T01:33:37.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.629+0000 7ff88bfd1740 1 -- start start 2024-04-25T01:33:37.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.629+0000 7ff88bfd1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 0x556652227960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.629+0000 7ff88bfd1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652227ea0 0x556652232030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.629+0000 7ff88bfd1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556652232570 0x556652234960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.629+0000 7ff88bfd1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55665218c670 con 0x556652227580 2024-04-25T01:33:37.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.629+0000 7ff88bfd1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556652134970 con 0x556652232570 2024-04-25T01:33:37.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.629+0000 7ff88bfd1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x556652188630 con 0x556652227ea0 2024-04-25T01:33:37.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.629+0000 7ff8837fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 0x556652227960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.629+0000 7ff8837fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 0x556652227960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40230/0 (socket says 172.21.15.28:40230) 2024-04-25T01:33:37.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.629+0000 7ff8837fe640 1 -- 172.21.15.28:0/2730938247 learned_addr learned my addr 172.21.15.28:0/2730938247 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.629+0000 7ff8827fc640 1 --2- 172.21.15.28:0/2730938247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556652232570 0x556652234960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.630+0000 7ff86bfff640 1 --2- 172.21.15.28:0/2909773523 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff86c0a2e10 0x7ff86c0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.630+0000 7ff8837fe640 1 -- 172.21.15.28:0/2730938247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652227ea0 msgr2=0x556652232030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.630+0000 7ff8837fe640 1 --2- 172.21.15.28:0/2730938247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652227ea0 0x556652232030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.630+0000 7ff8837fe640 1 -- 172.21.15.28:0/2730938247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556652232570 msgr2=0x556652234960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.630+0000 7ff8837fe640 1 --2- 172.21.15.28:0/2730938247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556652232570 0x556652234960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.630+0000 7ff8837fe640 1 -- 172.21.15.28:0/2730938247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556652158400 con 0x556652227580 2024-04-25T01:33:37.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.630+0000 7ff86bfff640 1 -- 172.21.15.28:0/2909773523 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff8780cf520 con 0x7ff884100fe0 2024-04-25T01:33:37.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.630+0000 7ff8837fe640 1 --2- 172.21.15.28:0/2909773523 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff86c0a2e10 0x7ff86c0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.630+0000 7ff8837fe640 1 --2- 172.21.15.28:0/2730938247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 0x556652227960 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ff87c00b6b0 tx=0x7ff87c004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2990235f45c48464 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.630+0000 7ff8697fa640 1 -- 172.21.15.28:0/2730938247 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c003d60 con 0x556652227580 2024-04-25T01:33:37.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.630+0000 7ff8697fa640 1 -- 172.21.15.28:0/2730938247 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff87c015070 con 0x556652227580 2024-04-25T01:33:37.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.630+0000 7ff8697fa640 1 -- 172.21.15.28:0/2730938247 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c01bc10 con 0x556652227580 2024-04-25T01:33:37.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.631+0000 7ff88bfd1740 1 -- 172.21.15.28:0/2730938247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 msgr2=0x556652227960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.631+0000 7ff8837fe640 1 --2- 172.21.15.28:0/2909773523 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff86c0a2e10 0x7ff86c0a52d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff87c01c000 tx=0x7ff87c010810 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.631+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/2730938247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 0x556652227960 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ff87c00b6b0 tx=0x7ff87c004610 comp rx=0 tx=0).stop 2024-04-25T01:33:37.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.631+0000 7ff88bfd1740 1 -- 172.21.15.28:0/2730938247 shutdown_connections 2024-04-25T01:33:37.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.631+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/2730938247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556652232570 0x556652234960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.631+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/2730938247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652227ea0 0x556652232030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.631+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/2730938247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 0x556652227960 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.631+0000 7ff88bfd1740 1 -- 172.21.15.28:0/2730938247 >> 172.21.15.28:0/2730938247 conn(0x5566522238a0 msgr2=0x556652223c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.631+0000 7ff88bfd1740 1 -- 172.21.15.28:0/2730938247 shutdown_connections 2024-04-25T01:33:37.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.631+0000 7ff88bfd1740 1 -- 172.21.15.28:0/2730938247 wait complete. 2024-04-25T01:33:37.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.631+0000 7ff88bfd1740 1 -- start start 2024-04-25T01:33:37.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.631+0000 7ff88bfd1740 1 Processor -- start 2024-04-25T01:33:37.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.631+0000 7ff88bfd1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 0x556652231dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.631+0000 7ff88bfd1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652227ea0 0x55665222f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.631+0000 7ff88bfd1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556652232570 0x55665222f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.631+0000 7ff88bfd1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556652235360 con 0x556652227580 2024-04-25T01:33:37.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.631+0000 7ff88bfd1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556652148050 con 0x556652232570 2024-04-25T01:33:37.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.631+0000 7ff88bfd1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5566522354e0 con 0x556652227ea0 2024-04-25T01:33:37.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.631+0000 7ff88bfd1740 10 client.0 _ll_register_callbacks cb 0x556651e9fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:37.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.632+0000 7ff882ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 0x556652231dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.632+0000 7ff882ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 0x556652231dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40244/0 (socket says 172.21.15.28:40244) 2024-04-25T01:33:37.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.632+0000 7ff882ffd640 1 -- 172.21.15.28:0/3763160730 learned_addr learned my addr 172.21.15.28:0/3763160730 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.632+0000 7ff8827fc640 1 --2- 172.21.15.28:0/3763160730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652227ea0 0x55665222f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.632+0000 7ff882ffd640 1 -- 172.21.15.28:0/3763160730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652227ea0 msgr2=0x55665222f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.632+0000 7ff882ffd640 1 --2- 172.21.15.28:0/3763160730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652227ea0 0x55665222f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.632+0000 7ff882ffd640 1 -- 172.21.15.28:0/3763160730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556652232570 msgr2=0x55665222f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.632+0000 7ff882ffd640 1 --2- 172.21.15.28:0/3763160730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556652232570 0x55665222f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.632+0000 7ff882ffd640 1 -- 172.21.15.28:0/3763160730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556652119dc0 con 0x556652227580 2024-04-25T01:33:37.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.632+0000 7ff882ffd640 1 --2- 172.21.15.28:0/3763160730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 0x556652231dc0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ff874009430 tx=0x7ff874031af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd14edeffdd1fa3a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.632+0000 7ff866ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:37.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.632+0000 7ff866ff5640 1 -- 172.21.15.28:0/3763160730 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff874003b70 con 0x556652227580 2024-04-25T01:33:37.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.632+0000 7ff866ff5640 1 -- 172.21.15.28:0/3763160730 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff874003d10 con 0x556652227580 2024-04-25T01:33:37.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.632+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3763160730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff87c015070 con 0x556652227580 2024-04-25T01:33:37.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.632+0000 7ff866ff5640 1 -- 172.21.15.28:0/3763160730 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff87404e050 con 0x556652227580 2024-04-25T01:33:37.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.632+0000 7ff88bfd1740 10 client.15903 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:37.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.632+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3763160730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556651ea03f0 con 0x556652227580 2024-04-25T01:33:37.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.633+0000 7ff866ff5640 1 -- 172.21.15.28:0/3763160730 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff87c015070 con 0x556652227580 2024-04-25T01:33:37.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.633+0000 7ff866ff5640 1 -- 172.21.15.28:0/3763160730 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff874030070 con 0x556652227580 2024-04-25T01:33:37.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.633+0000 7ff88bfd1740 10 client.15903 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:37.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.633+0000 7ff88bfd1740 10 client.15903 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:33:37.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.633+0000 7ff88bfd1740 20 client.15903 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:37.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.633+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/3763160730 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5566522424a0 0x5566522448d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.633+0000 7ff88bfd1740 4 client.15903 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:37.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.633+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3763160730 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556652199bb0 con 0x5566522424a0 2024-04-25T01:33:37.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.633+0000 7ff8827fc640 1 --2- 172.21.15.28:0/3763160730 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5566522424a0 0x5566522448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.634+0000 7ff8827fc640 1 --2- 172.21.15.28:0/3763160730 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5566522424a0 0x5566522448d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=acfea653d0bbceec server_cookie=d182ba557527049c in_seq=0 out_seq=0 2024-04-25T01:33:37.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.634+0000 7ff866ff5640 10 client.15903 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:33:37.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.634+0000 7ff866ff5640 1 -- 172.21.15.28:0/3763160730 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556652199bb0 con 0x5566522424a0 2024-04-25T01:33:37.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.634+0000 7ff866ff5640 10 client.15903 handle_command_reply: tid=0 2024-04-25T01:33:37.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.634+0000 7ff88bfd1740 1 client.15903 shutdown 2024-04-25T01:33:37.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.634+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3763160730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 msgr2=0x556652231dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.634+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/3763160730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 0x556652231dc0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ff874009430 tx=0x7ff874031af0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.634+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3763160730 shutdown_connections 2024-04-25T01:33:37.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.634+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/3763160730 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5566522424a0 0x5566522448d0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.634+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/3763160730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556652232570 0x55665222f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.634+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/3763160730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652227ea0 0x55665222f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.634+0000 7ff866ff5640 0 client.15903 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:33:37.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.634+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/3763160730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 0x556652231dc0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.635+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3763160730 >> 172.21.15.28:0/3763160730 conn(0x5566522238a0 msgr2=0x556652222b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.635+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3763160730 shutdown_connections 2024-04-25T01:33:37.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.635+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3763160730 wait complete. 2024-04-25T01:33:37.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.635+0000 7ff88bfd1740 20 client.15903 trim_cache size 0 max 16384 2024-04-25T01:33:37.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.648+0000 7ff88bfd1740 1 Processor -- start 2024-04-25T01:33:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.648+0000 7ff88bfd1740 1 -- start start 2024-04-25T01:33:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.648+0000 7ff88bfd1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 0x5566522318f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.648+0000 7ff88bfd1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652227ea0 0x55665223a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.648+0000 7ff88bfd1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556652232570 0x55665223ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.648+0000 7ff88bfd1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8841aeb30 con 0x556652227580 2024-04-25T01:33:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.648+0000 7ff88bfd1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff8841b0670 con 0x556652232570 2024-04-25T01:33:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.648+0000 7ff88bfd1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff8841af200 con 0x556652227ea0 2024-04-25T01:33:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.648+0000 7ff882ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 0x5566522318f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.648+0000 7ff882ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 0x5566522318f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40252/0 (socket says 172.21.15.28:40252) 2024-04-25T01:33:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.648+0000 7ff882ffd640 1 -- 172.21.15.28:0/3905207322 learned_addr learned my addr 172.21.15.28:0/3905207322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.648+0000 7ff882ffd640 1 -- 172.21.15.28:0/3905207322 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652227ea0 msgr2=0x55665223a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.648+0000 7ff882ffd640 1 --2- 172.21.15.28:0/3905207322 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652227ea0 0x55665223a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.648+0000 7ff882ffd640 1 -- 172.21.15.28:0/3905207322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556652232570 msgr2=0x55665223ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.648+0000 7ff882ffd640 1 --2- 172.21.15.28:0/3905207322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556652232570 0x55665223ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.648+0000 7ff882ffd640 1 -- 172.21.15.28:0/3905207322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556651f6a7d0 con 0x556652227580 2024-04-25T01:33:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.649+0000 7ff882ffd640 1 --2- 172.21.15.28:0/3905207322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 0x5566522318f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7ff874000f80 tx=0x7ff874009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7a218f247a2bb46 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.649+0000 7ff868ff9640 1 -- 172.21.15.28:0/3905207322 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff874057070 con 0x556652227580 2024-04-25T01:33:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.649+0000 7ff868ff9640 1 -- 172.21.15.28:0/3905207322 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff874036040 con 0x556652227580 2024-04-25T01:33:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.649+0000 7ff868ff9640 1 -- 172.21.15.28:0/3905207322 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff874052b70 con 0x556652227580 2024-04-25T01:33:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.649+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3905207322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 msgr2=0x5566522318f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.649+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/3905207322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 0x5566522318f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7ff874000f80 tx=0x7ff874009a10 comp rx=0 tx=0).stop 2024-04-25T01:33:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.649+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3905207322 shutdown_connections 2024-04-25T01:33:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.649+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/3905207322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556652232570 0x55665223ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.649+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/3905207322 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652227ea0 0x55665223a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.649+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/3905207322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227580 0x5566522318f0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.649+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3905207322 >> 172.21.15.28:0/3905207322 conn(0x5566522238a0 msgr2=0x55665223d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.649+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3905207322 shutdown_connections 2024-04-25T01:33:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.649+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3905207322 wait complete. 2024-04-25T01:33:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.650+0000 7ff88bfd1740 1 -- start start 2024-04-25T01:33:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.650+0000 7ff88bfd1740 1 Processor -- start 2024-04-25T01:33:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.650+0000 7ff88bfd1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556652227580 0x556652238530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.650+0000 7ff88bfd1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227ea0 0x556652238a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.650+0000 7ff88bfd1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652232570 0x5566522a7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.650+0000 7ff88bfd1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8841affa0 con 0x556652227ea0 2024-04-25T01:33:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.650+0000 7ff88bfd1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff8841af8d0 con 0x556652227580 2024-04-25T01:33:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.650+0000 7ff88bfd1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff8841acb10 con 0x556652232570 2024-04-25T01:33:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.650+0000 7ff88bfd1740 10 client.0 _ll_register_callbacks cb 0x7ff874009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.650+0000 7ff8827fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227ea0 0x556652238a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.650+0000 7ff8827fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227ea0 0x556652238a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40254/0 (socket says 172.21.15.28:40254) 2024-04-25T01:33:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.650+0000 7ff8827fc640 1 -- 172.21.15.28:0/3674235020 learned_addr learned my addr 172.21.15.28:0/3674235020 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.650+0000 7ff882ffd640 1 --2- 172.21.15.28:0/3674235020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556652227580 0x556652238530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.650+0000 7ff8827fc640 1 -- 172.21.15.28:0/3674235020 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652232570 msgr2=0x5566522a7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.650+0000 7ff8827fc640 1 --2- 172.21.15.28:0/3674235020 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652232570 0x5566522a7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.650+0000 7ff8827fc640 1 -- 172.21.15.28:0/3674235020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556652227580 msgr2=0x556652238530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.650+0000 7ff8827fc640 1 --2- 172.21.15.28:0/3674235020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556652227580 0x556652238530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.650+0000 7ff8827fc640 1 -- 172.21.15.28:0/3674235020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566522a7f60 con 0x556652227ea0 2024-04-25T01:33:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.650+0000 7ff882ffd640 1 --2- 172.21.15.28:0/3674235020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556652227580 0x556652238530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.651+0000 7ff8827fc640 1 --2- 172.21.15.28:0/3674235020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227ea0 0x556652238a70 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7ff8780cc950 tx=0x7ff878010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aaac827b9cbc793 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.651+0000 7ff866ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.651+0000 7ff866ff5640 1 -- 172.21.15.28:0/3674235020 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff878020040 con 0x556652227ea0 2024-04-25T01:33:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.651+0000 7ff866ff5640 1 -- 172.21.15.28:0/3674235020 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff878099890 con 0x556652227ea0 2024-04-25T01:33:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.651+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3674235020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff874036040 con 0x556652227ea0 2024-04-25T01:33:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.651+0000 7ff866ff5640 1 -- 172.21.15.28:0/3674235020 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff87809aef0 con 0x556652227ea0 2024-04-25T01:33:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.651+0000 7ff88bfd1740 10 client.15909 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.651+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3674235020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5566522a9120 con 0x556652227ea0 2024-04-25T01:33:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.651+0000 7ff866ff5640 1 -- 172.21.15.28:0/3674235020 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff874036040 con 0x556652227ea0 2024-04-25T01:33:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.651+0000 7ff866ff5640 1 -- 172.21.15.28:0/3674235020 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff878097c00 con 0x556652227ea0 2024-04-25T01:33:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.651+0000 7ff88bfd1740 10 client.15909 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.651+0000 7ff88bfd1740 10 client.15909 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:33:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.651+0000 7ff88bfd1740 20 client.15909 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.651+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/3674235020 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5566522a9880 0x5566522abc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.651+0000 7ff88bfd1740 4 client.15909 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:33:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.651+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3674235020 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556652193f00 con 0x5566522a9880 2024-04-25T01:33:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.652+0000 7ff882ffd640 1 --2- 172.21.15.28:0/3674235020 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5566522a9880 0x5566522abc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.652+0000 7ff882ffd640 1 --2- 172.21.15.28:0/3674235020 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5566522a9880 0x5566522abc70 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fd08f36db92b7d7c server_cookie=b204f4477e71f6f6 in_seq=0 out_seq=0 2024-04-25T01:33:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.652+0000 7ff866ff5640 10 client.15909 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:33:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.652+0000 7ff866ff5640 1 -- 172.21.15.28:0/3674235020 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556652193f00 con 0x5566522a9880 2024-04-25T01:33:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.652+0000 7ff866ff5640 10 client.15909 handle_command_reply: tid=0 2024-04-25T01:33:37.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.652+0000 7ff88bfd1740 1 client.15909 shutdown 2024-04-25T01:33:37.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.652+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3674235020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227ea0 msgr2=0x556652238a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.652+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/3674235020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227ea0 0x556652238a70 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7ff8780cc950 tx=0x7ff878010d70 comp rx=0 tx=0).stop 2024-04-25T01:33:37.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.652+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3674235020 shutdown_connections 2024-04-25T01:33:37.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.652+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/3674235020 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5566522a9880 0x5566522abc70 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.652+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/3674235020 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556652232570 0x5566522a7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.652+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/3674235020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556652227ea0 0x556652238a70 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.652+0000 7ff866ff5640 0 client.15909 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:33:37.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.652+0000 7ff88bfd1740 1 --2- 172.21.15.28:0/3674235020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556652227580 0x556652238530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.652+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3674235020 >> 172.21.15.28:0/3674235020 conn(0x5566522238a0 msgr2=0x55665223daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.652+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3674235020 shutdown_connections 2024-04-25T01:33:37.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.652+0000 7ff88bfd1740 1 -- 172.21.15.28:0/3674235020 wait complete. 2024-04-25T01:33:37.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.652+0000 7ff88bfd1740 20 client.15909 trim_cache size 0 max 16384 2024-04-25T01:33:37.639 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:33:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.653+0000 7ff8892bd640 1 -- 172.21.15.28:0/2909773523 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff86c0a2e10 msgr2=0x7ff86c0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.653+0000 7ff8892bd640 1 --2- 172.21.15.28:0/2909773523 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff86c0a2e10 0x7ff86c0a52d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff87c01c000 tx=0x7ff87c010810 comp rx=0 tx=0).stop 2024-04-25T01:33:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.653+0000 7ff8892bd640 1 -- 172.21.15.28:0/2909773523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff884100fe0 msgr2=0x7ff884199490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.653+0000 7ff8892bd640 1 --2- 172.21.15.28:0/2909773523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff884100fe0 0x7ff884199490 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ff87800d950 tx=0x7ff87800de10 comp rx=0 tx=0).stop 2024-04-25T01:33:37.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.653+0000 7ff8892bd640 1 -- 172.21.15.28:0/2909773523 shutdown_connections 2024-04-25T01:33:37.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.653+0000 7ff8892bd640 1 --2- 172.21.15.28:0/2909773523 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff86c0a2e10 0x7ff86c0a52d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.653+0000 7ff8892bd640 1 --2- 172.21.15.28:0/2909773523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff884106630 0x7ff884199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.653+0000 7ff8892bd640 1 --2- 172.21.15.28:0/2909773523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff884100fe0 0x7ff884199490 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.653+0000 7ff8892bd640 1 --2- 172.21.15.28:0/2909773523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff884100620 0x7ff884198f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.653+0000 7ff8892bd640 1 -- 172.21.15.28:0/2909773523 >> 172.21.15.28:0/2909773523 conn(0x7ff8840776a0 msgr2=0x7ff8841031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.653+0000 7ff8892bd640 1 -- 172.21.15.28:0/2909773523 shutdown_connections 2024-04-25T01:33:37.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.653+0000 7ff8892bd640 1 -- 172.21.15.28:0/2909773523 wait complete. 2024-04-25T01:33:37.643 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:33:37.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.777+0000 7f2ac03b7640 1 -- 172.21.15.28:0/3018610487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ab8100620 msgr2=0x7f2ab8100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.777+0000 7f2ac03b7640 1 --2- 172.21.15.28:0/3018610487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ab8100620 0x7f2ab8100aa0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4009930 tx=0x7f2ab402a420 comp rx=0 tx=0).stop 2024-04-25T01:33:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.777+0000 7f2ac03b7640 1 -- 172.21.15.28:0/3018610487 shutdown_connections 2024-04-25T01:33:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.777+0000 7f2ac03b7640 1 --2- 172.21.15.28:0/3018610487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ab8100fe0 0x7f2ab810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.777+0000 7f2ac03b7640 1 --2- 172.21.15.28:0/3018610487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ab8100620 0x7f2ab8100aa0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.777+0000 7f2ac03b7640 1 --2- 172.21.15.28:0/3018610487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ab8106630 0x7f2ab8106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.777+0000 7f2ac03b7640 1 -- 172.21.15.28:0/3018610487 >> 172.21.15.28:0/3018610487 conn(0x7f2ab80776a0 msgr2=0x7f2ab8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.777+0000 7f2ac03b7640 1 -- 172.21.15.28:0/3018610487 shutdown_connections 2024-04-25T01:33:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.777+0000 7f2ac03b7640 1 -- 172.21.15.28:0/3018610487 wait complete. 2024-04-25T01:33:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.778+0000 7f2ac03b7640 1 Processor -- start 2024-04-25T01:33:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.778+0000 7f2ac03b7640 1 -- start start 2024-04-25T01:33:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.778+0000 7f2ac03b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ab8100620 0x7f2ab8198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.778+0000 7f2ac03b7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ab8100fe0 0x7f2ab8198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.778+0000 7f2ac03b7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ab8106630 0x7f2ab819c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.778+0000 7f2ac03b7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab8110a40 con 0x7f2ab8100620 2024-04-25T01:33:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.778+0000 7f2ac03b7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab81108c0 con 0x7f2ab8100fe0 2024-04-25T01:33:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.778+0000 7f2ac03b7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab8110bc0 con 0x7f2ab8106630 2024-04-25T01:33:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.778+0000 7f2abe153640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ab8100620 0x7f2ab8198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.778+0000 7f2abe153640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ab8100620 0x7f2ab8198150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40272/0 (socket says 172.21.15.28:40272) 2024-04-25T01:33:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.778+0000 7f2abe153640 1 -- 172.21.15.28:0/2313503959 learned_addr learned my addr 172.21.15.28:0/2313503959 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.778+0000 7f2abd952640 1 --2- 172.21.15.28:0/2313503959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ab8100fe0 0x7f2ab8198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.778+0000 7f2abe153640 1 -- 172.21.15.28:0/2313503959 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ab8106630 msgr2=0x7f2ab819c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.778+0000 7f2abe153640 1 --2- 172.21.15.28:0/2313503959 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ab8106630 0x7f2ab819c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.778+0000 7f2abe153640 1 -- 172.21.15.28:0/2313503959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ab8100fe0 msgr2=0x7f2ab8198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.778+0000 7f2abe153640 1 --2- 172.21.15.28:0/2313503959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ab8100fe0 0x7f2ab8198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.778+0000 7f2abe153640 1 -- 172.21.15.28:0/2313503959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab819d0f0 con 0x7f2ab8100620 2024-04-25T01:33:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.779+0000 7f2abe153640 1 --2- 172.21.15.28:0/2313503959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ab8100620 0x7f2ab8198150 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800c970 tx=0x7f2aa800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.779+0000 7f2aaf7fe640 1 -- 172.21.15.28:0/2313503959 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa8007bb0 con 0x7f2ab8100620 2024-04-25T01:33:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.779+0000 7f2ac03b7640 1 -- 172.21.15.28:0/2313503959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab819d3e0 con 0x7f2ab8100620 2024-04-25T01:33:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.779+0000 7f2aaf7fe640 1 -- 172.21.15.28:0/2313503959 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2aa8007d50 con 0x7f2ab8100620 2024-04-25T01:33:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.779+0000 7f2aaf7fe640 1 -- 172.21.15.28:0/2313503959 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa800f450 con 0x7f2ab8100620 2024-04-25T01:33:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.779+0000 7f2ac03b7640 1 -- 172.21.15.28:0/2313503959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab81a4c70 con 0x7f2ab8100620 2024-04-25T01:33:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.781+0000 7f2aaf7fe640 1 -- 172.21.15.28:0/2313503959 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f2aa8005020 con 0x7f2ab8100620 2024-04-25T01:33:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.781+0000 7f2ac31a1740 1 Processor -- start 2024-04-25T01:33:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2ac31a1740 1 -- start start 2024-04-25T01:33:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2ac31a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ed990 0x555e3a4edd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2ac31a1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4ee2b0 0x555e3a4f8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2ac31a1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4f8980 0x555e3a4fad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2ac31a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555e3a44e670 con 0x555e3a4ed990 2024-04-25T01:33:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2ac31a1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555e3a3f6970 con 0x555e3a4ee2b0 2024-04-25T01:33:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2ac31a1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x555e3a44a630 con 0x555e3a4f8980 2024-04-25T01:33:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2abe954640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ed990 0x555e3a4edd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2abe954640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ed990 0x555e3a4edd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40286/0 (socket says 172.21.15.28:40286) 2024-04-25T01:33:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2abe954640 1 -- 172.21.15.28:0/51182988 learned_addr learned my addr 172.21.15.28:0/51182988 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2aaf7fe640 1 --2- 172.21.15.28:0/2313503959 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f2aa40a2d90 0x7f2aa40a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2abd952640 1 --2- 172.21.15.28:0/51182988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4f8980 0x555e3a4fad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2abe954640 1 --2- 172.21.15.28:0/2313503959 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f2aa40a2d90 0x7f2aa40a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2aaf7fe640 1 -- 172.21.15.28:0/2313503959 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f2aa80cf720 con 0x7f2ab8100620 2024-04-25T01:33:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2abe153640 1 --2- 172.21.15.28:0/51182988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4ee2b0 0x555e3a4f8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2abe954640 1 -- 172.21.15.28:0/51182988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4f8980 msgr2=0x555e3a4fad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2abe954640 1 --2- 172.21.15.28:0/51182988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4f8980 0x555e3a4fad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2abe954640 1 -- 172.21.15.28:0/51182988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4ee2b0 msgr2=0x555e3a4f8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2abe954640 1 --2- 172.21.15.28:0/51182988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4ee2b0 0x555e3a4f8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2abe954640 1 -- 172.21.15.28:0/51182988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e3a41a400 con 0x555e3a4ed990 2024-04-25T01:33:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2abe153640 1 --2- 172.21.15.28:0/51182988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4ee2b0 0x555e3a4f8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.782+0000 7f2abe954640 1 --2- 172.21.15.28:0/51182988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ed990 0x555e3a4edd70 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0015900 tx=0x7f2ab0015d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb099204f6809815 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.783+0000 7f2aacbfe640 1 -- 172.21.15.28:0/51182988 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0022070 con 0x555e3a4ed990 2024-04-25T01:33:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.783+0000 7f2aacbfe640 1 -- 172.21.15.28:0/51182988 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ab0015270 con 0x555e3a4ed990 2024-04-25T01:33:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.783+0000 7f2aacbfe640 1 -- 172.21.15.28:0/51182988 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab00058e0 con 0x555e3a4ed990 2024-04-25T01:33:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.783+0000 7f2ac31a1740 1 -- 172.21.15.28:0/51182988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ed990 msgr2=0x555e3a4edd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.783+0000 7f2abe954640 1 --2- 172.21.15.28:0/2313503959 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f2aa40a2d90 0x7f2aa40a5250 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2ab001f000 tx=0x7f2ab001b000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.783+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/51182988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ed990 0x555e3a4edd70 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0015900 tx=0x7f2ab0015d00 comp rx=0 tx=0).stop 2024-04-25T01:33:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.783+0000 7f2ac31a1740 1 -- 172.21.15.28:0/51182988 shutdown_connections 2024-04-25T01:33:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.783+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/51182988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4f8980 0x555e3a4fad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.783+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/51182988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4ee2b0 0x555e3a4f8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.783+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/51182988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ed990 0x555e3a4edd70 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.783+0000 7f2ac31a1740 1 -- 172.21.15.28:0/51182988 >> 172.21.15.28:0/51182988 conn(0x555e3a4e9cb0 msgr2=0x555e3a4ea090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.784+0000 7f2ac31a1740 1 -- 172.21.15.28:0/51182988 shutdown_connections 2024-04-25T01:33:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.784+0000 7f2ac31a1740 1 -- 172.21.15.28:0/51182988 wait complete. 2024-04-25T01:33:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.784+0000 7f2ac31a1740 1 -- start start 2024-04-25T01:33:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.784+0000 7f2ac31a1740 1 Processor -- start 2024-04-25T01:33:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.784+0000 7f2ac31a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ed990 0x555e3a4f82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.784+0000 7f2ac31a1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4ee2b0 0x555e3a4f55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.784+0000 7f2ac31a1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4f8980 0x555e3a4f5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.784+0000 7f2ac31a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555e3a4fb770 con 0x555e3a4ed990 2024-04-25T01:33:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.784+0000 7f2ac31a1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555e3a40a050 con 0x555e3a4f8980 2024-04-25T01:33:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.784+0000 7f2ac31a1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x555e3a4fb8f0 con 0x555e3a4ee2b0 2024-04-25T01:33:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.784+0000 7f2ac31a1740 10 client.0 _ll_register_callbacks cb 0x555e3a161be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.784+0000 7f2abe153640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ed990 0x555e3a4f82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.784+0000 7f2abe153640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ed990 0x555e3a4f82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40290/0 (socket says 172.21.15.28:40290) 2024-04-25T01:33:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.784+0000 7f2abe153640 1 -- 172.21.15.28:0/3406967727 learned_addr learned my addr 172.21.15.28:0/3406967727 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.784+0000 7f2abd952640 1 --2- 172.21.15.28:0/3406967727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4ee2b0 0x555e3a4f55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.785+0000 7f2abe954640 1 --2- 172.21.15.28:0/3406967727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4f8980 0x555e3a4f5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.785+0000 7f2abe153640 1 -- 172.21.15.28:0/3406967727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4ee2b0 msgr2=0x555e3a4f55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.785+0000 7f2abe153640 1 --2- 172.21.15.28:0/3406967727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4ee2b0 0x555e3a4f55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.785+0000 7f2abe153640 1 -- 172.21.15.28:0/3406967727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4f8980 msgr2=0x555e3a4f5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.785+0000 7f2abe153640 1 --2- 172.21.15.28:0/3406967727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4f8980 0x555e3a4f5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.785+0000 7f2abe153640 1 -- 172.21.15.28:0/3406967727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e3a3dbdc0 con 0x555e3a4ed990 2024-04-25T01:33:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.785+0000 7f2abe153640 1 --2- 172.21.15.28:0/3406967727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ed990 0x555e3a4f82d0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8098100 tx=0x7f2aa809b090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f34be3d8a98b71c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.785+0000 7f2aa1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.785+0000 7f2aa1ffb640 1 -- 172.21.15.28:0/3406967727 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa801d070 con 0x555e3a4ed990 2024-04-25T01:33:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.785+0000 7f2aa1ffb640 1 -- 172.21.15.28:0/3406967727 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f2aa8005310 con 0x555e3a4ed990 2024-04-25T01:33:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.785+0000 7f2ac31a1740 1 -- 172.21.15.28:0/3406967727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ab0015270 con 0x555e3a4ed990 2024-04-25T01:33:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.785+0000 7f2aa1ffb640 1 -- 172.21.15.28:0/3406967727 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2aa809fae0 con 0x555e3a4ed990 2024-04-25T01:33:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.785+0000 7f2ac31a1740 10 client.15921 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.785+0000 7f2ac31a1740 1 -- 172.21.15.28:0/3406967727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555e3a1623f0 con 0x555e3a4ed990 2024-04-25T01:33:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.785+0000 7f2aa1ffb640 1 -- 172.21.15.28:0/3406967727 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ab0015270 con 0x555e3a4ed990 2024-04-25T01:33:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.785+0000 7f2aa1ffb640 1 -- 172.21.15.28:0/3406967727 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2aa80ccde0 con 0x555e3a4ed990 2024-04-25T01:33:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.785+0000 7f2ac31a1740 10 client.15921 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.786+0000 7f2ac31a1740 10 client.15921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.786+0000 7f2ac31a1740 20 client.15921 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.786+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/3406967727 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555e3a508890 0x555e3a50ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.786+0000 7f2ac31a1740 4 client.15921 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.786+0000 7f2ac31a1740 1 -- 172.21.15.28:0/3406967727 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e3a45bbb0 con 0x555e3a508890 2024-04-25T01:33:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.786+0000 7f2abd952640 1 --2- 172.21.15.28:0/3406967727 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555e3a508890 0x555e3a50ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.786+0000 7f2aa1ffb640 10 client.15921 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.786+0000 7f2abd952640 1 --2- 172.21.15.28:0/3406967727 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555e3a508890 0x555e3a50ac80 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9de873b7186ac710 server_cookie=1a5346e5b95a8d24 in_seq=0 out_seq=0 2024-04-25T01:33:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.787+0000 7f2aa1ffb640 1 -- 172.21.15.28:0/3406967727 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555e3a45bbb0 con 0x555e3a508890 2024-04-25T01:33:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.787+0000 7f2aa1ffb640 10 client.15921 handle_command_reply: tid=0 2024-04-25T01:33:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.787+0000 7f2ac31a1740 1 client.15921 shutdown 2024-04-25T01:33:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.787+0000 7f2ac31a1740 1 -- 172.21.15.28:0/3406967727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ed990 msgr2=0x555e3a4f82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.787+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/3406967727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ed990 0x555e3a4f82d0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8098100 tx=0x7f2aa809b090 comp rx=0 tx=0).stop 2024-04-25T01:33:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.787+0000 7f2ac31a1740 1 -- 172.21.15.28:0/3406967727 shutdown_connections 2024-04-25T01:33:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.787+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/3406967727 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555e3a508890 0x555e3a50ac80 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.787+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/3406967727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4f8980 0x555e3a4f5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.787+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/3406967727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4ee2b0 0x555e3a4f55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.787+0000 7f2aa1ffb640 0 client.15921 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.787+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/3406967727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ed990 0x555e3a4f82d0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.787+0000 7f2ac31a1740 1 -- 172.21.15.28:0/3406967727 >> 172.21.15.28:0/3406967727 conn(0x555e3a4e9cb0 msgr2=0x555e3a4ef100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.787+0000 7f2ac31a1740 1 -- 172.21.15.28:0/3406967727 shutdown_connections 2024-04-25T01:33:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.787+0000 7f2ac31a1740 1 -- 172.21.15.28:0/3406967727 wait complete. 2024-04-25T01:33:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.787+0000 7f2ac31a1740 20 client.15921 trim_cache size 0 max 16384 2024-04-25T01:33:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2ac31a1740 1 Processor -- start 2024-04-25T01:33:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2ac31a1740 1 -- start start 2024-04-25T01:33:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2ac31a1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4ed990 0x555e3a504630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2ac31a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ee2b0 0x555e3a4e4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2ac31a1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4f8980 0x555e3a510520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2ac31a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab81ac790 con 0x555e3a4ee2b0 2024-04-25T01:33:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2ac31a1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab81ae2d0 con 0x555e3a4ed990 2024-04-25T01:33:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2ac31a1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab81ace60 con 0x555e3a4f8980 2024-04-25T01:33:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2abd952640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ee2b0 0x555e3a4e4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2abd952640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ee2b0 0x555e3a4e4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40300/0 (socket says 172.21.15.28:40300) 2024-04-25T01:33:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2abd952640 1 -- 172.21.15.28:0/4248216260 learned_addr learned my addr 172.21.15.28:0/4248216260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2abe954640 1 --2- 172.21.15.28:0/4248216260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4f8980 0x555e3a510520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2abe153640 1 --2- 172.21.15.28:0/4248216260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4ed990 0x555e3a504630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2abd952640 1 -- 172.21.15.28:0/4248216260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4f8980 msgr2=0x555e3a510520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2abd952640 1 --2- 172.21.15.28:0/4248216260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4f8980 0x555e3a510520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2abd952640 1 -- 172.21.15.28:0/4248216260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4ed990 msgr2=0x555e3a504630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2abd952640 1 --2- 172.21.15.28:0/4248216260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4ed990 0x555e3a504630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2abd952640 1 -- 172.21.15.28:0/4248216260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e3a4ee0e0 con 0x555e3a4ee2b0 2024-04-25T01:33:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2abe954640 1 --2- 172.21.15.28:0/4248216260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4f8980 0x555e3a510520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.801+0000 7f2abe153640 1 --2- 172.21.15.28:0/4248216260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4ed990 0x555e3a504630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.802+0000 7f2abd952640 1 --2- 172.21.15.28:0/4248216260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ee2b0 0x555e3a4e4b40 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2ab402abb0 tx=0x7f2ab4031a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d1cf6343c5c6831 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.802+0000 7f2aa3fff640 1 -- 172.21.15.28:0/4248216260 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab404c070 con 0x555e3a4ee2b0 2024-04-25T01:33:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.802+0000 7f2aa3fff640 1 -- 172.21.15.28:0/4248216260 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ab4048900 con 0x555e3a4ee2b0 2024-04-25T01:33:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.802+0000 7f2aa3fff640 1 -- 172.21.15.28:0/4248216260 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4035040 con 0x555e3a4ee2b0 2024-04-25T01:33:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.802+0000 7f2ac31a1740 1 -- 172.21.15.28:0/4248216260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ee2b0 msgr2=0x555e3a4e4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.802+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/4248216260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ee2b0 0x555e3a4e4b40 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2ab402abb0 tx=0x7f2ab4031a10 comp rx=0 tx=0).stop 2024-04-25T01:33:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.802+0000 7f2ac31a1740 1 -- 172.21.15.28:0/4248216260 shutdown_connections 2024-04-25T01:33:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.802+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/4248216260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4f8980 0x555e3a510520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.802+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/4248216260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4ee2b0 0x555e3a4e4b40 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.802+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/4248216260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4ed990 0x555e3a504630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.802+0000 7f2ac31a1740 1 -- 172.21.15.28:0/4248216260 >> 172.21.15.28:0/4248216260 conn(0x555e3a4e9cb0 msgr2=0x555e3a4efef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.803+0000 7f2ac31a1740 1 -- 172.21.15.28:0/4248216260 shutdown_connections 2024-04-25T01:33:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.803+0000 7f2ac31a1740 1 -- 172.21.15.28:0/4248216260 wait complete. 2024-04-25T01:33:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.803+0000 7f2ac31a1740 1 -- start start 2024-04-25T01:33:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.803+0000 7f2ac31a1740 1 Processor -- start 2024-04-25T01:33:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.803+0000 7f2ac31a1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4ed990 0x555e3a500b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.803+0000 7f2ac31a1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4ee2b0 0x555e3a501080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.803+0000 7f2ac31a1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4f8980 0x555e3a4fdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.803+0000 7f2ac31a1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab81adc00 con 0x555e3a4f8980 2024-04-25T01:33:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.803+0000 7f2ac31a1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab81ad530 con 0x555e3a4ee2b0 2024-04-25T01:33:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.803+0000 7f2ac31a1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab81aa770 con 0x555e3a4ed990 2024-04-25T01:33:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.803+0000 7f2ac31a1740 10 client.0 _ll_register_callbacks cb 0x7f2aa80cc8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.803+0000 7f2abe954640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4f8980 0x555e3a4fdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.803+0000 7f2abe954640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4f8980 0x555e3a4fdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40312/0 (socket says 172.21.15.28:40312) 2024-04-25T01:33:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.803+0000 7f2abe954640 1 -- 172.21.15.28:0/1818137976 learned_addr learned my addr 172.21.15.28:0/1818137976 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.803+0000 7f2abe153640 1 --2- 172.21.15.28:0/1818137976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4ed990 0x555e3a500b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.803+0000 7f2abd952640 1 --2- 172.21.15.28:0/1818137976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4ee2b0 0x555e3a501080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.804+0000 7f2abe954640 1 -- 172.21.15.28:0/1818137976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4ed990 msgr2=0x555e3a500b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.804+0000 7f2abe954640 1 --2- 172.21.15.28:0/1818137976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4ed990 0x555e3a500b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.804+0000 7f2abe954640 1 -- 172.21.15.28:0/1818137976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4ee2b0 msgr2=0x555e3a501080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.804+0000 7f2abe954640 1 --2- 172.21.15.28:0/1818137976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4ee2b0 0x555e3a501080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.804+0000 7f2abe954640 1 -- 172.21.15.28:0/1818137976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e3a4f5e90 con 0x555e3a4f8980 2024-04-25T01:33:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.804+0000 7f2abe954640 1 --2- 172.21.15.28:0/1818137976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4f8980 0x555e3a4fdb50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0018040 tx=0x7f2ab00128a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f0d92ab4bfb7fc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.804+0000 7f2aa1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.804+0000 7f2aa1ffb640 1 -- 172.21.15.28:0/1818137976 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0022040 con 0x555e3a4f8980 2024-04-25T01:33:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.804+0000 7f2aa1ffb640 1 -- 172.21.15.28:0/1818137976 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f2ab0004d30 con 0x555e3a4f8980 2024-04-25T01:33:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.804+0000 7f2ac31a1740 1 -- 172.21.15.28:0/1818137976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ab4048900 con 0x555e3a4f8980 2024-04-25T01:33:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.804+0000 7f2aa1ffb640 1 -- 172.21.15.28:0/1818137976 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2ab001d8c0 con 0x555e3a4f8980 2024-04-25T01:33:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.804+0000 7f2ac31a1740 10 client.15927 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.804+0000 7f2ac31a1740 1 -- 172.21.15.28:0/1818137976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555e3a22c7d0 con 0x555e3a4f8980 2024-04-25T01:33:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.804+0000 7f2aa1ffb640 1 -- 172.21.15.28:0/1818137976 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ab4048900 con 0x555e3a4f8980 2024-04-25T01:33:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.804+0000 7f2aa1ffb640 1 -- 172.21.15.28:0/1818137976 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2ab0005550 con 0x555e3a4f8980 2024-04-25T01:33:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.804+0000 7f2ac31a1740 10 client.15927 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.804+0000 7f2ac31a1740 10 client.15927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.804+0000 7f2ac31a1740 20 client.15927 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.804+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/1818137976 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555e3a56fa70 0x555e3a571e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.805+0000 7f2ac31a1740 4 client.15927 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:33:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.805+0000 7f2ac31a1740 1 -- 172.21.15.28:0/1818137976 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555e3a455f00 con 0x555e3a56fa70 2024-04-25T01:33:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.805+0000 7f2abe153640 1 --2- 172.21.15.28:0/1818137976 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555e3a56fa70 0x555e3a571e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.805+0000 7f2abe153640 1 --2- 172.21.15.28:0/1818137976 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555e3a56fa70 0x555e3a571e90 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f0f3a1f452653b6 server_cookie=76e3b327e28103c2 in_seq=0 out_seq=0 2024-04-25T01:33:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.805+0000 7f2aa1ffb640 10 client.15927 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.805+0000 7f2aa1ffb640 1 -- 172.21.15.28:0/1818137976 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555e3a455f00 con 0x555e3a56fa70 2024-04-25T01:33:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.805+0000 7f2aa1ffb640 10 client.15927 handle_command_reply: tid=0 2024-04-25T01:33:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.805+0000 7f2ac31a1740 1 client.15927 shutdown 2024-04-25T01:33:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.805+0000 7f2ac31a1740 1 -- 172.21.15.28:0/1818137976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4f8980 msgr2=0x555e3a4fdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.805+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/1818137976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4f8980 0x555e3a4fdb50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0018040 tx=0x7f2ab00128a0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac31a1740 1 -- 172.21.15.28:0/1818137976 shutdown_connections 2024-04-25T01:33:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/1818137976 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555e3a56fa70 0x555e3a571e90 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/1818137976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e3a4f8980 0x555e3a4fdb50 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/1818137976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e3a4ee2b0 0x555e3a501080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2aa1ffb640 0 client.15927 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac31a1740 1 --2- 172.21.15.28:0/1818137976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e3a4ed990 0x555e3a500b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac31a1740 1 -- 172.21.15.28:0/1818137976 >> 172.21.15.28:0/1818137976 conn(0x555e3a4e9cb0 msgr2=0x555e3a4e49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac31a1740 1 -- 172.21.15.28:0/1818137976 shutdown_connections 2024-04-25T01:33:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac31a1740 1 -- 172.21.15.28:0/1818137976 wait complete. 2024-04-25T01:33:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac31a1740 20 client.15927 trim_cache size 0 max 16384 2024-04-25T01:33:37.792 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:33:37.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac03b7640 1 -- 172.21.15.28:0/2313503959 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f2aa40a2d90 msgr2=0x7f2aa40a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac03b7640 1 --2- 172.21.15.28:0/2313503959 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f2aa40a2d90 0x7f2aa40a5250 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2ab001f000 tx=0x7f2ab001b000 comp rx=0 tx=0).stop 2024-04-25T01:33:37.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac03b7640 1 -- 172.21.15.28:0/2313503959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ab8100620 msgr2=0x7f2ab8198150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:37.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac03b7640 1 --2- 172.21.15.28:0/2313503959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ab8100620 0x7f2ab8198150 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800c970 tx=0x7f2aa800ce30 comp rx=0 tx=0).stop 2024-04-25T01:33:37.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac03b7640 1 -- 172.21.15.28:0/2313503959 shutdown_connections 2024-04-25T01:33:37.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac03b7640 1 --2- 172.21.15.28:0/2313503959 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f2aa40a2d90 0x7f2aa40a5250 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac03b7640 1 --2- 172.21.15.28:0/2313503959 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2ab8106630 0x7f2ab819c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac03b7640 1 --2- 172.21.15.28:0/2313503959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2ab8100fe0 0x7f2ab8198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac03b7640 1 --2- 172.21.15.28:0/2313503959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ab8100620 0x7f2ab8198150 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac03b7640 1 -- 172.21.15.28:0/2313503959 >> 172.21.15.28:0/2313503959 conn(0x7f2ab80776a0 msgr2=0x7f2ab806a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac03b7640 1 -- 172.21.15.28:0/2313503959 shutdown_connections 2024-04-25T01:33:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:37.806+0000 7f2ac03b7640 1 -- 172.21.15.28:0/2313503959 wait complete. 2024-04-25T01:33:37.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:37 smithi130 ceph-mon[32702]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-25T01:33:37.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:37 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/238256142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:33:37.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:37 smithi032 ceph-mon[32461]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-25T01:33:37.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:37 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/238256142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:33:38.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:37 smithi028 ceph-mon[29390]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-25T01:33:38.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:37 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/238256142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:33:38.796 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:33:38.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:33:38.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:33:38.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.929+0000 7ff03fb14640 1 -- 172.21.15.28:0/3819163815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0381029c0 msgr2=0x7ff038102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.929+0000 7ff03fb14640 1 --2- 172.21.15.28:0/3819163815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0381029c0 0x7ff038102da0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff034009900 tx=0x7ff03402a510 comp rx=0 tx=0).stop 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.929+0000 7ff03fb14640 1 -- 172.21.15.28:0/3819163815 shutdown_connections 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.929+0000 7ff03fb14640 1 --2- 172.21.15.28:0/3819163815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff03810bd00 0x7ff03810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.929+0000 7ff03fb14640 1 --2- 172.21.15.28:0/3819163815 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0381032e0 0x7ff03810b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.929+0000 7ff03fb14640 1 --2- 172.21.15.28:0/3819163815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0381029c0 0x7ff038102da0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.929+0000 7ff03fb14640 1 -- 172.21.15.28:0/3819163815 >> 172.21.15.28:0/3819163815 conn(0x7ff038077740 msgr2=0x7ff038077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.929+0000 7ff03fb14640 1 -- 172.21.15.28:0/3819163815 shutdown_connections 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.929+0000 7ff03fb14640 1 -- 172.21.15.28:0/3819163815 wait complete. 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.930+0000 7ff03fb14640 1 Processor -- start 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.930+0000 7ff03fb14640 1 -- start start 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.930+0000 7ff03fb14640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0381029c0 0x7ff038198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.930+0000 7ff03fb14640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0381032e0 0x7ff038198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.930+0000 7ff03fb14640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff03810bd00 0x7ff03819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.930+0000 7ff03fb14640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff038111b00 con 0x7ff0381029c0 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.930+0000 7ff03fb14640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff038111980 con 0x7ff03810bd00 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.930+0000 7ff03fb14640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff038111c80 con 0x7ff0381032e0 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.930+0000 7ff03d8b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0381029c0 0x7ff038198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.930+0000 7ff03d8b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0381029c0 0x7ff038198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40338/0 (socket says 172.21.15.28:40338) 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.930+0000 7ff03d8b0640 1 -- 172.21.15.28:0/395596204 learned_addr learned my addr 172.21.15.28:0/395596204 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.930+0000 7ff03e0b1640 1 --2- 172.21.15.28:0/395596204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff03810bd00 0x7ff03819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.930+0000 7ff03d0af640 1 --2- 172.21.15.28:0/395596204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0381032e0 0x7ff038198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.930+0000 7ff03d8b0640 1 -- 172.21.15.28:0/395596204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0381032e0 msgr2=0x7ff038198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.930+0000 7ff03d8b0640 1 --2- 172.21.15.28:0/395596204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0381032e0 0x7ff038198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.930+0000 7ff03d8b0640 1 -- 172.21.15.28:0/395596204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff03810bd00 msgr2=0x7ff03819c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:38.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.930+0000 7ff03d8b0640 1 --2- 172.21.15.28:0/395596204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff03810bd00 0x7ff03819c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.930+0000 7ff03d8b0640 1 -- 172.21.15.28:0/395596204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03819cfa0 con 0x7ff0381029c0 2024-04-25T01:33:38.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.930+0000 7ff03e0b1640 1 --2- 172.21.15.28:0/395596204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff03810bd00 0x7ff03819c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.931+0000 7ff03d8b0640 1 --2- 172.21.15.28:0/395596204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0381029c0 0x7ff038198000 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff034005ab0 tx=0x7ff0340042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.931+0000 7ff02affd640 1 -- 172.21.15.28:0/395596204 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03403f030 con 0x7ff0381029c0 2024-04-25T01:33:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.931+0000 7ff03fb14640 1 -- 172.21.15.28:0/395596204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff03819d230 con 0x7ff0381029c0 2024-04-25T01:33:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.931+0000 7ff02affd640 1 -- 172.21.15.28:0/395596204 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff034003ce0 con 0x7ff0381029c0 2024-04-25T01:33:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.931+0000 7ff02affd640 1 -- 172.21.15.28:0/395596204 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff034035070 con 0x7ff0381029c0 2024-04-25T01:33:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.931+0000 7ff03fb14640 1 -- 172.21.15.28:0/395596204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0381a4b20 con 0x7ff0381029c0 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.934+0000 7ff02affd640 1 -- 172.21.15.28:0/395596204 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7ff034042030 con 0x7ff0381029c0 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.934+0000 7ff042825740 1 Processor -- start 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.934+0000 7ff042825740 1 -- start start 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.934+0000 7ff042825740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 0x55744e878cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff042825740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e879290 0x55744e879710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff042825740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e884010 0x55744e886400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff042825740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55744e7dd2c0 con 0x55744e87a980 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff042825740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55744e785910 con 0x55744e884010 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff042825740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55744e7d9280 con 0x55744e879290 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff02affd640 1 --2- 172.21.15.28:0/395596204 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff0240a2d90 0x7ff0240a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff03e0b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 0x55744e878cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff03e0b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 0x55744e878cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40348/0 (socket says 172.21.15.28:40348) 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff03e0b1640 1 -- 172.21.15.28:0/3504096811 learned_addr learned my addr 172.21.15.28:0/3504096811 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff03e0b1640 1 --2- 172.21.15.28:0/395596204 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff0240a2d90 0x7ff0240a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff02affd640 1 -- 172.21.15.28:0/395596204 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff0340f02f0 con 0x7ff0381029c0 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff03d8b0640 1 --2- 172.21.15.28:0/3504096811 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e879290 0x55744e879710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff03d0af640 1 --2- 172.21.15.28:0/3504096811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e884010 0x55744e886400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff03e0b1640 1 -- 172.21.15.28:0/3504096811 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e879290 msgr2=0x55744e879710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff03e0b1640 1 --2- 172.21.15.28:0/3504096811 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e879290 0x55744e879710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff03e0b1640 1 -- 172.21.15.28:0/3504096811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e884010 msgr2=0x55744e886400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff03e0b1640 1 --2- 172.21.15.28:0/3504096811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e884010 0x55744e886400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff03e0b1640 1 -- 172.21.15.28:0/3504096811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55744e796fb0 con 0x55744e87a980 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff03d8b0640 1 --2- 172.21.15.28:0/3504096811 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e879290 0x55744e879710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff03e0b1640 1 --2- 172.21.15.28:0/3504096811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 0x55744e878cc0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff030013e40 tx=0x7ff0300107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce1a44248479c3f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.935+0000 7ff03e0b1640 1 --2- 172.21.15.28:0/395596204 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff0240a2d90 0x7ff0240a5250 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff030010ca0 tx=0x7ff03001b000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.936+0000 7ff023fff640 1 -- 172.21.15.28:0/3504096811 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03000bd50 con 0x55744e87a980 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.936+0000 7ff023fff640 1 -- 172.21.15.28:0/3504096811 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff030003970 con 0x55744e87a980 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.936+0000 7ff023fff640 1 -- 172.21.15.28:0/3504096811 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff030004330 con 0x55744e87a980 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.936+0000 7ff042825740 1 -- 172.21.15.28:0/3504096811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 msgr2=0x55744e878cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.936+0000 7ff042825740 1 --2- 172.21.15.28:0/3504096811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 0x55744e878cc0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff030013e40 tx=0x7ff0300107a0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.936+0000 7ff042825740 1 -- 172.21.15.28:0/3504096811 shutdown_connections 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.936+0000 7ff042825740 1 --2- 172.21.15.28:0/3504096811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e884010 0x55744e886400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.936+0000 7ff042825740 1 --2- 172.21.15.28:0/3504096811 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e879290 0x55744e879710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.936+0000 7ff042825740 1 --2- 172.21.15.28:0/3504096811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 0x55744e878cc0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.936+0000 7ff042825740 1 -- 172.21.15.28:0/3504096811 >> 172.21.15.28:0/3504096811 conn(0x55744e841950 msgr2=0x55744e8757f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.937+0000 7ff042825740 1 -- 172.21.15.28:0/3504096811 shutdown_connections 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.937+0000 7ff042825740 1 -- 172.21.15.28:0/3504096811 wait complete. 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.937+0000 7ff042825740 1 -- start start 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.937+0000 7ff042825740 1 Processor -- start 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.937+0000 7ff042825740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e879290 0x55744e880f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.937+0000 7ff042825740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 0x55744e881490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.937+0000 7ff042825740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e884010 0x55744e8819d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.937+0000 7ff042825740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55744e886e40 con 0x55744e87a980 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.937+0000 7ff042825740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55744e798e90 con 0x55744e884010 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.937+0000 7ff042825740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55744e886fc0 con 0x55744e879290 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.937+0000 7ff042825740 10 client.0 _ll_register_callbacks cb 0x55744e7d4970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.937+0000 7ff03d0af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 0x55744e881490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.937+0000 7ff03d0af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 0x55744e881490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40350/0 (socket says 172.21.15.28:40350) 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.937+0000 7ff03d0af640 1 -- 172.21.15.28:0/1313030528 learned_addr learned my addr 172.21.15.28:0/1313030528 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.938+0000 7ff03e0b1640 1 --2- 172.21.15.28:0/1313030528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e884010 0x55744e8819d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.938+0000 7ff03d8b0640 1 --2- 172.21.15.28:0/1313030528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e879290 0x55744e880f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.938+0000 7ff03d0af640 1 -- 172.21.15.28:0/1313030528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e879290 msgr2=0x55744e880f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.938+0000 7ff03d0af640 1 --2- 172.21.15.28:0/1313030528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e879290 0x55744e880f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.938+0000 7ff03d0af640 1 -- 172.21.15.28:0/1313030528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e884010 msgr2=0x55744e8819d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.938+0000 7ff03d0af640 1 --2- 172.21.15.28:0/1313030528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e884010 0x55744e8819d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.938+0000 7ff03d0af640 1 -- 172.21.15.28:0/1313030528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55744e5b97e0 con 0x55744e87a980 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.938+0000 7ff03d0af640 1 --2- 172.21.15.28:0/1313030528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 0x55744e881490 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff02c00f640 tx=0x7ff02c00fb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=960a769539b72d2a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.938+0000 7ff0217fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.938+0000 7ff0217fa640 1 -- 172.21.15.28:0/1313030528 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c016070 con 0x55744e87a980 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.938+0000 7ff0217fa640 1 -- 172.21.15.28:0/1313030528 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff02c00ba60 con 0x55744e87a980 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.938+0000 7ff042825740 1 -- 172.21.15.28:0/1313030528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff030003970 con 0x55744e87a980 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.938+0000 7ff0217fa640 1 -- 172.21.15.28:0/1313030528 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff02c025050 con 0x55744e87a980 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.938+0000 7ff042825740 10 client.15939 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.938+0000 7ff042825740 1 -- 172.21.15.28:0/1313030528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55744e619f70 con 0x55744e87a980 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.939+0000 7ff0217fa640 1 -- 172.21.15.28:0/1313030528 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff030003970 con 0x55744e87a980 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.939+0000 7ff0217fa640 1 -- 172.21.15.28:0/1313030528 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff02c01a400 con 0x55744e87a980 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.939+0000 7ff042825740 10 client.15939 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.939+0000 7ff042825740 10 client.15939 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.939+0000 7ff042825740 20 client.15939 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.939+0000 7ff042825740 1 --2- 172.21.15.28:0/1313030528 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55744e8941b0 0x55744e896630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.939+0000 7ff042825740 4 client.15939 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.939+0000 7ff042825740 1 -- 172.21.15.28:0/1313030528 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55744e7b15a0 con 0x55744e8941b0 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.939+0000 7ff03d8b0640 1 --2- 172.21.15.28:0/1313030528 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55744e8941b0 0x55744e896630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.940+0000 7ff0217fa640 10 client.15939 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.940+0000 7ff03d8b0640 1 --2- 172.21.15.28:0/1313030528 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55744e8941b0 0x55744e896630 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54944e6e570294cc server_cookie=209a4d7be2413cf0 in_seq=0 out_seq=0 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.940+0000 7ff0217fa640 1 -- 172.21.15.28:0/1313030528 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7ff0340a1ae0 con 0x55744e8941b0 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.940+0000 7ff0217fa640 10 client.15939 handle_command_reply: tid=0 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.940+0000 7ff042825740 1 client.15939 shutdown 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.940+0000 7ff042825740 1 -- 172.21.15.28:0/1313030528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 msgr2=0x55744e881490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.940+0000 7ff042825740 1 --2- 172.21.15.28:0/1313030528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 0x55744e881490 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff02c00f640 tx=0x7ff02c00fb00 comp rx=0 tx=0).stop 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.941+0000 7ff042825740 1 -- 172.21.15.28:0/1313030528 shutdown_connections 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.941+0000 7ff042825740 1 --2- 172.21.15.28:0/1313030528 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55744e8941b0 0x55744e896630 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.941+0000 7ff042825740 1 --2- 172.21.15.28:0/1313030528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e884010 0x55744e8819d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.941+0000 7ff042825740 1 --2- 172.21.15.28:0/1313030528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 0x55744e881490 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.941+0000 7ff042825740 1 --2- 172.21.15.28:0/1313030528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e879290 0x55744e880f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.941+0000 7ff042825740 1 -- 172.21.15.28:0/1313030528 >> 172.21.15.28:0/1313030528 conn(0x55744e841950 msgr2=0x55744e87b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.941+0000 7ff0217fa640 0 client.15939 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.941+0000 7ff042825740 1 -- 172.21.15.28:0/1313030528 shutdown_connections 2024-04-25T01:33:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.941+0000 7ff042825740 1 -- 172.21.15.28:0/1313030528 wait complete. 2024-04-25T01:33:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.941+0000 7ff042825740 20 client.15939 trim_cache size 0 max 16384 2024-04-25T01:33:38.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.961+0000 7ff042825740 1 Processor -- start 2024-04-25T01:33:38.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.961+0000 7ff042825740 1 -- start start 2024-04-25T01:33:38.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.961+0000 7ff042825740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e879290 0x55744e88fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:38.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.961+0000 7ff042825740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e87a980 0x55744e874950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:38.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.962+0000 7ff042825740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e884010 0x55744e89bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:38.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.962+0000 7ff042825740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0381ae9d0 con 0x55744e879290 2024-04-25T01:33:38.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.962+0000 7ff042825740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff0381b0510 con 0x55744e884010 2024-04-25T01:33:38.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.962+0000 7ff042825740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff0381af0a0 con 0x55744e87a980 2024-04-25T01:33:38.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.962+0000 7ff03d8b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e879290 0x55744e88fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:38.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.962+0000 7ff03d8b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e879290 0x55744e88fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40360/0 (socket says 172.21.15.28:40360) 2024-04-25T01:33:38.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.962+0000 7ff03d8b0640 1 -- 172.21.15.28:0/3096886598 learned_addr learned my addr 172.21.15.28:0/3096886598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:38.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.962+0000 7ff03e0b1640 1 --2- 172.21.15.28:0/3096886598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e884010 0x55744e89bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:38.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.962+0000 7ff03d8b0640 1 -- 172.21.15.28:0/3096886598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e87a980 msgr2=0x55744e874950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:38.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.962+0000 7ff03d8b0640 1 --2- 172.21.15.28:0/3096886598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e87a980 0x55744e874950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.962+0000 7ff03d8b0640 1 -- 172.21.15.28:0/3096886598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e884010 msgr2=0x55744e89bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:38.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.962+0000 7ff03d8b0640 1 --2- 172.21.15.28:0/3096886598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e884010 0x55744e89bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.962+0000 7ff03d8b0640 1 -- 172.21.15.28:0/3096886598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55744e4b1170 con 0x55744e879290 2024-04-25T01:33:38.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.962+0000 7ff03d8b0640 1 --2- 172.21.15.28:0/3096886598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e879290 0x55744e88fd70 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ff0340b8040 tx=0x7ff03408b290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d10d90485f354d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:38.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.962+0000 7ff0237fe640 1 -- 172.21.15.28:0/3096886598 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03408b070 con 0x55744e879290 2024-04-25T01:33:38.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.962+0000 7ff0237fe640 1 -- 172.21.15.28:0/3096886598 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55744e7b15a0 con 0x55744e879290 2024-04-25T01:33:38.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.962+0000 7ff0237fe640 1 -- 172.21.15.28:0/3096886598 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff034049040 con 0x55744e879290 2024-04-25T01:33:38.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.963+0000 7ff042825740 1 -- 172.21.15.28:0/3096886598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e879290 msgr2=0x55744e88fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:38.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.963+0000 7ff042825740 1 --2- 172.21.15.28:0/3096886598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e879290 0x55744e88fd70 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ff0340b8040 tx=0x7ff03408b290 comp rx=0 tx=0).stop 2024-04-25T01:33:38.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.963+0000 7ff042825740 1 -- 172.21.15.28:0/3096886598 shutdown_connections 2024-04-25T01:33:38.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.963+0000 7ff042825740 1 --2- 172.21.15.28:0/3096886598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e884010 0x55744e89bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.963+0000 7ff042825740 1 --2- 172.21.15.28:0/3096886598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e87a980 0x55744e874950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.963+0000 7ff042825740 1 --2- 172.21.15.28:0/3096886598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e879290 0x55744e88fd70 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.963+0000 7ff042825740 1 -- 172.21.15.28:0/3096886598 >> 172.21.15.28:0/3096886598 conn(0x55744e841950 msgr2=0x55744e8757f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.963+0000 7ff042825740 1 -- 172.21.15.28:0/3096886598 shutdown_connections 2024-04-25T01:33:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.963+0000 7ff042825740 1 -- 172.21.15.28:0/3096886598 wait complete. 2024-04-25T01:33:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.963+0000 7ff042825740 1 -- start start 2024-04-25T01:33:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.964+0000 7ff042825740 1 Processor -- start 2024-04-25T01:33:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.964+0000 7ff042825740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e879290 0x55744e88c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.964+0000 7ff042825740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 0x55744e88c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.964+0000 7ff042825740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e884010 0x55744e889250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.964+0000 7ff042825740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0381afe40 con 0x55744e87a980 2024-04-25T01:33:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.964+0000 7ff042825740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff0381af770 con 0x55744e879290 2024-04-25T01:33:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.964+0000 7ff042825740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff0381ac9b0 con 0x55744e884010 2024-04-25T01:33:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.964+0000 7ff042825740 10 client.0 _ll_register_callbacks cb 0x7ff02c00de40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.964+0000 7ff03d0af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 0x55744e88c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.964+0000 7ff03d0af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 0x55744e88c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40364/0 (socket says 172.21.15.28:40364) 2024-04-25T01:33:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.964+0000 7ff03d0af640 1 -- 172.21.15.28:0/1870362191 learned_addr learned my addr 172.21.15.28:0/1870362191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.964+0000 7ff03d8b0640 1 --2- 172.21.15.28:0/1870362191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e879290 0x55744e88c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.964+0000 7ff03e0b1640 1 --2- 172.21.15.28:0/1870362191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e884010 0x55744e889250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.964+0000 7ff03d0af640 1 -- 172.21.15.28:0/1870362191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e884010 msgr2=0x55744e889250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.964+0000 7ff03d0af640 1 --2- 172.21.15.28:0/1870362191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e884010 0x55744e889250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.964+0000 7ff03d0af640 1 -- 172.21.15.28:0/1870362191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e879290 msgr2=0x55744e88c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.964+0000 7ff03d0af640 1 --2- 172.21.15.28:0/1870362191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e879290 0x55744e88c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.964+0000 7ff03d0af640 1 -- 172.21.15.28:0/1870362191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55744e889990 con 0x55744e87a980 2024-04-25T01:33:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.964+0000 7ff03d8b0640 1 --2- 172.21.15.28:0/1870362191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e879290 0x55744e88c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:33:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.965+0000 7ff03d0af640 1 --2- 172.21.15.28:0/1870362191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 0x55744e88c7c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff02c01aa10 tx=0x7ff02c01acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=459b446912e4cf59 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.965+0000 7ff0217fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.965+0000 7ff0217fa640 1 -- 172.21.15.28:0/1870362191 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c016040 con 0x55744e87a980 2024-04-25T01:33:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.965+0000 7ff0217fa640 1 -- 172.21.15.28:0/1870362191 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff02c005240 con 0x55744e87a980 2024-04-25T01:33:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.965+0000 7ff042825740 1 -- 172.21.15.28:0/1870362191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55744e7b15a0 con 0x55744e87a980 2024-04-25T01:33:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.965+0000 7ff0217fa640 1 -- 172.21.15.28:0/1870362191 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff02c039050 con 0x55744e87a980 2024-04-25T01:33:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.965+0000 7ff042825740 10 client.15945 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.965+0000 7ff042825740 1 -- 172.21.15.28:0/1870362191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55744e8fb240 con 0x55744e87a980 2024-04-25T01:33:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.965+0000 7ff0217fa640 1 -- 172.21.15.28:0/1870362191 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55744e7b15a0 con 0x55744e87a980 2024-04-25T01:33:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.965+0000 7ff0217fa640 1 -- 172.21.15.28:0/1870362191 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff02c004640 con 0x55744e87a980 2024-04-25T01:33:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.965+0000 7ff042825740 10 client.15945 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.965+0000 7ff042825740 10 client.15945 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.965+0000 7ff042825740 20 client.15945 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.965+0000 7ff042825740 1 --2- 172.21.15.28:0/1870362191 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55744e8fb9e0 0x55744e8fde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.965+0000 7ff042825740 4 client.15945 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:33:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.966+0000 7ff042825740 1 -- 172.21.15.28:0/1870362191 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55744e7ea800 con 0x55744e8fb9e0 2024-04-25T01:33:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.966+0000 7ff03d8b0640 1 --2- 172.21.15.28:0/1870362191 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55744e8fb9e0 0x55744e8fde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.966+0000 7ff0217fa640 10 client.15945 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.966+0000 7ff03d8b0640 1 --2- 172.21.15.28:0/1870362191 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55744e8fb9e0 0x55744e8fde00 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c30cf950055759b0 server_cookie=93f27badd9642d99 in_seq=0 out_seq=0 2024-04-25T01:33:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.967+0000 7ff0217fa640 1 -- 172.21.15.28:0/1870362191 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55744e7ea800 con 0x55744e8fb9e0 2024-04-25T01:33:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.967+0000 7ff0217fa640 10 client.15945 handle_command_reply: tid=0 2024-04-25T01:33:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.967+0000 7ff042825740 1 client.15945 shutdown 2024-04-25T01:33:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.967+0000 7ff042825740 1 -- 172.21.15.28:0/1870362191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 msgr2=0x55744e88c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.967+0000 7ff042825740 1 --2- 172.21.15.28:0/1870362191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 0x55744e88c7c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff02c01aa10 tx=0x7ff02c01acc0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.968+0000 7ff042825740 1 -- 172.21.15.28:0/1870362191 shutdown_connections 2024-04-25T01:33:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.968+0000 7ff042825740 1 --2- 172.21.15.28:0/1870362191 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55744e8fb9e0 0x55744e8fde00 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.968+0000 7ff042825740 1 --2- 172.21.15.28:0/1870362191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744e884010 0x55744e889250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.968+0000 7ff042825740 1 --2- 172.21.15.28:0/1870362191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744e87a980 0x55744e88c7c0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.968+0000 7ff0217fa640 0 client.15945 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.968+0000 7ff042825740 1 --2- 172.21.15.28:0/1870362191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744e879290 0x55744e88c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.968+0000 7ff042825740 1 -- 172.21.15.28:0/1870362191 >> 172.21.15.28:0/1870362191 conn(0x55744e841950 msgr2=0x55744e87b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.968+0000 7ff042825740 1 -- 172.21.15.28:0/1870362191 shutdown_connections 2024-04-25T01:33:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.968+0000 7ff042825740 1 -- 172.21.15.28:0/1870362191 wait complete. 2024-04-25T01:33:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.968+0000 7ff042825740 20 client.15945 trim_cache size 0 max 16384 2024-04-25T01:33:38.954 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:33:38.954 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:33:38.954 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "9855a9d7-41ea-4000-9673-0aef7bf6e045", 2024-04-25T01:33:38.954 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:33:38.954 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:33:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.968+0000 7ff03fb14640 1 -- 172.21.15.28:0/395596204 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff0240a2d90 msgr2=0x7ff0240a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.968+0000 7ff03fb14640 1 --2- 172.21.15.28:0/395596204 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff0240a2d90 0x7ff0240a5250 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff030010ca0 tx=0x7ff03001b000 comp rx=0 tx=0).stop 2024-04-25T01:33:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.968+0000 7ff03fb14640 1 -- 172.21.15.28:0/395596204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0381029c0 msgr2=0x7ff038198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.968+0000 7ff03fb14640 1 --2- 172.21.15.28:0/395596204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0381029c0 0x7ff038198000 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff034005ab0 tx=0x7ff0340042f0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.968+0000 7ff03fb14640 1 -- 172.21.15.28:0/395596204 shutdown_connections 2024-04-25T01:33:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.968+0000 7ff03fb14640 1 --2- 172.21.15.28:0/395596204 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff0240a2d90 0x7ff0240a5250 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.968+0000 7ff03fb14640 1 --2- 172.21.15.28:0/395596204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff03810bd00 0x7ff03819c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.968+0000 7ff03fb14640 1 --2- 172.21.15.28:0/395596204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0381032e0 0x7ff038198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.968+0000 7ff03fb14640 1 --2- 172.21.15.28:0/395596204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0381029c0 0x7ff038198000 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:38.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.969+0000 7ff03fb14640 1 -- 172.21.15.28:0/395596204 >> 172.21.15.28:0/395596204 conn(0x7ff038077740 msgr2=0x7ff0380fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:38.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.969+0000 7ff03fb14640 1 -- 172.21.15.28:0/395596204 shutdown_connections 2024-04-25T01:33:38.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:38.969+0000 7ff03fb14640 1 -- 172.21.15.28:0/395596204 wait complete. 2024-04-25T01:33:38.957 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:33:38.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:33:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.088+0000 7f29d2777640 1 -- 172.21.15.28:0/1006061842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29cc10bd00 msgr2=0x7f29cc10e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.088+0000 7f29d2777640 1 --2- 172.21.15.28:0/1006061842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29cc10bd00 0x7f29cc10e1a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f29c4009ca0 tx=0x7f29c402a9e0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.088+0000 7f29d2777640 1 -- 172.21.15.28:0/1006061842 shutdown_connections 2024-04-25T01:33:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.088+0000 7f29d2777640 1 --2- 172.21.15.28:0/1006061842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29cc10bd00 0x7f29cc10e1a0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.088+0000 7f29d2777640 1 --2- 172.21.15.28:0/1006061842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f29cc1032e0 0x7f29cc10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.088+0000 7f29d2777640 1 --2- 172.21.15.28:0/1006061842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29cc1029c0 0x7f29cc102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.088+0000 7f29d2777640 1 -- 172.21.15.28:0/1006061842 >> 172.21.15.28:0/1006061842 conn(0x7f29cc077740 msgr2=0x7f29cc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.088+0000 7f29d2777640 1 -- 172.21.15.28:0/1006061842 shutdown_connections 2024-04-25T01:33:39.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.088+0000 7f29d2777640 1 -- 172.21.15.28:0/1006061842 wait complete. 2024-04-25T01:33:39.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.089+0000 7f29d2777640 1 Processor -- start 2024-04-25T01:33:39.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.089+0000 7f29d2777640 1 -- start start 2024-04-25T01:33:39.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.089+0000 7f29d2777640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29cc1029c0 0x7f29cc197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:39.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.089+0000 7f29d2777640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29cc1032e0 0x7f29cc198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:39.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.089+0000 7f29d2777640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f29cc10bd00 0x7f29cc19c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:39.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.089+0000 7f29d2777640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f29cc111b00 con 0x7f29cc1029c0 2024-04-25T01:33:39.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.089+0000 7f29d2777640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f29cc111980 con 0x7f29cc1032e0 2024-04-25T01:33:39.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.089+0000 7f29d2777640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f29cc111c80 con 0x7f29cc10bd00 2024-04-25T01:33:39.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.089+0000 7f29cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29cc1029c0 0x7f29cc197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:39.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.090+0000 7f29cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29cc1029c0 0x7f29cc197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40370/0 (socket says 172.21.15.28:40370) 2024-04-25T01:33:39.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.090+0000 7f29cbfff640 1 -- 172.21.15.28:0/1061956441 learned_addr learned my addr 172.21.15.28:0/1061956441 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:39.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.090+0000 7f29d0d14640 1 --2- 172.21.15.28:0/1061956441 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f29cc10bd00 0x7f29cc19c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:39.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.090+0000 7f29cb7fe640 1 --2- 172.21.15.28:0/1061956441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29cc1032e0 0x7f29cc198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:39.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.090+0000 7f29cbfff640 1 -- 172.21.15.28:0/1061956441 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f29cc10bd00 msgr2=0x7f29cc19c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:39.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.090+0000 7f29cbfff640 1 --2- 172.21.15.28:0/1061956441 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f29cc10bd00 0x7f29cc19c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.090+0000 7f29cbfff640 1 -- 172.21.15.28:0/1061956441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29cc1032e0 msgr2=0x7f29cc198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:39.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.090+0000 7f29cbfff640 1 --2- 172.21.15.28:0/1061956441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29cc1032e0 0x7f29cc198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.090+0000 7f29cbfff640 1 -- 172.21.15.28:0/1061956441 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29cc19ced0 con 0x7f29cc1029c0 2024-04-25T01:33:39.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.090+0000 7f29d0d14640 1 --2- 172.21.15.28:0/1061956441 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f29cc10bd00 0x7f29cc19c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:39.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.090+0000 7f29cbfff640 1 --2- 172.21.15.28:0/1061956441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29cc1029c0 0x7f29cc197ee0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00c920 tx=0x7f29bc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:39.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.090+0000 7f29c97fa640 1 -- 172.21.15.28:0/1061956441 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc007c70 con 0x7f29cc1029c0 2024-04-25T01:33:39.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.090+0000 7f29d2777640 1 -- 172.21.15.28:0/1061956441 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29cc19d1c0 con 0x7f29cc1029c0 2024-04-25T01:33:39.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.090+0000 7f29c97fa640 1 -- 172.21.15.28:0/1061956441 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29bc007e10 con 0x7f29cc1029c0 2024-04-25T01:33:39.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.090+0000 7f29d2777640 1 -- 172.21.15.28:0/1061956441 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29cc1a4a50 con 0x7f29cc1029c0 2024-04-25T01:33:39.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.090+0000 7f29c97fa640 1 -- 172.21.15.28:0/1061956441 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc018890 con 0x7f29cc1029c0 2024-04-25T01:33:39.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.093+0000 7f29c97fa640 1 -- 172.21.15.28:0/1061956441 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138009+0+0 (secure 0 0 0) 0x7f29bc018b10 con 0x7f29cc1029c0 2024-04-25T01:33:39.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.093+0000 7f29c97fa640 1 --2- 172.21.15.28:0/1061956441 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f29b40a2d90 0x7f29b40a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:39.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29cb7fe640 1 --2- 172.21.15.28:0/1061956441 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f29b40a2d90 0x7f29b40a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:39.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29c97fa640 1 -- 172.21.15.28:0/1061956441 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f29bc0cfa60 con 0x7f29cc1029c0 2024-04-25T01:33:39.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29d5564740 1 Processor -- start 2024-04-25T01:33:39.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29d5564740 1 -- start start 2024-04-25T01:33:39.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29d5564740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c09688a360 0x55c09688a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:39.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29d5564740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c09688ad10 0x55c0968994f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:39.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29d5564740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c096899a30 0x55c09689be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:39.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29d5564740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c0967ef6b0 con 0x55c09688ad10 2024-04-25T01:33:39.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29d5564740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c096797880 con 0x55c096899a30 2024-04-25T01:33:39.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29d5564740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c0967eb670 con 0x55c09688a360 2024-04-25T01:33:39.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29cb7fe640 1 --2- 172.21.15.28:0/1061956441 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f29b40a2d90 0x7f29b40a5250 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f29cc0fff20 tx=0x7f29c000c4a0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:39.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c09688ad10 0x55c0968994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:39.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c09688ad10 0x55c0968994f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40384/0 (socket says 172.21.15.28:40384) 2024-04-25T01:33:39.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29cb7fe640 1 -- 172.21.15.28:0/2181406732 learned_addr learned my addr 172.21.15.28:0/2181406732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:39.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29d0d14640 1 --2- 172.21.15.28:0/2181406732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c096899a30 0x55c09689be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:39.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29cbfff640 1 --2- 172.21.15.28:0/2181406732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c09688a360 0x55c09688a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29cb7fe640 1 -- 172.21.15.28:0/2181406732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c09688a360 msgr2=0x55c09688a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29cb7fe640 1 --2- 172.21.15.28:0/2181406732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c09688a360 0x55c09688a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29cb7fe640 1 -- 172.21.15.28:0/2181406732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c096899a30 msgr2=0x55c09689be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29cb7fe640 1 --2- 172.21.15.28:0/2181406732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c096899a30 0x55c09689be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29cb7fe640 1 -- 172.21.15.28:0/2181406732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0965cd7d0 con 0x55c09688ad10 2024-04-25T01:33:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.094+0000 7f29cbfff640 1 --2- 172.21.15.28:0/2181406732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c09688a360 0x55c09688a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.095+0000 7f29cb7fe640 1 --2- 172.21.15.28:0/2181406732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c09688ad10 0x55c0968994f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f29c000ee50 tx=0x7f29c0011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c59ef5257781cbc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.095+0000 7f29b27fc640 1 -- 172.21.15.28:0/2181406732 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c0021070 con 0x55c09688ad10 2024-04-25T01:33:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.095+0000 7f29b27fc640 1 -- 172.21.15.28:0/2181406732 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29c0010e40 con 0x55c09688ad10 2024-04-25T01:33:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.095+0000 7f29b27fc640 1 -- 172.21.15.28:0/2181406732 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c001cc50 con 0x55c09688ad10 2024-04-25T01:33:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.096+0000 7f29d5564740 1 -- 172.21.15.28:0/2181406732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c09688ad10 msgr2=0x55c0968994f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.096+0000 7f29d5564740 1 --2- 172.21.15.28:0/2181406732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c09688ad10 0x55c0968994f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f29c000ee50 tx=0x7f29c0011000 comp rx=0 tx=0).stop 2024-04-25T01:33:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.096+0000 7f29d5564740 1 -- 172.21.15.28:0/2181406732 shutdown_connections 2024-04-25T01:33:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.096+0000 7f29d5564740 1 --2- 172.21.15.28:0/2181406732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c096899a30 0x55c09689be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.096+0000 7f29d5564740 1 --2- 172.21.15.28:0/2181406732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c09688ad10 0x55c0968994f0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.096+0000 7f29d5564740 1 --2- 172.21.15.28:0/2181406732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c09688a360 0x55c09688a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.096+0000 7f29d5564740 1 -- 172.21.15.28:0/2181406732 >> 172.21.15.28:0/2181406732 conn(0x55c096886950 msgr2=0x55c096886d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:39.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.096+0000 7f29d5564740 1 -- 172.21.15.28:0/2181406732 shutdown_connections 2024-04-25T01:33:39.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.096+0000 7f29d5564740 1 -- 172.21.15.28:0/2181406732 wait complete. 2024-04-25T01:33:39.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.096+0000 7f29d5564740 1 -- start start 2024-04-25T01:33:39.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.097+0000 7f29d5564740 1 Processor -- start 2024-04-25T01:33:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.097+0000 7f29d5564740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c09688a360 0x55c096896990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.097+0000 7f29d5564740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c09688ad10 0x55c096896ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.097+0000 7f29d5564740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c096899a30 0x55c096897410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.097+0000 7f29d5564740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c09689c7f0 con 0x55c096899a30 2024-04-25T01:33:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.097+0000 7f29d5564740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c0967aaf50 con 0x55c09688ad10 2024-04-25T01:33:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.097+0000 7f29d5564740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c09689c970 con 0x55c09688a360 2024-04-25T01:33:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.097+0000 7f29d5564740 10 client.0 _ll_register_callbacks cb 0x55c0967e6000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.097+0000 7f29d0d14640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c096899a30 0x55c096897410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.097+0000 7f29d0d14640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c096899a30 0x55c096897410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40396/0 (socket says 172.21.15.28:40396) 2024-04-25T01:33:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.097+0000 7f29d0d14640 1 -- 172.21.15.28:0/2801303376 learned_addr learned my addr 172.21.15.28:0/2801303376 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.097+0000 7f29cb7fe640 1 --2- 172.21.15.28:0/2801303376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c09688ad10 0x55c096896ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.097+0000 7f29cbfff640 1 --2- 172.21.15.28:0/2801303376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c09688a360 0x55c096896990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.097+0000 7f29d0d14640 1 -- 172.21.15.28:0/2801303376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c09688a360 msgr2=0x55c096896990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.097+0000 7f29d0d14640 1 --2- 172.21.15.28:0/2801303376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c09688a360 0x55c096896990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.097+0000 7f29d0d14640 1 -- 172.21.15.28:0/2801303376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c09688ad10 msgr2=0x55c096896ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.097+0000 7f29d0d14640 1 --2- 172.21.15.28:0/2801303376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c09688ad10 0x55c096896ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.097+0000 7f29d0d14640 1 -- 172.21.15.28:0/2801303376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0965033f0 con 0x55c096899a30 2024-04-25T01:33:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.098+0000 7f29d0d14640 1 --2- 172.21.15.28:0/2801303376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c096899a30 0x55c096897410 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f29c402f040 tx=0x7f29c4003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a92792cea592d5e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.098+0000 7f29ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.098+0000 7f29ab7fe640 1 -- 172.21.15.28:0/2801303376 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c4033040 con 0x55c096899a30 2024-04-25T01:33:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.098+0000 7f29ab7fe640 1 -- 172.21.15.28:0/2801303376 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f29c402e390 con 0x55c096899a30 2024-04-25T01:33:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.098+0000 7f29d5564740 1 -- 172.21.15.28:0/2801303376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c0967fcbf0 con 0x55c096899a30 2024-04-25T01:33:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.098+0000 7f29ab7fe640 1 -- 172.21.15.28:0/2801303376 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f29c4050050 con 0x55c096899a30 2024-04-25T01:33:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.098+0000 7f29d5564740 10 client.15957 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.098+0000 7f29d5564740 1 -- 172.21.15.28:0/2801303376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c0964c30f0 con 0x55c096899a30 2024-04-25T01:33:39.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.098+0000 7f29ab7fe640 1 -- 172.21.15.28:0/2801303376 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c0967fcbf0 con 0x55c096899a30 2024-04-25T01:33:39.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.098+0000 7f29ab7fe640 1 -- 172.21.15.28:0/2801303376 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f29c404b030 con 0x55c096899a30 2024-04-25T01:33:39.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.098+0000 7f29d5564740 10 client.15957 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:39.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.098+0000 7f29d5564740 10 client.15957 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:39.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.098+0000 7f29d5564740 20 client.15957 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:39.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.098+0000 7f29d5564740 1 --2- 172.21.15.28:0/2801303376 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c0968a9d90 0x55c0968ac1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:39.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.098+0000 7f29d5564740 4 client.15957 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:33:39.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.098+0000 7f29d5564740 1 -- 172.21.15.28:0/2801303376 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0967f6f40 con 0x55c0968a9d90 2024-04-25T01:33:39.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.099+0000 7f29cbfff640 1 --2- 172.21.15.28:0/2801303376 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c0968a9d90 0x55c0968ac1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:39.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.099+0000 7f29ab7fe640 10 client.15957 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:39.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.099+0000 7f29cbfff640 1 --2- 172.21.15.28:0/2801303376 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c0968a9d90 0x55c0968ac1b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8d4816c2baab9a5 server_cookie=3f52c4af28936d7c in_seq=0 out_seq=0 2024-04-25T01:33:39.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.100+0000 7f29ab7fe640 1 -- 172.21.15.28:0/2801303376 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c0967f6f40 con 0x55c0968a9d90 2024-04-25T01:33:39.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.100+0000 7f29ab7fe640 10 client.15957 handle_command_reply: tid=0 2024-04-25T01:33:39.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.100+0000 7f29d5564740 1 client.15957 shutdown 2024-04-25T01:33:39.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.100+0000 7f29d5564740 1 -- 172.21.15.28:0/2801303376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c096899a30 msgr2=0x55c096897410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:39.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.100+0000 7f29d5564740 1 --2- 172.21.15.28:0/2801303376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c096899a30 0x55c096897410 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f29c402f040 tx=0x7f29c4003970 comp rx=0 tx=0).stop 2024-04-25T01:33:39.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.100+0000 7f29d5564740 1 -- 172.21.15.28:0/2801303376 shutdown_connections 2024-04-25T01:33:39.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.100+0000 7f29d5564740 1 --2- 172.21.15.28:0/2801303376 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c0968a9d90 0x55c0968ac1b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.100+0000 7f29d5564740 1 --2- 172.21.15.28:0/2801303376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c096899a30 0x55c096897410 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.100+0000 7f29d5564740 1 --2- 172.21.15.28:0/2801303376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c09688ad10 0x55c096896ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.100+0000 7f29ab7fe640 0 client.15957 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:39.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.100+0000 7f29d5564740 1 --2- 172.21.15.28:0/2801303376 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c09688a360 0x55c096896990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.100+0000 7f29d5564740 1 -- 172.21.15.28:0/2801303376 >> 172.21.15.28:0/2801303376 conn(0x55c096886950 msgr2=0x55c09688c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:39.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.100+0000 7f29d5564740 1 -- 172.21.15.28:0/2801303376 shutdown_connections 2024-04-25T01:33:39.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.100+0000 7f29d5564740 1 -- 172.21.15.28:0/2801303376 wait complete. 2024-04-25T01:33:39.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.100+0000 7f29d5564740 20 client.15957 trim_cache size 0 max 16384 2024-04-25T01:33:39.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.135+0000 7f29d5564740 1 Processor -- start 2024-04-25T01:33:39.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.135+0000 7f29d5564740 1 -- start start 2024-04-25T01:33:39.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.135+0000 7f29d5564740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c09688a360 0x55c0968a5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.135+0000 7f29d5564740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c09688ad10 0x55c0968958f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.135+0000 7f29d5564740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c096899a30 0x55c096897200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.135+0000 7f29d5564740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f29b800fd20 con 0x55c09688a360 2024-04-25T01:33:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.135+0000 7f29d5564740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f29b8011860 con 0x55c09688ad10 2024-04-25T01:33:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.135+0000 7f29d5564740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f29b80103f0 con 0x55c096899a30 2024-04-25T01:33:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.135+0000 7f29cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c09688a360 0x55c0968a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.135+0000 7f29cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c09688a360 0x55c0968a5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40406/0 (socket says 172.21.15.28:40406) 2024-04-25T01:33:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.135+0000 7f29cbfff640 1 -- 172.21.15.28:0/3129379253 learned_addr learned my addr 172.21.15.28:0/3129379253 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.135+0000 7f29cb7fe640 1 --2- 172.21.15.28:0/3129379253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c09688ad10 0x55c0968958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.135+0000 7f29d0d14640 1 --2- 172.21.15.28:0/3129379253 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c096899a30 0x55c096897200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.135+0000 7f29cbfff640 1 -- 172.21.15.28:0/3129379253 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c096899a30 msgr2=0x55c096897200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.135+0000 7f29cbfff640 1 --2- 172.21.15.28:0/3129379253 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c096899a30 0x55c096897200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.135+0000 7f29cbfff640 1 -- 172.21.15.28:0/3129379253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c09688ad10 msgr2=0x55c0968958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:39.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.135+0000 7f29cbfff640 1 --2- 172.21.15.28:0/3129379253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c09688ad10 0x55c0968958f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.136+0000 7f29cbfff640 1 -- 172.21.15.28:0/3129379253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c096886780 con 0x55c09688a360 2024-04-25T01:33:39.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.136+0000 7f29d0d14640 1 --2- 172.21.15.28:0/3129379253 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c096899a30 0x55c096897200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:39.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.136+0000 7f29cbfff640 1 --2- 172.21.15.28:0/3129379253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c09688a360 0x55c0968a5780 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f29bc0984a0 tx=0x7f29bc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8163b35d6c5e5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:39.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.136+0000 7f29b1ffb640 1 -- 172.21.15.28:0/3129379253 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc099b70 con 0x55c09688a360 2024-04-25T01:33:39.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.136+0000 7f29b1ffb640 1 -- 172.21.15.28:0/3129379253 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29bc099dd0 con 0x55c09688a360 2024-04-25T01:33:39.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.136+0000 7f29b1ffb640 1 -- 172.21.15.28:0/3129379253 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc0a42b0 con 0x55c09688a360 2024-04-25T01:33:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.137+0000 7f29d5564740 1 -- 172.21.15.28:0/3129379253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c09688a360 msgr2=0x55c0968a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.137+0000 7f29d5564740 1 --2- 172.21.15.28:0/3129379253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c09688a360 0x55c0968a5780 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f29bc0984a0 tx=0x7f29bc00cf90 comp rx=0 tx=0).stop 2024-04-25T01:33:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.137+0000 7f29d5564740 1 -- 172.21.15.28:0/3129379253 shutdown_connections 2024-04-25T01:33:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.137+0000 7f29d5564740 1 --2- 172.21.15.28:0/3129379253 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c096899a30 0x55c096897200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.137+0000 7f29d5564740 1 --2- 172.21.15.28:0/3129379253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c09688ad10 0x55c0968958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.137+0000 7f29d5564740 1 --2- 172.21.15.28:0/3129379253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c09688a360 0x55c0968a5780 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.137+0000 7f29d5564740 1 -- 172.21.15.28:0/3129379253 >> 172.21.15.28:0/3129379253 conn(0x55c096886950 msgr2=0x55c096885af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.137+0000 7f29d5564740 1 -- 172.21.15.28:0/3129379253 shutdown_connections 2024-04-25T01:33:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.137+0000 7f29d5564740 1 -- 172.21.15.28:0/3129379253 wait complete. 2024-04-25T01:33:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.137+0000 7f29d5564740 1 -- start start 2024-04-25T01:33:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29d5564740 1 Processor -- start 2024-04-25T01:33:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29d5564740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c09688a360 0x55c0968a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29d5564740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c09688ad10 0x55c096893240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29d5564740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c096899a30 0x55c096893780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29d5564740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f29b8011190 con 0x55c096899a30 2024-04-25T01:33:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29d5564740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f29b8010ac0 con 0x55c09688a360 2024-04-25T01:33:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29d5564740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f29b800dd00 con 0x55c09688ad10 2024-04-25T01:33:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29d5564740 10 client.0 _ll_register_callbacks cb 0x7f29c4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:33:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29d0d14640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c096899a30 0x55c096893780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29d0d14640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c096899a30 0x55c096893780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40416/0 (socket says 172.21.15.28:40416) 2024-04-25T01:33:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29d0d14640 1 -- 172.21.15.28:0/3053418645 learned_addr learned my addr 172.21.15.28:0/3053418645 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:33:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29cb7fe640 1 --2- 172.21.15.28:0/3053418645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c09688ad10 0x55c096893240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29cbfff640 1 --2- 172.21.15.28:0/3053418645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c09688a360 0x55c0968a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29d0d14640 1 -- 172.21.15.28:0/3053418645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c09688ad10 msgr2=0x55c096893240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29d0d14640 1 --2- 172.21.15.28:0/3053418645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c09688ad10 0x55c096893240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29d0d14640 1 -- 172.21.15.28:0/3053418645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c09688a360 msgr2=0x55c0968a1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:33:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29d0d14640 1 --2- 172.21.15.28:0/3053418645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c09688a360 0x55c0968a1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29d0d14640 1 -- 172.21.15.28:0/3053418645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c096893d40 con 0x55c096899a30 2024-04-25T01:33:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29cb7fe640 1 --2- 172.21.15.28:0/3053418645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c09688ad10 0x55c096893240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:33:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29cbfff640 1 --2- 172.21.15.28:0/3053418645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c09688a360 0x55c0968a1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:33:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29d0d14640 1 --2- 172.21.15.28:0/3053418645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c096899a30 0x55c096893780 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f29c4002230 tx=0x7f29c4003fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75eaac892c21fd72 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:33:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.138+0000 7f29ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:33:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.139+0000 7f29ab7fe640 1 -- 172.21.15.28:0/3053418645 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c4058020 con 0x55c096899a30 2024-04-25T01:33:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.139+0000 7f29ab7fe640 1 -- 172.21.15.28:0/3053418645 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f29c4004690 con 0x55c096899a30 2024-04-25T01:33:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.139+0000 7f29d5564740 1 -- 172.21.15.28:0/3053418645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29bc099dd0 con 0x55c096899a30 2024-04-25T01:33:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.139+0000 7f29ab7fe640 1 -- 172.21.15.28:0/3053418645 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f29c4063050 con 0x55c096899a30 2024-04-25T01:33:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.139+0000 7f29d5564740 10 client.15963 fetch_fsmap learned FSMap version 28 2024-04-25T01:33:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.139+0000 7f29d5564740 1 -- 172.21.15.28:0/3053418645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c096910060 con 0x55c096899a30 2024-04-25T01:33:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.139+0000 7f29ab7fe640 1 -- 172.21.15.28:0/3053418645 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29bc099dd0 con 0x55c096899a30 2024-04-25T01:33:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.139+0000 7f29ab7fe640 1 -- 172.21.15.28:0/3053418645 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f29c404b070 con 0x55c096899a30 2024-04-25T01:33:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.139+0000 7f29d5564740 10 client.15963 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:33:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.139+0000 7f29d5564740 10 client.15963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:33:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.139+0000 7f29d5564740 20 client.15963 populate_metadata read hostname 'smithi028' 2024-04-25T01:33:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.139+0000 7f29d5564740 1 --2- 172.21.15.28:0/3053418645 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c096910860 0x55c096912c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:33:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.139+0000 7f29d5564740 4 client.15963 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:33:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.139+0000 7f29d5564740 1 -- 172.21.15.28:0/3053418645 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c0967746f0 con 0x55c096910860 2024-04-25T01:33:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.139+0000 7f29cbfff640 1 --2- 172.21.15.28:0/3053418645 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c096910860 0x55c096912c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:33:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.140+0000 7f29cbfff640 1 --2- 172.21.15.28:0/3053418645 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c096910860 0x55c096912c80 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a14a13498761b52 server_cookie=4353c0f6d1f53669 in_seq=0 out_seq=0 2024-04-25T01:33:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.140+0000 7f29ab7fe640 10 client.15963 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.140+0000 7f29ab7fe640 1 -- 172.21.15.28:0/3053418645 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c0967746f0 con 0x55c096910860 2024-04-25T01:33:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.140+0000 7f29ab7fe640 10 client.15963 handle_command_reply: tid=0 2024-04-25T01:33:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.140+0000 7f29d5564740 1 client.15963 shutdown 2024-04-25T01:33:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.140+0000 7f29d5564740 1 -- 172.21.15.28:0/3053418645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c096899a30 msgr2=0x55c096893780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.140+0000 7f29d5564740 1 --2- 172.21.15.28:0/3053418645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c096899a30 0x55c096893780 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f29c4002230 tx=0x7f29c4003fd0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.140+0000 7f29d5564740 1 -- 172.21.15.28:0/3053418645 shutdown_connections 2024-04-25T01:33:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.140+0000 7f29d5564740 1 --2- 172.21.15.28:0/3053418645 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c096910860 0x55c096912c80 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.140+0000 7f29d5564740 1 --2- 172.21.15.28:0/3053418645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c096899a30 0x55c096893780 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.140+0000 7f29d5564740 1 --2- 172.21.15.28:0/3053418645 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c09688ad10 0x55c096893240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.140+0000 7f29d5564740 1 --2- 172.21.15.28:0/3053418645 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c09688a360 0x55c0968a1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.140+0000 7f29d5564740 1 -- 172.21.15.28:0/3053418645 >> 172.21.15.28:0/3053418645 conn(0x55c096886950 msgr2=0x55c09688ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:39.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.140+0000 7f29ab7fe640 0 client.15963 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:33:39.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.140+0000 7f29d5564740 1 -- 172.21.15.28:0/3053418645 shutdown_connections 2024-04-25T01:33:39.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.140+0000 7f29d5564740 1 -- 172.21.15.28:0/3053418645 wait complete. 2024-04-25T01:33:39.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.141+0000 7f29d5564740 20 client.15963 trim_cache size 0 max 16384 2024-04-25T01:33:39.124 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:33:39.124 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:33:39.124 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-25T01:33:39.125 INFO:teuthology.orchestra.run.smithi028.stdout: "9855a9d7-41ea-4000-9673-0aef7bf6e045": { 2024-04-25T01:33:39.125 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-25T01:33:39.125 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "9855a9d7-41ea-4000-9673-0aef7bf6e045", 2024-04-25T01:33:39.125 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-25T01:33:39.125 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:33:39.125 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:33:39.125 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:33:39.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.141+0000 7f29d2777640 1 -- 172.21.15.28:0/1061956441 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f29b40a2d90 msgr2=0x7f29b40a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:39.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.141+0000 7f29d2777640 1 --2- 172.21.15.28:0/1061956441 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f29b40a2d90 0x7f29b40a5250 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f29cc0fff20 tx=0x7f29c000c4a0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.141+0000 7f29d2777640 1 -- 172.21.15.28:0/1061956441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29cc1029c0 msgr2=0x7f29cc197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:33:39.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.141+0000 7f29d2777640 1 --2- 172.21.15.28:0/1061956441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29cc1029c0 0x7f29cc197ee0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00c920 tx=0x7f29bc00cde0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.141+0000 7f29d2777640 1 -- 172.21.15.28:0/1061956441 shutdown_connections 2024-04-25T01:33:39.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.141+0000 7f29d2777640 1 --2- 172.21.15.28:0/1061956441 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f29b40a2d90 0x7f29b40a5250 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.141+0000 7f29d2777640 1 --2- 172.21.15.28:0/1061956441 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f29cc10bd00 0x7f29cc19c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.141+0000 7f29d2777640 1 --2- 172.21.15.28:0/1061956441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29cc1032e0 0x7f29cc198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.141+0000 7f29d2777640 1 --2- 172.21.15.28:0/1061956441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29cc1029c0 0x7f29cc197ee0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:33:39.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.141+0000 7f29d2777640 1 -- 172.21.15.28:0/1061956441 >> 172.21.15.28:0/1061956441 conn(0x7f29cc077740 msgr2=0x7f29cc0fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:33:39.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.141+0000 7f29d2777640 1 -- 172.21.15.28:0/1061956441 shutdown_connections 2024-04-25T01:33:39.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:33:39.141+0000 7f29d2777640 1 -- 172.21.15.28:0/1061956441 wait complete. 2024-04-25T01:33:39.132 INFO:tasks.cephfs.filesystem:scrub status for tag:9855a9d7-41ea-4000-9673-0aef7bf6e045 - {'path': '/', 'tag': '9855a9d7-41ea-4000-9673-0aef7bf6e045', 'options': 'recursive,force'} 2024-04-25T01:33:39.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:39 smithi130 ceph-mon[32702]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:33:39.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:39 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:33:39.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:39 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:39.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:39 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:33:39.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:39 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:39.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:39 smithi032 ceph-mon[32461]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:33:39.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:39 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:33:39.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:39 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:39.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:39 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:33:39.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:39 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:40.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:39 smithi028 ceph-mon[29390]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:33:40.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:39 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:33:40.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:39 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:40.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:39 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:33:40.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:39 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:33:41.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:41 smithi130 ceph-mon[32702]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:33:41.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:41 smithi032 ceph-mon[32461]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:33:42.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:41 smithi028 ceph-mon[29390]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:33:43.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:43 smithi130 ceph-mon[32702]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-04-25T01:33:43.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:43 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:33:43.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:43 smithi032 ceph-mon[32461]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-04-25T01:33:43.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:43 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:33:44.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:43 smithi028 ceph-mon[29390]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-04-25T01:33:44.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:43 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:33:45.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:45 smithi130 ceph-mon[32702]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:33:45.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:45 smithi032 ceph-mon[32461]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:33:46.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:45 smithi028 ceph-mon[29390]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:33:47.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:47 smithi130 ceph-mon[32702]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 682 B/s wr, 12 op/s 2024-04-25T01:33:47.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:47 smithi032 ceph-mon[32461]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 682 B/s wr, 12 op/s 2024-04-25T01:33:48.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:47 smithi028 ceph-mon[29390]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 682 B/s wr, 12 op/s 2024-04-25T01:33:49.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:49 smithi130 ceph-mon[32702]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-25T01:33:49.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:49 smithi032 ceph-mon[32461]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-25T01:33:50.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:49 smithi028 ceph-mon[29390]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-25T01:33:51.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:51 smithi130 ceph-mon[32702]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 8 op/s 2024-04-25T01:33:51.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:51 smithi032 ceph-mon[32461]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 8 op/s 2024-04-25T01:33:52.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:51 smithi028 ceph-mon[29390]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 8 op/s 2024-04-25T01:33:53.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:53 smithi130 ceph-mon[32702]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 682 B/s wr, 13 op/s 2024-04-25T01:33:53.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:53 smithi032 ceph-mon[32461]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 682 B/s wr, 13 op/s 2024-04-25T01:33:54.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:53 smithi028 ceph-mon[29390]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 682 B/s wr, 13 op/s 2024-04-25T01:33:55.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:55 smithi130 ceph-mon[32702]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-04-25T01:33:55.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:55 smithi032 ceph-mon[32461]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-04-25T01:33:56.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:55 smithi028 ceph-mon[29390]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-04-25T01:33:57.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:57 smithi130 ceph-mon[32702]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-04-25T01:33:57.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:57 smithi032 ceph-mon[32461]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-04-25T01:33:58.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:57 smithi028 ceph-mon[29390]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 12 op/s 2024-04-25T01:33:59.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:33:59 smithi032 ceph-mon[32461]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:34:00.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:33:59 smithi028 ceph-mon[29390]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:34:00.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:33:59 smithi130 ceph-mon[32702]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:34:00.561 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:34:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:34:00.105+0000 7fae223ea700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:34:00.561 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:34:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:34:00.105+0000 7fae223ea700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:34:00.561 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:34:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:34:00.105+0000 7fae223ea700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:34:00.561 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:34:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:34:00.105+0000 7fae223ea700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:34:00.561 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:34:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:34:00.105+0000 7fae223ea700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:34:01.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:01 smithi032 ceph-mon[32461]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:34:01.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:01 smithi032 ceph-mon[32461]: mgrmap e24: x(active, since 60s), standbys: z, y 2024-04-25T01:34:02.010 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:01 smithi130 ceph-mon[32702]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:34:02.010 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:01 smithi130 ceph-mon[32702]: mgrmap e24: x(active, since 60s), standbys: z, y 2024-04-25T01:34:02.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:01 smithi028 ceph-mon[29390]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:34:02.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:01 smithi028 ceph-mon[29390]: mgrmap e24: x(active, since 60s), standbys: z, y 2024-04-25T01:34:03.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:03 smithi032 ceph-mon[32461]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:34:04.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:03 smithi028 ceph-mon[29390]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:34:04.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:03 smithi130 ceph-mon[32702]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:34:05.982 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:05 smithi032 ceph-mon[32461]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:34:05.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:05 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:34:06.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:05 smithi028 ceph-mon[29390]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:34:06.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:05 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:34:06.207 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:05 smithi130 ceph-mon[32702]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:34:06.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:05 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:34:07.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:07 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:07 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:07 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:07 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:07 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:07 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:07 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:34:07.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:07 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:34:07.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:07 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:07 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:07 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:07 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:07 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:07 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:07 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:07 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:34:07.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:07 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:34:07.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:07 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:07 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:07 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:07 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:07 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:07 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:07 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:07.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:07 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:34:07.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:07 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:34:07.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:07 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:34:08.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:08 smithi032 ceph-mon[32461]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-04-25T01:34:08.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:08 smithi028 ceph-mon[29390]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-04-25T01:34:08.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:08 smithi130 ceph-mon[32702]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-04-25T01:34:09.134 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:34:09.134 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:34:09.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.269+0000 7f89ee3ce640 1 -- 172.21.15.28:0/2576286137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89e8102150 msgr2=0x7f89e810e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.269+0000 7f89ee3ce640 1 --2- 172.21.15.28:0/2576286137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89e8102150 0x7f89e810e660 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f89e0009cd0 tx=0x7f89e002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.269+0000 7f89ee3ce640 1 -- 172.21.15.28:0/2576286137 shutdown_connections 2024-04-25T01:34:09.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.269+0000 7f89ee3ce640 1 --2- 172.21.15.28:0/2576286137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89e8102150 0x7f89e810e660 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.269+0000 7f89ee3ce640 1 --2- 172.21.15.28:0/2576286137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e8101790 0x7f89e8101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.269+0000 7f89ee3ce640 1 --2- 172.21.15.28:0/2576286137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f89e81077a0 0x7f89e8107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.269+0000 7f89ee3ce640 1 -- 172.21.15.28:0/2576286137 >> 172.21.15.28:0/2576286137 conn(0x7f89e80fd2a0 msgr2=0x7f89e80ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.269+0000 7f89ee3ce640 1 -- 172.21.15.28:0/2576286137 shutdown_connections 2024-04-25T01:34:09.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.269+0000 7f89ee3ce640 1 -- 172.21.15.28:0/2576286137 wait complete. 2024-04-25T01:34:09.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.270+0000 7f89ee3ce640 1 Processor -- start 2024-04-25T01:34:09.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.270+0000 7f89ee3ce640 1 -- start start 2024-04-25T01:34:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.271+0000 7f89ee3ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89e8101790 0x7f89e8198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.271+0000 7f89ee3ce640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e8102150 0x7f89e8198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.271+0000 7f89ee3ce640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f89e81077a0 0x7f89e819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.271+0000 7f89ee3ce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89e8111bb0 con 0x7f89e8101790 2024-04-25T01:34:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.271+0000 7f89ee3ce640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f89e8111a30 con 0x7f89e8102150 2024-04-25T01:34:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.271+0000 7f89ee3ce640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f89e8111d30 con 0x7f89e81077a0 2024-04-25T01:34:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.271+0000 7f89e7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89e8101790 0x7f89e8198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.271+0000 7f89e7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89e8101790 0x7f89e8198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58688/0 (socket says 172.21.15.28:58688) 2024-04-25T01:34:09.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.271+0000 7f89e7fff640 1 -- 172.21.15.28:0/2894091883 learned_addr learned my addr 172.21.15.28:0/2894091883 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:09.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.271+0000 7f89ec96b640 1 --2- 172.21.15.28:0/2894091883 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f89e81077a0 0x7f89e819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.271+0000 7f89e77fe640 1 --2- 172.21.15.28:0/2894091883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e8102150 0x7f89e8198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.271+0000 7f89e7fff640 1 -- 172.21.15.28:0/2894091883 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f89e81077a0 msgr2=0x7f89e819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.271+0000 7f89e7fff640 1 --2- 172.21.15.28:0/2894091883 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f89e81077a0 0x7f89e819c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.271+0000 7f89e7fff640 1 -- 172.21.15.28:0/2894091883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e8102150 msgr2=0x7f89e8198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.271+0000 7f89e7fff640 1 --2- 172.21.15.28:0/2894091883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e8102150 0x7f89e8198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.271+0000 7f89e7fff640 1 -- 172.21.15.28:0/2894091883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e819d000 con 0x7f89e8101790 2024-04-25T01:34:09.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.272+0000 7f89e7fff640 1 --2- 172.21.15.28:0/2894091883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89e8101790 0x7f89e8198010 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f89d800d770 tx=0x7f89d800db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.272+0000 7f89e57fa640 1 -- 172.21.15.28:0/2894091883 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d801c070 con 0x7f89e8101790 2024-04-25T01:34:09.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.272+0000 7f89e57fa640 1 -- 172.21.15.28:0/2894091883 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89d8016d60 con 0x7f89e8101790 2024-04-25T01:34:09.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.272+0000 7f89e57fa640 1 -- 172.21.15.28:0/2894091883 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8004a00 con 0x7f89e8101790 2024-04-25T01:34:09.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.272+0000 7f89ee3ce640 1 -- 172.21.15.28:0/2894091883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89e819d2f0 con 0x7f89e8101790 2024-04-25T01:34:09.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.272+0000 7f89ee3ce640 1 -- 172.21.15.28:0/2894091883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89e8073970 con 0x7f89e8101790 2024-04-25T01:34:09.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.275+0000 7f89f11ba740 1 Processor -- start 2024-04-25T01:34:09.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.275+0000 7f89f11ba740 1 -- start start 2024-04-25T01:34:09.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.275+0000 7f89e57fa640 1 -- 172.21.15.28:0/2894091883 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f89d8004ba0 con 0x7f89e8101790 2024-04-25T01:34:09.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.275+0000 7f89f11ba740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d389630 0x556a4d389a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.275+0000 7f89f11ba740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389f50 0x556a4d3940e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.275+0000 7f89f11ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d394620 0x556a4d396a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.275+0000 7f89f11ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556a4d2ee6b0 con 0x556a4d394620 2024-04-25T01:34:09.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.275+0000 7f89f11ba740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556a4d296880 con 0x556a4d389f50 2024-04-25T01:34:09.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.275+0000 7f89f11ba740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x556a4d2ea670 con 0x556a4d389630 2024-04-25T01:34:09.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.276+0000 7f89e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d394620 0x556a4d396a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.276+0000 7f89e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d394620 0x556a4d396a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58702/0 (socket says 172.21.15.28:58702) 2024-04-25T01:34:09.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.276+0000 7f89e77fe640 1 -- 172.21.15.28:0/2097121719 learned_addr learned my addr 172.21.15.28:0/2097121719 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:09.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.276+0000 7f89e7fff640 1 --2- 172.21.15.28:0/2097121719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389f50 0x556a4d3940e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.276+0000 7f89ec96b640 1 --2- 172.21.15.28:0/2097121719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d389630 0x556a4d389a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.276+0000 7f89e77fe640 1 -- 172.21.15.28:0/2097121719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d389630 msgr2=0x556a4d389a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.276+0000 7f89e77fe640 1 --2- 172.21.15.28:0/2097121719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d389630 0x556a4d389a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.276+0000 7f89e77fe640 1 -- 172.21.15.28:0/2097121719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389f50 msgr2=0x556a4d3940e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.276+0000 7f89e77fe640 1 --2- 172.21.15.28:0/2097121719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389f50 0x556a4d3940e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.276+0000 7f89e77fe640 1 -- 172.21.15.28:0/2097121719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a4d0cc7d0 con 0x556a4d394620 2024-04-25T01:34:09.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.276+0000 7f89e77fe640 1 --2- 172.21.15.28:0/2097121719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d394620 0x556a4d396a10 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00aca0 tx=0x7f89dc00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a54270ae44e05300 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.277+0000 7f89e57fa640 1 --2- 172.21.15.28:0/2894091883 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f89d00a2ea0 0x7f89d00a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.277+0000 7f89ce7fc640 1 -- 172.21.15.28:0/2097121719 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc01d070 con 0x556a4d394620 2024-04-25T01:34:09.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.277+0000 7f89ce7fc640 1 -- 172.21.15.28:0/2097121719 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89dc018b20 con 0x556a4d394620 2024-04-25T01:34:09.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.277+0000 7f89ce7fc640 1 -- 172.21.15.28:0/2097121719 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc010cb0 con 0x556a4d394620 2024-04-25T01:34:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.277+0000 7f89ec96b640 1 --2- 172.21.15.28:0/2894091883 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f89d00a2ea0 0x7f89d00a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.277+0000 7f89e57fa640 1 -- 172.21.15.28:0/2894091883 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f89d80d7950 con 0x7f89e8101790 2024-04-25T01:34:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.278+0000 7f89f11ba740 1 -- 172.21.15.28:0/2097121719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d394620 msgr2=0x556a4d396a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.278+0000 7f89ec96b640 1 --2- 172.21.15.28:0/2894091883 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f89d00a2ea0 0x7f89d00a5360 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f89e0009a60 tx=0x7f89e0005fa0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.278+0000 7f89f11ba740 1 --2- 172.21.15.28:0/2097121719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d394620 0x556a4d396a10 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00aca0 tx=0x7f89dc00f790 comp rx=0 tx=0).stop 2024-04-25T01:34:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.278+0000 7f89f11ba740 1 -- 172.21.15.28:0/2097121719 shutdown_connections 2024-04-25T01:34:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.278+0000 7f89f11ba740 1 --2- 172.21.15.28:0/2097121719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d394620 0x556a4d396a10 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.278+0000 7f89f11ba740 1 --2- 172.21.15.28:0/2097121719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389f50 0x556a4d3940e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.278+0000 7f89f11ba740 1 --2- 172.21.15.28:0/2097121719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d389630 0x556a4d389a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.278+0000 7f89f11ba740 1 -- 172.21.15.28:0/2097121719 >> 172.21.15.28:0/2097121719 conn(0x556a4d385950 msgr2=0x556a4d385d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.278+0000 7f89f11ba740 1 -- 172.21.15.28:0/2097121719 shutdown_connections 2024-04-25T01:34:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.278+0000 7f89f11ba740 1 -- 172.21.15.28:0/2097121719 wait complete. 2024-04-25T01:34:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.278+0000 7f89f11ba740 1 -- start start 2024-04-25T01:34:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.278+0000 7f89f11ba740 1 Processor -- start 2024-04-25T01:34:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.279+0000 7f89f11ba740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389630 0x556a4d3915a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.279+0000 7f89f11ba740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d389f50 0x556a4d391ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.279+0000 7f89f11ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d394620 0x556a4d392020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.279+0000 7f89f11ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556a4d397410 con 0x556a4d394620 2024-04-25T01:34:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.279+0000 7f89f11ba740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556a4d2a9f50 con 0x556a4d389630 2024-04-25T01:34:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.279+0000 7f89f11ba740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x556a4d397590 con 0x556a4d389f50 2024-04-25T01:34:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.279+0000 7f89f11ba740 10 client.0 _ll_register_callbacks cb 0x556a4d2e5000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.279+0000 7f89e7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389630 0x556a4d3915a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.279+0000 7f89ec96b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d394620 0x556a4d392020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.279+0000 7f89e7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389630 0x556a4d3915a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:45836/0 (socket says 172.21.15.28:45836) 2024-04-25T01:34:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.279+0000 7f89e7fff640 1 -- 172.21.15.28:0/4032887704 learned_addr learned my addr 172.21.15.28:0/4032887704 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.279+0000 7f89e77fe640 1 --2- 172.21.15.28:0/4032887704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d389f50 0x556a4d391ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.279+0000 7f89ec96b640 1 -- 172.21.15.28:0/4032887704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d389f50 msgr2=0x556a4d391ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.279+0000 7f89ec96b640 1 --2- 172.21.15.28:0/4032887704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d389f50 0x556a4d391ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.279+0000 7f89ec96b640 1 -- 172.21.15.28:0/4032887704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389630 msgr2=0x556a4d3915a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.279+0000 7f89ec96b640 1 --2- 172.21.15.28:0/4032887704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389630 0x556a4d3915a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.279+0000 7f89ec96b640 1 -- 172.21.15.28:0/4032887704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a4d0023f0 con 0x556a4d394620 2024-04-25T01:34:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.279+0000 7f89e7fff640 1 --2- 172.21.15.28:0/4032887704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389630 0x556a4d3915a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.279+0000 7f89e77fe640 1 --2- 172.21.15.28:0/4032887704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d389f50 0x556a4d391ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.280+0000 7f89ec96b640 1 --2- 172.21.15.28:0/4032887704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d394620 0x556a4d392020 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f89e0009ca0 tx=0x7f89e002a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83000b1863b2ceda server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.280+0000 7f89c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.280+0000 7f89c77fe640 1 -- 172.21.15.28:0/4032887704 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e0055060 con 0x556a4d394620 2024-04-25T01:34:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.280+0000 7f89f11ba740 1 -- 172.21.15.28:0/4032887704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89dc018b20 con 0x556a4d394620 2024-04-25T01:34:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.280+0000 7f89c77fe640 1 -- 172.21.15.28:0/4032887704 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f89e0003e00 con 0x556a4d394620 2024-04-25T01:34:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.280+0000 7f89c77fe640 1 -- 172.21.15.28:0/4032887704 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f89e005c050 con 0x556a4d394620 2024-04-25T01:34:09.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.280+0000 7f89f11ba740 10 client.15981 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:09.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.280+0000 7f89f11ba740 1 -- 172.21.15.28:0/4032887704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556a4cfc20f0 con 0x556a4d394620 2024-04-25T01:34:09.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.280+0000 7f89c77fe640 1 -- 172.21.15.28:0/4032887704 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89dc018b20 con 0x556a4d394620 2024-04-25T01:34:09.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.281+0000 7f89c77fe640 1 -- 172.21.15.28:0/4032887704 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f89e004f030 con 0x556a4d394620 2024-04-25T01:34:09.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.281+0000 7f89f11ba740 10 client.15981 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:09.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.281+0000 7f89f11ba740 10 client.15981 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:09.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.281+0000 7f89f11ba740 20 client.15981 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:09.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.281+0000 7f89f11ba740 1 --2- 172.21.15.28:0/4032887704 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556a4d3a4960 0x556a4d3a6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.281+0000 7f89f11ba740 4 client.15981 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:09.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.281+0000 7f89f11ba740 1 -- 172.21.15.28:0/4032887704 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a4d2fbbf0 con 0x556a4d3a4960 2024-04-25T01:34:09.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.281+0000 7f89e7fff640 1 --2- 172.21.15.28:0/4032887704 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556a4d3a4960 0x556a4d3a6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.281+0000 7f89c77fe640 10 client.15981 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:09.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.281+0000 7f89e7fff640 1 --2- 172.21.15.28:0/4032887704 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556a4d3a4960 0x556a4d3a6d70 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4f3b325031ffb4 server_cookie=1018b81ccd5c618c in_seq=0 out_seq=0 2024-04-25T01:34:09.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.282+0000 7f89c77fe640 1 -- 172.21.15.28:0/4032887704 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556a4d2fbbf0 con 0x556a4d3a4960 2024-04-25T01:34:09.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.282+0000 7f89c77fe640 10 client.15981 handle_command_reply: tid=0 2024-04-25T01:34:09.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.282+0000 7f89f11ba740 1 client.15981 shutdown 2024-04-25T01:34:09.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.282+0000 7f89f11ba740 1 -- 172.21.15.28:0/4032887704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d394620 msgr2=0x556a4d392020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.282+0000 7f89f11ba740 1 --2- 172.21.15.28:0/4032887704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d394620 0x556a4d392020 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f89e0009ca0 tx=0x7f89e002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.282+0000 7f89f11ba740 1 -- 172.21.15.28:0/4032887704 shutdown_connections 2024-04-25T01:34:09.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.282+0000 7f89f11ba740 1 --2- 172.21.15.28:0/4032887704 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556a4d3a4960 0x556a4d3a6d70 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.282+0000 7f89f11ba740 1 --2- 172.21.15.28:0/4032887704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d394620 0x556a4d392020 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.282+0000 7f89f11ba740 1 --2- 172.21.15.28:0/4032887704 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d389f50 0x556a4d391ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.282+0000 7f89c77fe640 0 client.15981 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:09.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.282+0000 7f89f11ba740 1 --2- 172.21.15.28:0/4032887704 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389630 0x556a4d3915a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.282+0000 7f89f11ba740 1 -- 172.21.15.28:0/4032887704 >> 172.21.15.28:0/4032887704 conn(0x556a4d385950 msgr2=0x556a4d38b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.283+0000 7f89f11ba740 1 -- 172.21.15.28:0/4032887704 shutdown_connections 2024-04-25T01:34:09.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.283+0000 7f89f11ba740 1 -- 172.21.15.28:0/4032887704 wait complete. 2024-04-25T01:34:09.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.283+0000 7f89f11ba740 20 client.15981 trim_cache size 0 max 16384 2024-04-25T01:34:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.318+0000 7f89f11ba740 1 Processor -- start 2024-04-25T01:34:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.318+0000 7f89f11ba740 1 -- start start 2024-04-25T01:34:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.318+0000 7f89f11ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d389630 0x556a4d3a0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.318+0000 7f89f11ba740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389f50 0x556a4d38bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.318+0000 7f89f11ba740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d394620 0x556a4d384970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.318+0000 7f89f11ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89e81ac7b0 con 0x556a4d389630 2024-04-25T01:34:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.318+0000 7f89f11ba740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f89e81ae2f0 con 0x556a4d389f50 2024-04-25T01:34:09.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.318+0000 7f89f11ba740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f89e81ace80 con 0x556a4d394620 2024-04-25T01:34:09.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.318+0000 7f89e7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d389630 0x556a4d3a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.318+0000 7f89e7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d389630 0x556a4d3a0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58718/0 (socket says 172.21.15.28:58718) 2024-04-25T01:34:09.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.318+0000 7f89e7fff640 1 -- 172.21.15.28:0/1482819917 learned_addr learned my addr 172.21.15.28:0/1482819917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:09.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.318+0000 7f89ec96b640 1 --2- 172.21.15.28:0/1482819917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d394620 0x556a4d384970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.319+0000 7f89e77fe640 1 --2- 172.21.15.28:0/1482819917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389f50 0x556a4d38bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.319+0000 7f89e7fff640 1 -- 172.21.15.28:0/1482819917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d394620 msgr2=0x556a4d384970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.319+0000 7f89e7fff640 1 --2- 172.21.15.28:0/1482819917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d394620 0x556a4d384970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.319+0000 7f89e7fff640 1 -- 172.21.15.28:0/1482819917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389f50 msgr2=0x556a4d38bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.319+0000 7f89e7fff640 1 --2- 172.21.15.28:0/1482819917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389f50 0x556a4d38bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.319+0000 7f89e7fff640 1 -- 172.21.15.28:0/1482819917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a4d396d80 con 0x556a4d389630 2024-04-25T01:34:09.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.319+0000 7f89e7fff640 1 --2- 172.21.15.28:0/1482819917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d389630 0x556a4d3a0300 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f89d800d740 tx=0x7f89d808e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=748b7aff90440c25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.319+0000 7f89cdffb640 1 -- 172.21.15.28:0/1482819917 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d80a3350 con 0x556a4d389630 2024-04-25T01:34:09.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.319+0000 7f89cdffb640 1 -- 172.21.15.28:0/1482819917 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89d80a40f0 con 0x556a4d389630 2024-04-25T01:34:09.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.319+0000 7f89cdffb640 1 -- 172.21.15.28:0/1482819917 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d80a1cf0 con 0x556a4d389630 2024-04-25T01:34:09.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.320+0000 7f89f11ba740 1 -- 172.21.15.28:0/1482819917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d389630 msgr2=0x556a4d3a0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.320+0000 7f89f11ba740 1 --2- 172.21.15.28:0/1482819917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d389630 0x556a4d3a0300 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f89d800d740 tx=0x7f89d808e040 comp rx=0 tx=0).stop 2024-04-25T01:34:09.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.320+0000 7f89f11ba740 1 -- 172.21.15.28:0/1482819917 shutdown_connections 2024-04-25T01:34:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.320+0000 7f89f11ba740 1 --2- 172.21.15.28:0/1482819917 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d394620 0x556a4d384970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.320+0000 7f89f11ba740 1 --2- 172.21.15.28:0/1482819917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389f50 0x556a4d38bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.320+0000 7f89f11ba740 1 --2- 172.21.15.28:0/1482819917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d389630 0x556a4d3a0300 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.320+0000 7f89f11ba740 1 -- 172.21.15.28:0/1482819917 >> 172.21.15.28:0/1482819917 conn(0x556a4d385950 msgr2=0x556a4d384260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.320+0000 7f89f11ba740 1 -- 172.21.15.28:0/1482819917 shutdown_connections 2024-04-25T01:34:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.320+0000 7f89f11ba740 1 -- 172.21.15.28:0/1482819917 wait complete. 2024-04-25T01:34:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.321+0000 7f89f11ba740 1 -- start start 2024-04-25T01:34:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.321+0000 7f89f11ba740 1 Processor -- start 2024-04-25T01:34:09.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.321+0000 7f89f11ba740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d389630 0x556a4d38e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.321+0000 7f89f11ba740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389f50 0x556a4d38e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.321+0000 7f89f11ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d394620 0x556a4d38ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.321+0000 7f89f11ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89e81adc20 con 0x556a4d394620 2024-04-25T01:34:09.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.321+0000 7f89f11ba740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f89e81ad550 con 0x556a4d389f50 2024-04-25T01:34:09.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.321+0000 7f89f11ba740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f89e81aa790 con 0x556a4d389630 2024-04-25T01:34:09.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.321+0000 7f89f11ba740 10 client.0 _ll_register_callbacks cb 0x7f89e0009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.322+0000 7f89e7fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d389630 0x556a4d38e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.322+0000 7f89ec96b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d394620 0x556a4d38ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.322+0000 7f89e7fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d389630 0x556a4d38e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.28:46600/0 (socket says 172.21.15.28:46600) 2024-04-25T01:34:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.322+0000 7f89e7fff640 1 -- 172.21.15.28:0/1210156263 learned_addr learned my addr 172.21.15.28:0/1210156263 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.322+0000 7f89e77fe640 1 --2- 172.21.15.28:0/1210156263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389f50 0x556a4d38e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.322+0000 7f89e7fff640 1 -- 172.21.15.28:0/1210156263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389f50 msgr2=0x556a4d38e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.322+0000 7f89e7fff640 1 --2- 172.21.15.28:0/1210156263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389f50 0x556a4d38e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.322+0000 7f89e7fff640 1 -- 172.21.15.28:0/1210156263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d394620 msgr2=0x556a4d38ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.322+0000 7f89e7fff640 1 --2- 172.21.15.28:0/1210156263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d394620 0x556a4d38ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.322+0000 7f89e7fff640 1 -- 172.21.15.28:0/1210156263 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a4d409fe0 con 0x556a4d389630 2024-04-25T01:34:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.322+0000 7f89ec96b640 1 --2- 172.21.15.28:0/1210156263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d394620 0x556a4d38ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.322+0000 7f89e7fff640 1 --2- 172.21.15.28:0/1210156263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d389630 0x556a4d38e380 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009610 tx=0x7f89d80a3350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd9f5f3e7dd1e6f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.322+0000 7f89c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-25T01:34:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.322+0000 7f89c77fe640 1 -- 172.21.15.28:0/1210156263 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d80bb060 con 0x556a4d389630 2024-04-25T01:34:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.322+0000 7f89f11ba740 1 -- 172.21.15.28:0/1210156263 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89d80a40f0 con 0x556a4d389630 2024-04-25T01:34:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.322+0000 7f89c77fe640 1 -- 172.21.15.28:0/1210156263 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f89d80dd1a0 con 0x556a4d389630 2024-04-25T01:34:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.323+0000 7f89c77fe640 1 -- 172.21.15.28:0/1210156263 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f89d80a39f0 con 0x556a4d389630 2024-04-25T01:34:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.323+0000 7f89f11ba740 10 client.25459 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.323+0000 7f89f11ba740 1 -- 172.21.15.28:0/1210156263 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556a4d40b2d0 con 0x556a4d389630 2024-04-25T01:34:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.323+0000 7f89c77fe640 1 -- 172.21.15.28:0/1210156263 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89d80a40f0 con 0x556a4d389630 2024-04-25T01:34:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.323+0000 7f89c77fe640 1 -- 172.21.15.28:0/1210156263 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f89d80c2200 con 0x556a4d389630 2024-04-25T01:34:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.324+0000 7f89f11ba740 10 client.25459 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.324+0000 7f89f11ba740 10 client.25459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.324+0000 7f89f11ba740 20 client.25459 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.324+0000 7f89f11ba740 1 --2- 172.21.15.28:0/1210156263 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556a4d3a4960 0x556a4d40d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.324+0000 7f89f11ba740 4 client.25459 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:34:09.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.324+0000 7f89f11ba740 1 -- 172.21.15.28:0/1210156263 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556a4d2f5f40 con 0x556a4d3a4960 2024-04-25T01:34:09.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.324+0000 7f89e77fe640 1 --2- 172.21.15.28:0/1210156263 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556a4d3a4960 0x556a4d40d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.324+0000 7f89e77fe640 1 --2- 172.21.15.28:0/1210156263 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556a4d3a4960 0x556a4d40d9b0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=684cc971f149442f server_cookie=39e80629f83a04d6 in_seq=0 out_seq=0 2024-04-25T01:34:09.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.324+0000 7f89c77fe640 10 client.25459 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:09.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.324+0000 7f89c77fe640 1 -- 172.21.15.28:0/1210156263 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556a4d2f5f40 con 0x556a4d3a4960 2024-04-25T01:34:09.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.324+0000 7f89c77fe640 10 client.25459 handle_command_reply: tid=0 2024-04-25T01:34:09.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.325+0000 7f89f11ba740 1 client.25459 shutdown 2024-04-25T01:34:09.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.325+0000 7f89f11ba740 1 -- 172.21.15.28:0/1210156263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d389630 msgr2=0x556a4d38e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.325+0000 7f89f11ba740 1 --2- 172.21.15.28:0/1210156263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d389630 0x556a4d38e380 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009610 tx=0x7f89d80a3350 comp rx=0 tx=0).stop 2024-04-25T01:34:09.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.325+0000 7f89f11ba740 1 -- 172.21.15.28:0/1210156263 shutdown_connections 2024-04-25T01:34:09.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.325+0000 7f89f11ba740 1 --2- 172.21.15.28:0/1210156263 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556a4d3a4960 0x556a4d40d9b0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.325+0000 7f89c77fe640 0 client.25459 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:09.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.325+0000 7f89f11ba740 1 --2- 172.21.15.28:0/1210156263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a4d394620 0x556a4d38ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.325+0000 7f89f11ba740 1 --2- 172.21.15.28:0/1210156263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556a4d389f50 0x556a4d38e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.325+0000 7f89f11ba740 1 --2- 172.21.15.28:0/1210156263 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556a4d389630 0x556a4d38e380 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.325+0000 7f89f11ba740 1 -- 172.21.15.28:0/1210156263 >> 172.21.15.28:0/1210156263 conn(0x556a4d385950 msgr2=0x556a4d384af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.325+0000 7f89f11ba740 1 -- 172.21.15.28:0/1210156263 shutdown_connections 2024-04-25T01:34:09.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.325+0000 7f89f11ba740 1 -- 172.21.15.28:0/1210156263 wait complete. 2024-04-25T01:34:09.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.325+0000 7f89f11ba740 20 client.25459 trim_cache size 0 max 16384 2024-04-25T01:34:09.316 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:34:09.316 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:34:09.316 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:34:09.316 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:34:09.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.326+0000 7f89ee3ce640 1 -- 172.21.15.28:0/2894091883 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f89d00a2ea0 msgr2=0x7f89d00a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.326+0000 7f89ee3ce640 1 --2- 172.21.15.28:0/2894091883 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f89d00a2ea0 0x7f89d00a5360 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f89e0009a60 tx=0x7f89e0005fa0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.326+0000 7f89ee3ce640 1 -- 172.21.15.28:0/2894091883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89e8101790 msgr2=0x7f89e8198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.326+0000 7f89ee3ce640 1 --2- 172.21.15.28:0/2894091883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89e8101790 0x7f89e8198010 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f89d800d770 tx=0x7f89d800db70 comp rx=0 tx=0).stop 2024-04-25T01:34:09.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.326+0000 7f89ee3ce640 1 -- 172.21.15.28:0/2894091883 shutdown_connections 2024-04-25T01:34:09.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.326+0000 7f89ee3ce640 1 --2- 172.21.15.28:0/2894091883 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f89d00a2ea0 0x7f89d00a5360 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.326+0000 7f89ee3ce640 1 --2- 172.21.15.28:0/2894091883 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f89e81077a0 0x7f89e819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.326+0000 7f89ee3ce640 1 --2- 172.21.15.28:0/2894091883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89e8102150 0x7f89e8198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.326+0000 7f89ee3ce640 1 --2- 172.21.15.28:0/2894091883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89e8101790 0x7f89e8198010 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.326+0000 7f89ee3ce640 1 -- 172.21.15.28:0/2894091883 >> 172.21.15.28:0/2894091883 conn(0x7f89e80fd2a0 msgr2=0x7f89e80fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.326+0000 7f89ee3ce640 1 -- 172.21.15.28:0/2894091883 shutdown_connections 2024-04-25T01:34:09.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.326+0000 7f89ee3ce640 1 -- 172.21.15.28:0/2894091883 wait complete. 2024-04-25T01:34:09.319 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:34:09.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:34:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.450+0000 7f97bd1ee640 1 -- 172.21.15.28:0/833485262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b8100fe0 msgr2=0x7f97b810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.450+0000 7f97bd1ee640 1 --2- 172.21.15.28:0/833485262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b8100fe0 0x7f97b810d4f0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f97b0009cd0 tx=0x7f97b002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.451+0000 7f97bd1ee640 1 -- 172.21.15.28:0/833485262 shutdown_connections 2024-04-25T01:34:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.451+0000 7f97bd1ee640 1 --2- 172.21.15.28:0/833485262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b8100fe0 0x7f97b810d4f0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.451+0000 7f97bd1ee640 1 --2- 172.21.15.28:0/833485262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97b8100620 0x7f97b8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.451+0000 7f97bd1ee640 1 --2- 172.21.15.28:0/833485262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97b8106630 0x7f97b8106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.451+0000 7f97bd1ee640 1 -- 172.21.15.28:0/833485262 >> 172.21.15.28:0/833485262 conn(0x7f97b8077740 msgr2=0x7f97b8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.451+0000 7f97bd1ee640 1 -- 172.21.15.28:0/833485262 shutdown_connections 2024-04-25T01:34:09.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.451+0000 7f97bd1ee640 1 -- 172.21.15.28:0/833485262 wait complete. 2024-04-25T01:34:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.452+0000 7f97bd1ee640 1 Processor -- start 2024-04-25T01:34:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.452+0000 7f97bd1ee640 1 -- start start 2024-04-25T01:34:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.452+0000 7f97bd1ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b8100620 0x7f97b8198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.452+0000 7f97bd1ee640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97b8100fe0 0x7f97b8198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.452+0000 7f97bd1ee640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97b8106630 0x7f97b819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.452+0000 7f97bd1ee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97b8111b40 con 0x7f97b8100620 2024-04-25T01:34:09.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.452+0000 7f97bd1ee640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f97b81119c0 con 0x7f97b8100fe0 2024-04-25T01:34:09.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.452+0000 7f97bd1ee640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f97b8111cc0 con 0x7f97b8106630 2024-04-25T01:34:09.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.452+0000 7f97b6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b8100620 0x7f97b8198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.452+0000 7f97b659c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97b8100fe0 0x7f97b8198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.452+0000 7f97b6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b8100620 0x7f97b8198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58752/0 (socket says 172.21.15.28:58752) 2024-04-25T01:34:09.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.452+0000 7f97b6d9d640 1 -- 172.21.15.28:0/917213984 learned_addr learned my addr 172.21.15.28:0/917213984 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:09.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.453+0000 7f97b759e640 1 --2- 172.21.15.28:0/917213984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97b8106630 0x7f97b819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.453+0000 7f97b659c640 1 -- 172.21.15.28:0/917213984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97b8106630 msgr2=0x7f97b819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.453+0000 7f97b659c640 1 --2- 172.21.15.28:0/917213984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97b8106630 0x7f97b819c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.453+0000 7f97b659c640 1 -- 172.21.15.28:0/917213984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b8100620 msgr2=0x7f97b8198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.453+0000 7f97b659c640 1 --2- 172.21.15.28:0/917213984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b8100620 0x7f97b8198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.453+0000 7f97b659c640 1 -- 172.21.15.28:0/917213984 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b819cfc0 con 0x7f97b8100fe0 2024-04-25T01:34:09.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.453+0000 7f97b6d9d640 1 --2- 172.21.15.28:0/917213984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b8100620 0x7f97b8198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:09.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.453+0000 7f97b659c640 1 --2- 172.21.15.28:0/917213984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97b8100fe0 0x7f97b8198560 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00e9f0 tx=0x7f97ac00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.453+0000 7f979ffff640 1 -- 172.21.15.28:0/917213984 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac00ccc0 con 0x7f97b8100fe0 2024-04-25T01:34:09.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.453+0000 7f979ffff640 1 -- 172.21.15.28:0/917213984 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97ac005020 con 0x7f97b8100fe0 2024-04-25T01:34:09.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.454+0000 7f97bd1ee640 1 -- 172.21.15.28:0/917213984 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b819d2b0 con 0x7f97b8100fe0 2024-04-25T01:34:09.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.454+0000 7f979ffff640 1 -- 172.21.15.28:0/917213984 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac0049e0 con 0x7f97b8100fe0 2024-04-25T01:34:09.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.454+0000 7f97bd1ee640 1 -- 172.21.15.28:0/917213984 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b81a4b40 con 0x7f97b8100fe0 2024-04-25T01:34:09.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.455+0000 7f97bd1ee640 1 -- 172.21.15.28:0/917213984 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97b8068560 con 0x7f97b8100fe0 2024-04-25T01:34:09.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.457+0000 7f979ffff640 1 -- 172.21.15.28:0/917213984 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f97ac005560 con 0x7f97b8100fe0 2024-04-25T01:34:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.458+0000 7f979ffff640 1 --2- 172.21.15.28:0/917213984 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97a00a2e20 0x7f97a00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.458+0000 7f97b6d9d640 1 --2- 172.21.15.28:0/917213984 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97a00a2e20 0x7f97a00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.458+0000 7f979ffff640 1 -- 172.21.15.28:0/917213984 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f97ac0cf9b0 con 0x7f97b8100fe0 2024-04-25T01:34:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.458+0000 7f97b6d9d640 1 --2- 172.21.15.28:0/917213984 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97a00a2e20 0x7f97a00a52e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f97a8011440 tx=0x7f97a8005f90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.461+0000 7f979ffff640 1 -- 172.21.15.28:0/917213984 <== mon.2 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f97ac0051d0 con 0x7f97b8100fe0 2024-04-25T01:34:09.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.740+0000 7f97bd1ee640 1 -- 172.21.15.28:0/917213984 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97b8199320 con 0x7f97b8100fe0 2024-04-25T01:34:09.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.741+0000 7f979ffff640 1 -- 172.21.15.28:0/917213984 <== mon.2 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7f97b8199320 con 0x7f97b8100fe0 2024-04-25T01:34:09.718 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:34:09.718 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:34:09.718 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:34:09.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.743+0000 7f97bd1ee640 1 -- 172.21.15.28:0/917213984 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97a00a2e20 msgr2=0x7f97a00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.743+0000 7f97bd1ee640 1 --2- 172.21.15.28:0/917213984 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97a00a2e20 0x7f97a00a52e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f97a8011440 tx=0x7f97a8005f90 comp rx=0 tx=0).stop 2024-04-25T01:34:09.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.743+0000 7f97bd1ee640 1 -- 172.21.15.28:0/917213984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97b8100fe0 msgr2=0x7f97b8198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.743+0000 7f97bd1ee640 1 --2- 172.21.15.28:0/917213984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97b8100fe0 0x7f97b8198560 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00e9f0 tx=0x7f97ac00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.743+0000 7f97bd1ee640 1 -- 172.21.15.28:0/917213984 shutdown_connections 2024-04-25T01:34:09.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.743+0000 7f97bd1ee640 1 --2- 172.21.15.28:0/917213984 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97a00a2e20 0x7f97a00a52e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.743+0000 7f97bd1ee640 1 --2- 172.21.15.28:0/917213984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97b8106630 0x7f97b819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.743+0000 7f97bd1ee640 1 --2- 172.21.15.28:0/917213984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97b8100fe0 0x7f97b8198560 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.743+0000 7f97bd1ee640 1 --2- 172.21.15.28:0/917213984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97b8100620 0x7f97b8198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.743+0000 7f97bd1ee640 1 -- 172.21.15.28:0/917213984 >> 172.21.15.28:0/917213984 conn(0x7f97b8077740 msgr2=0x7f97b8102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.743+0000 7f97bd1ee640 1 -- 172.21.15.28:0/917213984 shutdown_connections 2024-04-25T01:34:09.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.743+0000 7f97bd1ee640 1 -- 172.21.15.28:0/917213984 wait complete. 2024-04-25T01:34:09.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:34:09.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.905+0000 7f8cfac64640 1 -- 172.21.15.28:0/927256308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cf4100610 msgr2=0x7f8cf4100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.905+0000 7f8cfac64640 1 --2- 172.21.15.28:0/927256308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cf4100610 0x7f8cf4100a90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0009860 tx=0x7f8ce002a340 comp rx=0 tx=0).stop 2024-04-25T01:34:09.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.905+0000 7f8cfac64640 1 -- 172.21.15.28:0/927256308 shutdown_connections 2024-04-25T01:34:09.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.905+0000 7f8cfac64640 1 --2- 172.21.15.28:0/927256308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8cf4100fd0 0x7f8cf410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.905+0000 7f8cfac64640 1 --2- 172.21.15.28:0/927256308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cf4100610 0x7f8cf4100a90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.905+0000 7f8cfac64640 1 --2- 172.21.15.28:0/927256308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cf41066b0 0x7f8cf4106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.905+0000 7f8cfac64640 1 -- 172.21.15.28:0/927256308 >> 172.21.15.28:0/927256308 conn(0x7f8cf4077740 msgr2=0x7f8cf4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.905+0000 7f8cfac64640 1 -- 172.21.15.28:0/927256308 shutdown_connections 2024-04-25T01:34:09.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.905+0000 7f8cfac64640 1 -- 172.21.15.28:0/927256308 wait complete. 2024-04-25T01:34:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.906+0000 7f8cfac64640 1 Processor -- start 2024-04-25T01:34:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.906+0000 7f8cfac64640 1 -- start start 2024-04-25T01:34:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.907+0000 7f8cfac64640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cf4100610 0x7f8cf4196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.907+0000 7f8cfac64640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8cf4100fd0 0x7f8cf41973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.907+0000 7f8cfac64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cf41066b0 0x7f8cf419b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.907+0000 7f8cfac64640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf4110a30 con 0x7f8cf41066b0 2024-04-25T01:34:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.907+0000 7f8cfac64640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf41108b0 con 0x7f8cf4100610 2024-04-25T01:34:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.907+0000 7f8cfac64640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf4110bb0 con 0x7f8cf4100fd0 2024-04-25T01:34:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.907+0000 7f8cf8a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cf4100610 0x7f8cf4196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.907+0000 7f8cf8a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cf4100610 0x7f8cf4196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:45878/0 (socket says 172.21.15.28:45878) 2024-04-25T01:34:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.907+0000 7f8cf8a00640 1 -- 172.21.15.28:0/1161461768 learned_addr learned my addr 172.21.15.28:0/1161461768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.907+0000 7f8cf3fff640 1 --2- 172.21.15.28:0/1161461768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8cf4100fd0 0x7f8cf41973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.907+0000 7f8cf9201640 1 --2- 172.21.15.28:0/1161461768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cf41066b0 0x7f8cf419b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.908+0000 7f8cf3fff640 1 -- 172.21.15.28:0/1161461768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cf4100610 msgr2=0x7f8cf4196e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.908+0000 7f8cf3fff640 1 --2- 172.21.15.28:0/1161461768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cf4100610 0x7f8cf4196e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.908+0000 7f8cf3fff640 1 -- 172.21.15.28:0/1161461768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cf41066b0 msgr2=0x7f8cf419b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.908+0000 7f8cf3fff640 1 --2- 172.21.15.28:0/1161461768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cf41066b0 0x7f8cf419b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.908+0000 7f8cf3fff640 1 -- 172.21.15.28:0/1161461768 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cf419be80 con 0x7f8cf4100fd0 2024-04-25T01:34:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.908+0000 7f8cf8a00640 1 --2- 172.21.15.28:0/1161461768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cf4100610 0x7f8cf4196e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.908+0000 7f8cf9201640 1 --2- 172.21.15.28:0/1161461768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cf41066b0 0x7f8cf419b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.908+0000 7f8cf3fff640 1 --2- 172.21.15.28:0/1161461768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8cf4100fd0 0x7f8cf41973d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0035040 tx=0x7f8ce0030430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.908+0000 7f8cf1ffb640 1 -- 172.21.15.28:0/1161461768 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0042020 con 0x7f8cf4100fd0 2024-04-25T01:34:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.908+0000 7f8cfac64640 1 -- 172.21.15.28:0/1161461768 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cf419c110 con 0x7f8cf4100fd0 2024-04-25T01:34:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.908+0000 7f8cf1ffb640 1 -- 172.21.15.28:0/1161461768 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ce0033de0 con 0x7f8cf4100fd0 2024-04-25T01:34:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.908+0000 7f8cf1ffb640 1 -- 172.21.15.28:0/1161461768 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce0037040 con 0x7f8cf4100fd0 2024-04-25T01:34:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.909+0000 7f8cfac64640 1 -- 172.21.15.28:0/1161461768 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cf40688d0 con 0x7f8cf4100fd0 2024-04-25T01:34:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.912+0000 7f8cfd974740 1 Processor -- start 2024-04-25T01:34:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.912+0000 7f8cfd974740 1 -- start start 2024-04-25T01:34:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.912+0000 7f8cfd974740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25580 0x56260bd25960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.912+0000 7f8cfd974740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd25ea0 0x56260bd30030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.912+0000 7f8cfd974740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd30570 0x56260bd32960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.912+0000 7f8cfd974740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56260bc8a670 con 0x56260bd25580 2024-04-25T01:34:09.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.912+0000 7f8cfd974740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56260bc32970 con 0x56260bd30570 2024-04-25T01:34:09.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.912+0000 7f8cfd974740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56260bc86630 con 0x56260bd25ea0 2024-04-25T01:34:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.913+0000 7f8cf9201640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25580 0x56260bd25960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.913+0000 7f8cf9201640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25580 0x56260bd25960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58782/0 (socket says 172.21.15.28:58782) 2024-04-25T01:34:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.913+0000 7f8cf9201640 1 -- 172.21.15.28:0/3839066540 learned_addr learned my addr 172.21.15.28:0/3839066540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.913+0000 7f8cf8a00640 1 --2- 172.21.15.28:0/3839066540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd25ea0 0x56260bd30030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.913+0000 7f8cf9201640 1 -- 172.21.15.28:0/3839066540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd25ea0 msgr2=0x56260bd30030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.913+0000 7f8cf9201640 1 --2- 172.21.15.28:0/3839066540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd25ea0 0x56260bd30030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.913+0000 7f8cf9201640 1 -- 172.21.15.28:0/3839066540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd30570 msgr2=0x56260bd32960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:34:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.913+0000 7f8cf9201640 1 --2- 172.21.15.28:0/3839066540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd30570 0x56260bd32960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.913+0000 7f8cf9201640 1 -- 172.21.15.28:0/3839066540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56260bc56400 con 0x56260bd25580 2024-04-25T01:34:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.913+0000 7f8cf1ffb640 1 -- 172.21.15.28:0/1161461768 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f8ce0045020 con 0x7f8cf4100fd0 2024-04-25T01:34:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.913+0000 7f8cf9201640 1 --2- 172.21.15.28:0/3839066540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25580 0x56260bd25960 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0117e0 tx=0x7f8cec011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbd62bfce46d78c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.913+0000 7f8cdeffd640 1 -- 172.21.15.28:0/3839066540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cec011310 con 0x56260bd25580 2024-04-25T01:34:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.913+0000 7f8cdeffd640 1 -- 172.21.15.28:0/3839066540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cec003e50 con 0x56260bd25580 2024-04-25T01:34:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.913+0000 7f8cdeffd640 1 -- 172.21.15.28:0/3839066540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cec004590 con 0x56260bd25580 2024-04-25T01:34:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.914+0000 7f8cf1ffb640 1 --2- 172.21.15.28:0/1161461768 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8ce40a2f20 0x7f8ce40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.914+0000 7f8cf8a00640 1 --2- 172.21.15.28:0/1161461768 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8ce40a2f20 0x7f8ce40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.914+0000 7f8cf1ffb640 1 -- 172.21.15.28:0/1161461768 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8ce00f1820 con 0x7f8cf4100fd0 2024-04-25T01:34:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.914+0000 7f8cfd974740 1 -- 172.21.15.28:0/3839066540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25580 msgr2=0x56260bd25960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.914+0000 7f8cfd974740 1 --2- 172.21.15.28:0/3839066540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25580 0x56260bd25960 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0117e0 tx=0x7f8cec011ca0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.914+0000 7f8cfd974740 1 -- 172.21.15.28:0/3839066540 shutdown_connections 2024-04-25T01:34:09.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.914+0000 7f8cfd974740 1 --2- 172.21.15.28:0/3839066540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd30570 0x56260bd32960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.914+0000 7f8cfd974740 1 --2- 172.21.15.28:0/3839066540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd25ea0 0x56260bd30030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.914+0000 7f8cfd974740 1 --2- 172.21.15.28:0/3839066540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25580 0x56260bd25960 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.915+0000 7f8cfd974740 1 -- 172.21.15.28:0/3839066540 >> 172.21.15.28:0/3839066540 conn(0x56260bd218a0 msgr2=0x56260bd21c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.915+0000 7f8cfd974740 1 -- 172.21.15.28:0/3839066540 shutdown_connections 2024-04-25T01:34:09.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.915+0000 7f8cf8a00640 1 --2- 172.21.15.28:0/1161461768 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8ce40a2f20 0x7f8ce40a53e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x56260bd26b40 tx=0x7f8ce8009260 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.915+0000 7f8cfd974740 1 -- 172.21.15.28:0/3839066540 wait complete. 2024-04-25T01:34:09.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.915+0000 7f8cfd974740 1 -- start start 2024-04-25T01:34:09.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.915+0000 7f8cfd974740 1 Processor -- start 2024-04-25T01:34:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.916+0000 7f8cfd974740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd25580 0x56260bd2ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.916+0000 7f8cfd974740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25ea0 0x56260bd2d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.916+0000 7f8cfd974740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd30570 0x56260bd2d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.916+0000 7f8cf3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25ea0 0x56260bd2d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.916+0000 7f8cf3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25ea0 0x56260bd2d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58784/0 (socket says 172.21.15.28:58784) 2024-04-25T01:34:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.916+0000 7f8cf3fff640 1 -- 172.21.15.28:0/1182815934 learned_addr learned my addr 172.21.15.28:0/1182815934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.916+0000 7f8cfd974740 1 -- 172.21.15.28:0/1182815934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56260bd33360 con 0x56260bd25ea0 2024-04-25T01:34:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.916+0000 7f8cfd974740 1 -- 172.21.15.28:0/1182815934 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56260bc46050 con 0x56260bd25580 2024-04-25T01:34:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.916+0000 7f8cf9201640 1 --2- 172.21.15.28:0/1182815934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd30570 0x56260bd2d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.916+0000 7f8cfd974740 1 -- 172.21.15.28:0/1182815934 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56260bd334e0 con 0x56260bd30570 2024-04-25T01:34:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.916+0000 7f8cf8a00640 1 --2- 172.21.15.28:0/1182815934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd25580 0x56260bd2ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.916+0000 7f8cfd974740 10 client.0 _ll_register_callbacks cb 0x56260b99dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.916+0000 7f8cf9201640 1 -- 172.21.15.28:0/1182815934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd25580 msgr2=0x56260bd2ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.916+0000 7f8cf9201640 1 --2- 172.21.15.28:0/1182815934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd25580 0x56260bd2ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.916+0000 7f8cf9201640 1 -- 172.21.15.28:0/1182815934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25ea0 msgr2=0x56260bd2d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.916+0000 7f8cf9201640 1 --2- 172.21.15.28:0/1182815934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25ea0 0x56260bd2d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.917+0000 7f8cf9201640 1 -- 172.21.15.28:0/1182815934 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56260bc17dc0 con 0x56260bd30570 2024-04-25T01:34:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.917+0000 7f8cf3fff640 1 --2- 172.21.15.28:0/1182815934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25ea0 0x56260bd2d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.917+0000 7f8cf9201640 1 --2- 172.21.15.28:0/1182815934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd30570 0x56260bd2d830 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0117e0 tx=0x7f8cec003de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51eb48b31439815 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.917+0000 7f8cd3fff640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-25T01:34:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.917+0000 7f8cd3fff640 1 -- 172.21.15.28:0/1182815934 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cec02a060 con 0x56260bd30570 2024-04-25T01:34:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.917+0000 7f8cfd974740 1 -- 172.21.15.28:0/1182815934 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cec003e50 con 0x56260bd30570 2024-04-25T01:34:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.917+0000 7f8cd3fff640 1 -- 172.21.15.28:0/1182815934 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8cec004a40 con 0x56260bd30570 2024-04-25T01:34:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.917+0000 7f8cd3fff640 1 -- 172.21.15.28:0/1182815934 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8cec031050 con 0x56260bd30570 2024-04-25T01:34:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.917+0000 7f8cfd974740 10 client.25477 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.917+0000 7f8cfd974740 1 -- 172.21.15.28:0/1182815934 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56260b99e3f0 con 0x56260bd30570 2024-04-25T01:34:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.918+0000 7f8cd3fff640 1 -- 172.21.15.28:0/1182815934 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cec003e50 con 0x56260bd30570 2024-04-25T01:34:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.918+0000 7f8cd3fff640 1 -- 172.21.15.28:0/1182815934 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8cec018020 con 0x56260bd30570 2024-04-25T01:34:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.918+0000 7f8cfd974740 10 client.25477 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.918+0000 7f8cfd974740 10 client.25477 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:34:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.918+0000 7f8cfd974740 20 client.25477 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.919+0000 7f8cfd974740 1 --2- 172.21.15.28:0/1182815934 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x56260bd40490 0x56260bd42880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.919+0000 7f8cfd974740 4 client.25477 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.919+0000 7f8cfd974740 1 -- 172.21.15.28:0/1182815934 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56260bc97bb0 con 0x56260bd40490 2024-04-25T01:34:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.919+0000 7f8cf8a00640 1 --2- 172.21.15.28:0/1182815934 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x56260bd40490 0x56260bd42880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.919+0000 7f8cd3fff640 10 client.25477 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:34:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.919+0000 7f8cf8a00640 1 --2- 172.21.15.28:0/1182815934 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x56260bd40490 0x56260bd42880 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bef1066c87f7f05 server_cookie=9f0bf7ff99601eb0 in_seq=0 out_seq=0 2024-04-25T01:34:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.920+0000 7f8cd3fff640 1 -- 172.21.15.28:0/1182815934 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56260bc97bb0 con 0x56260bd40490 2024-04-25T01:34:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.920+0000 7f8cd3fff640 10 client.25477 handle_command_reply: tid=0 2024-04-25T01:34:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.920+0000 7f8cfd974740 1 client.25477 shutdown 2024-04-25T01:34:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.920+0000 7f8cfd974740 1 -- 172.21.15.28:0/1182815934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd30570 msgr2=0x56260bd2d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.920+0000 7f8cfd974740 1 --2- 172.21.15.28:0/1182815934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd30570 0x56260bd2d830 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8cec0117e0 tx=0x7f8cec003de0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.920+0000 7f8cfd974740 1 -- 172.21.15.28:0/1182815934 shutdown_connections 2024-04-25T01:34:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.920+0000 7f8cfd974740 1 --2- 172.21.15.28:0/1182815934 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x56260bd40490 0x56260bd42880 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.920+0000 7f8cfd974740 1 --2- 172.21.15.28:0/1182815934 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd30570 0x56260bd2d830 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.920+0000 7f8cfd974740 1 --2- 172.21.15.28:0/1182815934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25ea0 0x56260bd2d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.920+0000 7f8cd3fff640 0 client.25477 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:34:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.920+0000 7f8cfd974740 1 --2- 172.21.15.28:0/1182815934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd25580 0x56260bd2ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.920+0000 7f8cfd974740 1 -- 172.21.15.28:0/1182815934 >> 172.21.15.28:0/1182815934 conn(0x56260bd218a0 msgr2=0x56260bd276a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.920+0000 7f8cfd974740 1 -- 172.21.15.28:0/1182815934 shutdown_connections 2024-04-25T01:34:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.921+0000 7f8cfd974740 1 -- 172.21.15.28:0/1182815934 wait complete. 2024-04-25T01:34:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.921+0000 7f8cfd974740 20 client.25477 trim_cache size 0 max 16384 2024-04-25T01:34:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.946+0000 7f8cfd974740 1 Processor -- start 2024-04-25T01:34:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.946+0000 7f8cfd974740 1 -- start start 2024-04-25T01:34:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.947+0000 7f8cfd974740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25580 0x56260bd2cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.947+0000 7f8cfd974740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd25ea0 0x56260bd27eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.947+0000 7f8cfd974740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd30570 0x56260bd2fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.947+0000 7f8cfd974740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf41ac890 con 0x56260bd25580 2024-04-25T01:34:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.947+0000 7f8cf8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25580 0x56260bd2cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.947+0000 7f8cfd974740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf41ae3d0 con 0x56260bd30570 2024-04-25T01:34:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.947+0000 7f8cfd974740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf41acf60 con 0x56260bd25ea0 2024-04-25T01:34:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.947+0000 7f8cf8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25580 0x56260bd2cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58794/0 (socket says 172.21.15.28:58794) 2024-04-25T01:34:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.947+0000 7f8cf8a00640 1 -- 172.21.15.28:0/3378870800 learned_addr learned my addr 172.21.15.28:0/3378870800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:09.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.947+0000 7f8cf3fff640 1 --2- 172.21.15.28:0/3378870800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd25ea0 0x56260bd27eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.947+0000 7f8cf8a00640 1 -- 172.21.15.28:0/3378870800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd25ea0 msgr2=0x56260bd27eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.947+0000 7f8cf8a00640 1 --2- 172.21.15.28:0/3378870800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd25ea0 0x56260bd27eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.947+0000 7f8cf8a00640 1 -- 172.21.15.28:0/3378870800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd30570 msgr2=0x56260bd2fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.947+0000 7f8cf9201640 1 --2- 172.21.15.28:0/3378870800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd30570 0x56260bd2fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.947+0000 7f8cf8a00640 1 --2- 172.21.15.28:0/3378870800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd30570 0x56260bd2fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.947+0000 7f8cf8a00640 1 -- 172.21.15.28:0/3378870800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56260ba687d0 con 0x56260bd25580 2024-04-25T01:34:09.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.947+0000 7f8cf9201640 1 --2- 172.21.15.28:0/3378870800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd30570 0x56260bd2fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:09.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.947+0000 7f8cf8a00640 1 --2- 172.21.15.28:0/3378870800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25580 0x56260bd2cb00 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x56260bd42fb0 tx=0x7f8ce801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd02ded2badd8320 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.948+0000 7f8cde7fc640 1 -- 172.21.15.28:0/3378870800 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8012020 con 0x56260bd25580 2024-04-25T01:34:09.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.948+0000 7f8cde7fc640 1 -- 172.21.15.28:0/3378870800 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ce80121c0 con 0x56260bd25580 2024-04-25T01:34:09.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.948+0000 7f8cde7fc640 1 -- 172.21.15.28:0/3378870800 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8011440 con 0x56260bd25580 2024-04-25T01:34:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.949+0000 7f8cfd974740 1 -- 172.21.15.28:0/3378870800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25580 msgr2=0x56260bd2cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.949+0000 7f8cfd974740 1 --2- 172.21.15.28:0/3378870800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25580 0x56260bd2cb00 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x56260bd42fb0 tx=0x7f8ce801b000 comp rx=0 tx=0).stop 2024-04-25T01:34:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.949+0000 7f8cfd974740 1 -- 172.21.15.28:0/3378870800 shutdown_connections 2024-04-25T01:34:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.949+0000 7f8cfd974740 1 --2- 172.21.15.28:0/3378870800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd30570 0x56260bd2fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.949+0000 7f8cfd974740 1 --2- 172.21.15.28:0/3378870800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd25ea0 0x56260bd27eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.949+0000 7f8cfd974740 1 --2- 172.21.15.28:0/3378870800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25580 0x56260bd2cb00 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.949+0000 7f8cfd974740 1 -- 172.21.15.28:0/3378870800 >> 172.21.15.28:0/3378870800 conn(0x56260bd218a0 msgr2=0x56260bd275a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.949+0000 7f8cfd974740 1 -- 172.21.15.28:0/3378870800 shutdown_connections 2024-04-25T01:34:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.949+0000 7f8cfd974740 1 -- 172.21.15.28:0/3378870800 wait complete. 2024-04-25T01:34:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.949+0000 7f8cfd974740 1 -- start start 2024-04-25T01:34:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.949+0000 7f8cfd974740 1 Processor -- start 2024-04-25T01:34:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.950+0000 7f8cfd974740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd25580 0x56260bd33a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.950+0000 7f8cfd974740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25ea0 0x56260bd33f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.950+0000 7f8cfd974740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd30570 0x56260bd9e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.950+0000 7f8cfd974740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf41add00 con 0x56260bd25ea0 2024-04-25T01:34:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.950+0000 7f8cfd974740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf41ad630 con 0x56260bd30570 2024-04-25T01:34:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.950+0000 7f8cf3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25ea0 0x56260bd33f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.950+0000 7f8cfd974740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf41aa870 con 0x56260bd25580 2024-04-25T01:34:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.950+0000 7f8cf3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25ea0 0x56260bd33f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58796/0 (socket says 172.21.15.28:58796) 2024-04-25T01:34:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.950+0000 7f8cf3fff640 1 -- 172.21.15.28:0/429824061 learned_addr learned my addr 172.21.15.28:0/429824061 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.950+0000 7f8cfd974740 10 client.0 _ll_register_callbacks cb 0x7f8ce8009b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.950+0000 7f8cf9201640 1 --2- 172.21.15.28:0/429824061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd30570 0x56260bd9e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.950+0000 7f8cf3fff640 1 -- 172.21.15.28:0/429824061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd25580 msgr2=0x56260bd33a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.950+0000 7f8cf3fff640 1 --2- 172.21.15.28:0/429824061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd25580 0x56260bd33a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.950+0000 7f8cf3fff640 1 -- 172.21.15.28:0/429824061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd30570 msgr2=0x56260bd9e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.950+0000 7f8cf3fff640 1 --2- 172.21.15.28:0/429824061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd30570 0x56260bd9e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.950+0000 7f8cf3fff640 1 -- 172.21.15.28:0/429824061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56260bd9eef0 con 0x56260bd25ea0 2024-04-25T01:34:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.950+0000 7f8cf3fff640 1 --2- 172.21.15.28:0/429824061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25ea0 0x56260bd33f40 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f8ce00bd8a0 tx=0x7f8ce0030af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0da98627b2e3d2a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.950+0000 7f8cd3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.951+0000 7f8cd3fff640 1 -- 172.21.15.28:0/429824061 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce004c040 con 0x56260bd25ea0 2024-04-25T01:34:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.951+0000 7f8cd3fff640 1 -- 172.21.15.28:0/429824061 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8ce00bdd70 con 0x56260bd25ea0 2024-04-25T01:34:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.951+0000 7f8cfd974740 1 -- 172.21.15.28:0/429824061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ce80121c0 con 0x56260bd25ea0 2024-04-25T01:34:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.951+0000 7f8cd3fff640 1 -- 172.21.15.28:0/429824061 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8ce00c1c10 con 0x56260bd25ea0 2024-04-25T01:34:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.951+0000 7f8cfd974740 10 client.16011 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.951+0000 7f8cfd974740 1 -- 172.21.15.28:0/429824061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56260bda6210 con 0x56260bd25ea0 2024-04-25T01:34:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.951+0000 7f8cd3fff640 1 -- 172.21.15.28:0/429824061 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ce80121c0 con 0x56260bd25ea0 2024-04-25T01:34:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.951+0000 7f8cd3fff640 1 -- 172.21.15.28:0/429824061 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8ce00beb90 con 0x56260bd25ea0 2024-04-25T01:34:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.951+0000 7f8cfd974740 10 client.16011 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.951+0000 7f8cfd974740 10 client.16011 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:34:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.951+0000 7f8cfd974740 20 client.16011 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.951+0000 7f8cfd974740 1 --2- 172.21.15.28:0/429824061 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x56260bda6940 0x56260bda8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.951+0000 7f8cfd974740 4 client.16011 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:34:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.951+0000 7f8cfd974740 1 -- 172.21.15.28:0/429824061 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56260bc91f00 con 0x56260bda6940 2024-04-25T01:34:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.952+0000 7f8cf8a00640 1 --2- 172.21.15.28:0/429824061 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x56260bda6940 0x56260bda8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.952+0000 7f8cd3fff640 10 client.16011 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:34:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.952+0000 7f8cf8a00640 1 --2- 172.21.15.28:0/429824061 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x56260bda6940 0x56260bda8d60 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2c1c84bed82cccc2 server_cookie=6e7548d9f2305690 in_seq=0 out_seq=0 2024-04-25T01:34:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.953+0000 7f8cd3fff640 1 -- 172.21.15.28:0/429824061 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56260bc91f00 con 0x56260bda6940 2024-04-25T01:34:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.953+0000 7f8cd3fff640 10 client.16011 handle_command_reply: tid=0 2024-04-25T01:34:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.953+0000 7f8cfd974740 1 client.16011 shutdown 2024-04-25T01:34:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.953+0000 7f8cfd974740 1 -- 172.21.15.28:0/429824061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25ea0 msgr2=0x56260bd33f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.953+0000 7f8cfd974740 1 --2- 172.21.15.28:0/429824061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25ea0 0x56260bd33f40 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f8ce00bd8a0 tx=0x7f8ce0030af0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.953+0000 7f8cfd974740 1 -- 172.21.15.28:0/429824061 shutdown_connections 2024-04-25T01:34:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.953+0000 7f8cfd974740 1 --2- 172.21.15.28:0/429824061 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x56260bda6940 0x56260bda8d60 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.953+0000 7f8cfd974740 1 --2- 172.21.15.28:0/429824061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56260bd30570 0x56260bd9e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.953+0000 7f8cfd974740 1 --2- 172.21.15.28:0/429824061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56260bd25ea0 0x56260bd33f40 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.953+0000 7f8cfd974740 1 --2- 172.21.15.28:0/429824061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56260bd25580 0x56260bd33a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.953+0000 7f8cd3fff640 0 client.16011 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:34:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.953+0000 7f8cfd974740 1 -- 172.21.15.28:0/429824061 >> 172.21.15.28:0/429824061 conn(0x56260bd218a0 msgr2=0x56260bd3b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.953+0000 7f8cfd974740 1 -- 172.21.15.28:0/429824061 shutdown_connections 2024-04-25T01:34:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.953+0000 7f8cfd974740 1 -- 172.21.15.28:0/429824061 wait complete. 2024-04-25T01:34:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.953+0000 7f8cfd974740 20 client.16011 trim_cache size 0 max 16384 2024-04-25T01:34:09.934 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:34:09.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.954+0000 7f8cfac64640 1 -- 172.21.15.28:0/1161461768 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8ce40a2f20 msgr2=0x7f8ce40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.954+0000 7f8cfac64640 1 --2- 172.21.15.28:0/1161461768 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8ce40a2f20 0x7f8ce40a53e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x56260bd26b40 tx=0x7f8ce8009260 comp rx=0 tx=0).stop 2024-04-25T01:34:09.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.954+0000 7f8cfac64640 1 -- 172.21.15.28:0/1161461768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8cf4100fd0 msgr2=0x7f8cf41973d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:09.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.954+0000 7f8cfac64640 1 --2- 172.21.15.28:0/1161461768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8cf4100fd0 0x7f8cf41973d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0035040 tx=0x7f8ce0030430 comp rx=0 tx=0).stop 2024-04-25T01:34:09.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.954+0000 7f8cfac64640 1 -- 172.21.15.28:0/1161461768 shutdown_connections 2024-04-25T01:34:09.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.954+0000 7f8cfac64640 1 --2- 172.21.15.28:0/1161461768 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8ce40a2f20 0x7f8ce40a53e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.954+0000 7f8cfac64640 1 --2- 172.21.15.28:0/1161461768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cf41066b0 0x7f8cf419b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.954+0000 7f8cfac64640 1 --2- 172.21.15.28:0/1161461768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8cf4100fd0 0x7f8cf41973d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.954+0000 7f8cfac64640 1 --2- 172.21.15.28:0/1161461768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8cf4100610 0x7f8cf4196e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:09.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.954+0000 7f8cfac64640 1 -- 172.21.15.28:0/1161461768 >> 172.21.15.28:0/1161461768 conn(0x7f8cf4077740 msgr2=0x7f8cf40ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:09.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.954+0000 7f8cfac64640 1 -- 172.21.15.28:0/1161461768 shutdown_connections 2024-04-25T01:34:09.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:09.954+0000 7f8cfac64640 1 -- 172.21.15.28:0/1161461768 wait complete. 2024-04-25T01:34:09.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:34:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.074+0000 7f66e36cd640 1 -- 172.21.15.28:0/2298323166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66dc1066b0 msgr2=0x7f66dc106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.074+0000 7f66e36cd640 1 --2- 172.21.15.28:0/2298323166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66dc1066b0 0x7f66dc106a90 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f66d800fb40 tx=0x7f66d802e5b0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.074+0000 7f66e36cd640 1 -- 172.21.15.28:0/2298323166 shutdown_connections 2024-04-25T01:34:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.074+0000 7f66e36cd640 1 --2- 172.21.15.28:0/2298323166 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66dc100fd0 0x7f66dc10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.074+0000 7f66e36cd640 1 --2- 172.21.15.28:0/2298323166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66dc100610 0x7f66dc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.074+0000 7f66e36cd640 1 --2- 172.21.15.28:0/2298323166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66dc1066b0 0x7f66dc106a90 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.074+0000 7f66e36cd640 1 -- 172.21.15.28:0/2298323166 >> 172.21.15.28:0/2298323166 conn(0x7f66dc077740 msgr2=0x7f66dc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.074+0000 7f66e36cd640 1 -- 172.21.15.28:0/2298323166 shutdown_connections 2024-04-25T01:34:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.074+0000 7f66e36cd640 1 -- 172.21.15.28:0/2298323166 wait complete. 2024-04-25T01:34:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.075+0000 7f66e36cd640 1 Processor -- start 2024-04-25T01:34:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.075+0000 7f66e36cd640 1 -- start start 2024-04-25T01:34:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.075+0000 7f66e36cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66dc100610 0x7f66dc195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.075+0000 7f66e36cd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66dc100fd0 0x7f66dc196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.075+0000 7f66e36cd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66dc19a850 0x7f66dc19ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.075+0000 7f66e36cd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66dc10f970 con 0x7f66dc100610 2024-04-25T01:34:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.075+0000 7f66e36cd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f66dc10f7f0 con 0x7f66dc19a850 2024-04-25T01:34:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.075+0000 7f66e36cd640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f66dc10faf0 con 0x7f66dc100fd0 2024-04-25T01:34:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.075+0000 7f66e1469640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66dc100610 0x7f66dc195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.075+0000 7f66e1469640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66dc100610 0x7f66dc195fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58816/0 (socket says 172.21.15.28:58816) 2024-04-25T01:34:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.075+0000 7f66e1469640 1 -- 172.21.15.28:0/216551719 learned_addr learned my addr 172.21.15.28:0/216551719 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.075+0000 7f66e0c68640 1 --2- 172.21.15.28:0/216551719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66dc100fd0 0x7f66dc196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.075+0000 7f66e1c6a640 1 --2- 172.21.15.28:0/216551719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66dc19a850 0x7f66dc19ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.076+0000 7f66e0c68640 1 -- 172.21.15.28:0/216551719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66dc19a850 msgr2=0x7f66dc19ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.076+0000 7f66e0c68640 1 --2- 172.21.15.28:0/216551719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66dc19a850 0x7f66dc19ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.076+0000 7f66e0c68640 1 -- 172.21.15.28:0/216551719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66dc100610 msgr2=0x7f66dc195fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.076+0000 7f66e0c68640 1 --2- 172.21.15.28:0/216551719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66dc100610 0x7f66dc195fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.076+0000 7f66e0c68640 1 -- 172.21.15.28:0/216551719 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66dc19b340 con 0x7f66dc100fd0 2024-04-25T01:34:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.076+0000 7f66e1469640 1 --2- 172.21.15.28:0/216551719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66dc100610 0x7f66dc195fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.076+0000 7f66e0c68640 1 --2- 172.21.15.28:0/216551719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66dc100fd0 0x7f66dc196520 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f66cc007860 tx=0x7f66cc007c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.076+0000 7f66d27fc640 1 -- 172.21.15.28:0/216551719 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66cc00f710 con 0x7f66dc100fd0 2024-04-25T01:34:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.076+0000 7f66e36cd640 1 -- 172.21.15.28:0/216551719 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66dc19b630 con 0x7f66dc100fd0 2024-04-25T01:34:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.076+0000 7f66d27fc640 1 -- 172.21.15.28:0/216551719 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66cc01a070 con 0x7f66dc100fd0 2024-04-25T01:34:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.077+0000 7f66d27fc640 1 -- 172.21.15.28:0/216551719 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66cc015d50 con 0x7f66dc100fd0 2024-04-25T01:34:10.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.077+0000 7f66e36cd640 1 -- 172.21.15.28:0/216551719 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66dc1a2e20 con 0x7f66dc100fd0 2024-04-25T01:34:10.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.079+0000 7f66d27fc640 1 -- 172.21.15.28:0/216551719 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f66cc00f8b0 con 0x7f66dc100fd0 2024-04-25T01:34:10.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.080+0000 7f66e63d9740 1 Processor -- start 2024-04-25T01:34:10.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.080+0000 7f66e63d9740 1 -- start start 2024-04-25T01:34:10.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.080+0000 7f66e63d9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b0354580 0x55f2b0354960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.080+0000 7f66e63d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354ea0 0x55f2b035f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.080+0000 7f66e63d9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b035f570 0x55f2b0361960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.080+0000 7f66e63d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f2b02b9670 con 0x55f2b0354ea0 2024-04-25T01:34:10.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.080+0000 7f66e63d9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f2b0261970 con 0x55f2b035f570 2024-04-25T01:34:10.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.080+0000 7f66e63d9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55f2b02b5630 con 0x55f2b0354580 2024-04-25T01:34:10.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.081+0000 7f66d27fc640 1 --2- 172.21.15.28:0/216551719 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f66c80a2d00 0x7f66c80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.081+0000 7f66e1c6a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b0354580 0x55f2b0354960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.081+0000 7f66e1c6a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b0354580 0x55f2b0354960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.28:46706/0 (socket says 172.21.15.28:46706) 2024-04-25T01:34:10.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.081+0000 7f66e1c6a640 1 -- 172.21.15.28:0/3277788135 learned_addr learned my addr 172.21.15.28:0/3277788135 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.081+0000 7f66e1c6a640 1 --2- 172.21.15.28:0/216551719 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f66c80a2d00 0x7f66c80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.081+0000 7f66e0c68640 1 --2- 172.21.15.28:0/3277788135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b035f570 0x55f2b0361960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.081+0000 7f66e1469640 1 --2- 172.21.15.28:0/3277788135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354ea0 0x55f2b035f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.081+0000 7f66e0c68640 1 -- 172.21.15.28:0/3277788135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b0354580 msgr2=0x55f2b0354960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.081+0000 7f66e0c68640 1 --2- 172.21.15.28:0/3277788135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b0354580 0x55f2b0354960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.081+0000 7f66e0c68640 1 -- 172.21.15.28:0/3277788135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354ea0 msgr2=0x55f2b035f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.081+0000 7f66e0c68640 1 --2- 172.21.15.28:0/3277788135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354ea0 0x55f2b035f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.081+0000 7f66e0c68640 1 -- 172.21.15.28:0/3277788135 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2b0285400 con 0x55f2b035f570 2024-04-25T01:34:10.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.081+0000 7f66e1469640 1 --2- 172.21.15.28:0/3277788135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354ea0 0x55f2b035f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:10.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.082+0000 7f66e0c68640 1 --2- 172.21.15.28:0/3277788135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b035f570 0x55f2b0361960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f66cc0cc7a0 tx=0x7f66cc097800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdb90a5927ccf2e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.082+0000 7f66c77fe640 1 -- 172.21.15.28:0/3277788135 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66cc09b270 con 0x55f2b035f570 2024-04-25T01:34:10.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.082+0000 7f66d27fc640 1 -- 172.21.15.28:0/216551719 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f66cc0cf510 con 0x7f66dc100fd0 2024-04-25T01:34:10.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.082+0000 7f66e1c6a640 1 --2- 172.21.15.28:0/216551719 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f66c80a2d00 0x7f66c80a51c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f66d4009270 tx=0x7f66d4014d10 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.082+0000 7f66c77fe640 1 -- 172.21.15.28:0/3277788135 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66cc052250 con 0x55f2b035f570 2024-04-25T01:34:10.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.082+0000 7f66c77fe640 1 -- 172.21.15.28:0/3277788135 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66cc074830 con 0x55f2b035f570 2024-04-25T01:34:10.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.083+0000 7f66e63d9740 1 -- 172.21.15.28:0/3277788135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b035f570 msgr2=0x55f2b0361960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.083+0000 7f66e63d9740 1 --2- 172.21.15.28:0/3277788135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b035f570 0x55f2b0361960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f66cc0cc7a0 tx=0x7f66cc097800 comp rx=0 tx=0).stop 2024-04-25T01:34:10.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.083+0000 7f66e63d9740 1 -- 172.21.15.28:0/3277788135 shutdown_connections 2024-04-25T01:34:10.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.083+0000 7f66e63d9740 1 --2- 172.21.15.28:0/3277788135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b035f570 0x55f2b0361960 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.083+0000 7f66e63d9740 1 --2- 172.21.15.28:0/3277788135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354ea0 0x55f2b035f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.083+0000 7f66e63d9740 1 --2- 172.21.15.28:0/3277788135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b0354580 0x55f2b0354960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.083+0000 7f66e63d9740 1 -- 172.21.15.28:0/3277788135 >> 172.21.15.28:0/3277788135 conn(0x55f2b03508a0 msgr2=0x55f2b0350c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.083+0000 7f66e63d9740 1 -- 172.21.15.28:0/3277788135 shutdown_connections 2024-04-25T01:34:10.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.083+0000 7f66e63d9740 1 -- 172.21.15.28:0/3277788135 wait complete. 2024-04-25T01:34:10.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.083+0000 7f66e63d9740 1 -- start start 2024-04-25T01:34:10.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.084+0000 7f66e63d9740 1 Processor -- start 2024-04-25T01:34:10.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.085+0000 7f66e63d9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b0354580 0x55f2b035ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.085+0000 7f66e63d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354ea0 0x55f2b035c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.085+0000 7f66e63d9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b035f570 0x55f2b035c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.085+0000 7f66e0c68640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354ea0 0x55f2b035c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.085+0000 7f66e0c68640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354ea0 0x55f2b035c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58840/0 (socket says 172.21.15.28:58840) 2024-04-25T01:34:10.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.085+0000 7f66e0c68640 1 -- 172.21.15.28:0/644460146 learned_addr learned my addr 172.21.15.28:0/644460146 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.085+0000 7f66e1469640 1 --2- 172.21.15.28:0/644460146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b0354580 0x55f2b035ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.085+0000 7f66e63d9740 1 -- 172.21.15.28:0/644460146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f2b0362360 con 0x55f2b0354ea0 2024-04-25T01:34:10.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.085+0000 7f66e63d9740 1 -- 172.21.15.28:0/644460146 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f2b0275050 con 0x55f2b0354580 2024-04-25T01:34:10.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.085+0000 7f66e63d9740 1 -- 172.21.15.28:0/644460146 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55f2b03624e0 con 0x55f2b035f570 2024-04-25T01:34:10.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.085+0000 7f66e1c6a640 1 --2- 172.21.15.28:0/644460146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b035f570 0x55f2b035c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.085+0000 7f66e63d9740 10 client.0 _ll_register_callbacks cb 0x55f2affccbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:10.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.086+0000 7f66e0c68640 1 -- 172.21.15.28:0/644460146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b035f570 msgr2=0x55f2b035c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.086+0000 7f66e0c68640 1 --2- 172.21.15.28:0/644460146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b035f570 0x55f2b035c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.086+0000 7f66e0c68640 1 -- 172.21.15.28:0/644460146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b0354580 msgr2=0x55f2b035ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.086+0000 7f66e0c68640 1 --2- 172.21.15.28:0/644460146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b0354580 0x55f2b035ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.086+0000 7f66e0c68640 1 -- 172.21.15.28:0/644460146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2b0246dc0 con 0x55f2b0354ea0 2024-04-25T01:34:10.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.086+0000 7f66e63d9740 1 -- 172.21.15.28:0/644460146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66cc052250 con 0x55f2b0354ea0 2024-04-25T01:34:10.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.086+0000 7f66e0c68640 1 --2- 172.21.15.28:0/644460146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354ea0 0x55f2b035c250 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f66cc00b480 tx=0x7f66cc0999b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=136b159869019d27 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.086+0000 7f66c4ff9640 10 client.16023 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:10.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.086+0000 7f66c4ff9640 1 -- 172.21.15.28:0/644460146 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66cc074a20 con 0x55f2b0354ea0 2024-04-25T01:34:10.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.086+0000 7f66c4ff9640 1 -- 172.21.15.28:0/644460146 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f66cc0971f0 con 0x55f2b0354ea0 2024-04-25T01:34:10.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.086+0000 7f66c4ff9640 1 -- 172.21.15.28:0/644460146 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f66cc0b1e60 con 0x55f2b0354ea0 2024-04-25T01:34:10.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.086+0000 7f66e63d9740 10 client.16023 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:10.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.086+0000 7f66e63d9740 1 -- 172.21.15.28:0/644460146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=28}) -- 0x55f2affcd3f0 con 0x55f2b0354ea0 2024-04-25T01:34:10.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.087+0000 7f66c4ff9640 1 -- 172.21.15.28:0/644460146 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f66cc0cf120 con 0x55f2b0354ea0 2024-04-25T01:34:10.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.087+0000 7f66e63d9740 10 client.16023 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:10.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.087+0000 7f66e63d9740 10 client.16023 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:34:10.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.087+0000 7f66e63d9740 20 client.16023 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:10.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.087+0000 7f66e63d9740 1 --2- 172.21.15.28:0/644460146 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55f2b036f3a0 0x55f2b0371790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.087+0000 7f66e63d9740 4 client.16023 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:10.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.087+0000 7f66e63d9740 1 -- 172.21.15.28:0/644460146 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2b02c6bb0 con 0x55f2b036f3a0 2024-04-25T01:34:10.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.087+0000 7f66e1469640 1 --2- 172.21.15.28:0/644460146 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55f2b036f3a0 0x55f2b0371790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.088+0000 7f66e1469640 1 --2- 172.21.15.28:0/644460146 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55f2b036f3a0 0x55f2b0371790 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=68ae177158534d48 server_cookie=83e45f83fce6bcb in_seq=0 out_seq=0 2024-04-25T01:34:10.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.088+0000 7f66c4ff9640 10 client.16023 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:34:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.089+0000 7f66c4ff9640 1 -- 172.21.15.28:0/644460146 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f2b02c6bb0 con 0x55f2b036f3a0 2024-04-25T01:34:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.089+0000 7f66c4ff9640 10 client.16023 handle_command_reply: tid=0 2024-04-25T01:34:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.089+0000 7f66e63d9740 1 client.16023 shutdown 2024-04-25T01:34:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.089+0000 7f66e63d9740 1 -- 172.21.15.28:0/644460146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354ea0 msgr2=0x55f2b035c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.089+0000 7f66e63d9740 1 --2- 172.21.15.28:0/644460146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354ea0 0x55f2b035c250 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f66cc00b480 tx=0x7f66cc0999b0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.089+0000 7f66e63d9740 1 -- 172.21.15.28:0/644460146 shutdown_connections 2024-04-25T01:34:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.089+0000 7f66e63d9740 1 --2- 172.21.15.28:0/644460146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b035f570 0x55f2b035c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.089+0000 7f66e63d9740 1 --2- 172.21.15.28:0/644460146 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55f2b036f3a0 0x55f2b0371790 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.089+0000 7f66e63d9740 1 --2- 172.21.15.28:0/644460146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354ea0 0x55f2b035c250 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.089+0000 7f66c4ff9640 0 client.16023 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:34:10.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.089+0000 7f66e63d9740 1 --2- 172.21.15.28:0/644460146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b0354580 0x55f2b035ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.089+0000 7f66e63d9740 1 -- 172.21.15.28:0/644460146 >> 172.21.15.28:0/644460146 conn(0x55f2b03508a0 msgr2=0x55f2b03566a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.089+0000 7f66e63d9740 1 -- 172.21.15.28:0/644460146 shutdown_connections 2024-04-25T01:34:10.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.089+0000 7f66e63d9740 1 -- 172.21.15.28:0/644460146 wait complete. 2024-04-25T01:34:10.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.089+0000 7f66e63d9740 20 client.16023 trim_cache size 0 max 16384 2024-04-25T01:34:10.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.116+0000 7f66e63d9740 1 Processor -- start 2024-04-25T01:34:10.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.116+0000 7f66e63d9740 1 -- start start 2024-04-25T01:34:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.116+0000 7f66e63d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354580 0x55f2b036ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.116+0000 7f66e63d9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b0354ea0 0x55f2b035ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.116+0000 7f66e63d9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b035f570 0x55f2b03594d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.116+0000 7f66e63d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66dc1aca10 con 0x55f2b0354580 2024-04-25T01:34:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.116+0000 7f66e63d9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f66dc1ae550 con 0x55f2b0354ea0 2024-04-25T01:34:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.116+0000 7f66e63d9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f66dc1ad0e0 con 0x55f2b035f570 2024-04-25T01:34:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.117+0000 7f66e1469640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354580 0x55f2b036ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.117+0000 7f66e1469640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354580 0x55f2b036ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58852/0 (socket says 172.21.15.28:58852) 2024-04-25T01:34:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.117+0000 7f66e1469640 1 -- 172.21.15.28:0/3611452556 learned_addr learned my addr 172.21.15.28:0/3611452556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.117+0000 7f66e0c68640 1 --2- 172.21.15.28:0/3611452556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b0354ea0 0x55f2b035ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.117+0000 7f66e1c6a640 1 --2- 172.21.15.28:0/3611452556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b035f570 0x55f2b03594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.117+0000 7f66e1469640 1 -- 172.21.15.28:0/3611452556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b035f570 msgr2=0x55f2b03594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.117+0000 7f66e1469640 1 --2- 172.21.15.28:0/3611452556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b035f570 0x55f2b03594d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.117+0000 7f66e1469640 1 -- 172.21.15.28:0/3611452556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b0354ea0 msgr2=0x55f2b035ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.117+0000 7f66e1469640 1 --2- 172.21.15.28:0/3611452556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b0354ea0 0x55f2b035ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.117+0000 7f66e1469640 1 -- 172.21.15.28:0/3611452556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2b0354cd0 con 0x55f2b0354580 2024-04-25T01:34:10.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.117+0000 7f66e1469640 1 --2- 172.21.15.28:0/3611452556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354580 0x55f2b036ae40 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f66d8009230 tx=0x7f66d8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f01b7abd78f359 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.117+0000 7f66c6ffd640 1 -- 172.21.15.28:0/3611452556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d8059b50 con 0x55f2b0354580 2024-04-25T01:34:10.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.117+0000 7f66c6ffd640 1 -- 172.21.15.28:0/3611452556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66d8059cf0 con 0x55f2b0354580 2024-04-25T01:34:10.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.117+0000 7f66c6ffd640 1 -- 172.21.15.28:0/3611452556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d800ec80 con 0x55f2b0354580 2024-04-25T01:34:10.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.118+0000 7f66e63d9740 1 -- 172.21.15.28:0/3611452556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354580 msgr2=0x55f2b036ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.118+0000 7f66e63d9740 1 --2- 172.21.15.28:0/3611452556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354580 0x55f2b036ae40 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f66d8009230 tx=0x7f66d8000bc0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.119+0000 7f66e63d9740 1 -- 172.21.15.28:0/3611452556 shutdown_connections 2024-04-25T01:34:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.119+0000 7f66e63d9740 1 --2- 172.21.15.28:0/3611452556 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b035f570 0x55f2b03594d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.119+0000 7f66e63d9740 1 --2- 172.21.15.28:0/3611452556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b0354ea0 0x55f2b035ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.119+0000 7f66e63d9740 1 --2- 172.21.15.28:0/3611452556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b0354580 0x55f2b036ae40 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.119+0000 7f66e63d9740 1 -- 172.21.15.28:0/3611452556 >> 172.21.15.28:0/3611452556 conn(0x55f2b03508a0 msgr2=0x55f2b036a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.119+0000 7f66e63d9740 1 -- 172.21.15.28:0/3611452556 shutdown_connections 2024-04-25T01:34:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.119+0000 7f66e63d9740 1 -- 172.21.15.28:0/3611452556 wait complete. 2024-04-25T01:34:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.119+0000 7f66e63d9740 1 -- start start 2024-04-25T01:34:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.119+0000 7f66e63d9740 1 Processor -- start 2024-04-25T01:34:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.120+0000 7f66e63d9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b0354580 0x55f2b03745b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.120+0000 7f66e63d9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b0354ea0 0x55f2b0374af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.120+0000 7f66e63d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b035f570 0x55f2b0372a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.120+0000 7f66e63d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66dc1ade80 con 0x55f2b035f570 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.120+0000 7f66e1c6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b035f570 0x55f2b0372a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.120+0000 7f66e1c6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b035f570 0x55f2b0372a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58866/0 (socket says 172.21.15.28:58866) 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.120+0000 7f66e1c6a640 1 -- 172.21.15.28:0/265458408 learned_addr learned my addr 172.21.15.28:0/265458408 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.120+0000 7f66e63d9740 1 -- 172.21.15.28:0/265458408 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f66dc1ad7b0 con 0x55f2b0354580 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.120+0000 7f66e63d9740 1 -- 172.21.15.28:0/265458408 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f66dc1aa9f0 con 0x55f2b0354ea0 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.120+0000 7f66e0c68640 1 --2- 172.21.15.28:0/265458408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b0354ea0 0x55f2b0374af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.120+0000 7f66e1469640 1 --2- 172.21.15.28:0/265458408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b0354580 0x55f2b03745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.120+0000 7f66e63d9740 10 client.0 _ll_register_callbacks cb 0x7f66d40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.120+0000 7f66e1c6a640 1 -- 172.21.15.28:0/265458408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b0354ea0 msgr2=0x55f2b0374af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.120+0000 7f66e1c6a640 1 --2- 172.21.15.28:0/265458408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b0354ea0 0x55f2b0374af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.120+0000 7f66e1c6a640 1 -- 172.21.15.28:0/265458408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b0354580 msgr2=0x55f2b03745b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.120+0000 7f66e1c6a640 1 --2- 172.21.15.28:0/265458408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b0354580 0x55f2b03745b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.120+0000 7f66e1c6a640 1 -- 172.21.15.28:0/265458408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2b00977d0 con 0x55f2b035f570 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.120+0000 7f66e0c68640 1 --2- 172.21.15.28:0/265458408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b0354ea0 0x55f2b0374af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:10.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.121+0000 7f66e1c6a640 1 --2- 172.21.15.28:0/265458408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b035f570 0x55f2b0372a00 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f66d400db40 tx=0x7f66d4014be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8146c79e969e1f77 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.121+0000 7f66c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.121+0000 7f66c4ff9640 1 -- 172.21.15.28:0/265458408 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d4014040 con 0x55f2b035f570 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.121+0000 7f66c4ff9640 1 -- 172.21.15.28:0/265458408 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f66d4003b70 con 0x55f2b035f570 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.121+0000 7f66e63d9740 1 -- 172.21.15.28:0/265458408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66d8059cf0 con 0x55f2b035f570 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.121+0000 7f66c4ff9640 1 -- 172.21.15.28:0/265458408 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f66d401d040 con 0x55f2b035f570 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.121+0000 7f66e63d9740 10 client.16029 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.121+0000 7f66e63d9740 1 -- 172.21.15.28:0/265458408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f2b03d5e60 con 0x55f2b035f570 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.121+0000 7f66c4ff9640 1 -- 172.21.15.28:0/265458408 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66d4003d70 con 0x55f2b035f570 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.122+0000 7f66c4ff9640 1 -- 172.21.15.28:0/265458408 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f66d4004590 con 0x55f2b035f570 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.122+0000 7f66e63d9740 10 client.16029 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.122+0000 7f66e63d9740 10 client.16029 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.122+0000 7f66e63d9740 20 client.16029 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.122+0000 7f66e63d9740 1 --2- 172.21.15.28:0/265458408 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55f2b03d6630 0x55f2b03d8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.122+0000 7f66e63d9740 4 client.16029 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:34:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.122+0000 7f66e63d9740 1 -- 172.21.15.28:0/265458408 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f66d8059cf0 con 0x55f2b03d6630 2024-04-25T01:34:10.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.122+0000 7f66e1469640 1 --2- 172.21.15.28:0/265458408 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55f2b03d6630 0x55f2b03d8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.123+0000 7f66c4ff9640 10 client.16029 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:34:10.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.123+0000 7f66e1469640 1 --2- 172.21.15.28:0/265458408 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55f2b03d6630 0x55f2b03d8a50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ce18f9e25b1a86de server_cookie=8eaa4e43e349c98a in_seq=0 out_seq=0 2024-04-25T01:34:10.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.123+0000 7f66c4ff9640 1 -- 172.21.15.28:0/265458408 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f66d8059cf0 con 0x55f2b03d6630 2024-04-25T01:34:10.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.123+0000 7f66c4ff9640 10 client.16029 handle_command_reply: tid=0 2024-04-25T01:34:10.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.123+0000 7f66e63d9740 1 client.16029 shutdown 2024-04-25T01:34:10.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.123+0000 7f66e63d9740 1 -- 172.21.15.28:0/265458408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b035f570 msgr2=0x55f2b0372a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.123+0000 7f66e63d9740 1 --2- 172.21.15.28:0/265458408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b035f570 0x55f2b0372a00 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f66d400db40 tx=0x7f66d4014be0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.123+0000 7f66e63d9740 1 -- 172.21.15.28:0/265458408 shutdown_connections 2024-04-25T01:34:10.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.123+0000 7f66e63d9740 1 --2- 172.21.15.28:0/265458408 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55f2b03d6630 0x55f2b03d8a50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.123+0000 7f66e63d9740 1 --2- 172.21.15.28:0/265458408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b035f570 0x55f2b0372a00 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.123+0000 7f66e63d9740 1 --2- 172.21.15.28:0/265458408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2b0354ea0 0x55f2b0374af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.123+0000 7f66c4ff9640 0 client.16029 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:34:10.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.123+0000 7f66e63d9740 1 --2- 172.21.15.28:0/265458408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2b0354580 0x55f2b03745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.123+0000 7f66e63d9740 1 -- 172.21.15.28:0/265458408 >> 172.21.15.28:0/265458408 conn(0x55f2b03508a0 msgr2=0x55f2b036afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.123+0000 7f66e63d9740 1 -- 172.21.15.28:0/265458408 shutdown_connections 2024-04-25T01:34:10.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.123+0000 7f66e63d9740 1 -- 172.21.15.28:0/265458408 wait complete. 2024-04-25T01:34:10.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.123+0000 7f66e63d9740 20 client.16029 trim_cache size 0 max 16384 2024-04-25T01:34:10.105 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:34:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.124+0000 7f66e36cd640 1 -- 172.21.15.28:0/216551719 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f66c80a2d00 msgr2=0x7f66c80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.124+0000 7f66e36cd640 1 --2- 172.21.15.28:0/216551719 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f66c80a2d00 0x7f66c80a51c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f66d4009270 tx=0x7f66d4014d10 comp rx=0 tx=0).stop 2024-04-25T01:34:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.124+0000 7f66e36cd640 1 -- 172.21.15.28:0/216551719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66dc100fd0 msgr2=0x7f66dc196520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.124+0000 7f66e36cd640 1 --2- 172.21.15.28:0/216551719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66dc100fd0 0x7f66dc196520 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f66cc007860 tx=0x7f66cc007c60 comp rx=0 tx=0).stop 2024-04-25T01:34:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.124+0000 7f66e36cd640 1 -- 172.21.15.28:0/216551719 shutdown_connections 2024-04-25T01:34:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.124+0000 7f66e36cd640 1 --2- 172.21.15.28:0/216551719 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f66c80a2d00 0x7f66c80a51c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.124+0000 7f66e36cd640 1 --2- 172.21.15.28:0/216551719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66dc19a850 0x7f66dc19ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.124+0000 7f66e36cd640 1 --2- 172.21.15.28:0/216551719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f66dc100fd0 0x7f66dc196520 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.124+0000 7f66e36cd640 1 --2- 172.21.15.28:0/216551719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66dc100610 0x7f66dc195fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.124+0000 7f66e36cd640 1 -- 172.21.15.28:0/216551719 >> 172.21.15.28:0/216551719 conn(0x7f66dc077740 msgr2=0x7f66dc0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.124+0000 7f66e36cd640 1 -- 172.21.15.28:0/216551719 shutdown_connections 2024-04-25T01:34:10.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.124+0000 7f66e36cd640 1 -- 172.21.15.28:0/216551719 wait complete. 2024-04-25T01:34:10.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:34:10.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.248+0000 7fec0d858640 1 -- 172.21.15.28:0/3857530002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec08100610 msgr2=0x7fec08100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.248+0000 7fec0d858640 1 --2- 172.21.15.28:0/3857530002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec08100610 0x7fec08100a90 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7febfc009930 tx=0x7febfc02a420 comp rx=0 tx=0).stop 2024-04-25T01:34:10.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.248+0000 7fec0d858640 1 -- 172.21.15.28:0/3857530002 shutdown_connections 2024-04-25T01:34:10.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.248+0000 7fec0d858640 1 --2- 172.21.15.28:0/3857530002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec08100fd0 0x7fec0810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.248+0000 7fec0d858640 1 --2- 172.21.15.28:0/3857530002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec08100610 0x7fec08100a90 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.248+0000 7fec0d858640 1 --2- 172.21.15.28:0/3857530002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec081066b0 0x7fec08106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.248+0000 7fec0d858640 1 -- 172.21.15.28:0/3857530002 >> 172.21.15.28:0/3857530002 conn(0x7fec080769c0 msgr2=0x7fec08076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.248+0000 7fec0d858640 1 -- 172.21.15.28:0/3857530002 shutdown_connections 2024-04-25T01:34:10.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.248+0000 7fec0d858640 1 -- 172.21.15.28:0/3857530002 wait complete. 2024-04-25T01:34:10.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.249+0000 7fec0d858640 1 Processor -- start 2024-04-25T01:34:10.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.249+0000 7fec0d858640 1 -- start start 2024-04-25T01:34:10.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.249+0000 7fec0d858640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec08100610 0x7fec08196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.249+0000 7fec0d858640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec08100fd0 0x7fec08197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.250+0000 7fec0d858640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec081066b0 0x7fec0819b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.250+0000 7fec0d858640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fec08110a40 con 0x7fec081066b0 2024-04-25T01:34:10.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.250+0000 7fec06ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec08100610 0x7fec08196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.250+0000 7fec077fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec081066b0 0x7fec0819b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.250+0000 7fec06ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec08100610 0x7fec08196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.28:46748/0 (socket says 172.21.15.28:46748) 2024-04-25T01:34:10.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.250+0000 7fec06ffd640 1 -- 172.21.15.28:0/2369960751 learned_addr learned my addr 172.21.15.28:0/2369960751 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.250+0000 7fec0d858640 1 -- 172.21.15.28:0/2369960751 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fec081108c0 con 0x7fec08100fd0 2024-04-25T01:34:10.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.250+0000 7fec067fc640 1 --2- 172.21.15.28:0/2369960751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec08100fd0 0x7fec08197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.250+0000 7fec0d858640 1 -- 172.21.15.28:0/2369960751 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fec08110bc0 con 0x7fec08100610 2024-04-25T01:34:10.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.250+0000 7fec077fe640 1 -- 172.21.15.28:0/2369960751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec08100610 msgr2=0x7fec08196ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.250+0000 7fec077fe640 1 --2- 172.21.15.28:0/2369960751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec08100610 0x7fec08196ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.250+0000 7fec077fe640 1 -- 172.21.15.28:0/2369960751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec08100fd0 msgr2=0x7fec08197420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.250+0000 7fec077fe640 1 --2- 172.21.15.28:0/2369960751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec08100fd0 0x7fec08197420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.250+0000 7fec077fe640 1 -- 172.21.15.28:0/2369960751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec0819bed0 con 0x7fec081066b0 2024-04-25T01:34:10.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.250+0000 7fec077fe640 1 --2- 172.21.15.28:0/2369960751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec081066b0 0x7fec0819b7a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fec00009fd0 tx=0x7fec0000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.251+0000 7fec0c856640 1 -- 172.21.15.28:0/2369960751 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec0000ce20 con 0x7fec081066b0 2024-04-25T01:34:10.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.251+0000 7fec0c856640 1 -- 172.21.15.28:0/2369960751 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec00005020 con 0x7fec081066b0 2024-04-25T01:34:10.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.251+0000 7fec0d858640 1 -- 172.21.15.28:0/2369960751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec0819c1c0 con 0x7fec081066b0 2024-04-25T01:34:10.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.251+0000 7fec0c856640 1 -- 172.21.15.28:0/2369960751 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec00010720 con 0x7fec081066b0 2024-04-25T01:34:10.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.251+0000 7fec0d858640 1 -- 172.21.15.28:0/2369960751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec08069d40 con 0x7fec081066b0 2024-04-25T01:34:10.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.254+0000 7fec10563740 1 Processor -- start 2024-04-25T01:34:10.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.254+0000 7fec0c856640 1 -- 172.21.15.28:0/2369960751 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7fec000108e0 con 0x7fec081066b0 2024-04-25T01:34:10.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.254+0000 7fec10563740 1 -- start start 2024-04-25T01:34:10.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.255+0000 7fec10563740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6bd580 0x55bf2f6bd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.255+0000 7fec10563740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6bdea0 0x55bf2f6c8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.255+0000 7fec10563740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6c8570 0x55bf2f6ca960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.255+0000 7fec10563740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bf2f622670 con 0x55bf2f6bdea0 2024-04-25T01:34:10.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.255+0000 7fec10563740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55bf2f5ca970 con 0x55bf2f6bd580 2024-04-25T01:34:10.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.255+0000 7fec10563740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55bf2f61e630 con 0x55bf2f6c8570 2024-04-25T01:34:10.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.255+0000 7fec06ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6bdea0 0x55bf2f6c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.255+0000 7fec067fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6bd580 0x55bf2f6bd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.255+0000 7fec06ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6bdea0 0x55bf2f6c8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58898/0 (socket says 172.21.15.28:58898) 2024-04-25T01:34:10.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.255+0000 7fec06ffd640 1 -- 172.21.15.28:0/35163088 learned_addr learned my addr 172.21.15.28:0/35163088 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.255+0000 7fec067fc640 1 --2- 172.21.15.28:0/35163088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6c8570 0x55bf2f6ca960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.255+0000 7fec0c856640 1 --2- 172.21.15.28:0/2369960751 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7febf00a2ea0 0x7febf00a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.255+0000 7fec06ffd640 1 -- 172.21.15.28:0/35163088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6c8570 msgr2=0x55bf2f6ca960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.255+0000 7fec06ffd640 1 --2- 172.21.15.28:0/35163088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6c8570 0x55bf2f6ca960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.255+0000 7fec06ffd640 1 -- 172.21.15.28:0/35163088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6bd580 msgr2=0x55bf2f6bd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.255+0000 7fec06ffd640 1 --2- 172.21.15.28:0/35163088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6bd580 0x55bf2f6bd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.255+0000 7fec06ffd640 1 -- 172.21.15.28:0/35163088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf2f5ee400 con 0x55bf2f6bdea0 2024-04-25T01:34:10.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.255+0000 7fec077fe640 1 --2- 172.21.15.28:0/2369960751 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7febf00a2ea0 0x7febf00a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.256+0000 7fec0c856640 1 -- 172.21.15.28:0/2369960751 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fec000d0130 con 0x7fec081066b0 2024-04-25T01:34:10.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.256+0000 7fec06ffd640 1 --2- 172.21.15.28:0/35163088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6bdea0 0x55bf2f6c8030 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7febf800d980 tx=0x7febf800c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7914302ad47db1df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.256+0000 7fec077fe640 1 --2- 172.21.15.28:0/2369960751 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7febf00a2ea0 0x7febf00a5360 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fec000cd3c0 tx=0x7fec00067130 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.256+0000 7febedffb640 1 -- 172.21.15.28:0/35163088 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf800cd00 con 0x55bf2f6bdea0 2024-04-25T01:34:10.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.256+0000 7febedffb640 1 -- 172.21.15.28:0/35163088 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febf801b070 con 0x55bf2f6bdea0 2024-04-25T01:34:10.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.257+0000 7febedffb640 1 -- 172.21.15.28:0/35163088 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf8015760 con 0x55bf2f6bdea0 2024-04-25T01:34:10.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.257+0000 7fec10563740 1 -- 172.21.15.28:0/35163088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6bdea0 msgr2=0x55bf2f6c8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.257+0000 7fec10563740 1 --2- 172.21.15.28:0/35163088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6bdea0 0x55bf2f6c8030 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7febf800d980 tx=0x7febf800c2f0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.257+0000 7fec10563740 1 -- 172.21.15.28:0/35163088 shutdown_connections 2024-04-25T01:34:10.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.257+0000 7fec10563740 1 --2- 172.21.15.28:0/35163088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6c8570 0x55bf2f6ca960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.257+0000 7fec10563740 1 --2- 172.21.15.28:0/35163088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6bdea0 0x55bf2f6c8030 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.257+0000 7fec10563740 1 --2- 172.21.15.28:0/35163088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6bd580 0x55bf2f6bd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.257+0000 7fec10563740 1 -- 172.21.15.28:0/35163088 >> 172.21.15.28:0/35163088 conn(0x55bf2f6b98a0 msgr2=0x55bf2f6b9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.257+0000 7fec10563740 1 -- 172.21.15.28:0/35163088 shutdown_connections 2024-04-25T01:34:10.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.257+0000 7fec10563740 1 -- 172.21.15.28:0/35163088 wait complete. 2024-04-25T01:34:10.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.257+0000 7fec10563740 1 -- start start 2024-04-25T01:34:10.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.258+0000 7fec10563740 1 Processor -- start 2024-04-25T01:34:10.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.258+0000 7fec10563740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6bd580 0x55bf2f6c7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.258+0000 7fec10563740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6bdea0 0x55bf2f6c50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.258+0000 7fec10563740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6c8570 0x55bf2f6c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.258+0000 7fec10563740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bf2f6cb400 con 0x55bf2f6bdea0 2024-04-25T01:34:10.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.258+0000 7fec10563740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55bf2f5de050 con 0x55bf2f6c8570 2024-04-25T01:34:10.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.258+0000 7fec10563740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55bf2f6cb580 con 0x55bf2f6bd580 2024-04-25T01:34:10.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.258+0000 7fec10563740 10 client.0 _ll_register_callbacks cb 0x55bf2f335be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:10.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.258+0000 7fec06ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6bdea0 0x55bf2f6c50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.258+0000 7fec06ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6bdea0 0x55bf2f6c50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58904/0 (socket says 172.21.15.28:58904) 2024-04-25T01:34:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.258+0000 7fec06ffd640 1 -- 172.21.15.28:0/3130463932 learned_addr learned my addr 172.21.15.28:0/3130463932 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.258+0000 7fec067fc640 1 --2- 172.21.15.28:0/3130463932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6bd580 0x55bf2f6c7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.258+0000 7fec067fc640 1 --2- 172.21.15.28:0/3130463932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6c8570 0x55bf2f6c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.259+0000 7fec06ffd640 1 -- 172.21.15.28:0/3130463932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6bd580 msgr2=0x55bf2f6c7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.259+0000 7fec06ffd640 1 --2- 172.21.15.28:0/3130463932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6bd580 0x55bf2f6c7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.259+0000 7fec06ffd640 1 -- 172.21.15.28:0/3130463932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6c8570 msgr2=0x55bf2f6c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.259+0000 7fec06ffd640 1 --2- 172.21.15.28:0/3130463932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6c8570 0x55bf2f6c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.259+0000 7fec06ffd640 1 -- 172.21.15.28:0/3130463932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf2f5afdc0 con 0x55bf2f6bdea0 2024-04-25T01:34:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.259+0000 7fec06ffd640 1 --2- 172.21.15.28:0/3130463932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6bdea0 0x55bf2f6c50d0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7febf800d010 tx=0x7febf800d4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9f1e91c8a5a551 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.259+0000 7febe6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.259+0000 7febe6ffd640 1 -- 172.21.15.28:0/3130463932 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf8005460 con 0x55bf2f6bdea0 2024-04-25T01:34:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.259+0000 7febe6ffd640 1 -- 172.21.15.28:0/3130463932 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7febf8016480 con 0x55bf2f6bdea0 2024-04-25T01:34:10.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.259+0000 7fec10563740 1 -- 172.21.15.28:0/3130463932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febf801b070 con 0x55bf2f6bdea0 2024-04-25T01:34:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.259+0000 7febe6ffd640 1 -- 172.21.15.28:0/3130463932 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7febf8005600 con 0x55bf2f6bdea0 2024-04-25T01:34:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.260+0000 7fec10563740 10 client.16041 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.260+0000 7fec10563740 1 -- 172.21.15.28:0/3130463932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bf2f3363f0 con 0x55bf2f6bdea0 2024-04-25T01:34:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.260+0000 7febe6ffd640 1 -- 172.21.15.28:0/3130463932 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febf801b070 con 0x55bf2f6bdea0 2024-04-25T01:34:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.260+0000 7febe6ffd640 1 -- 172.21.15.28:0/3130463932 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7febf802a470 con 0x55bf2f6bdea0 2024-04-25T01:34:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.260+0000 7fec10563740 10 client.16041 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.260+0000 7fec10563740 10 client.16041 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:34:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.260+0000 7fec10563740 20 client.16041 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.260+0000 7fec10563740 1 --2- 172.21.15.28:0/3130463932 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55bf2f6e0dc0 0x55bf2f6e31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.260+0000 7fec10563740 4 client.16041 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.260+0000 7fec10563740 1 -- 172.21.15.28:0/3130463932 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf2f62fbb0 con 0x55bf2f6e0dc0 2024-04-25T01:34:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.260+0000 7fec067fc640 1 --2- 172.21.15.28:0/3130463932 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55bf2f6e0dc0 0x55bf2f6e31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.261+0000 7fec067fc640 1 --2- 172.21.15.28:0/3130463932 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55bf2f6e0dc0 0x55bf2f6e31b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9b7488af91a3bc3b server_cookie=2285845ccd543ef2 in_seq=0 out_seq=0 2024-04-25T01:34:10.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.261+0000 7febe6ffd640 10 client.16041 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:34:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.262+0000 7febe6ffd640 1 -- 172.21.15.28:0/3130463932 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bf2f62fbb0 con 0x55bf2f6e0dc0 2024-04-25T01:34:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.262+0000 7febe6ffd640 10 client.16041 handle_command_reply: tid=0 2024-04-25T01:34:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.262+0000 7fec10563740 1 client.16041 shutdown 2024-04-25T01:34:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.262+0000 7fec10563740 1 -- 172.21.15.28:0/3130463932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6bdea0 msgr2=0x55bf2f6c50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.262+0000 7fec10563740 1 --2- 172.21.15.28:0/3130463932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6bdea0 0x55bf2f6c50d0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7febf800d010 tx=0x7febf800d4b0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.262+0000 7fec10563740 1 -- 172.21.15.28:0/3130463932 shutdown_connections 2024-04-25T01:34:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.262+0000 7fec10563740 1 --2- 172.21.15.28:0/3130463932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6c8570 0x55bf2f6c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.262+0000 7fec10563740 1 --2- 172.21.15.28:0/3130463932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6bdea0 0x55bf2f6c50d0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.262+0000 7fec10563740 1 --2- 172.21.15.28:0/3130463932 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55bf2f6e0dc0 0x55bf2f6e31b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.262+0000 7fec10563740 1 --2- 172.21.15.28:0/3130463932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6bd580 0x55bf2f6c7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.262+0000 7fec10563740 1 -- 172.21.15.28:0/3130463932 >> 172.21.15.28:0/3130463932 conn(0x55bf2f6b98a0 msgr2=0x55bf2f6bf6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.262+0000 7febe6ffd640 0 client.16041 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:34:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.262+0000 7fec10563740 1 -- 172.21.15.28:0/3130463932 shutdown_connections 2024-04-25T01:34:10.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.262+0000 7fec10563740 1 -- 172.21.15.28:0/3130463932 wait complete. 2024-04-25T01:34:10.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.262+0000 7fec10563740 20 client.16041 trim_cache size 0 max 16384 2024-04-25T01:34:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.289+0000 7fec10563740 1 Processor -- start 2024-04-25T01:34:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.289+0000 7fec10563740 1 -- start start 2024-04-25T01:34:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.290+0000 7fec10563740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6bd580 0x55bf2f6c0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.290+0000 7fec10563740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6c8570 0x55bf2f6b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.290+0000 7fec10563740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6c7910 0x55bf2f6c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.290+0000 7fec10563740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fec081ac8b0 con 0x55bf2f6c7910 2024-04-25T01:34:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.290+0000 7fec10563740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fec081ae3f0 con 0x55bf2f6c8570 2024-04-25T01:34:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.290+0000 7fec10563740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fec081acf80 con 0x55bf2f6bd580 2024-04-25T01:34:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.290+0000 7fec067fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6c7910 0x55bf2f6c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.290+0000 7fec067fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6c7910 0x55bf2f6c7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58918/0 (socket says 172.21.15.28:58918) 2024-04-25T01:34:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.290+0000 7fec067fc640 1 -- 172.21.15.28:0/1437854834 learned_addr learned my addr 172.21.15.28:0/1437854834 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.290+0000 7fec067fc640 1 --2- 172.21.15.28:0/1437854834 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6bd580 0x55bf2f6c0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.290+0000 7fec06ffd640 1 --2- 172.21.15.28:0/1437854834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6c8570 0x55bf2f6b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.290+0000 7fec067fc640 1 -- 172.21.15.28:0/1437854834 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6bd580 msgr2=0x55bf2f6c0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.290+0000 7fec067fc640 1 --2- 172.21.15.28:0/1437854834 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6bd580 0x55bf2f6c0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.290+0000 7fec067fc640 1 -- 172.21.15.28:0/1437854834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6c8570 msgr2=0x55bf2f6b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.290+0000 7fec067fc640 1 --2- 172.21.15.28:0/1437854834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6c8570 0x55bf2f6b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.290+0000 7fec067fc640 1 -- 172.21.15.28:0/1437854834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf2f6c4080 con 0x55bf2f6c7910 2024-04-25T01:34:10.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.290+0000 7fec067fc640 1 --2- 172.21.15.28:0/1437854834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6c7910 0x55bf2f6c7d30 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7febfc006010 tx=0x7febfc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19af0aea7f2f8348 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.291+0000 7febed7fa640 1 -- 172.21.15.28:0/1437854834 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc04ee60 con 0x55bf2f6c7910 2024-04-25T01:34:10.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.291+0000 7febed7fa640 1 -- 172.21.15.28:0/1437854834 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febfc052070 con 0x55bf2f6c7910 2024-04-25T01:34:10.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.291+0000 7febed7fa640 1 -- 172.21.15.28:0/1437854834 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc035040 con 0x55bf2f6c7910 2024-04-25T01:34:10.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.292+0000 7fec10563740 1 -- 172.21.15.28:0/1437854834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6c7910 msgr2=0x55bf2f6c7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.292+0000 7fec10563740 1 --2- 172.21.15.28:0/1437854834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6c7910 0x55bf2f6c7d30 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7febfc006010 tx=0x7febfc000bc0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.292+0000 7fec10563740 1 -- 172.21.15.28:0/1437854834 shutdown_connections 2024-04-25T01:34:10.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.292+0000 7fec10563740 1 --2- 172.21.15.28:0/1437854834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6c7910 0x55bf2f6c7d30 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.292+0000 7fec10563740 1 --2- 172.21.15.28:0/1437854834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6c8570 0x55bf2f6b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.292+0000 7fec10563740 1 --2- 172.21.15.28:0/1437854834 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6bd580 0x55bf2f6c0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.292+0000 7fec10563740 1 -- 172.21.15.28:0/1437854834 >> 172.21.15.28:0/1437854834 conn(0x55bf2f6b98a0 msgr2=0x55bf2f6b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.292+0000 7fec10563740 1 -- 172.21.15.28:0/1437854834 shutdown_connections 2024-04-25T01:34:10.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.292+0000 7fec10563740 1 -- 172.21.15.28:0/1437854834 wait complete. 2024-04-25T01:34:10.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.292+0000 7fec10563740 1 -- start start 2024-04-25T01:34:10.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec10563740 1 Processor -- start 2024-04-25T01:34:10.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec10563740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6bd580 0x55bf2f6c2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec10563740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6c7910 0x55bf2f6c26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec10563740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6c8570 0x55bf2f746020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec10563740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fec081add20 con 0x55bf2f6c8570 2024-04-25T01:34:10.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec10563740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fec081ad650 con 0x55bf2f6c7910 2024-04-25T01:34:10.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec10563740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fec081aa890 con 0x55bf2f6bd580 2024-04-25T01:34:10.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec10563740 10 client.0 _ll_register_callbacks cb 0x7febfc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:10.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec067fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6c8570 0x55bf2f746020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec06ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6c7910 0x55bf2f6c26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec06ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6c7910 0x55bf2f6c26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:46008/0 (socket says 172.21.15.28:46008) 2024-04-25T01:34:10.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec06ffd640 1 -- 172.21.15.28:0/1819863656 learned_addr learned my addr 172.21.15.28:0/1819863656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec067fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6c8570 0x55bf2f746020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58920/0 (socket says 172.21.15.28:58920) 2024-04-25T01:34:10.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec067fc640 1 --2- 172.21.15.28:0/1819863656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6bd580 0x55bf2f6c2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec06ffd640 1 -- 172.21.15.28:0/1819863656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6bd580 msgr2=0x55bf2f6c2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec06ffd640 1 --2- 172.21.15.28:0/1819863656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6bd580 0x55bf2f6c2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec06ffd640 1 -- 172.21.15.28:0/1819863656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6c8570 msgr2=0x55bf2f746020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec06ffd640 1 --2- 172.21.15.28:0/1819863656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6c8570 0x55bf2f746020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec06ffd640 1 -- 172.21.15.28:0/1819863656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf2f4007d0 con 0x55bf2f6c7910 2024-04-25T01:34:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.293+0000 7fec067fc640 1 --2- 172.21.15.28:0/1819863656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6bd580 0x55bf2f6c2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.294+0000 7fec06ffd640 1 --2- 172.21.15.28:0/1819863656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6c7910 0x55bf2f6c26b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7febf800d520 tx=0x7febf802a6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5898825da709afb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.294+0000 7febe6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-25T01:34:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.294+0000 7febe6ffd640 1 -- 172.21.15.28:0/1819863656 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf8005a70 con 0x55bf2f6c7910 2024-04-25T01:34:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.294+0000 7febe6ffd640 1 -- 172.21.15.28:0/1819863656 <== mon.2 v2:172.21.15.32:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7febf802ac20 con 0x55bf2f6c7910 2024-04-25T01:34:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.294+0000 7fec10563740 1 -- 172.21.15.28:0/1819863656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febfc052070 con 0x55bf2f6c7910 2024-04-25T01:34:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.294+0000 7febe6ffd640 1 -- 172.21.15.28:0/1819863656 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7febf8016480 con 0x55bf2f6c7910 2024-04-25T01:34:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.294+0000 7fec10563740 10 client.25571 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.294+0000 7fec10563740 1 -- 172.21.15.28:0/1819863656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bf2f747490 con 0x55bf2f6c7910 2024-04-25T01:34:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.294+0000 7febe6ffd640 1 -- 172.21.15.28:0/1819863656 <== mon.2 v2:172.21.15.32:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febf801b040 con 0x55bf2f6c7910 2024-04-25T01:34:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.295+0000 7febe6ffd640 1 -- 172.21.15.28:0/1819863656 <== mon.2 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7febf8004a60 con 0x55bf2f6c7910 2024-04-25T01:34:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.295+0000 7fec10563740 10 client.25571 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:10.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.295+0000 7fec10563740 10 client.25571 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:34:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.295+0000 7fec10563740 20 client.25571 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.295+0000 7fec10563740 1 --2- 172.21.15.28:0/1819863656 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55bf2f747c40 0x55bf2f74a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.295+0000 7fec10563740 4 client.25571 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:34:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.295+0000 7fec10563740 1 -- 172.21.15.28:0/1819863656 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7febfc052070 con 0x55bf2f747c40 2024-04-25T01:34:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.295+0000 7fec067fc640 1 --2- 172.21.15.28:0/1819863656 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55bf2f747c40 0x55bf2f74a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.296+0000 7fec067fc640 1 --2- 172.21.15.28:0/1819863656 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55bf2f747c40 0x55bf2f74a030 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bad8e3507f06b39e server_cookie=10832087df912ab in_seq=0 out_seq=0 2024-04-25T01:34:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.296+0000 7febe6ffd640 10 client.25571 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:34:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.296+0000 7febe6ffd640 1 -- 172.21.15.28:0/1819863656 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7febfc052070 con 0x55bf2f747c40 2024-04-25T01:34:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.296+0000 7febe6ffd640 10 client.25571 handle_command_reply: tid=0 2024-04-25T01:34:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.296+0000 7fec10563740 1 client.25571 shutdown 2024-04-25T01:34:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.296+0000 7fec10563740 1 -- 172.21.15.28:0/1819863656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6c7910 msgr2=0x55bf2f6c26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.296+0000 7fec10563740 1 --2- 172.21.15.28:0/1819863656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6c7910 0x55bf2f6c26b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7febf800d520 tx=0x7febf802a6a0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.296+0000 7fec10563740 1 -- 172.21.15.28:0/1819863656 shutdown_connections 2024-04-25T01:34:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.296+0000 7fec10563740 1 --2- 172.21.15.28:0/1819863656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf2f6c8570 0x55bf2f746020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.296+0000 7fec10563740 1 --2- 172.21.15.28:0/1819863656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf2f6c7910 0x55bf2f6c26b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.296+0000 7fec10563740 1 --2- 172.21.15.28:0/1819863656 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55bf2f747c40 0x55bf2f74a030 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.296+0000 7fec10563740 1 --2- 172.21.15.28:0/1819863656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf2f6bd580 0x55bf2f6c2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.296+0000 7fec10563740 1 -- 172.21.15.28:0/1819863656 >> 172.21.15.28:0/1819863656 conn(0x55bf2f6b98a0 msgr2=0x55bf2f6c0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.296+0000 7febe6ffd640 0 client.25571 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:34:10.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.296+0000 7fec10563740 1 -- 172.21.15.28:0/1819863656 shutdown_connections 2024-04-25T01:34:10.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.296+0000 7fec10563740 1 -- 172.21.15.28:0/1819863656 wait complete. 2024-04-25T01:34:10.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.297+0000 7fec10563740 20 client.25571 trim_cache size 0 max 16384 2024-04-25T01:34:10.278 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:34:10.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.297+0000 7fec0d858640 1 -- 172.21.15.28:0/2369960751 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7febf00a2ea0 msgr2=0x7febf00a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.297+0000 7fec0d858640 1 --2- 172.21.15.28:0/2369960751 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7febf00a2ea0 0x7febf00a5360 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fec000cd3c0 tx=0x7fec00067130 comp rx=0 tx=0).stop 2024-04-25T01:34:10.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.297+0000 7fec0d858640 1 -- 172.21.15.28:0/2369960751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec081066b0 msgr2=0x7fec0819b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.297+0000 7fec0d858640 1 --2- 172.21.15.28:0/2369960751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec081066b0 0x7fec0819b7a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fec00009fd0 tx=0x7fec0000ef10 comp rx=0 tx=0).stop 2024-04-25T01:34:10.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.298+0000 7fec0d858640 1 -- 172.21.15.28:0/2369960751 shutdown_connections 2024-04-25T01:34:10.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.298+0000 7fec0d858640 1 --2- 172.21.15.28:0/2369960751 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7febf00a2ea0 0x7febf00a5360 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.298+0000 7fec0d858640 1 --2- 172.21.15.28:0/2369960751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec081066b0 0x7fec0819b7a0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.298+0000 7fec0d858640 1 --2- 172.21.15.28:0/2369960751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fec08100fd0 0x7fec08197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.298+0000 7fec0d858640 1 --2- 172.21.15.28:0/2369960751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fec08100610 0x7fec08196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.298+0000 7fec0d858640 1 -- 172.21.15.28:0/2369960751 >> 172.21.15.28:0/2369960751 conn(0x7fec080769c0 msgr2=0x7fec080feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.298+0000 7fec0d858640 1 -- 172.21.15.28:0/2369960751 shutdown_connections 2024-04-25T01:34:10.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.298+0000 7fec0d858640 1 -- 172.21.15.28:0/2369960751 wait complete. 2024-04-25T01:34:10.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:34:10.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.426+0000 7fec04495640 1 -- 172.21.15.28:0/2580113981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc100510 msgr2=0x7febfc1008f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.426+0000 7fec04495640 1 --2- 172.21.15.28:0/2580113981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc100510 0x7febfc1008f0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7febf8009900 tx=0x7febf802a410 comp rx=0 tx=0).stop 2024-04-25T01:34:10.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.426+0000 7fec04495640 1 -- 172.21.15.28:0/2580113981 shutdown_connections 2024-04-25T01:34:10.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.426+0000 7fec04495640 1 --2- 172.21.15.28:0/2580113981 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7febfc10bce0 0x7febfc10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.426+0000 7fec04495640 1 --2- 172.21.15.28:0/2580113981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febfc100ec0 0x7febfc10b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.426+0000 7fec04495640 1 --2- 172.21.15.28:0/2580113981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc100510 0x7febfc1008f0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.426+0000 7fec04495640 1 -- 172.21.15.28:0/2580113981 >> 172.21.15.28:0/2580113981 conn(0x7febfc075940 msgr2=0x7febfc075d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.426+0000 7fec04495640 1 -- 172.21.15.28:0/2580113981 shutdown_connections 2024-04-25T01:34:10.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.426+0000 7fec04495640 1 -- 172.21.15.28:0/2580113981 wait complete. 2024-04-25T01:34:10.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.427+0000 7fec04495640 1 Processor -- start 2024-04-25T01:34:10.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.427+0000 7fec04495640 1 -- start start 2024-04-25T01:34:10.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.427+0000 7fec04495640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febfc100510 0x7febfc101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.427+0000 7fec04495640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc10bce0 0x7febfc1019c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.427+0000 7fec04495640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7febfc101f00 0x7febfc1a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.427+0000 7fec04495640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7febfc111b20 con 0x7febfc10bce0 2024-04-25T01:34:10.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.427+0000 7fec04495640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7febfc1119a0 con 0x7febfc100510 2024-04-25T01:34:10.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.427+0000 7fec04495640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7febfc111ca0 con 0x7febfc101f00 2024-04-25T01:34:10.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.427+0000 7fec02231640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febfc100510 0x7febfc101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.427+0000 7fec02231640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febfc100510 0x7febfc101480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:46020/0 (socket says 172.21.15.28:46020) 2024-04-25T01:34:10.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.427+0000 7fec02231640 1 -- 172.21.15.28:0/719278886 learned_addr learned my addr 172.21.15.28:0/719278886 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.427+0000 7fec01a30640 1 --2- 172.21.15.28:0/719278886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc10bce0 0x7febfc1019c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.427+0000 7fec02a32640 1 --2- 172.21.15.28:0/719278886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7febfc101f00 0x7febfc1a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.428+0000 7fec02231640 1 -- 172.21.15.28:0/719278886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7febfc101f00 msgr2=0x7febfc1a6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.428+0000 7fec02231640 1 --2- 172.21.15.28:0/719278886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7febfc101f00 0x7febfc1a6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.428+0000 7fec02231640 1 -- 172.21.15.28:0/719278886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc10bce0 msgr2=0x7febfc1019c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.428+0000 7fec02231640 1 --2- 172.21.15.28:0/719278886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc10bce0 0x7febfc1019c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.428+0000 7fec02231640 1 -- 172.21.15.28:0/719278886 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febfc1a68b0 con 0x7febfc100510 2024-04-25T01:34:10.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.428+0000 7fec01a30640 1 --2- 172.21.15.28:0/719278886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc10bce0 0x7febfc1019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:10.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.428+0000 7fec02231640 1 --2- 172.21.15.28:0/719278886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febfc100510 0x7febfc101480 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7febf8006010 tx=0x7febf80042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.428+0000 7febf37fe640 1 -- 172.21.15.28:0/719278886 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf803f030 con 0x7febfc100510 2024-04-25T01:34:10.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.428+0000 7febf37fe640 1 -- 172.21.15.28:0/719278886 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febf8003ce0 con 0x7febfc100510 2024-04-25T01:34:10.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.428+0000 7fec04495640 1 -- 172.21.15.28:0/719278886 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febfc1a6b40 con 0x7febfc100510 2024-04-25T01:34:10.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.428+0000 7febf37fe640 1 -- 172.21.15.28:0/719278886 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf8035070 con 0x7febfc100510 2024-04-25T01:34:10.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.429+0000 7fec04495640 1 -- 172.21.15.28:0/719278886 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febfc1a7080 con 0x7febfc100510 2024-04-25T01:34:10.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.431+0000 7fec0719c740 1 Processor -- start 2024-04-25T01:34:10.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.432+0000 7fec0719c740 1 -- start start 2024-04-25T01:34:10.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.432+0000 7fec0719c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56433147fd90 0x564331480170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.432+0000 7fec0719c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5643314806b0 0x56433148a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.432+0000 7fec0719c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433148ad80 0x56433148d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.432+0000 7fec0719c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5643313dc670 con 0x56433148ad80 2024-04-25T01:34:10.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.432+0000 7fec0719c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564331384970 con 0x56433147fd90 2024-04-25T01:34:10.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.432+0000 7fec0719c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5643313d8630 con 0x5643314806b0 2024-04-25T01:34:10.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.432+0000 7febf37fe640 1 -- 172.21.15.28:0/719278886 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7febf8042030 con 0x7febfc100510 2024-04-25T01:34:10.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.432+0000 7fec02a32640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56433147fd90 0x564331480170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.432+0000 7fec01a30640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433148ad80 0x56433148d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.432+0000 7fec02a32640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56433147fd90 0x564331480170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:46034/0 (socket says 172.21.15.28:46034) 2024-04-25T01:34:10.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.432+0000 7fec02a32640 1 -- 172.21.15.28:0/3733116542 learned_addr learned my addr 172.21.15.28:0/3733116542 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.433+0000 7fec02a32640 1 -- 172.21.15.28:0/3733116542 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5643314806b0 msgr2=0x56433148a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.433+0000 7fec02231640 1 --2- 172.21.15.28:0/3733116542 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5643314806b0 0x56433148a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.433+0000 7fec02a32640 1 --2- 172.21.15.28:0/3733116542 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5643314806b0 0x56433148a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.433+0000 7fec02a32640 1 -- 172.21.15.28:0/3733116542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433148ad80 msgr2=0x56433148d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.433+0000 7fec02a32640 1 --2- 172.21.15.28:0/3733116542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433148ad80 0x56433148d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.433+0000 7fec02a32640 1 -- 172.21.15.28:0/3733116542 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643313a8400 con 0x56433147fd90 2024-04-25T01:34:10.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.433+0000 7fec01a30640 1 --2- 172.21.15.28:0/3733116542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433148ad80 0x56433148d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:10.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.433+0000 7fec02231640 1 --2- 172.21.15.28:0/3733116542 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5643314806b0 0x56433148a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:10.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.433+0000 7febf37fe640 1 --2- 172.21.15.28:0/719278886 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7febe80a2ea0 0x7febe80a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.433+0000 7fec02a32640 1 --2- 172.21.15.28:0/3733116542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56433147fd90 0x564331480170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7febf4009f70 tx=0x7febf4010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9239815d31d45d6a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.433+0000 7febf37fe640 1 -- 172.21.15.28:0/719278886 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7febf80efe10 con 0x7febfc100510 2024-04-25T01:34:10.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.433+0000 7fec02231640 1 --2- 172.21.15.28:0/719278886 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7febe80a2ea0 0x7febe80a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.433+0000 7febf0bfe640 1 -- 172.21.15.28:0/3733116542 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf4016070 con 0x56433147fd90 2024-04-25T01:34:10.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.433+0000 7febf0bfe640 1 -- 172.21.15.28:0/3733116542 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febf4004d20 con 0x56433147fd90 2024-04-25T01:34:10.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.433+0000 7febf0bfe640 1 -- 172.21.15.28:0/3733116542 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf4012910 con 0x56433147fd90 2024-04-25T01:34:10.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.434+0000 7fec0719c740 1 -- 172.21.15.28:0/3733116542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56433147fd90 msgr2=0x564331480170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.434+0000 7fec0719c740 1 --2- 172.21.15.28:0/3733116542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56433147fd90 0x564331480170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7febf4009f70 tx=0x7febf4010800 comp rx=0 tx=0).stop 2024-04-25T01:34:10.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.434+0000 7fec02231640 1 --2- 172.21.15.28:0/719278886 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7febe80a2ea0 0x7febe80a5360 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7febf80ed100 tx=0x7febf802a410 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.434+0000 7fec0719c740 1 -- 172.21.15.28:0/3733116542 shutdown_connections 2024-04-25T01:34:10.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.434+0000 7fec0719c740 1 --2- 172.21.15.28:0/3733116542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433148ad80 0x56433148d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.434+0000 7fec0719c740 1 --2- 172.21.15.28:0/3733116542 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5643314806b0 0x56433148a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.434+0000 7fec0719c740 1 --2- 172.21.15.28:0/3733116542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56433147fd90 0x564331480170 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.434+0000 7fec0719c740 1 -- 172.21.15.28:0/3733116542 >> 172.21.15.28:0/3733116542 conn(0x56433147c0b0 msgr2=0x56433147c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.434+0000 7fec0719c740 1 -- 172.21.15.28:0/3733116542 shutdown_connections 2024-04-25T01:34:10.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.434+0000 7fec0719c740 1 -- 172.21.15.28:0/3733116542 wait complete. 2024-04-25T01:34:10.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.435+0000 7fec0719c740 1 -- start start 2024-04-25T01:34:10.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.435+0000 7fec0719c740 1 Processor -- start 2024-04-25T01:34:10.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.435+0000 7fec0719c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 0x56433148a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.435+0000 7fec0719c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5643314806b0 0x564331487a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.435+0000 7fec0719c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56433148ad80 0x564331487fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.435+0000 7fec0719c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56433148db70 con 0x56433147fd90 2024-04-25T01:34:10.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.435+0000 7fec0719c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564331398050 con 0x56433148ad80 2024-04-25T01:34:10.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.435+0000 7fec0719c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56433148dcf0 con 0x5643314806b0 2024-04-25T01:34:10.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.435+0000 7fec0719c740 10 client.0 _ll_register_callbacks cb 0x5643310efbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:10.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.435+0000 7fec02a32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 0x56433148a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.435+0000 7fec02a32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 0x56433148a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58964/0 (socket says 172.21.15.28:58964) 2024-04-25T01:34:10.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.435+0000 7fec02a32640 1 -- 172.21.15.28:0/2395369872 learned_addr learned my addr 172.21.15.28:0/2395369872 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.435+0000 7fec01a30640 1 --2- 172.21.15.28:0/2395369872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5643314806b0 0x564331487a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.436+0000 7fec01a30640 1 -- 172.21.15.28:0/2395369872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56433148ad80 msgr2=0x564331487fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:34:10.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.436+0000 7fec01a30640 1 --2- 172.21.15.28:0/2395369872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56433148ad80 0x564331487fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.436+0000 7fec01a30640 1 -- 172.21.15.28:0/2395369872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 msgr2=0x56433148a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.436+0000 7fec01a30640 1 --2- 172.21.15.28:0/2395369872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 0x56433148a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.436+0000 7fec01a30640 1 -- 172.21.15.28:0/2395369872 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564331369dc0 con 0x5643314806b0 2024-04-25T01:34:10.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.436+0000 7fec02a32640 1 --2- 172.21.15.28:0/2395369872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 0x56433148a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:10.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.436+0000 7fec01a30640 1 --2- 172.21.15.28:0/2395369872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5643314806b0 0x564331487a10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7febfc102ec0 tx=0x7febec00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db5c16460807b3bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.436+0000 7febe5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-25T01:34:10.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.436+0000 7febe5ffb640 1 -- 172.21.15.28:0/2395369872 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febec0120d0 con 0x5643314806b0 2024-04-25T01:34:10.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.436+0000 7fec0719c740 1 -- 172.21.15.28:0/2395369872 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febf4004d20 con 0x5643314806b0 2024-04-25T01:34:10.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.436+0000 7febe5ffb640 1 -- 172.21.15.28:0/2395369872 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7febec004030 con 0x5643314806b0 2024-04-25T01:34:10.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.436+0000 7febe5ffb640 1 -- 172.21.15.28:0/2395369872 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7febec025050 con 0x5643314806b0 2024-04-25T01:34:10.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.437+0000 7fec0719c740 10 client.25525 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:10.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.437+0000 7fec0719c740 1 -- 172.21.15.28:0/2395369872 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5643310f03f0 con 0x5643314806b0 2024-04-25T01:34:10.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.437+0000 7febe5ffb640 1 -- 172.21.15.28:0/2395369872 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febf4004d20 con 0x5643314806b0 2024-04-25T01:34:10.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.437+0000 7febe5ffb640 1 -- 172.21.15.28:0/2395369872 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7febec01c460 con 0x5643314806b0 2024-04-25T01:34:10.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.437+0000 7fec0719c740 10 client.25525 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:10.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.437+0000 7fec0719c740 10 client.25525 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:34:10.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.437+0000 7fec0719c740 20 client.25525 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:10.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.437+0000 7fec0719c740 1 --2- 172.21.15.28:0/2395369872 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56433149b1f0 0x56433149d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.437+0000 7fec0719c740 4 client.25525 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:10.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.437+0000 7fec0719c740 1 -- 172.21.15.28:0/2395369872 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643313e9bb0 con 0x56433149b1f0 2024-04-25T01:34:10.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.437+0000 7fec02a32640 1 --2- 172.21.15.28:0/2395369872 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56433149b1f0 0x56433149d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.438+0000 7febe5ffb640 10 client.25525 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:34:10.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.438+0000 7fec02a32640 1 --2- 172.21.15.28:0/2395369872 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56433149b1f0 0x56433149d650 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6b60a94e8095d294 server_cookie=cce27da59fc764fd in_seq=0 out_seq=0 2024-04-25T01:34:10.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.439+0000 7febe5ffb640 1 -- 172.21.15.28:0/2395369872 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5643313e9bb0 con 0x56433149b1f0 2024-04-25T01:34:10.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.439+0000 7febe5ffb640 10 client.25525 handle_command_reply: tid=0 2024-04-25T01:34:10.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.439+0000 7fec0719c740 1 client.25525 shutdown 2024-04-25T01:34:10.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.439+0000 7fec0719c740 1 -- 172.21.15.28:0/2395369872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5643314806b0 msgr2=0x564331487a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.439+0000 7fec0719c740 1 --2- 172.21.15.28:0/2395369872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5643314806b0 0x564331487a10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7febfc102ec0 tx=0x7febec00f790 comp rx=0 tx=0).stop 2024-04-25T01:34:10.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.439+0000 7fec0719c740 1 -- 172.21.15.28:0/2395369872 shutdown_connections 2024-04-25T01:34:10.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.439+0000 7fec0719c740 1 --2- 172.21.15.28:0/2395369872 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56433149b1f0 0x56433149d650 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.439+0000 7febe5ffb640 0 client.25525 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:34:10.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.439+0000 7fec0719c740 1 --2- 172.21.15.28:0/2395369872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56433148ad80 0x564331487fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.439+0000 7fec0719c740 1 --2- 172.21.15.28:0/2395369872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5643314806b0 0x564331487a10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.439+0000 7fec0719c740 1 --2- 172.21.15.28:0/2395369872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 0x56433148a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.439+0000 7fec0719c740 1 -- 172.21.15.28:0/2395369872 >> 172.21.15.28:0/2395369872 conn(0x56433147c0b0 msgr2=0x564331472b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.439+0000 7fec0719c740 1 -- 172.21.15.28:0/2395369872 shutdown_connections 2024-04-25T01:34:10.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.439+0000 7fec0719c740 1 -- 172.21.15.28:0/2395369872 wait complete. 2024-04-25T01:34:10.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.439+0000 7fec0719c740 20 client.25525 trim_cache size 0 max 16384 2024-04-25T01:34:10.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.453+0000 7fec0719c740 1 Processor -- start 2024-04-25T01:34:10.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.453+0000 7fec0719c740 1 -- start start 2024-04-25T01:34:10.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.453+0000 7fec0719c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 0x56433148a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.453+0000 7fec0719c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643314806b0 0x5643314a2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.453+0000 7fec0719c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56433148ad80 0x5643314a31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.453+0000 7fec0719c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7febfc1ac8c0 con 0x56433147fd90 2024-04-25T01:34:10.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.453+0000 7fec0719c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7febfc1ae400 con 0x5643314806b0 2024-04-25T01:34:10.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.453+0000 7fec0719c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7febfc1acf90 con 0x56433148ad80 2024-04-25T01:34:10.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.453+0000 7fec02a32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 0x56433148a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.453+0000 7fec02a32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 0x56433148a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58980/0 (socket says 172.21.15.28:58980) 2024-04-25T01:34:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.453+0000 7fec02a32640 1 -- 172.21.15.28:0/1282427763 learned_addr learned my addr 172.21.15.28:0/1282427763 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.453+0000 7fec01a30640 1 --2- 172.21.15.28:0/1282427763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643314806b0 0x5643314a2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.453+0000 7fec02a32640 1 --2- 172.21.15.28:0/1282427763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56433148ad80 0x5643314a31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.454+0000 7fec02a32640 1 -- 172.21.15.28:0/1282427763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56433148ad80 msgr2=0x5643314a31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.454+0000 7fec02a32640 1 --2- 172.21.15.28:0/1282427763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56433148ad80 0x5643314a31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.454+0000 7fec02a32640 1 -- 172.21.15.28:0/1282427763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643314806b0 msgr2=0x5643314a2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.454+0000 7fec02a32640 1 --2- 172.21.15.28:0/1282427763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643314806b0 0x5643314a2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.454+0000 7fec02a32640 1 -- 172.21.15.28:0/1282427763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643314869c0 con 0x56433147fd90 2024-04-25T01:34:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.454+0000 7fec01a30640 1 --2- 172.21.15.28:0/1282427763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643314806b0 0x5643314a2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.454+0000 7fec02a32640 1 --2- 172.21.15.28:0/1282427763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 0x56433148a250 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7febf4005ae0 tx=0x7febf4009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f1c266165c75e3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.454+0000 7febe7fff640 1 -- 172.21.15.28:0/1282427763 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf402e070 con 0x56433147fd90 2024-04-25T01:34:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.454+0000 7febe7fff640 1 -- 172.21.15.28:0/1282427763 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febf4003e50 con 0x56433147fd90 2024-04-25T01:34:10.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.454+0000 7febe7fff640 1 -- 172.21.15.28:0/1282427763 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf4016070 con 0x56433147fd90 2024-04-25T01:34:10.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.455+0000 7fec0719c740 1 -- 172.21.15.28:0/1282427763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 msgr2=0x56433148a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.455+0000 7fec0719c740 1 --2- 172.21.15.28:0/1282427763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 0x56433148a250 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7febf4005ae0 tx=0x7febf4009600 comp rx=0 tx=0).stop 2024-04-25T01:34:10.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.455+0000 7fec0719c740 1 -- 172.21.15.28:0/1282427763 shutdown_connections 2024-04-25T01:34:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.455+0000 7fec0719c740 1 --2- 172.21.15.28:0/1282427763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56433148ad80 0x5643314a31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.455+0000 7fec0719c740 1 --2- 172.21.15.28:0/1282427763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5643314806b0 0x5643314a2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.455+0000 7fec0719c740 1 --2- 172.21.15.28:0/1282427763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 0x56433148a250 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.455+0000 7fec0719c740 1 -- 172.21.15.28:0/1282427763 >> 172.21.15.28:0/1282427763 conn(0x56433147c0b0 msgr2=0x564331472b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.455+0000 7fec0719c740 1 -- 172.21.15.28:0/1282427763 shutdown_connections 2024-04-25T01:34:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.455+0000 7fec0719c740 1 -- 172.21.15.28:0/1282427763 wait complete. 2024-04-25T01:34:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.456+0000 7fec0719c740 1 -- start start 2024-04-25T01:34:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.456+0000 7fec0719c740 1 Processor -- start 2024-04-25T01:34:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.456+0000 7fec0719c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 0x5643314a0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.456+0000 7fec0719c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5643314806b0 0x5643314a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.456+0000 7fec0719c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56433148ad80 0x5643314a1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.456+0000 7fec0719c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7febfc1add30 con 0x56433147fd90 2024-04-25T01:34:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.456+0000 7fec0719c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7febfc1ad660 con 0x56433148ad80 2024-04-25T01:34:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.456+0000 7fec0719c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7febfc1aa8c0 con 0x5643314806b0 2024-04-25T01:34:10.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.456+0000 7fec02a32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 0x5643314a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.456+0000 7fec02a32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 0x5643314a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58988/0 (socket says 172.21.15.28:58988) 2024-04-25T01:34:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.456+0000 7fec0719c740 10 client.0 _ll_register_callbacks cb 0x7febec009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.456+0000 7fec02a32640 1 -- 172.21.15.28:0/3692363810 learned_addr learned my addr 172.21.15.28:0/3692363810 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.456+0000 7fec01a30640 1 --2- 172.21.15.28:0/3692363810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5643314806b0 0x5643314a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.456+0000 7fec02a32640 1 --2- 172.21.15.28:0/3692363810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56433148ad80 0x5643314a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.457+0000 7fec02a32640 1 -- 172.21.15.28:0/3692363810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5643314806b0 msgr2=0x5643314a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.457+0000 7fec02a32640 1 --2- 172.21.15.28:0/3692363810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5643314806b0 0x5643314a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.457+0000 7fec02a32640 1 -- 172.21.15.28:0/3692363810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56433148ad80 msgr2=0x5643314a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.457+0000 7fec02a32640 1 --2- 172.21.15.28:0/3692363810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56433148ad80 0x5643314a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.457+0000 7fec02a32640 1 -- 172.21.15.28:0/3692363810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643314804e0 con 0x56433147fd90 2024-04-25T01:34:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.457+0000 7fec02a32640 1 --2- 172.21.15.28:0/3692363810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 0x5643314a0810 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7febf4009d30 tx=0x7febf40056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff920e559d91ff26 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.457+0000 7febe5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.457+0000 7febe5ffb640 1 -- 172.21.15.28:0/3692363810 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf402e070 con 0x56433147fd90 2024-04-25T01:34:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.457+0000 7febe5ffb640 1 -- 172.21.15.28:0/3692363810 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7febf4003b70 con 0x56433147fd90 2024-04-25T01:34:10.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.457+0000 7fec0719c740 1 -- 172.21.15.28:0/3692363810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febf4003e50 con 0x56433147fd90 2024-04-25T01:34:10.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.457+0000 7febe5ffb640 1 -- 172.21.15.28:0/3692363810 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7febf4016070 con 0x56433147fd90 2024-04-25T01:34:10.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.457+0000 7fec0719c740 10 client.16065 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:10.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.457+0000 7fec0719c740 1 -- 172.21.15.28:0/3692363810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5643311ba7d0 con 0x56433147fd90 2024-04-25T01:34:10.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.457+0000 7febe5ffb640 1 -- 172.21.15.28:0/3692363810 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febf4003e50 con 0x56433147fd90 2024-04-25T01:34:10.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.458+0000 7febe5ffb640 1 -- 172.21.15.28:0/3692363810 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7febf400f040 con 0x56433147fd90 2024-04-25T01:34:10.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.458+0000 7fec0719c740 10 client.16065 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:10.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.458+0000 7fec0719c740 10 client.16065 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:34:10.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.458+0000 7fec0719c740 20 client.16065 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:10.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.458+0000 7fec0719c740 1 --2- 172.21.15.28:0/3692363810 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56433149b1f0 0x564331503b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.458+0000 7fec0719c740 4 client.16065 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:34:10.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.458+0000 7fec0719c740 1 -- 172.21.15.28:0/3692363810 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5643313e3f00 con 0x56433149b1f0 2024-04-25T01:34:10.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.458+0000 7fec01a30640 1 --2- 172.21.15.28:0/3692363810 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56433149b1f0 0x564331503b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.458+0000 7febe5ffb640 10 client.16065 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:34:10.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.458+0000 7fec01a30640 1 --2- 172.21.15.28:0/3692363810 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56433149b1f0 0x564331503b30 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=81653b6934ac379f server_cookie=381c724146b1484a in_seq=0 out_seq=0 2024-04-25T01:34:10.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.459+0000 7febe5ffb640 1 -- 172.21.15.28:0/3692363810 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5643313e3f00 con 0x56433149b1f0 2024-04-25T01:34:10.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.459+0000 7febe5ffb640 10 client.16065 handle_command_reply: tid=0 2024-04-25T01:34:10.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.459+0000 7fec0719c740 1 client.16065 shutdown 2024-04-25T01:34:10.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.459+0000 7fec0719c740 1 -- 172.21.15.28:0/3692363810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 msgr2=0x5643314a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.459+0000 7fec0719c740 1 --2- 172.21.15.28:0/3692363810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 0x5643314a0810 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7febf4009d30 tx=0x7febf40056f0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.459+0000 7fec0719c740 1 -- 172.21.15.28:0/3692363810 shutdown_connections 2024-04-25T01:34:10.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.459+0000 7fec0719c740 1 --2- 172.21.15.28:0/3692363810 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56433149b1f0 0x564331503b30 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.459+0000 7fec0719c740 1 --2- 172.21.15.28:0/3692363810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56433148ad80 0x5643314a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.459+0000 7febe5ffb640 0 client.16065 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:34:10.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.459+0000 7fec0719c740 1 --2- 172.21.15.28:0/3692363810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5643314806b0 0x5643314a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.459+0000 7fec0719c740 1 --2- 172.21.15.28:0/3692363810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56433147fd90 0x5643314a0810 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.459+0000 7fec0719c740 1 -- 172.21.15.28:0/3692363810 >> 172.21.15.28:0/3692363810 conn(0x56433147c0b0 msgr2=0x564331495f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.459+0000 7fec0719c740 1 -- 172.21.15.28:0/3692363810 shutdown_connections 2024-04-25T01:34:10.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.459+0000 7fec0719c740 1 -- 172.21.15.28:0/3692363810 wait complete. 2024-04-25T01:34:10.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.459+0000 7fec0719c740 20 client.16065 trim_cache size 0 max 16384 2024-04-25T01:34:10.441 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:34:10.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.460+0000 7fec04495640 1 -- 172.21.15.28:0/719278886 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7febe80a2ea0 msgr2=0x7febe80a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.460+0000 7fec04495640 1 --2- 172.21.15.28:0/719278886 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7febe80a2ea0 0x7febe80a5360 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7febf80ed100 tx=0x7febf802a410 comp rx=0 tx=0).stop 2024-04-25T01:34:10.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.460+0000 7fec04495640 1 -- 172.21.15.28:0/719278886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febfc100510 msgr2=0x7febfc101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.460+0000 7fec04495640 1 --2- 172.21.15.28:0/719278886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febfc100510 0x7febfc101480 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7febf8006010 tx=0x7febf80042f0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.460+0000 7fec04495640 1 -- 172.21.15.28:0/719278886 shutdown_connections 2024-04-25T01:34:10.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.460+0000 7fec04495640 1 --2- 172.21.15.28:0/719278886 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7febe80a2ea0 0x7febe80a5360 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.460+0000 7fec04495640 1 --2- 172.21.15.28:0/719278886 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7febfc101f00 0x7febfc1a6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.460+0000 7fec04495640 1 --2- 172.21.15.28:0/719278886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febfc10bce0 0x7febfc1019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.460+0000 7fec04495640 1 --2- 172.21.15.28:0/719278886 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7febfc100510 0x7febfc101480 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.460+0000 7fec04495640 1 -- 172.21.15.28:0/719278886 >> 172.21.15.28:0/719278886 conn(0x7febfc075940 msgr2=0x7febfc0fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.460+0000 7fec04495640 1 -- 172.21.15.28:0/719278886 shutdown_connections 2024-04-25T01:34:10.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.460+0000 7fec04495640 1 -- 172.21.15.28:0/719278886 wait complete. 2024-04-25T01:34:10.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:10 smithi028 ceph-mon[29390]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-25T01:34:10.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:10 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/917213984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:34:10.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:34:10.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:10 smithi032 ceph-mon[32461]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-25T01:34:10.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:10 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/917213984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:34:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.582+0000 7f4575bd5640 1 -- 172.21.15.28:0/3923909542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f457010bd00 msgr2=0x7f457010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.582+0000 7f4575bd5640 1 --2- 172.21.15.28:0/3923909542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f457010bd00 0x7f457010e1a0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f4568009ca0 tx=0x7f456802a9e0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.582+0000 7f4575bd5640 1 -- 172.21.15.28:0/3923909542 shutdown_connections 2024-04-25T01:34:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.582+0000 7f4575bd5640 1 --2- 172.21.15.28:0/3923909542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f457010bd00 0x7f457010e1a0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.582+0000 7f4575bd5640 1 --2- 172.21.15.28:0/3923909542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45701032e0 0x7f457010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.582+0000 7f4575bd5640 1 --2- 172.21.15.28:0/3923909542 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45701029c0 0x7f4570102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.582+0000 7f4575bd5640 1 -- 172.21.15.28:0/3923909542 >> 172.21.15.28:0/3923909542 conn(0x7f4570077740 msgr2=0x7f4570077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.582+0000 7f4575bd5640 1 -- 172.21.15.28:0/3923909542 shutdown_connections 2024-04-25T01:34:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.582+0000 7f4575bd5640 1 -- 172.21.15.28:0/3923909542 wait complete. 2024-04-25T01:34:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.583+0000 7f4575bd5640 1 Processor -- start 2024-04-25T01:34:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.583+0000 7f4575bd5640 1 -- start start 2024-04-25T01:34:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.583+0000 7f4575bd5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45701029c0 0x7f4570198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.583+0000 7f4575bd5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45701032e0 0x7f4570198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.583+0000 7f4575bd5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f457010bd00 0x7f457019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.583+0000 7f4575bd5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4570111b00 con 0x7f457010bd00 2024-04-25T01:34:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.583+0000 7f4575bd5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4570111980 con 0x7f45701029c0 2024-04-25T01:34:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.583+0000 7f4575bd5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4570111c80 con 0x7f45701032e0 2024-04-25T01:34:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.583+0000 7f456f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45701029c0 0x7f4570198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.583+0000 7f456f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45701029c0 0x7f4570198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:46086/0 (socket says 172.21.15.28:46086) 2024-04-25T01:34:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.583+0000 7f456f7fe640 1 -- 172.21.15.28:0/3034384594 learned_addr learned my addr 172.21.15.28:0/3034384594 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.583+0000 7f456ffff640 1 --2- 172.21.15.28:0/3034384594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f457010bd00 0x7f457019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.583+0000 7f456effd640 1 --2- 172.21.15.28:0/3034384594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45701032e0 0x7f4570198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.584+0000 7f456f7fe640 1 -- 172.21.15.28:0/3034384594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45701032e0 msgr2=0x7f4570198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.584+0000 7f456f7fe640 1 --2- 172.21.15.28:0/3034384594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45701032e0 0x7f4570198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.584+0000 7f456f7fe640 1 -- 172.21.15.28:0/3034384594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f457010bd00 msgr2=0x7f457019c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.584+0000 7f456f7fe640 1 --2- 172.21.15.28:0/3034384594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f457010bd00 0x7f457019c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.584+0000 7f456f7fe640 1 -- 172.21.15.28:0/3034384594 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f457019cfa0 con 0x7f45701029c0 2024-04-25T01:34:10.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.584+0000 7f456f7fe640 1 --2- 172.21.15.28:0/3034384594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45701029c0 0x7f4570198000 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f456000d770 tx=0x7f456000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.584+0000 7f456cff9640 1 -- 172.21.15.28:0/3034384594 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4560017070 con 0x7f45701029c0 2024-04-25T01:34:10.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.584+0000 7f4575bd5640 1 -- 172.21.15.28:0/3034384594 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f457019d290 con 0x7f45701029c0 2024-04-25T01:34:10.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.584+0000 7f456cff9640 1 -- 172.21.15.28:0/3034384594 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4560005500 con 0x7f45701029c0 2024-04-25T01:34:10.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.584+0000 7f456cff9640 1 -- 172.21.15.28:0/3034384594 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4560004e00 con 0x7f45701029c0 2024-04-25T01:34:10.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.584+0000 7f4575bd5640 1 -- 172.21.15.28:0/3034384594 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45701a4b20 con 0x7f45701029c0 2024-04-25T01:34:10.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.587+0000 7f45789b6740 1 Processor -- start 2024-04-25T01:34:10.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.587+0000 7f45789b6740 1 -- start start 2024-04-25T01:34:10.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.587+0000 7f456cff9640 1 -- 172.21.15.28:0/3034384594 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f4560010460 con 0x7f45701029c0 2024-04-25T01:34:10.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.588+0000 7f45789b6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01d580 0x55bacd01d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.588+0000 7f45789b6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd01dea0 0x55bacd028030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.588+0000 7f45789b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd028570 0x55bacd02a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.588+0000 7f45789b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55baccf82670 con 0x55bacd028570 2024-04-25T01:34:10.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.588+0000 7f45789b6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55baccf2a970 con 0x55bacd01dea0 2024-04-25T01:34:10.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.588+0000 7f45789b6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55baccf7e630 con 0x55bacd01d580 2024-04-25T01:34:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.588+0000 7f456effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd028570 0x55bacd02a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.588+0000 7f456effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd028570 0x55bacd02a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59012/0 (socket says 172.21.15.28:59012) 2024-04-25T01:34:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.588+0000 7f456effd640 1 -- 172.21.15.28:0/3482378088 learned_addr learned my addr 172.21.15.28:0/3482378088 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.588+0000 7f456ffff640 1 --2- 172.21.15.28:0/3482378088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01d580 0x55bacd01d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.588+0000 7f456f7fe640 1 --2- 172.21.15.28:0/3482378088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd01dea0 0x55bacd028030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.588+0000 7f456effd640 1 -- 172.21.15.28:0/3482378088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01d580 msgr2=0x55bacd01d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.588+0000 7f456effd640 1 --2- 172.21.15.28:0/3482378088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01d580 0x55bacd01d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.588+0000 7f456effd640 1 -- 172.21.15.28:0/3482378088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd01dea0 msgr2=0x55bacd028030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.588+0000 7f456effd640 1 --2- 172.21.15.28:0/3482378088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd01dea0 0x55bacd028030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.588+0000 7f456effd640 1 -- 172.21.15.28:0/3482378088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55baccf4e400 con 0x55bacd028570 2024-04-25T01:34:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.588+0000 7f456f7fe640 1 --2- 172.21.15.28:0/3482378088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd01dea0 0x55bacd028030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.588+0000 7f456cff9640 1 --2- 172.21.15.28:0/3034384594 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f45580a2e20 0x7f45580a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.589+0000 7f456effd640 1 --2- 172.21.15.28:0/3482378088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd028570 0x55bacd02a960 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4564009c10 tx=0x7f4564011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25b7b8ba6aa37f66 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.589+0000 7f456cff9640 1 -- 172.21.15.28:0/3034384594 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f45600cf430 con 0x7f45701029c0 2024-04-25T01:34:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.589+0000 7f456ffff640 1 --2- 172.21.15.28:0/3034384594 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f45580a2e20 0x7f45580a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.589+0000 7f4555ffb640 1 -- 172.21.15.28:0/3482378088 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4564004230 con 0x55bacd028570 2024-04-25T01:34:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.589+0000 7f4555ffb640 1 -- 172.21.15.28:0/3482378088 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45640043d0 con 0x55bacd028570 2024-04-25T01:34:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.589+0000 7f4555ffb640 1 -- 172.21.15.28:0/3482378088 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4564004e60 con 0x55bacd028570 2024-04-25T01:34:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.590+0000 7f45789b6740 1 -- 172.21.15.28:0/3482378088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd028570 msgr2=0x55bacd02a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.590+0000 7f456ffff640 1 --2- 172.21.15.28:0/3034384594 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f45580a2e20 0x7f45580a52e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4568005e30 tx=0x7f456803a040 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.590+0000 7f45789b6740 1 --2- 172.21.15.28:0/3482378088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd028570 0x55bacd02a960 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4564009c10 tx=0x7f4564011810 comp rx=0 tx=0).stop 2024-04-25T01:34:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.590+0000 7f45789b6740 1 -- 172.21.15.28:0/3482378088 shutdown_connections 2024-04-25T01:34:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.590+0000 7f45789b6740 1 --2- 172.21.15.28:0/3482378088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd028570 0x55bacd02a960 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.590+0000 7f45789b6740 1 --2- 172.21.15.28:0/3482378088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd01dea0 0x55bacd028030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.590+0000 7f45789b6740 1 --2- 172.21.15.28:0/3482378088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01d580 0x55bacd01d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.590+0000 7f45789b6740 1 -- 172.21.15.28:0/3482378088 >> 172.21.15.28:0/3482378088 conn(0x55bacd0198a0 msgr2=0x55bacd019c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.590+0000 7f45789b6740 1 -- 172.21.15.28:0/3482378088 shutdown_connections 2024-04-25T01:34:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.590+0000 7f45789b6740 1 -- 172.21.15.28:0/3482378088 wait complete. 2024-04-25T01:34:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.591+0000 7f45789b6740 1 -- start start 2024-04-25T01:34:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.591+0000 7f45789b6740 1 Processor -- start 2024-04-25T01:34:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.591+0000 7f45789b6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd01d580 0x55bacd027ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.591+0000 7f45789b6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01dea0 0x55bacd0251c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.591+0000 7f45789b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd028570 0x55bacd025720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.591+0000 7f45789b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bacd02b360 con 0x55bacd028570 2024-04-25T01:34:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.591+0000 7f45789b6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55baccf3e050 con 0x55bacd01d580 2024-04-25T01:34:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.591+0000 7f45789b6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55bacd02b4e0 con 0x55bacd01dea0 2024-04-25T01:34:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.591+0000 7f45789b6740 10 client.0 _ll_register_callbacks cb 0x55baccc95be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.592+0000 7f456ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd028570 0x55bacd025720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.592+0000 7f456f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd01d580 0x55bacd027ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.592+0000 7f456ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd028570 0x55bacd025720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59028/0 (socket says 172.21.15.28:59028) 2024-04-25T01:34:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.592+0000 7f456effd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01dea0 0x55bacd0251c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.592+0000 7f456ffff640 1 -- 172.21.15.28:0/1357447084 learned_addr learned my addr 172.21.15.28:0/1357447084 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.592+0000 7f456f7fe640 1 -- 172.21.15.28:0/1357447084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01dea0 msgr2=0x55bacd0251c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.592+0000 7f456f7fe640 1 --2- 172.21.15.28:0/1357447084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01dea0 0x55bacd0251c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.592+0000 7f456f7fe640 1 -- 172.21.15.28:0/1357447084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd028570 msgr2=0x55bacd025720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.592+0000 7f456f7fe640 1 --2- 172.21.15.28:0/1357447084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd028570 0x55bacd025720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.592+0000 7f456f7fe640 1 -- 172.21.15.28:0/1357447084 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55baccf0fdc0 con 0x55bacd01d580 2024-04-25T01:34:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.592+0000 7f456f7fe640 1 --2- 172.21.15.28:0/1357447084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd01d580 0x55bacd027ed0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f45600cc6c0 tx=0x7f4560097710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52346603135a60c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.592+0000 7f454effd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-25T01:34:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.592+0000 7f454effd640 1 -- 172.21.15.28:0/1357447084 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4560020050 con 0x55bacd01d580 2024-04-25T01:34:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.592+0000 7f45789b6740 1 -- 172.21.15.28:0/1357447084 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45640043d0 con 0x55bacd01d580 2024-04-25T01:34:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.592+0000 7f454effd640 1 -- 172.21.15.28:0/1357447084 <== mon.2 v2:172.21.15.32:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f4560097ae0 con 0x55bacd01d580 2024-04-25T01:34:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.593+0000 7f454effd640 1 -- 172.21.15.28:0/1357447084 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f456009add0 con 0x55bacd01d580 2024-04-25T01:34:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.593+0000 7f45789b6740 10 client.25598 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.593+0000 7f45789b6740 1 -- 172.21.15.28:0/1357447084 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55baccc963f0 con 0x55bacd01d580 2024-04-25T01:34:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.593+0000 7f454effd640 1 -- 172.21.15.28:0/1357447084 <== mon.2 v2:172.21.15.32:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45640043d0 con 0x55bacd01d580 2024-04-25T01:34:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.593+0000 7f454effd640 1 -- 172.21.15.28:0/1357447084 <== mon.2 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f45600cc9a0 con 0x55bacd01d580 2024-04-25T01:34:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.593+0000 7f45789b6740 10 client.25598 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.593+0000 7f45789b6740 10 client.25598 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.593+0000 7f45789b6740 20 client.25598 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.593+0000 7f45789b6740 1 --2- 172.21.15.28:0/1357447084 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bacd038a40 0x55bacd03aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.593+0000 7f45789b6740 4 client.25598 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.593+0000 7f45789b6740 1 -- 172.21.15.28:0/1357447084 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55baccf8fbb0 con 0x55bacd038a40 2024-04-25T01:34:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.593+0000 7f456effd640 1 --2- 172.21.15.28:0/1357447084 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bacd038a40 0x55bacd03aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.594+0000 7f454effd640 10 client.25598 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.594+0000 7f456effd640 1 --2- 172.21.15.28:0/1357447084 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bacd038a40 0x55bacd03aed0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b04a0cc2d9f7dfb server_cookie=7d509ff07b2815ce in_seq=0 out_seq=0 2024-04-25T01:34:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.594+0000 7f454effd640 1 -- 172.21.15.28:0/1357447084 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55baccf8fbb0 con 0x55bacd038a40 2024-04-25T01:34:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.594+0000 7f454effd640 10 client.25598 handle_command_reply: tid=0 2024-04-25T01:34:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.595+0000 7f45789b6740 1 client.25598 shutdown 2024-04-25T01:34:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.595+0000 7f45789b6740 1 -- 172.21.15.28:0/1357447084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd01d580 msgr2=0x55bacd027ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.595+0000 7f45789b6740 1 --2- 172.21.15.28:0/1357447084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd01d580 0x55bacd027ed0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f45600cc6c0 tx=0x7f4560097710 comp rx=0 tx=0).stop 2024-04-25T01:34:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.595+0000 7f45789b6740 1 -- 172.21.15.28:0/1357447084 shutdown_connections 2024-04-25T01:34:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.595+0000 7f45789b6740 1 --2- 172.21.15.28:0/1357447084 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bacd038a40 0x55bacd03aed0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.595+0000 7f45789b6740 1 --2- 172.21.15.28:0/1357447084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd028570 0x55bacd025720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.595+0000 7f45789b6740 1 --2- 172.21.15.28:0/1357447084 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01dea0 0x55bacd0251c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.595+0000 7f45789b6740 1 --2- 172.21.15.28:0/1357447084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd01d580 0x55bacd027ed0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.595+0000 7f454effd640 0 client.25598 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.595+0000 7f45789b6740 1 -- 172.21.15.28:0/1357447084 >> 172.21.15.28:0/1357447084 conn(0x55bacd0198a0 msgr2=0x55bacd018b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.595+0000 7f45789b6740 1 -- 172.21.15.28:0/1357447084 shutdown_connections 2024-04-25T01:34:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.595+0000 7f45789b6740 1 -- 172.21.15.28:0/1357447084 wait complete. 2024-04-25T01:34:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.595+0000 7f45789b6740 20 client.25598 trim_cache size 0 max 16384 2024-04-25T01:34:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.608+0000 7f45789b6740 1 Processor -- start 2024-04-25T01:34:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.608+0000 7f45789b6740 1 -- start start 2024-04-25T01:34:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.609+0000 7f45789b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd01d580 0x55bacd034220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.609+0000 7f45789b6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01dea0 0x55bacd027a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.609+0000 7f45789b6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd028570 0x55bacd0407b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.609+0000 7f45789b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45701b0c90 con 0x55bacd01d580 2024-04-25T01:34:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.609+0000 7f45789b6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f45701b27d0 con 0x55bacd028570 2024-04-25T01:34:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.609+0000 7f45789b6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f45701b1360 con 0x55bacd01dea0 2024-04-25T01:34:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.609+0000 7f456f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd01d580 0x55bacd034220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.609+0000 7f456f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd01d580 0x55bacd034220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59044/0 (socket says 172.21.15.28:59044) 2024-04-25T01:34:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.609+0000 7f456f7fe640 1 -- 172.21.15.28:0/3104529388 learned_addr learned my addr 172.21.15.28:0/3104529388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.609+0000 7f456effd640 1 --2- 172.21.15.28:0/3104529388 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01dea0 0x55bacd027a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.609+0000 7f456ffff640 1 --2- 172.21.15.28:0/3104529388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd028570 0x55bacd0407b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.609+0000 7f456f7fe640 1 -- 172.21.15.28:0/3104529388 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01dea0 msgr2=0x55bacd027a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.609+0000 7f456f7fe640 1 --2- 172.21.15.28:0/3104529388 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01dea0 0x55bacd027a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.609+0000 7f456f7fe640 1 -- 172.21.15.28:0/3104529388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd028570 msgr2=0x55bacd0407b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.609+0000 7f456f7fe640 1 --2- 172.21.15.28:0/3104529388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd028570 0x55bacd0407b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.609+0000 7f456f7fe640 1 -- 172.21.15.28:0/3104529388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bacd01dcd0 con 0x55bacd01d580 2024-04-25T01:34:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.609+0000 7f456ffff640 1 --2- 172.21.15.28:0/3104529388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd028570 0x55bacd0407b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.609+0000 7f456f7fe640 1 --2- 172.21.15.28:0/3104529388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd01d580 0x55bacd034220 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4560099720 tx=0x7f45600ccf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdab2aa7ea536b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.610+0000 7f45557fa640 1 -- 172.21.15.28:0/3104529388 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45600b4070 con 0x55bacd01d580 2024-04-25T01:34:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.610+0000 7f45557fa640 1 -- 172.21.15.28:0/3104529388 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45600046c0 con 0x55bacd01d580 2024-04-25T01:34:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.610+0000 7f45557fa640 1 -- 172.21.15.28:0/3104529388 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4560020070 con 0x55bacd01d580 2024-04-25T01:34:10.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.611+0000 7f45789b6740 1 -- 172.21.15.28:0/3104529388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd01d580 msgr2=0x55bacd034220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.611+0000 7f45789b6740 1 --2- 172.21.15.28:0/3104529388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd01d580 0x55bacd034220 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4560099720 tx=0x7f45600ccf90 comp rx=0 tx=0).stop 2024-04-25T01:34:10.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.611+0000 7f45789b6740 1 -- 172.21.15.28:0/3104529388 shutdown_connections 2024-04-25T01:34:10.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.611+0000 7f45789b6740 1 --2- 172.21.15.28:0/3104529388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd028570 0x55bacd0407b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.611+0000 7f45789b6740 1 --2- 172.21.15.28:0/3104529388 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01dea0 0x55bacd027a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.611+0000 7f45789b6740 1 --2- 172.21.15.28:0/3104529388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd01d580 0x55bacd034220 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.611+0000 7f45789b6740 1 -- 172.21.15.28:0/3104529388 >> 172.21.15.28:0/3104529388 conn(0x55bacd0198a0 msgr2=0x55bacd03aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.611+0000 7f45789b6740 1 -- 172.21.15.28:0/3104529388 shutdown_connections 2024-04-25T01:34:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.611+0000 7f45789b6740 1 -- 172.21.15.28:0/3104529388 wait complete. 2024-04-25T01:34:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.611+0000 7f45789b6740 1 -- start start 2024-04-25T01:34:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.612+0000 7f45789b6740 1 Processor -- start 2024-04-25T01:34:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.612+0000 7f45789b6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01d580 0x55bacd097320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.612+0000 7f45789b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd01dea0 0x55bacd097860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.612+0000 7f45789b6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd028570 0x55bacd02bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.613+0000 7f45789b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45701b2100 con 0x55bacd01dea0 2024-04-25T01:34:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.613+0000 7f45789b6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f45701b1a30 con 0x55bacd028570 2024-04-25T01:34:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.613+0000 7f456effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd01dea0 0x55bacd097860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.613+0000 7f456f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01d580 0x55bacd097320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.613+0000 7f45789b6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f45701aec70 con 0x55bacd01d580 2024-04-25T01:34:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.613+0000 7f456f7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01d580 0x55bacd097320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.28:46884/0 (socket says 172.21.15.28:46884) 2024-04-25T01:34:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.613+0000 7f456effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd01dea0 0x55bacd097860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59046/0 (socket says 172.21.15.28:59046) 2024-04-25T01:34:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.613+0000 7f456f7fe640 1 -- 172.21.15.28:0/2468767579 learned_addr learned my addr 172.21.15.28:0/2468767579 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.613+0000 7f45789b6740 10 client.0 _ll_register_callbacks cb 0x7f45600970b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.613+0000 7f456effd640 1 -- 172.21.15.28:0/2468767579 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01d580 msgr2=0x55bacd097320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.613+0000 7f456effd640 1 --2- 172.21.15.28:0/2468767579 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01d580 0x55bacd097320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.613+0000 7f456effd640 1 -- 172.21.15.28:0/2468767579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd028570 msgr2=0x55bacd02bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.613+0000 7f456effd640 1 --2- 172.21.15.28:0/2468767579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd028570 0x55bacd02bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.613+0000 7f456effd640 1 -- 172.21.15.28:0/2468767579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55baccd607d0 con 0x55bacd01dea0 2024-04-25T01:34:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.613+0000 7f456effd640 1 --2- 172.21.15.28:0/2468767579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd01dea0 0x55bacd097860 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4564000c00 tx=0x7f4564005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a333bc42a64d2751 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.613+0000 7f454effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.613+0000 7f454effd640 1 -- 172.21.15.28:0/2468767579 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456401e040 con 0x55bacd01dea0 2024-04-25T01:34:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.613+0000 7f454effd640 1 -- 172.21.15.28:0/2468767579 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f4564004460 con 0x55bacd01dea0 2024-04-25T01:34:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.613+0000 7f45789b6740 1 -- 172.21.15.28:0/2468767579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55baccf89f00 con 0x55bacd01dea0 2024-04-25T01:34:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.614+0000 7f454effd640 1 -- 172.21.15.28:0/2468767579 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4564004660 con 0x55bacd01dea0 2024-04-25T01:34:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.614+0000 7f45789b6740 10 client.16083 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.614+0000 7f45789b6740 1 -- 172.21.15.28:0/2468767579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bacd09e900 con 0x55bacd01dea0 2024-04-25T01:34:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.614+0000 7f454effd640 1 -- 172.21.15.28:0/2468767579 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55baccf89f00 con 0x55bacd01dea0 2024-04-25T01:34:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.614+0000 7f454effd640 1 -- 172.21.15.28:0/2468767579 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4564024040 con 0x55bacd01dea0 2024-04-25T01:34:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.615+0000 7f45789b6740 10 client.16083 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.615+0000 7f45789b6740 10 client.16083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.615+0000 7f45789b6740 20 client.16083 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.615+0000 7f45789b6740 1 --2- 172.21.15.28:0/2468767579 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bacd09ee60 0x55bacd0a1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.615+0000 7f45789b6740 4 client.16083 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:34:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.615+0000 7f45789b6740 1 -- 172.21.15.28:0/2468767579 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55baccf07820 con 0x55bacd09ee60 2024-04-25T01:34:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.615+0000 7f456f7fe640 1 --2- 172.21.15.28:0/2468767579 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bacd09ee60 0x55bacd0a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:10.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.616+0000 7f454effd640 10 client.16083 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:10.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.616+0000 7f456f7fe640 1 --2- 172.21.15.28:0/2468767579 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bacd09ee60 0x55bacd0a1280 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67be406c57f85ae server_cookie=fa342368013446d8 in_seq=0 out_seq=0 2024-04-25T01:34:10.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.616+0000 7f454effd640 1 -- 172.21.15.28:0/2468767579 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55baccf07820 con 0x55bacd09ee60 2024-04-25T01:34:10.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.616+0000 7f454effd640 10 client.16083 handle_command_reply: tid=0 2024-04-25T01:34:10.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.616+0000 7f45789b6740 1 client.16083 shutdown 2024-04-25T01:34:10.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.616+0000 7f45789b6740 1 -- 172.21.15.28:0/2468767579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd01dea0 msgr2=0x55bacd097860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.616+0000 7f45789b6740 1 --2- 172.21.15.28:0/2468767579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd01dea0 0x55bacd097860 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4564000c00 tx=0x7f4564005d80 comp rx=0 tx=0).stop 2024-04-25T01:34:10.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.616+0000 7f45789b6740 1 -- 172.21.15.28:0/2468767579 shutdown_connections 2024-04-25T01:34:10.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.616+0000 7f45789b6740 1 --2- 172.21.15.28:0/2468767579 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bacd09ee60 0x55bacd0a1280 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.616+0000 7f45789b6740 1 --2- 172.21.15.28:0/2468767579 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bacd028570 0x55bacd02bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.616+0000 7f45789b6740 1 --2- 172.21.15.28:0/2468767579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bacd01dea0 0x55bacd097860 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.616+0000 7f45789b6740 1 --2- 172.21.15.28:0/2468767579 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bacd01d580 0x55bacd097320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.616+0000 7f454effd640 0 client.16083 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:10.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.616+0000 7f45789b6740 1 -- 172.21.15.28:0/2468767579 >> 172.21.15.28:0/2468767579 conn(0x55bacd0198a0 msgr2=0x55bacd0182b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.617+0000 7f45789b6740 1 -- 172.21.15.28:0/2468767579 shutdown_connections 2024-04-25T01:34:10.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.617+0000 7f45789b6740 1 -- 172.21.15.28:0/2468767579 wait complete. 2024-04-25T01:34:10.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.617+0000 7f45789b6740 20 client.16083 trim_cache size 0 max 16384 2024-04-25T01:34:10.598 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:34:10.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.617+0000 7f4575bd5640 1 -- 172.21.15.28:0/3034384594 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f45580a2e20 msgr2=0x7f45580a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.617+0000 7f4575bd5640 1 --2- 172.21.15.28:0/3034384594 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f45580a2e20 0x7f45580a52e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4568005e30 tx=0x7f456803a040 comp rx=0 tx=0).stop 2024-04-25T01:34:10.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.617+0000 7f4575bd5640 1 -- 172.21.15.28:0/3034384594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45701029c0 msgr2=0x7f4570198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:10.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.617+0000 7f4575bd5640 1 --2- 172.21.15.28:0/3034384594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45701029c0 0x7f4570198000 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f456000d770 tx=0x7f456000dc30 comp rx=0 tx=0).stop 2024-04-25T01:34:10.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.617+0000 7f4575bd5640 1 -- 172.21.15.28:0/3034384594 shutdown_connections 2024-04-25T01:34:10.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.617+0000 7f4575bd5640 1 --2- 172.21.15.28:0/3034384594 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f45580a2e20 0x7f45580a52e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.617+0000 7f4575bd5640 1 --2- 172.21.15.28:0/3034384594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f457010bd00 0x7f457019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.617+0000 7f4575bd5640 1 --2- 172.21.15.28:0/3034384594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45701032e0 0x7f4570198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.617+0000 7f4575bd5640 1 --2- 172.21.15.28:0/3034384594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45701029c0 0x7f4570198000 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:10.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.617+0000 7f4575bd5640 1 -- 172.21.15.28:0/3034384594 >> 172.21.15.28:0/3034384594 conn(0x7f4570077740 msgr2=0x7f45700fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:10.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.618+0000 7f4575bd5640 1 -- 172.21.15.28:0/3034384594 shutdown_connections 2024-04-25T01:34:10.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:10.618+0000 7f4575bd5640 1 -- 172.21.15.28:0/3034384594 wait complete. 2024-04-25T01:34:10.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:10 smithi130 ceph-mon[32702]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-04-25T01:34:10.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:10 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/917213984' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:34:11.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:11 smithi028 ceph-mon[29390]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-25T01:34:11.607 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:34:11.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:34:11.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:34:11.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:11 smithi130 ceph-mon[32702]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-25T01:34:11.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.742+0000 7fe9920a5640 1 -- 172.21.15.28:0/1308077277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98c1032e0 msgr2=0x7fe98c10b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:11.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.742+0000 7fe9920a5640 1 --2- 172.21.15.28:0/1308077277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98c1032e0 0x7fe98c10b7c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fe974009900 tx=0x7fe97402a510 comp rx=0 tx=0).stop 2024-04-25T01:34:11.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.742+0000 7fe9920a5640 1 -- 172.21.15.28:0/1308077277 shutdown_connections 2024-04-25T01:34:11.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.742+0000 7fe9920a5640 1 --2- 172.21.15.28:0/1308077277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe98c10bd00 0x7fe98c10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.742+0000 7fe9920a5640 1 --2- 172.21.15.28:0/1308077277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98c1032e0 0x7fe98c10b7c0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.742+0000 7fe9920a5640 1 --2- 172.21.15.28:0/1308077277 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe98c1029c0 0x7fe98c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.742+0000 7fe9920a5640 1 -- 172.21.15.28:0/1308077277 >> 172.21.15.28:0/1308077277 conn(0x7fe98c077740 msgr2=0x7fe98c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:11.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.743+0000 7fe9920a5640 1 -- 172.21.15.28:0/1308077277 shutdown_connections 2024-04-25T01:34:11.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.743+0000 7fe9920a5640 1 -- 172.21.15.28:0/1308077277 wait complete. 2024-04-25T01:34:11.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.743+0000 7fe9920a5640 1 Processor -- start 2024-04-25T01:34:11.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.743+0000 7fe9920a5640 1 -- start start 2024-04-25T01:34:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe9920a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98c1029c0 0x7fe98c100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe9920a5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe98c1032e0 0x7fe98c100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe9920a5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe98c10bd00 0x7fe98c101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe9920a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe98c111b50 con 0x7fe98c1029c0 2024-04-25T01:34:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe9920a5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe98c1119d0 con 0x7fe98c1032e0 2024-04-25T01:34:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe9920a5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe98c111cd0 con 0x7fe98c10bd00 2024-04-25T01:34:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe9837fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98c1029c0 0x7fe98c100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe9837fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98c1029c0 0x7fe98c100610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59068/0 (socket says 172.21.15.28:59068) 2024-04-25T01:34:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe9837fe640 1 -- 172.21.15.28:0/1691215333 learned_addr learned my addr 172.21.15.28:0/1691215333 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe9837fe640 1 -- 172.21.15.28:0/1691215333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe98c10bd00 msgr2=0x7fe98c101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe98bfff640 1 --2- 172.21.15.28:0/1691215333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe98c10bd00 0x7fe98c101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe98b7fe640 1 --2- 172.21.15.28:0/1691215333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe98c1032e0 0x7fe98c100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe9837fe640 1 --2- 172.21.15.28:0/1691215333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe98c10bd00 0x7fe98c101090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe9837fe640 1 -- 172.21.15.28:0/1691215333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe98c1032e0 msgr2=0x7fe98c100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe9837fe640 1 --2- 172.21.15.28:0/1691215333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe98c1032e0 0x7fe98c100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe9837fe640 1 -- 172.21.15.28:0/1691215333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe98c1a6550 con 0x7fe98c1029c0 2024-04-25T01:34:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe98b7fe640 1 --2- 172.21.15.28:0/1691215333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe98c1032e0 0x7fe98c100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe9837fe640 1 --2- 172.21.15.28:0/1691215333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98c1029c0 0x7fe98c100610 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00d770 tx=0x7fe97c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe9897fa640 1 -- 172.21.15.28:0/1691215333 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c017070 con 0x7fe98c1029c0 2024-04-25T01:34:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe9920a5640 1 -- 172.21.15.28:0/1691215333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe98c1a6840 con 0x7fe98c1029c0 2024-04-25T01:34:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.744+0000 7fe9897fa640 1 -- 172.21.15.28:0/1691215333 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe97c005500 con 0x7fe98c1029c0 2024-04-25T01:34:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.745+0000 7fe9897fa640 1 -- 172.21.15.28:0/1691215333 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c004e00 con 0x7fe98c1029c0 2024-04-25T01:34:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.745+0000 7fe9920a5640 1 -- 172.21.15.28:0/1691215333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe98c1a6d90 con 0x7fe98c1029c0 2024-04-25T01:34:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.747+0000 7fe994db5740 1 Processor -- start 2024-04-25T01:34:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.747+0000 7fe994db5740 1 -- start start 2024-04-25T01:34:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.747+0000 7fe994db5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56301da70980 0x56301da6ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.747+0000 7fe994db5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56301da6f290 0x56301da6f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.747+0000 7fe994db5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da7a010 0x56301da7c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.747+0000 7fe994db5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56301d9d32c0 con 0x56301da7a010 2024-04-25T01:34:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.747+0000 7fe994db5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56301d97b910 con 0x56301da70980 2024-04-25T01:34:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.747+0000 7fe994db5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56301d9cf280 con 0x56301da6f290 2024-04-25T01:34:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.748+0000 7fe98b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da7a010 0x56301da7c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.748+0000 7fe98b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da7a010 0x56301da7c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59080/0 (socket says 172.21.15.28:59080) 2024-04-25T01:34:11.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.748+0000 7fe98b7fe640 1 -- 172.21.15.28:0/3183739547 learned_addr learned my addr 172.21.15.28:0/3183739547 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:11.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.748+0000 7fe9897fa640 1 -- 172.21.15.28:0/1691215333 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7fe97c004fa0 con 0x7fe98c1029c0 2024-04-25T01:34:11.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.748+0000 7fe98bfff640 1 --2- 172.21.15.28:0/3183739547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56301da70980 0x56301da6ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.748+0000 7fe98bfff640 1 -- 172.21.15.28:0/3183739547 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56301da6f290 msgr2=0x56301da6f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:11.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.748+0000 7fe98bfff640 1 --2- 172.21.15.28:0/3183739547 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56301da6f290 0x56301da6f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.748+0000 7fe98bfff640 1 -- 172.21.15.28:0/3183739547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da7a010 msgr2=0x56301da7c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:11.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.748+0000 7fe98bfff640 1 --2- 172.21.15.28:0/3183739547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da7a010 0x56301da7c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.748+0000 7fe98bfff640 1 -- 172.21.15.28:0/3183739547 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56301d98cfb0 con 0x56301da70980 2024-04-25T01:34:11.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.748+0000 7fe98bfff640 1 --2- 172.21.15.28:0/3183739547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56301da70980 0x56301da6ecc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe984009d20 tx=0x7fe98400e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=298b4e6a4b65968d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:11.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.748+0000 7fe9897fa640 1 --2- 172.21.15.28:0/1691215333 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe9780a2e20 0x7fe9780a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.749+0000 7fe981ffb640 1 -- 172.21.15.28:0/3183739547 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe984016070 con 0x56301da70980 2024-04-25T01:34:11.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.749+0000 7fe981ffb640 1 -- 172.21.15.28:0/3183739547 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe984005020 con 0x56301da70980 2024-04-25T01:34:11.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.749+0000 7fe981ffb640 1 -- 172.21.15.28:0/3183739547 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9840116b0 con 0x56301da70980 2024-04-25T01:34:11.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.749+0000 7fe9897fa640 1 -- 172.21.15.28:0/1691215333 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fe97c0cf590 con 0x7fe98c1029c0 2024-04-25T01:34:11.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.749+0000 7fe9837fe640 1 --2- 172.21.15.28:0/1691215333 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe9780a2e20 0x7fe9780a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.750+0000 7fe994db5740 1 -- 172.21.15.28:0/3183739547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56301da70980 msgr2=0x56301da6ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.750+0000 7fe9837fe640 1 --2- 172.21.15.28:0/1691215333 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe9780a2e20 0x7fe9780a52e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe97c0cc880 tx=0x7fe97c00df00 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.750+0000 7fe994db5740 1 --2- 172.21.15.28:0/3183739547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56301da70980 0x56301da6ecc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe984009d20 tx=0x7fe98400e6c0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.750+0000 7fe994db5740 1 -- 172.21.15.28:0/3183739547 shutdown_connections 2024-04-25T01:34:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.750+0000 7fe994db5740 1 --2- 172.21.15.28:0/3183739547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da7a010 0x56301da7c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.750+0000 7fe994db5740 1 --2- 172.21.15.28:0/3183739547 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56301da6f290 0x56301da6f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.750+0000 7fe994db5740 1 --2- 172.21.15.28:0/3183739547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56301da70980 0x56301da6ecc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.750+0000 7fe994db5740 1 -- 172.21.15.28:0/3183739547 >> 172.21.15.28:0/3183739547 conn(0x56301da37950 msgr2=0x56301da6b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.750+0000 7fe994db5740 1 -- 172.21.15.28:0/3183739547 shutdown_connections 2024-04-25T01:34:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.750+0000 7fe994db5740 1 -- 172.21.15.28:0/3183739547 wait complete. 2024-04-25T01:34:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.750+0000 7fe994db5740 1 -- start start 2024-04-25T01:34:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.750+0000 7fe994db5740 1 Processor -- start 2024-04-25T01:34:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.751+0000 7fe994db5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56301da6f290 0x56301da76fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.751+0000 7fe994db5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da70980 0x56301da774e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.751+0000 7fe994db5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56301da7a010 0x56301da77a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.751+0000 7fe994db5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56301da7cea0 con 0x56301da70980 2024-04-25T01:34:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.751+0000 7fe994db5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56301d98ee90 con 0x56301da7a010 2024-04-25T01:34:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.751+0000 7fe98b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da70980 0x56301da774e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.751+0000 7fe98b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da70980 0x56301da774e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59086/0 (socket says 172.21.15.28:59086) 2024-04-25T01:34:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.751+0000 7fe98b7fe640 1 -- 172.21.15.28:0/1587291042 learned_addr learned my addr 172.21.15.28:0/1587291042 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.751+0000 7fe994db5740 1 -- 172.21.15.28:0/1587291042 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56301da7d020 con 0x56301da6f290 2024-04-25T01:34:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.751+0000 7fe98bfff640 1 --2- 172.21.15.28:0/1587291042 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56301da6f290 0x56301da76fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.751+0000 7fe994db5740 10 client.0 _ll_register_callbacks cb 0x56301d9ca970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.751+0000 7fe98bfff640 1 --2- 172.21.15.28:0/1587291042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56301da7a010 0x56301da77a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.751+0000 7fe98b7fe640 1 -- 172.21.15.28:0/1587291042 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56301da6f290 msgr2=0x56301da76fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.751+0000 7fe98b7fe640 1 --2- 172.21.15.28:0/1587291042 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56301da6f290 0x56301da76fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.751+0000 7fe98b7fe640 1 -- 172.21.15.28:0/1587291042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56301da7a010 msgr2=0x56301da77a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.751+0000 7fe98b7fe640 1 --2- 172.21.15.28:0/1587291042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56301da7a010 0x56301da77a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.751+0000 7fe98b7fe640 1 -- 172.21.15.28:0/1587291042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56301d7af7e0 con 0x56301da70980 2024-04-25T01:34:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.752+0000 7fe98b7fe640 1 --2- 172.21.15.28:0/1587291042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da70980 0x56301da774e0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe97402aa30 tx=0x7fe9740043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fabdd6591f9395b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.752+0000 7fe96affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.752+0000 7fe96affd640 1 -- 172.21.15.28:0/1587291042 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe974035040 con 0x56301da70980 2024-04-25T01:34:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.752+0000 7fe96affd640 1 -- 172.21.15.28:0/1587291042 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fe974005450 con 0x56301da70980 2024-04-25T01:34:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.752+0000 7fe994db5740 1 -- 172.21.15.28:0/1587291042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe984005020 con 0x56301da70980 2024-04-25T01:34:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.752+0000 7fe96affd640 1 -- 172.21.15.28:0/1587291042 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe97404d050 con 0x56301da70980 2024-04-25T01:34:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.752+0000 7fe994db5740 10 client.16095 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.752+0000 7fe994db5740 1 -- 172.21.15.28:0/1587291042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56301d80ff70 con 0x56301da70980 2024-04-25T01:34:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.752+0000 7fe96affd640 1 -- 172.21.15.28:0/1587291042 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe984005020 con 0x56301da70980 2024-04-25T01:34:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.753+0000 7fe96affd640 1 -- 172.21.15.28:0/1587291042 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe974003820 con 0x56301da70980 2024-04-25T01:34:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.753+0000 7fe994db5740 10 client.16095 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.753+0000 7fe994db5740 10 client.16095 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.753+0000 7fe994db5740 20 client.16095 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.753+0000 7fe994db5740 1 --2- 172.21.15.28:0/1587291042 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56301da8a1c0 0x56301da8c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.753+0000 7fe994db5740 4 client.16095 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.753+0000 7fe994db5740 1 -- 172.21.15.28:0/1587291042 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56301d9a75a0 con 0x56301da8a1c0 2024-04-25T01:34:11.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.753+0000 7fe98bfff640 1 --2- 172.21.15.28:0/1587291042 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56301da8a1c0 0x56301da8c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.736 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:11 smithi032 ceph-mon[32461]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-25T01:34:11.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.754+0000 7fe96affd640 10 client.16095 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:11.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.754+0000 7fe98bfff640 1 --2- 172.21.15.28:0/1587291042 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56301da8a1c0 0x56301da8c5e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70a4b17b566f3bfd server_cookie=fe543cc61f76fbe5 in_seq=0 out_seq=0 2024-04-25T01:34:11.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.754+0000 7fe96affd640 1 -- 172.21.15.28:0/1587291042 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56301d9a75a0 con 0x56301da8a1c0 2024-04-25T01:34:11.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.754+0000 7fe96affd640 10 client.16095 handle_command_reply: tid=0 2024-04-25T01:34:11.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.754+0000 7fe994db5740 1 client.16095 shutdown 2024-04-25T01:34:11.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.754+0000 7fe994db5740 1 -- 172.21.15.28:0/1587291042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da70980 msgr2=0x56301da774e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:11.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.754+0000 7fe994db5740 1 --2- 172.21.15.28:0/1587291042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da70980 0x56301da774e0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe97402aa30 tx=0x7fe9740043b0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.755+0000 7fe994db5740 1 -- 172.21.15.28:0/1587291042 shutdown_connections 2024-04-25T01:34:11.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.755+0000 7fe994db5740 1 --2- 172.21.15.28:0/1587291042 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56301da8a1c0 0x56301da8c5e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.755+0000 7fe96affd640 0 client.16095 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:11.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.755+0000 7fe994db5740 1 --2- 172.21.15.28:0/1587291042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56301da7a010 0x56301da77a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.755+0000 7fe994db5740 1 --2- 172.21.15.28:0/1587291042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da70980 0x56301da774e0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.755+0000 7fe994db5740 1 --2- 172.21.15.28:0/1587291042 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56301da6f290 0x56301da76fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.755+0000 7fe994db5740 1 -- 172.21.15.28:0/1587291042 >> 172.21.15.28:0/1587291042 conn(0x56301da37950 msgr2=0x56301da71660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:11.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.755+0000 7fe994db5740 1 -- 172.21.15.28:0/1587291042 shutdown_connections 2024-04-25T01:34:11.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.755+0000 7fe994db5740 1 -- 172.21.15.28:0/1587291042 wait complete. 2024-04-25T01:34:11.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.755+0000 7fe994db5740 20 client.16095 trim_cache size 0 max 16384 2024-04-25T01:34:11.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.773+0000 7fe994db5740 1 Processor -- start 2024-04-25T01:34:11.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.773+0000 7fe994db5740 1 -- start start 2024-04-25T01:34:11.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.773+0000 7fe994db5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da6f290 0x56301da85da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.773+0000 7fe994db5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56301da7a010 0x56301da77840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.773+0000 7fe994db5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56301da91eb0 0x56301da922f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.773+0000 7fe994db5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe98c1b0d40 con 0x56301da6f290 2024-04-25T01:34:11.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.773+0000 7fe994db5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe98c1b2880 con 0x56301da91eb0 2024-04-25T01:34:11.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.773+0000 7fe994db5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe98c1b1410 con 0x56301da7a010 2024-04-25T01:34:11.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.773+0000 7fe98bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da6f290 0x56301da85da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.773+0000 7fe98bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da6f290 0x56301da85da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59096/0 (socket says 172.21.15.28:59096) 2024-04-25T01:34:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.773+0000 7fe98bfff640 1 -- 172.21.15.28:0/3498735145 learned_addr learned my addr 172.21.15.28:0/3498735145 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.773+0000 7fe98b7fe640 1 --2- 172.21.15.28:0/3498735145 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56301da7a010 0x56301da77840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.774+0000 7fe98bfff640 1 -- 172.21.15.28:0/3498735145 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56301da7a010 msgr2=0x56301da77840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.774+0000 7fe98bfff640 1 --2- 172.21.15.28:0/3498735145 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56301da7a010 0x56301da77840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.774+0000 7fe98bfff640 1 -- 172.21.15.28:0/3498735145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56301da91eb0 msgr2=0x56301da922f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.774+0000 7fe98bfff640 1 --2- 172.21.15.28:0/3498735145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56301da91eb0 0x56301da922f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.774+0000 7fe98bfff640 1 -- 172.21.15.28:0/3498735145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56301d6a7170 con 0x56301da6f290 2024-04-25T01:34:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.774+0000 7fe98b7fe640 1 --2- 172.21.15.28:0/3498735145 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56301da7a010 0x56301da77840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.774+0000 7fe98bfff640 1 --2- 172.21.15.28:0/3498735145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da6f290 0x56301da85da0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe984005ab0 tx=0x7fe984009440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=896e058006a06f83 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.774+0000 7fe9817fa640 1 -- 172.21.15.28:0/3498735145 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe984016040 con 0x56301da6f290 2024-04-25T01:34:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.774+0000 7fe9817fa640 1 -- 172.21.15.28:0/3498735145 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe98402de50 con 0x56301da6f290 2024-04-25T01:34:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.774+0000 7fe9817fa640 1 -- 172.21.15.28:0/3498735145 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe984011950 con 0x56301da6f290 2024-04-25T01:34:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.775+0000 7fe994db5740 1 -- 172.21.15.28:0/3498735145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da6f290 msgr2=0x56301da85da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.775+0000 7fe994db5740 1 --2- 172.21.15.28:0/3498735145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da6f290 0x56301da85da0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe984005ab0 tx=0x7fe984009440 comp rx=0 tx=0).stop 2024-04-25T01:34:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.775+0000 7fe994db5740 1 -- 172.21.15.28:0/3498735145 shutdown_connections 2024-04-25T01:34:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.775+0000 7fe994db5740 1 --2- 172.21.15.28:0/3498735145 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56301da91eb0 0x56301da922f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.775+0000 7fe994db5740 1 --2- 172.21.15.28:0/3498735145 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56301da7a010 0x56301da77840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.775+0000 7fe994db5740 1 --2- 172.21.15.28:0/3498735145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da6f290 0x56301da85da0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.775+0000 7fe994db5740 1 -- 172.21.15.28:0/3498735145 >> 172.21.15.28:0/3498735145 conn(0x56301da37950 msgr2=0x56301da6a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.775+0000 7fe994db5740 1 -- 172.21.15.28:0/3498735145 shutdown_connections 2024-04-25T01:34:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.775+0000 7fe994db5740 1 -- 172.21.15.28:0/3498735145 wait complete. 2024-04-25T01:34:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.775+0000 7fe994db5740 1 -- start start 2024-04-25T01:34:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.776+0000 7fe994db5740 1 Processor -- start 2024-04-25T01:34:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.776+0000 7fe994db5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56301da6f290 0x56301da82650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.776+0000 7fe994db5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da7a010 0x56301da7f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.776+0000 7fe994db5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56301da91eb0 0x56301da7f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.776+0000 7fe994db5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe98c1b21b0 con 0x56301da7a010 2024-04-25T01:34:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.776+0000 7fe994db5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe98c1b1ae0 con 0x56301da91eb0 2024-04-25T01:34:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.776+0000 7fe994db5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe98c1aed20 con 0x56301da6f290 2024-04-25T01:34:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.776+0000 7fe994db5740 10 client.0 _ll_register_callbacks cb 0x7fe974009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.776+0000 7fe98b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da7a010 0x56301da7f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.776+0000 7fe98b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da7a010 0x56301da7f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59098/0 (socket says 172.21.15.28:59098) 2024-04-25T01:34:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.776+0000 7fe98b7fe640 1 -- 172.21.15.28:0/2054622321 learned_addr learned my addr 172.21.15.28:0/2054622321 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.777+0000 7fe98b7fe640 1 -- 172.21.15.28:0/2054622321 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56301da6f290 msgr2=0x56301da82650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:34:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.777+0000 7fe98b7fe640 1 --2- 172.21.15.28:0/2054622321 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56301da6f290 0x56301da82650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.777+0000 7fe98b7fe640 1 -- 172.21.15.28:0/2054622321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56301da91eb0 msgr2=0x56301da7f7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:34:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.777+0000 7fe98b7fe640 1 --2- 172.21.15.28:0/2054622321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56301da91eb0 0x56301da7f7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.777+0000 7fe98b7fe640 1 -- 172.21.15.28:0/2054622321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56301da7ffa0 con 0x56301da7a010 2024-04-25T01:34:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.777+0000 7fe98b7fe640 1 --2- 172.21.15.28:0/2054622321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da7a010 0x56301da7f280 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe974031550 tx=0x7fe974000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dfd5492ed83648f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.777+0000 7fe96affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.777+0000 7fe96affd640 1 -- 172.21.15.28:0/2054622321 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe974035070 con 0x56301da7a010 2024-04-25T01:34:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.777+0000 7fe96affd640 1 -- 172.21.15.28:0/2054622321 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fe974005250 con 0x56301da7a010 2024-04-25T01:34:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.777+0000 7fe994db5740 1 -- 172.21.15.28:0/2054622321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe98402de50 con 0x56301da7a010 2024-04-25T01:34:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.777+0000 7fe96affd640 1 -- 172.21.15.28:0/2054622321 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe97402ab90 con 0x56301da7a010 2024-04-25T01:34:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.777+0000 7fe994db5740 10 client.16101 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.777+0000 7fe994db5740 1 -- 172.21.15.28:0/2054622321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56301daf9ed0 con 0x56301da7a010 2024-04-25T01:34:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.778+0000 7fe96affd640 1 -- 172.21.15.28:0/2054622321 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe98402de50 con 0x56301da7a010 2024-04-25T01:34:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.778+0000 7fe96affd640 1 -- 172.21.15.28:0/2054622321 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe974031a70 con 0x56301da7a010 2024-04-25T01:34:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.778+0000 7fe994db5740 10 client.16101 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.778+0000 7fe994db5740 10 client.16101 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.778+0000 7fe994db5740 20 client.16101 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.778+0000 7fe994db5740 1 --2- 172.21.15.28:0/2054622321 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56301dafa590 0x56301dafc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.778+0000 7fe994db5740 4 client.16101 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:34:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.778+0000 7fe994db5740 1 -- 172.21.15.28:0/2054622321 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56301d9e0800 con 0x56301dafa590 2024-04-25T01:34:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.778+0000 7fe98bfff640 1 --2- 172.21.15.28:0/2054622321 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56301dafa590 0x56301dafc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.779+0000 7fe96affd640 10 client.16101 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.779+0000 7fe98bfff640 1 --2- 172.21.15.28:0/2054622321 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56301dafa590 0x56301dafc9b0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b53321550ad8e8a0 server_cookie=78c83cbfe0f993a3 in_seq=0 out_seq=0 2024-04-25T01:34:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.780+0000 7fe96affd640 1 -- 172.21.15.28:0/2054622321 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56301d9e0800 con 0x56301dafa590 2024-04-25T01:34:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.780+0000 7fe96affd640 10 client.16101 handle_command_reply: tid=0 2024-04-25T01:34:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.780+0000 7fe994db5740 1 client.16101 shutdown 2024-04-25T01:34:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.780+0000 7fe994db5740 1 -- 172.21.15.28:0/2054622321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da7a010 msgr2=0x56301da7f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.780+0000 7fe994db5740 1 --2- 172.21.15.28:0/2054622321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da7a010 0x56301da7f280 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe974031550 tx=0x7fe974000bc0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.780+0000 7fe994db5740 1 -- 172.21.15.28:0/2054622321 shutdown_connections 2024-04-25T01:34:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.780+0000 7fe994db5740 1 --2- 172.21.15.28:0/2054622321 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56301dafa590 0x56301dafc9b0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.780+0000 7fe994db5740 1 --2- 172.21.15.28:0/2054622321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56301da91eb0 0x56301da7f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.780+0000 7fe96affd640 0 client.16101 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.780+0000 7fe994db5740 1 --2- 172.21.15.28:0/2054622321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56301da7a010 0x56301da7f280 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.780+0000 7fe994db5740 1 --2- 172.21.15.28:0/2054622321 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56301da6f290 0x56301da82650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.780+0000 7fe994db5740 1 -- 172.21.15.28:0/2054622321 >> 172.21.15.28:0/2054622321 conn(0x56301da37950 msgr2=0x56301da90ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.780+0000 7fe994db5740 1 -- 172.21.15.28:0/2054622321 shutdown_connections 2024-04-25T01:34:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.781+0000 7fe994db5740 1 -- 172.21.15.28:0/2054622321 wait complete. 2024-04-25T01:34:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.781+0000 7fe994db5740 20 client.16101 trim_cache size 0 max 16384 2024-04-25T01:34:11.761 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:34:11.761 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:34:11.761 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "01791c66-14b0-4e29-b1b7-181e98ea519b", 2024-04-25T01:34:11.761 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:34:11.762 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:34:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.781+0000 7fe9920a5640 1 -- 172.21.15.28:0/1691215333 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe9780a2e20 msgr2=0x7fe9780a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.781+0000 7fe9920a5640 1 --2- 172.21.15.28:0/1691215333 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe9780a2e20 0x7fe9780a52e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe97c0cc880 tx=0x7fe97c00df00 comp rx=0 tx=0).stop 2024-04-25T01:34:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.781+0000 7fe9920a5640 1 -- 172.21.15.28:0/1691215333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98c1029c0 msgr2=0x7fe98c100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.781+0000 7fe9920a5640 1 --2- 172.21.15.28:0/1691215333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98c1029c0 0x7fe98c100610 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00d770 tx=0x7fe97c00dc30 comp rx=0 tx=0).stop 2024-04-25T01:34:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.781+0000 7fe9920a5640 1 -- 172.21.15.28:0/1691215333 shutdown_connections 2024-04-25T01:34:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.781+0000 7fe9920a5640 1 --2- 172.21.15.28:0/1691215333 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe9780a2e20 0x7fe9780a52e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.781+0000 7fe9920a5640 1 --2- 172.21.15.28:0/1691215333 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe98c10bd00 0x7fe98c101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.781+0000 7fe9920a5640 1 --2- 172.21.15.28:0/1691215333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe98c1032e0 0x7fe98c100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.781+0000 7fe9920a5640 1 --2- 172.21.15.28:0/1691215333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98c1029c0 0x7fe98c100610 secure :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00d770 tx=0x7fe97c00dc30 comp rx=0 tx=0).stop 2024-04-25T01:34:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.781+0000 7fe9920a5640 1 -- 172.21.15.28:0/1691215333 >> 172.21.15.28:0/1691215333 conn(0x7fe98c077740 msgr2=0x7fe98c0fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.781+0000 7fe9920a5640 1 -- 172.21.15.28:0/1691215333 shutdown_connections 2024-04-25T01:34:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.782+0000 7fe9920a5640 1 -- 172.21.15.28:0/1691215333 wait complete. 2024-04-25T01:34:11.771 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:34:11.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:34:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.902+0000 7f51c2fa9640 1 -- 172.21.15.28:0/1319249872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51bc06a410 msgr2=0x7f51bc10e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:11.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.902+0000 7f51c2fa9640 1 --2- 172.21.15.28:0/1319249872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51bc06a410 0x7f51bc10e5a0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f51b4009ca0 tx=0x7f51b402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.902+0000 7f51c2fa9640 1 -- 172.21.15.28:0/1319249872 shutdown_connections 2024-04-25T01:34:11.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.902+0000 7f51c2fa9640 1 --2- 172.21.15.28:0/1319249872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51bc06a410 0x7f51bc10e5a0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.902+0000 7f51c2fa9640 1 --2- 172.21.15.28:0/1319249872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51bc069a50 0x7f51bc069ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.902+0000 7f51c2fa9640 1 --2- 172.21.15.28:0/1319249872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51bc107790 0x7f51bc107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.902+0000 7f51c2fa9640 1 -- 172.21.15.28:0/1319249872 >> 172.21.15.28:0/1319249872 conn(0x7f51bc0761c0 msgr2=0x7f51bc0765d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:11.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.902+0000 7f51c2fa9640 1 -- 172.21.15.28:0/1319249872 shutdown_connections 2024-04-25T01:34:11.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.902+0000 7f51c2fa9640 1 -- 172.21.15.28:0/1319249872 wait complete. 2024-04-25T01:34:11.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.903+0000 7f51c2fa9640 1 Processor -- start 2024-04-25T01:34:11.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.903+0000 7f51c2fa9640 1 -- start start 2024-04-25T01:34:11.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.903+0000 7f51c2fa9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51bc069a50 0x7f51bc197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.903+0000 7f51c2fa9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51bc06a410 0x7f51bc1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.903+0000 7f51c2fa9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51bc107790 0x7f51bc19c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.903+0000 7f51c2fa9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc111ae0 con 0x7f51bc107790 2024-04-25T01:34:11.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.903+0000 7f51c2fa9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc111960 con 0x7f51bc06a410 2024-04-25T01:34:11.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.903+0000 7f51c2fa9640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc111c60 con 0x7f51bc069a50 2024-04-25T01:34:11.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.903+0000 7f51bbfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51bc06a410 0x7f51bc1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.903+0000 7f51bbfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51bc06a410 0x7f51bc1984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:46178/0 (socket says 172.21.15.28:46178) 2024-04-25T01:34:11.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.903+0000 7f51bbfff640 1 -- 172.21.15.28:0/2264917532 learned_addr learned my addr 172.21.15.28:0/2264917532 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:11.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.904+0000 7f51c1546640 1 --2- 172.21.15.28:0/2264917532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51bc107790 0x7f51bc19c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.904+0000 7f51c0d45640 1 --2- 172.21.15.28:0/2264917532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51bc069a50 0x7f51bc197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.904+0000 7f51bbfff640 1 -- 172.21.15.28:0/2264917532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51bc069a50 msgr2=0x7f51bc197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:11.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.904+0000 7f51bbfff640 1 --2- 172.21.15.28:0/2264917532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51bc069a50 0x7f51bc197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.904+0000 7f51bbfff640 1 -- 172.21.15.28:0/2264917532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51bc107790 msgr2=0x7f51bc19c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:11.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.904+0000 7f51bbfff640 1 --2- 172.21.15.28:0/2264917532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51bc107790 0x7f51bc19c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.904+0000 7f51bbfff640 1 -- 172.21.15.28:0/2264917532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51bc19cf20 con 0x7f51bc06a410 2024-04-25T01:34:11.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.904+0000 7f51c1546640 1 --2- 172.21.15.28:0/2264917532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51bc107790 0x7f51bc19c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:11.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.904+0000 7f51bbfff640 1 --2- 172.21.15.28:0/2264917532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51bc06a410 0x7f51bc1984c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f51b000d950 tx=0x7f51b000dd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:11.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.904+0000 7f51b9ffb640 1 -- 172.21.15.28:0/2264917532 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0017030 con 0x7f51bc06a410 2024-04-25T01:34:11.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.905+0000 7f51b9ffb640 1 -- 172.21.15.28:0/2264917532 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51b0004020 con 0x7f51bc06a410 2024-04-25T01:34:11.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.905+0000 7f51b9ffb640 1 -- 172.21.15.28:0/2264917532 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0005650 con 0x7f51bc06a410 2024-04-25T01:34:11.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.905+0000 7f51c2fa9640 1 -- 172.21.15.28:0/2264917532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51bc19d210 con 0x7f51bc06a410 2024-04-25T01:34:11.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.905+0000 7f51c2fa9640 1 -- 172.21.15.28:0/2264917532 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51bc100b20 con 0x7f51bc06a410 2024-04-25T01:34:11.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.907+0000 7f51b9ffb640 1 -- 172.21.15.28:0/2264917532 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 24) ==== 138066+0+0 (secure 0 0 0) 0x7f51b00041d0 con 0x7f51bc06a410 2024-04-25T01:34:11.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.908+0000 7f51b9ffb640 1 --2- 172.21.15.28:0/2264917532 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f51a40a7230 0x7f51a40a96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.908+0000 7f51c0d45640 1 --2- 172.21.15.28:0/2264917532 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f51a40a7230 0x7f51a40a96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.908+0000 7f51b9ffb640 1 -- 172.21.15.28:0/2264917532 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f51b00cf3c0 con 0x7f51bc06a410 2024-04-25T01:34:11.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.909+0000 7f51c0d45640 1 --2- 172.21.15.28:0/2264917532 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f51a40a7230 0x7f51a40a96f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f51ac00ab60 tx=0x7f51ac00a2d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:11.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.909+0000 7f51c5d9a740 1 Processor -- start 2024-04-25T01:34:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.909+0000 7f51c5d9a740 1 -- start start 2024-04-25T01:34:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.909+0000 7f51c5d9a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072630 0x555e75072a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.909+0000 7f51c5d9a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e75072f50 0x555e7507d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.909+0000 7f51c5d9a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 0x555e7507fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.909+0000 7f51c5d9a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555e74fd76b0 con 0x555e75072630 2024-04-25T01:34:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.909+0000 7f51c5d9a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555e74f7f880 con 0x555e7507d620 2024-04-25T01:34:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.909+0000 7f51c5d9a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x555e74fd3670 con 0x555e75072f50 2024-04-25T01:34:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.910+0000 7f51c0d45640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072630 0x555e75072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.910+0000 7f51c0d45640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072630 0x555e75072a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59132/0 (socket says 172.21.15.28:59132) 2024-04-25T01:34:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.910+0000 7f51c0d45640 1 -- 172.21.15.28:0/1241850118 learned_addr learned my addr 172.21.15.28:0/1241850118 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.910+0000 7f51c1546640 1 --2- 172.21.15.28:0/1241850118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 0x555e7507fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.910+0000 7f51bbfff640 1 --2- 172.21.15.28:0/1241850118 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e75072f50 0x555e7507d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.910+0000 7f51c0d45640 1 -- 172.21.15.28:0/1241850118 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e75072f50 msgr2=0x555e7507d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.910+0000 7f51c0d45640 1 --2- 172.21.15.28:0/1241850118 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e75072f50 0x555e7507d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.910+0000 7f51c0d45640 1 -- 172.21.15.28:0/1241850118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 msgr2=0x555e7507fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.910+0000 7f51c0d45640 1 --2- 172.21.15.28:0/1241850118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 0x555e7507fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.910+0000 7f51c0d45640 1 -- 172.21.15.28:0/1241850118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e74db57d0 con 0x555e75072630 2024-04-25T01:34:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.910+0000 7f51c1546640 1 --2- 172.21.15.28:0/1241850118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 0x555e7507fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.910+0000 7f51c0d45640 1 --2- 172.21.15.28:0/1241850118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072630 0x555e75072a10 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f51ac0107c0 tx=0x7f51ac012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87faa53d9104934d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.911+0000 7f51a2ffd640 1 -- 172.21.15.28:0/1241850118 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac016070 con 0x555e75072630 2024-04-25T01:34:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.911+0000 7f51a2ffd640 1 -- 172.21.15.28:0/1241850118 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51ac01be10 con 0x555e75072630 2024-04-25T01:34:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.911+0000 7f51a2ffd640 1 -- 172.21.15.28:0/1241850118 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac027e10 con 0x555e75072630 2024-04-25T01:34:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.911+0000 7f51c5d9a740 1 -- 172.21.15.28:0/1241850118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072630 msgr2=0x555e75072a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.911+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/1241850118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072630 0x555e75072a10 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f51ac0107c0 tx=0x7f51ac012000 comp rx=0 tx=0).stop 2024-04-25T01:34:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.912+0000 7f51c5d9a740 1 -- 172.21.15.28:0/1241850118 shutdown_connections 2024-04-25T01:34:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.912+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/1241850118 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 0x555e7507fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.912+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/1241850118 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e75072f50 0x555e7507d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.912+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/1241850118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072630 0x555e75072a10 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.912+0000 7f51c5d9a740 1 -- 172.21.15.28:0/1241850118 >> 172.21.15.28:0/1241850118 conn(0x555e7506e950 msgr2=0x555e7506ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.912+0000 7f51c5d9a740 1 -- 172.21.15.28:0/1241850118 shutdown_connections 2024-04-25T01:34:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.912+0000 7f51c5d9a740 1 -- 172.21.15.28:0/1241850118 wait complete. 2024-04-25T01:34:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.912+0000 7f51c5d9a740 1 -- start start 2024-04-25T01:34:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.912+0000 7f51c5d9a740 1 Processor -- start 2024-04-25T01:34:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.913+0000 7f51c5d9a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e75072630 0x555e7507a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.913+0000 7f51c5d9a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072f50 0x555e7507aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.913+0000 7f51c5d9a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 0x555e7507b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.913+0000 7f51c5d9a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555e75080410 con 0x555e75072f50 2024-04-25T01:34:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.913+0000 7f51c5d9a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555e74f92f50 con 0x555e7507d620 2024-04-25T01:34:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.913+0000 7f51c5d9a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x555e75080590 con 0x555e75072630 2024-04-25T01:34:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.913+0000 7f51bbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072f50 0x555e7507aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.913+0000 7f51bbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072f50 0x555e7507aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59134/0 (socket says 172.21.15.28:59134) 2024-04-25T01:34:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.913+0000 7f51bbfff640 1 -- 172.21.15.28:0/2614929611 learned_addr learned my addr 172.21.15.28:0/2614929611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.913+0000 7f51c5d9a740 10 client.0 _ll_register_callbacks cb 0x555e74fce000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.913+0000 7f51c0d45640 1 --2- 172.21.15.28:0/2614929611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e75072630 0x555e7507a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.913+0000 7f51c1546640 1 --2- 172.21.15.28:0/2614929611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 0x555e7507b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.914+0000 7f51bbfff640 1 -- 172.21.15.28:0/2614929611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e75072630 msgr2=0x555e7507a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:11.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.914+0000 7f51bbfff640 1 --2- 172.21.15.28:0/2614929611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e75072630 0x555e7507a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.914+0000 7f51bbfff640 1 -- 172.21.15.28:0/2614929611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 msgr2=0x555e7507b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:11.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.914+0000 7f51bbfff640 1 --2- 172.21.15.28:0/2614929611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 0x555e7507b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.914+0000 7f51bbfff640 1 -- 172.21.15.28:0/2614929611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e74ceb3f0 con 0x555e75072f50 2024-04-25T01:34:11.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.914+0000 7f51bbfff640 1 --2- 172.21.15.28:0/2614929611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072f50 0x555e7507aac0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f51b0097d20 tx=0x7f51b009ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9502c727feb96df4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:11.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.914+0000 7f519bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:11.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.914+0000 7f519bfff640 1 -- 172.21.15.28:0/2614929611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b009f700 con 0x555e75072f50 2024-04-25T01:34:11.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.914+0000 7f519bfff640 1 -- 172.21.15.28:0/2614929611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f51b00979c0 con 0x555e75072f50 2024-04-25T01:34:11.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.914+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2614929611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51ac01be10 con 0x555e75072f50 2024-04-25T01:34:11.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.915+0000 7f519bfff640 1 -- 172.21.15.28:0/2614929611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f51b00a3fc0 con 0x555e75072f50 2024-04-25T01:34:11.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.915+0000 7f51c5d9a740 10 client.16113 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:11.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.915+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2614929611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555e74cab0f0 con 0x555e75072f50 2024-04-25T01:34:11.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.915+0000 7f519bfff640 1 -- 172.21.15.28:0/2614929611 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51ac01be10 con 0x555e75072f50 2024-04-25T01:34:11.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.915+0000 7f519bfff640 1 -- 172.21.15.28:0/2614929611 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f51b009f170 con 0x555e75072f50 2024-04-25T01:34:11.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.915+0000 7f51c5d9a740 10 client.16113 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:11.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.915+0000 7f51c5d9a740 10 client.16113 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:11.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.915+0000 7f51c5d9a740 20 client.16113 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:11.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.915+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/2614929611 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555e7508d980 0x555e7508fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.915+0000 7f51c5d9a740 4 client.16113 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:11.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.915+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2614929611 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e74fe4bf0 con 0x555e7508d980 2024-04-25T01:34:11.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.916+0000 7f51c0d45640 1 --2- 172.21.15.28:0/2614929611 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555e7508d980 0x555e7508fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.916+0000 7f519bfff640 10 client.16113 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:11.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.916+0000 7f51c0d45640 1 --2- 172.21.15.28:0/2614929611 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555e7508d980 0x555e7508fd90 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5d69112fc4603ea server_cookie=64db2dcecfcd98e3 in_seq=0 out_seq=0 2024-04-25T01:34:11.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.917+0000 7f519bfff640 1 -- 172.21.15.28:0/2614929611 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555e74fe4bf0 con 0x555e7508d980 2024-04-25T01:34:11.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.917+0000 7f519bfff640 10 client.16113 handle_command_reply: tid=0 2024-04-25T01:34:11.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.917+0000 7f51c5d9a740 1 client.16113 shutdown 2024-04-25T01:34:11.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.917+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2614929611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072f50 msgr2=0x555e7507aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:11.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.917+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/2614929611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072f50 0x555e7507aac0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f51b0097d20 tx=0x7f51b009ad70 comp rx=0 tx=0).stop 2024-04-25T01:34:11.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.917+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2614929611 shutdown_connections 2024-04-25T01:34:11.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.917+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/2614929611 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555e7508d980 0x555e7508fd90 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.917+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/2614929611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 0x555e7507b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.917+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/2614929611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072f50 0x555e7507aac0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.917+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/2614929611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e75072630 0x555e7507a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.917+0000 7f519bfff640 0 client.16113 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:11.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.917+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2614929611 >> 172.21.15.28:0/2614929611 conn(0x555e7506e950 msgr2=0x555e75074750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:11.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.917+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2614929611 shutdown_connections 2024-04-25T01:34:11.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.917+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2614929611 wait complete. 2024-04-25T01:34:11.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.917+0000 7f51c5d9a740 20 client.16113 trim_cache size 0 max 16384 2024-04-25T01:34:11.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.935+0000 7f51c5d9a740 1 Processor -- start 2024-04-25T01:34:11.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.935+0000 7f51c5d9a740 1 -- start start 2024-04-25T01:34:11.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.936+0000 7f51c5d9a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072630 0x555e7507ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.936+0000 7f51c5d9a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 0x555e750892d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.936+0000 7f51c5d9a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e7508d980 0x555e750794c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.936+0000 7f51c5d9a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc1ac8b0 con 0x555e75072630 2024-04-25T01:34:11.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.936+0000 7f51c5d9a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc1ae3f0 con 0x555e7507d620 2024-04-25T01:34:11.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.936+0000 7f51c5d9a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc1acf80 con 0x555e7508d980 2024-04-25T01:34:11.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.936+0000 7f51c0d45640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072630 0x555e7507ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.936+0000 7f51c0d45640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072630 0x555e7507ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59150/0 (socket says 172.21.15.28:59150) 2024-04-25T01:34:11.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.936+0000 7f51c0d45640 1 -- 172.21.15.28:0/2751885609 learned_addr learned my addr 172.21.15.28:0/2751885609 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:11.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.936+0000 7f51c1546640 1 --2- 172.21.15.28:0/2751885609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e7508d980 0x555e750794c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.936+0000 7f51bbfff640 1 --2- 172.21.15.28:0/2751885609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 0x555e750892d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.936+0000 7f51c0d45640 1 -- 172.21.15.28:0/2751885609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e7508d980 msgr2=0x555e750794c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:11.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.936+0000 7f51c0d45640 1 --2- 172.21.15.28:0/2751885609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e7508d980 0x555e750794c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.936+0000 7f51c0d45640 1 -- 172.21.15.28:0/2751885609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 msgr2=0x555e750892d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:11.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.936+0000 7f51c0d45640 1 --2- 172.21.15.28:0/2751885609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 0x555e750892d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.936+0000 7f51c0d45640 1 -- 172.21.15.28:0/2751885609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e75060090 con 0x555e75072630 2024-04-25T01:34:11.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.936+0000 7f51c1546640 1 --2- 172.21.15.28:0/2751885609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e7508d980 0x555e750794c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:11.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.936+0000 7f51c0d45640 1 --2- 172.21.15.28:0/2751885609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072630 0x555e7507ccb0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f51ac0107c0 tx=0x7f51ac027950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c3f8991cdeefe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:11.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.937+0000 7f51a27fc640 1 -- 172.21.15.28:0/2751885609 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac044070 con 0x555e75072630 2024-04-25T01:34:11.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.937+0000 7f51a27fc640 1 -- 172.21.15.28:0/2751885609 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51ac027680 con 0x555e75072630 2024-04-25T01:34:11.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.937+0000 7f51a27fc640 1 -- 172.21.15.28:0/2751885609 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac016050 con 0x555e75072630 2024-04-25T01:34:11.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.938+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2751885609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072630 msgr2=0x555e7507ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:11.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.938+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/2751885609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072630 0x555e7507ccb0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f51ac0107c0 tx=0x7f51ac027950 comp rx=0 tx=0).stop 2024-04-25T01:34:11.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.938+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2751885609 shutdown_connections 2024-04-25T01:34:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.938+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/2751885609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e7508d980 0x555e750794c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.938+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/2751885609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 0x555e750892d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.938+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/2751885609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e75072630 0x555e7507ccb0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.938+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2751885609 >> 172.21.15.28:0/2751885609 conn(0x555e7506e950 msgr2=0x555e7507add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.938+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2751885609 shutdown_connections 2024-04-25T01:34:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.938+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2751885609 wait complete. 2024-04-25T01:34:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.939+0000 7f51c5d9a740 1 -- start start 2024-04-25T01:34:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.939+0000 7f51c5d9a740 1 Processor -- start 2024-04-25T01:34:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.940+0000 7f51c5d9a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e75072630 0x555e750814a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.940+0000 7f51c5d9a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 0x555e750819e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.940+0000 7f51c5d9a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e7508d980 0x555e75081f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.940+0000 7f51c5d9a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc1add20 con 0x555e7508d980 2024-04-25T01:34:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.940+0000 7f51c5d9a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc1ad650 con 0x555e7507d620 2024-04-25T01:34:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.940+0000 7f51c5d9a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc1aa890 con 0x555e75072630 2024-04-25T01:34:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.940+0000 7f51c5d9a740 10 client.0 _ll_register_callbacks cb 0x7f51b0004520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.940+0000 7f51c1546640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e7508d980 0x555e75081f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.940+0000 7f51c0d45640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e75072630 0x555e750814a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.940+0000 7f51c1546640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e7508d980 0x555e75081f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59152/0 (socket says 172.21.15.28:59152) 2024-04-25T01:34:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.940+0000 7f51c1546640 1 -- 172.21.15.28:0/2073404618 learned_addr learned my addr 172.21.15.28:0/2073404618 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.940+0000 7f51bbfff640 1 --2- 172.21.15.28:0/2073404618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 0x555e750819e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.940+0000 7f51c0d45640 1 -- 172.21.15.28:0/2073404618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 msgr2=0x555e750819e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.940+0000 7f51c0d45640 1 --2- 172.21.15.28:0/2073404618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 0x555e750819e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.940+0000 7f51c0d45640 1 -- 172.21.15.28:0/2073404618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e7508d980 msgr2=0x555e75081f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.940+0000 7f51c0d45640 1 --2- 172.21.15.28:0/2073404618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e7508d980 0x555e75081f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.940+0000 7f51c0d45640 1 -- 172.21.15.28:0/2073404618 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e750f3170 con 0x555e75072630 2024-04-25T01:34:11.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.940+0000 7f51c0d45640 1 --2- 172.21.15.28:0/2073404618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e75072630 0x555e750814a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f51ac00a940 tx=0x7f51ac00f920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5282949b2bcba109 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.940+0000 7f519bfff640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-25T01:34:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.941+0000 7f519bfff640 1 -- 172.21.15.28:0/2073404618 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac044070 con 0x555e75072630 2024-04-25T01:34:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.941+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2073404618 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51ac027680 con 0x555e75072630 2024-04-25T01:34:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.941+0000 7f519bfff640 1 -- 172.21.15.28:0/2073404618 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f51ac01bc30 con 0x555e75072630 2024-04-25T01:34:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.941+0000 7f519bfff640 1 -- 172.21.15.28:0/2073404618 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f51ac016020 con 0x555e75072630 2024-04-25T01:34:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.941+0000 7f51c5d9a740 10 client.25573 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.941+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2073404618 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555e750f4480 con 0x555e75072630 2024-04-25T01:34:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.941+0000 7f519bfff640 1 -- 172.21.15.28:0/2073404618 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51ac027680 con 0x555e75072630 2024-04-25T01:34:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.941+0000 7f519bfff640 1 -- 172.21.15.28:0/2073404618 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f51ac005400 con 0x555e75072630 2024-04-25T01:34:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.941+0000 7f51c5d9a740 10 client.25573 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.941+0000 7f51c5d9a740 10 client.25573 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.941+0000 7f51c5d9a740 20 client.25573 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.941+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/2073404618 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555e750f4960 0x555e750f6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.941+0000 7f51c5d9a740 4 client.25573 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:34:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.941+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2073404618 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555e74fdef40 con 0x555e750f4960 2024-04-25T01:34:11.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.942+0000 7f51bbfff640 1 --2- 172.21.15.28:0/2073404618 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555e750f4960 0x555e750f6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.942+0000 7f519bfff640 10 client.25573 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.942+0000 7f51bbfff640 1 --2- 172.21.15.28:0/2073404618 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555e750f4960 0x555e750f6d50 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=956c3b5c13aba168 server_cookie=2d2e5e12498b18d8 in_seq=0 out_seq=0 2024-04-25T01:34:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.942+0000 7f519bfff640 1 -- 172.21.15.28:0/2073404618 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x555e74fdef40 con 0x555e750f4960 2024-04-25T01:34:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.942+0000 7f519bfff640 10 client.25573 handle_command_reply: tid=0 2024-04-25T01:34:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.942+0000 7f51c5d9a740 1 client.25573 shutdown 2024-04-25T01:34:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.942+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2073404618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e75072630 msgr2=0x555e750814a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.942+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/2073404618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e75072630 0x555e750814a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f51ac00a940 tx=0x7f51ac00f920 comp rx=0 tx=0).stop 2024-04-25T01:34:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.942+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2073404618 shutdown_connections 2024-04-25T01:34:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.942+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/2073404618 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555e750f4960 0x555e750f6d50 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.943+0000 7f519bfff640 0 client.25573 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.943+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/2073404618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e7508d980 0x555e75081f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.943+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/2073404618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555e7507d620 0x555e750819e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.943+0000 7f51c5d9a740 1 --2- 172.21.15.28:0/2073404618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555e75072630 0x555e750814a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.943+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2073404618 >> 172.21.15.28:0/2073404618 conn(0x555e7506e950 msgr2=0x555e7507a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:11.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.943+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2073404618 shutdown_connections 2024-04-25T01:34:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.943+0000 7f51c5d9a740 1 -- 172.21.15.28:0/2073404618 wait complete. 2024-04-25T01:34:11.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.943+0000 7f51c5d9a740 20 client.25573 trim_cache size 0 max 16384 2024-04-25T01:34:11.926 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:34:11.926 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:34:11.926 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-25T01:34:11.926 INFO:teuthology.orchestra.run.smithi028.stdout: "01791c66-14b0-4e29-b1b7-181e98ea519b": { 2024-04-25T01:34:11.926 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-25T01:34:11.926 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "01791c66-14b0-4e29-b1b7-181e98ea519b", 2024-04-25T01:34:11.926 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-25T01:34:11.926 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:34:11.927 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:34:11.927 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:34:11.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.943+0000 7f51c2fa9640 1 -- 172.21.15.28:0/2264917532 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f51a40a7230 msgr2=0x7f51a40a96f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:11.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.943+0000 7f51c2fa9640 1 --2- 172.21.15.28:0/2264917532 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f51a40a7230 0x7f51a40a96f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f51ac00ab60 tx=0x7f51ac00a2d0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.943+0000 7f51c2fa9640 1 -- 172.21.15.28:0/2264917532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51bc06a410 msgr2=0x7f51bc1984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.943+0000 7f51c2fa9640 1 --2- 172.21.15.28:0/2264917532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51bc06a410 0x7f51bc1984c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f51b000d950 tx=0x7f51b000dd50 comp rx=0 tx=0).stop 2024-04-25T01:34:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.943+0000 7f51c2fa9640 1 -- 172.21.15.28:0/2264917532 shutdown_connections 2024-04-25T01:34:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.943+0000 7f51c2fa9640 1 --2- 172.21.15.28:0/2264917532 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f51a40a7230 0x7f51a40a96f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.943+0000 7f51c2fa9640 1 --2- 172.21.15.28:0/2264917532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51bc107790 0x7f51bc19c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.943+0000 7f51c2fa9640 1 --2- 172.21.15.28:0/2264917532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51bc06a410 0x7f51bc1984c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.943+0000 7f51c2fa9640 1 --2- 172.21.15.28:0/2264917532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f51bc069a50 0x7f51bc197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.943+0000 7f51c2fa9640 1 -- 172.21.15.28:0/2264917532 >> 172.21.15.28:0/2264917532 conn(0x7f51bc0761c0 msgr2=0x7f51bc0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.944+0000 7f51c2fa9640 1 -- 172.21.15.28:0/2264917532 shutdown_connections 2024-04-25T01:34:11.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:11.944+0000 7f51c2fa9640 1 -- 172.21.15.28:0/2264917532 wait complete. 2024-04-25T01:34:11.932 INFO:tasks.cephfs.filesystem:scrub status for tag:01791c66-14b0-4e29-b1b7-181e98ea519b - {'path': '/', 'tag': '01791c66-14b0-4e29-b1b7-181e98ea519b', 'options': 'recursive,force'} 2024-04-25T01:34:12.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:12 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:34:12.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:12 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:12.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:12 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:34:12.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:12 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:12.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:12 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:34:12.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:12 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:12.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:12 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:34:12.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:12 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:12.733 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:12 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:34:12.733 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:12 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:12.733 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:12 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:34:12.733 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:12 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:13.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:13 smithi028 ceph-mon[29390]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-04-25T01:34:13.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:13 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:34:13.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:13 smithi130 ceph-mon[32702]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-04-25T01:34:13.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:13 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:34:13.733 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:13 smithi032 ceph-mon[32461]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-04-25T01:34:13.733 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:13 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:34:15.957 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:15 smithi130 ceph-mon[32702]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-25T01:34:15.982 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:15 smithi032 ceph-mon[32461]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-25T01:34:16.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:15 smithi028 ceph-mon[29390]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-25T01:34:17.957 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:17 smithi130 ceph-mon[32702]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 170 B/s wr, 12 op/s 2024-04-25T01:34:17.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:17 smithi032 ceph-mon[32461]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 170 B/s wr, 12 op/s 2024-04-25T01:34:18.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:17 smithi028 ceph-mon[29390]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 170 B/s wr, 12 op/s 2024-04-25T01:34:19.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:19 smithi130 ceph-mon[32702]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T01:34:19.982 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:19 smithi032 ceph-mon[32461]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T01:34:20.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:19 smithi028 ceph-mon[29390]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T01:34:20.103 INFO:tasks.workunit.client.0.smithi028.stderr:Updating files: 96% (12109/12492) Updating files: 97% (12118/12492) Updating files: 98% (12243/12492) Updating files: 99% (12368/12492) Updating files: 100% (12492/12492) Updating files: 100% (12492/12492), done. 2024-04-25T01:34:20.379 INFO:tasks.workunit.client.0.smithi028.stderr:Note: switching to 'd2c45d977142e024a59b68a8f1dce8f0b9840551'. 2024-04-25T01:34:20.379 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-25T01:34:20.379 INFO:tasks.workunit.client.0.smithi028.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T01:34:20.379 INFO:tasks.workunit.client.0.smithi028.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T01:34:20.379 INFO:tasks.workunit.client.0.smithi028.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T01:34:20.379 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-25T01:34:20.380 INFO:tasks.workunit.client.0.smithi028.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T01:34:20.380 INFO:tasks.workunit.client.0.smithi028.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T01:34:20.380 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-25T01:34:20.380 INFO:tasks.workunit.client.0.smithi028.stderr: git switch -c 2024-04-25T01:34:20.380 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-25T01:34:20.380 INFO:tasks.workunit.client.0.smithi028.stderr:Or undo this operation with: 2024-04-25T01:34:20.380 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-25T01:34:20.380 INFO:tasks.workunit.client.0.smithi028.stderr: git switch - 2024-04-25T01:34:20.380 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-25T01:34:20.380 INFO:tasks.workunit.client.0.smithi028.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T01:34:20.380 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-25T01:34:20.380 INFO:tasks.workunit.client.0.smithi028.stderr:HEAD is now at d2c45d97714 Merge pull request #56528 from adk3798/squid-mypy-bump 2024-04-25T01:34:20.383 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-25T01:34:20.413 INFO:tasks.workunit.client.0.smithi028.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-25T01:34:20.415 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T01:34:20.416 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-25T01:34:21.043 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-25T01:34:21.096 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-25T01:34:21.137 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T01:34:21.138 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T01:34:21.138 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-25T01:34:21.178 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T01:34:21.181 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:34:21.181 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-25T01:34:21.238 INFO:tasks.workunit:Running workunits matching suites/iozone.sh on client.0... 2024-04-25T01:34:21.239 INFO:tasks.workunit:Running workunit suites/iozone.sh... 2024-04-25T01:34:21.239 DEBUG:teuthology.orchestra.run.smithi028:workunit test suites/iozone.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d2c45d977142e024a59b68a8f1dce8f0b9840551 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh 2024-04-25T01:34:21.299 INFO:tasks.workunit.client.0.smithi028.stderr:+ iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-04-25T01:34:21.377 INFO:tasks.workunit.client.0.smithi028.stdout: Iozone: Performance Test of File I/O 2024-04-25T01:34:21.377 INFO:tasks.workunit.client.0.smithi028.stdout: Version $Revision: 3.494 $ 2024-04-25T01:34:21.377 INFO:tasks.workunit.client.0.smithi028.stdout: Compiled for 64 bit mode. 2024-04-25T01:34:21.377 INFO:tasks.workunit.client.0.smithi028.stdout: Build: linux-AMD64 2024-04-25T01:34:21.377 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:21.377 INFO:tasks.workunit.client.0.smithi028.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-04-25T01:34:21.378 INFO:tasks.workunit.client.0.smithi028.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-04-25T01:34:21.378 INFO:tasks.workunit.client.0.smithi028.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-04-25T01:34:21.378 INFO:tasks.workunit.client.0.smithi028.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-04-25T01:34:21.378 INFO:tasks.workunit.client.0.smithi028.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-04-25T01:34:21.378 INFO:tasks.workunit.client.0.smithi028.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-04-25T01:34:21.378 INFO:tasks.workunit.client.0.smithi028.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-04-25T01:34:21.378 INFO:tasks.workunit.client.0.smithi028.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-04-25T01:34:21.378 INFO:tasks.workunit.client.0.smithi028.stdout: Alexey Skidanov, Sudhir Kumar. 2024-04-25T01:34:21.378 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:21.378 INFO:tasks.workunit.client.0.smithi028.stdout: Run began: Thu Apr 25 01:34:21 2024 2024-04-25T01:34:21.378 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:21.378 INFO:tasks.workunit.client.0.smithi028.stdout: Include close in write timing 2024-04-25T01:34:21.378 INFO:tasks.workunit.client.0.smithi028.stdout: Include fsync in write timing 2024-04-25T01:34:21.378 INFO:tasks.workunit.client.0.smithi028.stdout: File size set to 1048576 kB 2024-04-25T01:34:21.379 INFO:tasks.workunit.client.0.smithi028.stdout: Record Size 16 kB 2024-04-25T01:34:21.379 INFO:tasks.workunit.client.0.smithi028.stdout: Command line used: iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 2024-04-25T01:34:21.379 INFO:tasks.workunit.client.0.smithi028.stdout: Output is in kBytes/sec 2024-04-25T01:34:21.379 INFO:tasks.workunit.client.0.smithi028.stdout: Time Resolution = 0.000001 seconds. 2024-04-25T01:34:21.379 INFO:tasks.workunit.client.0.smithi028.stdout: Processor cache size set to 1024 kBytes. 2024-04-25T01:34:21.379 INFO:tasks.workunit.client.0.smithi028.stdout: Processor cache line size set to 32 bytes. 2024-04-25T01:34:21.379 INFO:tasks.workunit.client.0.smithi028.stdout: File stride size set to 17 * record size. 2024-04-25T01:34:21.379 INFO:tasks.workunit.client.0.smithi028.stdout: Throughput test with 1 process 2024-04-25T01:34:21.379 INFO:tasks.workunit.client.0.smithi028.stdout: Each process writes a 1048576 kByte file in 16 kByte records 2024-04-25T01:34:21.957 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:21 smithi130 ceph-mon[32702]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T01:34:21.982 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:21 smithi032 ceph-mon[32461]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T01:34:22.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:21 smithi028 ceph-mon[29390]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T01:34:22.727 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:22.728 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 initial writers = 777590.94 kB/sec 2024-04-25T01:34:22.728 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 initial writers = 777454.32 kB/sec 2024-04-25T01:34:22.728 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 777590.94 kB/sec 2024-04-25T01:34:22.728 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 777590.94 kB/sec 2024-04-25T01:34:22.728 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 777590.94 kB/sec 2024-04-25T01:34:22.728 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 1048576.00 kB 2024-04-25T01:34:23.958 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:23 smithi130 ceph-mon[32702]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.6 KiB/s wr, 14 op/s 2024-04-25T01:34:23.983 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:23 smithi032 ceph-mon[32461]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.6 KiB/s wr, 14 op/s 2024-04-25T01:34:24.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:23 smithi028 ceph-mon[29390]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.6 KiB/s wr, 14 op/s 2024-04-25T01:34:25.957 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:25 smithi130 ceph-mon[32702]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 4.5 KiB/s wr, 9 op/s 2024-04-25T01:34:25.982 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:25 smithi032 ceph-mon[32461]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 4.5 KiB/s wr, 9 op/s 2024-04-25T01:34:26.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:25 smithi028 ceph-mon[29390]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 4.5 KiB/s wr, 9 op/s 2024-04-25T01:34:27.942 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:27.942 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 rewriters = 870232.38 kB/sec 2024-04-25T01:34:27.942 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 rewriters = 870108.10 kB/sec 2024-04-25T01:34:27.942 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 870232.38 kB/sec 2024-04-25T01:34:27.942 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 870232.38 kB/sec 2024-04-25T01:34:27.942 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 870232.38 kB/sec 2024-04-25T01:34:27.943 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 1048576.00 kB 2024-04-25T01:34:28.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:27 smithi130 ceph-mon[32702]: pgmap v46: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 85 MiB/s wr, 35 op/s 2024-04-25T01:34:28.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:27 smithi032 ceph-mon[32461]: pgmap v46: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 85 MiB/s wr, 35 op/s 2024-04-25T01:34:28.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:27 smithi028 ceph-mon[29390]: pgmap v46: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 85 MiB/s wr, 35 op/s 2024-04-25T01:34:30.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:29 smithi130 ceph-mon[32702]: pgmap v47: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 85 MiB/s wr, 31 op/s 2024-04-25T01:34:30.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:29 smithi032 ceph-mon[32461]: pgmap v47: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 85 MiB/s wr, 31 op/s 2024-04-25T01:34:30.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:29 smithi028 ceph-mon[29390]: pgmap v47: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 85 MiB/s wr, 31 op/s 2024-04-25T01:34:30.776 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:30.776 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 readers = 6714238.50 kB/sec 2024-04-25T01:34:30.776 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 readers = 6705005.55 kB/sec 2024-04-25T01:34:30.776 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 6714238.50 kB/sec 2024-04-25T01:34:30.776 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 6714238.50 kB/sec 2024-04-25T01:34:30.776 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 6714238.50 kB/sec 2024-04-25T01:34:30.776 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 1048576.00 kB 2024-04-25T01:34:32.207 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:31 smithi130 ceph-mon[32702]: pgmap v48: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 85 MiB/s wr, 31 op/s 2024-04-25T01:34:32.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:31 smithi032 ceph-mon[32461]: pgmap v48: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 85 MiB/s wr, 31 op/s 2024-04-25T01:34:32.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:31 smithi028 ceph-mon[29390]: pgmap v48: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 85 MiB/s wr, 31 op/s 2024-04-25T01:34:33.013 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:33.014 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 re-readers = 6970064.50 kB/sec 2024-04-25T01:34:33.014 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 re-readers = 6959476.07 kB/sec 2024-04-25T01:34:33.014 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 6970064.50 kB/sec 2024-04-25T01:34:33.014 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 6970064.50 kB/sec 2024-04-25T01:34:33.014 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 6970064.50 kB/sec 2024-04-25T01:34:33.014 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 1048576.00 kB 2024-04-25T01:34:34.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:33 smithi130 ceph-mon[32702]: pgmap v49: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 57 op/s 2024-04-25T01:34:34.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:33 smithi130 ceph-mon[32702]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-04-25T01:34:34.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:33 smithi032 ceph-mon[32461]: pgmap v49: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 57 op/s 2024-04-25T01:34:34.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:33 smithi032 ceph-mon[32461]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-04-25T01:34:34.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:33 smithi028 ceph-mon[29390]: pgmap v49: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 57 op/s 2024-04-25T01:34:34.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:33 smithi028 ceph-mon[29390]: mgrmap e25: x(active, since 92s), standbys: z, y 2024-04-25T01:34:36.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:35 smithi130 ceph-mon[32702]: pgmap v50: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 171 MiB/s wr, 52 op/s 2024-04-25T01:34:36.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:35 smithi032 ceph-mon[32461]: pgmap v50: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 171 MiB/s wr, 52 op/s 2024-04-25T01:34:36.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:35 smithi028 ceph-mon[29390]: pgmap v50: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 171 MiB/s wr, 52 op/s 2024-04-25T01:34:38.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:37 smithi130 ceph-mon[32702]: pgmap v51: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 56 op/s 2024-04-25T01:34:38.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:37 smithi032 ceph-mon[32461]: pgmap v51: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 56 op/s 2024-04-25T01:34:38.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:37 smithi028 ceph-mon[29390]: pgmap v51: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 171 MiB/s wr, 56 op/s 2024-04-25T01:34:39.312 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:39.312 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:39.312 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:39.312 INFO:tasks.workunit.client.0.smithi028.stdout:iozone test complete. 2024-04-25T01:34:39.312 INFO:tasks.workunit.client.0.smithi028.stderr:+ iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-04-25T01:34:39.379 INFO:tasks.workunit.client.0.smithi028.stdout: Iozone: Performance Test of File I/O 2024-04-25T01:34:39.380 INFO:tasks.workunit.client.0.smithi028.stdout: Version $Revision: 3.494 $ 2024-04-25T01:34:39.380 INFO:tasks.workunit.client.0.smithi028.stdout: Compiled for 64 bit mode. 2024-04-25T01:34:39.380 INFO:tasks.workunit.client.0.smithi028.stdout: Build: linux-AMD64 2024-04-25T01:34:39.380 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:39.380 INFO:tasks.workunit.client.0.smithi028.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-04-25T01:34:39.380 INFO:tasks.workunit.client.0.smithi028.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-04-25T01:34:39.380 INFO:tasks.workunit.client.0.smithi028.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-04-25T01:34:39.380 INFO:tasks.workunit.client.0.smithi028.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-04-25T01:34:39.380 INFO:tasks.workunit.client.0.smithi028.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-04-25T01:34:39.380 INFO:tasks.workunit.client.0.smithi028.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-04-25T01:34:39.380 INFO:tasks.workunit.client.0.smithi028.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-04-25T01:34:39.380 INFO:tasks.workunit.client.0.smithi028.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-04-25T01:34:39.381 INFO:tasks.workunit.client.0.smithi028.stdout: Alexey Skidanov, Sudhir Kumar. 2024-04-25T01:34:39.381 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:39.381 INFO:tasks.workunit.client.0.smithi028.stdout: Run began: Thu Apr 25 01:34:39 2024 2024-04-25T01:34:39.381 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:39.381 INFO:tasks.workunit.client.0.smithi028.stdout: Include close in write timing 2024-04-25T01:34:39.381 INFO:tasks.workunit.client.0.smithi028.stdout: Include fsync in write timing 2024-04-25T01:34:39.381 INFO:tasks.workunit.client.0.smithi028.stdout: File size set to 1048576 kB 2024-04-25T01:34:39.381 INFO:tasks.workunit.client.0.smithi028.stdout: Record Size 1024 kB 2024-04-25T01:34:39.381 INFO:tasks.workunit.client.0.smithi028.stdout: Command line used: iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 2024-04-25T01:34:39.381 INFO:tasks.workunit.client.0.smithi028.stdout: Output is in kBytes/sec 2024-04-25T01:34:39.381 INFO:tasks.workunit.client.0.smithi028.stdout: Time Resolution = 0.000001 seconds. 2024-04-25T01:34:39.381 INFO:tasks.workunit.client.0.smithi028.stdout: Processor cache size set to 1024 kBytes. 2024-04-25T01:34:39.382 INFO:tasks.workunit.client.0.smithi028.stdout: Processor cache line size set to 32 bytes. 2024-04-25T01:34:39.382 INFO:tasks.workunit.client.0.smithi028.stdout: File stride size set to 17 * record size. 2024-04-25T01:34:39.382 INFO:tasks.workunit.client.0.smithi028.stdout: Throughput test with 1 process 2024-04-25T01:34:39.382 INFO:tasks.workunit.client.0.smithi028.stdout: Each process writes a 1048576 kByte file in 1024 kByte records 2024-04-25T01:34:40.207 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:39 smithi130 ceph-mon[32702]: pgmap v52: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 MiB/s wr, 30 op/s 2024-04-25T01:34:40.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:39 smithi032 ceph-mon[32461]: pgmap v52: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 MiB/s wr, 30 op/s 2024-04-25T01:34:40.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:39 smithi028 ceph-mon[29390]: pgmap v52: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 MiB/s wr, 30 op/s 2024-04-25T01:34:40.714 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:40.714 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 initial writers = 807220.38 kB/sec 2024-04-25T01:34:40.714 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 initial writers = 807062.63 kB/sec 2024-04-25T01:34:40.714 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 807220.38 kB/sec 2024-04-25T01:34:40.714 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 807220.38 kB/sec 2024-04-25T01:34:40.714 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 807220.38 kB/sec 2024-04-25T01:34:40.714 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 1048576.00 kB 2024-04-25T01:34:41.933 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:34:41.933 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:34:42.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.076+0000 7f9802122640 1 -- 172.21.15.28:0/1657281779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97fc06a780 msgr2=0x7f97fc10e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.076+0000 7f9802122640 1 --2- 172.21.15.28:0/1657281779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97fc06a780 0x7f97fc10e5f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009cd0 tx=0x7f97f402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.076+0000 7f9802122640 1 -- 172.21.15.28:0/1657281779 shutdown_connections 2024-04-25T01:34:42.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.076+0000 7f9802122640 1 --2- 172.21.15.28:0/1657281779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97fc06a780 0x7f97fc10e5f0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.076+0000 7f9802122640 1 --2- 172.21.15.28:0/1657281779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97fc069dc0 0x7f97fc06a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.076+0000 7f9802122640 1 --2- 172.21.15.28:0/1657281779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97fc107730 0x7f97fc107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.076+0000 7f9802122640 1 -- 172.21.15.28:0/1657281779 >> 172.21.15.28:0/1657281779 conn(0x7f97fc0776a0 msgr2=0x7f97fc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.076+0000 7f9802122640 1 -- 172.21.15.28:0/1657281779 shutdown_connections 2024-04-25T01:34:42.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.076+0000 7f9802122640 1 -- 172.21.15.28:0/1657281779 wait complete. 2024-04-25T01:34:42.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.076+0000 7f9802122640 1 Processor -- start 2024-04-25T01:34:42.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.077+0000 7f9802122640 1 -- start start 2024-04-25T01:34:42.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.077+0000 7f9802122640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97fc069dc0 0x7f97fc100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.077+0000 7f9802122640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97fc06a780 0x7f97fc100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.077+0000 7f9802122640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97fc107730 0x7f97fc1010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.077+0000 7f9802122640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc111b00 con 0x7f97fc06a780 2024-04-25T01:34:42.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.077+0000 7f9802122640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc111980 con 0x7f97fc069dc0 2024-04-25T01:34:42.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.077+0000 7f9802122640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc111c80 con 0x7f97fc107730 2024-04-25T01:34:42.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.077+0000 7f97fb7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97fc069dc0 0x7f97fc100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.077+0000 7f97faffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97fc06a780 0x7f97fc100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.077+0000 7f97fb7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97fc069dc0 0x7f97fc100670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:39744/0 (socket says 172.21.15.28:39744) 2024-04-25T01:34:42.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.077+0000 7f97fb7fe640 1 -- 172.21.15.28:0/2753040756 learned_addr learned my addr 172.21.15.28:0/2753040756 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.077+0000 7f97fbfff640 1 --2- 172.21.15.28:0/2753040756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97fc107730 0x7f97fc1010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.077+0000 7f97faffd640 1 -- 172.21.15.28:0/2753040756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97fc107730 msgr2=0x7f97fc1010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.077+0000 7f97faffd640 1 --2- 172.21.15.28:0/2753040756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97fc107730 0x7f97fc1010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.077+0000 7f97faffd640 1 -- 172.21.15.28:0/2753040756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97fc069dc0 msgr2=0x7f97fc100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.077+0000 7f97faffd640 1 --2- 172.21.15.28:0/2753040756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97fc069dc0 0x7f97fc100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.077+0000 7f97faffd640 1 -- 172.21.15.28:0/2753040756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97fc1a6460 con 0x7f97fc06a780 2024-04-25T01:34:42.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.077+0000 7f97fb7fe640 1 --2- 172.21.15.28:0/2753040756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97fc069dc0 0x7f97fc100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:42.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.077+0000 7f97fbfff640 1 --2- 172.21.15.28:0/2753040756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97fc107730 0x7f97fc1010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:42.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.078+0000 7f97faffd640 1 --2- 172.21.15.28:0/2753040756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97fc06a780 0x7f97fc100bb0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f97f000d6d0 tx=0x7f97f000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.078+0000 7f97f8ff9640 1 -- 172.21.15.28:0/2753040756 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0017070 con 0x7f97fc06a780 2024-04-25T01:34:42.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.078+0000 7f9802122640 1 -- 172.21.15.28:0/2753040756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97fc1a6750 con 0x7f97fc06a780 2024-04-25T01:34:42.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.078+0000 7f97f8ff9640 1 -- 172.21.15.28:0/2753040756 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97f0009820 con 0x7f97fc06a780 2024-04-25T01:34:42.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.078+0000 7f97f8ff9640 1 -- 172.21.15.28:0/2753040756 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0004cd0 con 0x7f97fc06a780 2024-04-25T01:34:42.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.078+0000 7f9802122640 1 -- 172.21.15.28:0/2753040756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97fc1a6ca0 con 0x7f97fc06a780 2024-04-25T01:34:42.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.080+0000 7f97f8ff9640 1 -- 172.21.15.28:0/2753040756 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f97f00099d0 con 0x7f97fc06a780 2024-04-25T01:34:42.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.081+0000 7f9804e32740 1 Processor -- start 2024-04-25T01:34:42.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.081+0000 7f9804e32740 1 -- start start 2024-04-25T01:34:42.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.081+0000 7f9804e32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444f630 0x563ac444fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.081+0000 7f9804e32740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444ff50 0x563ac445a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.081+0000 7f9804e32740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac445a620 0x563ac445ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.081+0000 7f9804e32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563ac43b46b0 con 0x563ac444f630 2024-04-25T01:34:42.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.081+0000 7f9804e32740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563ac435c880 con 0x563ac445a620 2024-04-25T01:34:42.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.081+0000 7f9804e32740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x563ac43b0670 con 0x563ac444ff50 2024-04-25T01:34:42.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.081+0000 7f97fbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444f630 0x563ac444fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.081+0000 7f97fbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444f630 0x563ac444fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41766/0 (socket says 172.21.15.28:41766) 2024-04-25T01:34:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.081+0000 7f97fbfff640 1 -- 172.21.15.28:0/503387142 learned_addr learned my addr 172.21.15.28:0/503387142 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.081+0000 7f97fb7fe640 1 --2- 172.21.15.28:0/503387142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444ff50 0x563ac445a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.081+0000 7f97f8ff9640 1 --2- 172.21.15.28:0/2753040756 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97e40a2d00 0x7f97e40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.081+0000 7f97faffd640 1 --2- 172.21.15.28:0/503387142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac445a620 0x563ac445ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.081+0000 7f97fbfff640 1 -- 172.21.15.28:0/503387142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444ff50 msgr2=0x563ac445a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.081+0000 7f97fbfff640 1 --2- 172.21.15.28:0/503387142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444ff50 0x563ac445a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.081+0000 7f97fbfff640 1 -- 172.21.15.28:0/503387142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac445a620 msgr2=0x563ac445ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.081+0000 7f97fbfff640 1 --2- 172.21.15.28:0/503387142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac445a620 0x563ac445ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.081+0000 7f97fbfff640 1 -- 172.21.15.28:0/503387142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ac41927d0 con 0x563ac444f630 2024-04-25T01:34:42.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.082+0000 7f97f8ff9640 1 -- 172.21.15.28:0/2753040756 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f97f00cf390 con 0x7f97fc06a780 2024-04-25T01:34:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.082+0000 7f97fbfff640 1 --2- 172.21.15.28:0/2753040756 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97e40a2d00 0x7f97e40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.082+0000 7f97fbfff640 1 --2- 172.21.15.28:0/503387142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444f630 0x563ac444fa10 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009430 tx=0x7f97f4005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ff48ddec17befb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.082+0000 7f97e1ffb640 1 -- 172.21.15.28:0/503387142 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f4031ba0 con 0x563ac444f630 2024-04-25T01:34:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.082+0000 7f97e1ffb640 1 -- 172.21.15.28:0/503387142 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97f4036040 con 0x563ac444f630 2024-04-25T01:34:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.082+0000 7f97e1ffb640 1 -- 172.21.15.28:0/503387142 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f4030810 con 0x563ac444f630 2024-04-25T01:34:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.083+0000 7f9804e32740 1 -- 172.21.15.28:0/503387142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444f630 msgr2=0x563ac444fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.083+0000 7f9804e32740 1 --2- 172.21.15.28:0/503387142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444f630 0x563ac444fa10 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009430 tx=0x7f97f4005fa0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.083+0000 7f97fbfff640 1 --2- 172.21.15.28:0/2753040756 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97e40a2d00 0x7f97e40a51c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f97f404b000 tx=0x7f97f4048000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.083+0000 7f9804e32740 1 -- 172.21.15.28:0/503387142 shutdown_connections 2024-04-25T01:34:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.083+0000 7f9804e32740 1 --2- 172.21.15.28:0/503387142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac445a620 0x563ac445ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.083+0000 7f9804e32740 1 --2- 172.21.15.28:0/503387142 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444ff50 0x563ac445a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.083+0000 7f9804e32740 1 --2- 172.21.15.28:0/503387142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444f630 0x563ac444fa10 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.083+0000 7f9804e32740 1 -- 172.21.15.28:0/503387142 >> 172.21.15.28:0/503387142 conn(0x563ac444b950 msgr2=0x563ac444bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.083+0000 7f9804e32740 1 -- 172.21.15.28:0/503387142 shutdown_connections 2024-04-25T01:34:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.083+0000 7f9804e32740 1 -- 172.21.15.28:0/503387142 wait complete. 2024-04-25T01:34:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.083+0000 7f9804e32740 1 -- start start 2024-04-25T01:34:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.083+0000 7f9804e32740 1 Processor -- start 2024-04-25T01:34:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.084+0000 7f9804e32740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444f630 0x563ac4457590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.084+0000 7f9804e32740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac444ff50 0x563ac4457ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.084+0000 7f9804e32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac445a620 0x563ac4458010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.084+0000 7f9804e32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563ac445d410 con 0x563ac445a620 2024-04-25T01:34:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.084+0000 7f9804e32740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563ac436ff50 con 0x563ac444ff50 2024-04-25T01:34:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.084+0000 7f9804e32740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x563ac445d590 con 0x563ac444f630 2024-04-25T01:34:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.084+0000 7f9804e32740 10 client.0 _ll_register_callbacks cb 0x563ac43ab000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.084+0000 7f97fbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac445a620 0x563ac4458010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.084+0000 7f97fbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac445a620 0x563ac4458010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41774/0 (socket says 172.21.15.28:41774) 2024-04-25T01:34:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.084+0000 7f97fbfff640 1 -- 172.21.15.28:0/1759431308 learned_addr learned my addr 172.21.15.28:0/1759431308 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.084+0000 7f97faffd640 1 --2- 172.21.15.28:0/1759431308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac444ff50 0x563ac4457ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.084+0000 7f97fb7fe640 1 --2- 172.21.15.28:0/1759431308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444f630 0x563ac4457590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.084+0000 7f97fbfff640 1 -- 172.21.15.28:0/1759431308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444f630 msgr2=0x563ac4457590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.084+0000 7f97fbfff640 1 --2- 172.21.15.28:0/1759431308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444f630 0x563ac4457590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.084+0000 7f97fbfff640 1 -- 172.21.15.28:0/1759431308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac444ff50 msgr2=0x563ac4457ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.084+0000 7f97fbfff640 1 --2- 172.21.15.28:0/1759431308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac444ff50 0x563ac4457ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.084+0000 7f97fbfff640 1 -- 172.21.15.28:0/1759431308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ac40c83f0 con 0x563ac445a620 2024-04-25T01:34:42.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.084+0000 7f97fbfff640 1 --2- 172.21.15.28:0/1759431308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac445a620 0x563ac4458010 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f97f402ae00 tx=0x7f97f4004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8eb196e41fcb6e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.084+0000 7f97daffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:42.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.085+0000 7f97daffd640 1 -- 172.21.15.28:0/1759431308 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f405f060 con 0x563ac445a620 2024-04-25T01:34:42.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.085+0000 7f97daffd640 1 -- 172.21.15.28:0/1759431308 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f97f4030a70 con 0x563ac445a620 2024-04-25T01:34:42.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.085+0000 7f9804e32740 1 -- 172.21.15.28:0/1759431308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97f4036040 con 0x563ac445a620 2024-04-25T01:34:42.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.085+0000 7f97daffd640 1 -- 172.21.15.28:0/1759431308 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f97f4066050 con 0x563ac445a620 2024-04-25T01:34:42.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.085+0000 7f9804e32740 10 client.16131 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:42.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.085+0000 7f9804e32740 1 -- 172.21.15.28:0/1759431308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563ac40880f0 con 0x563ac445a620 2024-04-25T01:34:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.085+0000 7f97daffd640 1 -- 172.21.15.28:0/1759431308 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97f4036040 con 0x563ac445a620 2024-04-25T01:34:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.085+0000 7f97daffd640 1 -- 172.21.15.28:0/1759431308 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f97f4030070 con 0x563ac445a620 2024-04-25T01:34:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.085+0000 7f9804e32740 10 client.16131 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.085+0000 7f9804e32740 10 client.16131 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.085+0000 7f9804e32740 20 client.16131 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.085+0000 7f9804e32740 1 --2- 172.21.15.28:0/1759431308 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563ac446a760 0x563ac446cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.085+0000 7f9804e32740 4 client.16131 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.085+0000 7f9804e32740 1 -- 172.21.15.28:0/1759431308 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ac43c1bf0 con 0x563ac446a760 2024-04-25T01:34:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.086+0000 7f97fb7fe640 1 --2- 172.21.15.28:0/1759431308 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563ac446a760 0x563ac446cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.086+0000 7f97fb7fe640 1 --2- 172.21.15.28:0/1759431308 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563ac446a760 0x563ac446cb50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfd1edffdf7dab3 server_cookie=b53b1df1a36c0365 in_seq=0 out_seq=0 2024-04-25T01:34:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.086+0000 7f97daffd640 10 client.16131 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.087+0000 7f97daffd640 1 -- 172.21.15.28:0/1759431308 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563ac43c1bf0 con 0x563ac446a760 2024-04-25T01:34:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.087+0000 7f97daffd640 10 client.16131 handle_command_reply: tid=0 2024-04-25T01:34:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.087+0000 7f9804e32740 1 client.16131 shutdown 2024-04-25T01:34:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.087+0000 7f9804e32740 1 -- 172.21.15.28:0/1759431308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac445a620 msgr2=0x563ac4458010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.087+0000 7f9804e32740 1 --2- 172.21.15.28:0/1759431308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac445a620 0x563ac4458010 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f97f402ae00 tx=0x7f97f4004960 comp rx=0 tx=0).stop 2024-04-25T01:34:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.087+0000 7f9804e32740 1 -- 172.21.15.28:0/1759431308 shutdown_connections 2024-04-25T01:34:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.087+0000 7f9804e32740 1 --2- 172.21.15.28:0/1759431308 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563ac446a760 0x563ac446cb50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.087+0000 7f9804e32740 1 --2- 172.21.15.28:0/1759431308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac445a620 0x563ac4458010 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.087+0000 7f9804e32740 1 --2- 172.21.15.28:0/1759431308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac444ff50 0x563ac4457ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.087+0000 7f97daffd640 0 client.16131 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.087+0000 7f9804e32740 1 --2- 172.21.15.28:0/1759431308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444f630 0x563ac4457590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.087+0000 7f9804e32740 1 -- 172.21.15.28:0/1759431308 >> 172.21.15.28:0/1759431308 conn(0x563ac444b950 msgr2=0x563ac4451750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.087+0000 7f9804e32740 1 -- 172.21.15.28:0/1759431308 shutdown_connections 2024-04-25T01:34:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.087+0000 7f9804e32740 1 -- 172.21.15.28:0/1759431308 wait complete. 2024-04-25T01:34:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.087+0000 7f9804e32740 20 client.16131 trim_cache size 0 max 16384 2024-04-25T01:34:42.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f9804e32740 1 Processor -- start 2024-04-25T01:34:42.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f9804e32740 1 -- start start 2024-04-25T01:34:42.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f9804e32740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444f630 0x563ac44662d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f9804e32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444ff50 0x563ac444bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f9804e32740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac445a620 0x563ac4457e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f9804e32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc1b0c00 con 0x563ac444ff50 2024-04-25T01:34:42.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f9804e32740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc1b2740 con 0x563ac445a620 2024-04-25T01:34:42.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f9804e32740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc1b12d0 con 0x563ac444f630 2024-04-25T01:34:42.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f97faffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444ff50 0x563ac444bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f97faffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444ff50 0x563ac444bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41790/0 (socket says 172.21.15.28:41790) 2024-04-25T01:34:42.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f97faffd640 1 -- 172.21.15.28:0/106751846 learned_addr learned my addr 172.21.15.28:0/106751846 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f97fb7fe640 1 --2- 172.21.15.28:0/106751846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444f630 0x563ac44662d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f97faffd640 1 -- 172.21.15.28:0/106751846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444f630 msgr2=0x563ac44662d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f97faffd640 1 --2- 172.21.15.28:0/106751846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444f630 0x563ac44662d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f97fbfff640 1 --2- 172.21.15.28:0/106751846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac445a620 0x563ac4457e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f97faffd640 1 -- 172.21.15.28:0/106751846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac445a620 msgr2=0x563ac4457e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f97faffd640 1 --2- 172.21.15.28:0/106751846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac445a620 0x563ac4457e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f97faffd640 1 -- 172.21.15.28:0/106751846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ac444fd80 con 0x563ac444ff50 2024-04-25T01:34:42.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f97fb7fe640 1 --2- 172.21.15.28:0/106751846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444f630 0x563ac44662d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:42.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f97fbfff640 1 --2- 172.21.15.28:0/106751846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac445a620 0x563ac4457e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:42.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.105+0000 7f97faffd640 1 --2- 172.21.15.28:0/106751846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444ff50 0x563ac444bd30 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f97f0009610 tx=0x7f97f00d17e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21bcb9d353a555e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.106+0000 7f97e17fa640 1 -- 172.21.15.28:0/106751846 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0020040 con 0x563ac444ff50 2024-04-25T01:34:42.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.106+0000 7f97e17fa640 1 -- 172.21.15.28:0/106751846 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97f00a4180 con 0x563ac444ff50 2024-04-25T01:34:42.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.106+0000 7f97e17fa640 1 -- 172.21.15.28:0/106751846 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f00cc9c0 con 0x563ac444ff50 2024-04-25T01:34:42.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.106+0000 7f9804e32740 1 -- 172.21.15.28:0/106751846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444ff50 msgr2=0x563ac444bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.106+0000 7f9804e32740 1 --2- 172.21.15.28:0/106751846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444ff50 0x563ac444bd30 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f97f0009610 tx=0x7f97f00d17e0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.106+0000 7f9804e32740 1 -- 172.21.15.28:0/106751846 shutdown_connections 2024-04-25T01:34:42.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.106+0000 7f9804e32740 1 --2- 172.21.15.28:0/106751846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac445a620 0x563ac4457e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.106+0000 7f9804e32740 1 --2- 172.21.15.28:0/106751846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444ff50 0x563ac444bd30 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.106+0000 7f9804e32740 1 --2- 172.21.15.28:0/106751846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444f630 0x563ac44662d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.106+0000 7f9804e32740 1 -- 172.21.15.28:0/106751846 >> 172.21.15.28:0/106751846 conn(0x563ac444b950 msgr2=0x563ac444a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.106+0000 7f9804e32740 1 -- 172.21.15.28:0/106751846 shutdown_connections 2024-04-25T01:34:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.106+0000 7f9804e32740 1 -- 172.21.15.28:0/106751846 wait complete. 2024-04-25T01:34:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.107+0000 7f9804e32740 1 -- start start 2024-04-25T01:34:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.107+0000 7f9804e32740 1 Processor -- start 2024-04-25T01:34:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.107+0000 7f9804e32740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444f630 0x563ac445da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.107+0000 7f9804e32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444ff50 0x563ac445df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.107+0000 7f9804e32740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac445a620 0x563ac44c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.107+0000 7f9804e32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc1b2070 con 0x563ac444ff50 2024-04-25T01:34:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.107+0000 7f9804e32740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc1b19a0 con 0x563ac445a620 2024-04-25T01:34:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.107+0000 7f9804e32740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc1aebe0 con 0x563ac444f630 2024-04-25T01:34:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.107+0000 7f9804e32740 10 client.0 _ll_register_callbacks cb 0x7f97f4053040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.107+0000 7f97faffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444ff50 0x563ac445df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.107+0000 7f97faffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444ff50 0x563ac445df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41806/0 (socket says 172.21.15.28:41806) 2024-04-25T01:34:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.107+0000 7f97fbfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac445a620 0x563ac44c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.107+0000 7f97faffd640 1 -- 172.21.15.28:0/2558290119 learned_addr learned my addr 172.21.15.28:0/2558290119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.107+0000 7f97fb7fe640 1 --2- 172.21.15.28:0/2558290119 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444f630 0x563ac445da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.108+0000 7f97fbfff640 1 -- 172.21.15.28:0/2558290119 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444f630 msgr2=0x563ac445da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.108+0000 7f97fbfff640 1 --2- 172.21.15.28:0/2558290119 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444f630 0x563ac445da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.108+0000 7f97fbfff640 1 -- 172.21.15.28:0/2558290119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444ff50 msgr2=0x563ac445df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.108+0000 7f97fbfff640 1 --2- 172.21.15.28:0/2558290119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444ff50 0x563ac445df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.108+0000 7f97fbfff640 1 -- 172.21.15.28:0/2558290119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ac44c8f40 con 0x563ac445a620 2024-04-25T01:34:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.108+0000 7f97faffd640 1 --2- 172.21.15.28:0/2558290119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444ff50 0x563ac445df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.108+0000 7f97fbfff640 1 --2- 172.21.15.28:0/2558290119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac445a620 0x563ac44c8980 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009430 tx=0x7f97f4030d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea916ca538a59970 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.108+0000 7f97daffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-25T01:34:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.108+0000 7f97daffd640 1 -- 172.21.15.28:0/2558290119 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f406f060 con 0x563ac445a620 2024-04-25T01:34:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.108+0000 7f9804e32740 1 -- 172.21.15.28:0/2558290119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97f00a4180 con 0x563ac445a620 2024-04-25T01:34:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.108+0000 7f97daffd640 1 -- 172.21.15.28:0/2558290119 <== mon.2 v2:172.21.15.32:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f97f4004040 con 0x563ac445a620 2024-04-25T01:34:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.108+0000 7f97daffd640 1 -- 172.21.15.28:0/2558290119 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f97f4076050 con 0x563ac445a620 2024-04-25T01:34:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.108+0000 7f9804e32740 10 client.25643 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.108+0000 7f9804e32740 1 -- 172.21.15.28:0/2558290119 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563ac44d0270 con 0x563ac445a620 2024-04-25T01:34:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.108+0000 7f97daffd640 1 -- 172.21.15.28:0/2558290119 <== mon.2 v2:172.21.15.32:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97f00a4180 con 0x563ac445a620 2024-04-25T01:34:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.109+0000 7f97daffd640 1 -- 172.21.15.28:0/2558290119 <== mon.2 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f97f405a750 con 0x563ac445a620 2024-04-25T01:34:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.109+0000 7f9804e32740 10 client.25643 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.109+0000 7f9804e32740 10 client.25643 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.109+0000 7f9804e32740 20 client.25643 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.109+0000 7f9804e32740 1 --2- 172.21.15.28:0/2558290119 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563ac44d0930 0x563ac44d2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.109+0000 7f9804e32740 4 client.25643 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:34:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.109+0000 7f9804e32740 1 -- 172.21.15.28:0/2558290119 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563ac43bbf40 con 0x563ac44d0930 2024-04-25T01:34:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.109+0000 7f97fb7fe640 1 --2- 172.21.15.28:0/2558290119 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563ac44d0930 0x563ac44d2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.109+0000 7f97daffd640 10 client.25643 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:42.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.109+0000 7f97fb7fe640 1 --2- 172.21.15.28:0/2558290119 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563ac44d0930 0x563ac44d2d20 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18f7c9fd83fd770a server_cookie=9cfb018552582da8 in_seq=0 out_seq=0 2024-04-25T01:34:42.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.109+0000 7f97daffd640 1 -- 172.21.15.28:0/2558290119 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563ac43bbf40 con 0x563ac44d0930 2024-04-25T01:34:42.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.109+0000 7f97daffd640 10 client.25643 handle_command_reply: tid=0 2024-04-25T01:34:42.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.109+0000 7f9804e32740 1 client.25643 shutdown 2024-04-25T01:34:42.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9804e32740 1 -- 172.21.15.28:0/2558290119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac445a620 msgr2=0x563ac44c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9804e32740 1 --2- 172.21.15.28:0/2558290119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac445a620 0x563ac44c8980 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009430 tx=0x7f97f4030d20 comp rx=0 tx=0).stop 2024-04-25T01:34:42.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9804e32740 1 -- 172.21.15.28:0/2558290119 shutdown_connections 2024-04-25T01:34:42.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9804e32740 1 --2- 172.21.15.28:0/2558290119 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563ac44d0930 0x563ac44d2d20 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9804e32740 1 --2- 172.21.15.28:0/2558290119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ac445a620 0x563ac44c8980 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9804e32740 1 --2- 172.21.15.28:0/2558290119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ac444ff50 0x563ac445df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9804e32740 1 --2- 172.21.15.28:0/2558290119 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ac444f630 0x563ac445da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f97daffd640 0 client.25643 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:42.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9804e32740 1 -- 172.21.15.28:0/2558290119 >> 172.21.15.28:0/2558290119 conn(0x563ac444b950 msgr2=0x563ac444a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9804e32740 1 -- 172.21.15.28:0/2558290119 shutdown_connections 2024-04-25T01:34:42.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9804e32740 1 -- 172.21.15.28:0/2558290119 wait complete. 2024-04-25T01:34:42.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9804e32740 20 client.25643 trim_cache size 0 max 16384 2024-04-25T01:34:42.093 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:34:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:34:42.093 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:34:42.093 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:34:42.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9802122640 1 -- 172.21.15.28:0/2753040756 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97e40a2d00 msgr2=0x7f97e40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9802122640 1 --2- 172.21.15.28:0/2753040756 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97e40a2d00 0x7f97e40a51c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f97f404b000 tx=0x7f97f4048000 comp rx=0 tx=0).stop 2024-04-25T01:34:42.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9802122640 1 -- 172.21.15.28:0/2753040756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97fc06a780 msgr2=0x7f97fc100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9802122640 1 --2- 172.21.15.28:0/2753040756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97fc06a780 0x7f97fc100bb0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f97f000d6d0 tx=0x7f97f000db90 comp rx=0 tx=0).stop 2024-04-25T01:34:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9802122640 1 -- 172.21.15.28:0/2753040756 shutdown_connections 2024-04-25T01:34:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9802122640 1 --2- 172.21.15.28:0/2753040756 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97e40a2d00 0x7f97e40a51c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9802122640 1 --2- 172.21.15.28:0/2753040756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97fc107730 0x7f97fc1010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9802122640 1 --2- 172.21.15.28:0/2753040756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97fc06a780 0x7f97fc100bb0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9802122640 1 --2- 172.21.15.28:0/2753040756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97fc069dc0 0x7f97fc100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9802122640 1 -- 172.21.15.28:0/2753040756 >> 172.21.15.28:0/2753040756 conn(0x7f97fc0776a0 msgr2=0x7f97fc0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9802122640 1 -- 172.21.15.28:0/2753040756 shutdown_connections 2024-04-25T01:34:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.110+0000 7f9802122640 1 -- 172.21.15.28:0/2753040756 wait complete. 2024-04-25T01:34:42.098 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:34:42.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:34:42.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:41 smithi028 ceph-mon[29390]: pgmap v53: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 MiB/s wr, 30 op/s 2024-04-25T01:34:42.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.224+0000 7fe18e21c640 1 -- 172.21.15.28:0/1028076450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188102160 msgr2=0x7fe18810e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.224+0000 7fe18e21c640 1 --2- 172.21.15.28:0/1028076450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188102160 0x7fe18810e5a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fe180009ca0 tx=0x7fe18002a950 comp rx=0 tx=0).stop 2024-04-25T01:34:42.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.224+0000 7fe18e21c640 1 -- 172.21.15.28:0/1028076450 shutdown_connections 2024-04-25T01:34:42.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.224+0000 7fe18e21c640 1 --2- 172.21.15.28:0/1028076450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188102160 0x7fe18810e5a0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.224+0000 7fe18e21c640 1 --2- 172.21.15.28:0/1028076450 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe1881017c0 0x7fe188101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.224+0000 7fe18e21c640 1 --2- 172.21.15.28:0/1028076450 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe188107750 0x7fe188107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.224+0000 7fe18e21c640 1 -- 172.21.15.28:0/1028076450 >> 172.21.15.28:0/1028076450 conn(0x7fe1880788d0 msgr2=0x7fe188078ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.224+0000 7fe18e21c640 1 -- 172.21.15.28:0/1028076450 shutdown_connections 2024-04-25T01:34:42.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.224+0000 7fe18e21c640 1 -- 172.21.15.28:0/1028076450 wait complete. 2024-04-25T01:34:42.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.225+0000 7fe18e21c640 1 Processor -- start 2024-04-25T01:34:42.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.225+0000 7fe18e21c640 1 -- start start 2024-04-25T01:34:42.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.225+0000 7fe18e21c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe1881017c0 0x7fe188198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.225+0000 7fe18e21c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188102160 0x7fe188198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.225+0000 7fe18e21c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe188107750 0x7fe18819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.225+0000 7fe18e21c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe188111b10 con 0x7fe188102160 2024-04-25T01:34:42.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.225+0000 7fe18e21c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe188111990 con 0x7fe188107750 2024-04-25T01:34:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.225+0000 7fe18e21c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe188111c90 con 0x7fe1881017c0 2024-04-25T01:34:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.225+0000 7fe186ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188102160 0x7fe188198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.225+0000 7fe186ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188102160 0x7fe188198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41814/0 (socket says 172.21.15.28:41814) 2024-04-25T01:34:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.225+0000 7fe186ffd640 1 -- 172.21.15.28:0/903031445 learned_addr learned my addr 172.21.15.28:0/903031445 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.225+0000 7fe187fff640 1 --2- 172.21.15.28:0/903031445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe188107750 0x7fe18819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.225+0000 7fe1877fe640 1 --2- 172.21.15.28:0/903031445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe1881017c0 0x7fe188198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.225+0000 7fe186ffd640 1 -- 172.21.15.28:0/903031445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe1881017c0 msgr2=0x7fe188198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.225+0000 7fe186ffd640 1 --2- 172.21.15.28:0/903031445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe1881017c0 0x7fe188198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.225+0000 7fe186ffd640 1 -- 172.21.15.28:0/903031445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe188107750 msgr2=0x7fe18819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.225+0000 7fe186ffd640 1 --2- 172.21.15.28:0/903031445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe188107750 0x7fe18819c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.225+0000 7fe186ffd640 1 -- 172.21.15.28:0/903031445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe18819d000 con 0x7fe188102160 2024-04-25T01:34:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.226+0000 7fe186ffd640 1 --2- 172.21.15.28:0/903031445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188102160 0x7fe188198550 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00c890 tx=0x7fe17c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.226+0000 7fe184ff9640 1 -- 172.21.15.28:0/903031445 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c004220 con 0x7fe188102160 2024-04-25T01:34:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.226+0000 7fe184ff9640 1 -- 172.21.15.28:0/903031445 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe17c0043c0 con 0x7fe188102160 2024-04-25T01:34:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.226+0000 7fe18e21c640 1 -- 172.21.15.28:0/903031445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe18819d2f0 con 0x7fe188102160 2024-04-25T01:34:42.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.226+0000 7fe184ff9640 1 -- 172.21.15.28:0/903031445 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c0049e0 con 0x7fe188102160 2024-04-25T01:34:42.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.226+0000 7fe18e21c640 1 -- 172.21.15.28:0/903031445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1881a4b80 con 0x7fe188102160 2024-04-25T01:34:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.227+0000 7fe18e21c640 1 -- 172.21.15.28:0/903031445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1880689b0 con 0x7fe188102160 2024-04-25T01:34:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.228+0000 7fe184ff9640 1 -- 172.21.15.28:0/903031445 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe17c0079b0 con 0x7fe188102160 2024-04-25T01:34:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.229+0000 7fe184ff9640 1 --2- 172.21.15.28:0/903031445 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe1700a2d00 0x7fe1700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.229+0000 7fe1877fe640 1 --2- 172.21.15.28:0/903031445 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe1700a2d00 0x7fe1700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.229+0000 7fe184ff9640 1 -- 172.21.15.28:0/903031445 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fe17c0cf7a0 con 0x7fe188102160 2024-04-25T01:34:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.229+0000 7fe1877fe640 1 --2- 172.21.15.28:0/903031445 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe1700a2d00 0x7fe1700a51c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe17800bb60 tx=0x7fe17800b390 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.207 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:41 smithi130 ceph-mon[32702]: pgmap v53: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 MiB/s wr, 30 op/s 2024-04-25T01:34:42.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.233+0000 7fe184ff9640 1 -- 172.21.15.28:0/903031445 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe17c014030 con 0x7fe188102160 2024-04-25T01:34:42.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:41 smithi032 ceph-mon[32461]: pgmap v53: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 MiB/s wr, 30 op/s 2024-04-25T01:34:42.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.459+0000 7fe18e21c640 1 -- 172.21.15.28:0/903031445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe188109c50 con 0x7fe188102160 2024-04-25T01:34:42.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.460+0000 7fe184ff9640 1 -- 172.21.15.28:0/903031445 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7fe17c09b9d0 con 0x7fe188102160 2024-04-25T01:34:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:34:42.437 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:34:42.437 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:34:42.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.462+0000 7fe18e21c640 1 -- 172.21.15.28:0/903031445 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe1700a2d00 msgr2=0x7fe1700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.462+0000 7fe18e21c640 1 --2- 172.21.15.28:0/903031445 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe1700a2d00 0x7fe1700a51c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe17800bb60 tx=0x7fe17800b390 comp rx=0 tx=0).stop 2024-04-25T01:34:42.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.462+0000 7fe18e21c640 1 -- 172.21.15.28:0/903031445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188102160 msgr2=0x7fe188198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.462+0000 7fe18e21c640 1 --2- 172.21.15.28:0/903031445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188102160 0x7fe188198550 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00c890 tx=0x7fe17c00cd50 comp rx=0 tx=0).stop 2024-04-25T01:34:42.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.462+0000 7fe18e21c640 1 -- 172.21.15.28:0/903031445 shutdown_connections 2024-04-25T01:34:42.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.462+0000 7fe18e21c640 1 --2- 172.21.15.28:0/903031445 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe1700a2d00 0x7fe1700a51c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.462+0000 7fe18e21c640 1 --2- 172.21.15.28:0/903031445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe188107750 0x7fe18819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.462+0000 7fe18e21c640 1 --2- 172.21.15.28:0/903031445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe188102160 0x7fe188198550 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.462+0000 7fe18e21c640 1 --2- 172.21.15.28:0/903031445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe1881017c0 0x7fe188198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.462+0000 7fe18e21c640 1 -- 172.21.15.28:0/903031445 >> 172.21.15.28:0/903031445 conn(0x7fe1880788d0 msgr2=0x7fe1881004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.462+0000 7fe18e21c640 1 -- 172.21.15.28:0/903031445 shutdown_connections 2024-04-25T01:34:42.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.462+0000 7fe18e21c640 1 -- 172.21.15.28:0/903031445 wait complete. 2024-04-25T01:34:42.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:34:42.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.619+0000 7f34342fc640 1 -- 172.21.15.28:0/1949941943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f342c106690 msgr2=0x7f342c106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.619+0000 7f34342fc640 1 --2- 172.21.15.28:0/1949941943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f342c106690 0x7f342c106a70 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f3428009900 tx=0x7f342802a410 comp rx=0 tx=0).stop 2024-04-25T01:34:42.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.619+0000 7f34342fc640 1 -- 172.21.15.28:0/1949941943 shutdown_connections 2024-04-25T01:34:42.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.619+0000 7f34342fc640 1 --2- 172.21.15.28:0/1949941943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f342c101330 0x7f342c10d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.619+0000 7f34342fc640 1 --2- 172.21.15.28:0/1949941943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f342c100990 0x7f342c100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.619+0000 7f34342fc640 1 --2- 172.21.15.28:0/1949941943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f342c106690 0x7f342c106a70 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.619+0000 7f34342fc640 1 -- 172.21.15.28:0/1949941943 >> 172.21.15.28:0/1949941943 conn(0x7f342c076440 msgr2=0x7f342c076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.620+0000 7f34342fc640 1 -- 172.21.15.28:0/1949941943 shutdown_connections 2024-04-25T01:34:42.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.620+0000 7f34342fc640 1 -- 172.21.15.28:0/1949941943 wait complete. 2024-04-25T01:34:42.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.620+0000 7f34342fc640 1 Processor -- start 2024-04-25T01:34:42.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.620+0000 7f34342fc640 1 -- start start 2024-04-25T01:34:42.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.620+0000 7f34342fc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f342c100990 0x7f342c196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.620+0000 7f34342fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f342c101330 0x7f342c1972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.620+0000 7f34342fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f342c106690 0x7f342c19b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.620+0000 7f34342fc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f342c1109e0 con 0x7f342c101330 2024-04-25T01:34:42.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.620+0000 7f34342fc640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f342c110860 con 0x7f342c106690 2024-04-25T01:34:42.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.620+0000 7f34342fc640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f342c110b60 con 0x7f342c100990 2024-04-25T01:34:42.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.621+0000 7f3431897640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f342c101330 0x7f342c1972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.621+0000 7f3431897640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f342c101330 0x7f342c1972e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41830/0 (socket says 172.21.15.28:41830) 2024-04-25T01:34:42.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.621+0000 7f3431897640 1 -- 172.21.15.28:0/1745661247 learned_addr learned my addr 172.21.15.28:0/1745661247 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.621+0000 7f3432098640 1 --2- 172.21.15.28:0/1745661247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f342c100990 0x7f342c196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.621+0000 7f3432899640 1 --2- 172.21.15.28:0/1745661247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f342c106690 0x7f342c19b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.621+0000 7f3431897640 1 -- 172.21.15.28:0/1745661247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f342c100990 msgr2=0x7f342c196da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.621+0000 7f3431897640 1 --2- 172.21.15.28:0/1745661247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f342c100990 0x7f342c196da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.621+0000 7f3431897640 1 -- 172.21.15.28:0/1745661247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f342c106690 msgr2=0x7f342c19b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.621+0000 7f3431897640 1 --2- 172.21.15.28:0/1745661247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f342c106690 0x7f342c19b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.621+0000 7f3431897640 1 -- 172.21.15.28:0/1745661247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f342c19bd90 con 0x7f342c101330 2024-04-25T01:34:42.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.621+0000 7f3431897640 1 --2- 172.21.15.28:0/1745661247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f342c101330 0x7f342c1972e0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f341c00c920 tx=0x7f341c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.621+0000 7f34237fe640 1 -- 172.21.15.28:0/1745661247 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c007c70 con 0x7f342c101330 2024-04-25T01:34:42.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.621+0000 7f34237fe640 1 -- 172.21.15.28:0/1745661247 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f341c007e10 con 0x7f342c101330 2024-04-25T01:34:42.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.621+0000 7f34342fc640 1 -- 172.21.15.28:0/1745661247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f342c19c080 con 0x7f342c101330 2024-04-25T01:34:42.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.621+0000 7f34237fe640 1 -- 172.21.15.28:0/1745661247 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c018890 con 0x7f342c101330 2024-04-25T01:34:42.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.621+0000 7f34342fc640 1 -- 172.21.15.28:0/1745661247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f342c1a3910 con 0x7f342c101330 2024-04-25T01:34:42.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.623+0000 7f34237fe640 1 -- 172.21.15.28:0/1745661247 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f341c018a70 con 0x7f342c101330 2024-04-25T01:34:42.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.624+0000 7f3437009740 1 Processor -- start 2024-04-25T01:34:42.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.624+0000 7f3437009740 1 -- start start 2024-04-25T01:34:42.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.624+0000 7f3437009740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565349191580 0x565349191960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.624+0000 7f34237fe640 1 --2- 172.21.15.28:0/1745661247 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f34180a2d80 0x7f34180a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.624+0000 7f3437009740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191ea0 0x56534919c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.624+0000 7f3437009740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56534919c570 0x56534919e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.624+0000 7f3437009740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5653490f6670 con 0x565349191ea0 2024-04-25T01:34:42.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.624+0000 7f3437009740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56534909e970 con 0x56534919c570 2024-04-25T01:34:42.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.624+0000 7f3437009740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5653490f2630 con 0x565349191580 2024-04-25T01:34:42.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.625+0000 7f3431897640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191ea0 0x56534919c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.625+0000 7f3432098640 1 --2- 172.21.15.28:0/1745661247 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f34180a2d80 0x7f34180a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.625+0000 7f3432899640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565349191580 0x565349191960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.625+0000 7f3431897640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191ea0 0x56534919c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41836/0 (socket says 172.21.15.28:41836) 2024-04-25T01:34:42.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.625+0000 7f3431897640 1 -- 172.21.15.28:0/1523498095 learned_addr learned my addr 172.21.15.28:0/1523498095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.625+0000 7f34237fe640 1 -- 172.21.15.28:0/1745661247 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f341c0cf800 con 0x7f342c101330 2024-04-25T01:34:42.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.625+0000 7f3432899640 1 --2- 172.21.15.28:0/1523498095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56534919c570 0x56534919e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.625+0000 7f3431897640 1 -- 172.21.15.28:0/1523498095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565349191580 msgr2=0x565349191960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.625+0000 7f3431897640 1 --2- 172.21.15.28:0/1523498095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565349191580 0x565349191960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.625+0000 7f3431897640 1 -- 172.21.15.28:0/1523498095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56534919c570 msgr2=0x56534919e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.625+0000 7f3431897640 1 --2- 172.21.15.28:0/1523498095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56534919c570 0x56534919e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.625+0000 7f3431897640 1 -- 172.21.15.28:0/1523498095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653490c2400 con 0x565349191ea0 2024-04-25T01:34:42.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.625+0000 7f3432899640 1 --2- 172.21.15.28:0/1523498095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565349191580 0x565349191960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:42.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.625+0000 7f3432098640 1 --2- 172.21.15.28:0/1745661247 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f34180a2d80 0x7f34180a5240 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3428009230 tx=0x7f34280058d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.625+0000 7f3431897640 1 --2- 172.21.15.28:0/1523498095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191ea0 0x56534919c030 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f341c098350 tx=0x7f341c06d940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec9a0c1b38f4381 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.625+0000 7f3420bfe640 1 -- 172.21.15.28:0/1523498095 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c099f40 con 0x565349191ea0 2024-04-25T01:34:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.625+0000 7f3420bfe640 1 -- 172.21.15.28:0/1523498095 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f341c09c080 con 0x565349191ea0 2024-04-25T01:34:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.625+0000 7f3420bfe640 1 -- 172.21.15.28:0/1523498095 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c098fd0 con 0x565349191ea0 2024-04-25T01:34:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.626+0000 7f3437009740 1 -- 172.21.15.28:0/1523498095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191ea0 msgr2=0x56534919c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.626+0000 7f3437009740 1 --2- 172.21.15.28:0/1523498095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191ea0 0x56534919c030 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f341c098350 tx=0x7f341c06d940 comp rx=0 tx=0).stop 2024-04-25T01:34:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.626+0000 7f3437009740 1 -- 172.21.15.28:0/1523498095 shutdown_connections 2024-04-25T01:34:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.626+0000 7f3437009740 1 --2- 172.21.15.28:0/1523498095 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56534919c570 0x56534919e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.626+0000 7f3437009740 1 --2- 172.21.15.28:0/1523498095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191ea0 0x56534919c030 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.626+0000 7f3437009740 1 --2- 172.21.15.28:0/1523498095 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565349191580 0x565349191960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.626+0000 7f3437009740 1 -- 172.21.15.28:0/1523498095 >> 172.21.15.28:0/1523498095 conn(0x56534918d8a0 msgr2=0x56534918dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.626+0000 7f3437009740 1 -- 172.21.15.28:0/1523498095 shutdown_connections 2024-04-25T01:34:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.626+0000 7f3437009740 1 -- 172.21.15.28:0/1523498095 wait complete. 2024-04-25T01:34:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.626+0000 7f3437009740 1 -- start start 2024-04-25T01:34:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.626+0000 7f3437009740 1 Processor -- start 2024-04-25T01:34:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3437009740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565349191580 0x56534919bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3437009740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191ea0 0x565349199230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3437009740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56534919c570 0x565349199840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3437009740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56534919f360 con 0x565349191ea0 2024-04-25T01:34:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3437009740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5653490b2050 con 0x56534919c570 2024-04-25T01:34:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3437009740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56534919f4e0 con 0x565349191580 2024-04-25T01:34:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3437009740 10 client.0 _ll_register_callbacks cb 0x565348e09be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3431897640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191ea0 0x565349199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3431897640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191ea0 0x565349199230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41848/0 (socket says 172.21.15.28:41848) 2024-04-25T01:34:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3431897640 1 -- 172.21.15.28:0/864066702 learned_addr learned my addr 172.21.15.28:0/864066702 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3432098640 1 --2- 172.21.15.28:0/864066702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565349191580 0x56534919bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3432899640 1 --2- 172.21.15.28:0/864066702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56534919c570 0x565349199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3431897640 1 -- 172.21.15.28:0/864066702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565349191580 msgr2=0x56534919bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3431897640 1 --2- 172.21.15.28:0/864066702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565349191580 0x56534919bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3431897640 1 -- 172.21.15.28:0/864066702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56534919c570 msgr2=0x565349199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3431897640 1 --2- 172.21.15.28:0/864066702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56534919c570 0x565349199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3431897640 1 -- 172.21.15.28:0/864066702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565349083dc0 con 0x565349191ea0 2024-04-25T01:34:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3432899640 1 --2- 172.21.15.28:0/864066702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56534919c570 0x565349199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3431897640 1 --2- 172.21.15.28:0/864066702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191ea0 0x565349199230 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f341c051010 tx=0x7f341c00fea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d61f63a5f51deac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3415ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3415ffb640 1 -- 172.21.15.28:0/864066702 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c0b8020 con 0x565349191ea0 2024-04-25T01:34:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3415ffb640 1 -- 172.21.15.28:0/864066702 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f341c09a4b0 con 0x565349191ea0 2024-04-25T01:34:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.627+0000 7f3437009740 1 -- 172.21.15.28:0/864066702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f341c09c080 con 0x565349191ea0 2024-04-25T01:34:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.628+0000 7f3415ffb640 1 -- 172.21.15.28:0/864066702 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f341c0991b0 con 0x565349191ea0 2024-04-25T01:34:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.628+0000 7f3437009740 10 client.16155 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.628+0000 7f3437009740 1 -- 172.21.15.28:0/864066702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x565348e0a3f0 con 0x565349191ea0 2024-04-25T01:34:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.628+0000 7f3415ffb640 1 -- 172.21.15.28:0/864066702 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f341c09c080 con 0x565349191ea0 2024-04-25T01:34:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.628+0000 7f3415ffb640 1 -- 172.21.15.28:0/864066702 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f341c0cf410 con 0x565349191ea0 2024-04-25T01:34:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.628+0000 7f3437009740 10 client.16155 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.628+0000 7f3437009740 10 client.16155 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:34:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.628+0000 7f3437009740 20 client.16155 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.628+0000 7f3437009740 1 --2- 172.21.15.28:0/864066702 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5653491ac510 0x5653491ae900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.628+0000 7f3437009740 4 client.16155 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.628+0000 7f3437009740 1 -- 172.21.15.28:0/864066702 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565349103bb0 con 0x5653491ac510 2024-04-25T01:34:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.628+0000 7f3432098640 1 --2- 172.21.15.28:0/864066702 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5653491ac510 0x5653491ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.629+0000 7f3432098640 1 --2- 172.21.15.28:0/864066702 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5653491ac510 0x5653491ae900 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6cb398ead32c2cad server_cookie=4774e062f69cd583 in_seq=0 out_seq=0 2024-04-25T01:34:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.629+0000 7f3415ffb640 10 client.16155 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:34:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.630+0000 7f3415ffb640 1 -- 172.21.15.28:0/864066702 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x565349103bb0 con 0x5653491ac510 2024-04-25T01:34:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.630+0000 7f3415ffb640 10 client.16155 handle_command_reply: tid=0 2024-04-25T01:34:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.630+0000 7f3437009740 1 client.16155 shutdown 2024-04-25T01:34:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.630+0000 7f3437009740 1 -- 172.21.15.28:0/864066702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191ea0 msgr2=0x565349199230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.630+0000 7f3437009740 1 --2- 172.21.15.28:0/864066702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191ea0 0x565349199230 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f341c051010 tx=0x7f341c00fea0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.630+0000 7f3437009740 1 -- 172.21.15.28:0/864066702 shutdown_connections 2024-04-25T01:34:42.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.630+0000 7f3437009740 1 --2- 172.21.15.28:0/864066702 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5653491ac510 0x5653491ae900 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.630+0000 7f3437009740 1 --2- 172.21.15.28:0/864066702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56534919c570 0x565349199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.630+0000 7f3415ffb640 0 client.16155 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:34:42.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.630+0000 7f3437009740 1 --2- 172.21.15.28:0/864066702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191ea0 0x565349199230 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.630+0000 7f3437009740 1 --2- 172.21.15.28:0/864066702 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565349191580 0x56534919bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.630+0000 7f3437009740 1 -- 172.21.15.28:0/864066702 >> 172.21.15.28:0/864066702 conn(0x56534918d8a0 msgr2=0x56534918cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.630+0000 7f3437009740 1 -- 172.21.15.28:0/864066702 shutdown_connections 2024-04-25T01:34:42.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.630+0000 7f3437009740 1 -- 172.21.15.28:0/864066702 wait complete. 2024-04-25T01:34:42.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.630+0000 7f3437009740 20 client.16155 trim_cache size 0 max 16384 2024-04-25T01:34:42.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.643+0000 7f3437009740 1 Processor -- start 2024-04-25T01:34:42.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.643+0000 7f3437009740 1 -- start start 2024-04-25T01:34:42.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3437009740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191580 0x5653491991d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3437009740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565349191ea0 0x56534919ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3437009740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56534919c570 0x5653491b1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3437009740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f342c1ac6d0 con 0x565349191580 2024-04-25T01:34:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3437009740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f342c1ae210 con 0x56534919c570 2024-04-25T01:34:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3437009740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f342c1acda0 con 0x565349191ea0 2024-04-25T01:34:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3432098640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191580 0x5653491991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3432098640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191580 0x5653491991d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41850/0 (socket says 172.21.15.28:41850) 2024-04-25T01:34:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3432098640 1 -- 172.21.15.28:0/4165775923 learned_addr learned my addr 172.21.15.28:0/4165775923 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3431897640 1 --2- 172.21.15.28:0/4165775923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565349191ea0 0x56534919ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3432098640 1 -- 172.21.15.28:0/4165775923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565349191ea0 msgr2=0x56534919ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3432098640 1 --2- 172.21.15.28:0/4165775923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565349191ea0 0x56534919ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3432098640 1 -- 172.21.15.28:0/4165775923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56534919c570 msgr2=0x5653491b1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3432098640 1 --2- 172.21.15.28:0/4165775923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56534919c570 0x5653491b1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3432098640 1 -- 172.21.15.28:0/4165775923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565349199bb0 con 0x565349191580 2024-04-25T01:34:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3431897640 1 --2- 172.21.15.28:0/4165775923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565349191ea0 0x56534919ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3432098640 1 --2- 172.21.15.28:0/4165775923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191580 0x5653491991d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f342802a930 tx=0x7f3428040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e7d430c29b80f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3417fff640 1 -- 172.21.15.28:0/4165775923 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3428035040 con 0x565349191580 2024-04-25T01:34:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3417fff640 1 -- 172.21.15.28:0/4165775923 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3428057070 con 0x565349191580 2024-04-25T01:34:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.644+0000 7f3417fff640 1 -- 172.21.15.28:0/4165775923 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34280035f0 con 0x565349191580 2024-04-25T01:34:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.645+0000 7f3437009740 1 -- 172.21.15.28:0/4165775923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191580 msgr2=0x5653491991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.645+0000 7f3437009740 1 --2- 172.21.15.28:0/4165775923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191580 0x5653491991d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f342802a930 tx=0x7f3428040000 comp rx=0 tx=0).stop 2024-04-25T01:34:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.645+0000 7f3437009740 1 -- 172.21.15.28:0/4165775923 shutdown_connections 2024-04-25T01:34:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.645+0000 7f3437009740 1 --2- 172.21.15.28:0/4165775923 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56534919c570 0x5653491b1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.645+0000 7f3437009740 1 --2- 172.21.15.28:0/4165775923 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x565349191ea0 0x56534919ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.645+0000 7f3437009740 1 --2- 172.21.15.28:0/4165775923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191580 0x5653491991d0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.645+0000 7f3437009740 1 -- 172.21.15.28:0/4165775923 >> 172.21.15.28:0/4165775923 conn(0x56534918d8a0 msgr2=0x565349198ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.645+0000 7f3437009740 1 -- 172.21.15.28:0/4165775923 shutdown_connections 2024-04-25T01:34:42.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.645+0000 7f3437009740 1 -- 172.21.15.28:0/4165775923 wait complete. 2024-04-25T01:34:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.645+0000 7f3437009740 1 -- start start 2024-04-25T01:34:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.645+0000 7f3437009740 1 Processor -- start 2024-04-25T01:34:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.645+0000 7f3437009740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191580 0x5653491afa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.645+0000 7f3437009740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565349191ea0 0x5653491aff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.645+0000 7f3437009740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56534919c570 0x5653492116f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.645+0000 7f3437009740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f342c1adb40 con 0x565349191580 2024-04-25T01:34:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.645+0000 7f3437009740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f342c1ad470 con 0x565349191ea0 2024-04-25T01:34:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.645+0000 7f3437009740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f342c1aa6b0 con 0x56534919c570 2024-04-25T01:34:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.645+0000 7f3437009740 10 client.0 _ll_register_callbacks cb 0x7f34240124f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3432098640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191580 0x5653491afa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3432098640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191580 0x5653491afa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41858/0 (socket says 172.21.15.28:41858) 2024-04-25T01:34:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3432098640 1 -- 172.21.15.28:0/2483880736 learned_addr learned my addr 172.21.15.28:0/2483880736 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3432899640 1 --2- 172.21.15.28:0/2483880736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56534919c570 0x5653492116f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3431897640 1 --2- 172.21.15.28:0/2483880736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565349191ea0 0x5653491aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3432098640 1 -- 172.21.15.28:0/2483880736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56534919c570 msgr2=0x5653492116f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3432098640 1 --2- 172.21.15.28:0/2483880736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56534919c570 0x5653492116f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3432098640 1 -- 172.21.15.28:0/2483880736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565349191ea0 msgr2=0x5653491aff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3432098640 1 --2- 172.21.15.28:0/2483880736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565349191ea0 0x5653491aff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3432098640 1 -- 172.21.15.28:0/2483880736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565348ed47d0 con 0x565349191580 2024-04-25T01:34:42.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3432899640 1 --2- 172.21.15.28:0/2483880736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56534919c570 0x5653492116f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:42.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3431897640 1 --2- 172.21.15.28:0/2483880736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565349191ea0 0x5653491aff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:42.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3432098640 1 --2- 172.21.15.28:0/2483880736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191580 0x5653491afa10 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f3428004770 tx=0x7f3428009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f98ce74d40ffdd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3415ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:42.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3415ffb640 1 -- 172.21.15.28:0/2483880736 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3428035070 con 0x565349191580 2024-04-25T01:34:42.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3415ffb640 1 -- 172.21.15.28:0/2483880736 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f34280518d0 con 0x565349191580 2024-04-25T01:34:42.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3437009740 1 -- 172.21.15.28:0/2483880736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3428057070 con 0x565349191580 2024-04-25T01:34:42.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3415ffb640 1 -- 172.21.15.28:0/2483880736 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3428069050 con 0x565349191580 2024-04-25T01:34:42.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3437009740 10 client.16161 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:42.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.646+0000 7f3437009740 1 -- 172.21.15.28:0/2483880736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x565349212c40 con 0x565349191580 2024-04-25T01:34:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.647+0000 7f3415ffb640 1 -- 172.21.15.28:0/2483880736 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3428057070 con 0x565349191580 2024-04-25T01:34:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.647+0000 7f3415ffb640 1 -- 172.21.15.28:0/2483880736 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3428003c50 con 0x565349191580 2024-04-25T01:34:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.647+0000 7f3437009740 10 client.16161 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.647+0000 7f3437009740 10 client.16161 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:34:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.647+0000 7f3437009740 20 client.16161 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.647+0000 7f3437009740 1 --2- 172.21.15.28:0/2483880736 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x565349213320 0x565349215740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.647+0000 7f3437009740 4 client.16161 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:34:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.647+0000 7f3437009740 1 -- 172.21.15.28:0/2483880736 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5653490fdf00 con 0x565349213320 2024-04-25T01:34:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.647+0000 7f3431897640 1 --2- 172.21.15.28:0/2483880736 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x565349213320 0x565349215740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.647+0000 7f3431897640 1 --2- 172.21.15.28:0/2483880736 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x565349213320 0x565349215740 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5c63c00b401ccb31 server_cookie=d70743795fa41aad in_seq=0 out_seq=0 2024-04-25T01:34:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.647+0000 7f3415ffb640 10 client.16161 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:34:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f3415ffb640 1 -- 172.21.15.28:0/2483880736 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5653490fdf00 con 0x565349213320 2024-04-25T01:34:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f3415ffb640 10 client.16161 handle_command_reply: tid=0 2024-04-25T01:34:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f3437009740 1 client.16161 shutdown 2024-04-25T01:34:42.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f3437009740 1 -- 172.21.15.28:0/2483880736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191580 msgr2=0x5653491afa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f3437009740 1 --2- 172.21.15.28:0/2483880736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191580 0x5653491afa10 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f3428004770 tx=0x7f3428009260 comp rx=0 tx=0).stop 2024-04-25T01:34:42.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f3437009740 1 -- 172.21.15.28:0/2483880736 shutdown_connections 2024-04-25T01:34:42.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f3437009740 1 --2- 172.21.15.28:0/2483880736 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x565349213320 0x565349215740 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f3437009740 1 --2- 172.21.15.28:0/2483880736 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56534919c570 0x5653492116f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f3437009740 1 --2- 172.21.15.28:0/2483880736 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565349191ea0 0x5653491aff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f3437009740 1 --2- 172.21.15.28:0/2483880736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x565349191580 0x5653491afa10 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f3415ffb640 0 client.16161 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:34:42.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f3437009740 1 -- 172.21.15.28:0/2483880736 >> 172.21.15.28:0/2483880736 conn(0x56534918d8a0 msgr2=0x5653491a7790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f3437009740 1 -- 172.21.15.28:0/2483880736 shutdown_connections 2024-04-25T01:34:42.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f3437009740 1 -- 172.21.15.28:0/2483880736 wait complete. 2024-04-25T01:34:42.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f3437009740 20 client.16161 trim_cache size 0 max 16384 2024-04-25T01:34:42.632 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:34:42.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f34342fc640 1 -- 172.21.15.28:0/1745661247 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f34180a2d80 msgr2=0x7f34180a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f34342fc640 1 --2- 172.21.15.28:0/1745661247 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f34180a2d80 0x7f34180a5240 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3428009230 tx=0x7f34280058d0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f34342fc640 1 -- 172.21.15.28:0/1745661247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f342c101330 msgr2=0x7f342c1972e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f34342fc640 1 --2- 172.21.15.28:0/1745661247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f342c101330 0x7f342c1972e0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f341c00c920 tx=0x7f341c00cde0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f34342fc640 1 -- 172.21.15.28:0/1745661247 shutdown_connections 2024-04-25T01:34:42.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f34342fc640 1 --2- 172.21.15.28:0/1745661247 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f34180a2d80 0x7f34180a5240 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f34342fc640 1 --2- 172.21.15.28:0/1745661247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f342c106690 0x7f342c19b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f34342fc640 1 --2- 172.21.15.28:0/1745661247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f342c101330 0x7f342c1972e0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f34342fc640 1 --2- 172.21.15.28:0/1745661247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f342c100990 0x7f342c196da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.648+0000 7f34342fc640 1 -- 172.21.15.28:0/1745661247 >> 172.21.15.28:0/1745661247 conn(0x7f342c076440 msgr2=0x7f342c0ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.649+0000 7f34342fc640 1 -- 172.21.15.28:0/1745661247 shutdown_connections 2024-04-25T01:34:42.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.649+0000 7f34342fc640 1 -- 172.21.15.28:0/1745661247 wait complete. 2024-04-25T01:34:42.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:34:42.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.764+0000 7f99ee770640 1 -- 172.21.15.28:0/468572595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e81020d0 msgr2=0x7f99e810e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.764+0000 7f99ee770640 1 --2- 172.21.15.28:0/468572595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e81020d0 0x7f99e810e5e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f99e0009cd0 tx=0x7f99e002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.764+0000 7f99ee770640 1 -- 172.21.15.28:0/468572595 shutdown_connections 2024-04-25T01:34:42.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.764+0000 7f99ee770640 1 --2- 172.21.15.28:0/468572595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e81020d0 0x7f99e810e5e0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.764+0000 7f99ee770640 1 --2- 172.21.15.28:0/468572595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e8101710 0x7f99e8101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.764+0000 7f99ee770640 1 --2- 172.21.15.28:0/468572595 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f99e8107720 0x7f99e8107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.764+0000 7f99ee770640 1 -- 172.21.15.28:0/468572595 >> 172.21.15.28:0/468572595 conn(0x7f99e80fd180 msgr2=0x7f99e80ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.764+0000 7f99ee770640 1 -- 172.21.15.28:0/468572595 shutdown_connections 2024-04-25T01:34:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.764+0000 7f99ee770640 1 -- 172.21.15.28:0/468572595 wait complete. 2024-04-25T01:34:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.765+0000 7f99ee770640 1 Processor -- start 2024-04-25T01:34:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.765+0000 7f99ee770640 1 -- start start 2024-04-25T01:34:42.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.765+0000 7f99ee770640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e8101710 0x7f99e8197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.765+0000 7f99ee770640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e81020d0 0x7f99e8198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.765+0000 7f99ee770640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f99e8107720 0x7f99e819c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.765+0000 7f99ee770640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f99e8111af0 con 0x7f99e81020d0 2024-04-25T01:34:42.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.765+0000 7f99ee770640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f99e8111970 con 0x7f99e8101710 2024-04-25T01:34:42.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.765+0000 7f99ee770640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f99e8111c70 con 0x7f99e8107720 2024-04-25T01:34:42.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.765+0000 7f99e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e81020d0 0x7f99e8198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.765+0000 7f99e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e81020d0 0x7f99e8198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41864/0 (socket says 172.21.15.28:41864) 2024-04-25T01:34:42.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.765+0000 7f99e77fe640 1 -- 172.21.15.28:0/82169834 learned_addr learned my addr 172.21.15.28:0/82169834 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.765+0000 7f99e7fff640 1 --2- 172.21.15.28:0/82169834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e8101710 0x7f99e8197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.765+0000 7f99ecd0d640 1 --2- 172.21.15.28:0/82169834 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f99e8107720 0x7f99e819c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.766+0000 7f99e77fe640 1 -- 172.21.15.28:0/82169834 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f99e8107720 msgr2=0x7f99e819c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.766+0000 7f99e77fe640 1 --2- 172.21.15.28:0/82169834 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f99e8107720 0x7f99e819c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.766+0000 7f99e77fe640 1 -- 172.21.15.28:0/82169834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e8101710 msgr2=0x7f99e8197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.766+0000 7f99e77fe640 1 --2- 172.21.15.28:0/82169834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e8101710 0x7f99e8197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.766+0000 7f99e77fe640 1 -- 172.21.15.28:0/82169834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99e819cfe0 con 0x7f99e81020d0 2024-04-25T01:34:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.766+0000 7f99e77fe640 1 --2- 172.21.15.28:0/82169834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e81020d0 0x7f99e8198530 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f99dc00d950 tx=0x7f99dc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.766+0000 7f99e57fa640 1 -- 172.21.15.28:0/82169834 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99dc017030 con 0x7f99e81020d0 2024-04-25T01:34:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.766+0000 7f99ee770640 1 -- 172.21.15.28:0/82169834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99e819d2d0 con 0x7f99e81020d0 2024-04-25T01:34:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.766+0000 7f99e57fa640 1 -- 172.21.15.28:0/82169834 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99dc004020 con 0x7f99e81020d0 2024-04-25T01:34:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.766+0000 7f99e57fa640 1 -- 172.21.15.28:0/82169834 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99dc0047a0 con 0x7f99e81020d0 2024-04-25T01:34:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.766+0000 7f99ee770640 1 -- 172.21.15.28:0/82169834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99e81a4b60 con 0x7f99e81020d0 2024-04-25T01:34:42.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.768+0000 7f99e57fa640 1 -- 172.21.15.28:0/82169834 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f99dc0041d0 con 0x7f99e81020d0 2024-04-25T01:34:42.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99f1482740 1 Processor -- start 2024-04-25T01:34:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99e57fa640 1 --2- 172.21.15.28:0/82169834 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f99d00a2d00 0x7f99d00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99f1482740 1 -- start start 2024-04-25T01:34:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99e57fa640 1 -- 172.21.15.28:0/82169834 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f99dc0cf170 con 0x7f99e81020d0 2024-04-25T01:34:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99ecd0d640 1 --2- 172.21.15.28:0/82169834 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f99d00a2d00 0x7f99d00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99f1482740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c74580 0x559185c74960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99f1482740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c74ea0 0x559185c7f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99f1482740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c7f570 0x559185c81960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99f1482740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559185bd9670 con 0x559185c7f570 2024-04-25T01:34:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99f1482740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559185b81970 con 0x559185c74ea0 2024-04-25T01:34:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99f1482740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x559185bd5630 con 0x559185c74580 2024-04-25T01:34:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99ecd0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c7f570 0x559185c81960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99ecd0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c7f570 0x559185c81960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41876/0 (socket says 172.21.15.28:41876) 2024-04-25T01:34:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99ecd0d640 1 -- 172.21.15.28:0/977025919 learned_addr learned my addr 172.21.15.28:0/977025919 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99e77fe640 1 --2- 172.21.15.28:0/977025919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c74ea0 0x559185c7f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99e7fff640 1 --2- 172.21.15.28:0/977025919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c74580 0x559185c74960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99ecd0d640 1 --2- 172.21.15.28:0/82169834 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f99d00a2d00 0x7f99d00a51c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f99e0009cd0 tx=0x7f99e00316c0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99ecd0d640 1 -- 172.21.15.28:0/977025919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c74580 msgr2=0x559185c74960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99ecd0d640 1 --2- 172.21.15.28:0/977025919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c74580 0x559185c74960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99ecd0d640 1 -- 172.21.15.28:0/977025919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c74ea0 msgr2=0x559185c7f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99ecd0d640 1 --2- 172.21.15.28:0/977025919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c74ea0 0x559185c7f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.769+0000 7f99ecd0d640 1 -- 172.21.15.28:0/977025919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559185ba5400 con 0x559185c7f570 2024-04-25T01:34:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.770+0000 7f99ecd0d640 1 --2- 172.21.15.28:0/977025919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c7f570 0x559185c81960 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f99e0035000 tx=0x7f99e0049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f26159f3e292d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.770+0000 7f99ce7fc640 1 -- 172.21.15.28:0/977025919 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e00044c0 con 0x559185c7f570 2024-04-25T01:34:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.770+0000 7f99ce7fc640 1 -- 172.21.15.28:0/977025919 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99e0050070 con 0x559185c7f570 2024-04-25T01:34:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.770+0000 7f99ce7fc640 1 -- 172.21.15.28:0/977025919 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e004a450 con 0x559185c7f570 2024-04-25T01:34:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.770+0000 7f99f1482740 1 -- 172.21.15.28:0/977025919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c7f570 msgr2=0x559185c81960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.770+0000 7f99f1482740 1 --2- 172.21.15.28:0/977025919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c7f570 0x559185c81960 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f99e0035000 tx=0x7f99e0049000 comp rx=0 tx=0).stop 2024-04-25T01:34:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.770+0000 7f99f1482740 1 -- 172.21.15.28:0/977025919 shutdown_connections 2024-04-25T01:34:42.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.770+0000 7f99f1482740 1 --2- 172.21.15.28:0/977025919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c7f570 0x559185c81960 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.770+0000 7f99f1482740 1 --2- 172.21.15.28:0/977025919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c74ea0 0x559185c7f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.770+0000 7f99f1482740 1 --2- 172.21.15.28:0/977025919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c74580 0x559185c74960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.770+0000 7f99f1482740 1 -- 172.21.15.28:0/977025919 >> 172.21.15.28:0/977025919 conn(0x559185c708a0 msgr2=0x559185c70c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.771+0000 7f99f1482740 1 -- 172.21.15.28:0/977025919 shutdown_connections 2024-04-25T01:34:42.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.771+0000 7f99f1482740 1 -- 172.21.15.28:0/977025919 wait complete. 2024-04-25T01:34:42.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.771+0000 7f99f1482740 1 -- start start 2024-04-25T01:34:42.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.771+0000 7f99f1482740 1 Processor -- start 2024-04-25T01:34:42.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.771+0000 7f99f1482740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c74580 0x559185c7ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.771+0000 7f99f1482740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c74ea0 0x559185c7c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.771+0000 7f99f1482740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c7f570 0x559185c7c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.771+0000 7f99f1482740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559185c82300 con 0x559185c74580 2024-04-25T01:34:42.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.771+0000 7f99f1482740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559185b95050 con 0x559185c7f570 2024-04-25T01:34:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.771+0000 7f99f1482740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x559185c82480 con 0x559185c74ea0 2024-04-25T01:34:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.771+0000 7f99e7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c74580 0x559185c7ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.771+0000 7f99f1482740 10 client.0 _ll_register_callbacks cb 0x5591858ecbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.772+0000 7f99e7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c74580 0x559185c7ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41888/0 (socket says 172.21.15.28:41888) 2024-04-25T01:34:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.772+0000 7f99e7fff640 1 -- 172.21.15.28:0/3212687611 learned_addr learned my addr 172.21.15.28:0/3212687611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.772+0000 7f99ecd0d640 1 --2- 172.21.15.28:0/3212687611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c7f570 0x559185c7c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.772+0000 7f99e77fe640 1 --2- 172.21.15.28:0/3212687611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c74ea0 0x559185c7c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.772+0000 7f99e7fff640 1 -- 172.21.15.28:0/3212687611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c74ea0 msgr2=0x559185c7c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.772+0000 7f99e7fff640 1 --2- 172.21.15.28:0/3212687611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c74ea0 0x559185c7c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.772+0000 7f99e7fff640 1 -- 172.21.15.28:0/3212687611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c7f570 msgr2=0x559185c7c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.772+0000 7f99e7fff640 1 --2- 172.21.15.28:0/3212687611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c7f570 0x559185c7c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.772+0000 7f99e7fff640 1 -- 172.21.15.28:0/3212687611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559185b66dc0 con 0x559185c74580 2024-04-25T01:34:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.772+0000 7f99e7fff640 1 --2- 172.21.15.28:0/3212687611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c74580 0x559185c7ef00 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f99d800beb0 tx=0x7f99d8012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8575749f2ea36766 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.772+0000 7f99c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.772+0000 7f99c77fe640 1 -- 172.21.15.28:0/3212687611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d8004220 con 0x559185c74580 2024-04-25T01:34:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.772+0000 7f99c77fe640 1 -- 172.21.15.28:0/3212687611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f99d80043c0 con 0x559185c74580 2024-04-25T01:34:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.772+0000 7f99f1482740 1 -- 172.21.15.28:0/3212687611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99e0050070 con 0x559185c74580 2024-04-25T01:34:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.772+0000 7f99c77fe640 1 -- 172.21.15.28:0/3212687611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f99d8024050 con 0x559185c74580 2024-04-25T01:34:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.772+0000 7f99f1482740 10 client.16173 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.772+0000 7f99f1482740 1 -- 172.21.15.28:0/3212687611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5591858ed3f0 con 0x559185c74580 2024-04-25T01:34:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.773+0000 7f99c77fe640 1 -- 172.21.15.28:0/3212687611 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99e0050070 con 0x559185c74580 2024-04-25T01:34:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.773+0000 7f99c77fe640 1 -- 172.21.15.28:0/3212687611 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f99d8019030 con 0x559185c74580 2024-04-25T01:34:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.773+0000 7f99f1482740 10 client.16173 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.773+0000 7f99f1482740 10 client.16173 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:34:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.773+0000 7f99f1482740 20 client.16173 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.773+0000 7f99f1482740 1 --2- 172.21.15.28:0/3212687611 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x559185c8f5e0 0x559185c91a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.773+0000 7f99f1482740 4 client.16173 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.773+0000 7f99f1482740 1 -- 172.21.15.28:0/3212687611 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559185be6bb0 con 0x559185c8f5e0 2024-04-25T01:34:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.773+0000 7f99e77fe640 1 --2- 172.21.15.28:0/3212687611 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x559185c8f5e0 0x559185c91a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.773+0000 7f99c77fe640 10 client.16173 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:34:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.773+0000 7f99e77fe640 1 --2- 172.21.15.28:0/3212687611 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x559185c8f5e0 0x559185c91a40 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=59a28a4b15c1bd38 server_cookie=8bdd577031bbf162 in_seq=0 out_seq=0 2024-04-25T01:34:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.774+0000 7f99c77fe640 1 -- 172.21.15.28:0/3212687611 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559185be6bb0 con 0x559185c8f5e0 2024-04-25T01:34:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.774+0000 7f99c77fe640 10 client.16173 handle_command_reply: tid=0 2024-04-25T01:34:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.774+0000 7f99f1482740 1 client.16173 shutdown 2024-04-25T01:34:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.774+0000 7f99f1482740 1 -- 172.21.15.28:0/3212687611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c74580 msgr2=0x559185c7ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.774+0000 7f99f1482740 1 --2- 172.21.15.28:0/3212687611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c74580 0x559185c7ef00 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f99d800beb0 tx=0x7f99d8012820 comp rx=0 tx=0).stop 2024-04-25T01:34:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.774+0000 7f99f1482740 1 -- 172.21.15.28:0/3212687611 shutdown_connections 2024-04-25T01:34:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.774+0000 7f99f1482740 1 --2- 172.21.15.28:0/3212687611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c7f570 0x559185c7c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.774+0000 7f99f1482740 1 --2- 172.21.15.28:0/3212687611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c74ea0 0x559185c7c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.774+0000 7f99f1482740 1 --2- 172.21.15.28:0/3212687611 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x559185c8f5e0 0x559185c91a40 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.774+0000 7f99f1482740 1 --2- 172.21.15.28:0/3212687611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c74580 0x559185c7ef00 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.774+0000 7f99f1482740 1 -- 172.21.15.28:0/3212687611 >> 172.21.15.28:0/3212687611 conn(0x559185c708a0 msgr2=0x559185c6fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.774+0000 7f99c77fe640 0 client.16173 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:34:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.774+0000 7f99f1482740 1 -- 172.21.15.28:0/3212687611 shutdown_connections 2024-04-25T01:34:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.775+0000 7f99f1482740 1 -- 172.21.15.28:0/3212687611 wait complete. 2024-04-25T01:34:42.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.775+0000 7f99f1482740 20 client.16173 trim_cache size 0 max 16384 2024-04-25T01:34:42.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.788+0000 7f99f1482740 1 Processor -- start 2024-04-25T01:34:42.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.788+0000 7f99f1482740 1 -- start start 2024-04-25T01:34:42.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.788+0000 7f99f1482740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c74580 0x559185c8b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.788+0000 7f99f1482740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c74ea0 0x559185c97310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.788+0000 7f99f1482740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c7f570 0x559185c97850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.788+0000 7f99f1482740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f99e81aeaf0 con 0x559185c7f570 2024-04-25T01:34:42.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.788+0000 7f99f1482740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f99e81b0630 con 0x559185c74580 2024-04-25T01:34:42.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.788+0000 7f99f1482740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f99e81af1c0 con 0x559185c74ea0 2024-04-25T01:34:42.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.788+0000 7f99ecd0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c7f570 0x559185c97850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.788+0000 7f99ecd0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c7f570 0x559185c97850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41898/0 (socket says 172.21.15.28:41898) 2024-04-25T01:34:42.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.788+0000 7f99ecd0d640 1 -- 172.21.15.28:0/3004813431 learned_addr learned my addr 172.21.15.28:0/3004813431 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.788+0000 7f99e7fff640 1 --2- 172.21.15.28:0/3004813431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c74580 0x559185c8b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.788+0000 7f99e77fe640 1 --2- 172.21.15.28:0/3004813431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c74ea0 0x559185c97310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.788+0000 7f99ecd0d640 1 -- 172.21.15.28:0/3004813431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c74ea0 msgr2=0x559185c97310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.788+0000 7f99ecd0d640 1 --2- 172.21.15.28:0/3004813431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c74ea0 0x559185c97310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.788+0000 7f99ecd0d640 1 -- 172.21.15.28:0/3004813431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c74580 msgr2=0x559185c8b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.788+0000 7f99ecd0d640 1 --2- 172.21.15.28:0/3004813431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c74580 0x559185c8b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.788+0000 7f99ecd0d640 1 -- 172.21.15.28:0/3004813431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559185c7cae0 con 0x559185c7f570 2024-04-25T01:34:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.788+0000 7f99e77fe640 1 --2- 172.21.15.28:0/3004813431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c74ea0 0x559185c97310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.789+0000 7f99ecd0d640 1 --2- 172.21.15.28:0/3004813431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c7f570 0x559185c97850 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x559185c71a30 tx=0x7f99e002a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12614f1b9b9c44f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.789+0000 7f99cdffb640 1 -- 172.21.15.28:0/3004813431 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e00044c0 con 0x559185c7f570 2024-04-25T01:34:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.789+0000 7f99cdffb640 1 -- 172.21.15.28:0/3004813431 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99e0050070 con 0x559185c7f570 2024-04-25T01:34:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.789+0000 7f99cdffb640 1 -- 172.21.15.28:0/3004813431 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e0030e50 con 0x559185c7f570 2024-04-25T01:34:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.789+0000 7f99f1482740 1 -- 172.21.15.28:0/3004813431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c7f570 msgr2=0x559185c97850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.789+0000 7f99f1482740 1 --2- 172.21.15.28:0/3004813431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c7f570 0x559185c97850 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x559185c71a30 tx=0x7f99e002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.789+0000 7f99f1482740 1 -- 172.21.15.28:0/3004813431 shutdown_connections 2024-04-25T01:34:42.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.789+0000 7f99f1482740 1 --2- 172.21.15.28:0/3004813431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c7f570 0x559185c97850 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.789+0000 7f99f1482740 1 --2- 172.21.15.28:0/3004813431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c74ea0 0x559185c97310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.789+0000 7f99f1482740 1 --2- 172.21.15.28:0/3004813431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c74580 0x559185c8b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.789+0000 7f99f1482740 1 -- 172.21.15.28:0/3004813431 >> 172.21.15.28:0/3004813431 conn(0x559185c708a0 msgr2=0x559185c6f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.789+0000 7f99f1482740 1 -- 172.21.15.28:0/3004813431 shutdown_connections 2024-04-25T01:34:42.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.789+0000 7f99f1482740 1 -- 172.21.15.28:0/3004813431 wait complete. 2024-04-25T01:34:42.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.790+0000 7f99f1482740 1 -- start start 2024-04-25T01:34:42.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.790+0000 7f99f1482740 1 Processor -- start 2024-04-25T01:34:42.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.790+0000 7f99f1482740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c74580 0x559185c85130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.790+0000 7f99f1482740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c74ea0 0x559185c85670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.790+0000 7f99f1482740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c7f570 0x559185c85bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.790+0000 7f99f1482740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f99e81aff60 con 0x559185c74580 2024-04-25T01:34:42.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.790+0000 7f99f1482740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f99e81af890 con 0x559185c74ea0 2024-04-25T01:34:42.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.790+0000 7f99f1482740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f99e81acad0 con 0x559185c7f570 2024-04-25T01:34:42.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.790+0000 7f99f1482740 10 client.0 _ll_register_callbacks cb 0x7f99d800bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:42.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.790+0000 7f99e7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c74580 0x559185c85130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.790+0000 7f99e7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c74580 0x559185c85130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41902/0 (socket says 172.21.15.28:41902) 2024-04-25T01:34:42.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.790+0000 7f99e7fff640 1 -- 172.21.15.28:0/2909968557 learned_addr learned my addr 172.21.15.28:0/2909968557 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.790+0000 7f99e77fe640 1 --2- 172.21.15.28:0/2909968557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c74ea0 0x559185c85670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.790+0000 7f99ecd0d640 1 --2- 172.21.15.28:0/2909968557 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c7f570 0x559185c85bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.791+0000 7f99e7fff640 1 -- 172.21.15.28:0/2909968557 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c7f570 msgr2=0x559185c85bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.791+0000 7f99e7fff640 1 --2- 172.21.15.28:0/2909968557 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c7f570 0x559185c85bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.791+0000 7f99e7fff640 1 -- 172.21.15.28:0/2909968557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c74ea0 msgr2=0x559185c85670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.791+0000 7f99e7fff640 1 --2- 172.21.15.28:0/2909968557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c74ea0 0x559185c85670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.791+0000 7f99e7fff640 1 -- 172.21.15.28:0/2909968557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591859b77d0 con 0x559185c74580 2024-04-25T01:34:42.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.791+0000 7f99e7fff640 1 --2- 172.21.15.28:0/2909968557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c74580 0x559185c85130 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f99d8015040 tx=0x7f99d8003f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9299a8a201b806bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.791+0000 7f99c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:42.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.791+0000 7f99c77fe640 1 -- 172.21.15.28:0/2909968557 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d8004000 con 0x559185c74580 2024-04-25T01:34:42.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.791+0000 7f99c77fe640 1 -- 172.21.15.28:0/2909968557 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f99d80041a0 con 0x559185c74580 2024-04-25T01:34:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.791+0000 7f99f1482740 1 -- 172.21.15.28:0/2909968557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99e0050070 con 0x559185c74580 2024-04-25T01:34:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.791+0000 7f99c77fe640 1 -- 172.21.15.28:0/2909968557 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f99d8037050 con 0x559185c74580 2024-04-25T01:34:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.791+0000 7f99f1482740 10 client.16179 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.791+0000 7f99f1482740 1 -- 172.21.15.28:0/2909968557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559185cf5c40 con 0x559185c74580 2024-04-25T01:34:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.791+0000 7f99c77fe640 1 -- 172.21.15.28:0/2909968557 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99e0050070 con 0x559185c74580 2024-04-25T01:34:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.791+0000 7f99c77fe640 1 -- 172.21.15.28:0/2909968557 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f99d8019040 con 0x559185c74580 2024-04-25T01:34:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.791+0000 7f99f1482740 10 client.16179 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.791+0000 7f99f1482740 10 client.16179 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:34:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.791+0000 7f99f1482740 20 client.16179 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.791+0000 7f99f1482740 1 --2- 172.21.15.28:0/2909968557 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x559185cf6410 0x559185cf8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.792+0000 7f99f1482740 4 client.16179 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:34:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.792+0000 7f99f1482740 1 -- 172.21.15.28:0/2909968557 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559185be0f00 con 0x559185cf6410 2024-04-25T01:34:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.792+0000 7f99e77fe640 1 --2- 172.21.15.28:0/2909968557 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x559185cf6410 0x559185cf8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.792+0000 7f99e77fe640 1 --2- 172.21.15.28:0/2909968557 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x559185cf6410 0x559185cf8830 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6e2ff938e667eebf server_cookie=758962d9474da02e in_seq=0 out_seq=0 2024-04-25T01:34:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.792+0000 7f99c77fe640 10 client.16179 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:34:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.792+0000 7f99c77fe640 1 -- 172.21.15.28:0/2909968557 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559185be0f00 con 0x559185cf6410 2024-04-25T01:34:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.792+0000 7f99c77fe640 10 client.16179 handle_command_reply: tid=0 2024-04-25T01:34:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.792+0000 7f99f1482740 1 client.16179 shutdown 2024-04-25T01:34:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.792+0000 7f99f1482740 1 -- 172.21.15.28:0/2909968557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c74580 msgr2=0x559185c85130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.792+0000 7f99f1482740 1 --2- 172.21.15.28:0/2909968557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c74580 0x559185c85130 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f99d8015040 tx=0x7f99d8003f30 comp rx=0 tx=0).stop 2024-04-25T01:34:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99f1482740 1 -- 172.21.15.28:0/2909968557 shutdown_connections 2024-04-25T01:34:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99f1482740 1 --2- 172.21.15.28:0/2909968557 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559185c7f570 0x559185c85bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99f1482740 1 --2- 172.21.15.28:0/2909968557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559185c74ea0 0x559185c85670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99f1482740 1 --2- 172.21.15.28:0/2909968557 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x559185cf6410 0x559185cf8830 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99f1482740 1 --2- 172.21.15.28:0/2909968557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559185c74580 0x559185c85130 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99c77fe640 0 client.16179 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:34:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99f1482740 1 -- 172.21.15.28:0/2909968557 >> 172.21.15.28:0/2909968557 conn(0x559185c708a0 msgr2=0x559185c6fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99f1482740 1 -- 172.21.15.28:0/2909968557 shutdown_connections 2024-04-25T01:34:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99f1482740 1 -- 172.21.15.28:0/2909968557 wait complete. 2024-04-25T01:34:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99f1482740 20 client.16179 trim_cache size 0 max 16384 2024-04-25T01:34:42.777 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:34:42.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99ee770640 1 -- 172.21.15.28:0/82169834 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f99d00a2d00 msgr2=0x7f99d00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99ee770640 1 --2- 172.21.15.28:0/82169834 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f99d00a2d00 0x7f99d00a51c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f99e0009cd0 tx=0x7f99e00316c0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99ee770640 1 -- 172.21.15.28:0/82169834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e81020d0 msgr2=0x7f99e8198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99ee770640 1 --2- 172.21.15.28:0/82169834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e81020d0 0x7f99e8198530 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f99dc00d950 tx=0x7f99dc00de10 comp rx=0 tx=0).stop 2024-04-25T01:34:42.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99ee770640 1 -- 172.21.15.28:0/82169834 shutdown_connections 2024-04-25T01:34:42.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99ee770640 1 --2- 172.21.15.28:0/82169834 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f99d00a2d00 0x7f99d00a51c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99ee770640 1 --2- 172.21.15.28:0/82169834 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f99e8107720 0x7f99e819c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99ee770640 1 --2- 172.21.15.28:0/82169834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99e81020d0 0x7f99e8198530 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99ee770640 1 --2- 172.21.15.28:0/82169834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99e8101710 0x7f99e8197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99ee770640 1 -- 172.21.15.28:0/82169834 >> 172.21.15.28:0/82169834 conn(0x7f99e80fd180 msgr2=0x7f99e80fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99ee770640 1 -- 172.21.15.28:0/82169834 shutdown_connections 2024-04-25T01:34:42.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.793+0000 7f99ee770640 1 -- 172.21.15.28:0/82169834 wait complete. 2024-04-25T01:34:42.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:34:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.909+0000 7fe99cf89640 1 -- 172.21.15.28:0/1558416027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9981032e0 msgr2=0x7fe99810b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.909+0000 7fe99cf89640 1 --2- 172.21.15.28:0/1558416027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9981032e0 0x7fe99810b7c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe98c009900 tx=0x7fe98c02a510 comp rx=0 tx=0).stop 2024-04-25T01:34:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.909+0000 7fe99cf89640 1 -- 172.21.15.28:0/1558416027 shutdown_connections 2024-04-25T01:34:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.909+0000 7fe99cf89640 1 --2- 172.21.15.28:0/1558416027 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe99810bd00 0x7fe99810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.909+0000 7fe99cf89640 1 --2- 172.21.15.28:0/1558416027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9981032e0 0x7fe99810b7c0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.909+0000 7fe99cf89640 1 --2- 172.21.15.28:0/1558416027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9981029c0 0x7fe998102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.909+0000 7fe99cf89640 1 -- 172.21.15.28:0/1558416027 >> 172.21.15.28:0/1558416027 conn(0x7fe998077740 msgr2=0x7fe998077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.909+0000 7fe99cf89640 1 -- 172.21.15.28:0/1558416027 shutdown_connections 2024-04-25T01:34:42.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.909+0000 7fe99cf89640 1 -- 172.21.15.28:0/1558416027 wait complete. 2024-04-25T01:34:42.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.910+0000 7fe99cf89640 1 Processor -- start 2024-04-25T01:34:42.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.910+0000 7fe99cf89640 1 -- start start 2024-04-25T01:34:42.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.910+0000 7fe99cf89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9981029c0 0x7fe998198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.910+0000 7fe99cf89640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9981032e0 0x7fe998198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.910+0000 7fe99cf89640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe99810bd00 0x7fe99819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.910+0000 7fe99cf89640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe998111b00 con 0x7fe9981029c0 2024-04-25T01:34:42.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.910+0000 7fe99cf89640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe998111980 con 0x7fe99810bd00 2024-04-25T01:34:42.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.910+0000 7fe99cf89640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe998111c80 con 0x7fe9981032e0 2024-04-25T01:34:42.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.910+0000 7fe99659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9981029c0 0x7fe998198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.910+0000 7fe99659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9981029c0 0x7fe998198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41932/0 (socket says 172.21.15.28:41932) 2024-04-25T01:34:42.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.910+0000 7fe99659c640 1 -- 172.21.15.28:0/2947140877 learned_addr learned my addr 172.21.15.28:0/2947140877 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.910+0000 7fe995d9b640 1 --2- 172.21.15.28:0/2947140877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9981032e0 0x7fe998198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.910+0000 7fe996d9d640 1 --2- 172.21.15.28:0/2947140877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe99810bd00 0x7fe99819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.910+0000 7fe99659c640 1 -- 172.21.15.28:0/2947140877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9981032e0 msgr2=0x7fe998198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.910+0000 7fe99659c640 1 --2- 172.21.15.28:0/2947140877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9981032e0 0x7fe998198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.910+0000 7fe99659c640 1 -- 172.21.15.28:0/2947140877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe99810bd00 msgr2=0x7fe99819c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.910+0000 7fe99659c640 1 --2- 172.21.15.28:0/2947140877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe99810bd00 0x7fe99819c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.910+0000 7fe99659c640 1 -- 172.21.15.28:0/2947140877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe99819d010 con 0x7fe9981029c0 2024-04-25T01:34:42.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.910+0000 7fe995d9b640 1 --2- 172.21.15.28:0/2947140877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9981032e0 0x7fe998198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:42.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.911+0000 7fe99659c640 1 --2- 172.21.15.28:0/2947140877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9981029c0 0x7fe998198020 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe98800d950 tx=0x7fe98800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.911+0000 7fe9877fe640 1 -- 172.21.15.28:0/2947140877 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe988017030 con 0x7fe9981029c0 2024-04-25T01:34:42.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.911+0000 7fe99cf89640 1 -- 172.21.15.28:0/2947140877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe99819d300 con 0x7fe9981029c0 2024-04-25T01:34:42.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.911+0000 7fe9877fe640 1 -- 172.21.15.28:0/2947140877 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe988004020 con 0x7fe9981029c0 2024-04-25T01:34:42.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.911+0000 7fe9877fe640 1 -- 172.21.15.28:0/2947140877 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9880047a0 con 0x7fe9981029c0 2024-04-25T01:34:42.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.911+0000 7fe99cf89640 1 -- 172.21.15.28:0/2947140877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9981a4b90 con 0x7fe9981029c0 2024-04-25T01:34:42.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.913+0000 7fe9877fe640 1 -- 172.21.15.28:0/2947140877 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe9880041d0 con 0x7fe9981029c0 2024-04-25T01:34:42.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.913+0000 7fe99fd73740 1 Processor -- start 2024-04-25T01:34:42.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe99fd73740 1 -- start start 2024-04-25T01:34:42.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe99fd73740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b182fbe2b0 0x55b182fbe690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe99fd73740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbec60 0x55b182fcd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe99fd73740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b182fcd980 0x55b182fcfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe99fd73740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b182f23670 con 0x55b182fbec60 2024-04-25T01:34:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe99fd73740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b182ecb970 con 0x55b182fbe2b0 2024-04-25T01:34:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe99fd73740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b182f1f630 con 0x55b182fcd980 2024-04-25T01:34:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe9877fe640 1 --2- 172.21.15.28:0/2947140877 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe9800a2d00 0x7fe9800a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe99659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbec60 0x55b182fcd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe99659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbec60 0x55b182fcd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41942/0 (socket says 172.21.15.28:41942) 2024-04-25T01:34:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe99659c640 1 -- 172.21.15.28:0/6767108 learned_addr learned my addr 172.21.15.28:0/6767108 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe996d9d640 1 --2- 172.21.15.28:0/2947140877 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe9800a2d00 0x7fe9800a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe995d9b640 1 --2- 172.21.15.28:0/6767108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b182fcd980 0x55b182fcfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe996d9d640 1 --2- 172.21.15.28:0/6767108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b182fbe2b0 0x55b182fbe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe9877fe640 1 -- 172.21.15.28:0/2947140877 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fe9880cf280 con 0x7fe9981029c0 2024-04-25T01:34:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe99659c640 1 -- 172.21.15.28:0/6767108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b182fcd980 msgr2=0x55b182fcfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe99659c640 1 --2- 172.21.15.28:0/6767108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b182fcd980 0x55b182fcfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe99659c640 1 -- 172.21.15.28:0/6767108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b182fbe2b0 msgr2=0x55b182fbe690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe99659c640 1 --2- 172.21.15.28:0/6767108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b182fbe2b0 0x55b182fbe690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.914+0000 7fe99659c640 1 -- 172.21.15.28:0/6767108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b182eef400 con 0x55b182fbec60 2024-04-25T01:34:42.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.915+0000 7fe99659c640 1 --2- 172.21.15.28:0/6767108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbec60 0x55b182fcd440 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe9880ccb20 tx=0x7fe988005130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56c957ee1e6af88 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.915+0000 7fe996d9d640 1 --2- 172.21.15.28:0/2947140877 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe9800a2d00 0x7fe9800a51c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe9900092f0 tx=0x7fe990009c30 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.915+0000 7fe984ff9640 1 -- 172.21.15.28:0/6767108 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98809bb00 con 0x55b182fbec60 2024-04-25T01:34:42.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.915+0000 7fe984ff9640 1 -- 172.21.15.28:0/6767108 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe98809b400 con 0x55b182fbec60 2024-04-25T01:34:42.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.915+0000 7fe984ff9640 1 -- 172.21.15.28:0/6767108 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98809b8c0 con 0x55b182fbec60 2024-04-25T01:34:42.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.915+0000 7fe99fd73740 1 -- 172.21.15.28:0/6767108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbec60 msgr2=0x55b182fcd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.915+0000 7fe99fd73740 1 --2- 172.21.15.28:0/6767108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbec60 0x55b182fcd440 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe9880ccb20 tx=0x7fe988005130 comp rx=0 tx=0).stop 2024-04-25T01:34:42.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.915+0000 7fe99fd73740 1 -- 172.21.15.28:0/6767108 shutdown_connections 2024-04-25T01:34:42.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.915+0000 7fe99fd73740 1 --2- 172.21.15.28:0/6767108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b182fcd980 0x55b182fcfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.915+0000 7fe99fd73740 1 --2- 172.21.15.28:0/6767108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbec60 0x55b182fcd440 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.915+0000 7fe99fd73740 1 --2- 172.21.15.28:0/6767108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b182fbe2b0 0x55b182fbe690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.915+0000 7fe99fd73740 1 -- 172.21.15.28:0/6767108 >> 172.21.15.28:0/6767108 conn(0x55b182fba8a0 msgr2=0x55b182fbac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.916+0000 7fe99fd73740 1 -- 172.21.15.28:0/6767108 shutdown_connections 2024-04-25T01:34:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.916+0000 7fe99fd73740 1 -- 172.21.15.28:0/6767108 wait complete. 2024-04-25T01:34:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.916+0000 7fe99fd73740 1 -- start start 2024-04-25T01:34:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.916+0000 7fe99fd73740 1 Processor -- start 2024-04-25T01:34:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.916+0000 7fe99fd73740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b182fbe2b0 0x55b182fcd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.916+0000 7fe99fd73740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbec60 0x55b182fca610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.916+0000 7fe99fd73740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b182fcd980 0x55b182fcabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.916+0000 7fe99fd73740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b182fd06e0 con 0x55b182fbec60 2024-04-25T01:34:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.916+0000 7fe99fd73740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b182edf050 con 0x55b182fbe2b0 2024-04-25T01:34:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.916+0000 7fe99fd73740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b182fd0860 con 0x55b182fcd980 2024-04-25T01:34:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.916+0000 7fe995d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbec60 0x55b182fca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.916+0000 7fe99fd73740 10 client.0 _ll_register_callbacks cb 0x55b182c36be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.916+0000 7fe995d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbec60 0x55b182fca610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41944/0 (socket says 172.21.15.28:41944) 2024-04-25T01:34:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.916+0000 7fe995d9b640 1 -- 172.21.15.28:0/664555877 learned_addr learned my addr 172.21.15.28:0/664555877 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.917+0000 7fe996d9d640 1 --2- 172.21.15.28:0/664555877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b182fcd980 0x55b182fcabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.917+0000 7fe99659c640 1 --2- 172.21.15.28:0/664555877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b182fbe2b0 0x55b182fcd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.917+0000 7fe995d9b640 1 -- 172.21.15.28:0/664555877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b182fcd980 msgr2=0x55b182fcabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.917+0000 7fe995d9b640 1 --2- 172.21.15.28:0/664555877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b182fcd980 0x55b182fcabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.917+0000 7fe995d9b640 1 -- 172.21.15.28:0/664555877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b182fbe2b0 msgr2=0x55b182fcd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.917+0000 7fe995d9b640 1 --2- 172.21.15.28:0/664555877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b182fbe2b0 0x55b182fcd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.917+0000 7fe995d9b640 1 -- 172.21.15.28:0/664555877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b182eb0dc0 con 0x55b182fbec60 2024-04-25T01:34:42.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.917+0000 7fe995d9b640 1 --2- 172.21.15.28:0/664555877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbec60 0x55b182fca610 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe98c009900 tx=0x7fe98c031860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da06e40e0cb4862d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.917+0000 7fe97dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:42.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.917+0000 7fe97dffb640 1 -- 172.21.15.28:0/664555877 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c035040 con 0x55b182fbec60 2024-04-25T01:34:42.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.917+0000 7fe97dffb640 1 -- 172.21.15.28:0/664555877 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fe98c004240 con 0x55b182fbec60 2024-04-25T01:34:42.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.917+0000 7fe99fd73740 1 -- 172.21.15.28:0/664555877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe98809b400 con 0x55b182fbec60 2024-04-25T01:34:42.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.917+0000 7fe97dffb640 1 -- 172.21.15.28:0/664555877 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe98c004440 con 0x55b182fbec60 2024-04-25T01:34:42.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.917+0000 7fe99fd73740 10 client.16191 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:42.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.917+0000 7fe99fd73740 1 -- 172.21.15.28:0/664555877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b182c373f0 con 0x55b182fbec60 2024-04-25T01:34:42.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.917+0000 7fe97dffb640 1 -- 172.21.15.28:0/664555877 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe98809b400 con 0x55b182fbec60 2024-04-25T01:34:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.918+0000 7fe97dffb640 1 -- 172.21.15.28:0/664555877 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe98c004b50 con 0x55b182fbec60 2024-04-25T01:34:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.918+0000 7fe99fd73740 10 client.16191 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.918+0000 7fe99fd73740 10 client.16191 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:34:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.918+0000 7fe99fd73740 20 client.16191 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.918+0000 7fe99fd73740 1 --2- 172.21.15.28:0/664555877 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b182fdda10 0x55b182fdfe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.918+0000 7fe99fd73740 4 client.16191 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.918+0000 7fe99fd73740 1 -- 172.21.15.28:0/664555877 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b182f30bb0 con 0x55b182fdda10 2024-04-25T01:34:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.918+0000 7fe99659c640 1 --2- 172.21.15.28:0/664555877 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b182fdda10 0x55b182fdfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.918+0000 7fe99659c640 1 --2- 172.21.15.28:0/664555877 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b182fdda10 0x55b182fdfe60 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fc36140d13e8893f server_cookie=dae4a36245b612fe in_seq=0 out_seq=0 2024-04-25T01:34:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.918+0000 7fe97dffb640 10 client.16191 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:34:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.919+0000 7fe97dffb640 1 -- 172.21.15.28:0/664555877 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b182f30bb0 con 0x55b182fdda10 2024-04-25T01:34:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.919+0000 7fe97dffb640 10 client.16191 handle_command_reply: tid=0 2024-04-25T01:34:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.919+0000 7fe99fd73740 1 client.16191 shutdown 2024-04-25T01:34:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.919+0000 7fe99fd73740 1 -- 172.21.15.28:0/664555877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbec60 msgr2=0x55b182fca610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.919+0000 7fe99fd73740 1 --2- 172.21.15.28:0/664555877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbec60 0x55b182fca610 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe98c009900 tx=0x7fe98c031860 comp rx=0 tx=0).stop 2024-04-25T01:34:42.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.919+0000 7fe99fd73740 1 -- 172.21.15.28:0/664555877 shutdown_connections 2024-04-25T01:34:42.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.919+0000 7fe99fd73740 1 --2- 172.21.15.28:0/664555877 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b182fdda10 0x55b182fdfe60 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.919+0000 7fe99fd73740 1 --2- 172.21.15.28:0/664555877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b182fcd980 0x55b182fcabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.919+0000 7fe99fd73740 1 --2- 172.21.15.28:0/664555877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbec60 0x55b182fca610 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.919+0000 7fe99fd73740 1 --2- 172.21.15.28:0/664555877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b182fbe2b0 0x55b182fcd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.919+0000 7fe97dffb640 0 client.16191 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:34:42.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.919+0000 7fe99fd73740 1 -- 172.21.15.28:0/664555877 >> 172.21.15.28:0/664555877 conn(0x55b182fba8a0 msgr2=0x55b182fc06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.919+0000 7fe99fd73740 1 -- 172.21.15.28:0/664555877 shutdown_connections 2024-04-25T01:34:42.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.919+0000 7fe99fd73740 1 -- 172.21.15.28:0/664555877 wait complete. 2024-04-25T01:34:42.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.919+0000 7fe99fd73740 20 client.16191 trim_cache size 0 max 16384 2024-04-25T01:34:42.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.933+0000 7fe99fd73740 1 Processor -- start 2024-04-25T01:34:42.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.933+0000 7fe99fd73740 1 -- start start 2024-04-25T01:34:42.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.933+0000 7fe99fd73740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbe2b0 0x55b182fd95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.933+0000 7fe99fd73740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b182fbec60 0x55b182fbac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.933+0000 7fe99fd73740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b182fcd980 0x55b182fcce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.933+0000 7fe99fd73740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9981ac9b0 con 0x55b182fbe2b0 2024-04-25T01:34:42.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.933+0000 7fe99fd73740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe9981ae4f0 con 0x55b182fbec60 2024-04-25T01:34:42.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.933+0000 7fe99fd73740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe9981ad080 con 0x55b182fcd980 2024-04-25T01:34:42.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.933+0000 7fe99659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbe2b0 0x55b182fd95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.933+0000 7fe99659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbe2b0 0x55b182fd95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41954/0 (socket says 172.21.15.28:41954) 2024-04-25T01:34:42.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.933+0000 7fe99659c640 1 -- 172.21.15.28:0/3218450727 learned_addr learned my addr 172.21.15.28:0/3218450727 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.933+0000 7fe99659c640 1 -- 172.21.15.28:0/3218450727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b182fcd980 msgr2=0x55b182fcce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.933+0000 7fe99659c640 1 --2- 172.21.15.28:0/3218450727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b182fcd980 0x55b182fcce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.933+0000 7fe99659c640 1 -- 172.21.15.28:0/3218450727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b182fbec60 msgr2=0x55b182fbac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.933+0000 7fe99659c640 1 --2- 172.21.15.28:0/3218450727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b182fbec60 0x55b182fbac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.933+0000 7fe99659c640 1 -- 172.21.15.28:0/3218450727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b182fc95c0 con 0x55b182fbe2b0 2024-04-25T01:34:42.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.933+0000 7fe99659c640 1 --2- 172.21.15.28:0/3218450727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbe2b0 0x55b182fd95e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe9880ccc30 tx=0x7fe988097040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0332d6f1f321b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.933+0000 7fe97ffff640 1 -- 172.21.15.28:0/3218450727 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9880976a0 con 0x55b182fbe2b0 2024-04-25T01:34:42.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.934+0000 7fe97ffff640 1 -- 172.21.15.28:0/3218450727 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe98809b400 con 0x55b182fbe2b0 2024-04-25T01:34:42.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.934+0000 7fe97ffff640 1 -- 172.21.15.28:0/3218450727 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98809b8c0 con 0x55b182fbe2b0 2024-04-25T01:34:42.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.934+0000 7fe99fd73740 1 -- 172.21.15.28:0/3218450727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbe2b0 msgr2=0x55b182fd95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.934+0000 7fe99fd73740 1 --2- 172.21.15.28:0/3218450727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbe2b0 0x55b182fd95e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe9880ccc30 tx=0x7fe988097040 comp rx=0 tx=0).stop 2024-04-25T01:34:42.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.934+0000 7fe99fd73740 1 -- 172.21.15.28:0/3218450727 shutdown_connections 2024-04-25T01:34:42.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.934+0000 7fe99fd73740 1 --2- 172.21.15.28:0/3218450727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b182fcd980 0x55b182fcce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.934+0000 7fe99fd73740 1 --2- 172.21.15.28:0/3218450727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b182fbec60 0x55b182fbac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.934+0000 7fe99fd73740 1 --2- 172.21.15.28:0/3218450727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fbe2b0 0x55b182fd95e0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.934+0000 7fe99fd73740 1 -- 172.21.15.28:0/3218450727 >> 172.21.15.28:0/3218450727 conn(0x55b182fba8a0 msgr2=0x55b182fc5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.934+0000 7fe99fd73740 1 -- 172.21.15.28:0/3218450727 shutdown_connections 2024-04-25T01:34:42.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.934+0000 7fe99fd73740 1 -- 172.21.15.28:0/3218450727 wait complete. 2024-04-25T01:34:42.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.934+0000 7fe99fd73740 1 -- start start 2024-04-25T01:34:42.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.934+0000 7fe99fd73740 1 Processor -- start 2024-04-25T01:34:42.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe99fd73740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b182fbe2b0 0x55b182fd1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe99fd73740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b182fbec60 0x55b182fd1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe99fd73740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fcd980 0x55b182fd21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe99fd73740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9981ade20 con 0x55b182fcd980 2024-04-25T01:34:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe99fd73740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe9981ad750 con 0x55b182fbe2b0 2024-04-25T01:34:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe99fd73740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe9981aa990 con 0x55b182fbec60 2024-04-25T01:34:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe99fd73740 10 client.0 _ll_register_callbacks cb 0x7fe98c005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe996d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fcd980 0x55b182fd21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe99659c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b182fbe2b0 0x55b182fd1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe996d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fcd980 0x55b182fd21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41962/0 (socket says 172.21.15.28:41962) 2024-04-25T01:34:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe996d9d640 1 -- 172.21.15.28:0/450099893 learned_addr learned my addr 172.21.15.28:0/450099893 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe995d9b640 1 --2- 172.21.15.28:0/450099893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b182fbec60 0x55b182fd1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe996d9d640 1 -- 172.21.15.28:0/450099893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b182fbec60 msgr2=0x55b182fd1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe996d9d640 1 --2- 172.21.15.28:0/450099893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b182fbec60 0x55b182fd1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe996d9d640 1 -- 172.21.15.28:0/450099893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b182fbe2b0 msgr2=0x55b182fd1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe996d9d640 1 --2- 172.21.15.28:0/450099893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b182fbe2b0 0x55b182fd1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe996d9d640 1 -- 172.21.15.28:0/450099893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b182fbea00 con 0x55b182fcd980 2024-04-25T01:34:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe996d9d640 1 --2- 172.21.15.28:0/450099893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fcd980 0x55b182fd21e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe990014340 tx=0x7fe990004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=665df8c3ad77a442 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe97dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.935+0000 7fe97dffb640 1 -- 172.21.15.28:0/450099893 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe990026070 con 0x55b182fcd980 2024-04-25T01:34:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.936+0000 7fe97dffb640 1 -- 172.21.15.28:0/450099893 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fe990004d20 con 0x55b182fcd980 2024-04-25T01:34:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.936+0000 7fe99fd73740 1 -- 172.21.15.28:0/450099893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe98809b400 con 0x55b182fcd980 2024-04-25T01:34:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.936+0000 7fe97dffb640 1 -- 172.21.15.28:0/450099893 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe990033050 con 0x55b182fcd980 2024-04-25T01:34:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.936+0000 7fe99fd73740 10 client.16197 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.936+0000 7fe99fd73740 1 -- 172.21.15.28:0/450099893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b182d017d0 con 0x55b182fcd980 2024-04-25T01:34:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.936+0000 7fe97dffb640 1 -- 172.21.15.28:0/450099893 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe98809b400 con 0x55b182fcd980 2024-04-25T01:34:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.936+0000 7fe97dffb640 1 -- 172.21.15.28:0/450099893 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe99000f050 con 0x55b182fcd980 2024-04-25T01:34:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.936+0000 7fe99fd73740 10 client.16197 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.936+0000 7fe99fd73740 10 client.16197 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:34:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.936+0000 7fe99fd73740 20 client.16197 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.936+0000 7fe99fd73740 1 --2- 172.21.15.28:0/450099893 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b183044910 0x55b183046d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.936+0000 7fe99fd73740 4 client.16197 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:34:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.936+0000 7fe99fd73740 1 -- 172.21.15.28:0/450099893 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b182f2af00 con 0x55b183044910 2024-04-25T01:34:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.936+0000 7fe99659c640 1 --2- 172.21.15.28:0/450099893 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b183044910 0x55b183046d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.937+0000 7fe97dffb640 10 client.16197 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:34:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.937+0000 7fe99659c640 1 --2- 172.21.15.28:0/450099893 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b183044910 0x55b183046d30 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4cddcbebb777ca04 server_cookie=2dfa21c0c9943740 in_seq=0 out_seq=0 2024-04-25T01:34:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.937+0000 7fe97dffb640 1 -- 172.21.15.28:0/450099893 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b182f2af00 con 0x55b183044910 2024-04-25T01:34:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.937+0000 7fe97dffb640 10 client.16197 handle_command_reply: tid=0 2024-04-25T01:34:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.937+0000 7fe99fd73740 1 client.16197 shutdown 2024-04-25T01:34:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.937+0000 7fe99fd73740 1 -- 172.21.15.28:0/450099893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fcd980 msgr2=0x55b182fd21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.937+0000 7fe99fd73740 1 --2- 172.21.15.28:0/450099893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fcd980 0x55b182fd21e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe990014340 tx=0x7fe990004a40 comp rx=0 tx=0).stop 2024-04-25T01:34:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.937+0000 7fe99fd73740 1 -- 172.21.15.28:0/450099893 shutdown_connections 2024-04-25T01:34:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.937+0000 7fe99fd73740 1 --2- 172.21.15.28:0/450099893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b182fcd980 0x55b182fd21e0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.937+0000 7fe99fd73740 1 --2- 172.21.15.28:0/450099893 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b183044910 0x55b183046d30 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.937+0000 7fe99fd73740 1 --2- 172.21.15.28:0/450099893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b182fbec60 0x55b182fd1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.937+0000 7fe99fd73740 1 --2- 172.21.15.28:0/450099893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b182fbe2b0 0x55b182fd1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.937+0000 7fe99fd73740 1 -- 172.21.15.28:0/450099893 >> 172.21.15.28:0/450099893 conn(0x55b182fba8a0 msgr2=0x55b182fc5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.937+0000 7fe97dffb640 0 client.16197 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:34:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.937+0000 7fe99fd73740 1 -- 172.21.15.28:0/450099893 shutdown_connections 2024-04-25T01:34:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.937+0000 7fe99fd73740 1 -- 172.21.15.28:0/450099893 wait complete. 2024-04-25T01:34:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.937+0000 7fe99fd73740 20 client.16197 trim_cache size 0 max 16384 2024-04-25T01:34:42.920 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:34:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.938+0000 7fe99cf89640 1 -- 172.21.15.28:0/2947140877 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe9800a2d00 msgr2=0x7fe9800a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.938+0000 7fe99cf89640 1 --2- 172.21.15.28:0/2947140877 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe9800a2d00 0x7fe9800a51c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe9900092f0 tx=0x7fe990009c30 comp rx=0 tx=0).stop 2024-04-25T01:34:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.938+0000 7fe99cf89640 1 -- 172.21.15.28:0/2947140877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9981029c0 msgr2=0x7fe998198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.938+0000 7fe99cf89640 1 --2- 172.21.15.28:0/2947140877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9981029c0 0x7fe998198020 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe98800d950 tx=0x7fe98800de10 comp rx=0 tx=0).stop 2024-04-25T01:34:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.938+0000 7fe99cf89640 1 -- 172.21.15.28:0/2947140877 shutdown_connections 2024-04-25T01:34:42.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.938+0000 7fe99cf89640 1 --2- 172.21.15.28:0/2947140877 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe9800a2d00 0x7fe9800a51c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.938+0000 7fe99cf89640 1 --2- 172.21.15.28:0/2947140877 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe99810bd00 0x7fe99819c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.938+0000 7fe99cf89640 1 --2- 172.21.15.28:0/2947140877 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9981032e0 0x7fe998198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.938+0000 7fe99cf89640 1 --2- 172.21.15.28:0/2947140877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9981029c0 0x7fe998198020 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.938+0000 7fe99cf89640 1 -- 172.21.15.28:0/2947140877 >> 172.21.15.28:0/2947140877 conn(0x7fe998077740 msgr2=0x7fe9980fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.938+0000 7fe99cf89640 1 -- 172.21.15.28:0/2947140877 shutdown_connections 2024-04-25T01:34:42.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:42.938+0000 7fe99cf89640 1 -- 172.21.15.28:0/2947140877 wait complete. 2024-04-25T01:34:42.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:34:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.052+0000 7fc14b716640 1 -- 172.21.15.28:0/585813238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc144101710 msgr2=0x7fc144101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.052+0000 7fc14b716640 1 --2- 172.21.15.28:0/585813238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc144101710 0x7fc144101b90 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fc134009930 tx=0x7fc13402a420 comp rx=0 tx=0).stop 2024-04-25T01:34:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.053+0000 7fc14b716640 1 -- 172.21.15.28:0/585813238 shutdown_connections 2024-04-25T01:34:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.053+0000 7fc14b716640 1 --2- 172.21.15.28:0/585813238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc1441020d0 0x7fc14410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.053+0000 7fc14b716640 1 --2- 172.21.15.28:0/585813238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc144101710 0x7fc144101b90 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.053+0000 7fc14b716640 1 --2- 172.21.15.28:0/585813238 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc144107720 0x7fc144107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.053+0000 7fc14b716640 1 -- 172.21.15.28:0/585813238 >> 172.21.15.28:0/585813238 conn(0x7fc1440fd180 msgr2=0x7fc1440ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.053+0000 7fc14b716640 1 -- 172.21.15.28:0/585813238 shutdown_connections 2024-04-25T01:34:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.053+0000 7fc14b716640 1 -- 172.21.15.28:0/585813238 wait complete. 2024-04-25T01:34:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.053+0000 7fc14b716640 1 Processor -- start 2024-04-25T01:34:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.053+0000 7fc14b716640 1 -- start start 2024-04-25T01:34:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.053+0000 7fc14b716640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc144101710 0x7fc144198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.053+0000 7fc14b716640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1441020d0 0x7fc1441985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.053+0000 7fc14b716640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc144107720 0x7fc14419c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.053+0000 7fc14b716640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc144111af0 con 0x7fc144101710 2024-04-25T01:34:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.053+0000 7fc14b716640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc144111970 con 0x7fc144107720 2024-04-25T01:34:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.053+0000 7fc14b716640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc144111c70 con 0x7fc1441020d0 2024-04-25T01:34:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.053+0000 7fc1494b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc144101710 0x7fc144198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.053+0000 7fc1494b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc144101710 0x7fc144198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41974/0 (socket says 172.21.15.28:41974) 2024-04-25T01:34:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.054+0000 7fc1494b2640 1 -- 172.21.15.28:0/3483726349 learned_addr learned my addr 172.21.15.28:0/3483726349 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.054+0000 7fc149cb3640 1 --2- 172.21.15.28:0/3483726349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc144107720 0x7fc14419c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.054+0000 7fc148cb1640 1 --2- 172.21.15.28:0/3483726349 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1441020d0 0x7fc1441985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.054+0000 7fc1494b2640 1 -- 172.21.15.28:0/3483726349 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1441020d0 msgr2=0x7fc1441985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.054+0000 7fc1494b2640 1 --2- 172.21.15.28:0/3483726349 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1441020d0 0x7fc1441985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.054+0000 7fc1494b2640 1 -- 172.21.15.28:0/3483726349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc144107720 msgr2=0x7fc14419c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.054+0000 7fc1494b2640 1 --2- 172.21.15.28:0/3483726349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc144107720 0x7fc14419c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.054+0000 7fc1494b2640 1 -- 172.21.15.28:0/3483726349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc14419d080 con 0x7fc144101710 2024-04-25T01:34:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.054+0000 7fc149cb3640 1 --2- 172.21.15.28:0/3483726349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc144107720 0x7fc14419c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.054+0000 7fc1494b2640 1 --2- 172.21.15.28:0/3483726349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc144101710 0x7fc144198090 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fc14000d950 tx=0x7fc14000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.054+0000 7fc13a7fc640 1 -- 172.21.15.28:0/3483726349 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc140017030 con 0x7fc144101710 2024-04-25T01:34:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.054+0000 7fc14b716640 1 -- 172.21.15.28:0/3483726349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc14419d370 con 0x7fc144101710 2024-04-25T01:34:43.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.054+0000 7fc13a7fc640 1 -- 172.21.15.28:0/3483726349 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc140004020 con 0x7fc144101710 2024-04-25T01:34:43.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.054+0000 7fc13a7fc640 1 -- 172.21.15.28:0/3483726349 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1400047a0 con 0x7fc144101710 2024-04-25T01:34:43.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.054+0000 7fc14b716640 1 -- 172.21.15.28:0/3483726349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1441a4c00 con 0x7fc144101710 2024-04-25T01:34:43.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.056+0000 7fc13a7fc640 1 -- 172.21.15.28:0/3483726349 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc1400041d0 con 0x7fc144101710 2024-04-25T01:34:43.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.057+0000 7fc14e41c740 1 Processor -- start 2024-04-25T01:34:43.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.057+0000 7fc14e41c740 1 -- start start 2024-04-25T01:34:43.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.057+0000 7fc14e41c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38b580 0x55db8e38b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.057+0000 7fc14e41c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e38bea0 0x55db8e396030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.057+0000 7fc14e41c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e396570 0x55db8e398960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.057+0000 7fc14e41c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db8e2f0670 con 0x55db8e396570 2024-04-25T01:34:43.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.057+0000 7fc14e41c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55db8e298970 con 0x55db8e38b580 2024-04-25T01:34:43.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.057+0000 7fc14e41c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55db8e2ec630 con 0x55db8e38bea0 2024-04-25T01:34:43.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.058+0000 7fc13a7fc640 1 --2- 172.21.15.28:0/3483726349 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc1300a2d00 0x7fc1300a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.058+0000 7fc148cb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e396570 0x55db8e398960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.058+0000 7fc148cb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e396570 0x55db8e398960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41984/0 (socket says 172.21.15.28:41984) 2024-04-25T01:34:43.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.058+0000 7fc148cb1640 1 -- 172.21.15.28:0/573773467 learned_addr learned my addr 172.21.15.28:0/573773467 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:43.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.058+0000 7fc149cb3640 1 --2- 172.21.15.28:0/573773467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38b580 0x55db8e38b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.058+0000 7fc1494b2640 1 --2- 172.21.15.28:0/573773467 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e38bea0 0x55db8e396030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.058+0000 7fc149cb3640 1 --2- 172.21.15.28:0/3483726349 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc1300a2d00 0x7fc1300a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.058+0000 7fc13a7fc640 1 -- 172.21.15.28:0/3483726349 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc1400cf220 con 0x7fc144101710 2024-04-25T01:34:43.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.058+0000 7fc148cb1640 1 -- 172.21.15.28:0/573773467 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e38bea0 msgr2=0x55db8e396030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:43.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.058+0000 7fc148cb1640 1 --2- 172.21.15.28:0/573773467 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e38bea0 0x55db8e396030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.058+0000 7fc148cb1640 1 -- 172.21.15.28:0/573773467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38b580 msgr2=0x55db8e38b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:43.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.058+0000 7fc148cb1640 1 --2- 172.21.15.28:0/573773467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38b580 0x55db8e38b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.058+0000 7fc148cb1640 1 -- 172.21.15.28:0/573773467 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db8e2bc400 con 0x55db8e396570 2024-04-25T01:34:43.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.058+0000 7fc149cb3640 1 --2- 172.21.15.28:0/3483726349 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc1300a2d00 0x7fc1300a51c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc13c009270 tx=0x7fc13c009500 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:43.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.058+0000 7fc148cb1640 1 --2- 172.21.15.28:0/573773467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e396570 0x55db8e398960 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc13402a940 tx=0x7fc134005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bbadb44c2381f74 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:43.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.058+0000 7fc12f7fe640 1 -- 172.21.15.28:0/573773467 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13403f070 con 0x55db8e396570 2024-04-25T01:34:43.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.058+0000 7fc12f7fe640 1 -- 172.21.15.28:0/573773467 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc134004030 con 0x55db8e396570 2024-04-25T01:34:43.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.058+0000 7fc12f7fe640 1 -- 172.21.15.28:0/573773467 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134035070 con 0x55db8e396570 2024-04-25T01:34:43.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.059+0000 7fc14e41c740 1 -- 172.21.15.28:0/573773467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e396570 msgr2=0x55db8e398960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:43.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.059+0000 7fc14e41c740 1 --2- 172.21.15.28:0/573773467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e396570 0x55db8e398960 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc13402a940 tx=0x7fc134005dd0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.059+0000 7fc14e41c740 1 -- 172.21.15.28:0/573773467 shutdown_connections 2024-04-25T01:34:43.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.059+0000 7fc14e41c740 1 --2- 172.21.15.28:0/573773467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e396570 0x55db8e398960 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.059+0000 7fc14e41c740 1 --2- 172.21.15.28:0/573773467 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e38bea0 0x55db8e396030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.059+0000 7fc14e41c740 1 --2- 172.21.15.28:0/573773467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38b580 0x55db8e38b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.059+0000 7fc14e41c740 1 -- 172.21.15.28:0/573773467 >> 172.21.15.28:0/573773467 conn(0x55db8e3878a0 msgr2=0x55db8e387c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:43.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.059+0000 7fc14e41c740 1 -- 172.21.15.28:0/573773467 shutdown_connections 2024-04-25T01:34:43.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.059+0000 7fc14e41c740 1 -- 172.21.15.28:0/573773467 wait complete. 2024-04-25T01:34:43.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.059+0000 7fc14e41c740 1 -- start start 2024-04-25T01:34:43.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.060+0000 7fc14e41c740 1 Processor -- start 2024-04-25T01:34:43.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.060+0000 7fc14e41c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38b580 0x55db8e395e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.060+0000 7fc14e41c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e38bea0 0x55db8e393170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.060+0000 7fc14e41c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e396570 0x55db8e3936b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.060+0000 7fc14e41c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db8e399300 con 0x55db8e38bea0 2024-04-25T01:34:43.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.060+0000 7fc14e41c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55db8e2ac050 con 0x55db8e38b580 2024-04-25T01:34:43.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.060+0000 7fc14e41c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55db8e399480 con 0x55db8e396570 2024-04-25T01:34:43.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.060+0000 7fc14e41c740 10 client.0 _ll_register_callbacks cb 0x55db8e003be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:43.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.060+0000 7fc148cb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e38bea0 0x55db8e393170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.060+0000 7fc148cb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e38bea0 0x55db8e393170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41986/0 (socket says 172.21.15.28:41986) 2024-04-25T01:34:43.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.060+0000 7fc148cb1640 1 -- 172.21.15.28:0/1238297231 learned_addr learned my addr 172.21.15.28:0/1238297231 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:43.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.060+0000 7fc1494b2640 1 --2- 172.21.15.28:0/1238297231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38b580 0x55db8e395e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.060+0000 7fc149cb3640 1 --2- 172.21.15.28:0/1238297231 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e396570 0x55db8e3936b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.060+0000 7fc148cb1640 1 -- 172.21.15.28:0/1238297231 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e396570 msgr2=0x55db8e3936b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:43.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.060+0000 7fc148cb1640 1 --2- 172.21.15.28:0/1238297231 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e396570 0x55db8e3936b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.060+0000 7fc148cb1640 1 -- 172.21.15.28:0/1238297231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38b580 msgr2=0x55db8e395e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:43.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.060+0000 7fc148cb1640 1 --2- 172.21.15.28:0/1238297231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38b580 0x55db8e395e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.060+0000 7fc148cb1640 1 -- 172.21.15.28:0/1238297231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db8e27ddc0 con 0x55db8e38bea0 2024-04-25T01:34:43.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.061+0000 7fc148cb1640 1 --2- 172.21.15.28:0/1238297231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e38bea0 0x55db8e393170 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fc134006010 tx=0x7fc1340056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe8b4c1adc49cff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:43.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.061+0000 7fc12cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:43.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.061+0000 7fc12cff9640 1 -- 172.21.15.28:0/1238297231 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134035070 con 0x55db8e38bea0 2024-04-25T01:34:43.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.061+0000 7fc12cff9640 1 -- 172.21.15.28:0/1238297231 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc1340315b0 con 0x55db8e38bea0 2024-04-25T01:34:43.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.061+0000 7fc14e41c740 1 -- 172.21.15.28:0/1238297231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc134004030 con 0x55db8e38bea0 2024-04-25T01:34:43.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.061+0000 7fc12cff9640 1 -- 172.21.15.28:0/1238297231 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc134055050 con 0x55db8e38bea0 2024-04-25T01:34:43.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.061+0000 7fc14e41c740 10 client.16209 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:43.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.061+0000 7fc14e41c740 1 -- 172.21.15.28:0/1238297231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55db8e0043f0 con 0x55db8e38bea0 2024-04-25T01:34:43.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.061+0000 7fc12cff9640 1 -- 172.21.15.28:0/1238297231 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc134004030 con 0x55db8e38bea0 2024-04-25T01:34:43.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.061+0000 7fc12cff9640 1 -- 172.21.15.28:0/1238297231 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc1340046c0 con 0x55db8e38bea0 2024-04-25T01:34:43.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.061+0000 7fc14e41c740 10 client.16209 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:43.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.061+0000 7fc14e41c740 10 client.16209 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:34:43.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.061+0000 7fc14e41c740 20 client.16209 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:43.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.061+0000 7fc14e41c740 1 --2- 172.21.15.28:0/1238297231 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55db8e3a63a0 0x55db8e3a8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.061+0000 7fc14e41c740 4 client.16209 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:43.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.061+0000 7fc14e41c740 1 -- 172.21.15.28:0/1238297231 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db8e2fdbb0 con 0x55db8e3a63a0 2024-04-25T01:34:43.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.062+0000 7fc1494b2640 1 --2- 172.21.15.28:0/1238297231 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55db8e3a63a0 0x55db8e3a8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.062+0000 7fc1494b2640 1 --2- 172.21.15.28:0/1238297231 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55db8e3a63a0 0x55db8e3a8790 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dca1cc7212672367 server_cookie=6c51902e8c5d304b in_seq=0 out_seq=0 2024-04-25T01:34:43.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.062+0000 7fc12cff9640 10 client.16209 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:34:43.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.063+0000 7fc12cff9640 1 -- 172.21.15.28:0/1238297231 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55db8e2fdbb0 con 0x55db8e3a63a0 2024-04-25T01:34:43.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.063+0000 7fc12cff9640 10 client.16209 handle_command_reply: tid=0 2024-04-25T01:34:43.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.063+0000 7fc14e41c740 1 client.16209 shutdown 2024-04-25T01:34:43.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.063+0000 7fc14e41c740 1 -- 172.21.15.28:0/1238297231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e38bea0 msgr2=0x55db8e393170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:43.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.063+0000 7fc14e41c740 1 --2- 172.21.15.28:0/1238297231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e38bea0 0x55db8e393170 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fc134006010 tx=0x7fc1340056d0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.063+0000 7fc14e41c740 1 -- 172.21.15.28:0/1238297231 shutdown_connections 2024-04-25T01:34:43.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.063+0000 7fc14e41c740 1 --2- 172.21.15.28:0/1238297231 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55db8e3a63a0 0x55db8e3a8790 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.063+0000 7fc14e41c740 1 --2- 172.21.15.28:0/1238297231 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e396570 0x55db8e3936b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.063+0000 7fc14e41c740 1 --2- 172.21.15.28:0/1238297231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e38bea0 0x55db8e393170 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.063+0000 7fc12cff9640 0 client.16209 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:34:43.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.063+0000 7fc14e41c740 1 --2- 172.21.15.28:0/1238297231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38b580 0x55db8e395e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.063+0000 7fc14e41c740 1 -- 172.21.15.28:0/1238297231 >> 172.21.15.28:0/1238297231 conn(0x55db8e3878a0 msgr2=0x55db8e38d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:43.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.063+0000 7fc14e41c740 1 -- 172.21.15.28:0/1238297231 shutdown_connections 2024-04-25T01:34:43.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.063+0000 7fc14e41c740 1 -- 172.21.15.28:0/1238297231 wait complete. 2024-04-25T01:34:43.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.063+0000 7fc14e41c740 20 client.16209 trim_cache size 0 max 16384 2024-04-25T01:34:43.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.077+0000 7fc14e41c740 1 Processor -- start 2024-04-25T01:34:43.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.077+0000 7fc14e41c740 1 -- start start 2024-04-25T01:34:43.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.077+0000 7fc14e41c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e38b580 0x55db8e3a21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.077+0000 7fc14e41c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38bea0 0x55db8e38e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.077+0000 7fc14e41c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e396570 0x55db8e3959b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.077+0000 7fc14e41c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441aeb30 con 0x55db8e396570 2024-04-25T01:34:43.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.077+0000 7fc14e41c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441b0670 con 0x55db8e38bea0 2024-04-25T01:34:43.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.077+0000 7fc14e41c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441af200 con 0x55db8e38b580 2024-04-25T01:34:43.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.077+0000 7fc149cb3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e396570 0x55db8e3959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.077+0000 7fc148cb1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38bea0 0x55db8e38e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.077+0000 7fc149cb3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e396570 0x55db8e3959b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41998/0 (socket says 172.21.15.28:41998) 2024-04-25T01:34:43.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.077+0000 7fc1494b2640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e38b580 0x55db8e3a21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.077+0000 7fc149cb3640 1 -- 172.21.15.28:0/3488582854 learned_addr learned my addr 172.21.15.28:0/3488582854 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:43.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.078+0000 7fc1494b2640 1 -- 172.21.15.28:0/3488582854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38bea0 msgr2=0x55db8e38e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.078+0000 7fc1494b2640 1 --2- 172.21.15.28:0/3488582854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38bea0 0x55db8e38e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.078+0000 7fc1494b2640 1 -- 172.21.15.28:0/3488582854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e396570 msgr2=0x55db8e3959b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.078+0000 7fc1494b2640 1 --2- 172.21.15.28:0/3488582854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e396570 0x55db8e3959b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.078+0000 7fc1494b2640 1 -- 172.21.15.28:0/3488582854 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db8e0ce7d0 con 0x55db8e38b580 2024-04-25T01:34:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.078+0000 7fc1494b2640 1 --2- 172.21.15.28:0/3488582854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e38b580 0x55db8e3a21c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc1400cc5d0 tx=0x7fc1400207f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33a1b5c72cc418c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.078+0000 7fc12effd640 1 -- 172.21.15.28:0/3488582854 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14009baa0 con 0x55db8e38b580 2024-04-25T01:34:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.078+0000 7fc12effd640 1 -- 172.21.15.28:0/3488582854 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc14009b3a0 con 0x55db8e38b580 2024-04-25T01:34:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.078+0000 7fc12effd640 1 -- 172.21.15.28:0/3488582854 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1400cc850 con 0x55db8e38b580 2024-04-25T01:34:43.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.079+0000 7fc14e41c740 1 -- 172.21.15.28:0/3488582854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e38b580 msgr2=0x55db8e3a21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:43.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.079+0000 7fc14e41c740 1 --2- 172.21.15.28:0/3488582854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e38b580 0x55db8e3a21c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc1400cc5d0 tx=0x7fc1400207f0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.079+0000 7fc14e41c740 1 -- 172.21.15.28:0/3488582854 shutdown_connections 2024-04-25T01:34:43.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.079+0000 7fc14e41c740 1 --2- 172.21.15.28:0/3488582854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e396570 0x55db8e3959b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.079+0000 7fc14e41c740 1 --2- 172.21.15.28:0/3488582854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38bea0 0x55db8e38e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.079+0000 7fc14e41c740 1 --2- 172.21.15.28:0/3488582854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e38b580 0x55db8e3a21c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.079+0000 7fc14e41c740 1 -- 172.21.15.28:0/3488582854 >> 172.21.15.28:0/3488582854 conn(0x55db8e3878a0 msgr2=0x55db8e38dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:43.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.079+0000 7fc14e41c740 1 -- 172.21.15.28:0/3488582854 shutdown_connections 2024-04-25T01:34:43.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.079+0000 7fc14e41c740 1 -- 172.21.15.28:0/3488582854 wait complete. 2024-04-25T01:34:43.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.079+0000 7fc14e41c740 1 -- start start 2024-04-25T01:34:43.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.079+0000 7fc14e41c740 1 Processor -- start 2024-04-25T01:34:43.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.080+0000 7fc14e41c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e38b580 0x55db8e39eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.080+0000 7fc14e41c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38bea0 0x55db8e399e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.080+0000 7fc14e41c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e396570 0x55db8e39a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.080+0000 7fc14e41c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441affa0 con 0x55db8e38b580 2024-04-25T01:34:43.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.080+0000 7fc14e41c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441af8d0 con 0x55db8e38bea0 2024-04-25T01:34:43.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.080+0000 7fc14e41c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc1441acb10 con 0x55db8e396570 2024-04-25T01:34:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.080+0000 7fc14e41c740 10 client.0 _ll_register_callbacks cb 0x7fc13c012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.080+0000 7fc1494b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e38b580 0x55db8e39eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.080+0000 7fc1494b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e38b580 0x55db8e39eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42000/0 (socket says 172.21.15.28:42000) 2024-04-25T01:34:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.080+0000 7fc1494b2640 1 -- 172.21.15.28:0/2290632180 learned_addr learned my addr 172.21.15.28:0/2290632180 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.080+0000 7fc149cb3640 1 --2- 172.21.15.28:0/2290632180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e396570 0x55db8e39a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.080+0000 7fc148cb1640 1 --2- 172.21.15.28:0/2290632180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38bea0 0x55db8e399e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.080+0000 7fc149cb3640 1 -- 172.21.15.28:0/2290632180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38bea0 msgr2=0x55db8e399e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.080+0000 7fc149cb3640 1 --2- 172.21.15.28:0/2290632180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38bea0 0x55db8e399e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.080+0000 7fc149cb3640 1 -- 172.21.15.28:0/2290632180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e38b580 msgr2=0x55db8e39eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.080+0000 7fc149cb3640 1 --2- 172.21.15.28:0/2290632180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e38b580 0x55db8e39eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.080+0000 7fc149cb3640 1 -- 172.21.15.28:0/2290632180 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db8e39abc0 con 0x55db8e396570 2024-04-25T01:34:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.080+0000 7fc1494b2640 1 --2- 172.21.15.28:0/2290632180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e38b580 0x55db8e39eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.081+0000 7fc149cb3640 1 --2- 172.21.15.28:0/2290632180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e396570 0x55db8e39a390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55db8e38be70 tx=0x7fc13c004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7532ea0d6e7e50dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.081+0000 7fc12cff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-25T01:34:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.081+0000 7fc12cff9640 1 -- 172.21.15.28:0/2290632180 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c004b20 con 0x55db8e396570 2024-04-25T01:34:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.081+0000 7fc12cff9640 1 -- 172.21.15.28:0/2290632180 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc13c01b070 con 0x55db8e396570 2024-04-25T01:34:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.081+0000 7fc14e41c740 1 -- 172.21.15.28:0/2290632180 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc14009b3a0 con 0x55db8e396570 2024-04-25T01:34:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.081+0000 7fc12cff9640 1 -- 172.21.15.28:0/2290632180 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc13c004cc0 con 0x55db8e396570 2024-04-25T01:34:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.081+0000 7fc14e41c740 10 client.25657 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.081+0000 7fc14e41c740 1 -- 172.21.15.28:0/2290632180 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55db8e40c6c0 con 0x55db8e396570 2024-04-25T01:34:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.081+0000 7fc12cff9640 1 -- 172.21.15.28:0/2290632180 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc14009b3a0 con 0x55db8e396570 2024-04-25T01:34:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.081+0000 7fc12cff9640 1 -- 172.21.15.28:0/2290632180 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc13c02b460 con 0x55db8e396570 2024-04-25T01:34:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.081+0000 7fc14e41c740 10 client.25657 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.081+0000 7fc14e41c740 10 client.25657 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:34:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.081+0000 7fc14e41c740 20 client.25657 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.081+0000 7fc14e41c740 1 --2- 172.21.15.28:0/2290632180 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55db8e40ce30 0x55db8e40f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.081+0000 7fc14e41c740 4 client.25657 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:34:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.081+0000 7fc14e41c740 1 -- 172.21.15.28:0/2290632180 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55db8e393a40 con 0x55db8e40ce30 2024-04-25T01:34:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.082+0000 7fc1494b2640 1 --2- 172.21.15.28:0/2290632180 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55db8e40ce30 0x55db8e40f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.082+0000 7fc1494b2640 1 --2- 172.21.15.28:0/2290632180 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55db8e40ce30 0x55db8e40f250 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c2fdba1b3396f54e server_cookie=a2e3c08aa15f3bcf in_seq=0 out_seq=0 2024-04-25T01:34:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.082+0000 7fc12cff9640 10 client.25657 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:34:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.082+0000 7fc12cff9640 1 -- 172.21.15.28:0/2290632180 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55db8e393a40 con 0x55db8e40ce30 2024-04-25T01:34:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.082+0000 7fc12cff9640 10 client.25657 handle_command_reply: tid=0 2024-04-25T01:34:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.082+0000 7fc14e41c740 1 client.25657 shutdown 2024-04-25T01:34:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.082+0000 7fc14e41c740 1 -- 172.21.15.28:0/2290632180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e396570 msgr2=0x55db8e39a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.082+0000 7fc14e41c740 1 --2- 172.21.15.28:0/2290632180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e396570 0x55db8e39a390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55db8e38be70 tx=0x7fc13c004ab0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.083+0000 7fc14e41c740 1 -- 172.21.15.28:0/2290632180 shutdown_connections 2024-04-25T01:34:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.083+0000 7fc14e41c740 1 --2- 172.21.15.28:0/2290632180 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55db8e40ce30 0x55db8e40f250 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.083+0000 7fc14e41c740 1 --2- 172.21.15.28:0/2290632180 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db8e396570 0x55db8e39a390 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.083+0000 7fc14e41c740 1 --2- 172.21.15.28:0/2290632180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db8e38bea0 0x55db8e399e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.083+0000 7fc12cff9640 0 client.25657 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:34:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.083+0000 7fc14e41c740 1 --2- 172.21.15.28:0/2290632180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db8e38b580 0x55db8e39eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.083+0000 7fc14e41c740 1 -- 172.21.15.28:0/2290632180 >> 172.21.15.28:0/2290632180 conn(0x55db8e3878a0 msgr2=0x55db8e386b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:43.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.083+0000 7fc14e41c740 1 -- 172.21.15.28:0/2290632180 shutdown_connections 2024-04-25T01:34:43.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.083+0000 7fc14e41c740 1 -- 172.21.15.28:0/2290632180 wait complete. 2024-04-25T01:34:43.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.083+0000 7fc14e41c740 20 client.25657 trim_cache size 0 max 16384 2024-04-25T01:34:43.065 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:34:43.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.083+0000 7fc14b716640 1 -- 172.21.15.28:0/3483726349 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc1300a2d00 msgr2=0x7fc1300a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:43.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.083+0000 7fc14b716640 1 --2- 172.21.15.28:0/3483726349 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc1300a2d00 0x7fc1300a51c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc13c009270 tx=0x7fc13c009500 comp rx=0 tx=0).stop 2024-04-25T01:34:43.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.083+0000 7fc14b716640 1 -- 172.21.15.28:0/3483726349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc144101710 msgr2=0x7fc144198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.083+0000 7fc14b716640 1 --2- 172.21.15.28:0/3483726349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc144101710 0x7fc144198090 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fc14000d950 tx=0x7fc14000de10 comp rx=0 tx=0).stop 2024-04-25T01:34:43.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:42 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/903031445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:34:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.084+0000 7fc14b716640 1 -- 172.21.15.28:0/3483726349 shutdown_connections 2024-04-25T01:34:43.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.084+0000 7fc14b716640 1 --2- 172.21.15.28:0/3483726349 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc1300a2d00 0x7fc1300a51c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.084+0000 7fc14b716640 1 --2- 172.21.15.28:0/3483726349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc144107720 0x7fc14419c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.084+0000 7fc14b716640 1 --2- 172.21.15.28:0/3483726349 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc1441020d0 0x7fc1441985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.084+0000 7fc14b716640 1 --2- 172.21.15.28:0/3483726349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc144101710 0x7fc144198090 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.084+0000 7fc14b716640 1 -- 172.21.15.28:0/3483726349 >> 172.21.15.28:0/3483726349 conn(0x7fc1440fd180 msgr2=0x7fc1440fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:43.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.084+0000 7fc14b716640 1 -- 172.21.15.28:0/3483726349 shutdown_connections 2024-04-25T01:34:43.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.084+0000 7fc14b716640 1 -- 172.21.15.28:0/3483726349 wait complete. 2024-04-25T01:34:43.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:34:43.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.214+0000 7fb564b61640 1 -- 172.21.15.28:0/2067929853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb560100fd0 msgr2=0x7fb56010d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:43.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.214+0000 7fb564b61640 1 --2- 172.21.15.28:0/2067929853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb560100fd0 0x7fb56010d4e0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fb558009cd0 tx=0x7fb55802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.215+0000 7fb564b61640 1 -- 172.21.15.28:0/2067929853 shutdown_connections 2024-04-25T01:34:43.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.215+0000 7fb564b61640 1 --2- 172.21.15.28:0/2067929853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb560100fd0 0x7fb56010d4e0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.215+0000 7fb564b61640 1 --2- 172.21.15.28:0/2067929853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb560100610 0x7fb560100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.215+0000 7fb564b61640 1 --2- 172.21.15.28:0/2067929853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb5601066b0 0x7fb560106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.215+0000 7fb564b61640 1 -- 172.21.15.28:0/2067929853 >> 172.21.15.28:0/2067929853 conn(0x7fb560077740 msgr2=0x7fb560077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:43.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.215+0000 7fb564b61640 1 -- 172.21.15.28:0/2067929853 shutdown_connections 2024-04-25T01:34:43.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.215+0000 7fb564b61640 1 -- 172.21.15.28:0/2067929853 wait complete. 2024-04-25T01:34:43.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.215+0000 7fb564b61640 1 Processor -- start 2024-04-25T01:34:43.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.215+0000 7fb564b61640 1 -- start start 2024-04-25T01:34:43.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.216+0000 7fb564b61640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb560100610 0x7fb560198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.216+0000 7fb564b61640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb560100fd0 0x7fb5601992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.216+0000 7fb564b61640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5601066b0 0x7fb5601a3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.216+0000 7fb564b61640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb560110a40 con 0x7fb5601066b0 2024-04-25T01:34:43.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.216+0000 7fb564b61640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb5601108c0 con 0x7fb560100610 2024-04-25T01:34:43.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.216+0000 7fb564b61640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb560110bc0 con 0x7fb560100fd0 2024-04-25T01:34:43.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.216+0000 7fb55ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5601066b0 0x7fb5601a3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.216+0000 7fb55ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5601066b0 0x7fb5601a3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42006/0 (socket says 172.21.15.28:42006) 2024-04-25T01:34:43.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.216+0000 7fb55ed9d640 1 -- 172.21.15.28:0/2891933527 learned_addr learned my addr 172.21.15.28:0/2891933527 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:43.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.216+0000 7fb55e59c640 1 --2- 172.21.15.28:0/2891933527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb560100610 0x7fb560198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.216+0000 7fb557fff640 1 --2- 172.21.15.28:0/2891933527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb560100fd0 0x7fb5601992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.216+0000 7fb55ed9d640 1 -- 172.21.15.28:0/2891933527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb560100fd0 msgr2=0x7fb5601992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:43.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.216+0000 7fb55ed9d640 1 --2- 172.21.15.28:0/2891933527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb560100fd0 0x7fb5601992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.216+0000 7fb55ed9d640 1 -- 172.21.15.28:0/2891933527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb560100610 msgr2=0x7fb560198d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:43.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.216+0000 7fb55ed9d640 1 --2- 172.21.15.28:0/2891933527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb560100610 0x7fb560198d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.216+0000 7fb55ed9d640 1 -- 172.21.15.28:0/2891933527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5601a4340 con 0x7fb5601066b0 2024-04-25T01:34:43.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.216+0000 7fb55ed9d640 1 --2- 172.21.15.28:0/2891933527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5601066b0 0x7fb5601a3e00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fb558033040 tx=0x7fb558031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:43.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.217+0000 7fb5577fe640 1 -- 172.21.15.28:0/2891933527 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb558004040 con 0x7fb5601066b0 2024-04-25T01:34:43.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.217+0000 7fb5577fe640 1 -- 172.21.15.28:0/2891933527 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5580046c0 con 0x7fb5601066b0 2024-04-25T01:34:43.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.217+0000 7fb564b61640 1 -- 172.21.15.28:0/2891933527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5601a4510 con 0x7fb5601066b0 2024-04-25T01:34:43.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.217+0000 7fb5577fe640 1 -- 172.21.15.28:0/2891933527 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb558046480 con 0x7fb5601066b0 2024-04-25T01:34:43.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.217+0000 7fb564b61640 1 -- 172.21.15.28:0/2891933527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5601a49d0 con 0x7fb5601066b0 2024-04-25T01:34:43.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.219+0000 7fb5577fe640 1 -- 172.21.15.28:0/2891933527 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb5580305a0 con 0x7fb5601066b0 2024-04-25T01:34:43.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.220+0000 7fb5577fe640 1 --2- 172.21.15.28:0/2891933527 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb54c0a2d00 0x7fb54c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.220+0000 7fb5577fe640 1 -- 172.21.15.28:0/2891933527 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fb55802e030 con 0x7fb5601066b0 2024-04-25T01:34:43.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.220+0000 7fb55e59c640 1 --2- 172.21.15.28:0/2891933527 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb54c0a2d00 0x7fb54c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.220+0000 7fb55e59c640 1 --2- 172.21.15.28:0/2891933527 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb54c0a2d00 0x7fb54c0a51c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb550005fd0 tx=0x7fb55000b410 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:43.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.221+0000 7fb56786f740 1 Processor -- start 2024-04-25T01:34:43.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.221+0000 7fb56786f740 1 -- start start 2024-04-25T01:34:43.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.221+0000 7fb56786f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211308580 0x55c211308960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.221+0000 7fb56786f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211308ea0 0x55c211313030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.222+0000 7fb56786f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211313570 0x55c211315960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.222+0000 7fb56786f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c21126d670 con 0x55c211313570 2024-04-25T01:34:43.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.222+0000 7fb56786f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c211215970 con 0x55c211308ea0 2024-04-25T01:34:43.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.222+0000 7fb56786f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c211269630 con 0x55c211308580 2024-04-25T01:34:43.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.222+0000 7fb55ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211313570 0x55c211315960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.222+0000 7fb557fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211308ea0 0x55c211313030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.222+0000 7fb55ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211313570 0x55c211315960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42022/0 (socket says 172.21.15.28:42022) 2024-04-25T01:34:43.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.222+0000 7fb557fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211308ea0 0x55c211313030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:40038/0 (socket says 172.21.15.28:40038) 2024-04-25T01:34:43.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.222+0000 7fb55ed9d640 1 -- 172.21.15.28:0/4210977849 learned_addr learned my addr 172.21.15.28:0/4210977849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:43.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.222+0000 7fb55e59c640 1 --2- 172.21.15.28:0/4210977849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211308580 0x55c211308960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.222+0000 7fb55ed9d640 1 -- 172.21.15.28:0/4210977849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211308580 msgr2=0x55c211308960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:43.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.222+0000 7fb55ed9d640 1 --2- 172.21.15.28:0/4210977849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211308580 0x55c211308960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.222+0000 7fb55ed9d640 1 -- 172.21.15.28:0/4210977849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211308ea0 msgr2=0x55c211313030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:43.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.222+0000 7fb55ed9d640 1 --2- 172.21.15.28:0/4210977849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211308ea0 0x55c211313030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.222+0000 7fb55ed9d640 1 -- 172.21.15.28:0/4210977849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c211239400 con 0x55c211313570 2024-04-25T01:34:43.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.222+0000 7fb557fff640 1 --2- 172.21.15.28:0/4210977849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211308ea0 0x55c211313030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:43.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.222+0000 7fb55ed9d640 1 --2- 172.21.15.28:0/4210977849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211313570 0x55c211315960 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb5580bcff0 tx=0x7fb5580bce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b68475db8a9c2c5b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:43.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.223+0000 7fb554ff9640 1 -- 172.21.15.28:0/4210977849 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5580bd760 con 0x55c211313570 2024-04-25T01:34:43.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.223+0000 7fb554ff9640 1 -- 172.21.15.28:0/4210977849 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb558036040 con 0x55c211313570 2024-04-25T01:34:43.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.223+0000 7fb554ff9640 1 -- 172.21.15.28:0/4210977849 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5580b96d0 con 0x55c211313570 2024-04-25T01:34:43.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.223+0000 7fb56786f740 1 -- 172.21.15.28:0/4210977849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211313570 msgr2=0x55c211315960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:43.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.223+0000 7fb56786f740 1 --2- 172.21.15.28:0/4210977849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211313570 0x55c211315960 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb5580bcff0 tx=0x7fb5580bce30 comp rx=0 tx=0).stop 2024-04-25T01:34:43.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.223+0000 7fb56786f740 1 -- 172.21.15.28:0/4210977849 shutdown_connections 2024-04-25T01:34:43.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.223+0000 7fb56786f740 1 --2- 172.21.15.28:0/4210977849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211313570 0x55c211315960 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.223+0000 7fb56786f740 1 --2- 172.21.15.28:0/4210977849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211308ea0 0x55c211313030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.223+0000 7fb56786f740 1 --2- 172.21.15.28:0/4210977849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211308580 0x55c211308960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.223+0000 7fb56786f740 1 -- 172.21.15.28:0/4210977849 >> 172.21.15.28:0/4210977849 conn(0x55c2113048a0 msgr2=0x55c211304c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.224+0000 7fb56786f740 1 -- 172.21.15.28:0/4210977849 shutdown_connections 2024-04-25T01:34:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.224+0000 7fb56786f740 1 -- 172.21.15.28:0/4210977849 wait complete. 2024-04-25T01:34:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.224+0000 7fb56786f740 1 -- start start 2024-04-25T01:34:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.224+0000 7fb56786f740 1 Processor -- start 2024-04-25T01:34:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.224+0000 7fb56786f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211308580 0x55c211312f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.224+0000 7fb56786f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211308ea0 0x55c211310270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.224+0000 7fb56786f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211313570 0x55c211310880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.224+0000 7fb56786f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c211316120 con 0x55c211308580 2024-04-25T01:34:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.224+0000 7fb56786f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c211229050 con 0x55c211308ea0 2024-04-25T01:34:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.224+0000 7fb56786f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c2113162a0 con 0x55c211313570 2024-04-25T01:34:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.224+0000 7fb56786f740 10 client.0 _ll_register_callbacks cb 0x55c210f80be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.224+0000 7fb55e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211308580 0x55c211312f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.225+0000 7fb55e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211308580 0x55c211312f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42034/0 (socket says 172.21.15.28:42034) 2024-04-25T01:34:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.225+0000 7fb55e59c640 1 -- 172.21.15.28:0/610958505 learned_addr learned my addr 172.21.15.28:0/610958505 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.225+0000 7fb557fff640 1 --2- 172.21.15.28:0/610958505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211308ea0 0x55c211310270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.225+0000 7fb55ed9d640 1 --2- 172.21.15.28:0/610958505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211313570 0x55c211310880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.225+0000 7fb55e59c640 1 -- 172.21.15.28:0/610958505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211313570 msgr2=0x55c211310880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.225+0000 7fb55e59c640 1 --2- 172.21.15.28:0/610958505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211313570 0x55c211310880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.225+0000 7fb55e59c640 1 -- 172.21.15.28:0/610958505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211308ea0 msgr2=0x55c211310270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.225+0000 7fb55e59c640 1 --2- 172.21.15.28:0/610958505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211308ea0 0x55c211310270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.225+0000 7fb55e59c640 1 -- 172.21.15.28:0/610958505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2111fadc0 con 0x55c211308580 2024-04-25T01:34:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.225+0000 7fb55e59c640 1 --2- 172.21.15.28:0/610958505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211308580 0x55c211312f80 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fb550003fa0 tx=0x7fb550004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1206c874aeb788c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.225+0000 7fb545ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.225+0000 7fb545ffb640 1 -- 172.21.15.28:0/610958505 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55001e830 con 0x55c211308580 2024-04-25T01:34:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.225+0000 7fb545ffb640 1 -- 172.21.15.28:0/610958505 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fb55001e9d0 con 0x55c211308580 2024-04-25T01:34:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.225+0000 7fb56786f740 1 -- 172.21.15.28:0/610958505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb558036040 con 0x55c211308580 2024-04-25T01:34:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.225+0000 7fb545ffb640 1 -- 172.21.15.28:0/610958505 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb55001ec10 con 0x55c211308580 2024-04-25T01:34:43.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.225+0000 7fb56786f740 10 client.16227 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:43.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.225+0000 7fb56786f740 1 -- 172.21.15.28:0/610958505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c210f813f0 con 0x55c211308580 2024-04-25T01:34:43.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.226+0000 7fb545ffb640 1 -- 172.21.15.28:0/610958505 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb558036040 con 0x55c211308580 2024-04-25T01:34:43.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.226+0000 7fb545ffb640 1 -- 172.21.15.28:0/610958505 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb550026020 con 0x55c211308580 2024-04-25T01:34:43.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.226+0000 7fb56786f740 10 client.16227 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:43.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.226+0000 7fb56786f740 10 client.16227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:43.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.226+0000 7fb56786f740 20 client.16227 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:43.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.226+0000 7fb56786f740 1 --2- 172.21.15.28:0/610958505 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c211327840 0x55c211329c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.226+0000 7fb56786f740 4 client.16227 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:43.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.226+0000 7fb56786f740 1 -- 172.21.15.28:0/610958505 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c21127abb0 con 0x55c211327840 2024-04-25T01:34:43.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.226+0000 7fb557fff640 1 --2- 172.21.15.28:0/610958505 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c211327840 0x55c211329c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.227+0000 7fb557fff640 1 --2- 172.21.15.28:0/610958505 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c211327840 0x55c211329c60 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccf401c51b73b4aa server_cookie=6558dd1d44d77195 in_seq=0 out_seq=0 2024-04-25T01:34:43.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.227+0000 7fb545ffb640 10 client.16227 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:43.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.228+0000 7fb545ffb640 1 -- 172.21.15.28:0/610958505 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c21127abb0 con 0x55c211327840 2024-04-25T01:34:43.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.228+0000 7fb545ffb640 10 client.16227 handle_command_reply: tid=0 2024-04-25T01:34:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.228+0000 7fb56786f740 1 client.16227 shutdown 2024-04-25T01:34:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.228+0000 7fb56786f740 1 -- 172.21.15.28:0/610958505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211308580 msgr2=0x55c211312f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.228+0000 7fb56786f740 1 --2- 172.21.15.28:0/610958505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211308580 0x55c211312f80 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fb550003fa0 tx=0x7fb550004730 comp rx=0 tx=0).stop 2024-04-25T01:34:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.228+0000 7fb56786f740 1 -- 172.21.15.28:0/610958505 shutdown_connections 2024-04-25T01:34:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.228+0000 7fb56786f740 1 --2- 172.21.15.28:0/610958505 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c211327840 0x55c211329c60 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.228+0000 7fb56786f740 1 --2- 172.21.15.28:0/610958505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211313570 0x55c211310880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.228+0000 7fb56786f740 1 --2- 172.21.15.28:0/610958505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211308ea0 0x55c211310270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.228+0000 7fb545ffb640 0 client.16227 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.228+0000 7fb56786f740 1 --2- 172.21.15.28:0/610958505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211308580 0x55c211312f80 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.228+0000 7fb56786f740 1 -- 172.21.15.28:0/610958505 >> 172.21.15.28:0/610958505 conn(0x55c2113048a0 msgr2=0x55c21130a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.228+0000 7fb56786f740 1 -- 172.21.15.28:0/610958505 shutdown_connections 2024-04-25T01:34:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.228+0000 7fb56786f740 1 -- 172.21.15.28:0/610958505 wait complete. 2024-04-25T01:34:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.228+0000 7fb56786f740 20 client.16227 trim_cache size 0 max 16384 2024-04-25T01:34:43.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:42 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/903031445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:34:43.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.242+0000 7fb56786f740 1 Processor -- start 2024-04-25T01:34:43.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.242+0000 7fb56786f740 1 -- start start 2024-04-25T01:34:43.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.242+0000 7fb56786f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211308580 0x55c211312ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.242+0000 7fb56786f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211308ea0 0x55c21132f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.242+0000 7fb56786f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211313570 0x55c21132fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.242+0000 7fb56786f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb5601ac7c0 con 0x55c211308580 2024-04-25T01:34:43.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.242+0000 7fb56786f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb5601ae300 con 0x55c211313570 2024-04-25T01:34:43.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.242+0000 7fb56786f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb5601ace90 con 0x55c211308ea0 2024-04-25T01:34:43.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.243+0000 7fb55e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211308580 0x55c211312ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.243+0000 7fb55e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211308580 0x55c211312ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42042/0 (socket says 172.21.15.28:42042) 2024-04-25T01:34:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.243+0000 7fb55e59c640 1 -- 172.21.15.28:0/3620046595 learned_addr learned my addr 172.21.15.28:0/3620046595 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.243+0000 7fb55ed9d640 1 --2- 172.21.15.28:0/3620046595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211313570 0x55c21132fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.243+0000 7fb557fff640 1 --2- 172.21.15.28:0/3620046595 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211308ea0 0x55c21132f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.243+0000 7fb55e59c640 1 -- 172.21.15.28:0/3620046595 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211308ea0 msgr2=0x55c21132f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.243+0000 7fb55e59c640 1 --2- 172.21.15.28:0/3620046595 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211308ea0 0x55c21132f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.243+0000 7fb55e59c640 1 -- 172.21.15.28:0/3620046595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211313570 msgr2=0x55c21132fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.243+0000 7fb55e59c640 1 --2- 172.21.15.28:0/3620046595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211313570 0x55c21132fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.243+0000 7fb55e59c640 1 -- 172.21.15.28:0/3620046595 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c211310bf0 con 0x55c211308580 2024-04-25T01:34:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.243+0000 7fb55e59c640 1 --2- 172.21.15.28:0/3620046595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211308580 0x55c211312ab0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fb550029fd0 tx=0x7fb550004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d12bfc6ffaa97b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.243+0000 7fb547fff640 1 -- 172.21.15.28:0/3620046595 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55003b070 con 0x55c211308580 2024-04-25T01:34:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.243+0000 7fb547fff640 1 -- 172.21.15.28:0/3620046595 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb550028440 con 0x55c211308580 2024-04-25T01:34:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.243+0000 7fb547fff640 1 -- 172.21.15.28:0/3620046595 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55001a7e0 con 0x55c211308580 2024-04-25T01:34:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.244+0000 7fb56786f740 1 -- 172.21.15.28:0/3620046595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211308580 msgr2=0x55c211312ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:43.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.244+0000 7fb56786f740 1 --2- 172.21.15.28:0/3620046595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211308580 0x55c211312ab0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fb550029fd0 tx=0x7fb550004570 comp rx=0 tx=0).stop 2024-04-25T01:34:43.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.244+0000 7fb56786f740 1 -- 172.21.15.28:0/3620046595 shutdown_connections 2024-04-25T01:34:43.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.244+0000 7fb56786f740 1 --2- 172.21.15.28:0/3620046595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211313570 0x55c21132fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.244+0000 7fb56786f740 1 --2- 172.21.15.28:0/3620046595 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211308ea0 0x55c21132f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.244+0000 7fb56786f740 1 --2- 172.21.15.28:0/3620046595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211308580 0x55c211312ab0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.244+0000 7fb56786f740 1 -- 172.21.15.28:0/3620046595 >> 172.21.15.28:0/3620046595 conn(0x55c2113048a0 msgr2=0x55c211309cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:43.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.244+0000 7fb56786f740 1 -- 172.21.15.28:0/3620046595 shutdown_connections 2024-04-25T01:34:43.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.244+0000 7fb56786f740 1 -- 172.21.15.28:0/3620046595 wait complete. 2024-04-25T01:34:43.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.245+0000 7fb56786f740 1 -- start start 2024-04-25T01:34:43.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.245+0000 7fb56786f740 1 Processor -- start 2024-04-25T01:34:43.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.245+0000 7fb56786f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211308580 0x55c21131b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.245+0000 7fb56786f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211308ea0 0x55c21131ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.245+0000 7fb56786f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211313570 0x55c21131bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.245+0000 7fb56786f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb5601adc30 con 0x55c211313570 2024-04-25T01:34:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.245+0000 7fb56786f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb5601ad560 con 0x55c211308ea0 2024-04-25T01:34:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.245+0000 7fb56786f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb5601aa7a0 con 0x55c211308580 2024-04-25T01:34:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.245+0000 7fb56786f740 10 client.0 _ll_register_callbacks cb 0x7fb550003c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.245+0000 7fb55ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211313570 0x55c21131bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.245+0000 7fb55ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211313570 0x55c21131bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42048/0 (socket says 172.21.15.28:42048) 2024-04-25T01:34:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.245+0000 7fb557fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211308ea0 0x55c21131ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.245+0000 7fb55ed9d640 1 -- 172.21.15.28:0/3096517088 learned_addr learned my addr 172.21.15.28:0/3096517088 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.246+0000 7fb55e59c640 1 --2- 172.21.15.28:0/3096517088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211308580 0x55c21131b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.246+0000 7fb557fff640 1 -- 172.21.15.28:0/3096517088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211308580 msgr2=0x55c21131b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.246+0000 7fb557fff640 1 --2- 172.21.15.28:0/3096517088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211308580 0x55c21131b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.246+0000 7fb557fff640 1 -- 172.21.15.28:0/3096517088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211313570 msgr2=0x55c21131bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.246+0000 7fb557fff640 1 --2- 172.21.15.28:0/3096517088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211313570 0x55c21131bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.246+0000 7fb557fff640 1 -- 172.21.15.28:0/3096517088 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c21104b7d0 con 0x55c211308ea0 2024-04-25T01:34:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.246+0000 7fb55ed9d640 1 --2- 172.21.15.28:0/3096517088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211313570 0x55c21131bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.246+0000 7fb557fff640 1 --2- 172.21.15.28:0/3096517088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211308ea0 0x55c21131ba10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x55c21132a490 tx=0x7fb5480204f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7b38ebc5bf42f8d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.246+0000 7fb545ffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-25T01:34:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.246+0000 7fb545ffb640 1 -- 172.21.15.28:0/3096517088 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb548013600 con 0x55c211308ea0 2024-04-25T01:34:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.246+0000 7fb545ffb640 1 -- 172.21.15.28:0/3096517088 <== mon.2 v2:172.21.15.32:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fb5480137a0 con 0x55c211308ea0 2024-04-25T01:34:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.246+0000 7fb56786f740 1 -- 172.21.15.28:0/3096517088 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb550028440 con 0x55c211308ea0 2024-04-25T01:34:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.247+0000 7fb545ffb640 1 -- 172.21.15.28:0/3096517088 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb548030050 con 0x55c211308ea0 2024-04-25T01:34:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.247+0000 7fb56786f740 10 client.25721 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.247+0000 7fb56786f740 1 -- 172.21.15.28:0/3096517088 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c21138e120 con 0x55c211308ea0 2024-04-25T01:34:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.247+0000 7fb545ffb640 1 -- 172.21.15.28:0/3096517088 <== mon.2 v2:172.21.15.32:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb550028440 con 0x55c211308ea0 2024-04-25T01:34:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.247+0000 7fb545ffb640 1 -- 172.21.15.28:0/3096517088 <== mon.2 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb54801f480 con 0x55c211308ea0 2024-04-25T01:34:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.247+0000 7fb56786f740 10 client.25721 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.247+0000 7fb56786f740 10 client.25721 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.247+0000 7fb56786f740 20 client.25721 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.247+0000 7fb56786f740 1 --2- 172.21.15.28:0/3096517088 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c21138e870 0x55c211390c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.247+0000 7fb56786f740 4 client.25721 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:34:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.247+0000 7fb56786f740 1 -- 172.21.15.28:0/3096517088 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c211274f00 con 0x55c21138e870 2024-04-25T01:34:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.247+0000 7fb55e59c640 1 --2- 172.21.15.28:0/3096517088 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c21138e870 0x55c211390c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.248+0000 7fb545ffb640 10 client.25721 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.248+0000 7fb55e59c640 1 --2- 172.21.15.28:0/3096517088 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c21138e870 0x55c211390c90 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da74d0a66c8cf433 server_cookie=cafd2d6cbb710791 in_seq=0 out_seq=0 2024-04-25T01:34:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.248+0000 7fb545ffb640 1 -- 172.21.15.28:0/3096517088 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c211274f00 con 0x55c21138e870 2024-04-25T01:34:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.248+0000 7fb545ffb640 10 client.25721 handle_command_reply: tid=0 2024-04-25T01:34:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.248+0000 7fb56786f740 1 client.25721 shutdown 2024-04-25T01:34:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.248+0000 7fb56786f740 1 -- 172.21.15.28:0/3096517088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211308ea0 msgr2=0x55c21131ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.248+0000 7fb56786f740 1 --2- 172.21.15.28:0/3096517088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211308ea0 0x55c21131ba10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x55c21132a490 tx=0x7fb5480204f0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.248+0000 7fb56786f740 1 -- 172.21.15.28:0/3096517088 shutdown_connections 2024-04-25T01:34:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.248+0000 7fb56786f740 1 --2- 172.21.15.28:0/3096517088 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c21138e870 0x55c211390c90 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.248+0000 7fb56786f740 1 --2- 172.21.15.28:0/3096517088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c211313570 0x55c21131bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.248+0000 7fb545ffb640 0 client.25721 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.248+0000 7fb56786f740 1 --2- 172.21.15.28:0/3096517088 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c211308ea0 0x55c21131ba10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.248+0000 7fb56786f740 1 --2- 172.21.15.28:0/3096517088 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c211308580 0x55c21131b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.248+0000 7fb56786f740 1 -- 172.21.15.28:0/3096517088 >> 172.21.15.28:0/3096517088 conn(0x55c2113048a0 msgr2=0x55c21130a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:43.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.248+0000 7fb56786f740 1 -- 172.21.15.28:0/3096517088 shutdown_connections 2024-04-25T01:34:43.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.248+0000 7fb56786f740 1 -- 172.21.15.28:0/3096517088 wait complete. 2024-04-25T01:34:43.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.249+0000 7fb56786f740 20 client.25721 trim_cache size 0 max 16384 2024-04-25T01:34:43.231 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:34:43.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.249+0000 7fb564b61640 1 -- 172.21.15.28:0/2891933527 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb54c0a2d00 msgr2=0x7fb54c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:43.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.249+0000 7fb564b61640 1 --2- 172.21.15.28:0/2891933527 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb54c0a2d00 0x7fb54c0a51c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb550005fd0 tx=0x7fb55000b410 comp rx=0 tx=0).stop 2024-04-25T01:34:43.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.249+0000 7fb564b61640 1 -- 172.21.15.28:0/2891933527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5601066b0 msgr2=0x7fb5601a3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:43.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.249+0000 7fb564b61640 1 --2- 172.21.15.28:0/2891933527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5601066b0 0x7fb5601a3e00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fb558033040 tx=0x7fb558031e60 comp rx=0 tx=0).stop 2024-04-25T01:34:43.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.249+0000 7fb564b61640 1 -- 172.21.15.28:0/2891933527 shutdown_connections 2024-04-25T01:34:43.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.249+0000 7fb564b61640 1 --2- 172.21.15.28:0/2891933527 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb54c0a2d00 0x7fb54c0a51c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.249+0000 7fb564b61640 1 --2- 172.21.15.28:0/2891933527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5601066b0 0x7fb5601a3e00 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.249+0000 7fb564b61640 1 --2- 172.21.15.28:0/2891933527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb560100fd0 0x7fb5601992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.249+0000 7fb564b61640 1 --2- 172.21.15.28:0/2891933527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb560100610 0x7fb560198d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:43.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.249+0000 7fb564b61640 1 -- 172.21.15.28:0/2891933527 >> 172.21.15.28:0/2891933527 conn(0x7fb560077740 msgr2=0x7fb5600fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:43.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.250+0000 7fb564b61640 1 -- 172.21.15.28:0/2891933527 shutdown_connections 2024-04-25T01:34:43.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:43.250+0000 7fb564b61640 1 -- 172.21.15.28:0/2891933527 wait complete. 2024-04-25T01:34:43.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:42 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/903031445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:34:44.207 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:43 smithi130 ceph-mon[32702]: pgmap v54: 129 pgs: 129 active+clean; 2.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 171 MiB/s wr, 58 op/s 2024-04-25T01:34:44.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:43 smithi032 ceph-mon[32461]: pgmap v54: 129 pgs: 129 active+clean; 2.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 171 MiB/s wr, 58 op/s 2024-04-25T01:34:44.241 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:34:44.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:34:44.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:34:44.278 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:43 smithi028 ceph-mon[29390]: pgmap v54: 129 pgs: 129 active+clean; 2.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 171 MiB/s wr, 58 op/s 2024-04-25T01:34:44.289 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:44.289 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 rewriters = 844430.00 kB/sec 2024-04-25T01:34:44.289 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 rewriters = 844252.54 kB/sec 2024-04-25T01:34:44.289 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 844430.00 kB/sec 2024-04-25T01:34:44.289 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 844430.00 kB/sec 2024-04-25T01:34:44.289 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 844430.00 kB/sec 2024-04-25T01:34:44.289 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 1048576.00 kB 2024-04-25T01:34:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.404+0000 7ff4dca6b640 1 -- 172.21.15.28:0/2853598271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d810bd00 msgr2=0x7ff4d810e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.404+0000 7ff4dca6b640 1 --2- 172.21.15.28:0/2853598271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d810bd00 0x7ff4d810e1a0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0009ca0 tx=0x7ff4d002a9e0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.404+0000 7ff4dca6b640 1 -- 172.21.15.28:0/2853598271 shutdown_connections 2024-04-25T01:34:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.404+0000 7ff4dca6b640 1 --2- 172.21.15.28:0/2853598271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d810bd00 0x7ff4d810e1a0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.404+0000 7ff4dca6b640 1 --2- 172.21.15.28:0/2853598271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4d81032e0 0x7ff4d810b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.404+0000 7ff4dca6b640 1 --2- 172.21.15.28:0/2853598271 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4d81029c0 0x7ff4d8102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.404+0000 7ff4dca6b640 1 -- 172.21.15.28:0/2853598271 >> 172.21.15.28:0/2853598271 conn(0x7ff4d8077740 msgr2=0x7ff4d8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.404+0000 7ff4dca6b640 1 -- 172.21.15.28:0/2853598271 shutdown_connections 2024-04-25T01:34:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.404+0000 7ff4dca6b640 1 -- 172.21.15.28:0/2853598271 wait complete. 2024-04-25T01:34:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4dca6b640 1 Processor -- start 2024-04-25T01:34:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4dca6b640 1 -- start start 2024-04-25T01:34:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4dca6b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4d81029c0 0x7ff4d8197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4dca6b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d81032e0 0x7ff4d8198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4dca6b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4d810bd00 0x7ff4d819c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4dca6b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d8111b50 con 0x7ff4d81032e0 2024-04-25T01:34:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4dca6b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d81119d0 con 0x7ff4d81029c0 2024-04-25T01:34:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4dca6b640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d8111cd0 con 0x7ff4d810bd00 2024-04-25T01:34:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4d5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d81032e0 0x7ff4d8198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4d5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d81032e0 0x7ff4d8198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42066/0 (socket says 172.21.15.28:42066) 2024-04-25T01:34:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4d5d9b640 1 -- 172.21.15.28:0/1518976993 learned_addr learned my addr 172.21.15.28:0/1518976993 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4d6d9d640 1 --2- 172.21.15.28:0/1518976993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4d810bd00 0x7ff4d819c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4d659c640 1 --2- 172.21.15.28:0/1518976993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4d81029c0 0x7ff4d8197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4d5d9b640 1 -- 172.21.15.28:0/1518976993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4d810bd00 msgr2=0x7ff4d819c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4d5d9b640 1 --2- 172.21.15.28:0/1518976993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4d810bd00 0x7ff4d819c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4d5d9b640 1 -- 172.21.15.28:0/1518976993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4d81029c0 msgr2=0x7ff4d8197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4d5d9b640 1 --2- 172.21.15.28:0/1518976993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4d81029c0 0x7ff4d8197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4d5d9b640 1 -- 172.21.15.28:0/1518976993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d819cf20 con 0x7ff4d81032e0 2024-04-25T01:34:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4d6d9d640 1 --2- 172.21.15.28:0/1518976993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4d810bd00 0x7ff4d819c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:34:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.405+0000 7ff4d5d9b640 1 --2- 172.21.15.28:0/1518976993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d81032e0 0x7ff4d8198470 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc00c920 tx=0x7ff4cc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.406+0000 7ff4c77fe640 1 -- 172.21.15.28:0/1518976993 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc007c70 con 0x7ff4d81032e0 2024-04-25T01:34:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.406+0000 7ff4c77fe640 1 -- 172.21.15.28:0/1518976993 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4cc007e10 con 0x7ff4d81032e0 2024-04-25T01:34:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.406+0000 7ff4dca6b640 1 -- 172.21.15.28:0/1518976993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4d819d210 con 0x7ff4d81032e0 2024-04-25T01:34:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.406+0000 7ff4c77fe640 1 -- 172.21.15.28:0/1518976993 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc018890 con 0x7ff4d81032e0 2024-04-25T01:34:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.406+0000 7ff4dca6b640 1 -- 172.21.15.28:0/1518976993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4d81a4aa0 con 0x7ff4d81032e0 2024-04-25T01:34:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.408+0000 7ff4c77fe640 1 -- 172.21.15.28:0/1518976993 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff4cc018a70 con 0x7ff4d81032e0 2024-04-25T01:34:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.408+0000 7ff4df77b740 1 Processor -- start 2024-04-25T01:34:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.408+0000 7ff4df77b740 1 -- start start 2024-04-25T01:34:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4df77b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fba980 0x55a377fb8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4df77b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fb9290 0x55a377fb9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4df77b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fc4010 0x55a377fc6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4df77b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a377f1d2c0 con 0x55a377fc4010 2024-04-25T01:34:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4df77b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a377ec5910 con 0x55a377fba980 2024-04-25T01:34:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4df77b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55a377f19280 con 0x55a377fb9290 2024-04-25T01:34:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4c77fe640 1 --2- 172.21.15.28:0/1518976993 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4c00a2d00 0x7ff4c00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4d5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fc4010 0x55a377fc6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4d6d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fba980 0x55a377fb8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4d6d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fba980 0x55a377fb8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:40090/0 (socket says 172.21.15.28:40090) 2024-04-25T01:34:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4d5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fc4010 0x55a377fc6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42074/0 (socket says 172.21.15.28:42074) 2024-04-25T01:34:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4d6d9d640 1 -- 172.21.15.28:0/3579733819 learned_addr learned my addr 172.21.15.28:0/3579733819 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4d659c640 1 --2- 172.21.15.28:0/3579733819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fb9290 0x55a377fb9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4d6d9d640 1 --2- 172.21.15.28:0/1518976993 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4c00a2d00 0x7ff4c00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4c77fe640 1 -- 172.21.15.28:0/1518976993 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff4cc0cf970 con 0x7ff4d81032e0 2024-04-25T01:34:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4d6d9d640 1 -- 172.21.15.28:0/3579733819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fb9290 msgr2=0x55a377fb9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4d6d9d640 1 --2- 172.21.15.28:0/3579733819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fb9290 0x55a377fb9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4d6d9d640 1 -- 172.21.15.28:0/3579733819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fc4010 msgr2=0x55a377fc6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4d6d9d640 1 --2- 172.21.15.28:0/3579733819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fc4010 0x55a377fc6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4d6d9d640 1 -- 172.21.15.28:0/3579733819 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a377ed6fb0 con 0x55a377fba980 2024-04-25T01:34:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.409+0000 7ff4d6d9d640 1 --2- 172.21.15.28:0/3579733819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fba980 0x55a377fb8cc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff4d002e410 tx=0x7ff4d002e440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d0dca9e74ac902 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.410+0000 7ff4d6d9d640 1 --2- 172.21.15.28:0/1518976993 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4c00a2d00 0x7ff4c00a51c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0009a60 tx=0x7ff4d0033000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.410+0000 7ff4c4ff9640 1 -- 172.21.15.28:0/3579733819 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d004c070 con 0x55a377fba980 2024-04-25T01:34:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.410+0000 7ff4c4ff9640 1 -- 172.21.15.28:0/3579733819 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4d002f640 con 0x55a377fba980 2024-04-25T01:34:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.410+0000 7ff4c4ff9640 1 -- 172.21.15.28:0/3579733819 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0033070 con 0x55a377fba980 2024-04-25T01:34:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.410+0000 7ff4df77b740 1 -- 172.21.15.28:0/3579733819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fba980 msgr2=0x55a377fb8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.410+0000 7ff4df77b740 1 --2- 172.21.15.28:0/3579733819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fba980 0x55a377fb8cc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff4d002e410 tx=0x7ff4d002e440 comp rx=0 tx=0).stop 2024-04-25T01:34:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.410+0000 7ff4df77b740 1 -- 172.21.15.28:0/3579733819 shutdown_connections 2024-04-25T01:34:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.410+0000 7ff4df77b740 1 --2- 172.21.15.28:0/3579733819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fc4010 0x55a377fc6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.410+0000 7ff4df77b740 1 --2- 172.21.15.28:0/3579733819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fb9290 0x55a377fb9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.410+0000 7ff4df77b740 1 --2- 172.21.15.28:0/3579733819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fba980 0x55a377fb8cc0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.410+0000 7ff4df77b740 1 -- 172.21.15.28:0/3579733819 >> 172.21.15.28:0/3579733819 conn(0x55a377f81950 msgr2=0x55a377fb57f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.411+0000 7ff4df77b740 1 -- 172.21.15.28:0/3579733819 shutdown_connections 2024-04-25T01:34:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.411+0000 7ff4df77b740 1 -- 172.21.15.28:0/3579733819 wait complete. 2024-04-25T01:34:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.411+0000 7ff4df77b740 1 -- start start 2024-04-25T01:34:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.411+0000 7ff4df77b740 1 Processor -- start 2024-04-25T01:34:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.411+0000 7ff4df77b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fb9290 0x55a377fc0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.411+0000 7ff4df77b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fba980 0x55a377fc1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.411+0000 7ff4df77b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fc4010 0x55a377fc1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.411+0000 7ff4df77b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a377fc6de0 con 0x55a377fba980 2024-04-25T01:34:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.411+0000 7ff4df77b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a377ed8e90 con 0x55a377fb9290 2024-04-25T01:34:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.411+0000 7ff4df77b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55a377fc6f60 con 0x55a377fc4010 2024-04-25T01:34:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.411+0000 7ff4df77b740 10 client.0 _ll_register_callbacks cb 0x55a377f14970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.411+0000 7ff4d5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fba980 0x55a377fc1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.411+0000 7ff4d5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fba980 0x55a377fc1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42086/0 (socket says 172.21.15.28:42086) 2024-04-25T01:34:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.411+0000 7ff4d5d9b640 1 -- 172.21.15.28:0/4070057908 learned_addr learned my addr 172.21.15.28:0/4070057908 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.411+0000 7ff4d659c640 1 --2- 172.21.15.28:0/4070057908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fb9290 0x55a377fc0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.411+0000 7ff4d5d9b640 1 -- 172.21.15.28:0/4070057908 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fc4010 msgr2=0x55a377fc1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.411+0000 7ff4d6d9d640 1 --2- 172.21.15.28:0/4070057908 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fc4010 0x55a377fc1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.412+0000 7ff4d5d9b640 1 --2- 172.21.15.28:0/4070057908 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fc4010 0x55a377fc1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.412+0000 7ff4d5d9b640 1 -- 172.21.15.28:0/4070057908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fb9290 msgr2=0x55a377fc0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.412+0000 7ff4d5d9b640 1 --2- 172.21.15.28:0/4070057908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fb9290 0x55a377fc0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.412+0000 7ff4d5d9b640 1 -- 172.21.15.28:0/4070057908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a377cf97e0 con 0x55a377fba980 2024-04-25T01:34:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.412+0000 7ff4d5d9b640 1 --2- 172.21.15.28:0/4070057908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fba980 0x55a377fc1500 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc0984c0 tx=0x7ff4cc00ff30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4cc034dc8acf741 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.412+0000 7ff4bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.412+0000 7ff4bdffb640 1 -- 172.21.15.28:0/4070057908 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc0a4760 con 0x55a377fba980 2024-04-25T01:34:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.412+0000 7ff4df77b740 1 -- 172.21.15.28:0/4070057908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4d002f640 con 0x55a377fba980 2024-04-25T01:34:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.412+0000 7ff4bdffb640 1 -- 172.21.15.28:0/4070057908 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff4cc00fb30 con 0x55a377fba980 2024-04-25T01:34:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.412+0000 7ff4bdffb640 1 -- 172.21.15.28:0/4070057908 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff4cc0ccfa0 con 0x55a377fba980 2024-04-25T01:34:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.412+0000 7ff4df77b740 10 client.16245 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.412+0000 7ff4df77b740 1 -- 172.21.15.28:0/4070057908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a377d59f70 con 0x55a377fba980 2024-04-25T01:34:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.412+0000 7ff4bdffb640 1 -- 172.21.15.28:0/4070057908 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4d002f640 con 0x55a377fba980 2024-04-25T01:34:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.412+0000 7ff4bdffb640 1 -- 172.21.15.28:0/4070057908 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff4cc09af40 con 0x55a377fba980 2024-04-25T01:34:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.412+0000 7ff4df77b740 10 client.16245 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.412+0000 7ff4df77b740 10 client.16245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.412+0000 7ff4df77b740 20 client.16245 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.412+0000 7ff4df77b740 1 --2- 172.21.15.28:0/4070057908 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55a377fd3f60 0x55a377fd6350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.413+0000 7ff4df77b740 4 client.16245 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.413+0000 7ff4df77b740 1 -- 172.21.15.28:0/4070057908 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a377ef15a0 con 0x55a377fd3f60 2024-04-25T01:34:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.413+0000 7ff4d659c640 1 --2- 172.21.15.28:0/4070057908 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55a377fd3f60 0x55a377fd6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.413+0000 7ff4d659c640 1 --2- 172.21.15.28:0/4070057908 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55a377fd3f60 0x55a377fd6350 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fdc825dcc748dfa server_cookie=90a7715a7f6a7395 in_seq=0 out_seq=0 2024-04-25T01:34:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.413+0000 7ff4bdffb640 10 client.16245 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.414+0000 7ff4bdffb640 1 -- 172.21.15.28:0/4070057908 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a377ef15a0 con 0x55a377fd3f60 2024-04-25T01:34:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.414+0000 7ff4bdffb640 10 client.16245 handle_command_reply: tid=0 2024-04-25T01:34:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.414+0000 7ff4df77b740 1 client.16245 shutdown 2024-04-25T01:34:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.414+0000 7ff4df77b740 1 -- 172.21.15.28:0/4070057908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fba980 msgr2=0x55a377fc1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.414+0000 7ff4df77b740 1 --2- 172.21.15.28:0/4070057908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fba980 0x55a377fc1500 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc0984c0 tx=0x7ff4cc00ff30 comp rx=0 tx=0).stop 2024-04-25T01:34:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.414+0000 7ff4df77b740 1 -- 172.21.15.28:0/4070057908 shutdown_connections 2024-04-25T01:34:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.414+0000 7ff4df77b740 1 --2- 172.21.15.28:0/4070057908 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55a377fd3f60 0x55a377fd6350 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.414+0000 7ff4df77b740 1 --2- 172.21.15.28:0/4070057908 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fc4010 0x55a377fc1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.414+0000 7ff4df77b740 1 --2- 172.21.15.28:0/4070057908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fba980 0x55a377fc1500 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.414+0000 7ff4df77b740 1 --2- 172.21.15.28:0/4070057908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fb9290 0x55a377fc0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.414+0000 7ff4bdffb640 0 client.16245 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.414+0000 7ff4df77b740 1 -- 172.21.15.28:0/4070057908 >> 172.21.15.28:0/4070057908 conn(0x55a377f81950 msgr2=0x55a377fbb660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.414+0000 7ff4df77b740 1 -- 172.21.15.28:0/4070057908 shutdown_connections 2024-04-25T01:34:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.414+0000 7ff4df77b740 1 -- 172.21.15.28:0/4070057908 wait complete. 2024-04-25T01:34:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.414+0000 7ff4df77b740 20 client.16245 trim_cache size 0 max 16384 2024-04-25T01:34:44.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.432+0000 7ff4df77b740 1 Processor -- start 2024-04-25T01:34:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.432+0000 7ff4df77b740 1 -- start start 2024-04-25T01:34:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.432+0000 7ff4df77b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fb9290 0x55a377fcfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.432+0000 7ff4df77b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fba980 0x55a377fc08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.432+0000 7ff4df77b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fc4010 0x55a377fdbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.432+0000 7ff4df77b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d81aea80 con 0x55a377fc4010 2024-04-25T01:34:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.432+0000 7ff4df77b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d81b05c0 con 0x55a377fb9290 2024-04-25T01:34:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.432+0000 7ff4df77b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d81af150 con 0x55a377fba980 2024-04-25T01:34:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.432+0000 7ff4d6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fc4010 0x55a377fdbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.432+0000 7ff4d6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fc4010 0x55a377fdbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42100/0 (socket says 172.21.15.28:42100) 2024-04-25T01:34:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.432+0000 7ff4d6d9d640 1 -- 172.21.15.28:0/1369118870 learned_addr learned my addr 172.21.15.28:0/1369118870 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.432+0000 7ff4d659c640 1 --2- 172.21.15.28:0/1369118870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fb9290 0x55a377fcfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.432+0000 7ff4d5d9b640 1 --2- 172.21.15.28:0/1369118870 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fba980 0x55a377fc08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.432+0000 7ff4d6d9d640 1 -- 172.21.15.28:0/1369118870 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fba980 msgr2=0x55a377fc08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.432+0000 7ff4d6d9d640 1 --2- 172.21.15.28:0/1369118870 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fba980 0x55a377fc08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.432+0000 7ff4d6d9d640 1 -- 172.21.15.28:0/1369118870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fb9290 msgr2=0x55a377fcfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.432+0000 7ff4d6d9d640 1 --2- 172.21.15.28:0/1369118870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fb9290 0x55a377fcfce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.432+0000 7ff4d6d9d640 1 -- 172.21.15.28:0/1369118870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a377bf1170 con 0x55a377fc4010 2024-04-25T01:34:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.432+0000 7ff4d659c640 1 --2- 172.21.15.28:0/1369118870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fb9290 0x55a377fcfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.433+0000 7ff4d6d9d640 1 --2- 172.21.15.28:0/1369118870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fc4010 0x55a377fdbc20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff4d002aa20 tx=0x7ff4d002eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b76f1f5ae34ec10 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.433+0000 7ff4bffff640 1 -- 172.21.15.28:0/1369118870 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d005e020 con 0x55a377fc4010 2024-04-25T01:34:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.433+0000 7ff4bffff640 1 -- 172.21.15.28:0/1369118870 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4d0009540 con 0x55a377fc4010 2024-04-25T01:34:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.433+0000 7ff4bffff640 1 -- 172.21.15.28:0/1369118870 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d004c040 con 0x55a377fc4010 2024-04-25T01:34:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.433+0000 7ff4df77b740 1 -- 172.21.15.28:0/1369118870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fc4010 msgr2=0x55a377fdbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.433+0000 7ff4df77b740 1 --2- 172.21.15.28:0/1369118870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fc4010 0x55a377fdbc20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff4d002aa20 tx=0x7ff4d002eef0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.433+0000 7ff4df77b740 1 -- 172.21.15.28:0/1369118870 shutdown_connections 2024-04-25T01:34:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.433+0000 7ff4df77b740 1 --2- 172.21.15.28:0/1369118870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fc4010 0x55a377fdbc20 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.433+0000 7ff4df77b740 1 --2- 172.21.15.28:0/1369118870 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fba980 0x55a377fc08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.433+0000 7ff4df77b740 1 --2- 172.21.15.28:0/1369118870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fb9290 0x55a377fcfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.433+0000 7ff4df77b740 1 -- 172.21.15.28:0/1369118870 >> 172.21.15.28:0/1369118870 conn(0x55a377f81950 msgr2=0x55a377fb4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.433+0000 7ff4df77b740 1 -- 172.21.15.28:0/1369118870 shutdown_connections 2024-04-25T01:34:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.433+0000 7ff4df77b740 1 -- 172.21.15.28:0/1369118870 wait complete. 2024-04-25T01:34:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.434+0000 7ff4df77b740 1 -- start start 2024-04-25T01:34:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.434+0000 7ff4df77b740 1 Processor -- start 2024-04-25T01:34:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.434+0000 7ff4df77b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fb9290 0x55a377fc7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.434+0000 7ff4df77b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fba980 0x55a377fc8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.434+0000 7ff4df77b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fc4010 0x55a378039490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.434+0000 7ff4df77b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d81afef0 con 0x55a377fba980 2024-04-25T01:34:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.434+0000 7ff4df77b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d81af820 con 0x55a377fc4010 2024-04-25T01:34:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.434+0000 7ff4df77b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d81aca60 con 0x55a377fb9290 2024-04-25T01:34:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.434+0000 7ff4df77b740 10 client.0 _ll_register_callbacks cb 0x7ff4cc00cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.434+0000 7ff4d5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fba980 0x55a377fc8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.434+0000 7ff4d5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fba980 0x55a377fc8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42104/0 (socket says 172.21.15.28:42104) 2024-04-25T01:34:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.434+0000 7ff4d5d9b640 1 -- 172.21.15.28:0/2815727532 learned_addr learned my addr 172.21.15.28:0/2815727532 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.434+0000 7ff4d659c640 1 --2- 172.21.15.28:0/2815727532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fb9290 0x55a377fc7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.434+0000 7ff4d6d9d640 1 --2- 172.21.15.28:0/2815727532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fc4010 0x55a378039490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.434+0000 7ff4d5d9b640 1 -- 172.21.15.28:0/2815727532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fb9290 msgr2=0x55a377fc7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.434+0000 7ff4d5d9b640 1 --2- 172.21.15.28:0/2815727532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fb9290 0x55a377fc7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.434+0000 7ff4d5d9b640 1 -- 172.21.15.28:0/2815727532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fc4010 msgr2=0x55a378039490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.434+0000 7ff4d5d9b640 1 --2- 172.21.15.28:0/2815727532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fc4010 0x55a378039490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.434+0000 7ff4d5d9b640 1 -- 172.21.15.28:0/2815727532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3780399d0 con 0x55a377fba980 2024-04-25T01:34:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.435+0000 7ff4d5d9b640 1 --2- 172.21.15.28:0/2815727532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fba980 0x55a377fc8520 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc09b7f0 tx=0x7ff4cc00fd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93b620e4e57d1441 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.435+0000 7ff4bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.435+0000 7ff4bdffb640 1 -- 172.21.15.28:0/2815727532 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc09aa50 con 0x55a377fba980 2024-04-25T01:34:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.435+0000 7ff4bdffb640 1 -- 172.21.15.28:0/2815727532 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff4cc09ad00 con 0x55a377fba980 2024-04-25T01:34:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.435+0000 7ff4df77b740 1 -- 172.21.15.28:0/2815727532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4d0009540 con 0x55a377fba980 2024-04-25T01:34:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.435+0000 7ff4bdffb640 1 -- 172.21.15.28:0/2815727532 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff4cc0cd140 con 0x55a377fba980 2024-04-25T01:34:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.435+0000 7ff4df77b740 10 client.16251 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.435+0000 7ff4df77b740 1 -- 172.21.15.28:0/2815727532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a37803ab20 con 0x55a377fba980 2024-04-25T01:34:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.435+0000 7ff4bdffb640 1 -- 172.21.15.28:0/2815727532 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4d0009540 con 0x55a377fba980 2024-04-25T01:34:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.435+0000 7ff4bdffb640 1 -- 172.21.15.28:0/2815727532 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff4cc097840 con 0x55a377fba980 2024-04-25T01:34:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.435+0000 7ff4df77b740 10 client.16251 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.435+0000 7ff4df77b740 10 client.16251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.435+0000 7ff4df77b740 20 client.16251 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.435+0000 7ff4df77b740 1 --2- 172.21.15.28:0/2815727532 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55a37803b250 0x55a37803d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.435+0000 7ff4df77b740 4 client.16251 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:34:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.435+0000 7ff4df77b740 1 -- 172.21.15.28:0/2815727532 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a377f2a800 con 0x55a37803b250 2024-04-25T01:34:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.436+0000 7ff4d659c640 1 --2- 172.21.15.28:0/2815727532 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55a37803b250 0x55a37803d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.436+0000 7ff4d659c640 1 --2- 172.21.15.28:0/2815727532 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55a37803b250 0x55a37803d670 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab8068354873fb6d server_cookie=5599686cbec954e9 in_seq=0 out_seq=0 2024-04-25T01:34:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.436+0000 7ff4bdffb640 10 client.16251 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.437+0000 7ff4bdffb640 1 -- 172.21.15.28:0/2815727532 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a377f2a800 con 0x55a37803b250 2024-04-25T01:34:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.437+0000 7ff4bdffb640 10 client.16251 handle_command_reply: tid=0 2024-04-25T01:34:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.437+0000 7ff4df77b740 1 client.16251 shutdown 2024-04-25T01:34:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.437+0000 7ff4df77b740 1 -- 172.21.15.28:0/2815727532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fba980 msgr2=0x55a377fc8520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.437+0000 7ff4df77b740 1 --2- 172.21.15.28:0/2815727532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fba980 0x55a377fc8520 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc09b7f0 tx=0x7ff4cc00fd60 comp rx=0 tx=0).stop 2024-04-25T01:34:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.437+0000 7ff4df77b740 1 -- 172.21.15.28:0/2815727532 shutdown_connections 2024-04-25T01:34:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.437+0000 7ff4df77b740 1 --2- 172.21.15.28:0/2815727532 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55a37803b250 0x55a37803d670 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.437+0000 7ff4df77b740 1 --2- 172.21.15.28:0/2815727532 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a377fc4010 0x55a378039490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.437+0000 7ff4df77b740 1 --2- 172.21.15.28:0/2815727532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a377fba980 0x55a377fc8520 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.437+0000 7ff4bdffb640 0 client.16251 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.437+0000 7ff4df77b740 1 --2- 172.21.15.28:0/2815727532 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a377fb9290 0x55a377fc7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.437+0000 7ff4df77b740 1 -- 172.21.15.28:0/2815727532 >> 172.21.15.28:0/2815727532 conn(0x55a377f81950 msgr2=0x55a377fc0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.437+0000 7ff4df77b740 1 -- 172.21.15.28:0/2815727532 shutdown_connections 2024-04-25T01:34:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.437+0000 7ff4df77b740 1 -- 172.21.15.28:0/2815727532 wait complete. 2024-04-25T01:34:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.437+0000 7ff4df77b740 20 client.16251 trim_cache size 0 max 16384 2024-04-25T01:34:44.421 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:34:44.421 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:34:44.421 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "b6cd89c8-194a-4cc1-aba9-352c6d2e67b5", 2024-04-25T01:34:44.421 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:34:44.421 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:34:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.438+0000 7ff4dca6b640 1 -- 172.21.15.28:0/1518976993 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4c00a2d00 msgr2=0x7ff4c00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.438+0000 7ff4dca6b640 1 --2- 172.21.15.28:0/1518976993 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4c00a2d00 0x7ff4c00a51c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0009a60 tx=0x7ff4d0033000 comp rx=0 tx=0).stop 2024-04-25T01:34:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.438+0000 7ff4dca6b640 1 -- 172.21.15.28:0/1518976993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d81032e0 msgr2=0x7ff4d8198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.438+0000 7ff4dca6b640 1 --2- 172.21.15.28:0/1518976993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d81032e0 0x7ff4d8198470 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc00c920 tx=0x7ff4cc00cde0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.438+0000 7ff4dca6b640 1 -- 172.21.15.28:0/1518976993 shutdown_connections 2024-04-25T01:34:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.438+0000 7ff4dca6b640 1 --2- 172.21.15.28:0/1518976993 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4c00a2d00 0x7ff4c00a51c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.438+0000 7ff4dca6b640 1 --2- 172.21.15.28:0/1518976993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff4d810bd00 0x7ff4d819c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.438+0000 7ff4dca6b640 1 --2- 172.21.15.28:0/1518976993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d81032e0 0x7ff4d8198470 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.438+0000 7ff4dca6b640 1 --2- 172.21.15.28:0/1518976993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff4d81029c0 0x7ff4d8197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.438+0000 7ff4dca6b640 1 -- 172.21.15.28:0/1518976993 >> 172.21.15.28:0/1518976993 conn(0x7ff4d8077740 msgr2=0x7ff4d80fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.438+0000 7ff4dca6b640 1 -- 172.21.15.28:0/1518976993 shutdown_connections 2024-04-25T01:34:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.438+0000 7ff4dca6b640 1 -- 172.21.15.28:0/1518976993 wait complete. 2024-04-25T01:34:44.425 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:34:44.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:34:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.594+0000 7fbef14da640 1 -- 172.21.15.28:0/908257465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec06a780 msgr2=0x7fbeec10e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.594+0000 7fbef14da640 1 --2- 172.21.15.28:0/908257465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec06a780 0x7fbeec10e5f0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fbee4009cd0 tx=0x7fbee402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.594+0000 7fbef14da640 1 -- 172.21.15.28:0/908257465 shutdown_connections 2024-04-25T01:34:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.594+0000 7fbef14da640 1 --2- 172.21.15.28:0/908257465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec06a780 0x7fbeec10e5f0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.594+0000 7fbef14da640 1 --2- 172.21.15.28:0/908257465 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec069dc0 0x7fbeec06a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.594+0000 7fbef14da640 1 --2- 172.21.15.28:0/908257465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeec107730 0x7fbeec107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.594+0000 7fbef14da640 1 -- 172.21.15.28:0/908257465 >> 172.21.15.28:0/908257465 conn(0x7fbeec0776a0 msgr2=0x7fbeec077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.594+0000 7fbef14da640 1 -- 172.21.15.28:0/908257465 shutdown_connections 2024-04-25T01:34:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.594+0000 7fbef14da640 1 -- 172.21.15.28:0/908257465 wait complete. 2024-04-25T01:34:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.594+0000 7fbef14da640 1 Processor -- start 2024-04-25T01:34:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbef14da640 1 -- start start 2024-04-25T01:34:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbef14da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec069dc0 0x7fbeec1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbef14da640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeec06a780 0x7fbeec198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbef14da640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec107730 0x7fbeec19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbef14da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec111b40 con 0x7fbeec069dc0 2024-04-25T01:34:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbef14da640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec1119c0 con 0x7fbeec06a780 2024-04-25T01:34:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbef14da640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec111cc0 con 0x7fbeec107730 2024-04-25T01:34:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbeeaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec069dc0 0x7fbeec1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbeeaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec069dc0 0x7fbeec1980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42124/0 (socket says 172.21.15.28:42124) 2024-04-25T01:34:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbeeaffd640 1 -- 172.21.15.28:0/452159452 learned_addr learned my addr 172.21.15.28:0/452159452 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbeeb7fe640 1 --2- 172.21.15.28:0/452159452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec107730 0x7fbeec19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbeeaffd640 1 -- 172.21.15.28:0/452159452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec107730 msgr2=0x7fbeec19c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:44.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbeeaffd640 1 --2- 172.21.15.28:0/452159452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec107730 0x7fbeec19c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbee3fff640 1 --2- 172.21.15.28:0/452159452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeec06a780 0x7fbeec198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbeeaffd640 1 -- 172.21.15.28:0/452159452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeec06a780 msgr2=0x7fbeec198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:44.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbeeaffd640 1 --2- 172.21.15.28:0/452159452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeec06a780 0x7fbeec198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbeeaffd640 1 -- 172.21.15.28:0/452159452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeec19d0b0 con 0x7fbeec069dc0 2024-04-25T01:34:44.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbeeb7fe640 1 --2- 172.21.15.28:0/452159452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec107730 0x7fbeec19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:44.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbeeaffd640 1 --2- 172.21.15.28:0/452159452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec069dc0 0x7fbeec1980c0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00d950 tx=0x7fbedc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:44.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.595+0000 7fbee8ff9640 1 -- 172.21.15.28:0/452159452 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc017030 con 0x7fbeec069dc0 2024-04-25T01:34:44.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.596+0000 7fbee8ff9640 1 -- 172.21.15.28:0/452159452 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbedc004020 con 0x7fbeec069dc0 2024-04-25T01:34:44.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.596+0000 7fbef14da640 1 -- 172.21.15.28:0/452159452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeec19d3a0 con 0x7fbeec069dc0 2024-04-25T01:34:44.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.596+0000 7fbee8ff9640 1 -- 172.21.15.28:0/452159452 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc0047a0 con 0x7fbeec069dc0 2024-04-25T01:34:44.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.596+0000 7fbef14da640 1 -- 172.21.15.28:0/452159452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeec1a4c30 con 0x7fbeec069dc0 2024-04-25T01:34:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.598+0000 7fbee8ff9640 1 -- 172.21.15.28:0/452159452 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fbedc0041d0 con 0x7fbeec069dc0 2024-04-25T01:34:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.598+0000 7fbef41ed740 1 Processor -- start 2024-04-25T01:34:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.598+0000 7fbef41ed740 1 -- start start 2024-04-25T01:34:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbef41ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad48a40 0x55d69ad48e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbef41ed740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad49360 0x55d69ad534f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbee8ff9640 1 --2- 172.21.15.28:0/452159452 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbed80a2d00 0x7fbed80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbef41ed740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad53a30 0x55d69ad55e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbef41ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d69aca96b0 con 0x55d69ad48a40 2024-04-25T01:34:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbef41ed740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d69ac51880 con 0x55d69ad53a30 2024-04-25T01:34:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbef41ed740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55d69aca5670 con 0x55d69ad49360 2024-04-25T01:34:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbeeb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad48a40 0x55d69ad48e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbee3fff640 1 --2- 172.21.15.28:0/452159452 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbed80a2d00 0x7fbed80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbeeb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad48a40 0x55d69ad48e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42130/0 (socket says 172.21.15.28:42130) 2024-04-25T01:34:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbeeb7fe640 1 -- 172.21.15.28:0/3821589536 learned_addr learned my addr 172.21.15.28:0/3821589536 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbeeaffd640 1 --2- 172.21.15.28:0/3821589536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad49360 0x55d69ad534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbee8ff9640 1 -- 172.21.15.28:0/452159452 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fbedc0cf1a0 con 0x7fbeec069dc0 2024-04-25T01:34:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbeeb7fe640 1 --2- 172.21.15.28:0/3821589536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad53a30 0x55d69ad55e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbeeaffd640 1 -- 172.21.15.28:0/3821589536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad53a30 msgr2=0x55d69ad55e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbeeaffd640 1 --2- 172.21.15.28:0/3821589536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad53a30 0x55d69ad55e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbeeaffd640 1 -- 172.21.15.28:0/3821589536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad48a40 msgr2=0x55d69ad48e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbeeaffd640 1 --2- 172.21.15.28:0/3821589536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad48a40 0x55d69ad48e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbeeaffd640 1 -- 172.21.15.28:0/3821589536 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d69aa877d0 con 0x55d69ad49360 2024-04-25T01:34:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbeeb7fe640 1 --2- 172.21.15.28:0/3821589536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad53a30 0x55d69ad55e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:34:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbee3fff640 1 --2- 172.21.15.28:0/452159452 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbed80a2d00 0x7fbed80a51c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbed4005fd0 tx=0x7fbed4005850 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.599+0000 7fbeeaffd640 1 --2- 172.21.15.28:0/3821589536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad49360 0x55d69ad534f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0cc5b0 tx=0x7fbedc020350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1ba5e4bde233b87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.600+0000 7fbee17fa640 1 -- 172.21.15.28:0/3821589536 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc09aec0 con 0x55d69ad49360 2024-04-25T01:34:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.600+0000 7fbee17fa640 1 -- 172.21.15.28:0/3821589536 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbedc09ba80 con 0x55d69ad49360 2024-04-25T01:34:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.600+0000 7fbee17fa640 1 -- 172.21.15.28:0/3821589536 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc09ace0 con 0x55d69ad49360 2024-04-25T01:34:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.600+0000 7fbef41ed740 1 -- 172.21.15.28:0/3821589536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad49360 msgr2=0x55d69ad534f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.600+0000 7fbef41ed740 1 --2- 172.21.15.28:0/3821589536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad49360 0x55d69ad534f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0cc5b0 tx=0x7fbedc020350 comp rx=0 tx=0).stop 2024-04-25T01:34:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.600+0000 7fbef41ed740 1 -- 172.21.15.28:0/3821589536 shutdown_connections 2024-04-25T01:34:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.600+0000 7fbef41ed740 1 --2- 172.21.15.28:0/3821589536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad53a30 0x55d69ad55e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.600+0000 7fbef41ed740 1 --2- 172.21.15.28:0/3821589536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad49360 0x55d69ad534f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.600+0000 7fbef41ed740 1 --2- 172.21.15.28:0/3821589536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad48a40 0x55d69ad48e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.600+0000 7fbef41ed740 1 -- 172.21.15.28:0/3821589536 >> 172.21.15.28:0/3821589536 conn(0x55d69ad44d60 msgr2=0x55d69ad45140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.600+0000 7fbef41ed740 1 -- 172.21.15.28:0/3821589536 shutdown_connections 2024-04-25T01:34:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.600+0000 7fbef41ed740 1 -- 172.21.15.28:0/3821589536 wait complete. 2024-04-25T01:34:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbef41ed740 1 -- start start 2024-04-25T01:34:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbef41ed740 1 Processor -- start 2024-04-25T01:34:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbef41ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad48a40 0x55d69ad508c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbef41ed740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad49360 0x55d69ad50e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbef41ed740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad53a30 0x55d69ad51340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbef41ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d69ad56820 con 0x55d69ad48a40 2024-04-25T01:34:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbef41ed740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d69ac64f50 con 0x55d69ad53a30 2024-04-25T01:34:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbef41ed740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55d69ad569a0 con 0x55d69ad49360 2024-04-25T01:34:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbef41ed740 10 client.0 _ll_register_callbacks cb 0x55d69aca0000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbeeaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad48a40 0x55d69ad508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbeeaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad48a40 0x55d69ad508c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42144/0 (socket says 172.21.15.28:42144) 2024-04-25T01:34:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbeeaffd640 1 -- 172.21.15.28:0/145970954 learned_addr learned my addr 172.21.15.28:0/145970954 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbee3fff640 1 --2- 172.21.15.28:0/145970954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad49360 0x55d69ad50e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbeeb7fe640 1 --2- 172.21.15.28:0/145970954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad53a30 0x55d69ad51340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbeeaffd640 1 -- 172.21.15.28:0/145970954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad49360 msgr2=0x55d69ad50e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbeeaffd640 1 --2- 172.21.15.28:0/145970954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad49360 0x55d69ad50e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbeeaffd640 1 -- 172.21.15.28:0/145970954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad53a30 msgr2=0x55d69ad51340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbeeaffd640 1 --2- 172.21.15.28:0/145970954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad53a30 0x55d69ad51340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbeeaffd640 1 -- 172.21.15.28:0/145970954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d69a9bd3f0 con 0x55d69ad48a40 2024-04-25T01:34:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.601+0000 7fbee3fff640 1 --2- 172.21.15.28:0/145970954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad49360 0x55d69ad50e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:34:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.602+0000 7fbeeaffd640 1 --2- 172.21.15.28:0/145970954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad48a40 0x55d69ad508c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0cc660 tx=0x7fbedc097ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc38605ac19e4028 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.602+0000 7fbed27fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.602+0000 7fbed27fc640 1 -- 172.21.15.28:0/145970954 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc09b380 con 0x55d69ad48a40 2024-04-25T01:34:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.602+0000 7fbed27fc640 1 -- 172.21.15.28:0/145970954 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fbedc050de0 con 0x55d69ad48a40 2024-04-25T01:34:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.602+0000 7fbef41ed740 1 -- 172.21.15.28:0/145970954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbedc09ba80 con 0x55d69ad48a40 2024-04-25T01:34:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.602+0000 7fbed27fc640 1 -- 172.21.15.28:0/145970954 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbedc09f730 con 0x55d69ad48a40 2024-04-25T01:34:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.602+0000 7fbef41ed740 10 client.16263 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.602+0000 7fbef41ed740 1 -- 172.21.15.28:0/145970954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d69a97d0f0 con 0x55d69ad48a40 2024-04-25T01:34:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.602+0000 7fbed27fc640 1 -- 172.21.15.28:0/145970954 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbedc09ba80 con 0x55d69ad48a40 2024-04-25T01:34:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.602+0000 7fbed27fc640 1 -- 172.21.15.28:0/145970954 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbedc0a3bf0 con 0x55d69ad48a40 2024-04-25T01:34:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.602+0000 7fbef41ed740 10 client.16263 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.602+0000 7fbef41ed740 10 client.16263 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.602+0000 7fbef41ed740 20 client.16263 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.602+0000 7fbef41ed740 1 --2- 172.21.15.28:0/145970954 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55d69ad63a10 0x55d69ad65e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.602+0000 7fbef41ed740 4 client.16263 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:34:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.602+0000 7fbef41ed740 1 -- 172.21.15.28:0/145970954 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d69acb6bf0 con 0x55d69ad63a10 2024-04-25T01:34:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.603+0000 7fbee3fff640 1 --2- 172.21.15.28:0/145970954 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55d69ad63a10 0x55d69ad65e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.603+0000 7fbee3fff640 1 --2- 172.21.15.28:0/145970954 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55d69ad63a10 0x55d69ad65e00 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef76d6df0853ccd server_cookie=a93afd501aafe3e7 in_seq=0 out_seq=0 2024-04-25T01:34:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.603+0000 7fbed27fc640 10 client.16263 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.603+0000 7fbed27fc640 1 -- 172.21.15.28:0/145970954 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d69acb6bf0 con 0x55d69ad63a10 2024-04-25T01:34:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.603+0000 7fbed27fc640 10 client.16263 handle_command_reply: tid=0 2024-04-25T01:34:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.604+0000 7fbef41ed740 1 client.16263 shutdown 2024-04-25T01:34:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.604+0000 7fbef41ed740 1 -- 172.21.15.28:0/145970954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad48a40 msgr2=0x55d69ad508c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.604+0000 7fbef41ed740 1 --2- 172.21.15.28:0/145970954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad48a40 0x55d69ad508c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0cc660 tx=0x7fbedc097ca0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.604+0000 7fbef41ed740 1 -- 172.21.15.28:0/145970954 shutdown_connections 2024-04-25T01:34:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.604+0000 7fbef41ed740 1 --2- 172.21.15.28:0/145970954 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55d69ad63a10 0x55d69ad65e00 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.604+0000 7fbef41ed740 1 --2- 172.21.15.28:0/145970954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad53a30 0x55d69ad51340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.604+0000 7fbed27fc640 0 client.16263 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.604+0000 7fbef41ed740 1 --2- 172.21.15.28:0/145970954 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad49360 0x55d69ad50e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.604+0000 7fbef41ed740 1 --2- 172.21.15.28:0/145970954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad48a40 0x55d69ad508c0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.604+0000 7fbef41ed740 1 -- 172.21.15.28:0/145970954 >> 172.21.15.28:0/145970954 conn(0x55d69ad44d60 msgr2=0x55d69ad4ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.604+0000 7fbef41ed740 1 -- 172.21.15.28:0/145970954 shutdown_connections 2024-04-25T01:34:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.604+0000 7fbef41ed740 1 -- 172.21.15.28:0/145970954 wait complete. 2024-04-25T01:34:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.604+0000 7fbef41ed740 20 client.16263 trim_cache size 0 max 16384 2024-04-25T01:34:44.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.621+0000 7fbef41ed740 1 Processor -- start 2024-04-25T01:34:44.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.621+0000 7fbef41ed740 1 -- start start 2024-04-25T01:34:44.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.621+0000 7fbef41ed740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad48a40 0x55d69ad52ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.621+0000 7fbef41ed740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad49360 0x55d69ad4fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.621+0000 7fbef41ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad53a30 0x55d69ad5f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.621+0000 7fbef41ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec1ac8e0 con 0x55d69ad53a30 2024-04-25T01:34:44.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.621+0000 7fbef41ed740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec1ae420 con 0x55d69ad49360 2024-04-25T01:34:44.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.621+0000 7fbef41ed740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec1acfb0 con 0x55d69ad48a40 2024-04-25T01:34:44.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.622+0000 7fbeeb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad53a30 0x55d69ad5f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.622+0000 7fbeeb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad53a30 0x55d69ad5f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42158/0 (socket says 172.21.15.28:42158) 2024-04-25T01:34:44.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.622+0000 7fbeeb7fe640 1 -- 172.21.15.28:0/466166586 learned_addr learned my addr 172.21.15.28:0/466166586 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:44.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.622+0000 7fbee3fff640 1 --2- 172.21.15.28:0/466166586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad49360 0x55d69ad4fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.622+0000 7fbeeaffd640 1 --2- 172.21.15.28:0/466166586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad48a40 0x55d69ad52ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.622+0000 7fbeeb7fe640 1 -- 172.21.15.28:0/466166586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad48a40 msgr2=0x55d69ad52ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:44.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.622+0000 7fbeeb7fe640 1 --2- 172.21.15.28:0/466166586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad48a40 0x55d69ad52ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.622+0000 7fbeeb7fe640 1 -- 172.21.15.28:0/466166586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad49360 msgr2=0x55d69ad4fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:44.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.622+0000 7fbeeb7fe640 1 --2- 172.21.15.28:0/466166586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad49360 0x55d69ad4fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.622+0000 7fbeeb7fe640 1 -- 172.21.15.28:0/466166586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d69ad49190 con 0x55d69ad53a30 2024-04-25T01:34:44.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.622+0000 7fbeeb7fe640 1 --2- 172.21.15.28:0/466166586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad53a30 0x55d69ad5f300 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fbee4031730 tx=0x7fbee4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a25de78d0984 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:44.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.622+0000 7fbee0ff9640 1 -- 172.21.15.28:0/466166586 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee4031980 con 0x55d69ad53a30 2024-04-25T01:34:44.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.622+0000 7fbee0ff9640 1 -- 172.21.15.28:0/466166586 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbee4036040 con 0x55d69ad53a30 2024-04-25T01:34:44.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.622+0000 7fbee0ff9640 1 -- 172.21.15.28:0/466166586 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee40046a0 con 0x55d69ad53a30 2024-04-25T01:34:44.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.623+0000 7fbef41ed740 1 -- 172.21.15.28:0/466166586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad53a30 msgr2=0x55d69ad5f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.623+0000 7fbef41ed740 1 --2- 172.21.15.28:0/466166586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad53a30 0x55d69ad5f300 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fbee4031730 tx=0x7fbee4005dc0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.623+0000 7fbef41ed740 1 -- 172.21.15.28:0/466166586 shutdown_connections 2024-04-25T01:34:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.623+0000 7fbef41ed740 1 --2- 172.21.15.28:0/466166586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad53a30 0x55d69ad5f300 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.623+0000 7fbef41ed740 1 --2- 172.21.15.28:0/466166586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad49360 0x55d69ad4fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.623+0000 7fbef41ed740 1 --2- 172.21.15.28:0/466166586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad48a40 0x55d69ad52ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.623+0000 7fbef41ed740 1 -- 172.21.15.28:0/466166586 >> 172.21.15.28:0/466166586 conn(0x55d69ad44d60 msgr2=0x55d69ad5ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.623+0000 7fbef41ed740 1 -- 172.21.15.28:0/466166586 shutdown_connections 2024-04-25T01:34:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.623+0000 7fbef41ed740 1 -- 172.21.15.28:0/466166586 wait complete. 2024-04-25T01:34:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.623+0000 7fbef41ed740 1 -- start start 2024-04-25T01:34:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.623+0000 7fbef41ed740 1 Processor -- start 2024-04-25T01:34:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbef41ed740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad48a40 0x55d69ad66040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbef41ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad49360 0x55d69ad66580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbef41ed740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad53a30 0x55d69adc7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbef41ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec1add50 con 0x55d69ad49360 2024-04-25T01:34:44.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbef41ed740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec1ad680 con 0x55d69ad48a40 2024-04-25T01:34:44.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbef41ed740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec1aa8c0 con 0x55d69ad53a30 2024-04-25T01:34:44.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbef41ed740 10 client.0 _ll_register_callbacks cb 0x7fbedc009740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:34:44.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbee3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad49360 0x55d69ad66580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbee3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad49360 0x55d69ad66580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42170/0 (socket says 172.21.15.28:42170) 2024-04-25T01:34:44.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbee3fff640 1 -- 172.21.15.28:0/4031807634 learned_addr learned my addr 172.21.15.28:0/4031807634 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:34:44.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbeeb7fe640 1 --2- 172.21.15.28:0/4031807634 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad53a30 0x55d69adc7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbeeaffd640 1 --2- 172.21.15.28:0/4031807634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad48a40 0x55d69ad66040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbee3fff640 1 -- 172.21.15.28:0/4031807634 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad53a30 msgr2=0x55d69adc7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:44.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbee3fff640 1 --2- 172.21.15.28:0/4031807634 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad53a30 0x55d69adc7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbee3fff640 1 -- 172.21.15.28:0/4031807634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad48a40 msgr2=0x55d69ad66040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:34:44.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbee3fff640 1 --2- 172.21.15.28:0/4031807634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad48a40 0x55d69ad66040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbee3fff640 1 -- 172.21.15.28:0/4031807634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d69adc81d0 con 0x55d69ad49360 2024-04-25T01:34:44.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbee3fff640 1 --2- 172.21.15.28:0/4031807634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad49360 0x55d69ad66580 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fbed40058c0 tx=0x7fbed4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60e71a2f45354a3a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:34:44.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbed27fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:34:44.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.624+0000 7fbed27fc640 1 -- 172.21.15.28:0/4031807634 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed4027dc0 con 0x55d69ad49360 2024-04-25T01:34:44.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.625+0000 7fbed27fc640 1 -- 172.21.15.28:0/4031807634 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fbed4011ab0 con 0x55d69ad49360 2024-04-25T01:34:44.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.625+0000 7fbef41ed740 1 -- 172.21.15.28:0/4031807634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbee4036040 con 0x55d69ad49360 2024-04-25T01:34:44.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.625+0000 7fbed27fc640 1 -- 172.21.15.28:0/4031807634 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbed4038050 con 0x55d69ad49360 2024-04-25T01:34:44.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.625+0000 7fbef41ed740 10 client.16269 fetch_fsmap learned FSMap version 28 2024-04-25T01:34:44.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.625+0000 7fbef41ed740 1 -- 172.21.15.28:0/4031807634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d69adc9550 con 0x55d69ad49360 2024-04-25T01:34:44.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.625+0000 7fbed27fc640 1 -- 172.21.15.28:0/4031807634 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbee4036040 con 0x55d69ad49360 2024-04-25T01:34:44.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.625+0000 7fbed27fc640 1 -- 172.21.15.28:0/4031807634 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbed400f040 con 0x55d69ad49360 2024-04-25T01:34:44.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.625+0000 7fbef41ed740 10 client.16269 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:34:44.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.625+0000 7fbef41ed740 10 client.16269 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:34:44.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.625+0000 7fbef41ed740 20 client.16269 populate_metadata read hostname 'smithi028' 2024-04-25T01:34:44.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.625+0000 7fbef41ed740 1 --2- 172.21.15.28:0/4031807634 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55d69adc9cb0 0x55d69adcc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:34:44.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.625+0000 7fbef41ed740 4 client.16269 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:34:44.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.625+0000 7fbef41ed740 1 -- 172.21.15.28:0/4031807634 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d69acb0f40 con 0x55d69adc9cb0 2024-04-25T01:34:44.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.625+0000 7fbeeaffd640 1 --2- 172.21.15.28:0/4031807634 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55d69adc9cb0 0x55d69adcc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:34:44.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.626+0000 7fbeeaffd640 1 --2- 172.21.15.28:0/4031807634 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55d69adc9cb0 0x55d69adcc0d0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c37a08f9f3e7839 server_cookie=2c20410f429e2be5 in_seq=0 out_seq=0 2024-04-25T01:34:44.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.626+0000 7fbed27fc640 10 client.16269 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:44.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.626+0000 7fbed27fc640 1 -- 172.21.15.28:0/4031807634 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d69acb0f40 con 0x55d69adc9cb0 2024-04-25T01:34:44.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.626+0000 7fbed27fc640 10 client.16269 handle_command_reply: tid=0 2024-04-25T01:34:44.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.626+0000 7fbef41ed740 1 client.16269 shutdown 2024-04-25T01:34:44.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.626+0000 7fbef41ed740 1 -- 172.21.15.28:0/4031807634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad49360 msgr2=0x55d69ad66580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:44.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.626+0000 7fbef41ed740 1 --2- 172.21.15.28:0/4031807634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad49360 0x55d69ad66580 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fbed40058c0 tx=0x7fbed4004730 comp rx=0 tx=0).stop 2024-04-25T01:34:44.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.626+0000 7fbef41ed740 1 -- 172.21.15.28:0/4031807634 shutdown_connections 2024-04-25T01:34:44.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.626+0000 7fbef41ed740 1 --2- 172.21.15.28:0/4031807634 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55d69adc9cb0 0x55d69adcc0d0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.626+0000 7fbef41ed740 1 --2- 172.21.15.28:0/4031807634 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d69ad53a30 0x55d69adc7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.626+0000 7fbef41ed740 1 --2- 172.21.15.28:0/4031807634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d69ad49360 0x55d69ad66580 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.626+0000 7fbed27fc640 0 client.16269 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:34:44.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.626+0000 7fbef41ed740 1 --2- 172.21.15.28:0/4031807634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d69ad48a40 0x55d69ad66040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.626+0000 7fbef41ed740 1 -- 172.21.15.28:0/4031807634 >> 172.21.15.28:0/4031807634 conn(0x55d69ad44d60 msgr2=0x55d69ad5f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:44.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.626+0000 7fbef41ed740 1 -- 172.21.15.28:0/4031807634 shutdown_connections 2024-04-25T01:34:44.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.626+0000 7fbef41ed740 1 -- 172.21.15.28:0/4031807634 wait complete. 2024-04-25T01:34:44.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.626+0000 7fbef41ed740 20 client.16269 trim_cache size 0 max 16384 2024-04-25T01:34:44.610 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:34:44.610 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:34:44.610 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-25T01:34:44.610 INFO:teuthology.orchestra.run.smithi028.stdout: "b6cd89c8-194a-4cc1-aba9-352c6d2e67b5": { 2024-04-25T01:34:44.611 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-25T01:34:44.611 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "b6cd89c8-194a-4cc1-aba9-352c6d2e67b5", 2024-04-25T01:34:44.611 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-25T01:34:44.611 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:34:44.611 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:34:44.611 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:34:44.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.626+0000 7fbef14da640 1 -- 172.21.15.28:0/452159452 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbed80a2d00 msgr2=0x7fbed80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:44.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.627+0000 7fbef14da640 1 --2- 172.21.15.28:0/452159452 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbed80a2d00 0x7fbed80a51c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbed4005fd0 tx=0x7fbed4005850 comp rx=0 tx=0).stop 2024-04-25T01:34:44.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.627+0000 7fbef14da640 1 -- 172.21.15.28:0/452159452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec069dc0 msgr2=0x7fbeec1980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:34:44.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.627+0000 7fbef14da640 1 --2- 172.21.15.28:0/452159452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec069dc0 0x7fbeec1980c0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00d950 tx=0x7fbedc00de10 comp rx=0 tx=0).stop 2024-04-25T01:34:44.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.627+0000 7fbef14da640 1 -- 172.21.15.28:0/452159452 shutdown_connections 2024-04-25T01:34:44.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.627+0000 7fbef14da640 1 --2- 172.21.15.28:0/452159452 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbed80a2d00 0x7fbed80a51c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.627+0000 7fbef14da640 1 --2- 172.21.15.28:0/452159452 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbeec107730 0x7fbeec19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.627+0000 7fbef14da640 1 --2- 172.21.15.28:0/452159452 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeec06a780 0x7fbeec198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.627+0000 7fbef14da640 1 --2- 172.21.15.28:0/452159452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec069dc0 0x7fbeec1980c0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:34:44.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.627+0000 7fbef14da640 1 -- 172.21.15.28:0/452159452 >> 172.21.15.28:0/452159452 conn(0x7fbeec0776a0 msgr2=0x7fbeec0fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:34:44.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.627+0000 7fbef14da640 1 -- 172.21.15.28:0/452159452 shutdown_connections 2024-04-25T01:34:44.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:34:44.627+0000 7fbef14da640 1 -- 172.21.15.28:0/452159452 wait complete. 2024-04-25T01:34:44.614 INFO:tasks.cephfs.filesystem:scrub status for tag:b6cd89c8-194a-4cc1-aba9-352c6d2e67b5 - {'path': '/', 'tag': 'b6cd89c8-194a-4cc1-aba9-352c6d2e67b5', 'options': 'recursive,force'} 2024-04-25T01:34:46.207 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:45 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:34:46.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:45 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:46.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:45 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:34:46.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:45 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:46.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:45 smithi130 ceph-mon[32702]: pgmap v55: 129 pgs: 129 active+clean; 2.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 85 MiB/s wr, 31 op/s 2024-04-25T01:34:46.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:45 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:34:46.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:45 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:46.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:45 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:34:46.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:45 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:46.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:45 smithi032 ceph-mon[32461]: pgmap v55: 129 pgs: 129 active+clean; 2.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 85 MiB/s wr, 31 op/s 2024-04-25T01:34:46.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:45 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:34:46.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:45 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:46.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:45 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:34:46.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:45 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:34:46.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:45 smithi028 ceph-mon[29390]: pgmap v55: 129 pgs: 129 active+clean; 2.0 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 85 MiB/s wr, 31 op/s 2024-04-25T01:34:47.469 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:47.469 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 readers = 7663226.00 kB/sec 2024-04-25T01:34:47.469 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 readers = 7650708.44 kB/sec 2024-04-25T01:34:47.470 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 7663226.00 kB/sec 2024-04-25T01:34:47.470 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 7663226.00 kB/sec 2024-04-25T01:34:47.470 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 7663226.00 kB/sec 2024-04-25T01:34:47.470 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 1048576.00 kB 2024-04-25T01:34:48.207 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:47 smithi130 ceph-mon[32702]: pgmap v56: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 171 MiB/s wr, 79 op/s 2024-04-25T01:34:48.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:47 smithi032 ceph-mon[32461]: pgmap v56: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 171 MiB/s wr, 79 op/s 2024-04-25T01:34:48.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:47 smithi028 ceph-mon[29390]: pgmap v56: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 171 MiB/s wr, 79 op/s 2024-04-25T01:34:49.207 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:48 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:34:49.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:48 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:34:49.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:48 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:34:49.687 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:49.687 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 re-readers = 7621012.00 kB/sec 2024-04-25T01:34:49.687 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 re-readers = 7610620.13 kB/sec 2024-04-25T01:34:49.687 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 7621012.00 kB/sec 2024-04-25T01:34:49.687 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 7621012.00 kB/sec 2024-04-25T01:34:49.687 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 7621012.00 kB/sec 2024-04-25T01:34:49.687 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 1048576.00 kB 2024-04-25T01:34:50.207 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:49 smithi130 ceph-mon[32702]: pgmap v57: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 171 MiB/s wr, 75 op/s 2024-04-25T01:34:50.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:49 smithi032 ceph-mon[32461]: pgmap v57: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 171 MiB/s wr, 75 op/s 2024-04-25T01:34:50.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:49 smithi028 ceph-mon[29390]: pgmap v57: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 171 MiB/s wr, 75 op/s 2024-04-25T01:34:52.207 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:51 smithi130 ceph-mon[32702]: pgmap v58: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 171 MiB/s wr, 75 op/s 2024-04-25T01:34:52.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:51 smithi032 ceph-mon[32461]: pgmap v58: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 171 MiB/s wr, 75 op/s 2024-04-25T01:34:52.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:51 smithi028 ceph-mon[29390]: pgmap v58: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 171 MiB/s wr, 75 op/s 2024-04-25T01:34:54.207 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:53 smithi130 ceph-mon[32702]: pgmap v59: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 171 MiB/s wr, 80 op/s 2024-04-25T01:34:54.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:53 smithi032 ceph-mon[32461]: pgmap v59: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 171 MiB/s wr, 80 op/s 2024-04-25T01:34:54.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:53 smithi028 ceph-mon[29390]: pgmap v59: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 171 MiB/s wr, 80 op/s 2024-04-25T01:34:56.207 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:55 smithi130 ceph-mon[32702]: pgmap v60: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 85 MiB/s wr, 52 op/s 2024-04-25T01:34:56.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:55 smithi032 ceph-mon[32461]: pgmap v60: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 85 MiB/s wr, 52 op/s 2024-04-25T01:34:56.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:55 smithi028 ceph-mon[29390]: pgmap v60: 129 pgs: 129 active+clean; 1.0 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 85 MiB/s wr, 52 op/s 2024-04-25T01:34:56.331 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:56.331 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:56.332 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:56.332 INFO:tasks.workunit.client.0.smithi028.stdout:iozone test complete. 2024-04-25T01:34:56.332 INFO:tasks.workunit.client.0.smithi028.stderr:+ iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-04-25T01:34:56.399 INFO:tasks.workunit.client.0.smithi028.stdout: Iozone: Performance Test of File I/O 2024-04-25T01:34:56.399 INFO:tasks.workunit.client.0.smithi028.stdout: Version $Revision: 3.494 $ 2024-04-25T01:34:56.399 INFO:tasks.workunit.client.0.smithi028.stdout: Compiled for 64 bit mode. 2024-04-25T01:34:56.399 INFO:tasks.workunit.client.0.smithi028.stdout: Build: linux-AMD64 2024-04-25T01:34:56.399 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:56.399 INFO:tasks.workunit.client.0.smithi028.stdout: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins 2024-04-25T01:34:56.399 INFO:tasks.workunit.client.0.smithi028.stdout: Al Slater, Scott Rhine, Mike Wisner, Ken Goss 2024-04-25T01:34:56.400 INFO:tasks.workunit.client.0.smithi028.stdout: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, 2024-04-25T01:34:56.400 INFO:tasks.workunit.client.0.smithi028.stdout: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, 2024-04-25T01:34:56.400 INFO:tasks.workunit.client.0.smithi028.stdout: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, 2024-04-25T01:34:56.400 INFO:tasks.workunit.client.0.smithi028.stdout: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, 2024-04-25T01:34:56.400 INFO:tasks.workunit.client.0.smithi028.stdout: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, 2024-04-25T01:34:56.400 INFO:tasks.workunit.client.0.smithi028.stdout: Vangel Bojaxhi, Ben England, Vikentsi Lapa, 2024-04-25T01:34:56.400 INFO:tasks.workunit.client.0.smithi028.stdout: Alexey Skidanov, Sudhir Kumar. 2024-04-25T01:34:56.400 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:56.400 INFO:tasks.workunit.client.0.smithi028.stdout: Run began: Thu Apr 25 01:34:56 2024 2024-04-25T01:34:56.400 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:34:56.400 INFO:tasks.workunit.client.0.smithi028.stdout: Include close in write timing 2024-04-25T01:34:56.400 INFO:tasks.workunit.client.0.smithi028.stdout: Include fsync in write timing 2024-04-25T01:34:56.400 INFO:tasks.workunit.client.0.smithi028.stdout: File size set to 10485760 kB 2024-04-25T01:34:56.400 INFO:tasks.workunit.client.0.smithi028.stdout: Record Size 1024 kB 2024-04-25T01:34:56.400 INFO:tasks.workunit.client.0.smithi028.stdout: Command line used: iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 2024-04-25T01:34:56.401 INFO:tasks.workunit.client.0.smithi028.stdout: Output is in kBytes/sec 2024-04-25T01:34:56.401 INFO:tasks.workunit.client.0.smithi028.stdout: Time Resolution = 0.000001 seconds. 2024-04-25T01:34:56.401 INFO:tasks.workunit.client.0.smithi028.stdout: Processor cache size set to 1024 kBytes. 2024-04-25T01:34:56.401 INFO:tasks.workunit.client.0.smithi028.stdout: Processor cache line size set to 32 bytes. 2024-04-25T01:34:56.401 INFO:tasks.workunit.client.0.smithi028.stdout: File stride size set to 17 * record size. 2024-04-25T01:34:56.401 INFO:tasks.workunit.client.0.smithi028.stdout: Throughput test with 1 process 2024-04-25T01:34:56.401 INFO:tasks.workunit.client.0.smithi028.stdout: Each process writes a 10485760 kByte file in 1024 kByte records 2024-04-25T01:34:58.207 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:57 smithi130 ceph-mon[32702]: pgmap v61: 129 pgs: 129 active+clean; 1.6 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 85 MiB/s wr, 79 op/s 2024-04-25T01:34:58.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:57 smithi032 ceph-mon[32461]: pgmap v61: 129 pgs: 129 active+clean; 1.6 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 85 MiB/s wr, 79 op/s 2024-04-25T01:34:58.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:57 smithi028 ceph-mon[29390]: pgmap v61: 129 pgs: 129 active+clean; 1.6 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 85 MiB/s wr, 79 op/s 2024-04-25T01:35:00.207 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:34:59 smithi130 ceph-mon[32702]: pgmap v62: 129 pgs: 129 active+clean; 1.6 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.5 KiB/s wr, 31 op/s 2024-04-25T01:35:00.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:34:59 smithi032 ceph-mon[32461]: pgmap v62: 129 pgs: 129 active+clean; 1.6 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.5 KiB/s wr, 31 op/s 2024-04-25T01:35:00.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:34:59 smithi028 ceph-mon[29390]: pgmap v62: 129 pgs: 129 active+clean; 1.6 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.5 KiB/s wr, 31 op/s 2024-04-25T01:35:02.207 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:01 smithi130 ceph-mon[32702]: pgmap v63: 129 pgs: 129 active+clean; 1.6 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.5 KiB/s wr, 31 op/s 2024-04-25T01:35:02.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:01 smithi032 ceph-mon[32461]: pgmap v63: 129 pgs: 129 active+clean; 1.6 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.5 KiB/s wr, 31 op/s 2024-04-25T01:35:02.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:01 smithi028 ceph-mon[29390]: pgmap v63: 129 pgs: 129 active+clean; 1.6 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.5 KiB/s wr, 31 op/s 2024-04-25T01:35:04.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:03 smithi130 ceph-mon[32702]: pgmap v64: 129 pgs: 129 active+clean; 4.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 347 MiB/s wr, 310 op/s 2024-04-25T01:35:04.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:03 smithi032 ceph-mon[32461]: pgmap v64: 129 pgs: 129 active+clean; 4.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 347 MiB/s wr, 310 op/s 2024-04-25T01:35:04.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:03 smithi028 ceph-mon[29390]: pgmap v64: 129 pgs: 129 active+clean; 4.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 347 MiB/s wr, 310 op/s 2024-04-25T01:35:06.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:05 smithi130 ceph-mon[32702]: pgmap v65: 129 pgs: 129 active+clean; 4.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 347 MiB/s wr, 306 op/s 2024-04-25T01:35:06.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:05 smithi032 ceph-mon[32461]: pgmap v65: 129 pgs: 129 active+clean; 4.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 347 MiB/s wr, 306 op/s 2024-04-25T01:35:06.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:05 smithi028 ceph-mon[29390]: pgmap v65: 129 pgs: 129 active+clean; 4.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 347 MiB/s wr, 306 op/s 2024-04-25T01:35:07.572 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:35:07.572 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 initial writers = 941454.19 kB/sec 2024-04-25T01:35:07.572 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 initial writers = 941437.82 kB/sec 2024-04-25T01:35:07.572 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 941454.19 kB/sec 2024-04-25T01:35:07.572 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 941454.19 kB/sec 2024-04-25T01:35:07.572 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 941454.19 kB/sec 2024-04-25T01:35:07.572 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 10485760.00 kB 2024-04-25T01:35:08.207 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:07 smithi130 ceph-mon[32702]: pgmap v66: 129 pgs: 129 active+clean; 9.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 774 MiB/s wr, 461 op/s 2024-04-25T01:35:08.207 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:07 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:35:08.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:07 smithi032 ceph-mon[32461]: pgmap v66: 129 pgs: 129 active+clean; 9.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 774 MiB/s wr, 461 op/s 2024-04-25T01:35:08.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:07 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:35:08.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:07 smithi028 ceph-mon[29390]: pgmap v66: 129 pgs: 129 active+clean; 9.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 774 MiB/s wr, 461 op/s 2024-04-25T01:35:08.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:07 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:35:09.207 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:08 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:35:09.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:08 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:35:09.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:08 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:35:09.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:08 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:35:09.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:08 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:35:09.232 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:08 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:35:09.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:08 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:35:09.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:08 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:35:09.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:08 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:35:10.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:10 smithi028 ceph-mon[29390]: pgmap v67: 129 pgs: 129 active+clean; 9.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 774 MiB/s wr, 434 op/s 2024-04-25T01:35:10.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:10 smithi130 ceph-mon[32702]: pgmap v67: 129 pgs: 129 active+clean; 9.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 774 MiB/s wr, 434 op/s 2024-04-25T01:35:10.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:10 smithi032 ceph-mon[32461]: pgmap v67: 129 pgs: 129 active+clean; 9.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 774 MiB/s wr, 434 op/s 2024-04-25T01:35:12.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:12 smithi028 ceph-mon[29390]: pgmap v68: 129 pgs: 129 active+clean; 9.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 774 MiB/s wr, 434 op/s 2024-04-25T01:35:12.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:12 smithi130 ceph-mon[32702]: pgmap v68: 129 pgs: 129 active+clean; 9.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 774 MiB/s wr, 434 op/s 2024-04-25T01:35:12.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:12 smithi032 ceph-mon[32461]: pgmap v68: 129 pgs: 129 active+clean; 9.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 774 MiB/s wr, 434 op/s 2024-04-25T01:35:14.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:14 smithi028 ceph-mon[29390]: pgmap v69: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 896 MiB/s wr, 491 op/s 2024-04-25T01:35:14.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:14 smithi130 ceph-mon[32702]: pgmap v69: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 896 MiB/s wr, 491 op/s 2024-04-25T01:35:14.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:14 smithi032 ceph-mon[32461]: pgmap v69: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 896 MiB/s wr, 491 op/s 2024-04-25T01:35:14.615 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:35:14.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:35:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.875+0000 7f8e825b7640 1 -- 172.21.15.28:0/731901903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e7c11af20 msgr2=0x7f8e7c11d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.875+0000 7f8e825b7640 1 --2- 172.21.15.28:0/731901903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e7c11af20 0x7f8e7c11d310 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8e74009cd0 tx=0x7f8e7402a930 comp rx=0 tx=0).stop 2024-04-25T01:35:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.875+0000 7f8e825b7640 1 -- 172.21.15.28:0/731901903 shutdown_connections 2024-04-25T01:35:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.875+0000 7f8e825b7640 1 --2- 172.21.15.28:0/731901903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e7c11af20 0x7f8e7c11d310 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.875+0000 7f8e825b7640 1 --2- 172.21.15.28:0/731901903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e7c10dfc0 0x7f8e7c10e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.875+0000 7f8e825b7640 1 --2- 172.21.15.28:0/731901903 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e7c10ff90 0x7f8e7c10da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.875+0000 7f8e825b7640 1 -- 172.21.15.28:0/731901903 >> 172.21.15.28:0/731901903 conn(0x7f8e7c06c850 msgr2=0x7f8e7c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.876+0000 7f8e825b7640 1 -- 172.21.15.28:0/731901903 shutdown_connections 2024-04-25T01:35:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.876+0000 7f8e825b7640 1 -- 172.21.15.28:0/731901903 wait complete. 2024-04-25T01:35:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.878+0000 7f8e825b7640 1 Processor -- start 2024-04-25T01:35:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.878+0000 7f8e825b7640 1 -- start start 2024-04-25T01:35:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.878+0000 7f8e825b7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e7c10dfc0 0x7f8e7c078930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.878+0000 7f8e825b7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e7c10ff90 0x7f8e7c078e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.878+0000 7f8e825b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e7c11af20 0x7f8e7c073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.878+0000 7f8e825b7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c1208d0 con 0x7f8e7c11af20 2024-04-25T01:35:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.878+0000 7f8e825b7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c120750 con 0x7f8e7c10dfc0 2024-04-25T01:35:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.878+0000 7f8e825b7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7c120a50 con 0x7f8e7c10ff90 2024-04-25T01:35:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.879+0000 7f8e80b54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e7c11af20 0x7f8e7c073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.879+0000 7f8e80b54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e7c11af20 0x7f8e7c073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55958/0 (socket says 172.21.15.28:55958) 2024-04-25T01:35:14.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.879+0000 7f8e80b54640 1 -- 172.21.15.28:0/1450452841 learned_addr learned my addr 172.21.15.28:0/1450452841 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:14.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.880+0000 7f8e80b54640 1 -- 172.21.15.28:0/1450452841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e7c10ff90 msgr2=0x7f8e7c078e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.880+0000 7f8e80b54640 1 --2- 172.21.15.28:0/1450452841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e7c10ff90 0x7f8e7c078e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.880+0000 7f8e80b54640 1 -- 172.21.15.28:0/1450452841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e7c10dfc0 msgr2=0x7f8e7c078930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.880+0000 7f8e80b54640 1 --2- 172.21.15.28:0/1450452841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e7c10dfc0 0x7f8e7c078930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.880+0000 7f8e80b54640 1 -- 172.21.15.28:0/1450452841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e7c074300 con 0x7f8e7c11af20 2024-04-25T01:35:14.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.881+0000 7f8e80b54640 1 --2- 172.21.15.28:0/1450452841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e7c11af20 0x7f8e7c073a40 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f8e74004af0 tx=0x7f8e74031720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:14.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.881+0000 7f8e797fa640 1 -- 172.21.15.28:0/1450452841 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7402fa50 con 0x7f8e7c11af20 2024-04-25T01:35:14.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.882+0000 7f8e825b7640 1 -- 172.21.15.28:0/1450452841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e7c074590 con 0x7f8e7c11af20 2024-04-25T01:35:14.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.882+0000 7f8e825b7640 1 -- 172.21.15.28:0/1450452841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e7c080830 con 0x7f8e7c11af20 2024-04-25T01:35:14.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.882+0000 7f8e797fa640 1 -- 172.21.15.28:0/1450452841 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e74036030 con 0x7f8e7c11af20 2024-04-25T01:35:14.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.882+0000 7f8e797fa640 1 -- 172.21.15.28:0/1450452841 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7403b400 con 0x7f8e7c11af20 2024-04-25T01:35:14.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.887+0000 7f8e797fa640 1 -- 172.21.15.28:0/1450452841 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8e7403b5a0 con 0x7f8e7c11af20 2024-04-25T01:35:14.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.888+0000 7f8e853a7740 1 Processor -- start 2024-04-25T01:35:14.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.888+0000 7f8e853a7740 1 -- start start 2024-04-25T01:35:14.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.888+0000 7f8e853a7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 0x55c611adfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.888+0000 7f8e853a7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611adff50 0x55c611aea0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.888+0000 7f8e853a7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c611aea620 0x55c611aeca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.888+0000 7f8e853a7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c611a446b0 con 0x55c611adf630 2024-04-25T01:35:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.888+0000 7f8e853a7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c6119ec880 con 0x55c611adff50 2024-04-25T01:35:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.888+0000 7f8e853a7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c611a40670 con 0x55c611aea620 2024-04-25T01:35:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.889+0000 7f8e797fa640 1 --2- 172.21.15.28:0/1450452841 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8e640e6d90 0x7f8e640e9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.890+0000 7f8e797fa640 1 -- 172.21.15.28:0/1450452841 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8e740f2080 con 0x7f8e7c11af20 2024-04-25T01:35:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.890+0000 7f8e7b7fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c611aea620 0x55c611aeca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.890+0000 7f8e7bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611adff50 0x55c611aea0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.890+0000 7f8e7bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611adff50 0x55c611aea0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:53760/0 (socket says 172.21.15.28:53760) 2024-04-25T01:35:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.890+0000 7f8e7bfff640 1 -- 172.21.15.28:0/4190658856 learned_addr learned my addr 172.21.15.28:0/4190658856 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.891+0000 7f8e80b54640 1 --2- 172.21.15.28:0/1450452841 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8e640e6d90 0x7f8e640e9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.891+0000 7f8e7b7fe640 1 --2- 172.21.15.28:0/4190658856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 0x55c611adfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.892+0000 7f8e7bfff640 1 -- 172.21.15.28:0/4190658856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c611aea620 msgr2=0x55c611aeca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.892+0000 7f8e7bfff640 1 --2- 172.21.15.28:0/4190658856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c611aea620 0x55c611aeca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.892+0000 7f8e7bfff640 1 -- 172.21.15.28:0/4190658856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 msgr2=0x55c611adfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.892+0000 7f8e7bfff640 1 --2- 172.21.15.28:0/4190658856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 0x55c611adfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.892+0000 7f8e7bfff640 1 -- 172.21.15.28:0/4190658856 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6118227d0 con 0x55c611adff50 2024-04-25T01:35:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.892+0000 7f8e7b7fe640 1 --2- 172.21.15.28:0/4190658856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 0x55c611adfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.894+0000 7f8e7bfff640 1 --2- 172.21.15.28:0/4190658856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611adff50 0x55c611aea0e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8e7000ba80 tx=0x7f8e7000bf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60df453cc0f9a499 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:14.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.895+0000 7f8e627fc640 1 -- 172.21.15.28:0/4190658856 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e70016070 con 0x55c611adff50 2024-04-25T01:35:14.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.895+0000 7f8e627fc640 1 -- 172.21.15.28:0/4190658856 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e70005500 con 0x55c611adff50 2024-04-25T01:35:14.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.896+0000 7f8e853a7740 1 -- 172.21.15.28:0/4190658856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611adff50 msgr2=0x55c611aea0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.896+0000 7f8e853a7740 1 --2- 172.21.15.28:0/4190658856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611adff50 0x55c611aea0e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8e7000ba80 tx=0x7f8e7000bf40 comp rx=0 tx=0).stop 2024-04-25T01:35:14.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.897+0000 7f8e80b54640 1 --2- 172.21.15.28:0/1450452841 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8e640e6d90 0x7f8e640e9250 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8e740ba770 tx=0x7f8e74005ef0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.899+0000 7f8e853a7740 1 -- 172.21.15.28:0/4190658856 shutdown_connections 2024-04-25T01:35:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.899+0000 7f8e853a7740 1 --2- 172.21.15.28:0/4190658856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c611aea620 0x55c611aeca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.899+0000 7f8e853a7740 1 --2- 172.21.15.28:0/4190658856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611adff50 0x55c611aea0e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.899+0000 7f8e853a7740 1 --2- 172.21.15.28:0/4190658856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 0x55c611adfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.899+0000 7f8e853a7740 1 -- 172.21.15.28:0/4190658856 >> 172.21.15.28:0/4190658856 conn(0x55c611adb950 msgr2=0x55c611adbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.899+0000 7f8e853a7740 1 -- 172.21.15.28:0/4190658856 shutdown_connections 2024-04-25T01:35:14.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.900+0000 7f8e853a7740 1 -- 172.21.15.28:0/4190658856 wait complete. 2024-04-25T01:35:14.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.901+0000 7f8e853a7740 1 -- start start 2024-04-25T01:35:14.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.902+0000 7f8e853a7740 1 Processor -- start 2024-04-25T01:35:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.906+0000 7f8e853a7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 0x55c611aea030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.906+0000 7f8e853a7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c611adff50 0x55c611ae73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.906+0000 7f8e853a7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611aea620 0x55c611ae79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.906+0000 7f8e853a7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c611afe4c0 con 0x55c611adf630 2024-04-25T01:35:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.906+0000 7f8e853a7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c6119fff50 con 0x55c611aea620 2024-04-25T01:35:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.906+0000 7f8e853a7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c611afe640 con 0x55c611adff50 2024-04-25T01:35:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.906+0000 7f8e853a7740 10 client.0 _ll_register_callbacks cb 0x55c611a3b000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.908+0000 7f8e7b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 0x55c611aea030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.908+0000 7f8e7b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 0x55c611aea030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55978/0 (socket says 172.21.15.28:55978) 2024-04-25T01:35:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.908+0000 7f8e7b7fe640 1 -- 172.21.15.28:0/1409721881 learned_addr learned my addr 172.21.15.28:0/1409721881 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.909+0000 7f8e7b7fe640 1 --2- 172.21.15.28:0/1409721881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611aea620 0x55c611ae79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.910+0000 7f8e7b7fe640 1 -- 172.21.15.28:0/1409721881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c611adff50 msgr2=0x55c611ae73c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:35:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.910+0000 7f8e7b7fe640 1 --2- 172.21.15.28:0/1409721881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c611adff50 0x55c611ae73c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.910+0000 7f8e7b7fe640 1 -- 172.21.15.28:0/1409721881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611aea620 msgr2=0x55c611ae79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.910+0000 7f8e7b7fe640 1 --2- 172.21.15.28:0/1409721881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611aea620 0x55c611ae79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.910+0000 7f8e7b7fe640 1 -- 172.21.15.28:0/1409721881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6117583f0 con 0x55c611adf630 2024-04-25T01:35:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.911+0000 7f8e7b7fe640 1 --2- 172.21.15.28:0/1409721881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 0x55c611aea030 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c014ac0 tx=0x7f8e6c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6287dedc027a6c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.911+0000 7f8e5b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.916+0000 7f8e5b7fe640 1 -- 172.21.15.28:0/1409721881 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c010c40 con 0x55c611adf630 2024-04-25T01:35:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.916+0000 7f8e853a7740 1 -- 172.21.15.28:0/1409721881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e70005500 con 0x55c611adf630 2024-04-25T01:35:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.916+0000 7f8e5b7fe640 1 -- 172.21.15.28:0/1409721881 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8e6c010de0 con 0x55c611adf630 2024-04-25T01:35:14.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.921+0000 7f8e5b7fe640 1 -- 172.21.15.28:0/1409721881 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8e6c026050 con 0x55c611adf630 2024-04-25T01:35:14.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.926+0000 7f8e853a7740 10 client.16281 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:14.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.927+0000 7f8e853a7740 1 -- 172.21.15.28:0/1409721881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c6117180f0 con 0x55c611adf630 2024-04-25T01:35:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.937+0000 7f8e5b7fe640 1 -- 172.21.15.28:0/1409721881 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e70005500 con 0x55c611adf630 2024-04-25T01:35:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.937+0000 7f8e5b7fe640 1 -- 172.21.15.28:0/1409721881 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8e6c01f060 con 0x55c611adf630 2024-04-25T01:35:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.939+0000 7f8e853a7740 10 client.16281 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.939+0000 7f8e853a7740 10 client.16281 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.939+0000 7f8e853a7740 20 client.16281 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:14.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.939+0000 7f8e853a7740 1 --2- 172.21.15.28:0/1409721881 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c611b0b9f0 0x55c611b0de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.939+0000 7f8e853a7740 4 client.16281 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:14.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.939+0000 7f8e853a7740 1 -- 172.21.15.28:0/1409721881 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c611a51bf0 con 0x55c611b0b9f0 2024-04-25T01:35:14.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.945+0000 7f8e7bfff640 1 --2- 172.21.15.28:0/1409721881 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c611b0b9f0 0x55c611b0de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.953+0000 7f8e7bfff640 1 --2- 172.21.15.28:0/1409721881 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c611b0b9f0 0x55c611b0de70 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9037c5563a50fe5 server_cookie=36d5690e0e734397 in_seq=0 out_seq=0 2024-04-25T01:35:14.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.956+0000 7f8e5b7fe640 10 client.16281 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:14.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.956+0000 7f8e5b7fe640 1 -- 172.21.15.28:0/1409721881 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c611a51bf0 con 0x55c611b0b9f0 2024-04-25T01:35:14.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.956+0000 7f8e5b7fe640 10 client.16281 handle_command_reply: tid=0 2024-04-25T01:35:14.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.956+0000 7f8e853a7740 1 client.16281 shutdown 2024-04-25T01:35:14.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.958+0000 7f8e853a7740 1 -- 172.21.15.28:0/1409721881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 msgr2=0x55c611aea030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:14.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.958+0000 7f8e853a7740 1 --2- 172.21.15.28:0/1409721881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 0x55c611aea030 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c014ac0 tx=0x7f8e6c005f90 comp rx=0 tx=0).stop 2024-04-25T01:35:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.962+0000 7f8e853a7740 1 -- 172.21.15.28:0/1409721881 shutdown_connections 2024-04-25T01:35:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.962+0000 7f8e853a7740 1 --2- 172.21.15.28:0/1409721881 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c611b0b9f0 0x55c611b0de70 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.962+0000 7f8e853a7740 1 --2- 172.21.15.28:0/1409721881 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611aea620 0x55c611ae79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.962+0000 7f8e853a7740 1 --2- 172.21.15.28:0/1409721881 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c611adff50 0x55c611ae73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.962+0000 7f8e853a7740 1 --2- 172.21.15.28:0/1409721881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 0x55c611aea030 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.962+0000 7f8e853a7740 1 -- 172.21.15.28:0/1409721881 >> 172.21.15.28:0/1409721881 conn(0x55c611adb950 msgr2=0x55c611ae1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:14.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.962+0000 7f8e5b7fe640 0 client.16281 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:14.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.963+0000 7f8e853a7740 1 -- 172.21.15.28:0/1409721881 shutdown_connections 2024-04-25T01:35:14.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.964+0000 7f8e853a7740 1 -- 172.21.15.28:0/1409721881 wait complete. 2024-04-25T01:35:14.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:14.967+0000 7f8e853a7740 20 client.16281 trim_cache size 0 max 16384 2024-04-25T01:35:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.015+0000 7f8e853a7740 1 Processor -- start 2024-04-25T01:35:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.015+0000 7f8e853a7740 1 -- start start 2024-04-25T01:35:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.015+0000 7f8e853a7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 0x55c611ae72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.015+0000 7f8e853a7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c611adff50 0x55c611ae2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.015+0000 7f8e853a7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611aea620 0x55c611ae6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.015+0000 7f8e853a7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e6800fed0 con 0x55c611adf630 2024-04-25T01:35:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.015+0000 7f8e853a7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8e68011a10 con 0x55c611aea620 2024-04-25T01:35:14.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.015+0000 7f8e853a7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8e680105a0 con 0x55c611adff50 2024-04-25T01:35:14.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.016+0000 7f8e7b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 0x55c611ae72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.016+0000 7f8e7b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 0x55c611ae72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55980/0 (socket says 172.21.15.28:55980) 2024-04-25T01:35:14.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.016+0000 7f8e7b7fe640 1 -- 172.21.15.28:0/1260965130 learned_addr learned my addr 172.21.15.28:0/1260965130 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.020+0000 7f8e7bfff640 1 --2- 172.21.15.28:0/1260965130 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c611adff50 0x55c611ae2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.020+0000 7f8e7b7fe640 1 --2- 172.21.15.28:0/1260965130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611aea620 0x55c611ae6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:14.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.022+0000 7f8e7b7fe640 1 -- 172.21.15.28:0/1260965130 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c611adff50 msgr2=0x55c611ae2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:14.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.022+0000 7f8e7b7fe640 1 --2- 172.21.15.28:0/1260965130 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c611adff50 0x55c611ae2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.022+0000 7f8e7b7fe640 1 -- 172.21.15.28:0/1260965130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611aea620 msgr2=0x55c611ae6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:14.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.022+0000 7f8e7b7fe640 1 --2- 172.21.15.28:0/1260965130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611aea620 0x55c611ae6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:14.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.022+0000 7f8e7b7fe640 1 -- 172.21.15.28:0/1260965130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c611adfd80 con 0x55c611adf630 2024-04-25T01:35:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.024+0000 7f8e7b7fe640 1 --2- 172.21.15.28:0/1260965130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 0x55c611ae72c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c000c00 tx=0x7f8e6c0093b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cceccd18e96a76eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.029+0000 7f8e61ffb640 1 -- 172.21.15.28:0/1260965130 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c010e40 con 0x55c611adf630 2024-04-25T01:35:15.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.029+0000 7f8e61ffb640 1 -- 172.21.15.28:0/1260965130 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e6c005860 con 0x55c611adf630 2024-04-25T01:35:15.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.029+0000 7f8e61ffb640 1 -- 172.21.15.28:0/1260965130 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c011660 con 0x55c611adf630 2024-04-25T01:35:15.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.030+0000 7f8e853a7740 1 -- 172.21.15.28:0/1260965130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 msgr2=0x55c611ae72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:15.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.030+0000 7f8e853a7740 1 --2- 172.21.15.28:0/1260965130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 0x55c611ae72c0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c000c00 tx=0x7f8e6c0093b0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.032+0000 7f8e853a7740 1 -- 172.21.15.28:0/1260965130 shutdown_connections 2024-04-25T01:35:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.032+0000 7f8e853a7740 1 --2- 172.21.15.28:0/1260965130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611aea620 0x55c611ae6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.032+0000 7f8e853a7740 1 --2- 172.21.15.28:0/1260965130 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c611adff50 0x55c611ae2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.032+0000 7f8e853a7740 1 --2- 172.21.15.28:0/1260965130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611adf630 0x55c611ae72c0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.032+0000 7f8e853a7740 1 -- 172.21.15.28:0/1260965130 >> 172.21.15.28:0/1260965130 conn(0x55c611adb950 msgr2=0x55c611ada260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:15.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.034+0000 7f8e853a7740 1 -- 172.21.15.28:0/1260965130 shutdown_connections 2024-04-25T01:35:15.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.036+0000 7f8e853a7740 1 -- 172.21.15.28:0/1260965130 wait complete. 2024-04-25T01:35:15.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.036+0000 7f8e853a7740 1 -- start start 2024-04-25T01:35:15.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.037+0000 7f8e853a7740 1 Processor -- start 2024-04-25T01:35:15.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.037+0000 7f8e853a7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611adf630 0x55c611ae3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:15.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.037+0000 7f8e853a7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c611adff50 0x55c611ae3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:15.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.037+0000 7f8e853a7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611aea620 0x55c611ae40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:15.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.037+0000 7f8e853a7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e68011340 con 0x55c611aea620 2024-04-25T01:35:15.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.037+0000 7f8e853a7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8e68010c70 con 0x55c611adf630 2024-04-25T01:35:15.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.037+0000 7f8e853a7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8e6800deb0 con 0x55c611adff50 2024-04-25T01:35:15.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.037+0000 7f8e853a7740 10 client.0 _ll_register_callbacks cb 0x7f8e6c014680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:15.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.040+0000 7f8e7b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611aea620 0x55c611ae40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:15.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.040+0000 7f8e7b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611aea620 0x55c611ae40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55996/0 (socket says 172.21.15.28:55996) 2024-04-25T01:35:15.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.040+0000 7f8e7b7fe640 1 -- 172.21.15.28:0/1234259014 learned_addr learned my addr 172.21.15.28:0/1234259014 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:15.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.040+0000 7f8e7b7fe640 1 --2- 172.21.15.28:0/1234259014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611adf630 0x55c611ae3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:15.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.041+0000 7f8e7b7fe640 1 -- 172.21.15.28:0/1234259014 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c611adff50 msgr2=0x55c611ae3b70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:35:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.041+0000 7f8e7b7fe640 1 --2- 172.21.15.28:0/1234259014 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c611adff50 0x55c611ae3b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.041+0000 7f8e7b7fe640 1 -- 172.21.15.28:0/1234259014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611adf630 msgr2=0x55c611ae3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.041+0000 7f8e7b7fe640 1 --2- 172.21.15.28:0/1234259014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611adf630 0x55c611ae3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.041+0000 7f8e7b7fe640 1 -- 172.21.15.28:0/1234259014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c611b02a70 con 0x55c611aea620 2024-04-25T01:35:15.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.043+0000 7f8e7b7fe640 1 --2- 172.21.15.28:0/1234259014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611aea620 0x55c611ae40b0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c009670 tx=0x7f8e6c017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11d5b3b582bd2d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:15.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.044+0000 7f8e5b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:15.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.045+0000 7f8e5b7fe640 1 -- 172.21.15.28:0/1234259014 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c003c00 con 0x55c611aea620 2024-04-25T01:35:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.045+0000 7f8e5b7fe640 1 -- 172.21.15.28:0/1234259014 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8e6c003fc0 con 0x55c611aea620 2024-04-25T01:35:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.045+0000 7f8e853a7740 1 -- 172.21.15.28:0/1234259014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c611a4bf40 con 0x55c611aea620 2024-04-25T01:35:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.045+0000 7f8e5b7fe640 1 -- 172.21.15.28:0/1234259014 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8e6c0041c0 con 0x55c611aea620 2024-04-25T01:35:15.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.049+0000 7f8e853a7740 10 client.16287 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:15.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.049+0000 7f8e853a7740 1 -- 172.21.15.28:0/1234259014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c611b03d00 con 0x55c611aea620 2024-04-25T01:35:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.057+0000 7f8e5b7fe640 1 -- 172.21.15.28:0/1234259014 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c611a4bf40 con 0x55c611aea620 2024-04-25T01:35:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.057+0000 7f8e5b7fe640 1 -- 172.21.15.28:0/1234259014 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8e6c021430 con 0x55c611aea620 2024-04-25T01:35:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.058+0000 7f8e853a7740 10 client.16287 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.058+0000 7f8e853a7740 10 client.16287 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.058+0000 7f8e853a7740 20 client.16287 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.058+0000 7f8e853a7740 1 --2- 172.21.15.28:0/1234259014 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c611b71fb0 0x55c611b743a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.058+0000 7f8e853a7740 4 client.16287 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:35:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.058+0000 7f8e853a7740 1 -- 172.21.15.28:0/1234259014 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c6119c96f0 con 0x55c611b71fb0 2024-04-25T01:35:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.059+0000 7f8e7bfff640 1 --2- 172.21.15.28:0/1234259014 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c611b71fb0 0x55c611b743a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:15.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.066+0000 7f8e7bfff640 1 --2- 172.21.15.28:0/1234259014 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c611b71fb0 0x55c611b743a0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d41445042d30692f server_cookie=a3dfaa0ad2e773 in_seq=0 out_seq=0 2024-04-25T01:35:15.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.066+0000 7f8e5b7fe640 10 client.16287 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:15.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.066+0000 7f8e5b7fe640 1 -- 172.21.15.28:0/1234259014 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c6119c96f0 con 0x55c611b71fb0 2024-04-25T01:35:15.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.067+0000 7f8e5b7fe640 10 client.16287 handle_command_reply: tid=0 2024-04-25T01:35:15.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.068+0000 7f8e853a7740 1 client.16287 shutdown 2024-04-25T01:35:15.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.074+0000 7f8e853a7740 1 -- 172.21.15.28:0/1234259014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611aea620 msgr2=0x55c611ae40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:15.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.074+0000 7f8e853a7740 1 --2- 172.21.15.28:0/1234259014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611aea620 0x55c611ae40b0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c009670 tx=0x7f8e6c017040 comp rx=0 tx=0).stop 2024-04-25T01:35:15.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.075+0000 7f8e853a7740 1 -- 172.21.15.28:0/1234259014 shutdown_connections 2024-04-25T01:35:15.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.075+0000 7f8e853a7740 1 --2- 172.21.15.28:0/1234259014 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c611b71fb0 0x55c611b743a0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.075+0000 7f8e853a7740 1 --2- 172.21.15.28:0/1234259014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c611aea620 0x55c611ae40b0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.075+0000 7f8e853a7740 1 --2- 172.21.15.28:0/1234259014 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c611adff50 0x55c611ae3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.075+0000 7f8e853a7740 1 --2- 172.21.15.28:0/1234259014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c611adf630 0x55c611ae3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.075+0000 7f8e5b7fe640 0 client.16287 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:15.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.075+0000 7f8e853a7740 1 -- 172.21.15.28:0/1234259014 >> 172.21.15.28:0/1234259014 conn(0x55c611adb950 msgr2=0x55c611adaaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:15.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.075+0000 7f8e853a7740 1 -- 172.21.15.28:0/1234259014 shutdown_connections 2024-04-25T01:35:15.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.075+0000 7f8e853a7740 1 -- 172.21.15.28:0/1234259014 wait complete. 2024-04-25T01:35:15.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.076+0000 7f8e853a7740 20 client.16287 trim_cache size 0 max 16384 2024-04-25T01:35:15.054 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:35:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:35:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:35:15.055 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:35:15.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.080+0000 7f8e825b7640 1 -- 172.21.15.28:0/1450452841 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8e640e6d90 msgr2=0x7f8e640e9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:15.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.080+0000 7f8e825b7640 1 --2- 172.21.15.28:0/1450452841 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8e640e6d90 0x7f8e640e9250 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8e740ba770 tx=0x7f8e74005ef0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.080+0000 7f8e825b7640 1 -- 172.21.15.28:0/1450452841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e7c11af20 msgr2=0x7f8e7c073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:15.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.080+0000 7f8e825b7640 1 --2- 172.21.15.28:0/1450452841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e7c11af20 0x7f8e7c073a40 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f8e74004af0 tx=0x7f8e74031720 comp rx=0 tx=0).stop 2024-04-25T01:35:15.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.080+0000 7f8e825b7640 1 -- 172.21.15.28:0/1450452841 shutdown_connections 2024-04-25T01:35:15.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.080+0000 7f8e825b7640 1 --2- 172.21.15.28:0/1450452841 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8e640e6d90 0x7f8e640e9250 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.080+0000 7f8e825b7640 1 --2- 172.21.15.28:0/1450452841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e7c11af20 0x7f8e7c073a40 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.080+0000 7f8e825b7640 1 --2- 172.21.15.28:0/1450452841 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e7c10ff90 0x7f8e7c078e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.080+0000 7f8e825b7640 1 --2- 172.21.15.28:0/1450452841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e7c10dfc0 0x7f8e7c078930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.080+0000 7f8e825b7640 1 -- 172.21.15.28:0/1450452841 >> 172.21.15.28:0/1450452841 conn(0x7f8e7c06c850 msgr2=0x7f8e7c070230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:15.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.080+0000 7f8e825b7640 1 -- 172.21.15.28:0/1450452841 shutdown_connections 2024-04-25T01:35:15.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.080+0000 7f8e825b7640 1 -- 172.21.15.28:0/1450452841 wait complete. 2024-04-25T01:35:15.093 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:35:15.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:35:15.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.340+0000 7f239d02f640 1 -- 172.21.15.28:0/2940663072 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f239002e330 con 0x7f2398106650 2024-04-25T01:35:15.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.341+0000 7f23a0295640 1 -- 172.21.15.28:0/2940663072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2398106650 msgr2=0x7f2398106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:15.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.341+0000 7f23a0295640 1 --2- 172.21.15.28:0/2940663072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2398106650 0x7f2398106a30 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f2390009ca0 tx=0x7f239002a980 comp rx=0 tx=0).stop 2024-04-25T01:35:15.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.341+0000 7f23a0295640 1 -- 172.21.15.28:0/2940663072 shutdown_connections 2024-04-25T01:35:15.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.341+0000 7f23a0295640 1 --2- 172.21.15.28:0/2940663072 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2398101720 0x7f239810d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.341+0000 7f23a0295640 1 --2- 172.21.15.28:0/2940663072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2398100d60 0x7f23981011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.341+0000 7f23a0295640 1 --2- 172.21.15.28:0/2940663072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2398106650 0x7f2398106a30 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.341+0000 7f23a0295640 1 -- 172.21.15.28:0/2940663072 >> 172.21.15.28:0/2940663072 conn(0x7f2398076be0 msgr2=0x7f23980fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:15.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.341+0000 7f23a0295640 1 -- 172.21.15.28:0/2940663072 shutdown_connections 2024-04-25T01:35:15.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.342+0000 7f23a0295640 1 -- 172.21.15.28:0/2940663072 wait complete. 2024-04-25T01:35:15.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.343+0000 7f23a0295640 1 Processor -- start 2024-04-25T01:35:15.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.346+0000 7f23a0295640 1 -- start start 2024-04-25T01:35:15.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.346+0000 7f23a0295640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2398100d60 0x7f239806dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:15.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.346+0000 7f23a0295640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2398101720 0x7f239806e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:15.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.346+0000 7f23a0295640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2398106650 0x7f23981ac510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:15.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.347+0000 7f23a0295640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f239806b4f0 con 0x7f2398101720 2024-04-25T01:35:15.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.347+0000 7f23a0295640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f239806b370 con 0x7f2398100d60 2024-04-25T01:35:15.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.347+0000 7f23a0295640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f239806b670 con 0x7f2398106650 2024-04-25T01:35:15.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.350+0000 7f239e031640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2398100d60 0x7f239806dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.351+0000 7f239e832640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2398106650 0x7f23981ac510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:15.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.351+0000 7f239d830640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2398101720 0x7f239806e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:15.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.351+0000 7f239d830640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2398101720 0x7f239806e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56024/0 (socket says 172.21.15.28:56024) 2024-04-25T01:35:15.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.351+0000 7f239d830640 1 -- 172.21.15.28:0/874287157 learned_addr learned my addr 172.21.15.28:0/874287157 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:15.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.354+0000 7f239d830640 1 -- 172.21.15.28:0/874287157 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2398106650 msgr2=0x7f23981ac510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:15.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.354+0000 7f239d830640 1 --2- 172.21.15.28:0/874287157 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2398106650 0x7f23981ac510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.354+0000 7f239d830640 1 -- 172.21.15.28:0/874287157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2398100d60 msgr2=0x7f239806dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:15.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.354+0000 7f239d830640 1 --2- 172.21.15.28:0/874287157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2398100d60 0x7f239806dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.354+0000 7f239d830640 1 -- 172.21.15.28:0/874287157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23981acbb0 con 0x7f2398101720 2024-04-25T01:35:15.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.357+0000 7f239d830640 1 --2- 172.21.15.28:0/874287157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2398101720 0x7f239806e2d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f239400b480 tx=0x7f239400b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.358+0000 7f238f7fe640 1 -- 172.21.15.28:0/874287157 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2394014070 con 0x7f2398101720 2024-04-25T01:35:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.358+0000 7f23a0295640 1 -- 172.21.15.28:0/874287157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23981074d0 con 0x7f2398101720 2024-04-25T01:35:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.358+0000 7f23a0295640 1 -- 172.21.15.28:0/874287157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2398107a20 con 0x7f2398101720 2024-04-25T01:35:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.359+0000 7f238f7fe640 1 -- 172.21.15.28:0/874287157 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2394010d50 con 0x7f2398101720 2024-04-25T01:35:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.360+0000 7f238f7fe640 1 -- 172.21.15.28:0/874287157 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2394017360 con 0x7f2398101720 2024-04-25T01:35:15.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.365+0000 7f238f7fe640 1 -- 172.21.15.28:0/874287157 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2394017070 con 0x7f2398101720 2024-04-25T01:35:15.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.366+0000 7f238f7fe640 1 --2- 172.21.15.28:0/874287157 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23840a2f00 0x7f23840a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:15.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.367+0000 7f238f7fe640 1 -- 172.21.15.28:0/874287157 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f23940ceb70 con 0x7f2398101720 2024-04-25T01:35:15.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.367+0000 7f23a0295640 1 -- 172.21.15.28:0/874287157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f237c005290 con 0x7f2398101720 2024-04-25T01:35:15.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.371+0000 7f239e031640 1 --2- 172.21.15.28:0/874287157 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23840a2f00 0x7f23840a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:15.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.383+0000 7f239e031640 1 --2- 172.21.15.28:0/874287157 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23840a2f00 0x7f23840a53c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2390030040 tx=0x7f239002fc70 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:15.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.385+0000 7f238f7fe640 1 -- 172.21.15.28:0/874287157 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f239409b590 con 0x7f2398101720 2024-04-25T01:35:15.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.926+0000 7f23a0295640 1 -- 172.21.15.28:0/874287157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f237c005850 con 0x7f2398101720 2024-04-25T01:35:15.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.930+0000 7f238f7fe640 1 -- 172.21.15.28:0/874287157 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7f239409ae90 con 0x7f2398101720 2024-04-25T01:35:15.908 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:35:15.908 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:35:15.908 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:35:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.937+0000 7f23a0295640 1 -- 172.21.15.28:0/874287157 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23840a2f00 msgr2=0x7f23840a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.937+0000 7f23a0295640 1 --2- 172.21.15.28:0/874287157 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23840a2f00 0x7f23840a53c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2390030040 tx=0x7f239002fc70 comp rx=0 tx=0).stop 2024-04-25T01:35:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.937+0000 7f23a0295640 1 -- 172.21.15.28:0/874287157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2398101720 msgr2=0x7f239806e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.937+0000 7f23a0295640 1 --2- 172.21.15.28:0/874287157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2398101720 0x7f239806e2d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f239400b480 tx=0x7f239400b940 comp rx=0 tx=0).stop 2024-04-25T01:35:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.938+0000 7f23a0295640 1 -- 172.21.15.28:0/874287157 shutdown_connections 2024-04-25T01:35:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.938+0000 7f23a0295640 1 --2- 172.21.15.28:0/874287157 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23840a2f00 0x7f23840a53c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.938+0000 7f23a0295640 1 --2- 172.21.15.28:0/874287157 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2398106650 0x7f23981ac510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.938+0000 7f23a0295640 1 --2- 172.21.15.28:0/874287157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2398101720 0x7f239806e2d0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.938+0000 7f23a0295640 1 --2- 172.21.15.28:0/874287157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2398100d60 0x7f239806dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.938+0000 7f23a0295640 1 -- 172.21.15.28:0/874287157 >> 172.21.15.28:0/874287157 conn(0x7f2398076be0 msgr2=0x7f23980fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:15.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.940+0000 7f23a0295640 1 -- 172.21.15.28:0/874287157 shutdown_connections 2024-04-25T01:35:15.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:15.940+0000 7f23a0295640 1 -- 172.21.15.28:0/874287157 wait complete. 2024-04-25T01:35:15.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:35:16.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.222+0000 7f9c072d2640 1 -- 172.21.15.28:0/1330953286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0010dfc0 msgr2=0x7f9c0010e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:16.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.223+0000 7f9c072d2640 1 --2- 172.21.15.28:0/1330953286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0010dfc0 0x7f9c0010e440 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f9bf001a940 tx=0x7f9bf003b470 comp rx=0 tx=0).stop 2024-04-25T01:35:16.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.224+0000 7f9c072d2640 1 -- 172.21.15.28:0/1330953286 shutdown_connections 2024-04-25T01:35:16.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.224+0000 7f9c072d2640 1 --2- 172.21.15.28:0/1330953286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c0011af20 0x7f9c0011d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.224+0000 7f9c072d2640 1 --2- 172.21.15.28:0/1330953286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0010dfc0 0x7f9c0010e440 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.224+0000 7f9c072d2640 1 --2- 172.21.15.28:0/1330953286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c0010ff90 0x7f9c0010da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.224+0000 7f9c072d2640 1 -- 172.21.15.28:0/1330953286 >> 172.21.15.28:0/1330953286 conn(0x7f9c0006c850 msgr2=0x7f9c0006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:16.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.224+0000 7f9c072d2640 1 -- 172.21.15.28:0/1330953286 shutdown_connections 2024-04-25T01:35:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.224+0000 7f9c072d2640 1 -- 172.21.15.28:0/1330953286 wait complete. 2024-04-25T01:35:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.225+0000 7f9c072d2640 1 Processor -- start 2024-04-25T01:35:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.225+0000 7f9c072d2640 1 -- start start 2024-04-25T01:35:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.226+0000 7f9c072d2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0010dfc0 0x7f9c0011a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.226+0000 7f9c072d2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c0010ff90 0x7f9c00116560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.226+0000 7f9c072d2640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c0011af20 0x7f9c00116aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.226+0000 7f9c072d2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9c001130d0 con 0x7f9c0010dfc0 2024-04-25T01:35:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.226+0000 7f9c072d2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9c00112f50 con 0x7f9c0010ff90 2024-04-25T01:35:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.226+0000 7f9c072d2640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9c00113250 con 0x7f9c0011af20 2024-04-25T01:35:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.226+0000 7f9c0506e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0010dfc0 0x7f9c0011a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.226+0000 7f9c0506e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0010dfc0 0x7f9c0011a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56038/0 (socket says 172.21.15.28:56038) 2024-04-25T01:35:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.226+0000 7f9c0506e640 1 -- 172.21.15.28:0/489094217 learned_addr learned my addr 172.21.15.28:0/489094217 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:16.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.227+0000 7f9c0586f640 1 --2- 172.21.15.28:0/489094217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c0011af20 0x7f9c00116aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.227+0000 7f9c0506e640 1 -- 172.21.15.28:0/489094217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c0011af20 msgr2=0x7f9c00116aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:16.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.227+0000 7f9c0506e640 1 --2- 172.21.15.28:0/489094217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c0011af20 0x7f9c00116aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.227+0000 7f9c0506e640 1 -- 172.21.15.28:0/489094217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c0010ff90 msgr2=0x7f9c00116560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:16.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.227+0000 7f9c0506e640 1 --2- 172.21.15.28:0/489094217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c0010ff90 0x7f9c00116560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.227+0000 7f9c0506e640 1 -- 172.21.15.28:0/489094217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c00116fe0 con 0x7f9c0010dfc0 2024-04-25T01:35:16.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.228+0000 7f9c0506e640 1 --2- 172.21.15.28:0/489094217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0010dfc0 0x7f9c0011a4d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00ea20 tx=0x7f9bfc00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:16.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.230+0000 7f9bf67fc640 1 -- 172.21.15.28:0/489094217 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc00cd20 con 0x7f9c0010dfc0 2024-04-25T01:35:16.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.230+0000 7f9bf67fc640 1 -- 172.21.15.28:0/489094217 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bfc010910 con 0x7f9c0010dfc0 2024-04-25T01:35:16.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.230+0000 7f9bf67fc640 1 -- 172.21.15.28:0/489094217 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc018860 con 0x7f9c0010dfc0 2024-04-25T01:35:16.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.230+0000 7f9c072d2640 1 -- 172.21.15.28:0/489094217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c001172d0 con 0x7f9c0010dfc0 2024-04-25T01:35:16.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.230+0000 7f9c072d2640 1 -- 172.21.15.28:0/489094217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c001ccfe0 con 0x7f9c0010dfc0 2024-04-25T01:35:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.236+0000 7f9bf67fc640 1 -- 172.21.15.28:0/489094217 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9bfc018a00 con 0x7f9c0010dfc0 2024-04-25T01:35:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.238+0000 7f9bf67fc640 1 --2- 172.21.15.28:0/489094217 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9bec0a2ed0 0x7f9bec0a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.238+0000 7f9bf67fc640 1 -- 172.21.15.28:0/489094217 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f9bfc0d0b10 con 0x7f9c0010dfc0 2024-04-25T01:35:16.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.239+0000 7f9c0486d640 1 --2- 172.21.15.28:0/489094217 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9bec0a2ed0 0x7f9bec0a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.242+0000 7f9c0486d640 1 --2- 172.21.15.28:0/489094217 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9bec0a2ed0 0x7f9bec0a5390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9bf001a860 tx=0x7f9bf001a600 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.243+0000 7f9c09fde740 1 Processor -- start 2024-04-25T01:35:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.243+0000 7f9c09fde740 1 -- start start 2024-04-25T01:35:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.243+0000 7f9c09fde740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d59580 0x564783d59960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.243+0000 7f9c09fde740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59ea0 0x564783d64030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.243+0000 7f9c09fde740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d64570 0x564783d66960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.244+0000 7f9c09fde740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564783cbe670 con 0x564783d59580 2024-04-25T01:35:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.244+0000 7f9c09fde740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564783c66970 con 0x564783d64570 2024-04-25T01:35:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.244+0000 7f9c09fde740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x564783cba630 con 0x564783d59ea0 2024-04-25T01:35:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.245+0000 7f9c0586f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d64570 0x564783d66960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.245+0000 7f9c0586f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d64570 0x564783d66960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:53812/0 (socket says 172.21.15.28:53812) 2024-04-25T01:35:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.245+0000 7f9c0586f640 1 -- 172.21.15.28:0/300510027 learned_addr learned my addr 172.21.15.28:0/300510027 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.245+0000 7f9c0486d640 1 --2- 172.21.15.28:0/300510027 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59ea0 0x564783d64030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.246+0000 7f9c0506e640 1 --2- 172.21.15.28:0/300510027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d59580 0x564783d59960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.248+0000 7f9c0586f640 1 -- 172.21.15.28:0/300510027 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59ea0 msgr2=0x564783d64030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.248+0000 7f9c0586f640 1 --2- 172.21.15.28:0/300510027 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59ea0 0x564783d64030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.248+0000 7f9c0586f640 1 -- 172.21.15.28:0/300510027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d59580 msgr2=0x564783d59960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.248+0000 7f9c0586f640 1 --2- 172.21.15.28:0/300510027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d59580 0x564783d59960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.248+0000 7f9c0586f640 1 -- 172.21.15.28:0/300510027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564783c8a400 con 0x564783d64570 2024-04-25T01:35:16.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.252+0000 7f9c0586f640 1 --2- 172.21.15.28:0/300510027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d64570 0x564783d66960 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8009f60 tx=0x7f9bf8009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17e4369b2fe9b59c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:16.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.254+0000 7f9beb7fe640 1 -- 172.21.15.28:0/300510027 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf801a070 con 0x564783d64570 2024-04-25T01:35:16.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.254+0000 7f9beb7fe640 1 -- 172.21.15.28:0/300510027 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bf80158f0 con 0x564783d64570 2024-04-25T01:35:16.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.256+0000 7f9c09fde740 1 -- 172.21.15.28:0/300510027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d64570 msgr2=0x564783d66960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:16.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.256+0000 7f9c09fde740 1 --2- 172.21.15.28:0/300510027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d64570 0x564783d66960 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8009f60 tx=0x7f9bf8009f90 comp rx=0 tx=0).stop 2024-04-25T01:35:16.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.257+0000 7f9c09fde740 1 -- 172.21.15.28:0/300510027 shutdown_connections 2024-04-25T01:35:16.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.257+0000 7f9c09fde740 1 --2- 172.21.15.28:0/300510027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d64570 0x564783d66960 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.257+0000 7f9c09fde740 1 --2- 172.21.15.28:0/300510027 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59ea0 0x564783d64030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.257+0000 7f9c09fde740 1 --2- 172.21.15.28:0/300510027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d59580 0x564783d59960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.257+0000 7f9c09fde740 1 -- 172.21.15.28:0/300510027 >> 172.21.15.28:0/300510027 conn(0x564783d558a0 msgr2=0x564783d55c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:16.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.262+0000 7f9c09fde740 1 -- 172.21.15.28:0/300510027 shutdown_connections 2024-04-25T01:35:16.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.263+0000 7f9c09fde740 1 -- 172.21.15.28:0/300510027 wait complete. 2024-04-25T01:35:16.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.263+0000 7f9c09fde740 1 -- start start 2024-04-25T01:35:16.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.263+0000 7f9c09fde740 1 Processor -- start 2024-04-25T01:35:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.265+0000 7f9c09fde740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59580 0x564783d63e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.265+0000 7f9c09fde740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d59ea0 0x564783d610f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.265+0000 7f9c09fde740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 0x564783d61630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.266+0000 7f9c09fde740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564783d673b0 con 0x564783d64570 2024-04-25T01:35:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.266+0000 7f9c09fde740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564783c7a050 con 0x564783d59ea0 2024-04-25T01:35:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.266+0000 7f9c09fde740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x564783d67530 con 0x564783d59580 2024-04-25T01:35:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.266+0000 7f9c0586f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 0x564783d61630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.266+0000 7f9c0586f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 0x564783d61630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56054/0 (socket says 172.21.15.28:56054) 2024-04-25T01:35:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.266+0000 7f9c0586f640 1 -- 172.21.15.28:0/4190393328 learned_addr learned my addr 172.21.15.28:0/4190393328 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.266+0000 7f9c09fde740 10 client.0 _ll_register_callbacks cb 0x5647839d1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.266+0000 7f9c0506e640 1 --2- 172.21.15.28:0/4190393328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59580 0x564783d63e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.266+0000 7f9c0586f640 1 -- 172.21.15.28:0/4190393328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59580 msgr2=0x564783d63e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.266+0000 7f9c0586f640 1 --2- 172.21.15.28:0/4190393328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59580 0x564783d63e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.267+0000 7f9c0586f640 1 -- 172.21.15.28:0/4190393328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d59ea0 msgr2=0x564783d610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.267+0000 7f9c0586f640 1 --2- 172.21.15.28:0/4190393328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d59ea0 0x564783d610f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.267+0000 7f9c0586f640 1 -- 172.21.15.28:0/4190393328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564783c4bdc0 con 0x564783d64570 2024-04-25T01:35:16.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.267+0000 7f9c0586f640 1 --2- 172.21.15.28:0/4190393328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 0x564783d61630 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8018040 tx=0x7f9bf8015640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=238b8e4b30282a70 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:16.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.267+0000 7f9be8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:16.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.268+0000 7f9be8ff9640 1 -- 172.21.15.28:0/4190393328 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf801a070 con 0x564783d64570 2024-04-25T01:35:16.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.268+0000 7f9be8ff9640 1 -- 172.21.15.28:0/4190393328 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f9bf800acf0 con 0x564783d64570 2024-04-25T01:35:16.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.268+0000 7f9c09fde740 1 -- 172.21.15.28:0/4190393328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bf80158f0 con 0x564783d64570 2024-04-25T01:35:16.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.271+0000 7f9be8ff9640 1 -- 172.21.15.28:0/4190393328 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9bf8015730 con 0x564783d64570 2024-04-25T01:35:16.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.273+0000 7f9c09fde740 10 client.16305 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:16.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.273+0000 7f9c09fde740 1 -- 172.21.15.28:0/4190393328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5647839d23f0 con 0x564783d64570 2024-04-25T01:35:16.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.274+0000 7f9be8ff9640 1 -- 172.21.15.28:0/4190393328 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bf8024070 con 0x564783d64570 2024-04-25T01:35:16.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.274+0000 7f9be8ff9640 1 -- 172.21.15.28:0/4190393328 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9bf80045a0 con 0x564783d64570 2024-04-25T01:35:16.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.274+0000 7f9c09fde740 10 client.16305 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:16.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.274+0000 7f9c09fde740 10 client.16305 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:35:16.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.274+0000 7f9c09fde740 20 client.16305 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:16.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.274+0000 7f9c09fde740 1 --2- 172.21.15.28:0/4190393328 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564783d742c0 0x564783d766b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.274+0000 7f9c09fde740 4 client.16305 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:16.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.274+0000 7f9c09fde740 1 -- 172.21.15.28:0/4190393328 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9bf80158f0 con 0x564783d742c0 2024-04-25T01:35:16.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.278+0000 7f9c0506e640 1 --2- 172.21.15.28:0/4190393328 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564783d742c0 0x564783d766b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.291+0000 7f9c0506e640 1 --2- 172.21.15.28:0/4190393328 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564783d742c0 0x564783d766b0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=337fadbbb8d02478 server_cookie=3868b599e33309ad in_seq=0 out_seq=0 2024-04-25T01:35:16.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.291+0000 7f9be8ff9640 10 client.16305 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:35:16.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.294+0000 7f9be8ff9640 1 -- 172.21.15.28:0/4190393328 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f9bf80158f0 con 0x564783d742c0 2024-04-25T01:35:16.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.294+0000 7f9be8ff9640 10 client.16305 handle_command_reply: tid=0 2024-04-25T01:35:16.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.294+0000 7f9c09fde740 1 client.16305 shutdown 2024-04-25T01:35:16.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.297+0000 7f9c09fde740 1 -- 172.21.15.28:0/4190393328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 msgr2=0x564783d61630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:16.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.297+0000 7f9c09fde740 1 --2- 172.21.15.28:0/4190393328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 0x564783d61630 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8018040 tx=0x7f9bf8015640 comp rx=0 tx=0).stop 2024-04-25T01:35:16.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.298+0000 7f9c09fde740 1 -- 172.21.15.28:0/4190393328 shutdown_connections 2024-04-25T01:35:16.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.298+0000 7f9c09fde740 1 --2- 172.21.15.28:0/4190393328 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564783d742c0 0x564783d766b0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.299+0000 7f9c09fde740 1 --2- 172.21.15.28:0/4190393328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 0x564783d61630 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.299+0000 7f9c09fde740 1 --2- 172.21.15.28:0/4190393328 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d59ea0 0x564783d610f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.299+0000 7f9c09fde740 1 --2- 172.21.15.28:0/4190393328 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59580 0x564783d63e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.299+0000 7f9c09fde740 1 -- 172.21.15.28:0/4190393328 >> 172.21.15.28:0/4190393328 conn(0x564783d558a0 msgr2=0x564783d54b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:16.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.299+0000 7f9be8ff9640 0 client.16305 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:35:16.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.299+0000 7f9c09fde740 1 -- 172.21.15.28:0/4190393328 shutdown_connections 2024-04-25T01:35:16.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.299+0000 7f9c09fde740 1 -- 172.21.15.28:0/4190393328 wait complete. 2024-04-25T01:35:16.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.299+0000 7f9c09fde740 20 client.16305 trim_cache size 0 max 16384 2024-04-25T01:35:16.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.333+0000 7f9c09fde740 1 Processor -- start 2024-04-25T01:35:16.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:16 smithi028 ceph-mon[29390]: pgmap v70: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 549 MiB/s wr, 213 op/s 2024-04-25T01:35:16.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:16 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/874287157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:35:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.334+0000 7f9c09fde740 1 -- start start 2024-04-25T01:35:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.335+0000 7f9c09fde740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59580 0x564783d5b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.335+0000 7f9c09fde740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d59ea0 0x564783d70350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.335+0000 7f9c09fde740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 0x564783d63930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.335+0000 7f9c09fde740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9be400fed0 con 0x564783d64570 2024-04-25T01:35:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.335+0000 7f9c09fde740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9be4011a10 con 0x564783d59ea0 2024-04-25T01:35:16.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.335+0000 7f9c09fde740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9be40105a0 con 0x564783d59580 2024-04-25T01:35:16.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.336+0000 7f9c0486d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d59ea0 0x564783d70350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.336+0000 7f9c0486d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d59ea0 0x564783d70350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:53826/0 (socket says 172.21.15.28:53826) 2024-04-25T01:35:16.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.336+0000 7f9c0486d640 1 -- 172.21.15.28:0/2467207551 learned_addr learned my addr 172.21.15.28:0/2467207551 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:16.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.336+0000 7f9c0586f640 1 --2- 172.21.15.28:0/2467207551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 0x564783d63930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.336+0000 7f9c0506e640 1 --2- 172.21.15.28:0/2467207551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59580 0x564783d5b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.337+0000 7f9c0586f640 1 -- 172.21.15.28:0/2467207551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59580 msgr2=0x564783d5b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.337+0000 7f9c0586f640 1 --2- 172.21.15.28:0/2467207551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59580 0x564783d5b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.337+0000 7f9c0586f640 1 -- 172.21.15.28:0/2467207551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d59ea0 msgr2=0x564783d70350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:16.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.337+0000 7f9c0586f640 1 --2- 172.21.15.28:0/2467207551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d59ea0 0x564783d70350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.337+0000 7f9c0586f640 1 -- 172.21.15.28:0/2467207551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564783d59cd0 con 0x564783d64570 2024-04-25T01:35:16.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.343+0000 7f9c0586f640 1 --2- 172.21.15.28:0/2467207551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 0x564783d63930 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8004b10 tx=0x7f9bf80096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c451ca489b54a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:16.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.347+0000 7f9beaffd640 1 -- 172.21.15.28:0/2467207551 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf801a070 con 0x564783d64570 2024-04-25T01:35:16.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.347+0000 7f9beaffd640 1 -- 172.21.15.28:0/2467207551 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bf8024040 con 0x564783d64570 2024-04-25T01:35:16.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.347+0000 7f9beaffd640 1 -- 172.21.15.28:0/2467207551 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf801f900 con 0x564783d64570 2024-04-25T01:35:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.349+0000 7f9c09fde740 1 -- 172.21.15.28:0/2467207551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 msgr2=0x564783d63930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.349+0000 7f9c09fde740 1 --2- 172.21.15.28:0/2467207551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 0x564783d63930 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8004b10 tx=0x7f9bf80096d0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.354+0000 7f9c09fde740 1 -- 172.21.15.28:0/2467207551 shutdown_connections 2024-04-25T01:35:16.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.354+0000 7f9c09fde740 1 --2- 172.21.15.28:0/2467207551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 0x564783d63930 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.354+0000 7f9c09fde740 1 --2- 172.21.15.28:0/2467207551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d59ea0 0x564783d70350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.354+0000 7f9c09fde740 1 --2- 172.21.15.28:0/2467207551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59580 0x564783d5b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.354+0000 7f9c09fde740 1 -- 172.21.15.28:0/2467207551 >> 172.21.15.28:0/2467207551 conn(0x564783d558a0 msgr2=0x564783d5c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:16.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.357+0000 7f9c09fde740 1 -- 172.21.15.28:0/2467207551 shutdown_connections 2024-04-25T01:35:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.357+0000 7f9c09fde740 1 -- 172.21.15.28:0/2467207551 wait complete. 2024-04-25T01:35:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.357+0000 7f9c09fde740 1 -- start start 2024-04-25T01:35:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.357+0000 7f9c09fde740 1 Processor -- start 2024-04-25T01:35:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.358+0000 7f9c09fde740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d59580 0x564783d6cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.358+0000 7f9c09fde740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59ea0 0x564783d67ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.358+0000 7f9c09fde740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 0x564783d683e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.358+0000 7f9c09fde740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9be4011340 con 0x564783d64570 2024-04-25T01:35:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.358+0000 7f9c09fde740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9be4010c70 con 0x564783d59580 2024-04-25T01:35:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.358+0000 7f9c09fde740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9be400deb0 con 0x564783d59ea0 2024-04-25T01:35:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.358+0000 7f9c09fde740 10 client.0 _ll_register_callbacks cb 0x7f9bf8018070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.359+0000 7f9c0586f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 0x564783d683e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.359+0000 7f9c0586f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 0x564783d683e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56078/0 (socket says 172.21.15.28:56078) 2024-04-25T01:35:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.359+0000 7f9c0586f640 1 -- 172.21.15.28:0/919359248 learned_addr learned my addr 172.21.15.28:0/919359248 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.359+0000 7f9c0486d640 1 --2- 172.21.15.28:0/919359248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59ea0 0x564783d67ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.360+0000 7f9c0586f640 1 -- 172.21.15.28:0/919359248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59ea0 msgr2=0x564783d67ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.360+0000 7f9c0586f640 1 --2- 172.21.15.28:0/919359248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59ea0 0x564783d67ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.360+0000 7f9c0586f640 1 -- 172.21.15.28:0/919359248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d59580 msgr2=0x564783d6cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.360+0000 7f9c0586f640 1 --2- 172.21.15.28:0/919359248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d59580 0x564783d6cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.360+0000 7f9c0586f640 1 -- 172.21.15.28:0/919359248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564783a9c7d0 con 0x564783d64570 2024-04-25T01:35:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.360+0000 7f9c0586f640 1 --2- 172.21.15.28:0/919359248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 0x564783d683e0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8000f80 tx=0x7f9bf8009cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f6848d271715b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.360+0000 7f9be8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:16.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.361+0000 7f9be8ff9640 1 -- 172.21.15.28:0/919359248 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf801a040 con 0x564783d64570 2024-04-25T01:35:16.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.361+0000 7f9be8ff9640 1 -- 172.21.15.28:0/919359248 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f9bf8029b40 con 0x564783d64570 2024-04-25T01:35:16.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.362+0000 7f9c09fde740 1 -- 172.21.15.28:0/919359248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bf8024040 con 0x564783d64570 2024-04-25T01:35:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.363+0000 7f9be8ff9640 1 -- 172.21.15.28:0/919359248 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9bf8029d40 con 0x564783d64570 2024-04-25T01:35:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.364+0000 7f9c09fde740 10 client.16311 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.364+0000 7f9c09fde740 1 -- 172.21.15.28:0/919359248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564783dda3c0 con 0x564783d64570 2024-04-25T01:35:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.365+0000 7f9be8ff9640 1 -- 172.21.15.28:0/919359248 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bf8024040 con 0x564783d64570 2024-04-25T01:35:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.367+0000 7f9be8ff9640 1 -- 172.21.15.28:0/919359248 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9bf8031070 con 0x564783d64570 2024-04-25T01:35:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.367+0000 7f9c09fde740 10 client.16311 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.367+0000 7f9c09fde740 10 client.16311 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:35:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.368+0000 7f9c09fde740 20 client.16311 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.368+0000 7f9c09fde740 1 --2- 172.21.15.28:0/919359248 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564783dda920 0x564783ddcd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.368+0000 7f9c09fde740 4 client.16311 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:35:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.368+0000 7f9c09fde740 1 -- 172.21.15.28:0/919359248 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564783ccbbb0 con 0x564783dda920 2024-04-25T01:35:16.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.372+0000 7f9c0506e640 1 --2- 172.21.15.28:0/919359248 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564783dda920 0x564783ddcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.391+0000 7f9c0506e640 1 --2- 172.21.15.28:0/919359248 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564783dda920 0x564783ddcd30 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a0d0f5ae6a21c071 server_cookie=87328e72f3845c77 in_seq=0 out_seq=0 2024-04-25T01:35:16.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.393+0000 7f9be8ff9640 10 client.16311 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:35:16.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.393+0000 7f9be8ff9640 1 -- 172.21.15.28:0/919359248 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564783ccbbb0 con 0x564783dda920 2024-04-25T01:35:16.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.393+0000 7f9be8ff9640 10 client.16311 handle_command_reply: tid=0 2024-04-25T01:35:16.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.393+0000 7f9c09fde740 1 client.16311 shutdown 2024-04-25T01:35:16.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.399+0000 7f9c09fde740 1 -- 172.21.15.28:0/919359248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 msgr2=0x564783d683e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:16.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.399+0000 7f9c09fde740 1 --2- 172.21.15.28:0/919359248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 0x564783d683e0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8000f80 tx=0x7f9bf8009cc0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.402+0000 7f9c09fde740 1 -- 172.21.15.28:0/919359248 shutdown_connections 2024-04-25T01:35:16.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.403+0000 7f9c09fde740 1 --2- 172.21.15.28:0/919359248 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564783dda920 0x564783ddcd30 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.403+0000 7f9c09fde740 1 --2- 172.21.15.28:0/919359248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564783d64570 0x564783d683e0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.403+0000 7f9c09fde740 1 --2- 172.21.15.28:0/919359248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564783d59ea0 0x564783d67ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.403+0000 7f9c09fde740 1 --2- 172.21.15.28:0/919359248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564783d59580 0x564783d6cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.403+0000 7f9c09fde740 1 -- 172.21.15.28:0/919359248 >> 172.21.15.28:0/919359248 conn(0x564783d558a0 msgr2=0x564783d60920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:16.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.405+0000 7f9be8ff9640 0 client.16311 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:35:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.411+0000 7f9c09fde740 1 -- 172.21.15.28:0/919359248 shutdown_connections 2024-04-25T01:35:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.411+0000 7f9c09fde740 1 -- 172.21.15.28:0/919359248 wait complete. 2024-04-25T01:35:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.412+0000 7f9c09fde740 20 client.16311 trim_cache size 0 max 16384 2024-04-25T01:35:16.391 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:35:16.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.416+0000 7f9c072d2640 1 -- 172.21.15.28:0/489094217 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9bec0a2ed0 msgr2=0x7f9bec0a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:16.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.417+0000 7f9c072d2640 1 --2- 172.21.15.28:0/489094217 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9bec0a2ed0 0x7f9bec0a5390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9bf001a860 tx=0x7f9bf001a600 comp rx=0 tx=0).stop 2024-04-25T01:35:16.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.417+0000 7f9c072d2640 1 -- 172.21.15.28:0/489094217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0010dfc0 msgr2=0x7f9c0011a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:16.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.417+0000 7f9c072d2640 1 --2- 172.21.15.28:0/489094217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0010dfc0 0x7f9c0011a4d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00ea20 tx=0x7f9bfc00eee0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.418+0000 7f9c072d2640 1 -- 172.21.15.28:0/489094217 shutdown_connections 2024-04-25T01:35:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.418+0000 7f9c072d2640 1 --2- 172.21.15.28:0/489094217 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9bec0a2ed0 0x7f9bec0a5390 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.418+0000 7f9c072d2640 1 --2- 172.21.15.28:0/489094217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9c0011af20 0x7f9c00116aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.418+0000 7f9c072d2640 1 --2- 172.21.15.28:0/489094217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9c0010ff90 0x7f9c00116560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.419+0000 7f9c072d2640 1 --2- 172.21.15.28:0/489094217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c0010dfc0 0x7f9c0011a4d0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.419+0000 7f9c072d2640 1 -- 172.21.15.28:0/489094217 >> 172.21.15.28:0/489094217 conn(0x7f9c0006c850 msgr2=0x7f9c0011c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.421+0000 7f9c072d2640 1 -- 172.21.15.28:0/489094217 shutdown_connections 2024-04-25T01:35:16.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.421+0000 7f9c072d2640 1 -- 172.21.15.28:0/489094217 wait complete. 2024-04-25T01:35:16.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:35:16.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:16 smithi130 ceph-mon[32702]: pgmap v70: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 549 MiB/s wr, 213 op/s 2024-04-25T01:35:16.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:16 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/874287157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:35:16.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:16 smithi032 ceph-mon[32461]: pgmap v70: 129 pgs: 129 active+clean; 11 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 549 MiB/s wr, 213 op/s 2024-04-25T01:35:16.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:16 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/874287157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:35:16.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.720+0000 7fd482777640 1 -- 172.21.15.28:0/990611927 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd47c102150 msgr2=0x7fd47c10e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:16.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.720+0000 7fd482777640 1 --2- 172.21.15.28:0/990611927 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd47c102150 0x7fd47c10e660 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd474009cd0 tx=0x7fd474032e10 comp rx=0 tx=0).stop 2024-04-25T01:35:16.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.725+0000 7fd482777640 1 -- 172.21.15.28:0/990611927 shutdown_connections 2024-04-25T01:35:16.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.725+0000 7fd482777640 1 --2- 172.21.15.28:0/990611927 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd47c102150 0x7fd47c10e660 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.725+0000 7fd482777640 1 --2- 172.21.15.28:0/990611927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c101790 0x7fd47c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.725+0000 7fd482777640 1 --2- 172.21.15.28:0/990611927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd47c1077a0 0x7fd47c107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.725+0000 7fd482777640 1 -- 172.21.15.28:0/990611927 >> 172.21.15.28:0/990611927 conn(0x7fd47c076250 msgr2=0x7fd47c0ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:16.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.729+0000 7fd482777640 1 -- 172.21.15.28:0/990611927 shutdown_connections 2024-04-25T01:35:16.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.733+0000 7fd482777640 1 -- 172.21.15.28:0/990611927 wait complete. 2024-04-25T01:35:16.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.738+0000 7fd482777640 1 Processor -- start 2024-04-25T01:35:16.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.744+0000 7fd482777640 1 -- start start 2024-04-25T01:35:16.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.745+0000 7fd482777640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c101790 0x7fd47c072cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.745+0000 7fd482777640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd47c102150 0x7fd47c073200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.745+0000 7fd482777640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd47c1077a0 0x7fd47c06cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.745+0000 7fd482777640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c06b750 con 0x7fd47c101790 2024-04-25T01:35:16.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.745+0000 7fd482777640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c06b5d0 con 0x7fd47c1077a0 2024-04-25T01:35:16.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.745+0000 7fd482777640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c06b8d0 con 0x7fd47c102150 2024-04-25T01:35:16.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.746+0000 7fd47bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c101790 0x7fd47c072cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.746+0000 7fd47bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c101790 0x7fd47c072cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47440/0 (socket says 172.21.15.28:47440) 2024-04-25T01:35:16.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.746+0000 7fd47bfff640 1 -- 172.21.15.28:0/3731136462 learned_addr learned my addr 172.21.15.28:0/3731136462 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:16.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.747+0000 7fd47b7fe640 1 --2- 172.21.15.28:0/3731136462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd47c102150 0x7fd47c073200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.747+0000 7fd47bfff640 1 -- 172.21.15.28:0/3731136462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd47c102150 msgr2=0x7fd47c073200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:16.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.747+0000 7fd47bfff640 1 --2- 172.21.15.28:0/3731136462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd47c102150 0x7fd47c073200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.747+0000 7fd47bfff640 1 -- 172.21.15.28:0/3731136462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd47c1077a0 msgr2=0x7fd47c06cd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:35:16.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.747+0000 7fd47bfff640 1 --2- 172.21.15.28:0/3731136462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd47c1077a0 0x7fd47c06cd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.747+0000 7fd47bfff640 1 -- 172.21.15.28:0/3731136462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd47c06d640 con 0x7fd47c101790 2024-04-25T01:35:16.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.748+0000 7fd47bfff640 1 --2- 172.21.15.28:0/3731136462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c101790 0x7fd47c072cc0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd47000b450 tx=0x7fd47000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.753+0000 7fd4797fa640 1 -- 172.21.15.28:0/3731136462 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd470015020 con 0x7fd47c101790 2024-04-25T01:35:16.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.754+0000 7fd4797fa640 1 -- 172.21.15.28:0/3731136462 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd470011d70 con 0x7fd47c101790 2024-04-25T01:35:16.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.754+0000 7fd4797fa640 1 -- 172.21.15.28:0/3731136462 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47000f340 con 0x7fd47c101790 2024-04-25T01:35:16.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.755+0000 7fd482777640 1 -- 172.21.15.28:0/3731136462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd47c06d930 con 0x7fd47c101790 2024-04-25T01:35:16.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.755+0000 7fd482777640 1 -- 172.21.15.28:0/3731136462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd47c1bc370 con 0x7fd47c101790 2024-04-25T01:35:16.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.773+0000 7fd485567740 1 Processor -- start 2024-04-25T01:35:16.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.778+0000 7fd4797fa640 1 -- 172.21.15.28:0/3731136462 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd470018020 con 0x7fd47c101790 2024-04-25T01:35:16.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.779+0000 7fd4797fa640 1 --2- 172.21.15.28:0/3731136462 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd4640a2f10 0x7fd4640a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.780+0000 7fd485567740 1 -- start start 2024-04-25T01:35:16.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.780+0000 7fd4797fa640 1 -- 172.21.15.28:0/3731136462 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fd4700cf660 con 0x7fd47c101790 2024-04-25T01:35:16.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.780+0000 7fd485567740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594fcb82580 0x5594fcb82960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.780+0000 7fd485567740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82ea0 0x5594fcb8d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.780+0000 7fd485567740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594fcb8d570 0x5594fcb8f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.780+0000 7fd485567740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5594fcae7670 con 0x5594fcb82ea0 2024-04-25T01:35:16.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.780+0000 7fd485567740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5594fca8f970 con 0x5594fcb82580 2024-04-25T01:35:16.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.780+0000 7fd485567740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5594fcae3630 con 0x5594fcb8d570 2024-04-25T01:35:16.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.782+0000 7fd47b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82ea0 0x5594fcb8d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.782+0000 7fd47b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82ea0 0x5594fcb8d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47444/0 (socket says 172.21.15.28:47444) 2024-04-25T01:35:16.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.782+0000 7fd47b7fe640 1 -- 172.21.15.28:0/62245507 learned_addr learned my addr 172.21.15.28:0/62245507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:16.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.783+0000 7fd47b7fe640 1 -- 172.21.15.28:0/62245507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594fcb8d570 msgr2=0x5594fcb8f960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:35:16.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.783+0000 7fd47b7fe640 1 --2- 172.21.15.28:0/62245507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594fcb8d570 0x5594fcb8f960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.783+0000 7fd47b7fe640 1 -- 172.21.15.28:0/62245507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594fcb82580 msgr2=0x5594fcb82960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:35:16.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.783+0000 7fd47b7fe640 1 --2- 172.21.15.28:0/62245507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594fcb82580 0x5594fcb82960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.783+0000 7fd47b7fe640 1 -- 172.21.15.28:0/62245507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594fcab3400 con 0x5594fcb82ea0 2024-04-25T01:35:16.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.783+0000 7fd47b7fe640 1 --2- 172.21.15.28:0/62245507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82ea0 0x5594fcb8d030 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd46c009bf0 tx=0x7fd46c010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6de7017afa8a1788 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:16.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.783+0000 7fd480d14640 1 --2- 172.21.15.28:0/3731136462 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd4640a2f10 0x7fd4640a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.786+0000 7fd4627fc640 1 -- 172.21.15.28:0/62245507 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd46c004020 con 0x5594fcb82ea0 2024-04-25T01:35:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.786+0000 7fd4627fc640 1 -- 172.21.15.28:0/62245507 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd46c0041c0 con 0x5594fcb82ea0 2024-04-25T01:35:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.786+0000 7fd4627fc640 1 -- 172.21.15.28:0/62245507 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd46c0126b0 con 0x5594fcb82ea0 2024-04-25T01:35:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.787+0000 7fd485567740 1 -- 172.21.15.28:0/62245507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82ea0 msgr2=0x5594fcb8d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.787+0000 7fd485567740 1 --2- 172.21.15.28:0/62245507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82ea0 0x5594fcb8d030 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd46c009bf0 tx=0x7fd46c010740 comp rx=0 tx=0).stop 2024-04-25T01:35:16.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.792+0000 7fd485567740 1 -- 172.21.15.28:0/62245507 shutdown_connections 2024-04-25T01:35:16.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.792+0000 7fd485567740 1 --2- 172.21.15.28:0/62245507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594fcb8d570 0x5594fcb8f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.792+0000 7fd485567740 1 --2- 172.21.15.28:0/62245507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82ea0 0x5594fcb8d030 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.792+0000 7fd485567740 1 --2- 172.21.15.28:0/62245507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594fcb82580 0x5594fcb82960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.792+0000 7fd485567740 1 -- 172.21.15.28:0/62245507 >> 172.21.15.28:0/62245507 conn(0x5594fcb7e8a0 msgr2=0x5594fcb7ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:16.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.795+0000 7fd480d14640 1 --2- 172.21.15.28:0/3731136462 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd4640a2f10 0x7fd4640a53d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd474009a60 tx=0x7fd474039360 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.797+0000 7fd485567740 1 -- 172.21.15.28:0/62245507 shutdown_connections 2024-04-25T01:35:16.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.805+0000 7fd485567740 1 -- 172.21.15.28:0/62245507 wait complete. 2024-04-25T01:35:16.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.806+0000 7fd485567740 1 -- start start 2024-04-25T01:35:16.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.806+0000 7fd485567740 1 Processor -- start 2024-04-25T01:35:16.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.807+0000 7fd485567740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594fcb82580 0x5594fcb8cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.807+0000 7fd485567740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82ea0 0x5594fcb8a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.807+0000 7fd485567740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594fcb8d570 0x5594fcb8a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.807+0000 7fd485567740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5594fcb903b0 con 0x5594fcb82ea0 2024-04-25T01:35:16.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.807+0000 7fd485567740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5594fcaa3050 con 0x5594fcb82580 2024-04-25T01:35:16.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.807+0000 7fd485567740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5594fcb90530 con 0x5594fcb8d570 2024-04-25T01:35:16.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.807+0000 7fd485567740 10 client.0 _ll_register_callbacks cb 0x5594fc7fabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:16.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.810+0000 7fd480d14640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594fcb8d570 0x5594fcb8a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.810+0000 7fd47bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594fcb82580 0x5594fcb8cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.811+0000 7fd47b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82ea0 0x5594fcb8a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.811+0000 7fd47b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82ea0 0x5594fcb8a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47456/0 (socket says 172.21.15.28:47456) 2024-04-25T01:35:16.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.811+0000 7fd47b7fe640 1 -- 172.21.15.28:0/3326578891 learned_addr learned my addr 172.21.15.28:0/3326578891 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:16.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.815+0000 7fd47bfff640 1 -- 172.21.15.28:0/3326578891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594fcb8d570 msgr2=0x5594fcb8a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:16.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.815+0000 7fd47bfff640 1 --2- 172.21.15.28:0/3326578891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594fcb8d570 0x5594fcb8a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.815+0000 7fd47bfff640 1 -- 172.21.15.28:0/3326578891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82ea0 msgr2=0x5594fcb8a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:16.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.815+0000 7fd47bfff640 1 --2- 172.21.15.28:0/3326578891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82ea0 0x5594fcb8a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.815+0000 7fd47bfff640 1 -- 172.21.15.28:0/3326578891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594fca74dc0 con 0x5594fcb82580 2024-04-25T01:35:16.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.817+0000 7fd47bfff640 1 --2- 172.21.15.28:0/3326578891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594fcb82580 0x5594fcb8cfc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd4700ccb20 tx=0x7fd47009bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89f8ca92b195deea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:16.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.817+0000 7fd45b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-25T01:35:16.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.819+0000 7fd45b7fe640 1 -- 172.21.15.28:0/3326578891 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4700a4560 con 0x5594fcb82580 2024-04-25T01:35:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.819+0000 7fd485567740 1 -- 172.21.15.28:0/3326578891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd46c0041c0 con 0x5594fcb82580 2024-04-25T01:35:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.819+0000 7fd45b7fe640 1 -- 172.21.15.28:0/3326578891 <== mon.2 v2:172.21.15.32:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fd4700cf270 con 0x5594fcb82580 2024-04-25T01:35:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.819+0000 7fd45b7fe640 1 -- 172.21.15.28:0/3326578891 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd4700ccc90 con 0x5594fcb82580 2024-04-25T01:35:16.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.820+0000 7fd485567740 10 client.25775 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:16.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.821+0000 7fd485567740 1 -- 172.21.15.28:0/3326578891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5594fc7fb3f0 con 0x5594fcb82580 2024-04-25T01:35:16.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.823+0000 7fd45b7fe640 1 -- 172.21.15.28:0/3326578891 <== mon.2 v2:172.21.15.32:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd46c0041c0 con 0x5594fcb82580 2024-04-25T01:35:16.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.823+0000 7fd45b7fe640 1 -- 172.21.15.28:0/3326578891 <== mon.2 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd47009ad40 con 0x5594fcb82580 2024-04-25T01:35:16.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.824+0000 7fd485567740 10 client.25775 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:16.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.824+0000 7fd485567740 10 client.25775 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:35:16.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.824+0000 7fd485567740 20 client.25775 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:16.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.824+0000 7fd485567740 1 --2- 172.21.15.28:0/3326578891 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5594fcb9d720 0x5594fcb9fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.824+0000 7fd485567740 4 client.25775 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:16.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.824+0000 7fd485567740 1 -- 172.21.15.28:0/3326578891 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594fcaf4bb0 con 0x5594fcb9d720 2024-04-25T01:35:16.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.828+0000 7fd47b7fe640 1 --2- 172.21.15.28:0/3326578891 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5594fcb9d720 0x5594fcb9fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.839+0000 7fd47b7fe640 1 --2- 172.21.15.28:0/3326578891 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5594fcb9d720 0x5594fcb9fb10 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2ebf740b7bba2113 server_cookie=c02a1b5098296c5f in_seq=0 out_seq=0 2024-04-25T01:35:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.840+0000 7fd45b7fe640 10 client.25775 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:35:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.841+0000 7fd45b7fe640 1 -- 172.21.15.28:0/3326578891 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5594fcaf4bb0 con 0x5594fcb9d720 2024-04-25T01:35:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.842+0000 7fd45b7fe640 10 client.25775 handle_command_reply: tid=0 2024-04-25T01:35:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.842+0000 7fd485567740 1 client.25775 shutdown 2024-04-25T01:35:16.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.845+0000 7fd485567740 1 -- 172.21.15.28:0/3326578891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594fcb82580 msgr2=0x5594fcb8cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:16.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.845+0000 7fd485567740 1 --2- 172.21.15.28:0/3326578891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594fcb82580 0x5594fcb8cfc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd4700ccb20 tx=0x7fd47009bff0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.847+0000 7fd485567740 1 -- 172.21.15.28:0/3326578891 shutdown_connections 2024-04-25T01:35:16.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.848+0000 7fd485567740 1 --2- 172.21.15.28:0/3326578891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594fcb8d570 0x5594fcb8a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.848+0000 7fd485567740 1 --2- 172.21.15.28:0/3326578891 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5594fcb9d720 0x5594fcb9fb10 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.848+0000 7fd485567740 1 --2- 172.21.15.28:0/3326578891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82ea0 0x5594fcb8a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.848+0000 7fd485567740 1 --2- 172.21.15.28:0/3326578891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594fcb82580 0x5594fcb8cfc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.848+0000 7fd485567740 1 -- 172.21.15.28:0/3326578891 >> 172.21.15.28:0/3326578891 conn(0x5594fcb7e8a0 msgr2=0x5594fcb7db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:16.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.848+0000 7fd45b7fe640 0 client.25775 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:35:16.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.850+0000 7fd485567740 1 -- 172.21.15.28:0/3326578891 shutdown_connections 2024-04-25T01:35:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.850+0000 7fd485567740 1 -- 172.21.15.28:0/3326578891 wait complete. 2024-04-25T01:35:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.851+0000 7fd485567740 20 client.25775 trim_cache size 0 max 16384 2024-04-25T01:35:16.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.886+0000 7fd485567740 1 Processor -- start 2024-04-25T01:35:16.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.886+0000 7fd485567740 1 -- start start 2024-04-25T01:35:16.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.886+0000 7fd485567740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594fcb82580 0x5594fcb992a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.886+0000 7fd485567740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594fcb82ea0 0x5594fcb8caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.886+0000 7fd485567740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb8d570 0x5594fcb8d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.886+0000 7fd485567740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c1c6140 con 0x5594fcb8d570 2024-04-25T01:35:16.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.886+0000 7fd485567740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c1c7c80 con 0x5594fcb82ea0 2024-04-25T01:35:16.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.886+0000 7fd485567740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c1c6810 con 0x5594fcb82580 2024-04-25T01:35:16.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.888+0000 7fd480d14640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb8d570 0x5594fcb8d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.888+0000 7fd480d14640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb8d570 0x5594fcb8d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47458/0 (socket says 172.21.15.28:47458) 2024-04-25T01:35:16.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.888+0000 7fd480d14640 1 -- 172.21.15.28:0/1543675740 learned_addr learned my addr 172.21.15.28:0/1543675740 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:16.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.889+0000 7fd47bfff640 1 --2- 172.21.15.28:0/1543675740 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594fcb82580 0x5594fcb992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.889+0000 7fd480d14640 1 -- 172.21.15.28:0/1543675740 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594fcb82580 msgr2=0x5594fcb992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:16.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.889+0000 7fd480d14640 1 --2- 172.21.15.28:0/1543675740 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594fcb82580 0x5594fcb992a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.889+0000 7fd480d14640 1 -- 172.21.15.28:0/1543675740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594fcb82ea0 msgr2=0x5594fcb8caf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:35:16.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.889+0000 7fd480d14640 1 --2- 172.21.15.28:0/1543675740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594fcb82ea0 0x5594fcb8caf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.889+0000 7fd480d14640 1 -- 172.21.15.28:0/1543675740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594fcb8fcd0 con 0x5594fcb8d570 2024-04-25T01:35:16.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.890+0000 7fd480d14640 1 --2- 172.21.15.28:0/1543675740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb8d570 0x5594fcb8d030 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd47403d000 tx=0x7fd474012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a610e06ca3e0cec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:16.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.892+0000 7fd461ffb640 1 -- 172.21.15.28:0/1543675740 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47404c070 con 0x5594fcb8d570 2024-04-25T01:35:16.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.892+0000 7fd461ffb640 1 -- 172.21.15.28:0/1543675740 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd474041040 con 0x5594fcb8d570 2024-04-25T01:35:16.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.894+0000 7fd485567740 1 -- 172.21.15.28:0/1543675740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb8d570 msgr2=0x5594fcb8d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:16.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.894+0000 7fd485567740 1 --2- 172.21.15.28:0/1543675740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb8d570 0x5594fcb8d030 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd47403d000 tx=0x7fd474012000 comp rx=0 tx=0).stop 2024-04-25T01:35:16.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.898+0000 7fd485567740 1 -- 172.21.15.28:0/1543675740 shutdown_connections 2024-04-25T01:35:16.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.898+0000 7fd485567740 1 --2- 172.21.15.28:0/1543675740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb8d570 0x5594fcb8d030 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.898+0000 7fd485567740 1 --2- 172.21.15.28:0/1543675740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594fcb82ea0 0x5594fcb8caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.898+0000 7fd485567740 1 --2- 172.21.15.28:0/1543675740 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594fcb82580 0x5594fcb992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.898+0000 7fd485567740 1 -- 172.21.15.28:0/1543675740 >> 172.21.15.28:0/1543675740 conn(0x5594fcb7e8a0 msgr2=0x5594fcb85740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.900+0000 7fd485567740 1 -- 172.21.15.28:0/1543675740 shutdown_connections 2024-04-25T01:35:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.904+0000 7fd485567740 1 -- 172.21.15.28:0/1543675740 wait complete. 2024-04-25T01:35:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.904+0000 7fd485567740 1 -- start start 2024-04-25T01:35:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.904+0000 7fd485567740 1 Processor -- start 2024-04-25T01:35:16.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.908+0000 7fd485567740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82580 0x5594fcb957b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.908+0000 7fd485567740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594fcb82ea0 0x5594fcb95cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.909+0000 7fd485567740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594fcb8d570 0x5594fcb92780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.909+0000 7fd485567740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c1c75b0 con 0x5594fcb82580 2024-04-25T01:35:16.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.909+0000 7fd485567740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c1c6ee0 con 0x5594fcb82ea0 2024-04-25T01:35:16.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.909+0000 7fd485567740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c1c4120 con 0x5594fcb8d570 2024-04-25T01:35:16.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.909+0000 7fd485567740 10 client.0 _ll_register_callbacks cb 0x7fd4700092c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:16.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.911+0000 7fd47bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82580 0x5594fcb957b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.911+0000 7fd47bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82580 0x5594fcb957b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47470/0 (socket says 172.21.15.28:47470) 2024-04-25T01:35:16.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.911+0000 7fd47bfff640 1 -- 172.21.15.28:0/2907101035 learned_addr learned my addr 172.21.15.28:0/2907101035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:16.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.911+0000 7fd480d14640 1 --2- 172.21.15.28:0/2907101035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594fcb8d570 0x5594fcb92780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.911+0000 7fd47b7fe640 1 --2- 172.21.15.28:0/2907101035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594fcb82ea0 0x5594fcb95cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.912+0000 7fd47bfff640 1 -- 172.21.15.28:0/2907101035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594fcb8d570 msgr2=0x5594fcb92780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:16.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.912+0000 7fd47bfff640 1 --2- 172.21.15.28:0/2907101035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594fcb8d570 0x5594fcb92780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.912+0000 7fd47bfff640 1 -- 172.21.15.28:0/2907101035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594fcb82ea0 msgr2=0x5594fcb95cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:16.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.912+0000 7fd47bfff640 1 --2- 172.21.15.28:0/2907101035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594fcb82ea0 0x5594fcb95cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.912+0000 7fd47bfff640 1 -- 172.21.15.28:0/2907101035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594fc8c57d0 con 0x5594fcb82580 2024-04-25T01:35:16.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.912+0000 7fd47bfff640 1 --2- 172.21.15.28:0/2907101035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82580 0x5594fcb957b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd470097d50 tx=0x7fd4700b4af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c9bd0faa0fc4f8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:16.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.912+0000 7fd45b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:16.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.913+0000 7fd45b7fe640 1 -- 172.21.15.28:0/2907101035 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47009b430 con 0x5594fcb82580 2024-04-25T01:35:16.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.913+0000 7fd45b7fe640 1 -- 172.21.15.28:0/2907101035 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fd4700ba2b0 con 0x5594fcb82580 2024-04-25T01:35:16.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.913+0000 7fd485567740 1 -- 172.21.15.28:0/2907101035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd474041040 con 0x5594fcb82580 2024-04-25T01:35:16.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.915+0000 7fd45b7fe640 1 -- 172.21.15.28:0/2907101035 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd4700a4560 con 0x5594fcb82580 2024-04-25T01:35:16.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.916+0000 7fd485567740 10 client.16329 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:16.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.916+0000 7fd485567740 1 -- 172.21.15.28:0/2907101035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5594fcc042e0 con 0x5594fcb82580 2024-04-25T01:35:16.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.917+0000 7fd45b7fe640 1 -- 172.21.15.28:0/2907101035 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd474041040 con 0x5594fcb82580 2024-04-25T01:35:16.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.917+0000 7fd45b7fe640 1 -- 172.21.15.28:0/2907101035 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd47009a7b0 con 0x5594fcb82580 2024-04-25T01:35:16.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.917+0000 7fd485567740 10 client.16329 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:16.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.917+0000 7fd485567740 10 client.16329 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:35:16.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.917+0000 7fd485567740 20 client.16329 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:16.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.917+0000 7fd485567740 1 --2- 172.21.15.28:0/2907101035 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5594fcc04870 0x5594fcc06cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:16.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.917+0000 7fd485567740 4 client.16329 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:35:16.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.917+0000 7fd485567740 1 -- 172.21.15.28:0/2907101035 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5594fcaeef00 con 0x5594fcc04870 2024-04-25T01:35:16.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.918+0000 7fd47b7fe640 1 --2- 172.21.15.28:0/2907101035 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5594fcc04870 0x5594fcc06cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:16.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.921+0000 7fd45b7fe640 10 client.16329 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:35:16.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.922+0000 7fd47b7fe640 1 --2- 172.21.15.28:0/2907101035 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5594fcc04870 0x5594fcc06cb0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ca83f0740e323dc0 server_cookie=6aa820eadd74c2e in_seq=0 out_seq=0 2024-04-25T01:35:16.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.923+0000 7fd45b7fe640 1 -- 172.21.15.28:0/2907101035 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5594fcaeef00 con 0x5594fcc04870 2024-04-25T01:35:16.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.923+0000 7fd45b7fe640 10 client.16329 handle_command_reply: tid=0 2024-04-25T01:35:16.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.923+0000 7fd485567740 1 client.16329 shutdown 2024-04-25T01:35:16.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.924+0000 7fd485567740 1 -- 172.21.15.28:0/2907101035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82580 msgr2=0x5594fcb957b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:16.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.924+0000 7fd485567740 1 --2- 172.21.15.28:0/2907101035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82580 0x5594fcb957b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd470097d50 tx=0x7fd4700b4af0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.925+0000 7fd485567740 1 -- 172.21.15.28:0/2907101035 shutdown_connections 2024-04-25T01:35:16.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.925+0000 7fd485567740 1 --2- 172.21.15.28:0/2907101035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5594fcb8d570 0x5594fcb92780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.925+0000 7fd485567740 1 --2- 172.21.15.28:0/2907101035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5594fcb82ea0 0x5594fcb95cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.925+0000 7fd485567740 1 --2- 172.21.15.28:0/2907101035 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5594fcc04870 0x5594fcc06cb0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.925+0000 7fd485567740 1 --2- 172.21.15.28:0/2907101035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5594fcb82580 0x5594fcb957b0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.925+0000 7fd485567740 1 -- 172.21.15.28:0/2907101035 >> 172.21.15.28:0/2907101035 conn(0x5594fcb7e8a0 msgr2=0x5594fcb98290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:16.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.925+0000 7fd45b7fe640 0 client.16329 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:35:16.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.926+0000 7fd485567740 1 -- 172.21.15.28:0/2907101035 shutdown_connections 2024-04-25T01:35:16.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.926+0000 7fd485567740 1 -- 172.21.15.28:0/2907101035 wait complete. 2024-04-25T01:35:16.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.927+0000 7fd485567740 20 client.16329 trim_cache size 0 max 16384 2024-04-25T01:35:16.903 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:35:16.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.928+0000 7fd482777640 1 -- 172.21.15.28:0/3731136462 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd4640a2f10 msgr2=0x7fd4640a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:16.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.928+0000 7fd482777640 1 --2- 172.21.15.28:0/3731136462 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd4640a2f10 0x7fd4640a53d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd474009a60 tx=0x7fd474039360 comp rx=0 tx=0).stop 2024-04-25T01:35:16.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.928+0000 7fd482777640 1 -- 172.21.15.28:0/3731136462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c101790 msgr2=0x7fd47c072cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:16.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.928+0000 7fd482777640 1 --2- 172.21.15.28:0/3731136462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c101790 0x7fd47c072cc0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd47000b450 tx=0x7fd47000b910 comp rx=0 tx=0).stop 2024-04-25T01:35:16.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.928+0000 7fd482777640 1 -- 172.21.15.28:0/3731136462 shutdown_connections 2024-04-25T01:35:16.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.928+0000 7fd482777640 1 --2- 172.21.15.28:0/3731136462 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd4640a2f10 0x7fd4640a53d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.928+0000 7fd482777640 1 --2- 172.21.15.28:0/3731136462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd47c1077a0 0x7fd47c06cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.928+0000 7fd482777640 1 --2- 172.21.15.28:0/3731136462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd47c102150 0x7fd47c073200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.928+0000 7fd482777640 1 --2- 172.21.15.28:0/3731136462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47c101790 0x7fd47c072cc0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:16.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.928+0000 7fd482777640 1 -- 172.21.15.28:0/3731136462 >> 172.21.15.28:0/3731136462 conn(0x7fd47c076250 msgr2=0x7fd47c076810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:16.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.931+0000 7fd482777640 1 -- 172.21.15.28:0/3731136462 shutdown_connections 2024-04-25T01:35:16.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:16.931+0000 7fd482777640 1 -- 172.21.15.28:0/3731136462 wait complete. 2024-04-25T01:35:16.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:35:17.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.201+0000 7fdbff541640 1 -- 172.21.15.28:0/4021584811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbf810d040 msgr2=0x7fdbf8114c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:17.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.201+0000 7fdbff541640 1 --2- 172.21.15.28:0/4021584811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbf810d040 0x7fdbf8114c10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0009ca0 tx=0x7fdbf002a8f0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.201+0000 7fdbff541640 1 -- 172.21.15.28:0/4021584811 shutdown_connections 2024-04-25T01:35:17.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.201+0000 7fdbff541640 1 --2- 172.21.15.28:0/4021584811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbf810d040 0x7fdbf8114c10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.201+0000 7fdbff541640 1 --2- 172.21.15.28:0/4021584811 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdbf8108970 0x7fdbf810c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.201+0000 7fdbff541640 1 --2- 172.21.15.28:0/4021584811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf8107fc0 0x7fdbf81083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.201+0000 7fdbff541640 1 -- 172.21.15.28:0/4021584811 >> 172.21.15.28:0/4021584811 conn(0x7fdbf8075c80 msgr2=0x7fdbf80780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:17.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.207+0000 7fdbff541640 1 -- 172.21.15.28:0/4021584811 shutdown_connections 2024-04-25T01:35:17.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.211+0000 7fdbff541640 1 -- 172.21.15.28:0/4021584811 wait complete. 2024-04-25T01:35:17.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.212+0000 7fdbff541640 1 Processor -- start 2024-04-25T01:35:17.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.213+0000 7fdbff541640 1 -- start start 2024-04-25T01:35:17.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.213+0000 7fdbff541640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdbf8107fc0 0x7fdbf8072f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.213+0000 7fdbff541640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbf810d040 0x7fdbf806bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.213+0000 7fdbff541640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf806c510 0x7fdbf806c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.213+0000 7fdbff541640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf8118150 con 0x7fdbf806c510 2024-04-25T01:35:17.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.213+0000 7fdbff541640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf8117fd0 con 0x7fdbf810d040 2024-04-25T01:35:17.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.213+0000 7fdbff541640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf81182d0 con 0x7fdbf8107fc0 2024-04-25T01:35:17.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.215+0000 7fdbfcadc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbf810d040 0x7fdbf806bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.215+0000 7fdbfcadc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbf810d040 0x7fdbf806bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:44384/0 (socket says 172.21.15.28:44384) 2024-04-25T01:35:17.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.215+0000 7fdbfcadc640 1 -- 172.21.15.28:0/379909129 learned_addr learned my addr 172.21.15.28:0/379909129 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:17.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.219+0000 7fdbfd2dd640 1 --2- 172.21.15.28:0/379909129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdbf8107fc0 0x7fdbf8072f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.219+0000 7fdbfcadc640 1 -- 172.21.15.28:0/379909129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdbf8107fc0 msgr2=0x7fdbf8072f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:17.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.219+0000 7fdbfcadc640 1 --2- 172.21.15.28:0/379909129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdbf8107fc0 0x7fdbf8072f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.219+0000 7fdbfcadc640 1 -- 172.21.15.28:0/379909129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf806c510 msgr2=0x7fdbf806c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:17.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.219+0000 7fdbfcadc640 1 --2- 172.21.15.28:0/379909129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf806c510 0x7fdbf806c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.219+0000 7fdbfcadc640 1 -- 172.21.15.28:0/379909129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbf8070a70 con 0x7fdbf810d040 2024-04-25T01:35:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.225+0000 7fdbfcadc640 1 --2- 172.21.15.28:0/379909129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbf810d040 0x7fdbf806bfd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400d770 tx=0x7fdbf400dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:17.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.229+0000 7fdbea7fc640 1 -- 172.21.15.28:0/379909129 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4017070 con 0x7fdbf810d040 2024-04-25T01:35:17.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.230+0000 7fdbea7fc640 1 -- 172.21.15.28:0/379909129 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf4009c30 con 0x7fdbf810d040 2024-04-25T01:35:17.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.230+0000 7fdbea7fc640 1 -- 172.21.15.28:0/379909129 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf400f620 con 0x7fdbf810d040 2024-04-25T01:35:17.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.231+0000 7fdbff541640 1 -- 172.21.15.28:0/379909129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbf8070d60 con 0x7fdbf810d040 2024-04-25T01:35:17.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.231+0000 7fdbff541640 1 -- 172.21.15.28:0/379909129 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbf8071260 con 0x7fdbf810d040 2024-04-25T01:35:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.238+0000 7fdc02245740 1 Processor -- start 2024-04-25T01:35:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.238+0000 7fdc02245740 1 -- start start 2024-04-25T01:35:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.238+0000 7fdc02245740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83d5580 0x55aae83d5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.238+0000 7fdc02245740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83d5ea0 0x55aae83e0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.239+0000 7fdc02245740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83e0570 0x55aae83e2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.239+0000 7fdc02245740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aae833a670 con 0x55aae83d5ea0 2024-04-25T01:35:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.239+0000 7fdc02245740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55aae82e2970 con 0x55aae83e0570 2024-04-25T01:35:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.239+0000 7fdc02245740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55aae8336630 con 0x55aae83d5580 2024-04-25T01:35:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.243+0000 7fdbfcadc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83e0570 0x55aae83e2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.244+0000 7fdbfcadc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83e0570 0x55aae83e2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:44396/0 (socket says 172.21.15.28:44396) 2024-04-25T01:35:17.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.244+0000 7fdbfcadc640 1 -- 172.21.15.28:0/951838173 learned_addr learned my addr 172.21.15.28:0/951838173 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:17.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.245+0000 7fdbfdade640 1 --2- 172.21.15.28:0/951838173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83d5580 0x55aae83d5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.245+0000 7fdbfd2dd640 1 --2- 172.21.15.28:0/951838173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83d5ea0 0x55aae83e0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.245+0000 7fdbea7fc640 1 -- 172.21.15.28:0/379909129 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fdbf401a070 con 0x7fdbf810d040 2024-04-25T01:35:17.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.250+0000 7fdbfdade640 1 -- 172.21.15.28:0/951838173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83e0570 msgr2=0x55aae83e2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:17.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.250+0000 7fdbfdade640 1 --2- 172.21.15.28:0/951838173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83e0570 0x55aae83e2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.250+0000 7fdbfdade640 1 -- 172.21.15.28:0/951838173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83d5ea0 msgr2=0x55aae83e0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.250+0000 7fdbfdade640 1 --2- 172.21.15.28:0/951838173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83d5ea0 0x55aae83e0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.250+0000 7fdbfdade640 1 -- 172.21.15.28:0/951838173 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aae8306400 con 0x55aae83d5580 2024-04-25T01:35:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.252+0000 7fdbea7fc640 1 --2- 172.21.15.28:0/379909129 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdbe40a2f00 0x7fdbe40a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.252+0000 7fdbea7fc640 1 -- 172.21.15.28:0/379909129 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fdbf40cf010 con 0x7fdbf810d040 2024-04-25T01:35:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.253+0000 7fdbfdade640 1 --2- 172.21.15.28:0/951838173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83d5580 0x55aae83d5960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0009bf0 tx=0x7fdbf0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10d4da4ac93bcd99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.255+0000 7fdbe37fe640 1 -- 172.21.15.28:0/951838173 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf00098f0 con 0x55aae83d5580 2024-04-25T01:35:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.255+0000 7fdbe37fe640 1 -- 172.21.15.28:0/951838173 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf003f070 con 0x55aae83d5580 2024-04-25T01:35:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.255+0000 7fdbe37fe640 1 -- 172.21.15.28:0/951838173 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0004040 con 0x55aae83d5580 2024-04-25T01:35:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.256+0000 7fdbfd2dd640 1 --2- 172.21.15.28:0/379909129 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdbe40a2f00 0x7fdbe40a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.257+0000 7fdc02245740 1 -- 172.21.15.28:0/951838173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83d5580 msgr2=0x55aae83d5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.258+0000 7fdc02245740 1 --2- 172.21.15.28:0/951838173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83d5580 0x55aae83d5960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0009bf0 tx=0x7fdbf0009780 comp rx=0 tx=0).stop 2024-04-25T01:35:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.258+0000 7fdc02245740 1 -- 172.21.15.28:0/951838173 shutdown_connections 2024-04-25T01:35:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.258+0000 7fdc02245740 1 --2- 172.21.15.28:0/951838173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83e0570 0x55aae83e2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.258+0000 7fdc02245740 1 --2- 172.21.15.28:0/951838173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83d5ea0 0x55aae83e0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.258+0000 7fdc02245740 1 --2- 172.21.15.28:0/951838173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83d5580 0x55aae83d5960 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.258+0000 7fdc02245740 1 -- 172.21.15.28:0/951838173 >> 172.21.15.28:0/951838173 conn(0x55aae83d18a0 msgr2=0x55aae83d1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:17.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.259+0000 7fdc02245740 1 -- 172.21.15.28:0/951838173 shutdown_connections 2024-04-25T01:35:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.261+0000 7fdbfd2dd640 1 --2- 172.21.15.28:0/379909129 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdbe40a2f00 0x7fdbe40a53c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x55aae83d6b40 tx=0x7fdbec008040 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.263+0000 7fdc02245740 1 -- 172.21.15.28:0/951838173 wait complete. 2024-04-25T01:35:17.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.263+0000 7fdc02245740 1 -- start start 2024-04-25T01:35:17.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.264+0000 7fdc02245740 1 Processor -- start 2024-04-25T01:35:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.265+0000 7fdc02245740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83d5580 0x55aae83dfee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.265+0000 7fdc02245740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83d5ea0 0x55aae83dd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.266+0000 7fdc02245740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83e0570 0x55aae83dd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.266+0000 7fdc02245740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aae83e3300 con 0x55aae83d5580 2024-04-25T01:35:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.266+0000 7fdc02245740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55aae82f6050 con 0x55aae83e0570 2024-04-25T01:35:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.266+0000 7fdc02245740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55aae83e3480 con 0x55aae83d5ea0 2024-04-25T01:35:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.266+0000 7fdbfcadc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83d5ea0 0x55aae83dd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.266+0000 7fdbfcadc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83d5ea0 0x55aae83dd1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.28:39356/0 (socket says 172.21.15.28:39356) 2024-04-25T01:35:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.266+0000 7fdbfcadc640 1 -- 172.21.15.28:0/1776230817 learned_addr learned my addr 172.21.15.28:0/1776230817 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.266+0000 7fdbfdade640 1 --2- 172.21.15.28:0/1776230817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83e0570 0x55aae83dd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.267+0000 7fdc02245740 10 client.0 _ll_register_callbacks cb 0x55aae804dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.267+0000 7fdbfcadc640 1 -- 172.21.15.28:0/1776230817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83e0570 msgr2=0x55aae83dd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.267+0000 7fdbfcadc640 1 --2- 172.21.15.28:0/1776230817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83e0570 0x55aae83dd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.267+0000 7fdbfcadc640 1 -- 172.21.15.28:0/1776230817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83d5580 msgr2=0x55aae83dfee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:35:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.267+0000 7fdbfcadc640 1 --2- 172.21.15.28:0/1776230817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83d5580 0x55aae83dfee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.267+0000 7fdbfcadc640 1 -- 172.21.15.28:0/1776230817 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aae82c7dc0 con 0x55aae83d5ea0 2024-04-25T01:35:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.269+0000 7fdbfcadc640 1 --2- 172.21.15.28:0/1776230817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83d5ea0 0x55aae83dd1d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdbf409af10 tx=0x7fdbf409afe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20a6f39cdeec2587 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.270+0000 7fdbe0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-25T01:35:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.270+0000 7fdbe0ff9640 1 -- 172.21.15.28:0/1776230817 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4009de0 con 0x55aae83d5ea0 2024-04-25T01:35:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.271+0000 7fdc02245740 1 -- 172.21.15.28:0/1776230817 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbf003f070 con 0x55aae83d5ea0 2024-04-25T01:35:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.271+0000 7fdbe0ff9640 1 -- 172.21.15.28:0/1776230817 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fdbf40cc640 con 0x55aae83d5ea0 2024-04-25T01:35:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.272+0000 7fdbe0ff9640 1 -- 172.21.15.28:0/1776230817 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdbf40b9050 con 0x55aae83d5ea0 2024-04-25T01:35:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.274+0000 7fdc02245740 10 client.25741 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.274+0000 7fdc02245740 1 -- 172.21.15.28:0/1776230817 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55aae804e3f0 con 0x55aae83d5ea0 2024-04-25T01:35:17.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.277+0000 7fdbe0ff9640 1 -- 172.21.15.28:0/1776230817 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf003f070 con 0x55aae83d5ea0 2024-04-25T01:35:17.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.277+0000 7fdbe0ff9640 1 -- 172.21.15.28:0/1776230817 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fdbf409f120 con 0x55aae83d5ea0 2024-04-25T01:35:17.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.277+0000 7fdc02245740 10 client.25741 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:17.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.277+0000 7fdc02245740 10 client.25741 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:35:17.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.277+0000 7fdc02245740 20 client.25741 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:17.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.277+0000 7fdc02245740 1 --2- 172.21.15.28:0/1776230817 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55aae83f0420 0x55aae83f28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.277+0000 7fdc02245740 4 client.25741 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:17.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.277+0000 7fdc02245740 1 -- 172.21.15.28:0/1776230817 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aae8347bb0 con 0x55aae83f0420 2024-04-25T01:35:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.279+0000 7fdbfd2dd640 1 --2- 172.21.15.28:0/1776230817 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55aae83f0420 0x55aae83f28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.290+0000 7fdbfd2dd640 1 --2- 172.21.15.28:0/1776230817 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55aae83f0420 0x55aae83f28b0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=88cf7b18b7f97675 server_cookie=4c208873a73be092 in_seq=0 out_seq=0 2024-04-25T01:35:17.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.294+0000 7fdbe0ff9640 10 client.25741 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:35:17.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.297+0000 7fdbe0ff9640 1 -- 172.21.15.28:0/1776230817 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aae8347bb0 con 0x55aae83f0420 2024-04-25T01:35:17.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.297+0000 7fdbe0ff9640 10 client.25741 handle_command_reply: tid=0 2024-04-25T01:35:17.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.298+0000 7fdc02245740 1 client.25741 shutdown 2024-04-25T01:35:17.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.303+0000 7fdc02245740 1 -- 172.21.15.28:0/1776230817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83d5ea0 msgr2=0x55aae83dd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:17.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.303+0000 7fdc02245740 1 --2- 172.21.15.28:0/1776230817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83d5ea0 0x55aae83dd1d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdbf409af10 tx=0x7fdbf409afe0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.305+0000 7fdc02245740 1 -- 172.21.15.28:0/1776230817 shutdown_connections 2024-04-25T01:35:17.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.305+0000 7fdc02245740 1 --2- 172.21.15.28:0/1776230817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83e0570 0x55aae83dd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.305+0000 7fdc02245740 1 --2- 172.21.15.28:0/1776230817 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55aae83f0420 0x55aae83f28b0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.305+0000 7fdc02245740 1 --2- 172.21.15.28:0/1776230817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83d5ea0 0x55aae83dd1d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.305+0000 7fdc02245740 1 --2- 172.21.15.28:0/1776230817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83d5580 0x55aae83dfee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.305+0000 7fdc02245740 1 -- 172.21.15.28:0/1776230817 >> 172.21.15.28:0/1776230817 conn(0x55aae83d18a0 msgr2=0x55aae83d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:17.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.305+0000 7fdbe0ff9640 0 client.25741 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:35:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.306+0000 7fdc02245740 1 -- 172.21.15.28:0/1776230817 shutdown_connections 2024-04-25T01:35:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.308+0000 7fdc02245740 1 -- 172.21.15.28:0/1776230817 wait complete. 2024-04-25T01:35:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.309+0000 7fdc02245740 20 client.25741 trim_cache size 0 max 16384 2024-04-25T01:35:17.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.350+0000 7fdc02245740 1 Processor -- start 2024-04-25T01:35:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.352+0000 7fdc02245740 1 -- start start 2024-04-25T01:35:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.352+0000 7fdc02245740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83d5580 0x55aae83d77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.352+0000 7fdc02245740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83d5ea0 0x55aae83dfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.352+0000 7fdc02245740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83e0570 0x55aae83f81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.352+0000 7fdc02245740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdbdc00fe90 con 0x55aae83d5ea0 2024-04-25T01:35:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.352+0000 7fdc02245740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdbdc0119d0 con 0x55aae83d5580 2024-04-25T01:35:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.352+0000 7fdc02245740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fdbdc010560 con 0x55aae83e0570 2024-04-25T01:35:17.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.355+0000 7fdbfdade640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83e0570 0x55aae83f81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.355+0000 7fdbfd2dd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83d5580 0x55aae83d77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.356+0000 7fdbfcadc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83d5ea0 0x55aae83dfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.356+0000 7fdbfcadc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83d5ea0 0x55aae83dfa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47494/0 (socket says 172.21.15.28:47494) 2024-04-25T01:35:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.356+0000 7fdbfcadc640 1 -- 172.21.15.28:0/3136128338 learned_addr learned my addr 172.21.15.28:0/3136128338 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.357+0000 7fdbfdade640 1 -- 172.21.15.28:0/3136128338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83d5580 msgr2=0x55aae83d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.357+0000 7fdbfdade640 1 --2- 172.21.15.28:0/3136128338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83d5580 0x55aae83d77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.357+0000 7fdbfdade640 1 -- 172.21.15.28:0/3136128338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83d5ea0 msgr2=0x55aae83dfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.357+0000 7fdbfdade640 1 --2- 172.21.15.28:0/3136128338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83d5ea0 0x55aae83dfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.357+0000 7fdbfdade640 1 -- 172.21.15.28:0/3136128338 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aae83ddaa0 con 0x55aae83e0570 2024-04-25T01:35:17.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.359+0000 7fdbfdade640 1 --2- 172.21.15.28:0/3136128338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83e0570 0x55aae83f81c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0009ca0 tx=0x7fdbf0009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99e27b35e086a900 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:17.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.360+0000 7fdbe2ffd640 1 -- 172.21.15.28:0/3136128338 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0050070 con 0x55aae83e0570 2024-04-25T01:35:17.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.360+0000 7fdbe2ffd640 1 -- 172.21.15.28:0/3136128338 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf003f050 con 0x55aae83e0570 2024-04-25T01:35:17.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.360+0000 7fdbe2ffd640 1 -- 172.21.15.28:0/3136128338 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0004040 con 0x55aae83e0570 2024-04-25T01:35:17.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.364+0000 7fdc02245740 1 -- 172.21.15.28:0/3136128338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83e0570 msgr2=0x55aae83f81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:17.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.364+0000 7fdc02245740 1 --2- 172.21.15.28:0/3136128338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83e0570 0x55aae83f81c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0009ca0 tx=0x7fdbf0009560 comp rx=0 tx=0).stop 2024-04-25T01:35:17.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.366+0000 7fdc02245740 1 -- 172.21.15.28:0/3136128338 shutdown_connections 2024-04-25T01:35:17.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.366+0000 7fdc02245740 1 --2- 172.21.15.28:0/3136128338 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83e0570 0x55aae83f81c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.366+0000 7fdc02245740 1 --2- 172.21.15.28:0/3136128338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83d5ea0 0x55aae83dfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.366+0000 7fdc02245740 1 --2- 172.21.15.28:0/3136128338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83d5580 0x55aae83d77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.366+0000 7fdc02245740 1 -- 172.21.15.28:0/3136128338 >> 172.21.15.28:0/3136128338 conn(0x55aae83d18a0 msgr2=0x55aae83d8740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:17.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.374+0000 7fdc02245740 1 -- 172.21.15.28:0/3136128338 shutdown_connections 2024-04-25T01:35:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.374+0000 7fdc02245740 1 -- 172.21.15.28:0/3136128338 wait complete. 2024-04-25T01:35:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.375+0000 7fdc02245740 1 -- start start 2024-04-25T01:35:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.375+0000 7fdc02245740 1 Processor -- start 2024-04-25T01:35:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.376+0000 7fdc02245740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83d5580 0x55aae83e87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.377+0000 7fdc02245740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83d5ea0 0x55aae83e4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.377+0000 7fdc02245740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83e0570 0x55aae83e50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.377+0000 7fdc02245740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdbdc011300 con 0x55aae83e0570 2024-04-25T01:35:17.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.377+0000 7fdc02245740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdbdc010c30 con 0x55aae83d5ea0 2024-04-25T01:35:17.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.377+0000 7fdc02245740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fdbdc00de70 con 0x55aae83d5580 2024-04-25T01:35:17.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.378+0000 7fdc02245740 10 client.0 _ll_register_callbacks cb 0x7fdbf40cc360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.384+0000 7fdbfdade640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83e0570 0x55aae83e50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.384+0000 7fdbfdade640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83e0570 0x55aae83e50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47502/0 (socket says 172.21.15.28:47502) 2024-04-25T01:35:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.384+0000 7fdbfdade640 1 -- 172.21.15.28:0/27161601 learned_addr learned my addr 172.21.15.28:0/27161601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.385+0000 7fdbfcadc640 1 --2- 172.21.15.28:0/27161601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83d5ea0 0x55aae83e4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.385+0000 7fdbfd2dd640 1 --2- 172.21.15.28:0/27161601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83d5580 0x55aae83e87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.385+0000 7fdbfdade640 1 -- 172.21.15.28:0/27161601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83d5580 msgr2=0x55aae83e87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.385+0000 7fdbfdade640 1 --2- 172.21.15.28:0/27161601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83d5580 0x55aae83e87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.385+0000 7fdbfdade640 1 -- 172.21.15.28:0/27161601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83d5ea0 msgr2=0x55aae83e4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.385+0000 7fdbfdade640 1 --2- 172.21.15.28:0/27161601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83d5ea0 0x55aae83e4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.385+0000 7fdbfdade640 1 -- 172.21.15.28:0/27161601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aae81187d0 con 0x55aae83e0570 2024-04-25T01:35:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.386+0000 7fdbfdade640 1 --2- 172.21.15.28:0/27161601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83e0570 0x55aae83e50a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0004af0 tx=0x7fdbf0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e37b4351d51868e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.386+0000 7fdbe0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:17.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.411+0000 7fdbe0ff9640 1 -- 172.21.15.28:0/27161601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0050040 con 0x55aae83e0570 2024-04-25T01:35:17.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.411+0000 7fdc02245740 1 -- 172.21.15.28:0/27161601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbf003f050 con 0x55aae83e0570 2024-04-25T01:35:17.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.412+0000 7fdbe0ff9640 1 -- 172.21.15.28:0/27161601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fdbf0003b70 con 0x55aae83e0570 2024-04-25T01:35:17.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.417+0000 7fdbe0ff9640 1 -- 172.21.15.28:0/27161601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdbf0003d70 con 0x55aae83e0570 2024-04-25T01:35:17.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.421+0000 7fdc02245740 10 client.16338 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:17.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.421+0000 7fdc02245740 1 -- 172.21.15.28:0/27161601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55aae8457540 con 0x55aae83e0570 2024-04-25T01:35:17.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.426+0000 7fdbe0ff9640 1 -- 172.21.15.28:0/27161601 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf003f050 con 0x55aae83e0570 2024-04-25T01:35:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.427+0000 7fdbe0ff9640 1 -- 172.21.15.28:0/27161601 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fdbf002fa50 con 0x55aae83e0570 2024-04-25T01:35:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.427+0000 7fdc02245740 10 client.16338 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.427+0000 7fdc02245740 10 client.16338 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:35:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.427+0000 7fdc02245740 20 client.16338 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.428+0000 7fdc02245740 1 --2- 172.21.15.28:0/27161601 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55aae84577d0 0x55aae8459bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.428+0000 7fdc02245740 4 client.16338 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:35:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.428+0000 7fdc02245740 1 -- 172.21.15.28:0/27161601 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55aae8341f00 con 0x55aae84577d0 2024-04-25T01:35:17.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.434+0000 7fdbfd2dd640 1 --2- 172.21.15.28:0/27161601 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55aae84577d0 0x55aae8459bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.449+0000 7fdbfd2dd640 1 --2- 172.21.15.28:0/27161601 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55aae84577d0 0x55aae8459bc0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=618abc5c307f32e4 server_cookie=271da440799f2791 in_seq=0 out_seq=0 2024-04-25T01:35:17.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.449+0000 7fdbe0ff9640 10 client.16338 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:35:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.450+0000 7fdbe0ff9640 1 -- 172.21.15.28:0/27161601 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55aae8341f00 con 0x55aae84577d0 2024-04-25T01:35:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.450+0000 7fdbe0ff9640 10 client.16338 handle_command_reply: tid=0 2024-04-25T01:35:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.450+0000 7fdc02245740 1 client.16338 shutdown 2024-04-25T01:35:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.451+0000 7fdc02245740 1 -- 172.21.15.28:0/27161601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83e0570 msgr2=0x55aae83e50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.451+0000 7fdc02245740 1 --2- 172.21.15.28:0/27161601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83e0570 0x55aae83e50a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0004af0 tx=0x7fdbf0009780 comp rx=0 tx=0).stop 2024-04-25T01:35:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.452+0000 7fdc02245740 1 -- 172.21.15.28:0/27161601 shutdown_connections 2024-04-25T01:35:17.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.452+0000 7fdc02245740 1 --2- 172.21.15.28:0/27161601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aae83e0570 0x55aae83e50a0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.452+0000 7fdc02245740 1 --2- 172.21.15.28:0/27161601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aae83d5ea0 0x55aae83e4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.452+0000 7fdc02245740 1 --2- 172.21.15.28:0/27161601 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55aae84577d0 0x55aae8459bc0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.453+0000 7fdc02245740 1 --2- 172.21.15.28:0/27161601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aae83d5580 0x55aae83e87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.453+0000 7fdc02245740 1 -- 172.21.15.28:0/27161601 >> 172.21.15.28:0/27161601 conn(0x55aae83d18a0 msgr2=0x55aae83d02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:17.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.453+0000 7fdbe0ff9640 0 client.16338 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:35:17.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.453+0000 7fdc02245740 1 -- 172.21.15.28:0/27161601 shutdown_connections 2024-04-25T01:35:17.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.453+0000 7fdc02245740 1 -- 172.21.15.28:0/27161601 wait complete. 2024-04-25T01:35:17.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.453+0000 7fdc02245740 20 client.16338 trim_cache size 0 max 16384 2024-04-25T01:35:17.432 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:35:17.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.456+0000 7fdbff541640 1 -- 172.21.15.28:0/379909129 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdbe40a2f00 msgr2=0x7fdbe40a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:17.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.456+0000 7fdbff541640 1 --2- 172.21.15.28:0/379909129 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdbe40a2f00 0x7fdbe40a53c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x55aae83d6b40 tx=0x7fdbec008040 comp rx=0 tx=0).stop 2024-04-25T01:35:17.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.456+0000 7fdbff541640 1 -- 172.21.15.28:0/379909129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbf810d040 msgr2=0x7fdbf806bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:17.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.456+0000 7fdbff541640 1 --2- 172.21.15.28:0/379909129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbf810d040 0x7fdbf806bfd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400d770 tx=0x7fdbf400dc30 comp rx=0 tx=0).stop 2024-04-25T01:35:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.456+0000 7fdbff541640 1 -- 172.21.15.28:0/379909129 shutdown_connections 2024-04-25T01:35:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.456+0000 7fdbff541640 1 --2- 172.21.15.28:0/379909129 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdbe40a2f00 0x7fdbe40a53c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.456+0000 7fdbff541640 1 --2- 172.21.15.28:0/379909129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf806c510 0x7fdbf806c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.456+0000 7fdbff541640 1 --2- 172.21.15.28:0/379909129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdbf810d040 0x7fdbf806bfd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.456+0000 7fdbff541640 1 --2- 172.21.15.28:0/379909129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdbf8107fc0 0x7fdbf8072f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.456+0000 7fdbff541640 1 -- 172.21.15.28:0/379909129 >> 172.21.15.28:0/379909129 conn(0x7fdbf8075c80 msgr2=0x7fdbf8078030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.456+0000 7fdbff541640 1 -- 172.21.15.28:0/379909129 shutdown_connections 2024-04-25T01:35:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.456+0000 7fdbff541640 1 -- 172.21.15.28:0/379909129 wait complete. 2024-04-25T01:35:17.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:35:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.822+0000 7f1febfff640 1 -- 172.21.15.28:0/2955699944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fec10dd00 msgr2=0x7f1fec10e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:17.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.822+0000 7f1febfff640 1 --2- 172.21.15.28:0/2955699944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fec10dd00 0x7f1fec10e0e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0009900 tx=0x7f1fe002a4d0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.826+0000 7f1febfff640 1 -- 172.21.15.28:0/2955699944 shutdown_connections 2024-04-25T01:35:17.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.827+0000 7f1febfff640 1 --2- 172.21.15.28:0/2955699944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fec108680 0x7f1fec114aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.827+0000 7f1febfff640 1 --2- 172.21.15.28:0/2955699944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fec107ce0 0x7f1fec108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.827+0000 7f1febfff640 1 --2- 172.21.15.28:0/2955699944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fec10dd00 0x7f1fec10e0e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.827+0000 7f1febfff640 1 -- 172.21.15.28:0/2955699944 >> 172.21.15.28:0/2955699944 conn(0x7f1fec077560 msgr2=0x7f1fec106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:17.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.834+0000 7f1febfff640 1 -- 172.21.15.28:0/2955699944 shutdown_connections 2024-04-25T01:35:17.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.837+0000 7f1febfff640 1 -- 172.21.15.28:0/2955699944 wait complete. 2024-04-25T01:35:17.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.838+0000 7f1febfff640 1 Processor -- start 2024-04-25T01:35:17.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.840+0000 7f1febfff640 1 -- start start 2024-04-25T01:35:17.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.840+0000 7f1febfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fec107ce0 0x7f1fec07bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.840+0000 7f1febfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fec108680 0x7f1fec07c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.840+0000 7f1febfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fec10dd00 0x7f1fec080490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.840+0000 7f1febfff640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec07a5a0 con 0x7f1fec107ce0 2024-04-25T01:35:17.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.840+0000 7f1febfff640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec07a420 con 0x7f1fec10dd00 2024-04-25T01:35:17.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.840+0000 7f1febfff640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec07a720 con 0x7f1fec108680 2024-04-25T01:35:17.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.842+0000 7f1feaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fec107ce0 0x7f1fec07bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.842+0000 7f1feaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fec107ce0 0x7f1fec07bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47514/0 (socket says 172.21.15.28:47514) 2024-04-25T01:35:17.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.842+0000 7f1feaffd640 1 -- 172.21.15.28:0/3849739626 learned_addr learned my addr 172.21.15.28:0/3849739626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:17.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.843+0000 7f1fea7fc640 1 --2- 172.21.15.28:0/3849739626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fec108680 0x7f1fec07c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.845+0000 7f1feb7fe640 1 --2- 172.21.15.28:0/3849739626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fec10dd00 0x7f1fec080490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.852+0000 7f1fea7fc640 1 -- 172.21.15.28:0/3849739626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fec10dd00 msgr2=0x7f1fec080490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:17.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.852+0000 7f1fea7fc640 1 --2- 172.21.15.28:0/3849739626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fec10dd00 0x7f1fec080490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.852+0000 7f1fea7fc640 1 -- 172.21.15.28:0/3849739626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fec107ce0 msgr2=0x7f1fec07bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:17.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.852+0000 7f1fea7fc640 1 --2- 172.21.15.28:0/3849739626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fec107ce0 0x7f1fec07bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.852+0000 7f1fea7fc640 1 -- 172.21.15.28:0/3849739626 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fec080b40 con 0x7f1fec108680 2024-04-25T01:35:17.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.852+0000 7f1feaffd640 1 --2- 172.21.15.28:0/3849739626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fec107ce0 0x7f1fec07bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:35:17.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.857+0000 7f1fea7fc640 1 --2- 172.21.15.28:0/3849739626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fec108680 0x7f1fec07c110 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400be50 tx=0x7f1fe4009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:17.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.865+0000 7f1ff0a36640 1 -- 172.21.15.28:0/3849739626 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4019070 con 0x7f1fec108680 2024-04-25T01:35:17.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.865+0000 7f1febfff640 1 -- 172.21.15.28:0/3849739626 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fec080e30 con 0x7f1fec108680 2024-04-25T01:35:17.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.866+0000 7f1febfff640 1 -- 172.21.15.28:0/3849739626 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fec06b160 con 0x7f1fec108680 2024-04-25T01:35:17.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.866+0000 7f1ff0a36640 1 -- 172.21.15.28:0/3849739626 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fe4014920 con 0x7f1fec108680 2024-04-25T01:35:17.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.866+0000 7f1ff0a36640 1 -- 172.21.15.28:0/3849739626 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe40129e0 con 0x7f1fec108680 2024-04-25T01:35:17.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.876+0000 7f1ff4828740 1 Processor -- start 2024-04-25T01:35:17.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.879+0000 7f1ff4828740 1 -- start start 2024-04-25T01:35:17.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.879+0000 7f1ff4828740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03ec720 0x555fc03ecb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.879+0000 7f1ff4828740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fc03ed040 0x555fc03f71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.879+0000 7f1ff4828740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03f7710 0x555fc03f9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.879+0000 7f1ff4828740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555fc0351670 con 0x555fc03ec720 2024-04-25T01:35:17.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.879+0000 7f1ff4828740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555fc02f9970 con 0x555fc03f7710 2024-04-25T01:35:17.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.879+0000 7f1ff4828740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x555fc034d630 con 0x555fc03ed040 2024-04-25T01:35:17.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.882+0000 7f1feb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03ec720 0x555fc03ecb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.883+0000 7f1feaffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fc03ed040 0x555fc03f71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.883+0000 7f1feb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03ec720 0x555fc03ecb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47528/0 (socket says 172.21.15.28:47528) 2024-04-25T01:35:17.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.883+0000 7f1feb7fe640 1 -- 172.21.15.28:0/2376711990 learned_addr learned my addr 172.21.15.28:0/2376711990 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:17.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.886+0000 7f1feb7fe640 1 -- 172.21.15.28:0/2376711990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fc03ed040 msgr2=0x555fc03f71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:17.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.886+0000 7f1feb7fe640 1 --2- 172.21.15.28:0/2376711990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fc03ed040 0x555fc03f71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.886+0000 7f1feb7fe640 1 -- 172.21.15.28:0/2376711990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03f7710 msgr2=0x555fc03f9b00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:35:17.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.886+0000 7f1feb7fe640 1 --2- 172.21.15.28:0/2376711990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03f7710 0x555fc03f9b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.886+0000 7f1feb7fe640 1 -- 172.21.15.28:0/2376711990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fc031d400 con 0x555fc03ec720 2024-04-25T01:35:17.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.890+0000 7f1feb7fe640 1 --2- 172.21.15.28:0/2376711990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03ec720 0x555fc03ecb00 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0099d0 tx=0x7f1fdc009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc4e86131d6dbfd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:17.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.898+0000 7f1ff0a36640 1 -- 172.21.15.28:0/3849739626 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1fe4012b80 con 0x7f1fec108680 2024-04-25T01:35:17.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.899+0000 7f1fd1ffb640 1 -- 172.21.15.28:0/2376711990 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc017070 con 0x555fc03ec720 2024-04-25T01:35:17.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.899+0000 7f1fd1ffb640 1 -- 172.21.15.28:0/2376711990 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fdc013470 con 0x555fc03ec720 2024-04-25T01:35:17.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.899+0000 7f1fd1ffb640 1 -- 172.21.15.28:0/2376711990 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0125f0 con 0x555fc03ec720 2024-04-25T01:35:17.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.900+0000 7f1ff0a36640 1 --2- 172.21.15.28:0/3849739626 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1fd40a2f10 0x7f1fd40a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.900+0000 7f1ff0a36640 1 -- 172.21.15.28:0/3849739626 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1fe40cee70 con 0x7f1fec108680 2024-04-25T01:35:17.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.901+0000 7f1ff4828740 1 -- 172.21.15.28:0/2376711990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03ec720 msgr2=0x555fc03ecb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:17.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.901+0000 7f1ff4828740 1 --2- 172.21.15.28:0/2376711990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03ec720 0x555fc03ecb00 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0099d0 tx=0x7f1fdc009e90 comp rx=0 tx=0).stop 2024-04-25T01:35:17.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.902+0000 7f1feaffd640 1 --2- 172.21.15.28:0/3849739626 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1fd40a2f10 0x7f1fd40a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.906+0000 7f1ff4828740 1 -- 172.21.15.28:0/2376711990 shutdown_connections 2024-04-25T01:35:17.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.906+0000 7f1ff4828740 1 --2- 172.21.15.28:0/2376711990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03f7710 0x555fc03f9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.906+0000 7f1ff4828740 1 --2- 172.21.15.28:0/2376711990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fc03ed040 0x555fc03f71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.906+0000 7f1ff4828740 1 --2- 172.21.15.28:0/2376711990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03ec720 0x555fc03ecb00 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.906+0000 7f1ff4828740 1 -- 172.21.15.28:0/2376711990 >> 172.21.15.28:0/2376711990 conn(0x555fc03e8a40 msgr2=0x555fc03e8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:17.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.908+0000 7f1feaffd640 1 --2- 172.21.15.28:0/3849739626 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1fd40a2f10 0x7f1fd40a53d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00096c0 tx=0x7f1fe0005bd0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:17.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.910+0000 7f1ff4828740 1 -- 172.21.15.28:0/2376711990 shutdown_connections 2024-04-25T01:35:17.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.914+0000 7f1ff4828740 1 -- 172.21.15.28:0/2376711990 wait complete. 2024-04-25T01:35:17.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.915+0000 7f1ff4828740 1 -- start start 2024-04-25T01:35:17.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.916+0000 7f1ff4828740 1 Processor -- start 2024-04-25T01:35:17.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.916+0000 7f1ff4828740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fc03ec720 0x555fc03f70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.917+0000 7f1ff4828740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03ed040 0x555fc03f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.917+0000 7f1ff4828740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03f7710 0x555fc03f4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.917+0000 7f1ff4828740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555fc03fa500 con 0x555fc03f7710 2024-04-25T01:35:17.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.917+0000 7f1ff4828740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555fc030d050 con 0x555fc03ed040 2024-04-25T01:35:17.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.917+0000 7f1ff4828740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x555fc03fa680 con 0x555fc03ec720 2024-04-25T01:35:17.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.917+0000 7f1ff4828740 10 client.0 _ll_register_callbacks cb 0x555fc0064be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:17.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.919+0000 7f1feb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03f7710 0x555fc03f4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.919+0000 7f1feb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03f7710 0x555fc03f4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47532/0 (socket says 172.21.15.28:47532) 2024-04-25T01:35:17.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.919+0000 7f1feb7fe640 1 -- 172.21.15.28:0/886256564 learned_addr learned my addr 172.21.15.28:0/886256564 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:17.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.920+0000 7f1fea7fc640 1 --2- 172.21.15.28:0/886256564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03ed040 0x555fc03f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.920+0000 7f1feb7fe640 1 -- 172.21.15.28:0/886256564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fc03ec720 msgr2=0x555fc03f70c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:35:17.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.920+0000 7f1feb7fe640 1 --2- 172.21.15.28:0/886256564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fc03ec720 0x555fc03f70c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.920+0000 7f1feb7fe640 1 -- 172.21.15.28:0/886256564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03ed040 msgr2=0x555fc03f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:17.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.920+0000 7f1feb7fe640 1 --2- 172.21.15.28:0/886256564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03ed040 0x555fc03f43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.920+0000 7f1feb7fe640 1 -- 172.21.15.28:0/886256564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fc02dedc0 con 0x555fc03f7710 2024-04-25T01:35:17.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.926+0000 7f1feb7fe640 1 --2- 172.21.15.28:0/886256564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03f7710 0x555fc03f4950 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00eec0 tx=0x7f1fdc00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1bb0396b2e86436 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:17.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.929+0000 7f1fcaffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:17.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.929+0000 7f1fcaffd640 1 -- 172.21.15.28:0/886256564 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc029060 con 0x555fc03f7710 2024-04-25T01:35:17.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.930+0000 7f1fcaffd640 1 -- 172.21.15.28:0/886256564 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1fdc01c450 con 0x555fc03f7710 2024-04-25T01:35:17.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.931+0000 7f1ff4828740 1 -- 172.21.15.28:0/886256564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fdc013470 con 0x555fc03f7710 2024-04-25T01:35:17.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.933+0000 7f1fcaffd640 1 -- 172.21.15.28:0/886256564 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1fdc017040 con 0x555fc03f7710 2024-04-25T01:35:17.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.933+0000 7f1ff4828740 10 client.16347 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:17.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.933+0000 7f1ff4828740 1 -- 172.21.15.28:0/886256564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555fc00653f0 con 0x555fc03f7710 2024-04-25T01:35:17.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.935+0000 7f1fcaffd640 1 -- 172.21.15.28:0/886256564 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fdc013470 con 0x555fc03f7710 2024-04-25T01:35:17.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.936+0000 7f1fcaffd640 1 -- 172.21.15.28:0/886256564 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1fdc02e7a0 con 0x555fc03f7710 2024-04-25T01:35:17.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.936+0000 7f1ff4828740 10 client.16347 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:17.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.936+0000 7f1ff4828740 10 client.16347 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:35:17.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.936+0000 7f1ff4828740 20 client.16347 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:17.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.936+0000 7f1ff4828740 1 --2- 172.21.15.28:0/886256564 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x555fc0407970 0x555fc0409e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.936+0000 7f1ff4828740 4 client.16347 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:17.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.936+0000 7f1ff4828740 1 -- 172.21.15.28:0/886256564 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555fc035ebb0 con 0x555fc0407970 2024-04-25T01:35:17.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.937+0000 7f1feaffd640 1 --2- 172.21.15.28:0/886256564 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x555fc0407970 0x555fc0409e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.945+0000 7f1feaffd640 1 --2- 172.21.15.28:0/886256564 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x555fc0407970 0x555fc0409e10 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=df8696e27c213c7 server_cookie=e96004c29834a93e in_seq=0 out_seq=0 2024-04-25T01:35:17.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.945+0000 7f1fcaffd640 10 client.16347 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:35:17.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.947+0000 7f1fcaffd640 1 -- 172.21.15.28:0/886256564 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555fc035ebb0 con 0x555fc0407970 2024-04-25T01:35:17.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.947+0000 7f1fcaffd640 10 client.16347 handle_command_reply: tid=0 2024-04-25T01:35:17.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.947+0000 7f1ff4828740 1 client.16347 shutdown 2024-04-25T01:35:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.953+0000 7f1ff4828740 1 -- 172.21.15.28:0/886256564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03f7710 msgr2=0x555fc03f4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.953+0000 7f1ff4828740 1 --2- 172.21.15.28:0/886256564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03f7710 0x555fc03f4950 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00eec0 tx=0x7f1fdc00eef0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.956+0000 7f1ff4828740 1 -- 172.21.15.28:0/886256564 shutdown_connections 2024-04-25T01:35:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.956+0000 7f1ff4828740 1 --2- 172.21.15.28:0/886256564 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x555fc0407970 0x555fc0409e10 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.956+0000 7f1ff4828740 1 --2- 172.21.15.28:0/886256564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03f7710 0x555fc03f4950 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.956+0000 7f1ff4828740 1 --2- 172.21.15.28:0/886256564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03ed040 0x555fc03f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.956+0000 7f1ff4828740 1 --2- 172.21.15.28:0/886256564 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fc03ec720 0x555fc03f70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.956+0000 7f1ff4828740 1 -- 172.21.15.28:0/886256564 >> 172.21.15.28:0/886256564 conn(0x555fc03e8a40 msgr2=0x555fc03ee740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.957+0000 7f1fcaffd640 0 client.16347 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:35:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.962+0000 7f1ff4828740 1 -- 172.21.15.28:0/886256564 shutdown_connections 2024-04-25T01:35:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.962+0000 7f1ff4828740 1 -- 172.21.15.28:0/886256564 wait complete. 2024-04-25T01:35:17.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:17.966+0000 7f1ff4828740 20 client.16347 trim_cache size 0 max 16384 2024-04-25T01:35:17.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.008+0000 7f1ff4828740 1 Processor -- start 2024-04-25T01:35:17.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.010+0000 7f1ff4828740 1 -- start start 2024-04-25T01:35:17.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.010+0000 7f1ff4828740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03ec720 0x555fc0403450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.010+0000 7f1ff4828740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fc03ed040 0x555fc03e7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.010+0000 7f1ff4828740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03f7710 0x555fc03f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:17.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.010+0000 7f1ff4828740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd800fe80 con 0x555fc03f7710 2024-04-25T01:35:17.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.010+0000 7f1ff4828740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd80119c0 con 0x555fc03ec720 2024-04-25T01:35:17.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.010+0000 7f1ff4828740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd8010550 con 0x555fc03ed040 2024-04-25T01:35:17.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.011+0000 7f1feaffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03ec720 0x555fc0403450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:17.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.011+0000 7f1feaffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03ec720 0x555fc0403450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:44490/0 (socket says 172.21.15.28:44490) 2024-04-25T01:35:17.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.011+0000 7f1feaffd640 1 -- 172.21.15.28:0/594905686 learned_addr learned my addr 172.21.15.28:0/594905686 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:17.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.012+0000 7f1feaffd640 1 -- 172.21.15.28:0/594905686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fc03ed040 msgr2=0x555fc03e7a40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:35:17.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.012+0000 7f1feaffd640 1 --2- 172.21.15.28:0/594905686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fc03ed040 0x555fc03e7a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.012+0000 7f1feaffd640 1 -- 172.21.15.28:0/594905686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03f7710 msgr2=0x555fc03f6bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:35:17.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.012+0000 7f1feaffd640 1 --2- 172.21.15.28:0/594905686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03f7710 0x555fc03f6bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.012+0000 7f1feaffd640 1 -- 172.21.15.28:0/594905686 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fc012f7d0 con 0x555fc03ec720 2024-04-25T01:35:17.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.013+0000 7f1feaffd640 1 --2- 172.21.15.28:0/594905686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03ec720 0x555fc0403450 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0009230 tx=0x7f1fe003f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31f88d482225655e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.015+0000 7f1fd17fa640 1 -- 172.21.15.28:0/594905686 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe0056070 con 0x555fc03ec720 2024-04-25T01:35:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.015+0000 7f1fd17fa640 1 -- 172.21.15.28:0/594905686 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fe00517b0 con 0x555fc03ec720 2024-04-25T01:35:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.016+0000 7f1ff4828740 1 -- 172.21.15.28:0/594905686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03ec720 msgr2=0x555fc0403450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.016+0000 7f1ff4828740 1 --2- 172.21.15.28:0/594905686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03ec720 0x555fc0403450 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0009230 tx=0x7f1fe003f000 comp rx=0 tx=0).stop 2024-04-25T01:35:17.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.019+0000 7f1ff4828740 1 -- 172.21.15.28:0/594905686 shutdown_connections 2024-04-25T01:35:17.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.019+0000 7f1ff4828740 1 --2- 172.21.15.28:0/594905686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03f7710 0x555fc03f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.019+0000 7f1ff4828740 1 --2- 172.21.15.28:0/594905686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fc03ed040 0x555fc03e7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.019+0000 7f1ff4828740 1 --2- 172.21.15.28:0/594905686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03ec720 0x555fc0403450 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:17.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.019+0000 7f1ff4828740 1 -- 172.21.15.28:0/594905686 >> 172.21.15.28:0/594905686 conn(0x555fc03e8a40 msgr2=0x555fc03e7330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:17.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.020+0000 7f1ff4828740 1 -- 172.21.15.28:0/594905686 shutdown_connections 2024-04-25T01:35:17.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.022+0000 7f1ff4828740 1 -- 172.21.15.28:0/594905686 wait complete. 2024-04-25T01:35:18.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.023+0000 7f1ff4828740 1 -- start start 2024-04-25T01:35:18.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.023+0000 7f1ff4828740 1 Processor -- start 2024-04-25T01:35:18.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.028+0000 7f1ff4828740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fc03ec720 0x555fc03fa920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.028+0000 7f1ff4828740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03ed040 0x555fc03fae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.028+0000 7f1ff4828740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03f7710 0x555fc046bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.028+0000 7f1ff4828740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd80112f0 con 0x555fc03ed040 2024-04-25T01:35:18.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.028+0000 7f1ff4828740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd8010c20 con 0x555fc03f7710 2024-04-25T01:35:18.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.028+0000 7f1ff4828740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd800de60 con 0x555fc03ec720 2024-04-25T01:35:18.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.028+0000 7f1ff4828740 10 client.0 _ll_register_callbacks cb 0x7f1fdc009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:18.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.029+0000 7f1fea7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03ed040 0x555fc03fae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.029+0000 7f1fea7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03ed040 0x555fc03fae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47534/0 (socket says 172.21.15.28:47534) 2024-04-25T01:35:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.029+0000 7f1fea7fc640 1 -- 172.21.15.28:0/1552240015 learned_addr learned my addr 172.21.15.28:0/1552240015 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.029+0000 7f1fea7fc640 1 -- 172.21.15.28:0/1552240015 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fc03ec720 msgr2=0x555fc03fa920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:35:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.030+0000 7f1fea7fc640 1 --2- 172.21.15.28:0/1552240015 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fc03ec720 0x555fc03fa920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.030+0000 7f1fea7fc640 1 -- 172.21.15.28:0/1552240015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03f7710 msgr2=0x555fc046bf80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:35:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.030+0000 7f1fea7fc640 1 --2- 172.21.15.28:0/1552240015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03f7710 0x555fc046bf80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.030+0000 7f1fea7fc640 1 -- 172.21.15.28:0/1552240015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fc046c4c0 con 0x555fc03ed040 2024-04-25T01:35:18.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.030+0000 7f1fea7fc640 1 --2- 172.21.15.28:0/1552240015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03ed040 0x555fc03fae60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f1fe409b140 tx=0x7f1fe4012340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d077b62104d0eedd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:18.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.030+0000 7f1fcaffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:18.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.031+0000 7f1fcaffd640 1 -- 172.21.15.28:0/1552240015 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe40b4060 con 0x555fc03ed040 2024-04-25T01:35:18.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.031+0000 7f1ff4828740 1 -- 172.21.15.28:0/1552240015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555fc0358f00 con 0x555fc03ed040 2024-04-25T01:35:18.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.031+0000 7f1fcaffd640 1 -- 172.21.15.28:0/1552240015 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1fe4097820 con 0x555fc03ed040 2024-04-25T01:35:18.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.032+0000 7f1fcaffd640 1 -- 172.21.15.28:0/1552240015 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1fe409b1b0 con 0x555fc03ed040 2024-04-25T01:35:18.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.033+0000 7f1ff4828740 10 client.16350 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:18.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.033+0000 7f1ff4828740 1 -- 172.21.15.28:0/1552240015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555fc046d780 con 0x555fc03ed040 2024-04-25T01:35:18.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.033+0000 7f1fcaffd640 1 -- 172.21.15.28:0/1552240015 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555fc0358f00 con 0x555fc03ed040 2024-04-25T01:35:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.035+0000 7f1fcaffd640 1 -- 172.21.15.28:0/1552240015 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1fe40a38e0 con 0x555fc03ed040 2024-04-25T01:35:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.035+0000 7f1ff4828740 10 client.16350 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.035+0000 7f1ff4828740 10 client.16350 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:35:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.035+0000 7f1ff4828740 20 client.16350 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.035+0000 7f1ff4828740 1 --2- 172.21.15.28:0/1552240015 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x555fc046df50 0x555fc0470370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.035+0000 7f1ff4828740 4 client.16350 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:35:18.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.035+0000 7f1ff4828740 1 -- 172.21.15.28:0/1552240015 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555fc02d6820 con 0x555fc046df50 2024-04-25T01:35:18.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.042+0000 7f1feaffd640 1 --2- 172.21.15.28:0/1552240015 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x555fc046df50 0x555fc0470370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:18.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.061+0000 7f1feaffd640 1 --2- 172.21.15.28:0/1552240015 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x555fc046df50 0x555fc0470370 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1f8dedabbfa619bd server_cookie=70e95387893237bb in_seq=0 out_seq=0 2024-04-25T01:35:18.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.061+0000 7f1fcaffd640 10 client.16350 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:35:18.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.063+0000 7f1fcaffd640 1 -- 172.21.15.28:0/1552240015 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555fc02d6820 con 0x555fc046df50 2024-04-25T01:35:18.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.064+0000 7f1fcaffd640 10 client.16350 handle_command_reply: tid=0 2024-04-25T01:35:18.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.064+0000 7f1ff4828740 1 client.16350 shutdown 2024-04-25T01:35:18.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.068+0000 7f1ff4828740 1 -- 172.21.15.28:0/1552240015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03ed040 msgr2=0x555fc03fae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:18.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.068+0000 7f1ff4828740 1 --2- 172.21.15.28:0/1552240015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03ed040 0x555fc03fae60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f1fe409b140 tx=0x7f1fe4012340 comp rx=0 tx=0).stop 2024-04-25T01:35:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.076+0000 7f1ff4828740 1 -- 172.21.15.28:0/1552240015 shutdown_connections 2024-04-25T01:35:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.076+0000 7f1ff4828740 1 --2- 172.21.15.28:0/1552240015 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x555fc046df50 0x555fc0470370 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.076+0000 7f1ff4828740 1 --2- 172.21.15.28:0/1552240015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555fc03f7710 0x555fc046bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.076+0000 7f1ff4828740 1 --2- 172.21.15.28:0/1552240015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555fc03ed040 0x555fc03fae60 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.076+0000 7f1ff4828740 1 --2- 172.21.15.28:0/1552240015 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555fc03ec720 0x555fc03fa920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.076+0000 7f1ff4828740 1 -- 172.21.15.28:0/1552240015 >> 172.21.15.28:0/1552240015 conn(0x555fc03e8a40 msgr2=0x555fc03ef8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.076+0000 7f1fcaffd640 0 client.16350 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:35:18.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.083+0000 7f1ff4828740 1 -- 172.21.15.28:0/1552240015 shutdown_connections 2024-04-25T01:35:18.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.083+0000 7f1ff4828740 1 -- 172.21.15.28:0/1552240015 wait complete. 2024-04-25T01:35:18.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.084+0000 7f1ff4828740 20 client.16350 trim_cache size 0 max 16384 2024-04-25T01:35:18.061 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:35:18.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.088+0000 7f1febfff640 1 -- 172.21.15.28:0/3849739626 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1fd40a2f10 msgr2=0x7f1fd40a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:18.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.088+0000 7f1febfff640 1 --2- 172.21.15.28:0/3849739626 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1fd40a2f10 0x7f1fd40a53d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00096c0 tx=0x7f1fe0005bd0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.088+0000 7f1febfff640 1 -- 172.21.15.28:0/3849739626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fec108680 msgr2=0x7f1fec07c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:18.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.088+0000 7f1febfff640 1 --2- 172.21.15.28:0/3849739626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fec108680 0x7f1fec07c110 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400be50 tx=0x7f1fe4009f70 comp rx=0 tx=0).stop 2024-04-25T01:35:18.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.088+0000 7f1febfff640 1 -- 172.21.15.28:0/3849739626 shutdown_connections 2024-04-25T01:35:18.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.088+0000 7f1febfff640 1 --2- 172.21.15.28:0/3849739626 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1fd40a2f10 0x7f1fd40a53d0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00096c0 tx=0x7f1fe0005bd0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.088+0000 7f1febfff640 1 --2- 172.21.15.28:0/3849739626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fec10dd00 0x7f1fec080490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.088+0000 7f1febfff640 1 --2- 172.21.15.28:0/3849739626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fec108680 0x7f1fec07c110 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.088+0000 7f1febfff640 1 --2- 172.21.15.28:0/3849739626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fec107ce0 0x7f1fec07bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.088+0000 7f1febfff640 1 -- 172.21.15.28:0/3849739626 >> 172.21.15.28:0/3849739626 conn(0x7f1fec077560 msgr2=0x7f1fec078330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:18.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.089+0000 7f1febfff640 1 -- 172.21.15.28:0/3849739626 shutdown_connections 2024-04-25T01:35:18.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.091+0000 7f1febfff640 1 -- 172.21.15.28:0/3849739626 wait complete. 2024-04-25T01:35:18.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:35:18.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:18 smithi028 ceph-mon[29390]: pgmap v71: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 675 MiB/s wr, 344 op/s 2024-04-25T01:35:18.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.396+0000 7f0565826640 1 -- 172.21.15.28:0/4097894172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05580051f0 msgr2=0x7f0558005670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:18.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.396+0000 7f0565826640 1 --2- 172.21.15.28:0/4097894172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05580051f0 0x7f0558005670 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f054c009860 tx=0x7f054c02a340 comp rx=0 tx=0).stop 2024-04-25T01:35:18.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.397+0000 7f0565826640 1 -- 172.21.15.28:0/4097894172 shutdown_connections 2024-04-25T01:35:18.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.397+0000 7f0565826640 1 --2- 172.21.15.28:0/4097894172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05580986e0 0x7f0558098b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.397+0000 7f0565826640 1 --2- 172.21.15.28:0/4097894172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05580051f0 0x7f0558005670 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f054c009860 tx=0x7f054c02a340 comp rx=0 tx=0).stop 2024-04-25T01:35:18.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.397+0000 7f0565826640 1 --2- 172.21.15.28:0/4097894172 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f055809acf0 0x7f0558004cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.397+0000 7f0565826640 1 -- 172.21.15.28:0/4097894172 >> 172.21.15.28:0/4097894172 conn(0x7f055800a740 msgr2=0x7f055800ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:18.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.402+0000 7f0565826640 1 -- 172.21.15.28:0/4097894172 shutdown_connections 2024-04-25T01:35:18.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.406+0000 7f0565826640 1 -- 172.21.15.28:0/4097894172 wait complete. 2024-04-25T01:35:18.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.407+0000 7f0565826640 1 Processor -- start 2024-04-25T01:35:18.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.407+0000 7f0565826640 1 -- start start 2024-04-25T01:35:18.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.407+0000 7f0565826640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05580986e0 0x7f05580a9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.407+0000 7f0565826640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055809acf0 0x7f05580a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.407+0000 7f0565826640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05580a5970 0x7f05580a5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.408+0000 7f0565826640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f055809dc20 con 0x7f055809acf0 2024-04-25T01:35:18.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.408+0000 7f0565826640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f055809daa0 con 0x7f05580986e0 2024-04-25T01:35:18.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.408+0000 7f0565826640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f055809dda0 con 0x7f05580a5970 2024-04-25T01:35:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.411+0000 7f0565025640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05580a5970 0x7f05580a5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.411+0000 7f0565025640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05580a5970 0x7f05580a5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.28:39436/0 (socket says 172.21.15.28:39436) 2024-04-25T01:35:18.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.411+0000 7f0565025640 1 -- 172.21.15.28:0/237009894 learned_addr learned my addr 172.21.15.28:0/237009894 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:18.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.411+0000 7f0564824640 1 --2- 172.21.15.28:0/237009894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05580986e0 0x7f05580a9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.412+0000 7f0565025640 1 -- 172.21.15.28:0/237009894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05580986e0 msgr2=0x7f05580a9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.412+0000 7f0565025640 1 --2- 172.21.15.28:0/237009894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05580986e0 0x7f05580a9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.412+0000 7f0565025640 1 -- 172.21.15.28:0/237009894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055809acf0 msgr2=0x7f05580a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.412+0000 7f0565025640 1 --2- 172.21.15.28:0/237009894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055809acf0 0x7f05580a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.412+0000 7f0565025640 1 -- 172.21.15.28:0/237009894 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0558147bf0 con 0x7f05580a5970 2024-04-25T01:35:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.413+0000 7f0565025640 1 --2- 172.21.15.28:0/237009894 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05580a5970 0x7f05580a5e20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f056002d700 tx=0x7f056006e4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:18.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.413+0000 7f055dffb640 1 -- 172.21.15.28:0/237009894 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0560070040 con 0x7f05580a5970 2024-04-25T01:35:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.414+0000 7f0565826640 1 -- 172.21.15.28:0/237009894 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0558147e80 con 0x7f05580a5970 2024-04-25T01:35:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.414+0000 7f0565826640 1 -- 172.21.15.28:0/237009894 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05581482e0 con 0x7f05580a5970 2024-04-25T01:35:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.415+0000 7f055dffb640 1 -- 172.21.15.28:0/237009894 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f056008a070 con 0x7f05580a5970 2024-04-25T01:35:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.415+0000 7f055dffb640 1 -- 172.21.15.28:0/237009894 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f056008da20 con 0x7f05580a5970 2024-04-25T01:35:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.424+0000 7f055dffb640 1 -- 172.21.15.28:0/237009894 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f056008d370 con 0x7f05580a5970 2024-04-25T01:35:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.426+0000 7f0569793740 1 Processor -- start 2024-04-25T01:35:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.426+0000 7f0569793740 1 -- start start 2024-04-25T01:35:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.427+0000 7f0569793740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da0720 0x564c94da0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.427+0000 7f0569793740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da1040 0x564c94dab1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.427+0000 7f0569793740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c94dab710 0x564c94dadb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.427+0000 7f0569793740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564c94d05670 con 0x564c94da0720 2024-04-25T01:35:18.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.428+0000 7f0569793740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564c94cad970 con 0x564c94da1040 2024-04-25T01:35:18.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.428+0000 7f0569793740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x564c94d01630 con 0x564c94dab710 2024-04-25T01:35:18.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.428+0000 7f0564824640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da1040 0x564c94dab1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:18.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.428+0000 7f0564824640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da1040 0x564c94dab1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:44516/0 (socket says 172.21.15.28:44516) 2024-04-25T01:35:18.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.428+0000 7f0564824640 1 -- 172.21.15.28:0/2666853094 learned_addr learned my addr 172.21.15.28:0/2666853094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:18.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.428+0000 7f055ffff640 1 --2- 172.21.15.28:0/2666853094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da0720 0x564c94da0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:18.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.429+0000 7f055dffb640 1 --2- 172.21.15.28:0/237009894 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f05500a2e40 0x7f05500a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.429+0000 7f0565025640 1 --2- 172.21.15.28:0/237009894 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f05500a2e40 0x7f05500a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.430+0000 7f055ffff640 1 -- 172.21.15.28:0/2666853094 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c94dab710 msgr2=0x564c94dadb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.430+0000 7f055ffff640 1 --2- 172.21.15.28:0/2666853094 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c94dab710 0x564c94dadb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.430+0000 7f055ffff640 1 -- 172.21.15.28:0/2666853094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da1040 msgr2=0x564c94dab1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.430+0000 7f055ffff640 1 --2- 172.21.15.28:0/2666853094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da1040 0x564c94dab1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.430+0000 7f055ffff640 1 -- 172.21.15.28:0/2666853094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c94cd1400 con 0x564c94da0720 2024-04-25T01:35:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.430+0000 7f0565025640 1 --2- 172.21.15.28:0/237009894 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f05500a2e40 0x7f05500a5300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f056010c1f0 tx=0x7f056010bbb0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.430+0000 7f055dffb640 1 -- 172.21.15.28:0/237009894 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f0560143590 con 0x7f05580a5970 2024-04-25T01:35:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.430+0000 7f0564824640 1 --2- 172.21.15.28:0/2666853094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da1040 0x564c94dab1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:35:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.431+0000 7f055ffff640 1 --2- 172.21.15.28:0/2666853094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da0720 0x564c94da0b00 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f054c008000 tx=0x7f054c030340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44b810ae0dbf2962 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:18.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.439+0000 7f054affd640 1 -- 172.21.15.28:0/2666853094 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054c037040 con 0x564c94da0720 2024-04-25T01:35:18.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.439+0000 7f054affd640 1 -- 172.21.15.28:0/2666853094 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f054c030e50 con 0x564c94da0720 2024-04-25T01:35:18.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.439+0000 7f054affd640 1 -- 172.21.15.28:0/2666853094 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054c005280 con 0x564c94da0720 2024-04-25T01:35:18.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.440+0000 7f0569793740 1 -- 172.21.15.28:0/2666853094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da0720 msgr2=0x564c94da0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:18.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.440+0000 7f0569793740 1 --2- 172.21.15.28:0/2666853094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da0720 0x564c94da0b00 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f054c008000 tx=0x7f054c030340 comp rx=0 tx=0).stop 2024-04-25T01:35:18.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.445+0000 7f0569793740 1 -- 172.21.15.28:0/2666853094 shutdown_connections 2024-04-25T01:35:18.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.445+0000 7f0569793740 1 --2- 172.21.15.28:0/2666853094 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c94dab710 0x564c94dadb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.445+0000 7f0569793740 1 --2- 172.21.15.28:0/2666853094 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da1040 0x564c94dab1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.445+0000 7f0569793740 1 --2- 172.21.15.28:0/2666853094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da0720 0x564c94da0b00 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.445+0000 7f0569793740 1 -- 172.21.15.28:0/2666853094 >> 172.21.15.28:0/2666853094 conn(0x564c94d9ca40 msgr2=0x564c94d9ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:18.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.445+0000 7f0569793740 1 -- 172.21.15.28:0/2666853094 shutdown_connections 2024-04-25T01:35:18.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.445+0000 7f0569793740 1 -- 172.21.15.28:0/2666853094 wait complete. 2024-04-25T01:35:18.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.446+0000 7f0569793740 1 -- start start 2024-04-25T01:35:18.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.446+0000 7f0569793740 1 Processor -- start 2024-04-25T01:35:18.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.447+0000 7f0569793740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da0720 0x564c94dab0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.447+0000 7f0569793740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da1040 0x564c94da83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.447+0000 7f0569793740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c94dab710 0x564c94da89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.447+0000 7f0569793740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564c94dae500 con 0x564c94da1040 2024-04-25T01:35:18.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.447+0000 7f0569793740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564c94cc1050 con 0x564c94da0720 2024-04-25T01:35:18.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.447+0000 7f0569793740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x564c94dae680 con 0x564c94dab710 2024-04-25T01:35:18.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.447+0000 7f0569793740 10 client.0 _ll_register_callbacks cb 0x564c94a18be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:18.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.449+0000 7f0564824640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da1040 0x564c94da83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:18.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.449+0000 7f0564824640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da1040 0x564c94da83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47562/0 (socket says 172.21.15.28:47562) 2024-04-25T01:35:18.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.449+0000 7f0564824640 1 -- 172.21.15.28:0/1680702337 learned_addr learned my addr 172.21.15.28:0/1680702337 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:18.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.450+0000 7f055ffff640 1 --2- 172.21.15.28:0/1680702337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da0720 0x564c94dab0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:18.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.450+0000 7f055ffff640 1 --2- 172.21.15.28:0/1680702337 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c94dab710 0x564c94da89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:18.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.450+0000 7f0564824640 1 -- 172.21.15.28:0/1680702337 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c94dab710 msgr2=0x564c94da89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:18.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.450+0000 7f0564824640 1 --2- 172.21.15.28:0/1680702337 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c94dab710 0x564c94da89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.451+0000 7f0564824640 1 -- 172.21.15.28:0/1680702337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da0720 msgr2=0x564c94dab0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:18.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.451+0000 7f0564824640 1 --2- 172.21.15.28:0/1680702337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da0720 0x564c94dab0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.451+0000 7f0564824640 1 -- 172.21.15.28:0/1680702337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c94c92dc0 con 0x564c94da1040 2024-04-25T01:35:18.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.452+0000 7f0564824640 1 --2- 172.21.15.28:0/1680702337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da1040 0x564c94da83c0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f055400bac0 tx=0x7f0554004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b01b135d42bf335c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:18.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.452+0000 7f053ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:18.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.456+0000 7f053ffff640 1 -- 172.21.15.28:0/1680702337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0554011440 con 0x564c94da1040 2024-04-25T01:35:18.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.456+0000 7f0569793740 1 -- 172.21.15.28:0/1680702337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f054c030e50 con 0x564c94da1040 2024-04-25T01:35:18.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.456+0000 7f053ffff640 1 -- 172.21.15.28:0/1680702337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f05540115e0 con 0x564c94da1040 2024-04-25T01:35:18.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.459+0000 7f053ffff640 1 -- 172.21.15.28:0/1680702337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f05540117e0 con 0x564c94da1040 2024-04-25T01:35:18.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.459+0000 7f0569793740 10 client.16359 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:18.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.459+0000 7f0569793740 1 -- 172.21.15.28:0/1680702337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564c94a193f0 con 0x564c94da1040 2024-04-25T01:35:18.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.461+0000 7f053ffff640 1 -- 172.21.15.28:0/1680702337 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f054c030e50 con 0x564c94da1040 2024-04-25T01:35:18.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.462+0000 7f053ffff640 1 -- 172.21.15.28:0/1680702337 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f055401baa0 con 0x564c94da1040 2024-04-25T01:35:18.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.462+0000 7f0569793740 10 client.16359 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:18.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.462+0000 7f0569793740 10 client.16359 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:18.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.462+0000 7f0569793740 20 client.16359 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:18.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.462+0000 7f0569793740 1 --2- 172.21.15.28:0/1680702337 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564c94dbbd50 0x564c94dbe1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.462+0000 7f0569793740 4 client.16359 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:18.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.462+0000 7f0569793740 1 -- 172.21.15.28:0/1680702337 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c94d12bb0 con 0x564c94dbbd50 2024-04-25T01:35:18.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.462+0000 7f055ffff640 1 --2- 172.21.15.28:0/1680702337 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564c94dbbd50 0x564c94dbe1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:18.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.464+0000 7f053ffff640 10 client.16359 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:18.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.464+0000 7f055ffff640 1 --2- 172.21.15.28:0/1680702337 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564c94dbbd50 0x564c94dbe1f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cf90da3b1faf7e8 server_cookie=b88c7867ba3ad08e in_seq=0 out_seq=0 2024-04-25T01:35:18.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.465+0000 7f053ffff640 1 -- 172.21.15.28:0/1680702337 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564c94d12bb0 con 0x564c94dbbd50 2024-04-25T01:35:18.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.465+0000 7f053ffff640 10 client.16359 handle_command_reply: tid=0 2024-04-25T01:35:18.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.465+0000 7f0569793740 1 client.16359 shutdown 2024-04-25T01:35:18.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.469+0000 7f0569793740 1 -- 172.21.15.28:0/1680702337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da1040 msgr2=0x564c94da83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:18.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.469+0000 7f0569793740 1 --2- 172.21.15.28:0/1680702337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da1040 0x564c94da83c0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f055400bac0 tx=0x7f0554004600 comp rx=0 tx=0).stop 2024-04-25T01:35:18.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.470+0000 7f0569793740 1 -- 172.21.15.28:0/1680702337 shutdown_connections 2024-04-25T01:35:18.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.470+0000 7f0569793740 1 --2- 172.21.15.28:0/1680702337 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564c94dbbd50 0x564c94dbe1f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.470+0000 7f0569793740 1 --2- 172.21.15.28:0/1680702337 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c94dab710 0x564c94da89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.470+0000 7f0569793740 1 --2- 172.21.15.28:0/1680702337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da1040 0x564c94da83c0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.470+0000 7f0569793740 1 --2- 172.21.15.28:0/1680702337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da0720 0x564c94dab0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.470+0000 7f0569793740 1 -- 172.21.15.28:0/1680702337 >> 172.21.15.28:0/1680702337 conn(0x564c94d9ca40 msgr2=0x564c94da2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:18.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.470+0000 7f053ffff640 0 client.16359 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.471+0000 7f0569793740 1 -- 172.21.15.28:0/1680702337 shutdown_connections 2024-04-25T01:35:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.471+0000 7f0569793740 1 -- 172.21.15.28:0/1680702337 wait complete. 2024-04-25T01:35:18.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.471+0000 7f0569793740 20 client.16359 trim_cache size 0 max 16384 2024-04-25T01:35:18.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:18 smithi130 ceph-mon[32702]: pgmap v71: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 675 MiB/s wr, 344 op/s 2024-04-25T01:35:18.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.501+0000 7f0569793740 1 Processor -- start 2024-04-25T01:35:18.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.501+0000 7f0569793740 1 -- start start 2024-04-25T01:35:18.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.502+0000 7f0569793740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da0720 0x564c94d9c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.502+0000 7f0569793740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da1040 0x564c94daac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.502+0000 7f0569793740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c94dab710 0x564c94da3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.503+0000 7f0569793740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f054400fec0 con 0x564c94da0720 2024-04-25T01:35:18.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.503+0000 7f055ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da0720 0x564c94d9c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:18.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.503+0000 7f055ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da0720 0x564c94d9c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47564/0 (socket says 172.21.15.28:47564) 2024-04-25T01:35:18.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.503+0000 7f055ffff640 1 -- 172.21.15.28:0/1715975200 learned_addr learned my addr 172.21.15.28:0/1715975200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:18.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.503+0000 7f0564824640 1 --2- 172.21.15.28:0/1715975200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da1040 0x564c94daac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:18.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.503+0000 7f0569793740 1 -- 172.21.15.28:0/1715975200 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0544011a00 con 0x564c94da1040 2024-04-25T01:35:18.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.503+0000 7f0569793740 1 -- 172.21.15.28:0/1715975200 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0544010590 con 0x564c94dab710 2024-04-25T01:35:18.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.503+0000 7f055ffff640 1 --2- 172.21.15.28:0/1715975200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c94dab710 0x564c94da3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:18.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.504+0000 7f055ffff640 1 -- 172.21.15.28:0/1715975200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c94dab710 msgr2=0x564c94da3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:18.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.504+0000 7f055ffff640 1 --2- 172.21.15.28:0/1715975200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c94dab710 0x564c94da3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.504+0000 7f055ffff640 1 -- 172.21.15.28:0/1715975200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da1040 msgr2=0x564c94daac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:18.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.504+0000 7f055ffff640 1 --2- 172.21.15.28:0/1715975200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da1040 0x564c94daac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.504+0000 7f055ffff640 1 -- 172.21.15.28:0/1715975200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c94ae37d0 con 0x564c94da0720 2024-04-25T01:35:18.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.504+0000 7f055ffff640 1 --2- 172.21.15.28:0/1715975200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da0720 0x564c94d9c340 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f054c009250 tx=0x7f054c005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df865bca5de3490 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:18.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:18 smithi032 ceph-mon[32461]: pgmap v71: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 675 MiB/s wr, 344 op/s 2024-04-25T01:35:18.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.505+0000 7f054a7fc640 1 -- 172.21.15.28:0/1715975200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054c037040 con 0x564c94da0720 2024-04-25T01:35:18.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.505+0000 7f054a7fc640 1 -- 172.21.15.28:0/1715975200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f054c030910 con 0x564c94da0720 2024-04-25T01:35:18.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.505+0000 7f054a7fc640 1 -- 172.21.15.28:0/1715975200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054c0035f0 con 0x564c94da0720 2024-04-25T01:35:18.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.507+0000 7f0569793740 1 -- 172.21.15.28:0/1715975200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da0720 msgr2=0x564c94d9c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:18.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.507+0000 7f0569793740 1 --2- 172.21.15.28:0/1715975200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da0720 0x564c94d9c340 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f054c009250 tx=0x7f054c005c90 comp rx=0 tx=0).stop 2024-04-25T01:35:18.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.510+0000 7f0569793740 1 -- 172.21.15.28:0/1715975200 shutdown_connections 2024-04-25T01:35:18.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.510+0000 7f0569793740 1 --2- 172.21.15.28:0/1715975200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c94dab710 0x564c94da3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.510+0000 7f0569793740 1 --2- 172.21.15.28:0/1715975200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da1040 0x564c94daac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.510+0000 7f0569793740 1 --2- 172.21.15.28:0/1715975200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da0720 0x564c94d9c340 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.510+0000 7f0569793740 1 -- 172.21.15.28:0/1715975200 >> 172.21.15.28:0/1715975200 conn(0x564c94d9ca40 msgr2=0x564c94da3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:18.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.510+0000 7f0569793740 1 -- 172.21.15.28:0/1715975200 shutdown_connections 2024-04-25T01:35:18.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.510+0000 7f0569793740 1 -- 172.21.15.28:0/1715975200 wait complete. 2024-04-25T01:35:18.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.510+0000 7f0569793740 1 -- start start 2024-04-25T01:35:18.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.511+0000 7f0569793740 1 Processor -- start 2024-04-25T01:35:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.511+0000 7f0569793740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da0720 0x564c94db3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.511+0000 7f0569793740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da1040 0x564c94da4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.511+0000 7f0569793740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c94dab710 0x564c94da5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.511+0000 7f0569793740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0544011330 con 0x564c94da1040 2024-04-25T01:35:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.511+0000 7f0569793740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0544010c60 con 0x564c94da0720 2024-04-25T01:35:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.511+0000 7f0569793740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f054400dea0 con 0x564c94dab710 2024-04-25T01:35:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.511+0000 7f0569793740 10 client.0 _ll_register_callbacks cb 0x7f05540040c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.512+0000 7f0564824640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da1040 0x564c94da4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:18.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.512+0000 7f0564824640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da1040 0x564c94da4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47568/0 (socket says 172.21.15.28:47568) 2024-04-25T01:35:18.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.512+0000 7f0564824640 1 -- 172.21.15.28:0/159426363 learned_addr learned my addr 172.21.15.28:0/159426363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:18.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.513+0000 7f0564824640 1 -- 172.21.15.28:0/159426363 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c94dab710 msgr2=0x564c94da5650 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:35:18.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.513+0000 7f0564824640 1 --2- 172.21.15.28:0/159426363 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c94dab710 0x564c94da5650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.513+0000 7f0564824640 1 -- 172.21.15.28:0/159426363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da0720 msgr2=0x564c94db3e00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:35:18.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.513+0000 7f0564824640 1 --2- 172.21.15.28:0/159426363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da0720 0x564c94db3e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.513+0000 7f0564824640 1 -- 172.21.15.28:0/159426363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c94e21370 con 0x564c94da1040 2024-04-25T01:35:18.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.513+0000 7f0564824640 1 --2- 172.21.15.28:0/159426363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da1040 0x564c94da4f10 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f05540042c0 tx=0x7f0554011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3affd9e6fa76f197 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:18.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.514+0000 7f053ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:18.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.515+0000 7f053ffff640 1 -- 172.21.15.28:0/159426363 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0554005030 con 0x564c94da1040 2024-04-25T01:35:18.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.515+0000 7f0569793740 1 -- 172.21.15.28:0/159426363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f054c030910 con 0x564c94da1040 2024-04-25T01:35:18.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.515+0000 7f053ffff640 1 -- 172.21.15.28:0/159426363 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f05540051d0 con 0x564c94da1040 2024-04-25T01:35:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.516+0000 7f053ffff640 1 -- 172.21.15.28:0/159426363 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f05540053d0 con 0x564c94da1040 2024-04-25T01:35:18.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.518+0000 7f0569793740 10 client.16365 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:18.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.518+0000 7f0569793740 1 -- 172.21.15.28:0/159426363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564c94e22570 con 0x564c94da1040 2024-04-25T01:35:18.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.523+0000 7f053ffff640 1 -- 172.21.15.28:0/159426363 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f054c030910 con 0x564c94da1040 2024-04-25T01:35:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.523+0000 7f053ffff640 1 -- 172.21.15.28:0/159426363 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f055401b660 con 0x564c94da1040 2024-04-25T01:35:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.523+0000 7f0569793740 10 client.16365 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.523+0000 7f0569793740 10 client.16365 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.523+0000 7f0569793740 20 client.16365 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.523+0000 7f0569793740 1 --2- 172.21.15.28:0/159426363 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564c94e22cd0 0x564c94e250f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.523+0000 7f0569793740 4 client.16365 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:35:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.523+0000 7f0569793740 1 -- 172.21.15.28:0/159426363 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564c94d0cf00 con 0x564c94e22cd0 2024-04-25T01:35:18.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.530+0000 7f055ffff640 1 --2- 172.21.15.28:0/159426363 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564c94e22cd0 0x564c94e250f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:18.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.553+0000 7f055ffff640 1 --2- 172.21.15.28:0/159426363 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564c94e22cd0 0x564c94e250f0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e201eb2e4fea3f42 server_cookie=70a0cfe458dfc153 in_seq=0 out_seq=0 2024-04-25T01:35:18.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.554+0000 7f053ffff640 10 client.16365 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:18.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.558+0000 7f053ffff640 1 -- 172.21.15.28:0/159426363 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564c94d0cf00 con 0x564c94e22cd0 2024-04-25T01:35:18.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.558+0000 7f053ffff640 10 client.16365 handle_command_reply: tid=0 2024-04-25T01:35:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.558+0000 7f0569793740 1 client.16365 shutdown 2024-04-25T01:35:18.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.565+0000 7f0569793740 1 -- 172.21.15.28:0/159426363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da1040 msgr2=0x564c94da4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:18.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.565+0000 7f0569793740 1 --2- 172.21.15.28:0/159426363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da1040 0x564c94da4f10 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f05540042c0 tx=0x7f0554011bb0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.569+0000 7f0569793740 1 -- 172.21.15.28:0/159426363 shutdown_connections 2024-04-25T01:35:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.569+0000 7f0569793740 1 --2- 172.21.15.28:0/159426363 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564c94e22cd0 0x564c94e250f0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.569+0000 7f0569793740 1 --2- 172.21.15.28:0/159426363 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c94dab710 0x564c94da5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.569+0000 7f0569793740 1 --2- 172.21.15.28:0/159426363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c94da1040 0x564c94da4f10 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.569+0000 7f0569793740 1 --2- 172.21.15.28:0/159426363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c94da0720 0x564c94db3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.569+0000 7f0569793740 1 -- 172.21.15.28:0/159426363 >> 172.21.15.28:0/159426363 conn(0x564c94d9ca40 msgr2=0x564c94da2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.569+0000 7f053ffff640 0 client.16365 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.573+0000 7f0569793740 1 -- 172.21.15.28:0/159426363 shutdown_connections 2024-04-25T01:35:18.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.575+0000 7f0569793740 1 -- 172.21.15.28:0/159426363 wait complete. 2024-04-25T01:35:18.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.578+0000 7f0569793740 20 client.16365 trim_cache size 0 max 16384 2024-04-25T01:35:18.558 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:35:18.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.585+0000 7f0565826640 1 -- 172.21.15.28:0/237009894 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f05500a2e40 msgr2=0x7f05500a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:18.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.585+0000 7f0565826640 1 --2- 172.21.15.28:0/237009894 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f05500a2e40 0x7f05500a5300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f056010c1f0 tx=0x7f056010bbb0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.585+0000 7f0565826640 1 -- 172.21.15.28:0/237009894 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05580a5970 msgr2=0x7f05580a5e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:18.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.585+0000 7f0565826640 1 --2- 172.21.15.28:0/237009894 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05580a5970 0x7f05580a5e20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f056002d700 tx=0x7f056006e4d0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.586+0000 7f0565826640 1 -- 172.21.15.28:0/237009894 shutdown_connections 2024-04-25T01:35:18.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.586+0000 7f0565826640 1 --2- 172.21.15.28:0/237009894 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f05500a2e40 0x7f05500a5300 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.586+0000 7f0565826640 1 --2- 172.21.15.28:0/237009894 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f05580a5970 0x7f05580a5e20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.586+0000 7f0565826640 1 --2- 172.21.15.28:0/237009894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055809acf0 0x7f05580a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.586+0000 7f0565826640 1 --2- 172.21.15.28:0/237009894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f05580986e0 0x7f05580a9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:18.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.586+0000 7f0565826640 1 -- 172.21.15.28:0/237009894 >> 172.21.15.28:0/237009894 conn(0x7f055800a740 msgr2=0x7f05580976c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:18.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.588+0000 7f0565826640 1 -- 172.21.15.28:0/237009894 shutdown_connections 2024-04-25T01:35:18.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:18.588+0000 7f0565826640 1 -- 172.21.15.28:0/237009894 wait complete. 2024-04-25T01:35:19.595 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:35:19.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:35:19.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:35:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.881+0000 7f2f6a61c640 1 -- 172.21.15.28:0/3174023533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f64112730 msgr2=0x7f2f64114b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.881+0000 7f2f6a61c640 1 --2- 172.21.15.28:0/3174023533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f64112730 0x7f2f64114b20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c009be0 tx=0x7f2f5c02a770 comp rx=0 tx=0).stop 2024-04-25T01:35:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.887+0000 7f2f6a61c640 1 -- 172.21.15.28:0/3174023533 shutdown_connections 2024-04-25T01:35:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.887+0000 7f2f6a61c640 1 --2- 172.21.15.28:0/3174023533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f64112730 0x7f2f64114b20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.887+0000 7f2f6a61c640 1 --2- 172.21.15.28:0/3174023533 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f64074400 0x7f2f641121f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.887+0000 7f2f6a61c640 1 --2- 172.21.15.28:0/3174023533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f64073ae0 0x7f2f64073ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.887+0000 7f2f6a61c640 1 -- 172.21.15.28:0/3174023533 >> 172.21.15.28:0/3174023533 conn(0x7f2f64078b40 msgr2=0x7f2f64078f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.892+0000 7f2f6a61c640 1 -- 172.21.15.28:0/3174023533 shutdown_connections 2024-04-25T01:35:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.893+0000 7f2f6a61c640 1 -- 172.21.15.28:0/3174023533 wait complete. 2024-04-25T01:35:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.894+0000 7f2f6a61c640 1 Processor -- start 2024-04-25T01:35:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.895+0000 7f2f6a61c640 1 -- start start 2024-04-25T01:35:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.896+0000 7f2f6a61c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f64073ae0 0x7f2f64073060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.896+0000 7f2f6a61c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f64074400 0x7f2f6406c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.896+0000 7f2f6a61c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f64112730 0x7f2f6406c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.896+0000 7f2f6a61c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f641180d0 con 0x7f2f64074400 2024-04-25T01:35:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.896+0000 7f2f6a61c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2f64117f50 con 0x7f2f64073ae0 2024-04-25T01:35:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.896+0000 7f2f6a61c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2f64118250 con 0x7f2f64112730 2024-04-25T01:35:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.897+0000 7f2f68bb9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f64112730 0x7f2f6406c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.897+0000 7f2f68bb9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f64112730 0x7f2f6406c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.28:39494/0 (socket says 172.21.15.28:39494) 2024-04-25T01:35:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.897+0000 7f2f68bb9640 1 -- 172.21.15.28:0/2710639990 learned_addr learned my addr 172.21.15.28:0/2710639990 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.898+0000 7f2f63fff640 1 --2- 172.21.15.28:0/2710639990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f64073ae0 0x7f2f64073060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.898+0000 7f2f637fe640 1 --2- 172.21.15.28:0/2710639990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f64074400 0x7f2f6406c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.898+0000 7f2f68bb9640 1 -- 172.21.15.28:0/2710639990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f64073ae0 msgr2=0x7f2f64073060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.898+0000 7f2f68bb9640 1 --2- 172.21.15.28:0/2710639990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f64073ae0 0x7f2f64073060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.898+0000 7f2f68bb9640 1 -- 172.21.15.28:0/2710639990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f64074400 msgr2=0x7f2f6406c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.898+0000 7f2f68bb9640 1 --2- 172.21.15.28:0/2710639990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f64074400 0x7f2f6406c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.898+0000 7f2f68bb9640 1 -- 172.21.15.28:0/2710639990 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f6406cc80 con 0x7f2f64112730 2024-04-25T01:35:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.898+0000 7f2f637fe640 1 --2- 172.21.15.28:0/2710639990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f64074400 0x7f2f6406c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.899+0000 7f2f68bb9640 1 --2- 172.21.15.28:0/2710639990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f64112730 0x7f2f6406c6c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c006010 tx=0x7f2f5c003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:19.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.900+0000 7f2f617fa640 1 -- 172.21.15.28:0/2710639990 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c041020 con 0x7f2f64112730 2024-04-25T01:35:19.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.901+0000 7f2f617fa640 1 -- 172.21.15.28:0/2710639990 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f5c030880 con 0x7f2f64112730 2024-04-25T01:35:19.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.901+0000 7f2f617fa640 1 -- 172.21.15.28:0/2710639990 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c036040 con 0x7f2f64112730 2024-04-25T01:35:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.901+0000 7f2f6a61c640 1 -- 172.21.15.28:0/2710639990 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f6406cf10 con 0x7f2f64112730 2024-04-25T01:35:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.901+0000 7f2f6a61c640 1 -- 172.21.15.28:0/2710639990 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f64070dc0 con 0x7f2f64112730 2024-04-25T01:35:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.907+0000 7f2f6d409740 1 Processor -- start 2024-04-25T01:35:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.907+0000 7f2f6d409740 1 -- start start 2024-04-25T01:35:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.908+0000 7f2f6d409740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bc0980 0x558e07bbecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.908+0000 7f2f6d409740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e07bbf290 0x558e07bbf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.908+0000 7f2f6d409740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 0x558e07bcc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.908+0000 7f2f6d409740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558e07b232c0 con 0x558e07bca010 2024-04-25T01:35:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.908+0000 7f2f6d409740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558e07acb910 con 0x558e07bc0980 2024-04-25T01:35:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.908+0000 7f2f6d409740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x558e07b1f280 con 0x558e07bbf290 2024-04-25T01:35:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.908+0000 7f2f617fa640 1 -- 172.21.15.28:0/2710639990 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2f5c044020 con 0x7f2f64112730 2024-04-25T01:35:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.909+0000 7f2f617fa640 1 --2- 172.21.15.28:0/2710639990 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f2f4c0a2ed0 0x7f2f4c0a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.910+0000 7f2f617fa640 1 -- 172.21.15.28:0/2710639990 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f2f5c0f0050 con 0x7f2f64112730 2024-04-25T01:35:19.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.910+0000 7f2f68bb9640 1 --2- 172.21.15.28:0/2710639990 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f2f4c0a2ed0 0x7f2f4c0a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.910+0000 7f2f637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 0x558e07bcc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.912+0000 7f2f68bb9640 1 --2- 172.21.15.28:0/2710639990 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f2f4c0a2ed0 0x7f2f4c0a5390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c0b8ba0 tx=0x7f2f5c0098c0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.914+0000 7f2f63fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e07bbf290 0x558e07bbf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.914+0000 7f2f637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 0x558e07bcc400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47596/0 (socket says 172.21.15.28:47596) 2024-04-25T01:35:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.914+0000 7f2f637fe640 1 -- 172.21.15.28:0/1411784827 learned_addr learned my addr 172.21.15.28:0/1411784827 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.914+0000 7f2f637fe640 1 --2- 172.21.15.28:0/1411784827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bc0980 0x558e07bbecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.921+0000 7f2f637fe640 1 -- 172.21.15.28:0/1411784827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e07bbf290 msgr2=0x558e07bbf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.921+0000 7f2f637fe640 1 --2- 172.21.15.28:0/1411784827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e07bbf290 0x558e07bbf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.921+0000 7f2f637fe640 1 -- 172.21.15.28:0/1411784827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bc0980 msgr2=0x558e07bbecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.921+0000 7f2f637fe640 1 --2- 172.21.15.28:0/1411784827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bc0980 0x558e07bbecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.921+0000 7f2f637fe640 1 -- 172.21.15.28:0/1411784827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e07adcfb0 con 0x558e07bca010 2024-04-25T01:35:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.922+0000 7f2f637fe640 1 --2- 172.21.15.28:0/1411784827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 0x558e07bcc400 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f2f580158d0 tx=0x7f2f58009370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1340af2bacff55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.922+0000 7f2f4a7fc640 1 -- 172.21.15.28:0/1411784827 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f58005020 con 0x558e07bca010 2024-04-25T01:35:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.923+0000 7f2f4a7fc640 1 -- 172.21.15.28:0/1411784827 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f580047a0 con 0x558e07bca010 2024-04-25T01:35:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.923+0000 7f2f4a7fc640 1 -- 172.21.15.28:0/1411784827 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f58005020 con 0x558e07bca010 2024-04-25T01:35:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.925+0000 7f2f6d409740 1 -- 172.21.15.28:0/1411784827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 msgr2=0x558e07bcc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:19.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.925+0000 7f2f6d409740 1 --2- 172.21.15.28:0/1411784827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 0x558e07bcc400 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f2f580158d0 tx=0x7f2f58009370 comp rx=0 tx=0).stop 2024-04-25T01:35:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.926+0000 7f2f6d409740 1 -- 172.21.15.28:0/1411784827 shutdown_connections 2024-04-25T01:35:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.926+0000 7f2f6d409740 1 --2- 172.21.15.28:0/1411784827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 0x558e07bcc400 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.926+0000 7f2f6d409740 1 --2- 172.21.15.28:0/1411784827 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e07bbf290 0x558e07bbf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.926+0000 7f2f6d409740 1 --2- 172.21.15.28:0/1411784827 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bc0980 0x558e07bbecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.926+0000 7f2f6d409740 1 -- 172.21.15.28:0/1411784827 >> 172.21.15.28:0/1411784827 conn(0x558e07b87950 msgr2=0x558e07bbb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:19.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.928+0000 7f2f6d409740 1 -- 172.21.15.28:0/1411784827 shutdown_connections 2024-04-25T01:35:19.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.928+0000 7f2f6d409740 1 -- 172.21.15.28:0/1411784827 wait complete. 2024-04-25T01:35:19.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.929+0000 7f2f6d409740 1 -- start start 2024-04-25T01:35:19.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.929+0000 7f2f6d409740 1 Processor -- start 2024-04-25T01:35:19.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.930+0000 7f2f6d409740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bbf290 0x558e07bc6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:19.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.930+0000 7f2f6d409740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e07bc0980 0x558e07bc74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:19.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.930+0000 7f2f6d409740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 0x558e07bc79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.931+0000 7f2f637fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bbf290 0x558e07bc6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.931+0000 7f2f637fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bbf290 0x558e07bc6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:44584/0 (socket says 172.21.15.28:44584) 2024-04-25T01:35:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.931+0000 7f2f637fe640 1 -- 172.21.15.28:0/3262760004 learned_addr learned my addr 172.21.15.28:0/3262760004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.931+0000 7f2f6d409740 1 -- 172.21.15.28:0/3262760004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558e07bccea0 con 0x558e07bca010 2024-04-25T01:35:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.931+0000 7f2f6d409740 1 -- 172.21.15.28:0/3262760004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558e07adee90 con 0x558e07bbf290 2024-04-25T01:35:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.931+0000 7f2f6d409740 1 -- 172.21.15.28:0/3262760004 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x558e07bcd020 con 0x558e07bc0980 2024-04-25T01:35:19.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.931+0000 7f2f637fe640 1 --2- 172.21.15.28:0/3262760004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 0x558e07bc79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:19.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.932+0000 7f2f6d409740 10 client.0 _ll_register_callbacks cb 0x558e07b1a970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:19.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.932+0000 7f2f637fe640 1 -- 172.21.15.28:0/3262760004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e07bc0980 msgr2=0x558e07bc74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:19.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.932+0000 7f2f637fe640 1 --2- 172.21.15.28:0/3262760004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e07bc0980 0x558e07bc74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.932+0000 7f2f637fe640 1 -- 172.21.15.28:0/3262760004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bbf290 msgr2=0x558e07bc6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:19.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.932+0000 7f2f637fe640 1 --2- 172.21.15.28:0/3262760004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bbf290 0x558e07bc6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.932+0000 7f2f637fe640 1 -- 172.21.15.28:0/3262760004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e078ff7e0 con 0x558e07bca010 2024-04-25T01:35:19.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.932+0000 7f2f637fe640 1 --2- 172.21.15.28:0/3262760004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 0x558e07bc79e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f2f580153f0 tx=0x7f2f58005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=defccbdad1677386 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:19.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.932+0000 7f2f437fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:19.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.933+0000 7f2f437fe640 1 -- 172.21.15.28:0/3262760004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f58005020 con 0x558e07bca010 2024-04-25T01:35:19.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.933+0000 7f2f6d409740 1 -- 172.21.15.28:0/3262760004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f580047a0 con 0x558e07bca010 2024-04-25T01:35:19.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.933+0000 7f2f437fe640 1 -- 172.21.15.28:0/3262760004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f2f580150f0 con 0x558e07bca010 2024-04-25T01:35:19.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.935+0000 7f2f437fe640 1 -- 172.21.15.28:0/3262760004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2f58004b60 con 0x558e07bca010 2024-04-25T01:35:19.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.935+0000 7f2f6d409740 10 client.16374 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:19.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.935+0000 7f2f6d409740 1 -- 172.21.15.28:0/3262760004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558e0795ff70 con 0x558e07bca010 2024-04-25T01:35:19.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.936+0000 7f2f437fe640 1 -- 172.21.15.28:0/3262760004 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f580047a0 con 0x558e07bca010 2024-04-25T01:35:19.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.936+0000 7f2f437fe640 1 -- 172.21.15.28:0/3262760004 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2f5800c030 con 0x558e07bca010 2024-04-25T01:35:19.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.937+0000 7f2f6d409740 10 client.16374 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:19.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.937+0000 7f2f6d409740 10 client.16374 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:19.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.937+0000 7f2f6d409740 20 client.16374 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:19.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.937+0000 7f2f6d409740 1 --2- 172.21.15.28:0/3262760004 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558e07bda020 0x558e07bdc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:19.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.937+0000 7f2f6d409740 4 client.16374 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:19.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.937+0000 7f2f6d409740 1 -- 172.21.15.28:0/3262760004 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e07af75a0 con 0x558e07bda020 2024-04-25T01:35:19.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.939+0000 7f2f63fff640 1 --2- 172.21.15.28:0/3262760004 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558e07bda020 0x558e07bdc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:19.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.941+0000 7f2f63fff640 1 --2- 172.21.15.28:0/3262760004 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558e07bda020 0x558e07bdc410 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fb51998d6731367 server_cookie=173aa8bc3b4d9578 in_seq=0 out_seq=0 2024-04-25T01:35:19.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.943+0000 7f2f437fe640 10 client.16374 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:19.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.944+0000 7f2f437fe640 1 -- 172.21.15.28:0/3262760004 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558e07af75a0 con 0x558e07bda020 2024-04-25T01:35:19.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.944+0000 7f2f437fe640 10 client.16374 handle_command_reply: tid=0 2024-04-25T01:35:19.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.944+0000 7f2f6d409740 1 client.16374 shutdown 2024-04-25T01:35:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.944+0000 7f2f6d409740 1 -- 172.21.15.28:0/3262760004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 msgr2=0x558e07bc79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.944+0000 7f2f6d409740 1 --2- 172.21.15.28:0/3262760004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 0x558e07bc79e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f2f580153f0 tx=0x7f2f58005fd0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.945+0000 7f2f6d409740 1 -- 172.21.15.28:0/3262760004 shutdown_connections 2024-04-25T01:35:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.945+0000 7f2f6d409740 1 --2- 172.21.15.28:0/3262760004 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558e07bda020 0x558e07bdc410 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.945+0000 7f2f6d409740 1 --2- 172.21.15.28:0/3262760004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 0x558e07bc79e0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.945+0000 7f2f6d409740 1 --2- 172.21.15.28:0/3262760004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e07bc0980 0x558e07bc74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.945+0000 7f2f6d409740 1 --2- 172.21.15.28:0/3262760004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bbf290 0x558e07bc6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.945+0000 7f2f6d409740 1 -- 172.21.15.28:0/3262760004 >> 172.21.15.28:0/3262760004 conn(0x558e07b87950 msgr2=0x558e07bc15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:19.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.945+0000 7f2f437fe640 0 client.16374 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:19.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.945+0000 7f2f6d409740 1 -- 172.21.15.28:0/3262760004 shutdown_connections 2024-04-25T01:35:19.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.945+0000 7f2f6d409740 1 -- 172.21.15.28:0/3262760004 wait complete. 2024-04-25T01:35:19.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.946+0000 7f2f6d409740 20 client.16374 trim_cache size 0 max 16384 2024-04-25T01:35:19.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.997+0000 7f2f6d409740 1 Processor -- start 2024-04-25T01:35:19.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.997+0000 7f2f6d409740 1 -- start start 2024-04-25T01:35:19.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.997+0000 7f2f6d409740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e07bbf290 0x558e07bba730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:19.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.997+0000 7f2f6d409740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bc0980 0x558e07bc7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:19.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.998+0000 7f2f6d409740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bca010 0x558e07bc1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:19.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.998+0000 7f2f6d409740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f641bd8b0 con 0x558e07bc0980 2024-04-25T01:35:19.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.998+0000 7f2f6d409740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2f641bf3f0 con 0x558e07bca010 2024-04-25T01:35:19.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.998+0000 7f2f6d409740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2f641bdf80 con 0x558e07bbf290 2024-04-25T01:35:19.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.998+0000 7f2f63fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bc0980 0x558e07bc7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:19.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.998+0000 7f2f63fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bc0980 0x558e07bc7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47622/0 (socket says 172.21.15.28:47622) 2024-04-25T01:35:19.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.998+0000 7f2f63fff640 1 -- 172.21.15.28:0/2248461274 learned_addr learned my addr 172.21.15.28:0/2248461274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:19.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.999+0000 7f2f63fff640 1 -- 172.21.15.28:0/2248461274 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e07bbf290 msgr2=0x558e07bba730 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:35:19.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.999+0000 7f2f63fff640 1 --2- 172.21.15.28:0/2248461274 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e07bbf290 0x558e07bba730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.999+0000 7f2f63fff640 1 -- 172.21.15.28:0/2248461274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bca010 msgr2=0x558e07bc1470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:35:19.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.999+0000 7f2f63fff640 1 --2- 172.21.15.28:0/2248461274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bca010 0x558e07bc1470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:19.999+0000 7f2f63fff640 1 -- 172.21.15.28:0/2248461274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e077f7170 con 0x558e07bc0980 2024-04-25T01:35:19.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.000+0000 7f2f63fff640 1 --2- 172.21.15.28:0/2248461274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bc0980 0x558e07bc7800 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f2f5400dea0 tx=0x7f2f54005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d307337fb01b89 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:19.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.000+0000 7f2f49ffb640 1 -- 172.21.15.28:0/2248461274 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f54003d50 con 0x558e07bc0980 2024-04-25T01:35:19.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.001+0000 7f2f49ffb640 1 -- 172.21.15.28:0/2248461274 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f54020070 con 0x558e07bc0980 2024-04-25T01:35:19.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.001+0000 7f2f49ffb640 1 -- 172.21.15.28:0/2248461274 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f54012d10 con 0x558e07bc0980 2024-04-25T01:35:19.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.002+0000 7f2f6d409740 1 -- 172.21.15.28:0/2248461274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bc0980 msgr2=0x558e07bc7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:19.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.002+0000 7f2f6d409740 1 --2- 172.21.15.28:0/2248461274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bc0980 0x558e07bc7800 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f2f5400dea0 tx=0x7f2f54005d80 comp rx=0 tx=0).stop 2024-04-25T01:35:19.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.002+0000 7f2f6d409740 1 -- 172.21.15.28:0/2248461274 shutdown_connections 2024-04-25T01:35:19.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.002+0000 7f2f6d409740 1 --2- 172.21.15.28:0/2248461274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bca010 0x558e07bc1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.002+0000 7f2f6d409740 1 --2- 172.21.15.28:0/2248461274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bc0980 0x558e07bc7800 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.002+0000 7f2f6d409740 1 --2- 172.21.15.28:0/2248461274 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e07bbf290 0x558e07bba730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.002+0000 7f2f6d409740 1 -- 172.21.15.28:0/2248461274 >> 172.21.15.28:0/2248461274 conn(0x558e07b87950 msgr2=0x558e07bc0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:19.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.003+0000 7f2f6d409740 1 -- 172.21.15.28:0/2248461274 shutdown_connections 2024-04-25T01:35:19.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.003+0000 7f2f6d409740 1 -- 172.21.15.28:0/2248461274 wait complete. 2024-04-25T01:35:19.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.004+0000 7f2f6d409740 1 -- start start 2024-04-25T01:35:19.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.004+0000 7f2f6d409740 1 Processor -- start 2024-04-25T01:35:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.008+0000 7f2f6d409740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bbf290 0x558e07bd22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.009+0000 7f2f6d409740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e07bc0980 0x558e07bd27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.009+0000 7f2f6d409740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 0x558e07bcf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.009+0000 7f2f6d409740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f641bed20 con 0x558e07bca010 2024-04-25T01:35:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.009+0000 7f2f6d409740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2f641be650 con 0x558e07bbf290 2024-04-25T01:35:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.009+0000 7f2f6d409740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2f641bb890 con 0x558e07bc0980 2024-04-25T01:35:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.009+0000 7f2f6d409740 10 client.0 _ll_register_callbacks cb 0x7f2f58009db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.010+0000 7f2f637fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bbf290 0x558e07bd22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.010+0000 7f2f637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 0x558e07bcf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.010+0000 7f2f637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 0x558e07bcf280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47624/0 (socket says 172.21.15.28:47624) 2024-04-25T01:35:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.010+0000 7f2f637fe640 1 -- 172.21.15.28:0/2350609049 learned_addr learned my addr 172.21.15.28:0/2350609049 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.010+0000 7f2f63fff640 1 --2- 172.21.15.28:0/2350609049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e07bc0980 0x558e07bd27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.014+0000 7f2f637fe640 1 -- 172.21.15.28:0/2350609049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e07bc0980 msgr2=0x558e07bd27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.014+0000 7f2f637fe640 1 --2- 172.21.15.28:0/2350609049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e07bc0980 0x558e07bd27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.014+0000 7f2f637fe640 1 -- 172.21.15.28:0/2350609049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bbf290 msgr2=0x558e07bd22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.014+0000 7f2f637fe640 1 --2- 172.21.15.28:0/2350609049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bbf290 0x558e07bd22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.014+0000 7f2f637fe640 1 -- 172.21.15.28:0/2350609049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e07bcf8c0 con 0x558e07bca010 2024-04-25T01:35:19.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.016+0000 7f2f637fe640 1 --2- 172.21.15.28:0/2350609049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 0x558e07bcf280 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f2f58020780 tx=0x7f2f58000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8a3e650de3f64b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:19.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.016+0000 7f2f437fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:19.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.018+0000 7f2f437fe640 1 -- 172.21.15.28:0/2350609049 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f58005bb0 con 0x558e07bca010 2024-04-25T01:35:19.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.018+0000 7f2f6d409740 1 -- 172.21.15.28:0/2350609049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f54020070 con 0x558e07bca010 2024-04-25T01:35:19.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.018+0000 7f2f437fe640 1 -- 172.21.15.28:0/2350609049 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f2f58009440 con 0x558e07bca010 2024-04-25T01:35:19.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.020+0000 7f2f437fe640 1 -- 172.21.15.28:0/2350609049 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2f58009e60 con 0x558e07bca010 2024-04-25T01:35:19.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.022+0000 7f2f6d409740 10 client.16380 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:19.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.022+0000 7f2f6d409740 1 -- 172.21.15.28:0/2350609049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558e07c51d70 con 0x558e07bca010 2024-04-25T01:35:20.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.024+0000 7f2f437fe640 1 -- 172.21.15.28:0/2350609049 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f54020070 con 0x558e07bca010 2024-04-25T01:35:20.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.024+0000 7f2f437fe640 1 -- 172.21.15.28:0/2350609049 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2f5800c050 con 0x558e07bca010 2024-04-25T01:35:20.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.024+0000 7f2f6d409740 10 client.16380 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:20.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.024+0000 7f2f6d409740 10 client.16380 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:20.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.024+0000 7f2f6d409740 20 client.16380 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:20.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.024+0000 7f2f6d409740 1 --2- 172.21.15.28:0/2350609049 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558e07c523b0 0x558e07c547d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.024+0000 7f2f6d409740 4 client.16380 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:35:20.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.024+0000 7f2f6d409740 1 -- 172.21.15.28:0/2350609049 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558e07b30800 con 0x558e07c523b0 2024-04-25T01:35:20.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.026+0000 7f2f63fff640 1 --2- 172.21.15.28:0/2350609049 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558e07c523b0 0x558e07c547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:20.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.035+0000 7f2f63fff640 1 --2- 172.21.15.28:0/2350609049 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558e07c523b0 0x558e07c547d0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ea18a14845ac87f server_cookie=2ac057fd48a09548 in_seq=0 out_seq=0 2024-04-25T01:35:20.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.036+0000 7f2f437fe640 10 client.16380 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:20.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.039+0000 7f2f437fe640 1 -- 172.21.15.28:0/2350609049 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558e07b30800 con 0x558e07c523b0 2024-04-25T01:35:20.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.039+0000 7f2f437fe640 10 client.16380 handle_command_reply: tid=0 2024-04-25T01:35:20.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.039+0000 7f2f6d409740 1 client.16380 shutdown 2024-04-25T01:35:20.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.039+0000 7f2f6d409740 1 -- 172.21.15.28:0/2350609049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 msgr2=0x558e07bcf280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:20.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.039+0000 7f2f6d409740 1 --2- 172.21.15.28:0/2350609049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 0x558e07bcf280 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f2f58020780 tx=0x7f2f58000bc0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.039+0000 7f2f6d409740 1 -- 172.21.15.28:0/2350609049 shutdown_connections 2024-04-25T01:35:20.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.039+0000 7f2f6d409740 1 --2- 172.21.15.28:0/2350609049 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558e07c523b0 0x558e07c547d0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.039+0000 7f2f6d409740 1 --2- 172.21.15.28:0/2350609049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e07bca010 0x558e07bcf280 secure :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f2f58020780 tx=0x7f2f58000bc0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.039+0000 7f2f6d409740 1 --2- 172.21.15.28:0/2350609049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558e07bc0980 0x558e07bd27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.039+0000 7f2f6d409740 1 --2- 172.21.15.28:0/2350609049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558e07bbf290 0x558e07bd22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.039+0000 7f2f6d409740 1 -- 172.21.15.28:0/2350609049 >> 172.21.15.28:0/2350609049 conn(0x558e07b87950 msgr2=0x558e07bc15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:20.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.039+0000 7f2f437fe640 0 client.16380 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:20.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.044+0000 7f2f6d409740 1 -- 172.21.15.28:0/2350609049 shutdown_connections 2024-04-25T01:35:20.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.045+0000 7f2f6d409740 1 -- 172.21.15.28:0/2350609049 wait complete. 2024-04-25T01:35:20.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.045+0000 7f2f6d409740 20 client.16380 trim_cache size 0 max 16384 2024-04-25T01:35:20.022 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:35:20.022 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:35:20.023 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "bf5c612d-f962-4e92-81dc-3753b38385a3", 2024-04-25T01:35:20.023 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:35:20.023 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:35:20.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.048+0000 7f2f6a61c640 1 -- 172.21.15.28:0/2710639990 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f2f4c0a2ed0 msgr2=0x7f2f4c0a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:20.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.048+0000 7f2f6a61c640 1 --2- 172.21.15.28:0/2710639990 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f2f4c0a2ed0 0x7f2f4c0a5390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c0b8ba0 tx=0x7f2f5c0098c0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.048+0000 7f2f6a61c640 1 -- 172.21.15.28:0/2710639990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f64112730 msgr2=0x7f2f6406c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:20.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.048+0000 7f2f6a61c640 1 --2- 172.21.15.28:0/2710639990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f64112730 0x7f2f6406c6c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c006010 tx=0x7f2f5c003c10 comp rx=0 tx=0).stop 2024-04-25T01:35:20.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.049+0000 7f2f6a61c640 1 -- 172.21.15.28:0/2710639990 shutdown_connections 2024-04-25T01:35:20.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.049+0000 7f2f6a61c640 1 --2- 172.21.15.28:0/2710639990 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f2f4c0a2ed0 0x7f2f4c0a5390 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.049+0000 7f2f6a61c640 1 --2- 172.21.15.28:0/2710639990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f64112730 0x7f2f6406c6c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.049+0000 7f2f6a61c640 1 --2- 172.21.15.28:0/2710639990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f64074400 0x7f2f6406c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.049+0000 7f2f6a61c640 1 --2- 172.21.15.28:0/2710639990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f64073ae0 0x7f2f64073060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.049+0000 7f2f6a61c640 1 -- 172.21.15.28:0/2710639990 >> 172.21.15.28:0/2710639990 conn(0x7f2f64078b40 msgr2=0x7f2f640ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:20.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.049+0000 7f2f6a61c640 1 -- 172.21.15.28:0/2710639990 shutdown_connections 2024-04-25T01:35:20.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.049+0000 7f2f6a61c640 1 -- 172.21.15.28:0/2710639990 wait complete. 2024-04-25T01:35:20.055 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:35:20.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:35:20.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.311+0000 7f3bfe912640 1 -- 172.21.15.28:0/809993788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bf806a2b0 msgr2=0x7f3bf806a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:20.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.311+0000 7f3bfe912640 1 --2- 172.21.15.28:0/809993788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bf806a2b0 0x7f3bf806a710 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f3be8009900 tx=0x7f3be802a420 comp rx=0 tx=0).stop 2024-04-25T01:35:20.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.312+0000 7f3bfe912640 1 -- 172.21.15.28:0/809993788 shutdown_connections 2024-04-25T01:35:20.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.312+0000 7f3bfe912640 1 --2- 172.21.15.28:0/809993788 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bf8104150 0x7f3bf8104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.312+0000 7f3bfe912640 1 --2- 172.21.15.28:0/809993788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bf806a2b0 0x7f3bf806a710 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.312+0000 7f3bfe912640 1 --2- 172.21.15.28:0/809993788 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3bf8106650 0x7f3bf8069d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.312+0000 7f3bfe912640 1 -- 172.21.15.28:0/809993788 >> 172.21.15.28:0/809993788 conn(0x7f3bf80777d0 msgr2=0x7f3bf8077be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:20.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.312+0000 7f3bfe912640 1 -- 172.21.15.28:0/809993788 shutdown_connections 2024-04-25T01:35:20.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.314+0000 7f3bfe912640 1 -- 172.21.15.28:0/809993788 wait complete. 2024-04-25T01:35:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.315+0000 7f3bfe912640 1 Processor -- start 2024-04-25T01:35:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.315+0000 7f3bfe912640 1 -- start start 2024-04-25T01:35:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.316+0000 7f3bfe912640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3bf806a2b0 0x7f3bf8114bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.316+0000 7f3bfe912640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bf8104150 0x7f3bf810ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.316+0000 7f3bfe912640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bf8106650 0x7f3bf810f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.316+0000 7f3bfe912640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf81075c0 con 0x7f3bf8106650 2024-04-25T01:35:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.316+0000 7f3bfe912640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf8107440 con 0x7f3bf8104150 2024-04-25T01:35:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.316+0000 7f3bfe912640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf8107740 con 0x7f3bf806a2b0 2024-04-25T01:35:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.317+0000 7f3bfceaf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bf8106650 0x7f3bf810f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.317+0000 7f3bfceaf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bf8106650 0x7f3bf810f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47638/0 (socket says 172.21.15.28:47638) 2024-04-25T01:35:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.317+0000 7f3bfceaf640 1 -- 172.21.15.28:0/2914709847 learned_addr learned my addr 172.21.15.28:0/2914709847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:20.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.319+0000 7f3bf7fff640 1 --2- 172.21.15.28:0/2914709847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3bf806a2b0 0x7f3bf8114bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:20.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.320+0000 7f3bfceaf640 1 -- 172.21.15.28:0/2914709847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3bf806a2b0 msgr2=0x7f3bf8114bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:20.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.320+0000 7f3bfceaf640 1 --2- 172.21.15.28:0/2914709847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3bf806a2b0 0x7f3bf8114bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.320+0000 7f3bfceaf640 1 -- 172.21.15.28:0/2914709847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bf8104150 msgr2=0x7f3bf810ec20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T01:35:20.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.320+0000 7f3bfceaf640 1 --2- 172.21.15.28:0/2914709847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bf8104150 0x7f3bf810ec20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.320+0000 7f3bfceaf640 1 -- 172.21.15.28:0/2914709847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bf810f990 con 0x7f3bf8106650 2024-04-25T01:35:20.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.321+0000 7f3bfceaf640 1 --2- 172.21.15.28:0/2914709847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bf8106650 0x7f3bf810f160 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000cad0 tx=0x7f3bf000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.322+0000 7f3bf57fa640 1 -- 172.21.15.28:0/2914709847 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf000ae20 con 0x7f3bf8106650 2024-04-25T01:35:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.323+0000 7f3bf57fa640 1 -- 172.21.15.28:0/2914709847 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bf0005020 con 0x7f3bf8106650 2024-04-25T01:35:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.323+0000 7f3bf57fa640 1 -- 172.21.15.28:0/2914709847 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf000f6f0 con 0x7f3bf8106650 2024-04-25T01:35:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.323+0000 7f3bfe912640 1 -- 172.21.15.28:0/2914709847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bf81b34a0 con 0x7f3bf8106650 2024-04-25T01:35:20.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.323+0000 7f3bfe912640 1 -- 172.21.15.28:0/2914709847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bf81b39a0 con 0x7f3bf8106650 2024-04-25T01:35:20.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.329+0000 7f3c01617740 1 Processor -- start 2024-04-25T01:35:20.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.332+0000 7f3c01617740 1 -- start start 2024-04-25T01:35:20.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.333+0000 7f3c01617740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6a1630 0x55adfd6a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.333+0000 7f3c01617740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 0x55adfd6ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.333+0000 7f3c01617740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55adfd6ac620 0x55adfd6aea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.333+0000 7f3c01617740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55adfd6066b0 con 0x55adfd6a1f50 2024-04-25T01:35:20.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.333+0000 7f3bf57fa640 1 -- 172.21.15.28:0/2914709847 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3bf0005560 con 0x7f3bf8106650 2024-04-25T01:35:20.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.333+0000 7f3c01617740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55adfd5ae880 con 0x55adfd6a1630 2024-04-25T01:35:20.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.333+0000 7f3c01617740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55adfd602670 con 0x55adfd6ac620 2024-04-25T01:35:20.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:20 smithi028 ceph-mon[29390]: pgmap v72: 129 pgs: 129 active+clean; 15 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 541 MiB/s wr, 263 op/s 2024-04-25T01:35:20.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.335+0000 7f3bf7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 0x55adfd6ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:20.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.335+0000 7f3bf77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6a1630 0x55adfd6a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:20.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.335+0000 7f3bf77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55adfd6ac620 0x55adfd6aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:20.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.335+0000 7f3bf7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 0x55adfd6ac0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47648/0 (socket says 172.21.15.28:47648) 2024-04-25T01:35:20.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.335+0000 7f3bf7fff640 1 -- 172.21.15.28:0/296568487 learned_addr learned my addr 172.21.15.28:0/296568487 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:20.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.335+0000 7f3bf77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55adfd6ac620 0x55adfd6aea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.28:39572/0 (socket says 172.21.15.28:39572) 2024-04-25T01:35:20.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.336+0000 7f3bf57fa640 1 --2- 172.21.15.28:0/2914709847 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3be00a2f00 0x7f3be00a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.337+0000 7f3bf7fff640 1 -- 172.21.15.28:0/296568487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55adfd6ac620 msgr2=0x55adfd6aea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:20.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.337+0000 7f3bf7fff640 1 --2- 172.21.15.28:0/296568487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55adfd6ac620 0x55adfd6aea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.337+0000 7f3bf7fff640 1 -- 172.21.15.28:0/296568487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6a1630 msgr2=0x55adfd6a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:20.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.337+0000 7f3bf7fff640 1 --2- 172.21.15.28:0/296568487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6a1630 0x55adfd6a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.337+0000 7f3bf7fff640 1 -- 172.21.15.28:0/296568487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adfd3e47d0 con 0x55adfd6a1f50 2024-04-25T01:35:20.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.337+0000 7f3bf57fa640 1 -- 172.21.15.28:0/2914709847 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f3bf00cf8b0 con 0x7f3bf8106650 2024-04-25T01:35:20.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.337+0000 7f3bf7fff640 1 --2- 172.21.15.28:0/296568487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 0x55adfd6ac0e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f3bec00a8d0 tx=0x7f3bec00ad90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d64dba67b900e35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:20.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.338+0000 7f3bde7fc640 1 -- 172.21.15.28:0/296568487 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bec015070 con 0x55adfd6a1f50 2024-04-25T01:35:20.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.338+0000 7f3bde7fc640 1 -- 172.21.15.28:0/296568487 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bec00cb40 con 0x55adfd6a1f50 2024-04-25T01:35:20.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.340+0000 7f3bfceaf640 1 --2- 172.21.15.28:0/2914709847 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3be00a2f00 0x7f3be00a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:20.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.340+0000 7f3c01617740 1 -- 172.21.15.28:0/296568487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 msgr2=0x55adfd6ac0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:20.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.340+0000 7f3c01617740 1 --2- 172.21.15.28:0/296568487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 0x55adfd6ac0e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f3bec00a8d0 tx=0x7f3bec00ad90 comp rx=0 tx=0).stop 2024-04-25T01:35:20.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.340+0000 7f3c01617740 1 -- 172.21.15.28:0/296568487 shutdown_connections 2024-04-25T01:35:20.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.340+0000 7f3c01617740 1 --2- 172.21.15.28:0/296568487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55adfd6ac620 0x55adfd6aea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.340+0000 7f3c01617740 1 --2- 172.21.15.28:0/296568487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 0x55adfd6ac0e0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.340+0000 7f3c01617740 1 --2- 172.21.15.28:0/296568487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6a1630 0x55adfd6a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.340+0000 7f3c01617740 1 -- 172.21.15.28:0/296568487 >> 172.21.15.28:0/296568487 conn(0x55adfd69d950 msgr2=0x55adfd69dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:20.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.340+0000 7f3c01617740 1 -- 172.21.15.28:0/296568487 shutdown_connections 2024-04-25T01:35:20.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.340+0000 7f3c01617740 1 -- 172.21.15.28:0/296568487 wait complete. 2024-04-25T01:35:20.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.341+0000 7f3c01617740 1 -- start start 2024-04-25T01:35:20.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.341+0000 7f3c01617740 1 Processor -- start 2024-04-25T01:35:20.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.342+0000 7f3c01617740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6a1630 0x55adfd6a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.342+0000 7f3c01617740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 0x55adfd6a99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.342+0000 7f3c01617740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55adfd6ac620 0x55adfd6a9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.342+0000 7f3c01617740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55adfd6af460 con 0x55adfd6a1f50 2024-04-25T01:35:20.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.342+0000 7f3c01617740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55adfd5c1f50 con 0x55adfd6a1630 2024-04-25T01:35:20.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.342+0000 7f3c01617740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55adfd6af5e0 con 0x55adfd6ac620 2024-04-25T01:35:20.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.342+0000 7f3c01617740 10 client.0 _ll_register_callbacks cb 0x55adfd5fd000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:20.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.343+0000 7f3bf7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 0x55adfd6a99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:20.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.343+0000 7f3bf7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 0x55adfd6a99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47652/0 (socket says 172.21.15.28:47652) 2024-04-25T01:35:20.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.343+0000 7f3bf7fff640 1 -- 172.21.15.28:0/799037067 learned_addr learned my addr 172.21.15.28:0/799037067 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:20.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.343+0000 7f3bf77fe640 1 --2- 172.21.15.28:0/799037067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6a1630 0x55adfd6a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:20.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.343+0000 7f3bfceaf640 1 --2- 172.21.15.28:0/2914709847 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3be00a2f00 0x7f3be00a53c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00cccc0 tx=0x7f3bf0005850 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:20.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.344+0000 7f3bf7fff640 1 -- 172.21.15.28:0/799037067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55adfd6ac620 msgr2=0x55adfd6a9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:20.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.344+0000 7f3bf7fff640 1 --2- 172.21.15.28:0/799037067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55adfd6ac620 0x55adfd6a9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.344+0000 7f3bf7fff640 1 -- 172.21.15.28:0/799037067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6a1630 msgr2=0x55adfd6a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:20.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.344+0000 7f3bf7fff640 1 --2- 172.21.15.28:0/799037067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6a1630 0x55adfd6a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.344+0000 7f3bf7fff640 1 -- 172.21.15.28:0/799037067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adfd31a3f0 con 0x55adfd6a1f50 2024-04-25T01:35:20.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.344+0000 7f3bf7fff640 1 --2- 172.21.15.28:0/799037067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 0x55adfd6a99e0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f3bec011cd0 tx=0x7f3bec006bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6cc033ebca4f5e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:20.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.345+0000 7f3bd77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:20.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.345+0000 7f3bd77fe640 1 -- 172.21.15.28:0/799037067 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bec015040 con 0x55adfd6a1f50 2024-04-25T01:35:20.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.345+0000 7f3bd77fe640 1 -- 172.21.15.28:0/799037067 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f3bec011800 con 0x55adfd6a1f50 2024-04-25T01:35:20.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.345+0000 7f3c01617740 1 -- 172.21.15.28:0/799037067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bec00cb40 con 0x55adfd6a1f50 2024-04-25T01:35:20.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.346+0000 7f3bd77fe640 1 -- 172.21.15.28:0/799037067 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3bec006a20 con 0x55adfd6a1f50 2024-04-25T01:35:20.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.346+0000 7f3c01617740 10 client.16392 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:20.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.346+0000 7f3c01617740 1 -- 172.21.15.28:0/799037067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55adfd2da0f0 con 0x55adfd6a1f50 2024-04-25T01:35:20.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.347+0000 7f3bd77fe640 1 -- 172.21.15.28:0/799037067 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bec00cb40 con 0x55adfd6a1f50 2024-04-25T01:35:20.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.347+0000 7f3bd77fe640 1 -- 172.21.15.28:0/799037067 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3bec003690 con 0x55adfd6a1f50 2024-04-25T01:35:20.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.348+0000 7f3c01617740 10 client.16392 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:20.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.348+0000 7f3c01617740 10 client.16392 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:20.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.348+0000 7f3c01617740 20 client.16392 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:20.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.348+0000 7f3c01617740 1 --2- 172.21.15.28:0/799037067 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55adfd6bc720 0x55adfd6beb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.348+0000 7f3c01617740 4 client.16392 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:20.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.348+0000 7f3c01617740 1 -- 172.21.15.28:0/799037067 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adfd613bf0 con 0x55adfd6bc720 2024-04-25T01:35:20.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.350+0000 7f3bf77fe640 1 --2- 172.21.15.28:0/799037067 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55adfd6bc720 0x55adfd6beb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:20.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.352+0000 7f3bd77fe640 10 client.16392 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:20.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.352+0000 7f3bf77fe640 1 --2- 172.21.15.28:0/799037067 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55adfd6bc720 0x55adfd6beb10 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88c19e9a9c233ffb server_cookie=3f08d903bf9c8b1d in_seq=0 out_seq=0 2024-04-25T01:35:20.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.354+0000 7f3bd77fe640 1 -- 172.21.15.28:0/799037067 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55adfd613bf0 con 0x55adfd6bc720 2024-04-25T01:35:20.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.354+0000 7f3bd77fe640 10 client.16392 handle_command_reply: tid=0 2024-04-25T01:35:20.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.354+0000 7f3c01617740 1 client.16392 shutdown 2024-04-25T01:35:20.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.357+0000 7f3c01617740 1 -- 172.21.15.28:0/799037067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 msgr2=0x55adfd6a99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:20.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.357+0000 7f3c01617740 1 --2- 172.21.15.28:0/799037067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 0x55adfd6a99e0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f3bec011cd0 tx=0x7f3bec006bf0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.358+0000 7f3c01617740 1 -- 172.21.15.28:0/799037067 shutdown_connections 2024-04-25T01:35:20.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.358+0000 7f3c01617740 1 --2- 172.21.15.28:0/799037067 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55adfd6bc720 0x55adfd6beb10 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.358+0000 7f3c01617740 1 --2- 172.21.15.28:0/799037067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55adfd6ac620 0x55adfd6a9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.358+0000 7f3c01617740 1 --2- 172.21.15.28:0/799037067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 0x55adfd6a99e0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.358+0000 7f3c01617740 1 --2- 172.21.15.28:0/799037067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6a1630 0x55adfd6a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.358+0000 7f3c01617740 1 -- 172.21.15.28:0/799037067 >> 172.21.15.28:0/799037067 conn(0x55adfd69d950 msgr2=0x55adfd6a3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:20.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.358+0000 7f3bd77fe640 0 client.16392 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:20.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.358+0000 7f3c01617740 1 -- 172.21.15.28:0/799037067 shutdown_connections 2024-04-25T01:35:20.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.359+0000 7f3c01617740 1 -- 172.21.15.28:0/799037067 wait complete. 2024-04-25T01:35:20.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.359+0000 7f3c01617740 20 client.16392 trim_cache size 0 max 16384 2024-04-25T01:35:20.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.398+0000 7f3c01617740 1 Processor -- start 2024-04-25T01:35:20.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.398+0000 7f3c01617740 1 -- start start 2024-04-25T01:35:20.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.398+0000 7f3c01617740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6a1630 0x55adfd6b8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.398+0000 7f3c01617740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55adfd6a1f50 0x55adfd6a35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.398+0000 7f3c01617740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6ac620 0x55adfd6c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.399+0000 7f3c01617740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf81bdb10 con 0x55adfd6ac620 2024-04-25T01:35:20.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.399+0000 7f3c01617740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf81bf650 con 0x55adfd6a1630 2024-04-25T01:35:20.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.399+0000 7f3c01617740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf81be1e0 con 0x55adfd6a1f50 2024-04-25T01:35:20.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.399+0000 7f3bf7fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55adfd6a1f50 0x55adfd6a35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:20.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.399+0000 7f3bf77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6a1630 0x55adfd6b8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:20.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.399+0000 7f3bf77fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6a1630 0x55adfd6b8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:44650/0 (socket says 172.21.15.28:44650) 2024-04-25T01:35:20.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.399+0000 7f3bf77fe640 1 -- 172.21.15.28:0/3507786046 learned_addr learned my addr 172.21.15.28:0/3507786046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:20.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.399+0000 7f3bf77fe640 1 --2- 172.21.15.28:0/3507786046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6ac620 0x55adfd6c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:20.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.401+0000 7f3bf77fe640 1 -- 172.21.15.28:0/3507786046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55adfd6a1f50 msgr2=0x55adfd6a35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:20.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.401+0000 7f3bf77fe640 1 --2- 172.21.15.28:0/3507786046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55adfd6a1f50 0x55adfd6a35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.401+0000 7f3bf77fe640 1 -- 172.21.15.28:0/3507786046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6a1630 msgr2=0x55adfd6b8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:20.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.401+0000 7f3bf77fe640 1 --2- 172.21.15.28:0/3507786046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6a1630 0x55adfd6b8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.401+0000 7f3bf77fe640 1 -- 172.21.15.28:0/3507786046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adfd69d780 con 0x55adfd6ac620 2024-04-25T01:35:20.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.401+0000 7f3bf77fe640 1 --2- 172.21.15.28:0/3507786046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6ac620 0x55adfd6c41c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f3be8009850 tx=0x7f3be8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a413a23c79b4907b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:20.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.401+0000 7f3bddffb640 1 -- 172.21.15.28:0/3507786046 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be8035040 con 0x55adfd6ac620 2024-04-25T01:35:20.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.402+0000 7f3bddffb640 1 -- 172.21.15.28:0/3507786046 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55adfd613bf0 con 0x55adfd6ac620 2024-04-25T01:35:20.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.402+0000 7f3bddffb640 1 -- 172.21.15.28:0/3507786046 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be8044520 con 0x55adfd6ac620 2024-04-25T01:35:20.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.403+0000 7f3c01617740 1 -- 172.21.15.28:0/3507786046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6ac620 msgr2=0x55adfd6c41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:20.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.403+0000 7f3c01617740 1 --2- 172.21.15.28:0/3507786046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6ac620 0x55adfd6c41c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f3be8009850 tx=0x7f3be8000bc0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.403+0000 7f3c01617740 1 -- 172.21.15.28:0/3507786046 shutdown_connections 2024-04-25T01:35:20.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.403+0000 7f3c01617740 1 --2- 172.21.15.28:0/3507786046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6ac620 0x55adfd6c41c0 secure :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f3be8009850 tx=0x7f3be8000bc0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.403+0000 7f3c01617740 1 --2- 172.21.15.28:0/3507786046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55adfd6a1f50 0x55adfd6a35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.403+0000 7f3c01617740 1 --2- 172.21.15.28:0/3507786046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6a1630 0x55adfd6b8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.403+0000 7f3c01617740 1 -- 172.21.15.28:0/3507786046 >> 172.21.15.28:0/3507786046 conn(0x55adfd69d950 msgr2=0x55adfd6a2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:20.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.403+0000 7f3c01617740 1 -- 172.21.15.28:0/3507786046 shutdown_connections 2024-04-25T01:35:20.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.403+0000 7f3c01617740 1 -- 172.21.15.28:0/3507786046 wait complete. 2024-04-25T01:35:20.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.404+0000 7f3c01617740 1 -- start start 2024-04-25T01:35:20.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.404+0000 7f3c01617740 1 Processor -- start 2024-04-25T01:35:20.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.404+0000 7f3c01617740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55adfd6a1630 0x55adfd6b0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.405+0000 7f3c01617740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 0x55adfd6b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.405+0000 7f3c01617740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6b0df0 0x55adfd721a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.405+0000 7f3c01617740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf81bef80 con 0x55adfd6a1f50 2024-04-25T01:35:20.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.405+0000 7f3c01617740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf81be8b0 con 0x55adfd6b0df0 2024-04-25T01:35:20.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.405+0000 7f3c01617740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf81bbaf0 con 0x55adfd6a1630 2024-04-25T01:35:20.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.405+0000 7f3c01617740 10 client.0 _ll_register_callbacks cb 0x7f3bec00a720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:20.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.405+0000 7f3bf7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 0x55adfd6b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:20.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.405+0000 7f3bf7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 0x55adfd6b08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47682/0 (socket says 172.21.15.28:47682) 2024-04-25T01:35:20.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.405+0000 7f3bf7fff640 1 -- 172.21.15.28:0/2555149896 learned_addr learned my addr 172.21.15.28:0/2555149896 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:20.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.406+0000 7f3bf7fff640 1 -- 172.21.15.28:0/2555149896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55adfd6a1630 msgr2=0x55adfd6b0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:20.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.406+0000 7f3bf7fff640 1 --2- 172.21.15.28:0/2555149896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55adfd6a1630 0x55adfd6b0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.406+0000 7f3bf7fff640 1 -- 172.21.15.28:0/2555149896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6b0df0 msgr2=0x55adfd721a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:20.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.406+0000 7f3bf7fff640 1 --2- 172.21.15.28:0/2555149896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6b0df0 0x55adfd721a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.406+0000 7f3bf7fff640 1 -- 172.21.15.28:0/2555149896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adfd721f70 con 0x55adfd6a1f50 2024-04-25T01:35:20.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.407+0000 7f3bf7fff640 1 --2- 172.21.15.28:0/2555149896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 0x55adfd6b08b0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f3bec003bd0 tx=0x7f3bec003c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bee1601c816756a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:20.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.407+0000 7f3bd77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:20.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.408+0000 7f3bd77fe640 1 -- 172.21.15.28:0/2555149896 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bec034020 con 0x55adfd6a1f50 2024-04-25T01:35:20.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.408+0000 7f3bd77fe640 1 -- 172.21.15.28:0/2555149896 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f3bec018760 con 0x55adfd6a1f50 2024-04-25T01:35:20.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.408+0000 7f3c01617740 1 -- 172.21.15.28:0/2555149896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55adfd613bf0 con 0x55adfd6a1f50 2024-04-25T01:35:20.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.408+0000 7f3bd77fe640 1 -- 172.21.15.28:0/2555149896 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3bec015040 con 0x55adfd6a1f50 2024-04-25T01:35:20.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.409+0000 7f3c01617740 10 client.16398 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:20.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.409+0000 7f3c01617740 1 -- 172.21.15.28:0/2555149896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55adfd723230 con 0x55adfd6a1f50 2024-04-25T01:35:20.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.410+0000 7f3bd77fe640 1 -- 172.21.15.28:0/2555149896 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55adfd613bf0 con 0x55adfd6a1f50 2024-04-25T01:35:20.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.410+0000 7f3bd77fe640 1 -- 172.21.15.28:0/2555149896 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3bec004490 con 0x55adfd6a1f50 2024-04-25T01:35:20.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.410+0000 7f3c01617740 10 client.16398 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:20.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.410+0000 7f3c01617740 10 client.16398 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:20.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.410+0000 7f3c01617740 20 client.16398 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:20.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.410+0000 7f3c01617740 1 --2- 172.21.15.28:0/2555149896 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55adfd723920 0x55adfd725d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:20.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.410+0000 7f3c01617740 4 client.16398 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:35:20.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.410+0000 7f3c01617740 1 -- 172.21.15.28:0/2555149896 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55adfd60df40 con 0x55adfd723920 2024-04-25T01:35:20.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.411+0000 7f3bf77fe640 1 --2- 172.21.15.28:0/2555149896 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55adfd723920 0x55adfd725d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:20.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.415+0000 7f3bf77fe640 1 --2- 172.21.15.28:0/2555149896 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55adfd723920 0x55adfd725d40 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edbd89f9366812f9 server_cookie=928367c198166443 in_seq=0 out_seq=0 2024-04-25T01:35:20.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.415+0000 7f3bd77fe640 10 client.16398 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:20.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.415+0000 7f3bd77fe640 1 -- 172.21.15.28:0/2555149896 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55adfd60df40 con 0x55adfd723920 2024-04-25T01:35:20.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.415+0000 7f3bd77fe640 10 client.16398 handle_command_reply: tid=0 2024-04-25T01:35:20.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.415+0000 7f3c01617740 1 client.16398 shutdown 2024-04-25T01:35:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.416+0000 7f3c01617740 1 -- 172.21.15.28:0/2555149896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 msgr2=0x55adfd6b08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.416+0000 7f3c01617740 1 --2- 172.21.15.28:0/2555149896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 0x55adfd6b08b0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f3bec003bd0 tx=0x7f3bec003c50 comp rx=0 tx=0).stop 2024-04-25T01:35:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.416+0000 7f3c01617740 1 -- 172.21.15.28:0/2555149896 shutdown_connections 2024-04-25T01:35:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.416+0000 7f3c01617740 1 --2- 172.21.15.28:0/2555149896 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55adfd723920 0x55adfd725d40 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.416+0000 7f3c01617740 1 --2- 172.21.15.28:0/2555149896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55adfd6b0df0 0x55adfd721a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.416+0000 7f3c01617740 1 --2- 172.21.15.28:0/2555149896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55adfd6a1f50 0x55adfd6b08b0 secure :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f3bec003bd0 tx=0x7f3bec003c50 comp rx=0 tx=0).stop 2024-04-25T01:35:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.416+0000 7f3c01617740 1 --2- 172.21.15.28:0/2555149896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55adfd6a1630 0x55adfd6b0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.416+0000 7f3c01617740 1 -- 172.21.15.28:0/2555149896 >> 172.21.15.28:0/2555149896 conn(0x55adfd69d950 msgr2=0x55adfd69caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.416+0000 7f3bd77fe640 0 client.16398 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:20.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.420+0000 7f3c01617740 1 -- 172.21.15.28:0/2555149896 shutdown_connections 2024-04-25T01:35:20.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.420+0000 7f3c01617740 1 -- 172.21.15.28:0/2555149896 wait complete. 2024-04-25T01:35:20.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.420+0000 7f3c01617740 20 client.16398 trim_cache size 0 max 16384 2024-04-25T01:35:20.397 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:35:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:35:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-25T01:35:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "bf5c612d-f962-4e92-81dc-3753b38385a3": { 2024-04-25T01:35:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-25T01:35:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "bf5c612d-f962-4e92-81dc-3753b38385a3", 2024-04-25T01:35:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-25T01:35:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:35:20.398 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:35:20.398 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:35:20.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.422+0000 7f3bfe912640 1 -- 172.21.15.28:0/2914709847 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3be00a2f00 msgr2=0x7f3be00a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:20.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.422+0000 7f3bfe912640 1 --2- 172.21.15.28:0/2914709847 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3be00a2f00 0x7f3be00a53c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00cccc0 tx=0x7f3bf0005850 comp rx=0 tx=0).stop 2024-04-25T01:35:20.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.422+0000 7f3bfe912640 1 -- 172.21.15.28:0/2914709847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bf8106650 msgr2=0x7f3bf810f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:20.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.422+0000 7f3bfe912640 1 --2- 172.21.15.28:0/2914709847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bf8106650 0x7f3bf810f160 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000cad0 tx=0x7f3bf000cf90 comp rx=0 tx=0).stop 2024-04-25T01:35:20.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.423+0000 7f3bfe912640 1 -- 172.21.15.28:0/2914709847 shutdown_connections 2024-04-25T01:35:20.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.423+0000 7f3bfe912640 1 --2- 172.21.15.28:0/2914709847 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3be00a2f00 0x7f3be00a53c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.423+0000 7f3bfe912640 1 --2- 172.21.15.28:0/2914709847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bf8106650 0x7f3bf810f160 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.423+0000 7f3bfe912640 1 --2- 172.21.15.28:0/2914709847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3bf8104150 0x7f3bf810ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.423+0000 7f3bfe912640 1 --2- 172.21.15.28:0/2914709847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3bf806a2b0 0x7f3bf8114bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:20.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.423+0000 7f3bfe912640 1 -- 172.21.15.28:0/2914709847 >> 172.21.15.28:0/2914709847 conn(0x7f3bf80777d0 msgr2=0x7f3bf80fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:20.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.425+0000 7f3bfe912640 1 -- 172.21.15.28:0/2914709847 shutdown_connections 2024-04-25T01:35:20.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:20.426+0000 7f3bfe912640 1 -- 172.21.15.28:0/2914709847 wait complete. 2024-04-25T01:35:20.427 INFO:tasks.cephfs.filesystem:scrub status for tag:bf5c612d-f962-4e92-81dc-3753b38385a3 - {'path': '/', 'tag': 'bf5c612d-f962-4e92-81dc-3753b38385a3', 'options': 'recursive,force'} 2024-04-25T01:35:20.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:20 smithi130 ceph-mon[32702]: pgmap v72: 129 pgs: 129 active+clean; 15 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 541 MiB/s wr, 263 op/s 2024-04-25T01:35:20.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:20 smithi032 ceph-mon[32461]: pgmap v72: 129 pgs: 129 active+clean; 15 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 541 MiB/s wr, 263 op/s 2024-04-25T01:35:21.185 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:35:21.185 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 rewriters = 989882.00 kB/sec 2024-04-25T01:35:21.185 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 rewriters = 989858.59 kB/sec 2024-04-25T01:35:21.185 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 989882.00 kB/sec 2024-04-25T01:35:21.185 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 989882.00 kB/sec 2024-04-25T01:35:21.185 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 989882.00 kB/sec 2024-04-25T01:35:21.186 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 10485760.00 kB 2024-04-25T01:35:21.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:21 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:35:21.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:21 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:21.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:21 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:35:21.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:21 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:21.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:21 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:35:21.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:21 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:21.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:21 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:35:21.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:21 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:21.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:21 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:35:21.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:21 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:21.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:21 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:35:21.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:21 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:22.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:22 smithi028 ceph-mon[29390]: pgmap v73: 129 pgs: 129 active+clean; 15 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 541 MiB/s wr, 263 op/s 2024-04-25T01:35:22.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:22 smithi130 ceph-mon[32702]: pgmap v73: 129 pgs: 129 active+clean; 15 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 541 MiB/s wr, 263 op/s 2024-04-25T01:35:22.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:22 smithi032 ceph-mon[32461]: pgmap v73: 129 pgs: 129 active+clean; 15 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 541 MiB/s wr, 263 op/s 2024-04-25T01:35:24.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:24 smithi028 ceph-mon[29390]: pgmap v74: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 933 MiB/s wr, 379 op/s 2024-04-25T01:35:24.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:24 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:35:24.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:24 smithi130 ceph-mon[32702]: pgmap v74: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 933 MiB/s wr, 379 op/s 2024-04-25T01:35:24.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:24 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:35:24.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:24 smithi032 ceph-mon[32461]: pgmap v74: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 933 MiB/s wr, 379 op/s 2024-04-25T01:35:24.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:24 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:35:26.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:26 smithi028 ceph-mon[29390]: pgmap v75: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 810 MiB/s wr, 322 op/s 2024-04-25T01:35:26.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:26 smithi130 ceph-mon[32702]: pgmap v75: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 810 MiB/s wr, 322 op/s 2024-04-25T01:35:26.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:26 smithi032 ceph-mon[32461]: pgmap v75: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 810 MiB/s wr, 322 op/s 2024-04-25T01:35:27.313 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:35:27.313 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 readers = 8962050.00 kB/sec 2024-04-25T01:35:27.313 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 readers = 8960164.74 kB/sec 2024-04-25T01:35:27.313 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 8962050.00 kB/sec 2024-04-25T01:35:27.313 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 8962050.00 kB/sec 2024-04-25T01:35:27.313 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 8962050.00 kB/sec 2024-04-25T01:35:27.313 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 10485760.00 kB 2024-04-25T01:35:28.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:28 smithi028 ceph-mon[29390]: pgmap v76: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 811 MiB/s wr, 326 op/s 2024-04-25T01:35:28.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:28 smithi130 ceph-mon[32702]: pgmap v76: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 811 MiB/s wr, 326 op/s 2024-04-25T01:35:28.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:28 smithi032 ceph-mon[32461]: pgmap v76: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 811 MiB/s wr, 326 op/s 2024-04-25T01:35:30.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:30 smithi028 ceph-mon[29390]: pgmap v77: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 685 MiB/s wr, 196 op/s 2024-04-25T01:35:30.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:30 smithi130 ceph-mon[32702]: pgmap v77: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 685 MiB/s wr, 196 op/s 2024-04-25T01:35:30.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:30 smithi032 ceph-mon[32461]: pgmap v77: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 685 MiB/s wr, 196 op/s 2024-04-25T01:35:30.586 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:35:30.586 INFO:tasks.workunit.client.0.smithi028.stdout: Children see throughput for 1 re-readers = 8976503.00 kB/sec 2024-04-25T01:35:30.586 INFO:tasks.workunit.client.0.smithi028.stdout: Parent sees throughput for 1 re-readers = 8974614.61 kB/sec 2024-04-25T01:35:30.586 INFO:tasks.workunit.client.0.smithi028.stdout: Min throughput per process = 8976503.00 kB/sec 2024-04-25T01:35:30.586 INFO:tasks.workunit.client.0.smithi028.stdout: Max throughput per process = 8976503.00 kB/sec 2024-04-25T01:35:30.587 INFO:tasks.workunit.client.0.smithi028.stdout: Avg throughput per process = 8976503.00 kB/sec 2024-04-25T01:35:30.587 INFO:tasks.workunit.client.0.smithi028.stdout: Min xfer = 10485760.00 kB 2024-04-25T01:35:32.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:32 smithi028 ceph-mon[29390]: pgmap v78: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 392 MiB/s wr, 121 op/s 2024-04-25T01:35:32.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:32 smithi130 ceph-mon[32702]: pgmap v78: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 392 MiB/s wr, 121 op/s 2024-04-25T01:35:32.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:32 smithi032 ceph-mon[32461]: pgmap v78: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 392 MiB/s wr, 121 op/s 2024-04-25T01:35:34.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:34 smithi130 ceph-mon[32702]: pgmap v79: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 392 MiB/s wr, 125 op/s 2024-04-25T01:35:34.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:34 smithi032 ceph-mon[32461]: pgmap v79: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 392 MiB/s wr, 125 op/s 2024-04-25T01:35:34.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:34 smithi028 ceph-mon[29390]: pgmap v79: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 392 MiB/s wr, 125 op/s 2024-04-25T01:35:36.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:36 smithi130 ceph-mon[32702]: pgmap v80: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-25T01:35:36.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:36 smithi032 ceph-mon[32461]: pgmap v80: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-25T01:35:36.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:36 smithi028 ceph-mon[29390]: pgmap v80: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-25T01:35:37.400 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:35:37.401 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:35:37.401 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-25T01:35:37.401 INFO:tasks.workunit.client.0.smithi028.stdout:iozone test complete. 2024-04-25T01:35:37.402 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T01:35:37.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T01:35:37.435 INFO:tasks.workunit:Stopping ['suites/iozone.sh'] on client.0... 2024-04-25T01:35:37.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-25T01:35:37.784 DEBUG:teuthology.parallel:result is None 2024-04-25T01:35:37.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T01:35:37.818 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T01:35:37.818 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-25T01:35:37.829 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-25T01:35:37.829 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T01:35:37.829 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T01:35:37.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-25T01:35:37.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-25T01:35:38.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:38 smithi130 ceph-mon[32702]: pgmap v81: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.0 KiB/s wr, 13 op/s 2024-04-25T01:35:38.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:38 smithi032 ceph-mon[32461]: pgmap v81: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.0 KiB/s wr, 13 op/s 2024-04-25T01:35:38.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:38 smithi028 ceph-mon[29390]: pgmap v81: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.0 KiB/s wr, 13 op/s 2024-04-25T01:35:40.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:40 smithi130 ceph-mon[32702]: pgmap v82: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.6 KiB/s wr, 10 op/s 2024-04-25T01:35:40.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:40 smithi032 ceph-mon[32461]: pgmap v82: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.6 KiB/s wr, 10 op/s 2024-04-25T01:35:40.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:40 smithi028 ceph-mon[29390]: pgmap v82: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.6 KiB/s wr, 10 op/s 2024-04-25T01:35:42.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:42 smithi130 ceph-mon[32702]: pgmap v83: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.6 KiB/s wr, 9 op/s 2024-04-25T01:35:42.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:42 smithi032 ceph-mon[32461]: pgmap v83: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.6 KiB/s wr, 9 op/s 2024-04-25T01:35:42.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:42 smithi028 ceph-mon[29390]: pgmap v83: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.6 KiB/s wr, 9 op/s 2024-04-25T01:35:44.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:44 smithi130 ceph-mon[32702]: pgmap v84: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.7 KiB/s wr, 227 op/s 2024-04-25T01:35:44.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:44 smithi032 ceph-mon[32461]: pgmap v84: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.7 KiB/s wr, 227 op/s 2024-04-25T01:35:44.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:44 smithi028 ceph-mon[29390]: pgmap v84: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.7 KiB/s wr, 227 op/s 2024-04-25T01:35:46.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:46 smithi130 ceph-mon[32702]: pgmap v85: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 4.7 KiB/s wr, 223 op/s 2024-04-25T01:35:46.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:46 smithi032 ceph-mon[32461]: pgmap v85: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 4.7 KiB/s wr, 223 op/s 2024-04-25T01:35:46.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:46 smithi028 ceph-mon[29390]: pgmap v85: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 4.7 KiB/s wr, 223 op/s 2024-04-25T01:35:48.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:48 smithi028 ceph-mon[29390]: pgmap v86: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 226 op/s 2024-04-25T01:35:48.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:48 smithi130 ceph-mon[32702]: pgmap v86: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 226 op/s 2024-04-25T01:35:48.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:48 smithi032 ceph-mon[32461]: pgmap v86: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 226 op/s 2024-04-25T01:35:50.428 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:35:50.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:35:50.455 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:50 smithi028 ceph-mon[29390]: pgmap v87: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 3.2 KiB/s wr, 224 op/s 2024-04-25T01:35:50.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:50 smithi130 ceph-mon[32702]: pgmap v87: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 3.2 KiB/s wr, 224 op/s 2024-04-25T01:35:50.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:50 smithi032 ceph-mon[32461]: pgmap v87: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 3.2 KiB/s wr, 224 op/s 2024-04-25T01:35:50.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.570+0000 7f7e4604c640 1 -- 172.21.15.28:0/2118387030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e40100540 msgr2=0x7f7e40100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:50.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.570+0000 7f7e4604c640 1 --2- 172.21.15.28:0/2118387030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e40100540 0x7f7e40100920 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f7e30009900 tx=0x7f7e3002a410 comp rx=0 tx=0).stop 2024-04-25T01:35:50.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.570+0000 7f7e4604c640 1 -- 172.21.15.28:0/2118387030 shutdown_connections 2024-04-25T01:35:50.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.570+0000 7f7e4604c640 1 --2- 172.21.15.28:0/2118387030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e4010e0f0 0x7f7e401104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.570+0000 7f7e4604c640 1 --2- 172.21.15.28:0/2118387030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e40100e60 0x7f7e4010dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.570+0000 7f7e4604c640 1 --2- 172.21.15.28:0/2118387030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e40100540 0x7f7e40100920 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.570+0000 7f7e4604c640 1 -- 172.21.15.28:0/2118387030 >> 172.21.15.28:0/2118387030 conn(0x7f7e40077740 msgr2=0x7f7e40077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:50.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.570+0000 7f7e4604c640 1 -- 172.21.15.28:0/2118387030 shutdown_connections 2024-04-25T01:35:50.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.570+0000 7f7e4604c640 1 -- 172.21.15.28:0/2118387030 wait complete. 2024-04-25T01:35:50.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.571+0000 7f7e4604c640 1 Processor -- start 2024-04-25T01:35:50.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.571+0000 7f7e4604c640 1 -- start start 2024-04-25T01:35:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.571+0000 7f7e4604c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e40100540 0x7f7e4019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.571+0000 7f7e4604c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e40100e60 0x7f7e4019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.571+0000 7f7e4604c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e4010e0f0 0x7f7e4019eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.571+0000 7f7e4604c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e40113db0 con 0x7f7e40100e60 2024-04-25T01:35:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.571+0000 7f7e4604c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7e40113c30 con 0x7f7e4010e0f0 2024-04-25T01:35:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.571+0000 7f7e4604c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7e40113f30 con 0x7f7e40100540 2024-04-25T01:35:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.571+0000 7f7e3effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e40100e60 0x7f7e4019a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.571+0000 7f7e3effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e40100e60 0x7f7e4019a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46870/0 (socket says 172.21.15.28:46870) 2024-04-25T01:35:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.571+0000 7f7e3effd640 1 -- 172.21.15.28:0/150567708 learned_addr learned my addr 172.21.15.28:0/150567708 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.571+0000 7f7e3ffff640 1 --2- 172.21.15.28:0/150567708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e4010e0f0 0x7f7e4019eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.572+0000 7f7e3f7fe640 1 --2- 172.21.15.28:0/150567708 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e40100540 0x7f7e4019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.572+0000 7f7e3effd640 1 -- 172.21.15.28:0/150567708 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e40100540 msgr2=0x7f7e4019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.572+0000 7f7e3effd640 1 --2- 172.21.15.28:0/150567708 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e40100540 0x7f7e4019a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.572+0000 7f7e3effd640 1 -- 172.21.15.28:0/150567708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e4010e0f0 msgr2=0x7f7e4019eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.572+0000 7f7e3effd640 1 --2- 172.21.15.28:0/150567708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e4010e0f0 0x7f7e4019eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.572+0000 7f7e3effd640 1 -- 172.21.15.28:0/150567708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e4019f2a0 con 0x7f7e40100e60 2024-04-25T01:35:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.572+0000 7f7e3effd640 1 --2- 172.21.15.28:0/150567708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e40100e60 0x7f7e4019a7f0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400b4a0 tx=0x7f7e3400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.572+0000 7f7e3cff9640 1 -- 172.21.15.28:0/150567708 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34014070 con 0x7f7e40100e60 2024-04-25T01:35:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.572+0000 7f7e3cff9640 1 -- 172.21.15.28:0/150567708 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e34010af0 con 0x7f7e40100e60 2024-04-25T01:35:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.572+0000 7f7e4604c640 1 -- 172.21.15.28:0/150567708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e4019f590 con 0x7f7e40100e60 2024-04-25T01:35:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.572+0000 7f7e3cff9640 1 -- 172.21.15.28:0/150567708 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34017d10 con 0x7f7e40100e60 2024-04-25T01:35:50.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.572+0000 7f7e4604c640 1 -- 172.21.15.28:0/150567708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e401a6e20 con 0x7f7e40100e60 2024-04-25T01:35:50.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.574+0000 7f7e3cff9640 1 -- 172.21.15.28:0/150567708 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7e34017070 con 0x7f7e40100e60 2024-04-25T01:35:50.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.575+0000 7f7e48e30740 1 Processor -- start 2024-04-25T01:35:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.575+0000 7f7e48e30740 1 -- start start 2024-04-25T01:35:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.575+0000 7f7e3cff9640 1 --2- 172.21.15.28:0/150567708 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7e280a2d00 0x7f7e280a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.575+0000 7f7e48e30740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e935630 0x55d63e935a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.575+0000 7f7e48e30740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e935f50 0x55d63e9400e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.575+0000 7f7e48e30740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e940620 0x55d63e942a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.575+0000 7f7e48e30740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d63e89a6b0 con 0x55d63e940620 2024-04-25T01:35:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.575+0000 7f7e48e30740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d63e842880 con 0x55d63e935630 2024-04-25T01:35:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.575+0000 7f7e48e30740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55d63e896670 con 0x55d63e935f50 2024-04-25T01:35:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.575+0000 7f7e3ffff640 1 --2- 172.21.15.28:0/150567708 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7e280a2d00 0x7f7e280a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.575+0000 7f7e3cff9640 1 -- 172.21.15.28:0/150567708 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f7e340cea50 con 0x7f7e40100e60 2024-04-25T01:35:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.575+0000 7f7e3f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e935630 0x55d63e935a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.575+0000 7f7e3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e940620 0x55d63e942a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.575+0000 7f7e3f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e935630 0x55d63e935a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:38948/0 (socket says 172.21.15.28:38948) 2024-04-25T01:35:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.575+0000 7f7e3f7fe640 1 -- 172.21.15.28:0/1513566963 learned_addr learned my addr 172.21.15.28:0/1513566963 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:50.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.575+0000 7f7e3effd640 1 --2- 172.21.15.28:0/1513566963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e935f50 0x55d63e9400e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.576+0000 7f7e3f7fe640 1 -- 172.21.15.28:0/1513566963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e935f50 msgr2=0x55d63e9400e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:50.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.576+0000 7f7e3f7fe640 1 --2- 172.21.15.28:0/1513566963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e935f50 0x55d63e9400e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.576+0000 7f7e3f7fe640 1 -- 172.21.15.28:0/1513566963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e940620 msgr2=0x55d63e942a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:50.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.576+0000 7f7e3f7fe640 1 --2- 172.21.15.28:0/1513566963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e940620 0x55d63e942a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.576+0000 7f7e3f7fe640 1 -- 172.21.15.28:0/1513566963 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d63e6787d0 con 0x55d63e935630 2024-04-25T01:35:50.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.576+0000 7f7e3ffff640 1 --2- 172.21.15.28:0/150567708 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7e280a2d00 0x7f7e280a51c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7e38005fd0 tx=0x7f7e3800add0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:50.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.576+0000 7f7e3f7fe640 1 --2- 172.21.15.28:0/1513566963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e935630 0x55d63e935a10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7e30009900 tx=0x7f7e300040f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eca05fb37758493e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:50.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.576+0000 7f7e25ffb640 1 -- 172.21.15.28:0/1513566963 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3003f070 con 0x55d63e935630 2024-04-25T01:35:50.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.576+0000 7f7e25ffb640 1 -- 172.21.15.28:0/1513566963 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e30005500 con 0x55d63e935630 2024-04-25T01:35:50.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.576+0000 7f7e25ffb640 1 -- 172.21.15.28:0/1513566963 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e30035070 con 0x55d63e935630 2024-04-25T01:35:50.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.577+0000 7f7e48e30740 1 -- 172.21.15.28:0/1513566963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e935630 msgr2=0x55d63e935a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.577+0000 7f7e48e30740 1 --2- 172.21.15.28:0/1513566963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e935630 0x55d63e935a10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7e30009900 tx=0x7f7e300040f0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.577+0000 7f7e48e30740 1 -- 172.21.15.28:0/1513566963 shutdown_connections 2024-04-25T01:35:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.577+0000 7f7e48e30740 1 --2- 172.21.15.28:0/1513566963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e940620 0x55d63e942a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.577+0000 7f7e48e30740 1 --2- 172.21.15.28:0/1513566963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e935f50 0x55d63e9400e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.577+0000 7f7e48e30740 1 --2- 172.21.15.28:0/1513566963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e935630 0x55d63e935a10 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.577+0000 7f7e48e30740 1 -- 172.21.15.28:0/1513566963 >> 172.21.15.28:0/1513566963 conn(0x55d63e931950 msgr2=0x55d63e931d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.577+0000 7f7e48e30740 1 -- 172.21.15.28:0/1513566963 shutdown_connections 2024-04-25T01:35:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.577+0000 7f7e48e30740 1 -- 172.21.15.28:0/1513566963 wait complete. 2024-04-25T01:35:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.577+0000 7f7e48e30740 1 -- start start 2024-04-25T01:35:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.577+0000 7f7e48e30740 1 Processor -- start 2024-04-25T01:35:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e48e30740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e935630 0x55d63e93d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e48e30740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e935f50 0x55d63e93da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e48e30740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e940620 0x55d63e93df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e48e30740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d63e9433b0 con 0x55d63e935f50 2024-04-25T01:35:50.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e48e30740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d63e855f50 con 0x55d63e935630 2024-04-25T01:35:50.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e48e30740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55d63e943530 con 0x55d63e940620 2024-04-25T01:35:50.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e48e30740 10 client.0 _ll_register_callbacks cb 0x55d63e891000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:50.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e3effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e935f50 0x55d63e93da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e3effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e935f50 0x55d63e93da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46898/0 (socket says 172.21.15.28:46898) 2024-04-25T01:35:50.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e3f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e935630 0x55d63e93d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e3effd640 1 -- 172.21.15.28:0/3046880211 learned_addr learned my addr 172.21.15.28:0/3046880211 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:50.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e3ffff640 1 --2- 172.21.15.28:0/3046880211 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e940620 0x55d63e93df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e3effd640 1 -- 172.21.15.28:0/3046880211 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e940620 msgr2=0x55d63e93df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:50.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e3effd640 1 --2- 172.21.15.28:0/3046880211 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e940620 0x55d63e93df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e3effd640 1 -- 172.21.15.28:0/3046880211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e935630 msgr2=0x55d63e93d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:50.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e3effd640 1 --2- 172.21.15.28:0/3046880211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e935630 0x55d63e93d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e3effd640 1 -- 172.21.15.28:0/3046880211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d63e5ae3f0 con 0x55d63e935f50 2024-04-25T01:35:50.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e3f7fe640 1 --2- 172.21.15.28:0/3046880211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e935630 0x55d63e93d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:35:50.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e3effd640 1 --2- 172.21.15.28:0/3046880211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e935f50 0x55d63e93da40 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400be60 tx=0x7f7e340970b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71f124055524f736 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:50.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e1effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:50.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e1effd640 1 -- 172.21.15.28:0/3046880211 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3401e040 con 0x55d63e935f50 2024-04-25T01:35:50.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.578+0000 7f7e1effd640 1 -- 172.21.15.28:0/3046880211 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f7e34097350 con 0x55d63e935f50 2024-04-25T01:35:50.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.579+0000 7f7e48e30740 1 -- 172.21.15.28:0/3046880211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e30005500 con 0x55d63e935f50 2024-04-25T01:35:50.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.579+0000 7f7e1effd640 1 -- 172.21.15.28:0/3046880211 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7e340b8050 con 0x55d63e935f50 2024-04-25T01:35:50.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.579+0000 7f7e48e30740 10 client.16410 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:50.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.579+0000 7f7e48e30740 1 -- 172.21.15.28:0/3046880211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d63e56e0f0 con 0x55d63e935f50 2024-04-25T01:35:50.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.579+0000 7f7e1effd640 1 -- 172.21.15.28:0/3046880211 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e30005500 con 0x55d63e935f50 2024-04-25T01:35:50.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.579+0000 7f7e1effd640 1 -- 172.21.15.28:0/3046880211 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7e34063ce0 con 0x55d63e935f50 2024-04-25T01:35:50.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.579+0000 7f7e48e30740 10 client.16410 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:50.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.579+0000 7f7e48e30740 10 client.16410 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:50.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.579+0000 7f7e48e30740 20 client.16410 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:50.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.579+0000 7f7e48e30740 1 --2- 172.21.15.28:0/3046880211 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55d63e9507d0 0x55d63e952c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.579+0000 7f7e48e30740 4 client.16410 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:50.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.579+0000 7f7e48e30740 1 -- 172.21.15.28:0/3046880211 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d63e8a7bf0 con 0x55d63e9507d0 2024-04-25T01:35:50.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.579+0000 7f7e3f7fe640 1 --2- 172.21.15.28:0/3046880211 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55d63e9507d0 0x55d63e952c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.580+0000 7f7e3f7fe640 1 --2- 172.21.15.28:0/3046880211 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55d63e9507d0 0x55d63e952c40 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cf4136f44800c15 server_cookie=77a31d858d439c0d in_seq=0 out_seq=0 2024-04-25T01:35:50.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.580+0000 7f7e1effd640 10 client.16410 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:50.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.581+0000 7f7e1effd640 1 -- 172.21.15.28:0/3046880211 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d63e8a7bf0 con 0x55d63e9507d0 2024-04-25T01:35:50.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.581+0000 7f7e1effd640 10 client.16410 handle_command_reply: tid=0 2024-04-25T01:35:50.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.581+0000 7f7e48e30740 1 client.16410 shutdown 2024-04-25T01:35:50.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.581+0000 7f7e48e30740 1 -- 172.21.15.28:0/3046880211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e935f50 msgr2=0x55d63e93da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:50.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.581+0000 7f7e48e30740 1 --2- 172.21.15.28:0/3046880211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e935f50 0x55d63e93da40 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400be60 tx=0x7f7e340970b0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.581+0000 7f7e48e30740 1 -- 172.21.15.28:0/3046880211 shutdown_connections 2024-04-25T01:35:50.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.581+0000 7f7e48e30740 1 --2- 172.21.15.28:0/3046880211 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55d63e9507d0 0x55d63e952c40 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.581+0000 7f7e48e30740 1 --2- 172.21.15.28:0/3046880211 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e940620 0x55d63e93df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.581+0000 7f7e48e30740 1 --2- 172.21.15.28:0/3046880211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e935f50 0x55d63e93da40 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.581+0000 7f7e1effd640 0 client.16410 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.581+0000 7f7e48e30740 1 --2- 172.21.15.28:0/3046880211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e935630 0x55d63e93d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.581+0000 7f7e48e30740 1 -- 172.21.15.28:0/3046880211 >> 172.21.15.28:0/3046880211 conn(0x55d63e931950 msgr2=0x55d63e937750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.581+0000 7f7e48e30740 1 -- 172.21.15.28:0/3046880211 shutdown_connections 2024-04-25T01:35:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.581+0000 7f7e48e30740 1 -- 172.21.15.28:0/3046880211 wait complete. 2024-04-25T01:35:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.581+0000 7f7e48e30740 20 client.16410 trim_cache size 0 max 16384 2024-04-25T01:35:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.599+0000 7f7e48e30740 1 Processor -- start 2024-04-25T01:35:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.599+0000 7f7e48e30740 1 -- start start 2024-04-25T01:35:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.599+0000 7f7e48e30740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e935630 0x55d63e93fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.599+0000 7f7e48e30740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e935f50 0x55d63e94c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.599+0000 7f7e48e30740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e940620 0x55d63e930970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.599+0000 7f7e48e30740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e401b0d00 con 0x55d63e940620 2024-04-25T01:35:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.599+0000 7f7e48e30740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7e401b2840 con 0x55d63e935630 2024-04-25T01:35:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.599+0000 7f7e48e30740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7e401b13d0 con 0x55d63e935f50 2024-04-25T01:35:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.599+0000 7f7e3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e940620 0x55d63e930970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.599+0000 7f7e3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e940620 0x55d63e930970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46902/0 (socket says 172.21.15.28:46902) 2024-04-25T01:35:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.599+0000 7f7e3ffff640 1 -- 172.21.15.28:0/1073536813 learned_addr learned my addr 172.21.15.28:0/1073536813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.599+0000 7f7e3f7fe640 1 --2- 172.21.15.28:0/1073536813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e935630 0x55d63e93fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.599+0000 7f7e3effd640 1 --2- 172.21.15.28:0/1073536813 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e935f50 0x55d63e94c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.600+0000 7f7e3ffff640 1 -- 172.21.15.28:0/1073536813 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e935f50 msgr2=0x55d63e94c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:50.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.600+0000 7f7e3ffff640 1 --2- 172.21.15.28:0/1073536813 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e935f50 0x55d63e94c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.600+0000 7f7e3ffff640 1 -- 172.21.15.28:0/1073536813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e935630 msgr2=0x55d63e93fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:50.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.600+0000 7f7e3ffff640 1 --2- 172.21.15.28:0/1073536813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e935630 0x55d63e93fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.600+0000 7f7e3ffff640 1 -- 172.21.15.28:0/1073536813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d63e931780 con 0x55d63e940620 2024-04-25T01:35:50.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.600+0000 7f7e3f7fe640 1 --2- 172.21.15.28:0/1073536813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e935630 0x55d63e93fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:50.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.600+0000 7f7e3ffff640 1 --2- 172.21.15.28:0/1073536813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e940620 0x55d63e930970 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f7e38014320 tx=0x7f7e38009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d6dcd24b1eaa3a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:50.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.600+0000 7f7e257fa640 1 -- 172.21.15.28:0/1073536813 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e38004440 con 0x55d63e940620 2024-04-25T01:35:50.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.600+0000 7f7e257fa640 1 -- 172.21.15.28:0/1073536813 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e38026070 con 0x55d63e940620 2024-04-25T01:35:50.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.600+0000 7f7e257fa640 1 -- 172.21.15.28:0/1073536813 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e38012450 con 0x55d63e940620 2024-04-25T01:35:50.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.601+0000 7f7e48e30740 1 -- 172.21.15.28:0/1073536813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e940620 msgr2=0x55d63e930970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:50.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.601+0000 7f7e48e30740 1 --2- 172.21.15.28:0/1073536813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e940620 0x55d63e930970 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f7e38014320 tx=0x7f7e38009000 comp rx=0 tx=0).stop 2024-04-25T01:35:50.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.601+0000 7f7e48e30740 1 -- 172.21.15.28:0/1073536813 shutdown_connections 2024-04-25T01:35:50.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.601+0000 7f7e48e30740 1 --2- 172.21.15.28:0/1073536813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e940620 0x55d63e930970 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.601+0000 7f7e48e30740 1 --2- 172.21.15.28:0/1073536813 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e935f50 0x55d63e94c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.601+0000 7f7e48e30740 1 --2- 172.21.15.28:0/1073536813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e935630 0x55d63e93fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.601+0000 7f7e48e30740 1 -- 172.21.15.28:0/1073536813 >> 172.21.15.28:0/1073536813 conn(0x55d63e931950 msgr2=0x55d63e930260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.601+0000 7f7e48e30740 1 -- 172.21.15.28:0/1073536813 shutdown_connections 2024-04-25T01:35:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.601+0000 7f7e48e30740 1 -- 172.21.15.28:0/1073536813 wait complete. 2024-04-25T01:35:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.601+0000 7f7e48e30740 1 -- start start 2024-04-25T01:35:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.601+0000 7f7e48e30740 1 Processor -- start 2024-04-25T01:35:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.601+0000 7f7e48e30740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e935630 0x55d63e948cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e48e30740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e935f50 0x55d63e939fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e48e30740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e940620 0x55d63e93a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e48e30740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e401b2170 con 0x55d63e935f50 2024-04-25T01:35:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e48e30740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7e401b1aa0 con 0x55d63e940620 2024-04-25T01:35:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e48e30740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7e401aece0 con 0x55d63e935630 2024-04-25T01:35:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e48e30740 10 client.0 _ll_register_callbacks cb 0x7f7e34096fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:50.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e3effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e935f50 0x55d63e939fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e3effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e935f50 0x55d63e939fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46906/0 (socket says 172.21.15.28:46906) 2024-04-25T01:35:50.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e3effd640 1 -- 172.21.15.28:0/4217407688 learned_addr learned my addr 172.21.15.28:0/4217407688 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:50.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e3f7fe640 1 --2- 172.21.15.28:0/4217407688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e935630 0x55d63e948cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e3ffff640 1 --2- 172.21.15.28:0/4217407688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e940620 0x55d63e93a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e3effd640 1 -- 172.21.15.28:0/4217407688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e935630 msgr2=0x55d63e948cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:50.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e3effd640 1 --2- 172.21.15.28:0/4217407688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e935630 0x55d63e948cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e3effd640 1 -- 172.21.15.28:0/4217407688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e940620 msgr2=0x55d63e93a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:50.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e3effd640 1 --2- 172.21.15.28:0/4217407688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e940620 0x55d63e93a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e3effd640 1 -- 172.21.15.28:0/4217407688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d63e9b5d50 con 0x55d63e935f50 2024-04-25T01:35:50.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e3effd640 1 --2- 172.21.15.28:0/4217407688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e935f50 0x55d63e939fd0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f7e340976b0 tx=0x7f7e3409a820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4007ee7311510e85 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:50.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e1effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:50.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e1effd640 1 -- 172.21.15.28:0/4217407688 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3401e070 con 0x55d63e935f50 2024-04-25T01:35:50.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e1effd640 1 -- 172.21.15.28:0/4217407688 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f7e3409a130 con 0x55d63e935f50 2024-04-25T01:35:50.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e48e30740 1 -- 172.21.15.28:0/4217407688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e38026070 con 0x55d63e935f50 2024-04-25T01:35:50.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.602+0000 7f7e1effd640 1 -- 172.21.15.28:0/4217407688 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7e340a3950 con 0x55d63e935f50 2024-04-25T01:35:50.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.603+0000 7f7e48e30740 10 client.16416 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:50.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.603+0000 7f7e48e30740 1 -- 172.21.15.28:0/4217407688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d63e9b6e40 con 0x55d63e935f50 2024-04-25T01:35:50.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.603+0000 7f7e1effd640 1 -- 172.21.15.28:0/4217407688 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e38026070 con 0x55d63e935f50 2024-04-25T01:35:50.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.603+0000 7f7e1effd640 1 -- 172.21.15.28:0/4217407688 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7e340a3c30 con 0x55d63e935f50 2024-04-25T01:35:50.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.603+0000 7f7e48e30740 10 client.16416 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:50.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.603+0000 7f7e48e30740 10 client.16416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:50.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.603+0000 7f7e48e30740 20 client.16416 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:50.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.603+0000 7f7e48e30740 1 --2- 172.21.15.28:0/4217407688 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55d63e9b7500 0x55d63e9b9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.603+0000 7f7e48e30740 4 client.16416 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:35:50.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.603+0000 7f7e48e30740 1 -- 172.21.15.28:0/4217407688 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d63e8a1f40 con 0x55d63e9b7500 2024-04-25T01:35:50.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.603+0000 7f7e3f7fe640 1 --2- 172.21.15.28:0/4217407688 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55d63e9b7500 0x55d63e9b9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.604+0000 7f7e3f7fe640 1 --2- 172.21.15.28:0/4217407688 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55d63e9b7500 0x55d63e9b9920 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65b205197e13366e server_cookie=77b96b2d7d7306b4 in_seq=0 out_seq=0 2024-04-25T01:35:50.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.604+0000 7f7e1effd640 10 client.16416 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:50.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.604+0000 7f7e1effd640 1 -- 172.21.15.28:0/4217407688 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d63e8a1f40 con 0x55d63e9b7500 2024-04-25T01:35:50.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.604+0000 7f7e1effd640 10 client.16416 handle_command_reply: tid=0 2024-04-25T01:35:50.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.604+0000 7f7e48e30740 1 client.16416 shutdown 2024-04-25T01:35:50.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.604+0000 7f7e48e30740 1 -- 172.21.15.28:0/4217407688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e935f50 msgr2=0x55d63e939fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:50.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.604+0000 7f7e48e30740 1 --2- 172.21.15.28:0/4217407688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e935f50 0x55d63e939fd0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f7e340976b0 tx=0x7f7e3409a820 comp rx=0 tx=0).stop 2024-04-25T01:35:50.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.604+0000 7f7e48e30740 1 -- 172.21.15.28:0/4217407688 shutdown_connections 2024-04-25T01:35:50.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.604+0000 7f7e48e30740 1 --2- 172.21.15.28:0/4217407688 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55d63e9b7500 0x55d63e9b9920 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.604+0000 7f7e48e30740 1 --2- 172.21.15.28:0/4217407688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d63e940620 0x55d63e93a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.604+0000 7f7e48e30740 1 --2- 172.21.15.28:0/4217407688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d63e935f50 0x55d63e939fd0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.604+0000 7f7e48e30740 1 --2- 172.21.15.28:0/4217407688 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d63e935630 0x55d63e948cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.604+0000 7f7e1effd640 0 client.16416 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:50.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.604+0000 7f7e48e30740 1 -- 172.21.15.28:0/4217407688 >> 172.21.15.28:0/4217407688 conn(0x55d63e931950 msgr2=0x55d63e93dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:50.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.604+0000 7f7e48e30740 1 -- 172.21.15.28:0/4217407688 shutdown_connections 2024-04-25T01:35:50.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.604+0000 7f7e48e30740 1 -- 172.21.15.28:0/4217407688 wait complete. 2024-04-25T01:35:50.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.604+0000 7f7e48e30740 20 client.16416 trim_cache size 0 max 16384 2024-04-25T01:35:50.587 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:35:50.587 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:35:50.587 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:35:50.587 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:35:50.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.605+0000 7f7e4604c640 1 -- 172.21.15.28:0/150567708 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7e280a2d00 msgr2=0x7f7e280a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:50.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.605+0000 7f7e4604c640 1 --2- 172.21.15.28:0/150567708 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7e280a2d00 0x7f7e280a51c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7e38005fd0 tx=0x7f7e3800add0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.605+0000 7f7e4604c640 1 -- 172.21.15.28:0/150567708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e40100e60 msgr2=0x7f7e4019a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:50.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.605+0000 7f7e4604c640 1 --2- 172.21.15.28:0/150567708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e40100e60 0x7f7e4019a7f0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400b4a0 tx=0x7f7e3400b960 comp rx=0 tx=0).stop 2024-04-25T01:35:50.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.605+0000 7f7e4604c640 1 -- 172.21.15.28:0/150567708 shutdown_connections 2024-04-25T01:35:50.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.605+0000 7f7e4604c640 1 --2- 172.21.15.28:0/150567708 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7e280a2d00 0x7f7e280a51c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.605+0000 7f7e4604c640 1 --2- 172.21.15.28:0/150567708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e4010e0f0 0x7f7e4019eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.605+0000 7f7e4604c640 1 --2- 172.21.15.28:0/150567708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e40100e60 0x7f7e4019a7f0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.605+0000 7f7e4604c640 1 --2- 172.21.15.28:0/150567708 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e40100540 0x7f7e4019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.605+0000 7f7e4604c640 1 -- 172.21.15.28:0/150567708 >> 172.21.15.28:0/150567708 conn(0x7f7e40077740 msgr2=0x7f7e4010f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:50.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.605+0000 7f7e4604c640 1 -- 172.21.15.28:0/150567708 shutdown_connections 2024-04-25T01:35:50.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.605+0000 7f7e4604c640 1 -- 172.21.15.28:0/150567708 wait complete. 2024-04-25T01:35:50.592 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:35:50.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:35:50.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.758+0000 7f3ddb82a640 1 -- 172.21.15.28:0/2327752141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dd41029d0 msgr2=0x7f3dd4102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:50.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.758+0000 7f3ddb82a640 1 --2- 172.21.15.28:0/2327752141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dd41029d0 0x7f3dd4102db0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009ca0 tx=0x7f3dc402a9e0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.759+0000 7f3ddb82a640 1 -- 172.21.15.28:0/2327752141 shutdown_connections 2024-04-25T01:35:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.759+0000 7f3ddb82a640 1 --2- 172.21.15.28:0/2327752141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3dd410bd10 0x7f3dd410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.759+0000 7f3ddb82a640 1 --2- 172.21.15.28:0/2327752141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dd41032f0 0x7f3dd410b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.759+0000 7f3ddb82a640 1 --2- 172.21.15.28:0/2327752141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dd41029d0 0x7f3dd4102db0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.759+0000 7f3ddb82a640 1 -- 172.21.15.28:0/2327752141 >> 172.21.15.28:0/2327752141 conn(0x7f3dd4077740 msgr2=0x7f3dd4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.759+0000 7f3ddb82a640 1 -- 172.21.15.28:0/2327752141 shutdown_connections 2024-04-25T01:35:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.759+0000 7f3ddb82a640 1 -- 172.21.15.28:0/2327752141 wait complete. 2024-04-25T01:35:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.759+0000 7f3ddb82a640 1 Processor -- start 2024-04-25T01:35:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.759+0000 7f3ddb82a640 1 -- start start 2024-04-25T01:35:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.759+0000 7f3ddb82a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dd41029d0 0x7f3dd4197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.759+0000 7f3ddb82a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3dd41032f0 0x7f3dd4198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.759+0000 7f3ddb82a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dd410bd10 0x7f3dd419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.759+0000 7f3ddb82a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd4111b10 con 0x7f3dd410bd10 2024-04-25T01:35:50.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.759+0000 7f3ddb82a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd4111990 con 0x7f3dd41029d0 2024-04-25T01:35:50.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.759+0000 7f3ddb82a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd4111c90 con 0x7f3dd41032f0 2024-04-25T01:35:50.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.760+0000 7f3dd9dc7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dd410bd10 0x7f3dd419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.760+0000 7f3dd9dc7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dd410bd10 0x7f3dd419c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46922/0 (socket says 172.21.15.28:46922) 2024-04-25T01:35:50.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.760+0000 7f3dd9dc7640 1 -- 172.21.15.28:0/2627805839 learned_addr learned my addr 172.21.15.28:0/2627805839 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:50.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.760+0000 7f3dd8dc5640 1 --2- 172.21.15.28:0/2627805839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3dd41032f0 0x7f3dd4198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.760+0000 7f3dd95c6640 1 --2- 172.21.15.28:0/2627805839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dd41029d0 0x7f3dd4197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.760+0000 7f3dd9dc7640 1 -- 172.21.15.28:0/2627805839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3dd41032f0 msgr2=0x7f3dd4198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:50.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.760+0000 7f3dd9dc7640 1 --2- 172.21.15.28:0/2627805839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3dd41032f0 0x7f3dd4198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.760+0000 7f3dd9dc7640 1 -- 172.21.15.28:0/2627805839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dd41029d0 msgr2=0x7f3dd4197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:50.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.760+0000 7f3dd9dc7640 1 --2- 172.21.15.28:0/2627805839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dd41029d0 0x7f3dd4197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.760+0000 7f3dd9dc7640 1 -- 172.21.15.28:0/2627805839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dd419cfe0 con 0x7f3dd410bd10 2024-04-25T01:35:50.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.760+0000 7f3dd9dc7640 1 --2- 172.21.15.28:0/2627805839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dd410bd10 0x7f3dd419c8b0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc00d750 tx=0x7f3dcc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:50.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.760+0000 7f3dd27fc640 1 -- 172.21.15.28:0/2627805839 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc017070 con 0x7f3dd410bd10 2024-04-25T01:35:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.760+0000 7f3dd27fc640 1 -- 172.21.15.28:0/2627805839 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dcc005500 con 0x7f3dd410bd10 2024-04-25T01:35:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.760+0000 7f3ddb82a640 1 -- 172.21.15.28:0/2627805839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dd419d2d0 con 0x7f3dd410bd10 2024-04-25T01:35:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.760+0000 7f3dd27fc640 1 -- 172.21.15.28:0/2627805839 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc004d90 con 0x7f3dd410bd10 2024-04-25T01:35:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.760+0000 7f3ddb82a640 1 -- 172.21.15.28:0/2627805839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dd41a4b60 con 0x7f3dd410bd10 2024-04-25T01:35:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.761+0000 7f3ddb82a640 1 -- 172.21.15.28:0/2627805839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dd4068560 con 0x7f3dd410bd10 2024-04-25T01:35:50.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.762+0000 7f3dd27fc640 1 -- 172.21.15.28:0/2627805839 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3dcc004f30 con 0x7f3dd410bd10 2024-04-25T01:35:50.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.763+0000 7f3dd27fc640 1 --2- 172.21.15.28:0/2627805839 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3dbc0a2d80 0x7f3dbc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:50.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.763+0000 7f3dd95c6640 1 --2- 172.21.15.28:0/2627805839 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3dbc0a2d80 0x7f3dbc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:50.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.764+0000 7f3dd27fc640 1 -- 172.21.15.28:0/2627805839 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f3dcc0cf440 con 0x7f3dd410bd10 2024-04-25T01:35:50.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.764+0000 7f3dd95c6640 1 --2- 172.21.15.28:0/2627805839 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3dbc0a2d80 0x7f3dbc0a5240 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4000f80 tx=0x7f3dc403a040 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:50.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.767+0000 7f3dd27fc640 1 -- 172.21.15.28:0/2627805839 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3dcc09bda0 con 0x7f3dd410bd10 2024-04-25T01:35:50.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.989+0000 7f3ddb82a640 1 -- 172.21.15.28:0/2627805839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3dd402d170 con 0x7f3dd410bd10 2024-04-25T01:35:50.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.989+0000 7f3dd27fc640 1 -- 172.21.15.28:0/2627805839 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7f3dcc09b6a0 con 0x7f3dd410bd10 2024-04-25T01:35:50.966 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:35:50.966 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:35:50.967 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:35:50.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.992+0000 7f3ddb82a640 1 -- 172.21.15.28:0/2627805839 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3dbc0a2d80 msgr2=0x7f3dbc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:50.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.992+0000 7f3ddb82a640 1 --2- 172.21.15.28:0/2627805839 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3dbc0a2d80 0x7f3dbc0a5240 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4000f80 tx=0x7f3dc403a040 comp rx=0 tx=0).stop 2024-04-25T01:35:50.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.992+0000 7f3ddb82a640 1 -- 172.21.15.28:0/2627805839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dd410bd10 msgr2=0x7f3dd419c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:50.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.992+0000 7f3ddb82a640 1 --2- 172.21.15.28:0/2627805839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dd410bd10 0x7f3dd419c8b0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc00d750 tx=0x7f3dcc00dc10 comp rx=0 tx=0).stop 2024-04-25T01:35:50.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.992+0000 7f3ddb82a640 1 -- 172.21.15.28:0/2627805839 shutdown_connections 2024-04-25T01:35:50.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.992+0000 7f3ddb82a640 1 --2- 172.21.15.28:0/2627805839 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3dbc0a2d80 0x7f3dbc0a5240 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.992+0000 7f3ddb82a640 1 --2- 172.21.15.28:0/2627805839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3dd410bd10 0x7f3dd419c8b0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.992+0000 7f3ddb82a640 1 --2- 172.21.15.28:0/2627805839 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3dd41032f0 0x7f3dd4198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.992+0000 7f3ddb82a640 1 --2- 172.21.15.28:0/2627805839 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3dd41029d0 0x7f3dd4197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:50.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.992+0000 7f3ddb82a640 1 -- 172.21.15.28:0/2627805839 >> 172.21.15.28:0/2627805839 conn(0x7f3dd4077740 msgr2=0x7f3dd40fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:50.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.992+0000 7f3ddb82a640 1 -- 172.21.15.28:0/2627805839 shutdown_connections 2024-04-25T01:35:50.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:50.992+0000 7f3ddb82a640 1 -- 172.21.15.28:0/2627805839 wait complete. 2024-04-25T01:35:50.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:35:51.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.107+0000 7f7d6c39e640 1 -- 172.21.15.28:0/503497534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d64069dc0 msgr2=0x7f7d6406a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.107+0000 7f7d6c39e640 1 --2- 172.21.15.28:0/503497534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d64069dc0 0x7f7d6406a240 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f7d54009930 tx=0x7f7d5402a420 comp rx=0 tx=0).stop 2024-04-25T01:35:51.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.107+0000 7f7d6c39e640 1 -- 172.21.15.28:0/503497534 shutdown_connections 2024-04-25T01:35:51.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.107+0000 7f7d6c39e640 1 --2- 172.21.15.28:0/503497534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d6406a780 0x7f7d6410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.107+0000 7f7d6c39e640 1 --2- 172.21.15.28:0/503497534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d64069dc0 0x7f7d6406a240 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.107+0000 7f7d6c39e640 1 --2- 172.21.15.28:0/503497534 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d64107730 0x7f7d64107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.107+0000 7f7d6c39e640 1 -- 172.21.15.28:0/503497534 >> 172.21.15.28:0/503497534 conn(0x7f7d640776a0 msgr2=0x7f7d64077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.107+0000 7f7d6c39e640 1 -- 172.21.15.28:0/503497534 shutdown_connections 2024-04-25T01:35:51.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.107+0000 7f7d6c39e640 1 -- 172.21.15.28:0/503497534 wait complete. 2024-04-25T01:35:51.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.108+0000 7f7d6c39e640 1 Processor -- start 2024-04-25T01:35:51.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.108+0000 7f7d6c39e640 1 -- start start 2024-04-25T01:35:51.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.108+0000 7f7d6c39e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d64069dc0 0x7f7d641980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.108+0000 7f7d6c39e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d6406a780 0x7f7d64198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.108+0000 7f7d6c39e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d64107730 0x7f7d6419c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.108+0000 7f7d6c39e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d64111b40 con 0x7f7d64107730 2024-04-25T01:35:51.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.108+0000 7f7d6c39e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7d641119c0 con 0x7f7d64069dc0 2024-04-25T01:35:51.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.108+0000 7f7d6c39e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7d64111cc0 con 0x7f7d6406a780 2024-04-25T01:35:51.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.108+0000 7f7d6a93b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d64107730 0x7f7d6419c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.108+0000 7f7d6a93b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d64107730 0x7f7d6419c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46944/0 (socket says 172.21.15.28:46944) 2024-04-25T01:35:51.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.108+0000 7f7d6a13a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d64069dc0 0x7f7d641980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.108+0000 7f7d6a93b640 1 -- 172.21.15.28:0/195026213 learned_addr learned my addr 172.21.15.28:0/195026213 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.108+0000 7f7d69939640 1 --2- 172.21.15.28:0/195026213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d6406a780 0x7f7d64198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.108+0000 7f7d6a93b640 1 -- 172.21.15.28:0/195026213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d6406a780 msgr2=0x7f7d64198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.108+0000 7f7d6a93b640 1 --2- 172.21.15.28:0/195026213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d6406a780 0x7f7d64198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.108+0000 7f7d6a93b640 1 -- 172.21.15.28:0/195026213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d64069dc0 msgr2=0x7f7d641980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.108+0000 7f7d6a93b640 1 --2- 172.21.15.28:0/195026213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d64069dc0 0x7f7d641980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.108+0000 7f7d6a93b640 1 -- 172.21.15.28:0/195026213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6419d080 con 0x7f7d64107730 2024-04-25T01:35:51.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.109+0000 7f7d6a93b640 1 --2- 172.21.15.28:0/195026213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d64107730 0x7f7d6419c950 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c00eb10 tx=0x7f7d5c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.110+0000 7f7d5b7fe640 1 -- 172.21.15.28:0/195026213 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c004b20 con 0x7f7d64107730 2024-04-25T01:35:51.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.110+0000 7f7d5b7fe640 1 -- 172.21.15.28:0/195026213 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d5c004cc0 con 0x7f7d64107730 2024-04-25T01:35:51.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.111+0000 7f7d6c39e640 1 -- 172.21.15.28:0/195026213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d6419d370 con 0x7f7d64107730 2024-04-25T01:35:51.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.111+0000 7f7d5b7fe640 1 -- 172.21.15.28:0/195026213 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c010760 con 0x7f7d64107730 2024-04-25T01:35:51.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.111+0000 7f7d6c39e640 1 -- 172.21.15.28:0/195026213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d641a4c00 con 0x7f7d64107730 2024-04-25T01:35:51.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.113+0000 7f7d5b7fe640 1 -- 172.21.15.28:0/195026213 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7d5c005020 con 0x7f7d64107730 2024-04-25T01:35:51.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.113+0000 7f7d6f184740 1 Processor -- start 2024-04-25T01:35:51.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d6f184740 1 -- start start 2024-04-25T01:35:51.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d5b7fe640 1 --2- 172.21.15.28:0/195026213 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7d500a2d80 0x7f7d500a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d6f184740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243d4580 0x55e2243d4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d6f184740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4ea0 0x55e2243df030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d6f184740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2243df570 0x55e2243e1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d6f184740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e224339670 con 0x55e2243d4ea0 2024-04-25T01:35:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d6f184740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e2242e1970 con 0x55e2243df570 2024-04-25T01:35:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d6f184740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55e224335630 con 0x55e2243d4580 2024-04-25T01:35:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d5b7fe640 1 -- 172.21.15.28:0/195026213 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f7d5c0cfcc0 con 0x7f7d64107730 2024-04-25T01:35:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d69939640 1 --2- 172.21.15.28:0/195026213 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7d500a2d80 0x7f7d500a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d69939640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4ea0 0x55e2243df030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d6a13a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243d4580 0x55e2243d4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d69939640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4ea0 0x55e2243df030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46954/0 (socket says 172.21.15.28:46954) 2024-04-25T01:35:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d69939640 1 -- 172.21.15.28:0/3461854799 learned_addr learned my addr 172.21.15.28:0/3461854799 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d6a93b640 1 --2- 172.21.15.28:0/3461854799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2243df570 0x55e2243e1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d6a13a640 1 -- 172.21.15.28:0/3461854799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2243df570 msgr2=0x55e2243e1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d6a13a640 1 --2- 172.21.15.28:0/3461854799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2243df570 0x55e2243e1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d6a13a640 1 -- 172.21.15.28:0/3461854799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4ea0 msgr2=0x55e2243df030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d6a13a640 1 --2- 172.21.15.28:0/3461854799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4ea0 0x55e2243df030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d6a13a640 1 -- 172.21.15.28:0/3461854799 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e224305400 con 0x55e2243d4580 2024-04-25T01:35:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.114+0000 7f7d69939640 1 --2- 172.21.15.28:0/3461854799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4ea0 0x55e2243df030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:35:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.115+0000 7f7d69939640 1 --2- 172.21.15.28:0/195026213 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7d500a2d80 0x7f7d500a5240 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7d54032fd0 tx=0x7f7d54005fd0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.115+0000 7f7d6a13a640 1 --2- 172.21.15.28:0/3461854799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243d4580 0x55e2243d4960 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7d6000aac0 tx=0x7f7d6000af80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f50a5cca227dfde0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.115+0000 7f7d58bfe640 1 -- 172.21.15.28:0/3461854799 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d60004020 con 0x55e2243d4580 2024-04-25T01:35:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.115+0000 7f7d58bfe640 1 -- 172.21.15.28:0/3461854799 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d600041c0 con 0x55e2243d4580 2024-04-25T01:35:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.115+0000 7f7d58bfe640 1 -- 172.21.15.28:0/3461854799 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d60018770 con 0x55e2243d4580 2024-04-25T01:35:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.116+0000 7f7d6f184740 1 -- 172.21.15.28:0/3461854799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243d4580 msgr2=0x55e2243d4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.116+0000 7f7d6f184740 1 --2- 172.21.15.28:0/3461854799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243d4580 0x55e2243d4960 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7d6000aac0 tx=0x7f7d6000af80 comp rx=0 tx=0).stop 2024-04-25T01:35:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.116+0000 7f7d6f184740 1 -- 172.21.15.28:0/3461854799 shutdown_connections 2024-04-25T01:35:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.116+0000 7f7d6f184740 1 --2- 172.21.15.28:0/3461854799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2243df570 0x55e2243e1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.116+0000 7f7d6f184740 1 --2- 172.21.15.28:0/3461854799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4ea0 0x55e2243df030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.116+0000 7f7d6f184740 1 --2- 172.21.15.28:0/3461854799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243d4580 0x55e2243d4960 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.116+0000 7f7d6f184740 1 -- 172.21.15.28:0/3461854799 >> 172.21.15.28:0/3461854799 conn(0x55e2243d08a0 msgr2=0x55e2243d0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.116+0000 7f7d6f184740 1 -- 172.21.15.28:0/3461854799 shutdown_connections 2024-04-25T01:35:51.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.116+0000 7f7d6f184740 1 -- 172.21.15.28:0/3461854799 wait complete. 2024-04-25T01:35:51.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.116+0000 7f7d6f184740 1 -- start start 2024-04-25T01:35:51.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.116+0000 7f7d6f184740 1 Processor -- start 2024-04-25T01:35:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d6f184740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243d4580 0x55e2243deee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d6f184740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4ea0 0x55e2243dc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d6f184740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2243df570 0x55e2243dc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d6f184740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e2243e2360 con 0x55e2243d4ea0 2024-04-25T01:35:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d6f184740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e2242f5050 con 0x55e2243df570 2024-04-25T01:35:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d6f184740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55e2243e24e0 con 0x55e2243d4580 2024-04-25T01:35:51.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d6f184740 10 client.0 _ll_register_callbacks cb 0x55e22404cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:51.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d69939640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4ea0 0x55e2243dc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d69939640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4ea0 0x55e2243dc1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46970/0 (socket says 172.21.15.28:46970) 2024-04-25T01:35:51.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d69939640 1 -- 172.21.15.28:0/1601967866 learned_addr learned my addr 172.21.15.28:0/1601967866 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d6a93b640 1 --2- 172.21.15.28:0/1601967866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2243df570 0x55e2243dc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d6a13a640 1 --2- 172.21.15.28:0/1601967866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243d4580 0x55e2243deee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d69939640 1 -- 172.21.15.28:0/1601967866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243d4580 msgr2=0x55e2243deee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d69939640 1 --2- 172.21.15.28:0/1601967866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243d4580 0x55e2243deee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d69939640 1 -- 172.21.15.28:0/1601967866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2243df570 msgr2=0x55e2243dc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d69939640 1 --2- 172.21.15.28:0/1601967866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2243df570 0x55e2243dc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d69939640 1 -- 172.21.15.28:0/1601967866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2242c6dc0 con 0x55e2243d4ea0 2024-04-25T01:35:51.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d6a13a640 1 --2- 172.21.15.28:0/1601967866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243d4580 0x55e2243deee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:51.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d69939640 1 --2- 172.21.15.28:0/1601967866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4ea0 0x55e2243dc1d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f7d54005780 tx=0x7f7d540058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4094a9babdd4c840 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:51.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d4dffb640 1 -- 172.21.15.28:0/1601967866 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5404f030 con 0x55e2243d4ea0 2024-04-25T01:35:51.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d4dffb640 1 -- 172.21.15.28:0/1601967866 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f7d54003980 con 0x55e2243d4ea0 2024-04-25T01:35:51.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.117+0000 7f7d6f184740 1 -- 172.21.15.28:0/1601967866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d600041c0 con 0x55e2243d4ea0 2024-04-25T01:35:51.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.118+0000 7f7d4dffb640 1 -- 172.21.15.28:0/1601967866 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7d5405c050 con 0x55e2243d4ea0 2024-04-25T01:35:51.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.118+0000 7f7d6f184740 10 client.16434 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:51.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.118+0000 7f7d6f184740 1 -- 172.21.15.28:0/1601967866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e22404d3f0 con 0x55e2243d4ea0 2024-04-25T01:35:51.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.118+0000 7f7d4dffb640 1 -- 172.21.15.28:0/1601967866 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d600041c0 con 0x55e2243d4ea0 2024-04-25T01:35:51.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.118+0000 7f7d4dffb640 1 -- 172.21.15.28:0/1601967866 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7d540564d0 con 0x55e2243d4ea0 2024-04-25T01:35:51.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.118+0000 7f7d6f184740 10 client.16434 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:51.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.118+0000 7f7d6f184740 10 client.16434 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:35:51.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.118+0000 7f7d6f184740 20 client.16434 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:51.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.118+0000 7f7d6f184740 1 --2- 172.21.15.28:0/1601967866 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55e2243ef560 0x55e2243f1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.118+0000 7f7d6f184740 4 client.16434 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:51.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.118+0000 7f7d6f184740 1 -- 172.21.15.28:0/1601967866 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e224346bb0 con 0x55e2243ef560 2024-04-25T01:35:51.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.118+0000 7f7d6a13a640 1 --2- 172.21.15.28:0/1601967866 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55e2243ef560 0x55e2243f1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.119+0000 7f7d4dffb640 10 client.16434 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:35:51.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.119+0000 7f7d6a13a640 1 --2- 172.21.15.28:0/1601967866 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55e2243ef560 0x55e2243f1990 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3bdfa471bfcc9fab server_cookie=3e6fcc2341c7f01b in_seq=0 out_seq=0 2024-04-25T01:35:51.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.120+0000 7f7d4dffb640 1 -- 172.21.15.28:0/1601967866 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e224346bb0 con 0x55e2243ef560 2024-04-25T01:35:51.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.120+0000 7f7d4dffb640 10 client.16434 handle_command_reply: tid=0 2024-04-25T01:35:51.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.120+0000 7f7d6f184740 1 client.16434 shutdown 2024-04-25T01:35:51.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.120+0000 7f7d6f184740 1 -- 172.21.15.28:0/1601967866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4ea0 msgr2=0x55e2243dc1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.120+0000 7f7d6f184740 1 --2- 172.21.15.28:0/1601967866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4ea0 0x55e2243dc1d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f7d54005780 tx=0x7f7d540058d0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.120+0000 7f7d6f184740 1 -- 172.21.15.28:0/1601967866 shutdown_connections 2024-04-25T01:35:51.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.120+0000 7f7d6f184740 1 --2- 172.21.15.28:0/1601967866 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55e2243ef560 0x55e2243f1990 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.120+0000 7f7d6f184740 1 --2- 172.21.15.28:0/1601967866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2243df570 0x55e2243dc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.120+0000 7f7d4dffb640 0 client.16434 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:35:51.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.120+0000 7f7d6f184740 1 --2- 172.21.15.28:0/1601967866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4ea0 0x55e2243dc1d0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.120+0000 7f7d6f184740 1 --2- 172.21.15.28:0/1601967866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243d4580 0x55e2243deee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.120+0000 7f7d6f184740 1 -- 172.21.15.28:0/1601967866 >> 172.21.15.28:0/1601967866 conn(0x55e2243d08a0 msgr2=0x55e2243d66a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.120+0000 7f7d6f184740 1 -- 172.21.15.28:0/1601967866 shutdown_connections 2024-04-25T01:35:51.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.120+0000 7f7d6f184740 1 -- 172.21.15.28:0/1601967866 wait complete. 2024-04-25T01:35:51.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.120+0000 7f7d6f184740 20 client.16434 trim_cache size 0 max 16384 2024-04-25T01:35:51.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.134+0000 7f7d6f184740 1 Processor -- start 2024-04-25T01:35:51.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.134+0000 7f7d6f184740 1 -- start start 2024-04-25T01:35:51.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.134+0000 7f7d6f184740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4580 0x55e2243dea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.134+0000 7f7d6f184740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2243d4ea0 0x55e2243e7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.134+0000 7f7d6f184740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243df570 0x55e2243e7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.134+0000 7f7d6f184740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d641aea70 con 0x55e2243d4580 2024-04-25T01:35:51.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.134+0000 7f7d6f184740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7d641b05b0 con 0x55e2243d4ea0 2024-04-25T01:35:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.134+0000 7f7d6f184740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7d641af140 con 0x55e2243df570 2024-04-25T01:35:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.134+0000 7f7d6a13a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4580 0x55e2243dea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.134+0000 7f7d6a13a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4580 0x55e2243dea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46986/0 (socket says 172.21.15.28:46986) 2024-04-25T01:35:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.134+0000 7f7d6a13a640 1 -- 172.21.15.28:0/479664690 learned_addr learned my addr 172.21.15.28:0/479664690 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.134+0000 7f7d6a13a640 1 -- 172.21.15.28:0/479664690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243df570 msgr2=0x55e2243e7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.134+0000 7f7d6a13a640 1 --2- 172.21.15.28:0/479664690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243df570 0x55e2243e7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.134+0000 7f7d6a13a640 1 -- 172.21.15.28:0/479664690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2243d4ea0 msgr2=0x55e2243e7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.134+0000 7f7d6a13a640 1 --2- 172.21.15.28:0/479664690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2243d4ea0 0x55e2243e7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.134+0000 7f7d6a13a640 1 -- 172.21.15.28:0/479664690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2241177d0 con 0x55e2243d4580 2024-04-25T01:35:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.134+0000 7f7d6a13a640 1 --2- 172.21.15.28:0/479664690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4580 0x55e2243dea10 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f7d6000f850 tx=0x7f7d60003ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7899fbd6a863951a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.134+0000 7f7d4ffff640 1 -- 172.21.15.28:0/479664690 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6000fd90 con 0x55e2243d4580 2024-04-25T01:35:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.135+0000 7f7d4ffff640 1 -- 172.21.15.28:0/479664690 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d60031070 con 0x55e2243d4580 2024-04-25T01:35:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.135+0000 7f7d4ffff640 1 -- 172.21.15.28:0/479664690 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d60003fe0 con 0x55e2243d4580 2024-04-25T01:35:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.135+0000 7f7d6f184740 1 -- 172.21.15.28:0/479664690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4580 msgr2=0x55e2243dea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.135+0000 7f7d6f184740 1 --2- 172.21.15.28:0/479664690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4580 0x55e2243dea10 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f7d6000f850 tx=0x7f7d60003ec0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.135+0000 7f7d6f184740 1 -- 172.21.15.28:0/479664690 shutdown_connections 2024-04-25T01:35:51.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.135+0000 7f7d6f184740 1 --2- 172.21.15.28:0/479664690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243df570 0x55e2243e7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.135+0000 7f7d6f184740 1 --2- 172.21.15.28:0/479664690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2243d4ea0 0x55e2243e7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.135+0000 7f7d6f184740 1 --2- 172.21.15.28:0/479664690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243d4580 0x55e2243dea10 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.135+0000 7f7d6f184740 1 -- 172.21.15.28:0/479664690 >> 172.21.15.28:0/479664690 conn(0x55e2243d08a0 msgr2=0x55e2243ea210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.135+0000 7f7d6f184740 1 -- 172.21.15.28:0/479664690 shutdown_connections 2024-04-25T01:35:51.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.135+0000 7f7d6f184740 1 -- 172.21.15.28:0/479664690 wait complete. 2024-04-25T01:35:51.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.135+0000 7f7d6f184740 1 -- start start 2024-04-25T01:35:51.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.135+0000 7f7d6f184740 1 Processor -- start 2024-04-25T01:35:51.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d6f184740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2243d4580 0x55e2243e5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d6f184740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243d4ea0 0x55e2243e5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d6f184740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243df570 0x55e224454ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d6f184740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d641afee0 con 0x55e2243df570 2024-04-25T01:35:51.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d6f184740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7d641af810 con 0x55e2243d4580 2024-04-25T01:35:51.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d6f184740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7d641aca50 con 0x55e2243d4ea0 2024-04-25T01:35:51.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d6f184740 10 client.0 _ll_register_callbacks cb 0x7f7d54005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:51.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d6a93b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243df570 0x55e224454ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d6a93b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243df570 0x55e224454ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46990/0 (socket says 172.21.15.28:46990) 2024-04-25T01:35:51.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d6a93b640 1 -- 172.21.15.28:0/117158603 learned_addr learned my addr 172.21.15.28:0/117158603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d6a13a640 1 --2- 172.21.15.28:0/117158603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2243d4580 0x55e2243e5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d6a93b640 1 -- 172.21.15.28:0/117158603 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243d4ea0 msgr2=0x55e2243e5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d6a93b640 1 --2- 172.21.15.28:0/117158603 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243d4ea0 0x55e2243e5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d6a93b640 1 -- 172.21.15.28:0/117158603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2243d4580 msgr2=0x55e2243e5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d6a93b640 1 --2- 172.21.15.28:0/117158603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2243d4580 0x55e2243e5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d6a93b640 1 -- 172.21.15.28:0/117158603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e224455020 con 0x55e2243df570 2024-04-25T01:35:51.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d6a93b640 1 --2- 172.21.15.28:0/117158603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243df570 0x55e224454ae0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c098870 tx=0x7f7d5c09b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f3daadcb65d16 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d4dffb640 1 -- 172.21.15.28:0/117158603 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c0a0250 con 0x55e2243df570 2024-04-25T01:35:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d4dffb640 1 -- 172.21.15.28:0/117158603 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f7d5c098510 con 0x55e2243df570 2024-04-25T01:35:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.136+0000 7f7d6f184740 1 -- 172.21.15.28:0/117158603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d60031070 con 0x55e2243df570 2024-04-25T01:35:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.137+0000 7f7d4dffb640 1 -- 172.21.15.28:0/117158603 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7d5c084050 con 0x55e2243df570 2024-04-25T01:35:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.137+0000 7f7d6f184740 10 client.16440 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.137+0000 7f7d6f184740 1 -- 172.21.15.28:0/117158603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e224456220 con 0x55e2243df570 2024-04-25T01:35:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.137+0000 7f7d4dffb640 1 -- 172.21.15.28:0/117158603 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d60031070 con 0x55e2243df570 2024-04-25T01:35:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.137+0000 7f7d4dffb640 1 -- 172.21.15.28:0/117158603 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7d5c005310 con 0x55e2243df570 2024-04-25T01:35:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.137+0000 7f7d6f184740 10 client.16440 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.137+0000 7f7d6f184740 10 client.16440 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:35:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.137+0000 7f7d6f184740 20 client.16440 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.137+0000 7f7d6f184740 1 --2- 172.21.15.28:0/117158603 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55e224456940 0x55e224458d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.137+0000 7f7d6f184740 4 client.16440 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:35:51.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.137+0000 7f7d6f184740 1 -- 172.21.15.28:0/117158603 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e224340f00 con 0x55e224456940 2024-04-25T01:35:51.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.137+0000 7f7d6a13a640 1 --2- 172.21.15.28:0/117158603 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55e224456940 0x55e224458d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.138+0000 7f7d6a13a640 1 --2- 172.21.15.28:0/117158603 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55e224456940 0x55e224458d30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a357a93d92f41904 server_cookie=5731822f89ef5044 in_seq=0 out_seq=0 2024-04-25T01:35:51.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.138+0000 7f7d4dffb640 10 client.16440 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:35:51.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.138+0000 7f7d4dffb640 1 -- 172.21.15.28:0/117158603 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e224340f00 con 0x55e224456940 2024-04-25T01:35:51.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.138+0000 7f7d4dffb640 10 client.16440 handle_command_reply: tid=0 2024-04-25T01:35:51.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.138+0000 7f7d6f184740 1 client.16440 shutdown 2024-04-25T01:35:51.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.138+0000 7f7d6f184740 1 -- 172.21.15.28:0/117158603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243df570 msgr2=0x55e224454ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.138+0000 7f7d6f184740 1 --2- 172.21.15.28:0/117158603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243df570 0x55e224454ae0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c098870 tx=0x7f7d5c09b8c0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.138+0000 7f7d6f184740 1 -- 172.21.15.28:0/117158603 shutdown_connections 2024-04-25T01:35:51.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.138+0000 7f7d6f184740 1 --2- 172.21.15.28:0/117158603 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55e224456940 0x55e224458d30 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.138+0000 7f7d6f184740 1 --2- 172.21.15.28:0/117158603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2243df570 0x55e224454ae0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.138+0000 7f7d6f184740 1 --2- 172.21.15.28:0/117158603 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e2243d4ea0 0x55e2243e5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.138+0000 7f7d4dffb640 0 client.16440 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:35:51.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.138+0000 7f7d6f184740 1 --2- 172.21.15.28:0/117158603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e2243d4580 0x55e2243e5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.139+0000 7f7d6f184740 1 -- 172.21.15.28:0/117158603 >> 172.21.15.28:0/117158603 conn(0x55e2243d08a0 msgr2=0x55e2243eaaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.139+0000 7f7d6f184740 1 -- 172.21.15.28:0/117158603 shutdown_connections 2024-04-25T01:35:51.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.139+0000 7f7d6f184740 1 -- 172.21.15.28:0/117158603 wait complete. 2024-04-25T01:35:51.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.139+0000 7f7d6f184740 20 client.16440 trim_cache size 0 max 16384 2024-04-25T01:35:51.122 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:35:51.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.139+0000 7f7d6c39e640 1 -- 172.21.15.28:0/195026213 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7d500a2d80 msgr2=0x7f7d500a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.139+0000 7f7d6c39e640 1 --2- 172.21.15.28:0/195026213 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7d500a2d80 0x7f7d500a5240 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7d54032fd0 tx=0x7f7d54005fd0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.139+0000 7f7d6c39e640 1 -- 172.21.15.28:0/195026213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d64107730 msgr2=0x7f7d6419c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.139+0000 7f7d6c39e640 1 --2- 172.21.15.28:0/195026213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d64107730 0x7f7d6419c950 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c00eb10 tx=0x7f7d5c00c2d0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.139+0000 7f7d6c39e640 1 -- 172.21.15.28:0/195026213 shutdown_connections 2024-04-25T01:35:51.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.139+0000 7f7d6c39e640 1 --2- 172.21.15.28:0/195026213 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7d500a2d80 0x7f7d500a5240 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.139+0000 7f7d6c39e640 1 --2- 172.21.15.28:0/195026213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d64107730 0x7f7d6419c950 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.139+0000 7f7d6c39e640 1 --2- 172.21.15.28:0/195026213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7d6406a780 0x7f7d64198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.139+0000 7f7d6c39e640 1 --2- 172.21.15.28:0/195026213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7d64069dc0 0x7f7d641980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.139+0000 7f7d6c39e640 1 -- 172.21.15.28:0/195026213 >> 172.21.15.28:0/195026213 conn(0x7f7d640776a0 msgr2=0x7f7d640fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.139+0000 7f7d6c39e640 1 -- 172.21.15.28:0/195026213 shutdown_connections 2024-04-25T01:35:51.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.139+0000 7f7d6c39e640 1 -- 172.21.15.28:0/195026213 wait complete. 2024-04-25T01:35:51.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:35:51.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.255+0000 7fd86b122640 1 -- 172.21.15.28:0/3823021067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd864100fd0 msgr2=0x7fd86410d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.255+0000 7fd86b122640 1 --2- 172.21.15.28:0/3823021067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd864100fd0 0x7fd86410d4e0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fd85c009cd0 tx=0x7fd85c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.255+0000 7fd86b122640 1 -- 172.21.15.28:0/3823021067 shutdown_connections 2024-04-25T01:35:51.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.256+0000 7fd86b122640 1 --2- 172.21.15.28:0/3823021067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd864100fd0 0x7fd86410d4e0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.256+0000 7fd86b122640 1 --2- 172.21.15.28:0/3823021067 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd864100610 0x7fd864100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.256+0000 7fd86b122640 1 --2- 172.21.15.28:0/3823021067 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8641066b0 0x7fd864106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.256+0000 7fd86b122640 1 -- 172.21.15.28:0/3823021067 >> 172.21.15.28:0/3823021067 conn(0x7fd864077740 msgr2=0x7fd864077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.256+0000 7fd86b122640 1 -- 172.21.15.28:0/3823021067 shutdown_connections 2024-04-25T01:35:51.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.256+0000 7fd86b122640 1 -- 172.21.15.28:0/3823021067 wait complete. 2024-04-25T01:35:51.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.256+0000 7fd86b122640 1 Processor -- start 2024-04-25T01:35:51.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.256+0000 7fd86b122640 1 -- start start 2024-04-25T01:35:51.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.256+0000 7fd86b122640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd864100610 0x7fd864197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.256+0000 7fd86b122640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd864100fd0 0x7fd8641984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.256+0000 7fd86b122640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8641066b0 0x7fd86419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.256+0000 7fd86b122640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd864110a30 con 0x7fd864100610 2024-04-25T01:35:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.256+0000 7fd86b122640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641108b0 con 0x7fd864100fd0 2024-04-25T01:35:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.256+0000 7fd86b122640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd864110bb0 con 0x7fd8641066b0 2024-04-25T01:35:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.256+0000 7fd868ebe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd864100610 0x7fd864197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.257+0000 7fd868ebe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd864100610 0x7fd864197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47014/0 (socket says 172.21.15.28:47014) 2024-04-25T01:35:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.257+0000 7fd868ebe640 1 -- 172.21.15.28:0/94687135 learned_addr learned my addr 172.21.15.28:0/94687135 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.257+0000 7fd863fff640 1 --2- 172.21.15.28:0/94687135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd864100fd0 0x7fd8641984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.257+0000 7fd8696bf640 1 --2- 172.21.15.28:0/94687135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8641066b0 0x7fd86419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.257+0000 7fd868ebe640 1 -- 172.21.15.28:0/94687135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8641066b0 msgr2=0x7fd86419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.257+0000 7fd868ebe640 1 --2- 172.21.15.28:0/94687135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8641066b0 0x7fd86419c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.257+0000 7fd868ebe640 1 -- 172.21.15.28:0/94687135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd864100fd0 msgr2=0x7fd8641984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.257+0000 7fd868ebe640 1 --2- 172.21.15.28:0/94687135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd864100fd0 0x7fd8641984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.257+0000 7fd868ebe640 1 -- 172.21.15.28:0/94687135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd86419cf50 con 0x7fd864100610 2024-04-25T01:35:51.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.257+0000 7fd863fff640 1 --2- 172.21.15.28:0/94687135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd864100fd0 0x7fd8641984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:51.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.257+0000 7fd868ebe640 1 --2- 172.21.15.28:0/94687135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd864100610 0x7fd864197f60 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd85400e9c0 tx=0x7fd85400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.257+0000 7fd861ffb640 1 -- 172.21.15.28:0/94687135 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85400cd80 con 0x7fd864100610 2024-04-25T01:35:51.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.257+0000 7fd86b122640 1 -- 172.21.15.28:0/94687135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd86419d240 con 0x7fd864100610 2024-04-25T01:35:51.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.257+0000 7fd861ffb640 1 -- 172.21.15.28:0/94687135 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8540035f0 con 0x7fd864100610 2024-04-25T01:35:51.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.257+0000 7fd861ffb640 1 -- 172.21.15.28:0/94687135 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd854010690 con 0x7fd864100610 2024-04-25T01:35:51.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.257+0000 7fd86b122640 1 -- 172.21.15.28:0/94687135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8641a4ad0 con 0x7fd864100610 2024-04-25T01:35:51.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.260+0000 7fd861ffb640 1 -- 172.21.15.28:0/94687135 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd8540037a0 con 0x7fd864100610 2024-04-25T01:35:51.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.260+0000 7fd86de35740 1 Processor -- start 2024-04-25T01:35:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.260+0000 7fd86de35740 1 -- start start 2024-04-25T01:35:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.260+0000 7fd86de35740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177bc2b0 0x55ef177bc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.260+0000 7fd86de35740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bcc60 0x55ef177cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.260+0000 7fd86de35740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 0x55ef177cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.260+0000 7fd86de35740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ef17721670 con 0x55ef177cb980 2024-04-25T01:35:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.260+0000 7fd86de35740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ef176c9970 con 0x55ef177bcc60 2024-04-25T01:35:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.260+0000 7fd86de35740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ef1771d630 con 0x55ef177bc2b0 2024-04-25T01:35:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.260+0000 7fd861ffb640 1 --2- 172.21.15.28:0/94687135 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd84c0a2d00 0x7fd84c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.260+0000 7fd863fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 0x55ef177cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.260+0000 7fd863fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 0x55ef177cdd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47020/0 (socket says 172.21.15.28:47020) 2024-04-25T01:35:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.260+0000 7fd863fff640 1 -- 172.21.15.28:0/3474863250 learned_addr learned my addr 172.21.15.28:0/3474863250 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.260+0000 7fd868ebe640 1 --2- 172.21.15.28:0/3474863250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bcc60 0x55ef177cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.260+0000 7fd8696bf640 1 --2- 172.21.15.28:0/94687135 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd84c0a2d00 0x7fd84c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.260+0000 7fd861ffb640 1 -- 172.21.15.28:0/94687135 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fd8540cfc40 con 0x7fd864100610 2024-04-25T01:35:51.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.261+0000 7fd8696bf640 1 --2- 172.21.15.28:0/3474863250 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177bc2b0 0x55ef177bc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.261+0000 7fd863fff640 1 -- 172.21.15.28:0/3474863250 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177bc2b0 msgr2=0x55ef177bc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.261+0000 7fd863fff640 1 --2- 172.21.15.28:0/3474863250 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177bc2b0 0x55ef177bc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.261+0000 7fd863fff640 1 -- 172.21.15.28:0/3474863250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bcc60 msgr2=0x55ef177cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.261+0000 7fd863fff640 1 --2- 172.21.15.28:0/3474863250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bcc60 0x55ef177cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.261+0000 7fd863fff640 1 -- 172.21.15.28:0/3474863250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef176ed400 con 0x55ef177cb980 2024-04-25T01:35:51.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.261+0000 7fd8696bf640 1 --2- 172.21.15.28:0/94687135 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd84c0a2d00 0x7fd84c0a51c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd85c009a60 tx=0x7fd85c033000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.261+0000 7fd863fff640 1 --2- 172.21.15.28:0/3474863250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 0x55ef177cdd70 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fd85800bd30 tx=0x7fd85800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6edd6441564e3966 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.261+0000 7fd84affd640 1 -- 172.21.15.28:0/3474863250 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd858015070 con 0x55ef177cb980 2024-04-25T01:35:51.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.261+0000 7fd84affd640 1 -- 172.21.15.28:0/3474863250 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd85800be20 con 0x55ef177cb980 2024-04-25T01:35:51.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.261+0000 7fd84affd640 1 -- 172.21.15.28:0/3474863250 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8580118e0 con 0x55ef177cb980 2024-04-25T01:35:51.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.262+0000 7fd86de35740 1 -- 172.21.15.28:0/3474863250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 msgr2=0x55ef177cdd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.262+0000 7fd86de35740 1 --2- 172.21.15.28:0/3474863250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 0x55ef177cdd70 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fd85800bd30 tx=0x7fd85800f730 comp rx=0 tx=0).stop 2024-04-25T01:35:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.262+0000 7fd86de35740 1 -- 172.21.15.28:0/3474863250 shutdown_connections 2024-04-25T01:35:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.262+0000 7fd86de35740 1 --2- 172.21.15.28:0/3474863250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 0x55ef177cdd70 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.262+0000 7fd86de35740 1 --2- 172.21.15.28:0/3474863250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bcc60 0x55ef177cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.262+0000 7fd86de35740 1 --2- 172.21.15.28:0/3474863250 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177bc2b0 0x55ef177bc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.262+0000 7fd86de35740 1 -- 172.21.15.28:0/3474863250 >> 172.21.15.28:0/3474863250 conn(0x55ef177b88a0 msgr2=0x55ef177b8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.262+0000 7fd86de35740 1 -- 172.21.15.28:0/3474863250 shutdown_connections 2024-04-25T01:35:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.262+0000 7fd86de35740 1 -- 172.21.15.28:0/3474863250 wait complete. 2024-04-25T01:35:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.262+0000 7fd86de35740 1 -- start start 2024-04-25T01:35:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.262+0000 7fd86de35740 1 Processor -- start 2024-04-25T01:35:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd86de35740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bc2b0 0x55ef177cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd86de35740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177bcc60 0x55ef177c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd86de35740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177cb980 0x55ef177c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd86de35740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ef177ce740 con 0x55ef177bcc60 2024-04-25T01:35:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd86de35740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ef176dd050 con 0x55ef177bc2b0 2024-04-25T01:35:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd86de35740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ef177ce8c0 con 0x55ef177cb980 2024-04-25T01:35:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd86de35740 10 client.0 _ll_register_callbacks cb 0x55ef17434be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd863fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177bcc60 0x55ef177c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd863fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177bcc60 0x55ef177c85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47030/0 (socket says 172.21.15.28:47030) 2024-04-25T01:35:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd863fff640 1 -- 172.21.15.28:0/2718393171 learned_addr learned my addr 172.21.15.28:0/2718393171 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd868ebe640 1 --2- 172.21.15.28:0/2718393171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bc2b0 0x55ef177cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd8696bf640 1 --2- 172.21.15.28:0/2718393171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177cb980 0x55ef177c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd863fff640 1 -- 172.21.15.28:0/2718393171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177cb980 msgr2=0x55ef177c8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd863fff640 1 --2- 172.21.15.28:0/2718393171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177cb980 0x55ef177c8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd863fff640 1 -- 172.21.15.28:0/2718393171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bc2b0 msgr2=0x55ef177cb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd863fff640 1 --2- 172.21.15.28:0/2718393171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bc2b0 0x55ef177cb2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd863fff640 1 -- 172.21.15.28:0/2718393171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef176aedc0 con 0x55ef177bcc60 2024-04-25T01:35:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd863fff640 1 --2- 172.21.15.28:0/2718393171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177bcc60 0x55ef177c85e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fd858000c00 tx=0x7fd858005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6369aedbd10992d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd843fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd843fff640 1 -- 172.21.15.28:0/2718393171 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd858015070 con 0x55ef177bcc60 2024-04-25T01:35:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd843fff640 1 -- 172.21.15.28:0/2718393171 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fd858005500 con 0x55ef177bcc60 2024-04-25T01:35:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.263+0000 7fd86de35740 1 -- 172.21.15.28:0/2718393171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd85800be20 con 0x55ef177bcc60 2024-04-25T01:35:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.264+0000 7fd843fff640 1 -- 172.21.15.28:0/2718393171 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd8580046c0 con 0x55ef177bcc60 2024-04-25T01:35:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.264+0000 7fd86de35740 10 client.16452 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.264+0000 7fd86de35740 1 -- 172.21.15.28:0/2718393171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ef174353f0 con 0x55ef177bcc60 2024-04-25T01:35:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.264+0000 7fd843fff640 1 -- 172.21.15.28:0/2718393171 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd85800be20 con 0x55ef177bcc60 2024-04-25T01:35:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.264+0000 7fd843fff640 1 -- 172.21.15.28:0/2718393171 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd858011ca0 con 0x55ef177bcc60 2024-04-25T01:35:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.264+0000 7fd86de35740 10 client.16452 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.264+0000 7fd86de35740 10 client.16452 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:35:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.264+0000 7fd86de35740 20 client.16452 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.264+0000 7fd86de35740 1 --2- 172.21.15.28:0/2718393171 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55ef177db870 0x55ef177ddc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.264+0000 7fd86de35740 4 client.16452 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.264+0000 7fd86de35740 1 -- 172.21.15.28:0/2718393171 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef1772ebb0 con 0x55ef177db870 2024-04-25T01:35:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.264+0000 7fd868ebe640 1 --2- 172.21.15.28:0/2718393171 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55ef177db870 0x55ef177ddc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.265+0000 7fd843fff640 10 client.16452 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:35:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.265+0000 7fd868ebe640 1 --2- 172.21.15.28:0/2718393171 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55ef177db870 0x55ef177ddc60 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7323f36e2df76625 server_cookie=53acc50618c46229 in_seq=0 out_seq=0 2024-04-25T01:35:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.265+0000 7fd843fff640 1 -- 172.21.15.28:0/2718393171 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ef1772ebb0 con 0x55ef177db870 2024-04-25T01:35:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.265+0000 7fd843fff640 10 client.16452 handle_command_reply: tid=0 2024-04-25T01:35:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.265+0000 7fd86de35740 1 client.16452 shutdown 2024-04-25T01:35:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.266+0000 7fd86de35740 1 -- 172.21.15.28:0/2718393171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177bcc60 msgr2=0x55ef177c85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.266+0000 7fd86de35740 1 --2- 172.21.15.28:0/2718393171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177bcc60 0x55ef177c85e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fd858000c00 tx=0x7fd858005040 comp rx=0 tx=0).stop 2024-04-25T01:35:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.266+0000 7fd86de35740 1 -- 172.21.15.28:0/2718393171 shutdown_connections 2024-04-25T01:35:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.266+0000 7fd86de35740 1 --2- 172.21.15.28:0/2718393171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177cb980 0x55ef177c8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.266+0000 7fd86de35740 1 --2- 172.21.15.28:0/2718393171 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55ef177db870 0x55ef177ddc60 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.266+0000 7fd86de35740 1 --2- 172.21.15.28:0/2718393171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177bcc60 0x55ef177c85e0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.266+0000 7fd86de35740 1 --2- 172.21.15.28:0/2718393171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bc2b0 0x55ef177cb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.266+0000 7fd843fff640 0 client.16452 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:35:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.266+0000 7fd86de35740 1 -- 172.21.15.28:0/2718393171 >> 172.21.15.28:0/2718393171 conn(0x55ef177b88a0 msgr2=0x55ef177be6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.266+0000 7fd86de35740 1 -- 172.21.15.28:0/2718393171 shutdown_connections 2024-04-25T01:35:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.266+0000 7fd86de35740 1 -- 172.21.15.28:0/2718393171 wait complete. 2024-04-25T01:35:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.266+0000 7fd86de35740 20 client.16452 trim_cache size 0 max 16384 2024-04-25T01:35:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.279+0000 7fd86de35740 1 Processor -- start 2024-04-25T01:35:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.279+0000 7fd86de35740 1 -- start start 2024-04-25T01:35:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.279+0000 7fd86de35740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bc2b0 0x55ef177d7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.279+0000 7fd86de35740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177bcc60 0x55ef177cae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.279+0000 7fd86de35740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 0x55ef177e3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.279+0000 7fd86de35740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641ac8b0 con 0x55ef177cb980 2024-04-25T01:35:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.279+0000 7fd86de35740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641ae3f0 con 0x55ef177bc2b0 2024-04-25T01:35:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.279+0000 7fd86de35740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641acf80 con 0x55ef177bcc60 2024-04-25T01:35:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.279+0000 7fd8696bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 0x55ef177e3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.279+0000 7fd8696bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 0x55ef177e3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47032/0 (socket says 172.21.15.28:47032) 2024-04-25T01:35:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.279+0000 7fd8696bf640 1 -- 172.21.15.28:0/1153189700 learned_addr learned my addr 172.21.15.28:0/1153189700 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.280+0000 7fd868ebe640 1 --2- 172.21.15.28:0/1153189700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bc2b0 0x55ef177d7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.280+0000 7fd863fff640 1 --2- 172.21.15.28:0/1153189700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177bcc60 0x55ef177cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.280+0000 7fd8696bf640 1 -- 172.21.15.28:0/1153189700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177bcc60 msgr2=0x55ef177cae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.280+0000 7fd8696bf640 1 --2- 172.21.15.28:0/1153189700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177bcc60 0x55ef177cae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.280+0000 7fd8696bf640 1 -- 172.21.15.28:0/1153189700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bc2b0 msgr2=0x55ef177d7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.280+0000 7fd8696bf640 1 --2- 172.21.15.28:0/1153189700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bc2b0 0x55ef177d7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.280+0000 7fd8696bf640 1 -- 172.21.15.28:0/1153189700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef174ff7d0 con 0x55ef177cb980 2024-04-25T01:35:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.280+0000 7fd8696bf640 1 --2- 172.21.15.28:0/1153189700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 0x55ef177e3450 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fd85c009cd0 tx=0x7fd85c02a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c560a8f18467c542 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.280+0000 7fd84a7fc640 1 -- 172.21.15.28:0/1153189700 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c003d40 con 0x55ef177cb980 2024-04-25T01:35:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.280+0000 7fd84a7fc640 1 -- 172.21.15.28:0/1153189700 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd85c036040 con 0x55ef177cb980 2024-04-25T01:35:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.280+0000 7fd84a7fc640 1 -- 172.21.15.28:0/1153189700 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c004860 con 0x55ef177cb980 2024-04-25T01:35:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.281+0000 7fd86de35740 1 -- 172.21.15.28:0/1153189700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 msgr2=0x55ef177e3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.281+0000 7fd86de35740 1 --2- 172.21.15.28:0/1153189700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 0x55ef177e3450 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fd85c009cd0 tx=0x7fd85c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.281+0000 7fd86de35740 1 -- 172.21.15.28:0/1153189700 shutdown_connections 2024-04-25T01:35:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.281+0000 7fd86de35740 1 --2- 172.21.15.28:0/1153189700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 0x55ef177e3450 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.281+0000 7fd86de35740 1 --2- 172.21.15.28:0/1153189700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177bcc60 0x55ef177cae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.281+0000 7fd86de35740 1 --2- 172.21.15.28:0/1153189700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bc2b0 0x55ef177d7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.281+0000 7fd86de35740 1 -- 172.21.15.28:0/1153189700 >> 172.21.15.28:0/1153189700 conn(0x55ef177b88a0 msgr2=0x55ef177c7610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.281+0000 7fd86de35740 1 -- 172.21.15.28:0/1153189700 shutdown_connections 2024-04-25T01:35:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.281+0000 7fd86de35740 1 -- 172.21.15.28:0/1153189700 wait complete. 2024-04-25T01:35:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.281+0000 7fd86de35740 1 -- start start 2024-04-25T01:35:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.281+0000 7fd86de35740 1 Processor -- start 2024-04-25T01:35:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.281+0000 7fd86de35740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177bc2b0 0x55ef17839e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.281+0000 7fd86de35740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bcc60 0x55ef1783a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.281+0000 7fd86de35740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 0x55ef177d34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.281+0000 7fd86de35740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641add20 con 0x55ef177cb980 2024-04-25T01:35:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.281+0000 7fd86de35740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641ad650 con 0x55ef177bcc60 2024-04-25T01:35:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.281+0000 7fd86de35740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641aa890 con 0x55ef177bc2b0 2024-04-25T01:35:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.281+0000 7fd86de35740 10 client.0 _ll_register_callbacks cb 0x7fd854003e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.282+0000 7fd8696bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 0x55ef177d34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.282+0000 7fd8696bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 0x55ef177d34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47042/0 (socket says 172.21.15.28:47042) 2024-04-25T01:35:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.282+0000 7fd8696bf640 1 -- 172.21.15.28:0/3469547007 learned_addr learned my addr 172.21.15.28:0/3469547007 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.282+0000 7fd863fff640 1 --2- 172.21.15.28:0/3469547007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bcc60 0x55ef1783a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.282+0000 7fd868ebe640 1 --2- 172.21.15.28:0/3469547007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177bc2b0 0x55ef17839e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.282+0000 7fd8696bf640 1 -- 172.21.15.28:0/3469547007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177bc2b0 msgr2=0x55ef17839e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.282+0000 7fd8696bf640 1 --2- 172.21.15.28:0/3469547007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177bc2b0 0x55ef17839e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.282+0000 7fd8696bf640 1 -- 172.21.15.28:0/3469547007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bcc60 msgr2=0x55ef1783a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.282+0000 7fd8696bf640 1 --2- 172.21.15.28:0/3469547007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bcc60 0x55ef1783a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.282+0000 7fd8696bf640 1 -- 172.21.15.28:0/3469547007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef177d3b60 con 0x55ef177cb980 2024-04-25T01:35:51.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.282+0000 7fd8696bf640 1 --2- 172.21.15.28:0/3469547007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 0x55ef177d34f0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0047a0 tx=0x7fd85c031bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3556eeeee7661819 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.282+0000 7fd843fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:51.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.282+0000 7fd843fff640 1 -- 172.21.15.28:0/3469547007 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c004010 con 0x55ef177cb980 2024-04-25T01:35:51.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.282+0000 7fd843fff640 1 -- 172.21.15.28:0/3469547007 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fd85c0041b0 con 0x55ef177cb980 2024-04-25T01:35:51.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.282+0000 7fd86de35740 1 -- 172.21.15.28:0/3469547007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd85c036040 con 0x55ef177cb980 2024-04-25T01:35:51.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.282+0000 7fd843fff640 1 -- 172.21.15.28:0/3469547007 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd85c0043b0 con 0x55ef177cb980 2024-04-25T01:35:51.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.282+0000 7fd86de35740 10 client.16458 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:51.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.282+0000 7fd86de35740 1 -- 172.21.15.28:0/3469547007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ef17841760 con 0x55ef177cb980 2024-04-25T01:35:51.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.282+0000 7fd843fff640 1 -- 172.21.15.28:0/3469547007 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd85c036040 con 0x55ef177cb980 2024-04-25T01:35:51.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.283+0000 7fd843fff640 1 -- 172.21.15.28:0/3469547007 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd85c04a070 con 0x55ef177cb980 2024-04-25T01:35:51.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.283+0000 7fd86de35740 10 client.16458 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:51.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.283+0000 7fd86de35740 10 client.16458 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:35:51.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.283+0000 7fd86de35740 20 client.16458 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:51.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.283+0000 7fd86de35740 1 --2- 172.21.15.28:0/3469547007 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55ef17841e50 0x55ef17844270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.283+0000 7fd86de35740 4 client.16458 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:35:51.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.283+0000 7fd86de35740 1 -- 172.21.15.28:0/3469547007 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ef17728f00 con 0x55ef17841e50 2024-04-25T01:35:51.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.283+0000 7fd868ebe640 1 --2- 172.21.15.28:0/3469547007 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55ef17841e50 0x55ef17844270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.283+0000 7fd843fff640 10 client.16458 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:35:51.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.283+0000 7fd868ebe640 1 --2- 172.21.15.28:0/3469547007 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55ef17841e50 0x55ef17844270 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8d749fce917f39d5 server_cookie=d9c609874a22878a in_seq=0 out_seq=0 2024-04-25T01:35:51.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.284+0000 7fd843fff640 1 -- 172.21.15.28:0/3469547007 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ef17728f00 con 0x55ef17841e50 2024-04-25T01:35:51.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.284+0000 7fd843fff640 10 client.16458 handle_command_reply: tid=0 2024-04-25T01:35:51.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.284+0000 7fd86de35740 1 client.16458 shutdown 2024-04-25T01:35:51.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.284+0000 7fd86de35740 1 -- 172.21.15.28:0/3469547007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 msgr2=0x55ef177d34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.284+0000 7fd86de35740 1 --2- 172.21.15.28:0/3469547007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 0x55ef177d34f0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0047a0 tx=0x7fd85c031bd0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.284+0000 7fd86de35740 1 -- 172.21.15.28:0/3469547007 shutdown_connections 2024-04-25T01:35:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.284+0000 7fd86de35740 1 --2- 172.21.15.28:0/3469547007 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55ef17841e50 0x55ef17844270 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.284+0000 7fd86de35740 1 --2- 172.21.15.28:0/3469547007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef177cb980 0x55ef177d34f0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.284+0000 7fd86de35740 1 --2- 172.21.15.28:0/3469547007 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef177bcc60 0x55ef1783a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.284+0000 7fd843fff640 0 client.16458 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:35:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.284+0000 7fd86de35740 1 --2- 172.21.15.28:0/3469547007 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef177bc2b0 0x55ef17839e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.284+0000 7fd86de35740 1 -- 172.21.15.28:0/3469547007 >> 172.21.15.28:0/3469547007 conn(0x55ef177b88a0 msgr2=0x55ef177bde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.284+0000 7fd86de35740 1 -- 172.21.15.28:0/3469547007 shutdown_connections 2024-04-25T01:35:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.284+0000 7fd86de35740 1 -- 172.21.15.28:0/3469547007 wait complete. 2024-04-25T01:35:51.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.284+0000 7fd86de35740 20 client.16458 trim_cache size 0 max 16384 2024-04-25T01:35:51.268 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:35:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.285+0000 7fd86b122640 1 -- 172.21.15.28:0/94687135 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd84c0a2d00 msgr2=0x7fd84c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.285+0000 7fd86b122640 1 --2- 172.21.15.28:0/94687135 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd84c0a2d00 0x7fd84c0a51c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd85c009a60 tx=0x7fd85c033000 comp rx=0 tx=0).stop 2024-04-25T01:35:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.285+0000 7fd86b122640 1 -- 172.21.15.28:0/94687135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd864100610 msgr2=0x7fd864197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.285+0000 7fd86b122640 1 --2- 172.21.15.28:0/94687135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd864100610 0x7fd864197f60 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd85400e9c0 tx=0x7fd85400ee80 comp rx=0 tx=0).stop 2024-04-25T01:35:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.285+0000 7fd86b122640 1 -- 172.21.15.28:0/94687135 shutdown_connections 2024-04-25T01:35:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.285+0000 7fd86b122640 1 --2- 172.21.15.28:0/94687135 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd84c0a2d00 0x7fd84c0a51c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.285+0000 7fd86b122640 1 --2- 172.21.15.28:0/94687135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd8641066b0 0x7fd86419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.285+0000 7fd86b122640 1 --2- 172.21.15.28:0/94687135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd864100fd0 0x7fd8641984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.285+0000 7fd86b122640 1 --2- 172.21.15.28:0/94687135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd864100610 0x7fd864197f60 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.285+0000 7fd86b122640 1 -- 172.21.15.28:0/94687135 >> 172.21.15.28:0/94687135 conn(0x7fd864077740 msgr2=0x7fd8640ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.285+0000 7fd86b122640 1 -- 172.21.15.28:0/94687135 shutdown_connections 2024-04-25T01:35:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.285+0000 7fd86b122640 1 -- 172.21.15.28:0/94687135 wait complete. 2024-04-25T01:35:51.271 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:51 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2627805839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:35:51.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:35:51.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.400+0000 7f8085e31640 1 -- 172.21.15.28:0/1192720897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80801029c0 msgr2=0x7f8080102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.400+0000 7f8085e31640 1 --2- 172.21.15.28:0/1192720897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80801029c0 0x7f8080102da0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f8070009ca0 tx=0x7f807002a9e0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.400+0000 7f8085e31640 1 -- 172.21.15.28:0/1192720897 shutdown_connections 2024-04-25T01:35:51.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.400+0000 7f8085e31640 1 --2- 172.21.15.28:0/1192720897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f808010bd00 0x7f808010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.400+0000 7f8085e31640 1 --2- 172.21.15.28:0/1192720897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f80801032e0 0x7f808010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.400+0000 7f8085e31640 1 --2- 172.21.15.28:0/1192720897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80801029c0 0x7f8080102da0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.400+0000 7f8085e31640 1 -- 172.21.15.28:0/1192720897 >> 172.21.15.28:0/1192720897 conn(0x7f8080077740 msgr2=0x7f8080077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.400+0000 7f8085e31640 1 -- 172.21.15.28:0/1192720897 shutdown_connections 2024-04-25T01:35:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.400+0000 7f8085e31640 1 -- 172.21.15.28:0/1192720897 wait complete. 2024-04-25T01:35:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.400+0000 7f8085e31640 1 Processor -- start 2024-04-25T01:35:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.400+0000 7f8085e31640 1 -- start start 2024-04-25T01:35:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f8085e31640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f80801029c0 0x7f8080100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f8085e31640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80801032e0 0x7f8080100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f8085e31640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f808010bd00 0x7f8080101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f8085e31640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8080111b00 con 0x7f808010bd00 2024-04-25T01:35:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f8085e31640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8080111980 con 0x7f80801032e0 2024-04-25T01:35:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f8085e31640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8080111c80 con 0x7f80801029c0 2024-04-25T01:35:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f807ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f808010bd00 0x7f8080101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f807ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f808010bd00 0x7f8080101090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47064/0 (socket says 172.21.15.28:47064) 2024-04-25T01:35:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f807ffff640 1 -- 172.21.15.28:0/810023936 learned_addr learned my addr 172.21.15.28:0/810023936 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f807f7fe640 1 --2- 172.21.15.28:0/810023936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f80801029c0 0x7f8080100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f807effd640 1 --2- 172.21.15.28:0/810023936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80801032e0 0x7f8080100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f807ffff640 1 -- 172.21.15.28:0/810023936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f80801029c0 msgr2=0x7f8080100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f807ffff640 1 --2- 172.21.15.28:0/810023936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f80801029c0 0x7f8080100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f807ffff640 1 -- 172.21.15.28:0/810023936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80801032e0 msgr2=0x7f8080100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f807ffff640 1 --2- 172.21.15.28:0/810023936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80801032e0 0x7f8080100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f807ffff640 1 -- 172.21.15.28:0/810023936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80801a6440 con 0x7f808010bd00 2024-04-25T01:35:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f807ffff640 1 --2- 172.21.15.28:0/810023936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f808010bd00 0x7f8080101090 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f807800d750 tx=0x7f807800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f807cff9640 1 -- 172.21.15.28:0/810023936 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8078017070 con 0x7f808010bd00 2024-04-25T01:35:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f807cff9640 1 -- 172.21.15.28:0/810023936 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8078005500 con 0x7f808010bd00 2024-04-25T01:35:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f8085e31640 1 -- 172.21.15.28:0/810023936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80801a6730 con 0x7f808010bd00 2024-04-25T01:35:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f807cff9640 1 -- 172.21.15.28:0/810023936 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8078004d90 con 0x7f808010bd00 2024-04-25T01:35:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.401+0000 7f8085e31640 1 -- 172.21.15.28:0/810023936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80801a6c30 con 0x7f808010bd00 2024-04-25T01:35:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.404+0000 7f807cff9640 1 -- 172.21.15.28:0/810023936 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8078004f30 con 0x7f808010bd00 2024-04-25T01:35:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.404+0000 7f8088c1a740 1 Processor -- start 2024-04-25T01:35:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.404+0000 7f8088c1a740 1 -- start start 2024-04-25T01:35:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.404+0000 7f8088c1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0580 0x556b4cdc0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.404+0000 7f8088c1a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdc0ea0 0x556b4cdcb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.404+0000 7f807cff9640 1 --2- 172.21.15.28:0/810023936 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f80680a2d30 0x7f80680a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.404+0000 7f8088c1a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdcb570 0x556b4cdcd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.404+0000 7f8088c1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556b4cd25670 con 0x556b4cdc0580 2024-04-25T01:35:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.404+0000 7f8088c1a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556b4cccd970 con 0x556b4cdc0ea0 2024-04-25T01:35:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.404+0000 7f8088c1a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x556b4cd21630 con 0x556b4cdcb570 2024-04-25T01:35:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.404+0000 7f807effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0580 0x556b4cdc0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.404+0000 7f807f7fe640 1 --2- 172.21.15.28:0/810023936 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f80680a2d30 0x7f80680a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.404+0000 7f807effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0580 0x556b4cdc0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47066/0 (socket says 172.21.15.28:47066) 2024-04-25T01:35:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.404+0000 7f807effd640 1 -- 172.21.15.28:0/3950983835 learned_addr learned my addr 172.21.15.28:0/3950983835 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.404+0000 7f807cff9640 1 -- 172.21.15.28:0/810023936 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f80780cf1a0 con 0x7f808010bd00 2024-04-25T01:35:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.404+0000 7f807ffff640 1 --2- 172.21.15.28:0/3950983835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdcb570 0x556b4cdcd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.405+0000 7f807effd640 1 --2- 172.21.15.28:0/3950983835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdc0ea0 0x556b4cdcb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.405+0000 7f807effd640 1 -- 172.21.15.28:0/3950983835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdcb570 msgr2=0x556b4cdcd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.405+0000 7f807effd640 1 --2- 172.21.15.28:0/3950983835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdcb570 0x556b4cdcd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.405+0000 7f807effd640 1 -- 172.21.15.28:0/3950983835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdc0ea0 msgr2=0x556b4cdcb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.405+0000 7f807effd640 1 --2- 172.21.15.28:0/3950983835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdc0ea0 0x556b4cdcb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.405+0000 7f807effd640 1 -- 172.21.15.28:0/3950983835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b4ccf1400 con 0x556b4cdc0580 2024-04-25T01:35:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.405+0000 7f807effd640 1 --2- 172.21.15.28:0/3950983835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0580 0x556b4cdc0960 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f80740144f0 tx=0x7f807400b500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a870ff068346289 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.405+0000 7f807f7fe640 1 --2- 172.21.15.28:0/810023936 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f80680a2d30 0x7f80680a51f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8070000f80 tx=0x7f807003a040 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.405+0000 7f8065ffb640 1 -- 172.21.15.28:0/3950983835 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074005020 con 0x556b4cdc0580 2024-04-25T01:35:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.405+0000 7f8065ffb640 1 -- 172.21.15.28:0/3950983835 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80740047a0 con 0x556b4cdc0580 2024-04-25T01:35:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.405+0000 7f8065ffb640 1 -- 172.21.15.28:0/3950983835 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074012770 con 0x556b4cdc0580 2024-04-25T01:35:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.406+0000 7f8088c1a740 1 -- 172.21.15.28:0/3950983835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0580 msgr2=0x556b4cdc0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.406+0000 7f8088c1a740 1 --2- 172.21.15.28:0/3950983835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0580 0x556b4cdc0960 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f80740144f0 tx=0x7f807400b500 comp rx=0 tx=0).stop 2024-04-25T01:35:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.406+0000 7f8088c1a740 1 -- 172.21.15.28:0/3950983835 shutdown_connections 2024-04-25T01:35:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.406+0000 7f8088c1a740 1 --2- 172.21.15.28:0/3950983835 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdcb570 0x556b4cdcd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.406+0000 7f8088c1a740 1 --2- 172.21.15.28:0/3950983835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdc0ea0 0x556b4cdcb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.406+0000 7f8088c1a740 1 --2- 172.21.15.28:0/3950983835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0580 0x556b4cdc0960 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.406+0000 7f8088c1a740 1 -- 172.21.15.28:0/3950983835 >> 172.21.15.28:0/3950983835 conn(0x556b4cdbc8a0 msgr2=0x556b4cdbcc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.406+0000 7f8088c1a740 1 -- 172.21.15.28:0/3950983835 shutdown_connections 2024-04-25T01:35:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.406+0000 7f8088c1a740 1 -- 172.21.15.28:0/3950983835 wait complete. 2024-04-25T01:35:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.406+0000 7f8088c1a740 1 -- start start 2024-04-25T01:35:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.406+0000 7f8088c1a740 1 Processor -- start 2024-04-25T01:35:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.406+0000 7f8088c1a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdc0580 0x556b4cdcaea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.406+0000 7f8088c1a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdc0ea0 0x556b4cdc8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f8088c1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdcb570 0x556b4cdc86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f8088c1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556b4cdce300 con 0x556b4cdcb570 2024-04-25T01:35:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f8088c1a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556b4cce1050 con 0x556b4cdc0580 2024-04-25T01:35:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f8088c1a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x556b4cdce480 con 0x556b4cdc0ea0 2024-04-25T01:35:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f8088c1a740 10 client.0 _ll_register_callbacks cb 0x556b4ca38be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f807ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdcb570 0x556b4cdc86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f807ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdcb570 0x556b4cdc86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47074/0 (socket says 172.21.15.28:47074) 2024-04-25T01:35:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f807ffff640 1 -- 172.21.15.28:0/3092120847 learned_addr learned my addr 172.21.15.28:0/3092120847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f807effd640 1 --2- 172.21.15.28:0/3092120847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdc0ea0 0x556b4cdc8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f807f7fe640 1 --2- 172.21.15.28:0/3092120847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdc0580 0x556b4cdcaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f807ffff640 1 -- 172.21.15.28:0/3092120847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdc0ea0 msgr2=0x556b4cdc8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f807ffff640 1 --2- 172.21.15.28:0/3092120847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdc0ea0 0x556b4cdc8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f807ffff640 1 -- 172.21.15.28:0/3092120847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdc0580 msgr2=0x556b4cdcaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f807ffff640 1 --2- 172.21.15.28:0/3092120847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdc0580 0x556b4cdcaea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f807ffff640 1 -- 172.21.15.28:0/3092120847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b4ccb2dc0 con 0x556b4cdcb570 2024-04-25T01:35:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f807f7fe640 1 --2- 172.21.15.28:0/3092120847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdc0580 0x556b4cdcaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f807ffff640 1 --2- 172.21.15.28:0/3092120847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdcb570 0x556b4cdc86d0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f80780cc610 tx=0x7f807809afe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4285719783b4a720 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f805effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f805effd640 1 -- 172.21.15.28:0/3092120847 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8078020070 con 0x556b4cdcb570 2024-04-25T01:35:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f805effd640 1 -- 172.21.15.28:0/3092120847 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f80780cc830 con 0x556b4cdcb570 2024-04-25T01:35:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f8088c1a740 1 -- 172.21.15.28:0/3092120847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80740047a0 con 0x556b4cdcb570 2024-04-25T01:35:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f805effd640 1 -- 172.21.15.28:0/3092120847 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f807800de60 con 0x556b4cdcb570 2024-04-25T01:35:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f8088c1a740 10 client.16470 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.407+0000 7f8088c1a740 1 -- 172.21.15.28:0/3092120847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556b4ca393f0 con 0x556b4cdcb570 2024-04-25T01:35:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.408+0000 7f805effd640 1 -- 172.21.15.28:0/3092120847 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80740047a0 con 0x556b4cdcb570 2024-04-25T01:35:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.408+0000 7f805effd640 1 -- 172.21.15.28:0/3092120847 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f80780979f0 con 0x556b4cdcb570 2024-04-25T01:35:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.408+0000 7f8088c1a740 10 client.16470 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.408+0000 7f8088c1a740 10 client.16470 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:35:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.408+0000 7f8088c1a740 20 client.16470 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.408+0000 7f8088c1a740 1 --2- 172.21.15.28:0/3092120847 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x556b4cddb4e0 0x556b4cddd910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.408+0000 7f8088c1a740 4 client.16470 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.408+0000 7f8088c1a740 1 -- 172.21.15.28:0/3092120847 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b4cd32bb0 con 0x556b4cddb4e0 2024-04-25T01:35:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.408+0000 7f807f7fe640 1 --2- 172.21.15.28:0/3092120847 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x556b4cddb4e0 0x556b4cddd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.409+0000 7f805effd640 10 client.16470 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:35:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.409+0000 7f807f7fe640 1 --2- 172.21.15.28:0/3092120847 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x556b4cddb4e0 0x556b4cddd910 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cb8202d1db09a7f1 server_cookie=b8d8ac6d4ee0aaf3 in_seq=0 out_seq=0 2024-04-25T01:35:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.409+0000 7f805effd640 1 -- 172.21.15.28:0/3092120847 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556b4cd32bb0 con 0x556b4cddb4e0 2024-04-25T01:35:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.409+0000 7f805effd640 10 client.16470 handle_command_reply: tid=0 2024-04-25T01:35:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.409+0000 7f8088c1a740 1 client.16470 shutdown 2024-04-25T01:35:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.410+0000 7f8088c1a740 1 -- 172.21.15.28:0/3092120847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdcb570 msgr2=0x556b4cdc86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.410+0000 7f8088c1a740 1 --2- 172.21.15.28:0/3092120847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdcb570 0x556b4cdc86d0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f80780cc610 tx=0x7f807809afe0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.410+0000 7f8088c1a740 1 -- 172.21.15.28:0/3092120847 shutdown_connections 2024-04-25T01:35:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.410+0000 7f8088c1a740 1 --2- 172.21.15.28:0/3092120847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdcb570 0x556b4cdc86d0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.410+0000 7f8088c1a740 1 --2- 172.21.15.28:0/3092120847 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x556b4cddb4e0 0x556b4cddd910 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.410+0000 7f8088c1a740 1 --2- 172.21.15.28:0/3092120847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdc0ea0 0x556b4cdc8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.410+0000 7f8088c1a740 1 --2- 172.21.15.28:0/3092120847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdc0580 0x556b4cdcaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.410+0000 7f805effd640 0 client.16470 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:35:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.410+0000 7f8088c1a740 1 -- 172.21.15.28:0/3092120847 >> 172.21.15.28:0/3092120847 conn(0x556b4cdbc8a0 msgr2=0x556b4cdbbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.410+0000 7f8088c1a740 1 -- 172.21.15.28:0/3092120847 shutdown_connections 2024-04-25T01:35:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.410+0000 7f8088c1a740 1 -- 172.21.15.28:0/3092120847 wait complete. 2024-04-25T01:35:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.410+0000 7f8088c1a740 20 client.16470 trim_cache size 0 max 16384 2024-04-25T01:35:51.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.423+0000 7f8088c1a740 1 Processor -- start 2024-04-25T01:35:51.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.423+0000 7f8088c1a740 1 -- start start 2024-04-25T01:35:51.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.423+0000 7f8088c1a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdc0580 0x556b4cdd71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.423+0000 7f8088c1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0ea0 0x556b4cdc6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.423+0000 7f8088c1a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdcb570 0x556b4cdca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.423+0000 7f8088c1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f80801b0bd0 con 0x556b4cdc0ea0 2024-04-25T01:35:51.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.423+0000 7f8088c1a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f80801b2710 con 0x556b4cdc0580 2024-04-25T01:35:51.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.423+0000 7f8088c1a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f80801b12a0 con 0x556b4cdcb570 2024-04-25T01:35:51.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.423+0000 7f807effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0ea0 0x556b4cdc6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.423+0000 7f807effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0ea0 0x556b4cdc6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47076/0 (socket says 172.21.15.28:47076) 2024-04-25T01:35:51.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.423+0000 7f807effd640 1 -- 172.21.15.28:0/2903834686 learned_addr learned my addr 172.21.15.28:0/2903834686 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.424+0000 7f807f7fe640 1 --2- 172.21.15.28:0/2903834686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdc0580 0x556b4cdd71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.424+0000 7f807ffff640 1 --2- 172.21.15.28:0/2903834686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdcb570 0x556b4cdca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.424+0000 7f807effd640 1 -- 172.21.15.28:0/2903834686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdcb570 msgr2=0x556b4cdca9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.424+0000 7f807effd640 1 --2- 172.21.15.28:0/2903834686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdcb570 0x556b4cdca9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.424+0000 7f807effd640 1 -- 172.21.15.28:0/2903834686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdc0580 msgr2=0x556b4cdd71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.424+0000 7f807effd640 1 --2- 172.21.15.28:0/2903834686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdc0580 0x556b4cdd71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.424+0000 7f807effd640 1 -- 172.21.15.28:0/2903834686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b4cb037d0 con 0x556b4cdc0ea0 2024-04-25T01:35:51.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.424+0000 7f807f7fe640 1 --2- 172.21.15.28:0/2903834686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdc0580 0x556b4cdd71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:51.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.424+0000 7f807effd640 1 --2- 172.21.15.28:0/2903834686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0ea0 0x556b4cdc6930 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f8074014e40 tx=0x7f8074005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c81da6b5071752 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.424+0000 7f80657fa640 1 -- 172.21.15.28:0/2903834686 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074005000 con 0x556b4cdc0ea0 2024-04-25T01:35:51.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.424+0000 7f80657fa640 1 -- 172.21.15.28:0/2903834686 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8074005680 con 0x556b4cdc0ea0 2024-04-25T01:35:51.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.424+0000 7f80657fa640 1 -- 172.21.15.28:0/2903834686 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074012510 con 0x556b4cdc0ea0 2024-04-25T01:35:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.425+0000 7f8088c1a740 1 -- 172.21.15.28:0/2903834686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0ea0 msgr2=0x556b4cdc6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.425+0000 7f8088c1a740 1 --2- 172.21.15.28:0/2903834686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0ea0 0x556b4cdc6930 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f8074014e40 tx=0x7f8074005f30 comp rx=0 tx=0).stop 2024-04-25T01:35:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.425+0000 7f8088c1a740 1 -- 172.21.15.28:0/2903834686 shutdown_connections 2024-04-25T01:35:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.425+0000 7f8088c1a740 1 --2- 172.21.15.28:0/2903834686 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdcb570 0x556b4cdca9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.425+0000 7f8088c1a740 1 --2- 172.21.15.28:0/2903834686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0ea0 0x556b4cdc6930 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.425+0000 7f8088c1a740 1 --2- 172.21.15.28:0/2903834686 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdc0580 0x556b4cdd71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.425+0000 7f8088c1a740 1 -- 172.21.15.28:0/2903834686 >> 172.21.15.28:0/2903834686 conn(0x556b4cdbc8a0 msgr2=0x556b4cdc36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.425+0000 7f8088c1a740 1 -- 172.21.15.28:0/2903834686 shutdown_connections 2024-04-25T01:35:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.425+0000 7f8088c1a740 1 -- 172.21.15.28:0/2903834686 wait complete. 2024-04-25T01:35:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.425+0000 7f8088c1a740 1 -- start start 2024-04-25T01:35:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.425+0000 7f8088c1a740 1 Processor -- start 2024-04-25T01:35:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.425+0000 7f8088c1a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdc0580 0x556b4cdcf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.425+0000 7f8088c1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0ea0 0x556b4cdcf820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.425+0000 7f8088c1a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdcb570 0x556b4cdcfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f8088c1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f80801b2040 con 0x556b4cdc0ea0 2024-04-25T01:35:51.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f8088c1a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f80801b1970 con 0x556b4cdcb570 2024-04-25T01:35:51.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f8088c1a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f80801aebb0 con 0x556b4cdc0580 2024-04-25T01:35:51.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f8088c1a740 10 client.0 _ll_register_callbacks cb 0x7f807800d6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:51.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f807effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0ea0 0x556b4cdcf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f807effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0ea0 0x556b4cdcf820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47082/0 (socket says 172.21.15.28:47082) 2024-04-25T01:35:51.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f807effd640 1 -- 172.21.15.28:0/4216696138 learned_addr learned my addr 172.21.15.28:0/4216696138 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f807ffff640 1 --2- 172.21.15.28:0/4216696138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdcb570 0x556b4cdcfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f807f7fe640 1 --2- 172.21.15.28:0/4216696138 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdc0580 0x556b4cdcf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f807effd640 1 -- 172.21.15.28:0/4216696138 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdc0580 msgr2=0x556b4cdcf2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f807effd640 1 --2- 172.21.15.28:0/4216696138 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdc0580 0x556b4cdcf2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f807effd640 1 -- 172.21.15.28:0/4216696138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdcb570 msgr2=0x556b4cdcfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f807effd640 1 --2- 172.21.15.28:0/4216696138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdcb570 0x556b4cdcfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f807effd640 1 -- 172.21.15.28:0/4216696138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b4ce40c50 con 0x556b4cdc0ea0 2024-04-25T01:35:51.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f807effd640 1 --2- 172.21.15.28:0/4216696138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0ea0 0x556b4cdcf820 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f8074005e10 tx=0x7f8074004f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c38b8346ea9e457 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f805effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:51.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f805effd640 1 -- 172.21.15.28:0/4216696138 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807402c070 con 0x556b4cdc0ea0 2024-04-25T01:35:51.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f805effd640 1 -- 172.21.15.28:0/4216696138 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8074005300 con 0x556b4cdc0ea0 2024-04-25T01:35:51.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f8088c1a740 1 -- 172.21.15.28:0/4216696138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8074005680 con 0x556b4cdc0ea0 2024-04-25T01:35:51.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f805effd640 1 -- 172.21.15.28:0/4216696138 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8074038050 con 0x556b4cdc0ea0 2024-04-25T01:35:51.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f8088c1a740 10 client.16476 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:51.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.426+0000 7f8088c1a740 1 -- 172.21.15.28:0/4216696138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556b4ce41ea0 con 0x556b4cdc0ea0 2024-04-25T01:35:51.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.427+0000 7f805effd640 1 -- 172.21.15.28:0/4216696138 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8074005680 con 0x556b4cdc0ea0 2024-04-25T01:35:51.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.427+0000 7f805effd640 1 -- 172.21.15.28:0/4216696138 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f807400e500 con 0x556b4cdc0ea0 2024-04-25T01:35:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.427+0000 7f8088c1a740 10 client.16476 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.427+0000 7f8088c1a740 10 client.16476 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:35:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.427+0000 7f8088c1a740 20 client.16476 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.427+0000 7f8088c1a740 1 --2- 172.21.15.28:0/4216696138 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x556b4ce423a0 0x556b4ce447c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.427+0000 7f8088c1a740 4 client.16476 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:35:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.427+0000 7f8088c1a740 1 -- 172.21.15.28:0/4216696138 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556b4cdc8a60 con 0x556b4ce423a0 2024-04-25T01:35:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.427+0000 7f807f7fe640 1 --2- 172.21.15.28:0/4216696138 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x556b4ce423a0 0x556b4ce447c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.427+0000 7f805effd640 10 client.16476 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:35:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.427+0000 7f807f7fe640 1 --2- 172.21.15.28:0/4216696138 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x556b4ce423a0 0x556b4ce447c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5ba03f35fcac4c31 server_cookie=e454aee7d5fec1c3 in_seq=0 out_seq=0 2024-04-25T01:35:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.428+0000 7f805effd640 1 -- 172.21.15.28:0/4216696138 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556b4cdc8a60 con 0x556b4ce423a0 2024-04-25T01:35:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.428+0000 7f805effd640 10 client.16476 handle_command_reply: tid=0 2024-04-25T01:35:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.428+0000 7f8088c1a740 1 client.16476 shutdown 2024-04-25T01:35:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.428+0000 7f8088c1a740 1 -- 172.21.15.28:0/4216696138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0ea0 msgr2=0x556b4cdcf820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.428+0000 7f8088c1a740 1 --2- 172.21.15.28:0/4216696138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0ea0 0x556b4cdcf820 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f8074005e10 tx=0x7f8074004f00 comp rx=0 tx=0).stop 2024-04-25T01:35:51.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.428+0000 7f8088c1a740 1 -- 172.21.15.28:0/4216696138 shutdown_connections 2024-04-25T01:35:51.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.428+0000 7f8088c1a740 1 --2- 172.21.15.28:0/4216696138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556b4cdcb570 0x556b4cdcfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.428+0000 7f8088c1a740 1 --2- 172.21.15.28:0/4216696138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4cdc0ea0 0x556b4cdcf820 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.428+0000 7f8088c1a740 1 --2- 172.21.15.28:0/4216696138 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x556b4ce423a0 0x556b4ce447c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.428+0000 7f8088c1a740 1 --2- 172.21.15.28:0/4216696138 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556b4cdc0580 0x556b4cdcf2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.428+0000 7f805effd640 0 client.16476 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:35:51.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.428+0000 7f8088c1a740 1 -- 172.21.15.28:0/4216696138 >> 172.21.15.28:0/4216696138 conn(0x556b4cdbc8a0 msgr2=0x556b4cdc1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.428+0000 7f8088c1a740 1 -- 172.21.15.28:0/4216696138 shutdown_connections 2024-04-25T01:35:51.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.428+0000 7f8088c1a740 1 -- 172.21.15.28:0/4216696138 wait complete. 2024-04-25T01:35:51.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.428+0000 7f8088c1a740 20 client.16476 trim_cache size 0 max 16384 2024-04-25T01:35:51.412 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:35:51.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.428+0000 7f8085e31640 1 -- 172.21.15.28:0/810023936 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f80680a2d30 msgr2=0x7f80680a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.429+0000 7f8085e31640 1 --2- 172.21.15.28:0/810023936 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f80680a2d30 0x7f80680a51f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8070000f80 tx=0x7f807003a040 comp rx=0 tx=0).stop 2024-04-25T01:35:51.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.429+0000 7f8085e31640 1 -- 172.21.15.28:0/810023936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f808010bd00 msgr2=0x7f8080101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.429+0000 7f8085e31640 1 --2- 172.21.15.28:0/810023936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f808010bd00 0x7f8080101090 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f807800d750 tx=0x7f807800dc10 comp rx=0 tx=0).stop 2024-04-25T01:35:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.429+0000 7f8085e31640 1 -- 172.21.15.28:0/810023936 shutdown_connections 2024-04-25T01:35:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.429+0000 7f8085e31640 1 --2- 172.21.15.28:0/810023936 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f80680a2d30 0x7f80680a51f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.429+0000 7f8085e31640 1 --2- 172.21.15.28:0/810023936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f808010bd00 0x7f8080101090 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.429+0000 7f8085e31640 1 --2- 172.21.15.28:0/810023936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f80801032e0 0x7f8080100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.429+0000 7f8085e31640 1 --2- 172.21.15.28:0/810023936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f80801029c0 0x7f8080100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.429+0000 7f8085e31640 1 -- 172.21.15.28:0/810023936 >> 172.21.15.28:0/810023936 conn(0x7f8080077740 msgr2=0x7f80800fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.429+0000 7f8085e31640 1 -- 172.21.15.28:0/810023936 shutdown_connections 2024-04-25T01:35:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.429+0000 7f8085e31640 1 -- 172.21.15.28:0/810023936 wait complete. 2024-04-25T01:35:51.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:35:51.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:51 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2627805839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:35:51.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:51 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2627805839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:35:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.544+0000 7f4f39a10640 1 -- 172.21.15.28:0/475170500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f34100fe0 msgr2=0x7f4f3410d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.544+0000 7f4f39a10640 1 --2- 172.21.15.28:0/475170500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f34100fe0 0x7f4f3410d4f0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c009cd0 tx=0x7f4f2c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.544+0000 7f4f39a10640 1 -- 172.21.15.28:0/475170500 shutdown_connections 2024-04-25T01:35:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.544+0000 7f4f39a10640 1 --2- 172.21.15.28:0/475170500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f34100fe0 0x7f4f3410d4f0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.544+0000 7f4f39a10640 1 --2- 172.21.15.28:0/475170500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f34100620 0x7f4f34100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.544+0000 7f4f39a10640 1 --2- 172.21.15.28:0/475170500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f34106630 0x7f4f34106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.544+0000 7f4f39a10640 1 -- 172.21.15.28:0/475170500 >> 172.21.15.28:0/475170500 conn(0x7f4f340776a0 msgr2=0x7f4f34077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.544+0000 7f4f39a10640 1 -- 172.21.15.28:0/475170500 shutdown_connections 2024-04-25T01:35:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.544+0000 7f4f39a10640 1 -- 172.21.15.28:0/475170500 wait complete. 2024-04-25T01:35:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.544+0000 7f4f39a10640 1 Processor -- start 2024-04-25T01:35:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f39a10640 1 -- start start 2024-04-25T01:35:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f39a10640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f34100620 0x7f4f3419a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f39a10640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f34100fe0 0x7f4f3419a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f39a10640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f34106630 0x7f4f3419e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f39a10640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f34110a40 con 0x7f4f34100620 2024-04-25T01:35:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f39a10640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4f341108c0 con 0x7f4f34100fe0 2024-04-25T01:35:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f39a10640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4f34110bc0 con 0x7f4f34106630 2024-04-25T01:35:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f32ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f34100620 0x7f4f3419a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f32ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f34100620 0x7f4f3419a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47098/0 (socket says 172.21.15.28:47098) 2024-04-25T01:35:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f32ffd640 1 -- 172.21.15.28:0/2612291674 learned_addr learned my addr 172.21.15.28:0/2612291674 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f32ffd640 1 -- 172.21.15.28:0/2612291674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f34106630 msgr2=0x7f4f3419e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f32ffd640 1 --2- 172.21.15.28:0/2612291674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f34106630 0x7f4f3419e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f32ffd640 1 -- 172.21.15.28:0/2612291674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f34100fe0 msgr2=0x7f4f3419a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f32ffd640 1 --2- 172.21.15.28:0/2612291674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f34100fe0 0x7f4f3419a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f32ffd640 1 -- 172.21.15.28:0/2612291674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3419f0b0 con 0x7f4f34100620 2024-04-25T01:35:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f32ffd640 1 --2- 172.21.15.28:0/2612291674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f34100620 0x7f4f3419a0c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f4f2400d950 tx=0x7f4f2400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f38a0e640 1 -- 172.21.15.28:0/2612291674 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24017030 con 0x7f4f34100620 2024-04-25T01:35:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f39a10640 1 -- 172.21.15.28:0/2612291674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f3419f3a0 con 0x7f4f34100620 2024-04-25T01:35:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f38a0e640 1 -- 172.21.15.28:0/2612291674 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f24004020 con 0x7f4f34100620 2024-04-25T01:35:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f38a0e640 1 -- 172.21.15.28:0/2612291674 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f240047a0 con 0x7f4f34100620 2024-04-25T01:35:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.545+0000 7f4f39a10640 1 -- 172.21.15.28:0/2612291674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f341a6c30 con 0x7f4f34100620 2024-04-25T01:35:51.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.548+0000 7f4f38a0e640 1 -- 172.21.15.28:0/2612291674 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f4f240041d0 con 0x7f4f34100620 2024-04-25T01:35:51.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.548+0000 7f4f3c7f4740 1 Processor -- start 2024-04-25T01:35:51.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.548+0000 7f4f3c7f4740 1 -- start start 2024-04-25T01:35:51.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.548+0000 7f4f3c7f4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1580 0x55ba98dd1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.548+0000 7f4f3c7f4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98dd1ea0 0x55ba98ddc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.548+0000 7f4f3c7f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98ddc570 0x55ba98dde960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.548+0000 7f4f3c7f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ba98d36670 con 0x55ba98ddc570 2024-04-25T01:35:51.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.548+0000 7f4f3c7f4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ba98cde970 con 0x55ba98dd1580 2024-04-25T01:35:51.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.548+0000 7f4f3c7f4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ba98d32630 con 0x55ba98dd1ea0 2024-04-25T01:35:51.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.548+0000 7f4f38a0e640 1 --2- 172.21.15.28:0/2612291674 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4f1c0a2d00 0x7f4f1c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.549+0000 7f4f327fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98ddc570 0x55ba98dde960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.549+0000 7f4f327fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98ddc570 0x55ba98dde960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47102/0 (socket says 172.21.15.28:47102) 2024-04-25T01:35:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.549+0000 7f4f327fc640 1 -- 172.21.15.28:0/1438841243 learned_addr learned my addr 172.21.15.28:0/1438841243 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.549+0000 7f4f32ffd640 1 --2- 172.21.15.28:0/1438841243 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98dd1ea0 0x55ba98ddc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.549+0000 7f4f337fe640 1 --2- 172.21.15.28:0/1438841243 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1580 0x55ba98dd1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.549+0000 7f4f337fe640 1 --2- 172.21.15.28:0/2612291674 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4f1c0a2d00 0x7f4f1c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.549+0000 7f4f38a0e640 1 -- 172.21.15.28:0/2612291674 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f4f240cf200 con 0x7f4f34100620 2024-04-25T01:35:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.549+0000 7f4f327fc640 1 -- 172.21.15.28:0/1438841243 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98dd1ea0 msgr2=0x55ba98ddc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.549+0000 7f4f327fc640 1 --2- 172.21.15.28:0/1438841243 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98dd1ea0 0x55ba98ddc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.549+0000 7f4f327fc640 1 -- 172.21.15.28:0/1438841243 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1580 msgr2=0x55ba98dd1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.549+0000 7f4f327fc640 1 --2- 172.21.15.28:0/1438841243 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1580 0x55ba98dd1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.549+0000 7f4f327fc640 1 -- 172.21.15.28:0/1438841243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba98d02400 con 0x55ba98ddc570 2024-04-25T01:35:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.549+0000 7f4f327fc640 1 --2- 172.21.15.28:0/1438841243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98ddc570 0x55ba98dde960 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f4f2800aa40 tx=0x7f4f2800af00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=707957e39b0e8de0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.549+0000 7f4f337fe640 1 --2- 172.21.15.28:0/2612291674 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4f1c0a2d00 0x7f4f1c0a51c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c009ca0 tx=0x7f4f2c0095c0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.549+0000 7f4f19ffb640 1 -- 172.21.15.28:0/1438841243 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2800ce40 con 0x55ba98ddc570 2024-04-25T01:35:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.549+0000 7f4f19ffb640 1 -- 172.21.15.28:0/1438841243 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f280035f0 con 0x55ba98ddc570 2024-04-25T01:35:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.549+0000 7f4f19ffb640 1 -- 172.21.15.28:0/1438841243 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2800f690 con 0x55ba98ddc570 2024-04-25T01:35:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.550+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/1438841243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98ddc570 msgr2=0x55ba98dde960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.550+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/1438841243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98ddc570 0x55ba98dde960 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f4f2800aa40 tx=0x7f4f2800af00 comp rx=0 tx=0).stop 2024-04-25T01:35:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.550+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/1438841243 shutdown_connections 2024-04-25T01:35:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.550+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/1438841243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98ddc570 0x55ba98dde960 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.550+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/1438841243 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98dd1ea0 0x55ba98ddc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.550+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/1438841243 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1580 0x55ba98dd1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.550+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/1438841243 >> 172.21.15.28:0/1438841243 conn(0x55ba98dcd8a0 msgr2=0x55ba98dcdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.550+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/1438841243 shutdown_connections 2024-04-25T01:35:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.550+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/1438841243 wait complete. 2024-04-25T01:35:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.550+0000 7f4f3c7f4740 1 -- start start 2024-04-25T01:35:51.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.551+0000 7f4f3c7f4740 1 Processor -- start 2024-04-25T01:35:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.551+0000 7f4f3c7f4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1580 0x55ba98ddbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.551+0000 7f4f3c7f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98dd1ea0 0x55ba98dd9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.551+0000 7f4f3c7f4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98ddc570 0x55ba98dd9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.551+0000 7f4f3c7f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ba98ddf360 con 0x55ba98dd1ea0 2024-04-25T01:35:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.551+0000 7f4f3c7f4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ba98cf2050 con 0x55ba98dd1580 2024-04-25T01:35:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.551+0000 7f4f3c7f4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ba98ddf4e0 con 0x55ba98ddc570 2024-04-25T01:35:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.551+0000 7f4f3c7f4740 10 client.0 _ll_register_callbacks cb 0x55ba98a49be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.551+0000 7f4f327fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98dd1ea0 0x55ba98dd9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.551+0000 7f4f327fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98dd1ea0 0x55ba98dd9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47106/0 (socket says 172.21.15.28:47106) 2024-04-25T01:35:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.551+0000 7f4f327fc640 1 -- 172.21.15.28:0/3206851190 learned_addr learned my addr 172.21.15.28:0/3206851190 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.551+0000 7f4f327fc640 1 -- 172.21.15.28:0/3206851190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98ddc570 msgr2=0x55ba98dd9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.551+0000 7f4f327fc640 1 --2- 172.21.15.28:0/3206851190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98ddc570 0x55ba98dd9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.551+0000 7f4f327fc640 1 -- 172.21.15.28:0/3206851190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1580 msgr2=0x55ba98ddbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.551+0000 7f4f32ffd640 1 --2- 172.21.15.28:0/3206851190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1580 0x55ba98ddbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.551+0000 7f4f327fc640 1 --2- 172.21.15.28:0/3206851190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1580 0x55ba98ddbe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.551+0000 7f4f327fc640 1 -- 172.21.15.28:0/3206851190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba98cc3dc0 con 0x55ba98dd1ea0 2024-04-25T01:35:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.551+0000 7f4f327fc640 1 --2- 172.21.15.28:0/3206851190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98dd1ea0 0x55ba98dd9150 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4f28004250 tx=0x7f4f28003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab18d89d2c3d7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.551+0000 7f4f12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.552+0000 7f4f12ffd640 1 -- 172.21.15.28:0/3206851190 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f280043b0 con 0x55ba98dd1ea0 2024-04-25T01:35:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.552+0000 7f4f12ffd640 1 -- 172.21.15.28:0/3206851190 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f4f280039b0 con 0x55ba98dd1ea0 2024-04-25T01:35:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.552+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/3206851190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f280035f0 con 0x55ba98dd1ea0 2024-04-25T01:35:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.552+0000 7f4f12ffd640 1 -- 172.21.15.28:0/3206851190 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4f2802e050 con 0x55ba98dd1ea0 2024-04-25T01:35:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.552+0000 7f4f3c7f4740 10 client.16488 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.552+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/3206851190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ba98a4a3f0 con 0x55ba98dd1ea0 2024-04-25T01:35:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.552+0000 7f4f12ffd640 1 -- 172.21.15.28:0/3206851190 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f280035f0 con 0x55ba98dd1ea0 2024-04-25T01:35:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.552+0000 7f4f12ffd640 1 -- 172.21.15.28:0/3206851190 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4f28004930 con 0x55ba98dd1ea0 2024-04-25T01:35:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.552+0000 7f4f3c7f4740 10 client.16488 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.552+0000 7f4f3c7f4740 10 client.16488 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:35:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.552+0000 7f4f3c7f4740 20 client.16488 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.552+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/3206851190 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55ba98dec450 0x55ba98dee840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.552+0000 7f4f3c7f4740 4 client.16488 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.552+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/3206851190 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba98d43bb0 con 0x55ba98dec450 2024-04-25T01:35:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.553+0000 7f4f32ffd640 1 --2- 172.21.15.28:0/3206851190 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55ba98dec450 0x55ba98dee840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.553+0000 7f4f12ffd640 10 client.16488 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:35:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.553+0000 7f4f32ffd640 1 --2- 172.21.15.28:0/3206851190 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55ba98dec450 0x55ba98dee840 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a47edbcaf7271f9a server_cookie=c6df8caa619d039f in_seq=0 out_seq=0 2024-04-25T01:35:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.554+0000 7f4f12ffd640 1 -- 172.21.15.28:0/3206851190 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ba98d43bb0 con 0x55ba98dec450 2024-04-25T01:35:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.554+0000 7f4f12ffd640 10 client.16488 handle_command_reply: tid=0 2024-04-25T01:35:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.554+0000 7f4f3c7f4740 1 client.16488 shutdown 2024-04-25T01:35:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.554+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/3206851190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98dd1ea0 msgr2=0x55ba98dd9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.554+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/3206851190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98dd1ea0 0x55ba98dd9150 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f4f28004250 tx=0x7f4f28003d60 comp rx=0 tx=0).stop 2024-04-25T01:35:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.554+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/3206851190 shutdown_connections 2024-04-25T01:35:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.554+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/3206851190 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55ba98dec450 0x55ba98dee840 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.554+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/3206851190 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98ddc570 0x55ba98dd9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.554+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/3206851190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98dd1ea0 0x55ba98dd9150 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.554+0000 7f4f12ffd640 0 client.16488 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:35:51.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.554+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/3206851190 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1580 0x55ba98ddbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.554+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/3206851190 >> 172.21.15.28:0/3206851190 conn(0x55ba98dcd8a0 msgr2=0x55ba98dccb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.554+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/3206851190 shutdown_connections 2024-04-25T01:35:51.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.554+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/3206851190 wait complete. 2024-04-25T01:35:51.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.554+0000 7f4f3c7f4740 20 client.16488 trim_cache size 0 max 16384 2024-04-25T01:35:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.567+0000 7f4f3c7f4740 1 Processor -- start 2024-04-25T01:35:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.567+0000 7f4f3c7f4740 1 -- start start 2024-04-25T01:35:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.567+0000 7f4f3c7f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98dd1580 0x55ba98de8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.567+0000 7f4f3c7f4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1ea0 0x55ba98df3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.567+0000 7f4f3c7f4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98ddc570 0x55ba98df4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.567+0000 7f4f3c7f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f341b0c60 con 0x55ba98dd1580 2024-04-25T01:35:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.567+0000 7f4f3c7f4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4f341b27a0 con 0x55ba98dd1ea0 2024-04-25T01:35:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.567+0000 7f4f3c7f4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4f341b1330 con 0x55ba98ddc570 2024-04-25T01:35:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.568+0000 7f4f32ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98dd1580 0x55ba98de8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.568+0000 7f4f32ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98dd1580 0x55ba98de8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47108/0 (socket says 172.21.15.28:47108) 2024-04-25T01:35:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.568+0000 7f4f32ffd640 1 -- 172.21.15.28:0/826050537 learned_addr learned my addr 172.21.15.28:0/826050537 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.568+0000 7f4f327fc640 1 --2- 172.21.15.28:0/826050537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1ea0 0x55ba98df3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.568+0000 7f4f337fe640 1 --2- 172.21.15.28:0/826050537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98ddc570 0x55ba98df4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.568+0000 7f4f32ffd640 1 -- 172.21.15.28:0/826050537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98ddc570 msgr2=0x55ba98df4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.568+0000 7f4f32ffd640 1 --2- 172.21.15.28:0/826050537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98ddc570 0x55ba98df4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.568+0000 7f4f32ffd640 1 -- 172.21.15.28:0/826050537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1ea0 msgr2=0x55ba98df3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.568+0000 7f4f32ffd640 1 --2- 172.21.15.28:0/826050537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1ea0 0x55ba98df3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.568+0000 7f4f32ffd640 1 -- 172.21.15.28:0/826050537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba98dd1cd0 con 0x55ba98dd1580 2024-04-25T01:35:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.568+0000 7f4f337fe640 1 --2- 172.21.15.28:0/826050537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98ddc570 0x55ba98df4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.568+0000 7f4f32ffd640 1 --2- 172.21.15.28:0/826050537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98dd1580 0x55ba98de8220 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f4f24005fd0 tx=0x7f4f2406d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9288ee346da8361 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.568+0000 7f4f197fa640 1 -- 172.21.15.28:0/826050537 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2409ba80 con 0x55ba98dd1580 2024-04-25T01:35:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.568+0000 7f4f197fa640 1 -- 172.21.15.28:0/826050537 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f2409b380 con 0x55ba98dd1580 2024-04-25T01:35:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.568+0000 7f4f197fa640 1 -- 172.21.15.28:0/826050537 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f240cc830 con 0x55ba98dd1580 2024-04-25T01:35:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.569+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/826050537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98dd1580 msgr2=0x55ba98de8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.569+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/826050537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98dd1580 0x55ba98de8220 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f4f24005fd0 tx=0x7f4f2406d340 comp rx=0 tx=0).stop 2024-04-25T01:35:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.569+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/826050537 shutdown_connections 2024-04-25T01:35:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.569+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/826050537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98ddc570 0x55ba98df4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.569+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/826050537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1ea0 0x55ba98df3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.569+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/826050537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98dd1580 0x55ba98de8220 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.569+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/826050537 >> 172.21.15.28:0/826050537 conn(0x55ba98dcd8a0 msgr2=0x55ba98dcc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.569+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/826050537 shutdown_connections 2024-04-25T01:35:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.569+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/826050537 wait complete. 2024-04-25T01:35:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.569+0000 7f4f3c7f4740 1 -- start start 2024-04-25T01:35:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.569+0000 7f4f3c7f4740 1 Processor -- start 2024-04-25T01:35:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.569+0000 7f4f3c7f4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1580 0x55ba98de4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f3c7f4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98dd1ea0 0x55ba98de48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f3c7f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98ddc570 0x55ba98de1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f3c7f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f341b20d0 con 0x55ba98ddc570 2024-04-25T01:35:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f3c7f4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4f341b1a00 con 0x55ba98dd1580 2024-04-25T01:35:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f3c7f4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4f341aec40 con 0x55ba98dd1ea0 2024-04-25T01:35:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f3c7f4740 10 client.0 _ll_register_callbacks cb 0x7f4f2c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98ddc570 0x55ba98de1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f337fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98ddc570 0x55ba98de1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47118/0 (socket says 172.21.15.28:47118) 2024-04-25T01:35:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f337fe640 1 -- 172.21.15.28:0/1247238545 learned_addr learned my addr 172.21.15.28:0/1247238545 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f32ffd640 1 --2- 172.21.15.28:0/1247238545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1580 0x55ba98de4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f327fc640 1 --2- 172.21.15.28:0/1247238545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98dd1ea0 0x55ba98de48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f337fe640 1 -- 172.21.15.28:0/1247238545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98dd1ea0 msgr2=0x55ba98de48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f337fe640 1 --2- 172.21.15.28:0/1247238545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98dd1ea0 0x55ba98de48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f337fe640 1 -- 172.21.15.28:0/1247238545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1580 msgr2=0x55ba98de4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f337fe640 1 --2- 172.21.15.28:0/1247238545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1580 0x55ba98de4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f337fe640 1 -- 172.21.15.28:0/1247238545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba98b147d0 con 0x55ba98ddc570 2024-04-25T01:35:51.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f337fe640 1 --2- 172.21.15.28:0/1247238545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98ddc570 0x55ba98de1b10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c009cd0 tx=0x7f4f2c003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa31af33bf2e497 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:51.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f12ffd640 1 -- 172.21.15.28:0/1247238545 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c0047a0 con 0x55ba98ddc570 2024-04-25T01:35:51.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f12ffd640 1 -- 172.21.15.28:0/1247238545 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f4f2c0305a0 con 0x55ba98ddc570 2024-04-25T01:35:51.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.570+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/1247238545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f2409b380 con 0x55ba98ddc570 2024-04-25T01:35:51.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.571+0000 7f4f12ffd640 1 -- 172.21.15.28:0/1247238545 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4f2c0307a0 con 0x55ba98ddc570 2024-04-25T01:35:51.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.571+0000 7f4f3c7f4740 10 client.16494 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:51.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.571+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/1247238545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ba98e52090 con 0x55ba98ddc570 2024-04-25T01:35:51.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.571+0000 7f4f12ffd640 1 -- 172.21.15.28:0/1247238545 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f2409b380 con 0x55ba98ddc570 2024-04-25T01:35:51.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.571+0000 7f4f12ffd640 1 -- 172.21.15.28:0/1247238545 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4f2c030070 con 0x55ba98ddc570 2024-04-25T01:35:51.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.571+0000 7f4f3c7f4740 10 client.16494 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:51.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.571+0000 7f4f3c7f4740 10 client.16494 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:35:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.571+0000 7f4f3c7f4740 20 client.16494 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.571+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/1247238545 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55ba98e52890 0x55ba98e54cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.571+0000 7f4f3c7f4740 4 client.16494 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:35:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.571+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/1247238545 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ba98dd9a20 con 0x55ba98e52890 2024-04-25T01:35:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.571+0000 7f4f32ffd640 1 --2- 172.21.15.28:0/1247238545 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55ba98e52890 0x55ba98e54cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.571+0000 7f4f32ffd640 1 --2- 172.21.15.28:0/1247238545 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55ba98e52890 0x55ba98e54cb0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=694d90aaa174c50b server_cookie=d61de0372e646e4 in_seq=0 out_seq=0 2024-04-25T01:35:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.571+0000 7f4f12ffd640 10 client.16494 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:35:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.572+0000 7f4f12ffd640 1 -- 172.21.15.28:0/1247238545 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ba98dd9a20 con 0x55ba98e52890 2024-04-25T01:35:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.572+0000 7f4f12ffd640 10 client.16494 handle_command_reply: tid=0 2024-04-25T01:35:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.572+0000 7f4f3c7f4740 1 client.16494 shutdown 2024-04-25T01:35:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.572+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/1247238545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98ddc570 msgr2=0x55ba98de1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.572+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/1247238545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98ddc570 0x55ba98de1b10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c009cd0 tx=0x7f4f2c003d40 comp rx=0 tx=0).stop 2024-04-25T01:35:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.572+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/1247238545 shutdown_connections 2024-04-25T01:35:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.572+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/1247238545 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55ba98e52890 0x55ba98e54cb0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.572+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/1247238545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba98ddc570 0x55ba98de1b10 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.572+0000 7f4f12ffd640 0 client.16494 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:35:51.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.572+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/1247238545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ba98dd1ea0 0x55ba98de48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.572+0000 7f4f3c7f4740 1 --2- 172.21.15.28:0/1247238545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ba98dd1580 0x55ba98de4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.572+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/1247238545 >> 172.21.15.28:0/1247238545 conn(0x55ba98dcd8a0 msgr2=0x55ba98dccb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.572+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/1247238545 shutdown_connections 2024-04-25T01:35:51.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.572+0000 7f4f3c7f4740 1 -- 172.21.15.28:0/1247238545 wait complete. 2024-04-25T01:35:51.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.572+0000 7f4f3c7f4740 20 client.16494 trim_cache size 0 max 16384 2024-04-25T01:35:51.555 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:35:51.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.573+0000 7f4f39a10640 1 -- 172.21.15.28:0/2612291674 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4f1c0a2d00 msgr2=0x7f4f1c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.573+0000 7f4f39a10640 1 --2- 172.21.15.28:0/2612291674 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4f1c0a2d00 0x7f4f1c0a51c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c009ca0 tx=0x7f4f2c0095c0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.573+0000 7f4f39a10640 1 -- 172.21.15.28:0/2612291674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f34100620 msgr2=0x7f4f3419a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.573+0000 7f4f39a10640 1 --2- 172.21.15.28:0/2612291674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f34100620 0x7f4f3419a0c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f4f2400d950 tx=0x7f4f2400de10 comp rx=0 tx=0).stop 2024-04-25T01:35:51.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.573+0000 7f4f39a10640 1 -- 172.21.15.28:0/2612291674 shutdown_connections 2024-04-25T01:35:51.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.573+0000 7f4f39a10640 1 --2- 172.21.15.28:0/2612291674 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4f1c0a2d00 0x7f4f1c0a51c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.573+0000 7f4f39a10640 1 --2- 172.21.15.28:0/2612291674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f34106630 0x7f4f3419e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.573+0000 7f4f39a10640 1 --2- 172.21.15.28:0/2612291674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f34100fe0 0x7f4f3419a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.573+0000 7f4f39a10640 1 --2- 172.21.15.28:0/2612291674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f34100620 0x7f4f3419a0c0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.573+0000 7f4f39a10640 1 -- 172.21.15.28:0/2612291674 >> 172.21.15.28:0/2612291674 conn(0x7f4f340776a0 msgr2=0x7f4f34069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.573+0000 7f4f39a10640 1 -- 172.21.15.28:0/2612291674 shutdown_connections 2024-04-25T01:35:51.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.573+0000 7f4f39a10640 1 -- 172.21.15.28:0/2612291674 wait complete. 2024-04-25T01:35:51.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:35:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.688+0000 7fa66bae1640 1 -- 172.21.15.28:0/2451778853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa664106670 msgr2=0x7fa664069dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.688+0000 7fa66bae1640 1 --2- 172.21.15.28:0/2451778853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa664106670 0x7fa664069dc0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fa660009900 tx=0x7fa66002a410 comp rx=0 tx=0).stop 2024-04-25T01:35:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.688+0000 7fa66bae1640 1 -- 172.21.15.28:0/2451778853 shutdown_connections 2024-04-25T01:35:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.688+0000 7fa66bae1640 1 --2- 172.21.15.28:0/2451778853 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa664104170 0x7fa6641045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.688+0000 7fa66bae1640 1 --2- 172.21.15.28:0/2451778853 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa66406a300 0x7fa66406a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.688+0000 7fa66bae1640 1 --2- 172.21.15.28:0/2451778853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa664106670 0x7fa664069dc0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.688+0000 7fa66bae1640 1 -- 172.21.15.28:0/2451778853 >> 172.21.15.28:0/2451778853 conn(0x7fa6640776a0 msgr2=0x7fa664077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.688+0000 7fa66bae1640 1 -- 172.21.15.28:0/2451778853 shutdown_connections 2024-04-25T01:35:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.688+0000 7fa66bae1640 1 -- 172.21.15.28:0/2451778853 wait complete. 2024-04-25T01:35:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.689+0000 7fa66bae1640 1 Processor -- start 2024-04-25T01:35:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.689+0000 7fa66bae1640 1 -- start start 2024-04-25T01:35:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.689+0000 7fa66bae1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa66406a300 0x7fa66419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.689+0000 7fa66bae1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa664104170 0x7fa66419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.689+0000 7fa66bae1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa664106670 0x7fa6641a2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.689+0000 7fa66bae1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641180f0 con 0x7fa66406a300 2024-04-25T01:35:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.689+0000 7fa66bae1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa664117f70 con 0x7fa664106670 2024-04-25T01:35:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.689+0000 7fa66bae1640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa664118270 con 0x7fa664104170 2024-04-25T01:35:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.689+0000 7fa66987d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa66406a300 0x7fa66419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.689+0000 7fa66987d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa66406a300 0x7fa66419e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47144/0 (socket says 172.21.15.28:47144) 2024-04-25T01:35:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.689+0000 7fa66987d640 1 -- 172.21.15.28:0/4043275248 learned_addr learned my addr 172.21.15.28:0/4043275248 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.689+0000 7fa66907c640 1 --2- 172.21.15.28:0/4043275248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa664104170 0x7fa66419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.689+0000 7fa66a07e640 1 --2- 172.21.15.28:0/4043275248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa664106670 0x7fa6641a2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.689+0000 7fa66987d640 1 -- 172.21.15.28:0/4043275248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa664104170 msgr2=0x7fa66419eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.689+0000 7fa66987d640 1 --2- 172.21.15.28:0/4043275248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa664104170 0x7fa66419eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.689+0000 7fa66987d640 1 -- 172.21.15.28:0/4043275248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa664106670 msgr2=0x7fa6641a2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.689+0000 7fa66987d640 1 --2- 172.21.15.28:0/4043275248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa664106670 0x7fa6641a2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.689+0000 7fa66987d640 1 -- 172.21.15.28:0/4043275248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6641a35e0 con 0x7fa66406a300 2024-04-25T01:35:51.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.689+0000 7fa66907c640 1 --2- 172.21.15.28:0/4043275248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa664104170 0x7fa66419eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:51.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.690+0000 7fa66987d640 1 --2- 172.21.15.28:0/4043275248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa66406a300 0x7fa66419e5f0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fa660006010 tx=0x7fa6600042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.690+0000 7fa65affd640 1 -- 172.21.15.28:0/4043275248 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66003f030 con 0x7fa66406a300 2024-04-25T01:35:51.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.690+0000 7fa66bae1640 1 -- 172.21.15.28:0/4043275248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6641a3870 con 0x7fa66406a300 2024-04-25T01:35:51.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.690+0000 7fa65affd640 1 -- 172.21.15.28:0/4043275248 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa660003ce0 con 0x7fa66406a300 2024-04-25T01:35:51.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.690+0000 7fa65affd640 1 -- 172.21.15.28:0/4043275248 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660035070 con 0x7fa66406a300 2024-04-25T01:35:51.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.690+0000 7fa66bae1640 1 -- 172.21.15.28:0/4043275248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6641ab160 con 0x7fa66406a300 2024-04-25T01:35:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.692+0000 7fa65affd640 1 -- 172.21.15.28:0/4043275248 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa660042030 con 0x7fa66406a300 2024-04-25T01:35:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.692+0000 7fa66e7f5740 1 Processor -- start 2024-04-25T01:35:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa66e7f5740 1 -- start start 2024-04-25T01:35:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa66e7f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76e2580 0x5612c76e2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa66e7f5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76e2ea0 0x5612c76ed030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa66e7f5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5612c76ed570 0x5612c76ef960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa66e7f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5612c7647670 con 0x5612c76e2580 2024-04-25T01:35:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa66e7f5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5612c75ef970 con 0x5612c76e2ea0 2024-04-25T01:35:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa66e7f5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5612c7643630 con 0x5612c76ed570 2024-04-25T01:35:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa65affd640 1 --2- 172.21.15.28:0/4043275248 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa6500a2d00 0x7fa6500a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa66a07e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76e2580 0x5612c76e2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa66a07e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76e2580 0x5612c76e2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47158/0 (socket says 172.21.15.28:47158) 2024-04-25T01:35:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa66a07e640 1 -- 172.21.15.28:0/1136447966 learned_addr learned my addr 172.21.15.28:0/1136447966 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa66a07e640 1 --2- 172.21.15.28:0/4043275248 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa6500a2d00 0x7fa6500a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa65affd640 1 -- 172.21.15.28:0/4043275248 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fa6600efd50 con 0x7fa66406a300 2024-04-25T01:35:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa66987d640 1 --2- 172.21.15.28:0/1136447966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76e2ea0 0x5612c76ed030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa66907c640 1 --2- 172.21.15.28:0/1136447966 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5612c76ed570 0x5612c76ef960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa66a07e640 1 -- 172.21.15.28:0/1136447966 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5612c76ed570 msgr2=0x5612c76ef960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa66a07e640 1 --2- 172.21.15.28:0/1136447966 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5612c76ed570 0x5612c76ef960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa66a07e640 1 -- 172.21.15.28:0/1136447966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76e2ea0 msgr2=0x5612c76ed030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa66a07e640 1 --2- 172.21.15.28:0/1136447966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76e2ea0 0x5612c76ed030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa66a07e640 1 -- 172.21.15.28:0/1136447966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612c7613400 con 0x5612c76e2580 2024-04-25T01:35:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.693+0000 7fa66987d640 1 --2- 172.21.15.28:0/1136447966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76e2ea0 0x5612c76ed030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:35:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.694+0000 7fa66a07e640 1 --2- 172.21.15.28:0/1136447966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76e2580 0x5612c76e2960 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fa65c012c50 tx=0x7fa65c010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44147665a6817645 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.694+0000 7fa66a07e640 1 --2- 172.21.15.28:0/4043275248 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa6500a2d00 0x7fa6500a51c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa65c010c40 tx=0x7fa65c017000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.694+0000 7fa64ffff640 1 -- 172.21.15.28:0/1136447966 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c023030 con 0x5612c76e2580 2024-04-25T01:35:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.694+0000 7fa64ffff640 1 -- 172.21.15.28:0/1136447966 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa65c012e00 con 0x5612c76e2580 2024-04-25T01:35:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.694+0000 7fa64ffff640 1 -- 172.21.15.28:0/1136447966 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c005020 con 0x5612c76e2580 2024-04-25T01:35:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.695+0000 7fa66e7f5740 1 -- 172.21.15.28:0/1136447966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76e2580 msgr2=0x5612c76e2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.695+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/1136447966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76e2580 0x5612c76e2960 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fa65c012c50 tx=0x7fa65c010740 comp rx=0 tx=0).stop 2024-04-25T01:35:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.695+0000 7fa66e7f5740 1 -- 172.21.15.28:0/1136447966 shutdown_connections 2024-04-25T01:35:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.695+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/1136447966 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5612c76ed570 0x5612c76ef960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.695+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/1136447966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76e2ea0 0x5612c76ed030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.695+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/1136447966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76e2580 0x5612c76e2960 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.695+0000 7fa66e7f5740 1 -- 172.21.15.28:0/1136447966 >> 172.21.15.28:0/1136447966 conn(0x5612c76de8a0 msgr2=0x5612c76dec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.695+0000 7fa66e7f5740 1 -- 172.21.15.28:0/1136447966 shutdown_connections 2024-04-25T01:35:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.695+0000 7fa66e7f5740 1 -- 172.21.15.28:0/1136447966 wait complete. 2024-04-25T01:35:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.695+0000 7fa66e7f5740 1 -- start start 2024-04-25T01:35:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.695+0000 7fa66e7f5740 1 Processor -- start 2024-04-25T01:35:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.695+0000 7fa66e7f5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76e2580 0x5612c76ecf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.695+0000 7fa66e7f5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5612c76e2ea0 0x5612c76ea250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.695+0000 7fa66e7f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76ed570 0x5612c76ea860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa66e7f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5612c76f0300 con 0x5612c76ed570 2024-04-25T01:35:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa66e7f5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5612c7603050 con 0x5612c76e2580 2024-04-25T01:35:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa66e7f5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5612c76f0480 con 0x5612c76e2ea0 2024-04-25T01:35:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa66e7f5740 10 client.0 _ll_register_callbacks cb 0x5612c735abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa66a07e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76ed570 0x5612c76ea860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa66a07e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76ed570 0x5612c76ea860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47166/0 (socket says 172.21.15.28:47166) 2024-04-25T01:35:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa66a07e640 1 -- 172.21.15.28:0/4286612572 learned_addr learned my addr 172.21.15.28:0/4286612572 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa66987d640 1 --2- 172.21.15.28:0/4286612572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76e2580 0x5612c76ecf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa66a07e640 1 -- 172.21.15.28:0/4286612572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5612c76e2ea0 msgr2=0x5612c76ea250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa66a07e640 1 --2- 172.21.15.28:0/4286612572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5612c76e2ea0 0x5612c76ea250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa66a07e640 1 -- 172.21.15.28:0/4286612572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76e2580 msgr2=0x5612c76ecf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa66a07e640 1 --2- 172.21.15.28:0/4286612572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76e2580 0x5612c76ecf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa66a07e640 1 -- 172.21.15.28:0/4286612572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612c75d4dc0 con 0x5612c76ed570 2024-04-25T01:35:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa66a07e640 1 --2- 172.21.15.28:0/4286612572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76ed570 0x5612c76ea860 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa65c010fd0 tx=0x7fa65c0127e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22442faa4c1a6b96 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa64d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa64d7fa640 1 -- 172.21.15.28:0/4286612572 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c02c070 con 0x5612c76ed570 2024-04-25T01:35:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa64d7fa640 1 -- 172.21.15.28:0/4286612572 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fa65c005240 con 0x5612c76ed570 2024-04-25T01:35:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa66e7f5740 1 -- 172.21.15.28:0/4286612572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa65c012e00 con 0x5612c76ed570 2024-04-25T01:35:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa64d7fa640 1 -- 172.21.15.28:0/4286612572 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa65c038050 con 0x5612c76ed570 2024-04-25T01:35:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa66e7f5740 10 client.16506 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.696+0000 7fa66e7f5740 1 -- 172.21.15.28:0/4286612572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5612c735b3f0 con 0x5612c76ed570 2024-04-25T01:35:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.697+0000 7fa64d7fa640 1 -- 172.21.15.28:0/4286612572 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa65c012e00 con 0x5612c76ed570 2024-04-25T01:35:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.697+0000 7fa64d7fa640 1 -- 172.21.15.28:0/4286612572 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa65c01f7e0 con 0x5612c76ed570 2024-04-25T01:35:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.697+0000 7fa66e7f5740 10 client.16506 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.697+0000 7fa66e7f5740 10 client.16506 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.697+0000 7fa66e7f5740 20 client.16506 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.697+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/4286612572 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5612c76fd4a0 0x5612c76ff890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.697+0000 7fa66e7f5740 4 client.16506 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.697+0000 7fa66e7f5740 1 -- 172.21.15.28:0/4286612572 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612c7654bb0 con 0x5612c76fd4a0 2024-04-25T01:35:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.697+0000 7fa66987d640 1 --2- 172.21.15.28:0/4286612572 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5612c76fd4a0 0x5612c76ff890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.698+0000 7fa64d7fa640 10 client.16506 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.698+0000 7fa66987d640 1 --2- 172.21.15.28:0/4286612572 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5612c76fd4a0 0x5612c76ff890 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e11c2d2c7d468d68 server_cookie=89f6f8d3c2ba80b2 in_seq=0 out_seq=0 2024-04-25T01:35:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.698+0000 7fa64d7fa640 1 -- 172.21.15.28:0/4286612572 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5612c7654bb0 con 0x5612c76fd4a0 2024-04-25T01:35:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.698+0000 7fa64d7fa640 10 client.16506 handle_command_reply: tid=0 2024-04-25T01:35:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.698+0000 7fa66e7f5740 1 client.16506 shutdown 2024-04-25T01:35:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.698+0000 7fa66e7f5740 1 -- 172.21.15.28:0/4286612572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76ed570 msgr2=0x5612c76ea860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.698+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/4286612572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76ed570 0x5612c76ea860 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa65c010fd0 tx=0x7fa65c0127e0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.699+0000 7fa66e7f5740 1 -- 172.21.15.28:0/4286612572 shutdown_connections 2024-04-25T01:35:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.699+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/4286612572 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5612c76fd4a0 0x5612c76ff890 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.699+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/4286612572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76ed570 0x5612c76ea860 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.699+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/4286612572 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5612c76e2ea0 0x5612c76ea250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.699+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/4286612572 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76e2580 0x5612c76ecf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.699+0000 7fa64d7fa640 0 client.16506 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:51.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.699+0000 7fa66e7f5740 1 -- 172.21.15.28:0/4286612572 >> 172.21.15.28:0/4286612572 conn(0x5612c76de8a0 msgr2=0x5612c76ddb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.699+0000 7fa66e7f5740 1 -- 172.21.15.28:0/4286612572 shutdown_connections 2024-04-25T01:35:51.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.699+0000 7fa66e7f5740 1 -- 172.21.15.28:0/4286612572 wait complete. 2024-04-25T01:35:51.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.699+0000 7fa66e7f5740 20 client.16506 trim_cache size 0 max 16384 2024-04-25T01:35:51.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.712+0000 7fa66e7f5740 1 Processor -- start 2024-04-25T01:35:51.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.712+0000 7fa66e7f5740 1 -- start start 2024-04-25T01:35:51.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.713+0000 7fa66e7f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76e2580 0x5612c76f91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.713+0000 7fa66e7f5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5612c76e2ea0 0x5612c76eca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.713+0000 7fa66e7f5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76ed570 0x5612c7705000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.713+0000 7fa66e7f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641b50a0 con 0x5612c76e2580 2024-04-25T01:35:51.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.713+0000 7fa66e7f5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641b6be0 con 0x5612c76ed570 2024-04-25T01:35:51.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.713+0000 7fa66e7f5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641b5770 con 0x5612c76e2ea0 2024-04-25T01:35:51.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.713+0000 7fa66987d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76e2580 0x5612c76f91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.713+0000 7fa66987d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76e2580 0x5612c76f91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47174/0 (socket says 172.21.15.28:47174) 2024-04-25T01:35:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.713+0000 7fa66987d640 1 -- 172.21.15.28:0/3224779383 learned_addr learned my addr 172.21.15.28:0/3224779383 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.713+0000 7fa66a07e640 1 --2- 172.21.15.28:0/3224779383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76ed570 0x5612c7705000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.713+0000 7fa66907c640 1 --2- 172.21.15.28:0/3224779383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5612c76e2ea0 0x5612c76eca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.713+0000 7fa66987d640 1 -- 172.21.15.28:0/3224779383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5612c76e2ea0 msgr2=0x5612c76eca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.713+0000 7fa66987d640 1 --2- 172.21.15.28:0/3224779383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5612c76e2ea0 0x5612c76eca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.713+0000 7fa66987d640 1 -- 172.21.15.28:0/3224779383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76ed570 msgr2=0x5612c7705000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.713+0000 7fa66987d640 1 --2- 172.21.15.28:0/3224779383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76ed570 0x5612c7705000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.713+0000 7fa66987d640 1 -- 172.21.15.28:0/3224779383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612c74257d0 con 0x5612c76e2580 2024-04-25T01:35:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.713+0000 7fa66a07e640 1 --2- 172.21.15.28:0/3224779383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76ed570 0x5612c7705000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.713+0000 7fa66987d640 1 --2- 172.21.15.28:0/3224779383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76e2580 0x5612c76f91c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa6600b89c0 tx=0x7fa660009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f61b047ef98124 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.713+0000 7fa64f7fe640 1 -- 172.21.15.28:0/3224779383 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660049040 con 0x5612c76e2580 2024-04-25T01:35:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.714+0000 7fa64f7fe640 1 -- 172.21.15.28:0/3224779383 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa66003b580 con 0x5612c76e2580 2024-04-25T01:35:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.714+0000 7fa64f7fe640 1 -- 172.21.15.28:0/3224779383 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6600ba2f0 con 0x5612c76e2580 2024-04-25T01:35:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.714+0000 7fa66e7f5740 1 -- 172.21.15.28:0/3224779383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76e2580 msgr2=0x5612c76f91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.714+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/3224779383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76e2580 0x5612c76f91c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa6600b89c0 tx=0x7fa660009680 comp rx=0 tx=0).stop 2024-04-25T01:35:51.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.714+0000 7fa66e7f5740 1 -- 172.21.15.28:0/3224779383 shutdown_connections 2024-04-25T01:35:51.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.714+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/3224779383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76ed570 0x5612c7705000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.714+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/3224779383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5612c76e2ea0 0x5612c76eca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.714+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/3224779383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76e2580 0x5612c76f91c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.714+0000 7fa66e7f5740 1 -- 172.21.15.28:0/3224779383 >> 172.21.15.28:0/3224779383 conn(0x5612c76de8a0 msgr2=0x5612c76e5740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.714+0000 7fa66e7f5740 1 -- 172.21.15.28:0/3224779383 shutdown_connections 2024-04-25T01:35:51.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.714+0000 7fa66e7f5740 1 -- 172.21.15.28:0/3224779383 wait complete. 2024-04-25T01:35:51.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.715+0000 7fa66e7f5740 1 -- start start 2024-04-25T01:35:51.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.715+0000 7fa66e7f5740 1 Processor -- start 2024-04-25T01:35:51.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.715+0000 7fa66e7f5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76e2580 0x5612c76f58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.715+0000 7fa66e7f5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5612c76e2ea0 0x5612c76f0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.715+0000 7fa66e7f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76ed570 0x5612c76f11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.715+0000 7fa66e7f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641b6510 con 0x5612c76ed570 2024-04-25T01:35:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.715+0000 7fa66e7f5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641b5e40 con 0x5612c76e2580 2024-04-25T01:35:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.715+0000 7fa66e7f5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641b3080 con 0x5612c76e2ea0 2024-04-25T01:35:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.715+0000 7fa66e7f5740 10 client.0 _ll_register_callbacks cb 0x7fa65c012a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.715+0000 7fa66a07e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76ed570 0x5612c76f11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.715+0000 7fa66a07e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76ed570 0x5612c76f11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47184/0 (socket says 172.21.15.28:47184) 2024-04-25T01:35:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.715+0000 7fa66a07e640 1 -- 172.21.15.28:0/784981783 learned_addr learned my addr 172.21.15.28:0/784981783 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.715+0000 7fa66987d640 1 --2- 172.21.15.28:0/784981783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76e2580 0x5612c76f58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.715+0000 7fa66a07e640 1 -- 172.21.15.28:0/784981783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5612c76e2ea0 msgr2=0x5612c76f0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.715+0000 7fa66a07e640 1 --2- 172.21.15.28:0/784981783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5612c76e2ea0 0x5612c76f0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.715+0000 7fa66a07e640 1 -- 172.21.15.28:0/784981783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76e2580 msgr2=0x5612c76f58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:51.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.715+0000 7fa66a07e640 1 --2- 172.21.15.28:0/784981783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76e2580 0x5612c76f58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.715+0000 7fa66a07e640 1 -- 172.21.15.28:0/784981783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612c76f17f0 con 0x5612c76ed570 2024-04-25T01:35:51.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.715+0000 7fa66987d640 1 --2- 172.21.15.28:0/784981783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76e2580 0x5612c76f58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:51.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.716+0000 7fa66a07e640 1 --2- 172.21.15.28:0/784981783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76ed570 0x5612c76f11b0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009ca0 tx=0x7fa65c010c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5f9bc81e5b13b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:51.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.716+0000 7fa64d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:51.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.716+0000 7fa64d7fa640 1 -- 172.21.15.28:0/784981783 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c02c070 con 0x5612c76ed570 2024-04-25T01:35:51.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.716+0000 7fa64d7fa640 1 -- 172.21.15.28:0/784981783 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fa65c0120b0 con 0x5612c76ed570 2024-04-25T01:35:51.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.716+0000 7fa66e7f5740 1 -- 172.21.15.28:0/784981783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5612c764ef00 con 0x5612c76ed570 2024-04-25T01:35:51.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.716+0000 7fa64d7fa640 1 -- 172.21.15.28:0/784981783 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa65c01b040 con 0x5612c76ed570 2024-04-25T01:35:51.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.716+0000 7fa66e7f5740 10 client.16512 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:51.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.716+0000 7fa66e7f5740 1 -- 172.21.15.28:0/784981783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5612c77635f0 con 0x5612c76ed570 2024-04-25T01:35:51.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.716+0000 7fa64d7fa640 1 -- 172.21.15.28:0/784981783 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5612c764ef00 con 0x5612c76ed570 2024-04-25T01:35:51.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.716+0000 7fa64d7fa640 1 -- 172.21.15.28:0/784981783 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa65c028540 con 0x5612c76ed570 2024-04-25T01:35:51.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.716+0000 7fa66e7f5740 10 client.16512 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:51.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.716+0000 7fa66e7f5740 10 client.16512 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.716+0000 7fa66e7f5740 20 client.16512 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.716+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/784981783 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5612c7763cd0 0x5612c7766110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.716+0000 7fa66e7f5740 4 client.16512 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:35:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.716+0000 7fa66e7f5740 1 -- 172.21.15.28:0/784981783 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5612c75cc820 con 0x5612c7763cd0 2024-04-25T01:35:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.717+0000 7fa66987d640 1 --2- 172.21.15.28:0/784981783 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5612c7763cd0 0x5612c7766110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.717+0000 7fa66987d640 1 --2- 172.21.15.28:0/784981783 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5612c7763cd0 0x5612c7766110 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e20c9f09cd0b8f40 server_cookie=8b799642090b8fb0 in_seq=0 out_seq=0 2024-04-25T01:35:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.717+0000 7fa64d7fa640 10 client.16512 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.717+0000 7fa64d7fa640 1 -- 172.21.15.28:0/784981783 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5612c75cc820 con 0x5612c7763cd0 2024-04-25T01:35:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.717+0000 7fa64d7fa640 10 client.16512 handle_command_reply: tid=0 2024-04-25T01:35:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.717+0000 7fa66e7f5740 1 client.16512 shutdown 2024-04-25T01:35:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.717+0000 7fa66e7f5740 1 -- 172.21.15.28:0/784981783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76ed570 msgr2=0x5612c76f11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.717+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/784981783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76ed570 0x5612c76f11b0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009ca0 tx=0x7fa65c010c70 comp rx=0 tx=0).stop 2024-04-25T01:35:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.717+0000 7fa66e7f5740 1 -- 172.21.15.28:0/784981783 shutdown_connections 2024-04-25T01:35:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.717+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/784981783 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5612c7763cd0 0x5612c7766110 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.717+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/784981783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612c76ed570 0x5612c76f11b0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.717+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/784981783 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5612c76e2ea0 0x5612c76f0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.717+0000 7fa66e7f5740 1 --2- 172.21.15.28:0/784981783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612c76e2580 0x5612c76f58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.717+0000 7fa64d7fa640 0 client.16512 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:51.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.717+0000 7fa66e7f5740 1 -- 172.21.15.28:0/784981783 >> 172.21.15.28:0/784981783 conn(0x5612c76de8a0 msgr2=0x5612c76f81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.717+0000 7fa66e7f5740 1 -- 172.21.15.28:0/784981783 shutdown_connections 2024-04-25T01:35:51.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.718+0000 7fa66e7f5740 1 -- 172.21.15.28:0/784981783 wait complete. 2024-04-25T01:35:51.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.718+0000 7fa66e7f5740 20 client.16512 trim_cache size 0 max 16384 2024-04-25T01:35:51.700 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:35:51.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.718+0000 7fa66bae1640 1 -- 172.21.15.28:0/4043275248 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa6500a2d00 msgr2=0x7fa6500a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.718+0000 7fa66bae1640 1 --2- 172.21.15.28:0/4043275248 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa6500a2d00 0x7fa6500a51c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa65c010c40 tx=0x7fa65c017000 comp rx=0 tx=0).stop 2024-04-25T01:35:51.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.718+0000 7fa66bae1640 1 -- 172.21.15.28:0/4043275248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa66406a300 msgr2=0x7fa66419e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:51.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.718+0000 7fa66bae1640 1 --2- 172.21.15.28:0/4043275248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa66406a300 0x7fa66419e5f0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fa660006010 tx=0x7fa6600042f0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.718+0000 7fa66bae1640 1 -- 172.21.15.28:0/4043275248 shutdown_connections 2024-04-25T01:35:51.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.718+0000 7fa66bae1640 1 --2- 172.21.15.28:0/4043275248 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa6500a2d00 0x7fa6500a51c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.718+0000 7fa66bae1640 1 --2- 172.21.15.28:0/4043275248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa664106670 0x7fa6641a2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.718+0000 7fa66bae1640 1 --2- 172.21.15.28:0/4043275248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa664104170 0x7fa66419eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.718+0000 7fa66bae1640 1 --2- 172.21.15.28:0/4043275248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa66406a300 0x7fa66419e5f0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:51.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.718+0000 7fa66bae1640 1 -- 172.21.15.28:0/4043275248 >> 172.21.15.28:0/4043275248 conn(0x7fa6640776a0 msgr2=0x7fa664103150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:51.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.718+0000 7fa66bae1640 1 -- 172.21.15.28:0/4043275248 shutdown_connections 2024-04-25T01:35:51.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:51.718+0000 7fa66bae1640 1 -- 172.21.15.28:0/4043275248 wait complete. 2024-04-25T01:35:52.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:52 smithi130 ceph-mon[32702]: pgmap v88: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 3.1 KiB/s wr, 223 op/s 2024-04-25T01:35:52.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:52 smithi032 ceph-mon[32461]: pgmap v88: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 3.1 KiB/s wr, 223 op/s 2024-04-25T01:35:52.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:52 smithi028 ceph-mon[29390]: pgmap v88: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 3.1 KiB/s wr, 223 op/s 2024-04-25T01:35:52.706 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:35:52.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:35:52.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:35:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.850+0000 7ff85603f640 1 -- 172.21.15.28:0/135804590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff850100610 msgr2=0x7ff850100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.850+0000 7ff85603f640 1 --2- 172.21.15.28:0/135804590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff850100610 0x7ff850100a90 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ff844009930 tx=0x7ff84402a420 comp rx=0 tx=0).stop 2024-04-25T01:35:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.850+0000 7ff85603f640 1 -- 172.21.15.28:0/135804590 shutdown_connections 2024-04-25T01:35:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.850+0000 7ff85603f640 1 --2- 172.21.15.28:0/135804590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff850100fd0 0x7ff85010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.850+0000 7ff85603f640 1 --2- 172.21.15.28:0/135804590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff850100610 0x7ff850100a90 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.850+0000 7ff85603f640 1 --2- 172.21.15.28:0/135804590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8501066b0 0x7ff850106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.850+0000 7ff85603f640 1 -- 172.21.15.28:0/135804590 >> 172.21.15.28:0/135804590 conn(0x7ff850077740 msgr2=0x7ff850077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:52.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.850+0000 7ff85603f640 1 -- 172.21.15.28:0/135804590 shutdown_connections 2024-04-25T01:35:52.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.850+0000 7ff85603f640 1 -- 172.21.15.28:0/135804590 wait complete. 2024-04-25T01:35:52.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.851+0000 7ff85603f640 1 Processor -- start 2024-04-25T01:35:52.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.851+0000 7ff85603f640 1 -- start start 2024-04-25T01:35:52.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.851+0000 7ff85603f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff850100610 0x7ff850197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.851+0000 7ff85603f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff850100fd0 0x7ff850198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.851+0000 7ff85603f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8501066b0 0x7ff85019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.851+0000 7ff85603f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff850069400 con 0x7ff8501066b0 2024-04-25T01:35:52.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.851+0000 7ff85603f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff850069280 con 0x7ff850100610 2024-04-25T01:35:52.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.851+0000 7ff85603f640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff850069580 con 0x7ff850100fd0 2024-04-25T01:35:52.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.851+0000 7ff84ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8501066b0 0x7ff85019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.851+0000 7ff84ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8501066b0 0x7ff85019c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47194/0 (socket says 172.21.15.28:47194) 2024-04-25T01:35:52.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.851+0000 7ff84ffff640 1 -- 172.21.15.28:0/2586617809 learned_addr learned my addr 172.21.15.28:0/2586617809 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:52.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.851+0000 7ff84effd640 1 --2- 172.21.15.28:0/2586617809 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff850100fd0 0x7ff850198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.851+0000 7ff84f7fe640 1 --2- 172.21.15.28:0/2586617809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff850100610 0x7ff850197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.851+0000 7ff84ffff640 1 -- 172.21.15.28:0/2586617809 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff850100fd0 msgr2=0x7ff850198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:52.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.851+0000 7ff84ffff640 1 --2- 172.21.15.28:0/2586617809 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff850100fd0 0x7ff850198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.851+0000 7ff84ffff640 1 -- 172.21.15.28:0/2586617809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff850100610 msgr2=0x7ff850197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:52.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.852+0000 7ff84ffff640 1 --2- 172.21.15.28:0/2586617809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff850100610 0x7ff850197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.852+0000 7ff84ffff640 1 -- 172.21.15.28:0/2586617809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff85019cfb0 con 0x7ff8501066b0 2024-04-25T01:35:52.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.852+0000 7ff84ffff640 1 --2- 172.21.15.28:0/2586617809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8501066b0 0x7ff85019c880 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff84800eb10 tx=0x7ff84800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:52.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.852+0000 7ff84cff9640 1 -- 172.21.15.28:0/2586617809 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff848004b20 con 0x7ff8501066b0 2024-04-25T01:35:52.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.852+0000 7ff84cff9640 1 -- 172.21.15.28:0/2586617809 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff848004cc0 con 0x7ff8501066b0 2024-04-25T01:35:52.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.852+0000 7ff85603f640 1 -- 172.21.15.28:0/2586617809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff85019d2a0 con 0x7ff8501066b0 2024-04-25T01:35:52.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.852+0000 7ff84cff9640 1 -- 172.21.15.28:0/2586617809 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff848010760 con 0x7ff8501066b0 2024-04-25T01:35:52.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.852+0000 7ff85603f640 1 -- 172.21.15.28:0/2586617809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8501a4b30 con 0x7ff8501066b0 2024-04-25T01:35:52.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.854+0000 7ff84cff9640 1 -- 172.21.15.28:0/2586617809 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff848005020 con 0x7ff8501066b0 2024-04-25T01:35:52.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff858e2b740 1 Processor -- start 2024-04-25T01:35:52.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff858e2b740 1 -- start start 2024-04-25T01:35:52.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff84cff9640 1 --2- 172.21.15.28:0/2586617809 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8380a2d00 0x7ff8380a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff858e2b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d88980 0x555720d86cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff858e2b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d87290 0x555720d87710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff858e2b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d96420 0x555720d98810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff858e2b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555720ceb2c0 con 0x555720d96420 2024-04-25T01:35:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff858e2b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555720c93910 con 0x555720d88980 2024-04-25T01:35:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff858e2b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x555720ce7280 con 0x555720d87290 2024-04-25T01:35:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff84effd640 1 --2- 172.21.15.28:0/2586617809 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8380a2d00 0x7ff8380a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff84cff9640 1 -- 172.21.15.28:0/2586617809 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff8480cfda0 con 0x7ff8501066b0 2024-04-25T01:35:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff84f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d88980 0x555720d86cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff84ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d96420 0x555720d98810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff84f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d88980 0x555720d86cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:39274/0 (socket says 172.21.15.28:39274) 2024-04-25T01:35:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff84ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d96420 0x555720d98810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47208/0 (socket says 172.21.15.28:47208) 2024-04-25T01:35:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff84f7fe640 1 -- 172.21.15.28:0/2898021849 learned_addr learned my addr 172.21.15.28:0/2898021849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff84effd640 1 --2- 172.21.15.28:0/2898021849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d87290 0x555720d87710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff84ffff640 1 -- 172.21.15.28:0/2898021849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d87290 msgr2=0x555720d87710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff84ffff640 1 --2- 172.21.15.28:0/2898021849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d87290 0x555720d87710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff84ffff640 1 -- 172.21.15.28:0/2898021849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d88980 msgr2=0x555720d86cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff84ffff640 1 --2- 172.21.15.28:0/2898021849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d88980 0x555720d86cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff84ffff640 1 -- 172.21.15.28:0/2898021849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555720ca4fb0 con 0x555720d96420 2024-04-25T01:35:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.855+0000 7ff84f7fe640 1 --2- 172.21.15.28:0/2898021849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d88980 0x555720d86cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:35:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.856+0000 7ff84effd640 1 --2- 172.21.15.28:0/2586617809 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8380a2d00 0x7ff8380a51c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff844006010 tx=0x7ff844004310 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.856+0000 7ff84ffff640 1 --2- 172.21.15.28:0/2898021849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d96420 0x555720d98810 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ff8480cd150 tx=0x7ff84809bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=245896fc56df416c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.856+0000 7ff835ffb640 1 -- 172.21.15.28:0/2898021849 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84809c620 con 0x555720d96420 2024-04-25T01:35:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.856+0000 7ff835ffb640 1 -- 172.21.15.28:0/2898021849 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff84809bf20 con 0x555720d96420 2024-04-25T01:35:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.856+0000 7ff835ffb640 1 -- 172.21.15.28:0/2898021849 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8480a4700 con 0x555720d96420 2024-04-25T01:35:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.856+0000 7ff858e2b740 1 -- 172.21.15.28:0/2898021849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d96420 msgr2=0x555720d98810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.856+0000 7ff858e2b740 1 --2- 172.21.15.28:0/2898021849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d96420 0x555720d98810 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ff8480cd150 tx=0x7ff84809bcf0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.857+0000 7ff858e2b740 1 -- 172.21.15.28:0/2898021849 shutdown_connections 2024-04-25T01:35:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.857+0000 7ff858e2b740 1 --2- 172.21.15.28:0/2898021849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d96420 0x555720d98810 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.857+0000 7ff858e2b740 1 --2- 172.21.15.28:0/2898021849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d87290 0x555720d87710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.857+0000 7ff858e2b740 1 --2- 172.21.15.28:0/2898021849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d88980 0x555720d86cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.857+0000 7ff858e2b740 1 -- 172.21.15.28:0/2898021849 >> 172.21.15.28:0/2898021849 conn(0x555720d4f950 msgr2=0x555720d837f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.857+0000 7ff858e2b740 1 -- 172.21.15.28:0/2898021849 shutdown_connections 2024-04-25T01:35:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.857+0000 7ff858e2b740 1 -- 172.21.15.28:0/2898021849 wait complete. 2024-04-25T01:35:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.857+0000 7ff858e2b740 1 -- start start 2024-04-25T01:35:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.857+0000 7ff858e2b740 1 Processor -- start 2024-04-25T01:35:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.857+0000 7ff858e2b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 0x555720d933a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.857+0000 7ff858e2b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d88980 0x555720d938e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.857+0000 7ff858e2b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d96420 0x555720d93e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.857+0000 7ff858e2b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555720d992b0 con 0x555720d87290 2024-04-25T01:35:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.857+0000 7ff858e2b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555720ca6e90 con 0x555720d88980 2024-04-25T01:35:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.857+0000 7ff858e2b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x555720d99430 con 0x555720d96420 2024-04-25T01:35:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.857+0000 7ff858e2b740 10 client.0 _ll_register_callbacks cb 0x555720ce2970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.858+0000 7ff84f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 0x555720d933a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.858+0000 7ff84f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 0x555720d933a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47216/0 (socket says 172.21.15.28:47216) 2024-04-25T01:35:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.858+0000 7ff84f7fe640 1 -- 172.21.15.28:0/1449712451 learned_addr learned my addr 172.21.15.28:0/1449712451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.858+0000 7ff84effd640 1 --2- 172.21.15.28:0/1449712451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d88980 0x555720d938e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.858+0000 7ff84ffff640 1 --2- 172.21.15.28:0/1449712451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d96420 0x555720d93e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.858+0000 7ff84f7fe640 1 -- 172.21.15.28:0/1449712451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d96420 msgr2=0x555720d93e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.858+0000 7ff84f7fe640 1 --2- 172.21.15.28:0/1449712451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d96420 0x555720d93e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.858+0000 7ff84f7fe640 1 -- 172.21.15.28:0/1449712451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d88980 msgr2=0x555720d938e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.858+0000 7ff84f7fe640 1 --2- 172.21.15.28:0/1449712451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d88980 0x555720d938e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.858+0000 7ff84f7fe640 1 -- 172.21.15.28:0/1449712451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555720ac77e0 con 0x555720d87290 2024-04-25T01:35:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.858+0000 7ff84f7fe640 1 --2- 172.21.15.28:0/1449712451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 0x555720d933a0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7ff840012810 tx=0x7ff840012cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a32277bdcc67fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.858+0000 7ff82effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.858+0000 7ff82effd640 1 -- 172.21.15.28:0/1449712451 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff840004030 con 0x555720d87290 2024-04-25T01:35:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.858+0000 7ff82effd640 1 -- 172.21.15.28:0/1449712451 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff8400041d0 con 0x555720d87290 2024-04-25T01:35:52.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.858+0000 7ff858e2b740 1 -- 172.21.15.28:0/1449712451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff84809bf20 con 0x555720d87290 2024-04-25T01:35:52.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.858+0000 7ff82effd640 1 -- 172.21.15.28:0/1449712451 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff8400043d0 con 0x555720d87290 2024-04-25T01:35:52.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.858+0000 7ff858e2b740 10 client.16524 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:52.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.858+0000 7ff858e2b740 1 -- 172.21.15.28:0/1449712451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555720b27f70 con 0x555720d87290 2024-04-25T01:35:52.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.859+0000 7ff82effd640 1 -- 172.21.15.28:0/1449712451 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff84809bf20 con 0x555720d87290 2024-04-25T01:35:52.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.859+0000 7ff82effd640 1 -- 172.21.15.28:0/1449712451 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff840003b80 con 0x555720d87290 2024-04-25T01:35:52.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.859+0000 7ff858e2b740 10 client.16524 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:52.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.859+0000 7ff858e2b740 10 client.16524 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:52.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.859+0000 7ff858e2b740 20 client.16524 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:52.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.859+0000 7ff858e2b740 1 --2- 172.21.15.28:0/1449712451 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555720da65d0 0x555720da8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.859+0000 7ff858e2b740 4 client.16524 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:52.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.859+0000 7ff858e2b740 1 -- 172.21.15.28:0/1449712451 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555720cbf5a0 con 0x555720da65d0 2024-04-25T01:35:52.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.859+0000 7ff84effd640 1 --2- 172.21.15.28:0/1449712451 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555720da65d0 0x555720da8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.860+0000 7ff82effd640 10 client.16524 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:52.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.860+0000 7ff84effd640 1 --2- 172.21.15.28:0/1449712451 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555720da65d0 0x555720da8a50 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=142a7c3e11dbfa42 server_cookie=9018e7c665cff23b in_seq=0 out_seq=0 2024-04-25T01:35:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.860+0000 7ff82effd640 1 -- 172.21.15.28:0/1449712451 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555720cbf5a0 con 0x555720da65d0 2024-04-25T01:35:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.860+0000 7ff82effd640 10 client.16524 handle_command_reply: tid=0 2024-04-25T01:35:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.860+0000 7ff858e2b740 1 client.16524 shutdown 2024-04-25T01:35:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.860+0000 7ff858e2b740 1 -- 172.21.15.28:0/1449712451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 msgr2=0x555720d933a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.860+0000 7ff858e2b740 1 --2- 172.21.15.28:0/1449712451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 0x555720d933a0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7ff840012810 tx=0x7ff840012cd0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.860+0000 7ff858e2b740 1 -- 172.21.15.28:0/1449712451 shutdown_connections 2024-04-25T01:35:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.861+0000 7ff858e2b740 1 --2- 172.21.15.28:0/1449712451 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555720da65d0 0x555720da8a50 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.861+0000 7ff858e2b740 1 --2- 172.21.15.28:0/1449712451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d96420 0x555720d93e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.861+0000 7ff858e2b740 1 --2- 172.21.15.28:0/1449712451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d88980 0x555720d938e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.861+0000 7ff82effd640 0 client.16524 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.861+0000 7ff858e2b740 1 --2- 172.21.15.28:0/1449712451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 0x555720d933a0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.861+0000 7ff858e2b740 1 -- 172.21.15.28:0/1449712451 >> 172.21.15.28:0/1449712451 conn(0x555720d4f950 msgr2=0x555720d89660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.861+0000 7ff858e2b740 1 -- 172.21.15.28:0/1449712451 shutdown_connections 2024-04-25T01:35:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.861+0000 7ff858e2b740 1 -- 172.21.15.28:0/1449712451 wait complete. 2024-04-25T01:35:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.861+0000 7ff858e2b740 20 client.16524 trim_cache size 0 max 16384 2024-04-25T01:35:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.878+0000 7ff858e2b740 1 Processor -- start 2024-04-25T01:35:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff858e2b740 1 -- start start 2024-04-25T01:35:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff858e2b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 0x555720d95be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff858e2b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d88980 0x555720da21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff858e2b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d96420 0x555720d92500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff858e2b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8501ac850 con 0x555720d87290 2024-04-25T01:35:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff858e2b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff8501ae390 con 0x555720d88980 2024-04-25T01:35:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff858e2b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff8501acf20 con 0x555720d96420 2024-04-25T01:35:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff84f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 0x555720d95be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff84f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 0x555720d95be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47228/0 (socket says 172.21.15.28:47228) 2024-04-25T01:35:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff84f7fe640 1 -- 172.21.15.28:0/775650378 learned_addr learned my addr 172.21.15.28:0/775650378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff84ffff640 1 --2- 172.21.15.28:0/775650378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d96420 0x555720d92500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff84f7fe640 1 -- 172.21.15.28:0/775650378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d96420 msgr2=0x555720d92500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff84effd640 1 --2- 172.21.15.28:0/775650378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d88980 0x555720da21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff84f7fe640 1 --2- 172.21.15.28:0/775650378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d96420 0x555720d92500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff84f7fe640 1 -- 172.21.15.28:0/775650378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d88980 msgr2=0x555720da21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:52.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff84f7fe640 1 --2- 172.21.15.28:0/775650378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d88980 0x555720da21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff84f7fe640 1 -- 172.21.15.28:0/775650378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555720d83a00 con 0x555720d87290 2024-04-25T01:35:52.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff84effd640 1 --2- 172.21.15.28:0/775650378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d88980 0x555720da21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:52.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff84f7fe640 1 --2- 172.21.15.28:0/775650378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 0x555720d95be0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7ff8400059f0 tx=0x7ff8400122e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a63a51d564be5e44 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:52.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff8357fa640 1 -- 172.21.15.28:0/775650378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84001c5d0 con 0x555720d87290 2024-04-25T01:35:52.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff8357fa640 1 -- 172.21.15.28:0/775650378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff84001c770 con 0x555720d87290 2024-04-25T01:35:52.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.879+0000 7ff8357fa640 1 -- 172.21.15.28:0/775650378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84000fd20 con 0x555720d87290 2024-04-25T01:35:52.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.880+0000 7ff858e2b740 1 -- 172.21.15.28:0/775650378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 msgr2=0x555720d95be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:52.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.880+0000 7ff858e2b740 1 --2- 172.21.15.28:0/775650378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 0x555720d95be0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7ff8400059f0 tx=0x7ff8400122e0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.880+0000 7ff858e2b740 1 -- 172.21.15.28:0/775650378 shutdown_connections 2024-04-25T01:35:52.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.880+0000 7ff858e2b740 1 --2- 172.21.15.28:0/775650378 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d96420 0x555720d92500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.880+0000 7ff858e2b740 1 --2- 172.21.15.28:0/775650378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d88980 0x555720da21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.880+0000 7ff858e2b740 1 --2- 172.21.15.28:0/775650378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 0x555720d95be0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.880+0000 7ff858e2b740 1 -- 172.21.15.28:0/775650378 >> 172.21.15.28:0/775650378 conn(0x555720d4f950 msgr2=0x555720d820c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:52.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.880+0000 7ff858e2b740 1 -- 172.21.15.28:0/775650378 shutdown_connections 2024-04-25T01:35:52.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.880+0000 7ff858e2b740 1 -- 172.21.15.28:0/775650378 wait complete. 2024-04-25T01:35:52.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.880+0000 7ff858e2b740 1 -- start start 2024-04-25T01:35:52.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.881+0000 7ff858e2b740 1 Processor -- start 2024-04-25T01:35:52.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.881+0000 7ff858e2b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 0x555720d9e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.881+0000 7ff858e2b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d88980 0x555720d9ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.881+0000 7ff858e2b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d96420 0x555720d9b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.881+0000 7ff858e2b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8501adcc0 con 0x555720d87290 2024-04-25T01:35:52.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.881+0000 7ff858e2b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff8501ad5f0 con 0x555720d96420 2024-04-25T01:35:52.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.881+0000 7ff858e2b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff8501aa830 con 0x555720d88980 2024-04-25T01:35:52.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.881+0000 7ff858e2b740 10 client.0 _ll_register_callbacks cb 0x7ff840009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:52.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.881+0000 7ff84f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 0x555720d9e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.881+0000 7ff84f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 0x555720d9e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47230/0 (socket says 172.21.15.28:47230) 2024-04-25T01:35:52.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.881+0000 7ff84f7fe640 1 -- 172.21.15.28:0/3332889393 learned_addr learned my addr 172.21.15.28:0/3332889393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:52.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.881+0000 7ff84effd640 1 --2- 172.21.15.28:0/3332889393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d88980 0x555720d9ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.881+0000 7ff84ffff640 1 --2- 172.21.15.28:0/3332889393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d96420 0x555720d9b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.881+0000 7ff84f7fe640 1 -- 172.21.15.28:0/3332889393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d88980 msgr2=0x555720d9ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:52.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.881+0000 7ff84f7fe640 1 --2- 172.21.15.28:0/3332889393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d88980 0x555720d9ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.881+0000 7ff84f7fe640 1 -- 172.21.15.28:0/3332889393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d96420 msgr2=0x555720d9b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:52.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.881+0000 7ff84f7fe640 1 --2- 172.21.15.28:0/3332889393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d96420 0x555720d9b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.881+0000 7ff84f7fe640 1 -- 172.21.15.28:0/3332889393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557209bf170 con 0x555720d87290 2024-04-25T01:35:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.881+0000 7ff84ffff640 1 --2- 172.21.15.28:0/3332889393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d96420 0x555720d9b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.882+0000 7ff84f7fe640 1 --2- 172.21.15.28:0/3332889393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 0x555720d9e6c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7ff840009f00 tx=0x7ff8400041b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12be6449d85b3045 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.882+0000 7ff82effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.882+0000 7ff82effd640 1 -- 172.21.15.28:0/3332889393 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84001c5d0 con 0x555720d87290 2024-04-25T01:35:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.882+0000 7ff82effd640 1 -- 172.21.15.28:0/3332889393 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff84001cc80 con 0x555720d87290 2024-04-25T01:35:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.882+0000 7ff858e2b740 1 -- 172.21.15.28:0/3332889393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff84001c770 con 0x555720d87290 2024-04-25T01:35:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.882+0000 7ff82effd640 1 -- 172.21.15.28:0/3332889393 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff840005040 con 0x555720d87290 2024-04-25T01:35:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.882+0000 7ff858e2b740 10 client.16530 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.882+0000 7ff858e2b740 1 -- 172.21.15.28:0/3332889393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555720e0d200 con 0x555720d87290 2024-04-25T01:35:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.882+0000 7ff82effd640 1 -- 172.21.15.28:0/3332889393 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff84001c770 con 0x555720d87290 2024-04-25T01:35:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.882+0000 7ff82effd640 1 -- 172.21.15.28:0/3332889393 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff8400039d0 con 0x555720d87290 2024-04-25T01:35:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.882+0000 7ff858e2b740 10 client.16530 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.882+0000 7ff858e2b740 10 client.16530 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.882+0000 7ff858e2b740 20 client.16530 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.882+0000 7ff858e2b740 1 --2- 172.21.15.28:0/3332889393 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555720e0d6d0 0x555720e0fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.882+0000 7ff858e2b740 4 client.16530 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:35:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.882+0000 7ff858e2b740 1 -- 172.21.15.28:0/3332889393 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555720cf8800 con 0x555720e0d6d0 2024-04-25T01:35:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.882+0000 7ff84effd640 1 --2- 172.21.15.28:0/3332889393 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555720e0d6d0 0x555720e0fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.883+0000 7ff84effd640 1 --2- 172.21.15.28:0/3332889393 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555720e0d6d0 0x555720e0fac0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32dfa976285f1597 server_cookie=3f1fa641f13a9920 in_seq=0 out_seq=0 2024-04-25T01:35:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.883+0000 7ff82effd640 10 client.16530 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.884+0000 7ff82effd640 1 -- 172.21.15.28:0/3332889393 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555720cf8800 con 0x555720e0d6d0 2024-04-25T01:35:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.884+0000 7ff82effd640 10 client.16530 handle_command_reply: tid=0 2024-04-25T01:35:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.884+0000 7ff858e2b740 1 client.16530 shutdown 2024-04-25T01:35:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.884+0000 7ff858e2b740 1 -- 172.21.15.28:0/3332889393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 msgr2=0x555720d9e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.884+0000 7ff858e2b740 1 --2- 172.21.15.28:0/3332889393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 0x555720d9e6c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7ff840009f00 tx=0x7ff8400041b0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.884+0000 7ff858e2b740 1 -- 172.21.15.28:0/3332889393 shutdown_connections 2024-04-25T01:35:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.884+0000 7ff858e2b740 1 --2- 172.21.15.28:0/3332889393 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555720e0d6d0 0x555720e0fac0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.884+0000 7ff858e2b740 1 --2- 172.21.15.28:0/3332889393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555720d96420 0x555720d9b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.884+0000 7ff858e2b740 1 --2- 172.21.15.28:0/3332889393 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555720d88980 0x555720d9ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.884+0000 7ff82effd640 0 client.16530 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:52.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.884+0000 7ff858e2b740 1 --2- 172.21.15.28:0/3332889393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555720d87290 0x555720d9e6c0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.884+0000 7ff858e2b740 1 -- 172.21.15.28:0/3332889393 >> 172.21.15.28:0/3332889393 conn(0x555720d4f950 msgr2=0x555720d93340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:52.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.884+0000 7ff858e2b740 1 -- 172.21.15.28:0/3332889393 shutdown_connections 2024-04-25T01:35:52.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.884+0000 7ff858e2b740 1 -- 172.21.15.28:0/3332889393 wait complete. 2024-04-25T01:35:52.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.884+0000 7ff858e2b740 20 client.16530 trim_cache size 0 max 16384 2024-04-25T01:35:52.873 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:35:52.873 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:35:52.873 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "9a5379a3-b30d-462c-93d4-83c322c91233", 2024-04-25T01:35:52.874 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:35:52.874 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:35:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.885+0000 7ff85603f640 1 -- 172.21.15.28:0/2586617809 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8380a2d00 msgr2=0x7ff8380a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.885+0000 7ff85603f640 1 --2- 172.21.15.28:0/2586617809 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8380a2d00 0x7ff8380a51c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff844006010 tx=0x7ff844004310 comp rx=0 tx=0).stop 2024-04-25T01:35:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.885+0000 7ff85603f640 1 -- 172.21.15.28:0/2586617809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8501066b0 msgr2=0x7ff85019c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.885+0000 7ff85603f640 1 --2- 172.21.15.28:0/2586617809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8501066b0 0x7ff85019c880 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff84800eb10 tx=0x7ff84800c2d0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.885+0000 7ff85603f640 1 -- 172.21.15.28:0/2586617809 shutdown_connections 2024-04-25T01:35:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.885+0000 7ff85603f640 1 --2- 172.21.15.28:0/2586617809 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8380a2d00 0x7ff8380a51c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.885+0000 7ff85603f640 1 --2- 172.21.15.28:0/2586617809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8501066b0 0x7ff85019c880 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.885+0000 7ff85603f640 1 --2- 172.21.15.28:0/2586617809 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff850100fd0 0x7ff850198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.885+0000 7ff85603f640 1 --2- 172.21.15.28:0/2586617809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff850100610 0x7ff850197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.885+0000 7ff85603f640 1 -- 172.21.15.28:0/2586617809 >> 172.21.15.28:0/2586617809 conn(0x7ff850077740 msgr2=0x7ff8500fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.885+0000 7ff85603f640 1 -- 172.21.15.28:0/2586617809 shutdown_connections 2024-04-25T01:35:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:52.885+0000 7ff85603f640 1 -- 172.21.15.28:0/2586617809 wait complete. 2024-04-25T01:35:52.876 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:35:52.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:35:52.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.004+0000 7fe7f7493640 1 -- 172.21.15.28:0/2599460991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f0100fd0 msgr2=0x7fe7f010d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:52.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.004+0000 7fe7f7493640 1 --2- 172.21.15.28:0/2599460991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f0100fd0 0x7fe7f010d4e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8009cd0 tx=0x7fe7e802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.004+0000 7fe7f7493640 1 -- 172.21.15.28:0/2599460991 shutdown_connections 2024-04-25T01:35:52.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.004+0000 7fe7f7493640 1 --2- 172.21.15.28:0/2599460991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f0100fd0 0x7fe7f010d4e0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.004+0000 7fe7f7493640 1 --2- 172.21.15.28:0/2599460991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7f0100610 0x7fe7f0100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.004+0000 7fe7f7493640 1 --2- 172.21.15.28:0/2599460991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7f01066b0 0x7fe7f0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.004+0000 7fe7f7493640 1 -- 172.21.15.28:0/2599460991 >> 172.21.15.28:0/2599460991 conn(0x7fe7f00769c0 msgr2=0x7fe7f0076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:52.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.004+0000 7fe7f7493640 1 -- 172.21.15.28:0/2599460991 shutdown_connections 2024-04-25T01:35:52.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.004+0000 7fe7f7493640 1 -- 172.21.15.28:0/2599460991 wait complete. 2024-04-25T01:35:52.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.005+0000 7fe7f7493640 1 Processor -- start 2024-04-25T01:35:52.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.005+0000 7fe7f7493640 1 -- start start 2024-04-25T01:35:52.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.005+0000 7fe7f7493640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7f0100610 0x7fe7f0196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.005+0000 7fe7f7493640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f0100fd0 0x7fe7f0197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.005+0000 7fe7f7493640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7f01066b0 0x7fe7f019b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.005+0000 7fe7f7493640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f01109f0 con 0x7fe7f0100fd0 2024-04-25T01:35:52.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.005+0000 7fe7f7493640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f0110870 con 0x7fe7f01066b0 2024-04-25T01:35:52.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.005+0000 7fe7f7493640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f0110b70 con 0x7fe7f0100610 2024-04-25T01:35:52.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.005+0000 7fe7f4a2e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f0100fd0 0x7fe7f0197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.005+0000 7fe7f4a2e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f0100fd0 0x7fe7f0197390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47250/0 (socket says 172.21.15.28:47250) 2024-04-25T01:35:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.005+0000 7fe7f4a2e640 1 -- 172.21.15.28:0/2800094342 learned_addr learned my addr 172.21.15.28:0/2800094342 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.006+0000 7fe7f522f640 1 --2- 172.21.15.28:0/2800094342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7f0100610 0x7fe7f0196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.006+0000 7fe7f5a30640 1 --2- 172.21.15.28:0/2800094342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7f01066b0 0x7fe7f019b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.006+0000 7fe7f4a2e640 1 -- 172.21.15.28:0/2800094342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7f0100610 msgr2=0x7fe7f0196e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.006+0000 7fe7f4a2e640 1 --2- 172.21.15.28:0/2800094342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7f0100610 0x7fe7f0196e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.006+0000 7fe7f4a2e640 1 -- 172.21.15.28:0/2800094342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7f01066b0 msgr2=0x7fe7f019b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.006+0000 7fe7f4a2e640 1 --2- 172.21.15.28:0/2800094342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7f01066b0 0x7fe7f019b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.006+0000 7fe7f4a2e640 1 -- 172.21.15.28:0/2800094342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f019bdf0 con 0x7fe7f0100fd0 2024-04-25T01:35:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.006+0000 7fe7f5a30640 1 --2- 172.21.15.28:0/2800094342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7f01066b0 0x7fe7f019b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:35:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.006+0000 7fe7f4a2e640 1 --2- 172.21.15.28:0/2800094342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f0100fd0 0x7fe7f0197390 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fe7e000c890 tx=0x7fe7e000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.006+0000 7fe7e67fc640 1 -- 172.21.15.28:0/2800094342 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e0004220 con 0x7fe7f0100fd0 2024-04-25T01:35:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.006+0000 7fe7f7493640 1 -- 172.21.15.28:0/2800094342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7f019c0e0 con 0x7fe7f0100fd0 2024-04-25T01:35:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.006+0000 7fe7f7493640 1 -- 172.21.15.28:0/2800094342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7f01a3970 con 0x7fe7f0100fd0 2024-04-25T01:35:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.006+0000 7fe7e67fc640 1 -- 172.21.15.28:0/2800094342 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7e00043c0 con 0x7fe7f0100fd0 2024-04-25T01:35:52.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.006+0000 7fe7e67fc640 1 -- 172.21.15.28:0/2800094342 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e001d060 con 0x7fe7f0100fd0 2024-04-25T01:35:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.008+0000 7fe7e67fc640 1 -- 172.21.15.28:0/2800094342 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe7e0005020 con 0x7fe7f0100fd0 2024-04-25T01:35:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.009+0000 7fe7fa19f740 1 Processor -- start 2024-04-25T01:35:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.009+0000 7fe7fa19f740 1 -- start start 2024-04-25T01:35:52.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.009+0000 7fe7fa19f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f723942550 0x55f723942930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.009+0000 7fe7fa19f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f723942e70 0x55f72394f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.009+0000 7fe7fa19f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72394fc20 0x55f723952010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.009+0000 7fe7fa19f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f7238a56b0 con 0x55f72394fc20 2024-04-25T01:35:52.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.009+0000 7fe7fa19f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f72384d880 con 0x55f723942550 2024-04-25T01:35:52.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.009+0000 7fe7fa19f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55f7238a1670 con 0x55f723942e70 2024-04-25T01:35:52.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.009+0000 7fe7e67fc640 1 --2- 172.21.15.28:0/2800094342 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe7dc0a2d00 0x7fe7dc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.009+0000 7fe7f4a2e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72394fc20 0x55f723952010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.009+0000 7fe7f522f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f723942e70 0x55f72394f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.009+0000 7fe7f4a2e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72394fc20 0x55f723952010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47266/0 (socket says 172.21.15.28:47266) 2024-04-25T01:35:52.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.009+0000 7fe7f4a2e640 1 -- 172.21.15.28:0/1261094268 learned_addr learned my addr 172.21.15.28:0/1261094268 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:52.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.009+0000 7fe7f5a30640 1 --2- 172.21.15.28:0/1261094268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f723942550 0x55f723942930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.009+0000 7fe7f5a30640 1 --2- 172.21.15.28:0/2800094342 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe7dc0a2d00 0x7fe7dc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.009+0000 7fe7e67fc640 1 -- 172.21.15.28:0/2800094342 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fe7e00cf0c0 con 0x7fe7f0100fd0 2024-04-25T01:35:52.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.010+0000 7fe7f4a2e640 1 -- 172.21.15.28:0/1261094268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f723942e70 msgr2=0x55f72394f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:52.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.010+0000 7fe7f4a2e640 1 --2- 172.21.15.28:0/1261094268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f723942e70 0x55f72394f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.010+0000 7fe7f4a2e640 1 -- 172.21.15.28:0/1261094268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f723942550 msgr2=0x55f723942930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:52.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.010+0000 7fe7f4a2e640 1 --2- 172.21.15.28:0/1261094268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f723942550 0x55f723942930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.010+0000 7fe7f4a2e640 1 -- 172.21.15.28:0/1261094268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7236837d0 con 0x55f72394fc20 2024-04-25T01:35:52.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.010+0000 7fe7f5a30640 1 --2- 172.21.15.28:0/1261094268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f723942550 0x55f723942930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:35:52.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.010+0000 7fe7f4a2e640 1 --2- 172.21.15.28:0/1261094268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72394fc20 0x55f723952010 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0005550 tx=0x7fe7e0004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c10ae78d61a76c42 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:52.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.010+0000 7fe7f5a30640 1 --2- 172.21.15.28:0/2800094342 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe7dc0a2d00 0x7fe7dc0a51c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8009a60 tx=0x7fe7e80095c0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:52.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.010+0000 7fe7db7fe640 1 -- 172.21.15.28:0/1261094268 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e009b9a0 con 0x55f72394fc20 2024-04-25T01:35:52.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.010+0000 7fe7db7fe640 1 -- 172.21.15.28:0/1261094268 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7e009b2a0 con 0x55f72394fc20 2024-04-25T01:35:52.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.010+0000 7fe7db7fe640 1 -- 172.21.15.28:0/1261094268 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e00cc750 con 0x55f72394fc20 2024-04-25T01:35:52.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.011+0000 7fe7fa19f740 1 -- 172.21.15.28:0/1261094268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72394fc20 msgr2=0x55f723952010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.011+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/1261094268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72394fc20 0x55f723952010 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0005550 tx=0x7fe7e0004570 comp rx=0 tx=0).stop 2024-04-25T01:35:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.011+0000 7fe7fa19f740 1 -- 172.21.15.28:0/1261094268 shutdown_connections 2024-04-25T01:35:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.011+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/1261094268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72394fc20 0x55f723952010 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.011+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/1261094268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f723942e70 0x55f72394f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.011+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/1261094268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f723942550 0x55f723942930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.011+0000 7fe7fa19f740 1 -- 172.21.15.28:0/1261094268 >> 172.21.15.28:0/1261094268 conn(0x55f72393eb60 msgr2=0x55f72393ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.011+0000 7fe7fa19f740 1 -- 172.21.15.28:0/1261094268 shutdown_connections 2024-04-25T01:35:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.011+0000 7fe7fa19f740 1 -- 172.21.15.28:0/1261094268 wait complete. 2024-04-25T01:35:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.011+0000 7fe7fa19f740 1 -- start start 2024-04-25T01:35:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.011+0000 7fe7fa19f740 1 Processor -- start 2024-04-25T01:35:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7fa19f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f723942550 0x55f72394cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7fa19f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f723942e70 0x55f72394d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7fa19f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f72394fc20 0x55f72394d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7fa19f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f723952850 con 0x55f723942e70 2024-04-25T01:35:52.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7fa19f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f723860f50 con 0x55f723942550 2024-04-25T01:35:52.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7fa19f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55f7239529d0 con 0x55f72394fc20 2024-04-25T01:35:52.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7fa19f740 10 client.0 _ll_register_callbacks cb 0x55f72389c000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:52.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7f4a2e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f723942e70 0x55f72394d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7f4a2e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f723942e70 0x55f72394d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47270/0 (socket says 172.21.15.28:47270) 2024-04-25T01:35:52.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7f4a2e640 1 -- 172.21.15.28:0/2532396340 learned_addr learned my addr 172.21.15.28:0/2532396340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:52.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7f522f640 1 --2- 172.21.15.28:0/2532396340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f723942550 0x55f72394cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7f5a30640 1 --2- 172.21.15.28:0/2532396340 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f72394fc20 0x55f72394d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7f4a2e640 1 -- 172.21.15.28:0/2532396340 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f72394fc20 msgr2=0x55f72394d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:52.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7f4a2e640 1 --2- 172.21.15.28:0/2532396340 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f72394fc20 0x55f72394d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7f4a2e640 1 -- 172.21.15.28:0/2532396340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f723942550 msgr2=0x55f72394cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:52.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7f4a2e640 1 --2- 172.21.15.28:0/2532396340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f723942550 0x55f72394cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:52.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7f4a2e640 1 -- 172.21.15.28:0/2532396340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7235b93f0 con 0x55f723942e70 2024-04-25T01:35:52.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7f4a2e640 1 --2- 172.21.15.28:0/2532396340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f723942e70 0x55f72394d0d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fe7e00058c0 tx=0x7fe7e009ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6967f72c2b8aab15 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:52.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:52.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7d8ff9640 1 -- 172.21.15.28:0/2532396340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e009b9a0 con 0x55f723942e70 2024-04-25T01:35:52.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7d8ff9640 1 -- 172.21.15.28:0/2532396340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fe7e0097910 con 0x55f723942e70 2024-04-25T01:35:52.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.012+0000 7fe7fa19f740 1 -- 172.21.15.28:0/2532396340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7e009b2a0 con 0x55f723942e70 2024-04-25T01:35:52.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.013+0000 7fe7d8ff9640 1 -- 172.21.15.28:0/2532396340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe7e00b9050 con 0x55f723942e70 2024-04-25T01:35:52.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.013+0000 7fe7fa19f740 10 client.16542 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:52.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.013+0000 7fe7fa19f740 1 -- 172.21.15.28:0/2532396340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f7235790f0 con 0x55f723942e70 2024-04-25T01:35:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.013+0000 7fe7d8ff9640 1 -- 172.21.15.28:0/2532396340 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7e009b2a0 con 0x55f723942e70 2024-04-25T01:35:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.013+0000 7fe7d8ff9640 1 -- 172.21.15.28:0/2532396340 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe7e0020060 con 0x55f723942e70 2024-04-25T01:35:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.013+0000 7fe7fa19f740 10 client.16542 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.013+0000 7fe7fa19f740 10 client.16542 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.013+0000 7fe7fa19f740 20 client.16542 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.013+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/2532396340 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f72395fc40 0x55f723962030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.013+0000 7fe7fa19f740 4 client.16542 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:35:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.013+0000 7fe7fa19f740 1 -- 172.21.15.28:0/2532396340 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7238b2bf0 con 0x55f72395fc40 2024-04-25T01:35:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.013+0000 7fe7f522f640 1 --2- 172.21.15.28:0/2532396340 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f72395fc40 0x55f723962030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.013+0000 7fe7d8ff9640 10 client.16542 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.013+0000 7fe7f522f640 1 --2- 172.21.15.28:0/2532396340 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f72395fc40 0x55f723962030 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c05decfbab6e8902 server_cookie=2f2043f9a21d0642 in_seq=0 out_seq=0 2024-04-25T01:35:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.014+0000 7fe7d8ff9640 1 -- 172.21.15.28:0/2532396340 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f7238b2bf0 con 0x55f72395fc40 2024-04-25T01:35:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.014+0000 7fe7d8ff9640 10 client.16542 handle_command_reply: tid=0 2024-04-25T01:35:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.014+0000 7fe7fa19f740 1 client.16542 shutdown 2024-04-25T01:35:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.014+0000 7fe7fa19f740 1 -- 172.21.15.28:0/2532396340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f723942e70 msgr2=0x55f72394d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.014+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/2532396340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f723942e70 0x55f72394d0d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fe7e00058c0 tx=0x7fe7e009ac00 comp rx=0 tx=0).stop 2024-04-25T01:35:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.014+0000 7fe7fa19f740 1 -- 172.21.15.28:0/2532396340 shutdown_connections 2024-04-25T01:35:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.014+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/2532396340 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f72395fc40 0x55f723962030 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.014+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/2532396340 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f72394fc20 0x55f72394d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.014+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/2532396340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f723942e70 0x55f72394d0d0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.014+0000 7fe7d8ff9640 0 client.16542 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.014+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/2532396340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f723942550 0x55f72394cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.014+0000 7fe7fa19f740 1 -- 172.21.15.28:0/2532396340 >> 172.21.15.28:0/2532396340 conn(0x55f72393eb60 msgr2=0x55f723944930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.014+0000 7fe7fa19f740 1 -- 172.21.15.28:0/2532396340 shutdown_connections 2024-04-25T01:35:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.015+0000 7fe7fa19f740 1 -- 172.21.15.28:0/2532396340 wait complete. 2024-04-25T01:35:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.015+0000 7fe7fa19f740 20 client.16542 trim_cache size 0 max 16384 2024-04-25T01:35:53.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.032+0000 7fe7fa19f740 1 Processor -- start 2024-04-25T01:35:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.032+0000 7fe7fa19f740 1 -- start start 2024-04-25T01:35:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.032+0000 7fe7fa19f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f723942550 0x55f72394a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.032+0000 7fe7fa19f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f723942e70 0x55f72393bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.032+0000 7fe7fa19f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72394fc20 0x55f723965050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.032+0000 7fe7fa19f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d400fdb0 con 0x55f72394fc20 2024-04-25T01:35:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.032+0000 7fe7fa19f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d40118f0 con 0x55f723942550 2024-04-25T01:35:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.032+0000 7fe7fa19f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d4010480 con 0x55f723942e70 2024-04-25T01:35:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.032+0000 7fe7f5a30640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72394fc20 0x55f723965050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.032+0000 7fe7f5a30640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72394fc20 0x55f723965050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47284/0 (socket says 172.21.15.28:47284) 2024-04-25T01:35:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.032+0000 7fe7f5a30640 1 -- 172.21.15.28:0/2934279436 learned_addr learned my addr 172.21.15.28:0/2934279436 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.032+0000 7fe7f522f640 1 --2- 172.21.15.28:0/2934279436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f723942550 0x55f72394a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:53.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.032+0000 7fe7f4a2e640 1 --2- 172.21.15.28:0/2934279436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f723942e70 0x55f72393bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:53.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.033+0000 7fe7f5a30640 1 -- 172.21.15.28:0/2934279436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f723942e70 msgr2=0x55f72393bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:53.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.033+0000 7fe7f5a30640 1 --2- 172.21.15.28:0/2934279436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f723942e70 0x55f72393bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.033+0000 7fe7f5a30640 1 -- 172.21.15.28:0/2934279436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f723942550 msgr2=0x55f72394a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:53.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.033+0000 7fe7f5a30640 1 --2- 172.21.15.28:0/2934279436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f723942550 0x55f72394a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.033+0000 7fe7f5a30640 1 -- 172.21.15.28:0/2934279436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f72392e090 con 0x55f72394fc20 2024-04-25T01:35:53.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.033+0000 7fe7f5a30640 1 --2- 172.21.15.28:0/2934279436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72394fc20 0x55f723965050 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80314c0 tx=0x7fe7e8041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0535490dc600b18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:53.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.033+0000 7fe7daffd640 1 -- 172.21.15.28:0/2934279436 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8004040 con 0x55f72394fc20 2024-04-25T01:35:53.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.033+0000 7fe7daffd640 1 -- 172.21.15.28:0/2934279436 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7e8036040 con 0x55f72394fc20 2024-04-25T01:35:53.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.033+0000 7fe7daffd640 1 -- 172.21.15.28:0/2934279436 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8030d30 con 0x55f72394fc20 2024-04-25T01:35:53.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.033+0000 7fe7fa19f740 1 -- 172.21.15.28:0/2934279436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72394fc20 msgr2=0x55f723965050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:53.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.033+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/2934279436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72394fc20 0x55f723965050 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80314c0 tx=0x7fe7e8041040 comp rx=0 tx=0).stop 2024-04-25T01:35:53.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.034+0000 7fe7fa19f740 1 -- 172.21.15.28:0/2934279436 shutdown_connections 2024-04-25T01:35:53.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.034+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/2934279436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72394fc20 0x55f723965050 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.034+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/2934279436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f723942e70 0x55f72393bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.034+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/2934279436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f723942550 0x55f72394a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.034+0000 7fe7fa19f740 1 -- 172.21.15.28:0/2934279436 >> 172.21.15.28:0/2934279436 conn(0x55f72393eb60 msgr2=0x55f72393ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:53.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.034+0000 7fe7fa19f740 1 -- 172.21.15.28:0/2934279436 shutdown_connections 2024-04-25T01:35:53.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.034+0000 7fe7fa19f740 1 -- 172.21.15.28:0/2934279436 wait complete. 2024-04-25T01:35:53.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.034+0000 7fe7fa19f740 1 -- start start 2024-04-25T01:35:53.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.034+0000 7fe7fa19f740 1 Processor -- start 2024-04-25T01:35:53.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.034+0000 7fe7fa19f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f723942550 0x55f723962a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:53.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.034+0000 7fe7fa19f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f723942e70 0x55f723962f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:53.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.034+0000 7fe7fa19f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f72394fc20 0x55f7239634b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:53.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.034+0000 7fe7fa19f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d4011220 con 0x55f723942550 2024-04-25T01:35:53.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.034+0000 7fe7fa19f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d4010b50 con 0x55f72394fc20 2024-04-25T01:35:53.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.034+0000 7fe7fa19f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d400dd90 con 0x55f723942e70 2024-04-25T01:35:53.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.034+0000 7fe7fa19f740 10 client.0 _ll_register_callbacks cb 0x7fe7e8009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:35:53.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.034+0000 7fe7f522f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f723942550 0x55f723962a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:53.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.034+0000 7fe7f522f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f723942550 0x55f723962a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47298/0 (socket says 172.21.15.28:47298) 2024-04-25T01:35:53.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.034+0000 7fe7f522f640 1 -- 172.21.15.28:0/676372627 learned_addr learned my addr 172.21.15.28:0/676372627 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:35:53.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.035+0000 7fe7f4a2e640 1 --2- 172.21.15.28:0/676372627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f723942e70 0x55f723962f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:53.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.035+0000 7fe7f5a30640 1 --2- 172.21.15.28:0/676372627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f72394fc20 0x55f7239634b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:53.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.035+0000 7fe7f522f640 1 -- 172.21.15.28:0/676372627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f723942e70 msgr2=0x55f723962f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:53.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.035+0000 7fe7f522f640 1 --2- 172.21.15.28:0/676372627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f723942e70 0x55f723962f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.035+0000 7fe7f522f640 1 -- 172.21.15.28:0/676372627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f72394fc20 msgr2=0x55f7239634b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:35:53.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.035+0000 7fe7f522f640 1 --2- 172.21.15.28:0/676372627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f72394fc20 0x55f7239634b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.035+0000 7fe7f522f640 1 -- 172.21.15.28:0/676372627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7239c4e20 con 0x55f723942550 2024-04-25T01:35:53.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.035+0000 7fe7f522f640 1 --2- 172.21.15.28:0/676372627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f723942550 0x55f723962a30 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x55f7238a5740 tx=0x7fe7ec013960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ccdee7bf4922c1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:35:53.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.035+0000 7fe7d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:35:53.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.035+0000 7fe7d8ff9640 1 -- 172.21.15.28:0/676372627 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec004220 con 0x55f723942550 2024-04-25T01:35:53.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.035+0000 7fe7d8ff9640 1 -- 172.21.15.28:0/676372627 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fe7ec0043c0 con 0x55f723942550 2024-04-25T01:35:53.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.035+0000 7fe7fa19f740 1 -- 172.21.15.28:0/676372627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7e8036040 con 0x55f723942550 2024-04-25T01:35:53.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.035+0000 7fe7d8ff9640 1 -- 172.21.15.28:0/676372627 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe7ec01d380 con 0x55f723942550 2024-04-25T01:35:53.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.035+0000 7fe7fa19f740 10 client.16548 fetch_fsmap learned FSMap version 28 2024-04-25T01:35:53.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.035+0000 7fe7fa19f740 1 -- 172.21.15.28:0/676372627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f7239c6070 con 0x55f723942550 2024-04-25T01:35:53.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.035+0000 7fe7d8ff9640 1 -- 172.21.15.28:0/676372627 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7e8036040 con 0x55f723942550 2024-04-25T01:35:53.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.036+0000 7fe7d8ff9640 1 -- 172.21.15.28:0/676372627 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe7ec011440 con 0x55f723942550 2024-04-25T01:35:53.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.036+0000 7fe7fa19f740 10 client.16548 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:35:53.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.036+0000 7fe7fa19f740 10 client.16548 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:35:53.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.036+0000 7fe7fa19f740 20 client.16548 populate_metadata read hostname 'smithi028' 2024-04-25T01:35:53.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.036+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/676372627 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f7239c67a0 0x55f7239c8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:35:53.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.036+0000 7fe7fa19f740 4 client.16548 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:35:53.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.036+0000 7fe7fa19f740 1 -- 172.21.15.28:0/676372627 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f7238acf40 con 0x55f7239c67a0 2024-04-25T01:35:53.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.036+0000 7fe7f4a2e640 1 --2- 172.21.15.28:0/676372627 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f7239c67a0 0x55f7239c8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:35:53.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.036+0000 7fe7d8ff9640 10 client.16548 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:53.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.036+0000 7fe7f4a2e640 1 --2- 172.21.15.28:0/676372627 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f7239c67a0 0x55f7239c8bc0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ffed8efe8ae4a51 server_cookie=cbb340109b9c57b4 in_seq=0 out_seq=0 2024-04-25T01:35:53.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.036+0000 7fe7d8ff9640 1 -- 172.21.15.28:0/676372627 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f7238acf40 con 0x55f7239c67a0 2024-04-25T01:35:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.036+0000 7fe7d8ff9640 10 client.16548 handle_command_reply: tid=0 2024-04-25T01:35:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.036+0000 7fe7fa19f740 1 client.16548 shutdown 2024-04-25T01:35:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7fa19f740 1 -- 172.21.15.28:0/676372627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f723942550 msgr2=0x55f723962a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/676372627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f723942550 0x55f723962a30 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x55f7238a5740 tx=0x7fe7ec013960 comp rx=0 tx=0).stop 2024-04-25T01:35:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7fa19f740 1 -- 172.21.15.28:0/676372627 shutdown_connections 2024-04-25T01:35:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/676372627 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f7239c67a0 0x55f7239c8bc0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/676372627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f72394fc20 0x55f7239634b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7d8ff9640 0 client.16548 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:35:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/676372627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f723942e70 0x55f723962f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7fa19f740 1 --2- 172.21.15.28:0/676372627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f723942550 0x55f723962a30 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7fa19f740 1 -- 172.21.15.28:0/676372627 >> 172.21.15.28:0/676372627 conn(0x55f72393eb60 msgr2=0x55f72394b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7fa19f740 1 -- 172.21.15.28:0/676372627 shutdown_connections 2024-04-25T01:35:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7fa19f740 1 -- 172.21.15.28:0/676372627 wait complete. 2024-04-25T01:35:53.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7fa19f740 20 client.16548 trim_cache size 0 max 16384 2024-04-25T01:35:53.021 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:35:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:35:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-25T01:35:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: "9a5379a3-b30d-462c-93d4-83c322c91233": { 2024-04-25T01:35:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-25T01:35:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "9a5379a3-b30d-462c-93d4-83c322c91233", 2024-04-25T01:35:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-25T01:35:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:35:53.021 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:35:53.021 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:35:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7f7493640 1 -- 172.21.15.28:0/2800094342 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe7dc0a2d00 msgr2=0x7fe7dc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7f7493640 1 --2- 172.21.15.28:0/2800094342 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe7dc0a2d00 0x7fe7dc0a51c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8009a60 tx=0x7fe7e80095c0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7f7493640 1 -- 172.21.15.28:0/2800094342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f0100fd0 msgr2=0x7fe7f0197390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:35:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7f7493640 1 --2- 172.21.15.28:0/2800094342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f0100fd0 0x7fe7f0197390 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fe7e000c890 tx=0x7fe7e000cd50 comp rx=0 tx=0).stop 2024-04-25T01:35:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7f7493640 1 -- 172.21.15.28:0/2800094342 shutdown_connections 2024-04-25T01:35:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7f7493640 1 --2- 172.21.15.28:0/2800094342 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe7dc0a2d00 0x7fe7dc0a51c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7f7493640 1 --2- 172.21.15.28:0/2800094342 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe7f01066b0 0x7fe7f019b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7f7493640 1 --2- 172.21.15.28:0/2800094342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7f0100fd0 0x7fe7f0197390 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7f7493640 1 --2- 172.21.15.28:0/2800094342 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe7f0100610 0x7fe7f0196e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:35:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7f7493640 1 -- 172.21.15.28:0/2800094342 >> 172.21.15.28:0/2800094342 conn(0x7fe7f00769c0 msgr2=0x7fe7f00fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:35:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7f7493640 1 -- 172.21.15.28:0/2800094342 shutdown_connections 2024-04-25T01:35:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:35:53.037+0000 7fe7f7493640 1 -- 172.21.15.28:0/2800094342 wait complete. 2024-04-25T01:35:53.025 INFO:tasks.cephfs.filesystem:scrub status for tag:9a5379a3-b30d-462c-93d4-83c322c91233 - {'path': '/', 'tag': '9a5379a3-b30d-462c-93d4-83c322c91233', 'options': 'recursive,force'} 2024-04-25T01:35:54.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:54 smithi130 ceph-mon[32702]: pgmap v89: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 3.1 KiB/s wr, 227 op/s 2024-04-25T01:35:54.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:54 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:35:54.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:54 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:54.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:54 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:35:54.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:54 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:54.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:54 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:35:54.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:54 smithi032 ceph-mon[32461]: pgmap v89: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 3.1 KiB/s wr, 227 op/s 2024-04-25T01:35:54.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:54 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:35:54.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:54 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:54.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:54 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:35:54.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:54 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:54.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:54 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:35:54.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:54 smithi028 ceph-mon[29390]: pgmap v89: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 3.1 KiB/s wr, 227 op/s 2024-04-25T01:35:54.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:54 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:35:54.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:54 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:54.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:54 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:35:54.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:54 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:35:54.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:54 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:35:56.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:56 smithi130 ceph-mon[32702]: pgmap v90: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-25T01:35:56.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:56 smithi032 ceph-mon[32461]: pgmap v90: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-25T01:35:56.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:56 smithi028 ceph-mon[29390]: pgmap v90: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-25T01:35:58.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:35:58 smithi130 ceph-mon[32702]: pgmap v91: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1023 B/s wr, 12 op/s 2024-04-25T01:35:58.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:35:58 smithi032 ceph-mon[32461]: pgmap v91: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1023 B/s wr, 12 op/s 2024-04-25T01:35:58.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:35:58 smithi028 ceph-mon[29390]: pgmap v91: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1023 B/s wr, 12 op/s 2024-04-25T01:36:00.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:00 smithi130 ceph-mon[32702]: pgmap v92: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-25T01:36:00.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:00 smithi032 ceph-mon[32461]: pgmap v92: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-25T01:36:00.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:00 smithi028 ceph-mon[29390]: pgmap v92: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-25T01:36:02.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:02 smithi130 ceph-mon[32702]: pgmap v93: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:36:02.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:02 smithi032 ceph-mon[32461]: pgmap v93: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:36:02.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:02 smithi028 ceph-mon[29390]: pgmap v93: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:36:04.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:04 smithi130 ceph-mon[32702]: pgmap v94: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:36:04.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:04 smithi032 ceph-mon[32461]: pgmap v94: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:36:04.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:04 smithi028 ceph-mon[29390]: pgmap v94: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:36:06.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:06 smithi130 ceph-mon[32702]: pgmap v95: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:36:06.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:06 smithi032 ceph-mon[32461]: pgmap v95: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:36:06.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:06 smithi028 ceph-mon[29390]: pgmap v95: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:36:08.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:08 smithi130 ceph-mon[32702]: pgmap v96: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-04-25T01:36:08.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:08 smithi032 ceph-mon[32461]: pgmap v96: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-04-25T01:36:08.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:08 smithi028 ceph-mon[29390]: pgmap v96: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-04-25T01:36:09.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:09 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:36:09.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:09 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:36:09.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:09 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:36:09.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:09 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:36:09.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:09 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:36:09.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:09 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:36:09.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:09 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:36:09.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:09 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:36:09.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:09 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:36:09.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:09 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:36:09.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:09 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:36:09.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:09 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:36:10.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:10 smithi130 ceph-mon[32702]: pgmap v97: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-04-25T01:36:10.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:10 smithi032 ceph-mon[32461]: pgmap v97: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-04-25T01:36:10.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:10 smithi028 ceph-mon[29390]: pgmap v97: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-04-25T01:36:12.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:12 smithi130 ceph-mon[32702]: pgmap v98: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:36:12.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:12 smithi032 ceph-mon[32461]: pgmap v98: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:36:12.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:12 smithi028 ceph-mon[29390]: pgmap v98: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:36:14.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:14 smithi130 ceph-mon[32702]: pgmap v99: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:36:14.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:14 smithi032 ceph-mon[32461]: pgmap v99: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:36:14.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:14 smithi028 ceph-mon[29390]: pgmap v99: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:36:16.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:16 smithi130 ceph-mon[32702]: pgmap v100: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-25T01:36:16.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:16 smithi032 ceph-mon[32461]: pgmap v100: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-25T01:36:16.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:16 smithi028 ceph-mon[29390]: pgmap v100: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-25T01:36:18.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:18 smithi130 ceph-mon[32702]: pgmap v101: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 11 op/s 2024-04-25T01:36:18.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:18 smithi032 ceph-mon[32461]: pgmap v101: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 11 op/s 2024-04-25T01:36:18.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:18 smithi028 ceph-mon[29390]: pgmap v101: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 11 op/s 2024-04-25T01:36:20.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:20 smithi130 ceph-mon[32702]: pgmap v102: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T01:36:20.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:20 smithi032 ceph-mon[32461]: pgmap v102: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T01:36:20.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:20 smithi028 ceph-mon[29390]: pgmap v102: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T01:36:22.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:22 smithi130 ceph-mon[32702]: pgmap v103: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T01:36:22.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:22 smithi032 ceph-mon[32461]: pgmap v103: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T01:36:22.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:22 smithi028 ceph-mon[29390]: pgmap v103: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T01:36:23.026 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:36:23.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:36:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.169+0000 7f9661930640 1 -- 172.21.15.28:0/1704053882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f965c100fd0 msgr2=0x7f965c10d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.169+0000 7f9661930640 1 --2- 172.21.15.28:0/1704053882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f965c100fd0 0x7f965c10d4e0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f9654009cd0 tx=0x7f965402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.169+0000 7f9661930640 1 -- 172.21.15.28:0/1704053882 shutdown_connections 2024-04-25T01:36:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.169+0000 7f9661930640 1 --2- 172.21.15.28:0/1704053882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f965c100fd0 0x7f965c10d4e0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.169+0000 7f9661930640 1 --2- 172.21.15.28:0/1704053882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965c100610 0x7f965c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.169+0000 7f9661930640 1 --2- 172.21.15.28:0/1704053882 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f965c1066b0 0x7f965c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.169+0000 7f9661930640 1 -- 172.21.15.28:0/1704053882 >> 172.21.15.28:0/1704053882 conn(0x7f965c077740 msgr2=0x7f965c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.169+0000 7f9661930640 1 -- 172.21.15.28:0/1704053882 shutdown_connections 2024-04-25T01:36:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.169+0000 7f9661930640 1 -- 172.21.15.28:0/1704053882 wait complete. 2024-04-25T01:36:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.169+0000 7f9661930640 1 Processor -- start 2024-04-25T01:36:23.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.170+0000 7f9661930640 1 -- start start 2024-04-25T01:36:23.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.170+0000 7f9661930640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965c100610 0x7f965c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.170+0000 7f9661930640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f965c100fd0 0x7f965c1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.170+0000 7f9661930640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f965c1066b0 0x7f965c19c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.170+0000 7f9661930640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f965c110a40 con 0x7f965c1066b0 2024-04-25T01:36:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.170+0000 7f9661930640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1108c0 con 0x7f965c100610 2024-04-25T01:36:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.170+0000 7f9661930640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f965c110bc0 con 0x7f965c100fd0 2024-04-25T01:36:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.170+0000 7f965b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f965c1066b0 0x7f965c19c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.170+0000 7f965b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f965c1066b0 0x7f965c19c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48144/0 (socket says 172.21.15.28:48144) 2024-04-25T01:36:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.170+0000 7f965b7fe640 1 -- 172.21.15.28:0/4274037728 learned_addr learned my addr 172.21.15.28:0/4274037728 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.170+0000 7f965affd640 1 --2- 172.21.15.28:0/4274037728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965c100610 0x7f965c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.170+0000 7f965a7fc640 1 --2- 172.21.15.28:0/4274037728 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f965c100fd0 0x7f965c1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.170+0000 7f965b7fe640 1 -- 172.21.15.28:0/4274037728 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f965c100fd0 msgr2=0x7f965c1984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.170+0000 7f965b7fe640 1 --2- 172.21.15.28:0/4274037728 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f965c100fd0 0x7f965c1984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.170+0000 7f965b7fe640 1 -- 172.21.15.28:0/4274037728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965c100610 msgr2=0x7f965c197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.170+0000 7f965b7fe640 1 --2- 172.21.15.28:0/4274037728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965c100610 0x7f965c197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.170+0000 7f965b7fe640 1 -- 172.21.15.28:0/4274037728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965c19cf20 con 0x7f965c1066b0 2024-04-25T01:36:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.171+0000 7f965b7fe640 1 --2- 172.21.15.28:0/4274037728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f965c1066b0 0x7f965c19c7f0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9654033040 tx=0x7f9654003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.171+0000 7f966092e640 1 -- 172.21.15.28:0/4274037728 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9654004040 con 0x7f965c1066b0 2024-04-25T01:36:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.171+0000 7f966092e640 1 -- 172.21.15.28:0/4274037728 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96540046c0 con 0x7f965c1066b0 2024-04-25T01:36:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.171+0000 7f9661930640 1 -- 172.21.15.28:0/4274037728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f965c19d1b0 con 0x7f965c1066b0 2024-04-25T01:36:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.171+0000 7f966092e640 1 -- 172.21.15.28:0/4274037728 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9654046480 con 0x7f965c1066b0 2024-04-25T01:36:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.171+0000 7f9661930640 1 -- 172.21.15.28:0/4274037728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f965c1a4aa0 con 0x7f965c1066b0 2024-04-25T01:36:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.173+0000 7f966092e640 1 -- 172.21.15.28:0/4274037728 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f96540305a0 con 0x7f965c1066b0 2024-04-25T01:36:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.173+0000 7f966463e740 1 Processor -- start 2024-04-25T01:36:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.173+0000 7f966463e740 1 -- start start 2024-04-25T01:36:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f966092e640 1 --2- 172.21.15.28:0/4274037728 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f96440a2d00 0x7f96440a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f966463e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988630 0x55c1d0988a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f966463e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988f50 0x55c1d09930e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f966463e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0993620 0x55c1d0995a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f966463e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c1d08ed6b0 con 0x55c1d0988630 2024-04-25T01:36:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f966463e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c1d0895880 con 0x55c1d0993620 2024-04-25T01:36:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f966463e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c1d08e9670 con 0x55c1d0988f50 2024-04-25T01:36:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f965affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988630 0x55c1d0988a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f965affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988630 0x55c1d0988a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48150/0 (socket says 172.21.15.28:48150) 2024-04-25T01:36:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f965affd640 1 -- 172.21.15.28:0/4078680611 learned_addr learned my addr 172.21.15.28:0/4078680611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f965a7fc640 1 --2- 172.21.15.28:0/4274037728 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f96440a2d00 0x7f96440a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f966092e640 1 -- 172.21.15.28:0/4274037728 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f965402e030 con 0x7f965c1066b0 2024-04-25T01:36:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f965a7fc640 1 --2- 172.21.15.28:0/4078680611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988f50 0x55c1d09930e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f965b7fe640 1 --2- 172.21.15.28:0/4078680611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0993620 0x55c1d0995a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f965affd640 1 -- 172.21.15.28:0/4078680611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988f50 msgr2=0x55c1d09930e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f965affd640 1 --2- 172.21.15.28:0/4078680611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988f50 0x55c1d09930e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f965affd640 1 -- 172.21.15.28:0/4078680611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0993620 msgr2=0x55c1d0995a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f965affd640 1 --2- 172.21.15.28:0/4078680611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0993620 0x55c1d0995a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f965affd640 1 -- 172.21.15.28:0/4078680611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1d06cb7d0 con 0x55c1d0988630 2024-04-25T01:36:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f965affd640 1 --2- 172.21.15.28:0/4078680611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988630 0x55c1d0988a10 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f964c004590 tx=0x7f964c00bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91e21a65dee034ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f965a7fc640 1 --2- 172.21.15.28:0/4274037728 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f96440a2d00 0x7f96440a51c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f965c105d10 tx=0x7f96500128d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.174+0000 7f9641ffb640 1 -- 172.21.15.28:0/4078680611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c003bd0 con 0x55c1d0988630 2024-04-25T01:36:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.175+0000 7f9641ffb640 1 -- 172.21.15.28:0/4078680611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f964c015070 con 0x55c1d0988630 2024-04-25T01:36:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.175+0000 7f9641ffb640 1 -- 172.21.15.28:0/4078680611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c0116a0 con 0x55c1d0988630 2024-04-25T01:36:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.175+0000 7f966463e740 1 -- 172.21.15.28:0/4078680611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988630 msgr2=0x55c1d0988a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.175+0000 7f966463e740 1 --2- 172.21.15.28:0/4078680611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988630 0x55c1d0988a10 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f964c004590 tx=0x7f964c00bb20 comp rx=0 tx=0).stop 2024-04-25T01:36:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.175+0000 7f966463e740 1 -- 172.21.15.28:0/4078680611 shutdown_connections 2024-04-25T01:36:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.175+0000 7f966463e740 1 --2- 172.21.15.28:0/4078680611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0993620 0x55c1d0995a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.175+0000 7f966463e740 1 --2- 172.21.15.28:0/4078680611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988f50 0x55c1d09930e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.175+0000 7f966463e740 1 --2- 172.21.15.28:0/4078680611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988630 0x55c1d0988a10 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.175+0000 7f966463e740 1 -- 172.21.15.28:0/4078680611 >> 172.21.15.28:0/4078680611 conn(0x55c1d0984950 msgr2=0x55c1d0984d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.175+0000 7f966463e740 1 -- 172.21.15.28:0/4078680611 shutdown_connections 2024-04-25T01:36:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.175+0000 7f966463e740 1 -- 172.21.15.28:0/4078680611 wait complete. 2024-04-25T01:36:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.176+0000 7f966463e740 1 -- start start 2024-04-25T01:36:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.176+0000 7f966463e740 1 Processor -- start 2024-04-25T01:36:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.176+0000 7f966463e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988630 0x55c1d09905c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.176+0000 7f966463e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0988f50 0x55c1d0990b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.176+0000 7f966463e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0993620 0x55c1d0991040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.176+0000 7f966463e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c1d0996410 con 0x55c1d0993620 2024-04-25T01:36:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.176+0000 7f966463e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c1d08a8f50 con 0x55c1d0988f50 2024-04-25T01:36:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.176+0000 7f966463e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c1d0996590 con 0x55c1d0988630 2024-04-25T01:36:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.176+0000 7f966463e740 10 client.0 _ll_register_callbacks cb 0x55c1d08e4000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.176+0000 7f965b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0993620 0x55c1d0991040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.176+0000 7f965b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0993620 0x55c1d0991040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48162/0 (socket says 172.21.15.28:48162) 2024-04-25T01:36:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.176+0000 7f965b7fe640 1 -- 172.21.15.28:0/3589858336 learned_addr learned my addr 172.21.15.28:0/3589858336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.176+0000 7f965a7fc640 1 --2- 172.21.15.28:0/3589858336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0988f50 0x55c1d0990b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.176+0000 7f965affd640 1 --2- 172.21.15.28:0/3589858336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988630 0x55c1d09905c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.177+0000 7f965b7fe640 1 -- 172.21.15.28:0/3589858336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988630 msgr2=0x55c1d09905c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.177+0000 7f965b7fe640 1 --2- 172.21.15.28:0/3589858336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988630 0x55c1d09905c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.177+0000 7f965b7fe640 1 -- 172.21.15.28:0/3589858336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0988f50 msgr2=0x55c1d0990b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.177+0000 7f965b7fe640 1 --2- 172.21.15.28:0/3589858336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0988f50 0x55c1d0990b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.177+0000 7f965b7fe640 1 -- 172.21.15.28:0/3589858336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1d06013f0 con 0x55c1d0993620 2024-04-25T01:36:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.177+0000 7f965b7fe640 1 --2- 172.21.15.28:0/3589858336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0993620 0x55c1d0991040 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f9654009ca0 tx=0x7f96540b9420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7f98d74619ccf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.177+0000 7f963affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.177+0000 7f963affd640 1 -- 172.21.15.28:0/3589858336 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96540bd770 con 0x55c1d0993620 2024-04-25T01:36:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.177+0000 7f963affd640 1 -- 172.21.15.28:0/3589858336 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f96540b96e0 con 0x55c1d0993620 2024-04-25T01:36:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.177+0000 7f966463e740 1 -- 172.21.15.28:0/3589858336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f964c015070 con 0x55c1d0993620 2024-04-25T01:36:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.177+0000 7f963affd640 1 -- 172.21.15.28:0/3589858336 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f96540bd070 con 0x55c1d0993620 2024-04-25T01:36:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.177+0000 7f966463e740 10 client.16560 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.177+0000 7f966463e740 1 -- 172.21.15.28:0/3589858336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c1d05c10f0 con 0x55c1d0993620 2024-04-25T01:36:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.177+0000 7f963affd640 1 -- 172.21.15.28:0/3589858336 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f964c015070 con 0x55c1d0993620 2024-04-25T01:36:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.177+0000 7f963affd640 1 -- 172.21.15.28:0/3589858336 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f965404e070 con 0x55c1d0993620 2024-04-25T01:36:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.177+0000 7f966463e740 10 client.16560 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.177+0000 7f966463e740 10 client.16560 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.177+0000 7f966463e740 20 client.16560 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.178+0000 7f966463e740 1 --2- 172.21.15.28:0/3589858336 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c1d09a3920 0x55c1d09a5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.178+0000 7f966463e740 4 client.16560 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:23.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.178+0000 7f966463e740 1 -- 172.21.15.28:0/3589858336 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1d08fabf0 con 0x55c1d09a3920 2024-04-25T01:36:23.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.178+0000 7f965affd640 1 --2- 172.21.15.28:0/3589858336 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c1d09a3920 0x55c1d09a5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.178+0000 7f965affd640 1 --2- 172.21.15.28:0/3589858336 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c1d09a3920 0x55c1d09a5dc0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=720caf7af57867c9 server_cookie=9349019deef0b44b in_seq=0 out_seq=0 2024-04-25T01:36:23.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.178+0000 7f963affd640 10 client.16560 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:23.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.179+0000 7f963affd640 1 -- 172.21.15.28:0/3589858336 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c1d08fabf0 con 0x55c1d09a3920 2024-04-25T01:36:23.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.179+0000 7f963affd640 10 client.16560 handle_command_reply: tid=0 2024-04-25T01:36:23.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.179+0000 7f966463e740 1 client.16560 shutdown 2024-04-25T01:36:23.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.179+0000 7f966463e740 1 -- 172.21.15.28:0/3589858336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0993620 msgr2=0x55c1d0991040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.179+0000 7f966463e740 1 --2- 172.21.15.28:0/3589858336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0993620 0x55c1d0991040 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f9654009ca0 tx=0x7f96540b9420 comp rx=0 tx=0).stop 2024-04-25T01:36:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.179+0000 7f966463e740 1 -- 172.21.15.28:0/3589858336 shutdown_connections 2024-04-25T01:36:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.179+0000 7f966463e740 1 --2- 172.21.15.28:0/3589858336 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c1d09a3920 0x55c1d09a5dc0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.179+0000 7f966463e740 1 --2- 172.21.15.28:0/3589858336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0993620 0x55c1d0991040 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.179+0000 7f963affd640 0 client.16560 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.179+0000 7f966463e740 1 --2- 172.21.15.28:0/3589858336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0988f50 0x55c1d0990b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.179+0000 7f966463e740 1 --2- 172.21.15.28:0/3589858336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988630 0x55c1d09905c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.179+0000 7f966463e740 1 -- 172.21.15.28:0/3589858336 >> 172.21.15.28:0/3589858336 conn(0x55c1d0984950 msgr2=0x55c1d098a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.180+0000 7f966463e740 1 -- 172.21.15.28:0/3589858336 shutdown_connections 2024-04-25T01:36:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.180+0000 7f966463e740 1 -- 172.21.15.28:0/3589858336 wait complete. 2024-04-25T01:36:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.180+0000 7f966463e740 20 client.16560 trim_cache size 0 max 16384 2024-04-25T01:36:23.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.197+0000 7f966463e740 1 Processor -- start 2024-04-25T01:36:23.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.197+0000 7f966463e740 1 -- start start 2024-04-25T01:36:23.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f966463e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988630 0x55c1d099f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f966463e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988f50 0x55c1d098ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f966463e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0993620 0x55c1d0983970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f966463e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1ae9a0 con 0x55c1d0988630 2024-04-25T01:36:23.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f966463e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1b04e0 con 0x55c1d0993620 2024-04-25T01:36:23.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f966463e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1af070 con 0x55c1d0988f50 2024-04-25T01:36:23.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f965affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988630 0x55c1d099f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f965affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988630 0x55c1d099f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48174/0 (socket says 172.21.15.28:48174) 2024-04-25T01:36:23.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f965affd640 1 -- 172.21.15.28:0/1208954424 learned_addr learned my addr 172.21.15.28:0/1208954424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f965b7fe640 1 --2- 172.21.15.28:0/1208954424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0993620 0x55c1d0983970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f965affd640 1 -- 172.21.15.28:0/1208954424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988f50 msgr2=0x55c1d098ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f965affd640 1 --2- 172.21.15.28:0/1208954424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988f50 0x55c1d098ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f965affd640 1 -- 172.21.15.28:0/1208954424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0993620 msgr2=0x55c1d0983970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f965affd640 1 --2- 172.21.15.28:0/1208954424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0993620 0x55c1d0983970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f965affd640 1 -- 172.21.15.28:0/1208954424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1d0984780 con 0x55c1d0988630 2024-04-25T01:36:23.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f965b7fe640 1 --2- 172.21.15.28:0/1208954424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0993620 0x55c1d0983970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:23.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f965affd640 1 --2- 172.21.15.28:0/1208954424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988630 0x55c1d099f2d0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f964c004590 tx=0x7f964c004500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9743daf2f2aa09f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f96417fa640 1 -- 172.21.15.28:0/1208954424 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c003dd0 con 0x55c1d0988630 2024-04-25T01:36:23.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f96417fa640 1 -- 172.21.15.28:0/1208954424 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f964c015070 con 0x55c1d0988630 2024-04-25T01:36:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.198+0000 7f96417fa640 1 -- 172.21.15.28:0/1208954424 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c02a170 con 0x55c1d0988630 2024-04-25T01:36:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.199+0000 7f966463e740 1 -- 172.21.15.28:0/1208954424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988630 msgr2=0x55c1d099f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.199+0000 7f966463e740 1 --2- 172.21.15.28:0/1208954424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988630 0x55c1d099f2d0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f964c004590 tx=0x7f964c004500 comp rx=0 tx=0).stop 2024-04-25T01:36:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.199+0000 7f966463e740 1 -- 172.21.15.28:0/1208954424 shutdown_connections 2024-04-25T01:36:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.199+0000 7f966463e740 1 --2- 172.21.15.28:0/1208954424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0993620 0x55c1d0983970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.199+0000 7f966463e740 1 --2- 172.21.15.28:0/1208954424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988f50 0x55c1d098ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.199+0000 7f966463e740 1 --2- 172.21.15.28:0/1208954424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988630 0x55c1d099f2d0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.199+0000 7f966463e740 1 -- 172.21.15.28:0/1208954424 >> 172.21.15.28:0/1208954424 conn(0x55c1d0984950 msgr2=0x55c1d0983260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.199+0000 7f966463e740 1 -- 172.21.15.28:0/1208954424 shutdown_connections 2024-04-25T01:36:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.199+0000 7f966463e740 1 -- 172.21.15.28:0/1208954424 wait complete. 2024-04-25T01:36:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.199+0000 7f966463e740 1 -- start start 2024-04-25T01:36:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f966463e740 1 Processor -- start 2024-04-25T01:36:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f966463e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988630 0x55c1d09972a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f966463e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988f50 0x55c1d09977e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f966463e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0993620 0x55c1d0997d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f966463e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1afe10 con 0x55c1d0988f50 2024-04-25T01:36:23.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f966463e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1af740 con 0x55c1d0993620 2024-04-25T01:36:23.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f966463e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f965c1ac980 con 0x55c1d0988630 2024-04-25T01:36:23.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f966463e740 10 client.0 _ll_register_callbacks cb 0x7f9654030fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:23.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f965a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988f50 0x55c1d09977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f965a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988f50 0x55c1d09977e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48180/0 (socket says 172.21.15.28:48180) 2024-04-25T01:36:23.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f965a7fc640 1 -- 172.21.15.28:0/3289942552 learned_addr learned my addr 172.21.15.28:0/3289942552 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f965affd640 1 --2- 172.21.15.28:0/3289942552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988630 0x55c1d09972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f965b7fe640 1 --2- 172.21.15.28:0/3289942552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0993620 0x55c1d0997d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f965a7fc640 1 -- 172.21.15.28:0/3289942552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988630 msgr2=0x55c1d09972a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f965a7fc640 1 --2- 172.21.15.28:0/3289942552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988630 0x55c1d09972a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f965a7fc640 1 -- 172.21.15.28:0/3289942552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0993620 msgr2=0x55c1d0997d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f965a7fc640 1 --2- 172.21.15.28:0/3289942552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0993620 0x55c1d0997d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f965a7fc640 1 -- 172.21.15.28:0/3289942552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1d0a0d360 con 0x55c1d0988f50 2024-04-25T01:36:23.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f965affd640 1 --2- 172.21.15.28:0/3289942552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988630 0x55c1d09972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:23.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f965b7fe640 1 --2- 172.21.15.28:0/3289942552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0993620 0x55c1d0997d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:23.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f965a7fc640 1 --2- 172.21.15.28:0/3289942552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988f50 0x55c1d09977e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f9650005fd0 tx=0x7f96500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8d36842bb798935 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.200+0000 7f963affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:23.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.201+0000 7f963affd640 1 -- 172.21.15.28:0/3289942552 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650020a80 con 0x55c1d0988f50 2024-04-25T01:36:23.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.201+0000 7f963affd640 1 -- 172.21.15.28:0/3289942552 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f9650020c20 con 0x55c1d0988f50 2024-04-25T01:36:23.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.201+0000 7f966463e740 1 -- 172.21.15.28:0/3289942552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f964c015070 con 0x55c1d0988f50 2024-04-25T01:36:23.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.201+0000 7f963affd640 1 -- 172.21.15.28:0/3289942552 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9650020e60 con 0x55c1d0988f50 2024-04-25T01:36:23.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.201+0000 7f966463e740 10 client.16566 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:23.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.201+0000 7f966463e740 1 -- 172.21.15.28:0/3289942552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c1d0a0e610 con 0x55c1d0988f50 2024-04-25T01:36:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.201+0000 7f963affd640 1 -- 172.21.15.28:0/3289942552 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f964c015070 con 0x55c1d0988f50 2024-04-25T01:36:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.201+0000 7f963affd640 1 -- 172.21.15.28:0/3289942552 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9650027070 con 0x55c1d0988f50 2024-04-25T01:36:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.201+0000 7f966463e740 10 client.16566 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.201+0000 7f966463e740 10 client.16566 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.201+0000 7f966463e740 20 client.16566 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.201+0000 7f966463e740 1 --2- 172.21.15.28:0/3289942552 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c1d09a3920 0x55c1d0a10ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.201+0000 7f966463e740 4 client.16566 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:36:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.201+0000 7f966463e740 1 -- 172.21.15.28:0/3289942552 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c1d08f4f40 con 0x55c1d09a3920 2024-04-25T01:36:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.202+0000 7f965affd640 1 --2- 172.21.15.28:0/3289942552 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c1d09a3920 0x55c1d0a10ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.202+0000 7f965affd640 1 --2- 172.21.15.28:0/3289942552 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c1d09a3920 0x55c1d0a10ed0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3823ac717b8eaa31 server_cookie=7f9ec47822703f7c in_seq=0 out_seq=0 2024-04-25T01:36:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.202+0000 7f963affd640 10 client.16566 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.202+0000 7f963affd640 1 -- 172.21.15.28:0/3289942552 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c1d08f4f40 con 0x55c1d09a3920 2024-04-25T01:36:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.202+0000 7f963affd640 10 client.16566 handle_command_reply: tid=0 2024-04-25T01:36:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.202+0000 7f966463e740 1 client.16566 shutdown 2024-04-25T01:36:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.202+0000 7f966463e740 1 -- 172.21.15.28:0/3289942552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988f50 msgr2=0x55c1d09977e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.202+0000 7f966463e740 1 --2- 172.21.15.28:0/3289942552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988f50 0x55c1d09977e0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f9650005fd0 tx=0x7f96500046c0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.202+0000 7f966463e740 1 -- 172.21.15.28:0/3289942552 shutdown_connections 2024-04-25T01:36:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.202+0000 7f966463e740 1 --2- 172.21.15.28:0/3289942552 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c1d09a3920 0x55c1d0a10ed0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.202+0000 7f966463e740 1 --2- 172.21.15.28:0/3289942552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1d0993620 0x55c1d0997d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.202+0000 7f966463e740 1 --2- 172.21.15.28:0/3289942552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1d0988f50 0x55c1d09977e0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.202+0000 7f966463e740 1 --2- 172.21.15.28:0/3289942552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1d0988630 0x55c1d09972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.202+0000 7f963affd640 0 client.16566 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.202+0000 7f966463e740 1 -- 172.21.15.28:0/3289942552 >> 172.21.15.28:0/3289942552 conn(0x55c1d0984950 msgr2=0x55c1d0990df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.202+0000 7f966463e740 1 -- 172.21.15.28:0/3289942552 shutdown_connections 2024-04-25T01:36:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.202+0000 7f966463e740 1 -- 172.21.15.28:0/3289942552 wait complete. 2024-04-25T01:36:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.203+0000 7f966463e740 20 client.16566 trim_cache size 0 max 16384 2024-04-25T01:36:23.185 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:36:23.186 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:36:23.186 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:36:23.186 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:36:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.203+0000 7f9661930640 1 -- 172.21.15.28:0/4274037728 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f96440a2d00 msgr2=0x7f96440a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.203+0000 7f9661930640 1 --2- 172.21.15.28:0/4274037728 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f96440a2d00 0x7f96440a51c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f965c105d10 tx=0x7f96500128d0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.203+0000 7f9661930640 1 -- 172.21.15.28:0/4274037728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f965c1066b0 msgr2=0x7f965c19c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.203+0000 7f9661930640 1 --2- 172.21.15.28:0/4274037728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f965c1066b0 0x7f965c19c7f0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9654033040 tx=0x7f9654003970 comp rx=0 tx=0).stop 2024-04-25T01:36:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.203+0000 7f9661930640 1 -- 172.21.15.28:0/4274037728 shutdown_connections 2024-04-25T01:36:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.203+0000 7f9661930640 1 --2- 172.21.15.28:0/4274037728 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f96440a2d00 0x7f96440a51c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.203+0000 7f9661930640 1 --2- 172.21.15.28:0/4274037728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f965c1066b0 0x7f965c19c7f0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.203+0000 7f9661930640 1 --2- 172.21.15.28:0/4274037728 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f965c100fd0 0x7f965c1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.203+0000 7f9661930640 1 --2- 172.21.15.28:0/4274037728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f965c100610 0x7f965c197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.203+0000 7f9661930640 1 -- 172.21.15.28:0/4274037728 >> 172.21.15.28:0/4274037728 conn(0x7f965c077740 msgr2=0x7f965c0ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.203+0000 7f9661930640 1 -- 172.21.15.28:0/4274037728 shutdown_connections 2024-04-25T01:36:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.203+0000 7f9661930640 1 -- 172.21.15.28:0/4274037728 wait complete. 2024-04-25T01:36:23.190 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:36:23.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:36:23.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.317+0000 7f8580e88640 1 -- 172.21.15.28:0/1675252875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f857c102a20 msgr2=0x7f857c102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.317+0000 7f8580e88640 1 --2- 172.21.15.28:0/1675252875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f857c102a20 0x7f857c102e00 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f856c009900 tx=0x7f856c02a480 comp rx=0 tx=0).stop 2024-04-25T01:36:23.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.317+0000 7f8580e88640 1 -- 172.21.15.28:0/1675252875 shutdown_connections 2024-04-25T01:36:23.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.317+0000 7f8580e88640 1 --2- 172.21.15.28:0/1675252875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f857c10bd60 0x7f857c10e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.317+0000 7f8580e88640 1 --2- 172.21.15.28:0/1675252875 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f857c103340 0x7f857c10b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.317+0000 7f8580e88640 1 --2- 172.21.15.28:0/1675252875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f857c102a20 0x7f857c102e00 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.317+0000 7f8580e88640 1 -- 172.21.15.28:0/1675252875 >> 172.21.15.28:0/1675252875 conn(0x7f857c0777a0 msgr2=0x7f857c077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.317+0000 7f8580e88640 1 -- 172.21.15.28:0/1675252875 shutdown_connections 2024-04-25T01:36:23.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.317+0000 7f8580e88640 1 -- 172.21.15.28:0/1675252875 wait complete. 2024-04-25T01:36:23.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.317+0000 7f8580e88640 1 Processor -- start 2024-04-25T01:36:23.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f8580e88640 1 -- start start 2024-04-25T01:36:23.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f8580e88640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f857c102a20 0x7f857c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f8580e88640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f857c103340 0x7f857c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f8580e88640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f857c10bd60 0x7f857c19c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f8580e88640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f857c111b30 con 0x7f857c103340 2024-04-25T01:36:23.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f8580e88640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f857c1119b0 con 0x7f857c10bd60 2024-04-25T01:36:23.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f8580e88640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f857c111cb0 con 0x7f857c102a20 2024-04-25T01:36:23.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f8579d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f857c103340 0x7f857c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f8579d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f857c103340 0x7f857c198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48188/0 (socket says 172.21.15.28:48188) 2024-04-25T01:36:23.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f8579d9b640 1 -- 172.21.15.28:0/1753372091 learned_addr learned my addr 172.21.15.28:0/1753372091 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f857ad9d640 1 --2- 172.21.15.28:0/1753372091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f857c10bd60 0x7f857c19c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f857a59c640 1 --2- 172.21.15.28:0/1753372091 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f857c102a20 0x7f857c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f8579d9b640 1 -- 172.21.15.28:0/1753372091 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f857c102a20 msgr2=0x7f857c198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f8579d9b640 1 --2- 172.21.15.28:0/1753372091 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f857c102a20 0x7f857c198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f8579d9b640 1 -- 172.21.15.28:0/1753372091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f857c10bd60 msgr2=0x7f857c19c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f8579d9b640 1 --2- 172.21.15.28:0/1753372091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f857c10bd60 0x7f857c19c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f8579d9b640 1 -- 172.21.15.28:0/1753372091 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857c19cfd0 con 0x7f857c103340 2024-04-25T01:36:23.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f857ad9d640 1 --2- 172.21.15.28:0/1753372091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f857c10bd60 0x7f857c19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:23.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f8579d9b640 1 --2- 172.21.15.28:0/1753372091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f857c103340 0x7f857c198570 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f857000d770 tx=0x7f857000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f856b7fe640 1 -- 172.21.15.28:0/1753372091 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8570017070 con 0x7f857c103340 2024-04-25T01:36:23.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f856b7fe640 1 -- 172.21.15.28:0/1753372091 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8570005500 con 0x7f857c103340 2024-04-25T01:36:23.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.318+0000 7f8580e88640 1 -- 172.21.15.28:0/1753372091 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f857c19d2c0 con 0x7f857c103340 2024-04-25T01:36:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.319+0000 7f856b7fe640 1 -- 172.21.15.28:0/1753372091 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8570004e00 con 0x7f857c103340 2024-04-25T01:36:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.319+0000 7f8580e88640 1 -- 172.21.15.28:0/1753372091 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f857c1a4b50 con 0x7f857c103340 2024-04-25T01:36:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.319+0000 7f8580e88640 1 -- 172.21.15.28:0/1753372091 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f857c0689b0 con 0x7f857c103340 2024-04-25T01:36:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.321+0000 7f856b7fe640 1 -- 172.21.15.28:0/1753372091 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8570004fa0 con 0x7f857c103340 2024-04-25T01:36:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.321+0000 7f856b7fe640 1 --2- 172.21.15.28:0/1753372091 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f85640a2d80 0x7f85640a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.321+0000 7f857a59c640 1 --2- 172.21.15.28:0/1753372091 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f85640a2d80 0x7f85640a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.321+0000 7f856b7fe640 1 -- 172.21.15.28:0/1753372091 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f85700cf250 con 0x7f857c103340 2024-04-25T01:36:23.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.322+0000 7f857a59c640 1 --2- 172.21.15.28:0/1753372091 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f85640a2d80 0x7f85640a5240 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f856c0096c0 tx=0x7f856c000bc0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.326+0000 7f856b7fe640 1 -- 172.21.15.28:0/1753372091 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f857009bb30 con 0x7f857c103340 2024-04-25T01:36:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.548+0000 7f8580e88640 1 -- 172.21.15.28:0/1753372091 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f857c02d170 con 0x7f857c103340 2024-04-25T01:36:23.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.549+0000 7f856b7fe640 1 -- 172.21.15.28:0/1753372091 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7f857009b430 con 0x7f857c103340 2024-04-25T01:36:23.526 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:36:23.526 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:36:23.526 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:36:23.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.551+0000 7f8580e88640 1 -- 172.21.15.28:0/1753372091 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f85640a2d80 msgr2=0x7f85640a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.551+0000 7f8580e88640 1 --2- 172.21.15.28:0/1753372091 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f85640a2d80 0x7f85640a5240 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f856c0096c0 tx=0x7f856c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.551+0000 7f8580e88640 1 -- 172.21.15.28:0/1753372091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f857c103340 msgr2=0x7f857c198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.551+0000 7f8580e88640 1 --2- 172.21.15.28:0/1753372091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f857c103340 0x7f857c198570 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f857000d770 tx=0x7f857000dc30 comp rx=0 tx=0).stop 2024-04-25T01:36:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.551+0000 7f8580e88640 1 -- 172.21.15.28:0/1753372091 shutdown_connections 2024-04-25T01:36:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.551+0000 7f8580e88640 1 --2- 172.21.15.28:0/1753372091 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f85640a2d80 0x7f85640a5240 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.551+0000 7f8580e88640 1 --2- 172.21.15.28:0/1753372091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f857c10bd60 0x7f857c19c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.551+0000 7f8580e88640 1 --2- 172.21.15.28:0/1753372091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f857c103340 0x7f857c198570 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.551+0000 7f8580e88640 1 --2- 172.21.15.28:0/1753372091 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f857c102a20 0x7f857c198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.551+0000 7f8580e88640 1 -- 172.21.15.28:0/1753372091 >> 172.21.15.28:0/1753372091 conn(0x7f857c0777a0 msgr2=0x7f857c0fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.551+0000 7f8580e88640 1 -- 172.21.15.28:0/1753372091 shutdown_connections 2024-04-25T01:36:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.551+0000 7f8580e88640 1 -- 172.21.15.28:0/1753372091 wait complete. 2024-04-25T01:36:23.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:36:23.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.667+0000 7fd3c5aa7640 1 -- 172.21.15.28:0/2756321009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c0107730 msgr2=0x7fd3c0107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.667+0000 7fd3c5aa7640 1 --2- 172.21.15.28:0/2756321009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c0107730 0x7fd3c0107b10 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0009930 tx=0x7fd3b002a420 comp rx=0 tx=0).stop 2024-04-25T01:36:23.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.667+0000 7fd3c5aa7640 1 -- 172.21.15.28:0/2756321009 shutdown_connections 2024-04-25T01:36:23.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.667+0000 7fd3c5aa7640 1 --2- 172.21.15.28:0/2756321009 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3c006a780 0x7fd3c010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.667+0000 7fd3c5aa7640 1 --2- 172.21.15.28:0/2756321009 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3c0069dc0 0x7fd3c006a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.667+0000 7fd3c5aa7640 1 --2- 172.21.15.28:0/2756321009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c0107730 0x7fd3c0107b10 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.667+0000 7fd3c5aa7640 1 -- 172.21.15.28:0/2756321009 >> 172.21.15.28:0/2756321009 conn(0x7fd3c00776a0 msgr2=0x7fd3c0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.667+0000 7fd3c5aa7640 1 -- 172.21.15.28:0/2756321009 shutdown_connections 2024-04-25T01:36:23.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.667+0000 7fd3c5aa7640 1 -- 172.21.15.28:0/2756321009 wait complete. 2024-04-25T01:36:23.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.668+0000 7fd3c5aa7640 1 Processor -- start 2024-04-25T01:36:23.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.668+0000 7fd3c5aa7640 1 -- start start 2024-04-25T01:36:23.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.668+0000 7fd3c5aa7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3c0069dc0 0x7fd3c0100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.668+0000 7fd3c5aa7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c006a780 0x7fd3c0100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.668+0000 7fd3c5aa7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3c0107730 0x7fd3c01010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.668+0000 7fd3c5aa7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c0111b00 con 0x7fd3c006a780 2024-04-25T01:36:23.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.668+0000 7fd3c5aa7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c0111980 con 0x7fd3c0107730 2024-04-25T01:36:23.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.668+0000 7fd3c5aa7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c0111c80 con 0x7fd3c0069dc0 2024-04-25T01:36:23.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.668+0000 7fd3beffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c006a780 0x7fd3c0100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.668+0000 7fd3beffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c006a780 0x7fd3c0100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48214/0 (socket says 172.21.15.28:48214) 2024-04-25T01:36:23.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.669+0000 7fd3beffd640 1 -- 172.21.15.28:0/688307904 learned_addr learned my addr 172.21.15.28:0/688307904 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.669+0000 7fd3bf7fe640 1 --2- 172.21.15.28:0/688307904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3c0069dc0 0x7fd3c0100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.669+0000 7fd3bffff640 1 --2- 172.21.15.28:0/688307904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3c0107730 0x7fd3c01010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.669+0000 7fd3beffd640 1 -- 172.21.15.28:0/688307904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3c0069dc0 msgr2=0x7fd3c0100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.669+0000 7fd3beffd640 1 --2- 172.21.15.28:0/688307904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3c0069dc0 0x7fd3c0100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.669+0000 7fd3beffd640 1 -- 172.21.15.28:0/688307904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3c0107730 msgr2=0x7fd3c01010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.669+0000 7fd3beffd640 1 --2- 172.21.15.28:0/688307904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3c0107730 0x7fd3c01010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.669+0000 7fd3beffd640 1 -- 172.21.15.28:0/688307904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3c01a6420 con 0x7fd3c006a780 2024-04-25T01:36:23.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.669+0000 7fd3bffff640 1 --2- 172.21.15.28:0/688307904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3c0107730 0x7fd3c01010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:23.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.669+0000 7fd3beffd640 1 --2- 172.21.15.28:0/688307904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c006a780 0x7fd3c0100bb0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400e9f0 tx=0x7fd3b400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.669+0000 7fd3bcff9640 1 -- 172.21.15.28:0/688307904 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b400cd80 con 0x7fd3c006a780 2024-04-25T01:36:23.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.669+0000 7fd3bcff9640 1 -- 172.21.15.28:0/688307904 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3b4010910 con 0x7fd3c006a780 2024-04-25T01:36:23.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.669+0000 7fd3bcff9640 1 -- 172.21.15.28:0/688307904 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4018990 con 0x7fd3c006a780 2024-04-25T01:36:23.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.669+0000 7fd3c5aa7640 1 -- 172.21.15.28:0/688307904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3c01a6710 con 0x7fd3c006a780 2024-04-25T01:36:23.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.669+0000 7fd3c5aa7640 1 -- 172.21.15.28:0/688307904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3c01a6c60 con 0x7fd3c006a780 2024-04-25T01:36:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.671+0000 7fd3bcff9640 1 -- 172.21.15.28:0/688307904 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd3b4010ac0 con 0x7fd3c006a780 2024-04-25T01:36:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3c87b0740 1 Processor -- start 2024-04-25T01:36:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3c87b0740 1 -- start start 2024-04-25T01:36:23.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3c87b0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced322580 0x563ced322960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3c87b0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced322ea0 0x563ced32d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3c87b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced32d570 0x563ced32f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3c87b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563ced287670 con 0x563ced32d570 2024-04-25T01:36:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3c87b0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563ced22f970 con 0x563ced322580 2024-04-25T01:36:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3c87b0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x563ced283630 con 0x563ced322ea0 2024-04-25T01:36:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3bcff9640 1 --2- 172.21.15.28:0/688307904 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd3a80a2d30 0x7fd3a80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3beffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced32d570 0x563ced32f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3beffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced32d570 0x563ced32f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48230/0 (socket says 172.21.15.28:48230) 2024-04-25T01:36:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3beffd640 1 -- 172.21.15.28:0/3699450792 learned_addr learned my addr 172.21.15.28:0/3699450792 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3bffff640 1 --2- 172.21.15.28:0/688307904 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd3a80a2d30 0x7fd3a80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3bf7fe640 1 --2- 172.21.15.28:0/3699450792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced322ea0 0x563ced32d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3bffff640 1 --2- 172.21.15.28:0/3699450792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced322580 0x563ced322960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3bcff9640 1 -- 172.21.15.28:0/688307904 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fd3b40cfaf0 con 0x7fd3c006a780 2024-04-25T01:36:23.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3beffd640 1 -- 172.21.15.28:0/3699450792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced322ea0 msgr2=0x563ced32d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3beffd640 1 --2- 172.21.15.28:0/3699450792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced322ea0 0x563ced32d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3beffd640 1 -- 172.21.15.28:0/3699450792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced322580 msgr2=0x563ced322960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.672+0000 7fd3beffd640 1 --2- 172.21.15.28:0/3699450792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced322580 0x563ced322960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.673+0000 7fd3beffd640 1 -- 172.21.15.28:0/3699450792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ced253400 con 0x563ced32d570 2024-04-25T01:36:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.673+0000 7fd3bffff640 1 --2- 172.21.15.28:0/3699450792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced322580 0x563ced322960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.673+0000 7fd3bf7fe640 1 --2- 172.21.15.28:0/3699450792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced322ea0 0x563ced32d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.673+0000 7fd3beffd640 1 --2- 172.21.15.28:0/3699450792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced32d570 0x563ced32f960 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400c6a0 tx=0x7fd3b40a61e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=960b8f32c9a0e332 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.673+0000 7fd3bffff640 1 --2- 172.21.15.28:0/688307904 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd3a80a2d30 0x7fd3a80a51f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8007fd0 tx=0x7fd3b8007450 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.673+0000 7fd3a5ffb640 1 -- 172.21.15.28:0/3699450792 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4026040 con 0x563ced32d570 2024-04-25T01:36:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.673+0000 7fd3a5ffb640 1 -- 172.21.15.28:0/3699450792 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3b409c3d0 con 0x563ced32d570 2024-04-25T01:36:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.673+0000 7fd3a5ffb640 1 -- 172.21.15.28:0/3699450792 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4098340 con 0x563ced32d570 2024-04-25T01:36:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.673+0000 7fd3c87b0740 1 -- 172.21.15.28:0/3699450792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced32d570 msgr2=0x563ced32f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.673+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/3699450792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced32d570 0x563ced32f960 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400c6a0 tx=0x7fd3b40a61e0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.674+0000 7fd3c87b0740 1 -- 172.21.15.28:0/3699450792 shutdown_connections 2024-04-25T01:36:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.674+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/3699450792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced32d570 0x563ced32f960 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.674+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/3699450792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced322ea0 0x563ced32d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.674+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/3699450792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced322580 0x563ced322960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.674+0000 7fd3c87b0740 1 -- 172.21.15.28:0/3699450792 >> 172.21.15.28:0/3699450792 conn(0x563ced31e8a0 msgr2=0x563ced31ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.674+0000 7fd3c87b0740 1 -- 172.21.15.28:0/3699450792 shutdown_connections 2024-04-25T01:36:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.674+0000 7fd3c87b0740 1 -- 172.21.15.28:0/3699450792 wait complete. 2024-04-25T01:36:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.674+0000 7fd3c87b0740 1 -- start start 2024-04-25T01:36:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.674+0000 7fd3c87b0740 1 Processor -- start 2024-04-25T01:36:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.674+0000 7fd3c87b0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced322580 0x563ced32cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.674+0000 7fd3c87b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced322ea0 0x563ced32a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.674+0000 7fd3c87b0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced32d570 0x563ced32a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.674+0000 7fd3c87b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563ced3302a0 con 0x563ced322ea0 2024-04-25T01:36:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.674+0000 7fd3c87b0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563ced243050 con 0x563ced32d570 2024-04-25T01:36:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.674+0000 7fd3c87b0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x563ced330420 con 0x563ced322580 2024-04-25T01:36:23.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.674+0000 7fd3c87b0740 10 client.0 _ll_register_callbacks cb 0x563cecf9abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.674+0000 7fd3beffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced322ea0 0x563ced32a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.675+0000 7fd3beffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced322ea0 0x563ced32a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48238/0 (socket says 172.21.15.28:48238) 2024-04-25T01:36:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.675+0000 7fd3beffd640 1 -- 172.21.15.28:0/2294433986 learned_addr learned my addr 172.21.15.28:0/2294433986 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.675+0000 7fd3bf7fe640 1 --2- 172.21.15.28:0/2294433986 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced322580 0x563ced32cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.675+0000 7fd3bffff640 1 --2- 172.21.15.28:0/2294433986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced32d570 0x563ced32a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.675+0000 7fd3beffd640 1 -- 172.21.15.28:0/2294433986 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced322580 msgr2=0x563ced32cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.675+0000 7fd3beffd640 1 --2- 172.21.15.28:0/2294433986 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced322580 0x563ced32cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.675+0000 7fd3beffd640 1 -- 172.21.15.28:0/2294433986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced32d570 msgr2=0x563ced32a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.675+0000 7fd3beffd640 1 --2- 172.21.15.28:0/2294433986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced32d570 0x563ced32a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.675+0000 7fd3beffd640 1 -- 172.21.15.28:0/2294433986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ced214dc0 con 0x563ced322ea0 2024-04-25T01:36:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.675+0000 7fd3bf7fe640 1 --2- 172.21.15.28:0/2294433986 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced322580 0x563ced32cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.675+0000 7fd3beffd640 1 --2- 172.21.15.28:0/2294433986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced322ea0 0x563ced32a190 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fd3b409bc60 tx=0x7fd3b409bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c31ddbe2ae29cf9a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.675+0000 7fd39effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.675+0000 7fd39effd640 1 -- 172.21.15.28:0/2294433986 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b40a0080 con 0x563ced322ea0 2024-04-25T01:36:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.675+0000 7fd39effd640 1 -- 172.21.15.28:0/2294433986 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fd3b40a44b0 con 0x563ced322ea0 2024-04-25T01:36:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.675+0000 7fd3c87b0740 1 -- 172.21.15.28:0/2294433986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3b409c3d0 con 0x563ced322ea0 2024-04-25T01:36:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.675+0000 7fd39effd640 1 -- 172.21.15.28:0/2294433986 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd3b40b9050 con 0x563ced322ea0 2024-04-25T01:36:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.675+0000 7fd3c87b0740 10 client.16584 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.675+0000 7fd3c87b0740 1 -- 172.21.15.28:0/2294433986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563cecf9b3f0 con 0x563ced322ea0 2024-04-25T01:36:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.675+0000 7fd39effd640 1 -- 172.21.15.28:0/2294433986 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3b409c3d0 con 0x563ced322ea0 2024-04-25T01:36:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.676+0000 7fd39effd640 1 -- 172.21.15.28:0/2294433986 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd3b40b3020 con 0x563ced322ea0 2024-04-25T01:36:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.676+0000 7fd3c87b0740 10 client.16584 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.676+0000 7fd3c87b0740 10 client.16584 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:36:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.676+0000 7fd3c87b0740 20 client.16584 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.676+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/2294433986 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x563ced33d370 0x563ced33f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.676+0000 7fd3c87b0740 4 client.16584 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.676+0000 7fd3c87b0740 1 -- 172.21.15.28:0/2294433986 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ced294bb0 con 0x563ced33d370 2024-04-25T01:36:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.676+0000 7fd3bf7fe640 1 --2- 172.21.15.28:0/2294433986 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x563ced33d370 0x563ced33f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.676+0000 7fd3bf7fe640 1 --2- 172.21.15.28:0/2294433986 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x563ced33d370 0x563ced33f760 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=16f5df4374526d37 server_cookie=fb55e20d9986a4e in_seq=0 out_seq=0 2024-04-25T01:36:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.676+0000 7fd39effd640 10 client.16584 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:36:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.677+0000 7fd39effd640 1 -- 172.21.15.28:0/2294433986 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563ced294bb0 con 0x563ced33d370 2024-04-25T01:36:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.677+0000 7fd39effd640 10 client.16584 handle_command_reply: tid=0 2024-04-25T01:36:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.677+0000 7fd3c87b0740 1 client.16584 shutdown 2024-04-25T01:36:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.677+0000 7fd3c87b0740 1 -- 172.21.15.28:0/2294433986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced322ea0 msgr2=0x563ced32a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.677+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/2294433986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced322ea0 0x563ced32a190 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fd3b409bc60 tx=0x7fd3b409bd40 comp rx=0 tx=0).stop 2024-04-25T01:36:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.677+0000 7fd3c87b0740 1 -- 172.21.15.28:0/2294433986 shutdown_connections 2024-04-25T01:36:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.677+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/2294433986 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x563ced33d370 0x563ced33f760 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.677+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/2294433986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced32d570 0x563ced32a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.677+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/2294433986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced322ea0 0x563ced32a190 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.677+0000 7fd39effd640 0 client.16584 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:36:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.677+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/2294433986 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced322580 0x563ced32cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.677+0000 7fd3c87b0740 1 -- 172.21.15.28:0/2294433986 >> 172.21.15.28:0/2294433986 conn(0x563ced31e8a0 msgr2=0x563ced31db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.678+0000 7fd3c87b0740 1 -- 172.21.15.28:0/2294433986 shutdown_connections 2024-04-25T01:36:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.678+0000 7fd3c87b0740 1 -- 172.21.15.28:0/2294433986 wait complete. 2024-04-25T01:36:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.678+0000 7fd3c87b0740 20 client.16584 trim_cache size 0 max 16384 2024-04-25T01:36:23.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.691+0000 7fd3c87b0740 1 Processor -- start 2024-04-25T01:36:23.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.691+0000 7fd3c87b0740 1 -- start start 2024-04-25T01:36:23.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.691+0000 7fd3c87b0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced322580 0x563ced32c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.691+0000 7fd3c87b0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced322ea0 0x563ced342780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.691+0000 7fd3c87b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced32d570 0x563ced342cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.691+0000 7fd3c87b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c01b0bc0 con 0x563ced32d570 2024-04-25T01:36:23.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.691+0000 7fd3c87b0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c01b2700 con 0x563ced322580 2024-04-25T01:36:23.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.691+0000 7fd3c87b0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c01b1290 con 0x563ced322ea0 2024-04-25T01:36:23.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.691+0000 7fd3bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced32d570 0x563ced342cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.691+0000 7fd3bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced32d570 0x563ced342cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48254/0 (socket says 172.21.15.28:48254) 2024-04-25T01:36:23.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.691+0000 7fd3bffff640 1 -- 172.21.15.28:0/3930748643 learned_addr learned my addr 172.21.15.28:0/3930748643 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.691+0000 7fd3bf7fe640 1 --2- 172.21.15.28:0/3930748643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced322580 0x563ced32c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.691+0000 7fd3beffd640 1 --2- 172.21.15.28:0/3930748643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced322ea0 0x563ced342780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.692+0000 7fd3bffff640 1 -- 172.21.15.28:0/3930748643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced322ea0 msgr2=0x563ced342780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.692+0000 7fd3bffff640 1 --2- 172.21.15.28:0/3930748643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced322ea0 0x563ced342780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.692+0000 7fd3bffff640 1 -- 172.21.15.28:0/3930748643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced322580 msgr2=0x563ced32c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.692+0000 7fd3bffff640 1 --2- 172.21.15.28:0/3930748643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced322580 0x563ced32c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.692+0000 7fd3bffff640 1 -- 172.21.15.28:0/3930748643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ced0657d0 con 0x563ced32d570 2024-04-25T01:36:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.692+0000 7fd3bffff640 1 --2- 172.21.15.28:0/3930748643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced32d570 0x563ced342cc0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8013600 tx=0x7fd3b80072a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c649a5cd013ed50 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.692+0000 7fd3a57fa640 1 -- 172.21.15.28:0/3930748643 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8023070 con 0x563ced32d570 2024-04-25T01:36:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.692+0000 7fd3a57fa640 1 -- 172.21.15.28:0/3930748643 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3b801f450 con 0x563ced32d570 2024-04-25T01:36:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.692+0000 7fd3a57fa640 1 -- 172.21.15.28:0/3930748643 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8011660 con 0x563ced32d570 2024-04-25T01:36:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.692+0000 7fd3c87b0740 1 -- 172.21.15.28:0/3930748643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced32d570 msgr2=0x563ced342cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.692+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/3930748643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced32d570 0x563ced342cc0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8013600 tx=0x7fd3b80072a0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.693+0000 7fd3c87b0740 1 -- 172.21.15.28:0/3930748643 shutdown_connections 2024-04-25T01:36:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.693+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/3930748643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced32d570 0x563ced342cc0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.693+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/3930748643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced322ea0 0x563ced342780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.693+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/3930748643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced322580 0x563ced32c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.693+0000 7fd3c87b0740 1 -- 172.21.15.28:0/3930748643 >> 172.21.15.28:0/3930748643 conn(0x563ced31e8a0 msgr2=0x563ced338f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.693+0000 7fd3c87b0740 1 -- 172.21.15.28:0/3930748643 shutdown_connections 2024-04-25T01:36:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.693+0000 7fd3c87b0740 1 -- 172.21.15.28:0/3930748643 wait complete. 2024-04-25T01:36:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.693+0000 7fd3c87b0740 1 -- start start 2024-04-25T01:36:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.693+0000 7fd3c87b0740 1 Processor -- start 2024-04-25T01:36:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.693+0000 7fd3c87b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced322580 0x563ced342a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.693+0000 7fd3c87b0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced322ea0 0x563ced33fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.693+0000 7fd3c87b0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced32d570 0x563ced340350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.693+0000 7fd3c87b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c01b2030 con 0x563ced322580 2024-04-25T01:36:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.693+0000 7fd3c87b0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c01b1960 con 0x563ced322ea0 2024-04-25T01:36:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.693+0000 7fd3c87b0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c01aeba0 con 0x563ced32d570 2024-04-25T01:36:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.693+0000 7fd3c87b0740 10 client.0 _ll_register_callbacks cb 0x7fd3b409b8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.693+0000 7fd3bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced322580 0x563ced342a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.694+0000 7fd3bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced322580 0x563ced342a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48256/0 (socket says 172.21.15.28:48256) 2024-04-25T01:36:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.694+0000 7fd3bf7fe640 1 -- 172.21.15.28:0/3786813261 learned_addr learned my addr 172.21.15.28:0/3786813261 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.694+0000 7fd3beffd640 1 --2- 172.21.15.28:0/3786813261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced322ea0 0x563ced33fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.694+0000 7fd3bffff640 1 --2- 172.21.15.28:0/3786813261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced32d570 0x563ced340350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.694+0000 7fd3bf7fe640 1 -- 172.21.15.28:0/3786813261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced32d570 msgr2=0x563ced340350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.694+0000 7fd3bf7fe640 1 --2- 172.21.15.28:0/3786813261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced32d570 0x563ced340350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.694+0000 7fd3bf7fe640 1 -- 172.21.15.28:0/3786813261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced322ea0 msgr2=0x563ced33fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.694+0000 7fd3bf7fe640 1 --2- 172.21.15.28:0/3786813261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced322ea0 0x563ced33fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.694+0000 7fd3bf7fe640 1 -- 172.21.15.28:0/3786813261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ced340b50 con 0x563ced322580 2024-04-25T01:36:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.694+0000 7fd3bf7fe640 1 --2- 172.21.15.28:0/3786813261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced322580 0x563ced342a80 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0009900 tx=0x7fd3b0031aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8625aa230cf56c43 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.694+0000 7fd39effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.694+0000 7fd39effd640 1 -- 172.21.15.28:0/3786813261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b0035040 con 0x563ced322580 2024-04-25T01:36:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.694+0000 7fd39effd640 1 -- 172.21.15.28:0/3786813261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fd3b0045dc0 con 0x563ced322580 2024-04-25T01:36:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.694+0000 7fd3c87b0740 1 -- 172.21.15.28:0/3786813261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3b801f450 con 0x563ced322580 2024-04-25T01:36:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.694+0000 7fd39effd640 1 -- 172.21.15.28:0/3786813261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd3b0057050 con 0x563ced322580 2024-04-25T01:36:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.694+0000 7fd3c87b0740 10 client.16590 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.694+0000 7fd3c87b0740 1 -- 172.21.15.28:0/3786813261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563ced3a3540 con 0x563ced322580 2024-04-25T01:36:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.694+0000 7fd39effd640 1 -- 172.21.15.28:0/3786813261 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3b801f450 con 0x563ced322580 2024-04-25T01:36:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.695+0000 7fd39effd640 1 -- 172.21.15.28:0/3786813261 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd3b00035f0 con 0x563ced322580 2024-04-25T01:36:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.695+0000 7fd3c87b0740 10 client.16590 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.695+0000 7fd3c87b0740 10 client.16590 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:36:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.695+0000 7fd3c87b0740 20 client.16590 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.695+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/3786813261 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x563ced3a3c10 0x563ced3a6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.695+0000 7fd3c87b0740 4 client.16590 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.695+0000 7fd3c87b0740 1 -- 172.21.15.28:0/3786813261 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563ced28ef00 con 0x563ced3a3c10 2024-04-25T01:36:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.695+0000 7fd3beffd640 1 --2- 172.21.15.28:0/3786813261 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x563ced3a3c10 0x563ced3a6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.695+0000 7fd39effd640 10 client.16590 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:36:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.695+0000 7fd3beffd640 1 --2- 172.21.15.28:0/3786813261 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x563ced3a3c10 0x563ced3a6000 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9279508ceb4bd18c server_cookie=d1cc5d7ab8ec5e3f in_seq=0 out_seq=0 2024-04-25T01:36:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.695+0000 7fd39effd640 1 -- 172.21.15.28:0/3786813261 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563ced28ef00 con 0x563ced3a3c10 2024-04-25T01:36:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.695+0000 7fd39effd640 10 client.16590 handle_command_reply: tid=0 2024-04-25T01:36:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.695+0000 7fd3c87b0740 1 client.16590 shutdown 2024-04-25T01:36:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c87b0740 1 -- 172.21.15.28:0/3786813261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced322580 msgr2=0x563ced342a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/3786813261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced322580 0x563ced342a80 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0009900 tx=0x7fd3b0031aa0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c87b0740 1 -- 172.21.15.28:0/3786813261 shutdown_connections 2024-04-25T01:36:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/3786813261 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x563ced3a3c10 0x563ced3a6000 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/3786813261 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ced32d570 0x563ced340350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/3786813261 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ced322ea0 0x563ced33fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd39effd640 0 client.16590 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:36:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c87b0740 1 --2- 172.21.15.28:0/3786813261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ced322580 0x563ced342a80 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c87b0740 1 -- 172.21.15.28:0/3786813261 >> 172.21.15.28:0/3786813261 conn(0x563ced31e8a0 msgr2=0x563ced324e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c87b0740 1 -- 172.21.15.28:0/3786813261 shutdown_connections 2024-04-25T01:36:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c87b0740 1 -- 172.21.15.28:0/3786813261 wait complete. 2024-04-25T01:36:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c87b0740 20 client.16590 trim_cache size 0 max 16384 2024-04-25T01:36:23.681 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:36:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c5aa7640 1 -- 172.21.15.28:0/688307904 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd3a80a2d30 msgr2=0x7fd3a80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c5aa7640 1 --2- 172.21.15.28:0/688307904 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd3a80a2d30 0x7fd3a80a51f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8007fd0 tx=0x7fd3b8007450 comp rx=0 tx=0).stop 2024-04-25T01:36:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c5aa7640 1 -- 172.21.15.28:0/688307904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c006a780 msgr2=0x7fd3c0100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c5aa7640 1 --2- 172.21.15.28:0/688307904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c006a780 0x7fd3c0100bb0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400e9f0 tx=0x7fd3b400eeb0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c5aa7640 1 -- 172.21.15.28:0/688307904 shutdown_connections 2024-04-25T01:36:23.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c5aa7640 1 --2- 172.21.15.28:0/688307904 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd3a80a2d30 0x7fd3a80a51f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c5aa7640 1 --2- 172.21.15.28:0/688307904 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd3c0107730 0x7fd3c01010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c5aa7640 1 --2- 172.21.15.28:0/688307904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3c006a780 0x7fd3c0100bb0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c5aa7640 1 --2- 172.21.15.28:0/688307904 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd3c0069dc0 0x7fd3c0100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.696+0000 7fd3c5aa7640 1 -- 172.21.15.28:0/688307904 >> 172.21.15.28:0/688307904 conn(0x7fd3c00776a0 msgr2=0x7fd3c00fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.697+0000 7fd3c5aa7640 1 -- 172.21.15.28:0/688307904 shutdown_connections 2024-04-25T01:36:23.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.697+0000 7fd3c5aa7640 1 -- 172.21.15.28:0/688307904 wait complete. 2024-04-25T01:36:23.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:36:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.811+0000 7f2f0c9f9640 1 -- 172.21.15.28:0/2844769762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f08100fd0 msgr2=0x7f2f0810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.811+0000 7f2f0c9f9640 1 --2- 172.21.15.28:0/2844769762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f08100fd0 0x7f2f0810d4e0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f2f00009cd0 tx=0x7f2f0002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.812+0000 7f2f0c9f9640 1 -- 172.21.15.28:0/2844769762 shutdown_connections 2024-04-25T01:36:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.812+0000 7f2f0c9f9640 1 --2- 172.21.15.28:0/2844769762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f08100fd0 0x7f2f0810d4e0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.812+0000 7f2f0c9f9640 1 --2- 172.21.15.28:0/2844769762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f08100610 0x7f2f08100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.812+0000 7f2f0c9f9640 1 --2- 172.21.15.28:0/2844769762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f081066b0 0x7f2f08106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.812+0000 7f2f0c9f9640 1 -- 172.21.15.28:0/2844769762 >> 172.21.15.28:0/2844769762 conn(0x7f2f08077740 msgr2=0x7f2f08077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.812+0000 7f2f0c9f9640 1 -- 172.21.15.28:0/2844769762 shutdown_connections 2024-04-25T01:36:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.812+0000 7f2f0c9f9640 1 -- 172.21.15.28:0/2844769762 wait complete. 2024-04-25T01:36:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.812+0000 7f2f0c9f9640 1 Processor -- start 2024-04-25T01:36:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.812+0000 7f2f0c9f9640 1 -- start start 2024-04-25T01:36:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.813+0000 7f2f0c9f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f08100610 0x7f2f0810df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.813+0000 7f2f0c9f9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f08100fd0 0x7f2f0810e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.813+0000 7f2f0c9f9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f081066b0 0x7f2f0810e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.813+0000 7f2f0c9f9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f08111b80 con 0x7f2f08100610 2024-04-25T01:36:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.813+0000 7f2f0c9f9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2f08111a00 con 0x7f2f081066b0 2024-04-25T01:36:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.813+0000 7f2f0c9f9640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2f08111d00 con 0x7f2f08100fd0 2024-04-25T01:36:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.813+0000 7f2f0659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f08100610 0x7f2f0810df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.813+0000 7f2f0659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f08100610 0x7f2f0810df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48272/0 (socket says 172.21.15.28:48272) 2024-04-25T01:36:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.813+0000 7f2f0659c640 1 -- 172.21.15.28:0/3311109297 learned_addr learned my addr 172.21.15.28:0/3311109297 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.813+0000 7f2f05d9b640 1 --2- 172.21.15.28:0/3311109297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f08100fd0 0x7f2f0810e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.813+0000 7f2f06d9d640 1 --2- 172.21.15.28:0/3311109297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f081066b0 0x7f2f0810e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.813+0000 7f2f0659c640 1 -- 172.21.15.28:0/3311109297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f08100fd0 msgr2=0x7f2f0810e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.813+0000 7f2f0659c640 1 --2- 172.21.15.28:0/3311109297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f08100fd0 0x7f2f0810e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.813+0000 7f2f0659c640 1 -- 172.21.15.28:0/3311109297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f081066b0 msgr2=0x7f2f0810e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.813+0000 7f2f0659c640 1 --2- 172.21.15.28:0/3311109297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f081066b0 0x7f2f0810e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.813+0000 7f2f0659c640 1 -- 172.21.15.28:0/3311109297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f081a64c0 con 0x7f2f08100610 2024-04-25T01:36:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.813+0000 7f2f06d9d640 1 --2- 172.21.15.28:0/3311109297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f081066b0 0x7f2f0810e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.813+0000 7f2f0659c640 1 --2- 172.21.15.28:0/3311109297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f08100610 0x7f2f0810df70 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f2ef800d950 tx=0x7f2ef800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.813+0000 7f2ef77fe640 1 -- 172.21.15.28:0/3311109297 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8017070 con 0x7f2f08100610 2024-04-25T01:36:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.814+0000 7f2f0c9f9640 1 -- 172.21.15.28:0/3311109297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f081a67b0 con 0x7f2f08100610 2024-04-25T01:36:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.814+0000 7f2ef77fe640 1 -- 172.21.15.28:0/3311109297 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ef8009c30 con 0x7f2f08100610 2024-04-25T01:36:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.814+0000 7f2ef77fe640 1 -- 172.21.15.28:0/3311109297 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef800f600 con 0x7f2f08100610 2024-04-25T01:36:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.814+0000 7f2f0c9f9640 1 -- 172.21.15.28:0/3311109297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f081a6d00 con 0x7f2f08100610 2024-04-25T01:36:23.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.816+0000 7f2ef77fe640 1 -- 172.21.15.28:0/3311109297 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2ef801a070 con 0x7f2f08100610 2024-04-25T01:36:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.816+0000 7f2f0f7e8740 1 Processor -- start 2024-04-25T01:36:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.816+0000 7f2f0f7e8740 1 -- start start 2024-04-25T01:36:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2f0f7e8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f376b580 0x5626f376b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2f0f7e8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f376bea0 0x5626f3776030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2f0f7e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f3776570 0x5626f3778960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2f0f7e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5626f36d0670 con 0x5626f3776570 2024-04-25T01:36:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2f0f7e8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5626f3678970 con 0x5626f376bea0 2024-04-25T01:36:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2f0f7e8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5626f36cc630 con 0x5626f376b580 2024-04-25T01:36:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2ef77fe640 1 --2- 172.21.15.28:0/3311109297 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f2ef00a2d30 0x7f2ef00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2f05d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f3776570 0x5626f3778960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2f05d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f3776570 0x5626f3778960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48284/0 (socket says 172.21.15.28:48284) 2024-04-25T01:36:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2f05d9b640 1 -- 172.21.15.28:0/1339877545 learned_addr learned my addr 172.21.15.28:0/1339877545 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2f06d9d640 1 --2- 172.21.15.28:0/3311109297 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f2ef00a2d30 0x7f2ef00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2f06d9d640 1 --2- 172.21.15.28:0/1339877545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f376b580 0x5626f376b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2ef77fe640 1 -- 172.21.15.28:0/3311109297 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f2ef80cf170 con 0x7f2f08100610 2024-04-25T01:36:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2f0659c640 1 --2- 172.21.15.28:0/1339877545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f376bea0 0x5626f3776030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2f05d9b640 1 -- 172.21.15.28:0/1339877545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f376b580 msgr2=0x5626f376b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2f05d9b640 1 --2- 172.21.15.28:0/1339877545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f376b580 0x5626f376b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2f05d9b640 1 -- 172.21.15.28:0/1339877545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f376bea0 msgr2=0x5626f3776030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2f05d9b640 1 --2- 172.21.15.28:0/1339877545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f376bea0 0x5626f3776030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2f05d9b640 1 -- 172.21.15.28:0/1339877545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626f369c400 con 0x5626f3776570 2024-04-25T01:36:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2f05d9b640 1 --2- 172.21.15.28:0/1339877545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f3776570 0x5626f3778960 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2efc00abf0 tx=0x7f2efc00f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3699feb315d9f40c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2f06d9d640 1 --2- 172.21.15.28:0/3311109297 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f2ef00a2d30 0x7f2ef00a51f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2f00006010 tx=0x7f2f00009980 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.817+0000 7f2ef4ff9640 1 -- 172.21.15.28:0/1339877545 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2efc01d070 con 0x5626f3776570 2024-04-25T01:36:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.818+0000 7f2ef4ff9640 1 -- 172.21.15.28:0/1339877545 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2efc018920 con 0x5626f3776570 2024-04-25T01:36:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.818+0000 7f2ef4ff9640 1 -- 172.21.15.28:0/1339877545 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2efc010a90 con 0x5626f3776570 2024-04-25T01:36:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.818+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/1339877545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f3776570 msgr2=0x5626f3778960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.818+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/1339877545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f3776570 0x5626f3778960 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2efc00abf0 tx=0x7f2efc00f3c0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.818+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/1339877545 shutdown_connections 2024-04-25T01:36:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.818+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/1339877545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f3776570 0x5626f3778960 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.818+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/1339877545 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f376bea0 0x5626f3776030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.818+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/1339877545 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f376b580 0x5626f376b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.818+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/1339877545 >> 172.21.15.28:0/1339877545 conn(0x5626f37678a0 msgr2=0x5626f3767c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.818+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/1339877545 shutdown_connections 2024-04-25T01:36:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.818+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/1339877545 wait complete. 2024-04-25T01:36:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f0f7e8740 1 -- start start 2024-04-25T01:36:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f0f7e8740 1 Processor -- start 2024-04-25T01:36:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f0f7e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376b580 0x5626f3775eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f0f7e8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f376bea0 0x5626f37731a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f0f7e8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f3776570 0x5626f37736e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f0f7e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5626f3779360 con 0x5626f376b580 2024-04-25T01:36:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f0f7e8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5626f368c050 con 0x5626f3776570 2024-04-25T01:36:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f0f7e8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5626f37794e0 con 0x5626f376bea0 2024-04-25T01:36:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f0f7e8740 10 client.0 _ll_register_callbacks cb 0x5626f33e3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f0659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376b580 0x5626f3775eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f0659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376b580 0x5626f3775eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48300/0 (socket says 172.21.15.28:48300) 2024-04-25T01:36:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f0659c640 1 -- 172.21.15.28:0/3345743448 learned_addr learned my addr 172.21.15.28:0/3345743448 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f05d9b640 1 --2- 172.21.15.28:0/3345743448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f376bea0 0x5626f37731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f06d9d640 1 --2- 172.21.15.28:0/3345743448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f3776570 0x5626f37736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f0659c640 1 -- 172.21.15.28:0/3345743448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f376bea0 msgr2=0x5626f37731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f0659c640 1 --2- 172.21.15.28:0/3345743448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f376bea0 0x5626f37731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f0659c640 1 -- 172.21.15.28:0/3345743448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f3776570 msgr2=0x5626f37736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f0659c640 1 --2- 172.21.15.28:0/3345743448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f3776570 0x5626f37736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f0659c640 1 -- 172.21.15.28:0/3345743448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626f365ddc0 con 0x5626f376b580 2024-04-25T01:36:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.819+0000 7f2f05d9b640 1 --2- 172.21.15.28:0/3345743448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f376bea0 0x5626f37731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.820+0000 7f2f0659c640 1 --2- 172.21.15.28:0/3345743448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376b580 0x5626f3775eb0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2ef800bf80 tx=0x7f2ef809afb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4797c98dde04fb6f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.820+0000 7f2eedffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.820+0000 7f2eedffb640 1 -- 172.21.15.28:0/3345743448 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8021040 con 0x5626f376b580 2024-04-25T01:36:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.820+0000 7f2eedffb640 1 -- 172.21.15.28:0/3345743448 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f2ef80979c0 con 0x5626f376b580 2024-04-25T01:36:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.820+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/3345743448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2efc018920 con 0x5626f376b580 2024-04-25T01:36:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.820+0000 7f2eedffb640 1 -- 172.21.15.28:0/3345743448 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2ef80a3fc0 con 0x5626f376b580 2024-04-25T01:36:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.820+0000 7f2f0f7e8740 10 client.16602 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.820+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/3345743448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5626f33e43f0 con 0x5626f376b580 2024-04-25T01:36:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.820+0000 7f2eedffb640 1 -- 172.21.15.28:0/3345743448 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2efc018920 con 0x5626f376b580 2024-04-25T01:36:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.820+0000 7f2eedffb640 1 -- 172.21.15.28:0/3345743448 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2ef80bb0b0 con 0x5626f376b580 2024-04-25T01:36:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.820+0000 7f2f0f7e8740 10 client.16602 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.820+0000 7f2f0f7e8740 10 client.16602 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:36:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.820+0000 7f2f0f7e8740 20 client.16602 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.820+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/3345743448 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5626f3786640 0x5626f3788a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.820+0000 7f2f0f7e8740 4 client.16602 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.820+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/3345743448 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626f36ddbb0 con 0x5626f3786640 2024-04-25T01:36:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.821+0000 7f2f05d9b640 1 --2- 172.21.15.28:0/3345743448 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5626f3786640 0x5626f3788a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.821+0000 7f2eedffb640 10 client.16602 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:36:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.821+0000 7f2f05d9b640 1 --2- 172.21.15.28:0/3345743448 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5626f3786640 0x5626f3788a90 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bcd107ed8b7da205 server_cookie=f450477bb3cc25b in_seq=0 out_seq=0 2024-04-25T01:36:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.822+0000 7f2eedffb640 1 -- 172.21.15.28:0/3345743448 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5626f36ddbb0 con 0x5626f3786640 2024-04-25T01:36:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.822+0000 7f2eedffb640 10 client.16602 handle_command_reply: tid=0 2024-04-25T01:36:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.822+0000 7f2f0f7e8740 1 client.16602 shutdown 2024-04-25T01:36:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.822+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/3345743448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376b580 msgr2=0x5626f3775eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.822+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/3345743448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376b580 0x5626f3775eb0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2ef800bf80 tx=0x7f2ef809afb0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.822+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/3345743448 shutdown_connections 2024-04-25T01:36:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.822+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/3345743448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f3776570 0x5626f37736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.822+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/3345743448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f376bea0 0x5626f37731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.822+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/3345743448 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5626f3786640 0x5626f3788a90 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.822+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/3345743448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376b580 0x5626f3775eb0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.822+0000 7f2eedffb640 0 client.16602 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:36:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.822+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/3345743448 >> 172.21.15.28:0/3345743448 conn(0x5626f37678a0 msgr2=0x5626f3766b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.822+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/3345743448 shutdown_connections 2024-04-25T01:36:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.822+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/3345743448 wait complete. 2024-04-25T01:36:23.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.822+0000 7f2f0f7e8740 20 client.16602 trim_cache size 0 max 16384 2024-04-25T01:36:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.835+0000 7f2f0f7e8740 1 Processor -- start 2024-04-25T01:36:23.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.835+0000 7f2f0f7e8740 1 -- start start 2024-04-25T01:36:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.835+0000 7f2f0f7e8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f37678a0 0x5626f3782220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.835+0000 7f2f0f7e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376b580 0x5626f376e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.835+0000 7f2f0f7e8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f376bea0 0x5626f37759e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.835+0000 7f2f0f7e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f081b0b80 con 0x5626f376b580 2024-04-25T01:36:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.835+0000 7f2f0f7e8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2f081b26c0 con 0x5626f376bea0 2024-04-25T01:36:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.835+0000 7f2f0f7e8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2f081b1250 con 0x5626f37678a0 2024-04-25T01:36:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.836+0000 7f2f05d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376b580 0x5626f376e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.836+0000 7f2f05d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376b580 0x5626f376e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48316/0 (socket says 172.21.15.28:48316) 2024-04-25T01:36:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.836+0000 7f2f05d9b640 1 -- 172.21.15.28:0/1543448357 learned_addr learned my addr 172.21.15.28:0/1543448357 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.836+0000 7f2f05d9b640 1 -- 172.21.15.28:0/1543448357 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f37678a0 msgr2=0x5626f3782220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.836+0000 7f2f05d9b640 1 --2- 172.21.15.28:0/1543448357 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f37678a0 0x5626f3782220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.836+0000 7f2f05d9b640 1 -- 172.21.15.28:0/1543448357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f376bea0 msgr2=0x5626f37759e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.836+0000 7f2f05d9b640 1 --2- 172.21.15.28:0/1543448357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f376bea0 0x5626f37759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.836+0000 7f2f05d9b640 1 -- 172.21.15.28:0/1543448357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626f34ae7d0 con 0x5626f376b580 2024-04-25T01:36:23.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.836+0000 7f2f05d9b640 1 --2- 172.21.15.28:0/1543448357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376b580 0x5626f376e600 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f2efc00abf0 tx=0x7f2efc00a900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8712b1ec1c758850 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.836+0000 7f2eeffff640 1 -- 172.21.15.28:0/1543448357 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2efc02a030 con 0x5626f376b580 2024-04-25T01:36:23.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.836+0000 7f2eeffff640 1 -- 172.21.15.28:0/1543448357 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2efc005040 con 0x5626f376b580 2024-04-25T01:36:23.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.836+0000 7f2eeffff640 1 -- 172.21.15.28:0/1543448357 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2efc01d040 con 0x5626f376b580 2024-04-25T01:36:23.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.837+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/1543448357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376b580 msgr2=0x5626f376e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.837+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/1543448357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376b580 0x5626f376e600 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f2efc00abf0 tx=0x7f2efc00a900 comp rx=0 tx=0).stop 2024-04-25T01:36:23.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.837+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/1543448357 shutdown_connections 2024-04-25T01:36:23.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.837+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/1543448357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f376bea0 0x5626f37759e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.837+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/1543448357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376b580 0x5626f376e600 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.837+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/1543448357 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f37678a0 0x5626f3782220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.837+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/1543448357 >> 172.21.15.28:0/1543448357 conn(0x5626f3773840 msgr2=0x5626f376def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.837+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/1543448357 shutdown_connections 2024-04-25T01:36:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.837+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/1543448357 wait complete. 2024-04-25T01:36:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.837+0000 7f2f0f7e8740 1 -- start start 2024-04-25T01:36:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.837+0000 7f2f0f7e8740 1 Processor -- start 2024-04-25T01:36:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.837+0000 7f2f0f7e8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f37678a0 0x5626f377e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.837+0000 7f2f0f7e8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f376b580 0x5626f376fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.837+0000 7f2f0f7e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376bea0 0x5626f376ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.837+0000 7f2f0f7e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f081b1ff0 con 0x5626f376bea0 2024-04-25T01:36:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.837+0000 7f2f0f7e8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2f081b1920 con 0x5626f376b580 2024-04-25T01:36:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.837+0000 7f2f0f7e8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2f081aeb60 con 0x5626f37678a0 2024-04-25T01:36:23.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.837+0000 7f2f0f7e8740 10 client.0 _ll_register_callbacks cb 0x7f2ef809ad80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.838+0000 7f2f06d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376bea0 0x5626f376ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.838+0000 7f2f06d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376bea0 0x5626f376ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48328/0 (socket says 172.21.15.28:48328) 2024-04-25T01:36:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.838+0000 7f2f06d9d640 1 -- 172.21.15.28:0/1693291269 learned_addr learned my addr 172.21.15.28:0/1693291269 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.838+0000 7f2f0659c640 1 --2- 172.21.15.28:0/1693291269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f37678a0 0x5626f377e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.838+0000 7f2f05d9b640 1 --2- 172.21.15.28:0/1693291269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f376b580 0x5626f376fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.838+0000 7f2f06d9d640 1 -- 172.21.15.28:0/1693291269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f37678a0 msgr2=0x5626f377e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.838+0000 7f2f06d9d640 1 --2- 172.21.15.28:0/1693291269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f37678a0 0x5626f377e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.838+0000 7f2f06d9d640 1 -- 172.21.15.28:0/1693291269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f376b580 msgr2=0x5626f376fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.838+0000 7f2f06d9d640 1 --2- 172.21.15.28:0/1693291269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f376b580 0x5626f376fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.838+0000 7f2f06d9d640 1 -- 172.21.15.28:0/1693291269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626f37705f0 con 0x5626f376bea0 2024-04-25T01:36:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.838+0000 7f2f05d9b640 1 --2- 172.21.15.28:0/1693291269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f376b580 0x5626f376fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.838+0000 7f2f06d9d640 1 --2- 172.21.15.28:0/1693291269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376bea0 0x5626f376ff50 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f2f00009cd0 tx=0x7f2f000095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0cd24d6952c3fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.838+0000 7f2eedffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.838+0000 7f2eedffb640 1 -- 172.21.15.28:0/1693291269 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f00031d30 con 0x5626f376bea0 2024-04-25T01:36:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.838+0000 7f2eedffb640 1 -- 172.21.15.28:0/1693291269 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f2f000305a0 con 0x5626f376bea0 2024-04-25T01:36:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.838+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/1693291269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2efc005040 con 0x5626f376bea0 2024-04-25T01:36:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.838+0000 7f2eedffb640 1 -- 172.21.15.28:0/1693291269 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2f000307a0 con 0x5626f376bea0 2024-04-25T01:36:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.838+0000 7f2f0f7e8740 10 client.16608 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.838+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/1693291269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5626f37eca30 con 0x5626f376bea0 2024-04-25T01:36:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.839+0000 7f2eedffb640 1 -- 172.21.15.28:0/1693291269 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2efc005040 con 0x5626f376bea0 2024-04-25T01:36:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.839+0000 7f2eedffb640 1 -- 172.21.15.28:0/1693291269 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2f00030070 con 0x5626f376bea0 2024-04-25T01:36:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.839+0000 7f2f0f7e8740 10 client.16608 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.839+0000 7f2f0f7e8740 10 client.16608 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:36:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.839+0000 7f2f0f7e8740 20 client.16608 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.839+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/1693291269 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5626f3776570 0x5626f37ef270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.839+0000 7f2f0f7e8740 4 client.16608 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.839+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/1693291269 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5626f36d7f00 con 0x5626f3776570 2024-04-25T01:36:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.839+0000 7f2f0659c640 1 --2- 172.21.15.28:0/1693291269 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5626f3776570 0x5626f37ef270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.839+0000 7f2f0659c640 1 --2- 172.21.15.28:0/1693291269 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5626f3776570 0x5626f37ef270 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d1a79e6c6088b868 server_cookie=dd3165c83761c12 in_seq=0 out_seq=0 2024-04-25T01:36:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.839+0000 7f2eedffb640 10 client.16608 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:36:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2eedffb640 1 -- 172.21.15.28:0/1693291269 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5626f36d7f00 con 0x5626f3776570 2024-04-25T01:36:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2eedffb640 10 client.16608 handle_command_reply: tid=0 2024-04-25T01:36:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0f7e8740 1 client.16608 shutdown 2024-04-25T01:36:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/1693291269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376bea0 msgr2=0x5626f376ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/1693291269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376bea0 0x5626f376ff50 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f2f00009cd0 tx=0x7f2f000095c0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/1693291269 shutdown_connections 2024-04-25T01:36:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/1693291269 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5626f3776570 0x5626f37ef270 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/1693291269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626f376bea0 0x5626f376ff50 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/1693291269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5626f376b580 0x5626f376fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2eedffb640 0 client.16608 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:36:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0f7e8740 1 --2- 172.21.15.28:0/1693291269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5626f37678a0 0x5626f377e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/1693291269 >> 172.21.15.28:0/1693291269 conn(0x5626f3773840 msgr2=0x5626f376def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/1693291269 shutdown_connections 2024-04-25T01:36:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0f7e8740 1 -- 172.21.15.28:0/1693291269 wait complete. 2024-04-25T01:36:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0f7e8740 20 client.16608 trim_cache size 0 max 16384 2024-04-25T01:36:23.826 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:36:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0c9f9640 1 -- 172.21.15.28:0/3311109297 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f2ef00a2d30 msgr2=0x7f2ef00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0c9f9640 1 --2- 172.21.15.28:0/3311109297 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f2ef00a2d30 0x7f2ef00a51f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2f00006010 tx=0x7f2f00009980 comp rx=0 tx=0).stop 2024-04-25T01:36:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0c9f9640 1 -- 172.21.15.28:0/3311109297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f08100610 msgr2=0x7f2f0810df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0c9f9640 1 --2- 172.21.15.28:0/3311109297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f08100610 0x7f2f0810df70 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f2ef800d950 tx=0x7f2ef800de10 comp rx=0 tx=0).stop 2024-04-25T01:36:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0c9f9640 1 -- 172.21.15.28:0/3311109297 shutdown_connections 2024-04-25T01:36:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0c9f9640 1 --2- 172.21.15.28:0/3311109297 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f2ef00a2d30 0x7f2ef00a51f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0c9f9640 1 --2- 172.21.15.28:0/3311109297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2f081066b0 0x7f2f0810e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0c9f9640 1 --2- 172.21.15.28:0/3311109297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2f08100fd0 0x7f2f0810e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0c9f9640 1 --2- 172.21.15.28:0/3311109297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f08100610 0x7f2f0810df70 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.840+0000 7f2f0c9f9640 1 -- 172.21.15.28:0/3311109297 >> 172.21.15.28:0/3311109297 conn(0x7f2f08077740 msgr2=0x7f2f080ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.841+0000 7f2f0c9f9640 1 -- 172.21.15.28:0/3311109297 shutdown_connections 2024-04-25T01:36:23.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.841+0000 7f2f0c9f9640 1 -- 172.21.15.28:0/3311109297 wait complete. 2024-04-25T01:36:23.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:36:23.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.957+0000 7fd97cde2640 1 -- 172.21.15.28:0/2117815610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd978069dc0 msgr2=0x7fd97806a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.957+0000 7fd97cde2640 1 --2- 172.21.15.28:0/2117815610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd978069dc0 0x7fd97806a240 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fd960009930 tx=0x7fd96002a420 comp rx=0 tx=0).stop 2024-04-25T01:36:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.957+0000 7fd97cde2640 1 -- 172.21.15.28:0/2117815610 shutdown_connections 2024-04-25T01:36:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.957+0000 7fd97cde2640 1 --2- 172.21.15.28:0/2117815610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd97806a780 0x7fd97810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.957+0000 7fd97cde2640 1 --2- 172.21.15.28:0/2117815610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd978069dc0 0x7fd97806a240 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.957+0000 7fd97cde2640 1 --2- 172.21.15.28:0/2117815610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd978107730 0x7fd978107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.957+0000 7fd97cde2640 1 -- 172.21.15.28:0/2117815610 >> 172.21.15.28:0/2117815610 conn(0x7fd9780776a0 msgr2=0x7fd978077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.957+0000 7fd97cde2640 1 -- 172.21.15.28:0/2117815610 shutdown_connections 2024-04-25T01:36:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.957+0000 7fd97cde2640 1 -- 172.21.15.28:0/2117815610 wait complete. 2024-04-25T01:36:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.958+0000 7fd97cde2640 1 Processor -- start 2024-04-25T01:36:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.958+0000 7fd97cde2640 1 -- start start 2024-04-25T01:36:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.958+0000 7fd97cde2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd978069dc0 0x7fd978198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.958+0000 7fd97cde2640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd97806a780 0x7fd978198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.958+0000 7fd97cde2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd978107730 0x7fd97819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.958+0000 7fd97cde2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd978111b40 con 0x7fd978069dc0 2024-04-25T01:36:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.958+0000 7fd97cde2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd9781119c0 con 0x7fd978107730 2024-04-25T01:36:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.958+0000 7fd97cde2640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd978111cc0 con 0x7fd97806a780 2024-04-25T01:36:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.958+0000 7fd97659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd978069dc0 0x7fd978198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.958+0000 7fd97659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd978069dc0 0x7fd978198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48344/0 (socket says 172.21.15.28:48344) 2024-04-25T01:36:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.958+0000 7fd97659c640 1 -- 172.21.15.28:0/1787802031 learned_addr learned my addr 172.21.15.28:0/1787802031 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.958+0000 7fd96dd9b640 1 --2- 172.21.15.28:0/1787802031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd97806a780 0x7fd978198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.958+0000 7fd976d9d640 1 --2- 172.21.15.28:0/1787802031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd978107730 0x7fd97819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.958+0000 7fd97659c640 1 -- 172.21.15.28:0/1787802031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd97806a780 msgr2=0x7fd978198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.958+0000 7fd97659c640 1 --2- 172.21.15.28:0/1787802031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd97806a780 0x7fd978198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.958+0000 7fd97659c640 1 -- 172.21.15.28:0/1787802031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd978107730 msgr2=0x7fd97819c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.958+0000 7fd97659c640 1 --2- 172.21.15.28:0/1787802031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd978107730 0x7fd97819c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.958+0000 7fd97659c640 1 -- 172.21.15.28:0/1787802031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd97819d030 con 0x7fd978069dc0 2024-04-25T01:36:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.958+0000 7fd976d9d640 1 --2- 172.21.15.28:0/1787802031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd978107730 0x7fd97819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.959+0000 7fd97659c640 1 --2- 172.21.15.28:0/1787802031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd978069dc0 0x7fd978198040 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fd96800d770 tx=0x7fd96800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.959+0000 7fd96ffff640 1 -- 172.21.15.28:0/1787802031 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd968017070 con 0x7fd978069dc0 2024-04-25T01:36:23.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.959+0000 7fd97cde2640 1 -- 172.21.15.28:0/1787802031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd97819d320 con 0x7fd978069dc0 2024-04-25T01:36:23.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.959+0000 7fd96ffff640 1 -- 172.21.15.28:0/1787802031 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd968005500 con 0x7fd978069dc0 2024-04-25T01:36:23.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.959+0000 7fd96ffff640 1 -- 172.21.15.28:0/1787802031 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd968004e00 con 0x7fd978069dc0 2024-04-25T01:36:23.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.959+0000 7fd97cde2640 1 -- 172.21.15.28:0/1787802031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9781a4bb0 con 0x7fd978069dc0 2024-04-25T01:36:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.961+0000 7fd96ffff640 1 -- 172.21.15.28:0/1787802031 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd968004fa0 con 0x7fd978069dc0 2024-04-25T01:36:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.961+0000 7fd97fbc5740 1 Processor -- start 2024-04-25T01:36:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd97fbc5740 1 -- start start 2024-04-25T01:36:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd96ffff640 1 --2- 172.21.15.28:0/1787802031 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd9640a2d00 0x7fd9640a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd97fbc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173191580 0x562173191960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd97fbc5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191ea0 0x56217319c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd97fbc5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56217319c570 0x56217319e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd97fbc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5621730f6670 con 0x562173191580 2024-04-25T01:36:23.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd97fbc5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56217309e970 con 0x56217319c570 2024-04-25T01:36:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd97fbc5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5621730f2630 con 0x562173191ea0 2024-04-25T01:36:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd97659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173191580 0x562173191960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd976d9d640 1 --2- 172.21.15.28:0/1787802031 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd9640a2d00 0x7fd9640a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd97659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173191580 0x562173191960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48346/0 (socket says 172.21.15.28:48346) 2024-04-25T01:36:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd97659c640 1 -- 172.21.15.28:0/456372250 learned_addr learned my addr 172.21.15.28:0/456372250 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd96ffff640 1 -- 172.21.15.28:0/1787802031 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fd9680cf1c0 con 0x7fd978069dc0 2024-04-25T01:36:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd976d9d640 1 --2- 172.21.15.28:0/456372250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56217319c570 0x56217319e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd96dd9b640 1 --2- 172.21.15.28:0/456372250 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191ea0 0x56217319c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd97659c640 1 -- 172.21.15.28:0/456372250 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191ea0 msgr2=0x56217319c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd97659c640 1 --2- 172.21.15.28:0/456372250 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191ea0 0x56217319c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd97659c640 1 -- 172.21.15.28:0/456372250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56217319c570 msgr2=0x56217319e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd97659c640 1 --2- 172.21.15.28:0/456372250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56217319c570 0x56217319e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd97659c640 1 -- 172.21.15.28:0/456372250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621730c2400 con 0x562173191580 2024-04-25T01:36:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd976d9d640 1 --2- 172.21.15.28:0/1787802031 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd9640a2d00 0x7fd9640a51c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd970005fd0 tx=0x7fd970014b70 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.962+0000 7fd97659c640 1 --2- 172.21.15.28:0/456372250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173191580 0x562173191960 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fd968097d10 tx=0x7fd968009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8285fb0d4a348a2f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.963+0000 7fd96cd99640 1 -- 172.21.15.28:0/456372250 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9680c09b0 con 0x562173191580 2024-04-25T01:36:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.963+0000 7fd96cd99640 1 -- 172.21.15.28:0/456372250 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9680046c0 con 0x562173191580 2024-04-25T01:36:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.963+0000 7fd96cd99640 1 -- 172.21.15.28:0/456372250 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd968016170 con 0x562173191580 2024-04-25T01:36:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.963+0000 7fd97fbc5740 1 -- 172.21.15.28:0/456372250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173191580 msgr2=0x562173191960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.963+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/456372250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173191580 0x562173191960 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fd968097d10 tx=0x7fd968009c30 comp rx=0 tx=0).stop 2024-04-25T01:36:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.963+0000 7fd97fbc5740 1 -- 172.21.15.28:0/456372250 shutdown_connections 2024-04-25T01:36:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.963+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/456372250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56217319c570 0x56217319e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.963+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/456372250 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191ea0 0x56217319c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.963+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/456372250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173191580 0x562173191960 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.963+0000 7fd97fbc5740 1 -- 172.21.15.28:0/456372250 >> 172.21.15.28:0/456372250 conn(0x56217318d8a0 msgr2=0x56217318dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.963+0000 7fd97fbc5740 1 -- 172.21.15.28:0/456372250 shutdown_connections 2024-04-25T01:36:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.964+0000 7fd97fbc5740 1 -- 172.21.15.28:0/456372250 wait complete. 2024-04-25T01:36:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.964+0000 7fd97fbc5740 1 -- start start 2024-04-25T01:36:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.964+0000 7fd97fbc5740 1 Processor -- start 2024-04-25T01:36:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.964+0000 7fd97fbc5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191580 0x56217319be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.964+0000 7fd97fbc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173191ea0 0x562173199150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.964+0000 7fd97fbc5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56217319c570 0x562173199690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.964+0000 7fd97fbc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56217319f360 con 0x562173191ea0 2024-04-25T01:36:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.964+0000 7fd97fbc5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5621730b2050 con 0x56217319c570 2024-04-25T01:36:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.964+0000 7fd97fbc5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56217319f4e0 con 0x562173191580 2024-04-25T01:36:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.964+0000 7fd97fbc5740 10 client.0 _ll_register_callbacks cb 0x562172e09be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.964+0000 7fd96dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173191ea0 0x562173199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.964+0000 7fd96dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173191ea0 0x562173199150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48354/0 (socket says 172.21.15.28:48354) 2024-04-25T01:36:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.964+0000 7fd96dd9b640 1 -- 172.21.15.28:0/3017932910 learned_addr learned my addr 172.21.15.28:0/3017932910 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.964+0000 7fd976d9d640 1 --2- 172.21.15.28:0/3017932910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56217319c570 0x562173199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.964+0000 7fd97659c640 1 --2- 172.21.15.28:0/3017932910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191580 0x56217319be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.964+0000 7fd96dd9b640 1 -- 172.21.15.28:0/3017932910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191580 msgr2=0x56217319be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.964+0000 7fd96dd9b640 1 --2- 172.21.15.28:0/3017932910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191580 0x56217319be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.965+0000 7fd96dd9b640 1 -- 172.21.15.28:0/3017932910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56217319c570 msgr2=0x562173199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.965+0000 7fd96dd9b640 1 --2- 172.21.15.28:0/3017932910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56217319c570 0x562173199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.965+0000 7fd96dd9b640 1 -- 172.21.15.28:0/3017932910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562173083dc0 con 0x562173191ea0 2024-04-25T01:36:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.965+0000 7fd96dd9b640 1 --2- 172.21.15.28:0/3017932910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173191ea0 0x562173199150 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fd96002a940 tx=0x7fd9600317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14199eee6bbc52b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.965+0000 7fd95dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.965+0000 7fd95dffb640 1 -- 172.21.15.28:0/3017932910 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd960040070 con 0x562173191ea0 2024-04-25T01:36:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.965+0000 7fd95dffb640 1 -- 172.21.15.28:0/3017932910 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fd960004450 con 0x562173191ea0 2024-04-25T01:36:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.965+0000 7fd97fbc5740 1 -- 172.21.15.28:0/3017932910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562173103bb0 con 0x562173191ea0 2024-04-25T01:36:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.965+0000 7fd95dffb640 1 -- 172.21.15.28:0/3017932910 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd96004e050 con 0x562173191ea0 2024-04-25T01:36:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.965+0000 7fd97fbc5740 10 client.16620 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.965+0000 7fd97fbc5740 1 -- 172.21.15.28:0/3017932910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562172e0a3f0 con 0x562173191ea0 2024-04-25T01:36:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.965+0000 7fd95dffb640 1 -- 172.21.15.28:0/3017932910 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x562173103bb0 con 0x562173191ea0 2024-04-25T01:36:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.965+0000 7fd95dffb640 1 -- 172.21.15.28:0/3017932910 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd960045690 con 0x562173191ea0 2024-04-25T01:36:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.965+0000 7fd97fbc5740 10 client.16620 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.965+0000 7fd97fbc5740 10 client.16620 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:36:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.965+0000 7fd97fbc5740 20 client.16620 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.965+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/3017932910 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5621731ac560 0x5621731ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.966+0000 7fd97fbc5740 4 client.16620 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.966+0000 7fd97fbc5740 1 -- 172.21.15.28:0/3017932910 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621730fdf00 con 0x5621731ac560 2024-04-25T01:36:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.966+0000 7fd97659c640 1 --2- 172.21.15.28:0/3017932910 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5621731ac560 0x5621731ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.966+0000 7fd95dffb640 10 client.16620 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:36:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.966+0000 7fd97659c640 1 --2- 172.21.15.28:0/3017932910 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5621731ac560 0x5621731ae9c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c434a9a62a0c55bb server_cookie=941ab7ec7bd9936 in_seq=0 out_seq=0 2024-04-25T01:36:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.967+0000 7fd95dffb640 1 -- 172.21.15.28:0/3017932910 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5621730fdf00 con 0x5621731ac560 2024-04-25T01:36:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.967+0000 7fd95dffb640 10 client.16620 handle_command_reply: tid=0 2024-04-25T01:36:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.967+0000 7fd97fbc5740 1 client.16620 shutdown 2024-04-25T01:36:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.967+0000 7fd97fbc5740 1 -- 172.21.15.28:0/3017932910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173191ea0 msgr2=0x562173199150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.967+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/3017932910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173191ea0 0x562173199150 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fd96002a940 tx=0x7fd9600317e0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.967+0000 7fd97fbc5740 1 -- 172.21.15.28:0/3017932910 shutdown_connections 2024-04-25T01:36:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.967+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/3017932910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56217319c570 0x562173199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.967+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/3017932910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173191ea0 0x562173199150 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.967+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/3017932910 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5621731ac560 0x5621731ae9c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.967+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/3017932910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191580 0x56217319be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.967+0000 7fd95dffb640 0 client.16620 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:36:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.967+0000 7fd97fbc5740 1 -- 172.21.15.28:0/3017932910 >> 172.21.15.28:0/3017932910 conn(0x56217318d8a0 msgr2=0x5621731936a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.967+0000 7fd97fbc5740 1 -- 172.21.15.28:0/3017932910 shutdown_connections 2024-04-25T01:36:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.967+0000 7fd97fbc5740 1 -- 172.21.15.28:0/3017932910 wait complete. 2024-04-25T01:36:23.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.967+0000 7fd97fbc5740 20 client.16620 trim_cache size 0 max 16384 2024-04-25T01:36:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.980+0000 7fd97fbc5740 1 Processor -- start 2024-04-25T01:36:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.980+0000 7fd97fbc5740 1 -- start start 2024-04-25T01:36:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd97fbc5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191580 0x5621731a81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd97fbc5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562173191ea0 0x56217318c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd97fbc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56217319c570 0x56217319b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd97fbc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9781aeaf0 con 0x56217319c570 2024-04-25T01:36:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd97fbc5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd9781b0630 con 0x562173191ea0 2024-04-25T01:36:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd97fbc5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd9781af1c0 con 0x562173191580 2024-04-25T01:36:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd976d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56217319c570 0x56217319b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd976d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56217319c570 0x56217319b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48370/0 (socket says 172.21.15.28:48370) 2024-04-25T01:36:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd976d9d640 1 -- 172.21.15.28:0/1672249047 learned_addr learned my addr 172.21.15.28:0/1672249047 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd96dd9b640 1 --2- 172.21.15.28:0/1672249047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562173191ea0 0x56217318c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd97659c640 1 --2- 172.21.15.28:0/1672249047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191580 0x5621731a81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd976d9d640 1 -- 172.21.15.28:0/1672249047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191580 msgr2=0x5621731a81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd976d9d640 1 --2- 172.21.15.28:0/1672249047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191580 0x5621731a81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd976d9d640 1 -- 172.21.15.28:0/1672249047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562173191ea0 msgr2=0x56217318c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd976d9d640 1 --2- 172.21.15.28:0/1672249047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562173191ea0 0x56217318c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd976d9d640 1 -- 172.21.15.28:0/1672249047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562172ed47d0 con 0x56217319c570 2024-04-25T01:36:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd96dd9b640 1 --2- 172.21.15.28:0/1672249047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562173191ea0 0x56217318c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd976d9d640 1 --2- 172.21.15.28:0/1672249047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56217319c570 0x56217319b990 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd970012c50 tx=0x7fd9700094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb5a127dbf829fca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd95ffff640 1 -- 172.21.15.28:0/1672249047 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd970026070 con 0x56217319c570 2024-04-25T01:36:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd95ffff640 1 -- 172.21.15.28:0/1672249047 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd970012c80 con 0x56217319c570 2024-04-25T01:36:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.981+0000 7fd95ffff640 1 -- 172.21.15.28:0/1672249047 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd970022660 con 0x56217319c570 2024-04-25T01:36:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.982+0000 7fd97fbc5740 1 -- 172.21.15.28:0/1672249047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56217319c570 msgr2=0x56217319b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.982+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/1672249047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56217319c570 0x56217319b990 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd970012c50 tx=0x7fd9700094c0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.982+0000 7fd97fbc5740 1 -- 172.21.15.28:0/1672249047 shutdown_connections 2024-04-25T01:36:23.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.982+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/1672249047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56217319c570 0x56217319b990 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.982+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/1672249047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562173191ea0 0x56217318c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.982+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/1672249047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191580 0x5621731a81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.982+0000 7fd97fbc5740 1 -- 172.21.15.28:0/1672249047 >> 172.21.15.28:0/1672249047 conn(0x56217318d8a0 msgr2=0x5621731a7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.982+0000 7fd97fbc5740 1 -- 172.21.15.28:0/1672249047 shutdown_connections 2024-04-25T01:36:23.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.982+0000 7fd97fbc5740 1 -- 172.21.15.28:0/1672249047 wait complete. 2024-04-25T01:36:23.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.982+0000 7fd97fbc5740 1 -- start start 2024-04-25T01:36:23.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.982+0000 7fd97fbc5740 1 Processor -- start 2024-04-25T01:36:23.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd97fbc5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191580 0x5621731a2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd97fbc5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562173191ea0 0x5621731a28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd97fbc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56217319c570 0x5621731a2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd97fbc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9781aff60 con 0x56217319c570 2024-04-25T01:36:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd97fbc5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd9781af890 con 0x562173191ea0 2024-04-25T01:36:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd97fbc5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd9781acad0 con 0x562173191580 2024-04-25T01:36:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd97fbc5740 10 client.0 _ll_register_callbacks cb 0x7fd960005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd976d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56217319c570 0x5621731a2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd976d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56217319c570 0x5621731a2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48380/0 (socket says 172.21.15.28:48380) 2024-04-25T01:36:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd976d9d640 1 -- 172.21.15.28:0/3467525590 learned_addr learned my addr 172.21.15.28:0/3467525590 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd97659c640 1 --2- 172.21.15.28:0/3467525590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191580 0x5621731a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd96dd9b640 1 --2- 172.21.15.28:0/3467525590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562173191ea0 0x5621731a28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd976d9d640 1 -- 172.21.15.28:0/3467525590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191580 msgr2=0x5621731a2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd976d9d640 1 --2- 172.21.15.28:0/3467525590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191580 0x5621731a2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd976d9d640 1 -- 172.21.15.28:0/3467525590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562173191ea0 msgr2=0x5621731a28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd976d9d640 1 --2- 172.21.15.28:0/3467525590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562173191ea0 0x5621731a28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd976d9d640 1 -- 172.21.15.28:0/3467525590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562173211d60 con 0x56217319c570 2024-04-25T01:36:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd96dd9b640 1 --2- 172.21.15.28:0/3467525590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562173191ea0 0x5621731a28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd976d9d640 1 --2- 172.21.15.28:0/3467525590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56217319c570 0x5621731a2df0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd97002bf60 tx=0x7fd97002bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac7680007ddd62f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd95dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd95dffb640 1 -- 172.21.15.28:0/3467525590 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd970026040 con 0x56217319c570 2024-04-25T01:36:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd95dffb640 1 -- 172.21.15.28:0/3467525590 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fd970004a40 con 0x56217319c570 2024-04-25T01:36:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.983+0000 7fd97fbc5740 1 -- 172.21.15.28:0/3467525590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd970012c80 con 0x56217319c570 2024-04-25T01:36:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.984+0000 7fd95dffb640 1 -- 172.21.15.28:0/3467525590 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd970004c60 con 0x56217319c570 2024-04-25T01:36:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.984+0000 7fd97fbc5740 10 client.16626 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.984+0000 7fd97fbc5740 1 -- 172.21.15.28:0/3467525590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562173213000 con 0x56217319c570 2024-04-25T01:36:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.984+0000 7fd95dffb640 1 -- 172.21.15.28:0/3467525590 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd970012c80 con 0x56217319c570 2024-04-25T01:36:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.984+0000 7fd95dffb640 1 -- 172.21.15.28:0/3467525590 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd97000f040 con 0x56217319c570 2024-04-25T01:36:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.984+0000 7fd97fbc5740 10 client.16626 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.984+0000 7fd97fbc5740 10 client.16626 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:36:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.984+0000 7fd97fbc5740 20 client.16626 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.984+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/3467525590 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5621732136f0 0x562173215b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.984+0000 7fd97fbc5740 4 client.16626 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.984+0000 7fd97fbc5740 1 -- 172.21.15.28:0/3467525590 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56217307b820 con 0x5621732136f0 2024-04-25T01:36:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.984+0000 7fd97659c640 1 --2- 172.21.15.28:0/3467525590 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5621732136f0 0x562173215b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.985+0000 7fd97659c640 1 --2- 172.21.15.28:0/3467525590 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5621732136f0 0x562173215b10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7a98328651db3460 server_cookie=b6ab3ba3ed06f1f8 in_seq=0 out_seq=0 2024-04-25T01:36:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.985+0000 7fd95dffb640 10 client.16626 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:36:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.985+0000 7fd95dffb640 1 -- 172.21.15.28:0/3467525590 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56217307b820 con 0x5621732136f0 2024-04-25T01:36:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.985+0000 7fd95dffb640 10 client.16626 handle_command_reply: tid=0 2024-04-25T01:36:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.985+0000 7fd97fbc5740 1 client.16626 shutdown 2024-04-25T01:36:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.985+0000 7fd97fbc5740 1 -- 172.21.15.28:0/3467525590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56217319c570 msgr2=0x5621731a2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.985+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/3467525590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56217319c570 0x5621731a2df0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd97002bf60 tx=0x7fd97002bcd0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.985+0000 7fd97fbc5740 1 -- 172.21.15.28:0/3467525590 shutdown_connections 2024-04-25T01:36:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.985+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/3467525590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56217319c570 0x5621731a2df0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.985+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/3467525590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562173191ea0 0x5621731a28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.985+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/3467525590 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5621732136f0 0x562173215b10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.985+0000 7fd97fbc5740 1 --2- 172.21.15.28:0/3467525590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562173191580 0x5621731a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.985+0000 7fd95dffb640 0 client.16626 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:36:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.985+0000 7fd97fbc5740 1 -- 172.21.15.28:0/3467525590 >> 172.21.15.28:0/3467525590 conn(0x56217318d8a0 msgr2=0x56217319a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.985+0000 7fd97fbc5740 1 -- 172.21.15.28:0/3467525590 shutdown_connections 2024-04-25T01:36:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.985+0000 7fd97fbc5740 1 -- 172.21.15.28:0/3467525590 wait complete. 2024-04-25T01:36:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.985+0000 7fd97fbc5740 20 client.16626 trim_cache size 0 max 16384 2024-04-25T01:36:23.971 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:36:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.986+0000 7fd97cde2640 1 -- 172.21.15.28:0/1787802031 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd9640a2d00 msgr2=0x7fd9640a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.986+0000 7fd97cde2640 1 --2- 172.21.15.28:0/1787802031 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd9640a2d00 0x7fd9640a51c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd970005fd0 tx=0x7fd970014b70 comp rx=0 tx=0).stop 2024-04-25T01:36:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.986+0000 7fd97cde2640 1 -- 172.21.15.28:0/1787802031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd978069dc0 msgr2=0x7fd978198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.986+0000 7fd97cde2640 1 --2- 172.21.15.28:0/1787802031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd978069dc0 0x7fd978198040 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fd96800d770 tx=0x7fd96800dc30 comp rx=0 tx=0).stop 2024-04-25T01:36:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.986+0000 7fd97cde2640 1 -- 172.21.15.28:0/1787802031 shutdown_connections 2024-04-25T01:36:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.986+0000 7fd97cde2640 1 --2- 172.21.15.28:0/1787802031 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd9640a2d00 0x7fd9640a51c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.986+0000 7fd97cde2640 1 --2- 172.21.15.28:0/1787802031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd978107730 0x7fd97819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.986+0000 7fd97cde2640 1 --2- 172.21.15.28:0/1787802031 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd97806a780 0x7fd978198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.986+0000 7fd97cde2640 1 --2- 172.21.15.28:0/1787802031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd978069dc0 0x7fd978198040 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.986+0000 7fd97cde2640 1 -- 172.21.15.28:0/1787802031 >> 172.21.15.28:0/1787802031 conn(0x7fd9780776a0 msgr2=0x7fd9780fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.986+0000 7fd97cde2640 1 -- 172.21.15.28:0/1787802031 shutdown_connections 2024-04-25T01:36:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:23.986+0000 7fd97cde2640 1 -- 172.21.15.28:0/1787802031 wait complete. 2024-04-25T01:36:23.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:36:24.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.101+0000 7f990a253640 1 -- 172.21.15.28:0/2671243664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904069dc0 msgr2=0x7f990406a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:24.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.101+0000 7f990a253640 1 --2- 172.21.15.28:0/2671243664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904069dc0 0x7f990406a240 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f98f8009930 tx=0x7f98f802a420 comp rx=0 tx=0).stop 2024-04-25T01:36:24.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.102+0000 7f990a253640 1 -- 172.21.15.28:0/2671243664 shutdown_connections 2024-04-25T01:36:24.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.102+0000 7f990a253640 1 --2- 172.21.15.28:0/2671243664 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f990406a780 0x7f990410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.102+0000 7f990a253640 1 --2- 172.21.15.28:0/2671243664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904069dc0 0x7f990406a240 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.102+0000 7f990a253640 1 --2- 172.21.15.28:0/2671243664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9904107730 0x7f9904107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.102+0000 7f990a253640 1 -- 172.21.15.28:0/2671243664 >> 172.21.15.28:0/2671243664 conn(0x7f99040776a0 msgr2=0x7f9904077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:24.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.102+0000 7f990a253640 1 -- 172.21.15.28:0/2671243664 shutdown_connections 2024-04-25T01:36:24.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.102+0000 7f990a253640 1 -- 172.21.15.28:0/2671243664 wait complete. 2024-04-25T01:36:24.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.102+0000 7f990a253640 1 Processor -- start 2024-04-25T01:36:24.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.102+0000 7f990a253640 1 -- start start 2024-04-25T01:36:24.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.102+0000 7f990a253640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9904069dc0 0x7f9904197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.102+0000 7f990a253640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f990406a780 0x7f99041984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.102+0000 7f990a253640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904107730 0x7f990419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.102+0000 7f990a253640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9904111b40 con 0x7f9904107730 2024-04-25T01:36:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.102+0000 7f990a253640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f99041119c0 con 0x7f990406a780 2024-04-25T01:36:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.102+0000 7f990a253640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9904111cc0 con 0x7f9904069dc0 2024-04-25T01:36:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.103+0000 7f9903fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904107730 0x7f990419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.103+0000 7f9903fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904107730 0x7f990419c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48392/0 (socket says 172.21.15.28:48392) 2024-04-25T01:36:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.103+0000 7f9903fff640 1 -- 172.21.15.28:0/4004045860 learned_addr learned my addr 172.21.15.28:0/4004045860 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.103+0000 7f9902ffd640 1 --2- 172.21.15.28:0/4004045860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f990406a780 0x7f99041984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.103+0000 7f99037fe640 1 --2- 172.21.15.28:0/4004045860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9904069dc0 0x7f9904197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.103+0000 7f9903fff640 1 -- 172.21.15.28:0/4004045860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9904069dc0 msgr2=0x7f9904197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.103+0000 7f9903fff640 1 --2- 172.21.15.28:0/4004045860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9904069dc0 0x7f9904197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.103+0000 7f9903fff640 1 -- 172.21.15.28:0/4004045860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f990406a780 msgr2=0x7f99041984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.103+0000 7f9903fff640 1 --2- 172.21.15.28:0/4004045860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f990406a780 0x7f99041984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.103+0000 7f9903fff640 1 -- 172.21.15.28:0/4004045860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990419cf90 con 0x7f9904107730 2024-04-25T01:36:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.103+0000 7f99037fe640 1 --2- 172.21.15.28:0/4004045860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9904069dc0 0x7f9904197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.103+0000 7f9902ffd640 1 --2- 172.21.15.28:0/4004045860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f990406a780 0x7f99041984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.103+0000 7f9903fff640 1 --2- 172.21.15.28:0/4004045860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904107730 0x7f990419c860 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f98fc00c970 tx=0x7f98fc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.103+0000 7f9900ff9640 1 -- 172.21.15.28:0/4004045860 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc007bb0 con 0x7f9904107730 2024-04-25T01:36:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.103+0000 7f9900ff9640 1 -- 172.21.15.28:0/4004045860 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98fc007d50 con 0x7f9904107730 2024-04-25T01:36:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.103+0000 7f9900ff9640 1 -- 172.21.15.28:0/4004045860 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc018800 con 0x7f9904107730 2024-04-25T01:36:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.103+0000 7f990a253640 1 -- 172.21.15.28:0/4004045860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f990419d280 con 0x7f9904107730 2024-04-25T01:36:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.103+0000 7f990a253640 1 -- 172.21.15.28:0/4004045860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99041a4b10 con 0x7f9904107730 2024-04-25T01:36:24.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.105+0000 7f9900ff9640 1 -- 172.21.15.28:0/4004045860 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f98fc00fa50 con 0x7f9904107730 2024-04-25T01:36:24.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f990d035740 1 Processor -- start 2024-04-25T01:36:24.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f990d035740 1 -- start start 2024-04-25T01:36:24.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f990d035740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb979580 0x55cbfb979960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f990d035740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb979ea0 0x55cbfb984030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f990d035740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb984570 0x55cbfb986960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f990d035740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cbfb8de670 con 0x55cbfb979580 2024-04-25T01:36:24.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f990d035740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cbfb886970 con 0x55cbfb979ea0 2024-04-25T01:36:24.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f990d035740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55cbfb8da630 con 0x55cbfb984570 2024-04-25T01:36:24.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f9902ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb979580 0x55cbfb979960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f9902ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb979580 0x55cbfb979960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48408/0 (socket says 172.21.15.28:48408) 2024-04-25T01:36:24.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f9902ffd640 1 -- 172.21.15.28:0/3893646730 learned_addr learned my addr 172.21.15.28:0/3893646730 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:24.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f99037fe640 1 --2- 172.21.15.28:0/3893646730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb979ea0 0x55cbfb984030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f9900ff9640 1 --2- 172.21.15.28:0/4004045860 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f98ec0a2d00 0x7f98ec0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f9902ffd640 1 -- 172.21.15.28:0/3893646730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb984570 msgr2=0x55cbfb986960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:24.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f9902ffd640 1 --2- 172.21.15.28:0/3893646730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb984570 0x55cbfb986960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f9902ffd640 1 -- 172.21.15.28:0/3893646730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb979ea0 msgr2=0x55cbfb984030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f9902ffd640 1 --2- 172.21.15.28:0/3893646730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb979ea0 0x55cbfb984030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f9902ffd640 1 -- 172.21.15.28:0/3893646730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbfb8aa400 con 0x55cbfb979580 2024-04-25T01:36:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f99037fe640 1 --2- 172.21.15.28:0/3893646730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb979ea0 0x55cbfb984030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.106+0000 7f9903fff640 1 --2- 172.21.15.28:0/4004045860 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f98ec0a2d00 0x7f98ec0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.107+0000 7f9902ffd640 1 --2- 172.21.15.28:0/3893646730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb979580 0x55cbfb979960 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f98f802a940 tx=0x7f98f8032f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd690c9f21d66626 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.107+0000 7f9900ff9640 1 -- 172.21.15.28:0/4004045860 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f98fc0cf870 con 0x7f9904107730 2024-04-25T01:36:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.107+0000 7f98e9ffb640 1 -- 172.21.15.28:0/3893646730 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8035050 con 0x55cbfb979580 2024-04-25T01:36:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.107+0000 7f98e9ffb640 1 -- 172.21.15.28:0/3893646730 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98f8031870 con 0x55cbfb979580 2024-04-25T01:36:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.107+0000 7f98e9ffb640 1 -- 172.21.15.28:0/3893646730 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f80048e0 con 0x55cbfb979580 2024-04-25T01:36:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.107+0000 7f990d035740 1 -- 172.21.15.28:0/3893646730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb979580 msgr2=0x55cbfb979960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.107+0000 7f9903fff640 1 --2- 172.21.15.28:0/4004045860 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f98ec0a2d00 0x7f98ec0a51c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0cd280 tx=0x7f98fc01baf0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.107+0000 7f990d035740 1 --2- 172.21.15.28:0/3893646730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb979580 0x55cbfb979960 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f98f802a940 tx=0x7f98f8032f00 comp rx=0 tx=0).stop 2024-04-25T01:36:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.107+0000 7f990d035740 1 -- 172.21.15.28:0/3893646730 shutdown_connections 2024-04-25T01:36:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.107+0000 7f990d035740 1 --2- 172.21.15.28:0/3893646730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb984570 0x55cbfb986960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.107+0000 7f990d035740 1 --2- 172.21.15.28:0/3893646730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb979ea0 0x55cbfb984030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.107+0000 7f990d035740 1 --2- 172.21.15.28:0/3893646730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb979580 0x55cbfb979960 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.107+0000 7f990d035740 1 -- 172.21.15.28:0/3893646730 >> 172.21.15.28:0/3893646730 conn(0x55cbfb9758a0 msgr2=0x55cbfb975c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.108+0000 7f990d035740 1 -- 172.21.15.28:0/3893646730 shutdown_connections 2024-04-25T01:36:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.108+0000 7f990d035740 1 -- 172.21.15.28:0/3893646730 wait complete. 2024-04-25T01:36:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.108+0000 7f990d035740 1 -- start start 2024-04-25T01:36:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.108+0000 7f990d035740 1 Processor -- start 2024-04-25T01:36:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.108+0000 7f990d035740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb979580 0x55cbfb983e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.108+0000 7f990d035740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb979ea0 0x55cbfb9811d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.108+0000 7f990d035740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb984570 0x55cbfb981710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.108+0000 7f990d035740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cbfb987300 con 0x55cbfb984570 2024-04-25T01:36:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.108+0000 7f990d035740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cbfb89a050 con 0x55cbfb979580 2024-04-25T01:36:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.108+0000 7f990d035740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55cbfb987480 con 0x55cbfb979ea0 2024-04-25T01:36:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.108+0000 7f990d035740 10 client.0 _ll_register_callbacks cb 0x55cbfb5f1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.108+0000 7f9902ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb984570 0x55cbfb981710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.109+0000 7f99037fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb979ea0 0x55cbfb9811d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.109+0000 7f9902ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb984570 0x55cbfb981710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48422/0 (socket says 172.21.15.28:48422) 2024-04-25T01:36:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.109+0000 7f9902ffd640 1 -- 172.21.15.28:0/2961883175 learned_addr learned my addr 172.21.15.28:0/2961883175 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.109+0000 7f9902ffd640 1 --2- 172.21.15.28:0/2961883175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb979580 0x55cbfb983e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.109+0000 7f99037fe640 1 -- 172.21.15.28:0/2961883175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb979580 msgr2=0x55cbfb983e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.109+0000 7f99037fe640 1 --2- 172.21.15.28:0/2961883175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb979580 0x55cbfb983e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.109+0000 7f99037fe640 1 -- 172.21.15.28:0/2961883175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb984570 msgr2=0x55cbfb981710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.109+0000 7f99037fe640 1 --2- 172.21.15.28:0/2961883175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb984570 0x55cbfb981710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.109+0000 7f99037fe640 1 -- 172.21.15.28:0/2961883175 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbfb86bdc0 con 0x55cbfb979ea0 2024-04-25T01:36:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.109+0000 7f9902ffd640 1 --2- 172.21.15.28:0/2961883175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb984570 0x55cbfb981710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:36:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.109+0000 7f99037fe640 1 --2- 172.21.15.28:0/2961883175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb979ea0 0x55cbfb9811d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f98f4005fd0 tx=0x7f98f400faf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da5a8c55f4000353 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.109+0000 7f98e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-25T01:36:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.109+0000 7f98e2ffd640 1 -- 172.21.15.28:0/2961883175 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f4003b70 con 0x55cbfb979ea0 2024-04-25T01:36:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.109+0000 7f990d035740 1 -- 172.21.15.28:0/2961883175 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98f8031870 con 0x55cbfb979ea0 2024-04-25T01:36:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.109+0000 7f98e2ffd640 1 -- 172.21.15.28:0/2961883175 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f98f4003d10 con 0x55cbfb979ea0 2024-04-25T01:36:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.109+0000 7f98e2ffd640 1 -- 172.21.15.28:0/2961883175 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f98f4011710 con 0x55cbfb979ea0 2024-04-25T01:36:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.109+0000 7f990d035740 10 client.25942 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.110+0000 7f990d035740 1 -- 172.21.15.28:0/2961883175 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cbfb5f23f0 con 0x55cbfb979ea0 2024-04-25T01:36:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.110+0000 7f98e2ffd640 1 -- 172.21.15.28:0/2961883175 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98f8031870 con 0x55cbfb979ea0 2024-04-25T01:36:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.110+0000 7f98e2ffd640 1 -- 172.21.15.28:0/2961883175 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f98f40045e0 con 0x55cbfb979ea0 2024-04-25T01:36:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.110+0000 7f990d035740 10 client.25942 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.110+0000 7f990d035740 10 client.25942 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:36:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.110+0000 7f990d035740 20 client.25942 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.110+0000 7f990d035740 1 --2- 172.21.15.28:0/2961883175 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55cbfb994500 0x55cbfb996940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.110+0000 7f990d035740 4 client.25942 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.110+0000 7f990d035740 1 -- 172.21.15.28:0/2961883175 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbfb8ebbb0 con 0x55cbfb994500 2024-04-25T01:36:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.110+0000 7f9902ffd640 1 --2- 172.21.15.28:0/2961883175 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55cbfb994500 0x55cbfb996940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.111+0000 7f9902ffd640 1 --2- 172.21.15.28:0/2961883175 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55cbfb994500 0x55cbfb996940 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ef0540a9c79f7674 server_cookie=98187bdffe79d64e in_seq=0 out_seq=0 2024-04-25T01:36:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.111+0000 7f98e2ffd640 10 client.25942 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:36:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.111+0000 7f98e2ffd640 1 -- 172.21.15.28:0/2961883175 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cbfb8ebbb0 con 0x55cbfb994500 2024-04-25T01:36:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.111+0000 7f98e2ffd640 10 client.25942 handle_command_reply: tid=0 2024-04-25T01:36:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.111+0000 7f990d035740 1 client.25942 shutdown 2024-04-25T01:36:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.112+0000 7f990d035740 1 -- 172.21.15.28:0/2961883175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb979ea0 msgr2=0x55cbfb9811d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.112+0000 7f990d035740 1 --2- 172.21.15.28:0/2961883175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb979ea0 0x55cbfb9811d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f98f4005fd0 tx=0x7f98f400faf0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.112+0000 7f990d035740 1 -- 172.21.15.28:0/2961883175 shutdown_connections 2024-04-25T01:36:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.112+0000 7f990d035740 1 --2- 172.21.15.28:0/2961883175 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55cbfb994500 0x55cbfb996940 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.112+0000 7f990d035740 1 --2- 172.21.15.28:0/2961883175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb984570 0x55cbfb981710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.112+0000 7f98e2ffd640 0 client.25942 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:36:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.112+0000 7f990d035740 1 --2- 172.21.15.28:0/2961883175 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb979ea0 0x55cbfb9811d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.112+0000 7f990d035740 1 --2- 172.21.15.28:0/2961883175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb979580 0x55cbfb983e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.112+0000 7f990d035740 1 -- 172.21.15.28:0/2961883175 >> 172.21.15.28:0/2961883175 conn(0x55cbfb9758a0 msgr2=0x55cbfb974b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.112+0000 7f990d035740 1 -- 172.21.15.28:0/2961883175 shutdown_connections 2024-04-25T01:36:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.112+0000 7f990d035740 1 -- 172.21.15.28:0/2961883175 wait complete. 2024-04-25T01:36:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.112+0000 7f990d035740 20 client.25942 trim_cache size 0 max 16384 2024-04-25T01:36:24.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.125+0000 7f990d035740 1 Processor -- start 2024-04-25T01:36:24.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.125+0000 7f990d035740 1 -- start start 2024-04-25T01:36:24.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.125+0000 7f990d035740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb979580 0x55cbfb983900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.125+0000 7f990d035740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb979ea0 0x55cbfb983e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.125+0000 7f990d035740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb984570 0x55cbfb99c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.125+0000 7f990d035740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f99041ae890 con 0x55cbfb979ea0 2024-04-25T01:36:24.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.125+0000 7f990d035740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f99041b03d0 con 0x55cbfb984570 2024-04-25T01:36:24.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.125+0000 7f990d035740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f99041aef60 con 0x55cbfb979580 2024-04-25T01:36:24.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.125+0000 7f99037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb979ea0 0x55cbfb983e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.125+0000 7f99037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb979ea0 0x55cbfb983e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48436/0 (socket says 172.21.15.28:48436) 2024-04-25T01:36:24.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.125+0000 7f99037fe640 1 -- 172.21.15.28:0/1417849943 learned_addr learned my addr 172.21.15.28:0/1417849943 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:24.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.126+0000 7f9902ffd640 1 --2- 172.21.15.28:0/1417849943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb984570 0x55cbfb99c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.126+0000 7f9902ffd640 1 --2- 172.21.15.28:0/1417849943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb979580 0x55cbfb983900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.126+0000 7f99037fe640 1 -- 172.21.15.28:0/1417849943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb979580 msgr2=0x55cbfb983900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:24.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.126+0000 7f99037fe640 1 --2- 172.21.15.28:0/1417849943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb979580 0x55cbfb983900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.126+0000 7f99037fe640 1 -- 172.21.15.28:0/1417849943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb984570 msgr2=0x55cbfb99c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:24.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.126+0000 7f99037fe640 1 --2- 172.21.15.28:0/1417849943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb984570 0x55cbfb99c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.126+0000 7f99037fe640 1 -- 172.21.15.28:0/1417849943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbfb979cd0 con 0x55cbfb979ea0 2024-04-25T01:36:24.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.126+0000 7f9902ffd640 1 --2- 172.21.15.28:0/1417849943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb979580 0x55cbfb983900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:24.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.126+0000 7f99037fe640 1 --2- 172.21.15.28:0/1417849943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb979ea0 0x55cbfb983e40 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f98f4002230 tx=0x7f98f4009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8869029eaab434c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:24.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.126+0000 7f98e97fa640 1 -- 172.21.15.28:0/1417849943 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f4009e50 con 0x55cbfb979ea0 2024-04-25T01:36:24.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.126+0000 7f98e97fa640 1 -- 172.21.15.28:0/1417849943 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98f4010720 con 0x55cbfb979ea0 2024-04-25T01:36:24.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.126+0000 7f98e97fa640 1 -- 172.21.15.28:0/1417849943 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f40052a0 con 0x55cbfb979ea0 2024-04-25T01:36:24.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.127+0000 7f990d035740 1 -- 172.21.15.28:0/1417849943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb979ea0 msgr2=0x55cbfb983e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:24.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.127+0000 7f990d035740 1 --2- 172.21.15.28:0/1417849943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb979ea0 0x55cbfb983e40 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f98f4002230 tx=0x7f98f4009ca0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.127+0000 7f990d035740 1 -- 172.21.15.28:0/1417849943 shutdown_connections 2024-04-25T01:36:24.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.127+0000 7f990d035740 1 --2- 172.21.15.28:0/1417849943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb984570 0x55cbfb99c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.127+0000 7f990d035740 1 --2- 172.21.15.28:0/1417849943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb979ea0 0x55cbfb983e40 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.127+0000 7f990d035740 1 --2- 172.21.15.28:0/1417849943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb979580 0x55cbfb983900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.127+0000 7f990d035740 1 -- 172.21.15.28:0/1417849943 >> 172.21.15.28:0/1417849943 conn(0x55cbfb9758a0 msgr2=0x55cbfb97faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:24.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.127+0000 7f990d035740 1 -- 172.21.15.28:0/1417849943 shutdown_connections 2024-04-25T01:36:24.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.127+0000 7f990d035740 1 -- 172.21.15.28:0/1417849943 wait complete. 2024-04-25T01:36:24.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.127+0000 7f990d035740 1 -- start start 2024-04-25T01:36:24.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.127+0000 7f990d035740 1 Processor -- start 2024-04-25T01:36:24.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.127+0000 7f990d035740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb979580 0x55cbfb98c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.127+0000 7f990d035740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb979ea0 0x55cbfb989cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.127+0000 7f990d035740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb984570 0x55cbfb98a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.127+0000 7f990d035740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f99041afd00 con 0x55cbfb984570 2024-04-25T01:36:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.127+0000 7f990d035740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f99041af630 con 0x55cbfb979ea0 2024-04-25T01:36:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.127+0000 7f990d035740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f99041ac870 con 0x55cbfb979580 2024-04-25T01:36:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.127+0000 7f990d035740 10 client.0 _ll_register_callbacks cb 0x7f98f4009630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.128+0000 7f9902ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb984570 0x55cbfb98a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.128+0000 7f9902ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb984570 0x55cbfb98a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48450/0 (socket says 172.21.15.28:48450) 2024-04-25T01:36:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.128+0000 7f9902ffd640 1 -- 172.21.15.28:0/3207296585 learned_addr learned my addr 172.21.15.28:0/3207296585 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.128+0000 7f9902ffd640 1 --2- 172.21.15.28:0/3207296585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb979580 0x55cbfb98c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.128+0000 7f99037fe640 1 --2- 172.21.15.28:0/3207296585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb979ea0 0x55cbfb989cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.128+0000 7f9902ffd640 1 -- 172.21.15.28:0/3207296585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb979580 msgr2=0x55cbfb98c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.128+0000 7f9902ffd640 1 --2- 172.21.15.28:0/3207296585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb979580 0x55cbfb98c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.128+0000 7f9902ffd640 1 -- 172.21.15.28:0/3207296585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb979ea0 msgr2=0x55cbfb989cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.128+0000 7f9902ffd640 1 --2- 172.21.15.28:0/3207296585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb979ea0 0x55cbfb989cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.128+0000 7f9902ffd640 1 -- 172.21.15.28:0/3207296585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbfb6bc7d0 con 0x55cbfb984570 2024-04-25T01:36:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.128+0000 7f9902ffd640 1 --2- 172.21.15.28:0/3207296585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb984570 0x55cbfb98a210 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f98f8005500 tx=0x7f98f8032f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3344a1f2f0b8ee6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.128+0000 7f98e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.128+0000 7f98e2ffd640 1 -- 172.21.15.28:0/3207296585 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f805b070 con 0x55cbfb984570 2024-04-25T01:36:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.128+0000 7f98e2ffd640 1 -- 172.21.15.28:0/3207296585 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f98f8054ae0 con 0x55cbfb984570 2024-04-25T01:36:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.128+0000 7f990d035740 1 -- 172.21.15.28:0/3207296585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98f4010720 con 0x55cbfb984570 2024-04-25T01:36:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.128+0000 7f98e2ffd640 1 -- 172.21.15.28:0/3207296585 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f98f8035040 con 0x55cbfb984570 2024-04-25T01:36:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.128+0000 7f990d035740 10 client.16644 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.128+0000 7f990d035740 1 -- 172.21.15.28:0/3207296585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cbfb9fa710 con 0x55cbfb984570 2024-04-25T01:36:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.129+0000 7f98e2ffd640 1 -- 172.21.15.28:0/3207296585 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98f4010720 con 0x55cbfb984570 2024-04-25T01:36:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.129+0000 7f98e2ffd640 1 -- 172.21.15.28:0/3207296585 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f98f805e020 con 0x55cbfb984570 2024-04-25T01:36:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.129+0000 7f990d035740 10 client.16644 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.129+0000 7f990d035740 10 client.16644 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:36:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.129+0000 7f990d035740 20 client.16644 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.129+0000 7f990d035740 1 --2- 172.21.15.28:0/3207296585 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55cbfb9faaa0 0x55cbfb9fce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.129+0000 7f990d035740 4 client.16644 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.129+0000 7f990d035740 1 -- 172.21.15.28:0/3207296585 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cbfb8e5f00 con 0x55cbfb9faaa0 2024-04-25T01:36:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.129+0000 7f99037fe640 1 --2- 172.21.15.28:0/3207296585 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55cbfb9faaa0 0x55cbfb9fce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.129+0000 7f99037fe640 1 --2- 172.21.15.28:0/3207296585 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55cbfb9faaa0 0x55cbfb9fce90 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e3f24930df80b578 server_cookie=728afb233ab18080 in_seq=0 out_seq=0 2024-04-25T01:36:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.129+0000 7f98e2ffd640 10 client.16644 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:36:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.130+0000 7f98e2ffd640 1 -- 172.21.15.28:0/3207296585 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cbfb8e5f00 con 0x55cbfb9faaa0 2024-04-25T01:36:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.130+0000 7f98e2ffd640 10 client.16644 handle_command_reply: tid=0 2024-04-25T01:36:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.130+0000 7f990d035740 1 client.16644 shutdown 2024-04-25T01:36:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.130+0000 7f990d035740 1 -- 172.21.15.28:0/3207296585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb984570 msgr2=0x55cbfb98a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.130+0000 7f990d035740 1 --2- 172.21.15.28:0/3207296585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb984570 0x55cbfb98a210 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f98f8005500 tx=0x7f98f8032f00 comp rx=0 tx=0).stop 2024-04-25T01:36:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.130+0000 7f990d035740 1 -- 172.21.15.28:0/3207296585 shutdown_connections 2024-04-25T01:36:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.130+0000 7f990d035740 1 --2- 172.21.15.28:0/3207296585 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55cbfb9faaa0 0x55cbfb9fce90 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.130+0000 7f990d035740 1 --2- 172.21.15.28:0/3207296585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbfb984570 0x55cbfb98a210 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.130+0000 7f990d035740 1 --2- 172.21.15.28:0/3207296585 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbfb979ea0 0x55cbfb989cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.130+0000 7f98e2ffd640 0 client.16644 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:36:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.130+0000 7f990d035740 1 --2- 172.21.15.28:0/3207296585 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbfb979580 0x55cbfb98c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.130+0000 7f990d035740 1 -- 172.21.15.28:0/3207296585 >> 172.21.15.28:0/3207296585 conn(0x55cbfb9758a0 msgr2=0x55cbfb9742b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:24.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.130+0000 7f990d035740 1 -- 172.21.15.28:0/3207296585 shutdown_connections 2024-04-25T01:36:24.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.130+0000 7f990d035740 1 -- 172.21.15.28:0/3207296585 wait complete. 2024-04-25T01:36:24.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.130+0000 7f990d035740 20 client.16644 trim_cache size 0 max 16384 2024-04-25T01:36:24.115 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:36:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.130+0000 7f990a253640 1 -- 172.21.15.28:0/4004045860 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f98ec0a2d00 msgr2=0x7f98ec0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.130+0000 7f990a253640 1 --2- 172.21.15.28:0/4004045860 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f98ec0a2d00 0x7f98ec0a51c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0cd280 tx=0x7f98fc01baf0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.130+0000 7f990a253640 1 -- 172.21.15.28:0/4004045860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904107730 msgr2=0x7f990419c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.130+0000 7f990a253640 1 --2- 172.21.15.28:0/4004045860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904107730 0x7f990419c860 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f98fc00c970 tx=0x7f98fc00ce30 comp rx=0 tx=0).stop 2024-04-25T01:36:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.131+0000 7f990a253640 1 -- 172.21.15.28:0/4004045860 shutdown_connections 2024-04-25T01:36:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.131+0000 7f990a253640 1 --2- 172.21.15.28:0/4004045860 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f98ec0a2d00 0x7f98ec0a51c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.131+0000 7f990a253640 1 --2- 172.21.15.28:0/4004045860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9904107730 0x7f990419c860 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.131+0000 7f990a253640 1 --2- 172.21.15.28:0/4004045860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f990406a780 0x7f99041984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.131+0000 7f990a253640 1 --2- 172.21.15.28:0/4004045860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9904069dc0 0x7f9904197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.131+0000 7f990a253640 1 -- 172.21.15.28:0/4004045860 >> 172.21.15.28:0/4004045860 conn(0x7f99040776a0 msgr2=0x7f99040fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:24.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.131+0000 7f990a253640 1 -- 172.21.15.28:0/4004045860 shutdown_connections 2024-04-25T01:36:24.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.131+0000 7f990a253640 1 -- 172.21.15.28:0/4004045860 wait complete. 2024-04-25T01:36:24.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:36:24.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.246+0000 7f8a62b9e640 1 -- 172.21.15.28:0/3966425928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a5c069dc0 msgr2=0x7f8a5c06a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:24.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.246+0000 7f8a62b9e640 1 --2- 172.21.15.28:0/3966425928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a5c069dc0 0x7f8a5c06a240 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f8a48009930 tx=0x7f8a4802a420 comp rx=0 tx=0).stop 2024-04-25T01:36:24.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.246+0000 7f8a62b9e640 1 -- 172.21.15.28:0/3966425928 shutdown_connections 2024-04-25T01:36:24.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.246+0000 7f8a62b9e640 1 --2- 172.21.15.28:0/3966425928 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a5c06a780 0x7f8a5c10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.246+0000 7f8a62b9e640 1 --2- 172.21.15.28:0/3966425928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a5c069dc0 0x7f8a5c06a240 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.246+0000 7f8a62b9e640 1 --2- 172.21.15.28:0/3966425928 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a5c107730 0x7f8a5c107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.246+0000 7f8a62b9e640 1 -- 172.21.15.28:0/3966425928 >> 172.21.15.28:0/3966425928 conn(0x7f8a5c0776a0 msgr2=0x7f8a5c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:24.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.246+0000 7f8a62b9e640 1 -- 172.21.15.28:0/3966425928 shutdown_connections 2024-04-25T01:36:24.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.246+0000 7f8a62b9e640 1 -- 172.21.15.28:0/3966425928 wait complete. 2024-04-25T01:36:24.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a62b9e640 1 Processor -- start 2024-04-25T01:36:24.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a62b9e640 1 -- start start 2024-04-25T01:36:24.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a62b9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a5c069dc0 0x7f8a5c197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a62b9e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a5c06a780 0x7f8a5c1984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a62b9e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a5c107730 0x7f8a5c19c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a62b9e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c111b90 con 0x7f8a5c069dc0 2024-04-25T01:36:24.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a62b9e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c111a10 con 0x7f8a5c107730 2024-04-25T01:36:24.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a62b9e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c111d10 con 0x7f8a5c06a780 2024-04-25T01:36:24.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a6093a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a5c069dc0 0x7f8a5c197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a6093a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a5c069dc0 0x7f8a5c197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48468/0 (socket says 172.21.15.28:48468) 2024-04-25T01:36:24.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a6093a640 1 -- 172.21.15.28:0/476547677 learned_addr learned my addr 172.21.15.28:0/476547677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:24.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a5bfff640 1 --2- 172.21.15.28:0/476547677 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a5c06a780 0x7f8a5c1984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a6113b640 1 --2- 172.21.15.28:0/476547677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a5c107730 0x7f8a5c19c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a6093a640 1 -- 172.21.15.28:0/476547677 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a5c06a780 msgr2=0x7f8a5c1984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:24.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a6093a640 1 --2- 172.21.15.28:0/476547677 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a5c06a780 0x7f8a5c1984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a6093a640 1 -- 172.21.15.28:0/476547677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a5c107730 msgr2=0x7f8a5c19c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:24.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a6093a640 1 --2- 172.21.15.28:0/476547677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a5c107730 0x7f8a5c19c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a6093a640 1 -- 172.21.15.28:0/476547677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a5c19cf50 con 0x7f8a5c069dc0 2024-04-25T01:36:24.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a6113b640 1 --2- 172.21.15.28:0/476547677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a5c107730 0x7f8a5c19c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:24.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.247+0000 7f8a6093a640 1 --2- 172.21.15.28:0/476547677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a5c069dc0 0x7f8a5c197fb0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f8a5000d950 tx=0x7f8a5000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:24.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.248+0000 7f8a59ffb640 1 -- 172.21.15.28:0/476547677 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a50017030 con 0x7f8a5c069dc0 2024-04-25T01:36:24.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.248+0000 7f8a62b9e640 1 -- 172.21.15.28:0/476547677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a5c19d240 con 0x7f8a5c069dc0 2024-04-25T01:36:24.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.248+0000 7f8a59ffb640 1 -- 172.21.15.28:0/476547677 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a50004020 con 0x7f8a5c069dc0 2024-04-25T01:36:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.248+0000 7f8a59ffb640 1 -- 172.21.15.28:0/476547677 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a500047a0 con 0x7f8a5c069dc0 2024-04-25T01:36:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.248+0000 7f8a62b9e640 1 -- 172.21.15.28:0/476547677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a5c1a4ad0 con 0x7f8a5c069dc0 2024-04-25T01:36:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.250+0000 7f8a59ffb640 1 -- 172.21.15.28:0/476547677 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8a500041d0 con 0x7f8a5c069dc0 2024-04-25T01:36:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.250+0000 7f8a65985740 1 Processor -- start 2024-04-25T01:36:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a65985740 1 -- start start 2024-04-25T01:36:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a59ffb640 1 --2- 172.21.15.28:0/476547677 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8a4c0a2d00 0x7f8a4c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a65985740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13580 0x55da9fb13960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a65985740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb13ea0 0x55da9fb1e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a65985740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 0x55da9fb20960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a65985740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55da9fa78670 con 0x55da9fb1e570 2024-04-25T01:36:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a65985740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55da9fa20970 con 0x55da9fb13ea0 2024-04-25T01:36:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a65985740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55da9fa74630 con 0x55da9fb13580 2024-04-25T01:36:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a6113b640 1 --2- 172.21.15.28:0/476547677 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8a4c0a2d00 0x7f8a4c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a6113b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 0x55da9fb20960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a59ffb640 1 -- 172.21.15.28:0/476547677 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8a500cf180 con 0x7f8a5c069dc0 2024-04-25T01:36:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a6113b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 0x55da9fb20960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48482/0 (socket says 172.21.15.28:48482) 2024-04-25T01:36:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a6113b640 1 -- 172.21.15.28:0/1413712627 learned_addr learned my addr 172.21.15.28:0/1413712627 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a6093a640 1 --2- 172.21.15.28:0/1413712627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13580 0x55da9fb13960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a5bfff640 1 --2- 172.21.15.28:0/1413712627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb13ea0 0x55da9fb1e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a6093a640 1 -- 172.21.15.28:0/1413712627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb13ea0 msgr2=0x55da9fb1e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a6093a640 1 --2- 172.21.15.28:0/1413712627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb13ea0 0x55da9fb1e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a6093a640 1 -- 172.21.15.28:0/1413712627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 msgr2=0x55da9fb20960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a6093a640 1 --2- 172.21.15.28:0/1413712627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 0x55da9fb20960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a6093a640 1 -- 172.21.15.28:0/1413712627 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da9fa44400 con 0x55da9fb13580 2024-04-25T01:36:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a6113b640 1 --2- 172.21.15.28:0/1413712627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 0x55da9fb20960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:36:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.251+0000 7f8a6113b640 1 --2- 172.21.15.28:0/476547677 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8a4c0a2d00 0x7f8a4c0a51c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8a5400afd0 tx=0x7f8a5400adc0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.252+0000 7f8a6093a640 1 --2- 172.21.15.28:0/1413712627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13580 0x55da9fb13960 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8a500cc530 tx=0x7f8a5009ae40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2fe9a81e6aba250 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.252+0000 7f8a46ffd640 1 -- 172.21.15.28:0/1413712627 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a500a0840 con 0x55da9fb13580 2024-04-25T01:36:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.252+0000 7f8a46ffd640 1 -- 172.21.15.28:0/1413712627 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a5009ba00 con 0x55da9fb13580 2024-04-25T01:36:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.252+0000 7f8a46ffd640 1 -- 172.21.15.28:0/1413712627 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a500a1020 con 0x55da9fb13580 2024-04-25T01:36:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.252+0000 7f8a65985740 1 -- 172.21.15.28:0/1413712627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13580 msgr2=0x55da9fb13960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.252+0000 7f8a65985740 1 --2- 172.21.15.28:0/1413712627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13580 0x55da9fb13960 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8a500cc530 tx=0x7f8a5009ae40 comp rx=0 tx=0).stop 2024-04-25T01:36:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.252+0000 7f8a65985740 1 -- 172.21.15.28:0/1413712627 shutdown_connections 2024-04-25T01:36:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.252+0000 7f8a65985740 1 --2- 172.21.15.28:0/1413712627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 0x55da9fb20960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.252+0000 7f8a65985740 1 --2- 172.21.15.28:0/1413712627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb13ea0 0x55da9fb1e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.252+0000 7f8a65985740 1 --2- 172.21.15.28:0/1413712627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13580 0x55da9fb13960 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.252+0000 7f8a65985740 1 -- 172.21.15.28:0/1413712627 >> 172.21.15.28:0/1413712627 conn(0x55da9fb0f8a0 msgr2=0x55da9fb0fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.252+0000 7f8a65985740 1 -- 172.21.15.28:0/1413712627 shutdown_connections 2024-04-25T01:36:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.253+0000 7f8a65985740 1 -- 172.21.15.28:0/1413712627 wait complete. 2024-04-25T01:36:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.253+0000 7f8a65985740 1 -- start start 2024-04-25T01:36:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.253+0000 7f8a65985740 1 Processor -- start 2024-04-25T01:36:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.253+0000 7f8a65985740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13580 0x55da9fb1df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.253+0000 7f8a65985740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb13ea0 0x55da9fb1b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.253+0000 7f8a65985740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 0x55da9fb1b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.253+0000 7f8a65985740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55da9fb21360 con 0x55da9fb1e570 2024-04-25T01:36:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.253+0000 7f8a65985740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55da9fa34050 con 0x55da9fb13ea0 2024-04-25T01:36:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.253+0000 7f8a65985740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55da9fb214e0 con 0x55da9fb13580 2024-04-25T01:36:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.253+0000 7f8a65985740 10 client.0 _ll_register_callbacks cb 0x55da9f78bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.253+0000 7f8a6113b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 0x55da9fb1b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.253+0000 7f8a6113b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 0x55da9fb1b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48498/0 (socket says 172.21.15.28:48498) 2024-04-25T01:36:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.253+0000 7f8a6113b640 1 -- 172.21.15.28:0/1463385554 learned_addr learned my addr 172.21.15.28:0/1463385554 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.253+0000 7f8a5bfff640 1 --2- 172.21.15.28:0/1463385554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb13ea0 0x55da9fb1b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.253+0000 7f8a6093a640 1 --2- 172.21.15.28:0/1463385554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13580 0x55da9fb1df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.254+0000 7f8a6113b640 1 -- 172.21.15.28:0/1463385554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13580 msgr2=0x55da9fb1df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.254+0000 7f8a6113b640 1 --2- 172.21.15.28:0/1463385554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13580 0x55da9fb1df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.254+0000 7f8a6113b640 1 -- 172.21.15.28:0/1463385554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb13ea0 msgr2=0x55da9fb1b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.254+0000 7f8a6113b640 1 --2- 172.21.15.28:0/1463385554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb13ea0 0x55da9fb1b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.254+0000 7f8a6113b640 1 -- 172.21.15.28:0/1463385554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da9fa05dc0 con 0x55da9fb1e570 2024-04-25T01:36:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.254+0000 7f8a6113b640 1 --2- 172.21.15.28:0/1463385554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 0x55da9fb1b860 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f8a5401e600 tx=0x7f8a54005f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b47b570ec9369da8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.254+0000 7f8a3bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.254+0000 7f8a3bfff640 1 -- 172.21.15.28:0/1463385554 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5401f800 con 0x55da9fb1e570 2024-04-25T01:36:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.254+0000 7f8a3bfff640 1 -- 172.21.15.28:0/1463385554 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8a5401f9a0 con 0x55da9fb1e570 2024-04-25T01:36:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.254+0000 7f8a65985740 1 -- 172.21.15.28:0/1463385554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a5009ba00 con 0x55da9fb1e570 2024-04-25T01:36:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.254+0000 7f8a3bfff640 1 -- 172.21.15.28:0/1463385554 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8a5400a6f0 con 0x55da9fb1e570 2024-04-25T01:36:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.254+0000 7f8a65985740 10 client.16656 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.254+0000 7f8a65985740 1 -- 172.21.15.28:0/1463385554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55da9f78c3f0 con 0x55da9fb1e570 2024-04-25T01:36:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.254+0000 7f8a3bfff640 1 -- 172.21.15.28:0/1463385554 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a5009ba00 con 0x55da9fb1e570 2024-04-25T01:36:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.255+0000 7f8a3bfff640 1 -- 172.21.15.28:0/1463385554 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8a5402ace0 con 0x55da9fb1e570 2024-04-25T01:36:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.255+0000 7f8a65985740 10 client.16656 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.255+0000 7f8a65985740 10 client.16656 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.255+0000 7f8a65985740 20 client.16656 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.255+0000 7f8a65985740 1 --2- 172.21.15.28:0/1463385554 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55da9fb2e640 0x55da9fb30a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.255+0000 7f8a65985740 4 client.16656 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.255+0000 7f8a65985740 1 -- 172.21.15.28:0/1463385554 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da9fa85bb0 con 0x55da9fb2e640 2024-04-25T01:36:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.255+0000 7f8a6093a640 1 --2- 172.21.15.28:0/1463385554 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55da9fb2e640 0x55da9fb30a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.255+0000 7f8a6093a640 1 --2- 172.21.15.28:0/1463385554 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55da9fb2e640 0x55da9fb30a30 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a55e1f7b77540304 server_cookie=8df264ee11b953f1 in_seq=0 out_seq=0 2024-04-25T01:36:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.255+0000 7f8a3bfff640 10 client.16656 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.256+0000 7f8a3bfff640 1 -- 172.21.15.28:0/1463385554 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55da9fa85bb0 con 0x55da9fb2e640 2024-04-25T01:36:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.256+0000 7f8a3bfff640 10 client.16656 handle_command_reply: tid=0 2024-04-25T01:36:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.256+0000 7f8a65985740 1 client.16656 shutdown 2024-04-25T01:36:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.256+0000 7f8a65985740 1 -- 172.21.15.28:0/1463385554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 msgr2=0x55da9fb1b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.256+0000 7f8a65985740 1 --2- 172.21.15.28:0/1463385554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 0x55da9fb1b860 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f8a5401e600 tx=0x7f8a54005f40 comp rx=0 tx=0).stop 2024-04-25T01:36:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.256+0000 7f8a65985740 1 -- 172.21.15.28:0/1463385554 shutdown_connections 2024-04-25T01:36:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.256+0000 7f8a65985740 1 --2- 172.21.15.28:0/1463385554 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55da9fb2e640 0x55da9fb30a30 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.256+0000 7f8a65985740 1 --2- 172.21.15.28:0/1463385554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 0x55da9fb1b860 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.256+0000 7f8a65985740 1 --2- 172.21.15.28:0/1463385554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb13ea0 0x55da9fb1b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.256+0000 7f8a3bfff640 0 client.16656 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.256+0000 7f8a65985740 1 --2- 172.21.15.28:0/1463385554 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13580 0x55da9fb1df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.256+0000 7f8a65985740 1 -- 172.21.15.28:0/1463385554 >> 172.21.15.28:0/1463385554 conn(0x55da9fb0f8a0 msgr2=0x55da9fb156a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.257+0000 7f8a65985740 1 -- 172.21.15.28:0/1463385554 shutdown_connections 2024-04-25T01:36:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.257+0000 7f8a65985740 1 -- 172.21.15.28:0/1463385554 wait complete. 2024-04-25T01:36:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.257+0000 7f8a65985740 20 client.16656 trim_cache size 0 max 16384 2024-04-25T01:36:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.270+0000 7f8a65985740 1 Processor -- start 2024-04-25T01:36:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.270+0000 7f8a65985740 1 -- start start 2024-04-25T01:36:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.270+0000 7f8a65985740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb13580 0x55da9fb2a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.270+0000 7f8a65985740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13ea0 0x55da9fb1da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.270+0000 7f8a65985740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb1e570 0x55da9fb362c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.270+0000 7f8a65985740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c1aea20 con 0x55da9fb13580 2024-04-25T01:36:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.270+0000 7f8a65985740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c1b0560 con 0x55da9fb1e570 2024-04-25T01:36:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.270+0000 7f8a65985740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c1af0f0 con 0x55da9fb13ea0 2024-04-25T01:36:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.270+0000 7f8a6093a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb13580 0x55da9fb2a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.270+0000 7f8a6093a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb13580 0x55da9fb2a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48508/0 (socket says 172.21.15.28:48508) 2024-04-25T01:36:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.270+0000 7f8a6093a640 1 -- 172.21.15.28:0/2316172026 learned_addr learned my addr 172.21.15.28:0/2316172026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.270+0000 7f8a6113b640 1 --2- 172.21.15.28:0/2316172026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb1e570 0x55da9fb362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.270+0000 7f8a6093a640 1 -- 172.21.15.28:0/2316172026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13ea0 msgr2=0x55da9fb1da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.270+0000 7f8a6093a640 1 --2- 172.21.15.28:0/2316172026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13ea0 0x55da9fb1da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.270+0000 7f8a6093a640 1 -- 172.21.15.28:0/2316172026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb1e570 msgr2=0x55da9fb362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.270+0000 7f8a6093a640 1 --2- 172.21.15.28:0/2316172026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb1e570 0x55da9fb362c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.270+0000 7f8a6093a640 1 -- 172.21.15.28:0/2316172026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da9fb30da0 con 0x55da9fb13580 2024-04-25T01:36:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.270+0000 7f8a6113b640 1 --2- 172.21.15.28:0/2316172026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb1e570 0x55da9fb362c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.271+0000 7f8a6093a640 1 --2- 172.21.15.28:0/2316172026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb13580 0x55da9fb2a220 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f8a5009b290 tx=0x7f8a50005cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94cf6074224fa1b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.271+0000 7f8a467fc640 1 -- 172.21.15.28:0/2316172026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5009ba00 con 0x55da9fb13580 2024-04-25T01:36:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.271+0000 7f8a467fc640 1 -- 172.21.15.28:0/2316172026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a500b4070 con 0x55da9fb13580 2024-04-25T01:36:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.271+0000 7f8a467fc640 1 -- 172.21.15.28:0/2316172026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a500a0f00 con 0x55da9fb13580 2024-04-25T01:36:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.271+0000 7f8a65985740 1 -- 172.21.15.28:0/2316172026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb13580 msgr2=0x55da9fb2a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.271+0000 7f8a65985740 1 --2- 172.21.15.28:0/2316172026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb13580 0x55da9fb2a220 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f8a5009b290 tx=0x7f8a50005cd0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.271+0000 7f8a65985740 1 -- 172.21.15.28:0/2316172026 shutdown_connections 2024-04-25T01:36:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.271+0000 7f8a65985740 1 --2- 172.21.15.28:0/2316172026 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb1e570 0x55da9fb362c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.271+0000 7f8a65985740 1 --2- 172.21.15.28:0/2316172026 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13ea0 0x55da9fb1da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.271+0000 7f8a65985740 1 --2- 172.21.15.28:0/2316172026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb13580 0x55da9fb2a220 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.271+0000 7f8a65985740 1 -- 172.21.15.28:0/2316172026 >> 172.21.15.28:0/2316172026 conn(0x55da9fb0f8a0 msgr2=0x55da9fb16860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.271+0000 7f8a65985740 1 -- 172.21.15.28:0/2316172026 shutdown_connections 2024-04-25T01:36:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.271+0000 7f8a65985740 1 -- 172.21.15.28:0/2316172026 wait complete. 2024-04-25T01:36:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.272+0000 7f8a65985740 1 -- start start 2024-04-25T01:36:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.272+0000 7f8a65985740 1 Processor -- start 2024-04-25T01:36:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.272+0000 7f8a65985740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13580 0x55da9fb22250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.272+0000 7f8a65985740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb13ea0 0x55da9fb22790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.272+0000 7f8a65985740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 0x55da9fb22cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.272+0000 7f8a65985740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c1afe90 con 0x55da9fb1e570 2024-04-25T01:36:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.272+0000 7f8a65985740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c1af7c0 con 0x55da9fb13ea0 2024-04-25T01:36:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.272+0000 7f8a65985740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c1aca00 con 0x55da9fb13580 2024-04-25T01:36:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.272+0000 7f8a65985740 10 client.0 _ll_register_callbacks cb 0x7f8a5400ac00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.272+0000 7f8a6113b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 0x55da9fb22cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.272+0000 7f8a6113b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 0x55da9fb22cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48514/0 (socket says 172.21.15.28:48514) 2024-04-25T01:36:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.272+0000 7f8a6113b640 1 -- 172.21.15.28:0/1450873506 learned_addr learned my addr 172.21.15.28:0/1450873506 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.272+0000 7f8a6093a640 1 --2- 172.21.15.28:0/1450873506 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13580 0x55da9fb22250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.272+0000 7f8a5bfff640 1 --2- 172.21.15.28:0/1450873506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb13ea0 0x55da9fb22790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.272+0000 7f8a6113b640 1 -- 172.21.15.28:0/1450873506 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13580 msgr2=0x55da9fb22250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.272+0000 7f8a6113b640 1 --2- 172.21.15.28:0/1450873506 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13580 0x55da9fb22250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a6113b640 1 -- 172.21.15.28:0/1450873506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb13ea0 msgr2=0x55da9fb22790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a6113b640 1 --2- 172.21.15.28:0/1450873506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb13ea0 0x55da9fb22790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a6113b640 1 -- 172.21.15.28:0/1450873506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da9f8567d0 con 0x55da9fb1e570 2024-04-25T01:36:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a5bfff640 1 --2- 172.21.15.28:0/1450873506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb13ea0 0x55da9fb22790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a6113b640 1 --2- 172.21.15.28:0/1450873506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 0x55da9fb22cd0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f8a54006010 tx=0x7f8a540122f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce78da220691d81b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a3bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a3bfff640 1 -- 172.21.15.28:0/1450873506 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5403b070 con 0x55da9fb1e570 2024-04-25T01:36:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a3bfff640 1 -- 172.21.15.28:0/1450873506 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8a5401f600 con 0x55da9fb1e570 2024-04-25T01:36:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a65985740 1 -- 172.21.15.28:0/1450873506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a500b4070 con 0x55da9fb1e570 2024-04-25T01:36:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a3bfff640 1 -- 172.21.15.28:0/1450873506 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8a5401ee90 con 0x55da9fb1e570 2024-04-25T01:36:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a65985740 10 client.16662 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a65985740 1 -- 172.21.15.28:0/1450873506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55da9fb94c60 con 0x55da9fb1e570 2024-04-25T01:36:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a3bfff640 1 -- 172.21.15.28:0/1450873506 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a500b4070 con 0x55da9fb1e570 2024-04-25T01:36:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a3bfff640 1 -- 172.21.15.28:0/1450873506 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8a54004cd0 con 0x55da9fb1e570 2024-04-25T01:36:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a65985740 10 client.16662 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a65985740 10 client.16662 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a65985740 20 client.16662 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a65985740 1 --2- 172.21.15.28:0/1450873506 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55da9fb95350 0x55da9fb97770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a65985740 4 client.16662 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.273+0000 7f8a65985740 1 -- 172.21.15.28:0/1450873506 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55da9fa7ff00 con 0x55da9fb95350 2024-04-25T01:36:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.274+0000 7f8a6093a640 1 --2- 172.21.15.28:0/1450873506 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55da9fb95350 0x55da9fb97770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.274+0000 7f8a6093a640 1 --2- 172.21.15.28:0/1450873506 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55da9fb95350 0x55da9fb97770 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e76a113a2b83d18 server_cookie=522dcea3b1fcc0a7 in_seq=0 out_seq=0 2024-04-25T01:36:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.274+0000 7f8a3bfff640 10 client.16662 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.274+0000 7f8a3bfff640 1 -- 172.21.15.28:0/1450873506 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55da9fa7ff00 con 0x55da9fb95350 2024-04-25T01:36:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.274+0000 7f8a3bfff640 10 client.16662 handle_command_reply: tid=0 2024-04-25T01:36:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.274+0000 7f8a65985740 1 client.16662 shutdown 2024-04-25T01:36:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.274+0000 7f8a65985740 1 -- 172.21.15.28:0/1450873506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 msgr2=0x55da9fb22cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.274+0000 7f8a65985740 1 --2- 172.21.15.28:0/1450873506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 0x55da9fb22cd0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f8a54006010 tx=0x7f8a540122f0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a65985740 1 -- 172.21.15.28:0/1450873506 shutdown_connections 2024-04-25T01:36:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a65985740 1 --2- 172.21.15.28:0/1450873506 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55da9fb95350 0x55da9fb97770 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a65985740 1 --2- 172.21.15.28:0/1450873506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55da9fb1e570 0x55da9fb22cd0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a65985740 1 --2- 172.21.15.28:0/1450873506 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da9fb13ea0 0x55da9fb22790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a3bfff640 0 client.16662 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a65985740 1 --2- 172.21.15.28:0/1450873506 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55da9fb13580 0x55da9fb22250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a65985740 1 -- 172.21.15.28:0/1450873506 >> 172.21.15.28:0/1450873506 conn(0x55da9fb0f8a0 msgr2=0x55da9fb29210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a65985740 1 -- 172.21.15.28:0/1450873506 shutdown_connections 2024-04-25T01:36:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a65985740 1 -- 172.21.15.28:0/1450873506 wait complete. 2024-04-25T01:36:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a65985740 20 client.16662 trim_cache size 0 max 16384 2024-04-25T01:36:24.258 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:36:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a62b9e640 1 -- 172.21.15.28:0/476547677 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8a4c0a2d00 msgr2=0x7f8a4c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a62b9e640 1 --2- 172.21.15.28:0/476547677 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8a4c0a2d00 0x7f8a4c0a51c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8a5400afd0 tx=0x7f8a5400adc0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a62b9e640 1 -- 172.21.15.28:0/476547677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a5c069dc0 msgr2=0x7f8a5c197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a62b9e640 1 --2- 172.21.15.28:0/476547677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a5c069dc0 0x7f8a5c197fb0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f8a5000d950 tx=0x7f8a5000de10 comp rx=0 tx=0).stop 2024-04-25T01:36:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a62b9e640 1 -- 172.21.15.28:0/476547677 shutdown_connections 2024-04-25T01:36:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a62b9e640 1 --2- 172.21.15.28:0/476547677 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8a4c0a2d00 0x7f8a4c0a51c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a62b9e640 1 --2- 172.21.15.28:0/476547677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a5c107730 0x7f8a5c19c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a62b9e640 1 --2- 172.21.15.28:0/476547677 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8a5c06a780 0x7f8a5c1984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a62b9e640 1 --2- 172.21.15.28:0/476547677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a5c069dc0 0x7f8a5c197fb0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a62b9e640 1 -- 172.21.15.28:0/476547677 >> 172.21.15.28:0/476547677 conn(0x7f8a5c0776a0 msgr2=0x7f8a5c0fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a62b9e640 1 -- 172.21.15.28:0/476547677 shutdown_connections 2024-04-25T01:36:24.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:24.275+0000 7f8a62b9e640 1 -- 172.21.15.28:0/476547677 wait complete. 2024-04-25T01:36:24.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:24 smithi130 ceph-mon[32702]: pgmap v104: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T01:36:24.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:24 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1753372091' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:36:24.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:24 smithi032 ceph-mon[32461]: pgmap v104: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T01:36:24.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:24 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1753372091' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:36:24.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:24 smithi028 ceph-mon[29390]: pgmap v104: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T01:36:24.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:24 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1753372091' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:36:25.263 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:36:25.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:36:25.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:36:25.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.408+0000 7f643486d640 1 -- 172.21.15.28:0/1215420906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6430100fd0 msgr2=0x7f643010d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:25.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.408+0000 7f643486d640 1 --2- 172.21.15.28:0/1215420906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6430100fd0 0x7f643010d4e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f6420009cd0 tx=0x7f642002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.408+0000 7f643486d640 1 -- 172.21.15.28:0/1215420906 shutdown_connections 2024-04-25T01:36:25.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.408+0000 7f643486d640 1 --2- 172.21.15.28:0/1215420906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6430100fd0 0x7f643010d4e0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.408+0000 7f643486d640 1 --2- 172.21.15.28:0/1215420906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6430100610 0x7f6430100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.408+0000 7f643486d640 1 --2- 172.21.15.28:0/1215420906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64301066b0 0x7f6430106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.408+0000 7f643486d640 1 -- 172.21.15.28:0/1215420906 >> 172.21.15.28:0/1215420906 conn(0x7f6430077740 msgr2=0x7f6430077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:25.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.408+0000 7f643486d640 1 -- 172.21.15.28:0/1215420906 shutdown_connections 2024-04-25T01:36:25.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.408+0000 7f643486d640 1 -- 172.21.15.28:0/1215420906 wait complete. 2024-04-25T01:36:25.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.409+0000 7f643486d640 1 Processor -- start 2024-04-25T01:36:25.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.409+0000 7f643486d640 1 -- start start 2024-04-25T01:36:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.409+0000 7f643486d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6430100610 0x7f6430197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.409+0000 7f643486d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6430100fd0 0x7f64301984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.409+0000 7f643486d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64301066b0 0x7f643019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.409+0000 7f643486d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64301109f0 con 0x7f6430100fd0 2024-04-25T01:36:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.409+0000 7f643486d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6430110870 con 0x7f6430100610 2024-04-25T01:36:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.409+0000 7f643486d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6430110b70 con 0x7f64301066b0 2024-04-25T01:36:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.409+0000 7f642dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6430100fd0 0x7f64301984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.409+0000 7f642dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6430100fd0 0x7f64301984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48534/0 (socket says 172.21.15.28:48534) 2024-04-25T01:36:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.409+0000 7f642dd9b640 1 -- 172.21.15.28:0/3799969214 learned_addr learned my addr 172.21.15.28:0/3799969214 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.409+0000 7f642ed9d640 1 --2- 172.21.15.28:0/3799969214 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64301066b0 0x7f643019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.409+0000 7f642e59c640 1 --2- 172.21.15.28:0/3799969214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6430100610 0x7f6430197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.410+0000 7f642dd9b640 1 -- 172.21.15.28:0/3799969214 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64301066b0 msgr2=0x7f643019c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.410+0000 7f642dd9b640 1 --2- 172.21.15.28:0/3799969214 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64301066b0 0x7f643019c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.410+0000 7f642dd9b640 1 -- 172.21.15.28:0/3799969214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6430100610 msgr2=0x7f6430197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.410+0000 7f642dd9b640 1 --2- 172.21.15.28:0/3799969214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6430100610 0x7f6430197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.410+0000 7f642dd9b640 1 -- 172.21.15.28:0/3799969214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f643019cf20 con 0x7f6430100fd0 2024-04-25T01:36:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.410+0000 7f642dd9b640 1 --2- 172.21.15.28:0/3799969214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6430100fd0 0x7f64301984c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f642400c890 tx=0x7f642400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.410+0000 7f641f7fe640 1 -- 172.21.15.28:0/3799969214 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6424004220 con 0x7f6430100fd0 2024-04-25T01:36:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.410+0000 7f641f7fe640 1 -- 172.21.15.28:0/3799969214 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64240043c0 con 0x7f6430100fd0 2024-04-25T01:36:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.410+0000 7f643486d640 1 -- 172.21.15.28:0/3799969214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f643019d210 con 0x7f6430100fd0 2024-04-25T01:36:25.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.410+0000 7f641f7fe640 1 -- 172.21.15.28:0/3799969214 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64240049e0 con 0x7f6430100fd0 2024-04-25T01:36:25.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.410+0000 7f643486d640 1 -- 172.21.15.28:0/3799969214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64301a4aa0 con 0x7f6430100fd0 2024-04-25T01:36:25.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.412+0000 7f641f7fe640 1 -- 172.21.15.28:0/3799969214 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f64240079b0 con 0x7f6430100fd0 2024-04-25T01:36:25.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.413+0000 7f6437576740 1 Processor -- start 2024-04-25T01:36:25.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.413+0000 7f6437576740 1 -- start start 2024-04-25T01:36:25.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.413+0000 7f6437576740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b1c980 0x55db48b1acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.413+0000 7f641f7fe640 1 --2- 172.21.15.28:0/3799969214 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f64180a2d00 0x7f64180a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.413+0000 7f6437576740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b1b290 0x55db48b1b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.413+0000 7f6437576740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b2a420 0x55db48b2c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.413+0000 7f6437576740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db48a7f2c0 con 0x55db48b2a420 2024-04-25T01:36:25.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.413+0000 7f6437576740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55db48a27910 con 0x55db48b1b290 2024-04-25T01:36:25.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.413+0000 7f6437576740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55db48a7b280 con 0x55db48b1c980 2024-04-25T01:36:25.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.413+0000 7f642e59c640 1 --2- 172.21.15.28:0/3799969214 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f64180a2d00 0x7f64180a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.413+0000 7f642ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b2a420 0x55db48b2c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.413+0000 7f642ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b2a420 0x55db48b2c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48536/0 (socket says 172.21.15.28:48536) 2024-04-25T01:36:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.413+0000 7f642ed9d640 1 -- 172.21.15.28:0/384684188 learned_addr learned my addr 172.21.15.28:0/384684188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.413+0000 7f641f7fe640 1 -- 172.21.15.28:0/3799969214 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f64240cf7f0 con 0x7f6430100fd0 2024-04-25T01:36:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.414+0000 7f642dd9b640 1 --2- 172.21.15.28:0/384684188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b1b290 0x55db48b1b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.414+0000 7f642ed9d640 1 -- 172.21.15.28:0/384684188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b1c980 msgr2=0x55db48b1acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.414+0000 7f642ed9d640 1 --2- 172.21.15.28:0/384684188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b1c980 0x55db48b1acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.414+0000 7f642ed9d640 1 -- 172.21.15.28:0/384684188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b1b290 msgr2=0x55db48b1b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.414+0000 7f642ed9d640 1 --2- 172.21.15.28:0/384684188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b1b290 0x55db48b1b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.414+0000 7f642ed9d640 1 -- 172.21.15.28:0/384684188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db48a38fb0 con 0x55db48b2a420 2024-04-25T01:36:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.414+0000 7f642dd9b640 1 --2- 172.21.15.28:0/384684188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b1b290 0x55db48b1b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.414+0000 7f642e59c640 1 --2- 172.21.15.28:0/3799969214 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f64180a2d00 0x7f64180a51c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6428005fd0 tx=0x7f6428005850 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.414+0000 7f642ed9d640 1 --2- 172.21.15.28:0/384684188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b2a420 0x55db48b2c810 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f6420009cd0 tx=0x7f6420009960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f1770a33b9e05e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.414+0000 7f641cff9640 1 -- 172.21.15.28:0/384684188 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6420031ac0 con 0x55db48b2a420 2024-04-25T01:36:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.414+0000 7f641cff9640 1 -- 172.21.15.28:0/384684188 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6420036040 con 0x55db48b2a420 2024-04-25T01:36:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.414+0000 7f641cff9640 1 -- 172.21.15.28:0/384684188 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64200047d0 con 0x55db48b2a420 2024-04-25T01:36:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.415+0000 7f6437576740 1 -- 172.21.15.28:0/384684188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b2a420 msgr2=0x55db48b2c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.415+0000 7f6437576740 1 --2- 172.21.15.28:0/384684188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b2a420 0x55db48b2c810 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f6420009cd0 tx=0x7f6420009960 comp rx=0 tx=0).stop 2024-04-25T01:36:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.415+0000 7f6437576740 1 -- 172.21.15.28:0/384684188 shutdown_connections 2024-04-25T01:36:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.415+0000 7f6437576740 1 --2- 172.21.15.28:0/384684188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b2a420 0x55db48b2c810 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.415+0000 7f6437576740 1 --2- 172.21.15.28:0/384684188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b1b290 0x55db48b1b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.415+0000 7f6437576740 1 --2- 172.21.15.28:0/384684188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b1c980 0x55db48b1acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.415+0000 7f6437576740 1 -- 172.21.15.28:0/384684188 >> 172.21.15.28:0/384684188 conn(0x55db48ae3950 msgr2=0x55db48b177f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.415+0000 7f6437576740 1 -- 172.21.15.28:0/384684188 shutdown_connections 2024-04-25T01:36:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.415+0000 7f6437576740 1 -- 172.21.15.28:0/384684188 wait complete. 2024-04-25T01:36:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.415+0000 7f6437576740 1 -- start start 2024-04-25T01:36:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.415+0000 7f6437576740 1 Processor -- start 2024-04-25T01:36:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.416+0000 7f6437576740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b1b290 0x55db48b27370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.416+0000 7f6437576740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b1c980 0x55db48b278b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.416+0000 7f6437576740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b2a420 0x55db48b27df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.416+0000 7f6437576740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db48b2d250 con 0x55db48b1b290 2024-04-25T01:36:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.416+0000 7f6437576740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55db48a3ae90 con 0x55db48b2a420 2024-04-25T01:36:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.416+0000 7f6437576740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55db48b2d3d0 con 0x55db48b1c980 2024-04-25T01:36:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.416+0000 7f6437576740 10 client.0 _ll_register_callbacks cb 0x55db48a76970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.416+0000 7f642e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b1b290 0x55db48b27370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.416+0000 7f642e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b1b290 0x55db48b27370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48542/0 (socket says 172.21.15.28:48542) 2024-04-25T01:36:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.416+0000 7f642e59c640 1 -- 172.21.15.28:0/2659873729 learned_addr learned my addr 172.21.15.28:0/2659873729 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.416+0000 7f642ed9d640 1 --2- 172.21.15.28:0/2659873729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b2a420 0x55db48b27df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.416+0000 7f642dd9b640 1 --2- 172.21.15.28:0/2659873729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b1c980 0x55db48b278b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.416+0000 7f642e59c640 1 -- 172.21.15.28:0/2659873729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b1c980 msgr2=0x55db48b278b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.416+0000 7f642e59c640 1 --2- 172.21.15.28:0/2659873729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b1c980 0x55db48b278b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.416+0000 7f642e59c640 1 -- 172.21.15.28:0/2659873729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b2a420 msgr2=0x55db48b27df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.416+0000 7f642e59c640 1 --2- 172.21.15.28:0/2659873729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b2a420 0x55db48b27df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.416+0000 7f642e59c640 1 -- 172.21.15.28:0/2659873729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db4885b7e0 con 0x55db48b1b290 2024-04-25T01:36:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.416+0000 7f642e59c640 1 --2- 172.21.15.28:0/2659873729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b1b290 0x55db48b27370 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f642800ffe0 tx=0x7f64280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdfd955baebf1743 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.416+0000 7f6415ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.417+0000 7f6415ffb640 1 -- 172.21.15.28:0/2659873729 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6428022070 con 0x55db48b1b290 2024-04-25T01:36:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.417+0000 7f6415ffb640 1 -- 172.21.15.28:0/2659873729 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f642801e860 con 0x55db48b1b290 2024-04-25T01:36:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.417+0000 7f6437576740 1 -- 172.21.15.28:0/2659873729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6420036040 con 0x55db48b1b290 2024-04-25T01:36:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.417+0000 7f6415ffb640 1 -- 172.21.15.28:0/2659873729 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f642802e050 con 0x55db48b1b290 2024-04-25T01:36:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.417+0000 7f6437576740 10 client.16674 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.417+0000 7f6437576740 1 -- 172.21.15.28:0/2659873729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55db488bbf70 con 0x55db48b1b290 2024-04-25T01:36:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.417+0000 7f6415ffb640 1 -- 172.21.15.28:0/2659873729 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6420036040 con 0x55db48b1b290 2024-04-25T01:36:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.417+0000 7f6415ffb640 1 -- 172.21.15.28:0/2659873729 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6428010b20 con 0x55db48b1b290 2024-04-25T01:36:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.417+0000 7f6437576740 10 client.16674 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.417+0000 7f6437576740 10 client.16674 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.417+0000 7f6437576740 20 client.16674 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.417+0000 7f6437576740 1 --2- 172.21.15.28:0/2659873729 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db48b3a4e0 0x55db48b3c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.417+0000 7f6437576740 4 client.16674 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.417+0000 7f6437576740 1 -- 172.21.15.28:0/2659873729 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db48a535a0 con 0x55db48b3a4e0 2024-04-25T01:36:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.417+0000 7f642dd9b640 1 --2- 172.21.15.28:0/2659873729 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db48b3a4e0 0x55db48b3c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.418+0000 7f6415ffb640 10 client.16674 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.418+0000 7f642dd9b640 1 --2- 172.21.15.28:0/2659873729 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db48b3a4e0 0x55db48b3c930 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2652f10a8552b725 server_cookie=b61223b6e987bfd9 in_seq=0 out_seq=0 2024-04-25T01:36:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.418+0000 7f6415ffb640 1 -- 172.21.15.28:0/2659873729 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55db48a535a0 con 0x55db48b3a4e0 2024-04-25T01:36:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.418+0000 7f6415ffb640 10 client.16674 handle_command_reply: tid=0 2024-04-25T01:36:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.418+0000 7f6437576740 1 client.16674 shutdown 2024-04-25T01:36:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.419+0000 7f6437576740 1 -- 172.21.15.28:0/2659873729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b1b290 msgr2=0x55db48b27370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.419+0000 7f6437576740 1 --2- 172.21.15.28:0/2659873729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b1b290 0x55db48b27370 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f642800ffe0 tx=0x7f64280046c0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.419+0000 7f6437576740 1 -- 172.21.15.28:0/2659873729 shutdown_connections 2024-04-25T01:36:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.419+0000 7f6437576740 1 --2- 172.21.15.28:0/2659873729 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db48b3a4e0 0x55db48b3c930 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.419+0000 7f6437576740 1 --2- 172.21.15.28:0/2659873729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b2a420 0x55db48b27df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.419+0000 7f6437576740 1 --2- 172.21.15.28:0/2659873729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b1c980 0x55db48b278b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.419+0000 7f6415ffb640 0 client.16674 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.419+0000 7f6437576740 1 --2- 172.21.15.28:0/2659873729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b1b290 0x55db48b27370 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.419+0000 7f6437576740 1 -- 172.21.15.28:0/2659873729 >> 172.21.15.28:0/2659873729 conn(0x55db48ae3950 msgr2=0x55db48b1d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.419+0000 7f6437576740 1 -- 172.21.15.28:0/2659873729 shutdown_connections 2024-04-25T01:36:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.419+0000 7f6437576740 1 -- 172.21.15.28:0/2659873729 wait complete. 2024-04-25T01:36:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.419+0000 7f6437576740 20 client.16674 trim_cache size 0 max 16384 2024-04-25T01:36:25.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f6437576740 1 Processor -- start 2024-04-25T01:36:25.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f6437576740 1 -- start start 2024-04-25T01:36:25.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f6437576740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b1b290 0x55db48b36150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f6437576740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b1c980 0x55db48b167d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f6437576740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b2a420 0x55db48b42200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f6437576740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64301ac7c0 con 0x55db48b1c980 2024-04-25T01:36:25.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f6437576740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f64301ae300 con 0x55db48b1b290 2024-04-25T01:36:25.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f6437576740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f64301ace90 con 0x55db48b2a420 2024-04-25T01:36:25.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f642dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b1c980 0x55db48b167d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f642dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b1c980 0x55db48b167d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48554/0 (socket says 172.21.15.28:48554) 2024-04-25T01:36:25.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f642dd9b640 1 -- 172.21.15.28:0/3720157931 learned_addr learned my addr 172.21.15.28:0/3720157931 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f642e59c640 1 --2- 172.21.15.28:0/3720157931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b1b290 0x55db48b36150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f642ed9d640 1 --2- 172.21.15.28:0/3720157931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b2a420 0x55db48b42200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f642dd9b640 1 -- 172.21.15.28:0/3720157931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b2a420 msgr2=0x55db48b42200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f642dd9b640 1 --2- 172.21.15.28:0/3720157931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b2a420 0x55db48b42200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f642dd9b640 1 -- 172.21.15.28:0/3720157931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b1b290 msgr2=0x55db48b36150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f642dd9b640 1 --2- 172.21.15.28:0/3720157931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b1b290 0x55db48b36150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f642dd9b640 1 -- 172.21.15.28:0/3720157931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db48753170 con 0x55db48b1c980 2024-04-25T01:36:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f642ed9d640 1 --2- 172.21.15.28:0/3720157931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b2a420 0x55db48b42200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.437+0000 7f642dd9b640 1 --2- 172.21.15.28:0/3720157931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b1c980 0x55db48b167d0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6424007590 tx=0x7f6424005d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b94e1a9c88679b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.438+0000 7f6417fff640 1 -- 172.21.15.28:0/3720157931 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f642401d040 con 0x55db48b1c980 2024-04-25T01:36:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.438+0000 7f6417fff640 1 -- 172.21.15.28:0/3720157931 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f642409b9d0 con 0x55db48b1c980 2024-04-25T01:36:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.438+0000 7f6417fff640 1 -- 172.21.15.28:0/3720157931 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6424007ca0 con 0x55db48b1c980 2024-04-25T01:36:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.438+0000 7f6437576740 1 -- 172.21.15.28:0/3720157931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b1c980 msgr2=0x55db48b167d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.438+0000 7f6437576740 1 --2- 172.21.15.28:0/3720157931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b1c980 0x55db48b167d0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6424007590 tx=0x7f6424005d20 comp rx=0 tx=0).stop 2024-04-25T01:36:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.438+0000 7f6437576740 1 -- 172.21.15.28:0/3720157931 shutdown_connections 2024-04-25T01:36:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.438+0000 7f6437576740 1 --2- 172.21.15.28:0/3720157931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b2a420 0x55db48b42200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.438+0000 7f6437576740 1 --2- 172.21.15.28:0/3720157931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b1c980 0x55db48b167d0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.438+0000 7f6437576740 1 --2- 172.21.15.28:0/3720157931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b1b290 0x55db48b36150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.438+0000 7f6437576740 1 -- 172.21.15.28:0/3720157931 >> 172.21.15.28:0/3720157931 conn(0x55db48ae3950 msgr2=0x55db48b160c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.438+0000 7f6437576740 1 -- 172.21.15.28:0/3720157931 shutdown_connections 2024-04-25T01:36:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.438+0000 7f6437576740 1 -- 172.21.15.28:0/3720157931 wait complete. 2024-04-25T01:36:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.438+0000 7f6437576740 1 -- start start 2024-04-25T01:36:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f6437576740 1 Processor -- start 2024-04-25T01:36:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f6437576740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b1b290 0x55db48b24440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f6437576740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b1c980 0x55db48b24980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f6437576740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b2a420 0x55db48b9fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f6437576740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64301adc30 con 0x55db48b2a420 2024-04-25T01:36:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f6437576740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f64301ad560 con 0x55db48b1c980 2024-04-25T01:36:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f6437576740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f64301aa7a0 con 0x55db48b1b290 2024-04-25T01:36:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f6437576740 10 client.0 _ll_register_callbacks cb 0x7f6428015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f642ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b2a420 0x55db48b9fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f642ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b2a420 0x55db48b9fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48556/0 (socket says 172.21.15.28:48556) 2024-04-25T01:36:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f642ed9d640 1 -- 172.21.15.28:0/1675951944 learned_addr learned my addr 172.21.15.28:0/1675951944 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f642e59c640 1 --2- 172.21.15.28:0/1675951944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b1b290 0x55db48b24440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f642ed9d640 1 -- 172.21.15.28:0/1675951944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b1b290 msgr2=0x55db48b24440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f642ed9d640 1 --2- 172.21.15.28:0/1675951944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b1b290 0x55db48b24440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f642ed9d640 1 -- 172.21.15.28:0/1675951944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b1c980 msgr2=0x55db48b24980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f642dd9b640 1 --2- 172.21.15.28:0/1675951944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b1c980 0x55db48b24980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f642ed9d640 1 --2- 172.21.15.28:0/1675951944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b1c980 0x55db48b24980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f642ed9d640 1 -- 172.21.15.28:0/1675951944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db48b9ffb0 con 0x55db48b2a420 2024-04-25T01:36:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f642e59c640 1 --2- 172.21.15.28:0/1675951944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b1b290 0x55db48b24440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f642dd9b640 1 --2- 172.21.15.28:0/1675951944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b1c980 0x55db48b24980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f642ed9d640 1 --2- 172.21.15.28:0/1675951944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b2a420 0x55db48b9fa70 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f64200043a0 tx=0x7f64200043d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb19a9ea377d616 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.439+0000 7f6415ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.440+0000 7f6415ffb640 1 -- 172.21.15.28:0/1675951944 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6420052060 con 0x55db48b2a420 2024-04-25T01:36:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.440+0000 7f6415ffb640 1 -- 172.21.15.28:0/1675951944 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f6420003b70 con 0x55db48b2a420 2024-04-25T01:36:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.440+0000 7f6437576740 1 -- 172.21.15.28:0/1675951944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f642409b9d0 con 0x55db48b2a420 2024-04-25T01:36:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.440+0000 7f6415ffb640 1 -- 172.21.15.28:0/1675951944 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6420031d80 con 0x55db48b2a420 2024-04-25T01:36:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.440+0000 7f6437576740 10 client.16680 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.440+0000 7f6437576740 1 -- 172.21.15.28:0/1675951944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55db48ba10a0 con 0x55db48b2a420 2024-04-25T01:36:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.440+0000 7f6415ffb640 1 -- 172.21.15.28:0/1675951944 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f642409b9d0 con 0x55db48b2a420 2024-04-25T01:36:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.440+0000 7f6415ffb640 1 -- 172.21.15.28:0/1675951944 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6420030070 con 0x55db48b2a420 2024-04-25T01:36:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.440+0000 7f6437576740 10 client.16680 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.440+0000 7f6437576740 10 client.16680 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.440+0000 7f6437576740 20 client.16680 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.440+0000 7f6437576740 1 --2- 172.21.15.28:0/1675951944 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db48ba13e0 0x55db48ba37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.440+0000 7f6437576740 4 client.16680 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:36:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.440+0000 7f6437576740 1 -- 172.21.15.28:0/1675951944 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55db48a8c800 con 0x55db48ba13e0 2024-04-25T01:36:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.440+0000 7f642e59c640 1 --2- 172.21.15.28:0/1675951944 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db48ba13e0 0x55db48ba37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.441+0000 7f6415ffb640 10 client.16680 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.441+0000 7f642e59c640 1 --2- 172.21.15.28:0/1675951944 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db48ba13e0 0x55db48ba37d0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4ee4930dc585402 server_cookie=f9a1cc14f4378f44 in_seq=0 out_seq=0 2024-04-25T01:36:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.442+0000 7f6415ffb640 1 -- 172.21.15.28:0/1675951944 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55db48a8c800 con 0x55db48ba13e0 2024-04-25T01:36:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.442+0000 7f6415ffb640 10 client.16680 handle_command_reply: tid=0 2024-04-25T01:36:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.442+0000 7f6437576740 1 client.16680 shutdown 2024-04-25T01:36:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.442+0000 7f6437576740 1 -- 172.21.15.28:0/1675951944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b2a420 msgr2=0x55db48b9fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.442+0000 7f6437576740 1 --2- 172.21.15.28:0/1675951944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b2a420 0x55db48b9fa70 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f64200043a0 tx=0x7f64200043d0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.442+0000 7f6437576740 1 -- 172.21.15.28:0/1675951944 shutdown_connections 2024-04-25T01:36:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.442+0000 7f6437576740 1 --2- 172.21.15.28:0/1675951944 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db48ba13e0 0x55db48ba37d0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.442+0000 7f6437576740 1 --2- 172.21.15.28:0/1675951944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db48b2a420 0x55db48b9fa70 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.442+0000 7f6437576740 1 --2- 172.21.15.28:0/1675951944 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db48b1c980 0x55db48b24980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.442+0000 7f6437576740 1 --2- 172.21.15.28:0/1675951944 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db48b1b290 0x55db48b24440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.442+0000 7f6415ffb640 0 client.16680 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.442+0000 7f6437576740 1 -- 172.21.15.28:0/1675951944 >> 172.21.15.28:0/1675951944 conn(0x55db48ae3950 msgr2=0x55db48b16950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.442+0000 7f6437576740 1 -- 172.21.15.28:0/1675951944 shutdown_connections 2024-04-25T01:36:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.442+0000 7f6437576740 1 -- 172.21.15.28:0/1675951944 wait complete. 2024-04-25T01:36:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.442+0000 7f6437576740 20 client.16680 trim_cache size 0 max 16384 2024-04-25T01:36:25.425 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:36:25.426 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:36:25.426 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "2b65b232-c036-46f6-90a8-6f14e32d4ed5", 2024-04-25T01:36:25.426 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:36:25.426 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:36:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.443+0000 7f643486d640 1 -- 172.21.15.28:0/3799969214 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f64180a2d00 msgr2=0x7f64180a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.443+0000 7f643486d640 1 --2- 172.21.15.28:0/3799969214 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f64180a2d00 0x7f64180a51c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6428005fd0 tx=0x7f6428005850 comp rx=0 tx=0).stop 2024-04-25T01:36:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.443+0000 7f643486d640 1 -- 172.21.15.28:0/3799969214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6430100fd0 msgr2=0x7f64301984c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.443+0000 7f643486d640 1 --2- 172.21.15.28:0/3799969214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6430100fd0 0x7f64301984c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f642400c890 tx=0x7f642400cd50 comp rx=0 tx=0).stop 2024-04-25T01:36:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.443+0000 7f643486d640 1 -- 172.21.15.28:0/3799969214 shutdown_connections 2024-04-25T01:36:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.443+0000 7f643486d640 1 --2- 172.21.15.28:0/3799969214 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f64180a2d00 0x7f64180a51c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.443+0000 7f643486d640 1 --2- 172.21.15.28:0/3799969214 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64301066b0 0x7f643019c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.443+0000 7f643486d640 1 --2- 172.21.15.28:0/3799969214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6430100fd0 0x7f64301984c0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.443+0000 7f643486d640 1 --2- 172.21.15.28:0/3799969214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6430100610 0x7f6430197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.443+0000 7f643486d640 1 -- 172.21.15.28:0/3799969214 >> 172.21.15.28:0/3799969214 conn(0x7f6430077740 msgr2=0x7f64300ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.443+0000 7f643486d640 1 -- 172.21.15.28:0/3799969214 shutdown_connections 2024-04-25T01:36:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.443+0000 7f643486d640 1 -- 172.21.15.28:0/3799969214 wait complete. 2024-04-25T01:36:25.430 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:36:25.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:36:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.559+0000 7fabbfe2a640 1 -- 172.21.15.28:0/2301535462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb81020d0 msgr2=0x7fabb810e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.559+0000 7fabbfe2a640 1 --2- 172.21.15.28:0/2301535462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb81020d0 0x7fabb810e5e0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fabb0009cd0 tx=0x7fabb002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.559+0000 7fabbfe2a640 1 -- 172.21.15.28:0/2301535462 shutdown_connections 2024-04-25T01:36:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.559+0000 7fabbfe2a640 1 --2- 172.21.15.28:0/2301535462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb81020d0 0x7fabb810e5e0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.559+0000 7fabbfe2a640 1 --2- 172.21.15.28:0/2301535462 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabb8101710 0x7fabb8101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.559+0000 7fabbfe2a640 1 --2- 172.21.15.28:0/2301535462 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabb8107720 0x7fabb8107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.559+0000 7fabbfe2a640 1 -- 172.21.15.28:0/2301535462 >> 172.21.15.28:0/2301535462 conn(0x7fabb8078b90 msgr2=0x7fabb8078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.559+0000 7fabbfe2a640 1 -- 172.21.15.28:0/2301535462 shutdown_connections 2024-04-25T01:36:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.559+0000 7fabbfe2a640 1 -- 172.21.15.28:0/2301535462 wait complete. 2024-04-25T01:36:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.560+0000 7fabbfe2a640 1 Processor -- start 2024-04-25T01:36:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.560+0000 7fabbfe2a640 1 -- start start 2024-04-25T01:36:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.560+0000 7fabbfe2a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabb8101710 0x7fabb8198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.560+0000 7fabbfe2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb81020d0 0x7fabb8198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.560+0000 7fabbfe2a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabb8107720 0x7fabb819c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.560+0000 7fabbfe2a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fabb8111b30 con 0x7fabb81020d0 2024-04-25T01:36:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.560+0000 7fabbfe2a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fabb81119b0 con 0x7fabb8107720 2024-04-25T01:36:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.560+0000 7fabbfe2a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fabb8111cb0 con 0x7fabb8101710 2024-04-25T01:36:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.560+0000 7fabbd3c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb81020d0 0x7fabb8198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.560+0000 7fabbd3c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb81020d0 0x7fabb8198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48568/0 (socket says 172.21.15.28:48568) 2024-04-25T01:36:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.560+0000 7fabbd3c5640 1 -- 172.21.15.28:0/1942124108 learned_addr learned my addr 172.21.15.28:0/1942124108 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.560+0000 7fabbdbc6640 1 --2- 172.21.15.28:0/1942124108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabb8101710 0x7fabb8198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.560+0000 7fabbe3c7640 1 --2- 172.21.15.28:0/1942124108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabb8107720 0x7fabb819c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.560+0000 7fabbd3c5640 1 -- 172.21.15.28:0/1942124108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabb8101710 msgr2=0x7fabb8198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.560+0000 7fabbd3c5640 1 --2- 172.21.15.28:0/1942124108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabb8101710 0x7fabb8198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.560+0000 7fabbd3c5640 1 -- 172.21.15.28:0/1942124108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabb8107720 msgr2=0x7fabb819c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.560+0000 7fabbd3c5640 1 --2- 172.21.15.28:0/1942124108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabb8107720 0x7fabb819c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.560+0000 7fabbd3c5640 1 -- 172.21.15.28:0/1942124108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb819d040 con 0x7fabb81020d0 2024-04-25T01:36:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.561+0000 7fabbe3c7640 1 --2- 172.21.15.28:0/1942124108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabb8107720 0x7fabb819c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.561+0000 7fabbd3c5640 1 --2- 172.21.15.28:0/1942124108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb81020d0 0x7fabb8198590 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7faba800d950 tx=0x7faba800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.561+0000 7fabaeffd640 1 -- 172.21.15.28:0/1942124108 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba8017070 con 0x7fabb81020d0 2024-04-25T01:36:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.561+0000 7fabaeffd640 1 -- 172.21.15.28:0/1942124108 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faba8009c30 con 0x7fabb81020d0 2024-04-25T01:36:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.561+0000 7fabaeffd640 1 -- 172.21.15.28:0/1942124108 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba800f6b0 con 0x7fabb81020d0 2024-04-25T01:36:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.561+0000 7fabbfe2a640 1 -- 172.21.15.28:0/1942124108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabb819d330 con 0x7fabb81020d0 2024-04-25T01:36:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.561+0000 7fabbfe2a640 1 -- 172.21.15.28:0/1942124108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabb81a4bc0 con 0x7fabb81020d0 2024-04-25T01:36:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.563+0000 7fabaeffd640 1 -- 172.21.15.28:0/1942124108 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7faba80109d0 con 0x7fabb81020d0 2024-04-25T01:36:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.563+0000 7fabc2c14740 1 Processor -- start 2024-04-25T01:36:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabc2c14740 1 -- start start 2024-04-25T01:36:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabc2c14740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580614630 0x559580614a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabaeffd640 1 --2- 172.21.15.28:0/1942124108 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7faba40a2d00 0x7faba40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabc2c14740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 0x55958061f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabc2c14740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55958061f620 0x559580621a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabc2c14740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5595805796b0 con 0x559580614630 2024-04-25T01:36:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabc2c14740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559580521880 con 0x55958061f620 2024-04-25T01:36:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabc2c14740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x559580575670 con 0x559580614f50 2024-04-25T01:36:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabbe3c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580614630 0x559580614a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabbdbc6640 1 --2- 172.21.15.28:0/1942124108 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7faba40a2d00 0x7faba40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabbe3c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580614630 0x559580614a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48582/0 (socket says 172.21.15.28:48582) 2024-04-25T01:36:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabbe3c7640 1 -- 172.21.15.28:0/4229365187 learned_addr learned my addr 172.21.15.28:0/4229365187 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabaeffd640 1 -- 172.21.15.28:0/1942124108 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7faba80cf3e0 con 0x7fabb81020d0 2024-04-25T01:36:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabbd3c5640 1 --2- 172.21.15.28:0/4229365187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 0x55958061f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabbe3c7640 1 --2- 172.21.15.28:0/4229365187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55958061f620 0x559580621a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabbe3c7640 1 -- 172.21.15.28:0/4229365187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 msgr2=0x55958061f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabbe3c7640 1 --2- 172.21.15.28:0/4229365187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 0x55958061f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabbe3c7640 1 -- 172.21.15.28:0/4229365187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55958061f620 msgr2=0x559580621a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabbe3c7640 1 --2- 172.21.15.28:0/4229365187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55958061f620 0x559580621a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.564+0000 7fabbe3c7640 1 -- 172.21.15.28:0/4229365187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595803577d0 con 0x559580614630 2024-04-25T01:36:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.565+0000 7fabbdbc6640 1 --2- 172.21.15.28:0/1942124108 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7faba40a2d00 0x7faba40a51c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fabb4005fd0 tx=0x7fabb400b390 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.565+0000 7fabbe3c7640 1 --2- 172.21.15.28:0/4229365187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580614630 0x559580614a10 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fabb0035000 tx=0x7fabb0005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5cf24ad8da930ed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.565+0000 7faba3fff640 1 -- 172.21.15.28:0/4229365187 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb0031ac0 con 0x559580614630 2024-04-25T01:36:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.565+0000 7faba3fff640 1 -- 172.21.15.28:0/4229365187 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabb0036040 con 0x559580614630 2024-04-25T01:36:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.565+0000 7faba3fff640 1 -- 172.21.15.28:0/4229365187 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb0004610 con 0x559580614630 2024-04-25T01:36:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.565+0000 7fabc2c14740 1 -- 172.21.15.28:0/4229365187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580614630 msgr2=0x559580614a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.566+0000 7fabc2c14740 1 --2- 172.21.15.28:0/4229365187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580614630 0x559580614a10 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fabb0035000 tx=0x7fabb0005df0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.566+0000 7fabc2c14740 1 -- 172.21.15.28:0/4229365187 shutdown_connections 2024-04-25T01:36:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.566+0000 7fabc2c14740 1 --2- 172.21.15.28:0/4229365187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55958061f620 0x559580621a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.566+0000 7fabc2c14740 1 --2- 172.21.15.28:0/4229365187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 0x55958061f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.566+0000 7fabc2c14740 1 --2- 172.21.15.28:0/4229365187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559580614630 0x559580614a10 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.566+0000 7fabc2c14740 1 -- 172.21.15.28:0/4229365187 >> 172.21.15.28:0/4229365187 conn(0x559580610950 msgr2=0x559580610d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.566+0000 7fabc2c14740 1 -- 172.21.15.28:0/4229365187 shutdown_connections 2024-04-25T01:36:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.566+0000 7fabc2c14740 1 -- 172.21.15.28:0/4229365187 wait complete. 2024-04-25T01:36:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.566+0000 7fabc2c14740 1 -- start start 2024-04-25T01:36:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.566+0000 7fabc2c14740 1 Processor -- start 2024-04-25T01:36:25.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.566+0000 7fabc2c14740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559580614630 0x55958061c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.566+0000 7fabc2c14740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 0x55958061cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabc2c14740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 0x55958061d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabc2c14740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5595806223b0 con 0x55958061f620 2024-04-25T01:36:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabc2c14740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559580534f50 con 0x559580614630 2024-04-25T01:36:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabc2c14740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x559580622530 con 0x559580614f50 2024-04-25T01:36:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabc2c14740 10 client.0 _ll_register_callbacks cb 0x559580570000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabbe3c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 0x55958061d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabbe3c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 0x55958061d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48598/0 (socket says 172.21.15.28:48598) 2024-04-25T01:36:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabbe3c7640 1 -- 172.21.15.28:0/1581252540 learned_addr learned my addr 172.21.15.28:0/1581252540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabbd3c5640 1 --2- 172.21.15.28:0/1581252540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 0x55958061cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabbdbc6640 1 --2- 172.21.15.28:0/1581252540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559580614630 0x55958061c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabbe3c7640 1 -- 172.21.15.28:0/1581252540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 msgr2=0x55958061cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabbe3c7640 1 --2- 172.21.15.28:0/1581252540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 0x55958061cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabbe3c7640 1 -- 172.21.15.28:0/1581252540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559580614630 msgr2=0x55958061c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabbe3c7640 1 --2- 172.21.15.28:0/1581252540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559580614630 0x55958061c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabbe3c7640 1 -- 172.21.15.28:0/1581252540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55958028d3f0 con 0x55958061f620 2024-04-25T01:36:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabbdbc6640 1 --2- 172.21.15.28:0/1581252540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559580614630 0x55958061c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabbe3c7640 1 --2- 172.21.15.28:0/1581252540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 0x55958061d020 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fabb0009a60 tx=0x7fabb0005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ceb90b547268dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7faba17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7faba17fa640 1 -- 172.21.15.28:0/1581252540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb0003b70 con 0x55958061f620 2024-04-25T01:36:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7faba17fa640 1 -- 172.21.15.28:0/1581252540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fabb0003d10 con 0x55958061f620 2024-04-25T01:36:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabc2c14740 1 -- 172.21.15.28:0/1581252540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabb0036040 con 0x55958061f620 2024-04-25T01:36:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7faba17fa640 1 -- 172.21.15.28:0/1581252540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fabb0003f30 con 0x55958061f620 2024-04-25T01:36:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabc2c14740 10 client.16692 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.567+0000 7fabc2c14740 1 -- 172.21.15.28:0/1581252540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55958024d0f0 con 0x55958061f620 2024-04-25T01:36:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.568+0000 7faba17fa640 1 -- 172.21.15.28:0/1581252540 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabb0036040 con 0x55958061f620 2024-04-25T01:36:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.568+0000 7faba17fa640 1 -- 172.21.15.28:0/1581252540 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fabb0030070 con 0x55958061f620 2024-04-25T01:36:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.568+0000 7fabc2c14740 10 client.16692 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.568+0000 7fabc2c14740 10 client.16692 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.568+0000 7fabc2c14740 20 client.16692 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.568+0000 7fabc2c14740 1 --2- 172.21.15.28:0/1581252540 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55958062f710 0x559580631b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.568+0000 7fabc2c14740 4 client.16692 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.568+0000 7fabc2c14740 1 -- 172.21.15.28:0/1581252540 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559580586bf0 con 0x55958062f710 2024-04-25T01:36:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.568+0000 7fabbdbc6640 1 --2- 172.21.15.28:0/1581252540 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55958062f710 0x559580631b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.569+0000 7faba17fa640 10 client.16692 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.569+0000 7fabbdbc6640 1 --2- 172.21.15.28:0/1581252540 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55958062f710 0x559580631b00 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd2e1b1b07361a2b server_cookie=62d8dff30d4bc2e1 in_seq=0 out_seq=0 2024-04-25T01:36:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.569+0000 7faba17fa640 1 -- 172.21.15.28:0/1581252540 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559580586bf0 con 0x55958062f710 2024-04-25T01:36:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.569+0000 7faba17fa640 10 client.16692 handle_command_reply: tid=0 2024-04-25T01:36:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.569+0000 7fabc2c14740 1 client.16692 shutdown 2024-04-25T01:36:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.569+0000 7fabc2c14740 1 -- 172.21.15.28:0/1581252540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 msgr2=0x55958061d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.569+0000 7fabc2c14740 1 --2- 172.21.15.28:0/1581252540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 0x55958061d020 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fabb0009a60 tx=0x7fabb0005d80 comp rx=0 tx=0).stop 2024-04-25T01:36:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.569+0000 7fabc2c14740 1 -- 172.21.15.28:0/1581252540 shutdown_connections 2024-04-25T01:36:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.569+0000 7fabc2c14740 1 --2- 172.21.15.28:0/1581252540 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55958062f710 0x559580631b00 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.569+0000 7fabc2c14740 1 --2- 172.21.15.28:0/1581252540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 0x55958061d020 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.569+0000 7fabc2c14740 1 --2- 172.21.15.28:0/1581252540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 0x55958061cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.569+0000 7faba17fa640 0 client.16692 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.569+0000 7fabc2c14740 1 --2- 172.21.15.28:0/1581252540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559580614630 0x55958061c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.569+0000 7fabc2c14740 1 -- 172.21.15.28:0/1581252540 >> 172.21.15.28:0/1581252540 conn(0x559580610950 msgr2=0x559580616750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.570+0000 7fabc2c14740 1 -- 172.21.15.28:0/1581252540 shutdown_connections 2024-04-25T01:36:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.570+0000 7fabc2c14740 1 -- 172.21.15.28:0/1581252540 wait complete. 2024-04-25T01:36:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.570+0000 7fabc2c14740 20 client.16692 trim_cache size 0 max 16384 2024-04-25T01:36:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.587+0000 7fabc2c14740 1 Processor -- start 2024-04-25T01:36:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.587+0000 7fabc2c14740 1 -- start start 2024-04-25T01:36:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.587+0000 7fabc2c14740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559580614630 0x55958061ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.587+0000 7fabc2c14740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 0x55958061b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.587+0000 7fabc2c14740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 0x55958062ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.587+0000 7fabc2c14740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fabb81aeb30 con 0x55958061f620 2024-04-25T01:36:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.587+0000 7fabc2c14740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fabb81b0670 con 0x559580614630 2024-04-25T01:36:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.587+0000 7fabc2c14740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fabb81af200 con 0x559580614f50 2024-04-25T01:36:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.588+0000 7fabbe3c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 0x55958062ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.588+0000 7fabbe3c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 0x55958062ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48606/0 (socket says 172.21.15.28:48606) 2024-04-25T01:36:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.588+0000 7fabbe3c7640 1 -- 172.21.15.28:0/864064626 learned_addr learned my addr 172.21.15.28:0/864064626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.588+0000 7fabbdbc6640 1 --2- 172.21.15.28:0/864064626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559580614630 0x55958061ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.588+0000 7fabbd3c5640 1 --2- 172.21.15.28:0/864064626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 0x55958061b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.588+0000 7fabbe3c7640 1 -- 172.21.15.28:0/864064626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 msgr2=0x55958061b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.588+0000 7fabbe3c7640 1 --2- 172.21.15.28:0/864064626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 0x55958061b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.588+0000 7fabbe3c7640 1 -- 172.21.15.28:0/864064626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559580614630 msgr2=0x55958061ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.588+0000 7fabbe3c7640 1 --2- 172.21.15.28:0/864064626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559580614630 0x55958061ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.588+0000 7fabbe3c7640 1 -- 172.21.15.28:0/864064626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559580614430 con 0x55958061f620 2024-04-25T01:36:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.588+0000 7fabbe3c7640 1 --2- 172.21.15.28:0/864064626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 0x55958062ae90 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fabb002ae00 tx=0x7fabb002a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2b4fa6784788ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.588+0000 7faba37fe640 1 -- 172.21.15.28:0/864064626 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb00045a0 con 0x55958061f620 2024-04-25T01:36:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.588+0000 7faba37fe640 1 -- 172.21.15.28:0/864064626 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabb0036040 con 0x55958061f620 2024-04-25T01:36:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.588+0000 7faba37fe640 1 -- 172.21.15.28:0/864064626 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb0030690 con 0x55958061f620 2024-04-25T01:36:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.589+0000 7fabc2c14740 1 -- 172.21.15.28:0/864064626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 msgr2=0x55958062ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.589+0000 7fabc2c14740 1 --2- 172.21.15.28:0/864064626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 0x55958062ae90 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fabb002ae00 tx=0x7fabb002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.589+0000 7fabc2c14740 1 -- 172.21.15.28:0/864064626 shutdown_connections 2024-04-25T01:36:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.589+0000 7fabc2c14740 1 --2- 172.21.15.28:0/864064626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 0x55958062ae90 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.589+0000 7fabc2c14740 1 --2- 172.21.15.28:0/864064626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 0x55958061b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.589+0000 7fabc2c14740 1 --2- 172.21.15.28:0/864064626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559580614630 0x55958061ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.589+0000 7fabc2c14740 1 -- 172.21.15.28:0/864064626 >> 172.21.15.28:0/864064626 conn(0x559580610950 msgr2=0x55958062a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.589+0000 7fabc2c14740 1 -- 172.21.15.28:0/864064626 shutdown_connections 2024-04-25T01:36:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.589+0000 7fabc2c14740 1 -- 172.21.15.28:0/864064626 wait complete. 2024-04-25T01:36:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.589+0000 7fabc2c14740 1 -- start start 2024-04-25T01:36:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.589+0000 7fabc2c14740 1 Processor -- start 2024-04-25T01:36:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabc2c14740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559580614630 0x5595806347a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabc2c14740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 0x559580634ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabc2c14740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 0x559580631cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabc2c14740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fabb81affa0 con 0x55958061f620 2024-04-25T01:36:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabc2c14740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fabb81af8d0 con 0x559580614630 2024-04-25T01:36:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabc2c14740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fabb81acb10 con 0x559580614f50 2024-04-25T01:36:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabc2c14740 10 client.0 _ll_register_callbacks cb 0x7fabb0031430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabbe3c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 0x559580631cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabbe3c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 0x559580631cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48620/0 (socket says 172.21.15.28:48620) 2024-04-25T01:36:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabbe3c7640 1 -- 172.21.15.28:0/2894881566 learned_addr learned my addr 172.21.15.28:0/2894881566 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabbd3c5640 1 --2- 172.21.15.28:0/2894881566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 0x559580634ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabbdbc6640 1 --2- 172.21.15.28:0/2894881566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559580614630 0x5595806347a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabbe3c7640 1 -- 172.21.15.28:0/2894881566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 msgr2=0x559580634ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabbe3c7640 1 --2- 172.21.15.28:0/2894881566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 0x559580634ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabbe3c7640 1 -- 172.21.15.28:0/2894881566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559580614630 msgr2=0x5595806347a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabbe3c7640 1 --2- 172.21.15.28:0/2894881566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559580614630 0x5595806347a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabbe3c7640 1 -- 172.21.15.28:0/2894881566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595806323b0 con 0x55958061f620 2024-04-25T01:36:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabbe3c7640 1 --2- 172.21.15.28:0/2894881566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 0x559580631cc0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fabb0009430 tx=0x7fabb0031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1990b46072fdc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7faba17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7faba17fa640 1 -- 172.21.15.28:0/2894881566 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb0064060 con 0x55958061f620 2024-04-25T01:36:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7faba17fa640 1 -- 172.21.15.28:0/2894881566 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fabb0003d30 con 0x55958061f620 2024-04-25T01:36:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabc2c14740 1 -- 172.21.15.28:0/2894881566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabb0036040 con 0x55958061f620 2024-04-25T01:36:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7faba17fa640 1 -- 172.21.15.28:0/2894881566 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fabb00045a0 con 0x55958061f620 2024-04-25T01:36:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabc2c14740 10 client.16698 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.590+0000 7fabc2c14740 1 -- 172.21.15.28:0/2894881566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5595806323b0 con 0x55958061f620 2024-04-25T01:36:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.591+0000 7faba17fa640 1 -- 172.21.15.28:0/2894881566 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabb0036040 con 0x55958061f620 2024-04-25T01:36:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.591+0000 7faba17fa640 1 -- 172.21.15.28:0/2894881566 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fabb005e030 con 0x55958061f620 2024-04-25T01:36:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.591+0000 7fabc2c14740 10 client.16698 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.591+0000 7fabc2c14740 10 client.16698 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.591+0000 7fabc2c14740 20 client.16698 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.591+0000 7fabc2c14740 1 --2- 172.21.15.28:0/2894881566 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x559580695600 0x5595806979f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.591+0000 7fabc2c14740 4 client.16698 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:36:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.591+0000 7fabc2c14740 1 -- 172.21.15.28:0/2894881566 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559580580f40 con 0x559580695600 2024-04-25T01:36:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.591+0000 7fabbdbc6640 1 --2- 172.21.15.28:0/2894881566 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x559580695600 0x5595806979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.591+0000 7fabbdbc6640 1 --2- 172.21.15.28:0/2894881566 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x559580695600 0x5595806979f0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dc40d678d8888a9 server_cookie=737dc899c004893a in_seq=0 out_seq=0 2024-04-25T01:36:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.591+0000 7faba17fa640 10 client.16698 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.592+0000 7faba17fa640 1 -- 172.21.15.28:0/2894881566 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559580580f40 con 0x559580695600 2024-04-25T01:36:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.592+0000 7faba17fa640 10 client.16698 handle_command_reply: tid=0 2024-04-25T01:36:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.592+0000 7fabc2c14740 1 client.16698 shutdown 2024-04-25T01:36:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.592+0000 7fabc2c14740 1 -- 172.21.15.28:0/2894881566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 msgr2=0x559580631cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.592+0000 7fabc2c14740 1 --2- 172.21.15.28:0/2894881566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 0x559580631cc0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fabb0009430 tx=0x7fabb0031e60 comp rx=0 tx=0).stop 2024-04-25T01:36:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.592+0000 7fabc2c14740 1 -- 172.21.15.28:0/2894881566 shutdown_connections 2024-04-25T01:36:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.592+0000 7fabc2c14740 1 --2- 172.21.15.28:0/2894881566 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x559580695600 0x5595806979f0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.592+0000 7fabc2c14740 1 --2- 172.21.15.28:0/2894881566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55958061f620 0x559580631cc0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.592+0000 7fabc2c14740 1 --2- 172.21.15.28:0/2894881566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559580614f50 0x559580634ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.592+0000 7faba17fa640 0 client.16698 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.592+0000 7fabc2c14740 1 --2- 172.21.15.28:0/2894881566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559580614630 0x5595806347a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.592+0000 7fabc2c14740 1 -- 172.21.15.28:0/2894881566 >> 172.21.15.28:0/2894881566 conn(0x559580610950 msgr2=0x55958062a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.592+0000 7fabc2c14740 1 -- 172.21.15.28:0/2894881566 shutdown_connections 2024-04-25T01:36:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.592+0000 7fabc2c14740 1 -- 172.21.15.28:0/2894881566 wait complete. 2024-04-25T01:36:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.592+0000 7fabc2c14740 20 client.16698 trim_cache size 0 max 16384 2024-04-25T01:36:25.575 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:36:25.575 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:36:25.575 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-25T01:36:25.575 INFO:teuthology.orchestra.run.smithi028.stdout: "2b65b232-c036-46f6-90a8-6f14e32d4ed5": { 2024-04-25T01:36:25.575 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-25T01:36:25.575 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "2b65b232-c036-46f6-90a8-6f14e32d4ed5", 2024-04-25T01:36:25.575 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-25T01:36:25.575 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:36:25.575 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:36:25.576 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:36:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.593+0000 7fabbfe2a640 1 -- 172.21.15.28:0/1942124108 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7faba40a2d00 msgr2=0x7faba40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.593+0000 7fabbfe2a640 1 --2- 172.21.15.28:0/1942124108 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7faba40a2d00 0x7faba40a51c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fabb4005fd0 tx=0x7fabb400b390 comp rx=0 tx=0).stop 2024-04-25T01:36:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.593+0000 7fabbfe2a640 1 -- 172.21.15.28:0/1942124108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb81020d0 msgr2=0x7fabb8198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.593+0000 7fabbfe2a640 1 --2- 172.21.15.28:0/1942124108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb81020d0 0x7fabb8198590 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7faba800d950 tx=0x7faba800de10 comp rx=0 tx=0).stop 2024-04-25T01:36:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.593+0000 7fabbfe2a640 1 -- 172.21.15.28:0/1942124108 shutdown_connections 2024-04-25T01:36:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.593+0000 7fabbfe2a640 1 --2- 172.21.15.28:0/1942124108 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7faba40a2d00 0x7faba40a51c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.593+0000 7fabbfe2a640 1 --2- 172.21.15.28:0/1942124108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fabb8107720 0x7fabb819c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.593+0000 7fabbfe2a640 1 --2- 172.21.15.28:0/1942124108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb81020d0 0x7fabb8198590 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.593+0000 7fabbfe2a640 1 --2- 172.21.15.28:0/1942124108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fabb8101710 0x7fabb8198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.593+0000 7fabbfe2a640 1 -- 172.21.15.28:0/1942124108 >> 172.21.15.28:0/1942124108 conn(0x7fabb8078b90 msgr2=0x7fabb80ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.593+0000 7fabbfe2a640 1 -- 172.21.15.28:0/1942124108 shutdown_connections 2024-04-25T01:36:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:25.593+0000 7fabbfe2a640 1 -- 172.21.15.28:0/1942124108 wait complete. 2024-04-25T01:36:25.580 INFO:tasks.cephfs.filesystem:scrub status for tag:2b65b232-c036-46f6-90a8-6f14e32d4ed5 - {'path': '/', 'tag': '2b65b232-c036-46f6-90a8-6f14e32d4ed5', 'options': 'recursive,force'} 2024-04-25T01:36:26.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:26 smithi130 ceph-mon[32702]: pgmap v105: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:36:26.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:26 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:36:26.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:26 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:26.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:26 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:36:26.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:26 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:26.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:26 smithi032 ceph-mon[32461]: pgmap v105: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:36:26.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:26 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:36:26.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:26 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:26.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:26 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:36:26.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:26 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:26.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:26 smithi028 ceph-mon[29390]: pgmap v105: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:36:26.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:26 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:36:26.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:26 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:26.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:26 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:36:26.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:26 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:28.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:28 smithi130 ceph-mon[32702]: pgmap v106: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-04-25T01:36:28.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:28 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:36:28.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:28 smithi032 ceph-mon[32461]: pgmap v106: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-04-25T01:36:28.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:28 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:36:28.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:28 smithi028 ceph-mon[29390]: pgmap v106: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-04-25T01:36:28.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:28 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:36:30.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:30 smithi130 ceph-mon[32702]: pgmap v107: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T01:36:30.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:30 smithi032 ceph-mon[32461]: pgmap v107: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T01:36:30.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:30 smithi028 ceph-mon[29390]: pgmap v107: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T01:36:32.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:32 smithi130 ceph-mon[32702]: pgmap v108: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:36:32.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:32 smithi032 ceph-mon[32461]: pgmap v108: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:36:32.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:32 smithi028 ceph-mon[29390]: pgmap v108: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:36:34.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:34 smithi130 ceph-mon[32702]: pgmap v109: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-04-25T01:36:34.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:34 smithi032 ceph-mon[32461]: pgmap v109: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-04-25T01:36:34.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:34 smithi028 ceph-mon[29390]: pgmap v109: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-04-25T01:36:36.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:36 smithi130 ceph-mon[32702]: pgmap v110: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:36:36.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:36 smithi032 ceph-mon[32461]: pgmap v110: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:36:36.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:36 smithi028 ceph-mon[29390]: pgmap v110: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:36:38.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:38 smithi130 ceph-mon[32702]: pgmap v111: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-04-25T01:36:38.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:38 smithi032 ceph-mon[32461]: pgmap v111: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-04-25T01:36:38.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:38 smithi028 ceph-mon[29390]: pgmap v111: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-04-25T01:36:40.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:40 smithi130 ceph-mon[32702]: pgmap v112: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-25T01:36:40.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:40 smithi032 ceph-mon[32461]: pgmap v112: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-25T01:36:40.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:40 smithi028 ceph-mon[29390]: pgmap v112: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-25T01:36:42.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:42 smithi130 ceph-mon[32702]: pgmap v113: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:36:42.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:42 smithi032 ceph-mon[32461]: pgmap v113: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:36:42.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:42 smithi028 ceph-mon[29390]: pgmap v113: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:36:44.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:44 smithi130 ceph-mon[32702]: pgmap v114: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:36:44.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:44 smithi032 ceph-mon[32461]: pgmap v114: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:36:44.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:44 smithi028 ceph-mon[29390]: pgmap v114: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:36:46.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:46 smithi130 ceph-mon[32702]: pgmap v115: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:36:46.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:46 smithi032 ceph-mon[32461]: pgmap v115: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:36:46.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:46 smithi028 ceph-mon[29390]: pgmap v115: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:36:48.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:48 smithi130 ceph-mon[32702]: pgmap v116: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:36:48.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:48 smithi032 ceph-mon[32461]: pgmap v116: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:36:48.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:48 smithi028 ceph-mon[29390]: pgmap v116: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:36:50.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:50 smithi130 ceph-mon[32702]: pgmap v117: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:36:50.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:50 smithi032 ceph-mon[32461]: pgmap v117: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:36:50.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:50 smithi028 ceph-mon[29390]: pgmap v117: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:36:52.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:52 smithi130 ceph-mon[32702]: pgmap v118: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:36:52.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:52 smithi032 ceph-mon[32461]: pgmap v118: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:36:52.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:52 smithi028 ceph-mon[29390]: pgmap v118: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:36:54.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:54 smithi130 ceph-mon[32702]: pgmap v119: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:36:54.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:54 smithi032 ceph-mon[32461]: pgmap v119: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:36:54.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:54 smithi028 ceph-mon[29390]: pgmap v119: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:36:55.580 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:36:55.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:36:55.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.722+0000 7fed00210640 1 -- 172.21.15.28:0/1701161312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf8100610 msgr2=0x7fecf8100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:55.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.722+0000 7fed00210640 1 --2- 172.21.15.28:0/1701161312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf8100610 0x7fecf8100a90 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fece8009930 tx=0x7fece802a420 comp rx=0 tx=0).stop 2024-04-25T01:36:55.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.722+0000 7fed00210640 1 -- 172.21.15.28:0/1701161312 shutdown_connections 2024-04-25T01:36:55.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.722+0000 7fed00210640 1 --2- 172.21.15.28:0/1701161312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8100fd0 0x7fecf810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.722+0000 7fed00210640 1 --2- 172.21.15.28:0/1701161312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf8100610 0x7fecf8100a90 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.722+0000 7fed00210640 1 --2- 172.21.15.28:0/1701161312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fecf81066b0 0x7fecf8106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.722+0000 7fed00210640 1 -- 172.21.15.28:0/1701161312 >> 172.21.15.28:0/1701161312 conn(0x7fecf8077740 msgr2=0x7fecf8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:55.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.723+0000 7fed00210640 1 -- 172.21.15.28:0/1701161312 shutdown_connections 2024-04-25T01:36:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.723+0000 7fed00210640 1 -- 172.21.15.28:0/1701161312 wait complete. 2024-04-25T01:36:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.723+0000 7fed00210640 1 Processor -- start 2024-04-25T01:36:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.723+0000 7fed00210640 1 -- start start 2024-04-25T01:36:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.723+0000 7fed00210640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8100610 0x7fecf8198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.723+0000 7fed00210640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf8100fd0 0x7fecf81993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.723+0000 7fed00210640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fecf81066b0 0x7fecf81a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.723+0000 7fed00210640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81109f0 con 0x7fecf8100fd0 2024-04-25T01:36:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.723+0000 7fed00210640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fecf8110870 con 0x7fecf8100610 2024-04-25T01:36:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.723+0000 7fed00210640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fecf8110b70 con 0x7fecf81066b0 2024-04-25T01:36:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.724+0000 7fecfd7ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf8100fd0 0x7fecf81993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.724+0000 7fecfd7ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf8100fd0 0x7fecf81993b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40840/0 (socket says 172.21.15.28:40840) 2024-04-25T01:36:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.724+0000 7fecfd7ab640 1 -- 172.21.15.28:0/2703743981 learned_addr learned my addr 172.21.15.28:0/2703743981 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.724+0000 7fecfe7ad640 1 --2- 172.21.15.28:0/2703743981 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fecf81066b0 0x7fecf81a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.724+0000 7fecfdfac640 1 --2- 172.21.15.28:0/2703743981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8100610 0x7fecf8198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.724+0000 7fecfd7ab640 1 -- 172.21.15.28:0/2703743981 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fecf81066b0 msgr2=0x7fecf81a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.724+0000 7fecfd7ab640 1 --2- 172.21.15.28:0/2703743981 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fecf81066b0 0x7fecf81a3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.724+0000 7fecfd7ab640 1 -- 172.21.15.28:0/2703743981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8100610 msgr2=0x7fecf8198e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.724+0000 7fecfd7ab640 1 --2- 172.21.15.28:0/2703743981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8100610 0x7fecf8198e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.724+0000 7fecfd7ab640 1 -- 172.21.15.28:0/2703743981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf81a4430 con 0x7fecf8100fd0 2024-04-25T01:36:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.724+0000 7fecfdfac640 1 --2- 172.21.15.28:0/2703743981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8100610 0x7fecf8198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.724+0000 7fecfd7ab640 1 --2- 172.21.15.28:0/2703743981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf8100fd0 0x7fecf81993b0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fece8009230 tx=0x7fece8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.724+0000 7feceeffd640 1 -- 172.21.15.28:0/2703743981 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece803f030 con 0x7fecf8100fd0 2024-04-25T01:36:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.724+0000 7fed00210640 1 -- 172.21.15.28:0/2703743981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecf81a4600 con 0x7fecf8100fd0 2024-04-25T01:36:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.724+0000 7feceeffd640 1 -- 172.21.15.28:0/2703743981 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fece8003ad0 con 0x7fecf8100fd0 2024-04-25T01:36:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.724+0000 7feceeffd640 1 -- 172.21.15.28:0/2703743981 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece8035070 con 0x7fecf8100fd0 2024-04-25T01:36:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.724+0000 7fed00210640 1 -- 172.21.15.28:0/2703743981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecf81a4ac0 con 0x7fecf8100fd0 2024-04-25T01:36:55.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.727+0000 7feceeffd640 1 -- 172.21.15.28:0/2703743981 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fece8042030 con 0x7fecf8100fd0 2024-04-25T01:36:55.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.727+0000 7fed02ff5740 1 Processor -- start 2024-04-25T01:36:55.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.727+0000 7fed02ff5740 1 -- start start 2024-04-25T01:36:55.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.727+0000 7fed02ff5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858630 0x55b9a2858a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.727+0000 7fed02ff5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2858f50 0x55b9a28630e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.727+0000 7fed02ff5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2863620 0x55b9a2865a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.727+0000 7fed02ff5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b9a27bd6b0 con 0x55b9a2858630 2024-04-25T01:36:55.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.727+0000 7fed02ff5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b9a2765880 con 0x55b9a2858f50 2024-04-25T01:36:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.727+0000 7fed02ff5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b9a27b9670 con 0x55b9a2863620 2024-04-25T01:36:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.727+0000 7fecfe7ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858630 0x55b9a2858a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.727+0000 7fecfe7ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858630 0x55b9a2858a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40850/0 (socket says 172.21.15.28:40850) 2024-04-25T01:36:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.727+0000 7fecfe7ad640 1 -- 172.21.15.28:0/1645295191 learned_addr learned my addr 172.21.15.28:0/1645295191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.727+0000 7fecfdfac640 1 --2- 172.21.15.28:0/1645295191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2858f50 0x55b9a28630e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.728+0000 7fecfd7ab640 1 --2- 172.21.15.28:0/1645295191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2863620 0x55b9a2865a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.728+0000 7feceeffd640 1 --2- 172.21.15.28:0/2703743981 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fece40a2d00 0x7fece40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.728+0000 7fecfe7ad640 1 -- 172.21.15.28:0/1645295191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2863620 msgr2=0x55b9a2865a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.728+0000 7fecfe7ad640 1 --2- 172.21.15.28:0/1645295191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2863620 0x55b9a2865a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.728+0000 7fecfe7ad640 1 -- 172.21.15.28:0/1645295191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2858f50 msgr2=0x55b9a28630e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.728+0000 7fecfe7ad640 1 --2- 172.21.15.28:0/1645295191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2858f50 0x55b9a28630e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.728+0000 7fecfe7ad640 1 -- 172.21.15.28:0/1645295191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9a259b7d0 con 0x55b9a2858630 2024-04-25T01:36:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.728+0000 7fecfdfac640 1 --2- 172.21.15.28:0/1645295191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2858f50 0x55b9a28630e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:36:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.728+0000 7feceeffd640 1 -- 172.21.15.28:0/2703743981 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fece80efe60 con 0x7fecf8100fd0 2024-04-25T01:36:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.728+0000 7fecfe7ad640 1 --2- 172.21.15.28:0/1645295191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858630 0x55b9a2858a10 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fecf0009fc0 tx=0x7fecf0010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9daefffc21c2bd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.728+0000 7fecfe7ad640 1 --2- 172.21.15.28:0/2703743981 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fece40a2d00 0x7fece40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.728+0000 7fece3fff640 1 -- 172.21.15.28:0/1645295191 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf0003cd0 con 0x55b9a2858630 2024-04-25T01:36:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.728+0000 7fece3fff640 1 -- 172.21.15.28:0/1645295191 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecf0004590 con 0x55b9a2858630 2024-04-25T01:36:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.728+0000 7fece3fff640 1 -- 172.21.15.28:0/1645295191 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf001b420 con 0x55b9a2858630 2024-04-25T01:36:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.729+0000 7fed02ff5740 1 -- 172.21.15.28:0/1645295191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858630 msgr2=0x55b9a2858a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.729+0000 7fed02ff5740 1 --2- 172.21.15.28:0/1645295191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858630 0x55b9a2858a10 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fecf0009fc0 tx=0x7fecf0010860 comp rx=0 tx=0).stop 2024-04-25T01:36:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.729+0000 7fecfe7ad640 1 --2- 172.21.15.28:0/2703743981 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fece40a2d00 0x7fece40a51c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fecf0011fd0 tx=0x7fecf001bb60 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.729+0000 7fed02ff5740 1 -- 172.21.15.28:0/1645295191 shutdown_connections 2024-04-25T01:36:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.729+0000 7fed02ff5740 1 --2- 172.21.15.28:0/1645295191 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2863620 0x55b9a2865a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.729+0000 7fed02ff5740 1 --2- 172.21.15.28:0/1645295191 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2858f50 0x55b9a28630e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.729+0000 7fed02ff5740 1 --2- 172.21.15.28:0/1645295191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858630 0x55b9a2858a10 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.729+0000 7fed02ff5740 1 -- 172.21.15.28:0/1645295191 >> 172.21.15.28:0/1645295191 conn(0x55b9a2854950 msgr2=0x55b9a2854d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.729+0000 7fed02ff5740 1 -- 172.21.15.28:0/1645295191 shutdown_connections 2024-04-25T01:36:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.729+0000 7fed02ff5740 1 -- 172.21.15.28:0/1645295191 wait complete. 2024-04-25T01:36:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.730+0000 7fed02ff5740 1 -- start start 2024-04-25T01:36:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.730+0000 7fed02ff5740 1 Processor -- start 2024-04-25T01:36:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.730+0000 7fed02ff5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2858630 0x55b9a2860550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.730+0000 7fed02ff5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858f50 0x55b9a2860a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.730+0000 7fed02ff5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2863620 0x55b9a2860fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.730+0000 7fed02ff5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b9a2866410 con 0x55b9a2858f50 2024-04-25T01:36:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.730+0000 7fed02ff5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b9a2778f50 con 0x55b9a2863620 2024-04-25T01:36:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.730+0000 7fed02ff5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b9a2866590 con 0x55b9a2858630 2024-04-25T01:36:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.730+0000 7fed02ff5740 10 client.0 _ll_register_callbacks cb 0x55b9a27b4000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.730+0000 7fecfd7ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858f50 0x55b9a2860a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.730+0000 7fecfd7ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858f50 0x55b9a2860a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40858/0 (socket says 172.21.15.28:40858) 2024-04-25T01:36:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.730+0000 7fecfd7ab640 1 -- 172.21.15.28:0/789977201 learned_addr learned my addr 172.21.15.28:0/789977201 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.730+0000 7fecfdfac640 1 --2- 172.21.15.28:0/789977201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2858630 0x55b9a2860550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.730+0000 7fecfe7ad640 1 --2- 172.21.15.28:0/789977201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2863620 0x55b9a2860fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.731+0000 7fecfd7ab640 1 -- 172.21.15.28:0/789977201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2858630 msgr2=0x55b9a2860550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.731+0000 7fecfd7ab640 1 --2- 172.21.15.28:0/789977201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2858630 0x55b9a2860550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.731+0000 7fecfd7ab640 1 -- 172.21.15.28:0/789977201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2863620 msgr2=0x55b9a2860fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.731+0000 7fecfd7ab640 1 --2- 172.21.15.28:0/789977201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2863620 0x55b9a2860fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.731+0000 7fecfd7ab640 1 -- 172.21.15.28:0/789977201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9a24d13f0 con 0x55b9a2858f50 2024-04-25T01:36:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.731+0000 7fecfdfac640 1 --2- 172.21.15.28:0/789977201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2858630 0x55b9a2860550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.731+0000 7fecfd7ab640 1 --2- 172.21.15.28:0/789977201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858f50 0x55b9a2860a90 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fece80ed2d0 tx=0x7fece802a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab3bd59b1dc343f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.731+0000 7fece17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.731+0000 7fece17fa640 1 -- 172.21.15.28:0/789977201 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece8049050 con 0x55b9a2858f50 2024-04-25T01:36:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.731+0000 7fece17fa640 1 -- 172.21.15.28:0/789977201 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fece80ed4f0 con 0x55b9a2858f50 2024-04-25T01:36:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.731+0000 7fed02ff5740 1 -- 172.21.15.28:0/789977201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecf0004590 con 0x55b9a2858f50 2024-04-25T01:36:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.731+0000 7fece17fa640 1 -- 172.21.15.28:0/789977201 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fece80bbb80 con 0x55b9a2858f50 2024-04-25T01:36:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.731+0000 7fed02ff5740 10 client.16710 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.731+0000 7fed02ff5740 1 -- 172.21.15.28:0/789977201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b9a24910f0 con 0x55b9a2858f50 2024-04-25T01:36:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.731+0000 7fece17fa640 1 -- 172.21.15.28:0/789977201 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecf0004590 con 0x55b9a2858f50 2024-04-25T01:36:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.731+0000 7fece17fa640 1 -- 172.21.15.28:0/789977201 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fece80b86b0 con 0x55b9a2858f50 2024-04-25T01:36:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.731+0000 7fed02ff5740 10 client.16710 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.731+0000 7fed02ff5740 10 client.16710 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.732+0000 7fed02ff5740 20 client.16710 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.732+0000 7fed02ff5740 1 --2- 172.21.15.28:0/789977201 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b9a2873730 0x55b9a2875b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.732+0000 7fed02ff5740 4 client.16710 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.732+0000 7fed02ff5740 1 -- 172.21.15.28:0/789977201 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9a27cabf0 con 0x55b9a2873730 2024-04-25T01:36:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.732+0000 7fecfdfac640 1 --2- 172.21.15.28:0/789977201 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b9a2873730 0x55b9a2875b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.732+0000 7fecfdfac640 1 --2- 172.21.15.28:0/789977201 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b9a2873730 0x55b9a2875b60 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55beb6afe09e3d6d server_cookie=178bc2e753232739 in_seq=0 out_seq=0 2024-04-25T01:36:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.732+0000 7fece17fa640 10 client.16710 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.733+0000 7fece17fa640 1 -- 172.21.15.28:0/789977201 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b9a27cabf0 con 0x55b9a2873730 2024-04-25T01:36:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.733+0000 7fece17fa640 10 client.16710 handle_command_reply: tid=0 2024-04-25T01:36:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.733+0000 7fed02ff5740 1 client.16710 shutdown 2024-04-25T01:36:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.733+0000 7fed02ff5740 1 -- 172.21.15.28:0/789977201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858f50 msgr2=0x55b9a2860a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.733+0000 7fed02ff5740 1 --2- 172.21.15.28:0/789977201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858f50 0x55b9a2860a90 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fece80ed2d0 tx=0x7fece802a420 comp rx=0 tx=0).stop 2024-04-25T01:36:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.733+0000 7fed02ff5740 1 -- 172.21.15.28:0/789977201 shutdown_connections 2024-04-25T01:36:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.733+0000 7fed02ff5740 1 --2- 172.21.15.28:0/789977201 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b9a2873730 0x55b9a2875b60 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.733+0000 7fed02ff5740 1 --2- 172.21.15.28:0/789977201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2863620 0x55b9a2860fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.733+0000 7fed02ff5740 1 --2- 172.21.15.28:0/789977201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858f50 0x55b9a2860a90 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.733+0000 7fece17fa640 0 client.16710 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.733+0000 7fed02ff5740 1 --2- 172.21.15.28:0/789977201 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2858630 0x55b9a2860550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.733+0000 7fed02ff5740 1 -- 172.21.15.28:0/789977201 >> 172.21.15.28:0/789977201 conn(0x55b9a2854950 msgr2=0x55b9a285a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.733+0000 7fed02ff5740 1 -- 172.21.15.28:0/789977201 shutdown_connections 2024-04-25T01:36:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.733+0000 7fed02ff5740 1 -- 172.21.15.28:0/789977201 wait complete. 2024-04-25T01:36:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.734+0000 7fed02ff5740 20 client.16710 trim_cache size 0 max 16384 2024-04-25T01:36:55.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.751+0000 7fed02ff5740 1 Processor -- start 2024-04-25T01:36:55.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.751+0000 7fed02ff5740 1 -- start start 2024-04-25T01:36:55.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.751+0000 7fed02ff5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2858630 0x55b9a286f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.751+0000 7fed02ff5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2858f50 0x55b9a2854d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.751+0000 7fed02ff5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2863620 0x55b9a2862380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.751+0000 7fed02ff5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81aea20 con 0x55b9a2863620 2024-04-25T01:36:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.751+0000 7fed02ff5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81b0560 con 0x55b9a2858630 2024-04-25T01:36:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.751+0000 7fed02ff5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81af0f0 con 0x55b9a2858f50 2024-04-25T01:36:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.752+0000 7fecfe7ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2863620 0x55b9a2862380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.752+0000 7fecfe7ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2863620 0x55b9a2862380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40872/0 (socket says 172.21.15.28:40872) 2024-04-25T01:36:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.752+0000 7fecfe7ad640 1 -- 172.21.15.28:0/4245843360 learned_addr learned my addr 172.21.15.28:0/4245843360 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.752+0000 7fecfdfac640 1 --2- 172.21.15.28:0/4245843360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2858630 0x55b9a286f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.752+0000 7fecfd7ab640 1 --2- 172.21.15.28:0/4245843360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2858f50 0x55b9a2854d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.752+0000 7fecfe7ad640 1 -- 172.21.15.28:0/4245843360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2858f50 msgr2=0x55b9a2854d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.752+0000 7fecfe7ad640 1 --2- 172.21.15.28:0/4245843360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2858f50 0x55b9a2854d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.752+0000 7fecfe7ad640 1 -- 172.21.15.28:0/4245843360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2858630 msgr2=0x55b9a286f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.752+0000 7fecfe7ad640 1 --2- 172.21.15.28:0/4245843360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2858630 0x55b9a286f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.752+0000 7fecfe7ad640 1 -- 172.21.15.28:0/4245843360 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9a28761e0 con 0x55b9a2863620 2024-04-25T01:36:55.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.752+0000 7fecfe7ad640 1 --2- 172.21.15.28:0/4245843360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2863620 0x55b9a2862380 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fecf0009d60 tx=0x7fecf0026000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c38c38eb47b12f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:55.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.752+0000 7fece37fe640 1 -- 172.21.15.28:0/4245843360 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf0004000 con 0x55b9a2863620 2024-04-25T01:36:55.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.752+0000 7fece37fe640 1 -- 172.21.15.28:0/4245843360 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecf00041a0 con 0x55b9a2863620 2024-04-25T01:36:55.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.752+0000 7fece37fe640 1 -- 172.21.15.28:0/4245843360 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf00047f0 con 0x55b9a2863620 2024-04-25T01:36:55.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.753+0000 7fed02ff5740 1 -- 172.21.15.28:0/4245843360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2863620 msgr2=0x55b9a2862380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:55.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.753+0000 7fed02ff5740 1 --2- 172.21.15.28:0/4245843360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2863620 0x55b9a2862380 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fecf0009d60 tx=0x7fecf0026000 comp rx=0 tx=0).stop 2024-04-25T01:36:55.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.753+0000 7fed02ff5740 1 -- 172.21.15.28:0/4245843360 shutdown_connections 2024-04-25T01:36:55.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.753+0000 7fed02ff5740 1 --2- 172.21.15.28:0/4245843360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2863620 0x55b9a2862380 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.753+0000 7fed02ff5740 1 --2- 172.21.15.28:0/4245843360 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2858f50 0x55b9a2854d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.753+0000 7fed02ff5740 1 --2- 172.21.15.28:0/4245843360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2858630 0x55b9a286f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.753+0000 7fed02ff5740 1 -- 172.21.15.28:0/4245843360 >> 172.21.15.28:0/4245843360 conn(0x55b9a2854950 msgr2=0x55b9a2861c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:55.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.753+0000 7fed02ff5740 1 -- 172.21.15.28:0/4245843360 shutdown_connections 2024-04-25T01:36:55.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.753+0000 7fed02ff5740 1 -- 172.21.15.28:0/4245843360 wait complete. 2024-04-25T01:36:55.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.753+0000 7fed02ff5740 1 -- start start 2024-04-25T01:36:55.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.753+0000 7fed02ff5740 1 Processor -- start 2024-04-25T01:36:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fed02ff5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858630 0x55b9a2867270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fed02ff5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2858f50 0x55b9a28677b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fed02ff5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2863620 0x55b9a2867cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fed02ff5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81afe90 con 0x55b9a2858630 2024-04-25T01:36:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fed02ff5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81af7c0 con 0x55b9a2858f50 2024-04-25T01:36:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fed02ff5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fecf81aca00 con 0x55b9a2863620 2024-04-25T01:36:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fed02ff5740 10 client.0 _ll_register_callbacks cb 0x7fece8005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fecfdfac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858630 0x55b9a2867270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fecfdfac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858630 0x55b9a2867270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40878/0 (socket says 172.21.15.28:40878) 2024-04-25T01:36:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fecfdfac640 1 -- 172.21.15.28:0/3289005771 learned_addr learned my addr 172.21.15.28:0/3289005771 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fecfe7ad640 1 --2- 172.21.15.28:0/3289005771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2863620 0x55b9a2867cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fecfd7ab640 1 --2- 172.21.15.28:0/3289005771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2858f50 0x55b9a28677b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fecfdfac640 1 -- 172.21.15.28:0/3289005771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2863620 msgr2=0x55b9a2867cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fecfdfac640 1 --2- 172.21.15.28:0/3289005771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2863620 0x55b9a2867cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fecfdfac640 1 -- 172.21.15.28:0/3289005771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2858f50 msgr2=0x55b9a28677b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fecfdfac640 1 --2- 172.21.15.28:0/3289005771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2858f50 0x55b9a28677b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fecfdfac640 1 -- 172.21.15.28:0/3289005771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9a28dd120 con 0x55b9a2858630 2024-04-25T01:36:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fecfdfac640 1 --2- 172.21.15.28:0/3289005771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858630 0x55b9a2867270 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x55b9a28763e0 tx=0x7fecf400fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77849e82e9749db5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fece17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.754+0000 7fece17fa640 1 -- 172.21.15.28:0/3289005771 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf4005020 con 0x55b9a2858630 2024-04-25T01:36:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.755+0000 7fece17fa640 1 -- 172.21.15.28:0/3289005771 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fecf40051c0 con 0x55b9a2858630 2024-04-25T01:36:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.755+0000 7fed02ff5740 1 -- 172.21.15.28:0/3289005771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecf00041a0 con 0x55b9a2858630 2024-04-25T01:36:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.755+0000 7fece17fa640 1 -- 172.21.15.28:0/3289005771 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fecf402c050 con 0x55b9a2858630 2024-04-25T01:36:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.755+0000 7fed02ff5740 10 client.16716 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.755+0000 7fed02ff5740 1 -- 172.21.15.28:0/3289005771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b9a28de470 con 0x55b9a2858630 2024-04-25T01:36:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.755+0000 7fece17fa640 1 -- 172.21.15.28:0/3289005771 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecf00041a0 con 0x55b9a2858630 2024-04-25T01:36:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.755+0000 7fece17fa640 1 -- 172.21.15.28:0/3289005771 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fecf4024400 con 0x55b9a2858630 2024-04-25T01:36:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.755+0000 7fed02ff5740 10 client.16716 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.755+0000 7fed02ff5740 10 client.16716 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.755+0000 7fed02ff5740 20 client.16716 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.755+0000 7fed02ff5740 1 --2- 172.21.15.28:0/3289005771 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b9a28debe0 0x55b9a28e0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.755+0000 7fed02ff5740 4 client.16716 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:36:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.755+0000 7fed02ff5740 1 -- 172.21.15.28:0/3289005771 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b9a27c4f40 con 0x55b9a28debe0 2024-04-25T01:36:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.755+0000 7fecfd7ab640 1 --2- 172.21.15.28:0/3289005771 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b9a28debe0 0x55b9a28e0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.756+0000 7fecfd7ab640 1 --2- 172.21.15.28:0/3289005771 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b9a28debe0 0x55b9a28e0fd0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4851a226ad93e368 server_cookie=20eb2721d902eb7 in_seq=0 out_seq=0 2024-04-25T01:36:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.756+0000 7fece17fa640 10 client.16716 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.756+0000 7fece17fa640 1 -- 172.21.15.28:0/3289005771 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b9a27c4f40 con 0x55b9a28debe0 2024-04-25T01:36:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.756+0000 7fece17fa640 10 client.16716 handle_command_reply: tid=0 2024-04-25T01:36:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.756+0000 7fed02ff5740 1 client.16716 shutdown 2024-04-25T01:36:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.756+0000 7fed02ff5740 1 -- 172.21.15.28:0/3289005771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858630 msgr2=0x55b9a2867270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.756+0000 7fed02ff5740 1 --2- 172.21.15.28:0/3289005771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858630 0x55b9a2867270 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x55b9a28763e0 tx=0x7fecf400fae0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.756+0000 7fed02ff5740 1 -- 172.21.15.28:0/3289005771 shutdown_connections 2024-04-25T01:36:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.756+0000 7fed02ff5740 1 --2- 172.21.15.28:0/3289005771 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b9a28debe0 0x55b9a28e0fd0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.756+0000 7fed02ff5740 1 --2- 172.21.15.28:0/3289005771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b9a2863620 0x55b9a2867cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.756+0000 7fed02ff5740 1 --2- 172.21.15.28:0/3289005771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b9a2858f50 0x55b9a28677b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.756+0000 7fece17fa640 0 client.16716 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.756+0000 7fed02ff5740 1 --2- 172.21.15.28:0/3289005771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9a2858630 0x55b9a2867270 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.756+0000 7fed02ff5740 1 -- 172.21.15.28:0/3289005771 >> 172.21.15.28:0/3289005771 conn(0x55b9a2854950 msgr2=0x55b9a2860da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.756+0000 7fed02ff5740 1 -- 172.21.15.28:0/3289005771 shutdown_connections 2024-04-25T01:36:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.756+0000 7fed02ff5740 1 -- 172.21.15.28:0/3289005771 wait complete. 2024-04-25T01:36:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.756+0000 7fed02ff5740 20 client.16716 trim_cache size 0 max 16384 2024-04-25T01:36:55.741 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:36:55.741 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:36:55.741 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:36:55.741 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:36:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.757+0000 7fed00210640 1 -- 172.21.15.28:0/2703743981 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fece40a2d00 msgr2=0x7fece40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.757+0000 7fed00210640 1 --2- 172.21.15.28:0/2703743981 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fece40a2d00 0x7fece40a51c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fecf0011fd0 tx=0x7fecf001bb60 comp rx=0 tx=0).stop 2024-04-25T01:36:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.757+0000 7fed00210640 1 -- 172.21.15.28:0/2703743981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf8100fd0 msgr2=0x7fecf81993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.757+0000 7fed00210640 1 --2- 172.21.15.28:0/2703743981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf8100fd0 0x7fecf81993b0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fece8009230 tx=0x7fece8004270 comp rx=0 tx=0).stop 2024-04-25T01:36:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.757+0000 7fed00210640 1 -- 172.21.15.28:0/2703743981 shutdown_connections 2024-04-25T01:36:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.757+0000 7fed00210640 1 --2- 172.21.15.28:0/2703743981 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fece40a2d00 0x7fece40a51c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.757+0000 7fed00210640 1 --2- 172.21.15.28:0/2703743981 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fecf81066b0 0x7fecf81a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.757+0000 7fed00210640 1 --2- 172.21.15.28:0/2703743981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf8100fd0 0x7fecf81993b0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.757+0000 7fed00210640 1 --2- 172.21.15.28:0/2703743981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecf8100610 0x7fecf8198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.757+0000 7fed00210640 1 -- 172.21.15.28:0/2703743981 >> 172.21.15.28:0/2703743981 conn(0x7fecf8077740 msgr2=0x7fecf80ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.757+0000 7fed00210640 1 -- 172.21.15.28:0/2703743981 shutdown_connections 2024-04-25T01:36:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.757+0000 7fed00210640 1 -- 172.21.15.28:0/2703743981 wait complete. 2024-04-25T01:36:55.744 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:36:55.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:36:55.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.871+0000 7f00d3472640 1 -- 172.21.15.28:0/3235984514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00cc107750 msgr2=0x7f00cc107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:55.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.871+0000 7f00d3472640 1 --2- 172.21.15.28:0/3235984514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00cc107750 0x7f00cc107b30 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f00c8009900 tx=0x7f00c802a480 comp rx=0 tx=0).stop 2024-04-25T01:36:55.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.871+0000 7f00d3472640 1 -- 172.21.15.28:0/3235984514 shutdown_connections 2024-04-25T01:36:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.871+0000 7f00d3472640 1 --2- 172.21.15.28:0/3235984514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00cc102160 0x7f00cc10e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.871+0000 7f00d3472640 1 --2- 172.21.15.28:0/3235984514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f00cc1017c0 0x7f00cc101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.871+0000 7f00d3472640 1 --2- 172.21.15.28:0/3235984514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00cc107750 0x7f00cc107b30 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.871+0000 7f00d3472640 1 -- 172.21.15.28:0/3235984514 >> 172.21.15.28:0/3235984514 conn(0x7f00cc0788d0 msgr2=0x7f00cc078ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.871+0000 7f00d3472640 1 -- 172.21.15.28:0/3235984514 shutdown_connections 2024-04-25T01:36:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.871+0000 7f00d3472640 1 -- 172.21.15.28:0/3235984514 wait complete. 2024-04-25T01:36:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.871+0000 7f00d3472640 1 Processor -- start 2024-04-25T01:36:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.871+0000 7f00d3472640 1 -- start start 2024-04-25T01:36:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00d3472640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f00cc1017c0 0x7f00cc073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00d3472640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00cc102160 0x7f00cc073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00d3472640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00cc107750 0x7f00cc0744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00d3472640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00cc111ad0 con 0x7f00cc102160 2024-04-25T01:36:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00d3472640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f00cc111950 con 0x7f00cc107750 2024-04-25T01:36:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00d3472640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f00cc111c50 con 0x7f00cc1017c0 2024-04-25T01:36:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00d0a0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00cc102160 0x7f00cc073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00d0a0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00cc102160 0x7f00cc073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40898/0 (socket says 172.21.15.28:40898) 2024-04-25T01:36:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00d0a0d640 1 -- 172.21.15.28:0/112296319 learned_addr learned my addr 172.21.15.28:0/112296319 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00d1a0f640 1 --2- 172.21.15.28:0/112296319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00cc107750 0x7f00cc0744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00d0a0d640 1 -- 172.21.15.28:0/112296319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f00cc1017c0 msgr2=0x7f00cc073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00d120e640 1 --2- 172.21.15.28:0/112296319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f00cc1017c0 0x7f00cc073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00d0a0d640 1 --2- 172.21.15.28:0/112296319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f00cc1017c0 0x7f00cc073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00d0a0d640 1 -- 172.21.15.28:0/112296319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00cc107750 msgr2=0x7f00cc0744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00d0a0d640 1 --2- 172.21.15.28:0/112296319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00cc107750 0x7f00cc0744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00d0a0d640 1 -- 172.21.15.28:0/112296319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00cc1a64d0 con 0x7f00cc102160 2024-04-25T01:36:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00d1a0f640 1 --2- 172.21.15.28:0/112296319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00cc107750 0x7f00cc0744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00d0a0d640 1 --2- 172.21.15.28:0/112296319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00cc102160 0x7f00cc073f80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f00bc00c920 tx=0x7f00bc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00c27fc640 1 -- 172.21.15.28:0/112296319 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc007c70 con 0x7f00cc102160 2024-04-25T01:36:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00c27fc640 1 -- 172.21.15.28:0/112296319 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00bc007e10 con 0x7f00cc102160 2024-04-25T01:36:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00d3472640 1 -- 172.21.15.28:0/112296319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00cc1a67c0 con 0x7f00cc102160 2024-04-25T01:36:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.872+0000 7f00c27fc640 1 -- 172.21.15.28:0/112296319 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc018890 con 0x7f00cc102160 2024-04-25T01:36:55.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.873+0000 7f00d3472640 1 -- 172.21.15.28:0/112296319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00cc1a6d10 con 0x7f00cc102160 2024-04-25T01:36:55.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.874+0000 7f00d3472640 1 -- 172.21.15.28:0/112296319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00cc0689b0 con 0x7f00cc102160 2024-04-25T01:36:55.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.875+0000 7f00c27fc640 1 -- 172.21.15.28:0/112296319 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f00bc018a30 con 0x7f00cc102160 2024-04-25T01:36:55.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.875+0000 7f00c27fc640 1 --2- 172.21.15.28:0/112296319 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f00b80a2d30 0x7f00b80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:55.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.875+0000 7f00c27fc640 1 -- 172.21.15.28:0/112296319 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f00bc0cf950 con 0x7f00cc102160 2024-04-25T01:36:55.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.875+0000 7f00d120e640 1 --2- 172.21.15.28:0/112296319 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f00b80a2d30 0x7f00b80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:55.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.876+0000 7f00d120e640 1 --2- 172.21.15.28:0/112296319 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f00b80a2d30 0x7f00b80a51f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f00c80096c0 tx=0x7f00c8000bc0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:55.880+0000 7f00c27fc640 1 -- 172.21.15.28:0/112296319 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f00bc014030 con 0x7f00cc102160 2024-04-25T01:36:56.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.101+0000 7f00d3472640 1 -- 172.21.15.28:0/112296319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00cc074f20 con 0x7f00cc102160 2024-04-25T01:36:56.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.102+0000 7f00c27fc640 1 -- 172.21.15.28:0/112296319 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7f00bc01f300 con 0x7f00cc102160 2024-04-25T01:36:56.079 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:36:56.079 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:36:56.079 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:36:56.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.104+0000 7f00d3472640 1 -- 172.21.15.28:0/112296319 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f00b80a2d30 msgr2=0x7f00b80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.104+0000 7f00d3472640 1 --2- 172.21.15.28:0/112296319 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f00b80a2d30 0x7f00b80a51f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f00c80096c0 tx=0x7f00c8000bc0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.104+0000 7f00d3472640 1 -- 172.21.15.28:0/112296319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00cc102160 msgr2=0x7f00cc073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.104+0000 7f00d3472640 1 --2- 172.21.15.28:0/112296319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00cc102160 0x7f00cc073f80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f00bc00c920 tx=0x7f00bc00cde0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.104+0000 7f00d3472640 1 -- 172.21.15.28:0/112296319 shutdown_connections 2024-04-25T01:36:56.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.104+0000 7f00d3472640 1 --2- 172.21.15.28:0/112296319 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f00b80a2d30 0x7f00b80a51f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.104+0000 7f00d3472640 1 --2- 172.21.15.28:0/112296319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f00cc107750 0x7f00cc0744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.104+0000 7f00d3472640 1 --2- 172.21.15.28:0/112296319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00cc102160 0x7f00cc073f80 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.104+0000 7f00d3472640 1 --2- 172.21.15.28:0/112296319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f00cc1017c0 0x7f00cc073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.104+0000 7f00d3472640 1 -- 172.21.15.28:0/112296319 >> 172.21.15.28:0/112296319 conn(0x7f00cc0788d0 msgr2=0x7f00cc100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.104+0000 7f00d3472640 1 -- 172.21.15.28:0/112296319 shutdown_connections 2024-04-25T01:36:56.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.104+0000 7f00d3472640 1 -- 172.21.15.28:0/112296319 wait complete. 2024-04-25T01:36:56.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:36:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.219+0000 7f1a703dc640 1 -- 172.21.15.28:0/3885982595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a68100540 msgr2=0x7f1a68100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.219+0000 7f1a703dc640 1 --2- 172.21.15.28:0/3885982595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a68100540 0x7f1a68100920 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f1a58009900 tx=0x7f1a5802a410 comp rx=0 tx=0).stop 2024-04-25T01:36:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.220+0000 7f1a703dc640 1 -- 172.21.15.28:0/3885982595 shutdown_connections 2024-04-25T01:36:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.220+0000 7f1a703dc640 1 --2- 172.21.15.28:0/3885982595 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a6810e0f0 0x7f1a681104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.220+0000 7f1a703dc640 1 --2- 172.21.15.28:0/3885982595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a68100e60 0x7f1a6810dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.220+0000 7f1a703dc640 1 --2- 172.21.15.28:0/3885982595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a68100540 0x7f1a68100920 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.220+0000 7f1a703dc640 1 -- 172.21.15.28:0/3885982595 >> 172.21.15.28:0/3885982595 conn(0x7f1a68077740 msgr2=0x7f1a68077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.220+0000 7f1a703dc640 1 -- 172.21.15.28:0/3885982595 shutdown_connections 2024-04-25T01:36:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.220+0000 7f1a703dc640 1 -- 172.21.15.28:0/3885982595 wait complete. 2024-04-25T01:36:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.220+0000 7f1a703dc640 1 Processor -- start 2024-04-25T01:36:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.220+0000 7f1a703dc640 1 -- start start 2024-04-25T01:36:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.220+0000 7f1a703dc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a68100540 0x7f1a6819a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.220+0000 7f1a703dc640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a68100e60 0x7f1a6819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.220+0000 7f1a703dc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a6810e0f0 0x7f1a6819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.220+0000 7f1a703dc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1a68113cf0 con 0x7f1a68100540 2024-04-25T01:36:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.220+0000 7f1a703dc640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1a68113b70 con 0x7f1a6810e0f0 2024-04-25T01:36:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.220+0000 7f1a703dc640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1a68113e70 con 0x7f1a68100e60 2024-04-25T01:36:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.221+0000 7f1a6e178640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a68100540 0x7f1a6819a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.221+0000 7f1a6e178640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a68100540 0x7f1a6819a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40906/0 (socket says 172.21.15.28:40906) 2024-04-25T01:36:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.221+0000 7f1a6e178640 1 -- 172.21.15.28:0/870131354 learned_addr learned my addr 172.21.15.28:0/870131354 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.221+0000 7f1a6e979640 1 --2- 172.21.15.28:0/870131354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a6810e0f0 0x7f1a6819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.221+0000 7f1a6d977640 1 --2- 172.21.15.28:0/870131354 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a68100e60 0x7f1a6819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.221+0000 7f1a6e178640 1 -- 172.21.15.28:0/870131354 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a68100e60 msgr2=0x7f1a6819a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.221+0000 7f1a6e178640 1 --2- 172.21.15.28:0/870131354 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a68100e60 0x7f1a6819a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.221+0000 7f1a6e178640 1 -- 172.21.15.28:0/870131354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a6810e0f0 msgr2=0x7f1a6819ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.221+0000 7f1a6e178640 1 --2- 172.21.15.28:0/870131354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a6810e0f0 0x7f1a6819ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.221+0000 7f1a6e178640 1 -- 172.21.15.28:0/870131354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a6819f200 con 0x7f1a68100540 2024-04-25T01:36:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.221+0000 7f1a6e178640 1 --2- 172.21.15.28:0/870131354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a68100540 0x7f1a6819a210 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f1a58006010 tx=0x7f1a580042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.221+0000 7f1a5f7fe640 1 -- 172.21.15.28:0/870131354 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5803f030 con 0x7f1a68100540 2024-04-25T01:36:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.221+0000 7f1a5f7fe640 1 -- 172.21.15.28:0/870131354 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a58003ce0 con 0x7f1a68100540 2024-04-25T01:36:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.221+0000 7f1a703dc640 1 -- 172.21.15.28:0/870131354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a6819f490 con 0x7f1a68100540 2024-04-25T01:36:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.221+0000 7f1a5f7fe640 1 -- 172.21.15.28:0/870131354 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a58035070 con 0x7f1a68100540 2024-04-25T01:36:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.221+0000 7f1a703dc640 1 -- 172.21.15.28:0/870131354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a681a6d80 con 0x7f1a68100540 2024-04-25T01:36:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.223+0000 7f1a5f7fe640 1 -- 172.21.15.28:0/870131354 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1a58042030 con 0x7f1a68100540 2024-04-25T01:36:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.224+0000 7f1a731ca740 1 Processor -- start 2024-04-25T01:36:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.224+0000 7f1a731ca740 1 -- start start 2024-04-25T01:36:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.224+0000 7f1a731ca740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0580 0x5654124e0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.224+0000 7f1a731ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0ea0 0x5654124eb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.224+0000 7f1a731ca740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124eb570 0x5654124ed960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.224+0000 7f1a731ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x565412445670 con 0x5654124e0ea0 2024-04-25T01:36:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.224+0000 7f1a731ca740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5654123ed970 con 0x5654124e0580 2024-04-25T01:36:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.224+0000 7f1a731ca740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x565412441630 con 0x5654124eb570 2024-04-25T01:36:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.224+0000 7f1a5f7fe640 1 --2- 172.21.15.28:0/870131354 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1a500a2d00 0x7f1a500a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.224+0000 7f1a6e178640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0ea0 0x5654124eb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.224+0000 7f1a6e979640 1 --2- 172.21.15.28:0/870131354 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1a500a2d00 0x7f1a500a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.224+0000 7f1a6e178640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0ea0 0x5654124eb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40916/0 (socket says 172.21.15.28:40916) 2024-04-25T01:36:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.225+0000 7f1a6e979640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0580 0x5654124e0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.225+0000 7f1a6e178640 1 -- 172.21.15.28:0/157611363 learned_addr learned my addr 172.21.15.28:0/157611363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.225+0000 7f1a6d977640 1 --2- 172.21.15.28:0/157611363 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124eb570 0x5654124ed960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.225+0000 7f1a5f7fe640 1 -- 172.21.15.28:0/870131354 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1a580efe00 con 0x7f1a68100540 2024-04-25T01:36:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.225+0000 7f1a6e178640 1 -- 172.21.15.28:0/157611363 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124eb570 msgr2=0x5654124ed960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.225+0000 7f1a6e178640 1 --2- 172.21.15.28:0/157611363 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124eb570 0x5654124ed960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.225+0000 7f1a6e178640 1 -- 172.21.15.28:0/157611363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0580 msgr2=0x5654124e0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.225+0000 7f1a6e178640 1 --2- 172.21.15.28:0/157611363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0580 0x5654124e0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.225+0000 7f1a6e178640 1 -- 172.21.15.28:0/157611363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565412411400 con 0x5654124e0ea0 2024-04-25T01:36:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.225+0000 7f1a6e979640 1 --2- 172.21.15.28:0/870131354 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1a500a2d00 0x7f1a500a51c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1a600098b0 tx=0x7f1a60009d90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.225+0000 7f1a6e178640 1 --2- 172.21.15.28:0/157611363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0ea0 0x5654124eb030 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f1a580bbfd0 tx=0x7f1a580044e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b09d16e2f02188f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.225+0000 7f1a5cbfe640 1 -- 172.21.15.28:0/157611363 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a580c4f00 con 0x5654124e0ea0 2024-04-25T01:36:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.225+0000 7f1a5cbfe640 1 -- 172.21.15.28:0/157611363 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a580bc740 con 0x5654124e0ea0 2024-04-25T01:36:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.225+0000 7f1a5cbfe640 1 -- 172.21.15.28:0/157611363 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a580b86b0 con 0x5654124e0ea0 2024-04-25T01:36:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.226+0000 7f1a731ca740 1 -- 172.21.15.28:0/157611363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0ea0 msgr2=0x5654124eb030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.226+0000 7f1a731ca740 1 --2- 172.21.15.28:0/157611363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0ea0 0x5654124eb030 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f1a580bbfd0 tx=0x7f1a580044e0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.226+0000 7f1a731ca740 1 -- 172.21.15.28:0/157611363 shutdown_connections 2024-04-25T01:36:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.226+0000 7f1a731ca740 1 --2- 172.21.15.28:0/157611363 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124eb570 0x5654124ed960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.226+0000 7f1a731ca740 1 --2- 172.21.15.28:0/157611363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0ea0 0x5654124eb030 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.226+0000 7f1a731ca740 1 --2- 172.21.15.28:0/157611363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0580 0x5654124e0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.226+0000 7f1a731ca740 1 -- 172.21.15.28:0/157611363 >> 172.21.15.28:0/157611363 conn(0x5654124dc8a0 msgr2=0x5654124dcc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.226+0000 7f1a731ca740 1 -- 172.21.15.28:0/157611363 shutdown_connections 2024-04-25T01:36:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.226+0000 7f1a731ca740 1 -- 172.21.15.28:0/157611363 wait complete. 2024-04-25T01:36:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.226+0000 7f1a731ca740 1 -- start start 2024-04-25T01:36:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a731ca740 1 Processor -- start 2024-04-25T01:36:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a731ca740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0580 0x5654124eaf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a731ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0ea0 0x5654124e8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a731ca740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124eb570 0x5654124e8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a731ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x565412445670 con 0x5654124e0ea0 2024-04-25T01:36:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a731ca740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x565412401050 con 0x5654124e0580 2024-04-25T01:36:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a731ca740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5654124ee120 con 0x5654124eb570 2024-04-25T01:36:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a731ca740 10 client.0 _ll_register_callbacks cb 0x565412158be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a6d977640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0ea0 0x5654124e8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a6d977640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0ea0 0x5654124e8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40926/0 (socket says 172.21.15.28:40926) 2024-04-25T01:36:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a6d977640 1 -- 172.21.15.28:0/2502162300 learned_addr learned my addr 172.21.15.28:0/2502162300 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a6e979640 1 --2- 172.21.15.28:0/2502162300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124eb570 0x5654124e8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a6e178640 1 --2- 172.21.15.28:0/2502162300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0580 0x5654124eaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a6d977640 1 -- 172.21.15.28:0/2502162300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124eb570 msgr2=0x5654124e8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a6d977640 1 --2- 172.21.15.28:0/2502162300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124eb570 0x5654124e8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a6d977640 1 -- 172.21.15.28:0/2502162300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0580 msgr2=0x5654124eaf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a6d977640 1 --2- 172.21.15.28:0/2502162300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0580 0x5654124eaf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a6d977640 1 -- 172.21.15.28:0/2502162300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654123d2dc0 con 0x5654124e0ea0 2024-04-25T01:36:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a6e979640 1 --2- 172.21.15.28:0/2502162300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124eb570 0x5654124e8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a6d977640 1 --2- 172.21.15.28:0/2502162300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0ea0 0x5654124e8230 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f1a6400da10 tx=0x7f1a6400ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfbbb451f98ae790 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.227+0000 7f1a4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.228+0000 7f1a4dffb640 1 -- 172.21.15.28:0/2502162300 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6401b440 con 0x5654124e0ea0 2024-04-25T01:36:56.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.228+0000 7f1a4dffb640 1 -- 172.21.15.28:0/2502162300 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1a64011440 con 0x5654124e0ea0 2024-04-25T01:36:56.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.228+0000 7f1a731ca740 1 -- 172.21.15.28:0/2502162300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a580bc740 con 0x5654124e0ea0 2024-04-25T01:36:56.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.228+0000 7f1a4dffb640 1 -- 172.21.15.28:0/2502162300 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1a64011640 con 0x5654124e0ea0 2024-04-25T01:36:56.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.228+0000 7f1a731ca740 10 client.16734 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:56.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.228+0000 7f1a731ca740 1 -- 172.21.15.28:0/2502162300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5654121593f0 con 0x5654124e0ea0 2024-04-25T01:36:56.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.228+0000 7f1a4dffb640 1 -- 172.21.15.28:0/2502162300 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a580bc740 con 0x5654124e0ea0 2024-04-25T01:36:56.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.228+0000 7f1a4dffb640 1 -- 172.21.15.28:0/2502162300 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1a64019020 con 0x5654124e0ea0 2024-04-25T01:36:56.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.228+0000 7f1a731ca740 10 client.16734 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:56.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.228+0000 7f1a731ca740 10 client.16734 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:36:56.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.228+0000 7f1a731ca740 20 client.16734 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:56.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.228+0000 7f1a731ca740 1 --2- 172.21.15.28:0/2502162300 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5654124fb3c0 0x5654124fd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.228+0000 7f1a731ca740 4 client.16734 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:56.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.228+0000 7f1a731ca740 1 -- 172.21.15.28:0/2502162300 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565412452bb0 con 0x5654124fb3c0 2024-04-25T01:36:56.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.229+0000 7f1a6e178640 1 --2- 172.21.15.28:0/2502162300 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5654124fb3c0 0x5654124fd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.229+0000 7f1a4dffb640 10 client.16734 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:36:56.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.229+0000 7f1a6e178640 1 --2- 172.21.15.28:0/2502162300 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5654124fb3c0 0x5654124fd800 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=51fde1249d9ae1c server_cookie=5a13951a8ef70c27 in_seq=0 out_seq=0 2024-04-25T01:36:56.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.230+0000 7f1a4dffb640 1 -- 172.21.15.28:0/2502162300 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x565412452bb0 con 0x5654124fb3c0 2024-04-25T01:36:56.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.230+0000 7f1a4dffb640 10 client.16734 handle_command_reply: tid=0 2024-04-25T01:36:56.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.230+0000 7f1a731ca740 1 client.16734 shutdown 2024-04-25T01:36:56.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.230+0000 7f1a731ca740 1 -- 172.21.15.28:0/2502162300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0ea0 msgr2=0x5654124e8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.230+0000 7f1a731ca740 1 --2- 172.21.15.28:0/2502162300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0ea0 0x5654124e8230 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f1a6400da10 tx=0x7f1a6400ded0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.230+0000 7f1a731ca740 1 -- 172.21.15.28:0/2502162300 shutdown_connections 2024-04-25T01:36:56.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.230+0000 7f1a731ca740 1 --2- 172.21.15.28:0/2502162300 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5654124fb3c0 0x5654124fd800 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.230+0000 7f1a731ca740 1 --2- 172.21.15.28:0/2502162300 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124eb570 0x5654124e8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.230+0000 7f1a4dffb640 0 client.16734 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:36:56.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.230+0000 7f1a731ca740 1 --2- 172.21.15.28:0/2502162300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0ea0 0x5654124e8230 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.230+0000 7f1a731ca740 1 --2- 172.21.15.28:0/2502162300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0580 0x5654124eaf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.230+0000 7f1a731ca740 1 -- 172.21.15.28:0/2502162300 >> 172.21.15.28:0/2502162300 conn(0x5654124dc8a0 msgr2=0x5654124dbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.230+0000 7f1a731ca740 1 -- 172.21.15.28:0/2502162300 shutdown_connections 2024-04-25T01:36:56.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.230+0000 7f1a731ca740 1 -- 172.21.15.28:0/2502162300 wait complete. 2024-04-25T01:36:56.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.230+0000 7f1a731ca740 20 client.16734 trim_cache size 0 max 16384 2024-04-25T01:36:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.243+0000 7f1a731ca740 1 Processor -- start 2024-04-25T01:36:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.243+0000 7f1a731ca740 1 -- start start 2024-04-25T01:36:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.243+0000 7f1a731ca740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0580 0x5654124f6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.243+0000 7f1a731ca740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124e0ea0 0x565412503110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.243+0000 7f1a731ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124eb570 0x565412503650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.243+0000 7f1a731ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1a681b0ca0 con 0x5654124eb570 2024-04-25T01:36:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.243+0000 7f1a731ca740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1a681b27e0 con 0x5654124e0580 2024-04-25T01:36:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.243+0000 7f1a731ca740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1a681b1370 con 0x5654124e0ea0 2024-04-25T01:36:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.244+0000 7f1a6e979640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124eb570 0x565412503650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.244+0000 7f1a6e979640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124eb570 0x565412503650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40928/0 (socket says 172.21.15.28:40928) 2024-04-25T01:36:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.244+0000 7f1a6e979640 1 -- 172.21.15.28:0/1307708932 learned_addr learned my addr 172.21.15.28:0/1307708932 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.244+0000 7f1a6e178640 1 --2- 172.21.15.28:0/1307708932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0580 0x5654124f6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.244+0000 7f1a6d977640 1 --2- 172.21.15.28:0/1307708932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124e0ea0 0x565412503110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.244+0000 7f1a6e979640 1 -- 172.21.15.28:0/1307708932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124e0ea0 msgr2=0x565412503110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.244+0000 7f1a6e979640 1 --2- 172.21.15.28:0/1307708932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124e0ea0 0x565412503110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.244+0000 7f1a6e979640 1 -- 172.21.15.28:0/1307708932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0580 msgr2=0x5654124f6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.244+0000 7f1a6e979640 1 --2- 172.21.15.28:0/1307708932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0580 0x5654124f6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.244+0000 7f1a6e979640 1 -- 172.21.15.28:0/1307708932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654122237d0 con 0x5654124eb570 2024-04-25T01:36:56.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.244+0000 7f1a6e979640 1 --2- 172.21.15.28:0/1307708932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124eb570 0x565412503650 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f1a60012660 tx=0x7f1a6001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37aa25eda8987d32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.244+0000 7f1a4ffff640 1 -- 172.21.15.28:0/1307708932 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a60003bf0 con 0x5654124eb570 2024-04-25T01:36:56.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.244+0000 7f1a4ffff640 1 -- 172.21.15.28:0/1307708932 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a60025070 con 0x5654124eb570 2024-04-25T01:36:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.244+0000 7f1a4ffff640 1 -- 172.21.15.28:0/1307708932 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a60005020 con 0x5654124eb570 2024-04-25T01:36:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.245+0000 7f1a731ca740 1 -- 172.21.15.28:0/1307708932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124eb570 msgr2=0x565412503650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.245+0000 7f1a731ca740 1 --2- 172.21.15.28:0/1307708932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124eb570 0x565412503650 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f1a60012660 tx=0x7f1a6001f000 comp rx=0 tx=0).stop 2024-04-25T01:36:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.245+0000 7f1a731ca740 1 -- 172.21.15.28:0/1307708932 shutdown_connections 2024-04-25T01:36:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.245+0000 7f1a731ca740 1 --2- 172.21.15.28:0/1307708932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124eb570 0x565412503650 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.245+0000 7f1a731ca740 1 --2- 172.21.15.28:0/1307708932 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124e0ea0 0x565412503110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.245+0000 7f1a731ca740 1 --2- 172.21.15.28:0/1307708932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0580 0x5654124f6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.245+0000 7f1a731ca740 1 -- 172.21.15.28:0/1307708932 >> 172.21.15.28:0/1307708932 conn(0x5654124dc8a0 msgr2=0x5654124db2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.245+0000 7f1a731ca740 1 -- 172.21.15.28:0/1307708932 shutdown_connections 2024-04-25T01:36:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.245+0000 7f1a731ca740 1 -- 172.21.15.28:0/1307708932 wait complete. 2024-04-25T01:36:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.245+0000 7f1a731ca740 1 -- start start 2024-04-25T01:36:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.245+0000 7f1a731ca740 1 Processor -- start 2024-04-25T01:36:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.245+0000 7f1a731ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0580 0x5654124f1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.245+0000 7f1a731ca740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0ea0 0x5654124f1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.245+0000 7f1a731ca740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124eb570 0x565412560990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.245+0000 7f1a731ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1a681b2110 con 0x5654124e0580 2024-04-25T01:36:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.245+0000 7f1a731ca740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1a681b1a40 con 0x5654124e0ea0 2024-04-25T01:36:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.245+0000 7f1a731ca740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1a681aec80 con 0x5654124eb570 2024-04-25T01:36:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.245+0000 7f1a731ca740 10 client.0 _ll_register_callbacks cb 0x7f1a64004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.246+0000 7f1a6e178640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0580 0x5654124f1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.246+0000 7f1a6e178640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0580 0x5654124f1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40936/0 (socket says 172.21.15.28:40936) 2024-04-25T01:36:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.246+0000 7f1a6e178640 1 -- 172.21.15.28:0/3461626699 learned_addr learned my addr 172.21.15.28:0/3461626699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.246+0000 7f1a6e178640 1 -- 172.21.15.28:0/3461626699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124eb570 msgr2=0x565412560990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.246+0000 7f1a6e178640 1 --2- 172.21.15.28:0/3461626699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124eb570 0x565412560990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.246+0000 7f1a6d977640 1 --2- 172.21.15.28:0/3461626699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0ea0 0x5654124f1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.246+0000 7f1a6e178640 1 -- 172.21.15.28:0/3461626699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0ea0 msgr2=0x5654124f1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.246+0000 7f1a6e178640 1 --2- 172.21.15.28:0/3461626699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0ea0 0x5654124f1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.246+0000 7f1a6e178640 1 -- 172.21.15.28:0/3461626699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565412560ed0 con 0x5654124e0580 2024-04-25T01:36:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.246+0000 7f1a6e178640 1 --2- 172.21.15.28:0/3461626699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0580 0x5654124f1440 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1a580b8360 tx=0x7f1a580092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f8f7e458d19badb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.246+0000 7f1a4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.246+0000 7f1a4dffb640 1 -- 172.21.15.28:0/3461626699 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a580c4f00 con 0x5654124e0580 2024-04-25T01:36:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.246+0000 7f1a4dffb640 1 -- 172.21.15.28:0/3461626699 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1a5803bd40 con 0x5654124e0580 2024-04-25T01:36:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.246+0000 7f1a731ca740 1 -- 172.21.15.28:0/3461626699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a60025070 con 0x5654124e0580 2024-04-25T01:36:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.246+0000 7f1a4dffb640 1 -- 172.21.15.28:0/3461626699 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1a5803b460 con 0x5654124e0580 2024-04-25T01:36:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.246+0000 7f1a731ca740 10 client.16740 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.246+0000 7f1a731ca740 1 -- 172.21.15.28:0/3461626699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x565412562130 con 0x5654124e0580 2024-04-25T01:36:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.246+0000 7f1a4dffb640 1 -- 172.21.15.28:0/3461626699 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a60025070 con 0x5654124e0580 2024-04-25T01:36:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.247+0000 7f1a4dffb640 1 -- 172.21.15.28:0/3461626699 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1a580b86b0 con 0x5654124e0580 2024-04-25T01:36:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.247+0000 7f1a731ca740 10 client.16740 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.247+0000 7f1a731ca740 10 client.16740 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:36:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.247+0000 7f1a731ca740 20 client.16740 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.247+0000 7f1a731ca740 1 --2- 172.21.15.28:0/3461626699 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5654124fb3c0 0x565412564560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.247+0000 7f1a731ca740 4 client.16740 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.247+0000 7f1a731ca740 1 -- 172.21.15.28:0/3461626699 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56541244cf00 con 0x5654124fb3c0 2024-04-25T01:36:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.247+0000 7f1a6d977640 1 --2- 172.21.15.28:0/3461626699 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5654124fb3c0 0x565412564560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.247+0000 7f1a6d977640 1 --2- 172.21.15.28:0/3461626699 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5654124fb3c0 0x565412564560 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4dd73e34250ded79 server_cookie=5be0c93e95fa653 in_seq=0 out_seq=0 2024-04-25T01:36:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.247+0000 7f1a4dffb640 10 client.16740 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:36:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.247+0000 7f1a4dffb640 1 -- 172.21.15.28:0/3461626699 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56541244cf00 con 0x5654124fb3c0 2024-04-25T01:36:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.247+0000 7f1a4dffb640 10 client.16740 handle_command_reply: tid=0 2024-04-25T01:36:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.247+0000 7f1a731ca740 1 client.16740 shutdown 2024-04-25T01:36:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a731ca740 1 -- 172.21.15.28:0/3461626699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0580 msgr2=0x5654124f1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a731ca740 1 --2- 172.21.15.28:0/3461626699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0580 0x5654124f1440 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1a580b8360 tx=0x7f1a580092d0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a731ca740 1 -- 172.21.15.28:0/3461626699 shutdown_connections 2024-04-25T01:36:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a731ca740 1 --2- 172.21.15.28:0/3461626699 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5654124fb3c0 0x565412564560 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a731ca740 1 --2- 172.21.15.28:0/3461626699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5654124eb570 0x565412560990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a731ca740 1 --2- 172.21.15.28:0/3461626699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5654124e0ea0 0x5654124f1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a4dffb640 0 client.16740 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:36:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a731ca740 1 --2- 172.21.15.28:0/3461626699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654124e0580 0x5654124f1440 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a731ca740 1 -- 172.21.15.28:0/3461626699 >> 172.21.15.28:0/3461626699 conn(0x5654124dc8a0 msgr2=0x5654124dbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a731ca740 1 -- 172.21.15.28:0/3461626699 shutdown_connections 2024-04-25T01:36:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a731ca740 1 -- 172.21.15.28:0/3461626699 wait complete. 2024-04-25T01:36:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a731ca740 20 client.16740 trim_cache size 0 max 16384 2024-04-25T01:36:56.232 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:36:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a703dc640 1 -- 172.21.15.28:0/870131354 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1a500a2d00 msgr2=0x7f1a500a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a703dc640 1 --2- 172.21.15.28:0/870131354 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1a500a2d00 0x7f1a500a51c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1a600098b0 tx=0x7f1a60009d90 comp rx=0 tx=0).stop 2024-04-25T01:36:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a703dc640 1 -- 172.21.15.28:0/870131354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a68100540 msgr2=0x7f1a6819a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a703dc640 1 --2- 172.21.15.28:0/870131354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a68100540 0x7f1a6819a210 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f1a58006010 tx=0x7f1a580042f0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a703dc640 1 -- 172.21.15.28:0/870131354 shutdown_connections 2024-04-25T01:36:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a703dc640 1 --2- 172.21.15.28:0/870131354 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1a500a2d00 0x7f1a500a51c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a703dc640 1 --2- 172.21.15.28:0/870131354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1a6810e0f0 0x7f1a6819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a703dc640 1 --2- 172.21.15.28:0/870131354 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1a68100e60 0x7f1a6819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a703dc640 1 --2- 172.21.15.28:0/870131354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a68100540 0x7f1a6819a210 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a703dc640 1 -- 172.21.15.28:0/870131354 >> 172.21.15.28:0/870131354 conn(0x7f1a68077740 msgr2=0x7f1a6810f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a703dc640 1 -- 172.21.15.28:0/870131354 shutdown_connections 2024-04-25T01:36:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.248+0000 7f1a703dc640 1 -- 172.21.15.28:0/870131354 wait complete. 2024-04-25T01:36:56.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:36:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.363+0000 7fc20c3ed640 1 -- 172.21.15.28:0/736036382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc20406a780 msgr2=0x7fc20410e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.363+0000 7fc20c3ed640 1 --2- 172.21.15.28:0/736036382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc20406a780 0x7fc20410e5f0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc009cd0 tx=0x7fc1fc02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.363+0000 7fc20c3ed640 1 -- 172.21.15.28:0/736036382 shutdown_connections 2024-04-25T01:36:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.363+0000 7fc20c3ed640 1 --2- 172.21.15.28:0/736036382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc20406a780 0x7fc20410e5f0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.363+0000 7fc20c3ed640 1 --2- 172.21.15.28:0/736036382 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc204069dc0 0x7fc20406a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.363+0000 7fc20c3ed640 1 --2- 172.21.15.28:0/736036382 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc204107730 0x7fc204107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.363+0000 7fc20c3ed640 1 -- 172.21.15.28:0/736036382 >> 172.21.15.28:0/736036382 conn(0x7fc2040776a0 msgr2=0x7fc204077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.363+0000 7fc20c3ed640 1 -- 172.21.15.28:0/736036382 shutdown_connections 2024-04-25T01:36:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.363+0000 7fc20c3ed640 1 -- 172.21.15.28:0/736036382 wait complete. 2024-04-25T01:36:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.364+0000 7fc20c3ed640 1 Processor -- start 2024-04-25T01:36:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.364+0000 7fc20c3ed640 1 -- start start 2024-04-25T01:36:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.364+0000 7fc20c3ed640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc204069dc0 0x7fc204197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.364+0000 7fc20c3ed640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc20406a780 0x7fc204198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.364+0000 7fc20c3ed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc204107730 0x7fc20419c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.364+0000 7fc20c3ed640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc204111b40 con 0x7fc204107730 2024-04-25T01:36:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.364+0000 7fc20c3ed640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc2041119c0 con 0x7fc204069dc0 2024-04-25T01:36:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.364+0000 7fc20c3ed640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc204111cc0 con 0x7fc20406a780 2024-04-25T01:36:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.364+0000 7fc20a98a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc204107730 0x7fc20419c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.364+0000 7fc20a98a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc204107730 0x7fc20419c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40952/0 (socket says 172.21.15.28:40952) 2024-04-25T01:36:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.364+0000 7fc20a98a640 1 -- 172.21.15.28:0/1649114239 learned_addr learned my addr 172.21.15.28:0/1649114239 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.364+0000 7fc20a189640 1 --2- 172.21.15.28:0/1649114239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc204069dc0 0x7fc204197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.364+0000 7fc20a98a640 1 -- 172.21.15.28:0/1649114239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc20406a780 msgr2=0x7fc204198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.364+0000 7fc209988640 1 --2- 172.21.15.28:0/1649114239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc20406a780 0x7fc204198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.364+0000 7fc20a98a640 1 --2- 172.21.15.28:0/1649114239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc20406a780 0x7fc204198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.364+0000 7fc20a98a640 1 -- 172.21.15.28:0/1649114239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc204069dc0 msgr2=0x7fc204197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.364+0000 7fc20a98a640 1 --2- 172.21.15.28:0/1649114239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc204069dc0 0x7fc204197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.364+0000 7fc20a98a640 1 -- 172.21.15.28:0/1649114239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc20419cfb0 con 0x7fc204107730 2024-04-25T01:36:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.364+0000 7fc209988640 1 --2- 172.21.15.28:0/1649114239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc20406a780 0x7fc204198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.365+0000 7fc20a98a640 1 --2- 172.21.15.28:0/1649114239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc204107730 0x7fc20419c880 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc005cc0 tx=0x7fc1fc003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.365+0000 7fc1fb7fe640 1 -- 172.21.15.28:0/1649114239 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc004040 con 0x7fc204107730 2024-04-25T01:36:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.365+0000 7fc20c3ed640 1 -- 172.21.15.28:0/1649114239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc20419d240 con 0x7fc204107730 2024-04-25T01:36:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.365+0000 7fc1fb7fe640 1 -- 172.21.15.28:0/1649114239 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1fc0046c0 con 0x7fc204107730 2024-04-25T01:36:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.365+0000 7fc1fb7fe640 1 -- 172.21.15.28:0/1649114239 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc046480 con 0x7fc204107730 2024-04-25T01:36:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.365+0000 7fc20c3ed640 1 -- 172.21.15.28:0/1649114239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2041a4b30 con 0x7fc204107730 2024-04-25T01:36:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.368+0000 7fc1fb7fe640 1 -- 172.21.15.28:0/1649114239 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc1fc0305a0 con 0x7fc204107730 2024-04-25T01:36:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.368+0000 7fc20f1ce740 1 Processor -- start 2024-04-25T01:36:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.368+0000 7fc20f1ce740 1 -- start start 2024-04-25T01:36:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.368+0000 7fc20f1ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333df4580 0x560333df4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.368+0000 7fc20f1ce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333df4ea0 0x560333dff030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.368+0000 7fc20f1ce740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333dff570 0x560333e01960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.368+0000 7fc20f1ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560333d59670 con 0x560333df4580 2024-04-25T01:36:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.368+0000 7fc20f1ce740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560333d01970 con 0x560333df4ea0 2024-04-25T01:36:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.368+0000 7fc20f1ce740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x560333d55630 con 0x560333dff570 2024-04-25T01:36:56.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.369+0000 7fc209988640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333df4580 0x560333df4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.369+0000 7fc1fb7fe640 1 --2- 172.21.15.28:0/1649114239 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc1f00a2d00 0x7fc1f00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.369+0000 7fc209988640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333df4580 0x560333df4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40968/0 (socket says 172.21.15.28:40968) 2024-04-25T01:36:56.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.369+0000 7fc209988640 1 -- 172.21.15.28:0/1458291490 learned_addr learned my addr 172.21.15.28:0/1458291490 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.369+0000 7fc20a189640 1 --2- 172.21.15.28:0/1458291490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333df4ea0 0x560333dff030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.369+0000 7fc20a98a640 1 --2- 172.21.15.28:0/1649114239 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc1f00a2d00 0x7fc1f00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.369+0000 7fc209988640 1 --2- 172.21.15.28:0/1458291490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333dff570 0x560333e01960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.369+0000 7fc1fb7fe640 1 -- 172.21.15.28:0/1649114239 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc1fc02e030 con 0x7fc204107730 2024-04-25T01:36:56.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.369+0000 7fc209988640 1 -- 172.21.15.28:0/1458291490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333dff570 msgr2=0x560333e01960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.369+0000 7fc209988640 1 --2- 172.21.15.28:0/1458291490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333dff570 0x560333e01960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.369+0000 7fc209988640 1 -- 172.21.15.28:0/1458291490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333df4ea0 msgr2=0x560333dff030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.369+0000 7fc209988640 1 --2- 172.21.15.28:0/1458291490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333df4ea0 0x560333dff030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.369+0000 7fc209988640 1 -- 172.21.15.28:0/1458291490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560333d25400 con 0x560333df4580 2024-04-25T01:36:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.369+0000 7fc20a189640 1 --2- 172.21.15.28:0/1458291490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333df4ea0 0x560333dff030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:36:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.369+0000 7fc209988640 1 --2- 172.21.15.28:0/1458291490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333df4580 0x560333df4960 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4015970 tx=0x7fc1f4009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59a2f112ca77cb09 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.369+0000 7fc20a98a640 1 --2- 172.21.15.28:0/1649114239 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc1f00a2d00 0x7fc1f00a51c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc0ee540 tx=0x7fc1fc03d040 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.369+0000 7fc1f8bfe640 1 -- 172.21.15.28:0/1458291490 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4005020 con 0x560333df4580 2024-04-25T01:36:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.369+0000 7fc1f8bfe640 1 -- 172.21.15.28:0/1458291490 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1f40047a0 con 0x560333df4580 2024-04-25T01:36:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.369+0000 7fc1f8bfe640 1 -- 172.21.15.28:0/1458291490 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4010970 con 0x560333df4580 2024-04-25T01:36:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.370+0000 7fc20f1ce740 1 -- 172.21.15.28:0/1458291490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333df4580 msgr2=0x560333df4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.370+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/1458291490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333df4580 0x560333df4960 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4015970 tx=0x7fc1f4009630 comp rx=0 tx=0).stop 2024-04-25T01:36:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.370+0000 7fc20f1ce740 1 -- 172.21.15.28:0/1458291490 shutdown_connections 2024-04-25T01:36:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.370+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/1458291490 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333dff570 0x560333e01960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.370+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/1458291490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333df4ea0 0x560333dff030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.370+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/1458291490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333df4580 0x560333df4960 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.370+0000 7fc20f1ce740 1 -- 172.21.15.28:0/1458291490 >> 172.21.15.28:0/1458291490 conn(0x560333df08a0 msgr2=0x560333df0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.370+0000 7fc20f1ce740 1 -- 172.21.15.28:0/1458291490 shutdown_connections 2024-04-25T01:36:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.370+0000 7fc20f1ce740 1 -- 172.21.15.28:0/1458291490 wait complete. 2024-04-25T01:36:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc20f1ce740 1 -- start start 2024-04-25T01:36:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc20f1ce740 1 Processor -- start 2024-04-25T01:36:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc20f1ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333df4580 0x560333dfef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc20f1ce740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333df4ea0 0x560333dfc210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc20f1ce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333dff570 0x560333dfc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc20f1ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560333e02300 con 0x560333df4580 2024-04-25T01:36:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc20f1ce740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560333d15050 con 0x560333dff570 2024-04-25T01:36:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc20f1ce740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x560333e02480 con 0x560333df4ea0 2024-04-25T01:36:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc20f1ce740 10 client.0 _ll_register_callbacks cb 0x560333a6cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc209988640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333df4580 0x560333dfef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc209988640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333df4580 0x560333dfef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40976/0 (socket says 172.21.15.28:40976) 2024-04-25T01:36:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc209988640 1 -- 172.21.15.28:0/3483418512 learned_addr learned my addr 172.21.15.28:0/3483418512 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc209988640 1 --2- 172.21.15.28:0/3483418512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333dff570 0x560333dfc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc209988640 1 -- 172.21.15.28:0/3483418512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333df4ea0 msgr2=0x560333dfc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc20a189640 1 --2- 172.21.15.28:0/3483418512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333df4ea0 0x560333dfc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc209988640 1 --2- 172.21.15.28:0/3483418512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333df4ea0 0x560333dfc210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc209988640 1 -- 172.21.15.28:0/3483418512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333dff570 msgr2=0x560333dfc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc209988640 1 --2- 172.21.15.28:0/3483418512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333dff570 0x560333dfc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc209988640 1 -- 172.21.15.28:0/3483418512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560333ce6dc0 con 0x560333df4580 2024-04-25T01:36:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.371+0000 7fc20a189640 1 --2- 172.21.15.28:0/3483418512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333df4ea0 0x560333dfc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.372+0000 7fc209988640 1 --2- 172.21.15.28:0/3483418512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333df4580 0x560333dfef20 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4006010 tx=0x7fc1f4009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceb6abf73ad07eec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.372+0000 7fc1edffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.372+0000 7fc1edffb640 1 -- 172.21.15.28:0/3483418512 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4004d60 con 0x560333df4580 2024-04-25T01:36:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.372+0000 7fc1edffb640 1 -- 172.21.15.28:0/3483418512 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc1f4005000 con 0x560333df4580 2024-04-25T01:36:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.372+0000 7fc20f1ce740 1 -- 172.21.15.28:0/3483418512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1f40047a0 con 0x560333df4580 2024-04-25T01:36:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.372+0000 7fc1edffb640 1 -- 172.21.15.28:0/3483418512 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc1f4033050 con 0x560333df4580 2024-04-25T01:36:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.372+0000 7fc20f1ce740 10 client.16752 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.372+0000 7fc20f1ce740 1 -- 172.21.15.28:0/3483418512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560333a6d3f0 con 0x560333df4580 2024-04-25T01:36:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.372+0000 7fc1edffb640 1 -- 172.21.15.28:0/3483418512 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1f40047a0 con 0x560333df4580 2024-04-25T01:36:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.372+0000 7fc1edffb640 1 -- 172.21.15.28:0/3483418512 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc1f4020470 con 0x560333df4580 2024-04-25T01:36:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.372+0000 7fc20f1ce740 10 client.16752 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.372+0000 7fc20f1ce740 10 client.16752 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:36:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.372+0000 7fc20f1ce740 20 client.16752 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.372+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/3483418512 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x560333e0f510 0x560333e11900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.372+0000 7fc20f1ce740 4 client.16752 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.372+0000 7fc20f1ce740 1 -- 172.21.15.28:0/3483418512 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560333d66bb0 con 0x560333e0f510 2024-04-25T01:36:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.373+0000 7fc20a189640 1 --2- 172.21.15.28:0/3483418512 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x560333e0f510 0x560333e11900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.373+0000 7fc20a189640 1 --2- 172.21.15.28:0/3483418512 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x560333e0f510 0x560333e11900 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bf58a74c2d3f5314 server_cookie=b1e1980a5c061630 in_seq=0 out_seq=0 2024-04-25T01:36:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.373+0000 7fc1edffb640 10 client.16752 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:36:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.374+0000 7fc1edffb640 1 -- 172.21.15.28:0/3483418512 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560333d66bb0 con 0x560333e0f510 2024-04-25T01:36:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.374+0000 7fc1edffb640 10 client.16752 handle_command_reply: tid=0 2024-04-25T01:36:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.374+0000 7fc20f1ce740 1 client.16752 shutdown 2024-04-25T01:36:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.374+0000 7fc20f1ce740 1 -- 172.21.15.28:0/3483418512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333df4580 msgr2=0x560333dfef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.374+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/3483418512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333df4580 0x560333dfef20 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4006010 tx=0x7fc1f4009380 comp rx=0 tx=0).stop 2024-04-25T01:36:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.374+0000 7fc20f1ce740 1 -- 172.21.15.28:0/3483418512 shutdown_connections 2024-04-25T01:36:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.374+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/3483418512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333dff570 0x560333dfc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.374+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/3483418512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333df4ea0 0x560333dfc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.374+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/3483418512 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x560333e0f510 0x560333e11900 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.374+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/3483418512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333df4580 0x560333dfef20 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.374+0000 7fc1edffb640 0 client.16752 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:36:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.374+0000 7fc20f1ce740 1 -- 172.21.15.28:0/3483418512 >> 172.21.15.28:0/3483418512 conn(0x560333df08a0 msgr2=0x560333defb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.374+0000 7fc20f1ce740 1 -- 172.21.15.28:0/3483418512 shutdown_connections 2024-04-25T01:36:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.374+0000 7fc20f1ce740 1 -- 172.21.15.28:0/3483418512 wait complete. 2024-04-25T01:36:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.374+0000 7fc20f1ce740 20 client.16752 trim_cache size 0 max 16384 2024-04-25T01:36:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.387+0000 7fc20f1ce740 1 Processor -- start 2024-04-25T01:36:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.387+0000 7fc20f1ce740 1 -- start start 2024-04-25T01:36:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc20f1ce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333df4580 0x560333e0b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc20f1ce740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333df4ea0 0x560333deb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc20f1ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333dff570 0x560333df7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc20f1ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2041ae9c0 con 0x560333dff570 2024-04-25T01:36:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc20f1ce740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc2041b0500 con 0x560333df4580 2024-04-25T01:36:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc20f1ce740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc2041af090 con 0x560333df4ea0 2024-04-25T01:36:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc209988640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333dff570 0x560333df7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc209988640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333dff570 0x560333df7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40982/0 (socket says 172.21.15.28:40982) 2024-04-25T01:36:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc209988640 1 -- 172.21.15.28:0/1644593199 learned_addr learned my addr 172.21.15.28:0/1644593199 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc209988640 1 --2- 172.21.15.28:0/1644593199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333df4580 0x560333e0b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc20a189640 1 --2- 172.21.15.28:0/1644593199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333df4ea0 0x560333deb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc209988640 1 -- 172.21.15.28:0/1644593199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333df4ea0 msgr2=0x560333deb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc209988640 1 --2- 172.21.15.28:0/1644593199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333df4ea0 0x560333deb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc209988640 1 -- 172.21.15.28:0/1644593199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333df4580 msgr2=0x560333e0b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc209988640 1 --2- 172.21.15.28:0/1644593199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333df4580 0x560333e0b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc209988640 1 -- 172.21.15.28:0/1644593199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560333dfcb20 con 0x560333dff570 2024-04-25T01:36:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc209988640 1 --2- 172.21.15.28:0/1644593199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333dff570 0x560333df7740 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fc1f40208c0 tx=0x7fc1f40098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5255fc5ecd00eb84 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc1effff640 1 -- 172.21.15.28:0/1644593199 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4004770 con 0x560333dff570 2024-04-25T01:36:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc1effff640 1 -- 172.21.15.28:0/1644593199 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1f4009650 con 0x560333dff570 2024-04-25T01:36:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.388+0000 7fc1effff640 1 -- 172.21.15.28:0/1644593199 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4010940 con 0x560333dff570 2024-04-25T01:36:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.389+0000 7fc20f1ce740 1 -- 172.21.15.28:0/1644593199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333dff570 msgr2=0x560333df7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.389+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/1644593199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333dff570 0x560333df7740 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fc1f40208c0 tx=0x7fc1f40098e0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.389+0000 7fc20f1ce740 1 -- 172.21.15.28:0/1644593199 shutdown_connections 2024-04-25T01:36:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.389+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/1644593199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333dff570 0x560333df7740 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.389+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/1644593199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333df4ea0 0x560333deb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.389+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/1644593199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333df4580 0x560333e0b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.389+0000 7fc20f1ce740 1 -- 172.21.15.28:0/1644593199 >> 172.21.15.28:0/1644593199 conn(0x560333df08a0 msgr2=0x560333dfb260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.389+0000 7fc20f1ce740 1 -- 172.21.15.28:0/1644593199 shutdown_connections 2024-04-25T01:36:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.389+0000 7fc20f1ce740 1 -- 172.21.15.28:0/1644593199 wait complete. 2024-04-25T01:36:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.389+0000 7fc20f1ce740 1 -- start start 2024-04-25T01:36:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.389+0000 7fc20f1ce740 1 Processor -- start 2024-04-25T01:36:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc20f1ce740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333df4580 0x560333e71df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc20f1ce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333df4ea0 0x560333e72330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc20f1ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333dff570 0x560333e165c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc20f1ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2041afe30 con 0x560333dff570 2024-04-25T01:36:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc20f1ce740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc2041af760 con 0x560333df4ea0 2024-04-25T01:36:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc20f1ce740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc2041ac9a0 con 0x560333df4580 2024-04-25T01:36:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc20f1ce740 10 client.0 _ll_register_callbacks cb 0x7fc20000b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc209988640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333dff570 0x560333e165c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc209988640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333dff570 0x560333e165c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40998/0 (socket says 172.21.15.28:40998) 2024-04-25T01:36:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc209988640 1 -- 172.21.15.28:0/1596309278 learned_addr learned my addr 172.21.15.28:0/1596309278 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc209988640 1 --2- 172.21.15.28:0/1596309278 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333df4580 0x560333e71df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc20a189640 1 --2- 172.21.15.28:0/1596309278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333df4ea0 0x560333e72330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc209988640 1 -- 172.21.15.28:0/1596309278 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333df4580 msgr2=0x560333e71df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc209988640 1 --2- 172.21.15.28:0/1596309278 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333df4580 0x560333e71df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc209988640 1 -- 172.21.15.28:0/1596309278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333df4ea0 msgr2=0x560333e72330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc209988640 1 --2- 172.21.15.28:0/1596309278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333df4ea0 0x560333e72330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc209988640 1 -- 172.21.15.28:0/1596309278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560333b377d0 con 0x560333dff570 2024-04-25T01:36:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc209988640 1 --2- 172.21.15.28:0/1596309278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333dff570 0x560333e165c0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4009460 tx=0x7fc1f4017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33d80f970c6f7f20 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc1edffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.390+0000 7fc1edffb640 1 -- 172.21.15.28:0/1596309278 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f403e040 con 0x560333dff570 2024-04-25T01:36:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.391+0000 7fc1edffb640 1 -- 172.21.15.28:0/1596309278 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc1f4004020 con 0x560333dff570 2024-04-25T01:36:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.391+0000 7fc20f1ce740 1 -- 172.21.15.28:0/1596309278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1f4009650 con 0x560333dff570 2024-04-25T01:36:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.391+0000 7fc1edffb640 1 -- 172.21.15.28:0/1596309278 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc1f404d050 con 0x560333dff570 2024-04-25T01:36:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.391+0000 7fc20f1ce740 10 client.16758 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.391+0000 7fc20f1ce740 1 -- 172.21.15.28:0/1596309278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560333e793d0 con 0x560333dff570 2024-04-25T01:36:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.391+0000 7fc1edffb640 1 -- 172.21.15.28:0/1596309278 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1f4009650 con 0x560333dff570 2024-04-25T01:36:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.391+0000 7fc1edffb640 1 -- 172.21.15.28:0/1596309278 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc1f4044030 con 0x560333dff570 2024-04-25T01:36:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.391+0000 7fc20f1ce740 10 client.16758 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.391+0000 7fc20f1ce740 10 client.16758 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:36:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.391+0000 7fc20f1ce740 20 client.16758 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.391+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/1596309278 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x560333e799e0 0x560333e7be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.391+0000 7fc20f1ce740 4 client.16758 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.391+0000 7fc20f1ce740 1 -- 172.21.15.28:0/1596309278 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560333d60f00 con 0x560333e799e0 2024-04-25T01:36:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.391+0000 7fc20a189640 1 --2- 172.21.15.28:0/1596309278 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x560333e799e0 0x560333e7be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.392+0000 7fc20a189640 1 --2- 172.21.15.28:0/1596309278 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x560333e799e0 0x560333e7be00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6fc04a52339f2c93 server_cookie=c329950c314d9225 in_seq=0 out_seq=0 2024-04-25T01:36:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.392+0000 7fc1edffb640 10 client.16758 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:36:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.392+0000 7fc1edffb640 1 -- 172.21.15.28:0/1596309278 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560333d60f00 con 0x560333e799e0 2024-04-25T01:36:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.392+0000 7fc1edffb640 10 client.16758 handle_command_reply: tid=0 2024-04-25T01:36:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.392+0000 7fc20f1ce740 1 client.16758 shutdown 2024-04-25T01:36:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.392+0000 7fc20f1ce740 1 -- 172.21.15.28:0/1596309278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333dff570 msgr2=0x560333e165c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.392+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/1596309278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333dff570 0x560333e165c0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4009460 tx=0x7fc1f4017040 comp rx=0 tx=0).stop 2024-04-25T01:36:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.392+0000 7fc20f1ce740 1 -- 172.21.15.28:0/1596309278 shutdown_connections 2024-04-25T01:36:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.392+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/1596309278 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x560333e799e0 0x560333e7be00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.392+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/1596309278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560333dff570 0x560333e165c0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.392+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/1596309278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560333df4ea0 0x560333e72330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.392+0000 7fc1edffb640 0 client.16758 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:36:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.392+0000 7fc20f1ce740 1 --2- 172.21.15.28:0/1596309278 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560333df4580 0x560333e71df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.392+0000 7fc20f1ce740 1 -- 172.21.15.28:0/1596309278 >> 172.21.15.28:0/1596309278 conn(0x560333df08a0 msgr2=0x560333df5b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.392+0000 7fc20f1ce740 1 -- 172.21.15.28:0/1596309278 shutdown_connections 2024-04-25T01:36:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.392+0000 7fc20f1ce740 1 -- 172.21.15.28:0/1596309278 wait complete. 2024-04-25T01:36:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.392+0000 7fc20f1ce740 20 client.16758 trim_cache size 0 max 16384 2024-04-25T01:36:56.376 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:36:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.393+0000 7fc20c3ed640 1 -- 172.21.15.28:0/1649114239 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc1f00a2d00 msgr2=0x7fc1f00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.393+0000 7fc20c3ed640 1 --2- 172.21.15.28:0/1649114239 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc1f00a2d00 0x7fc1f00a51c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc0ee540 tx=0x7fc1fc03d040 comp rx=0 tx=0).stop 2024-04-25T01:36:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.393+0000 7fc20c3ed640 1 -- 172.21.15.28:0/1649114239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc204107730 msgr2=0x7fc20419c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.393+0000 7fc20c3ed640 1 --2- 172.21.15.28:0/1649114239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc204107730 0x7fc20419c880 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc005cc0 tx=0x7fc1fc003970 comp rx=0 tx=0).stop 2024-04-25T01:36:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.393+0000 7fc20c3ed640 1 -- 172.21.15.28:0/1649114239 shutdown_connections 2024-04-25T01:36:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.393+0000 7fc20c3ed640 1 --2- 172.21.15.28:0/1649114239 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc1f00a2d00 0x7fc1f00a51c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.393+0000 7fc20c3ed640 1 --2- 172.21.15.28:0/1649114239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc204107730 0x7fc20419c880 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.393+0000 7fc20c3ed640 1 --2- 172.21.15.28:0/1649114239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc20406a780 0x7fc204198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.393+0000 7fc20c3ed640 1 --2- 172.21.15.28:0/1649114239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc204069dc0 0x7fc204197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.393+0000 7fc20c3ed640 1 -- 172.21.15.28:0/1649114239 >> 172.21.15.28:0/1649114239 conn(0x7fc2040776a0 msgr2=0x7fc2040fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.393+0000 7fc20c3ed640 1 -- 172.21.15.28:0/1649114239 shutdown_connections 2024-04-25T01:36:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.393+0000 7fc20c3ed640 1 -- 172.21.15.28:0/1649114239 wait complete. 2024-04-25T01:36:56.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:56 smithi028 ceph-mon[29390]: pgmap v120: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:36:56.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:56 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/112296319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:36:56.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:36:56.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:56 smithi130 ceph-mon[32702]: pgmap v120: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:36:56.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:56 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/112296319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:36:56.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:56 smithi032 ceph-mon[32461]: pgmap v120: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:36:56.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:56 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/112296319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:36:56.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.507+0000 7f5c5e0c3640 1 -- 172.21.15.28:0/3606143951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c58107730 msgr2=0x7f5c58107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.507+0000 7f5c5e0c3640 1 --2- 172.21.15.28:0/3606143951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c58107730 0x7f5c58107b10 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f5c48009930 tx=0x7f5c4802a420 comp rx=0 tx=0).stop 2024-04-25T01:36:56.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.507+0000 7f5c5e0c3640 1 -- 172.21.15.28:0/3606143951 shutdown_connections 2024-04-25T01:36:56.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.507+0000 7f5c5e0c3640 1 --2- 172.21.15.28:0/3606143951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c5806a780 0x7f5c5810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.507+0000 7f5c5e0c3640 1 --2- 172.21.15.28:0/3606143951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c58069dc0 0x7f5c5806a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.507+0000 7f5c5e0c3640 1 --2- 172.21.15.28:0/3606143951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c58107730 0x7f5c58107b10 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.507+0000 7f5c5e0c3640 1 -- 172.21.15.28:0/3606143951 >> 172.21.15.28:0/3606143951 conn(0x7f5c580776a0 msgr2=0x7f5c58077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.507+0000 7f5c5e0c3640 1 -- 172.21.15.28:0/3606143951 shutdown_connections 2024-04-25T01:36:56.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.507+0000 7f5c5e0c3640 1 -- 172.21.15.28:0/3606143951 wait complete. 2024-04-25T01:36:56.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.508+0000 7f5c5e0c3640 1 Processor -- start 2024-04-25T01:36:56.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.508+0000 7f5c5e0c3640 1 -- start start 2024-04-25T01:36:56.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.508+0000 7f5c5e0c3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c58069dc0 0x7f5c58197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.508+0000 7f5c5e0c3640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c5806a780 0x7f5c58198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.508+0000 7f5c5e0c3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c58107730 0x7f5c5819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.508+0000 7f5c5e0c3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c58111b50 con 0x7f5c58069dc0 2024-04-25T01:36:56.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.508+0000 7f5c5e0c3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5c581119d0 con 0x7f5c58107730 2024-04-25T01:36:56.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.508+0000 7f5c5e0c3640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f5c58111cd0 con 0x7f5c5806a780 2024-04-25T01:36:56.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.508+0000 7f5c577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c58069dc0 0x7f5c58197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.508+0000 7f5c577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c58069dc0 0x7f5c58197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41030/0 (socket says 172.21.15.28:41030) 2024-04-25T01:36:56.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.508+0000 7f5c577fe640 1 -- 172.21.15.28:0/4197479764 learned_addr learned my addr 172.21.15.28:0/4197479764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.508+0000 7f5c57fff640 1 --2- 172.21.15.28:0/4197479764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c58107730 0x7f5c5819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.508+0000 7f5c577fe640 1 -- 172.21.15.28:0/4197479764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c5806a780 msgr2=0x7f5c58198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.508+0000 7f5c577fe640 1 --2- 172.21.15.28:0/4197479764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c5806a780 0x7f5c58198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.508+0000 7f5c577fe640 1 -- 172.21.15.28:0/4197479764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c58107730 msgr2=0x7f5c5819c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.508+0000 7f5c577fe640 1 --2- 172.21.15.28:0/4197479764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c58107730 0x7f5c5819c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.508+0000 7f5c577fe640 1 -- 172.21.15.28:0/4197479764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c5819cf90 con 0x7f5c58069dc0 2024-04-25T01:36:56.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.509+0000 7f5c577fe640 1 --2- 172.21.15.28:0/4197479764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c58069dc0 0x7f5c58197ff0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f5c4802a940 tx=0x7f5c48003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.509+0000 7f5c54ff9640 1 -- 172.21.15.28:0/4197479764 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4803f070 con 0x7f5c58069dc0 2024-04-25T01:36:56.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.509+0000 7f5c54ff9640 1 -- 172.21.15.28:0/4197479764 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c48031c90 con 0x7f5c58069dc0 2024-04-25T01:36:56.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.509+0000 7f5c5e0c3640 1 -- 172.21.15.28:0/4197479764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c5819d220 con 0x7f5c58069dc0 2024-04-25T01:36:56.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.509+0000 7f5c54ff9640 1 -- 172.21.15.28:0/4197479764 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c48035030 con 0x7f5c58069dc0 2024-04-25T01:36:56.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.509+0000 7f5c5e0c3640 1 -- 172.21.15.28:0/4197479764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c581a4b10 con 0x7f5c58069dc0 2024-04-25T01:36:56.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.511+0000 7f5c54ff9640 1 -- 172.21.15.28:0/4197479764 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5c48042070 con 0x7f5c58069dc0 2024-04-25T01:36:56.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.511+0000 7f5c60dcc740 1 Processor -- start 2024-04-25T01:36:56.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.511+0000 7f5c60dcc740 1 -- start start 2024-04-25T01:36:56.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c60dcc740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 0x55716be67960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c60dcc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be67ea0 0x55716be72030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c60dcc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 0x55716be74960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c60dcc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55716bdcc670 con 0x55716be72570 2024-04-25T01:36:56.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c60dcc740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55716bd74970 con 0x55716be67580 2024-04-25T01:36:56.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c60dcc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55716bdc8630 con 0x55716be67ea0 2024-04-25T01:36:56.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c56ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 0x55716be74960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c57fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 0x55716be67960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c56ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 0x55716be74960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41034/0 (socket says 172.21.15.28:41034) 2024-04-25T01:36:56.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c56ffd640 1 -- 172.21.15.28:0/2055052732 learned_addr learned my addr 172.21.15.28:0/2055052732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c54ff9640 1 --2- 172.21.15.28:0/4197479764 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f5c400a2d80 0x7f5c400a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c577fe640 1 --2- 172.21.15.28:0/2055052732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be67ea0 0x55716be72030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c57fff640 1 --2- 172.21.15.28:0/4197479764 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f5c400a2d80 0x7f5c400a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c56ffd640 1 -- 172.21.15.28:0/2055052732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be67ea0 msgr2=0x55716be72030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c56ffd640 1 --2- 172.21.15.28:0/2055052732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be67ea0 0x55716be72030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c56ffd640 1 -- 172.21.15.28:0/2055052732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 msgr2=0x55716be67960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c56ffd640 1 --2- 172.21.15.28:0/2055052732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 0x55716be67960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c56ffd640 1 -- 172.21.15.28:0/2055052732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55716bd98400 con 0x55716be72570 2024-04-25T01:36:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c57fff640 1 --2- 172.21.15.28:0/2055052732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 0x55716be67960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:36:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c54ff9640 1 -- 172.21.15.28:0/4197479764 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f5c480effc0 con 0x7f5c58069dc0 2024-04-25T01:36:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.512+0000 7f5c56ffd640 1 --2- 172.21.15.28:0/2055052732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 0x55716be74960 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c00bae0 tx=0x7f5c4c00d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c823822c05e2ff4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.513+0000 7f5c57fff640 1 --2- 172.21.15.28:0/4197479764 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f5c400a2d80 0x7f5c400a5240 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5c50005cc0 tx=0x7f5c50004ab0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.513+0000 7f5c3dffb640 1 -- 172.21.15.28:0/2055052732 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c00dcb0 con 0x55716be72570 2024-04-25T01:36:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.513+0000 7f5c3dffb640 1 -- 172.21.15.28:0/2055052732 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c4c00de50 con 0x55716be72570 2024-04-25T01:36:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.513+0000 7f5c3dffb640 1 -- 172.21.15.28:0/2055052732 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c0106c0 con 0x55716be72570 2024-04-25T01:36:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.513+0000 7f5c60dcc740 1 -- 172.21.15.28:0/2055052732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 msgr2=0x55716be74960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.513+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/2055052732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 0x55716be74960 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c00bae0 tx=0x7f5c4c00d2e0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.513+0000 7f5c60dcc740 1 -- 172.21.15.28:0/2055052732 shutdown_connections 2024-04-25T01:36:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.513+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/2055052732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 0x55716be74960 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.513+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/2055052732 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be67ea0 0x55716be72030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.513+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/2055052732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 0x55716be67960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.513+0000 7f5c60dcc740 1 -- 172.21.15.28:0/2055052732 >> 172.21.15.28:0/2055052732 conn(0x55716be638a0 msgr2=0x55716be63c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.513+0000 7f5c60dcc740 1 -- 172.21.15.28:0/2055052732 shutdown_connections 2024-04-25T01:36:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.513+0000 7f5c60dcc740 1 -- 172.21.15.28:0/2055052732 wait complete. 2024-04-25T01:36:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.514+0000 7f5c60dcc740 1 -- start start 2024-04-25T01:36:56.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.514+0000 7f5c60dcc740 1 Processor -- start 2024-04-25T01:36:56.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.514+0000 7f5c60dcc740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 0x55716be71e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.514+0000 7f5c60dcc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be67ea0 0x55716be6f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.514+0000 7f5c60dcc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 0x55716be6f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.514+0000 7f5c60dcc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55716be75360 con 0x55716be72570 2024-04-25T01:36:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.514+0000 7f5c60dcc740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55716bd88050 con 0x55716be67580 2024-04-25T01:36:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.514+0000 7f5c60dcc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55716be754e0 con 0x55716be67ea0 2024-04-25T01:36:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.514+0000 7f5c60dcc740 10 client.0 _ll_register_callbacks cb 0x55716badfbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.514+0000 7f5c57fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 0x55716be6f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.514+0000 7f5c57fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 0x55716be6f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41050/0 (socket says 172.21.15.28:41050) 2024-04-25T01:36:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.514+0000 7f5c57fff640 1 -- 172.21.15.28:0/4011014308 learned_addr learned my addr 172.21.15.28:0/4011014308 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.514+0000 7f5c577fe640 1 --2- 172.21.15.28:0/4011014308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 0x55716be71e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.514+0000 7f5c56ffd640 1 --2- 172.21.15.28:0/4011014308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be67ea0 0x55716be6f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.515+0000 7f5c57fff640 1 -- 172.21.15.28:0/4011014308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be67ea0 msgr2=0x55716be6f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.515+0000 7f5c57fff640 1 --2- 172.21.15.28:0/4011014308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be67ea0 0x55716be6f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.515+0000 7f5c57fff640 1 -- 172.21.15.28:0/4011014308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 msgr2=0x55716be71e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.515+0000 7f5c57fff640 1 --2- 172.21.15.28:0/4011014308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 0x55716be71e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.515+0000 7f5c57fff640 1 -- 172.21.15.28:0/4011014308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55716bd59dc0 con 0x55716be72570 2024-04-25T01:36:56.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.515+0000 7f5c56ffd640 1 --2- 172.21.15.28:0/4011014308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be67ea0 0x55716be6f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:56.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.515+0000 7f5c57fff640 1 --2- 172.21.15.28:0/4011014308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 0x55716be6f660 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f5c5000e6b0 tx=0x7f5c50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b8fbb6d84a3398 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.515+0000 7f5c36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:56.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.515+0000 7f5c36ffd640 1 -- 172.21.15.28:0/4011014308 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c50022070 con 0x55716be72570 2024-04-25T01:36:56.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.515+0000 7f5c36ffd640 1 -- 172.21.15.28:0/4011014308 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f5c50013660 con 0x55716be72570 2024-04-25T01:36:56.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.515+0000 7f5c60dcc740 1 -- 172.21.15.28:0/4011014308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c4c00de50 con 0x55716be72570 2024-04-25T01:36:56.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.515+0000 7f5c36ffd640 1 -- 172.21.15.28:0/4011014308 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5c5002e050 con 0x55716be72570 2024-04-25T01:36:56.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.515+0000 7f5c60dcc740 10 client.16770 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:56.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.515+0000 7f5c60dcc740 1 -- 172.21.15.28:0/4011014308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55716bae03f0 con 0x55716be72570 2024-04-25T01:36:56.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.515+0000 7f5c36ffd640 1 -- 172.21.15.28:0/4011014308 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c4c00de50 con 0x55716be72570 2024-04-25T01:36:56.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.515+0000 7f5c36ffd640 1 -- 172.21.15.28:0/4011014308 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5c5000f8c0 con 0x55716be72570 2024-04-25T01:36:56.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.515+0000 7f5c60dcc740 10 client.16770 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:56.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.515+0000 7f5c60dcc740 10 client.16770 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:36:56.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.515+0000 7f5c60dcc740 20 client.16770 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:56.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.516+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/4011014308 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55716be825a0 0x55716be84a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.516+0000 7f5c60dcc740 4 client.16770 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:56.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.516+0000 7f5c60dcc740 1 -- 172.21.15.28:0/4011014308 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55716bdd9bb0 con 0x55716be825a0 2024-04-25T01:36:56.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.516+0000 7f5c577fe640 1 --2- 172.21.15.28:0/4011014308 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55716be825a0 0x55716be84a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.516+0000 7f5c577fe640 1 --2- 172.21.15.28:0/4011014308 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55716be825a0 0x55716be84a20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=614b62a7b347e6f6 server_cookie=57134e6e8cb8975b in_seq=0 out_seq=0 2024-04-25T01:36:56.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.516+0000 7f5c36ffd640 10 client.16770 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:36:56.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.517+0000 7f5c36ffd640 1 -- 172.21.15.28:0/4011014308 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55716bdd9bb0 con 0x55716be825a0 2024-04-25T01:36:56.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.517+0000 7f5c36ffd640 10 client.16770 handle_command_reply: tid=0 2024-04-25T01:36:56.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.517+0000 7f5c60dcc740 1 client.16770 shutdown 2024-04-25T01:36:56.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.517+0000 7f5c60dcc740 1 -- 172.21.15.28:0/4011014308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 msgr2=0x55716be6f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.517+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/4011014308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 0x55716be6f660 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f5c5000e6b0 tx=0x7f5c50004a40 comp rx=0 tx=0).stop 2024-04-25T01:36:56.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.517+0000 7f5c60dcc740 1 -- 172.21.15.28:0/4011014308 shutdown_connections 2024-04-25T01:36:56.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.517+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/4011014308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 0x55716be6f660 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.517+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/4011014308 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55716be825a0 0x55716be84a20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.517+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/4011014308 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be67ea0 0x55716be6f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.517+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/4011014308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 0x55716be71e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.517+0000 7f5c36ffd640 0 client.16770 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:36:56.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.517+0000 7f5c60dcc740 1 -- 172.21.15.28:0/4011014308 >> 172.21.15.28:0/4011014308 conn(0x55716be638a0 msgr2=0x55716be696a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.517+0000 7f5c60dcc740 1 -- 172.21.15.28:0/4011014308 shutdown_connections 2024-04-25T01:36:56.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.517+0000 7f5c60dcc740 1 -- 172.21.15.28:0/4011014308 wait complete. 2024-04-25T01:36:56.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.517+0000 7f5c60dcc740 20 client.16770 trim_cache size 0 max 16384 2024-04-25T01:36:56.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c60dcc740 1 Processor -- start 2024-04-25T01:36:56.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c60dcc740 1 -- start start 2024-04-25T01:36:56.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c60dcc740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 0x55716be7e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c60dcc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be67ea0 0x55716be8a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c60dcc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 0x55716be8a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c60dcc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c581aea20 con 0x55716be72570 2024-04-25T01:36:56.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c60dcc740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5c581b0560 con 0x55716be67580 2024-04-25T01:36:56.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c60dcc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f5c581af0f0 con 0x55716be67ea0 2024-04-25T01:36:56.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c57fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 0x55716be8a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c57fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 0x55716be8a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41060/0 (socket says 172.21.15.28:41060) 2024-04-25T01:36:56.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c57fff640 1 -- 172.21.15.28:0/458637444 learned_addr learned my addr 172.21.15.28:0/458637444 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c56ffd640 1 --2- 172.21.15.28:0/458637444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be67ea0 0x55716be8a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c57fff640 1 -- 172.21.15.28:0/458637444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be67ea0 msgr2=0x55716be8a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c57fff640 1 --2- 172.21.15.28:0/458637444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be67ea0 0x55716be8a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c577fe640 1 --2- 172.21.15.28:0/458637444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 0x55716be7e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c57fff640 1 -- 172.21.15.28:0/458637444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 msgr2=0x55716be7e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c57fff640 1 --2- 172.21.15.28:0/458637444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 0x55716be7e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c57fff640 1 -- 172.21.15.28:0/458637444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55716bbaa7d0 con 0x55716be72570 2024-04-25T01:36:56.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c56ffd640 1 --2- 172.21.15.28:0/458637444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be67ea0 0x55716be8a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:56.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.531+0000 7f5c57fff640 1 --2- 172.21.15.28:0/458637444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 0x55716be8a830 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f5c50011c60 tx=0x7f5c5000bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84bef31a159ae739 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.532+0000 7f5c3d7fa640 1 -- 172.21.15.28:0/458637444 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c50036030 con 0x55716be72570 2024-04-25T01:36:56.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.532+0000 7f5c3d7fa640 1 -- 172.21.15.28:0/458637444 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c50018070 con 0x55716be72570 2024-04-25T01:36:56.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.532+0000 7f5c3d7fa640 1 -- 172.21.15.28:0/458637444 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c50022040 con 0x55716be72570 2024-04-25T01:36:56.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.532+0000 7f5c60dcc740 1 -- 172.21.15.28:0/458637444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 msgr2=0x55716be8a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.532+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/458637444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 0x55716be8a830 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f5c50011c60 tx=0x7f5c5000bdd0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.532+0000 7f5c60dcc740 1 -- 172.21.15.28:0/458637444 shutdown_connections 2024-04-25T01:36:56.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.532+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/458637444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be72570 0x55716be8a830 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.532+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/458637444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be67ea0 0x55716be8a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.532+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/458637444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 0x55716be7e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.532+0000 7f5c60dcc740 1 -- 172.21.15.28:0/458637444 >> 172.21.15.28:0/458637444 conn(0x55716be638a0 msgr2=0x55716be62b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.532+0000 7f5c60dcc740 1 -- 172.21.15.28:0/458637444 shutdown_connections 2024-04-25T01:36:56.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.532+0000 7f5c60dcc740 1 -- 172.21.15.28:0/458637444 wait complete. 2024-04-25T01:36:56.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.532+0000 7f5c60dcc740 1 -- start start 2024-04-25T01:36:56.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.533+0000 7f5c60dcc740 1 Processor -- start 2024-04-25T01:36:56.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.533+0000 7f5c60dcc740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 0x55716be78490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.533+0000 7f5c60dcc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be67ea0 0x55716be789d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.533+0000 7f5c60dcc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be72570 0x55716be78f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.533+0000 7f5c60dcc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c581afe90 con 0x55716be67ea0 2024-04-25T01:36:56.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.533+0000 7f5c60dcc740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5c581af7c0 con 0x55716be67580 2024-04-25T01:36:56.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.533+0000 7f5c60dcc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f5c581aca00 con 0x55716be72570 2024-04-25T01:36:56.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.533+0000 7f5c60dcc740 10 client.0 _ll_register_callbacks cb 0x7f5c5000bd20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:56.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.533+0000 7f5c56ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be67ea0 0x55716be789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.533+0000 7f5c56ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be67ea0 0x55716be789d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41062/0 (socket says 172.21.15.28:41062) 2024-04-25T01:36:56.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.533+0000 7f5c56ffd640 1 -- 172.21.15.28:0/1764066164 learned_addr learned my addr 172.21.15.28:0/1764066164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.533+0000 7f5c57fff640 1 --2- 172.21.15.28:0/1764066164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be72570 0x55716be78f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.533+0000 7f5c577fe640 1 --2- 172.21.15.28:0/1764066164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 0x55716be78490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.533+0000 7f5c56ffd640 1 -- 172.21.15.28:0/1764066164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be72570 msgr2=0x55716be78f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.533+0000 7f5c56ffd640 1 --2- 172.21.15.28:0/1764066164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be72570 0x55716be78f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.533+0000 7f5c56ffd640 1 -- 172.21.15.28:0/1764066164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 msgr2=0x55716be78490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.533+0000 7f5c56ffd640 1 --2- 172.21.15.28:0/1764066164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 0x55716be78490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.533+0000 7f5c56ffd640 1 -- 172.21.15.28:0/1764066164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55716bee7dc0 con 0x55716be67ea0 2024-04-25T01:36:56.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.533+0000 7f5c57fff640 1 --2- 172.21.15.28:0/1764066164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be72570 0x55716be78f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:56.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.534+0000 7f5c56ffd640 1 --2- 172.21.15.28:0/1764066164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be67ea0 0x55716be789d0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c000c00 tx=0x7f5c4c00dce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1118d11b242d95fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.534+0000 7f5c36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:56.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.534+0000 7f5c36ffd640 1 -- 172.21.15.28:0/1764066164 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c004450 con 0x55716be67ea0 2024-04-25T01:36:56.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.534+0000 7f5c36ffd640 1 -- 172.21.15.28:0/1764066164 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f5c4c0035f0 con 0x55716be67ea0 2024-04-25T01:36:56.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.534+0000 7f5c60dcc740 1 -- 172.21.15.28:0/1764066164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c50018070 con 0x55716be67ea0 2024-04-25T01:36:56.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.534+0000 7f5c36ffd640 1 -- 172.21.15.28:0/1764066164 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5c4c0037f0 con 0x55716be67ea0 2024-04-25T01:36:56.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.534+0000 7f5c60dcc740 10 client.16776 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:56.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.534+0000 7f5c60dcc740 1 -- 172.21.15.28:0/1764066164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55716bee9030 con 0x55716be67ea0 2024-04-25T01:36:56.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.534+0000 7f5c36ffd640 1 -- 172.21.15.28:0/1764066164 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c50018070 con 0x55716be67ea0 2024-04-25T01:36:56.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.534+0000 7f5c36ffd640 1 -- 172.21.15.28:0/1764066164 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5c4c022070 con 0x55716be67ea0 2024-04-25T01:36:56.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.534+0000 7f5c60dcc740 10 client.16776 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.534+0000 7f5c60dcc740 10 client.16776 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:36:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.534+0000 7f5c60dcc740 20 client.16776 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.534+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/1764066164 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55716bee9590 0x55716beeb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.534+0000 7f5c60dcc740 4 client.16776 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.534+0000 7f5c60dcc740 1 -- 172.21.15.28:0/1764066164 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55716be6f9f0 con 0x55716bee9590 2024-04-25T01:36:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.535+0000 7f5c577fe640 1 --2- 172.21.15.28:0/1764066164 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55716bee9590 0x55716beeb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.535+0000 7f5c577fe640 1 --2- 172.21.15.28:0/1764066164 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55716bee9590 0x55716beeb9b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=703b014022b3d7dd server_cookie=18ccd4867bf69d78 in_seq=0 out_seq=0 2024-04-25T01:36:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.535+0000 7f5c36ffd640 10 client.16776 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:36:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.535+0000 7f5c36ffd640 1 -- 172.21.15.28:0/1764066164 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55716be6f9f0 con 0x55716bee9590 2024-04-25T01:36:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.535+0000 7f5c36ffd640 10 client.16776 handle_command_reply: tid=0 2024-04-25T01:36:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.535+0000 7f5c60dcc740 1 client.16776 shutdown 2024-04-25T01:36:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.535+0000 7f5c60dcc740 1 -- 172.21.15.28:0/1764066164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be67ea0 msgr2=0x55716be789d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.535+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/1764066164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be67ea0 0x55716be789d0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c000c00 tx=0x7f5c4c00dce0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c60dcc740 1 -- 172.21.15.28:0/1764066164 shutdown_connections 2024-04-25T01:36:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/1764066164 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55716bee9590 0x55716beeb9b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/1764066164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55716be72570 0x55716be78f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/1764066164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55716be67ea0 0x55716be789d0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c36ffd640 0 client.16776 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:36:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c60dcc740 1 --2- 172.21.15.28:0/1764066164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55716be67580 0x55716be78490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c60dcc740 1 -- 172.21.15.28:0/1764066164 >> 172.21.15.28:0/1764066164 conn(0x55716be638a0 msgr2=0x55716be7d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c60dcc740 1 -- 172.21.15.28:0/1764066164 shutdown_connections 2024-04-25T01:36:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c60dcc740 1 -- 172.21.15.28:0/1764066164 wait complete. 2024-04-25T01:36:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c60dcc740 20 client.16776 trim_cache size 0 max 16384 2024-04-25T01:36:56.523 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:36:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c5e0c3640 1 -- 172.21.15.28:0/4197479764 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f5c400a2d80 msgr2=0x7f5c400a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c5e0c3640 1 --2- 172.21.15.28:0/4197479764 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f5c400a2d80 0x7f5c400a5240 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5c50005cc0 tx=0x7f5c50004ab0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c5e0c3640 1 -- 172.21.15.28:0/4197479764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c58069dc0 msgr2=0x7f5c58197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c5e0c3640 1 --2- 172.21.15.28:0/4197479764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c58069dc0 0x7f5c58197ff0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f5c4802a940 tx=0x7f5c48003b90 comp rx=0 tx=0).stop 2024-04-25T01:36:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c5e0c3640 1 -- 172.21.15.28:0/4197479764 shutdown_connections 2024-04-25T01:36:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c5e0c3640 1 --2- 172.21.15.28:0/4197479764 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f5c400a2d80 0x7f5c400a5240 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c5e0c3640 1 --2- 172.21.15.28:0/4197479764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c58107730 0x7f5c5819c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c5e0c3640 1 --2- 172.21.15.28:0/4197479764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5c5806a780 0x7f5c58198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c5e0c3640 1 --2- 172.21.15.28:0/4197479764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c58069dc0 0x7f5c58197ff0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c5e0c3640 1 -- 172.21.15.28:0/4197479764 >> 172.21.15.28:0/4197479764 conn(0x7f5c580776a0 msgr2=0x7f5c580fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c5e0c3640 1 -- 172.21.15.28:0/4197479764 shutdown_connections 2024-04-25T01:36:56.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.536+0000 7f5c5e0c3640 1 -- 172.21.15.28:0/4197479764 wait complete. 2024-04-25T01:36:56.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:36:56.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.653+0000 7f1ee449e640 1 -- 172.21.15.28:0/723274757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1edc069dc0 msgr2=0x7f1edc06a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.653+0000 7f1ee449e640 1 --2- 172.21.15.28:0/723274757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1edc069dc0 0x7f1edc06a240 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc009930 tx=0x7f1ecc02a420 comp rx=0 tx=0).stop 2024-04-25T01:36:56.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.653+0000 7f1ee449e640 1 -- 172.21.15.28:0/723274757 shutdown_connections 2024-04-25T01:36:56.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.653+0000 7f1ee449e640 1 --2- 172.21.15.28:0/723274757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1edc06a780 0x7f1edc10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.653+0000 7f1ee449e640 1 --2- 172.21.15.28:0/723274757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1edc069dc0 0x7f1edc06a240 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.653+0000 7f1ee449e640 1 --2- 172.21.15.28:0/723274757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1edc107730 0x7f1edc107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.653+0000 7f1ee449e640 1 -- 172.21.15.28:0/723274757 >> 172.21.15.28:0/723274757 conn(0x7f1edc0776a0 msgr2=0x7f1edc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.653+0000 7f1ee449e640 1 -- 172.21.15.28:0/723274757 shutdown_connections 2024-04-25T01:36:56.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.653+0000 7f1ee449e640 1 -- 172.21.15.28:0/723274757 wait complete. 2024-04-25T01:36:56.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.654+0000 7f1ee449e640 1 Processor -- start 2024-04-25T01:36:56.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.654+0000 7f1ee449e640 1 -- start start 2024-04-25T01:36:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.654+0000 7f1ee449e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1edc069dc0 0x7f1edc19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.654+0000 7f1ee449e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1edc06a780 0x7f1edc19a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.654+0000 7f1ee449e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1edc107730 0x7f1edc19ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.654+0000 7f1ee449e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc111b40 con 0x7f1edc107730 2024-04-25T01:36:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.654+0000 7f1ee449e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc1119c0 con 0x7f1edc06a780 2024-04-25T01:36:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.654+0000 7f1ee449e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc111cc0 con 0x7f1edc069dc0 2024-04-25T01:36:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.654+0000 7f1ee2a3b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1edc107730 0x7f1edc19ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.654+0000 7f1ee223a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1edc069dc0 0x7f1edc19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.654+0000 7f1ee2a3b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1edc107730 0x7f1edc19ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41088/0 (socket says 172.21.15.28:41088) 2024-04-25T01:36:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.654+0000 7f1ee2a3b640 1 -- 172.21.15.28:0/2764474435 learned_addr learned my addr 172.21.15.28:0/2764474435 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.654+0000 7f1ee1a39640 1 --2- 172.21.15.28:0/2764474435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1edc06a780 0x7f1edc19a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.655+0000 7f1ee2a3b640 1 -- 172.21.15.28:0/2764474435 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1edc069dc0 msgr2=0x7f1edc19a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.655+0000 7f1ee2a3b640 1 --2- 172.21.15.28:0/2764474435 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1edc069dc0 0x7f1edc19a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.655+0000 7f1ee2a3b640 1 -- 172.21.15.28:0/2764474435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1edc06a780 msgr2=0x7f1edc19a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.655+0000 7f1ee2a3b640 1 --2- 172.21.15.28:0/2764474435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1edc06a780 0x7f1edc19a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.655+0000 7f1ee2a3b640 1 -- 172.21.15.28:0/2764474435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1edc19f1c0 con 0x7f1edc107730 2024-04-25T01:36:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.655+0000 7f1ee2a3b640 1 --2- 172.21.15.28:0/2764474435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1edc107730 0x7f1edc19ea90 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f1ed400eaa0 tx=0x7f1ed400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.655+0000 7f1ed37fe640 1 -- 172.21.15.28:0/2764474435 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed400cdb0 con 0x7f1edc107730 2024-04-25T01:36:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.655+0000 7f1ed37fe640 1 -- 172.21.15.28:0/2764474435 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ed4005020 con 0x7f1edc107730 2024-04-25T01:36:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.655+0000 7f1ee449e640 1 -- 172.21.15.28:0/2764474435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1edc19f4b0 con 0x7f1edc107730 2024-04-25T01:36:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.655+0000 7f1ed37fe640 1 -- 172.21.15.28:0/2764474435 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed40116b0 con 0x7f1edc107730 2024-04-25T01:36:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.655+0000 7f1ee449e640 1 -- 172.21.15.28:0/2764474435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1edc1a6d40 con 0x7f1edc107730 2024-04-25T01:36:56.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.657+0000 7f1ed37fe640 1 -- 172.21.15.28:0/2764474435 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1ed4005560 con 0x7f1edc107730 2024-04-25T01:36:56.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.658+0000 7f1ee71aa740 1 Processor -- start 2024-04-25T01:36:56.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.658+0000 7f1ee71aa740 1 -- start start 2024-04-25T01:36:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.658+0000 7f1ee71aa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8580 0x555d4c5c8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.658+0000 7f1ee71aa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8ea0 0x555d4c5d3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.658+0000 7f1ee71aa740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5d3570 0x555d4c5d5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.658+0000 7f1ee71aa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555d4c52d670 con 0x555d4c5c8ea0 2024-04-25T01:36:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.658+0000 7f1ee71aa740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555d4c4d5970 con 0x555d4c5d3570 2024-04-25T01:36:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.658+0000 7f1ee71aa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x555d4c529630 con 0x555d4c5c8580 2024-04-25T01:36:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.658+0000 7f1ee223a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8ea0 0x555d4c5d3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.658+0000 7f1ee223a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8ea0 0x555d4c5d3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41094/0 (socket says 172.21.15.28:41094) 2024-04-25T01:36:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.658+0000 7f1ee223a640 1 -- 172.21.15.28:0/1055433239 learned_addr learned my addr 172.21.15.28:0/1055433239 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.658+0000 7f1ee1a39640 1 --2- 172.21.15.28:0/1055433239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5d3570 0x555d4c5d5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.659+0000 7f1ed37fe640 1 --2- 172.21.15.28:0/2764474435 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1ec80a2d00 0x7f1ec80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.659+0000 7f1ee1a39640 1 --2- 172.21.15.28:0/1055433239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8580 0x555d4c5c8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.659+0000 7f1ee223a640 1 -- 172.21.15.28:0/1055433239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8580 msgr2=0x555d4c5c8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.659+0000 7f1ee223a640 1 --2- 172.21.15.28:0/1055433239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8580 0x555d4c5c8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.659+0000 7f1ee2a3b640 1 --2- 172.21.15.28:0/2764474435 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1ec80a2d00 0x7f1ec80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.659+0000 7f1ee223a640 1 -- 172.21.15.28:0/1055433239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5d3570 msgr2=0x555d4c5d5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.659+0000 7f1ee223a640 1 --2- 172.21.15.28:0/1055433239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5d3570 0x555d4c5d5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.659+0000 7f1ee223a640 1 -- 172.21.15.28:0/1055433239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d4c4f9400 con 0x555d4c5c8ea0 2024-04-25T01:36:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.659+0000 7f1ee1a39640 1 --2- 172.21.15.28:0/1055433239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5d3570 0x555d4c5d5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:36:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.659+0000 7f1ed37fe640 1 -- 172.21.15.28:0/2764474435 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1ed40cf930 con 0x7f1edc107730 2024-04-25T01:36:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.659+0000 7f1ee223a640 1 --2- 172.21.15.28:0/1055433239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8ea0 0x555d4c5d3030 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8009c70 tx=0x7f1ed8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0d63553fb6ea73 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.659+0000 7f1ed0bfe640 1 -- 172.21.15.28:0/1055433239 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8004230 con 0x555d4c5c8ea0 2024-04-25T01:36:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.659+0000 7f1ee2a3b640 1 --2- 172.21.15.28:0/2764474435 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1ec80a2d00 0x7f1ec80a51c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1ed409b8c0 tx=0x7f1ed4000f40 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.659+0000 7f1ed0bfe640 1 -- 172.21.15.28:0/1055433239 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ed80043d0 con 0x555d4c5c8ea0 2024-04-25T01:36:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.659+0000 7f1ed0bfe640 1 -- 172.21.15.28:0/1055433239 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8019740 con 0x555d4c5c8ea0 2024-04-25T01:36:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.660+0000 7f1ee71aa740 1 -- 172.21.15.28:0/1055433239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8ea0 msgr2=0x555d4c5d3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.660+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/1055433239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8ea0 0x555d4c5d3030 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8009c70 tx=0x7f1ed8011810 comp rx=0 tx=0).stop 2024-04-25T01:36:56.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.660+0000 7f1ee71aa740 1 -- 172.21.15.28:0/1055433239 shutdown_connections 2024-04-25T01:36:56.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.660+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/1055433239 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5d3570 0x555d4c5d5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.660+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/1055433239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8ea0 0x555d4c5d3030 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.660+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/1055433239 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8580 0x555d4c5c8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.660+0000 7f1ee71aa740 1 -- 172.21.15.28:0/1055433239 >> 172.21.15.28:0/1055433239 conn(0x555d4c5c48a0 msgr2=0x555d4c5c4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.660+0000 7f1ee71aa740 1 -- 172.21.15.28:0/1055433239 shutdown_connections 2024-04-25T01:36:56.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.660+0000 7f1ee71aa740 1 -- 172.21.15.28:0/1055433239 wait complete. 2024-04-25T01:36:56.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.660+0000 7f1ee71aa740 1 -- start start 2024-04-25T01:36:56.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.660+0000 7f1ee71aa740 1 Processor -- start 2024-04-25T01:36:56.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ee71aa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8580 0x555d4c5d2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ee71aa740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5c8ea0 0x555d4c5d01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ee71aa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5d3570 0x555d4c5d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ee71aa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555d4c5d6360 con 0x555d4c5d3570 2024-04-25T01:36:56.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ee71aa740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555d4c4e9050 con 0x555d4c5c8ea0 2024-04-25T01:36:56.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ee71aa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x555d4c5d64e0 con 0x555d4c5c8580 2024-04-25T01:36:56.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ee71aa740 10 client.0 _ll_register_callbacks cb 0x555d4c240be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:56.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ee1a39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5d3570 0x555d4c5d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ee1a39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5d3570 0x555d4c5d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41106/0 (socket says 172.21.15.28:41106) 2024-04-25T01:36:56.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ee1a39640 1 -- 172.21.15.28:0/2215060672 learned_addr learned my addr 172.21.15.28:0/2215060672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ee1a39640 1 --2- 172.21.15.28:0/2215060672 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8580 0x555d4c5d2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ee223a640 1 --2- 172.21.15.28:0/2215060672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5c8ea0 0x555d4c5d01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ee1a39640 1 -- 172.21.15.28:0/2215060672 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8580 msgr2=0x555d4c5d2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ee1a39640 1 --2- 172.21.15.28:0/2215060672 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8580 0x555d4c5d2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ee1a39640 1 -- 172.21.15.28:0/2215060672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5c8ea0 msgr2=0x555d4c5d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ee1a39640 1 --2- 172.21.15.28:0/2215060672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5c8ea0 0x555d4c5d01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ee1a39640 1 -- 172.21.15.28:0/2215060672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d4c4badc0 con 0x555d4c5d3570 2024-04-25T01:36:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ee1a39640 1 --2- 172.21.15.28:0/2215060672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5d3570 0x555d4c5d06e0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc005c40 tx=0x7f1ecc02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f551b24db951d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ec5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ec5ffb640 1 -- 172.21.15.28:0/2215060672 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc044030 con 0x555d4c5d3570 2024-04-25T01:36:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ec5ffb640 1 -- 172.21.15.28:0/2215060672 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1ecc004230 con 0x555d4c5d3570 2024-04-25T01:36:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.661+0000 7f1ee71aa740 1 -- 172.21.15.28:0/2215060672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ed80043d0 con 0x555d4c5d3570 2024-04-25T01:36:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.662+0000 7f1ec5ffb640 1 -- 172.21.15.28:0/2215060672 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1ecc047040 con 0x555d4c5d3570 2024-04-25T01:36:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.662+0000 7f1ee71aa740 10 client.16788 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.662+0000 7f1ee71aa740 1 -- 172.21.15.28:0/2215060672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555d4c2413f0 con 0x555d4c5d3570 2024-04-25T01:36:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.662+0000 7f1ec5ffb640 1 -- 172.21.15.28:0/2215060672 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ed80043d0 con 0x555d4c5d3570 2024-04-25T01:36:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.662+0000 7f1ec5ffb640 1 -- 172.21.15.28:0/2215060672 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1ecc004a00 con 0x555d4c5d3570 2024-04-25T01:36:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.662+0000 7f1ee71aa740 10 client.16788 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.662+0000 7f1ee71aa740 10 client.16788 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:36:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.662+0000 7f1ee71aa740 20 client.16788 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.662+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/2215060672 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x555d4c5e35a0 0x555d4c5e5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.662+0000 7f1ee71aa740 4 client.16788 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.662+0000 7f1ee71aa740 1 -- 172.21.15.28:0/2215060672 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d4c53abb0 con 0x555d4c5e35a0 2024-04-25T01:36:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.662+0000 7f1ee223a640 1 --2- 172.21.15.28:0/2215060672 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x555d4c5e35a0 0x555d4c5e5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.663+0000 7f1ee223a640 1 --2- 172.21.15.28:0/2215060672 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x555d4c5e35a0 0x555d4c5e5990 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=28cbcc41146efaef server_cookie=d74304ec84e7df28 in_seq=0 out_seq=0 2024-04-25T01:36:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.663+0000 7f1ec5ffb640 10 client.16788 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:36:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.663+0000 7f1ec5ffb640 1 -- 172.21.15.28:0/2215060672 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555d4c53abb0 con 0x555d4c5e35a0 2024-04-25T01:36:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.663+0000 7f1ec5ffb640 10 client.16788 handle_command_reply: tid=0 2024-04-25T01:36:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.664+0000 7f1ee71aa740 1 client.16788 shutdown 2024-04-25T01:36:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.664+0000 7f1ee71aa740 1 -- 172.21.15.28:0/2215060672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5d3570 msgr2=0x555d4c5d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.664+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/2215060672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5d3570 0x555d4c5d06e0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc005c40 tx=0x7f1ecc02a420 comp rx=0 tx=0).stop 2024-04-25T01:36:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.664+0000 7f1ee71aa740 1 -- 172.21.15.28:0/2215060672 shutdown_connections 2024-04-25T01:36:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.664+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/2215060672 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x555d4c5e35a0 0x555d4c5e5990 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.664+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/2215060672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5d3570 0x555d4c5d06e0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.664+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/2215060672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5c8ea0 0x555d4c5d01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.664+0000 7f1ec5ffb640 0 client.16788 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:36:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.664+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/2215060672 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8580 0x555d4c5d2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.664+0000 7f1ee71aa740 1 -- 172.21.15.28:0/2215060672 >> 172.21.15.28:0/2215060672 conn(0x555d4c5c48a0 msgr2=0x555d4c5ca6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.664+0000 7f1ee71aa740 1 -- 172.21.15.28:0/2215060672 shutdown_connections 2024-04-25T01:36:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.664+0000 7f1ee71aa740 1 -- 172.21.15.28:0/2215060672 wait complete. 2024-04-25T01:36:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.664+0000 7f1ee71aa740 20 client.16788 trim_cache size 0 max 16384 2024-04-25T01:36:56.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.677+0000 7f1ee71aa740 1 Processor -- start 2024-04-25T01:36:56.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.677+0000 7f1ee71aa740 1 -- start start 2024-04-25T01:36:56.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.677+0000 7f1ee71aa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8580 0x555d4c5df220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.678+0000 7f1ee71aa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8ea0 0x555d4c5eb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.678+0000 7f1ee71aa740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5d3570 0x555d4c5eb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.678+0000 7f1ee71aa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc1b0cf0 con 0x555d4c5c8580 2024-04-25T01:36:56.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.678+0000 7f1ee71aa740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc1b2830 con 0x555d4c5d3570 2024-04-25T01:36:56.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.678+0000 7f1ee71aa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc1b13c0 con 0x555d4c5c8ea0 2024-04-25T01:36:56.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.678+0000 7f1ee1a39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8580 0x555d4c5df220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.678+0000 7f1ee1a39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8580 0x555d4c5df220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49462/0 (socket says 172.21.15.28:49462) 2024-04-25T01:36:56.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.678+0000 7f1ee1a39640 1 -- 172.21.15.28:0/364282607 learned_addr learned my addr 172.21.15.28:0/364282607 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.678+0000 7f1ee1a39640 1 --2- 172.21.15.28:0/364282607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5d3570 0x555d4c5eb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.678+0000 7f1ee223a640 1 --2- 172.21.15.28:0/364282607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8ea0 0x555d4c5eb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.678+0000 7f1ee1a39640 1 -- 172.21.15.28:0/364282607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8ea0 msgr2=0x555d4c5eb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.678+0000 7f1ee1a39640 1 --2- 172.21.15.28:0/364282607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8ea0 0x555d4c5eb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.678+0000 7f1ee1a39640 1 -- 172.21.15.28:0/364282607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5d3570 msgr2=0x555d4c5eb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.678+0000 7f1ee1a39640 1 --2- 172.21.15.28:0/364282607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5d3570 0x555d4c5eb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.678+0000 7f1ee1a39640 1 -- 172.21.15.28:0/364282607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d4c30b7d0 con 0x555d4c5c8580 2024-04-25T01:36:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.678+0000 7f1ee1a39640 1 --2- 172.21.15.28:0/364282607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8580 0x555d4c5df220 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x555d4c5ebbd0 tx=0x7f1ecc031830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fbbf7eddc34d700 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.678+0000 7f1ec7fff640 1 -- 172.21.15.28:0/364282607 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc044070 con 0x555d4c5c8580 2024-04-25T01:36:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.678+0000 7f1ec7fff640 1 -- 172.21.15.28:0/364282607 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ecc031bb0 con 0x555d4c5c8580 2024-04-25T01:36:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.678+0000 7f1ec7fff640 1 -- 172.21.15.28:0/364282607 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc035040 con 0x555d4c5c8580 2024-04-25T01:36:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.679+0000 7f1ee71aa740 1 -- 172.21.15.28:0/364282607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8580 msgr2=0x555d4c5df220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.679+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/364282607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8580 0x555d4c5df220 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x555d4c5ebbd0 tx=0x7f1ecc031830 comp rx=0 tx=0).stop 2024-04-25T01:36:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.679+0000 7f1ee71aa740 1 -- 172.21.15.28:0/364282607 shutdown_connections 2024-04-25T01:36:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.679+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/364282607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5d3570 0x555d4c5eb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.679+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/364282607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8ea0 0x555d4c5eb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.679+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/364282607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8580 0x555d4c5df220 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.679+0000 7f1ee71aa740 1 -- 172.21.15.28:0/364282607 >> 172.21.15.28:0/364282607 conn(0x555d4c5c48a0 msgr2=0x555d4c5c3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.679+0000 7f1ee71aa740 1 -- 172.21.15.28:0/364282607 shutdown_connections 2024-04-25T01:36:56.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.679+0000 7f1ee71aa740 1 -- 172.21.15.28:0/364282607 wait complete. 2024-04-25T01:36:56.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.679+0000 7f1ee71aa740 1 -- start start 2024-04-25T01:36:56.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.679+0000 7f1ee71aa740 1 Processor -- start 2024-04-25T01:36:56.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee71aa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8580 0x555d4c5db880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee71aa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8ea0 0x555d4c5d8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee71aa740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5d3570 0x555d4c5d9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee71aa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc1b2160 con 0x555d4c5c8580 2024-04-25T01:36:56.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee71aa740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc1b1a90 con 0x555d4c5d3570 2024-04-25T01:36:56.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee71aa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc1aecd0 con 0x555d4c5c8ea0 2024-04-25T01:36:56.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee71aa740 10 client.0 _ll_register_callbacks cb 0x7f1ecc031780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:56.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee1a39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8580 0x555d4c5db880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee1a39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8580 0x555d4c5db880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49464/0 (socket says 172.21.15.28:49464) 2024-04-25T01:36:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee1a39640 1 -- 172.21.15.28:0/2884165916 learned_addr learned my addr 172.21.15.28:0/2884165916 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee223a640 1 --2- 172.21.15.28:0/2884165916 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8ea0 0x555d4c5d8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee1a39640 1 -- 172.21.15.28:0/2884165916 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8ea0 msgr2=0x555d4c5d8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee1a39640 1 --2- 172.21.15.28:0/2884165916 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8ea0 0x555d4c5d8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee1a39640 1 -- 172.21.15.28:0/2884165916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5d3570 msgr2=0x555d4c5d9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee1a39640 1 --2- 172.21.15.28:0/2884165916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5d3570 0x555d4c5d9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee1a39640 1 -- 172.21.15.28:0/2884165916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d4c5d96c0 con 0x555d4c5c8580 2024-04-25T01:36:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee1a39640 1 --2- 172.21.15.28:0/2884165916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8580 0x555d4c5db880 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc041040 tx=0x7f1ecc032f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88849c995279e6cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ec5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ec5ffb640 1 -- 172.21.15.28:0/2884165916 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc035070 con 0x555d4c5c8580 2024-04-25T01:36:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee71aa740 1 -- 172.21.15.28:0/2884165916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ecc031bb0 con 0x555d4c5c8580 2024-04-25T01:36:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ec5ffb640 1 -- 172.21.15.28:0/2884165916 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1ecc004ac0 con 0x555d4c5c8580 2024-04-25T01:36:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ec5ffb640 1 -- 172.21.15.28:0/2884165916 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1ecc067050 con 0x555d4c5c8580 2024-04-25T01:36:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee71aa740 10 client.16794 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.680+0000 7f1ee71aa740 1 -- 172.21.15.28:0/2884165916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555d4c649750 con 0x555d4c5c8580 2024-04-25T01:36:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.681+0000 7f1ec5ffb640 1 -- 172.21.15.28:0/2884165916 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ecc031bb0 con 0x555d4c5c8580 2024-04-25T01:36:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.681+0000 7f1ec5ffb640 1 -- 172.21.15.28:0/2884165916 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1ecc003920 con 0x555d4c5c8580 2024-04-25T01:36:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.681+0000 7f1ee71aa740 10 client.16794 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.681+0000 7f1ee71aa740 10 client.16794 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:36:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.681+0000 7f1ee71aa740 20 client.16794 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.681+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/2884165916 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x555d4c649e40 0x555d4c64c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.681+0000 7f1ee71aa740 4 client.16794 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.681+0000 7f1ee71aa740 1 -- 172.21.15.28:0/2884165916 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555d4c534f00 con 0x555d4c649e40 2024-04-25T01:36:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.681+0000 7f1ee223a640 1 --2- 172.21.15.28:0/2884165916 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x555d4c649e40 0x555d4c64c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.681+0000 7f1ee223a640 1 --2- 172.21.15.28:0/2884165916 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x555d4c649e40 0x555d4c64c260 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=231ee881f7ea62e7 server_cookie=fa6b1ae431bc868e in_seq=0 out_seq=0 2024-04-25T01:36:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.681+0000 7f1ec5ffb640 10 client.16794 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:36:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ec5ffb640 1 -- 172.21.15.28:0/2884165916 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555d4c534f00 con 0x555d4c649e40 2024-04-25T01:36:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ec5ffb640 10 client.16794 handle_command_reply: tid=0 2024-04-25T01:36:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee71aa740 1 client.16794 shutdown 2024-04-25T01:36:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee71aa740 1 -- 172.21.15.28:0/2884165916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8580 msgr2=0x555d4c5db880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/2884165916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8580 0x555d4c5db880 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc041040 tx=0x7f1ecc032f30 comp rx=0 tx=0).stop 2024-04-25T01:36:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee71aa740 1 -- 172.21.15.28:0/2884165916 shutdown_connections 2024-04-25T01:36:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/2884165916 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x555d4c649e40 0x555d4c64c260 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/2884165916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555d4c5d3570 0x555d4c5d9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/2884165916 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555d4c5c8ea0 0x555d4c5d8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee71aa740 1 --2- 172.21.15.28:0/2884165916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d4c5c8580 0x555d4c5db880 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ec5ffb640 0 client.16794 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:36:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee71aa740 1 -- 172.21.15.28:0/2884165916 >> 172.21.15.28:0/2884165916 conn(0x555d4c5c48a0 msgr2=0x555d4c5de210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee71aa740 1 -- 172.21.15.28:0/2884165916 shutdown_connections 2024-04-25T01:36:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee71aa740 1 -- 172.21.15.28:0/2884165916 wait complete. 2024-04-25T01:36:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee71aa740 20 client.16794 trim_cache size 0 max 16384 2024-04-25T01:36:56.667 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:36:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee449e640 1 -- 172.21.15.28:0/2764474435 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1ec80a2d00 msgr2=0x7f1ec80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee449e640 1 --2- 172.21.15.28:0/2764474435 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1ec80a2d00 0x7f1ec80a51c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1ed409b8c0 tx=0x7f1ed4000f40 comp rx=0 tx=0).stop 2024-04-25T01:36:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee449e640 1 -- 172.21.15.28:0/2764474435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1edc107730 msgr2=0x7f1edc19ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee449e640 1 --2- 172.21.15.28:0/2764474435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1edc107730 0x7f1edc19ea90 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f1ed400eaa0 tx=0x7f1ed400ef60 comp rx=0 tx=0).stop 2024-04-25T01:36:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee449e640 1 -- 172.21.15.28:0/2764474435 shutdown_connections 2024-04-25T01:36:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee449e640 1 --2- 172.21.15.28:0/2764474435 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1ec80a2d00 0x7f1ec80a51c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee449e640 1 --2- 172.21.15.28:0/2764474435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1edc107730 0x7f1edc19ea90 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee449e640 1 --2- 172.21.15.28:0/2764474435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1edc06a780 0x7f1edc19a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee449e640 1 --2- 172.21.15.28:0/2764474435 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1edc069dc0 0x7f1edc19a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.682+0000 7f1ee449e640 1 -- 172.21.15.28:0/2764474435 >> 172.21.15.28:0/2764474435 conn(0x7f1edc0776a0 msgr2=0x7f1edc10cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.683+0000 7f1ee449e640 1 -- 172.21.15.28:0/2764474435 shutdown_connections 2024-04-25T01:36:56.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.683+0000 7f1ee449e640 1 -- 172.21.15.28:0/2764474435 wait complete. 2024-04-25T01:36:56.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:36:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.798+0000 7f7679b09640 1 -- 172.21.15.28:0/3316438127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7674107770 msgr2=0x7f7674107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.798+0000 7f7679b09640 1 --2- 172.21.15.28:0/3316438127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7674107770 0x7f7674107b50 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7664009930 tx=0x7f766402a420 comp rx=0 tx=0).stop 2024-04-25T01:36:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.798+0000 7f7679b09640 1 -- 172.21.15.28:0/3316438127 shutdown_connections 2024-04-25T01:36:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.798+0000 7f7679b09640 1 --2- 172.21.15.28:0/3316438127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7674102120 0x7f767410e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.798+0000 7f7679b09640 1 --2- 172.21.15.28:0/3316438127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7674101760 0x7f7674101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.798+0000 7f7679b09640 1 --2- 172.21.15.28:0/3316438127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7674107770 0x7f7674107b50 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.798+0000 7f7679b09640 1 -- 172.21.15.28:0/3316438127 >> 172.21.15.28:0/3316438127 conn(0x7f76740fd250 msgr2=0x7f76740ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.798+0000 7f7679b09640 1 -- 172.21.15.28:0/3316438127 shutdown_connections 2024-04-25T01:36:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.798+0000 7f7679b09640 1 -- 172.21.15.28:0/3316438127 wait complete. 2024-04-25T01:36:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.799+0000 7f7679b09640 1 Processor -- start 2024-04-25T01:36:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.799+0000 7f7679b09640 1 -- start start 2024-04-25T01:36:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.799+0000 7f7679b09640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7674101760 0x7f7674073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.799+0000 7f7679b09640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7674102120 0x7f7674073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.799+0000 7f7679b09640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7674107770 0x7f76740744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.799+0000 7f7679b09640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7674111b80 con 0x7f7674101760 2024-04-25T01:36:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.799+0000 7f7679b09640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7674111a00 con 0x7f7674107770 2024-04-25T01:36:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.799+0000 7f7679b09640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7674111d00 con 0x7f7674102120 2024-04-25T01:36:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.799+0000 7f76737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7674101760 0x7f7674073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.799+0000 7f76737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7674101760 0x7f7674073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49482/0 (socket says 172.21.15.28:49482) 2024-04-25T01:36:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.799+0000 7f76737fe640 1 -- 172.21.15.28:0/434844543 learned_addr learned my addr 172.21.15.28:0/434844543 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.800+0000 7f7672ffd640 1 --2- 172.21.15.28:0/434844543 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7674102120 0x7f7674073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.800+0000 7f7673fff640 1 --2- 172.21.15.28:0/434844543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7674107770 0x7f76740744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.800+0000 7f76737fe640 1 -- 172.21.15.28:0/434844543 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7674102120 msgr2=0x7f7674073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.800+0000 7f76737fe640 1 --2- 172.21.15.28:0/434844543 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7674102120 0x7f7674073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.800+0000 7f76737fe640 1 -- 172.21.15.28:0/434844543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7674107770 msgr2=0x7f76740744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.800+0000 7f76737fe640 1 --2- 172.21.15.28:0/434844543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7674107770 0x7f76740744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.800+0000 7f76737fe640 1 -- 172.21.15.28:0/434844543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76741a6520 con 0x7f7674101760 2024-04-25T01:36:56.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.800+0000 7f7672ffd640 1 --2- 172.21.15.28:0/434844543 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7674102120 0x7f7674073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:56.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.800+0000 7f76737fe640 1 --2- 172.21.15.28:0/434844543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7674101760 0x7f7674073a40 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f7664009230 tx=0x7f7664004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.800+0000 7f7670ff9640 1 -- 172.21.15.28:0/434844543 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766403f030 con 0x7f7674101760 2024-04-25T01:36:56.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.800+0000 7f7679b09640 1 -- 172.21.15.28:0/434844543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76741a67b0 con 0x7f7674101760 2024-04-25T01:36:56.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.800+0000 7f7670ff9640 1 -- 172.21.15.28:0/434844543 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7664003ad0 con 0x7f7674101760 2024-04-25T01:36:56.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.800+0000 7f7670ff9640 1 -- 172.21.15.28:0/434844543 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7664035070 con 0x7f7674101760 2024-04-25T01:36:56.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.800+0000 7f7679b09640 1 -- 172.21.15.28:0/434844543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76741a6cf0 con 0x7f7674101760 2024-04-25T01:36:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.803+0000 7f7670ff9640 1 -- 172.21.15.28:0/434844543 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7664042030 con 0x7f7674101760 2024-04-25T01:36:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.803+0000 7f767c80b740 1 Processor -- start 2024-04-25T01:36:56.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.803+0000 7f767c80b740 1 -- start start 2024-04-25T01:36:56.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.803+0000 7f767c80b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0e580 0x563bf9e0e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.803+0000 7f767c80b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e0eea0 0x563bf9e19030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.803+0000 7f767c80b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 0x563bf9e1b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.803+0000 7f767c80b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563bf9d73670 con 0x563bf9e19570 2024-04-25T01:36:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.803+0000 7f767c80b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563bf9d1b970 con 0x563bf9e0e580 2024-04-25T01:36:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.803+0000 7f767c80b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x563bf9d6f630 con 0x563bf9e0eea0 2024-04-25T01:36:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.804+0000 7f7672ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 0x563bf9e1b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.804+0000 7f7672ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 0x563bf9e1b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49488/0 (socket says 172.21.15.28:49488) 2024-04-25T01:36:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.804+0000 7f7672ffd640 1 -- 172.21.15.28:0/1896529471 learned_addr learned my addr 172.21.15.28:0/1896529471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.804+0000 7f7673fff640 1 --2- 172.21.15.28:0/1896529471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0e580 0x563bf9e0e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.804+0000 7f7670ff9640 1 --2- 172.21.15.28:0/434844543 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f765c0a2d00 0x7f765c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.804+0000 7f76737fe640 1 --2- 172.21.15.28:0/1896529471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e0eea0 0x563bf9e19030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.804+0000 7f7672ffd640 1 -- 172.21.15.28:0/1896529471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e0eea0 msgr2=0x563bf9e19030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.804+0000 7f7672ffd640 1 --2- 172.21.15.28:0/1896529471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e0eea0 0x563bf9e19030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.804+0000 7f7673fff640 1 --2- 172.21.15.28:0/434844543 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f765c0a2d00 0x7f765c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.804+0000 7f7672ffd640 1 -- 172.21.15.28:0/1896529471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0e580 msgr2=0x563bf9e0e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.804+0000 7f7672ffd640 1 --2- 172.21.15.28:0/1896529471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0e580 0x563bf9e0e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.804+0000 7f7672ffd640 1 -- 172.21.15.28:0/1896529471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bf9d3f400 con 0x563bf9e19570 2024-04-25T01:36:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.804+0000 7f7670ff9640 1 -- 172.21.15.28:0/434844543 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f76640efd30 con 0x7f7674101760 2024-04-25T01:36:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.804+0000 7f7672ffd640 1 --2- 172.21.15.28:0/1896529471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 0x563bf9e1b960 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f7668008c10 tx=0x7f766800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d9634ab6a984a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.804+0000 7f7659ffb640 1 -- 172.21.15.28:0/1896529471 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7668014070 con 0x563bf9e19570 2024-04-25T01:36:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.804+0000 7f7673fff640 1 --2- 172.21.15.28:0/434844543 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f765c0a2d00 0x7f765c0a51c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f766c005fd0 tx=0x7f766c004ab0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.804+0000 7f7659ffb640 1 -- 172.21.15.28:0/1896529471 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76680047a0 con 0x563bf9e19570 2024-04-25T01:36:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.804+0000 7f7659ffb640 1 -- 172.21.15.28:0/1896529471 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7668010690 con 0x563bf9e19570 2024-04-25T01:36:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.805+0000 7f767c80b740 1 -- 172.21.15.28:0/1896529471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 msgr2=0x563bf9e1b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.805+0000 7f767c80b740 1 --2- 172.21.15.28:0/1896529471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 0x563bf9e1b960 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f7668008c10 tx=0x7f766800e720 comp rx=0 tx=0).stop 2024-04-25T01:36:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.805+0000 7f767c80b740 1 -- 172.21.15.28:0/1896529471 shutdown_connections 2024-04-25T01:36:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.805+0000 7f767c80b740 1 --2- 172.21.15.28:0/1896529471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 0x563bf9e1b960 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.805+0000 7f767c80b740 1 --2- 172.21.15.28:0/1896529471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e0eea0 0x563bf9e19030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.805+0000 7f767c80b740 1 --2- 172.21.15.28:0/1896529471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0e580 0x563bf9e0e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.805+0000 7f767c80b740 1 -- 172.21.15.28:0/1896529471 >> 172.21.15.28:0/1896529471 conn(0x563bf9e0a8a0 msgr2=0x563bf9e0ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.805+0000 7f767c80b740 1 -- 172.21.15.28:0/1896529471 shutdown_connections 2024-04-25T01:36:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.805+0000 7f767c80b740 1 -- 172.21.15.28:0/1896529471 wait complete. 2024-04-25T01:36:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.805+0000 7f767c80b740 1 -- start start 2024-04-25T01:36:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.806+0000 7f767c80b740 1 Processor -- start 2024-04-25T01:36:56.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.806+0000 7f767c80b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0e580 0x563bf9e18ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.806+0000 7f767c80b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e0eea0 0x563bf9e161b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.806+0000 7f767c80b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 0x563bf9e16710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.806+0000 7f767c80b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563bf9e1c360 con 0x563bf9e19570 2024-04-25T01:36:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.806+0000 7f767c80b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563bf9d2f050 con 0x563bf9e0e580 2024-04-25T01:36:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.806+0000 7f767c80b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x563bf9e1c4e0 con 0x563bf9e0eea0 2024-04-25T01:36:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.806+0000 7f767c80b740 10 client.0 _ll_register_callbacks cb 0x563bf9a86be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.806+0000 7f7673fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 0x563bf9e16710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.806+0000 7f7673fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 0x563bf9e16710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49494/0 (socket says 172.21.15.28:49494) 2024-04-25T01:36:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.806+0000 7f7673fff640 1 -- 172.21.15.28:0/3452639013 learned_addr learned my addr 172.21.15.28:0/3452639013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.806+0000 7f7672ffd640 1 --2- 172.21.15.28:0/3452639013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e0eea0 0x563bf9e161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.806+0000 7f76737fe640 1 --2- 172.21.15.28:0/3452639013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0e580 0x563bf9e18ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.806+0000 7f7673fff640 1 -- 172.21.15.28:0/3452639013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e0eea0 msgr2=0x563bf9e161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.806+0000 7f7673fff640 1 --2- 172.21.15.28:0/3452639013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e0eea0 0x563bf9e161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.806+0000 7f7673fff640 1 -- 172.21.15.28:0/3452639013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0e580 msgr2=0x563bf9e18ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.806+0000 7f7673fff640 1 --2- 172.21.15.28:0/3452639013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0e580 0x563bf9e18ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.806+0000 7f7673fff640 1 -- 172.21.15.28:0/3452639013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bf9d00dc0 con 0x563bf9e19570 2024-04-25T01:36:56.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.807+0000 7f7673fff640 1 --2- 172.21.15.28:0/3452639013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 0x563bf9e16710 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f766c012960 tx=0x7f766c0095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dec22353bc406c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.807+0000 7f7652ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.807+0000 7f7652ffd640 1 -- 172.21.15.28:0/3452639013 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766c011d70 con 0x563bf9e19570 2024-04-25T01:36:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.807+0000 7f7652ffd640 1 -- 172.21.15.28:0/3452639013 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f766c01f410 con 0x563bf9e19570 2024-04-25T01:36:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.807+0000 7f767c80b740 1 -- 172.21.15.28:0/3452639013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76680047a0 con 0x563bf9e19570 2024-04-25T01:36:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.807+0000 7f7652ffd640 1 -- 172.21.15.28:0/3452639013 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f766c01f650 con 0x563bf9e19570 2024-04-25T01:36:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.807+0000 7f767c80b740 10 client.16806 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.807+0000 7f767c80b740 1 -- 172.21.15.28:0/3452639013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563bf9a873f0 con 0x563bf9e19570 2024-04-25T01:36:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.807+0000 7f7652ffd640 1 -- 172.21.15.28:0/3452639013 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76680047a0 con 0x563bf9e19570 2024-04-25T01:36:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.807+0000 7f7652ffd640 1 -- 172.21.15.28:0/3452639013 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f766c01b070 con 0x563bf9e19570 2024-04-25T01:36:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.807+0000 7f767c80b740 10 client.16806 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.807+0000 7f767c80b740 10 client.16806 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.807+0000 7f767c80b740 20 client.16806 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.807+0000 7f767c80b740 1 --2- 172.21.15.28:0/3452639013 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563bf9e29620 0x563bf9e2ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.807+0000 7f767c80b740 4 client.16806 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.807+0000 7f767c80b740 1 -- 172.21.15.28:0/3452639013 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563bf9d80bb0 con 0x563bf9e29620 2024-04-25T01:36:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.808+0000 7f76737fe640 1 --2- 172.21.15.28:0/3452639013 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563bf9e29620 0x563bf9e2ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.808+0000 7f76737fe640 1 --2- 172.21.15.28:0/3452639013 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563bf9e29620 0x563bf9e2ba70 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fee7a118369d4a server_cookie=5fc1aaabc852c28a in_seq=0 out_seq=0 2024-04-25T01:36:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.808+0000 7f7652ffd640 10 client.16806 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.809+0000 7f7652ffd640 1 -- 172.21.15.28:0/3452639013 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563bf9d80bb0 con 0x563bf9e29620 2024-04-25T01:36:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.809+0000 7f7652ffd640 10 client.16806 handle_command_reply: tid=0 2024-04-25T01:36:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.809+0000 7f767c80b740 1 client.16806 shutdown 2024-04-25T01:36:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.809+0000 7f767c80b740 1 -- 172.21.15.28:0/3452639013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 msgr2=0x563bf9e16710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.809+0000 7f767c80b740 1 --2- 172.21.15.28:0/3452639013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 0x563bf9e16710 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f766c012960 tx=0x7f766c0095d0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.809+0000 7f767c80b740 1 -- 172.21.15.28:0/3452639013 shutdown_connections 2024-04-25T01:36:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.809+0000 7f767c80b740 1 --2- 172.21.15.28:0/3452639013 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563bf9e29620 0x563bf9e2ba70 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.809+0000 7f767c80b740 1 --2- 172.21.15.28:0/3452639013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 0x563bf9e16710 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.809+0000 7f767c80b740 1 --2- 172.21.15.28:0/3452639013 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e0eea0 0x563bf9e161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.809+0000 7f767c80b740 1 --2- 172.21.15.28:0/3452639013 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0e580 0x563bf9e18ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.809+0000 7f7652ffd640 0 client.16806 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.809+0000 7f767c80b740 1 -- 172.21.15.28:0/3452639013 >> 172.21.15.28:0/3452639013 conn(0x563bf9e0a8a0 msgr2=0x563bf9e09b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.809+0000 7f767c80b740 1 -- 172.21.15.28:0/3452639013 shutdown_connections 2024-04-25T01:36:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.809+0000 7f767c80b740 1 -- 172.21.15.28:0/3452639013 wait complete. 2024-04-25T01:36:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.809+0000 7f767c80b740 20 client.16806 trim_cache size 0 max 16384 2024-04-25T01:36:56.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.822+0000 7f767c80b740 1 Processor -- start 2024-04-25T01:36:56.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.822+0000 7f767c80b740 1 -- start start 2024-04-25T01:36:56.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.822+0000 7f767c80b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e0e580 0x563bf9e25220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.822+0000 7f767c80b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0eea0 0x563bf9e31340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.822+0000 7f767c80b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e19570 0x563bf9e31880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.822+0000 7f767c80b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76741b0c00 con 0x563bf9e0e580 2024-04-25T01:36:56.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.822+0000 7f767c80b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f76741b2740 con 0x563bf9e0eea0 2024-04-25T01:36:56.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.822+0000 7f767c80b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f76741b12d0 con 0x563bf9e19570 2024-04-25T01:36:56.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.823+0000 7f76737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e0e580 0x563bf9e25220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.823+0000 7f76737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e0e580 0x563bf9e25220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49500/0 (socket says 172.21.15.28:49500) 2024-04-25T01:36:56.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.823+0000 7f76737fe640 1 -- 172.21.15.28:0/2446385241 learned_addr learned my addr 172.21.15.28:0/2446385241 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.823+0000 7f7673fff640 1 --2- 172.21.15.28:0/2446385241 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e19570 0x563bf9e31880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.823+0000 7f7672ffd640 1 --2- 172.21.15.28:0/2446385241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0eea0 0x563bf9e31340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.823+0000 7f76737fe640 1 -- 172.21.15.28:0/2446385241 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e19570 msgr2=0x563bf9e31880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.823+0000 7f76737fe640 1 --2- 172.21.15.28:0/2446385241 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e19570 0x563bf9e31880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.823+0000 7f76737fe640 1 -- 172.21.15.28:0/2446385241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0eea0 msgr2=0x563bf9e31340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.823+0000 7f76737fe640 1 --2- 172.21.15.28:0/2446385241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0eea0 0x563bf9e31340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.823+0000 7f76737fe640 1 -- 172.21.15.28:0/2446385241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bf9e16a80 con 0x563bf9e0e580 2024-04-25T01:36:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.823+0000 7f7673fff640 1 --2- 172.21.15.28:0/2446385241 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e19570 0x563bf9e31880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.823+0000 7f76737fe640 1 --2- 172.21.15.28:0/2446385241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e0e580 0x563bf9e25220 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f7664009a50 tx=0x7f7664005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=293a12e02272c256 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.823+0000 7f76597fa640 1 -- 172.21.15.28:0/2446385241 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7664049040 con 0x563bf9e0e580 2024-04-25T01:36:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.823+0000 7f76597fa640 1 -- 172.21.15.28:0/2446385241 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76640bbf10 con 0x563bf9e0e580 2024-04-25T01:36:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.823+0000 7f76597fa640 1 -- 172.21.15.28:0/2446385241 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76640b8580 con 0x563bf9e0e580 2024-04-25T01:36:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.824+0000 7f767c80b740 1 -- 172.21.15.28:0/2446385241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e0e580 msgr2=0x563bf9e25220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.824+0000 7f767c80b740 1 --2- 172.21.15.28:0/2446385241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e0e580 0x563bf9e25220 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f7664009a50 tx=0x7f7664005e50 comp rx=0 tx=0).stop 2024-04-25T01:36:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.824+0000 7f767c80b740 1 -- 172.21.15.28:0/2446385241 shutdown_connections 2024-04-25T01:36:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.824+0000 7f767c80b740 1 --2- 172.21.15.28:0/2446385241 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e19570 0x563bf9e31880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.824+0000 7f767c80b740 1 --2- 172.21.15.28:0/2446385241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0eea0 0x563bf9e31340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.824+0000 7f767c80b740 1 --2- 172.21.15.28:0/2446385241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e0e580 0x563bf9e25220 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.824+0000 7f767c80b740 1 -- 172.21.15.28:0/2446385241 >> 172.21.15.28:0/2446385241 conn(0x563bf9e0a8a0 msgr2=0x563bf9e09b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.824+0000 7f767c80b740 1 -- 172.21.15.28:0/2446385241 shutdown_connections 2024-04-25T01:36:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.824+0000 7f767c80b740 1 -- 172.21.15.28:0/2446385241 wait complete. 2024-04-25T01:36:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.824+0000 7f767c80b740 1 -- start start 2024-04-25T01:36:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.824+0000 7f767c80b740 1 Processor -- start 2024-04-25T01:36:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.824+0000 7f767c80b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0e580 0x563bf9e1f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f767c80b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e0eea0 0x563bf9e1f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f767c80b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 0x563bf9e1fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f767c80b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76741b2070 con 0x563bf9e19570 2024-04-25T01:36:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f767c80b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f76741b19a0 con 0x563bf9e0e580 2024-04-25T01:36:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f767c80b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f76741aebe0 con 0x563bf9e0eea0 2024-04-25T01:36:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f767c80b740 10 client.0 _ll_register_callbacks cb 0x7f766c012610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:56.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f7673fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 0x563bf9e1fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f76737fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0e580 0x563bf9e1f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f7673fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 0x563bf9e1fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49512/0 (socket says 172.21.15.28:49512) 2024-04-25T01:36:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f7673fff640 1 -- 172.21.15.28:0/2087704249 learned_addr learned my addr 172.21.15.28:0/2087704249 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f7672ffd640 1 --2- 172.21.15.28:0/2087704249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e0eea0 0x563bf9e1f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f7673fff640 1 -- 172.21.15.28:0/2087704249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e0eea0 msgr2=0x563bf9e1f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f7673fff640 1 --2- 172.21.15.28:0/2087704249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e0eea0 0x563bf9e1f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f7673fff640 1 -- 172.21.15.28:0/2087704249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0e580 msgr2=0x563bf9e1f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f7673fff640 1 --2- 172.21.15.28:0/2087704249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0e580 0x563bf9e1f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f7673fff640 1 -- 172.21.15.28:0/2087704249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bf9b517d0 con 0x563bf9e19570 2024-04-25T01:36:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f7673fff640 1 --2- 172.21.15.28:0/2087704249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 0x563bf9e1fd30 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f766c014d30 tx=0x7f766c011830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86731bf4449645fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f7652ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f7652ffd640 1 -- 172.21.15.28:0/2087704249 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766c039070 con 0x563bf9e19570 2024-04-25T01:36:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f7652ffd640 1 -- 172.21.15.28:0/2087704249 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f766c011b70 con 0x563bf9e19570 2024-04-25T01:36:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.825+0000 7f767c80b740 1 -- 172.21.15.28:0/2087704249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76640bbf10 con 0x563bf9e19570 2024-04-25T01:36:56.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.826+0000 7f7652ffd640 1 -- 172.21.15.28:0/2087704249 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f766c011d70 con 0x563bf9e19570 2024-04-25T01:36:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.826+0000 7f767c80b740 10 client.16812 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.826+0000 7f767c80b740 1 -- 172.21.15.28:0/2087704249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563bf9e8fcf0 con 0x563bf9e19570 2024-04-25T01:36:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.826+0000 7f7652ffd640 1 -- 172.21.15.28:0/2087704249 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76640bbf10 con 0x563bf9e19570 2024-04-25T01:36:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.826+0000 7f7652ffd640 1 -- 172.21.15.28:0/2087704249 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f766c01b070 con 0x563bf9e19570 2024-04-25T01:36:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.826+0000 7f767c80b740 10 client.16812 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.826+0000 7f767c80b740 10 client.16812 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.826+0000 7f767c80b740 20 client.16812 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.826+0000 7f767c80b740 1 --2- 172.21.15.28:0/2087704249 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563bf9e90470 0x563bf9e92890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.826+0000 7f767c80b740 4 client.16812 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.826+0000 7f767c80b740 1 -- 172.21.15.28:0/2087704249 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563bf9d7af00 con 0x563bf9e90470 2024-04-25T01:36:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.826+0000 7f76737fe640 1 --2- 172.21.15.28:0/2087704249 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563bf9e90470 0x563bf9e92890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.827+0000 7f7652ffd640 10 client.16812 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.827+0000 7f76737fe640 1 --2- 172.21.15.28:0/2087704249 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563bf9e90470 0x563bf9e92890 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffbf4311cc9f71ac server_cookie=94933daccb21bef4 in_seq=0 out_seq=0 2024-04-25T01:36:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.827+0000 7f7652ffd640 1 -- 172.21.15.28:0/2087704249 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563bf9d7af00 con 0x563bf9e90470 2024-04-25T01:36:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.827+0000 7f7652ffd640 10 client.16812 handle_command_reply: tid=0 2024-04-25T01:36:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.827+0000 7f767c80b740 1 client.16812 shutdown 2024-04-25T01:36:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.827+0000 7f767c80b740 1 -- 172.21.15.28:0/2087704249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 msgr2=0x563bf9e1fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.827+0000 7f767c80b740 1 --2- 172.21.15.28:0/2087704249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 0x563bf9e1fd30 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f766c014d30 tx=0x7f766c011830 comp rx=0 tx=0).stop 2024-04-25T01:36:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.827+0000 7f767c80b740 1 -- 172.21.15.28:0/2087704249 shutdown_connections 2024-04-25T01:36:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.827+0000 7f767c80b740 1 --2- 172.21.15.28:0/2087704249 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563bf9e90470 0x563bf9e92890 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.827+0000 7f767c80b740 1 --2- 172.21.15.28:0/2087704249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bf9e19570 0x563bf9e1fd30 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.827+0000 7f767c80b740 1 --2- 172.21.15.28:0/2087704249 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563bf9e0eea0 0x563bf9e1f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.827+0000 7f767c80b740 1 --2- 172.21.15.28:0/2087704249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563bf9e0e580 0x563bf9e1f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.827+0000 7f767c80b740 1 -- 172.21.15.28:0/2087704249 >> 172.21.15.28:0/2087704249 conn(0x563bf9e0a8a0 msgr2=0x563bf9e24210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.827+0000 7f7652ffd640 0 client.16812 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.827+0000 7f767c80b740 1 -- 172.21.15.28:0/2087704249 shutdown_connections 2024-04-25T01:36:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.827+0000 7f767c80b740 1 -- 172.21.15.28:0/2087704249 wait complete. 2024-04-25T01:36:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.827+0000 7f767c80b740 20 client.16812 trim_cache size 0 max 16384 2024-04-25T01:36:56.810 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:36:56.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.828+0000 7f7679b09640 1 -- 172.21.15.28:0/434844543 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f765c0a2d00 msgr2=0x7f765c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.828+0000 7f7679b09640 1 --2- 172.21.15.28:0/434844543 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f765c0a2d00 0x7f765c0a51c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f766c005fd0 tx=0x7f766c004ab0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.828+0000 7f7679b09640 1 -- 172.21.15.28:0/434844543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7674101760 msgr2=0x7f7674073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.828+0000 7f7679b09640 1 --2- 172.21.15.28:0/434844543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7674101760 0x7f7674073a40 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f7664009230 tx=0x7f7664004270 comp rx=0 tx=0).stop 2024-04-25T01:36:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.828+0000 7f7679b09640 1 -- 172.21.15.28:0/434844543 shutdown_connections 2024-04-25T01:36:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.828+0000 7f7679b09640 1 --2- 172.21.15.28:0/434844543 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f765c0a2d00 0x7f765c0a51c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.828+0000 7f7679b09640 1 --2- 172.21.15.28:0/434844543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7674107770 0x7f76740744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.828+0000 7f7679b09640 1 --2- 172.21.15.28:0/434844543 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7674102120 0x7f7674073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.828+0000 7f7679b09640 1 --2- 172.21.15.28:0/434844543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7674101760 0x7f7674073a40 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.828+0000 7f7679b09640 1 -- 172.21.15.28:0/434844543 >> 172.21.15.28:0/434844543 conn(0x7f76740fd250 msgr2=0x7f76740fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.828+0000 7f7679b09640 1 -- 172.21.15.28:0/434844543 shutdown_connections 2024-04-25T01:36:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:56.828+0000 7f7679b09640 1 -- 172.21.15.28:0/434844543 wait complete. 2024-04-25T01:36:57.815 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:36:57.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:36:57.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:36:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.959+0000 7f54cce38640 1 -- 172.21.15.28:0/2389092795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54c8107730 msgr2=0x7f54c8107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.959+0000 7f54cce38640 1 --2- 172.21.15.28:0/2389092795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54c8107730 0x7f54c8107b10 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009930 tx=0x7f54b802a420 comp rx=0 tx=0).stop 2024-04-25T01:36:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.959+0000 7f54cce38640 1 -- 172.21.15.28:0/2389092795 shutdown_connections 2024-04-25T01:36:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.959+0000 7f54cce38640 1 --2- 172.21.15.28:0/2389092795 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54c806a780 0x7f54c810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.959+0000 7f54cce38640 1 --2- 172.21.15.28:0/2389092795 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54c8069dc0 0x7f54c806a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.959+0000 7f54cce38640 1 --2- 172.21.15.28:0/2389092795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54c8107730 0x7f54c8107b10 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.959+0000 7f54cce38640 1 -- 172.21.15.28:0/2389092795 >> 172.21.15.28:0/2389092795 conn(0x7f54c80776a0 msgr2=0x7f54c8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.959+0000 7f54cce38640 1 -- 172.21.15.28:0/2389092795 shutdown_connections 2024-04-25T01:36:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.959+0000 7f54cce38640 1 -- 172.21.15.28:0/2389092795 wait complete. 2024-04-25T01:36:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.960+0000 7f54cce38640 1 Processor -- start 2024-04-25T01:36:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.960+0000 7f54cce38640 1 -- start start 2024-04-25T01:36:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.960+0000 7f54cce38640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54c8069dc0 0x7f54c81981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:57.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.960+0000 7f54cce38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54c806a780 0x7f54c8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:57.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.960+0000 7f54cce38640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54c8107730 0x7f54c819ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:57.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.960+0000 7f54cce38640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54c8111b40 con 0x7f54c806a780 2024-04-25T01:36:57.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.960+0000 7f54cce38640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f54c81119c0 con 0x7f54c8107730 2024-04-25T01:36:57.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.960+0000 7f54cce38640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f54c8111cc0 con 0x7f54c8069dc0 2024-04-25T01:36:57.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.960+0000 7f54c5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54c806a780 0x7f54c8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:57.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.960+0000 7f54c5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54c806a780 0x7f54c8198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49518/0 (socket says 172.21.15.28:49518) 2024-04-25T01:36:57.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.960+0000 7f54c5d9b640 1 -- 172.21.15.28:0/1783635963 learned_addr learned my addr 172.21.15.28:0/1783635963 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:57.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.960+0000 7f54c659c640 1 --2- 172.21.15.28:0/1783635963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54c8069dc0 0x7f54c81981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:57.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.960+0000 7f54c6d9d640 1 --2- 172.21.15.28:0/1783635963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54c8107730 0x7f54c819ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:57.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.961+0000 7f54c5d9b640 1 -- 172.21.15.28:0/1783635963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54c8069dc0 msgr2=0x7f54c81981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:57.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.961+0000 7f54c5d9b640 1 --2- 172.21.15.28:0/1783635963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54c8069dc0 0x7f54c81981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.961+0000 7f54c5d9b640 1 -- 172.21.15.28:0/1783635963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54c8107730 msgr2=0x7f54c819ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:57.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.961+0000 7f54c5d9b640 1 --2- 172.21.15.28:0/1783635963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54c8107730 0x7f54c819ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.961+0000 7f54c5d9b640 1 -- 172.21.15.28:0/1783635963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54c819d190 con 0x7f54c806a780 2024-04-25T01:36:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.961+0000 7f54c6d9d640 1 --2- 172.21.15.28:0/1783635963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54c8107730 0x7f54c819ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.961+0000 7f54c5d9b640 1 --2- 172.21.15.28:0/1783635963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54c806a780 0x7f54c8198730 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00d950 tx=0x7f54bc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.961+0000 7f54b77fe640 1 -- 172.21.15.28:0/1783635963 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc017070 con 0x7f54c806a780 2024-04-25T01:36:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.961+0000 7f54b77fe640 1 -- 172.21.15.28:0/1783635963 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54bc009c30 con 0x7f54c806a780 2024-04-25T01:36:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.961+0000 7f54cce38640 1 -- 172.21.15.28:0/1783635963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54c819d480 con 0x7f54c806a780 2024-04-25T01:36:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.961+0000 7f54b77fe640 1 -- 172.21.15.28:0/1783635963 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc00f6b0 con 0x7f54c806a780 2024-04-25T01:36:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.961+0000 7f54cce38640 1 -- 172.21.15.28:0/1783635963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54c81a4d10 con 0x7f54c806a780 2024-04-25T01:36:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.963+0000 7f54b77fe640 1 -- 172.21.15.28:0/1783635963 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f54bc0109d0 con 0x7f54c806a780 2024-04-25T01:36:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.964+0000 7f54cfc1d740 1 Processor -- start 2024-04-25T01:36:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.964+0000 7f54cfc1d740 1 -- start start 2024-04-25T01:36:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.964+0000 7f54cfc1d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 0x562a7b88ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.964+0000 7f54cfc1d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b88f290 0x562a7b88f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.964+0000 7f54b77fe640 1 --2- 172.21.15.28:0/1783635963 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f54b00a2d00 0x7f54b00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.964+0000 7f54cfc1d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b89e420 0x562a7b8a0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54cfc1d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562a7b7f32c0 con 0x562a7b890980 2024-04-25T01:36:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54cfc1d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x562a7b79b910 con 0x562a7b88f290 2024-04-25T01:36:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54cfc1d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x562a7b7ef280 con 0x562a7b89e420 2024-04-25T01:36:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54c6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 0x562a7b88ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54c659c640 1 --2- 172.21.15.28:0/1783635963 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f54b00a2d00 0x7f54b00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54c6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 0x562a7b88ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49532/0 (socket says 172.21.15.28:49532) 2024-04-25T01:36:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54c6d9d640 1 -- 172.21.15.28:0/2571549776 learned_addr learned my addr 172.21.15.28:0/2571549776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54b77fe640 1 -- 172.21.15.28:0/1783635963 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f54bc0cf440 con 0x7f54c806a780 2024-04-25T01:36:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54c5d9b640 1 --2- 172.21.15.28:0/2571549776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b88f290 0x562a7b88f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54c6d9d640 1 --2- 172.21.15.28:0/2571549776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b89e420 0x562a7b8a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54c6d9d640 1 -- 172.21.15.28:0/2571549776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b89e420 msgr2=0x562a7b8a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54c6d9d640 1 --2- 172.21.15.28:0/2571549776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b89e420 0x562a7b8a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54c6d9d640 1 -- 172.21.15.28:0/2571549776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b88f290 msgr2=0x562a7b88f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54c6d9d640 1 --2- 172.21.15.28:0/2571549776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b88f290 0x562a7b88f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54c6d9d640 1 -- 172.21.15.28:0/2571549776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a7b7acfb0 con 0x562a7b890980 2024-04-25T01:36:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54c6d9d640 1 --2- 172.21.15.28:0/2571549776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 0x562a7b88ecc0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f54c0014800 tx=0x7f54c0009910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dd64d310973678a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54c659c640 1 --2- 172.21.15.28:0/1783635963 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f54b00a2d00 0x7f54b00a51c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009900 tx=0x7f54b80058d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54b4ff9640 1 -- 172.21.15.28:0/2571549776 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c001a030 con 0x562a7b890980 2024-04-25T01:36:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54b4ff9640 1 -- 172.21.15.28:0/2571549776 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54c0014e40 con 0x562a7b890980 2024-04-25T01:36:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.965+0000 7f54b4ff9640 1 -- 172.21.15.28:0/2571549776 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c00053a0 con 0x562a7b890980 2024-04-25T01:36:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.966+0000 7f54cfc1d740 1 -- 172.21.15.28:0/2571549776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 msgr2=0x562a7b88ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.966+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/2571549776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 0x562a7b88ecc0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f54c0014800 tx=0x7f54c0009910 comp rx=0 tx=0).stop 2024-04-25T01:36:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.966+0000 7f54cfc1d740 1 -- 172.21.15.28:0/2571549776 shutdown_connections 2024-04-25T01:36:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.966+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/2571549776 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b89e420 0x562a7b8a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.966+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/2571549776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b88f290 0x562a7b88f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.966+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/2571549776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 0x562a7b88ecc0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.966+0000 7f54cfc1d740 1 -- 172.21.15.28:0/2571549776 >> 172.21.15.28:0/2571549776 conn(0x562a7b857950 msgr2=0x562a7b88b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.966+0000 7f54cfc1d740 1 -- 172.21.15.28:0/2571549776 shutdown_connections 2024-04-25T01:36:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.966+0000 7f54cfc1d740 1 -- 172.21.15.28:0/2571549776 wait complete. 2024-04-25T01:36:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.967+0000 7f54cfc1d740 1 -- start start 2024-04-25T01:36:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.967+0000 7f54cfc1d740 1 Processor -- start 2024-04-25T01:36:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.967+0000 7f54cfc1d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b88f290 0x562a7b89b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.967+0000 7f54cfc1d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 0x562a7b89b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.967+0000 7f54cfc1d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b89e420 0x562a7b89bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.967+0000 7f54cfc1d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562a7b8a12b0 con 0x562a7b890980 2024-04-25T01:36:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.967+0000 7f54cfc1d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x562a7b7aee90 con 0x562a7b89e420 2024-04-25T01:36:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.967+0000 7f54cfc1d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x562a7b8a1430 con 0x562a7b88f290 2024-04-25T01:36:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.967+0000 7f54cfc1d740 10 client.0 _ll_register_callbacks cb 0x562a7b7ea970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.967+0000 7f54c5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 0x562a7b89b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.967+0000 7f54c5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 0x562a7b89b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49542/0 (socket says 172.21.15.28:49542) 2024-04-25T01:36:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.967+0000 7f54c5d9b640 1 -- 172.21.15.28:0/3797244069 learned_addr learned my addr 172.21.15.28:0/3797244069 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.967+0000 7f54c659c640 1 --2- 172.21.15.28:0/3797244069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b88f290 0x562a7b89b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.967+0000 7f54c6d9d640 1 --2- 172.21.15.28:0/3797244069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b89e420 0x562a7b89bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.967+0000 7f54c5d9b640 1 -- 172.21.15.28:0/3797244069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b88f290 msgr2=0x562a7b89b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.967+0000 7f54c5d9b640 1 --2- 172.21.15.28:0/3797244069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b88f290 0x562a7b89b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54c5d9b640 1 -- 172.21.15.28:0/3797244069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b89e420 msgr2=0x562a7b89bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54c5d9b640 1 --2- 172.21.15.28:0/3797244069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b89e420 0x562a7b89bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54c5d9b640 1 -- 172.21.15.28:0/3797244069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a7b5cf7e0 con 0x562a7b890980 2024-04-25T01:36:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54c659c640 1 --2- 172.21.15.28:0/3797244069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b88f290 0x562a7b89b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54c6d9d640 1 --2- 172.21.15.28:0/3797244069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b89e420 0x562a7b89bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54c5d9b640 1 --2- 172.21.15.28:0/3797244069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 0x562a7b89b810 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f54bc009610 tx=0x7f54bc09b490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70d768115886c58d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54adffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54adffb640 1 -- 172.21.15.28:0/3797244069 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc020040 con 0x562a7b890980 2024-04-25T01:36:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54adffb640 1 -- 172.21.15.28:0/3797244069 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f54bc0b5d90 con 0x562a7b890980 2024-04-25T01:36:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54cfc1d740 1 -- 172.21.15.28:0/3797244069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562a7b7c75a0 con 0x562a7b890980 2024-04-25T01:36:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54adffb640 1 -- 172.21.15.28:0/3797244069 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f54bc09bcc0 con 0x562a7b890980 2024-04-25T01:36:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54cfc1d740 10 client.16824 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54cfc1d740 1 -- 172.21.15.28:0/3797244069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562a7b62ff70 con 0x562a7b890980 2024-04-25T01:36:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54adffb640 1 -- 172.21.15.28:0/3797244069 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x562a7b7c75a0 con 0x562a7b890980 2024-04-25T01:36:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54adffb640 1 -- 172.21.15.28:0/3797244069 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f54bc0ccb00 con 0x562a7b890980 2024-04-25T01:36:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54cfc1d740 10 client.16824 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:57.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54cfc1d740 10 client.16824 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:57.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54cfc1d740 20 client.16824 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:57.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/3797244069 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x562a7b8ae3b0 0x562a7b8b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:57.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54cfc1d740 4 client.16824 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:57.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.968+0000 7f54cfc1d740 1 -- 172.21.15.28:0/3797244069 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a7b800800 con 0x562a7b8ae3b0 2024-04-25T01:36:57.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.969+0000 7f54c659c640 1 --2- 172.21.15.28:0/3797244069 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x562a7b8ae3b0 0x562a7b8b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:57.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.969+0000 7f54c659c640 1 --2- 172.21.15.28:0/3797244069 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x562a7b8ae3b0 0x562a7b8b07e0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c68bc6bf6b1ece01 server_cookie=d7b101feed337e72 in_seq=0 out_seq=0 2024-04-25T01:36:57.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.969+0000 7f54adffb640 10 client.16824 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:57.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.970+0000 7f54adffb640 1 -- 172.21.15.28:0/3797244069 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562a7b800800 con 0x562a7b8ae3b0 2024-04-25T01:36:57.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.970+0000 7f54adffb640 10 client.16824 handle_command_reply: tid=0 2024-04-25T01:36:57.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.970+0000 7f54cfc1d740 1 client.16824 shutdown 2024-04-25T01:36:57.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.970+0000 7f54cfc1d740 1 -- 172.21.15.28:0/3797244069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 msgr2=0x562a7b89b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:57.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.970+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/3797244069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 0x562a7b89b810 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f54bc009610 tx=0x7f54bc09b490 comp rx=0 tx=0).stop 2024-04-25T01:36:57.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.970+0000 7f54cfc1d740 1 -- 172.21.15.28:0/3797244069 shutdown_connections 2024-04-25T01:36:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.970+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/3797244069 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x562a7b8ae3b0 0x562a7b8b07e0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.970+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/3797244069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b89e420 0x562a7b89bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.970+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/3797244069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 0x562a7b89b810 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.970+0000 7f54adffb640 0 client.16824 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.970+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/3797244069 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b88f290 0x562a7b89b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.970+0000 7f54cfc1d740 1 -- 172.21.15.28:0/3797244069 >> 172.21.15.28:0/3797244069 conn(0x562a7b857950 msgr2=0x562a7b8915f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.970+0000 7f54cfc1d740 1 -- 172.21.15.28:0/3797244069 shutdown_connections 2024-04-25T01:36:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.970+0000 7f54cfc1d740 1 -- 172.21.15.28:0/3797244069 wait complete. 2024-04-25T01:36:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.970+0000 7f54cfc1d740 20 client.16824 trim_cache size 0 max 16384 2024-04-25T01:36:57.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.988+0000 7f54cfc1d740 1 Processor -- start 2024-04-25T01:36:57.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.988+0000 7f54cfc1d740 1 -- start start 2024-04-25T01:36:57.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.988+0000 7f54cfc1d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b88f290 0x562a7b8aa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:57.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.988+0000 7f54cfc1d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 0x562a7b89a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:57.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.988+0000 7f54cfc1d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b89e420 0x562a7b88a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:57.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.988+0000 7f54cfc1d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54c81ac950 con 0x562a7b890980 2024-04-25T01:36:57.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.988+0000 7f54cfc1d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f54c81ae490 con 0x562a7b88f290 2024-04-25T01:36:57.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.988+0000 7f54cfc1d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f54c81ad020 con 0x562a7b89e420 2024-04-25T01:36:57.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.988+0000 7f54c5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 0x562a7b89a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:57.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.989+0000 7f54c5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 0x562a7b89a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49552/0 (socket says 172.21.15.28:49552) 2024-04-25T01:36:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.989+0000 7f54c5d9b640 1 -- 172.21.15.28:0/4076493992 learned_addr learned my addr 172.21.15.28:0/4076493992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.989+0000 7f54c659c640 1 --2- 172.21.15.28:0/4076493992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b88f290 0x562a7b8aa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.989+0000 7f54c6d9d640 1 --2- 172.21.15.28:0/4076493992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b89e420 0x562a7b88a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.989+0000 7f54c5d9b640 1 -- 172.21.15.28:0/4076493992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b89e420 msgr2=0x562a7b88a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.989+0000 7f54c5d9b640 1 --2- 172.21.15.28:0/4076493992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b89e420 0x562a7b88a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.989+0000 7f54c5d9b640 1 -- 172.21.15.28:0/4076493992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b88f290 msgr2=0x562a7b8aa1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.989+0000 7f54c5d9b640 1 --2- 172.21.15.28:0/4076493992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b88f290 0x562a7b8aa1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.989+0000 7f54c5d9b640 1 -- 172.21.15.28:0/4076493992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a7b4c7170 con 0x562a7b890980 2024-04-25T01:36:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.989+0000 7f54c659c640 1 --2- 172.21.15.28:0/4076493992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b88f290 0x562a7b8aa1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.989+0000 7f54c5d9b640 1 --2- 172.21.15.28:0/4076493992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 0x562a7b89a430 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f54bc0058c0 tx=0x7f54bc0cf370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1017815c2dfdcac0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.989+0000 7f54affff640 1 -- 172.21.15.28:0/4076493992 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc020030 con 0x562a7b890980 2024-04-25T01:36:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.989+0000 7f54affff640 1 -- 172.21.15.28:0/4076493992 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54bc09b5c0 con 0x562a7b890980 2024-04-25T01:36:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.989+0000 7f54affff640 1 -- 172.21.15.28:0/4076493992 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc0afae0 con 0x562a7b890980 2024-04-25T01:36:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.990+0000 7f54cfc1d740 1 -- 172.21.15.28:0/4076493992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 msgr2=0x562a7b89a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.990+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/4076493992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 0x562a7b89a430 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f54bc0058c0 tx=0x7f54bc0cf370 comp rx=0 tx=0).stop 2024-04-25T01:36:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.990+0000 7f54cfc1d740 1 -- 172.21.15.28:0/4076493992 shutdown_connections 2024-04-25T01:36:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.990+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/4076493992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b89e420 0x562a7b88a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.990+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/4076493992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b890980 0x562a7b89a430 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.990+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/4076493992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b88f290 0x562a7b8aa1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.990+0000 7f54cfc1d740 1 -- 172.21.15.28:0/4076493992 >> 172.21.15.28:0/4076493992 conn(0x562a7b857950 msgr2=0x562a7b88a0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.990+0000 7f54cfc1d740 1 -- 172.21.15.28:0/4076493992 shutdown_connections 2024-04-25T01:36:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.990+0000 7f54cfc1d740 1 -- 172.21.15.28:0/4076493992 wait complete. 2024-04-25T01:36:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.990+0000 7f54cfc1d740 1 -- start start 2024-04-25T01:36:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.990+0000 7f54cfc1d740 1 Processor -- start 2024-04-25T01:36:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.990+0000 7f54cfc1d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b88f290 0x562a7b8982a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.990+0000 7f54cfc1d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b890980 0x562a7b8987e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.990+0000 7f54cfc1d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b89e420 0x562a7b913920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.990+0000 7f54cfc1d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54c81addc0 con 0x562a7b89e420 2024-04-25T01:36:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.990+0000 7f54cfc1d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f54c81ad6f0 con 0x562a7b88f290 2024-04-25T01:36:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.990+0000 7f54cfc1d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f54c81aa930 con 0x562a7b890980 2024-04-25T01:36:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.990+0000 7f54cfc1d740 10 client.0 _ll_register_callbacks cb 0x7f54bc097ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.991+0000 7f54c6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b89e420 0x562a7b913920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.991+0000 7f54c6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b89e420 0x562a7b913920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49560/0 (socket says 172.21.15.28:49560) 2024-04-25T01:36:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.991+0000 7f54c6d9d640 1 -- 172.21.15.28:0/2638947991 learned_addr learned my addr 172.21.15.28:0/2638947991 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.991+0000 7f54c5d9b640 1 --2- 172.21.15.28:0/2638947991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b890980 0x562a7b8987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.991+0000 7f54c659c640 1 --2- 172.21.15.28:0/2638947991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b88f290 0x562a7b8982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.991+0000 7f54c6d9d640 1 -- 172.21.15.28:0/2638947991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b890980 msgr2=0x562a7b8987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.991+0000 7f54c6d9d640 1 --2- 172.21.15.28:0/2638947991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b890980 0x562a7b8987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.991+0000 7f54c6d9d640 1 -- 172.21.15.28:0/2638947991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b88f290 msgr2=0x562a7b8982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.991+0000 7f54c6d9d640 1 --2- 172.21.15.28:0/2638947991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b88f290 0x562a7b8982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.991+0000 7f54c6d9d640 1 -- 172.21.15.28:0/2638947991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a7b899070 con 0x562a7b89e420 2024-04-25T01:36:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.991+0000 7f54c659c640 1 --2- 172.21.15.28:0/2638947991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b88f290 0x562a7b8982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.991+0000 7f54c6d9d640 1 --2- 172.21.15.28:0/2638947991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b89e420 0x562a7b913920 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f54c0005f00 tx=0x7f54c00097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da507a95ca01a1c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.991+0000 7f54adffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.991+0000 7f54adffb640 1 -- 172.21.15.28:0/2638947991 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c002e060 con 0x562a7b89e420 2024-04-25T01:36:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.991+0000 7f54adffb640 1 -- 172.21.15.28:0/2638947991 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f54c0014390 con 0x562a7b89e420 2024-04-25T01:36:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.991+0000 7f54cfc1d740 1 -- 172.21.15.28:0/2638947991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54bc09b5c0 con 0x562a7b89e420 2024-04-25T01:36:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.991+0000 7f54adffb640 1 -- 172.21.15.28:0/2638947991 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f54c0035050 con 0x562a7b89e420 2024-04-25T01:36:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.991+0000 7f54cfc1d740 10 client.16830 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.991+0000 7f54cfc1d740 1 -- 172.21.15.28:0/2638947991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562a7b914f70 con 0x562a7b89e420 2024-04-25T01:36:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.992+0000 7f54adffb640 1 -- 172.21.15.28:0/2638947991 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54bc09b5c0 con 0x562a7b89e420 2024-04-25T01:36:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.992+0000 7f54adffb640 1 -- 172.21.15.28:0/2638947991 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f54c000f040 con 0x562a7b89e420 2024-04-25T01:36:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.992+0000 7f54cfc1d740 10 client.16830 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.992+0000 7f54cfc1d740 10 client.16830 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.992+0000 7f54cfc1d740 20 client.16830 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.992+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/2638947991 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x562a7b915380 0x562a7b9177a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.992+0000 7f54cfc1d740 4 client.16830 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:36:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.992+0000 7f54cfc1d740 1 -- 172.21.15.28:0/2638947991 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562a7b7fab50 con 0x562a7b915380 2024-04-25T01:36:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.992+0000 7f54c659c640 1 --2- 172.21.15.28:0/2638947991 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x562a7b915380 0x562a7b9177a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.992+0000 7f54adffb640 10 client.16830 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.992+0000 7f54c659c640 1 --2- 172.21.15.28:0/2638947991 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x562a7b915380 0x562a7b9177a0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9284bf92c869fb54 server_cookie=f601b2107a9fb8bf in_seq=0 out_seq=0 2024-04-25T01:36:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.993+0000 7f54adffb640 1 -- 172.21.15.28:0/2638947991 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562a7b7fab50 con 0x562a7b915380 2024-04-25T01:36:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.993+0000 7f54adffb640 10 client.16830 handle_command_reply: tid=0 2024-04-25T01:36:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.993+0000 7f54cfc1d740 1 client.16830 shutdown 2024-04-25T01:36:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.994+0000 7f54cfc1d740 1 -- 172.21.15.28:0/2638947991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b89e420 msgr2=0x562a7b913920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.994+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/2638947991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b89e420 0x562a7b913920 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f54c0005f00 tx=0x7f54c00097b0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.994+0000 7f54cfc1d740 1 -- 172.21.15.28:0/2638947991 shutdown_connections 2024-04-25T01:36:57.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.994+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/2638947991 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x562a7b915380 0x562a7b9177a0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.994+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/2638947991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a7b89e420 0x562a7b913920 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.994+0000 7f54adffb640 0 client.16830 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:57.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.994+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/2638947991 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a7b890980 0x562a7b8987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.994+0000 7f54cfc1d740 1 --2- 172.21.15.28:0/2638947991 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a7b88f290 0x562a7b8982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.994+0000 7f54cfc1d740 1 -- 172.21.15.28:0/2638947991 >> 172.21.15.28:0/2638947991 conn(0x562a7b857950 msgr2=0x562a7b89cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:57.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.994+0000 7f54cfc1d740 1 -- 172.21.15.28:0/2638947991 shutdown_connections 2024-04-25T01:36:57.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.994+0000 7f54cfc1d740 1 -- 172.21.15.28:0/2638947991 wait complete. 2024-04-25T01:36:57.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.994+0000 7f54cfc1d740 20 client.16830 trim_cache size 0 max 16384 2024-04-25T01:36:57.980 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:36:57.980 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:36:57.980 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "b1bdf961-5ed2-474a-8bb9-f82b7bf3b794", 2024-04-25T01:36:57.980 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:36:57.980 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:36:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.994+0000 7f54cce38640 1 -- 172.21.15.28:0/1783635963 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f54b00a2d00 msgr2=0x7f54b00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.994+0000 7f54cce38640 1 --2- 172.21.15.28:0/1783635963 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f54b00a2d00 0x7f54b00a51c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009900 tx=0x7f54b80058d0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.994+0000 7f54cce38640 1 -- 172.21.15.28:0/1783635963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54c806a780 msgr2=0x7f54c8198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.994+0000 7f54cce38640 1 --2- 172.21.15.28:0/1783635963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54c806a780 0x7f54c8198730 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00d950 tx=0x7f54bc00de10 comp rx=0 tx=0).stop 2024-04-25T01:36:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.995+0000 7f54cce38640 1 -- 172.21.15.28:0/1783635963 shutdown_connections 2024-04-25T01:36:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.995+0000 7f54cce38640 1 --2- 172.21.15.28:0/1783635963 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f54b00a2d00 0x7f54b00a51c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.995+0000 7f54cce38640 1 --2- 172.21.15.28:0/1783635963 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f54c8107730 0x7f54c819ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.995+0000 7f54cce38640 1 --2- 172.21.15.28:0/1783635963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54c806a780 0x7f54c8198730 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.995+0000 7f54cce38640 1 --2- 172.21.15.28:0/1783635963 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f54c8069dc0 0x7f54c81981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.995+0000 7f54cce38640 1 -- 172.21.15.28:0/1783635963 >> 172.21.15.28:0/1783635963 conn(0x7f54c80776a0 msgr2=0x7f54c80fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.995+0000 7f54cce38640 1 -- 172.21.15.28:0/1783635963 shutdown_connections 2024-04-25T01:36:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:57.995+0000 7f54cce38640 1 -- 172.21.15.28:0/1783635963 wait complete. 2024-04-25T01:36:57.983 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:36:57.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:36:58.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.111+0000 7f48f799d640 1 -- 172.21.15.28:0/3036198285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f0107720 msgr2=0x7f48f0107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.111+0000 7f48f799d640 1 --2- 172.21.15.28:0/3036198285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f0107720 0x7f48f0107b00 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f48ec009930 tx=0x7f48ec02a420 comp rx=0 tx=0).stop 2024-04-25T01:36:58.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.111+0000 7f48f799d640 1 -- 172.21.15.28:0/3036198285 shutdown_connections 2024-04-25T01:36:58.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.111+0000 7f48f799d640 1 --2- 172.21.15.28:0/3036198285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48f01020d0 0x7f48f010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.111+0000 7f48f799d640 1 --2- 172.21.15.28:0/3036198285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48f0101710 0x7f48f0101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.111+0000 7f48f799d640 1 --2- 172.21.15.28:0/3036198285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f0107720 0x7f48f0107b00 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.111+0000 7f48f799d640 1 -- 172.21.15.28:0/3036198285 >> 172.21.15.28:0/3036198285 conn(0x7f48f00fd180 msgr2=0x7f48f00ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.111+0000 7f48f799d640 1 -- 172.21.15.28:0/3036198285 shutdown_connections 2024-04-25T01:36:58.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.111+0000 7f48f799d640 1 -- 172.21.15.28:0/3036198285 wait complete. 2024-04-25T01:36:58.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.112+0000 7f48f799d640 1 Processor -- start 2024-04-25T01:36:58.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.112+0000 7f48f799d640 1 -- start start 2024-04-25T01:36:58.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.112+0000 7f48f799d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48f0101710 0x7f48f0198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.112+0000 7f48f799d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48f01020d0 0x7f48f0198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.112+0000 7f48f799d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f0107720 0x7f48f019c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.112+0000 7f48f799d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48f0111b40 con 0x7f48f0107720 2024-04-25T01:36:58.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.112+0000 7f48f799d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f48f01119c0 con 0x7f48f0101710 2024-04-25T01:36:58.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.112+0000 7f48f799d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f48f0111cc0 con 0x7f48f01020d0 2024-04-25T01:36:58.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.112+0000 7f48f5f3a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f0107720 0x7f48f019c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.112+0000 7f48f5f3a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f0107720 0x7f48f019c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49578/0 (socket says 172.21.15.28:49578) 2024-04-25T01:36:58.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.112+0000 7f48f5f3a640 1 -- 172.21.15.28:0/254726681 learned_addr learned my addr 172.21.15.28:0/254726681 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.112+0000 7f48f5739640 1 --2- 172.21.15.28:0/254726681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48f0101710 0x7f48f0198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.112+0000 7f48f4f38640 1 --2- 172.21.15.28:0/254726681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48f01020d0 0x7f48f0198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.112+0000 7f48f5f3a640 1 -- 172.21.15.28:0/254726681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48f01020d0 msgr2=0x7f48f0198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.112+0000 7f48f5f3a640 1 --2- 172.21.15.28:0/254726681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48f01020d0 0x7f48f0198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.112+0000 7f48f5f3a640 1 -- 172.21.15.28:0/254726681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48f0101710 msgr2=0x7f48f0198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.112+0000 7f48f5f3a640 1 --2- 172.21.15.28:0/254726681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48f0101710 0x7f48f0198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.112+0000 7f48f5f3a640 1 -- 172.21.15.28:0/254726681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48f019d030 con 0x7f48f0107720 2024-04-25T01:36:58.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.112+0000 7f48f5739640 1 --2- 172.21.15.28:0/254726681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48f0101710 0x7f48f0198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:58.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.113+0000 7f48f5f3a640 1 --2- 172.21.15.28:0/254726681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f0107720 0x7f48f019c900 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f48e800eaa0 tx=0x7f48e800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.113+0000 7f48e67fc640 1 -- 172.21.15.28:0/254726681 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e800cdb0 con 0x7f48f0107720 2024-04-25T01:36:58.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.113+0000 7f48e67fc640 1 -- 172.21.15.28:0/254726681 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48e8005020 con 0x7f48f0107720 2024-04-25T01:36:58.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.113+0000 7f48e67fc640 1 -- 172.21.15.28:0/254726681 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e80116b0 con 0x7f48f0107720 2024-04-25T01:36:58.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.113+0000 7f48f799d640 1 -- 172.21.15.28:0/254726681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48f019d320 con 0x7f48f0107720 2024-04-25T01:36:58.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.113+0000 7f48f799d640 1 -- 172.21.15.28:0/254726681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48f01a4bb0 con 0x7f48f0107720 2024-04-25T01:36:58.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.115+0000 7f48e67fc640 1 -- 172.21.15.28:0/254726681 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f48e8005560 con 0x7f48f0107720 2024-04-25T01:36:58.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.115+0000 7f48fa781740 1 Processor -- start 2024-04-25T01:36:58.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.115+0000 7f48fa781740 1 -- start start 2024-04-25T01:36:58.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48fa781740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2e8114510 0x55f2e81148f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48fa781740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8114e30 0x55f2e81234f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48fa781740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8123a30 0x55f2e8125e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48fa781740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f2e80796b0 con 0x55f2e8123a30 2024-04-25T01:36:58.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48fa781740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f2e8021880 con 0x55f2e8114e30 2024-04-25T01:36:58.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48fa781740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55f2e8075670 con 0x55f2e8114510 2024-04-25T01:36:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48e67fc640 1 --2- 172.21.15.28:0/254726681 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f48dc0a2d80 0x7f48dc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48f5f3a640 1 --2- 172.21.15.28:0/254726681 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f48dc0a2d80 0x7f48dc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48f4f38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8123a30 0x55f2e8125e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48f4f38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8123a30 0x55f2e8125e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49582/0 (socket says 172.21.15.28:49582) 2024-04-25T01:36:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48f4f38640 1 -- 172.21.15.28:0/2268661324 learned_addr learned my addr 172.21.15.28:0/2268661324 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48f4f38640 1 --2- 172.21.15.28:0/2268661324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2e8114510 0x55f2e81148f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48f5739640 1 --2- 172.21.15.28:0/2268661324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8114e30 0x55f2e81234f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48e67fc640 1 -- 172.21.15.28:0/254726681 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f48e80cf9d0 con 0x7f48f0107720 2024-04-25T01:36:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48f4f38640 1 -- 172.21.15.28:0/2268661324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2e8114510 msgr2=0x55f2e81148f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48f4f38640 1 --2- 172.21.15.28:0/2268661324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2e8114510 0x55f2e81148f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48f4f38640 1 -- 172.21.15.28:0/2268661324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8114e30 msgr2=0x55f2e81234f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48f4f38640 1 --2- 172.21.15.28:0/2268661324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8114e30 0x55f2e81234f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48f4f38640 1 -- 172.21.15.28:0/2268661324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2e7e577d0 con 0x55f2e8123a30 2024-04-25T01:36:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48f5739640 1 --2- 172.21.15.28:0/2268661324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8114e30 0x55f2e81234f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:58.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.116+0000 7f48f5f3a640 1 --2- 172.21.15.28:0/254726681 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f48dc0a2d80 0x7f48dc0a5240 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f48e80cce90 tx=0x7f48e806ec20 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.117+0000 7f48f4f38640 1 --2- 172.21.15.28:0/2268661324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8123a30 0x55f2e8125e20 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f48e0005fd0 tx=0x7f48e000e3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6a75c4ef0c3899 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.117+0000 7f48db7fe640 1 -- 172.21.15.28:0/2268661324 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e000b680 con 0x55f2e8123a30 2024-04-25T01:36:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.117+0000 7f48db7fe640 1 -- 172.21.15.28:0/2268661324 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48e000bd00 con 0x55f2e8123a30 2024-04-25T01:36:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.117+0000 7f48db7fe640 1 -- 172.21.15.28:0/2268661324 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e0011e30 con 0x55f2e8123a30 2024-04-25T01:36:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.117+0000 7f48fa781740 1 -- 172.21.15.28:0/2268661324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8123a30 msgr2=0x55f2e8125e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.117+0000 7f48fa781740 1 --2- 172.21.15.28:0/2268661324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8123a30 0x55f2e8125e20 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f48e0005fd0 tx=0x7f48e000e3d0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.117+0000 7f48fa781740 1 -- 172.21.15.28:0/2268661324 shutdown_connections 2024-04-25T01:36:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.117+0000 7f48fa781740 1 --2- 172.21.15.28:0/2268661324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8123a30 0x55f2e8125e20 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.117+0000 7f48fa781740 1 --2- 172.21.15.28:0/2268661324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8114e30 0x55f2e81234f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.117+0000 7f48fa781740 1 --2- 172.21.15.28:0/2268661324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2e8114510 0x55f2e81148f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.117+0000 7f48fa781740 1 -- 172.21.15.28:0/2268661324 >> 172.21.15.28:0/2268661324 conn(0x55f2e8110950 msgr2=0x55f2e8110d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.117+0000 7f48fa781740 1 -- 172.21.15.28:0/2268661324 shutdown_connections 2024-04-25T01:36:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.117+0000 7f48fa781740 1 -- 172.21.15.28:0/2268661324 wait complete. 2024-04-25T01:36:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.118+0000 7f48fa781740 1 -- start start 2024-04-25T01:36:58.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.118+0000 7f48fa781740 1 Processor -- start 2024-04-25T01:36:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.118+0000 7f48fa781740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8114510 0x55f2e8120960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.118+0000 7f48fa781740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2e8114e30 0x55f2e8120ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.118+0000 7f48fa781740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8123a30 0x55f2e81213e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.118+0000 7f48fa781740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f2e8126870 con 0x55f2e8114510 2024-04-25T01:36:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.118+0000 7f48fa781740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f2e8034f50 con 0x55f2e8123a30 2024-04-25T01:36:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.118+0000 7f48fa781740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55f2e81269f0 con 0x55f2e8114e30 2024-04-25T01:36:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.118+0000 7f48fa781740 10 client.0 _ll_register_callbacks cb 0x55f2e8070000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.118+0000 7f48f4f38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8114510 0x55f2e8120960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.118+0000 7f48f4f38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8114510 0x55f2e8120960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49590/0 (socket says 172.21.15.28:49590) 2024-04-25T01:36:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.118+0000 7f48f4f38640 1 -- 172.21.15.28:0/2349252961 learned_addr learned my addr 172.21.15.28:0/2349252961 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.118+0000 7f48f5739640 1 --2- 172.21.15.28:0/2349252961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2e8114e30 0x55f2e8120ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.119+0000 7f48f4f38640 1 --2- 172.21.15.28:0/2349252961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8123a30 0x55f2e81213e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.119+0000 7f48f4f38640 1 -- 172.21.15.28:0/2349252961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2e8114e30 msgr2=0x55f2e8120ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.119+0000 7f48f4f38640 1 --2- 172.21.15.28:0/2349252961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2e8114e30 0x55f2e8120ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.119+0000 7f48f4f38640 1 -- 172.21.15.28:0/2349252961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8123a30 msgr2=0x55f2e81213e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.119+0000 7f48f4f38640 1 --2- 172.21.15.28:0/2349252961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8123a30 0x55f2e81213e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.119+0000 7f48f4f38640 1 -- 172.21.15.28:0/2349252961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2e7d8d3f0 con 0x55f2e8114510 2024-04-25T01:36:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.119+0000 7f48f4f38640 1 --2- 172.21.15.28:0/2349252961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8114510 0x55f2e8120960 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f48e0011b10 tx=0x7f48e0001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=392b9869d59b9b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.119+0000 7f48d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.119+0000 7f48d8ff9640 1 -- 172.21.15.28:0/2349252961 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e000b680 con 0x55f2e8114510 2024-04-25T01:36:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.119+0000 7f48d8ff9640 1 -- 172.21.15.28:0/2349252961 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f48e000b820 con 0x55f2e8114510 2024-04-25T01:36:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.119+0000 7f48fa781740 1 -- 172.21.15.28:0/2349252961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48e000bd00 con 0x55f2e8114510 2024-04-25T01:36:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.119+0000 7f48d8ff9640 1 -- 172.21.15.28:0/2349252961 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f48e000ba20 con 0x55f2e8114510 2024-04-25T01:36:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.119+0000 7f48fa781740 10 client.16842 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.119+0000 7f48fa781740 1 -- 172.21.15.28:0/2349252961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f2e7d4d0f0 con 0x55f2e8114510 2024-04-25T01:36:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.119+0000 7f48d8ff9640 1 -- 172.21.15.28:0/2349252961 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48e000bd00 con 0x55f2e8114510 2024-04-25T01:36:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.120+0000 7f48d8ff9640 1 -- 172.21.15.28:0/2349252961 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f48e000f040 con 0x55f2e8114510 2024-04-25T01:36:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.120+0000 7f48fa781740 10 client.16842 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.120+0000 7f48fa781740 10 client.16842 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.120+0000 7f48fa781740 20 client.16842 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.120+0000 7f48fa781740 1 --2- 172.21.15.28:0/2349252961 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f2e8133d10 0x55f2e8136100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.120+0000 7f48fa781740 4 client.16842 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.120+0000 7f48fa781740 1 -- 172.21.15.28:0/2349252961 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2e8086bf0 con 0x55f2e8133d10 2024-04-25T01:36:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.120+0000 7f48f5739640 1 --2- 172.21.15.28:0/2349252961 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f2e8133d10 0x55f2e8136100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.120+0000 7f48f5739640 1 --2- 172.21.15.28:0/2349252961 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f2e8133d10 0x55f2e8136100 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9579bfc402551f86 server_cookie=72b304d44339e0d3 in_seq=0 out_seq=0 2024-04-25T01:36:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.120+0000 7f48d8ff9640 10 client.16842 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:58.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.121+0000 7f48d8ff9640 1 -- 172.21.15.28:0/2349252961 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f2e8086bf0 con 0x55f2e8133d10 2024-04-25T01:36:58.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.121+0000 7f48d8ff9640 10 client.16842 handle_command_reply: tid=0 2024-04-25T01:36:58.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.121+0000 7f48fa781740 1 client.16842 shutdown 2024-04-25T01:36:58.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.121+0000 7f48fa781740 1 -- 172.21.15.28:0/2349252961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8114510 msgr2=0x55f2e8120960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.121+0000 7f48fa781740 1 --2- 172.21.15.28:0/2349252961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8114510 0x55f2e8120960 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f48e0011b10 tx=0x7f48e0001e50 comp rx=0 tx=0).stop 2024-04-25T01:36:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.121+0000 7f48fa781740 1 -- 172.21.15.28:0/2349252961 shutdown_connections 2024-04-25T01:36:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.121+0000 7f48fa781740 1 --2- 172.21.15.28:0/2349252961 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f2e8133d10 0x55f2e8136100 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.121+0000 7f48fa781740 1 --2- 172.21.15.28:0/2349252961 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8123a30 0x55f2e81213e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.121+0000 7f48d8ff9640 0 client.16842 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.121+0000 7f48fa781740 1 --2- 172.21.15.28:0/2349252961 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2e8114e30 0x55f2e8120ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.121+0000 7f48fa781740 1 --2- 172.21.15.28:0/2349252961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8114510 0x55f2e8120960 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.121+0000 7f48fa781740 1 -- 172.21.15.28:0/2349252961 >> 172.21.15.28:0/2349252961 conn(0x55f2e8110950 msgr2=0x55f2e8116700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.121+0000 7f48fa781740 1 -- 172.21.15.28:0/2349252961 shutdown_connections 2024-04-25T01:36:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.121+0000 7f48fa781740 1 -- 172.21.15.28:0/2349252961 wait complete. 2024-04-25T01:36:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.122+0000 7f48fa781740 20 client.16842 trim_cache size 0 max 16384 2024-04-25T01:36:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.139+0000 7f48fa781740 1 Processor -- start 2024-04-25T01:36:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.139+0000 7f48fa781740 1 -- start start 2024-04-25T01:36:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.139+0000 7f48fa781740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8114510 0x55f2e812f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.139+0000 7f48fa781740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8114e30 0x55f2e8120150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.139+0000 7f48fa781740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2e8123a30 0x55f2e8121030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.139+0000 7f48fa781740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48f01ac960 con 0x55f2e8114e30 2024-04-25T01:36:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.139+0000 7f48fa781740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f48f01ae4a0 con 0x55f2e8114510 2024-04-25T01:36:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.139+0000 7f48fa781740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f48f01ad030 con 0x55f2e8123a30 2024-04-25T01:36:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.139+0000 7f48f5739640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8114e30 0x55f2e8120150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.139+0000 7f48f5739640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8114e30 0x55f2e8120150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49604/0 (socket says 172.21.15.28:49604) 2024-04-25T01:36:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.139+0000 7f48f5739640 1 -- 172.21.15.28:0/1055744927 learned_addr learned my addr 172.21.15.28:0/1055744927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.139+0000 7f48f4f38640 1 --2- 172.21.15.28:0/1055744927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8114510 0x55f2e812f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.139+0000 7f48f5739640 1 -- 172.21.15.28:0/1055744927 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2e8123a30 msgr2=0x55f2e8121030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.139+0000 7f48f5739640 1 --2- 172.21.15.28:0/1055744927 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2e8123a30 0x55f2e8121030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.139+0000 7f48f5739640 1 -- 172.21.15.28:0/1055744927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8114510 msgr2=0x55f2e812f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.139+0000 7f48f5739640 1 --2- 172.21.15.28:0/1055744927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8114510 0x55f2e812f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.140+0000 7f48f5739640 1 -- 172.21.15.28:0/1055744927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2e811b800 con 0x55f2e8114e30 2024-04-25T01:36:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.140+0000 7f48f5739640 1 --2- 172.21.15.28:0/1055744927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8114e30 0x55f2e8120150 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f48ec02a9e0 tx=0x7f48ec005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5d559fa9cbaefcc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.140+0000 7f48daffd640 1 -- 172.21.15.28:0/1055744927 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ec035040 con 0x55f2e8114e30 2024-04-25T01:36:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.140+0000 7f48daffd640 1 -- 172.21.15.28:0/1055744927 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48ec049070 con 0x55f2e8114e30 2024-04-25T01:36:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.140+0000 7f48daffd640 1 -- 172.21.15.28:0/1055744927 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ec043600 con 0x55f2e8114e30 2024-04-25T01:36:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.140+0000 7f48fa781740 1 -- 172.21.15.28:0/1055744927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8114e30 msgr2=0x55f2e8120150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.140+0000 7f48fa781740 1 --2- 172.21.15.28:0/1055744927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8114e30 0x55f2e8120150 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f48ec02a9e0 tx=0x7f48ec005b50 comp rx=0 tx=0).stop 2024-04-25T01:36:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.140+0000 7f48fa781740 1 -- 172.21.15.28:0/1055744927 shutdown_connections 2024-04-25T01:36:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.140+0000 7f48fa781740 1 --2- 172.21.15.28:0/1055744927 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2e8123a30 0x55f2e8121030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.140+0000 7f48fa781740 1 --2- 172.21.15.28:0/1055744927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8114e30 0x55f2e8120150 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.140+0000 7f48fa781740 1 --2- 172.21.15.28:0/1055744927 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8114510 0x55f2e812f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.140+0000 7f48fa781740 1 -- 172.21.15.28:0/1055744927 >> 172.21.15.28:0/1055744927 conn(0x55f2e8110950 msgr2=0x55f2e8120920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.140+0000 7f48fa781740 1 -- 172.21.15.28:0/1055744927 shutdown_connections 2024-04-25T01:36:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.140+0000 7f48fa781740 1 -- 172.21.15.28:0/1055744927 wait complete. 2024-04-25T01:36:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.141+0000 7f48fa781740 1 -- start start 2024-04-25T01:36:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.141+0000 7f48fa781740 1 Processor -- start 2024-04-25T01:36:58.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.141+0000 7f48fa781740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2e8114510 0x55f2e812c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.141+0000 7f48fa781740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8114e30 0x55f2e811d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.141+0000 7f48fa781740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8123a30 0x55f2e811d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.141+0000 7f48fa781740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48f01addd0 con 0x55f2e8123a30 2024-04-25T01:36:58.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.141+0000 7f48fa781740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f48f01ad700 con 0x55f2e8114e30 2024-04-25T01:36:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.141+0000 7f48fa781740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f48f01aa940 con 0x55f2e8114510 2024-04-25T01:36:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.141+0000 7f48fa781740 10 client.0 _ll_register_callbacks cb 0x7f48e000ee10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.141+0000 7f48f4f38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8123a30 0x55f2e811d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.141+0000 7f48f4f38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8123a30 0x55f2e811d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49616/0 (socket says 172.21.15.28:49616) 2024-04-25T01:36:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.141+0000 7f48f4f38640 1 -- 172.21.15.28:0/2026323105 learned_addr learned my addr 172.21.15.28:0/2026323105 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.141+0000 7f48f5739640 1 --2- 172.21.15.28:0/2026323105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8114e30 0x55f2e811d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.141+0000 7f48f4f38640 1 --2- 172.21.15.28:0/2026323105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2e8114510 0x55f2e812c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.142+0000 7f48f4f38640 1 -- 172.21.15.28:0/2026323105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2e8114510 msgr2=0x55f2e812c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.142+0000 7f48f4f38640 1 --2- 172.21.15.28:0/2026323105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2e8114510 0x55f2e812c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.142+0000 7f48f4f38640 1 -- 172.21.15.28:0/2026323105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8114e30 msgr2=0x55f2e811d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.142+0000 7f48f4f38640 1 --2- 172.21.15.28:0/2026323105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8114e30 0x55f2e811d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.142+0000 7f48f4f38640 1 -- 172.21.15.28:0/2026323105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2e811e1b0 con 0x55f2e8123a30 2024-04-25T01:36:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.142+0000 7f48f4f38640 1 --2- 172.21.15.28:0/2026323105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8123a30 0x55f2e811d980 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f48e0011b10 tx=0x7f48e0011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a903106f6c62484 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.142+0000 7f48d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.142+0000 7f48d8ff9640 1 -- 172.21.15.28:0/2026323105 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e0022030 con 0x55f2e8123a30 2024-04-25T01:36:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.142+0000 7f48d8ff9640 1 -- 172.21.15.28:0/2026323105 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f48e0004620 con 0x55f2e8123a30 2024-04-25T01:36:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.142+0000 7f48fa781740 1 -- 172.21.15.28:0/2026323105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48ec049070 con 0x55f2e8123a30 2024-04-25T01:36:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.142+0000 7f48d8ff9640 1 -- 172.21.15.28:0/2026323105 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f48e0004820 con 0x55f2e8123a30 2024-04-25T01:36:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.142+0000 7f48fa781740 10 client.16848 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.142+0000 7f48fa781740 1 -- 172.21.15.28:0/2026323105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f2e819a110 con 0x55f2e8123a30 2024-04-25T01:36:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.142+0000 7f48d8ff9640 1 -- 172.21.15.28:0/2026323105 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48ec049070 con 0x55f2e8123a30 2024-04-25T01:36:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.142+0000 7f48d8ff9640 1 -- 172.21.15.28:0/2026323105 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f48e000bbc0 con 0x55f2e8123a30 2024-04-25T01:36:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.142+0000 7f48fa781740 10 client.16848 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.142+0000 7f48fa781740 10 client.16848 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.142+0000 7f48fa781740 20 client.16848 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.142+0000 7f48fa781740 1 --2- 172.21.15.28:0/2026323105 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f2e819a770 0x55f2e819cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.143+0000 7f48fa781740 4 client.16848 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:36:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.143+0000 7f48fa781740 1 -- 172.21.15.28:0/2026323105 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f2e8080f40 con 0x55f2e819a770 2024-04-25T01:36:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.143+0000 7f48f5739640 1 --2- 172.21.15.28:0/2026323105 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f2e819a770 0x55f2e819cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.143+0000 7f48f5739640 1 --2- 172.21.15.28:0/2026323105 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f2e819a770 0x55f2e819cb60 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cba54a8072eaac06 server_cookie=6d5b68b16300d03a in_seq=0 out_seq=0 2024-04-25T01:36:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.143+0000 7f48d8ff9640 10 client.16848 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.143+0000 7f48d8ff9640 1 -- 172.21.15.28:0/2026323105 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f2e8080f40 con 0x55f2e819a770 2024-04-25T01:36:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.143+0000 7f48d8ff9640 10 client.16848 handle_command_reply: tid=0 2024-04-25T01:36:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.143+0000 7f48fa781740 1 client.16848 shutdown 2024-04-25T01:36:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.143+0000 7f48fa781740 1 -- 172.21.15.28:0/2026323105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8123a30 msgr2=0x55f2e811d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.143+0000 7f48fa781740 1 --2- 172.21.15.28:0/2026323105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8123a30 0x55f2e811d980 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f48e0011b10 tx=0x7f48e0011b40 comp rx=0 tx=0).stop 2024-04-25T01:36:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.143+0000 7f48fa781740 1 -- 172.21.15.28:0/2026323105 shutdown_connections 2024-04-25T01:36:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.143+0000 7f48fa781740 1 --2- 172.21.15.28:0/2026323105 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55f2e819a770 0x55f2e819cb60 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.143+0000 7f48fa781740 1 --2- 172.21.15.28:0/2026323105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2e8123a30 0x55f2e811d980 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.143+0000 7f48fa781740 1 --2- 172.21.15.28:0/2026323105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f2e8114e30 0x55f2e811d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.143+0000 7f48d8ff9640 0 client.16848 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.143+0000 7f48fa781740 1 --2- 172.21.15.28:0/2026323105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f2e8114510 0x55f2e812c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.143+0000 7f48fa781740 1 -- 172.21.15.28:0/2026323105 >> 172.21.15.28:0/2026323105 conn(0x55f2e8110950 msgr2=0x55f2e81166e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.144+0000 7f48fa781740 1 -- 172.21.15.28:0/2026323105 shutdown_connections 2024-04-25T01:36:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.144+0000 7f48fa781740 1 -- 172.21.15.28:0/2026323105 wait complete. 2024-04-25T01:36:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.144+0000 7f48fa781740 20 client.16848 trim_cache size 0 max 16384 2024-04-25T01:36:58.126 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:36:58.126 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:36:58.126 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:36:58.126 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:36:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.144+0000 7f48f799d640 1 -- 172.21.15.28:0/254726681 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f48dc0a2d80 msgr2=0x7f48dc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.144+0000 7f48f799d640 1 --2- 172.21.15.28:0/254726681 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f48dc0a2d80 0x7f48dc0a5240 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f48e80cce90 tx=0x7f48e806ec20 comp rx=0 tx=0).stop 2024-04-25T01:36:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.144+0000 7f48f799d640 1 -- 172.21.15.28:0/254726681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f0107720 msgr2=0x7f48f019c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.144+0000 7f48f799d640 1 --2- 172.21.15.28:0/254726681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f0107720 0x7f48f019c900 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f48e800eaa0 tx=0x7f48e800ef60 comp rx=0 tx=0).stop 2024-04-25T01:36:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.144+0000 7f48f799d640 1 -- 172.21.15.28:0/254726681 shutdown_connections 2024-04-25T01:36:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.144+0000 7f48f799d640 1 --2- 172.21.15.28:0/254726681 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f48dc0a2d80 0x7f48dc0a5240 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.144+0000 7f48f799d640 1 --2- 172.21.15.28:0/254726681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f0107720 0x7f48f019c900 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.144+0000 7f48f799d640 1 --2- 172.21.15.28:0/254726681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f48f01020d0 0x7f48f0198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.144+0000 7f48f799d640 1 --2- 172.21.15.28:0/254726681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f48f0101710 0x7f48f0198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.144+0000 7f48f799d640 1 -- 172.21.15.28:0/254726681 >> 172.21.15.28:0/254726681 conn(0x7f48f00fd180 msgr2=0x7f48f00fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.144+0000 7f48f799d640 1 -- 172.21.15.28:0/254726681 shutdown_connections 2024-04-25T01:36:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.144+0000 7f48f799d640 1 -- 172.21.15.28:0/254726681 wait complete. 2024-04-25T01:36:58.132 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:36:58.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:36:58.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.258+0000 7f9a53c93640 1 -- 172.21.15.28:0/3610012247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a4c100fe0 msgr2=0x7f9a4c10d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.258+0000 7f9a53c93640 1 --2- 172.21.15.28:0/3610012247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a4c100fe0 0x7f9a4c10d4f0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f9a44009cd0 tx=0x7f9a4402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.258+0000 7f9a53c93640 1 -- 172.21.15.28:0/3610012247 shutdown_connections 2024-04-25T01:36:58.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.258+0000 7f9a53c93640 1 --2- 172.21.15.28:0/3610012247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a4c100fe0 0x7f9a4c10d4f0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.258+0000 7f9a53c93640 1 --2- 172.21.15.28:0/3610012247 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a4c100620 0x7f9a4c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.258+0000 7f9a53c93640 1 --2- 172.21.15.28:0/3610012247 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a4c106630 0x7f9a4c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.258+0000 7f9a53c93640 1 -- 172.21.15.28:0/3610012247 >> 172.21.15.28:0/3610012247 conn(0x7f9a4c077740 msgr2=0x7f9a4c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.258+0000 7f9a53c93640 1 -- 172.21.15.28:0/3610012247 shutdown_connections 2024-04-25T01:36:58.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.258+0000 7f9a53c93640 1 -- 172.21.15.28:0/3610012247 wait complete. 2024-04-25T01:36:58.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.258+0000 7f9a53c93640 1 Processor -- start 2024-04-25T01:36:58.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.258+0000 7f9a53c93640 1 -- start start 2024-04-25T01:36:58.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a53c93640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a4c100620 0x7f9a4c198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a53c93640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a4c100fe0 0x7f9a4c199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a53c93640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a4c106630 0x7f9a4c1a3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a53c93640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9a4c110a50 con 0x7f9a4c100fe0 2024-04-25T01:36:58.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a53c93640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9a4c1108d0 con 0x7f9a4c100620 2024-04-25T01:36:58.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a53c93640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9a4c110bd0 con 0x7f9a4c106630 2024-04-25T01:36:58.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a5122e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a4c100fe0 0x7f9a4c199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a5122e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a4c100fe0 0x7f9a4c199350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49640/0 (socket says 172.21.15.28:49640) 2024-04-25T01:36:58.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a5122e640 1 -- 172.21.15.28:0/3625749206 learned_addr learned my addr 172.21.15.28:0/3625749206 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a52230640 1 --2- 172.21.15.28:0/3625749206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a4c106630 0x7f9a4c1a3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a51a2f640 1 --2- 172.21.15.28:0/3625749206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a4c100620 0x7f9a4c198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a5122e640 1 -- 172.21.15.28:0/3625749206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a4c106630 msgr2=0x7f9a4c1a3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a5122e640 1 --2- 172.21.15.28:0/3625749206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a4c106630 0x7f9a4c1a3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a5122e640 1 -- 172.21.15.28:0/3625749206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a4c100620 msgr2=0x7f9a4c198e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a5122e640 1 --2- 172.21.15.28:0/3625749206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a4c100620 0x7f9a4c198e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a5122e640 1 -- 172.21.15.28:0/3625749206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a4c1a43d0 con 0x7f9a4c100fe0 2024-04-25T01:36:58.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a52230640 1 --2- 172.21.15.28:0/3625749206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a4c106630 0x7f9a4c1a3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:58.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a51a2f640 1 --2- 172.21.15.28:0/3625749206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a4c100620 0x7f9a4c198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:58.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a5122e640 1 --2- 172.21.15.28:0/3625749206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a4c100fe0 0x7f9a4c199350 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c00d950 tx=0x7f9a3c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a42ffd640 1 -- 172.21.15.28:0/3625749206 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c017070 con 0x7f9a4c100fe0 2024-04-25T01:36:58.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a53c93640 1 -- 172.21.15.28:0/3625749206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a4c1a45a0 con 0x7f9a4c100fe0 2024-04-25T01:36:58.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a42ffd640 1 -- 172.21.15.28:0/3625749206 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a3c009c30 con 0x7f9a4c100fe0 2024-04-25T01:36:58.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a42ffd640 1 -- 172.21.15.28:0/3625749206 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c00f6b0 con 0x7f9a4c100fe0 2024-04-25T01:36:58.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.259+0000 7f9a53c93640 1 -- 172.21.15.28:0/3625749206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a4c1a4ac0 con 0x7f9a4c100fe0 2024-04-25T01:36:58.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.261+0000 7f9a53c93640 1 -- 172.21.15.28:0/3625749206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a4c068560 con 0x7f9a4c100fe0 2024-04-25T01:36:58.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.261+0000 7f9a42ffd640 1 -- 172.21.15.28:0/3625749206 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9a3c0109d0 con 0x7f9a4c100fe0 2024-04-25T01:36:58.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.262+0000 7f9a42ffd640 1 --2- 172.21.15.28:0/3625749206 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9a380a2d00 0x7f9a380a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.262+0000 7f9a51a2f640 1 --2- 172.21.15.28:0/3625749206 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9a380a2d00 0x7f9a380a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.262+0000 7f9a42ffd640 1 -- 172.21.15.28:0/3625749206 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f9a3c0cf120 con 0x7f9a4c100fe0 2024-04-25T01:36:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.263+0000 7f9a51a2f640 1 --2- 172.21.15.28:0/3625749206 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9a380a2d00 0x7f9a380a51c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9a48005d90 tx=0x7f9a48005d20 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.267+0000 7f9a42ffd640 1 -- 172.21.15.28:0/3625749206 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9a3c09ba00 con 0x7f9a4c100fe0 2024-04-25T01:36:58.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:58 smithi130 ceph-mon[32702]: pgmap v121: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:36:58.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:58 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:36:58.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:58 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:58.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:58 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:36:58.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:58 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:58.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:58 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:36:58.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.493+0000 7f9a53c93640 1 -- 172.21.15.28:0/3625749206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a4c108bb0 con 0x7f9a4c100fe0 2024-04-25T01:36:58.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.494+0000 7f9a42ffd640 1 -- 172.21.15.28:0/3625749206 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7f9a3c009de0 con 0x7f9a4c100fe0 2024-04-25T01:36:58.470 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:36:58.470 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:36:58.471 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:36:58.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.496+0000 7f9a53c93640 1 -- 172.21.15.28:0/3625749206 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9a380a2d00 msgr2=0x7f9a380a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.496+0000 7f9a53c93640 1 --2- 172.21.15.28:0/3625749206 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9a380a2d00 0x7f9a380a51c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9a48005d90 tx=0x7f9a48005d20 comp rx=0 tx=0).stop 2024-04-25T01:36:58.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.496+0000 7f9a53c93640 1 -- 172.21.15.28:0/3625749206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a4c100fe0 msgr2=0x7f9a4c199350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.496+0000 7f9a53c93640 1 --2- 172.21.15.28:0/3625749206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a4c100fe0 0x7f9a4c199350 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c00d950 tx=0x7f9a3c00de10 comp rx=0 tx=0).stop 2024-04-25T01:36:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.496+0000 7f9a53c93640 1 -- 172.21.15.28:0/3625749206 shutdown_connections 2024-04-25T01:36:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.496+0000 7f9a53c93640 1 --2- 172.21.15.28:0/3625749206 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9a380a2d00 0x7f9a380a51c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.496+0000 7f9a53c93640 1 --2- 172.21.15.28:0/3625749206 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9a4c106630 0x7f9a4c1a3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.496+0000 7f9a53c93640 1 --2- 172.21.15.28:0/3625749206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a4c100fe0 0x7f9a4c199350 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.496+0000 7f9a53c93640 1 --2- 172.21.15.28:0/3625749206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a4c100620 0x7f9a4c198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.496+0000 7f9a53c93640 1 -- 172.21.15.28:0/3625749206 >> 172.21.15.28:0/3625749206 conn(0x7f9a4c077740 msgr2=0x7f9a4c102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.496+0000 7f9a53c93640 1 -- 172.21.15.28:0/3625749206 shutdown_connections 2024-04-25T01:36:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.496+0000 7f9a53c93640 1 -- 172.21.15.28:0/3625749206 wait complete. 2024-04-25T01:36:58.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:58 smithi032 ceph-mon[32461]: pgmap v121: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:36:58.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:58 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:36:58.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:58 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:58.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:58 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:36:58.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:58 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:58.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:58 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:36:58.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:58 smithi028 ceph-mon[29390]: pgmap v121: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:36:58.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:58 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:36:58.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:58 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:58.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:58 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:36:58.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:58 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:36:58.483 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:58 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:36:58.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:36:58.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.612+0000 7fda85d6a640 1 -- 172.21.15.28:0/1995733395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda80102840 msgr2=0x7fda800ffc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.612+0000 7fda85d6a640 1 --2- 172.21.15.28:0/1995733395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda80102840 0x7fda800ffc40 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fda70009900 tx=0x7fda7002a410 comp rx=0 tx=0).stop 2024-04-25T01:36:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.612+0000 7fda85d6a640 1 -- 172.21.15.28:0/1995733395 shutdown_connections 2024-04-25T01:36:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.612+0000 7fda85d6a640 1 --2- 172.21.15.28:0/1995733395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fda8010e3c0 0x7fda801107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.612+0000 7fda85d6a640 1 --2- 172.21.15.28:0/1995733395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda80100180 0x7fda801005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.612+0000 7fda85d6a640 1 --2- 172.21.15.28:0/1995733395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda80102840 0x7fda800ffc40 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.612+0000 7fda85d6a640 1 -- 172.21.15.28:0/1995733395 >> 172.21.15.28:0/1995733395 conn(0x7fda80076c00 msgr2=0x7fda800fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.612+0000 7fda85d6a640 1 -- 172.21.15.28:0/1995733395 shutdown_connections 2024-04-25T01:36:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.612+0000 7fda85d6a640 1 -- 172.21.15.28:0/1995733395 wait complete. 2024-04-25T01:36:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.613+0000 7fda85d6a640 1 Processor -- start 2024-04-25T01:36:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.613+0000 7fda85d6a640 1 -- start start 2024-04-25T01:36:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.613+0000 7fda85d6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda80100180 0x7fda8019a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.613+0000 7fda85d6a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda80102840 0x7fda8019a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.613+0000 7fda85d6a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fda8010e3c0 0x7fda8019eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.613+0000 7fda85d6a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda80113f20 con 0x7fda80100180 2024-04-25T01:36:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.613+0000 7fda85d6a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fda80113da0 con 0x7fda80102840 2024-04-25T01:36:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.613+0000 7fda85d6a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fda801140a0 con 0x7fda8010e3c0 2024-04-25T01:36:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.613+0000 7fda7f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda80100180 0x7fda8019a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.613+0000 7fda7f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda80100180 0x7fda8019a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49656/0 (socket says 172.21.15.28:49656) 2024-04-25T01:36:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.613+0000 7fda7f7fe640 1 -- 172.21.15.28:0/882037209 learned_addr learned my addr 172.21.15.28:0/882037209 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.613+0000 7fda7ffff640 1 --2- 172.21.15.28:0/882037209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fda8010e3c0 0x7fda8019eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.613+0000 7fda7effd640 1 --2- 172.21.15.28:0/882037209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda80102840 0x7fda8019a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.614+0000 7fda7f7fe640 1 -- 172.21.15.28:0/882037209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fda8010e3c0 msgr2=0x7fda8019eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.614+0000 7fda7f7fe640 1 --2- 172.21.15.28:0/882037209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fda8010e3c0 0x7fda8019eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.614+0000 7fda7f7fe640 1 -- 172.21.15.28:0/882037209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda80102840 msgr2=0x7fda8019a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.614+0000 7fda7f7fe640 1 --2- 172.21.15.28:0/882037209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda80102840 0x7fda8019a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.614+0000 7fda7f7fe640 1 -- 172.21.15.28:0/882037209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8019f3d0 con 0x7fda80100180 2024-04-25T01:36:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.614+0000 7fda7f7fe640 1 --2- 172.21.15.28:0/882037209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda80100180 0x7fda8019a3e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fda70006010 tx=0x7fda70003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.614+0000 7fda7cff9640 1 -- 172.21.15.28:0/882037209 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7003f070 con 0x7fda80100180 2024-04-25T01:36:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.614+0000 7fda7cff9640 1 -- 172.21.15.28:0/882037209 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda70031c30 con 0x7fda80100180 2024-04-25T01:36:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.614+0000 7fda85d6a640 1 -- 172.21.15.28:0/882037209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda8019f660 con 0x7fda80100180 2024-04-25T01:36:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.614+0000 7fda7cff9640 1 -- 172.21.15.28:0/882037209 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda70035030 con 0x7fda80100180 2024-04-25T01:36:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.614+0000 7fda85d6a640 1 -- 172.21.15.28:0/882037209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda801a6f50 con 0x7fda80100180 2024-04-25T01:36:58.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.616+0000 7fda7cff9640 1 -- 172.21.15.28:0/882037209 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fda70042070 con 0x7fda80100180 2024-04-25T01:36:58.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.617+0000 7fda88a71740 1 Processor -- start 2024-04-25T01:36:58.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.617+0000 7fda88a71740 1 -- start start 2024-04-25T01:36:58.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.617+0000 7fda88a71740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371580 0x55948e371960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.617+0000 7fda88a71740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371ea0 0x55948e37c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.617+0000 7fda7cff9640 1 --2- 172.21.15.28:0/882037209 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fda680a2d00 0x7fda680a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.617+0000 7fda88a71740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948e37c570 0x55948e37e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.617+0000 7fda88a71740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55948e2d6670 con 0x55948e371580 2024-04-25T01:36:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.617+0000 7fda88a71740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55948e27e970 con 0x55948e371ea0 2024-04-25T01:36:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.617+0000 7fda88a71740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55948e2d2630 con 0x55948e37c570 2024-04-25T01:36:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.617+0000 7fda7ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371580 0x55948e371960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.617+0000 7fda7ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371580 0x55948e371960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49658/0 (socket says 172.21.15.28:49658) 2024-04-25T01:36:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.617+0000 7fda7effd640 1 --2- 172.21.15.28:0/882037209 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fda680a2d00 0x7fda680a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.617+0000 7fda7ffff640 1 -- 172.21.15.28:0/3805312512 learned_addr learned my addr 172.21.15.28:0/3805312512 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.617+0000 7fda7f7fe640 1 --2- 172.21.15.28:0/3805312512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371ea0 0x55948e37c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.617+0000 7fda7cff9640 1 -- 172.21.15.28:0/882037209 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fda700effa0 con 0x7fda80100180 2024-04-25T01:36:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.617+0000 7fda7ffff640 1 -- 172.21.15.28:0/3805312512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948e37c570 msgr2=0x55948e37e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.617+0000 7fda7ffff640 1 --2- 172.21.15.28:0/3805312512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948e37c570 0x55948e37e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.617+0000 7fda7ffff640 1 -- 172.21.15.28:0/3805312512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371ea0 msgr2=0x55948e37c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.617+0000 7fda7ffff640 1 --2- 172.21.15.28:0/3805312512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371ea0 0x55948e37c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.618+0000 7fda7ffff640 1 -- 172.21.15.28:0/3805312512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55948e2a2400 con 0x55948e371580 2024-04-25T01:36:58.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.618+0000 7fda7effd640 1 --2- 172.21.15.28:0/882037209 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fda680a2d00 0x7fda680a51c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fda800fe9a0 tx=0x7fda7400b390 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.618+0000 7fda7ffff640 1 --2- 172.21.15.28:0/3805312512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371580 0x55948e371960 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fda78009fd0 tx=0x7fda78012870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8c752d6c9057669 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.618+0000 7fda65ffb640 1 -- 172.21.15.28:0/3805312512 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda780136c0 con 0x55948e371580 2024-04-25T01:36:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.618+0000 7fda65ffb640 1 -- 172.21.15.28:0/3805312512 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda78013d40 con 0x55948e371580 2024-04-25T01:36:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.618+0000 7fda65ffb640 1 -- 172.21.15.28:0/3805312512 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda78010e40 con 0x55948e371580 2024-04-25T01:36:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.619+0000 7fda88a71740 1 -- 172.21.15.28:0/3805312512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371580 msgr2=0x55948e371960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.619+0000 7fda88a71740 1 --2- 172.21.15.28:0/3805312512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371580 0x55948e371960 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fda78009fd0 tx=0x7fda78012870 comp rx=0 tx=0).stop 2024-04-25T01:36:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.619+0000 7fda88a71740 1 -- 172.21.15.28:0/3805312512 shutdown_connections 2024-04-25T01:36:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.619+0000 7fda88a71740 1 --2- 172.21.15.28:0/3805312512 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948e37c570 0x55948e37e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.619+0000 7fda88a71740 1 --2- 172.21.15.28:0/3805312512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371ea0 0x55948e37c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.619+0000 7fda88a71740 1 --2- 172.21.15.28:0/3805312512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371580 0x55948e371960 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.619+0000 7fda88a71740 1 -- 172.21.15.28:0/3805312512 >> 172.21.15.28:0/3805312512 conn(0x55948e36d8a0 msgr2=0x55948e36dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.619+0000 7fda88a71740 1 -- 172.21.15.28:0/3805312512 shutdown_connections 2024-04-25T01:36:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.619+0000 7fda88a71740 1 -- 172.21.15.28:0/3805312512 wait complete. 2024-04-25T01:36:58.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.619+0000 7fda88a71740 1 -- start start 2024-04-25T01:36:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.619+0000 7fda88a71740 1 Processor -- start 2024-04-25T01:36:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.619+0000 7fda88a71740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948e371580 0x55948e37bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.619+0000 7fda88a71740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371ea0 0x55948e3791d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.619+0000 7fda88a71740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e37c570 0x55948e379730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.619+0000 7fda88a71740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55948e37f360 con 0x55948e37c570 2024-04-25T01:36:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.619+0000 7fda88a71740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55948e292050 con 0x55948e371ea0 2024-04-25T01:36:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.619+0000 7fda88a71740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55948e37f4e0 con 0x55948e371580 2024-04-25T01:36:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.619+0000 7fda88a71740 10 client.0 _ll_register_callbacks cb 0x55948dfe9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.620+0000 7fda7ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e37c570 0x55948e379730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.620+0000 7fda7ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e37c570 0x55948e379730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49670/0 (socket says 172.21.15.28:49670) 2024-04-25T01:36:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.620+0000 7fda7ffff640 1 -- 172.21.15.28:0/844635369 learned_addr learned my addr 172.21.15.28:0/844635369 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.620+0000 7fda7f7fe640 1 --2- 172.21.15.28:0/844635369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948e371580 0x55948e37bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.620+0000 7fda7effd640 1 --2- 172.21.15.28:0/844635369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371ea0 0x55948e3791d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.620+0000 7fda7ffff640 1 -- 172.21.15.28:0/844635369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948e371580 msgr2=0x55948e37bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.620+0000 7fda7ffff640 1 --2- 172.21.15.28:0/844635369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948e371580 0x55948e37bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.620+0000 7fda7ffff640 1 -- 172.21.15.28:0/844635369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371ea0 msgr2=0x55948e3791d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.620+0000 7fda7ffff640 1 --2- 172.21.15.28:0/844635369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371ea0 0x55948e3791d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.620+0000 7fda7ffff640 1 -- 172.21.15.28:0/844635369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55948e263dc0 con 0x55948e37c570 2024-04-25T01:36:58.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.620+0000 7fda7ffff640 1 --2- 172.21.15.28:0/844635369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e37c570 0x55948e379730 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fda78009db0 tx=0x7fda78009de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8064ba70a8ec4c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.620+0000 7fda5effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:58.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.620+0000 7fda5effd640 1 -- 172.21.15.28:0/844635369 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7802b060 con 0x55948e37c570 2024-04-25T01:36:58.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.620+0000 7fda5effd640 1 -- 172.21.15.28:0/844635369 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fda780103d0 con 0x55948e37c570 2024-04-25T01:36:58.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.620+0000 7fda88a71740 1 -- 172.21.15.28:0/844635369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda78013d40 con 0x55948e37c570 2024-04-25T01:36:58.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.620+0000 7fda5effd640 1 -- 172.21.15.28:0/844635369 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fda78032050 con 0x55948e37c570 2024-04-25T01:36:58.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.620+0000 7fda88a71740 10 client.16866 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:58.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.620+0000 7fda88a71740 1 -- 172.21.15.28:0/844635369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55948dfea3f0 con 0x55948e37c570 2024-04-25T01:36:58.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.621+0000 7fda5effd640 1 -- 172.21.15.28:0/844635369 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda78013d40 con 0x55948e37c570 2024-04-25T01:36:58.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.621+0000 7fda5effd640 1 -- 172.21.15.28:0/844635369 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fda7800f040 con 0x55948e37c570 2024-04-25T01:36:58.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.621+0000 7fda88a71740 10 client.16866 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:58.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.621+0000 7fda88a71740 10 client.16866 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:36:58.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.621+0000 7fda88a71740 20 client.16866 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:58.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.621+0000 7fda88a71740 1 --2- 172.21.15.28:0/844635369 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55948e38c560 0x55948e38e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.621+0000 7fda88a71740 4 client.16866 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:58.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.621+0000 7fda88a71740 1 -- 172.21.15.28:0/844635369 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55948e2e3bb0 con 0x55948e38c560 2024-04-25T01:36:58.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.621+0000 7fda7f7fe640 1 --2- 172.21.15.28:0/844635369 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55948e38c560 0x55948e38e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.622+0000 7fda5effd640 10 client.16866 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:36:58.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.622+0000 7fda7f7fe640 1 --2- 172.21.15.28:0/844635369 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55948e38c560 0x55948e38e950 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bcf3eff1724ebf51 server_cookie=10ab6c6c8a0ab99 in_seq=0 out_seq=0 2024-04-25T01:36:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.622+0000 7fda5effd640 1 -- 172.21.15.28:0/844635369 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55948e2e3bb0 con 0x55948e38c560 2024-04-25T01:36:58.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.622+0000 7fda5effd640 10 client.16866 handle_command_reply: tid=0 2024-04-25T01:36:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.622+0000 7fda88a71740 1 client.16866 shutdown 2024-04-25T01:36:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.623+0000 7fda88a71740 1 -- 172.21.15.28:0/844635369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e37c570 msgr2=0x55948e379730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.623+0000 7fda88a71740 1 --2- 172.21.15.28:0/844635369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e37c570 0x55948e379730 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fda78009db0 tx=0x7fda78009de0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.623+0000 7fda88a71740 1 -- 172.21.15.28:0/844635369 shutdown_connections 2024-04-25T01:36:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.623+0000 7fda88a71740 1 --2- 172.21.15.28:0/844635369 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55948e38c560 0x55948e38e950 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.623+0000 7fda88a71740 1 --2- 172.21.15.28:0/844635369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e37c570 0x55948e379730 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.623+0000 7fda88a71740 1 --2- 172.21.15.28:0/844635369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371ea0 0x55948e3791d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.623+0000 7fda5effd640 0 client.16866 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:36:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.623+0000 7fda88a71740 1 --2- 172.21.15.28:0/844635369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948e371580 0x55948e37bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.623+0000 7fda88a71740 1 -- 172.21.15.28:0/844635369 >> 172.21.15.28:0/844635369 conn(0x55948e36d8a0 msgr2=0x55948e36cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.623+0000 7fda88a71740 1 -- 172.21.15.28:0/844635369 shutdown_connections 2024-04-25T01:36:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.623+0000 7fda88a71740 1 -- 172.21.15.28:0/844635369 wait complete. 2024-04-25T01:36:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.623+0000 7fda88a71740 20 client.16866 trim_cache size 0 max 16384 2024-04-25T01:36:58.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.636+0000 7fda88a71740 1 Processor -- start 2024-04-25T01:36:58.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.636+0000 7fda88a71740 1 -- start start 2024-04-25T01:36:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.636+0000 7fda88a71740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371580 0x55948e388220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.636+0000 7fda88a71740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371ea0 0x55948e37ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.636+0000 7fda88a71740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948e37c570 0x55948e393f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.636+0000 7fda88a71740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda801b0c40 con 0x55948e371580 2024-04-25T01:36:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.636+0000 7fda88a71740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fda801b2780 con 0x55948e371ea0 2024-04-25T01:36:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.636+0000 7fda88a71740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fda801b1310 con 0x55948e37c570 2024-04-25T01:36:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.636+0000 7fda7f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371580 0x55948e388220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.637+0000 7fda7f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371580 0x55948e388220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49672/0 (socket says 172.21.15.28:49672) 2024-04-25T01:36:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.637+0000 7fda7f7fe640 1 -- 172.21.15.28:0/3906073548 learned_addr learned my addr 172.21.15.28:0/3906073548 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.637+0000 7fda7effd640 1 --2- 172.21.15.28:0/3906073548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371ea0 0x55948e37ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.637+0000 7fda7ffff640 1 --2- 172.21.15.28:0/3906073548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948e37c570 0x55948e393f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.637+0000 7fda7f7fe640 1 -- 172.21.15.28:0/3906073548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948e37c570 msgr2=0x55948e393f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.637+0000 7fda7f7fe640 1 --2- 172.21.15.28:0/3906073548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948e37c570 0x55948e393f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.637+0000 7fda7f7fe640 1 -- 172.21.15.28:0/3906073548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371ea0 msgr2=0x55948e37ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.637+0000 7fda7f7fe640 1 --2- 172.21.15.28:0/3906073548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371ea0 0x55948e37ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.637+0000 7fda7f7fe640 1 -- 172.21.15.28:0/3906073548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55948e374500 con 0x55948e371580 2024-04-25T01:36:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.637+0000 7fda7effd640 1 --2- 172.21.15.28:0/3906073548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371ea0 0x55948e37ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.637+0000 7fda7f7fe640 1 --2- 172.21.15.28:0/3906073548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371580 0x55948e388220 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fda700bbed0 tx=0x7fda70049dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0b01aa7d0b88dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.637+0000 7fda657fa640 1 -- 172.21.15.28:0/3906073548 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda70049040 con 0x55948e371580 2024-04-25T01:36:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.637+0000 7fda657fa640 1 -- 172.21.15.28:0/3906073548 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda700bc1d0 con 0x55948e371580 2024-04-25T01:36:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.637+0000 7fda657fa640 1 -- 172.21.15.28:0/3906073548 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda700b8840 con 0x55948e371580 2024-04-25T01:36:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.638+0000 7fda88a71740 1 -- 172.21.15.28:0/3906073548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371580 msgr2=0x55948e388220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.638+0000 7fda88a71740 1 --2- 172.21.15.28:0/3906073548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371580 0x55948e388220 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fda700bbed0 tx=0x7fda70049dd0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.638+0000 7fda88a71740 1 -- 172.21.15.28:0/3906073548 shutdown_connections 2024-04-25T01:36:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.638+0000 7fda88a71740 1 --2- 172.21.15.28:0/3906073548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948e37c570 0x55948e393f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.638+0000 7fda88a71740 1 --2- 172.21.15.28:0/3906073548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371ea0 0x55948e37ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.638+0000 7fda88a71740 1 --2- 172.21.15.28:0/3906073548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371580 0x55948e388220 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.638+0000 7fda88a71740 1 -- 172.21.15.28:0/3906073548 >> 172.21.15.28:0/3906073548 conn(0x55948e36d8a0 msgr2=0x55948e378a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.638+0000 7fda88a71740 1 -- 172.21.15.28:0/3906073548 shutdown_connections 2024-04-25T01:36:58.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.638+0000 7fda88a71740 1 -- 172.21.15.28:0/3906073548 wait complete. 2024-04-25T01:36:58.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.638+0000 7fda88a71740 1 -- start start 2024-04-25T01:36:58.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.638+0000 7fda88a71740 1 Processor -- start 2024-04-25T01:36:58.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.638+0000 7fda88a71740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371580 0x55948e3848d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.638+0000 7fda88a71740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371ea0 0x55948e37fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.638+0000 7fda88a71740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948e37c570 0x55948e3801d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.638+0000 7fda88a71740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda801b20b0 con 0x55948e371ea0 2024-04-25T01:36:58.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.638+0000 7fda88a71740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fda801b19e0 con 0x55948e371580 2024-04-25T01:36:58.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.638+0000 7fda88a71740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fda801aec20 con 0x55948e37c570 2024-04-25T01:36:58.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.638+0000 7fda88a71740 10 client.0 _ll_register_callbacks cb 0x7fda700b7e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:58.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.639+0000 7fda7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371ea0 0x55948e37fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.639+0000 7fda7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371ea0 0x55948e37fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49676/0 (socket says 172.21.15.28:49676) 2024-04-25T01:36:58.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.639+0000 7fda7effd640 1 -- 172.21.15.28:0/3718615544 learned_addr learned my addr 172.21.15.28:0/3718615544 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.639+0000 7fda7ffff640 1 --2- 172.21.15.28:0/3718615544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948e37c570 0x55948e3801d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.639+0000 7fda7f7fe640 1 --2- 172.21.15.28:0/3718615544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371580 0x55948e3848d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.639+0000 7fda7effd640 1 -- 172.21.15.28:0/3718615544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948e37c570 msgr2=0x55948e3801d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.639+0000 7fda7effd640 1 --2- 172.21.15.28:0/3718615544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948e37c570 0x55948e3801d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.639+0000 7fda7effd640 1 -- 172.21.15.28:0/3718615544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371580 msgr2=0x55948e3848d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.639+0000 7fda7effd640 1 --2- 172.21.15.28:0/3718615544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371580 0x55948e3848d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.639+0000 7fda7effd640 1 -- 172.21.15.28:0/3718615544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55948e371cd0 con 0x55948e371ea0 2024-04-25T01:36:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.639+0000 7fda7f7fe640 1 --2- 172.21.15.28:0/3718615544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371580 0x55948e3848d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.639+0000 7fda7effd640 1 --2- 172.21.15.28:0/3718615544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371ea0 0x55948e37fc90 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x55948e378a30 tx=0x7fda74004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32c5e2b87038aa36 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.639+0000 7fda5effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.639+0000 7fda5effd640 1 -- 172.21.15.28:0/3718615544 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda74005220 con 0x55948e371ea0 2024-04-25T01:36:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.639+0000 7fda5effd640 1 -- 172.21.15.28:0/3718615544 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fda740053c0 con 0x55948e371ea0 2024-04-25T01:36:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.639+0000 7fda88a71740 1 -- 172.21.15.28:0/3718615544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda700bc1d0 con 0x55948e371ea0 2024-04-25T01:36:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.639+0000 7fda5effd640 1 -- 172.21.15.28:0/3718615544 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fda740055c0 con 0x55948e371ea0 2024-04-25T01:36:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.639+0000 7fda88a71740 10 client.16872 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.639+0000 7fda88a71740 1 -- 172.21.15.28:0/3718615544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55948e0b47d0 con 0x55948e371ea0 2024-04-25T01:36:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.640+0000 7fda5effd640 1 -- 172.21.15.28:0/3718615544 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda700bc1d0 con 0x55948e371ea0 2024-04-25T01:36:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.640+0000 7fda5effd640 1 -- 172.21.15.28:0/3718615544 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fda74027070 con 0x55948e371ea0 2024-04-25T01:36:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.640+0000 7fda88a71740 10 client.16872 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.640+0000 7fda88a71740 10 client.16872 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:36:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.640+0000 7fda88a71740 20 client.16872 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.640+0000 7fda88a71740 1 --2- 172.21.15.28:0/3718615544 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55948e3f2ae0 0x55948e3f4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.640+0000 7fda88a71740 4 client.16872 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.640+0000 7fda88a71740 1 -- 172.21.15.28:0/3718615544 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55948e2ddf00 con 0x55948e3f2ae0 2024-04-25T01:36:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.640+0000 7fda7f7fe640 1 --2- 172.21.15.28:0/3718615544 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55948e3f2ae0 0x55948e3f4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.641+0000 7fda5effd640 10 client.16872 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:36:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.641+0000 7fda7f7fe640 1 --2- 172.21.15.28:0/3718615544 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55948e3f2ae0 0x55948e3f4f00 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fc4941d0e9dece55 server_cookie=9ded2a79034c9f83 in_seq=0 out_seq=0 2024-04-25T01:36:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.641+0000 7fda5effd640 1 -- 172.21.15.28:0/3718615544 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55948e2ddf00 con 0x55948e3f2ae0 2024-04-25T01:36:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.641+0000 7fda5effd640 10 client.16872 handle_command_reply: tid=0 2024-04-25T01:36:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.641+0000 7fda88a71740 1 client.16872 shutdown 2024-04-25T01:36:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.641+0000 7fda88a71740 1 -- 172.21.15.28:0/3718615544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371ea0 msgr2=0x55948e37fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.641+0000 7fda88a71740 1 --2- 172.21.15.28:0/3718615544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371ea0 0x55948e37fc90 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x55948e378a30 tx=0x7fda74004730 comp rx=0 tx=0).stop 2024-04-25T01:36:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.641+0000 7fda88a71740 1 -- 172.21.15.28:0/3718615544 shutdown_connections 2024-04-25T01:36:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.641+0000 7fda88a71740 1 --2- 172.21.15.28:0/3718615544 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55948e3f2ae0 0x55948e3f4f00 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.641+0000 7fda88a71740 1 --2- 172.21.15.28:0/3718615544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55948e37c570 0x55948e3801d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.641+0000 7fda5effd640 0 client.16872 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:36:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.641+0000 7fda88a71740 1 --2- 172.21.15.28:0/3718615544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948e371ea0 0x55948e37fc90 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.641+0000 7fda88a71740 1 --2- 172.21.15.28:0/3718615544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55948e371580 0x55948e3848d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.641+0000 7fda88a71740 1 -- 172.21.15.28:0/3718615544 >> 172.21.15.28:0/3718615544 conn(0x55948e36d8a0 msgr2=0x55948e378a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.641+0000 7fda88a71740 1 -- 172.21.15.28:0/3718615544 shutdown_connections 2024-04-25T01:36:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.641+0000 7fda88a71740 1 -- 172.21.15.28:0/3718615544 wait complete. 2024-04-25T01:36:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.641+0000 7fda88a71740 20 client.16872 trim_cache size 0 max 16384 2024-04-25T01:36:58.625 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:36:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.642+0000 7fda85d6a640 1 -- 172.21.15.28:0/882037209 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fda680a2d00 msgr2=0x7fda680a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.642+0000 7fda85d6a640 1 --2- 172.21.15.28:0/882037209 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fda680a2d00 0x7fda680a51c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fda800fe9a0 tx=0x7fda7400b390 comp rx=0 tx=0).stop 2024-04-25T01:36:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.642+0000 7fda85d6a640 1 -- 172.21.15.28:0/882037209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda80100180 msgr2=0x7fda8019a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.642+0000 7fda85d6a640 1 --2- 172.21.15.28:0/882037209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda80100180 0x7fda8019a3e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fda70006010 tx=0x7fda70003c10 comp rx=0 tx=0).stop 2024-04-25T01:36:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.642+0000 7fda85d6a640 1 -- 172.21.15.28:0/882037209 shutdown_connections 2024-04-25T01:36:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.642+0000 7fda85d6a640 1 --2- 172.21.15.28:0/882037209 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fda680a2d00 0x7fda680a51c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.642+0000 7fda85d6a640 1 --2- 172.21.15.28:0/882037209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fda8010e3c0 0x7fda8019eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.642+0000 7fda85d6a640 1 --2- 172.21.15.28:0/882037209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda80102840 0x7fda8019a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.642+0000 7fda85d6a640 1 --2- 172.21.15.28:0/882037209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda80100180 0x7fda8019a3e0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.642+0000 7fda85d6a640 1 -- 172.21.15.28:0/882037209 >> 172.21.15.28:0/882037209 conn(0x7fda80076c00 msgr2=0x7fda8010f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.642+0000 7fda85d6a640 1 -- 172.21.15.28:0/882037209 shutdown_connections 2024-04-25T01:36:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.642+0000 7fda85d6a640 1 -- 172.21.15.28:0/882037209 wait complete. 2024-04-25T01:36:58.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:36:58.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.757+0000 7fc05d109640 1 -- 172.21.15.28:0/474393637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0581020d0 msgr2=0x7fc05810e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.757+0000 7fc05d109640 1 --2- 172.21.15.28:0/474393637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0581020d0 0x7fc05810e5e0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fc050009cd0 tx=0x7fc05002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.757+0000 7fc05d109640 1 -- 172.21.15.28:0/474393637 shutdown_connections 2024-04-25T01:36:58.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.757+0000 7fc05d109640 1 --2- 172.21.15.28:0/474393637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0581020d0 0x7fc05810e5e0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.757+0000 7fc05d109640 1 --2- 172.21.15.28:0/474393637 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc058101710 0x7fc058101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.757+0000 7fc05d109640 1 --2- 172.21.15.28:0/474393637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc058107720 0x7fc058107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.757+0000 7fc05d109640 1 -- 172.21.15.28:0/474393637 >> 172.21.15.28:0/474393637 conn(0x7fc0580fd180 msgr2=0x7fc0580ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.757+0000 7fc05d109640 1 -- 172.21.15.28:0/474393637 shutdown_connections 2024-04-25T01:36:58.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.757+0000 7fc05d109640 1 -- 172.21.15.28:0/474393637 wait complete. 2024-04-25T01:36:58.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.757+0000 7fc05d109640 1 Processor -- start 2024-04-25T01:36:58.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.758+0000 7fc05d109640 1 -- start start 2024-04-25T01:36:58.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.758+0000 7fc05d109640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058101710 0x7fc058198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.758+0000 7fc05d109640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc0581020d0 0x7fc058198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.758+0000 7fc05d109640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc058107720 0x7fc05819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.758+0000 7fc05d109640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc058111b30 con 0x7fc058101710 2024-04-25T01:36:58.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.758+0000 7fc05d109640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581119b0 con 0x7fc058107720 2024-04-25T01:36:58.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.758+0000 7fc05d109640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc058111cb0 con 0x7fc0581020d0 2024-04-25T01:36:58.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.758+0000 7fc056d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058101710 0x7fc058198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.758+0000 7fc056d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058101710 0x7fc058198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49692/0 (socket says 172.21.15.28:49692) 2024-04-25T01:36:58.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.758+0000 7fc056d9d640 1 -- 172.21.15.28:0/3934404179 learned_addr learned my addr 172.21.15.28:0/3934404179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.758+0000 7fc04e59c640 1 --2- 172.21.15.28:0/3934404179 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc0581020d0 0x7fc058198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.758+0000 7fc05759e640 1 --2- 172.21.15.28:0/3934404179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc058107720 0x7fc05819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.758+0000 7fc056d9d640 1 -- 172.21.15.28:0/3934404179 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc0581020d0 msgr2=0x7fc058198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.758+0000 7fc056d9d640 1 --2- 172.21.15.28:0/3934404179 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc0581020d0 0x7fc058198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.758+0000 7fc056d9d640 1 -- 172.21.15.28:0/3934404179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc058107720 msgr2=0x7fc05819c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.758+0000 7fc056d9d640 1 --2- 172.21.15.28:0/3934404179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc058107720 0x7fc05819c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.758+0000 7fc056d9d640 1 -- 172.21.15.28:0/3934404179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc05819d020 con 0x7fc058101710 2024-04-25T01:36:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.758+0000 7fc05759e640 1 --2- 172.21.15.28:0/3934404179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc058107720 0x7fc05819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.759+0000 7fc056d9d640 1 --2- 172.21.15.28:0/3934404179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058101710 0x7fc058198030 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fc04800c970 tx=0x7fc04800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.759+0000 7fc054d99640 1 -- 172.21.15.28:0/3934404179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc048007bb0 con 0x7fc058101710 2024-04-25T01:36:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.759+0000 7fc05d109640 1 -- 172.21.15.28:0/3934404179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc05819d310 con 0x7fc058101710 2024-04-25T01:36:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.759+0000 7fc054d99640 1 -- 172.21.15.28:0/3934404179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc048007d50 con 0x7fc058101710 2024-04-25T01:36:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.759+0000 7fc054d99640 1 -- 172.21.15.28:0/3934404179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc048018800 con 0x7fc058101710 2024-04-25T01:36:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.759+0000 7fc05d109640 1 -- 172.21.15.28:0/3934404179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0581a4ba0 con 0x7fc058101710 2024-04-25T01:36:58.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.762+0000 7fc05fe13740 1 Processor -- start 2024-04-25T01:36:58.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.762+0000 7fc054d99640 1 -- 172.21.15.28:0/3934404179 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc04800f450 con 0x7fc058101710 2024-04-25T01:36:58.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.762+0000 7fc05fe13740 1 -- start start 2024-04-25T01:36:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.762+0000 7fc05fe13740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd0840460 0x55dfd0840840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.762+0000 7fc05fe13740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd0840d80 0x55dfd084f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.762+0000 7fc05fe13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd084f980 0x55dfd0851d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.762+0000 7fc05fe13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dfd07a5670 con 0x55dfd084f980 2024-04-25T01:36:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.762+0000 7fc05fe13740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55dfd074d970 con 0x55dfd0840460 2024-04-25T01:36:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.762+0000 7fc05fe13740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55dfd07a1630 con 0x55dfd0840d80 2024-04-25T01:36:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.762+0000 7fc04e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd084f980 0x55dfd0851d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.762+0000 7fc04e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd084f980 0x55dfd0851d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49704/0 (socket says 172.21.15.28:49704) 2024-04-25T01:36:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.762+0000 7fc04e59c640 1 -- 172.21.15.28:0/3904936948 learned_addr learned my addr 172.21.15.28:0/3904936948 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.762+0000 7fc05759e640 1 --2- 172.21.15.28:0/3904936948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd0840460 0x55dfd0840840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.762+0000 7fc056d9d640 1 --2- 172.21.15.28:0/3904936948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd0840d80 0x55dfd084f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.763+0000 7fc054d99640 1 --2- 172.21.15.28:0/3934404179 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc0440a2d00 0x7fc0440a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.763+0000 7fc04e59c640 1 -- 172.21.15.28:0/3904936948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd0840d80 msgr2=0x55dfd084f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.763+0000 7fc04e59c640 1 --2- 172.21.15.28:0/3904936948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd0840d80 0x55dfd084f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.763+0000 7fc04e59c640 1 -- 172.21.15.28:0/3904936948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd0840460 msgr2=0x55dfd0840840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.763+0000 7fc04e59c640 1 --2- 172.21.15.28:0/3904936948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd0840460 0x55dfd0840840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.763+0000 7fc04e59c640 1 -- 172.21.15.28:0/3904936948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfd0771400 con 0x55dfd084f980 2024-04-25T01:36:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.763+0000 7fc05759e640 1 --2- 172.21.15.28:0/3904936948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd0840460 0x55dfd0840840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:36:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.763+0000 7fc05759e640 1 --2- 172.21.15.28:0/3934404179 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc0440a2d00 0x7fc0440a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.763+0000 7fc054d99640 1 -- 172.21.15.28:0/3934404179 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc0480cf750 con 0x7fc058101710 2024-04-25T01:36:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.763+0000 7fc04e59c640 1 --2- 172.21.15.28:0/3904936948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd084f980 0x55dfd0851d70 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fc04000b740 tx=0x7fc04000bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6a7898407cf14b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.763+0000 7fc04d59a640 1 -- 172.21.15.28:0/3904936948 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc040003dd0 con 0x55dfd084f980 2024-04-25T01:36:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.763+0000 7fc04d59a640 1 -- 172.21.15.28:0/3904936948 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc040015070 con 0x55dfd084f980 2024-04-25T01:36:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.763+0000 7fc04d59a640 1 -- 172.21.15.28:0/3904936948 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc040004e00 con 0x55dfd084f980 2024-04-25T01:36:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.763+0000 7fc05759e640 1 --2- 172.21.15.28:0/3934404179 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc0440a2d00 0x7fc0440a51c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc050004af0 tx=0x7fc0500095c0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.764+0000 7fc05fe13740 1 -- 172.21.15.28:0/3904936948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd084f980 msgr2=0x55dfd0851d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.764+0000 7fc05fe13740 1 --2- 172.21.15.28:0/3904936948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd084f980 0x55dfd0851d70 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fc04000b740 tx=0x7fc04000bc00 comp rx=0 tx=0).stop 2024-04-25T01:36:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.764+0000 7fc05fe13740 1 -- 172.21.15.28:0/3904936948 shutdown_connections 2024-04-25T01:36:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.764+0000 7fc05fe13740 1 --2- 172.21.15.28:0/3904936948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd084f980 0x55dfd0851d70 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.764+0000 7fc05fe13740 1 --2- 172.21.15.28:0/3904936948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd0840d80 0x55dfd084f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.764+0000 7fc05fe13740 1 --2- 172.21.15.28:0/3904936948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd0840460 0x55dfd0840840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.764+0000 7fc05fe13740 1 -- 172.21.15.28:0/3904936948 >> 172.21.15.28:0/3904936948 conn(0x55dfd083c8a0 msgr2=0x55dfd083cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.764+0000 7fc05fe13740 1 -- 172.21.15.28:0/3904936948 shutdown_connections 2024-04-25T01:36:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.764+0000 7fc05fe13740 1 -- 172.21.15.28:0/3904936948 wait complete. 2024-04-25T01:36:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.764+0000 7fc05fe13740 1 -- start start 2024-04-25T01:36:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.764+0000 7fc05fe13740 1 Processor -- start 2024-04-25T01:36:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc05fe13740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd0840460 0x55dfd084f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc05fe13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840d80 0x55dfd084c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc05fe13740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd084f980 0x55dfd084ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc05fe13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dfd08527c0 con 0x55dfd0840d80 2024-04-25T01:36:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc05fe13740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55dfd0761050 con 0x55dfd084f980 2024-04-25T01:36:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc05fe13740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55dfd0852940 con 0x55dfd0840460 2024-04-25T01:36:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc05fe13740 10 client.0 _ll_register_callbacks cb 0x55dfd04b8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc04e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840d80 0x55dfd084c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc04e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840d80 0x55dfd084c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49714/0 (socket says 172.21.15.28:49714) 2024-04-25T01:36:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc04e59c640 1 -- 172.21.15.28:0/930228002 learned_addr learned my addr 172.21.15.28:0/930228002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc05759e640 1 --2- 172.21.15.28:0/930228002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd084f980 0x55dfd084ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc056d9d640 1 --2- 172.21.15.28:0/930228002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd0840460 0x55dfd084f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc04e59c640 1 -- 172.21.15.28:0/930228002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd0840460 msgr2=0x55dfd084f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc04e59c640 1 --2- 172.21.15.28:0/930228002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd0840460 0x55dfd084f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc04e59c640 1 -- 172.21.15.28:0/930228002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd084f980 msgr2=0x55dfd084ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc04e59c640 1 --2- 172.21.15.28:0/930228002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd084f980 0x55dfd084ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc04e59c640 1 -- 172.21.15.28:0/930228002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfd0732dc0 con 0x55dfd0840d80 2024-04-25T01:36:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc056d9d640 1 --2- 172.21.15.28:0/930228002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd0840460 0x55dfd084f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc04e59c640 1 --2- 172.21.15.28:0/930228002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840d80 0x55dfd084c540 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fc0400039c0 tx=0x7fc040004380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c644c90682c61a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.765+0000 7fc03e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.766+0000 7fc03e7fc640 1 -- 172.21.15.28:0/930228002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc040028060 con 0x55dfd0840d80 2024-04-25T01:36:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.766+0000 7fc03e7fc640 1 -- 172.21.15.28:0/930228002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc040004b40 con 0x55dfd0840d80 2024-04-25T01:36:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.766+0000 7fc05fe13740 1 -- 172.21.15.28:0/930228002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc040015070 con 0x55dfd0840d80 2024-04-25T01:36:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.766+0000 7fc03e7fc640 1 -- 172.21.15.28:0/930228002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc04002f050 con 0x55dfd0840d80 2024-04-25T01:36:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.766+0000 7fc05fe13740 10 client.16884 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.766+0000 7fc05fe13740 1 -- 172.21.15.28:0/930228002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dfd04b93f0 con 0x55dfd0840d80 2024-04-25T01:36:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.766+0000 7fc03e7fc640 1 -- 172.21.15.28:0/930228002 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc040015070 con 0x55dfd0840d80 2024-04-25T01:36:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.766+0000 7fc03e7fc640 1 -- 172.21.15.28:0/930228002 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc040022070 con 0x55dfd0840d80 2024-04-25T01:36:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.766+0000 7fc05fe13740 10 client.16884 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.766+0000 7fc05fe13740 10 client.16884 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:36:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.766+0000 7fc05fe13740 20 client.16884 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.766+0000 7fc05fe13740 1 --2- 172.21.15.28:0/930228002 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55dfd085f920 0x55dfd0861d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.766+0000 7fc05fe13740 4 client.16884 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.766+0000 7fc05fe13740 1 -- 172.21.15.28:0/930228002 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfd07b2bb0 con 0x55dfd085f920 2024-04-25T01:36:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.766+0000 7fc056d9d640 1 --2- 172.21.15.28:0/930228002 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55dfd085f920 0x55dfd0861d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.767+0000 7fc056d9d640 1 --2- 172.21.15.28:0/930228002 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55dfd085f920 0x55dfd0861d10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=356c49a2dafc6c37 server_cookie=e105042d9c9a369a in_seq=0 out_seq=0 2024-04-25T01:36:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.767+0000 7fc03e7fc640 10 client.16884 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:36:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.767+0000 7fc03e7fc640 1 -- 172.21.15.28:0/930228002 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dfd07b2bb0 con 0x55dfd085f920 2024-04-25T01:36:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.767+0000 7fc03e7fc640 10 client.16884 handle_command_reply: tid=0 2024-04-25T01:36:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.767+0000 7fc05fe13740 1 client.16884 shutdown 2024-04-25T01:36:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.768+0000 7fc05fe13740 1 -- 172.21.15.28:0/930228002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840d80 msgr2=0x55dfd084c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.768+0000 7fc05fe13740 1 --2- 172.21.15.28:0/930228002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840d80 0x55dfd084c540 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fc0400039c0 tx=0x7fc040004380 comp rx=0 tx=0).stop 2024-04-25T01:36:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.768+0000 7fc05fe13740 1 -- 172.21.15.28:0/930228002 shutdown_connections 2024-04-25T01:36:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.768+0000 7fc05fe13740 1 --2- 172.21.15.28:0/930228002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd084f980 0x55dfd084ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.768+0000 7fc05fe13740 1 --2- 172.21.15.28:0/930228002 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55dfd085f920 0x55dfd0861d10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.768+0000 7fc05fe13740 1 --2- 172.21.15.28:0/930228002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840d80 0x55dfd084c540 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.768+0000 7fc03e7fc640 0 client.16884 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:36:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.768+0000 7fc05fe13740 1 --2- 172.21.15.28:0/930228002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd0840460 0x55dfd084f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.768+0000 7fc05fe13740 1 -- 172.21.15.28:0/930228002 >> 172.21.15.28:0/930228002 conn(0x55dfd083c8a0 msgr2=0x55dfd0842650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.768+0000 7fc05fe13740 1 -- 172.21.15.28:0/930228002 shutdown_connections 2024-04-25T01:36:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.768+0000 7fc05fe13740 1 -- 172.21.15.28:0/930228002 wait complete. 2024-04-25T01:36:58.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.768+0000 7fc05fe13740 20 client.16884 trim_cache size 0 max 16384 2024-04-25T01:36:58.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.781+0000 7fc05fe13740 1 Processor -- start 2024-04-25T01:36:58.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.781+0000 7fc05fe13740 1 -- start start 2024-04-25T01:36:58.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.781+0000 7fc05fe13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840460 0x55dfd085b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.781+0000 7fc05fe13740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd0840d80 0x55dfd0847af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.781+0000 7fc05fe13740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd084f980 0x55dfd084ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.781+0000 7fc05fe13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581ac8a0 con 0x55dfd0840460 2024-04-25T01:36:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.781+0000 7fc05fe13740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581ae3e0 con 0x55dfd084f980 2024-04-25T01:36:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.782+0000 7fc05fe13740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581acf70 con 0x55dfd0840d80 2024-04-25T01:36:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.782+0000 7fc056d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840460 0x55dfd085b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.782+0000 7fc056d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840460 0x55dfd085b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49716/0 (socket says 172.21.15.28:49716) 2024-04-25T01:36:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.782+0000 7fc056d9d640 1 -- 172.21.15.28:0/833278208 learned_addr learned my addr 172.21.15.28:0/833278208 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.782+0000 7fc04e59c640 1 --2- 172.21.15.28:0/833278208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd0840d80 0x55dfd0847af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.782+0000 7fc05759e640 1 --2- 172.21.15.28:0/833278208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd084f980 0x55dfd084ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.782+0000 7fc056d9d640 1 -- 172.21.15.28:0/833278208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd0840d80 msgr2=0x55dfd0847af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.782+0000 7fc056d9d640 1 --2- 172.21.15.28:0/833278208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd0840d80 0x55dfd0847af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.782+0000 7fc056d9d640 1 -- 172.21.15.28:0/833278208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd084f980 msgr2=0x55dfd084ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.782+0000 7fc056d9d640 1 --2- 172.21.15.28:0/833278208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd084f980 0x55dfd084ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.782+0000 7fc056d9d640 1 -- 172.21.15.28:0/833278208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfd05837d0 con 0x55dfd0840460 2024-04-25T01:36:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.782+0000 7fc056d9d640 1 --2- 172.21.15.28:0/833278208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840460 0x55dfd085b720 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fc04800cfd0 tx=0x7fc0480d1c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68d9bc9c3a760ce3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.782+0000 7fc04cd99640 1 -- 172.21.15.28:0/833278208 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0480b3070 con 0x55dfd0840460 2024-04-25T01:36:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.782+0000 7fc04cd99640 1 -- 172.21.15.28:0/833278208 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0480afe50 con 0x55dfd0840460 2024-04-25T01:36:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.782+0000 7fc04cd99640 1 -- 172.21.15.28:0/833278208 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0480a45a0 con 0x55dfd0840460 2024-04-25T01:36:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.783+0000 7fc05fe13740 1 -- 172.21.15.28:0/833278208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840460 msgr2=0x55dfd085b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.783+0000 7fc05fe13740 1 --2- 172.21.15.28:0/833278208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840460 0x55dfd085b720 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fc04800cfd0 tx=0x7fc0480d1c60 comp rx=0 tx=0).stop 2024-04-25T01:36:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.783+0000 7fc05fe13740 1 -- 172.21.15.28:0/833278208 shutdown_connections 2024-04-25T01:36:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.783+0000 7fc05fe13740 1 --2- 172.21.15.28:0/833278208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd084f980 0x55dfd084ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.783+0000 7fc05fe13740 1 --2- 172.21.15.28:0/833278208 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd0840d80 0x55dfd0847af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.783+0000 7fc05fe13740 1 --2- 172.21.15.28:0/833278208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840460 0x55dfd085b720 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.783+0000 7fc05fe13740 1 -- 172.21.15.28:0/833278208 >> 172.21.15.28:0/833278208 conn(0x55dfd083c8a0 msgr2=0x55dfd08473e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.783+0000 7fc05fe13740 1 -- 172.21.15.28:0/833278208 shutdown_connections 2024-04-25T01:36:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.783+0000 7fc05fe13740 1 -- 172.21.15.28:0/833278208 wait complete. 2024-04-25T01:36:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.783+0000 7fc05fe13740 1 -- start start 2024-04-25T01:36:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.783+0000 7fc05fe13740 1 Processor -- start 2024-04-25T01:36:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc05fe13740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840460 0x55dfd08bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc05fe13740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd0840d80 0x55dfd08be480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc05fe13740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd084f980 0x55dfd0852d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc05fe13740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581add10 con 0x55dfd0840460 2024-04-25T01:36:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc05fe13740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581ad640 con 0x55dfd0840d80 2024-04-25T01:36:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc05fe13740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581aa880 con 0x55dfd084f980 2024-04-25T01:36:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc05fe13740 10 client.0 _ll_register_callbacks cb 0x7fc050009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc056d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840460 0x55dfd08bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc056d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840460 0x55dfd08bdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49730/0 (socket says 172.21.15.28:49730) 2024-04-25T01:36:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc056d9d640 1 -- 172.21.15.28:0/693055838 learned_addr learned my addr 172.21.15.28:0/693055838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc05759e640 1 --2- 172.21.15.28:0/693055838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd084f980 0x55dfd0852d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc056d9d640 1 -- 172.21.15.28:0/693055838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd084f980 msgr2=0x55dfd0852d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc04e59c640 1 --2- 172.21.15.28:0/693055838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd0840d80 0x55dfd08be480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc056d9d640 1 --2- 172.21.15.28:0/693055838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd084f980 0x55dfd0852d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc056d9d640 1 -- 172.21.15.28:0/693055838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd0840d80 msgr2=0x55dfd08be480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc056d9d640 1 --2- 172.21.15.28:0/693055838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd0840d80 0x55dfd08be480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc056d9d640 1 -- 172.21.15.28:0/693055838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfd0853450 con 0x55dfd0840460 2024-04-25T01:36:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc056d9d640 1 --2- 172.21.15.28:0/693055838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840460 0x55dfd08bdf40 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fc0480d1010 tx=0x7fc0480ccad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d66937f1d22582c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc03e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc03e7fc640 1 -- 172.21.15.28:0/693055838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0480b9020 con 0x55dfd0840460 2024-04-25T01:36:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc03e7fc640 1 -- 172.21.15.28:0/693055838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc0480ba250 con 0x55dfd0840460 2024-04-25T01:36:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc05fe13740 1 -- 172.21.15.28:0/693055838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0480afe50 con 0x55dfd0840460 2024-04-25T01:36:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc03e7fc640 1 -- 172.21.15.28:0/693055838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc0480b3070 con 0x55dfd0840460 2024-04-25T01:36:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc05fe13740 10 client.16890 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.784+0000 7fc05fe13740 1 -- 172.21.15.28:0/693055838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dfd08c5670 con 0x55dfd0840460 2024-04-25T01:36:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.785+0000 7fc03e7fc640 1 -- 172.21.15.28:0/693055838 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0480afe50 con 0x55dfd0840460 2024-04-25T01:36:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.785+0000 7fc03e7fc640 1 -- 172.21.15.28:0/693055838 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc048020030 con 0x55dfd0840460 2024-04-25T01:36:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.785+0000 7fc05fe13740 10 client.16890 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.785+0000 7fc05fe13740 10 client.16890 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:36:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.785+0000 7fc05fe13740 20 client.16890 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.785+0000 7fc05fe13740 1 --2- 172.21.15.28:0/693055838 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55dfd08c5cf0 0x55dfd08c8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.785+0000 7fc05fe13740 4 client.16890 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.785+0000 7fc05fe13740 1 -- 172.21.15.28:0/693055838 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dfd07acf00 con 0x55dfd08c5cf0 2024-04-25T01:36:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.785+0000 7fc04e59c640 1 --2- 172.21.15.28:0/693055838 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55dfd08c5cf0 0x55dfd08c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.785+0000 7fc04e59c640 1 --2- 172.21.15.28:0/693055838 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55dfd08c5cf0 0x55dfd08c8110 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3c8345ac2a9a2abc server_cookie=268e7b698e5753d8 in_seq=0 out_seq=0 2024-04-25T01:36:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.785+0000 7fc03e7fc640 10 client.16890 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:36:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc03e7fc640 1 -- 172.21.15.28:0/693055838 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dfd07acf00 con 0x55dfd08c5cf0 2024-04-25T01:36:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc03e7fc640 10 client.16890 handle_command_reply: tid=0 2024-04-25T01:36:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05fe13740 1 client.16890 shutdown 2024-04-25T01:36:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05fe13740 1 -- 172.21.15.28:0/693055838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840460 msgr2=0x55dfd08bdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05fe13740 1 --2- 172.21.15.28:0/693055838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840460 0x55dfd08bdf40 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fc0480d1010 tx=0x7fc0480ccad0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05fe13740 1 -- 172.21.15.28:0/693055838 shutdown_connections 2024-04-25T01:36:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05fe13740 1 --2- 172.21.15.28:0/693055838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dfd084f980 0x55dfd0852d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05fe13740 1 --2- 172.21.15.28:0/693055838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dfd0840d80 0x55dfd08be480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05fe13740 1 --2- 172.21.15.28:0/693055838 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55dfd08c5cf0 0x55dfd08c8110 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05fe13740 1 --2- 172.21.15.28:0/693055838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd0840460 0x55dfd08bdf40 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc03e7fc640 0 client.16890 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:36:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05fe13740 1 -- 172.21.15.28:0/693055838 >> 172.21.15.28:0/693055838 conn(0x55dfd083c8a0 msgr2=0x55dfd0842630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05fe13740 1 -- 172.21.15.28:0/693055838 shutdown_connections 2024-04-25T01:36:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05fe13740 1 -- 172.21.15.28:0/693055838 wait complete. 2024-04-25T01:36:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05fe13740 20 client.16890 trim_cache size 0 max 16384 2024-04-25T01:36:58.770 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:36:58.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05d109640 1 -- 172.21.15.28:0/3934404179 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc0440a2d00 msgr2=0x7fc0440a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05d109640 1 --2- 172.21.15.28:0/3934404179 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc0440a2d00 0x7fc0440a51c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc050004af0 tx=0x7fc0500095c0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05d109640 1 -- 172.21.15.28:0/3934404179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058101710 msgr2=0x7fc058198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05d109640 1 --2- 172.21.15.28:0/3934404179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058101710 0x7fc058198030 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fc04800c970 tx=0x7fc04800ce30 comp rx=0 tx=0).stop 2024-04-25T01:36:58.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05d109640 1 -- 172.21.15.28:0/3934404179 shutdown_connections 2024-04-25T01:36:58.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05d109640 1 --2- 172.21.15.28:0/3934404179 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc0440a2d00 0x7fc0440a51c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05d109640 1 --2- 172.21.15.28:0/3934404179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc058107720 0x7fc05819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05d109640 1 --2- 172.21.15.28:0/3934404179 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc0581020d0 0x7fc058198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05d109640 1 --2- 172.21.15.28:0/3934404179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc058101710 0x7fc058198030 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.786+0000 7fc05d109640 1 -- 172.21.15.28:0/3934404179 >> 172.21.15.28:0/3934404179 conn(0x7fc0580fd180 msgr2=0x7fc0580fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.787+0000 7fc05d109640 1 -- 172.21.15.28:0/3934404179 shutdown_connections 2024-04-25T01:36:58.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.787+0000 7fc05d109640 1 -- 172.21.15.28:0/3934404179 wait complete. 2024-04-25T01:36:58.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:36:58.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.901+0000 7ff140f42640 1 -- 172.21.15.28:0/158512854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff13c100520 msgr2=0x7ff13c100900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.901+0000 7ff140f42640 1 --2- 172.21.15.28:0/158512854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff13c100520 0x7ff13c100900 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ff12c009900 tx=0x7ff12c02a410 comp rx=0 tx=0).stop 2024-04-25T01:36:58.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.902+0000 7ff140f42640 1 -- 172.21.15.28:0/158512854 shutdown_connections 2024-04-25T01:36:58.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.902+0000 7ff140f42640 1 --2- 172.21.15.28:0/158512854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff13c10e0d0 0x7ff13c1104c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.902+0000 7ff140f42640 1 --2- 172.21.15.28:0/158512854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff13c100e40 0x7ff13c10db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.902+0000 7ff140f42640 1 --2- 172.21.15.28:0/158512854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff13c100520 0x7ff13c100900 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.902+0000 7ff140f42640 1 -- 172.21.15.28:0/158512854 >> 172.21.15.28:0/158512854 conn(0x7ff13c0769c0 msgr2=0x7ff13c076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.902+0000 7ff140f42640 1 -- 172.21.15.28:0/158512854 shutdown_connections 2024-04-25T01:36:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.902+0000 7ff140f42640 1 -- 172.21.15.28:0/158512854 wait complete. 2024-04-25T01:36:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.902+0000 7ff140f42640 1 Processor -- start 2024-04-25T01:36:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.902+0000 7ff140f42640 1 -- start start 2024-04-25T01:36:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff140f42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff13c100520 0x7ff13c19a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff140f42640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff13c100e40 0x7ff13c19a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff140f42640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff13c10e0d0 0x7ff13c19ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff140f42640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c113d30 con 0x7ff13c100520 2024-04-25T01:36:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff140f42640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c113bb0 con 0x7ff13c10e0d0 2024-04-25T01:36:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff140f42640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c113eb0 con 0x7ff13c100e40 2024-04-25T01:36:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff13a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff13c100520 0x7ff13c19a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff13a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff13c100520 0x7ff13c19a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49744/0 (socket says 172.21.15.28:49744) 2024-04-25T01:36:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff13a59c640 1 -- 172.21.15.28:0/1160177507 learned_addr learned my addr 172.21.15.28:0/1160177507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff133fff640 1 --2- 172.21.15.28:0/1160177507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff13c100e40 0x7ff13c19a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff13ad9d640 1 --2- 172.21.15.28:0/1160177507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff13c10e0d0 0x7ff13c19ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff13a59c640 1 -- 172.21.15.28:0/1160177507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff13c100e40 msgr2=0x7ff13c19a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff13a59c640 1 --2- 172.21.15.28:0/1160177507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff13c100e40 0x7ff13c19a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff13a59c640 1 -- 172.21.15.28:0/1160177507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff13c10e0d0 msgr2=0x7ff13c19ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff13a59c640 1 --2- 172.21.15.28:0/1160177507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff13c10e0d0 0x7ff13c19ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff13a59c640 1 -- 172.21.15.28:0/1160177507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13c19f160 con 0x7ff13c100520 2024-04-25T01:36:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff13a59c640 1 --2- 172.21.15.28:0/1160177507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff13c100520 0x7ff13c19a170 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff12c005ab0 tx=0x7ff12c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff1337fe640 1 -- 172.21.15.28:0/1160177507 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c03f030 con 0x7ff13c100520 2024-04-25T01:36:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff1337fe640 1 -- 172.21.15.28:0/1160177507 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff12c003ce0 con 0x7ff13c100520 2024-04-25T01:36:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff140f42640 1 -- 172.21.15.28:0/1160177507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff13c19f3f0 con 0x7ff13c100520 2024-04-25T01:36:58.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff1337fe640 1 -- 172.21.15.28:0/1160177507 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c035070 con 0x7ff13c100520 2024-04-25T01:36:58.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.903+0000 7ff140f42640 1 -- 172.21.15.28:0/1160177507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff13c1a6ce0 con 0x7ff13c100520 2024-04-25T01:36:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.905+0000 7ff1337fe640 1 -- 172.21.15.28:0/1160177507 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff12c042030 con 0x7ff13c100520 2024-04-25T01:36:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.906+0000 7ff143c53740 1 Processor -- start 2024-04-25T01:36:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.906+0000 7ff143c53740 1 -- start start 2024-04-25T01:36:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.906+0000 7ff1337fe640 1 --2- 172.21.15.28:0/1160177507 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff1280a2d00 0x7ff1280a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.906+0000 7ff143c53740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a75494580 0x559a75494960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.906+0000 7ff143c53740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a75494ea0 0x559a7549f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.906+0000 7ff143c53740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7549f570 0x559a754a1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.906+0000 7ff143c53740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559a753f9670 con 0x559a7549f570 2024-04-25T01:36:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.906+0000 7ff143c53740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559a753a1970 con 0x559a75494580 2024-04-25T01:36:58.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.906+0000 7ff143c53740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x559a753f5630 con 0x559a75494ea0 2024-04-25T01:36:58.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.907+0000 7ff13ad9d640 1 --2- 172.21.15.28:0/1160177507 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff1280a2d00 0x7ff1280a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.907+0000 7ff133fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a75494ea0 0x559a7549f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.907+0000 7ff1337fe640 1 -- 172.21.15.28:0/1160177507 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff12c0f01b0 con 0x7ff13c100520 2024-04-25T01:36:58.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.907+0000 7ff13ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7549f570 0x559a754a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.907+0000 7ff133fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a75494ea0 0x559a7549f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.28:45726/0 (socket says 172.21.15.28:45726) 2024-04-25T01:36:58.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.907+0000 7ff133fff640 1 -- 172.21.15.28:0/3333065451 learned_addr learned my addr 172.21.15.28:0/3333065451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.907+0000 7ff13a59c640 1 --2- 172.21.15.28:0/3333065451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a75494580 0x559a75494960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.907+0000 7ff133fff640 1 -- 172.21.15.28:0/3333065451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a75494580 msgr2=0x559a75494960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.907+0000 7ff133fff640 1 --2- 172.21.15.28:0/3333065451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a75494580 0x559a75494960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.907+0000 7ff133fff640 1 -- 172.21.15.28:0/3333065451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7549f570 msgr2=0x559a754a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.907+0000 7ff133fff640 1 --2- 172.21.15.28:0/3333065451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7549f570 0x559a754a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.907+0000 7ff133fff640 1 -- 172.21.15.28:0/3333065451 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a753c5400 con 0x559a75494ea0 2024-04-25T01:36:58.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.907+0000 7ff13ad9d640 1 --2- 172.21.15.28:0/3333065451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7549f570 0x559a754a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:36:58.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.907+0000 7ff13ad9d640 1 --2- 172.21.15.28:0/1160177507 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff1280a2d00 0x7ff1280a51c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff13c069b70 tx=0x7ff13400aba0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.907+0000 7ff133fff640 1 --2- 172.21.15.28:0/3333065451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a75494ea0 0x559a7549f030 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff12400cbc0 tx=0x7ff12400e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5049e634862ac1a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.907+0000 7ff130ff9640 1 -- 172.21.15.28:0/3333065451 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff124010630 con 0x559a75494ea0 2024-04-25T01:36:58.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.907+0000 7ff130ff9640 1 -- 172.21.15.28:0/3333065451 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff124010cb0 con 0x559a75494ea0 2024-04-25T01:36:58.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.907+0000 7ff130ff9640 1 -- 172.21.15.28:0/3333065451 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12401ab30 con 0x559a75494ea0 2024-04-25T01:36:58.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.908+0000 7ff143c53740 1 -- 172.21.15.28:0/3333065451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a75494ea0 msgr2=0x559a7549f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.908+0000 7ff143c53740 1 --2- 172.21.15.28:0/3333065451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a75494ea0 0x559a7549f030 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff12400cbc0 tx=0x7ff12400e720 comp rx=0 tx=0).stop 2024-04-25T01:36:58.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.908+0000 7ff143c53740 1 -- 172.21.15.28:0/3333065451 shutdown_connections 2024-04-25T01:36:58.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.908+0000 7ff143c53740 1 --2- 172.21.15.28:0/3333065451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a7549f570 0x559a754a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.908+0000 7ff143c53740 1 --2- 172.21.15.28:0/3333065451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a75494ea0 0x559a7549f030 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.908+0000 7ff143c53740 1 --2- 172.21.15.28:0/3333065451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a75494580 0x559a75494960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.908+0000 7ff143c53740 1 -- 172.21.15.28:0/3333065451 >> 172.21.15.28:0/3333065451 conn(0x559a754908a0 msgr2=0x559a75490c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.908+0000 7ff143c53740 1 -- 172.21.15.28:0/3333065451 shutdown_connections 2024-04-25T01:36:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.908+0000 7ff143c53740 1 -- 172.21.15.28:0/3333065451 wait complete. 2024-04-25T01:36:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.909+0000 7ff143c53740 1 -- start start 2024-04-25T01:36:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.909+0000 7ff143c53740 1 Processor -- start 2024-04-25T01:36:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.909+0000 7ff143c53740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a75494580 0x559a7549eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.909+0000 7ff143c53740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494ea0 0x559a7549c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.909+0000 7ff143c53740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a7549f570 0x559a7549c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.909+0000 7ff143c53740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559a754a2360 con 0x559a75494ea0 2024-04-25T01:36:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.909+0000 7ff143c53740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559a753b5050 con 0x559a75494580 2024-04-25T01:36:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.909+0000 7ff143c53740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x559a754a24e0 con 0x559a7549f570 2024-04-25T01:36:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.909+0000 7ff143c53740 10 client.0 _ll_register_callbacks cb 0x559a7510cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.909+0000 7ff133fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494ea0 0x559a7549c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.909+0000 7ff133fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494ea0 0x559a7549c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49762/0 (socket says 172.21.15.28:49762) 2024-04-25T01:36:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.909+0000 7ff133fff640 1 -- 172.21.15.28:0/2003011413 learned_addr learned my addr 172.21.15.28:0/2003011413 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.909+0000 7ff13ad9d640 1 --2- 172.21.15.28:0/2003011413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a7549f570 0x559a7549c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.909+0000 7ff13a59c640 1 --2- 172.21.15.28:0/2003011413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a75494580 0x559a7549eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.909+0000 7ff133fff640 1 -- 172.21.15.28:0/2003011413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a7549f570 msgr2=0x559a7549c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.909+0000 7ff133fff640 1 --2- 172.21.15.28:0/2003011413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a7549f570 0x559a7549c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.909+0000 7ff133fff640 1 -- 172.21.15.28:0/2003011413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a75494580 msgr2=0x559a7549eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.909+0000 7ff133fff640 1 --2- 172.21.15.28:0/2003011413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a75494580 0x559a7549eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.909+0000 7ff133fff640 1 -- 172.21.15.28:0/2003011413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a75386dc0 con 0x559a75494ea0 2024-04-25T01:36:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.910+0000 7ff133fff640 1 --2- 172.21.15.28:0/2003011413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494ea0 0x559a7549c1c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ff12400ec20 tx=0x7ff12401aea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36ac25b8e5114ddb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.910+0000 7ff121ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.910+0000 7ff121ffb640 1 -- 172.21.15.28:0/2003011413 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff124029060 con 0x559a75494ea0 2024-04-25T01:36:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.910+0000 7ff121ffb640 1 -- 172.21.15.28:0/2003011413 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff12400c6d0 con 0x559a75494ea0 2024-04-25T01:36:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.910+0000 7ff143c53740 1 -- 172.21.15.28:0/2003011413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff124010cb0 con 0x559a75494ea0 2024-04-25T01:36:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.910+0000 7ff121ffb640 1 -- 172.21.15.28:0/2003011413 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff124010e60 con 0x559a75494ea0 2024-04-25T01:36:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.910+0000 7ff143c53740 10 client.16902 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.910+0000 7ff143c53740 1 -- 172.21.15.28:0/2003011413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559a7510d3f0 con 0x559a75494ea0 2024-04-25T01:36:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.910+0000 7ff121ffb640 1 -- 172.21.15.28:0/2003011413 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff124010cb0 con 0x559a75494ea0 2024-04-25T01:36:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.910+0000 7ff121ffb640 1 -- 172.21.15.28:0/2003011413 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff124024990 con 0x559a75494ea0 2024-04-25T01:36:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.910+0000 7ff143c53740 10 client.16902 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.910+0000 7ff143c53740 10 client.16902 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:36:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.910+0000 7ff143c53740 20 client.16902 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.910+0000 7ff143c53740 1 --2- 172.21.15.28:0/2003011413 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x559a754af4e0 0x559a754b18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.910+0000 7ff143c53740 4 client.16902 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.910+0000 7ff143c53740 1 -- 172.21.15.28:0/2003011413 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a75406bb0 con 0x559a754af4e0 2024-04-25T01:36:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.911+0000 7ff13a59c640 1 --2- 172.21.15.28:0/2003011413 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x559a754af4e0 0x559a754b18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.911+0000 7ff13a59c640 1 --2- 172.21.15.28:0/2003011413 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x559a754af4e0 0x559a754b18d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f2cdd95885bce846 server_cookie=83a910251223ec4a in_seq=0 out_seq=0 2024-04-25T01:36:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.911+0000 7ff121ffb640 10 client.16902 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:36:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.912+0000 7ff121ffb640 1 -- 172.21.15.28:0/2003011413 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559a75406bb0 con 0x559a754af4e0 2024-04-25T01:36:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.912+0000 7ff121ffb640 10 client.16902 handle_command_reply: tid=0 2024-04-25T01:36:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.912+0000 7ff143c53740 1 client.16902 shutdown 2024-04-25T01:36:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.912+0000 7ff143c53740 1 -- 172.21.15.28:0/2003011413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494ea0 msgr2=0x559a7549c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.912+0000 7ff143c53740 1 --2- 172.21.15.28:0/2003011413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494ea0 0x559a7549c1c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ff12400ec20 tx=0x7ff12401aea0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.912+0000 7ff143c53740 1 -- 172.21.15.28:0/2003011413 shutdown_connections 2024-04-25T01:36:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.912+0000 7ff143c53740 1 --2- 172.21.15.28:0/2003011413 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x559a754af4e0 0x559a754b18d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.912+0000 7ff143c53740 1 --2- 172.21.15.28:0/2003011413 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a7549f570 0x559a7549c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.912+0000 7ff143c53740 1 --2- 172.21.15.28:0/2003011413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494ea0 0x559a7549c1c0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.912+0000 7ff143c53740 1 --2- 172.21.15.28:0/2003011413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a75494580 0x559a7549eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.912+0000 7ff121ffb640 0 client.16902 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:36:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.912+0000 7ff143c53740 1 -- 172.21.15.28:0/2003011413 >> 172.21.15.28:0/2003011413 conn(0x559a754908a0 msgr2=0x559a754966a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.912+0000 7ff143c53740 1 -- 172.21.15.28:0/2003011413 shutdown_connections 2024-04-25T01:36:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.912+0000 7ff143c53740 1 -- 172.21.15.28:0/2003011413 wait complete. 2024-04-25T01:36:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.912+0000 7ff143c53740 20 client.16902 trim_cache size 0 max 16384 2024-04-25T01:36:58.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.925+0000 7ff143c53740 1 Processor -- start 2024-04-25T01:36:58.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.925+0000 7ff143c53740 1 -- start start 2024-04-25T01:36:58.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.925+0000 7ff143c53740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a75494580 0x559a754ab1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.925+0000 7ff143c53740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494ea0 0x559a7548fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.925+0000 7ff143c53740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a7549f570 0x559a7549ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.925+0000 7ff143c53740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c1b0c10 con 0x559a75494ea0 2024-04-25T01:36:58.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.925+0000 7ff143c53740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c1b2750 con 0x559a75494580 2024-04-25T01:36:58.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.925+0000 7ff143c53740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c1b12e0 con 0x559a7549f570 2024-04-25T01:36:58.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.925+0000 7ff133fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494ea0 0x559a7548fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.926+0000 7ff133fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494ea0 0x559a7548fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49768/0 (socket says 172.21.15.28:49768) 2024-04-25T01:36:58.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.926+0000 7ff133fff640 1 -- 172.21.15.28:0/3224917551 learned_addr learned my addr 172.21.15.28:0/3224917551 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.926+0000 7ff13a59c640 1 --2- 172.21.15.28:0/3224917551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a75494580 0x559a754ab1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.926+0000 7ff13ad9d640 1 --2- 172.21.15.28:0/3224917551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a7549f570 0x559a7549ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.926+0000 7ff133fff640 1 -- 172.21.15.28:0/3224917551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a7549f570 msgr2=0x559a7549ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.926+0000 7ff133fff640 1 --2- 172.21.15.28:0/3224917551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a7549f570 0x559a7549ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.926+0000 7ff133fff640 1 -- 172.21.15.28:0/3224917551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a75494580 msgr2=0x559a754ab1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.926+0000 7ff133fff640 1 --2- 172.21.15.28:0/3224917551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a75494580 0x559a754ab1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.926+0000 7ff133fff640 1 -- 172.21.15.28:0/3224917551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a751d77d0 con 0x559a75494ea0 2024-04-25T01:36:58.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.926+0000 7ff133fff640 1 --2- 172.21.15.28:0/3224917551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494ea0 0x559a7548fd80 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7ff124024f00 tx=0x7ff12401abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f273129ef91894 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.926+0000 7ff123fff640 1 -- 172.21.15.28:0/3224917551 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff124029040 con 0x559a75494ea0 2024-04-25T01:36:58.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.926+0000 7ff123fff640 1 -- 172.21.15.28:0/3224917551 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff12400c430 con 0x559a75494ea0 2024-04-25T01:36:58.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.926+0000 7ff123fff640 1 -- 172.21.15.28:0/3224917551 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff124005330 con 0x559a75494ea0 2024-04-25T01:36:58.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff143c53740 1 -- 172.21.15.28:0/3224917551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494ea0 msgr2=0x559a7548fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff143c53740 1 --2- 172.21.15.28:0/3224917551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494ea0 0x559a7548fd80 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7ff124024f00 tx=0x7ff12401abc0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff143c53740 1 -- 172.21.15.28:0/3224917551 shutdown_connections 2024-04-25T01:36:58.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff143c53740 1 --2- 172.21.15.28:0/3224917551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a7549f570 0x559a7549ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff143c53740 1 --2- 172.21.15.28:0/3224917551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494ea0 0x559a7548fd80 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff143c53740 1 --2- 172.21.15.28:0/3224917551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a75494580 0x559a754ab1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff143c53740 1 -- 172.21.15.28:0/3224917551 >> 172.21.15.28:0/3224917551 conn(0x559a754908a0 msgr2=0x559a754aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff143c53740 1 -- 172.21.15.28:0/3224917551 shutdown_connections 2024-04-25T01:36:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff143c53740 1 -- 172.21.15.28:0/3224917551 wait complete. 2024-04-25T01:36:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff143c53740 1 -- start start 2024-04-25T01:36:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff143c53740 1 Processor -- start 2024-04-25T01:36:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff143c53740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494580 0x559a754a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff143c53740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a75494ea0 0x559a754a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff143c53740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a7549f570 0x559a754a5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff143c53740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c1b2080 con 0x559a75494580 2024-04-25T01:36:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff143c53740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c1b19b0 con 0x559a7549f570 2024-04-25T01:36:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff143c53740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c1aebf0 con 0x559a75494ea0 2024-04-25T01:36:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff143c53740 10 client.0 _ll_register_callbacks cb 0x7ff124014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff13a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494580 0x559a754a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff13a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494580 0x559a754a53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49778/0 (socket says 172.21.15.28:49778) 2024-04-25T01:36:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.927+0000 7ff13a59c640 1 -- 172.21.15.28:0/2428653523 learned_addr learned my addr 172.21.15.28:0/2428653523 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.928+0000 7ff133fff640 1 --2- 172.21.15.28:0/2428653523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a75494ea0 0x559a754a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.928+0000 7ff13ad9d640 1 --2- 172.21.15.28:0/2428653523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a7549f570 0x559a754a5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.928+0000 7ff13a59c640 1 -- 172.21.15.28:0/2428653523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a75494ea0 msgr2=0x559a754a58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.928+0000 7ff13a59c640 1 --2- 172.21.15.28:0/2428653523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a75494ea0 0x559a754a58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.928+0000 7ff13a59c640 1 -- 172.21.15.28:0/2428653523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a7549f570 msgr2=0x559a754a5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.928+0000 7ff13a59c640 1 --2- 172.21.15.28:0/2428653523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a7549f570 0x559a754a5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.928+0000 7ff13a59c640 1 -- 172.21.15.28:0/2428653523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a75514a00 con 0x559a75494580 2024-04-25T01:36:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.928+0000 7ff13ad9d640 1 --2- 172.21.15.28:0/2428653523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a7549f570 0x559a754a5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.928+0000 7ff13a59c640 1 --2- 172.21.15.28:0/2428653523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494580 0x559a754a53b0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7ff12c0bc2c0 tx=0x7ff12c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63f1c014992db230 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.928+0000 7ff121ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.928+0000 7ff121ffb640 1 -- 172.21.15.28:0/2428653523 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c049040 con 0x559a75494580 2024-04-25T01:36:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.928+0000 7ff121ffb640 1 -- 172.21.15.28:0/2428653523 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff12c03bd70 con 0x559a75494580 2024-04-25T01:36:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.928+0000 7ff143c53740 1 -- 172.21.15.28:0/2428653523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff12400c430 con 0x559a75494580 2024-04-25T01:36:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.928+0000 7ff121ffb640 1 -- 172.21.15.28:0/2428653523 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff12c0c06e0 con 0x559a75494580 2024-04-25T01:36:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.928+0000 7ff143c53740 10 client.16908 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.928+0000 7ff143c53740 1 -- 172.21.15.28:0/2428653523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559a75515d20 con 0x559a75494580 2024-04-25T01:36:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.928+0000 7ff121ffb640 1 -- 172.21.15.28:0/2428653523 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff12400c430 con 0x559a75494580 2024-04-25T01:36:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.929+0000 7ff121ffb640 1 -- 172.21.15.28:0/2428653523 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff12c03b460 con 0x559a75494580 2024-04-25T01:36:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.929+0000 7ff143c53740 10 client.16908 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.929+0000 7ff143c53740 10 client.16908 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:36:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.929+0000 7ff143c53740 20 client.16908 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.929+0000 7ff143c53740 1 --2- 172.21.15.28:0/2428653523 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x559a755163f0 0x559a75518800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.929+0000 7ff143c53740 4 client.16908 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.929+0000 7ff143c53740 1 -- 172.21.15.28:0/2428653523 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559a75400f00 con 0x559a755163f0 2024-04-25T01:36:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.929+0000 7ff133fff640 1 --2- 172.21.15.28:0/2428653523 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x559a755163f0 0x559a75518800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff133fff640 1 --2- 172.21.15.28:0/2428653523 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x559a755163f0 0x559a75518800 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=567df2bcf3a040f3 server_cookie=919921ee72a33b55 in_seq=0 out_seq=0 2024-04-25T01:36:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff121ffb640 10 client.16908 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:36:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff121ffb640 1 -- 172.21.15.28:0/2428653523 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559a75400f00 con 0x559a755163f0 2024-04-25T01:36:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff121ffb640 10 client.16908 handle_command_reply: tid=0 2024-04-25T01:36:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff143c53740 1 client.16908 shutdown 2024-04-25T01:36:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff143c53740 1 -- 172.21.15.28:0/2428653523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494580 msgr2=0x559a754a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff143c53740 1 --2- 172.21.15.28:0/2428653523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494580 0x559a754a53b0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7ff12c0bc2c0 tx=0x7ff12c02a410 comp rx=0 tx=0).stop 2024-04-25T01:36:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff143c53740 1 -- 172.21.15.28:0/2428653523 shutdown_connections 2024-04-25T01:36:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff143c53740 1 --2- 172.21.15.28:0/2428653523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559a7549f570 0x559a754a5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff143c53740 1 --2- 172.21.15.28:0/2428653523 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x559a755163f0 0x559a75518800 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff143c53740 1 --2- 172.21.15.28:0/2428653523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559a75494ea0 0x559a754a58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff143c53740 1 --2- 172.21.15.28:0/2428653523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559a75494580 0x559a754a53b0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff121ffb640 0 client.16908 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:36:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff143c53740 1 -- 172.21.15.28:0/2428653523 >> 172.21.15.28:0/2428653523 conn(0x559a754908a0 msgr2=0x559a7549d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff143c53740 1 -- 172.21.15.28:0/2428653523 shutdown_connections 2024-04-25T01:36:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff143c53740 1 -- 172.21.15.28:0/2428653523 wait complete. 2024-04-25T01:36:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff143c53740 20 client.16908 trim_cache size 0 max 16384 2024-04-25T01:36:58.915 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:36:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff140f42640 1 -- 172.21.15.28:0/1160177507 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff1280a2d00 msgr2=0x7ff1280a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff140f42640 1 --2- 172.21.15.28:0/1160177507 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff1280a2d00 0x7ff1280a51c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff13c069b70 tx=0x7ff13400aba0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff140f42640 1 -- 172.21.15.28:0/1160177507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff13c100520 msgr2=0x7ff13c19a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.930+0000 7ff140f42640 1 --2- 172.21.15.28:0/1160177507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff13c100520 0x7ff13c19a170 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff12c005ab0 tx=0x7ff12c0042f0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.931+0000 7ff140f42640 1 -- 172.21.15.28:0/1160177507 shutdown_connections 2024-04-25T01:36:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.931+0000 7ff140f42640 1 --2- 172.21.15.28:0/1160177507 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff1280a2d00 0x7ff1280a51c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.931+0000 7ff140f42640 1 --2- 172.21.15.28:0/1160177507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff13c10e0d0 0x7ff13c19ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.931+0000 7ff140f42640 1 --2- 172.21.15.28:0/1160177507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff13c100e40 0x7ff13c19a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.931+0000 7ff140f42640 1 --2- 172.21.15.28:0/1160177507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff13c100520 0x7ff13c19a170 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.931+0000 7ff140f42640 1 -- 172.21.15.28:0/1160177507 >> 172.21.15.28:0/1160177507 conn(0x7ff13c0769c0 msgr2=0x7ff13c10f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.931+0000 7ff140f42640 1 -- 172.21.15.28:0/1160177507 shutdown_connections 2024-04-25T01:36:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:58.931+0000 7ff140f42640 1 -- 172.21.15.28:0/1160177507 wait complete. 2024-04-25T01:36:58.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:36:59.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.046+0000 7f6e85dee640 1 -- 172.21.15.28:0/1420952548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e80100990 msgr2=0x7f6e80100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:59.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.046+0000 7f6e85dee640 1 --2- 172.21.15.28:0/1420952548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e80100990 0x7f6e80100df0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f6e74009900 tx=0x7f6e7402a410 comp rx=0 tx=0).stop 2024-04-25T01:36:59.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.046+0000 7f6e85dee640 1 -- 172.21.15.28:0/1420952548 shutdown_connections 2024-04-25T01:36:59.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.046+0000 7f6e85dee640 1 --2- 172.21.15.28:0/1420952548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e80101330 0x7f6e8010d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.046+0000 7f6e85dee640 1 --2- 172.21.15.28:0/1420952548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e80100990 0x7f6e80100df0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.046+0000 7f6e85dee640 1 --2- 172.21.15.28:0/1420952548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e80106690 0x7f6e80106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.046+0000 7f6e85dee640 1 -- 172.21.15.28:0/1420952548 >> 172.21.15.28:0/1420952548 conn(0x7f6e80076440 msgr2=0x7f6e80076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:59.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.046+0000 7f6e85dee640 1 -- 172.21.15.28:0/1420952548 shutdown_connections 2024-04-25T01:36:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.046+0000 7f6e85dee640 1 -- 172.21.15.28:0/1420952548 wait complete. 2024-04-25T01:36:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.047+0000 7f6e85dee640 1 Processor -- start 2024-04-25T01:36:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.047+0000 7f6e85dee640 1 -- start start 2024-04-25T01:36:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.047+0000 7f6e85dee640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e80100990 0x7f6e80198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.047+0000 7f6e85dee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e80101330 0x7f6e80199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.047+0000 7f6e85dee640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e80106690 0x7f6e801a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.047+0000 7f6e85dee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6e801109a0 con 0x7f6e80101330 2024-04-25T01:36:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.047+0000 7f6e85dee640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6e80110820 con 0x7f6e80100990 2024-04-25T01:36:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.047+0000 7f6e85dee640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6e80110b20 con 0x7f6e80106690 2024-04-25T01:36:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.048+0000 7f6e7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e80101330 0x7f6e80199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.048+0000 7f6e7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e80101330 0x7f6e80199360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49792/0 (socket says 172.21.15.28:49792) 2024-04-25T01:36:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.048+0000 7f6e7effd640 1 -- 172.21.15.28:0/1263960456 learned_addr learned my addr 172.21.15.28:0/1263960456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.048+0000 7f6e7f7fe640 1 --2- 172.21.15.28:0/1263960456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e80100990 0x7f6e80198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.048+0000 7f6e7ffff640 1 --2- 172.21.15.28:0/1263960456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e80106690 0x7f6e801a3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.048+0000 7f6e7effd640 1 -- 172.21.15.28:0/1263960456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e80106690 msgr2=0x7f6e801a3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.048+0000 7f6e7effd640 1 --2- 172.21.15.28:0/1263960456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e80106690 0x7f6e801a3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.048+0000 7f6e7effd640 1 -- 172.21.15.28:0/1263960456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e80100990 msgr2=0x7f6e80198e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.048+0000 7f6e7effd640 1 --2- 172.21.15.28:0/1263960456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e80100990 0x7f6e80198e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.048+0000 7f6e7effd640 1 -- 172.21.15.28:0/1263960456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e801a43e0 con 0x7f6e80101330 2024-04-25T01:36:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.048+0000 7f6e7effd640 1 --2- 172.21.15.28:0/1263960456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e80101330 0x7f6e80199360 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f6e74006010 tx=0x7f6e740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.048+0000 7f6e7cff9640 1 -- 172.21.15.28:0/1263960456 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7403f030 con 0x7f6e80101330 2024-04-25T01:36:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.048+0000 7f6e7cff9640 1 -- 172.21.15.28:0/1263960456 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e74003ce0 con 0x7f6e80101330 2024-04-25T01:36:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.048+0000 7f6e85dee640 1 -- 172.21.15.28:0/1263960456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e801a45b0 con 0x7f6e80101330 2024-04-25T01:36:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.048+0000 7f6e7cff9640 1 -- 172.21.15.28:0/1263960456 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e74035070 con 0x7f6e80101330 2024-04-25T01:36:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.048+0000 7f6e85dee640 1 -- 172.21.15.28:0/1263960456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e801a4a10 con 0x7f6e80101330 2024-04-25T01:36:59.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.050+0000 7f6e7cff9640 1 -- 172.21.15.28:0/1263960456 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6e74042030 con 0x7f6e80101330 2024-04-25T01:36:59.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e88bda740 1 Processor -- start 2024-04-25T01:36:59.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e88bda740 1 -- start start 2024-04-25T01:36:59.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e7cff9640 1 --2- 172.21.15.28:0/1263960456 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6e680a2d00 0x7f6e680a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e88bda740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7fa580 0x55dc8e7fa960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e88bda740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 0x55dc8e805030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e88bda740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 0x55dc8e807960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e88bda740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dc8e75f670 con 0x55dc8e7faea0 2024-04-25T01:36:59.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e88bda740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55dc8e707970 con 0x55dc8e7fa580 2024-04-25T01:36:59.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e88bda740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55dc8e75b630 con 0x55dc8e805570 2024-04-25T01:36:59.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e7ffff640 1 --2- 172.21.15.28:0/1263960456 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6e680a2d00 0x7f6e680a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e7cff9640 1 -- 172.21.15.28:0/1263960456 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f6e740efdd0 con 0x7f6e80101330 2024-04-25T01:36:59.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 0x55dc8e805030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 0x55dc8e805030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49804/0 (socket says 172.21.15.28:49804) 2024-04-25T01:36:59.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e7effd640 1 -- 172.21.15.28:0/1848464828 learned_addr learned my addr 172.21.15.28:0/1848464828 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:59.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e7f7fe640 1 --2- 172.21.15.28:0/1848464828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7fa580 0x55dc8e7fa960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e7ffff640 1 --2- 172.21.15.28:0/1848464828 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 0x55dc8e807960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e7effd640 1 -- 172.21.15.28:0/1848464828 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 msgr2=0x55dc8e807960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:59.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e7effd640 1 --2- 172.21.15.28:0/1848464828 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 0x55dc8e807960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e7effd640 1 -- 172.21.15.28:0/1848464828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7fa580 msgr2=0x55dc8e7fa960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.051+0000 7f6e7effd640 1 --2- 172.21.15.28:0/1848464828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7fa580 0x55dc8e7fa960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.052+0000 7f6e7effd640 1 -- 172.21.15.28:0/1848464828 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc8e72b400 con 0x55dc8e7faea0 2024-04-25T01:36:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.052+0000 7f6e7ffff640 1 --2- 172.21.15.28:0/1848464828 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 0x55dc8e807960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:36:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.052+0000 7f6e7effd640 1 --2- 172.21.15.28:0/1848464828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 0x55dc8e805030 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f6e740c4750 tx=0x7f6e74003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10f1fb3eb9c0f6f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.052+0000 7f6e7ffff640 1 --2- 172.21.15.28:0/1263960456 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6e680a2d00 0x7f6e680a51c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x55dc8e7f7a30 tx=0x7f6e780098c0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.052+0000 7f6e65ffb640 1 -- 172.21.15.28:0/1848464828 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e74049070 con 0x55dc8e7faea0 2024-04-25T01:36:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.052+0000 7f6e65ffb640 1 -- 172.21.15.28:0/1848464828 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e740bc710 con 0x55dc8e7faea0 2024-04-25T01:36:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.052+0000 7f6e65ffb640 1 -- 172.21.15.28:0/1848464828 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e740c4c80 con 0x55dc8e7faea0 2024-04-25T01:36:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.052+0000 7f6e88bda740 1 -- 172.21.15.28:0/1848464828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 msgr2=0x55dc8e805030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.052+0000 7f6e88bda740 1 --2- 172.21.15.28:0/1848464828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 0x55dc8e805030 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f6e740c4750 tx=0x7f6e74003f10 comp rx=0 tx=0).stop 2024-04-25T01:36:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.053+0000 7f6e88bda740 1 -- 172.21.15.28:0/1848464828 shutdown_connections 2024-04-25T01:36:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.053+0000 7f6e88bda740 1 --2- 172.21.15.28:0/1848464828 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 0x55dc8e807960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.053+0000 7f6e88bda740 1 --2- 172.21.15.28:0/1848464828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 0x55dc8e805030 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.053+0000 7f6e88bda740 1 --2- 172.21.15.28:0/1848464828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7fa580 0x55dc8e7fa960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.053+0000 7f6e88bda740 1 -- 172.21.15.28:0/1848464828 >> 172.21.15.28:0/1848464828 conn(0x55dc8e7f68a0 msgr2=0x55dc8e7f6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.053+0000 7f6e88bda740 1 -- 172.21.15.28:0/1848464828 shutdown_connections 2024-04-25T01:36:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.053+0000 7f6e88bda740 1 -- 172.21.15.28:0/1848464828 wait complete. 2024-04-25T01:36:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.053+0000 7f6e88bda740 1 -- start start 2024-04-25T01:36:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.053+0000 7f6e88bda740 1 Processor -- start 2024-04-25T01:36:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.053+0000 7f6e88bda740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7fa580 0x55dc8e804ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.053+0000 7f6e88bda740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 0x55dc8e8021b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.053+0000 7f6e88bda740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 0x55dc8e802710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.053+0000 7f6e88bda740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dc8e808120 con 0x55dc8e7faea0 2024-04-25T01:36:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.053+0000 7f6e88bda740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55dc8e71b050 con 0x55dc8e7fa580 2024-04-25T01:36:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.053+0000 7f6e88bda740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55dc8e8082a0 con 0x55dc8e805570 2024-04-25T01:36:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.054+0000 7f6e88bda740 10 client.0 _ll_register_callbacks cb 0x55dc8e472be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.054+0000 7f6e7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 0x55dc8e8021b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.054+0000 7f6e7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 0x55dc8e8021b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49806/0 (socket says 172.21.15.28:49806) 2024-04-25T01:36:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.054+0000 7f6e7f7fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7fa580 0x55dc8e804ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.054+0000 7f6e7effd640 1 -- 172.21.15.28:0/1369857102 learned_addr learned my addr 172.21.15.28:0/1369857102 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.054+0000 7f6e7ffff640 1 --2- 172.21.15.28:0/1369857102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 0x55dc8e802710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.054+0000 7f6e7effd640 1 -- 172.21.15.28:0/1369857102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 msgr2=0x55dc8e802710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.054+0000 7f6e7effd640 1 --2- 172.21.15.28:0/1369857102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 0x55dc8e802710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.054+0000 7f6e7effd640 1 -- 172.21.15.28:0/1369857102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7fa580 msgr2=0x55dc8e804ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.054+0000 7f6e7effd640 1 --2- 172.21.15.28:0/1369857102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7fa580 0x55dc8e804ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.054+0000 7f6e7effd640 1 -- 172.21.15.28:0/1369857102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc8e6ecdc0 con 0x55dc8e7faea0 2024-04-25T01:36:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.054+0000 7f6e7effd640 1 --2- 172.21.15.28:0/1369857102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 0x55dc8e8021b0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f6e740b89e0 tx=0x7f6e740bbec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71f5f7954ecccedb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.054+0000 7f6e5effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.054+0000 7f6e5effd640 1 -- 172.21.15.28:0/1369857102 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e74049050 con 0x55dc8e7faea0 2024-04-25T01:36:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.054+0000 7f6e5effd640 1 -- 172.21.15.28:0/1369857102 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f6e740b8680 con 0x55dc8e7faea0 2024-04-25T01:36:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.054+0000 7f6e88bda740 1 -- 172.21.15.28:0/1369857102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e740bc710 con 0x55dc8e7faea0 2024-04-25T01:36:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.054+0000 7f6e5effd640 1 -- 172.21.15.28:0/1369857102 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6e740bb970 con 0x55dc8e7faea0 2024-04-25T01:36:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.054+0000 7f6e88bda740 10 client.16920 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.054+0000 7f6e88bda740 1 -- 172.21.15.28:0/1369857102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dc8e4733f0 con 0x55dc8e7faea0 2024-04-25T01:36:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.055+0000 7f6e5effd640 1 -- 172.21.15.28:0/1369857102 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e740bc710 con 0x55dc8e7faea0 2024-04-25T01:36:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.055+0000 7f6e5effd640 1 -- 172.21.15.28:0/1369857102 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6e740a5120 con 0x55dc8e7faea0 2024-04-25T01:36:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.055+0000 7f6e88bda740 10 client.16920 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.055+0000 7f6e88bda740 10 client.16920 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:36:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.055+0000 7f6e88bda740 20 client.16920 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.055+0000 7f6e88bda740 1 --2- 172.21.15.28:0/1369857102 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55dc8e8150b0 0x55dc8e8174a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.055+0000 7f6e88bda740 4 client.16920 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.055+0000 7f6e88bda740 1 -- 172.21.15.28:0/1369857102 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc8e76cbb0 con 0x55dc8e8150b0 2024-04-25T01:36:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.055+0000 7f6e7f7fe640 1 --2- 172.21.15.28:0/1369857102 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55dc8e8150b0 0x55dc8e8174a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.055+0000 7f6e7f7fe640 1 --2- 172.21.15.28:0/1369857102 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55dc8e8150b0 0x55dc8e8174a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9424ac2096b0c665 server_cookie=355cb3d25cd16c62 in_seq=0 out_seq=0 2024-04-25T01:36:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.055+0000 7f6e5effd640 10 client.16920 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:36:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.056+0000 7f6e5effd640 1 -- 172.21.15.28:0/1369857102 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dc8e76cbb0 con 0x55dc8e8150b0 2024-04-25T01:36:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.056+0000 7f6e5effd640 10 client.16920 handle_command_reply: tid=0 2024-04-25T01:36:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.056+0000 7f6e88bda740 1 client.16920 shutdown 2024-04-25T01:36:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.056+0000 7f6e88bda740 1 -- 172.21.15.28:0/1369857102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 msgr2=0x55dc8e8021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.056+0000 7f6e88bda740 1 --2- 172.21.15.28:0/1369857102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 0x55dc8e8021b0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f6e740b89e0 tx=0x7f6e740bbec0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.056+0000 7f6e88bda740 1 -- 172.21.15.28:0/1369857102 shutdown_connections 2024-04-25T01:36:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.056+0000 7f6e88bda740 1 --2- 172.21.15.28:0/1369857102 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55dc8e8150b0 0x55dc8e8174a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.056+0000 7f6e88bda740 1 --2- 172.21.15.28:0/1369857102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 0x55dc8e802710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.056+0000 7f6e88bda740 1 --2- 172.21.15.28:0/1369857102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 0x55dc8e8021b0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.056+0000 7f6e5effd640 0 client.16920 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:36:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.056+0000 7f6e88bda740 1 --2- 172.21.15.28:0/1369857102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7fa580 0x55dc8e804ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.056+0000 7f6e88bda740 1 -- 172.21.15.28:0/1369857102 >> 172.21.15.28:0/1369857102 conn(0x55dc8e7f68a0 msgr2=0x55dc8e7f5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.056+0000 7f6e88bda740 1 -- 172.21.15.28:0/1369857102 shutdown_connections 2024-04-25T01:36:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.057+0000 7f6e88bda740 1 -- 172.21.15.28:0/1369857102 wait complete. 2024-04-25T01:36:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.057+0000 7f6e88bda740 20 client.16920 trim_cache size 0 max 16384 2024-04-25T01:36:59.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.070+0000 7f6e88bda740 1 Processor -- start 2024-04-25T01:36:59.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.070+0000 7f6e88bda740 1 -- start start 2024-04-25T01:36:59.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.070+0000 7f6e88bda740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7fa580 0x55dc8e810f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.070+0000 7f6e88bda740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7faea0 0x55dc8e81cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.070+0000 7f6e88bda740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 0x55dc8e81d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.070+0000 7f6e88bda740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6e801aea80 con 0x55dc8e7fa580 2024-04-25T01:36:59.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.070+0000 7f6e88bda740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6e801b05c0 con 0x55dc8e7faea0 2024-04-25T01:36:59.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.070+0000 7f6e88bda740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6e801af150 con 0x55dc8e805570 2024-04-25T01:36:59.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.070+0000 7f6e7f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7fa580 0x55dc8e810f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.070+0000 7f6e7f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7fa580 0x55dc8e810f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49812/0 (socket says 172.21.15.28:49812) 2024-04-25T01:36:59.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.070+0000 7f6e7f7fe640 1 -- 172.21.15.28:0/2897412584 learned_addr learned my addr 172.21.15.28:0/2897412584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:59.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.070+0000 7f6e7f7fe640 1 -- 172.21.15.28:0/2897412584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 msgr2=0x55dc8e81d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:59.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.070+0000 7f6e7f7fe640 1 --2- 172.21.15.28:0/2897412584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 0x55dc8e81d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.070+0000 7f6e7f7fe640 1 -- 172.21.15.28:0/2897412584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7faea0 msgr2=0x55dc8e81cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.070+0000 7f6e7effd640 1 --2- 172.21.15.28:0/2897412584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7faea0 0x55dc8e81cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.070+0000 7f6e7f7fe640 1 --2- 172.21.15.28:0/2897412584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7faea0 0x55dc8e81cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.070+0000 7f6e7f7fe640 1 -- 172.21.15.28:0/2897412584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc8e53d7d0 con 0x55dc8e7fa580 2024-04-25T01:36:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.071+0000 7f6e7f7fe640 1 --2- 172.21.15.28:0/2897412584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7fa580 0x55dc8e810f10 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f6e70009e00 tx=0x7f6e70005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f35c889cb2c609f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.071+0000 7f6e657fa640 1 -- 172.21.15.28:0/2897412584 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e700200f0 con 0x55dc8e7fa580 2024-04-25T01:36:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.071+0000 7f6e657fa640 1 -- 172.21.15.28:0/2897412584 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e70020770 con 0x55dc8e7fa580 2024-04-25T01:36:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.071+0000 7f6e657fa640 1 -- 172.21.15.28:0/2897412584 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e70010c00 con 0x55dc8e7fa580 2024-04-25T01:36:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.071+0000 7f6e88bda740 1 -- 172.21.15.28:0/2897412584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7fa580 msgr2=0x55dc8e810f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.071+0000 7f6e88bda740 1 --2- 172.21.15.28:0/2897412584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7fa580 0x55dc8e810f10 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f6e70009e00 tx=0x7f6e70005f90 comp rx=0 tx=0).stop 2024-04-25T01:36:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.071+0000 7f6e88bda740 1 -- 172.21.15.28:0/2897412584 shutdown_connections 2024-04-25T01:36:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.071+0000 7f6e88bda740 1 --2- 172.21.15.28:0/2897412584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 0x55dc8e81d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.071+0000 7f6e88bda740 1 --2- 172.21.15.28:0/2897412584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7faea0 0x55dc8e81cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.071+0000 7f6e88bda740 1 --2- 172.21.15.28:0/2897412584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7fa580 0x55dc8e810f10 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.071+0000 7f6e88bda740 1 -- 172.21.15.28:0/2897412584 >> 172.21.15.28:0/2897412584 conn(0x55dc8e7f68a0 msgr2=0x55dc8e7f5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.071+0000 7f6e88bda740 1 -- 172.21.15.28:0/2897412584 shutdown_connections 2024-04-25T01:36:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.071+0000 7f6e88bda740 1 -- 172.21.15.28:0/2897412584 wait complete. 2024-04-25T01:36:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.072+0000 7f6e88bda740 1 -- start start 2024-04-25T01:36:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.072+0000 7f6e88bda740 1 Processor -- start 2024-04-25T01:36:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.072+0000 7f6e88bda740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7fa580 0x55dc8e80b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.072+0000 7f6e88bda740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 0x55dc8e80b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.072+0000 7f6e88bda740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 0x55dc8e80bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.072+0000 7f6e88bda740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6e801afef0 con 0x55dc8e7faea0 2024-04-25T01:36:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.072+0000 7f6e88bda740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6e801af820 con 0x55dc8e7fa580 2024-04-25T01:36:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.072+0000 7f6e88bda740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6e801aca60 con 0x55dc8e805570 2024-04-25T01:36:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.072+0000 7f6e88bda740 10 client.0 _ll_register_callbacks cb 0x7f6e700097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.072+0000 7f6e7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 0x55dc8e80b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.072+0000 7f6e7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 0x55dc8e80b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49826/0 (socket says 172.21.15.28:49826) 2024-04-25T01:36:59.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.072+0000 7f6e7effd640 1 -- 172.21.15.28:0/2191337840 learned_addr learned my addr 172.21.15.28:0/2191337840 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.072+0000 7f6e7f7fe640 1 --2- 172.21.15.28:0/2191337840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7fa580 0x55dc8e80b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.072+0000 7f6e7ffff640 1 --2- 172.21.15.28:0/2191337840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 0x55dc8e80bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e7effd640 1 -- 172.21.15.28:0/2191337840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 msgr2=0x55dc8e80bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e7effd640 1 --2- 172.21.15.28:0/2191337840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 0x55dc8e80bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e7effd640 1 -- 172.21.15.28:0/2191337840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7fa580 msgr2=0x55dc8e80b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e7effd640 1 --2- 172.21.15.28:0/2191337840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7fa580 0x55dc8e80b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e7effd640 1 -- 172.21.15.28:0/2191337840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc8e87a820 con 0x55dc8e7faea0 2024-04-25T01:36:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e7ffff640 1 --2- 172.21.15.28:0/2191337840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 0x55dc8e80bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:36:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e7effd640 1 --2- 172.21.15.28:0/2191337840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 0x55dc8e80b610 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f6e74005ab0 tx=0x7f6e740b82b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10152e1c7cb83be3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e5effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e5effd640 1 -- 172.21.15.28:0/2191337840 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e740bbb50 con 0x55dc8e7faea0 2024-04-25T01:36:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e5effd640 1 -- 172.21.15.28:0/2191337840 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f6e740b8680 con 0x55dc8e7faea0 2024-04-25T01:36:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e88bda740 1 -- 172.21.15.28:0/2191337840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e70020770 con 0x55dc8e7faea0 2024-04-25T01:36:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e5effd640 1 -- 172.21.15.28:0/2191337840 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6e74049070 con 0x55dc8e7faea0 2024-04-25T01:36:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e88bda740 10 client.16926 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e88bda740 1 -- 172.21.15.28:0/2191337840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dc8e87a820 con 0x55dc8e7faea0 2024-04-25T01:36:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e5effd640 1 -- 172.21.15.28:0/2191337840 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e70020770 con 0x55dc8e7faea0 2024-04-25T01:36:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e5effd640 1 -- 172.21.15.28:0/2191337840 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6e740ed400 con 0x55dc8e7faea0 2024-04-25T01:36:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e88bda740 10 client.16926 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e88bda740 10 client.16926 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:36:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e88bda740 20 client.16926 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e88bda740 1 --2- 172.21.15.28:0/2191337840 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55dc8e87bf70 0x55dc8e87e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e88bda740 4 client.16926 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.073+0000 7f6e88bda740 1 -- 172.21.15.28:0/2191337840 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dc8e766f00 con 0x55dc8e87bf70 2024-04-25T01:36:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.074+0000 7f6e7f7fe640 1 --2- 172.21.15.28:0/2191337840 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55dc8e87bf70 0x55dc8e87e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.074+0000 7f6e7f7fe640 1 --2- 172.21.15.28:0/2191337840 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55dc8e87bf70 0x55dc8e87e390 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1bf1e695dcebc7ad server_cookie=48e921bbdfa845bb in_seq=0 out_seq=0 2024-04-25T01:36:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.074+0000 7f6e5effd640 10 client.16926 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:36:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.074+0000 7f6e5effd640 1 -- 172.21.15.28:0/2191337840 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dc8e766f00 con 0x55dc8e87bf70 2024-04-25T01:36:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.074+0000 7f6e5effd640 10 client.16926 handle_command_reply: tid=0 2024-04-25T01:36:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.074+0000 7f6e88bda740 1 client.16926 shutdown 2024-04-25T01:36:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e88bda740 1 -- 172.21.15.28:0/2191337840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 msgr2=0x55dc8e80b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e88bda740 1 --2- 172.21.15.28:0/2191337840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 0x55dc8e80b610 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f6e74005ab0 tx=0x7f6e740b82b0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e88bda740 1 -- 172.21.15.28:0/2191337840 shutdown_connections 2024-04-25T01:36:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e88bda740 1 --2- 172.21.15.28:0/2191337840 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55dc8e87bf70 0x55dc8e87e390 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e88bda740 1 --2- 172.21.15.28:0/2191337840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55dc8e805570 0x55dc8e80bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e88bda740 1 --2- 172.21.15.28:0/2191337840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc8e7faea0 0x55dc8e80b610 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e5effd640 0 client.16926 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:36:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e88bda740 1 --2- 172.21.15.28:0/2191337840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55dc8e7fa580 0x55dc8e80b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e88bda740 1 -- 172.21.15.28:0/2191337840 >> 172.21.15.28:0/2191337840 conn(0x55dc8e7f68a0 msgr2=0x55dc8e80ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e88bda740 1 -- 172.21.15.28:0/2191337840 shutdown_connections 2024-04-25T01:36:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e88bda740 1 -- 172.21.15.28:0/2191337840 wait complete. 2024-04-25T01:36:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e88bda740 20 client.16926 trim_cache size 0 max 16384 2024-04-25T01:36:59.058 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:36:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e85dee640 1 -- 172.21.15.28:0/1263960456 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6e680a2d00 msgr2=0x7f6e680a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e85dee640 1 --2- 172.21.15.28:0/1263960456 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6e680a2d00 0x7f6e680a51c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x55dc8e7f7a30 tx=0x7f6e780098c0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e85dee640 1 -- 172.21.15.28:0/1263960456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e80101330 msgr2=0x7f6e80199360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e85dee640 1 --2- 172.21.15.28:0/1263960456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e80101330 0x7f6e80199360 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f6e74006010 tx=0x7f6e740042f0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e85dee640 1 -- 172.21.15.28:0/1263960456 shutdown_connections 2024-04-25T01:36:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e85dee640 1 --2- 172.21.15.28:0/1263960456 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6e680a2d00 0x7f6e680a51c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e85dee640 1 --2- 172.21.15.28:0/1263960456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6e80106690 0x7f6e801a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e85dee640 1 --2- 172.21.15.28:0/1263960456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e80101330 0x7f6e80199360 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e85dee640 1 --2- 172.21.15.28:0/1263960456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6e80100990 0x7f6e80198e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e85dee640 1 -- 172.21.15.28:0/1263960456 >> 172.21.15.28:0/1263960456 conn(0x7f6e80076440 msgr2=0x7f6e800ff8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e85dee640 1 -- 172.21.15.28:0/1263960456 shutdown_connections 2024-04-25T01:36:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.075+0000 7f6e85dee640 1 -- 172.21.15.28:0/1263960456 wait complete. 2024-04-25T01:36:59.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:36:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.190+0000 7fcf8f61c640 1 -- 172.21.15.28:0/1194892135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf88100620 msgr2=0x7fcf88100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.190+0000 7fcf8f61c640 1 --2- 172.21.15.28:0/1194892135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf88100620 0x7fcf88100aa0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009930 tx=0x7fcf7802a420 comp rx=0 tx=0).stop 2024-04-25T01:36:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.190+0000 7fcf8f61c640 1 -- 172.21.15.28:0/1194892135 shutdown_connections 2024-04-25T01:36:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.190+0000 7fcf8f61c640 1 --2- 172.21.15.28:0/1194892135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf88100fe0 0x7fcf8810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.190+0000 7fcf8f61c640 1 --2- 172.21.15.28:0/1194892135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf88100620 0x7fcf88100aa0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.190+0000 7fcf8f61c640 1 --2- 172.21.15.28:0/1194892135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf88106630 0x7fcf88106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.190+0000 7fcf8f61c640 1 -- 172.21.15.28:0/1194892135 >> 172.21.15.28:0/1194892135 conn(0x7fcf880776a0 msgr2=0x7fcf88077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.190+0000 7fcf8f61c640 1 -- 172.21.15.28:0/1194892135 shutdown_connections 2024-04-25T01:36:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.190+0000 7fcf8f61c640 1 -- 172.21.15.28:0/1194892135 wait complete. 2024-04-25T01:36:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.191+0000 7fcf8f61c640 1 Processor -- start 2024-04-25T01:36:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.191+0000 7fcf8f61c640 1 -- start start 2024-04-25T01:36:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.191+0000 7fcf8f61c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf88100620 0x7fcf88198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.191+0000 7fcf8f61c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf88100fe0 0x7fcf88198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.191+0000 7fcf8f61c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf88106630 0x7fcf8819c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.191+0000 7fcf8f61c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf88111b40 con 0x7fcf88100620 2024-04-25T01:36:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.191+0000 7fcf8f61c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcf881119c0 con 0x7fcf88100fe0 2024-04-25T01:36:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.191+0000 7fcf8f61c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fcf88111cc0 con 0x7fcf88106630 2024-04-25T01:36:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.191+0000 7fcf8d3b8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf88100620 0x7fcf88198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.191+0000 7fcf8d3b8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf88100620 0x7fcf88198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49844/0 (socket says 172.21.15.28:49844) 2024-04-25T01:36:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.191+0000 7fcf8d3b8640 1 -- 172.21.15.28:0/2359940471 learned_addr learned my addr 172.21.15.28:0/2359940471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.191+0000 7fcf8cbb7640 1 --2- 172.21.15.28:0/2359940471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf88100fe0 0x7fcf88198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.191+0000 7fcf8d3b8640 1 -- 172.21.15.28:0/2359940471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf88106630 msgr2=0x7fcf8819c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.191+0000 7fcf8d3b8640 1 --2- 172.21.15.28:0/2359940471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf88106630 0x7fcf8819c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.191+0000 7fcf8d3b8640 1 -- 172.21.15.28:0/2359940471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf88100fe0 msgr2=0x7fcf88198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.191+0000 7fcf8d3b8640 1 --2- 172.21.15.28:0/2359940471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf88100fe0 0x7fcf88198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.191+0000 7fcf8d3b8640 1 -- 172.21.15.28:0/2359940471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf8819d030 con 0x7fcf88100620 2024-04-25T01:36:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.191+0000 7fcf8d3b8640 1 --2- 172.21.15.28:0/2359940471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf88100620 0x7fcf88198040 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fcf8400cce0 tx=0x7fcf8400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.192+0000 7fcf7e7fc640 1 -- 172.21.15.28:0/2359940471 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf8400f450 con 0x7fcf88100620 2024-04-25T01:36:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.192+0000 7fcf7e7fc640 1 -- 172.21.15.28:0/2359940471 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf8400fad0 con 0x7fcf88100620 2024-04-25T01:36:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.192+0000 7fcf8f61c640 1 -- 172.21.15.28:0/2359940471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf8819d320 con 0x7fcf88100620 2024-04-25T01:36:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.192+0000 7fcf7e7fc640 1 -- 172.21.15.28:0/2359940471 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf84018ce0 con 0x7fcf88100620 2024-04-25T01:36:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.192+0000 7fcf8f61c640 1 -- 172.21.15.28:0/2359940471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf881a4bb0 con 0x7fcf88100620 2024-04-25T01:36:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.194+0000 7fcf7e7fc640 1 -- 172.21.15.28:0/2359940471 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fcf8400f5f0 con 0x7fcf88100620 2024-04-25T01:36:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.194+0000 7fcf92408740 1 Processor -- start 2024-04-25T01:36:59.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.194+0000 7fcf92408740 1 -- start start 2024-04-25T01:36:59.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf92408740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564db9df580 0x5564db9df960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf92408740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9dfea0 0x5564db9ea030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf92408740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9ea570 0x5564db9ec960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf92408740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5564db944670 con 0x5564db9ea570 2024-04-25T01:36:59.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf92408740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5564db8ec970 con 0x5564db9dfea0 2024-04-25T01:36:59.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf92408740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5564db940630 con 0x5564db9df580 2024-04-25T01:36:59.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf7e7fc640 1 --2- 172.21.15.28:0/2359940471 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcf740a2d00 0x7fcf740a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf8cbb7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9ea570 0x5564db9ec960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf8cbb7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9ea570 0x5564db9ec960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49848/0 (socket says 172.21.15.28:49848) 2024-04-25T01:36:59.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf8cbb7640 1 -- 172.21.15.28:0/250812220 learned_addr learned my addr 172.21.15.28:0/250812220 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:59.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf8d3b8640 1 --2- 172.21.15.28:0/250812220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9dfea0 0x5564db9ea030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf8dbb9640 1 --2- 172.21.15.28:0/2359940471 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcf740a2d00 0x7fcf740a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf8dbb9640 1 --2- 172.21.15.28:0/250812220 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564db9df580 0x5564db9df960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf7e7fc640 1 -- 172.21.15.28:0/2359940471 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fcf840cfd50 con 0x7fcf88100620 2024-04-25T01:36:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf8cbb7640 1 -- 172.21.15.28:0/250812220 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564db9df580 msgr2=0x5564db9df960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf8cbb7640 1 --2- 172.21.15.28:0/250812220 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564db9df580 0x5564db9df960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf8cbb7640 1 -- 172.21.15.28:0/250812220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9dfea0 msgr2=0x5564db9ea030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf8cbb7640 1 --2- 172.21.15.28:0/250812220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9dfea0 0x5564db9ea030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf8cbb7640 1 -- 172.21.15.28:0/250812220 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564db910400 con 0x5564db9ea570 2024-04-25T01:36:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf8cbb7640 1 --2- 172.21.15.28:0/250812220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9ea570 0x5564db9ec960 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fcf7802a940 tx=0x7fcf780316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e8d1c028a210974 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.195+0000 7fcf8dbb9640 1 --2- 172.21.15.28:0/2359940471 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcf740a2d00 0x7fcf740a51c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcf80013690 tx=0x7fcf8000c830 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.196+0000 7fcf737fe640 1 -- 172.21.15.28:0/250812220 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf78040070 con 0x5564db9ea570 2024-04-25T01:36:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.196+0000 7fcf737fe640 1 -- 172.21.15.28:0/250812220 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf78005020 con 0x5564db9ea570 2024-04-25T01:36:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.196+0000 7fcf737fe640 1 -- 172.21.15.28:0/250812220 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf78035040 con 0x5564db9ea570 2024-04-25T01:36:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.196+0000 7fcf92408740 1 -- 172.21.15.28:0/250812220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9ea570 msgr2=0x5564db9ec960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.196+0000 7fcf92408740 1 --2- 172.21.15.28:0/250812220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9ea570 0x5564db9ec960 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fcf7802a940 tx=0x7fcf780316e0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.196+0000 7fcf92408740 1 -- 172.21.15.28:0/250812220 shutdown_connections 2024-04-25T01:36:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.196+0000 7fcf92408740 1 --2- 172.21.15.28:0/250812220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9ea570 0x5564db9ec960 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.196+0000 7fcf92408740 1 --2- 172.21.15.28:0/250812220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9dfea0 0x5564db9ea030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.196+0000 7fcf92408740 1 --2- 172.21.15.28:0/250812220 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564db9df580 0x5564db9df960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.196+0000 7fcf92408740 1 -- 172.21.15.28:0/250812220 >> 172.21.15.28:0/250812220 conn(0x5564db9db8a0 msgr2=0x5564db9dbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.196+0000 7fcf92408740 1 -- 172.21.15.28:0/250812220 shutdown_connections 2024-04-25T01:36:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.196+0000 7fcf92408740 1 -- 172.21.15.28:0/250812220 wait complete. 2024-04-25T01:36:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.197+0000 7fcf92408740 1 -- start start 2024-04-25T01:36:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.197+0000 7fcf92408740 1 Processor -- start 2024-04-25T01:36:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.197+0000 7fcf92408740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9df580 0x5564db9e9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.197+0000 7fcf92408740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564db9dfea0 0x5564db9e70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.197+0000 7fcf92408740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9ea570 0x5564db9e75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.197+0000 7fcf92408740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5564db9ed300 con 0x5564db9ea570 2024-04-25T01:36:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.197+0000 7fcf92408740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5564db900050 con 0x5564db9df580 2024-04-25T01:36:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.197+0000 7fcf92408740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5564db9ed480 con 0x5564db9dfea0 2024-04-25T01:36:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.197+0000 7fcf92408740 10 client.0 _ll_register_callbacks cb 0x5564db657be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.197+0000 7fcf8dbb9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9ea570 0x5564db9e75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.197+0000 7fcf8dbb9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9ea570 0x5564db9e75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49850/0 (socket says 172.21.15.28:49850) 2024-04-25T01:36:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.197+0000 7fcf8dbb9640 1 -- 172.21.15.28:0/1653389807 learned_addr learned my addr 172.21.15.28:0/1653389807 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.198+0000 7fcf8cbb7640 1 --2- 172.21.15.28:0/1653389807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564db9dfea0 0x5564db9e70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.198+0000 7fcf8d3b8640 1 --2- 172.21.15.28:0/1653389807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9df580 0x5564db9e9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.198+0000 7fcf8dbb9640 1 -- 172.21.15.28:0/1653389807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564db9dfea0 msgr2=0x5564db9e70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.198+0000 7fcf8dbb9640 1 --2- 172.21.15.28:0/1653389807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564db9dfea0 0x5564db9e70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.198+0000 7fcf8dbb9640 1 -- 172.21.15.28:0/1653389807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9df580 msgr2=0x5564db9e9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.198+0000 7fcf8dbb9640 1 --2- 172.21.15.28:0/1653389807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9df580 0x5564db9e9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.198+0000 7fcf8dbb9640 1 -- 172.21.15.28:0/1653389807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564db8d1dc0 con 0x5564db9ea570 2024-04-25T01:36:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.198+0000 7fcf8dbb9640 1 --2- 172.21.15.28:0/1653389807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9ea570 0x5564db9e75f0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fcf800117a0 tx=0x7fcf80005a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1899c5c8cca961fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.198+0000 7fcf70ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.198+0000 7fcf70ff9640 1 -- 172.21.15.28:0/1653389807 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf80024070 con 0x5564db9ea570 2024-04-25T01:36:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.198+0000 7fcf70ff9640 1 -- 172.21.15.28:0/1653389807 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fcf80009070 con 0x5564db9ea570 2024-04-25T01:36:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.198+0000 7fcf92408740 1 -- 172.21.15.28:0/1653389807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf78005020 con 0x5564db9ea570 2024-04-25T01:36:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.198+0000 7fcf70ff9640 1 -- 172.21.15.28:0/1653389807 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcf80030050 con 0x5564db9ea570 2024-04-25T01:36:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.198+0000 7fcf92408740 10 client.16938 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.198+0000 7fcf92408740 1 -- 172.21.15.28:0/1653389807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5564db6583f0 con 0x5564db9ea570 2024-04-25T01:36:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.198+0000 7fcf70ff9640 1 -- 172.21.15.28:0/1653389807 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf78005020 con 0x5564db9ea570 2024-04-25T01:36:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.199+0000 7fcf70ff9640 1 -- 172.21.15.28:0/1653389807 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcf8000f040 con 0x5564db9ea570 2024-04-25T01:36:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.199+0000 7fcf92408740 10 client.16938 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.199+0000 7fcf92408740 10 client.16938 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.199+0000 7fcf92408740 20 client.16938 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.199+0000 7fcf92408740 1 --2- 172.21.15.28:0/1653389807 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5564db9fa3e0 0x5564db9fc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.199+0000 7fcf92408740 4 client.16938 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:36:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.199+0000 7fcf92408740 1 -- 172.21.15.28:0/1653389807 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564db951bb0 con 0x5564db9fa3e0 2024-04-25T01:36:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.199+0000 7fcf8d3b8640 1 --2- 172.21.15.28:0/1653389807 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5564db9fa3e0 0x5564db9fc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.199+0000 7fcf8d3b8640 1 --2- 172.21.15.28:0/1653389807 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5564db9fa3e0 0x5564db9fc830 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce2a35d4b1c2b0e3 server_cookie=1dd38643dcdbc78 in_seq=0 out_seq=0 2024-04-25T01:36:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.199+0000 7fcf70ff9640 10 client.16938 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.200+0000 7fcf70ff9640 1 -- 172.21.15.28:0/1653389807 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5564db951bb0 con 0x5564db9fa3e0 2024-04-25T01:36:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.200+0000 7fcf70ff9640 10 client.16938 handle_command_reply: tid=0 2024-04-25T01:36:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.200+0000 7fcf92408740 1 client.16938 shutdown 2024-04-25T01:36:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.200+0000 7fcf92408740 1 -- 172.21.15.28:0/1653389807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9ea570 msgr2=0x5564db9e75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.200+0000 7fcf92408740 1 --2- 172.21.15.28:0/1653389807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9ea570 0x5564db9e75f0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fcf800117a0 tx=0x7fcf80005a80 comp rx=0 tx=0).stop 2024-04-25T01:36:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.200+0000 7fcf92408740 1 -- 172.21.15.28:0/1653389807 shutdown_connections 2024-04-25T01:36:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.200+0000 7fcf92408740 1 --2- 172.21.15.28:0/1653389807 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5564db9fa3e0 0x5564db9fc830 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.200+0000 7fcf92408740 1 --2- 172.21.15.28:0/1653389807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9ea570 0x5564db9e75f0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.200+0000 7fcf92408740 1 --2- 172.21.15.28:0/1653389807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564db9dfea0 0x5564db9e70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.200+0000 7fcf70ff9640 0 client.16938 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.200+0000 7fcf92408740 1 --2- 172.21.15.28:0/1653389807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9df580 0x5564db9e9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.200+0000 7fcf92408740 1 -- 172.21.15.28:0/1653389807 >> 172.21.15.28:0/1653389807 conn(0x5564db9db8a0 msgr2=0x5564db9dab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.200+0000 7fcf92408740 1 -- 172.21.15.28:0/1653389807 shutdown_connections 2024-04-25T01:36:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.200+0000 7fcf92408740 1 -- 172.21.15.28:0/1653389807 wait complete. 2024-04-25T01:36:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.200+0000 7fcf92408740 20 client.16938 trim_cache size 0 max 16384 2024-04-25T01:36:59.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf92408740 1 Processor -- start 2024-04-25T01:36:59.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf92408740 1 -- start start 2024-04-25T01:36:59.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf92408740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9df580 0x5564db9f61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf92408740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564db9dfea0 0x5564dba02100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf92408740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9ea570 0x5564dba02640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf92408740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf881ae9d0 con 0x5564db9df580 2024-04-25T01:36:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf92408740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcf881b0510 con 0x5564db9ea570 2024-04-25T01:36:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf92408740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fcf881af0a0 con 0x5564db9dfea0 2024-04-25T01:36:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf8d3b8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9df580 0x5564db9f61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf8d3b8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9df580 0x5564db9f61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49860/0 (socket says 172.21.15.28:49860) 2024-04-25T01:36:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf8d3b8640 1 -- 172.21.15.28:0/641210424 learned_addr learned my addr 172.21.15.28:0/641210424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf8dbb9640 1 --2- 172.21.15.28:0/641210424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9ea570 0x5564dba02640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf8d3b8640 1 -- 172.21.15.28:0/641210424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564db9dfea0 msgr2=0x5564dba02100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf8d3b8640 1 --2- 172.21.15.28:0/641210424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564db9dfea0 0x5564dba02100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf8d3b8640 1 -- 172.21.15.28:0/641210424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9ea570 msgr2=0x5564dba02640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf8d3b8640 1 --2- 172.21.15.28:0/641210424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9ea570 0x5564dba02640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf8d3b8640 1 -- 172.21.15.28:0/641210424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564db7227d0 con 0x5564db9df580 2024-04-25T01:36:59.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf8d3b8640 1 --2- 172.21.15.28:0/641210424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9df580 0x5564db9f61c0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fcf840988a0 tx=0x7fcf8401b580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd128f0aa692c950 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf72ffd640 1 -- 172.21.15.28:0/641210424 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf8409bed0 con 0x5564db9df580 2024-04-25T01:36:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.214+0000 7fcf72ffd640 1 -- 172.21.15.28:0/641210424 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf840a0280 con 0x5564db9df580 2024-04-25T01:36:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.215+0000 7fcf72ffd640 1 -- 172.21.15.28:0/641210424 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf840cd380 con 0x5564db9df580 2024-04-25T01:36:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.215+0000 7fcf92408740 1 -- 172.21.15.28:0/641210424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9df580 msgr2=0x5564db9f61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.215+0000 7fcf92408740 1 --2- 172.21.15.28:0/641210424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9df580 0x5564db9f61c0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fcf840988a0 tx=0x7fcf8401b580 comp rx=0 tx=0).stop 2024-04-25T01:36:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.215+0000 7fcf92408740 1 -- 172.21.15.28:0/641210424 shutdown_connections 2024-04-25T01:36:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.215+0000 7fcf92408740 1 --2- 172.21.15.28:0/641210424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9ea570 0x5564dba02640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.215+0000 7fcf92408740 1 --2- 172.21.15.28:0/641210424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564db9dfea0 0x5564dba02100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.215+0000 7fcf92408740 1 --2- 172.21.15.28:0/641210424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9df580 0x5564db9f61c0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.215+0000 7fcf92408740 1 -- 172.21.15.28:0/641210424 >> 172.21.15.28:0/641210424 conn(0x5564db9db8a0 msgr2=0x5564db9dab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.215+0000 7fcf92408740 1 -- 172.21.15.28:0/641210424 shutdown_connections 2024-04-25T01:36:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.215+0000 7fcf92408740 1 -- 172.21.15.28:0/641210424 wait complete. 2024-04-25T01:36:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.215+0000 7fcf92408740 1 -- start start 2024-04-25T01:36:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.216+0000 7fcf92408740 1 Processor -- start 2024-04-25T01:36:59.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.216+0000 7fcf92408740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9df580 0x5564db9f02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.216+0000 7fcf92408740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9dfea0 0x5564db9f0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.216+0000 7fcf92408740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564db9ea570 0x5564db9f0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.216+0000 7fcf92408740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf881afe40 con 0x5564db9df580 2024-04-25T01:36:59.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.216+0000 7fcf92408740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcf881af770 con 0x5564db9dfea0 2024-04-25T01:36:59.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.216+0000 7fcf92408740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fcf881ac9b0 con 0x5564db9ea570 2024-04-25T01:36:59.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.216+0000 7fcf92408740 10 client.0 _ll_register_callbacks cb 0x7fcf8000ca60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:36:59.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.216+0000 7fcf8d3b8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9df580 0x5564db9f02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.216+0000 7fcf8d3b8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9df580 0x5564db9f02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49870/0 (socket says 172.21.15.28:49870) 2024-04-25T01:36:59.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.216+0000 7fcf8d3b8640 1 -- 172.21.15.28:0/1095481288 learned_addr learned my addr 172.21.15.28:0/1095481288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:36:59.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.216+0000 7fcf8cbb7640 1 --2- 172.21.15.28:0/1095481288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9dfea0 0x5564db9f0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.216+0000 7fcf8dbb9640 1 --2- 172.21.15.28:0/1095481288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564db9ea570 0x5564db9f0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.216+0000 7fcf8d3b8640 1 -- 172.21.15.28:0/1095481288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564db9ea570 msgr2=0x5564db9f0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:59.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.216+0000 7fcf8d3b8640 1 --2- 172.21.15.28:0/1095481288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564db9ea570 0x5564db9f0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.216+0000 7fcf8d3b8640 1 -- 172.21.15.28:0/1095481288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9dfea0 msgr2=0x5564db9f0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:36:59.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.216+0000 7fcf8d3b8640 1 --2- 172.21.15.28:0/1095481288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9dfea0 0x5564db9f0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.216+0000 7fcf8d3b8640 1 -- 172.21.15.28:0/1095481288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564dba63ef0 con 0x5564db9df580 2024-04-25T01:36:59.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.217+0000 7fcf8d3b8640 1 --2- 172.21.15.28:0/1095481288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9df580 0x5564db9f02c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fcf840515d0 tx=0x7fcf8409c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c12b90bd21b51fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:36:59.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.217+0000 7fcf70ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:36:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.219+0000 7fcf70ff9640 1 -- 172.21.15.28:0/1095481288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf840b7070 con 0x5564db9df580 2024-04-25T01:36:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.219+0000 7fcf70ff9640 1 -- 172.21.15.28:0/1095481288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fcf84098540 con 0x5564db9df580 2024-04-25T01:36:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.219+0000 7fcf92408740 1 -- 172.21.15.28:0/1095481288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf840a0280 con 0x5564db9df580 2024-04-25T01:36:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.219+0000 7fcf70ff9640 1 -- 172.21.15.28:0/1095481288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcf8400f8e0 con 0x5564db9df580 2024-04-25T01:36:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.219+0000 7fcf92408740 10 client.16944 fetch_fsmap learned FSMap version 28 2024-04-25T01:36:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.219+0000 7fcf92408740 1 -- 172.21.15.28:0/1095481288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5564dba651d0 con 0x5564db9df580 2024-04-25T01:36:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.219+0000 7fcf70ff9640 1 -- 172.21.15.28:0/1095481288 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf840a0280 con 0x5564db9df580 2024-04-25T01:36:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.219+0000 7fcf70ff9640 1 -- 172.21.15.28:0/1095481288 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcf8401b120 con 0x5564db9df580 2024-04-25T01:36:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.219+0000 7fcf92408740 10 client.16944 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:36:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.219+0000 7fcf92408740 10 client.16944 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:36:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.219+0000 7fcf92408740 20 client.16944 populate_metadata read hostname 'smithi028' 2024-04-25T01:36:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.219+0000 7fcf92408740 1 --2- 172.21.15.28:0/1095481288 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5564dba658f0 0x5564dba67ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:36:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.219+0000 7fcf92408740 4 client.16944 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:36:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.219+0000 7fcf92408740 1 -- 172.21.15.28:0/1095481288 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5564db94bf00 con 0x5564dba658f0 2024-04-25T01:36:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.219+0000 7fcf8cbb7640 1 --2- 172.21.15.28:0/1095481288 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5564dba658f0 0x5564dba67ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:36:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.220+0000 7fcf8cbb7640 1 --2- 172.21.15.28:0/1095481288 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5564dba658f0 0x5564dba67ce0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e0cb1edceb8eb55 server_cookie=37a86a45451104fb in_seq=0 out_seq=0 2024-04-25T01:36:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.220+0000 7fcf70ff9640 10 client.16944 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.220+0000 7fcf70ff9640 1 -- 172.21.15.28:0/1095481288 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5564db94bf00 con 0x5564dba658f0 2024-04-25T01:36:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.220+0000 7fcf70ff9640 10 client.16944 handle_command_reply: tid=0 2024-04-25T01:36:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.220+0000 7fcf92408740 1 client.16944 shutdown 2024-04-25T01:36:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.220+0000 7fcf92408740 1 -- 172.21.15.28:0/1095481288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9df580 msgr2=0x5564db9f02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.220+0000 7fcf92408740 1 --2- 172.21.15.28:0/1095481288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9df580 0x5564db9f02c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fcf840515d0 tx=0x7fcf8409c010 comp rx=0 tx=0).stop 2024-04-25T01:36:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.220+0000 7fcf92408740 1 -- 172.21.15.28:0/1095481288 shutdown_connections 2024-04-25T01:36:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.220+0000 7fcf92408740 1 --2- 172.21.15.28:0/1095481288 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5564dba658f0 0x5564dba67ce0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.220+0000 7fcf92408740 1 --2- 172.21.15.28:0/1095481288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5564db9ea570 0x5564db9f0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.220+0000 7fcf92408740 1 --2- 172.21.15.28:0/1095481288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5564db9dfea0 0x5564db9f0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.220+0000 7fcf70ff9640 0 client.16944 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:36:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.220+0000 7fcf92408740 1 --2- 172.21.15.28:0/1095481288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564db9df580 0x5564db9f02c0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.220+0000 7fcf92408740 1 -- 172.21.15.28:0/1095481288 >> 172.21.15.28:0/1095481288 conn(0x5564db9db8a0 msgr2=0x5564db9f51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.220+0000 7fcf92408740 1 -- 172.21.15.28:0/1095481288 shutdown_connections 2024-04-25T01:36:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.221+0000 7fcf92408740 1 -- 172.21.15.28:0/1095481288 wait complete. 2024-04-25T01:36:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.221+0000 7fcf92408740 20 client.16944 trim_cache size 0 max 16384 2024-04-25T01:36:59.202 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:36:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.221+0000 7fcf8f61c640 1 -- 172.21.15.28:0/2359940471 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcf740a2d00 msgr2=0x7fcf740a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.221+0000 7fcf8f61c640 1 --2- 172.21.15.28:0/2359940471 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcf740a2d00 0x7fcf740a51c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcf80013690 tx=0x7fcf8000c830 comp rx=0 tx=0).stop 2024-04-25T01:36:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.221+0000 7fcf8f61c640 1 -- 172.21.15.28:0/2359940471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf88100620 msgr2=0x7fcf88198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:36:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.221+0000 7fcf8f61c640 1 --2- 172.21.15.28:0/2359940471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf88100620 0x7fcf88198040 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fcf8400cce0 tx=0x7fcf8400a620 comp rx=0 tx=0).stop 2024-04-25T01:36:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.221+0000 7fcf8f61c640 1 -- 172.21.15.28:0/2359940471 shutdown_connections 2024-04-25T01:36:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.221+0000 7fcf8f61c640 1 --2- 172.21.15.28:0/2359940471 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcf740a2d00 0x7fcf740a51c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.221+0000 7fcf8f61c640 1 --2- 172.21.15.28:0/2359940471 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcf88106630 0x7fcf8819c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.221+0000 7fcf8f61c640 1 --2- 172.21.15.28:0/2359940471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcf88100fe0 0x7fcf88198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.221+0000 7fcf8f61c640 1 --2- 172.21.15.28:0/2359940471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf88100620 0x7fcf88198040 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:36:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.221+0000 7fcf8f61c640 1 -- 172.21.15.28:0/2359940471 >> 172.21.15.28:0/2359940471 conn(0x7fcf880776a0 msgr2=0x7fcf88103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:36:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.221+0000 7fcf8f61c640 1 -- 172.21.15.28:0/2359940471 shutdown_connections 2024-04-25T01:36:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:36:59.221+0000 7fcf8f61c640 1 -- 172.21.15.28:0/2359940471 wait complete. 2024-04-25T01:36:59.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:36:59 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3625749206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:36:59.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:36:59 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3625749206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:36:59.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:36:59 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3625749206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:37:00.209 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:37:00.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:37:00.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:37:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.352+0000 7f1ebdd22640 1 -- 172.21.15.28:0/262097237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb8106630 msgr2=0x7f1eb8106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.352+0000 7f1ebdd22640 1 --2- 172.21.15.28:0/262097237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb8106630 0x7f1eb8106a10 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0009cd0 tx=0x7f1eb002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.352+0000 7f1ebdd22640 1 -- 172.21.15.28:0/262097237 shutdown_connections 2024-04-25T01:37:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.352+0000 7f1ebdd22640 1 --2- 172.21.15.28:0/262097237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eb8100fe0 0x7f1eb810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.352+0000 7f1ebdd22640 1 --2- 172.21.15.28:0/262097237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1eb8100620 0x7f1eb8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.352+0000 7f1ebdd22640 1 --2- 172.21.15.28:0/262097237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb8106630 0x7f1eb8106a10 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.352+0000 7f1ebdd22640 1 -- 172.21.15.28:0/262097237 >> 172.21.15.28:0/262097237 conn(0x7f1eb80776a0 msgr2=0x7f1eb8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.352+0000 7f1ebdd22640 1 -- 172.21.15.28:0/262097237 shutdown_connections 2024-04-25T01:37:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.352+0000 7f1ebdd22640 1 -- 172.21.15.28:0/262097237 wait complete. 2024-04-25T01:37:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.353+0000 7f1ebdd22640 1 Processor -- start 2024-04-25T01:37:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.353+0000 7f1ebdd22640 1 -- start start 2024-04-25T01:37:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.353+0000 7f1ebdd22640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb8100620 0x7f1eb8198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.353+0000 7f1ebdd22640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eb8100fe0 0x7f1eb8199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.353+0000 7f1ebdd22640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1eb8106630 0x7f1eb81a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.353+0000 7f1ebdd22640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb8110a40 con 0x7f1eb8100620 2024-04-25T01:37:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.353+0000 7f1ebdd22640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb81108c0 con 0x7f1eb8100fe0 2024-04-25T01:37:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.353+0000 7f1ebdd22640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb8110bc0 con 0x7f1eb8106630 2024-04-25T01:37:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.354+0000 7f1eb77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb8100620 0x7f1eb8198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.354+0000 7f1eb77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb8100620 0x7f1eb8198e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49898/0 (socket says 172.21.15.28:49898) 2024-04-25T01:37:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.354+0000 7f1eb77fe640 1 -- 172.21.15.28:0/485366520 learned_addr learned my addr 172.21.15.28:0/485366520 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.354+0000 7f1eb6ffd640 1 --2- 172.21.15.28:0/485366520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eb8100fe0 0x7f1eb8199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.354+0000 7f1eb7fff640 1 --2- 172.21.15.28:0/485366520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1eb8106630 0x7f1eb81a3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.354+0000 7f1eb77fe640 1 -- 172.21.15.28:0/485366520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1eb8106630 msgr2=0x7f1eb81a3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.354+0000 7f1eb77fe640 1 --2- 172.21.15.28:0/485366520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1eb8106630 0x7f1eb81a3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.354+0000 7f1eb77fe640 1 -- 172.21.15.28:0/485366520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eb8100fe0 msgr2=0x7f1eb8199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.354+0000 7f1eb77fe640 1 --2- 172.21.15.28:0/485366520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eb8100fe0 0x7f1eb8199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.354+0000 7f1eb77fe640 1 -- 172.21.15.28:0/485366520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb81a43e0 con 0x7f1eb8100620 2024-04-25T01:37:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.354+0000 7f1eb7fff640 1 --2- 172.21.15.28:0/485366520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1eb8106630 0x7f1eb81a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.354+0000 7f1eb77fe640 1 --2- 172.21.15.28:0/485366520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb8100620 0x7f1eb8198e20 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f1eb002ae00 tx=0x7f1eb0003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.354+0000 7f1eb4ff9640 1 -- 172.21.15.28:0/485366520 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0004040 con 0x7f1eb8100620 2024-04-25T01:37:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.354+0000 7f1ebdd22640 1 -- 172.21.15.28:0/485366520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eb81a45b0 con 0x7f1eb8100620 2024-04-25T01:37:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.354+0000 7f1eb4ff9640 1 -- 172.21.15.28:0/485366520 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1eb00046c0 con 0x7f1eb8100620 2024-04-25T01:37:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.354+0000 7f1eb4ff9640 1 -- 172.21.15.28:0/485366520 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0046480 con 0x7f1eb8100620 2024-04-25T01:37:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.354+0000 7f1ebdd22640 1 -- 172.21.15.28:0/485366520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eb81a4ad0 con 0x7f1eb8100620 2024-04-25T01:37:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.356+0000 7f1eb4ff9640 1 -- 172.21.15.28:0/485366520 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1eb00305a0 con 0x7f1eb8100620 2024-04-25T01:37:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.357+0000 7f1ec0a32740 1 Processor -- start 2024-04-25T01:37:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.357+0000 7f1ec0a32740 1 -- start start 2024-04-25T01:37:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.357+0000 7f1eb4ff9640 1 --2- 172.21.15.28:0/485366520 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1ea00a2d00 0x7f1ea00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.357+0000 7f1ec0a32740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56319f46f980 0x56319f46dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.357+0000 7f1ec0a32740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f46e290 0x56319f46e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.357+0000 7f1ec0a32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f47d420 0x56319f47f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.357+0000 7f1ec0a32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56319f3d22c0 con 0x56319f47d420 2024-04-25T01:37:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.357+0000 7f1ec0a32740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56319f37a910 con 0x56319f46e290 2024-04-25T01:37:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.357+0000 7f1ec0a32740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56319f3ce280 con 0x56319f46f980 2024-04-25T01:37:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.357+0000 7f1eb7fff640 1 --2- 172.21.15.28:0/485366520 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1ea00a2d00 0x7f1ea00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.357+0000 7f1eb7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f47d420 0x56319f47f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.357+0000 7f1eb7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f47d420 0x56319f47f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49910/0 (socket says 172.21.15.28:49910) 2024-04-25T01:37:00.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.357+0000 7f1eb7fff640 1 -- 172.21.15.28:0/541554842 learned_addr learned my addr 172.21.15.28:0/541554842 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:00.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.357+0000 7f1eb4ff9640 1 -- 172.21.15.28:0/485366520 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1eb002e030 con 0x7f1eb8100620 2024-04-25T01:37:00.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.358+0000 7f1eb77fe640 1 --2- 172.21.15.28:0/541554842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56319f46f980 0x56319f46dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.358+0000 7f1eb6ffd640 1 --2- 172.21.15.28:0/541554842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f46e290 0x56319f46e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.358+0000 7f1eb7fff640 1 -- 172.21.15.28:0/541554842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56319f46f980 msgr2=0x56319f46dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:00.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.358+0000 7f1eb7fff640 1 --2- 172.21.15.28:0/541554842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56319f46f980 0x56319f46dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.358+0000 7f1eb7fff640 1 -- 172.21.15.28:0/541554842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f46e290 msgr2=0x56319f46e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:00.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.358+0000 7f1eb7fff640 1 --2- 172.21.15.28:0/541554842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f46e290 0x56319f46e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.358+0000 7f1eb7fff640 1 -- 172.21.15.28:0/541554842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56319f38bfb0 con 0x56319f47d420 2024-04-25T01:37:00.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.358+0000 7f1eb7fff640 1 --2- 172.21.15.28:0/485366520 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1ea00a2d00 0x7f1ea00a51c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1ea80137e0 tx=0x7f1ea800ba20 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:00.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.358+0000 7f1eb7fff640 1 --2- 172.21.15.28:0/541554842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f47d420 0x56319f47f810 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8004640 tx=0x7f1ea801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d1670e4244399d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:00.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.358+0000 7f1e9dffb640 1 -- 172.21.15.28:0/541554842 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea800e760 con 0x56319f47d420 2024-04-25T01:37:00.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.358+0000 7f1e9dffb640 1 -- 172.21.15.28:0/541554842 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ea800ede0 con 0x56319f47d420 2024-04-25T01:37:00.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.358+0000 7f1e9dffb640 1 -- 172.21.15.28:0/541554842 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea801fe10 con 0x56319f47d420 2024-04-25T01:37:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.359+0000 7f1ec0a32740 1 -- 172.21.15.28:0/541554842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f47d420 msgr2=0x56319f47f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.359+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/541554842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f47d420 0x56319f47f810 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8004640 tx=0x7f1ea801a000 comp rx=0 tx=0).stop 2024-04-25T01:37:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.359+0000 7f1ec0a32740 1 -- 172.21.15.28:0/541554842 shutdown_connections 2024-04-25T01:37:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.359+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/541554842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f47d420 0x56319f47f810 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.359+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/541554842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f46e290 0x56319f46e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.359+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/541554842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56319f46f980 0x56319f46dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.359+0000 7f1ec0a32740 1 -- 172.21.15.28:0/541554842 >> 172.21.15.28:0/541554842 conn(0x56319f436950 msgr2=0x56319f46a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.359+0000 7f1ec0a32740 1 -- 172.21.15.28:0/541554842 shutdown_connections 2024-04-25T01:37:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.359+0000 7f1ec0a32740 1 -- 172.21.15.28:0/541554842 wait complete. 2024-04-25T01:37:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.359+0000 7f1ec0a32740 1 -- start start 2024-04-25T01:37:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1ec0a32740 1 Processor -- start 2024-04-25T01:37:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1ec0a32740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f46e290 0x56319f47a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1ec0a32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f46f980 0x56319f47a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1ec0a32740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56319f47d420 0x56319f47add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1ec0a32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56319f480250 con 0x56319f46f980 2024-04-25T01:37:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1ec0a32740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56319f38de90 con 0x56319f46e290 2024-04-25T01:37:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1ec0a32740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56319f4803d0 con 0x56319f47d420 2024-04-25T01:37:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1ec0a32740 10 client.0 _ll_register_callbacks cb 0x56319f3c9970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1eb6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f46f980 0x56319f47a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1eb6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f46f980 0x56319f47a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49924/0 (socket says 172.21.15.28:49924) 2024-04-25T01:37:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1eb6ffd640 1 -- 172.21.15.28:0/2405364214 learned_addr learned my addr 172.21.15.28:0/2405364214 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1eb77fe640 1 --2- 172.21.15.28:0/2405364214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f46e290 0x56319f47a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1eb7fff640 1 --2- 172.21.15.28:0/2405364214 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56319f47d420 0x56319f47add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1eb6ffd640 1 -- 172.21.15.28:0/2405364214 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56319f47d420 msgr2=0x56319f47add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1eb6ffd640 1 --2- 172.21.15.28:0/2405364214 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56319f47d420 0x56319f47add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1eb6ffd640 1 -- 172.21.15.28:0/2405364214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f46e290 msgr2=0x56319f47a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1eb6ffd640 1 --2- 172.21.15.28:0/2405364214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f46e290 0x56319f47a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1eb6ffd640 1 -- 172.21.15.28:0/2405364214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56319f1ae7e0 con 0x56319f46f980 2024-04-25T01:37:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1eb6ffd640 1 --2- 172.21.15.28:0/2405364214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f46f980 0x56319f47a890 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f1eac00ba60 tx=0x7f1eac00bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c54455107d4b631b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.360+0000 7f1e96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.361+0000 7f1e96ffd640 1 -- 172.21.15.28:0/2405364214 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eac005020 con 0x56319f46f980 2024-04-25T01:37:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.361+0000 7f1e96ffd640 1 -- 172.21.15.28:0/2405364214 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1eac0051c0 con 0x56319f46f980 2024-04-25T01:37:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.361+0000 7f1ec0a32740 1 -- 172.21.15.28:0/2405364214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ea800ede0 con 0x56319f46f980 2024-04-25T01:37:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.361+0000 7f1e96ffd640 1 -- 172.21.15.28:0/2405364214 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1eac0053c0 con 0x56319f46f980 2024-04-25T01:37:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.361+0000 7f1ec0a32740 10 client.16956 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.361+0000 7f1ec0a32740 1 -- 172.21.15.28:0/2405364214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56319f20ef70 con 0x56319f46f980 2024-04-25T01:37:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.361+0000 7f1e96ffd640 1 -- 172.21.15.28:0/2405364214 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ea800ede0 con 0x56319f46f980 2024-04-25T01:37:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.361+0000 7f1e96ffd640 1 -- 172.21.15.28:0/2405364214 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1eac01b670 con 0x56319f46f980 2024-04-25T01:37:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.361+0000 7f1ec0a32740 10 client.16956 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.361+0000 7f1ec0a32740 10 client.16956 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.361+0000 7f1ec0a32740 20 client.16956 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.361+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/2405364214 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56319f48d570 0x56319f48f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.361+0000 7f1ec0a32740 4 client.16956 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:37:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.361+0000 7f1ec0a32740 1 -- 172.21.15.28:0/2405364214 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56319f3a65a0 con 0x56319f48d570 2024-04-25T01:37:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.362+0000 7f1eb77fe640 1 --2- 172.21.15.28:0/2405364214 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56319f48d570 0x56319f48f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.362+0000 7f1e96ffd640 10 client.16956 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.362+0000 7f1eb77fe640 1 --2- 172.21.15.28:0/2405364214 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56319f48d570 0x56319f48f9c0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d26154161e2cf8a5 server_cookie=679d21655b8408b8 in_seq=0 out_seq=0 2024-04-25T01:37:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.363+0000 7f1e96ffd640 1 -- 172.21.15.28:0/2405364214 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56319f3a65a0 con 0x56319f48d570 2024-04-25T01:37:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.363+0000 7f1e96ffd640 10 client.16956 handle_command_reply: tid=0 2024-04-25T01:37:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.363+0000 7f1ec0a32740 1 client.16956 shutdown 2024-04-25T01:37:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.363+0000 7f1ec0a32740 1 -- 172.21.15.28:0/2405364214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f46f980 msgr2=0x56319f47a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.363+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/2405364214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f46f980 0x56319f47a890 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f1eac00ba60 tx=0x7f1eac00bf20 comp rx=0 tx=0).stop 2024-04-25T01:37:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.363+0000 7f1ec0a32740 1 -- 172.21.15.28:0/2405364214 shutdown_connections 2024-04-25T01:37:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.363+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/2405364214 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56319f48d570 0x56319f48f9c0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.363+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/2405364214 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56319f47d420 0x56319f47add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.363+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/2405364214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f46f980 0x56319f47a890 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.363+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/2405364214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f46e290 0x56319f47a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.363+0000 7f1e96ffd640 0 client.16956 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.363+0000 7f1ec0a32740 1 -- 172.21.15.28:0/2405364214 >> 172.21.15.28:0/2405364214 conn(0x56319f436950 msgr2=0x56319f470660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.363+0000 7f1ec0a32740 1 -- 172.21.15.28:0/2405364214 shutdown_connections 2024-04-25T01:37:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.363+0000 7f1ec0a32740 1 -- 172.21.15.28:0/2405364214 wait complete. 2024-04-25T01:37:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.363+0000 7f1ec0a32740 20 client.16956 trim_cache size 0 max 16384 2024-04-25T01:37:00.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.381+0000 7f1ec0a32740 1 Processor -- start 2024-04-25T01:37:00.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.381+0000 7f1ec0a32740 1 -- start start 2024-04-25T01:37:00.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.381+0000 7f1ec0a32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f46e290 0x56319f4891a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.381+0000 7f1ec0a32740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56319f46f980 0x56319f4952d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.381+0000 7f1ec0a32740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f47d420 0x56319f495810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.381+0000 7f1ec0a32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb81ac850 con 0x56319f46e290 2024-04-25T01:37:00.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.381+0000 7f1ec0a32740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb81ae390 con 0x56319f47d420 2024-04-25T01:37:00.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.381+0000 7f1ec0a32740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb81acf20 con 0x56319f46f980 2024-04-25T01:37:00.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.381+0000 7f1eb77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f46e290 0x56319f4891a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.381+0000 7f1eb77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f46e290 0x56319f4891a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49940/0 (socket says 172.21.15.28:49940) 2024-04-25T01:37:00.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.381+0000 7f1eb77fe640 1 -- 172.21.15.28:0/2872041081 learned_addr learned my addr 172.21.15.28:0/2872041081 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:00.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.382+0000 7f1eb7fff640 1 --2- 172.21.15.28:0/2872041081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f47d420 0x56319f495810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.382+0000 7f1eb77fe640 1 -- 172.21.15.28:0/2872041081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56319f46f980 msgr2=0x56319f4952d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:00.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.382+0000 7f1eb77fe640 1 --2- 172.21.15.28:0/2872041081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56319f46f980 0x56319f4952d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.382+0000 7f1eb77fe640 1 -- 172.21.15.28:0/2872041081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f47d420 msgr2=0x56319f495810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:00.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.382+0000 7f1eb77fe640 1 --2- 172.21.15.28:0/2872041081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f47d420 0x56319f495810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.382+0000 7f1eb77fe640 1 -- 172.21.15.28:0/2872041081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56319f0a6170 con 0x56319f46e290 2024-04-25T01:37:00.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.382+0000 7f1eb77fe640 1 --2- 172.21.15.28:0/2872041081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f46e290 0x56319f4891a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00b9580 tx=0x7f1eb00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457f3cd7b2eb2aad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.382+0000 7f1e9d7fa640 1 -- 172.21.15.28:0/2872041081 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb00bcef0 con 0x56319f46e290 2024-04-25T01:37:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.382+0000 7f1e9d7fa640 1 -- 172.21.15.28:0/2872041081 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1eb0036040 con 0x56319f46e290 2024-04-25T01:37:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.382+0000 7f1e9d7fa640 1 -- 172.21.15.28:0/2872041081 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb00c1170 con 0x56319f46e290 2024-04-25T01:37:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.383+0000 7f1ec0a32740 1 -- 172.21.15.28:0/2872041081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f46e290 msgr2=0x56319f4891a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.383+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/2872041081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f46e290 0x56319f4891a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00b9580 tx=0x7f1eb00095c0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.383+0000 7f1ec0a32740 1 -- 172.21.15.28:0/2872041081 shutdown_connections 2024-04-25T01:37:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.383+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/2872041081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f47d420 0x56319f495810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.383+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/2872041081 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56319f46f980 0x56319f4952d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.383+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/2872041081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f46e290 0x56319f4891a0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.383+0000 7f1ec0a32740 1 -- 172.21.15.28:0/2872041081 >> 172.21.15.28:0/2872041081 conn(0x56319f436950 msgr2=0x56319f469950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.383+0000 7f1ec0a32740 1 -- 172.21.15.28:0/2872041081 shutdown_connections 2024-04-25T01:37:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.383+0000 7f1ec0a32740 1 -- 172.21.15.28:0/2872041081 wait complete. 2024-04-25T01:37:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.383+0000 7f1ec0a32740 1 -- start start 2024-04-25T01:37:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.383+0000 7f1ec0a32740 1 Processor -- start 2024-04-25T01:37:00.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.383+0000 7f1ec0a32740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56319f46e290 0x56319f4859a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.383+0000 7f1ec0a32740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f46f980 0x56319f482680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.383+0000 7f1ec0a32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f47d420 0x56319f482bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.383+0000 7f1ec0a32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb81adcc0 con 0x56319f47d420 2024-04-25T01:37:00.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.383+0000 7f1ec0a32740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb81ad5f0 con 0x56319f46f980 2024-04-25T01:37:00.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.383+0000 7f1ec0a32740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb81aa830 con 0x56319f46e290 2024-04-25T01:37:00.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.383+0000 7f1ec0a32740 10 client.0 _ll_register_callbacks cb 0x7f1eac0042b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:00.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.384+0000 7f1eb7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f47d420 0x56319f482bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.384+0000 7f1eb7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f47d420 0x56319f482bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49952/0 (socket says 172.21.15.28:49952) 2024-04-25T01:37:00.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.384+0000 7f1eb7fff640 1 -- 172.21.15.28:0/4059586099 learned_addr learned my addr 172.21.15.28:0/4059586099 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:00.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.384+0000 7f1eb77fe640 1 --2- 172.21.15.28:0/4059586099 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56319f46e290 0x56319f4859a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.384+0000 7f1eb6ffd640 1 --2- 172.21.15.28:0/4059586099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f46f980 0x56319f482680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.384+0000 7f1eb7fff640 1 -- 172.21.15.28:0/4059586099 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56319f46e290 msgr2=0x56319f4859a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:00.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.384+0000 7f1eb7fff640 1 --2- 172.21.15.28:0/4059586099 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56319f46e290 0x56319f4859a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.384+0000 7f1eb7fff640 1 -- 172.21.15.28:0/4059586099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f46f980 msgr2=0x56319f482680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.384+0000 7f1eb7fff640 1 --2- 172.21.15.28:0/4059586099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f46f980 0x56319f482680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.384+0000 7f1eb7fff640 1 -- 172.21.15.28:0/4059586099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56319f483300 con 0x56319f47d420 2024-04-25T01:37:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.384+0000 7f1eb7fff640 1 --2- 172.21.15.28:0/4059586099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f47d420 0x56319f482bc0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8000c00 tx=0x7f1ea8005ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43410f85bbc15a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.384+0000 7f1e96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.384+0000 7f1e96ffd640 1 -- 172.21.15.28:0/4059586099 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8013040 con 0x56319f47d420 2024-04-25T01:37:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.384+0000 7f1e96ffd640 1 -- 172.21.15.28:0/4059586099 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1ea80131e0 con 0x56319f47d420 2024-04-25T01:37:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.384+0000 7f1ec0a32740 1 -- 172.21.15.28:0/4059586099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1eb0036040 con 0x56319f47d420 2024-04-25T01:37:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.384+0000 7f1e96ffd640 1 -- 172.21.15.28:0/4059586099 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1ea8041050 con 0x56319f47d420 2024-04-25T01:37:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.384+0000 7f1ec0a32740 10 client.16962 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.384+0000 7f1ec0a32740 1 -- 172.21.15.28:0/4059586099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56319f4f4ac0 con 0x56319f47d420 2024-04-25T01:37:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.385+0000 7f1e96ffd640 1 -- 172.21.15.28:0/4059586099 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1eb0036040 con 0x56319f47d420 2024-04-25T01:37:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.385+0000 7f1e96ffd640 1 -- 172.21.15.28:0/4059586099 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1ea800f040 con 0x56319f47d420 2024-04-25T01:37:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.385+0000 7f1ec0a32740 10 client.16962 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.385+0000 7f1ec0a32740 10 client.16962 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.385+0000 7f1ec0a32740 20 client.16962 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.385+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/4059586099 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56319f4f4ea0 0x56319f4f72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.385+0000 7f1ec0a32740 4 client.16962 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:37:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.385+0000 7f1ec0a32740 1 -- 172.21.15.28:0/4059586099 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56319f3df800 con 0x56319f4f4ea0 2024-04-25T01:37:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.385+0000 7f1eb77fe640 1 --2- 172.21.15.28:0/4059586099 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56319f4f4ea0 0x56319f4f72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.385+0000 7f1eb77fe640 1 --2- 172.21.15.28:0/4059586099 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56319f4f4ea0 0x56319f4f72c0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c1f5d3e2dc84e11 server_cookie=65e9e257246fa2ee in_seq=0 out_seq=0 2024-04-25T01:37:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.385+0000 7f1e96ffd640 10 client.16962 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.386+0000 7f1e96ffd640 1 -- 172.21.15.28:0/4059586099 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56319f3df800 con 0x56319f4f4ea0 2024-04-25T01:37:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.386+0000 7f1e96ffd640 10 client.16962 handle_command_reply: tid=0 2024-04-25T01:37:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.386+0000 7f1ec0a32740 1 client.16962 shutdown 2024-04-25T01:37:00.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.387+0000 7f1ec0a32740 1 -- 172.21.15.28:0/4059586099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f47d420 msgr2=0x56319f482bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:00.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.387+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/4059586099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f47d420 0x56319f482bc0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8000c00 tx=0x7f1ea8005ed0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.387+0000 7f1ec0a32740 1 -- 172.21.15.28:0/4059586099 shutdown_connections 2024-04-25T01:37:00.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.387+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/4059586099 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56319f4f4ea0 0x56319f4f72c0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.387+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/4059586099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56319f47d420 0x56319f482bc0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.387+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/4059586099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56319f46f980 0x56319f482680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.387+0000 7f1e96ffd640 0 client.16962 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.387+0000 7f1ec0a32740 1 --2- 172.21.15.28:0/4059586099 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56319f46e290 0x56319f4859a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.387+0000 7f1ec0a32740 1 -- 172.21.15.28:0/4059586099 >> 172.21.15.28:0/4059586099 conn(0x56319f436950 msgr2=0x56319f47bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.387+0000 7f1ec0a32740 1 -- 172.21.15.28:0/4059586099 shutdown_connections 2024-04-25T01:37:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.387+0000 7f1ec0a32740 1 -- 172.21.15.28:0/4059586099 wait complete. 2024-04-25T01:37:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.387+0000 7f1ec0a32740 20 client.16962 trim_cache size 0 max 16384 2024-04-25T01:37:00.369 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:37:00.369 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:37:00.369 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "3b359342-de0a-4c6e-8816-753490d77cb2", 2024-04-25T01:37:00.370 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:37:00.370 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:37:00.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.387+0000 7f1ebdd22640 1 -- 172.21.15.28:0/485366520 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1ea00a2d00 msgr2=0x7f1ea00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:00.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.388+0000 7f1ebdd22640 1 --2- 172.21.15.28:0/485366520 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1ea00a2d00 0x7f1ea00a51c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1ea80137e0 tx=0x7f1ea800ba20 comp rx=0 tx=0).stop 2024-04-25T01:37:00.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.388+0000 7f1ebdd22640 1 -- 172.21.15.28:0/485366520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb8100620 msgr2=0x7f1eb8198e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:00.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.388+0000 7f1ebdd22640 1 --2- 172.21.15.28:0/485366520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb8100620 0x7f1eb8198e20 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f1eb002ae00 tx=0x7f1eb0003970 comp rx=0 tx=0).stop 2024-04-25T01:37:00.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.388+0000 7f1ebdd22640 1 -- 172.21.15.28:0/485366520 shutdown_connections 2024-04-25T01:37:00.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.388+0000 7f1ebdd22640 1 --2- 172.21.15.28:0/485366520 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1ea00a2d00 0x7f1ea00a51c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.388+0000 7f1ebdd22640 1 --2- 172.21.15.28:0/485366520 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1eb8106630 0x7f1eb81a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.388+0000 7f1ebdd22640 1 --2- 172.21.15.28:0/485366520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1eb8100fe0 0x7f1eb8199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.388+0000 7f1ebdd22640 1 --2- 172.21.15.28:0/485366520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb8100620 0x7f1eb8198e20 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.388+0000 7f1ebdd22640 1 -- 172.21.15.28:0/485366520 >> 172.21.15.28:0/485366520 conn(0x7f1eb80776a0 msgr2=0x7f1eb8103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:00.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.388+0000 7f1ebdd22640 1 -- 172.21.15.28:0/485366520 shutdown_connections 2024-04-25T01:37:00.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.388+0000 7f1ebdd22640 1 -- 172.21.15.28:0/485366520 wait complete. 2024-04-25T01:37:00.375 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:37:00.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:37:00.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:00 smithi130 ceph-mon[32702]: pgmap v122: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:37:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.503+0000 7f25c47ce640 1 -- 172.21.15.28:0/295298337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25bc069dc0 msgr2=0x7f25bc06a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.503+0000 7f25c47ce640 1 --2- 172.21.15.28:0/295298337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25bc069dc0 0x7f25bc06a240 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f25ac009930 tx=0x7f25ac02a420 comp rx=0 tx=0).stop 2024-04-25T01:37:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.503+0000 7f25c47ce640 1 -- 172.21.15.28:0/295298337 shutdown_connections 2024-04-25T01:37:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.503+0000 7f25c47ce640 1 --2- 172.21.15.28:0/295298337 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f25bc06a780 0x7f25bc10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.503+0000 7f25c47ce640 1 --2- 172.21.15.28:0/295298337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25bc069dc0 0x7f25bc06a240 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.503+0000 7f25c47ce640 1 --2- 172.21.15.28:0/295298337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25bc107730 0x7f25bc107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.503+0000 7f25c47ce640 1 -- 172.21.15.28:0/295298337 >> 172.21.15.28:0/295298337 conn(0x7f25bc0776a0 msgr2=0x7f25bc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.503+0000 7f25c47ce640 1 -- 172.21.15.28:0/295298337 shutdown_connections 2024-04-25T01:37:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.503+0000 7f25c47ce640 1 -- 172.21.15.28:0/295298337 wait complete. 2024-04-25T01:37:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.504+0000 7f25c47ce640 1 Processor -- start 2024-04-25T01:37:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.504+0000 7f25c47ce640 1 -- start start 2024-04-25T01:37:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.504+0000 7f25c47ce640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25bc069dc0 0x7f25bc197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.504+0000 7f25c47ce640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f25bc06a780 0x7f25bc1984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.504+0000 7f25c47ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25bc107730 0x7f25bc19c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.504+0000 7f25c47ce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc111b40 con 0x7f25bc107730 2024-04-25T01:37:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.504+0000 7f25c47ce640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc1119c0 con 0x7f25bc069dc0 2024-04-25T01:37:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.504+0000 7f25c47ce640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc111cc0 con 0x7f25bc06a780 2024-04-25T01:37:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.504+0000 7f25c2d6b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25bc107730 0x7f25bc19c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.504+0000 7f25c2d6b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25bc107730 0x7f25bc19c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49966/0 (socket says 172.21.15.28:49966) 2024-04-25T01:37:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.504+0000 7f25c2d6b640 1 -- 172.21.15.28:0/1157301648 learned_addr learned my addr 172.21.15.28:0/1157301648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.504+0000 7f25c256a640 1 --2- 172.21.15.28:0/1157301648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25bc069dc0 0x7f25bc197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.504+0000 7f25c2d6b640 1 -- 172.21.15.28:0/1157301648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f25bc06a780 msgr2=0x7f25bc1984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.504+0000 7f25c2d6b640 1 --2- 172.21.15.28:0/1157301648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f25bc06a780 0x7f25bc1984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.504+0000 7f25c2d6b640 1 -- 172.21.15.28:0/1157301648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25bc069dc0 msgr2=0x7f25bc197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.504+0000 7f25c2d6b640 1 --2- 172.21.15.28:0/1157301648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25bc069dc0 0x7f25bc197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.504+0000 7f25c2d6b640 1 -- 172.21.15.28:0/1157301648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25bc19cf90 con 0x7f25bc107730 2024-04-25T01:37:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.505+0000 7f25c2d6b640 1 --2- 172.21.15.28:0/1157301648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25bc107730 0x7f25bc19c860 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f25b400cce0 tx=0x7f25b400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.505+0000 7f25b37fe640 1 -- 172.21.15.28:0/1157301648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b400f450 con 0x7f25bc107730 2024-04-25T01:37:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.505+0000 7f25c47ce640 1 -- 172.21.15.28:0/1157301648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25bc19d280 con 0x7f25bc107730 2024-04-25T01:37:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.505+0000 7f25b37fe640 1 -- 172.21.15.28:0/1157301648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25b400fad0 con 0x7f25bc107730 2024-04-25T01:37:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.505+0000 7f25b37fe640 1 -- 172.21.15.28:0/1157301648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b4018c10 con 0x7f25bc107730 2024-04-25T01:37:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.505+0000 7f25c47ce640 1 -- 172.21.15.28:0/1157301648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25bc1a4b10 con 0x7f25bc107730 2024-04-25T01:37:00.484 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:00 smithi032 ceph-mon[32461]: pgmap v122: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:37:00.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.507+0000 7f25b37fe640 1 -- 172.21.15.28:0/1157301648 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f25b400f5f0 con 0x7f25bc107730 2024-04-25T01:37:00.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.507+0000 7f25c75b8740 1 Processor -- start 2024-04-25T01:37:00.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.507+0000 7f25c75b8740 1 -- start start 2024-04-25T01:37:00.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25c75b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b64530 0x5620b2b64910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25c75b8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620b2b64e50 0x5620b2b734f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25c75b8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b73a30 0x5620b2b75e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25c75b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5620b2ac96b0 con 0x5620b2b64530 2024-04-25T01:37:00.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25c75b8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5620b2a71880 con 0x5620b2b73a30 2024-04-25T01:37:00.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25c75b8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5620b2ac5670 con 0x5620b2b64e50 2024-04-25T01:37:00.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25b37fe640 1 --2- 172.21.15.28:0/1157301648 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f25a80a2d00 0x7f25a80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25c1d69640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b64530 0x5620b2b64910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25c1d69640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b64530 0x5620b2b64910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49974/0 (socket says 172.21.15.28:49974) 2024-04-25T01:37:00.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25c1d69640 1 -- 172.21.15.28:0/1480531260 learned_addr learned my addr 172.21.15.28:0/1480531260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:00.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25c2d6b640 1 --2- 172.21.15.28:0/1157301648 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f25a80a2d00 0x7f25a80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25b37fe640 1 -- 172.21.15.28:0/1157301648 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f25b40cff00 con 0x7f25bc107730 2024-04-25T01:37:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25c1d69640 1 --2- 172.21.15.28:0/1480531260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b73a30 0x5620b2b75e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25c256a640 1 --2- 172.21.15.28:0/1480531260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620b2b64e50 0x5620b2b734f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25c1d69640 1 -- 172.21.15.28:0/1480531260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620b2b64e50 msgr2=0x5620b2b734f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25c1d69640 1 --2- 172.21.15.28:0/1480531260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620b2b64e50 0x5620b2b734f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25c1d69640 1 -- 172.21.15.28:0/1480531260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b73a30 msgr2=0x5620b2b75e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25c1d69640 1 --2- 172.21.15.28:0/1480531260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b73a30 0x5620b2b75e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25c1d69640 1 -- 172.21.15.28:0/1480531260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620b28a77d0 con 0x5620b2b64530 2024-04-25T01:37:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25c1d69640 1 --2- 172.21.15.28:0/1480531260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b64530 0x5620b2b64910 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f25ac032fd0 tx=0x7f25ac0314a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbe9d7487ac628fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25c2d6b640 1 --2- 172.21.15.28:0/1157301648 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f25a80a2d00 0x7f25a80a51c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f25b40cd3c0 tx=0x7f25b400af30 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.508+0000 7f25b0ff9640 1 -- 172.21.15.28:0/1480531260 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ac043020 con 0x5620b2b64530 2024-04-25T01:37:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.509+0000 7f25b0ff9640 1 -- 172.21.15.28:0/1480531260 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25ac0318b0 con 0x5620b2b64530 2024-04-25T01:37:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.509+0000 7f25b0ff9640 1 -- 172.21.15.28:0/1480531260 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ac035040 con 0x5620b2b64530 2024-04-25T01:37:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.509+0000 7f25c75b8740 1 -- 172.21.15.28:0/1480531260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b64530 msgr2=0x5620b2b64910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:00.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.509+0000 7f25c75b8740 1 --2- 172.21.15.28:0/1480531260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b64530 0x5620b2b64910 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f25ac032fd0 tx=0x7f25ac0314a0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.509+0000 7f25c75b8740 1 -- 172.21.15.28:0/1480531260 shutdown_connections 2024-04-25T01:37:00.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.509+0000 7f25c75b8740 1 --2- 172.21.15.28:0/1480531260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b73a30 0x5620b2b75e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.509+0000 7f25c75b8740 1 --2- 172.21.15.28:0/1480531260 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620b2b64e50 0x5620b2b734f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.509+0000 7f25c75b8740 1 --2- 172.21.15.28:0/1480531260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b64530 0x5620b2b64910 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.509+0000 7f25c75b8740 1 -- 172.21.15.28:0/1480531260 >> 172.21.15.28:0/1480531260 conn(0x5620b2b60950 msgr2=0x5620b2b60d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:00.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.509+0000 7f25c75b8740 1 -- 172.21.15.28:0/1480531260 shutdown_connections 2024-04-25T01:37:00.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.509+0000 7f25c75b8740 1 -- 172.21.15.28:0/1480531260 wait complete. 2024-04-25T01:37:00.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.510+0000 7f25c75b8740 1 -- start start 2024-04-25T01:37:00.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.510+0000 7f25c75b8740 1 Processor -- start 2024-04-25T01:37:00.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.510+0000 7f25c75b8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b64530 0x5620b2b709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.510+0000 7f25c75b8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620b2b64e50 0x5620b2b70ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.510+0000 7f25c75b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b73a30 0x5620b2b71430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.510+0000 7f25c75b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5620b2b76790 con 0x5620b2b73a30 2024-04-25T01:37:00.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.510+0000 7f25c75b8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5620b2a84f50 con 0x5620b2b64530 2024-04-25T01:37:00.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.510+0000 7f25c75b8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5620b2b76910 con 0x5620b2b64e50 2024-04-25T01:37:00.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.510+0000 7f25c75b8740 10 client.0 _ll_register_callbacks cb 0x5620b2ac0000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:00.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.510+0000 7f25c1d69640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b73a30 0x5620b2b71430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.510+0000 7f25c1d69640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b73a30 0x5620b2b71430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49978/0 (socket says 172.21.15.28:49978) 2024-04-25T01:37:00.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.510+0000 7f25c1d69640 1 -- 172.21.15.28:0/902930914 learned_addr learned my addr 172.21.15.28:0/902930914 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:00.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.510+0000 7f25c256a640 1 --2- 172.21.15.28:0/902930914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620b2b64e50 0x5620b2b70ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.510+0000 7f25c1d69640 1 --2- 172.21.15.28:0/902930914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b64530 0x5620b2b709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25c1d69640 1 -- 172.21.15.28:0/902930914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620b2b64e50 msgr2=0x5620b2b70ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:00.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25c1d69640 1 --2- 172.21.15.28:0/902930914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620b2b64e50 0x5620b2b70ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25c1d69640 1 -- 172.21.15.28:0/902930914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b64530 msgr2=0x5620b2b709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:00.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25c1d69640 1 --2- 172.21.15.28:0/902930914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b64530 0x5620b2b709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25c1d69640 1 -- 172.21.15.28:0/902930914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620b27dd3f0 con 0x5620b2b73a30 2024-04-25T01:37:00.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25c1d69640 1 --2- 172.21.15.28:0/902930914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b73a30 0x5620b2b71430 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f25ac031850 tx=0x7f25ac0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae4630124b0cd5c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:00.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:00.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25a5ffb640 1 -- 172.21.15.28:0/902930914 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ac035070 con 0x5620b2b73a30 2024-04-25T01:37:00.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25a5ffb640 1 -- 172.21.15.28:0/902930914 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f25ac004310 con 0x5620b2b73a30 2024-04-25T01:37:00.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25c75b8740 1 -- 172.21.15.28:0/902930914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25ac0318b0 con 0x5620b2b73a30 2024-04-25T01:37:00.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25a5ffb640 1 -- 172.21.15.28:0/902930914 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f25ac043030 con 0x5620b2b73a30 2024-04-25T01:37:00.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25c75b8740 10 client.16974 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:00.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25c75b8740 1 -- 172.21.15.28:0/902930914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5620b279d0f0 con 0x5620b2b73a30 2024-04-25T01:37:00.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25a5ffb640 1 -- 172.21.15.28:0/902930914 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25ac0318b0 con 0x5620b2b73a30 2024-04-25T01:37:00.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25a5ffb640 1 -- 172.21.15.28:0/902930914 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f25ac048420 con 0x5620b2b73a30 2024-04-25T01:37:00.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25c75b8740 10 client.16974 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:00.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25c75b8740 10 client.16974 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:00.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25c75b8740 20 client.16974 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:00.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25c75b8740 1 --2- 172.21.15.28:0/902930914 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5620b2b83b10 0x5620b2b85f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25c75b8740 4 client.16974 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:37:00.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.511+0000 7f25c75b8740 1 -- 172.21.15.28:0/902930914 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5620b2ad6bf0 con 0x5620b2b83b10 2024-04-25T01:37:00.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.512+0000 7f25c256a640 1 --2- 172.21.15.28:0/902930914 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5620b2b83b10 0x5620b2b85f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.512+0000 7f25c256a640 1 --2- 172.21.15.28:0/902930914 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5620b2b83b10 0x5620b2b85f00 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=857a7a1341e156f3 server_cookie=325127a092bd4cce in_seq=0 out_seq=0 2024-04-25T01:37:00.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.512+0000 7f25a5ffb640 10 client.16974 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:00.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.513+0000 7f25a5ffb640 1 -- 172.21.15.28:0/902930914 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5620b2ad6bf0 con 0x5620b2b83b10 2024-04-25T01:37:00.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.513+0000 7f25a5ffb640 10 client.16974 handle_command_reply: tid=0 2024-04-25T01:37:00.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.513+0000 7f25c75b8740 1 client.16974 shutdown 2024-04-25T01:37:00.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.513+0000 7f25c75b8740 1 -- 172.21.15.28:0/902930914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b73a30 msgr2=0x5620b2b71430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:00.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.513+0000 7f25c75b8740 1 --2- 172.21.15.28:0/902930914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b73a30 0x5620b2b71430 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f25ac031850 tx=0x7f25ac0039c0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.513+0000 7f25c75b8740 1 -- 172.21.15.28:0/902930914 shutdown_connections 2024-04-25T01:37:00.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.513+0000 7f25c75b8740 1 --2- 172.21.15.28:0/902930914 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5620b2b83b10 0x5620b2b85f00 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.513+0000 7f25c75b8740 1 --2- 172.21.15.28:0/902930914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b73a30 0x5620b2b71430 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.513+0000 7f25c75b8740 1 --2- 172.21.15.28:0/902930914 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620b2b64e50 0x5620b2b70ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.513+0000 7f25c75b8740 1 --2- 172.21.15.28:0/902930914 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b64530 0x5620b2b709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.513+0000 7f25a5ffb640 0 client.16974 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:00.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.513+0000 7f25c75b8740 1 -- 172.21.15.28:0/902930914 >> 172.21.15.28:0/902930914 conn(0x5620b2b60950 msgr2=0x5620b2b66780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:00.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.513+0000 7f25c75b8740 1 -- 172.21.15.28:0/902930914 shutdown_connections 2024-04-25T01:37:00.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.513+0000 7f25c75b8740 1 -- 172.21.15.28:0/902930914 wait complete. 2024-04-25T01:37:00.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.513+0000 7f25c75b8740 20 client.16974 trim_cache size 0 max 16384 2024-04-25T01:37:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.531+0000 7f25c75b8740 1 Processor -- start 2024-04-25T01:37:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.531+0000 7f25c75b8740 1 -- start start 2024-04-25T01:37:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.531+0000 7f25c75b8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620b2b64530 0x5620b2b730e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.531+0000 7f25c75b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b64e50 0x5620b2b60d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.531+0000 7f25c75b8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b73a30 0x5620b2b7f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.531+0000 7f25c75b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc1ac700 con 0x5620b2b64e50 2024-04-25T01:37:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.531+0000 7f25c75b8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc1ae240 con 0x5620b2b73a30 2024-04-25T01:37:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.531+0000 7f25c75b8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc1acdd0 con 0x5620b2b64530 2024-04-25T01:37:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.531+0000 7f25c256a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b64e50 0x5620b2b60d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.531+0000 7f25c256a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b64e50 0x5620b2b60d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49982/0 (socket says 172.21.15.28:49982) 2024-04-25T01:37:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.531+0000 7f25c256a640 1 -- 172.21.15.28:0/3466793092 learned_addr learned my addr 172.21.15.28:0/3466793092 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.531+0000 7f25c1d69640 1 --2- 172.21.15.28:0/3466793092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b73a30 0x5620b2b7f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.531+0000 7f25c256a640 1 -- 172.21.15.28:0/3466793092 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620b2b64530 msgr2=0x5620b2b730e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.531+0000 7f25c256a640 1 --2- 172.21.15.28:0/3466793092 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620b2b64530 0x5620b2b730e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.531+0000 7f25c256a640 1 -- 172.21.15.28:0/3466793092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b73a30 msgr2=0x5620b2b7f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.531+0000 7f25c256a640 1 --2- 172.21.15.28:0/3466793092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b73a30 0x5620b2b7f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.531+0000 7f25c256a640 1 -- 172.21.15.28:0/3466793092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620b2b64c80 con 0x5620b2b64e50 2024-04-25T01:37:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.531+0000 7f25c1d69640 1 --2- 172.21.15.28:0/3466793092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b73a30 0x5620b2b7f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.531+0000 7f25c256a640 1 --2- 172.21.15.28:0/3466793092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b64e50 0x5620b2b60d30 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x5620b2b70210 tx=0x7f25b800ff80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=436b4d6fe5a5bc7b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.532+0000 7f25a7fff640 1 -- 172.21.15.28:0/3466793092 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b8003fc0 con 0x5620b2b64e50 2024-04-25T01:37:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.532+0000 7f25a7fff640 1 -- 172.21.15.28:0/3466793092 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25b8005500 con 0x5620b2b64e50 2024-04-25T01:37:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.532+0000 7f25a7fff640 1 -- 172.21.15.28:0/3466793092 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b801e2b0 con 0x5620b2b64e50 2024-04-25T01:37:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.532+0000 7f25c75b8740 1 -- 172.21.15.28:0/3466793092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b64e50 msgr2=0x5620b2b60d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.532+0000 7f25c75b8740 1 --2- 172.21.15.28:0/3466793092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b64e50 0x5620b2b60d30 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x5620b2b70210 tx=0x7f25b800ff80 comp rx=0 tx=0).stop 2024-04-25T01:37:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.532+0000 7f25c75b8740 1 -- 172.21.15.28:0/3466793092 shutdown_connections 2024-04-25T01:37:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.532+0000 7f25c75b8740 1 --2- 172.21.15.28:0/3466793092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b73a30 0x5620b2b7f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.532+0000 7f25c75b8740 1 --2- 172.21.15.28:0/3466793092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b64e50 0x5620b2b60d30 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.532+0000 7f25c75b8740 1 --2- 172.21.15.28:0/3466793092 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620b2b64530 0x5620b2b730e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.532+0000 7f25c75b8740 1 -- 172.21.15.28:0/3466793092 >> 172.21.15.28:0/3466793092 conn(0x5620b2b60950 msgr2=0x5620b2b7eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.532+0000 7f25c75b8740 1 -- 172.21.15.28:0/3466793092 shutdown_connections 2024-04-25T01:37:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.532+0000 7f25c75b8740 1 -- 172.21.15.28:0/3466793092 wait complete. 2024-04-25T01:37:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.532+0000 7f25c75b8740 1 -- start start 2024-04-25T01:37:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.532+0000 7f25c75b8740 1 Processor -- start 2024-04-25T01:37:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25c75b8740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620b2b64530 0x5620b2b890d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25c75b8740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b64e50 0x5620b2b86460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25c75b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b73a30 0x5620b2b869a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25c75b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc1adb70 con 0x5620b2b73a30 2024-04-25T01:37:00.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25c75b8740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc1ad4a0 con 0x5620b2b64e50 2024-04-25T01:37:00.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25c75b8740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc1aa6e0 con 0x5620b2b64530 2024-04-25T01:37:00.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25c75b8740 10 client.0 _ll_register_callbacks cb 0x7f25ac032f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:00.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25c1d69640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b73a30 0x5620b2b869a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25c256a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b64e50 0x5620b2b86460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25c1d69640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b73a30 0x5620b2b869a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49998/0 (socket says 172.21.15.28:49998) 2024-04-25T01:37:00.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25c1d69640 1 -- 172.21.15.28:0/2522198752 learned_addr learned my addr 172.21.15.28:0/2522198752 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:00.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25c1d69640 1 --2- 172.21.15.28:0/2522198752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620b2b64530 0x5620b2b890d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25c1d69640 1 -- 172.21.15.28:0/2522198752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620b2b64530 msgr2=0x5620b2b890d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:00.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25c1d69640 1 --2- 172.21.15.28:0/2522198752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620b2b64530 0x5620b2b890d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25c1d69640 1 -- 172.21.15.28:0/2522198752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b64e50 msgr2=0x5620b2b86460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25c1d69640 1 --2- 172.21.15.28:0/2522198752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b64e50 0x5620b2b86460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25c1d69640 1 -- 172.21.15.28:0/2522198752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620b2b86f60 con 0x5620b2b73a30 2024-04-25T01:37:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25c1d69640 1 --2- 172.21.15.28:0/2522198752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b73a30 0x5620b2b869a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f25ac02a940 tx=0x7f25ac0314a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63431e3e0a121b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.533+0000 7f25a5ffb640 1 -- 172.21.15.28:0/2522198752 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ac067060 con 0x5620b2b73a30 2024-04-25T01:37:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.534+0000 7f25a5ffb640 1 -- 172.21.15.28:0/2522198752 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f25ac031880 con 0x5620b2b73a30 2024-04-25T01:37:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.534+0000 7f25c75b8740 1 -- 172.21.15.28:0/2522198752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25b8005500 con 0x5620b2b73a30 2024-04-25T01:37:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.534+0000 7f25a5ffb640 1 -- 172.21.15.28:0/2522198752 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f25ac06e050 con 0x5620b2b73a30 2024-04-25T01:37:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.534+0000 7f25c75b8740 10 client.16980 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.534+0000 7f25c75b8740 1 -- 172.21.15.28:0/2522198752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5620b2be9a40 con 0x5620b2b73a30 2024-04-25T01:37:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.534+0000 7f25a5ffb640 1 -- 172.21.15.28:0/2522198752 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25b8005500 con 0x5620b2b73a30 2024-04-25T01:37:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.534+0000 7f25a5ffb640 1 -- 172.21.15.28:0/2522198752 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f25ac03e480 con 0x5620b2b73a30 2024-04-25T01:37:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.534+0000 7f25c75b8740 10 client.16980 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:00.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.534+0000 7f25c75b8740 10 client.16980 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.534+0000 7f25c75b8740 20 client.16980 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.534+0000 7f25c75b8740 1 --2- 172.21.15.28:0/2522198752 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5620b2bea040 0x5620b2bec430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.534+0000 7f25c75b8740 4 client.16980 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:37:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.534+0000 7f25c75b8740 1 -- 172.21.15.28:0/2522198752 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5620b2ad0f40 con 0x5620b2bea040 2024-04-25T01:37:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.534+0000 7f25c256a640 1 --2- 172.21.15.28:0/2522198752 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5620b2bea040 0x5620b2bec430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.535+0000 7f25a5ffb640 10 client.16980 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.535+0000 7f25c256a640 1 --2- 172.21.15.28:0/2522198752 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5620b2bea040 0x5620b2bec430 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55333cddd8f6902a server_cookie=37e2ab96255459a1 in_seq=0 out_seq=0 2024-04-25T01:37:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.535+0000 7f25a5ffb640 1 -- 172.21.15.28:0/2522198752 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5620b2ad0f40 con 0x5620b2bea040 2024-04-25T01:37:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.535+0000 7f25a5ffb640 10 client.16980 handle_command_reply: tid=0 2024-04-25T01:37:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.535+0000 7f25c75b8740 1 client.16980 shutdown 2024-04-25T01:37:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.535+0000 7f25c75b8740 1 -- 172.21.15.28:0/2522198752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b73a30 msgr2=0x5620b2b869a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.535+0000 7f25c75b8740 1 --2- 172.21.15.28:0/2522198752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b73a30 0x5620b2b869a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f25ac02a940 tx=0x7f25ac0314a0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.535+0000 7f25c75b8740 1 -- 172.21.15.28:0/2522198752 shutdown_connections 2024-04-25T01:37:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.535+0000 7f25c75b8740 1 --2- 172.21.15.28:0/2522198752 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5620b2bea040 0x5620b2bec430 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.535+0000 7f25c75b8740 1 --2- 172.21.15.28:0/2522198752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5620b2b73a30 0x5620b2b869a0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.535+0000 7f25c75b8740 1 --2- 172.21.15.28:0/2522198752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5620b2b64e50 0x5620b2b86460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.535+0000 7f25c75b8740 1 --2- 172.21.15.28:0/2522198752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5620b2b64530 0x5620b2b890d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.535+0000 7f25c75b8740 1 -- 172.21.15.28:0/2522198752 >> 172.21.15.28:0/2522198752 conn(0x5620b2b60950 msgr2=0x5620b2b71200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.535+0000 7f25a5ffb640 0 client.16980 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.535+0000 7f25c75b8740 1 -- 172.21.15.28:0/2522198752 shutdown_connections 2024-04-25T01:37:00.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.535+0000 7f25c75b8740 1 -- 172.21.15.28:0/2522198752 wait complete. 2024-04-25T01:37:00.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.535+0000 7f25c75b8740 20 client.16980 trim_cache size 0 max 16384 2024-04-25T01:37:00.519 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:37:00.519 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:37:00.519 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-25T01:37:00.519 INFO:teuthology.orchestra.run.smithi028.stdout: "3b359342-de0a-4c6e-8816-753490d77cb2": { 2024-04-25T01:37:00.519 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-25T01:37:00.519 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "3b359342-de0a-4c6e-8816-753490d77cb2", 2024-04-25T01:37:00.519 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-25T01:37:00.519 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:37:00.520 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:37:00.520 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:37:00.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.536+0000 7f25c47ce640 1 -- 172.21.15.28:0/1157301648 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f25a80a2d00 msgr2=0x7f25a80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:00.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.536+0000 7f25c47ce640 1 --2- 172.21.15.28:0/1157301648 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f25a80a2d00 0x7f25a80a51c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f25b40cd3c0 tx=0x7f25b400af30 comp rx=0 tx=0).stop 2024-04-25T01:37:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.536+0000 7f25c47ce640 1 -- 172.21.15.28:0/1157301648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25bc107730 msgr2=0x7f25bc19c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.536+0000 7f25c47ce640 1 --2- 172.21.15.28:0/1157301648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25bc107730 0x7f25bc19c860 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f25b400cce0 tx=0x7f25b400a620 comp rx=0 tx=0).stop 2024-04-25T01:37:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.536+0000 7f25c47ce640 1 -- 172.21.15.28:0/1157301648 shutdown_connections 2024-04-25T01:37:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.536+0000 7f25c47ce640 1 --2- 172.21.15.28:0/1157301648 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f25a80a2d00 0x7f25a80a51c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.536+0000 7f25c47ce640 1 --2- 172.21.15.28:0/1157301648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25bc107730 0x7f25bc19c860 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.536+0000 7f25c47ce640 1 --2- 172.21.15.28:0/1157301648 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f25bc06a780 0x7f25bc1984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.536+0000 7f25c47ce640 1 --2- 172.21.15.28:0/1157301648 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f25bc069dc0 0x7f25bc197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.536+0000 7f25c47ce640 1 -- 172.21.15.28:0/1157301648 >> 172.21.15.28:0/1157301648 conn(0x7f25bc0776a0 msgr2=0x7f25bc0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.536+0000 7f25c47ce640 1 -- 172.21.15.28:0/1157301648 shutdown_connections 2024-04-25T01:37:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:00.536+0000 7f25c47ce640 1 -- 172.21.15.28:0/1157301648 wait complete. 2024-04-25T01:37:00.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:00 smithi028 ceph-mon[29390]: pgmap v122: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:37:00.522 INFO:tasks.cephfs.filesystem:scrub status for tag:3b359342-de0a-4c6e-8816-753490d77cb2 - {'path': '/', 'tag': '3b359342-de0a-4c6e-8816-753490d77cb2', 'options': 'recursive,force'} 2024-04-25T01:37:01.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:01 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:37:01.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:01 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:37:01.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:01 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:37:01.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:01 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:37:01.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:01 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:37:01.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:01 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:37:01.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:01 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:37:01.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:01 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:37:01.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:01 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:37:01.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:01 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:37:01.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:01 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:37:01.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:01 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:37:02.457 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:02 smithi130 ceph-mon[32702]: pgmap v123: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:37:02.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:02 smithi032 ceph-mon[32461]: pgmap v123: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:37:02.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:02 smithi028 ceph-mon[29390]: pgmap v123: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:37:03.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:03 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:37:03.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:03 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:37:03.707 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:03 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:37:04.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:04 smithi032 ceph-mon[32461]: pgmap v124: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:37:04.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:04 smithi028 ceph-mon[29390]: pgmap v124: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:37:04.707 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:04 smithi130 ceph-mon[32702]: pgmap v124: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:37:06.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:06 smithi032 ceph-mon[32461]: pgmap v125: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:37:06.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:06 smithi028 ceph-mon[29390]: pgmap v125: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:37:06.707 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:06 smithi130 ceph-mon[32702]: pgmap v125: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:37:08.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:08 smithi032 ceph-mon[32461]: pgmap v126: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:37:08.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:08 smithi028 ceph-mon[29390]: pgmap v126: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:37:08.707 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:08 smithi130 ceph-mon[32702]: pgmap v126: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:37:09.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:09 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:37:09.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:09 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:37:09.707 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:09 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:37:10.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:10 smithi032 ceph-mon[32461]: pgmap v127: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-04-25T01:37:10.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:10 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:37:10.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:10 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:37:10.483 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:10 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:37:10.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:10 smithi028 ceph-mon[29390]: pgmap v127: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-04-25T01:37:10.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:10 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:37:10.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:10 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:37:10.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:10 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:37:10.707 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:10 smithi130 ceph-mon[32702]: pgmap v127: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-04-25T01:37:10.707 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:10 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:37:10.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:10 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:37:10.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:10 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:37:12.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:12 smithi032 ceph-mon[32461]: pgmap v128: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:37:12.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:12 smithi028 ceph-mon[29390]: pgmap v128: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:37:12.707 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:12 smithi130 ceph-mon[32702]: pgmap v128: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:37:14.482 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:14 smithi032 ceph-mon[32461]: pgmap v129: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-04-25T01:37:14.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:14 smithi028 ceph-mon[29390]: pgmap v129: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-04-25T01:37:14.707 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:14 smithi130 ceph-mon[32702]: pgmap v129: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 28 op/s 2024-04-25T01:37:16.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:16 smithi028 ceph-mon[29390]: pgmap v130: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-25T01:37:16.707 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:16 smithi130 ceph-mon[32702]: pgmap v130: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-25T01:37:16.733 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:16 smithi032 ceph-mon[32461]: pgmap v130: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-25T01:37:18.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:18 smithi028 ceph-mon[29390]: pgmap v131: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 11 op/s 2024-04-25T01:37:18.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:18 smithi130 ceph-mon[32702]: pgmap v131: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 11 op/s 2024-04-25T01:37:18.733 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:18 smithi032 ceph-mon[32461]: pgmap v131: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 11 op/s 2024-04-25T01:37:20.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:20 smithi028 ceph-mon[29390]: pgmap v132: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T01:37:20.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:20 smithi130 ceph-mon[32702]: pgmap v132: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T01:37:20.733 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:20 smithi032 ceph-mon[32461]: pgmap v132: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T01:37:22.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:22 smithi028 ceph-mon[29390]: pgmap v133: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T01:37:22.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:22 smithi130 ceph-mon[32702]: pgmap v133: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T01:37:22.733 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:22 smithi032 ceph-mon[32461]: pgmap v133: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T01:37:24.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:24 smithi028 ceph-mon[29390]: pgmap v134: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T01:37:24.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:24 smithi130 ceph-mon[32702]: pgmap v134: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T01:37:24.733 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:24 smithi032 ceph-mon[32461]: pgmap v134: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T01:37:26.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:26 smithi028 ceph-mon[29390]: pgmap v135: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:37:26.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:26 smithi130 ceph-mon[32702]: pgmap v135: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:37:26.733 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:26 smithi032 ceph-mon[32461]: pgmap v135: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:37:27.561 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:27 smithi028 ceph-mon[29390]: pgmap v136: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:37:27.708 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:27 smithi130 ceph-mon[32702]: pgmap v136: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:37:27.733 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:27 smithi032 ceph-mon[32461]: pgmap v136: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:37:30.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:29 smithi028 ceph-mon[29390]: pgmap v137: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:37:30.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:29 smithi130 ceph-mon[32702]: pgmap v137: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:37:30.233 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:29 smithi032 ceph-mon[32461]: pgmap v137: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:37:30.523 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:37:30.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:37:30.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.663+0000 7f2949b5c640 1 -- 172.21.15.28:0/3371650893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2944101710 msgr2=0x7f2944101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:30.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.663+0000 7f2949b5c640 1 --2- 172.21.15.28:0/3371650893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2944101710 0x7f2944101b90 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f2934009930 tx=0x7f293402a420 comp rx=0 tx=0).stop 2024-04-25T01:37:30.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.663+0000 7f2949b5c640 1 -- 172.21.15.28:0/3371650893 shutdown_connections 2024-04-25T01:37:30.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.663+0000 7f2949b5c640 1 --2- 172.21.15.28:0/3371650893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f29441020d0 0x7f294410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.663+0000 7f2949b5c640 1 --2- 172.21.15.28:0/3371650893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2944101710 0x7f2944101b90 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.663+0000 7f2949b5c640 1 --2- 172.21.15.28:0/3371650893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2944107720 0x7f2944107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.663+0000 7f2949b5c640 1 -- 172.21.15.28:0/3371650893 >> 172.21.15.28:0/3371650893 conn(0x7f29440fd180 msgr2=0x7f29440ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:30.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.663+0000 7f2949b5c640 1 -- 172.21.15.28:0/3371650893 shutdown_connections 2024-04-25T01:37:30.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.663+0000 7f2949b5c640 1 -- 172.21.15.28:0/3371650893 wait complete. 2024-04-25T01:37:30.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.664+0000 7f2949b5c640 1 Processor -- start 2024-04-25T01:37:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.664+0000 7f2949b5c640 1 -- start start 2024-04-25T01:37:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.664+0000 7f2949b5c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2944101710 0x7f2944198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.664+0000 7f2949b5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29441020d0 0x7f2944198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.664+0000 7f2949b5c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2944107720 0x7f294419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.664+0000 7f2949b5c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2944111b30 con 0x7f29441020d0 2024-04-25T01:37:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.664+0000 7f2949b5c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f29441119b0 con 0x7f2944107720 2024-04-25T01:37:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.664+0000 7f2949b5c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2944111cb0 con 0x7f2944101710 2024-04-25T01:37:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.665+0000 7f2942ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29441020d0 0x7f2944198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.665+0000 7f2942ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29441020d0 0x7f2944198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57570/0 (socket says 172.21.15.28:57570) 2024-04-25T01:37:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.665+0000 7f2942ffd640 1 -- 172.21.15.28:0/1719714723 learned_addr learned my addr 172.21.15.28:0/1719714723 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.665+0000 7f29437fe640 1 --2- 172.21.15.28:0/1719714723 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2944101710 0x7f2944198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.665+0000 7f2943fff640 1 --2- 172.21.15.28:0/1719714723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2944107720 0x7f294419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.665+0000 7f2942ffd640 1 -- 172.21.15.28:0/1719714723 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2944101710 msgr2=0x7f2944198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.665+0000 7f2942ffd640 1 --2- 172.21.15.28:0/1719714723 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2944101710 0x7f2944198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.665+0000 7f2942ffd640 1 -- 172.21.15.28:0/1719714723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2944107720 msgr2=0x7f294419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:30.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.665+0000 7f2942ffd640 1 --2- 172.21.15.28:0/1719714723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2944107720 0x7f294419c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.665+0000 7f2942ffd640 1 -- 172.21.15.28:0/1719714723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294419d000 con 0x7f29441020d0 2024-04-25T01:37:30.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.665+0000 7f2942ffd640 1 --2- 172.21.15.28:0/1719714723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29441020d0 0x7f2944198550 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f2934009230 tx=0x7f2934004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:30.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.665+0000 7f2940ff9640 1 -- 172.21.15.28:0/1719714723 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293403f030 con 0x7f29441020d0 2024-04-25T01:37:30.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.665+0000 7f2940ff9640 1 -- 172.21.15.28:0/1719714723 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2934003ad0 con 0x7f29441020d0 2024-04-25T01:37:30.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.665+0000 7f2949b5c640 1 -- 172.21.15.28:0/1719714723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f294419d290 con 0x7f29441020d0 2024-04-25T01:37:30.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.665+0000 7f2940ff9640 1 -- 172.21.15.28:0/1719714723 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2934035070 con 0x7f29441020d0 2024-04-25T01:37:30.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.665+0000 7f2949b5c640 1 -- 172.21.15.28:0/1719714723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29441a4b80 con 0x7f29441020d0 2024-04-25T01:37:30.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.668+0000 7f2940ff9640 1 -- 172.21.15.28:0/1719714723 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2934042030 con 0x7f29441020d0 2024-04-25T01:37:30.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.668+0000 7f294c85f740 1 Processor -- start 2024-04-25T01:37:30.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.668+0000 7f294c85f740 1 -- start start 2024-04-25T01:37:30.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.668+0000 7f294c85f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461b630 0x563c4461ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.668+0000 7f294c85f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461bf50 0x563c446260e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.668+0000 7f294c85f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 0x563c44628a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.668+0000 7f294c85f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563c445806b0 con 0x563c4461bf50 2024-04-25T01:37:30.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.668+0000 7f294c85f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563c44528880 con 0x563c44626620 2024-04-25T01:37:30.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.668+0000 7f294c85f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x563c4457c670 con 0x563c4461b630 2024-04-25T01:37:30.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.668+0000 7f29437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461bf50 0x563c446260e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.669+0000 7f2940ff9640 1 --2- 172.21.15.28:0/1719714723 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f292c0a2d00 0x7f292c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.669+0000 7f2943fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461b630 0x563c4461ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.669+0000 7f29437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461bf50 0x563c446260e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57584/0 (socket says 172.21.15.28:57584) 2024-04-25T01:37:30.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.669+0000 7f29437fe640 1 -- 172.21.15.28:0/2279070234 learned_addr learned my addr 172.21.15.28:0/2279070234 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:30.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.669+0000 7f2942ffd640 1 --2- 172.21.15.28:0/2279070234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 0x563c44628a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.669+0000 7f2940ff9640 1 -- 172.21.15.28:0/1719714723 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f29340efc40 con 0x7f29441020d0 2024-04-25T01:37:30.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.669+0000 7f2943fff640 1 --2- 172.21.15.28:0/1719714723 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f292c0a2d00 0x7f292c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.669+0000 7f2943fff640 1 -- 172.21.15.28:0/2279070234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 msgr2=0x563c44628a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:30.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.669+0000 7f2943fff640 1 --2- 172.21.15.28:0/2279070234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 0x563c44628a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.669+0000 7f2943fff640 1 -- 172.21.15.28:0/2279070234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461bf50 msgr2=0x563c446260e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:30.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.669+0000 7f2943fff640 1 --2- 172.21.15.28:0/2279070234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461bf50 0x563c446260e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.669+0000 7f2943fff640 1 -- 172.21.15.28:0/2279070234 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c4435e7d0 con 0x563c4461b630 2024-04-25T01:37:30.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.669+0000 7f29437fe640 1 --2- 172.21.15.28:0/2279070234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461bf50 0x563c446260e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:37:30.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.669+0000 7f2942ffd640 1 --2- 172.21.15.28:0/2279070234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 0x563c44628a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:37:30.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.669+0000 7f2943fff640 1 --2- 172.21.15.28:0/2279070234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461b630 0x563c4461ba10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f293c013970 tx=0x7f293c013e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ac327c659c5d90c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:30.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.669+0000 7f2943fff640 1 --2- 172.21.15.28:0/1719714723 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f292c0a2d00 0x7f292c0a51c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f293c00a000 tx=0x7f293c01d000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:30.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.669+0000 7f2929ffb640 1 -- 172.21.15.28:0/2279070234 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c022070 con 0x563c4461b630 2024-04-25T01:37:30.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.670+0000 7f2929ffb640 1 -- 172.21.15.28:0/2279070234 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f293c01cc50 con 0x563c4461b630 2024-04-25T01:37:30.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.670+0000 7f2929ffb640 1 -- 172.21.15.28:0/2279070234 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c01dac0 con 0x563c4461b630 2024-04-25T01:37:30.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.670+0000 7f294c85f740 1 -- 172.21.15.28:0/2279070234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461b630 msgr2=0x563c4461ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:30.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.670+0000 7f294c85f740 1 --2- 172.21.15.28:0/2279070234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461b630 0x563c4461ba10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f293c013970 tx=0x7f293c013e30 comp rx=0 tx=0).stop 2024-04-25T01:37:30.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.670+0000 7f294c85f740 1 -- 172.21.15.28:0/2279070234 shutdown_connections 2024-04-25T01:37:30.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.670+0000 7f294c85f740 1 --2- 172.21.15.28:0/2279070234 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 0x563c44628a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.670+0000 7f294c85f740 1 --2- 172.21.15.28:0/2279070234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461bf50 0x563c446260e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.670+0000 7f294c85f740 1 --2- 172.21.15.28:0/2279070234 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461b630 0x563c4461ba10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.670+0000 7f294c85f740 1 -- 172.21.15.28:0/2279070234 >> 172.21.15.28:0/2279070234 conn(0x563c44617950 msgr2=0x563c44617d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:30.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.670+0000 7f294c85f740 1 -- 172.21.15.28:0/2279070234 shutdown_connections 2024-04-25T01:37:30.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.670+0000 7f294c85f740 1 -- 172.21.15.28:0/2279070234 wait complete. 2024-04-25T01:37:30.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.671+0000 7f294c85f740 1 -- start start 2024-04-25T01:37:30.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.671+0000 7f294c85f740 1 Processor -- start 2024-04-25T01:37:30.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.671+0000 7f294c85f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 0x563c446235b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.671+0000 7f294c85f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461bf50 0x563c44623af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.671+0000 7f294c85f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 0x563c44624030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.671+0000 7f294c85f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563c446293b0 con 0x563c4461b630 2024-04-25T01:37:30.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.671+0000 7f294c85f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563c4453bf50 con 0x563c44626620 2024-04-25T01:37:30.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.671+0000 7f294c85f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x563c44629530 con 0x563c4461bf50 2024-04-25T01:37:30.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.671+0000 7f294c85f740 10 client.0 _ll_register_callbacks cb 0x563c44577000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:30.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.671+0000 7f29437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 0x563c446235b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.671+0000 7f29437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 0x563c446235b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57592/0 (socket says 172.21.15.28:57592) 2024-04-25T01:37:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.671+0000 7f29437fe640 1 -- 172.21.15.28:0/1955555637 learned_addr learned my addr 172.21.15.28:0/1955555637 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.671+0000 7f2942ffd640 1 --2- 172.21.15.28:0/1955555637 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461bf50 0x563c44623af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.671+0000 7f2943fff640 1 --2- 172.21.15.28:0/1955555637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 0x563c44624030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.672+0000 7f29437fe640 1 -- 172.21.15.28:0/1955555637 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461bf50 msgr2=0x563c44623af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.672+0000 7f29437fe640 1 --2- 172.21.15.28:0/1955555637 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461bf50 0x563c44623af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.672+0000 7f29437fe640 1 -- 172.21.15.28:0/1955555637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 msgr2=0x563c44624030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.672+0000 7f29437fe640 1 --2- 172.21.15.28:0/1955555637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 0x563c44624030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.672+0000 7f29437fe640 1 -- 172.21.15.28:0/1955555637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c442943f0 con 0x563c4461b630 2024-04-25T01:37:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.672+0000 7f2943fff640 1 --2- 172.21.15.28:0/1955555637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 0x563c44624030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:37:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.672+0000 7f29437fe640 1 --2- 172.21.15.28:0/1955555637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 0x563c446235b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f293800bf90 tx=0x7f29380118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87991be70b8b9c68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.672+0000 7f2922ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.672+0000 7f2922ffd640 1 -- 172.21.15.28:0/1955555637 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293801f060 con 0x563c4461b630 2024-04-25T01:37:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.672+0000 7f2922ffd640 1 -- 172.21.15.28:0/1955555637 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f29380107c0 con 0x563c4461b630 2024-04-25T01:37:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.672+0000 7f294c85f740 1 -- 172.21.15.28:0/1955555637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f293c01cc50 con 0x563c4461b630 2024-04-25T01:37:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.672+0000 7f2922ffd640 1 -- 172.21.15.28:0/1955555637 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2938026050 con 0x563c4461b630 2024-04-25T01:37:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.672+0000 7f294c85f740 10 client.16992 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.672+0000 7f294c85f740 1 -- 172.21.15.28:0/1955555637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563c442540f0 con 0x563c4461b630 2024-04-25T01:37:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.672+0000 7f2922ffd640 1 -- 172.21.15.28:0/1955555637 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f293c01cc50 con 0x563c4461b630 2024-04-25T01:37:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.673+0000 7f2922ffd640 1 -- 172.21.15.28:0/1955555637 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f293800fbb0 con 0x563c4461b630 2024-04-25T01:37:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.673+0000 7f294c85f740 10 client.16992 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.673+0000 7f294c85f740 10 client.16992 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.673+0000 7f294c85f740 20 client.16992 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.673+0000 7f294c85f740 1 --2- 172.21.15.28:0/1955555637 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563c44636900 0x563c44638d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.673+0000 7f294c85f740 4 client.16992 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:37:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.673+0000 7f294c85f740 1 -- 172.21.15.28:0/1955555637 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c4458dbf0 con 0x563c44636900 2024-04-25T01:37:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.673+0000 7f2942ffd640 1 --2- 172.21.15.28:0/1955555637 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563c44636900 0x563c44638d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.673+0000 7f2942ffd640 1 --2- 172.21.15.28:0/1955555637 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563c44636900 0x563c44638d10 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26779a12d26aab9a server_cookie=cdf3ef9f5c25465a in_seq=0 out_seq=0 2024-04-25T01:37:30.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.673+0000 7f2922ffd640 10 client.16992 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:30.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.674+0000 7f2922ffd640 1 -- 172.21.15.28:0/1955555637 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563c4458dbf0 con 0x563c44636900 2024-04-25T01:37:30.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.674+0000 7f2922ffd640 10 client.16992 handle_command_reply: tid=0 2024-04-25T01:37:30.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.674+0000 7f294c85f740 1 client.16992 shutdown 2024-04-25T01:37:30.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.674+0000 7f294c85f740 1 -- 172.21.15.28:0/1955555637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 msgr2=0x563c446235b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:30.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.674+0000 7f294c85f740 1 --2- 172.21.15.28:0/1955555637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 0x563c446235b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f293800bf90 tx=0x7f29380118d0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.674+0000 7f294c85f740 1 -- 172.21.15.28:0/1955555637 shutdown_connections 2024-04-25T01:37:30.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.674+0000 7f294c85f740 1 --2- 172.21.15.28:0/1955555637 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563c44636900 0x563c44638d10 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.674+0000 7f294c85f740 1 --2- 172.21.15.28:0/1955555637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 0x563c44624030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.674+0000 7f2922ffd640 0 client.16992 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:30.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.674+0000 7f294c85f740 1 --2- 172.21.15.28:0/1955555637 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461bf50 0x563c44623af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.674+0000 7f294c85f740 1 --2- 172.21.15.28:0/1955555637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 0x563c446235b0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.674+0000 7f294c85f740 1 -- 172.21.15.28:0/1955555637 >> 172.21.15.28:0/1955555637 conn(0x563c44617950 msgr2=0x563c4461d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:30.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.674+0000 7f294c85f740 1 -- 172.21.15.28:0/1955555637 shutdown_connections 2024-04-25T01:37:30.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.674+0000 7f294c85f740 1 -- 172.21.15.28:0/1955555637 wait complete. 2024-04-25T01:37:30.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.675+0000 7f294c85f740 20 client.16992 trim_cache size 0 max 16384 2024-04-25T01:37:30.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.692+0000 7f294c85f740 1 Processor -- start 2024-04-25T01:37:30.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.692+0000 7f294c85f740 1 -- start start 2024-04-25T01:37:30.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.692+0000 7f294c85f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 0x563c44632270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.692+0000 7f294c85f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461bf50 0x563c4461ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.692+0000 7f294c85f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 0x563c44622e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.692+0000 7f294c85f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f29441aea10 con 0x563c4461b630 2024-04-25T01:37:30.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.692+0000 7f294c85f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f29441b0550 con 0x563c44626620 2024-04-25T01:37:30.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.692+0000 7f294c85f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f29441af0e0 con 0x563c4461bf50 2024-04-25T01:37:30.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.693+0000 7f29437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 0x563c44632270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.693+0000 7f29437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 0x563c44632270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57600/0 (socket says 172.21.15.28:57600) 2024-04-25T01:37:30.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.693+0000 7f29437fe640 1 -- 172.21.15.28:0/3933371011 learned_addr learned my addr 172.21.15.28:0/3933371011 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:30.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.693+0000 7f2943fff640 1 --2- 172.21.15.28:0/3933371011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 0x563c44622e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.693+0000 7f2942ffd640 1 --2- 172.21.15.28:0/3933371011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461bf50 0x563c4461ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.693+0000 7f29437fe640 1 -- 172.21.15.28:0/3933371011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461bf50 msgr2=0x563c4461ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:30.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.693+0000 7f29437fe640 1 --2- 172.21.15.28:0/3933371011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461bf50 0x563c4461ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.693+0000 7f29437fe640 1 -- 172.21.15.28:0/3933371011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 msgr2=0x563c44622e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:30.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.693+0000 7f29437fe640 1 --2- 172.21.15.28:0/3933371011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 0x563c44622e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.693+0000 7f29437fe640 1 -- 172.21.15.28:0/3933371011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c44609090 con 0x563c4461b630 2024-04-25T01:37:30.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.693+0000 7f2942ffd640 1 --2- 172.21.15.28:0/3933371011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461bf50 0x563c4461ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:30.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.693+0000 7f29437fe640 1 --2- 172.21.15.28:0/3933371011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 0x563c44632270 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f2938005d00 tx=0x7f2938019720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b6f268d8f0e90d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:30.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.693+0000 7f29297fa640 1 -- 172.21.15.28:0/3933371011 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293802f070 con 0x563c4461b630 2024-04-25T01:37:30.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.693+0000 7f29297fa640 1 -- 172.21.15.28:0/3933371011 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2938019dc0 con 0x563c4461b630 2024-04-25T01:37:30.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.693+0000 7f29297fa640 1 -- 172.21.15.28:0/3933371011 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29380107c0 con 0x563c4461b630 2024-04-25T01:37:30.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.694+0000 7f294c85f740 1 -- 172.21.15.28:0/3933371011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 msgr2=0x563c44632270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:30.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.694+0000 7f294c85f740 1 --2- 172.21.15.28:0/3933371011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 0x563c44632270 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f2938005d00 tx=0x7f2938019720 comp rx=0 tx=0).stop 2024-04-25T01:37:30.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.694+0000 7f294c85f740 1 -- 172.21.15.28:0/3933371011 shutdown_connections 2024-04-25T01:37:30.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.694+0000 7f294c85f740 1 --2- 172.21.15.28:0/3933371011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 0x563c44622e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.694+0000 7f294c85f740 1 --2- 172.21.15.28:0/3933371011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461bf50 0x563c4461ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.694+0000 7f294c85f740 1 --2- 172.21.15.28:0/3933371011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 0x563c44632270 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.694+0000 7f294c85f740 1 -- 172.21.15.28:0/3933371011 >> 172.21.15.28:0/3933371011 conn(0x563c44617950 msgr2=0x563c44623e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:30.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.694+0000 7f294c85f740 1 -- 172.21.15.28:0/3933371011 shutdown_connections 2024-04-25T01:37:30.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.694+0000 7f294c85f740 1 -- 172.21.15.28:0/3933371011 wait complete. 2024-04-25T01:37:30.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.694+0000 7f294c85f740 1 -- start start 2024-04-25T01:37:30.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.694+0000 7f294c85f740 1 Processor -- start 2024-04-25T01:37:30.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f294c85f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 0x563c4462e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f294c85f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461bf50 0x563c44629d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f294c85f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 0x563c4462a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f294c85f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f29441afe80 con 0x563c4461b630 2024-04-25T01:37:30.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f294c85f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f29441af7b0 con 0x563c44626620 2024-04-25T01:37:30.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f294c85f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f29441ac9f0 con 0x563c4461bf50 2024-04-25T01:37:30.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f294c85f740 10 client.0 _ll_register_callbacks cb 0x7f293800b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:30.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f29437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 0x563c4462e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f29437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 0x563c4462e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57608/0 (socket says 172.21.15.28:57608) 2024-04-25T01:37:30.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f29437fe640 1 -- 172.21.15.28:0/3892190893 learned_addr learned my addr 172.21.15.28:0/3892190893 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:30.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f2942ffd640 1 --2- 172.21.15.28:0/3892190893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461bf50 0x563c44629d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f2943fff640 1 --2- 172.21.15.28:0/3892190893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 0x563c4462a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f29437fe640 1 -- 172.21.15.28:0/3892190893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461bf50 msgr2=0x563c44629d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:30.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f29437fe640 1 --2- 172.21.15.28:0/3892190893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461bf50 0x563c44629d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f29437fe640 1 -- 172.21.15.28:0/3892190893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 msgr2=0x563c4462a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:30.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f29437fe640 1 --2- 172.21.15.28:0/3892190893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 0x563c4462a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f29437fe640 1 -- 172.21.15.28:0/3892190893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c4462a870 con 0x563c4461b630 2024-04-25T01:37:30.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f2942ffd640 1 --2- 172.21.15.28:0/3892190893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461bf50 0x563c44629d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:30.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f2943fff640 1 --2- 172.21.15.28:0/3892190893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 0x563c4462a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:30.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f29437fe640 1 --2- 172.21.15.28:0/3892190893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 0x563c4462e9f0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f29380052c0 tx=0x7f2938005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65aa5b801012a387 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:30.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f2922ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:30.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f2922ffd640 1 -- 172.21.15.28:0/3892190893 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2938038020 con 0x563c4461b630 2024-04-25T01:37:30.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f2922ffd640 1 -- 172.21.15.28:0/3892190893 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f293800b410 con 0x563c4461b630 2024-04-25T01:37:30.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.695+0000 7f294c85f740 1 -- 172.21.15.28:0/3892190893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2938019dc0 con 0x563c4461b630 2024-04-25T01:37:30.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.696+0000 7f2922ffd640 1 -- 172.21.15.28:0/3892190893 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2938043050 con 0x563c4461b630 2024-04-25T01:37:30.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.696+0000 7f294c85f740 10 client.16998 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:30.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.696+0000 7f294c85f740 1 -- 172.21.15.28:0/3892190893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563c4469cbf0 con 0x563c4461b630 2024-04-25T01:37:30.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.696+0000 7f2922ffd640 1 -- 172.21.15.28:0/3892190893 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2938019dc0 con 0x563c4461b630 2024-04-25T01:37:30.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.696+0000 7f2922ffd640 1 -- 172.21.15.28:0/3892190893 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f293800f3c0 con 0x563c4461b630 2024-04-25T01:37:30.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.696+0000 7f294c85f740 10 client.16998 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:30.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.696+0000 7f294c85f740 10 client.16998 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:30.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.696+0000 7f294c85f740 20 client.16998 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:30.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.696+0000 7f294c85f740 1 --2- 172.21.15.28:0/3892190893 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563c4469d2d0 0x563c4469f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.696+0000 7f294c85f740 4 client.16998 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:37:30.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.696+0000 7f294c85f740 1 -- 172.21.15.28:0/3892190893 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563c4461f1b0 con 0x563c4469d2d0 2024-04-25T01:37:30.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.696+0000 7f2942ffd640 1 --2- 172.21.15.28:0/3892190893 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563c4469d2d0 0x563c4469f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.697+0000 7f2942ffd640 1 --2- 172.21.15.28:0/3892190893 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563c4469d2d0 0x563c4469f6f0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47e81b040eec2b83 server_cookie=92fea505cc10280f in_seq=0 out_seq=0 2024-04-25T01:37:30.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.697+0000 7f2922ffd640 10 client.16998 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:30.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.697+0000 7f2922ffd640 1 -- 172.21.15.28:0/3892190893 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563c4461f1b0 con 0x563c4469d2d0 2024-04-25T01:37:30.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.697+0000 7f2922ffd640 10 client.16998 handle_command_reply: tid=0 2024-04-25T01:37:30.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.697+0000 7f294c85f740 1 client.16998 shutdown 2024-04-25T01:37:30.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.697+0000 7f294c85f740 1 -- 172.21.15.28:0/3892190893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 msgr2=0x563c4462e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:30.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.697+0000 7f294c85f740 1 --2- 172.21.15.28:0/3892190893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 0x563c4462e9f0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f29380052c0 tx=0x7f2938005770 comp rx=0 tx=0).stop 2024-04-25T01:37:30.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.697+0000 7f294c85f740 1 -- 172.21.15.28:0/3892190893 shutdown_connections 2024-04-25T01:37:30.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.697+0000 7f294c85f740 1 --2- 172.21.15.28:0/3892190893 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563c4469d2d0 0x563c4469f6f0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.697+0000 7f294c85f740 1 --2- 172.21.15.28:0/3892190893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563c44626620 0x563c4462a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.697+0000 7f294c85f740 1 --2- 172.21.15.28:0/3892190893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563c4461bf50 0x563c44629d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.697+0000 7f2922ffd640 0 client.16998 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:30.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.697+0000 7f294c85f740 1 --2- 172.21.15.28:0/3892190893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4461b630 0x563c4462e9f0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.697+0000 7f294c85f740 1 -- 172.21.15.28:0/3892190893 >> 172.21.15.28:0/3892190893 conn(0x563c44617950 msgr2=0x563c44616260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:30.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.697+0000 7f294c85f740 1 -- 172.21.15.28:0/3892190893 shutdown_connections 2024-04-25T01:37:30.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.697+0000 7f294c85f740 1 -- 172.21.15.28:0/3892190893 wait complete. 2024-04-25T01:37:30.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.697+0000 7f294c85f740 20 client.16998 trim_cache size 0 max 16384 2024-04-25T01:37:30.682 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:37:30.682 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:37:30.682 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:37:30.682 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:37:30.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.698+0000 7f2949b5c640 1 -- 172.21.15.28:0/1719714723 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f292c0a2d00 msgr2=0x7f292c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:30.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.698+0000 7f2949b5c640 1 --2- 172.21.15.28:0/1719714723 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f292c0a2d00 0x7f292c0a51c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f293c00a000 tx=0x7f293c01d000 comp rx=0 tx=0).stop 2024-04-25T01:37:30.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.698+0000 7f2949b5c640 1 -- 172.21.15.28:0/1719714723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29441020d0 msgr2=0x7f2944198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:30.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.698+0000 7f2949b5c640 1 --2- 172.21.15.28:0/1719714723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29441020d0 0x7f2944198550 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f2934009230 tx=0x7f2934004270 comp rx=0 tx=0).stop 2024-04-25T01:37:30.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.698+0000 7f2949b5c640 1 -- 172.21.15.28:0/1719714723 shutdown_connections 2024-04-25T01:37:30.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.698+0000 7f2949b5c640 1 --2- 172.21.15.28:0/1719714723 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f292c0a2d00 0x7f292c0a51c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.698+0000 7f2949b5c640 1 --2- 172.21.15.28:0/1719714723 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2944107720 0x7f294419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.698+0000 7f2949b5c640 1 --2- 172.21.15.28:0/1719714723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29441020d0 0x7f2944198550 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.698+0000 7f2949b5c640 1 --2- 172.21.15.28:0/1719714723 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2944101710 0x7f2944198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.698+0000 7f2949b5c640 1 -- 172.21.15.28:0/1719714723 >> 172.21.15.28:0/1719714723 conn(0x7f29440fd180 msgr2=0x7f29440fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:30.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.698+0000 7f2949b5c640 1 -- 172.21.15.28:0/1719714723 shutdown_connections 2024-04-25T01:37:30.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.698+0000 7f2949b5c640 1 -- 172.21.15.28:0/1719714723 wait complete. 2024-04-25T01:37:30.687 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:37:30.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:37:30.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.812+0000 7f9ccf472640 1 -- 172.21.15.28:0/2474341182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cc8106630 msgr2=0x7f9cc8106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:30.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.812+0000 7f9ccf472640 1 --2- 172.21.15.28:0/2474341182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cc8106630 0x7f9cc8106a10 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4009930 tx=0x7f9cc402a420 comp rx=0 tx=0).stop 2024-04-25T01:37:30.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.812+0000 7f9ccf472640 1 -- 172.21.15.28:0/2474341182 shutdown_connections 2024-04-25T01:37:30.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.812+0000 7f9ccf472640 1 --2- 172.21.15.28:0/2474341182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cc8100fe0 0x7f9cc810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.812+0000 7f9ccf472640 1 --2- 172.21.15.28:0/2474341182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cc8100620 0x7f9cc8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.812+0000 7f9ccf472640 1 --2- 172.21.15.28:0/2474341182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cc8106630 0x7f9cc8106a10 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.812+0000 7f9ccf472640 1 -- 172.21.15.28:0/2474341182 >> 172.21.15.28:0/2474341182 conn(0x7f9cc8077740 msgr2=0x7f9cc8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:30.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.812+0000 7f9ccf472640 1 -- 172.21.15.28:0/2474341182 shutdown_connections 2024-04-25T01:37:30.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.812+0000 7f9ccf472640 1 -- 172.21.15.28:0/2474341182 wait complete. 2024-04-25T01:37:30.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.812+0000 7f9ccf472640 1 Processor -- start 2024-04-25T01:37:30.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.812+0000 7f9ccf472640 1 -- start start 2024-04-25T01:37:30.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9ccf472640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cc8100620 0x7f9cc8198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9ccf472640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cc8100fe0 0x7f9cc8198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9ccf472640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cc8106630 0x7f9cc819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9ccf472640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc8110a40 con 0x7f9cc8100620 2024-04-25T01:37:30.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9ccf472640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc81108c0 con 0x7f9cc8100fe0 2024-04-25T01:37:30.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9ccf472640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc8110bc0 con 0x7f9cc8106630 2024-04-25T01:37:30.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9ccd20e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cc8100620 0x7f9cc8198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9ccd20e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cc8100620 0x7f9cc8198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57618/0 (socket says 172.21.15.28:57618) 2024-04-25T01:37:30.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9ccd20e640 1 -- 172.21.15.28:0/3150519237 learned_addr learned my addr 172.21.15.28:0/3150519237 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:30.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9ccda0f640 1 --2- 172.21.15.28:0/3150519237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cc8106630 0x7f9cc819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9ccd20e640 1 -- 172.21.15.28:0/3150519237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cc8106630 msgr2=0x7f9cc819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:30.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9ccca0d640 1 --2- 172.21.15.28:0/3150519237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cc8100fe0 0x7f9cc8198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9ccd20e640 1 --2- 172.21.15.28:0/3150519237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cc8106630 0x7f9cc819c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9ccd20e640 1 -- 172.21.15.28:0/3150519237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cc8100fe0 msgr2=0x7f9cc8198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:30.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9ccd20e640 1 --2- 172.21.15.28:0/3150519237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cc8100fe0 0x7f9cc8198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:30.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9ccd20e640 1 -- 172.21.15.28:0/3150519237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc819cff0 con 0x7f9cc8100620 2024-04-25T01:37:30.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9ccd20e640 1 --2- 172.21.15.28:0/3150519237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cc8100620 0x7f9cc8198000 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4009230 tx=0x7f9cc4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:30.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9cbe7fc640 1 -- 172.21.15.28:0/3150519237 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc403f030 con 0x7f9cc8100620 2024-04-25T01:37:30.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9ccf472640 1 -- 172.21.15.28:0/3150519237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cc819d280 con 0x7f9cc8100620 2024-04-25T01:37:30.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9cbe7fc640 1 -- 172.21.15.28:0/3150519237 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cc4004440 con 0x7f9cc8100620 2024-04-25T01:37:30.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9cbe7fc640 1 -- 172.21.15.28:0/3150519237 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc4035070 con 0x7f9cc8100620 2024-04-25T01:37:30.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.813+0000 7f9ccf472640 1 -- 172.21.15.28:0/3150519237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cc81a4b70 con 0x7f9cc8100620 2024-04-25T01:37:30.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.814+0000 7f9ccf472640 1 -- 172.21.15.28:0/3150519237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cc8068560 con 0x7f9cc8100620 2024-04-25T01:37:30.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.816+0000 7f9cbe7fc640 1 -- 172.21.15.28:0/3150519237 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9cc4042030 con 0x7f9cc8100620 2024-04-25T01:37:30.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.816+0000 7f9cbe7fc640 1 --2- 172.21.15.28:0/3150519237 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9cb40a2d00 0x7f9cb40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:30.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.816+0000 7f9ccca0d640 1 --2- 172.21.15.28:0/3150519237 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9cb40a2d00 0x7f9cb40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:30.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.816+0000 7f9cbe7fc640 1 -- 172.21.15.28:0/3150519237 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f9cc40effe0 con 0x7f9cc8100620 2024-04-25T01:37:30.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.817+0000 7f9ccca0d640 1 --2- 172.21.15.28:0/3150519237 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9cb40a2d00 0x7f9cb40a51c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8102770 tx=0x7f9cb8005f90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:30.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:30.820+0000 7f9cbe7fc640 1 -- 172.21.15.28:0/3150519237 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9cc40bc8c0 con 0x7f9cc8100620 2024-04-25T01:37:31.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.041+0000 7f9ccf472640 1 -- 172.21.15.28:0/3150519237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9cc8109ba0 con 0x7f9cc8100620 2024-04-25T01:37:31.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.042+0000 7f9cbe7fc640 1 -- 172.21.15.28:0/3150519237 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7f9cc40bc1c0 con 0x7f9cc8100620 2024-04-25T01:37:31.020 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:37:31.020 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:37:31.020 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:37:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.044+0000 7f9ccf472640 1 -- 172.21.15.28:0/3150519237 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9cb40a2d00 msgr2=0x7f9cb40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.044+0000 7f9ccf472640 1 --2- 172.21.15.28:0/3150519237 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9cb40a2d00 0x7f9cb40a51c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8102770 tx=0x7f9cb8005f90 comp rx=0 tx=0).stop 2024-04-25T01:37:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.044+0000 7f9ccf472640 1 -- 172.21.15.28:0/3150519237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cc8100620 msgr2=0x7f9cc8198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.044+0000 7f9ccf472640 1 --2- 172.21.15.28:0/3150519237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cc8100620 0x7f9cc8198000 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4009230 tx=0x7f9cc4004040 comp rx=0 tx=0).stop 2024-04-25T01:37:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.044+0000 7f9ccf472640 1 -- 172.21.15.28:0/3150519237 shutdown_connections 2024-04-25T01:37:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.044+0000 7f9ccf472640 1 --2- 172.21.15.28:0/3150519237 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9cb40a2d00 0x7f9cb40a51c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.044+0000 7f9ccf472640 1 --2- 172.21.15.28:0/3150519237 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cc8106630 0x7f9cc819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.044+0000 7f9ccf472640 1 --2- 172.21.15.28:0/3150519237 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cc8100fe0 0x7f9cc8198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.044+0000 7f9ccf472640 1 --2- 172.21.15.28:0/3150519237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cc8100620 0x7f9cc8198000 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.044+0000 7f9ccf472640 1 -- 172.21.15.28:0/3150519237 >> 172.21.15.28:0/3150519237 conn(0x7f9cc8077740 msgr2=0x7f9cc8068c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.044+0000 7f9ccf472640 1 -- 172.21.15.28:0/3150519237 shutdown_connections 2024-04-25T01:37:31.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.044+0000 7f9ccf472640 1 -- 172.21.15.28:0/3150519237 wait complete. 2024-04-25T01:37:31.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:37:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.160+0000 7f855e2ee640 1 -- 172.21.15.28:0/2019584242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8558100620 msgr2=0x7f8558100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.160+0000 7f855e2ee640 1 --2- 172.21.15.28:0/2019584242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8558100620 0x7f8558100aa0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f8540009930 tx=0x7f854002a420 comp rx=0 tx=0).stop 2024-04-25T01:37:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.160+0000 7f855e2ee640 1 -- 172.21.15.28:0/2019584242 shutdown_connections 2024-04-25T01:37:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.160+0000 7f855e2ee640 1 --2- 172.21.15.28:0/2019584242 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8558100fe0 0x7f855810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.160+0000 7f855e2ee640 1 --2- 172.21.15.28:0/2019584242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8558100620 0x7f8558100aa0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.160+0000 7f855e2ee640 1 --2- 172.21.15.28:0/2019584242 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8558106630 0x7f8558106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.160+0000 7f855e2ee640 1 -- 172.21.15.28:0/2019584242 >> 172.21.15.28:0/2019584242 conn(0x7f85580776a0 msgr2=0x7f8558077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.160+0000 7f855e2ee640 1 -- 172.21.15.28:0/2019584242 shutdown_connections 2024-04-25T01:37:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.160+0000 7f855e2ee640 1 -- 172.21.15.28:0/2019584242 wait complete. 2024-04-25T01:37:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.160+0000 7f855e2ee640 1 Processor -- start 2024-04-25T01:37:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.160+0000 7f855e2ee640 1 -- start start 2024-04-25T01:37:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f855e2ee640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8558100620 0x7f8558197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f855e2ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8558100fe0 0x7f85581984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f855e2ee640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8558106630 0x7f855819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f855e2ee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8558110a40 con 0x7f8558100fe0 2024-04-25T01:37:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f855e2ee640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f85581108c0 con 0x7f8558106630 2024-04-25T01:37:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f855e2ee640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8558110bc0 con 0x7f8558100620 2024-04-25T01:37:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f85577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8558100fe0 0x7f85581984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f85577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8558100fe0 0x7f85581984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57642/0 (socket says 172.21.15.28:57642) 2024-04-25T01:37:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f85577fe640 1 -- 172.21.15.28:0/1164670851 learned_addr learned my addr 172.21.15.28:0/1164670851 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f8557fff640 1 --2- 172.21.15.28:0/1164670851 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8558100620 0x7f8558197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f855c88b640 1 --2- 172.21.15.28:0/1164670851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8558106630 0x7f855819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f85577fe640 1 -- 172.21.15.28:0/1164670851 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8558100620 msgr2=0x7f8558197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f85577fe640 1 --2- 172.21.15.28:0/1164670851 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8558100620 0x7f8558197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f85577fe640 1 -- 172.21.15.28:0/1164670851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8558106630 msgr2=0x7f855819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f85577fe640 1 --2- 172.21.15.28:0/1164670851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8558106630 0x7f855819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f85577fe640 1 -- 172.21.15.28:0/1164670851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855819cf50 con 0x7f8558100fe0 2024-04-25T01:37:31.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f85577fe640 1 --2- 172.21.15.28:0/1164670851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8558100fe0 0x7f85581984f0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f8540009230 tx=0x7f8540004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f85557fa640 1 -- 172.21.15.28:0/1164670851 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854003f030 con 0x7f8558100fe0 2024-04-25T01:37:31.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f85557fa640 1 -- 172.21.15.28:0/1164670851 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8540003ad0 con 0x7f8558100fe0 2024-04-25T01:37:31.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f855e2ee640 1 -- 172.21.15.28:0/1164670851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f855819d1e0 con 0x7f8558100fe0 2024-04-25T01:37:31.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f85557fa640 1 -- 172.21.15.28:0/1164670851 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8540035070 con 0x7f8558100fe0 2024-04-25T01:37:31.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.161+0000 7f855e2ee640 1 -- 172.21.15.28:0/1164670851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85581a4ad0 con 0x7f8558100fe0 2024-04-25T01:37:31.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.164+0000 7f85557fa640 1 -- 172.21.15.28:0/1164670851 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8540042030 con 0x7f8558100fe0 2024-04-25T01:37:31.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.164+0000 7f8560ffa740 1 Processor -- start 2024-04-25T01:37:31.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.164+0000 7f8560ffa740 1 -- start start 2024-04-25T01:37:31.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.164+0000 7f8560ffa740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a60580 0x561305a60960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.164+0000 7f8560ffa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60ea0 0x561305a6b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.164+0000 7f8560ffa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 0x561305a6d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.164+0000 7f8560ffa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5613059c5670 con 0x561305a6b570 2024-04-25T01:37:31.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.164+0000 7f8560ffa740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56130596d970 con 0x561305a60580 2024-04-25T01:37:31.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.164+0000 7f8560ffa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5613059c1630 con 0x561305a60ea0 2024-04-25T01:37:31.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.164+0000 7f85557fa640 1 --2- 172.21.15.28:0/1164670851 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f85440a2d00 0x7f85440a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.164+0000 7f85577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 0x561305a6d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.165+0000 7f855c88b640 1 --2- 172.21.15.28:0/1164670851 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f85440a2d00 0x7f85440a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.165+0000 7f8557fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60ea0 0x561305a6b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.165+0000 7f855c88b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a60580 0x561305a60960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.165+0000 7f85577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 0x561305a6d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57654/0 (socket says 172.21.15.28:57654) 2024-04-25T01:37:31.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.165+0000 7f85577fe640 1 -- 172.21.15.28:0/2380481472 learned_addr learned my addr 172.21.15.28:0/2380481472 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.165+0000 7f8557fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60ea0 0x561305a6b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.28:42736/0 (socket says 172.21.15.28:42736) 2024-04-25T01:37:31.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.165+0000 7f85557fa640 1 -- 172.21.15.28:0/1164670851 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f85400efc60 con 0x7f8558100fe0 2024-04-25T01:37:31.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.165+0000 7f85577fe640 1 -- 172.21.15.28:0/2380481472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60ea0 msgr2=0x561305a6b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.165+0000 7f85577fe640 1 --2- 172.21.15.28:0/2380481472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60ea0 0x561305a6b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.165+0000 7f85577fe640 1 -- 172.21.15.28:0/2380481472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a60580 msgr2=0x561305a60960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.165+0000 7f85577fe640 1 --2- 172.21.15.28:0/2380481472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a60580 0x561305a60960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.165+0000 7f85577fe640 1 -- 172.21.15.28:0/2380481472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561305991400 con 0x561305a6b570 2024-04-25T01:37:31.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.165+0000 7f855c88b640 1 --2- 172.21.15.28:0/2380481472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a60580 0x561305a60960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:37:31.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.165+0000 7f85577fe640 1 --2- 172.21.15.28:0/2380481472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 0x561305a6d960 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f854008ee80 tx=0x7f854008de00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=577f6ce2ff8f02f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.165+0000 7f855c88b640 1 --2- 172.21.15.28:0/1164670851 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f85440a2d00 0x7f85440a51c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8550013d20 tx=0x7f8550007390 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.165+0000 7f854e7fc640 1 -- 172.21.15.28:0/2380481472 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8540049070 con 0x561305a6b570 2024-04-25T01:37:31.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.165+0000 7f854e7fc640 1 -- 172.21.15.28:0/2380481472 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85400a5040 con 0x561305a6b570 2024-04-25T01:37:31.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.165+0000 7f854e7fc640 1 -- 172.21.15.28:0/2380481472 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85400ed2f0 con 0x561305a6b570 2024-04-25T01:37:31.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.166+0000 7f8560ffa740 1 -- 172.21.15.28:0/2380481472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 msgr2=0x561305a6d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.166+0000 7f8560ffa740 1 --2- 172.21.15.28:0/2380481472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 0x561305a6d960 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f854008ee80 tx=0x7f854008de00 comp rx=0 tx=0).stop 2024-04-25T01:37:31.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.166+0000 7f8560ffa740 1 -- 172.21.15.28:0/2380481472 shutdown_connections 2024-04-25T01:37:31.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.166+0000 7f8560ffa740 1 --2- 172.21.15.28:0/2380481472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 0x561305a6d960 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.166+0000 7f8560ffa740 1 --2- 172.21.15.28:0/2380481472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60ea0 0x561305a6b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.166+0000 7f8560ffa740 1 --2- 172.21.15.28:0/2380481472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a60580 0x561305a60960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.166+0000 7f8560ffa740 1 -- 172.21.15.28:0/2380481472 >> 172.21.15.28:0/2380481472 conn(0x561305a5c8a0 msgr2=0x561305a5cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.166+0000 7f8560ffa740 1 -- 172.21.15.28:0/2380481472 shutdown_connections 2024-04-25T01:37:31.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.166+0000 7f8560ffa740 1 -- 172.21.15.28:0/2380481472 wait complete. 2024-04-25T01:37:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.166+0000 7f8560ffa740 1 -- start start 2024-04-25T01:37:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.167+0000 7f8560ffa740 1 Processor -- start 2024-04-25T01:37:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.167+0000 7f8560ffa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60580 0x561305a6aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.167+0000 7f8560ffa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a60ea0 0x561305a681c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.167+0000 7f8560ffa740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a6b570 0x561305a68720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.167+0000 7f8560ffa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561305a6e120 con 0x561305a60ea0 2024-04-25T01:37:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.167+0000 7f8560ffa740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561305981050 con 0x561305a6b570 2024-04-25T01:37:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.167+0000 7f8560ffa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x561305a6e2a0 con 0x561305a60580 2024-04-25T01:37:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.167+0000 7f8560ffa740 10 client.0 _ll_register_callbacks cb 0x5613056d8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.167+0000 7f85577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a60ea0 0x561305a681c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.167+0000 7f85577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a60ea0 0x561305a681c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57668/0 (socket says 172.21.15.28:57668) 2024-04-25T01:37:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.167+0000 7f85577fe640 1 -- 172.21.15.28:0/779646097 learned_addr learned my addr 172.21.15.28:0/779646097 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.167+0000 7f8557fff640 1 --2- 172.21.15.28:0/779646097 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60580 0x561305a6aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.167+0000 7f85577fe640 1 -- 172.21.15.28:0/779646097 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60580 msgr2=0x561305a6aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.167+0000 7f85577fe640 1 --2- 172.21.15.28:0/779646097 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60580 0x561305a6aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.167+0000 7f85577fe640 1 -- 172.21.15.28:0/779646097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a6b570 msgr2=0x561305a68720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.167+0000 7f85577fe640 1 --2- 172.21.15.28:0/779646097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a6b570 0x561305a68720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.167+0000 7f85577fe640 1 -- 172.21.15.28:0/779646097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561305952dc0 con 0x561305a60ea0 2024-04-25T01:37:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.167+0000 7f85577fe640 1 --2- 172.21.15.28:0/779646097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a60ea0 0x561305a681c0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f85400ed120 tx=0x7f85400ed010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8acea43c5dc8c8ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.167+0000 7f85377fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.168+0000 7f85377fe640 1 -- 172.21.15.28:0/779646097 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85400be070 con 0x561305a60ea0 2024-04-25T01:37:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.168+0000 7f85377fe640 1 -- 172.21.15.28:0/779646097 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f85400ed2f0 con 0x561305a60ea0 2024-04-25T01:37:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.168+0000 7f8560ffa740 1 -- 172.21.15.28:0/779646097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85400a5040 con 0x561305a60ea0 2024-04-25T01:37:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.168+0000 7f85377fe640 1 -- 172.21.15.28:0/779646097 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8540049040 con 0x561305a60ea0 2024-04-25T01:37:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.168+0000 7f8560ffa740 10 client.17016 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.168+0000 7f8560ffa740 1 -- 172.21.15.28:0/779646097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5613056d93f0 con 0x561305a60ea0 2024-04-25T01:37:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.168+0000 7f85377fe640 1 -- 172.21.15.28:0/779646097 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85400a5040 con 0x561305a60ea0 2024-04-25T01:37:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.168+0000 7f85377fe640 1 -- 172.21.15.28:0/779646097 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f854008b020 con 0x561305a60ea0 2024-04-25T01:37:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.168+0000 7f8560ffa740 10 client.17016 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.168+0000 7f8560ffa740 10 client.17016 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:37:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.168+0000 7f8560ffa740 20 client.17016 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.168+0000 7f8560ffa740 1 --2- 172.21.15.28:0/779646097 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x561305a7b1b0 0x561305a7d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.168+0000 7f8560ffa740 4 client.17016 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:37:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.168+0000 7f8560ffa740 1 -- 172.21.15.28:0/779646097 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613059d2bb0 con 0x561305a7b1b0 2024-04-25T01:37:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.168+0000 7f8557fff640 1 --2- 172.21.15.28:0/779646097 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x561305a7b1b0 0x561305a7d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.169+0000 7f8557fff640 1 --2- 172.21.15.28:0/779646097 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x561305a7b1b0 0x561305a7d5e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=705d412ab34f5149 server_cookie=3ce8a7ba2915ad30 in_seq=0 out_seq=0 2024-04-25T01:37:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.169+0000 7f85377fe640 10 client.17016 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:37:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.170+0000 7f85377fe640 1 -- 172.21.15.28:0/779646097 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5613059d2bb0 con 0x561305a7b1b0 2024-04-25T01:37:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.170+0000 7f85377fe640 10 client.17016 handle_command_reply: tid=0 2024-04-25T01:37:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.170+0000 7f8560ffa740 1 client.17016 shutdown 2024-04-25T01:37:31.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.170+0000 7f8560ffa740 1 -- 172.21.15.28:0/779646097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a60ea0 msgr2=0x561305a681c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.170+0000 7f8560ffa740 1 --2- 172.21.15.28:0/779646097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a60ea0 0x561305a681c0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f85400ed120 tx=0x7f85400ed010 comp rx=0 tx=0).stop 2024-04-25T01:37:31.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.170+0000 7f8560ffa740 1 -- 172.21.15.28:0/779646097 shutdown_connections 2024-04-25T01:37:31.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.170+0000 7f8560ffa740 1 --2- 172.21.15.28:0/779646097 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x561305a7b1b0 0x561305a7d5e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.170+0000 7f8560ffa740 1 --2- 172.21.15.28:0/779646097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a6b570 0x561305a68720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.170+0000 7f8560ffa740 1 --2- 172.21.15.28:0/779646097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a60ea0 0x561305a681c0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.170+0000 7f85377fe640 0 client.17016 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:37:31.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.170+0000 7f8560ffa740 1 --2- 172.21.15.28:0/779646097 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60580 0x561305a6aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.170+0000 7f8560ffa740 1 -- 172.21.15.28:0/779646097 >> 172.21.15.28:0/779646097 conn(0x561305a5c8a0 msgr2=0x561305a5bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.170+0000 7f8560ffa740 1 -- 172.21.15.28:0/779646097 shutdown_connections 2024-04-25T01:37:31.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.170+0000 7f8560ffa740 1 -- 172.21.15.28:0/779646097 wait complete. 2024-04-25T01:37:31.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.170+0000 7f8560ffa740 20 client.17016 trim_cache size 0 max 16384 2024-04-25T01:37:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.183+0000 7f8560ffa740 1 Processor -- start 2024-04-25T01:37:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.183+0000 7f8560ffa740 1 -- start start 2024-04-25T01:37:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.183+0000 7f8560ffa740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a60580 0x561305a76f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.183+0000 7f8560ffa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60ea0 0x561305a6aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.183+0000 7f8560ffa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 0x561305a82eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.183+0000 7f8560ffa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85581ae8d0 con 0x561305a6b570 2024-04-25T01:37:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.183+0000 7f8560ffa740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f85581b0410 con 0x561305a60580 2024-04-25T01:37:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.183+0000 7f8560ffa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f85581aefa0 con 0x561305a60ea0 2024-04-25T01:37:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.184+0000 7f855c88b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 0x561305a82eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.184+0000 7f855c88b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 0x561305a82eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57672/0 (socket says 172.21.15.28:57672) 2024-04-25T01:37:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.184+0000 7f855c88b640 1 -- 172.21.15.28:0/1119563179 learned_addr learned my addr 172.21.15.28:0/1119563179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.184+0000 7f8557fff640 1 --2- 172.21.15.28:0/1119563179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a60580 0x561305a76f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.184+0000 7f85577fe640 1 --2- 172.21.15.28:0/1119563179 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60ea0 0x561305a6aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.184+0000 7f855c88b640 1 -- 172.21.15.28:0/1119563179 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60ea0 msgr2=0x561305a6aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.184+0000 7f855c88b640 1 --2- 172.21.15.28:0/1119563179 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60ea0 0x561305a6aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.184+0000 7f855c88b640 1 -- 172.21.15.28:0/1119563179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a60580 msgr2=0x561305a76f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.184+0000 7f855c88b640 1 --2- 172.21.15.28:0/1119563179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a60580 0x561305a76f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.184+0000 7f855c88b640 1 -- 172.21.15.28:0/1119563179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613057a37d0 con 0x561305a6b570 2024-04-25T01:37:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.184+0000 7f8557fff640 1 --2- 172.21.15.28:0/1119563179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a60580 0x561305a76f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:37:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.184+0000 7f85577fe640 1 --2- 172.21.15.28:0/1119563179 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60ea0 0x561305a6aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.184+0000 7f855c88b640 1 --2- 172.21.15.28:0/1119563179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 0x561305a82eb0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f8550005c00 tx=0x7f8550004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3de1c33e6bc3aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.184+0000 7f854dffb640 1 -- 172.21.15.28:0/1119563179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8550004b20 con 0x561305a6b570 2024-04-25T01:37:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.184+0000 7f854dffb640 1 -- 172.21.15.28:0/1119563179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8550004cc0 con 0x561305a6b570 2024-04-25T01:37:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.184+0000 7f854dffb640 1 -- 172.21.15.28:0/1119563179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8550005380 con 0x561305a6b570 2024-04-25T01:37:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.185+0000 7f8560ffa740 1 -- 172.21.15.28:0/1119563179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 msgr2=0x561305a82eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.185+0000 7f8560ffa740 1 --2- 172.21.15.28:0/1119563179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 0x561305a82eb0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f8550005c00 tx=0x7f8550004a40 comp rx=0 tx=0).stop 2024-04-25T01:37:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.185+0000 7f8560ffa740 1 -- 172.21.15.28:0/1119563179 shutdown_connections 2024-04-25T01:37:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.185+0000 7f8560ffa740 1 --2- 172.21.15.28:0/1119563179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 0x561305a82eb0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.185+0000 7f8560ffa740 1 --2- 172.21.15.28:0/1119563179 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60ea0 0x561305a6aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.185+0000 7f8560ffa740 1 --2- 172.21.15.28:0/1119563179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a60580 0x561305a76f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.185+0000 7f8560ffa740 1 -- 172.21.15.28:0/1119563179 >> 172.21.15.28:0/1119563179 conn(0x561305a5c8a0 msgr2=0x561305a637a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.185+0000 7f8560ffa740 1 -- 172.21.15.28:0/1119563179 shutdown_connections 2024-04-25T01:37:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.185+0000 7f8560ffa740 1 -- 172.21.15.28:0/1119563179 wait complete. 2024-04-25T01:37:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.185+0000 7f8560ffa740 1 -- start start 2024-04-25T01:37:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.185+0000 7f8560ffa740 1 Processor -- start 2024-04-25T01:37:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f8560ffa740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a60580 0x561305a739f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f8560ffa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60ea0 0x561305a6ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f8560ffa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 0x561305a6f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f8560ffa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85581afd40 con 0x561305a6b570 2024-04-25T01:37:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f8560ffa740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f85581af670 con 0x561305a60580 2024-04-25T01:37:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f8560ffa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f85581ac8b0 con 0x561305a60ea0 2024-04-25T01:37:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f8560ffa740 10 client.0 _ll_register_callbacks cb 0x7f8550007d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f855c88b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 0x561305a6f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f855c88b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 0x561305a6f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57680/0 (socket says 172.21.15.28:57680) 2024-04-25T01:37:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f855c88b640 1 -- 172.21.15.28:0/1691384938 learned_addr learned my addr 172.21.15.28:0/1691384938 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f85577fe640 1 --2- 172.21.15.28:0/1691384938 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60ea0 0x561305a6ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f8557fff640 1 --2- 172.21.15.28:0/1691384938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a60580 0x561305a739f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f855c88b640 1 -- 172.21.15.28:0/1691384938 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60ea0 msgr2=0x561305a6ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f855c88b640 1 --2- 172.21.15.28:0/1691384938 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60ea0 0x561305a6ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f855c88b640 1 -- 172.21.15.28:0/1691384938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a60580 msgr2=0x561305a739f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f855c88b640 1 --2- 172.21.15.28:0/1691384938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a60580 0x561305a739f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f855c88b640 1 -- 172.21.15.28:0/1691384938 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561305ae0720 con 0x561305a6b570 2024-04-25T01:37:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f8557fff640 1 --2- 172.21.15.28:0/1691384938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a60580 0x561305a739f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f855c88b640 1 --2- 172.21.15.28:0/1691384938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 0x561305a6f4a0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f855000bb20 tx=0x7f8550005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692a806025fbefde server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f85377fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f85377fe640 1 -- 172.21.15.28:0/1691384938 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f855002d070 con 0x561305a6b570 2024-04-25T01:37:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f85377fe640 1 -- 172.21.15.28:0/1691384938 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8550003b80 con 0x561305a6b570 2024-04-25T01:37:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.186+0000 7f8560ffa740 1 -- 172.21.15.28:0/1691384938 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8550004cc0 con 0x561305a6b570 2024-04-25T01:37:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.187+0000 7f85377fe640 1 -- 172.21.15.28:0/1691384938 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8550003d80 con 0x561305a6b570 2024-04-25T01:37:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.187+0000 7f8560ffa740 10 client.17022 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.187+0000 7f8560ffa740 1 -- 172.21.15.28:0/1691384938 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561305ae1850 con 0x561305a6b570 2024-04-25T01:37:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.187+0000 7f85377fe640 1 -- 172.21.15.28:0/1691384938 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8550004cc0 con 0x561305a6b570 2024-04-25T01:37:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.187+0000 7f85377fe640 1 -- 172.21.15.28:0/1691384938 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8550004050 con 0x561305a6b570 2024-04-25T01:37:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.187+0000 7f8560ffa740 10 client.17022 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.187+0000 7f8560ffa740 10 client.17022 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:37:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.187+0000 7f8560ffa740 20 client.17022 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.187+0000 7f8560ffa740 1 --2- 172.21.15.28:0/1691384938 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x561305ae1ef0 0x561305ae4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.187+0000 7f8560ffa740 4 client.17022 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:37:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.187+0000 7f8560ffa740 1 -- 172.21.15.28:0/1691384938 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5613059ccf00 con 0x561305ae1ef0 2024-04-25T01:37:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.187+0000 7f8557fff640 1 --2- 172.21.15.28:0/1691384938 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x561305ae1ef0 0x561305ae4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.188+0000 7f85377fe640 10 client.17022 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:37:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.188+0000 7f8557fff640 1 --2- 172.21.15.28:0/1691384938 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x561305ae1ef0 0x561305ae4310 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=32e8a914c5c75697 server_cookie=16f68c9b4d3890a8 in_seq=0 out_seq=0 2024-04-25T01:37:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.188+0000 7f85377fe640 1 -- 172.21.15.28:0/1691384938 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5613059ccf00 con 0x561305ae1ef0 2024-04-25T01:37:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.188+0000 7f85377fe640 10 client.17022 handle_command_reply: tid=0 2024-04-25T01:37:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.188+0000 7f8560ffa740 1 client.17022 shutdown 2024-04-25T01:37:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.188+0000 7f8560ffa740 1 -- 172.21.15.28:0/1691384938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 msgr2=0x561305a6f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.188+0000 7f8560ffa740 1 --2- 172.21.15.28:0/1691384938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 0x561305a6f4a0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f855000bb20 tx=0x7f8550005850 comp rx=0 tx=0).stop 2024-04-25T01:37:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.188+0000 7f8560ffa740 1 -- 172.21.15.28:0/1691384938 shutdown_connections 2024-04-25T01:37:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.188+0000 7f8560ffa740 1 --2- 172.21.15.28:0/1691384938 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x561305ae1ef0 0x561305ae4310 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.188+0000 7f8560ffa740 1 --2- 172.21.15.28:0/1691384938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561305a6b570 0x561305a6f4a0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.188+0000 7f85377fe640 0 client.17022 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:37:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.188+0000 7f8560ffa740 1 --2- 172.21.15.28:0/1691384938 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561305a60ea0 0x561305a6ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.188+0000 7f8560ffa740 1 --2- 172.21.15.28:0/1691384938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561305a60580 0x561305a739f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.188+0000 7f8560ffa740 1 -- 172.21.15.28:0/1691384938 >> 172.21.15.28:0/1691384938 conn(0x561305a5c8a0 msgr2=0x561305a5b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.188+0000 7f8560ffa740 1 -- 172.21.15.28:0/1691384938 shutdown_connections 2024-04-25T01:37:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.188+0000 7f8560ffa740 1 -- 172.21.15.28:0/1691384938 wait complete. 2024-04-25T01:37:31.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.188+0000 7f8560ffa740 20 client.17022 trim_cache size 0 max 16384 2024-04-25T01:37:31.173 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:37:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.189+0000 7f855e2ee640 1 -- 172.21.15.28:0/1164670851 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f85440a2d00 msgr2=0x7f85440a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.189+0000 7f855e2ee640 1 --2- 172.21.15.28:0/1164670851 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f85440a2d00 0x7f85440a51c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8550013d20 tx=0x7f8550007390 comp rx=0 tx=0).stop 2024-04-25T01:37:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.189+0000 7f855e2ee640 1 -- 172.21.15.28:0/1164670851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8558100fe0 msgr2=0x7f85581984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.189+0000 7f855e2ee640 1 --2- 172.21.15.28:0/1164670851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8558100fe0 0x7f85581984f0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f8540009230 tx=0x7f8540004270 comp rx=0 tx=0).stop 2024-04-25T01:37:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.189+0000 7f855e2ee640 1 -- 172.21.15.28:0/1164670851 shutdown_connections 2024-04-25T01:37:31.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.189+0000 7f855e2ee640 1 --2- 172.21.15.28:0/1164670851 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f85440a2d00 0x7f85440a51c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.189+0000 7f855e2ee640 1 --2- 172.21.15.28:0/1164670851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8558106630 0x7f855819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.189+0000 7f855e2ee640 1 --2- 172.21.15.28:0/1164670851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8558100fe0 0x7f85581984f0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.189+0000 7f855e2ee640 1 --2- 172.21.15.28:0/1164670851 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8558100620 0x7f8558197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.189+0000 7f855e2ee640 1 -- 172.21.15.28:0/1164670851 >> 172.21.15.28:0/1164670851 conn(0x7f85580776a0 msgr2=0x7f85580ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.189+0000 7f855e2ee640 1 -- 172.21.15.28:0/1164670851 shutdown_connections 2024-04-25T01:37:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.189+0000 7f855e2ee640 1 -- 172.21.15.28:0/1164670851 wait complete. 2024-04-25T01:37:31.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:37:31.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.303+0000 7f693bbe2640 1 -- 172.21.15.28:0/567247515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f693406a780 msgr2=0x7f693410e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.303+0000 7f693bbe2640 1 --2- 172.21.15.28:0/567247515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f693406a780 0x7f693410e5f0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f692c009cd0 tx=0x7f692c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.303+0000 7f693bbe2640 1 -- 172.21.15.28:0/567247515 shutdown_connections 2024-04-25T01:37:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.303+0000 7f693bbe2640 1 --2- 172.21.15.28:0/567247515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f693406a780 0x7f693410e5f0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.303+0000 7f693bbe2640 1 --2- 172.21.15.28:0/567247515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6934069dc0 0x7f693406a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.303+0000 7f693bbe2640 1 --2- 172.21.15.28:0/567247515 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6934107730 0x7f6934107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.303+0000 7f693bbe2640 1 -- 172.21.15.28:0/567247515 >> 172.21.15.28:0/567247515 conn(0x7f69340776a0 msgr2=0x7f6934077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.303+0000 7f693bbe2640 1 -- 172.21.15.28:0/567247515 shutdown_connections 2024-04-25T01:37:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.303+0000 7f693bbe2640 1 -- 172.21.15.28:0/567247515 wait complete. 2024-04-25T01:37:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.304+0000 7f693bbe2640 1 Processor -- start 2024-04-25T01:37:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.304+0000 7f693bbe2640 1 -- start start 2024-04-25T01:37:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.304+0000 7f693bbe2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6934069dc0 0x7f6934197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.304+0000 7f693bbe2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f693406a780 0x7f6934198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.304+0000 7f693bbe2640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6934107730 0x7f693419c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.304+0000 7f693bbe2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6934111b00 con 0x7f693406a780 2024-04-25T01:37:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.304+0000 7f693bbe2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6934111980 con 0x7f6934069dc0 2024-04-25T01:37:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.304+0000 7f693bbe2640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6934111c80 con 0x7f6934107730 2024-04-25T01:37:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.304+0000 7f693917d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f693406a780 0x7f6934198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.304+0000 7f693917d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f693406a780 0x7f6934198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57696/0 (socket says 172.21.15.28:57696) 2024-04-25T01:37:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.304+0000 7f693917d640 1 -- 172.21.15.28:0/3249223838 learned_addr learned my addr 172.21.15.28:0/3249223838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.304+0000 7f693997e640 1 --2- 172.21.15.28:0/3249223838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6934069dc0 0x7f6934197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.305+0000 7f693a17f640 1 --2- 172.21.15.28:0/3249223838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6934107730 0x7f693419c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.305+0000 7f693917d640 1 -- 172.21.15.28:0/3249223838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6934107730 msgr2=0x7f693419c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.305+0000 7f693917d640 1 --2- 172.21.15.28:0/3249223838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6934107730 0x7f693419c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.305+0000 7f693917d640 1 -- 172.21.15.28:0/3249223838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6934069dc0 msgr2=0x7f6934197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.305+0000 7f693917d640 1 --2- 172.21.15.28:0/3249223838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6934069dc0 0x7f6934197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.305+0000 7f693917d640 1 -- 172.21.15.28:0/3249223838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f693419cf80 con 0x7f693406a780 2024-04-25T01:37:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.305+0000 7f693917d640 1 --2- 172.21.15.28:0/3249223838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f693406a780 0x7f6934198520 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f692400e9f0 tx=0x7f692400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.305+0000 7f692affd640 1 -- 172.21.15.28:0/3249223838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692400cd80 con 0x7f693406a780 2024-04-25T01:37:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.305+0000 7f692affd640 1 -- 172.21.15.28:0/3249223838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6924010910 con 0x7f693406a780 2024-04-25T01:37:31.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.305+0000 7f693bbe2640 1 -- 172.21.15.28:0/3249223838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f693419d270 con 0x7f693406a780 2024-04-25T01:37:31.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.305+0000 7f692affd640 1 -- 172.21.15.28:0/3249223838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6924018990 con 0x7f693406a780 2024-04-25T01:37:31.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.305+0000 7f693bbe2640 1 -- 172.21.15.28:0/3249223838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69341a4b00 con 0x7f693406a780 2024-04-25T01:37:31.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.308+0000 7f692affd640 1 -- 172.21.15.28:0/3249223838 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6924010ac0 con 0x7f693406a780 2024-04-25T01:37:31.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.308+0000 7f693e9c5740 1 Processor -- start 2024-04-25T01:37:31.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.308+0000 7f693e9c5740 1 -- start start 2024-04-25T01:37:31.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.308+0000 7f693e9c5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214a0580 0x55b1214a0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.308+0000 7f693e9c5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 0x55b1214ab030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.308+0000 7f693e9c5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214ab570 0x55b1214ad960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.308+0000 7f693e9c5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b121405670 con 0x55b1214a0ea0 2024-04-25T01:37:31.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.308+0000 7f693e9c5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b1213ad970 con 0x55b1214a0580 2024-04-25T01:37:31.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.308+0000 7f693e9c5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b121401630 con 0x55b1214ab570 2024-04-25T01:37:31.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.309+0000 7f693997e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 0x55b1214ab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.309+0000 7f693997e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 0x55b1214ab030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57700/0 (socket says 172.21.15.28:57700) 2024-04-25T01:37:31.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.309+0000 7f693997e640 1 -- 172.21.15.28:0/609204771 learned_addr learned my addr 172.21.15.28:0/609204771 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.309+0000 7f693a17f640 1 --2- 172.21.15.28:0/609204771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214a0580 0x55b1214a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.309+0000 7f693917d640 1 --2- 172.21.15.28:0/609204771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214ab570 0x55b1214ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.309+0000 7f693997e640 1 -- 172.21.15.28:0/609204771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214ab570 msgr2=0x55b1214ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.309+0000 7f693997e640 1 --2- 172.21.15.28:0/609204771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214ab570 0x55b1214ad960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.309+0000 7f692affd640 1 --2- 172.21.15.28:0/3249223838 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f69200a2d00 0x7f69200a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.309+0000 7f693997e640 1 -- 172.21.15.28:0/609204771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214a0580 msgr2=0x55b1214a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.309+0000 7f693997e640 1 --2- 172.21.15.28:0/609204771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214a0580 0x55b1214a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.309+0000 7f693997e640 1 -- 172.21.15.28:0/609204771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1213d1400 con 0x55b1214a0ea0 2024-04-25T01:37:31.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.309+0000 7f693a17f640 1 --2- 172.21.15.28:0/3249223838 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f69200a2d00 0x7f69200a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.309+0000 7f692affd640 1 -- 172.21.15.28:0/3249223838 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f69240cfae0 con 0x7f693406a780 2024-04-25T01:37:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.309+0000 7f693997e640 1 --2- 172.21.15.28:0/609204771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 0x55b1214ab030 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f693000bc50 tx=0x7f69300107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b125aa417b17302 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.309+0000 7f691ffff640 1 -- 172.21.15.28:0/609204771 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693001c480 con 0x55b1214a0ea0 2024-04-25T01:37:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.309+0000 7f691ffff640 1 -- 172.21.15.28:0/609204771 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f693001cb00 con 0x55b1214a0ea0 2024-04-25T01:37:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.309+0000 7f691ffff640 1 -- 172.21.15.28:0/609204771 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6930011b90 con 0x55b1214a0ea0 2024-04-25T01:37:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.310+0000 7f693e9c5740 1 -- 172.21.15.28:0/609204771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 msgr2=0x55b1214ab030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.310+0000 7f693a17f640 1 --2- 172.21.15.28:0/3249223838 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f69200a2d00 0x7f69200a51c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f692c004ab0 tx=0x7f692c0095c0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.310+0000 7f693e9c5740 1 --2- 172.21.15.28:0/609204771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 0x55b1214ab030 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f693000bc50 tx=0x7f69300107a0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.310+0000 7f693e9c5740 1 -- 172.21.15.28:0/609204771 shutdown_connections 2024-04-25T01:37:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.310+0000 7f693e9c5740 1 --2- 172.21.15.28:0/609204771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214ab570 0x55b1214ad960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.310+0000 7f693e9c5740 1 --2- 172.21.15.28:0/609204771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 0x55b1214ab030 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.310+0000 7f693e9c5740 1 --2- 172.21.15.28:0/609204771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214a0580 0x55b1214a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.310+0000 7f693e9c5740 1 -- 172.21.15.28:0/609204771 >> 172.21.15.28:0/609204771 conn(0x55b12149c8a0 msgr2=0x55b12149cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.310+0000 7f693e9c5740 1 -- 172.21.15.28:0/609204771 shutdown_connections 2024-04-25T01:37:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.310+0000 7f693e9c5740 1 -- 172.21.15.28:0/609204771 wait complete. 2024-04-25T01:37:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693e9c5740 1 -- start start 2024-04-25T01:37:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693e9c5740 1 Processor -- start 2024-04-25T01:37:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693e9c5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214a0580 0x55b1214aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693e9c5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 0x55b1214a8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693e9c5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214ab570 0x55b1214a8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693e9c5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b1214ae360 con 0x55b1214a0ea0 2024-04-25T01:37:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693e9c5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b1213c1050 con 0x55b1214ab570 2024-04-25T01:37:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693e9c5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b1214ae4e0 con 0x55b1214a0580 2024-04-25T01:37:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693e9c5740 10 client.0 _ll_register_callbacks cb 0x55b121118be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693917d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 0x55b1214a8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693917d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 0x55b1214a8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57710/0 (socket says 172.21.15.28:57710) 2024-04-25T01:37:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693917d640 1 -- 172.21.15.28:0/3689705942 learned_addr learned my addr 172.21.15.28:0/3689705942 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693a17f640 1 --2- 172.21.15.28:0/3689705942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214ab570 0x55b1214a8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693997e640 1 --2- 172.21.15.28:0/3689705942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214a0580 0x55b1214aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693917d640 1 -- 172.21.15.28:0/3689705942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214a0580 msgr2=0x55b1214aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693917d640 1 --2- 172.21.15.28:0/3689705942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214a0580 0x55b1214aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693917d640 1 -- 172.21.15.28:0/3689705942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214ab570 msgr2=0x55b1214a8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693917d640 1 --2- 172.21.15.28:0/3689705942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214ab570 0x55b1214a8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693917d640 1 -- 172.21.15.28:0/3689705942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b121392dc0 con 0x55b1214a0ea0 2024-04-25T01:37:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.311+0000 7f693997e640 1 --2- 172.21.15.28:0/3689705942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214a0580 0x55b1214aaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:37:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.312+0000 7f693917d640 1 --2- 172.21.15.28:0/3689705942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 0x55b1214a8220 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f692400c6a0 tx=0x7f692409bae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56401d86bda5fd18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.312+0000 7f691d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.312+0000 7f691d7fa640 1 -- 172.21.15.28:0/3689705942 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69240cd320 con 0x55b1214a0ea0 2024-04-25T01:37:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.312+0000 7f691d7fa640 1 -- 172.21.15.28:0/3689705942 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f6924018b30 con 0x55b1214a0ea0 2024-04-25T01:37:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.312+0000 7f693e9c5740 1 -- 172.21.15.28:0/3689705942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f693001cb00 con 0x55b1214a0ea0 2024-04-25T01:37:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.312+0000 7f691d7fa640 1 -- 172.21.15.28:0/3689705942 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f692409bd10 con 0x55b1214a0ea0 2024-04-25T01:37:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.312+0000 7f693e9c5740 10 client.17034 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.312+0000 7f693e9c5740 1 -- 172.21.15.28:0/3689705942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b1211193f0 con 0x55b1214a0ea0 2024-04-25T01:37:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.312+0000 7f691d7fa640 1 -- 172.21.15.28:0/3689705942 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f693001cb00 con 0x55b1214a0ea0 2024-04-25T01:37:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.312+0000 7f691d7fa640 1 -- 172.21.15.28:0/3689705942 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6924097a60 con 0x55b1214a0ea0 2024-04-25T01:37:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.312+0000 7f693e9c5740 10 client.17034 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.312+0000 7f693e9c5740 10 client.17034 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:37:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.312+0000 7f693e9c5740 20 client.17034 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.312+0000 7f693e9c5740 1 --2- 172.21.15.28:0/3689705942 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55b1214bb5f0 0x55b1214bda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.312+0000 7f693e9c5740 4 client.17034 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:37:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.312+0000 7f693e9c5740 1 -- 172.21.15.28:0/3689705942 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b121412bb0 con 0x55b1214bb5f0 2024-04-25T01:37:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.313+0000 7f693997e640 1 --2- 172.21.15.28:0/3689705942 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55b1214bb5f0 0x55b1214bda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.313+0000 7f693997e640 1 --2- 172.21.15.28:0/3689705942 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55b1214bb5f0 0x55b1214bda60 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3cd9dc4e92b6dd8e server_cookie=eff5db94ac723eb3 in_seq=0 out_seq=0 2024-04-25T01:37:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.313+0000 7f691d7fa640 10 client.17034 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:37:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.314+0000 7f691d7fa640 1 -- 172.21.15.28:0/3689705942 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b121412bb0 con 0x55b1214bb5f0 2024-04-25T01:37:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.314+0000 7f691d7fa640 10 client.17034 handle_command_reply: tid=0 2024-04-25T01:37:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.314+0000 7f693e9c5740 1 client.17034 shutdown 2024-04-25T01:37:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.314+0000 7f693e9c5740 1 -- 172.21.15.28:0/3689705942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 msgr2=0x55b1214a8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.314+0000 7f693e9c5740 1 --2- 172.21.15.28:0/3689705942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 0x55b1214a8220 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f692400c6a0 tx=0x7f692409bae0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.314+0000 7f693e9c5740 1 -- 172.21.15.28:0/3689705942 shutdown_connections 2024-04-25T01:37:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.314+0000 7f693e9c5740 1 --2- 172.21.15.28:0/3689705942 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214ab570 0x55b1214a8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.314+0000 7f693e9c5740 1 --2- 172.21.15.28:0/3689705942 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55b1214bb5f0 0x55b1214bda60 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.314+0000 7f693e9c5740 1 --2- 172.21.15.28:0/3689705942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 0x55b1214a8220 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.314+0000 7f693e9c5740 1 --2- 172.21.15.28:0/3689705942 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214a0580 0x55b1214aaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.314+0000 7f691d7fa640 0 client.17034 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:37:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.314+0000 7f693e9c5740 1 -- 172.21.15.28:0/3689705942 >> 172.21.15.28:0/3689705942 conn(0x55b12149c8a0 msgr2=0x55b12149bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.314+0000 7f693e9c5740 1 -- 172.21.15.28:0/3689705942 shutdown_connections 2024-04-25T01:37:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.314+0000 7f693e9c5740 1 -- 172.21.15.28:0/3689705942 wait complete. 2024-04-25T01:37:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.314+0000 7f693e9c5740 20 client.17034 trim_cache size 0 max 16384 2024-04-25T01:37:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693e9c5740 1 Processor -- start 2024-04-25T01:37:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693e9c5740 1 -- start start 2024-04-25T01:37:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693e9c5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214a0580 0x55b1214a7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693e9c5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 0x55b1214aaa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693e9c5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214ab570 0x55b1214b3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693e9c5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f69341aeb00 con 0x55b1214a0ea0 2024-04-25T01:37:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693e9c5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f69341b0640 con 0x55b1214ab570 2024-04-25T01:37:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693e9c5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f69341af1d0 con 0x55b1214a0580 2024-04-25T01:37:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693917d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 0x55b1214aaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693917d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 0x55b1214aaa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57718/0 (socket says 172.21.15.28:57718) 2024-04-25T01:37:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693917d640 1 -- 172.21.15.28:0/1665929236 learned_addr learned my addr 172.21.15.28:0/1665929236 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693a17f640 1 --2- 172.21.15.28:0/1665929236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214ab570 0x55b1214b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693997e640 1 --2- 172.21.15.28:0/1665929236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214a0580 0x55b1214a7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693917d640 1 -- 172.21.15.28:0/1665929236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214a0580 msgr2=0x55b1214a7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693917d640 1 --2- 172.21.15.28:0/1665929236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214a0580 0x55b1214a7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693917d640 1 -- 172.21.15.28:0/1665929236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214ab570 msgr2=0x55b1214b3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693917d640 1 --2- 172.21.15.28:0/1665929236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214ab570 0x55b1214b3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693917d640 1 -- 172.21.15.28:0/1665929236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1211e37d0 con 0x55b1214a0ea0 2024-04-25T01:37:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693997e640 1 --2- 172.21.15.28:0/1665929236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214a0580 0x55b1214a7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.328+0000 7f693917d640 1 --2- 172.21.15.28:0/1665929236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 0x55b1214aaa60 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f69240cfea0 tx=0x7f692409cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb5661e5db13cbd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.329+0000 7f691f7fe640 1 -- 172.21.15.28:0/1665929236 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69240b4e60 con 0x55b1214a0ea0 2024-04-25T01:37:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.329+0000 7f691f7fe640 1 -- 172.21.15.28:0/1665929236 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69240b1020 con 0x55b1214a0ea0 2024-04-25T01:37:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.329+0000 7f691f7fe640 1 -- 172.21.15.28:0/1665929236 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692409b3c0 con 0x55b1214a0ea0 2024-04-25T01:37:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.329+0000 7f693e9c5740 1 -- 172.21.15.28:0/1665929236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 msgr2=0x55b1214aaa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.329+0000 7f693e9c5740 1 --2- 172.21.15.28:0/1665929236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 0x55b1214aaa60 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f69240cfea0 tx=0x7f692409cf90 comp rx=0 tx=0).stop 2024-04-25T01:37:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.329+0000 7f693e9c5740 1 -- 172.21.15.28:0/1665929236 shutdown_connections 2024-04-25T01:37:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.329+0000 7f693e9c5740 1 --2- 172.21.15.28:0/1665929236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214ab570 0x55b1214b3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.329+0000 7f693e9c5740 1 --2- 172.21.15.28:0/1665929236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0ea0 0x55b1214aaa60 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.329+0000 7f693e9c5740 1 --2- 172.21.15.28:0/1665929236 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214a0580 0x55b1214a7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.329+0000 7f693e9c5740 1 -- 172.21.15.28:0/1665929236 >> 172.21.15.28:0/1665929236 conn(0x55b12149c8a0 msgr2=0x55b1214b6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.329+0000 7f693e9c5740 1 -- 172.21.15.28:0/1665929236 shutdown_connections 2024-04-25T01:37:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.329+0000 7f693e9c5740 1 -- 172.21.15.28:0/1665929236 wait complete. 2024-04-25T01:37:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.329+0000 7f693e9c5740 1 -- start start 2024-04-25T01:37:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693e9c5740 1 Processor -- start 2024-04-25T01:37:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693e9c5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0580 0x55b1214b1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693e9c5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214a0ea0 0x55b1214b1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693e9c5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214ab570 0x55b121520bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693e9c5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f69341aff70 con 0x55b1214a0580 2024-04-25T01:37:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693e9c5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f69341af8a0 con 0x55b1214ab570 2024-04-25T01:37:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693e9c5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f69341acae0 con 0x55b1214a0ea0 2024-04-25T01:37:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693e9c5740 10 client.0 _ll_register_callbacks cb 0x7f692400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693997e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0580 0x55b1214b1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693997e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0580 0x55b1214b1530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57728/0 (socket says 172.21.15.28:57728) 2024-04-25T01:37:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693997e640 1 -- 172.21.15.28:0/1215158618 learned_addr learned my addr 172.21.15.28:0/1215158618 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693917d640 1 --2- 172.21.15.28:0/1215158618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214a0ea0 0x55b1214b1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693a17f640 1 --2- 172.21.15.28:0/1215158618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214ab570 0x55b121520bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693997e640 1 -- 172.21.15.28:0/1215158618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214a0ea0 msgr2=0x55b1214b1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693997e640 1 --2- 172.21.15.28:0/1215158618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214a0ea0 0x55b1214b1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693997e640 1 -- 172.21.15.28:0/1215158618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214ab570 msgr2=0x55b121520bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693997e640 1 --2- 172.21.15.28:0/1215158618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214ab570 0x55b121520bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693997e640 1 -- 172.21.15.28:0/1215158618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1214b2370 con 0x55b1214a0580 2024-04-25T01:37:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693a17f640 1 --2- 172.21.15.28:0/1215158618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214ab570 0x55b121520bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f693997e640 1 --2- 172.21.15.28:0/1215158618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0580 0x55b1214b1530 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f693000bb70 tx=0x7f6930005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c70328e3a4d998e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.330+0000 7f691d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.331+0000 7f691d7fa640 1 -- 172.21.15.28:0/1215158618 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693002e070 con 0x55b1214a0580 2024-04-25T01:37:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.331+0000 7f691d7fa640 1 -- 172.21.15.28:0/1215158618 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f6930005450 con 0x55b1214a0580 2024-04-25T01:37:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.331+0000 7f693e9c5740 1 -- 172.21.15.28:0/1215158618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69240b1020 con 0x55b1214a0580 2024-04-25T01:37:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.331+0000 7f691d7fa640 1 -- 172.21.15.28:0/1215158618 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f693003a050 con 0x55b1214a0580 2024-04-25T01:37:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.331+0000 7f693e9c5740 10 client.17040 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.331+0000 7f693e9c5740 1 -- 172.21.15.28:0/1215158618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b1215221c0 con 0x55b1214a0580 2024-04-25T01:37:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.331+0000 7f691d7fa640 1 -- 172.21.15.28:0/1215158618 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69240b1020 con 0x55b1214a0580 2024-04-25T01:37:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.331+0000 7f691d7fa640 1 -- 172.21.15.28:0/1215158618 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6930011760 con 0x55b1214a0580 2024-04-25T01:37:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.331+0000 7f693e9c5740 10 client.17040 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.331+0000 7f693e9c5740 10 client.17040 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:37:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.331+0000 7f693e9c5740 20 client.17040 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.331+0000 7f693e9c5740 1 --2- 172.21.15.28:0/1215158618 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55b121522610 0x55b121524a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.331+0000 7f693e9c5740 4 client.17040 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:37:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.331+0000 7f693e9c5740 1 -- 172.21.15.28:0/1215158618 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b12140cf00 con 0x55b121522610 2024-04-25T01:37:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.332+0000 7f693917d640 1 --2- 172.21.15.28:0/1215158618 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55b121522610 0x55b121524a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.332+0000 7f693917d640 1 --2- 172.21.15.28:0/1215158618 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55b121522610 0x55b121524a00 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=21d5d34332eb005f server_cookie=c75957c1d7bebf67 in_seq=0 out_seq=0 2024-04-25T01:37:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.332+0000 7f691d7fa640 10 client.17040 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:37:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.332+0000 7f691d7fa640 1 -- 172.21.15.28:0/1215158618 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b12140cf00 con 0x55b121522610 2024-04-25T01:37:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.332+0000 7f691d7fa640 10 client.17040 handle_command_reply: tid=0 2024-04-25T01:37:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.332+0000 7f693e9c5740 1 client.17040 shutdown 2024-04-25T01:37:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.332+0000 7f693e9c5740 1 -- 172.21.15.28:0/1215158618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0580 msgr2=0x55b1214b1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.332+0000 7f693e9c5740 1 --2- 172.21.15.28:0/1215158618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0580 0x55b1214b1530 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f693000bb70 tx=0x7f6930005850 comp rx=0 tx=0).stop 2024-04-25T01:37:31.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.332+0000 7f693e9c5740 1 -- 172.21.15.28:0/1215158618 shutdown_connections 2024-04-25T01:37:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.332+0000 7f693e9c5740 1 --2- 172.21.15.28:0/1215158618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b1214ab570 0x55b121520bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.332+0000 7f693e9c5740 1 --2- 172.21.15.28:0/1215158618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b1214a0ea0 0x55b1214b1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.332+0000 7f693e9c5740 1 --2- 172.21.15.28:0/1215158618 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55b121522610 0x55b121524a00 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.332+0000 7f693e9c5740 1 --2- 172.21.15.28:0/1215158618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b1214a0580 0x55b1214b1530 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.332+0000 7f691d7fa640 0 client.17040 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:37:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.332+0000 7f693e9c5740 1 -- 172.21.15.28:0/1215158618 >> 172.21.15.28:0/1215158618 conn(0x55b12149c8a0 msgr2=0x55b1214b6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.332+0000 7f693e9c5740 1 -- 172.21.15.28:0/1215158618 shutdown_connections 2024-04-25T01:37:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.332+0000 7f693e9c5740 1 -- 172.21.15.28:0/1215158618 wait complete. 2024-04-25T01:37:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.333+0000 7f693e9c5740 20 client.17040 trim_cache size 0 max 16384 2024-04-25T01:37:31.317 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:37:31.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.333+0000 7f693bbe2640 1 -- 172.21.15.28:0/3249223838 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f69200a2d00 msgr2=0x7f69200a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.333+0000 7f693bbe2640 1 --2- 172.21.15.28:0/3249223838 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f69200a2d00 0x7f69200a51c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f692c004ab0 tx=0x7f692c0095c0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.333+0000 7f693bbe2640 1 -- 172.21.15.28:0/3249223838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f693406a780 msgr2=0x7f6934198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.333+0000 7f693bbe2640 1 --2- 172.21.15.28:0/3249223838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f693406a780 0x7f6934198520 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f692400e9f0 tx=0x7f692400eeb0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.333+0000 7f693bbe2640 1 -- 172.21.15.28:0/3249223838 shutdown_connections 2024-04-25T01:37:31.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.333+0000 7f693bbe2640 1 --2- 172.21.15.28:0/3249223838 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f69200a2d00 0x7f69200a51c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.333+0000 7f693bbe2640 1 --2- 172.21.15.28:0/3249223838 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6934107730 0x7f693419c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.333+0000 7f693bbe2640 1 --2- 172.21.15.28:0/3249223838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f693406a780 0x7f6934198520 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.333+0000 7f693bbe2640 1 --2- 172.21.15.28:0/3249223838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6934069dc0 0x7f6934197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.333+0000 7f693bbe2640 1 -- 172.21.15.28:0/3249223838 >> 172.21.15.28:0/3249223838 conn(0x7f69340776a0 msgr2=0x7f69340fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.333+0000 7f693bbe2640 1 -- 172.21.15.28:0/3249223838 shutdown_connections 2024-04-25T01:37:31.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.333+0000 7f693bbe2640 1 -- 172.21.15.28:0/3249223838 wait complete. 2024-04-25T01:37:31.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:37:31.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.448+0000 7f76fd838640 1 -- 172.21.15.28:0/1012123850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f8100620 msgr2=0x7f76f8100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.448+0000 7f76fd838640 1 --2- 172.21.15.28:0/1012123850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f8100620 0x7f76f8100aa0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009930 tx=0x7f76ec02a420 comp rx=0 tx=0).stop 2024-04-25T01:37:31.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.448+0000 7f76fd838640 1 -- 172.21.15.28:0/1012123850 shutdown_connections 2024-04-25T01:37:31.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.448+0000 7f76fd838640 1 --2- 172.21.15.28:0/1012123850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76f8100fe0 0x7f76f810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.448+0000 7f76fd838640 1 --2- 172.21.15.28:0/1012123850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f8100620 0x7f76f8100aa0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.448+0000 7f76fd838640 1 --2- 172.21.15.28:0/1012123850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f76f8106630 0x7f76f8106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.448+0000 7f76fd838640 1 -- 172.21.15.28:0/1012123850 >> 172.21.15.28:0/1012123850 conn(0x7f76f80776a0 msgr2=0x7f76f8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.448+0000 7f76fd838640 1 -- 172.21.15.28:0/1012123850 shutdown_connections 2024-04-25T01:37:31.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.448+0000 7f76fd838640 1 -- 172.21.15.28:0/1012123850 wait complete. 2024-04-25T01:37:31.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.449+0000 7f76fd838640 1 Processor -- start 2024-04-25T01:37:31.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.449+0000 7f76fd838640 1 -- start start 2024-04-25T01:37:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.449+0000 7f76fd838640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f8100620 0x7f76f8198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.449+0000 7f76fd838640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f76f8100fe0 0x7f76f8199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.449+0000 7f76fd838640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76f8106630 0x7f76f81a3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.449+0000 7f76fd838640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76f8110a40 con 0x7f76f8100620 2024-04-25T01:37:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.449+0000 7f76fd838640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f76f81108c0 con 0x7f76f8106630 2024-04-25T01:37:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.449+0000 7f76fd838640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f76f8110bc0 con 0x7f76f8100fe0 2024-04-25T01:37:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.449+0000 7f76f6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f8100620 0x7f76f8198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.449+0000 7f76f6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f8100620 0x7f76f8198ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57734/0 (socket says 172.21.15.28:57734) 2024-04-25T01:37:31.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.449+0000 7f76f6ffd640 1 -- 172.21.15.28:0/3106222676 learned_addr learned my addr 172.21.15.28:0/3106222676 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.449+0000 7f76f77fe640 1 --2- 172.21.15.28:0/3106222676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76f8106630 0x7f76f81a3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.450+0000 7f76f67fc640 1 --2- 172.21.15.28:0/3106222676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f76f8100fe0 0x7f76f8199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.450+0000 7f76f6ffd640 1 -- 172.21.15.28:0/3106222676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f76f8100fe0 msgr2=0x7f76f8199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.450+0000 7f76f6ffd640 1 --2- 172.21.15.28:0/3106222676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f76f8100fe0 0x7f76f8199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.450+0000 7f76f6ffd640 1 -- 172.21.15.28:0/3106222676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76f8106630 msgr2=0x7f76f81a3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.450+0000 7f76f6ffd640 1 --2- 172.21.15.28:0/3106222676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76f8106630 0x7f76f81a3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.450+0000 7f76f6ffd640 1 -- 172.21.15.28:0/3106222676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f81a4480 con 0x7f76f8100620 2024-04-25T01:37:31.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.450+0000 7f76f67fc640 1 --2- 172.21.15.28:0/3106222676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f76f8100fe0 0x7f76f8199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:31.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.450+0000 7f76f6ffd640 1 --2- 172.21.15.28:0/3106222676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f8100620 0x7f76f8198ec0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f76e800d770 tx=0x7f76e800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.450+0000 7f76fc836640 1 -- 172.21.15.28:0/3106222676 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e8017070 con 0x7f76f8100620 2024-04-25T01:37:31.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.450+0000 7f76fd838640 1 -- 172.21.15.28:0/3106222676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76f81a4650 con 0x7f76f8100620 2024-04-25T01:37:31.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.450+0000 7f76fc836640 1 -- 172.21.15.28:0/3106222676 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76e8005500 con 0x7f76f8100620 2024-04-25T01:37:31.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.450+0000 7f76fc836640 1 -- 172.21.15.28:0/3106222676 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e8004e00 con 0x7f76f8100620 2024-04-25T01:37:31.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.450+0000 7f76fd838640 1 -- 172.21.15.28:0/3106222676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76f81a4b20 con 0x7f76f8100620 2024-04-25T01:37:31.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.452+0000 7f76fc836640 1 -- 172.21.15.28:0/3106222676 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f76e8004fa0 con 0x7f76f8100620 2024-04-25T01:37:31.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f7700629740 1 Processor -- start 2024-04-25T01:37:31.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f7700629740 1 -- start start 2024-04-25T01:37:31.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f7700629740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f554580 0x55b21f554960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f7700629740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f554ea0 0x55b21f55f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f7700629740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f55f570 0x55b21f561960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f7700629740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b21f4b9670 con 0x55b21f55f570 2024-04-25T01:37:31.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f7700629740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b21f461970 con 0x55b21f554ea0 2024-04-25T01:37:31.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f7700629740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b21f4b5630 con 0x55b21f554580 2024-04-25T01:37:31.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f76fc836640 1 --2- 172.21.15.28:0/3106222676 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f76e00a2d00 0x7f76e00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f76f67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f55f570 0x55b21f561960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f76f67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f55f570 0x55b21f561960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57738/0 (socket says 172.21.15.28:57738) 2024-04-25T01:37:31.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f76f67fc640 1 -- 172.21.15.28:0/2920239279 learned_addr learned my addr 172.21.15.28:0/2920239279 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f76f77fe640 1 --2- 172.21.15.28:0/3106222676 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f76e00a2d00 0x7f76e00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f76f77fe640 1 --2- 172.21.15.28:0/2920239279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f554580 0x55b21f554960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f76fc836640 1 -- 172.21.15.28:0/3106222676 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f76e80cf160 con 0x7f76f8100620 2024-04-25T01:37:31.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f76f6ffd640 1 --2- 172.21.15.28:0/2920239279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f554ea0 0x55b21f55f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f76f67fc640 1 -- 172.21.15.28:0/2920239279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f554580 msgr2=0x55b21f554960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f76f67fc640 1 --2- 172.21.15.28:0/2920239279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f554580 0x55b21f554960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f76f67fc640 1 -- 172.21.15.28:0/2920239279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f554ea0 msgr2=0x55b21f55f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f76f67fc640 1 --2- 172.21.15.28:0/2920239279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f554ea0 0x55b21f55f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.453+0000 7f76f67fc640 1 -- 172.21.15.28:0/2920239279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b21f485400 con 0x55b21f55f570 2024-04-25T01:37:31.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.454+0000 7f76f67fc640 1 --2- 172.21.15.28:0/2920239279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f55f570 0x55b21f561960 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f76ec006010 tx=0x7f76ec009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef6b225507a6968 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.454+0000 7f76f77fe640 1 --2- 172.21.15.28:0/3106222676 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f76e00a2d00 0x7f76e00a51c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f76f0005fd0 tx=0x7f76f0004a40 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.454+0000 7f76ddffb640 1 -- 172.21.15.28:0/2920239279 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ec03f070 con 0x55b21f55f570 2024-04-25T01:37:31.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.454+0000 7f76ddffb640 1 -- 172.21.15.28:0/2920239279 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76ec0317b0 con 0x55b21f55f570 2024-04-25T01:37:31.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.454+0000 7f76ddffb640 1 -- 172.21.15.28:0/2920239279 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ec035070 con 0x55b21f55f570 2024-04-25T01:37:31.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.454+0000 7f7700629740 1 -- 172.21.15.28:0/2920239279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f55f570 msgr2=0x55b21f561960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.454+0000 7f7700629740 1 --2- 172.21.15.28:0/2920239279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f55f570 0x55b21f561960 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f76ec006010 tx=0x7f76ec009680 comp rx=0 tx=0).stop 2024-04-25T01:37:31.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.454+0000 7f7700629740 1 -- 172.21.15.28:0/2920239279 shutdown_connections 2024-04-25T01:37:31.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.454+0000 7f7700629740 1 --2- 172.21.15.28:0/2920239279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f55f570 0x55b21f561960 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.454+0000 7f7700629740 1 --2- 172.21.15.28:0/2920239279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f554ea0 0x55b21f55f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.454+0000 7f7700629740 1 --2- 172.21.15.28:0/2920239279 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f554580 0x55b21f554960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.454+0000 7f7700629740 1 -- 172.21.15.28:0/2920239279 >> 172.21.15.28:0/2920239279 conn(0x55b21f5508a0 msgr2=0x55b21f550c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.454+0000 7f7700629740 1 -- 172.21.15.28:0/2920239279 shutdown_connections 2024-04-25T01:37:31.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.455+0000 7f7700629740 1 -- 172.21.15.28:0/2920239279 wait complete. 2024-04-25T01:37:31.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.455+0000 7f7700629740 1 -- start start 2024-04-25T01:37:31.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.455+0000 7f7700629740 1 Processor -- start 2024-04-25T01:37:31.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.455+0000 7f7700629740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f554580 0x55b21f55ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.455+0000 7f7700629740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f554ea0 0x55b21f55c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.455+0000 7f7700629740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f55f570 0x55b21f55c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.455+0000 7f7700629740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b21f562300 con 0x55b21f55f570 2024-04-25T01:37:31.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.455+0000 7f7700629740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b21f475050 con 0x55b21f554580 2024-04-25T01:37:31.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.455+0000 7f7700629740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b21f562480 con 0x55b21f554ea0 2024-04-25T01:37:31.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.455+0000 7f7700629740 10 client.0 _ll_register_callbacks cb 0x55b21f1ccbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:31.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.455+0000 7f76f77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f55f570 0x55b21f55c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.455+0000 7f76f77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f55f570 0x55b21f55c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57742/0 (socket says 172.21.15.28:57742) 2024-04-25T01:37:31.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.455+0000 7f76f77fe640 1 -- 172.21.15.28:0/2160689126 learned_addr learned my addr 172.21.15.28:0/2160689126 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.456+0000 7f76f6ffd640 1 --2- 172.21.15.28:0/2160689126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f554580 0x55b21f55ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.456+0000 7f76f67fc640 1 --2- 172.21.15.28:0/2160689126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f554ea0 0x55b21f55c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.456+0000 7f76f77fe640 1 -- 172.21.15.28:0/2160689126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f554ea0 msgr2=0x55b21f55c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.456+0000 7f76f77fe640 1 --2- 172.21.15.28:0/2160689126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f554ea0 0x55b21f55c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.456+0000 7f76f77fe640 1 -- 172.21.15.28:0/2160689126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f554580 msgr2=0x55b21f55ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.456+0000 7f76f77fe640 1 --2- 172.21.15.28:0/2160689126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f554580 0x55b21f55ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.456+0000 7f76f77fe640 1 -- 172.21.15.28:0/2160689126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b21f446dc0 con 0x55b21f55f570 2024-04-25T01:37:31.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.456+0000 7f76f67fc640 1 --2- 172.21.15.28:0/2160689126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f554ea0 0x55b21f55c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.456+0000 7f76f77fe640 1 --2- 172.21.15.28:0/2160689126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f55f570 0x55b21f55c760 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f76f0003e70 tx=0x7f76f0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89b11db36fa4d18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.456+0000 7f76d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.456+0000 7f76d6ffd640 1 -- 172.21.15.28:0/2160689126 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f0025070 con 0x55b21f55f570 2024-04-25T01:37:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.456+0000 7f76d6ffd640 1 -- 172.21.15.28:0/2160689126 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f76f00119d0 con 0x55b21f55f570 2024-04-25T01:37:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.456+0000 7f7700629740 1 -- 172.21.15.28:0/2160689126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76ec0317b0 con 0x55b21f55f570 2024-04-25T01:37:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.456+0000 7f76d6ffd640 1 -- 172.21.15.28:0/2160689126 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f76f0031050 con 0x55b21f55f570 2024-04-25T01:37:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.456+0000 7f7700629740 10 client.17052 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.456+0000 7f7700629740 1 -- 172.21.15.28:0/2160689126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b21f1cd3f0 con 0x55b21f55f570 2024-04-25T01:37:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.456+0000 7f76d6ffd640 1 -- 172.21.15.28:0/2160689126 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76ec0317b0 con 0x55b21f55f570 2024-04-25T01:37:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.456+0000 7f76d6ffd640 1 -- 172.21.15.28:0/2160689126 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f76f00293b0 con 0x55b21f55f570 2024-04-25T01:37:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.457+0000 7f7700629740 10 client.17052 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.457+0000 7f7700629740 10 client.17052 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:37:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.457+0000 7f7700629740 20 client.17052 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.457+0000 7f7700629740 1 --2- 172.21.15.28:0/2160689126 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b21f56f5d0 0x55b21f571a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.457+0000 7f7700629740 4 client.17052 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:37:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.457+0000 7f7700629740 1 -- 172.21.15.28:0/2160689126 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b21f4c6bb0 con 0x55b21f56f5d0 2024-04-25T01:37:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.457+0000 7f76f6ffd640 1 --2- 172.21.15.28:0/2160689126 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b21f56f5d0 0x55b21f571a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.457+0000 7f76f6ffd640 1 --2- 172.21.15.28:0/2160689126 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b21f56f5d0 0x55b21f571a50 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3aff5857b8055141 server_cookie=7f9e251fd0b7e666 in_seq=0 out_seq=0 2024-04-25T01:37:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.457+0000 7f76d6ffd640 10 client.17052 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:37:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.458+0000 7f76d6ffd640 1 -- 172.21.15.28:0/2160689126 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b21f4c6bb0 con 0x55b21f56f5d0 2024-04-25T01:37:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.458+0000 7f76d6ffd640 10 client.17052 handle_command_reply: tid=0 2024-04-25T01:37:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.458+0000 7f7700629740 1 client.17052 shutdown 2024-04-25T01:37:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.458+0000 7f7700629740 1 -- 172.21.15.28:0/2160689126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f55f570 msgr2=0x55b21f55c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.458+0000 7f7700629740 1 --2- 172.21.15.28:0/2160689126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f55f570 0x55b21f55c760 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f76f0003e70 tx=0x7f76f0004ab0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.458+0000 7f7700629740 1 -- 172.21.15.28:0/2160689126 shutdown_connections 2024-04-25T01:37:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.458+0000 7f7700629740 1 --2- 172.21.15.28:0/2160689126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f55f570 0x55b21f55c760 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.458+0000 7f7700629740 1 --2- 172.21.15.28:0/2160689126 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b21f56f5d0 0x55b21f571a50 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.458+0000 7f7700629740 1 --2- 172.21.15.28:0/2160689126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f554ea0 0x55b21f55c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.458+0000 7f7700629740 1 --2- 172.21.15.28:0/2160689126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f554580 0x55b21f55ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.458+0000 7f76d6ffd640 0 client.17052 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:37:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.458+0000 7f7700629740 1 -- 172.21.15.28:0/2160689126 >> 172.21.15.28:0/2160689126 conn(0x55b21f5508a0 msgr2=0x55b21f54fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.458+0000 7f7700629740 1 -- 172.21.15.28:0/2160689126 shutdown_connections 2024-04-25T01:37:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.458+0000 7f7700629740 1 -- 172.21.15.28:0/2160689126 wait complete. 2024-04-25T01:37:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.458+0000 7f7700629740 20 client.17052 trim_cache size 0 max 16384 2024-04-25T01:37:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.471+0000 7f7700629740 1 Processor -- start 2024-04-25T01:37:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.471+0000 7f7700629740 1 -- start start 2024-04-25T01:37:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.472+0000 7f7700629740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f554580 0x55b21f55e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.472+0000 7f7700629740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f554ea0 0x55b21f577320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.472+0000 7f7700629740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f55f570 0x55b21f577860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.472+0000 7f7700629740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76f81ae9d0 con 0x55b21f554580 2024-04-25T01:37:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.472+0000 7f7700629740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f76f81b0510 con 0x55b21f55f570 2024-04-25T01:37:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.472+0000 7f7700629740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f76f81af0a0 con 0x55b21f554ea0 2024-04-25T01:37:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.472+0000 7f76f6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f554580 0x55b21f55e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.472+0000 7f76f6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f554580 0x55b21f55e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57746/0 (socket says 172.21.15.28:57746) 2024-04-25T01:37:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.472+0000 7f76f6ffd640 1 -- 172.21.15.28:0/1232711147 learned_addr learned my addr 172.21.15.28:0/1232711147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.472+0000 7f76f77fe640 1 --2- 172.21.15.28:0/1232711147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f55f570 0x55b21f577860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.472+0000 7f76f6ffd640 1 -- 172.21.15.28:0/1232711147 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f554ea0 msgr2=0x55b21f577320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.472+0000 7f76f6ffd640 1 --2- 172.21.15.28:0/1232711147 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f554ea0 0x55b21f577320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.472+0000 7f76f6ffd640 1 -- 172.21.15.28:0/1232711147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f55f570 msgr2=0x55b21f577860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.472+0000 7f76f6ffd640 1 --2- 172.21.15.28:0/1232711147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f55f570 0x55b21f577860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.472+0000 7f76f6ffd640 1 -- 172.21.15.28:0/1232711147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b21f2977d0 con 0x55b21f554580 2024-04-25T01:37:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.472+0000 7f76f6ffd640 1 --2- 172.21.15.28:0/1232711147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f554580 0x55b21f55e120 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f76e8009610 tx=0x7f76e8097620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b9d3a3eb07937c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.472+0000 7f76dd7fa640 1 -- 172.21.15.28:0/1232711147 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e80b4070 con 0x55b21f554580 2024-04-25T01:37:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.472+0000 7f76dd7fa640 1 -- 172.21.15.28:0/1232711147 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76e80a3fb0 con 0x55b21f554580 2024-04-25T01:37:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.472+0000 7f76dd7fa640 1 -- 172.21.15.28:0/1232711147 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e8084070 con 0x55b21f554580 2024-04-25T01:37:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.473+0000 7f7700629740 1 -- 172.21.15.28:0/1232711147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f554580 msgr2=0x55b21f55e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.473+0000 7f7700629740 1 --2- 172.21.15.28:0/1232711147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f554580 0x55b21f55e120 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f76e8009610 tx=0x7f76e8097620 comp rx=0 tx=0).stop 2024-04-25T01:37:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.473+0000 7f7700629740 1 -- 172.21.15.28:0/1232711147 shutdown_connections 2024-04-25T01:37:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.473+0000 7f7700629740 1 --2- 172.21.15.28:0/1232711147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f55f570 0x55b21f577860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.473+0000 7f7700629740 1 --2- 172.21.15.28:0/1232711147 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f554ea0 0x55b21f577320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.473+0000 7f7700629740 1 --2- 172.21.15.28:0/1232711147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f554580 0x55b21f55e120 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.473+0000 7f7700629740 1 -- 172.21.15.28:0/1232711147 >> 172.21.15.28:0/1232711147 conn(0x55b21f5508a0 msgr2=0x55b21f55da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.473+0000 7f7700629740 1 -- 172.21.15.28:0/1232711147 shutdown_connections 2024-04-25T01:37:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.473+0000 7f7700629740 1 -- 172.21.15.28:0/1232711147 wait complete. 2024-04-25T01:37:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.473+0000 7f7700629740 1 -- start start 2024-04-25T01:37:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f7700629740 1 Processor -- start 2024-04-25T01:37:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f7700629740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f554580 0x55b21f5652d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f7700629740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f554ea0 0x55b21f565810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f7700629740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f55f570 0x55b21f565d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f7700629740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76f81afe40 con 0x55b21f554580 2024-04-25T01:37:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f7700629740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f76f81af770 con 0x55b21f554ea0 2024-04-25T01:37:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f7700629740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f76f81ac9b0 con 0x55b21f55f570 2024-04-25T01:37:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f7700629740 10 client.0 _ll_register_callbacks cb 0x7f76f0003970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f76f6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f554580 0x55b21f5652d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f76f6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f554580 0x55b21f5652d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57752/0 (socket says 172.21.15.28:57752) 2024-04-25T01:37:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f76f6ffd640 1 -- 172.21.15.28:0/1244412617 learned_addr learned my addr 172.21.15.28:0/1244412617 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f76f77fe640 1 --2- 172.21.15.28:0/1244412617 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f55f570 0x55b21f565d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f76f67fc640 1 --2- 172.21.15.28:0/1244412617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f554ea0 0x55b21f565810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f76f6ffd640 1 -- 172.21.15.28:0/1244412617 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f55f570 msgr2=0x55b21f565d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f76f6ffd640 1 --2- 172.21.15.28:0/1244412617 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f55f570 0x55b21f565d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f76f6ffd640 1 -- 172.21.15.28:0/1244412617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f554ea0 msgr2=0x55b21f565810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f76f6ffd640 1 --2- 172.21.15.28:0/1244412617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f554ea0 0x55b21f565810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f76f6ffd640 1 -- 172.21.15.28:0/1244412617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b21f5d4da0 con 0x55b21f554580 2024-04-25T01:37:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f76f67fc640 1 --2- 172.21.15.28:0/1244412617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f554ea0 0x55b21f565810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:37:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f76f6ffd640 1 --2- 172.21.15.28:0/1244412617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f554580 0x55b21f5652d0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f76e80cc620 tx=0x7f76e809f780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b3e1e43097362bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.474+0000 7f76d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.475+0000 7f76d6ffd640 1 -- 172.21.15.28:0/1244412617 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e8084070 con 0x55b21f554580 2024-04-25T01:37:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.475+0000 7f76d6ffd640 1 -- 172.21.15.28:0/1244412617 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f76e8009cc0 con 0x55b21f554580 2024-04-25T01:37:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.475+0000 7f7700629740 1 -- 172.21.15.28:0/1244412617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76e80a3fb0 con 0x55b21f554580 2024-04-25T01:37:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.475+0000 7f76d6ffd640 1 -- 172.21.15.28:0/1244412617 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f76e80b4040 con 0x55b21f554580 2024-04-25T01:37:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.475+0000 7f7700629740 10 client.17058 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.475+0000 7f7700629740 1 -- 172.21.15.28:0/1244412617 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b21f5d6060 con 0x55b21f554580 2024-04-25T01:37:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.475+0000 7f76d6ffd640 1 -- 172.21.15.28:0/1244412617 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76e80a3fb0 con 0x55b21f554580 2024-04-25T01:37:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.475+0000 7f76d6ffd640 1 -- 172.21.15.28:0/1244412617 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f76e809a790 con 0x55b21f554580 2024-04-25T01:37:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.475+0000 7f7700629740 10 client.17058 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.475+0000 7f7700629740 10 client.17058 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:37:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.475+0000 7f7700629740 20 client.17058 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.475+0000 7f7700629740 1 --2- 172.21.15.28:0/1244412617 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b21f5d67f0 0x55b21f5d8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.475+0000 7f7700629740 4 client.17058 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:37:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.475+0000 7f7700629740 1 -- 172.21.15.28:0/1244412617 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b21f4c0f00 con 0x55b21f5d67f0 2024-04-25T01:37:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.475+0000 7f76f67fc640 1 --2- 172.21.15.28:0/1244412617 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b21f5d67f0 0x55b21f5d8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.476+0000 7f76d6ffd640 10 client.17058 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:37:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.476+0000 7f76f67fc640 1 --2- 172.21.15.28:0/1244412617 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b21f5d67f0 0x55b21f5d8c10 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7d40240d86ff1d1f server_cookie=e4729a38ae0f3a4c in_seq=0 out_seq=0 2024-04-25T01:37:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.476+0000 7f76d6ffd640 1 -- 172.21.15.28:0/1244412617 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b21f4c0f00 con 0x55b21f5d67f0 2024-04-25T01:37:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.476+0000 7f76d6ffd640 10 client.17058 handle_command_reply: tid=0 2024-04-25T01:37:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.476+0000 7f7700629740 1 client.17058 shutdown 2024-04-25T01:37:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.476+0000 7f7700629740 1 -- 172.21.15.28:0/1244412617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f554580 msgr2=0x55b21f5652d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.476+0000 7f7700629740 1 --2- 172.21.15.28:0/1244412617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f554580 0x55b21f5652d0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f76e80cc620 tx=0x7f76e809f780 comp rx=0 tx=0).stop 2024-04-25T01:37:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.476+0000 7f7700629740 1 -- 172.21.15.28:0/1244412617 shutdown_connections 2024-04-25T01:37:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.476+0000 7f7700629740 1 --2- 172.21.15.28:0/1244412617 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b21f5d67f0 0x55b21f5d8c10 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.476+0000 7f7700629740 1 --2- 172.21.15.28:0/1244412617 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b21f55f570 0x55b21f565d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.476+0000 7f7700629740 1 --2- 172.21.15.28:0/1244412617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b21f554ea0 0x55b21f565810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.476+0000 7f7700629740 1 --2- 172.21.15.28:0/1244412617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b21f554580 0x55b21f5652d0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.476+0000 7f76d6ffd640 0 client.17058 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:37:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.476+0000 7f7700629740 1 -- 172.21.15.28:0/1244412617 >> 172.21.15.28:0/1244412617 conn(0x55b21f5508a0 msgr2=0x55b21f55e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.477+0000 7f7700629740 1 -- 172.21.15.28:0/1244412617 shutdown_connections 2024-04-25T01:37:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.477+0000 7f7700629740 1 -- 172.21.15.28:0/1244412617 wait complete. 2024-04-25T01:37:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.477+0000 7f7700629740 20 client.17058 trim_cache size 0 max 16384 2024-04-25T01:37:31.460 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:37:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.477+0000 7f76fd838640 1 -- 172.21.15.28:0/3106222676 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f76e00a2d00 msgr2=0x7f76e00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.477+0000 7f76fd838640 1 --2- 172.21.15.28:0/3106222676 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f76e00a2d00 0x7f76e00a51c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f76f0005fd0 tx=0x7f76f0004a40 comp rx=0 tx=0).stop 2024-04-25T01:37:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.477+0000 7f76fd838640 1 -- 172.21.15.28:0/3106222676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f8100620 msgr2=0x7f76f8198ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.477+0000 7f76fd838640 1 --2- 172.21.15.28:0/3106222676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f8100620 0x7f76f8198ec0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f76e800d770 tx=0x7f76e800dc30 comp rx=0 tx=0).stop 2024-04-25T01:37:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.477+0000 7f76fd838640 1 -- 172.21.15.28:0/3106222676 shutdown_connections 2024-04-25T01:37:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.477+0000 7f76fd838640 1 --2- 172.21.15.28:0/3106222676 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f76e00a2d00 0x7f76e00a51c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.477+0000 7f76fd838640 1 --2- 172.21.15.28:0/3106222676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76f8106630 0x7f76f81a3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.477+0000 7f76fd838640 1 --2- 172.21.15.28:0/3106222676 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f76f8100fe0 0x7f76f8199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.477+0000 7f76fd838640 1 --2- 172.21.15.28:0/3106222676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f8100620 0x7f76f8198ec0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.477+0000 7f76fd838640 1 -- 172.21.15.28:0/3106222676 >> 172.21.15.28:0/3106222676 conn(0x7f76f80776a0 msgr2=0x7f76f8103230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.477+0000 7f76fd838640 1 -- 172.21.15.28:0/3106222676 shutdown_connections 2024-04-25T01:37:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.477+0000 7f76fd838640 1 -- 172.21.15.28:0/3106222676 wait complete. 2024-04-25T01:37:31.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:37:31.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.592+0000 7f75deea5640 1 -- 172.21.15.28:0/738542817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75d8101710 msgr2=0x7f75d8101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.592+0000 7f75deea5640 1 --2- 172.21.15.28:0/738542817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75d8101710 0x7f75d8101b90 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009930 tx=0x7f75c002a420 comp rx=0 tx=0).stop 2024-04-25T01:37:31.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.592+0000 7f75deea5640 1 -- 172.21.15.28:0/738542817 shutdown_connections 2024-04-25T01:37:31.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.592+0000 7f75deea5640 1 --2- 172.21.15.28:0/738542817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75d81020d0 0x7f75d810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.592+0000 7f75deea5640 1 --2- 172.21.15.28:0/738542817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75d8101710 0x7f75d8101b90 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.592+0000 7f75deea5640 1 --2- 172.21.15.28:0/738542817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75d8107720 0x7f75d8107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.592+0000 7f75deea5640 1 -- 172.21.15.28:0/738542817 >> 172.21.15.28:0/738542817 conn(0x7f75d80fd180 msgr2=0x7f75d80ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.592+0000 7f75deea5640 1 -- 172.21.15.28:0/738542817 shutdown_connections 2024-04-25T01:37:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.592+0000 7f75deea5640 1 -- 172.21.15.28:0/738542817 wait complete. 2024-04-25T01:37:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.593+0000 7f75deea5640 1 Processor -- start 2024-04-25T01:37:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.593+0000 7f75deea5640 1 -- start start 2024-04-25T01:37:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.593+0000 7f75deea5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75d8101710 0x7f75d81980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.593+0000 7f75deea5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75d81020d0 0x7f75d8198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.593+0000 7f75deea5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75d8107720 0x7f75d819c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.593+0000 7f75deea5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75d8111b80 con 0x7f75d8101710 2024-04-25T01:37:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.593+0000 7f75deea5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f75d8111a00 con 0x7f75d81020d0 2024-04-25T01:37:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.593+0000 7f75deea5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f75d8111d00 con 0x7f75d8107720 2024-04-25T01:37:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.593+0000 7f75dcc41640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75d8101710 0x7f75d81980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.593+0000 7f75dcc41640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75d8101710 0x7f75d81980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57766/0 (socket says 172.21.15.28:57766) 2024-04-25T01:37:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.593+0000 7f75dcc41640 1 -- 172.21.15.28:0/893537569 learned_addr learned my addr 172.21.15.28:0/893537569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.593+0000 7f75d7fff640 1 --2- 172.21.15.28:0/893537569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75d81020d0 0x7f75d8198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.593+0000 7f75dd442640 1 --2- 172.21.15.28:0/893537569 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75d8107720 0x7f75d819c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.593+0000 7f75dcc41640 1 -- 172.21.15.28:0/893537569 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75d8107720 msgr2=0x7f75d819c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.593+0000 7f75dcc41640 1 --2- 172.21.15.28:0/893537569 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75d8107720 0x7f75d819c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.593+0000 7f75dcc41640 1 -- 172.21.15.28:0/893537569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75d81020d0 msgr2=0x7f75d8198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.593+0000 7f75dcc41640 1 --2- 172.21.15.28:0/893537569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75d81020d0 0x7f75d8198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.593+0000 7f75dcc41640 1 -- 172.21.15.28:0/893537569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d819d060 con 0x7f75d8101710 2024-04-25T01:37:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.594+0000 7f75dcc41640 1 --2- 172.21.15.28:0/893537569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75d8101710 0x7f75d81980c0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f75d000ebc0 tx=0x7f75d000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.594+0000 7f75d5ffb640 1 -- 172.21.15.28:0/893537569 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d0010630 con 0x7f75d8101710 2024-04-25T01:37:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.594+0000 7f75d5ffb640 1 -- 172.21.15.28:0/893537569 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75d0010cb0 con 0x7f75d8101710 2024-04-25T01:37:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.594+0000 7f75deea5640 1 -- 172.21.15.28:0/893537569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75d819d350 con 0x7f75d8101710 2024-04-25T01:37:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.594+0000 7f75d5ffb640 1 -- 172.21.15.28:0/893537569 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d0018bd0 con 0x7f75d8101710 2024-04-25T01:37:31.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.594+0000 7f75deea5640 1 -- 172.21.15.28:0/893537569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75d81a4be0 con 0x7f75d8101710 2024-04-25T01:37:31.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.596+0000 7f75d5ffb640 1 -- 172.21.15.28:0/893537569 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f75d00107d0 con 0x7f75d8101710 2024-04-25T01:37:31.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.596+0000 7f75e1bbc740 1 Processor -- start 2024-04-25T01:37:31.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.596+0000 7f75e1bbc740 1 -- start start 2024-04-25T01:37:31.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.596+0000 7f75e1bbc740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f4c580 0x55ea49f4c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75e1bbc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4cea0 0x55ea49f57030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75e1bbc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f57570 0x55ea49f59960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75e1bbc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ea49eb1670 con 0x55ea49f57570 2024-04-25T01:37:31.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75e1bbc740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ea49e59970 con 0x55ea49f4c580 2024-04-25T01:37:31.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75e1bbc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ea49ead630 con 0x55ea49f4cea0 2024-04-25T01:37:31.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f57570 0x55ea49f59960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75d5ffb640 1 --2- 172.21.15.28:0/893537569 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f75c40a2d00 0x7f75c40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f57570 0x55ea49f59960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57780/0 (socket says 172.21.15.28:57780) 2024-04-25T01:37:31.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75d7fff640 1 -- 172.21.15.28:0/1710432383 learned_addr learned my addr 172.21.15.28:0/1710432383 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75dd442640 1 --2- 172.21.15.28:0/1710432383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f4c580 0x55ea49f4c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75dcc41640 1 --2- 172.21.15.28:0/1710432383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4cea0 0x55ea49f57030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75d5ffb640 1 -- 172.21.15.28:0/893537569 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f75d00cfe50 con 0x7f75d8101710 2024-04-25T01:37:31.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75dd442640 1 --2- 172.21.15.28:0/893537569 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f75c40a2d00 0x7f75c40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75d7fff640 1 -- 172.21.15.28:0/1710432383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4cea0 msgr2=0x55ea49f57030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75d7fff640 1 --2- 172.21.15.28:0/1710432383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4cea0 0x55ea49f57030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75d7fff640 1 -- 172.21.15.28:0/1710432383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f4c580 msgr2=0x55ea49f4c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75d7fff640 1 --2- 172.21.15.28:0/1710432383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f4c580 0x55ea49f4c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75d7fff640 1 -- 172.21.15.28:0/1710432383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea49e7d400 con 0x55ea49f57570 2024-04-25T01:37:31.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75d7fff640 1 --2- 172.21.15.28:0/1710432383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f57570 0x55ea49f59960 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f75c002aa10 tx=0x7f75c0005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f589659448c9a5c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75dd442640 1 --2- 172.21.15.28:0/893537569 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f75c40a2d00 0x7f75c40a51c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f75c800efd0 tx=0x7f75c80046c0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.597+0000 7f75ceffd640 1 -- 172.21.15.28:0/1710432383 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c0035030 con 0x55ea49f57570 2024-04-25T01:37:31.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.598+0000 7f75ceffd640 1 -- 172.21.15.28:0/1710432383 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75c00043c0 con 0x55ea49f57570 2024-04-25T01:37:31.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.598+0000 7f75ceffd640 1 -- 172.21.15.28:0/1710432383 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c0004de0 con 0x55ea49f57570 2024-04-25T01:37:31.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.598+0000 7f75e1bbc740 1 -- 172.21.15.28:0/1710432383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f57570 msgr2=0x55ea49f59960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.598+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/1710432383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f57570 0x55ea49f59960 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f75c002aa10 tx=0x7f75c0005e60 comp rx=0 tx=0).stop 2024-04-25T01:37:31.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.598+0000 7f75e1bbc740 1 -- 172.21.15.28:0/1710432383 shutdown_connections 2024-04-25T01:37:31.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.598+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/1710432383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f57570 0x55ea49f59960 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.598+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/1710432383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4cea0 0x55ea49f57030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.598+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/1710432383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f4c580 0x55ea49f4c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.598+0000 7f75e1bbc740 1 -- 172.21.15.28:0/1710432383 >> 172.21.15.28:0/1710432383 conn(0x55ea49f488a0 msgr2=0x55ea49f48c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.598+0000 7f75e1bbc740 1 -- 172.21.15.28:0/1710432383 shutdown_connections 2024-04-25T01:37:31.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.598+0000 7f75e1bbc740 1 -- 172.21.15.28:0/1710432383 wait complete. 2024-04-25T01:37:31.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.599+0000 7f75e1bbc740 1 -- start start 2024-04-25T01:37:31.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.599+0000 7f75e1bbc740 1 Processor -- start 2024-04-25T01:37:31.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.599+0000 7f75e1bbc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4c580 0x55ea49f56eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.599+0000 7f75e1bbc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4cea0 0x55ea49f541a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.599+0000 7f75e1bbc740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f57570 0x55ea49f546e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.599+0000 7f75e1bbc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ea49f5a300 con 0x55ea49f4c580 2024-04-25T01:37:31.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.599+0000 7f75e1bbc740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ea49e6d050 con 0x55ea49f57570 2024-04-25T01:37:31.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.599+0000 7f75e1bbc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ea49f5a480 con 0x55ea49f4cea0 2024-04-25T01:37:31.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.599+0000 7f75e1bbc740 10 client.0 _ll_register_callbacks cb 0x55ea49bc4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:31.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.599+0000 7f75dcc41640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4c580 0x55ea49f56eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.599+0000 7f75dcc41640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4c580 0x55ea49f56eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57782/0 (socket says 172.21.15.28:57782) 2024-04-25T01:37:31.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.599+0000 7f75dcc41640 1 -- 172.21.15.28:0/726542757 learned_addr learned my addr 172.21.15.28:0/726542757 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.599+0000 7f75dd442640 1 --2- 172.21.15.28:0/726542757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f57570 0x55ea49f546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.599+0000 7f75dcc41640 1 -- 172.21.15.28:0/726542757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4cea0 msgr2=0x55ea49f541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.599+0000 7f75d7fff640 1 --2- 172.21.15.28:0/726542757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4cea0 0x55ea49f541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.599+0000 7f75dcc41640 1 --2- 172.21.15.28:0/726542757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4cea0 0x55ea49f541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.599+0000 7f75dcc41640 1 -- 172.21.15.28:0/726542757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f57570 msgr2=0x55ea49f546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.599+0000 7f75dcc41640 1 --2- 172.21.15.28:0/726542757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f57570 0x55ea49f546e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.600+0000 7f75dcc41640 1 -- 172.21.15.28:0/726542757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea49e3edc0 con 0x55ea49f4c580 2024-04-25T01:37:31.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.600+0000 7f75d7fff640 1 --2- 172.21.15.28:0/726542757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4cea0 0x55ea49f541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:31.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.600+0000 7f75dcc41640 1 --2- 172.21.15.28:0/726542757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4c580 0x55ea49f56eb0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f75d009bd30 tx=0x7f75d001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ffc82a6d600c79 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.600+0000 7f75b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:31.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.600+0000 7f75b7fff640 1 -- 172.21.15.28:0/726542757 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d0010e60 con 0x55ea49f4c580 2024-04-25T01:37:31.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.600+0000 7f75b7fff640 1 -- 172.21.15.28:0/726542757 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f75d00986a0 con 0x55ea49f4c580 2024-04-25T01:37:31.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.600+0000 7f75e1bbc740 1 -- 172.21.15.28:0/726542757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75c00043c0 con 0x55ea49f4c580 2024-04-25T01:37:31.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.600+0000 7f75b7fff640 1 -- 172.21.15.28:0/726542757 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f75d009c030 con 0x55ea49f4c580 2024-04-25T01:37:31.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.600+0000 7f75e1bbc740 10 client.17070 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:31.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.600+0000 7f75e1bbc740 1 -- 172.21.15.28:0/726542757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ea49bc53f0 con 0x55ea49f4c580 2024-04-25T01:37:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.600+0000 7f75b7fff640 1 -- 172.21.15.28:0/726542757 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75c00043c0 con 0x55ea49f4c580 2024-04-25T01:37:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.600+0000 7f75b7fff640 1 -- 172.21.15.28:0/726542757 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f75d001dcb0 con 0x55ea49f4c580 2024-04-25T01:37:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.600+0000 7f75e1bbc740 10 client.17070 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.600+0000 7f75e1bbc740 10 client.17070 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:37:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.600+0000 7f75e1bbc740 20 client.17070 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.600+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/726542757 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55ea49f67540 0x55ea49f699d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.600+0000 7f75e1bbc740 4 client.17070 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:37:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.600+0000 7f75e1bbc740 1 -- 172.21.15.28:0/726542757 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea49ebebb0 con 0x55ea49f67540 2024-04-25T01:37:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.601+0000 7f75d7fff640 1 --2- 172.21.15.28:0/726542757 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55ea49f67540 0x55ea49f699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.601+0000 7f75d7fff640 1 --2- 172.21.15.28:0/726542757 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55ea49f67540 0x55ea49f699d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9e41247d9483744c server_cookie=4632bb1dc06b7da1 in_seq=0 out_seq=0 2024-04-25T01:37:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.601+0000 7f75b7fff640 10 client.17070 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:37:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.602+0000 7f75b7fff640 1 -- 172.21.15.28:0/726542757 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ea49ebebb0 con 0x55ea49f67540 2024-04-25T01:37:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.602+0000 7f75b7fff640 10 client.17070 handle_command_reply: tid=0 2024-04-25T01:37:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.602+0000 7f75e1bbc740 1 client.17070 shutdown 2024-04-25T01:37:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.602+0000 7f75e1bbc740 1 -- 172.21.15.28:0/726542757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4c580 msgr2=0x55ea49f56eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.602+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/726542757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4c580 0x55ea49f56eb0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f75d009bd30 tx=0x7f75d001f000 comp rx=0 tx=0).stop 2024-04-25T01:37:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.602+0000 7f75e1bbc740 1 -- 172.21.15.28:0/726542757 shutdown_connections 2024-04-25T01:37:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.602+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/726542757 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55ea49f67540 0x55ea49f699d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.602+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/726542757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f57570 0x55ea49f546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.602+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/726542757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4cea0 0x55ea49f541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.602+0000 7f75b7fff640 0 client.17070 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:37:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.602+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/726542757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4c580 0x55ea49f56eb0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.602+0000 7f75e1bbc740 1 -- 172.21.15.28:0/726542757 >> 172.21.15.28:0/726542757 conn(0x55ea49f488a0 msgr2=0x55ea49f47b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.602+0000 7f75e1bbc740 1 -- 172.21.15.28:0/726542757 shutdown_connections 2024-04-25T01:37:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.602+0000 7f75e1bbc740 1 -- 172.21.15.28:0/726542757 wait complete. 2024-04-25T01:37:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.602+0000 7f75e1bbc740 20 client.17070 trim_cache size 0 max 16384 2024-04-25T01:37:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75e1bbc740 1 Processor -- start 2024-04-25T01:37:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75e1bbc740 1 -- start start 2024-04-25T01:37:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75e1bbc740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f488a0 0x55ea49f631c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75e1bbc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4c580 0x55ea49f6f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75e1bbc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4cea0 0x55ea49f6f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75e1bbc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75d81aea30 con 0x55ea49f4cea0 2024-04-25T01:37:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75e1bbc740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f75d81b0570 con 0x55ea49f488a0 2024-04-25T01:37:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75e1bbc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f75d81af100 con 0x55ea49f4c580 2024-04-25T01:37:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75dd442640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4cea0 0x55ea49f6f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75dd442640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4cea0 0x55ea49f6f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57796/0 (socket says 172.21.15.28:57796) 2024-04-25T01:37:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75dd442640 1 -- 172.21.15.28:0/3034347636 learned_addr learned my addr 172.21.15.28:0/3034347636 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75d7fff640 1 --2- 172.21.15.28:0/3034347636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4c580 0x55ea49f6f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75dd442640 1 -- 172.21.15.28:0/3034347636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4c580 msgr2=0x55ea49f6f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75dd442640 1 --2- 172.21.15.28:0/3034347636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4c580 0x55ea49f6f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75dcc41640 1 --2- 172.21.15.28:0/3034347636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f488a0 0x55ea49f631c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75dd442640 1 -- 172.21.15.28:0/3034347636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f488a0 msgr2=0x55ea49f631c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75dd442640 1 --2- 172.21.15.28:0/3034347636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f488a0 0x55ea49f631c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75dd442640 1 -- 172.21.15.28:0/3034347636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea49c8f7d0 con 0x55ea49f4cea0 2024-04-25T01:37:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75dcc41640 1 --2- 172.21.15.28:0/3034347636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f488a0 0x55ea49f631c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.616+0000 7f75dd442640 1 --2- 172.21.15.28:0/3034347636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4cea0 0x55ea49f6f830 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f75c8010910 tx=0x7f75c8015070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a8d636446bcee81 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.617+0000 7f75ce7fc640 1 -- 172.21.15.28:0/3034347636 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c801eb40 con 0x55ea49f4cea0 2024-04-25T01:37:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.617+0000 7f75ce7fc640 1 -- 172.21.15.28:0/3034347636 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75c8005500 con 0x55ea49f4cea0 2024-04-25T01:37:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.617+0000 7f75ce7fc640 1 -- 172.21.15.28:0/3034347636 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c8004df0 con 0x55ea49f4cea0 2024-04-25T01:37:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.617+0000 7f75e1bbc740 1 -- 172.21.15.28:0/3034347636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4cea0 msgr2=0x55ea49f6f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.617+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/3034347636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4cea0 0x55ea49f6f830 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f75c8010910 tx=0x7f75c8015070 comp rx=0 tx=0).stop 2024-04-25T01:37:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.617+0000 7f75e1bbc740 1 -- 172.21.15.28:0/3034347636 shutdown_connections 2024-04-25T01:37:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.617+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/3034347636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4cea0 0x55ea49f6f830 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.617+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/3034347636 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4c580 0x55ea49f6f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.617+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/3034347636 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f488a0 0x55ea49f631c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.617+0000 7f75e1bbc740 1 -- 172.21.15.28:0/3034347636 >> 172.21.15.28:0/3034347636 conn(0x55ea49f54840 msgr2=0x55ea49f47b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.617+0000 7f75e1bbc740 1 -- 172.21.15.28:0/3034347636 shutdown_connections 2024-04-25T01:37:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.617+0000 7f75e1bbc740 1 -- 172.21.15.28:0/3034347636 wait complete. 2024-04-25T01:37:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75e1bbc740 1 -- start start 2024-04-25T01:37:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75e1bbc740 1 Processor -- start 2024-04-25T01:37:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75e1bbc740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f488a0 0x55ea49f5d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75e1bbc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4c580 0x55ea49f5d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75e1bbc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4cea0 0x55ea49f5ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75e1bbc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75d81afea0 con 0x55ea49f4c580 2024-04-25T01:37:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75e1bbc740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f75d81af7d0 con 0x55ea49f488a0 2024-04-25T01:37:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75e1bbc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f75d81aca10 con 0x55ea49f4cea0 2024-04-25T01:37:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75e1bbc740 10 client.0 _ll_register_callbacks cb 0x7f75d000eb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4c580 0x55ea49f5d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4c580 0x55ea49f5d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57808/0 (socket says 172.21.15.28:57808) 2024-04-25T01:37:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75d7fff640 1 -- 172.21.15.28:0/3951066829 learned_addr learned my addr 172.21.15.28:0/3951066829 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75dd442640 1 --2- 172.21.15.28:0/3951066829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4cea0 0x55ea49f5ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75dcc41640 1 --2- 172.21.15.28:0/3951066829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f488a0 0x55ea49f5d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75d7fff640 1 -- 172.21.15.28:0/3951066829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4cea0 msgr2=0x55ea49f5ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75d7fff640 1 --2- 172.21.15.28:0/3951066829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4cea0 0x55ea49f5ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75d7fff640 1 -- 172.21.15.28:0/3951066829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f488a0 msgr2=0x55ea49f5d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75d7fff640 1 --2- 172.21.15.28:0/3951066829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f488a0 0x55ea49f5d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75d7fff640 1 -- 172.21.15.28:0/3951066829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea49fccd70 con 0x55ea49f4c580 2024-04-25T01:37:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.618+0000 7f75dcc41640 1 --2- 172.21.15.28:0/3951066829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f488a0 0x55ea49f5d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.619+0000 7f75d7fff640 1 --2- 172.21.15.28:0/3951066829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4c580 0x55ea49f5d870 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f75c0005ab0 tx=0x7f75c0031490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83aa85b051b60779 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.619+0000 7f75b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.619+0000 7f75b7fff640 1 -- 172.21.15.28:0/3951066829 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c0057060 con 0x55ea49f4c580 2024-04-25T01:37:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.619+0000 7f75b7fff640 1 -- 172.21.15.28:0/3951066829 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f75c00315b0 con 0x55ea49f4c580 2024-04-25T01:37:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.619+0000 7f75e1bbc740 1 -- 172.21.15.28:0/3951066829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75c8005500 con 0x55ea49f4c580 2024-04-25T01:37:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.619+0000 7f75b7fff640 1 -- 172.21.15.28:0/3951066829 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f75c005e050 con 0x55ea49f4c580 2024-04-25T01:37:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.619+0000 7f75e1bbc740 10 client.17076 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.619+0000 7f75e1bbc740 1 -- 172.21.15.28:0/3951066829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ea49fce000 con 0x55ea49f4c580 2024-04-25T01:37:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.619+0000 7f75b7fff640 1 -- 172.21.15.28:0/3951066829 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75c8005500 con 0x55ea49f4c580 2024-04-25T01:37:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.619+0000 7f75b7fff640 1 -- 172.21.15.28:0/3951066829 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f75c004e040 con 0x55ea49f4c580 2024-04-25T01:37:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.619+0000 7f75e1bbc740 10 client.17076 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.619+0000 7f75e1bbc740 10 client.17076 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:37:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.619+0000 7f75e1bbc740 20 client.17076 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.619+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/3951066829 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55ea49f57570 0x55ea49fd0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.619+0000 7f75e1bbc740 4 client.17076 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:37:31.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.619+0000 7f75e1bbc740 1 -- 172.21.15.28:0/3951066829 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ea49eb8f00 con 0x55ea49f57570 2024-04-25T01:37:31.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.620+0000 7f75dcc41640 1 --2- 172.21.15.28:0/3951066829 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55ea49f57570 0x55ea49fd0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.620+0000 7f75dcc41640 1 --2- 172.21.15.28:0/3951066829 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55ea49f57570 0x55ea49fd0350 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=65d87437340b6d9c server_cookie=1f490f5d6dfbab45 in_seq=0 out_seq=0 2024-04-25T01:37:31.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.620+0000 7f75b7fff640 10 client.17076 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:37:31.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.620+0000 7f75b7fff640 1 -- 172.21.15.28:0/3951066829 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ea49eb8f00 con 0x55ea49f57570 2024-04-25T01:37:31.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.620+0000 7f75b7fff640 10 client.17076 handle_command_reply: tid=0 2024-04-25T01:37:31.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.620+0000 7f75e1bbc740 1 client.17076 shutdown 2024-04-25T01:37:31.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.620+0000 7f75e1bbc740 1 -- 172.21.15.28:0/3951066829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4c580 msgr2=0x55ea49f5d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.620+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/3951066829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4c580 0x55ea49f5d870 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f75c0005ab0 tx=0x7f75c0031490 comp rx=0 tx=0).stop 2024-04-25T01:37:31.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.620+0000 7f75e1bbc740 1 -- 172.21.15.28:0/3951066829 shutdown_connections 2024-04-25T01:37:31.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.620+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/3951066829 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55ea49f57570 0x55ea49fd0350 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.620+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/3951066829 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea49f4cea0 0x55ea49f5ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.620+0000 7f75b7fff640 0 client.17076 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:37:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.620+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/3951066829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea49f4c580 0x55ea49f5d870 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.620+0000 7f75e1bbc740 1 --2- 172.21.15.28:0/3951066829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea49f488a0 0x55ea49f5d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.620+0000 7f75e1bbc740 1 -- 172.21.15.28:0/3951066829 >> 172.21.15.28:0/3951066829 conn(0x55ea49f54840 msgr2=0x55ea49f621b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.620+0000 7f75e1bbc740 1 -- 172.21.15.28:0/3951066829 shutdown_connections 2024-04-25T01:37:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.621+0000 7f75e1bbc740 1 -- 172.21.15.28:0/3951066829 wait complete. 2024-04-25T01:37:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.621+0000 7f75e1bbc740 20 client.17076 trim_cache size 0 max 16384 2024-04-25T01:37:31.606 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:37:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.621+0000 7f75deea5640 1 -- 172.21.15.28:0/893537569 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f75c40a2d00 msgr2=0x7f75c40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.621+0000 7f75deea5640 1 --2- 172.21.15.28:0/893537569 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f75c40a2d00 0x7f75c40a51c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f75c800efd0 tx=0x7f75c80046c0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.621+0000 7f75deea5640 1 -- 172.21.15.28:0/893537569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75d8101710 msgr2=0x7f75d81980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.621+0000 7f75deea5640 1 --2- 172.21.15.28:0/893537569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75d8101710 0x7f75d81980c0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f75d000ebc0 tx=0x7f75d000c6a0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.621+0000 7f75deea5640 1 -- 172.21.15.28:0/893537569 shutdown_connections 2024-04-25T01:37:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.621+0000 7f75deea5640 1 --2- 172.21.15.28:0/893537569 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f75c40a2d00 0x7f75c40a51c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.621+0000 7f75deea5640 1 --2- 172.21.15.28:0/893537569 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75d8107720 0x7f75d819c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.621+0000 7f75deea5640 1 --2- 172.21.15.28:0/893537569 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75d81020d0 0x7f75d8198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.621+0000 7f75deea5640 1 --2- 172.21.15.28:0/893537569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75d8101710 0x7f75d81980c0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.621+0000 7f75deea5640 1 -- 172.21.15.28:0/893537569 >> 172.21.15.28:0/893537569 conn(0x7f75d80fd180 msgr2=0x7f75d80fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.621+0000 7f75deea5640 1 -- 172.21.15.28:0/893537569 shutdown_connections 2024-04-25T01:37:31.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.621+0000 7f75deea5640 1 -- 172.21.15.28:0/893537569 wait complete. 2024-04-25T01:37:31.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:37:31.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.737+0000 7f02f6b3b640 1 -- 172.21.15.28:0/4241288047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02f0100fe0 msgr2=0x7f02f010d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.737+0000 7f02f6b3b640 1 --2- 172.21.15.28:0/4241288047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02f0100fe0 0x7f02f010d4f0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009cd0 tx=0x7f02e802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.737+0000 7f02f6b3b640 1 -- 172.21.15.28:0/4241288047 shutdown_connections 2024-04-25T01:37:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.737+0000 7f02f6b3b640 1 --2- 172.21.15.28:0/4241288047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02f0100fe0 0x7f02f010d4f0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.737+0000 7f02f6b3b640 1 --2- 172.21.15.28:0/4241288047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02f0100620 0x7f02f0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.737+0000 7f02f6b3b640 1 --2- 172.21.15.28:0/4241288047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02f0106630 0x7f02f0106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.737+0000 7f02f6b3b640 1 -- 172.21.15.28:0/4241288047 >> 172.21.15.28:0/4241288047 conn(0x7f02f00776a0 msgr2=0x7f02f0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.737+0000 7f02f6b3b640 1 -- 172.21.15.28:0/4241288047 shutdown_connections 2024-04-25T01:37:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.737+0000 7f02f6b3b640 1 -- 172.21.15.28:0/4241288047 wait complete. 2024-04-25T01:37:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.737+0000 7f02f6b3b640 1 Processor -- start 2024-04-25T01:37:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.737+0000 7f02f6b3b640 1 -- start start 2024-04-25T01:37:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02f6b3b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02f0100620 0x7f02f0198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02f6b3b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02f0100fe0 0x7f02f0198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02f6b3b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02f0106630 0x7f02f019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02f6b3b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f02f0110a40 con 0x7f02f0100620 2024-04-25T01:37:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02f6b3b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01108c0 con 0x7f02f0100fe0 2024-04-25T01:37:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02f6b3b640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f02f0110bc0 con 0x7f02f0106630 2024-04-25T01:37:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02f48d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02f0100620 0x7f02f0198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02f48d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02f0100620 0x7f02f0198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57822/0 (socket says 172.21.15.28:57822) 2024-04-25T01:37:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02f48d7640 1 -- 172.21.15.28:0/300806337 learned_addr learned my addr 172.21.15.28:0/300806337 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02e7fff640 1 --2- 172.21.15.28:0/300806337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02f0100fe0 0x7f02f0198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02f50d8640 1 --2- 172.21.15.28:0/300806337 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02f0106630 0x7f02f019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02f48d7640 1 -- 172.21.15.28:0/300806337 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02f0106630 msgr2=0x7f02f019c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02f48d7640 1 --2- 172.21.15.28:0/300806337 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02f0106630 0x7f02f019c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02f48d7640 1 -- 172.21.15.28:0/300806337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02f0100fe0 msgr2=0x7f02f0198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02f48d7640 1 --2- 172.21.15.28:0/300806337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02f0100fe0 0x7f02f0198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02f48d7640 1 -- 172.21.15.28:0/300806337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f019d000 con 0x7f02f0100620 2024-04-25T01:37:31.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02f50d8640 1 --2- 172.21.15.28:0/300806337 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02f0106630 0x7f02f019c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:31.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02f48d7640 1 --2- 172.21.15.28:0/300806337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02f0100620 0x7f02f0198010 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00d770 tx=0x7f02ec00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02e5ffb640 1 -- 172.21.15.28:0/300806337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec017070 con 0x7f02f0100620 2024-04-25T01:37:31.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02e5ffb640 1 -- 172.21.15.28:0/300806337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02ec005500 con 0x7f02f0100620 2024-04-25T01:37:31.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02f6b3b640 1 -- 172.21.15.28:0/300806337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f019d2f0 con 0x7f02f0100620 2024-04-25T01:37:31.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.738+0000 7f02e5ffb640 1 -- 172.21.15.28:0/300806337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec004e00 con 0x7f02f0100620 2024-04-25T01:37:31.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.739+0000 7f02f6b3b640 1 -- 172.21.15.28:0/300806337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f01a4b80 con 0x7f02f0100620 2024-04-25T01:37:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.741+0000 7f02e5ffb640 1 -- 172.21.15.28:0/300806337 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f02ec004fa0 con 0x7f02f0100620 2024-04-25T01:37:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.741+0000 7f02f984a740 1 Processor -- start 2024-04-25T01:37:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.741+0000 7f02f984a740 1 -- start start 2024-04-25T01:37:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.741+0000 7f02e5ffb640 1 --2- 172.21.15.28:0/300806337 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f02d80a2d80 0x7f02d80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.741+0000 7f02f984a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d292b0 0x55c549d29690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.741+0000 7f02f984a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d29c60 0x55c549d38440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.741+0000 7f02f984a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d38980 0x55c549d3ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.741+0000 7f02f984a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c549c8e670 con 0x55c549d38980 2024-04-25T01:37:31.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.741+0000 7f02f984a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c549c36970 con 0x55c549d292b0 2024-04-25T01:37:31.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.741+0000 7f02f984a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c549c8a630 con 0x55c549d29c60 2024-04-25T01:37:31.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.741+0000 7f02e5ffb640 1 -- 172.21.15.28:0/300806337 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f02ec0cf1d0 con 0x7f02f0100620 2024-04-25T01:37:31.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.741+0000 7f02f50d8640 1 --2- 172.21.15.28:0/300806337 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f02d80a2d80 0x7f02d80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.742+0000 7f02f48d7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d292b0 0x55c549d29690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.742+0000 7f02f48d7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d292b0 0x55c549d29690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:33908/0 (socket says 172.21.15.28:33908) 2024-04-25T01:37:31.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.742+0000 7f02e7fff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d29c60 0x55c549d38440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.742+0000 7f02f48d7640 1 -- 172.21.15.28:0/938042850 learned_addr learned my addr 172.21.15.28:0/938042850 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.742+0000 7f02f50d8640 1 --2- 172.21.15.28:0/938042850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d38980 0x55c549d3ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.742+0000 7f02f48d7640 1 -- 172.21.15.28:0/938042850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d29c60 msgr2=0x55c549d38440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.742+0000 7f02f48d7640 1 --2- 172.21.15.28:0/938042850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d29c60 0x55c549d38440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.742+0000 7f02f48d7640 1 -- 172.21.15.28:0/938042850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d38980 msgr2=0x55c549d3ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.742+0000 7f02f48d7640 1 --2- 172.21.15.28:0/938042850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d38980 0x55c549d3ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.742+0000 7f02f48d7640 1 -- 172.21.15.28:0/938042850 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c549c5a400 con 0x55c549d292b0 2024-04-25T01:37:31.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.742+0000 7f02e7fff640 1 --2- 172.21.15.28:0/938042850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d29c60 0x55c549d38440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:37:31.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.742+0000 7f02f50d8640 1 --2- 172.21.15.28:0/300806337 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f02d80a2d80 0x7f02d80a5240 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f02e8031860 tx=0x7f02e8009910 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.742+0000 7f02f48d7640 1 --2- 172.21.15.28:0/938042850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d292b0 0x55c549d29690 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0cc640 tx=0x7f02ec065760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=193050b581449074 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.742+0000 7f02d6ffd640 1 -- 172.21.15.28:0/938042850 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec020070 con 0x55c549d292b0 2024-04-25T01:37:31.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.742+0000 7f02d6ffd640 1 -- 172.21.15.28:0/938042850 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02ec09bab0 con 0x55c549d292b0 2024-04-25T01:37:31.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.742+0000 7f02d6ffd640 1 -- 172.21.15.28:0/938042850 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec0996b0 con 0x55c549d292b0 2024-04-25T01:37:31.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.743+0000 7f02f984a740 1 -- 172.21.15.28:0/938042850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d292b0 msgr2=0x55c549d29690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.743+0000 7f02f984a740 1 --2- 172.21.15.28:0/938042850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d292b0 0x55c549d29690 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0cc640 tx=0x7f02ec065760 comp rx=0 tx=0).stop 2024-04-25T01:37:31.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.743+0000 7f02f984a740 1 -- 172.21.15.28:0/938042850 shutdown_connections 2024-04-25T01:37:31.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.743+0000 7f02f984a740 1 --2- 172.21.15.28:0/938042850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d38980 0x55c549d3ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.743+0000 7f02f984a740 1 --2- 172.21.15.28:0/938042850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d29c60 0x55c549d38440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.743+0000 7f02f984a740 1 --2- 172.21.15.28:0/938042850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d292b0 0x55c549d29690 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.743+0000 7f02f984a740 1 -- 172.21.15.28:0/938042850 >> 172.21.15.28:0/938042850 conn(0x55c549d258a0 msgr2=0x55c549d25c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.743+0000 7f02f984a740 1 -- 172.21.15.28:0/938042850 shutdown_connections 2024-04-25T01:37:31.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.743+0000 7f02f984a740 1 -- 172.21.15.28:0/938042850 wait complete. 2024-04-25T01:37:31.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.743+0000 7f02f984a740 1 -- start start 2024-04-25T01:37:31.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.743+0000 7f02f984a740 1 Processor -- start 2024-04-25T01:37:31.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02f984a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d292b0 0x55c549d382c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02f984a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d29c60 0x55c549d355b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02f984a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d38980 0x55c549d35af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02f984a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c549d3b6e0 con 0x55c549d29c60 2024-04-25T01:37:31.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02f984a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c549c4a050 con 0x55c549d292b0 2024-04-25T01:37:31.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02f984a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c549d3b860 con 0x55c549d38980 2024-04-25T01:37:31.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02f984a740 10 client.0 _ll_register_callbacks cb 0x55c5499a1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:31.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02e7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d29c60 0x55c549d355b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02e7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d29c60 0x55c549d355b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57838/0 (socket says 172.21.15.28:57838) 2024-04-25T01:37:31.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02e7fff640 1 -- 172.21.15.28:0/2091289820 learned_addr learned my addr 172.21.15.28:0/2091289820 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02f48d7640 1 --2- 172.21.15.28:0/2091289820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d292b0 0x55c549d382c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02f50d8640 1 --2- 172.21.15.28:0/2091289820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d38980 0x55c549d35af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02e7fff640 1 -- 172.21.15.28:0/2091289820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d38980 msgr2=0x55c549d35af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02e7fff640 1 --2- 172.21.15.28:0/2091289820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d38980 0x55c549d35af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02e7fff640 1 -- 172.21.15.28:0/2091289820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d292b0 msgr2=0x55c549d382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02e7fff640 1 --2- 172.21.15.28:0/2091289820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d292b0 0x55c549d382c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02e7fff640 1 -- 172.21.15.28:0/2091289820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c549c1bdc0 con 0x55c549d29c60 2024-04-25T01:37:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02e7fff640 1 --2- 172.21.15.28:0/2091289820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d29c60 0x55c549d355b0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f02e0012700 tx=0x7f02e0012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b677d1228ea9e3d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02cffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.744+0000 7f02cffff640 1 -- 172.21.15.28:0/2091289820 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e0004260 con 0x55c549d29c60 2024-04-25T01:37:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.745+0000 7f02cffff640 1 -- 172.21.15.28:0/2091289820 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f02e0004400 con 0x55c549d29c60 2024-04-25T01:37:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.745+0000 7f02f984a740 1 -- 172.21.15.28:0/2091289820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02ec09bab0 con 0x55c549d29c60 2024-04-25T01:37:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.745+0000 7f02cffff640 1 -- 172.21.15.28:0/2091289820 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f02e00119a0 con 0x55c549d29c60 2024-04-25T01:37:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.745+0000 7f02f984a740 10 client.17088 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.745+0000 7f02f984a740 1 -- 172.21.15.28:0/2091289820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c5499a23f0 con 0x55c549d29c60 2024-04-25T01:37:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.745+0000 7f02cffff640 1 -- 172.21.15.28:0/2091289820 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02ec09bab0 con 0x55c549d29c60 2024-04-25T01:37:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.745+0000 7f02cffff640 1 -- 172.21.15.28:0/2091289820 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f02e000f440 con 0x55c549d29c60 2024-04-25T01:37:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.745+0000 7f02f984a740 10 client.17088 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.745+0000 7f02f984a740 10 client.17088 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.745+0000 7f02f984a740 20 client.17088 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.745+0000 7f02f984a740 1 --2- 172.21.15.28:0/2091289820 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c549d489c0 0x55c549d4ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.745+0000 7f02f984a740 4 client.17088 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:37:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.745+0000 7f02f984a740 1 -- 172.21.15.28:0/2091289820 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c549c9bbb0 con 0x55c549d489c0 2024-04-25T01:37:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.745+0000 7f02f48d7640 1 --2- 172.21.15.28:0/2091289820 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c549d489c0 0x55c549d4ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.746+0000 7f02f48d7640 1 --2- 172.21.15.28:0/2091289820 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c549d489c0 0x55c549d4ae20 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e441f5223b84fe19 server_cookie=d86b79328e194e94 in_seq=0 out_seq=0 2024-04-25T01:37:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.746+0000 7f02cffff640 10 client.17088 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.746+0000 7f02cffff640 1 -- 172.21.15.28:0/2091289820 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c549c9bbb0 con 0x55c549d489c0 2024-04-25T01:37:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.746+0000 7f02cffff640 10 client.17088 handle_command_reply: tid=0 2024-04-25T01:37:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.747+0000 7f02f984a740 1 client.17088 shutdown 2024-04-25T01:37:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.747+0000 7f02f984a740 1 -- 172.21.15.28:0/2091289820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d29c60 msgr2=0x55c549d355b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.747+0000 7f02f984a740 1 --2- 172.21.15.28:0/2091289820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d29c60 0x55c549d355b0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f02e0012700 tx=0x7f02e0012bc0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.747+0000 7f02f984a740 1 -- 172.21.15.28:0/2091289820 shutdown_connections 2024-04-25T01:37:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.747+0000 7f02f984a740 1 --2- 172.21.15.28:0/2091289820 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c549d489c0 0x55c549d4ae20 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.747+0000 7f02f984a740 1 --2- 172.21.15.28:0/2091289820 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d38980 0x55c549d35af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.747+0000 7f02f984a740 1 --2- 172.21.15.28:0/2091289820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d29c60 0x55c549d355b0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.747+0000 7f02f984a740 1 --2- 172.21.15.28:0/2091289820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d292b0 0x55c549d382c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.747+0000 7f02cffff640 0 client.17088 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:31.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.747+0000 7f02f984a740 1 -- 172.21.15.28:0/2091289820 >> 172.21.15.28:0/2091289820 conn(0x55c549d258a0 msgr2=0x55c549d2b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.747+0000 7f02f984a740 1 -- 172.21.15.28:0/2091289820 shutdown_connections 2024-04-25T01:37:31.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.747+0000 7f02f984a740 1 -- 172.21.15.28:0/2091289820 wait complete. 2024-04-25T01:37:31.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.747+0000 7f02f984a740 20 client.17088 trim_cache size 0 max 16384 2024-04-25T01:37:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.760+0000 7f02f984a740 1 Processor -- start 2024-04-25T01:37:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.760+0000 7f02f984a740 1 -- start start 2024-04-25T01:37:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.761+0000 7f02f984a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d292b0 0x55c549d303e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.761+0000 7f02f984a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d29c60 0x55c549d37df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.761+0000 7f02f984a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d38980 0x55c549d50730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.761+0000 7f02f984a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01ac850 con 0x55c549d292b0 2024-04-25T01:37:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.761+0000 7f02f984a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01ae390 con 0x55c549d29c60 2024-04-25T01:37:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.761+0000 7f02f984a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01acf20 con 0x55c549d38980 2024-04-25T01:37:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.761+0000 7f02f48d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d292b0 0x55c549d303e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.761+0000 7f02f48d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d292b0 0x55c549d303e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57846/0 (socket says 172.21.15.28:57846) 2024-04-25T01:37:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.761+0000 7f02f48d7640 1 -- 172.21.15.28:0/1429834412 learned_addr learned my addr 172.21.15.28:0/1429834412 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.761+0000 7f02f48d7640 1 -- 172.21.15.28:0/1429834412 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d38980 msgr2=0x55c549d50730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.761+0000 7f02f48d7640 1 --2- 172.21.15.28:0/1429834412 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d38980 0x55c549d50730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.761+0000 7f02f48d7640 1 -- 172.21.15.28:0/1429834412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d29c60 msgr2=0x55c549d37df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.761+0000 7f02f48d7640 1 --2- 172.21.15.28:0/1429834412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d29c60 0x55c549d37df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.761+0000 7f02f48d7640 1 -- 172.21.15.28:0/1429834412 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c549d34560 con 0x55c549d292b0 2024-04-25T01:37:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.761+0000 7f02f48d7640 1 --2- 172.21.15.28:0/1429834412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d292b0 0x55c549d303e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0ccbe0 tx=0x7f02ec0a4020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f759e1337e3c433e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.761+0000 7f02d67fc640 1 -- 172.21.15.28:0/1429834412 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec020040 con 0x55c549d292b0 2024-04-25T01:37:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.761+0000 7f02d67fc640 1 -- 172.21.15.28:0/1429834412 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02ec005290 con 0x55c549d292b0 2024-04-25T01:37:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.761+0000 7f02d67fc640 1 -- 172.21.15.28:0/1429834412 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec0653d0 con 0x55c549d292b0 2024-04-25T01:37:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.762+0000 7f02f984a740 1 -- 172.21.15.28:0/1429834412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d292b0 msgr2=0x55c549d303e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.762+0000 7f02f984a740 1 --2- 172.21.15.28:0/1429834412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d292b0 0x55c549d303e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0ccbe0 tx=0x7f02ec0a4020 comp rx=0 tx=0).stop 2024-04-25T01:37:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.762+0000 7f02f984a740 1 -- 172.21.15.28:0/1429834412 shutdown_connections 2024-04-25T01:37:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.762+0000 7f02f984a740 1 --2- 172.21.15.28:0/1429834412 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d38980 0x55c549d50730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.762+0000 7f02f984a740 1 --2- 172.21.15.28:0/1429834412 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d29c60 0x55c549d37df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.762+0000 7f02f984a740 1 --2- 172.21.15.28:0/1429834412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d292b0 0x55c549d303e0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.762+0000 7f02f984a740 1 -- 172.21.15.28:0/1429834412 >> 172.21.15.28:0/1429834412 conn(0x55c549d258a0 msgr2=0x55c549d2b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.762+0000 7f02f984a740 1 -- 172.21.15.28:0/1429834412 shutdown_connections 2024-04-25T01:37:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.762+0000 7f02f984a740 1 -- 172.21.15.28:0/1429834412 wait complete. 2024-04-25T01:37:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.762+0000 7f02f984a740 1 -- start start 2024-04-25T01:37:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.762+0000 7f02f984a740 1 Processor -- start 2024-04-25T01:37:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02f984a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d292b0 0x55c549d40ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02f984a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d29c60 0x55c549d3dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02f984a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d38980 0x55c549d3e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02f984a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01adcc0 con 0x55c549d29c60 2024-04-25T01:37:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02f984a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01ad5f0 con 0x55c549d292b0 2024-04-25T01:37:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02f984a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f02f01aa830 con 0x55c549d38980 2024-04-25T01:37:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02f984a740 10 client.0 _ll_register_callbacks cb 0x7f02e0009dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02e7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d29c60 0x55c549d3dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02e7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d29c60 0x55c549d3dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57856/0 (socket says 172.21.15.28:57856) 2024-04-25T01:37:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02e7fff640 1 -- 172.21.15.28:0/2787412132 learned_addr learned my addr 172.21.15.28:0/2787412132 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02f48d7640 1 --2- 172.21.15.28:0/2787412132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d292b0 0x55c549d40ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02f50d8640 1 --2- 172.21.15.28:0/2787412132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d38980 0x55c549d3e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02e7fff640 1 -- 172.21.15.28:0/2787412132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d38980 msgr2=0x55c549d3e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02e7fff640 1 --2- 172.21.15.28:0/2787412132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d38980 0x55c549d3e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02e7fff640 1 -- 172.21.15.28:0/2787412132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d292b0 msgr2=0x55c549d40ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02e7fff640 1 --2- 172.21.15.28:0/2787412132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d292b0 0x55c549d40ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02e7fff640 1 -- 172.21.15.28:0/2787412132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c549a6c7d0 con 0x55c549d29c60 2024-04-25T01:37:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02f48d7640 1 --2- 172.21.15.28:0/2787412132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d292b0 0x55c549d40ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:37:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02e7fff640 1 --2- 172.21.15.28:0/2787412132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d29c60 0x55c549d3dac0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f02e000f8e0 tx=0x7f02e000fd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da912a4cf69f11f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.763+0000 7f02cffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.764+0000 7f02cffff640 1 -- 172.21.15.28:0/2787412132 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e000f650 con 0x55c549d29c60 2024-04-25T01:37:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.764+0000 7f02f984a740 1 -- 172.21.15.28:0/2787412132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02ec005290 con 0x55c549d29c60 2024-04-25T01:37:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.764+0000 7f02cffff640 1 -- 172.21.15.28:0/2787412132 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f02e0011b40 con 0x55c549d29c60 2024-04-25T01:37:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.764+0000 7f02cffff640 1 -- 172.21.15.28:0/2787412132 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f02e0037050 con 0x55c549d29c60 2024-04-25T01:37:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.764+0000 7f02f984a740 10 client.17094 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.764+0000 7f02f984a740 1 -- 172.21.15.28:0/2787412132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c549dafb10 con 0x55c549d29c60 2024-04-25T01:37:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.764+0000 7f02cffff640 1 -- 172.21.15.28:0/2787412132 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02ec005290 con 0x55c549d29c60 2024-04-25T01:37:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.764+0000 7f02cffff640 1 -- 172.21.15.28:0/2787412132 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f02e00046b0 con 0x55c549d29c60 2024-04-25T01:37:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.764+0000 7f02f984a740 10 client.17094 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.764+0000 7f02f984a740 10 client.17094 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.764+0000 7f02f984a740 20 client.17094 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.764+0000 7f02f984a740 1 --2- 172.21.15.28:0/2787412132 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c549db02e0 0x55c549db2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.764+0000 7f02f984a740 4 client.17094 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:37:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.764+0000 7f02f984a740 1 -- 172.21.15.28:0/2787412132 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c549c95f00 con 0x55c549db02e0 2024-04-25T01:37:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.764+0000 7f02f48d7640 1 --2- 172.21.15.28:0/2787412132 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c549db02e0 0x55c549db2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.765+0000 7f02cffff640 10 client.17094 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.765+0000 7f02f48d7640 1 --2- 172.21.15.28:0/2787412132 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c549db02e0 0x55c549db2700 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9278eba237c310f server_cookie=ed2eff329c306186 in_seq=0 out_seq=0 2024-04-25T01:37:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.765+0000 7f02cffff640 1 -- 172.21.15.28:0/2787412132 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c549c95f00 con 0x55c549db02e0 2024-04-25T01:37:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.765+0000 7f02cffff640 10 client.17094 handle_command_reply: tid=0 2024-04-25T01:37:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.765+0000 7f02f984a740 1 client.17094 shutdown 2024-04-25T01:37:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.765+0000 7f02f984a740 1 -- 172.21.15.28:0/2787412132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d29c60 msgr2=0x55c549d3dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.765+0000 7f02f984a740 1 --2- 172.21.15.28:0/2787412132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d29c60 0x55c549d3dac0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f02e000f8e0 tx=0x7f02e000fd80 comp rx=0 tx=0).stop 2024-04-25T01:37:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.765+0000 7f02f984a740 1 -- 172.21.15.28:0/2787412132 shutdown_connections 2024-04-25T01:37:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.765+0000 7f02f984a740 1 --2- 172.21.15.28:0/2787412132 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c549db02e0 0x55c549db2700 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.765+0000 7f02f984a740 1 --2- 172.21.15.28:0/2787412132 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c549d38980 0x55c549d3e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.765+0000 7f02f984a740 1 --2- 172.21.15.28:0/2787412132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c549d29c60 0x55c549d3dac0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.765+0000 7f02cffff640 0 client.17094 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.765+0000 7f02f984a740 1 --2- 172.21.15.28:0/2787412132 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c549d292b0 0x55c549d40ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.765+0000 7f02f984a740 1 -- 172.21.15.28:0/2787412132 >> 172.21.15.28:0/2787412132 conn(0x55c549d258a0 msgr2=0x55c549d242b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.765+0000 7f02f984a740 1 -- 172.21.15.28:0/2787412132 shutdown_connections 2024-04-25T01:37:31.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.765+0000 7f02f984a740 1 -- 172.21.15.28:0/2787412132 wait complete. 2024-04-25T01:37:31.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.765+0000 7f02f984a740 20 client.17094 trim_cache size 0 max 16384 2024-04-25T01:37:31.749 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:37:31.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.766+0000 7f02f6b3b640 1 -- 172.21.15.28:0/300806337 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f02d80a2d80 msgr2=0x7f02d80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.766+0000 7f02f6b3b640 1 --2- 172.21.15.28:0/300806337 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f02d80a2d80 0x7f02d80a5240 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f02e8031860 tx=0x7f02e8009910 comp rx=0 tx=0).stop 2024-04-25T01:37:31.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.766+0000 7f02f6b3b640 1 -- 172.21.15.28:0/300806337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02f0100620 msgr2=0x7f02f0198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:31.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.766+0000 7f02f6b3b640 1 --2- 172.21.15.28:0/300806337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02f0100620 0x7f02f0198010 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00d770 tx=0x7f02ec00dc30 comp rx=0 tx=0).stop 2024-04-25T01:37:31.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.766+0000 7f02f6b3b640 1 -- 172.21.15.28:0/300806337 shutdown_connections 2024-04-25T01:37:31.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.766+0000 7f02f6b3b640 1 --2- 172.21.15.28:0/300806337 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f02d80a2d80 0x7f02d80a5240 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.766+0000 7f02f6b3b640 1 --2- 172.21.15.28:0/300806337 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f02f0106630 0x7f02f019c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.766+0000 7f02f6b3b640 1 --2- 172.21.15.28:0/300806337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02f0100fe0 0x7f02f0198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.766+0000 7f02f6b3b640 1 --2- 172.21.15.28:0/300806337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02f0100620 0x7f02f0198010 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:31.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.766+0000 7f02f6b3b640 1 -- 172.21.15.28:0/300806337 >> 172.21.15.28:0/300806337 conn(0x7f02f00776a0 msgr2=0x7f02f006a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:31.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.766+0000 7f02f6b3b640 1 -- 172.21.15.28:0/300806337 shutdown_connections 2024-04-25T01:37:31.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:31.766+0000 7f02f6b3b640 1 -- 172.21.15.28:0/300806337 wait complete. 2024-04-25T01:37:32.011 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:31 smithi130 ceph-mon[32702]: pgmap v138: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:37:32.011 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:31 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3150519237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:37:32.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:31 smithi028 ceph-mon[29390]: pgmap v138: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:37:32.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:31 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3150519237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:37:32.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:31 smithi032 ceph-mon[32461]: pgmap v138: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:37:32.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:31 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3150519237' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:37:32.755 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:37:32.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:37:32.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:37:32.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.896+0000 7f4e5dea5640 1 -- 172.21.15.28:0/1375040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e58107720 msgr2=0x7f4e58107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.896+0000 7f4e5dea5640 1 --2- 172.21.15.28:0/1375040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e58107720 0x7f4e58107b00 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f4e48009930 tx=0x7f4e4802a420 comp rx=0 tx=0).stop 2024-04-25T01:37:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.896+0000 7f4e5dea5640 1 -- 172.21.15.28:0/1375040 shutdown_connections 2024-04-25T01:37:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.896+0000 7f4e5dea5640 1 --2- 172.21.15.28:0/1375040 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e581020d0 0x7f4e5810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.896+0000 7f4e5dea5640 1 --2- 172.21.15.28:0/1375040 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e58101710 0x7f4e58101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.896+0000 7f4e5dea5640 1 --2- 172.21.15.28:0/1375040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e58107720 0x7f4e58107b00 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.896+0000 7f4e5dea5640 1 -- 172.21.15.28:0/1375040 >> 172.21.15.28:0/1375040 conn(0x7f4e58078b90 msgr2=0x7f4e58078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.896+0000 7f4e5dea5640 1 -- 172.21.15.28:0/1375040 shutdown_connections 2024-04-25T01:37:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.896+0000 7f4e5dea5640 1 -- 172.21.15.28:0/1375040 wait complete. 2024-04-25T01:37:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.897+0000 7f4e5dea5640 1 Processor -- start 2024-04-25T01:37:32.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.897+0000 7f4e5dea5640 1 -- start start 2024-04-25T01:37:32.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.897+0000 7f4e5dea5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e58101710 0x7f4e58073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:32.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.897+0000 7f4e5dea5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e581020d0 0x7f4e58073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:32.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.897+0000 7f4e5dea5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e58107720 0x7f4e580744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:32.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.897+0000 7f4e5dea5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4e58111b80 con 0x7f4e58101710 2024-04-25T01:37:32.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.897+0000 7f4e5dea5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4e58111a00 con 0x7f4e58107720 2024-04-25T01:37:32.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.897+0000 7f4e5dea5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4e58111d00 con 0x7f4e581020d0 2024-04-25T01:37:32.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.897+0000 7f4e577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e58101710 0x7f4e58073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:32.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.897+0000 7f4e577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e58101710 0x7f4e58073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57876/0 (socket says 172.21.15.28:57876) 2024-04-25T01:37:32.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.897+0000 7f4e577fe640 1 -- 172.21.15.28:0/3836749559 learned_addr learned my addr 172.21.15.28:0/3836749559 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:32.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.897+0000 7f4e56ffd640 1 --2- 172.21.15.28:0/3836749559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e581020d0 0x7f4e58073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:32.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.898+0000 7f4e57fff640 1 --2- 172.21.15.28:0/3836749559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e58107720 0x7f4e580744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:32.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.898+0000 7f4e577fe640 1 -- 172.21.15.28:0/3836749559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e581020d0 msgr2=0x7f4e58073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:32.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.898+0000 7f4e577fe640 1 --2- 172.21.15.28:0/3836749559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e581020d0 0x7f4e58073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.898+0000 7f4e577fe640 1 -- 172.21.15.28:0/3836749559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e58107720 msgr2=0x7f4e580744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:32.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.898+0000 7f4e577fe640 1 --2- 172.21.15.28:0/3836749559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e58107720 0x7f4e580744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.898+0000 7f4e577fe640 1 -- 172.21.15.28:0/3836749559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e581a65a0 con 0x7f4e58101710 2024-04-25T01:37:32.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.898+0000 7f4e57fff640 1 --2- 172.21.15.28:0/3836749559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e58107720 0x7f4e580744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:37:32.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.898+0000 7f4e577fe640 1 --2- 172.21.15.28:0/3836749559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e58101710 0x7f4e58073a40 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f4e4802a940 tx=0x7f4e48003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:32.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.898+0000 7f4e54ff9640 1 -- 172.21.15.28:0/3836749559 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4803f070 con 0x7f4e58101710 2024-04-25T01:37:32.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.898+0000 7f4e54ff9640 1 -- 172.21.15.28:0/3836749559 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e48031c90 con 0x7f4e58101710 2024-04-25T01:37:32.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.898+0000 7f4e5dea5640 1 -- 172.21.15.28:0/3836749559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e581a6830 con 0x7f4e58101710 2024-04-25T01:37:32.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.898+0000 7f4e54ff9640 1 -- 172.21.15.28:0/3836749559 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e48035030 con 0x7f4e58101710 2024-04-25T01:37:32.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.898+0000 7f4e5dea5640 1 -- 172.21.15.28:0/3836749559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e581a6d20 con 0x7f4e58101710 2024-04-25T01:37:32.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.900+0000 7f4e54ff9640 1 -- 172.21.15.28:0/3836749559 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f4e48042070 con 0x7f4e58101710 2024-04-25T01:37:32.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.901+0000 7f4e60bb3740 1 Processor -- start 2024-04-25T01:37:32.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.901+0000 7f4e60bb3740 1 -- start start 2024-04-25T01:37:32.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.901+0000 7f4e54ff9640 1 --2- 172.21.15.28:0/3836749559 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4e400a2d00 0x7f4e400a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:32.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.901+0000 7f4e60bb3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86e0980 0x55c8e86decc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:32.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.901+0000 7f4e60bb3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86df290 0x55c8e86df710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:32.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.901+0000 7f4e60bb3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86ea010 0x55c8e86ec400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:32.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.901+0000 7f4e60bb3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c8e86432c0 con 0x55c8e86ea010 2024-04-25T01:37:32.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.901+0000 7f4e60bb3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c8e85eb910 con 0x55c8e86df290 2024-04-25T01:37:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.901+0000 7f4e60bb3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c8e863f280 con 0x55c8e86e0980 2024-04-25T01:37:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.901+0000 7f4e57fff640 1 --2- 172.21.15.28:0/3836749559 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4e400a2d00 0x7f4e400a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.901+0000 7f4e54ff9640 1 -- 172.21.15.28:0/3836749559 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f4e480effb0 con 0x7f4e58101710 2024-04-25T01:37:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.901+0000 7f4e57fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86ea010 0x55c8e86ec400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.901+0000 7f4e57fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86ea010 0x55c8e86ec400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57892/0 (socket says 172.21.15.28:57892) 2024-04-25T01:37:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.901+0000 7f4e57fff640 1 -- 172.21.15.28:0/1041487920 learned_addr learned my addr 172.21.15.28:0/1041487920 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.901+0000 7f4e56ffd640 1 --2- 172.21.15.28:0/1041487920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86df290 0x55c8e86df710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.901+0000 7f4e577fe640 1 --2- 172.21.15.28:0/1041487920 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86e0980 0x55c8e86decc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.902+0000 7f4e57fff640 1 -- 172.21.15.28:0/1041487920 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86e0980 msgr2=0x55c8e86decc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.902+0000 7f4e57fff640 1 --2- 172.21.15.28:0/1041487920 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86e0980 0x55c8e86decc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.902+0000 7f4e57fff640 1 -- 172.21.15.28:0/1041487920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86df290 msgr2=0x55c8e86df710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.902+0000 7f4e57fff640 1 --2- 172.21.15.28:0/1041487920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86df290 0x55c8e86df710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.902+0000 7f4e57fff640 1 -- 172.21.15.28:0/1041487920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8e85fcfb0 con 0x55c8e86ea010 2024-04-25T01:37:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.902+0000 7f4e56ffd640 1 --2- 172.21.15.28:0/1041487920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86df290 0x55c8e86df710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:37:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.902+0000 7f4e577fe640 1 --2- 172.21.15.28:0/1041487920 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86e0980 0x55c8e86decc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:37:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.902+0000 7f4e57fff640 1 --2- 172.21.15.28:0/3836749559 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4e400a2d00 0x7f4e400a51c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4e50012be0 tx=0x7f4e50009db0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.902+0000 7f4e57fff640 1 --2- 172.21.15.28:0/1041487920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86ea010 0x55c8e86ec400 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f4e50005b30 tx=0x7f4e5001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f463d521bb06a16 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.902+0000 7f4e3dffb640 1 -- 172.21.15.28:0/1041487920 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5000c7a0 con 0x55c8e86ea010 2024-04-25T01:37:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.902+0000 7f4e3dffb640 1 -- 172.21.15.28:0/1041487920 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e50025070 con 0x55c8e86ea010 2024-04-25T01:37:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.902+0000 7f4e3dffb640 1 -- 172.21.15.28:0/1041487920 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e50020d00 con 0x55c8e86ea010 2024-04-25T01:37:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.903+0000 7f4e60bb3740 1 -- 172.21.15.28:0/1041487920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86ea010 msgr2=0x55c8e86ec400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.903+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/1041487920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86ea010 0x55c8e86ec400 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f4e50005b30 tx=0x7f4e5001c000 comp rx=0 tx=0).stop 2024-04-25T01:37:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.903+0000 7f4e60bb3740 1 -- 172.21.15.28:0/1041487920 shutdown_connections 2024-04-25T01:37:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.903+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/1041487920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86ea010 0x55c8e86ec400 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.903+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/1041487920 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86df290 0x55c8e86df710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.903+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/1041487920 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86e0980 0x55c8e86decc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.903+0000 7f4e60bb3740 1 -- 172.21.15.28:0/1041487920 >> 172.21.15.28:0/1041487920 conn(0x55c8e86a7950 msgr2=0x55c8e86db7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.903+0000 7f4e60bb3740 1 -- 172.21.15.28:0/1041487920 shutdown_connections 2024-04-25T01:37:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.903+0000 7f4e60bb3740 1 -- 172.21.15.28:0/1041487920 wait complete. 2024-04-25T01:37:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.903+0000 7f4e60bb3740 1 -- start start 2024-04-25T01:37:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e60bb3740 1 Processor -- start 2024-04-25T01:37:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e60bb3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86df290 0x55c8e86e6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e60bb3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86e0980 0x55c8e86e7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e60bb3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86ea010 0x55c8e86e79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e60bb3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c8e86ece40 con 0x55c8e86e0980 2024-04-25T01:37:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e60bb3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c8e85fee90 con 0x55c8e86df290 2024-04-25T01:37:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e60bb3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c8e86ecfc0 con 0x55c8e86ea010 2024-04-25T01:37:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e60bb3740 10 client.0 _ll_register_callbacks cb 0x55c8e863a970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e56ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86e0980 0x55c8e86e7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e56ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86e0980 0x55c8e86e7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57908/0 (socket says 172.21.15.28:57908) 2024-04-25T01:37:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e56ffd640 1 -- 172.21.15.28:0/3173562035 learned_addr learned my addr 172.21.15.28:0/3173562035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e577fe640 1 --2- 172.21.15.28:0/3173562035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86df290 0x55c8e86e6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e57fff640 1 --2- 172.21.15.28:0/3173562035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86ea010 0x55c8e86e79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e56ffd640 1 -- 172.21.15.28:0/3173562035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86ea010 msgr2=0x55c8e86e79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e56ffd640 1 --2- 172.21.15.28:0/3173562035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86ea010 0x55c8e86e79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e56ffd640 1 -- 172.21.15.28:0/3173562035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86df290 msgr2=0x55c8e86e6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e56ffd640 1 --2- 172.21.15.28:0/3173562035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86df290 0x55c8e86e6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e56ffd640 1 -- 172.21.15.28:0/3173562035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8e841f7e0 con 0x55c8e86e0980 2024-04-25T01:37:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e577fe640 1 --2- 172.21.15.28:0/3173562035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86df290 0x55c8e86e6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.904+0000 7f4e57fff640 1 --2- 172.21.15.28:0/3173562035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86ea010 0x55c8e86e79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.905+0000 7f4e56ffd640 1 --2- 172.21.15.28:0/3173562035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86e0980 0x55c8e86e7480 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c00ba60 tx=0x7f4e4c00bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d058cb57ab29e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.905+0000 7f4e36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.905+0000 7f4e36ffd640 1 -- 172.21.15.28:0/3173562035 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c011440 con 0x55c8e86e0980 2024-04-25T01:37:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.905+0000 7f4e36ffd640 1 -- 172.21.15.28:0/3173562035 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f4e4c0115e0 con 0x55c8e86e0980 2024-04-25T01:37:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.905+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3173562035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e50025070 con 0x55c8e86e0980 2024-04-25T01:37:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.905+0000 7f4e36ffd640 1 -- 172.21.15.28:0/3173562035 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4e4c005020 con 0x55c8e86e0980 2024-04-25T01:37:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.905+0000 7f4e60bb3740 10 client.17106 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.905+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3173562035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c8e847ff70 con 0x55c8e86e0980 2024-04-25T01:37:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.905+0000 7f4e36ffd640 1 -- 172.21.15.28:0/3173562035 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e50025070 con 0x55c8e86e0980 2024-04-25T01:37:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.905+0000 7f4e36ffd640 1 -- 172.21.15.28:0/3173562035 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4e4c0047a0 con 0x55c8e86e0980 2024-04-25T01:37:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.905+0000 7f4e60bb3740 10 client.17106 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.905+0000 7f4e60bb3740 10 client.17106 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.905+0000 7f4e60bb3740 20 client.17106 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.905+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/3173562035 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c8e86fa200 0x55c8e86fc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.905+0000 7f4e60bb3740 4 client.17106 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:37:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.905+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3173562035 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8e86175a0 con 0x55c8e86fa200 2024-04-25T01:37:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.906+0000 7f4e577fe640 1 --2- 172.21.15.28:0/3173562035 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c8e86fa200 0x55c8e86fc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.906+0000 7f4e36ffd640 10 client.17106 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.906+0000 7f4e577fe640 1 --2- 172.21.15.28:0/3173562035 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c8e86fa200 0x55c8e86fc660 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f51e88857ad6f83 server_cookie=c48f8f5747dc4e42 in_seq=0 out_seq=0 2024-04-25T01:37:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.907+0000 7f4e36ffd640 1 -- 172.21.15.28:0/3173562035 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c8e86175a0 con 0x55c8e86fa200 2024-04-25T01:37:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.907+0000 7f4e36ffd640 10 client.17106 handle_command_reply: tid=0 2024-04-25T01:37:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.907+0000 7f4e60bb3740 1 client.17106 shutdown 2024-04-25T01:37:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.907+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3173562035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86e0980 msgr2=0x55c8e86e7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.907+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/3173562035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86e0980 0x55c8e86e7480 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c00ba60 tx=0x7f4e4c00bf20 comp rx=0 tx=0).stop 2024-04-25T01:37:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.907+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3173562035 shutdown_connections 2024-04-25T01:37:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.907+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/3173562035 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c8e86fa200 0x55c8e86fc660 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.907+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/3173562035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86ea010 0x55c8e86e79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.907+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/3173562035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86e0980 0x55c8e86e7480 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.907+0000 7f4e36ffd640 0 client.17106 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.907+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/3173562035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86df290 0x55c8e86e6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.907+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3173562035 >> 172.21.15.28:0/3173562035 conn(0x55c8e86a7950 msgr2=0x55c8e86e1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.907+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3173562035 shutdown_connections 2024-04-25T01:37:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.907+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3173562035 wait complete. 2024-04-25T01:37:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.907+0000 7f4e60bb3740 20 client.17106 trim_cache size 0 max 16384 2024-04-25T01:37:32.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.925+0000 7f4e60bb3740 1 Processor -- start 2024-04-25T01:37:32.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.925+0000 7f4e60bb3740 1 -- start start 2024-04-25T01:37:32.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.925+0000 7f4e60bb3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86df290 0x55c8e86f5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:32.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e60bb3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86e0980 0x55c8e86da7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:32.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e60bb3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86ea010 0x55c8e8701f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:32.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e60bb3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4e581b0ca0 con 0x55c8e86e0980 2024-04-25T01:37:32.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e60bb3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4e581b27e0 con 0x55c8e86df290 2024-04-25T01:37:32.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e60bb3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4e581b1370 con 0x55c8e86ea010 2024-04-25T01:37:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e56ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86e0980 0x55c8e86da7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e56ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86e0980 0x55c8e86da7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57918/0 (socket says 172.21.15.28:57918) 2024-04-25T01:37:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e56ffd640 1 -- 172.21.15.28:0/3067612502 learned_addr learned my addr 172.21.15.28:0/3067612502 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e57fff640 1 --2- 172.21.15.28:0/3067612502 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86ea010 0x55c8e8701f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e577fe640 1 --2- 172.21.15.28:0/3067612502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86df290 0x55c8e86f5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e56ffd640 1 -- 172.21.15.28:0/3067612502 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86ea010 msgr2=0x55c8e8701f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e56ffd640 1 --2- 172.21.15.28:0/3067612502 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86ea010 0x55c8e8701f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e56ffd640 1 -- 172.21.15.28:0/3067612502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86df290 msgr2=0x55c8e86f5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e56ffd640 1 --2- 172.21.15.28:0/3067612502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86df290 0x55c8e86f5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e56ffd640 1 -- 172.21.15.28:0/3067612502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8e8317170 con 0x55c8e86e0980 2024-04-25T01:37:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e577fe640 1 --2- 172.21.15.28:0/3067612502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86df290 0x55c8e86f5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e56ffd640 1 --2- 172.21.15.28:0/3067612502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86e0980 0x55c8e86da7d0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c001eb0 tx=0x7f4e4c003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad90ca5443663781 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e3d7fa640 1 -- 172.21.15.28:0/3067612502 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c000e60 con 0x55c8e86e0980 2024-04-25T01:37:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e3d7fa640 1 -- 172.21.15.28:0/3067612502 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e4c0051c0 con 0x55c8e86e0980 2024-04-25T01:37:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.926+0000 7f4e3d7fa640 1 -- 172.21.15.28:0/3067612502 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4c01b850 con 0x55c8e86e0980 2024-04-25T01:37:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.927+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3067612502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86e0980 msgr2=0x55c8e86da7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.927+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/3067612502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86e0980 0x55c8e86da7d0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c001eb0 tx=0x7f4e4c003f60 comp rx=0 tx=0).stop 2024-04-25T01:37:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.927+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3067612502 shutdown_connections 2024-04-25T01:37:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.927+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/3067612502 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86ea010 0x55c8e8701f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.927+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/3067612502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86e0980 0x55c8e86da7d0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.927+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/3067612502 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86df290 0x55c8e86f5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.927+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3067612502 >> 172.21.15.28:0/3067612502 conn(0x55c8e86a7950 msgr2=0x55c8e86da0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.927+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3067612502 shutdown_connections 2024-04-25T01:37:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.927+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3067612502 wait complete. 2024-04-25T01:37:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.927+0000 7f4e60bb3740 1 -- start start 2024-04-25T01:37:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e60bb3740 1 Processor -- start 2024-04-25T01:37:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e60bb3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86df290 0x55c8e86e42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e60bb3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86e0980 0x55c8e86e47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e60bb3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86ea010 0x55c8e875f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e60bb3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4e581b2110 con 0x55c8e86df290 2024-04-25T01:37:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e60bb3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4e581b1a40 con 0x55c8e86e0980 2024-04-25T01:37:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e60bb3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4e581aec80 con 0x55c8e86ea010 2024-04-25T01:37:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e60bb3740 10 client.0 _ll_register_callbacks cb 0x7f4e4c0042b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86df290 0x55c8e86e42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86df290 0x55c8e86e42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57926/0 (socket says 172.21.15.28:57926) 2024-04-25T01:37:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e577fe640 1 -- 172.21.15.28:0/3503755865 learned_addr learned my addr 172.21.15.28:0/3503755865 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e56ffd640 1 --2- 172.21.15.28:0/3503755865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86e0980 0x55c8e86e47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e57fff640 1 --2- 172.21.15.28:0/3503755865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86ea010 0x55c8e875f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e577fe640 1 -- 172.21.15.28:0/3503755865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86ea010 msgr2=0x55c8e875f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e577fe640 1 --2- 172.21.15.28:0/3503755865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86ea010 0x55c8e875f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e577fe640 1 -- 172.21.15.28:0/3503755865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86e0980 msgr2=0x55c8e86e47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e577fe640 1 --2- 172.21.15.28:0/3503755865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86e0980 0x55c8e86e47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e577fe640 1 -- 172.21.15.28:0/3503755865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8e875fce0 con 0x55c8e86df290 2024-04-25T01:37:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e57fff640 1 --2- 172.21.15.28:0/3503755865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86ea010 0x55c8e875f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:37:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e577fe640 1 --2- 172.21.15.28:0/3503755865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86df290 0x55c8e86e42b0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f4e480b8590 tx=0x7f4e48009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aae7d0a806de826 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.928+0000 7f4e36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.929+0000 7f4e36ffd640 1 -- 172.21.15.28:0/3503755865 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e48049040 con 0x55c8e86df290 2024-04-25T01:37:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.929+0000 7f4e36ffd640 1 -- 172.21.15.28:0/3503755865 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f4e480ba430 con 0x55c8e86df290 2024-04-25T01:37:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.929+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3503755865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e4c0051c0 con 0x55c8e86df290 2024-04-25T01:37:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.929+0000 7f4e36ffd640 1 -- 172.21.15.28:0/3503755865 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4e480bc830 con 0x55c8e86df290 2024-04-25T01:37:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.929+0000 7f4e60bb3740 10 client.17112 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.929+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3503755865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c8e8760f00 con 0x55c8e86df290 2024-04-25T01:37:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.929+0000 7f4e36ffd640 1 -- 172.21.15.28:0/3503755865 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e4c0051c0 con 0x55c8e86df290 2024-04-25T01:37:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.929+0000 7f4e36ffd640 1 -- 172.21.15.28:0/3503755865 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4e480b87a0 con 0x55c8e86df290 2024-04-25T01:37:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.929+0000 7f4e60bb3740 10 client.17112 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.929+0000 7f4e60bb3740 10 client.17112 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.929+0000 7f4e60bb3740 20 client.17112 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.929+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/3503755865 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c8e87615d0 0x55c8e87639f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.929+0000 7f4e60bb3740 4 client.17112 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:37:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.929+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3503755865 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c8e8650800 con 0x55c8e87615d0 2024-04-25T01:37:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.929+0000 7f4e56ffd640 1 --2- 172.21.15.28:0/3503755865 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c8e87615d0 0x55c8e87639f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.930+0000 7f4e56ffd640 1 --2- 172.21.15.28:0/3503755865 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c8e87615d0 0x55c8e87639f0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a97dfc71769bc85b server_cookie=214c0f2e0bc192c0 in_seq=0 out_seq=0 2024-04-25T01:37:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.930+0000 7f4e36ffd640 10 client.17112 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:32.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.931+0000 7f4e36ffd640 1 -- 172.21.15.28:0/3503755865 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c8e8650800 con 0x55c8e87615d0 2024-04-25T01:37:32.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.931+0000 7f4e36ffd640 10 client.17112 handle_command_reply: tid=0 2024-04-25T01:37:32.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.931+0000 7f4e60bb3740 1 client.17112 shutdown 2024-04-25T01:37:32.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.931+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3503755865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86df290 msgr2=0x55c8e86e42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:32.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.931+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/3503755865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86df290 0x55c8e86e42b0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f4e480b8590 tx=0x7f4e48009260 comp rx=0 tx=0).stop 2024-04-25T01:37:32.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.931+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3503755865 shutdown_connections 2024-04-25T01:37:32.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.931+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/3503755865 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c8e87615d0 0x55c8e87639f0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.931+0000 7f4e36ffd640 0 client.17112 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:32.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.931+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/3503755865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c8e86ea010 0x55c8e875f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.931+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/3503755865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c8e86e0980 0x55c8e86e47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.931+0000 7f4e60bb3740 1 --2- 172.21.15.28:0/3503755865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8e86df290 0x55c8e86e42b0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.931+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3503755865 >> 172.21.15.28:0/3503755865 conn(0x55c8e86a7950 msgr2=0x55c8e86da950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:32.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.931+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3503755865 shutdown_connections 2024-04-25T01:37:32.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.931+0000 7f4e60bb3740 1 -- 172.21.15.28:0/3503755865 wait complete. 2024-04-25T01:37:32.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.932+0000 7f4e60bb3740 20 client.17112 trim_cache size 0 max 16384 2024-04-25T01:37:32.915 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:37:32.915 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:37:32.915 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "12bcd52d-dead-40c0-a880-fe68faa01255", 2024-04-25T01:37:32.915 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:37:32.915 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:37:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.932+0000 7f4e5dea5640 1 -- 172.21.15.28:0/3836749559 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4e400a2d00 msgr2=0x7f4e400a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.932+0000 7f4e5dea5640 1 --2- 172.21.15.28:0/3836749559 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4e400a2d00 0x7f4e400a51c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4e50012be0 tx=0x7f4e50009db0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.932+0000 7f4e5dea5640 1 -- 172.21.15.28:0/3836749559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e58101710 msgr2=0x7f4e58073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.932+0000 7f4e5dea5640 1 --2- 172.21.15.28:0/3836749559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e58101710 0x7f4e58073a40 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f4e4802a940 tx=0x7f4e48003b90 comp rx=0 tx=0).stop 2024-04-25T01:37:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.932+0000 7f4e5dea5640 1 -- 172.21.15.28:0/3836749559 shutdown_connections 2024-04-25T01:37:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.932+0000 7f4e5dea5640 1 --2- 172.21.15.28:0/3836749559 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4e400a2d00 0x7f4e400a51c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.932+0000 7f4e5dea5640 1 --2- 172.21.15.28:0/3836749559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e58107720 0x7f4e580744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.932+0000 7f4e5dea5640 1 --2- 172.21.15.28:0/3836749559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4e581020d0 0x7f4e58073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.932+0000 7f4e5dea5640 1 --2- 172.21.15.28:0/3836749559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e58101710 0x7f4e58073a40 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.932+0000 7f4e5dea5640 1 -- 172.21.15.28:0/3836749559 >> 172.21.15.28:0/3836749559 conn(0x7f4e58078b90 msgr2=0x7f4e580ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.932+0000 7f4e5dea5640 1 -- 172.21.15.28:0/3836749559 shutdown_connections 2024-04-25T01:37:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:32.932+0000 7f4e5dea5640 1 -- 172.21.15.28:0/3836749559 wait complete. 2024-04-25T01:37:32.921 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:37:32.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:37:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.048+0000 7fc2df716640 1 -- 172.21.15.28:0/2081944349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d8100fe0 msgr2=0x7fc2d810d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.048+0000 7fc2df716640 1 --2- 172.21.15.28:0/2081944349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d8100fe0 0x7fc2d810d4f0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0009cd0 tx=0x7fc2d002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.048+0000 7fc2df716640 1 -- 172.21.15.28:0/2081944349 shutdown_connections 2024-04-25T01:37:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.048+0000 7fc2df716640 1 --2- 172.21.15.28:0/2081944349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d8100fe0 0x7fc2d810d4f0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.048+0000 7fc2df716640 1 --2- 172.21.15.28:0/2081944349 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc2d8100620 0x7fc2d8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.048+0000 7fc2df716640 1 --2- 172.21.15.28:0/2081944349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2d8106630 0x7fc2d8106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.048+0000 7fc2df716640 1 -- 172.21.15.28:0/2081944349 >> 172.21.15.28:0/2081944349 conn(0x7fc2d80776a0 msgr2=0x7fc2d8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.048+0000 7fc2df716640 1 -- 172.21.15.28:0/2081944349 shutdown_connections 2024-04-25T01:37:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.048+0000 7fc2df716640 1 -- 172.21.15.28:0/2081944349 wait complete. 2024-04-25T01:37:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.049+0000 7fc2df716640 1 Processor -- start 2024-04-25T01:37:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.049+0000 7fc2df716640 1 -- start start 2024-04-25T01:37:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.049+0000 7fc2df716640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d8100620 0x7fc2d8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.049+0000 7fc2df716640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc2d8100fe0 0x7fc2d8198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.049+0000 7fc2df716640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2d8106630 0x7fc2d819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.049+0000 7fc2df716640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d8110a40 con 0x7fc2d8100620 2024-04-25T01:37:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.049+0000 7fc2df716640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d81108c0 con 0x7fc2d8106630 2024-04-25T01:37:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.049+0000 7fc2df716640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d8110bc0 con 0x7fc2d8100fe0 2024-04-25T01:37:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.049+0000 7fc2dd4b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d8100620 0x7fc2d8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.049+0000 7fc2dd4b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d8100620 0x7fc2d8197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57948/0 (socket says 172.21.15.28:57948) 2024-04-25T01:37:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.049+0000 7fc2dd4b2640 1 -- 172.21.15.28:0/2377399831 learned_addr learned my addr 172.21.15.28:0/2377399831 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.049+0000 7fc2dd4b2640 1 -- 172.21.15.28:0/2377399831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc2d8100fe0 msgr2=0x7fc2d8198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.049+0000 7fc2dd4b2640 1 --2- 172.21.15.28:0/2377399831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc2d8100fe0 0x7fc2d8198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.049+0000 7fc2dd4b2640 1 -- 172.21.15.28:0/2377399831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2d8106630 msgr2=0x7fc2d819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.049+0000 7fc2dd4b2640 1 --2- 172.21.15.28:0/2377399831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2d8106630 0x7fc2d819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.049+0000 7fc2dd4b2640 1 -- 172.21.15.28:0/2377399831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d819cfc0 con 0x7fc2d8100620 2024-04-25T01:37:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.049+0000 7fc2dd4b2640 1 --2- 172.21.15.28:0/2377399831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d8100620 0x7fc2d8197fd0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400ea60 tx=0x7fc2d400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.050+0000 7fc2ce7fc640 1 -- 172.21.15.28:0/2377399831 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d400cde0 con 0x7fc2d8100620 2024-04-25T01:37:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.050+0000 7fc2ce7fc640 1 -- 172.21.15.28:0/2377399831 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2d40035f0 con 0x7fc2d8100620 2024-04-25T01:37:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.050+0000 7fc2ce7fc640 1 -- 172.21.15.28:0/2377399831 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4010670 con 0x7fc2d8100620 2024-04-25T01:37:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.050+0000 7fc2df716640 1 -- 172.21.15.28:0/2377399831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2d819d2b0 con 0x7fc2d8100620 2024-04-25T01:37:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.050+0000 7fc2df716640 1 -- 172.21.15.28:0/2377399831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2d81a4b40 con 0x7fc2d8100620 2024-04-25T01:37:33.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.052+0000 7fc2ce7fc640 1 -- 172.21.15.28:0/2377399831 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc2d40037a0 con 0x7fc2d8100620 2024-04-25T01:37:33.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.052+0000 7fc2e241b740 1 Processor -- start 2024-04-25T01:37:33.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.052+0000 7fc2e241b740 1 -- start start 2024-04-25T01:37:33.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.052+0000 7fc2e241b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac00630 0x561e1ac00a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.052+0000 7fc2e241b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00f50 0x561e1ac0b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.052+0000 7fc2e241b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac0b620 0x561e1ac0da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.052+0000 7fc2ce7fc640 1 --2- 172.21.15.28:0/2377399831 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc2c40a2d00 0x7fc2c40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.052+0000 7fc2e241b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561e1ab656b0 con 0x561e1ac0b620 2024-04-25T01:37:33.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.052+0000 7fc2e241b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561e1ab0d880 con 0x561e1ac00630 2024-04-25T01:37:33.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.052+0000 7fc2e241b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x561e1ab61670 con 0x561e1ac00f50 2024-04-25T01:37:33.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.053+0000 7fc2ddcb3640 1 --2- 172.21.15.28:0/2377399831 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc2c40a2d00 0x7fc2c40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.053+0000 7fc2dccb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac0b620 0x561e1ac0da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.053+0000 7fc2dccb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac0b620 0x561e1ac0da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57962/0 (socket says 172.21.15.28:57962) 2024-04-25T01:37:33.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.053+0000 7fc2dccb1640 1 -- 172.21.15.28:0/778059456 learned_addr learned my addr 172.21.15.28:0/778059456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:33.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.053+0000 7fc2ce7fc640 1 -- 172.21.15.28:0/2377399831 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc2d40cfb50 con 0x7fc2d8100620 2024-04-25T01:37:33.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.053+0000 7fc2dd4b2640 1 --2- 172.21.15.28:0/778059456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00f50 0x561e1ac0b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.053+0000 7fc2ddcb3640 1 --2- 172.21.15.28:0/778059456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac00630 0x561e1ac00a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.053+0000 7fc2dccb1640 1 -- 172.21.15.28:0/778059456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00f50 msgr2=0x561e1ac0b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:33.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.053+0000 7fc2dccb1640 1 --2- 172.21.15.28:0/778059456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00f50 0x561e1ac0b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.053+0000 7fc2dccb1640 1 -- 172.21.15.28:0/778059456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac00630 msgr2=0x561e1ac00a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:33.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.053+0000 7fc2dccb1640 1 --2- 172.21.15.28:0/778059456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac00630 0x561e1ac00a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.053+0000 7fc2dccb1640 1 -- 172.21.15.28:0/778059456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e1a9437d0 con 0x561e1ac0b620 2024-04-25T01:37:33.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.053+0000 7fc2ddcb3640 1 --2- 172.21.15.28:0/778059456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac00630 0x561e1ac00a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:37:33.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.053+0000 7fc2dccb1640 1 --2- 172.21.15.28:0/778059456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac0b620 0x561e1ac0da10 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8009be0 tx=0x7fc2c800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=723bef4a30c7c6cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:33.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.053+0000 7fc2ddcb3640 1 --2- 172.21.15.28:0/2377399831 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc2c40a2d00 0x7fc2c40a51c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0009a60 tx=0x7fc2d00093a0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:33.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.053+0000 7fc2c37fe640 1 -- 172.21.15.28:0/778059456 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c8005020 con 0x561e1ac0b620 2024-04-25T01:37:33.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.053+0000 7fc2c37fe640 1 -- 172.21.15.28:0/778059456 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2c80047a0 con 0x561e1ac0b620 2024-04-25T01:37:33.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.053+0000 7fc2c37fe640 1 -- 172.21.15.28:0/778059456 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c8010670 con 0x561e1ac0b620 2024-04-25T01:37:33.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.054+0000 7fc2e241b740 1 -- 172.21.15.28:0/778059456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac0b620 msgr2=0x561e1ac0da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:33.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.054+0000 7fc2e241b740 1 --2- 172.21.15.28:0/778059456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac0b620 0x561e1ac0da10 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8009be0 tx=0x7fc2c800e720 comp rx=0 tx=0).stop 2024-04-25T01:37:33.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.054+0000 7fc2e241b740 1 -- 172.21.15.28:0/778059456 shutdown_connections 2024-04-25T01:37:33.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.054+0000 7fc2e241b740 1 --2- 172.21.15.28:0/778059456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac0b620 0x561e1ac0da10 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.054+0000 7fc2e241b740 1 --2- 172.21.15.28:0/778059456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00f50 0x561e1ac0b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.054+0000 7fc2e241b740 1 --2- 172.21.15.28:0/778059456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac00630 0x561e1ac00a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.054+0000 7fc2e241b740 1 -- 172.21.15.28:0/778059456 >> 172.21.15.28:0/778059456 conn(0x561e1abfc950 msgr2=0x561e1abfcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:33.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.054+0000 7fc2e241b740 1 -- 172.21.15.28:0/778059456 shutdown_connections 2024-04-25T01:37:33.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.054+0000 7fc2e241b740 1 -- 172.21.15.28:0/778059456 wait complete. 2024-04-25T01:37:33.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.054+0000 7fc2e241b740 1 -- start start 2024-04-25T01:37:33.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.055+0000 7fc2e241b740 1 Processor -- start 2024-04-25T01:37:33.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.055+0000 7fc2e241b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00630 0x561e1ac08570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.055+0000 7fc2e241b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac00f50 0x561e1ac08ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.055+0000 7fc2e241b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac0b620 0x561e1ac08ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.055+0000 7fc2e241b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561e1ac0e410 con 0x561e1ac00f50 2024-04-25T01:37:33.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.055+0000 7fc2e241b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561e1ab20f50 con 0x561e1ac0b620 2024-04-25T01:37:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.055+0000 7fc2e241b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x561e1ac0e590 con 0x561e1ac00630 2024-04-25T01:37:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.055+0000 7fc2e241b740 10 client.0 _ll_register_callbacks cb 0x561e1ab5c000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.055+0000 7fc2dccb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac00f50 0x561e1ac08ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.055+0000 7fc2dccb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac00f50 0x561e1ac08ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57974/0 (socket says 172.21.15.28:57974) 2024-04-25T01:37:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.055+0000 7fc2dccb1640 1 -- 172.21.15.28:0/867884797 learned_addr learned my addr 172.21.15.28:0/867884797 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.055+0000 7fc2ddcb3640 1 --2- 172.21.15.28:0/867884797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac0b620 0x561e1ac08ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.055+0000 7fc2dd4b2640 1 --2- 172.21.15.28:0/867884797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00630 0x561e1ac08570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.055+0000 7fc2dccb1640 1 -- 172.21.15.28:0/867884797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00630 msgr2=0x561e1ac08570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.055+0000 7fc2dccb1640 1 --2- 172.21.15.28:0/867884797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00630 0x561e1ac08570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.055+0000 7fc2dccb1640 1 -- 172.21.15.28:0/867884797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac0b620 msgr2=0x561e1ac08ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.055+0000 7fc2dccb1640 1 --2- 172.21.15.28:0/867884797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac0b620 0x561e1ac08ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.055+0000 7fc2dccb1640 1 -- 172.21.15.28:0/867884797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e1a8793f0 con 0x561e1ac00f50 2024-04-25T01:37:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.056+0000 7fc2dccb1640 1 --2- 172.21.15.28:0/867884797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac00f50 0x561e1ac08ab0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fc2c800ef90 tx=0x7fc2c8005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82dc4b4f8e39c541 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.056+0000 7fc2c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.056+0000 7fc2c0ff9640 1 -- 172.21.15.28:0/867884797 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c8026020 con 0x561e1ac00f50 2024-04-25T01:37:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.056+0000 7fc2c0ff9640 1 -- 172.21.15.28:0/867884797 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc2c8005220 con 0x561e1ac00f50 2024-04-25T01:37:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.056+0000 7fc2e241b740 1 -- 172.21.15.28:0/867884797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2c80047a0 con 0x561e1ac00f50 2024-04-25T01:37:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.056+0000 7fc2c0ff9640 1 -- 172.21.15.28:0/867884797 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc2c8031050 con 0x561e1ac00f50 2024-04-25T01:37:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.056+0000 7fc2e241b740 10 client.17124 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.056+0000 7fc2e241b740 1 -- 172.21.15.28:0/867884797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561e1a8390f0 con 0x561e1ac00f50 2024-04-25T01:37:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.056+0000 7fc2c0ff9640 1 -- 172.21.15.28:0/867884797 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2c80047a0 con 0x561e1ac00f50 2024-04-25T01:37:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.056+0000 7fc2c0ff9640 1 -- 172.21.15.28:0/867884797 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc2c802d4b0 con 0x561e1ac00f50 2024-04-25T01:37:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.056+0000 7fc2e241b740 10 client.17124 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.056+0000 7fc2e241b740 10 client.17124 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.056+0000 7fc2e241b740 20 client.17124 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.056+0000 7fc2e241b740 1 --2- 172.21.15.28:0/867884797 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x561e1ac1b860 0x561e1ac1dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.056+0000 7fc2e241b740 4 client.17124 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:37:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.056+0000 7fc2e241b740 1 -- 172.21.15.28:0/867884797 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e1ab72bf0 con 0x561e1ac1b860 2024-04-25T01:37:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.057+0000 7fc2dd4b2640 1 --2- 172.21.15.28:0/867884797 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x561e1ac1b860 0x561e1ac1dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.057+0000 7fc2dd4b2640 1 --2- 172.21.15.28:0/867884797 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x561e1ac1b860 0x561e1ac1dc50 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71391634f0df46b4 server_cookie=7014e47c16f73b0e in_seq=0 out_seq=0 2024-04-25T01:37:33.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.057+0000 7fc2c0ff9640 10 client.17124 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:33.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.058+0000 7fc2c0ff9640 1 -- 172.21.15.28:0/867884797 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561e1ab72bf0 con 0x561e1ac1b860 2024-04-25T01:37:33.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.058+0000 7fc2c0ff9640 10 client.17124 handle_command_reply: tid=0 2024-04-25T01:37:33.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.058+0000 7fc2e241b740 1 client.17124 shutdown 2024-04-25T01:37:33.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.058+0000 7fc2e241b740 1 -- 172.21.15.28:0/867884797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac00f50 msgr2=0x561e1ac08ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:33.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.058+0000 7fc2e241b740 1 --2- 172.21.15.28:0/867884797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac00f50 0x561e1ac08ab0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fc2c800ef90 tx=0x7fc2c8005d30 comp rx=0 tx=0).stop 2024-04-25T01:37:33.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.058+0000 7fc2e241b740 1 -- 172.21.15.28:0/867884797 shutdown_connections 2024-04-25T01:37:33.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.058+0000 7fc2e241b740 1 --2- 172.21.15.28:0/867884797 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x561e1ac1b860 0x561e1ac1dc50 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.058+0000 7fc2e241b740 1 --2- 172.21.15.28:0/867884797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac0b620 0x561e1ac08ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.058+0000 7fc2e241b740 1 --2- 172.21.15.28:0/867884797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac00f50 0x561e1ac08ab0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.058+0000 7fc2c0ff9640 0 client.17124 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:33.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.058+0000 7fc2e241b740 1 --2- 172.21.15.28:0/867884797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00630 0x561e1ac08570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.058+0000 7fc2e241b740 1 -- 172.21.15.28:0/867884797 >> 172.21.15.28:0/867884797 conn(0x561e1abfc950 msgr2=0x561e1ac02750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:33.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.058+0000 7fc2e241b740 1 -- 172.21.15.28:0/867884797 shutdown_connections 2024-04-25T01:37:33.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.058+0000 7fc2e241b740 1 -- 172.21.15.28:0/867884797 wait complete. 2024-04-25T01:37:33.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.058+0000 7fc2e241b740 20 client.17124 trim_cache size 0 max 16384 2024-04-25T01:37:33.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.076+0000 7fc2e241b740 1 Processor -- start 2024-04-25T01:37:33.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.076+0000 7fc2e241b740 1 -- start start 2024-04-25T01:37:33.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.076+0000 7fc2e241b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00630 0x561e1ac172d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.076+0000 7fc2e241b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac00f50 0x561e1abfc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.076+0000 7fc2e241b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac0b620 0x561e1abfcd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.076+0000 7fc2e241b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d81aeb10 con 0x561e1ac0b620 2024-04-25T01:37:33.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.076+0000 7fc2e241b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d81b0650 con 0x561e1ac00f50 2024-04-25T01:37:33.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.076+0000 7fc2e241b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d81af1e0 con 0x561e1ac00630 2024-04-25T01:37:33.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.076+0000 7fc2ddcb3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac0b620 0x561e1abfcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.076+0000 7fc2ddcb3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac0b620 0x561e1abfcd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57986/0 (socket says 172.21.15.28:57986) 2024-04-25T01:37:33.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.076+0000 7fc2ddcb3640 1 -- 172.21.15.28:0/186576605 learned_addr learned my addr 172.21.15.28:0/186576605 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:33.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.076+0000 7fc2dd4b2640 1 --2- 172.21.15.28:0/186576605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00630 0x561e1ac172d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.076+0000 7fc2dccb1640 1 --2- 172.21.15.28:0/186576605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac00f50 0x561e1abfc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.076+0000 7fc2ddcb3640 1 -- 172.21.15.28:0/186576605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00630 msgr2=0x561e1ac172d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:33.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.076+0000 7fc2ddcb3640 1 --2- 172.21.15.28:0/186576605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00630 0x561e1ac172d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.076+0000 7fc2ddcb3640 1 -- 172.21.15.28:0/186576605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac00f50 msgr2=0x561e1abfc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:33.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.076+0000 7fc2ddcb3640 1 --2- 172.21.15.28:0/186576605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac00f50 0x561e1abfc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.076+0000 7fc2ddcb3640 1 -- 172.21.15.28:0/186576605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e1ac0e2f0 con 0x561e1ac0b620 2024-04-25T01:37:33.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.076+0000 7fc2dccb1640 1 --2- 172.21.15.28:0/186576605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac00f50 0x561e1abfc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:37:33.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.077+0000 7fc2ddcb3640 1 --2- 172.21.15.28:0/186576605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac0b620 0x561e1abfcd30 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0031610 tx=0x7fc2d003c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5628080d63b29b60 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:33.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.077+0000 7fc2c2ffd640 1 -- 172.21.15.28:0/186576605 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d0030b20 con 0x561e1ac0b620 2024-04-25T01:37:33.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.077+0000 7fc2c2ffd640 1 -- 172.21.15.28:0/186576605 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2d0036050 con 0x561e1ac0b620 2024-04-25T01:37:33.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.077+0000 7fc2c2ffd640 1 -- 172.21.15.28:0/186576605 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d0048df0 con 0x561e1ac0b620 2024-04-25T01:37:33.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.077+0000 7fc2e241b740 1 -- 172.21.15.28:0/186576605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac0b620 msgr2=0x561e1abfcd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:33.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.077+0000 7fc2e241b740 1 --2- 172.21.15.28:0/186576605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac0b620 0x561e1abfcd30 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0031610 tx=0x7fc2d003c000 comp rx=0 tx=0).stop 2024-04-25T01:37:33.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.077+0000 7fc2e241b740 1 -- 172.21.15.28:0/186576605 shutdown_connections 2024-04-25T01:37:33.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.077+0000 7fc2e241b740 1 --2- 172.21.15.28:0/186576605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac0b620 0x561e1abfcd30 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.077+0000 7fc2e241b740 1 --2- 172.21.15.28:0/186576605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac00f50 0x561e1abfc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.077+0000 7fc2e241b740 1 --2- 172.21.15.28:0/186576605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00630 0x561e1ac172d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.077+0000 7fc2e241b740 1 -- 172.21.15.28:0/186576605 >> 172.21.15.28:0/186576605 conn(0x561e1abfc950 msgr2=0x561e1ac08530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:33.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.077+0000 7fc2e241b740 1 -- 172.21.15.28:0/186576605 shutdown_connections 2024-04-25T01:37:33.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.077+0000 7fc2e241b740 1 -- 172.21.15.28:0/186576605 wait complete. 2024-04-25T01:37:33.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.078+0000 7fc2e241b740 1 -- start start 2024-04-25T01:37:33.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.078+0000 7fc2e241b740 1 Processor -- start 2024-04-25T01:37:33.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.078+0000 7fc2e241b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00630 0x561e1ac13d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.078+0000 7fc2e241b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac00f50 0x561e1ac05080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.078+0000 7fc2e241b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac0b620 0x561e1ac057a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.078+0000 7fc2e241b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d81aff80 con 0x561e1ac00f50 2024-04-25T01:37:33.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.078+0000 7fc2e241b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d81af8b0 con 0x561e1ac0b620 2024-04-25T01:37:33.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.078+0000 7fc2e241b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc2d81acaf0 con 0x561e1ac00630 2024-04-25T01:37:33.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.078+0000 7fc2e241b740 10 client.0 _ll_register_callbacks cb 0x7fc2d4097920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:37:33.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.078+0000 7fc2dccb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac00f50 0x561e1ac05080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.078+0000 7fc2dccb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac00f50 0x561e1ac05080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57998/0 (socket says 172.21.15.28:57998) 2024-04-25T01:37:33.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.078+0000 7fc2dccb1640 1 -- 172.21.15.28:0/1042855497 learned_addr learned my addr 172.21.15.28:0/1042855497 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:37:33.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.078+0000 7fc2ddcb3640 1 --2- 172.21.15.28:0/1042855497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac0b620 0x561e1ac057a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.078+0000 7fc2dd4b2640 1 --2- 172.21.15.28:0/1042855497 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00630 0x561e1ac13d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2dccb1640 1 -- 172.21.15.28:0/1042855497 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00630 msgr2=0x561e1ac13d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:33.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2dccb1640 1 --2- 172.21.15.28:0/1042855497 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00630 0x561e1ac13d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2dccb1640 1 -- 172.21.15.28:0/1042855497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac0b620 msgr2=0x561e1ac057a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:37:33.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2dccb1640 1 --2- 172.21.15.28:0/1042855497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac0b620 0x561e1ac057a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2dccb1640 1 -- 172.21.15.28:0/1042855497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e1ac80b80 con 0x561e1ac00f50 2024-04-25T01:37:33.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2dccb1640 1 --2- 172.21.15.28:0/1042855497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac00f50 0x561e1ac05080 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fc2c800e760 tx=0x7fc2c8009690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8d8b94af57a4f30 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:37:33.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:37:33.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2c0ff9640 1 -- 172.21.15.28:0/1042855497 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2c8026070 con 0x561e1ac00f50 2024-04-25T01:37:33.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2c0ff9640 1 -- 172.21.15.28:0/1042855497 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc2c8004240 con 0x561e1ac00f50 2024-04-25T01:37:33.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2e241b740 1 -- 172.21.15.28:0/1042855497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2d0036050 con 0x561e1ac00f50 2024-04-25T01:37:33.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2c0ff9640 1 -- 172.21.15.28:0/1042855497 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc2c801d040 con 0x561e1ac00f50 2024-04-25T01:37:33.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2e241b740 10 client.17130 fetch_fsmap learned FSMap version 28 2024-04-25T01:37:33.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2e241b740 1 -- 172.21.15.28:0/1042855497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561e1ac81c20 con 0x561e1ac00f50 2024-04-25T01:37:33.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2c0ff9640 1 -- 172.21.15.28:0/1042855497 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2d0036050 con 0x561e1ac00f50 2024-04-25T01:37:33.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2c0ff9640 1 -- 172.21.15.28:0/1042855497 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc2c8003680 con 0x561e1ac00f50 2024-04-25T01:37:33.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2e241b740 10 client.17130 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:37:33.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2e241b740 10 client.17130 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:37:33.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2e241b740 20 client.17130 populate_metadata read hostname 'smithi028' 2024-04-25T01:37:33.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2e241b740 1 --2- 172.21.15.28:0/1042855497 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x561e1ac82390 0x561e1ac847b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:37:33.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2e241b740 4 client.17130 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:37:33.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.079+0000 7fc2e241b740 1 -- 172.21.15.28:0/1042855497 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561e1ab6cf40 con 0x561e1ac82390 2024-04-25T01:37:33.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.080+0000 7fc2dd4b2640 1 --2- 172.21.15.28:0/1042855497 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x561e1ac82390 0x561e1ac847b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:37:33.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.080+0000 7fc2c0ff9640 10 client.17130 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:33.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.080+0000 7fc2dd4b2640 1 --2- 172.21.15.28:0/1042855497 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x561e1ac82390 0x561e1ac847b0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19c2b6b615c913b server_cookie=2bdf97ce99513d7e in_seq=0 out_seq=0 2024-04-25T01:37:33.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.080+0000 7fc2c0ff9640 1 -- 172.21.15.28:0/1042855497 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561e1ab6cf40 con 0x561e1ac82390 2024-04-25T01:37:33.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.080+0000 7fc2c0ff9640 10 client.17130 handle_command_reply: tid=0 2024-04-25T01:37:33.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.080+0000 7fc2e241b740 1 client.17130 shutdown 2024-04-25T01:37:33.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.080+0000 7fc2e241b740 1 -- 172.21.15.28:0/1042855497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac00f50 msgr2=0x561e1ac05080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:33.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.080+0000 7fc2e241b740 1 --2- 172.21.15.28:0/1042855497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac00f50 0x561e1ac05080 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fc2c800e760 tx=0x7fc2c8009690 comp rx=0 tx=0).stop 2024-04-25T01:37:33.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.080+0000 7fc2e241b740 1 -- 172.21.15.28:0/1042855497 shutdown_connections 2024-04-25T01:37:33.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.080+0000 7fc2e241b740 1 --2- 172.21.15.28:0/1042855497 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x561e1ac82390 0x561e1ac847b0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.080+0000 7fc2e241b740 1 --2- 172.21.15.28:0/1042855497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1ac0b620 0x561e1ac057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.080+0000 7fc2c0ff9640 0 client.17130 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:37:33.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.080+0000 7fc2e241b740 1 --2- 172.21.15.28:0/1042855497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561e1ac00f50 0x561e1ac05080 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.080+0000 7fc2e241b740 1 --2- 172.21.15.28:0/1042855497 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561e1ac00630 0x561e1ac13d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.080+0000 7fc2e241b740 1 -- 172.21.15.28:0/1042855497 >> 172.21.15.28:0/1042855497 conn(0x561e1abfc950 msgr2=0x561e1ac08da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:33.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.081+0000 7fc2e241b740 1 -- 172.21.15.28:0/1042855497 shutdown_connections 2024-04-25T01:37:33.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.081+0000 7fc2e241b740 1 -- 172.21.15.28:0/1042855497 wait complete. 2024-04-25T01:37:33.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.081+0000 7fc2e241b740 20 client.17130 trim_cache size 0 max 16384 2024-04-25T01:37:33.065 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:37:33.065 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:37:33.065 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-25T01:37:33.065 INFO:teuthology.orchestra.run.smithi028.stdout: "12bcd52d-dead-40c0-a880-fe68faa01255": { 2024-04-25T01:37:33.065 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-25T01:37:33.065 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "12bcd52d-dead-40c0-a880-fe68faa01255", 2024-04-25T01:37:33.065 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-25T01:37:33.065 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:37:33.065 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:37:33.065 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:37:33.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.081+0000 7fc2df716640 1 -- 172.21.15.28:0/2377399831 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc2c40a2d00 msgr2=0x7fc2c40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:33.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.081+0000 7fc2df716640 1 --2- 172.21.15.28:0/2377399831 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc2c40a2d00 0x7fc2c40a51c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0009a60 tx=0x7fc2d00093a0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.081+0000 7fc2df716640 1 -- 172.21.15.28:0/2377399831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d8100620 msgr2=0x7fc2d8197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:37:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.081+0000 7fc2df716640 1 --2- 172.21.15.28:0/2377399831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d8100620 0x7fc2d8197fd0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400ea60 tx=0x7fc2d400ef20 comp rx=0 tx=0).stop 2024-04-25T01:37:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.081+0000 7fc2df716640 1 -- 172.21.15.28:0/2377399831 shutdown_connections 2024-04-25T01:37:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.081+0000 7fc2df716640 1 --2- 172.21.15.28:0/2377399831 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc2c40a2d00 0x7fc2c40a51c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.081+0000 7fc2df716640 1 --2- 172.21.15.28:0/2377399831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2d8106630 0x7fc2d819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.081+0000 7fc2df716640 1 --2- 172.21.15.28:0/2377399831 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc2d8100fe0 0x7fc2d8198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.081+0000 7fc2df716640 1 --2- 172.21.15.28:0/2377399831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d8100620 0x7fc2d8197fd0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:37:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.081+0000 7fc2df716640 1 -- 172.21.15.28:0/2377399831 >> 172.21.15.28:0/2377399831 conn(0x7fc2d80776a0 msgr2=0x7fc2d806a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:37:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.081+0000 7fc2df716640 1 -- 172.21.15.28:0/2377399831 shutdown_connections 2024-04-25T01:37:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:37:33.081+0000 7fc2df716640 1 -- 172.21.15.28:0/2377399831 wait complete. 2024-04-25T01:37:33.070 INFO:tasks.cephfs.filesystem:scrub status for tag:12bcd52d-dead-40c0-a880-fe68faa01255 - {'path': '/', 'tag': '12bcd52d-dead-40c0-a880-fe68faa01255', 'options': 'recursive,force'} 2024-04-25T01:37:34.036 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:33 smithi032 ceph-mon[32461]: pgmap v139: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:37:34.036 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:33 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:37:34.036 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:33 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:37:34.037 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:33 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:37:34.037 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:33 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:37:34.037 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:33 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:37:34.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:33 smithi028 ceph-mon[29390]: pgmap v139: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:37:34.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:33 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:37:34.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:33 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:37:34.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:33 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:37:34.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:33 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:37:34.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:33 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:37:34.208 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:33 smithi130 ceph-mon[32702]: pgmap v139: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:37:34.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:33 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:37:34.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:33 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:37:34.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:33 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:37:34.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:33 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:37:34.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:33 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:37:36.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:35 smithi028 ceph-mon[29390]: pgmap v140: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:37:36.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:35 smithi130 ceph-mon[32702]: pgmap v140: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:37:36.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:35 smithi032 ceph-mon[32461]: pgmap v140: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:37:38.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:37 smithi028 ceph-mon[29390]: pgmap v141: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-04-25T01:37:38.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:37 smithi130 ceph-mon[32702]: pgmap v141: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-04-25T01:37:38.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:37 smithi032 ceph-mon[32461]: pgmap v141: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-04-25T01:37:40.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:39 smithi028 ceph-mon[29390]: pgmap v142: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T01:37:40.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:39 smithi130 ceph-mon[32702]: pgmap v142: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T01:37:40.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:39 smithi032 ceph-mon[32461]: pgmap v142: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T01:37:42.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:41 smithi028 ceph-mon[29390]: pgmap v143: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:37:42.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:41 smithi130 ceph-mon[32702]: pgmap v143: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:37:42.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:41 smithi032 ceph-mon[32461]: pgmap v143: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:37:44.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:43 smithi028 ceph-mon[29390]: pgmap v144: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-04-25T01:37:44.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:43 smithi130 ceph-mon[32702]: pgmap v144: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-04-25T01:37:44.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:43 smithi032 ceph-mon[32461]: pgmap v144: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-04-25T01:37:46.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:45 smithi028 ceph-mon[29390]: pgmap v145: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:37:46.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:45 smithi130 ceph-mon[32702]: pgmap v145: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:37:46.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:45 smithi032 ceph-mon[32461]: pgmap v145: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:37:48.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:47 smithi028 ceph-mon[29390]: pgmap v146: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-04-25T01:37:48.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:47 smithi130 ceph-mon[32702]: pgmap v146: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-04-25T01:37:48.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:47 smithi032 ceph-mon[32461]: pgmap v146: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-04-25T01:37:50.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:49 smithi028 ceph-mon[29390]: pgmap v147: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-25T01:37:50.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:49 smithi130 ceph-mon[32702]: pgmap v147: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-25T01:37:50.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:49 smithi032 ceph-mon[32461]: pgmap v147: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-25T01:37:52.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:51 smithi028 ceph-mon[29390]: pgmap v148: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:37:52.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:51 smithi130 ceph-mon[32702]: pgmap v148: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:37:52.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:51 smithi032 ceph-mon[32461]: pgmap v148: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:37:54.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:53 smithi028 ceph-mon[29390]: pgmap v149: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:37:54.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:53 smithi130 ceph-mon[32702]: pgmap v149: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:37:54.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:53 smithi032 ceph-mon[32461]: pgmap v149: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:37:56.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:55 smithi028 ceph-mon[29390]: pgmap v150: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:37:56.209 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:55 smithi130 ceph-mon[32702]: pgmap v150: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:37:56.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:55 smithi032 ceph-mon[32461]: pgmap v150: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:37:58.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:57 smithi028 ceph-mon[29390]: pgmap v151: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:37:58.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:57 smithi130 ceph-mon[32702]: pgmap v151: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:37:58.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:57 smithi032 ceph-mon[32461]: pgmap v151: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:38:00.062 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:37:59 smithi028 ceph-mon[29390]: pgmap v152: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-04-25T01:38:00.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:37:59 smithi130 ceph-mon[32702]: pgmap v152: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-04-25T01:38:00.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:37:59 smithi032 ceph-mon[32461]: pgmap v152: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-04-25T01:38:02.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:01 smithi028 ceph-mon[29390]: pgmap v153: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-25T01:38:02.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:01 smithi130 ceph-mon[32702]: pgmap v153: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-25T01:38:02.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:01 smithi032 ceph-mon[32461]: pgmap v153: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-25T01:38:03.071 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:38:03.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:38:03.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.209+0000 7fbc13509640 1 -- 172.21.15.28:0/3087709484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0c06a6e0 msgr2=0x7fbc0c10dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.209+0000 7fbc13509640 1 --2- 172.21.15.28:0/3087709484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0c06a6e0 0x7fbc0c10dc10 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009900 tx=0x7fbbfc02a410 comp rx=0 tx=0).stop 2024-04-25T01:38:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.209+0000 7fbc13509640 1 -- 172.21.15.28:0/3087709484 shutdown_connections 2024-04-25T01:38:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.209+0000 7fbc13509640 1 --2- 172.21.15.28:0/3087709484 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc0c10e150 0x7fbc0c110540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.209+0000 7fbc13509640 1 --2- 172.21.15.28:0/3087709484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0c06a6e0 0x7fbc0c10dc10 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.209+0000 7fbc13509640 1 --2- 172.21.15.28:0/3087709484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc0c069dc0 0x7fbc0c06a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.209+0000 7fbc13509640 1 -- 172.21.15.28:0/3087709484 >> 172.21.15.28:0/3087709484 conn(0x7fbc0c0776a0 msgr2=0x7fbc0c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.209+0000 7fbc13509640 1 -- 172.21.15.28:0/3087709484 shutdown_connections 2024-04-25T01:38:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.209+0000 7fbc13509640 1 -- 172.21.15.28:0/3087709484 wait complete. 2024-04-25T01:38:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.209+0000 7fbc13509640 1 Processor -- start 2024-04-25T01:38:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.209+0000 7fbc13509640 1 -- start start 2024-04-25T01:38:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.210+0000 7fbc13509640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc0c069dc0 0x7fbc0c19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.210+0000 7fbc13509640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc0c06a6e0 0x7fbc0c19a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.210+0000 7fbc13509640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0c10e150 0x7fbc0c19ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.210+0000 7fbc13509640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c113d10 con 0x7fbc0c10e150 2024-04-25T01:38:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.210+0000 7fbc13509640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c113b90 con 0x7fbc0c06a6e0 2024-04-25T01:38:03.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.210+0000 7fbc13509640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c113e90 con 0x7fbc0c069dc0 2024-04-25T01:38:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.210+0000 7fbc11aa6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0c10e150 0x7fbc0c19ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.210+0000 7fbc11aa6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0c10e150 0x7fbc0c19ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42714/0 (socket says 172.21.15.28:42714) 2024-04-25T01:38:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.210+0000 7fbc11aa6640 1 -- 172.21.15.28:0/982768451 learned_addr learned my addr 172.21.15.28:0/982768451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.210+0000 7fbc10aa4640 1 --2- 172.21.15.28:0/982768451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc0c06a6e0 0x7fbc0c19a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.210+0000 7fbc112a5640 1 --2- 172.21.15.28:0/982768451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc0c069dc0 0x7fbc0c19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.210+0000 7fbc11aa6640 1 -- 172.21.15.28:0/982768451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc0c069dc0 msgr2=0x7fbc0c19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.210+0000 7fbc11aa6640 1 --2- 172.21.15.28:0/982768451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc0c069dc0 0x7fbc0c19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.210+0000 7fbc11aa6640 1 -- 172.21.15.28:0/982768451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc0c06a6e0 msgr2=0x7fbc0c19a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.210+0000 7fbc11aa6640 1 --2- 172.21.15.28:0/982768451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc0c06a6e0 0x7fbc0c19a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.210+0000 7fbc11aa6640 1 -- 172.21.15.28:0/982768451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0c19f150 con 0x7fbc0c10e150 2024-04-25T01:38:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.210+0000 7fbc10aa4640 1 --2- 172.21.15.28:0/982768451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc0c06a6e0 0x7fbc0c19a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.210+0000 7fbc11aa6640 1 --2- 172.21.15.28:0/982768451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0c10e150 0x7fbc0c19ea20 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fbc0400ccb0 tx=0x7fbc0400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.210+0000 7fbc027fc640 1 -- 172.21.15.28:0/982768451 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc0400f450 con 0x7fbc0c10e150 2024-04-25T01:38:03.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.211+0000 7fbc13509640 1 -- 172.21.15.28:0/982768451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc0c19f440 con 0x7fbc0c10e150 2024-04-25T01:38:03.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.211+0000 7fbc027fc640 1 -- 172.21.15.28:0/982768451 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc0400fad0 con 0x7fbc0c10e150 2024-04-25T01:38:03.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.211+0000 7fbc027fc640 1 -- 172.21.15.28:0/982768451 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04018bf0 con 0x7fbc0c10e150 2024-04-25T01:38:03.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.211+0000 7fbc13509640 1 -- 172.21.15.28:0/982768451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc0c1a6cd0 con 0x7fbc0c10e150 2024-04-25T01:38:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.213+0000 7fbc027fc640 1 -- 172.21.15.28:0/982768451 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fbc0400f5f0 con 0x7fbc0c10e150 2024-04-25T01:38:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.214+0000 7fbc1620b740 1 Processor -- start 2024-04-25T01:38:03.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.214+0000 7fbc1620b740 1 -- start start 2024-04-25T01:38:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.214+0000 7fbc1620b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26cc630 0x55aef26cca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.214+0000 7fbc1620b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26ccf50 0x55aef26d70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.214+0000 7fbc1620b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26d7620 0x55aef26d9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.214+0000 7fbc1620b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aef26316b0 con 0x55aef26d7620 2024-04-25T01:38:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.214+0000 7fbc1620b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55aef25d9880 con 0x55aef26ccf50 2024-04-25T01:38:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.214+0000 7fbc1620b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55aef262d670 con 0x55aef26cc630 2024-04-25T01:38:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.214+0000 7fbc10aa4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26d7620 0x55aef26d9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.214+0000 7fbc027fc640 1 --2- 172.21.15.28:0/982768451 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbbf80a2d00 0x7fbbf80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.214+0000 7fbc10aa4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26d7620 0x55aef26d9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42722/0 (socket says 172.21.15.28:42722) 2024-04-25T01:38:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.214+0000 7fbc112a5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26ccf50 0x55aef26d70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.214+0000 7fbc10aa4640 1 -- 172.21.15.28:0/991858004 learned_addr learned my addr 172.21.15.28:0/991858004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.215+0000 7fbc10aa4640 1 --2- 172.21.15.28:0/991858004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26cc630 0x55aef26cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.215+0000 7fbc11aa6640 1 --2- 172.21.15.28:0/982768451 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbbf80a2d00 0x7fbbf80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.215+0000 7fbc027fc640 1 -- 172.21.15.28:0/982768451 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fbc040cfe60 con 0x7fbc0c10e150 2024-04-25T01:38:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.215+0000 7fbc112a5640 1 -- 172.21.15.28:0/991858004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26cc630 msgr2=0x55aef26cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.215+0000 7fbc112a5640 1 --2- 172.21.15.28:0/991858004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26cc630 0x55aef26cca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.215+0000 7fbc112a5640 1 -- 172.21.15.28:0/991858004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26d7620 msgr2=0x55aef26d9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.215+0000 7fbc112a5640 1 --2- 172.21.15.28:0/991858004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26d7620 0x55aef26d9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.215+0000 7fbc112a5640 1 -- 172.21.15.28:0/991858004 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aef240f7d0 con 0x55aef26ccf50 2024-04-25T01:38:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.215+0000 7fbc112a5640 1 --2- 172.21.15.28:0/991858004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26ccf50 0x55aef26d70e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbc08009c20 tx=0x7fbc08011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d31ac1bfb6fcc58 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.215+0000 7fbc11aa6640 1 --2- 172.21.15.28:0/982768451 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbbf80a2d00 0x7fbbf80a51c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbc0409bd40 tx=0x7fbc040914d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.215+0000 7fbbf77fe640 1 -- 172.21.15.28:0/991858004 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc08004050 con 0x55aef26ccf50 2024-04-25T01:38:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.215+0000 7fbbf77fe640 1 -- 172.21.15.28:0/991858004 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc08005500 con 0x55aef26ccf50 2024-04-25T01:38:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.215+0000 7fbbf77fe640 1 -- 172.21.15.28:0/991858004 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc08004d60 con 0x55aef26ccf50 2024-04-25T01:38:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.216+0000 7fbc1620b740 1 -- 172.21.15.28:0/991858004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26ccf50 msgr2=0x55aef26d70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.216+0000 7fbc1620b740 1 --2- 172.21.15.28:0/991858004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26ccf50 0x55aef26d70e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbc08009c20 tx=0x7fbc08011810 comp rx=0 tx=0).stop 2024-04-25T01:38:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.216+0000 7fbc1620b740 1 -- 172.21.15.28:0/991858004 shutdown_connections 2024-04-25T01:38:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.216+0000 7fbc1620b740 1 --2- 172.21.15.28:0/991858004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26d7620 0x55aef26d9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.216+0000 7fbc1620b740 1 --2- 172.21.15.28:0/991858004 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26ccf50 0x55aef26d70e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.216+0000 7fbc1620b740 1 --2- 172.21.15.28:0/991858004 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26cc630 0x55aef26cca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.216+0000 7fbc1620b740 1 -- 172.21.15.28:0/991858004 >> 172.21.15.28:0/991858004 conn(0x55aef26c8950 msgr2=0x55aef26c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.216+0000 7fbc1620b740 1 -- 172.21.15.28:0/991858004 shutdown_connections 2024-04-25T01:38:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.216+0000 7fbc1620b740 1 -- 172.21.15.28:0/991858004 wait complete. 2024-04-25T01:38:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.216+0000 7fbc1620b740 1 -- start start 2024-04-25T01:38:03.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.216+0000 7fbc1620b740 1 Processor -- start 2024-04-25T01:38:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.217+0000 7fbc1620b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26cc630 0x55aef26d44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.217+0000 7fbc1620b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26ccf50 0x55aef26d4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.217+0000 7fbc1620b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26d7620 0x55aef26d4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.217+0000 7fbc1620b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aef26da410 con 0x55aef26cc630 2024-04-25T01:38:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.217+0000 7fbc1620b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55aef25ecf50 con 0x55aef26ccf50 2024-04-25T01:38:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.217+0000 7fbc1620b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55aef26da590 con 0x55aef26d7620 2024-04-25T01:38:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.217+0000 7fbc1620b740 10 client.0 _ll_register_callbacks cb 0x55aef2628000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.217+0000 7fbc10aa4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26cc630 0x55aef26d44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.217+0000 7fbc10aa4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26cc630 0x55aef26d44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42734/0 (socket says 172.21.15.28:42734) 2024-04-25T01:38:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.217+0000 7fbc10aa4640 1 -- 172.21.15.28:0/3175745252 learned_addr learned my addr 172.21.15.28:0/3175745252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.217+0000 7fbc112a5640 1 --2- 172.21.15.28:0/3175745252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26ccf50 0x55aef26d4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.217+0000 7fbc10aa4640 1 -- 172.21.15.28:0/3175745252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26d7620 msgr2=0x55aef26d4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.217+0000 7fbc10aa4640 1 --2- 172.21.15.28:0/3175745252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26d7620 0x55aef26d4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.217+0000 7fbc10aa4640 1 -- 172.21.15.28:0/3175745252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26ccf50 msgr2=0x55aef26d4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.217+0000 7fbc10aa4640 1 --2- 172.21.15.28:0/3175745252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26ccf50 0x55aef26d4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.217+0000 7fbc10aa4640 1 -- 172.21.15.28:0/3175745252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aef23453f0 con 0x55aef26cc630 2024-04-25T01:38:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.218+0000 7fbc10aa4640 1 --2- 172.21.15.28:0/3175745252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26cc630 0x55aef26d44c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc006010 tx=0x7fbbfc005a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=233eca1b5194f18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.218+0000 7fbbf4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.218+0000 7fbbf4ff9640 1 -- 172.21.15.28:0/3175745252 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc045030 con 0x55aef26cc630 2024-04-25T01:38:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.218+0000 7fbbf4ff9640 1 -- 172.21.15.28:0/3175745252 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fbbfc004040 con 0x55aef26cc630 2024-04-25T01:38:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.218+0000 7fbc1620b740 1 -- 172.21.15.28:0/3175745252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc08005500 con 0x55aef26cc630 2024-04-25T01:38:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.218+0000 7fbbf4ff9640 1 -- 172.21.15.28:0/3175745252 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbbfc052050 con 0x55aef26cc630 2024-04-25T01:38:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.218+0000 7fbc1620b740 10 client.17142 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.218+0000 7fbc1620b740 1 -- 172.21.15.28:0/3175745252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55aef23050f0 con 0x55aef26cc630 2024-04-25T01:38:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.218+0000 7fbbf4ff9640 1 -- 172.21.15.28:0/3175745252 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc08005500 con 0x55aef26cc630 2024-04-25T01:38:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.218+0000 7fbbf4ff9640 1 -- 172.21.15.28:0/3175745252 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbbfc03a040 con 0x55aef26cc630 2024-04-25T01:38:03.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.218+0000 7fbc1620b740 10 client.17142 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.218+0000 7fbc1620b740 10 client.17142 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.218+0000 7fbc1620b740 20 client.17142 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.218+0000 7fbc1620b740 1 --2- 172.21.15.28:0/3175745252 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55aef26e7810 0x55aef26e9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.218+0000 7fbc1620b740 4 client.17142 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.218+0000 7fbc1620b740 1 -- 172.21.15.28:0/3175745252 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aef263ebf0 con 0x55aef26e7810 2024-04-25T01:38:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.219+0000 7fbc112a5640 1 --2- 172.21.15.28:0/3175745252 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55aef26e7810 0x55aef26e9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.219+0000 7fbc112a5640 1 --2- 172.21.15.28:0/3175745252 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55aef26e7810 0x55aef26e9c60 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16bf1bcb5e77844f server_cookie=a1023fa4313a03c0 in_seq=0 out_seq=0 2024-04-25T01:38:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.219+0000 7fbbf4ff9640 10 client.17142 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.220+0000 7fbbf4ff9640 1 -- 172.21.15.28:0/3175745252 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aef263ebf0 con 0x55aef26e7810 2024-04-25T01:38:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.220+0000 7fbbf4ff9640 10 client.17142 handle_command_reply: tid=0 2024-04-25T01:38:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.220+0000 7fbc1620b740 1 client.17142 shutdown 2024-04-25T01:38:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.220+0000 7fbc1620b740 1 -- 172.21.15.28:0/3175745252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26cc630 msgr2=0x55aef26d44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.220+0000 7fbc1620b740 1 --2- 172.21.15.28:0/3175745252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26cc630 0x55aef26d44c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc006010 tx=0x7fbbfc005a40 comp rx=0 tx=0).stop 2024-04-25T01:38:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.220+0000 7fbc1620b740 1 -- 172.21.15.28:0/3175745252 shutdown_connections 2024-04-25T01:38:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.220+0000 7fbc1620b740 1 --2- 172.21.15.28:0/3175745252 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55aef26e7810 0x55aef26e9c60 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.220+0000 7fbc1620b740 1 --2- 172.21.15.28:0/3175745252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26d7620 0x55aef26d4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.220+0000 7fbbf4ff9640 0 client.17142 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.220+0000 7fbc1620b740 1 --2- 172.21.15.28:0/3175745252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26ccf50 0x55aef26d4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.220+0000 7fbc1620b740 1 --2- 172.21.15.28:0/3175745252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26cc630 0x55aef26d44c0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.220+0000 7fbc1620b740 1 -- 172.21.15.28:0/3175745252 >> 172.21.15.28:0/3175745252 conn(0x55aef26c8950 msgr2=0x55aef26ce750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.220+0000 7fbc1620b740 1 -- 172.21.15.28:0/3175745252 shutdown_connections 2024-04-25T01:38:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.220+0000 7fbc1620b740 1 -- 172.21.15.28:0/3175745252 wait complete. 2024-04-25T01:38:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.220+0000 7fbc1620b740 20 client.17142 trim_cache size 0 max 16384 2024-04-25T01:38:03.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.238+0000 7fbc1620b740 1 Processor -- start 2024-04-25T01:38:03.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.238+0000 7fbc1620b740 1 -- start start 2024-04-25T01:38:03.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.238+0000 7fbc1620b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26cc630 0x55aef26e32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.238+0000 7fbc1620b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26ccf50 0x55aef26d2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.238+0000 7fbc1620b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26d7620 0x55aef26ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.238+0000 7fbc1620b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c1b0c10 con 0x55aef26cc630 2024-04-25T01:38:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.238+0000 7fbc1620b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c1b2750 con 0x55aef26d7620 2024-04-25T01:38:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.238+0000 7fbc1620b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c1b12e0 con 0x55aef26ccf50 2024-04-25T01:38:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.238+0000 7fbc10aa4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26cc630 0x55aef26e32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.238+0000 7fbc10aa4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26cc630 0x55aef26e32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42736/0 (socket says 172.21.15.28:42736) 2024-04-25T01:38:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.238+0000 7fbc10aa4640 1 -- 172.21.15.28:0/2271663501 learned_addr learned my addr 172.21.15.28:0/2271663501 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.238+0000 7fbc10aa4640 1 --2- 172.21.15.28:0/2271663501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26d7620 0x55aef26ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.238+0000 7fbc10aa4640 1 -- 172.21.15.28:0/2271663501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26ccf50 msgr2=0x55aef26d2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.238+0000 7fbc112a5640 1 --2- 172.21.15.28:0/2271663501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26ccf50 0x55aef26d2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.239+0000 7fbc10aa4640 1 --2- 172.21.15.28:0/2271663501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26ccf50 0x55aef26d2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.239+0000 7fbc10aa4640 1 -- 172.21.15.28:0/2271663501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26d7620 msgr2=0x55aef26ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.239+0000 7fbc10aa4640 1 --2- 172.21.15.28:0/2271663501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26d7620 0x55aef26ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.239+0000 7fbc10aa4640 1 -- 172.21.15.28:0/2271663501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aef26d5420 con 0x55aef26cc630 2024-04-25T01:38:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.239+0000 7fbc112a5640 1 --2- 172.21.15.28:0/2271663501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26ccf50 0x55aef26d2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.239+0000 7fbc10aa4640 1 --2- 172.21.15.28:0/2271663501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26cc630 0x55aef26e32d0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009900 tx=0x7fbbfc009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=745a4c82d742e676 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.239+0000 7fbbf6ffd640 1 -- 172.21.15.28:0/2271663501 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc05b020 con 0x55aef26cc630 2024-04-25T01:38:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.239+0000 7fbbf6ffd640 1 -- 172.21.15.28:0/2271663501 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbfc04f070 con 0x55aef26cc630 2024-04-25T01:38:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.239+0000 7fbbf6ffd640 1 -- 172.21.15.28:0/2271663501 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc045040 con 0x55aef26cc630 2024-04-25T01:38:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.239+0000 7fbc1620b740 1 -- 172.21.15.28:0/2271663501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26cc630 msgr2=0x55aef26e32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.239+0000 7fbc1620b740 1 --2- 172.21.15.28:0/2271663501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26cc630 0x55aef26e32d0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009900 tx=0x7fbbfc009230 comp rx=0 tx=0).stop 2024-04-25T01:38:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.239+0000 7fbc1620b740 1 -- 172.21.15.28:0/2271663501 shutdown_connections 2024-04-25T01:38:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.239+0000 7fbc1620b740 1 --2- 172.21.15.28:0/2271663501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26d7620 0x55aef26ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.239+0000 7fbc1620b740 1 --2- 172.21.15.28:0/2271663501 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26ccf50 0x55aef26d2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.239+0000 7fbc1620b740 1 --2- 172.21.15.28:0/2271663501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26cc630 0x55aef26e32d0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.239+0000 7fbc1620b740 1 -- 172.21.15.28:0/2271663501 >> 172.21.15.28:0/2271663501 conn(0x55aef26c8950 msgr2=0x55aef26d4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.240+0000 7fbc1620b740 1 -- 172.21.15.28:0/2271663501 shutdown_connections 2024-04-25T01:38:03.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.240+0000 7fbc1620b740 1 -- 172.21.15.28:0/2271663501 wait complete. 2024-04-25T01:38:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.240+0000 7fbc1620b740 1 -- start start 2024-04-25T01:38:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.240+0000 7fbc1620b740 1 Processor -- start 2024-04-25T01:38:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.240+0000 7fbc1620b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26cc630 0x55aef26dfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.240+0000 7fbc1620b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26ccf50 0x55aef26d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.240+0000 7fbc1620b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26d7620 0x55aef26d1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.240+0000 7fbc1620b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c1b2080 con 0x55aef26ccf50 2024-04-25T01:38:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.240+0000 7fbc1620b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c1b19b0 con 0x55aef26d7620 2024-04-25T01:38:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.240+0000 7fbc1620b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c1aebf0 con 0x55aef26cc630 2024-04-25T01:38:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.240+0000 7fbc1620b740 10 client.0 _ll_register_callbacks cb 0x7fbbfc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.240+0000 7fbc112a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26ccf50 0x55aef26d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.240+0000 7fbc112a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26ccf50 0x55aef26d0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42748/0 (socket says 172.21.15.28:42748) 2024-04-25T01:38:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.240+0000 7fbc112a5640 1 -- 172.21.15.28:0/2320552671 learned_addr learned my addr 172.21.15.28:0/2320552671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.240+0000 7fbc10aa4640 1 --2- 172.21.15.28:0/2320552671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26d7620 0x55aef26d1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.241+0000 7fbc10aa4640 1 --2- 172.21.15.28:0/2320552671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26cc630 0x55aef26dfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.241+0000 7fbc112a5640 1 -- 172.21.15.28:0/2320552671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26cc630 msgr2=0x55aef26dfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.241+0000 7fbc112a5640 1 --2- 172.21.15.28:0/2320552671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26cc630 0x55aef26dfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.241+0000 7fbc112a5640 1 -- 172.21.15.28:0/2320552671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26d7620 msgr2=0x55aef26d1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.241+0000 7fbc112a5640 1 --2- 172.21.15.28:0/2320552671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26d7620 0x55aef26d1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.241+0000 7fbc112a5640 1 -- 172.21.15.28:0/2320552671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aef26d1ac0 con 0x55aef26ccf50 2024-04-25T01:38:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.241+0000 7fbc10aa4640 1 --2- 172.21.15.28:0/2320552671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26cc630 0x55aef26dfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.241+0000 7fbc112a5640 1 --2- 172.21.15.28:0/2320552671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26ccf50 0x55aef26d0e10 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fbc080047c0 tx=0x7fbc08004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e84dac18e594582 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.241+0000 7fbbf4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.241+0000 7fbbf4ff9640 1 -- 172.21.15.28:0/2320552671 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc08032060 con 0x55aef26ccf50 2024-04-25T01:38:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.241+0000 7fbbf4ff9640 1 -- 172.21.15.28:0/2320552671 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fbc0802cdc0 con 0x55aef26ccf50 2024-04-25T01:38:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.241+0000 7fbc1620b740 1 -- 172.21.15.28:0/2320552671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbfc04f070 con 0x55aef26ccf50 2024-04-25T01:38:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.241+0000 7fbbf4ff9640 1 -- 172.21.15.28:0/2320552671 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbc08039050 con 0x55aef26ccf50 2024-04-25T01:38:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.241+0000 7fbc1620b740 10 client.17148 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.241+0000 7fbc1620b740 1 -- 172.21.15.28:0/2320552671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55aef274dd20 con 0x55aef26ccf50 2024-04-25T01:38:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.241+0000 7fbbf4ff9640 1 -- 172.21.15.28:0/2320552671 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbfc04f070 con 0x55aef26ccf50 2024-04-25T01:38:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.242+0000 7fbbf4ff9640 1 -- 172.21.15.28:0/2320552671 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbc08019a00 con 0x55aef26ccf50 2024-04-25T01:38:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.242+0000 7fbc1620b740 10 client.17148 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.242+0000 7fbc1620b740 10 client.17148 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.242+0000 7fbc1620b740 20 client.17148 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.242+0000 7fbc1620b740 1 --2- 172.21.15.28:0/2320552671 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55aef274e310 0x55aef2750730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.242+0000 7fbc1620b740 4 client.17148 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:38:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.242+0000 7fbc1620b740 1 -- 172.21.15.28:0/2320552671 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55aef2638f40 con 0x55aef274e310 2024-04-25T01:38:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.242+0000 7fbc10aa4640 1 --2- 172.21.15.28:0/2320552671 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55aef274e310 0x55aef2750730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.242+0000 7fbc10aa4640 1 --2- 172.21.15.28:0/2320552671 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55aef274e310 0x55aef2750730 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a74dbec125189f97 server_cookie=b86d8fc4603fe6ab in_seq=0 out_seq=0 2024-04-25T01:38:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.242+0000 7fbbf4ff9640 10 client.17148 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbbf4ff9640 1 -- 172.21.15.28:0/2320552671 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55aef2638f40 con 0x55aef274e310 2024-04-25T01:38:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbbf4ff9640 10 client.17148 handle_command_reply: tid=0 2024-04-25T01:38:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc1620b740 1 client.17148 shutdown 2024-04-25T01:38:03.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc1620b740 1 -- 172.21.15.28:0/2320552671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26ccf50 msgr2=0x55aef26d0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc1620b740 1 --2- 172.21.15.28:0/2320552671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26ccf50 0x55aef26d0e10 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fbc080047c0 tx=0x7fbc08004600 comp rx=0 tx=0).stop 2024-04-25T01:38:03.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc1620b740 1 -- 172.21.15.28:0/2320552671 shutdown_connections 2024-04-25T01:38:03.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc1620b740 1 --2- 172.21.15.28:0/2320552671 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55aef274e310 0x55aef2750730 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc1620b740 1 --2- 172.21.15.28:0/2320552671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aef26d7620 0x55aef26d1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc1620b740 1 --2- 172.21.15.28:0/2320552671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aef26ccf50 0x55aef26d0e10 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbbf4ff9640 0 client.17148 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:03.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc1620b740 1 --2- 172.21.15.28:0/2320552671 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aef26cc630 0x55aef26dfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc1620b740 1 -- 172.21.15.28:0/2320552671 >> 172.21.15.28:0/2320552671 conn(0x55aef26c8950 msgr2=0x55aef26c7260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc1620b740 1 -- 172.21.15.28:0/2320552671 shutdown_connections 2024-04-25T01:38:03.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc1620b740 1 -- 172.21.15.28:0/2320552671 wait complete. 2024-04-25T01:38:03.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc1620b740 20 client.17148 trim_cache size 0 max 16384 2024-04-25T01:38:03.230 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:38:03.230 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:38:03.230 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:38:03.230 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:38:03.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc13509640 1 -- 172.21.15.28:0/982768451 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbbf80a2d00 msgr2=0x7fbbf80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc13509640 1 --2- 172.21.15.28:0/982768451 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbbf80a2d00 0x7fbbf80a51c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbc0409bd40 tx=0x7fbc040914d0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc13509640 1 -- 172.21.15.28:0/982768451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0c10e150 msgr2=0x7fbc0c19ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc13509640 1 --2- 172.21.15.28:0/982768451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0c10e150 0x7fbc0c19ea20 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fbc0400ccb0 tx=0x7fbc0400a620 comp rx=0 tx=0).stop 2024-04-25T01:38:03.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc13509640 1 -- 172.21.15.28:0/982768451 shutdown_connections 2024-04-25T01:38:03.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc13509640 1 --2- 172.21.15.28:0/982768451 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbbf80a2d00 0x7fbbf80a51c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc13509640 1 --2- 172.21.15.28:0/982768451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc0c10e150 0x7fbc0c19ea20 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc13509640 1 --2- 172.21.15.28:0/982768451 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbc0c06a6e0 0x7fbc0c19a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc13509640 1 --2- 172.21.15.28:0/982768451 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbc0c069dc0 0x7fbc0c19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.243+0000 7fbc13509640 1 -- 172.21.15.28:0/982768451 >> 172.21.15.28:0/982768451 conn(0x7fbc0c0776a0 msgr2=0x7fbc0c10f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.244+0000 7fbc13509640 1 -- 172.21.15.28:0/982768451 shutdown_connections 2024-04-25T01:38:03.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.244+0000 7fbc13509640 1 -- 172.21.15.28:0/982768451 wait complete. 2024-04-25T01:38:03.234 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:38:03.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:38:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.357+0000 7fd67ca64640 1 -- 172.21.15.28:0/3616479321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd678100690 msgr2=0x7fd678100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.357+0000 7fd67ca64640 1 --2- 172.21.15.28:0/3616479321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd678100690 0x7fd678100b10 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fd660009930 tx=0x7fd66002a420 comp rx=0 tx=0).stop 2024-04-25T01:38:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.357+0000 7fd67ca64640 1 -- 172.21.15.28:0/3616479321 shutdown_connections 2024-04-25T01:38:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.357+0000 7fd67ca64640 1 --2- 172.21.15.28:0/3616479321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd678101050 0x7fd67810d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.357+0000 7fd67ca64640 1 --2- 172.21.15.28:0/3616479321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd678100690 0x7fd678100b10 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.357+0000 7fd67ca64640 1 --2- 172.21.15.28:0/3616479321 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd678106650 0x7fd678106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.357+0000 7fd67ca64640 1 -- 172.21.15.28:0/3616479321 >> 172.21.15.28:0/3616479321 conn(0x7fd6780777a0 msgr2=0x7fd678077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.357+0000 7fd67ca64640 1 -- 172.21.15.28:0/3616479321 shutdown_connections 2024-04-25T01:38:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.357+0000 7fd67ca64640 1 -- 172.21.15.28:0/3616479321 wait complete. 2024-04-25T01:38:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.357+0000 7fd67ca64640 1 Processor -- start 2024-04-25T01:38:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.357+0000 7fd67ca64640 1 -- start start 2024-04-25T01:38:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd67ca64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd678100690 0x7fd6780fd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd67ca64640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd678101050 0x7fd6780fdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd67ca64640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd678106650 0x7fd6780fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd67ca64640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd678110a60 con 0x7fd678100690 2024-04-25T01:38:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd67ca64640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd6781108e0 con 0x7fd678106650 2024-04-25T01:38:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd67ca64640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd678110be0 con 0x7fd678101050 2024-04-25T01:38:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd67659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd678100690 0x7fd6780fd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd67659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd678100690 0x7fd6780fd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42760/0 (socket says 172.21.15.28:42760) 2024-04-25T01:38:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd67659c640 1 -- 172.21.15.28:0/572610936 learned_addr learned my addr 172.21.15.28:0/572610936 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd66ffff640 1 --2- 172.21.15.28:0/572610936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd678101050 0x7fd6780fdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd676d9d640 1 --2- 172.21.15.28:0/572610936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd678106650 0x7fd6780fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd67659c640 1 -- 172.21.15.28:0/572610936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd678101050 msgr2=0x7fd6780fdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd67659c640 1 --2- 172.21.15.28:0/572610936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd678101050 0x7fd6780fdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd67659c640 1 -- 172.21.15.28:0/572610936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd678106650 msgr2=0x7fd6780fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd67659c640 1 --2- 172.21.15.28:0/572610936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd678106650 0x7fd6780fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd67659c640 1 -- 172.21.15.28:0/572610936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6781a6540 con 0x7fd678100690 2024-04-25T01:38:03.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd67659c640 1 --2- 172.21.15.28:0/572610936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd678100690 0x7fd6780fd650 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fd66800d770 tx=0x7fd66800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd66f7fe640 1 -- 172.21.15.28:0/572610936 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd668017070 con 0x7fd678100690 2024-04-25T01:38:03.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd66f7fe640 1 -- 172.21.15.28:0/572610936 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd668005500 con 0x7fd678100690 2024-04-25T01:38:03.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd67ca64640 1 -- 172.21.15.28:0/572610936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6781a6830 con 0x7fd678100690 2024-04-25T01:38:03.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.358+0000 7fd66f7fe640 1 -- 172.21.15.28:0/572610936 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd668004e00 con 0x7fd678100690 2024-04-25T01:38:03.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.359+0000 7fd67ca64640 1 -- 172.21.15.28:0/572610936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6781a6d00 con 0x7fd678100690 2024-04-25T01:38:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.359+0000 7fd67ca64640 1 -- 172.21.15.28:0/572610936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6780689b0 con 0x7fd678100690 2024-04-25T01:38:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.361+0000 7fd66f7fe640 1 -- 172.21.15.28:0/572610936 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd668004fa0 con 0x7fd678100690 2024-04-25T01:38:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.361+0000 7fd66f7fe640 1 --2- 172.21.15.28:0/572610936 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd6640a2d00 0x7fd6640a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.361+0000 7fd66ffff640 1 --2- 172.21.15.28:0/572610936 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd6640a2d00 0x7fd6640a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.361+0000 7fd66f7fe640 1 -- 172.21.15.28:0/572610936 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fd6680cf1b0 con 0x7fd678100690 2024-04-25T01:38:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.362+0000 7fd66ffff640 1 --2- 172.21.15.28:0/572610936 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd6640a2d00 0x7fd6640a51c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd660009900 tx=0x7fd6600058d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.366+0000 7fd66f7fe640 1 -- 172.21.15.28:0/572610936 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd66809ba90 con 0x7fd678100690 2024-04-25T01:38:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.590+0000 7fd67ca64640 1 -- 172.21.15.28:0/572610936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd678109be0 con 0x7fd678100690 2024-04-25T01:38:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.591+0000 7fd66f7fe640 1 -- 172.21.15.28:0/572610936 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7fd66809b390 con 0x7fd678100690 2024-04-25T01:38:03.571 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:38:03.571 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:38:03.571 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:38:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.593+0000 7fd67ca64640 1 -- 172.21.15.28:0/572610936 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd6640a2d00 msgr2=0x7fd6640a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.593+0000 7fd67ca64640 1 --2- 172.21.15.28:0/572610936 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd6640a2d00 0x7fd6640a51c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd660009900 tx=0x7fd6600058d0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.593+0000 7fd67ca64640 1 -- 172.21.15.28:0/572610936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd678100690 msgr2=0x7fd6780fd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.593+0000 7fd67ca64640 1 --2- 172.21.15.28:0/572610936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd678100690 0x7fd6780fd650 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fd66800d770 tx=0x7fd66800dc30 comp rx=0 tx=0).stop 2024-04-25T01:38:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.593+0000 7fd67ca64640 1 -- 172.21.15.28:0/572610936 shutdown_connections 2024-04-25T01:38:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.593+0000 7fd67ca64640 1 --2- 172.21.15.28:0/572610936 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd6640a2d00 0x7fd6640a51c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.593+0000 7fd67ca64640 1 --2- 172.21.15.28:0/572610936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd678106650 0x7fd6780fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.593+0000 7fd67ca64640 1 --2- 172.21.15.28:0/572610936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd678101050 0x7fd6780fdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.593+0000 7fd67ca64640 1 --2- 172.21.15.28:0/572610936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd678100690 0x7fd6780fd650 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.593+0000 7fd67ca64640 1 -- 172.21.15.28:0/572610936 >> 172.21.15.28:0/572610936 conn(0x7fd6780777a0 msgr2=0x7fd67806a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.593+0000 7fd67ca64640 1 -- 172.21.15.28:0/572610936 shutdown_connections 2024-04-25T01:38:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.594+0000 7fd67ca64640 1 -- 172.21.15.28:0/572610936 wait complete. 2024-04-25T01:38:03.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:38:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.709+0000 7f7ec5363640 1 -- 172.21.15.28:0/2215668757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ec0100fe0 msgr2=0x7f7ec010d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.709+0000 7f7ec5363640 1 --2- 172.21.15.28:0/2215668757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ec0100fe0 0x7f7ec010d4f0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009cd0 tx=0x7f7eb802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.709+0000 7f7ec5363640 1 -- 172.21.15.28:0/2215668757 shutdown_connections 2024-04-25T01:38:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.709+0000 7f7ec5363640 1 --2- 172.21.15.28:0/2215668757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ec0100fe0 0x7f7ec010d4f0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.709+0000 7f7ec5363640 1 --2- 172.21.15.28:0/2215668757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec0100620 0x7f7ec0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.709+0000 7f7ec5363640 1 --2- 172.21.15.28:0/2215668757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ec0106630 0x7f7ec0106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.709+0000 7f7ec5363640 1 -- 172.21.15.28:0/2215668757 >> 172.21.15.28:0/2215668757 conn(0x7f7ec00776a0 msgr2=0x7f7ec0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.709+0000 7f7ec5363640 1 -- 172.21.15.28:0/2215668757 shutdown_connections 2024-04-25T01:38:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.709+0000 7f7ec5363640 1 -- 172.21.15.28:0/2215668757 wait complete. 2024-04-25T01:38:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.710+0000 7f7ec5363640 1 Processor -- start 2024-04-25T01:38:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.710+0000 7f7ec5363640 1 -- start start 2024-04-25T01:38:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.710+0000 7f7ec5363640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ec0100620 0x7f7ec0196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.710+0000 7f7ec5363640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ec0100fe0 0x7f7ec0197170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.710+0000 7f7ec5363640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec0106630 0x7f7ec01976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.710+0000 7f7ec5363640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec0110a40 con 0x7f7ec0100620 2024-04-25T01:38:03.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.710+0000 7f7ec5363640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec01108c0 con 0x7f7ec0106630 2024-04-25T01:38:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.710+0000 7f7ec5363640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec0110bc0 con 0x7f7ec0100fe0 2024-04-25T01:38:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.710+0000 7f7ebeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ec0100620 0x7f7ec0196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.710+0000 7f7ebeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ec0100620 0x7f7ec0196c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42780/0 (socket says 172.21.15.28:42780) 2024-04-25T01:38:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.710+0000 7f7ebeffd640 1 -- 172.21.15.28:0/1977335584 learned_addr learned my addr 172.21.15.28:0/1977335584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.710+0000 7f7eb7fff640 1 --2- 172.21.15.28:0/1977335584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ec0100fe0 0x7f7ec0197170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.710+0000 7f7ebeffd640 1 -- 172.21.15.28:0/1977335584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ec0100fe0 msgr2=0x7f7ec0197170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.710+0000 7f7ebeffd640 1 --2- 172.21.15.28:0/1977335584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ec0100fe0 0x7f7ec0197170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.710+0000 7f7ebeffd640 1 -- 172.21.15.28:0/1977335584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec0106630 msgr2=0x7f7ec01976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.710+0000 7f7ebeffd640 1 --2- 172.21.15.28:0/1977335584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec0106630 0x7f7ec01976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.710+0000 7f7ebeffd640 1 -- 172.21.15.28:0/1977335584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec01a64a0 con 0x7f7ec0100620 2024-04-25T01:38:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.710+0000 7f7ebeffd640 1 --2- 172.21.15.28:0/1977335584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ec0100620 0x7f7ec0196c30 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f7eb000d770 tx=0x7f7eb000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.710+0000 7f7ebcff9640 1 -- 172.21.15.28:0/1977335584 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb0017070 con 0x7f7ec0100620 2024-04-25T01:38:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.711+0000 7f7ebcff9640 1 -- 172.21.15.28:0/1977335584 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7eb0005500 con 0x7f7ec0100620 2024-04-25T01:38:03.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.711+0000 7f7ebcff9640 1 -- 172.21.15.28:0/1977335584 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb0004e00 con 0x7f7ec0100620 2024-04-25T01:38:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.711+0000 7f7ec5363640 1 -- 172.21.15.28:0/1977335584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ec01a6790 con 0x7f7ec0100620 2024-04-25T01:38:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.711+0000 7f7ec5363640 1 -- 172.21.15.28:0/1977335584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ec01a6c90 con 0x7f7ec0100620 2024-04-25T01:38:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.713+0000 7f7ebcff9640 1 -- 172.21.15.28:0/1977335584 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7eb0004fa0 con 0x7f7ec0100620 2024-04-25T01:38:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.713+0000 7f7ec8075740 1 Processor -- start 2024-04-25T01:38:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.713+0000 7f7ec8075740 1 -- start start 2024-04-25T01:38:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.713+0000 7f7ec8075740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 0x55f691f13960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.713+0000 7f7ec8075740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f13ea0 0x55f691f1e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.713+0000 7f7ec8075740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f691f1e570 0x55f691f20960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.713+0000 7f7ec8075740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f691e78670 con 0x55f691f13580 2024-04-25T01:38:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.713+0000 7f7ec8075740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f691e20970 con 0x55f691f1e570 2024-04-25T01:38:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.713+0000 7f7ec8075740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55f691e74630 con 0x55f691f13ea0 2024-04-25T01:38:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.714+0000 7f7ebf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 0x55f691f13960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.714+0000 7f7ebf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 0x55f691f13960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42790/0 (socket says 172.21.15.28:42790) 2024-04-25T01:38:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.714+0000 7f7ebf7fe640 1 -- 172.21.15.28:0/3868540984 learned_addr learned my addr 172.21.15.28:0/3868540984 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.714+0000 7f7ebeffd640 1 --2- 172.21.15.28:0/3868540984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f13ea0 0x55f691f1e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.714+0000 7f7eb7fff640 1 --2- 172.21.15.28:0/3868540984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f691f1e570 0x55f691f20960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.714+0000 7f7ebcff9640 1 --2- 172.21.15.28:0/1977335584 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7eac0a2d00 0x7f7eac0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.714+0000 7f7ebf7fe640 1 -- 172.21.15.28:0/3868540984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f13ea0 msgr2=0x55f691f1e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.714+0000 7f7ebf7fe640 1 --2- 172.21.15.28:0/3868540984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f13ea0 0x55f691f1e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.714+0000 7f7ebf7fe640 1 -- 172.21.15.28:0/3868540984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f691f1e570 msgr2=0x55f691f20960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.714+0000 7f7ebf7fe640 1 --2- 172.21.15.28:0/3868540984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f691f1e570 0x55f691f20960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.714+0000 7f7ebf7fe640 1 -- 172.21.15.28:0/3868540984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f691e44400 con 0x55f691f13580 2024-04-25T01:38:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.714+0000 7f7ebeffd640 1 --2- 172.21.15.28:0/3868540984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f13ea0 0x55f691f1e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.714+0000 7f7ebf7fe640 1 --2- 172.21.15.28:0/1977335584 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7eac0a2d00 0x7f7eac0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.714+0000 7f7ebcff9640 1 -- 172.21.15.28:0/1977335584 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f7eb00cf260 con 0x7f7ec0100620 2024-04-25T01:38:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.714+0000 7f7ebf7fe640 1 --2- 172.21.15.28:0/3868540984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 0x55f691f13960 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009df0 tx=0x7f7eb8005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad9e3ad827e5851 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.714+0000 7f7eb57fa640 1 -- 172.21.15.28:0/3868540984 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb804d070 con 0x55f691f13580 2024-04-25T01:38:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.714+0000 7f7eb57fa640 1 -- 172.21.15.28:0/3868540984 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7eb8043070 con 0x55f691f13580 2024-04-25T01:38:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.714+0000 7f7eb57fa640 1 -- 172.21.15.28:0/3868540984 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8030aa0 con 0x55f691f13580 2024-04-25T01:38:03.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.715+0000 7f7ec8075740 1 -- 172.21.15.28:0/3868540984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 msgr2=0x55f691f13960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.715+0000 7f7ec8075740 1 --2- 172.21.15.28:0/3868540984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 0x55f691f13960 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009df0 tx=0x7f7eb8005ef0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.715+0000 7f7ebf7fe640 1 --2- 172.21.15.28:0/1977335584 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7eac0a2d00 0x7f7eac0a51c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7eb804a000 tx=0x7f7eb804d000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.715+0000 7f7ec8075740 1 -- 172.21.15.28:0/3868540984 shutdown_connections 2024-04-25T01:38:03.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.715+0000 7f7ec8075740 1 --2- 172.21.15.28:0/3868540984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f691f1e570 0x55f691f20960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.715+0000 7f7ec8075740 1 --2- 172.21.15.28:0/3868540984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f13ea0 0x55f691f1e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.715+0000 7f7ec8075740 1 --2- 172.21.15.28:0/3868540984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 0x55f691f13960 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.715+0000 7f7ec8075740 1 -- 172.21.15.28:0/3868540984 >> 172.21.15.28:0/3868540984 conn(0x55f691f0f8a0 msgr2=0x55f691f0fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.715+0000 7f7ec8075740 1 -- 172.21.15.28:0/3868540984 shutdown_connections 2024-04-25T01:38:03.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.715+0000 7f7ec8075740 1 -- 172.21.15.28:0/3868540984 wait complete. 2024-04-25T01:38:03.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.715+0000 7f7ec8075740 1 -- start start 2024-04-25T01:38:03.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7ec8075740 1 Processor -- start 2024-04-25T01:38:03.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7ec8075740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f13580 0x55f691f1df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7ec8075740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f691f13ea0 0x55f691f1b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7ec8075740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f1e570 0x55f691f1b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7ec8075740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f691f21360 con 0x55f691f1e570 2024-04-25T01:38:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7ec8075740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f691e34050 con 0x55f691f13ea0 2024-04-25T01:38:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7ec8075740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55f691f214e0 con 0x55f691f13580 2024-04-25T01:38:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7ec8075740 10 client.0 _ll_register_callbacks cb 0x55f691b8bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7ebf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f1e570 0x55f691f1b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7ebf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f1e570 0x55f691f1b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42802/0 (socket says 172.21.15.28:42802) 2024-04-25T01:38:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7ebf7fe640 1 -- 172.21.15.28:0/2804465575 learned_addr learned my addr 172.21.15.28:0/2804465575 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7eb7fff640 1 --2- 172.21.15.28:0/2804465575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f691f13ea0 0x55f691f1b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7ebeffd640 1 --2- 172.21.15.28:0/2804465575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f13580 0x55f691f1df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7ebf7fe640 1 -- 172.21.15.28:0/2804465575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f13580 msgr2=0x55f691f1df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7ebf7fe640 1 --2- 172.21.15.28:0/2804465575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f13580 0x55f691f1df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7ebf7fe640 1 -- 172.21.15.28:0/2804465575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f691f13ea0 msgr2=0x55f691f1b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7ebf7fe640 1 --2- 172.21.15.28:0/2804465575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f691f13ea0 0x55f691f1b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7ebf7fe640 1 -- 172.21.15.28:0/2804465575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f691e05dc0 con 0x55f691f1e570 2024-04-25T01:38:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7ebf7fe640 1 --2- 172.21.15.28:0/2804465575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f1e570 0x55f691f1b7b0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8031040 tx=0x7f7eb8030640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff1649e824472e80 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.716+0000 7f7ea67fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.717+0000 7f7ea67fc640 1 -- 172.21.15.28:0/2804465575 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8058030 con 0x55f691f1e570 2024-04-25T01:38:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.717+0000 7f7ea67fc640 1 -- 172.21.15.28:0/2804465575 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f7eb8004040 con 0x55f691f1e570 2024-04-25T01:38:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.717+0000 7f7ec8075740 1 -- 172.21.15.28:0/2804465575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7eb8043070 con 0x55f691f1e570 2024-04-25T01:38:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.717+0000 7f7ea67fc640 1 -- 172.21.15.28:0/2804465575 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7eb8065050 con 0x55f691f1e570 2024-04-25T01:38:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.717+0000 7f7ec8075740 10 client.17166 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.717+0000 7f7ec8075740 1 -- 172.21.15.28:0/2804465575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f691b8c3f0 con 0x55f691f1e570 2024-04-25T01:38:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.717+0000 7f7ea67fc640 1 -- 172.21.15.28:0/2804465575 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7eb8043070 con 0x55f691f1e570 2024-04-25T01:38:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.717+0000 7f7ea67fc640 1 -- 172.21.15.28:0/2804465575 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7eb8030070 con 0x55f691f1e570 2024-04-25T01:38:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.717+0000 7f7ec8075740 10 client.17166 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.717+0000 7f7ec8075740 10 client.17166 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:38:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.717+0000 7f7ec8075740 20 client.17166 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.717+0000 7f7ec8075740 1 --2- 172.21.15.28:0/2804465575 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f691f2e490 0x55f691f30880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.717+0000 7f7ec8075740 4 client.17166 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.717+0000 7f7ec8075740 1 -- 172.21.15.28:0/2804465575 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f691e85bb0 con 0x55f691f2e490 2024-04-25T01:38:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.717+0000 7f7ebeffd640 1 --2- 172.21.15.28:0/2804465575 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f691f2e490 0x55f691f30880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.718+0000 7f7ebeffd640 1 --2- 172.21.15.28:0/2804465575 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f691f2e490 0x55f691f30880 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=726af6c14f32160b server_cookie=3253dd2ef66db382 in_seq=0 out_seq=0 2024-04-25T01:38:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.718+0000 7f7ea67fc640 10 client.17166 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:38:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.719+0000 7f7ea67fc640 1 -- 172.21.15.28:0/2804465575 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f691e85bb0 con 0x55f691f2e490 2024-04-25T01:38:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.719+0000 7f7ea67fc640 10 client.17166 handle_command_reply: tid=0 2024-04-25T01:38:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.719+0000 7f7ec8075740 1 client.17166 shutdown 2024-04-25T01:38:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.719+0000 7f7ec8075740 1 -- 172.21.15.28:0/2804465575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f1e570 msgr2=0x55f691f1b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.719+0000 7f7ec8075740 1 --2- 172.21.15.28:0/2804465575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f1e570 0x55f691f1b7b0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8031040 tx=0x7f7eb8030640 comp rx=0 tx=0).stop 2024-04-25T01:38:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.719+0000 7f7ec8075740 1 -- 172.21.15.28:0/2804465575 shutdown_connections 2024-04-25T01:38:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.719+0000 7f7ec8075740 1 --2- 172.21.15.28:0/2804465575 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f691f2e490 0x55f691f30880 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.719+0000 7f7ec8075740 1 --2- 172.21.15.28:0/2804465575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f1e570 0x55f691f1b7b0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.719+0000 7f7ec8075740 1 --2- 172.21.15.28:0/2804465575 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f691f13ea0 0x55f691f1b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.719+0000 7f7ea67fc640 0 client.17166 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:38:03.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.719+0000 7f7ec8075740 1 --2- 172.21.15.28:0/2804465575 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f13580 0x55f691f1df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.719+0000 7f7ec8075740 1 -- 172.21.15.28:0/2804465575 >> 172.21.15.28:0/2804465575 conn(0x55f691f0f8a0 msgr2=0x55f691f156a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.719+0000 7f7ec8075740 1 -- 172.21.15.28:0/2804465575 shutdown_connections 2024-04-25T01:38:03.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.719+0000 7f7ec8075740 1 -- 172.21.15.28:0/2804465575 wait complete. 2024-04-25T01:38:03.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.719+0000 7f7ec8075740 20 client.17166 trim_cache size 0 max 16384 2024-04-25T01:38:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.732+0000 7f7ec8075740 1 Processor -- start 2024-04-25T01:38:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.732+0000 7f7ec8075740 1 -- start start 2024-04-25T01:38:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.732+0000 7f7ec8075740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 0x55f691f1da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.732+0000 7f7ec8075740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f13ea0 0x55f691f165c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.733+0000 7f7ec8075740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f691f1e570 0x55f691f338a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.733+0000 7f7ec8075740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec01b0c70 con 0x55f691f13580 2024-04-25T01:38:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.733+0000 7f7ec8075740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec01b27b0 con 0x55f691f1e570 2024-04-25T01:38:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.733+0000 7f7ec8075740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec01b1340 con 0x55f691f13ea0 2024-04-25T01:38:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.733+0000 7f7ebeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 0x55f691f1da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.733+0000 7f7ebeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 0x55f691f1da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42808/0 (socket says 172.21.15.28:42808) 2024-04-25T01:38:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.733+0000 7f7ebeffd640 1 -- 172.21.15.28:0/816785668 learned_addr learned my addr 172.21.15.28:0/816785668 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.733+0000 7f7eb7fff640 1 --2- 172.21.15.28:0/816785668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f13ea0 0x55f691f165c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.733+0000 7f7ebeffd640 1 -- 172.21.15.28:0/816785668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f13ea0 msgr2=0x55f691f165c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.733+0000 7f7ebeffd640 1 --2- 172.21.15.28:0/816785668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f13ea0 0x55f691f165c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.733+0000 7f7ebeffd640 1 -- 172.21.15.28:0/816785668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f691f1e570 msgr2=0x55f691f338a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.733+0000 7f7ebeffd640 1 --2- 172.21.15.28:0/816785668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f691f1e570 0x55f691f338a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.733+0000 7f7ebeffd640 1 -- 172.21.15.28:0/816785668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f691c567d0 con 0x55f691f13580 2024-04-25T01:38:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.733+0000 7f7ebeffd640 1 --2- 172.21.15.28:0/816785668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 0x55f691f1da50 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f7eb000d740 tx=0x7f7eb0097040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b5937acef48f2e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.733+0000 7f7eb4ff9640 1 -- 172.21.15.28:0/816785668 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb0020040 con 0x55f691f13580 2024-04-25T01:38:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.733+0000 7f7eb4ff9640 1 -- 172.21.15.28:0/816785668 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7eb009bb40 con 0x55f691f13580 2024-04-25T01:38:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.733+0000 7f7eb4ff9640 1 -- 172.21.15.28:0/816785668 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb0099740 con 0x55f691f13580 2024-04-25T01:38:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.734+0000 7f7ec8075740 1 -- 172.21.15.28:0/816785668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 msgr2=0x55f691f1da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.734+0000 7f7ec8075740 1 --2- 172.21.15.28:0/816785668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 0x55f691f1da50 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f7eb000d740 tx=0x7f7eb0097040 comp rx=0 tx=0).stop 2024-04-25T01:38:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.734+0000 7f7ec8075740 1 -- 172.21.15.28:0/816785668 shutdown_connections 2024-04-25T01:38:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.734+0000 7f7ec8075740 1 --2- 172.21.15.28:0/816785668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f691f1e570 0x55f691f338a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.734+0000 7f7ec8075740 1 --2- 172.21.15.28:0/816785668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f13ea0 0x55f691f165c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.734+0000 7f7ec8075740 1 --2- 172.21.15.28:0/816785668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 0x55f691f1da50 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.734+0000 7f7ec8075740 1 -- 172.21.15.28:0/816785668 >> 172.21.15.28:0/816785668 conn(0x55f691f0f8a0 msgr2=0x55f691f15eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.734+0000 7f7ec8075740 1 -- 172.21.15.28:0/816785668 shutdown_connections 2024-04-25T01:38:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.734+0000 7f7ec8075740 1 -- 172.21.15.28:0/816785668 wait complete. 2024-04-25T01:38:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.734+0000 7f7ec8075740 1 -- start start 2024-04-25T01:38:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.734+0000 7f7ec8075740 1 Processor -- start 2024-04-25T01:38:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ec8075740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 0x55f691f33620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ec8075740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f691f13ea0 0x55f691f33b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ec8075740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f1e570 0x55f691f30b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ec8075740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec01b20e0 con 0x55f691f13580 2024-04-25T01:38:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ec8075740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec01b1a10 con 0x55f691f13ea0 2024-04-25T01:38:03.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ec8075740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec01aec50 con 0x55f691f1e570 2024-04-25T01:38:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ec8075740 10 client.0 _ll_register_callbacks cb 0x7f7eb804b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ebeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 0x55f691f33620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ebeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 0x55f691f33620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42820/0 (socket says 172.21.15.28:42820) 2024-04-25T01:38:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ebeffd640 1 -- 172.21.15.28:0/547502968 learned_addr learned my addr 172.21.15.28:0/547502968 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7eb7fff640 1 --2- 172.21.15.28:0/547502968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f691f13ea0 0x55f691f33b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ebf7fe640 1 --2- 172.21.15.28:0/547502968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f1e570 0x55f691f30b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ebeffd640 1 -- 172.21.15.28:0/547502968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f1e570 msgr2=0x55f691f30b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ebeffd640 1 --2- 172.21.15.28:0/547502968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f1e570 0x55f691f30b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ebeffd640 1 -- 172.21.15.28:0/547502968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f691f13ea0 msgr2=0x55f691f33b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ebeffd640 1 --2- 172.21.15.28:0/547502968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f691f13ea0 0x55f691f33b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ebeffd640 1 -- 172.21.15.28:0/547502968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f691f31260 con 0x55f691f13580 2024-04-25T01:38:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ebeffd640 1 --2- 172.21.15.28:0/547502968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 0x55f691f33620 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0097e10 tx=0x7f7eb009b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b813dba160b439 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ea67fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ea67fc640 1 -- 172.21.15.28:0/547502968 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb0084070 con 0x55f691f13580 2024-04-25T01:38:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ea67fc640 1 -- 172.21.15.28:0/547502968 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f7eb00cc8f0 con 0x55f691f13580 2024-04-25T01:38:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ec8075740 1 -- 172.21.15.28:0/547502968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7eb009bb40 con 0x55f691f13580 2024-04-25T01:38:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ea67fc640 1 -- 172.21.15.28:0/547502968 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7eb0020050 con 0x55f691f13580 2024-04-25T01:38:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ec8075740 10 client.17172 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.735+0000 7f7ec8075740 1 -- 172.21.15.28:0/547502968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f691f941c0 con 0x55f691f13580 2024-04-25T01:38:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.736+0000 7f7ea67fc640 1 -- 172.21.15.28:0/547502968 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7eb009bb40 con 0x55f691f13580 2024-04-25T01:38:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.736+0000 7f7ea67fc640 1 -- 172.21.15.28:0/547502968 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7eb009a300 con 0x55f691f13580 2024-04-25T01:38:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.736+0000 7f7ec8075740 10 client.17172 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.736+0000 7f7ec8075740 10 client.17172 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:38:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.736+0000 7f7ec8075740 20 client.17172 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.736+0000 7f7ec8075740 1 --2- 172.21.15.28:0/547502968 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f691f948b0 0x55f691f96cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.736+0000 7f7ec8075740 4 client.17172 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:38:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.736+0000 7f7ec8075740 1 -- 172.21.15.28:0/547502968 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f691e7ff00 con 0x55f691f948b0 2024-04-25T01:38:03.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.736+0000 7f7eb7fff640 1 --2- 172.21.15.28:0/547502968 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f691f948b0 0x55f691f96cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.736+0000 7f7ea67fc640 10 client.17172 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.736+0000 7f7eb7fff640 1 --2- 172.21.15.28:0/547502968 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f691f948b0 0x55f691f96cd0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b15e97e92d16e77a server_cookie=4509db6e00240536 in_seq=0 out_seq=0 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ea67fc640 1 -- 172.21.15.28:0/547502968 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f691e7ff00 con 0x55f691f948b0 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ea67fc640 10 client.17172 handle_command_reply: tid=0 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec8075740 1 client.17172 shutdown 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec8075740 1 -- 172.21.15.28:0/547502968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 msgr2=0x55f691f33620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec8075740 1 --2- 172.21.15.28:0/547502968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 0x55f691f33620 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0097e10 tx=0x7f7eb009b4b0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec8075740 1 -- 172.21.15.28:0/547502968 shutdown_connections 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec8075740 1 --2- 172.21.15.28:0/547502968 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f691f948b0 0x55f691f96cd0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec8075740 1 --2- 172.21.15.28:0/547502968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f691f1e570 0x55f691f30b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec8075740 1 --2- 172.21.15.28:0/547502968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f691f13ea0 0x55f691f33b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec8075740 1 --2- 172.21.15.28:0/547502968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f691f13580 0x55f691f33620 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ea67fc640 0 client.17172 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:38:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec8075740 1 -- 172.21.15.28:0/547502968 >> 172.21.15.28:0/547502968 conn(0x55f691f0f8a0 msgr2=0x55f691f1b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec8075740 1 -- 172.21.15.28:0/547502968 shutdown_connections 2024-04-25T01:38:03.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec8075740 1 -- 172.21.15.28:0/547502968 wait complete. 2024-04-25T01:38:03.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec8075740 20 client.17172 trim_cache size 0 max 16384 2024-04-25T01:38:03.723 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:38:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec5363640 1 -- 172.21.15.28:0/1977335584 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7eac0a2d00 msgr2=0x7f7eac0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec5363640 1 --2- 172.21.15.28:0/1977335584 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7eac0a2d00 0x7f7eac0a51c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7eb804a000 tx=0x7f7eb804d000 comp rx=0 tx=0).stop 2024-04-25T01:38:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec5363640 1 -- 172.21.15.28:0/1977335584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ec0100620 msgr2=0x7f7ec0196c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec5363640 1 --2- 172.21.15.28:0/1977335584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ec0100620 0x7f7ec0196c30 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f7eb000d770 tx=0x7f7eb000dc30 comp rx=0 tx=0).stop 2024-04-25T01:38:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec5363640 1 -- 172.21.15.28:0/1977335584 shutdown_connections 2024-04-25T01:38:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec5363640 1 --2- 172.21.15.28:0/1977335584 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7eac0a2d00 0x7f7eac0a51c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec5363640 1 --2- 172.21.15.28:0/1977335584 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ec0106630 0x7f7ec01976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec5363640 1 --2- 172.21.15.28:0/1977335584 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7ec0100fe0 0x7f7ec0197170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec5363640 1 --2- 172.21.15.28:0/1977335584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ec0100620 0x7f7ec0196c30 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.737+0000 7f7ec5363640 1 -- 172.21.15.28:0/1977335584 >> 172.21.15.28:0/1977335584 conn(0x7f7ec00776a0 msgr2=0x7f7ec01030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.738+0000 7f7ec5363640 1 -- 172.21.15.28:0/1977335584 shutdown_connections 2024-04-25T01:38:03.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.738+0000 7f7ec5363640 1 -- 172.21.15.28:0/1977335584 wait complete. 2024-04-25T01:38:03.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:38:03.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.853+0000 7f0672b1d640 1 -- 172.21.15.28:0/721722255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c069dc0 msgr2=0x7f066c06a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.853+0000 7f0672b1d640 1 --2- 172.21.15.28:0/721722255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c069dc0 0x7f066c06a240 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f0660009930 tx=0x7f066002a420 comp rx=0 tx=0).stop 2024-04-25T01:38:03.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.853+0000 7f0672b1d640 1 -- 172.21.15.28:0/721722255 shutdown_connections 2024-04-25T01:38:03.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.853+0000 7f0672b1d640 1 --2- 172.21.15.28:0/721722255 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f066c06a780 0x7f066c10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.853+0000 7f0672b1d640 1 --2- 172.21.15.28:0/721722255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c069dc0 0x7f066c06a240 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.853+0000 7f0672b1d640 1 --2- 172.21.15.28:0/721722255 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f066c107730 0x7f066c107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.853+0000 7f0672b1d640 1 -- 172.21.15.28:0/721722255 >> 172.21.15.28:0/721722255 conn(0x7f066c0776a0 msgr2=0x7f066c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.853+0000 7f0672b1d640 1 -- 172.21.15.28:0/721722255 shutdown_connections 2024-04-25T01:38:03.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.853+0000 7f0672b1d640 1 -- 172.21.15.28:0/721722255 wait complete. 2024-04-25T01:38:03.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f0672b1d640 1 Processor -- start 2024-04-25T01:38:03.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f0672b1d640 1 -- start start 2024-04-25T01:38:03.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f0672b1d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f066c069dc0 0x7f066c1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f0672b1d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c06a780 0x7f066c198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f0672b1d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f066c107730 0x7f066c19c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f0672b1d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f066c111b40 con 0x7f066c06a780 2024-04-25T01:38:03.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f0672b1d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f066c1119c0 con 0x7f066c107730 2024-04-25T01:38:03.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f0672b1d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f066c111cc0 con 0x7f066c069dc0 2024-04-25T01:38:03.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f066bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c06a780 0x7f066c198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f066bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c06a780 0x7f066c198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42836/0 (socket says 172.21.15.28:42836) 2024-04-25T01:38:03.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f066bfff640 1 -- 172.21.15.28:0/958049200 learned_addr learned my addr 172.21.15.28:0/958049200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f06710ba640 1 --2- 172.21.15.28:0/958049200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f066c107730 0x7f066c19c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f06708b9640 1 --2- 172.21.15.28:0/958049200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f066c069dc0 0x7f066c1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f066bfff640 1 -- 172.21.15.28:0/958049200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f066c069dc0 msgr2=0x7f066c1980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f066bfff640 1 --2- 172.21.15.28:0/958049200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f066c069dc0 0x7f066c1980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f066bfff640 1 -- 172.21.15.28:0/958049200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f066c107730 msgr2=0x7f066c19c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f066bfff640 1 --2- 172.21.15.28:0/958049200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f066c107730 0x7f066c19c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f066bfff640 1 -- 172.21.15.28:0/958049200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f066c19d060 con 0x7f066c06a780 2024-04-25T01:38:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f06708b9640 1 --2- 172.21.15.28:0/958049200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f066c069dc0 0x7f066c1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f06710ba640 1 --2- 172.21.15.28:0/958049200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f066c107730 0x7f066c19c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.854+0000 7f066bfff640 1 --2- 172.21.15.28:0/958049200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c06a780 0x7f066c198600 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f0660009a50 tx=0x7f0660004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.855+0000 7f0669ffb640 1 -- 172.21.15.28:0/958049200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066003f030 con 0x7f066c06a780 2024-04-25T01:38:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.855+0000 7f0669ffb640 1 -- 172.21.15.28:0/958049200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0660003ce0 con 0x7f066c06a780 2024-04-25T01:38:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.855+0000 7f0672b1d640 1 -- 172.21.15.28:0/958049200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f066c19d2f0 con 0x7f066c06a780 2024-04-25T01:38:03.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.855+0000 7f0669ffb640 1 -- 172.21.15.28:0/958049200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0660035070 con 0x7f066c06a780 2024-04-25T01:38:03.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.855+0000 7f0672b1d640 1 -- 172.21.15.28:0/958049200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f066c1a4be0 con 0x7f066c06a780 2024-04-25T01:38:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.857+0000 7f0669ffb640 1 -- 172.21.15.28:0/958049200 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0660042030 con 0x7f066c06a780 2024-04-25T01:38:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.857+0000 7f067581f740 1 Processor -- start 2024-04-25T01:38:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.857+0000 7f067581f740 1 -- start start 2024-04-25T01:38:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.857+0000 7f067581f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578259052b0 0x557825905690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.857+0000 7f067581f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825905c60 0x557825914440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.857+0000 7f067581f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557825914980 0x557825916d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.857+0000 7f067581f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55782586a670 con 0x557825905c60 2024-04-25T01:38:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.857+0000 7f067581f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557825812970 con 0x5578259052b0 2024-04-25T01:38:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.857+0000 7f067581f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x557825866630 con 0x557825914980 2024-04-25T01:38:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.858+0000 7f06708b9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825905c60 0x557825914440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.858+0000 7f06708b9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825905c60 0x557825914440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42844/0 (socket says 172.21.15.28:42844) 2024-04-25T01:38:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.858+0000 7f06708b9640 1 -- 172.21.15.28:0/547174144 learned_addr learned my addr 172.21.15.28:0/547174144 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.858+0000 7f066bfff640 1 --2- 172.21.15.28:0/547174144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557825914980 0x557825916d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.858+0000 7f06710ba640 1 --2- 172.21.15.28:0/547174144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578259052b0 0x557825905690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.858+0000 7f0669ffb640 1 --2- 172.21.15.28:0/958049200 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f06540a2d80 0x7f06540a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.858+0000 7f06708b9640 1 -- 172.21.15.28:0/547174144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557825914980 msgr2=0x557825916d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.858+0000 7f06708b9640 1 --2- 172.21.15.28:0/547174144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557825914980 0x557825916d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.858+0000 7f06708b9640 1 -- 172.21.15.28:0/547174144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578259052b0 msgr2=0x557825905690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.858+0000 7f06708b9640 1 --2- 172.21.15.28:0/547174144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578259052b0 0x557825905690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.858+0000 7f06708b9640 1 -- 172.21.15.28:0/547174144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557825836400 con 0x557825905c60 2024-04-25T01:38:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.858+0000 7f06710ba640 1 --2- 172.21.15.28:0/958049200 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f06540a2d80 0x7f06540a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.858+0000 7f0669ffb640 1 -- 172.21.15.28:0/958049200 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f06600efc50 con 0x7f066c06a780 2024-04-25T01:38:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.858+0000 7f06708b9640 1 --2- 172.21.15.28:0/547174144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825905c60 0x557825914440 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f065c00aa10 tx=0x7f065c00aed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d70a405867f901f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.858+0000 7f0652ffd640 1 -- 172.21.15.28:0/547174144 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065c015070 con 0x557825905c60 2024-04-25T01:38:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.858+0000 7f0652ffd640 1 -- 172.21.15.28:0/547174144 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f065c00cc60 con 0x557825905c60 2024-04-25T01:38:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.858+0000 7f0652ffd640 1 -- 172.21.15.28:0/547174144 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065c018690 con 0x557825905c60 2024-04-25T01:38:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.859+0000 7f067581f740 1 -- 172.21.15.28:0/547174144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825905c60 msgr2=0x557825914440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.859+0000 7f067581f740 1 --2- 172.21.15.28:0/547174144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825905c60 0x557825914440 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f065c00aa10 tx=0x7f065c00aed0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.859+0000 7f06710ba640 1 --2- 172.21.15.28:0/958049200 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f06540a2d80 0x7f06540a5240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0664007f10 tx=0x7f0664004a40 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.859+0000 7f067581f740 1 -- 172.21.15.28:0/547174144 shutdown_connections 2024-04-25T01:38:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.859+0000 7f067581f740 1 --2- 172.21.15.28:0/547174144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557825914980 0x557825916d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.859+0000 7f067581f740 1 --2- 172.21.15.28:0/547174144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825905c60 0x557825914440 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.859+0000 7f067581f740 1 --2- 172.21.15.28:0/547174144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5578259052b0 0x557825905690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.859+0000 7f067581f740 1 -- 172.21.15.28:0/547174144 >> 172.21.15.28:0/547174144 conn(0x5578259018a0 msgr2=0x557825901c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.859+0000 7f067581f740 1 -- 172.21.15.28:0/547174144 shutdown_connections 2024-04-25T01:38:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.859+0000 7f067581f740 1 -- 172.21.15.28:0/547174144 wait complete. 2024-04-25T01:38:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.859+0000 7f067581f740 1 -- start start 2024-04-25T01:38:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f067581f740 1 Processor -- start 2024-04-25T01:38:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f067581f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578259052b0 0x5578259142b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f067581f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557825905c60 0x5578259115a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f067581f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825914980 0x557825911ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f067581f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557825917740 con 0x557825914980 2024-04-25T01:38:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f067581f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557825826050 con 0x557825905c60 2024-04-25T01:38:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f067581f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5578259178c0 con 0x5578259052b0 2024-04-25T01:38:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f067581f740 10 client.0 _ll_register_callbacks cb 0x55782557dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f06710ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825914980 0x557825911ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f06710ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825914980 0x557825911ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42850/0 (socket says 172.21.15.28:42850) 2024-04-25T01:38:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f06710ba640 1 -- 172.21.15.28:0/35893030 learned_addr learned my addr 172.21.15.28:0/35893030 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f066bfff640 1 --2- 172.21.15.28:0/35893030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557825905c60 0x5578259115a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f06708b9640 1 --2- 172.21.15.28:0/35893030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578259052b0 0x5578259142b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f06710ba640 1 -- 172.21.15.28:0/35893030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578259052b0 msgr2=0x5578259142b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f06710ba640 1 --2- 172.21.15.28:0/35893030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578259052b0 0x5578259142b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f06710ba640 1 -- 172.21.15.28:0/35893030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557825905c60 msgr2=0x5578259115a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f06710ba640 1 --2- 172.21.15.28:0/35893030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557825905c60 0x5578259115a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f06710ba640 1 -- 172.21.15.28:0/35893030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578257f7dc0 con 0x557825914980 2024-04-25T01:38:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f06710ba640 1 --2- 172.21.15.28:0/35893030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825914980 0x557825911ae0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f0664012a80 tx=0x7f0664007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=142bbf9c37b2cf15 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.860+0000 7f064bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.861+0000 7f064bfff640 1 -- 172.21.15.28:0/35893030 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066401de40 con 0x557825914980 2024-04-25T01:38:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.861+0000 7f064bfff640 1 -- 172.21.15.28:0/35893030 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f066401b450 con 0x557825914980 2024-04-25T01:38:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.861+0000 7f067581f740 1 -- 172.21.15.28:0/35893030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f065c00cc60 con 0x557825914980 2024-04-25T01:38:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.861+0000 7f064bfff640 1 -- 172.21.15.28:0/35893030 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f066401b650 con 0x557825914980 2024-04-25T01:38:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.861+0000 7f067581f740 10 client.17184 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.861+0000 7f067581f740 1 -- 172.21.15.28:0/35893030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55782557e3f0 con 0x557825914980 2024-04-25T01:38:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.861+0000 7f064bfff640 1 -- 172.21.15.28:0/35893030 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f065c00cc60 con 0x557825914980 2024-04-25T01:38:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.861+0000 7f064bfff640 1 -- 172.21.15.28:0/35893030 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0664024070 con 0x557825914980 2024-04-25T01:38:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.861+0000 7f067581f740 10 client.17184 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.861+0000 7f067581f740 10 client.17184 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:38:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.861+0000 7f067581f740 20 client.17184 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.861+0000 7f067581f740 1 --2- 172.21.15.28:0/35893030 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x557825924a20 0x557825926ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.861+0000 7f067581f740 4 client.17184 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.861+0000 7f067581f740 1 -- 172.21.15.28:0/35893030 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557825877bb0 con 0x557825924a20 2024-04-25T01:38:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.862+0000 7f06708b9640 1 --2- 172.21.15.28:0/35893030 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x557825924a20 0x557825926ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.862+0000 7f06708b9640 1 --2- 172.21.15.28:0/35893030 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x557825924a20 0x557825926ea0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1e902efd9db9f06d server_cookie=73b5605742bf729 in_seq=0 out_seq=0 2024-04-25T01:38:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.862+0000 7f064bfff640 10 client.17184 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:38:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.863+0000 7f064bfff640 1 -- 172.21.15.28:0/35893030 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557825877bb0 con 0x557825924a20 2024-04-25T01:38:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.863+0000 7f064bfff640 10 client.17184 handle_command_reply: tid=0 2024-04-25T01:38:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.863+0000 7f067581f740 1 client.17184 shutdown 2024-04-25T01:38:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.863+0000 7f067581f740 1 -- 172.21.15.28:0/35893030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825914980 msgr2=0x557825911ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.863+0000 7f067581f740 1 --2- 172.21.15.28:0/35893030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825914980 0x557825911ae0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f0664012a80 tx=0x7f0664007590 comp rx=0 tx=0).stop 2024-04-25T01:38:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.863+0000 7f067581f740 1 -- 172.21.15.28:0/35893030 shutdown_connections 2024-04-25T01:38:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.863+0000 7f067581f740 1 --2- 172.21.15.28:0/35893030 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x557825924a20 0x557825926ea0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.863+0000 7f067581f740 1 --2- 172.21.15.28:0/35893030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825914980 0x557825911ae0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.863+0000 7f067581f740 1 --2- 172.21.15.28:0/35893030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557825905c60 0x5578259115a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.863+0000 7f067581f740 1 --2- 172.21.15.28:0/35893030 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578259052b0 0x5578259142b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.863+0000 7f067581f740 1 -- 172.21.15.28:0/35893030 >> 172.21.15.28:0/35893030 conn(0x5578259018a0 msgr2=0x5578259076f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.863+0000 7f064bfff640 0 client.17184 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:38:03.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.863+0000 7f067581f740 1 -- 172.21.15.28:0/35893030 shutdown_connections 2024-04-25T01:38:03.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.863+0000 7f067581f740 1 -- 172.21.15.28:0/35893030 wait complete. 2024-04-25T01:38:03.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.863+0000 7f067581f740 20 client.17184 trim_cache size 0 max 16384 2024-04-25T01:38:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.876+0000 7f067581f740 1 Processor -- start 2024-04-25T01:38:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.876+0000 7f067581f740 1 -- start start 2024-04-25T01:38:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.877+0000 7f067581f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578259052b0 0x55782590cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.877+0000 7f067581f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557825905c60 0x557825907570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.877+0000 7f067581f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557825914980 0x557825913de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.877+0000 7f067581f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f066c1ac840 con 0x5578259052b0 2024-04-25T01:38:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.877+0000 7f067581f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f066c1ae380 con 0x557825905c60 2024-04-25T01:38:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.877+0000 7f067581f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f066c1acf10 con 0x557825914980 2024-04-25T01:38:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.877+0000 7f06708b9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578259052b0 0x55782590cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.877+0000 7f06708b9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578259052b0 0x55782590cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42852/0 (socket says 172.21.15.28:42852) 2024-04-25T01:38:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.877+0000 7f06708b9640 1 -- 172.21.15.28:0/2118482633 learned_addr learned my addr 172.21.15.28:0/2118482633 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.877+0000 7f066bfff640 1 --2- 172.21.15.28:0/2118482633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557825905c60 0x557825907570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.877+0000 7f06708b9640 1 -- 172.21.15.28:0/2118482633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557825914980 msgr2=0x557825913de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.877+0000 7f06708b9640 1 --2- 172.21.15.28:0/2118482633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557825914980 0x557825913de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.877+0000 7f06708b9640 1 -- 172.21.15.28:0/2118482633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557825905c60 msgr2=0x557825907570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.877+0000 7f06708b9640 1 --2- 172.21.15.28:0/2118482633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557825905c60 0x557825907570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.877+0000 7f06708b9640 1 -- 172.21.15.28:0/2118482633 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578256487d0 con 0x5578259052b0 2024-04-25T01:38:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.877+0000 7f06708b9640 1 --2- 172.21.15.28:0/2118482633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578259052b0 0x55782590cc70 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f065c00ca80 tx=0x7f065c013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21da2e6c6f061bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.877+0000 7f06527fc640 1 -- 172.21.15.28:0/2118482633 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065c015040 con 0x5578259052b0 2024-04-25T01:38:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.877+0000 7f06527fc640 1 -- 172.21.15.28:0/2118482633 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f065c00cdb0 con 0x5578259052b0 2024-04-25T01:38:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.877+0000 7f06527fc640 1 -- 172.21.15.28:0/2118482633 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065c027690 con 0x5578259052b0 2024-04-25T01:38:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.878+0000 7f067581f740 1 -- 172.21.15.28:0/2118482633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578259052b0 msgr2=0x55782590cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.878+0000 7f067581f740 1 --2- 172.21.15.28:0/2118482633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578259052b0 0x55782590cc70 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f065c00ca80 tx=0x7f065c013040 comp rx=0 tx=0).stop 2024-04-25T01:38:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.878+0000 7f067581f740 1 -- 172.21.15.28:0/2118482633 shutdown_connections 2024-04-25T01:38:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.878+0000 7f067581f740 1 --2- 172.21.15.28:0/2118482633 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557825914980 0x557825913de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.878+0000 7f067581f740 1 --2- 172.21.15.28:0/2118482633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557825905c60 0x557825907570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.878+0000 7f067581f740 1 --2- 172.21.15.28:0/2118482633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578259052b0 0x55782590cc70 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.878+0000 7f067581f740 1 -- 172.21.15.28:0/2118482633 >> 172.21.15.28:0/2118482633 conn(0x5578259018a0 msgr2=0x557825906e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.878+0000 7f067581f740 1 -- 172.21.15.28:0/2118482633 shutdown_connections 2024-04-25T01:38:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.878+0000 7f067581f740 1 -- 172.21.15.28:0/2118482633 wait complete. 2024-04-25T01:38:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.878+0000 7f067581f740 1 -- start start 2024-04-25T01:38:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.878+0000 7f067581f740 1 Processor -- start 2024-04-25T01:38:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f067581f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578259052b0 0x55782591cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f067581f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557825905c60 0x55782591d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f067581f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825914980 0x557825918140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f067581f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f066c1adcb0 con 0x557825914980 2024-04-25T01:38:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f067581f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f066c1ad5e0 con 0x557825905c60 2024-04-25T01:38:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f067581f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f066c1aa820 con 0x5578259052b0 2024-04-25T01:38:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f067581f740 10 client.0 _ll_register_callbacks cb 0x7f0664010ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f06710ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825914980 0x557825918140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f06710ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825914980 0x557825918140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42864/0 (socket says 172.21.15.28:42864) 2024-04-25T01:38:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f066bfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557825905c60 0x55782591d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f06710ba640 1 -- 172.21.15.28:0/2589288390 learned_addr learned my addr 172.21.15.28:0/2589288390 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f06708b9640 1 --2- 172.21.15.28:0/2589288390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578259052b0 0x55782591cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f06710ba640 1 -- 172.21.15.28:0/2589288390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578259052b0 msgr2=0x55782591cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f06710ba640 1 --2- 172.21.15.28:0/2589288390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578259052b0 0x55782591cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f06710ba640 1 -- 172.21.15.28:0/2589288390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557825905c60 msgr2=0x55782591d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f06710ba640 1 --2- 172.21.15.28:0/2589288390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557825905c60 0x55782591d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f06710ba640 1 -- 172.21.15.28:0/2589288390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557825918970 con 0x557825914980 2024-04-25T01:38:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f06710ba640 1 --2- 172.21.15.28:0/2589288390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825914980 0x557825918140 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f0664026ca0 tx=0x7f0664007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1720b1226a53300 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f064bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f064bfff640 1 -- 172.21.15.28:0/2589288390 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0664038070 con 0x557825914980 2024-04-25T01:38:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f064bfff640 1 -- 172.21.15.28:0/2589288390 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f0664004ab0 con 0x557825914980 2024-04-25T01:38:03.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.879+0000 7f067581f740 1 -- 172.21.15.28:0/2589288390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f065c00cdb0 con 0x557825914980 2024-04-25T01:38:03.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.880+0000 7f064bfff640 1 -- 172.21.15.28:0/2589288390 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0664040050 con 0x557825914980 2024-04-25T01:38:03.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.880+0000 7f067581f740 10 client.17190 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:03.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.880+0000 7f067581f740 1 -- 172.21.15.28:0/2589288390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55782598ac70 con 0x557825914980 2024-04-25T01:38:03.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.880+0000 7f064bfff640 1 -- 172.21.15.28:0/2589288390 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f065c00cdb0 con 0x557825914980 2024-04-25T01:38:03.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.880+0000 7f064bfff640 1 -- 172.21.15.28:0/2589288390 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f066403b030 con 0x557825914980 2024-04-25T01:38:03.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.880+0000 7f067581f740 10 client.17190 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:03.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.880+0000 7f067581f740 10 client.17190 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:38:03.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.880+0000 7f067581f740 20 client.17190 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:03.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.880+0000 7f067581f740 1 --2- 172.21.15.28:0/2589288390 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55782598b3e0 0x55782598d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.880+0000 7f067581f740 4 client.17190 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:38:03.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.880+0000 7f067581f740 1 -- 172.21.15.28:0/2589288390 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557825871f00 con 0x55782598b3e0 2024-04-25T01:38:03.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.880+0000 7f06708b9640 1 --2- 172.21.15.28:0/2589288390 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55782598b3e0 0x55782598d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.881+0000 7f064bfff640 10 client.17190 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:38:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.881+0000 7f06708b9640 1 --2- 172.21.15.28:0/2589288390 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55782598b3e0 0x55782598d800 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2643f8cb57f6763a server_cookie=2d8d4334743b450 in_seq=0 out_seq=0 2024-04-25T01:38:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.881+0000 7f064bfff640 1 -- 172.21.15.28:0/2589288390 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557825871f00 con 0x55782598b3e0 2024-04-25T01:38:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.881+0000 7f064bfff640 10 client.17190 handle_command_reply: tid=0 2024-04-25T01:38:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.881+0000 7f067581f740 1 client.17190 shutdown 2024-04-25T01:38:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.881+0000 7f067581f740 1 -- 172.21.15.28:0/2589288390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825914980 msgr2=0x557825918140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.881+0000 7f067581f740 1 --2- 172.21.15.28:0/2589288390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825914980 0x557825918140 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f0664026ca0 tx=0x7f0664007430 comp rx=0 tx=0).stop 2024-04-25T01:38:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.881+0000 7f067581f740 1 -- 172.21.15.28:0/2589288390 shutdown_connections 2024-04-25T01:38:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.881+0000 7f067581f740 1 --2- 172.21.15.28:0/2589288390 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55782598b3e0 0x55782598d800 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.881+0000 7f067581f740 1 --2- 172.21.15.28:0/2589288390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557825914980 0x557825918140 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.881+0000 7f067581f740 1 --2- 172.21.15.28:0/2589288390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557825905c60 0x55782591d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.881+0000 7f064bfff640 0 client.17190 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:38:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.881+0000 7f067581f740 1 --2- 172.21.15.28:0/2589288390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5578259052b0 0x55782591cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.881+0000 7f067581f740 1 -- 172.21.15.28:0/2589288390 >> 172.21.15.28:0/2589288390 conn(0x5578259018a0 msgr2=0x5578259076f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.881+0000 7f067581f740 1 -- 172.21.15.28:0/2589288390 shutdown_connections 2024-04-25T01:38:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.881+0000 7f067581f740 1 -- 172.21.15.28:0/2589288390 wait complete. 2024-04-25T01:38:03.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.881+0000 7f067581f740 20 client.17190 trim_cache size 0 max 16384 2024-04-25T01:38:03.868 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:38:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.882+0000 7f0672b1d640 1 -- 172.21.15.28:0/958049200 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f06540a2d80 msgr2=0x7f06540a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.882+0000 7f0672b1d640 1 --2- 172.21.15.28:0/958049200 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f06540a2d80 0x7f06540a5240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0664007f10 tx=0x7f0664004a40 comp rx=0 tx=0).stop 2024-04-25T01:38:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.882+0000 7f0672b1d640 1 -- 172.21.15.28:0/958049200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c06a780 msgr2=0x7f066c198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.882+0000 7f0672b1d640 1 --2- 172.21.15.28:0/958049200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c06a780 0x7f066c198600 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f0660009a50 tx=0x7f0660004270 comp rx=0 tx=0).stop 2024-04-25T01:38:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.882+0000 7f0672b1d640 1 -- 172.21.15.28:0/958049200 shutdown_connections 2024-04-25T01:38:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.882+0000 7f0672b1d640 1 --2- 172.21.15.28:0/958049200 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f06540a2d80 0x7f06540a5240 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.882+0000 7f0672b1d640 1 --2- 172.21.15.28:0/958049200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f066c107730 0x7f066c19c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.882+0000 7f0672b1d640 1 --2- 172.21.15.28:0/958049200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066c06a780 0x7f066c198600 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.882+0000 7f0672b1d640 1 --2- 172.21.15.28:0/958049200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f066c069dc0 0x7f066c1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.882+0000 7f0672b1d640 1 -- 172.21.15.28:0/958049200 >> 172.21.15.28:0/958049200 conn(0x7f066c0776a0 msgr2=0x7f066c0fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.882+0000 7f0672b1d640 1 -- 172.21.15.28:0/958049200 shutdown_connections 2024-04-25T01:38:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.882+0000 7f0672b1d640 1 -- 172.21.15.28:0/958049200 wait complete. 2024-04-25T01:38:03.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:03 smithi028 ceph-mon[29390]: pgmap v154: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:38:03.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:03 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/572610936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:38:03.872 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:38:03.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.997+0000 7f684a509640 1 -- 172.21.15.28:0/579329302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440ffc40 msgr2=0x7f68441000c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.997+0000 7f684a509640 1 --2- 172.21.15.28:0/579329302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440ffc40 0x7f68441000c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f6838009930 tx=0x7f683802a420 comp rx=0 tx=0).stop 2024-04-25T01:38:03.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.997+0000 7f684a509640 1 -- 172.21.15.28:0/579329302 shutdown_connections 2024-04-25T01:38:03.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.997+0000 7f684a509640 1 --2- 172.21.15.28:0/579329302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6844100600 0x7f684410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.997+0000 7f684a509640 1 --2- 172.21.15.28:0/579329302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440ffc40 0x7f68441000c0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.997+0000 7f684a509640 1 --2- 172.21.15.28:0/579329302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6844107720 0x7f6844107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.997+0000 7f684a509640 1 -- 172.21.15.28:0/579329302 >> 172.21.15.28:0/579329302 conn(0x7f6844076bd0 msgr2=0x7f68440fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.997+0000 7f684a509640 1 -- 172.21.15.28:0/579329302 shutdown_connections 2024-04-25T01:38:03.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.997+0000 7f684a509640 1 -- 172.21.15.28:0/579329302 wait complete. 2024-04-25T01:38:03.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f684a509640 1 Processor -- start 2024-04-25T01:38:03.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f684a509640 1 -- start start 2024-04-25T01:38:03.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f684a509640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440ffc40 0x7f6844197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f684a509640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6844100600 0x7f68441984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f684a509640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6844107720 0x7f684419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f684a509640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6844111b30 con 0x7f68440ffc40 2024-04-25T01:38:03.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f684a509640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f68441119b0 con 0x7f6844107720 2024-04-25T01:38:03.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f684a509640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6844111cb0 con 0x7f6844100600 2024-04-25T01:38:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f6843fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440ffc40 0x7f6844197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f6843fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440ffc40 0x7f6844197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42890/0 (socket says 172.21.15.28:42890) 2024-04-25T01:38:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f6843fff640 1 -- 172.21.15.28:0/246153372 learned_addr learned my addr 172.21.15.28:0/246153372 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f68437fe640 1 --2- 172.21.15.28:0/246153372 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6844100600 0x7f68441984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f6848aa6640 1 --2- 172.21.15.28:0/246153372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6844107720 0x7f684419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f6843fff640 1 -- 172.21.15.28:0/246153372 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6844100600 msgr2=0x7f68441984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f6843fff640 1 --2- 172.21.15.28:0/246153372 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6844100600 0x7f68441984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f6843fff640 1 -- 172.21.15.28:0/246153372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6844107720 msgr2=0x7f684419c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f6843fff640 1 --2- 172.21.15.28:0/246153372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6844107720 0x7f684419c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f6843fff640 1 -- 172.21.15.28:0/246153372 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f684419cfa0 con 0x7f68440ffc40 2024-04-25T01:38:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f6848aa6640 1 --2- 172.21.15.28:0/246153372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6844107720 0x7f684419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.998+0000 7f68437fe640 1 --2- 172.21.15.28:0/246153372 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6844100600 0x7f68441984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.999+0000 7f6843fff640 1 --2- 172.21.15.28:0/246153372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440ffc40 0x7f6844197fb0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f683400c920 tx=0x7f683400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.999+0000 7f68417fa640 1 -- 172.21.15.28:0/246153372 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6834007c70 con 0x7f68440ffc40 2024-04-25T01:38:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.999+0000 7f684a509640 1 -- 172.21.15.28:0/246153372 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f684419d290 con 0x7f68440ffc40 2024-04-25T01:38:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.999+0000 7f68417fa640 1 -- 172.21.15.28:0/246153372 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6834007e10 con 0x7f68440ffc40 2024-04-25T01:38:03.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.999+0000 7f68417fa640 1 -- 172.21.15.28:0/246153372 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6834018890 con 0x7f68440ffc40 2024-04-25T01:38:03.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:03.999+0000 7f684a509640 1 -- 172.21.15.28:0/246153372 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68441a4b20 con 0x7f68440ffc40 2024-04-25T01:38:03.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.002+0000 7f68417fa640 1 -- 172.21.15.28:0/246153372 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6834018a70 con 0x7f68440ffc40 2024-04-25T01:38:03.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.002+0000 7f684d20e740 1 Processor -- start 2024-04-25T01:38:03.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.002+0000 7f684d20e740 1 -- start start 2024-04-25T01:38:03.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.002+0000 7f684d20e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 0x562a47ead960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.002+0000 7f684d20e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 0x562a47eb8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.002+0000 7f684d20e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47eb8570 0x562a47eba960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.002+0000 7f684d20e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562a47e12670 con 0x562a47ead580 2024-04-25T01:38:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.002+0000 7f684d20e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x562a47dba970 con 0x562a47eadea0 2024-04-25T01:38:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.002+0000 7f684d20e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x562a47e0e630 con 0x562a47eb8570 2024-04-25T01:38:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.002+0000 7f6848aa6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 0x562a47ead960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.002+0000 7f6848aa6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 0x562a47ead960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42904/0 (socket says 172.21.15.28:42904) 2024-04-25T01:38:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.002+0000 7f6848aa6640 1 -- 172.21.15.28:0/1667240029 learned_addr learned my addr 172.21.15.28:0/1667240029 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.002+0000 7f6843fff640 1 --2- 172.21.15.28:0/1667240029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 0x562a47eb8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.002+0000 7f68417fa640 1 --2- 172.21.15.28:0/246153372 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f682c0a2d00 0x7f682c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.002+0000 7f68437fe640 1 --2- 172.21.15.28:0/1667240029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47eb8570 0x562a47eba960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.002+0000 7f6848aa6640 1 -- 172.21.15.28:0/1667240029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47eb8570 msgr2=0x562a47eba960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.002+0000 7f6848aa6640 1 --2- 172.21.15.28:0/1667240029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47eb8570 0x562a47eba960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.002+0000 7f6848aa6640 1 -- 172.21.15.28:0/1667240029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 msgr2=0x562a47eb8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.002+0000 7f6848aa6640 1 --2- 172.21.15.28:0/1667240029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 0x562a47eb8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.003+0000 7f6848aa6640 1 -- 172.21.15.28:0/1667240029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a47dde400 con 0x562a47ead580 2024-04-25T01:38:03.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.003+0000 7f68417fa640 1 -- 172.21.15.28:0/246153372 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f68340cf910 con 0x7f68440ffc40 2024-04-25T01:38:03.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.003+0000 7f6848aa6640 1 --2- 172.21.15.28:0/246153372 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f682c0a2d00 0x7f682c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.003+0000 7f6848aa6640 1 --2- 172.21.15.28:0/1667240029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 0x562a47ead960 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f683c00f390 tx=0x7f683c00f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be4347f36d47373d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.003+0000 7f682a7fc640 1 -- 172.21.15.28:0/1667240029 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c01a610 con 0x562a47ead580 2024-04-25T01:38:03.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.003+0000 7f682a7fc640 1 -- 172.21.15.28:0/1667240029 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f683c01ac90 con 0x562a47ead580 2024-04-25T01:38:03.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.003+0000 7f682a7fc640 1 -- 172.21.15.28:0/1667240029 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c010b20 con 0x562a47ead580 2024-04-25T01:38:03.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.004+0000 7f684d20e740 1 -- 172.21.15.28:0/1667240029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 msgr2=0x562a47ead960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.004+0000 7f6848aa6640 1 --2- 172.21.15.28:0/246153372 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f682c0a2d00 0x7f682c0a51c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f683c020000 tx=0x7f683c01d000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.004+0000 7f684d20e740 1 --2- 172.21.15.28:0/1667240029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 0x562a47ead960 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f683c00f390 tx=0x7f683c00f850 comp rx=0 tx=0).stop 2024-04-25T01:38:03.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.004+0000 7f684d20e740 1 -- 172.21.15.28:0/1667240029 shutdown_connections 2024-04-25T01:38:03.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.004+0000 7f684d20e740 1 --2- 172.21.15.28:0/1667240029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47eb8570 0x562a47eba960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.004+0000 7f684d20e740 1 --2- 172.21.15.28:0/1667240029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 0x562a47eb8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.004+0000 7f684d20e740 1 --2- 172.21.15.28:0/1667240029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 0x562a47ead960 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.004+0000 7f684d20e740 1 -- 172.21.15.28:0/1667240029 >> 172.21.15.28:0/1667240029 conn(0x562a47ea98a0 msgr2=0x562a47ea9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.004+0000 7f684d20e740 1 -- 172.21.15.28:0/1667240029 shutdown_connections 2024-04-25T01:38:03.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.004+0000 7f684d20e740 1 -- 172.21.15.28:0/1667240029 wait complete. 2024-04-25T01:38:03.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.004+0000 7f684d20e740 1 -- start start 2024-04-25T01:38:03.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.004+0000 7f684d20e740 1 Processor -- start 2024-04-25T01:38:03.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.004+0000 7f684d20e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 0x562a47eb7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f684d20e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 0x562a47eb5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f684d20e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47eb8570 0x562a47eb56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f684d20e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562a47ebb360 con 0x562a47ead580 2024-04-25T01:38:03.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f684d20e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x562a47dce050 con 0x562a47eadea0 2024-04-25T01:38:03.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f684d20e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x562a47ebb4e0 con 0x562a47eb8570 2024-04-25T01:38:03.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f684d20e740 10 client.0 _ll_register_callbacks cb 0x562a47b25be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:03.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f6843fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 0x562a47eb7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f6843fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 0x562a47eb7ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42918/0 (socket says 172.21.15.28:42918) 2024-04-25T01:38:03.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f6843fff640 1 -- 172.21.15.28:0/344885605 learned_addr learned my addr 172.21.15.28:0/344885605 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:03.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f6848aa6640 1 --2- 172.21.15.28:0/344885605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47eb8570 0x562a47eb56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f68437fe640 1 --2- 172.21.15.28:0/344885605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 0x562a47eb5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f6843fff640 1 -- 172.21.15.28:0/344885605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47eb8570 msgr2=0x562a47eb56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f6843fff640 1 --2- 172.21.15.28:0/344885605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47eb8570 0x562a47eb56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f6843fff640 1 -- 172.21.15.28:0/344885605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 msgr2=0x562a47eb5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:03.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f6843fff640 1 --2- 172.21.15.28:0/344885605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 0x562a47eb5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f6843fff640 1 -- 172.21.15.28:0/344885605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a47d9fdc0 con 0x562a47ead580 2024-04-25T01:38:03.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f6843fff640 1 --2- 172.21.15.28:0/344885605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 0x562a47eb7ea0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f6834007590 tx=0x7f683409b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56cc543fa668f20 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:03.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f68237fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:03.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f68237fe640 1 -- 172.21.15.28:0/344885605 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683409baf0 con 0x562a47ead580 2024-04-25T01:38:03.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f68237fe640 1 -- 172.21.15.28:0/344885605 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f683400fb30 con 0x562a47ead580 2024-04-25T01:38:03.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f684d20e740 1 -- 172.21.15.28:0/344885605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f683c01ac90 con 0x562a47ead580 2024-04-25T01:38:03.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f68237fe640 1 -- 172.21.15.28:0/344885605 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f683409fea0 con 0x562a47ead580 2024-04-25T01:38:03.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f684d20e740 10 client.17202 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.005+0000 7f684d20e740 1 -- 172.21.15.28:0/344885605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562a47b263f0 con 0x562a47ead580 2024-04-25T01:38:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.006+0000 7f68237fe640 1 -- 172.21.15.28:0/344885605 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f683c01ac90 con 0x562a47ead580 2024-04-25T01:38:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.006+0000 7f68237fe640 1 -- 172.21.15.28:0/344885605 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f68340ba2b0 con 0x562a47ead580 2024-04-25T01:38:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.006+0000 7f684d20e740 10 client.17202 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.006+0000 7f684d20e740 10 client.17202 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:38:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.006+0000 7f684d20e740 20 client.17202 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.006+0000 7f684d20e740 1 --2- 172.21.15.28:0/344885605 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x562a47ec85a0 0x562a47eca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.006+0000 7f684d20e740 4 client.17202 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.006+0000 7f684d20e740 1 -- 172.21.15.28:0/344885605 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a47e1fbb0 con 0x562a47ec85a0 2024-04-25T01:38:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.006+0000 7f68437fe640 1 --2- 172.21.15.28:0/344885605 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x562a47ec85a0 0x562a47eca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.007+0000 7f68237fe640 10 client.17202 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:38:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.007+0000 7f68437fe640 1 --2- 172.21.15.28:0/344885605 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x562a47ec85a0 0x562a47eca9f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ff520c98ca2e8c4 server_cookie=a60842cbb89428d2 in_seq=0 out_seq=0 2024-04-25T01:38:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.007+0000 7f68237fe640 1 -- 172.21.15.28:0/344885605 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562a47e1fbb0 con 0x562a47ec85a0 2024-04-25T01:38:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.007+0000 7f68237fe640 10 client.17202 handle_command_reply: tid=0 2024-04-25T01:38:03.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.007+0000 7f684d20e740 1 client.17202 shutdown 2024-04-25T01:38:03.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.008+0000 7f684d20e740 1 -- 172.21.15.28:0/344885605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 msgr2=0x562a47eb7ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:03.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.008+0000 7f684d20e740 1 --2- 172.21.15.28:0/344885605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 0x562a47eb7ea0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f6834007590 tx=0x7f683409b730 comp rx=0 tx=0).stop 2024-04-25T01:38:03.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.008+0000 7f684d20e740 1 -- 172.21.15.28:0/344885605 shutdown_connections 2024-04-25T01:38:03.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.008+0000 7f684d20e740 1 --2- 172.21.15.28:0/344885605 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x562a47ec85a0 0x562a47eca9f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.008+0000 7f684d20e740 1 --2- 172.21.15.28:0/344885605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47eb8570 0x562a47eb56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.008+0000 7f684d20e740 1 --2- 172.21.15.28:0/344885605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 0x562a47eb5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.008+0000 7f68237fe640 0 client.17202 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:38:03.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.008+0000 7f684d20e740 1 --2- 172.21.15.28:0/344885605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 0x562a47eb7ea0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:03.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.008+0000 7f684d20e740 1 -- 172.21.15.28:0/344885605 >> 172.21.15.28:0/344885605 conn(0x562a47ea98a0 msgr2=0x562a47ea8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:03.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.008+0000 7f684d20e740 1 -- 172.21.15.28:0/344885605 shutdown_connections 2024-04-25T01:38:03.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.008+0000 7f684d20e740 1 -- 172.21.15.28:0/344885605 wait complete. 2024-04-25T01:38:03.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.008+0000 7f684d20e740 20 client.17202 trim_cache size 0 max 16384 2024-04-25T01:38:04.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.021+0000 7f684d20e740 1 Processor -- start 2024-04-25T01:38:04.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.021+0000 7f684d20e740 1 -- start start 2024-04-25T01:38:04.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.021+0000 7f684d20e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47ead580 0x562a47ec4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.021+0000 7f684d20e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 0x562a47eb79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.021+0000 7f684d20e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47eb8570 0x562a47ed0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.021+0000 7f684d20e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f68441aea30 con 0x562a47eb8570 2024-04-25T01:38:04.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.021+0000 7f684d20e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f68441b0570 con 0x562a47eadea0 2024-04-25T01:38:04.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.021+0000 7f684d20e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f68441af100 con 0x562a47ead580 2024-04-25T01:38:04.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.021+0000 7f6848aa6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47eb8570 0x562a47ed0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.021+0000 7f6848aa6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47eb8570 0x562a47ed0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42930/0 (socket says 172.21.15.28:42930) 2024-04-25T01:38:04.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.021+0000 7f6848aa6640 1 -- 172.21.15.28:0/2427517115 learned_addr learned my addr 172.21.15.28:0/2427517115 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:04.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.021+0000 7f68437fe640 1 --2- 172.21.15.28:0/2427517115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 0x562a47eb79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.021+0000 7f6843fff640 1 --2- 172.21.15.28:0/2427517115 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47ead580 0x562a47ec4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.022+0000 7f6848aa6640 1 -- 172.21.15.28:0/2427517115 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47ead580 msgr2=0x562a47ec4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.022+0000 7f6848aa6640 1 --2- 172.21.15.28:0/2427517115 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47ead580 0x562a47ec4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.022+0000 7f6848aa6640 1 -- 172.21.15.28:0/2427517115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 msgr2=0x562a47eb79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.022+0000 7f6848aa6640 1 --2- 172.21.15.28:0/2427517115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 0x562a47eb79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.022+0000 7f6848aa6640 1 -- 172.21.15.28:0/2427517115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a47bf07d0 con 0x562a47eb8570 2024-04-25T01:38:04.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.022+0000 7f6843fff640 1 --2- 172.21.15.28:0/2427517115 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47ead580 0x562a47ec4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:04.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.022+0000 7f6848aa6640 1 --2- 172.21.15.28:0/2427517115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47eb8570 0x562a47ed0240 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f683c0093c0 tx=0x7f683c0124b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b4ee470c3b64684 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:04.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.022+0000 7f6829ffb640 1 -- 172.21.15.28:0/2427517115 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c009710 con 0x562a47eb8570 2024-04-25T01:38:04.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.022+0000 7f6829ffb640 1 -- 172.21.15.28:0/2427517115 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f683c01be50 con 0x562a47eb8570 2024-04-25T01:38:04.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.022+0000 7f6829ffb640 1 -- 172.21.15.28:0/2427517115 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c0256a0 con 0x562a47eb8570 2024-04-25T01:38:04.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.022+0000 7f684d20e740 1 -- 172.21.15.28:0/2427517115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47eb8570 msgr2=0x562a47ed0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.022+0000 7f684d20e740 1 --2- 172.21.15.28:0/2427517115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47eb8570 0x562a47ed0240 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f683c0093c0 tx=0x7f683c0124b0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.022+0000 7f684d20e740 1 -- 172.21.15.28:0/2427517115 shutdown_connections 2024-04-25T01:38:04.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.022+0000 7f684d20e740 1 --2- 172.21.15.28:0/2427517115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47eb8570 0x562a47ed0240 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.022+0000 7f684d20e740 1 --2- 172.21.15.28:0/2427517115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 0x562a47eb79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.022+0000 7f684d20e740 1 --2- 172.21.15.28:0/2427517115 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47ead580 0x562a47ec4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.022+0000 7f684d20e740 1 -- 172.21.15.28:0/2427517115 >> 172.21.15.28:0/2427517115 conn(0x562a47ea98a0 msgr2=0x562a47eb0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:04.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.023+0000 7f684d20e740 1 -- 172.21.15.28:0/2427517115 shutdown_connections 2024-04-25T01:38:04.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.023+0000 7f684d20e740 1 -- 172.21.15.28:0/2427517115 wait complete. 2024-04-25T01:38:04.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.023+0000 7f684d20e740 1 -- start start 2024-04-25T01:38:04.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.023+0000 7f684d20e740 1 Processor -- start 2024-04-25T01:38:04.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.023+0000 7f684d20e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 0x562a47ec0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.023+0000 7f684d20e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 0x562a47ebbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.023+0000 7f684d20e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47eb8570 0x562a47ebc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.023+0000 7f684d20e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f68441afea0 con 0x562a47ead580 2024-04-25T01:38:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.023+0000 7f684d20e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f68441af7d0 con 0x562a47eadea0 2024-04-25T01:38:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.023+0000 7f684d20e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f68441aca10 con 0x562a47eb8570 2024-04-25T01:38:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.023+0000 7f684d20e740 10 client.0 _ll_register_callbacks cb 0x7f683400c870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.023+0000 7f6843fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 0x562a47ec0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.023+0000 7f6843fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 0x562a47ec0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42940/0 (socket says 172.21.15.28:42940) 2024-04-25T01:38:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.023+0000 7f6843fff640 1 -- 172.21.15.28:0/552932607 learned_addr learned my addr 172.21.15.28:0/552932607 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.024+0000 7f68437fe640 1 --2- 172.21.15.28:0/552932607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 0x562a47ebbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.024+0000 7f6848aa6640 1 --2- 172.21.15.28:0/552932607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47eb8570 0x562a47ebc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.024+0000 7f6843fff640 1 -- 172.21.15.28:0/552932607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47eb8570 msgr2=0x562a47ebc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.024+0000 7f6843fff640 1 --2- 172.21.15.28:0/552932607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47eb8570 0x562a47ebc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.024+0000 7f6843fff640 1 -- 172.21.15.28:0/552932607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 msgr2=0x562a47ebbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.024+0000 7f6843fff640 1 --2- 172.21.15.28:0/552932607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 0x562a47ebbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.024+0000 7f6843fff640 1 -- 172.21.15.28:0/552932607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a47ebcc70 con 0x562a47ead580 2024-04-25T01:38:04.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.024+0000 7f6843fff640 1 --2- 172.21.15.28:0/552932607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 0x562a47ec0b40 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f683409b7f0 tx=0x7f683400fd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38efcccfab2061d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:04.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.024+0000 7f68237fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:04.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.024+0000 7f68237fe640 1 -- 172.21.15.28:0/552932607 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683409af90 con 0x562a47ead580 2024-04-25T01:38:04.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.024+0000 7f68237fe640 1 -- 172.21.15.28:0/552932607 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f68340a42d0 con 0x562a47ead580 2024-04-25T01:38:04.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.024+0000 7f684d20e740 1 -- 172.21.15.28:0/552932607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f683c01be50 con 0x562a47ead580 2024-04-25T01:38:04.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.024+0000 7f68237fe640 1 -- 172.21.15.28:0/552932607 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f683409baf0 con 0x562a47ead580 2024-04-25T01:38:04.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.024+0000 7f684d20e740 10 client.17208 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:04.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.024+0000 7f684d20e740 1 -- 172.21.15.28:0/552932607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562a47f2ea80 con 0x562a47ead580 2024-04-25T01:38:04.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.024+0000 7f68237fe640 1 -- 172.21.15.28:0/552932607 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f683c01be50 con 0x562a47ead580 2024-04-25T01:38:04.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.025+0000 7f68237fe640 1 -- 172.21.15.28:0/552932607 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f683409aa40 con 0x562a47ead580 2024-04-25T01:38:04.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.025+0000 7f684d20e740 10 client.17208 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:04.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.025+0000 7f684d20e740 10 client.17208 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:38:04.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.025+0000 7f684d20e740 20 client.17208 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:04.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.025+0000 7f684d20e740 1 --2- 172.21.15.28:0/552932607 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x562a47f2f0e0 0x562a47f314d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.025+0000 7f684d20e740 4 client.17208 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:38:04.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.025+0000 7f684d20e740 1 -- 172.21.15.28:0/552932607 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562a47e19f00 con 0x562a47f2f0e0 2024-04-25T01:38:04.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.025+0000 7f68437fe640 1 --2- 172.21.15.28:0/552932607 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x562a47f2f0e0 0x562a47f314d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.025+0000 7f68437fe640 1 --2- 172.21.15.28:0/552932607 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x562a47f2f0e0 0x562a47f314d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f4c1ae1edc19ff28 server_cookie=59acfe0171fd9384 in_seq=0 out_seq=0 2024-04-25T01:38:04.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.025+0000 7f68237fe640 10 client.17208 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:38:04.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.026+0000 7f68237fe640 1 -- 172.21.15.28:0/552932607 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562a47e19f00 con 0x562a47f2f0e0 2024-04-25T01:38:04.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.026+0000 7f68237fe640 10 client.17208 handle_command_reply: tid=0 2024-04-25T01:38:04.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.026+0000 7f684d20e740 1 client.17208 shutdown 2024-04-25T01:38:04.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.026+0000 7f684d20e740 1 -- 172.21.15.28:0/552932607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 msgr2=0x562a47ec0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.026+0000 7f684d20e740 1 --2- 172.21.15.28:0/552932607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 0x562a47ec0b40 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f683409b7f0 tx=0x7f683400fd60 comp rx=0 tx=0).stop 2024-04-25T01:38:04.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.026+0000 7f684d20e740 1 -- 172.21.15.28:0/552932607 shutdown_connections 2024-04-25T01:38:04.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.026+0000 7f684d20e740 1 --2- 172.21.15.28:0/552932607 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x562a47f2f0e0 0x562a47f314d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.026+0000 7f684d20e740 1 --2- 172.21.15.28:0/552932607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562a47eb8570 0x562a47ebc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.026+0000 7f684d20e740 1 --2- 172.21.15.28:0/552932607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562a47eadea0 0x562a47ebbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.026+0000 7f68237fe640 0 client.17208 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:38:04.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.026+0000 7f684d20e740 1 --2- 172.21.15.28:0/552932607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a47ead580 0x562a47ec0b40 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.026+0000 7f684d20e740 1 -- 172.21.15.28:0/552932607 >> 172.21.15.28:0/552932607 conn(0x562a47ea98a0 msgr2=0x562a47ec3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:04.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.026+0000 7f684d20e740 1 -- 172.21.15.28:0/552932607 shutdown_connections 2024-04-25T01:38:04.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.026+0000 7f684d20e740 1 -- 172.21.15.28:0/552932607 wait complete. 2024-04-25T01:38:04.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.026+0000 7f684d20e740 20 client.17208 trim_cache size 0 max 16384 2024-04-25T01:38:04.012 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:38:04.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.026+0000 7f684a509640 1 -- 172.21.15.28:0/246153372 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f682c0a2d00 msgr2=0x7f682c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.026+0000 7f684a509640 1 --2- 172.21.15.28:0/246153372 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f682c0a2d00 0x7f682c0a51c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f683c020000 tx=0x7f683c01d000 comp rx=0 tx=0).stop 2024-04-25T01:38:04.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.026+0000 7f684a509640 1 -- 172.21.15.28:0/246153372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440ffc40 msgr2=0x7f6844197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.026+0000 7f684a509640 1 --2- 172.21.15.28:0/246153372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440ffc40 0x7f6844197fb0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f683400c920 tx=0x7f683400cde0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.027+0000 7f684a509640 1 -- 172.21.15.28:0/246153372 shutdown_connections 2024-04-25T01:38:04.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.027+0000 7f684a509640 1 --2- 172.21.15.28:0/246153372 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f682c0a2d00 0x7f682c0a51c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.027+0000 7f684a509640 1 --2- 172.21.15.28:0/246153372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6844107720 0x7f684419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.027+0000 7f684a509640 1 --2- 172.21.15.28:0/246153372 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6844100600 0x7f68441984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.027+0000 7f684a509640 1 --2- 172.21.15.28:0/246153372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440ffc40 0x7f6844197fb0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.027+0000 7f684a509640 1 -- 172.21.15.28:0/246153372 >> 172.21.15.28:0/246153372 conn(0x7f6844076bd0 msgr2=0x7f68440fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:04.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.027+0000 7f684a509640 1 -- 172.21.15.28:0/246153372 shutdown_connections 2024-04-25T01:38:04.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.027+0000 7f684a509640 1 -- 172.21.15.28:0/246153372 wait complete. 2024-04-25T01:38:04.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:38:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.141+0000 7fbb15c6b640 1 -- 172.21.15.28:0/598874792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb10106630 msgr2=0x7fbb10106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.141+0000 7fbb15c6b640 1 --2- 172.21.15.28:0/598874792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb10106630 0x7fbb10106a10 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009930 tx=0x7fbb0002a420 comp rx=0 tx=0).stop 2024-04-25T01:38:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.141+0000 7fbb15c6b640 1 -- 172.21.15.28:0/598874792 shutdown_connections 2024-04-25T01:38:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.141+0000 7fbb15c6b640 1 --2- 172.21.15.28:0/598874792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb10100fe0 0x7fbb1010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.141+0000 7fbb15c6b640 1 --2- 172.21.15.28:0/598874792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb10100620 0x7fbb10100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.141+0000 7fbb15c6b640 1 --2- 172.21.15.28:0/598874792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb10106630 0x7fbb10106a10 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.141+0000 7fbb15c6b640 1 -- 172.21.15.28:0/598874792 >> 172.21.15.28:0/598874792 conn(0x7fbb100776a0 msgr2=0x7fbb10077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.141+0000 7fbb15c6b640 1 -- 172.21.15.28:0/598874792 shutdown_connections 2024-04-25T01:38:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.141+0000 7fbb15c6b640 1 -- 172.21.15.28:0/598874792 wait complete. 2024-04-25T01:38:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.141+0000 7fbb15c6b640 1 Processor -- start 2024-04-25T01:38:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb15c6b640 1 -- start start 2024-04-25T01:38:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb15c6b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb10100620 0x7fbb1010df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb15c6b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb10100fe0 0x7fbb1010e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb15c6b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb10106630 0x7fbb1010ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb15c6b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb10111b40 con 0x7fbb10100fe0 2024-04-25T01:38:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb15c6b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbb101119c0 con 0x7fbb10106630 2024-04-25T01:38:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb15c6b640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbb10111cc0 con 0x7fbb10100620 2024-04-25T01:38:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb0effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb10100fe0 0x7fbb1010e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb0effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb10100fe0 0x7fbb1010e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42952/0 (socket says 172.21.15.28:42952) 2024-04-25T01:38:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb0effd640 1 -- 172.21.15.28:0/2975589641 learned_addr learned my addr 172.21.15.28:0/2975589641 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb0ffff640 1 --2- 172.21.15.28:0/2975589641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb10106630 0x7fbb1010ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb0f7fe640 1 --2- 172.21.15.28:0/2975589641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb10100620 0x7fbb1010df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb0effd640 1 -- 172.21.15.28:0/2975589641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb10100620 msgr2=0x7fbb1010df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb0effd640 1 --2- 172.21.15.28:0/2975589641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb10100620 0x7fbb1010df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb0effd640 1 -- 172.21.15.28:0/2975589641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb10106630 msgr2=0x7fbb1010ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb0effd640 1 --2- 172.21.15.28:0/2975589641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb10106630 0x7fbb1010ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb0effd640 1 -- 172.21.15.28:0/2975589641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb101a6440 con 0x7fbb10100fe0 2024-04-25T01:38:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb0f7fe640 1 --2- 172.21.15.28:0/2975589641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb10100620 0x7fbb1010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb0ffff640 1 --2- 172.21.15.28:0/2975589641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb10106630 0x7fbb1010ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.142+0000 7fbb0effd640 1 --2- 172.21.15.28:0/2975589641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb10100fe0 0x7fbb1010e4c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fbb0400c920 tx=0x7fbb0400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.143+0000 7fbb0cff9640 1 -- 172.21.15.28:0/2975589641 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb04007c70 con 0x7fbb10100fe0 2024-04-25T01:38:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.143+0000 7fbb0cff9640 1 -- 172.21.15.28:0/2975589641 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb04007e10 con 0x7fbb10100fe0 2024-04-25T01:38:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.143+0000 7fbb15c6b640 1 -- 172.21.15.28:0/2975589641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb101a6730 con 0x7fbb10100fe0 2024-04-25T01:38:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.143+0000 7fbb0cff9640 1 -- 172.21.15.28:0/2975589641 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb04018890 con 0x7fbb10100fe0 2024-04-25T01:38:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.143+0000 7fbb15c6b640 1 -- 172.21.15.28:0/2975589641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb101a6c80 con 0x7fbb10100fe0 2024-04-25T01:38:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.145+0000 7fbb0cff9640 1 -- 172.21.15.28:0/2975589641 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fbb04018a70 con 0x7fbb10100fe0 2024-04-25T01:38:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.145+0000 7fbb18978740 1 Processor -- start 2024-04-25T01:38:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.145+0000 7fbb18978740 1 -- start start 2024-04-25T01:38:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb18978740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a76580 0x560df4a76960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb18978740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76ea0 0x560df4a81030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb18978740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a81570 0x560df4a83960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb18978740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560df49db670 con 0x560df4a76ea0 2024-04-25T01:38:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb18978740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560df4983970 con 0x560df4a81570 2024-04-25T01:38:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb18978740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x560df49d7630 con 0x560df4a76580 2024-04-25T01:38:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb0f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76ea0 0x560df4a81030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb0f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76ea0 0x560df4a81030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42962/0 (socket says 172.21.15.28:42962) 2024-04-25T01:38:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb0f7fe640 1 -- 172.21.15.28:0/2627751680 learned_addr learned my addr 172.21.15.28:0/2627751680 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:04.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb0cff9640 1 --2- 172.21.15.28:0/2975589641 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbaf80a2d00 0x7fbaf80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb0ffff640 1 --2- 172.21.15.28:0/2627751680 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a76580 0x560df4a76960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb0cff9640 1 -- 172.21.15.28:0/2975589641 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fbb040cf9f0 con 0x7fbb10100fe0 2024-04-25T01:38:04.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb0effd640 1 --2- 172.21.15.28:0/2627751680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a81570 0x560df4a83960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb0ffff640 1 --2- 172.21.15.28:0/2975589641 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbaf80a2d00 0x7fbaf80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb0f7fe640 1 -- 172.21.15.28:0/2627751680 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a76580 msgr2=0x560df4a76960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb0f7fe640 1 --2- 172.21.15.28:0/2627751680 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a76580 0x560df4a76960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb0f7fe640 1 -- 172.21.15.28:0/2627751680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a81570 msgr2=0x560df4a83960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb0f7fe640 1 --2- 172.21.15.28:0/2627751680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a81570 0x560df4a83960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb0f7fe640 1 -- 172.21.15.28:0/2627751680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560df49a7400 con 0x560df4a76ea0 2024-04-25T01:38:04.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.146+0000 7fbb0f7fe640 1 --2- 172.21.15.28:0/2627751680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76ea0 0x560df4a81030 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fbb00006010 tx=0x7fbb00009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8602150bcd982da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:04.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.147+0000 7fbb0ffff640 1 --2- 172.21.15.28:0/2975589641 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbaf80a2d00 0x7fbaf80a51c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbb08005c40 tx=0x7fbb08009200 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:04.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.147+0000 7fbaf5ffb640 1 -- 172.21.15.28:0/2627751680 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0003f070 con 0x560df4a76ea0 2024-04-25T01:38:04.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.147+0000 7fbaf5ffb640 1 -- 172.21.15.28:0/2627751680 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb000317b0 con 0x560df4a76ea0 2024-04-25T01:38:04.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.147+0000 7fbaf5ffb640 1 -- 172.21.15.28:0/2627751680 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb00035070 con 0x560df4a76ea0 2024-04-25T01:38:04.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.147+0000 7fbb18978740 1 -- 172.21.15.28:0/2627751680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76ea0 msgr2=0x560df4a81030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.147+0000 7fbb18978740 1 --2- 172.21.15.28:0/2627751680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76ea0 0x560df4a81030 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fbb00006010 tx=0x7fbb00009680 comp rx=0 tx=0).stop 2024-04-25T01:38:04.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.147+0000 7fbb18978740 1 -- 172.21.15.28:0/2627751680 shutdown_connections 2024-04-25T01:38:04.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.147+0000 7fbb18978740 1 --2- 172.21.15.28:0/2627751680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a81570 0x560df4a83960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.147+0000 7fbb18978740 1 --2- 172.21.15.28:0/2627751680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76ea0 0x560df4a81030 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.147+0000 7fbb18978740 1 --2- 172.21.15.28:0/2627751680 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a76580 0x560df4a76960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.147+0000 7fbb18978740 1 -- 172.21.15.28:0/2627751680 >> 172.21.15.28:0/2627751680 conn(0x560df4a728a0 msgr2=0x560df4a72c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:04.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.147+0000 7fbb18978740 1 -- 172.21.15.28:0/2627751680 shutdown_connections 2024-04-25T01:38:04.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.147+0000 7fbb18978740 1 -- 172.21.15.28:0/2627751680 wait complete. 2024-04-25T01:38:04.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.148+0000 7fbb18978740 1 -- start start 2024-04-25T01:38:04.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.148+0000 7fbb18978740 1 Processor -- start 2024-04-25T01:38:04.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.148+0000 7fbb18978740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76580 0x560df4a80e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.148+0000 7fbb18978740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a76ea0 0x560df4a7e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.148+0000 7fbb18978740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a81570 0x560df4a7e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.148+0000 7fbb18978740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560df4a84300 con 0x560df4a76580 2024-04-25T01:38:04.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.148+0000 7fbb18978740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560df4997050 con 0x560df4a76ea0 2024-04-25T01:38:04.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.148+0000 7fbb18978740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x560df4a84480 con 0x560df4a81570 2024-04-25T01:38:04.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.148+0000 7fbb18978740 10 client.0 _ll_register_callbacks cb 0x560df46eebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:04.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.148+0000 7fbb0f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76580 0x560df4a80e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.148+0000 7fbb0f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76580 0x560df4a80e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42978/0 (socket says 172.21.15.28:42978) 2024-04-25T01:38:04.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.148+0000 7fbb0f7fe640 1 -- 172.21.15.28:0/3180122685 learned_addr learned my addr 172.21.15.28:0/3180122685 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:04.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.148+0000 7fbb0effd640 1 --2- 172.21.15.28:0/3180122685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a76ea0 0x560df4a7e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.148+0000 7fbb0ffff640 1 --2- 172.21.15.28:0/3180122685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a81570 0x560df4a7e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.149+0000 7fbb0f7fe640 1 -- 172.21.15.28:0/3180122685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a81570 msgr2=0x560df4a7e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.149+0000 7fbb0f7fe640 1 --2- 172.21.15.28:0/3180122685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a81570 0x560df4a7e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.149+0000 7fbb0f7fe640 1 -- 172.21.15.28:0/3180122685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a76ea0 msgr2=0x560df4a7e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.149+0000 7fbb0f7fe640 1 --2- 172.21.15.28:0/3180122685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a76ea0 0x560df4a7e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.149+0000 7fbb0f7fe640 1 -- 172.21.15.28:0/3180122685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560df4968dc0 con 0x560df4a76580 2024-04-25T01:38:04.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.149+0000 7fbb0f7fe640 1 --2- 172.21.15.28:0/3180122685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76580 0x560df4a80e30 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fbb00000c00 tx=0x7fbb00009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e1e4d21563e36a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:04.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.149+0000 7fbaeeffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:04.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.149+0000 7fbaeeffd640 1 -- 172.21.15.28:0/3180122685 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0003f070 con 0x560df4a76580 2024-04-25T01:38:04.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.149+0000 7fbb18978740 1 -- 172.21.15.28:0/3180122685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb000317b0 con 0x560df4a76580 2024-04-25T01:38:04.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.149+0000 7fbaeeffd640 1 -- 172.21.15.28:0/3180122685 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fbb00004250 con 0x560df4a76580 2024-04-25T01:38:04.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.149+0000 7fbaeeffd640 1 -- 172.21.15.28:0/3180122685 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbb00055050 con 0x560df4a76580 2024-04-25T01:38:04.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.149+0000 7fbb18978740 10 client.17220 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:04.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.149+0000 7fbb18978740 1 -- 172.21.15.28:0/3180122685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560df46ef3f0 con 0x560df4a76580 2024-04-25T01:38:04.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.149+0000 7fbaeeffd640 1 -- 172.21.15.28:0/3180122685 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb000317b0 con 0x560df4a76580 2024-04-25T01:38:04.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.149+0000 7fbaeeffd640 1 -- 172.21.15.28:0/3180122685 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbb00004730 con 0x560df4a76580 2024-04-25T01:38:04.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.149+0000 7fbb18978740 10 client.17220 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:04.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.149+0000 7fbb18978740 10 client.17220 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:38:04.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.149+0000 7fbb18978740 20 client.17220 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:04.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.150+0000 7fbb18978740 1 --2- 172.21.15.28:0/3180122685 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x560df4a91350 0x560df4a93740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.150+0000 7fbb18978740 4 client.17220 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:04.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.150+0000 7fbb18978740 1 -- 172.21.15.28:0/3180122685 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560df49e8bb0 con 0x560df4a91350 2024-04-25T01:38:04.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.150+0000 7fbb0effd640 1 --2- 172.21.15.28:0/3180122685 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x560df4a91350 0x560df4a93740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.150+0000 7fbb0effd640 1 --2- 172.21.15.28:0/3180122685 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x560df4a91350 0x560df4a93740 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fca09fd2b83b0143 server_cookie=16bef6b74b739087 in_seq=0 out_seq=0 2024-04-25T01:38:04.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.150+0000 7fbaeeffd640 10 client.17220 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:38:04.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.151+0000 7fbaeeffd640 1 -- 172.21.15.28:0/3180122685 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560df49e8bb0 con 0x560df4a91350 2024-04-25T01:38:04.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.151+0000 7fbaeeffd640 10 client.17220 handle_command_reply: tid=0 2024-04-25T01:38:04.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.151+0000 7fbb18978740 1 client.17220 shutdown 2024-04-25T01:38:04.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.151+0000 7fbb18978740 1 -- 172.21.15.28:0/3180122685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76580 msgr2=0x560df4a80e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.151+0000 7fbb18978740 1 --2- 172.21.15.28:0/3180122685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76580 0x560df4a80e30 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fbb00000c00 tx=0x7fbb00009260 comp rx=0 tx=0).stop 2024-04-25T01:38:04.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.151+0000 7fbb18978740 1 -- 172.21.15.28:0/3180122685 shutdown_connections 2024-04-25T01:38:04.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.151+0000 7fbb18978740 1 --2- 172.21.15.28:0/3180122685 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x560df4a91350 0x560df4a93740 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.151+0000 7fbb18978740 1 --2- 172.21.15.28:0/3180122685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a81570 0x560df4a7e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.151+0000 7fbb18978740 1 --2- 172.21.15.28:0/3180122685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a76ea0 0x560df4a7e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.151+0000 7fbaeeffd640 0 client.17220 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:38:04.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.151+0000 7fbb18978740 1 --2- 172.21.15.28:0/3180122685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76580 0x560df4a80e30 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.151+0000 7fbb18978740 1 -- 172.21.15.28:0/3180122685 >> 172.21.15.28:0/3180122685 conn(0x560df4a728a0 msgr2=0x560df4a71b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:04.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.151+0000 7fbb18978740 1 -- 172.21.15.28:0/3180122685 shutdown_connections 2024-04-25T01:38:04.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.152+0000 7fbb18978740 1 -- 172.21.15.28:0/3180122685 wait complete. 2024-04-25T01:38:04.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.152+0000 7fbb18978740 20 client.17220 trim_cache size 0 max 16384 2024-04-25T01:38:04.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.165+0000 7fbb18978740 1 Processor -- start 2024-04-25T01:38:04.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.165+0000 7fbb18978740 1 -- start start 2024-04-25T01:38:04.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.165+0000 7fbb18978740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a76580 0x560df4a79860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.165+0000 7fbb18978740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a76ea0 0x560df4a719c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.165+0000 7fbb18978740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a81570 0x560df4a80960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.165+0000 7fbb18978740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb101b0c10 con 0x560df4a81570 2024-04-25T01:38:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.165+0000 7fbb18978740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbb101b2750 con 0x560df4a76580 2024-04-25T01:38:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.165+0000 7fbb18978740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbb101b12e0 con 0x560df4a76ea0 2024-04-25T01:38:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.165+0000 7fbb0ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a81570 0x560df4a80960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.165+0000 7fbb0ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a81570 0x560df4a80960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42990/0 (socket says 172.21.15.28:42990) 2024-04-25T01:38:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.165+0000 7fbb0ffff640 1 -- 172.21.15.28:0/946363092 learned_addr learned my addr 172.21.15.28:0/946363092 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.165+0000 7fbb0f7fe640 1 --2- 172.21.15.28:0/946363092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a76580 0x560df4a79860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.165+0000 7fbb0effd640 1 --2- 172.21.15.28:0/946363092 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a76ea0 0x560df4a719c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.165+0000 7fbb0ffff640 1 -- 172.21.15.28:0/946363092 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a76ea0 msgr2=0x560df4a719c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.165+0000 7fbb0ffff640 1 --2- 172.21.15.28:0/946363092 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a76ea0 0x560df4a719c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.166+0000 7fbb0ffff640 1 -- 172.21.15.28:0/946363092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a76580 msgr2=0x560df4a79860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.166+0000 7fbb0ffff640 1 --2- 172.21.15.28:0/946363092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a76580 0x560df4a79860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.166+0000 7fbb0ffff640 1 -- 172.21.15.28:0/946363092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560df47b97d0 con 0x560df4a81570 2024-04-25T01:38:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.166+0000 7fbb0effd640 1 --2- 172.21.15.28:0/946363092 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a76ea0 0x560df4a719c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.166+0000 7fbb0f7fe640 1 --2- 172.21.15.28:0/946363092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a76580 0x560df4a79860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.166+0000 7fbb0ffff640 1 --2- 172.21.15.28:0/946363092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a81570 0x560df4a80960 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fbb0800f800 tx=0x7fbb08009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea294e5cd3286a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.166+0000 7fbaf57fa640 1 -- 172.21.15.28:0/946363092 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb08018040 con 0x560df4a81570 2024-04-25T01:38:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.166+0000 7fbaf57fa640 1 -- 172.21.15.28:0/946363092 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb08011970 con 0x560df4a81570 2024-04-25T01:38:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.166+0000 7fbaf57fa640 1 -- 172.21.15.28:0/946363092 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb08005740 con 0x560df4a81570 2024-04-25T01:38:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.166+0000 7fbb18978740 1 -- 172.21.15.28:0/946363092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a81570 msgr2=0x560df4a80960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.166+0000 7fbb18978740 1 --2- 172.21.15.28:0/946363092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a81570 0x560df4a80960 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fbb0800f800 tx=0x7fbb08009650 comp rx=0 tx=0).stop 2024-04-25T01:38:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.166+0000 7fbb18978740 1 -- 172.21.15.28:0/946363092 shutdown_connections 2024-04-25T01:38:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.166+0000 7fbb18978740 1 --2- 172.21.15.28:0/946363092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a81570 0x560df4a80960 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.166+0000 7fbb18978740 1 --2- 172.21.15.28:0/946363092 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a76ea0 0x560df4a719c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.166+0000 7fbb18978740 1 --2- 172.21.15.28:0/946363092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a76580 0x560df4a79860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.166+0000 7fbb18978740 1 -- 172.21.15.28:0/946363092 >> 172.21.15.28:0/946363092 conn(0x560df4a728a0 msgr2=0x560df4a712b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.166+0000 7fbb18978740 1 -- 172.21.15.28:0/946363092 shutdown_connections 2024-04-25T01:38:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.167+0000 7fbb18978740 1 -- 172.21.15.28:0/946363092 wait complete. 2024-04-25T01:38:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.167+0000 7fbb18978740 1 -- start start 2024-04-25T01:38:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.167+0000 7fbb18978740 1 Processor -- start 2024-04-25T01:38:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.167+0000 7fbb18978740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a76580 0x560df4a899e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.167+0000 7fbb18978740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76ea0 0x560df4a84d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.167+0000 7fbb18978740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a81570 0x560df4a85290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.167+0000 7fbb18978740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb101b2080 con 0x560df4a76ea0 2024-04-25T01:38:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.167+0000 7fbb18978740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbb101b19b0 con 0x560df4a76580 2024-04-25T01:38:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.167+0000 7fbb18978740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbb101aebf0 con 0x560df4a81570 2024-04-25T01:38:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.167+0000 7fbb18978740 10 client.0 _ll_register_callbacks cb 0x7fbb00032c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.167+0000 7fbb0effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76ea0 0x560df4a84d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.167+0000 7fbb0effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76ea0 0x560df4a84d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42998/0 (socket says 172.21.15.28:42998) 2024-04-25T01:38:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.167+0000 7fbb0effd640 1 -- 172.21.15.28:0/3147509590 learned_addr learned my addr 172.21.15.28:0/3147509590 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.167+0000 7fbb0f7fe640 1 --2- 172.21.15.28:0/3147509590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a76580 0x560df4a899e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbb0effd640 1 -- 172.21.15.28:0/3147509590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a81570 msgr2=0x560df4a85290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbb0effd640 1 --2- 172.21.15.28:0/3147509590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a81570 0x560df4a85290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbb0effd640 1 -- 172.21.15.28:0/3147509590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a76580 msgr2=0x560df4a899e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbb0effd640 1 --2- 172.21.15.28:0/3147509590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a76580 0x560df4a899e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbb0effd640 1 -- 172.21.15.28:0/3147509590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560df4a85a90 con 0x560df4a76ea0 2024-04-25T01:38:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbb0f7fe640 1 --2- 172.21.15.28:0/3147509590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a76580 0x560df4a899e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbb0effd640 1 --2- 172.21.15.28:0/3147509590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76ea0 0x560df4a84d50 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fbb0409bb00 tx=0x7fbb04000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e1d5f6d7c85a45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbaeeffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbaeeffd640 1 -- 172.21.15.28:0/3147509590 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb040b8020 con 0x560df4a76ea0 2024-04-25T01:38:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbaeeffd640 1 -- 172.21.15.28:0/3147509590 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fbb0400fb30 con 0x560df4a76ea0 2024-04-25T01:38:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbb18978740 1 -- 172.21.15.28:0/3147509590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb08011970 con 0x560df4a76ea0 2024-04-25T01:38:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbaeeffd640 1 -- 172.21.15.28:0/3147509590 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbb0409ff20 con 0x560df4a76ea0 2024-04-25T01:38:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbb18978740 10 client.17226 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbb18978740 1 -- 172.21.15.28:0/3147509590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560df4af7500 con 0x560df4a76ea0 2024-04-25T01:38:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbaeeffd640 1 -- 172.21.15.28:0/3147509590 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb08011970 con 0x560df4a76ea0 2024-04-25T01:38:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbaeeffd640 1 -- 172.21.15.28:0/3147509590 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbb040a4350 con 0x560df4a76ea0 2024-04-25T01:38:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbb18978740 10 client.17226 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbb18978740 10 client.17226 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:38:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbb18978740 20 client.17226 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbb18978740 1 --2- 172.21.15.28:0/3147509590 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x560df4af7c30 0x560df4afa050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbb18978740 4 client.17226 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:38:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.168+0000 7fbb18978740 1 -- 172.21.15.28:0/3147509590 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560df4a7e9f0 con 0x560df4af7c30 2024-04-25T01:38:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.169+0000 7fbb0f7fe640 1 --2- 172.21.15.28:0/3147509590 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x560df4af7c30 0x560df4afa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.169+0000 7fbb0f7fe640 1 --2- 172.21.15.28:0/3147509590 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x560df4af7c30 0x560df4afa050 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=62e53049ddf73f6e server_cookie=c45a9eb7a16d2171 in_seq=0 out_seq=0 2024-04-25T01:38:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.169+0000 7fbaeeffd640 10 client.17226 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:38:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.169+0000 7fbaeeffd640 1 -- 172.21.15.28:0/3147509590 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560df4a7e9f0 con 0x560df4af7c30 2024-04-25T01:38:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.169+0000 7fbaeeffd640 10 client.17226 handle_command_reply: tid=0 2024-04-25T01:38:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.169+0000 7fbb18978740 1 client.17226 shutdown 2024-04-25T01:38:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.169+0000 7fbb18978740 1 -- 172.21.15.28:0/3147509590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76ea0 msgr2=0x560df4a84d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.169+0000 7fbb18978740 1 --2- 172.21.15.28:0/3147509590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76ea0 0x560df4a84d50 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fbb0409bb00 tx=0x7fbb04000bc0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb18978740 1 -- 172.21.15.28:0/3147509590 shutdown_connections 2024-04-25T01:38:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb18978740 1 --2- 172.21.15.28:0/3147509590 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x560df4af7c30 0x560df4afa050 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb18978740 1 --2- 172.21.15.28:0/3147509590 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560df4a81570 0x560df4a85290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb18978740 1 --2- 172.21.15.28:0/3147509590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560df4a76ea0 0x560df4a84d50 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbaeeffd640 0 client.17226 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:38:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb18978740 1 --2- 172.21.15.28:0/3147509590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560df4a76580 0x560df4a899e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb18978740 1 -- 172.21.15.28:0/3147509590 >> 172.21.15.28:0/3147509590 conn(0x560df4a728a0 msgr2=0x560df4a8c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb18978740 1 -- 172.21.15.28:0/3147509590 shutdown_connections 2024-04-25T01:38:04.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb18978740 1 -- 172.21.15.28:0/3147509590 wait complete. 2024-04-25T01:38:04.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb18978740 20 client.17226 trim_cache size 0 max 16384 2024-04-25T01:38:04.156 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:38:04.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb15c6b640 1 -- 172.21.15.28:0/2975589641 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbaf80a2d00 msgr2=0x7fbaf80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb15c6b640 1 --2- 172.21.15.28:0/2975589641 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbaf80a2d00 0x7fbaf80a51c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbb08005c40 tx=0x7fbb08009200 comp rx=0 tx=0).stop 2024-04-25T01:38:04.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb15c6b640 1 -- 172.21.15.28:0/2975589641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb10100fe0 msgr2=0x7fbb1010e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb15c6b640 1 --2- 172.21.15.28:0/2975589641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb10100fe0 0x7fbb1010e4c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fbb0400c920 tx=0x7fbb0400cde0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb15c6b640 1 -- 172.21.15.28:0/2975589641 shutdown_connections 2024-04-25T01:38:04.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb15c6b640 1 --2- 172.21.15.28:0/2975589641 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbaf80a2d00 0x7fbaf80a51c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb15c6b640 1 --2- 172.21.15.28:0/2975589641 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbb10106630 0x7fbb1010ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb15c6b640 1 --2- 172.21.15.28:0/2975589641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb10100fe0 0x7fbb1010e4c0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb15c6b640 1 --2- 172.21.15.28:0/2975589641 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbb10100620 0x7fbb1010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb15c6b640 1 -- 172.21.15.28:0/2975589641 >> 172.21.15.28:0/2975589641 conn(0x7fbb100776a0 msgr2=0x7fbb101030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:04.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb15c6b640 1 -- 172.21.15.28:0/2975589641 shutdown_connections 2024-04-25T01:38:04.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.170+0000 7fbb15c6b640 1 -- 172.21.15.28:0/2975589641 wait complete. 2024-04-25T01:38:04.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:38:04.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:03 smithi130 ceph-mon[32702]: pgmap v154: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:38:04.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:03 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/572610936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:38:04.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:03 smithi032 ceph-mon[32461]: pgmap v154: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:38:04.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:03 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/572610936' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:38:04.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.285+0000 7f8fbd7a5640 1 -- 172.21.15.28:0/1516151757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb806a780 msgr2=0x7f8fb810e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.285+0000 7f8fbd7a5640 1 --2- 172.21.15.28:0/1516151757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb806a780 0x7f8fb810e5f0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f8fb0009cd0 tx=0x7f8fb002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.285+0000 7f8fbd7a5640 1 -- 172.21.15.28:0/1516151757 shutdown_connections 2024-04-25T01:38:04.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.285+0000 7f8fbd7a5640 1 --2- 172.21.15.28:0/1516151757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb806a780 0x7f8fb810e5f0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.285+0000 7f8fbd7a5640 1 --2- 172.21.15.28:0/1516151757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fb8069dc0 0x7f8fb806a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.285+0000 7f8fbd7a5640 1 --2- 172.21.15.28:0/1516151757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8fb8107730 0x7f8fb8107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.285+0000 7f8fbd7a5640 1 -- 172.21.15.28:0/1516151757 >> 172.21.15.28:0/1516151757 conn(0x7f8fb80776a0 msgr2=0x7f8fb8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:04.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.285+0000 7f8fbd7a5640 1 -- 172.21.15.28:0/1516151757 shutdown_connections 2024-04-25T01:38:04.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.285+0000 7f8fbd7a5640 1 -- 172.21.15.28:0/1516151757 wait complete. 2024-04-25T01:38:04.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.285+0000 7f8fbd7a5640 1 Processor -- start 2024-04-25T01:38:04.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.286+0000 7f8fbd7a5640 1 -- start start 2024-04-25T01:38:04.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.286+0000 7f8fbd7a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb8069dc0 0x7f8fb8197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.286+0000 7f8fbd7a5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8fb806a780 0x7f8fb81984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.286+0000 7f8fbd7a5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fb8107730 0x7f8fb819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.286+0000 7f8fbd7a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb8111b40 con 0x7f8fb8069dc0 2024-04-25T01:38:04.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.286+0000 7f8fbd7a5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb81119c0 con 0x7f8fb8107730 2024-04-25T01:38:04.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.286+0000 7f8fbd7a5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb8111cc0 con 0x7f8fb806a780 2024-04-25T01:38:04.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.286+0000 7f8fb6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb8069dc0 0x7f8fb8197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.286+0000 7f8fb6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb8069dc0 0x7f8fb8197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43016/0 (socket says 172.21.15.28:43016) 2024-04-25T01:38:04.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.286+0000 7f8fb6ffd640 1 -- 172.21.15.28:0/2466433098 learned_addr learned my addr 172.21.15.28:0/2466433098 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:04.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.286+0000 7f8fb77fe640 1 --2- 172.21.15.28:0/2466433098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fb8107730 0x7f8fb819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.286+0000 7f8fb67fc640 1 --2- 172.21.15.28:0/2466433098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8fb806a780 0x7f8fb81984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.286+0000 7f8fb6ffd640 1 -- 172.21.15.28:0/2466433098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8fb806a780 msgr2=0x7f8fb81984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.286+0000 7f8fb6ffd640 1 --2- 172.21.15.28:0/2466433098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8fb806a780 0x7f8fb81984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.286+0000 7f8fb6ffd640 1 -- 172.21.15.28:0/2466433098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fb8107730 msgr2=0x7f8fb819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.286+0000 7f8fb6ffd640 1 --2- 172.21.15.28:0/2466433098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fb8107730 0x7f8fb819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.286+0000 7f8fb6ffd640 1 -- 172.21.15.28:0/2466433098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb819cf50 con 0x7f8fb8069dc0 2024-04-25T01:38:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.286+0000 7f8fb6ffd640 1 --2- 172.21.15.28:0/2466433098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb8069dc0 0x7f8fb8197f60 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f8fa800e9f0 tx=0x7f8fa800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.287+0000 7f8f9ffff640 1 -- 172.21.15.28:0/2466433098 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa800cd80 con 0x7f8fb8069dc0 2024-04-25T01:38:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.287+0000 7f8fbd7a5640 1 -- 172.21.15.28:0/2466433098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fb819d240 con 0x7f8fb8069dc0 2024-04-25T01:38:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.287+0000 7f8f9ffff640 1 -- 172.21.15.28:0/2466433098 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fa8005500 con 0x7f8fb8069dc0 2024-04-25T01:38:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.287+0000 7f8f9ffff640 1 -- 172.21.15.28:0/2466433098 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8010690 con 0x7f8fb8069dc0 2024-04-25T01:38:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.287+0000 7f8fbd7a5640 1 -- 172.21.15.28:0/2466433098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fb81a4ad0 con 0x7f8fb8069dc0 2024-04-25T01:38:04.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.289+0000 7f8f9ffff640 1 -- 172.21.15.28:0/2466433098 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8fa80046c0 con 0x7f8fb8069dc0 2024-04-25T01:38:04.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.289+0000 7f8fc058f740 1 Processor -- start 2024-04-25T01:38:04.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.289+0000 7f8fc058f740 1 -- start start 2024-04-25T01:38:04.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.289+0000 7f8f9ffff640 1 --2- 172.21.15.28:0/2466433098 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8fa00a2d00 0x7f8fa00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8fc058f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 0x558815f99960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8fc058f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815f99ea0 0x558815fa4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8fc058f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815fa4570 0x558815fa6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8fc058f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558815efe670 con 0x558815f99580 2024-04-25T01:38:04.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8fc058f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558815ea6970 con 0x558815fa4570 2024-04-25T01:38:04.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8fc058f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x558815efa630 con 0x558815f99ea0 2024-04-25T01:38:04.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8fb6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 0x558815f99960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8fb77fe640 1 --2- 172.21.15.28:0/2466433098 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8fa00a2d00 0x7f8fa00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8fb6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 0x558815f99960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43020/0 (socket says 172.21.15.28:43020) 2024-04-25T01:38:04.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8fb6ffd640 1 -- 172.21.15.28:0/1675226898 learned_addr learned my addr 172.21.15.28:0/1675226898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:04.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8f9ffff640 1 -- 172.21.15.28:0/2466433098 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8fa80cfbf0 con 0x7f8fb8069dc0 2024-04-25T01:38:04.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8fb67fc640 1 --2- 172.21.15.28:0/1675226898 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815f99ea0 0x558815fa4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8fb6ffd640 1 -- 172.21.15.28:0/1675226898 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815f99ea0 msgr2=0x558815fa4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8fb6ffd640 1 --2- 172.21.15.28:0/1675226898 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815f99ea0 0x558815fa4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8fb6ffd640 1 -- 172.21.15.28:0/1675226898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815fa4570 msgr2=0x558815fa6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8fb6ffd640 1 --2- 172.21.15.28:0/1675226898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815fa4570 0x558815fa6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8fb6ffd640 1 -- 172.21.15.28:0/1675226898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558815eca400 con 0x558815f99580 2024-04-25T01:38:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8fb67fc640 1 --2- 172.21.15.28:0/1675226898 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815f99ea0 0x558815fa4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8fb77fe640 1 --2- 172.21.15.28:0/2466433098 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8fa00a2d00 0x7f8fa00a51c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8fb0004ab0 tx=0x7f8fb0009800 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8fb6ffd640 1 --2- 172.21.15.28:0/1675226898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 0x558815f99960 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f8fa80a4510 tx=0x7f8fa8097a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4622db181c31eb7a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8f9d7fa640 1 -- 172.21.15.28:0/1675226898 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa809b730 con 0x558815f99580 2024-04-25T01:38:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8f9d7fa640 1 -- 172.21.15.28:0/1675226898 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fa809c4d0 con 0x558815f99580 2024-04-25T01:38:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.290+0000 7f8f9d7fa640 1 -- 172.21.15.28:0/1675226898 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa80bd350 con 0x558815f99580 2024-04-25T01:38:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.291+0000 7f8fc058f740 1 -- 172.21.15.28:0/1675226898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 msgr2=0x558815f99960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.291+0000 7f8fc058f740 1 --2- 172.21.15.28:0/1675226898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 0x558815f99960 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f8fa80a4510 tx=0x7f8fa8097a10 comp rx=0 tx=0).stop 2024-04-25T01:38:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.291+0000 7f8fc058f740 1 -- 172.21.15.28:0/1675226898 shutdown_connections 2024-04-25T01:38:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.291+0000 7f8fc058f740 1 --2- 172.21.15.28:0/1675226898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815fa4570 0x558815fa6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.291+0000 7f8fc058f740 1 --2- 172.21.15.28:0/1675226898 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815f99ea0 0x558815fa4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.291+0000 7f8fc058f740 1 --2- 172.21.15.28:0/1675226898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 0x558815f99960 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.291+0000 7f8fc058f740 1 -- 172.21.15.28:0/1675226898 >> 172.21.15.28:0/1675226898 conn(0x558815f958a0 msgr2=0x558815f95c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.291+0000 7f8fc058f740 1 -- 172.21.15.28:0/1675226898 shutdown_connections 2024-04-25T01:38:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.291+0000 7f8fc058f740 1 -- 172.21.15.28:0/1675226898 wait complete. 2024-04-25T01:38:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.291+0000 7f8fc058f740 1 -- start start 2024-04-25T01:38:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fc058f740 1 Processor -- start 2024-04-25T01:38:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fc058f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815f99580 0x558815fa3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fc058f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99ea0 0x558815fa11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fc058f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815fa4570 0x558815fa1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fc058f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558815fa7360 con 0x558815f99ea0 2024-04-25T01:38:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fc058f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558815eba050 con 0x558815fa4570 2024-04-25T01:38:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fc058f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x558815fa74e0 con 0x558815f99580 2024-04-25T01:38:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fc058f740 10 client.0 _ll_register_callbacks cb 0x558815c11be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fb67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99ea0 0x558815fa11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fb67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99ea0 0x558815fa11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43032/0 (socket says 172.21.15.28:43032) 2024-04-25T01:38:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fb67fc640 1 -- 172.21.15.28:0/1602581363 learned_addr learned my addr 172.21.15.28:0/1602581363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fb77fe640 1 --2- 172.21.15.28:0/1602581363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815fa4570 0x558815fa1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fb6ffd640 1 --2- 172.21.15.28:0/1602581363 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815f99580 0x558815fa3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fb67fc640 1 -- 172.21.15.28:0/1602581363 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815f99580 msgr2=0x558815fa3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fb67fc640 1 --2- 172.21.15.28:0/1602581363 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815f99580 0x558815fa3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fb67fc640 1 -- 172.21.15.28:0/1602581363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815fa4570 msgr2=0x558815fa1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fb67fc640 1 --2- 172.21.15.28:0/1602581363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815fa4570 0x558815fa1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fb67fc640 1 -- 172.21.15.28:0/1602581363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558815e8bdc0 con 0x558815f99ea0 2024-04-25T01:38:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fb77fe640 1 --2- 172.21.15.28:0/1602581363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815fa4570 0x558815fa1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fb6ffd640 1 --2- 172.21.15.28:0/1602581363 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815f99580 0x558815fa3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8fb67fc640 1 --2- 172.21.15.28:0/1602581363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99ea0 0x558815fa11f0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f8fac00be90 tx=0x7f8fac0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=772db08dd48f758a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8f9aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.292+0000 7f8f9aff5640 1 -- 172.21.15.28:0/1602581363 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac012650 con 0x558815f99ea0 2024-04-25T01:38:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.293+0000 7f8fc058f740 1 -- 172.21.15.28:0/1602581363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fa809c4d0 con 0x558815f99ea0 2024-04-25T01:38:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.293+0000 7f8f9aff5640 1 -- 172.21.15.28:0/1602581363 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8fac0127f0 con 0x558815f99ea0 2024-04-25T01:38:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.293+0000 7f8f9aff5640 1 -- 172.21.15.28:0/1602581363 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8fac0129f0 con 0x558815f99ea0 2024-04-25T01:38:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.293+0000 7f8fc058f740 10 client.17238 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.293+0000 7f8fc058f740 1 -- 172.21.15.28:0/1602581363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558815c123f0 con 0x558815f99ea0 2024-04-25T01:38:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.293+0000 7f8f9aff5640 1 -- 172.21.15.28:0/1602581363 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fa809c4d0 con 0x558815f99ea0 2024-04-25T01:38:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.293+0000 7f8f9aff5640 1 -- 172.21.15.28:0/1602581363 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8fac01a3d0 con 0x558815f99ea0 2024-04-25T01:38:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.293+0000 7f8fc058f740 10 client.17238 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.293+0000 7f8fc058f740 10 client.17238 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.293+0000 7f8fc058f740 20 client.17238 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.293+0000 7f8fc058f740 1 --2- 172.21.15.28:0/1602581363 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558815fb4630 0x558815fb6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.293+0000 7f8fc058f740 4 client.17238 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.293+0000 7f8fc058f740 1 -- 172.21.15.28:0/1602581363 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558815f0bbb0 con 0x558815fb4630 2024-04-25T01:38:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.293+0000 7f8fb6ffd640 1 --2- 172.21.15.28:0/1602581363 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558815fb4630 0x558815fb6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.294+0000 7f8f9aff5640 10 client.17238 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.294+0000 7f8fb6ffd640 1 --2- 172.21.15.28:0/1602581363 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558815fb4630 0x558815fb6a70 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45aa9b7c70e2f58c server_cookie=3527e879c9805b17 in_seq=0 out_seq=0 2024-04-25T01:38:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.294+0000 7f8f9aff5640 1 -- 172.21.15.28:0/1602581363 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558815f0bbb0 con 0x558815fb4630 2024-04-25T01:38:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.294+0000 7f8f9aff5640 10 client.17238 handle_command_reply: tid=0 2024-04-25T01:38:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.294+0000 7f8fc058f740 1 client.17238 shutdown 2024-04-25T01:38:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.295+0000 7f8fc058f740 1 -- 172.21.15.28:0/1602581363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99ea0 msgr2=0x558815fa11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.295+0000 7f8fc058f740 1 --2- 172.21.15.28:0/1602581363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99ea0 0x558815fa11f0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f8fac00be90 tx=0x7f8fac0117b0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.295+0000 7f8fc058f740 1 -- 172.21.15.28:0/1602581363 shutdown_connections 2024-04-25T01:38:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.295+0000 7f8fc058f740 1 --2- 172.21.15.28:0/1602581363 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558815fb4630 0x558815fb6a70 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.295+0000 7f8fc058f740 1 --2- 172.21.15.28:0/1602581363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815fa4570 0x558815fa1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.295+0000 7f8fc058f740 1 --2- 172.21.15.28:0/1602581363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99ea0 0x558815fa11f0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.295+0000 7f8f9aff5640 0 client.17238 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.295+0000 7f8fc058f740 1 --2- 172.21.15.28:0/1602581363 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815f99580 0x558815fa3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.295+0000 7f8fc058f740 1 -- 172.21.15.28:0/1602581363 >> 172.21.15.28:0/1602581363 conn(0x558815f958a0 msgr2=0x558815f9b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.295+0000 7f8fc058f740 1 -- 172.21.15.28:0/1602581363 shutdown_connections 2024-04-25T01:38:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.295+0000 7f8fc058f740 1 -- 172.21.15.28:0/1602581363 wait complete. 2024-04-25T01:38:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.295+0000 7f8fc058f740 20 client.17238 trim_cache size 0 max 16384 2024-04-25T01:38:04.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.308+0000 7f8fc058f740 1 Processor -- start 2024-04-25T01:38:04.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.308+0000 7f8fc058f740 1 -- start start 2024-04-25T01:38:04.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.308+0000 7f8fc058f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 0x558815fb0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.308+0000 7f8fc058f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815f99ea0 0x558815fa0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.308+0000 7f8fc058f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815fa4570 0x558815fbc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.308+0000 7f8fc058f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb81aead0 con 0x558815f99580 2024-04-25T01:38:04.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.308+0000 7f8fc058f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb81b0610 con 0x558815f99ea0 2024-04-25T01:38:04.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.308+0000 7f8fc058f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb81af1a0 con 0x558815fa4570 2024-04-25T01:38:04.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.308+0000 7f8fb6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 0x558815fb0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.308+0000 7f8fb6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 0x558815fb0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43042/0 (socket says 172.21.15.28:43042) 2024-04-25T01:38:04.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.308+0000 7f8fb6ffd640 1 -- 172.21.15.28:0/1812762352 learned_addr learned my addr 172.21.15.28:0/1812762352 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:04.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.308+0000 7f8fb67fc640 1 --2- 172.21.15.28:0/1812762352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815f99ea0 0x558815fa0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.309+0000 7f8fb6ffd640 1 -- 172.21.15.28:0/1812762352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815fa4570 msgr2=0x558815fbc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.309+0000 7f8fb6ffd640 1 --2- 172.21.15.28:0/1812762352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815fa4570 0x558815fbc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.309+0000 7f8fb6ffd640 1 -- 172.21.15.28:0/1812762352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815f99ea0 msgr2=0x558815fa0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.309+0000 7f8fb6ffd640 1 --2- 172.21.15.28:0/1812762352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815f99ea0 0x558815fa0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.309+0000 7f8fb6ffd640 1 -- 172.21.15.28:0/1812762352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558815f99cd0 con 0x558815f99580 2024-04-25T01:38:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.309+0000 7f8fb67fc640 1 --2- 172.21.15.28:0/1812762352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815f99ea0 0x558815fa0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.309+0000 7f8fb6ffd640 1 --2- 172.21.15.28:0/1812762352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 0x558815fb0220 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8004a80 tx=0x7f8fa809bca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7feb40e6dc38c33b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.309+0000 7f8f9cff9640 1 -- 172.21.15.28:0/1812762352 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa80b4fd0 con 0x558815f99580 2024-04-25T01:38:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.309+0000 7f8f9cff9640 1 -- 172.21.15.28:0/1812762352 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fa809c4d0 con 0x558815f99580 2024-04-25T01:38:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.309+0000 7f8f9cff9640 1 -- 172.21.15.28:0/1812762352 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8098440 con 0x558815f99580 2024-04-25T01:38:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.309+0000 7f8fc058f740 1 -- 172.21.15.28:0/1812762352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 msgr2=0x558815fb0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.309+0000 7f8fc058f740 1 --2- 172.21.15.28:0/1812762352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 0x558815fb0220 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8004a80 tx=0x7f8fa809bca0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.309+0000 7f8fc058f740 1 -- 172.21.15.28:0/1812762352 shutdown_connections 2024-04-25T01:38:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.309+0000 7f8fc058f740 1 --2- 172.21.15.28:0/1812762352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815fa4570 0x558815fbc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.309+0000 7f8fc058f740 1 --2- 172.21.15.28:0/1812762352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815f99ea0 0x558815fa0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.309+0000 7f8fc058f740 1 --2- 172.21.15.28:0/1812762352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 0x558815fb0220 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.309+0000 7f8fc058f740 1 -- 172.21.15.28:0/1812762352 >> 172.21.15.28:0/1812762352 conn(0x558815f958a0 msgr2=0x558815fa0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:04.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.310+0000 7f8fc058f740 1 -- 172.21.15.28:0/1812762352 shutdown_connections 2024-04-25T01:38:04.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.310+0000 7f8fc058f740 1 -- 172.21.15.28:0/1812762352 wait complete. 2024-04-25T01:38:04.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.310+0000 7f8fc058f740 1 -- start start 2024-04-25T01:38:04.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.310+0000 7f8fc058f740 1 Processor -- start 2024-04-25T01:38:04.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.310+0000 7f8fc058f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 0x558815fac700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.310+0000 7f8fc058f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815f99ea0 0x558815facc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.310+0000 7f8fc058f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815fa4570 0x558815fa7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.310+0000 7f8fc058f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb81aff40 con 0x558815f99580 2024-04-25T01:38:04.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.310+0000 7f8fc058f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb81af870 con 0x558815f99ea0 2024-04-25T01:38:04.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.310+0000 7f8fc058f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb81acab0 con 0x558815fa4570 2024-04-25T01:38:04.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.310+0000 7f8fc058f740 10 client.0 _ll_register_callbacks cb 0x7f8fac005f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:04.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.310+0000 7f8fb6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 0x558815fac700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.310+0000 7f8fb6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 0x558815fac700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43054/0 (socket says 172.21.15.28:43054) 2024-04-25T01:38:04.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.310+0000 7f8fb6ffd640 1 -- 172.21.15.28:0/3219908339 learned_addr learned my addr 172.21.15.28:0/3219908339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:04.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.310+0000 7f8fb67fc640 1 --2- 172.21.15.28:0/3219908339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815f99ea0 0x558815facc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.310+0000 7f8fb77fe640 1 --2- 172.21.15.28:0/3219908339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815fa4570 0x558815fa7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.310+0000 7f8fb6ffd640 1 -- 172.21.15.28:0/3219908339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815fa4570 msgr2=0x558815fa7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.310+0000 7f8fb6ffd640 1 --2- 172.21.15.28:0/3219908339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815fa4570 0x558815fa7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.311+0000 7f8fb6ffd640 1 -- 172.21.15.28:0/3219908339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815f99ea0 msgr2=0x558815facc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:04.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.311+0000 7f8fb6ffd640 1 --2- 172.21.15.28:0/3219908339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815f99ea0 0x558815facc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.311+0000 7f8fb6ffd640 1 -- 172.21.15.28:0/3219908339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558815cdc7d0 con 0x558815f99580 2024-04-25T01:38:04.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.311+0000 7f8fb6ffd640 1 --2- 172.21.15.28:0/3219908339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 0x558815fac700 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f8fa809bd10 tx=0x7f8fa80cd530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708cdd479df548dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:04.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.311+0000 7f8f9aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:04.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.311+0000 7f8f9aff5640 1 -- 172.21.15.28:0/3219908339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8066030 con 0x558815f99580 2024-04-25T01:38:04.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.311+0000 7f8f9aff5640 1 -- 172.21.15.28:0/3219908339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8fa80a45b0 con 0x558815f99580 2024-04-25T01:38:04.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.311+0000 7f8fc058f740 1 -- 172.21.15.28:0/3219908339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fa809c4d0 con 0x558815f99580 2024-04-25T01:38:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.311+0000 7f8f9aff5640 1 -- 172.21.15.28:0/3219908339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8fa8099e70 con 0x558815f99580 2024-04-25T01:38:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.311+0000 7f8fc058f740 10 client.17244 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.311+0000 7f8fc058f740 1 -- 172.21.15.28:0/3219908339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55881601a5f0 con 0x558815f99580 2024-04-25T01:38:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.311+0000 7f8f9aff5640 1 -- 172.21.15.28:0/3219908339 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fa809c4d0 con 0x558815f99580 2024-04-25T01:38:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.311+0000 7f8f9aff5640 1 -- 172.21.15.28:0/3219908339 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8fa80b1070 con 0x558815f99580 2024-04-25T01:38:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.311+0000 7f8fc058f740 10 client.17244 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.311+0000 7f8fc058f740 10 client.17244 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.311+0000 7f8fc058f740 20 client.17244 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.311+0000 7f8fc058f740 1 --2- 172.21.15.28:0/3219908339 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55881601a820 0x55881601cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.311+0000 7f8fc058f740 4 client.17244 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:38:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.311+0000 7f8fc058f740 1 -- 172.21.15.28:0/3219908339 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558815f05f00 con 0x55881601a820 2024-04-25T01:38:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.312+0000 7f8fb67fc640 1 --2- 172.21.15.28:0/3219908339 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55881601a820 0x55881601cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.312+0000 7f8fb67fc640 1 --2- 172.21.15.28:0/3219908339 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55881601a820 0x55881601cc10 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f78e28b14e1a01 server_cookie=971de970fd1cb437 in_seq=0 out_seq=0 2024-04-25T01:38:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.312+0000 7f8f9aff5640 10 client.17244 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.312+0000 7f8f9aff5640 1 -- 172.21.15.28:0/3219908339 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558815f05f00 con 0x55881601a820 2024-04-25T01:38:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.312+0000 7f8f9aff5640 10 client.17244 handle_command_reply: tid=0 2024-04-25T01:38:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.312+0000 7f8fc058f740 1 client.17244 shutdown 2024-04-25T01:38:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.312+0000 7f8fc058f740 1 -- 172.21.15.28:0/3219908339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 msgr2=0x558815fac700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.312+0000 7f8fc058f740 1 --2- 172.21.15.28:0/3219908339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 0x558815fac700 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f8fa809bd10 tx=0x7f8fa80cd530 comp rx=0 tx=0).stop 2024-04-25T01:38:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.312+0000 7f8fc058f740 1 -- 172.21.15.28:0/3219908339 shutdown_connections 2024-04-25T01:38:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.312+0000 7f8fc058f740 1 --2- 172.21.15.28:0/3219908339 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55881601a820 0x55881601cc10 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.312+0000 7f8fc058f740 1 --2- 172.21.15.28:0/3219908339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558815fa4570 0x558815fa7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.312+0000 7f8fc058f740 1 --2- 172.21.15.28:0/3219908339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558815f99ea0 0x558815facc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.312+0000 7f8f9aff5640 0 client.17244 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.312+0000 7f8fc058f740 1 --2- 172.21.15.28:0/3219908339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558815f99580 0x558815fac700 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.312+0000 7f8fc058f740 1 -- 172.21.15.28:0/3219908339 >> 172.21.15.28:0/3219908339 conn(0x558815f958a0 msgr2=0x558815fa0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.312+0000 7f8fc058f740 1 -- 172.21.15.28:0/3219908339 shutdown_connections 2024-04-25T01:38:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.312+0000 7f8fc058f740 1 -- 172.21.15.28:0/3219908339 wait complete. 2024-04-25T01:38:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.313+0000 7f8fc058f740 20 client.17244 trim_cache size 0 max 16384 2024-04-25T01:38:04.299 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:38:04.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.313+0000 7f8fbd7a5640 1 -- 172.21.15.28:0/2466433098 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8fa00a2d00 msgr2=0x7f8fa00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.313+0000 7f8fbd7a5640 1 --2- 172.21.15.28:0/2466433098 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8fa00a2d00 0x7f8fa00a51c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8fb0004ab0 tx=0x7f8fb0009800 comp rx=0 tx=0).stop 2024-04-25T01:38:04.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.313+0000 7f8fbd7a5640 1 -- 172.21.15.28:0/2466433098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb8069dc0 msgr2=0x7f8fb8197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:04.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.313+0000 7f8fbd7a5640 1 --2- 172.21.15.28:0/2466433098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb8069dc0 0x7f8fb8197f60 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f8fa800e9f0 tx=0x7f8fa800edf0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.313+0000 7f8fbd7a5640 1 -- 172.21.15.28:0/2466433098 shutdown_connections 2024-04-25T01:38:04.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.313+0000 7f8fbd7a5640 1 --2- 172.21.15.28:0/2466433098 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8fa00a2d00 0x7f8fa00a51c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.313+0000 7f8fbd7a5640 1 --2- 172.21.15.28:0/2466433098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8fb8107730 0x7f8fb819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.313+0000 7f8fbd7a5640 1 --2- 172.21.15.28:0/2466433098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8fb806a780 0x7f8fb81984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.313+0000 7f8fbd7a5640 1 --2- 172.21.15.28:0/2466433098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb8069dc0 0x7f8fb8197f60 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:04.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.313+0000 7f8fbd7a5640 1 -- 172.21.15.28:0/2466433098 >> 172.21.15.28:0/2466433098 conn(0x7f8fb80776a0 msgr2=0x7f8fb80fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:04.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.313+0000 7f8fbd7a5640 1 -- 172.21.15.28:0/2466433098 shutdown_connections 2024-04-25T01:38:04.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:04.313+0000 7f8fbd7a5640 1 -- 172.21.15.28:0/2466433098 wait complete. 2024-04-25T01:38:05.303 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:38:05.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:38:05.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:38:05.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.443+0000 7f514d0da640 1 -- 172.21.15.28:0/1963989209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f514806a780 msgr2=0x7f514810e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.443+0000 7f514d0da640 1 --2- 172.21.15.28:0/1963989209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f514806a780 0x7f514810e5f0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f5140009cd0 tx=0x7f514002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.443+0000 7f514d0da640 1 -- 172.21.15.28:0/1963989209 shutdown_connections 2024-04-25T01:38:05.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.443+0000 7f514d0da640 1 --2- 172.21.15.28:0/1963989209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f514806a780 0x7f514810e5f0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.443+0000 7f514d0da640 1 --2- 172.21.15.28:0/1963989209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5148069dc0 0x7f514806a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.443+0000 7f514d0da640 1 --2- 172.21.15.28:0/1963989209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5148107730 0x7f5148107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.443+0000 7f514d0da640 1 -- 172.21.15.28:0/1963989209 >> 172.21.15.28:0/1963989209 conn(0x7f51480776a0 msgr2=0x7f5148077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:05.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.443+0000 7f514d0da640 1 -- 172.21.15.28:0/1963989209 shutdown_connections 2024-04-25T01:38:05.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.443+0000 7f514d0da640 1 -- 172.21.15.28:0/1963989209 wait complete. 2024-04-25T01:38:05.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.444+0000 7f514d0da640 1 Processor -- start 2024-04-25T01:38:05.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.444+0000 7f514d0da640 1 -- start start 2024-04-25T01:38:05.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.444+0000 7f514d0da640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5148069dc0 0x7f5148197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.444+0000 7f514d0da640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f514806a780 0x7f51481984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.444+0000 7f514d0da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5148107730 0x7f514819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.444+0000 7f514d0da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5148111b40 con 0x7f5148107730 2024-04-25T01:38:05.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.444+0000 7f514d0da640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f51481119c0 con 0x7f5148069dc0 2024-04-25T01:38:05.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.444+0000 7f514d0da640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f5148111cc0 con 0x7f514806a780 2024-04-25T01:38:05.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.445+0000 7f514759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5148107730 0x7f514819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.445+0000 7f514759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5148107730 0x7f514819c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43074/0 (socket says 172.21.15.28:43074) 2024-04-25T01:38:05.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.445+0000 7f514759e640 1 -- 172.21.15.28:0/1546045678 learned_addr learned my addr 172.21.15.28:0/1546045678 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:05.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.445+0000 7f5146d9d640 1 --2- 172.21.15.28:0/1546045678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5148069dc0 0x7f5148197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.445+0000 7f514659c640 1 --2- 172.21.15.28:0/1546045678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f514806a780 0x7f51481984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.445+0000 7f514759e640 1 -- 172.21.15.28:0/1546045678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f514806a780 msgr2=0x7f51481984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.445+0000 7f514759e640 1 --2- 172.21.15.28:0/1546045678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f514806a780 0x7f51481984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.445+0000 7f514759e640 1 -- 172.21.15.28:0/1546045678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5148069dc0 msgr2=0x7f5148197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.445+0000 7f514759e640 1 --2- 172.21.15.28:0/1546045678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5148069dc0 0x7f5148197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.445+0000 7f514759e640 1 -- 172.21.15.28:0/1546045678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f514819cf50 con 0x7f5148107730 2024-04-25T01:38:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.445+0000 7f514759e640 1 --2- 172.21.15.28:0/1546045678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5148107730 0x7f514819c820 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f5140005cc0 tx=0x7f5140003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.445+0000 7f512ffff640 1 -- 172.21.15.28:0/1546045678 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5140004040 con 0x7f5148107730 2024-04-25T01:38:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.445+0000 7f514d0da640 1 -- 172.21.15.28:0/1546045678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f514819d1e0 con 0x7f5148107730 2024-04-25T01:38:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.445+0000 7f512ffff640 1 -- 172.21.15.28:0/1546045678 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51400046c0 con 0x7f5148107730 2024-04-25T01:38:05.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.445+0000 7f512ffff640 1 -- 172.21.15.28:0/1546045678 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5140046480 con 0x7f5148107730 2024-04-25T01:38:05.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.445+0000 7f514d0da640 1 -- 172.21.15.28:0/1546045678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51481a4ad0 con 0x7f5148107730 2024-04-25T01:38:05.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.448+0000 7f512ffff640 1 -- 172.21.15.28:0/1546045678 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f51400305a0 con 0x7f5148107730 2024-04-25T01:38:05.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.448+0000 7f514fdec740 1 Processor -- start 2024-04-25T01:38:05.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.448+0000 7f514fdec740 1 -- start start 2024-04-25T01:38:05.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.448+0000 7f514fdec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865db980 0x55be865d9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.448+0000 7f512ffff640 1 --2- 172.21.15.28:0/1546045678 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f51300a2d00 0x7f51300a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.448+0000 7f514fdec740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865da290 0x55be865da710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.448+0000 7f514fdec740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865e9420 0x55be865eb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.448+0000 7f514fdec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55be8653e2c0 con 0x55be865db980 2024-04-25T01:38:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.448+0000 7f514fdec740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55be864e6910 con 0x55be865da290 2024-04-25T01:38:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.448+0000 7f514fdec740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55be8653a280 con 0x55be865e9420 2024-04-25T01:38:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.448+0000 7f514659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865db980 0x55be865d9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.448+0000 7f5146d9d640 1 --2- 172.21.15.28:0/1546045678 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f51300a2d00 0x7f51300a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.448+0000 7f512ffff640 1 -- 172.21.15.28:0/1546045678 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f514002e030 con 0x7f5148107730 2024-04-25T01:38:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.448+0000 7f514659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865db980 0x55be865d9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43086/0 (socket says 172.21.15.28:43086) 2024-04-25T01:38:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.449+0000 7f514659c640 1 -- 172.21.15.28:0/999865919 learned_addr learned my addr 172.21.15.28:0/999865919 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.449+0000 7f514759e640 1 --2- 172.21.15.28:0/999865919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865e9420 0x55be865eb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.449+0000 7f514659c640 1 --2- 172.21.15.28:0/999865919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865da290 0x55be865da710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.449+0000 7f514659c640 1 -- 172.21.15.28:0/999865919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865e9420 msgr2=0x55be865eb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:05.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.449+0000 7f514659c640 1 --2- 172.21.15.28:0/999865919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865e9420 0x55be865eb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.449+0000 7f514659c640 1 -- 172.21.15.28:0/999865919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865da290 msgr2=0x55be865da710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.449+0000 7f514659c640 1 --2- 172.21.15.28:0/999865919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865da290 0x55be865da710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.449+0000 7f514659c640 1 -- 172.21.15.28:0/999865919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be864f7fb0 con 0x55be865db980 2024-04-25T01:38:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.449+0000 7f5146d9d640 1 --2- 172.21.15.28:0/1546045678 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f51300a2d00 0x7f51300a51c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5138005fd0 tx=0x7f513800b390 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.449+0000 7f514659c640 1 --2- 172.21.15.28:0/999865919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865db980 0x55be865d9cc0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f513c00bfc0 tx=0x7f513c00b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd90ac5f740a1574 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.449+0000 7f512d7fa640 1 -- 172.21.15.28:0/999865919 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c019030 con 0x55be865db980 2024-04-25T01:38:05.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.449+0000 7f512d7fa640 1 -- 172.21.15.28:0/999865919 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f513c0119a0 con 0x55be865db980 2024-04-25T01:38:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.449+0000 7f512d7fa640 1 -- 172.21.15.28:0/999865919 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c01d820 con 0x55be865db980 2024-04-25T01:38:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.450+0000 7f514fdec740 1 -- 172.21.15.28:0/999865919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865db980 msgr2=0x55be865d9cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.450+0000 7f514fdec740 1 --2- 172.21.15.28:0/999865919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865db980 0x55be865d9cc0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f513c00bfc0 tx=0x7f513c00b670 comp rx=0 tx=0).stop 2024-04-25T01:38:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.450+0000 7f514fdec740 1 -- 172.21.15.28:0/999865919 shutdown_connections 2024-04-25T01:38:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.450+0000 7f514fdec740 1 --2- 172.21.15.28:0/999865919 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865e9420 0x55be865eb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.450+0000 7f514fdec740 1 --2- 172.21.15.28:0/999865919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865da290 0x55be865da710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.450+0000 7f514fdec740 1 --2- 172.21.15.28:0/999865919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865db980 0x55be865d9cc0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.450+0000 7f514fdec740 1 -- 172.21.15.28:0/999865919 >> 172.21.15.28:0/999865919 conn(0x55be865a2950 msgr2=0x55be865d67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.450+0000 7f514fdec740 1 -- 172.21.15.28:0/999865919 shutdown_connections 2024-04-25T01:38:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.450+0000 7f514fdec740 1 -- 172.21.15.28:0/999865919 wait complete. 2024-04-25T01:38:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.450+0000 7f514fdec740 1 -- start start 2024-04-25T01:38:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.450+0000 7f514fdec740 1 Processor -- start 2024-04-25T01:38:05.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f514fdec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865da290 0x55be865e6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f514fdec740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865db980 0x55be865e6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f514fdec740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865e9420 0x55be865e6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f514fdec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55be865ec2b0 con 0x55be865da290 2024-04-25T01:38:05.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f514fdec740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55be864f9e90 con 0x55be865e9420 2024-04-25T01:38:05.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f514fdec740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55be865ec430 con 0x55be865db980 2024-04-25T01:38:05.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f514fdec740 10 client.0 _ll_register_callbacks cb 0x55be86535970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:05.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f5146d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865da290 0x55be865e6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f5146d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865da290 0x55be865e6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43090/0 (socket says 172.21.15.28:43090) 2024-04-25T01:38:05.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f5146d9d640 1 -- 172.21.15.28:0/2838489720 learned_addr learned my addr 172.21.15.28:0/2838489720 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:05.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f514659c640 1 --2- 172.21.15.28:0/2838489720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865db980 0x55be865e6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f5146d9d640 1 -- 172.21.15.28:0/2838489720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865db980 msgr2=0x55be865e6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:05.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f5146d9d640 1 --2- 172.21.15.28:0/2838489720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865db980 0x55be865e6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f5146d9d640 1 -- 172.21.15.28:0/2838489720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865e9420 msgr2=0x55be865e6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:05.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f514759e640 1 --2- 172.21.15.28:0/2838489720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865e9420 0x55be865e6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f5146d9d640 1 --2- 172.21.15.28:0/2838489720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865e9420 0x55be865e6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f5146d9d640 1 -- 172.21.15.28:0/2838489720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be8631a7e0 con 0x55be865da290 2024-04-25T01:38:05.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f514759e640 1 --2- 172.21.15.28:0/2838489720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865e9420 0x55be865e6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:05.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f5146d9d640 1 --2- 172.21.15.28:0/2838489720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865da290 0x55be865e6310 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f5138003e20 tx=0x7f5138004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d72f7d81810306a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:05.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f512aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:05.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f512aff5640 1 -- 172.21.15.28:0/2838489720 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513801e620 con 0x55be865da290 2024-04-25T01:38:05.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f512aff5640 1 -- 172.21.15.28:0/2838489720 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f513801e7c0 con 0x55be865da290 2024-04-25T01:38:05.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.451+0000 7f514fdec740 1 -- 172.21.15.28:0/2838489720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f513c0119a0 con 0x55be865da290 2024-04-25T01:38:05.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.452+0000 7f512aff5640 1 -- 172.21.15.28:0/2838489720 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f513802f050 con 0x55be865da290 2024-04-25T01:38:05.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.452+0000 7f514fdec740 10 client.17256 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:05.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.452+0000 7f514fdec740 1 -- 172.21.15.28:0/2838489720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55be8637af70 con 0x55be865da290 2024-04-25T01:38:05.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.452+0000 7f512aff5640 1 -- 172.21.15.28:0/2838489720 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f513c0119a0 con 0x55be865da290 2024-04-25T01:38:05.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.452+0000 7f512aff5640 1 -- 172.21.15.28:0/2838489720 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f513801cba0 con 0x55be865da290 2024-04-25T01:38:05.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.452+0000 7f514fdec740 10 client.17256 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:05.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.452+0000 7f514fdec740 10 client.17256 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:05.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.452+0000 7f514fdec740 20 client.17256 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:05.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.452+0000 7f514fdec740 1 --2- 172.21.15.28:0/2838489720 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55be865f95d0 0x55be865fb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.452+0000 7f514fdec740 4 client.17256 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.452+0000 7f514fdec740 1 -- 172.21.15.28:0/2838489720 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be865125a0 con 0x55be865f95d0 2024-04-25T01:38:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.452+0000 7f514659c640 1 --2- 172.21.15.28:0/2838489720 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55be865f95d0 0x55be865fb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.453+0000 7f514659c640 1 --2- 172.21.15.28:0/2838489720 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55be865f95d0 0x55be865fb9c0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3aa0a2e8a5cd11ca server_cookie=75abf1a9805f5871 in_seq=0 out_seq=0 2024-04-25T01:38:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.453+0000 7f512aff5640 10 client.17256 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.453+0000 7f512aff5640 1 -- 172.21.15.28:0/2838489720 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55be865125a0 con 0x55be865f95d0 2024-04-25T01:38:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.453+0000 7f512aff5640 10 client.17256 handle_command_reply: tid=0 2024-04-25T01:38:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.453+0000 7f514fdec740 1 client.17256 shutdown 2024-04-25T01:38:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.454+0000 7f514fdec740 1 -- 172.21.15.28:0/2838489720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865da290 msgr2=0x55be865e6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.454+0000 7f514fdec740 1 --2- 172.21.15.28:0/2838489720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865da290 0x55be865e6310 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f5138003e20 tx=0x7f5138004730 comp rx=0 tx=0).stop 2024-04-25T01:38:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.454+0000 7f514fdec740 1 -- 172.21.15.28:0/2838489720 shutdown_connections 2024-04-25T01:38:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.454+0000 7f514fdec740 1 --2- 172.21.15.28:0/2838489720 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55be865f95d0 0x55be865fb9c0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.454+0000 7f514fdec740 1 --2- 172.21.15.28:0/2838489720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865e9420 0x55be865e6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.454+0000 7f514fdec740 1 --2- 172.21.15.28:0/2838489720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865db980 0x55be865e6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.454+0000 7f514fdec740 1 --2- 172.21.15.28:0/2838489720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865da290 0x55be865e6310 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.454+0000 7f512aff5640 0 client.17256 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.454+0000 7f514fdec740 1 -- 172.21.15.28:0/2838489720 >> 172.21.15.28:0/2838489720 conn(0x55be865a2950 msgr2=0x55be865dc5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.454+0000 7f514fdec740 1 -- 172.21.15.28:0/2838489720 shutdown_connections 2024-04-25T01:38:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.454+0000 7f514fdec740 1 -- 172.21.15.28:0/2838489720 wait complete. 2024-04-25T01:38:05.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.454+0000 7f514fdec740 20 client.17256 trim_cache size 0 max 16384 2024-04-25T01:38:05.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.472+0000 7f514fdec740 1 Processor -- start 2024-04-25T01:38:05.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.472+0000 7f514fdec740 1 -- start start 2024-04-25T01:38:05.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.472+0000 7f514fdec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865da290 0x55be865f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.472+0000 7f514fdec740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865db980 0x55be866012d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.472+0000 7f514fdec740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865e9420 0x55be86601810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.472+0000 7f514fdec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51481ac7d0 con 0x55be865da290 2024-04-25T01:38:05.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.472+0000 7f514fdec740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f51481ae310 con 0x55be865db980 2024-04-25T01:38:05.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.472+0000 7f514fdec740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f51481acea0 con 0x55be865e9420 2024-04-25T01:38:05.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.472+0000 7f5146d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865da290 0x55be865f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.472+0000 7f5146d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865da290 0x55be865f51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43104/0 (socket says 172.21.15.28:43104) 2024-04-25T01:38:05.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.472+0000 7f5146d9d640 1 -- 172.21.15.28:0/2619005275 learned_addr learned my addr 172.21.15.28:0/2619005275 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.472+0000 7f514759e640 1 --2- 172.21.15.28:0/2619005275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865e9420 0x55be86601810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.472+0000 7f5146d9d640 1 -- 172.21.15.28:0/2619005275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865e9420 msgr2=0x55be86601810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.472+0000 7f5146d9d640 1 --2- 172.21.15.28:0/2619005275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865e9420 0x55be86601810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.472+0000 7f5146d9d640 1 -- 172.21.15.28:0/2619005275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865db980 msgr2=0x55be866012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.472+0000 7f514659c640 1 --2- 172.21.15.28:0/2619005275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865db980 0x55be866012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.472+0000 7f5146d9d640 1 --2- 172.21.15.28:0/2619005275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865db980 0x55be866012d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.472+0000 7f5146d9d640 1 -- 172.21.15.28:0/2619005275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be86212170 con 0x55be865da290 2024-04-25T01:38:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.473+0000 7f5146d9d640 1 --2- 172.21.15.28:0/2619005275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865da290 0x55be865f51b0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f5138005d80 tx=0x7f5138003fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=730f424b91578a97 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.473+0000 7f512cff9640 1 -- 172.21.15.28:0/2619005275 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513801d520 con 0x55be865da290 2024-04-25T01:38:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.473+0000 7f512cff9640 1 -- 172.21.15.28:0/2619005275 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f513801d6c0 con 0x55be865da290 2024-04-25T01:38:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.473+0000 7f512cff9640 1 -- 172.21.15.28:0/2619005275 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513801ccf0 con 0x55be865da290 2024-04-25T01:38:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.473+0000 7f514fdec740 1 -- 172.21.15.28:0/2619005275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865da290 msgr2=0x55be865f51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.473+0000 7f514fdec740 1 --2- 172.21.15.28:0/2619005275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865da290 0x55be865f51b0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f5138005d80 tx=0x7f5138003fd0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.473+0000 7f514fdec740 1 -- 172.21.15.28:0/2619005275 shutdown_connections 2024-04-25T01:38:05.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.473+0000 7f514fdec740 1 --2- 172.21.15.28:0/2619005275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865e9420 0x55be86601810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.473+0000 7f514fdec740 1 --2- 172.21.15.28:0/2619005275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865db980 0x55be866012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.473+0000 7f514fdec740 1 --2- 172.21.15.28:0/2619005275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865da290 0x55be865f51b0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.473+0000 7f514fdec740 1 -- 172.21.15.28:0/2619005275 >> 172.21.15.28:0/2619005275 conn(0x55be865a2950 msgr2=0x55be865e4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:05.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.473+0000 7f514fdec740 1 -- 172.21.15.28:0/2619005275 shutdown_connections 2024-04-25T01:38:05.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.473+0000 7f514fdec740 1 -- 172.21.15.28:0/2619005275 wait complete. 2024-04-25T01:38:05.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f514fdec740 1 -- start start 2024-04-25T01:38:05.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f514fdec740 1 Processor -- start 2024-04-25T01:38:05.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f514fdec740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865da290 0x55be865e32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f514fdec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865db980 0x55be865e3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f514fdec740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865e9420 0x55be865e3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f514fdec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51481adc40 con 0x55be865db980 2024-04-25T01:38:05.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f514fdec740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f51481ad570 con 0x55be865e9420 2024-04-25T01:38:05.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f514fdec740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f51481aa7b0 con 0x55be865da290 2024-04-25T01:38:05.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f514fdec740 10 client.0 _ll_register_callbacks cb 0x7f5138005c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:05.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f514659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865db980 0x55be865e3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f514659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865db980 0x55be865e3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43120/0 (socket says 172.21.15.28:43120) 2024-04-25T01:38:05.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f514659c640 1 -- 172.21.15.28:0/2444136994 learned_addr learned my addr 172.21.15.28:0/2444136994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:05.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f514759e640 1 --2- 172.21.15.28:0/2444136994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865e9420 0x55be865e3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f5146d9d640 1 --2- 172.21.15.28:0/2444136994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865da290 0x55be865e32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f514659c640 1 -- 172.21.15.28:0/2444136994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865da290 msgr2=0x55be865e32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:05.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f514659c640 1 --2- 172.21.15.28:0/2444136994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865da290 0x55be865e32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f514659c640 1 -- 172.21.15.28:0/2444136994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865e9420 msgr2=0x55be865e3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:05.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f514659c640 1 --2- 172.21.15.28:0/2444136994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865e9420 0x55be865e3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f514659c640 1 -- 172.21.15.28:0/2444136994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be8665ede0 con 0x55be865db980 2024-04-25T01:38:05.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f5146d9d640 1 --2- 172.21.15.28:0/2444136994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865da290 0x55be865e32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:05.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.474+0000 7f514759e640 1 --2- 172.21.15.28:0/2444136994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865e9420 0x55be865e3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:05.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.475+0000 7f514659c640 1 --2- 172.21.15.28:0/2444136994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865db980 0x55be865e3810 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f513c01d430 tx=0x7f513c01d500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e8cd22dc44360d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.475+0000 7f512aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.475+0000 7f512aff5640 1 -- 172.21.15.28:0/2444136994 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c02e060 con 0x55be865db980 2024-04-25T01:38:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.475+0000 7f514fdec740 1 -- 172.21.15.28:0/2444136994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f513801d6c0 con 0x55be865db980 2024-04-25T01:38:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.475+0000 7f512aff5640 1 -- 172.21.15.28:0/2444136994 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f513c004230 con 0x55be865db980 2024-04-25T01:38:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.475+0000 7f512aff5640 1 -- 172.21.15.28:0/2444136994 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f513c035050 con 0x55be865db980 2024-04-25T01:38:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.475+0000 7f514fdec740 10 client.17262 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.475+0000 7f514fdec740 1 -- 172.21.15.28:0/2444136994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55be86660100 con 0x55be865db980 2024-04-25T01:38:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.475+0000 7f512aff5640 1 -- 172.21.15.28:0/2444136994 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f513801d6c0 con 0x55be865db980 2024-04-25T01:38:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.475+0000 7f512aff5640 1 -- 172.21.15.28:0/2444136994 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f513c0284a0 con 0x55be865db980 2024-04-25T01:38:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.475+0000 7f514fdec740 10 client.17262 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.475+0000 7f514fdec740 10 client.17262 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.475+0000 7f514fdec740 20 client.17262 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.475+0000 7f514fdec740 1 --2- 172.21.15.28:0/2444136994 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55be86660550 0x55be86662940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.475+0000 7f514fdec740 4 client.17262 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:38:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.475+0000 7f514fdec740 1 -- 172.21.15.28:0/2444136994 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55be8654b800 con 0x55be86660550 2024-04-25T01:38:05.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.476+0000 7f5146d9d640 1 --2- 172.21.15.28:0/2444136994 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55be86660550 0x55be86662940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.476+0000 7f5146d9d640 1 --2- 172.21.15.28:0/2444136994 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55be86660550 0x55be86662940 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=852e29051e57be99 server_cookie=5ffd3ac4d4473501 in_seq=0 out_seq=0 2024-04-25T01:38:05.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.476+0000 7f512aff5640 10 client.17262 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:05.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.477+0000 7f512aff5640 1 -- 172.21.15.28:0/2444136994 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55be8654b800 con 0x55be86660550 2024-04-25T01:38:05.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.477+0000 7f512aff5640 10 client.17262 handle_command_reply: tid=0 2024-04-25T01:38:05.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.477+0000 7f514fdec740 1 client.17262 shutdown 2024-04-25T01:38:05.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.477+0000 7f514fdec740 1 -- 172.21.15.28:0/2444136994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865db980 msgr2=0x55be865e3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.477+0000 7f514fdec740 1 --2- 172.21.15.28:0/2444136994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865db980 0x55be865e3810 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f513c01d430 tx=0x7f513c01d500 comp rx=0 tx=0).stop 2024-04-25T01:38:05.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.477+0000 7f514fdec740 1 -- 172.21.15.28:0/2444136994 shutdown_connections 2024-04-25T01:38:05.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.477+0000 7f514fdec740 1 --2- 172.21.15.28:0/2444136994 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55be86660550 0x55be86662940 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.477+0000 7f514fdec740 1 --2- 172.21.15.28:0/2444136994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be865e9420 0x55be865e3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.477+0000 7f512aff5640 0 client.17262 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:05.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.477+0000 7f514fdec740 1 --2- 172.21.15.28:0/2444136994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be865db980 0x55be865e3810 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.478+0000 7f514fdec740 1 --2- 172.21.15.28:0/2444136994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55be865da290 0x55be865e32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.478+0000 7f514fdec740 1 -- 172.21.15.28:0/2444136994 >> 172.21.15.28:0/2444136994 conn(0x55be865a2950 msgr2=0x55be865d50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:05.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.478+0000 7f514fdec740 1 -- 172.21.15.28:0/2444136994 shutdown_connections 2024-04-25T01:38:05.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.478+0000 7f514fdec740 1 -- 172.21.15.28:0/2444136994 wait complete. 2024-04-25T01:38:05.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.478+0000 7f514fdec740 20 client.17262 trim_cache size 0 max 16384 2024-04-25T01:38:05.463 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:38:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:38:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "a951342f-628d-45ed-b974-6f395a60a19a", 2024-04-25T01:38:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:38:05.463 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:38:05.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.478+0000 7f514d0da640 1 -- 172.21.15.28:0/1546045678 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f51300a2d00 msgr2=0x7f51300a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.478+0000 7f514d0da640 1 --2- 172.21.15.28:0/1546045678 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f51300a2d00 0x7f51300a51c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5138005fd0 tx=0x7f513800b390 comp rx=0 tx=0).stop 2024-04-25T01:38:05.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.478+0000 7f514d0da640 1 -- 172.21.15.28:0/1546045678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5148107730 msgr2=0x7f514819c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.478+0000 7f514d0da640 1 --2- 172.21.15.28:0/1546045678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5148107730 0x7f514819c820 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f5140005cc0 tx=0x7f5140003970 comp rx=0 tx=0).stop 2024-04-25T01:38:05.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.478+0000 7f514d0da640 1 -- 172.21.15.28:0/1546045678 shutdown_connections 2024-04-25T01:38:05.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.478+0000 7f514d0da640 1 --2- 172.21.15.28:0/1546045678 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f51300a2d00 0x7f51300a51c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.478+0000 7f514d0da640 1 --2- 172.21.15.28:0/1546045678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5148107730 0x7f514819c820 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.478+0000 7f514d0da640 1 --2- 172.21.15.28:0/1546045678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f514806a780 0x7f51481984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.478+0000 7f514d0da640 1 --2- 172.21.15.28:0/1546045678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5148069dc0 0x7f5148197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.478+0000 7f514d0da640 1 -- 172.21.15.28:0/1546045678 >> 172.21.15.28:0/1546045678 conn(0x7f51480776a0 msgr2=0x7f51480fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:05.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.478+0000 7f514d0da640 1 -- 172.21.15.28:0/1546045678 shutdown_connections 2024-04-25T01:38:05.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.478+0000 7f514d0da640 1 -- 172.21.15.28:0/1546045678 wait complete. 2024-04-25T01:38:05.469 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:38:05.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:38:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.594+0000 7f0ee12f5640 1 -- 172.21.15.28:0/1692111408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc100fe0 msgr2=0x7f0edc10d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.594+0000 7f0ee12f5640 1 --2- 172.21.15.28:0/1692111408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc100fe0 0x7f0edc10d4f0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4009cd0 tx=0x7f0ed402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.594+0000 7f0ee12f5640 1 -- 172.21.15.28:0/1692111408 shutdown_connections 2024-04-25T01:38:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.594+0000 7f0ee12f5640 1 --2- 172.21.15.28:0/1692111408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc100fe0 0x7f0edc10d4f0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.594+0000 7f0ee12f5640 1 --2- 172.21.15.28:0/1692111408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0edc100620 0x7f0edc100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.594+0000 7f0ee12f5640 1 --2- 172.21.15.28:0/1692111408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0edc106630 0x7f0edc106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.594+0000 7f0ee12f5640 1 -- 172.21.15.28:0/1692111408 >> 172.21.15.28:0/1692111408 conn(0x7f0edc0776a0 msgr2=0x7f0edc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.594+0000 7f0ee12f5640 1 -- 172.21.15.28:0/1692111408 shutdown_connections 2024-04-25T01:38:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.594+0000 7f0ee12f5640 1 -- 172.21.15.28:0/1692111408 wait complete. 2024-04-25T01:38:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.595+0000 7f0ee12f5640 1 Processor -- start 2024-04-25T01:38:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.595+0000 7f0ee12f5640 1 -- start start 2024-04-25T01:38:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.595+0000 7f0ee12f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc100620 0x7f0edc197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.595+0000 7f0ee12f5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0edc100fe0 0x7f0edc198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.595+0000 7f0ee12f5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0edc106630 0x7f0edc19c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.595+0000 7f0ee12f5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc110a40 con 0x7f0edc100620 2024-04-25T01:38:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.595+0000 7f0ee12f5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc1108c0 con 0x7f0edc100fe0 2024-04-25T01:38:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.595+0000 7f0ee12f5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc110bc0 con 0x7f0edc106630 2024-04-25T01:38:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.595+0000 7f0edaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc100620 0x7f0edc197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.595+0000 7f0edaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc100620 0x7f0edc197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43136/0 (socket says 172.21.15.28:43136) 2024-04-25T01:38:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.595+0000 7f0edaffd640 1 -- 172.21.15.28:0/735500423 learned_addr learned my addr 172.21.15.28:0/735500423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.595+0000 7f0edaffd640 1 -- 172.21.15.28:0/735500423 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0edc106630 msgr2=0x7f0edc19c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.595+0000 7f0edaffd640 1 --2- 172.21.15.28:0/735500423 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0edc106630 0x7f0edc19c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.595+0000 7f0edaffd640 1 -- 172.21.15.28:0/735500423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0edc100fe0 msgr2=0x7f0edc198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.595+0000 7f0ed3fff640 1 --2- 172.21.15.28:0/735500423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0edc100fe0 0x7f0edc198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.595+0000 7f0edaffd640 1 --2- 172.21.15.28:0/735500423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0edc100fe0 0x7f0edc198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.595+0000 7f0edaffd640 1 -- 172.21.15.28:0/735500423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0edc19cfe0 con 0x7f0edc100620 2024-04-25T01:38:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.596+0000 7f0edaffd640 1 --2- 172.21.15.28:0/735500423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc100620 0x7f0edc197ff0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00d950 tx=0x7f0ecc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.596+0000 7f0ed8ff9640 1 -- 172.21.15.28:0/735500423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc017030 con 0x7f0edc100620 2024-04-25T01:38:05.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.596+0000 7f0ed8ff9640 1 -- 172.21.15.28:0/735500423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ecc004020 con 0x7f0edc100620 2024-04-25T01:38:05.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.596+0000 7f0ed8ff9640 1 -- 172.21.15.28:0/735500423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc0047a0 con 0x7f0edc100620 2024-04-25T01:38:05.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.596+0000 7f0ee12f5640 1 -- 172.21.15.28:0/735500423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0edc19d2d0 con 0x7f0edc100620 2024-04-25T01:38:05.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.596+0000 7f0ee12f5640 1 -- 172.21.15.28:0/735500423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0edc1a4b60 con 0x7f0edc100620 2024-04-25T01:38:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.598+0000 7f0ed8ff9640 1 -- 172.21.15.28:0/735500423 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0ecc0041d0 con 0x7f0edc100620 2024-04-25T01:38:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.598+0000 7f0ee3ffe740 1 Processor -- start 2024-04-25T01:38:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.598+0000 7f0ee3ffe740 1 -- start start 2024-04-25T01:38:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0ee3ffe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 0x55e3e9b7e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0ee3ffe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b7ee50 0x55e3e9b8d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0ee3ffe740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b8da30 0x55e3e9b8fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0ee3ffe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e3e9ae36b0 con 0x55e3e9b7e530 2024-04-25T01:38:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0ee3ffe740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e3e9a8b880 con 0x55e3e9b8da30 2024-04-25T01:38:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0ee3ffe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55e3e9adf670 con 0x55e3e9b7ee50 2024-04-25T01:38:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0ed8ff9640 1 --2- 172.21.15.28:0/735500423 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0ec80a2d00 0x7f0ec80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0edb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 0x55e3e9b7e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0edb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 0x55e3e9b7e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43144/0 (socket says 172.21.15.28:43144) 2024-04-25T01:38:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0edb7fe640 1 -- 172.21.15.28:0/2667720842 learned_addr learned my addr 172.21.15.28:0/2667720842 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/735500423 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0ec80a2d00 0x7f0ec80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0ed3fff640 1 --2- 172.21.15.28:0/2667720842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b8da30 0x55e3e9b8fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0ed8ff9640 1 -- 172.21.15.28:0/735500423 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f0ecc0cf200 con 0x7f0edc100620 2024-04-25T01:38:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0edaffd640 1 --2- 172.21.15.28:0/2667720842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b7ee50 0x55e3e9b8d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0edb7fe640 1 -- 172.21.15.28:0/2667720842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b7ee50 msgr2=0x55e3e9b8d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/2667720842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b7ee50 0x55e3e9b8d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0edb7fe640 1 -- 172.21.15.28:0/2667720842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b8da30 msgr2=0x55e3e9b8fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/2667720842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b8da30 0x55e3e9b8fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0edb7fe640 1 -- 172.21.15.28:0/2667720842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3e98c17d0 con 0x55e3e9b7e530 2024-04-25T01:38:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.599+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/2667720842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 0x55e3e9b7e910 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4004af0 tx=0x7f0ed40318c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b44c6a3f3afa5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.600+0000 7f0ed17fa640 1 -- 172.21.15.28:0/2667720842 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4031d10 con 0x55e3e9b7e530 2024-04-25T01:38:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.600+0000 7f0ed17fa640 1 -- 172.21.15.28:0/2667720842 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ed4036040 con 0x55e3e9b7e530 2024-04-25T01:38:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.600+0000 7f0ed17fa640 1 -- 172.21.15.28:0/2667720842 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4030900 con 0x55e3e9b7e530 2024-04-25T01:38:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.600+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/735500423 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0ec80a2d00 0x7f0ec80a51c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0ed404b000 tx=0x7f0ed4049000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.600+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/2667720842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 msgr2=0x55e3e9b7e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.600+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/2667720842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 0x55e3e9b7e910 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4004af0 tx=0x7f0ed40318c0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.600+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/2667720842 shutdown_connections 2024-04-25T01:38:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.600+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/2667720842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b8da30 0x55e3e9b8fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.600+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/2667720842 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b7ee50 0x55e3e9b8d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.600+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/2667720842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 0x55e3e9b7e910 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.600+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/2667720842 >> 172.21.15.28:0/2667720842 conn(0x55e3e9b7a950 msgr2=0x55e3e9b7ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.601+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/2667720842 shutdown_connections 2024-04-25T01:38:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.601+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/2667720842 wait complete. 2024-04-25T01:38:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.601+0000 7f0ee3ffe740 1 -- start start 2024-04-25T01:38:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.601+0000 7f0ee3ffe740 1 Processor -- start 2024-04-25T01:38:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.601+0000 7f0ee3ffe740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b7e530 0x55e3e9b8a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.601+0000 7f0ee3ffe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7ee50 0x55e3e9b8af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.601+0000 7f0ee3ffe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b8da30 0x55e3e9b8b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.601+0000 7f0ee3ffe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e3e9b907f0 con 0x55e3e9b7ee50 2024-04-25T01:38:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.601+0000 7f0ee3ffe740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e3e9a9ef50 con 0x55e3e9b7e530 2024-04-25T01:38:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.601+0000 7f0ee3ffe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55e3e9b90970 con 0x55e3e9b8da30 2024-04-25T01:38:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.601+0000 7f0ee3ffe740 10 client.0 _ll_register_callbacks cb 0x55e3e9ada000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.601+0000 7f0ed3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7ee50 0x55e3e9b8af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.601+0000 7f0ed3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7ee50 0x55e3e9b8af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43156/0 (socket says 172.21.15.28:43156) 2024-04-25T01:38:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.601+0000 7f0ed3fff640 1 -- 172.21.15.28:0/2317133492 learned_addr learned my addr 172.21.15.28:0/2317133492 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.601+0000 7f0edaffd640 1 --2- 172.21.15.28:0/2317133492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b7e530 0x55e3e9b8a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/2317133492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b8da30 0x55e3e9b8b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0ed3fff640 1 -- 172.21.15.28:0/2317133492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b8da30 msgr2=0x55e3e9b8b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0ed3fff640 1 --2- 172.21.15.28:0/2317133492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b8da30 0x55e3e9b8b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0ed3fff640 1 -- 172.21.15.28:0/2317133492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b7e530 msgr2=0x55e3e9b8a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0ed3fff640 1 --2- 172.21.15.28:0/2317133492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b7e530 0x55e3e9b8a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0ed3fff640 1 -- 172.21.15.28:0/2317133492 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3e97f73f0 con 0x55e3e9b7ee50 2024-04-25T01:38:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/2317133492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b8da30 0x55e3e9b8b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0ed3fff640 1 --2- 172.21.15.28:0/2317133492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7ee50 0x55e3e9b8af00 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4009df0 tx=0x7f0ec40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f7d4da138f48453 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0ec27fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0ec27fc640 1 -- 172.21.15.28:0/2317133492 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4012860 con 0x55e3e9b7ee50 2024-04-25T01:38:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0ec27fc640 1 -- 172.21.15.28:0/2317133492 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f0ec4012a00 con 0x55e3e9b7ee50 2024-04-25T01:38:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/2317133492 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ed4036040 con 0x55e3e9b7ee50 2024-04-25T01:38:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0ec27fc640 1 -- 172.21.15.28:0/2317133492 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0ec4012c00 con 0x55e3e9b7ee50 2024-04-25T01:38:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0ee3ffe740 10 client.17274 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/2317133492 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e3e97b70f0 con 0x55e3e9b7ee50 2024-04-25T01:38:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0ec27fc640 1 -- 172.21.15.28:0/2317133492 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ed4036040 con 0x55e3e9b7ee50 2024-04-25T01:38:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0ec27fc640 1 -- 172.21.15.28:0/2317133492 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0ec400f040 con 0x55e3e9b7ee50 2024-04-25T01:38:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0ee3ffe740 10 client.17274 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0ee3ffe740 10 client.17274 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.602+0000 7f0ee3ffe740 20 client.17274 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.603+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/2317133492 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e3e9b9de20 0x55e3e9ba0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.603+0000 7f0ee3ffe740 4 client.17274 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.603+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/2317133492 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3e9af0bf0 con 0x55e3e9b9de20 2024-04-25T01:38:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.603+0000 7f0edaffd640 1 --2- 172.21.15.28:0/2317133492 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e3e9b9de20 0x55e3e9ba0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.603+0000 7f0edaffd640 1 --2- 172.21.15.28:0/2317133492 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e3e9b9de20 0x55e3e9ba0280 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da5610d6278211f4 server_cookie=aded3068a658374d in_seq=0 out_seq=0 2024-04-25T01:38:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.603+0000 7f0ec27fc640 10 client.17274 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.604+0000 7f0ec27fc640 1 -- 172.21.15.28:0/2317133492 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e3e9af0bf0 con 0x55e3e9b9de20 2024-04-25T01:38:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.604+0000 7f0ec27fc640 10 client.17274 handle_command_reply: tid=0 2024-04-25T01:38:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.604+0000 7f0ee3ffe740 1 client.17274 shutdown 2024-04-25T01:38:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.604+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/2317133492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7ee50 msgr2=0x55e3e9b8af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.604+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/2317133492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7ee50 0x55e3e9b8af00 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4009df0 tx=0x7f0ec40117b0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.604+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/2317133492 shutdown_connections 2024-04-25T01:38:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.604+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/2317133492 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e3e9b9de20 0x55e3e9ba0280 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.604+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/2317133492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b8da30 0x55e3e9b8b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.604+0000 7f0ec27fc640 0 client.17274 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.604+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/2317133492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7ee50 0x55e3e9b8af00 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.604+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/2317133492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b7e530 0x55e3e9b8a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.604+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/2317133492 >> 172.21.15.28:0/2317133492 conn(0x55e3e9b7a950 msgr2=0x55e3e9b80780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.604+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/2317133492 shutdown_connections 2024-04-25T01:38:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.604+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/2317133492 wait complete. 2024-04-25T01:38:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.604+0000 7f0ee3ffe740 20 client.17274 trim_cache size 0 max 16384 2024-04-25T01:38:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.621+0000 7f0ee3ffe740 1 Processor -- start 2024-04-25T01:38:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.622+0000 7f0ee3ffe740 1 -- start start 2024-04-25T01:38:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.622+0000 7f0ee3ffe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 0x55e3e9b996f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.622+0000 7f0ee3ffe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b7ee50 0x55e3e9b8b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.622+0000 7f0ee3ffe740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b8da30 0x55e3e9ba5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.622+0000 7f0ee3ffe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc1ac880 con 0x55e3e9b7e530 2024-04-25T01:38:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.622+0000 7f0ee3ffe740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc1ae3c0 con 0x55e3e9b8da30 2024-04-25T01:38:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.622+0000 7f0ee3ffe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc1acf50 con 0x55e3e9b7ee50 2024-04-25T01:38:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.622+0000 7f0edaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 0x55e3e9b996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.622+0000 7f0edaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 0x55e3e9b996f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43160/0 (socket says 172.21.15.28:43160) 2024-04-25T01:38:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.622+0000 7f0edaffd640 1 -- 172.21.15.28:0/2701385213 learned_addr learned my addr 172.21.15.28:0/2701385213 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.622+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/2701385213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b8da30 0x55e3e9ba5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.622+0000 7f0ed3fff640 1 --2- 172.21.15.28:0/2701385213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b7ee50 0x55e3e9b8b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.622+0000 7f0edaffd640 1 -- 172.21.15.28:0/2701385213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b7ee50 msgr2=0x55e3e9b8b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.622+0000 7f0edaffd640 1 --2- 172.21.15.28:0/2701385213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b7ee50 0x55e3e9b8b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.622+0000 7f0edaffd640 1 -- 172.21.15.28:0/2701385213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b8da30 msgr2=0x55e3e9ba5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.622+0000 7f0edaffd640 1 --2- 172.21.15.28:0/2701385213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b8da30 0x55e3e9ba5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.622+0000 7f0edaffd640 1 -- 172.21.15.28:0/2701385213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3e9b7ec80 con 0x55e3e9b7e530 2024-04-25T01:38:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.622+0000 7f0ed3fff640 1 --2- 172.21.15.28:0/2701385213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b7ee50 0x55e3e9b8b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.622+0000 7f0edaffd640 1 --2- 172.21.15.28:0/2701385213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 0x55e3e9b996f0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc009610 tx=0x7f0ecc0ce040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2558494d79088282 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.623+0000 7f0ed0ff9640 1 -- 172.21.15.28:0/2701385213 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc09b380 con 0x55e3e9b7e530 2024-04-25T01:38:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.623+0000 7f0ed0ff9640 1 -- 172.21.15.28:0/2701385213 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ecc09f730 con 0x55e3e9b7e530 2024-04-25T01:38:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.623+0000 7f0ed0ff9640 1 -- 172.21.15.28:0/2701385213 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc0a3b60 con 0x55e3e9b7e530 2024-04-25T01:38:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.623+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/2701385213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 msgr2=0x55e3e9b996f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.623+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/2701385213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 0x55e3e9b996f0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc009610 tx=0x7f0ecc0ce040 comp rx=0 tx=0).stop 2024-04-25T01:38:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.623+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/2701385213 shutdown_connections 2024-04-25T01:38:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.623+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/2701385213 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b8da30 0x55e3e9ba5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.623+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/2701385213 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b7ee50 0x55e3e9b8b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.623+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/2701385213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 0x55e3e9b996f0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.623+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/2701385213 >> 172.21.15.28:0/2701385213 conn(0x55e3e9b7a950 msgr2=0x55e3e9b8a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.623+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/2701385213 shutdown_connections 2024-04-25T01:38:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.623+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/2701385213 wait complete. 2024-04-25T01:38:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0ee3ffe740 1 -- start start 2024-04-25T01:38:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0ee3ffe740 1 Processor -- start 2024-04-25T01:38:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0ee3ffe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 0x55e3e9b96180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0ee3ffe740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b7ee50 0x55e3e9b914f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0ee3ffe740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b8da30 0x55e3e9b91c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0ee3ffe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc1adcf0 con 0x55e3e9b7e530 2024-04-25T01:38:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0ee3ffe740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc1ad620 con 0x55e3e9b7ee50 2024-04-25T01:38:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0ee3ffe740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc1aa860 con 0x55e3e9b8da30 2024-04-25T01:38:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0ee3ffe740 10 client.0 _ll_register_callbacks cb 0x7f0ec4009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0edaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 0x55e3e9b96180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0edaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 0x55e3e9b96180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43166/0 (socket says 172.21.15.28:43166) 2024-04-25T01:38:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0edaffd640 1 -- 172.21.15.28:0/3668184540 learned_addr learned my addr 172.21.15.28:0/3668184540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0ed3fff640 1 --2- 172.21.15.28:0/3668184540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b7ee50 0x55e3e9b914f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/3668184540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b8da30 0x55e3e9b91c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0edaffd640 1 -- 172.21.15.28:0/3668184540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b8da30 msgr2=0x55e3e9b91c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0edaffd640 1 --2- 172.21.15.28:0/3668184540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b8da30 0x55e3e9b91c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0edaffd640 1 -- 172.21.15.28:0/3668184540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b7ee50 msgr2=0x55e3e9b914f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0edaffd640 1 --2- 172.21.15.28:0/3668184540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b7ee50 0x55e3e9b914f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0edaffd640 1 -- 172.21.15.28:0/3668184540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3e9c033c0 con 0x55e3e9b7e530 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.624+0000 7f0edb7fe640 1 --2- 172.21.15.28:0/3668184540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b8da30 0x55e3e9b91c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.625+0000 7f0edaffd640 1 --2- 172.21.15.28:0/3668184540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 0x55e3e9b96180 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc0bafd0 tx=0x7f0ecc0097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=316a6e53bf399645 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.625+0000 7f0ec27fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.625+0000 7f0ec27fc640 1 -- 172.21.15.28:0/3668184540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc0979f0 con 0x55e3e9b7e530 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.625+0000 7f0ec27fc640 1 -- 172.21.15.28:0/3668184540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f0ecc0cc830 con 0x55e3e9b7e530 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.625+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/3668184540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ecc09f730 con 0x55e3e9b7e530 2024-04-25T01:38:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.625+0000 7f0ec27fc640 1 -- 172.21.15.28:0/3668184540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0ecc0a3b60 con 0x55e3e9b7e530 2024-04-25T01:38:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.625+0000 7f0ee3ffe740 10 client.17280 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.625+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/3668184540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e3e9c04500 con 0x55e3e9b7e530 2024-04-25T01:38:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.625+0000 7f0ec27fc640 1 -- 172.21.15.28:0/3668184540 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ecc09f730 con 0x55e3e9b7e530 2024-04-25T01:38:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.625+0000 7f0ec27fc640 1 -- 172.21.15.28:0/3668184540 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0ecc0660a0 con 0x55e3e9b7e530 2024-04-25T01:38:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.625+0000 7f0ee3ffe740 10 client.17280 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.625+0000 7f0ee3ffe740 10 client.17280 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.625+0000 7f0ee3ffe740 20 client.17280 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.625+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/3668184540 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e3e9c04be0 0x55e3e9c07000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.625+0000 7f0ee3ffe740 4 client.17280 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:38:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.625+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/3668184540 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e3e9aeaf40 con 0x55e3e9c04be0 2024-04-25T01:38:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.625+0000 7f0ed3fff640 1 --2- 172.21.15.28:0/3668184540 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e3e9c04be0 0x55e3e9c07000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.626+0000 7f0ed3fff640 1 --2- 172.21.15.28:0/3668184540 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e3e9c04be0 0x55e3e9c07000 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ecd1e5c3dd53205 server_cookie=5b91ea40a30e5e51 in_seq=0 out_seq=0 2024-04-25T01:38:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.626+0000 7f0ec27fc640 10 client.17280 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.626+0000 7f0ec27fc640 1 -- 172.21.15.28:0/3668184540 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e3e9aeaf40 con 0x55e3e9c04be0 2024-04-25T01:38:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.626+0000 7f0ec27fc640 10 client.17280 handle_command_reply: tid=0 2024-04-25T01:38:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.626+0000 7f0ee3ffe740 1 client.17280 shutdown 2024-04-25T01:38:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.626+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/3668184540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 msgr2=0x55e3e9b96180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.626+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/3668184540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 0x55e3e9b96180 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc0bafd0 tx=0x7f0ecc0097a0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.626+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/3668184540 shutdown_connections 2024-04-25T01:38:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.626+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/3668184540 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e3e9c04be0 0x55e3e9c07000 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.626+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/3668184540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e3e9b8da30 0x55e3e9b91c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.626+0000 7f0ec27fc640 0 client.17280 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.626+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/3668184540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e3e9b7ee50 0x55e3e9b914f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.626+0000 7f0ee3ffe740 1 --2- 172.21.15.28:0/3668184540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3e9b7e530 0x55e3e9b96180 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.626+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/3668184540 >> 172.21.15.28:0/3668184540 conn(0x55e3e9b7a950 msgr2=0x55e3e9b8b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.626+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/3668184540 shutdown_connections 2024-04-25T01:38:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.626+0000 7f0ee3ffe740 1 -- 172.21.15.28:0/3668184540 wait complete. 2024-04-25T01:38:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.626+0000 7f0ee3ffe740 20 client.17280 trim_cache size 0 max 16384 2024-04-25T01:38:05.612 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:38:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:38:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-25T01:38:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "a951342f-628d-45ed-b974-6f395a60a19a": { 2024-04-25T01:38:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-25T01:38:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "a951342f-628d-45ed-b974-6f395a60a19a", 2024-04-25T01:38:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-25T01:38:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:38:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:38:05.613 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:38:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.627+0000 7f0ee12f5640 1 -- 172.21.15.28:0/735500423 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0ec80a2d00 msgr2=0x7f0ec80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.627+0000 7f0ee12f5640 1 --2- 172.21.15.28:0/735500423 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0ec80a2d00 0x7f0ec80a51c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0ed404b000 tx=0x7f0ed4049000 comp rx=0 tx=0).stop 2024-04-25T01:38:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.627+0000 7f0ee12f5640 1 -- 172.21.15.28:0/735500423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc100620 msgr2=0x7f0edc197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.627+0000 7f0ee12f5640 1 --2- 172.21.15.28:0/735500423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc100620 0x7f0edc197ff0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00d950 tx=0x7f0ecc00de10 comp rx=0 tx=0).stop 2024-04-25T01:38:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.627+0000 7f0ee12f5640 1 -- 172.21.15.28:0/735500423 shutdown_connections 2024-04-25T01:38:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.627+0000 7f0ee12f5640 1 --2- 172.21.15.28:0/735500423 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0ec80a2d00 0x7f0ec80a51c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.627+0000 7f0ee12f5640 1 --2- 172.21.15.28:0/735500423 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0edc106630 0x7f0edc19c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.627+0000 7f0ee12f5640 1 --2- 172.21.15.28:0/735500423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0edc100fe0 0x7f0edc198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.627+0000 7f0ee12f5640 1 --2- 172.21.15.28:0/735500423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc100620 0x7f0edc197ff0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.627+0000 7f0ee12f5640 1 -- 172.21.15.28:0/735500423 >> 172.21.15.28:0/735500423 conn(0x7f0edc0776a0 msgr2=0x7f0edc069fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.627+0000 7f0ee12f5640 1 -- 172.21.15.28:0/735500423 shutdown_connections 2024-04-25T01:38:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:05.627+0000 7f0ee12f5640 1 -- 172.21.15.28:0/735500423 wait complete. 2024-04-25T01:38:05.617 INFO:tasks.cephfs.filesystem:scrub status for tag:a951342f-628d-45ed-b974-6f395a60a19a - {'path': '/', 'tag': 'a951342f-628d-45ed-b974-6f395a60a19a', 'options': 'recursive,force'} 2024-04-25T01:38:06.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:05 smithi028 ceph-mon[29390]: pgmap v155: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:38:06.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:05 smithi130 ceph-mon[32702]: pgmap v155: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:38:06.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:05 smithi032 ceph-mon[32461]: pgmap v155: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:38:07.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:06 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:38:07.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:06 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:38:07.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:06 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:38:07.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:06 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:38:07.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:06 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:38:07.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:06 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:38:07.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:06 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:38:07.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:06 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:38:07.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:06 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:38:07.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:06 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:38:07.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:06 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:38:07.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:06 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:38:08.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:07 smithi130 ceph-mon[32702]: pgmap v156: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-04-25T01:38:08.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:07 smithi032 ceph-mon[32461]: pgmap v156: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-04-25T01:38:08.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:07 smithi028 ceph-mon[29390]: pgmap v156: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-04-25T01:38:09.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:08 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:38:09.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:08 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:38:09.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:08 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:38:10.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:09 smithi130 ceph-mon[32702]: pgmap v157: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-04-25T01:38:10.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:09 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:38:10.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:09 smithi032 ceph-mon[32461]: pgmap v157: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-04-25T01:38:10.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:09 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:38:10.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:09 smithi028 ceph-mon[29390]: pgmap v157: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-04-25T01:38:10.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:09 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:38:11.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:10 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:38:11.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:10 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:38:11.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:10 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:38:11.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:10 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:38:11.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:10 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:38:11.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:10 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:38:11.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:10 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:38:11.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:10 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:38:11.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:10 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:38:12.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:11 smithi130 ceph-mon[32702]: pgmap v158: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:38:12.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:11 smithi032 ceph-mon[32461]: pgmap v158: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:38:12.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:11 smithi028 ceph-mon[29390]: pgmap v158: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:38:14.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:13 smithi130 ceph-mon[32702]: pgmap v159: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:38:14.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:13 smithi032 ceph-mon[32461]: pgmap v159: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:38:14.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:13 smithi028 ceph-mon[29390]: pgmap v159: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:38:16.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:15 smithi130 ceph-mon[32702]: pgmap v160: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-25T01:38:16.235 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:15 smithi032 ceph-mon[32461]: pgmap v160: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-25T01:38:16.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:15 smithi028 ceph-mon[29390]: pgmap v160: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-25T01:38:18.210 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:17 smithi130 ceph-mon[32702]: pgmap v161: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 170 B/s wr, 12 op/s 2024-04-25T01:38:18.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:17 smithi032 ceph-mon[32461]: pgmap v161: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 170 B/s wr, 12 op/s 2024-04-25T01:38:18.313 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:17 smithi028 ceph-mon[29390]: pgmap v161: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 170 B/s wr, 12 op/s 2024-04-25T01:38:20.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:19 smithi028 ceph-mon[29390]: pgmap v162: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T01:38:20.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:19 smithi130 ceph-mon[32702]: pgmap v162: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T01:38:20.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:19 smithi032 ceph-mon[32461]: pgmap v162: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T01:38:22.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:21 smithi130 ceph-mon[32702]: pgmap v163: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T01:38:22.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:21 smithi032 ceph-mon[32461]: pgmap v163: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T01:38:22.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:21 smithi028 ceph-mon[29390]: pgmap v163: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T01:38:24.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:23 smithi130 ceph-mon[32702]: pgmap v164: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T01:38:24.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:23 smithi032 ceph-mon[32461]: pgmap v164: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T01:38:24.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:23 smithi028 ceph-mon[29390]: pgmap v164: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T01:38:26.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:25 smithi130 ceph-mon[32702]: pgmap v165: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:38:26.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:25 smithi032 ceph-mon[32461]: pgmap v165: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:38:26.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:25 smithi028 ceph-mon[29390]: pgmap v165: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:38:28.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:27 smithi130 ceph-mon[32702]: pgmap v166: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:38:28.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:27 smithi032 ceph-mon[32461]: pgmap v166: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:38:28.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:27 smithi028 ceph-mon[29390]: pgmap v166: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:38:30.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:29 smithi130 ceph-mon[32702]: pgmap v167: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:38:30.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:29 smithi032 ceph-mon[32461]: pgmap v167: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:38:30.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:29 smithi028 ceph-mon[29390]: pgmap v167: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:38:32.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:31 smithi130 ceph-mon[32702]: pgmap v168: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:38:32.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:31 smithi032 ceph-mon[32461]: pgmap v168: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:38:32.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:31 smithi028 ceph-mon[29390]: pgmap v168: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:38:34.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:33 smithi130 ceph-mon[32702]: pgmap v169: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:38:34.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:33 smithi032 ceph-mon[32461]: pgmap v169: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:38:34.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:33 smithi028 ceph-mon[29390]: pgmap v169: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:38:35.622 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:38:35.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:38:35.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.760+0000 7f06ade83640 1 -- 172.21.15.28:0/481330583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a81029c0 msgr2=0x7f06a8102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:35.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.760+0000 7f06ade83640 1 --2- 172.21.15.28:0/481330583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a81029c0 0x7f06a8102da0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f0698009900 tx=0x7f069802a510 comp rx=0 tx=0).stop 2024-04-25T01:38:35.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.760+0000 7f06ade83640 1 -- 172.21.15.28:0/481330583 shutdown_connections 2024-04-25T01:38:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.760+0000 7f06ade83640 1 --2- 172.21.15.28:0/481330583 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06a810bd00 0x7f06a810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.760+0000 7f06ade83640 1 --2- 172.21.15.28:0/481330583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06a81032e0 0x7f06a810b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.760+0000 7f06ade83640 1 --2- 172.21.15.28:0/481330583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a81029c0 0x7f06a8102da0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.760+0000 7f06ade83640 1 -- 172.21.15.28:0/481330583 >> 172.21.15.28:0/481330583 conn(0x7f06a8077740 msgr2=0x7f06a8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.760+0000 7f06ade83640 1 -- 172.21.15.28:0/481330583 shutdown_connections 2024-04-25T01:38:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.760+0000 7f06ade83640 1 -- 172.21.15.28:0/481330583 wait complete. 2024-04-25T01:38:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.760+0000 7f06ade83640 1 Processor -- start 2024-04-25T01:38:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.761+0000 7f06ade83640 1 -- start start 2024-04-25T01:38:35.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.761+0000 7f06ade83640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a81029c0 0x7f06a8100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.761+0000 7f06ade83640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06a81032e0 0x7f06a8100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.761+0000 7f06ade83640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06a810bd00 0x7f06a8101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.761+0000 7f06ade83640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06a8111b50 con 0x7f06a81029c0 2024-04-25T01:38:35.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.761+0000 7f06ade83640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f06a81119d0 con 0x7f06a810bd00 2024-04-25T01:38:35.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.761+0000 7f06ade83640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f06a8111cd0 con 0x7f06a81032e0 2024-04-25T01:38:35.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.761+0000 7f06a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a81029c0 0x7f06a8100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.761+0000 7f06a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a81029c0 0x7f06a8100610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36750/0 (socket says 172.21.15.28:36750) 2024-04-25T01:38:35.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.761+0000 7f06a77fe640 1 -- 172.21.15.28:0/917338445 learned_addr learned my addr 172.21.15.28:0/917338445 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:35.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.761+0000 7f06a7fff640 1 --2- 172.21.15.28:0/917338445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06a810bd00 0x7f06a8101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.761+0000 7f06a77fe640 1 -- 172.21.15.28:0/917338445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06a81032e0 msgr2=0x7f06a8100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:35.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.761+0000 7f06a6ffd640 1 --2- 172.21.15.28:0/917338445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06a81032e0 0x7f06a8100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.761+0000 7f06a77fe640 1 --2- 172.21.15.28:0/917338445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06a81032e0 0x7f06a8100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.761+0000 7f06a77fe640 1 -- 172.21.15.28:0/917338445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06a810bd00 msgr2=0x7f06a8101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:35.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.761+0000 7f06a77fe640 1 --2- 172.21.15.28:0/917338445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06a810bd00 0x7f06a8101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.761+0000 7f06a77fe640 1 -- 172.21.15.28:0/917338445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a81a64b0 con 0x7f06a81029c0 2024-04-25T01:38:35.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.761+0000 7f06a7fff640 1 --2- 172.21.15.28:0/917338445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06a810bd00 0x7f06a8101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:35.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.762+0000 7f06a77fe640 1 --2- 172.21.15.28:0/917338445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a81029c0 0x7f06a8100610 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f0698004770 tx=0x7f0698003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:35.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.762+0000 7f06a4ff9640 1 -- 172.21.15.28:0/917338445 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069803f070 con 0x7f06a81029c0 2024-04-25T01:38:35.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.762+0000 7f06ade83640 1 -- 172.21.15.28:0/917338445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06a81a6740 con 0x7f06a81029c0 2024-04-25T01:38:35.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.762+0000 7f06a4ff9640 1 -- 172.21.15.28:0/917338445 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0698031ca0 con 0x7f06a81029c0 2024-04-25T01:38:35.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.762+0000 7f06a4ff9640 1 -- 172.21.15.28:0/917338445 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0698035030 con 0x7f06a81029c0 2024-04-25T01:38:35.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.762+0000 7f06ade83640 1 -- 172.21.15.28:0/917338445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06a81a6c30 con 0x7f06a81029c0 2024-04-25T01:38:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.765+0000 7f06a4ff9640 1 -- 172.21.15.28:0/917338445 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0698042070 con 0x7f06a81029c0 2024-04-25T01:38:35.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.765+0000 7f06b0b8a740 1 Processor -- start 2024-04-25T01:38:35.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.765+0000 7f06b0b8a740 1 -- start start 2024-04-25T01:38:35.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.765+0000 7f06b0b8a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba72a630 0x5583ba72aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.765+0000 7f06b0b8a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583ba72af50 0x5583ba7350e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.765+0000 7f06b0b8a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba735620 0x5583ba737a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.765+0000 7f06b0b8a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5583ba68f6b0 con 0x5583ba72a630 2024-04-25T01:38:35.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.765+0000 7f06b0b8a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5583ba637880 con 0x5583ba735620 2024-04-25T01:38:35.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.765+0000 7f06b0b8a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5583ba68b670 con 0x5583ba72af50 2024-04-25T01:38:35.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.765+0000 7f06a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba72a630 0x5583ba72aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.766+0000 7f06a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba72a630 0x5583ba72aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36758/0 (socket says 172.21.15.28:36758) 2024-04-25T01:38:35.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.766+0000 7f06a7fff640 1 -- 172.21.15.28:0/1271722288 learned_addr learned my addr 172.21.15.28:0/1271722288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:35.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.766+0000 7f06a77fe640 1 --2- 172.21.15.28:0/1271722288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583ba72af50 0x5583ba7350e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.766+0000 7f06a6ffd640 1 --2- 172.21.15.28:0/1271722288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba735620 0x5583ba737a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.766+0000 7f06a4ff9640 1 --2- 172.21.15.28:0/917338445 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f06900a2d30 0x7f06900a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.766+0000 7f06a7fff640 1 -- 172.21.15.28:0/1271722288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583ba72af50 msgr2=0x5583ba7350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:35.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.766+0000 7f06a7fff640 1 --2- 172.21.15.28:0/1271722288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583ba72af50 0x5583ba7350e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.766+0000 7f06a7fff640 1 -- 172.21.15.28:0/1271722288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba735620 msgr2=0x5583ba737a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:35.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.766+0000 7f06a7fff640 1 --2- 172.21.15.28:0/1271722288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba735620 0x5583ba737a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.766+0000 7f06a7fff640 1 -- 172.21.15.28:0/1271722288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583ba46d7d0 con 0x5583ba72a630 2024-04-25T01:38:35.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.766+0000 7f06a6ffd640 1 --2- 172.21.15.28:0/1271722288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba735620 0x5583ba737a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:35.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.766+0000 7f06a4ff9640 1 -- 172.21.15.28:0/917338445 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f06980effd0 con 0x7f06a81029c0 2024-04-25T01:38:35.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.766+0000 7f06a7fff640 1 --2- 172.21.15.28:0/917338445 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f06900a2d30 0x7f06900a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.766+0000 7f06a7fff640 1 --2- 172.21.15.28:0/1271722288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba72a630 0x5583ba72aa10 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009fa0 tx=0x7f06a000f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c80b961bc12cd422 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:35.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.766+0000 7f068dffb640 1 -- 172.21.15.28:0/1271722288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a001a610 con 0x5583ba72a630 2024-04-25T01:38:35.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.766+0000 7f068dffb640 1 -- 172.21.15.28:0/1271722288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06a001ac90 con 0x5583ba72a630 2024-04-25T01:38:35.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.766+0000 7f068dffb640 1 -- 172.21.15.28:0/1271722288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a0010960 con 0x5583ba72a630 2024-04-25T01:38:35.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.767+0000 7f06b0b8a740 1 -- 172.21.15.28:0/1271722288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba72a630 msgr2=0x5583ba72aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.767+0000 7f06a7fff640 1 --2- 172.21.15.28:0/917338445 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f06900a2d30 0x7f06900a51f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f06a0020000 tx=0x7f06a001d000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.767+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/1271722288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba72a630 0x5583ba72aa10 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009fa0 tx=0x7f06a000f7f0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.767+0000 7f06b0b8a740 1 -- 172.21.15.28:0/1271722288 shutdown_connections 2024-04-25T01:38:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.767+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/1271722288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba735620 0x5583ba737a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.767+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/1271722288 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583ba72af50 0x5583ba7350e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.767+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/1271722288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba72a630 0x5583ba72aa10 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.767+0000 7f06b0b8a740 1 -- 172.21.15.28:0/1271722288 >> 172.21.15.28:0/1271722288 conn(0x5583ba726950 msgr2=0x5583ba726d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.767+0000 7f06b0b8a740 1 -- 172.21.15.28:0/1271722288 shutdown_connections 2024-04-25T01:38:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.767+0000 7f06b0b8a740 1 -- 172.21.15.28:0/1271722288 wait complete. 2024-04-25T01:38:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.768+0000 7f06b0b8a740 1 -- start start 2024-04-25T01:38:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.768+0000 7f06b0b8a740 1 Processor -- start 2024-04-25T01:38:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.768+0000 7f06b0b8a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583ba72a630 0x5583ba732560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.768+0000 7f06b0b8a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba72af50 0x5583ba732aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.768+0000 7f06b0b8a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba735620 0x5583ba732fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.768+0000 7f06b0b8a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5583ba738410 con 0x5583ba735620 2024-04-25T01:38:35.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.768+0000 7f06b0b8a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5583ba64af50 con 0x5583ba72af50 2024-04-25T01:38:35.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.768+0000 7f06b0b8a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5583ba738590 con 0x5583ba72a630 2024-04-25T01:38:35.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.768+0000 7f06b0b8a740 10 client.0 _ll_register_callbacks cb 0x5583ba686000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:35.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.768+0000 7f06a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba735620 0x5583ba732fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.768+0000 7f06a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba735620 0x5583ba732fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36766/0 (socket says 172.21.15.28:36766) 2024-04-25T01:38:35.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.768+0000 7f06a7fff640 1 -- 172.21.15.28:0/786831434 learned_addr learned my addr 172.21.15.28:0/786831434 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:35.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.768+0000 7f06a6ffd640 1 --2- 172.21.15.28:0/786831434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba72af50 0x5583ba732aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.768+0000 7f06a77fe640 1 --2- 172.21.15.28:0/786831434 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583ba72a630 0x5583ba732560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.769+0000 7f06a7fff640 1 -- 172.21.15.28:0/786831434 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583ba72a630 msgr2=0x5583ba732560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:35.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.769+0000 7f06a7fff640 1 --2- 172.21.15.28:0/786831434 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583ba72a630 0x5583ba732560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.769+0000 7f06a7fff640 1 -- 172.21.15.28:0/786831434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba72af50 msgr2=0x5583ba732aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:35.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.769+0000 7f06a7fff640 1 --2- 172.21.15.28:0/786831434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba72af50 0x5583ba732aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.769+0000 7f06a7fff640 1 -- 172.21.15.28:0/786831434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583ba3a33f0 con 0x5583ba735620 2024-04-25T01:38:35.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.769+0000 7f06a6ffd640 1 --2- 172.21.15.28:0/786831434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba72af50 0x5583ba732aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:35.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.769+0000 7f06a7fff640 1 --2- 172.21.15.28:0/786831434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba735620 0x5583ba732fe0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f06a0005b70 tx=0x7f06a0009990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e57535abce5c0393 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:35.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.769+0000 7f0686ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:35.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.769+0000 7f0686ffd640 1 -- 172.21.15.28:0/786831434 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a0032060 con 0x5583ba735620 2024-04-25T01:38:35.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.769+0000 7f0686ffd640 1 -- 172.21.15.28:0/786831434 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f06a0012240 con 0x5583ba735620 2024-04-25T01:38:35.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.769+0000 7f06b0b8a740 1 -- 172.21.15.28:0/786831434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06a001ac90 con 0x5583ba735620 2024-04-25T01:38:35.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.769+0000 7f0686ffd640 1 -- 172.21.15.28:0/786831434 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f06a0039050 con 0x5583ba735620 2024-04-25T01:38:35.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.769+0000 7f06b0b8a740 10 client.17292 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:35.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.769+0000 7f06b0b8a740 1 -- 172.21.15.28:0/786831434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5583ba3630f0 con 0x5583ba735620 2024-04-25T01:38:35.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.769+0000 7f0686ffd640 1 -- 172.21.15.28:0/786831434 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06a001ac90 con 0x5583ba735620 2024-04-25T01:38:35.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.770+0000 7f0686ffd640 1 -- 172.21.15.28:0/786831434 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f06a0025440 con 0x5583ba735620 2024-04-25T01:38:35.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.770+0000 7f06b0b8a740 10 client.17292 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:35.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.770+0000 7f06b0b8a740 10 client.17292 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:35.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.770+0000 7f06b0b8a740 20 client.17292 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:35.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.770+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/786831434 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5583ba7457b0 0x5583ba747ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.770+0000 7f06b0b8a740 4 client.17292 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:35.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.770+0000 7f06b0b8a740 1 -- 172.21.15.28:0/786831434 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5583ba69cbf0 con 0x5583ba7457b0 2024-04-25T01:38:35.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.770+0000 7f06a77fe640 1 --2- 172.21.15.28:0/786831434 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5583ba7457b0 0x5583ba747ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.770+0000 7f0686ffd640 10 client.17292 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:35.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.770+0000 7f06a77fe640 1 --2- 172.21.15.28:0/786831434 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5583ba7457b0 0x5583ba747ba0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c0e812e684d778e server_cookie=432a6cbf9feab959 in_seq=0 out_seq=0 2024-04-25T01:38:35.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.771+0000 7f0686ffd640 1 -- 172.21.15.28:0/786831434 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5583ba69cbf0 con 0x5583ba7457b0 2024-04-25T01:38:35.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.771+0000 7f0686ffd640 10 client.17292 handle_command_reply: tid=0 2024-04-25T01:38:35.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.771+0000 7f06b0b8a740 1 client.17292 shutdown 2024-04-25T01:38:35.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.771+0000 7f06b0b8a740 1 -- 172.21.15.28:0/786831434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba735620 msgr2=0x5583ba732fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:35.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.771+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/786831434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba735620 0x5583ba732fe0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f06a0005b70 tx=0x7f06a0009990 comp rx=0 tx=0).stop 2024-04-25T01:38:35.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.771+0000 7f06b0b8a740 1 -- 172.21.15.28:0/786831434 shutdown_connections 2024-04-25T01:38:35.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.771+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/786831434 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5583ba7457b0 0x5583ba747ba0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.771+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/786831434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba735620 0x5583ba732fe0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.771+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/786831434 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba72af50 0x5583ba732aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.771+0000 7f0686ffd640 0 client.17292 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:35.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.771+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/786831434 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583ba72a630 0x5583ba732560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.771+0000 7f06b0b8a740 1 -- 172.21.15.28:0/786831434 >> 172.21.15.28:0/786831434 conn(0x5583ba726950 msgr2=0x5583ba72c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:35.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.771+0000 7f06b0b8a740 1 -- 172.21.15.28:0/786831434 shutdown_connections 2024-04-25T01:38:35.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.772+0000 7f06b0b8a740 1 -- 172.21.15.28:0/786831434 wait complete. 2024-04-25T01:38:35.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.772+0000 7f06b0b8a740 20 client.17292 trim_cache size 0 max 16384 2024-04-25T01:38:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.789+0000 7f06b0b8a740 1 Processor -- start 2024-04-25T01:38:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.789+0000 7f06b0b8a740 1 -- start start 2024-04-25T01:38:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.789+0000 7f06b0b8a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba72a630 0x5583ba734c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.789+0000 7f06b0b8a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba72af50 0x5583ba725af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.789+0000 7f06b0b8a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583ba735620 0x5583ba732e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.789+0000 7f06b0b8a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06a81b0c70 con 0x5583ba72a630 2024-04-25T01:38:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.789+0000 7f06b0b8a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f06a81b27b0 con 0x5583ba72af50 2024-04-25T01:38:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.789+0000 7f06b0b8a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f06a81b1340 con 0x5583ba735620 2024-04-25T01:38:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.789+0000 7f06a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba72a630 0x5583ba734c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.790+0000 7f06a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba72a630 0x5583ba734c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36768/0 (socket says 172.21.15.28:36768) 2024-04-25T01:38:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.790+0000 7f06a77fe640 1 -- 172.21.15.28:0/3846098456 learned_addr learned my addr 172.21.15.28:0/3846098456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.790+0000 7f06a6ffd640 1 --2- 172.21.15.28:0/3846098456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba72af50 0x5583ba725af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.790+0000 7f06a77fe640 1 -- 172.21.15.28:0/3846098456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583ba735620 msgr2=0x5583ba732e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:35.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.790+0000 7f06a77fe640 1 --2- 172.21.15.28:0/3846098456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583ba735620 0x5583ba732e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.790+0000 7f06a77fe640 1 -- 172.21.15.28:0/3846098456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba72af50 msgr2=0x5583ba725af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:35.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.790+0000 7f06a77fe640 1 --2- 172.21.15.28:0/3846098456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba72af50 0x5583ba725af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.790+0000 7f06a77fe640 1 -- 172.21.15.28:0/3846098456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583ba718090 con 0x5583ba72a630 2024-04-25T01:38:35.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.790+0000 7f06a6ffd640 1 --2- 172.21.15.28:0/3846098456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba72af50 0x5583ba725af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:35.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.790+0000 7f06a77fe640 1 --2- 172.21.15.28:0/3846098456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba72a630 0x5583ba734c90 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f0698000c00 tx=0x7f06980f2420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f3394ac2a20607 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:35.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.790+0000 7f068d7fa640 1 -- 172.21.15.28:0/3846098456 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0698049040 con 0x5583ba72a630 2024-04-25T01:38:35.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.790+0000 7f068d7fa640 1 -- 172.21.15.28:0/3846098456 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06980bc150 con 0x5583ba72a630 2024-04-25T01:38:35.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.790+0000 7f068d7fa640 1 -- 172.21.15.28:0/3846098456 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06980b87c0 con 0x5583ba72a630 2024-04-25T01:38:35.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.791+0000 7f06b0b8a740 1 -- 172.21.15.28:0/3846098456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba72a630 msgr2=0x5583ba734c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:35.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.791+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/3846098456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba72a630 0x5583ba734c90 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f0698000c00 tx=0x7f06980f2420 comp rx=0 tx=0).stop 2024-04-25T01:38:35.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.791+0000 7f06b0b8a740 1 -- 172.21.15.28:0/3846098456 shutdown_connections 2024-04-25T01:38:35.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.791+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/3846098456 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583ba735620 0x5583ba732e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.791+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/3846098456 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba72af50 0x5583ba725af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.791+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/3846098456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba72a630 0x5583ba734c90 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.791+0000 7f06b0b8a740 1 -- 172.21.15.28:0/3846098456 >> 172.21.15.28:0/3846098456 conn(0x5583ba726950 msgr2=0x5583ba72bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.791+0000 7f06b0b8a740 1 -- 172.21.15.28:0/3846098456 shutdown_connections 2024-04-25T01:38:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.791+0000 7f06b0b8a740 1 -- 172.21.15.28:0/3846098456 wait complete. 2024-04-25T01:38:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.791+0000 7f06b0b8a740 1 -- start start 2024-04-25T01:38:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.791+0000 7f06b0b8a740 1 Processor -- start 2024-04-25T01:38:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.791+0000 7f06b0b8a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba72a630 0x5583ba72f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.791+0000 7f06b0b8a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583ba72af50 0x5583ba72f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f06b0b8a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba735620 0x5583ba72ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f06b0b8a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06a81b20e0 con 0x5583ba735620 2024-04-25T01:38:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f06b0b8a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f06a81b1a10 con 0x5583ba72a630 2024-04-25T01:38:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f06b0b8a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f06a81aec50 con 0x5583ba72af50 2024-04-25T01:38:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f06b0b8a740 10 client.0 _ll_register_callbacks cb 0x7f06980b8220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f06a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba735620 0x5583ba72ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f06a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba735620 0x5583ba72ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36784/0 (socket says 172.21.15.28:36784) 2024-04-25T01:38:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f06a7fff640 1 -- 172.21.15.28:0/3710940410 learned_addr learned my addr 172.21.15.28:0/3710940410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f06a6ffd640 1 --2- 172.21.15.28:0/3710940410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583ba72af50 0x5583ba72f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f06a77fe640 1 --2- 172.21.15.28:0/3710940410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba72a630 0x5583ba72f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f06a7fff640 1 -- 172.21.15.28:0/3710940410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583ba72af50 msgr2=0x5583ba72f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f06a7fff640 1 --2- 172.21.15.28:0/3710940410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583ba72af50 0x5583ba72f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f06a7fff640 1 -- 172.21.15.28:0/3710940410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba72a630 msgr2=0x5583ba72f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f06a7fff640 1 --2- 172.21.15.28:0/3710940410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba72a630 0x5583ba72f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f06a7fff640 1 -- 172.21.15.28:0/3710940410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583ba7aad40 con 0x5583ba735620 2024-04-25T01:38:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f06a7fff640 1 --2- 172.21.15.28:0/3710940410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba735620 0x5583ba72ff10 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f06a0005ba0 tx=0x7f06a0012dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ae99b4a9217a2b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f0686ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f0686ffd640 1 -- 172.21.15.28:0/3710940410 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a0032050 con 0x5583ba735620 2024-04-25T01:38:35.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f0686ffd640 1 -- 172.21.15.28:0/3710940410 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f06a0025440 con 0x5583ba735620 2024-04-25T01:38:35.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f06b0b8a740 1 -- 172.21.15.28:0/3710940410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06980bc150 con 0x5583ba735620 2024-04-25T01:38:35.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f0686ffd640 1 -- 172.21.15.28:0/3710940410 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f06a0009af0 con 0x5583ba735620 2024-04-25T01:38:35.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f06b0b8a740 10 client.17298 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:35.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.792+0000 7f06b0b8a740 1 -- 172.21.15.28:0/3710940410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5583ba7ac090 con 0x5583ba735620 2024-04-25T01:38:35.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.793+0000 7f0686ffd640 1 -- 172.21.15.28:0/3710940410 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06980bc150 con 0x5583ba735620 2024-04-25T01:38:35.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.793+0000 7f0686ffd640 1 -- 172.21.15.28:0/3710940410 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f06a0012040 con 0x5583ba735620 2024-04-25T01:38:35.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.793+0000 7f06b0b8a740 10 client.17298 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:35.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.793+0000 7f06b0b8a740 10 client.17298 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:35.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.793+0000 7f06b0b8a740 20 client.17298 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:35.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.793+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/3710940410 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5583ba7ac860 0x5583ba7aec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.793+0000 7f06b0b8a740 4 client.17298 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:38:35.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.793+0000 7f06b0b8a740 1 -- 172.21.15.28:0/3710940410 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5583ba696f40 con 0x5583ba7ac860 2024-04-25T01:38:35.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.793+0000 7f06a77fe640 1 --2- 172.21.15.28:0/3710940410 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5583ba7ac860 0x5583ba7aec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.794+0000 7f06a77fe640 1 --2- 172.21.15.28:0/3710940410 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5583ba7ac860 0x5583ba7aec80 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5277b64319131a0 server_cookie=b11ae4ebf4edb313 in_seq=0 out_seq=0 2024-04-25T01:38:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.794+0000 7f0686ffd640 10 client.17298 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.794+0000 7f0686ffd640 1 -- 172.21.15.28:0/3710940410 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5583ba696f40 con 0x5583ba7ac860 2024-04-25T01:38:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.794+0000 7f0686ffd640 10 client.17298 handle_command_reply: tid=0 2024-04-25T01:38:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.794+0000 7f06b0b8a740 1 client.17298 shutdown 2024-04-25T01:38:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.794+0000 7f06b0b8a740 1 -- 172.21.15.28:0/3710940410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba735620 msgr2=0x5583ba72ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.794+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/3710940410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba735620 0x5583ba72ff10 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f06a0005ba0 tx=0x7f06a0012dd0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.794+0000 7f06b0b8a740 1 -- 172.21.15.28:0/3710940410 shutdown_connections 2024-04-25T01:38:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.794+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/3710940410 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5583ba7ac860 0x5583ba7aec80 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.794+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/3710940410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583ba735620 0x5583ba72ff10 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.794+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/3710940410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5583ba72af50 0x5583ba72f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.794+0000 7f06b0b8a740 1 --2- 172.21.15.28:0/3710940410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5583ba72a630 0x5583ba72f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.794+0000 7f0686ffd640 0 client.17298 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.794+0000 7f06b0b8a740 1 -- 172.21.15.28:0/3710940410 >> 172.21.15.28:0/3710940410 conn(0x5583ba726950 msgr2=0x5583ba725970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.794+0000 7f06b0b8a740 1 -- 172.21.15.28:0/3710940410 shutdown_connections 2024-04-25T01:38:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.794+0000 7f06b0b8a740 1 -- 172.21.15.28:0/3710940410 wait complete. 2024-04-25T01:38:35.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.794+0000 7f06b0b8a740 20 client.17298 trim_cache size 0 max 16384 2024-04-25T01:38:35.782 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:38:35.782 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:38:35.782 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:38:35.782 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:38:35.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.795+0000 7f06ade83640 1 -- 172.21.15.28:0/917338445 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f06900a2d30 msgr2=0x7f06900a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:35.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.795+0000 7f06ade83640 1 --2- 172.21.15.28:0/917338445 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f06900a2d30 0x7f06900a51f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f06a0020000 tx=0x7f06a001d000 comp rx=0 tx=0).stop 2024-04-25T01:38:35.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.795+0000 7f06ade83640 1 -- 172.21.15.28:0/917338445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a81029c0 msgr2=0x7f06a8100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:35.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.795+0000 7f06ade83640 1 --2- 172.21.15.28:0/917338445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a81029c0 0x7f06a8100610 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f0698004770 tx=0x7f0698003c10 comp rx=0 tx=0).stop 2024-04-25T01:38:35.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.795+0000 7f06ade83640 1 -- 172.21.15.28:0/917338445 shutdown_connections 2024-04-25T01:38:35.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.795+0000 7f06ade83640 1 --2- 172.21.15.28:0/917338445 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f06900a2d30 0x7f06900a51f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.795+0000 7f06ade83640 1 --2- 172.21.15.28:0/917338445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f06a810bd00 0x7f06a8101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.795+0000 7f06ade83640 1 --2- 172.21.15.28:0/917338445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f06a81032e0 0x7f06a8100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.795+0000 7f06ade83640 1 --2- 172.21.15.28:0/917338445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a81029c0 0x7f06a8100610 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.795+0000 7f06ade83640 1 -- 172.21.15.28:0/917338445 >> 172.21.15.28:0/917338445 conn(0x7f06a8077740 msgr2=0x7f06a80fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:35.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.795+0000 7f06ade83640 1 -- 172.21.15.28:0/917338445 shutdown_connections 2024-04-25T01:38:35.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.795+0000 7f06ade83640 1 -- 172.21.15.28:0/917338445 wait complete. 2024-04-25T01:38:35.786 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:38:35.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:38:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.908+0000 7f792e9ed640 1 -- 172.21.15.28:0/4186323608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7928102a20 msgr2=0x7f7928102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.908+0000 7f792e9ed640 1 --2- 172.21.15.28:0/4186323608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7928102a20 0x7f7928102e00 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f7910009900 tx=0x7f791002a480 comp rx=0 tx=0).stop 2024-04-25T01:38:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.908+0000 7f792e9ed640 1 -- 172.21.15.28:0/4186323608 shutdown_connections 2024-04-25T01:38:35.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.908+0000 7f792e9ed640 1 --2- 172.21.15.28:0/4186323608 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f792810bd60 0x7f792810e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.908+0000 7f792e9ed640 1 --2- 172.21.15.28:0/4186323608 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7928103340 0x7f792810b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.908+0000 7f792e9ed640 1 --2- 172.21.15.28:0/4186323608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7928102a20 0x7f7928102e00 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.908+0000 7f792e9ed640 1 -- 172.21.15.28:0/4186323608 >> 172.21.15.28:0/4186323608 conn(0x7f79280777a0 msgr2=0x7f7928077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.908+0000 7f792e9ed640 1 -- 172.21.15.28:0/4186323608 shutdown_connections 2024-04-25T01:38:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.908+0000 7f792e9ed640 1 -- 172.21.15.28:0/4186323608 wait complete. 2024-04-25T01:38:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.909+0000 7f792e9ed640 1 Processor -- start 2024-04-25T01:38:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.909+0000 7f792e9ed640 1 -- start start 2024-04-25T01:38:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.909+0000 7f792e9ed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7928102a20 0x7f7928198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.909+0000 7f792e9ed640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7928103340 0x7f7928198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.909+0000 7f792e9ed640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f792810bd60 0x7f792819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.909+0000 7f792e9ed640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7928111b30 con 0x7f7928102a20 2024-04-25T01:38:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.909+0000 7f792e9ed640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f79281119b0 con 0x7f792810bd60 2024-04-25T01:38:35.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.909+0000 7f792e9ed640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7928111cb0 con 0x7f7928103340 2024-04-25T01:38:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.909+0000 7f7927fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7928102a20 0x7f7928198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.909+0000 7f7927fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7928102a20 0x7f7928198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36800/0 (socket says 172.21.15.28:36800) 2024-04-25T01:38:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.909+0000 7f7927fff640 1 -- 172.21.15.28:0/1268457187 learned_addr learned my addr 172.21.15.28:0/1268457187 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.910+0000 7f792cf8a640 1 --2- 172.21.15.28:0/1268457187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f792810bd60 0x7f792819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.910+0000 7f79277fe640 1 --2- 172.21.15.28:0/1268457187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7928103340 0x7f7928198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.910+0000 7f7927fff640 1 -- 172.21.15.28:0/1268457187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7928103340 msgr2=0x7f7928198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.910+0000 7f7927fff640 1 --2- 172.21.15.28:0/1268457187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7928103340 0x7f7928198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.910+0000 7f7927fff640 1 -- 172.21.15.28:0/1268457187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f792810bd60 msgr2=0x7f792819c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.910+0000 7f7927fff640 1 --2- 172.21.15.28:0/1268457187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f792810bd60 0x7f792819c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.910+0000 7f7927fff640 1 -- 172.21.15.28:0/1268457187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f792819d020 con 0x7f7928102a20 2024-04-25T01:38:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.910+0000 7f792cf8a640 1 --2- 172.21.15.28:0/1268457187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f792810bd60 0x7f792819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.910+0000 7f7927fff640 1 --2- 172.21.15.28:0/1268457187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7928102a20 0x7f7928198030 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f7910006010 tx=0x7f79100042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.910+0000 7f79257fa640 1 -- 172.21.15.28:0/1268457187 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791003f030 con 0x7f7928102a20 2024-04-25T01:38:35.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.910+0000 7f792e9ed640 1 -- 172.21.15.28:0/1268457187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f792819d2b0 con 0x7f7928102a20 2024-04-25T01:38:35.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.910+0000 7f79257fa640 1 -- 172.21.15.28:0/1268457187 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7910003ce0 con 0x7f7928102a20 2024-04-25T01:38:35.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.910+0000 7f79257fa640 1 -- 172.21.15.28:0/1268457187 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7910035070 con 0x7f7928102a20 2024-04-25T01:38:35.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.910+0000 7f792e9ed640 1 -- 172.21.15.28:0/1268457187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79281a4ba0 con 0x7f7928102a20 2024-04-25T01:38:35.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.911+0000 7f792e9ed640 1 -- 172.21.15.28:0/1268457187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79280689b0 con 0x7f7928102a20 2024-04-25T01:38:35.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.912+0000 7f79257fa640 1 -- 172.21.15.28:0/1268457187 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7910042030 con 0x7f7928102a20 2024-04-25T01:38:35.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.913+0000 7f79257fa640 1 --2- 172.21.15.28:0/1268457187 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f79140a2d00 0x7f79140a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:35.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.913+0000 7f79257fa640 1 -- 172.21.15.28:0/1268457187 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f79100efd50 con 0x7f7928102a20 2024-04-25T01:38:35.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.913+0000 7f79277fe640 1 --2- 172.21.15.28:0/1268457187 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f79140a2d00 0x7f79140a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:35.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.913+0000 7f79277fe640 1 --2- 172.21.15.28:0/1268457187 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f79140a2d00 0x7f79140a51c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7918005fd0 tx=0x7f7918005850 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:35.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:35.918+0000 7f79257fa640 1 -- 172.21.15.28:0/1268457187 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f79100f3030 con 0x7f7928102a20 2024-04-25T01:38:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.143+0000 7f792e9ed640 1 -- 172.21.15.28:0/1268457187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7928106ee0 con 0x7f7928102a20 2024-04-25T01:38:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.143+0000 7f79257fa640 1 -- 172.21.15.28:0/1268457187 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7f79100bc6f0 con 0x7f7928102a20 2024-04-25T01:38:36.124 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:38:36.124 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:38:36.125 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:38:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.145+0000 7f792e9ed640 1 -- 172.21.15.28:0/1268457187 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f79140a2d00 msgr2=0x7f79140a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.145+0000 7f792e9ed640 1 --2- 172.21.15.28:0/1268457187 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f79140a2d00 0x7f79140a51c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7918005fd0 tx=0x7f7918005850 comp rx=0 tx=0).stop 2024-04-25T01:38:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.145+0000 7f792e9ed640 1 -- 172.21.15.28:0/1268457187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7928102a20 msgr2=0x7f7928198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.145+0000 7f792e9ed640 1 --2- 172.21.15.28:0/1268457187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7928102a20 0x7f7928198030 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f7910006010 tx=0x7f79100042f0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.146+0000 7f792e9ed640 1 -- 172.21.15.28:0/1268457187 shutdown_connections 2024-04-25T01:38:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.146+0000 7f792e9ed640 1 --2- 172.21.15.28:0/1268457187 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f79140a2d00 0x7f79140a51c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.146+0000 7f792e9ed640 1 --2- 172.21.15.28:0/1268457187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f792810bd60 0x7f792819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.146+0000 7f792e9ed640 1 --2- 172.21.15.28:0/1268457187 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7928103340 0x7f7928198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.146+0000 7f792e9ed640 1 --2- 172.21.15.28:0/1268457187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7928102a20 0x7f7928198030 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.146+0000 7f792e9ed640 1 -- 172.21.15.28:0/1268457187 >> 172.21.15.28:0/1268457187 conn(0x7f79280777a0 msgr2=0x7f79280fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.146+0000 7f792e9ed640 1 -- 172.21.15.28:0/1268457187 shutdown_connections 2024-04-25T01:38:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.146+0000 7f792e9ed640 1 -- 172.21.15.28:0/1268457187 wait complete. 2024-04-25T01:38:36.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:38:36.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:35 smithi028 ceph-mon[29390]: pgmap v170: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:38:36.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:35 smithi130 ceph-mon[32702]: pgmap v170: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:38:36.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:35 smithi032 ceph-mon[32461]: pgmap v170: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:38:36.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.261+0000 7fa41cfc1640 1 -- 172.21.15.28:0/3452559102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa418101710 msgr2=0x7fa418101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.261+0000 7fa41cfc1640 1 --2- 172.21.15.28:0/3452559102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa418101710 0x7fa418101b90 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fa400009930 tx=0x7fa40002a420 comp rx=0 tx=0).stop 2024-04-25T01:38:36.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.261+0000 7fa41cfc1640 1 -- 172.21.15.28:0/3452559102 shutdown_connections 2024-04-25T01:38:36.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.261+0000 7fa41cfc1640 1 --2- 172.21.15.28:0/3452559102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa4181020d0 0x7fa41810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.261+0000 7fa41cfc1640 1 --2- 172.21.15.28:0/3452559102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa418101710 0x7fa418101b90 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.261+0000 7fa41cfc1640 1 --2- 172.21.15.28:0/3452559102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa418107720 0x7fa418107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.261+0000 7fa41cfc1640 1 -- 172.21.15.28:0/3452559102 >> 172.21.15.28:0/3452559102 conn(0x7fa4180fd180 msgr2=0x7fa4180ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.261+0000 7fa41cfc1640 1 -- 172.21.15.28:0/3452559102 shutdown_connections 2024-04-25T01:38:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.261+0000 7fa41cfc1640 1 -- 172.21.15.28:0/3452559102 wait complete. 2024-04-25T01:38:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.261+0000 7fa41cfc1640 1 Processor -- start 2024-04-25T01:38:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.261+0000 7fa41cfc1640 1 -- start start 2024-04-25T01:38:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa41cfc1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa418101710 0x7fa418197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa41cfc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4181020d0 0x7fa4181984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa41cfc1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa418107720 0x7fa41819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa41cfc1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa418111af0 con 0x7fa4181020d0 2024-04-25T01:38:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa41cfc1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa418111970 con 0x7fa418101710 2024-04-25T01:38:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa41cfc1640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa418111c70 con 0x7fa418107720 2024-04-25T01:38:36.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa40ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4181020d0 0x7fa4181984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa40ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4181020d0 0x7fa4181984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36812/0 (socket says 172.21.15.28:36812) 2024-04-25T01:38:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa40ffff640 1 -- 172.21.15.28:0/183889203 learned_addr learned my addr 172.21.15.28:0/183889203 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa416d9d640 1 --2- 172.21.15.28:0/183889203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa418107720 0x7fa41819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa41659c640 1 --2- 172.21.15.28:0/183889203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa418101710 0x7fa418197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa40ffff640 1 -- 172.21.15.28:0/183889203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa418107720 msgr2=0x7fa41819c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa40ffff640 1 --2- 172.21.15.28:0/183889203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa418107720 0x7fa41819c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa40ffff640 1 -- 172.21.15.28:0/183889203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa418101710 msgr2=0x7fa418197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa40ffff640 1 --2- 172.21.15.28:0/183889203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa418101710 0x7fa418197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa40ffff640 1 -- 172.21.15.28:0/183889203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa41819cfa0 con 0x7fa4181020d0 2024-04-25T01:38:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa40ffff640 1 --2- 172.21.15.28:0/183889203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4181020d0 0x7fa4181984f0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fa400009230 tx=0x7fa400004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa40f7fe640 1 -- 172.21.15.28:0/183889203 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40003f030 con 0x7fa4181020d0 2024-04-25T01:38:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa40f7fe640 1 -- 172.21.15.28:0/183889203 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa400003ad0 con 0x7fa4181020d0 2024-04-25T01:38:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa41cfc1640 1 -- 172.21.15.28:0/183889203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa41819d230 con 0x7fa4181020d0 2024-04-25T01:38:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.262+0000 7fa40f7fe640 1 -- 172.21.15.28:0/183889203 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa400035070 con 0x7fa4181020d0 2024-04-25T01:38:36.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.263+0000 7fa41cfc1640 1 -- 172.21.15.28:0/183889203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4181a4b20 con 0x7fa4181020d0 2024-04-25T01:38:36.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.265+0000 7fa40f7fe640 1 -- 172.21.15.28:0/183889203 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa400042030 con 0x7fa4181020d0 2024-04-25T01:38:36.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.265+0000 7fa41fdaa740 1 Processor -- start 2024-04-25T01:38:36.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.265+0000 7fa41fdaa740 1 -- start start 2024-04-25T01:38:36.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.265+0000 7fa41fdaa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b595c580 0x5589b595c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.265+0000 7fa41fdaa740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b595cea0 0x5589b5967030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa41fdaa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b5967570 0x5589b5969960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa41fdaa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5589b58c1670 con 0x5589b5967570 2024-04-25T01:38:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa41fdaa740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5589b5869970 con 0x5589b595cea0 2024-04-25T01:38:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa41fdaa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5589b58bd630 con 0x5589b595c580 2024-04-25T01:38:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa40f7fe640 1 --2- 172.21.15.28:0/183889203 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa4040a2d80 0x7fa4040a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa40ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b5967570 0x5589b5969960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa40ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b5967570 0x5589b5969960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36816/0 (socket says 172.21.15.28:36816) 2024-04-25T01:38:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa40ffff640 1 -- 172.21.15.28:0/3430847027 learned_addr learned my addr 172.21.15.28:0/3430847027 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa416d9d640 1 --2- 172.21.15.28:0/183889203 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa4040a2d80 0x7fa4040a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa41659c640 1 --2- 172.21.15.28:0/3430847027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b595cea0 0x5589b5967030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa416d9d640 1 --2- 172.21.15.28:0/3430847027 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b595c580 0x5589b595c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa40f7fe640 1 -- 172.21.15.28:0/183889203 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fa4000efc90 con 0x7fa4181020d0 2024-04-25T01:38:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa40ffff640 1 -- 172.21.15.28:0/3430847027 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b595c580 msgr2=0x5589b595c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa40ffff640 1 --2- 172.21.15.28:0/3430847027 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b595c580 0x5589b595c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa40ffff640 1 -- 172.21.15.28:0/3430847027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b595cea0 msgr2=0x5589b5967030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa40ffff640 1 --2- 172.21.15.28:0/3430847027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b595cea0 0x5589b5967030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa40ffff640 1 -- 172.21.15.28:0/3430847027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589b588d400 con 0x5589b5967570 2024-04-25T01:38:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa416d9d640 1 --2- 172.21.15.28:0/3430847027 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b595c580 0x5589b595c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa41659c640 1 --2- 172.21.15.28:0/3430847027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b595cea0 0x5589b5967030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa40ffff640 1 --2- 172.21.15.28:0/3430847027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b5967570 0x5589b5969960 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fa4000b8280 tx=0x7fa40004b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=accc273beb19c9c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa416d9d640 1 --2- 172.21.15.28:0/183889203 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa4040a2d80 0x7fa4040a5240 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa410009fd0 tx=0x7fa410009560 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.266+0000 7fa40cff9640 1 -- 172.21.15.28:0/3430847027 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa400049020 con 0x5589b5967570 2024-04-25T01:38:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.267+0000 7fa40cff9640 1 -- 172.21.15.28:0/3430847027 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4000bc570 con 0x5589b5967570 2024-04-25T01:38:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.267+0000 7fa40cff9640 1 -- 172.21.15.28:0/3430847027 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa400003d90 con 0x5589b5967570 2024-04-25T01:38:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.267+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3430847027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b5967570 msgr2=0x5589b5969960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.267+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/3430847027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b5967570 0x5589b5969960 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fa4000b8280 tx=0x7fa40004b000 comp rx=0 tx=0).stop 2024-04-25T01:38:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.267+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3430847027 shutdown_connections 2024-04-25T01:38:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.267+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/3430847027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b5967570 0x5589b5969960 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.267+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/3430847027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b595cea0 0x5589b5967030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.267+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/3430847027 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b595c580 0x5589b595c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.267+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3430847027 >> 172.21.15.28:0/3430847027 conn(0x5589b59588a0 msgr2=0x5589b5958c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.267+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3430847027 shutdown_connections 2024-04-25T01:38:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.267+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3430847027 wait complete. 2024-04-25T01:38:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.268+0000 7fa41fdaa740 1 -- start start 2024-04-25T01:38:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.268+0000 7fa41fdaa740 1 Processor -- start 2024-04-25T01:38:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.268+0000 7fa41fdaa740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b595c580 0x5589b5966ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.268+0000 7fa41fdaa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b595cea0 0x5589b5964190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.268+0000 7fa41fdaa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b5967570 0x5589b59646d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.268+0000 7fa41fdaa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5589b596a120 con 0x5589b5967570 2024-04-25T01:38:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.268+0000 7fa41fdaa740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5589b587d050 con 0x5589b595c580 2024-04-25T01:38:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.268+0000 7fa41fdaa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5589b596a2a0 con 0x5589b595cea0 2024-04-25T01:38:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.268+0000 7fa41fdaa740 10 client.0 _ll_register_callbacks cb 0x5589b55d4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.268+0000 7fa416d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b5967570 0x5589b59646d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.268+0000 7fa416d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b5967570 0x5589b59646d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36818/0 (socket says 172.21.15.28:36818) 2024-04-25T01:38:36.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.268+0000 7fa416d9d640 1 -- 172.21.15.28:0/3928119719 learned_addr learned my addr 172.21.15.28:0/3928119719 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.268+0000 7fa40ffff640 1 --2- 172.21.15.28:0/3928119719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b595cea0 0x5589b5964190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.268+0000 7fa41659c640 1 --2- 172.21.15.28:0/3928119719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b595c580 0x5589b5966ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.268+0000 7fa416d9d640 1 -- 172.21.15.28:0/3928119719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b595cea0 msgr2=0x5589b5964190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.268+0000 7fa416d9d640 1 --2- 172.21.15.28:0/3928119719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b595cea0 0x5589b5964190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.268+0000 7fa416d9d640 1 -- 172.21.15.28:0/3928119719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b595c580 msgr2=0x5589b5966ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.268+0000 7fa416d9d640 1 --2- 172.21.15.28:0/3928119719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b595c580 0x5589b5966ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.268+0000 7fa416d9d640 1 -- 172.21.15.28:0/3928119719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589b584edc0 con 0x5589b5967570 2024-04-25T01:38:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.269+0000 7fa416d9d640 1 --2- 172.21.15.28:0/3928119719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b5967570 0x5589b59646d0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fa410010af0 tx=0x7fa410004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45db62f0f50d3aeb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.269+0000 7fa3fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.269+0000 7fa3fdffb640 1 -- 172.21.15.28:0/3928119719 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41002c060 con 0x5589b5967570 2024-04-25T01:38:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.269+0000 7fa3fdffb640 1 -- 172.21.15.28:0/3928119719 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fa41001d970 con 0x5589b5967570 2024-04-25T01:38:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.269+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3928119719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4000bc570 con 0x5589b5967570 2024-04-25T01:38:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.269+0000 7fa3fdffb640 1 -- 172.21.15.28:0/3928119719 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa410033050 con 0x5589b5967570 2024-04-25T01:38:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.269+0000 7fa41fdaa740 10 client.17316 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.269+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3928119719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5589b55d53f0 con 0x5589b5967570 2024-04-25T01:38:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.269+0000 7fa3fdffb640 1 -- 172.21.15.28:0/3928119719 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4000bc570 con 0x5589b5967570 2024-04-25T01:38:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.269+0000 7fa3fdffb640 1 -- 172.21.15.28:0/3928119719 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa41001c380 con 0x5589b5967570 2024-04-25T01:38:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.269+0000 7fa41fdaa740 10 client.17316 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.269+0000 7fa41fdaa740 10 client.17316 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:38:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.269+0000 7fa41fdaa740 20 client.17316 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.269+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/3928119719 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5589b5977330 0x5589b5979760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.269+0000 7fa41fdaa740 4 client.17316 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.269+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3928119719 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589b58cebb0 con 0x5589b5977330 2024-04-25T01:38:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.270+0000 7fa41659c640 1 --2- 172.21.15.28:0/3928119719 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5589b5977330 0x5589b5979760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.270+0000 7fa3fdffb640 10 client.17316 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:38:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.270+0000 7fa41659c640 1 --2- 172.21.15.28:0/3928119719 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5589b5977330 0x5589b5979760 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=513d1cca41674bf7 server_cookie=f604b7cf076fdd53 in_seq=0 out_seq=0 2024-04-25T01:38:36.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.271+0000 7fa3fdffb640 1 -- 172.21.15.28:0/3928119719 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5589b58cebb0 con 0x5589b5977330 2024-04-25T01:38:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.271+0000 7fa3fdffb640 10 client.17316 handle_command_reply: tid=0 2024-04-25T01:38:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.271+0000 7fa41fdaa740 1 client.17316 shutdown 2024-04-25T01:38:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.271+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3928119719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b5967570 msgr2=0x5589b59646d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.271+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/3928119719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b5967570 0x5589b59646d0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fa410010af0 tx=0x7fa410004a40 comp rx=0 tx=0).stop 2024-04-25T01:38:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.271+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3928119719 shutdown_connections 2024-04-25T01:38:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.271+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/3928119719 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5589b5977330 0x5589b5979760 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.271+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/3928119719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b5967570 0x5589b59646d0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.271+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/3928119719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b595cea0 0x5589b5964190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.271+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/3928119719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b595c580 0x5589b5966ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.271+0000 7fa3fdffb640 0 client.17316 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:38:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.271+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3928119719 >> 172.21.15.28:0/3928119719 conn(0x5589b59588a0 msgr2=0x5589b595e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.271+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3928119719 shutdown_connections 2024-04-25T01:38:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.271+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3928119719 wait complete. 2024-04-25T01:38:36.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.271+0000 7fa41fdaa740 20 client.17316 trim_cache size 0 max 16384 2024-04-25T01:38:36.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.284+0000 7fa41fdaa740 1 Processor -- start 2024-04-25T01:38:36.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.284+0000 7fa41fdaa740 1 -- start start 2024-04-25T01:38:36.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa41fdaa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b595c580 0x5589b5972f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa41fdaa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b595cea0 0x5589b597f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa41fdaa740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b5967570 0x5589b597f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa41fdaa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa4181aea40 con 0x5589b595cea0 2024-04-25T01:38:36.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa41fdaa740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa4181b0580 con 0x5589b5967570 2024-04-25T01:38:36.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa41fdaa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa4181af110 con 0x5589b595c580 2024-04-25T01:38:36.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa40ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b595cea0 0x5589b597f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa40ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b595cea0 0x5589b597f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36822/0 (socket says 172.21.15.28:36822) 2024-04-25T01:38:36.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa40ffff640 1 -- 172.21.15.28:0/1827113033 learned_addr learned my addr 172.21.15.28:0/1827113033 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa416d9d640 1 --2- 172.21.15.28:0/1827113033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b5967570 0x5589b597f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa41659c640 1 --2- 172.21.15.28:0/1827113033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b595c580 0x5589b5972f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa40ffff640 1 -- 172.21.15.28:0/1827113033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b595c580 msgr2=0x5589b5972f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa40ffff640 1 --2- 172.21.15.28:0/1827113033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b595c580 0x5589b5972f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa40ffff640 1 -- 172.21.15.28:0/1827113033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b5967570 msgr2=0x5589b597f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa40ffff640 1 --2- 172.21.15.28:0/1827113033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b5967570 0x5589b597f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa40ffff640 1 -- 172.21.15.28:0/1827113033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589b569f7d0 con 0x5589b595cea0 2024-04-25T01:38:36.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa41659c640 1 --2- 172.21.15.28:0/1827113033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b595c580 0x5589b5972f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:36.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa416d9d640 1 --2- 172.21.15.28:0/1827113033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b5967570 0x5589b597f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:36.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa40ffff640 1 --2- 172.21.15.28:0/1827113033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b595cea0 0x5589b597f030 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fa4000b82d0 tx=0x7fa4000b8110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a41137e83cb134d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa3fffff640 1 -- 172.21.15.28:0/1827113033 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa400049070 con 0x5589b595cea0 2024-04-25T01:38:36.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa3fffff640 1 -- 172.21.15.28:0/1827113033 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4000bc570 con 0x5589b595cea0 2024-04-25T01:38:36.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.285+0000 7fa3fffff640 1 -- 172.21.15.28:0/1827113033 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4000ed320 con 0x5589b595cea0 2024-04-25T01:38:36.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.286+0000 7fa41fdaa740 1 -- 172.21.15.28:0/1827113033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b595cea0 msgr2=0x5589b597f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.286+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/1827113033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b595cea0 0x5589b597f030 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fa4000b82d0 tx=0x7fa4000b8110 comp rx=0 tx=0).stop 2024-04-25T01:38:36.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.286+0000 7fa41fdaa740 1 -- 172.21.15.28:0/1827113033 shutdown_connections 2024-04-25T01:38:36.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.286+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/1827113033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b5967570 0x5589b597f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.286+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/1827113033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b595cea0 0x5589b597f030 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.286+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/1827113033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b595c580 0x5589b5972f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.286+0000 7fa41fdaa740 1 -- 172.21.15.28:0/1827113033 >> 172.21.15.28:0/1827113033 conn(0x5589b59588a0 msgr2=0x5589b5957b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.286+0000 7fa41fdaa740 1 -- 172.21.15.28:0/1827113033 shutdown_connections 2024-04-25T01:38:36.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.286+0000 7fa41fdaa740 1 -- 172.21.15.28:0/1827113033 wait complete. 2024-04-25T01:38:36.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.286+0000 7fa41fdaa740 1 -- start start 2024-04-25T01:38:36.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.286+0000 7fa41fdaa740 1 Processor -- start 2024-04-25T01:38:36.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa41fdaa740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b595c580 0x5589b596f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa41fdaa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b595cea0 0x5589b596cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa41fdaa740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b5967570 0x5589b596d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa41fdaa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa4181afeb0 con 0x5589b595cea0 2024-04-25T01:38:36.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa41fdaa740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa4181af7e0 con 0x5589b595c580 2024-04-25T01:38:36.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa41fdaa740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa4181aca20 con 0x5589b5967570 2024-04-25T01:38:36.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa41fdaa740 10 client.0 _ll_register_callbacks cb 0x7fa410009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:36.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa40ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b595cea0 0x5589b596cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa40ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b595cea0 0x5589b596cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36826/0 (socket says 172.21.15.28:36826) 2024-04-25T01:38:36.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa40ffff640 1 -- 172.21.15.28:0/3693442988 learned_addr learned my addr 172.21.15.28:0/3693442988 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa41659c640 1 --2- 172.21.15.28:0/3693442988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b595c580 0x5589b596f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa416d9d640 1 --2- 172.21.15.28:0/3693442988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b5967570 0x5589b596d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa40ffff640 1 -- 172.21.15.28:0/3693442988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b5967570 msgr2=0x5589b596d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa40ffff640 1 --2- 172.21.15.28:0/3693442988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b5967570 0x5589b596d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa40ffff640 1 -- 172.21.15.28:0/3693442988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b595c580 msgr2=0x5589b596f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa40ffff640 1 --2- 172.21.15.28:0/3693442988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b595c580 0x5589b596f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa40ffff640 1 -- 172.21.15.28:0/3693442988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589b596d8b0 con 0x5589b595cea0 2024-04-25T01:38:36.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa40ffff640 1 --2- 172.21.15.28:0/3693442988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b595cea0 0x5589b596cba0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fa4000bbb70 tx=0x7fa40002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5bd03af81894af6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa3fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:36.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa3fdffb640 1 -- 172.21.15.28:0/3693442988 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa400049040 con 0x5589b595cea0 2024-04-25T01:38:36.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa3fdffb640 1 -- 172.21.15.28:0/3693442988 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fa4000b84e0 con 0x5589b595cea0 2024-04-25T01:38:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3693442988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa4000bc570 con 0x5589b595cea0 2024-04-25T01:38:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa3fdffb640 1 -- 172.21.15.28:0/3693442988 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa40003b560 con 0x5589b595cea0 2024-04-25T01:38:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa41fdaa740 10 client.17322 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.287+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3693442988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5589b596d8b0 con 0x5589b595cea0 2024-04-25T01:38:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.288+0000 7fa3fdffb640 1 -- 172.21.15.28:0/3693442988 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4000bc570 con 0x5589b595cea0 2024-04-25T01:38:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.288+0000 7fa3fdffb640 1 -- 172.21.15.28:0/3693442988 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa400031bd0 con 0x5589b595cea0 2024-04-25T01:38:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.288+0000 7fa41fdaa740 10 client.17322 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.288+0000 7fa41fdaa740 10 client.17322 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:38:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.288+0000 7fa41fdaa740 20 client.17322 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.288+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/3693442988 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5589b59dd8d0 0x5589b59dfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.288+0000 7fa41fdaa740 4 client.17322 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:38:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.288+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3693442988 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5589b58c8f00 con 0x5589b59dd8d0 2024-04-25T01:38:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.288+0000 7fa41659c640 1 --2- 172.21.15.28:0/3693442988 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5589b59dd8d0 0x5589b59dfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.288+0000 7fa41659c640 1 --2- 172.21.15.28:0/3693442988 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5589b59dd8d0 0x5589b59dfcf0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6dbab06ac03f3715 server_cookie=84ea9f727b6d092 in_seq=0 out_seq=0 2024-04-25T01:38:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.288+0000 7fa3fdffb640 10 client.17322 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:38:36.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa3fdffb640 1 -- 172.21.15.28:0/3693442988 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5589b58c8f00 con 0x5589b59dd8d0 2024-04-25T01:38:36.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa3fdffb640 10 client.17322 handle_command_reply: tid=0 2024-04-25T01:38:36.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41fdaa740 1 client.17322 shutdown 2024-04-25T01:38:36.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3693442988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b595cea0 msgr2=0x5589b596cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/3693442988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b595cea0 0x5589b596cba0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fa4000bbb70 tx=0x7fa40002a420 comp rx=0 tx=0).stop 2024-04-25T01:38:36.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3693442988 shutdown_connections 2024-04-25T01:38:36.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/3693442988 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5589b59dd8d0 0x5589b59dfcf0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/3693442988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5589b5967570 0x5589b596d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa3fdffb640 0 client.17322 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:38:36.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/3693442988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589b595cea0 0x5589b596cba0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41fdaa740 1 --2- 172.21.15.28:0/3693442988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589b595c580 0x5589b596f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3693442988 >> 172.21.15.28:0/3693442988 conn(0x5589b59588a0 msgr2=0x5589b5971f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3693442988 shutdown_connections 2024-04-25T01:38:36.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41fdaa740 1 -- 172.21.15.28:0/3693442988 wait complete. 2024-04-25T01:38:36.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41fdaa740 20 client.17322 trim_cache size 0 max 16384 2024-04-25T01:38:36.278 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:38:36.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41cfc1640 1 -- 172.21.15.28:0/183889203 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa4040a2d80 msgr2=0x7fa4040a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41cfc1640 1 --2- 172.21.15.28:0/183889203 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa4040a2d80 0x7fa4040a5240 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa410009fd0 tx=0x7fa410009560 comp rx=0 tx=0).stop 2024-04-25T01:38:36.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41cfc1640 1 -- 172.21.15.28:0/183889203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4181020d0 msgr2=0x7fa4181984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41cfc1640 1 --2- 172.21.15.28:0/183889203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4181020d0 0x7fa4181984f0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fa400009230 tx=0x7fa400004270 comp rx=0 tx=0).stop 2024-04-25T01:38:36.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41cfc1640 1 -- 172.21.15.28:0/183889203 shutdown_connections 2024-04-25T01:38:36.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41cfc1640 1 --2- 172.21.15.28:0/183889203 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa4040a2d80 0x7fa4040a5240 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41cfc1640 1 --2- 172.21.15.28:0/183889203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa418107720 0x7fa41819c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41cfc1640 1 --2- 172.21.15.28:0/183889203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4181020d0 0x7fa4181984f0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41cfc1640 1 --2- 172.21.15.28:0/183889203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa418101710 0x7fa418197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.289+0000 7fa41cfc1640 1 -- 172.21.15.28:0/183889203 >> 172.21.15.28:0/183889203 conn(0x7fa4180fd180 msgr2=0x7fa4180fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.290+0000 7fa41cfc1640 1 -- 172.21.15.28:0/183889203 shutdown_connections 2024-04-25T01:38:36.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.290+0000 7fa41cfc1640 1 -- 172.21.15.28:0/183889203 wait complete. 2024-04-25T01:38:36.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:38:36.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.445+0000 7f72e0a10640 1 -- 172.21.15.28:0/1100832395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72dc106630 msgr2=0x7f72dc106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.445+0000 7f72e0a10640 1 --2- 172.21.15.28:0/1100832395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72dc106630 0x7f72dc106a10 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009930 tx=0x7f72cc02a420 comp rx=0 tx=0).stop 2024-04-25T01:38:36.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.445+0000 7f72e0a10640 1 -- 172.21.15.28:0/1100832395 shutdown_connections 2024-04-25T01:38:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.445+0000 7f72e0a10640 1 --2- 172.21.15.28:0/1100832395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72dc100fe0 0x7f72dc10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.445+0000 7f72e0a10640 1 --2- 172.21.15.28:0/1100832395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72dc100620 0x7f72dc100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.445+0000 7f72e0a10640 1 --2- 172.21.15.28:0/1100832395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72dc106630 0x7f72dc106a10 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.445+0000 7f72e0a10640 1 -- 172.21.15.28:0/1100832395 >> 172.21.15.28:0/1100832395 conn(0x7f72dc0776a0 msgr2=0x7f72dc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.445+0000 7f72e0a10640 1 -- 172.21.15.28:0/1100832395 shutdown_connections 2024-04-25T01:38:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.445+0000 7f72e0a10640 1 -- 172.21.15.28:0/1100832395 wait complete. 2024-04-25T01:38:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.446+0000 7f72e0a10640 1 Processor -- start 2024-04-25T01:38:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.446+0000 7f72e0a10640 1 -- start start 2024-04-25T01:38:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.446+0000 7f72e0a10640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72dc100620 0x7f72dc1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.446+0000 7f72e0a10640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72dc100fe0 0x7f72dc198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.446+0000 7f72e0a10640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72dc106630 0x7f72dc19c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.446+0000 7f72e0a10640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc111b40 con 0x7f72dc106630 2024-04-25T01:38:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.446+0000 7f72e0a10640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc1119c0 con 0x7f72dc100620 2024-04-25T01:38:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.446+0000 7f72e0a10640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc111cc0 con 0x7f72dc100fe0 2024-04-25T01:38:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.446+0000 7f72dad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72dc106630 0x7f72dc19c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.446+0000 7f72dad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72dc106630 0x7f72dc19c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36842/0 (socket says 172.21.15.28:36842) 2024-04-25T01:38:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.446+0000 7f72dad9d640 1 -- 172.21.15.28:0/4188992984 learned_addr learned my addr 172.21.15.28:0/4188992984 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.446+0000 7f72d3fff640 1 --2- 172.21.15.28:0/4188992984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72dc100fe0 0x7f72dc198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.446+0000 7f72da59c640 1 --2- 172.21.15.28:0/4188992984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72dc100620 0x7f72dc1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.447+0000 7f72dad9d640 1 -- 172.21.15.28:0/4188992984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72dc100fe0 msgr2=0x7f72dc198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.447+0000 7f72dad9d640 1 --2- 172.21.15.28:0/4188992984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72dc100fe0 0x7f72dc198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.447+0000 7f72dad9d640 1 -- 172.21.15.28:0/4188992984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72dc100620 msgr2=0x7f72dc1980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.447+0000 7f72dad9d640 1 --2- 172.21.15.28:0/4188992984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72dc100620 0x7f72dc1980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.447+0000 7f72dad9d640 1 -- 172.21.15.28:0/4188992984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72dc19d060 con 0x7f72dc106630 2024-04-25T01:38:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.447+0000 7f72dad9d640 1 --2- 172.21.15.28:0/4188992984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72dc106630 0x7f72dc19c930 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f72d400eb10 tx=0x7f72d400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.447+0000 7f72d37fe640 1 -- 172.21.15.28:0/4188992984 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d4004b20 con 0x7f72dc106630 2024-04-25T01:38:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.447+0000 7f72e0a10640 1 -- 172.21.15.28:0/4188992984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72dc19d350 con 0x7f72dc106630 2024-04-25T01:38:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.447+0000 7f72d37fe640 1 -- 172.21.15.28:0/4188992984 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72d4004cc0 con 0x7f72dc106630 2024-04-25T01:38:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.447+0000 7f72d37fe640 1 -- 172.21.15.28:0/4188992984 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d4010760 con 0x7f72dc106630 2024-04-25T01:38:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.447+0000 7f72e0a10640 1 -- 172.21.15.28:0/4188992984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72dc1a4be0 con 0x7f72dc106630 2024-04-25T01:38:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.449+0000 7f72d37fe640 1 -- 172.21.15.28:0/4188992984 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f72d4005020 con 0x7f72dc106630 2024-04-25T01:38:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.449+0000 7f72e37f4740 1 Processor -- start 2024-04-25T01:38:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.450+0000 7f72e37f4740 1 -- start start 2024-04-25T01:38:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.450+0000 7f72e37f4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df079580 0x55d8df079960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.450+0000 7f72e37f4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079ea0 0x55d8df084030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.450+0000 7f72e37f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df084570 0x55d8df086960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.450+0000 7f72e37f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d8defde670 con 0x55d8df084570 2024-04-25T01:38:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.450+0000 7f72e37f4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d8def86970 con 0x55d8df079580 2024-04-25T01:38:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.450+0000 7f72e37f4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55d8defda630 con 0x55d8df079ea0 2024-04-25T01:38:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.450+0000 7f72d37fe640 1 --2- 172.21.15.28:0/4188992984 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f72c80a2d00 0x7f72c80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.450+0000 7f72d3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df084570 0x55d8df086960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.450+0000 7f72d3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df084570 0x55d8df086960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36858/0 (socket says 172.21.15.28:36858) 2024-04-25T01:38:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.450+0000 7f72dad9d640 1 --2- 172.21.15.28:0/4188992984 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f72c80a2d00 0x7f72c80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.450+0000 7f72d3fff640 1 -- 172.21.15.28:0/381045345 learned_addr learned my addr 172.21.15.28:0/381045345 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.450+0000 7f72d3fff640 1 --2- 172.21.15.28:0/381045345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df079580 0x55d8df079960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.450+0000 7f72d37fe640 1 -- 172.21.15.28:0/4188992984 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f72d40cfd90 con 0x7f72dc106630 2024-04-25T01:38:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.450+0000 7f72da59c640 1 --2- 172.21.15.28:0/381045345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079ea0 0x55d8df084030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.451+0000 7f72d3fff640 1 -- 172.21.15.28:0/381045345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079ea0 msgr2=0x55d8df084030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.451+0000 7f72d3fff640 1 --2- 172.21.15.28:0/381045345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079ea0 0x55d8df084030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.451+0000 7f72d3fff640 1 -- 172.21.15.28:0/381045345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df079580 msgr2=0x55d8df079960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.451+0000 7f72d3fff640 1 --2- 172.21.15.28:0/381045345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df079580 0x55d8df079960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.451+0000 7f72d3fff640 1 -- 172.21.15.28:0/381045345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8defaa400 con 0x55d8df084570 2024-04-25T01:38:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.451+0000 7f72dad9d640 1 --2- 172.21.15.28:0/4188992984 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f72c80a2d00 0x7f72c80a51c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f72d409bea0 tx=0x7f72d40bc040 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.451+0000 7f72d3fff640 1 --2- 172.21.15.28:0/381045345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df084570 0x55d8df086960 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f72c40127a0 tx=0x7f72c400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db3e7f2edb2941f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.451+0000 7f72d0ff9640 1 -- 172.21.15.28:0/381045345 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c4012d20 con 0x55d8df084570 2024-04-25T01:38:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.451+0000 7f72d0ff9640 1 -- 172.21.15.28:0/381045345 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72c4005020 con 0x55d8df084570 2024-04-25T01:38:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.451+0000 7f72d0ff9640 1 -- 172.21.15.28:0/381045345 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c4004ae0 con 0x55d8df084570 2024-04-25T01:38:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.452+0000 7f72e37f4740 1 -- 172.21.15.28:0/381045345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df084570 msgr2=0x55d8df086960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.452+0000 7f72e37f4740 1 --2- 172.21.15.28:0/381045345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df084570 0x55d8df086960 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f72c40127a0 tx=0x7f72c400b7c0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.452+0000 7f72e37f4740 1 -- 172.21.15.28:0/381045345 shutdown_connections 2024-04-25T01:38:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.452+0000 7f72e37f4740 1 --2- 172.21.15.28:0/381045345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df084570 0x55d8df086960 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.452+0000 7f72e37f4740 1 --2- 172.21.15.28:0/381045345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079ea0 0x55d8df084030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.452+0000 7f72e37f4740 1 --2- 172.21.15.28:0/381045345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df079580 0x55d8df079960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.452+0000 7f72e37f4740 1 -- 172.21.15.28:0/381045345 >> 172.21.15.28:0/381045345 conn(0x55d8df0758a0 msgr2=0x55d8df075c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.452+0000 7f72e37f4740 1 -- 172.21.15.28:0/381045345 shutdown_connections 2024-04-25T01:38:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.452+0000 7f72e37f4740 1 -- 172.21.15.28:0/381045345 wait complete. 2024-04-25T01:38:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.452+0000 7f72e37f4740 1 -- start start 2024-04-25T01:38:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.452+0000 7f72e37f4740 1 Processor -- start 2024-04-25T01:38:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72e37f4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079580 0x55d8df083ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72e37f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df079ea0 0x55d8df0811b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72e37f4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df084570 0x55d8df081710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72e37f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d8df087360 con 0x55d8df079ea0 2024-04-25T01:38:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72e37f4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d8def9a050 con 0x55d8df084570 2024-04-25T01:38:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72e37f4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55d8df0874e0 con 0x55d8df079580 2024-04-25T01:38:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72e37f4740 10 client.0 _ll_register_callbacks cb 0x55d8decf1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72da59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df079ea0 0x55d8df0811b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72da59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df079ea0 0x55d8df0811b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36872/0 (socket says 172.21.15.28:36872) 2024-04-25T01:38:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72da59c640 1 -- 172.21.15.28:0/2846866684 learned_addr learned my addr 172.21.15.28:0/2846866684 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72d3fff640 1 --2- 172.21.15.28:0/2846866684 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079580 0x55d8df083ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72d3fff640 1 --2- 172.21.15.28:0/2846866684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df084570 0x55d8df081710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72da59c640 1 -- 172.21.15.28:0/2846866684 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079580 msgr2=0x55d8df083ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72da59c640 1 --2- 172.21.15.28:0/2846866684 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079580 0x55d8df083ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72da59c640 1 -- 172.21.15.28:0/2846866684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df084570 msgr2=0x55d8df081710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72da59c640 1 --2- 172.21.15.28:0/2846866684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df084570 0x55d8df081710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72da59c640 1 -- 172.21.15.28:0/2846866684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8def6bdc0 con 0x55d8df079ea0 2024-04-25T01:38:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72d3fff640 1 --2- 172.21.15.28:0/2846866684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df084570 0x55d8df081710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72da59c640 1 --2- 172.21.15.28:0/2846866684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df079ea0 0x55d8df0811b0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009930 tx=0x7f72cc0317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76bb7f34444c64b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72c1ffb640 1 -- 172.21.15.28:0/2846866684 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc035070 con 0x55d8df079ea0 2024-04-25T01:38:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72c1ffb640 1 -- 172.21.15.28:0/2846866684 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f72cc004040 con 0x55d8df079ea0 2024-04-25T01:38:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.453+0000 7f72e37f4740 1 -- 172.21.15.28:0/2846866684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72c4005020 con 0x55d8df079ea0 2024-04-25T01:38:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.454+0000 7f72c1ffb640 1 -- 172.21.15.28:0/2846866684 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f72cc004240 con 0x55d8df079ea0 2024-04-25T01:38:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.454+0000 7f72e37f4740 10 client.17334 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.454+0000 7f72e37f4740 1 -- 172.21.15.28:0/2846866684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d8decf23f0 con 0x55d8df079ea0 2024-04-25T01:38:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.454+0000 7f72c1ffb640 1 -- 172.21.15.28:0/2846866684 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72c4005020 con 0x55d8df079ea0 2024-04-25T01:38:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.454+0000 7f72c1ffb640 1 -- 172.21.15.28:0/2846866684 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f72cc004a90 con 0x55d8df079ea0 2024-04-25T01:38:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.454+0000 7f72e37f4740 10 client.17334 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.454+0000 7f72e37f4740 10 client.17334 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:38:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.454+0000 7f72e37f4740 20 client.17334 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.454+0000 7f72e37f4740 1 --2- 172.21.15.28:0/2846866684 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55d8df0945c0 0x55d8df0969b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.454+0000 7f72e37f4740 4 client.17334 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.454+0000 7f72e37f4740 1 -- 172.21.15.28:0/2846866684 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8defebbb0 con 0x55d8df0945c0 2024-04-25T01:38:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.454+0000 7f72d3fff640 1 --2- 172.21.15.28:0/2846866684 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55d8df0945c0 0x55d8df0969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.455+0000 7f72c1ffb640 10 client.17334 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:38:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.455+0000 7f72d3fff640 1 --2- 172.21.15.28:0/2846866684 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55d8df0945c0 0x55d8df0969b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9c5f1219d9c0f575 server_cookie=db55ae9ebf23486f in_seq=0 out_seq=0 2024-04-25T01:38:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.455+0000 7f72c1ffb640 1 -- 172.21.15.28:0/2846866684 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d8defebbb0 con 0x55d8df0945c0 2024-04-25T01:38:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.455+0000 7f72c1ffb640 10 client.17334 handle_command_reply: tid=0 2024-04-25T01:38:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.455+0000 7f72e37f4740 1 client.17334 shutdown 2024-04-25T01:38:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.455+0000 7f72e37f4740 1 -- 172.21.15.28:0/2846866684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df079ea0 msgr2=0x55d8df0811b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.455+0000 7f72e37f4740 1 --2- 172.21.15.28:0/2846866684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df079ea0 0x55d8df0811b0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009930 tx=0x7f72cc0317e0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.455+0000 7f72e37f4740 1 -- 172.21.15.28:0/2846866684 shutdown_connections 2024-04-25T01:38:36.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.456+0000 7f72e37f4740 1 --2- 172.21.15.28:0/2846866684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df084570 0x55d8df081710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.456+0000 7f72e37f4740 1 --2- 172.21.15.28:0/2846866684 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55d8df0945c0 0x55d8df0969b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.456+0000 7f72e37f4740 1 --2- 172.21.15.28:0/2846866684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df079ea0 0x55d8df0811b0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.456+0000 7f72e37f4740 1 --2- 172.21.15.28:0/2846866684 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079580 0x55d8df083ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.456+0000 7f72c1ffb640 0 client.17334 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:38:36.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.456+0000 7f72e37f4740 1 -- 172.21.15.28:0/2846866684 >> 172.21.15.28:0/2846866684 conn(0x55d8df0758a0 msgr2=0x55d8df07b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.456+0000 7f72e37f4740 1 -- 172.21.15.28:0/2846866684 shutdown_connections 2024-04-25T01:38:36.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.456+0000 7f72e37f4740 1 -- 172.21.15.28:0/2846866684 wait complete. 2024-04-25T01:38:36.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.456+0000 7f72e37f4740 20 client.17334 trim_cache size 0 max 16384 2024-04-25T01:38:36.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.469+0000 7f72e37f4740 1 Processor -- start 2024-04-25T01:38:36.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.469+0000 7f72e37f4740 1 -- start start 2024-04-25T01:38:36.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.469+0000 7f72e37f4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df079580 0x55d8df0839f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.469+0000 7f72e37f4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079ea0 0x55d8df09c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.469+0000 7f72e37f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df084570 0x55d8df09c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.469+0000 7f72e37f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc1aea50 con 0x55d8df084570 2024-04-25T01:38:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.469+0000 7f72e37f4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc1b0590 con 0x55d8df079580 2024-04-25T01:38:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.469+0000 7f72e37f4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc1af120 con 0x55d8df079ea0 2024-04-25T01:38:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.469+0000 7f72d3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df084570 0x55d8df09c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.469+0000 7f72d3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df084570 0x55d8df09c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36882/0 (socket says 172.21.15.28:36882) 2024-04-25T01:38:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.469+0000 7f72d3fff640 1 -- 172.21.15.28:0/3743729458 learned_addr learned my addr 172.21.15.28:0/3743729458 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.469+0000 7f72d3fff640 1 --2- 172.21.15.28:0/3743729458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df079580 0x55d8df0839f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.469+0000 7f72da59c640 1 --2- 172.21.15.28:0/3743729458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079ea0 0x55d8df09c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.470+0000 7f72d3fff640 1 -- 172.21.15.28:0/3743729458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079ea0 msgr2=0x55d8df09c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.470+0000 7f72d3fff640 1 --2- 172.21.15.28:0/3743729458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079ea0 0x55d8df09c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.470+0000 7f72d3fff640 1 -- 172.21.15.28:0/3743729458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df079580 msgr2=0x55d8df0839f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.470+0000 7f72d3fff640 1 --2- 172.21.15.28:0/3743729458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df079580 0x55d8df0839f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.470+0000 7f72d3fff640 1 -- 172.21.15.28:0/3743729458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8dedbc7d0 con 0x55d8df084570 2024-04-25T01:38:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.470+0000 7f72da59c640 1 --2- 172.21.15.28:0/3743729458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079ea0 0x55d8df09c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.470+0000 7f72d3fff640 1 --2- 172.21.15.28:0/3743729458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df084570 0x55d8df09c7c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x55d8df076a30 tx=0x7f72c400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ebe9da7cdeb872 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.470+0000 7f72c3fff640 1 -- 172.21.15.28:0/3743729458 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c4004030 con 0x55d8df084570 2024-04-25T01:38:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.470+0000 7f72c3fff640 1 -- 172.21.15.28:0/3743729458 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72c402f830 con 0x55d8df084570 2024-04-25T01:38:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.470+0000 7f72c3fff640 1 -- 172.21.15.28:0/3743729458 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c4010880 con 0x55d8df084570 2024-04-25T01:38:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.470+0000 7f72e37f4740 1 -- 172.21.15.28:0/3743729458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df084570 msgr2=0x55d8df09c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.470+0000 7f72e37f4740 1 --2- 172.21.15.28:0/3743729458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df084570 0x55d8df09c7c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x55d8df076a30 tx=0x7f72c400b7c0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.470+0000 7f72e37f4740 1 -- 172.21.15.28:0/3743729458 shutdown_connections 2024-04-25T01:38:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.470+0000 7f72e37f4740 1 --2- 172.21.15.28:0/3743729458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df084570 0x55d8df09c7c0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.470+0000 7f72e37f4740 1 --2- 172.21.15.28:0/3743729458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079ea0 0x55d8df09c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.470+0000 7f72e37f4740 1 --2- 172.21.15.28:0/3743729458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df079580 0x55d8df0839f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.470+0000 7f72e37f4740 1 -- 172.21.15.28:0/3743729458 >> 172.21.15.28:0/3743729458 conn(0x55d8df0758a0 msgr2=0x55d8df0742b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.471+0000 7f72e37f4740 1 -- 172.21.15.28:0/3743729458 shutdown_connections 2024-04-25T01:38:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.471+0000 7f72e37f4740 1 -- 172.21.15.28:0/3743729458 wait complete. 2024-04-25T01:38:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.471+0000 7f72e37f4740 1 -- start start 2024-04-25T01:38:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.471+0000 7f72e37f4740 1 Processor -- start 2024-04-25T01:38:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.471+0000 7f72e37f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df079580 0x55d8df088440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.471+0000 7f72e37f4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079ea0 0x55d8df088980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.471+0000 7f72e37f4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df084570 0x55d8df088f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.471+0000 7f72e37f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc1afec0 con 0x55d8df079580 2024-04-25T01:38:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.471+0000 7f72e37f4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc1af7f0 con 0x55d8df084570 2024-04-25T01:38:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.471+0000 7f72e37f4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc1aca30 con 0x55d8df079ea0 2024-04-25T01:38:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.471+0000 7f72e37f4740 10 client.0 _ll_register_callbacks cb 0x7f72d40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.471+0000 7f72d3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df079580 0x55d8df088440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.471+0000 7f72d3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df079580 0x55d8df088440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36884/0 (socket says 172.21.15.28:36884) 2024-04-25T01:38:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.471+0000 7f72d3fff640 1 -- 172.21.15.28:0/2268410926 learned_addr learned my addr 172.21.15.28:0/2268410926 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.471+0000 7f72d3fff640 1 --2- 172.21.15.28:0/2268410926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df084570 0x55d8df088f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.471+0000 7f72da59c640 1 --2- 172.21.15.28:0/2268410926 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079ea0 0x55d8df088980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.471+0000 7f72d3fff640 1 -- 172.21.15.28:0/2268410926 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079ea0 msgr2=0x55d8df088980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72d3fff640 1 --2- 172.21.15.28:0/2268410926 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079ea0 0x55d8df088980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72d3fff640 1 -- 172.21.15.28:0/2268410926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df084570 msgr2=0x55d8df088f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72d3fff640 1 --2- 172.21.15.28:0/2268410926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df084570 0x55d8df088f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72d3fff640 1 -- 172.21.15.28:0/2268410926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8df0f9de0 con 0x55d8df079580 2024-04-25T01:38:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72da59c640 1 --2- 172.21.15.28:0/2268410926 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079ea0 0x55d8df088980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72d3fff640 1 --2- 172.21.15.28:0/2268410926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df079580 0x55d8df088440 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f72c400b6b0 tx=0x7f72c400b520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce443fd68c92eb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72c1ffb640 1 -- 172.21.15.28:0/2268410926 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c402f5e0 con 0x55d8df079580 2024-04-25T01:38:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72c1ffb640 1 -- 172.21.15.28:0/2268410926 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f72c4010740 con 0x55d8df079580 2024-04-25T01:38:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72e37f4740 1 -- 172.21.15.28:0/2268410926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72c402f830 con 0x55d8df079580 2024-04-25T01:38:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72c1ffb640 1 -- 172.21.15.28:0/2268410926 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f72c4010940 con 0x55d8df079580 2024-04-25T01:38:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72e37f4740 10 client.17340 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72e37f4740 1 -- 172.21.15.28:0/2268410926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d8df0fb050 con 0x55d8df079580 2024-04-25T01:38:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72c1ffb640 1 -- 172.21.15.28:0/2268410926 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72c402f830 con 0x55d8df079580 2024-04-25T01:38:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72c1ffb640 1 -- 172.21.15.28:0/2268410926 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f72c402fce0 con 0x55d8df079580 2024-04-25T01:38:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72e37f4740 10 client.17340 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72e37f4740 10 client.17340 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:38:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72e37f4740 20 client.17340 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72e37f4740 1 --2- 172.21.15.28:0/2268410926 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55d8df0fb530 0x55d8df0fd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72e37f4740 4 client.17340 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:38:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.472+0000 7f72e37f4740 1 -- 172.21.15.28:0/2268410926 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d8defe5f00 con 0x55d8df0fb530 2024-04-25T01:38:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.473+0000 7f72da59c640 1 --2- 172.21.15.28:0/2268410926 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55d8df0fb530 0x55d8df0fd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.473+0000 7f72da59c640 1 --2- 172.21.15.28:0/2268410926 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55d8df0fb530 0x55d8df0fd950 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=486fc96ba4638c79 server_cookie=1398a14f44aaa53c in_seq=0 out_seq=0 2024-04-25T01:38:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.473+0000 7f72c1ffb640 10 client.17340 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:38:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.473+0000 7f72c1ffb640 1 -- 172.21.15.28:0/2268410926 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d8defe5f00 con 0x55d8df0fb530 2024-04-25T01:38:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.473+0000 7f72c1ffb640 10 client.17340 handle_command_reply: tid=0 2024-04-25T01:38:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.473+0000 7f72e37f4740 1 client.17340 shutdown 2024-04-25T01:38:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.473+0000 7f72e37f4740 1 -- 172.21.15.28:0/2268410926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df079580 msgr2=0x55d8df088440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.473+0000 7f72e37f4740 1 --2- 172.21.15.28:0/2268410926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df079580 0x55d8df088440 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f72c400b6b0 tx=0x7f72c400b520 comp rx=0 tx=0).stop 2024-04-25T01:38:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.473+0000 7f72e37f4740 1 -- 172.21.15.28:0/2268410926 shutdown_connections 2024-04-25T01:38:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.473+0000 7f72e37f4740 1 --2- 172.21.15.28:0/2268410926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d8df084570 0x55d8df088f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.473+0000 7f72e37f4740 1 --2- 172.21.15.28:0/2268410926 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d8df079ea0 0x55d8df088980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.473+0000 7f72e37f4740 1 --2- 172.21.15.28:0/2268410926 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55d8df0fb530 0x55d8df0fd950 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.473+0000 7f72e37f4740 1 --2- 172.21.15.28:0/2268410926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8df079580 0x55d8df088440 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.473+0000 7f72e37f4740 1 -- 172.21.15.28:0/2268410926 >> 172.21.15.28:0/2268410926 conn(0x55d8df0758a0 msgr2=0x55d8df074b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.473+0000 7f72c1ffb640 0 client.17340 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:38:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.473+0000 7f72e37f4740 1 -- 172.21.15.28:0/2268410926 shutdown_connections 2024-04-25T01:38:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.474+0000 7f72e37f4740 1 -- 172.21.15.28:0/2268410926 wait complete. 2024-04-25T01:38:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.474+0000 7f72e37f4740 20 client.17340 trim_cache size 0 max 16384 2024-04-25T01:38:36.463 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:38:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.474+0000 7f72e0a10640 1 -- 172.21.15.28:0/4188992984 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f72c80a2d00 msgr2=0x7f72c80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.474+0000 7f72e0a10640 1 --2- 172.21.15.28:0/4188992984 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f72c80a2d00 0x7f72c80a51c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f72d409bea0 tx=0x7f72d40bc040 comp rx=0 tx=0).stop 2024-04-25T01:38:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.474+0000 7f72e0a10640 1 -- 172.21.15.28:0/4188992984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72dc106630 msgr2=0x7f72dc19c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.474+0000 7f72e0a10640 1 --2- 172.21.15.28:0/4188992984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72dc106630 0x7f72dc19c930 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f72d400eb10 tx=0x7f72d400c2d0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.474+0000 7f72e0a10640 1 -- 172.21.15.28:0/4188992984 shutdown_connections 2024-04-25T01:38:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.474+0000 7f72e0a10640 1 --2- 172.21.15.28:0/4188992984 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f72c80a2d00 0x7f72c80a51c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.474+0000 7f72e0a10640 1 --2- 172.21.15.28:0/4188992984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72dc106630 0x7f72dc19c930 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.474+0000 7f72e0a10640 1 --2- 172.21.15.28:0/4188992984 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f72dc100fe0 0x7f72dc198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.474+0000 7f72e0a10640 1 --2- 172.21.15.28:0/4188992984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f72dc100620 0x7f72dc1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.474+0000 7f72e0a10640 1 -- 172.21.15.28:0/4188992984 >> 172.21.15.28:0/4188992984 conn(0x7f72dc0776a0 msgr2=0x7f72dc103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.474+0000 7f72e0a10640 1 -- 172.21.15.28:0/4188992984 shutdown_connections 2024-04-25T01:38:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.474+0000 7f72e0a10640 1 -- 172.21.15.28:0/4188992984 wait complete. 2024-04-25T01:38:36.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:38:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.591+0000 7f8278c09640 1 -- 172.21.15.28:0/4083441833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8274107720 msgr2=0x7f8274107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.591+0000 7f8278c09640 1 --2- 172.21.15.28:0/4083441833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8274107720 0x7f8274107b00 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f8264009930 tx=0x7f826402a420 comp rx=0 tx=0).stop 2024-04-25T01:38:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.591+0000 7f8278c09640 1 -- 172.21.15.28:0/4083441833 shutdown_connections 2024-04-25T01:38:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.591+0000 7f8278c09640 1 --2- 172.21.15.28:0/4083441833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f82741020d0 0x7f827410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.591+0000 7f8278c09640 1 --2- 172.21.15.28:0/4083441833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8274101710 0x7f8274101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.591+0000 7f8278c09640 1 --2- 172.21.15.28:0/4083441833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8274107720 0x7f8274107b00 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.591+0000 7f8278c09640 1 -- 172.21.15.28:0/4083441833 >> 172.21.15.28:0/4083441833 conn(0x7f82740fd180 msgr2=0x7f82740ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.591+0000 7f8278c09640 1 -- 172.21.15.28:0/4083441833 shutdown_connections 2024-04-25T01:38:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.591+0000 7f8278c09640 1 -- 172.21.15.28:0/4083441833 wait complete. 2024-04-25T01:38:36.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.592+0000 7f8278c09640 1 Processor -- start 2024-04-25T01:38:36.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.592+0000 7f8278c09640 1 -- start start 2024-04-25T01:38:36.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.592+0000 7f8278c09640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8274101710 0x7f8274198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.592+0000 7f8278c09640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82741020d0 0x7f8274198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.592+0000 7f8278c09640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8274107720 0x7f827419c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.592+0000 7f8278c09640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8274111b30 con 0x7f82741020d0 2024-04-25T01:38:36.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.592+0000 7f8278c09640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f82741119b0 con 0x7f8274107720 2024-04-25T01:38:36.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.592+0000 7f8278c09640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8274111cb0 con 0x7f8274101710 2024-04-25T01:38:36.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.592+0000 7f8271d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82741020d0 0x7f8274198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.592+0000 7f8271d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82741020d0 0x7f8274198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36892/0 (socket says 172.21.15.28:36892) 2024-04-25T01:38:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.592+0000 7f8271d9b640 1 -- 172.21.15.28:0/3522420472 learned_addr learned my addr 172.21.15.28:0/3522420472 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.592+0000 7f8272d9d640 1 --2- 172.21.15.28:0/3522420472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8274107720 0x7f827419c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.592+0000 7f827259c640 1 --2- 172.21.15.28:0/3522420472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8274101710 0x7f8274198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.593+0000 7f8271d9b640 1 -- 172.21.15.28:0/3522420472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8274101710 msgr2=0x7f8274198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.593+0000 7f8271d9b640 1 --2- 172.21.15.28:0/3522420472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8274101710 0x7f8274198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.593+0000 7f8271d9b640 1 -- 172.21.15.28:0/3522420472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8274107720 msgr2=0x7f827419c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.593+0000 7f8271d9b640 1 --2- 172.21.15.28:0/3522420472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8274107720 0x7f827419c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.593+0000 7f8271d9b640 1 -- 172.21.15.28:0/3522420472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827419d0b0 con 0x7f82741020d0 2024-04-25T01:38:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.593+0000 7f8271d9b640 1 --2- 172.21.15.28:0/3522420472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82741020d0 0x7f8274198650 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f826800c890 tx=0x7f826800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.593+0000 7f82637fe640 1 -- 172.21.15.28:0/3522420472 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8268004220 con 0x7f82741020d0 2024-04-25T01:38:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.593+0000 7f8278c09640 1 -- 172.21.15.28:0/3522420472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f827419d3a0 con 0x7f82741020d0 2024-04-25T01:38:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.593+0000 7f82637fe640 1 -- 172.21.15.28:0/3522420472 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82680043c0 con 0x7f82741020d0 2024-04-25T01:38:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.593+0000 7f82637fe640 1 -- 172.21.15.28:0/3522420472 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82680049e0 con 0x7f82741020d0 2024-04-25T01:38:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.593+0000 7f8278c09640 1 -- 172.21.15.28:0/3522420472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82741a4c30 con 0x7f82741020d0 2024-04-25T01:38:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.595+0000 7f82637fe640 1 -- 172.21.15.28:0/3522420472 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f82680079b0 con 0x7f82741020d0 2024-04-25T01:38:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f827b9f1740 1 Processor -- start 2024-04-25T01:38:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f827b9f1740 1 -- start start 2024-04-25T01:38:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f827b9f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849580 0x559512849960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f827b9f1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512849ea0 0x559512854030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f827b9f1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512854570 0x559512856960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f827b9f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5595127ae670 con 0x559512849580 2024-04-25T01:38:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f827b9f1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559512756970 con 0x559512854570 2024-04-25T01:38:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f827b9f1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5595127aa630 con 0x559512849ea0 2024-04-25T01:38:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f8272d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849580 0x559512849960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f8272d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849580 0x559512849960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36902/0 (socket says 172.21.15.28:36902) 2024-04-25T01:38:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f8272d9d640 1 -- 172.21.15.28:0/1207063685 learned_addr learned my addr 172.21.15.28:0/1207063685 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f8271d9b640 1 --2- 172.21.15.28:0/1207063685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512854570 0x559512856960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f82637fe640 1 --2- 172.21.15.28:0/3522420472 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f825c0a2d00 0x7f825c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f827259c640 1 --2- 172.21.15.28:0/1207063685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512849ea0 0x559512854030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f8272d9d640 1 -- 172.21.15.28:0/1207063685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512849ea0 msgr2=0x559512854030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f8272d9d640 1 --2- 172.21.15.28:0/1207063685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512849ea0 0x559512854030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f8272d9d640 1 -- 172.21.15.28:0/1207063685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512854570 msgr2=0x559512856960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f8272d9d640 1 --2- 172.21.15.28:0/1207063685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512854570 0x559512856960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f8272d9d640 1 -- 172.21.15.28:0/1207063685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55951277a400 con 0x559512849580 2024-04-25T01:38:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.596+0000 7f8271d9b640 1 --2- 172.21.15.28:0/1207063685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512854570 0x559512856960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.597+0000 7f8272d9d640 1 --2- 172.21.15.28:0/3522420472 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f825c0a2d00 0x7f825c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.597+0000 7f82637fe640 1 -- 172.21.15.28:0/3522420472 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f82680cf850 con 0x7f82741020d0 2024-04-25T01:38:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.597+0000 7f8272d9d640 1 --2- 172.21.15.28:0/1207063685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849580 0x559512849960 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f826c00a5d0 tx=0x7f826c00aa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4cdfdb64c62d2f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.597+0000 7f8260ff9640 1 -- 172.21.15.28:0/1207063685 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c01a040 con 0x559512849580 2024-04-25T01:38:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.597+0000 7f8260ff9640 1 -- 172.21.15.28:0/1207063685 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f826c021ba0 con 0x559512849580 2024-04-25T01:38:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.597+0000 7f8260ff9640 1 -- 172.21.15.28:0/1207063685 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c01f350 con 0x559512849580 2024-04-25T01:38:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.597+0000 7f8272d9d640 1 --2- 172.21.15.28:0/3522420472 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f825c0a2d00 0x7f825c0a51c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f826c020000 tx=0x7f826c021000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.598+0000 7f827b9f1740 1 -- 172.21.15.28:0/1207063685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849580 msgr2=0x559512849960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.598+0000 7f827b9f1740 1 --2- 172.21.15.28:0/1207063685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849580 0x559512849960 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f826c00a5d0 tx=0x7f826c00aa90 comp rx=0 tx=0).stop 2024-04-25T01:38:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.598+0000 7f827b9f1740 1 -- 172.21.15.28:0/1207063685 shutdown_connections 2024-04-25T01:38:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.598+0000 7f827b9f1740 1 --2- 172.21.15.28:0/1207063685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512854570 0x559512856960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.598+0000 7f827b9f1740 1 --2- 172.21.15.28:0/1207063685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512849ea0 0x559512854030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.598+0000 7f827b9f1740 1 --2- 172.21.15.28:0/1207063685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849580 0x559512849960 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.598+0000 7f827b9f1740 1 -- 172.21.15.28:0/1207063685 >> 172.21.15.28:0/1207063685 conn(0x5595128458a0 msgr2=0x559512845c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.598+0000 7f827b9f1740 1 -- 172.21.15.28:0/1207063685 shutdown_connections 2024-04-25T01:38:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.598+0000 7f827b9f1740 1 -- 172.21.15.28:0/1207063685 wait complete. 2024-04-25T01:38:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.598+0000 7f827b9f1740 1 -- start start 2024-04-25T01:38:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.598+0000 7f827b9f1740 1 Processor -- start 2024-04-25T01:38:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.598+0000 7f827b9f1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512849580 0x559512853ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.598+0000 7f827b9f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849ea0 0x559512851190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.598+0000 7f827b9f1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512854570 0x5595128516d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.598+0000 7f827b9f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559512857360 con 0x559512849ea0 2024-04-25T01:38:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.598+0000 7f827b9f1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55951276a050 con 0x559512849580 2024-04-25T01:38:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.598+0000 7f827b9f1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5595128574e0 con 0x559512854570 2024-04-25T01:38:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.598+0000 7f827b9f1740 10 client.0 _ll_register_callbacks cb 0x5595124c1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.598+0000 7f8271d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849ea0 0x559512851190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.599+0000 7f8271d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849ea0 0x559512851190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36912/0 (socket says 172.21.15.28:36912) 2024-04-25T01:38:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.599+0000 7f8271d9b640 1 -- 172.21.15.28:0/2368971800 learned_addr learned my addr 172.21.15.28:0/2368971800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.599+0000 7f8272d9d640 1 --2- 172.21.15.28:0/2368971800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512854570 0x5595128516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.599+0000 7f827259c640 1 --2- 172.21.15.28:0/2368971800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512849580 0x559512853ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.599+0000 7f8271d9b640 1 -- 172.21.15.28:0/2368971800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512854570 msgr2=0x5595128516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.599+0000 7f8271d9b640 1 --2- 172.21.15.28:0/2368971800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512854570 0x5595128516d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.599+0000 7f8271d9b640 1 -- 172.21.15.28:0/2368971800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512849580 msgr2=0x559512853ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.599+0000 7f8271d9b640 1 --2- 172.21.15.28:0/2368971800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512849580 0x559512853ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.599+0000 7f8271d9b640 1 -- 172.21.15.28:0/2368971800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55951273bdc0 con 0x559512849ea0 2024-04-25T01:38:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.599+0000 7f8271d9b640 1 --2- 172.21.15.28:0/2368971800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849ea0 0x559512851190 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f826809b6d0 tx=0x7f826809b510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31bc98a1629f7980 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.599+0000 7f8259ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.599+0000 7f8259ffb640 1 -- 172.21.15.28:0/2368971800 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826801d040 con 0x559512849ea0 2024-04-25T01:38:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.599+0000 7f8259ffb640 1 -- 172.21.15.28:0/2368971800 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f82680cce80 con 0x559512849ea0 2024-04-25T01:38:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.599+0000 7f827b9f1740 1 -- 172.21.15.28:0/2368971800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f826c021ba0 con 0x559512849ea0 2024-04-25T01:38:36.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.599+0000 7f8259ffb640 1 -- 172.21.15.28:0/2368971800 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f82680cd080 con 0x559512849ea0 2024-04-25T01:38:36.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.599+0000 7f827b9f1740 10 client.17352 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:36.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.599+0000 7f827b9f1740 1 -- 172.21.15.28:0/2368971800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5595124c23f0 con 0x559512849ea0 2024-04-25T01:38:36.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.599+0000 7f8259ffb640 1 -- 172.21.15.28:0/2368971800 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f826c021ba0 con 0x559512849ea0 2024-04-25T01:38:36.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.600+0000 7f8259ffb640 1 -- 172.21.15.28:0/2368971800 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f82680a4240 con 0x559512849ea0 2024-04-25T01:38:36.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.600+0000 7f827b9f1740 10 client.17352 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:36.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.600+0000 7f827b9f1740 10 client.17352 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:38:36.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.600+0000 7f827b9f1740 20 client.17352 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:36.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.600+0000 7f827b9f1740 1 --2- 172.21.15.28:0/2368971800 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5595128645a0 0x559512866990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.600+0000 7f827b9f1740 4 client.17352 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:36.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.600+0000 7f827b9f1740 1 -- 172.21.15.28:0/2368971800 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595127bbbb0 con 0x5595128645a0 2024-04-25T01:38:36.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.600+0000 7f827259c640 1 --2- 172.21.15.28:0/2368971800 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5595128645a0 0x559512866990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.600+0000 7f8259ffb640 10 client.17352 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:38:36.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.600+0000 7f827259c640 1 --2- 172.21.15.28:0/2368971800 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5595128645a0 0x559512866990 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a120d09e5e3fda23 server_cookie=afdfb9fd52db3d5f in_seq=0 out_seq=0 2024-04-25T01:38:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.601+0000 7f8259ffb640 1 -- 172.21.15.28:0/2368971800 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5595127bbbb0 con 0x5595128645a0 2024-04-25T01:38:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.601+0000 7f8259ffb640 10 client.17352 handle_command_reply: tid=0 2024-04-25T01:38:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.601+0000 7f827b9f1740 1 client.17352 shutdown 2024-04-25T01:38:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.601+0000 7f827b9f1740 1 -- 172.21.15.28:0/2368971800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849ea0 msgr2=0x559512851190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.601+0000 7f827b9f1740 1 --2- 172.21.15.28:0/2368971800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849ea0 0x559512851190 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f826809b6d0 tx=0x7f826809b510 comp rx=0 tx=0).stop 2024-04-25T01:38:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.601+0000 7f827b9f1740 1 -- 172.21.15.28:0/2368971800 shutdown_connections 2024-04-25T01:38:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.601+0000 7f827b9f1740 1 --2- 172.21.15.28:0/2368971800 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5595128645a0 0x559512866990 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.601+0000 7f827b9f1740 1 --2- 172.21.15.28:0/2368971800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512854570 0x5595128516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.601+0000 7f827b9f1740 1 --2- 172.21.15.28:0/2368971800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849ea0 0x559512851190 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.601+0000 7f8259ffb640 0 client.17352 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:38:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.601+0000 7f827b9f1740 1 --2- 172.21.15.28:0/2368971800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512849580 0x559512853ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.601+0000 7f827b9f1740 1 -- 172.21.15.28:0/2368971800 >> 172.21.15.28:0/2368971800 conn(0x5595128458a0 msgr2=0x559512844b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.601+0000 7f827b9f1740 1 -- 172.21.15.28:0/2368971800 shutdown_connections 2024-04-25T01:38:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.601+0000 7f827b9f1740 1 -- 172.21.15.28:0/2368971800 wait complete. 2024-04-25T01:38:36.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.601+0000 7f827b9f1740 20 client.17352 trim_cache size 0 max 16384 2024-04-25T01:38:36.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.615+0000 7f827b9f1740 1 Processor -- start 2024-04-25T01:38:36.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.615+0000 7f827b9f1740 1 -- start start 2024-04-25T01:38:36.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.615+0000 7f827b9f1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512849580 0x559512860220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.615+0000 7f827b9f1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512849ea0 0x559512850170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.615+0000 7f827b9f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512854570 0x5595128539d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.615+0000 7f827b9f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f82741aeb50 con 0x559512854570 2024-04-25T01:38:36.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.615+0000 7f827b9f1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f82741b0690 con 0x559512849ea0 2024-04-25T01:38:36.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.615+0000 7f827b9f1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f82741af220 con 0x559512849580 2024-04-25T01:38:36.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.615+0000 7f8272d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512854570 0x5595128539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.615+0000 7f8272d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512854570 0x5595128539d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36928/0 (socket says 172.21.15.28:36928) 2024-04-25T01:38:36.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.615+0000 7f8272d9d640 1 -- 172.21.15.28:0/862913265 learned_addr learned my addr 172.21.15.28:0/862913265 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.615+0000 7f8271d9b640 1 --2- 172.21.15.28:0/862913265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512849ea0 0x559512850170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.615+0000 7f827259c640 1 --2- 172.21.15.28:0/862913265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512849580 0x559512860220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.615+0000 7f8272d9d640 1 -- 172.21.15.28:0/862913265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512849580 msgr2=0x559512860220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.615+0000 7f8272d9d640 1 --2- 172.21.15.28:0/862913265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512849580 0x559512860220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.615+0000 7f8272d9d640 1 -- 172.21.15.28:0/862913265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512849ea0 msgr2=0x559512850170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.615+0000 7f8272d9d640 1 --2- 172.21.15.28:0/862913265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512849ea0 0x559512850170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.615+0000 7f8272d9d640 1 -- 172.21.15.28:0/862913265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55951258c7d0 con 0x559512854570 2024-04-25T01:38:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.615+0000 7f8271d9b640 1 --2- 172.21.15.28:0/862913265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512849ea0 0x559512850170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.616+0000 7f8272d9d640 1 --2- 172.21.15.28:0/862913265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512854570 0x5595128539d0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f826c00af90 tx=0x7f826c016f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b00b23a697aeca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.616+0000 7f825bfff640 1 -- 172.21.15.28:0/862913265 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c01a040 con 0x559512854570 2024-04-25T01:38:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.616+0000 7f825bfff640 1 -- 172.21.15.28:0/862913265 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f826c009970 con 0x559512854570 2024-04-25T01:38:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.616+0000 7f825bfff640 1 -- 172.21.15.28:0/862913265 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c01f6e0 con 0x559512854570 2024-04-25T01:38:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.616+0000 7f827b9f1740 1 -- 172.21.15.28:0/862913265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512854570 msgr2=0x5595128539d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.616+0000 7f827b9f1740 1 --2- 172.21.15.28:0/862913265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512854570 0x5595128539d0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f826c00af90 tx=0x7f826c016f90 comp rx=0 tx=0).stop 2024-04-25T01:38:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.616+0000 7f827b9f1740 1 -- 172.21.15.28:0/862913265 shutdown_connections 2024-04-25T01:38:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.616+0000 7f827b9f1740 1 --2- 172.21.15.28:0/862913265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512854570 0x5595128539d0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.616+0000 7f827b9f1740 1 --2- 172.21.15.28:0/862913265 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512849ea0 0x559512850170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.616+0000 7f827b9f1740 1 --2- 172.21.15.28:0/862913265 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512849580 0x559512860220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.616+0000 7f827b9f1740 1 -- 172.21.15.28:0/862913265 >> 172.21.15.28:0/862913265 conn(0x5595128458a0 msgr2=0x55951284fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.616+0000 7f827b9f1740 1 -- 172.21.15.28:0/862913265 shutdown_connections 2024-04-25T01:38:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.616+0000 7f827b9f1740 1 -- 172.21.15.28:0/862913265 wait complete. 2024-04-25T01:38:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.617+0000 7f827b9f1740 1 -- start start 2024-04-25T01:38:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.617+0000 7f827b9f1740 1 Processor -- start 2024-04-25T01:38:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.617+0000 7f827b9f1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512849580 0x55951285c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.617+0000 7f827b9f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849ea0 0x559512857c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.617+0000 7f827b9f1740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512854570 0x5595128581b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.617+0000 7f827b9f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f82741affc0 con 0x559512849ea0 2024-04-25T01:38:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.617+0000 7f827b9f1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f82741af8f0 con 0x559512849580 2024-04-25T01:38:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.617+0000 7f827b9f1740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f82741acb30 con 0x559512854570 2024-04-25T01:38:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.617+0000 7f827b9f1740 10 client.0 _ll_register_callbacks cb 0x7f82680058c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.617+0000 7f8271d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849ea0 0x559512857c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.617+0000 7f8271d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849ea0 0x559512857c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36938/0 (socket says 172.21.15.28:36938) 2024-04-25T01:38:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.617+0000 7f8271d9b640 1 -- 172.21.15.28:0/2413259818 learned_addr learned my addr 172.21.15.28:0/2413259818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.617+0000 7f827259c640 1 --2- 172.21.15.28:0/2413259818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512849580 0x55951285c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.617+0000 7f8272d9d640 1 --2- 172.21.15.28:0/2413259818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512854570 0x5595128581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.617+0000 7f8271d9b640 1 -- 172.21.15.28:0/2413259818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512854570 msgr2=0x5595128581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.617+0000 7f8271d9b640 1 --2- 172.21.15.28:0/2413259818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512854570 0x5595128581b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.617+0000 7f8271d9b640 1 -- 172.21.15.28:0/2413259818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512849580 msgr2=0x55951285c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.617+0000 7f8271d9b640 1 --2- 172.21.15.28:0/2413259818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512849580 0x55951285c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.617+0000 7f8271d9b640 1 -- 172.21.15.28:0/2413259818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595128586f0 con 0x559512849ea0 2024-04-25T01:38:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.618+0000 7f8271d9b640 1 --2- 172.21.15.28:0/2413259818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849ea0 0x559512857c70 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f82680a45b0 tx=0x7f82680cd130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a048b2ed88aab85 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.618+0000 7f8259ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.618+0000 7f8259ffb640 1 -- 172.21.15.28:0/2413259818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826809b330 con 0x559512849ea0 2024-04-25T01:38:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.618+0000 7f8259ffb640 1 -- 172.21.15.28:0/2413259818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f82680b4dd0 con 0x559512849ea0 2024-04-25T01:38:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.618+0000 7f827b9f1740 1 -- 172.21.15.28:0/2413259818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f826c009970 con 0x559512849ea0 2024-04-25T01:38:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.618+0000 7f8259ffb640 1 -- 172.21.15.28:0/2413259818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f82680a43c0 con 0x559512849ea0 2024-04-25T01:38:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.618+0000 7f827b9f1740 10 client.17358 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.618+0000 7f827b9f1740 1 -- 172.21.15.28:0/2413259818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5595128ca770 con 0x559512849ea0 2024-04-25T01:38:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.618+0000 7f8259ffb640 1 -- 172.21.15.28:0/2413259818 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f826c009970 con 0x559512849ea0 2024-04-25T01:38:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.618+0000 7f8259ffb640 1 -- 172.21.15.28:0/2413259818 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f826809ae20 con 0x559512849ea0 2024-04-25T01:38:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.618+0000 7f827b9f1740 10 client.17358 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.618+0000 7f827b9f1740 10 client.17358 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:38:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.618+0000 7f827b9f1740 20 client.17358 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.618+0000 7f827b9f1740 1 --2- 172.21.15.28:0/2413259818 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5595128cad20 0x5595128cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.618+0000 7f827b9f1740 4 client.17358 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:38:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.618+0000 7f827b9f1740 1 -- 172.21.15.28:0/2413259818 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5595127b5f00 con 0x5595128cad20 2024-04-25T01:38:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.618+0000 7f827259c640 1 --2- 172.21.15.28:0/2413259818 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5595128cad20 0x5595128cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.619+0000 7f827259c640 1 --2- 172.21.15.28:0/2413259818 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5595128cad20 0x5595128cd140 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4684422490151afb server_cookie=f194581abdfb12be in_seq=0 out_seq=0 2024-04-25T01:38:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.619+0000 7f8259ffb640 10 client.17358 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:38:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.619+0000 7f8259ffb640 1 -- 172.21.15.28:0/2413259818 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5595127b5f00 con 0x5595128cad20 2024-04-25T01:38:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.619+0000 7f8259ffb640 10 client.17358 handle_command_reply: tid=0 2024-04-25T01:38:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.619+0000 7f827b9f1740 1 client.17358 shutdown 2024-04-25T01:38:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.619+0000 7f827b9f1740 1 -- 172.21.15.28:0/2413259818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849ea0 msgr2=0x559512857c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.619+0000 7f827b9f1740 1 --2- 172.21.15.28:0/2413259818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849ea0 0x559512857c70 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f82680a45b0 tx=0x7f82680cd130 comp rx=0 tx=0).stop 2024-04-25T01:38:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.619+0000 7f827b9f1740 1 -- 172.21.15.28:0/2413259818 shutdown_connections 2024-04-25T01:38:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.619+0000 7f827b9f1740 1 --2- 172.21.15.28:0/2413259818 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5595128cad20 0x5595128cd140 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.619+0000 7f827b9f1740 1 --2- 172.21.15.28:0/2413259818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559512854570 0x5595128581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.619+0000 7f827b9f1740 1 --2- 172.21.15.28:0/2413259818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559512849ea0 0x559512857c70 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.619+0000 7f8259ffb640 0 client.17358 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:38:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.619+0000 7f827b9f1740 1 --2- 172.21.15.28:0/2413259818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559512849580 0x55951285c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.619+0000 7f827b9f1740 1 -- 172.21.15.28:0/2413259818 >> 172.21.15.28:0/2413259818 conn(0x5595128458a0 msgr2=0x5595128442b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.619+0000 7f827b9f1740 1 -- 172.21.15.28:0/2413259818 shutdown_connections 2024-04-25T01:38:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.619+0000 7f827b9f1740 1 -- 172.21.15.28:0/2413259818 wait complete. 2024-04-25T01:38:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.619+0000 7f827b9f1740 20 client.17358 trim_cache size 0 max 16384 2024-04-25T01:38:36.608 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:38:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.620+0000 7f8278c09640 1 -- 172.21.15.28:0/3522420472 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f825c0a2d00 msgr2=0x7f825c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.620+0000 7f8278c09640 1 --2- 172.21.15.28:0/3522420472 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f825c0a2d00 0x7f825c0a51c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f826c020000 tx=0x7f826c021000 comp rx=0 tx=0).stop 2024-04-25T01:38:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.620+0000 7f8278c09640 1 -- 172.21.15.28:0/3522420472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82741020d0 msgr2=0x7f8274198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.620+0000 7f8278c09640 1 --2- 172.21.15.28:0/3522420472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82741020d0 0x7f8274198650 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f826800c890 tx=0x7f826800cd50 comp rx=0 tx=0).stop 2024-04-25T01:38:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.620+0000 7f8278c09640 1 -- 172.21.15.28:0/3522420472 shutdown_connections 2024-04-25T01:38:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.620+0000 7f8278c09640 1 --2- 172.21.15.28:0/3522420472 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f825c0a2d00 0x7f825c0a51c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.620+0000 7f8278c09640 1 --2- 172.21.15.28:0/3522420472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8274107720 0x7f827419c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.620+0000 7f8278c09640 1 --2- 172.21.15.28:0/3522420472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82741020d0 0x7f8274198650 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.620+0000 7f8278c09640 1 --2- 172.21.15.28:0/3522420472 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8274101710 0x7f8274198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.620+0000 7f8278c09640 1 -- 172.21.15.28:0/3522420472 >> 172.21.15.28:0/3522420472 conn(0x7f82740fd180 msgr2=0x7f82740fd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.620+0000 7f8278c09640 1 -- 172.21.15.28:0/3522420472 shutdown_connections 2024-04-25T01:38:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.620+0000 7f8278c09640 1 -- 172.21.15.28:0/3522420472 wait complete. 2024-04-25T01:38:36.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:38:36.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.735+0000 7f9725438640 1 -- 172.21.15.28:0/1134759438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9720074490 msgr2=0x7f972010da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.735+0000 7f9725438640 1 --2- 172.21.15.28:0/1134759438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9720074490 0x7f972010da10 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f9714009900 tx=0x7f971402a480 comp rx=0 tx=0).stop 2024-04-25T01:38:36.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.735+0000 7f9725438640 1 -- 172.21.15.28:0/1134759438 shutdown_connections 2024-04-25T01:38:36.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.735+0000 7f9725438640 1 --2- 172.21.15.28:0/1134759438 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f972010df50 0x7f97201103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.735+0000 7f9725438640 1 --2- 172.21.15.28:0/1134759438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9720074490 0x7f972010da10 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.735+0000 7f9725438640 1 --2- 172.21.15.28:0/1134759438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9720073b70 0x7f9720073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.735+0000 7f9725438640 1 -- 172.21.15.28:0/1134759438 >> 172.21.15.28:0/1134759438 conn(0x7f97200fd180 msgr2=0x7f97200ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.735+0000 7f9725438640 1 -- 172.21.15.28:0/1134759438 shutdown_connections 2024-04-25T01:38:36.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.735+0000 7f9725438640 1 -- 172.21.15.28:0/1134759438 wait complete. 2024-04-25T01:38:36.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.735+0000 7f9725438640 1 Processor -- start 2024-04-25T01:38:36.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.736+0000 7f9725438640 1 -- start start 2024-04-25T01:38:36.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.736+0000 7f9725438640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9720073b70 0x7f972019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.736+0000 7f9725438640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9720074490 0x7f972019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.736+0000 7f9725438640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f972010df50 0x7f972019ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.736+0000 7f9725438640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9720113cf0 con 0x7f9720074490 2024-04-25T01:38:36.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.736+0000 7f9725438640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9720113b70 con 0x7f972010df50 2024-04-25T01:38:36.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.736+0000 7f9725438640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9720113e70 con 0x7f9720073b70 2024-04-25T01:38:36.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.736+0000 7f971e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9720074490 0x7f972019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.736+0000 7f971e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9720074490 0x7f972019a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45370/0 (socket says 172.21.15.28:45370) 2024-04-25T01:38:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.736+0000 7f971e7fc640 1 -- 172.21.15.28:0/446229097 learned_addr learned my addr 172.21.15.28:0/446229097 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.736+0000 7f971f7fe640 1 --2- 172.21.15.28:0/446229097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f972010df50 0x7f972019ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.736+0000 7f971effd640 1 --2- 172.21.15.28:0/446229097 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9720073b70 0x7f972019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.736+0000 7f971e7fc640 1 -- 172.21.15.28:0/446229097 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9720073b70 msgr2=0x7f972019a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.736+0000 7f971e7fc640 1 --2- 172.21.15.28:0/446229097 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9720073b70 0x7f972019a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.736+0000 7f971e7fc640 1 -- 172.21.15.28:0/446229097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f972010df50 msgr2=0x7f972019ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.736+0000 7f971e7fc640 1 --2- 172.21.15.28:0/446229097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f972010df50 0x7f972019ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.736+0000 7f971e7fc640 1 -- 172.21.15.28:0/446229097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f972019f1b0 con 0x7f9720074490 2024-04-25T01:38:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.736+0000 7f971e7fc640 1 --2- 172.21.15.28:0/446229097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9720074490 0x7f972019a750 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f9714006010 tx=0x7f97140042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.737+0000 7f9707fff640 1 -- 172.21.15.28:0/446229097 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971403f030 con 0x7f9720074490 2024-04-25T01:38:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.737+0000 7f9707fff640 1 -- 172.21.15.28:0/446229097 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9714003ce0 con 0x7f9720074490 2024-04-25T01:38:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.737+0000 7f9725438640 1 -- 172.21.15.28:0/446229097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f972019f440 con 0x7f9720074490 2024-04-25T01:38:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.737+0000 7f9707fff640 1 -- 172.21.15.28:0/446229097 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9714035070 con 0x7f9720074490 2024-04-25T01:38:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.737+0000 7f9725438640 1 -- 172.21.15.28:0/446229097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97201a6d30 con 0x7f9720074490 2024-04-25T01:38:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.739+0000 7f9707fff640 1 -- 172.21.15.28:0/446229097 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9714042030 con 0x7f9720074490 2024-04-25T01:38:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.739+0000 7f9728222740 1 Processor -- start 2024-04-25T01:38:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.739+0000 7f9728222740 1 -- start start 2024-04-25T01:38:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.739+0000 7f9728222740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08a580 0x564f2d08a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.739+0000 7f9728222740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d08aea0 0x564f2d095030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.739+0000 7f9728222740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d095570 0x564f2d097960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.739+0000 7f9728222740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564f2cfef670 con 0x564f2d08aea0 2024-04-25T01:38:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.739+0000 7f9728222740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564f2cf97970 con 0x564f2d08a580 2024-04-25T01:38:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.739+0000 7f9728222740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x564f2cfeb630 con 0x564f2d095570 2024-04-25T01:38:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.740+0000 7f971effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d08aea0 0x564f2d095030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.740+0000 7f971effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d08aea0 0x564f2d095030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45380/0 (socket says 172.21.15.28:45380) 2024-04-25T01:38:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.740+0000 7f971effd640 1 -- 172.21.15.28:0/2876106843 learned_addr learned my addr 172.21.15.28:0/2876106843 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.740+0000 7f971e7fc640 1 --2- 172.21.15.28:0/2876106843 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d095570 0x564f2d097960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.740+0000 7f971f7fe640 1 --2- 172.21.15.28:0/2876106843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08a580 0x564f2d08a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.740+0000 7f971effd640 1 -- 172.21.15.28:0/2876106843 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d095570 msgr2=0x564f2d097960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.740+0000 7f971effd640 1 --2- 172.21.15.28:0/2876106843 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d095570 0x564f2d097960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.740+0000 7f971effd640 1 -- 172.21.15.28:0/2876106843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08a580 msgr2=0x564f2d08a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.740+0000 7f971effd640 1 --2- 172.21.15.28:0/2876106843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08a580 0x564f2d08a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.740+0000 7f9707fff640 1 --2- 172.21.15.28:0/446229097 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97080a2d00 0x7f97080a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.740+0000 7f971effd640 1 -- 172.21.15.28:0/2876106843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f2cfbb400 con 0x564f2d08aea0 2024-04-25T01:38:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.740+0000 7f971f7fe640 1 --2- 172.21.15.28:0/2876106843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08a580 0x564f2d08a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.740+0000 7f971e7fc640 1 --2- 172.21.15.28:0/446229097 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97080a2d00 0x7f97080a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.740+0000 7f971effd640 1 --2- 172.21.15.28:0/2876106843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d08aea0 0x564f2d095030 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f971000bcb0 tx=0x7f971000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c06ddbedd91492 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.740+0000 7f9707fff640 1 -- 172.21.15.28:0/446229097 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f97140efca0 con 0x7f9720074490 2024-04-25T01:38:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.740+0000 7f97057fa640 1 -- 172.21.15.28:0/2876106843 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9710015070 con 0x564f2d08aea0 2024-04-25T01:38:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.740+0000 7f97057fa640 1 -- 172.21.15.28:0/2876106843 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f971001a940 con 0x564f2d08aea0 2024-04-25T01:38:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.740+0000 7f97057fa640 1 -- 172.21.15.28:0/2876106843 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9710011aa0 con 0x564f2d08aea0 2024-04-25T01:38:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.741+0000 7f9728222740 1 -- 172.21.15.28:0/2876106843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d08aea0 msgr2=0x564f2d095030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.741+0000 7f9728222740 1 --2- 172.21.15.28:0/2876106843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d08aea0 0x564f2d095030 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f971000bcb0 tx=0x7f971000f730 comp rx=0 tx=0).stop 2024-04-25T01:38:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.741+0000 7f971e7fc640 1 --2- 172.21.15.28:0/446229097 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97080a2d00 0x7f97080a51c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f97140ed160 tx=0x7f97140f37f0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.741+0000 7f9728222740 1 -- 172.21.15.28:0/2876106843 shutdown_connections 2024-04-25T01:38:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.741+0000 7f9728222740 1 --2- 172.21.15.28:0/2876106843 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d095570 0x564f2d097960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.741+0000 7f9728222740 1 --2- 172.21.15.28:0/2876106843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d08aea0 0x564f2d095030 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.741+0000 7f9728222740 1 --2- 172.21.15.28:0/2876106843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08a580 0x564f2d08a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.741+0000 7f9728222740 1 -- 172.21.15.28:0/2876106843 >> 172.21.15.28:0/2876106843 conn(0x564f2d0868a0 msgr2=0x564f2d086c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.741+0000 7f9728222740 1 -- 172.21.15.28:0/2876106843 shutdown_connections 2024-04-25T01:38:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.741+0000 7f9728222740 1 -- 172.21.15.28:0/2876106843 wait complete. 2024-04-25T01:38:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.741+0000 7f9728222740 1 -- start start 2024-04-25T01:38:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f9728222740 1 Processor -- start 2024-04-25T01:38:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f9728222740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d08a580 0x564f2d094f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f9728222740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08aea0 0x564f2d092200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f9728222740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d095570 0x564f2d0927a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f9728222740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564f2d098360 con 0x564f2d095570 2024-04-25T01:38:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f9728222740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564f2cfab050 con 0x564f2d08aea0 2024-04-25T01:38:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f9728222740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x564f2d0984e0 con 0x564f2d08a580 2024-04-25T01:38:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f9728222740 10 client.0 _ll_register_callbacks cb 0x564f2cd02be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f971f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d095570 0x564f2d0927a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f971f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d095570 0x564f2d0927a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45386/0 (socket says 172.21.15.28:45386) 2024-04-25T01:38:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f971f7fe640 1 -- 172.21.15.28:0/1037487127 learned_addr learned my addr 172.21.15.28:0/1037487127 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f971effd640 1 --2- 172.21.15.28:0/1037487127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08aea0 0x564f2d092200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f971f7fe640 1 --2- 172.21.15.28:0/1037487127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d08a580 0x564f2d094f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f971f7fe640 1 -- 172.21.15.28:0/1037487127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d08a580 msgr2=0x564f2d094f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f971f7fe640 1 --2- 172.21.15.28:0/1037487127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d08a580 0x564f2d094f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f971f7fe640 1 -- 172.21.15.28:0/1037487127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08aea0 msgr2=0x564f2d092200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f971f7fe640 1 --2- 172.21.15.28:0/1037487127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08aea0 0x564f2d092200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f971f7fe640 1 -- 172.21.15.28:0/1037487127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f2cf7cdc0 con 0x564f2d095570 2024-04-25T01:38:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f971f7fe640 1 --2- 172.21.15.28:0/1037487127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d095570 0x564f2d0927a0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f97180077f0 tx=0x7f9718007a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=626e469ce2042bf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.742+0000 7f9702ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.743+0000 7f9702ff5640 1 -- 172.21.15.28:0/1037487127 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97180110d0 con 0x564f2d095570 2024-04-25T01:38:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.743+0000 7f9702ff5640 1 -- 172.21.15.28:0/1037487127 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f9718004b20 con 0x564f2d095570 2024-04-25T01:38:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.743+0000 7f9728222740 1 -- 172.21.15.28:0/1037487127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f971001a940 con 0x564f2d095570 2024-04-25T01:38:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.743+0000 7f9702ff5640 1 -- 172.21.15.28:0/1037487127 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9718024050 con 0x564f2d095570 2024-04-25T01:38:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.743+0000 7f9728222740 10 client.17370 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.743+0000 7f9728222740 1 -- 172.21.15.28:0/1037487127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564f2cd033f0 con 0x564f2d095570 2024-04-25T01:38:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.743+0000 7f9702ff5640 1 -- 172.21.15.28:0/1037487127 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f971001a940 con 0x564f2d095570 2024-04-25T01:38:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.743+0000 7f9702ff5640 1 -- 172.21.15.28:0/1037487127 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f971800f420 con 0x564f2d095570 2024-04-25T01:38:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.743+0000 7f9728222740 10 client.17370 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.743+0000 7f9728222740 10 client.17370 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:38:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.743+0000 7f9728222740 20 client.17370 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.743+0000 7f9728222740 1 --2- 172.21.15.28:0/1037487127 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x564f2d0a5660 0x564f2d0a7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.743+0000 7f9728222740 4 client.17370 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.743+0000 7f9728222740 1 -- 172.21.15.28:0/1037487127 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f2cffcbb0 con 0x564f2d0a5660 2024-04-25T01:38:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.744+0000 7f971effd640 1 --2- 172.21.15.28:0/1037487127 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x564f2d0a5660 0x564f2d0a7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.744+0000 7f971effd640 1 --2- 172.21.15.28:0/1037487127 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x564f2d0a5660 0x564f2d0a7a90 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3266be61a51fa3ae server_cookie=1a2e43bebe0aaeb8 in_seq=0 out_seq=0 2024-04-25T01:38:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.744+0000 7f9702ff5640 10 client.17370 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:38:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.745+0000 7f9702ff5640 1 -- 172.21.15.28:0/1037487127 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564f2cffcbb0 con 0x564f2d0a5660 2024-04-25T01:38:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.745+0000 7f9702ff5640 10 client.17370 handle_command_reply: tid=0 2024-04-25T01:38:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.745+0000 7f9728222740 1 client.17370 shutdown 2024-04-25T01:38:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.745+0000 7f9728222740 1 -- 172.21.15.28:0/1037487127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d095570 msgr2=0x564f2d0927a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.745+0000 7f9728222740 1 --2- 172.21.15.28:0/1037487127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d095570 0x564f2d0927a0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f97180077f0 tx=0x7f9718007a70 comp rx=0 tx=0).stop 2024-04-25T01:38:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.745+0000 7f9728222740 1 -- 172.21.15.28:0/1037487127 shutdown_connections 2024-04-25T01:38:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.745+0000 7f9728222740 1 --2- 172.21.15.28:0/1037487127 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x564f2d0a5660 0x564f2d0a7a90 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.745+0000 7f9728222740 1 --2- 172.21.15.28:0/1037487127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d095570 0x564f2d0927a0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.745+0000 7f9728222740 1 --2- 172.21.15.28:0/1037487127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08aea0 0x564f2d092200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.745+0000 7f9728222740 1 --2- 172.21.15.28:0/1037487127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d08a580 0x564f2d094f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.745+0000 7f9702ff5640 0 client.17370 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:38:36.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.745+0000 7f9728222740 1 -- 172.21.15.28:0/1037487127 >> 172.21.15.28:0/1037487127 conn(0x564f2d0868a0 msgr2=0x564f2d085b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.745+0000 7f9728222740 1 -- 172.21.15.28:0/1037487127 shutdown_connections 2024-04-25T01:38:36.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.745+0000 7f9728222740 1 -- 172.21.15.28:0/1037487127 wait complete. 2024-04-25T01:38:36.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.745+0000 7f9728222740 20 client.17370 trim_cache size 0 max 16384 2024-04-25T01:38:36.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.758+0000 7f9728222740 1 Processor -- start 2024-04-25T01:38:36.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.758+0000 7f9728222740 1 -- start start 2024-04-25T01:38:36.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.758+0000 7f9728222740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d08a580 0x564f2d0a1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.758+0000 7f9728222740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08aea0 0x564f2d08c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.758+0000 7f9728222740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d095570 0x564f2d094a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.758+0000 7f9728222740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97201b0c00 con 0x564f2d095570 2024-04-25T01:38:36.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.758+0000 7f9728222740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f97201b2740 con 0x564f2d08aea0 2024-04-25T01:38:36.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.758+0000 7f9728222740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f97201b12d0 con 0x564f2d08a580 2024-04-25T01:38:36.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.758+0000 7f971f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d095570 0x564f2d094a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.758+0000 7f971f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d095570 0x564f2d094a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45392/0 (socket says 172.21.15.28:45392) 2024-04-25T01:38:36.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.759+0000 7f971f7fe640 1 -- 172.21.15.28:0/4089188219 learned_addr learned my addr 172.21.15.28:0/4089188219 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.759+0000 7f971f7fe640 1 --2- 172.21.15.28:0/4089188219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d08a580 0x564f2d0a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.759+0000 7f971effd640 1 --2- 172.21.15.28:0/4089188219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08aea0 0x564f2d08c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.759+0000 7f971f7fe640 1 -- 172.21.15.28:0/4089188219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d08a580 msgr2=0x564f2d0a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.759+0000 7f971f7fe640 1 --2- 172.21.15.28:0/4089188219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d08a580 0x564f2d0a1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.759+0000 7f971f7fe640 1 -- 172.21.15.28:0/4089188219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08aea0 msgr2=0x564f2d08c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.759+0000 7f971f7fe640 1 --2- 172.21.15.28:0/4089188219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08aea0 0x564f2d08c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.759+0000 7f971f7fe640 1 -- 172.21.15.28:0/4089188219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f2cdcd7d0 con 0x564f2d095570 2024-04-25T01:38:36.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.759+0000 7f971f7fe640 1 --2- 172.21.15.28:0/4089188219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d095570 0x564f2d094a40 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f9718007fd0 tx=0x7f9718000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f91f71271a096f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.759+0000 7f9704ff9640 1 -- 172.21.15.28:0/4089188219 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971800ecd0 con 0x564f2d095570 2024-04-25T01:38:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.759+0000 7f9704ff9640 1 -- 172.21.15.28:0/4089188219 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f971800f420 con 0x564f2d095570 2024-04-25T01:38:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.759+0000 7f9704ff9640 1 -- 172.21.15.28:0/4089188219 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9718004020 con 0x564f2d095570 2024-04-25T01:38:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f9728222740 1 -- 172.21.15.28:0/4089188219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d095570 msgr2=0x564f2d094a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f9728222740 1 --2- 172.21.15.28:0/4089188219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d095570 0x564f2d094a40 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f9718007fd0 tx=0x7f9718000f40 comp rx=0 tx=0).stop 2024-04-25T01:38:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f9728222740 1 -- 172.21.15.28:0/4089188219 shutdown_connections 2024-04-25T01:38:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f9728222740 1 --2- 172.21.15.28:0/4089188219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d095570 0x564f2d094a40 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f9728222740 1 --2- 172.21.15.28:0/4089188219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08aea0 0x564f2d08c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f9728222740 1 --2- 172.21.15.28:0/4089188219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d08a580 0x564f2d0a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f9728222740 1 -- 172.21.15.28:0/4089188219 >> 172.21.15.28:0/4089188219 conn(0x564f2d0868a0 msgr2=0x564f2d08bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f9728222740 1 -- 172.21.15.28:0/4089188219 shutdown_connections 2024-04-25T01:38:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f9728222740 1 -- 172.21.15.28:0/4089188219 wait complete. 2024-04-25T01:38:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f9728222740 1 -- start start 2024-04-25T01:38:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f9728222740 1 Processor -- start 2024-04-25T01:38:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f9728222740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08a580 0x564f2d09dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f9728222740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d08aea0 0x564f2d098ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f9728222740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d095570 0x564f2d099600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f9728222740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97201b2070 con 0x564f2d08aea0 2024-04-25T01:38:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f9728222740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f97201b19a0 con 0x564f2d08a580 2024-04-25T01:38:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f9728222740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f97201aebe0 con 0x564f2d095570 2024-04-25T01:38:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f9728222740 10 client.0 _ll_register_callbacks cb 0x7f97180079c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f971effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d08aea0 0x564f2d098ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f971effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d08aea0 0x564f2d098ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45394/0 (socket says 172.21.15.28:45394) 2024-04-25T01:38:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f971effd640 1 -- 172.21.15.28:0/3626641431 learned_addr learned my addr 172.21.15.28:0/3626641431 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.760+0000 7f971f7fe640 1 --2- 172.21.15.28:0/3626641431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08a580 0x564f2d09dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f971f7fe640 1 --2- 172.21.15.28:0/3626641431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d095570 0x564f2d099600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f971effd640 1 -- 172.21.15.28:0/3626641431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d095570 msgr2=0x564f2d099600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f971effd640 1 --2- 172.21.15.28:0/3626641431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d095570 0x564f2d099600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f971effd640 1 -- 172.21.15.28:0/3626641431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08a580 msgr2=0x564f2d09dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f971effd640 1 --2- 172.21.15.28:0/3626641431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08a580 0x564f2d09dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f971effd640 1 -- 172.21.15.28:0/3626641431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f2d099e30 con 0x564f2d08aea0 2024-04-25T01:38:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f971f7fe640 1 --2- 172.21.15.28:0/3626641431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08a580 0x564f2d09dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f971effd640 1 --2- 172.21.15.28:0/3626641431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d08aea0 0x564f2d098ff0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f971000fc30 tx=0x7f9710005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f613e265501628 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f9702ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f9702ff5640 1 -- 172.21.15.28:0/3626641431 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9710015040 con 0x564f2d08aea0 2024-04-25T01:38:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f9728222740 1 -- 172.21.15.28:0/3626641431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f971800f420 con 0x564f2d08aea0 2024-04-25T01:38:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f9702ff5640 1 -- 172.21.15.28:0/3626641431 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f9710004240 con 0x564f2d08aea0 2024-04-25T01:38:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f9702ff5640 1 -- 172.21.15.28:0/3626641431 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9710035050 con 0x564f2d08aea0 2024-04-25T01:38:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f9728222740 10 client.17376 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f9728222740 1 -- 172.21.15.28:0/3626641431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564f2d10bcd0 con 0x564f2d08aea0 2024-04-25T01:38:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f9702ff5640 1 -- 172.21.15.28:0/3626641431 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f971800f420 con 0x564f2d08aea0 2024-04-25T01:38:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f9702ff5640 1 -- 172.21.15.28:0/3626641431 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9710030030 con 0x564f2d08aea0 2024-04-25T01:38:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f9728222740 10 client.17376 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f9728222740 10 client.17376 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:38:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f9728222740 20 client.17376 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f9728222740 1 --2- 172.21.15.28:0/3626641431 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x564f2d10c340 0x564f2d10e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f9728222740 4 client.17376 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:38:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.761+0000 7f9728222740 1 -- 172.21.15.28:0/3626641431 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564f2cff6f00 con 0x564f2d10c340 2024-04-25T01:38:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.762+0000 7f971f7fe640 1 --2- 172.21.15.28:0/3626641431 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x564f2d10c340 0x564f2d10e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.762+0000 7f9702ff5640 10 client.17376 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:38:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.762+0000 7f971f7fe640 1 --2- 172.21.15.28:0/3626641431 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x564f2d10c340 0x564f2d10e760 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b79dc5bccfcda92b server_cookie=1868afb4a02e0c5 in_seq=0 out_seq=0 2024-04-25T01:38:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.762+0000 7f9702ff5640 1 -- 172.21.15.28:0/3626641431 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564f2cff6f00 con 0x564f2d10c340 2024-04-25T01:38:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.762+0000 7f9702ff5640 10 client.17376 handle_command_reply: tid=0 2024-04-25T01:38:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.762+0000 7f9728222740 1 client.17376 shutdown 2024-04-25T01:38:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.762+0000 7f9728222740 1 -- 172.21.15.28:0/3626641431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d08aea0 msgr2=0x564f2d098ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.762+0000 7f9728222740 1 --2- 172.21.15.28:0/3626641431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d08aea0 0x564f2d098ff0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f971000fc30 tx=0x7f9710005c70 comp rx=0 tx=0).stop 2024-04-25T01:38:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9728222740 1 -- 172.21.15.28:0/3626641431 shutdown_connections 2024-04-25T01:38:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9728222740 1 --2- 172.21.15.28:0/3626641431 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x564f2d10c340 0x564f2d10e760 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9728222740 1 --2- 172.21.15.28:0/3626641431 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564f2d095570 0x564f2d099600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9728222740 1 --2- 172.21.15.28:0/3626641431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564f2d08aea0 0x564f2d098ff0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9728222740 1 --2- 172.21.15.28:0/3626641431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564f2d08a580 0x564f2d09dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9702ff5640 0 client.17376 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:38:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9728222740 1 -- 172.21.15.28:0/3626641431 >> 172.21.15.28:0/3626641431 conn(0x564f2d0868a0 msgr2=0x564f2d085b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9728222740 1 -- 172.21.15.28:0/3626641431 shutdown_connections 2024-04-25T01:38:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9728222740 1 -- 172.21.15.28:0/3626641431 wait complete. 2024-04-25T01:38:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9728222740 20 client.17376 trim_cache size 0 max 16384 2024-04-25T01:38:36.752 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:38:36.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9725438640 1 -- 172.21.15.28:0/446229097 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97080a2d00 msgr2=0x7f97080a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9725438640 1 --2- 172.21.15.28:0/446229097 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97080a2d00 0x7f97080a51c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f97140ed160 tx=0x7f97140f37f0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9725438640 1 -- 172.21.15.28:0/446229097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9720074490 msgr2=0x7f972019a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9725438640 1 --2- 172.21.15.28:0/446229097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9720074490 0x7f972019a750 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f9714006010 tx=0x7f97140042f0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9725438640 1 -- 172.21.15.28:0/446229097 shutdown_connections 2024-04-25T01:38:36.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9725438640 1 --2- 172.21.15.28:0/446229097 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97080a2d00 0x7f97080a51c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9725438640 1 --2- 172.21.15.28:0/446229097 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f972010df50 0x7f972019ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9725438640 1 --2- 172.21.15.28:0/446229097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9720074490 0x7f972019a750 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9725438640 1 --2- 172.21.15.28:0/446229097 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9720073b70 0x7f972019a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9725438640 1 -- 172.21.15.28:0/446229097 >> 172.21.15.28:0/446229097 conn(0x7f97200fd180 msgr2=0x7f97200fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9725438640 1 -- 172.21.15.28:0/446229097 shutdown_connections 2024-04-25T01:38:36.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.763+0000 7f9725438640 1 -- 172.21.15.28:0/446229097 wait complete. 2024-04-25T01:38:36.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:38:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.879+0000 7fade32da640 1 -- 172.21.15.28:0/610377144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faddc100e60 msgr2=0x7faddc10dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.879+0000 7fade32da640 1 --2- 172.21.15.28:0/610377144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faddc100e60 0x7faddc10dbb0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009900 tx=0x7fadcc02a410 comp rx=0 tx=0).stop 2024-04-25T01:38:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.879+0000 7fade32da640 1 -- 172.21.15.28:0/610377144 shutdown_connections 2024-04-25T01:38:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.879+0000 7fade32da640 1 --2- 172.21.15.28:0/610377144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faddc10e0f0 0x7faddc1104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.879+0000 7fade32da640 1 --2- 172.21.15.28:0/610377144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faddc100e60 0x7faddc10dbb0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.879+0000 7fade32da640 1 --2- 172.21.15.28:0/610377144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faddc100540 0x7faddc100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.879+0000 7fade32da640 1 -- 172.21.15.28:0/610377144 >> 172.21.15.28:0/610377144 conn(0x7faddc077740 msgr2=0x7faddc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.879+0000 7fade32da640 1 -- 172.21.15.28:0/610377144 shutdown_connections 2024-04-25T01:38:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.879+0000 7fade32da640 1 -- 172.21.15.28:0/610377144 wait complete. 2024-04-25T01:38:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.879+0000 7fade32da640 1 Processor -- start 2024-04-25T01:38:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.879+0000 7fade32da640 1 -- start start 2024-04-25T01:38:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.880+0000 7fade32da640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faddc100540 0x7faddc19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.880+0000 7fade32da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faddc100e60 0x7faddc19a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.880+0000 7fade32da640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faddc10e0f0 0x7faddc19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.880+0000 7fade32da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faddc113d20 con 0x7faddc100e60 2024-04-25T01:38:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.880+0000 7fade32da640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faddc113ba0 con 0x7faddc100540 2024-04-25T01:38:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.880+0000 7fade32da640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7faddc113ea0 con 0x7faddc10e0f0 2024-04-25T01:38:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.880+0000 7fade0875640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faddc100e60 0x7faddc19a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.880+0000 7fade0875640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faddc100e60 0x7faddc19a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45416/0 (socket says 172.21.15.28:45416) 2024-04-25T01:38:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.880+0000 7fade0875640 1 -- 172.21.15.28:0/88891968 learned_addr learned my addr 172.21.15.28:0/88891968 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.880+0000 7fade1877640 1 --2- 172.21.15.28:0/88891968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faddc10e0f0 0x7faddc19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.880+0000 7fade1076640 1 --2- 172.21.15.28:0/88891968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faddc100540 0x7faddc19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.880+0000 7fade0875640 1 -- 172.21.15.28:0/88891968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faddc10e0f0 msgr2=0x7faddc19eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.880+0000 7fade0875640 1 --2- 172.21.15.28:0/88891968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faddc10e0f0 0x7faddc19eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.880+0000 7fade0875640 1 -- 172.21.15.28:0/88891968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faddc100540 msgr2=0x7faddc19a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.880+0000 7fade0875640 1 --2- 172.21.15.28:0/88891968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faddc100540 0x7faddc19a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.880+0000 7fade0875640 1 -- 172.21.15.28:0/88891968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faddc19f2d0 con 0x7faddc100e60 2024-04-25T01:38:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.880+0000 7fade1877640 1 --2- 172.21.15.28:0/88891968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faddc10e0f0 0x7faddc19eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:38:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.880+0000 7fade0875640 1 --2- 172.21.15.28:0/88891968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faddc100e60 0x7faddc19a820 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fadcc006010 tx=0x7fadcc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.880+0000 7fadd27fc640 1 -- 172.21.15.28:0/88891968 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc03f030 con 0x7faddc100e60 2024-04-25T01:38:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.881+0000 7fadd27fc640 1 -- 172.21.15.28:0/88891968 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadcc003ad0 con 0x7faddc100e60 2024-04-25T01:38:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.881+0000 7fade32da640 1 -- 172.21.15.28:0/88891968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faddc19f560 con 0x7faddc100e60 2024-04-25T01:38:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.881+0000 7fadd27fc640 1 -- 172.21.15.28:0/88891968 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc035070 con 0x7faddc100e60 2024-04-25T01:38:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.881+0000 7fade32da640 1 -- 172.21.15.28:0/88891968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faddc1a6e50 con 0x7faddc100e60 2024-04-25T01:38:36.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.883+0000 7fadd27fc640 1 -- 172.21.15.28:0/88891968 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fadcc042030 con 0x7faddc100e60 2024-04-25T01:38:36.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.883+0000 7fade5fef740 1 Processor -- start 2024-04-25T01:38:36.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.883+0000 7fade5fef740 1 -- start start 2024-04-25T01:38:36.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.883+0000 7fade5fef740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd31580 0x556c6fd31960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.883+0000 7fade5fef740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd31ea0 0x556c6fd3c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.883+0000 7fade5fef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd3c570 0x556c6fd3e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.883+0000 7fade5fef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556c6fc96670 con 0x556c6fd3c570 2024-04-25T01:38:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.883+0000 7fade5fef740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556c6fc3e970 con 0x556c6fd31580 2024-04-25T01:38:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.883+0000 7fade5fef740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x556c6fc92630 con 0x556c6fd31ea0 2024-04-25T01:38:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.884+0000 7fade0875640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd3c570 0x556c6fd3e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.884+0000 7fade0875640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd3c570 0x556c6fd3e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45422/0 (socket says 172.21.15.28:45422) 2024-04-25T01:38:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.884+0000 7fade0875640 1 -- 172.21.15.28:0/2540053769 learned_addr learned my addr 172.21.15.28:0/2540053769 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.884+0000 7fade1076640 1 --2- 172.21.15.28:0/2540053769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd31ea0 0x556c6fd3c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.884+0000 7fade1877640 1 --2- 172.21.15.28:0/2540053769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd31580 0x556c6fd31960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.884+0000 7fadd27fc640 1 --2- 172.21.15.28:0/88891968 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fadc80a2d80 0x7fadc80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.884+0000 7fade1877640 1 --2- 172.21.15.28:0/88891968 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fadc80a2d80 0x7fadc80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.884+0000 7fade0875640 1 -- 172.21.15.28:0/2540053769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd31ea0 msgr2=0x556c6fd3c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.884+0000 7fade0875640 1 --2- 172.21.15.28:0/2540053769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd31ea0 0x556c6fd3c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.884+0000 7fade0875640 1 -- 172.21.15.28:0/2540053769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd31580 msgr2=0x556c6fd31960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.884+0000 7fade0875640 1 --2- 172.21.15.28:0/2540053769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd31580 0x556c6fd31960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.884+0000 7fade0875640 1 -- 172.21.15.28:0/2540053769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c6fc62400 con 0x556c6fd3c570 2024-04-25T01:38:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.884+0000 7fade1076640 1 --2- 172.21.15.28:0/2540053769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd31ea0 0x556c6fd3c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:38:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.884+0000 7fadd27fc640 1 -- 172.21.15.28:0/88891968 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fadcc0efd70 con 0x7faddc100e60 2024-04-25T01:38:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.884+0000 7fade0875640 1 --2- 172.21.15.28:0/2540053769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd3c570 0x556c6fd3e960 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fadcc0ed0f0 tx=0x7fadcc03b640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa71e30b2c1e56db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.884+0000 7fade1877640 1 --2- 172.21.15.28:0/88891968 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fadc80a2d80 0x7fadc80a5240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fadd40094d0 tx=0x7fadd4004ab0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.884+0000 7fadc77fe640 1 -- 172.21.15.28:0/2540053769 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc049030 con 0x556c6fd3c570 2024-04-25T01:38:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.884+0000 7fadc77fe640 1 -- 172.21.15.28:0/2540053769 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadcc03b460 con 0x556c6fd3c570 2024-04-25T01:38:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.884+0000 7fadc77fe640 1 -- 172.21.15.28:0/2540053769 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc03be30 con 0x556c6fd3c570 2024-04-25T01:38:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.885+0000 7fade5fef740 1 -- 172.21.15.28:0/2540053769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd3c570 msgr2=0x556c6fd3e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.885+0000 7fade5fef740 1 --2- 172.21.15.28:0/2540053769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd3c570 0x556c6fd3e960 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fadcc0ed0f0 tx=0x7fadcc03b640 comp rx=0 tx=0).stop 2024-04-25T01:38:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.885+0000 7fade5fef740 1 -- 172.21.15.28:0/2540053769 shutdown_connections 2024-04-25T01:38:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.885+0000 7fade5fef740 1 --2- 172.21.15.28:0/2540053769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd3c570 0x556c6fd3e960 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.885+0000 7fade5fef740 1 --2- 172.21.15.28:0/2540053769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd31ea0 0x556c6fd3c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.885+0000 7fade5fef740 1 --2- 172.21.15.28:0/2540053769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd31580 0x556c6fd31960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.885+0000 7fade5fef740 1 -- 172.21.15.28:0/2540053769 >> 172.21.15.28:0/2540053769 conn(0x556c6fd2d8a0 msgr2=0x556c6fd2dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.885+0000 7fade5fef740 1 -- 172.21.15.28:0/2540053769 shutdown_connections 2024-04-25T01:38:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.885+0000 7fade5fef740 1 -- 172.21.15.28:0/2540053769 wait complete. 2024-04-25T01:38:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.885+0000 7fade5fef740 1 -- start start 2024-04-25T01:38:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.885+0000 7fade5fef740 1 Processor -- start 2024-04-25T01:38:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.886+0000 7fade5fef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31580 0x556c6fd3bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.886+0000 7fade5fef740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd31ea0 0x556c6fd390e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.886+0000 7fade5fef740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd3c570 0x556c6fd39620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.886+0000 7fade5fef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556c6fd3f120 con 0x556c6fd31580 2024-04-25T01:38:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.886+0000 7fade5fef740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556c6fc52050 con 0x556c6fd31ea0 2024-04-25T01:38:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.886+0000 7fade5fef740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x556c6fd3f2a0 con 0x556c6fd3c570 2024-04-25T01:38:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.886+0000 7fade5fef740 10 client.0 _ll_register_callbacks cb 0x556c6f9a9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.886+0000 7fade1076640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31580 0x556c6fd3bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.886+0000 7fade1076640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31580 0x556c6fd3bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45430/0 (socket says 172.21.15.28:45430) 2024-04-25T01:38:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.886+0000 7fade1076640 1 -- 172.21.15.28:0/4261298123 learned_addr learned my addr 172.21.15.28:0/4261298123 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.886+0000 7fade1877640 1 --2- 172.21.15.28:0/4261298123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd3c570 0x556c6fd39620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.886+0000 7fade0875640 1 --2- 172.21.15.28:0/4261298123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd31ea0 0x556c6fd390e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.886+0000 7fade1076640 1 -- 172.21.15.28:0/4261298123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd3c570 msgr2=0x556c6fd39620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.886+0000 7fade1076640 1 --2- 172.21.15.28:0/4261298123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd3c570 0x556c6fd39620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.886+0000 7fade1076640 1 -- 172.21.15.28:0/4261298123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd31ea0 msgr2=0x556c6fd390e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.886+0000 7fade1076640 1 --2- 172.21.15.28:0/4261298123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd31ea0 0x556c6fd390e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.886+0000 7fade1076640 1 -- 172.21.15.28:0/4261298123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c6fc23dc0 con 0x556c6fd31580 2024-04-25T01:38:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.886+0000 7fade0875640 1 --2- 172.21.15.28:0/4261298123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd31ea0 0x556c6fd390e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.887+0000 7fade1076640 1 --2- 172.21.15.28:0/4261298123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31580 0x556c6fd3bdf0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fadd800baa0 tx=0x7fadd800bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=145d5318247fa37a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.887+0000 7fadc4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.887+0000 7fadc4ff9640 1 -- 172.21.15.28:0/4261298123 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd8003df0 con 0x556c6fd31580 2024-04-25T01:38:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.887+0000 7fadc4ff9640 1 -- 172.21.15.28:0/4261298123 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fadd8011440 con 0x556c6fd31580 2024-04-25T01:38:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.887+0000 7fade5fef740 1 -- 172.21.15.28:0/4261298123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556c6fca3bb0 con 0x556c6fd31580 2024-04-25T01:38:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.887+0000 7fadc4ff9640 1 -- 172.21.15.28:0/4261298123 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fadd8011640 con 0x556c6fd31580 2024-04-25T01:38:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.887+0000 7fade5fef740 10 client.17388 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.887+0000 7fade5fef740 1 -- 172.21.15.28:0/4261298123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556c6f9aa3f0 con 0x556c6fd31580 2024-04-25T01:38:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.887+0000 7fadc4ff9640 1 -- 172.21.15.28:0/4261298123 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556c6fca3bb0 con 0x556c6fd31580 2024-04-25T01:38:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.887+0000 7fadc4ff9640 1 -- 172.21.15.28:0/4261298123 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fadd801b930 con 0x556c6fd31580 2024-04-25T01:38:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.887+0000 7fade5fef740 10 client.17388 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.887+0000 7fade5fef740 10 client.17388 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.887+0000 7fade5fef740 20 client.17388 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.887+0000 7fade5fef740 1 --2- 172.21.15.28:0/4261298123 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556c6fd4c280 0x556c6fd4e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.887+0000 7fade5fef740 4 client.17388 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.887+0000 7fade5fef740 1 -- 172.21.15.28:0/4261298123 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c6fc9df00 con 0x556c6fd4c280 2024-04-25T01:38:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.888+0000 7fade0875640 1 --2- 172.21.15.28:0/4261298123 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556c6fd4c280 0x556c6fd4e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.888+0000 7fadc4ff9640 10 client.17388 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.888+0000 7fade0875640 1 --2- 172.21.15.28:0/4261298123 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556c6fd4c280 0x556c6fd4e710 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d7cb0b7947b0627 server_cookie=1eff9ebfc42084f3 in_seq=0 out_seq=0 2024-04-25T01:38:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.889+0000 7fadc4ff9640 1 -- 172.21.15.28:0/4261298123 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556c6fc9df00 con 0x556c6fd4c280 2024-04-25T01:38:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.889+0000 7fadc4ff9640 10 client.17388 handle_command_reply: tid=0 2024-04-25T01:38:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.889+0000 7fade5fef740 1 client.17388 shutdown 2024-04-25T01:38:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.889+0000 7fade5fef740 1 -- 172.21.15.28:0/4261298123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31580 msgr2=0x556c6fd3bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.889+0000 7fade5fef740 1 --2- 172.21.15.28:0/4261298123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31580 0x556c6fd3bdf0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fadd800baa0 tx=0x7fadd800bf60 comp rx=0 tx=0).stop 2024-04-25T01:38:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.889+0000 7fade5fef740 1 -- 172.21.15.28:0/4261298123 shutdown_connections 2024-04-25T01:38:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.889+0000 7fade5fef740 1 --2- 172.21.15.28:0/4261298123 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556c6fd4c280 0x556c6fd4e710 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.889+0000 7fade5fef740 1 --2- 172.21.15.28:0/4261298123 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd3c570 0x556c6fd39620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.889+0000 7fade5fef740 1 --2- 172.21.15.28:0/4261298123 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd31ea0 0x556c6fd390e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.889+0000 7fadc4ff9640 0 client.17388 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:36.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.889+0000 7fade5fef740 1 --2- 172.21.15.28:0/4261298123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31580 0x556c6fd3bdf0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.889+0000 7fade5fef740 1 -- 172.21.15.28:0/4261298123 >> 172.21.15.28:0/4261298123 conn(0x556c6fd2d8a0 msgr2=0x556c6fd2cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.889+0000 7fade5fef740 1 -- 172.21.15.28:0/4261298123 shutdown_connections 2024-04-25T01:38:36.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.889+0000 7fade5fef740 1 -- 172.21.15.28:0/4261298123 wait complete. 2024-04-25T01:38:36.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.889+0000 7fade5fef740 20 client.17388 trim_cache size 0 max 16384 2024-04-25T01:38:36.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.902+0000 7fade5fef740 1 Processor -- start 2024-04-25T01:38:36.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.902+0000 7fade5fef740 1 -- start start 2024-04-25T01:38:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fade5fef740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd31580 0x556c6fd38260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fade5fef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31ea0 0x556c6fd3b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fade5fef740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd3c570 0x556c6fd53fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fade5fef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faddc1b0d10 con 0x556c6fd31ea0 2024-04-25T01:38:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fade5fef740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faddc1b2850 con 0x556c6fd3c570 2024-04-25T01:38:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fade5fef740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7faddc1b13e0 con 0x556c6fd31580 2024-04-25T01:38:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fade0875640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31ea0 0x556c6fd3b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fade0875640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31ea0 0x556c6fd3b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45436/0 (socket says 172.21.15.28:45436) 2024-04-25T01:38:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fade0875640 1 -- 172.21.15.28:0/378137607 learned_addr learned my addr 172.21.15.28:0/378137607 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fade1076640 1 --2- 172.21.15.28:0/378137607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd31580 0x556c6fd38260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fade0875640 1 -- 172.21.15.28:0/378137607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd31580 msgr2=0x556c6fd38260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fade0875640 1 --2- 172.21.15.28:0/378137607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd31580 0x556c6fd38260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fade0875640 1 -- 172.21.15.28:0/378137607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd3c570 msgr2=0x556c6fd53fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fade0875640 1 --2- 172.21.15.28:0/378137607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd3c570 0x556c6fd53fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fade0875640 1 -- 172.21.15.28:0/378137607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c6fa747d0 con 0x556c6fd31ea0 2024-04-25T01:38:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fade1076640 1 --2- 172.21.15.28:0/378137607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd31580 0x556c6fd38260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fade0875640 1 --2- 172.21.15.28:0/378137607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31ea0 0x556c6fd3b920 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fadcc005ab0 tx=0x7fadcc0f0100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f2b92396e9779c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fadc6ffd640 1 -- 172.21.15.28:0/378137607 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc076070 con 0x556c6fd31ea0 2024-04-25T01:38:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fadc6ffd640 1 -- 172.21.15.28:0/378137607 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadcc03be30 con 0x556c6fd31ea0 2024-04-25T01:38:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.903+0000 7fadc6ffd640 1 -- 172.21.15.28:0/378137607 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc049040 con 0x556c6fd31ea0 2024-04-25T01:38:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.904+0000 7fade5fef740 1 -- 172.21.15.28:0/378137607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31ea0 msgr2=0x556c6fd3b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.904+0000 7fade5fef740 1 --2- 172.21.15.28:0/378137607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31ea0 0x556c6fd3b920 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fadcc005ab0 tx=0x7fadcc0f0100 comp rx=0 tx=0).stop 2024-04-25T01:38:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.904+0000 7fade5fef740 1 -- 172.21.15.28:0/378137607 shutdown_connections 2024-04-25T01:38:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.904+0000 7fade5fef740 1 --2- 172.21.15.28:0/378137607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd3c570 0x556c6fd53fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.904+0000 7fade5fef740 1 --2- 172.21.15.28:0/378137607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31ea0 0x556c6fd3b920 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.904+0000 7fade5fef740 1 --2- 172.21.15.28:0/378137607 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd31580 0x556c6fd38260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.904+0000 7fade5fef740 1 -- 172.21.15.28:0/378137607 >> 172.21.15.28:0/378137607 conn(0x556c6fd2d8a0 msgr2=0x556c6fd2c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.904+0000 7fade5fef740 1 -- 172.21.15.28:0/378137607 shutdown_connections 2024-04-25T01:38:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.904+0000 7fade5fef740 1 -- 172.21.15.28:0/378137607 wait complete. 2024-04-25T01:38:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.904+0000 7fade5fef740 1 -- start start 2024-04-25T01:38:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.904+0000 7fade5fef740 1 Processor -- start 2024-04-25T01:38:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade5fef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31580 0x556c6fd36060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade5fef740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd31ea0 0x556c6fd365a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade5fef740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd3c570 0x556c6fd36ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade5fef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faddc1b2180 con 0x556c6fd31580 2024-04-25T01:38:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade5fef740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faddc1b1ab0 con 0x556c6fd3c570 2024-04-25T01:38:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade5fef740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7faddc1aecf0 con 0x556c6fd31ea0 2024-04-25T01:38:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade5fef740 10 client.0 _ll_register_callbacks cb 0x7fadd8004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade1076640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31580 0x556c6fd36060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade1076640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31580 0x556c6fd36060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45442/0 (socket says 172.21.15.28:45442) 2024-04-25T01:38:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade1076640 1 -- 172.21.15.28:0/99718664 learned_addr learned my addr 172.21.15.28:0/99718664 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade0875640 1 --2- 172.21.15.28:0/99718664 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd31ea0 0x556c6fd365a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade1076640 1 -- 172.21.15.28:0/99718664 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd31ea0 msgr2=0x556c6fd365a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade1076640 1 --2- 172.21.15.28:0/99718664 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd31ea0 0x556c6fd365a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade1076640 1 -- 172.21.15.28:0/99718664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd3c570 msgr2=0x556c6fd36ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade1877640 1 --2- 172.21.15.28:0/99718664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd3c570 0x556c6fd36ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade1076640 1 --2- 172.21.15.28:0/99718664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd3c570 0x556c6fd36ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade1076640 1 -- 172.21.15.28:0/99718664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c6fdb1a50 con 0x556c6fd31580 2024-04-25T01:38:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade0875640 1 --2- 172.21.15.28:0/99718664 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd31ea0 0x556c6fd365a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade1076640 1 --2- 172.21.15.28:0/99718664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31580 0x556c6fd36060 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fadd8011600 tx=0x7fadd800b400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b4497fb1b9c5e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fadc4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fadc4ff9640 1 -- 172.21.15.28:0/99718664 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd802d020 con 0x556c6fd31580 2024-04-25T01:38:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fadc4ff9640 1 -- 172.21.15.28:0/99718664 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fadd8005250 con 0x556c6fd31580 2024-04-25T01:38:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.905+0000 7fade5fef740 1 -- 172.21.15.28:0/99718664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadcc03be30 con 0x556c6fd31580 2024-04-25T01:38:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.906+0000 7fadc4ff9640 1 -- 172.21.15.28:0/99718664 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fadd80043a0 con 0x556c6fd31580 2024-04-25T01:38:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.906+0000 7fade5fef740 10 client.17394 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.906+0000 7fade5fef740 1 -- 172.21.15.28:0/99718664 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556c6fdb2cc0 con 0x556c6fd31580 2024-04-25T01:38:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.906+0000 7fadc4ff9640 1 -- 172.21.15.28:0/99718664 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadcc03be30 con 0x556c6fd31580 2024-04-25T01:38:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.906+0000 7fadc4ff9640 1 -- 172.21.15.28:0/99718664 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fadd8011a70 con 0x556c6fd31580 2024-04-25T01:38:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.906+0000 7fade5fef740 10 client.17394 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.906+0000 7fade5fef740 10 client.17394 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.906+0000 7fade5fef740 20 client.17394 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.906+0000 7fade5fef740 1 --2- 172.21.15.28:0/99718664 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556c6fdb34c0 0x556c6fdb58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.906+0000 7fade5fef740 4 client.17394 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:38:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.906+0000 7fade5fef740 1 -- 172.21.15.28:0/99718664 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556c6fd399b0 con 0x556c6fdb34c0 2024-04-25T01:38:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.906+0000 7fade0875640 1 --2- 172.21.15.28:0/99718664 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556c6fdb34c0 0x556c6fdb58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.907+0000 7fadc4ff9640 10 client.17394 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.907+0000 7fade0875640 1 --2- 172.21.15.28:0/99718664 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556c6fdb34c0 0x556c6fdb58e0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d9e8d683232c056 server_cookie=885e9989d3ec15ab in_seq=0 out_seq=0 2024-04-25T01:38:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.907+0000 7fadc4ff9640 1 -- 172.21.15.28:0/99718664 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556c6fd399b0 con 0x556c6fdb34c0 2024-04-25T01:38:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.907+0000 7fadc4ff9640 10 client.17394 handle_command_reply: tid=0 2024-04-25T01:38:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.907+0000 7fade5fef740 1 client.17394 shutdown 2024-04-25T01:38:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.907+0000 7fade5fef740 1 -- 172.21.15.28:0/99718664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31580 msgr2=0x556c6fd36060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.907+0000 7fade5fef740 1 --2- 172.21.15.28:0/99718664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31580 0x556c6fd36060 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fadd8011600 tx=0x7fadd800b400 comp rx=0 tx=0).stop 2024-04-25T01:38:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.907+0000 7fade5fef740 1 -- 172.21.15.28:0/99718664 shutdown_connections 2024-04-25T01:38:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.907+0000 7fade5fef740 1 --2- 172.21.15.28:0/99718664 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556c6fdb34c0 0x556c6fdb58e0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.907+0000 7fade5fef740 1 --2- 172.21.15.28:0/99718664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556c6fd3c570 0x556c6fd36ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.907+0000 7fade5fef740 1 --2- 172.21.15.28:0/99718664 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556c6fd31ea0 0x556c6fd365a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.907+0000 7fadc4ff9640 0 client.17394 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.907+0000 7fade5fef740 1 --2- 172.21.15.28:0/99718664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c6fd31580 0x556c6fd36060 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.907+0000 7fade5fef740 1 -- 172.21.15.28:0/99718664 >> 172.21.15.28:0/99718664 conn(0x556c6fd2d8a0 msgr2=0x556c6fd2cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.907+0000 7fade5fef740 1 -- 172.21.15.28:0/99718664 shutdown_connections 2024-04-25T01:38:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.907+0000 7fade5fef740 1 -- 172.21.15.28:0/99718664 wait complete. 2024-04-25T01:38:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.907+0000 7fade5fef740 20 client.17394 trim_cache size 0 max 16384 2024-04-25T01:38:36.897 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:38:36.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.908+0000 7fade32da640 1 -- 172.21.15.28:0/88891968 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fadc80a2d80 msgr2=0x7fadc80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.908+0000 7fade32da640 1 --2- 172.21.15.28:0/88891968 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fadc80a2d80 0x7fadc80a5240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fadd40094d0 tx=0x7fadd4004ab0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.908+0000 7fade32da640 1 -- 172.21.15.28:0/88891968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faddc100e60 msgr2=0x7faddc19a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:36.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.908+0000 7fade32da640 1 --2- 172.21.15.28:0/88891968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faddc100e60 0x7faddc19a820 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fadcc006010 tx=0x7fadcc004270 comp rx=0 tx=0).stop 2024-04-25T01:38:36.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.908+0000 7fade32da640 1 -- 172.21.15.28:0/88891968 shutdown_connections 2024-04-25T01:38:36.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.908+0000 7fade32da640 1 --2- 172.21.15.28:0/88891968 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fadc80a2d80 0x7fadc80a5240 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.908+0000 7fade32da640 1 --2- 172.21.15.28:0/88891968 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faddc10e0f0 0x7faddc19eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.908+0000 7fade32da640 1 --2- 172.21.15.28:0/88891968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faddc100e60 0x7faddc19a820 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.908+0000 7fade32da640 1 --2- 172.21.15.28:0/88891968 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faddc100540 0x7faddc19a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:36.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.908+0000 7fade32da640 1 -- 172.21.15.28:0/88891968 >> 172.21.15.28:0/88891968 conn(0x7faddc077740 msgr2=0x7faddc10f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:36.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.908+0000 7fade32da640 1 -- 172.21.15.28:0/88891968 shutdown_connections 2024-04-25T01:38:36.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:36.908+0000 7fade32da640 1 -- 172.21.15.28:0/88891968 wait complete. 2024-04-25T01:38:37.064 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:36 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1268457187' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:38:37.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:36 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1268457187' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:38:37.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:36 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1268457187' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:38:37.901 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:38:37.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:38:37.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:38:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.041+0000 7f45b3fed640 1 -- 172.21.15.28:0/670028185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac1007b0 msgr2=0x7f45ac100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.041+0000 7f45b3fed640 1 --2- 172.21.15.28:0/670028185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac1007b0 0x7f45ac100b90 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009ca0 tx=0x7f45a802a9e0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.041+0000 7f45b3fed640 1 -- 172.21.15.28:0/670028185 shutdown_connections 2024-04-25T01:38:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.041+0000 7f45b3fed640 1 --2- 172.21.15.28:0/670028185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45ac105ad0 0x7f45ac105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.041+0000 7f45b3fed640 1 --2- 172.21.15.28:0/670028185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45ac1010d0 0x7f45ac105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.041+0000 7f45b3fed640 1 --2- 172.21.15.28:0/670028185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac1007b0 0x7f45ac100b90 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.041+0000 7f45b3fed640 1 -- 172.21.15.28:0/670028185 >> 172.21.15.28:0/670028185 conn(0x7f45ac077740 msgr2=0x7f45ac077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.041+0000 7f45b3fed640 1 -- 172.21.15.28:0/670028185 shutdown_connections 2024-04-25T01:38:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.041+0000 7f45b3fed640 1 -- 172.21.15.28:0/670028185 wait complete. 2024-04-25T01:38:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.041+0000 7f45b3fed640 1 Processor -- start 2024-04-25T01:38:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.041+0000 7f45b3fed640 1 -- start start 2024-04-25T01:38:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45b3fed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac1007b0 0x7f45ac1a4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45b3fed640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45ac1010d0 0x7f45ac1a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45b3fed640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45ac105ad0 0x7f45ac19e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45b3fed640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1181c0 con 0x7f45ac1007b0 2024-04-25T01:38:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45b3fed640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac118040 con 0x7f45ac105ad0 2024-04-25T01:38:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45b3fed640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac118340 con 0x7f45ac1010d0 2024-04-25T01:38:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45b1d89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac1007b0 0x7f45ac1a4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45b1d89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac1007b0 0x7f45ac1a4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45452/0 (socket says 172.21.15.28:45452) 2024-04-25T01:38:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45b1d89640 1 -- 172.21.15.28:0/3829108227 learned_addr learned my addr 172.21.15.28:0/3829108227 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45b1588640 1 --2- 172.21.15.28:0/3829108227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45ac1010d0 0x7f45ac1a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45b258a640 1 --2- 172.21.15.28:0/3829108227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45ac105ad0 0x7f45ac19e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45b1d89640 1 -- 172.21.15.28:0/3829108227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45ac1010d0 msgr2=0x7f45ac1a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45b1d89640 1 --2- 172.21.15.28:0/3829108227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45ac1010d0 0x7f45ac1a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45b1d89640 1 -- 172.21.15.28:0/3829108227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45ac105ad0 msgr2=0x7f45ac19e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45b1d89640 1 --2- 172.21.15.28:0/3829108227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45ac105ad0 0x7f45ac19e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45b1d89640 1 -- 172.21.15.28:0/3829108227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ac19ed40 con 0x7f45ac1007b0 2024-04-25T01:38:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45b1d89640 1 --2- 172.21.15.28:0/3829108227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac1007b0 0x7f45ac1a4410 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009430 tx=0x7f45a8003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45a2ffd640 1 -- 172.21.15.28:0/3829108227 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a8046070 con 0x7f45ac1007b0 2024-04-25T01:38:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45a2ffd640 1 -- 172.21.15.28:0/3829108227 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45a802e870 con 0x7f45ac1007b0 2024-04-25T01:38:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45b3fed640 1 -- 172.21.15.28:0/3829108227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45ac19efd0 con 0x7f45ac1007b0 2024-04-25T01:38:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45a2ffd640 1 -- 172.21.15.28:0/3829108227 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a8033070 con 0x7f45ac1007b0 2024-04-25T01:38:38.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.042+0000 7f45b3fed640 1 -- 172.21.15.28:0/3829108227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45ac1ab2d0 con 0x7f45ac1007b0 2024-04-25T01:38:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.045+0000 7f45a2ffd640 1 -- 172.21.15.28:0/3829108227 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f45a8049070 con 0x7f45ac1007b0 2024-04-25T01:38:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.045+0000 7f45b6dce740 1 Processor -- start 2024-04-25T01:38:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.045+0000 7f45b6dce740 1 -- start start 2024-04-25T01:38:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.045+0000 7f45b6dce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857d4980 0x55b0857d2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.045+0000 7f45b6dce740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857d3290 0x55b0857d3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.045+0000 7f45b6dce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857de010 0x55b0857e0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.045+0000 7f45b6dce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b0857372c0 con 0x55b0857de010 2024-04-25T01:38:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.045+0000 7f45b6dce740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b0856df910 con 0x55b0857d4980 2024-04-25T01:38:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.046+0000 7f45b6dce740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b085733280 con 0x55b0857d3290 2024-04-25T01:38:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.046+0000 7f45a2ffd640 1 --2- 172.21.15.28:0/3829108227 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f45980a2d00 0x7f45980a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.046+0000 7f45b1588640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857de010 0x55b0857e0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.046+0000 7f45b258a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857d4980 0x55b0857d2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.046+0000 7f45b1588640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857de010 0x55b0857e0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45466/0 (socket says 172.21.15.28:45466) 2024-04-25T01:38:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.046+0000 7f45b1588640 1 -- 172.21.15.28:0/1038855019 learned_addr learned my addr 172.21.15.28:0/1038855019 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.046+0000 7f45b1d89640 1 --2- 172.21.15.28:0/1038855019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857d3290 0x55b0857d3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.046+0000 7f45b258a640 1 --2- 172.21.15.28:0/3829108227 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f45980a2d00 0x7f45980a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.046+0000 7f45a2ffd640 1 -- 172.21.15.28:0/3829108227 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f45a80f0530 con 0x7f45ac1007b0 2024-04-25T01:38:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.046+0000 7f45b1588640 1 -- 172.21.15.28:0/1038855019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857d3290 msgr2=0x55b0857d3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.046+0000 7f45b1588640 1 --2- 172.21.15.28:0/1038855019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857d3290 0x55b0857d3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.046+0000 7f45b1588640 1 -- 172.21.15.28:0/1038855019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857d4980 msgr2=0x55b0857d2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.046+0000 7f45b1588640 1 --2- 172.21.15.28:0/1038855019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857d4980 0x55b0857d2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.046+0000 7f45b1588640 1 -- 172.21.15.28:0/1038855019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0856f0fb0 con 0x55b0857de010 2024-04-25T01:38:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.046+0000 7f45b258a640 1 --2- 172.21.15.28:0/3829108227 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f45980a2d00 0x7f45980a51c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f45a400dfd0 tx=0x7f45a400d4f0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.046+0000 7f45b1588640 1 --2- 172.21.15.28:0/1038855019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857de010 0x55b0857e0400 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f459c00b7c0 tx=0x7f459c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b10e6f67949a1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.046+0000 7f4597fff640 1 -- 172.21.15.28:0/1038855019 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c00d950 con 0x55b0857de010 2024-04-25T01:38:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.047+0000 7f4597fff640 1 -- 172.21.15.28:0/1038855019 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f459c018070 con 0x55b0857de010 2024-04-25T01:38:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.047+0000 7f4597fff640 1 -- 172.21.15.28:0/1038855019 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c0123e0 con 0x55b0857de010 2024-04-25T01:38:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.047+0000 7f45b6dce740 1 -- 172.21.15.28:0/1038855019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857de010 msgr2=0x55b0857e0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.047+0000 7f45b6dce740 1 --2- 172.21.15.28:0/1038855019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857de010 0x55b0857e0400 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f459c00b7c0 tx=0x7f459c00bc80 comp rx=0 tx=0).stop 2024-04-25T01:38:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.047+0000 7f45b6dce740 1 -- 172.21.15.28:0/1038855019 shutdown_connections 2024-04-25T01:38:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.047+0000 7f45b6dce740 1 --2- 172.21.15.28:0/1038855019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857de010 0x55b0857e0400 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.047+0000 7f45b6dce740 1 --2- 172.21.15.28:0/1038855019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857d3290 0x55b0857d3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.047+0000 7f45b6dce740 1 --2- 172.21.15.28:0/1038855019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857d4980 0x55b0857d2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.047+0000 7f45b6dce740 1 -- 172.21.15.28:0/1038855019 >> 172.21.15.28:0/1038855019 conn(0x55b08579b950 msgr2=0x55b0857cf7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.047+0000 7f45b6dce740 1 -- 172.21.15.28:0/1038855019 shutdown_connections 2024-04-25T01:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.047+0000 7f45b6dce740 1 -- 172.21.15.28:0/1038855019 wait complete. 2024-04-25T01:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.048+0000 7f45b6dce740 1 -- start start 2024-04-25T01:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.048+0000 7f45b6dce740 1 Processor -- start 2024-04-25T01:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.048+0000 7f45b6dce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857d3290 0x55b0857daf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.048+0000 7f45b6dce740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857d4980 0x55b0857db470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.048+0000 7f45b6dce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857de010 0x55b0857db9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.048+0000 7f45b6dce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b0857e0ea0 con 0x55b0857d3290 2024-04-25T01:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.048+0000 7f45b6dce740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b0856f2e90 con 0x55b0857de010 2024-04-25T01:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.048+0000 7f45b6dce740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b0857e1020 con 0x55b0857d4980 2024-04-25T01:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.048+0000 7f45b6dce740 10 client.0 _ll_register_callbacks cb 0x55b08572e970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.048+0000 7f45b1d89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857d3290 0x55b0857daf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.048+0000 7f45b1d89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857d3290 0x55b0857daf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45476/0 (socket says 172.21.15.28:45476) 2024-04-25T01:38:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.048+0000 7f45b1d89640 1 -- 172.21.15.28:0/3890790336 learned_addr learned my addr 172.21.15.28:0/3890790336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.048+0000 7f45b258a640 1 --2- 172.21.15.28:0/3890790336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857de010 0x55b0857db9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.049+0000 7f45b1588640 1 --2- 172.21.15.28:0/3890790336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857d4980 0x55b0857db470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.049+0000 7f45b1d89640 1 -- 172.21.15.28:0/3890790336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857d4980 msgr2=0x55b0857db470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.049+0000 7f45b1d89640 1 --2- 172.21.15.28:0/3890790336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857d4980 0x55b0857db470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.049+0000 7f45b1d89640 1 -- 172.21.15.28:0/3890790336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857de010 msgr2=0x55b0857db9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.049+0000 7f45b1d89640 1 --2- 172.21.15.28:0/3890790336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857de010 0x55b0857db9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.049+0000 7f45b1d89640 1 -- 172.21.15.28:0/3890790336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0855137e0 con 0x55b0857d3290 2024-04-25T01:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.049+0000 7f45b1d89640 1 --2- 172.21.15.28:0/3890790336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857d3290 0x55b0857daf30 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f45a80b8b70 tx=0x7f45a808b640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c0d1b2ad0b24073 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.049+0000 7f45957fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.049+0000 7f45957fa640 1 -- 172.21.15.28:0/3890790336 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a80bc4e0 con 0x55b0857d3290 2024-04-25T01:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.049+0000 7f45b6dce740 1 -- 172.21.15.28:0/3890790336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f459c018070 con 0x55b0857d3290 2024-04-25T01:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.049+0000 7f45957fa640 1 -- 172.21.15.28:0/3890790336 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f45a80f01d0 con 0x55b0857d3290 2024-04-25T01:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.049+0000 7f45957fa640 1 -- 172.21.15.28:0/3890790336 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f45a80bc070 con 0x55b0857d3290 2024-04-25T01:38:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.049+0000 7f45b6dce740 10 client.17406 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.049+0000 7f45b6dce740 1 -- 172.21.15.28:0/3890790336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b085573f70 con 0x55b0857d3290 2024-04-25T01:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.049+0000 7f45957fa640 1 -- 172.21.15.28:0/3890790336 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f459c018070 con 0x55b0857d3290 2024-04-25T01:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.050+0000 7f45957fa640 1 -- 172.21.15.28:0/3890790336 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f45a802fa90 con 0x55b0857d3290 2024-04-25T01:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.050+0000 7f45b6dce740 10 client.17406 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.050+0000 7f45b6dce740 10 client.17406 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.050+0000 7f45b6dce740 20 client.17406 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.050+0000 7f45b6dce740 1 --2- 172.21.15.28:0/3890790336 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b0857ee0a0 0x55b0857f04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.050+0000 7f45b6dce740 4 client.17406 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.050+0000 7f45b6dce740 1 -- 172.21.15.28:0/3890790336 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b08570b5a0 con 0x55b0857ee0a0 2024-04-25T01:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.050+0000 7f45b1588640 1 --2- 172.21.15.28:0/3890790336 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b0857ee0a0 0x55b0857f04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.050+0000 7f45b1588640 1 --2- 172.21.15.28:0/3890790336 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b0857ee0a0 0x55b0857f04f0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9eff8542370ffb5c server_cookie=72308711d726fe94 in_seq=0 out_seq=0 2024-04-25T01:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.050+0000 7f45957fa640 10 client.17406 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.051+0000 7f45957fa640 1 -- 172.21.15.28:0/3890790336 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b08570b5a0 con 0x55b0857ee0a0 2024-04-25T01:38:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.051+0000 7f45957fa640 10 client.17406 handle_command_reply: tid=0 2024-04-25T01:38:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.051+0000 7f45b6dce740 1 client.17406 shutdown 2024-04-25T01:38:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.051+0000 7f45b6dce740 1 -- 172.21.15.28:0/3890790336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857d3290 msgr2=0x55b0857daf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.051+0000 7f45b6dce740 1 --2- 172.21.15.28:0/3890790336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857d3290 0x55b0857daf30 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f45a80b8b70 tx=0x7f45a808b640 comp rx=0 tx=0).stop 2024-04-25T01:38:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.051+0000 7f45b6dce740 1 -- 172.21.15.28:0/3890790336 shutdown_connections 2024-04-25T01:38:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.051+0000 7f45b6dce740 1 --2- 172.21.15.28:0/3890790336 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b0857ee0a0 0x55b0857f04f0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.051+0000 7f45b6dce740 1 --2- 172.21.15.28:0/3890790336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857de010 0x55b0857db9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.051+0000 7f45b6dce740 1 --2- 172.21.15.28:0/3890790336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857d4980 0x55b0857db470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.051+0000 7f45b6dce740 1 --2- 172.21.15.28:0/3890790336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857d3290 0x55b0857daf30 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.051+0000 7f45957fa640 0 client.17406 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.051+0000 7f45b6dce740 1 -- 172.21.15.28:0/3890790336 >> 172.21.15.28:0/3890790336 conn(0x55b08579b950 msgr2=0x55b0857d5660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.051+0000 7f45b6dce740 1 -- 172.21.15.28:0/3890790336 shutdown_connections 2024-04-25T01:38:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.051+0000 7f45b6dce740 1 -- 172.21.15.28:0/3890790336 wait complete. 2024-04-25T01:38:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.051+0000 7f45b6dce740 20 client.17406 trim_cache size 0 max 16384 2024-04-25T01:38:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.069+0000 7f45b6dce740 1 Processor -- start 2024-04-25T01:38:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.069+0000 7f45b6dce740 1 -- start start 2024-04-25T01:38:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.069+0000 7f45b6dce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857d3290 0x55b0857e9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.069+0000 7f45b6dce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857d4980 0x55b0857ce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.069+0000 7f45b6dce740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857de010 0x55b0857f5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.069+0000 7f45b6dce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1b50a0 con 0x55b0857d3290 2024-04-25T01:38:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.069+0000 7f45b6dce740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1b6be0 con 0x55b0857d4980 2024-04-25T01:38:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.069+0000 7f45b6dce740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1b5770 con 0x55b0857de010 2024-04-25T01:38:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.070+0000 7f45b1d89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857d3290 0x55b0857e9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.070+0000 7f45b1d89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857d3290 0x55b0857e9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45486/0 (socket says 172.21.15.28:45486) 2024-04-25T01:38:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.070+0000 7f45b1d89640 1 -- 172.21.15.28:0/2919155530 learned_addr learned my addr 172.21.15.28:0/2919155530 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.070+0000 7f45b258a640 1 --2- 172.21.15.28:0/2919155530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857de010 0x55b0857f5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.070+0000 7f45b1588640 1 --2- 172.21.15.28:0/2919155530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857d4980 0x55b0857ce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.070+0000 7f45b1d89640 1 -- 172.21.15.28:0/2919155530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857de010 msgr2=0x55b0857f5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.070+0000 7f45b1d89640 1 --2- 172.21.15.28:0/2919155530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857de010 0x55b0857f5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.070+0000 7f45b1d89640 1 -- 172.21.15.28:0/2919155530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857d4980 msgr2=0x55b0857ce7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.070+0000 7f45b1d89640 1 --2- 172.21.15.28:0/2919155530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857d4980 0x55b0857ce7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.070+0000 7f45b1d89640 1 -- 172.21.15.28:0/2919155530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b08540b170 con 0x55b0857d3290 2024-04-25T01:38:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.070+0000 7f45b1588640 1 --2- 172.21.15.28:0/2919155530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857d4980 0x55b0857ce7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.070+0000 7f45b1d89640 1 --2- 172.21.15.28:0/2919155530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857d3290 0x55b0857e9da0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f45a8005cc0 tx=0x7f45a80a7050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36b410897483dfa8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.070+0000 7f45977fe640 1 -- 172.21.15.28:0/2919155530 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a80bc250 con 0x55b0857d3290 2024-04-25T01:38:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.070+0000 7f45977fe640 1 -- 172.21.15.28:0/2919155530 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45a80bce10 con 0x55b0857d3290 2024-04-25T01:38:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.070+0000 7f45977fe640 1 -- 172.21.15.28:0/2919155530 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a808b190 con 0x55b0857d3290 2024-04-25T01:38:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.071+0000 7f45b6dce740 1 -- 172.21.15.28:0/2919155530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857d3290 msgr2=0x55b0857e9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.071+0000 7f45b6dce740 1 --2- 172.21.15.28:0/2919155530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857d3290 0x55b0857e9da0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f45a8005cc0 tx=0x7f45a80a7050 comp rx=0 tx=0).stop 2024-04-25T01:38:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.071+0000 7f45b6dce740 1 -- 172.21.15.28:0/2919155530 shutdown_connections 2024-04-25T01:38:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.071+0000 7f45b6dce740 1 --2- 172.21.15.28:0/2919155530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857de010 0x55b0857f5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.071+0000 7f45b6dce740 1 --2- 172.21.15.28:0/2919155530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857d4980 0x55b0857ce7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.071+0000 7f45b6dce740 1 --2- 172.21.15.28:0/2919155530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857d3290 0x55b0857e9da0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.071+0000 7f45b6dce740 1 -- 172.21.15.28:0/2919155530 >> 172.21.15.28:0/2919155530 conn(0x55b08579b950 msgr2=0x55b0857ce0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.071+0000 7f45b6dce740 1 -- 172.21.15.28:0/2919155530 shutdown_connections 2024-04-25T01:38:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.071+0000 7f45b6dce740 1 -- 172.21.15.28:0/2919155530 wait complete. 2024-04-25T01:38:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.071+0000 7f45b6dce740 1 -- start start 2024-04-25T01:38:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.071+0000 7f45b6dce740 1 Processor -- start 2024-04-25T01:38:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45b6dce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857d3290 0x55b0857e64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45b6dce740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857d4980 0x55b0857d71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45b6dce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857de010 0x55b0857d76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45b6dce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1b6510 con 0x55b0857de010 2024-04-25T01:38:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45b6dce740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1b5e40 con 0x55b0857d3290 2024-04-25T01:38:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45b6dce740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac1b3080 con 0x55b0857d4980 2024-04-25T01:38:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45b6dce740 10 client.0 _ll_register_callbacks cb 0x7f45a802f990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45b258a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857de010 0x55b0857d76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45b258a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857de010 0x55b0857d76e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45502/0 (socket says 172.21.15.28:45502) 2024-04-25T01:38:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45b258a640 1 -- 172.21.15.28:0/1108632036 learned_addr learned my addr 172.21.15.28:0/1108632036 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45b1588640 1 --2- 172.21.15.28:0/1108632036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857d4980 0x55b0857d71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45b1d89640 1 --2- 172.21.15.28:0/1108632036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857d3290 0x55b0857e64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45b258a640 1 -- 172.21.15.28:0/1108632036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857d4980 msgr2=0x55b0857d71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45b258a640 1 --2- 172.21.15.28:0/1108632036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857d4980 0x55b0857d71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45b258a640 1 -- 172.21.15.28:0/1108632036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857d3290 msgr2=0x55b0857e64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45b258a640 1 --2- 172.21.15.28:0/1108632036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857d3290 0x55b0857e64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45b258a640 1 -- 172.21.15.28:0/1108632036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0857d7c20 con 0x55b0857de010 2024-04-25T01:38:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45b258a640 1 --2- 172.21.15.28:0/1108632036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857de010 0x55b0857d76e0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f45a4014cc0 tx=0x7f45a400d240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e051c946b55ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45957fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45957fa640 1 -- 172.21.15.28:0/1108632036 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a4029070 con 0x55b0857de010 2024-04-25T01:38:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45b6dce740 1 -- 172.21.15.28:0/1108632036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45a80bce10 con 0x55b0857de010 2024-04-25T01:38:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.072+0000 7f45957fa640 1 -- 172.21.15.28:0/1108632036 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f45a400f070 con 0x55b0857de010 2024-04-25T01:38:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.073+0000 7f45957fa640 1 -- 172.21.15.28:0/1108632036 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f45a4014cf0 con 0x55b0857de010 2024-04-25T01:38:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.073+0000 7f45b6dce740 10 client.17412 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.073+0000 7f45b6dce740 1 -- 172.21.15.28:0/1108632036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b0857d8f20 con 0x55b0857de010 2024-04-25T01:38:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.073+0000 7f45957fa640 1 -- 172.21.15.28:0/1108632036 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45a80bce10 con 0x55b0857de010 2024-04-25T01:38:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.073+0000 7f45957fa640 1 -- 172.21.15.28:0/1108632036 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f45a4015040 con 0x55b0857de010 2024-04-25T01:38:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.073+0000 7f45b6dce740 10 client.17412 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.073+0000 7f45b6dce740 10 client.17412 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.073+0000 7f45b6dce740 20 client.17412 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.073+0000 7f45b6dce740 1 --2- 172.21.15.28:0/1108632036 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b0858559e0 0x55b085857e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.073+0000 7f45b6dce740 4 client.17412 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:38:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.073+0000 7f45b6dce740 1 -- 172.21.15.28:0/1108632036 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b085744800 con 0x55b0858559e0 2024-04-25T01:38:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.073+0000 7f45b1d89640 1 --2- 172.21.15.28:0/1108632036 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b0858559e0 0x55b085857e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.074+0000 7f45b1d89640 1 --2- 172.21.15.28:0/1108632036 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b0858559e0 0x55b085857e00 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46161f1a9450dba6 server_cookie=3bd40742cb022309 in_seq=0 out_seq=0 2024-04-25T01:38:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.074+0000 7f45957fa640 10 client.17412 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:38.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.075+0000 7f45957fa640 1 -- 172.21.15.28:0/1108632036 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b085744800 con 0x55b0858559e0 2024-04-25T01:38:38.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.075+0000 7f45957fa640 10 client.17412 handle_command_reply: tid=0 2024-04-25T01:38:38.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.075+0000 7f45b6dce740 1 client.17412 shutdown 2024-04-25T01:38:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.075+0000 7f45b6dce740 1 -- 172.21.15.28:0/1108632036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857de010 msgr2=0x55b0857d76e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.075+0000 7f45b6dce740 1 --2- 172.21.15.28:0/1108632036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857de010 0x55b0857d76e0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f45a4014cc0 tx=0x7f45a400d240 comp rx=0 tx=0).stop 2024-04-25T01:38:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.075+0000 7f45b6dce740 1 -- 172.21.15.28:0/1108632036 shutdown_connections 2024-04-25T01:38:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.075+0000 7f45b6dce740 1 --2- 172.21.15.28:0/1108632036 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b0858559e0 0x55b085857e00 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.075+0000 7f45b6dce740 1 --2- 172.21.15.28:0/1108632036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0857de010 0x55b0857d76e0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.075+0000 7f45b6dce740 1 --2- 172.21.15.28:0/1108632036 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b0857d4980 0x55b0857d71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.075+0000 7f45957fa640 0 client.17412 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.075+0000 7f45b6dce740 1 --2- 172.21.15.28:0/1108632036 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0857d3290 0x55b0857e64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.075+0000 7f45b6dce740 1 -- 172.21.15.28:0/1108632036 >> 172.21.15.28:0/1108632036 conn(0x55b08579b950 msgr2=0x55b0857ce0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.075+0000 7f45b6dce740 1 -- 172.21.15.28:0/1108632036 shutdown_connections 2024-04-25T01:38:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.075+0000 7f45b6dce740 1 -- 172.21.15.28:0/1108632036 wait complete. 2024-04-25T01:38:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.075+0000 7f45b6dce740 20 client.17412 trim_cache size 0 max 16384 2024-04-25T01:38:38.063 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:38:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:38:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "8e59f8de-521e-443a-b4f6-3dcb8c837111", 2024-04-25T01:38:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:38:38.063 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:38:38.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.076+0000 7f45b3fed640 1 -- 172.21.15.28:0/3829108227 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f45980a2d00 msgr2=0x7f45980a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.076+0000 7f45b3fed640 1 --2- 172.21.15.28:0/3829108227 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f45980a2d00 0x7f45980a51c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f45a400dfd0 tx=0x7f45a400d4f0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.076+0000 7f45b3fed640 1 -- 172.21.15.28:0/3829108227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac1007b0 msgr2=0x7f45ac1a4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.076+0000 7f45b3fed640 1 --2- 172.21.15.28:0/3829108227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac1007b0 0x7f45ac1a4410 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009430 tx=0x7f45a8003f10 comp rx=0 tx=0).stop 2024-04-25T01:38:38.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.076+0000 7f45b3fed640 1 -- 172.21.15.28:0/3829108227 shutdown_connections 2024-04-25T01:38:38.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.076+0000 7f45b3fed640 1 --2- 172.21.15.28:0/3829108227 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f45980a2d00 0x7f45980a51c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.076+0000 7f45b3fed640 1 --2- 172.21.15.28:0/3829108227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45ac105ad0 0x7f45ac19e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.076+0000 7f45b3fed640 1 --2- 172.21.15.28:0/3829108227 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f45ac1010d0 0x7f45ac1a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.076+0000 7f45b3fed640 1 --2- 172.21.15.28:0/3829108227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45ac1007b0 0x7f45ac1a4410 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.076+0000 7f45b3fed640 1 -- 172.21.15.28:0/3829108227 >> 172.21.15.28:0/3829108227 conn(0x7f45ac077740 msgr2=0x7f45ac103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.076+0000 7f45b3fed640 1 -- 172.21.15.28:0/3829108227 shutdown_connections 2024-04-25T01:38:38.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.076+0000 7f45b3fed640 1 -- 172.21.15.28:0/3829108227 wait complete. 2024-04-25T01:38:38.067 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:38:38.068 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:38:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.191+0000 7f5f12129640 1 -- 172.21.15.28:0/2241372595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c106630 msgr2=0x7f5f0c106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.191+0000 7f5f12129640 1 --2- 172.21.15.28:0/2241372595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c106630 0x7f5f0c106a10 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009930 tx=0x7f5efc02a420 comp rx=0 tx=0).stop 2024-04-25T01:38:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.191+0000 7f5f12129640 1 -- 172.21.15.28:0/2241372595 shutdown_connections 2024-04-25T01:38:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.191+0000 7f5f12129640 1 --2- 172.21.15.28:0/2241372595 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f0c100fe0 0x7f5f0c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.191+0000 7f5f12129640 1 --2- 172.21.15.28:0/2241372595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f0c100620 0x7f5f0c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.191+0000 7f5f12129640 1 --2- 172.21.15.28:0/2241372595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c106630 0x7f5f0c106a10 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.191+0000 7f5f12129640 1 -- 172.21.15.28:0/2241372595 >> 172.21.15.28:0/2241372595 conn(0x7f5f0c0776a0 msgr2=0x7f5f0c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.191+0000 7f5f12129640 1 -- 172.21.15.28:0/2241372595 shutdown_connections 2024-04-25T01:38:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.191+0000 7f5f12129640 1 -- 172.21.15.28:0/2241372595 wait complete. 2024-04-25T01:38:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.191+0000 7f5f12129640 1 Processor -- start 2024-04-25T01:38:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.191+0000 7f5f12129640 1 -- start start 2024-04-25T01:38:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f12129640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f0c100620 0x7f5f0c197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f12129640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c100fe0 0x7f5f0c1984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f12129640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f0c106630 0x7f5f0c19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f12129640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c110a90 con 0x7f5f0c100fe0 2024-04-25T01:38:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f12129640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c110910 con 0x7f5f0c106630 2024-04-25T01:38:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f12129640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c110c10 con 0x7f5f0c100620 2024-04-25T01:38:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f0affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c100fe0 0x7f5f0c1984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f0affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c100fe0 0x7f5f0c1984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45512/0 (socket says 172.21.15.28:45512) 2024-04-25T01:38:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f0affd640 1 -- 172.21.15.28:0/4211014551 learned_addr learned my addr 172.21.15.28:0/4211014551 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f0b7fe640 1 --2- 172.21.15.28:0/4211014551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f0c100620 0x7f5f0c197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f0bfff640 1 --2- 172.21.15.28:0/4211014551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f0c106630 0x7f5f0c19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f0affd640 1 -- 172.21.15.28:0/4211014551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f0c100620 msgr2=0x7f5f0c197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f0affd640 1 --2- 172.21.15.28:0/4211014551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f0c100620 0x7f5f0c197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f0affd640 1 -- 172.21.15.28:0/4211014551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f0c106630 msgr2=0x7f5f0c19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f0affd640 1 --2- 172.21.15.28:0/4211014551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f0c106630 0x7f5f0c19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f0affd640 1 -- 172.21.15.28:0/4211014551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f0c19cfa0 con 0x7f5f0c100fe0 2024-04-25T01:38:38.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f0bfff640 1 --2- 172.21.15.28:0/4211014551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f0c106630 0x7f5f0c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:38.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f0b7fe640 1 --2- 172.21.15.28:0/4211014551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f0c100620 0x7f5f0c197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:38.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f0affd640 1 --2- 172.21.15.28:0/4211014551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c100fe0 0x7f5f0c1984f0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f5f0000c920 tx=0x7f5f0000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f08ff9640 1 -- 172.21.15.28:0/4211014551 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f00007c70 con 0x7f5f0c100fe0 2024-04-25T01:38:38.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f12129640 1 -- 172.21.15.28:0/4211014551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f0c19d290 con 0x7f5f0c100fe0 2024-04-25T01:38:38.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f08ff9640 1 -- 172.21.15.28:0/4211014551 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f00007e10 con 0x7f5f0c100fe0 2024-04-25T01:38:38.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.192+0000 7f5f08ff9640 1 -- 172.21.15.28:0/4211014551 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f00018890 con 0x7f5f0c100fe0 2024-04-25T01:38:38.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.193+0000 7f5f12129640 1 -- 172.21.15.28:0/4211014551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f0c1a4b20 con 0x7f5f0c100fe0 2024-04-25T01:38:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.195+0000 7f5f08ff9640 1 -- 172.21.15.28:0/4211014551 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f5f00018a70 con 0x7f5f0c100fe0 2024-04-25T01:38:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.195+0000 7f5f14e39740 1 Processor -- start 2024-04-25T01:38:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.195+0000 7f5f14e39740 1 -- start start 2024-04-25T01:38:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.195+0000 7f5f14e39740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2f360 0x55bd18f2f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.195+0000 7f5f14e39740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f2fd10 0x55bd18f3e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.195+0000 7f5f14e39740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f3ea30 0x55bd18f40e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.195+0000 7f5f14e39740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bd18e946b0 con 0x55bd18f2f360 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.195+0000 7f5f14e39740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55bd18e3c880 con 0x55bd18f2fd10 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.195+0000 7f5f14e39740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55bd18e90670 con 0x55bd18f3ea30 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.195+0000 7f5f08ff9640 1 --2- 172.21.15.28:0/4211014551 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f5ef40a2d00 0x7f5ef40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.195+0000 7f5f0bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2f360 0x55bd18f2f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.195+0000 7f5f0bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2f360 0x55bd18f2f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45520/0 (socket says 172.21.15.28:45520) 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.196+0000 7f5f0bfff640 1 -- 172.21.15.28:0/1535934598 learned_addr learned my addr 172.21.15.28:0/1535934598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.196+0000 7f5f0bfff640 1 --2- 172.21.15.28:0/4211014551 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f5ef40a2d00 0x7f5ef40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.196+0000 7f5f0b7fe640 1 --2- 172.21.15.28:0/1535934598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f2fd10 0x55bd18f3e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.196+0000 7f5f08ff9640 1 -- 172.21.15.28:0/4211014551 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f5f000cf990 con 0x7f5f0c100fe0 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.196+0000 7f5f0affd640 1 --2- 172.21.15.28:0/1535934598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f3ea30 0x55bd18f40e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.196+0000 7f5f0bfff640 1 -- 172.21.15.28:0/1535934598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f3ea30 msgr2=0x55bd18f40e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.196+0000 7f5f0bfff640 1 --2- 172.21.15.28:0/1535934598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f3ea30 0x55bd18f40e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.196+0000 7f5f0bfff640 1 -- 172.21.15.28:0/1535934598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f2fd10 msgr2=0x55bd18f3e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.196+0000 7f5f0bfff640 1 --2- 172.21.15.28:0/1535934598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f2fd10 0x55bd18f3e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.196+0000 7f5f0bfff640 1 -- 172.21.15.28:0/1535934598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd18c727d0 con 0x55bd18f2f360 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.196+0000 7f5f0b7fe640 1 --2- 172.21.15.28:0/1535934598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f2fd10 0x55bd18f3e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.196+0000 7f5f0bfff640 1 --2- 172.21.15.28:0/1535934598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2f360 0x55bd18f2f740 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f5f040092f0 tx=0x7f5f04011750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef49382ba323277 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.196+0000 7f5f0bfff640 1 --2- 172.21.15.28:0/4211014551 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f5ef40a2d00 0x7f5ef40a51c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5f04011c50 tx=0x7f5f04016000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.196+0000 7f5ef1ffb640 1 -- 172.21.15.28:0/1535934598 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f04022070 con 0x55bd18f2f360 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.196+0000 7f5ef1ffb640 1 -- 172.21.15.28:0/1535934598 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f04009bc0 con 0x55bd18f2f360 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.196+0000 7f5ef1ffb640 1 -- 172.21.15.28:0/1535934598 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f04005400 con 0x55bd18f2f360 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.197+0000 7f5f14e39740 1 -- 172.21.15.28:0/1535934598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2f360 msgr2=0x55bd18f2f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.197+0000 7f5f14e39740 1 --2- 172.21.15.28:0/1535934598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2f360 0x55bd18f2f740 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f5f040092f0 tx=0x7f5f04011750 comp rx=0 tx=0).stop 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.197+0000 7f5f14e39740 1 -- 172.21.15.28:0/1535934598 shutdown_connections 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.197+0000 7f5f14e39740 1 --2- 172.21.15.28:0/1535934598 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f3ea30 0x55bd18f40e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.197+0000 7f5f14e39740 1 --2- 172.21.15.28:0/1535934598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f2fd10 0x55bd18f3e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.197+0000 7f5f14e39740 1 --2- 172.21.15.28:0/1535934598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2f360 0x55bd18f2f740 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.197+0000 7f5f14e39740 1 -- 172.21.15.28:0/1535934598 >> 172.21.15.28:0/1535934598 conn(0x55bd18f2b950 msgr2=0x55bd18f2bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.197+0000 7f5f14e39740 1 -- 172.21.15.28:0/1535934598 shutdown_connections 2024-04-25T01:38:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.197+0000 7f5f14e39740 1 -- 172.21.15.28:0/1535934598 wait complete. 2024-04-25T01:38:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.197+0000 7f5f14e39740 1 -- start start 2024-04-25T01:38:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.197+0000 7f5f14e39740 1 Processor -- start 2024-04-25T01:38:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5f14e39740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f2f360 0x55bd18f3ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5f14e39740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2fd10 0x55bd18f3bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5f14e39740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f3ea30 0x55bd18f3c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5f14e39740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bd18f41790 con 0x55bd18f2fd10 2024-04-25T01:38:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5f14e39740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55bd18e4ff50 con 0x55bd18f2f360 2024-04-25T01:38:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5f14e39740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55bd18f41910 con 0x55bd18f3ea30 2024-04-25T01:38:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5f14e39740 10 client.0 _ll_register_callbacks cb 0x55bd18e8b000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5f0affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2fd10 0x55bd18f3bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5f0affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2fd10 0x55bd18f3bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45536/0 (socket says 172.21.15.28:45536) 2024-04-25T01:38:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5f0affd640 1 -- 172.21.15.28:0/1032145705 learned_addr learned my addr 172.21.15.28:0/1032145705 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5f0bfff640 1 --2- 172.21.15.28:0/1032145705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f3ea30 0x55bd18f3c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5f0b7fe640 1 --2- 172.21.15.28:0/1032145705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f2f360 0x55bd18f3ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5f0affd640 1 -- 172.21.15.28:0/1032145705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f3ea30 msgr2=0x55bd18f3c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5f0affd640 1 --2- 172.21.15.28:0/1032145705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f3ea30 0x55bd18f3c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5f0affd640 1 -- 172.21.15.28:0/1032145705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f2f360 msgr2=0x55bd18f3ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5f0affd640 1 --2- 172.21.15.28:0/1032145705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f2f360 0x55bd18f3ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5f0affd640 1 -- 172.21.15.28:0/1032145705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd18ba83f0 con 0x55bd18f2fd10 2024-04-25T01:38:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5f0affd640 1 --2- 172.21.15.28:0/1032145705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2fd10 0x55bd18f3bf50 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f5f0009b870 tx=0x7f5f0009b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b00dae95e467a00 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5eeaffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5eeaffd640 1 -- 172.21.15.28:0/1032145705 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f00064030 con 0x55bd18f2fd10 2024-04-25T01:38:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.198+0000 7f5eeaffd640 1 -- 172.21.15.28:0/1032145705 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f5f0000fb30 con 0x55bd18f2fd10 2024-04-25T01:38:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.199+0000 7f5f14e39740 1 -- 172.21.15.28:0/1032145705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f04009bc0 con 0x55bd18f2fd10 2024-04-25T01:38:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.199+0000 7f5eeaffd640 1 -- 172.21.15.28:0/1032145705 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5f0000de20 con 0x55bd18f2fd10 2024-04-25T01:38:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.199+0000 7f5f14e39740 10 client.17424 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.199+0000 7f5f14e39740 1 -- 172.21.15.28:0/1032145705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bd18b680f0 con 0x55bd18f2fd10 2024-04-25T01:38:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.199+0000 7f5eeaffd640 1 -- 172.21.15.28:0/1032145705 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f04009bc0 con 0x55bd18f2fd10 2024-04-25T01:38:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.199+0000 7f5eeaffd640 1 -- 172.21.15.28:0/1032145705 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5f000981e0 con 0x55bd18f2fd10 2024-04-25T01:38:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.199+0000 7f5f14e39740 10 client.17424 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.199+0000 7f5f14e39740 10 client.17424 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.199+0000 7f5f14e39740 20 client.17424 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.199+0000 7f5f14e39740 1 --2- 172.21.15.28:0/1032145705 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bd18f4ed20 0x55bd18f51110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.199+0000 7f5f14e39740 4 client.17424 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.199+0000 7f5f14e39740 1 -- 172.21.15.28:0/1032145705 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd18ea1bf0 con 0x55bd18f4ed20 2024-04-25T01:38:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.199+0000 7f5f0b7fe640 1 --2- 172.21.15.28:0/1032145705 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bd18f4ed20 0x55bd18f51110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.200+0000 7f5f0b7fe640 1 --2- 172.21.15.28:0/1032145705 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bd18f4ed20 0x55bd18f51110 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=304300f03cd9ca2d server_cookie=c6c12d4ece09a08d in_seq=0 out_seq=0 2024-04-25T01:38:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.200+0000 7f5eeaffd640 10 client.17424 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.200+0000 7f5eeaffd640 1 -- 172.21.15.28:0/1032145705 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bd18ea1bf0 con 0x55bd18f4ed20 2024-04-25T01:38:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.200+0000 7f5eeaffd640 10 client.17424 handle_command_reply: tid=0 2024-04-25T01:38:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.200+0000 7f5f14e39740 1 client.17424 shutdown 2024-04-25T01:38:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.201+0000 7f5f14e39740 1 -- 172.21.15.28:0/1032145705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2fd10 msgr2=0x55bd18f3bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.201+0000 7f5f14e39740 1 --2- 172.21.15.28:0/1032145705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2fd10 0x55bd18f3bf50 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f5f0009b870 tx=0x7f5f0009b7a0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.201+0000 7f5f14e39740 1 -- 172.21.15.28:0/1032145705 shutdown_connections 2024-04-25T01:38:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.201+0000 7f5f14e39740 1 --2- 172.21.15.28:0/1032145705 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bd18f4ed20 0x55bd18f51110 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.201+0000 7f5f14e39740 1 --2- 172.21.15.28:0/1032145705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f3ea30 0x55bd18f3c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.201+0000 7f5f14e39740 1 --2- 172.21.15.28:0/1032145705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2fd10 0x55bd18f3bf50 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.201+0000 7f5eeaffd640 0 client.17424 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.201+0000 7f5f14e39740 1 --2- 172.21.15.28:0/1032145705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f2f360 0x55bd18f3ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.201+0000 7f5f14e39740 1 -- 172.21.15.28:0/1032145705 >> 172.21.15.28:0/1032145705 conn(0x55bd18f2b950 msgr2=0x55bd18f317c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.201+0000 7f5f14e39740 1 -- 172.21.15.28:0/1032145705 shutdown_connections 2024-04-25T01:38:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.201+0000 7f5f14e39740 1 -- 172.21.15.28:0/1032145705 wait complete. 2024-04-25T01:38:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.201+0000 7f5f14e39740 20 client.17424 trim_cache size 0 max 16384 2024-04-25T01:38:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.218+0000 7f5f14e39740 1 Processor -- start 2024-04-25T01:38:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.218+0000 7f5f14e39740 1 -- start start 2024-04-25T01:38:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.218+0000 7f5f14e39740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f2f360 0x55bd18f3a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.218+0000 7f5f14e39740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f2fd10 0x55bd18f4a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.218+0000 7f5f14e39740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f3ea30 0x55bd18f36c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.218+0000 7f5f14e39740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c1ac780 con 0x55bd18f3ea30 2024-04-25T01:38:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.218+0000 7f5f14e39740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c1ae2c0 con 0x55bd18f2f360 2024-04-25T01:38:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.218+0000 7f5f14e39740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c1ace50 con 0x55bd18f2fd10 2024-04-25T01:38:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.219+0000 7f5f0bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f3ea30 0x55bd18f36c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.219+0000 7f5f0bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f3ea30 0x55bd18f36c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45548/0 (socket says 172.21.15.28:45548) 2024-04-25T01:38:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.219+0000 7f5f0bfff640 1 -- 172.21.15.28:0/907723090 learned_addr learned my addr 172.21.15.28:0/907723090 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.219+0000 7f5f0b7fe640 1 --2- 172.21.15.28:0/907723090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f2f360 0x55bd18f3a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.219+0000 7f5f0affd640 1 --2- 172.21.15.28:0/907723090 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f2fd10 0x55bd18f4a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.219+0000 7f5f0bfff640 1 -- 172.21.15.28:0/907723090 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f2fd10 msgr2=0x55bd18f4a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.219+0000 7f5f0bfff640 1 --2- 172.21.15.28:0/907723090 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f2fd10 0x55bd18f4a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.219+0000 7f5f0bfff640 1 -- 172.21.15.28:0/907723090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f2f360 msgr2=0x55bd18f3a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.219+0000 7f5f0bfff640 1 --2- 172.21.15.28:0/907723090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f2f360 0x55bd18f3a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.219+0000 7f5f0bfff640 1 -- 172.21.15.28:0/907723090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd18f2fab0 con 0x55bd18f3ea30 2024-04-25T01:38:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.219+0000 7f5f0affd640 1 --2- 172.21.15.28:0/907723090 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f2fd10 0x55bd18f4a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.219+0000 7f5f0bfff640 1 --2- 172.21.15.28:0/907723090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f3ea30 0x55bd18f36c70 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f5f04009240 tx=0x7f5f04020040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a2428212efd2b10 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.219+0000 7f5ef17fa640 1 -- 172.21.15.28:0/907723090 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f04022040 con 0x55bd18f3ea30 2024-04-25T01:38:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.219+0000 7f5ef17fa640 1 -- 172.21.15.28:0/907723090 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f04004ee0 con 0x55bd18f3ea30 2024-04-25T01:38:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.219+0000 7f5ef17fa640 1 -- 172.21.15.28:0/907723090 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f04005130 con 0x55bd18f3ea30 2024-04-25T01:38:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.220+0000 7f5f14e39740 1 -- 172.21.15.28:0/907723090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f3ea30 msgr2=0x55bd18f36c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.220+0000 7f5f14e39740 1 --2- 172.21.15.28:0/907723090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f3ea30 0x55bd18f36c70 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f5f04009240 tx=0x7f5f04020040 comp rx=0 tx=0).stop 2024-04-25T01:38:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.220+0000 7f5f14e39740 1 -- 172.21.15.28:0/907723090 shutdown_connections 2024-04-25T01:38:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.220+0000 7f5f14e39740 1 --2- 172.21.15.28:0/907723090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f3ea30 0x55bd18f36c70 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.220+0000 7f5f14e39740 1 --2- 172.21.15.28:0/907723090 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f2fd10 0x55bd18f4a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.220+0000 7f5f14e39740 1 --2- 172.21.15.28:0/907723090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f2f360 0x55bd18f3a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.220+0000 7f5f14e39740 1 -- 172.21.15.28:0/907723090 >> 172.21.15.28:0/907723090 conn(0x55bd18f2b950 msgr2=0x55bd18f2aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.220+0000 7f5f14e39740 1 -- 172.21.15.28:0/907723090 shutdown_connections 2024-04-25T01:38:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.220+0000 7f5f14e39740 1 -- 172.21.15.28:0/907723090 wait complete. 2024-04-25T01:38:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.220+0000 7f5f14e39740 1 -- start start 2024-04-25T01:38:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.220+0000 7f5f14e39740 1 Processor -- start 2024-04-25T01:38:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.220+0000 7f5f14e39740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f2f360 0x55bd18f42740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.220+0000 7f5f14e39740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2fd10 0x55bd18f42c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.220+0000 7f5f14e39740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f3ea30 0x55bd18f431c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5f14e39740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c1adbf0 con 0x55bd18f2fd10 2024-04-25T01:38:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5f14e39740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c1ad520 con 0x55bd18f3ea30 2024-04-25T01:38:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5f14e39740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c1aa760 con 0x55bd18f2f360 2024-04-25T01:38:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5f14e39740 10 client.0 _ll_register_callbacks cb 0x7f5f0000c870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5f0affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2fd10 0x55bd18f42c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5f0affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2fd10 0x55bd18f42c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45562/0 (socket says 172.21.15.28:45562) 2024-04-25T01:38:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5f0affd640 1 -- 172.21.15.28:0/2153988994 learned_addr learned my addr 172.21.15.28:0/2153988994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5f0bfff640 1 --2- 172.21.15.28:0/2153988994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f3ea30 0x55bd18f431c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5f0affd640 1 -- 172.21.15.28:0/2153988994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f2f360 msgr2=0x55bd18f42740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5f0affd640 1 --2- 172.21.15.28:0/2153988994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f2f360 0x55bd18f42740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5f0affd640 1 -- 172.21.15.28:0/2153988994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f3ea30 msgr2=0x55bd18f431c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5f0affd640 1 --2- 172.21.15.28:0/2153988994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f3ea30 0x55bd18f431c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5f0affd640 1 -- 172.21.15.28:0/2153988994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd18fb43b0 con 0x55bd18f2fd10 2024-04-25T01:38:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5f0bfff640 1 --2- 172.21.15.28:0/2153988994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f3ea30 0x55bd18f431c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5f0affd640 1 --2- 172.21.15.28:0/2153988994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2fd10 0x55bd18f42c80 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f5f00098540 tx=0x7f5f0000fe30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=584b8d1e2f0ec77c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5eeaffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5eeaffd640 1 -- 172.21.15.28:0/2153988994 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f00064070 con 0x55bd18f2fd10 2024-04-25T01:38:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5eeaffd640 1 -- 172.21.15.28:0/2153988994 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f5f000a4350 con 0x55bd18f2fd10 2024-04-25T01:38:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5f14e39740 1 -- 172.21.15.28:0/2153988994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f04004ee0 con 0x55bd18f2fd10 2024-04-25T01:38:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5eeaffd640 1 -- 172.21.15.28:0/2153988994 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5f0009b940 con 0x55bd18f2fd10 2024-04-25T01:38:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5f14e39740 10 client.17430 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.221+0000 7f5f14e39740 1 -- 172.21.15.28:0/2153988994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bd18fb55f0 con 0x55bd18f2fd10 2024-04-25T01:38:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.222+0000 7f5eeaffd640 1 -- 172.21.15.28:0/2153988994 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f04004ee0 con 0x55bd18f2fd10 2024-04-25T01:38:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.222+0000 7f5eeaffd640 1 -- 172.21.15.28:0/2153988994 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5f0009afc0 con 0x55bd18f2fd10 2024-04-25T01:38:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.222+0000 7f5f14e39740 10 client.17430 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.222+0000 7f5f14e39740 10 client.17430 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.222+0000 7f5f14e39740 20 client.17430 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.222+0000 7f5f14e39740 1 --2- 172.21.15.28:0/2153988994 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bd18fb5b50 0x55bd18fb7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.222+0000 7f5f14e39740 4 client.17430 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:38:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.222+0000 7f5f14e39740 1 -- 172.21.15.28:0/2153988994 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bd18e9bf40 con 0x55bd18fb5b50 2024-04-25T01:38:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.222+0000 7f5f0b7fe640 1 --2- 172.21.15.28:0/2153988994 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bd18fb5b50 0x55bd18fb7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.222+0000 7f5eeaffd640 10 client.17430 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.222+0000 7f5f0b7fe640 1 --2- 172.21.15.28:0/2153988994 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bd18fb5b50 0x55bd18fb7f70 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bdbc861251345eb server_cookie=51121c5833be9331 in_seq=0 out_seq=0 2024-04-25T01:38:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5eeaffd640 1 -- 172.21.15.28:0/2153988994 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bd18e9bf40 con 0x55bd18fb5b50 2024-04-25T01:38:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5eeaffd640 10 client.17430 handle_command_reply: tid=0 2024-04-25T01:38:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f14e39740 1 client.17430 shutdown 2024-04-25T01:38:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f14e39740 1 -- 172.21.15.28:0/2153988994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2fd10 msgr2=0x55bd18f42c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f14e39740 1 --2- 172.21.15.28:0/2153988994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2fd10 0x55bd18f42c80 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f5f00098540 tx=0x7f5f0000fe30 comp rx=0 tx=0).stop 2024-04-25T01:38:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f14e39740 1 -- 172.21.15.28:0/2153988994 shutdown_connections 2024-04-25T01:38:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f14e39740 1 --2- 172.21.15.28:0/2153988994 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bd18fb5b50 0x55bd18fb7f70 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f14e39740 1 --2- 172.21.15.28:0/2153988994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bd18f3ea30 0x55bd18f431c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f14e39740 1 --2- 172.21.15.28:0/2153988994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd18f2fd10 0x55bd18f42c80 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5eeaffd640 0 client.17430 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f14e39740 1 --2- 172.21.15.28:0/2153988994 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bd18f2f360 0x55bd18f42740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f14e39740 1 -- 172.21.15.28:0/2153988994 >> 172.21.15.28:0/2153988994 conn(0x55bd18f2b950 msgr2=0x55bd18f3d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f14e39740 1 -- 172.21.15.28:0/2153988994 shutdown_connections 2024-04-25T01:38:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f14e39740 1 -- 172.21.15.28:0/2153988994 wait complete. 2024-04-25T01:38:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f14e39740 20 client.17430 trim_cache size 0 max 16384 2024-04-25T01:38:38.210 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:38:38.210 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:38:38.210 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:38:38.211 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:38:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f12129640 1 -- 172.21.15.28:0/4211014551 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f5ef40a2d00 msgr2=0x7f5ef40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f12129640 1 --2- 172.21.15.28:0/4211014551 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f5ef40a2d00 0x7f5ef40a51c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5f04011c50 tx=0x7f5f04016000 comp rx=0 tx=0).stop 2024-04-25T01:38:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f12129640 1 -- 172.21.15.28:0/4211014551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c100fe0 msgr2=0x7f5f0c1984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f12129640 1 --2- 172.21.15.28:0/4211014551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c100fe0 0x7f5f0c1984f0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f5f0000c920 tx=0x7f5f0000cde0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f12129640 1 -- 172.21.15.28:0/4211014551 shutdown_connections 2024-04-25T01:38:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f12129640 1 --2- 172.21.15.28:0/4211014551 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f5ef40a2d00 0x7f5ef40a51c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f12129640 1 --2- 172.21.15.28:0/4211014551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5f0c106630 0x7f5f0c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f12129640 1 --2- 172.21.15.28:0/4211014551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c100fe0 0x7f5f0c1984f0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.223+0000 7f5f12129640 1 --2- 172.21.15.28:0/4211014551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5f0c100620 0x7f5f0c197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.224+0000 7f5f12129640 1 -- 172.21.15.28:0/4211014551 >> 172.21.15.28:0/4211014551 conn(0x7f5f0c0776a0 msgr2=0x7f5f0c069f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.224+0000 7f5f12129640 1 -- 172.21.15.28:0/4211014551 shutdown_connections 2024-04-25T01:38:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.224+0000 7f5f12129640 1 -- 172.21.15.28:0/4211014551 wait complete. 2024-04-25T01:38:38.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:37 smithi130 ceph-mon[32702]: pgmap v171: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:38:38.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:37 smithi028 ceph-mon[29390]: pgmap v171: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:38:38.215 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:38:38.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:38:38.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:37 smithi032 ceph-mon[32461]: pgmap v171: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:38:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.336+0000 7f8de924d640 1 -- 172.21.15.28:0/2064691217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8de4107750 msgr2=0x7f8de4107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.336+0000 7f8de924d640 1 --2- 172.21.15.28:0/2064691217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8de4107750 0x7f8de4107b30 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4009900 tx=0x7f8dd402a480 comp rx=0 tx=0).stop 2024-04-25T01:38:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.336+0000 7f8de924d640 1 -- 172.21.15.28:0/2064691217 shutdown_connections 2024-04-25T01:38:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.336+0000 7f8de924d640 1 --2- 172.21.15.28:0/2064691217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8de4102160 0x7f8de410e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.336+0000 7f8de924d640 1 --2- 172.21.15.28:0/2064691217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de41017c0 0x7f8de4101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.336+0000 7f8de924d640 1 --2- 172.21.15.28:0/2064691217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8de4107750 0x7f8de4107b30 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.336+0000 7f8de924d640 1 -- 172.21.15.28:0/2064691217 >> 172.21.15.28:0/2064691217 conn(0x7f8de40788d0 msgr2=0x7f8de4078ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.336+0000 7f8de924d640 1 -- 172.21.15.28:0/2064691217 shutdown_connections 2024-04-25T01:38:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.336+0000 7f8de924d640 1 -- 172.21.15.28:0/2064691217 wait complete. 2024-04-25T01:38:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.336+0000 7f8de924d640 1 Processor -- start 2024-04-25T01:38:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.337+0000 7f8de924d640 1 -- start start 2024-04-25T01:38:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.337+0000 7f8de924d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8de41017c0 0x7f8de4197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.337+0000 7f8de924d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de4102160 0x7f8de4198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.337+0000 7f8de924d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8de4107750 0x7f8de419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.337+0000 7f8de924d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8de4111b10 con 0x7f8de4107750 2024-04-25T01:38:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.337+0000 7f8de924d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8de4111990 con 0x7f8de4102160 2024-04-25T01:38:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.337+0000 7f8de924d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8de4111c90 con 0x7f8de41017c0 2024-04-25T01:38:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.337+0000 7f8de359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8de4107750 0x7f8de419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.337+0000 7f8de359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8de4107750 0x7f8de419c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45588/0 (socket says 172.21.15.28:45588) 2024-04-25T01:38:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.337+0000 7f8de359e640 1 -- 172.21.15.28:0/3743771856 learned_addr learned my addr 172.21.15.28:0/3743771856 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.337+0000 7f8de259c640 1 --2- 172.21.15.28:0/3743771856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de4102160 0x7f8de4198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.337+0000 7f8de2d9d640 1 --2- 172.21.15.28:0/3743771856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8de41017c0 0x7f8de4197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.337+0000 7f8de359e640 1 -- 172.21.15.28:0/3743771856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8de41017c0 msgr2=0x7f8de4197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.337+0000 7f8de359e640 1 --2- 172.21.15.28:0/3743771856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8de41017c0 0x7f8de4197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.337+0000 7f8de359e640 1 -- 172.21.15.28:0/3743771856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de4102160 msgr2=0x7f8de4198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.337+0000 7f8de359e640 1 --2- 172.21.15.28:0/3743771856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de4102160 0x7f8de4198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.337+0000 7f8de359e640 1 -- 172.21.15.28:0/3743771856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de419cfe0 con 0x7f8de4107750 2024-04-25T01:38:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.337+0000 7f8de259c640 1 --2- 172.21.15.28:0/3743771856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de4102160 0x7f8de4198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.337+0000 7f8de359e640 1 --2- 172.21.15.28:0/3743771856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8de4107750 0x7f8de419c8b0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc00eaa0 tx=0x7f8ddc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.338+0000 7f8dcbfff640 1 -- 172.21.15.28:0/3743771856 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc00cdb0 con 0x7f8de4107750 2024-04-25T01:38:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.338+0000 7f8dcbfff640 1 -- 172.21.15.28:0/3743771856 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ddc005020 con 0x7f8de4107750 2024-04-25T01:38:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.338+0000 7f8de924d640 1 -- 172.21.15.28:0/3743771856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8de419d2d0 con 0x7f8de4107750 2024-04-25T01:38:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.338+0000 7f8dcbfff640 1 -- 172.21.15.28:0/3743771856 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc0116b0 con 0x7f8de4107750 2024-04-25T01:38:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.338+0000 7f8de924d640 1 -- 172.21.15.28:0/3743771856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8de41a4b60 con 0x7f8de4107750 2024-04-25T01:38:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.339+0000 7f8de924d640 1 -- 172.21.15.28:0/3743771856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8de40689b0 con 0x7f8de4107750 2024-04-25T01:38:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.340+0000 7f8dcbfff640 1 -- 172.21.15.28:0/3743771856 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8ddc005560 con 0x7f8de4107750 2024-04-25T01:38:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.341+0000 7f8dcbfff640 1 --2- 172.21.15.28:0/3743771856 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8dcc0a2d00 0x7f8dcc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.341+0000 7f8de2d9d640 1 --2- 172.21.15.28:0/3743771856 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8dcc0a2d00 0x7f8dcc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.341+0000 7f8dcbfff640 1 -- 172.21.15.28:0/3743771856 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8ddc0cfa20 con 0x7f8de4107750 2024-04-25T01:38:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.341+0000 7f8de2d9d640 1 --2- 172.21.15.28:0/3743771856 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8dcc0a2d00 0x7f8dcc0a51c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40096c0 tx=0x7f8dd4000bc0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.345+0000 7f8dcbfff640 1 -- 172.21.15.28:0/3743771856 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8ddc016030 con 0x7f8de4107750 2024-04-25T01:38:38.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.570+0000 7f8de924d640 1 -- 172.21.15.28:0/3743771856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8de402d170 con 0x7f8de4107750 2024-04-25T01:38:38.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.571+0000 7f8dcbfff640 1 -- 172.21.15.28:0/3743771856 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7f8ddc09c300 con 0x7f8de4107750 2024-04-25T01:38:38.552 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:38:38.552 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:38:38.552 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:38:38.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.573+0000 7f8de924d640 1 -- 172.21.15.28:0/3743771856 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8dcc0a2d00 msgr2=0x7f8dcc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.573+0000 7f8de924d640 1 --2- 172.21.15.28:0/3743771856 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8dcc0a2d00 0x7f8dcc0a51c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40096c0 tx=0x7f8dd4000bc0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.573+0000 7f8de924d640 1 -- 172.21.15.28:0/3743771856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8de4107750 msgr2=0x7f8de419c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.573+0000 7f8de924d640 1 --2- 172.21.15.28:0/3743771856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8de4107750 0x7f8de419c8b0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc00eaa0 tx=0x7f8ddc00ef60 comp rx=0 tx=0).stop 2024-04-25T01:38:38.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.573+0000 7f8de924d640 1 -- 172.21.15.28:0/3743771856 shutdown_connections 2024-04-25T01:38:38.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.573+0000 7f8de924d640 1 --2- 172.21.15.28:0/3743771856 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8dcc0a2d00 0x7f8dcc0a51c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.573+0000 7f8de924d640 1 --2- 172.21.15.28:0/3743771856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8de4107750 0x7f8de419c8b0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.573+0000 7f8de924d640 1 --2- 172.21.15.28:0/3743771856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8de4102160 0x7f8de4198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.573+0000 7f8de924d640 1 --2- 172.21.15.28:0/3743771856 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8de41017c0 0x7f8de4197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.573+0000 7f8de924d640 1 -- 172.21.15.28:0/3743771856 >> 172.21.15.28:0/3743771856 conn(0x7f8de40788d0 msgr2=0x7f8de41003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.573+0000 7f8de924d640 1 -- 172.21.15.28:0/3743771856 shutdown_connections 2024-04-25T01:38:38.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.573+0000 7f8de924d640 1 -- 172.21.15.28:0/3743771856 wait complete. 2024-04-25T01:38:38.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:38:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.690+0000 7f5866d70640 1 -- 172.21.15.28:0/971218312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58601002e0 msgr2=0x7f58601006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.690+0000 7f5866d70640 1 --2- 172.21.15.28:0/971218312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58601002e0 0x7f58601006c0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f5850009900 tx=0x7f585002a410 comp rx=0 tx=0).stop 2024-04-25T01:38:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.690+0000 7f5866d70640 1 -- 172.21.15.28:0/971218312 shutdown_connections 2024-04-25T01:38:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.690+0000 7f5866d70640 1 --2- 172.21.15.28:0/971218312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f5860106aa0 0x7f586010e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.690+0000 7f5866d70640 1 --2- 172.21.15.28:0/971218312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5860100c90 0x7f58601063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.690+0000 7f5866d70640 1 --2- 172.21.15.28:0/971218312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58601002e0 0x7f58601006c0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.690+0000 7f5866d70640 1 -- 172.21.15.28:0/971218312 >> 172.21.15.28:0/971218312 conn(0x7f58600758d0 msgr2=0x7f5860075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.690+0000 7f5866d70640 1 -- 172.21.15.28:0/971218312 shutdown_connections 2024-04-25T01:38:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.690+0000 7f5866d70640 1 -- 172.21.15.28:0/971218312 wait complete. 2024-04-25T01:38:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.691+0000 7f5866d70640 1 Processor -- start 2024-04-25T01:38:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.691+0000 7f5866d70640 1 -- start start 2024-04-25T01:38:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.691+0000 7f5866d70640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f58601002e0 0x7f5860197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.691+0000 7f5866d70640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5860106aa0 0x7f5860198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.691+0000 7f5866d70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f586019c8b0 0x7f586019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.691+0000 7f5866d70640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5860111b90 con 0x7f586019c8b0 2024-04-25T01:38:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.691+0000 7f5866d70640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5860111a10 con 0x7f5860106aa0 2024-04-25T01:38:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.691+0000 7f5866d70640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f5860111d10 con 0x7f58601002e0 2024-04-25T01:38:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.691+0000 7f586530d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f586019c8b0 0x7f586019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.691+0000 7f586530d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f586019c8b0 0x7f586019cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45592/0 (socket says 172.21.15.28:45592) 2024-04-25T01:38:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.691+0000 7f586530d640 1 -- 172.21.15.28:0/57647864 learned_addr learned my addr 172.21.15.28:0/57647864 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.691+0000 7f5864b0c640 1 --2- 172.21.15.28:0/57647864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f58601002e0 0x7f5860197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.691+0000 7f585ffff640 1 --2- 172.21.15.28:0/57647864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5860106aa0 0x7f5860198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.691+0000 7f586530d640 1 -- 172.21.15.28:0/57647864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f58601002e0 msgr2=0x7f5860197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.691+0000 7f586530d640 1 --2- 172.21.15.28:0/57647864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f58601002e0 0x7f5860197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.691+0000 7f586530d640 1 -- 172.21.15.28:0/57647864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5860106aa0 msgr2=0x7f5860198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.691+0000 7f586530d640 1 --2- 172.21.15.28:0/57647864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5860106aa0 0x7f5860198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.691+0000 7f586530d640 1 -- 172.21.15.28:0/57647864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f586019d3a0 con 0x7f586019c8b0 2024-04-25T01:38:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.692+0000 7f586530d640 1 --2- 172.21.15.28:0/57647864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f586019c8b0 0x7f586019cd60 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f585800cce0 tx=0x7f585800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.692+0000 7f585dffb640 1 -- 172.21.15.28:0/57647864 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585800f450 con 0x7f586019c8b0 2024-04-25T01:38:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.692+0000 7f5866d70640 1 -- 172.21.15.28:0/57647864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f586019d690 con 0x7f586019c8b0 2024-04-25T01:38:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.692+0000 7f585dffb640 1 -- 172.21.15.28:0/57647864 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f585800fad0 con 0x7f586019c8b0 2024-04-25T01:38:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.692+0000 7f585dffb640 1 -- 172.21.15.28:0/57647864 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5858018c10 con 0x7f586019c8b0 2024-04-25T01:38:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.692+0000 7f5866d70640 1 -- 172.21.15.28:0/57647864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58601a4e80 con 0x7f586019c8b0 2024-04-25T01:38:38.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.694+0000 7f585dffb640 1 -- 172.21.15.28:0/57647864 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f585800f5f0 con 0x7f586019c8b0 2024-04-25T01:38:38.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f5869a7d740 1 Processor -- start 2024-04-25T01:38:38.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f5869a7d740 1 -- start start 2024-04-25T01:38:38.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f5869a7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9e580 0x55619da9e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f5869a7d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619da9eea0 0x55619daa9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f5869a7d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55619daa9570 0x55619daab960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f5869a7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55619da03670 con 0x55619da9e580 2024-04-25T01:38:38.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f5869a7d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55619d9ab970 con 0x55619da9eea0 2024-04-25T01:38:38.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f5869a7d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55619d9ff630 con 0x55619daa9570 2024-04-25T01:38:38.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f585dffb640 1 --2- 172.21.15.28:0/57647864 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f584c0a2d00 0x7f584c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f585ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9e580 0x55619da9e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f585ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9e580 0x55619da9e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45608/0 (socket says 172.21.15.28:45608) 2024-04-25T01:38:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f585ffff640 1 -- 172.21.15.28:0/205576764 learned_addr learned my addr 172.21.15.28:0/205576764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f586530d640 1 --2- 172.21.15.28:0/57647864 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f584c0a2d00 0x7f584c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f5864b0c640 1 --2- 172.21.15.28:0/205576764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619da9eea0 0x55619daa9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f585ffff640 1 -- 172.21.15.28:0/205576764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55619daa9570 msgr2=0x55619daab960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f585ffff640 1 --2- 172.21.15.28:0/205576764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55619daa9570 0x55619daab960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f585ffff640 1 -- 172.21.15.28:0/205576764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619da9eea0 msgr2=0x55619daa9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f585ffff640 1 --2- 172.21.15.28:0/205576764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619da9eea0 0x55619daa9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f585ffff640 1 -- 172.21.15.28:0/205576764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55619d9cf400 con 0x55619da9e580 2024-04-25T01:38:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.695+0000 7f585dffb640 1 -- 172.21.15.28:0/57647864 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f58580cfde0 con 0x7f586019c8b0 2024-04-25T01:38:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.696+0000 7f585ffff640 1 --2- 172.21.15.28:0/205576764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9e580 0x55619da9e960 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f584800cca0 tx=0x7f5848010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4291c0b3d3c0219 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.696+0000 7f586530d640 1 --2- 172.21.15.28:0/57647864 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f584c0a2d00 0x7f584c0a51c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f585809bca0 tx=0x7f585809bd70 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.696+0000 7f5856ffd640 1 -- 172.21.15.28:0/205576764 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5848016070 con 0x55619da9e580 2024-04-25T01:38:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.696+0000 7f5856ffd640 1 -- 172.21.15.28:0/205576764 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5848012b60 con 0x55619da9e580 2024-04-25T01:38:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.696+0000 7f5856ffd640 1 -- 172.21.15.28:0/205576764 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5848011c10 con 0x55619da9e580 2024-04-25T01:38:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.696+0000 7f5869a7d740 1 -- 172.21.15.28:0/205576764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9e580 msgr2=0x55619da9e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.696+0000 7f5869a7d740 1 --2- 172.21.15.28:0/205576764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9e580 0x55619da9e960 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f584800cca0 tx=0x7f5848010740 comp rx=0 tx=0).stop 2024-04-25T01:38:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.696+0000 7f5869a7d740 1 -- 172.21.15.28:0/205576764 shutdown_connections 2024-04-25T01:38:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.696+0000 7f5869a7d740 1 --2- 172.21.15.28:0/205576764 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55619daa9570 0x55619daab960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.696+0000 7f5869a7d740 1 --2- 172.21.15.28:0/205576764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619da9eea0 0x55619daa9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.696+0000 7f5869a7d740 1 --2- 172.21.15.28:0/205576764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9e580 0x55619da9e960 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.696+0000 7f5869a7d740 1 -- 172.21.15.28:0/205576764 >> 172.21.15.28:0/205576764 conn(0x55619da9a8a0 msgr2=0x55619da9ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.697+0000 7f5869a7d740 1 -- 172.21.15.28:0/205576764 shutdown_connections 2024-04-25T01:38:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.697+0000 7f5869a7d740 1 -- 172.21.15.28:0/205576764 wait complete. 2024-04-25T01:38:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.697+0000 7f5869a7d740 1 -- start start 2024-04-25T01:38:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.697+0000 7f5869a7d740 1 Processor -- start 2024-04-25T01:38:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.697+0000 7f5869a7d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55619da9e580 0x55619daa8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.697+0000 7f5869a7d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619da9eea0 0x55619daa61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.697+0000 7f5869a7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619daa9570 0x55619daa6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.697+0000 7f5869a7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55619daac360 con 0x55619daa9570 2024-04-25T01:38:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.697+0000 7f5869a7d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55619d9bf050 con 0x55619da9eea0 2024-04-25T01:38:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.697+0000 7f5869a7d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55619daac4e0 con 0x55619da9e580 2024-04-25T01:38:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.697+0000 7f5869a7d740 10 client.0 _ll_register_callbacks cb 0x55619d716be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.697+0000 7f585ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619daa9570 0x55619daa6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.697+0000 7f585ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619daa9570 0x55619daa6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45612/0 (socket says 172.21.15.28:45612) 2024-04-25T01:38:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.698+0000 7f585ffff640 1 -- 172.21.15.28:0/1045441563 learned_addr learned my addr 172.21.15.28:0/1045441563 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.698+0000 7f585ffff640 1 --2- 172.21.15.28:0/1045441563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55619da9e580 0x55619daa8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.698+0000 7f5864b0c640 1 --2- 172.21.15.28:0/1045441563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619da9eea0 0x55619daa61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.698+0000 7f585ffff640 1 -- 172.21.15.28:0/1045441563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55619da9e580 msgr2=0x55619daa8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.698+0000 7f585ffff640 1 --2- 172.21.15.28:0/1045441563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55619da9e580 0x55619daa8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.698+0000 7f585ffff640 1 -- 172.21.15.28:0/1045441563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619da9eea0 msgr2=0x55619daa61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.698+0000 7f585ffff640 1 --2- 172.21.15.28:0/1045441563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619da9eea0 0x55619daa61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.698+0000 7f585ffff640 1 -- 172.21.15.28:0/1045441563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55619d990dc0 con 0x55619daa9570 2024-04-25T01:38:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.698+0000 7f5864b0c640 1 --2- 172.21.15.28:0/1045441563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619da9eea0 0x55619daa61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.698+0000 7f585ffff640 1 --2- 172.21.15.28:0/1045441563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619daa9570 0x55619daa6710 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f5848011fe0 tx=0x7f5848010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed2deb210e9b9ff6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.698+0000 7f583ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.698+0000 7f583ffff640 1 -- 172.21.15.28:0/1045441563 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584801f070 con 0x55619daa9570 2024-04-25T01:38:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.698+0000 7f583ffff640 1 -- 172.21.15.28:0/1045441563 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f5848004050 con 0x55619daa9570 2024-04-25T01:38:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.698+0000 7f5869a7d740 1 -- 172.21.15.28:0/1045441563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5848012b60 con 0x55619daa9570 2024-04-25T01:38:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.698+0000 7f583ffff640 1 -- 172.21.15.28:0/1045441563 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5848030050 con 0x55619daa9570 2024-04-25T01:38:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.698+0000 7f5869a7d740 10 client.17448 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.698+0000 7f5869a7d740 1 -- 172.21.15.28:0/1045441563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55619d7173f0 con 0x55619daa9570 2024-04-25T01:38:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.698+0000 7f583ffff640 1 -- 172.21.15.28:0/1045441563 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5848012b60 con 0x55619daa9570 2024-04-25T01:38:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.699+0000 7f583ffff640 1 -- 172.21.15.28:0/1045441563 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5848003680 con 0x55619daa9570 2024-04-25T01:38:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.699+0000 7f5869a7d740 10 client.17448 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.699+0000 7f5869a7d740 10 client.17448 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:38:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.699+0000 7f5869a7d740 20 client.17448 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.699+0000 7f5869a7d740 1 --2- 172.21.15.28:0/1045441563 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55619dab9590 0x55619dabb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.699+0000 7f5869a7d740 4 client.17448 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.699+0000 7f5869a7d740 1 -- 172.21.15.28:0/1045441563 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55619da10bb0 con 0x55619dab9590 2024-04-25T01:38:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.699+0000 7f5864b0c640 1 --2- 172.21.15.28:0/1045441563 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55619dab9590 0x55619dabb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.699+0000 7f5864b0c640 1 --2- 172.21.15.28:0/1045441563 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55619dab9590 0x55619dabb980 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1ac6139299463e9c server_cookie=fb13c7ec3309a38b in_seq=0 out_seq=0 2024-04-25T01:38:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.699+0000 7f583ffff640 10 client.17448 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:38:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.700+0000 7f583ffff640 1 -- 172.21.15.28:0/1045441563 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55619da10bb0 con 0x55619dab9590 2024-04-25T01:38:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.700+0000 7f583ffff640 10 client.17448 handle_command_reply: tid=0 2024-04-25T01:38:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.700+0000 7f5869a7d740 1 client.17448 shutdown 2024-04-25T01:38:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.700+0000 7f5869a7d740 1 -- 172.21.15.28:0/1045441563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619daa9570 msgr2=0x55619daa6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.700+0000 7f5869a7d740 1 --2- 172.21.15.28:0/1045441563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619daa9570 0x55619daa6710 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f5848011fe0 tx=0x7f5848010f90 comp rx=0 tx=0).stop 2024-04-25T01:38:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.700+0000 7f5869a7d740 1 -- 172.21.15.28:0/1045441563 shutdown_connections 2024-04-25T01:38:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.700+0000 7f5869a7d740 1 --2- 172.21.15.28:0/1045441563 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55619dab9590 0x55619dabb980 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.700+0000 7f5869a7d740 1 --2- 172.21.15.28:0/1045441563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619daa9570 0x55619daa6710 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.700+0000 7f5869a7d740 1 --2- 172.21.15.28:0/1045441563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619da9eea0 0x55619daa61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.700+0000 7f5869a7d740 1 --2- 172.21.15.28:0/1045441563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55619da9e580 0x55619daa8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.700+0000 7f583ffff640 0 client.17448 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:38:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.700+0000 7f5869a7d740 1 -- 172.21.15.28:0/1045441563 >> 172.21.15.28:0/1045441563 conn(0x55619da9a8a0 msgr2=0x55619da99b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.700+0000 7f5869a7d740 1 -- 172.21.15.28:0/1045441563 shutdown_connections 2024-04-25T01:38:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.700+0000 7f5869a7d740 1 -- 172.21.15.28:0/1045441563 wait complete. 2024-04-25T01:38:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.701+0000 7f5869a7d740 20 client.17448 trim_cache size 0 max 16384 2024-04-25T01:38:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.714+0000 7f5869a7d740 1 Processor -- start 2024-04-25T01:38:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.714+0000 7f5869a7d740 1 -- start start 2024-04-25T01:38:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.714+0000 7f5869a7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9e580 0x55619dab5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.714+0000 7f5869a7d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55619da9eea0 0x55619daa89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.714+0000 7f5869a7d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619daa9570 0x55619dac1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.714+0000 7f5869a7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f58601ae9f0 con 0x55619da9e580 2024-04-25T01:38:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.714+0000 7f5869a7d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f58601b0530 con 0x55619daa9570 2024-04-25T01:38:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.714+0000 7f5869a7d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f58601af0c0 con 0x55619da9eea0 2024-04-25T01:38:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.714+0000 7f585ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9e580 0x55619dab5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.714+0000 7f585ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9e580 0x55619dab5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45624/0 (socket says 172.21.15.28:45624) 2024-04-25T01:38:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.714+0000 7f585ffff640 1 -- 172.21.15.28:0/3297425677 learned_addr learned my addr 172.21.15.28:0/3297425677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.714+0000 7f5864b0c640 1 --2- 172.21.15.28:0/3297425677 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55619da9eea0 0x55619daa89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.714+0000 7f585ffff640 1 -- 172.21.15.28:0/3297425677 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55619da9eea0 msgr2=0x55619daa89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.714+0000 7f585ffff640 1 --2- 172.21.15.28:0/3297425677 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55619da9eea0 0x55619daa89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.714+0000 7f585ffff640 1 -- 172.21.15.28:0/3297425677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619daa9570 msgr2=0x55619dac1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.714+0000 7f585ffff640 1 --2- 172.21.15.28:0/3297425677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619daa9570 0x55619dac1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.714+0000 7f585ffff640 1 -- 172.21.15.28:0/3297425677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55619d7e17d0 con 0x55619da9e580 2024-04-25T01:38:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.715+0000 7f585ffff640 1 --2- 172.21.15.28:0/3297425677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9e580 0x55619dab5250 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f5848006010 tx=0x7f584800c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a476b57a495793c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.715+0000 7f58567fc640 1 -- 172.21.15.28:0/3297425677 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584801f070 con 0x55619da9e580 2024-04-25T01:38:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.715+0000 7f58567fc640 1 -- 172.21.15.28:0/3297425677 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5848010c40 con 0x55619da9e580 2024-04-25T01:38:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.715+0000 7f58567fc640 1 -- 172.21.15.28:0/3297425677 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5848016030 con 0x55619da9e580 2024-04-25T01:38:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.715+0000 7f5869a7d740 1 -- 172.21.15.28:0/3297425677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9e580 msgr2=0x55619dab5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.715+0000 7f5869a7d740 1 --2- 172.21.15.28:0/3297425677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9e580 0x55619dab5250 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f5848006010 tx=0x7f584800c450 comp rx=0 tx=0).stop 2024-04-25T01:38:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.715+0000 7f5869a7d740 1 -- 172.21.15.28:0/3297425677 shutdown_connections 2024-04-25T01:38:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.715+0000 7f5869a7d740 1 --2- 172.21.15.28:0/3297425677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619daa9570 0x55619dac1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.715+0000 7f5869a7d740 1 --2- 172.21.15.28:0/3297425677 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55619da9eea0 0x55619daa89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.715+0000 7f5869a7d740 1 --2- 172.21.15.28:0/3297425677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9e580 0x55619dab5250 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.715+0000 7f5869a7d740 1 -- 172.21.15.28:0/3297425677 >> 172.21.15.28:0/3297425677 conn(0x55619da9a8a0 msgr2=0x55619daa0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.715+0000 7f5869a7d740 1 -- 172.21.15.28:0/3297425677 shutdown_connections 2024-04-25T01:38:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.716+0000 7f5869a7d740 1 -- 172.21.15.28:0/3297425677 wait complete. 2024-04-25T01:38:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.716+0000 7f5869a7d740 1 -- start start 2024-04-25T01:38:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.716+0000 7f5869a7d740 1 Processor -- start 2024-04-25T01:38:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.716+0000 7f5869a7d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619da9e580 0x55619dab1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.716+0000 7f5869a7d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9eea0 0x55619daa3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.716+0000 7f5869a7d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55619daa9570 0x55619daa2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.716+0000 7f5869a7d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f58601afe60 con 0x55619da9eea0 2024-04-25T01:38:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.716+0000 7f5869a7d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f58601af790 con 0x55619da9e580 2024-04-25T01:38:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.716+0000 7f5869a7d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f58601ac9d0 con 0x55619daa9570 2024-04-25T01:38:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.716+0000 7f5869a7d740 10 client.0 _ll_register_callbacks cb 0x7f584800cba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.716+0000 7f5864b0c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9eea0 0x55619daa3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.716+0000 7f5864b0c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9eea0 0x55619daa3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45636/0 (socket says 172.21.15.28:45636) 2024-04-25T01:38:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.716+0000 7f5864b0c640 1 -- 172.21.15.28:0/3328875903 learned_addr learned my addr 172.21.15.28:0/3328875903 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.716+0000 7f585ffff640 1 --2- 172.21.15.28:0/3328875903 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55619daa9570 0x55619daa2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f585ffff640 1 --2- 172.21.15.28:0/3328875903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619da9e580 0x55619dab1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f5864b0c640 1 -- 172.21.15.28:0/3328875903 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55619daa9570 msgr2=0x55619daa2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f5864b0c640 1 --2- 172.21.15.28:0/3328875903 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55619daa9570 0x55619daa2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f5864b0c640 1 -- 172.21.15.28:0/3328875903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619da9e580 msgr2=0x55619dab1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f5864b0c640 1 --2- 172.21.15.28:0/3328875903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619da9e580 0x55619dab1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f5864b0c640 1 -- 172.21.15.28:0/3328875903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55619daa2960 con 0x55619da9eea0 2024-04-25T01:38:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f585ffff640 1 --2- 172.21.15.28:0/3328875903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619da9e580 0x55619dab1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f5864b0c640 1 --2- 172.21.15.28:0/3328875903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9eea0 0x55619daa3890 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f585002a930 tx=0x7f5850031a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4bca988c94de9c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f583ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f583ffff640 1 -- 172.21.15.28:0/3328875903 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5850035040 con 0x55619da9eea0 2024-04-25T01:38:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f583ffff640 1 -- 172.21.15.28:0/3328875903 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f5850004040 con 0x55619da9eea0 2024-04-25T01:38:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f5869a7d740 1 -- 172.21.15.28:0/3328875903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5848010c40 con 0x55619da9eea0 2024-04-25T01:38:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f583ffff640 1 -- 172.21.15.28:0/3328875903 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5850054050 con 0x55619da9eea0 2024-04-25T01:38:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f5869a7d740 10 client.17454 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f5869a7d740 1 -- 172.21.15.28:0/3328875903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55619db1efd0 con 0x55619da9eea0 2024-04-25T01:38:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f583ffff640 1 -- 172.21.15.28:0/3328875903 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5848010c40 con 0x55619da9eea0 2024-04-25T01:38:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f583ffff640 1 -- 172.21.15.28:0/3328875903 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5850003680 con 0x55619da9eea0 2024-04-25T01:38:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f5869a7d740 10 client.17454 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f5869a7d740 10 client.17454 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:38:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f5869a7d740 20 client.17454 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f5869a7d740 1 --2- 172.21.15.28:0/3328875903 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55619db1f730 0x55619db21b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f5869a7d740 4 client.17454 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:38:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.717+0000 7f5869a7d740 1 -- 172.21.15.28:0/3328875903 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55619da0af00 con 0x55619db1f730 2024-04-25T01:38:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.718+0000 7f585ffff640 1 --2- 172.21.15.28:0/3328875903 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55619db1f730 0x55619db21b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.718+0000 7f585ffff640 1 --2- 172.21.15.28:0/3328875903 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55619db1f730 0x55619db21b50 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=52b84a8337682ff6 server_cookie=370f8ea3e07de32f in_seq=0 out_seq=0 2024-04-25T01:38:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.718+0000 7f583ffff640 10 client.17454 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:38:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.718+0000 7f583ffff640 1 -- 172.21.15.28:0/3328875903 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55619da0af00 con 0x55619db1f730 2024-04-25T01:38:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.718+0000 7f583ffff640 10 client.17454 handle_command_reply: tid=0 2024-04-25T01:38:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.718+0000 7f5869a7d740 1 client.17454 shutdown 2024-04-25T01:38:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.718+0000 7f5869a7d740 1 -- 172.21.15.28:0/3328875903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9eea0 msgr2=0x55619daa3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.718+0000 7f5869a7d740 1 --2- 172.21.15.28:0/3328875903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9eea0 0x55619daa3890 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f585002a930 tx=0x7f5850031a10 comp rx=0 tx=0).stop 2024-04-25T01:38:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.718+0000 7f5869a7d740 1 -- 172.21.15.28:0/3328875903 shutdown_connections 2024-04-25T01:38:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.718+0000 7f5869a7d740 1 --2- 172.21.15.28:0/3328875903 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55619db1f730 0x55619db21b50 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.718+0000 7f5869a7d740 1 --2- 172.21.15.28:0/3328875903 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55619daa9570 0x55619daa2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.718+0000 7f5869a7d740 1 --2- 172.21.15.28:0/3328875903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55619da9eea0 0x55619daa3890 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.718+0000 7f583ffff640 0 client.17454 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:38:38.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.718+0000 7f5869a7d740 1 --2- 172.21.15.28:0/3328875903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55619da9e580 0x55619dab1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.718+0000 7f5869a7d740 1 -- 172.21.15.28:0/3328875903 >> 172.21.15.28:0/3328875903 conn(0x55619da9a8a0 msgr2=0x55619daa0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.719+0000 7f5869a7d740 1 -- 172.21.15.28:0/3328875903 shutdown_connections 2024-04-25T01:38:38.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.719+0000 7f5869a7d740 1 -- 172.21.15.28:0/3328875903 wait complete. 2024-04-25T01:38:38.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.719+0000 7f5869a7d740 20 client.17454 trim_cache size 0 max 16384 2024-04-25T01:38:38.707 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:38:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.719+0000 7f5866d70640 1 -- 172.21.15.28:0/57647864 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f584c0a2d00 msgr2=0x7f584c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.719+0000 7f5866d70640 1 --2- 172.21.15.28:0/57647864 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f584c0a2d00 0x7f584c0a51c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f585809bca0 tx=0x7f585809bd70 comp rx=0 tx=0).stop 2024-04-25T01:38:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.719+0000 7f5866d70640 1 -- 172.21.15.28:0/57647864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f586019c8b0 msgr2=0x7f586019cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.719+0000 7f5866d70640 1 --2- 172.21.15.28:0/57647864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f586019c8b0 0x7f586019cd60 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f585800cce0 tx=0x7f585800a620 comp rx=0 tx=0).stop 2024-04-25T01:38:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.719+0000 7f5866d70640 1 -- 172.21.15.28:0/57647864 shutdown_connections 2024-04-25T01:38:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.719+0000 7f5866d70640 1 --2- 172.21.15.28:0/57647864 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f584c0a2d00 0x7f584c0a51c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.719+0000 7f5866d70640 1 --2- 172.21.15.28:0/57647864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f586019c8b0 0x7f586019cd60 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.719+0000 7f5866d70640 1 --2- 172.21.15.28:0/57647864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5860106aa0 0x7f5860198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.719+0000 7f5866d70640 1 --2- 172.21.15.28:0/57647864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f58601002e0 0x7f5860197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.719+0000 7f5866d70640 1 -- 172.21.15.28:0/57647864 >> 172.21.15.28:0/57647864 conn(0x7f58600758d0 msgr2=0x7f58600fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.719+0000 7f5866d70640 1 -- 172.21.15.28:0/57647864 shutdown_connections 2024-04-25T01:38:38.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.719+0000 7f5866d70640 1 -- 172.21.15.28:0/57647864 wait complete. 2024-04-25T01:38:38.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:38:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.834+0000 7fa24bcb3640 1 -- 172.21.15.28:0/3519542035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa244069dc0 msgr2=0x7fa24406a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.834+0000 7fa24bcb3640 1 --2- 172.21.15.28:0/3519542035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa244069dc0 0x7fa24406a240 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fa234009930 tx=0x7fa23402a420 comp rx=0 tx=0).stop 2024-04-25T01:38:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.834+0000 7fa24bcb3640 1 -- 172.21.15.28:0/3519542035 shutdown_connections 2024-04-25T01:38:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.834+0000 7fa24bcb3640 1 --2- 172.21.15.28:0/3519542035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa24406a780 0x7fa24410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.834+0000 7fa24bcb3640 1 --2- 172.21.15.28:0/3519542035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa244069dc0 0x7fa24406a240 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.834+0000 7fa24bcb3640 1 --2- 172.21.15.28:0/3519542035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa244107730 0x7fa244107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.834+0000 7fa24bcb3640 1 -- 172.21.15.28:0/3519542035 >> 172.21.15.28:0/3519542035 conn(0x7fa2440776a0 msgr2=0x7fa244077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.834+0000 7fa24bcb3640 1 -- 172.21.15.28:0/3519542035 shutdown_connections 2024-04-25T01:38:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.834+0000 7fa24bcb3640 1 -- 172.21.15.28:0/3519542035 wait complete. 2024-04-25T01:38:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.835+0000 7fa24bcb3640 1 Processor -- start 2024-04-25T01:38:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.835+0000 7fa24bcb3640 1 -- start start 2024-04-25T01:38:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.835+0000 7fa24bcb3640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa244069dc0 0x7fa244197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.835+0000 7fa24bcb3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa24406a780 0x7fa2441984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.835+0000 7fa24bcb3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa244107730 0x7fa24419c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.835+0000 7fa24bcb3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa244111b40 con 0x7fa244107730 2024-04-25T01:38:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.835+0000 7fa24bcb3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa2441119c0 con 0x7fa24406a780 2024-04-25T01:38:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.835+0000 7fa24bcb3640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa244111cc0 con 0x7fa244069dc0 2024-04-25T01:38:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.835+0000 7fa24a250640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa244107730 0x7fa24419c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.835+0000 7fa24a250640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa244107730 0x7fa24419c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45654/0 (socket says 172.21.15.28:45654) 2024-04-25T01:38:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.835+0000 7fa24a250640 1 -- 172.21.15.28:0/2149755864 learned_addr learned my addr 172.21.15.28:0/2149755864 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.835+0000 7fa249a4f640 1 --2- 172.21.15.28:0/2149755864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa244069dc0 0x7fa244197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.835+0000 7fa24924e640 1 --2- 172.21.15.28:0/2149755864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa24406a780 0x7fa2441984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.836+0000 7fa24a250640 1 -- 172.21.15.28:0/2149755864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa244069dc0 msgr2=0x7fa244197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.836+0000 7fa24a250640 1 --2- 172.21.15.28:0/2149755864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa244069dc0 0x7fa244197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.836+0000 7fa24a250640 1 -- 172.21.15.28:0/2149755864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa24406a780 msgr2=0x7fa2441984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.836+0000 7fa24a250640 1 --2- 172.21.15.28:0/2149755864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa24406a780 0x7fa2441984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.836+0000 7fa24a250640 1 -- 172.21.15.28:0/2149755864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa24419cf20 con 0x7fa244107730 2024-04-25T01:38:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.836+0000 7fa24a250640 1 --2- 172.21.15.28:0/2149755864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa244107730 0x7fa24419c7f0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fa23c00eaa0 tx=0x7fa23c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.836+0000 7fa23affd640 1 -- 172.21.15.28:0/2149755864 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23c004d20 con 0x7fa244107730 2024-04-25T01:38:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.836+0000 7fa23affd640 1 -- 172.21.15.28:0/2149755864 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa23c005020 con 0x7fa244107730 2024-04-25T01:38:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.836+0000 7fa24bcb3640 1 -- 172.21.15.28:0/2149755864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa24419d210 con 0x7fa244107730 2024-04-25T01:38:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.836+0000 7fa23affd640 1 -- 172.21.15.28:0/2149755864 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23c0116d0 con 0x7fa244107730 2024-04-25T01:38:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.836+0000 7fa24bcb3640 1 -- 172.21.15.28:0/2149755864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2441a4aa0 con 0x7fa244107730 2024-04-25T01:38:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.838+0000 7fa23affd640 1 -- 172.21.15.28:0/2149755864 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa23c0051d0 con 0x7fa244107730 2024-04-25T01:38:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.839+0000 7fa24e9c2740 1 Processor -- start 2024-04-25T01:38:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.839+0000 7fa24e9c2740 1 -- start start 2024-04-25T01:38:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.839+0000 7fa24e9c2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1ef2d0 0x56429e1ef6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.839+0000 7fa24e9c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1efc80 0x56429e1fe440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.839+0000 7fa24e9c2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1fe980 0x56429e200d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.839+0000 7fa24e9c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56429e154670 con 0x56429e1efc80 2024-04-25T01:38:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.839+0000 7fa24e9c2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56429e0fc970 con 0x56429e1fe980 2024-04-25T01:38:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.839+0000 7fa24e9c2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56429e150630 con 0x56429e1ef2d0 2024-04-25T01:38:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.839+0000 7fa23affd640 1 --2- 172.21.15.28:0/2149755864 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa2300a2e00 0x7fa2300a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.839+0000 7fa249a4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1efc80 0x56429e1fe440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.839+0000 7fa249a4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1efc80 0x56429e1fe440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45660/0 (socket says 172.21.15.28:45660) 2024-04-25T01:38:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.839+0000 7fa249a4f640 1 -- 172.21.15.28:0/3521822248 learned_addr learned my addr 172.21.15.28:0/3521822248 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.839+0000 7fa24a250640 1 --2- 172.21.15.28:0/2149755864 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa2300a2e00 0x7fa2300a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.839+0000 7fa24924e640 1 --2- 172.21.15.28:0/3521822248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1fe980 0x56429e200d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.839+0000 7fa24924e640 1 --2- 172.21.15.28:0/3521822248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1ef2d0 0x56429e1ef6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.839+0000 7fa23affd640 1 -- 172.21.15.28:0/2149755864 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fa23c0cf8b0 con 0x7fa244107730 2024-04-25T01:38:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.840+0000 7fa249a4f640 1 -- 172.21.15.28:0/3521822248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1ef2d0 msgr2=0x56429e1ef6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.840+0000 7fa249a4f640 1 --2- 172.21.15.28:0/3521822248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1ef2d0 0x56429e1ef6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.840+0000 7fa249a4f640 1 -- 172.21.15.28:0/3521822248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1fe980 msgr2=0x56429e200d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.840+0000 7fa249a4f640 1 --2- 172.21.15.28:0/3521822248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1fe980 0x56429e200d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.840+0000 7fa249a4f640 1 -- 172.21.15.28:0/3521822248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56429e120400 con 0x56429e1efc80 2024-04-25T01:38:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.840+0000 7fa24a250640 1 --2- 172.21.15.28:0/2149755864 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa2300a2e00 0x7fa2300a52c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa23c09b730 tx=0x7fa23c06d9f0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.840+0000 7fa249a4f640 1 --2- 172.21.15.28:0/3521822248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1efc80 0x56429e1fe440 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fa24000ac90 tx=0x7fa24000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7bf415e9be9ee3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.840+0000 7fa22ffff640 1 -- 172.21.15.28:0/3521822248 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24001d070 con 0x56429e1efc80 2024-04-25T01:38:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.840+0000 7fa22ffff640 1 -- 172.21.15.28:0/3521822248 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa240018b20 con 0x56429e1efc80 2024-04-25T01:38:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.840+0000 7fa22ffff640 1 -- 172.21.15.28:0/3521822248 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa240010b00 con 0x56429e1efc80 2024-04-25T01:38:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.841+0000 7fa24e9c2740 1 -- 172.21.15.28:0/3521822248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1efc80 msgr2=0x56429e1fe440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.841+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/3521822248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1efc80 0x56429e1fe440 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fa24000ac90 tx=0x7fa24000f790 comp rx=0 tx=0).stop 2024-04-25T01:38:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.841+0000 7fa24e9c2740 1 -- 172.21.15.28:0/3521822248 shutdown_connections 2024-04-25T01:38:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.841+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/3521822248 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1fe980 0x56429e200d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.841+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/3521822248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1efc80 0x56429e1fe440 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.841+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/3521822248 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1ef2d0 0x56429e1ef6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.841+0000 7fa24e9c2740 1 -- 172.21.15.28:0/3521822248 >> 172.21.15.28:0/3521822248 conn(0x56429e1eb8a0 msgr2=0x56429e1ebc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.841+0000 7fa24e9c2740 1 -- 172.21.15.28:0/3521822248 shutdown_connections 2024-04-25T01:38:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.841+0000 7fa24e9c2740 1 -- 172.21.15.28:0/3521822248 wait complete. 2024-04-25T01:38:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.841+0000 7fa24e9c2740 1 -- start start 2024-04-25T01:38:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.841+0000 7fa24e9c2740 1 Processor -- start 2024-04-25T01:38:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa24e9c2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1ef2d0 0x56429e1fe2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa24e9c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1efc80 0x56429e1fb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa24e9c2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1fe980 0x56429e1fbb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa24e9c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56429e201740 con 0x56429e1efc80 2024-04-25T01:38:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa24e9c2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56429e110050 con 0x56429e1ef2d0 2024-04-25T01:38:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa24e9c2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56429e2018c0 con 0x56429e1fe980 2024-04-25T01:38:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa24e9c2740 10 client.0 _ll_register_callbacks cb 0x56429de67be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa249a4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1efc80 0x56429e1fb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa249a4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1efc80 0x56429e1fb5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45672/0 (socket says 172.21.15.28:45672) 2024-04-25T01:38:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa249a4f640 1 -- 172.21.15.28:0/792064217 learned_addr learned my addr 172.21.15.28:0/792064217 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa24924e640 1 --2- 172.21.15.28:0/792064217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1ef2d0 0x56429e1fe2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa24924e640 1 --2- 172.21.15.28:0/792064217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1fe980 0x56429e1fbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa249a4f640 1 -- 172.21.15.28:0/792064217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1fe980 msgr2=0x56429e1fbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa249a4f640 1 --2- 172.21.15.28:0/792064217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1fe980 0x56429e1fbb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa249a4f640 1 -- 172.21.15.28:0/792064217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1ef2d0 msgr2=0x56429e1fe2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa249a4f640 1 --2- 172.21.15.28:0/792064217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1ef2d0 0x56429e1fe2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa249a4f640 1 -- 172.21.15.28:0/792064217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56429e0e1dc0 con 0x56429e1efc80 2024-04-25T01:38:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa24924e640 1 --2- 172.21.15.28:0/792064217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1ef2d0 0x56429e1fe2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa249a4f640 1 --2- 172.21.15.28:0/792064217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1efc80 0x56429e1fb5c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fa240000c00 tx=0x7fa2400186f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48960dacc4608a6a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa22d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa22d7fa640 1 -- 172.21.15.28:0/792064217 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24001d040 con 0x56429e1efc80 2024-04-25T01:38:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa22d7fa640 1 -- 172.21.15.28:0/792064217 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fa2400108e0 con 0x56429e1efc80 2024-04-25T01:38:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.842+0000 7fa24e9c2740 1 -- 172.21.15.28:0/792064217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa240018b20 con 0x56429e1efc80 2024-04-25T01:38:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.843+0000 7fa22d7fa640 1 -- 172.21.15.28:0/792064217 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa240031050 con 0x56429e1efc80 2024-04-25T01:38:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.843+0000 7fa24e9c2740 10 client.17466 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.843+0000 7fa24e9c2740 1 -- 172.21.15.28:0/792064217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56429de683f0 con 0x56429e1efc80 2024-04-25T01:38:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.843+0000 7fa22d7fa640 1 -- 172.21.15.28:0/792064217 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa240018b20 con 0x56429e1efc80 2024-04-25T01:38:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.843+0000 7fa22d7fa640 1 -- 172.21.15.28:0/792064217 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa240020030 con 0x56429e1efc80 2024-04-25T01:38:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.843+0000 7fa24e9c2740 10 client.17466 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.843+0000 7fa24e9c2740 10 client.17466 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:38:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.843+0000 7fa24e9c2740 20 client.17466 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.843+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/792064217 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x56429e20e920 0x56429e210d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.843+0000 7fa24e9c2740 4 client.17466 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.843+0000 7fa24e9c2740 1 -- 172.21.15.28:0/792064217 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56429e161bb0 con 0x56429e20e920 2024-04-25T01:38:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.843+0000 7fa24924e640 1 --2- 172.21.15.28:0/792064217 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x56429e20e920 0x56429e210d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.844+0000 7fa22d7fa640 10 client.17466 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:38:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.844+0000 7fa24924e640 1 --2- 172.21.15.28:0/792064217 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x56429e20e920 0x56429e210d10 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c98269f4fbee5f02 server_cookie=87c114748b49ddf8 in_seq=0 out_seq=0 2024-04-25T01:38:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.844+0000 7fa22d7fa640 1 -- 172.21.15.28:0/792064217 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56429e161bb0 con 0x56429e20e920 2024-04-25T01:38:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.844+0000 7fa22d7fa640 10 client.17466 handle_command_reply: tid=0 2024-04-25T01:38:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.844+0000 7fa24e9c2740 1 client.17466 shutdown 2024-04-25T01:38:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.844+0000 7fa24e9c2740 1 -- 172.21.15.28:0/792064217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1efc80 msgr2=0x56429e1fb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.844+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/792064217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1efc80 0x56429e1fb5c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fa240000c00 tx=0x7fa2400186f0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.845+0000 7fa24e9c2740 1 -- 172.21.15.28:0/792064217 shutdown_connections 2024-04-25T01:38:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.845+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/792064217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1fe980 0x56429e1fbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.845+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/792064217 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x56429e20e920 0x56429e210d10 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.845+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/792064217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1efc80 0x56429e1fb5c0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.845+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/792064217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1ef2d0 0x56429e1fe2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.845+0000 7fa24e9c2740 1 -- 172.21.15.28:0/792064217 >> 172.21.15.28:0/792064217 conn(0x56429e1eb8a0 msgr2=0x56429e1f1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.845+0000 7fa22d7fa640 0 client.17466 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:38:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.845+0000 7fa24e9c2740 1 -- 172.21.15.28:0/792064217 shutdown_connections 2024-04-25T01:38:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.845+0000 7fa24e9c2740 1 -- 172.21.15.28:0/792064217 wait complete. 2024-04-25T01:38:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.845+0000 7fa24e9c2740 20 client.17466 trim_cache size 0 max 16384 2024-04-25T01:38:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.858+0000 7fa24e9c2740 1 Processor -- start 2024-04-25T01:38:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.858+0000 7fa24e9c2740 1 -- start start 2024-04-25T01:38:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.858+0000 7fa24e9c2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1ef2d0 0x56429e20a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.858+0000 7fa24e9c2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1efc80 0x56429e1fde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.858+0000 7fa24e9c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1fe980 0x56429e216340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.858+0000 7fa24e9c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2441ac8a0 con 0x56429e1fe980 2024-04-25T01:38:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.858+0000 7fa24e9c2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa2441ae3e0 con 0x56429e1efc80 2024-04-25T01:38:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.858+0000 7fa24e9c2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa2441acf70 con 0x56429e1ef2d0 2024-04-25T01:38:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.858+0000 7fa24924e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1fe980 0x56429e216340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.858+0000 7fa24924e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1fe980 0x56429e216340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45674/0 (socket says 172.21.15.28:45674) 2024-04-25T01:38:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.858+0000 7fa24924e640 1 -- 172.21.15.28:0/2281857133 learned_addr learned my addr 172.21.15.28:0/2281857133 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.859+0000 7fa24924e640 1 --2- 172.21.15.28:0/2281857133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1ef2d0 0x56429e20a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.859+0000 7fa249a4f640 1 --2- 172.21.15.28:0/2281857133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1efc80 0x56429e1fde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.859+0000 7fa24924e640 1 -- 172.21.15.28:0/2281857133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1ef2d0 msgr2=0x56429e20a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.859+0000 7fa24924e640 1 --2- 172.21.15.28:0/2281857133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1ef2d0 0x56429e20a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.859+0000 7fa24924e640 1 -- 172.21.15.28:0/2281857133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1efc80 msgr2=0x56429e1fde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.859+0000 7fa24924e640 1 --2- 172.21.15.28:0/2281857133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1efc80 0x56429e1fde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.859+0000 7fa24924e640 1 -- 172.21.15.28:0/2281857133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56429df327d0 con 0x56429e1fe980 2024-04-25T01:38:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.859+0000 7fa24924e640 1 --2- 172.21.15.28:0/2281857133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1fe980 0x56429e216340 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fa23402a940 tx=0x7fa234009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc1d010137e9e3ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.859+0000 7fa22f7fe640 1 -- 172.21.15.28:0/2281857133 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa234035040 con 0x56429e1fe980 2024-04-25T01:38:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.859+0000 7fa22f7fe640 1 -- 172.21.15.28:0/2281857133 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa23404b3d0 con 0x56429e1fe980 2024-04-25T01:38:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.859+0000 7fa22f7fe640 1 -- 172.21.15.28:0/2281857133 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa234005380 con 0x56429e1fe980 2024-04-25T01:38:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.860+0000 7fa24e9c2740 1 -- 172.21.15.28:0/2281857133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1fe980 msgr2=0x56429e216340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.860+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/2281857133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1fe980 0x56429e216340 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fa23402a940 tx=0x7fa234009680 comp rx=0 tx=0).stop 2024-04-25T01:38:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.860+0000 7fa24e9c2740 1 -- 172.21.15.28:0/2281857133 shutdown_connections 2024-04-25T01:38:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.860+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/2281857133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1fe980 0x56429e216340 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.860+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/2281857133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1efc80 0x56429e1fde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.860+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/2281857133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1ef2d0 0x56429e20a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.860+0000 7fa24e9c2740 1 -- 172.21.15.28:0/2281857133 >> 172.21.15.28:0/2281857133 conn(0x56429e1eb8a0 msgr2=0x56429e1f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.860+0000 7fa24e9c2740 1 -- 172.21.15.28:0/2281857133 shutdown_connections 2024-04-25T01:38:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.860+0000 7fa24e9c2740 1 -- 172.21.15.28:0/2281857133 wait complete. 2024-04-25T01:38:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.860+0000 7fa24e9c2740 1 -- start start 2024-04-25T01:38:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.860+0000 7fa24e9c2740 1 Processor -- start 2024-04-25T01:38:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.860+0000 7fa24e9c2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1ef2d0 0x56429e206ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.860+0000 7fa24e9c2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1efc80 0x56429e202280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.860+0000 7fa24e9c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1fe980 0x56429e2027c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.860+0000 7fa24e9c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa2441add10 con 0x56429e1fe980 2024-04-25T01:38:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.860+0000 7fa24e9c2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa2441ad640 con 0x56429e1efc80 2024-04-25T01:38:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.860+0000 7fa24e9c2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa2441aa880 con 0x56429e1ef2d0 2024-04-25T01:38:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.861+0000 7fa24e9c2740 10 client.0 _ll_register_callbacks cb 0x7fa24001b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.861+0000 7fa24924e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1fe980 0x56429e2027c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.861+0000 7fa24924e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1fe980 0x56429e2027c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45678/0 (socket says 172.21.15.28:45678) 2024-04-25T01:38:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.861+0000 7fa24924e640 1 -- 172.21.15.28:0/3494914896 learned_addr learned my addr 172.21.15.28:0/3494914896 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.861+0000 7fa249a4f640 1 --2- 172.21.15.28:0/3494914896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1efc80 0x56429e202280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.861+0000 7fa24924e640 1 --2- 172.21.15.28:0/3494914896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1ef2d0 0x56429e206ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.861+0000 7fa24924e640 1 -- 172.21.15.28:0/3494914896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1ef2d0 msgr2=0x56429e206ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.861+0000 7fa24924e640 1 --2- 172.21.15.28:0/3494914896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1ef2d0 0x56429e206ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.861+0000 7fa24924e640 1 -- 172.21.15.28:0/3494914896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1efc80 msgr2=0x56429e202280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.861+0000 7fa24924e640 1 --2- 172.21.15.28:0/3494914896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1efc80 0x56429e202280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.861+0000 7fa24924e640 1 -- 172.21.15.28:0/3494914896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56429e202ff0 con 0x56429e1fe980 2024-04-25T01:38:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.861+0000 7fa24924e640 1 --2- 172.21.15.28:0/3494914896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1fe980 0x56429e2027c0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fa234009230 tx=0x7fa234005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ae69bf5e879ad02 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.861+0000 7fa22d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.861+0000 7fa22d7fa640 1 -- 172.21.15.28:0/3494914896 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa234035070 con 0x56429e1fe980 2024-04-25T01:38:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.861+0000 7fa22d7fa640 1 -- 172.21.15.28:0/3494914896 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fa234005030 con 0x56429e1fe980 2024-04-25T01:38:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.861+0000 7fa24e9c2740 1 -- 172.21.15.28:0/3494914896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa23404b3d0 con 0x56429e1fe980 2024-04-25T01:38:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.861+0000 7fa22d7fa640 1 -- 172.21.15.28:0/3494914896 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa23404b880 con 0x56429e1fe980 2024-04-25T01:38:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.861+0000 7fa24e9c2740 10 client.17472 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.861+0000 7fa24e9c2740 1 -- 172.21.15.28:0/3494914896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56429e274b80 con 0x56429e1fe980 2024-04-25T01:38:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.862+0000 7fa22d7fa640 1 -- 172.21.15.28:0/3494914896 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa23404b3d0 con 0x56429e1fe980 2024-04-25T01:38:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.862+0000 7fa22d7fa640 1 -- 172.21.15.28:0/3494914896 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa240019ce0 con 0x56429e1fe980 2024-04-25T01:38:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.862+0000 7fa24e9c2740 10 client.17472 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.862+0000 7fa24e9c2740 10 client.17472 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:38:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.862+0000 7fa24e9c2740 20 client.17472 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.862+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/3494914896 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x56429e275270 0x56429e277690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.862+0000 7fa24e9c2740 4 client.17472 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:38:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.862+0000 7fa24e9c2740 1 -- 172.21.15.28:0/3494914896 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56429e15bf00 con 0x56429e275270 2024-04-25T01:38:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.862+0000 7fa249a4f640 1 --2- 172.21.15.28:0/3494914896 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x56429e275270 0x56429e277690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.863+0000 7fa249a4f640 1 --2- 172.21.15.28:0/3494914896 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x56429e275270 0x56429e277690 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=58bbe8f0ce860a48 server_cookie=e0fec738e99b38c1 in_seq=0 out_seq=0 2024-04-25T01:38:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.863+0000 7fa22d7fa640 10 client.17472 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:38:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.863+0000 7fa22d7fa640 1 -- 172.21.15.28:0/3494914896 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56429e15bf00 con 0x56429e275270 2024-04-25T01:38:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.863+0000 7fa22d7fa640 10 client.17472 handle_command_reply: tid=0 2024-04-25T01:38:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.863+0000 7fa24e9c2740 1 client.17472 shutdown 2024-04-25T01:38:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.863+0000 7fa24e9c2740 1 -- 172.21.15.28:0/3494914896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1fe980 msgr2=0x56429e2027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.863+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/3494914896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1fe980 0x56429e2027c0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fa234009230 tx=0x7fa234005fd0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.863+0000 7fa24e9c2740 1 -- 172.21.15.28:0/3494914896 shutdown_connections 2024-04-25T01:38:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.863+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/3494914896 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x56429e275270 0x56429e277690 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.863+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/3494914896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56429e1fe980 0x56429e2027c0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.863+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/3494914896 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56429e1efc80 0x56429e202280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.863+0000 7fa22d7fa640 0 client.17472 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:38:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.863+0000 7fa24e9c2740 1 --2- 172.21.15.28:0/3494914896 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56429e1ef2d0 0x56429e206ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.863+0000 7fa24e9c2740 1 -- 172.21.15.28:0/3494914896 >> 172.21.15.28:0/3494914896 conn(0x56429e1eb8a0 msgr2=0x56429e1ea2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.863+0000 7fa24e9c2740 1 -- 172.21.15.28:0/3494914896 shutdown_connections 2024-04-25T01:38:38.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.863+0000 7fa24e9c2740 1 -- 172.21.15.28:0/3494914896 wait complete. 2024-04-25T01:38:38.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.863+0000 7fa24e9c2740 20 client.17472 trim_cache size 0 max 16384 2024-04-25T01:38:38.851 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:38:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.864+0000 7fa24bcb3640 1 -- 172.21.15.28:0/2149755864 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa2300a2e00 msgr2=0x7fa2300a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.864+0000 7fa24bcb3640 1 --2- 172.21.15.28:0/2149755864 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa2300a2e00 0x7fa2300a52c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa23c09b730 tx=0x7fa23c06d9f0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.864+0000 7fa24bcb3640 1 -- 172.21.15.28:0/2149755864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa244107730 msgr2=0x7fa24419c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.864+0000 7fa24bcb3640 1 --2- 172.21.15.28:0/2149755864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa244107730 0x7fa24419c7f0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fa23c00eaa0 tx=0x7fa23c00eea0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.864+0000 7fa24bcb3640 1 -- 172.21.15.28:0/2149755864 shutdown_connections 2024-04-25T01:38:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.864+0000 7fa24bcb3640 1 --2- 172.21.15.28:0/2149755864 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa2300a2e00 0x7fa2300a52c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.864+0000 7fa24bcb3640 1 --2- 172.21.15.28:0/2149755864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa244107730 0x7fa24419c7f0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.864+0000 7fa24bcb3640 1 --2- 172.21.15.28:0/2149755864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa24406a780 0x7fa2441984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.864+0000 7fa24bcb3640 1 --2- 172.21.15.28:0/2149755864 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa244069dc0 0x7fa244197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.864+0000 7fa24bcb3640 1 -- 172.21.15.28:0/2149755864 >> 172.21.15.28:0/2149755864 conn(0x7fa2440776a0 msgr2=0x7fa2440fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.864+0000 7fa24bcb3640 1 -- 172.21.15.28:0/2149755864 shutdown_connections 2024-04-25T01:38:38.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.864+0000 7fa24bcb3640 1 -- 172.21.15.28:0/2149755864 wait complete. 2024-04-25T01:38:38.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:38:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.981+0000 7fed2186d640 1 -- 172.21.15.28:0/3020980871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed1c100610 msgr2=0x7fed1c100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.981+0000 7fed2186d640 1 --2- 172.21.15.28:0/3020980871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed1c100610 0x7fed1c100a90 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fed10009930 tx=0x7fed1002a420 comp rx=0 tx=0).stop 2024-04-25T01:38:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.981+0000 7fed2186d640 1 -- 172.21.15.28:0/3020980871 shutdown_connections 2024-04-25T01:38:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.981+0000 7fed2186d640 1 --2- 172.21.15.28:0/3020980871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed1c100fd0 0x7fed1c10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.981+0000 7fed2186d640 1 --2- 172.21.15.28:0/3020980871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed1c100610 0x7fed1c100a90 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.981+0000 7fed2186d640 1 --2- 172.21.15.28:0/3020980871 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed1c1066b0 0x7fed1c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.981+0000 7fed2186d640 1 -- 172.21.15.28:0/3020980871 >> 172.21.15.28:0/3020980871 conn(0x7fed1c077740 msgr2=0x7fed1c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.981+0000 7fed2186d640 1 -- 172.21.15.28:0/3020980871 shutdown_connections 2024-04-25T01:38:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.981+0000 7fed2186d640 1 -- 172.21.15.28:0/3020980871 wait complete. 2024-04-25T01:38:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.982+0000 7fed2186d640 1 Processor -- start 2024-04-25T01:38:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.982+0000 7fed2186d640 1 -- start start 2024-04-25T01:38:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.982+0000 7fed2186d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed1c100610 0x7fed1c102540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.982+0000 7fed2186d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed1c100fd0 0x7fed1c102a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.982+0000 7fed2186d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed1c1066b0 0x7fed1c102fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.982+0000 7fed2186d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c110a30 con 0x7fed1c100fd0 2024-04-25T01:38:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.982+0000 7fed2186d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c1108b0 con 0x7fed1c1066b0 2024-04-25T01:38:38.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.982+0000 7fed2186d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c110bb0 con 0x7fed1c100610 2024-04-25T01:38:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.983+0000 7fed1a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed1c100fd0 0x7fed1c102a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.983+0000 7fed1a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed1c100fd0 0x7fed1c102a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45700/0 (socket says 172.21.15.28:45700) 2024-04-25T01:38:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.983+0000 7fed1a7fc640 1 -- 172.21.15.28:0/2812609252 learned_addr learned my addr 172.21.15.28:0/2812609252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.983+0000 7fed1affd640 1 --2- 172.21.15.28:0/2812609252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed1c100610 0x7fed1c102540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.983+0000 7fed1b7fe640 1 --2- 172.21.15.28:0/2812609252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed1c1066b0 0x7fed1c102fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.983+0000 7fed1a7fc640 1 -- 172.21.15.28:0/2812609252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed1c100610 msgr2=0x7fed1c102540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.983+0000 7fed1a7fc640 1 --2- 172.21.15.28:0/2812609252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed1c100610 0x7fed1c102540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.983+0000 7fed1a7fc640 1 -- 172.21.15.28:0/2812609252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed1c1066b0 msgr2=0x7fed1c102fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.983+0000 7fed1a7fc640 1 --2- 172.21.15.28:0/2812609252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed1c1066b0 0x7fed1c102fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.983+0000 7fed1a7fc640 1 -- 172.21.15.28:0/2812609252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1c1a64e0 con 0x7fed1c100fd0 2024-04-25T01:38:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.983+0000 7fed1b7fe640 1 --2- 172.21.15.28:0/2812609252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed1c1066b0 0x7fed1c102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:38.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.983+0000 7fed1a7fc640 1 --2- 172.21.15.28:0/2812609252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed1c100fd0 0x7fed1c102a80 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fed10009230 tx=0x7fed10004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.983+0000 7fed2086b640 1 -- 172.21.15.28:0/2812609252 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1003f030 con 0x7fed1c100fd0 2024-04-25T01:38:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.983+0000 7fed2086b640 1 -- 172.21.15.28:0/2812609252 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed10004440 con 0x7fed1c100fd0 2024-04-25T01:38:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.983+0000 7fed2186d640 1 -- 172.21.15.28:0/2812609252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed1c1a6770 con 0x7fed1c100fd0 2024-04-25T01:38:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.983+0000 7fed2086b640 1 -- 172.21.15.28:0/2812609252 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed10035070 con 0x7fed1c100fd0 2024-04-25T01:38:38.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.983+0000 7fed2186d640 1 -- 172.21.15.28:0/2812609252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed1c1a6c60 con 0x7fed1c100fd0 2024-04-25T01:38:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.985+0000 7fed2086b640 1 -- 172.21.15.28:0/2812609252 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fed10042030 con 0x7fed1c100fd0 2024-04-25T01:38:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.986+0000 7fed24576740 1 Processor -- start 2024-04-25T01:38:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.986+0000 7fed24576740 1 -- start start 2024-04-25T01:38:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.986+0000 7fed24576740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9b8580 0x5599ca9b8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.986+0000 7fed24576740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8ea0 0x5599ca9c3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.986+0000 7fed24576740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 0x5599ca9c5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.986+0000 7fed24576740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5599ca91d670 con 0x5599ca9c3570 2024-04-25T01:38:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.986+0000 7fed24576740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5599ca8c5970 con 0x5599ca9b8ea0 2024-04-25T01:38:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.986+0000 7fed24576740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5599ca919630 con 0x5599ca9b8580 2024-04-25T01:38:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.986+0000 7fed2086b640 1 --2- 172.21.15.28:0/2812609252 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fed040a2d00 0x7fed040a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.986+0000 7fed1a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 0x5599ca9c5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.986+0000 7fed1b7fe640 1 --2- 172.21.15.28:0/2812609252 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fed040a2d00 0x7fed040a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.986+0000 7fed1affd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8ea0 0x5599ca9c3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.986+0000 7fed1a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 0x5599ca9c5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45710/0 (socket says 172.21.15.28:45710) 2024-04-25T01:38:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.986+0000 7fed1a7fc640 1 -- 172.21.15.28:0/746929701 learned_addr learned my addr 172.21.15.28:0/746929701 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.986+0000 7fed2086b640 1 -- 172.21.15.28:0/2812609252 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fed100efc30 con 0x7fed1c100fd0 2024-04-25T01:38:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.986+0000 7fed1b7fe640 1 --2- 172.21.15.28:0/746929701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9b8580 0x5599ca9b8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.987+0000 7fed1affd640 1 -- 172.21.15.28:0/746929701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9b8580 msgr2=0x5599ca9b8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.987+0000 7fed1affd640 1 --2- 172.21.15.28:0/746929701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9b8580 0x5599ca9b8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.987+0000 7fed1affd640 1 -- 172.21.15.28:0/746929701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 msgr2=0x5599ca9c5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.987+0000 7fed1affd640 1 --2- 172.21.15.28:0/746929701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 0x5599ca9c5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.987+0000 7fed1affd640 1 -- 172.21.15.28:0/746929701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599ca8e9400 con 0x5599ca9b8ea0 2024-04-25T01:38:38.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.987+0000 7fed1a7fc640 1 --2- 172.21.15.28:0/746929701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 0x5599ca9c5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:38:38.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.987+0000 7fed1b7fe640 1 --2- 172.21.15.28:0/2812609252 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fed040a2d00 0x7fed040a51c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fed14009810 tx=0x7fed14013ba0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.987+0000 7fed1affd640 1 --2- 172.21.15.28:0/746929701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8ea0 0x5599ca9c3030 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009ac0 tx=0x7fed0c009f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d54a7aacc465ec3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.987+0000 7fed01ffb640 1 -- 172.21.15.28:0/746929701 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c011da0 con 0x5599ca9b8ea0 2024-04-25T01:38:38.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.987+0000 7fed01ffb640 1 -- 172.21.15.28:0/746929701 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed0c016070 con 0x5599ca9b8ea0 2024-04-25T01:38:38.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.987+0000 7fed01ffb640 1 -- 172.21.15.28:0/746929701 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c012450 con 0x5599ca9b8ea0 2024-04-25T01:38:38.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.988+0000 7fed24576740 1 -- 172.21.15.28:0/746929701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8ea0 msgr2=0x5599ca9c3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.988+0000 7fed24576740 1 --2- 172.21.15.28:0/746929701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8ea0 0x5599ca9c3030 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009ac0 tx=0x7fed0c009f80 comp rx=0 tx=0).stop 2024-04-25T01:38:38.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.988+0000 7fed24576740 1 -- 172.21.15.28:0/746929701 shutdown_connections 2024-04-25T01:38:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.988+0000 7fed24576740 1 --2- 172.21.15.28:0/746929701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 0x5599ca9c5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.988+0000 7fed24576740 1 --2- 172.21.15.28:0/746929701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8ea0 0x5599ca9c3030 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.988+0000 7fed24576740 1 --2- 172.21.15.28:0/746929701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9b8580 0x5599ca9b8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.988+0000 7fed24576740 1 -- 172.21.15.28:0/746929701 >> 172.21.15.28:0/746929701 conn(0x5599ca9b48a0 msgr2=0x5599ca9b4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.988+0000 7fed24576740 1 -- 172.21.15.28:0/746929701 shutdown_connections 2024-04-25T01:38:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.988+0000 7fed24576740 1 -- 172.21.15.28:0/746929701 wait complete. 2024-04-25T01:38:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.988+0000 7fed24576740 1 -- start start 2024-04-25T01:38:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.988+0000 7fed24576740 1 Processor -- start 2024-04-25T01:38:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.988+0000 7fed24576740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9b8580 0x5599ca9c2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.988+0000 7fed24576740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8ea0 0x5599ca9c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.988+0000 7fed24576740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 0x5599ca9c0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.988+0000 7fed24576740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5599ca9c6360 con 0x5599ca9c3570 2024-04-25T01:38:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.988+0000 7fed24576740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5599ca8d9050 con 0x5599ca9b8ea0 2024-04-25T01:38:38.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.988+0000 7fed24576740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5599ca9c64e0 con 0x5599ca9b8580 2024-04-25T01:38:38.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fed24576740 10 client.0 _ll_register_callbacks cb 0x5599ca630be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:38.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fed1b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 0x5599ca9c0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fed1b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 0x5599ca9c0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45720/0 (socket says 172.21.15.28:45720) 2024-04-25T01:38:38.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fed1b7fe640 1 -- 172.21.15.28:0/4015653101 learned_addr learned my addr 172.21.15.28:0/4015653101 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fed1affd640 1 --2- 172.21.15.28:0/4015653101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9b8580 0x5599ca9c2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fed1a7fc640 1 --2- 172.21.15.28:0/4015653101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8ea0 0x5599ca9c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fed1b7fe640 1 -- 172.21.15.28:0/4015653101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9b8580 msgr2=0x5599ca9c2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fed1b7fe640 1 --2- 172.21.15.28:0/4015653101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9b8580 0x5599ca9c2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fed1b7fe640 1 -- 172.21.15.28:0/4015653101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8ea0 msgr2=0x5599ca9c01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fed1b7fe640 1 --2- 172.21.15.28:0/4015653101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8ea0 0x5599ca9c01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fed1b7fe640 1 -- 172.21.15.28:0/4015653101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599ca8aadc0 con 0x5599ca9c3570 2024-04-25T01:38:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fed1a7fc640 1 --2- 172.21.15.28:0/4015653101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8ea0 0x5599ca9c01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fed1b7fe640 1 --2- 172.21.15.28:0/4015653101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 0x5599ca9c0710 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fed14011a60 tx=0x7fed14009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47bef31873991ec6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fecfaffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fecfaffd640 1 -- 172.21.15.28:0/4015653101 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed14024070 con 0x5599ca9c3570 2024-04-25T01:38:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fecfaffd640 1 -- 172.21.15.28:0/4015653101 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fed14012c90 con 0x5599ca9c3570 2024-04-25T01:38:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fed24576740 1 -- 172.21.15.28:0/4015653101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed0c016070 con 0x5599ca9c3570 2024-04-25T01:38:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fecfaffd640 1 -- 172.21.15.28:0/4015653101 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fed1402f050 con 0x5599ca9c3570 2024-04-25T01:38:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fed24576740 10 client.17484 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.989+0000 7fed24576740 1 -- 172.21.15.28:0/4015653101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5599ca6313f0 con 0x5599ca9c3570 2024-04-25T01:38:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.990+0000 7fecfaffd640 1 -- 172.21.15.28:0/4015653101 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed0c016070 con 0x5599ca9c3570 2024-04-25T01:38:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.990+0000 7fecfaffd640 1 -- 172.21.15.28:0/4015653101 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fed14004620 con 0x5599ca9c3570 2024-04-25T01:38:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.990+0000 7fed24576740 10 client.17484 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.990+0000 7fed24576740 10 client.17484 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:38:38.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.990+0000 7fed24576740 20 client.17484 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.990+0000 7fed24576740 1 --2- 172.21.15.28:0/4015653101 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5599ca9d35b0 0x5599ca9d5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.990+0000 7fed24576740 4 client.17484 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.990+0000 7fed24576740 1 -- 172.21.15.28:0/4015653101 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599ca92abb0 con 0x5599ca9d35b0 2024-04-25T01:38:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.990+0000 7fed1affd640 1 --2- 172.21.15.28:0/4015653101 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5599ca9d35b0 0x5599ca9d5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.990+0000 7fecfaffd640 10 client.17484 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:38:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.990+0000 7fed1affd640 1 --2- 172.21.15.28:0/4015653101 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5599ca9d35b0 0x5599ca9d5a00 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ef69d54c5cc19554 server_cookie=97b9f3f08396fc32 in_seq=0 out_seq=0 2024-04-25T01:38:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.991+0000 7fecfaffd640 1 -- 172.21.15.28:0/4015653101 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5599ca92abb0 con 0x5599ca9d35b0 2024-04-25T01:38:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.991+0000 7fecfaffd640 10 client.17484 handle_command_reply: tid=0 2024-04-25T01:38:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.991+0000 7fed24576740 1 client.17484 shutdown 2024-04-25T01:38:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.991+0000 7fed24576740 1 -- 172.21.15.28:0/4015653101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 msgr2=0x5599ca9c0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.991+0000 7fed24576740 1 --2- 172.21.15.28:0/4015653101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 0x5599ca9c0710 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fed14011a60 tx=0x7fed14009500 comp rx=0 tx=0).stop 2024-04-25T01:38:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.991+0000 7fed24576740 1 -- 172.21.15.28:0/4015653101 shutdown_connections 2024-04-25T01:38:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.991+0000 7fed24576740 1 --2- 172.21.15.28:0/4015653101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 0x5599ca9c0710 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.991+0000 7fed24576740 1 --2- 172.21.15.28:0/4015653101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8ea0 0x5599ca9c01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.991+0000 7fed24576740 1 --2- 172.21.15.28:0/4015653101 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5599ca9d35b0 0x5599ca9d5a00 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.991+0000 7fed24576740 1 --2- 172.21.15.28:0/4015653101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9b8580 0x5599ca9c2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.991+0000 7fecfaffd640 0 client.17484 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:38:38.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.991+0000 7fed24576740 1 -- 172.21.15.28:0/4015653101 >> 172.21.15.28:0/4015653101 conn(0x5599ca9b48a0 msgr2=0x5599ca9b3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.991+0000 7fed24576740 1 -- 172.21.15.28:0/4015653101 shutdown_connections 2024-04-25T01:38:38.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.992+0000 7fed24576740 1 -- 172.21.15.28:0/4015653101 wait complete. 2024-04-25T01:38:38.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:38.992+0000 7fed24576740 20 client.17484 trim_cache size 0 max 16384 2024-04-25T01:38:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.005+0000 7fed24576740 1 Processor -- start 2024-04-25T01:38:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.005+0000 7fed24576740 1 -- start start 2024-04-25T01:38:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.005+0000 7fed24576740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8580 0x5599ca9cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.005+0000 7fed24576740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9b8ea0 0x5599ca9db2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.005+0000 7fed24576740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 0x5599ca9db810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.005+0000 7fed24576740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c1b0c30 con 0x5599ca9c3570 2024-04-25T01:38:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.005+0000 7fed24576740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c1b2770 con 0x5599ca9b8580 2024-04-25T01:38:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.005+0000 7fed24576740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c1b1300 con 0x5599ca9b8ea0 2024-04-25T01:38:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.005+0000 7fed1b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 0x5599ca9db810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.005+0000 7fed1b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 0x5599ca9db810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45722/0 (socket says 172.21.15.28:45722) 2024-04-25T01:38:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.005+0000 7fed1b7fe640 1 -- 172.21.15.28:0/294904141 learned_addr learned my addr 172.21.15.28:0/294904141 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.005+0000 7fed1a7fc640 1 --2- 172.21.15.28:0/294904141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9b8ea0 0x5599ca9db2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.005+0000 7fed1b7fe640 1 -- 172.21.15.28:0/294904141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9b8ea0 msgr2=0x5599ca9db2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.005+0000 7fed1affd640 1 --2- 172.21.15.28:0/294904141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8580 0x5599ca9cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.005+0000 7fed1b7fe640 1 --2- 172.21.15.28:0/294904141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9b8ea0 0x5599ca9db2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.005+0000 7fed1b7fe640 1 -- 172.21.15.28:0/294904141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8580 msgr2=0x5599ca9cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.005+0000 7fed1b7fe640 1 --2- 172.21.15.28:0/294904141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8580 0x5599ca9cf220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.005+0000 7fed1b7fe640 1 -- 172.21.15.28:0/294904141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599ca9bf160 con 0x5599ca9c3570 2024-04-25T01:38:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.006+0000 7fed1b7fe640 1 --2- 172.21.15.28:0/294904141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 0x5599ca9db810 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fed14004a80 tx=0x7fed14009960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c4fe0510f1b62a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.006+0000 7fed017fa640 1 -- 172.21.15.28:0/294904141 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed14024040 con 0x5599ca9c3570 2024-04-25T01:38:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.006+0000 7fed017fa640 1 -- 172.21.15.28:0/294904141 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed14038070 con 0x5599ca9c3570 2024-04-25T01:38:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.006+0000 7fed017fa640 1 -- 172.21.15.28:0/294904141 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed14033430 con 0x5599ca9c3570 2024-04-25T01:38:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.006+0000 7fed24576740 1 -- 172.21.15.28:0/294904141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 msgr2=0x5599ca9db810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.006+0000 7fed24576740 1 --2- 172.21.15.28:0/294904141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 0x5599ca9db810 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fed14004a80 tx=0x7fed14009960 comp rx=0 tx=0).stop 2024-04-25T01:38:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.006+0000 7fed24576740 1 -- 172.21.15.28:0/294904141 shutdown_connections 2024-04-25T01:38:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.006+0000 7fed24576740 1 --2- 172.21.15.28:0/294904141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9c3570 0x5599ca9db810 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.006+0000 7fed24576740 1 --2- 172.21.15.28:0/294904141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9b8ea0 0x5599ca9db2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.006+0000 7fed24576740 1 --2- 172.21.15.28:0/294904141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8580 0x5599ca9cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.006+0000 7fed24576740 1 -- 172.21.15.28:0/294904141 >> 172.21.15.28:0/294904141 conn(0x5599ca9b48a0 msgr2=0x5599ca9b3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.006+0000 7fed24576740 1 -- 172.21.15.28:0/294904141 shutdown_connections 2024-04-25T01:38:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.006+0000 7fed24576740 1 -- 172.21.15.28:0/294904141 wait complete. 2024-04-25T01:38:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed24576740 1 -- start start 2024-04-25T01:38:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed24576740 1 Processor -- start 2024-04-25T01:38:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed24576740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9b8580 0x5599ca9c9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed24576740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8ea0 0x5599ca9c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed24576740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9c3570 0x5599ca9c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed24576740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c1b20a0 con 0x5599ca9b8580 2024-04-25T01:38:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed24576740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c1b19d0 con 0x5599ca9b8ea0 2024-04-25T01:38:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed24576740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c1aec10 con 0x5599ca9c3570 2024-04-25T01:38:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed24576740 10 client.0 _ll_register_callbacks cb 0x7fed14013030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed1affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9b8580 0x5599ca9c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed1affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9b8580 0x5599ca9c9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45730/0 (socket says 172.21.15.28:45730) 2024-04-25T01:38:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed1affd640 1 -- 172.21.15.28:0/2025769819 learned_addr learned my addr 172.21.15.28:0/2025769819 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed1b7fe640 1 --2- 172.21.15.28:0/2025769819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9c3570 0x5599ca9c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed1a7fc640 1 --2- 172.21.15.28:0/2025769819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8ea0 0x5599ca9c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed1affd640 1 -- 172.21.15.28:0/2025769819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9c3570 msgr2=0x5599ca9c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed1affd640 1 --2- 172.21.15.28:0/2025769819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9c3570 0x5599ca9c9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed1affd640 1 -- 172.21.15.28:0/2025769819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8ea0 msgr2=0x5599ca9c96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed1affd640 1 --2- 172.21.15.28:0/2025769819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8ea0 0x5599ca9c96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed1affd640 1 -- 172.21.15.28:0/2025769819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599ca6fb7d0 con 0x5599ca9b8580 2024-04-25T01:38:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.007+0000 7fed1a7fc640 1 --2- 172.21.15.28:0/2025769819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8ea0 0x5599ca9c96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.008+0000 7fed1affd640 1 --2- 172.21.15.28:0/2025769819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9b8580 0x5599ca9c9180 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009ac0 tx=0x7fed0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ae74e3d43fd56b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.008+0000 7fecfaffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.008+0000 7fecfaffd640 1 -- 172.21.15.28:0/2025769819 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c011da0 con 0x5599ca9b8580 2024-04-25T01:38:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.008+0000 7fecfaffd640 1 -- 172.21.15.28:0/2025769819 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fed0c029430 con 0x5599ca9b8580 2024-04-25T01:38:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.008+0000 7fed24576740 1 -- 172.21.15.28:0/2025769819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed14038070 con 0x5599ca9b8580 2024-04-25T01:38:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.008+0000 7fecfaffd640 1 -- 172.21.15.28:0/2025769819 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fed0c034050 con 0x5599ca9b8580 2024-04-25T01:38:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.008+0000 7fed24576740 10 client.17490 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.008+0000 7fed24576740 1 -- 172.21.15.28:0/2025769819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5599caa39c30 con 0x5599ca9b8580 2024-04-25T01:38:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.008+0000 7fecfaffd640 1 -- 172.21.15.28:0/2025769819 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed14038070 con 0x5599ca9b8580 2024-04-25T01:38:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.008+0000 7fecfaffd640 1 -- 172.21.15.28:0/2025769819 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fed0c012450 con 0x5599ca9b8580 2024-04-25T01:38:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.008+0000 7fed24576740 10 client.17490 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.008+0000 7fed24576740 10 client.17490 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:38:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.008+0000 7fed24576740 20 client.17490 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.008+0000 7fed24576740 1 --2- 172.21.15.28:0/2025769819 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5599caa3a130 0x5599caa3c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.008+0000 7fed24576740 4 client.17490 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:38:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.008+0000 7fed24576740 1 -- 172.21.15.28:0/2025769819 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5599ca924f00 con 0x5599caa3a130 2024-04-25T01:38:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.009+0000 7fed1a7fc640 1 --2- 172.21.15.28:0/2025769819 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5599caa3a130 0x5599caa3c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.009+0000 7fecfaffd640 10 client.17490 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:38:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.009+0000 7fed1a7fc640 1 --2- 172.21.15.28:0/2025769819 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5599caa3a130 0x5599caa3c550 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=885b5be12a6f0973 server_cookie=8c5b0633267e3f41 in_seq=0 out_seq=0 2024-04-25T01:38:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.009+0000 7fecfaffd640 1 -- 172.21.15.28:0/2025769819 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5599ca924f00 con 0x5599caa3a130 2024-04-25T01:38:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.009+0000 7fecfaffd640 10 client.17490 handle_command_reply: tid=0 2024-04-25T01:38:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.009+0000 7fed24576740 1 client.17490 shutdown 2024-04-25T01:38:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.009+0000 7fed24576740 1 -- 172.21.15.28:0/2025769819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9b8580 msgr2=0x5599ca9c9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.009+0000 7fed24576740 1 --2- 172.21.15.28:0/2025769819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9b8580 0x5599ca9c9180 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009ac0 tx=0x7fed0c0057f0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.009+0000 7fed24576740 1 -- 172.21.15.28:0/2025769819 shutdown_connections 2024-04-25T01:38:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.009+0000 7fed24576740 1 --2- 172.21.15.28:0/2025769819 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5599caa3a130 0x5599caa3c550 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.009+0000 7fed24576740 1 --2- 172.21.15.28:0/2025769819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5599ca9c3570 0x5599ca9c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.009+0000 7fecfaffd640 0 client.17490 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:38:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.009+0000 7fed24576740 1 --2- 172.21.15.28:0/2025769819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5599ca9b8ea0 0x5599ca9c96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.009+0000 7fed24576740 1 --2- 172.21.15.28:0/2025769819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599ca9b8580 0x5599ca9c9180 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.010+0000 7fed24576740 1 -- 172.21.15.28:0/2025769819 >> 172.21.15.28:0/2025769819 conn(0x5599ca9b48a0 msgr2=0x5599ca9ce210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.010+0000 7fed24576740 1 -- 172.21.15.28:0/2025769819 shutdown_connections 2024-04-25T01:38:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.010+0000 7fed24576740 1 -- 172.21.15.28:0/2025769819 wait complete. 2024-04-25T01:38:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.010+0000 7fed24576740 20 client.17490 trim_cache size 0 max 16384 2024-04-25T01:38:38.998 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:38:38.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.010+0000 7fed2186d640 1 -- 172.21.15.28:0/2812609252 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fed040a2d00 msgr2=0x7fed040a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.010+0000 7fed2186d640 1 --2- 172.21.15.28:0/2812609252 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fed040a2d00 0x7fed040a51c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fed14009810 tx=0x7fed14013ba0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.010+0000 7fed2186d640 1 -- 172.21.15.28:0/2812609252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed1c100fd0 msgr2=0x7fed1c102a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.010+0000 7fed2186d640 1 --2- 172.21.15.28:0/2812609252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed1c100fd0 0x7fed1c102a80 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fed10009230 tx=0x7fed10004040 comp rx=0 tx=0).stop 2024-04-25T01:38:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.010+0000 7fed2186d640 1 -- 172.21.15.28:0/2812609252 shutdown_connections 2024-04-25T01:38:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.010+0000 7fed2186d640 1 --2- 172.21.15.28:0/2812609252 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fed040a2d00 0x7fed040a51c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.010+0000 7fed2186d640 1 --2- 172.21.15.28:0/2812609252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed1c1066b0 0x7fed1c102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.010+0000 7fed2186d640 1 --2- 172.21.15.28:0/2812609252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed1c100fd0 0x7fed1c102a80 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.010+0000 7fed2186d640 1 --2- 172.21.15.28:0/2812609252 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed1c100610 0x7fed1c102540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.010+0000 7fed2186d640 1 -- 172.21.15.28:0/2812609252 >> 172.21.15.28:0/2812609252 conn(0x7fed1c077740 msgr2=0x7fed1c104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.010+0000 7fed2186d640 1 -- 172.21.15.28:0/2812609252 shutdown_connections 2024-04-25T01:38:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.010+0000 7fed2186d640 1 -- 172.21.15.28:0/2812609252 wait complete. 2024-04-25T01:38:39.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:38:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.125+0000 7f464a2d4640 1 -- 172.21.15.28:0/3236195712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4644100fd0 msgr2=0x7f464410d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.125+0000 7f464a2d4640 1 --2- 172.21.15.28:0/3236195712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4644100fd0 0x7f464410d4e0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f463c009cd0 tx=0x7f463c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.126+0000 7f464a2d4640 1 -- 172.21.15.28:0/3236195712 shutdown_connections 2024-04-25T01:38:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.126+0000 7f464a2d4640 1 --2- 172.21.15.28:0/3236195712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4644100fd0 0x7f464410d4e0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.126+0000 7f464a2d4640 1 --2- 172.21.15.28:0/3236195712 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4644100610 0x7f4644100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.126+0000 7f464a2d4640 1 --2- 172.21.15.28:0/3236195712 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46441066b0 0x7f4644106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.126+0000 7f464a2d4640 1 -- 172.21.15.28:0/3236195712 >> 172.21.15.28:0/3236195712 conn(0x7f4644077740 msgr2=0x7f4644077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.126+0000 7f464a2d4640 1 -- 172.21.15.28:0/3236195712 shutdown_connections 2024-04-25T01:38:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.126+0000 7f464a2d4640 1 -- 172.21.15.28:0/3236195712 wait complete. 2024-04-25T01:38:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.126+0000 7f464a2d4640 1 Processor -- start 2024-04-25T01:38:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.126+0000 7f464a2d4640 1 -- start start 2024-04-25T01:38:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f464a2d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4644100610 0x7f464419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f464a2d4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4644100fd0 0x7f464419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f464a2d4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46441066b0 0x7f464419e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f464a2d4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46441109f0 con 0x7f4644100610 2024-04-25T01:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f464a2d4640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4644110870 con 0x7f4644100fd0 2024-04-25T01:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f464a2d4640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4644110b70 con 0x7f46441066b0 2024-04-25T01:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f4643fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4644100610 0x7f464419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f4643fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4644100610 0x7f464419a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45736/0 (socket says 172.21.15.28:45736) 2024-04-25T01:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f4643fff640 1 -- 172.21.15.28:0/2878243158 learned_addr learned my addr 172.21.15.28:0/2878243158 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f4648871640 1 --2- 172.21.15.28:0/2878243158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46441066b0 0x7f464419e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f46437fe640 1 --2- 172.21.15.28:0/2878243158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4644100fd0 0x7f464419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f4643fff640 1 -- 172.21.15.28:0/2878243158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46441066b0 msgr2=0x7f464419e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f4643fff640 1 --2- 172.21.15.28:0/2878243158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46441066b0 0x7f464419e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f4643fff640 1 -- 172.21.15.28:0/2878243158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4644100fd0 msgr2=0x7f464419a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f4643fff640 1 --2- 172.21.15.28:0/2878243158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4644100fd0 0x7f464419a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f4643fff640 1 -- 172.21.15.28:0/2878243158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f464419f110 con 0x7f4644100610 2024-04-25T01:38:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f4643fff640 1 --2- 172.21.15.28:0/2878243158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4644100610 0x7f464419a120 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f463400d950 tx=0x7f463400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f46417fa640 1 -- 172.21.15.28:0/2878243158 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4634017070 con 0x7f4644100610 2024-04-25T01:38:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f464a2d4640 1 -- 172.21.15.28:0/2878243158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f464419f400 con 0x7f4644100610 2024-04-25T01:38:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f46417fa640 1 -- 172.21.15.28:0/2878243158 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4634009c30 con 0x7f4644100610 2024-04-25T01:38:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f46417fa640 1 -- 172.21.15.28:0/2878243158 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463400f600 con 0x7f4644100610 2024-04-25T01:38:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.127+0000 7f464a2d4640 1 -- 172.21.15.28:0/2878243158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46441a6c90 con 0x7f4644100610 2024-04-25T01:38:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.129+0000 7f46417fa640 1 -- 172.21.15.28:0/2878243158 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f463401a070 con 0x7f4644100610 2024-04-25T01:38:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.130+0000 7f464cfe9740 1 Processor -- start 2024-04-25T01:38:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.130+0000 7f464cfe9740 1 -- start start 2024-04-25T01:38:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.130+0000 7f464cfe9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 0x55e01c138960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.130+0000 7f464cfe9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138ea0 0x55e01c143030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.130+0000 7f46417fa640 1 --2- 172.21.15.28:0/2878243158 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f462c0a2d00 0x7f462c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.130+0000 7f464cfe9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01c143570 0x55e01c145960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.130+0000 7f464cfe9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e01c09d670 con 0x55e01c138580 2024-04-25T01:38:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.130+0000 7f464cfe9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e01c045970 con 0x55e01c143570 2024-04-25T01:38:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.130+0000 7f464cfe9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55e01c099630 con 0x55e01c138ea0 2024-04-25T01:38:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.130+0000 7f4648871640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 0x55e01c138960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.130+0000 7f4648871640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 0x55e01c138960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45738/0 (socket says 172.21.15.28:45738) 2024-04-25T01:38:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.130+0000 7f4648871640 1 -- 172.21.15.28:0/779533402 learned_addr learned my addr 172.21.15.28:0/779533402 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.130+0000 7f46437fe640 1 --2- 172.21.15.28:0/2878243158 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f462c0a2d00 0x7f462c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.130+0000 7f46417fa640 1 -- 172.21.15.28:0/2878243158 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f46340cf290 con 0x7f4644100610 2024-04-25T01:38:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.130+0000 7f4643fff640 1 --2- 172.21.15.28:0/779533402 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138ea0 0x55e01c143030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.131+0000 7f4648871640 1 --2- 172.21.15.28:0/779533402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01c143570 0x55e01c145960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.131+0000 7f4648871640 1 -- 172.21.15.28:0/779533402 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138ea0 msgr2=0x55e01c143030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.131+0000 7f4648871640 1 --2- 172.21.15.28:0/779533402 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138ea0 0x55e01c143030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.131+0000 7f4648871640 1 -- 172.21.15.28:0/779533402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01c143570 msgr2=0x55e01c145960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.131+0000 7f4648871640 1 --2- 172.21.15.28:0/779533402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01c143570 0x55e01c145960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.131+0000 7f4648871640 1 -- 172.21.15.28:0/779533402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e01c069400 con 0x55e01c138580 2024-04-25T01:38:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.131+0000 7f46437fe640 1 --2- 172.21.15.28:0/2878243158 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f462c0a2d00 0x7f462c0a51c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f464410d8e0 tx=0x7f4638005f90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.131+0000 7f4648871640 1 --2- 172.21.15.28:0/779533402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 0x55e01c138960 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f463c004ab0 tx=0x7f463c009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=583aea49a6ecf87e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.131+0000 7f462a7fc640 1 -- 172.21.15.28:0/779533402 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463c0318c0 con 0x55e01c138580 2024-04-25T01:38:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.131+0000 7f462a7fc640 1 -- 172.21.15.28:0/779533402 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f463c036040 con 0x55e01c138580 2024-04-25T01:38:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.131+0000 7f462a7fc640 1 -- 172.21.15.28:0/779533402 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463c004610 con 0x55e01c138580 2024-04-25T01:38:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.132+0000 7f464cfe9740 1 -- 172.21.15.28:0/779533402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 msgr2=0x55e01c138960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.132+0000 7f464cfe9740 1 --2- 172.21.15.28:0/779533402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 0x55e01c138960 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f463c004ab0 tx=0x7f463c009360 comp rx=0 tx=0).stop 2024-04-25T01:38:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.132+0000 7f464cfe9740 1 -- 172.21.15.28:0/779533402 shutdown_connections 2024-04-25T01:38:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.132+0000 7f464cfe9740 1 --2- 172.21.15.28:0/779533402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01c143570 0x55e01c145960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.132+0000 7f464cfe9740 1 --2- 172.21.15.28:0/779533402 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138ea0 0x55e01c143030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.132+0000 7f464cfe9740 1 --2- 172.21.15.28:0/779533402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 0x55e01c138960 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.132+0000 7f464cfe9740 1 -- 172.21.15.28:0/779533402 >> 172.21.15.28:0/779533402 conn(0x55e01c1348a0 msgr2=0x55e01c134c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.132+0000 7f464cfe9740 1 -- 172.21.15.28:0/779533402 shutdown_connections 2024-04-25T01:38:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.132+0000 7f464cfe9740 1 -- 172.21.15.28:0/779533402 wait complete. 2024-04-25T01:38:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.132+0000 7f464cfe9740 1 -- start start 2024-04-25T01:38:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.132+0000 7f464cfe9740 1 Processor -- start 2024-04-25T01:38:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.132+0000 7f464cfe9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 0x55e01c142e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f464cfe9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138ea0 0x55e01c140120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f464cfe9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01c143570 0x55e01c140660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f464cfe9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e01c146300 con 0x55e01c138580 2024-04-25T01:38:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f464cfe9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e01c059050 con 0x55e01c143570 2024-04-25T01:38:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f464cfe9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55e01c146480 con 0x55e01c138ea0 2024-04-25T01:38:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f464cfe9740 10 client.0 _ll_register_callbacks cb 0x55e01bdb0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f4643fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 0x55e01c142e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f4643fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 0x55e01c142e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45748/0 (socket says 172.21.15.28:45748) 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f4643fff640 1 -- 172.21.15.28:0/3868747314 learned_addr learned my addr 172.21.15.28:0/3868747314 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f46437fe640 1 --2- 172.21.15.28:0/3868747314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138ea0 0x55e01c140120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f4643fff640 1 -- 172.21.15.28:0/3868747314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138ea0 msgr2=0x55e01c140120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f4643fff640 1 --2- 172.21.15.28:0/3868747314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138ea0 0x55e01c140120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f4643fff640 1 -- 172.21.15.28:0/3868747314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01c143570 msgr2=0x55e01c140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f4643fff640 1 --2- 172.21.15.28:0/3868747314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01c143570 0x55e01c140660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f4643fff640 1 -- 172.21.15.28:0/3868747314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e01c02adc0 con 0x55e01c138580 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f4643fff640 1 --2- 172.21.15.28:0/3868747314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 0x55e01c142e30 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f4634097820 tx=0x7f4634009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b7f4e5e97ed46a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f46237fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f46237fe640 1 -- 172.21.15.28:0/3868747314 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4634021040 con 0x55e01c138580 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f46237fe640 1 -- 172.21.15.28:0/3868747314 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f46340cc8c0 con 0x55e01c138580 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f464cfe9740 1 -- 172.21.15.28:0/3868747314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f463c036040 con 0x55e01c138580 2024-04-25T01:38:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f46237fe640 1 -- 172.21.15.28:0/3868747314 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f46340ccac0 con 0x55e01c138580 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f464cfe9740 10 client.17502 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f464cfe9740 1 -- 172.21.15.28:0/3868747314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e01bdb13f0 con 0x55e01c138580 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.133+0000 7f46237fe640 1 -- 172.21.15.28:0/3868747314 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f463c036040 con 0x55e01c138580 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.134+0000 7f46237fe640 1 -- 172.21.15.28:0/3868747314 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f46340b4210 con 0x55e01c138580 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.134+0000 7f464cfe9740 10 client.17502 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.134+0000 7f464cfe9740 10 client.17502 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.134+0000 7f464cfe9740 20 client.17502 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.134+0000 7f464cfe9740 1 --2- 172.21.15.28:0/3868747314 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55e01c1534c0 0x55e01c155930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.134+0000 7f464cfe9740 4 client.17502 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.134+0000 7f464cfe9740 1 -- 172.21.15.28:0/3868747314 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e01c0aabb0 con 0x55e01c1534c0 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.134+0000 7f46437fe640 1 --2- 172.21.15.28:0/3868747314 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55e01c1534c0 0x55e01c155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.135+0000 7f46237fe640 10 client.17502 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.135+0000 7f46437fe640 1 --2- 172.21.15.28:0/3868747314 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55e01c1534c0 0x55e01c155930 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a7800bb1f1023075 server_cookie=5b5c61d32ec4e5f3 in_seq=0 out_seq=0 2024-04-25T01:38:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.135+0000 7f46237fe640 1 -- 172.21.15.28:0/3868747314 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e01c0aabb0 con 0x55e01c1534c0 2024-04-25T01:38:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.135+0000 7f46237fe640 10 client.17502 handle_command_reply: tid=0 2024-04-25T01:38:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.135+0000 7f464cfe9740 1 client.17502 shutdown 2024-04-25T01:38:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.136+0000 7f464cfe9740 1 -- 172.21.15.28:0/3868747314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 msgr2=0x55e01c142e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.136+0000 7f464cfe9740 1 --2- 172.21.15.28:0/3868747314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 0x55e01c142e30 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f4634097820 tx=0x7f4634009f10 comp rx=0 tx=0).stop 2024-04-25T01:38:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.136+0000 7f464cfe9740 1 -- 172.21.15.28:0/3868747314 shutdown_connections 2024-04-25T01:38:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.136+0000 7f464cfe9740 1 --2- 172.21.15.28:0/3868747314 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55e01c1534c0 0x55e01c155930 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.136+0000 7f464cfe9740 1 --2- 172.21.15.28:0/3868747314 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01c143570 0x55e01c140660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.136+0000 7f464cfe9740 1 --2- 172.21.15.28:0/3868747314 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138ea0 0x55e01c140120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.136+0000 7f46237fe640 0 client.17502 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:38:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.136+0000 7f464cfe9740 1 --2- 172.21.15.28:0/3868747314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 0x55e01c142e30 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.136+0000 7f464cfe9740 1 -- 172.21.15.28:0/3868747314 >> 172.21.15.28:0/3868747314 conn(0x55e01c1348a0 msgr2=0x55e01c13a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.136+0000 7f464cfe9740 1 -- 172.21.15.28:0/3868747314 shutdown_connections 2024-04-25T01:38:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.136+0000 7f464cfe9740 1 -- 172.21.15.28:0/3868747314 wait complete. 2024-04-25T01:38:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.136+0000 7f464cfe9740 20 client.17502 trim_cache size 0 max 16384 2024-04-25T01:38:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.149+0000 7f464cfe9740 1 Processor -- start 2024-04-25T01:38:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.149+0000 7f464cfe9740 1 -- start start 2024-04-25T01:38:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.149+0000 7f464cfe9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138580 0x55e01c13b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.149+0000 7f464cfe9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01c138ea0 0x55e01c1339c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.149+0000 7f464cfe9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c143570 0x55e01c142960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.149+0000 7f464cfe9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46441b0a90 con 0x55e01c143570 2024-04-25T01:38:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.149+0000 7f464cfe9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f46441b25d0 con 0x55e01c138ea0 2024-04-25T01:38:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.149+0000 7f464cfe9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f46441b1160 con 0x55e01c138580 2024-04-25T01:38:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.149+0000 7f4648871640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c143570 0x55e01c142960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.149+0000 7f4648871640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c143570 0x55e01c142960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45758/0 (socket says 172.21.15.28:45758) 2024-04-25T01:38:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.149+0000 7f4648871640 1 -- 172.21.15.28:0/756042137 learned_addr learned my addr 172.21.15.28:0/756042137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.149+0000 7f46437fe640 1 --2- 172.21.15.28:0/756042137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01c138ea0 0x55e01c1339c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.149+0000 7f4643fff640 1 --2- 172.21.15.28:0/756042137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138580 0x55e01c13b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.150+0000 7f4648871640 1 -- 172.21.15.28:0/756042137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138580 msgr2=0x55e01c13b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.150+0000 7f4648871640 1 --2- 172.21.15.28:0/756042137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138580 0x55e01c13b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.150+0000 7f4648871640 1 -- 172.21.15.28:0/756042137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01c138ea0 msgr2=0x55e01c1339c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.150+0000 7f4648871640 1 --2- 172.21.15.28:0/756042137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01c138ea0 0x55e01c1339c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.150+0000 7f4648871640 1 -- 172.21.15.28:0/756042137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e01be7b7d0 con 0x55e01c143570 2024-04-25T01:38:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.150+0000 7f4648871640 1 --2- 172.21.15.28:0/756042137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c143570 0x55e01c142960 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f463c02af20 tx=0x7f463c0093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa5602c94075f01 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.150+0000 7f4629ffb640 1 -- 172.21.15.28:0/756042137 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463c0318c0 con 0x55e01c143570 2024-04-25T01:38:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.150+0000 7f4629ffb640 1 -- 172.21.15.28:0/756042137 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f463c036040 con 0x55e01c143570 2024-04-25T01:38:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.150+0000 7f4629ffb640 1 -- 172.21.15.28:0/756042137 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463c004500 con 0x55e01c143570 2024-04-25T01:38:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.150+0000 7f464cfe9740 1 -- 172.21.15.28:0/756042137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c143570 msgr2=0x55e01c142960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.150+0000 7f464cfe9740 1 --2- 172.21.15.28:0/756042137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c143570 0x55e01c142960 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f463c02af20 tx=0x7f463c0093d0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.150+0000 7f464cfe9740 1 -- 172.21.15.28:0/756042137 shutdown_connections 2024-04-25T01:38:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.150+0000 7f464cfe9740 1 --2- 172.21.15.28:0/756042137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c143570 0x55e01c142960 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.150+0000 7f464cfe9740 1 --2- 172.21.15.28:0/756042137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01c138ea0 0x55e01c1339c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.150+0000 7f464cfe9740 1 --2- 172.21.15.28:0/756042137 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138580 0x55e01c13b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.150+0000 7f464cfe9740 1 -- 172.21.15.28:0/756042137 >> 172.21.15.28:0/756042137 conn(0x55e01c1348a0 msgr2=0x55e01c1332b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.151+0000 7f464cfe9740 1 -- 172.21.15.28:0/756042137 shutdown_connections 2024-04-25T01:38:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.151+0000 7f464cfe9740 1 -- 172.21.15.28:0/756042137 wait complete. 2024-04-25T01:38:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.151+0000 7f464cfe9740 1 -- start start 2024-04-25T01:38:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.151+0000 7f464cfe9740 1 Processor -- start 2024-04-25T01:38:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.151+0000 7f464cfe9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 0x55e01c14b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.151+0000 7f464cfe9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138ea0 0x55e01c146cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.151+0000 7f464cfe9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01c143570 0x55e01c1471f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.151+0000 7f464cfe9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46441b1f00 con 0x55e01c138580 2024-04-25T01:38:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.151+0000 7f464cfe9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f46441b1830 con 0x55e01c143570 2024-04-25T01:38:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.151+0000 7f464cfe9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f46441aea70 con 0x55e01c138ea0 2024-04-25T01:38:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.151+0000 7f464cfe9740 10 client.0 _ll_register_callbacks cb 0x7f463409ae40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.151+0000 7f4643fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 0x55e01c14b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.151+0000 7f4643fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 0x55e01c14b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45766/0 (socket says 172.21.15.28:45766) 2024-04-25T01:38:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.152+0000 7f4643fff640 1 -- 172.21.15.28:0/3646507262 learned_addr learned my addr 172.21.15.28:0/3646507262 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.152+0000 7f4648871640 1 --2- 172.21.15.28:0/3646507262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01c143570 0x55e01c1471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.152+0000 7f46437fe640 1 --2- 172.21.15.28:0/3646507262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138ea0 0x55e01c146cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.152+0000 7f4643fff640 1 -- 172.21.15.28:0/3646507262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138ea0 msgr2=0x55e01c146cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.152+0000 7f4643fff640 1 --2- 172.21.15.28:0/3646507262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138ea0 0x55e01c146cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.152+0000 7f4643fff640 1 -- 172.21.15.28:0/3646507262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01c143570 msgr2=0x55e01c1471f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.152+0000 7f4643fff640 1 --2- 172.21.15.28:0/3646507262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01c143570 0x55e01c1471f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.152+0000 7f4643fff640 1 -- 172.21.15.28:0/3646507262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e01c147760 con 0x55e01c138580 2024-04-25T01:38:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.152+0000 7f4643fff640 1 --2- 172.21.15.28:0/3646507262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 0x55e01c14b840 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f463409a900 tx=0x7f463409abb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d66c045e41790b40 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.152+0000 7f46237fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.152+0000 7f46237fe640 1 -- 172.21.15.28:0/3646507262 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463409b1e0 con 0x55e01c138580 2024-04-25T01:38:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.152+0000 7f46237fe640 1 -- 172.21.15.28:0/3646507262 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f463409a2d0 con 0x55e01c138580 2024-04-25T01:38:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.152+0000 7f464cfe9740 1 -- 172.21.15.28:0/3646507262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f463c036040 con 0x55e01c138580 2024-04-25T01:38:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.152+0000 7f46237fe640 1 -- 172.21.15.28:0/3646507262 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f463409f7c0 con 0x55e01c138580 2024-04-25T01:38:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.152+0000 7f464cfe9740 10 client.17508 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.152+0000 7f464cfe9740 1 -- 172.21.15.28:0/3646507262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e01c1b97f0 con 0x55e01c138580 2024-04-25T01:38:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.152+0000 7f46237fe640 1 -- 172.21.15.28:0/3646507262 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f463c036040 con 0x55e01c138580 2024-04-25T01:38:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.152+0000 7f46237fe640 1 -- 172.21.15.28:0/3646507262 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f46340bb0c0 con 0x55e01c138580 2024-04-25T01:38:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.153+0000 7f464cfe9740 10 client.17508 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.153+0000 7f464cfe9740 10 client.17508 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:38:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.153+0000 7f464cfe9740 20 client.17508 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.153+0000 7f464cfe9740 1 --2- 172.21.15.28:0/3646507262 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55e01c1534c0 0x55e01c1bbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.153+0000 7f464cfe9740 4 client.17508 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:38:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.153+0000 7f464cfe9740 1 -- 172.21.15.28:0/3646507262 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e01c0a4f00 con 0x55e01c1534c0 2024-04-25T01:38:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.153+0000 7f46437fe640 1 --2- 172.21.15.28:0/3646507262 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55e01c1534c0 0x55e01c1bbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.153+0000 7f46437fe640 1 --2- 172.21.15.28:0/3646507262 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55e01c1534c0 0x55e01c1bbfd0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=38f31d0ade23c3e5 server_cookie=87b142c86e6c3e30 in_seq=0 out_seq=0 2024-04-25T01:38:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.153+0000 7f46237fe640 10 client.17508 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:38:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f46237fe640 1 -- 172.21.15.28:0/3646507262 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e01c0a4f00 con 0x55e01c1534c0 2024-04-25T01:38:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f46237fe640 10 client.17508 handle_command_reply: tid=0 2024-04-25T01:38:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464cfe9740 1 client.17508 shutdown 2024-04-25T01:38:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464cfe9740 1 -- 172.21.15.28:0/3646507262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 msgr2=0x55e01c14b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464cfe9740 1 --2- 172.21.15.28:0/3646507262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 0x55e01c14b840 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f463409a900 tx=0x7f463409abb0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464cfe9740 1 -- 172.21.15.28:0/3646507262 shutdown_connections 2024-04-25T01:38:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464cfe9740 1 --2- 172.21.15.28:0/3646507262 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55e01c1534c0 0x55e01c1bbfd0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464cfe9740 1 --2- 172.21.15.28:0/3646507262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01c143570 0x55e01c1471f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464cfe9740 1 --2- 172.21.15.28:0/3646507262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01c138ea0 0x55e01c146cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f46237fe640 0 client.17508 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:38:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464cfe9740 1 --2- 172.21.15.28:0/3646507262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01c138580 0x55e01c14b840 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464cfe9740 1 -- 172.21.15.28:0/3646507262 >> 172.21.15.28:0/3646507262 conn(0x55e01c1348a0 msgr2=0x55e01c14e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:39.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464cfe9740 1 -- 172.21.15.28:0/3646507262 shutdown_connections 2024-04-25T01:38:39.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464cfe9740 1 -- 172.21.15.28:0/3646507262 wait complete. 2024-04-25T01:38:39.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464cfe9740 20 client.17508 trim_cache size 0 max 16384 2024-04-25T01:38:39.141 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:38:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464a2d4640 1 -- 172.21.15.28:0/2878243158 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f462c0a2d00 msgr2=0x7f462c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464a2d4640 1 --2- 172.21.15.28:0/2878243158 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f462c0a2d00 0x7f462c0a51c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f464410d8e0 tx=0x7f4638005f90 comp rx=0 tx=0).stop 2024-04-25T01:38:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464a2d4640 1 -- 172.21.15.28:0/2878243158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4644100610 msgr2=0x7f464419a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464a2d4640 1 --2- 172.21.15.28:0/2878243158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4644100610 0x7f464419a120 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f463400d950 tx=0x7f463400de10 comp rx=0 tx=0).stop 2024-04-25T01:38:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464a2d4640 1 -- 172.21.15.28:0/2878243158 shutdown_connections 2024-04-25T01:38:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464a2d4640 1 --2- 172.21.15.28:0/2878243158 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f462c0a2d00 0x7f462c0a51c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464a2d4640 1 --2- 172.21.15.28:0/2878243158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f46441066b0 0x7f464419e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464a2d4640 1 --2- 172.21.15.28:0/2878243158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4644100fd0 0x7f464419a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.154+0000 7f464a2d4640 1 --2- 172.21.15.28:0/2878243158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4644100610 0x7f464419a120 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.155+0000 7f464a2d4640 1 -- 172.21.15.28:0/2878243158 >> 172.21.15.28:0/2878243158 conn(0x7f4644077740 msgr2=0x7f46440688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:39.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.155+0000 7f464a2d4640 1 -- 172.21.15.28:0/2878243158 shutdown_connections 2024-04-25T01:38:39.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.155+0000 7f464a2d4640 1 -- 172.21.15.28:0/2878243158 wait complete. 2024-04-25T01:38:39.145 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:38 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:38:39.145 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:38 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:38:39.145 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:38 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:38:39.145 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:38 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:38:39.145 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:38 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:38:39.145 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:38 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3743771856' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:38:39.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:38:39.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:38 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:38:39.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:38 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:38:39.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:38 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:38:39.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:38 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:38:39.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:38 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:38:39.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:38 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3743771856' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:38:39.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:38 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:38:39.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:38 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:38:39.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:38 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:38:39.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:38 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:38:39.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:38 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:38:39.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:38 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3743771856' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:38:39.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.270+0000 7fa0eadc1640 1 -- 172.21.15.28:0/1675063445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0e41050d0 msgr2=0x7fa0e410cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.270+0000 7fa0eadc1640 1 --2- 172.21.15.28:0/1675063445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0e41050d0 0x7fa0e410cca0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc009ca0 tx=0x7fa0dc02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.270+0000 7fa0eadc1640 1 -- 172.21.15.28:0/1675063445 shutdown_connections 2024-04-25T01:38:39.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.270+0000 7fa0eadc1640 1 --2- 172.21.15.28:0/1675063445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0e41050d0 0x7fa0e410cca0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.270+0000 7fa0eadc1640 1 --2- 172.21.15.28:0/1675063445 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0e41009b0 0x7fa0e41049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.270+0000 7fa0eadc1640 1 --2- 172.21.15.28:0/1675063445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0e4100000 0x7fa0e41003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.270+0000 7fa0eadc1640 1 -- 172.21.15.28:0/1675063445 >> 172.21.15.28:0/1675063445 conn(0x7fa0e4076c00 msgr2=0x7fa0e40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:39.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.270+0000 7fa0eadc1640 1 -- 172.21.15.28:0/1675063445 shutdown_connections 2024-04-25T01:38:39.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.270+0000 7fa0eadc1640 1 -- 172.21.15.28:0/1675063445 wait complete. 2024-04-25T01:38:39.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.270+0000 7fa0eadc1640 1 Processor -- start 2024-04-25T01:38:39.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.270+0000 7fa0eadc1640 1 -- start start 2024-04-25T01:38:39.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0eadc1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0e4100000 0x7fa0e4197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0eadc1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0e41050d0 0x7fa0e4198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0eadc1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0e419c800 0x7fa0e419ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0eadc1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e4110a60 con 0x7fa0e4100000 2024-04-25T01:38:39.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0eadc1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e41108e0 con 0x7fa0e419c800 2024-04-25T01:38:39.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0eadc1640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e4110be0 con 0x7fa0e41050d0 2024-04-25T01:38:39.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0e8b5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0e4100000 0x7fa0e4197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0e8b5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0e4100000 0x7fa0e4197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45794/0 (socket says 172.21.15.28:45794) 2024-04-25T01:38:39.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0e8b5d640 1 -- 172.21.15.28:0/3698541267 learned_addr learned my addr 172.21.15.28:0/3698541267 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:39.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0e935e640 1 --2- 172.21.15.28:0/3698541267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0e419c800 0x7fa0e419ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0e3fff640 1 --2- 172.21.15.28:0/3698541267 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0e41050d0 0x7fa0e4198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0e8b5d640 1 -- 172.21.15.28:0/3698541267 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0e41050d0 msgr2=0x7fa0e4198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0e8b5d640 1 --2- 172.21.15.28:0/3698541267 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0e41050d0 0x7fa0e4198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0e8b5d640 1 -- 172.21.15.28:0/3698541267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0e419c800 msgr2=0x7fa0e419ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0e8b5d640 1 --2- 172.21.15.28:0/3698541267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0e419c800 0x7fa0e419ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0e8b5d640 1 -- 172.21.15.28:0/3698541267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0e419d2f0 con 0x7fa0e4100000 2024-04-25T01:38:39.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0e8b5d640 1 --2- 172.21.15.28:0/3698541267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0e4100000 0x7fa0e4197f40 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fa0d800c920 tx=0x7fa0d800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:39.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0e1ffb640 1 -- 172.21.15.28:0/3698541267 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8007c70 con 0x7fa0e4100000 2024-04-25T01:38:39.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0eadc1640 1 -- 172.21.15.28:0/3698541267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0e419d5e0 con 0x7fa0e4100000 2024-04-25T01:38:39.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0e1ffb640 1 -- 172.21.15.28:0/3698541267 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0d8007e10 con 0x7fa0e4100000 2024-04-25T01:38:39.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.271+0000 7fa0e1ffb640 1 -- 172.21.15.28:0/3698541267 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8018890 con 0x7fa0e4100000 2024-04-25T01:38:39.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.272+0000 7fa0eadc1640 1 -- 172.21.15.28:0/3698541267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0e41a4dd0 con 0x7fa0e4100000 2024-04-25T01:38:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.274+0000 7fa0e1ffb640 1 -- 172.21.15.28:0/3698541267 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa0d8018a70 con 0x7fa0e4100000 2024-04-25T01:38:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0edbad740 1 Processor -- start 2024-04-25T01:38:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0edbad740 1 -- start start 2024-04-25T01:38:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0edbad740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e7f580 0x555ce6e7f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0edbad740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7fea0 0x555ce6e8a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0edbad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e8a570 0x555ce6e8c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0edbad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555ce6de4670 con 0x555ce6e8a570 2024-04-25T01:38:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0edbad740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555ce6d8c970 con 0x555ce6e7f580 2024-04-25T01:38:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0edbad740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x555ce6de0630 con 0x555ce6e7fea0 2024-04-25T01:38:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0e1ffb640 1 --2- 172.21.15.28:0/3698541267 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa0d40a2d00 0x7fa0d40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0e3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e8a570 0x555ce6e8c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0e3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e8a570 0x555ce6e8c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45800/0 (socket says 172.21.15.28:45800) 2024-04-25T01:38:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0e935e640 1 --2- 172.21.15.28:0/3698541267 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa0d40a2d00 0x7fa0d40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0e3fff640 1 -- 172.21.15.28:0/4124851685 learned_addr learned my addr 172.21.15.28:0/4124851685 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0e1ffb640 1 -- 172.21.15.28:0/3698541267 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fa0d80cf9f0 con 0x7fa0e4100000 2024-04-25T01:38:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0e935e640 1 --2- 172.21.15.28:0/4124851685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e7f580 0x555ce6e7f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0e8b5d640 1 --2- 172.21.15.28:0/4124851685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7fea0 0x555ce6e8a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0e3fff640 1 -- 172.21.15.28:0/4124851685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7fea0 msgr2=0x555ce6e8a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0e3fff640 1 --2- 172.21.15.28:0/4124851685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7fea0 0x555ce6e8a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0e3fff640 1 -- 172.21.15.28:0/4124851685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e7f580 msgr2=0x555ce6e7f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0e3fff640 1 --2- 172.21.15.28:0/4124851685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e7f580 0x555ce6e7f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0e3fff640 1 -- 172.21.15.28:0/4124851685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ce6db0400 con 0x555ce6e8a570 2024-04-25T01:38:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.275+0000 7fa0e8b5d640 1 --2- 172.21.15.28:0/4124851685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7fea0 0x555ce6e8a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.276+0000 7fa0e935e640 1 --2- 172.21.15.28:0/3698541267 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa0d40a2d00 0x7fa0d40a51c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc005e30 tx=0x7fa0dc02f3d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.276+0000 7fa0e3fff640 1 --2- 172.21.15.28:0/4124851685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e8a570 0x555ce6e8c960 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0005fd0 tx=0x7fa0d0009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f4f85771e4daa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.276+0000 7fa0ceffd640 1 -- 172.21.15.28:0/4124851685 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0016070 con 0x555ce6e8a570 2024-04-25T01:38:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.276+0000 7fa0ceffd640 1 -- 172.21.15.28:0/4124851685 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0d0005500 con 0x555ce6e8a570 2024-04-25T01:38:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.276+0000 7fa0ceffd640 1 -- 172.21.15.28:0/4124851685 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0004e60 con 0x555ce6e8a570 2024-04-25T01:38:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.276+0000 7fa0edbad740 1 -- 172.21.15.28:0/4124851685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e8a570 msgr2=0x555ce6e8c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.276+0000 7fa0edbad740 1 --2- 172.21.15.28:0/4124851685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e8a570 0x555ce6e8c960 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0005fd0 tx=0x7fa0d0009f20 comp rx=0 tx=0).stop 2024-04-25T01:38:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.277+0000 7fa0edbad740 1 -- 172.21.15.28:0/4124851685 shutdown_connections 2024-04-25T01:38:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.277+0000 7fa0edbad740 1 --2- 172.21.15.28:0/4124851685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e8a570 0x555ce6e8c960 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.277+0000 7fa0edbad740 1 --2- 172.21.15.28:0/4124851685 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7fea0 0x555ce6e8a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.277+0000 7fa0edbad740 1 --2- 172.21.15.28:0/4124851685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e7f580 0x555ce6e7f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.277+0000 7fa0edbad740 1 -- 172.21.15.28:0/4124851685 >> 172.21.15.28:0/4124851685 conn(0x555ce6e7b8a0 msgr2=0x555ce6e7bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.277+0000 7fa0edbad740 1 -- 172.21.15.28:0/4124851685 shutdown_connections 2024-04-25T01:38:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.277+0000 7fa0edbad740 1 -- 172.21.15.28:0/4124851685 wait complete. 2024-04-25T01:38:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.277+0000 7fa0edbad740 1 -- start start 2024-04-25T01:38:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.277+0000 7fa0edbad740 1 Processor -- start 2024-04-25T01:38:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.277+0000 7fa0edbad740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7f580 0x555ce6e89ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.277+0000 7fa0edbad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7fea0 0x555ce6e87190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.277+0000 7fa0edbad740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e8a570 0x555ce6e876d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.277+0000 7fa0edbad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555ce6e8d360 con 0x555ce6e7fea0 2024-04-25T01:38:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.277+0000 7fa0edbad740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555ce6da0050 con 0x555ce6e8a570 2024-04-25T01:38:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.277+0000 7fa0edbad740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x555ce6e8d4e0 con 0x555ce6e7f580 2024-04-25T01:38:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.277+0000 7fa0edbad740 10 client.0 _ll_register_callbacks cb 0x555ce6af7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0e3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7fea0 0x555ce6e87190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0e3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7fea0 0x555ce6e87190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45808/0 (socket says 172.21.15.28:45808) 2024-04-25T01:38:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0e3fff640 1 -- 172.21.15.28:0/776241369 learned_addr learned my addr 172.21.15.28:0/776241369 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0e935e640 1 --2- 172.21.15.28:0/776241369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e8a570 0x555ce6e876d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0e8b5d640 1 --2- 172.21.15.28:0/776241369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7f580 0x555ce6e89ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0e3fff640 1 -- 172.21.15.28:0/776241369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7f580 msgr2=0x555ce6e89ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0e3fff640 1 --2- 172.21.15.28:0/776241369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7f580 0x555ce6e89ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0e3fff640 1 -- 172.21.15.28:0/776241369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e8a570 msgr2=0x555ce6e876d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0e3fff640 1 --2- 172.21.15.28:0/776241369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e8a570 0x555ce6e876d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0e3fff640 1 -- 172.21.15.28:0/776241369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ce6d71dc0 con 0x555ce6e7fea0 2024-04-25T01:38:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0e8b5d640 1 --2- 172.21.15.28:0/776241369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7f580 0x555ce6e89ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0e3fff640 1 --2- 172.21.15.28:0/776241369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7fea0 0x555ce6e87190 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0004a00 tx=0x7fa0d00058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee305db79a05b77 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0c3fff640 1 -- 172.21.15.28:0/776241369 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0028060 con 0x555ce6e7fea0 2024-04-25T01:38:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0c3fff640 1 -- 172.21.15.28:0/776241369 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fa0d0004df0 con 0x555ce6e7fea0 2024-04-25T01:38:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0edbad740 1 -- 172.21.15.28:0/776241369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0d0005500 con 0x555ce6e7fea0 2024-04-25T01:38:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0c3fff640 1 -- 172.21.15.28:0/776241369 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa0d0016040 con 0x555ce6e7fea0 2024-04-25T01:38:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0edbad740 10 client.17520 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0edbad740 1 -- 172.21.15.28:0/776241369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555ce6af83f0 con 0x555ce6e7fea0 2024-04-25T01:38:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.278+0000 7fa0c3fff640 1 -- 172.21.15.28:0/776241369 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0d0005500 con 0x555ce6e7fea0 2024-04-25T01:38:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.279+0000 7fa0c3fff640 1 -- 172.21.15.28:0/776241369 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa0d000f040 con 0x555ce6e7fea0 2024-04-25T01:38:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.279+0000 7fa0edbad740 10 client.17520 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.279+0000 7fa0edbad740 10 client.17520 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.279+0000 7fa0edbad740 20 client.17520 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.279+0000 7fa0edbad740 1 --2- 172.21.15.28:0/776241369 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555ce6e9a540 0x555ce6e9c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.279+0000 7fa0edbad740 4 client.17520 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.279+0000 7fa0edbad740 1 -- 172.21.15.28:0/776241369 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ce6df1bb0 con 0x555ce6e9a540 2024-04-25T01:38:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.279+0000 7fa0e8b5d640 1 --2- 172.21.15.28:0/776241369 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555ce6e9a540 0x555ce6e9c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.279+0000 7fa0c3fff640 10 client.17520 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.279+0000 7fa0e8b5d640 1 --2- 172.21.15.28:0/776241369 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555ce6e9a540 0x555ce6e9c930 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee08d8a9d5716571 server_cookie=24fe6aeb6241d22c in_seq=0 out_seq=0 2024-04-25T01:38:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.280+0000 7fa0c3fff640 1 -- 172.21.15.28:0/776241369 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555ce6df1bb0 con 0x555ce6e9a540 2024-04-25T01:38:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.280+0000 7fa0c3fff640 10 client.17520 handle_command_reply: tid=0 2024-04-25T01:38:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.280+0000 7fa0edbad740 1 client.17520 shutdown 2024-04-25T01:38:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.280+0000 7fa0edbad740 1 -- 172.21.15.28:0/776241369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7fea0 msgr2=0x555ce6e87190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.280+0000 7fa0edbad740 1 --2- 172.21.15.28:0/776241369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7fea0 0x555ce6e87190 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0004a00 tx=0x7fa0d00058d0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.280+0000 7fa0edbad740 1 -- 172.21.15.28:0/776241369 shutdown_connections 2024-04-25T01:38:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.280+0000 7fa0edbad740 1 --2- 172.21.15.28:0/776241369 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555ce6e9a540 0x555ce6e9c930 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.280+0000 7fa0edbad740 1 --2- 172.21.15.28:0/776241369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e8a570 0x555ce6e876d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.280+0000 7fa0edbad740 1 --2- 172.21.15.28:0/776241369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7fea0 0x555ce6e87190 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.280+0000 7fa0edbad740 1 --2- 172.21.15.28:0/776241369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7f580 0x555ce6e89ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.280+0000 7fa0c3fff640 0 client.17520 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.280+0000 7fa0edbad740 1 -- 172.21.15.28:0/776241369 >> 172.21.15.28:0/776241369 conn(0x555ce6e7b8a0 msgr2=0x555ce6e7ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:39.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.280+0000 7fa0edbad740 1 -- 172.21.15.28:0/776241369 shutdown_connections 2024-04-25T01:38:39.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.280+0000 7fa0edbad740 1 -- 172.21.15.28:0/776241369 wait complete. 2024-04-25T01:38:39.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.280+0000 7fa0edbad740 20 client.17520 trim_cache size 0 max 16384 2024-04-25T01:38:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.294+0000 7fa0edbad740 1 Processor -- start 2024-04-25T01:38:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.294+0000 7fa0edbad740 1 -- start start 2024-04-25T01:38:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.294+0000 7fa0edbad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7b8a0 0x555ce6e96220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.294+0000 7fa0edbad740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e7f580 0x555ce6e86040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.294+0000 7fa0edbad740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7fea0 0x555ce6e899d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.294+0000 7fa0edbad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e41ae9c0 con 0x555ce6e7b8a0 2024-04-25T01:38:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.294+0000 7fa0edbad740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e41b0500 con 0x555ce6e7f580 2024-04-25T01:38:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.294+0000 7fa0edbad740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e41af090 con 0x555ce6e7fea0 2024-04-25T01:38:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.294+0000 7fa0e8b5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7b8a0 0x555ce6e96220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.294+0000 7fa0e8b5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7b8a0 0x555ce6e96220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45814/0 (socket says 172.21.15.28:45814) 2024-04-25T01:38:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.294+0000 7fa0e8b5d640 1 -- 172.21.15.28:0/361433215 learned_addr learned my addr 172.21.15.28:0/361433215 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.294+0000 7fa0e935e640 1 --2- 172.21.15.28:0/361433215 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7fea0 0x555ce6e899d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.294+0000 7fa0e8b5d640 1 -- 172.21.15.28:0/361433215 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7fea0 msgr2=0x555ce6e899d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.294+0000 7fa0e3fff640 1 --2- 172.21.15.28:0/361433215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e7f580 0x555ce6e86040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.294+0000 7fa0e8b5d640 1 --2- 172.21.15.28:0/361433215 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7fea0 0x555ce6e899d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.294+0000 7fa0e8b5d640 1 -- 172.21.15.28:0/361433215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e7f580 msgr2=0x555ce6e86040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.294+0000 7fa0e8b5d640 1 --2- 172.21.15.28:0/361433215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e7f580 0x555ce6e86040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.294+0000 7fa0e8b5d640 1 -- 172.21.15.28:0/361433215 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ce6bc27d0 con 0x555ce6e7b8a0 2024-04-25T01:38:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.294+0000 7fa0e935e640 1 --2- 172.21.15.28:0/361433215 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7fea0 0x555ce6e899d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.295+0000 7fa0e8b5d640 1 --2- 172.21.15.28:0/361433215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7b8a0 0x555ce6e96220 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8097fd0 tx=0x7fa0d8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39aa6ac65fe72f54 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.295+0000 7fa0ce7fc640 1 -- 172.21.15.28:0/361433215 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d80a47e0 con 0x555ce6e7b8a0 2024-04-25T01:38:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.295+0000 7fa0ce7fc640 1 -- 172.21.15.28:0/361433215 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0d8099e70 con 0x555ce6e7b8a0 2024-04-25T01:38:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.295+0000 7fa0ce7fc640 1 -- 172.21.15.28:0/361433215 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d80cd020 con 0x555ce6e7b8a0 2024-04-25T01:38:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.295+0000 7fa0edbad740 1 -- 172.21.15.28:0/361433215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7b8a0 msgr2=0x555ce6e96220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.295+0000 7fa0edbad740 1 --2- 172.21.15.28:0/361433215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7b8a0 0x555ce6e96220 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8097fd0 tx=0x7fa0d8000bc0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.295+0000 7fa0edbad740 1 -- 172.21.15.28:0/361433215 shutdown_connections 2024-04-25T01:38:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.295+0000 7fa0edbad740 1 --2- 172.21.15.28:0/361433215 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7fea0 0x555ce6e899d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.295+0000 7fa0edbad740 1 --2- 172.21.15.28:0/361433215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e7f580 0x555ce6e86040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.295+0000 7fa0edbad740 1 --2- 172.21.15.28:0/361433215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7b8a0 0x555ce6e96220 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.295+0000 7fa0edbad740 1 -- 172.21.15.28:0/361433215 >> 172.21.15.28:0/361433215 conn(0x555ce6e87830 msgr2=0x555ce6e85930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.295+0000 7fa0edbad740 1 -- 172.21.15.28:0/361433215 shutdown_connections 2024-04-25T01:38:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.295+0000 7fa0edbad740 1 -- 172.21.15.28:0/361433215 wait complete. 2024-04-25T01:38:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.295+0000 7fa0edbad740 1 -- start start 2024-04-25T01:38:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.296+0000 7fa0edbad740 1 Processor -- start 2024-04-25T01:38:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.296+0000 7fa0edbad740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e7b8a0 0x555ce6e92c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.296+0000 7fa0edbad740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7f580 0x555ce6e8df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.296+0000 7fa0edbad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7fea0 0x555ce6e8e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.296+0000 7fa0edbad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e41afe30 con 0x555ce6e7fea0 2024-04-25T01:38:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.296+0000 7fa0edbad740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e41af760 con 0x555ce6e7b8a0 2024-04-25T01:38:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.296+0000 7fa0edbad740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e41ac9a0 con 0x555ce6e7f580 2024-04-25T01:38:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.296+0000 7fa0edbad740 10 client.0 _ll_register_callbacks cb 0x7fa0dc009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.296+0000 7fa0e935e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7fea0 0x555ce6e8e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.296+0000 7fa0e935e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7fea0 0x555ce6e8e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45822/0 (socket says 172.21.15.28:45822) 2024-04-25T01:38:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.296+0000 7fa0e935e640 1 -- 172.21.15.28:0/3849909085 learned_addr learned my addr 172.21.15.28:0/3849909085 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.296+0000 7fa0e8b5d640 1 --2- 172.21.15.28:0/3849909085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e7b8a0 0x555ce6e92c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.296+0000 7fa0e3fff640 1 --2- 172.21.15.28:0/3849909085 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7f580 0x555ce6e8df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.296+0000 7fa0e935e640 1 -- 172.21.15.28:0/3849909085 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7f580 msgr2=0x555ce6e8df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.296+0000 7fa0e935e640 1 --2- 172.21.15.28:0/3849909085 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7f580 0x555ce6e8df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.296+0000 7fa0e935e640 1 -- 172.21.15.28:0/3849909085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e7b8a0 msgr2=0x555ce6e92c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.296+0000 7fa0e935e640 1 --2- 172.21.15.28:0/3849909085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e7b8a0 0x555ce6e92c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.296+0000 7fa0e935e640 1 -- 172.21.15.28:0/3849909085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ce6e8ed80 con 0x555ce6e7fea0 2024-04-25T01:38:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.296+0000 7fa0e3fff640 1 --2- 172.21.15.28:0/3849909085 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7f580 0x555ce6e8df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.297+0000 7fa0e935e640 1 --2- 172.21.15.28:0/3849909085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7fea0 0x555ce6e8e580 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc009ca0 tx=0x7fa0dc047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d5805cd06872ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.297+0000 7fa0c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.297+0000 7fa0c3fff640 1 -- 172.21.15.28:0/3849909085 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc031720 con 0x555ce6e7fea0 2024-04-25T01:38:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.297+0000 7fa0c3fff640 1 -- 172.21.15.28:0/3849909085 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fa0dc0318c0 con 0x555ce6e7fea0 2024-04-25T01:38:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.297+0000 7fa0edbad740 1 -- 172.21.15.28:0/3849909085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0d8099e70 con 0x555ce6e7fea0 2024-04-25T01:38:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.297+0000 7fa0c3fff640 1 -- 172.21.15.28:0/3849909085 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa0dc05e050 con 0x555ce6e7fea0 2024-04-25T01:38:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.297+0000 7fa0edbad740 10 client.17526 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.297+0000 7fa0edbad740 1 -- 172.21.15.28:0/3849909085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555ce6f00870 con 0x555ce6e7fea0 2024-04-25T01:38:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.297+0000 7fa0c3fff640 1 -- 172.21.15.28:0/3849909085 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0d8099e70 con 0x555ce6e7fea0 2024-04-25T01:38:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.297+0000 7fa0c3fff640 1 -- 172.21.15.28:0/3849909085 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa0dc02fa80 con 0x555ce6e7fea0 2024-04-25T01:38:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.297+0000 7fa0edbad740 10 client.17526 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.297+0000 7fa0edbad740 10 client.17526 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.297+0000 7fa0edbad740 20 client.17526 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.297+0000 7fa0edbad740 1 --2- 172.21.15.28:0/3849909085 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555ce6e8a570 0x555ce6f030b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.297+0000 7fa0edbad740 4 client.17526 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:38:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.297+0000 7fa0edbad740 1 -- 172.21.15.28:0/3849909085 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555ce6debf00 con 0x555ce6e8a570 2024-04-25T01:38:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.298+0000 7fa0e8b5d640 1 --2- 172.21.15.28:0/3849909085 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555ce6e8a570 0x555ce6f030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.298+0000 7fa0e8b5d640 1 --2- 172.21.15.28:0/3849909085 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555ce6e8a570 0x555ce6f030b0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb5eacbebbc6cba9 server_cookie=962717e9e88c0d34 in_seq=0 out_seq=0 2024-04-25T01:38:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.298+0000 7fa0c3fff640 10 client.17526 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.298+0000 7fa0c3fff640 1 -- 172.21.15.28:0/3849909085 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555ce6debf00 con 0x555ce6e8a570 2024-04-25T01:38:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.298+0000 7fa0c3fff640 10 client.17526 handle_command_reply: tid=0 2024-04-25T01:38:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.298+0000 7fa0edbad740 1 client.17526 shutdown 2024-04-25T01:38:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.298+0000 7fa0edbad740 1 -- 172.21.15.28:0/3849909085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7fea0 msgr2=0x555ce6e8e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.298+0000 7fa0edbad740 1 --2- 172.21.15.28:0/3849909085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7fea0 0x555ce6e8e580 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc009ca0 tx=0x7fa0dc047000 comp rx=0 tx=0).stop 2024-04-25T01:38:39.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.298+0000 7fa0edbad740 1 -- 172.21.15.28:0/3849909085 shutdown_connections 2024-04-25T01:38:39.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.298+0000 7fa0edbad740 1 --2- 172.21.15.28:0/3849909085 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555ce6e8a570 0x555ce6f030b0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.298+0000 7fa0edbad740 1 --2- 172.21.15.28:0/3849909085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ce6e7fea0 0x555ce6e8e580 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.298+0000 7fa0edbad740 1 --2- 172.21.15.28:0/3849909085 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ce6e7f580 0x555ce6e8df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.298+0000 7fa0c3fff640 0 client.17526 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:39.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.298+0000 7fa0edbad740 1 --2- 172.21.15.28:0/3849909085 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ce6e7b8a0 0x555ce6e92c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.298+0000 7fa0edbad740 1 -- 172.21.15.28:0/3849909085 >> 172.21.15.28:0/3849909085 conn(0x555ce6e87830 msgr2=0x555ce6e861c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:39.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.299+0000 7fa0edbad740 1 -- 172.21.15.28:0/3849909085 shutdown_connections 2024-04-25T01:38:39.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.299+0000 7fa0edbad740 1 -- 172.21.15.28:0/3849909085 wait complete. 2024-04-25T01:38:39.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.299+0000 7fa0edbad740 20 client.17526 trim_cache size 0 max 16384 2024-04-25T01:38:39.286 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:38:39.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.299+0000 7fa0eadc1640 1 -- 172.21.15.28:0/3698541267 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa0d40a2d00 msgr2=0x7fa0d40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.299+0000 7fa0eadc1640 1 --2- 172.21.15.28:0/3698541267 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa0d40a2d00 0x7fa0d40a51c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc005e30 tx=0x7fa0dc02f3d0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.299+0000 7fa0eadc1640 1 -- 172.21.15.28:0/3698541267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0e4100000 msgr2=0x7fa0e4197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:39.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.299+0000 7fa0eadc1640 1 --2- 172.21.15.28:0/3698541267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0e4100000 0x7fa0e4197f40 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fa0d800c920 tx=0x7fa0d800cde0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.299+0000 7fa0eadc1640 1 -- 172.21.15.28:0/3698541267 shutdown_connections 2024-04-25T01:38:39.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.299+0000 7fa0eadc1640 1 --2- 172.21.15.28:0/3698541267 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa0d40a2d00 0x7fa0d40a51c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.299+0000 7fa0eadc1640 1 --2- 172.21.15.28:0/3698541267 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa0e419c800 0x7fa0e419ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.299+0000 7fa0eadc1640 1 --2- 172.21.15.28:0/3698541267 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa0e41050d0 0x7fa0e4198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.299+0000 7fa0eadc1640 1 --2- 172.21.15.28:0/3698541267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0e4100000 0x7fa0e4197f40 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:39.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.299+0000 7fa0eadc1640 1 -- 172.21.15.28:0/3698541267 >> 172.21.15.28:0/3698541267 conn(0x7fa0e4076c00 msgr2=0x7fa0e41034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:39.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.299+0000 7fa0eadc1640 1 -- 172.21.15.28:0/3698541267 shutdown_connections 2024-04-25T01:38:39.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:39.299+0000 7fa0eadc1640 1 -- 172.21.15.28:0/3698541267 wait complete. 2024-04-25T01:38:40.211 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:39 smithi130 ceph-mon[32702]: pgmap v172: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:38:40.236 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:39 smithi032 ceph-mon[32461]: pgmap v172: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:38:40.291 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:38:40.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:38:40.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:38:40.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:39 smithi028 ceph-mon[29390]: pgmap v172: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:38:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.429+0000 7fb947cd4640 1 -- 172.21.15.28:0/4227193817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb94006a6e0 msgr2=0x7fb94010dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.429+0000 7fb947cd4640 1 --2- 172.21.15.28:0/4227193817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb94006a6e0 0x7fb94010dc10 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fb930009900 tx=0x7fb93002a410 comp rx=0 tx=0).stop 2024-04-25T01:38:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.429+0000 7fb947cd4640 1 -- 172.21.15.28:0/4227193817 shutdown_connections 2024-04-25T01:38:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.429+0000 7fb947cd4640 1 --2- 172.21.15.28:0/4227193817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb94010e150 0x7fb940110540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.429+0000 7fb947cd4640 1 --2- 172.21.15.28:0/4227193817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb94006a6e0 0x7fb94010dc10 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.429+0000 7fb947cd4640 1 --2- 172.21.15.28:0/4227193817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb940069dc0 0x7fb94006a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.429+0000 7fb947cd4640 1 -- 172.21.15.28:0/4227193817 >> 172.21.15.28:0/4227193817 conn(0x7fb9400776a0 msgr2=0x7fb940077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.430+0000 7fb947cd4640 1 -- 172.21.15.28:0/4227193817 shutdown_connections 2024-04-25T01:38:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.430+0000 7fb947cd4640 1 -- 172.21.15.28:0/4227193817 wait complete. 2024-04-25T01:38:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.430+0000 7fb947cd4640 1 Processor -- start 2024-04-25T01:38:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.430+0000 7fb947cd4640 1 -- start start 2024-04-25T01:38:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.430+0000 7fb947cd4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb940069dc0 0x7fb94019a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.430+0000 7fb947cd4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb94006a6e0 0x7fb94019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.430+0000 7fb947cd4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb94010e150 0x7fb94019ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.430+0000 7fb947cd4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb940113d10 con 0x7fb94006a6e0 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.430+0000 7fb947cd4640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb940113b90 con 0x7fb940069dc0 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.430+0000 7fb947cd4640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb940113e90 con 0x7fb94010e150 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.430+0000 7fb94526f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb94006a6e0 0x7fb94019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.430+0000 7fb94526f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb94006a6e0 0x7fb94019a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45838/0 (socket says 172.21.15.28:45838) 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.431+0000 7fb94526f640 1 -- 172.21.15.28:0/3025067387 learned_addr learned my addr 172.21.15.28:0/3025067387 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.431+0000 7fb946271640 1 --2- 172.21.15.28:0/3025067387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb94010e150 0x7fb94019ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.431+0000 7fb945a70640 1 --2- 172.21.15.28:0/3025067387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb940069dc0 0x7fb94019a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.431+0000 7fb94526f640 1 -- 172.21.15.28:0/3025067387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb94010e150 msgr2=0x7fb94019ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.431+0000 7fb94526f640 1 --2- 172.21.15.28:0/3025067387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb94010e150 0x7fb94019ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.431+0000 7fb94526f640 1 -- 172.21.15.28:0/3025067387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb940069dc0 msgr2=0x7fb94019a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.431+0000 7fb94526f640 1 --2- 172.21.15.28:0/3025067387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb940069dc0 0x7fb94019a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.431+0000 7fb94526f640 1 -- 172.21.15.28:0/3025067387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb94019f180 con 0x7fb94006a6e0 2024-04-25T01:38:40.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.431+0000 7fb94526f640 1 --2- 172.21.15.28:0/3025067387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb94006a6e0 0x7fb94019a6d0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fb930006010 tx=0x7fb9300042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:40.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.431+0000 7fb936ffd640 1 -- 172.21.15.28:0/3025067387 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93003f030 con 0x7fb94006a6e0 2024-04-25T01:38:40.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.431+0000 7fb936ffd640 1 -- 172.21.15.28:0/3025067387 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb930003ce0 con 0x7fb94006a6e0 2024-04-25T01:38:40.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.431+0000 7fb947cd4640 1 -- 172.21.15.28:0/3025067387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb94019f410 con 0x7fb94006a6e0 2024-04-25T01:38:40.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.431+0000 7fb936ffd640 1 -- 172.21.15.28:0/3025067387 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb930035070 con 0x7fb94006a6e0 2024-04-25T01:38:40.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.431+0000 7fb947cd4640 1 -- 172.21.15.28:0/3025067387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9401a6d00 con 0x7fb94006a6e0 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.434+0000 7fb936ffd640 1 -- 172.21.15.28:0/3025067387 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb930042030 con 0x7fb94006a6e0 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.434+0000 7fb94a9e9740 1 Processor -- start 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.435+0000 7fb94a9e9740 1 -- start start 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.435+0000 7fb94a9e9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674cc980 0x55ef674cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.435+0000 7fb94a9e9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cb290 0x55ef674cb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.435+0000 7fb94a9e9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674d6010 0x55ef674d8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.435+0000 7fb94a9e9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ef6742f2c0 con 0x55ef674cb290 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.435+0000 7fb94a9e9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ef673d7910 con 0x55ef674d6010 2024-04-25T01:38:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.435+0000 7fb94a9e9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ef6742b280 con 0x55ef674cc980 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.435+0000 7fb945a70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cb290 0x55ef674cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.435+0000 7fb945a70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cb290 0x55ef674cb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45844/0 (socket says 172.21.15.28:45844) 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.435+0000 7fb945a70640 1 -- 172.21.15.28:0/115637435 learned_addr learned my addr 172.21.15.28:0/115637435 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.435+0000 7fb946271640 1 --2- 172.21.15.28:0/115637435 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674cc980 0x55ef674cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.435+0000 7fb94526f640 1 --2- 172.21.15.28:0/115637435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674d6010 0x55ef674d8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.436+0000 7fb936ffd640 1 --2- 172.21.15.28:0/3025067387 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb92c0a2d80 0x7fb92c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.436+0000 7fb945a70640 1 -- 172.21.15.28:0/115637435 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674cc980 msgr2=0x55ef674cacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.436+0000 7fb945a70640 1 --2- 172.21.15.28:0/115637435 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674cc980 0x55ef674cacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.436+0000 7fb945a70640 1 -- 172.21.15.28:0/115637435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674d6010 msgr2=0x55ef674d8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.436+0000 7fb945a70640 1 --2- 172.21.15.28:0/115637435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674d6010 0x55ef674d8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.436+0000 7fb945a70640 1 -- 172.21.15.28:0/115637435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef673e8fb0 con 0x55ef674cb290 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.436+0000 7fb946271640 1 --2- 172.21.15.28:0/3025067387 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb92c0a2d80 0x7fb92c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.436+0000 7fb936ffd640 1 -- 172.21.15.28:0/3025067387 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fb9300efcd0 con 0x7fb94006a6e0 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.436+0000 7fb945a70640 1 --2- 172.21.15.28:0/115637435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cb290 0x55ef674cb710 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00cc60 tx=0x7fb93c00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5d57df58f2e754b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.436+0000 7fb92bfff640 1 -- 172.21.15.28:0/115637435 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93c014070 con 0x55ef674cb290 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.436+0000 7fb92bfff640 1 -- 172.21.15.28:0/115637435 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb93c0047a0 con 0x55ef674cb290 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.436+0000 7fb92bfff640 1 -- 172.21.15.28:0/115637435 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93c010790 con 0x55ef674cb290 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.437+0000 7fb94a9e9740 1 -- 172.21.15.28:0/115637435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cb290 msgr2=0x55ef674cb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.437+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/115637435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cb290 0x55ef674cb710 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00cc60 tx=0x7fb93c00e720 comp rx=0 tx=0).stop 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.437+0000 7fb946271640 1 --2- 172.21.15.28:0/3025067387 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb92c0a2d80 0x7fb92c0a5240 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb938005c40 tx=0x7fb938004a40 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.437+0000 7fb94a9e9740 1 -- 172.21.15.28:0/115637435 shutdown_connections 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.437+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/115637435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674d6010 0x55ef674d8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.437+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/115637435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cb290 0x55ef674cb710 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.437+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/115637435 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674cc980 0x55ef674cacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.437+0000 7fb94a9e9740 1 -- 172.21.15.28:0/115637435 >> 172.21.15.28:0/115637435 conn(0x55ef67493950 msgr2=0x55ef674c77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.437+0000 7fb94a9e9740 1 -- 172.21.15.28:0/115637435 shutdown_connections 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.437+0000 7fb94a9e9740 1 -- 172.21.15.28:0/115637435 wait complete. 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.437+0000 7fb94a9e9740 1 -- start start 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.437+0000 7fb94a9e9740 1 Processor -- start 2024-04-25T01:38:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.438+0000 7fb94a9e9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674cb290 0x55ef674d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.438+0000 7fb94a9e9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 0x55ef674d33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.438+0000 7fb94a9e9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674d6010 0x55ef674d3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.438+0000 7fb94a9e9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ef674d8ea0 con 0x55ef674cc980 2024-04-25T01:38:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.438+0000 7fb94a9e9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ef673eae90 con 0x55ef674d6010 2024-04-25T01:38:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.438+0000 7fb94a9e9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ef674d9020 con 0x55ef674cb290 2024-04-25T01:38:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.438+0000 7fb94a9e9740 10 client.0 _ll_register_callbacks cb 0x55ef67426970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.438+0000 7fb94526f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 0x55ef674d33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.438+0000 7fb94526f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 0x55ef674d33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45848/0 (socket says 172.21.15.28:45848) 2024-04-25T01:38:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.438+0000 7fb94526f640 1 -- 172.21.15.28:0/3859780343 learned_addr learned my addr 172.21.15.28:0/3859780343 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.438+0000 7fb946271640 1 --2- 172.21.15.28:0/3859780343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674d6010 0x55ef674d3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.438+0000 7fb945a70640 1 --2- 172.21.15.28:0/3859780343 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674cb290 0x55ef674d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.438+0000 7fb94526f640 1 -- 172.21.15.28:0/3859780343 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674cb290 msgr2=0x55ef674d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.438+0000 7fb94526f640 1 --2- 172.21.15.28:0/3859780343 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674cb290 0x55ef674d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.438+0000 7fb94526f640 1 -- 172.21.15.28:0/3859780343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674d6010 msgr2=0x55ef674d3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.438+0000 7fb94526f640 1 --2- 172.21.15.28:0/3859780343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674d6010 0x55ef674d3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.438+0000 7fb94526f640 1 -- 172.21.15.28:0/3859780343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef6720b7e0 con 0x55ef674cc980 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.438+0000 7fb946271640 1 --2- 172.21.15.28:0/3859780343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674d6010 0x55ef674d3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.439+0000 7fb94526f640 1 --2- 172.21.15.28:0/3859780343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 0x55ef674d33c0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fb9300bbc70 tx=0x7fb9300f3410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b2f4f75718f539 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.439+0000 7fb9297fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.439+0000 7fb9297fa640 1 -- 172.21.15.28:0/3859780343 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb930049070 con 0x55ef674cc980 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.439+0000 7fb9297fa640 1 -- 172.21.15.28:0/3859780343 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fb9300bc1d0 con 0x55ef674cc980 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.439+0000 7fb94a9e9740 1 -- 172.21.15.28:0/3859780343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb93c0047a0 con 0x55ef674cc980 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.439+0000 7fb9297fa640 1 -- 172.21.15.28:0/3859780343 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb9300b8210 con 0x55ef674cc980 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.439+0000 7fb94a9e9740 10 client.17538 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.439+0000 7fb94a9e9740 1 -- 172.21.15.28:0/3859780343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ef6726bf70 con 0x55ef674cc980 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.439+0000 7fb9297fa640 1 -- 172.21.15.28:0/3859780343 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9300d8020 con 0x55ef674cc980 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.439+0000 7fb9297fa640 1 -- 172.21.15.28:0/3859780343 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb9300c4750 con 0x55ef674cc980 2024-04-25T01:38:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.439+0000 7fb94a9e9740 10 client.17538 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.439+0000 7fb94a9e9740 10 client.17538 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.439+0000 7fb94a9e9740 20 client.17538 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.439+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/3859780343 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ef674e62e0 0x55ef674e8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.439+0000 7fb94a9e9740 4 client.17538 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.439+0000 7fb94a9e9740 1 -- 172.21.15.28:0/3859780343 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef674035a0 con 0x55ef674e62e0 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.440+0000 7fb945a70640 1 --2- 172.21.15.28:0/3859780343 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ef674e62e0 0x55ef674e8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.440+0000 7fb9297fa640 10 client.17538 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.440+0000 7fb945a70640 1 --2- 172.21.15.28:0/3859780343 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ef674e62e0 0x55ef674e8710 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12408dce4f39fd91 server_cookie=a77222e6cedcc313 in_seq=0 out_seq=0 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.441+0000 7fb9297fa640 1 -- 172.21.15.28:0/3859780343 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ef674035a0 con 0x55ef674e62e0 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.441+0000 7fb9297fa640 10 client.17538 handle_command_reply: tid=0 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.441+0000 7fb94a9e9740 1 client.17538 shutdown 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.441+0000 7fb94a9e9740 1 -- 172.21.15.28:0/3859780343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 msgr2=0x55ef674d33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.441+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/3859780343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 0x55ef674d33c0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fb9300bbc70 tx=0x7fb9300f3410 comp rx=0 tx=0).stop 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.441+0000 7fb94a9e9740 1 -- 172.21.15.28:0/3859780343 shutdown_connections 2024-04-25T01:38:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.441+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/3859780343 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ef674e62e0 0x55ef674e8710 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.441+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/3859780343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674d6010 0x55ef674d3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.441+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/3859780343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 0x55ef674d33c0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.441+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/3859780343 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674cb290 0x55ef674d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.441+0000 7fb9297fa640 0 client.17538 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.441+0000 7fb94a9e9740 1 -- 172.21.15.28:0/3859780343 >> 172.21.15.28:0/3859780343 conn(0x55ef67493950 msgr2=0x55ef674cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.441+0000 7fb94a9e9740 1 -- 172.21.15.28:0/3859780343 shutdown_connections 2024-04-25T01:38:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.441+0000 7fb94a9e9740 1 -- 172.21.15.28:0/3859780343 wait complete. 2024-04-25T01:38:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.441+0000 7fb94a9e9740 20 client.17538 trim_cache size 0 max 16384 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.460+0000 7fb94a9e9740 1 Processor -- start 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.460+0000 7fb94a9e9740 1 -- start start 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.460+0000 7fb94a9e9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674cb290 0x55ef674e1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.460+0000 7fb94a9e9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 0x55ef674edfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.460+0000 7fb94a9e9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674d6010 0x55ef674ee520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.460+0000 7fb94a9e9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9401b0c60 con 0x55ef674cc980 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.460+0000 7fb94a9e9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb9401b27a0 con 0x55ef674cb290 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.460+0000 7fb94a9e9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb9401b1330 con 0x55ef674d6010 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.460+0000 7fb94526f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 0x55ef674edfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.460+0000 7fb94526f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 0x55ef674edfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45856/0 (socket says 172.21.15.28:45856) 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.460+0000 7fb94526f640 1 -- 172.21.15.28:0/3215782327 learned_addr learned my addr 172.21.15.28:0/3215782327 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.460+0000 7fb946271640 1 --2- 172.21.15.28:0/3215782327 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674d6010 0x55ef674ee520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.460+0000 7fb94526f640 1 -- 172.21.15.28:0/3215782327 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674d6010 msgr2=0x55ef674ee520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.460+0000 7fb94526f640 1 --2- 172.21.15.28:0/3215782327 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674d6010 0x55ef674ee520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.460+0000 7fb94526f640 1 -- 172.21.15.28:0/3215782327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674cb290 msgr2=0x55ef674e1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.460+0000 7fb94526f640 1 --2- 172.21.15.28:0/3215782327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674cb290 0x55ef674e1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.460+0000 7fb94526f640 1 -- 172.21.15.28:0/3215782327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef67103170 con 0x55ef674cc980 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.460+0000 7fb94526f640 1 --2- 172.21.15.28:0/3215782327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 0x55ef674edfe0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fb9300bc130 tx=0x7fb9300c0490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f4d05f9b6536c7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.461+0000 7fb92b7fe640 1 -- 172.21.15.28:0/3215782327 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb930049070 con 0x55ef674cc980 2024-04-25T01:38:40.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.461+0000 7fb92b7fe640 1 -- 172.21.15.28:0/3215782327 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9300d8030 con 0x55ef674cc980 2024-04-25T01:38:40.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.461+0000 7fb92b7fe640 1 -- 172.21.15.28:0/3215782327 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9300044e0 con 0x55ef674cc980 2024-04-25T01:38:40.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.461+0000 7fb94a9e9740 1 -- 172.21.15.28:0/3215782327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 msgr2=0x55ef674edfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:40.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.461+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/3215782327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 0x55ef674edfe0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fb9300bc130 tx=0x7fb9300c0490 comp rx=0 tx=0).stop 2024-04-25T01:38:40.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.461+0000 7fb94a9e9740 1 -- 172.21.15.28:0/3215782327 shutdown_connections 2024-04-25T01:38:40.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.461+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/3215782327 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674d6010 0x55ef674ee520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.461+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/3215782327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 0x55ef674edfe0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.461+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/3215782327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674cb290 0x55ef674e1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.461+0000 7fb94a9e9740 1 -- 172.21.15.28:0/3215782327 >> 172.21.15.28:0/3215782327 conn(0x55ef67493950 msgr2=0x55ef674d1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:40.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.461+0000 7fb94a9e9740 1 -- 172.21.15.28:0/3215782327 shutdown_connections 2024-04-25T01:38:40.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.461+0000 7fb94a9e9740 1 -- 172.21.15.28:0/3215782327 wait complete. 2024-04-25T01:38:40.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.461+0000 7fb94a9e9740 1 -- start start 2024-04-25T01:38:40.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb94a9e9740 1 Processor -- start 2024-04-25T01:38:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb94a9e9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674cb290 0x55ef674d6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb94a9e9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 0x55ef674d7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb94a9e9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674d6010 0x55ef674d77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb94a9e9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9401b20d0 con 0x55ef674cc980 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb94a9e9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb9401b1a00 con 0x55ef674d6010 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb94a9e9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb9401aec40 con 0x55ef674cb290 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb94a9e9740 10 client.0 _ll_register_callbacks cb 0x7fb9300f3510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb94526f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 0x55ef674d7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb94526f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 0x55ef674d7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45870/0 (socket says 172.21.15.28:45870) 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb94526f640 1 -- 172.21.15.28:0/552088257 learned_addr learned my addr 172.21.15.28:0/552088257 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb945a70640 1 --2- 172.21.15.28:0/552088257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674cb290 0x55ef674d6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb946271640 1 --2- 172.21.15.28:0/552088257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674d6010 0x55ef674d77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb94526f640 1 -- 172.21.15.28:0/552088257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674cb290 msgr2=0x55ef674d6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb94526f640 1 --2- 172.21.15.28:0/552088257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674cb290 0x55ef674d6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb94526f640 1 -- 172.21.15.28:0/552088257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674d6010 msgr2=0x55ef674d77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb94526f640 1 --2- 172.21.15.28:0/552088257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674d6010 0x55ef674d77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb94526f640 1 -- 172.21.15.28:0/552088257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef674dc980 con 0x55ef674cc980 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb946271640 1 --2- 172.21.15.28:0/552088257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674d6010 0x55ef674d77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.462+0000 7fb945a70640 1 --2- 172.21.15.28:0/552088257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674cb290 0x55ef674d6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.463+0000 7fb94526f640 1 --2- 172.21.15.28:0/552088257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 0x55ef674d7290 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fb9300b83d0 tx=0x7fb9300f3640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f102a72e0654a6a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.463+0000 7fb9297fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.463+0000 7fb9297fa640 1 -- 172.21.15.28:0/552088257 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb930049070 con 0x55ef674cc980 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.463+0000 7fb9297fa640 1 -- 172.21.15.28:0/552088257 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fb9300b85e0 con 0x55ef674cc980 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.463+0000 7fb94a9e9740 1 -- 172.21.15.28:0/552088257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9300d8030 con 0x55ef674cc980 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.463+0000 7fb9297fa640 1 -- 172.21.15.28:0/552088257 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb9300b0050 con 0x55ef674cc980 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.463+0000 7fb94a9e9740 10 client.17544 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.463+0000 7fb94a9e9740 1 -- 172.21.15.28:0/552088257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ef674dc980 con 0x55ef674cc980 2024-04-25T01:38:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.463+0000 7fb9297fa640 1 -- 172.21.15.28:0/552088257 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9300d8030 con 0x55ef674cc980 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.463+0000 7fb9297fa640 1 -- 172.21.15.28:0/552088257 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb93003bc40 con 0x55ef674cc980 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.463+0000 7fb94a9e9740 10 client.17544 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.463+0000 7fb94a9e9740 10 client.17544 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.463+0000 7fb94a9e9740 20 client.17544 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.463+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/552088257 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ef6754d080 0x55ef6754f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.463+0000 7fb94a9e9740 4 client.17544 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.463+0000 7fb94a9e9740 1 -- 172.21.15.28:0/552088257 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ef6743c800 con 0x55ef6754d080 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.464+0000 7fb945a70640 1 --2- 172.21.15.28:0/552088257 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ef6754d080 0x55ef6754f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.464+0000 7fb945a70640 1 --2- 172.21.15.28:0/552088257 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ef6754d080 0x55ef6754f4a0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24c2e3c1dae28370 server_cookie=c35c883c52796e39 in_seq=0 out_seq=0 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.464+0000 7fb9297fa640 10 client.17544 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.465+0000 7fb9297fa640 1 -- 172.21.15.28:0/552088257 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ef6743c800 con 0x55ef6754d080 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.465+0000 7fb9297fa640 10 client.17544 handle_command_reply: tid=0 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.465+0000 7fb94a9e9740 1 client.17544 shutdown 2024-04-25T01:38:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.465+0000 7fb94a9e9740 1 -- 172.21.15.28:0/552088257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 msgr2=0x55ef674d7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.465+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/552088257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 0x55ef674d7290 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fb9300b83d0 tx=0x7fb9300f3640 comp rx=0 tx=0).stop 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.465+0000 7fb94a9e9740 1 -- 172.21.15.28:0/552088257 shutdown_connections 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.465+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/552088257 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ef6754d080 0x55ef6754f4a0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.465+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/552088257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ef674d6010 0x55ef674d77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.465+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/552088257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef674cc980 0x55ef674d7290 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.465+0000 7fb94a9e9740 1 --2- 172.21.15.28:0/552088257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ef674cb290 0x55ef674d6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.465+0000 7fb94a9e9740 1 -- 172.21.15.28:0/552088257 >> 172.21.15.28:0/552088257 conn(0x55ef67493950 msgr2=0x55ef674c6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.465+0000 7fb9297fa640 0 client.17544 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.465+0000 7fb94a9e9740 1 -- 172.21.15.28:0/552088257 shutdown_connections 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.465+0000 7fb94a9e9740 1 -- 172.21.15.28:0/552088257 wait complete. 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.465+0000 7fb94a9e9740 20 client.17544 trim_cache size 0 max 16384 2024-04-25T01:38:40.452 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:38:40.453 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:38:40.453 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "8e30f5ca-1e98-4adc-8c8d-11159372ef48", 2024-04-25T01:38:40.453 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:38:40.453 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.466+0000 7fb947cd4640 1 -- 172.21.15.28:0/3025067387 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb92c0a2d80 msgr2=0x7fb92c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.466+0000 7fb947cd4640 1 --2- 172.21.15.28:0/3025067387 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb92c0a2d80 0x7fb92c0a5240 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb938005c40 tx=0x7fb938004a40 comp rx=0 tx=0).stop 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.466+0000 7fb947cd4640 1 -- 172.21.15.28:0/3025067387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb94006a6e0 msgr2=0x7fb94019a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.466+0000 7fb947cd4640 1 --2- 172.21.15.28:0/3025067387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb94006a6e0 0x7fb94019a6d0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fb930006010 tx=0x7fb9300042f0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.466+0000 7fb947cd4640 1 -- 172.21.15.28:0/3025067387 shutdown_connections 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.466+0000 7fb947cd4640 1 --2- 172.21.15.28:0/3025067387 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb92c0a2d80 0x7fb92c0a5240 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.466+0000 7fb947cd4640 1 --2- 172.21.15.28:0/3025067387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb94010e150 0x7fb94019ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.466+0000 7fb947cd4640 1 --2- 172.21.15.28:0/3025067387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb94006a6e0 0x7fb94019a6d0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.466+0000 7fb947cd4640 1 --2- 172.21.15.28:0/3025067387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb940069dc0 0x7fb94019a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.466+0000 7fb947cd4640 1 -- 172.21.15.28:0/3025067387 >> 172.21.15.28:0/3025067387 conn(0x7fb9400776a0 msgr2=0x7fb94010f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.466+0000 7fb947cd4640 1 -- 172.21.15.28:0/3025067387 shutdown_connections 2024-04-25T01:38:40.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.466+0000 7fb947cd4640 1 -- 172.21.15.28:0/3025067387 wait complete. 2024-04-25T01:38:40.458 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:38:40.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:38:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.622+0000 7fed26aca640 1 -- 172.21.15.28:0/2554871380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed20100620 msgr2=0x7fed20100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.622+0000 7fed26aca640 1 --2- 172.21.15.28:0/2554871380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed20100620 0x7fed20100aa0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fed1000d940 tx=0x7fed1002c3b0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.622+0000 7fed26aca640 1 -- 172.21.15.28:0/2554871380 shutdown_connections 2024-04-25T01:38:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.622+0000 7fed26aca640 1 --2- 172.21.15.28:0/2554871380 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed20100fe0 0x7fed2010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.622+0000 7fed26aca640 1 --2- 172.21.15.28:0/2554871380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed20100620 0x7fed20100aa0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.622+0000 7fed26aca640 1 --2- 172.21.15.28:0/2554871380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed20106630 0x7fed20106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.622+0000 7fed26aca640 1 -- 172.21.15.28:0/2554871380 >> 172.21.15.28:0/2554871380 conn(0x7fed200776a0 msgr2=0x7fed20077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.622+0000 7fed26aca640 1 -- 172.21.15.28:0/2554871380 shutdown_connections 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.622+0000 7fed26aca640 1 -- 172.21.15.28:0/2554871380 wait complete. 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.623+0000 7fed26aca640 1 Processor -- start 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.623+0000 7fed26aca640 1 -- start start 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.623+0000 7fed26aca640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed20100620 0x7fed20195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.623+0000 7fed26aca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed20100fe0 0x7fed20196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.623+0000 7fed26aca640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed20106630 0x7fed2019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.623+0000 7fed26aca640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed2010f980 con 0x7fed20100fe0 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.623+0000 7fed26aca640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fed2010f800 con 0x7fed20106630 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.623+0000 7fed26aca640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fed2010fb00 con 0x7fed20100620 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.623+0000 7fed17fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed20100fe0 0x7fed20196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.623+0000 7fed17fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed20100fe0 0x7fed20196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45888/0 (socket says 172.21.15.28:45888) 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.623+0000 7fed17fff640 1 -- 172.21.15.28:0/2018872198 learned_addr learned my addr 172.21.15.28:0/2018872198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.623+0000 7fed24866640 1 --2- 172.21.15.28:0/2018872198 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed20100620 0x7fed20195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.623+0000 7fed25067640 1 --2- 172.21.15.28:0/2018872198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed20106630 0x7fed2019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.623+0000 7fed17fff640 1 -- 172.21.15.28:0/2018872198 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed20100620 msgr2=0x7fed20195de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.623+0000 7fed17fff640 1 --2- 172.21.15.28:0/2018872198 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed20100620 0x7fed20195de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.623+0000 7fed17fff640 1 -- 172.21.15.28:0/2018872198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed20106630 msgr2=0x7fed2019a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.623+0000 7fed17fff640 1 --2- 172.21.15.28:0/2018872198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed20106630 0x7fed2019a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.623+0000 7fed17fff640 1 -- 172.21.15.28:0/2018872198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed2019add0 con 0x7fed20100fe0 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.624+0000 7fed17fff640 1 --2- 172.21.15.28:0/2018872198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed20100fe0 0x7fed20196320 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fed10000c00 tx=0x7fed1000c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.624+0000 7fed15ffb640 1 -- 172.21.15.28:0/2018872198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed10041020 con 0x7fed20100fe0 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.624+0000 7fed15ffb640 1 -- 172.21.15.28:0/2018872198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed1000cd40 con 0x7fed20100fe0 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.624+0000 7fed26aca640 1 -- 172.21.15.28:0/2018872198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed2019b060 con 0x7fed20100fe0 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.624+0000 7fed15ffb640 1 -- 172.21.15.28:0/2018872198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1003d400 con 0x7fed20100fe0 2024-04-25T01:38:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.624+0000 7fed26aca640 1 -- 172.21.15.28:0/2018872198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed201a2950 con 0x7fed20100fe0 2024-04-25T01:38:40.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.626+0000 7fed15ffb640 1 -- 172.21.15.28:0/2018872198 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fed10044020 con 0x7fed20100fe0 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed297d7740 1 Processor -- start 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed297d7740 1 -- start start 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed15ffb640 1 --2- 172.21.15.28:0/2018872198 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fed080a2d80 0x7fed080a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed297d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 0x564c0d617910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed297d7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d617e50 0x564c0d6264f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed297d7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c0d626a30 0x564c0d628e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed297d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564c0d57c6b0 con 0x564c0d617530 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed297d7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564c0d524880 con 0x564c0d617e50 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed297d7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x564c0d578670 con 0x564c0d626a30 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed25067640 1 --2- 172.21.15.28:0/2018872198 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fed080a2d80 0x7fed080a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed24866640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 0x564c0d617910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed15ffb640 1 -- 172.21.15.28:0/2018872198 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fed100f1560 con 0x7fed20100fe0 2024-04-25T01:38:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed24866640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 0x564c0d617910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45894/0 (socket says 172.21.15.28:45894) 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed24866640 1 -- 172.21.15.28:0/2720039049 learned_addr learned my addr 172.21.15.28:0/2720039049 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed17fff640 1 --2- 172.21.15.28:0/2720039049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d617e50 0x564c0d6264f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed24866640 1 -- 172.21.15.28:0/2720039049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c0d626a30 msgr2=0x564c0d628e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed24866640 1 --2- 172.21.15.28:0/2720039049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c0d626a30 0x564c0d628e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed24866640 1 -- 172.21.15.28:0/2720039049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d617e50 msgr2=0x564c0d6264f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed24866640 1 --2- 172.21.15.28:0/2720039049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d617e50 0x564c0d6264f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed24866640 1 -- 172.21.15.28:0/2720039049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c0d35a7d0 con 0x564c0d617530 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed25067640 1 --2- 172.21.15.28:0/2018872198 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fed080a2d80 0x7fed080a5240 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fed18005fd0 tx=0x7fed180098c0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.627+0000 7fed24866640 1 --2- 172.21.15.28:0/2720039049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 0x564c0d617910 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fed1c009bf0 tx=0x7fed1c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f28338094259e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.628+0000 7fed06ffd640 1 -- 172.21.15.28:0/2720039049 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1c009e60 con 0x564c0d617530 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.628+0000 7fed06ffd640 1 -- 172.21.15.28:0/2720039049 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed1c004230 con 0x564c0d617530 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.628+0000 7fed06ffd640 1 -- 172.21.15.28:0/2720039049 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1c0197e0 con 0x564c0d617530 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.628+0000 7fed297d7740 1 -- 172.21.15.28:0/2720039049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 msgr2=0x564c0d617910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.628+0000 7fed297d7740 1 --2- 172.21.15.28:0/2720039049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 0x564c0d617910 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fed1c009bf0 tx=0x7fed1c011810 comp rx=0 tx=0).stop 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.628+0000 7fed297d7740 1 -- 172.21.15.28:0/2720039049 shutdown_connections 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.628+0000 7fed297d7740 1 --2- 172.21.15.28:0/2720039049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c0d626a30 0x564c0d628e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.628+0000 7fed297d7740 1 --2- 172.21.15.28:0/2720039049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d617e50 0x564c0d6264f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.628+0000 7fed297d7740 1 --2- 172.21.15.28:0/2720039049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 0x564c0d617910 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.628+0000 7fed297d7740 1 -- 172.21.15.28:0/2720039049 >> 172.21.15.28:0/2720039049 conn(0x564c0d613950 msgr2=0x564c0d613d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.628+0000 7fed297d7740 1 -- 172.21.15.28:0/2720039049 shutdown_connections 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.628+0000 7fed297d7740 1 -- 172.21.15.28:0/2720039049 wait complete. 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.629+0000 7fed297d7740 1 -- start start 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.629+0000 7fed297d7740 1 Processor -- start 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.629+0000 7fed297d7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c0d617530 0x564c0d6238f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.629+0000 7fed297d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617e50 0x564c0d623e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.629+0000 7fed297d7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d626a30 0x564c0d624370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.629+0000 7fed297d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564c0d6297f0 con 0x564c0d617e50 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.629+0000 7fed297d7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564c0d537f50 con 0x564c0d626a30 2024-04-25T01:38:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.629+0000 7fed297d7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x564c0d629970 con 0x564c0d617530 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.629+0000 7fed297d7740 10 client.0 _ll_register_callbacks cb 0x564c0d573000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.629+0000 7fed17fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617e50 0x564c0d623e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.629+0000 7fed17fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617e50 0x564c0d623e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45910/0 (socket says 172.21.15.28:45910) 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.629+0000 7fed17fff640 1 -- 172.21.15.28:0/3583370547 learned_addr learned my addr 172.21.15.28:0/3583370547 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.629+0000 7fed25067640 1 --2- 172.21.15.28:0/3583370547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d626a30 0x564c0d624370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.629+0000 7fed24866640 1 --2- 172.21.15.28:0/3583370547 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c0d617530 0x564c0d6238f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.630+0000 7fed17fff640 1 -- 172.21.15.28:0/3583370547 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c0d617530 msgr2=0x564c0d6238f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.630+0000 7fed17fff640 1 --2- 172.21.15.28:0/3583370547 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c0d617530 0x564c0d6238f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.630+0000 7fed17fff640 1 -- 172.21.15.28:0/3583370547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d626a30 msgr2=0x564c0d624370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.630+0000 7fed17fff640 1 --2- 172.21.15.28:0/3583370547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d626a30 0x564c0d624370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.630+0000 7fed17fff640 1 -- 172.21.15.28:0/3583370547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c0d2903f0 con 0x564c0d617e50 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.630+0000 7fed17fff640 1 --2- 172.21.15.28:0/3583370547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617e50 0x564c0d623e30 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fed10005ab0 tx=0x7fed100bd710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61321eb9be377a10 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.630+0000 7fecfffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.630+0000 7fecfffff640 1 -- 172.21.15.28:0/3583370547 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed100bd850 con 0x564c0d617e50 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.630+0000 7fecfffff640 1 -- 172.21.15.28:0/3583370547 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fed100a6ff0 con 0x564c0d617e50 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.630+0000 7fed297d7740 1 -- 172.21.15.28:0/3583370547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed1c004230 con 0x564c0d617e50 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.630+0000 7fecfffff640 1 -- 172.21.15.28:0/3583370547 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fed100c1c00 con 0x564c0d617e50 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.630+0000 7fed297d7740 10 client.17556 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.630+0000 7fed297d7740 1 -- 172.21.15.28:0/3583370547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564c0d2500f0 con 0x564c0d617e50 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.630+0000 7fecfffff640 1 -- 172.21.15.28:0/3583370547 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed1c004230 con 0x564c0d617e50 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.630+0000 7fecfffff640 1 -- 172.21.15.28:0/3583370547 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fed100a2070 con 0x564c0d617e50 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.630+0000 7fed297d7740 10 client.17556 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.630+0000 7fed297d7740 10 client.17556 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.631+0000 7fed297d7740 20 client.17556 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.631+0000 7fed297d7740 1 --2- 172.21.15.28:0/3583370547 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564c0d636ae0 0x564c0d638ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.631+0000 7fed297d7740 4 client.17556 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.631+0000 7fed297d7740 1 -- 172.21.15.28:0/3583370547 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c0d589bf0 con 0x564c0d636ae0 2024-04-25T01:38:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.631+0000 7fed24866640 1 --2- 172.21.15.28:0/3583370547 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564c0d636ae0 0x564c0d638ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.631+0000 7fed24866640 1 --2- 172.21.15.28:0/3583370547 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564c0d636ae0 0x564c0d638ef0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=939633160591694a server_cookie=df0fe75e1e01b21d in_seq=0 out_seq=0 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.631+0000 7fecfffff640 10 client.17556 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.632+0000 7fecfffff640 1 -- 172.21.15.28:0/3583370547 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564c0d589bf0 con 0x564c0d636ae0 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.632+0000 7fecfffff640 10 client.17556 handle_command_reply: tid=0 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.632+0000 7fed297d7740 1 client.17556 shutdown 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.632+0000 7fed297d7740 1 -- 172.21.15.28:0/3583370547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617e50 msgr2=0x564c0d623e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.632+0000 7fed297d7740 1 --2- 172.21.15.28:0/3583370547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617e50 0x564c0d623e30 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fed10005ab0 tx=0x7fed100bd710 comp rx=0 tx=0).stop 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.632+0000 7fed297d7740 1 -- 172.21.15.28:0/3583370547 shutdown_connections 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.632+0000 7fed297d7740 1 --2- 172.21.15.28:0/3583370547 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564c0d636ae0 0x564c0d638ef0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.632+0000 7fed297d7740 1 --2- 172.21.15.28:0/3583370547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d626a30 0x564c0d624370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.632+0000 7fed297d7740 1 --2- 172.21.15.28:0/3583370547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617e50 0x564c0d623e30 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.632+0000 7fecfffff640 0 client.17556 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.632+0000 7fed297d7740 1 --2- 172.21.15.28:0/3583370547 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c0d617530 0x564c0d6238f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.632+0000 7fed297d7740 1 -- 172.21.15.28:0/3583370547 >> 172.21.15.28:0/3583370547 conn(0x564c0d613950 msgr2=0x564c0d6196b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.632+0000 7fed297d7740 1 -- 172.21.15.28:0/3583370547 shutdown_connections 2024-04-25T01:38:40.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.632+0000 7fed297d7740 1 -- 172.21.15.28:0/3583370547 wait complete. 2024-04-25T01:38:40.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.632+0000 7fed297d7740 20 client.17556 trim_cache size 0 max 16384 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.650+0000 7fed297d7740 1 Processor -- start 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.650+0000 7fed297d7740 1 -- start start 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.650+0000 7fed297d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 0x564c0d626020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.650+0000 7fed297d7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c0d617e50 0x564c0d6326f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.650+0000 7fed297d7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d626a30 0x564c0d623fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.650+0000 7fed297d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed201ac770 con 0x564c0d617530 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.650+0000 7fed297d7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fed201ae2b0 con 0x564c0d626a30 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.650+0000 7fed297d7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fed201ace40 con 0x564c0d617e50 2024-04-25T01:38:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.650+0000 7fed24866640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 0x564c0d626020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.650+0000 7fed24866640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 0x564c0d626020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45916/0 (socket says 172.21.15.28:45916) 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.650+0000 7fed24866640 1 -- 172.21.15.28:0/1428688403 learned_addr learned my addr 172.21.15.28:0/1428688403 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.650+0000 7fed25067640 1 --2- 172.21.15.28:0/1428688403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d626a30 0x564c0d623fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.650+0000 7fed24866640 1 -- 172.21.15.28:0/1428688403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c0d617e50 msgr2=0x564c0d6326f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.650+0000 7fed24866640 1 --2- 172.21.15.28:0/1428688403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c0d617e50 0x564c0d6326f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.650+0000 7fed24866640 1 -- 172.21.15.28:0/1428688403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d626a30 msgr2=0x564c0d623fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.650+0000 7fed24866640 1 --2- 172.21.15.28:0/1428688403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d626a30 0x564c0d623fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.650+0000 7fed24866640 1 -- 172.21.15.28:0/1428688403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c0d613780 con 0x564c0d617530 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.650+0000 7fed24866640 1 --2- 172.21.15.28:0/1428688403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 0x564c0d626020 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fed1c019450 tx=0x7fed1c0193e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1c5656025f2d569 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.651+0000 7fed067fc640 1 -- 172.21.15.28:0/1428688403 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1c0050e0 con 0x564c0d617530 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.651+0000 7fed067fc640 1 -- 172.21.15.28:0/1428688403 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed1c005280 con 0x564c0d617530 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.651+0000 7fed067fc640 1 -- 172.21.15.28:0/1428688403 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1c019750 con 0x564c0d617530 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.651+0000 7fed297d7740 1 -- 172.21.15.28:0/1428688403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 msgr2=0x564c0d626020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.651+0000 7fed297d7740 1 --2- 172.21.15.28:0/1428688403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 0x564c0d626020 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fed1c019450 tx=0x7fed1c0193e0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.651+0000 7fed297d7740 1 -- 172.21.15.28:0/1428688403 shutdown_connections 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.651+0000 7fed297d7740 1 --2- 172.21.15.28:0/1428688403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d626a30 0x564c0d623fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.651+0000 7fed297d7740 1 --2- 172.21.15.28:0/1428688403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c0d617e50 0x564c0d6326f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.651+0000 7fed297d7740 1 --2- 172.21.15.28:0/1428688403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 0x564c0d626020 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.651+0000 7fed297d7740 1 -- 172.21.15.28:0/1428688403 >> 172.21.15.28:0/1428688403 conn(0x564c0d613950 msgr2=0x564c0d6238b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.651+0000 7fed297d7740 1 -- 172.21.15.28:0/1428688403 shutdown_connections 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.651+0000 7fed297d7740 1 -- 172.21.15.28:0/1428688403 wait complete. 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.652+0000 7fed297d7740 1 -- start start 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.652+0000 7fed297d7740 1 Processor -- start 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.652+0000 7fed297d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 0x564c0d62a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.652+0000 7fed297d7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c0d617e50 0x564c0d62ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.652+0000 7fed297d7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d626a30 0x564c0d62b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.652+0000 7fed297d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed201adbe0 con 0x564c0d617530 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.652+0000 7fed297d7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fed201ad510 con 0x564c0d626a30 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.652+0000 7fed297d7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fed201aa750 con 0x564c0d617e50 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.652+0000 7fed297d7740 10 client.0 _ll_register_callbacks cb 0x7fed1000d890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.652+0000 7fed24866640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 0x564c0d62a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.652+0000 7fed24866640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 0x564c0d62a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45924/0 (socket says 172.21.15.28:45924) 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.652+0000 7fed24866640 1 -- 172.21.15.28:0/3288850910 learned_addr learned my addr 172.21.15.28:0/3288850910 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.652+0000 7fed25067640 1 --2- 172.21.15.28:0/3288850910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d626a30 0x564c0d62b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fed17fff640 1 --2- 172.21.15.28:0/3288850910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c0d617e50 0x564c0d62ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fed24866640 1 -- 172.21.15.28:0/3288850910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c0d617e50 msgr2=0x564c0d62ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fed24866640 1 --2- 172.21.15.28:0/3288850910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c0d617e50 0x564c0d62ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fed24866640 1 -- 172.21.15.28:0/3288850910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d626a30 msgr2=0x564c0d62b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fed24866640 1 --2- 172.21.15.28:0/3288850910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d626a30 0x564c0d62b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fed24866640 1 -- 172.21.15.28:0/3288850910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c0d69c1d0 con 0x564c0d617530 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fed24866640 1 --2- 172.21.15.28:0/3288850910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 0x564c0d62a7c0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fed1c003d20 tx=0x7fed1c009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54dba2e06bd75409 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:38:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fecfffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fecfffff640 1 -- 172.21.15.28:0/3288850910 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1c0037f0 con 0x564c0d617530 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fecfffff640 1 -- 172.21.15.28:0/3288850910 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fed1c003990 con 0x564c0d617530 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fed297d7740 1 -- 172.21.15.28:0/3288850910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed1c005280 con 0x564c0d617530 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fecfffff640 1 -- 172.21.15.28:0/3288850910 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fed1c040050 con 0x564c0d617530 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fed297d7740 10 client.17562 fetch_fsmap learned FSMap version 28 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fed297d7740 1 -- 172.21.15.28:0/3288850910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564c0d69d4c0 con 0x564c0d617530 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fecfffff640 1 -- 172.21.15.28:0/3288850910 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed1c005280 con 0x564c0d617530 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fecfffff640 1 -- 172.21.15.28:0/3288850910 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fed1c018070 con 0x564c0d617530 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fed297d7740 10 client.17562 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fed297d7740 10 client.17562 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fed297d7740 20 client.17562 populate_metadata read hostname 'smithi028' 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.653+0000 7fed297d7740 1 --2- 172.21.15.28:0/3288850910 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564c0d69d970 0x564c0d69fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.654+0000 7fed297d7740 4 client.17562 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.654+0000 7fed297d7740 1 -- 172.21.15.28:0/3288850910 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564c0d583f40 con 0x564c0d69d970 2024-04-25T01:38:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.654+0000 7fed17fff640 1 --2- 172.21.15.28:0/3288850910 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564c0d69d970 0x564c0d69fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.654+0000 7fed17fff640 1 --2- 172.21.15.28:0/3288850910 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564c0d69d970 0x564c0d69fd90 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51f76d4ece766083 server_cookie=a49e80d2b1bfaa65 in_seq=0 out_seq=0 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.654+0000 7fecfffff640 10 client.17562 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.654+0000 7fecfffff640 1 -- 172.21.15.28:0/3288850910 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564c0d583f40 con 0x564c0d69d970 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.654+0000 7fecfffff640 10 client.17562 handle_command_reply: tid=0 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.654+0000 7fed297d7740 1 client.17562 shutdown 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.654+0000 7fed297d7740 1 -- 172.21.15.28:0/3288850910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 msgr2=0x564c0d62a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.654+0000 7fed297d7740 1 --2- 172.21.15.28:0/3288850910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 0x564c0d62a7c0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fed1c003d20 tx=0x7fed1c009830 comp rx=0 tx=0).stop 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.654+0000 7fed297d7740 1 -- 172.21.15.28:0/3288850910 shutdown_connections 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.654+0000 7fed297d7740 1 --2- 172.21.15.28:0/3288850910 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564c0d69d970 0x564c0d69fd90 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.654+0000 7fed297d7740 1 --2- 172.21.15.28:0/3288850910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c0d626a30 0x564c0d62b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.654+0000 7fed297d7740 1 --2- 172.21.15.28:0/3288850910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c0d617e50 0x564c0d62ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.654+0000 7fecfffff640 0 client.17562 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.654+0000 7fed297d7740 1 --2- 172.21.15.28:0/3288850910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c0d617530 0x564c0d62a7c0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.654+0000 7fed297d7740 1 -- 172.21.15.28:0/3288850910 >> 172.21.15.28:0/3288850910 conn(0x564c0d613950 msgr2=0x564c0d624140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.655+0000 7fed297d7740 1 -- 172.21.15.28:0/3288850910 shutdown_connections 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.655+0000 7fed297d7740 1 -- 172.21.15.28:0/3288850910 wait complete. 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.655+0000 7fed297d7740 20 client.17562 trim_cache size 0 max 16384 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi028.stdout: "8e30f5ca-1e98-4adc-8c8d-11159372ef48": { 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "8e30f5ca-1e98-4adc-8c8d-11159372ef48", 2024-04-25T01:38:40.643 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-25T01:38:40.644 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:38:40.644 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:38:40.644 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:38:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.655+0000 7fed26aca640 1 -- 172.21.15.28:0/2018872198 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fed080a2d80 msgr2=0x7fed080a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.655+0000 7fed26aca640 1 --2- 172.21.15.28:0/2018872198 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fed080a2d80 0x7fed080a5240 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fed18005fd0 tx=0x7fed180098c0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.655+0000 7fed26aca640 1 -- 172.21.15.28:0/2018872198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed20100fe0 msgr2=0x7fed20196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.655+0000 7fed26aca640 1 --2- 172.21.15.28:0/2018872198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed20100fe0 0x7fed20196320 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fed10000c00 tx=0x7fed1000c460 comp rx=0 tx=0).stop 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.655+0000 7fed26aca640 1 -- 172.21.15.28:0/2018872198 shutdown_connections 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.655+0000 7fed26aca640 1 --2- 172.21.15.28:0/2018872198 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fed080a2d80 0x7fed080a5240 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.655+0000 7fed26aca640 1 --2- 172.21.15.28:0/2018872198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fed20106630 0x7fed2019a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.655+0000 7fed26aca640 1 --2- 172.21.15.28:0/2018872198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed20100fe0 0x7fed20196320 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.655+0000 7fed26aca640 1 --2- 172.21.15.28:0/2018872198 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fed20100620 0x7fed20195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.655+0000 7fed26aca640 1 -- 172.21.15.28:0/2018872198 >> 172.21.15.28:0/2018872198 conn(0x7fed200776a0 msgr2=0x7fed20103150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.655+0000 7fed26aca640 1 -- 172.21.15.28:0/2018872198 shutdown_connections 2024-04-25T01:38:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:38:40.655+0000 7fed26aca640 1 -- 172.21.15.28:0/2018872198 wait complete. 2024-04-25T01:38:40.647 INFO:tasks.cephfs.filesystem:scrub status for tag:8e30f5ca-1e98-4adc-8c8d-11159372ef48 - {'path': '/', 'tag': '8e30f5ca-1e98-4adc-8c8d-11159372ef48', 'options': 'recursive,force'} 2024-04-25T01:38:42.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:41 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:38:42.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:41 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:38:42.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:41 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:38:42.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:41 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:38:42.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:41 smithi130 ceph-mon[32702]: pgmap v173: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:38:42.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:41 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:38:42.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:41 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:38:42.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:41 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:38:42.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:41 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:38:42.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:41 smithi032 ceph-mon[32461]: pgmap v173: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:38:42.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:41 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:38:42.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:41 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:38:42.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:41 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:38:42.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:41 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:38:42.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:41 smithi028 ceph-mon[29390]: pgmap v173: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:38:44.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:43 smithi130 ceph-mon[32702]: pgmap v174: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 12 op/s 2024-04-25T01:38:44.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:43 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:38:44.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:43 smithi032 ceph-mon[32461]: pgmap v174: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 12 op/s 2024-04-25T01:38:44.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:43 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:38:44.314 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:43 smithi028 ceph-mon[29390]: pgmap v174: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 12 op/s 2024-04-25T01:38:44.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:43 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:38:46.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:45 smithi130 ceph-mon[32702]: pgmap v175: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:38:46.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:45 smithi032 ceph-mon[32461]: pgmap v175: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:38:46.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:45 smithi028 ceph-mon[29390]: pgmap v175: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:38:48.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:47 smithi130 ceph-mon[32702]: pgmap v176: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-25T01:38:48.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:47 smithi032 ceph-mon[32461]: pgmap v176: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-25T01:38:48.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:47 smithi028 ceph-mon[29390]: pgmap v176: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-25T01:38:50.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:49 smithi130 ceph-mon[32702]: pgmap v177: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-04-25T01:38:50.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:49 smithi032 ceph-mon[32461]: pgmap v177: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-04-25T01:38:50.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:49 smithi028 ceph-mon[29390]: pgmap v177: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-04-25T01:38:52.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:51 smithi130 ceph-mon[32702]: pgmap v178: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:38:52.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:51 smithi032 ceph-mon[32461]: pgmap v178: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:38:52.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:51 smithi028 ceph-mon[29390]: pgmap v178: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:38:54.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:53 smithi130 ceph-mon[32702]: pgmap v179: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 12 op/s 2024-04-25T01:38:54.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:53 smithi032 ceph-mon[32461]: pgmap v179: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 12 op/s 2024-04-25T01:38:54.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:53 smithi028 ceph-mon[29390]: pgmap v179: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 12 op/s 2024-04-25T01:38:56.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:55 smithi130 ceph-mon[32702]: pgmap v180: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:38:56.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:55 smithi032 ceph-mon[32461]: pgmap v180: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:38:56.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:55 smithi028 ceph-mon[29390]: pgmap v180: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:38:58.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:57 smithi130 ceph-mon[32702]: pgmap v181: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:38:58.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:57 smithi032 ceph-mon[32461]: pgmap v181: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:38:58.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:57 smithi028 ceph-mon[29390]: pgmap v181: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:39:00.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:38:59 smithi130 ceph-mon[32702]: pgmap v182: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:39:00.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:38:59 smithi032 ceph-mon[32461]: pgmap v182: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:39:00.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:38:59 smithi028 ceph-mon[29390]: pgmap v182: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:39:02.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:01 smithi130 ceph-mon[32702]: pgmap v183: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:39:02.237 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:01 smithi032 ceph-mon[32461]: pgmap v183: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:39:02.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:01 smithi028 ceph-mon[29390]: pgmap v183: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:39:04.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:03 smithi130 ceph-mon[32702]: pgmap v184: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:39:04.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:03 smithi032 ceph-mon[32461]: pgmap v184: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:39:04.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:03 smithi028 ceph-mon[29390]: pgmap v184: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:39:06.212 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:05 smithi130 ceph-mon[32702]: pgmap v185: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:39:06.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:05 smithi032 ceph-mon[32461]: pgmap v185: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:39:06.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:05 smithi028 ceph-mon[29390]: pgmap v185: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:39:08.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:07 smithi130 ceph-mon[32702]: pgmap v186: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-04-25T01:39:08.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:07 smithi032 ceph-mon[32461]: pgmap v186: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-04-25T01:39:08.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:07 smithi028 ceph-mon[29390]: pgmap v186: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-04-25T01:39:10.213 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:09 smithi130 ceph-mon[32702]: pgmap v187: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-04-25T01:39:10.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:09 smithi032 ceph-mon[32461]: pgmap v187: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-04-25T01:39:10.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:09 smithi028 ceph-mon[29390]: pgmap v187: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-04-25T01:39:10.648 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:39:10.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:39:10.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.799+0000 7f6d4dcd4640 1 -- 172.21.15.28:0/4215830439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d48100610 msgr2=0x7f6d48100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:10.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.799+0000 7f6d4dcd4640 1 --2- 172.21.15.28:0/4215830439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d48100610 0x7f6d48100a90 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f6d38009930 tx=0x7f6d3802a420 comp rx=0 tx=0).stop 2024-04-25T01:39:10.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.799+0000 7f6d4dcd4640 1 -- 172.21.15.28:0/4215830439 shutdown_connections 2024-04-25T01:39:10.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.799+0000 7f6d4dcd4640 1 --2- 172.21.15.28:0/4215830439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d48100fd0 0x7f6d4810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.799+0000 7f6d4dcd4640 1 --2- 172.21.15.28:0/4215830439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d48100610 0x7f6d48100a90 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.799+0000 7f6d4dcd4640 1 --2- 172.21.15.28:0/4215830439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d481066b0 0x7f6d48106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.799+0000 7f6d4dcd4640 1 -- 172.21.15.28:0/4215830439 >> 172.21.15.28:0/4215830439 conn(0x7f6d48077740 msgr2=0x7f6d48077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:10.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.799+0000 7f6d4dcd4640 1 -- 172.21.15.28:0/4215830439 shutdown_connections 2024-04-25T01:39:10.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.799+0000 7f6d4dcd4640 1 -- 172.21.15.28:0/4215830439 wait complete. 2024-04-25T01:39:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.800+0000 7f6d4dcd4640 1 Processor -- start 2024-04-25T01:39:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.800+0000 7f6d4dcd4640 1 -- start start 2024-04-25T01:39:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.800+0000 7f6d4dcd4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d48100610 0x7f6d48196f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.800+0000 7f6d4dcd4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d48100fd0 0x7f6d48197470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.800+0000 7f6d4dcd4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d481066b0 0x7f6d4819b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.800+0000 7f6d4dcd4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d481109f0 con 0x7f6d48100610 2024-04-25T01:39:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.800+0000 7f6d4dcd4640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6d48110870 con 0x7f6d48100fd0 2024-04-25T01:39:10.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.800+0000 7f6d4dcd4640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6d48110b70 con 0x7f6d481066b0 2024-04-25T01:39:10.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.801+0000 7f6d477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d48100610 0x7f6d48196f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.801+0000 7f6d477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d48100610 0x7f6d48196f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48272/0 (socket says 172.21.15.28:48272) 2024-04-25T01:39:10.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.801+0000 7f6d477fe640 1 -- 172.21.15.28:0/1865228774 learned_addr learned my addr 172.21.15.28:0/1865228774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:10.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.801+0000 7f6d47fff640 1 --2- 172.21.15.28:0/1865228774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d481066b0 0x7f6d4819b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.801+0000 7f6d46ffd640 1 --2- 172.21.15.28:0/1865228774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d48100fd0 0x7f6d48197470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.801+0000 7f6d477fe640 1 -- 172.21.15.28:0/1865228774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d481066b0 msgr2=0x7f6d4819b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:10.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.801+0000 7f6d477fe640 1 --2- 172.21.15.28:0/1865228774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d481066b0 0x7f6d4819b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.801+0000 7f6d477fe640 1 -- 172.21.15.28:0/1865228774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d48100fd0 msgr2=0x7f6d48197470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:10.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.801+0000 7f6d477fe640 1 --2- 172.21.15.28:0/1865228774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d48100fd0 0x7f6d48197470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.801+0000 7f6d477fe640 1 -- 172.21.15.28:0/1865228774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4819bf20 con 0x7f6d48100610 2024-04-25T01:39:10.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.801+0000 7f6d477fe640 1 --2- 172.21.15.28:0/1865228774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d48100610 0x7f6d48196f30 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c009970 tx=0x7f6d3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:10.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.801+0000 7f6d44ff9640 1 -- 172.21.15.28:0/1865228774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c00d030 con 0x7f6d48100610 2024-04-25T01:39:10.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.801+0000 7f6d4dcd4640 1 -- 172.21.15.28:0/1865228774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d4819c210 con 0x7f6d48100610 2024-04-25T01:39:10.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.801+0000 7f6d44ff9640 1 -- 172.21.15.28:0/1865228774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d3c007d40 con 0x7f6d48100610 2024-04-25T01:39:10.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.801+0000 7f6d44ff9640 1 -- 172.21.15.28:0/1865228774 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c00cc50 con 0x7f6d48100610 2024-04-25T01:39:10.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.801+0000 7f6d4dcd4640 1 -- 172.21.15.28:0/1865228774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d480688d0 con 0x7f6d48100610 2024-04-25T01:39:10.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.804+0000 7f6d44ff9640 1 -- 172.21.15.28:0/1865228774 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6d3c007860 con 0x7f6d48100610 2024-04-25T01:39:10.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.804+0000 7f6d509e6740 1 Processor -- start 2024-04-25T01:39:10.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.804+0000 7f6d509e6740 1 -- start start 2024-04-25T01:39:10.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.804+0000 7f6d44ff9640 1 --2- 172.21.15.28:0/1865228774 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6d300a2d00 0x7f6d300a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d509e6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8510 0x55e01e3c88f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d509e6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3c8e30 0x55e01e3d74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d509e6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3d7a30 0x55e01e3d9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d509e6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e01e32d6b0 con 0x55e01e3d7a30 2024-04-25T01:39:10.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d509e6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e01e2d5880 con 0x55e01e3c8e30 2024-04-25T01:39:10.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d509e6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55e01e329670 con 0x55e01e3c8510 2024-04-25T01:39:10.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d44ff9640 1 -- 172.21.15.28:0/1865228774 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f6d3c0d0e80 con 0x7f6d48100610 2024-04-25T01:39:10.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d47fff640 1 --2- 172.21.15.28:0/1865228774 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6d300a2d00 0x7f6d300a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d46ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3c8e30 0x55e01e3d74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d46ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3c8e30 0x55e01e3d74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:41878/0 (socket says 172.21.15.28:41878) 2024-04-25T01:39:10.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d46ffd640 1 -- 172.21.15.28:0/2851532343 learned_addr learned my addr 172.21.15.28:0/2851532343 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:10.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d477fe640 1 --2- 172.21.15.28:0/2851532343 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8510 0x55e01e3c88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d47fff640 1 --2- 172.21.15.28:0/2851532343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3d7a30 0x55e01e3d9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d46ffd640 1 -- 172.21.15.28:0/2851532343 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8510 msgr2=0x55e01e3c88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:10.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d46ffd640 1 --2- 172.21.15.28:0/2851532343 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8510 0x55e01e3c88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d46ffd640 1 -- 172.21.15.28:0/2851532343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3d7a30 msgr2=0x55e01e3d9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:10.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d46ffd640 1 --2- 172.21.15.28:0/2851532343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3d7a30 0x55e01e3d9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d46ffd640 1 -- 172.21.15.28:0/2851532343 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e01e10b7d0 con 0x55e01e3c8e30 2024-04-25T01:39:10.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d47fff640 1 --2- 172.21.15.28:0/2851532343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3d7a30 0x55e01e3d9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:10.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d46ffd640 1 --2- 172.21.15.28:0/2851532343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3c8e30 0x55e01e3d74f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6d3802a940 tx=0x7f6d38005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=476dfb276d511fc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:10.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.805+0000 7f6d47fff640 1 --2- 172.21.15.28:0/1865228774 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6d300a2d00 0x7f6d300a51c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x55e01e3c9ac0 tx=0x7f6d40009bf0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:10.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.806+0000 7f6d2dffb640 1 -- 172.21.15.28:0/2851532343 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d38040070 con 0x55e01e3c8e30 2024-04-25T01:39:10.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.806+0000 7f6d2dffb640 1 -- 172.21.15.28:0/2851532343 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d380318f0 con 0x55e01e3c8e30 2024-04-25T01:39:10.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.806+0000 7f6d2dffb640 1 -- 172.21.15.28:0/2851532343 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d38035040 con 0x55e01e3c8e30 2024-04-25T01:39:10.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.806+0000 7f6d509e6740 1 -- 172.21.15.28:0/2851532343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3c8e30 msgr2=0x55e01e3d74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:10.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.806+0000 7f6d509e6740 1 --2- 172.21.15.28:0/2851532343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3c8e30 0x55e01e3d74f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6d3802a940 tx=0x7f6d38005c00 comp rx=0 tx=0).stop 2024-04-25T01:39:10.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.806+0000 7f6d509e6740 1 -- 172.21.15.28:0/2851532343 shutdown_connections 2024-04-25T01:39:10.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.806+0000 7f6d509e6740 1 --2- 172.21.15.28:0/2851532343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3d7a30 0x55e01e3d9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.806+0000 7f6d509e6740 1 --2- 172.21.15.28:0/2851532343 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3c8e30 0x55e01e3d74f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.806+0000 7f6d509e6740 1 --2- 172.21.15.28:0/2851532343 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8510 0x55e01e3c88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.806+0000 7f6d509e6740 1 -- 172.21.15.28:0/2851532343 >> 172.21.15.28:0/2851532343 conn(0x55e01e3c4950 msgr2=0x55e01e3c4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:10.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.806+0000 7f6d509e6740 1 -- 172.21.15.28:0/2851532343 shutdown_connections 2024-04-25T01:39:10.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.806+0000 7f6d509e6740 1 -- 172.21.15.28:0/2851532343 wait complete. 2024-04-25T01:39:10.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.807+0000 7f6d509e6740 1 -- start start 2024-04-25T01:39:10.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.807+0000 7f6d509e6740 1 Processor -- start 2024-04-25T01:39:10.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.807+0000 7f6d509e6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3c8510 0x55e01e3d48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.807+0000 7f6d509e6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8e30 0x55e01e3d4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.807+0000 7f6d509e6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3d7a30 0x55e01e3d5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.807+0000 7f6d509e6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e01e3da810 con 0x55e01e3c8510 2024-04-25T01:39:10.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.807+0000 7f6d509e6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e01e2e8f50 con 0x55e01e3d7a30 2024-04-25T01:39:10.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.807+0000 7f6d509e6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55e01e3da990 con 0x55e01e3c8e30 2024-04-25T01:39:10.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.807+0000 7f6d509e6740 10 client.0 _ll_register_callbacks cb 0x55e01e324000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:10.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3c8510 0x55e01e3d48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d46ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8e30 0x55e01e3d4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3c8510 0x55e01e3d48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48288/0 (socket says 172.21.15.28:48288) 2024-04-25T01:39:10.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d47fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3d7a30 0x55e01e3d5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d477fe640 1 -- 172.21.15.28:0/3634375232 learned_addr learned my addr 172.21.15.28:0/3634375232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:10.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d46ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8e30 0x55e01e3d4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.28:54516/0 (socket says 172.21.15.28:54516) 2024-04-25T01:39:10.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d477fe640 1 -- 172.21.15.28:0/3634375232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8e30 msgr2=0x55e01e3d4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:10.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d477fe640 1 --2- 172.21.15.28:0/3634375232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8e30 0x55e01e3d4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d477fe640 1 -- 172.21.15.28:0/3634375232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3d7a30 msgr2=0x55e01e3d5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:10.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d477fe640 1 --2- 172.21.15.28:0/3634375232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3d7a30 0x55e01e3d5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d477fe640 1 -- 172.21.15.28:0/3634375232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e01e0413f0 con 0x55e01e3c8510 2024-04-25T01:39:10.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d46ffd640 1 --2- 172.21.15.28:0/3634375232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8e30 0x55e01e3d4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:39:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d47fff640 1 --2- 172.21.15.28:0/3634375232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3d7a30 0x55e01e3d5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:39:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d477fe640 1 --2- 172.21.15.28:0/3634375232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3c8510 0x55e01e3d48e0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c005fd0 tx=0x7f6d3c099410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=987fe67307e7de36 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d26ffd640 1 -- 172.21.15.28:0/3634375232 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c0a1410 con 0x55e01e3c8510 2024-04-25T01:39:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d26ffd640 1 -- 172.21.15.28:0/3634375232 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f6d3c09b360 con 0x55e01e3c8510 2024-04-25T01:39:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d509e6740 1 -- 172.21.15.28:0/3634375232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d380318f0 con 0x55e01e3c8510 2024-04-25T01:39:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d26ffd640 1 -- 172.21.15.28:0/3634375232 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6d3c0a5cd0 con 0x55e01e3c8510 2024-04-25T01:39:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d509e6740 10 client.17571 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.808+0000 7f6d509e6740 1 -- 172.21.15.28:0/3634375232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e01e0010f0 con 0x55e01e3c8510 2024-04-25T01:39:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.809+0000 7f6d26ffd640 1 -- 172.21.15.28:0/3634375232 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d380318f0 con 0x55e01e3c8510 2024-04-25T01:39:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.809+0000 7f6d26ffd640 1 -- 172.21.15.28:0/3634375232 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6d3c0996d0 con 0x55e01e3c8510 2024-04-25T01:39:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.809+0000 7f6d509e6740 10 client.17571 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.809+0000 7f6d509e6740 10 client.17571 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.809+0000 7f6d509e6740 20 client.17571 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.809+0000 7f6d509e6740 1 --2- 172.21.15.28:0/3634375232 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e01e3e7b20 0x55e01e3e9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.809+0000 7f6d509e6740 4 client.17571 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.809+0000 7f6d509e6740 1 -- 172.21.15.28:0/3634375232 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e01e33abf0 con 0x55e01e3e7b20 2024-04-25T01:39:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.809+0000 7f6d46ffd640 1 --2- 172.21.15.28:0/3634375232 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e01e3e7b20 0x55e01e3e9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.809+0000 7f6d26ffd640 10 client.17571 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.809+0000 7f6d46ffd640 1 --2- 172.21.15.28:0/3634375232 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e01e3e7b20 0x55e01e3e9f30 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7daf845925b148d server_cookie=4ea3ed5a1dd90147 in_seq=0 out_seq=0 2024-04-25T01:39:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.810+0000 7f6d26ffd640 1 -- 172.21.15.28:0/3634375232 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e01e33abf0 con 0x55e01e3e7b20 2024-04-25T01:39:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.810+0000 7f6d26ffd640 10 client.17571 handle_command_reply: tid=0 2024-04-25T01:39:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.810+0000 7f6d509e6740 1 client.17571 shutdown 2024-04-25T01:39:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.810+0000 7f6d509e6740 1 -- 172.21.15.28:0/3634375232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3c8510 msgr2=0x55e01e3d48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.810+0000 7f6d509e6740 1 --2- 172.21.15.28:0/3634375232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3c8510 0x55e01e3d48e0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c005fd0 tx=0x7f6d3c099410 comp rx=0 tx=0).stop 2024-04-25T01:39:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.810+0000 7f6d509e6740 1 -- 172.21.15.28:0/3634375232 shutdown_connections 2024-04-25T01:39:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.810+0000 7f6d509e6740 1 --2- 172.21.15.28:0/3634375232 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e01e3e7b20 0x55e01e3e9f30 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.810+0000 7f6d509e6740 1 --2- 172.21.15.28:0/3634375232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3d7a30 0x55e01e3d5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.810+0000 7f6d509e6740 1 --2- 172.21.15.28:0/3634375232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8e30 0x55e01e3d4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.810+0000 7f6d509e6740 1 --2- 172.21.15.28:0/3634375232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3c8510 0x55e01e3d48e0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.810+0000 7f6d509e6740 1 -- 172.21.15.28:0/3634375232 >> 172.21.15.28:0/3634375232 conn(0x55e01e3c4950 msgr2=0x55e01e3ca690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.810+0000 7f6d26ffd640 0 client.17571 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.810+0000 7f6d509e6740 1 -- 172.21.15.28:0/3634375232 shutdown_connections 2024-04-25T01:39:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.810+0000 7f6d509e6740 1 -- 172.21.15.28:0/3634375232 wait complete. 2024-04-25T01:39:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.811+0000 7f6d509e6740 20 client.17571 trim_cache size 0 max 16384 2024-04-25T01:39:10.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.828+0000 7f6d509e6740 1 Processor -- start 2024-04-25T01:39:10.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.828+0000 7f6d509e6740 1 -- start start 2024-04-25T01:39:10.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.828+0000 7f6d509e6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3c8510 0x55e01e3d7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.828+0000 7f6d509e6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8e30 0x55e01e3e3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.828+0000 7f6d509e6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3d7a30 0x55e01e3d4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.828+0000 7f6d509e6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d481ac920 con 0x55e01e3d7a30 2024-04-25T01:39:10.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.828+0000 7f6d509e6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6d481ae460 con 0x55e01e3c8510 2024-04-25T01:39:10.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.828+0000 7f6d509e6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6d481acff0 con 0x55e01e3c8e30 2024-04-25T01:39:10.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.829+0000 7f6d477fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3c8510 0x55e01e3d7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.829+0000 7f6d47fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3d7a30 0x55e01e3d4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.829+0000 7f6d477fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3c8510 0x55e01e3d7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:41904/0 (socket says 172.21.15.28:41904) 2024-04-25T01:39:10.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.829+0000 7f6d47fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3d7a30 0x55e01e3d4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48292/0 (socket says 172.21.15.28:48292) 2024-04-25T01:39:10.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.829+0000 7f6d477fe640 1 -- 172.21.15.28:0/2379768500 learned_addr learned my addr 172.21.15.28:0/2379768500 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:10.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.829+0000 7f6d46ffd640 1 --2- 172.21.15.28:0/2379768500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8e30 0x55e01e3e3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.829+0000 7f6d47fff640 1 -- 172.21.15.28:0/2379768500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8e30 msgr2=0x55e01e3e3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:10.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.829+0000 7f6d47fff640 1 --2- 172.21.15.28:0/2379768500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8e30 0x55e01e3e3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.829+0000 7f6d47fff640 1 -- 172.21.15.28:0/2379768500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3c8510 msgr2=0x55e01e3d7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:10.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.829+0000 7f6d47fff640 1 --2- 172.21.15.28:0/2379768500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3c8510 0x55e01e3d7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.829+0000 7f6d47fff640 1 -- 172.21.15.28:0/2379768500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e01e3c9210 con 0x55e01e3d7a30 2024-04-25T01:39:10.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.829+0000 7f6d477fe640 1 --2- 172.21.15.28:0/2379768500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3c8510 0x55e01e3d7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:39:10.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.829+0000 7f6d47fff640 1 --2- 172.21.15.28:0/2379768500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3d7a30 0x55e01e3d4220 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f6d40012010 tx=0x7f6d4001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90538fdf7ac954b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:10.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.829+0000 7f6d2d7fa640 1 -- 172.21.15.28:0/2379768500 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d40004b20 con 0x55e01e3d7a30 2024-04-25T01:39:10.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.829+0000 7f6d2d7fa640 1 -- 172.21.15.28:0/2379768500 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d40004cc0 con 0x55e01e3d7a30 2024-04-25T01:39:10.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.829+0000 7f6d2d7fa640 1 -- 172.21.15.28:0/2379768500 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d40028940 con 0x55e01e3d7a30 2024-04-25T01:39:10.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.830+0000 7f6d509e6740 1 -- 172.21.15.28:0/2379768500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3d7a30 msgr2=0x55e01e3d4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:10.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.830+0000 7f6d509e6740 1 --2- 172.21.15.28:0/2379768500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3d7a30 0x55e01e3d4220 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f6d40012010 tx=0x7f6d4001a000 comp rx=0 tx=0).stop 2024-04-25T01:39:10.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.830+0000 7f6d509e6740 1 -- 172.21.15.28:0/2379768500 shutdown_connections 2024-04-25T01:39:10.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.830+0000 7f6d509e6740 1 --2- 172.21.15.28:0/2379768500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3d7a30 0x55e01e3d4220 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.830+0000 7f6d509e6740 1 --2- 172.21.15.28:0/2379768500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8e30 0x55e01e3e3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.830+0000 7f6d509e6740 1 --2- 172.21.15.28:0/2379768500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3c8510 0x55e01e3d7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.830+0000 7f6d509e6740 1 -- 172.21.15.28:0/2379768500 >> 172.21.15.28:0/2379768500 conn(0x55e01e3c4950 msgr2=0x55e01e3c3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:10.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.830+0000 7f6d509e6740 1 -- 172.21.15.28:0/2379768500 shutdown_connections 2024-04-25T01:39:10.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.830+0000 7f6d509e6740 1 -- 172.21.15.28:0/2379768500 wait complete. 2024-04-25T01:39:10.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.830+0000 7f6d509e6740 1 -- start start 2024-04-25T01:39:10.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.831+0000 7f6d509e6740 1 Processor -- start 2024-04-25T01:39:10.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.831+0000 7f6d509e6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3c8510 0x55e01e3dfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.831+0000 7f6d509e6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8e30 0x55e01e3e01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.831+0000 7f6d509e6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3d7a30 0x55e01e3d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.831+0000 7f6d509e6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d481add90 con 0x55e01e3c8510 2024-04-25T01:39:10.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.831+0000 7f6d509e6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6d481ad6c0 con 0x55e01e3d7a30 2024-04-25T01:39:10.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.831+0000 7f6d509e6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6d481aa900 con 0x55e01e3c8e30 2024-04-25T01:39:10.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.831+0000 7f6d509e6740 10 client.0 _ll_register_callbacks cb 0x7f6d3c09cba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:10.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.831+0000 7f6d477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3c8510 0x55e01e3dfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.831+0000 7f6d47fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3d7a30 0x55e01e3d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.831+0000 7f6d477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3c8510 0x55e01e3dfc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48304/0 (socket says 172.21.15.28:48304) 2024-04-25T01:39:10.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.831+0000 7f6d47fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3d7a30 0x55e01e3d0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:41906/0 (socket says 172.21.15.28:41906) 2024-04-25T01:39:10.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.831+0000 7f6d477fe640 1 -- 172.21.15.28:0/3990048460 learned_addr learned my addr 172.21.15.28:0/3990048460 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:10.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.831+0000 7f6d46ffd640 1 --2- 172.21.15.28:0/3990048460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8e30 0x55e01e3e01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.831+0000 7f6d47fff640 1 -- 172.21.15.28:0/3990048460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8e30 msgr2=0x55e01e3e01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:10.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.831+0000 7f6d47fff640 1 --2- 172.21.15.28:0/3990048460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8e30 0x55e01e3e01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.831+0000 7f6d47fff640 1 -- 172.21.15.28:0/3990048460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3c8510 msgr2=0x55e01e3dfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:10.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.831+0000 7f6d47fff640 1 --2- 172.21.15.28:0/3990048460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3c8510 0x55e01e3dfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.832+0000 7f6d47fff640 1 -- 172.21.15.28:0/3990048460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e01e3d1500 con 0x55e01e3d7a30 2024-04-25T01:39:10.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.832+0000 7f6d477fe640 1 --2- 172.21.15.28:0/3990048460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3c8510 0x55e01e3dfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:39:10.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.832+0000 7f6d47fff640 1 --2- 172.21.15.28:0/3990048460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3d7a30 0x55e01e3d0d50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6d4002a040 tx=0x7f6d40010d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcbbb9649e261b93 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:10.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.832+0000 7f6d26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-25T01:39:10.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.832+0000 7f6d26ffd640 1 -- 172.21.15.28:0/3990048460 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d40037070 con 0x55e01e3d7a30 2024-04-25T01:39:10.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.832+0000 7f6d509e6740 1 -- 172.21.15.28:0/3990048460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d40004cc0 con 0x55e01e3d7a30 2024-04-25T01:39:10.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.832+0000 7f6d26ffd640 1 -- 172.21.15.28:0/3990048460 <== mon.2 v2:172.21.15.32:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f6d40010fc0 con 0x55e01e3d7a30 2024-04-25T01:39:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.832+0000 7f6d26ffd640 1 -- 172.21.15.28:0/3990048460 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6d40009480 con 0x55e01e3d7a30 2024-04-25T01:39:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.832+0000 7f6d509e6740 10 client.26705 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.832+0000 7f6d509e6740 1 -- 172.21.15.28:0/3990048460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e01e44daf0 con 0x55e01e3d7a30 2024-04-25T01:39:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.832+0000 7f6d26ffd640 1 -- 172.21.15.28:0/3990048460 <== mon.2 v2:172.21.15.32:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d40004cc0 con 0x55e01e3d7a30 2024-04-25T01:39:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.833+0000 7f6d26ffd640 1 -- 172.21.15.28:0/3990048460 <== mon.2 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6d400335d0 con 0x55e01e3d7a30 2024-04-25T01:39:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.833+0000 7f6d509e6740 10 client.26705 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.833+0000 7f6d509e6740 10 client.26705 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.833+0000 7f6d509e6740 20 client.26705 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.833+0000 7f6d509e6740 1 --2- 172.21.15.28:0/3990048460 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e01e44e1c0 0x55e01e4505b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.833+0000 7f6d509e6740 4 client.26705 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:39:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.833+0000 7f6d509e6740 1 -- 172.21.15.28:0/3990048460 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e01e334f40 con 0x55e01e44e1c0 2024-04-25T01:39:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.833+0000 7f6d477fe640 1 --2- 172.21.15.28:0/3990048460 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e01e44e1c0 0x55e01e4505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.833+0000 7f6d26ffd640 10 client.26705 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.833+0000 7f6d477fe640 1 --2- 172.21.15.28:0/3990048460 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e01e44e1c0 0x55e01e4505b0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dddc3c73549c07f3 server_cookie=b49dcd8af5ce51a3 in_seq=0 out_seq=0 2024-04-25T01:39:10.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d26ffd640 1 -- 172.21.15.28:0/3990048460 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e01e334f40 con 0x55e01e44e1c0 2024-04-25T01:39:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d26ffd640 10 client.26705 handle_command_reply: tid=0 2024-04-25T01:39:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d509e6740 1 client.26705 shutdown 2024-04-25T01:39:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d509e6740 1 -- 172.21.15.28:0/3990048460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3d7a30 msgr2=0x55e01e3d0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d509e6740 1 --2- 172.21.15.28:0/3990048460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3d7a30 0x55e01e3d0d50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6d4002a040 tx=0x7f6d40010d20 comp rx=0 tx=0).stop 2024-04-25T01:39:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d509e6740 1 -- 172.21.15.28:0/3990048460 shutdown_connections 2024-04-25T01:39:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d509e6740 1 --2- 172.21.15.28:0/3990048460 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e01e44e1c0 0x55e01e4505b0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d509e6740 1 --2- 172.21.15.28:0/3990048460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e01e3d7a30 0x55e01e3d0d50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d509e6740 1 --2- 172.21.15.28:0/3990048460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e01e3c8e30 0x55e01e3e01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d509e6740 1 --2- 172.21.15.28:0/3990048460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e01e3c8510 0x55e01e3dfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d26ffd640 0 client.26705 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d509e6740 1 -- 172.21.15.28:0/3990048460 >> 172.21.15.28:0/3990048460 conn(0x55e01e3c4950 msgr2=0x55e01e3c9e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d509e6740 1 -- 172.21.15.28:0/3990048460 shutdown_connections 2024-04-25T01:39:10.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d509e6740 1 -- 172.21.15.28:0/3990048460 wait complete. 2024-04-25T01:39:10.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d509e6740 20 client.26705 trim_cache size 0 max 16384 2024-04-25T01:39:10.823 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:39:10.823 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:39:10.823 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:39:10.823 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:39:10.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d4dcd4640 1 -- 172.21.15.28:0/1865228774 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6d300a2d00 msgr2=0x7f6d300a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:10.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d4dcd4640 1 --2- 172.21.15.28:0/1865228774 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6d300a2d00 0x7f6d300a51c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x55e01e3c9ac0 tx=0x7f6d40009bf0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d4dcd4640 1 -- 172.21.15.28:0/1865228774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d48100610 msgr2=0x7f6d48196f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:10.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d4dcd4640 1 --2- 172.21.15.28:0/1865228774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d48100610 0x7f6d48196f30 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c009970 tx=0x7f6d3c009e30 comp rx=0 tx=0).stop 2024-04-25T01:39:10.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d4dcd4640 1 -- 172.21.15.28:0/1865228774 shutdown_connections 2024-04-25T01:39:10.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d4dcd4640 1 --2- 172.21.15.28:0/1865228774 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6d300a2d00 0x7f6d300a51c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d4dcd4640 1 --2- 172.21.15.28:0/1865228774 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6d481066b0 0x7f6d4819b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d4dcd4640 1 --2- 172.21.15.28:0/1865228774 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6d48100fd0 0x7f6d48197470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.834+0000 7f6d4dcd4640 1 --2- 172.21.15.28:0/1865228774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d48100610 0x7f6d48196f30 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.835+0000 7f6d4dcd4640 1 -- 172.21.15.28:0/1865228774 >> 172.21.15.28:0/1865228774 conn(0x7f6d48077740 msgr2=0x7f6d480ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.835+0000 7f6d4dcd4640 1 -- 172.21.15.28:0/1865228774 shutdown_connections 2024-04-25T01:39:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.835+0000 7f6d4dcd4640 1 -- 172.21.15.28:0/1865228774 wait complete. 2024-04-25T01:39:10.828 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:39:10.828 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:39:10.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.949+0000 7fdb696a5640 1 -- 172.21.15.28:0/3393220170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb6410bd10 msgr2=0x7fdb6410e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:10.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.949+0000 7fdb696a5640 1 --2- 172.21.15.28:0/3393220170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb6410bd10 0x7fdb6410e1b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c009be0 tx=0x7fdb5c02a7e0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.949+0000 7fdb696a5640 1 -- 172.21.15.28:0/3393220170 shutdown_connections 2024-04-25T01:39:10.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.949+0000 7fdb696a5640 1 --2- 172.21.15.28:0/3393220170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb6410bd10 0x7fdb6410e1b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.949+0000 7fdb696a5640 1 --2- 172.21.15.28:0/3393220170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb641032f0 0x7fdb6410b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.949+0000 7fdb696a5640 1 --2- 172.21.15.28:0/3393220170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb641029d0 0x7fdb64102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.949+0000 7fdb696a5640 1 -- 172.21.15.28:0/3393220170 >> 172.21.15.28:0/3393220170 conn(0x7fdb64077740 msgr2=0x7fdb64077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:10.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.949+0000 7fdb696a5640 1 -- 172.21.15.28:0/3393220170 shutdown_connections 2024-04-25T01:39:10.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.949+0000 7fdb696a5640 1 -- 172.21.15.28:0/3393220170 wait complete. 2024-04-25T01:39:10.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.950+0000 7fdb696a5640 1 Processor -- start 2024-04-25T01:39:10.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.950+0000 7fdb696a5640 1 -- start start 2024-04-25T01:39:10.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.950+0000 7fdb696a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb641029d0 0x7fdb64197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.950+0000 7fdb696a5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb641032f0 0x7fdb641984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.950+0000 7fdb696a5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb6410bd10 0x7fdb6419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.950+0000 7fdb696a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb64111ac0 con 0x7fdb641029d0 2024-04-25T01:39:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.950+0000 7fdb696a5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdb64111940 con 0x7fdb641032f0 2024-04-25T01:39:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.950+0000 7fdb696a5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fdb64111c40 con 0x7fdb6410bd10 2024-04-25T01:39:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.951+0000 7fdb62ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb641029d0 0x7fdb64197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.951+0000 7fdb62ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb641029d0 0x7fdb64197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48326/0 (socket says 172.21.15.28:48326) 2024-04-25T01:39:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.951+0000 7fdb62ffd640 1 -- 172.21.15.28:0/1515152319 learned_addr learned my addr 172.21.15.28:0/1515152319 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.951+0000 7fdb627fc640 1 --2- 172.21.15.28:0/1515152319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb641032f0 0x7fdb641984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.951+0000 7fdb637fe640 1 --2- 172.21.15.28:0/1515152319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb6410bd10 0x7fdb6419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.951+0000 7fdb62ffd640 1 -- 172.21.15.28:0/1515152319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb6410bd10 msgr2=0x7fdb6419c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.951+0000 7fdb62ffd640 1 --2- 172.21.15.28:0/1515152319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb6410bd10 0x7fdb6419c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.951+0000 7fdb62ffd640 1 -- 172.21.15.28:0/1515152319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb641032f0 msgr2=0x7fdb641984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:10.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.951+0000 7fdb62ffd640 1 --2- 172.21.15.28:0/1515152319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb641032f0 0x7fdb641984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:10.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.951+0000 7fdb62ffd640 1 -- 172.21.15.28:0/1515152319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb6419cf40 con 0x7fdb641029d0 2024-04-25T01:39:10.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.951+0000 7fdb62ffd640 1 --2- 172.21.15.28:0/1515152319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb641029d0 0x7fdb64197fa0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400e9f0 tx=0x7fdb5400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:10.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.951+0000 7fdb5bfff640 1 -- 172.21.15.28:0/1515152319 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb5400ccc0 con 0x7fdb641029d0 2024-04-25T01:39:10.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.951+0000 7fdb696a5640 1 -- 172.21.15.28:0/1515152319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb6419d230 con 0x7fdb641029d0 2024-04-25T01:39:10.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.951+0000 7fdb5bfff640 1 -- 172.21.15.28:0/1515152319 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb5400ce60 con 0x7fdb641029d0 2024-04-25T01:39:10.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.951+0000 7fdb5bfff640 1 -- 172.21.15.28:0/1515152319 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54010690 con 0x7fdb641029d0 2024-04-25T01:39:10.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.951+0000 7fdb696a5640 1 -- 172.21.15.28:0/1515152319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb641a4ac0 con 0x7fdb641029d0 2024-04-25T01:39:10.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.952+0000 7fdb696a5640 1 -- 172.21.15.28:0/1515152319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb64068560 con 0x7fdb641029d0 2024-04-25T01:39:10.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.954+0000 7fdb5bfff640 1 -- 172.21.15.28:0/1515152319 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fdb540046c0 con 0x7fdb641029d0 2024-04-25T01:39:10.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.954+0000 7fdb5bfff640 1 --2- 172.21.15.28:0/1515152319 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdb500a2d00 0x7fdb500a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:10.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.954+0000 7fdb5bfff640 1 -- 172.21.15.28:0/1515152319 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fdb540cfd10 con 0x7fdb641029d0 2024-04-25T01:39:10.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.954+0000 7fdb627fc640 1 --2- 172.21.15.28:0/1515152319 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdb500a2d00 0x7fdb500a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:10.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.955+0000 7fdb627fc640 1 --2- 172.21.15.28:0/1515152319 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdb500a2d00 0x7fdb500a51c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdb640ffff0 tx=0x7fdb4c009260 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:10.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:10.958+0000 7fdb5bfff640 1 -- 172.21.15.28:0/1515152319 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdb54053740 con 0x7fdb641029d0 2024-04-25T01:39:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.187+0000 7fdb696a5640 1 -- 172.21.15.28:0/1515152319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb64106ec0 con 0x7fdb641029d0 2024-04-25T01:39:11.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.188+0000 7fdb5bfff640 1 -- 172.21.15.28:0/1515152319 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7fdb5409c5f0 con 0x7fdb641029d0 2024-04-25T01:39:11.170 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:39:11.171 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:39:11.171 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:39:11.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.190+0000 7fdb696a5640 1 -- 172.21.15.28:0/1515152319 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdb500a2d00 msgr2=0x7fdb500a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.190+0000 7fdb696a5640 1 --2- 172.21.15.28:0/1515152319 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdb500a2d00 0x7fdb500a51c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdb640ffff0 tx=0x7fdb4c009260 comp rx=0 tx=0).stop 2024-04-25T01:39:11.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.190+0000 7fdb696a5640 1 -- 172.21.15.28:0/1515152319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb641029d0 msgr2=0x7fdb64197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.190+0000 7fdb696a5640 1 --2- 172.21.15.28:0/1515152319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb641029d0 0x7fdb64197fa0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400e9f0 tx=0x7fdb5400edf0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.190+0000 7fdb696a5640 1 -- 172.21.15.28:0/1515152319 shutdown_connections 2024-04-25T01:39:11.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.191+0000 7fdb696a5640 1 --2- 172.21.15.28:0/1515152319 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdb500a2d00 0x7fdb500a51c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.191+0000 7fdb696a5640 1 --2- 172.21.15.28:0/1515152319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdb6410bd10 0x7fdb6419c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.191+0000 7fdb696a5640 1 --2- 172.21.15.28:0/1515152319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdb641032f0 0x7fdb641984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.191+0000 7fdb696a5640 1 --2- 172.21.15.28:0/1515152319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb641029d0 0x7fdb64197fa0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.191+0000 7fdb696a5640 1 -- 172.21.15.28:0/1515152319 >> 172.21.15.28:0/1515152319 conn(0x7fdb64077740 msgr2=0x7fdb640fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.191+0000 7fdb696a5640 1 -- 172.21.15.28:0/1515152319 shutdown_connections 2024-04-25T01:39:11.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.191+0000 7fdb696a5640 1 -- 172.21.15.28:0/1515152319 wait complete. 2024-04-25T01:39:11.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:39:11.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:10 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:39:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.309+0000 7f42cfe66640 1 -- 172.21.15.28:0/236371033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42c8102150 msgr2=0x7f42c810e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.309+0000 7f42cfe66640 1 --2- 172.21.15.28:0/236371033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42c8102150 0x7f42c810e660 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f42c0009cd0 tx=0x7f42c002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.309+0000 7f42cfe66640 1 -- 172.21.15.28:0/236371033 shutdown_connections 2024-04-25T01:39:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.309+0000 7f42cfe66640 1 --2- 172.21.15.28:0/236371033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42c8102150 0x7f42c810e660 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.309+0000 7f42cfe66640 1 --2- 172.21.15.28:0/236371033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42c8101790 0x7f42c8101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.309+0000 7f42cfe66640 1 --2- 172.21.15.28:0/236371033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c81077a0 0x7f42c8107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.309+0000 7f42cfe66640 1 -- 172.21.15.28:0/236371033 >> 172.21.15.28:0/236371033 conn(0x7f42c80fd2a0 msgr2=0x7f42c80ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.309+0000 7f42cfe66640 1 -- 172.21.15.28:0/236371033 shutdown_connections 2024-04-25T01:39:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.310+0000 7f42cfe66640 1 -- 172.21.15.28:0/236371033 wait complete. 2024-04-25T01:39:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.310+0000 7f42cfe66640 1 Processor -- start 2024-04-25T01:39:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.310+0000 7f42cfe66640 1 -- start start 2024-04-25T01:39:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.310+0000 7f42cfe66640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c8101790 0x7f42c8198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.310+0000 7f42cfe66640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42c8102150 0x7f42c81985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.310+0000 7f42cfe66640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42c81077a0 0x7f42c819c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.310+0000 7f42cfe66640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42c8111bb0 con 0x7f42c81077a0 2024-04-25T01:39:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.310+0000 7f42cfe66640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f42c8111a30 con 0x7f42c8101790 2024-04-25T01:39:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.310+0000 7f42cfe66640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f42c8111d30 con 0x7f42c8102150 2024-04-25T01:39:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.311+0000 7f42ce403640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42c81077a0 0x7f42c819c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.311+0000 7f42ce403640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42c81077a0 0x7f42c819c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48348/0 (socket says 172.21.15.28:48348) 2024-04-25T01:39:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.311+0000 7f42ce403640 1 -- 172.21.15.28:0/29862690 learned_addr learned my addr 172.21.15.28:0/29862690 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.311+0000 7f42cd401640 1 --2- 172.21.15.28:0/29862690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42c8102150 0x7f42c81985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.311+0000 7f42cdc02640 1 --2- 172.21.15.28:0/29862690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c8101790 0x7f42c8198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.311+0000 7f42cd401640 1 -- 172.21.15.28:0/29862690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c8101790 msgr2=0x7f42c8198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.311+0000 7f42cd401640 1 --2- 172.21.15.28:0/29862690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c8101790 0x7f42c8198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.311+0000 7f42cd401640 1 -- 172.21.15.28:0/29862690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42c81077a0 msgr2=0x7f42c819c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.311+0000 7f42cd401640 1 --2- 172.21.15.28:0/29862690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42c81077a0 0x7f42c819c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.311+0000 7f42cd401640 1 -- 172.21.15.28:0/29862690 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c819d080 con 0x7f42c8102150 2024-04-25T01:39:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.311+0000 7f42ce403640 1 --2- 172.21.15.28:0/29862690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42c81077a0 0x7f42c819c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:39:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.311+0000 7f42cd401640 1 --2- 172.21.15.28:0/29862690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42c8102150 0x7f42c81985d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f42b800e9f0 tx=0x7f42b800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.311+0000 7f42beffd640 1 -- 172.21.15.28:0/29862690 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b800cd80 con 0x7f42c8102150 2024-04-25T01:39:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.311+0000 7f42cfe66640 1 -- 172.21.15.28:0/29862690 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42c819d370 con 0x7f42c8102150 2024-04-25T01:39:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.311+0000 7f42beffd640 1 -- 172.21.15.28:0/29862690 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42b80035f0 con 0x7f42c8102150 2024-04-25T01:39:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.311+0000 7f42beffd640 1 -- 172.21.15.28:0/29862690 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b8010670 con 0x7f42c8102150 2024-04-25T01:39:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.312+0000 7f42cfe66640 1 -- 172.21.15.28:0/29862690 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42c81a4c00 con 0x7f42c8102150 2024-04-25T01:39:11.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.314+0000 7f42d2b70740 1 Processor -- start 2024-04-25T01:39:11.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.314+0000 7f42d2b70740 1 -- start start 2024-04-25T01:39:11.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.314+0000 7f42d2b70740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c55d580 0x55f93c55d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.314+0000 7f42d2b70740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c55dea0 0x55f93c568030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.314+0000 7f42d2b70740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c568570 0x55f93c56a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.314+0000 7f42d2b70740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f93c4c2670 con 0x55f93c568570 2024-04-25T01:39:11.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.314+0000 7f42d2b70740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f93c46a970 con 0x55f93c55d580 2024-04-25T01:39:11.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.314+0000 7f42beffd640 1 -- 172.21.15.28:0/29862690 <== mon.1 v2:172.21.15.130:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f42b8010810 con 0x7f42c8102150 2024-04-25T01:39:11.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.314+0000 7f42d2b70740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55f93c4be630 con 0x55f93c55dea0 2024-04-25T01:39:11.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.315+0000 7f42ce403640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c55d580 0x55f93c55d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.315+0000 7f42ce403640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c55d580 0x55f93c55d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:41966/0 (socket says 172.21.15.28:41966) 2024-04-25T01:39:11.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.315+0000 7f42cd401640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c568570 0x55f93c56a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.315+0000 7f42ce403640 1 -- 172.21.15.28:0/1658769326 learned_addr learned my addr 172.21.15.28:0/1658769326 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.315+0000 7f42cdc02640 1 --2- 172.21.15.28:0/1658769326 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c55dea0 0x55f93c568030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.315+0000 7f42ce403640 1 -- 172.21.15.28:0/1658769326 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c55dea0 msgr2=0x55f93c568030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.315+0000 7f42ce403640 1 --2- 172.21.15.28:0/1658769326 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c55dea0 0x55f93c568030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.315+0000 7f42ce403640 1 -- 172.21.15.28:0/1658769326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c568570 msgr2=0x55f93c56a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.315+0000 7f42ce403640 1 --2- 172.21.15.28:0/1658769326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c568570 0x55f93c56a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.315+0000 7f42ce403640 1 -- 172.21.15.28:0/1658769326 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f93c48e400 con 0x55f93c55d580 2024-04-25T01:39:11.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.315+0000 7f42beffd640 1 --2- 172.21.15.28:0/29862690 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f42b40a2d00 0x7f42b40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.315+0000 7f42ce403640 1 --2- 172.21.15.28:0/1658769326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c55d580 0x55f93c55d960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f42c0009df0 tx=0x7f42c0009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5119f4e3ba0b3a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.315+0000 7f42cdc02640 1 --2- 172.21.15.28:0/29862690 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f42b40a2d00 0x7f42b40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.315+0000 7f42beffd640 1 -- 172.21.15.28:0/29862690 <== mon.1 v2:172.21.15.130:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f42b80cfa60 con 0x7f42c8102150 2024-04-25T01:39:11.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.316+0000 7f42b3fff640 1 -- 172.21.15.28:0/1658769326 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c0048070 con 0x55f93c55d580 2024-04-25T01:39:11.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.316+0000 7f42b3fff640 1 -- 172.21.15.28:0/1658769326 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42c0036070 con 0x55f93c55d580 2024-04-25T01:39:11.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.316+0000 7f42b3fff640 1 -- 172.21.15.28:0/1658769326 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c0004680 con 0x55f93c55d580 2024-04-25T01:39:11.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.316+0000 7f42d2b70740 1 -- 172.21.15.28:0/1658769326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c55d580 msgr2=0x55f93c55d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.316+0000 7f42cdc02640 1 --2- 172.21.15.28:0/29862690 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f42b40a2d00 0x7f42b40a51c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x55f93c55eb40 tx=0x7f42c4005f90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.316+0000 7f42d2b70740 1 --2- 172.21.15.28:0/1658769326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c55d580 0x55f93c55d960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f42c0009df0 tx=0x7f42c0009a20 comp rx=0 tx=0).stop 2024-04-25T01:39:11.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.316+0000 7f42d2b70740 1 -- 172.21.15.28:0/1658769326 shutdown_connections 2024-04-25T01:39:11.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.316+0000 7f42d2b70740 1 --2- 172.21.15.28:0/1658769326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c568570 0x55f93c56a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.316+0000 7f42d2b70740 1 --2- 172.21.15.28:0/1658769326 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c55dea0 0x55f93c568030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.316+0000 7f42d2b70740 1 --2- 172.21.15.28:0/1658769326 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c55d580 0x55f93c55d960 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.316+0000 7f42d2b70740 1 -- 172.21.15.28:0/1658769326 >> 172.21.15.28:0/1658769326 conn(0x55f93c5598a0 msgr2=0x55f93c559c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.316+0000 7f42d2b70740 1 -- 172.21.15.28:0/1658769326 shutdown_connections 2024-04-25T01:39:11.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.317+0000 7f42d2b70740 1 -- 172.21.15.28:0/1658769326 wait complete. 2024-04-25T01:39:11.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.317+0000 7f42d2b70740 1 -- start start 2024-04-25T01:39:11.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.317+0000 7f42d2b70740 1 Processor -- start 2024-04-25T01:39:11.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.317+0000 7f42d2b70740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c55d580 0x55f93c567f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.317+0000 7f42d2b70740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 0x55f93c565250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.317+0000 7f42d2b70740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c568570 0x55f93c565860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.317+0000 7f42d2b70740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f93c56f710 con 0x55f93c55dea0 2024-04-25T01:39:11.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.317+0000 7f42d2b70740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f93c47e050 con 0x55f93c55d580 2024-04-25T01:39:11.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.317+0000 7f42d2b70740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55f93c56f890 con 0x55f93c568570 2024-04-25T01:39:11.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.317+0000 7f42d2b70740 10 client.0 _ll_register_callbacks cb 0x55f93c1d5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:11.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.317+0000 7f42cd401640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 0x55f93c565250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.317+0000 7f42cd401640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 0x55f93c565250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48372/0 (socket says 172.21.15.28:48372) 2024-04-25T01:39:11.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.317+0000 7f42cdc02640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c55d580 0x55f93c567f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.317+0000 7f42cd401640 1 -- 172.21.15.28:0/947026538 learned_addr learned my addr 172.21.15.28:0/947026538 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.317+0000 7f42ce403640 1 --2- 172.21.15.28:0/947026538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c568570 0x55f93c565860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.318+0000 7f42cd401640 1 -- 172.21.15.28:0/947026538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c568570 msgr2=0x55f93c565860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.318+0000 7f42cd401640 1 --2- 172.21.15.28:0/947026538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c568570 0x55f93c565860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.318+0000 7f42cd401640 1 -- 172.21.15.28:0/947026538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c55d580 msgr2=0x55f93c567f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.318+0000 7f42cd401640 1 --2- 172.21.15.28:0/947026538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c55d580 0x55f93c567f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.318+0000 7f42cd401640 1 -- 172.21.15.28:0/947026538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f93c44fdc0 con 0x55f93c55dea0 2024-04-25T01:39:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.318+0000 7f42cdc02640 1 --2- 172.21.15.28:0/947026538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c55d580 0x55f93c567f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:39:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.318+0000 7f42cd401640 1 --2- 172.21.15.28:0/947026538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 0x55f93c565250 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f42b800c6a0 tx=0x7f42b8097880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3c0ee8c2fd95e0a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.318+0000 7f42b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.318+0000 7f42b17fa640 1 -- 172.21.15.28:0/947026538 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b809b5a0 con 0x55f93c55dea0 2024-04-25T01:39:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.318+0000 7f42b17fa640 1 -- 172.21.15.28:0/947026538 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f42b80093d0 con 0x55f93c55dea0 2024-04-25T01:39:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.318+0000 7f42d2b70740 1 -- 172.21.15.28:0/947026538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42c0036070 con 0x55f93c55dea0 2024-04-25T01:39:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.318+0000 7f42b17fa640 1 -- 172.21.15.28:0/947026538 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f42b809c340 con 0x55f93c55dea0 2024-04-25T01:39:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.318+0000 7f42d2b70740 10 client.17595 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.318+0000 7f42d2b70740 1 -- 172.21.15.28:0/947026538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f93c1d63f0 con 0x55f93c55dea0 2024-04-25T01:39:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.318+0000 7f42b17fa640 1 -- 172.21.15.28:0/947026538 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42c0036070 con 0x55f93c55dea0 2024-04-25T01:39:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.319+0000 7f42b17fa640 1 -- 172.21.15.28:0/947026538 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f42b80982b0 con 0x55f93c55dea0 2024-04-25T01:39:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.319+0000 7f42d2b70740 10 client.17595 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.319+0000 7f42d2b70740 10 client.17595 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:39:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.319+0000 7f42d2b70740 20 client.17595 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.319+0000 7f42d2b70740 1 --2- 172.21.15.28:0/947026538 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f93c57c990 0x55f93c57ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.319+0000 7f42d2b70740 4 client.17595 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.319+0000 7f42d2b70740 1 -- 172.21.15.28:0/947026538 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f93c4cfbb0 con 0x55f93c57c990 2024-04-25T01:39:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.319+0000 7f42cdc02640 1 --2- 172.21.15.28:0/947026538 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f93c57c990 0x55f93c57ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.319+0000 7f42cdc02640 1 --2- 172.21.15.28:0/947026538 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f93c57c990 0x55f93c57ed80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ec608d5b3c4231e7 server_cookie=e194736a2bfc542a in_seq=0 out_seq=0 2024-04-25T01:39:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.319+0000 7f42b17fa640 10 client.17595 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:39:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.320+0000 7f42b17fa640 1 -- 172.21.15.28:0/947026538 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f93c4cfbb0 con 0x55f93c57c990 2024-04-25T01:39:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.320+0000 7f42b17fa640 10 client.17595 handle_command_reply: tid=0 2024-04-25T01:39:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.320+0000 7f42d2b70740 1 client.17595 shutdown 2024-04-25T01:39:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.320+0000 7f42d2b70740 1 -- 172.21.15.28:0/947026538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 msgr2=0x55f93c565250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.320+0000 7f42d2b70740 1 --2- 172.21.15.28:0/947026538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 0x55f93c565250 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f42b800c6a0 tx=0x7f42b8097880 comp rx=0 tx=0).stop 2024-04-25T01:39:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.320+0000 7f42d2b70740 1 -- 172.21.15.28:0/947026538 shutdown_connections 2024-04-25T01:39:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.320+0000 7f42d2b70740 1 --2- 172.21.15.28:0/947026538 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f93c57c990 0x55f93c57ed80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.320+0000 7f42d2b70740 1 --2- 172.21.15.28:0/947026538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c568570 0x55f93c565860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.320+0000 7f42b17fa640 0 client.17595 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:39:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.320+0000 7f42d2b70740 1 --2- 172.21.15.28:0/947026538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 0x55f93c565250 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.320+0000 7f42d2b70740 1 --2- 172.21.15.28:0/947026538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c55d580 0x55f93c567f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.320+0000 7f42d2b70740 1 -- 172.21.15.28:0/947026538 >> 172.21.15.28:0/947026538 conn(0x55f93c5598a0 msgr2=0x55f93c55f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.320+0000 7f42d2b70740 1 -- 172.21.15.28:0/947026538 shutdown_connections 2024-04-25T01:39:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.320+0000 7f42d2b70740 1 -- 172.21.15.28:0/947026538 wait complete. 2024-04-25T01:39:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.320+0000 7f42d2b70740 20 client.17595 trim_cache size 0 max 16384 2024-04-25T01:39:11.315 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:10 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:39:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.334+0000 7f42d2b70740 1 Processor -- start 2024-04-25T01:39:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.334+0000 7f42d2b70740 1 -- start start 2024-04-25T01:39:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.334+0000 7f42d2b70740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c55d580 0x55f93c5785d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.334+0000 7f42d2b70740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 0x55f93c584610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.334+0000 7f42d2b70740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c568570 0x55f93c584b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.334+0000 7f42d2b70740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42c81ac780 con 0x55f93c55dea0 2024-04-25T01:39:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.334+0000 7f42d2b70740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f42c81ae2c0 con 0x55f93c568570 2024-04-25T01:39:11.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.334+0000 7f42d2b70740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f42c81ace50 con 0x55f93c55d580 2024-04-25T01:39:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.334+0000 7f42cd401640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 0x55f93c584610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.334+0000 7f42cd401640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 0x55f93c584610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48374/0 (socket says 172.21.15.28:48374) 2024-04-25T01:39:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.334+0000 7f42cd401640 1 -- 172.21.15.28:0/1850113500 learned_addr learned my addr 172.21.15.28:0/1850113500 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.334+0000 7f42cdc02640 1 --2- 172.21.15.28:0/1850113500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c55d580 0x55f93c5785d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.334+0000 7f42ce403640 1 --2- 172.21.15.28:0/1850113500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c568570 0x55f93c584b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.334+0000 7f42cd401640 1 -- 172.21.15.28:0/1850113500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c55d580 msgr2=0x55f93c5785d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.334+0000 7f42cd401640 1 --2- 172.21.15.28:0/1850113500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c55d580 0x55f93c5785d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.334+0000 7f42cd401640 1 -- 172.21.15.28:0/1850113500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c568570 msgr2=0x55f93c584b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.334+0000 7f42cd401640 1 --2- 172.21.15.28:0/1850113500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c568570 0x55f93c584b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.334+0000 7f42cd401640 1 -- 172.21.15.28:0/1850113500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f93c565bd0 con 0x55f93c55dea0 2024-04-25T01:39:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.335+0000 7f42cd401640 1 --2- 172.21.15.28:0/1850113500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 0x55f93c584610 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f42b809b940 tx=0x7f42b8005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a7085b7d8fe3cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.335+0000 7f42b37fe640 1 -- 172.21.15.28:0/1850113500 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b809b5a0 con 0x55f93c55dea0 2024-04-25T01:39:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.335+0000 7f42b37fe640 1 -- 172.21.15.28:0/1850113500 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42b8086040 con 0x55f93c55dea0 2024-04-25T01:39:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.335+0000 7f42b37fe640 1 -- 172.21.15.28:0/1850113500 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b809ba10 con 0x55f93c55dea0 2024-04-25T01:39:11.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.336+0000 7f42d2b70740 1 -- 172.21.15.28:0/1850113500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 msgr2=0x55f93c584610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.336+0000 7f42d2b70740 1 --2- 172.21.15.28:0/1850113500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 0x55f93c584610 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f42b809b940 tx=0x7f42b8005e30 comp rx=0 tx=0).stop 2024-04-25T01:39:11.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.336+0000 7f42d2b70740 1 -- 172.21.15.28:0/1850113500 shutdown_connections 2024-04-25T01:39:11.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.336+0000 7f42d2b70740 1 --2- 172.21.15.28:0/1850113500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c568570 0x55f93c584b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.336+0000 7f42d2b70740 1 --2- 172.21.15.28:0/1850113500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 0x55f93c584610 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.336+0000 7f42d2b70740 1 --2- 172.21.15.28:0/1850113500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c55d580 0x55f93c5785d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.336+0000 7f42d2b70740 1 -- 172.21.15.28:0/1850113500 >> 172.21.15.28:0/1850113500 conn(0x55f93c5598a0 msgr2=0x55f93c558b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.336+0000 7f42d2b70740 1 -- 172.21.15.28:0/1850113500 shutdown_connections 2024-04-25T01:39:11.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.336+0000 7f42d2b70740 1 -- 172.21.15.28:0/1850113500 wait complete. 2024-04-25T01:39:11.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.336+0000 7f42d2b70740 1 -- start start 2024-04-25T01:39:11.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.336+0000 7f42d2b70740 1 Processor -- start 2024-04-25T01:39:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.336+0000 7f42d2b70740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c55d580 0x55f93c5728f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.336+0000 7f42d2b70740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 0x55f93c572e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.336+0000 7f42d2b70740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c568570 0x55f93c5e1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.336+0000 7f42d2b70740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42c81adbf0 con 0x55f93c55dea0 2024-04-25T01:39:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.336+0000 7f42d2b70740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f42c81ad520 con 0x55f93c55d580 2024-04-25T01:39:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.336+0000 7f42d2b70740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f42c81aa760 con 0x55f93c568570 2024-04-25T01:39:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.336+0000 7f42d2b70740 10 client.0 _ll_register_callbacks cb 0x7f42b800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.337+0000 7f42cd401640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 0x55f93c572e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.337+0000 7f42cdc02640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c55d580 0x55f93c5728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.337+0000 7f42ce403640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c568570 0x55f93c5e1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.337+0000 7f42ce403640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c568570 0x55f93c5e1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.28:54594/0 (socket says 172.21.15.28:54594) 2024-04-25T01:39:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.337+0000 7f42ce403640 1 -- 172.21.15.28:0/3765977582 learned_addr learned my addr 172.21.15.28:0/3765977582 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.337+0000 7f42cdc02640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c55d580 0x55f93c5728f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:41996/0 (socket says 172.21.15.28:41996) 2024-04-25T01:39:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.337+0000 7f42cd401640 1 -- 172.21.15.28:0/3765977582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c568570 msgr2=0x55f93c5e1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.337+0000 7f42cd401640 1 --2- 172.21.15.28:0/3765977582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c568570 0x55f93c5e1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.337+0000 7f42cd401640 1 -- 172.21.15.28:0/3765977582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c55d580 msgr2=0x55f93c5728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.337+0000 7f42cd401640 1 --2- 172.21.15.28:0/3765977582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c55d580 0x55f93c5728f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.337+0000 7f42cd401640 1 -- 172.21.15.28:0/3765977582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f93c2a07d0 con 0x55f93c55dea0 2024-04-25T01:39:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.337+0000 7f42cd401640 1 --2- 172.21.15.28:0/3765977582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 0x55f93c572e30 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x55f93c57b6d0 tx=0x7f42b80cfe60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4384a6c365af4938 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.337+0000 7f42b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.337+0000 7f42b17fa640 1 -- 172.21.15.28:0/3765977582 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b809ba10 con 0x55f93c55dea0 2024-04-25T01:39:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.337+0000 7f42b17fa640 1 -- 172.21.15.28:0/3765977582 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f42b8010b00 con 0x55f93c55dea0 2024-04-25T01:39:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.337+0000 7f42d2b70740 1 -- 172.21.15.28:0/3765977582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42b8086040 con 0x55f93c55dea0 2024-04-25T01:39:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.337+0000 7f42b17fa640 1 -- 172.21.15.28:0/3765977582 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f42b809b5a0 con 0x55f93c55dea0 2024-04-25T01:39:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.337+0000 7f42d2b70740 10 client.17601 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.337+0000 7f42d2b70740 1 -- 172.21.15.28:0/3765977582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f93c2a07d0 con 0x55f93c55dea0 2024-04-25T01:39:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.338+0000 7f42b17fa640 1 -- 172.21.15.28:0/3765977582 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42b8086040 con 0x55f93c55dea0 2024-04-25T01:39:11.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.338+0000 7f42b17fa640 1 -- 172.21.15.28:0/3765977582 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f42b80b4050 con 0x55f93c55dea0 2024-04-25T01:39:11.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.338+0000 7f42d2b70740 10 client.17601 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:11.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.338+0000 7f42d2b70740 10 client.17601 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:39:11.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.338+0000 7f42d2b70740 20 client.17601 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:11.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.338+0000 7f42d2b70740 1 --2- 172.21.15.28:0/3765977582 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f93c5e3650 0x55f93c5e5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.338+0000 7f42d2b70740 4 client.17601 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:11.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.338+0000 7f42d2b70740 1 -- 172.21.15.28:0/3765977582 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f93c4c9f00 con 0x55f93c5e3650 2024-04-25T01:39:11.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.338+0000 7f42cdc02640 1 --2- 172.21.15.28:0/3765977582 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f93c5e3650 0x55f93c5e5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.339+0000 7f42b17fa640 10 client.17601 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:39:11.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.339+0000 7f42cdc02640 1 --2- 172.21.15.28:0/3765977582 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f93c5e3650 0x55f93c5e5a40 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a6b6edca4c91d296 server_cookie=738c4cefb1eecc25 in_seq=0 out_seq=0 2024-04-25T01:39:11.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.339+0000 7f42b17fa640 1 -- 172.21.15.28:0/3765977582 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f93c4c9f00 con 0x55f93c5e3650 2024-04-25T01:39:11.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.339+0000 7f42b17fa640 10 client.17601 handle_command_reply: tid=0 2024-04-25T01:39:11.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.339+0000 7f42d2b70740 1 client.17601 shutdown 2024-04-25T01:39:11.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.339+0000 7f42d2b70740 1 -- 172.21.15.28:0/3765977582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 msgr2=0x55f93c572e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.339+0000 7f42d2b70740 1 --2- 172.21.15.28:0/3765977582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 0x55f93c572e30 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x55f93c57b6d0 tx=0x7f42b80cfe60 comp rx=0 tx=0).stop 2024-04-25T01:39:11.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.339+0000 7f42d2b70740 1 -- 172.21.15.28:0/3765977582 shutdown_connections 2024-04-25T01:39:11.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.339+0000 7f42d2b70740 1 --2- 172.21.15.28:0/3765977582 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f93c5e3650 0x55f93c5e5a40 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.339+0000 7f42d2b70740 1 --2- 172.21.15.28:0/3765977582 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f93c568570 0x55f93c5e1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.339+0000 7f42b17fa640 0 client.17601 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:39:11.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.339+0000 7f42d2b70740 1 --2- 172.21.15.28:0/3765977582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f93c55dea0 0x55f93c572e30 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.339+0000 7f42d2b70740 1 --2- 172.21.15.28:0/3765977582 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f93c55d580 0x55f93c5728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.339+0000 7f42d2b70740 1 -- 172.21.15.28:0/3765977582 >> 172.21.15.28:0/3765977582 conn(0x55f93c5598a0 msgr2=0x55f93c5775c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.339+0000 7f42d2b70740 1 -- 172.21.15.28:0/3765977582 shutdown_connections 2024-04-25T01:39:11.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.339+0000 7f42d2b70740 1 -- 172.21.15.28:0/3765977582 wait complete. 2024-04-25T01:39:11.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.339+0000 7f42d2b70740 20 client.17601 trim_cache size 0 max 16384 2024-04-25T01:39:11.328 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.340+0000 7f42cfe66640 1 -- 172.21.15.28:0/29862690 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f42b40a2d00 msgr2=0x7f42b40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.340+0000 7f42cfe66640 1 --2- 172.21.15.28:0/29862690 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f42b40a2d00 0x7f42b40a51c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x55f93c55eb40 tx=0x7f42c4005f90 comp rx=0 tx=0).stop 2024-04-25T01:39:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.340+0000 7f42cfe66640 1 -- 172.21.15.28:0/29862690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42c8102150 msgr2=0x7f42c81985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.340+0000 7f42cfe66640 1 --2- 172.21.15.28:0/29862690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42c8102150 0x7f42c81985d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f42b800e9f0 tx=0x7f42b800eeb0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.340+0000 7f42cfe66640 1 -- 172.21.15.28:0/29862690 shutdown_connections 2024-04-25T01:39:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.340+0000 7f42cfe66640 1 --2- 172.21.15.28:0/29862690 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f42b40a2d00 0x7f42b40a51c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.340+0000 7f42cfe66640 1 --2- 172.21.15.28:0/29862690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42c81077a0 0x7f42c819c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.340+0000 7f42cfe66640 1 --2- 172.21.15.28:0/29862690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f42c8102150 0x7f42c81985d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.340+0000 7f42cfe66640 1 --2- 172.21.15.28:0/29862690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c8101790 0x7f42c8198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.340+0000 7f42cfe66640 1 -- 172.21.15.28:0/29862690 >> 172.21.15.28:0/29862690 conn(0x7f42c80fd2a0 msgr2=0x7f42c80fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.340+0000 7f42cfe66640 1 -- 172.21.15.28:0/29862690 shutdown_connections 2024-04-25T01:39:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.340+0000 7f42cfe66640 1 -- 172.21.15.28:0/29862690 wait complete. 2024-04-25T01:39:11.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:39:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.458+0000 7fc38939e640 1 -- 172.21.15.28:0/2753925027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3841066b0 msgr2=0x7fc384106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.458+0000 7fc38939e640 1 --2- 172.21.15.28:0/2753925027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3841066b0 0x7fc384106a90 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fc378009930 tx=0x7fc37802a420 comp rx=0 tx=0).stop 2024-04-25T01:39:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.458+0000 7fc38939e640 1 -- 172.21.15.28:0/2753925027 shutdown_connections 2024-04-25T01:39:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.458+0000 7fc38939e640 1 --2- 172.21.15.28:0/2753925027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc384100fd0 0x7fc38410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.458+0000 7fc38939e640 1 --2- 172.21.15.28:0/2753925027 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc384100610 0x7fc384100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.458+0000 7fc38939e640 1 --2- 172.21.15.28:0/2753925027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3841066b0 0x7fc384106a90 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.458+0000 7fc38939e640 1 -- 172.21.15.28:0/2753925027 >> 172.21.15.28:0/2753925027 conn(0x7fc384077740 msgr2=0x7fc384077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.458+0000 7fc38939e640 1 -- 172.21.15.28:0/2753925027 shutdown_connections 2024-04-25T01:39:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.458+0000 7fc38939e640 1 -- 172.21.15.28:0/2753925027 wait complete. 2024-04-25T01:39:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.458+0000 7fc38939e640 1 Processor -- start 2024-04-25T01:39:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.458+0000 7fc38939e640 1 -- start start 2024-04-25T01:39:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.459+0000 7fc38939e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc384100610 0x7fc384197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.459+0000 7fc38939e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc384100fd0 0x7fc384198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.459+0000 7fc38939e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3841066b0 0x7fc38419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.459+0000 7fc38939e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc384110a80 con 0x7fc3841066b0 2024-04-25T01:39:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.459+0000 7fc38939e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc384110900 con 0x7fc384100610 2024-04-25T01:39:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.459+0000 7fc38939e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc384110c00 con 0x7fc384100fd0 2024-04-25T01:39:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.459+0000 7fc3837fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3841066b0 0x7fc38419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.459+0000 7fc3837fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3841066b0 0x7fc38419c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48402/0 (socket says 172.21.15.28:48402) 2024-04-25T01:39:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.459+0000 7fc3837fe640 1 -- 172.21.15.28:0/81740372 learned_addr learned my addr 172.21.15.28:0/81740372 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.459+0000 7fc3827fc640 1 --2- 172.21.15.28:0/81740372 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc384100fd0 0x7fc384198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.459+0000 7fc382ffd640 1 --2- 172.21.15.28:0/81740372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc384100610 0x7fc384197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.459+0000 7fc3837fe640 1 -- 172.21.15.28:0/81740372 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc384100fd0 msgr2=0x7fc384198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.459+0000 7fc3837fe640 1 --2- 172.21.15.28:0/81740372 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc384100fd0 0x7fc384198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.459+0000 7fc3837fe640 1 -- 172.21.15.28:0/81740372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc384100610 msgr2=0x7fc384197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.459+0000 7fc3837fe640 1 --2- 172.21.15.28:0/81740372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc384100610 0x7fc384197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.459+0000 7fc3837fe640 1 -- 172.21.15.28:0/81740372 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc38419cfd0 con 0x7fc3841066b0 2024-04-25T01:39:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.459+0000 7fc3837fe640 1 --2- 172.21.15.28:0/81740372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3841066b0 0x7fc38419c8a0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fc37c00eb10 tx=0x7fc37c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.460+0000 7fc36bfff640 1 -- 172.21.15.28:0/81740372 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c004b20 con 0x7fc3841066b0 2024-04-25T01:39:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.460+0000 7fc36bfff640 1 -- 172.21.15.28:0/81740372 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc37c004cc0 con 0x7fc3841066b0 2024-04-25T01:39:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.460+0000 7fc38939e640 1 -- 172.21.15.28:0/81740372 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc38419d2c0 con 0x7fc3841066b0 2024-04-25T01:39:11.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.460+0000 7fc36bfff640 1 -- 172.21.15.28:0/81740372 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c010760 con 0x7fc3841066b0 2024-04-25T01:39:11.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.460+0000 7fc38939e640 1 -- 172.21.15.28:0/81740372 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3841a4b50 con 0x7fc3841066b0 2024-04-25T01:39:11.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.462+0000 7fc36bfff640 1 -- 172.21.15.28:0/81740372 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc37c005020 con 0x7fc3841066b0 2024-04-25T01:39:11.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.462+0000 7fc38c189740 1 Processor -- start 2024-04-25T01:39:11.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.462+0000 7fc38c189740 1 -- start start 2024-04-25T01:39:11.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.462+0000 7fc38c189740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 0x555c37da7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.462+0000 7fc38c189740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37da7ea0 0x555c37db2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.462+0000 7fc38c189740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37db2570 0x555c37db4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.463+0000 7fc38c189740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555c37d0c670 con 0x555c37da7580 2024-04-25T01:39:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.463+0000 7fc38c189740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555c37cb4970 con 0x555c37da7ea0 2024-04-25T01:39:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.463+0000 7fc38c189740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x555c37d08630 con 0x555c37db2570 2024-04-25T01:39:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.463+0000 7fc3827fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 0x555c37da7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.463+0000 7fc3827fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 0x555c37da7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48412/0 (socket says 172.21.15.28:48412) 2024-04-25T01:39:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.463+0000 7fc3827fc640 1 -- 172.21.15.28:0/3189105210 learned_addr learned my addr 172.21.15.28:0/3189105210 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.463+0000 7fc36bfff640 1 --2- 172.21.15.28:0/81740372 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc36c0a2d00 0x7fc36c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.463+0000 7fc382ffd640 1 --2- 172.21.15.28:0/3189105210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37da7ea0 0x555c37db2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.463+0000 7fc3827fc640 1 -- 172.21.15.28:0/3189105210 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37db2570 msgr2=0x555c37db4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.463+0000 7fc3827fc640 1 --2- 172.21.15.28:0/3189105210 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37db2570 0x555c37db4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.463+0000 7fc3827fc640 1 -- 172.21.15.28:0/3189105210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37da7ea0 msgr2=0x555c37db2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.463+0000 7fc3827fc640 1 --2- 172.21.15.28:0/3189105210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37da7ea0 0x555c37db2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.463+0000 7fc3837fe640 1 --2- 172.21.15.28:0/81740372 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc36c0a2d00 0x7fc36c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.463+0000 7fc3827fc640 1 -- 172.21.15.28:0/3189105210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c37cd8400 con 0x555c37da7580 2024-04-25T01:39:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.463+0000 7fc36bfff640 1 -- 172.21.15.28:0/81740372 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc37c0cfd40 con 0x7fc3841066b0 2024-04-25T01:39:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.463+0000 7fc3827fc640 1 --2- 172.21.15.28:0/3189105210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 0x555c37da7960 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fc374009d20 tx=0x7fc374013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2e3bb323ea2bf61 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.463+0000 7fc3697fa640 1 -- 172.21.15.28:0/3189105210 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc374004050 con 0x555c37da7580 2024-04-25T01:39:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.463+0000 7fc3697fa640 1 -- 172.21.15.28:0/3189105210 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc374017030 con 0x555c37da7580 2024-04-25T01:39:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.463+0000 7fc3697fa640 1 -- 172.21.15.28:0/3189105210 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc374010430 con 0x555c37da7580 2024-04-25T01:39:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.464+0000 7fc38c189740 1 -- 172.21.15.28:0/3189105210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 msgr2=0x555c37da7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.464+0000 7fc38c189740 1 --2- 172.21.15.28:0/3189105210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 0x555c37da7960 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fc374009d20 tx=0x7fc374013830 comp rx=0 tx=0).stop 2024-04-25T01:39:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.464+0000 7fc3837fe640 1 --2- 172.21.15.28:0/81740372 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc36c0a2d00 0x7fc36c0a51c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc37c09bc20 tx=0x7fc37c06dee0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.464+0000 7fc38c189740 1 -- 172.21.15.28:0/3189105210 shutdown_connections 2024-04-25T01:39:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.464+0000 7fc38c189740 1 --2- 172.21.15.28:0/3189105210 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37db2570 0x555c37db4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.464+0000 7fc38c189740 1 --2- 172.21.15.28:0/3189105210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37da7ea0 0x555c37db2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.464+0000 7fc38c189740 1 --2- 172.21.15.28:0/3189105210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 0x555c37da7960 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.464+0000 7fc38c189740 1 -- 172.21.15.28:0/3189105210 >> 172.21.15.28:0/3189105210 conn(0x555c37da38a0 msgr2=0x555c37da3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.464+0000 7fc38c189740 1 -- 172.21.15.28:0/3189105210 shutdown_connections 2024-04-25T01:39:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.464+0000 7fc38c189740 1 -- 172.21.15.28:0/3189105210 wait complete. 2024-04-25T01:39:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.464+0000 7fc38c189740 1 -- start start 2024-04-25T01:39:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc38c189740 1 Processor -- start 2024-04-25T01:39:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc38c189740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 0x555c37db1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc38c189740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37da7ea0 0x555c37daf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc38c189740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37db2570 0x555c37daf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc38c189740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555c37db5360 con 0x555c37da7580 2024-04-25T01:39:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc38c189740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555c37cc8050 con 0x555c37db2570 2024-04-25T01:39:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc38c189740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x555c37db54e0 con 0x555c37da7ea0 2024-04-25T01:39:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc38c189740 10 client.0 _ll_register_callbacks cb 0x555c37a1fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc3827fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 0x555c37db1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc3827fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 0x555c37db1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48424/0 (socket says 172.21.15.28:48424) 2024-04-25T01:39:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc3827fc640 1 -- 172.21.15.28:0/2587070245 learned_addr learned my addr 172.21.15.28:0/2587070245 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc382ffd640 1 --2- 172.21.15.28:0/2587070245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37da7ea0 0x555c37daf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc3827fc640 1 --2- 172.21.15.28:0/2587070245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37db2570 0x555c37daf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc3827fc640 1 -- 172.21.15.28:0/2587070245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37da7ea0 msgr2=0x555c37daf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc3827fc640 1 --2- 172.21.15.28:0/2587070245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37da7ea0 0x555c37daf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc3827fc640 1 -- 172.21.15.28:0/2587070245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37db2570 msgr2=0x555c37daf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc3827fc640 1 --2- 172.21.15.28:0/2587070245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37db2570 0x555c37daf6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc3827fc640 1 -- 172.21.15.28:0/2587070245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c37c99dc0 con 0x555c37da7580 2024-04-25T01:39:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc3827fc640 1 --2- 172.21.15.28:0/2587070245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 0x555c37db1ea0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fc374003c20 tx=0x7fc3740045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67d6c6856bad65d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.465+0000 7fc366ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.466+0000 7fc366ff5640 1 -- 172.21.15.28:0/2587070245 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc374003f80 con 0x555c37da7580 2024-04-25T01:39:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.466+0000 7fc366ff5640 1 -- 172.21.15.28:0/2587070245 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc374004120 con 0x555c37da7580 2024-04-25T01:39:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.466+0000 7fc38c189740 1 -- 172.21.15.28:0/2587070245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc374017030 con 0x555c37da7580 2024-04-25T01:39:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.466+0000 7fc366ff5640 1 -- 172.21.15.28:0/2587070245 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc374004320 con 0x555c37da7580 2024-04-25T01:39:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.466+0000 7fc38c189740 10 client.17613 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.466+0000 7fc38c189740 1 -- 172.21.15.28:0/2587070245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555c37a203f0 con 0x555c37da7580 2024-04-25T01:39:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.466+0000 7fc366ff5640 1 -- 172.21.15.28:0/2587070245 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc374017030 con 0x555c37da7580 2024-04-25T01:39:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.466+0000 7fc366ff5640 1 -- 172.21.15.28:0/2587070245 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc37401c520 con 0x555c37da7580 2024-04-25T01:39:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.466+0000 7fc38c189740 10 client.17613 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.466+0000 7fc38c189740 10 client.17613 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:39:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.466+0000 7fc38c189740 20 client.17613 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.466+0000 7fc38c189740 1 --2- 172.21.15.28:0/2587070245 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x555c37dc2490 0x555c37dc4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.466+0000 7fc38c189740 4 client.17613 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.466+0000 7fc38c189740 1 -- 172.21.15.28:0/2587070245 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c37d19bb0 con 0x555c37dc2490 2024-04-25T01:39:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.466+0000 7fc382ffd640 1 --2- 172.21.15.28:0/2587070245 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x555c37dc2490 0x555c37dc4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.467+0000 7fc382ffd640 1 --2- 172.21.15.28:0/2587070245 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x555c37dc2490 0x555c37dc4880 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2d838720762b62e1 server_cookie=7d287d0ee5b49bf6 in_seq=0 out_seq=0 2024-04-25T01:39:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.467+0000 7fc366ff5640 10 client.17613 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:39:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.467+0000 7fc366ff5640 1 -- 172.21.15.28:0/2587070245 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555c37d19bb0 con 0x555c37dc2490 2024-04-25T01:39:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.467+0000 7fc366ff5640 10 client.17613 handle_command_reply: tid=0 2024-04-25T01:39:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.467+0000 7fc38c189740 1 client.17613 shutdown 2024-04-25T01:39:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.467+0000 7fc38c189740 1 -- 172.21.15.28:0/2587070245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 msgr2=0x555c37db1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.467+0000 7fc38c189740 1 --2- 172.21.15.28:0/2587070245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 0x555c37db1ea0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fc374003c20 tx=0x7fc3740045a0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.467+0000 7fc38c189740 1 -- 172.21.15.28:0/2587070245 shutdown_connections 2024-04-25T01:39:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.468+0000 7fc38c189740 1 --2- 172.21.15.28:0/2587070245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37db2570 0x555c37daf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.468+0000 7fc38c189740 1 --2- 172.21.15.28:0/2587070245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37da7ea0 0x555c37daf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.468+0000 7fc38c189740 1 --2- 172.21.15.28:0/2587070245 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x555c37dc2490 0x555c37dc4880 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.468+0000 7fc38c189740 1 --2- 172.21.15.28:0/2587070245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 0x555c37db1ea0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.468+0000 7fc366ff5640 0 client.17613 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:39:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.468+0000 7fc38c189740 1 -- 172.21.15.28:0/2587070245 >> 172.21.15.28:0/2587070245 conn(0x555c37da38a0 msgr2=0x555c37da96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.468+0000 7fc38c189740 1 -- 172.21.15.28:0/2587070245 shutdown_connections 2024-04-25T01:39:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.468+0000 7fc38c189740 1 -- 172.21.15.28:0/2587070245 wait complete. 2024-04-25T01:39:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.468+0000 7fc38c189740 20 client.17613 trim_cache size 0 max 16384 2024-04-25T01:39:11.462 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:10 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:39:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.481+0000 7fc38c189740 1 Processor -- start 2024-04-25T01:39:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.481+0000 7fc38c189740 1 -- start start 2024-04-25T01:39:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.481+0000 7fc38c189740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 0x555c37dbe220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.481+0000 7fc38c189740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37da7ea0 0x555c37dae310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.481+0000 7fc38c189740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37db2570 0x555c37daa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.481+0000 7fc38c189740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3841aeb00 con 0x555c37da7580 2024-04-25T01:39:11.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.481+0000 7fc38c189740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc3841b0640 con 0x555c37da7ea0 2024-04-25T01:39:11.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.481+0000 7fc38c189740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc3841af1d0 con 0x555c37db2570 2024-04-25T01:39:11.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.481+0000 7fc3827fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 0x555c37dbe220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.481+0000 7fc3827fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 0x555c37dbe220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48438/0 (socket says 172.21.15.28:48438) 2024-04-25T01:39:11.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.481+0000 7fc3827fc640 1 -- 172.21.15.28:0/3131794232 learned_addr learned my addr 172.21.15.28:0/3131794232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.481+0000 7fc382ffd640 1 --2- 172.21.15.28:0/3131794232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37da7ea0 0x555c37dae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.481+0000 7fc3827fc640 1 --2- 172.21.15.28:0/3131794232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37db2570 0x555c37daa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.482+0000 7fc3827fc640 1 -- 172.21.15.28:0/3131794232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37db2570 msgr2=0x555c37daa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.482+0000 7fc3827fc640 1 --2- 172.21.15.28:0/3131794232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37db2570 0x555c37daa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.482+0000 7fc3827fc640 1 -- 172.21.15.28:0/3131794232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37da7ea0 msgr2=0x555c37dae310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.482+0000 7fc3827fc640 1 --2- 172.21.15.28:0/3131794232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37da7ea0 0x555c37dae310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.482+0000 7fc3827fc640 1 -- 172.21.15.28:0/3131794232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c37da7cd0 con 0x555c37da7580 2024-04-25T01:39:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.482+0000 7fc3827fc640 1 --2- 172.21.15.28:0/3131794232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 0x555c37dbe220 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fc374006010 tx=0x7fc374003b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c775361ea537c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.482+0000 7fc368ff9640 1 -- 172.21.15.28:0/3131794232 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3740099c0 con 0x555c37da7580 2024-04-25T01:39:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.482+0000 7fc368ff9640 1 -- 172.21.15.28:0/3131794232 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc374017040 con 0x555c37da7580 2024-04-25T01:39:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.482+0000 7fc368ff9640 1 -- 172.21.15.28:0/3131794232 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc374005030 con 0x555c37da7580 2024-04-25T01:39:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.482+0000 7fc38c189740 1 -- 172.21.15.28:0/3131794232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 msgr2=0x555c37dbe220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.482+0000 7fc38c189740 1 --2- 172.21.15.28:0/3131794232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 0x555c37dbe220 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fc374006010 tx=0x7fc374003b70 comp rx=0 tx=0).stop 2024-04-25T01:39:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc38c189740 1 -- 172.21.15.28:0/3131794232 shutdown_connections 2024-04-25T01:39:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc38c189740 1 --2- 172.21.15.28:0/3131794232 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37db2570 0x555c37daa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc38c189740 1 --2- 172.21.15.28:0/3131794232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37da7ea0 0x555c37dae310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc38c189740 1 --2- 172.21.15.28:0/3131794232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37da7580 0x555c37dbe220 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc38c189740 1 -- 172.21.15.28:0/3131794232 >> 172.21.15.28:0/3131794232 conn(0x555c37da38a0 msgr2=0x555c37da9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc38c189740 1 -- 172.21.15.28:0/3131794232 shutdown_connections 2024-04-25T01:39:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc38c189740 1 -- 172.21.15.28:0/3131794232 wait complete. 2024-04-25T01:39:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc38c189740 1 -- start start 2024-04-25T01:39:11.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc38c189740 1 Processor -- start 2024-04-25T01:39:11.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc38c189740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37da7580 0x555c37dace90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc38c189740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37da7ea0 0x555c37dba730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc38c189740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37db2570 0x555c37dab420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc38c189740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3841aff70 con 0x555c37db2570 2024-04-25T01:39:11.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc38c189740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc3841af8a0 con 0x555c37da7580 2024-04-25T01:39:11.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc38c189740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc3841acae0 con 0x555c37da7ea0 2024-04-25T01:39:11.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc38c189740 10 client.0 _ll_register_callbacks cb 0x7fc374009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:11.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc3827fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37db2570 0x555c37dab420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc382ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37da7ea0 0x555c37dba730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc3827fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37db2570 0x555c37dab420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48448/0 (socket says 172.21.15.28:48448) 2024-04-25T01:39:11.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc382ffd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37da7ea0 0x555c37dba730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.28:54638/0 (socket says 172.21.15.28:54638) 2024-04-25T01:39:11.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc3827fc640 1 -- 172.21.15.28:0/3439689794 learned_addr learned my addr 172.21.15.28:0/3439689794 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.483+0000 7fc3827fc640 1 --2- 172.21.15.28:0/3439689794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37da7580 0x555c37dace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.484+0000 7fc382ffd640 1 -- 172.21.15.28:0/3439689794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37da7580 msgr2=0x555c37dace90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.484+0000 7fc382ffd640 1 --2- 172.21.15.28:0/3439689794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37da7580 0x555c37dace90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.484+0000 7fc382ffd640 1 -- 172.21.15.28:0/3439689794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37db2570 msgr2=0x555c37dab420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.484+0000 7fc382ffd640 1 --2- 172.21.15.28:0/3439689794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37db2570 0x555c37dab420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.484+0000 7fc382ffd640 1 -- 172.21.15.28:0/3439689794 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c37aea7d0 con 0x555c37da7ea0 2024-04-25T01:39:11.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.484+0000 7fc382ffd640 1 --2- 172.21.15.28:0/3439689794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37da7ea0 0x555c37dba730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc378009230 tx=0x7fc3780043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76298278e5c5636d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.484+0000 7fc366ff5640 10 client.0 ms_handle_connect on v2:172.21.15.130:3300/0 2024-04-25T01:39:11.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.484+0000 7fc366ff5640 1 -- 172.21.15.28:0/3439689794 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc378035040 con 0x555c37da7ea0 2024-04-25T01:39:11.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.484+0000 7fc38c189740 1 -- 172.21.15.28:0/3439689794 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc374017040 con 0x555c37da7ea0 2024-04-25T01:39:11.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.484+0000 7fc366ff5640 1 -- 172.21.15.28:0/3439689794 <== mon.1 v2:172.21.15.130:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc378005040 con 0x555c37da7ea0 2024-04-25T01:39:11.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.484+0000 7fc366ff5640 1 -- 172.21.15.28:0/3439689794 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc378056050 con 0x555c37da7ea0 2024-04-25T01:39:11.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.484+0000 7fc38c189740 10 client.26629 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:11.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.484+0000 7fc38c189740 1 -- 172.21.15.28:0/3439689794 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555c37e27bd0 con 0x555c37da7ea0 2024-04-25T01:39:11.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.484+0000 7fc366ff5640 1 -- 172.21.15.28:0/3439689794 <== mon.1 v2:172.21.15.130:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc374017040 con 0x555c37da7ea0 2024-04-25T01:39:11.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc366ff5640 1 -- 172.21.15.28:0/3439689794 <== mon.1 v2:172.21.15.130:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc3780046c0 con 0x555c37da7ea0 2024-04-25T01:39:11.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc38c189740 10 client.26629 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:11.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc38c189740 10 client.26629 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:39:11.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc38c189740 20 client.26629 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:11.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc38c189740 1 --2- 172.21.15.28:0/3439689794 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x555c37e28300 0x555c37e2a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc38c189740 4 client.26629 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:11.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc38c189740 1 -- 172.21.15.28:0/3439689794 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555c37d13f00 con 0x555c37e28300 2024-04-25T01:39:11.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc3827fc640 1 --2- 172.21.15.28:0/3439689794 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x555c37e28300 0x555c37e2a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc366ff5640 10 client.26629 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:39:11.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc3827fc640 1 --2- 172.21.15.28:0/3439689794 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x555c37e28300 0x555c37e2a720 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=295a99b3d0d7f61a server_cookie=18fea4c307ebc4a8 in_seq=0 out_seq=0 2024-04-25T01:39:11.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc366ff5640 1 -- 172.21.15.28:0/3439689794 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555c37d13f00 con 0x555c37e28300 2024-04-25T01:39:11.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc366ff5640 10 client.26629 handle_command_reply: tid=0 2024-04-25T01:39:11.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc38c189740 1 client.26629 shutdown 2024-04-25T01:39:11.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc38c189740 1 -- 172.21.15.28:0/3439689794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37da7ea0 msgr2=0x555c37dba730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc38c189740 1 --2- 172.21.15.28:0/3439689794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37da7ea0 0x555c37dba730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc378009230 tx=0x7fc3780043b0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc38c189740 1 -- 172.21.15.28:0/3439689794 shutdown_connections 2024-04-25T01:39:11.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc38c189740 1 --2- 172.21.15.28:0/3439689794 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x555c37e28300 0x555c37e2a720 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc38c189740 1 --2- 172.21.15.28:0/3439689794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c37db2570 0x555c37dab420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc38c189740 1 --2- 172.21.15.28:0/3439689794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555c37da7ea0 0x555c37dba730 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc38c189740 1 --2- 172.21.15.28:0/3439689794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555c37da7580 0x555c37dace90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc366ff5640 0 client.26629 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:39:11.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc38c189740 1 -- 172.21.15.28:0/3439689794 >> 172.21.15.28:0/3439689794 conn(0x555c37da38a0 msgr2=0x555c37da95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.485+0000 7fc38c189740 1 -- 172.21.15.28:0/3439689794 shutdown_connections 2024-04-25T01:39:11.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.486+0000 7fc38c189740 1 -- 172.21.15.28:0/3439689794 wait complete. 2024-04-25T01:39:11.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.486+0000 7fc38c189740 20 client.26629 trim_cache size 0 max 16384 2024-04-25T01:39:11.478 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.486+0000 7fc38939e640 1 -- 172.21.15.28:0/81740372 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc36c0a2d00 msgr2=0x7fc36c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.486+0000 7fc38939e640 1 --2- 172.21.15.28:0/81740372 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc36c0a2d00 0x7fc36c0a51c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc37c09bc20 tx=0x7fc37c06dee0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.486+0000 7fc38939e640 1 -- 172.21.15.28:0/81740372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3841066b0 msgr2=0x7fc38419c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.486+0000 7fc38939e640 1 --2- 172.21.15.28:0/81740372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3841066b0 0x7fc38419c8a0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fc37c00eb10 tx=0x7fc37c00c2d0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.486+0000 7fc38939e640 1 -- 172.21.15.28:0/81740372 shutdown_connections 2024-04-25T01:39:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.486+0000 7fc38939e640 1 --2- 172.21.15.28:0/81740372 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc36c0a2d00 0x7fc36c0a51c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.486+0000 7fc38939e640 1 --2- 172.21.15.28:0/81740372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3841066b0 0x7fc38419c8a0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.486+0000 7fc38939e640 1 --2- 172.21.15.28:0/81740372 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc384100fd0 0x7fc384198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.486+0000 7fc38939e640 1 --2- 172.21.15.28:0/81740372 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc384100610 0x7fc384197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.486+0000 7fc38939e640 1 -- 172.21.15.28:0/81740372 >> 172.21.15.28:0/81740372 conn(0x7fc384077740 msgr2=0x7fc3840ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.486+0000 7fc38939e640 1 -- 172.21.15.28:0/81740372 shutdown_connections 2024-04-25T01:39:11.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.486+0000 7fc38939e640 1 -- 172.21.15.28:0/81740372 wait complete. 2024-04-25T01:39:11.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:39:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.603+0000 7f11cf7c8640 1 -- 172.21.15.28:0/1721809657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c8106630 msgr2=0x7f11c8106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.603+0000 7f11cf7c8640 1 --2- 172.21.15.28:0/1721809657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c8106630 0x7f11c8106a10 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f11c4009930 tx=0x7f11c402a420 comp rx=0 tx=0).stop 2024-04-25T01:39:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.603+0000 7f11cf7c8640 1 -- 172.21.15.28:0/1721809657 shutdown_connections 2024-04-25T01:39:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.603+0000 7f11cf7c8640 1 --2- 172.21.15.28:0/1721809657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11c8100fe0 0x7f11c810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.603+0000 7f11cf7c8640 1 --2- 172.21.15.28:0/1721809657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11c8100620 0x7f11c8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.603+0000 7f11cf7c8640 1 --2- 172.21.15.28:0/1721809657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c8106630 0x7f11c8106a10 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.603+0000 7f11cf7c8640 1 -- 172.21.15.28:0/1721809657 >> 172.21.15.28:0/1721809657 conn(0x7f11c80776a0 msgr2=0x7f11c8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.604+0000 7f11cf7c8640 1 -- 172.21.15.28:0/1721809657 shutdown_connections 2024-04-25T01:39:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.604+0000 7f11cf7c8640 1 -- 172.21.15.28:0/1721809657 wait complete. 2024-04-25T01:39:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.604+0000 7f11cf7c8640 1 Processor -- start 2024-04-25T01:39:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.604+0000 7f11cf7c8640 1 -- start start 2024-04-25T01:39:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.604+0000 7f11cf7c8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11c8100620 0x7f11c8196d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.604+0000 7f11cf7c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c8100fe0 0x7f11c8197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.604+0000 7f11cf7c8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11c8106630 0x7f11c81977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.604+0000 7f11cf7c8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11c8110a50 con 0x7f11c8100fe0 2024-04-25T01:39:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.604+0000 7f11cf7c8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f11c81108d0 con 0x7f11c8100620 2024-04-25T01:39:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.604+0000 7f11cf7c8640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f11c8110bd0 con 0x7f11c8106630 2024-04-25T01:39:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.605+0000 7f11ccd63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c8100fe0 0x7f11c8197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.605+0000 7f11ccd63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c8100fe0 0x7f11c8197280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48466/0 (socket says 172.21.15.28:48466) 2024-04-25T01:39:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.605+0000 7f11ccd63640 1 -- 172.21.15.28:0/2830332108 learned_addr learned my addr 172.21.15.28:0/2830332108 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.605+0000 7f11cd564640 1 --2- 172.21.15.28:0/2830332108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11c8100620 0x7f11c8196d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.605+0000 7f11cdd65640 1 --2- 172.21.15.28:0/2830332108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11c8106630 0x7f11c81977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.605+0000 7f11ccd63640 1 -- 172.21.15.28:0/2830332108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11c8106630 msgr2=0x7f11c81977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.605+0000 7f11ccd63640 1 --2- 172.21.15.28:0/2830332108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11c8106630 0x7f11c81977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.605+0000 7f11ccd63640 1 -- 172.21.15.28:0/2830332108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11c8100620 msgr2=0x7f11c8196d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.605+0000 7f11ccd63640 1 --2- 172.21.15.28:0/2830332108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11c8100620 0x7f11c8196d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.605+0000 7f11ccd63640 1 -- 172.21.15.28:0/2830332108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11c81a65b0 con 0x7f11c8100fe0 2024-04-25T01:39:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.605+0000 7f11ccd63640 1 --2- 172.21.15.28:0/2830332108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c8100fe0 0x7f11c8197280 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f11b800d950 tx=0x7f11b800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.605+0000 7f11be7fc640 1 -- 172.21.15.28:0/2830332108 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b8017070 con 0x7f11c8100fe0 2024-04-25T01:39:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.605+0000 7f11be7fc640 1 -- 172.21.15.28:0/2830332108 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11b8009c30 con 0x7f11c8100fe0 2024-04-25T01:39:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.605+0000 7f11cf7c8640 1 -- 172.21.15.28:0/2830332108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11c81a68a0 con 0x7f11c8100fe0 2024-04-25T01:39:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.605+0000 7f11be7fc640 1 -- 172.21.15.28:0/2830332108 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b800f6b0 con 0x7f11c8100fe0 2024-04-25T01:39:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.605+0000 7f11cf7c8640 1 -- 172.21.15.28:0/2830332108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11c81a6e40 con 0x7f11c8100fe0 2024-04-25T01:39:11.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.607+0000 7f11be7fc640 1 -- 172.21.15.28:0/2830332108 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f11b80109d0 con 0x7f11c8100fe0 2024-04-25T01:39:11.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.608+0000 7f11d25b4740 1 Processor -- start 2024-04-25T01:39:11.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.608+0000 7f11d25b4740 1 -- start start 2024-04-25T01:39:11.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.608+0000 7f11d25b4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae2022a580 0x55ae2022a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.608+0000 7f11d25b4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022aea0 0x55ae20235030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.608+0000 7f11d25b4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae20235570 0x55ae20237960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.608+0000 7f11d25b4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ae2018f670 con 0x55ae2022a580 2024-04-25T01:39:11.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.608+0000 7f11d25b4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ae20137970 con 0x55ae2022aea0 2024-04-25T01:39:11.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.608+0000 7f11d25b4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ae2018b630 con 0x55ae20235570 2024-04-25T01:39:11.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.608+0000 7f11be7fc640 1 --2- 172.21.15.28:0/2830332108 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f11b40a2d80 0x7f11b40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.608+0000 7f11cdd65640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae2022a580 0x55ae2022a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.608+0000 7f11cdd65640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae2022a580 0x55ae2022a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48482/0 (socket says 172.21.15.28:48482) 2024-04-25T01:39:11.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.608+0000 7f11cdd65640 1 -- 172.21.15.28:0/1452119857 learned_addr learned my addr 172.21.15.28:0/1452119857 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.608+0000 7f11ccd63640 1 --2- 172.21.15.28:0/1452119857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae20235570 0x55ae20237960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.608+0000 7f11cdd65640 1 --2- 172.21.15.28:0/2830332108 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f11b40a2d80 0x7f11b40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.608+0000 7f11be7fc640 1 -- 172.21.15.28:0/2830332108 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f11b80cf600 con 0x7f11c8100fe0 2024-04-25T01:39:11.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.608+0000 7f11cd564640 1 --2- 172.21.15.28:0/1452119857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022aea0 0x55ae20235030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.609+0000 7f11cdd65640 1 -- 172.21.15.28:0/1452119857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae20235570 msgr2=0x55ae20237960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.609+0000 7f11cdd65640 1 --2- 172.21.15.28:0/1452119857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae20235570 0x55ae20237960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.609+0000 7f11cdd65640 1 -- 172.21.15.28:0/1452119857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022aea0 msgr2=0x55ae20235030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.609+0000 7f11cdd65640 1 --2- 172.21.15.28:0/1452119857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022aea0 0x55ae20235030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.609+0000 7f11cdd65640 1 -- 172.21.15.28:0/1452119857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae2015b400 con 0x55ae2022a580 2024-04-25T01:39:11.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.609+0000 7f11ccd63640 1 --2- 172.21.15.28:0/1452119857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae20235570 0x55ae20237960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:39:11.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.609+0000 7f11cdd65640 1 --2- 172.21.15.28:0/1452119857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae2022a580 0x55ae2022a960 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f11c00099e0 tx=0x7f11c000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84531170c21af71 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.609+0000 7f11cdd65640 1 --2- 172.21.15.28:0/2830332108 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f11b40a2d80 0x7f11b40a5240 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f11c000fc90 tx=0x7f11c001e000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.609+0000 7f11b37fe640 1 -- 172.21.15.28:0/1452119857 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c001dbf0 con 0x55ae2022a580 2024-04-25T01:39:11.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.609+0000 7f11b37fe640 1 -- 172.21.15.28:0/1452119857 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11c0009d80 con 0x55ae2022a580 2024-04-25T01:39:11.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.609+0000 7f11b37fe640 1 -- 172.21.15.28:0/1452119857 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c00042b0 con 0x55ae2022a580 2024-04-25T01:39:11.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.610+0000 7f11d25b4740 1 -- 172.21.15.28:0/1452119857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae2022a580 msgr2=0x55ae2022a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.610+0000 7f11d25b4740 1 --2- 172.21.15.28:0/1452119857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae2022a580 0x55ae2022a960 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f11c00099e0 tx=0x7f11c000f790 comp rx=0 tx=0).stop 2024-04-25T01:39:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.610+0000 7f11d25b4740 1 -- 172.21.15.28:0/1452119857 shutdown_connections 2024-04-25T01:39:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.610+0000 7f11d25b4740 1 --2- 172.21.15.28:0/1452119857 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae20235570 0x55ae20237960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.610+0000 7f11d25b4740 1 --2- 172.21.15.28:0/1452119857 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022aea0 0x55ae20235030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.610+0000 7f11d25b4740 1 --2- 172.21.15.28:0/1452119857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae2022a580 0x55ae2022a960 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.610+0000 7f11d25b4740 1 -- 172.21.15.28:0/1452119857 >> 172.21.15.28:0/1452119857 conn(0x55ae202268a0 msgr2=0x55ae20226c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.610+0000 7f11d25b4740 1 -- 172.21.15.28:0/1452119857 shutdown_connections 2024-04-25T01:39:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.610+0000 7f11d25b4740 1 -- 172.21.15.28:0/1452119857 wait complete. 2024-04-25T01:39:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.610+0000 7f11d25b4740 1 -- start start 2024-04-25T01:39:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.610+0000 7f11d25b4740 1 Processor -- start 2024-04-25T01:39:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.610+0000 7f11d25b4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022a580 0x55ae20234f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11d25b4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae2022aea0 0x55ae20232220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11d25b4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae20235570 0x55ae20232830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11d25b4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ae20238300 con 0x55ae2022aea0 2024-04-25T01:39:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11d25b4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ae2014b050 con 0x55ae2022a580 2024-04-25T01:39:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11d25b4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ae20238480 con 0x55ae20235570 2024-04-25T01:39:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11ccd63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae2022aea0 0x55ae20232220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11d25b4740 10 client.0 _ll_register_callbacks cb 0x55ae1fea2be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11ccd63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae2022aea0 0x55ae20232220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48486/0 (socket says 172.21.15.28:48486) 2024-04-25T01:39:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11ccd63640 1 -- 172.21.15.28:0/1891200049 learned_addr learned my addr 172.21.15.28:0/1891200049 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11cd564640 1 --2- 172.21.15.28:0/1891200049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022a580 0x55ae20234f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11cdd65640 1 --2- 172.21.15.28:0/1891200049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae20235570 0x55ae20232830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11ccd63640 1 -- 172.21.15.28:0/1891200049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae20235570 msgr2=0x55ae20232830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11ccd63640 1 --2- 172.21.15.28:0/1891200049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae20235570 0x55ae20232830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11ccd63640 1 -- 172.21.15.28:0/1891200049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022a580 msgr2=0x55ae20234f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11ccd63640 1 --2- 172.21.15.28:0/1891200049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022a580 0x55ae20234f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11ccd63640 1 -- 172.21.15.28:0/1891200049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae2011cdc0 con 0x55ae2022aea0 2024-04-25T01:39:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11cdd65640 1 --2- 172.21.15.28:0/1891200049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae20235570 0x55ae20232830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11ccd63640 1 --2- 172.21.15.28:0/1891200049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae2022aea0 0x55ae20232220 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f11b8097cf0 tx=0x7f11b80a0de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b40c10a63838d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11b0ff9640 1 -- 172.21.15.28:0/1891200049 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b80b4060 con 0x55ae2022aea0 2024-04-25T01:39:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11b0ff9640 1 -- 172.21.15.28:0/1891200049 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f11b8097f00 con 0x55ae2022aea0 2024-04-25T01:39:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11d25b4740 1 -- 172.21.15.28:0/1891200049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11c0009d80 con 0x55ae2022aea0 2024-04-25T01:39:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11b0ff9640 1 -- 172.21.15.28:0/1891200049 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f11b8020050 con 0x55ae2022aea0 2024-04-25T01:39:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11d25b4740 10 client.17631 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.611+0000 7f11d25b4740 1 -- 172.21.15.28:0/1891200049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ae1fea33f0 con 0x55ae2022aea0 2024-04-25T01:39:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.612+0000 7f11b0ff9640 1 -- 172.21.15.28:0/1891200049 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11c0009d80 con 0x55ae2022aea0 2024-04-25T01:39:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.612+0000 7f11b0ff9640 1 -- 172.21.15.28:0/1891200049 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f11b80860b0 con 0x55ae2022aea0 2024-04-25T01:39:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.612+0000 7f11d25b4740 10 client.17631 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.612+0000 7f11d25b4740 10 client.17631 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:39:11.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.612+0000 7f11d25b4740 20 client.17631 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.612+0000 7f11d25b4740 1 --2- 172.21.15.28:0/1891200049 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ae20245510 0x55ae20247940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.612+0000 7f11d25b4740 4 client.17631 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.612+0000 7f11d25b4740 1 -- 172.21.15.28:0/1891200049 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae2019cbb0 con 0x55ae20245510 2024-04-25T01:39:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.612+0000 7f11cd564640 1 --2- 172.21.15.28:0/1891200049 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ae20245510 0x55ae20247940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.612+0000 7f11b0ff9640 10 client.17631 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:39:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.612+0000 7f11cd564640 1 --2- 172.21.15.28:0/1891200049 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ae20245510 0x55ae20247940 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=87045d08c2e093fe server_cookie=ec580e7fe0e37591 in_seq=0 out_seq=0 2024-04-25T01:39:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.613+0000 7f11b0ff9640 1 -- 172.21.15.28:0/1891200049 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ae2019cbb0 con 0x55ae20245510 2024-04-25T01:39:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.613+0000 7f11b0ff9640 10 client.17631 handle_command_reply: tid=0 2024-04-25T01:39:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.613+0000 7f11d25b4740 1 client.17631 shutdown 2024-04-25T01:39:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.613+0000 7f11d25b4740 1 -- 172.21.15.28:0/1891200049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae2022aea0 msgr2=0x55ae20232220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.613+0000 7f11d25b4740 1 --2- 172.21.15.28:0/1891200049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae2022aea0 0x55ae20232220 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f11b8097cf0 tx=0x7f11b80a0de0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.613+0000 7f11d25b4740 1 -- 172.21.15.28:0/1891200049 shutdown_connections 2024-04-25T01:39:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.613+0000 7f11d25b4740 1 --2- 172.21.15.28:0/1891200049 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ae20245510 0x55ae20247940 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.613+0000 7f11d25b4740 1 --2- 172.21.15.28:0/1891200049 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae20235570 0x55ae20232830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.613+0000 7f11b0ff9640 0 client.17631 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:39:11.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.613+0000 7f11d25b4740 1 --2- 172.21.15.28:0/1891200049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae2022aea0 0x55ae20232220 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.613+0000 7f11d25b4740 1 --2- 172.21.15.28:0/1891200049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022a580 0x55ae20234f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.613+0000 7f11d25b4740 1 -- 172.21.15.28:0/1891200049 >> 172.21.15.28:0/1891200049 conn(0x55ae202268a0 msgr2=0x55ae20225b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.613+0000 7f11d25b4740 1 -- 172.21.15.28:0/1891200049 shutdown_connections 2024-04-25T01:39:11.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.613+0000 7f11d25b4740 1 -- 172.21.15.28:0/1891200049 wait complete. 2024-04-25T01:39:11.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.614+0000 7f11d25b4740 20 client.17631 trim_cache size 0 max 16384 2024-04-25T01:39:11.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.627+0000 7f11d25b4740 1 Processor -- start 2024-04-25T01:39:11.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.627+0000 7f11d25b4740 1 -- start start 2024-04-25T01:39:11.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.627+0000 7f11d25b4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae2022a580 0x55ae202411c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.627+0000 7f11d25b4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022aea0 0x55ae20231250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.627+0000 7f11d25b4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae20235570 0x55ae20234a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.627+0000 7f11d25b4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11c81b0c40 con 0x55ae20235570 2024-04-25T01:39:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.627+0000 7f11d25b4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f11c81b2780 con 0x55ae2022aea0 2024-04-25T01:39:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.627+0000 7f11d25b4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f11c81b1310 con 0x55ae2022a580 2024-04-25T01:39:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.627+0000 7f11cdd65640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae20235570 0x55ae20234a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.627+0000 7f11cdd65640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae20235570 0x55ae20234a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48492/0 (socket says 172.21.15.28:48492) 2024-04-25T01:39:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.627+0000 7f11cdd65640 1 -- 172.21.15.28:0/2202948422 learned_addr learned my addr 172.21.15.28:0/2202948422 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.627+0000 7f11cd564640 1 --2- 172.21.15.28:0/2202948422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae2022a580 0x55ae202411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.627+0000 7f11cdd65640 1 -- 172.21.15.28:0/2202948422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae2022a580 msgr2=0x55ae202411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.627+0000 7f11cdd65640 1 --2- 172.21.15.28:0/2202948422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae2022a580 0x55ae202411c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.627+0000 7f11ccd63640 1 --2- 172.21.15.28:0/2202948422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022aea0 0x55ae20231250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.627+0000 7f11cdd65640 1 -- 172.21.15.28:0/2202948422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022aea0 msgr2=0x55ae20231250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.627+0000 7f11cdd65640 1 --2- 172.21.15.28:0/2202948422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022aea0 0x55ae20231250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.627+0000 7f11cdd65640 1 -- 172.21.15.28:0/2202948422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae20232ba0 con 0x55ae20235570 2024-04-25T01:39:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.627+0000 7f11ccd63640 1 --2- 172.21.15.28:0/2202948422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022aea0 0x55ae20231250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.628+0000 7f11cdd65640 1 --2- 172.21.15.28:0/2202948422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae20235570 0x55ae20234a60 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f11c0009800 tx=0x7f11c0011c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=615980ae70b9e865 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.628+0000 7f11b2ffd640 1 -- 172.21.15.28:0/2202948422 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c0011950 con 0x55ae20235570 2024-04-25T01:39:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.628+0000 7f11b2ffd640 1 -- 172.21.15.28:0/2202948422 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11c0003e50 con 0x55ae20235570 2024-04-25T01:39:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.628+0000 7f11b2ffd640 1 -- 172.21.15.28:0/2202948422 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c0005a10 con 0x55ae20235570 2024-04-25T01:39:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.628+0000 7f11d25b4740 1 -- 172.21.15.28:0/2202948422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae20235570 msgr2=0x55ae20234a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.628+0000 7f11d25b4740 1 --2- 172.21.15.28:0/2202948422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae20235570 0x55ae20234a60 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f11c0009800 tx=0x7f11c0011c30 comp rx=0 tx=0).stop 2024-04-25T01:39:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.628+0000 7f11d25b4740 1 -- 172.21.15.28:0/2202948422 shutdown_connections 2024-04-25T01:39:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.628+0000 7f11d25b4740 1 --2- 172.21.15.28:0/2202948422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae20235570 0x55ae20234a60 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.628+0000 7f11d25b4740 1 --2- 172.21.15.28:0/2202948422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022aea0 0x55ae20231250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.628+0000 7f11d25b4740 1 --2- 172.21.15.28:0/2202948422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae2022a580 0x55ae202411c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.628+0000 7f11d25b4740 1 -- 172.21.15.28:0/2202948422 >> 172.21.15.28:0/2202948422 conn(0x55ae202268a0 msgr2=0x55ae20226c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.628+0000 7f11d25b4740 1 -- 172.21.15.28:0/2202948422 shutdown_connections 2024-04-25T01:39:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.628+0000 7f11d25b4740 1 -- 172.21.15.28:0/2202948422 wait complete. 2024-04-25T01:39:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.629+0000 7f11d25b4740 1 -- start start 2024-04-25T01:39:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.629+0000 7f11d25b4740 1 Processor -- start 2024-04-25T01:39:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.629+0000 7f11d25b4740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae2022a580 0x55ae2023d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.629+0000 7f11d25b4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022aea0 0x55ae20238d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.629+0000 7f11d25b4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae20235570 0x55ae20239270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.629+0000 7f11d25b4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11c81b20b0 con 0x55ae20235570 2024-04-25T01:39:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.629+0000 7f11d25b4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f11c81b19e0 con 0x55ae2022aea0 2024-04-25T01:39:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.629+0000 7f11d25b4740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f11c81aec20 con 0x55ae2022a580 2024-04-25T01:39:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.629+0000 7f11d25b4740 10 client.0 _ll_register_callbacks cb 0x7f11b809b1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.629+0000 7f11cdd65640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae20235570 0x55ae20239270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.629+0000 7f11cdd65640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae20235570 0x55ae20239270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48502/0 (socket says 172.21.15.28:48502) 2024-04-25T01:39:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.629+0000 7f11cdd65640 1 -- 172.21.15.28:0/3496740771 learned_addr learned my addr 172.21.15.28:0/3496740771 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.629+0000 7f11cd564640 1 --2- 172.21.15.28:0/3496740771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae2022a580 0x55ae2023d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.629+0000 7f11ccd63640 1 --2- 172.21.15.28:0/3496740771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022aea0 0x55ae20238d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11cdd65640 1 -- 172.21.15.28:0/3496740771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae2022a580 msgr2=0x55ae2023d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11cdd65640 1 --2- 172.21.15.28:0/3496740771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae2022a580 0x55ae2023d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11cdd65640 1 -- 172.21.15.28:0/3496740771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022aea0 msgr2=0x55ae20238d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11cdd65640 1 --2- 172.21.15.28:0/3496740771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022aea0 0x55ae20238d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11cdd65640 1 -- 172.21.15.28:0/3496740771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae1ff6d7d0 con 0x55ae20235570 2024-04-25T01:39:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11cd564640 1 --2- 172.21.15.28:0/3496740771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae2022a580 0x55ae2023d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:39:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11cdd65640 1 --2- 172.21.15.28:0/3496740771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae20235570 0x55ae20239270 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f11c0005630 tx=0x7f11c0011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf6a19c340049b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11b0ff9640 1 -- 172.21.15.28:0/3496740771 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c003e060 con 0x55ae20235570 2024-04-25T01:39:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11b0ff9640 1 -- 172.21.15.28:0/3496740771 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f11c00050c0 con 0x55ae20235570 2024-04-25T01:39:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11d25b4740 1 -- 172.21.15.28:0/3496740771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11c0003e50 con 0x55ae20235570 2024-04-25T01:39:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11b0ff9640 1 -- 172.21.15.28:0/3496740771 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f11c0045050 con 0x55ae20235570 2024-04-25T01:39:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11d25b4740 10 client.17637 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:11.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11d25b4740 1 -- 172.21.15.28:0/3496740771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ae202ab6b0 con 0x55ae20235570 2024-04-25T01:39:11.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11b0ff9640 1 -- 172.21.15.28:0/3496740771 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11c0003e50 con 0x55ae20235570 2024-04-25T01:39:11.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11b0ff9640 1 -- 172.21.15.28:0/3496740771 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f11c002f480 con 0x55ae20235570 2024-04-25T01:39:11.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11d25b4740 10 client.17637 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:11.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11d25b4740 10 client.17637 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:39:11.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11d25b4740 20 client.17637 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:11.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11d25b4740 1 --2- 172.21.15.28:0/3496740771 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ae202abc00 0x55ae202ae020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11d25b4740 4 client.17637 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:11.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.630+0000 7f11d25b4740 1 -- 172.21.15.28:0/3496740771 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ae20196f00 con 0x55ae202abc00 2024-04-25T01:39:11.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.631+0000 7f11cd564640 1 --2- 172.21.15.28:0/3496740771 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ae202abc00 0x55ae202ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.631+0000 7f11b0ff9640 10 client.17637 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:39:11.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.631+0000 7f11cd564640 1 --2- 172.21.15.28:0/3496740771 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ae202abc00 0x55ae202ae020 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1a7b8459dc6eacd5 server_cookie=6469b1179c2de689 in_seq=0 out_seq=0 2024-04-25T01:39:11.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.632+0000 7f11b0ff9640 1 -- 172.21.15.28:0/3496740771 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ae20196f00 con 0x55ae202abc00 2024-04-25T01:39:11.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.632+0000 7f11b0ff9640 10 client.17637 handle_command_reply: tid=0 2024-04-25T01:39:11.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.632+0000 7f11d25b4740 1 client.17637 shutdown 2024-04-25T01:39:11.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.632+0000 7f11d25b4740 1 -- 172.21.15.28:0/3496740771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae20235570 msgr2=0x55ae20239270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.632+0000 7f11d25b4740 1 --2- 172.21.15.28:0/3496740771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae20235570 0x55ae20239270 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f11c0005630 tx=0x7f11c0011440 comp rx=0 tx=0).stop 2024-04-25T01:39:11.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.632+0000 7f11d25b4740 1 -- 172.21.15.28:0/3496740771 shutdown_connections 2024-04-25T01:39:11.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.632+0000 7f11d25b4740 1 --2- 172.21.15.28:0/3496740771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae20235570 0x55ae20239270 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.632+0000 7f11d25b4740 1 --2- 172.21.15.28:0/3496740771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae2022aea0 0x55ae20238d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.632+0000 7f11d25b4740 1 --2- 172.21.15.28:0/3496740771 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55ae202abc00 0x55ae202ae020 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.632+0000 7f11d25b4740 1 --2- 172.21.15.28:0/3496740771 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ae2022a580 0x55ae2023d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.632+0000 7f11b0ff9640 0 client.17637 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:39:11.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.632+0000 7f11d25b4740 1 -- 172.21.15.28:0/3496740771 >> 172.21.15.28:0/3496740771 conn(0x55ae202268a0 msgr2=0x55ae2022d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.632+0000 7f11d25b4740 1 -- 172.21.15.28:0/3496740771 shutdown_connections 2024-04-25T01:39:11.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.632+0000 7f11d25b4740 1 -- 172.21.15.28:0/3496740771 wait complete. 2024-04-25T01:39:11.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.632+0000 7f11d25b4740 20 client.17637 trim_cache size 0 max 16384 2024-04-25T01:39:11.621 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:11.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.632+0000 7f11cf7c8640 1 -- 172.21.15.28:0/2830332108 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f11b40a2d80 msgr2=0x7f11b40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.632+0000 7f11cf7c8640 1 --2- 172.21.15.28:0/2830332108 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f11b40a2d80 0x7f11b40a5240 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f11c000fc90 tx=0x7f11c001e000 comp rx=0 tx=0).stop 2024-04-25T01:39:11.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.632+0000 7f11cf7c8640 1 -- 172.21.15.28:0/2830332108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c8100fe0 msgr2=0x7f11c8197280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.632+0000 7f11cf7c8640 1 --2- 172.21.15.28:0/2830332108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c8100fe0 0x7f11c8197280 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f11b800d950 tx=0x7f11b800de10 comp rx=0 tx=0).stop 2024-04-25T01:39:11.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.633+0000 7f11cf7c8640 1 -- 172.21.15.28:0/2830332108 shutdown_connections 2024-04-25T01:39:11.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.633+0000 7f11cf7c8640 1 --2- 172.21.15.28:0/2830332108 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f11b40a2d80 0x7f11b40a5240 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.633+0000 7f11cf7c8640 1 --2- 172.21.15.28:0/2830332108 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11c8106630 0x7f11c81977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.633+0000 7f11cf7c8640 1 --2- 172.21.15.28:0/2830332108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c8100fe0 0x7f11c8197280 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.633+0000 7f11cf7c8640 1 --2- 172.21.15.28:0/2830332108 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11c8100620 0x7f11c8196d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.633+0000 7f11cf7c8640 1 -- 172.21.15.28:0/2830332108 >> 172.21.15.28:0/2830332108 conn(0x7f11c80776a0 msgr2=0x7f11c8103290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.633+0000 7f11cf7c8640 1 -- 172.21.15.28:0/2830332108 shutdown_connections 2024-04-25T01:39:11.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.633+0000 7f11cf7c8640 1 -- 172.21.15.28:0/2830332108 wait complete. 2024-04-25T01:39:11.625 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:39:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.748+0000 7f887aaf5640 1 -- 172.21.15.28:0/1747355594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8874100fd0 msgr2=0x7f887410d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.748+0000 7f887aaf5640 1 --2- 172.21.15.28:0/1747355594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8874100fd0 0x7f887410d4e0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f886c009cd0 tx=0x7f886c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.748+0000 7f887aaf5640 1 -- 172.21.15.28:0/1747355594 shutdown_connections 2024-04-25T01:39:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.748+0000 7f887aaf5640 1 --2- 172.21.15.28:0/1747355594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8874100fd0 0x7f887410d4e0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.748+0000 7f887aaf5640 1 --2- 172.21.15.28:0/1747355594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8874100610 0x7f8874100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.748+0000 7f887aaf5640 1 --2- 172.21.15.28:0/1747355594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88741066b0 0x7f8874106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.748+0000 7f887aaf5640 1 -- 172.21.15.28:0/1747355594 >> 172.21.15.28:0/1747355594 conn(0x7f8874077740 msgr2=0x7f8874077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.748+0000 7f887aaf5640 1 -- 172.21.15.28:0/1747355594 shutdown_connections 2024-04-25T01:39:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.749+0000 7f887aaf5640 1 -- 172.21.15.28:0/1747355594 wait complete. 2024-04-25T01:39:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.749+0000 7f887aaf5640 1 Processor -- start 2024-04-25T01:39:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.749+0000 7f887aaf5640 1 -- start start 2024-04-25T01:39:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.749+0000 7f887aaf5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8874100610 0x7f887410df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.749+0000 7f887aaf5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8874100fd0 0x7f887410e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.749+0000 7f887aaf5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88741066b0 0x7f887410e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.749+0000 7f887aaf5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8874111b80 con 0x7f8874100610 2024-04-25T01:39:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.749+0000 7f887aaf5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8874111a00 con 0x7f88741066b0 2024-04-25T01:39:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.749+0000 7f887aaf5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8874111d00 con 0x7f8874100fd0 2024-04-25T01:39:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.749+0000 7f8878891640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8874100610 0x7f887410df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.749+0000 7f8878891640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8874100610 0x7f887410df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48514/0 (socket says 172.21.15.28:48514) 2024-04-25T01:39:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.749+0000 7f8878891640 1 -- 172.21.15.28:0/2849383756 learned_addr learned my addr 172.21.15.28:0/2849383756 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.750+0000 7f8879092640 1 --2- 172.21.15.28:0/2849383756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88741066b0 0x7f887410e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.750+0000 7f8873fff640 1 --2- 172.21.15.28:0/2849383756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8874100fd0 0x7f887410e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.750+0000 7f8878891640 1 -- 172.21.15.28:0/2849383756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8874100fd0 msgr2=0x7f887410e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.750+0000 7f8878891640 1 --2- 172.21.15.28:0/2849383756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8874100fd0 0x7f887410e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.750+0000 7f8878891640 1 -- 172.21.15.28:0/2849383756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88741066b0 msgr2=0x7f887410e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.750+0000 7f8878891640 1 --2- 172.21.15.28:0/2849383756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88741066b0 0x7f887410e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.750+0000 7f8878891640 1 -- 172.21.15.28:0/2849383756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88741a6460 con 0x7f8874100610 2024-04-25T01:39:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.750+0000 7f8878891640 1 --2- 172.21.15.28:0/2849383756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8874100610 0x7f887410df70 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f886400e9c0 tx=0x7f886400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.750+0000 7f8871ffb640 1 -- 172.21.15.28:0/2849383756 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886400cde0 con 0x7f8874100610 2024-04-25T01:39:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.750+0000 7f8871ffb640 1 -- 172.21.15.28:0/2849383756 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8864010910 con 0x7f8874100610 2024-04-25T01:39:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.750+0000 7f887aaf5640 1 -- 172.21.15.28:0/2849383756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88741a6750 con 0x7f8874100610 2024-04-25T01:39:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.750+0000 7f8871ffb640 1 -- 172.21.15.28:0/2849383756 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8864018a00 con 0x7f8874100610 2024-04-25T01:39:11.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.750+0000 7f887aaf5640 1 -- 172.21.15.28:0/2849383756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88741a6ca0 con 0x7f8874100610 2024-04-25T01:39:11.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.753+0000 7f8871ffb640 1 -- 172.21.15.28:0/2849383756 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f886401f020 con 0x7f8874100610 2024-04-25T01:39:11.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.753+0000 7f887d804740 1 Processor -- start 2024-04-25T01:39:11.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.753+0000 7f887d804740 1 -- start start 2024-04-25T01:39:11.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.753+0000 7f887d804740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253e580 0x56459253e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.753+0000 7f887d804740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253eea0 0x564592549030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.754+0000 7f887d804740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564592549570 0x56459254b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.754+0000 7f887d804740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5645924a3670 con 0x56459253e580 2024-04-25T01:39:11.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.754+0000 7f887d804740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56459244b970 con 0x56459253eea0 2024-04-25T01:39:11.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.754+0000 7f887d804740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56459249f630 con 0x564592549570 2024-04-25T01:39:11.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.754+0000 7f8879092640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253e580 0x56459253e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.754+0000 7f8879092640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253e580 0x56459253e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48522/0 (socket says 172.21.15.28:48522) 2024-04-25T01:39:11.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.754+0000 7f8879092640 1 -- 172.21.15.28:0/927826733 learned_addr learned my addr 172.21.15.28:0/927826733 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.754+0000 7f8878891640 1 --2- 172.21.15.28:0/927826733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253eea0 0x564592549030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.754+0000 7f8873fff640 1 --2- 172.21.15.28:0/927826733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564592549570 0x56459254b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.754+0000 7f8871ffb640 1 --2- 172.21.15.28:0/2849383756 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f885c0a2d00 0x7f885c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.754+0000 7f8879092640 1 -- 172.21.15.28:0/927826733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564592549570 msgr2=0x56459254b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.754+0000 7f8879092640 1 --2- 172.21.15.28:0/927826733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564592549570 0x56459254b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.754+0000 7f8879092640 1 -- 172.21.15.28:0/927826733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253eea0 msgr2=0x564592549030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.754+0000 7f8879092640 1 --2- 172.21.15.28:0/927826733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253eea0 0x564592549030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.754+0000 7f8879092640 1 -- 172.21.15.28:0/927826733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56459246f400 con 0x56459253e580 2024-04-25T01:39:11.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.754+0000 7f8879092640 1 --2- 172.21.15.28:0/2849383756 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f885c0a2d00 0x7f885c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.754+0000 7f8871ffb640 1 -- 172.21.15.28:0/2849383756 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f88640cf8c0 con 0x7f8874100610 2024-04-25T01:39:11.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.754+0000 7f8879092640 1 --2- 172.21.15.28:0/927826733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253e580 0x56459253e960 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f886c009cd0 tx=0x7f886c031450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3599c0b0ac0d0072 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.755+0000 7f885affd640 1 -- 172.21.15.28:0/927826733 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886c042040 con 0x56459253e580 2024-04-25T01:39:11.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.755+0000 7f885affd640 1 -- 172.21.15.28:0/927826733 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f886c044070 con 0x56459253e580 2024-04-25T01:39:11.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.755+0000 7f885affd640 1 -- 172.21.15.28:0/927826733 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886c030c50 con 0x56459253e580 2024-04-25T01:39:11.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.755+0000 7f887d804740 1 -- 172.21.15.28:0/927826733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253e580 msgr2=0x56459253e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.755+0000 7f887d804740 1 --2- 172.21.15.28:0/927826733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253e580 0x56459253e960 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f886c009cd0 tx=0x7f886c031450 comp rx=0 tx=0).stop 2024-04-25T01:39:11.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.755+0000 7f8879092640 1 --2- 172.21.15.28:0/2849383756 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f885c0a2d00 0x7f885c0a51c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f886c04d000 tx=0x7f886c04e000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.755+0000 7f887d804740 1 -- 172.21.15.28:0/927826733 shutdown_connections 2024-04-25T01:39:11.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.755+0000 7f887d804740 1 --2- 172.21.15.28:0/927826733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564592549570 0x56459254b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.755+0000 7f887d804740 1 --2- 172.21.15.28:0/927826733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253eea0 0x564592549030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.755+0000 7f887d804740 1 --2- 172.21.15.28:0/927826733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253e580 0x56459253e960 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.755+0000 7f887d804740 1 -- 172.21.15.28:0/927826733 >> 172.21.15.28:0/927826733 conn(0x56459253a8a0 msgr2=0x56459253ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.755+0000 7f887d804740 1 -- 172.21.15.28:0/927826733 shutdown_connections 2024-04-25T01:39:11.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.755+0000 7f887d804740 1 -- 172.21.15.28:0/927826733 wait complete. 2024-04-25T01:39:11.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f887d804740 1 -- start start 2024-04-25T01:39:11.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f887d804740 1 Processor -- start 2024-04-25T01:39:11.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f887d804740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253e580 0x564592548ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f887d804740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253eea0 0x5645925461b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f887d804740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564592549570 0x564592546710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f887d804740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56459254c360 con 0x56459253eea0 2024-04-25T01:39:11.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f887d804740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56459245f050 con 0x56459253e580 2024-04-25T01:39:11.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f887d804740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56459254c4e0 con 0x564592549570 2024-04-25T01:39:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f887d804740 10 client.0 _ll_register_callbacks cb 0x5645921b6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f8873fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253eea0 0x5645925461b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f8878891640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253e580 0x564592548ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f8878891640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253e580 0x564592548ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:42164/0 (socket says 172.21.15.28:42164) 2024-04-25T01:39:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f8873fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253eea0 0x5645925461b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48526/0 (socket says 172.21.15.28:48526) 2024-04-25T01:39:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f8878891640 1 -- 172.21.15.28:0/4238224011 learned_addr learned my addr 172.21.15.28:0/4238224011 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f8879092640 1 --2- 172.21.15.28:0/4238224011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564592549570 0x564592546710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f8878891640 1 -- 172.21.15.28:0/4238224011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564592549570 msgr2=0x564592546710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f8878891640 1 --2- 172.21.15.28:0/4238224011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564592549570 0x564592546710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f8878891640 1 -- 172.21.15.28:0/4238224011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253eea0 msgr2=0x5645925461b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f8878891640 1 --2- 172.21.15.28:0/4238224011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253eea0 0x5645925461b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f8878891640 1 -- 172.21.15.28:0/4238224011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564592430dc0 con 0x56459253e580 2024-04-25T01:39:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.756+0000 7f8873fff640 1 --2- 172.21.15.28:0/4238224011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253eea0 0x5645925461b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:39:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.757+0000 7f8878891640 1 --2- 172.21.15.28:0/4238224011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253e580 0x564592548ec0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f886409ba30 tx=0x7f8864010ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a41d3c70666cb94 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.757+0000 7f8853fff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-25T01:39:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.757+0000 7f8853fff640 1 -- 172.21.15.28:0/4238224011 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886409c1a0 con 0x56459253e580 2024-04-25T01:39:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.757+0000 7f887d804740 1 -- 172.21.15.28:0/4238224011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f886c044070 con 0x56459253e580 2024-04-25T01:39:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.757+0000 7f8853fff640 1 -- 172.21.15.28:0/4238224011 <== mon.2 v2:172.21.15.32:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8864010ac0 con 0x56459253e580 2024-04-25T01:39:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.757+0000 7f8853fff640 1 -- 172.21.15.28:0/4238224011 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f886409baa0 con 0x56459253e580 2024-04-25T01:39:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.757+0000 7f887d804740 10 client.26771 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.757+0000 7f887d804740 1 -- 172.21.15.28:0/4238224011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5645921b73f0 con 0x56459253e580 2024-04-25T01:39:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.757+0000 7f8853fff640 1 -- 172.21.15.28:0/4238224011 <== mon.2 v2:172.21.15.32:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f886c044070 con 0x56459253e580 2024-04-25T01:39:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.757+0000 7f8853fff640 1 -- 172.21.15.28:0/4238224011 <== mon.2 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f88640a4310 con 0x56459253e580 2024-04-25T01:39:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.757+0000 7f887d804740 10 client.26771 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.757+0000 7f887d804740 10 client.26771 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:39:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.757+0000 7f887d804740 20 client.26771 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.757+0000 7f887d804740 1 --2- 172.21.15.28:0/4238224011 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5645925595c0 0x56459255b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.757+0000 7f887d804740 4 client.26771 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.757+0000 7f887d804740 1 -- 172.21.15.28:0/4238224011 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645924b0bb0 con 0x5645925595c0 2024-04-25T01:39:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.758+0000 7f8873fff640 1 --2- 172.21.15.28:0/4238224011 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5645925595c0 0x56459255b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.758+0000 7f8873fff640 1 --2- 172.21.15.28:0/4238224011 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5645925595c0 0x56459255b9b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5617eedf2228885e server_cookie=447e57acf3642bdd in_seq=0 out_seq=0 2024-04-25T01:39:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.758+0000 7f8853fff640 10 client.26771 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:39:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.759+0000 7f8853fff640 1 -- 172.21.15.28:0/4238224011 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5645924b0bb0 con 0x5645925595c0 2024-04-25T01:39:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.759+0000 7f8853fff640 10 client.26771 handle_command_reply: tid=0 2024-04-25T01:39:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.759+0000 7f887d804740 1 client.26771 shutdown 2024-04-25T01:39:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.759+0000 7f887d804740 1 -- 172.21.15.28:0/4238224011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253e580 msgr2=0x564592548ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.759+0000 7f887d804740 1 --2- 172.21.15.28:0/4238224011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253e580 0x564592548ec0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f886409ba30 tx=0x7f8864010ed0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.759+0000 7f887d804740 1 -- 172.21.15.28:0/4238224011 shutdown_connections 2024-04-25T01:39:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.759+0000 7f887d804740 1 --2- 172.21.15.28:0/4238224011 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5645925595c0 0x56459255b9b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.759+0000 7f887d804740 1 --2- 172.21.15.28:0/4238224011 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564592549570 0x564592546710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.759+0000 7f887d804740 1 --2- 172.21.15.28:0/4238224011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253eea0 0x5645925461b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.759+0000 7f887d804740 1 --2- 172.21.15.28:0/4238224011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253e580 0x564592548ec0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.759+0000 7f8853fff640 0 client.26771 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:39:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.759+0000 7f887d804740 1 -- 172.21.15.28:0/4238224011 >> 172.21.15.28:0/4238224011 conn(0x56459253a8a0 msgr2=0x564592539b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.759+0000 7f887d804740 1 -- 172.21.15.28:0/4238224011 shutdown_connections 2024-04-25T01:39:11.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.759+0000 7f887d804740 1 -- 172.21.15.28:0/4238224011 wait complete. 2024-04-25T01:39:11.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.759+0000 7f887d804740 20 client.26771 trim_cache size 0 max 16384 2024-04-25T01:39:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.772+0000 7f887d804740 1 Processor -- start 2024-04-25T01:39:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.772+0000 7f887d804740 1 -- start start 2024-04-25T01:39:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f887d804740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56459253e580 0x564592555220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f887d804740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253eea0 0x5645925415c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f887d804740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564592549570 0x5645925489f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f887d804740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f88741b0c20 con 0x564592549570 2024-04-25T01:39:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f887d804740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f88741b2760 con 0x56459253eea0 2024-04-25T01:39:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f887d804740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f88741b12f0 con 0x56459253e580 2024-04-25T01:39:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f8879092640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564592549570 0x5645925489f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f8879092640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564592549570 0x5645925489f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48534/0 (socket says 172.21.15.28:48534) 2024-04-25T01:39:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f8879092640 1 -- 172.21.15.28:0/2837982257 learned_addr learned my addr 172.21.15.28:0/2837982257 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f8878891640 1 --2- 172.21.15.28:0/2837982257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56459253e580 0x564592555220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f8873fff640 1 --2- 172.21.15.28:0/2837982257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253eea0 0x5645925415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f8879092640 1 -- 172.21.15.28:0/2837982257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56459253e580 msgr2=0x564592555220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f8879092640 1 --2- 172.21.15.28:0/2837982257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56459253e580 0x564592555220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f8879092640 1 -- 172.21.15.28:0/2837982257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253eea0 msgr2=0x5645925415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f8879092640 1 --2- 172.21.15.28:0/2837982257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253eea0 0x5645925415c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f8879092640 1 -- 172.21.15.28:0/2837982257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645922817d0 con 0x564592549570 2024-04-25T01:39:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f8873fff640 1 --2- 172.21.15.28:0/2837982257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253eea0 0x5645925415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f8879092640 1 --2- 172.21.15.28:0/2837982257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564592549570 0x5645925489f0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f886c02ae00 tx=0x7f886c03f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d61ea00310bf35fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f885a7fc640 1 -- 172.21.15.28:0/2837982257 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886c042040 con 0x564592549570 2024-04-25T01:39:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f885a7fc640 1 -- 172.21.15.28:0/2837982257 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f886c036040 con 0x564592549570 2024-04-25T01:39:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.773+0000 7f885a7fc640 1 -- 172.21.15.28:0/2837982257 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886c004310 con 0x564592549570 2024-04-25T01:39:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.774+0000 7f887d804740 1 -- 172.21.15.28:0/2837982257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564592549570 msgr2=0x5645925489f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.774+0000 7f887d804740 1 --2- 172.21.15.28:0/2837982257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564592549570 0x5645925489f0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f886c02ae00 tx=0x7f886c03f000 comp rx=0 tx=0).stop 2024-04-25T01:39:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.774+0000 7f887d804740 1 -- 172.21.15.28:0/2837982257 shutdown_connections 2024-04-25T01:39:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.774+0000 7f887d804740 1 --2- 172.21.15.28:0/2837982257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564592549570 0x5645925489f0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.774+0000 7f887d804740 1 --2- 172.21.15.28:0/2837982257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253eea0 0x5645925415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.774+0000 7f887d804740 1 --2- 172.21.15.28:0/2837982257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56459253e580 0x564592555220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.774+0000 7f887d804740 1 -- 172.21.15.28:0/2837982257 >> 172.21.15.28:0/2837982257 conn(0x56459253a8a0 msgr2=0x564592540eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.774+0000 7f887d804740 1 -- 172.21.15.28:0/2837982257 shutdown_connections 2024-04-25T01:39:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.774+0000 7f887d804740 1 -- 172.21.15.28:0/2837982257 wait complete. 2024-04-25T01:39:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.774+0000 7f887d804740 1 -- start start 2024-04-25T01:39:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.774+0000 7f887d804740 1 Processor -- start 2024-04-25T01:39:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f887d804740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253e580 0x564592551a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f887d804740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253eea0 0x564592542ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f887d804740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564592549570 0x564592543000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f887d804740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f88741b2090 con 0x56459253eea0 2024-04-25T01:39:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f887d804740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f88741b19c0 con 0x56459253e580 2024-04-25T01:39:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f887d804740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f88741aec00 con 0x564592549570 2024-04-25T01:39:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f887d804740 10 client.0 _ll_register_callbacks cb 0x7f8864018420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f8873fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253eea0 0x564592542ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f8873fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253eea0 0x564592542ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48546/0 (socket says 172.21.15.28:48546) 2024-04-25T01:39:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f8873fff640 1 -- 172.21.15.28:0/2519131353 learned_addr learned my addr 172.21.15.28:0/2519131353 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f8878891640 1 --2- 172.21.15.28:0/2519131353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253e580 0x564592551a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f8873fff640 1 -- 172.21.15.28:0/2519131353 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564592549570 msgr2=0x564592543000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f8879092640 1 --2- 172.21.15.28:0/2519131353 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564592549570 0x564592543000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f8873fff640 1 --2- 172.21.15.28:0/2519131353 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564592549570 0x564592543000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f8873fff640 1 -- 172.21.15.28:0/2519131353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253e580 msgr2=0x564592551a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f8873fff640 1 --2- 172.21.15.28:0/2519131353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253e580 0x564592551a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f8873fff640 1 -- 172.21.15.28:0/2519131353 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564592543740 con 0x56459253eea0 2024-04-25T01:39:11.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f8878891640 1 --2- 172.21.15.28:0/2519131353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253e580 0x564592551a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:11.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f8879092640 1 --2- 172.21.15.28:0/2519131353 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564592549570 0x564592543000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:11.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f8873fff640 1 --2- 172.21.15.28:0/2519131353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253eea0 0x564592542ac0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f887410fa00 tx=0x7f8868010c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82dc0e436eeaba23 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f8853fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:11.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f8853fff640 1 -- 172.21.15.28:0/2519131353 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886800fd60 con 0x56459253eea0 2024-04-25T01:39:11.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.775+0000 7f8853fff640 1 -- 172.21.15.28:0/2519131353 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8868004020 con 0x56459253eea0 2024-04-25T01:39:11.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.776+0000 7f887d804740 1 -- 172.21.15.28:0/2519131353 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f886c036040 con 0x56459253eea0 2024-04-25T01:39:11.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.776+0000 7f8853fff640 1 -- 172.21.15.28:0/2519131353 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8868030050 con 0x56459253eea0 2024-04-25T01:39:11.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.776+0000 7f887d804740 10 client.17655 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:11.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.776+0000 7f887d804740 1 -- 172.21.15.28:0/2519131353 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5645925bfa20 con 0x56459253eea0 2024-04-25T01:39:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.776+0000 7f8853fff640 1 -- 172.21.15.28:0/2519131353 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f886c036040 con 0x56459253eea0 2024-04-25T01:39:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.776+0000 7f8853fff640 1 -- 172.21.15.28:0/2519131353 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f88680035f0 con 0x56459253eea0 2024-04-25T01:39:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.776+0000 7f887d804740 10 client.17655 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.776+0000 7f887d804740 10 client.17655 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:39:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.776+0000 7f887d804740 20 client.17655 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.776+0000 7f887d804740 1 --2- 172.21.15.28:0/2519131353 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5645925c0150 0x5645925c2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.776+0000 7f887d804740 4 client.17655 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.776+0000 7f887d804740 1 -- 172.21.15.28:0/2519131353 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5645924aaf00 con 0x5645925c0150 2024-04-25T01:39:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.776+0000 7f8878891640 1 --2- 172.21.15.28:0/2519131353 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5645925c0150 0x5645925c2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.777+0000 7f8878891640 1 --2- 172.21.15.28:0/2519131353 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5645925c0150 0x5645925c2570 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6c13dc22e7654239 server_cookie=b3dbf2348a87a65b in_seq=0 out_seq=0 2024-04-25T01:39:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.777+0000 7f8853fff640 10 client.17655 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:39:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.777+0000 7f8853fff640 1 -- 172.21.15.28:0/2519131353 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5645924aaf00 con 0x5645925c0150 2024-04-25T01:39:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.777+0000 7f8853fff640 10 client.17655 handle_command_reply: tid=0 2024-04-25T01:39:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.777+0000 7f887d804740 1 client.17655 shutdown 2024-04-25T01:39:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.777+0000 7f887d804740 1 -- 172.21.15.28:0/2519131353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253eea0 msgr2=0x564592542ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.777+0000 7f887d804740 1 --2- 172.21.15.28:0/2519131353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253eea0 0x564592542ac0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f887410fa00 tx=0x7f8868010c70 comp rx=0 tx=0).stop 2024-04-25T01:39:11.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.777+0000 7f887d804740 1 -- 172.21.15.28:0/2519131353 shutdown_connections 2024-04-25T01:39:11.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.777+0000 7f887d804740 1 --2- 172.21.15.28:0/2519131353 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5645925c0150 0x5645925c2570 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.777+0000 7f887d804740 1 --2- 172.21.15.28:0/2519131353 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564592549570 0x564592543000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.777+0000 7f887d804740 1 --2- 172.21.15.28:0/2519131353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56459253eea0 0x564592542ac0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.777+0000 7f8853fff640 0 client.17655 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:39:11.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.777+0000 7f887d804740 1 --2- 172.21.15.28:0/2519131353 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56459253e580 0x564592551a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.777+0000 7f887d804740 1 -- 172.21.15.28:0/2519131353 >> 172.21.15.28:0/2519131353 conn(0x56459253a8a0 msgr2=0x564592540560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.777+0000 7f887d804740 1 -- 172.21.15.28:0/2519131353 shutdown_connections 2024-04-25T01:39:11.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.777+0000 7f887d804740 1 -- 172.21.15.28:0/2519131353 wait complete. 2024-04-25T01:39:11.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.777+0000 7f887d804740 20 client.17655 trim_cache size 0 max 16384 2024-04-25T01:39:11.767 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:11.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.778+0000 7f887aaf5640 1 -- 172.21.15.28:0/2849383756 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f885c0a2d00 msgr2=0x7f885c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.778+0000 7f887aaf5640 1 --2- 172.21.15.28:0/2849383756 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f885c0a2d00 0x7f885c0a51c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f886c04d000 tx=0x7f886c04e000 comp rx=0 tx=0).stop 2024-04-25T01:39:11.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.778+0000 7f887aaf5640 1 -- 172.21.15.28:0/2849383756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8874100610 msgr2=0x7f887410df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.778+0000 7f887aaf5640 1 --2- 172.21.15.28:0/2849383756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8874100610 0x7f887410df70 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f886400e9c0 tx=0x7f886400ee80 comp rx=0 tx=0).stop 2024-04-25T01:39:11.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.778+0000 7f887aaf5640 1 -- 172.21.15.28:0/2849383756 shutdown_connections 2024-04-25T01:39:11.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.778+0000 7f887aaf5640 1 --2- 172.21.15.28:0/2849383756 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f885c0a2d00 0x7f885c0a51c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.778+0000 7f887aaf5640 1 --2- 172.21.15.28:0/2849383756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f88741066b0 0x7f887410e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.778+0000 7f887aaf5640 1 --2- 172.21.15.28:0/2849383756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8874100fd0 0x7f887410e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.778+0000 7f887aaf5640 1 --2- 172.21.15.28:0/2849383756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8874100610 0x7f887410df70 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.778+0000 7f887aaf5640 1 -- 172.21.15.28:0/2849383756 >> 172.21.15.28:0/2849383756 conn(0x7f8874077740 msgr2=0x7f88740ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.778+0000 7f887aaf5640 1 -- 172.21.15.28:0/2849383756 shutdown_connections 2024-04-25T01:39:11.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.778+0000 7f887aaf5640 1 -- 172.21.15.28:0/2849383756 wait complete. 2024-04-25T01:39:11.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:39:11.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.894+0000 7fba8c215640 1 -- 172.21.15.28:0/3773757421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba841020d0 msgr2=0x7fba8410e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.894+0000 7fba8c215640 1 --2- 172.21.15.28:0/3773757421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba841020d0 0x7fba8410e5e0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fba7c009cd0 tx=0x7fba7c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.894+0000 7fba8c215640 1 -- 172.21.15.28:0/3773757421 shutdown_connections 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.894+0000 7fba8c215640 1 --2- 172.21.15.28:0/3773757421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba841020d0 0x7fba8410e5e0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.894+0000 7fba8c215640 1 --2- 172.21.15.28:0/3773757421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba84101710 0x7fba84101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.894+0000 7fba8c215640 1 --2- 172.21.15.28:0/3773757421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba84107720 0x7fba84107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.894+0000 7fba8c215640 1 -- 172.21.15.28:0/3773757421 >> 172.21.15.28:0/3773757421 conn(0x7fba840fd180 msgr2=0x7fba840ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.894+0000 7fba8c215640 1 -- 172.21.15.28:0/3773757421 shutdown_connections 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.894+0000 7fba8c215640 1 -- 172.21.15.28:0/3773757421 wait complete. 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba8c215640 1 Processor -- start 2024-04-25T01:39:11.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba8c215640 1 -- start start 2024-04-25T01:39:11.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba8c215640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba84101710 0x7fba84073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba8c215640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba841020d0 0x7fba84073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba8c215640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba84107720 0x7fba840744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba8c215640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fba84111af0 con 0x7fba84107720 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba8c215640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fba84111970 con 0x7fba841020d0 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba8c215640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fba84111c70 con 0x7fba84101710 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba8a7b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba84107720 0x7fba840744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba8a7b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba84107720 0x7fba840744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48560/0 (socket says 172.21.15.28:48560) 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba8a7b2640 1 -- 172.21.15.28:0/3584030183 learned_addr learned my addr 172.21.15.28:0/3584030183 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba897b0640 1 --2- 172.21.15.28:0/3584030183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba841020d0 0x7fba84073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba89fb1640 1 --2- 172.21.15.28:0/3584030183 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba84101710 0x7fba84073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba8a7b2640 1 -- 172.21.15.28:0/3584030183 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba84101710 msgr2=0x7fba84073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba8a7b2640 1 --2- 172.21.15.28:0/3584030183 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba84101710 0x7fba84073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba8a7b2640 1 -- 172.21.15.28:0/3584030183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba841020d0 msgr2=0x7fba84073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba8a7b2640 1 --2- 172.21.15.28:0/3584030183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba841020d0 0x7fba84073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba8a7b2640 1 -- 172.21.15.28:0/3584030183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba841a6390 con 0x7fba84107720 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba89fb1640 1 --2- 172.21.15.28:0/3584030183 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba84101710 0x7fba84073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.895+0000 7fba897b0640 1 --2- 172.21.15.28:0/3584030183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba841020d0 0x7fba84073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.896+0000 7fba8a7b2640 1 --2- 172.21.15.28:0/3584030183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba84107720 0x7fba840744c0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fba7c006010 tx=0x7fba7c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.896+0000 7fba7affd640 1 -- 172.21.15.28:0/3584030183 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c004450 con 0x7fba84107720 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.896+0000 7fba7affd640 1 -- 172.21.15.28:0/3584030183 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba7c003e10 con 0x7fba84107720 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.896+0000 7fba8c215640 1 -- 172.21.15.28:0/3584030183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba841a6620 con 0x7fba84107720 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.896+0000 7fba7affd640 1 -- 172.21.15.28:0/3584030183 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7c046480 con 0x7fba84107720 2024-04-25T01:39:11.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.896+0000 7fba8c215640 1 -- 172.21.15.28:0/3584030183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba841a6b70 con 0x7fba84107720 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.898+0000 7fba7affd640 1 -- 172.21.15.28:0/3584030183 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fba7c0045f0 con 0x7fba84107720 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.898+0000 7fba8f002740 1 Processor -- start 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.898+0000 7fba8f002740 1 -- start start 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.898+0000 7fba8f002740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154d580 0x55cf9154d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.898+0000 7fba8f002740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154dea0 0x55cf91558030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.898+0000 7fba8f002740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf91558570 0x55cf9155a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.898+0000 7fba8f002740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cf914b2670 con 0x55cf9154d580 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.898+0000 7fba8f002740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cf9145a970 con 0x55cf91558570 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.898+0000 7fba8f002740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55cf914ae630 con 0x55cf9154dea0 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.899+0000 7fba897b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154d580 0x55cf9154d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.899+0000 7fba897b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154d580 0x55cf9154d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48574/0 (socket says 172.21.15.28:48574) 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.899+0000 7fba897b0640 1 -- 172.21.15.28:0/3795339390 learned_addr learned my addr 172.21.15.28:0/3795339390 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.899+0000 7fba7affd640 1 --2- 172.21.15.28:0/3584030183 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fba700a2d00 0x7fba700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.899+0000 7fba89fb1640 1 --2- 172.21.15.28:0/3795339390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154dea0 0x55cf91558030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.899+0000 7fba897b0640 1 --2- 172.21.15.28:0/3795339390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf91558570 0x55cf9155a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.899+0000 7fba8a7b2640 1 --2- 172.21.15.28:0/3584030183 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fba700a2d00 0x7fba700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.899+0000 7fba897b0640 1 -- 172.21.15.28:0/3795339390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154dea0 msgr2=0x55cf91558030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.899+0000 7fba897b0640 1 --2- 172.21.15.28:0/3795339390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154dea0 0x55cf91558030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.899+0000 7fba897b0640 1 -- 172.21.15.28:0/3795339390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf91558570 msgr2=0x55cf9155a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.899+0000 7fba897b0640 1 --2- 172.21.15.28:0/3795339390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf91558570 0x55cf9155a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.899+0000 7fba7affd640 1 -- 172.21.15.28:0/3584030183 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fba7c02e030 con 0x7fba84107720 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.899+0000 7fba897b0640 1 -- 172.21.15.28:0/3795339390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf9147e400 con 0x55cf9154d580 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.899+0000 7fba897b0640 1 --2- 172.21.15.28:0/3795339390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154d580 0x55cf9154d960 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fba7400a5c0 tx=0x7fba7400abd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ecab6a3422785b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.899+0000 7fba8a7b2640 1 --2- 172.21.15.28:0/3584030183 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fba700a2d00 0x7fba700a51c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fba7c0bd020 tx=0x7fba7c0dd040 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.899+0000 7fba6ffff640 1 -- 172.21.15.28:0/3795339390 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba74013e60 con 0x55cf9154d580 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.899+0000 7fba6ffff640 1 -- 172.21.15.28:0/3795339390 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba74011e70 con 0x55cf9154d580 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.900+0000 7fba6ffff640 1 -- 172.21.15.28:0/3795339390 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7400e5f0 con 0x55cf9154d580 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.900+0000 7fba8f002740 1 -- 172.21.15.28:0/3795339390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154d580 msgr2=0x55cf9154d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.900+0000 7fba8f002740 1 --2- 172.21.15.28:0/3795339390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154d580 0x55cf9154d960 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fba7400a5c0 tx=0x7fba7400abd0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.900+0000 7fba8f002740 1 -- 172.21.15.28:0/3795339390 shutdown_connections 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.900+0000 7fba8f002740 1 --2- 172.21.15.28:0/3795339390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf91558570 0x55cf9155a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.900+0000 7fba8f002740 1 --2- 172.21.15.28:0/3795339390 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154dea0 0x55cf91558030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.900+0000 7fba8f002740 1 --2- 172.21.15.28:0/3795339390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154d580 0x55cf9154d960 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.900+0000 7fba8f002740 1 -- 172.21.15.28:0/3795339390 >> 172.21.15.28:0/3795339390 conn(0x55cf915498a0 msgr2=0x55cf91549c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.900+0000 7fba8f002740 1 -- 172.21.15.28:0/3795339390 shutdown_connections 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.900+0000 7fba8f002740 1 -- 172.21.15.28:0/3795339390 wait complete. 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.901+0000 7fba8f002740 1 -- start start 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.901+0000 7fba8f002740 1 Processor -- start 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.901+0000 7fba8f002740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154d580 0x55cf91557ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.901+0000 7fba8f002740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154dea0 0x55cf91555190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.901+0000 7fba8f002740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf91558570 0x55cf915556d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.901+0000 7fba8f002740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cf9155b360 con 0x55cf9154dea0 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.901+0000 7fba8f002740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cf9146e050 con 0x55cf91558570 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.901+0000 7fba8f002740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55cf9155b4e0 con 0x55cf9154d580 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.901+0000 7fba8f002740 10 client.0 _ll_register_callbacks cb 0x55cf911c5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.901+0000 7fba89fb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154dea0 0x55cf91555190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.901+0000 7fba89fb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154dea0 0x55cf91555190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48578/0 (socket says 172.21.15.28:48578) 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.901+0000 7fba89fb1640 1 -- 172.21.15.28:0/1284258754 learned_addr learned my addr 172.21.15.28:0/1284258754 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.901+0000 7fba897b0640 1 --2- 172.21.15.28:0/1284258754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf91558570 0x55cf915556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.901+0000 7fba897b0640 1 --2- 172.21.15.28:0/1284258754 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154d580 0x55cf91557ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.902+0000 7fba89fb1640 1 -- 172.21.15.28:0/1284258754 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154d580 msgr2=0x55cf91557ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.902+0000 7fba89fb1640 1 --2- 172.21.15.28:0/1284258754 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154d580 0x55cf91557ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.902+0000 7fba89fb1640 1 -- 172.21.15.28:0/1284258754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf91558570 msgr2=0x55cf915556d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.902+0000 7fba89fb1640 1 --2- 172.21.15.28:0/1284258754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf91558570 0x55cf915556d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.902+0000 7fba89fb1640 1 -- 172.21.15.28:0/1284258754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf9143fdc0 con 0x55cf9154dea0 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.902+0000 7fba897b0640 1 --2- 172.21.15.28:0/1284258754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf91558570 0x55cf915556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.902+0000 7fba89fb1640 1 --2- 172.21.15.28:0/1284258754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154dea0 0x55cf91555190 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fba80010a50 tx=0x7fba80010f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1eeb39c14ec8964 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.902+0000 7fba6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.902+0000 7fba6d7fa640 1 -- 172.21.15.28:0/1284258754 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8001d060 con 0x55cf9154dea0 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.902+0000 7fba6d7fa640 1 -- 172.21.15.28:0/1284258754 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fba8000ebe0 con 0x55cf9154dea0 2024-04-25T01:39:11.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.902+0000 7fba8f002740 1 -- 172.21.15.28:0/1284258754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba74011e70 con 0x55cf9154dea0 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.902+0000 7fba6d7fa640 1 -- 172.21.15.28:0/1284258754 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fba80024050 con 0x55cf9154dea0 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.902+0000 7fba8f002740 10 client.17667 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.902+0000 7fba8f002740 1 -- 172.21.15.28:0/1284258754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cf911c63f0 con 0x55cf9154dea0 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.902+0000 7fba6d7fa640 1 -- 172.21.15.28:0/1284258754 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba74011e70 con 0x55cf9154dea0 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.902+0000 7fba6d7fa640 1 -- 172.21.15.28:0/1284258754 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fba8000d3c0 con 0x55cf9154dea0 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.903+0000 7fba8f002740 10 client.17667 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.903+0000 7fba8f002740 10 client.17667 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.903+0000 7fba8f002740 20 client.17667 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.903+0000 7fba8f002740 1 --2- 172.21.15.28:0/1284258754 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cf915686a0 0x55cf9156aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.903+0000 7fba8f002740 4 client.17667 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.903+0000 7fba8f002740 1 -- 172.21.15.28:0/1284258754 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf914bfbb0 con 0x55cf915686a0 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.903+0000 7fba897b0640 1 --2- 172.21.15.28:0/1284258754 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cf915686a0 0x55cf9156aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.903+0000 7fba6d7fa640 10 client.17667 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.903+0000 7fba897b0640 1 --2- 172.21.15.28:0/1284258754 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cf915686a0 0x55cf9156aac0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a31929780581192d server_cookie=f991d9ddc6a9664e in_seq=0 out_seq=0 2024-04-25T01:39:11.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.904+0000 7fba6d7fa640 1 -- 172.21.15.28:0/1284258754 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cf914bfbb0 con 0x55cf915686a0 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.904+0000 7fba6d7fa640 10 client.17667 handle_command_reply: tid=0 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.904+0000 7fba8f002740 1 client.17667 shutdown 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.904+0000 7fba8f002740 1 -- 172.21.15.28:0/1284258754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154dea0 msgr2=0x55cf91555190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.904+0000 7fba8f002740 1 --2- 172.21.15.28:0/1284258754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154dea0 0x55cf91555190 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fba80010a50 tx=0x7fba80010f10 comp rx=0 tx=0).stop 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.904+0000 7fba8f002740 1 -- 172.21.15.28:0/1284258754 shutdown_connections 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.904+0000 7fba8f002740 1 --2- 172.21.15.28:0/1284258754 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cf915686a0 0x55cf9156aac0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.904+0000 7fba6d7fa640 0 client.17667 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.904+0000 7fba8f002740 1 --2- 172.21.15.28:0/1284258754 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf91558570 0x55cf915556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.904+0000 7fba8f002740 1 --2- 172.21.15.28:0/1284258754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154dea0 0x55cf91555190 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.904+0000 7fba8f002740 1 --2- 172.21.15.28:0/1284258754 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154d580 0x55cf91557ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.904+0000 7fba8f002740 1 -- 172.21.15.28:0/1284258754 >> 172.21.15.28:0/1284258754 conn(0x55cf915498a0 msgr2=0x55cf9154f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.904+0000 7fba8f002740 1 -- 172.21.15.28:0/1284258754 shutdown_connections 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.904+0000 7fba8f002740 1 -- 172.21.15.28:0/1284258754 wait complete. 2024-04-25T01:39:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.904+0000 7fba8f002740 20 client.17667 trim_cache size 0 max 16384 2024-04-25T01:39:11.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.917+0000 7fba8f002740 1 Processor -- start 2024-04-25T01:39:11.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.917+0000 7fba8f002740 1 -- start start 2024-04-25T01:39:11.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba8f002740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154d580 0x55cf91564220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba8f002740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9154dea0 0x55cf915579d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba8f002740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf91558570 0x55cf91570390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba8f002740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fba841b0a60 con 0x55cf91558570 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba8f002740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fba841b25a0 con 0x55cf9154dea0 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba8f002740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fba841b1130 con 0x55cf9154d580 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba897b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf91558570 0x55cf91570390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba897b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf91558570 0x55cf91570390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48594/0 (socket says 172.21.15.28:48594) 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba897b0640 1 -- 172.21.15.28:0/1712648324 learned_addr learned my addr 172.21.15.28:0/1712648324 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba897b0640 1 --2- 172.21.15.28:0/1712648324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154d580 0x55cf91564220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba89fb1640 1 --2- 172.21.15.28:0/1712648324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9154dea0 0x55cf915579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba897b0640 1 -- 172.21.15.28:0/1712648324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154d580 msgr2=0x55cf91564220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba897b0640 1 --2- 172.21.15.28:0/1712648324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154d580 0x55cf91564220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba897b0640 1 -- 172.21.15.28:0/1712648324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9154dea0 msgr2=0x55cf915579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba897b0640 1 --2- 172.21.15.28:0/1712648324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9154dea0 0x55cf915579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba897b0640 1 -- 172.21.15.28:0/1712648324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf912907d0 con 0x55cf91558570 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba89fb1640 1 --2- 172.21.15.28:0/1712648324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9154dea0 0x55cf915579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba897b0640 1 --2- 172.21.15.28:0/1712648324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf91558570 0x55cf91570390 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fba7400fed0 tx=0x7fba7400fe20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59a3de68c431f87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba6f7fe640 1 -- 172.21.15.28:0/1712648324 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba74011d00 con 0x55cf91558570 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba6f7fe640 1 -- 172.21.15.28:0/1712648324 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba7402d7b0 con 0x55cf91558570 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.918+0000 7fba6f7fe640 1 -- 172.21.15.28:0/1712648324 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7400fab0 con 0x55cf91558570 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.919+0000 7fba8f002740 1 -- 172.21.15.28:0/1712648324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf91558570 msgr2=0x55cf91570390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.919+0000 7fba8f002740 1 --2- 172.21.15.28:0/1712648324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf91558570 0x55cf91570390 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fba7400fed0 tx=0x7fba7400fe20 comp rx=0 tx=0).stop 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.919+0000 7fba8f002740 1 -- 172.21.15.28:0/1712648324 shutdown_connections 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.919+0000 7fba8f002740 1 --2- 172.21.15.28:0/1712648324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf91558570 0x55cf91570390 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.919+0000 7fba8f002740 1 --2- 172.21.15.28:0/1712648324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9154dea0 0x55cf915579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.919+0000 7fba8f002740 1 --2- 172.21.15.28:0/1712648324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154d580 0x55cf91564220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.919+0000 7fba8f002740 1 -- 172.21.15.28:0/1712648324 >> 172.21.15.28:0/1712648324 conn(0x55cf915498a0 msgr2=0x55cf9154f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.919+0000 7fba8f002740 1 -- 172.21.15.28:0/1712648324 shutdown_connections 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.919+0000 7fba8f002740 1 -- 172.21.15.28:0/1712648324 wait complete. 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.919+0000 7fba8f002740 1 -- start start 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.919+0000 7fba8f002740 1 Processor -- start 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba8f002740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154d580 0x55cf91560b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba8f002740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154dea0 0x55cf91551c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba8f002740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf91558570 0x55cf915521f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba8f002740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fba841b1ed0 con 0x55cf9154dea0 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba8f002740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fba841b1800 con 0x55cf91558570 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba8f002740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fba841aea40 con 0x55cf9154d580 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba8f002740 10 client.0 _ll_register_callbacks cb 0x7fba8000b6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba89fb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154dea0 0x55cf91551c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba89fb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154dea0 0x55cf91551c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48598/0 (socket says 172.21.15.28:48598) 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba89fb1640 1 -- 172.21.15.28:0/3951229199 learned_addr learned my addr 172.21.15.28:0/3951229199 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:11.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba897b0640 1 --2- 172.21.15.28:0/3951229199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf91558570 0x55cf915521f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba897b0640 1 --2- 172.21.15.28:0/3951229199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154d580 0x55cf91560b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba89fb1640 1 -- 172.21.15.28:0/3951229199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154d580 msgr2=0x55cf91560b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba89fb1640 1 --2- 172.21.15.28:0/3951229199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154d580 0x55cf91560b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba89fb1640 1 -- 172.21.15.28:0/3951229199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf91558570 msgr2=0x55cf915521f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba89fb1640 1 --2- 172.21.15.28:0/3951229199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf91558570 0x55cf915521f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba89fb1640 1 -- 172.21.15.28:0/3951229199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf91552a70 con 0x55cf9154dea0 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba897b0640 1 --2- 172.21.15.28:0/3951229199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf91558570 0x55cf915521f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba89fb1640 1 --2- 172.21.15.28:0/3951229199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154dea0 0x55cf91551c30 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fba80010a20 tx=0x7fba80010640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83dbb05a88c19c75 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba6d7fa640 1 -- 172.21.15.28:0/3951229199 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8002c070 con 0x55cf9154dea0 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba6d7fa640 1 -- 172.21.15.28:0/3951229199 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fba8000ede0 con 0x55cf9154dea0 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.920+0000 7fba8f002740 1 -- 172.21.15.28:0/3951229199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba7402d7b0 con 0x55cf9154dea0 2024-04-25T01:39:11.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.921+0000 7fba6d7fa640 1 -- 172.21.15.28:0/3951229199 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fba8001d020 con 0x55cf9154dea0 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.921+0000 7fba8f002740 10 client.17673 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.921+0000 7fba8f002740 1 -- 172.21.15.28:0/3951229199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cf915cec40 con 0x55cf9154dea0 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.921+0000 7fba6d7fa640 1 -- 172.21.15.28:0/3951229199 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba7402d7b0 con 0x55cf9154dea0 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.921+0000 7fba6d7fa640 1 -- 172.21.15.28:0/3951229199 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fba8002f020 con 0x55cf9154dea0 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.921+0000 7fba8f002740 10 client.17673 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.921+0000 7fba8f002740 10 client.17673 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.921+0000 7fba8f002740 20 client.17673 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.921+0000 7fba8f002740 1 --2- 172.21.15.28:0/3951229199 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cf915686a0 0x55cf915d13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.921+0000 7fba8f002740 4 client.17673 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.921+0000 7fba8f002740 1 -- 172.21.15.28:0/3951229199 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cf914b9f00 con 0x55cf915686a0 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.921+0000 7fba897b0640 1 --2- 172.21.15.28:0/3951229199 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cf915686a0 0x55cf915d13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.922+0000 7fba6d7fa640 10 client.17673 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.922+0000 7fba897b0640 1 --2- 172.21.15.28:0/3951229199 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cf915686a0 0x55cf915d13f0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5b2654aef6a03f5 server_cookie=66543167299fadfb in_seq=0 out_seq=0 2024-04-25T01:39:11.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.922+0000 7fba6d7fa640 1 -- 172.21.15.28:0/3951229199 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cf914b9f00 con 0x55cf915686a0 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.922+0000 7fba6d7fa640 10 client.17673 handle_command_reply: tid=0 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.922+0000 7fba8f002740 1 client.17673 shutdown 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.922+0000 7fba8f002740 1 -- 172.21.15.28:0/3951229199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154dea0 msgr2=0x55cf91551c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.922+0000 7fba8f002740 1 --2- 172.21.15.28:0/3951229199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154dea0 0x55cf91551c30 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fba80010a20 tx=0x7fba80010640 comp rx=0 tx=0).stop 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.922+0000 7fba8f002740 1 -- 172.21.15.28:0/3951229199 shutdown_connections 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.922+0000 7fba8f002740 1 --2- 172.21.15.28:0/3951229199 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cf915686a0 0x55cf915d13f0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.922+0000 7fba8f002740 1 --2- 172.21.15.28:0/3951229199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf91558570 0x55cf915521f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.922+0000 7fba8f002740 1 --2- 172.21.15.28:0/3951229199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9154dea0 0x55cf91551c30 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.922+0000 7fba8f002740 1 --2- 172.21.15.28:0/3951229199 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9154d580 0x55cf91560b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.922+0000 7fba6d7fa640 0 client.17673 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.922+0000 7fba8f002740 1 -- 172.21.15.28:0/3951229199 >> 172.21.15.28:0/3951229199 conn(0x55cf915498a0 msgr2=0x55cf91554a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.922+0000 7fba8f002740 1 -- 172.21.15.28:0/3951229199 shutdown_connections 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.922+0000 7fba8f002740 1 -- 172.21.15.28:0/3951229199 wait complete. 2024-04-25T01:39:11.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.922+0000 7fba8f002740 20 client.17673 trim_cache size 0 max 16384 2024-04-25T01:39:11.912 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:11.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.923+0000 7fba8c215640 1 -- 172.21.15.28:0/3584030183 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fba700a2d00 msgr2=0x7fba700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.923+0000 7fba8c215640 1 --2- 172.21.15.28:0/3584030183 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fba700a2d00 0x7fba700a51c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fba7c0bd020 tx=0x7fba7c0dd040 comp rx=0 tx=0).stop 2024-04-25T01:39:11.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.923+0000 7fba8c215640 1 -- 172.21.15.28:0/3584030183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba84107720 msgr2=0x7fba840744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:11.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.923+0000 7fba8c215640 1 --2- 172.21.15.28:0/3584030183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba84107720 0x7fba840744c0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fba7c006010 tx=0x7fba7c003c20 comp rx=0 tx=0).stop 2024-04-25T01:39:11.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.923+0000 7fba8c215640 1 -- 172.21.15.28:0/3584030183 shutdown_connections 2024-04-25T01:39:11.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.923+0000 7fba8c215640 1 --2- 172.21.15.28:0/3584030183 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fba700a2d00 0x7fba700a51c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.923+0000 7fba8c215640 1 --2- 172.21.15.28:0/3584030183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba84107720 0x7fba840744c0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.923+0000 7fba8c215640 1 --2- 172.21.15.28:0/3584030183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fba841020d0 0x7fba84073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.923+0000 7fba8c215640 1 --2- 172.21.15.28:0/3584030183 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fba84101710 0x7fba84073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:11.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.923+0000 7fba8c215640 1 -- 172.21.15.28:0/3584030183 >> 172.21.15.28:0/3584030183 conn(0x7fba840fd180 msgr2=0x7fba840fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:11.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.923+0000 7fba8c215640 1 -- 172.21.15.28:0/3584030183 shutdown_connections 2024-04-25T01:39:11.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:11.923+0000 7fba8c215640 1 -- 172.21.15.28:0/3584030183 wait complete. 2024-04-25T01:39:12.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:11 smithi032 ceph-mon[32461]: pgmap v188: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:39:12.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:11 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1515152319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:39:12.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:11 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:39:12.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:11 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:39:12.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:11 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:39:12.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:11 smithi028 ceph-mon[29390]: pgmap v188: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:39:12.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:11 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1515152319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:39:12.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:11 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:39:12.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:11 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:39:12.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:11 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:39:12.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:11 smithi130 ceph-mon[32702]: pgmap v188: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-04-25T01:39:12.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:11 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1515152319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:39:12.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:11 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:39:12.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:11 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:39:12.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:11 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:39:12.917 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:39:12.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:39:12.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:39:13.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.056+0000 7f80a1d7e640 1 -- 172.21.15.28:0/2207631071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c069dc0 msgr2=0x7f809c06a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.056+0000 7f80a1d7e640 1 --2- 172.21.15.28:0/2207631071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c069dc0 0x7f809c06a240 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f808c009930 tx=0x7f808c02a420 comp rx=0 tx=0).stop 2024-04-25T01:39:13.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.056+0000 7f80a1d7e640 1 -- 172.21.15.28:0/2207631071 shutdown_connections 2024-04-25T01:39:13.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.056+0000 7f80a1d7e640 1 --2- 172.21.15.28:0/2207631071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f809c06a780 0x7f809c10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.056+0000 7f80a1d7e640 1 --2- 172.21.15.28:0/2207631071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c069dc0 0x7f809c06a240 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.056+0000 7f80a1d7e640 1 --2- 172.21.15.28:0/2207631071 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f809c107730 0x7f809c107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.056+0000 7f80a1d7e640 1 -- 172.21.15.28:0/2207631071 >> 172.21.15.28:0/2207631071 conn(0x7f809c0776a0 msgr2=0x7f809c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.056+0000 7f80a1d7e640 1 -- 172.21.15.28:0/2207631071 shutdown_connections 2024-04-25T01:39:13.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.056+0000 7f80a1d7e640 1 -- 172.21.15.28:0/2207631071 wait complete. 2024-04-25T01:39:13.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.057+0000 7f80a1d7e640 1 Processor -- start 2024-04-25T01:39:13.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.057+0000 7f80a1d7e640 1 -- start start 2024-04-25T01:39:13.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.057+0000 7f80a1d7e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f809c069dc0 0x7f809c19a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.057+0000 7f80a1d7e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f809c06a780 0x7f809c19a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.057+0000 7f80a1d7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c107730 0x7f809c19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.057+0000 7f80a1d7e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f809c111b90 con 0x7f809c107730 2024-04-25T01:39:13.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.057+0000 7f80a1d7e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f809c111a10 con 0x7f809c069dc0 2024-04-25T01:39:13.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.057+0000 7f80a1d7e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f809c111d10 con 0x7f809c06a780 2024-04-25T01:39:13.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.057+0000 7f809bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c107730 0x7f809c19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.057+0000 7f809bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c107730 0x7f809c19eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48612/0 (socket says 172.21.15.28:48612) 2024-04-25T01:39:13.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.058+0000 7f809bfff640 1 -- 172.21.15.28:0/3534497601 learned_addr learned my addr 172.21.15.28:0/3534497601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.058+0000 7f809b7fe640 1 --2- 172.21.15.28:0/3534497601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f809c069dc0 0x7f809c19a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.058+0000 7f809affd640 1 --2- 172.21.15.28:0/3534497601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f809c06a780 0x7f809c19a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.058+0000 7f809bfff640 1 -- 172.21.15.28:0/3534497601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f809c06a780 msgr2=0x7f809c19a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.058+0000 7f809bfff640 1 --2- 172.21.15.28:0/3534497601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f809c06a780 0x7f809c19a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.058+0000 7f809bfff640 1 -- 172.21.15.28:0/3534497601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f809c069dc0 msgr2=0x7f809c19a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.058+0000 7f809bfff640 1 --2- 172.21.15.28:0/3534497601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f809c069dc0 0x7f809c19a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.058+0000 7f809bfff640 1 -- 172.21.15.28:0/3534497601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f809c19f2b0 con 0x7f809c107730 2024-04-25T01:39:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.058+0000 7f809affd640 1 --2- 172.21.15.28:0/3534497601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f809c06a780 0x7f809c19a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:13.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.058+0000 7f809bfff640 1 --2- 172.21.15.28:0/3534497601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c107730 0x7f809c19eb80 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f8094009fd0 tx=0x7f809400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.058+0000 7f8098ff9640 1 -- 172.21.15.28:0/3534497601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8094017070 con 0x7f809c107730 2024-04-25T01:39:13.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.058+0000 7f8098ff9640 1 -- 172.21.15.28:0/3534497601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8094003e50 con 0x7f809c107730 2024-04-25T01:39:13.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.058+0000 7f80a1d7e640 1 -- 172.21.15.28:0/3534497601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f809c19f5a0 con 0x7f809c107730 2024-04-25T01:39:13.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.058+0000 7f8098ff9640 1 -- 172.21.15.28:0/3534497601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80940106c0 con 0x7f809c107730 2024-04-25T01:39:13.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.058+0000 7f80a1d7e640 1 -- 172.21.15.28:0/3534497601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f809c1a6e30 con 0x7f809c107730 2024-04-25T01:39:13.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.060+0000 7f8098ff9640 1 -- 172.21.15.28:0/3534497601 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8094003970 con 0x7f809c107730 2024-04-25T01:39:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f80a4b6a740 1 Processor -- start 2024-04-25T01:39:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f80a4b6a740 1 -- start start 2024-04-25T01:39:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f80a4b6a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559bcfa57980 0x559bcfa55cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f80a4b6a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa56290 0x559bcfa56710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f80a4b6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa61010 0x559bcfa63400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f80a4b6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559bcf9ba2c0 con 0x559bcfa61010 2024-04-25T01:39:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f80a4b6a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559bcf962910 con 0x559bcfa57980 2024-04-25T01:39:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f80a4b6a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x559bcf9b6280 con 0x559bcfa56290 2024-04-25T01:39:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f809affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa61010 0x559bcfa63400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f809affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa61010 0x559bcfa63400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48614/0 (socket says 172.21.15.28:48614) 2024-04-25T01:39:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f809affd640 1 -- 172.21.15.28:0/1773971588 learned_addr learned my addr 172.21.15.28:0/1773971588 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f8098ff9640 1 --2- 172.21.15.28:0/3534497601 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f80840a2d80 0x7f80840a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f809affd640 1 --2- 172.21.15.28:0/1773971588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559bcfa57980 0x559bcfa55cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f809b7fe640 1 --2- 172.21.15.28:0/1773971588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa56290 0x559bcfa56710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f809bfff640 1 --2- 172.21.15.28:0/3534497601 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f80840a2d80 0x7f80840a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f809affd640 1 -- 172.21.15.28:0/1773971588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa56290 msgr2=0x559bcfa56710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f809affd640 1 --2- 172.21.15.28:0/1773971588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa56290 0x559bcfa56710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f809affd640 1 -- 172.21.15.28:0/1773971588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559bcfa57980 msgr2=0x559bcfa55cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f809affd640 1 --2- 172.21.15.28:0/1773971588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559bcfa57980 0x559bcfa55cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.061+0000 7f809affd640 1 -- 172.21.15.28:0/1773971588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bcf973fb0 con 0x559bcfa61010 2024-04-25T01:39:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.062+0000 7f8098ff9640 1 -- 172.21.15.28:0/3534497601 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f80940cf500 con 0x7f809c107730 2024-04-25T01:39:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.062+0000 7f809affd640 1 --2- 172.21.15.28:0/1773971588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa61010 0x559bcfa63400 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f808c031550 tx=0x7f808c009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a383bb28acdc881f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.062+0000 7f8081ffb640 1 -- 172.21.15.28:0/1773971588 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c043020 con 0x559bcfa61010 2024-04-25T01:39:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.062+0000 7f8081ffb640 1 -- 172.21.15.28:0/1773971588 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f808c0046c0 con 0x559bcfa61010 2024-04-25T01:39:13.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.062+0000 7f8081ffb640 1 -- 172.21.15.28:0/1773971588 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c035040 con 0x559bcfa61010 2024-04-25T01:39:13.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.062+0000 7f809bfff640 1 --2- 172.21.15.28:0/3534497601 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f80840a2d80 0x7f80840a5240 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f80940a3dc0 tx=0x7f809406d640 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.063+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1773971588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa61010 msgr2=0x559bcfa63400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.063+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/1773971588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa61010 0x559bcfa63400 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f808c031550 tx=0x7f808c009640 comp rx=0 tx=0).stop 2024-04-25T01:39:13.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.063+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1773971588 shutdown_connections 2024-04-25T01:39:13.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.063+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/1773971588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa61010 0x559bcfa63400 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.063+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/1773971588 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa56290 0x559bcfa56710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.063+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/1773971588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559bcfa57980 0x559bcfa55cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.063+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1773971588 >> 172.21.15.28:0/1773971588 conn(0x559bcfa1e950 msgr2=0x559bcfa527f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.063+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1773971588 shutdown_connections 2024-04-25T01:39:13.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.063+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1773971588 wait complete. 2024-04-25T01:39:13.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.063+0000 7f80a4b6a740 1 -- start start 2024-04-25T01:39:13.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.063+0000 7f80a4b6a740 1 Processor -- start 2024-04-25T01:39:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.063+0000 7f80a4b6a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa56290 0x559bcfa5df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f80a4b6a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559bcfa57980 0x559bcfa5e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f80a4b6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa61010 0x559bcfa5e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f80a4b6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559bcfa63e40 con 0x559bcfa61010 2024-04-25T01:39:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f80a4b6a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559bcf975e90 con 0x559bcfa57980 2024-04-25T01:39:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f80a4b6a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x559bcfa63fc0 con 0x559bcfa56290 2024-04-25T01:39:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f80a4b6a740 10 client.0 _ll_register_callbacks cb 0x559bcf9b1970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f809affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa61010 0x559bcfa5e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f809affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa61010 0x559bcfa5e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48628/0 (socket says 172.21.15.28:48628) 2024-04-25T01:39:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f809affd640 1 -- 172.21.15.28:0/1679209392 learned_addr learned my addr 172.21.15.28:0/1679209392 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f809b7fe640 1 --2- 172.21.15.28:0/1679209392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559bcfa57980 0x559bcfa5e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f809affd640 1 --2- 172.21.15.28:0/1679209392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa56290 0x559bcfa5df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f809affd640 1 -- 172.21.15.28:0/1679209392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa56290 msgr2=0x559bcfa5df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f809affd640 1 --2- 172.21.15.28:0/1679209392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa56290 0x559bcfa5df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f809affd640 1 -- 172.21.15.28:0/1679209392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559bcfa57980 msgr2=0x559bcfa5e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f809affd640 1 --2- 172.21.15.28:0/1679209392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559bcfa57980 0x559bcfa5e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f809affd640 1 -- 172.21.15.28:0/1679209392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bcf7967e0 con 0x559bcfa61010 2024-04-25T01:39:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f809affd640 1 --2- 172.21.15.28:0/1679209392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa61010 0x559bcfa5e9e0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f808c003650 tx=0x7f808c03f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c827d2573c28aff2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f807affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f807affd640 1 -- 172.21.15.28:0/1679209392 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c000e60 con 0x559bcfa61010 2024-04-25T01:39:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f807affd640 1 -- 172.21.15.28:0/1679209392 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f808c004cb0 con 0x559bcfa61010 2024-04-25T01:39:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.064+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1679209392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f808c0046c0 con 0x559bcfa61010 2024-04-25T01:39:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.065+0000 7f807affd640 1 -- 172.21.15.28:0/1679209392 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f808c05c050 con 0x559bcfa61010 2024-04-25T01:39:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.065+0000 7f80a4b6a740 10 client.17685 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.065+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1679209392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559bcf7f6f70 con 0x559bcfa61010 2024-04-25T01:39:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.065+0000 7f807affd640 1 -- 172.21.15.28:0/1679209392 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f808c0046c0 con 0x559bcfa61010 2024-04-25T01:39:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.065+0000 7f807affd640 1 -- 172.21.15.28:0/1679209392 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f808c03a040 con 0x559bcfa61010 2024-04-25T01:39:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.065+0000 7f80a4b6a740 10 client.17685 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.065+0000 7f80a4b6a740 10 client.17685 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.065+0000 7f80a4b6a740 20 client.17685 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.065+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/1679209392 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x559bcfa70f90 0x559bcfa73380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.065+0000 7f80a4b6a740 4 client.17685 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.065+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1679209392 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559bcf98e5a0 con 0x559bcfa70f90 2024-04-25T01:39:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.065+0000 7f809b7fe640 1 --2- 172.21.15.28:0/1679209392 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x559bcfa70f90 0x559bcfa73380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.066+0000 7f809b7fe640 1 --2- 172.21.15.28:0/1679209392 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x559bcfa70f90 0x559bcfa73380 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=376d4cecf3f7465 server_cookie=be4d40048a19e0d7 in_seq=0 out_seq=0 2024-04-25T01:39:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.066+0000 7f807affd640 10 client.17685 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.066+0000 7f807affd640 1 -- 172.21.15.28:0/1679209392 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559bcf98e5a0 con 0x559bcfa70f90 2024-04-25T01:39:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.066+0000 7f807affd640 10 client.17685 handle_command_reply: tid=0 2024-04-25T01:39:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.066+0000 7f80a4b6a740 1 client.17685 shutdown 2024-04-25T01:39:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.067+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1679209392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa61010 msgr2=0x559bcfa5e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.067+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/1679209392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa61010 0x559bcfa5e9e0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f808c003650 tx=0x7f808c03f040 comp rx=0 tx=0).stop 2024-04-25T01:39:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.067+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1679209392 shutdown_connections 2024-04-25T01:39:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.067+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/1679209392 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x559bcfa70f90 0x559bcfa73380 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.067+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/1679209392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa61010 0x559bcfa5e9e0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.067+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/1679209392 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559bcfa57980 0x559bcfa5e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.067+0000 7f807affd640 0 client.17685 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.067+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/1679209392 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa56290 0x559bcfa5df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.067+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1679209392 >> 172.21.15.28:0/1679209392 conn(0x559bcfa1e950 msgr2=0x559bcfa58660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.067+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1679209392 shutdown_connections 2024-04-25T01:39:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.067+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1679209392 wait complete. 2024-04-25T01:39:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.067+0000 7f80a4b6a740 20 client.17685 trim_cache size 0 max 16384 2024-04-25T01:39:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.085+0000 7f80a4b6a740 1 Processor -- start 2024-04-25T01:39:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.085+0000 7f80a4b6a740 1 -- start start 2024-04-25T01:39:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.085+0000 7f80a4b6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa56290 0x559bcfa517d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.085+0000 7f80a4b6a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559bcfa57980 0x559bcfa763a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.085+0000 7f80a4b6a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa61010 0x559bcfa768e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.085+0000 7f80a4b6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f809c1b0ba0 con 0x559bcfa56290 2024-04-25T01:39:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.085+0000 7f80a4b6a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f809c1b26e0 con 0x559bcfa57980 2024-04-25T01:39:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.085+0000 7f80a4b6a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f809c1b1270 con 0x559bcfa61010 2024-04-25T01:39:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.085+0000 7f809affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa56290 0x559bcfa517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.085+0000 7f809affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa56290 0x559bcfa517d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48632/0 (socket says 172.21.15.28:48632) 2024-04-25T01:39:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.085+0000 7f809affd640 1 -- 172.21.15.28:0/2499402987 learned_addr learned my addr 172.21.15.28:0/2499402987 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.085+0000 7f809affd640 1 -- 172.21.15.28:0/2499402987 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa61010 msgr2=0x559bcfa768e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.085+0000 7f809affd640 1 --2- 172.21.15.28:0/2499402987 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa61010 0x559bcfa768e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.085+0000 7f809affd640 1 -- 172.21.15.28:0/2499402987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559bcfa57980 msgr2=0x559bcfa763a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.085+0000 7f809affd640 1 --2- 172.21.15.28:0/2499402987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559bcfa57980 0x559bcfa763a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.085+0000 7f809affd640 1 -- 172.21.15.28:0/2499402987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bcf68e170 con 0x559bcfa56290 2024-04-25T01:39:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.085+0000 7f809affd640 1 --2- 172.21.15.28:0/2499402987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa56290 0x559bcfa517d0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f808c001eb0 tx=0x7f808c003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a14db7c365883e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.086+0000 7f80817fa640 1 -- 172.21.15.28:0/2499402987 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c035070 con 0x559bcfa56290 2024-04-25T01:39:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.086+0000 7f80817fa640 1 -- 172.21.15.28:0/2499402987 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f808c0046c0 con 0x559bcfa56290 2024-04-25T01:39:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.086+0000 7f80817fa640 1 -- 172.21.15.28:0/2499402987 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c043070 con 0x559bcfa56290 2024-04-25T01:39:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.086+0000 7f80a4b6a740 1 -- 172.21.15.28:0/2499402987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa56290 msgr2=0x559bcfa517d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.086+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/2499402987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa56290 0x559bcfa517d0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f808c001eb0 tx=0x7f808c003680 comp rx=0 tx=0).stop 2024-04-25T01:39:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.086+0000 7f80a4b6a740 1 -- 172.21.15.28:0/2499402987 shutdown_connections 2024-04-25T01:39:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.086+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/2499402987 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa61010 0x559bcfa768e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.086+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/2499402987 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559bcfa57980 0x559bcfa763a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.086+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/2499402987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa56290 0x559bcfa517d0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.086+0000 7f80a4b6a740 1 -- 172.21.15.28:0/2499402987 >> 172.21.15.28:0/2499402987 conn(0x559bcfa1e950 msgr2=0x559bcfa510c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.086+0000 7f80a4b6a740 1 -- 172.21.15.28:0/2499402987 shutdown_connections 2024-04-25T01:39:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.086+0000 7f80a4b6a740 1 -- 172.21.15.28:0/2499402987 wait complete. 2024-04-25T01:39:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f80a4b6a740 1 -- start start 2024-04-25T01:39:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f80a4b6a740 1 Processor -- start 2024-04-25T01:39:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f80a4b6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa56290 0x559bcfa741e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f80a4b6a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa57980 0x559bcfa74720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f80a4b6a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559bcfa61010 0x559bcfad6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f80a4b6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f809c1b2010 con 0x559bcfa56290 2024-04-25T01:39:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f80a4b6a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f809c1b1940 con 0x559bcfa61010 2024-04-25T01:39:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f80a4b6a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f809c1aeb80 con 0x559bcfa57980 2024-04-25T01:39:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f80a4b6a740 10 client.0 _ll_register_callbacks cb 0x7f808c004340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f809affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa56290 0x559bcfa741e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f809affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa56290 0x559bcfa741e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48640/0 (socket says 172.21.15.28:48640) 2024-04-25T01:39:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f809affd640 1 -- 172.21.15.28:0/1522327396 learned_addr learned my addr 172.21.15.28:0/1522327396 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f809affd640 1 --2- 172.21.15.28:0/1522327396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559bcfa61010 0x559bcfad6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f809b7fe640 1 --2- 172.21.15.28:0/1522327396 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa57980 0x559bcfa74720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f809affd640 1 -- 172.21.15.28:0/1522327396 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa57980 msgr2=0x559bcfa74720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f809affd640 1 --2- 172.21.15.28:0/1522327396 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa57980 0x559bcfa74720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f809affd640 1 -- 172.21.15.28:0/1522327396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559bcfa61010 msgr2=0x559bcfad6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f809affd640 1 --2- 172.21.15.28:0/1522327396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559bcfa61010 0x559bcfad6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f809affd640 1 -- 172.21.15.28:0/1522327396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bcfad66b0 con 0x559bcfa56290 2024-04-25T01:39:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.087+0000 7f809b7fe640 1 --2- 172.21.15.28:0/1522327396 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa57980 0x559bcfa74720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.088+0000 7f809affd640 1 --2- 172.21.15.28:0/1522327396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa56290 0x559bcfa741e0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f808c031830 tx=0x7f808c031730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9d7e13230a7c283 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.088+0000 7f807affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.088+0000 7f807affd640 1 -- 172.21.15.28:0/1522327396 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808c043070 con 0x559bcfa56290 2024-04-25T01:39:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.088+0000 7f807affd640 1 -- 172.21.15.28:0/1522327396 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f808c003f00 con 0x559bcfa56290 2024-04-25T01:39:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.088+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1522327396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f808c0046c0 con 0x559bcfa56290 2024-04-25T01:39:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.088+0000 7f807affd640 1 -- 172.21.15.28:0/1522327396 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f808c070050 con 0x559bcfa56290 2024-04-25T01:39:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.088+0000 7f80a4b6a740 10 client.17691 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.088+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1522327396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559bcfad7890 con 0x559bcfa56290 2024-04-25T01:39:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.088+0000 7f807affd640 1 -- 172.21.15.28:0/1522327396 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f808c0046c0 con 0x559bcfa56290 2024-04-25T01:39:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.088+0000 7f807affd640 1 -- 172.21.15.28:0/1522327396 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f808c048570 con 0x559bcfa56290 2024-04-25T01:39:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.088+0000 7f80a4b6a740 10 client.17691 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.088+0000 7f80a4b6a740 10 client.17691 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.088+0000 7f80a4b6a740 20 client.17691 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.088+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/1522327396 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x559bcfad7f40 0x559bcfada360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.088+0000 7f80a4b6a740 4 client.17691 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:39:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.088+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1522327396 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559bcf9c7800 con 0x559bcfad7f40 2024-04-25T01:39:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.089+0000 7f809b7fe640 1 --2- 172.21.15.28:0/1522327396 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x559bcfad7f40 0x559bcfada360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.089+0000 7f809b7fe640 1 --2- 172.21.15.28:0/1522327396 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x559bcfad7f40 0x559bcfada360 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94c8fa35da27af4 server_cookie=6cbb5711cbb6d755 in_seq=0 out_seq=0 2024-04-25T01:39:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.089+0000 7f807affd640 10 client.17691 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.090+0000 7f807affd640 1 -- 172.21.15.28:0/1522327396 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559bcf9c7800 con 0x559bcfad7f40 2024-04-25T01:39:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.090+0000 7f807affd640 10 client.17691 handle_command_reply: tid=0 2024-04-25T01:39:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.090+0000 7f80a4b6a740 1 client.17691 shutdown 2024-04-25T01:39:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.090+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1522327396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa56290 msgr2=0x559bcfa741e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.090+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/1522327396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa56290 0x559bcfa741e0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f808c031830 tx=0x7f808c031730 comp rx=0 tx=0).stop 2024-04-25T01:39:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1522327396 shutdown_connections 2024-04-25T01:39:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/1522327396 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x559bcfad7f40 0x559bcfada360 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/1522327396 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559bcfa61010 0x559bcfad6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/1522327396 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559bcfa57980 0x559bcfa74720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a4b6a740 1 --2- 172.21.15.28:0/1522327396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559bcfa56290 0x559bcfa741e0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1522327396 >> 172.21.15.28:0/1522327396 conn(0x559bcfa1e950 msgr2=0x559bcfa51950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f807affd640 0 client.17691 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1522327396 shutdown_connections 2024-04-25T01:39:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a4b6a740 1 -- 172.21.15.28:0/1522327396 wait complete. 2024-04-25T01:39:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a4b6a740 20 client.17691 trim_cache size 0 max 16384 2024-04-25T01:39:13.078 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:39:13.078 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:39:13.078 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "ff637efe-5d3b-41ea-bd02-c0d06bcbef0e", 2024-04-25T01:39:13.078 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:39:13.078 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:39:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a1d7e640 1 -- 172.21.15.28:0/3534497601 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f80840a2d80 msgr2=0x7f80840a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a1d7e640 1 --2- 172.21.15.28:0/3534497601 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f80840a2d80 0x7f80840a5240 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f80940a3dc0 tx=0x7f809406d640 comp rx=0 tx=0).stop 2024-04-25T01:39:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a1d7e640 1 -- 172.21.15.28:0/3534497601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c107730 msgr2=0x7f809c19eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a1d7e640 1 --2- 172.21.15.28:0/3534497601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c107730 0x7f809c19eb80 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f8094009fd0 tx=0x7f809400b260 comp rx=0 tx=0).stop 2024-04-25T01:39:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a1d7e640 1 -- 172.21.15.28:0/3534497601 shutdown_connections 2024-04-25T01:39:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a1d7e640 1 --2- 172.21.15.28:0/3534497601 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f80840a2d80 0x7f80840a5240 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a1d7e640 1 --2- 172.21.15.28:0/3534497601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c107730 0x7f809c19eb80 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a1d7e640 1 --2- 172.21.15.28:0/3534497601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f809c06a780 0x7f809c19a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a1d7e640 1 --2- 172.21.15.28:0/3534497601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f809c069dc0 0x7f809c19a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a1d7e640 1 -- 172.21.15.28:0/3534497601 >> 172.21.15.28:0/3534497601 conn(0x7f809c0776a0 msgr2=0x7f809c10cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.091+0000 7f80a1d7e640 1 -- 172.21.15.28:0/3534497601 shutdown_connections 2024-04-25T01:39:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.092+0000 7f80a1d7e640 1 -- 172.21.15.28:0/3534497601 wait complete. 2024-04-25T01:39:13.085 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:39:13.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:39:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.207+0000 7f75b9790640 1 -- 172.21.15.28:0/2236313778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75b4069dc0 msgr2=0x7f75b406a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.207+0000 7f75b9790640 1 --2- 172.21.15.28:0/2236313778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75b4069dc0 0x7f75b406a240 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f759c009930 tx=0x7f759c02a420 comp rx=0 tx=0).stop 2024-04-25T01:39:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.207+0000 7f75b9790640 1 -- 172.21.15.28:0/2236313778 shutdown_connections 2024-04-25T01:39:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.207+0000 7f75b9790640 1 --2- 172.21.15.28:0/2236313778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75b406a780 0x7f75b410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.207+0000 7f75b9790640 1 --2- 172.21.15.28:0/2236313778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75b4069dc0 0x7f75b406a240 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.207+0000 7f75b9790640 1 --2- 172.21.15.28:0/2236313778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75b4107730 0x7f75b4107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.207+0000 7f75b9790640 1 -- 172.21.15.28:0/2236313778 >> 172.21.15.28:0/2236313778 conn(0x7f75b40776a0 msgr2=0x7f75b4077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.207+0000 7f75b9790640 1 -- 172.21.15.28:0/2236313778 shutdown_connections 2024-04-25T01:39:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.207+0000 7f75b9790640 1 -- 172.21.15.28:0/2236313778 wait complete. 2024-04-25T01:39:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.208+0000 7f75b9790640 1 Processor -- start 2024-04-25T01:39:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.208+0000 7f75b9790640 1 -- start start 2024-04-25T01:39:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.208+0000 7f75b9790640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75b4069dc0 0x7f75b4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.208+0000 7f75b9790640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75b406a780 0x7f75b4198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.208+0000 7f75b9790640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75b4107730 0x7f75b419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.208+0000 7f75b9790640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75b4111b40 con 0x7f75b4069dc0 2024-04-25T01:39:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.208+0000 7f75b9790640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f75b41119c0 con 0x7f75b4107730 2024-04-25T01:39:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.208+0000 7f75b9790640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f75b4111cc0 con 0x7f75b406a780 2024-04-25T01:39:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.208+0000 7f75b2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75b4069dc0 0x7f75b4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.208+0000 7f75b2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75b4069dc0 0x7f75b4197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48666/0 (socket says 172.21.15.28:48666) 2024-04-25T01:39:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.208+0000 7f75b2ffd640 1 -- 172.21.15.28:0/2849900424 learned_addr learned my addr 172.21.15.28:0/2849900424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.208+0000 7f75b2ffd640 1 -- 172.21.15.28:0/2849900424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75b406a780 msgr2=0x7f75b4198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.208+0000 7f75b2ffd640 1 --2- 172.21.15.28:0/2849900424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75b406a780 0x7f75b4198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.208+0000 7f75b2ffd640 1 -- 172.21.15.28:0/2849900424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75b4107730 msgr2=0x7f75b419c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.208+0000 7f75b37fe640 1 --2- 172.21.15.28:0/2849900424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75b4107730 0x7f75b419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.208+0000 7f75b2ffd640 1 --2- 172.21.15.28:0/2849900424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75b4107730 0x7f75b419c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.208+0000 7f75b2ffd640 1 -- 172.21.15.28:0/2849900424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b419cfd0 con 0x7f75b4069dc0 2024-04-25T01:39:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.209+0000 7f75b2ffd640 1 --2- 172.21.15.28:0/2849900424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75b4069dc0 0x7f75b4197fe0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f75a400d770 tx=0x7f75a400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.209+0000 7f75b0ff9640 1 -- 172.21.15.28:0/2849900424 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a4017070 con 0x7f75b4069dc0 2024-04-25T01:39:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.209+0000 7f75b9790640 1 -- 172.21.15.28:0/2849900424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b419d2c0 con 0x7f75b4069dc0 2024-04-25T01:39:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.209+0000 7f75b0ff9640 1 -- 172.21.15.28:0/2849900424 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75a4005500 con 0x7f75b4069dc0 2024-04-25T01:39:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.209+0000 7f75b0ff9640 1 -- 172.21.15.28:0/2849900424 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a4004e00 con 0x7f75b4069dc0 2024-04-25T01:39:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.209+0000 7f75b9790640 1 -- 172.21.15.28:0/2849900424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b41a4b50 con 0x7f75b4069dc0 2024-04-25T01:39:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.212+0000 7f75b0ff9640 1 -- 172.21.15.28:0/2849900424 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f75a4004fa0 con 0x7f75b4069dc0 2024-04-25T01:39:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.212+0000 7f75bc57d740 1 Processor -- start 2024-04-25T01:39:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.212+0000 7f75bc57d740 1 -- start start 2024-04-25T01:39:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.212+0000 7f75bc57d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659360 0x56206a659740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.212+0000 7f75bc57d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a659d10 0x56206a6684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.212+0000 7f75bc57d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a668a30 0x56206a66ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.212+0000 7f75bc57d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56206a5be6b0 con 0x56206a659360 2024-04-25T01:39:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.212+0000 7f75bc57d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56206a566880 con 0x56206a659d10 2024-04-25T01:39:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.212+0000 7f75bc57d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56206a5ba670 con 0x56206a668a30 2024-04-25T01:39:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.212+0000 7f75b0ff9640 1 --2- 172.21.15.28:0/2849900424 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f75a00a2d00 0x7f75a00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.212+0000 7f75b2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a659d10 0x56206a6684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.212+0000 7f75b37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659360 0x56206a659740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.212+0000 7f75abfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a668a30 0x56206a66ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.213+0000 7f75abfff640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a668a30 0x56206a66ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.28:54856/0 (socket says 172.21.15.28:54856) 2024-04-25T01:39:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.213+0000 7f75b37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659360 0x56206a659740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48682/0 (socket says 172.21.15.28:48682) 2024-04-25T01:39:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.213+0000 7f75abfff640 1 -- 172.21.15.28:0/1679619769 learned_addr learned my addr 172.21.15.28:0/1679619769 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.213+0000 7f75b2ffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a659d10 0x56206a6684f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:42326/0 (socket says 172.21.15.28:42326) 2024-04-25T01:39:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.213+0000 7f75b0ff9640 1 -- 172.21.15.28:0/2849900424 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f75a40cf200 con 0x7f75b4069dc0 2024-04-25T01:39:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.213+0000 7f75b37fe640 1 --2- 172.21.15.28:0/2849900424 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f75a00a2d00 0x7f75a00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.213+0000 7f75b2ffd640 1 -- 172.21.15.28:0/1679619769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a668a30 msgr2=0x56206a66ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.213+0000 7f75b2ffd640 1 --2- 172.21.15.28:0/1679619769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a668a30 0x56206a66ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.213+0000 7f75b2ffd640 1 -- 172.21.15.28:0/1679619769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659360 msgr2=0x56206a659740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.213+0000 7f75b2ffd640 1 --2- 172.21.15.28:0/1679619769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659360 0x56206a659740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.213+0000 7f75b2ffd640 1 -- 172.21.15.28:0/1679619769 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56206a39c7d0 con 0x56206a659d10 2024-04-25T01:39:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.213+0000 7f75abfff640 1 --2- 172.21.15.28:0/1679619769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a668a30 0x56206a66ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:39:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.213+0000 7f75b37fe640 1 --2- 172.21.15.28:0/1679619769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659360 0x56206a659740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:39:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.213+0000 7f75b37fe640 1 --2- 172.21.15.28:0/2849900424 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f75a00a2d00 0x7f75a00a51c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f75ac009fd0 tx=0x7f75ac004a40 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.213+0000 7f75b2ffd640 1 --2- 172.21.15.28:0/1679619769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a659d10 0x56206a6684f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f75a40cc5e0 tx=0x7f75a40c5f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e64387239daff917 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.213+0000 7f75a97fa640 1 -- 172.21.15.28:0/1679619769 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a4020020 con 0x56206a659d10 2024-04-25T01:39:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.213+0000 7f75a97fa640 1 -- 172.21.15.28:0/1679619769 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75a4066470 con 0x56206a659d10 2024-04-25T01:39:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.213+0000 7f75a97fa640 1 -- 172.21.15.28:0/1679619769 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a4009cc0 con 0x56206a659d10 2024-04-25T01:39:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.214+0000 7f75bc57d740 1 -- 172.21.15.28:0/1679619769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a659d10 msgr2=0x56206a6684f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.214+0000 7f75bc57d740 1 --2- 172.21.15.28:0/1679619769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a659d10 0x56206a6684f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f75a40cc5e0 tx=0x7f75a40c5f70 comp rx=0 tx=0).stop 2024-04-25T01:39:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.214+0000 7f75bc57d740 1 -- 172.21.15.28:0/1679619769 shutdown_connections 2024-04-25T01:39:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.214+0000 7f75bc57d740 1 --2- 172.21.15.28:0/1679619769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a668a30 0x56206a66ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.214+0000 7f75bc57d740 1 --2- 172.21.15.28:0/1679619769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a659d10 0x56206a6684f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.214+0000 7f75bc57d740 1 --2- 172.21.15.28:0/1679619769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659360 0x56206a659740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.214+0000 7f75bc57d740 1 -- 172.21.15.28:0/1679619769 >> 172.21.15.28:0/1679619769 conn(0x56206a655950 msgr2=0x56206a655d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.214+0000 7f75bc57d740 1 -- 172.21.15.28:0/1679619769 shutdown_connections 2024-04-25T01:39:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.214+0000 7f75bc57d740 1 -- 172.21.15.28:0/1679619769 wait complete. 2024-04-25T01:39:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.214+0000 7f75bc57d740 1 -- start start 2024-04-25T01:39:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f75bc57d740 1 Processor -- start 2024-04-25T01:39:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f75bc57d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a659360 0x56206a665aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f75bc57d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a659d10 0x56206a665fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f75bc57d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a668a30 0x56206a666520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f75bc57d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56206a66b790 con 0x56206a668a30 2024-04-25T01:39:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f75bc57d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56206a579f50 con 0x56206a659360 2024-04-25T01:39:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f75bc57d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56206a66b910 con 0x56206a659d10 2024-04-25T01:39:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f75bc57d740 10 client.0 _ll_register_callbacks cb 0x56206a5b5000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f75b37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a668a30 0x56206a666520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f75b37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a668a30 0x56206a666520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48698/0 (socket says 172.21.15.28:48698) 2024-04-25T01:39:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f75b37fe640 1 -- 172.21.15.28:0/1975565519 learned_addr learned my addr 172.21.15.28:0/1975565519 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f75b2ffd640 1 --2- 172.21.15.28:0/1975565519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a659360 0x56206a665aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f75b37fe640 1 -- 172.21.15.28:0/1975565519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a659d10 msgr2=0x56206a665fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f75b37fe640 1 --2- 172.21.15.28:0/1975565519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a659d10 0x56206a665fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f75b37fe640 1 -- 172.21.15.28:0/1975565519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a659360 msgr2=0x56206a665aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f75b37fe640 1 --2- 172.21.15.28:0/1975565519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a659360 0x56206a665aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f75b37fe640 1 -- 172.21.15.28:0/1975565519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56206a2d23f0 con 0x56206a668a30 2024-04-25T01:39:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f75b37fe640 1 --2- 172.21.15.28:0/1975565519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a668a30 0x56206a666520 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f75ac011ee0 tx=0x7f75ac004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d80e272db229dbf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f759a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f759a7fc640 1 -- 172.21.15.28:0/1975565519 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ac024030 con 0x56206a668a30 2024-04-25T01:39:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.215+0000 7f759a7fc640 1 -- 172.21.15.28:0/1975565519 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f75ac0139c0 con 0x56206a668a30 2024-04-25T01:39:13.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.216+0000 7f75bc57d740 1 -- 172.21.15.28:0/1975565519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75a4066470 con 0x56206a668a30 2024-04-25T01:39:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.216+0000 7f759a7fc640 1 -- 172.21.15.28:0/1975565519 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f75ac031050 con 0x56206a668a30 2024-04-25T01:39:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.216+0000 7f75bc57d740 10 client.17703 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.216+0000 7f75bc57d740 1 -- 172.21.15.28:0/1975565519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56206a2920f0 con 0x56206a668a30 2024-04-25T01:39:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.216+0000 7f759a7fc640 1 -- 172.21.15.28:0/1975565519 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75a4066470 con 0x56206a668a30 2024-04-25T01:39:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.216+0000 7f759a7fc640 1 -- 172.21.15.28:0/1975565519 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f75ac020ba0 con 0x56206a668a30 2024-04-25T01:39:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.216+0000 7f75bc57d740 10 client.17703 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.216+0000 7f75bc57d740 10 client.17703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.216+0000 7f75bc57d740 20 client.17703 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.216+0000 7f75bc57d740 1 --2- 172.21.15.28:0/1975565519 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56206a678e30 0x56206a67b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.216+0000 7f75bc57d740 4 client.17703 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.216+0000 7f75bc57d740 1 -- 172.21.15.28:0/1975565519 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56206a5cbbf0 con 0x56206a678e30 2024-04-25T01:39:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.216+0000 7f75b2ffd640 1 --2- 172.21.15.28:0/1975565519 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56206a678e30 0x56206a67b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.217+0000 7f759a7fc640 10 client.17703 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.217+0000 7f75b2ffd640 1 --2- 172.21.15.28:0/1975565519 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56206a678e30 0x56206a67b2d0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=898fcc3e59171c91 server_cookie=dac2c7e7689cb04d in_seq=0 out_seq=0 2024-04-25T01:39:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.217+0000 7f759a7fc640 1 -- 172.21.15.28:0/1975565519 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56206a5cbbf0 con 0x56206a678e30 2024-04-25T01:39:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.217+0000 7f759a7fc640 10 client.17703 handle_command_reply: tid=0 2024-04-25T01:39:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.217+0000 7f75bc57d740 1 client.17703 shutdown 2024-04-25T01:39:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.217+0000 7f75bc57d740 1 -- 172.21.15.28:0/1975565519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a668a30 msgr2=0x56206a666520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.217+0000 7f75bc57d740 1 --2- 172.21.15.28:0/1975565519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a668a30 0x56206a666520 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f75ac011ee0 tx=0x7f75ac004ab0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.218+0000 7f75bc57d740 1 -- 172.21.15.28:0/1975565519 shutdown_connections 2024-04-25T01:39:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.218+0000 7f75bc57d740 1 --2- 172.21.15.28:0/1975565519 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56206a678e30 0x56206a67b2d0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.218+0000 7f75bc57d740 1 --2- 172.21.15.28:0/1975565519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a668a30 0x56206a666520 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.218+0000 7f75bc57d740 1 --2- 172.21.15.28:0/1975565519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a659d10 0x56206a665fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.218+0000 7f759a7fc640 0 client.17703 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:13.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.218+0000 7f75bc57d740 1 --2- 172.21.15.28:0/1975565519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a659360 0x56206a665aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.218+0000 7f75bc57d740 1 -- 172.21.15.28:0/1975565519 >> 172.21.15.28:0/1975565519 conn(0x56206a655950 msgr2=0x56206a65b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.218+0000 7f75bc57d740 1 -- 172.21.15.28:0/1975565519 shutdown_connections 2024-04-25T01:39:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.218+0000 7f75bc57d740 1 -- 172.21.15.28:0/1975565519 wait complete. 2024-04-25T01:39:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.218+0000 7f75bc57d740 20 client.17703 trim_cache size 0 max 16384 2024-04-25T01:39:13.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.235+0000 7f75bc57d740 1 Processor -- start 2024-04-25T01:39:13.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.235+0000 7f75bc57d740 1 -- start start 2024-04-25T01:39:13.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.235+0000 7f75bc57d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a655950 0x56206a6681d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.235+0000 7f75bc57d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659360 0x56206a674690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.235+0000 7f75bc57d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a659d10 0x56206a680ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.235+0000 7f75bc57d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75b41ac8b0 con 0x56206a659360 2024-04-25T01:39:13.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.235+0000 7f75bc57d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f75b41ae3f0 con 0x56206a659d10 2024-04-25T01:39:13.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.235+0000 7f75bc57d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f75b41acf80 con 0x56206a655950 2024-04-25T01:39:13.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.236+0000 7f75abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659360 0x56206a674690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.236+0000 7f75abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659360 0x56206a674690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48704/0 (socket says 172.21.15.28:48704) 2024-04-25T01:39:13.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.236+0000 7f75abfff640 1 -- 172.21.15.28:0/4273884158 learned_addr learned my addr 172.21.15.28:0/4273884158 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.236+0000 7f75b37fe640 1 --2- 172.21.15.28:0/4273884158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a659d10 0x56206a680ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.236+0000 7f75b2ffd640 1 --2- 172.21.15.28:0/4273884158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a655950 0x56206a6681d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.236+0000 7f75abfff640 1 -- 172.21.15.28:0/4273884158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a655950 msgr2=0x56206a6681d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.236+0000 7f75abfff640 1 --2- 172.21.15.28:0/4273884158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a655950 0x56206a6681d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.236+0000 7f75abfff640 1 -- 172.21.15.28:0/4273884158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a659d10 msgr2=0x56206a680ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.236+0000 7f75abfff640 1 --2- 172.21.15.28:0/4273884158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a659d10 0x56206a680ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.236+0000 7f75abfff640 1 -- 172.21.15.28:0/4273884158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56206a647090 con 0x56206a659360 2024-04-25T01:39:13.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.236+0000 7f75abfff640 1 --2- 172.21.15.28:0/4273884158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659360 0x56206a674690 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f759c009900 tx=0x7f759c005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=331bc75b7d0c2aac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.236+0000 7f75a8ff9640 1 -- 172.21.15.28:0/4273884158 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759c035040 con 0x56206a659360 2024-04-25T01:39:13.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.236+0000 7f75a8ff9640 1 -- 172.21.15.28:0/4273884158 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f759c004040 con 0x56206a659360 2024-04-25T01:39:13.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.236+0000 7f75a8ff9640 1 -- 172.21.15.28:0/4273884158 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759c005560 con 0x56206a659360 2024-04-25T01:39:13.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.237+0000 7f75bc57d740 1 -- 172.21.15.28:0/4273884158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659360 msgr2=0x56206a674690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.237+0000 7f75bc57d740 1 --2- 172.21.15.28:0/4273884158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659360 0x56206a674690 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f759c009900 tx=0x7f759c005e70 comp rx=0 tx=0).stop 2024-04-25T01:39:13.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.237+0000 7f75bc57d740 1 -- 172.21.15.28:0/4273884158 shutdown_connections 2024-04-25T01:39:13.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.237+0000 7f75bc57d740 1 --2- 172.21.15.28:0/4273884158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a659d10 0x56206a680ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.237+0000 7f75bc57d740 1 --2- 172.21.15.28:0/4273884158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659360 0x56206a674690 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.237+0000 7f75bc57d740 1 --2- 172.21.15.28:0/4273884158 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a655950 0x56206a6681d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.237+0000 7f75bc57d740 1 -- 172.21.15.28:0/4273884158 >> 172.21.15.28:0/4273884158 conn(0x56206a6646e0 msgr2=0x56206a654af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.237+0000 7f75bc57d740 1 -- 172.21.15.28:0/4273884158 shutdown_connections 2024-04-25T01:39:13.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.237+0000 7f75bc57d740 1 -- 172.21.15.28:0/4273884158 wait complete. 2024-04-25T01:39:13.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.237+0000 7f75bc57d740 1 -- start start 2024-04-25T01:39:13.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.237+0000 7f75bc57d740 1 Processor -- start 2024-04-25T01:39:13.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f75bc57d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a655950 0x56206a66c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f75bc57d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a659360 0x56206a66ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f75bc57d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659d10 0x56206a6de410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f75bc57d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75b41add20 con 0x56206a659d10 2024-04-25T01:39:13.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f75bc57d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f75b41ad650 con 0x56206a655950 2024-04-25T01:39:13.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f75bc57d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f75b41aa890 con 0x56206a659360 2024-04-25T01:39:13.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f75bc57d740 10 client.0 _ll_register_callbacks cb 0x7f75ac009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:13.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f75b37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659d10 0x56206a6de410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f75b37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659d10 0x56206a6de410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48706/0 (socket says 172.21.15.28:48706) 2024-04-25T01:39:13.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f75b37fe640 1 -- 172.21.15.28:0/758102865 learned_addr learned my addr 172.21.15.28:0/758102865 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f75b2ffd640 1 --2- 172.21.15.28:0/758102865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a655950 0x56206a66c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f75abfff640 1 --2- 172.21.15.28:0/758102865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a659360 0x56206a66ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f75b37fe640 1 -- 172.21.15.28:0/758102865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a659360 msgr2=0x56206a66ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f75b37fe640 1 --2- 172.21.15.28:0/758102865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a659360 0x56206a66ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f75b37fe640 1 -- 172.21.15.28:0/758102865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a655950 msgr2=0x56206a66c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f75b37fe640 1 --2- 172.21.15.28:0/758102865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a655950 0x56206a66c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f75b37fe640 1 -- 172.21.15.28:0/758102865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56206a66d6e0 con 0x56206a659d10 2024-04-25T01:39:13.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f75b37fe640 1 --2- 172.21.15.28:0/758102865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659d10 0x56206a6de410 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f75ac02b520 tx=0x7f75ac014780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1d3886d4613394 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f759a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:13.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f759a7fc640 1 -- 172.21.15.28:0/758102865 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ac039070 con 0x56206a659d10 2024-04-25T01:39:13.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f759a7fc640 1 -- 172.21.15.28:0/758102865 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f75ac004b20 con 0x56206a659d10 2024-04-25T01:39:13.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.238+0000 7f75bc57d740 1 -- 172.21.15.28:0/758102865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f759c004040 con 0x56206a659d10 2024-04-25T01:39:13.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.239+0000 7f759a7fc640 1 -- 172.21.15.28:0/758102865 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f75ac044050 con 0x56206a659d10 2024-04-25T01:39:13.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.239+0000 7f75bc57d740 10 client.17709 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:13.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.239+0000 7f75bc57d740 1 -- 172.21.15.28:0/758102865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56206a6dfa30 con 0x56206a659d10 2024-04-25T01:39:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.239+0000 7f759a7fc640 1 -- 172.21.15.28:0/758102865 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f759c004040 con 0x56206a659d10 2024-04-25T01:39:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.239+0000 7f759a7fc640 1 -- 172.21.15.28:0/758102865 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f75ac034040 con 0x56206a659d10 2024-04-25T01:39:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.239+0000 7f75bc57d740 10 client.17709 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.239+0000 7f75bc57d740 10 client.17709 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.239+0000 7f75bc57d740 20 client.17709 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.239+0000 7f75bc57d740 1 --2- 172.21.15.28:0/758102865 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56206a668a30 0x56206a6e2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.239+0000 7f75bc57d740 4 client.17709 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:39:13.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.239+0000 7f75bc57d740 1 -- 172.21.15.28:0/758102865 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56206a5c5f40 con 0x56206a668a30 2024-04-25T01:39:13.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.239+0000 7f75b2ffd640 1 --2- 172.21.15.28:0/758102865 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56206a668a30 0x56206a6e2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.240+0000 7f75b2ffd640 1 --2- 172.21.15.28:0/758102865 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56206a668a30 0x56206a6e2100 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ddbde86c46edf4f server_cookie=4cc9dd4afe98ae0b in_seq=0 out_seq=0 2024-04-25T01:39:13.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.240+0000 7f759a7fc640 10 client.17709 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:13.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.240+0000 7f759a7fc640 1 -- 172.21.15.28:0/758102865 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56206a5c5f40 con 0x56206a668a30 2024-04-25T01:39:13.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.240+0000 7f759a7fc640 10 client.17709 handle_command_reply: tid=0 2024-04-25T01:39:13.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.240+0000 7f75bc57d740 1 client.17709 shutdown 2024-04-25T01:39:13.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.240+0000 7f75bc57d740 1 -- 172.21.15.28:0/758102865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659d10 msgr2=0x56206a6de410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.240+0000 7f75bc57d740 1 --2- 172.21.15.28:0/758102865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659d10 0x56206a6de410 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f75ac02b520 tx=0x7f75ac014780 comp rx=0 tx=0).stop 2024-04-25T01:39:13.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.240+0000 7f75bc57d740 1 -- 172.21.15.28:0/758102865 shutdown_connections 2024-04-25T01:39:13.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.240+0000 7f75bc57d740 1 --2- 172.21.15.28:0/758102865 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56206a668a30 0x56206a6e2100 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.240+0000 7f75bc57d740 1 --2- 172.21.15.28:0/758102865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56206a659d10 0x56206a6de410 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.240+0000 7f75bc57d740 1 --2- 172.21.15.28:0/758102865 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56206a659360 0x56206a66ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.240+0000 7f75bc57d740 1 --2- 172.21.15.28:0/758102865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56206a655950 0x56206a66c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.240+0000 7f759a7fc640 0 client.17709 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:13.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.240+0000 7f75bc57d740 1 -- 172.21.15.28:0/758102865 >> 172.21.15.28:0/758102865 conn(0x56206a6646e0 msgr2=0x56206a665a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.240+0000 7f75bc57d740 1 -- 172.21.15.28:0/758102865 shutdown_connections 2024-04-25T01:39:13.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.240+0000 7f75bc57d740 1 -- 172.21.15.28:0/758102865 wait complete. 2024-04-25T01:39:13.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.240+0000 7f75bc57d740 20 client.17709 trim_cache size 0 max 16384 2024-04-25T01:39:13.240 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:39:13.240 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:39:13.240 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:39:13.240 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:39:13.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.241+0000 7f75b9790640 1 -- 172.21.15.28:0/2849900424 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f75a00a2d00 msgr2=0x7f75a00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.241+0000 7f75b9790640 1 --2- 172.21.15.28:0/2849900424 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f75a00a2d00 0x7f75a00a51c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f75ac009fd0 tx=0x7f75ac004a40 comp rx=0 tx=0).stop 2024-04-25T01:39:13.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.241+0000 7f75b9790640 1 -- 172.21.15.28:0/2849900424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75b4069dc0 msgr2=0x7f75b4197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.241+0000 7f75b9790640 1 --2- 172.21.15.28:0/2849900424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75b4069dc0 0x7f75b4197fe0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f75a400d770 tx=0x7f75a400dc30 comp rx=0 tx=0).stop 2024-04-25T01:39:13.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.241+0000 7f75b9790640 1 -- 172.21.15.28:0/2849900424 shutdown_connections 2024-04-25T01:39:13.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.241+0000 7f75b9790640 1 --2- 172.21.15.28:0/2849900424 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f75a00a2d00 0x7f75a00a51c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.241+0000 7f75b9790640 1 --2- 172.21.15.28:0/2849900424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f75b4107730 0x7f75b419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.241+0000 7f75b9790640 1 --2- 172.21.15.28:0/2849900424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f75b406a780 0x7f75b4198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.241+0000 7f75b9790640 1 --2- 172.21.15.28:0/2849900424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75b4069dc0 0x7f75b4197fe0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.241+0000 7f75b9790640 1 -- 172.21.15.28:0/2849900424 >> 172.21.15.28:0/2849900424 conn(0x7f75b40776a0 msgr2=0x7f75b40fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.241+0000 7f75b9790640 1 -- 172.21.15.28:0/2849900424 shutdown_connections 2024-04-25T01:39:13.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.241+0000 7f75b9790640 1 -- 172.21.15.28:0/2849900424 wait complete. 2024-04-25T01:39:13.242 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:39:13.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:39:13.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.363+0000 7f610bb34640 1 -- 172.21.15.28:0/3681117614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610410bd60 msgr2=0x7f610410e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.363+0000 7f610bb34640 1 --2- 172.21.15.28:0/3681117614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610410bd60 0x7f610410e1d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f60fc009ca0 tx=0x7f60fc02a950 comp rx=0 tx=0).stop 2024-04-25T01:39:13.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.363+0000 7f610bb34640 1 -- 172.21.15.28:0/3681117614 shutdown_connections 2024-04-25T01:39:13.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.363+0000 7f610bb34640 1 --2- 172.21.15.28:0/3681117614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610410bd60 0x7f610410e1d0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.363+0000 7f610bb34640 1 --2- 172.21.15.28:0/3681117614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6104103340 0x7f610410b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.363+0000 7f610bb34640 1 --2- 172.21.15.28:0/3681117614 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6104102a20 0x7f6104102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.363+0000 7f610bb34640 1 -- 172.21.15.28:0/3681117614 >> 172.21.15.28:0/3681117614 conn(0x7f61040777a0 msgr2=0x7f6104077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.363+0000 7f610bb34640 1 -- 172.21.15.28:0/3681117614 shutdown_connections 2024-04-25T01:39:13.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.363+0000 7f610bb34640 1 -- 172.21.15.28:0/3681117614 wait complete. 2024-04-25T01:39:13.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.364+0000 7f610bb34640 1 Processor -- start 2024-04-25T01:39:13.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.364+0000 7f610bb34640 1 -- start start 2024-04-25T01:39:13.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.364+0000 7f610bb34640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6104102a20 0x7f6104197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.364+0000 7f610bb34640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6104103340 0x7f61041984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.364+0000 7f610bb34640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610410bd60 0x7f610419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.364+0000 7f610bb34640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6104111b30 con 0x7f610410bd60 2024-04-25T01:39:13.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.364+0000 7f610bb34640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f61041119b0 con 0x7f6104103340 2024-04-25T01:39:13.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.364+0000 7f610bb34640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6104111cb0 con 0x7f6104102a20 2024-04-25T01:39:13.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.364+0000 7f610a0d1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610410bd60 0x7f610419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.364+0000 7f610a0d1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610410bd60 0x7f610419c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48724/0 (socket says 172.21.15.28:48724) 2024-04-25T01:39:13.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.364+0000 7f610a0d1640 1 -- 172.21.15.28:0/433263823 learned_addr learned my addr 172.21.15.28:0/433263823 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.364+0000 7f61098d0640 1 --2- 172.21.15.28:0/433263823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6104102a20 0x7f6104197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.364+0000 7f61090cf640 1 --2- 172.21.15.28:0/433263823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6104103340 0x7f61041984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.364+0000 7f610a0d1640 1 -- 172.21.15.28:0/433263823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6104102a20 msgr2=0x7f6104197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.364+0000 7f610a0d1640 1 --2- 172.21.15.28:0/433263823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6104102a20 0x7f6104197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.364+0000 7f610a0d1640 1 -- 172.21.15.28:0/433263823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6104103340 msgr2=0x7f61041984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.364+0000 7f610a0d1640 1 --2- 172.21.15.28:0/433263823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6104103340 0x7f61041984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.364+0000 7f610a0d1640 1 -- 172.21.15.28:0/433263823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610419cfa0 con 0x7f610410bd60 2024-04-25T01:39:13.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.365+0000 7f610a0d1640 1 --2- 172.21.15.28:0/433263823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610410bd60 0x7f610419c870 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f60fc0098d0 tx=0x7f60fc031f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.365+0000 7f60faffd640 1 -- 172.21.15.28:0/433263823 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60fc034040 con 0x7f610410bd60 2024-04-25T01:39:13.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.365+0000 7f60faffd640 1 -- 172.21.15.28:0/433263823 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60fc02f730 con 0x7f610410bd60 2024-04-25T01:39:13.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.365+0000 7f610bb34640 1 -- 172.21.15.28:0/433263823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f610419d230 con 0x7f610410bd60 2024-04-25T01:39:13.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.365+0000 7f60faffd640 1 -- 172.21.15.28:0/433263823 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60fc004600 con 0x7f610410bd60 2024-04-25T01:39:13.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.365+0000 7f610bb34640 1 -- 172.21.15.28:0/433263823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61041a4b20 con 0x7f610410bd60 2024-04-25T01:39:13.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.366+0000 7f610bb34640 1 -- 172.21.15.28:0/433263823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61040689b0 con 0x7f610410bd60 2024-04-25T01:39:13.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.367+0000 7f60faffd640 1 -- 172.21.15.28:0/433263823 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f60fc037020 con 0x7f610410bd60 2024-04-25T01:39:13.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.368+0000 7f60faffd640 1 --2- 172.21.15.28:0/433263823 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f60f00a2d00 0x7f60f00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.368+0000 7f61098d0640 1 --2- 172.21.15.28:0/433263823 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f60f00a2d00 0x7f60f00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.368+0000 7f60faffd640 1 -- 172.21.15.28:0/433263823 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f60fc0f0250 con 0x7f610410bd60 2024-04-25T01:39:13.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.368+0000 7f61098d0640 1 --2- 172.21.15.28:0/433263823 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f60f00a2d00 0x7f60f00a51c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6100009bd0 tx=0x7f6100005f90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.373+0000 7f60faffd640 1 -- 172.21.15.28:0/433263823 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f60fc0bcc40 con 0x7f610410bd60 2024-04-25T01:39:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.597+0000 7f610bb34640 1 -- 172.21.15.28:0/433263823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6104106ee0 con 0x7f610410bd60 2024-04-25T01:39:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.598+0000 7f60faffd640 1 -- 172.21.15.28:0/433263823 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7f60fc0bc540 con 0x7f610410bd60 2024-04-25T01:39:13.580 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:39:13.580 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:39:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:39:13.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.600+0000 7f610bb34640 1 -- 172.21.15.28:0/433263823 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f60f00a2d00 msgr2=0x7f60f00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.600+0000 7f610bb34640 1 --2- 172.21.15.28:0/433263823 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f60f00a2d00 0x7f60f00a51c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6100009bd0 tx=0x7f6100005f90 comp rx=0 tx=0).stop 2024-04-25T01:39:13.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.600+0000 7f610bb34640 1 -- 172.21.15.28:0/433263823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610410bd60 msgr2=0x7f610419c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.600+0000 7f610bb34640 1 --2- 172.21.15.28:0/433263823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610410bd60 0x7f610419c870 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f60fc0098d0 tx=0x7f60fc031f80 comp rx=0 tx=0).stop 2024-04-25T01:39:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.600+0000 7f610bb34640 1 -- 172.21.15.28:0/433263823 shutdown_connections 2024-04-25T01:39:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.600+0000 7f610bb34640 1 --2- 172.21.15.28:0/433263823 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f60f00a2d00 0x7f60f00a51c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.600+0000 7f610bb34640 1 --2- 172.21.15.28:0/433263823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f610410bd60 0x7f610419c870 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.600+0000 7f610bb34640 1 --2- 172.21.15.28:0/433263823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6104103340 0x7f61041984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.600+0000 7f610bb34640 1 --2- 172.21.15.28:0/433263823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6104102a20 0x7f6104197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.600+0000 7f610bb34640 1 -- 172.21.15.28:0/433263823 >> 172.21.15.28:0/433263823 conn(0x7f61040777a0 msgr2=0x7f61040fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.600+0000 7f610bb34640 1 -- 172.21.15.28:0/433263823 shutdown_connections 2024-04-25T01:39:13.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.600+0000 7f610bb34640 1 -- 172.21.15.28:0/433263823 wait complete. 2024-04-25T01:39:13.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:39:13.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.715+0000 7fe3f725d640 1 -- 172.21.15.28:0/297417832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0107720 msgr2=0x7fe3f0107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.715+0000 7fe3f725d640 1 --2- 172.21.15.28:0/297417832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0107720 0x7fe3f0107b00 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0009930 tx=0x7fe3e002a420 comp rx=0 tx=0).stop 2024-04-25T01:39:13.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.715+0000 7fe3f725d640 1 -- 172.21.15.28:0/297417832 shutdown_connections 2024-04-25T01:39:13.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.715+0000 7fe3f725d640 1 --2- 172.21.15.28:0/297417832 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3f01020d0 0x7fe3f010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.715+0000 7fe3f725d640 1 --2- 172.21.15.28:0/297417832 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3f0101710 0x7fe3f0101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.715+0000 7fe3f725d640 1 --2- 172.21.15.28:0/297417832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0107720 0x7fe3f0107b00 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.715+0000 7fe3f725d640 1 -- 172.21.15.28:0/297417832 >> 172.21.15.28:0/297417832 conn(0x7fe3f0078b90 msgr2=0x7fe3f0078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.715+0000 7fe3f725d640 1 -- 172.21.15.28:0/297417832 shutdown_connections 2024-04-25T01:39:13.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.715+0000 7fe3f725d640 1 -- 172.21.15.28:0/297417832 wait complete. 2024-04-25T01:39:13.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3f725d640 1 Processor -- start 2024-04-25T01:39:13.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3f725d640 1 -- start start 2024-04-25T01:39:13.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3f725d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0101710 0x7fe3f0198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3f725d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3f01020d0 0x7fe3f0198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3f725d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3f0107720 0x7fe3f019c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3f725d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f0111af0 con 0x7fe3f0101710 2024-04-25T01:39:13.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3f725d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f0111970 con 0x7fe3f0107720 2024-04-25T01:39:13.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3f725d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f0111c70 con 0x7fe3f01020d0 2024-04-25T01:39:13.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3f4ff9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0101710 0x7fe3f0198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3f4ff9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0101710 0x7fe3f0198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48736/0 (socket says 172.21.15.28:48736) 2024-04-25T01:39:13.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3f4ff9640 1 -- 172.21.15.28:0/946671860 learned_addr learned my addr 172.21.15.28:0/946671860 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3effff640 1 --2- 172.21.15.28:0/946671860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3f01020d0 0x7fe3f0198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3f4ff9640 1 -- 172.21.15.28:0/946671860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3f01020d0 msgr2=0x7fe3f0198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3f57fa640 1 --2- 172.21.15.28:0/946671860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3f0107720 0x7fe3f019c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3f4ff9640 1 --2- 172.21.15.28:0/946671860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3f01020d0 0x7fe3f0198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3f4ff9640 1 -- 172.21.15.28:0/946671860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3f0107720 msgr2=0x7fe3f019c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3f4ff9640 1 --2- 172.21.15.28:0/946671860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3f0107720 0x7fe3f019c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3f4ff9640 1 -- 172.21.15.28:0/946671860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f019d040 con 0x7fe3f0101710 2024-04-25T01:39:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3effff640 1 --2- 172.21.15.28:0/946671860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3f01020d0 0x7fe3f0198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3f57fa640 1 --2- 172.21.15.28:0/946671860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3f0107720 0x7fe3f019c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.716+0000 7fe3f4ff9640 1 --2- 172.21.15.28:0/946671860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0101710 0x7fe3f0198050 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0009230 tx=0x7fe3e0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.717+0000 7fe3edffb640 1 -- 172.21.15.28:0/946671860 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e003f030 con 0x7fe3f0101710 2024-04-25T01:39:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.717+0000 7fe3edffb640 1 -- 172.21.15.28:0/946671860 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3e0003ad0 con 0x7fe3f0101710 2024-04-25T01:39:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.717+0000 7fe3f725d640 1 -- 172.21.15.28:0/946671860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3f019d2d0 con 0x7fe3f0101710 2024-04-25T01:39:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.717+0000 7fe3edffb640 1 -- 172.21.15.28:0/946671860 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0035070 con 0x7fe3f0101710 2024-04-25T01:39:13.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.717+0000 7fe3f725d640 1 -- 172.21.15.28:0/946671860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3f01a4bc0 con 0x7fe3f0101710 2024-04-25T01:39:13.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.719+0000 7fe3edffb640 1 -- 172.21.15.28:0/946671860 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe3e0042030 con 0x7fe3f0101710 2024-04-25T01:39:13.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.719+0000 7fe3f9f6f740 1 Processor -- start 2024-04-25T01:39:13.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.719+0000 7fe3f9f6f740 1 -- start start 2024-04-25T01:39:13.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3f9f6f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558864572580 0x558864572960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3f9f6f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572ea0 0x55886457d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3f9f6f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55886457d570 0x55886457f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3edffb640 1 --2- 172.21.15.28:0/946671860 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe3d80a2d00 0x7fe3d80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3f9f6f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5588644d7670 con 0x558864572ea0 2024-04-25T01:39:13.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3f9f6f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55886447f970 con 0x55886457d570 2024-04-25T01:39:13.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3f9f6f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5588644d3630 con 0x558864572580 2024-04-25T01:39:13.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3f57fa640 1 --2- 172.21.15.28:0/946671860 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe3d80a2d00 0x7fe3d80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3f4ff9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572ea0 0x55886457d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3f4ff9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572ea0 0x55886457d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48738/0 (socket says 172.21.15.28:48738) 2024-04-25T01:39:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3f4ff9640 1 -- 172.21.15.28:0/1587914672 learned_addr learned my addr 172.21.15.28:0/1587914672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3f57fa640 1 --2- 172.21.15.28:0/1587914672 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558864572580 0x558864572960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3edffb640 1 -- 172.21.15.28:0/946671860 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fe3e00efee0 con 0x7fe3f0101710 2024-04-25T01:39:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3effff640 1 --2- 172.21.15.28:0/1587914672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55886457d570 0x55886457f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3f4ff9640 1 -- 172.21.15.28:0/1587914672 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558864572580 msgr2=0x558864572960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3f4ff9640 1 --2- 172.21.15.28:0/1587914672 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558864572580 0x558864572960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3f4ff9640 1 -- 172.21.15.28:0/1587914672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55886457d570 msgr2=0x55886457f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3f4ff9640 1 --2- 172.21.15.28:0/1587914672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55886457d570 0x55886457f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3f4ff9640 1 -- 172.21.15.28:0/1587914672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588644a3400 con 0x558864572ea0 2024-04-25T01:39:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3effff640 1 --2- 172.21.15.28:0/1587914672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55886457d570 0x55886457f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3f57fa640 1 --2- 172.21.15.28:0/946671860 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe3d80a2d00 0x7fe3d80a51c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8005c40 tx=0x7fe3e8009cd0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.720+0000 7fe3f4ff9640 1 --2- 172.21.15.28:0/1587914672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572ea0 0x55886457d030 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fe3e00bbd60 tx=0x7fe3e008e020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0091da6ecd27d91 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.721+0000 7fe3d6ffd640 1 -- 172.21.15.28:0/1587914672 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0049070 con 0x558864572ea0 2024-04-25T01:39:13.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.721+0000 7fe3d6ffd640 1 -- 172.21.15.28:0/1587914672 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3e00870f0 con 0x558864572ea0 2024-04-25T01:39:13.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.721+0000 7fe3d6ffd640 1 -- 172.21.15.28:0/1587914672 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e00b86d0 con 0x558864572ea0 2024-04-25T01:39:13.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.721+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/1587914672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572ea0 msgr2=0x55886457d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.721+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/1587914672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572ea0 0x55886457d030 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fe3e00bbd60 tx=0x7fe3e008e020 comp rx=0 tx=0).stop 2024-04-25T01:39:13.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.721+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/1587914672 shutdown_connections 2024-04-25T01:39:13.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.721+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/1587914672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55886457d570 0x55886457f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.721+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/1587914672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572ea0 0x55886457d030 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.721+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/1587914672 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558864572580 0x558864572960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.721+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/1587914672 >> 172.21.15.28:0/1587914672 conn(0x55886456e8a0 msgr2=0x55886456ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.721+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/1587914672 shutdown_connections 2024-04-25T01:39:13.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.721+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/1587914672 wait complete. 2024-04-25T01:39:13.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.722+0000 7fe3f9f6f740 1 -- start start 2024-04-25T01:39:13.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.722+0000 7fe3f9f6f740 1 Processor -- start 2024-04-25T01:39:13.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.722+0000 7fe3f9f6f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572580 0x55886457cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.722+0000 7fe3f9f6f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558864572ea0 0x55886457a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.722+0000 7fe3f9f6f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55886457d570 0x55886457a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.722+0000 7fe3f9f6f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558864580120 con 0x558864572580 2024-04-25T01:39:13.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.722+0000 7fe3f9f6f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558864493050 con 0x558864572ea0 2024-04-25T01:39:13.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.722+0000 7fe3f9f6f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5588645802a0 con 0x55886457d570 2024-04-25T01:39:13.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.722+0000 7fe3f9f6f740 10 client.0 _ll_register_callbacks cb 0x5588641eabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:13.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.722+0000 7fe3f4ff9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572580 0x55886457cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.722+0000 7fe3f4ff9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572580 0x55886457cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48748/0 (socket says 172.21.15.28:48748) 2024-04-25T01:39:13.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.722+0000 7fe3f4ff9640 1 -- 172.21.15.28:0/44454361 learned_addr learned my addr 172.21.15.28:0/44454361 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3effff640 1 --2- 172.21.15.28:0/44454361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558864572ea0 0x55886457a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3f57fa640 1 --2- 172.21.15.28:0/44454361 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55886457d570 0x55886457a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3f4ff9640 1 -- 172.21.15.28:0/44454361 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55886457d570 msgr2=0x55886457a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3f4ff9640 1 --2- 172.21.15.28:0/44454361 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55886457d570 0x55886457a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3f4ff9640 1 -- 172.21.15.28:0/44454361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558864572ea0 msgr2=0x55886457a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3f4ff9640 1 --2- 172.21.15.28:0/44454361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558864572ea0 0x55886457a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3f4ff9640 1 -- 172.21.15.28:0/44454361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558864464dc0 con 0x558864572580 2024-04-25T01:39:13.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3f57fa640 1 --2- 172.21.15.28:0/44454361 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55886457d570 0x55886457a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:13.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3f4ff9640 1 --2- 172.21.15.28:0/44454361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572580 0x55886457cea0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fe3e00bbff0 tx=0x7fe3e0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd79a4338432a0ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3cffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:13.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3cffff640 1 -- 172.21.15.28:0/44454361 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0049050 con 0x558864572580 2024-04-25T01:39:13.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3cffff640 1 -- 172.21.15.28:0/44454361 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fe3e00b86d0 con 0x558864572580 2024-04-25T01:39:13.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/44454361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3e00870f0 con 0x558864572580 2024-04-25T01:39:13.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3cffff640 1 -- 172.21.15.28:0/44454361 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe3e00c0410 con 0x558864572580 2024-04-25T01:39:13.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3f9f6f740 10 client.17727 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:13.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/44454361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5588641eb3f0 con 0x558864572580 2024-04-25T01:39:13.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3cffff640 1 -- 172.21.15.28:0/44454361 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3e00870f0 con 0x558864572580 2024-04-25T01:39:13.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3cffff640 1 -- 172.21.15.28:0/44454361 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe3e00b6bd0 con 0x558864572580 2024-04-25T01:39:13.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3f9f6f740 10 client.17727 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:13.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3f9f6f740 10 client.17727 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:39:13.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3f9f6f740 20 client.17727 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:13.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.723+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/44454361 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55886458d1b0 0x55886458f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.724+0000 7fe3f9f6f740 4 client.17727 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:13.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.724+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/44454361 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588644e4bb0 con 0x55886458d1b0 2024-04-25T01:39:13.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.724+0000 7fe3effff640 1 --2- 172.21.15.28:0/44454361 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55886458d1b0 0x55886458f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.724+0000 7fe3effff640 1 --2- 172.21.15.28:0/44454361 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55886458d1b0 0x55886458f5c0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=84607fd3a55080f7 server_cookie=be48afddb304408c in_seq=0 out_seq=0 2024-04-25T01:39:13.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.724+0000 7fe3cffff640 10 client.17727 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:39:13.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.725+0000 7fe3cffff640 1 -- 172.21.15.28:0/44454361 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5588644e4bb0 con 0x55886458d1b0 2024-04-25T01:39:13.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.725+0000 7fe3cffff640 10 client.17727 handle_command_reply: tid=0 2024-04-25T01:39:13.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.725+0000 7fe3f9f6f740 1 client.17727 shutdown 2024-04-25T01:39:13.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.725+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/44454361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572580 msgr2=0x55886457cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.725+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/44454361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572580 0x55886457cea0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fe3e00bbff0 tx=0x7fe3e0009260 comp rx=0 tx=0).stop 2024-04-25T01:39:13.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.725+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/44454361 shutdown_connections 2024-04-25T01:39:13.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.725+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/44454361 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55886458d1b0 0x55886458f5c0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.725+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/44454361 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55886457d570 0x55886457a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.725+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/44454361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558864572ea0 0x55886457a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.725+0000 7fe3cffff640 0 client.17727 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:39:13.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.725+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/44454361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572580 0x55886457cea0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.725+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/44454361 >> 172.21.15.28:0/44454361 conn(0x55886456e8a0 msgr2=0x55886456db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.725+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/44454361 shutdown_connections 2024-04-25T01:39:13.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.725+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/44454361 wait complete. 2024-04-25T01:39:13.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.725+0000 7fe3f9f6f740 20 client.17727 trim_cache size 0 max 16384 2024-04-25T01:39:13.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.739+0000 7fe3f9f6f740 1 Processor -- start 2024-04-25T01:39:13.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.739+0000 7fe3f9f6f740 1 -- start start 2024-04-25T01:39:13.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.739+0000 7fe3f9f6f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558864572580 0x558864588fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.739+0000 7fe3f9f6f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558864572ea0 0x5588645743e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.739+0000 7fe3f9f6f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55886457d570 0x55886457c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.739+0000 7fe3f9f6f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f01aea70 con 0x55886457d570 2024-04-25T01:39:13.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.739+0000 7fe3f9f6f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f01b05b0 con 0x558864572ea0 2024-04-25T01:39:13.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.739+0000 7fe3f9f6f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f01af140 con 0x558864572580 2024-04-25T01:39:13.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.739+0000 7fe3f57fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55886457d570 0x55886457c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.739+0000 7fe3f57fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55886457d570 0x55886457c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48752/0 (socket says 172.21.15.28:48752) 2024-04-25T01:39:13.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.739+0000 7fe3f57fa640 1 -- 172.21.15.28:0/2223416098 learned_addr learned my addr 172.21.15.28:0/2223416098 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.739+0000 7fe3f57fa640 1 -- 172.21.15.28:0/2223416098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558864572580 msgr2=0x558864588fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.739+0000 7fe3f57fa640 1 --2- 172.21.15.28:0/2223416098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558864572580 0x558864588fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.739+0000 7fe3f57fa640 1 -- 172.21.15.28:0/2223416098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558864572ea0 msgr2=0x5588645743e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.739+0000 7fe3f57fa640 1 --2- 172.21.15.28:0/2223416098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558864572ea0 0x5588645743e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.739+0000 7fe3f57fa640 1 -- 172.21.15.28:0/2223416098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588642b57d0 con 0x55886457d570 2024-04-25T01:39:13.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.739+0000 7fe3f57fa640 1 --2- 172.21.15.28:0/2223416098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55886457d570 0x55886457c9d0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8014820 tx=0x7fe3e8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d23c69adaf96b68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.739+0000 7fe3d67fc640 1 -- 172.21.15.28:0/2223416098 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8025070 con 0x55886457d570 2024-04-25T01:39:13.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.740+0000 7fe3d67fc640 1 -- 172.21.15.28:0/2223416098 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3e80128f0 con 0x55886457d570 2024-04-25T01:39:13.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.740+0000 7fe3d67fc640 1 -- 172.21.15.28:0/2223416098 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8005260 con 0x55886457d570 2024-04-25T01:39:13.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.740+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/2223416098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55886457d570 msgr2=0x55886457c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.740+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/2223416098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55886457d570 0x55886457c9d0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8014820 tx=0x7fe3e8004ab0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.740+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/2223416098 shutdown_connections 2024-04-25T01:39:13.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.740+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/2223416098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55886457d570 0x55886457c9d0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.740+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/2223416098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558864572ea0 0x5588645743e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.740+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/2223416098 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558864572580 0x558864588fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.740+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/2223416098 >> 172.21.15.28:0/2223416098 conn(0x55886456e8a0 msgr2=0x558864573cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.740+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/2223416098 shutdown_connections 2024-04-25T01:39:13.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.740+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/2223416098 wait complete. 2024-04-25T01:39:13.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.740+0000 7fe3f9f6f740 1 -- start start 2024-04-25T01:39:13.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.740+0000 7fe3f9f6f740 1 Processor -- start 2024-04-25T01:39:13.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3f9f6f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558864572580 0x558864585750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3f9f6f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572ea0 0x558864576a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3f9f6f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55886457d570 0x558864576f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3f9f6f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f01afee0 con 0x558864572ea0 2024-04-25T01:39:13.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3f9f6f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f01af810 con 0x55886457d570 2024-04-25T01:39:13.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3f9f6f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f01aca50 con 0x558864572580 2024-04-25T01:39:13.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3f9f6f740 10 client.0 _ll_register_callbacks cb 0x7fe3e80145b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:13.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3effff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572ea0 0x558864576a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3effff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572ea0 0x558864576a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48762/0 (socket says 172.21.15.28:48762) 2024-04-25T01:39:13.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3effff640 1 -- 172.21.15.28:0/1795505939 learned_addr learned my addr 172.21.15.28:0/1795505939 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3f4ff9640 1 --2- 172.21.15.28:0/1795505939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558864572580 0x558864585750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3effff640 1 -- 172.21.15.28:0/1795505939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558864572580 msgr2=0x558864585750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3effff640 1 --2- 172.21.15.28:0/1795505939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558864572580 0x558864585750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3effff640 1 -- 172.21.15.28:0/1795505939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55886457d570 msgr2=0x558864576f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3f57fa640 1 --2- 172.21.15.28:0/1795505939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55886457d570 0x558864576f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3effff640 1 --2- 172.21.15.28:0/1795505939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55886457d570 0x558864576f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3effff640 1 -- 172.21.15.28:0/1795505939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558864577600 con 0x558864572ea0 2024-04-25T01:39:13.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3f4ff9640 1 --2- 172.21.15.28:0/1795505939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558864572580 0x558864585750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:13.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3f57fa640 1 --2- 172.21.15.28:0/1795505939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55886457d570 0x558864576f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:13.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3effff640 1 --2- 172.21.15.28:0/1795505939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572ea0 0x558864576a20 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009b10 tx=0x7fe3e4014840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=784728b4ae6826f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.741+0000 7fe3cffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:13.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.742+0000 7fe3cffff640 1 -- 172.21.15.28:0/1795505939 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4004030 con 0x558864572ea0 2024-04-25T01:39:13.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.742+0000 7fe3cffff640 1 -- 172.21.15.28:0/1795505939 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fe3e40041d0 con 0x558864572ea0 2024-04-25T01:39:13.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.742+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/1795505939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3e80128f0 con 0x558864572ea0 2024-04-25T01:39:13.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.742+0000 7fe3cffff640 1 -- 172.21.15.28:0/1795505939 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe3e402d050 con 0x558864572ea0 2024-04-25T01:39:13.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.742+0000 7fe3f9f6f740 10 client.17733 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:13.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.742+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/1795505939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5588645f3620 con 0x558864572ea0 2024-04-25T01:39:13.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.742+0000 7fe3cffff640 1 -- 172.21.15.28:0/1795505939 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3e80128f0 con 0x558864572ea0 2024-04-25T01:39:13.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.742+0000 7fe3cffff640 1 -- 172.21.15.28:0/1795505939 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe3e400f040 con 0x558864572ea0 2024-04-25T01:39:13.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.742+0000 7fe3f9f6f740 10 client.17733 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:13.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.742+0000 7fe3f9f6f740 10 client.17733 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:39:13.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.742+0000 7fe3f9f6f740 20 client.17733 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:13.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.742+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/1795505939 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5588645f3dd0 0x5588645f61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.742+0000 7fe3f9f6f740 4 client.17733 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:13.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.742+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/1795505939 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55886457aa60 con 0x5588645f3dd0 2024-04-25T01:39:13.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.742+0000 7fe3f4ff9640 1 --2- 172.21.15.28:0/1795505939 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5588645f3dd0 0x5588645f61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.743+0000 7fe3cffff640 10 client.17733 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:39:13.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.743+0000 7fe3f4ff9640 1 --2- 172.21.15.28:0/1795505939 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5588645f3dd0 0x5588645f61f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=19491bfab873e325 server_cookie=ac1e7f56dbfd11e4 in_seq=0 out_seq=0 2024-04-25T01:39:13.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.743+0000 7fe3cffff640 1 -- 172.21.15.28:0/1795505939 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55886457aa60 con 0x5588645f3dd0 2024-04-25T01:39:13.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.743+0000 7fe3cffff640 10 client.17733 handle_command_reply: tid=0 2024-04-25T01:39:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.743+0000 7fe3f9f6f740 1 client.17733 shutdown 2024-04-25T01:39:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.743+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/1795505939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572ea0 msgr2=0x558864576a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.743+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/1795505939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572ea0 0x558864576a20 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009b10 tx=0x7fe3e4014840 comp rx=0 tx=0).stop 2024-04-25T01:39:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.743+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/1795505939 shutdown_connections 2024-04-25T01:39:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.743+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/1795505939 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5588645f3dd0 0x5588645f61f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.743+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/1795505939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55886457d570 0x558864576f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.743+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/1795505939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558864572ea0 0x558864576a20 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.743+0000 7fe3cffff640 0 client.17733 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:39:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.743+0000 7fe3f9f6f740 1 --2- 172.21.15.28:0/1795505939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558864572580 0x558864585750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.743+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/1795505939 >> 172.21.15.28:0/1795505939 conn(0x55886456e8a0 msgr2=0x558864574560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.743+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/1795505939 shutdown_connections 2024-04-25T01:39:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.743+0000 7fe3f9f6f740 1 -- 172.21.15.28:0/1795505939 wait complete. 2024-04-25T01:39:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.743+0000 7fe3f9f6f740 20 client.17733 trim_cache size 0 max 16384 2024-04-25T01:39:13.733 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:13.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.744+0000 7fe3f725d640 1 -- 172.21.15.28:0/946671860 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe3d80a2d00 msgr2=0x7fe3d80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.744+0000 7fe3f725d640 1 --2- 172.21.15.28:0/946671860 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe3d80a2d00 0x7fe3d80a51c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8005c40 tx=0x7fe3e8009cd0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.744+0000 7fe3f725d640 1 -- 172.21.15.28:0/946671860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0101710 msgr2=0x7fe3f0198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.744+0000 7fe3f725d640 1 --2- 172.21.15.28:0/946671860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0101710 0x7fe3f0198050 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0009230 tx=0x7fe3e0004270 comp rx=0 tx=0).stop 2024-04-25T01:39:13.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.744+0000 7fe3f725d640 1 -- 172.21.15.28:0/946671860 shutdown_connections 2024-04-25T01:39:13.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.744+0000 7fe3f725d640 1 --2- 172.21.15.28:0/946671860 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe3d80a2d00 0x7fe3d80a51c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.744+0000 7fe3f725d640 1 --2- 172.21.15.28:0/946671860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe3f0107720 0x7fe3f019c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.744+0000 7fe3f725d640 1 --2- 172.21.15.28:0/946671860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe3f01020d0 0x7fe3f0198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.744+0000 7fe3f725d640 1 --2- 172.21.15.28:0/946671860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3f0101710 0x7fe3f0198050 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.744+0000 7fe3f725d640 1 -- 172.21.15.28:0/946671860 >> 172.21.15.28:0/946671860 conn(0x7fe3f0078b90 msgr2=0x7fe3f00ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.744+0000 7fe3f725d640 1 -- 172.21.15.28:0/946671860 shutdown_connections 2024-04-25T01:39:13.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.744+0000 7fe3f725d640 1 -- 172.21.15.28:0/946671860 wait complete. 2024-04-25T01:39:13.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:39:13.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.860+0000 7fedbbef5640 1 -- 172.21.15.28:0/3870609662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb4100620 msgr2=0x7fedb4100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.860+0000 7fedbbef5640 1 --2- 172.21.15.28:0/3870609662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb4100620 0x7fedb4100aa0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7feda4009930 tx=0x7feda402a420 comp rx=0 tx=0).stop 2024-04-25T01:39:13.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.860+0000 7fedbbef5640 1 -- 172.21.15.28:0/3870609662 shutdown_connections 2024-04-25T01:39:13.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.860+0000 7fedbbef5640 1 --2- 172.21.15.28:0/3870609662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb4100fe0 0x7fedb410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.860+0000 7fedbbef5640 1 --2- 172.21.15.28:0/3870609662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb4100620 0x7fedb4100aa0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.860+0000 7fedbbef5640 1 --2- 172.21.15.28:0/3870609662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedb4106630 0x7fedb4106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.860+0000 7fedbbef5640 1 -- 172.21.15.28:0/3870609662 >> 172.21.15.28:0/3870609662 conn(0x7fedb40769c0 msgr2=0x7fedb4076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.860+0000 7fedbbef5640 1 -- 172.21.15.28:0/3870609662 shutdown_connections 2024-04-25T01:39:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.860+0000 7fedbbef5640 1 -- 172.21.15.28:0/3870609662 wait complete. 2024-04-25T01:39:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.860+0000 7fedbbef5640 1 Processor -- start 2024-04-25T01:39:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.860+0000 7fedbbef5640 1 -- start start 2024-04-25T01:39:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedbbef5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb4100620 0x7fedb41980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedbbef5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedb4100fe0 0x7fedb41985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedbbef5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb4106630 0x7fedb419c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedbbef5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fedb4110a40 con 0x7fedb4100620 2024-04-25T01:39:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedbbef5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41108c0 con 0x7fedb4106630 2024-04-25T01:39:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedbbef5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fedb4110bc0 con 0x7fedb4100fe0 2024-04-25T01:39:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedb9c91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb4100620 0x7fedb41980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedb9c91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb4100620 0x7fedb41980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48778/0 (socket says 172.21.15.28:48778) 2024-04-25T01:39:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedb9c91640 1 -- 172.21.15.28:0/3033618779 learned_addr learned my addr 172.21.15.28:0/3033618779 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedba492640 1 --2- 172.21.15.28:0/3033618779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb4106630 0x7fedb419c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedb9490640 1 --2- 172.21.15.28:0/3033618779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedb4100fe0 0x7fedb41985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedb9c91640 1 -- 172.21.15.28:0/3033618779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedb4100fe0 msgr2=0x7fedb41985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedb9c91640 1 --2- 172.21.15.28:0/3033618779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedb4100fe0 0x7fedb41985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedb9c91640 1 -- 172.21.15.28:0/3033618779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb4106630 msgr2=0x7fedb419c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedb9c91640 1 --2- 172.21.15.28:0/3033618779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb4106630 0x7fedb419c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedb9c91640 1 -- 172.21.15.28:0/3033618779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb419d0a0 con 0x7fedb4100620 2024-04-25T01:39:13.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedb9c91640 1 --2- 172.21.15.28:0/3033618779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb4100620 0x7fedb41980b0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fedb000e9f0 tx=0x7fedb000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedaaffd640 1 -- 172.21.15.28:0/3033618779 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb000cd80 con 0x7fedb4100620 2024-04-25T01:39:13.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedbbef5640 1 -- 172.21.15.28:0/3033618779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedb419d390 con 0x7fedb4100620 2024-04-25T01:39:13.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedaaffd640 1 -- 172.21.15.28:0/3033618779 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedb0005500 con 0x7fedb4100620 2024-04-25T01:39:13.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedaaffd640 1 -- 172.21.15.28:0/3033618779 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb0010690 con 0x7fedb4100620 2024-04-25T01:39:13.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.861+0000 7fedbbef5640 1 -- 172.21.15.28:0/3033618779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedb41a4c20 con 0x7fedb4100620 2024-04-25T01:39:13.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.864+0000 7fedaaffd640 1 -- 172.21.15.28:0/3033618779 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fedb00046c0 con 0x7fedb4100620 2024-04-25T01:39:13.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.864+0000 7fedbec03740 1 Processor -- start 2024-04-25T01:39:13.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.864+0000 7fedbec03740 1 -- start start 2024-04-25T01:39:13.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.864+0000 7fedbec03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 0x561dcc389860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.864+0000 7fedbec03740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc389da0 0x561dcc398440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.864+0000 7fedbec03740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc398980 0x561dcc39ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.864+0000 7fedbec03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561dcc2ee670 con 0x561dcc389480 2024-04-25T01:39:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.864+0000 7fedbec03740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561dcc296970 con 0x561dcc389da0 2024-04-25T01:39:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.864+0000 7fedbec03740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x561dcc2ea630 con 0x561dcc398980 2024-04-25T01:39:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.865+0000 7fedaaffd640 1 --2- 172.21.15.28:0/3033618779 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7feda00a2d00 0x7feda00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.865+0000 7fedba492640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 0x561dcc389860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.865+0000 7fedba492640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 0x561dcc389860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48786/0 (socket says 172.21.15.28:48786) 2024-04-25T01:39:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.865+0000 7fedba492640 1 -- 172.21.15.28:0/572054537 learned_addr learned my addr 172.21.15.28:0/572054537 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.865+0000 7fedba492640 1 --2- 172.21.15.28:0/3033618779 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7feda00a2d00 0x7feda00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.865+0000 7fedb9c91640 1 --2- 172.21.15.28:0/572054537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc389da0 0x561dcc398440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.865+0000 7fedb9490640 1 --2- 172.21.15.28:0/572054537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc398980 0x561dcc39ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.865+0000 7fedaaffd640 1 -- 172.21.15.28:0/3033618779 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fedb00cfc70 con 0x7fedb4100620 2024-04-25T01:39:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.865+0000 7fedba492640 1 -- 172.21.15.28:0/572054537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc398980 msgr2=0x561dcc39ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.865+0000 7fedba492640 1 --2- 172.21.15.28:0/572054537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc398980 0x561dcc39ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.865+0000 7fedba492640 1 -- 172.21.15.28:0/572054537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc389da0 msgr2=0x561dcc398440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.865+0000 7fedba492640 1 --2- 172.21.15.28:0/572054537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc389da0 0x561dcc398440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.865+0000 7fedba492640 1 -- 172.21.15.28:0/572054537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dcc2ba400 con 0x561dcc389480 2024-04-25T01:39:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.865+0000 7fedb9490640 1 --2- 172.21.15.28:0/572054537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc398980 0x561dcc39ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.865+0000 7fedba492640 1 --2- 172.21.15.28:0/572054537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 0x561dcc389860 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fedac013e60 tx=0x7fedac0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad21057ff15ed5c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.865+0000 7fedba492640 1 --2- 172.21.15.28:0/3033618779 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7feda00a2d00 0x7feda00a51c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fedac010ca0 tx=0x7fedac015000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.865+0000 7fed9ffff640 1 -- 172.21.15.28:0/572054537 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac00ae10 con 0x561dcc389480 2024-04-25T01:39:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.865+0000 7fed9ffff640 1 -- 172.21.15.28:0/572054537 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedac007be0 con 0x561dcc389480 2024-04-25T01:39:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.865+0000 7fed9ffff640 1 -- 172.21.15.28:0/572054537 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac00e440 con 0x561dcc389480 2024-04-25T01:39:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.866+0000 7fedbec03740 1 -- 172.21.15.28:0/572054537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 msgr2=0x561dcc389860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.866+0000 7fedbec03740 1 --2- 172.21.15.28:0/572054537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 0x561dcc389860 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fedac013e60 tx=0x7fedac0107a0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.866+0000 7fedbec03740 1 -- 172.21.15.28:0/572054537 shutdown_connections 2024-04-25T01:39:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.866+0000 7fedbec03740 1 --2- 172.21.15.28:0/572054537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc398980 0x561dcc39ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.866+0000 7fedbec03740 1 --2- 172.21.15.28:0/572054537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc389da0 0x561dcc398440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.866+0000 7fedbec03740 1 --2- 172.21.15.28:0/572054537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 0x561dcc389860 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.866+0000 7fedbec03740 1 -- 172.21.15.28:0/572054537 >> 172.21.15.28:0/572054537 conn(0x561dcc3858a0 msgr2=0x561dcc385c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.866+0000 7fedbec03740 1 -- 172.21.15.28:0/572054537 shutdown_connections 2024-04-25T01:39:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.866+0000 7fedbec03740 1 -- 172.21.15.28:0/572054537 wait complete. 2024-04-25T01:39:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.867+0000 7fedbec03740 1 -- start start 2024-04-25T01:39:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.867+0000 7fedbec03740 1 Processor -- start 2024-04-25T01:39:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.867+0000 7fedbec03740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc389480 0x561dcc398300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.867+0000 7fedbec03740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc389da0 0x561dcc3955f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.867+0000 7fedbec03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc398980 0x561dcc395b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.867+0000 7fedbec03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561dcc39b6e0 con 0x561dcc398980 2024-04-25T01:39:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.867+0000 7fedbec03740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561dcc2aa050 con 0x561dcc389480 2024-04-25T01:39:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.867+0000 7fedbec03740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x561dcc39b860 con 0x561dcc389da0 2024-04-25T01:39:13.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.867+0000 7fedbec03740 10 client.0 _ll_register_callbacks cb 0x561dcc001be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.867+0000 7fedba492640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc398980 0x561dcc395b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.867+0000 7fedba492640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc398980 0x561dcc395b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48794/0 (socket says 172.21.15.28:48794) 2024-04-25T01:39:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.867+0000 7fedba492640 1 -- 172.21.15.28:0/860310845 learned_addr learned my addr 172.21.15.28:0/860310845 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.867+0000 7fedb9c91640 1 --2- 172.21.15.28:0/860310845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc389480 0x561dcc398300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.867+0000 7fedb9490640 1 --2- 172.21.15.28:0/860310845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc389da0 0x561dcc3955f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.868+0000 7fedba492640 1 -- 172.21.15.28:0/860310845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc389da0 msgr2=0x561dcc3955f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.868+0000 7fedba492640 1 --2- 172.21.15.28:0/860310845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc389da0 0x561dcc3955f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.868+0000 7fedba492640 1 -- 172.21.15.28:0/860310845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc389480 msgr2=0x561dcc398300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.868+0000 7fedba492640 1 --2- 172.21.15.28:0/860310845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc389480 0x561dcc398300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.868+0000 7fedba492640 1 -- 172.21.15.28:0/860310845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dcc27bdc0 con 0x561dcc398980 2024-04-25T01:39:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.868+0000 7fedb9c91640 1 --2- 172.21.15.28:0/860310845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc389480 0x561dcc398300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:39:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.868+0000 7fedba492640 1 --2- 172.21.15.28:0/860310845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc398980 0x561dcc395b70 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fedac0076d0 tx=0x7fedac005910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=177f35cb634d0914 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.868+0000 7fed9d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:13.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.868+0000 7fed9d7fa640 1 -- 172.21.15.28:0/860310845 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac013220 con 0x561dcc398980 2024-04-25T01:39:13.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.868+0000 7fed9d7fa640 1 -- 172.21.15.28:0/860310845 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fedac003b80 con 0x561dcc398980 2024-04-25T01:39:13.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.868+0000 7fedbec03740 1 -- 172.21.15.28:0/860310845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedac007be0 con 0x561dcc398980 2024-04-25T01:39:13.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.868+0000 7fed9d7fa640 1 -- 172.21.15.28:0/860310845 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fedac0059b0 con 0x561dcc398980 2024-04-25T01:39:13.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.868+0000 7fedbec03740 10 client.17745 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:13.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.868+0000 7fedbec03740 1 -- 172.21.15.28:0/860310845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561dcc0023f0 con 0x561dcc398980 2024-04-25T01:39:13.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.868+0000 7fed9d7fa640 1 -- 172.21.15.28:0/860310845 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedac007be0 con 0x561dcc398980 2024-04-25T01:39:13.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.868+0000 7fed9d7fa640 1 -- 172.21.15.28:0/860310845 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fedac004040 con 0x561dcc398980 2024-04-25T01:39:13.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.868+0000 7fedbec03740 10 client.17745 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:13.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.868+0000 7fedbec03740 10 client.17745 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:39:13.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.868+0000 7fedbec03740 20 client.17745 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:13.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.869+0000 7fedbec03740 1 --2- 172.21.15.28:0/860310845 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x561dcc3a8960 0x561dcc3aad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.869+0000 7fedbec03740 4 client.17745 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:13.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.869+0000 7fedbec03740 1 -- 172.21.15.28:0/860310845 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561dcc2fbbb0 con 0x561dcc3a8960 2024-04-25T01:39:13.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.869+0000 7fedb9c91640 1 --2- 172.21.15.28:0/860310845 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x561dcc3a8960 0x561dcc3aad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.869+0000 7fedb9c91640 1 --2- 172.21.15.28:0/860310845 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x561dcc3a8960 0x561dcc3aad50 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a871bc31643074c8 server_cookie=f1305d12f79f6a07 in_seq=0 out_seq=0 2024-04-25T01:39:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.869+0000 7fed9d7fa640 10 client.17745 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:39:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.870+0000 7fed9d7fa640 1 -- 172.21.15.28:0/860310845 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561dcc2fbbb0 con 0x561dcc3a8960 2024-04-25T01:39:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.870+0000 7fed9d7fa640 10 client.17745 handle_command_reply: tid=0 2024-04-25T01:39:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.870+0000 7fedbec03740 1 client.17745 shutdown 2024-04-25T01:39:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.870+0000 7fedbec03740 1 -- 172.21.15.28:0/860310845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc398980 msgr2=0x561dcc395b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.870+0000 7fedbec03740 1 --2- 172.21.15.28:0/860310845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc398980 0x561dcc395b70 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fedac0076d0 tx=0x7fedac005910 comp rx=0 tx=0).stop 2024-04-25T01:39:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.870+0000 7fedbec03740 1 -- 172.21.15.28:0/860310845 shutdown_connections 2024-04-25T01:39:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.870+0000 7fedbec03740 1 --2- 172.21.15.28:0/860310845 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x561dcc3a8960 0x561dcc3aad50 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.870+0000 7fedbec03740 1 --2- 172.21.15.28:0/860310845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc398980 0x561dcc395b70 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.870+0000 7fedbec03740 1 --2- 172.21.15.28:0/860310845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc389da0 0x561dcc3955f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.870+0000 7fedbec03740 1 --2- 172.21.15.28:0/860310845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc389480 0x561dcc398300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.870+0000 7fed9d7fa640 0 client.17745 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:39:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.870+0000 7fedbec03740 1 -- 172.21.15.28:0/860310845 >> 172.21.15.28:0/860310845 conn(0x561dcc3858a0 msgr2=0x561dcc38b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.870+0000 7fedbec03740 1 -- 172.21.15.28:0/860310845 shutdown_connections 2024-04-25T01:39:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.870+0000 7fedbec03740 1 -- 172.21.15.28:0/860310845 wait complete. 2024-04-25T01:39:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.870+0000 7fedbec03740 20 client.17745 trim_cache size 0 max 16384 2024-04-25T01:39:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.884+0000 7fedbec03740 1 Processor -- start 2024-04-25T01:39:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.884+0000 7fedbec03740 1 -- start start 2024-04-25T01:39:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.884+0000 7fedbec03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 0x561dcc3a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.884+0000 7fedbec03740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc389da0 0x561dcc394750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.884+0000 7fedbec03740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc398980 0x561dcc390af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.884+0000 7fedbec03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41ac860 con 0x561dcc389480 2024-04-25T01:39:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.884+0000 7fedbec03740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41ae3a0 con 0x561dcc398980 2024-04-25T01:39:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.884+0000 7fedbec03740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41acf30 con 0x561dcc389da0 2024-04-25T01:39:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.884+0000 7fedb9c91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 0x561dcc3a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.884+0000 7fedb9c91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 0x561dcc3a45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48806/0 (socket says 172.21.15.28:48806) 2024-04-25T01:39:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.884+0000 7fedb9c91640 1 -- 172.21.15.28:0/1446570072 learned_addr learned my addr 172.21.15.28:0/1446570072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.884+0000 7fedba492640 1 --2- 172.21.15.28:0/1446570072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc398980 0x561dcc390af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.884+0000 7fedb9c91640 1 -- 172.21.15.28:0/1446570072 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc389da0 msgr2=0x561dcc394750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.884+0000 7fedb9490640 1 --2- 172.21.15.28:0/1446570072 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc389da0 0x561dcc394750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.884+0000 7fedb9c91640 1 --2- 172.21.15.28:0/1446570072 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc389da0 0x561dcc394750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.884+0000 7fedb9c91640 1 -- 172.21.15.28:0/1446570072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc398980 msgr2=0x561dcc390af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.884+0000 7fedb9c91640 1 --2- 172.21.15.28:0/1446570072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc398980 0x561dcc390af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.884+0000 7fedb9c91640 1 -- 172.21.15.28:0/1446570072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dcc0cc7d0 con 0x561dcc389480 2024-04-25T01:39:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.884+0000 7fedba492640 1 --2- 172.21.15.28:0/1446570072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc398980 0x561dcc390af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.885+0000 7fedb9c91640 1 --2- 172.21.15.28:0/1446570072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 0x561dcc3a45a0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fedb00cd0e0 tx=0x7fedb0007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db437f2eeb9b41fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.885+0000 7fed9f7fe640 1 -- 172.21.15.28:0/1446570072 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb009c550 con 0x561dcc389480 2024-04-25T01:39:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.885+0000 7fed9f7fe640 1 -- 172.21.15.28:0/1446570072 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedb009e040 con 0x561dcc389480 2024-04-25T01:39:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.885+0000 7fed9f7fe640 1 -- 172.21.15.28:0/1446570072 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb00984c0 con 0x561dcc389480 2024-04-25T01:39:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.885+0000 7fedbec03740 1 -- 172.21.15.28:0/1446570072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 msgr2=0x561dcc3a45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.885+0000 7fedbec03740 1 --2- 172.21.15.28:0/1446570072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 0x561dcc3a45a0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fedb00cd0e0 tx=0x7fedb0007850 comp rx=0 tx=0).stop 2024-04-25T01:39:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.885+0000 7fedbec03740 1 -- 172.21.15.28:0/1446570072 shutdown_connections 2024-04-25T01:39:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.885+0000 7fedbec03740 1 --2- 172.21.15.28:0/1446570072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc398980 0x561dcc390af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.885+0000 7fedbec03740 1 --2- 172.21.15.28:0/1446570072 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc389da0 0x561dcc394750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.885+0000 7fedbec03740 1 --2- 172.21.15.28:0/1446570072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 0x561dcc3a45a0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.885+0000 7fedbec03740 1 -- 172.21.15.28:0/1446570072 >> 172.21.15.28:0/1446570072 conn(0x561dcc3858a0 msgr2=0x561dcc3903e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.885+0000 7fedbec03740 1 -- 172.21.15.28:0/1446570072 shutdown_connections 2024-04-25T01:39:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.885+0000 7fedbec03740 1 -- 172.21.15.28:0/1446570072 wait complete. 2024-04-25T01:39:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedbec03740 1 -- start start 2024-04-25T01:39:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedbec03740 1 Processor -- start 2024-04-25T01:39:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedbec03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 0x561dcc3a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedbec03740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc389da0 0x561dcc3a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedbec03740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc398980 0x561dcc3919a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedbec03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41adcd0 con 0x561dcc389480 2024-04-25T01:39:13.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedbec03740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41ad600 con 0x561dcc389da0 2024-04-25T01:39:13.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedbec03740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fedb41aa840 con 0x561dcc398980 2024-04-25T01:39:13.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedbec03740 10 client.0 _ll_register_callbacks cb 0x7fedb0097a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:13.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedb9c91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 0x561dcc3a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedb9c91640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 0x561dcc3a0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48822/0 (socket says 172.21.15.28:48822) 2024-04-25T01:39:13.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedb9c91640 1 -- 172.21.15.28:0/1139894016 learned_addr learned my addr 172.21.15.28:0/1139894016 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedb9490640 1 --2- 172.21.15.28:0/1139894016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc389da0 0x561dcc3a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedba492640 1 --2- 172.21.15.28:0/1139894016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc398980 0x561dcc3919a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedb9c91640 1 -- 172.21.15.28:0/1139894016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc398980 msgr2=0x561dcc3919a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedb9c91640 1 --2- 172.21.15.28:0/1139894016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc398980 0x561dcc3919a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedb9c91640 1 -- 172.21.15.28:0/1139894016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc389da0 msgr2=0x561dcc3a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedb9c91640 1 --2- 172.21.15.28:0/1139894016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc389da0 0x561dcc3a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedb9c91640 1 -- 172.21.15.28:0/1139894016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dcc392090 con 0x561dcc389480 2024-04-25T01:39:13.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedb9490640 1 --2- 172.21.15.28:0/1139894016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc389da0 0x561dcc3a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.886+0000 7fedba492640 1 --2- 172.21.15.28:0/1139894016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc398980 0x561dcc3919a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.887+0000 7fedb9c91640 1 --2- 172.21.15.28:0/1139894016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 0x561dcc3a0ab0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fedb0004a80 tx=0x7fedb00cfe70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2cebae9196a9aaa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.887+0000 7fed9d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.887+0000 7fed9d7fa640 1 -- 172.21.15.28:0/1139894016 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb0065070 con 0x561dcc389480 2024-04-25T01:39:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.887+0000 7fed9d7fa640 1 -- 172.21.15.28:0/1139894016 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fedb00984c0 con 0x561dcc389480 2024-04-25T01:39:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.887+0000 7fedbec03740 1 -- 172.21.15.28:0/1139894016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedb009e040 con 0x561dcc389480 2024-04-25T01:39:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.887+0000 7fed9d7fa640 1 -- 172.21.15.28:0/1139894016 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fedb009c550 con 0x561dcc389480 2024-04-25T01:39:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.887+0000 7fedbec03740 10 client.17751 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.887+0000 7fedbec03740 1 -- 172.21.15.28:0/1139894016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561dcc40e4e0 con 0x561dcc389480 2024-04-25T01:39:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.887+0000 7fed9d7fa640 1 -- 172.21.15.28:0/1139894016 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedb009e040 con 0x561dcc389480 2024-04-25T01:39:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.887+0000 7fed9d7fa640 1 -- 172.21.15.28:0/1139894016 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fedb00a4730 con 0x561dcc389480 2024-04-25T01:39:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.887+0000 7fedbec03740 10 client.17751 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.887+0000 7fedbec03740 10 client.17751 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:39:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.887+0000 7fedbec03740 20 client.17751 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.887+0000 7fedbec03740 1 --2- 172.21.15.28:0/1139894016 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x561dcc40eb50 0x561dcc410f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.887+0000 7fedbec03740 4 client.17751 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.887+0000 7fedbec03740 1 -- 172.21.15.28:0/1139894016 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561dcc2f5f00 con 0x561dcc40eb50 2024-04-25T01:39:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.888+0000 7fedb9490640 1 --2- 172.21.15.28:0/1139894016 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x561dcc40eb50 0x561dcc410f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.888+0000 7fed9d7fa640 10 client.17751 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:39:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.888+0000 7fedb9490640 1 --2- 172.21.15.28:0/1139894016 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x561dcc40eb50 0x561dcc410f70 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=35a84f942360771e server_cookie=a14ba9add7b5874a in_seq=0 out_seq=0 2024-04-25T01:39:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.888+0000 7fed9d7fa640 1 -- 172.21.15.28:0/1139894016 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561dcc2f5f00 con 0x561dcc40eb50 2024-04-25T01:39:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.888+0000 7fed9d7fa640 10 client.17751 handle_command_reply: tid=0 2024-04-25T01:39:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.888+0000 7fedbec03740 1 client.17751 shutdown 2024-04-25T01:39:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.888+0000 7fedbec03740 1 -- 172.21.15.28:0/1139894016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 msgr2=0x561dcc3a0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.888+0000 7fedbec03740 1 --2- 172.21.15.28:0/1139894016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 0x561dcc3a0ab0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fedb0004a80 tx=0x7fedb00cfe70 comp rx=0 tx=0).stop 2024-04-25T01:39:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.888+0000 7fedbec03740 1 -- 172.21.15.28:0/1139894016 shutdown_connections 2024-04-25T01:39:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.888+0000 7fedbec03740 1 --2- 172.21.15.28:0/1139894016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561dcc398980 0x561dcc3919a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.888+0000 7fedbec03740 1 --2- 172.21.15.28:0/1139894016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561dcc389da0 0x561dcc3a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.888+0000 7fedbec03740 1 --2- 172.21.15.28:0/1139894016 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x561dcc40eb50 0x561dcc410f70 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.888+0000 7fedbec03740 1 --2- 172.21.15.28:0/1139894016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dcc389480 0x561dcc3a0ab0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.888+0000 7fed9d7fa640 0 client.17751 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:39:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.888+0000 7fedbec03740 1 -- 172.21.15.28:0/1139894016 >> 172.21.15.28:0/1139894016 conn(0x561dcc3858a0 msgr2=0x561dcc394750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.888+0000 7fedbec03740 1 -- 172.21.15.28:0/1139894016 shutdown_connections 2024-04-25T01:39:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.889+0000 7fedbec03740 1 -- 172.21.15.28:0/1139894016 wait complete. 2024-04-25T01:39:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.889+0000 7fedbec03740 20 client.17751 trim_cache size 0 max 16384 2024-04-25T01:39:13.878 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:13.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.889+0000 7fedbbef5640 1 -- 172.21.15.28:0/3033618779 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7feda00a2d00 msgr2=0x7feda00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.889+0000 7fedbbef5640 1 --2- 172.21.15.28:0/3033618779 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7feda00a2d00 0x7feda00a51c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fedac010ca0 tx=0x7fedac015000 comp rx=0 tx=0).stop 2024-04-25T01:39:13.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.889+0000 7fedbbef5640 1 -- 172.21.15.28:0/3033618779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb4100620 msgr2=0x7fedb41980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.889+0000 7fedbbef5640 1 --2- 172.21.15.28:0/3033618779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb4100620 0x7fedb41980b0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fedb000e9f0 tx=0x7fedb000edf0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.889+0000 7fedbbef5640 1 -- 172.21.15.28:0/3033618779 shutdown_connections 2024-04-25T01:39:13.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.889+0000 7fedbbef5640 1 --2- 172.21.15.28:0/3033618779 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7feda00a2d00 0x7feda00a51c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.889+0000 7fedbbef5640 1 --2- 172.21.15.28:0/3033618779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fedb4106630 0x7fedb419c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.889+0000 7fedbbef5640 1 --2- 172.21.15.28:0/3033618779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fedb4100fe0 0x7fedb41985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.889+0000 7fedbbef5640 1 --2- 172.21.15.28:0/3033618779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedb4100620 0x7fedb41980b0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.889+0000 7fedbbef5640 1 -- 172.21.15.28:0/3033618779 >> 172.21.15.28:0/3033618779 conn(0x7fedb40769c0 msgr2=0x7fedb406a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.889+0000 7fedbbef5640 1 -- 172.21.15.28:0/3033618779 shutdown_connections 2024-04-25T01:39:13.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:13.889+0000 7fedbbef5640 1 -- 172.21.15.28:0/3033618779 wait complete. 2024-04-25T01:39:13.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:39:13.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.005+0000 7fe51d129640 1 -- 172.21.15.28:0/614830672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe518100610 msgr2=0x7fe518100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.005+0000 7fe51d129640 1 --2- 172.21.15.28:0/614830672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe518100610 0x7fe518100a90 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fe500009930 tx=0x7fe50002a420 comp rx=0 tx=0).stop 2024-04-25T01:39:13.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.005+0000 7fe51d129640 1 -- 172.21.15.28:0/614830672 shutdown_connections 2024-04-25T01:39:13.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.005+0000 7fe51d129640 1 --2- 172.21.15.28:0/614830672 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe518100fd0 0x7fe51810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.005+0000 7fe51d129640 1 --2- 172.21.15.28:0/614830672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe518100610 0x7fe518100a90 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.005+0000 7fe51d129640 1 --2- 172.21.15.28:0/614830672 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5181066b0 0x7fe518106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.005+0000 7fe51d129640 1 -- 172.21.15.28:0/614830672 >> 172.21.15.28:0/614830672 conn(0x7fe518077740 msgr2=0x7fe518077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:13.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.005+0000 7fe51d129640 1 -- 172.21.15.28:0/614830672 shutdown_connections 2024-04-25T01:39:13.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.005+0000 7fe51d129640 1 -- 172.21.15.28:0/614830672 wait complete. 2024-04-25T01:39:13.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.005+0000 7fe51d129640 1 Processor -- start 2024-04-25T01:39:13.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.006+0000 7fe51d129640 1 -- start start 2024-04-25T01:39:13.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.006+0000 7fe51d129640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe518100610 0x7fe518196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.006+0000 7fe51d129640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe518100fd0 0x7fe518197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.006+0000 7fe51d129640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5181066b0 0x7fe51819b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.006+0000 7fe51d129640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe518110a80 con 0x7fe518100610 2024-04-25T01:39:13.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.006+0000 7fe51d129640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe518110900 con 0x7fe5181066b0 2024-04-25T01:39:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.006+0000 7fe51d129640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe518110c00 con 0x7fe518100fd0 2024-04-25T01:39:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.006+0000 7fe516d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe518100610 0x7fe518196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.006+0000 7fe516d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe518100610 0x7fe518196f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48852/0 (socket says 172.21.15.28:48852) 2024-04-25T01:39:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.006+0000 7fe516d9d640 1 -- 172.21.15.28:0/930878946 learned_addr learned my addr 172.21.15.28:0/930878946 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.006+0000 7fe51759e640 1 --2- 172.21.15.28:0/930878946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5181066b0 0x7fe51819b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.006+0000 7fe50ffff640 1 --2- 172.21.15.28:0/930878946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe518100fd0 0x7fe518197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.006+0000 7fe516d9d640 1 -- 172.21.15.28:0/930878946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe518100fd0 msgr2=0x7fe518197440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.006+0000 7fe516d9d640 1 --2- 172.21.15.28:0/930878946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe518100fd0 0x7fe518197440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.006+0000 7fe516d9d640 1 -- 172.21.15.28:0/930878946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5181066b0 msgr2=0x7fe51819b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.006+0000 7fe516d9d640 1 --2- 172.21.15.28:0/930878946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5181066b0 0x7fe51819b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.006+0000 7fe516d9d640 1 -- 172.21.15.28:0/930878946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51819bef0 con 0x7fe518100610 2024-04-25T01:39:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.006+0000 7fe516d9d640 1 --2- 172.21.15.28:0/930878946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe518100610 0x7fe518196f00 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fe50800d770 tx=0x7fe50800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.007+0000 7fe514d99640 1 -- 172.21.15.28:0/930878946 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508017070 con 0x7fe518100610 2024-04-25T01:39:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.007+0000 7fe514d99640 1 -- 172.21.15.28:0/930878946 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe508009c30 con 0x7fe518100610 2024-04-25T01:39:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.007+0000 7fe51d129640 1 -- 172.21.15.28:0/930878946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe51819c1e0 con 0x7fe518100610 2024-04-25T01:39:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.007+0000 7fe514d99640 1 -- 172.21.15.28:0/930878946 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508003dd0 con 0x7fe518100610 2024-04-25T01:39:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.007+0000 7fe51d129640 1 -- 172.21.15.28:0/930878946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5181a3af0 con 0x7fe518100610 2024-04-25T01:39:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.010+0000 7fe51fe3a740 1 Processor -- start 2024-04-25T01:39:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.010+0000 7fe514d99640 1 -- 172.21.15.28:0/930878946 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe508003fb0 con 0x7fe518100610 2024-04-25T01:39:13.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.010+0000 7fe51fe3a740 1 -- start start 2024-04-25T01:39:13.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.010+0000 7fe51fe3a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94c1f0 0x55843c94c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.010+0000 7fe51fe3a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c94cba0 0x55843c959240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.010+0000 7fe51fe3a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c959780 0x55843c95bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.010+0000 7fe51fe3a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55843c8b1670 con 0x55843c94c1f0 2024-04-25T01:39:13.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.010+0000 7fe51fe3a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55843c859970 con 0x55843c959780 2024-04-25T01:39:13.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.010+0000 7fe51fe3a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55843c8ad630 con 0x55843c94cba0 2024-04-25T01:39:13.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.010+0000 7fe51759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94c1f0 0x55843c94c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.010+0000 7fe51759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94c1f0 0x55843c94c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48868/0 (socket says 172.21.15.28:48868) 2024-04-25T01:39:13.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.010+0000 7fe51759e640 1 -- 172.21.15.28:0/368611189 learned_addr learned my addr 172.21.15.28:0/368611189 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:13.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.010+0000 7fe50ffff640 1 --2- 172.21.15.28:0/368611189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c959780 0x55843c95bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.010+0000 7fe516d9d640 1 --2- 172.21.15.28:0/368611189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c94cba0 0x55843c959240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.010+0000 7fe514d99640 1 --2- 172.21.15.28:0/930878946 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe5040a2d80 0x7fe5040a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.010+0000 7fe51759e640 1 -- 172.21.15.28:0/368611189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c94cba0 msgr2=0x55843c959240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.010+0000 7fe51759e640 1 --2- 172.21.15.28:0/368611189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c94cba0 0x55843c959240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.010+0000 7fe51759e640 1 -- 172.21.15.28:0/368611189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c959780 msgr2=0x55843c95bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.010+0000 7fe51759e640 1 --2- 172.21.15.28:0/368611189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c959780 0x55843c95bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.011+0000 7fe51759e640 1 -- 172.21.15.28:0/368611189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55843c87d400 con 0x55843c94c1f0 2024-04-25T01:39:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.011+0000 7fe516d9d640 1 --2- 172.21.15.28:0/368611189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c94cba0 0x55843c959240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.011+0000 7fe514d99640 1 -- 172.21.15.28:0/930878946 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fe5080cf0b0 con 0x7fe518100610 2024-04-25T01:39:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.011+0000 7fe51759e640 1 --2- 172.21.15.28:0/930878946 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe5040a2d80 0x7fe5040a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.011+0000 7fe51759e640 1 --2- 172.21.15.28:0/368611189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94c1f0 0x55843c94c5d0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fe51000a430 tx=0x7fe510009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b12868c6e357998 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.011+0000 7fe50d7fa640 1 -- 172.21.15.28:0/368611189 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe510010bb0 con 0x55843c94c1f0 2024-04-25T01:39:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.011+0000 7fe50d7fa640 1 -- 172.21.15.28:0/368611189 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe510010d50 con 0x55843c94c1f0 2024-04-25T01:39:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.011+0000 7fe50d7fa640 1 -- 172.21.15.28:0/368611189 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5100149a0 con 0x55843c94c1f0 2024-04-25T01:39:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.012+0000 7fe51fe3a740 1 -- 172.21.15.28:0/368611189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94c1f0 msgr2=0x55843c94c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.012+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/368611189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94c1f0 0x55843c94c5d0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fe51000a430 tx=0x7fe510009650 comp rx=0 tx=0).stop 2024-04-25T01:39:14.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.012+0000 7fe51759e640 1 --2- 172.21.15.28:0/930878946 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe5040a2d80 0x7fe5040a5240 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe51001b000 tx=0x7fe510014c40 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:14.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.012+0000 7fe51fe3a740 1 -- 172.21.15.28:0/368611189 shutdown_connections 2024-04-25T01:39:14.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.012+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/368611189 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c959780 0x55843c95bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.012+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/368611189 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c94cba0 0x55843c959240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.012+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/368611189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94c1f0 0x55843c94c5d0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.012+0000 7fe51fe3a740 1 -- 172.21.15.28:0/368611189 >> 172.21.15.28:0/368611189 conn(0x55843c9488a0 msgr2=0x55843c948c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:14.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.012+0000 7fe51fe3a740 1 -- 172.21.15.28:0/368611189 shutdown_connections 2024-04-25T01:39:14.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.012+0000 7fe51fe3a740 1 -- 172.21.15.28:0/368611189 wait complete. 2024-04-25T01:39:14.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.012+0000 7fe51fe3a740 1 -- start start 2024-04-25T01:39:14.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.012+0000 7fe51fe3a740 1 Processor -- start 2024-04-25T01:39:14.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.012+0000 7fe51fe3a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c94c1f0 0x55843c958f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe51fe3a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c94cba0 0x55843c9562f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe51fe3a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c959780 0x55843c956830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe51fe3a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55843c95c570 con 0x55843c959780 2024-04-25T01:39:14.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe51fe3a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55843c86d050 con 0x55843c94c1f0 2024-04-25T01:39:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe51fe3a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55843c95c6f0 con 0x55843c94cba0 2024-04-25T01:39:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe51fe3a740 10 client.0 _ll_register_callbacks cb 0x55843c5c4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe51759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c959780 0x55843c956830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe51759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c959780 0x55843c956830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48870/0 (socket says 172.21.15.28:48870) 2024-04-25T01:39:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe51759e640 1 -- 172.21.15.28:0/3019727304 learned_addr learned my addr 172.21.15.28:0/3019727304 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe50ffff640 1 --2- 172.21.15.28:0/3019727304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c94cba0 0x55843c9562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe516d9d640 1 --2- 172.21.15.28:0/3019727304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c94c1f0 0x55843c958f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe51759e640 1 -- 172.21.15.28:0/3019727304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c94cba0 msgr2=0x55843c9562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe51759e640 1 --2- 172.21.15.28:0/3019727304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c94cba0 0x55843c9562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe51759e640 1 -- 172.21.15.28:0/3019727304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c94c1f0 msgr2=0x55843c958f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe51759e640 1 --2- 172.21.15.28:0/3019727304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c94c1f0 0x55843c958f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe51759e640 1 -- 172.21.15.28:0/3019727304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55843c83edc0 con 0x55843c959780 2024-04-25T01:39:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe50ffff640 1 --2- 172.21.15.28:0/3019727304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c94cba0 0x55843c9562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe516d9d640 1 --2- 172.21.15.28:0/3019727304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c94c1f0 0x55843c958f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe51759e640 1 --2- 172.21.15.28:0/3019727304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c959780 0x55843c956830 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fe510014460 tx=0x7fe510022040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ccf97744e89d1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe4fe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe4fe7fc640 1 -- 172.21.15.28:0/3019727304 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe510014860 con 0x55843c959780 2024-04-25T01:39:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe4fe7fc640 1 -- 172.21.15.28:0/3019727304 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fe51001e940 con 0x55843c959780 2024-04-25T01:39:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.013+0000 7fe51fe3a740 1 -- 172.21.15.28:0/3019727304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe510010d50 con 0x55843c959780 2024-04-25T01:39:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.014+0000 7fe4fe7fc640 1 -- 172.21.15.28:0/3019727304 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe51003e050 con 0x55843c959780 2024-04-25T01:39:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.014+0000 7fe51fe3a740 10 client.17763 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.014+0000 7fe51fe3a740 1 -- 172.21.15.28:0/3019727304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55843c5c53f0 con 0x55843c959780 2024-04-25T01:39:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.014+0000 7fe4fe7fc640 1 -- 172.21.15.28:0/3019727304 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe510010d50 con 0x55843c959780 2024-04-25T01:39:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.014+0000 7fe4fe7fc640 1 -- 172.21.15.28:0/3019727304 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe510020900 con 0x55843c959780 2024-04-25T01:39:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.014+0000 7fe51fe3a740 10 client.17763 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.014+0000 7fe51fe3a740 10 client.17763 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:39:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.014+0000 7fe51fe3a740 20 client.17763 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:14.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.014+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/3019727304 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55843c9696f0 0x55843c96bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.014+0000 7fe51fe3a740 4 client.17763 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:14.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.014+0000 7fe51fe3a740 1 -- 172.21.15.28:0/3019727304 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55843c8bebb0 con 0x55843c9696f0 2024-04-25T01:39:14.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.014+0000 7fe516d9d640 1 --2- 172.21.15.28:0/3019727304 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55843c9696f0 0x55843c96bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.015+0000 7fe4fe7fc640 10 client.17763 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:39:14.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.015+0000 7fe516d9d640 1 --2- 172.21.15.28:0/3019727304 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55843c9696f0 0x55843c96bae0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ddcf4d793295caa server_cookie=c609f28b560098ce in_seq=0 out_seq=0 2024-04-25T01:39:14.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.015+0000 7fe4fe7fc640 1 -- 172.21.15.28:0/3019727304 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55843c8bebb0 con 0x55843c9696f0 2024-04-25T01:39:14.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.015+0000 7fe4fe7fc640 10 client.17763 handle_command_reply: tid=0 2024-04-25T01:39:14.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.016+0000 7fe51fe3a740 1 client.17763 shutdown 2024-04-25T01:39:14.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.016+0000 7fe51fe3a740 1 -- 172.21.15.28:0/3019727304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c959780 msgr2=0x55843c956830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.016+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/3019727304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c959780 0x55843c956830 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fe510014460 tx=0x7fe510022040 comp rx=0 tx=0).stop 2024-04-25T01:39:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.016+0000 7fe51fe3a740 1 -- 172.21.15.28:0/3019727304 shutdown_connections 2024-04-25T01:39:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.016+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/3019727304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c959780 0x55843c956830 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.016+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/3019727304 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55843c9696f0 0x55843c96bae0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.016+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/3019727304 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c94cba0 0x55843c9562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.016+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/3019727304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c94c1f0 0x55843c958f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.016+0000 7fe4fe7fc640 0 client.17763 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:39:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.016+0000 7fe51fe3a740 1 -- 172.21.15.28:0/3019727304 >> 172.21.15.28:0/3019727304 conn(0x55843c9488a0 msgr2=0x55843c947b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.016+0000 7fe51fe3a740 1 -- 172.21.15.28:0/3019727304 shutdown_connections 2024-04-25T01:39:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.016+0000 7fe51fe3a740 1 -- 172.21.15.28:0/3019727304 wait complete. 2024-04-25T01:39:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.016+0000 7fe51fe3a740 20 client.17763 trim_cache size 0 max 16384 2024-04-25T01:39:14.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.029+0000 7fe51fe3a740 1 Processor -- start 2024-04-25T01:39:14.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.029+0000 7fe51fe3a740 1 -- start start 2024-04-25T01:39:14.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.030+0000 7fe51fe3a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94c1f0 0x55843c965470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.030+0000 7fe51fe3a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c94cba0 0x55843c9712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.030+0000 7fe51fe3a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c959780 0x55843c9717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.030+0000 7fe51fe3a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe5181ac8e0 con 0x55843c94c1f0 2024-04-25T01:39:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.030+0000 7fe51fe3a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe5181ae420 con 0x55843c94cba0 2024-04-25T01:39:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.030+0000 7fe51fe3a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe5181acfb0 con 0x55843c959780 2024-04-25T01:39:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.030+0000 7fe516d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94c1f0 0x55843c965470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.030+0000 7fe516d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94c1f0 0x55843c965470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48874/0 (socket says 172.21.15.28:48874) 2024-04-25T01:39:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.030+0000 7fe516d9d640 1 -- 172.21.15.28:0/361637344 learned_addr learned my addr 172.21.15.28:0/361637344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.030+0000 7fe50ffff640 1 --2- 172.21.15.28:0/361637344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c94cba0 0x55843c9712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.030+0000 7fe516d9d640 1 -- 172.21.15.28:0/361637344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c959780 msgr2=0x55843c9717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.030+0000 7fe516d9d640 1 --2- 172.21.15.28:0/361637344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c959780 0x55843c9717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.030+0000 7fe516d9d640 1 -- 172.21.15.28:0/361637344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c94cba0 msgr2=0x55843c9712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.030+0000 7fe516d9d640 1 --2- 172.21.15.28:0/361637344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c94cba0 0x55843c9712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.030+0000 7fe516d9d640 1 -- 172.21.15.28:0/361637344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55843c94c940 con 0x55843c94c1f0 2024-04-25T01:39:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.030+0000 7fe50ffff640 1 --2- 172.21.15.28:0/361637344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c94cba0 0x55843c9712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.030+0000 7fe516d9d640 1 --2- 172.21.15.28:0/361637344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94c1f0 0x55843c965470 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fe5080cc520 tx=0x7fe5080bb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d384c5bc1598638 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:14.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.030+0000 7fe50cff9640 1 -- 172.21.15.28:0/361637344 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508020040 con 0x55843c94c1f0 2024-04-25T01:39:14.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.030+0000 7fe50cff9640 1 -- 172.21.15.28:0/361637344 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe50809b290 con 0x55843c94c1f0 2024-04-25T01:39:14.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.031+0000 7fe50cff9640 1 -- 172.21.15.28:0/361637344 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5080cc740 con 0x55843c94c1f0 2024-04-25T01:39:14.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.031+0000 7fe51fe3a740 1 -- 172.21.15.28:0/361637344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94c1f0 msgr2=0x55843c965470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.031+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/361637344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94c1f0 0x55843c965470 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fe5080cc520 tx=0x7fe5080bb040 comp rx=0 tx=0).stop 2024-04-25T01:39:14.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.031+0000 7fe51fe3a740 1 -- 172.21.15.28:0/361637344 shutdown_connections 2024-04-25T01:39:14.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.031+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/361637344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c959780 0x55843c9717e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.031+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/361637344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c94cba0 0x55843c9712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.031+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/361637344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94c1f0 0x55843c965470 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.031+0000 7fe51fe3a740 1 -- 172.21.15.28:0/361637344 >> 172.21.15.28:0/361637344 conn(0x55843c9488a0 msgr2=0x55843c9583b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:14.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.031+0000 7fe51fe3a740 1 -- 172.21.15.28:0/361637344 shutdown_connections 2024-04-25T01:39:14.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.031+0000 7fe51fe3a740 1 -- 172.21.15.28:0/361637344 wait complete. 2024-04-25T01:39:14.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe51fe3a740 1 -- start start 2024-04-25T01:39:14.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe51fe3a740 1 Processor -- start 2024-04-25T01:39:14.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe51fe3a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c94c1f0 0x55843c961a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe51fe3a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94cba0 0x55843c95ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe51fe3a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c959780 0x55843c95d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe51fe3a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe5181add50 con 0x55843c94cba0 2024-04-25T01:39:14.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe51fe3a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe5181ad680 con 0x55843c94c1f0 2024-04-25T01:39:14.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe51fe3a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe5181aa8c0 con 0x55843c959780 2024-04-25T01:39:14.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe51fe3a740 10 client.0 _ll_register_callbacks cb 0x7fe510009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:14.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe50ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94cba0 0x55843c95ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe50ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94cba0 0x55843c95ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48880/0 (socket says 172.21.15.28:48880) 2024-04-25T01:39:14.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe50ffff640 1 -- 172.21.15.28:0/3104673140 learned_addr learned my addr 172.21.15.28:0/3104673140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:14.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe516d9d640 1 --2- 172.21.15.28:0/3104673140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c94c1f0 0x55843c961a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe50ffff640 1 -- 172.21.15.28:0/3104673140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c959780 msgr2=0x55843c95d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe50ffff640 1 --2- 172.21.15.28:0/3104673140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c959780 0x55843c95d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe50ffff640 1 -- 172.21.15.28:0/3104673140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c94c1f0 msgr2=0x55843c961a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe50ffff640 1 --2- 172.21.15.28:0/3104673140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c94c1f0 0x55843c961a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe50ffff640 1 -- 172.21.15.28:0/3104673140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55843c68f7d0 con 0x55843c94cba0 2024-04-25T01:39:14.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe50ffff640 1 --2- 172.21.15.28:0/3104673140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94cba0 0x55843c95ce80 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fe500006010 tx=0x7fe500005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b041ec08aee1491 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:14.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.032+0000 7fe4fe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:14.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.033+0000 7fe4fe7fc640 1 -- 172.21.15.28:0/3104673140 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe500035070 con 0x55843c94cba0 2024-04-25T01:39:14.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.033+0000 7fe4fe7fc640 1 -- 172.21.15.28:0/3104673140 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fe5000048c0 con 0x55843c94cba0 2024-04-25T01:39:14.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.033+0000 7fe51fe3a740 1 -- 172.21.15.28:0/3104673140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe50809b290 con 0x55843c94cba0 2024-04-25T01:39:14.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.033+0000 7fe4fe7fc640 1 -- 172.21.15.28:0/3104673140 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe50004d050 con 0x55843c94cba0 2024-04-25T01:39:14.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.033+0000 7fe51fe3a740 10 client.17769 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:14.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.033+0000 7fe51fe3a740 1 -- 172.21.15.28:0/3104673140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55843c9d1a40 con 0x55843c94cba0 2024-04-25T01:39:14.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.033+0000 7fe4fe7fc640 1 -- 172.21.15.28:0/3104673140 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe50809b290 con 0x55843c94cba0 2024-04-25T01:39:14.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.033+0000 7fe4fe7fc640 1 -- 172.21.15.28:0/3104673140 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe500042020 con 0x55843c94cba0 2024-04-25T01:39:14.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.033+0000 7fe51fe3a740 10 client.17769 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:14.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.033+0000 7fe51fe3a740 10 client.17769 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:39:14.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.033+0000 7fe51fe3a740 20 client.17769 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:14.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.033+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/3104673140 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55843c9d21f0 0x55843c9d4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.033+0000 7fe51fe3a740 4 client.17769 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:14.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.033+0000 7fe51fe3a740 1 -- 172.21.15.28:0/3104673140 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55843c8b8f00 con 0x55843c9d21f0 2024-04-25T01:39:14.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.033+0000 7fe516d9d640 1 --2- 172.21.15.28:0/3104673140 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55843c9d21f0 0x55843c9d4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.034+0000 7fe4fe7fc640 10 client.17769 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:39:14.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.034+0000 7fe516d9d640 1 --2- 172.21.15.28:0/3104673140 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55843c9d21f0 0x55843c9d4610 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b43a879b0c5249dd server_cookie=47faffa0bf5e5653 in_seq=0 out_seq=0 2024-04-25T01:39:14.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.034+0000 7fe4fe7fc640 1 -- 172.21.15.28:0/3104673140 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55843c8b8f00 con 0x55843c9d21f0 2024-04-25T01:39:14.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.034+0000 7fe4fe7fc640 10 client.17769 handle_command_reply: tid=0 2024-04-25T01:39:14.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.034+0000 7fe51fe3a740 1 client.17769 shutdown 2024-04-25T01:39:14.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.034+0000 7fe51fe3a740 1 -- 172.21.15.28:0/3104673140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94cba0 msgr2=0x55843c95ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.034+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/3104673140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94cba0 0x55843c95ce80 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fe500006010 tx=0x7fe500005250 comp rx=0 tx=0).stop 2024-04-25T01:39:14.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.034+0000 7fe51fe3a740 1 -- 172.21.15.28:0/3104673140 shutdown_connections 2024-04-25T01:39:14.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.034+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/3104673140 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55843c9d21f0 0x55843c9d4610 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.034+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/3104673140 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55843c959780 0x55843c95d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.034+0000 7fe4fe7fc640 0 client.17769 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:39:14.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.034+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/3104673140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55843c94cba0 0x55843c95ce80 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.034+0000 7fe51fe3a740 1 --2- 172.21.15.28:0/3104673140 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55843c94c1f0 0x55843c961a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.034+0000 7fe51fe3a740 1 -- 172.21.15.28:0/3104673140 >> 172.21.15.28:0/3104673140 conn(0x55843c9488a0 msgr2=0x55843c9472b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:14.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.034+0000 7fe51fe3a740 1 -- 172.21.15.28:0/3104673140 shutdown_connections 2024-04-25T01:39:14.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.034+0000 7fe51fe3a740 1 -- 172.21.15.28:0/3104673140 wait complete. 2024-04-25T01:39:14.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.034+0000 7fe51fe3a740 20 client.17769 trim_cache size 0 max 16384 2024-04-25T01:39:14.023 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:14.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.035+0000 7fe51d129640 1 -- 172.21.15.28:0/930878946 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe5040a2d80 msgr2=0x7fe5040a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.035+0000 7fe51d129640 1 --2- 172.21.15.28:0/930878946 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe5040a2d80 0x7fe5040a5240 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe51001b000 tx=0x7fe510014c40 comp rx=0 tx=0).stop 2024-04-25T01:39:14.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.035+0000 7fe51d129640 1 -- 172.21.15.28:0/930878946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe518100610 msgr2=0x7fe518196f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.035+0000 7fe51d129640 1 --2- 172.21.15.28:0/930878946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe518100610 0x7fe518196f00 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fe50800d770 tx=0x7fe50800dc30 comp rx=0 tx=0).stop 2024-04-25T01:39:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.035+0000 7fe51d129640 1 -- 172.21.15.28:0/930878946 shutdown_connections 2024-04-25T01:39:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.035+0000 7fe51d129640 1 --2- 172.21.15.28:0/930878946 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe5040a2d80 0x7fe5040a5240 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.035+0000 7fe51d129640 1 --2- 172.21.15.28:0/930878946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5181066b0 0x7fe51819b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.035+0000 7fe51d129640 1 --2- 172.21.15.28:0/930878946 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe518100fd0 0x7fe518197440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.035+0000 7fe51d129640 1 --2- 172.21.15.28:0/930878946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe518100610 0x7fe518196f00 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.035+0000 7fe51d129640 1 -- 172.21.15.28:0/930878946 >> 172.21.15.28:0/930878946 conn(0x7fe518077740 msgr2=0x7fe5180ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.035+0000 7fe51d129640 1 -- 172.21.15.28:0/930878946 shutdown_connections 2024-04-25T01:39:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.035+0000 7fe51d129640 1 -- 172.21.15.28:0/930878946 wait complete. 2024-04-25T01:39:14.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:39:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.150+0000 7f8df6cc3640 1 -- 172.21.15.28:0/3345897718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df01020d0 msgr2=0x7f8df010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.150+0000 7f8df6cc3640 1 --2- 172.21.15.28:0/3345897718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df01020d0 0x7f8df010e5e0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009cd0 tx=0x7f8de802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.150+0000 7f8df6cc3640 1 -- 172.21.15.28:0/3345897718 shutdown_connections 2024-04-25T01:39:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.150+0000 7f8df6cc3640 1 --2- 172.21.15.28:0/3345897718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df01020d0 0x7f8df010e5e0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.150+0000 7f8df6cc3640 1 --2- 172.21.15.28:0/3345897718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8df0101710 0x7f8df0101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.150+0000 7f8df6cc3640 1 --2- 172.21.15.28:0/3345897718 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8df0107720 0x7f8df0107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.150+0000 7f8df6cc3640 1 -- 172.21.15.28:0/3345897718 >> 172.21.15.28:0/3345897718 conn(0x7f8df00fd180 msgr2=0x7f8df00ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.151+0000 7f8df6cc3640 1 -- 172.21.15.28:0/3345897718 shutdown_connections 2024-04-25T01:39:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.151+0000 7f8df6cc3640 1 -- 172.21.15.28:0/3345897718 wait complete. 2024-04-25T01:39:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.151+0000 7f8df6cc3640 1 Processor -- start 2024-04-25T01:39:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.151+0000 7f8df6cc3640 1 -- start start 2024-04-25T01:39:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.151+0000 7f8df6cc3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8df0101710 0x7f8df0198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.151+0000 7f8df6cc3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df01020d0 0x7f8df0198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.151+0000 7f8df6cc3640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8df0107720 0x7f8df019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.151+0000 7f8df6cc3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8df0111b30 con 0x7f8df01020d0 2024-04-25T01:39:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.151+0000 7f8df6cc3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8df01119b0 con 0x7f8df0101710 2024-04-25T01:39:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.151+0000 7f8df6cc3640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8df0111cb0 con 0x7f8df0107720 2024-04-25T01:39:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.151+0000 7f8de7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df01020d0 0x7f8df0198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.151+0000 7f8de7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df01020d0 0x7f8df0198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48898/0 (socket says 172.21.15.28:48898) 2024-04-25T01:39:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.151+0000 7f8de7fff640 1 -- 172.21.15.28:0/1619132312 learned_addr learned my addr 172.21.15.28:0/1619132312 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.152+0000 7f8df5260640 1 --2- 172.21.15.28:0/1619132312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8df0107720 0x7f8df019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.152+0000 7f8df4a5f640 1 --2- 172.21.15.28:0/1619132312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8df0101710 0x7f8df0198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.152+0000 7f8de7fff640 1 -- 172.21.15.28:0/1619132312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8df0107720 msgr2=0x7f8df019c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.152+0000 7f8de7fff640 1 --2- 172.21.15.28:0/1619132312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8df0107720 0x7f8df019c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.152+0000 7f8de7fff640 1 -- 172.21.15.28:0/1619132312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8df0101710 msgr2=0x7f8df0198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.152+0000 7f8de7fff640 1 --2- 172.21.15.28:0/1619132312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8df0101710 0x7f8df0198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.152+0000 7f8de7fff640 1 -- 172.21.15.28:0/1619132312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df019d000 con 0x7f8df01020d0 2024-04-25T01:39:14.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.152+0000 7f8df4a5f640 1 --2- 172.21.15.28:0/1619132312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8df0101710 0x7f8df0198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:14.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.152+0000 7f8de7fff640 1 --2- 172.21.15.28:0/1619132312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df01020d0 0x7f8df0198550 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f8de000d950 tx=0x7f8de000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:14.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.152+0000 7f8de5ffb640 1 -- 172.21.15.28:0/1619132312 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de0017070 con 0x7f8df01020d0 2024-04-25T01:39:14.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.152+0000 7f8df6cc3640 1 -- 172.21.15.28:0/1619132312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df019d2f0 con 0x7f8df01020d0 2024-04-25T01:39:14.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.152+0000 7f8de5ffb640 1 -- 172.21.15.28:0/1619132312 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8de0009c30 con 0x7f8df01020d0 2024-04-25T01:39:14.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.152+0000 7f8de5ffb640 1 -- 172.21.15.28:0/1619132312 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de000f6b0 con 0x7f8df01020d0 2024-04-25T01:39:14.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.152+0000 7f8df6cc3640 1 -- 172.21.15.28:0/1619132312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df01a4b80 con 0x7f8df01020d0 2024-04-25T01:39:14.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.154+0000 7f8de5ffb640 1 -- 172.21.15.28:0/1619132312 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8de00109d0 con 0x7f8df01020d0 2024-04-25T01:39:14.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.155+0000 7f8df99cc740 1 Processor -- start 2024-04-25T01:39:14.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.155+0000 7f8df99cc740 1 -- start start 2024-04-25T01:39:14.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.155+0000 7f8de5ffb640 1 --2- 172.21.15.28:0/1619132312 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8dd80a2d00 0x7f8dd80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.155+0000 7f8df99cc740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff542b0 0x56032ff54690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.155+0000 7f8df99cc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ff54c60 0x56032ff63440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.155+0000 7f8df99cc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 0x56032ff65d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.155+0000 7f8df99cc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56032feb9670 con 0x56032ff63980 2024-04-25T01:39:14.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.155+0000 7f8df99cc740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56032fe61970 con 0x56032ff542b0 2024-04-25T01:39:14.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.155+0000 7f8df99cc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56032feb5630 con 0x56032ff54c60 2024-04-25T01:39:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.155+0000 7f8de5ffb640 1 -- 172.21.15.28:0/1619132312 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8de00cf4b0 con 0x7f8df01020d0 2024-04-25T01:39:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.155+0000 7f8df5260640 1 --2- 172.21.15.28:0/1619132312 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8dd80a2d00 0x7f8dd80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.155+0000 7f8df5260640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 0x56032ff65d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.155+0000 7f8df5260640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 0x56032ff65d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48900/0 (socket says 172.21.15.28:48900) 2024-04-25T01:39:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.155+0000 7f8df5260640 1 -- 172.21.15.28:0/1292988868 learned_addr learned my addr 172.21.15.28:0/1292988868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.155+0000 7f8df4a5f640 1 --2- 172.21.15.28:0/1292988868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff542b0 0x56032ff54690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.155+0000 7f8de7fff640 1 --2- 172.21.15.28:0/1292988868 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ff54c60 0x56032ff63440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.156+0000 7f8df4a5f640 1 -- 172.21.15.28:0/1292988868 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ff54c60 msgr2=0x56032ff63440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.156+0000 7f8df4a5f640 1 --2- 172.21.15.28:0/1292988868 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ff54c60 0x56032ff63440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.156+0000 7f8df4a5f640 1 -- 172.21.15.28:0/1292988868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 msgr2=0x56032ff65d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.156+0000 7f8df4a5f640 1 --2- 172.21.15.28:0/1292988868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 0x56032ff65d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.156+0000 7f8df4a5f640 1 -- 172.21.15.28:0/1292988868 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56032fe85400 con 0x56032ff542b0 2024-04-25T01:39:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.156+0000 7f8df5260640 1 --2- 172.21.15.28:0/1292988868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 0x56032ff65d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:39:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.156+0000 7f8df5260640 1 --2- 172.21.15.28:0/1619132312 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8dd80a2d00 0x7f8dd80a51c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009a60 tx=0x7f8de8033000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.156+0000 7f8df4a5f640 1 --2- 172.21.15.28:0/1292988868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff542b0 0x56032ff54690 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8dec009be0 tx=0x7f8dec0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b047e4bb3ed69110 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.156+0000 7f8dd6ffd640 1 -- 172.21.15.28:0/1292988868 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec005020 con 0x56032ff542b0 2024-04-25T01:39:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.156+0000 7f8dd6ffd640 1 -- 172.21.15.28:0/1292988868 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dec0047a0 con 0x56032ff542b0 2024-04-25T01:39:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.156+0000 7f8dd6ffd640 1 -- 172.21.15.28:0/1292988868 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dec0126b0 con 0x56032ff542b0 2024-04-25T01:39:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.157+0000 7f8df99cc740 1 -- 172.21.15.28:0/1292988868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff542b0 msgr2=0x56032ff54690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.157+0000 7f8df99cc740 1 --2- 172.21.15.28:0/1292988868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff542b0 0x56032ff54690 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8dec009be0 tx=0x7f8dec0117b0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.157+0000 7f8df99cc740 1 -- 172.21.15.28:0/1292988868 shutdown_connections 2024-04-25T01:39:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.157+0000 7f8df99cc740 1 --2- 172.21.15.28:0/1292988868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 0x56032ff65d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.157+0000 7f8df99cc740 1 --2- 172.21.15.28:0/1292988868 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ff54c60 0x56032ff63440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.157+0000 7f8df99cc740 1 --2- 172.21.15.28:0/1292988868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff542b0 0x56032ff54690 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.157+0000 7f8df99cc740 1 -- 172.21.15.28:0/1292988868 >> 172.21.15.28:0/1292988868 conn(0x56032ff508a0 msgr2=0x56032ff50c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.157+0000 7f8df99cc740 1 -- 172.21.15.28:0/1292988868 shutdown_connections 2024-04-25T01:39:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.157+0000 7f8df99cc740 1 -- 172.21.15.28:0/1292988868 wait complete. 2024-04-25T01:39:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.157+0000 7f8df99cc740 1 -- start start 2024-04-25T01:39:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.157+0000 7f8df99cc740 1 Processor -- start 2024-04-25T01:39:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8df99cc740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff542b0 0x56032ff63290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8df99cc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ff54c60 0x56032ff60580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8df99cc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 0x56032ff60ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8df99cc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56032ff66740 con 0x56032ff63980 2024-04-25T01:39:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8df99cc740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56032fe75050 con 0x56032ff542b0 2024-04-25T01:39:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8df99cc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56032ff668c0 con 0x56032ff54c60 2024-04-25T01:39:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8df99cc740 10 client.0 _ll_register_callbacks cb 0x56032fbccbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8df5260640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 0x56032ff60ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8df5260640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 0x56032ff60ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48904/0 (socket says 172.21.15.28:48904) 2024-04-25T01:39:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8df5260640 1 -- 172.21.15.28:0/1955271047 learned_addr learned my addr 172.21.15.28:0/1955271047 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8df4a5f640 1 --2- 172.21.15.28:0/1955271047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff542b0 0x56032ff63290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8de7fff640 1 --2- 172.21.15.28:0/1955271047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ff54c60 0x56032ff60580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8df5260640 1 -- 172.21.15.28:0/1955271047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ff54c60 msgr2=0x56032ff60580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8df5260640 1 --2- 172.21.15.28:0/1955271047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ff54c60 0x56032ff60580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8df5260640 1 -- 172.21.15.28:0/1955271047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff542b0 msgr2=0x56032ff63290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8df5260640 1 --2- 172.21.15.28:0/1955271047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff542b0 0x56032ff63290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8df5260640 1 -- 172.21.15.28:0/1955271047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56032fe46dc0 con 0x56032ff63980 2024-04-25T01:39:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8df5260640 1 --2- 172.21.15.28:0/1955271047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 0x56032ff60ac0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f8de8003c70 tx=0x7f8de8047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da6fa70246d6d18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8dcffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8dcffff640 1 -- 172.21.15.28:0/1955271047 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de804f070 con 0x56032ff63980 2024-04-25T01:39:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8dcffff640 1 -- 172.21.15.28:0/1955271047 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8de8004640 con 0x56032ff63980 2024-04-25T01:39:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.158+0000 7f8df99cc740 1 -- 172.21.15.28:0/1955271047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8dec0047a0 con 0x56032ff63980 2024-04-25T01:39:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.159+0000 7f8dcffff640 1 -- 172.21.15.28:0/1955271047 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8de805b050 con 0x56032ff63980 2024-04-25T01:39:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.159+0000 7f8df99cc740 10 client.17781 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.159+0000 7f8df99cc740 1 -- 172.21.15.28:0/1955271047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56032fbcd3f0 con 0x56032ff63980 2024-04-25T01:39:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.159+0000 7f8dcffff640 1 -- 172.21.15.28:0/1955271047 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dec0047a0 con 0x56032ff63980 2024-04-25T01:39:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.159+0000 7f8dcffff640 1 -- 172.21.15.28:0/1955271047 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8de8049040 con 0x56032ff63980 2024-04-25T01:39:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.159+0000 7f8df99cc740 10 client.17781 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.159+0000 7f8df99cc740 10 client.17781 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:39:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.159+0000 7f8df99cc740 20 client.17781 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.159+0000 7f8df99cc740 1 --2- 172.21.15.28:0/1955271047 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56032ff739b0 0x56032ff75e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.159+0000 7f8df99cc740 4 client.17781 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.159+0000 7f8df99cc740 1 -- 172.21.15.28:0/1955271047 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56032fec6bb0 con 0x56032ff739b0 2024-04-25T01:39:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.159+0000 7f8df4a5f640 1 --2- 172.21.15.28:0/1955271047 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56032ff739b0 0x56032ff75e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.160+0000 7f8df4a5f640 1 --2- 172.21.15.28:0/1955271047 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56032ff739b0 0x56032ff75e00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6ee66891f61eaa34 server_cookie=1a78d4e3fbde92b6 in_seq=0 out_seq=0 2024-04-25T01:39:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.160+0000 7f8dcffff640 10 client.17781 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:39:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.160+0000 7f8dcffff640 1 -- 172.21.15.28:0/1955271047 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56032fec6bb0 con 0x56032ff739b0 2024-04-25T01:39:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.160+0000 7f8dcffff640 10 client.17781 handle_command_reply: tid=0 2024-04-25T01:39:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.160+0000 7f8df99cc740 1 client.17781 shutdown 2024-04-25T01:39:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.161+0000 7f8df99cc740 1 -- 172.21.15.28:0/1955271047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 msgr2=0x56032ff60ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.161+0000 7f8df99cc740 1 --2- 172.21.15.28:0/1955271047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 0x56032ff60ac0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f8de8003c70 tx=0x7f8de8047000 comp rx=0 tx=0).stop 2024-04-25T01:39:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.161+0000 7f8df99cc740 1 -- 172.21.15.28:0/1955271047 shutdown_connections 2024-04-25T01:39:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.161+0000 7f8df99cc740 1 --2- 172.21.15.28:0/1955271047 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56032ff739b0 0x56032ff75e00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.161+0000 7f8df99cc740 1 --2- 172.21.15.28:0/1955271047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 0x56032ff60ac0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.161+0000 7f8df99cc740 1 --2- 172.21.15.28:0/1955271047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ff54c60 0x56032ff60580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.161+0000 7f8dcffff640 0 client.17781 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:39:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.161+0000 7f8df99cc740 1 --2- 172.21.15.28:0/1955271047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff542b0 0x56032ff63290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.161+0000 7f8df99cc740 1 -- 172.21.15.28:0/1955271047 >> 172.21.15.28:0/1955271047 conn(0x56032ff508a0 msgr2=0x56032ff56710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.161+0000 7f8df99cc740 1 -- 172.21.15.28:0/1955271047 shutdown_connections 2024-04-25T01:39:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.161+0000 7f8df99cc740 1 -- 172.21.15.28:0/1955271047 wait complete. 2024-04-25T01:39:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.161+0000 7f8df99cc740 20 client.17781 trim_cache size 0 max 16384 2024-04-25T01:39:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.174+0000 7f8df99cc740 1 Processor -- start 2024-04-25T01:39:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.174+0000 7f8df99cc740 1 -- start start 2024-04-25T01:39:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.174+0000 7f8df99cc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ff542b0 0x56032ff5bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.174+0000 7f8df99cc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff54c60 0x56032ff62dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.174+0000 7f8df99cc740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff63980 0x56032ff56570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.174+0000 7f8df99cc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8df01ac860 con 0x56032ff54c60 2024-04-25T01:39:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.174+0000 7f8df99cc740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8df01ae3a0 con 0x56032ff63980 2024-04-25T01:39:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.174+0000 7f8df99cc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8df01acf30 con 0x56032ff542b0 2024-04-25T01:39:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.174+0000 7f8de7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff54c60 0x56032ff62dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.174+0000 7f8de7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff54c60 0x56032ff62dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48908/0 (socket says 172.21.15.28:48908) 2024-04-25T01:39:14.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.174+0000 7f8de7fff640 1 -- 172.21.15.28:0/2009710359 learned_addr learned my addr 172.21.15.28:0/2009710359 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:14.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.175+0000 7f8df5260640 1 --2- 172.21.15.28:0/2009710359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff63980 0x56032ff56570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.175+0000 7f8de7fff640 1 -- 172.21.15.28:0/2009710359 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ff542b0 msgr2=0x56032ff5bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.175+0000 7f8de7fff640 1 --2- 172.21.15.28:0/2009710359 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ff542b0 0x56032ff5bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.175+0000 7f8de7fff640 1 -- 172.21.15.28:0/2009710359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff63980 msgr2=0x56032ff56570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.175+0000 7f8de7fff640 1 --2- 172.21.15.28:0/2009710359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff63980 0x56032ff56570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.175+0000 7f8de7fff640 1 -- 172.21.15.28:0/2009710359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56032fc977d0 con 0x56032ff54c60 2024-04-25T01:39:14.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.175+0000 7f8de7fff640 1 --2- 172.21.15.28:0/2009710359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff54c60 0x56032ff62dc0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f8de000dfe0 tx=0x7f8de00978d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8652b310258b47f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:14.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.175+0000 7f8dd67fc640 1 -- 172.21.15.28:0/2009710359 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de00b3070 con 0x56032ff54c60 2024-04-25T01:39:14.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.175+0000 7f8dd67fc640 1 -- 172.21.15.28:0/2009710359 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8de009b630 con 0x56032ff54c60 2024-04-25T01:39:14.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.175+0000 7f8dd67fc640 1 -- 172.21.15.28:0/2009710359 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de0020040 con 0x56032ff54c60 2024-04-25T01:39:14.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.175+0000 7f8df99cc740 1 -- 172.21.15.28:0/2009710359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff54c60 msgr2=0x56032ff62dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.175+0000 7f8df99cc740 1 --2- 172.21.15.28:0/2009710359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff54c60 0x56032ff62dc0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f8de000dfe0 tx=0x7f8de00978d0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.176+0000 7f8df99cc740 1 -- 172.21.15.28:0/2009710359 shutdown_connections 2024-04-25T01:39:14.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.176+0000 7f8df99cc740 1 --2- 172.21.15.28:0/2009710359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff63980 0x56032ff56570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.176+0000 7f8df99cc740 1 --2- 172.21.15.28:0/2009710359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff54c60 0x56032ff62dc0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.176+0000 7f8df99cc740 1 --2- 172.21.15.28:0/2009710359 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ff542b0 0x56032ff5bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.176+0000 7f8df99cc740 1 -- 172.21.15.28:0/2009710359 >> 172.21.15.28:0/2009710359 conn(0x56032ff508a0 msgr2=0x56032ff55e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:14.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.176+0000 7f8df99cc740 1 -- 172.21.15.28:0/2009710359 shutdown_connections 2024-04-25T01:39:14.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.176+0000 7f8df99cc740 1 -- 172.21.15.28:0/2009710359 wait complete. 2024-04-25T01:39:14.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.176+0000 7f8df99cc740 1 -- start start 2024-04-25T01:39:14.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.176+0000 7f8df99cc740 1 Processor -- start 2024-04-25T01:39:14.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.176+0000 7f8df99cc740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff542b0 0x56032ff5d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.176+0000 7f8df99cc740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ff54c60 0x56032ff5dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.176+0000 7f8df99cc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 0x56032ff5e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.176+0000 7f8df99cc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8df01adcd0 con 0x56032ff63980 2024-04-25T01:39:14.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.176+0000 7f8df99cc740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8df01ad600 con 0x56032ff542b0 2024-04-25T01:39:14.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.176+0000 7f8df99cc740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8df01aa840 con 0x56032ff54c60 2024-04-25T01:39:14.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.176+0000 7f8df99cc740 10 client.0 _ll_register_callbacks cb 0x7f8de8009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:14.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.176+0000 7f8df5260640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 0x56032ff5e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.177+0000 7f8df5260640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 0x56032ff5e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48912/0 (socket says 172.21.15.28:48912) 2024-04-25T01:39:14.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.177+0000 7f8df5260640 1 -- 172.21.15.28:0/2285083522 learned_addr learned my addr 172.21.15.28:0/2285083522 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:14.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.177+0000 7f8df4a5f640 1 --2- 172.21.15.28:0/2285083522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff542b0 0x56032ff5d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.177+0000 7f8de7fff640 1 --2- 172.21.15.28:0/2285083522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ff54c60 0x56032ff5dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.177+0000 7f8df5260640 1 -- 172.21.15.28:0/2285083522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ff54c60 msgr2=0x56032ff5dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.177+0000 7f8df5260640 1 --2- 172.21.15.28:0/2285083522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ff54c60 0x56032ff5dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.177+0000 7f8df5260640 1 -- 172.21.15.28:0/2285083522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff542b0 msgr2=0x56032ff5d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.177+0000 7f8df5260640 1 --2- 172.21.15.28:0/2285083522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff542b0 0x56032ff5d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.177+0000 7f8df5260640 1 -- 172.21.15.28:0/2285083522 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56032ffd9210 con 0x56032ff63980 2024-04-25T01:39:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.177+0000 7f8df5260640 1 --2- 172.21.15.28:0/2285083522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 0x56032ff5e030 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f8de803c500 tx=0x7f8de8004870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986a907816bd9292 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.177+0000 7f8dcffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.177+0000 7f8dcffff640 1 -- 172.21.15.28:0/2285083522 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8058020 con 0x56032ff63980 2024-04-25T01:39:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.177+0000 7f8dcffff640 1 -- 172.21.15.28:0/2285083522 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8de8031430 con 0x56032ff63980 2024-04-25T01:39:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.177+0000 7f8df99cc740 1 -- 172.21.15.28:0/2285083522 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8de009b630 con 0x56032ff63980 2024-04-25T01:39:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.177+0000 7f8dcffff640 1 -- 172.21.15.28:0/2285083522 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8de804f040 con 0x56032ff63980 2024-04-25T01:39:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.177+0000 7f8df99cc740 10 client.17787 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.177+0000 7f8df99cc740 1 -- 172.21.15.28:0/2285083522 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56032ffda510 con 0x56032ff63980 2024-04-25T01:39:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.177+0000 7f8dcffff640 1 -- 172.21.15.28:0/2285083522 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8de009b630 con 0x56032ff63980 2024-04-25T01:39:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.178+0000 7f8dcffff640 1 -- 172.21.15.28:0/2285083522 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8de804a070 con 0x56032ff63980 2024-04-25T01:39:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.178+0000 7f8df99cc740 10 client.17787 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.178+0000 7f8df99cc740 10 client.17787 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:39:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.178+0000 7f8df99cc740 20 client.17787 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.178+0000 7f8df99cc740 1 --2- 172.21.15.28:0/2285083522 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56032ffdac50 0x56032ffdd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.178+0000 7f8df99cc740 4 client.17787 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.178+0000 7f8df99cc740 1 -- 172.21.15.28:0/2285083522 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56032ff60e50 con 0x56032ffdac50 2024-04-25T01:39:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.178+0000 7f8df4a5f640 1 --2- 172.21.15.28:0/2285083522 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56032ffdac50 0x56032ffdd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.178+0000 7f8dcffff640 10 client.17787 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:39:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.178+0000 7f8df4a5f640 1 --2- 172.21.15.28:0/2285083522 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56032ffdac50 0x56032ffdd070 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dd68baf397f45a22 server_cookie=52ce17d8e5c84b43 in_seq=0 out_seq=0 2024-04-25T01:39:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.178+0000 7f8dcffff640 1 -- 172.21.15.28:0/2285083522 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56032ff60e50 con 0x56032ffdac50 2024-04-25T01:39:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.178+0000 7f8dcffff640 10 client.17787 handle_command_reply: tid=0 2024-04-25T01:39:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.178+0000 7f8df99cc740 1 client.17787 shutdown 2024-04-25T01:39:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df99cc740 1 -- 172.21.15.28:0/2285083522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 msgr2=0x56032ff5e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df99cc740 1 --2- 172.21.15.28:0/2285083522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 0x56032ff5e030 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f8de803c500 tx=0x7f8de8004870 comp rx=0 tx=0).stop 2024-04-25T01:39:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df99cc740 1 -- 172.21.15.28:0/2285083522 shutdown_connections 2024-04-25T01:39:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df99cc740 1 --2- 172.21.15.28:0/2285083522 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56032ffdac50 0x56032ffdd070 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df99cc740 1 --2- 172.21.15.28:0/2285083522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56032ff63980 0x56032ff5e030 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df99cc740 1 --2- 172.21.15.28:0/2285083522 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56032ff54c60 0x56032ff5dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8dcffff640 0 client.17787 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:39:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df99cc740 1 --2- 172.21.15.28:0/2285083522 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56032ff542b0 0x56032ff5d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df99cc740 1 -- 172.21.15.28:0/2285083522 >> 172.21.15.28:0/2285083522 conn(0x56032ff508a0 msgr2=0x56032ff5baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df99cc740 1 -- 172.21.15.28:0/2285083522 shutdown_connections 2024-04-25T01:39:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df99cc740 1 -- 172.21.15.28:0/2285083522 wait complete. 2024-04-25T01:39:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df99cc740 20 client.17787 trim_cache size 0 max 16384 2024-04-25T01:39:14.167 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df6cc3640 1 -- 172.21.15.28:0/1619132312 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8dd80a2d00 msgr2=0x7f8dd80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df6cc3640 1 --2- 172.21.15.28:0/1619132312 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8dd80a2d00 0x7f8dd80a51c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009a60 tx=0x7f8de8033000 comp rx=0 tx=0).stop 2024-04-25T01:39:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df6cc3640 1 -- 172.21.15.28:0/1619132312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df01020d0 msgr2=0x7f8df0198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df6cc3640 1 --2- 172.21.15.28:0/1619132312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df01020d0 0x7f8df0198550 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f8de000d950 tx=0x7f8de000de10 comp rx=0 tx=0).stop 2024-04-25T01:39:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df6cc3640 1 -- 172.21.15.28:0/1619132312 shutdown_connections 2024-04-25T01:39:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df6cc3640 1 --2- 172.21.15.28:0/1619132312 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8dd80a2d00 0x7f8dd80a51c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df6cc3640 1 --2- 172.21.15.28:0/1619132312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8df0107720 0x7f8df019c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df6cc3640 1 --2- 172.21.15.28:0/1619132312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df01020d0 0x7f8df0198550 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df6cc3640 1 --2- 172.21.15.28:0/1619132312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8df0101710 0x7f8df0198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.179+0000 7f8df6cc3640 1 -- 172.21.15.28:0/1619132312 >> 172.21.15.28:0/1619132312 conn(0x7f8df00fd180 msgr2=0x7f8df00fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.180+0000 7f8df6cc3640 1 -- 172.21.15.28:0/1619132312 shutdown_connections 2024-04-25T01:39:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.180+0000 7f8df6cc3640 1 -- 172.21.15.28:0/1619132312 wait complete. 2024-04-25T01:39:14.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:13 smithi028 ceph-mon[29390]: pgmap v189: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:39:14.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:13 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:39:14.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:13 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:14.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:13 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:39:14.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:13 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:14.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:13 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:39:14.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:13 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/433263823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:39:14.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:39:14.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:13 smithi032 ceph-mon[32461]: pgmap v189: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:39:14.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:13 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:39:14.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:13 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:14.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:13 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:39:14.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:13 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:14.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:13 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:39:14.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:13 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/433263823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:39:14.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.294+0000 7fdec8909640 1 -- 172.21.15.28:0/109302309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdec410df50 msgr2=0x7fdec41103a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.294+0000 7fdec8909640 1 --2- 172.21.15.28:0/109302309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdec410df50 0x7fdec41103a0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fdebc009ca0 tx=0x7fdebc02a950 comp rx=0 tx=0).stop 2024-04-25T01:39:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.294+0000 7fdec8909640 1 -- 172.21.15.28:0/109302309 shutdown_connections 2024-04-25T01:39:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.294+0000 7fdec8909640 1 --2- 172.21.15.28:0/109302309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdec410df50 0x7fdec41103a0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.294+0000 7fdec8909640 1 --2- 172.21.15.28:0/109302309 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdec4074490 0x7fdec410da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.294+0000 7fdec8909640 1 --2- 172.21.15.28:0/109302309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdec4073b70 0x7fdec4073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.294+0000 7fdec8909640 1 -- 172.21.15.28:0/109302309 >> 172.21.15.28:0/109302309 conn(0x7fdec40fd180 msgr2=0x7fdec40ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.294+0000 7fdec8909640 1 -- 172.21.15.28:0/109302309 shutdown_connections 2024-04-25T01:39:14.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.294+0000 7fdec8909640 1 -- 172.21.15.28:0/109302309 wait complete. 2024-04-25T01:39:14.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.295+0000 7fdec8909640 1 Processor -- start 2024-04-25T01:39:14.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.295+0000 7fdec8909640 1 -- start start 2024-04-25T01:39:14.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.295+0000 7fdec8909640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdec4073b70 0x7fdec419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.295+0000 7fdec8909640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdec4074490 0x7fdec419a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.295+0000 7fdec8909640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdec410df50 0x7fdec419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.295+0000 7fdec8909640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdec4113cf0 con 0x7fdec4073b70 2024-04-25T01:39:14.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.295+0000 7fdec8909640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdec4113b70 con 0x7fdec410df50 2024-04-25T01:39:14.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.295+0000 7fdec8909640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fdec4113e70 con 0x7fdec4074490 2024-04-25T01:39:14.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.295+0000 7fdec259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdec4073b70 0x7fdec419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.295+0000 7fdec259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdec4073b70 0x7fdec419a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48934/0 (socket says 172.21.15.28:48934) 2024-04-25T01:39:14.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.295+0000 7fdec259c640 1 -- 172.21.15.28:0/1878992681 learned_addr learned my addr 172.21.15.28:0/1878992681 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:14.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.296+0000 7fdebbfff640 1 --2- 172.21.15.28:0/1878992681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdec4074490 0x7fdec419a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.296+0000 7fdec2d9d640 1 --2- 172.21.15.28:0/1878992681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdec410df50 0x7fdec419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.296+0000 7fdec259c640 1 -- 172.21.15.28:0/1878992681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdec4074490 msgr2=0x7fdec419a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.296+0000 7fdec259c640 1 --2- 172.21.15.28:0/1878992681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdec4074490 0x7fdec419a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.296+0000 7fdec259c640 1 -- 172.21.15.28:0/1878992681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdec410df50 msgr2=0x7fdec419eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.296+0000 7fdec259c640 1 --2- 172.21.15.28:0/1878992681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdec410df50 0x7fdec419eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.296+0000 7fdec259c640 1 -- 172.21.15.28:0/1878992681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec419f260 con 0x7fdec4073b70 2024-04-25T01:39:14.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.296+0000 7fdebbfff640 1 --2- 172.21.15.28:0/1878992681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdec4074490 0x7fdec419a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:14.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.296+0000 7fdec259c640 1 --2- 172.21.15.28:0/1878992681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdec4073b70 0x7fdec419a270 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400e9f0 tx=0x7fdeb400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:14.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.296+0000 7fdebb7fe640 1 -- 172.21.15.28:0/1878992681 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb400cd80 con 0x7fdec4073b70 2024-04-25T01:39:14.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.296+0000 7fdebb7fe640 1 -- 172.21.15.28:0/1878992681 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdeb4010910 con 0x7fdec4073b70 2024-04-25T01:39:14.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.296+0000 7fdec8909640 1 -- 172.21.15.28:0/1878992681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdec419f550 con 0x7fdec4073b70 2024-04-25T01:39:14.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.296+0000 7fdebb7fe640 1 -- 172.21.15.28:0/1878992681 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4018990 con 0x7fdec4073b70 2024-04-25T01:39:14.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.296+0000 7fdec8909640 1 -- 172.21.15.28:0/1878992681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdec41a6de0 con 0x7fdec4073b70 2024-04-25T01:39:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.299+0000 7fdebb7fe640 1 -- 172.21.15.28:0/1878992681 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fdeb4010ac0 con 0x7fdec4073b70 2024-04-25T01:39:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.299+0000 7fdecb60c740 1 Processor -- start 2024-04-25T01:39:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.299+0000 7fdecb60c740 1 -- start start 2024-04-25T01:39:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.299+0000 7fdecb60c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7f580 0x555892d7f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.299+0000 7fdecb60c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d7fea0 0x555892d8a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.299+0000 7fdecb60c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d8a570 0x555892d8c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.300+0000 7fdecb60c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555892ce4670 con 0x555892d7fea0 2024-04-25T01:39:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.300+0000 7fdecb60c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555892c8c970 con 0x555892d8a570 2024-04-25T01:39:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.300+0000 7fdecb60c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x555892ce0630 con 0x555892d7f580 2024-04-25T01:39:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.300+0000 7fdec259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d7fea0 0x555892d8a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.300+0000 7fdec259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d7fea0 0x555892d8a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48942/0 (socket says 172.21.15.28:48942) 2024-04-25T01:39:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.300+0000 7fdec259c640 1 -- 172.21.15.28:0/686850312 learned_addr learned my addr 172.21.15.28:0/686850312 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.300+0000 7fdebb7fe640 1 --2- 172.21.15.28:0/1878992681 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdeb00a2d00 0x7fdeb00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.300+0000 7fdec2d9d640 1 --2- 172.21.15.28:0/686850312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7f580 0x555892d7f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.300+0000 7fdebbfff640 1 --2- 172.21.15.28:0/686850312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d8a570 0x555892d8c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.300+0000 7fdec2d9d640 1 --2- 172.21.15.28:0/1878992681 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdeb00a2d00 0x7fdeb00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.300+0000 7fdebb7fe640 1 -- 172.21.15.28:0/1878992681 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fdeb40cfbe0 con 0x7fdec4073b70 2024-04-25T01:39:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.300+0000 7fdec259c640 1 -- 172.21.15.28:0/686850312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7f580 msgr2=0x555892d7f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.300+0000 7fdec259c640 1 --2- 172.21.15.28:0/686850312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7f580 0x555892d7f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.300+0000 7fdec259c640 1 -- 172.21.15.28:0/686850312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d8a570 msgr2=0x555892d8c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.300+0000 7fdec259c640 1 --2- 172.21.15.28:0/686850312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d8a570 0x555892d8c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.300+0000 7fdec259c640 1 -- 172.21.15.28:0/686850312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555892cb0400 con 0x555892d7fea0 2024-04-25T01:39:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.300+0000 7fdec259c640 1 --2- 172.21.15.28:0/686850312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d7fea0 0x555892d8a030 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4098790 tx=0x7fdeb4018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d41dc5888cb4a55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.300+0000 7fdec2d9d640 1 --2- 172.21.15.28:0/1878992681 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdeb00a2d00 0x7fdeb00a51c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdebc005fa0 tx=0x7fdebc02a950 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:14.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.300+0000 7fdeb8ff9640 1 -- 172.21.15.28:0/686850312 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb40985d0 con 0x555892d7fea0 2024-04-25T01:39:14.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.301+0000 7fdeb8ff9640 1 -- 172.21.15.28:0/686850312 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdeb409bdc0 con 0x555892d7fea0 2024-04-25T01:39:14.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.301+0000 7fdeb8ff9640 1 -- 172.21.15.28:0/686850312 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4018b30 con 0x555892d7fea0 2024-04-25T01:39:14.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.301+0000 7fdecb60c740 1 -- 172.21.15.28:0/686850312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d7fea0 msgr2=0x555892d8a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.301+0000 7fdecb60c740 1 --2- 172.21.15.28:0/686850312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d7fea0 0x555892d8a030 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4098790 tx=0x7fdeb4018420 comp rx=0 tx=0).stop 2024-04-25T01:39:14.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.301+0000 7fdecb60c740 1 -- 172.21.15.28:0/686850312 shutdown_connections 2024-04-25T01:39:14.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.301+0000 7fdecb60c740 1 --2- 172.21.15.28:0/686850312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d8a570 0x555892d8c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.301+0000 7fdecb60c740 1 --2- 172.21.15.28:0/686850312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d7fea0 0x555892d8a030 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.301+0000 7fdecb60c740 1 --2- 172.21.15.28:0/686850312 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7f580 0x555892d7f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.301+0000 7fdecb60c740 1 -- 172.21.15.28:0/686850312 >> 172.21.15.28:0/686850312 conn(0x555892d7b8a0 msgr2=0x555892d7bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:14.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.301+0000 7fdecb60c740 1 -- 172.21.15.28:0/686850312 shutdown_connections 2024-04-25T01:39:14.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.301+0000 7fdecb60c740 1 -- 172.21.15.28:0/686850312 wait complete. 2024-04-25T01:39:14.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdecb60c740 1 -- start start 2024-04-25T01:39:14.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdecb60c740 1 Processor -- start 2024-04-25T01:39:14.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdecb60c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d7f580 0x555892d89ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdecb60c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7fea0 0x555892d871c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdecb60c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d8a570 0x555892d87720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdecb60c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555892d8d2a0 con 0x555892d7f580 2024-04-25T01:39:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdecb60c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555892ca0050 con 0x555892d8a570 2024-04-25T01:39:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdecb60c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x555892d8d420 con 0x555892d7fea0 2024-04-25T01:39:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdecb60c740 10 client.0 _ll_register_callbacks cb 0x5558929f7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdec259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d7f580 0x555892d89ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdec259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d7f580 0x555892d89ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48950/0 (socket says 172.21.15.28:48950) 2024-04-25T01:39:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdec259c640 1 -- 172.21.15.28:0/3838215678 learned_addr learned my addr 172.21.15.28:0/3838215678 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdebbfff640 1 --2- 172.21.15.28:0/3838215678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7fea0 0x555892d871c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdec2d9d640 1 --2- 172.21.15.28:0/3838215678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d8a570 0x555892d87720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdec259c640 1 -- 172.21.15.28:0/3838215678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7fea0 msgr2=0x555892d871c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdec259c640 1 --2- 172.21.15.28:0/3838215678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7fea0 0x555892d871c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdec259c640 1 -- 172.21.15.28:0/3838215678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d8a570 msgr2=0x555892d87720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdec259c640 1 --2- 172.21.15.28:0/3838215678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d8a570 0x555892d87720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdec259c640 1 -- 172.21.15.28:0/3838215678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555892c71dc0 con 0x555892d7f580 2024-04-25T01:39:14.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.302+0000 7fdec2d9d640 1 --2- 172.21.15.28:0/3838215678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d8a570 0x555892d87720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:14.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.303+0000 7fdec259c640 1 --2- 172.21.15.28:0/3838215678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d7f580 0x555892d89ed0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fdeb409bd50 tx=0x7fdeb40984b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da150d73cffbafb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:14.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.303+0000 7fdea9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:14.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.303+0000 7fdea9ffb640 1 -- 172.21.15.28:0/3838215678 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb409b410 con 0x555892d7f580 2024-04-25T01:39:14.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.303+0000 7fdea9ffb640 1 -- 172.21.15.28:0/3838215678 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fdeb40ba650 con 0x555892d7f580 2024-04-25T01:39:14.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.303+0000 7fdecb60c740 1 -- 172.21.15.28:0/3838215678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdeb409bdc0 con 0x555892d7f580 2024-04-25T01:39:14.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.303+0000 7fdea9ffb640 1 -- 172.21.15.28:0/3838215678 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdeb40a0170 con 0x555892d7f580 2024-04-25T01:39:14.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.303+0000 7fdecb60c740 10 client.17799 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:14.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.303+0000 7fdecb60c740 1 -- 172.21.15.28:0/3838215678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5558929f83f0 con 0x555892d7f580 2024-04-25T01:39:14.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.303+0000 7fdea9ffb640 1 -- 172.21.15.28:0/3838215678 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdeb409bdc0 con 0x555892d7f580 2024-04-25T01:39:14.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.303+0000 7fdea9ffb640 1 -- 172.21.15.28:0/3838215678 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fdeb4097b80 con 0x555892d7f580 2024-04-25T01:39:14.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.303+0000 7fdecb60c740 10 client.17799 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:14.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.303+0000 7fdecb60c740 10 client.17799 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:14.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.303+0000 7fdecb60c740 20 client.17799 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:14.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.303+0000 7fdecb60c740 1 --2- 172.21.15.28:0/3838215678 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555892d9a390 0x555892d9c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.303+0000 7fdecb60c740 4 client.17799 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:14.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.303+0000 7fdecb60c740 1 -- 172.21.15.28:0/3838215678 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555892cf1bb0 con 0x555892d9a390 2024-04-25T01:39:14.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.304+0000 7fdebbfff640 1 --2- 172.21.15.28:0/3838215678 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555892d9a390 0x555892d9c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.304+0000 7fdebbfff640 1 --2- 172.21.15.28:0/3838215678 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555892d9a390 0x555892d9c780 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53445a3d8bf0ea49 server_cookie=83dce886a8da5d2e in_seq=0 out_seq=0 2024-04-25T01:39:14.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.304+0000 7fdea9ffb640 10 client.17799 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:14.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.305+0000 7fdea9ffb640 1 -- 172.21.15.28:0/3838215678 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555892cf1bb0 con 0x555892d9a390 2024-04-25T01:39:14.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.305+0000 7fdea9ffb640 10 client.17799 handle_command_reply: tid=0 2024-04-25T01:39:14.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.305+0000 7fdecb60c740 1 client.17799 shutdown 2024-04-25T01:39:14.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.305+0000 7fdecb60c740 1 -- 172.21.15.28:0/3838215678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d7f580 msgr2=0x555892d89ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.305+0000 7fdecb60c740 1 --2- 172.21.15.28:0/3838215678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d7f580 0x555892d89ed0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fdeb409bd50 tx=0x7fdeb40984b0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.305+0000 7fdecb60c740 1 -- 172.21.15.28:0/3838215678 shutdown_connections 2024-04-25T01:39:14.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.305+0000 7fdecb60c740 1 --2- 172.21.15.28:0/3838215678 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555892d9a390 0x555892d9c780 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.305+0000 7fdecb60c740 1 --2- 172.21.15.28:0/3838215678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d8a570 0x555892d87720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.305+0000 7fdecb60c740 1 --2- 172.21.15.28:0/3838215678 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7fea0 0x555892d871c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.305+0000 7fdea9ffb640 0 client.17799 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:14.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.305+0000 7fdecb60c740 1 --2- 172.21.15.28:0/3838215678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d7f580 0x555892d89ed0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.305+0000 7fdecb60c740 1 -- 172.21.15.28:0/3838215678 >> 172.21.15.28:0/3838215678 conn(0x555892d7b8a0 msgr2=0x555892d7ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:14.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.305+0000 7fdecb60c740 1 -- 172.21.15.28:0/3838215678 shutdown_connections 2024-04-25T01:39:14.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.305+0000 7fdecb60c740 1 -- 172.21.15.28:0/3838215678 wait complete. 2024-04-25T01:39:14.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.305+0000 7fdecb60c740 20 client.17799 trim_cache size 0 max 16384 2024-04-25T01:39:14.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.318+0000 7fdecb60c740 1 Processor -- start 2024-04-25T01:39:14.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.318+0000 7fdecb60c740 1 -- start start 2024-04-25T01:39:14.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.319+0000 7fdecb60c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d7f580 0x555892d7b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.319+0000 7fdecb60c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7fea0 0x555892d76a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.319+0000 7fdecb60c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d8a570 0x555892d89a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.319+0000 7fdecb60c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdec41b0d00 con 0x555892d8a570 2024-04-25T01:39:14.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.319+0000 7fdecb60c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdec41b2840 con 0x555892d7f580 2024-04-25T01:39:14.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.319+0000 7fdecb60c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fdec41b13d0 con 0x555892d7fea0 2024-04-25T01:39:14.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.319+0000 7fdec2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d8a570 0x555892d89a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.319+0000 7fdec259c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d7f580 0x555892d7b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.319+0000 7fdec2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d8a570 0x555892d89a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48960/0 (socket says 172.21.15.28:48960) 2024-04-25T01:39:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.319+0000 7fdec2d9d640 1 -- 172.21.15.28:0/491917860 learned_addr learned my addr 172.21.15.28:0/491917860 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.319+0000 7fdebbfff640 1 --2- 172.21.15.28:0/491917860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7fea0 0x555892d76a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.319+0000 7fdec2d9d640 1 -- 172.21.15.28:0/491917860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7fea0 msgr2=0x555892d76a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.319+0000 7fdec2d9d640 1 --2- 172.21.15.28:0/491917860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7fea0 0x555892d76a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.319+0000 7fdec2d9d640 1 -- 172.21.15.28:0/491917860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d7f580 msgr2=0x555892d7b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.319+0000 7fdec2d9d640 1 --2- 172.21.15.28:0/491917860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d7f580 0x555892d7b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.319+0000 7fdec2d9d640 1 -- 172.21.15.28:0/491917860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555892d87a90 con 0x555892d8a570 2024-04-25T01:39:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.319+0000 7fdec2d9d640 1 --2- 172.21.15.28:0/491917860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d8a570 0x555892d89a00 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fdebc035000 tx=0x7fdebc005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d43edde785bd059e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.320+0000 7fdeabfff640 1 -- 172.21.15.28:0/491917860 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdebc04e070 con 0x555892d8a570 2024-04-25T01:39:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.320+0000 7fdeabfff640 1 -- 172.21.15.28:0/491917860 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdebc003c30 con 0x555892d8a570 2024-04-25T01:39:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.320+0000 7fdeabfff640 1 -- 172.21.15.28:0/491917860 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdebc034040 con 0x555892d8a570 2024-04-25T01:39:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.320+0000 7fdecb60c740 1 -- 172.21.15.28:0/491917860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d8a570 msgr2=0x555892d89a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.320+0000 7fdecb60c740 1 --2- 172.21.15.28:0/491917860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d8a570 0x555892d89a00 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fdebc035000 tx=0x7fdebc005df0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.320+0000 7fdecb60c740 1 -- 172.21.15.28:0/491917860 shutdown_connections 2024-04-25T01:39:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.320+0000 7fdecb60c740 1 --2- 172.21.15.28:0/491917860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d8a570 0x555892d89a00 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.320+0000 7fdecb60c740 1 --2- 172.21.15.28:0/491917860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7fea0 0x555892d76a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.320+0000 7fdecb60c740 1 --2- 172.21.15.28:0/491917860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d7f580 0x555892d7b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.320+0000 7fdecb60c740 1 -- 172.21.15.28:0/491917860 >> 172.21.15.28:0/491917860 conn(0x555892d7b8a0 msgr2=0x555892d827d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.320+0000 7fdecb60c740 1 -- 172.21.15.28:0/491917860 shutdown_connections 2024-04-25T01:39:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.320+0000 7fdecb60c740 1 -- 172.21.15.28:0/491917860 wait complete. 2024-04-25T01:39:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdecb60c740 1 -- start start 2024-04-25T01:39:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdecb60c740 1 Processor -- start 2024-04-25T01:39:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdecb60c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7f580 0x555892d8df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdecb60c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d7fea0 0x555892d8e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdecb60c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d8a570 0x555892d8ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdecb60c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdec41b2170 con 0x555892d8a570 2024-04-25T01:39:14.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdecb60c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdec41b1aa0 con 0x555892d7fea0 2024-04-25T01:39:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdecb60c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fdec41aece0 con 0x555892d7f580 2024-04-25T01:39:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdecb60c740 10 client.0 _ll_register_callbacks cb 0x7fdeac00b420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdec2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d8a570 0x555892d8ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdec2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d8a570 0x555892d8ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48974/0 (socket says 172.21.15.28:48974) 2024-04-25T01:39:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdec2d9d640 1 -- 172.21.15.28:0/3885987610 learned_addr learned my addr 172.21.15.28:0/3885987610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdec259c640 1 --2- 172.21.15.28:0/3885987610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7f580 0x555892d8df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdebbfff640 1 --2- 172.21.15.28:0/3885987610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d7fea0 0x555892d8e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdec2d9d640 1 -- 172.21.15.28:0/3885987610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7f580 msgr2=0x555892d8df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdec2d9d640 1 --2- 172.21.15.28:0/3885987610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7f580 0x555892d8df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdec2d9d640 1 -- 172.21.15.28:0/3885987610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d7fea0 msgr2=0x555892d8e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdec2d9d640 1 --2- 172.21.15.28:0/3885987610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d7fea0 0x555892d8e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdec2d9d640 1 -- 172.21.15.28:0/3885987610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555892ac27d0 con 0x555892d8a570 2024-04-25T01:39:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdebbfff640 1 --2- 172.21.15.28:0/3885987610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d7fea0 0x555892d8e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdec2d9d640 1 --2- 172.21.15.28:0/3885987610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d8a570 0x555892d8ea00 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fdebc009a60 tx=0x7fdebc030c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d20748b472d946 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.321+0000 7fdea9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.322+0000 7fdea9ffb640 1 -- 172.21.15.28:0/3885987610 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdebc04e050 con 0x555892d8a570 2024-04-25T01:39:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.322+0000 7fdecb60c740 1 -- 172.21.15.28:0/3885987610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdebc003c30 con 0x555892d8a570 2024-04-25T01:39:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.322+0000 7fdea9ffb640 1 -- 172.21.15.28:0/3885987610 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fdebc053660 con 0x555892d8a570 2024-04-25T01:39:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.322+0000 7fdea9ffb640 1 -- 172.21.15.28:0/3885987610 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdebc064050 con 0x555892d8a570 2024-04-25T01:39:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.322+0000 7fdecb60c740 10 client.17805 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.322+0000 7fdecb60c740 1 -- 172.21.15.28:0/3885987610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555892e005d0 con 0x555892d8a570 2024-04-25T01:39:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.322+0000 7fdea9ffb640 1 -- 172.21.15.28:0/3885987610 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdebc003c30 con 0x555892d8a570 2024-04-25T01:39:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.322+0000 7fdea9ffb640 1 -- 172.21.15.28:0/3885987610 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fdebc049040 con 0x555892d8a570 2024-04-25T01:39:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.322+0000 7fdecb60c740 10 client.17805 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.322+0000 7fdecb60c740 10 client.17805 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.322+0000 7fdecb60c740 20 client.17805 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.322+0000 7fdecb60c740 1 --2- 172.21.15.28:0/3885987610 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555892e00bf0 0x555892e02fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.322+0000 7fdecb60c740 4 client.17805 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.322+0000 7fdecb60c740 1 -- 172.21.15.28:0/3885987610 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555892cebf00 con 0x555892e00bf0 2024-04-25T01:39:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.322+0000 7fdec259c640 1 --2- 172.21.15.28:0/3885987610 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555892e00bf0 0x555892e02fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.323+0000 7fdec259c640 1 --2- 172.21.15.28:0/3885987610 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555892e00bf0 0x555892e02fe0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61886697fdef1c3a server_cookie=10c6bb468b4788bd in_seq=0 out_seq=0 2024-04-25T01:39:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.323+0000 7fdea9ffb640 10 client.17805 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.323+0000 7fdea9ffb640 1 -- 172.21.15.28:0/3885987610 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555892cebf00 con 0x555892e00bf0 2024-04-25T01:39:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.323+0000 7fdea9ffb640 10 client.17805 handle_command_reply: tid=0 2024-04-25T01:39:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.323+0000 7fdecb60c740 1 client.17805 shutdown 2024-04-25T01:39:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.323+0000 7fdecb60c740 1 -- 172.21.15.28:0/3885987610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d8a570 msgr2=0x555892d8ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.323+0000 7fdecb60c740 1 --2- 172.21.15.28:0/3885987610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d8a570 0x555892d8ea00 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fdebc009a60 tx=0x7fdebc030c30 comp rx=0 tx=0).stop 2024-04-25T01:39:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.323+0000 7fdecb60c740 1 -- 172.21.15.28:0/3885987610 shutdown_connections 2024-04-25T01:39:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.323+0000 7fdecb60c740 1 --2- 172.21.15.28:0/3885987610 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555892e00bf0 0x555892e02fe0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.323+0000 7fdecb60c740 1 --2- 172.21.15.28:0/3885987610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555892d8a570 0x555892d8ea00 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.323+0000 7fdecb60c740 1 --2- 172.21.15.28:0/3885987610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555892d7fea0 0x555892d8e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.323+0000 7fdea9ffb640 0 client.17805 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.323+0000 7fdecb60c740 1 --2- 172.21.15.28:0/3885987610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555892d7f580 0x555892d8df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.323+0000 7fdecb60c740 1 -- 172.21.15.28:0/3885987610 >> 172.21.15.28:0/3885987610 conn(0x555892d7b8a0 msgr2=0x555892d81560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.323+0000 7fdecb60c740 1 -- 172.21.15.28:0/3885987610 shutdown_connections 2024-04-25T01:39:14.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.323+0000 7fdecb60c740 1 -- 172.21.15.28:0/3885987610 wait complete. 2024-04-25T01:39:14.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.323+0000 7fdecb60c740 20 client.17805 trim_cache size 0 max 16384 2024-04-25T01:39:14.312 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.324+0000 7fdec8909640 1 -- 172.21.15.28:0/1878992681 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdeb00a2d00 msgr2=0x7fdeb00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.324+0000 7fdec8909640 1 --2- 172.21.15.28:0/1878992681 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdeb00a2d00 0x7fdeb00a51c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdebc005fa0 tx=0x7fdebc02a950 comp rx=0 tx=0).stop 2024-04-25T01:39:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.324+0000 7fdec8909640 1 -- 172.21.15.28:0/1878992681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdec4073b70 msgr2=0x7fdec419a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.324+0000 7fdec8909640 1 --2- 172.21.15.28:0/1878992681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdec4073b70 0x7fdec419a270 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400e9f0 tx=0x7fdeb400eeb0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.324+0000 7fdec8909640 1 -- 172.21.15.28:0/1878992681 shutdown_connections 2024-04-25T01:39:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.324+0000 7fdec8909640 1 --2- 172.21.15.28:0/1878992681 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdeb00a2d00 0x7fdeb00a51c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.324+0000 7fdec8909640 1 --2- 172.21.15.28:0/1878992681 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdec410df50 0x7fdec419eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.324+0000 7fdec8909640 1 --2- 172.21.15.28:0/1878992681 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdec4074490 0x7fdec419a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.324+0000 7fdec8909640 1 --2- 172.21.15.28:0/1878992681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdec4073b70 0x7fdec419a270 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.324+0000 7fdec8909640 1 -- 172.21.15.28:0/1878992681 >> 172.21.15.28:0/1878992681 conn(0x7fdec40fd180 msgr2=0x7fdec40fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.324+0000 7fdec8909640 1 -- 172.21.15.28:0/1878992681 shutdown_connections 2024-04-25T01:39:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:14.324+0000 7fdec8909640 1 -- 172.21.15.28:0/1878992681 wait complete. 2024-04-25T01:39:14.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:13 smithi130 ceph-mon[32702]: pgmap v189: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-04-25T01:39:14.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:13 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:39:14.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:13 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:14.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:13 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:39:14.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:13 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:14.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:13 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:39:14.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:13 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/433263823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:39:15.317 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:39:15.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:39:15.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:39:15.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.454+0000 7f8eaa748640 1 -- 172.21.15.28:0/2085722860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ea4100fd0 msgr2=0x7f8ea410d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:15.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.454+0000 7f8eaa748640 1 --2- 172.21.15.28:0/2085722860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ea4100fd0 0x7f8ea410d4e0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c009cd0 tx=0x7f8e9c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.454+0000 7f8eaa748640 1 -- 172.21.15.28:0/2085722860 shutdown_connections 2024-04-25T01:39:15.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.454+0000 7f8eaa748640 1 --2- 172.21.15.28:0/2085722860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ea4100fd0 0x7f8ea410d4e0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.454+0000 7f8eaa748640 1 --2- 172.21.15.28:0/2085722860 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ea4100610 0x7f8ea4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.454+0000 7f8eaa748640 1 --2- 172.21.15.28:0/2085722860 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ea41066b0 0x7f8ea4106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.454+0000 7f8eaa748640 1 -- 172.21.15.28:0/2085722860 >> 172.21.15.28:0/2085722860 conn(0x7f8ea4077740 msgr2=0x7f8ea4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:15.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.454+0000 7f8eaa748640 1 -- 172.21.15.28:0/2085722860 shutdown_connections 2024-04-25T01:39:15.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.454+0000 7f8eaa748640 1 -- 172.21.15.28:0/2085722860 wait complete. 2024-04-25T01:39:15.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.454+0000 7f8eaa748640 1 Processor -- start 2024-04-25T01:39:15.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.454+0000 7f8eaa748640 1 -- start start 2024-04-25T01:39:15.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.455+0000 7f8eaa748640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ea4100610 0x7f8ea4198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.455+0000 7f8eaa748640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ea4100fd0 0x7f8ea4199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.455+0000 7f8eaa748640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ea41066b0 0x7f8ea41a3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.455+0000 7f8eaa748640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea41109f0 con 0x7f8ea4100fd0 2024-04-25T01:39:15.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.455+0000 7f8eaa748640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea4110870 con 0x7f8ea4100610 2024-04-25T01:39:15.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.455+0000 7f8eaa748640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea4110b70 con 0x7f8ea41066b0 2024-04-25T01:39:15.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.455+0000 7f8ea37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ea4100fd0 0x7f8ea4199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.455+0000 7f8ea37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ea4100fd0 0x7f8ea4199210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48994/0 (socket says 172.21.15.28:48994) 2024-04-25T01:39:15.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.455+0000 7f8ea37fe640 1 -- 172.21.15.28:0/704777596 learned_addr learned my addr 172.21.15.28:0/704777596 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:15.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.455+0000 7f8ea8ce5640 1 --2- 172.21.15.28:0/704777596 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ea41066b0 0x7f8ea41a3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.455+0000 7f8ea3fff640 1 --2- 172.21.15.28:0/704777596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ea4100610 0x7f8ea4198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.455+0000 7f8ea37fe640 1 -- 172.21.15.28:0/704777596 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ea41066b0 msgr2=0x7f8ea41a3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:15.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.455+0000 7f8ea37fe640 1 --2- 172.21.15.28:0/704777596 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ea41066b0 0x7f8ea41a3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.455+0000 7f8ea37fe640 1 -- 172.21.15.28:0/704777596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ea4100610 msgr2=0x7f8ea4198cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:15.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.455+0000 7f8ea37fe640 1 --2- 172.21.15.28:0/704777596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ea4100610 0x7f8ea4198cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.455+0000 7f8ea37fe640 1 -- 172.21.15.28:0/704777596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ea41a4290 con 0x7f8ea4100fd0 2024-04-25T01:39:15.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.455+0000 7f8ea8ce5640 1 --2- 172.21.15.28:0/704777596 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ea41066b0 0x7f8ea41a3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:15.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.456+0000 7f8ea37fe640 1 --2- 172.21.15.28:0/704777596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ea4100fd0 0x7f8ea4199210 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f8e9800c920 tx=0x7f8e9800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:15.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.456+0000 7f8ea17fa640 1 -- 172.21.15.28:0/704777596 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e98007c70 con 0x7f8ea4100fd0 2024-04-25T01:39:15.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.456+0000 7f8ea17fa640 1 -- 172.21.15.28:0/704777596 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e98007e10 con 0x7f8ea4100fd0 2024-04-25T01:39:15.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.456+0000 7f8eaa748640 1 -- 172.21.15.28:0/704777596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ea41a4460 con 0x7f8ea4100fd0 2024-04-25T01:39:15.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.456+0000 7f8ea17fa640 1 -- 172.21.15.28:0/704777596 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e98018890 con 0x7f8ea4100fd0 2024-04-25T01:39:15.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.456+0000 7f8eaa748640 1 -- 172.21.15.28:0/704777596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ea41a4a00 con 0x7f8ea4100fd0 2024-04-25T01:39:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.458+0000 7f8ea17fa640 1 -- 172.21.15.28:0/704777596 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8e98018a70 con 0x7f8ea4100fd0 2024-04-25T01:39:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ead44c740 1 Processor -- start 2024-04-25T01:39:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ead44c740 1 -- start start 2024-04-25T01:39:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ea17fa640 1 --2- 172.21.15.28:0/704777596 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8e8c0a2d80 0x7f8e8c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ea17fa640 1 -- 172.21.15.28:0/704777596 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8e980cf960 con 0x7f8ea4100fd0 2024-04-25T01:39:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ead44c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf29fa1980 0x55cf29f9fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ea8ce5640 1 --2- 172.21.15.28:0/704777596 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8e8c0a2d80 0x7f8e8c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ead44c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa0290 0x55cf29fa0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ead44c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29faf420 0x55cf29fb1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ead44c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cf29f042c0 con 0x55cf29faf420 2024-04-25T01:39:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ead44c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cf29eac910 con 0x55cf29fa0290 2024-04-25T01:39:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ead44c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55cf29f00280 con 0x55cf29fa1980 2024-04-25T01:39:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ea8ce5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29faf420 0x55cf29fb1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ea8ce5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29faf420 0x55cf29fb1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49006/0 (socket says 172.21.15.28:49006) 2024-04-25T01:39:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ea8ce5640 1 -- 172.21.15.28:0/2878090331 learned_addr learned my addr 172.21.15.28:0/2878090331 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ea3fff640 1 --2- 172.21.15.28:0/2878090331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf29fa1980 0x55cf29f9fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ea37fe640 1 --2- 172.21.15.28:0/2878090331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa0290 0x55cf29fa0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ea8ce5640 1 --2- 172.21.15.28:0/704777596 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8e8c0a2d80 0x7f8e8c0a5240 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c009cd0 tx=0x7f8e9c0318c0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ea8ce5640 1 -- 172.21.15.28:0/2878090331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf29fa1980 msgr2=0x55cf29f9fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ea8ce5640 1 --2- 172.21.15.28:0/2878090331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf29fa1980 0x55cf29f9fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ea8ce5640 1 -- 172.21.15.28:0/2878090331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa0290 msgr2=0x55cf29fa0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.459+0000 7f8ea8ce5640 1 --2- 172.21.15.28:0/2878090331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa0290 0x55cf29fa0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.460+0000 7f8ea8ce5640 1 -- 172.21.15.28:0/2878090331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf29ebdfb0 con 0x55cf29faf420 2024-04-25T01:39:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.460+0000 7f8ea8ce5640 1 --2- 172.21.15.28:0/2878090331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29faf420 0x55cf29fb1810 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c009a60 tx=0x7f8e9c049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba6cde861959f2cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.460+0000 7f8e8a7fc640 1 -- 172.21.15.28:0/2878090331 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c004690 con 0x55cf29faf420 2024-04-25T01:39:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.460+0000 7f8e8a7fc640 1 -- 172.21.15.28:0/2878090331 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e9c050070 con 0x55cf29faf420 2024-04-25T01:39:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.460+0000 7f8e8a7fc640 1 -- 172.21.15.28:0/2878090331 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c030ed0 con 0x55cf29faf420 2024-04-25T01:39:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.461+0000 7f8ead44c740 1 -- 172.21.15.28:0/2878090331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29faf420 msgr2=0x55cf29fb1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.461+0000 7f8ead44c740 1 --2- 172.21.15.28:0/2878090331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29faf420 0x55cf29fb1810 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c009a60 tx=0x7f8e9c049000 comp rx=0 tx=0).stop 2024-04-25T01:39:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.461+0000 7f8ead44c740 1 -- 172.21.15.28:0/2878090331 shutdown_connections 2024-04-25T01:39:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.461+0000 7f8ead44c740 1 --2- 172.21.15.28:0/2878090331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29faf420 0x55cf29fb1810 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.461+0000 7f8ead44c740 1 --2- 172.21.15.28:0/2878090331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa0290 0x55cf29fa0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.461+0000 7f8ead44c740 1 --2- 172.21.15.28:0/2878090331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf29fa1980 0x55cf29f9fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.461+0000 7f8ead44c740 1 -- 172.21.15.28:0/2878090331 >> 172.21.15.28:0/2878090331 conn(0x55cf29f68950 msgr2=0x55cf29f9c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.461+0000 7f8ead44c740 1 -- 172.21.15.28:0/2878090331 shutdown_connections 2024-04-25T01:39:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.461+0000 7f8ead44c740 1 -- 172.21.15.28:0/2878090331 wait complete. 2024-04-25T01:39:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.461+0000 7f8ead44c740 1 -- start start 2024-04-25T01:39:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.461+0000 7f8ead44c740 1 Processor -- start 2024-04-25T01:39:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.461+0000 7f8ead44c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29fa0290 0x55cf29fac320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8ead44c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa1980 0x55cf29fac860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8ead44c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf29faf420 0x55cf29facda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8ead44c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cf29fb2250 con 0x55cf29fa0290 2024-04-25T01:39:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8ead44c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cf29ebfe90 con 0x55cf29fa1980 2024-04-25T01:39:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8ead44c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55cf29fb23d0 con 0x55cf29faf420 2024-04-25T01:39:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8ead44c740 10 client.0 _ll_register_callbacks cb 0x55cf29efb970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8ea3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29fa0290 0x55cf29fac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8ea3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29fa0290 0x55cf29fac320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49016/0 (socket says 172.21.15.28:49016) 2024-04-25T01:39:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8ea3fff640 1 -- 172.21.15.28:0/1277428204 learned_addr learned my addr 172.21.15.28:0/1277428204 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8ea37fe640 1 --2- 172.21.15.28:0/1277428204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa1980 0x55cf29fac860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8ea8ce5640 1 --2- 172.21.15.28:0/1277428204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf29faf420 0x55cf29facda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8ea3fff640 1 -- 172.21.15.28:0/1277428204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf29faf420 msgr2=0x55cf29facda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8ea3fff640 1 --2- 172.21.15.28:0/1277428204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf29faf420 0x55cf29facda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8ea3fff640 1 -- 172.21.15.28:0/1277428204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa1980 msgr2=0x55cf29fac860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8ea3fff640 1 --2- 172.21.15.28:0/1277428204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa1980 0x55cf29fac860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8ea3fff640 1 -- 172.21.15.28:0/1277428204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf29ce07e0 con 0x55cf29fa0290 2024-04-25T01:39:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8ea3fff640 1 --2- 172.21.15.28:0/1277428204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29fa0290 0x55cf29fac320 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400beb0 tx=0x7f8e94012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eaac4fa202a358e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8e837fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8e837fe640 1 -- 172.21.15.28:0/1277428204 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9401f060 con 0x55cf29fa0290 2024-04-25T01:39:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8e837fe640 1 -- 172.21.15.28:0/1277428204 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8e940108a0 con 0x55cf29fa0290 2024-04-25T01:39:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.462+0000 7f8ead44c740 1 -- 172.21.15.28:0/1277428204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e9c050070 con 0x55cf29fa0290 2024-04-25T01:39:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.463+0000 7f8e837fe640 1 -- 172.21.15.28:0/1277428204 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8e94026050 con 0x55cf29fa0290 2024-04-25T01:39:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.463+0000 7f8ead44c740 10 client.17817 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.463+0000 7f8ead44c740 1 -- 172.21.15.28:0/1277428204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cf29d40f70 con 0x55cf29fa0290 2024-04-25T01:39:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.463+0000 7f8e837fe640 1 -- 172.21.15.28:0/1277428204 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e9c050070 con 0x55cf29fa0290 2024-04-25T01:39:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.463+0000 7f8e837fe640 1 -- 172.21.15.28:0/1277428204 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8e94019030 con 0x55cf29fa0290 2024-04-25T01:39:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.463+0000 7f8ead44c740 10 client.17817 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.463+0000 7f8ead44c740 10 client.17817 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.463+0000 7f8ead44c740 20 client.17817 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.463+0000 7f8ead44c740 1 --2- 172.21.15.28:0/1277428204 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cf29fbf4f0 0x55cf29fc1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.463+0000 7f8ead44c740 4 client.17817 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.463+0000 7f8ead44c740 1 -- 172.21.15.28:0/1277428204 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf29ed85a0 con 0x55cf29fbf4f0 2024-04-25T01:39:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.463+0000 7f8ea37fe640 1 --2- 172.21.15.28:0/1277428204 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cf29fbf4f0 0x55cf29fc1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.464+0000 7f8ea37fe640 1 --2- 172.21.15.28:0/1277428204 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cf29fbf4f0 0x55cf29fc1960 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45c61cea8c11a974 server_cookie=4ab6926905b36096 in_seq=0 out_seq=0 2024-04-25T01:39:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.464+0000 7f8e837fe640 10 client.17817 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.464+0000 7f8e837fe640 1 -- 172.21.15.28:0/1277428204 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cf29ed85a0 con 0x55cf29fbf4f0 2024-04-25T01:39:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.464+0000 7f8e837fe640 10 client.17817 handle_command_reply: tid=0 2024-04-25T01:39:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.465+0000 7f8ead44c740 1 client.17817 shutdown 2024-04-25T01:39:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.465+0000 7f8ead44c740 1 -- 172.21.15.28:0/1277428204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29fa0290 msgr2=0x55cf29fac320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.465+0000 7f8ead44c740 1 --2- 172.21.15.28:0/1277428204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29fa0290 0x55cf29fac320 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400beb0 tx=0x7f8e94012820 comp rx=0 tx=0).stop 2024-04-25T01:39:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.465+0000 7f8ead44c740 1 -- 172.21.15.28:0/1277428204 shutdown_connections 2024-04-25T01:39:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.465+0000 7f8ead44c740 1 --2- 172.21.15.28:0/1277428204 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cf29fbf4f0 0x55cf29fc1960 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.465+0000 7f8ead44c740 1 --2- 172.21.15.28:0/1277428204 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf29faf420 0x55cf29facda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.465+0000 7f8ead44c740 1 --2- 172.21.15.28:0/1277428204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa1980 0x55cf29fac860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.465+0000 7f8ead44c740 1 --2- 172.21.15.28:0/1277428204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29fa0290 0x55cf29fac320 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.465+0000 7f8e837fe640 0 client.17817 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.465+0000 7f8ead44c740 1 -- 172.21.15.28:0/1277428204 >> 172.21.15.28:0/1277428204 conn(0x55cf29f68950 msgr2=0x55cf29fa2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.465+0000 7f8ead44c740 1 -- 172.21.15.28:0/1277428204 shutdown_connections 2024-04-25T01:39:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.465+0000 7f8ead44c740 1 -- 172.21.15.28:0/1277428204 wait complete. 2024-04-25T01:39:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.465+0000 7f8ead44c740 20 client.17817 trim_cache size 0 max 16384 2024-04-25T01:39:15.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.483+0000 7f8ead44c740 1 Processor -- start 2024-04-25T01:39:15.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.483+0000 7f8ead44c740 1 -- start start 2024-04-25T01:39:15.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.483+0000 7f8ead44c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29fa0290 0x55cf29fbb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.483+0000 7f8ead44c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa1980 0x55cf29f9b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.483+0000 7f8ead44c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf29faf420 0x55cf29fc7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.483+0000 7f8ead44c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea41ac7f0 con 0x55cf29fa0290 2024-04-25T01:39:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.483+0000 7f8ead44c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea41ae330 con 0x55cf29fa1980 2024-04-25T01:39:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.483+0000 7f8ead44c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea41acec0 con 0x55cf29faf420 2024-04-25T01:39:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.483+0000 7f8ea3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29fa0290 0x55cf29fbb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.483+0000 7f8ea3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29fa0290 0x55cf29fbb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49028/0 (socket says 172.21.15.28:49028) 2024-04-25T01:39:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.483+0000 7f8ea3fff640 1 -- 172.21.15.28:0/3001380414 learned_addr learned my addr 172.21.15.28:0/3001380414 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.483+0000 7f8ea3fff640 1 -- 172.21.15.28:0/3001380414 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf29faf420 msgr2=0x55cf29fc7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.483+0000 7f8ea3fff640 1 --2- 172.21.15.28:0/3001380414 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf29faf420 0x55cf29fc7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.483+0000 7f8ea3fff640 1 -- 172.21.15.28:0/3001380414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa1980 msgr2=0x55cf29f9b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.483+0000 7f8ea3fff640 1 --2- 172.21.15.28:0/3001380414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa1980 0x55cf29f9b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.483+0000 7f8ea3fff640 1 -- 172.21.15.28:0/3001380414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf29bd8170 con 0x55cf29fa0290 2024-04-25T01:39:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.483+0000 7f8ea3fff640 1 --2- 172.21.15.28:0/3001380414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29fa0290 0x55cf29fbb150 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f8e94000c00 tx=0x7f8e9400ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a12fce050e40cef9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.484+0000 7f8e89ffb640 1 -- 172.21.15.28:0/3001380414 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e94005300 con 0x55cf29fa0290 2024-04-25T01:39:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.484+0000 7f8e89ffb640 1 -- 172.21.15.28:0/3001380414 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e940054a0 con 0x55cf29fa0290 2024-04-25T01:39:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.484+0000 7f8e89ffb640 1 -- 172.21.15.28:0/3001380414 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e94004a20 con 0x55cf29fa0290 2024-04-25T01:39:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.484+0000 7f8ead44c740 1 -- 172.21.15.28:0/3001380414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29fa0290 msgr2=0x55cf29fbb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.484+0000 7f8ead44c740 1 --2- 172.21.15.28:0/3001380414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29fa0290 0x55cf29fbb150 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f8e94000c00 tx=0x7f8e9400ba70 comp rx=0 tx=0).stop 2024-04-25T01:39:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.484+0000 7f8ead44c740 1 -- 172.21.15.28:0/3001380414 shutdown_connections 2024-04-25T01:39:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.484+0000 7f8ead44c740 1 --2- 172.21.15.28:0/3001380414 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf29faf420 0x55cf29fc7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.484+0000 7f8ead44c740 1 --2- 172.21.15.28:0/3001380414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa1980 0x55cf29f9b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.484+0000 7f8ead44c740 1 --2- 172.21.15.28:0/3001380414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29fa0290 0x55cf29fbb150 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.484+0000 7f8ead44c740 1 -- 172.21.15.28:0/3001380414 >> 172.21.15.28:0/3001380414 conn(0x55cf29f68950 msgr2=0x55cf29f9b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.484+0000 7f8ead44c740 1 -- 172.21.15.28:0/3001380414 shutdown_connections 2024-04-25T01:39:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.484+0000 7f8ead44c740 1 -- 172.21.15.28:0/3001380414 wait complete. 2024-04-25T01:39:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.484+0000 7f8ead44c740 1 -- start start 2024-04-25T01:39:15.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.485+0000 7f8ead44c740 1 Processor -- start 2024-04-25T01:39:15.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.485+0000 7f8ead44c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf29fa0290 0x55cf29fb3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.485+0000 7f8ead44c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa1980 0x55cf29fb3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.485+0000 7f8ead44c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29faf420 0x55cf2a024aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.485+0000 7f8ead44c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea41adc60 con 0x55cf29faf420 2024-04-25T01:39:15.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.485+0000 7f8ead44c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea41ad590 con 0x55cf29fa1980 2024-04-25T01:39:15.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.485+0000 7f8ead44c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea41aa7d0 con 0x55cf29fa0290 2024-04-25T01:39:15.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.485+0000 7f8ead44c740 10 client.0 _ll_register_callbacks cb 0x7f8e9400b6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:15.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.485+0000 7f8ea8ce5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29faf420 0x55cf2a024aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.485+0000 7f8ea8ce5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29faf420 0x55cf2a024aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49032/0 (socket says 172.21.15.28:49032) 2024-04-25T01:39:15.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.485+0000 7f8ea8ce5640 1 -- 172.21.15.28:0/2129167837 learned_addr learned my addr 172.21.15.28:0/2129167837 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:15.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.485+0000 7f8ea37fe640 1 --2- 172.21.15.28:0/2129167837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa1980 0x55cf29fb3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.485+0000 7f8ea3fff640 1 --2- 172.21.15.28:0/2129167837 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf29fa0290 0x55cf29fb3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.485+0000 7f8ea8ce5640 1 -- 172.21.15.28:0/2129167837 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf29fa0290 msgr2=0x55cf29fb3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:15.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.485+0000 7f8ea8ce5640 1 --2- 172.21.15.28:0/2129167837 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf29fa0290 0x55cf29fb3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.485+0000 7f8ea8ce5640 1 -- 172.21.15.28:0/2129167837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa1980 msgr2=0x55cf29fb3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.485+0000 7f8ea8ce5640 1 --2- 172.21.15.28:0/2129167837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa1980 0x55cf29fb3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.485+0000 7f8ea8ce5640 1 -- 172.21.15.28:0/2129167837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf2a025060 con 0x55cf29faf420 2024-04-25T01:39:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.486+0000 7f8ea37fe640 1 --2- 172.21.15.28:0/2129167837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa1980 0x55cf29fb3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.486+0000 7f8ea8ce5640 1 --2- 172.21.15.28:0/2129167837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29faf420 0x55cf2a024aa0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c0319a0 tx=0x7f8e9c005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b3324e628b1f8a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.486+0000 7f8e837fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.486+0000 7f8e837fe640 1 -- 172.21.15.28:0/2129167837 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c0307a0 con 0x55cf29faf420 2024-04-25T01:39:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.486+0000 7f8e837fe640 1 -- 172.21.15.28:0/2129167837 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8e9c030940 con 0x55cf29faf420 2024-04-25T01:39:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.486+0000 7f8ead44c740 1 -- 172.21.15.28:0/2129167837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e940054a0 con 0x55cf29faf420 2024-04-25T01:39:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.486+0000 7f8e837fe640 1 -- 172.21.15.28:0/2129167837 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8e9c030b40 con 0x55cf29faf420 2024-04-25T01:39:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.486+0000 7f8ead44c740 10 client.17823 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.486+0000 7f8ead44c740 1 -- 172.21.15.28:0/2129167837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cf2a026380 con 0x55cf29faf420 2024-04-25T01:39:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.486+0000 7f8e837fe640 1 -- 172.21.15.28:0/2129167837 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e940054a0 con 0x55cf29faf420 2024-04-25T01:39:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.486+0000 7f8e837fe640 1 -- 172.21.15.28:0/2129167837 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8e9c030070 con 0x55cf29faf420 2024-04-25T01:39:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.486+0000 7f8ead44c740 10 client.17823 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.486+0000 7f8ead44c740 10 client.17823 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.486+0000 7f8ead44c740 20 client.17823 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.486+0000 7f8ead44c740 1 --2- 172.21.15.28:0/2129167837 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cf2a0266b0 0x55cf2a028aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.487+0000 7f8ead44c740 4 client.17823 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:39:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.487+0000 7f8ead44c740 1 -- 172.21.15.28:0/2129167837 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cf29f11800 con 0x55cf2a0266b0 2024-04-25T01:39:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.487+0000 7f8ea3fff640 1 --2- 172.21.15.28:0/2129167837 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cf2a0266b0 0x55cf2a028aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.487+0000 7f8ea3fff640 1 --2- 172.21.15.28:0/2129167837 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cf2a0266b0 0x55cf2a028aa0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0281062ebfb2e18 server_cookie=4213246ae52c50a8 in_seq=0 out_seq=0 2024-04-25T01:39:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.487+0000 7f8e837fe640 10 client.17823 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.488+0000 7f8e837fe640 1 -- 172.21.15.28:0/2129167837 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cf29f11800 con 0x55cf2a0266b0 2024-04-25T01:39:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.488+0000 7f8e837fe640 10 client.17823 handle_command_reply: tid=0 2024-04-25T01:39:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.488+0000 7f8ead44c740 1 client.17823 shutdown 2024-04-25T01:39:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.488+0000 7f8ead44c740 1 -- 172.21.15.28:0/2129167837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29faf420 msgr2=0x55cf2a024aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.488+0000 7f8ead44c740 1 --2- 172.21.15.28:0/2129167837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29faf420 0x55cf2a024aa0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c0319a0 tx=0x7f8e9c005bd0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.488+0000 7f8ead44c740 1 -- 172.21.15.28:0/2129167837 shutdown_connections 2024-04-25T01:39:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.488+0000 7f8ead44c740 1 --2- 172.21.15.28:0/2129167837 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55cf2a0266b0 0x55cf2a028aa0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.488+0000 7f8ead44c740 1 --2- 172.21.15.28:0/2129167837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf29faf420 0x55cf2a024aa0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.488+0000 7f8ead44c740 1 --2- 172.21.15.28:0/2129167837 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf29fa1980 0x55cf29fb3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.488+0000 7f8ead44c740 1 --2- 172.21.15.28:0/2129167837 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf29fa0290 0x55cf29fb3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.488+0000 7f8ead44c740 1 -- 172.21.15.28:0/2129167837 >> 172.21.15.28:0/2129167837 conn(0x55cf29f68950 msgr2=0x55cf29f9b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.489+0000 7f8e837fe640 0 client.17823 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.489+0000 7f8ead44c740 1 -- 172.21.15.28:0/2129167837 shutdown_connections 2024-04-25T01:39:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.489+0000 7f8ead44c740 1 -- 172.21.15.28:0/2129167837 wait complete. 2024-04-25T01:39:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.489+0000 7f8ead44c740 20 client.17823 trim_cache size 0 max 16384 2024-04-25T01:39:15.477 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:39:15.477 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:39:15.477 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "a8cd9c38-e13c-4096-a640-dc20ca57b641", 2024-04-25T01:39:15.478 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:39:15.478 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:39:15.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.489+0000 7f8eaa748640 1 -- 172.21.15.28:0/704777596 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8e8c0a2d80 msgr2=0x7f8e8c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:15.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.489+0000 7f8eaa748640 1 --2- 172.21.15.28:0/704777596 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8e8c0a2d80 0x7f8e8c0a5240 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c009cd0 tx=0x7f8e9c0318c0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.489+0000 7f8eaa748640 1 -- 172.21.15.28:0/704777596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ea4100fd0 msgr2=0x7f8ea4199210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:15.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.489+0000 7f8eaa748640 1 --2- 172.21.15.28:0/704777596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ea4100fd0 0x7f8ea4199210 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f8e9800c920 tx=0x7f8e9800cde0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.489+0000 7f8eaa748640 1 -- 172.21.15.28:0/704777596 shutdown_connections 2024-04-25T01:39:15.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.489+0000 7f8eaa748640 1 --2- 172.21.15.28:0/704777596 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8e8c0a2d80 0x7f8e8c0a5240 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.489+0000 7f8eaa748640 1 --2- 172.21.15.28:0/704777596 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ea41066b0 0x7f8ea41a3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.489+0000 7f8eaa748640 1 --2- 172.21.15.28:0/704777596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ea4100fd0 0x7f8ea4199210 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.489+0000 7f8eaa748640 1 --2- 172.21.15.28:0/704777596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ea4100610 0x7f8ea4198cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.489+0000 7f8eaa748640 1 -- 172.21.15.28:0/704777596 >> 172.21.15.28:0/704777596 conn(0x7f8ea4077740 msgr2=0x7f8ea40fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:15.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.489+0000 7f8eaa748640 1 -- 172.21.15.28:0/704777596 shutdown_connections 2024-04-25T01:39:15.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.489+0000 7f8eaa748640 1 -- 172.21.15.28:0/704777596 wait complete. 2024-04-25T01:39:15.483 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:39:15.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:39:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.605+0000 7fcb8c458640 1 -- 172.21.15.28:0/2823996511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb84105160 msgr2=0x7fcb8410cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.605+0000 7fcb8c458640 1 --2- 172.21.15.28:0/2823996511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb84105160 0x7fcb8410cce0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c009ca0 tx=0x7fcb7c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.605+0000 7fcb8c458640 1 -- 172.21.15.28:0/2823996511 shutdown_connections 2024-04-25T01:39:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.605+0000 7fcb8c458640 1 --2- 172.21.15.28:0/2823996511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb84105160 0x7fcb8410cce0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.605+0000 7fcb8c458640 1 --2- 172.21.15.28:0/2823996511 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb84100c90 0x7fcb84104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.605+0000 7fcb8c458640 1 --2- 172.21.15.28:0/2823996511 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb841002e0 0x7fcb841006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.605+0000 7fcb8c458640 1 -- 172.21.15.28:0/2823996511 >> 172.21.15.28:0/2823996511 conn(0x7fcb840758d0 msgr2=0x7fcb84075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:15.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.605+0000 7fcb8c458640 1 -- 172.21.15.28:0/2823996511 shutdown_connections 2024-04-25T01:39:15.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.605+0000 7fcb8c458640 1 -- 172.21.15.28:0/2823996511 wait complete. 2024-04-25T01:39:15.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.606+0000 7fcb8c458640 1 Processor -- start 2024-04-25T01:39:15.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.606+0000 7fcb8c458640 1 -- start start 2024-04-25T01:39:15.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.606+0000 7fcb8c458640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb841002e0 0x7fcb84198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.606+0000 7fcb8c458640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb84105160 0x7fcb84198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.606+0000 7fcb8c458640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8419c880 0x7fcb8419cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.606+0000 7fcb8c458640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcb841101e0 con 0x7fcb8419c880 2024-04-25T01:39:15.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.606+0000 7fcb8c458640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcb84110060 con 0x7fcb841002e0 2024-04-25T01:39:15.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.606+0000 7fcb8c458640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fcb84110360 con 0x7fcb84105160 2024-04-25T01:39:15.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.606+0000 7fcb8a9f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8419c880 0x7fcb8419cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.606+0000 7fcb8a9f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8419c880 0x7fcb8419cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49040/0 (socket says 172.21.15.28:49040) 2024-04-25T01:39:15.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.606+0000 7fcb8a9f5640 1 -- 172.21.15.28:0/130124743 learned_addr learned my addr 172.21.15.28:0/130124743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:15.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.606+0000 7fcb8a1f4640 1 --2- 172.21.15.28:0/130124743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb841002e0 0x7fcb84198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.606+0000 7fcb899f3640 1 --2- 172.21.15.28:0/130124743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb84105160 0x7fcb84198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.606+0000 7fcb8a9f5640 1 -- 172.21.15.28:0/130124743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb84105160 msgr2=0x7fcb84198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:15.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.607+0000 7fcb8a9f5640 1 --2- 172.21.15.28:0/130124743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb84105160 0x7fcb84198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.607+0000 7fcb8a9f5640 1 -- 172.21.15.28:0/130124743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb841002e0 msgr2=0x7fcb84198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:15.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.607+0000 7fcb8a9f5640 1 --2- 172.21.15.28:0/130124743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb841002e0 0x7fcb84198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.607+0000 7fcb8a9f5640 1 -- 172.21.15.28:0/130124743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb8419d370 con 0x7fcb8419c880 2024-04-25T01:39:15.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.607+0000 7fcb899f3640 1 --2- 172.21.15.28:0/130124743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb84105160 0x7fcb84198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:15.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.607+0000 7fcb8a1f4640 1 --2- 172.21.15.28:0/130124743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb841002e0 0x7fcb84198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:39:15.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.607+0000 7fcb8a9f5640 1 --2- 172.21.15.28:0/130124743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8419c880 0x7fcb8419cd30 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c031a20 tx=0x7fcb7c031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:15.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.607+0000 7fcb7b7fe640 1 -- 172.21.15.28:0/130124743 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c003c30 con 0x7fcb8419c880 2024-04-25T01:39:15.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.607+0000 7fcb8c458640 1 -- 172.21.15.28:0/130124743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb8419d600 con 0x7fcb8419c880 2024-04-25T01:39:15.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.607+0000 7fcb7b7fe640 1 -- 172.21.15.28:0/130124743 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb7c003dd0 con 0x7fcb8419c880 2024-04-25T01:39:15.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.607+0000 7fcb7b7fe640 1 -- 172.21.15.28:0/130124743 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c02fc40 con 0x7fcb8419c880 2024-04-25T01:39:15.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.607+0000 7fcb8c458640 1 -- 172.21.15.28:0/130124743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb841a4e50 con 0x7fcb8419c880 2024-04-25T01:39:15.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.609+0000 7fcb7b7fe640 1 -- 172.21.15.28:0/130124743 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fcb7c004050 con 0x7fcb8419c880 2024-04-25T01:39:15.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.609+0000 7fcb8f165740 1 Processor -- start 2024-04-25T01:39:15.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb8f165740 1 -- start start 2024-04-25T01:39:15.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb8f165740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56383e840630 0x56383e840a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb8f165740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e840f50 0x56383e84b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb8f165740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e84b620 0x56383e84da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb8f165740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56383e7a56b0 con 0x56383e840f50 2024-04-25T01:39:15.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb8f165740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56383e74d880 con 0x56383e840630 2024-04-25T01:39:15.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb8f165740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56383e7a1670 con 0x56383e84b620 2024-04-25T01:39:15.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb7b7fe640 1 --2- 172.21.15.28:0/130124743 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcb700a2d00 0x7fcb700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb8a1f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e840f50 0x56383e84b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb8a1f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e840f50 0x56383e84b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49050/0 (socket says 172.21.15.28:49050) 2024-04-25T01:39:15.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb8a1f4640 1 -- 172.21.15.28:0/1952353706 learned_addr learned my addr 172.21.15.28:0/1952353706 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:15.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb8a9f5640 1 --2- 172.21.15.28:0/130124743 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcb700a2d00 0x7fcb700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb899f3640 1 --2- 172.21.15.28:0/1952353706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56383e840630 0x56383e840a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb7b7fe640 1 -- 172.21.15.28:0/130124743 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fcb7c0f08d0 con 0x7fcb8419c880 2024-04-25T01:39:15.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb899f3640 1 --2- 172.21.15.28:0/1952353706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e84b620 0x56383e84da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb8a1f4640 1 -- 172.21.15.28:0/1952353706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e84b620 msgr2=0x56383e84da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:15.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb8a1f4640 1 --2- 172.21.15.28:0/1952353706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e84b620 0x56383e84da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb8a1f4640 1 -- 172.21.15.28:0/1952353706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56383e840630 msgr2=0x56383e840a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:15.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb8a1f4640 1 --2- 172.21.15.28:0/1952353706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56383e840630 0x56383e840a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.610+0000 7fcb8a1f4640 1 -- 172.21.15.28:0/1952353706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56383e5837d0 con 0x56383e840f50 2024-04-25T01:39:15.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.611+0000 7fcb8a1f4640 1 --2- 172.21.15.28:0/1952353706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e840f50 0x56383e84b0e0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fcb800114c0 tx=0x7fcb80011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ad28228f974f804 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:15.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.611+0000 7fcb8a9f5640 1 --2- 172.21.15.28:0/130124743 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcb700a2d00 0x7fcb700a51c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0b9050 tx=0x7fcb7c02a8c0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:15.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.611+0000 7fcb78bfe640 1 -- 172.21.15.28:0/1952353706 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80004020 con 0x56383e840f50 2024-04-25T01:39:15.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.611+0000 7fcb78bfe640 1 -- 172.21.15.28:0/1952353706 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb80005500 con 0x56383e840f50 2024-04-25T01:39:15.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.611+0000 7fcb78bfe640 1 -- 172.21.15.28:0/1952353706 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80004db0 con 0x56383e840f50 2024-04-25T01:39:15.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.611+0000 7fcb8f165740 1 -- 172.21.15.28:0/1952353706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e840f50 msgr2=0x56383e84b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:15.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.611+0000 7fcb8f165740 1 --2- 172.21.15.28:0/1952353706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e840f50 0x56383e84b0e0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fcb800114c0 tx=0x7fcb80011980 comp rx=0 tx=0).stop 2024-04-25T01:39:15.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.612+0000 7fcb8f165740 1 -- 172.21.15.28:0/1952353706 shutdown_connections 2024-04-25T01:39:15.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.612+0000 7fcb8f165740 1 --2- 172.21.15.28:0/1952353706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e84b620 0x56383e84da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.612+0000 7fcb8f165740 1 --2- 172.21.15.28:0/1952353706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e840f50 0x56383e84b0e0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.612+0000 7fcb8f165740 1 --2- 172.21.15.28:0/1952353706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56383e840630 0x56383e840a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.612+0000 7fcb8f165740 1 -- 172.21.15.28:0/1952353706 >> 172.21.15.28:0/1952353706 conn(0x56383e83c950 msgr2=0x56383e83cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:15.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.612+0000 7fcb8f165740 1 -- 172.21.15.28:0/1952353706 shutdown_connections 2024-04-25T01:39:15.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.612+0000 7fcb8f165740 1 -- 172.21.15.28:0/1952353706 wait complete. 2024-04-25T01:39:15.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.612+0000 7fcb8f165740 1 -- start start 2024-04-25T01:39:15.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.612+0000 7fcb8f165740 1 Processor -- start 2024-04-25T01:39:15.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.612+0000 7fcb8f165740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56383e840630 0x56383e8484c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.612+0000 7fcb8f165740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e840f50 0x56383e848a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.612+0000 7fcb8f165740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e84b620 0x56383e848f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.612+0000 7fcb8f165740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56383e84e410 con 0x56383e84b620 2024-04-25T01:39:15.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.612+0000 7fcb8f165740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56383e760f50 con 0x56383e840630 2024-04-25T01:39:15.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.612+0000 7fcb8f165740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56383e84e590 con 0x56383e840f50 2024-04-25T01:39:15.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.612+0000 7fcb8f165740 10 client.0 _ll_register_callbacks cb 0x56383e79c000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:15.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.612+0000 7fcb899f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e84b620 0x56383e848f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.612+0000 7fcb899f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e84b620 0x56383e848f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49060/0 (socket says 172.21.15.28:49060) 2024-04-25T01:39:15.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.612+0000 7fcb899f3640 1 -- 172.21.15.28:0/1608762964 learned_addr learned my addr 172.21.15.28:0/1608762964 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:15.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.613+0000 7fcb899f3640 1 --2- 172.21.15.28:0/1608762964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56383e840630 0x56383e8484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.613+0000 7fcb8a1f4640 1 --2- 172.21.15.28:0/1608762964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e840f50 0x56383e848a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.613+0000 7fcb899f3640 1 -- 172.21.15.28:0/1608762964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e840f50 msgr2=0x56383e848a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:15.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.613+0000 7fcb899f3640 1 --2- 172.21.15.28:0/1608762964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e840f50 0x56383e848a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.613+0000 7fcb899f3640 1 -- 172.21.15.28:0/1608762964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56383e840630 msgr2=0x56383e8484c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:15.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.613+0000 7fcb899f3640 1 --2- 172.21.15.28:0/1608762964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56383e840630 0x56383e8484c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.613+0000 7fcb899f3640 1 -- 172.21.15.28:0/1608762964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56383e4b93f0 con 0x56383e84b620 2024-04-25T01:39:15.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.613+0000 7fcb899f3640 1 --2- 172.21.15.28:0/1608762964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e84b620 0x56383e848f40 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fcb840780f0 tx=0x7fcb74004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=790f08cf2fdc7ee7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:15.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.613+0000 7fcb6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:15.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.613+0000 7fcb6dffb640 1 -- 172.21.15.28:0/1608762964 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb74012e50 con 0x56383e84b620 2024-04-25T01:39:15.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.613+0000 7fcb6dffb640 1 -- 172.21.15.28:0/1608762964 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fcb740109b0 con 0x56383e84b620 2024-04-25T01:39:15.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.613+0000 7fcb8f165740 1 -- 172.21.15.28:0/1608762964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb80005500 con 0x56383e84b620 2024-04-25T01:39:15.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.613+0000 7fcb6dffb640 1 -- 172.21.15.28:0/1608762964 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcb74010bb0 con 0x56383e84b620 2024-04-25T01:39:15.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.613+0000 7fcb8f165740 10 client.17835 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:15.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.613+0000 7fcb8f165740 1 -- 172.21.15.28:0/1608762964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56383e4790f0 con 0x56383e84b620 2024-04-25T01:39:15.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.613+0000 7fcb6dffb640 1 -- 172.21.15.28:0/1608762964 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb80005500 con 0x56383e84b620 2024-04-25T01:39:15.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.614+0000 7fcb6dffb640 1 -- 172.21.15.28:0/1608762964 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcb7400f040 con 0x56383e84b620 2024-04-25T01:39:15.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.614+0000 7fcb8f165740 10 client.17835 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:15.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.614+0000 7fcb8f165740 10 client.17835 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:15.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.614+0000 7fcb8f165740 20 client.17835 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:15.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.614+0000 7fcb8f165740 1 --2- 172.21.15.28:0/1608762964 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56383e85b8f0 0x56383e85dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.614+0000 7fcb8f165740 4 client.17835 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:15.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.614+0000 7fcb8f165740 1 -- 172.21.15.28:0/1608762964 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56383e7b2bf0 con 0x56383e85b8f0 2024-04-25T01:39:15.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.614+0000 7fcb8a1f4640 1 --2- 172.21.15.28:0/1608762964 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56383e85b8f0 0x56383e85dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.614+0000 7fcb8a1f4640 1 --2- 172.21.15.28:0/1608762964 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56383e85b8f0 0x56383e85dd20 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bd315f68f43e822 server_cookie=4a2f71dfb8fd9bcc in_seq=0 out_seq=0 2024-04-25T01:39:15.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.614+0000 7fcb6dffb640 10 client.17835 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:15.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.615+0000 7fcb6dffb640 1 -- 172.21.15.28:0/1608762964 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56383e7b2bf0 con 0x56383e85b8f0 2024-04-25T01:39:15.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.615+0000 7fcb6dffb640 10 client.17835 handle_command_reply: tid=0 2024-04-25T01:39:15.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.615+0000 7fcb8f165740 1 client.17835 shutdown 2024-04-25T01:39:15.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.615+0000 7fcb8f165740 1 -- 172.21.15.28:0/1608762964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e84b620 msgr2=0x56383e848f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:15.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.615+0000 7fcb8f165740 1 --2- 172.21.15.28:0/1608762964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e84b620 0x56383e848f40 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fcb840780f0 tx=0x7fcb74004730 comp rx=0 tx=0).stop 2024-04-25T01:39:15.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.615+0000 7fcb8f165740 1 -- 172.21.15.28:0/1608762964 shutdown_connections 2024-04-25T01:39:15.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.615+0000 7fcb8f165740 1 --2- 172.21.15.28:0/1608762964 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56383e85b8f0 0x56383e85dd20 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.615+0000 7fcb8f165740 1 --2- 172.21.15.28:0/1608762964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e84b620 0x56383e848f40 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.615+0000 7fcb8f165740 1 --2- 172.21.15.28:0/1608762964 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e840f50 0x56383e848a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.615+0000 7fcb6dffb640 0 client.17835 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:15.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.615+0000 7fcb8f165740 1 --2- 172.21.15.28:0/1608762964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56383e840630 0x56383e8484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.616+0000 7fcb8f165740 1 -- 172.21.15.28:0/1608762964 >> 172.21.15.28:0/1608762964 conn(0x56383e83c950 msgr2=0x56383e842650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:15.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.616+0000 7fcb8f165740 1 -- 172.21.15.28:0/1608762964 shutdown_connections 2024-04-25T01:39:15.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.616+0000 7fcb8f165740 1 -- 172.21.15.28:0/1608762964 wait complete. 2024-04-25T01:39:15.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.616+0000 7fcb8f165740 20 client.17835 trim_cache size 0 max 16384 2024-04-25T01:39:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.633+0000 7fcb8f165740 1 Processor -- start 2024-04-25T01:39:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.633+0000 7fcb8f165740 1 -- start start 2024-04-25T01:39:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.633+0000 7fcb8f165740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e840630 0x56383e857300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.633+0000 7fcb8f165740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e840f50 0x56383e842dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.633+0000 7fcb8f165740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56383e84b620 0x56383e83b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.633+0000 7fcb8f165740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcb841aea10 con 0x56383e840630 2024-04-25T01:39:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.633+0000 7fcb8f165740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcb841b0550 con 0x56383e84b620 2024-04-25T01:39:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.633+0000 7fcb8f165740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fcb841af0e0 con 0x56383e840f50 2024-04-25T01:39:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.633+0000 7fcb899f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e840630 0x56383e857300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.633+0000 7fcb899f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e840630 0x56383e857300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49068/0 (socket says 172.21.15.28:49068) 2024-04-25T01:39:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.633+0000 7fcb899f3640 1 -- 172.21.15.28:0/3182228623 learned_addr learned my addr 172.21.15.28:0/3182228623 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.634+0000 7fcb8a1f4640 1 --2- 172.21.15.28:0/3182228623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e840f50 0x56383e842dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.634+0000 7fcb899f3640 1 -- 172.21.15.28:0/3182228623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e840f50 msgr2=0x56383e842dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.634+0000 7fcb899f3640 1 --2- 172.21.15.28:0/3182228623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e840f50 0x56383e842dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.634+0000 7fcb899f3640 1 -- 172.21.15.28:0/3182228623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56383e84b620 msgr2=0x56383e83b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.634+0000 7fcb899f3640 1 --2- 172.21.15.28:0/3182228623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56383e84b620 0x56383e83b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.634+0000 7fcb899f3640 1 -- 172.21.15.28:0/3182228623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56383e82e090 con 0x56383e840630 2024-04-25T01:39:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.634+0000 7fcb899f3640 1 --2- 172.21.15.28:0/3182228623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e840630 0x56383e857300 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fcb74005d00 tx=0x7fcb7400bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8af7d961001d02c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.634+0000 7fcb6ffff640 1 -- 172.21.15.28:0/3182228623 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb74014e60 con 0x56383e840630 2024-04-25T01:39:15.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.634+0000 7fcb6ffff640 1 -- 172.21.15.28:0/3182228623 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb74012e50 con 0x56383e840630 2024-04-25T01:39:15.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.634+0000 7fcb6ffff640 1 -- 172.21.15.28:0/3182228623 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb74020b20 con 0x56383e840630 2024-04-25T01:39:15.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.634+0000 7fcb8f165740 1 -- 172.21.15.28:0/3182228623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e840630 msgr2=0x56383e857300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:15.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.634+0000 7fcb8f165740 1 --2- 172.21.15.28:0/3182228623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e840630 0x56383e857300 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fcb74005d00 tx=0x7fcb7400bc90 comp rx=0 tx=0).stop 2024-04-25T01:39:15.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb8f165740 1 -- 172.21.15.28:0/3182228623 shutdown_connections 2024-04-25T01:39:15.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb8f165740 1 --2- 172.21.15.28:0/3182228623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56383e84b620 0x56383e83b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb8f165740 1 --2- 172.21.15.28:0/3182228623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e840f50 0x56383e842dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb8f165740 1 --2- 172.21.15.28:0/3182228623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e840630 0x56383e857300 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb8f165740 1 -- 172.21.15.28:0/3182228623 >> 172.21.15.28:0/3182228623 conn(0x56383e83c950 msgr2=0x56383e83b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:15.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb8f165740 1 -- 172.21.15.28:0/3182228623 shutdown_connections 2024-04-25T01:39:15.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb8f165740 1 -- 172.21.15.28:0/3182228623 wait complete. 2024-04-25T01:39:15.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb8f165740 1 -- start start 2024-04-25T01:39:15.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb8f165740 1 Processor -- start 2024-04-25T01:39:15.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb8f165740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56383e840630 0x56383e853a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb8f165740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e840f50 0x56383e844b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb8f165740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e84b620 0x56383e845080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb8f165740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcb841afe80 con 0x56383e84b620 2024-04-25T01:39:15.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb8f165740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcb841af7b0 con 0x56383e840630 2024-04-25T01:39:15.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb8f165740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fcb841ac9f0 con 0x56383e840f50 2024-04-25T01:39:15.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb8f165740 10 client.0 _ll_register_callbacks cb 0x7fcb7400ba70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:15.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb899f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e84b620 0x56383e845080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb899f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e84b620 0x56383e845080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49084/0 (socket says 172.21.15.28:49084) 2024-04-25T01:39:15.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb899f3640 1 -- 172.21.15.28:0/4156088131 learned_addr learned my addr 172.21.15.28:0/4156088131 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:15.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.635+0000 7fcb899f3640 1 --2- 172.21.15.28:0/4156088131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56383e840630 0x56383e853a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.636+0000 7fcb8a1f4640 1 --2- 172.21.15.28:0/4156088131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e840f50 0x56383e844b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.636+0000 7fcb899f3640 1 -- 172.21.15.28:0/4156088131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e840f50 msgr2=0x56383e844b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:15.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.636+0000 7fcb899f3640 1 --2- 172.21.15.28:0/4156088131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e840f50 0x56383e844b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.636+0000 7fcb899f3640 1 -- 172.21.15.28:0/4156088131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56383e840630 msgr2=0x56383e853a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:15.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.636+0000 7fcb899f3640 1 --2- 172.21.15.28:0/4156088131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56383e840630 0x56383e853a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.636+0000 7fcb899f3640 1 -- 172.21.15.28:0/4156088131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56383e8455c0 con 0x56383e84b620 2024-04-25T01:39:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.636+0000 7fcb899f3640 1 --2- 172.21.15.28:0/4156088131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e84b620 0x56383e845080 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fcb74005940 tx=0x7fcb74012250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33a7e7a95a156bc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.636+0000 7fcb6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.636+0000 7fcb6dffb640 1 -- 172.21.15.28:0/4156088131 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb740122f0 con 0x56383e84b620 2024-04-25T01:39:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.636+0000 7fcb6dffb640 1 -- 172.21.15.28:0/4156088131 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fcb74036040 con 0x56383e84b620 2024-04-25T01:39:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.636+0000 7fcb8f165740 1 -- 172.21.15.28:0/4156088131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb74012e50 con 0x56383e84b620 2024-04-25T01:39:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.636+0000 7fcb6dffb640 1 -- 172.21.15.28:0/4156088131 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcb740109b0 con 0x56383e84b620 2024-04-25T01:39:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.636+0000 7fcb8f165740 10 client.17841 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.636+0000 7fcb8f165740 1 -- 172.21.15.28:0/4156088131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56383e8c1ad0 con 0x56383e84b620 2024-04-25T01:39:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.636+0000 7fcb6dffb640 1 -- 172.21.15.28:0/4156088131 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb74012e50 con 0x56383e84b620 2024-04-25T01:39:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.636+0000 7fcb6dffb640 1 -- 172.21.15.28:0/4156088131 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcb7403b4a0 con 0x56383e84b620 2024-04-25T01:39:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.637+0000 7fcb8f165740 10 client.17841 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.637+0000 7fcb8f165740 10 client.17841 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.637+0000 7fcb8f165740 20 client.17841 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:15.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.637+0000 7fcb8f165740 1 --2- 172.21.15.28:0/4156088131 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56383e8c2080 0x56383e8c44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.637+0000 7fcb8f165740 4 client.17841 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:39:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.637+0000 7fcb8f165740 1 -- 172.21.15.28:0/4156088131 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56383e7acf40 con 0x56383e8c2080 2024-04-25T01:39:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.637+0000 7fcb8a1f4640 1 --2- 172.21.15.28:0/4156088131 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56383e8c2080 0x56383e8c44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.637+0000 7fcb8a1f4640 1 --2- 172.21.15.28:0/4156088131 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56383e8c2080 0x56383e8c44a0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=542583990530b223 server_cookie=f2fb9389f93c0f44 in_seq=0 out_seq=0 2024-04-25T01:39:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.637+0000 7fcb6dffb640 10 client.17841 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.637+0000 7fcb6dffb640 1 -- 172.21.15.28:0/4156088131 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56383e7acf40 con 0x56383e8c2080 2024-04-25T01:39:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.637+0000 7fcb6dffb640 10 client.17841 handle_command_reply: tid=0 2024-04-25T01:39:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.637+0000 7fcb8f165740 1 client.17841 shutdown 2024-04-25T01:39:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.637+0000 7fcb8f165740 1 -- 172.21.15.28:0/4156088131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e84b620 msgr2=0x56383e845080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.637+0000 7fcb8f165740 1 --2- 172.21.15.28:0/4156088131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e84b620 0x56383e845080 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fcb74005940 tx=0x7fcb74012250 comp rx=0 tx=0).stop 2024-04-25T01:39:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8f165740 1 -- 172.21.15.28:0/4156088131 shutdown_connections 2024-04-25T01:39:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8f165740 1 --2- 172.21.15.28:0/4156088131 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56383e8c2080 0x56383e8c44a0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8f165740 1 --2- 172.21.15.28:0/4156088131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56383e84b620 0x56383e845080 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb6dffb640 0 client.17841 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:15.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8f165740 1 --2- 172.21.15.28:0/4156088131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56383e840f50 0x56383e844b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8f165740 1 --2- 172.21.15.28:0/4156088131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56383e840630 0x56383e853a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8f165740 1 -- 172.21.15.28:0/4156088131 >> 172.21.15.28:0/4156088131 conn(0x56383e83c950 msgr2=0x56383e848d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:15.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8f165740 1 -- 172.21.15.28:0/4156088131 shutdown_connections 2024-04-25T01:39:15.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8f165740 1 -- 172.21.15.28:0/4156088131 wait complete. 2024-04-25T01:39:15.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8f165740 20 client.17841 trim_cache size 0 max 16384 2024-04-25T01:39:15.627 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:39:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:39:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-25T01:39:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "a8cd9c38-e13c-4096-a640-dc20ca57b641": { 2024-04-25T01:39:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-25T01:39:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "a8cd9c38-e13c-4096-a640-dc20ca57b641", 2024-04-25T01:39:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-25T01:39:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:39:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:39:15.628 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:39:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8c458640 1 -- 172.21.15.28:0/130124743 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcb700a2d00 msgr2=0x7fcb700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8c458640 1 --2- 172.21.15.28:0/130124743 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcb700a2d00 0x7fcb700a51c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0b9050 tx=0x7fcb7c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8c458640 1 -- 172.21.15.28:0/130124743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8419c880 msgr2=0x7fcb8419cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8c458640 1 --2- 172.21.15.28:0/130124743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8419c880 0x7fcb8419cd30 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c031a20 tx=0x7fcb7c031a50 comp rx=0 tx=0).stop 2024-04-25T01:39:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8c458640 1 -- 172.21.15.28:0/130124743 shutdown_connections 2024-04-25T01:39:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8c458640 1 --2- 172.21.15.28:0/130124743 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcb700a2d00 0x7fcb700a51c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8c458640 1 --2- 172.21.15.28:0/130124743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb8419c880 0x7fcb8419cd30 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8c458640 1 --2- 172.21.15.28:0/130124743 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb84105160 0x7fcb84198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8c458640 1 --2- 172.21.15.28:0/130124743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb841002e0 0x7fcb84198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8c458640 1 -- 172.21.15.28:0/130124743 >> 172.21.15.28:0/130124743 conn(0x7fcb840758d0 msgr2=0x7fcb84106c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8c458640 1 -- 172.21.15.28:0/130124743 shutdown_connections 2024-04-25T01:39:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:15.638+0000 7fcb8c458640 1 -- 172.21.15.28:0/130124743 wait complete. 2024-04-25T01:39:15.632 INFO:tasks.cephfs.filesystem:scrub status for tag:a8cd9c38-e13c-4096-a640-dc20ca57b641 - {'path': '/', 'tag': 'a8cd9c38-e13c-4096-a640-dc20ca57b641', 'options': 'recursive,force'} 2024-04-25T01:39:16.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:15 smithi032 ceph-mon[32461]: pgmap v190: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-25T01:39:16.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:15 smithi028 ceph-mon[29390]: pgmap v190: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-25T01:39:16.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:15 smithi130 ceph-mon[32702]: pgmap v190: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-25T01:39:17.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:16 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:39:17.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:16 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:17.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:16 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:39:17.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:16 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:17.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:16 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:39:17.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:16 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:17.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:16 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:39:17.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:16 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:17.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:16 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:39:17.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:16 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:17.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:16 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:39:17.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:16 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:18.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:17 smithi032 ceph-mon[32461]: pgmap v191: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 11 op/s 2024-04-25T01:39:18.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:17 smithi028 ceph-mon[29390]: pgmap v191: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 11 op/s 2024-04-25T01:39:18.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:17 smithi130 ceph-mon[32702]: pgmap v191: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 11 op/s 2024-04-25T01:39:19.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:19 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:39:19.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:19 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:39:19.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:19 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:39:20.238 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:20 smithi032 ceph-mon[32461]: pgmap v192: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T01:39:20.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:20 smithi028 ceph-mon[29390]: pgmap v192: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T01:39:20.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:20 smithi130 ceph-mon[32702]: pgmap v192: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T01:39:22.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:22 smithi028 ceph-mon[29390]: pgmap v193: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T01:39:22.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:22 smithi130 ceph-mon[32702]: pgmap v193: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T01:39:22.488 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:22 smithi032 ceph-mon[32461]: pgmap v193: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-04-25T01:39:24.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:24 smithi028 ceph-mon[29390]: pgmap v194: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T01:39:24.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:24 smithi130 ceph-mon[32702]: pgmap v194: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T01:39:24.488 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:24 smithi032 ceph-mon[32461]: pgmap v194: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 85 B/s wr, 12 op/s 2024-04-25T01:39:26.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:26 smithi028 ceph-mon[29390]: pgmap v195: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:39:26.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:26 smithi130 ceph-mon[32702]: pgmap v195: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:39:26.488 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:26 smithi032 ceph-mon[32461]: pgmap v195: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:39:28.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:28 smithi028 ceph-mon[29390]: pgmap v196: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-25T01:39:28.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:28 smithi130 ceph-mon[32702]: pgmap v196: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-25T01:39:28.489 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:28 smithi032 ceph-mon[32461]: pgmap v196: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-25T01:39:30.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:30 smithi028 ceph-mon[29390]: pgmap v197: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T01:39:30.463 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:30 smithi130 ceph-mon[32702]: pgmap v197: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T01:39:30.489 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:30 smithi032 ceph-mon[32461]: pgmap v197: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T01:39:32.266 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:32 smithi130 ceph-mon[32702]: pgmap v198: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:39:32.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:32 smithi028 ceph-mon[29390]: pgmap v198: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:39:32.489 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:32 smithi032 ceph-mon[32461]: pgmap v198: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:39:34.291 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:34 smithi032 ceph-mon[32461]: pgmap v199: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:39:34.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:34 smithi028 ceph-mon[29390]: pgmap v199: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:39:34.464 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:34 smithi130 ceph-mon[32702]: pgmap v199: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:39:36.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:36 smithi028 ceph-mon[29390]: pgmap v200: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:39:36.464 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:36 smithi130 ceph-mon[32702]: pgmap v200: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:39:36.489 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:36 smithi032 ceph-mon[32461]: pgmap v200: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:39:38.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:38 smithi028 ceph-mon[29390]: pgmap v201: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:39:38.464 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:38 smithi130 ceph-mon[32702]: pgmap v201: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:39:38.489 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:38 smithi032 ceph-mon[32461]: pgmap v201: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:39:40.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:40 smithi028 ceph-mon[29390]: pgmap v202: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:39:40.464 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:40 smithi130 ceph-mon[32702]: pgmap v202: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:39:40.489 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:40 smithi032 ceph-mon[32461]: pgmap v202: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:39:42.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:42 smithi028 ceph-mon[29390]: pgmap v203: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:39:42.464 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:42 smithi130 ceph-mon[32702]: pgmap v203: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:39:42.489 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:42 smithi032 ceph-mon[32461]: pgmap v203: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:39:44.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:44 smithi028 ceph-mon[29390]: pgmap v204: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:39:44.464 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:44 smithi130 ceph-mon[32702]: pgmap v204: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:39:44.489 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:44 smithi032 ceph-mon[32461]: pgmap v204: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:39:45.633 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:39:45.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:39:45.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.769+0000 7f53e6d63640 1 -- 172.21.15.28:0/4009276425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e010bd00 msgr2=0x7f53e010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:45.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.769+0000 7f53e6d63640 1 --2- 172.21.15.28:0/4009276425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e010bd00 0x7f53e010e1a0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f53d8009ca0 tx=0x7f53d802a9e0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.769+0000 7f53e6d63640 1 -- 172.21.15.28:0/4009276425 shutdown_connections 2024-04-25T01:39:45.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.769+0000 7f53e6d63640 1 --2- 172.21.15.28:0/4009276425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e010bd00 0x7f53e010e1a0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.769+0000 7f53e6d63640 1 --2- 172.21.15.28:0/4009276425 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53e01032e0 0x7f53e010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.769+0000 7f53e6d63640 1 --2- 172.21.15.28:0/4009276425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e01029c0 0x7f53e0102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.769+0000 7f53e6d63640 1 -- 172.21.15.28:0/4009276425 >> 172.21.15.28:0/4009276425 conn(0x7f53e0077740 msgr2=0x7f53e0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.769+0000 7f53e6d63640 1 -- 172.21.15.28:0/4009276425 shutdown_connections 2024-04-25T01:39:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.769+0000 7f53e6d63640 1 -- 172.21.15.28:0/4009276425 wait complete. 2024-04-25T01:39:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.770+0000 7f53e6d63640 1 Processor -- start 2024-04-25T01:39:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.770+0000 7f53e6d63640 1 -- start start 2024-04-25T01:39:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.770+0000 7f53e6d63640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e01029c0 0x7f53e0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.770+0000 7f53e6d63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e01032e0 0x7f53e0198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.770+0000 7f53e6d63640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53e010bd00 0x7f53e019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.770+0000 7f53e6d63640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f53e0111b50 con 0x7f53e01032e0 2024-04-25T01:39:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.770+0000 7f53e6d63640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f53e01119d0 con 0x7f53e01029c0 2024-04-25T01:39:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.770+0000 7f53e6d63640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f53e0111cd0 con 0x7f53e010bd00 2024-04-25T01:39:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.770+0000 7f53dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e01032e0 0x7f53e0198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.770+0000 7f53dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e01032e0 0x7f53e0198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51614/0 (socket says 172.21.15.28:51614) 2024-04-25T01:39:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.770+0000 7f53dffff640 1 -- 172.21.15.28:0/4246604906 learned_addr learned my addr 172.21.15.28:0/4246604906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.770+0000 7f53e5300640 1 --2- 172.21.15.28:0/4246604906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53e010bd00 0x7f53e019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.770+0000 7f53e4aff640 1 --2- 172.21.15.28:0/4246604906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e01029c0 0x7f53e0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.770+0000 7f53dffff640 1 -- 172.21.15.28:0/4246604906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53e010bd00 msgr2=0x7f53e019c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.770+0000 7f53dffff640 1 --2- 172.21.15.28:0/4246604906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53e010bd00 0x7f53e019c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.770+0000 7f53dffff640 1 -- 172.21.15.28:0/4246604906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e01029c0 msgr2=0x7f53e0197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.770+0000 7f53dffff640 1 --2- 172.21.15.28:0/4246604906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e01029c0 0x7f53e0197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.770+0000 7f53dffff640 1 -- 172.21.15.28:0/4246604906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53e019cf70 con 0x7f53e01032e0 2024-04-25T01:39:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.770+0000 7f53dffff640 1 --2- 172.21.15.28:0/4246604906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e01032e0 0x7f53e0198510 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f53d400d950 tx=0x7f53d400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.771+0000 7f53ddffb640 1 -- 172.21.15.28:0/4246604906 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d4017030 con 0x7f53e01032e0 2024-04-25T01:39:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.771+0000 7f53ddffb640 1 -- 172.21.15.28:0/4246604906 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53d4004020 con 0x7f53e01032e0 2024-04-25T01:39:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.771+0000 7f53e6d63640 1 -- 172.21.15.28:0/4246604906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53e019d260 con 0x7f53e01032e0 2024-04-25T01:39:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.771+0000 7f53ddffb640 1 -- 172.21.15.28:0/4246604906 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d40047a0 con 0x7f53e01032e0 2024-04-25T01:39:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.771+0000 7f53e6d63640 1 -- 172.21.15.28:0/4246604906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53e01a4af0 con 0x7f53e01032e0 2024-04-25T01:39:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.773+0000 7f53ddffb640 1 -- 172.21.15.28:0/4246604906 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f53d40041d0 con 0x7f53e01032e0 2024-04-25T01:39:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.773+0000 7f53e9a74740 1 Processor -- start 2024-04-25T01:39:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.773+0000 7f53e9a74740 1 -- start start 2024-04-25T01:39:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.773+0000 7f53ddffb640 1 --2- 172.21.15.28:0/4246604906 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f53c80a2d80 0x7f53c80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53e5300640 1 --2- 172.21.15.28:0/4246604906 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f53c80a2d80 0x7f53c80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53e9a74740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b89630 0x55db11b89a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53e9a74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89f50 0x55db11b940e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53e9a74740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b94620 0x55db11b96a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53e9a74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db11aee6b0 con 0x55db11b89f50 2024-04-25T01:39:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53ddffb640 1 -- 172.21.15.28:0/4246604906 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f53d40cf050 con 0x7f53e01032e0 2024-04-25T01:39:45.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53e9a74740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55db11a96880 con 0x55db11b94620 2024-04-25T01:39:45.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53e9a74740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55db11aea670 con 0x55db11b89630 2024-04-25T01:39:45.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89f50 0x55db11b940e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89f50 0x55db11b940e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51622/0 (socket says 172.21.15.28:51622) 2024-04-25T01:39:45.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53dffff640 1 -- 172.21.15.28:0/331085756 learned_addr learned my addr 172.21.15.28:0/331085756 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:45.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53e4aff640 1 --2- 172.21.15.28:0/331085756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b89630 0x55db11b89a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53e5300640 1 --2- 172.21.15.28:0/331085756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b94620 0x55db11b96a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53dffff640 1 -- 172.21.15.28:0/331085756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b89630 msgr2=0x55db11b89a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:45.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53dffff640 1 --2- 172.21.15.28:0/331085756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b89630 0x55db11b89a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53dffff640 1 -- 172.21.15.28:0/331085756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b94620 msgr2=0x55db11b96a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:45.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53dffff640 1 --2- 172.21.15.28:0/331085756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b94620 0x55db11b96a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53dffff640 1 -- 172.21.15.28:0/331085756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db118cc7d0 con 0x55db11b89f50 2024-04-25T01:39:45.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53e5300640 1 --2- 172.21.15.28:0/331085756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b94620 0x55db11b96a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:45.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53e4aff640 1 --2- 172.21.15.28:0/331085756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b89630 0x55db11b89a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:45.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53e5300640 1 --2- 172.21.15.28:0/4246604906 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f53c80a2d80 0x7f53c80a5240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f53d8006010 tx=0x7f53d8005d80 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:45.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.774+0000 7f53dffff640 1 --2- 172.21.15.28:0/331085756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89f50 0x55db11b940e0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f53d409b1c0 tx=0x7f53d409b120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c2dd2dd1827b1ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:45.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.775+0000 7f53c6ffd640 1 -- 172.21.15.28:0/331085756 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d409b230 con 0x55db11b89f50 2024-04-25T01:39:45.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.775+0000 7f53c6ffd640 1 -- 172.21.15.28:0/331085756 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53d409f5e0 con 0x55db11b89f50 2024-04-25T01:39:45.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.775+0000 7f53c6ffd640 1 -- 172.21.15.28:0/331085756 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d40ccbb0 con 0x55db11b89f50 2024-04-25T01:39:45.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.775+0000 7f53e9a74740 1 -- 172.21.15.28:0/331085756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89f50 msgr2=0x55db11b940e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:45.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.775+0000 7f53e9a74740 1 --2- 172.21.15.28:0/331085756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89f50 0x55db11b940e0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f53d409b1c0 tx=0x7f53d409b120 comp rx=0 tx=0).stop 2024-04-25T01:39:45.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.775+0000 7f53e9a74740 1 -- 172.21.15.28:0/331085756 shutdown_connections 2024-04-25T01:39:45.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.775+0000 7f53e9a74740 1 --2- 172.21.15.28:0/331085756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b94620 0x55db11b96a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.775+0000 7f53e9a74740 1 --2- 172.21.15.28:0/331085756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89f50 0x55db11b940e0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.775+0000 7f53e9a74740 1 --2- 172.21.15.28:0/331085756 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b89630 0x55db11b89a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.775+0000 7f53e9a74740 1 -- 172.21.15.28:0/331085756 >> 172.21.15.28:0/331085756 conn(0x55db11b85950 msgr2=0x55db11b85d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:45.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.775+0000 7f53e9a74740 1 -- 172.21.15.28:0/331085756 shutdown_connections 2024-04-25T01:39:45.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.775+0000 7f53e9a74740 1 -- 172.21.15.28:0/331085756 wait complete. 2024-04-25T01:39:45.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.776+0000 7f53e9a74740 1 -- start start 2024-04-25T01:39:45.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.776+0000 7f53e9a74740 1 Processor -- start 2024-04-25T01:39:45.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.776+0000 7f53e9a74740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b89630 0x55db11b91630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.776+0000 7f53e9a74740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b89f50 0x55db11b91b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.776+0000 7f53e9a74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b94620 0x55db11b920b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.776+0000 7f53e9a74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db11b97410 con 0x55db11b94620 2024-04-25T01:39:45.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.776+0000 7f53e9a74740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55db11aa9f50 con 0x55db11b89f50 2024-04-25T01:39:45.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.776+0000 7f53e9a74740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55db11b97590 con 0x55db11b89630 2024-04-25T01:39:45.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.776+0000 7f53e9a74740 10 client.0 _ll_register_callbacks cb 0x55db11ae5000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:45.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.776+0000 7f53e5300640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b94620 0x55db11b920b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.776+0000 7f53e5300640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b94620 0x55db11b920b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51636/0 (socket says 172.21.15.28:51636) 2024-04-25T01:39:45.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.776+0000 7f53e5300640 1 -- 172.21.15.28:0/3919921122 learned_addr learned my addr 172.21.15.28:0/3919921122 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:45.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.776+0000 7f53dffff640 1 --2- 172.21.15.28:0/3919921122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b89f50 0x55db11b91b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.777+0000 7f53e4aff640 1 --2- 172.21.15.28:0/3919921122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b89630 0x55db11b91630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.777+0000 7f53e5300640 1 -- 172.21.15.28:0/3919921122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b89630 msgr2=0x55db11b91630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:45.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.777+0000 7f53e5300640 1 --2- 172.21.15.28:0/3919921122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b89630 0x55db11b91630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.777+0000 7f53e5300640 1 -- 172.21.15.28:0/3919921122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b89f50 msgr2=0x55db11b91b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:45.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.777+0000 7f53e5300640 1 --2- 172.21.15.28:0/3919921122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b89f50 0x55db11b91b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.777+0000 7f53e5300640 1 -- 172.21.15.28:0/3919921122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db118023f0 con 0x55db11b94620 2024-04-25T01:39:45.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.777+0000 7f53e5300640 1 --2- 172.21.15.28:0/3919921122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b94620 0x55db11b920b0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f53d8035000 tx=0x7f53d802a9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=667b96245d8f83c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:45.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.777+0000 7f53bffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:45.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.777+0000 7f53bffff640 1 -- 172.21.15.28:0/3919921122 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d8033050 con 0x55db11b94620 2024-04-25T01:39:45.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.777+0000 7f53bffff640 1 -- 172.21.15.28:0/3919921122 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f53d802ee70 con 0x55db11b94620 2024-04-25T01:39:45.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.777+0000 7f53e9a74740 1 -- 172.21.15.28:0/3919921122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53d409f5e0 con 0x55db11b94620 2024-04-25T01:39:45.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.777+0000 7f53bffff640 1 -- 172.21.15.28:0/3919921122 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f53d8004040 con 0x55db11b94620 2024-04-25T01:39:45.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.777+0000 7f53e9a74740 10 client.17853 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:45.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.777+0000 7f53e9a74740 1 -- 172.21.15.28:0/3919921122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55db117c20f0 con 0x55db11b94620 2024-04-25T01:39:45.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.777+0000 7f53bffff640 1 -- 172.21.15.28:0/3919921122 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53d409f5e0 con 0x55db11b94620 2024-04-25T01:39:45.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.778+0000 7f53bffff640 1 -- 172.21.15.28:0/3919921122 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f53d802fb20 con 0x55db11b94620 2024-04-25T01:39:45.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.778+0000 7f53e9a74740 10 client.17853 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:45.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.778+0000 7f53e9a74740 10 client.17853 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:45.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.778+0000 7f53e9a74740 20 client.17853 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:45.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.778+0000 7f53e9a74740 1 --2- 172.21.15.28:0/3919921122 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db11ba48f0 0x55db11ba6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.778+0000 7f53e9a74740 4 client.17853 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:45.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.778+0000 7f53e9a74740 1 -- 172.21.15.28:0/3919921122 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db11afbbf0 con 0x55db11ba48f0 2024-04-25T01:39:45.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.778+0000 7f53e4aff640 1 --2- 172.21.15.28:0/3919921122 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db11ba48f0 0x55db11ba6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.778+0000 7f53e4aff640 1 --2- 172.21.15.28:0/3919921122 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db11ba48f0 0x55db11ba6ce0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c14727ff480e70b server_cookie=c192591557e6d61d in_seq=0 out_seq=0 2024-04-25T01:39:45.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.778+0000 7f53bffff640 10 client.17853 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:45.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.779+0000 7f53bffff640 1 -- 172.21.15.28:0/3919921122 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55db11afbbf0 con 0x55db11ba48f0 2024-04-25T01:39:45.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.779+0000 7f53bffff640 10 client.17853 handle_command_reply: tid=0 2024-04-25T01:39:45.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.779+0000 7f53e9a74740 1 client.17853 shutdown 2024-04-25T01:39:45.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.779+0000 7f53e9a74740 1 -- 172.21.15.28:0/3919921122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b94620 msgr2=0x55db11b920b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:45.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.779+0000 7f53e9a74740 1 --2- 172.21.15.28:0/3919921122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b94620 0x55db11b920b0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f53d8035000 tx=0x7f53d802a9e0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.779+0000 7f53e9a74740 1 -- 172.21.15.28:0/3919921122 shutdown_connections 2024-04-25T01:39:45.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.779+0000 7f53e9a74740 1 --2- 172.21.15.28:0/3919921122 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db11ba48f0 0x55db11ba6ce0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.779+0000 7f53e9a74740 1 --2- 172.21.15.28:0/3919921122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b94620 0x55db11b920b0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.779+0000 7f53e9a74740 1 --2- 172.21.15.28:0/3919921122 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b89f50 0x55db11b91b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.779+0000 7f53bffff640 0 client.17853 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:45.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.779+0000 7f53e9a74740 1 --2- 172.21.15.28:0/3919921122 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b89630 0x55db11b91630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.779+0000 7f53e9a74740 1 -- 172.21.15.28:0/3919921122 >> 172.21.15.28:0/3919921122 conn(0x55db11b85950 msgr2=0x55db11b8b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:45.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.779+0000 7f53e9a74740 1 -- 172.21.15.28:0/3919921122 shutdown_connections 2024-04-25T01:39:45.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.779+0000 7f53e9a74740 1 -- 172.21.15.28:0/3919921122 wait complete. 2024-04-25T01:39:45.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.780+0000 7f53e9a74740 20 client.17853 trim_cache size 0 max 16384 2024-04-25T01:39:45.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.797+0000 7f53e9a74740 1 Processor -- start 2024-04-25T01:39:45.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.797+0000 7f53e9a74740 1 -- start start 2024-04-25T01:39:45.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.797+0000 7f53e9a74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89630 0x55db11b93d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.797+0000 7f53e9a74740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b89f50 0x55db11ba02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.797+0000 7f53e9a74740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b94620 0x55db11b84970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.797+0000 7f53e9a74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f53e01aea50 con 0x55db11b89630 2024-04-25T01:39:45.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.797+0000 7f53e9a74740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f53e01b0590 con 0x55db11b89f50 2024-04-25T01:39:45.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.797+0000 7f53e9a74740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f53e01af120 con 0x55db11b94620 2024-04-25T01:39:45.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.798+0000 7f53e4aff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89630 0x55db11b93d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.798+0000 7f53e4aff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89630 0x55db11b93d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51648/0 (socket says 172.21.15.28:51648) 2024-04-25T01:39:45.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.798+0000 7f53e4aff640 1 -- 172.21.15.28:0/1392307922 learned_addr learned my addr 172.21.15.28:0/1392307922 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:45.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.798+0000 7f53dffff640 1 --2- 172.21.15.28:0/1392307922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b89f50 0x55db11ba02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.798+0000 7f53e4aff640 1 -- 172.21.15.28:0/1392307922 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b94620 msgr2=0x55db11b84970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:45.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.798+0000 7f53e4aff640 1 --2- 172.21.15.28:0/1392307922 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b94620 0x55db11b84970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.798+0000 7f53e4aff640 1 -- 172.21.15.28:0/1392307922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b89f50 msgr2=0x55db11ba02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:45.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.798+0000 7f53e4aff640 1 --2- 172.21.15.28:0/1392307922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b89f50 0x55db11ba02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.798+0000 7f53e4aff640 1 -- 172.21.15.28:0/1392307922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db11b85290 con 0x55db11b89630 2024-04-25T01:39:45.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.798+0000 7f53e4aff640 1 --2- 172.21.15.28:0/1392307922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89630 0x55db11b93d60 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x55db11ba7520 tx=0x7f53d00044d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=592300082d23bdfd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:45.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.798+0000 7f53c67fc640 1 -- 172.21.15.28:0/1392307922 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d0003dc0 con 0x55db11b89630 2024-04-25T01:39:45.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.798+0000 7f53c67fc640 1 -- 172.21.15.28:0/1392307922 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53d0023070 con 0x55db11b89630 2024-04-25T01:39:45.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.798+0000 7f53c67fc640 1 -- 172.21.15.28:0/1392307922 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d0012de0 con 0x55db11b89630 2024-04-25T01:39:45.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.799+0000 7f53e9a74740 1 -- 172.21.15.28:0/1392307922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89630 msgr2=0x55db11b93d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:45.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.799+0000 7f53e9a74740 1 --2- 172.21.15.28:0/1392307922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89630 0x55db11b93d60 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x55db11ba7520 tx=0x7f53d00044d0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.799+0000 7f53e9a74740 1 -- 172.21.15.28:0/1392307922 shutdown_connections 2024-04-25T01:39:45.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.799+0000 7f53e9a74740 1 --2- 172.21.15.28:0/1392307922 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b94620 0x55db11b84970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.799+0000 7f53e9a74740 1 --2- 172.21.15.28:0/1392307922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b89f50 0x55db11ba02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.799+0000 7f53e9a74740 1 --2- 172.21.15.28:0/1392307922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89630 0x55db11b93d60 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.799+0000 7f53e9a74740 1 -- 172.21.15.28:0/1392307922 >> 172.21.15.28:0/1392307922 conn(0x55db11b85950 msgr2=0x55db11b84260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:45.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.799+0000 7f53e9a74740 1 -- 172.21.15.28:0/1392307922 shutdown_connections 2024-04-25T01:39:45.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.799+0000 7f53e9a74740 1 -- 172.21.15.28:0/1392307922 wait complete. 2024-04-25T01:39:45.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.799+0000 7f53e9a74740 1 -- start start 2024-04-25T01:39:45.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.799+0000 7f53e9a74740 1 Processor -- start 2024-04-25T01:39:45.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53e9a74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89630 0x55db11b983c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53e9a74740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b89f50 0x55db11b98900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53e9a74740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b94620 0x55db11b98e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53e9a74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f53e01afec0 con 0x55db11b89630 2024-04-25T01:39:45.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53e9a74740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f53e01af7f0 con 0x55db11b94620 2024-04-25T01:39:45.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53e9a74740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f53e01aca30 con 0x55db11b89f50 2024-04-25T01:39:45.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53e9a74740 10 client.0 _ll_register_callbacks cb 0x7f53d8009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:45.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53e4aff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89630 0x55db11b983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53e4aff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89630 0x55db11b983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51654/0 (socket says 172.21.15.28:51654) 2024-04-25T01:39:45.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53e4aff640 1 -- 172.21.15.28:0/3191797540 learned_addr learned my addr 172.21.15.28:0/3191797540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:45.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53dffff640 1 --2- 172.21.15.28:0/3191797540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b89f50 0x55db11b98900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53e5300640 1 --2- 172.21.15.28:0/3191797540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b94620 0x55db11b98e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53e4aff640 1 -- 172.21.15.28:0/3191797540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b89f50 msgr2=0x55db11b98900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:45.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53e4aff640 1 --2- 172.21.15.28:0/3191797540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b89f50 0x55db11b98900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53e4aff640 1 -- 172.21.15.28:0/3191797540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b94620 msgr2=0x55db11b98e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:45.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53e4aff640 1 --2- 172.21.15.28:0/3191797540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b94620 0x55db11b98e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53e4aff640 1 -- 172.21.15.28:0/3191797540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db11c0a020 con 0x55db11b89630 2024-04-25T01:39:45.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53e5300640 1 --2- 172.21.15.28:0/3191797540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b94620 0x55db11b98e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:45.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53e4aff640 1 --2- 172.21.15.28:0/3191797540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89630 0x55db11b983c0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f53d0003ca0 tx=0x7f53d0003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b24673419562095 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:45.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53bffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:45.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53bffff640 1 -- 172.21.15.28:0/3191797540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d0012a60 con 0x55db11b89630 2024-04-25T01:39:45.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53bffff640 1 -- 172.21.15.28:0/3191797540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f53d0012c00 con 0x55db11b89630 2024-04-25T01:39:45.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.800+0000 7f53e9a74740 1 -- 172.21.15.28:0/3191797540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53d0023070 con 0x55db11b89630 2024-04-25T01:39:45.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.801+0000 7f53bffff640 1 -- 172.21.15.28:0/3191797540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f53d003a050 con 0x55db11b89630 2024-04-25T01:39:45.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.801+0000 7f53e9a74740 10 client.17859 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:45.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.801+0000 7f53e9a74740 1 -- 172.21.15.28:0/3191797540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55db11c0b2f0 con 0x55db11b89630 2024-04-25T01:39:45.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.801+0000 7f53bffff640 1 -- 172.21.15.28:0/3191797540 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53d0023070 con 0x55db11b89630 2024-04-25T01:39:45.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.801+0000 7f53bffff640 1 -- 172.21.15.28:0/3191797540 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f53d0004700 con 0x55db11b89630 2024-04-25T01:39:45.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.801+0000 7f53e9a74740 10 client.17859 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:45.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.801+0000 7f53e9a74740 10 client.17859 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:45.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.801+0000 7f53e9a74740 20 client.17859 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:45.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.801+0000 7f53e9a74740 1 --2- 172.21.15.28:0/3191797540 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db11c0b9e0 0x55db11c0de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.801+0000 7f53e9a74740 4 client.17859 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:39:45.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.801+0000 7f53e9a74740 1 -- 172.21.15.28:0/3191797540 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55db11af5f40 con 0x55db11c0b9e0 2024-04-25T01:39:45.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.801+0000 7f53dffff640 1 --2- 172.21.15.28:0/3191797540 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db11c0b9e0 0x55db11c0de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.802+0000 7f53dffff640 1 --2- 172.21.15.28:0/3191797540 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db11c0b9e0 0x55db11c0de00 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4762501a716a06fb server_cookie=6cc4867b81a1ca03 in_seq=0 out_seq=0 2024-04-25T01:39:45.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.802+0000 7f53bffff640 10 client.17859 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:45.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.802+0000 7f53bffff640 1 -- 172.21.15.28:0/3191797540 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55db11af5f40 con 0x55db11c0b9e0 2024-04-25T01:39:45.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.802+0000 7f53bffff640 10 client.17859 handle_command_reply: tid=0 2024-04-25T01:39:45.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.802+0000 7f53e9a74740 1 client.17859 shutdown 2024-04-25T01:39:45.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.802+0000 7f53e9a74740 1 -- 172.21.15.28:0/3191797540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89630 msgr2=0x55db11b983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:45.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.802+0000 7f53e9a74740 1 --2- 172.21.15.28:0/3191797540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89630 0x55db11b983c0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f53d0003ca0 tx=0x7f53d0003cd0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.802+0000 7f53e9a74740 1 -- 172.21.15.28:0/3191797540 shutdown_connections 2024-04-25T01:39:45.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.802+0000 7f53e9a74740 1 --2- 172.21.15.28:0/3191797540 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db11c0b9e0 0x55db11c0de00 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.802+0000 7f53e9a74740 1 --2- 172.21.15.28:0/3191797540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db11b94620 0x55db11b98e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.802+0000 7f53e9a74740 1 --2- 172.21.15.28:0/3191797540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db11b89f50 0x55db11b98900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.802+0000 7f53e9a74740 1 --2- 172.21.15.28:0/3191797540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db11b89630 0x55db11b983c0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.802+0000 7f53bffff640 0 client.17859 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:45.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.802+0000 7f53e9a74740 1 -- 172.21.15.28:0/3191797540 >> 172.21.15.28:0/3191797540 conn(0x55db11b85950 msgr2=0x55db11b91e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:45.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.802+0000 7f53e9a74740 1 -- 172.21.15.28:0/3191797540 shutdown_connections 2024-04-25T01:39:45.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.802+0000 7f53e9a74740 1 -- 172.21.15.28:0/3191797540 wait complete. 2024-04-25T01:39:45.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.802+0000 7f53e9a74740 20 client.17859 trim_cache size 0 max 16384 2024-04-25T01:39:45.791 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:39:45.791 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:39:45.791 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:39:45.791 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:39:45.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.803+0000 7f53e6d63640 1 -- 172.21.15.28:0/4246604906 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f53c80a2d80 msgr2=0x7f53c80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:45.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.803+0000 7f53e6d63640 1 --2- 172.21.15.28:0/4246604906 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f53c80a2d80 0x7f53c80a5240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f53d8006010 tx=0x7f53d8005d80 comp rx=0 tx=0).stop 2024-04-25T01:39:45.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.803+0000 7f53e6d63640 1 -- 172.21.15.28:0/4246604906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e01032e0 msgr2=0x7f53e0198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:45.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.803+0000 7f53e6d63640 1 --2- 172.21.15.28:0/4246604906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e01032e0 0x7f53e0198510 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f53d400d950 tx=0x7f53d400de10 comp rx=0 tx=0).stop 2024-04-25T01:39:45.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.803+0000 7f53e6d63640 1 -- 172.21.15.28:0/4246604906 shutdown_connections 2024-04-25T01:39:45.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.803+0000 7f53e6d63640 1 --2- 172.21.15.28:0/4246604906 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f53c80a2d80 0x7f53c80a5240 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.803+0000 7f53e6d63640 1 --2- 172.21.15.28:0/4246604906 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f53e010bd00 0x7f53e019c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.803+0000 7f53e6d63640 1 --2- 172.21.15.28:0/4246604906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e01032e0 0x7f53e0198510 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.803+0000 7f53e6d63640 1 --2- 172.21.15.28:0/4246604906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53e01029c0 0x7f53e0197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.803+0000 7f53e6d63640 1 -- 172.21.15.28:0/4246604906 >> 172.21.15.28:0/4246604906 conn(0x7f53e0077740 msgr2=0x7f53e00fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:45.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.803+0000 7f53e6d63640 1 -- 172.21.15.28:0/4246604906 shutdown_connections 2024-04-25T01:39:45.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.803+0000 7f53e6d63640 1 -- 172.21.15.28:0/4246604906 wait complete. 2024-04-25T01:39:45.797 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:39:45.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:39:45.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.918+0000 7f0ffa529640 1 -- 172.21.15.28:0/1816440331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ff4100690 msgr2=0x7f0ff4100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:45.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.918+0000 7f0ffa529640 1 --2- 172.21.15.28:0/1816440331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ff4100690 0x7f0ff4100b10 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8009930 tx=0x7f0fe802a420 comp rx=0 tx=0).stop 2024-04-25T01:39:45.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.918+0000 7f0ffa529640 1 -- 172.21.15.28:0/1816440331 shutdown_connections 2024-04-25T01:39:45.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.918+0000 7f0ffa529640 1 --2- 172.21.15.28:0/1816440331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ff4101050 0x7f0ff410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.918+0000 7f0ffa529640 1 --2- 172.21.15.28:0/1816440331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ff4100690 0x7f0ff4100b10 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.918+0000 7f0ffa529640 1 --2- 172.21.15.28:0/1816440331 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ff4106650 0x7f0ff4106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.918+0000 7f0ffa529640 1 -- 172.21.15.28:0/1816440331 >> 172.21.15.28:0/1816440331 conn(0x7f0ff40777a0 msgr2=0x7f0ff4077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:45.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.918+0000 7f0ffa529640 1 -- 172.21.15.28:0/1816440331 shutdown_connections 2024-04-25T01:39:45.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.918+0000 7f0ffa529640 1 -- 172.21.15.28:0/1816440331 wait complete. 2024-04-25T01:39:45.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.918+0000 7f0ffa529640 1 Processor -- start 2024-04-25T01:39:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.918+0000 7f0ffa529640 1 -- start start 2024-04-25T01:39:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ffa529640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ff4100690 0x7f0ff4197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ffa529640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ff4101050 0x7f0ff4198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ffa529640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ff4106650 0x7f0ff419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ffa529640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff4110a60 con 0x7f0ff4106650 2024-04-25T01:39:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ffa529640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff41108e0 con 0x7f0ff4101050 2024-04-25T01:39:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ffa529640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff4110be0 con 0x7f0ff4100690 2024-04-25T01:39:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ff8ac6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ff4106650 0x7f0ff419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ff8ac6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ff4106650 0x7f0ff419c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51670/0 (socket says 172.21.15.28:51670) 2024-04-25T01:39:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ff8ac6640 1 -- 172.21.15.28:0/2237886606 learned_addr learned my addr 172.21.15.28:0/2237886606 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ff3fff640 1 --2- 172.21.15.28:0/2237886606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ff4100690 0x7f0ff4197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ff37fe640 1 --2- 172.21.15.28:0/2237886606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ff4101050 0x7f0ff4198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ff8ac6640 1 -- 172.21.15.28:0/2237886606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ff4100690 msgr2=0x7f0ff4197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ff8ac6640 1 --2- 172.21.15.28:0/2237886606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ff4100690 0x7f0ff4197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ff8ac6640 1 -- 172.21.15.28:0/2237886606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ff4101050 msgr2=0x7f0ff4198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ff8ac6640 1 --2- 172.21.15.28:0/2237886606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ff4101050 0x7f0ff4198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ff8ac6640 1 -- 172.21.15.28:0/2237886606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff419cf90 con 0x7f0ff4106650 2024-04-25T01:39:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ff37fe640 1 --2- 172.21.15.28:0/2237886606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ff4101050 0x7f0ff4198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ff8ac6640 1 --2- 172.21.15.28:0/2237886606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ff4106650 0x7f0ff419c860 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f0fec00cab0 tx=0x7f0fec00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ff17fa640 1 -- 172.21.15.28:0/2237886606 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fec00ae20 con 0x7f0ff4106650 2024-04-25T01:39:45.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ff17fa640 1 -- 172.21.15.28:0/2237886606 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fec00f930 con 0x7f0ff4106650 2024-04-25T01:39:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ffa529640 1 -- 172.21.15.28:0/2237886606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ff419d280 con 0x7f0ff4106650 2024-04-25T01:39:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.919+0000 7f0ff17fa640 1 -- 172.21.15.28:0/2237886606 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fec018b00 con 0x7f0ff4106650 2024-04-25T01:39:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.920+0000 7f0ffa529640 1 -- 172.21.15.28:0/2237886606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ff41a4b10 con 0x7f0ff4106650 2024-04-25T01:39:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.921+0000 7f0ffa529640 1 -- 172.21.15.28:0/2237886606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ff40689b0 con 0x7f0ff4106650 2024-04-25T01:39:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.922+0000 7f0ff17fa640 1 -- 172.21.15.28:0/2237886606 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0fec00fae0 con 0x7f0ff4106650 2024-04-25T01:39:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.922+0000 7f0ff17fa640 1 --2- 172.21.15.28:0/2237886606 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0fdc0a2d00 0x7f0fdc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.922+0000 7f0ff17fa640 1 -- 172.21.15.28:0/2237886606 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f0fec0cfb40 con 0x7f0ff4106650 2024-04-25T01:39:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.922+0000 7f0ff3fff640 1 --2- 172.21.15.28:0/2237886606 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0fdc0a2d00 0x7f0fdc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.923+0000 7f0ff3fff640 1 --2- 172.21.15.28:0/2237886606 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0fdc0a2d00 0x7f0fdc0a51c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4009ae0 tx=0x7f0fe4005f90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:45.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:45.927+0000 7f0ff17fa640 1 -- 172.21.15.28:0/2237886606 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0fec014030 con 0x7f0ff4106650 2024-04-25T01:39:46.025 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:46 smithi028 ceph-mon[29390]: pgmap v205: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:39:46.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.153+0000 7f0ffa529640 1 -- 172.21.15.28:0/2237886606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ff41a4d70 con 0x7f0ff4106650 2024-04-25T01:39:46.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.153+0000 7f0ff17fa640 1 -- 172.21.15.28:0/2237886606 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7f0fec09c420 con 0x7f0ff4106650 2024-04-25T01:39:46.137 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:39:46.137 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:39:46.137 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:39:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.155+0000 7f0ffa529640 1 -- 172.21.15.28:0/2237886606 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0fdc0a2d00 msgr2=0x7f0fdc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.155+0000 7f0ffa529640 1 --2- 172.21.15.28:0/2237886606 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0fdc0a2d00 0x7f0fdc0a51c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4009ae0 tx=0x7f0fe4005f90 comp rx=0 tx=0).stop 2024-04-25T01:39:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.155+0000 7f0ffa529640 1 -- 172.21.15.28:0/2237886606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ff4106650 msgr2=0x7f0ff419c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.155+0000 7f0ffa529640 1 --2- 172.21.15.28:0/2237886606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ff4106650 0x7f0ff419c860 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f0fec00cab0 tx=0x7f0fec00cf70 comp rx=0 tx=0).stop 2024-04-25T01:39:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.155+0000 7f0ffa529640 1 -- 172.21.15.28:0/2237886606 shutdown_connections 2024-04-25T01:39:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.155+0000 7f0ffa529640 1 --2- 172.21.15.28:0/2237886606 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0fdc0a2d00 0x7f0fdc0a51c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.155+0000 7f0ffa529640 1 --2- 172.21.15.28:0/2237886606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ff4106650 0x7f0ff419c860 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.155+0000 7f0ffa529640 1 --2- 172.21.15.28:0/2237886606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ff4101050 0x7f0ff4198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.155+0000 7f0ffa529640 1 --2- 172.21.15.28:0/2237886606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0ff4100690 0x7f0ff4197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.156+0000 7f0ffa529640 1 -- 172.21.15.28:0/2237886606 >> 172.21.15.28:0/2237886606 conn(0x7f0ff40777a0 msgr2=0x7f0ff40ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.156+0000 7f0ffa529640 1 -- 172.21.15.28:0/2237886606 shutdown_connections 2024-04-25T01:39:46.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.156+0000 7f0ffa529640 1 -- 172.21.15.28:0/2237886606 wait complete. 2024-04-25T01:39:46.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:39:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.271+0000 7fc60c163640 1 -- 172.21.15.28:0/1687104674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc604074410 msgr2=0x7fc60410da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.271+0000 7fc60c163640 1 --2- 172.21.15.28:0/1687104674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc604074410 0x7fc60410da90 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4009900 tx=0x7fc5f402a480 comp rx=0 tx=0).stop 2024-04-25T01:39:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.271+0000 7fc60c163640 1 -- 172.21.15.28:0/1687104674 shutdown_connections 2024-04-25T01:39:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.271+0000 7fc60c163640 1 --2- 172.21.15.28:0/1687104674 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc60410dfd0 0x7fc6041103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.271+0000 7fc60c163640 1 --2- 172.21.15.28:0/1687104674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc604074410 0x7fc60410da90 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.271+0000 7fc60c163640 1 --2- 172.21.15.28:0/1687104674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc604073af0 0x7fc604073ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.271+0000 7fc60c163640 1 -- 172.21.15.28:0/1687104674 >> 172.21.15.28:0/1687104674 conn(0x7fc6040fd180 msgr2=0x7fc6040ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.271+0000 7fc60c163640 1 -- 172.21.15.28:0/1687104674 shutdown_connections 2024-04-25T01:39:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.271+0000 7fc60c163640 1 -- 172.21.15.28:0/1687104674 wait complete. 2024-04-25T01:39:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc60c163640 1 Processor -- start 2024-04-25T01:39:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc60c163640 1 -- start start 2024-04-25T01:39:46.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc60c163640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc604073af0 0x7fc60419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc60c163640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc604074410 0x7fc60419a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc60c163640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc60410dfd0 0x7fc60419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc60c163640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc604113d30 con 0x7fc60410dfd0 2024-04-25T01:39:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc60c163640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc604113bb0 con 0x7fc604073af0 2024-04-25T01:39:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc60c163640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc604113eb0 con 0x7fc604074410 2024-04-25T01:39:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc60a700640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc60410dfd0 0x7fc60419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc60a700640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc60410dfd0 0x7fc60419eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51678/0 (socket says 172.21.15.28:51678) 2024-04-25T01:39:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc60a700640 1 -- 172.21.15.28:0/945830536 learned_addr learned my addr 172.21.15.28:0/945830536 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc6096fe640 1 --2- 172.21.15.28:0/945830536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc604074410 0x7fc60419a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc609eff640 1 --2- 172.21.15.28:0/945830536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc604073af0 0x7fc60419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc60a700640 1 -- 172.21.15.28:0/945830536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc604074410 msgr2=0x7fc60419a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc60a700640 1 --2- 172.21.15.28:0/945830536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc604074410 0x7fc60419a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc60a700640 1 -- 172.21.15.28:0/945830536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc604073af0 msgr2=0x7fc60419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc60a700640 1 --2- 172.21.15.28:0/945830536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc604073af0 0x7fc60419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc60a700640 1 -- 172.21.15.28:0/945830536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc60419f290 con 0x7fc60410dfd0 2024-04-25T01:39:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc6096fe640 1 --2- 172.21.15.28:0/945830536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc604074410 0x7fc60419a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc609eff640 1 --2- 172.21.15.28:0/945830536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc604073af0 0x7fc60419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.272+0000 7fc60a700640 1 --2- 172.21.15.28:0/945830536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc60410dfd0 0x7fc60419eb60 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc00b970 tx=0x7fc5fc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.273+0000 7fc5faffd640 1 -- 172.21.15.28:0/945830536 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc019070 con 0x7fc60410dfd0 2024-04-25T01:39:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.273+0000 7fc5faffd640 1 -- 172.21.15.28:0/945830536 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5fc00cb40 con 0x7fc60410dfd0 2024-04-25T01:39:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.273+0000 7fc60c163640 1 -- 172.21.15.28:0/945830536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc60419f580 con 0x7fc60410dfd0 2024-04-25T01:39:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.273+0000 7fc5faffd640 1 -- 172.21.15.28:0/945830536 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc007650 con 0x7fc60410dfd0 2024-04-25T01:39:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.273+0000 7fc60c163640 1 -- 172.21.15.28:0/945830536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6041a6e10 con 0x7fc60410dfd0 2024-04-25T01:39:46.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.275+0000 7fc5faffd640 1 -- 172.21.15.28:0/945830536 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc5fc01c070 con 0x7fc60410dfd0 2024-04-25T01:39:46.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.275+0000 7fc60ee6a740 1 Processor -- start 2024-04-25T01:39:46.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.276+0000 7fc60ee6a740 1 -- start start 2024-04-25T01:39:46.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.276+0000 7fc60ee6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951c580 0x559e6951c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.276+0000 7fc60ee6a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e6951cea0 0x559e69527030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.276+0000 7fc60ee6a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559e69527570 0x559e69529960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.276+0000 7fc60ee6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559e69481670 con 0x559e6951c580 2024-04-25T01:39:46.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.276+0000 7fc60ee6a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559e69429970 con 0x559e6951cea0 2024-04-25T01:39:46.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.276+0000 7fc60ee6a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x559e6947d630 con 0x559e69527570 2024-04-25T01:39:46.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.276+0000 7fc6096fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951c580 0x559e6951c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.276+0000 7fc6096fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951c580 0x559e6951c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51692/0 (socket says 172.21.15.28:51692) 2024-04-25T01:39:46.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.276+0000 7fc6096fe640 1 -- 172.21.15.28:0/2098233054 learned_addr learned my addr 172.21.15.28:0/2098233054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.276+0000 7fc5faffd640 1 --2- 172.21.15.28:0/945830536 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc5f00a2d00 0x7fc5f00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.276+0000 7fc609eff640 1 --2- 172.21.15.28:0/2098233054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e6951cea0 0x559e69527030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.276+0000 7fc60a700640 1 --2- 172.21.15.28:0/945830536 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc5f00a2d00 0x7fc5f00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.276+0000 7fc6096fe640 1 -- 172.21.15.28:0/2098233054 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559e69527570 msgr2=0x559e69529960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.276+0000 7fc6096fe640 1 --2- 172.21.15.28:0/2098233054 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559e69527570 0x559e69529960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.276+0000 7fc6096fe640 1 -- 172.21.15.28:0/2098233054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e6951cea0 msgr2=0x559e69527030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.276+0000 7fc6096fe640 1 --2- 172.21.15.28:0/2098233054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e6951cea0 0x559e69527030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.276+0000 7fc6096fe640 1 -- 172.21.15.28:0/2098233054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e6944d400 con 0x559e6951c580 2024-04-25T01:39:46.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.276+0000 7fc5faffd640 1 -- 172.21.15.28:0/945830536 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc5fc0ced20 con 0x7fc60410dfd0 2024-04-25T01:39:46.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.277+0000 7fc6096fe640 1 --2- 172.21.15.28:0/2098233054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951c580 0x559e6951c960 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fc5f402a9a0 tx=0x7fc5f40316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=662d16769b88d2ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.277+0000 7fc60a700640 1 --2- 172.21.15.28:0/945830536 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc5f00a2d00 0x7fc5f00a51c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc09af40 tx=0x7fc5fc075040 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.277+0000 7fc5effff640 1 -- 172.21.15.28:0/2098233054 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4035050 con 0x559e6951c580 2024-04-25T01:39:46.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.277+0000 7fc5effff640 1 -- 172.21.15.28:0/2098233054 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5f4005020 con 0x559e6951c580 2024-04-25T01:39:46.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.277+0000 7fc5effff640 1 -- 172.21.15.28:0/2098233054 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4004ae0 con 0x559e6951c580 2024-04-25T01:39:46.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.277+0000 7fc60ee6a740 1 -- 172.21.15.28:0/2098233054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951c580 msgr2=0x559e6951c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.277+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/2098233054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951c580 0x559e6951c960 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fc5f402a9a0 tx=0x7fc5f40316f0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.278+0000 7fc60ee6a740 1 -- 172.21.15.28:0/2098233054 shutdown_connections 2024-04-25T01:39:46.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.278+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/2098233054 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559e69527570 0x559e69529960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.278+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/2098233054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e6951cea0 0x559e69527030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.278+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/2098233054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951c580 0x559e6951c960 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.278+0000 7fc60ee6a740 1 -- 172.21.15.28:0/2098233054 >> 172.21.15.28:0/2098233054 conn(0x559e695188a0 msgr2=0x559e69518c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.278+0000 7fc60ee6a740 1 -- 172.21.15.28:0/2098233054 shutdown_connections 2024-04-25T01:39:46.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.278+0000 7fc60ee6a740 1 -- 172.21.15.28:0/2098233054 wait complete. 2024-04-25T01:39:46.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.278+0000 7fc60ee6a740 1 -- start start 2024-04-25T01:39:46.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.278+0000 7fc60ee6a740 1 Processor -- start 2024-04-25T01:39:46.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.278+0000 7fc60ee6a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559e6951c580 0x559e69526ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.278+0000 7fc60ee6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951cea0 0x559e695241d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.278+0000 7fc60ee6a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e69527570 0x559e69524730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.278+0000 7fc60ee6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559e6952a300 con 0x559e6951cea0 2024-04-25T01:39:46.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.278+0000 7fc60ee6a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559e6943d050 con 0x559e69527570 2024-04-25T01:39:46.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.278+0000 7fc60ee6a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x559e6952a480 con 0x559e6951c580 2024-04-25T01:39:46.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.278+0000 7fc60ee6a740 10 client.0 _ll_register_callbacks cb 0x559e69194be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:46.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.279+0000 7fc609eff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951cea0 0x559e695241d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.279+0000 7fc609eff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951cea0 0x559e695241d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51708/0 (socket says 172.21.15.28:51708) 2024-04-25T01:39:46.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.279+0000 7fc609eff640 1 -- 172.21.15.28:0/3165087509 learned_addr learned my addr 172.21.15.28:0/3165087509 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.279+0000 7fc6096fe640 1 --2- 172.21.15.28:0/3165087509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e69527570 0x559e69524730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.279+0000 7fc6096fe640 1 --2- 172.21.15.28:0/3165087509 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559e6951c580 0x559e69526ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.279+0000 7fc609eff640 1 -- 172.21.15.28:0/3165087509 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559e6951c580 msgr2=0x559e69526ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.279+0000 7fc609eff640 1 --2- 172.21.15.28:0/3165087509 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559e6951c580 0x559e69526ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.279+0000 7fc609eff640 1 -- 172.21.15.28:0/3165087509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e69527570 msgr2=0x559e69524730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.279+0000 7fc609eff640 1 --2- 172.21.15.28:0/3165087509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e69527570 0x559e69524730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.279+0000 7fc609eff640 1 -- 172.21.15.28:0/3165087509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e6940edc0 con 0x559e6951cea0 2024-04-25T01:39:46.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.279+0000 7fc609eff640 1 --2- 172.21.15.28:0/3165087509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951cea0 0x559e695241d0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fc60000f740 tx=0x7fc60000fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf9aa15a23471774 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.279+0000 7fc5ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:46.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.279+0000 7fc5ed7fa640 1 -- 172.21.15.28:0/3165087509 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc600016070 con 0x559e6951cea0 2024-04-25T01:39:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.279+0000 7fc5ed7fa640 1 -- 172.21.15.28:0/3165087509 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc60000bd10 con 0x559e6951cea0 2024-04-25T01:39:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.279+0000 7fc60ee6a740 1 -- 172.21.15.28:0/3165087509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5f4005020 con 0x559e6951cea0 2024-04-25T01:39:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.279+0000 7fc5ed7fa640 1 -- 172.21.15.28:0/3165087509 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc60000fe50 con 0x559e6951cea0 2024-04-25T01:39:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.279+0000 7fc60ee6a740 10 client.17877 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.279+0000 7fc60ee6a740 1 -- 172.21.15.28:0/3165087509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559e691953f0 con 0x559e6951cea0 2024-04-25T01:39:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.280+0000 7fc5ed7fa640 1 -- 172.21.15.28:0/3165087509 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5f4005020 con 0x559e6951cea0 2024-04-25T01:39:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.280+0000 7fc5ed7fa640 1 -- 172.21.15.28:0/3165087509 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc60001a5d0 con 0x559e6951cea0 2024-04-25T01:39:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.280+0000 7fc60ee6a740 10 client.17877 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.280+0000 7fc60ee6a740 10 client.17877 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:39:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.280+0000 7fc60ee6a740 20 client.17877 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.280+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/3165087509 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x559e695375f0 0x559e69539a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.280+0000 7fc60ee6a740 4 client.17877 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.280+0000 7fc60ee6a740 1 -- 172.21.15.28:0/3165087509 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e6948ebb0 con 0x559e695375f0 2024-04-25T01:39:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.280+0000 7fc6096fe640 1 --2- 172.21.15.28:0/3165087509 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x559e695375f0 0x559e69539a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.281+0000 7fc5ed7fa640 10 client.17877 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:39:46.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.281+0000 7fc6096fe640 1 --2- 172.21.15.28:0/3165087509 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x559e695375f0 0x559e69539a10 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1191ee3341138910 server_cookie=cf85f3a8886b0d90 in_seq=0 out_seq=0 2024-04-25T01:39:46.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.281+0000 7fc5ed7fa640 1 -- 172.21.15.28:0/3165087509 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559e6948ebb0 con 0x559e695375f0 2024-04-25T01:39:46.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.281+0000 7fc5ed7fa640 10 client.17877 handle_command_reply: tid=0 2024-04-25T01:39:46.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.281+0000 7fc60ee6a740 1 client.17877 shutdown 2024-04-25T01:39:46.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.281+0000 7fc60ee6a740 1 -- 172.21.15.28:0/3165087509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951cea0 msgr2=0x559e695241d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.281+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/3165087509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951cea0 0x559e695241d0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fc60000f740 tx=0x7fc60000fc00 comp rx=0 tx=0).stop 2024-04-25T01:39:46.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.281+0000 7fc60ee6a740 1 -- 172.21.15.28:0/3165087509 shutdown_connections 2024-04-25T01:39:46.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.281+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/3165087509 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x559e695375f0 0x559e69539a10 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.281+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/3165087509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e69527570 0x559e69524730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.281+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/3165087509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951cea0 0x559e695241d0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.281+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/3165087509 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559e6951c580 0x559e69526ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.281+0000 7fc5ed7fa640 0 client.17877 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:39:46.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.281+0000 7fc60ee6a740 1 -- 172.21.15.28:0/3165087509 >> 172.21.15.28:0/3165087509 conn(0x559e695188a0 msgr2=0x559e69517b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.282+0000 7fc60ee6a740 1 -- 172.21.15.28:0/3165087509 shutdown_connections 2024-04-25T01:39:46.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.282+0000 7fc60ee6a740 1 -- 172.21.15.28:0/3165087509 wait complete. 2024-04-25T01:39:46.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.282+0000 7fc60ee6a740 20 client.17877 trim_cache size 0 max 16384 2024-04-25T01:39:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.295+0000 7fc60ee6a740 1 Processor -- start 2024-04-25T01:39:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.295+0000 7fc60ee6a740 1 -- start start 2024-04-25T01:39:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.295+0000 7fc60ee6a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e6951c580 0x559e695331f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.295+0000 7fc60ee6a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559e6951cea0 0x559e6951efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.295+0000 7fc60ee6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e69527570 0x559e69526a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.295+0000 7fc60ee6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6041b0cf0 con 0x559e69527570 2024-04-25T01:39:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.295+0000 7fc60ee6a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc6041b2830 con 0x559e6951c580 2024-04-25T01:39:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.295+0000 7fc60ee6a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc6041b13c0 con 0x559e6951cea0 2024-04-25T01:39:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.295+0000 7fc6096fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e69527570 0x559e69526a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.295+0000 7fc6096fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e69527570 0x559e69526a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51714/0 (socket says 172.21.15.28:51714) 2024-04-25T01:39:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.295+0000 7fc6096fe640 1 -- 172.21.15.28:0/2031855115 learned_addr learned my addr 172.21.15.28:0/2031855115 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.295+0000 7fc609eff640 1 --2- 172.21.15.28:0/2031855115 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559e6951cea0 0x559e6951efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.295+0000 7fc6096fe640 1 --2- 172.21.15.28:0/2031855115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e6951c580 0x559e695331f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.296+0000 7fc6096fe640 1 -- 172.21.15.28:0/2031855115 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559e6951cea0 msgr2=0x559e6951efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.296+0000 7fc6096fe640 1 --2- 172.21.15.28:0/2031855115 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559e6951cea0 0x559e6951efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.296+0000 7fc6096fe640 1 -- 172.21.15.28:0/2031855115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e6951c580 msgr2=0x559e695331f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.296+0000 7fc6096fe640 1 --2- 172.21.15.28:0/2031855115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e6951c580 0x559e695331f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.296+0000 7fc6096fe640 1 -- 172.21.15.28:0/2031855115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e6925f7d0 con 0x559e69527570 2024-04-25T01:39:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.296+0000 7fc6096fe640 1 --2- 172.21.15.28:0/2031855115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e69527570 0x559e69526a10 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4031b50 tx=0x7fc5f40047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2659a4a7752bf98e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.296+0000 7fc5ef7fe640 1 -- 172.21.15.28:0/2031855115 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4059070 con 0x559e69527570 2024-04-25T01:39:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.296+0000 7fc5ef7fe640 1 -- 172.21.15.28:0/2031855115 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5f40542d0 con 0x559e69527570 2024-04-25T01:39:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.296+0000 7fc5ef7fe640 1 -- 172.21.15.28:0/2031855115 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4051040 con 0x559e69527570 2024-04-25T01:39:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.296+0000 7fc60ee6a740 1 -- 172.21.15.28:0/2031855115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e69527570 msgr2=0x559e69526a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.296+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/2031855115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e69527570 0x559e69526a10 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4031b50 tx=0x7fc5f40047a0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.296+0000 7fc60ee6a740 1 -- 172.21.15.28:0/2031855115 shutdown_connections 2024-04-25T01:39:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.296+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/2031855115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e69527570 0x559e69526a10 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.296+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/2031855115 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559e6951cea0 0x559e6951efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.296+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/2031855115 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e6951c580 0x559e695331f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.296+0000 7fc60ee6a740 1 -- 172.21.15.28:0/2031855115 >> 172.21.15.28:0/2031855115 conn(0x559e695188a0 msgr2=0x559e695242f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.297+0000 7fc60ee6a740 1 -- 172.21.15.28:0/2031855115 shutdown_connections 2024-04-25T01:39:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.297+0000 7fc60ee6a740 1 -- 172.21.15.28:0/2031855115 wait complete. 2024-04-25T01:39:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.297+0000 7fc60ee6a740 1 -- start start 2024-04-25T01:39:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.297+0000 7fc60ee6a740 1 Processor -- start 2024-04-25T01:39:46.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.297+0000 7fc60ee6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951c580 0x559e6952b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.297+0000 7fc60ee6a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e6951cea0 0x559e6952b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.297+0000 7fc60ee6a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559e69527570 0x559e6952bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.297+0000 7fc60ee6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6041b2160 con 0x559e6951c580 2024-04-25T01:39:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.297+0000 7fc60ee6a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc6041b1a90 con 0x559e6951cea0 2024-04-25T01:39:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.297+0000 7fc60ee6a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc6041aecd0 con 0x559e69527570 2024-04-25T01:39:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.297+0000 7fc60ee6a740 10 client.0 _ll_register_callbacks cb 0x7fc60000d380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.297+0000 7fc6096fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951c580 0x559e6952b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.297+0000 7fc6096fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951c580 0x559e6952b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51722/0 (socket says 172.21.15.28:51722) 2024-04-25T01:39:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.297+0000 7fc6096fe640 1 -- 172.21.15.28:0/2148196833 learned_addr learned my addr 172.21.15.28:0/2148196833 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.298+0000 7fc6096fe640 1 --2- 172.21.15.28:0/2148196833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559e69527570 0x559e6952bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.298+0000 7fc609eff640 1 --2- 172.21.15.28:0/2148196833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e6951cea0 0x559e6952b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.298+0000 7fc6096fe640 1 -- 172.21.15.28:0/2148196833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559e69527570 msgr2=0x559e6952bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.298+0000 7fc6096fe640 1 --2- 172.21.15.28:0/2148196833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559e69527570 0x559e6952bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.298+0000 7fc6096fe640 1 -- 172.21.15.28:0/2148196833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e6951cea0 msgr2=0x559e6952b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.298+0000 7fc6096fe640 1 --2- 172.21.15.28:0/2148196833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e6951cea0 0x559e6952b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.298+0000 7fc6096fe640 1 -- 172.21.15.28:0/2148196833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e6959cb80 con 0x559e6951c580 2024-04-25T01:39:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.298+0000 7fc6096fe640 1 --2- 172.21.15.28:0/2148196833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951c580 0x559e6952b1e0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4057040 tx=0x7fc5f4049040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de98ca6207be5ccb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.298+0000 7fc5ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.298+0000 7fc5ed7fa640 1 -- 172.21.15.28:0/2148196833 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4059020 con 0x559e6951c580 2024-04-25T01:39:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.298+0000 7fc5ed7fa640 1 -- 172.21.15.28:0/2148196833 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc5f4054740 con 0x559e6951c580 2024-04-25T01:39:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.298+0000 7fc60ee6a740 1 -- 172.21.15.28:0/2148196833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5f40542d0 con 0x559e6951c580 2024-04-25T01:39:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.298+0000 7fc5ed7fa640 1 -- 172.21.15.28:0/2148196833 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc5f4035040 con 0x559e6951c580 2024-04-25T01:39:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.298+0000 7fc60ee6a740 10 client.17883 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.298+0000 7fc60ee6a740 1 -- 172.21.15.28:0/2148196833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559e6959de60 con 0x559e6951c580 2024-04-25T01:39:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.298+0000 7fc5ed7fa640 1 -- 172.21.15.28:0/2148196833 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5f40542d0 con 0x559e6951c580 2024-04-25T01:39:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.299+0000 7fc5ed7fa640 1 -- 172.21.15.28:0/2148196833 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc5f405e060 con 0x559e6951c580 2024-04-25T01:39:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.299+0000 7fc60ee6a740 10 client.17883 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.299+0000 7fc60ee6a740 10 client.17883 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:39:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.299+0000 7fc60ee6a740 20 client.17883 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.299+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/2148196833 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x559e6959e090 0x559e695a0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.299+0000 7fc60ee6a740 4 client.17883 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.299+0000 7fc60ee6a740 1 -- 172.21.15.28:0/2148196833 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559e69488f00 con 0x559e6959e090 2024-04-25T01:39:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.299+0000 7fc609eff640 1 --2- 172.21.15.28:0/2148196833 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x559e6959e090 0x559e695a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.299+0000 7fc5ed7fa640 10 client.17883 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:39:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.299+0000 7fc609eff640 1 --2- 172.21.15.28:0/2148196833 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x559e6959e090 0x559e695a0480 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f6457df23f8d4e59 server_cookie=8618e08e7e1c29c9 in_seq=0 out_seq=0 2024-04-25T01:39:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.299+0000 7fc5ed7fa640 1 -- 172.21.15.28:0/2148196833 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559e69488f00 con 0x559e6959e090 2024-04-25T01:39:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.299+0000 7fc5ed7fa640 10 client.17883 handle_command_reply: tid=0 2024-04-25T01:39:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.299+0000 7fc60ee6a740 1 client.17883 shutdown 2024-04-25T01:39:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60ee6a740 1 -- 172.21.15.28:0/2148196833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951c580 msgr2=0x559e6952b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/2148196833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951c580 0x559e6952b1e0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4057040 tx=0x7fc5f4049040 comp rx=0 tx=0).stop 2024-04-25T01:39:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60ee6a740 1 -- 172.21.15.28:0/2148196833 shutdown_connections 2024-04-25T01:39:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/2148196833 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x559e6959e090 0x559e695a0480 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/2148196833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559e69527570 0x559e6952bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/2148196833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559e6951cea0 0x559e6952b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc5ed7fa640 0 client.17883 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:39:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60ee6a740 1 --2- 172.21.15.28:0/2148196833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559e6951c580 0x559e6952b1e0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60ee6a740 1 -- 172.21.15.28:0/2148196833 >> 172.21.15.28:0/2148196833 conn(0x559e695188a0 msgr2=0x559e695172b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60ee6a740 1 -- 172.21.15.28:0/2148196833 shutdown_connections 2024-04-25T01:39:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60ee6a740 1 -- 172.21.15.28:0/2148196833 wait complete. 2024-04-25T01:39:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60ee6a740 20 client.17883 trim_cache size 0 max 16384 2024-04-25T01:39:46.290 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60c163640 1 -- 172.21.15.28:0/945830536 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc5f00a2d00 msgr2=0x7fc5f00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60c163640 1 --2- 172.21.15.28:0/945830536 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc5f00a2d00 0x7fc5f00a51c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc09af40 tx=0x7fc5fc075040 comp rx=0 tx=0).stop 2024-04-25T01:39:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60c163640 1 -- 172.21.15.28:0/945830536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc60410dfd0 msgr2=0x7fc60419eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60c163640 1 --2- 172.21.15.28:0/945830536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc60410dfd0 0x7fc60419eb60 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc00b970 tx=0x7fc5fc00be30 comp rx=0 tx=0).stop 2024-04-25T01:39:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60c163640 1 -- 172.21.15.28:0/945830536 shutdown_connections 2024-04-25T01:39:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60c163640 1 --2- 172.21.15.28:0/945830536 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc5f00a2d00 0x7fc5f00a51c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60c163640 1 --2- 172.21.15.28:0/945830536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc60410dfd0 0x7fc60419eb60 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60c163640 1 --2- 172.21.15.28:0/945830536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc604074410 0x7fc60419a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60c163640 1 --2- 172.21.15.28:0/945830536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc604073af0 0x7fc60419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60c163640 1 -- 172.21.15.28:0/945830536 >> 172.21.15.28:0/945830536 conn(0x7fc6040fd180 msgr2=0x7fc6040fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60c163640 1 -- 172.21.15.28:0/945830536 shutdown_connections 2024-04-25T01:39:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.300+0000 7fc60c163640 1 -- 172.21.15.28:0/945830536 wait complete. 2024-04-25T01:39:46.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:39:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.418+0000 7f1abcd63640 1 -- 172.21.15.28:0/808752303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab8107720 msgr2=0x7f1ab8107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.418+0000 7f1abcd63640 1 --2- 172.21.15.28:0/808752303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab8107720 0x7f1ab8107b00 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8009930 tx=0x7f1aa802a420 comp rx=0 tx=0).stop 2024-04-25T01:39:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.418+0000 7f1abcd63640 1 -- 172.21.15.28:0/808752303 shutdown_connections 2024-04-25T01:39:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.418+0000 7f1abcd63640 1 --2- 172.21.15.28:0/808752303 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ab81020d0 0x7f1ab810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.418+0000 7f1abcd63640 1 --2- 172.21.15.28:0/808752303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ab8101710 0x7f1ab8101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.418+0000 7f1abcd63640 1 --2- 172.21.15.28:0/808752303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab8107720 0x7f1ab8107b00 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.418+0000 7f1abcd63640 1 -- 172.21.15.28:0/808752303 >> 172.21.15.28:0/808752303 conn(0x7f1ab8078b90 msgr2=0x7f1ab8078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.418+0000 7f1abcd63640 1 -- 172.21.15.28:0/808752303 shutdown_connections 2024-04-25T01:39:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.418+0000 7f1abcd63640 1 -- 172.21.15.28:0/808752303 wait complete. 2024-04-25T01:39:46.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.418+0000 7f1abcd63640 1 Processor -- start 2024-04-25T01:39:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.419+0000 7f1abcd63640 1 -- start start 2024-04-25T01:39:46.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.419+0000 7f1abcd63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab8101710 0x7f1ab8073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.419+0000 7f1abcd63640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ab81020d0 0x7f1ab8073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.419+0000 7f1abcd63640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ab8107720 0x7f1ab80744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.419+0000 7f1abcd63640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab8111b80 con 0x7f1ab8101710 2024-04-25T01:39:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.419+0000 7f1abcd63640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab8111a00 con 0x7f1ab8107720 2024-04-25T01:39:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.419+0000 7f1abcd63640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab8111d00 con 0x7f1ab81020d0 2024-04-25T01:39:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.419+0000 7f1ab659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab8101710 0x7f1ab8073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.419+0000 7f1ab659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab8101710 0x7f1ab8073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51754/0 (socket says 172.21.15.28:51754) 2024-04-25T01:39:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.419+0000 7f1ab659c640 1 -- 172.21.15.28:0/4189006611 learned_addr learned my addr 172.21.15.28:0/4189006611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.419+0000 7f1aaffff640 1 --2- 172.21.15.28:0/4189006611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ab81020d0 0x7f1ab8073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.419+0000 7f1ab659c640 1 -- 172.21.15.28:0/4189006611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ab81020d0 msgr2=0x7f1ab8073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.419+0000 7f1ab659c640 1 --2- 172.21.15.28:0/4189006611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ab81020d0 0x7f1ab8073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.419+0000 7f1ab659c640 1 -- 172.21.15.28:0/4189006611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ab8107720 msgr2=0x7f1ab80744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.419+0000 7f1ab659c640 1 --2- 172.21.15.28:0/4189006611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ab8107720 0x7f1ab80744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.419+0000 7f1ab659c640 1 -- 172.21.15.28:0/4189006611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab81a6540 con 0x7f1ab8101710 2024-04-25T01:39:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.419+0000 7f1ab659c640 1 --2- 172.21.15.28:0/4189006611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab8101710 0x7f1ab8073a40 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f1aa802a940 tx=0x7f1aa8003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.419+0000 7f1aaf7fe640 1 -- 172.21.15.28:0/4189006611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa803f070 con 0x7f1ab8101710 2024-04-25T01:39:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.420+0000 7f1aaf7fe640 1 -- 172.21.15.28:0/4189006611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1aa8031c90 con 0x7f1ab8101710 2024-04-25T01:39:46.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.420+0000 7f1aaf7fe640 1 -- 172.21.15.28:0/4189006611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8035030 con 0x7f1ab8101710 2024-04-25T01:39:46.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.420+0000 7f1abcd63640 1 -- 172.21.15.28:0/4189006611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ab81a67d0 con 0x7f1ab8101710 2024-04-25T01:39:46.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.420+0000 7f1abcd63640 1 -- 172.21.15.28:0/4189006611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ab81a6d10 con 0x7f1ab8101710 2024-04-25T01:39:46.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.422+0000 7f1aaf7fe640 1 -- 172.21.15.28:0/4189006611 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1aa8042070 con 0x7f1ab8101710 2024-04-25T01:39:46.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.422+0000 7f1abfa75740 1 Processor -- start 2024-04-25T01:39:46.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.422+0000 7f1abfa75740 1 -- start start 2024-04-25T01:39:46.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1abfa75740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09f580 0x55cbbd09f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1abfa75740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd09fea0 0x55cbbd0aa030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1abfa75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd0aa570 0x55cbbd0ac960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1abfa75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cbbd004670 con 0x55cbbd0aa570 2024-04-25T01:39:46.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1abfa75740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cbbcfac970 con 0x55cbbd09fea0 2024-04-25T01:39:46.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1abfa75740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55cbbd000630 con 0x55cbbd09f580 2024-04-25T01:39:46.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1aaffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd0aa570 0x55cbbd0ac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1aaffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd0aa570 0x55cbbd0ac960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51770/0 (socket says 172.21.15.28:51770) 2024-04-25T01:39:46.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1aaffff640 1 -- 172.21.15.28:0/1123249915 learned_addr learned my addr 172.21.15.28:0/1123249915 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1ab6d9d640 1 --2- 172.21.15.28:0/1123249915 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09f580 0x55cbbd09f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1ab659c640 1 --2- 172.21.15.28:0/1123249915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd09fea0 0x55cbbd0aa030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1aaf7fe640 1 --2- 172.21.15.28:0/4189006611 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1aa40a2db0 0x7f1aa40a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1aaffff640 1 -- 172.21.15.28:0/1123249915 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09f580 msgr2=0x55cbbd09f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1aaffff640 1 --2- 172.21.15.28:0/1123249915 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09f580 0x55cbbd09f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1aaffff640 1 -- 172.21.15.28:0/1123249915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd09fea0 msgr2=0x55cbbd0aa030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1aaffff640 1 --2- 172.21.15.28:0/1123249915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd09fea0 0x55cbbd0aa030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1aaffff640 1 -- 172.21.15.28:0/1123249915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbbcfd0400 con 0x55cbbd0aa570 2024-04-25T01:39:46.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1ab659c640 1 --2- 172.21.15.28:0/1123249915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd09fea0 0x55cbbd0aa030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1ab6d9d640 1 --2- 172.21.15.28:0/4189006611 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1aa40a2db0 0x7f1aa40a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1aaf7fe640 1 -- 172.21.15.28:0/4189006611 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1aa80eff60 con 0x7f1ab8101710 2024-04-25T01:39:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1aaffff640 1 --2- 172.21.15.28:0/1123249915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd0aa570 0x55cbbd0ac960 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f1aa000dc80 tx=0x7f1aa000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b28a887cffe62f76 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1aacff9640 1 -- 172.21.15.28:0/1123249915 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0015070 con 0x55cbbd0aa570 2024-04-25T01:39:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1aacff9640 1 -- 172.21.15.28:0/1123249915 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1aa001a940 con 0x55cbbd0aa570 2024-04-25T01:39:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.423+0000 7f1aacff9640 1 -- 172.21.15.28:0/1123249915 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0011a40 con 0x55cbbd0aa570 2024-04-25T01:39:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.424+0000 7f1abfa75740 1 -- 172.21.15.28:0/1123249915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd0aa570 msgr2=0x55cbbd0ac960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.424+0000 7f1abfa75740 1 --2- 172.21.15.28:0/1123249915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd0aa570 0x55cbbd0ac960 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f1aa000dc80 tx=0x7f1aa000f730 comp rx=0 tx=0).stop 2024-04-25T01:39:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.424+0000 7f1ab6d9d640 1 --2- 172.21.15.28:0/4189006611 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1aa40a2db0 0x7f1aa40a5270 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0005c40 tx=0x7f1ab0004a40 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.424+0000 7f1abfa75740 1 -- 172.21.15.28:0/1123249915 shutdown_connections 2024-04-25T01:39:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.424+0000 7f1abfa75740 1 --2- 172.21.15.28:0/1123249915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd0aa570 0x55cbbd0ac960 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.424+0000 7f1abfa75740 1 --2- 172.21.15.28:0/1123249915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd09fea0 0x55cbbd0aa030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.424+0000 7f1abfa75740 1 --2- 172.21.15.28:0/1123249915 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09f580 0x55cbbd09f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.424+0000 7f1abfa75740 1 -- 172.21.15.28:0/1123249915 >> 172.21.15.28:0/1123249915 conn(0x55cbbd09b8a0 msgr2=0x55cbbd09bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.424+0000 7f1abfa75740 1 -- 172.21.15.28:0/1123249915 shutdown_connections 2024-04-25T01:39:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.424+0000 7f1abfa75740 1 -- 172.21.15.28:0/1123249915 wait complete. 2024-04-25T01:39:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1abfa75740 1 -- start start 2024-04-25T01:39:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1abfa75740 1 Processor -- start 2024-04-25T01:39:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1abfa75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09f580 0x55cbbd0a9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1abfa75740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09fea0 0x55cbbd0a71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1abfa75740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd0aa570 0x55cbbd0a76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1abfa75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cbbd0ad360 con 0x55cbbd09f580 2024-04-25T01:39:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1abfa75740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cbbcfc0050 con 0x55cbbd0aa570 2024-04-25T01:39:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1abfa75740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55cbbd0ad4e0 con 0x55cbbd09fea0 2024-04-25T01:39:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1abfa75740 10 client.0 _ll_register_callbacks cb 0x55cbbcd17be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1ab659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09f580 0x55cbbd0a9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1ab659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09f580 0x55cbbd0a9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51772/0 (socket says 172.21.15.28:51772) 2024-04-25T01:39:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1ab659c640 1 -- 172.21.15.28:0/849301344 learned_addr learned my addr 172.21.15.28:0/849301344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1aaffff640 1 --2- 172.21.15.28:0/849301344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09fea0 0x55cbbd0a71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1ab6d9d640 1 --2- 172.21.15.28:0/849301344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd0aa570 0x55cbbd0a76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1ab659c640 1 -- 172.21.15.28:0/849301344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09fea0 msgr2=0x55cbbd0a71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1ab659c640 1 --2- 172.21.15.28:0/849301344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09fea0 0x55cbbd0a71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1ab659c640 1 -- 172.21.15.28:0/849301344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd0aa570 msgr2=0x55cbbd0a76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1ab659c640 1 --2- 172.21.15.28:0/849301344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd0aa570 0x55cbbd0a76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1ab659c640 1 -- 172.21.15.28:0/849301344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbbcf91dc0 con 0x55cbbd09f580 2024-04-25T01:39:46.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1ab6d9d640 1 --2- 172.21.15.28:0/849301344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd0aa570 0x55cbbd0a76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:46.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.425+0000 7f1ab659c640 1 --2- 172.21.15.28:0/849301344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09f580 0x55cbbd0a9eb0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80bbe40 tx=0x7f1aa803b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0952538bd552a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.426+0000 7f1a9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:46.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.426+0000 7f1a9dffb640 1 -- 172.21.15.28:0/849301344 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8049070 con 0x55cbbd09f580 2024-04-25T01:39:46.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.426+0000 7f1a9dffb640 1 -- 172.21.15.28:0/849301344 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1aa80b87b0 con 0x55cbbd09f580 2024-04-25T01:39:46.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.426+0000 7f1abfa75740 1 -- 172.21.15.28:0/849301344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1aa001a940 con 0x55cbbd09f580 2024-04-25T01:39:46.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.426+0000 7f1a9dffb640 1 -- 172.21.15.28:0/849301344 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1aa80bbaa0 con 0x55cbbd09f580 2024-04-25T01:39:46.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.426+0000 7f1abfa75740 10 client.17895 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:46.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.426+0000 7f1abfa75740 1 -- 172.21.15.28:0/849301344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cbbcd183f0 con 0x55cbbd09f580 2024-04-25T01:39:46.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.426+0000 7f1a9dffb640 1 -- 172.21.15.28:0/849301344 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1aa001a940 con 0x55cbbd09f580 2024-04-25T01:39:46.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.426+0000 7f1a9dffb640 1 -- 172.21.15.28:0/849301344 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1aa80c0140 con 0x55cbbd09f580 2024-04-25T01:39:46.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.426+0000 7f1abfa75740 10 client.17895 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:46.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.426+0000 7f1abfa75740 10 client.17895 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:39:46.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.426+0000 7f1abfa75740 20 client.17895 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:46.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.426+0000 7f1abfa75740 1 --2- 172.21.15.28:0/849301344 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cbbd0ba430 0x55cbbd0bc860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.426+0000 7f1abfa75740 4 client.17895 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:46.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.426+0000 7f1abfa75740 1 -- 172.21.15.28:0/849301344 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbbd011bb0 con 0x55cbbd0ba430 2024-04-25T01:39:46.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.427+0000 7f1aaffff640 1 --2- 172.21.15.28:0/849301344 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cbbd0ba430 0x55cbbd0bc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.427+0000 7f1aaffff640 1 --2- 172.21.15.28:0/849301344 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cbbd0ba430 0x55cbbd0bc860 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2e52a533ae6dce33 server_cookie=26e8ccb6600cf9d1 in_seq=0 out_seq=0 2024-04-25T01:39:46.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.427+0000 7f1a9dffb640 10 client.17895 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:39:46.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.428+0000 7f1a9dffb640 1 -- 172.21.15.28:0/849301344 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cbbd011bb0 con 0x55cbbd0ba430 2024-04-25T01:39:46.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.428+0000 7f1a9dffb640 10 client.17895 handle_command_reply: tid=0 2024-04-25T01:39:46.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.428+0000 7f1abfa75740 1 client.17895 shutdown 2024-04-25T01:39:46.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.428+0000 7f1abfa75740 1 -- 172.21.15.28:0/849301344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09f580 msgr2=0x55cbbd0a9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.428+0000 7f1abfa75740 1 --2- 172.21.15.28:0/849301344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09f580 0x55cbbd0a9eb0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80bbe40 tx=0x7f1aa803b460 comp rx=0 tx=0).stop 2024-04-25T01:39:46.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.428+0000 7f1abfa75740 1 -- 172.21.15.28:0/849301344 shutdown_connections 2024-04-25T01:39:46.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.428+0000 7f1abfa75740 1 --2- 172.21.15.28:0/849301344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd0aa570 0x55cbbd0a76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.428+0000 7f1abfa75740 1 --2- 172.21.15.28:0/849301344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09fea0 0x55cbbd0a71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.428+0000 7f1abfa75740 1 --2- 172.21.15.28:0/849301344 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cbbd0ba430 0x55cbbd0bc860 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.428+0000 7f1abfa75740 1 --2- 172.21.15.28:0/849301344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09f580 0x55cbbd0a9eb0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.428+0000 7f1a9dffb640 0 client.17895 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:39:46.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.428+0000 7f1abfa75740 1 -- 172.21.15.28:0/849301344 >> 172.21.15.28:0/849301344 conn(0x55cbbd09b8a0 msgr2=0x55cbbd09ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.428+0000 7f1abfa75740 1 -- 172.21.15.28:0/849301344 shutdown_connections 2024-04-25T01:39:46.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.428+0000 7f1abfa75740 1 -- 172.21.15.28:0/849301344 wait complete. 2024-04-25T01:39:46.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.428+0000 7f1abfa75740 20 client.17895 trim_cache size 0 max 16384 2024-04-25T01:39:46.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.441+0000 7f1abfa75740 1 Processor -- start 2024-04-25T01:39:46.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.441+0000 7f1abfa75740 1 -- start start 2024-04-25T01:39:46.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1abfa75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09b8a0 0x55cbbd0b6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1abfa75740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd09f580 0x55cbbd0a26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1abfa75740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09fea0 0x55cbbd0a99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1abfa75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab81b0ce0 con 0x55cbbd09b8a0 2024-04-25T01:39:46.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1abfa75740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab81b2820 con 0x55cbbd09f580 2024-04-25T01:39:46.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1abfa75740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab81b13b0 con 0x55cbbd09fea0 2024-04-25T01:39:46.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1ab659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09b8a0 0x55cbbd0b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1ab659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09b8a0 0x55cbbd0b6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51776/0 (socket says 172.21.15.28:51776) 2024-04-25T01:39:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1ab659c640 1 -- 172.21.15.28:0/252848020 learned_addr learned my addr 172.21.15.28:0/252848020 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1ab6d9d640 1 --2- 172.21.15.28:0/252848020 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09fea0 0x55cbbd0a99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1aaffff640 1 --2- 172.21.15.28:0/252848020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd09f580 0x55cbbd0a26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1ab659c640 1 -- 172.21.15.28:0/252848020 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09fea0 msgr2=0x55cbbd0a99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1ab659c640 1 --2- 172.21.15.28:0/252848020 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09fea0 0x55cbbd0a99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1ab659c640 1 -- 172.21.15.28:0/252848020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd09f580 msgr2=0x55cbbd0a26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1ab659c640 1 --2- 172.21.15.28:0/252848020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd09f580 0x55cbbd0a26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1ab659c640 1 -- 172.21.15.28:0/252848020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbbcde27d0 con 0x55cbbd09b8a0 2024-04-25T01:39:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1aaffff640 1 --2- 172.21.15.28:0/252848020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd09f580 0x55cbbd0a26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1ab659c640 1 --2- 172.21.15.28:0/252848020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09b8a0 0x55cbbd0b6220 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8006010 tx=0x7f1aa8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=302fd7c86940ae2f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1a9ffff640 1 -- 172.21.15.28:0/252848020 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8049070 con 0x55cbbd09b8a0 2024-04-25T01:39:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1a9ffff640 1 -- 172.21.15.28:0/252848020 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1aa80bc840 con 0x55cbbd09b8a0 2024-04-25T01:39:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.442+0000 7f1a9ffff640 1 -- 172.21.15.28:0/252848020 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa80b87b0 con 0x55cbbd09b8a0 2024-04-25T01:39:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.443+0000 7f1abfa75740 1 -- 172.21.15.28:0/252848020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09b8a0 msgr2=0x55cbbd0b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.443+0000 7f1abfa75740 1 --2- 172.21.15.28:0/252848020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09b8a0 0x55cbbd0b6220 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8006010 tx=0x7f1aa8000bc0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.443+0000 7f1abfa75740 1 -- 172.21.15.28:0/252848020 shutdown_connections 2024-04-25T01:39:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.443+0000 7f1abfa75740 1 --2- 172.21.15.28:0/252848020 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09fea0 0x55cbbd0a99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.443+0000 7f1abfa75740 1 --2- 172.21.15.28:0/252848020 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd09f580 0x55cbbd0a26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.443+0000 7f1abfa75740 1 --2- 172.21.15.28:0/252848020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09b8a0 0x55cbbd0b6220 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.443+0000 7f1abfa75740 1 -- 172.21.15.28:0/252848020 >> 172.21.15.28:0/252848020 conn(0x55cbbd0a7840 msgr2=0x55cbbd0a1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.443+0000 7f1abfa75740 1 -- 172.21.15.28:0/252848020 shutdown_connections 2024-04-25T01:39:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.443+0000 7f1abfa75740 1 -- 172.21.15.28:0/252848020 wait complete. 2024-04-25T01:39:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.443+0000 7f1abfa75740 1 -- start start 2024-04-25T01:39:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.443+0000 7f1abfa75740 1 Processor -- start 2024-04-25T01:39:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.443+0000 7f1abfa75740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd09b8a0 0x55cbbd0a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.443+0000 7f1abfa75740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09f580 0x55cbbd0a4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.443+0000 7f1abfa75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09fea0 0x55cbbd0a49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.443+0000 7f1abfa75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab81b2150 con 0x55cbbd09fea0 2024-04-25T01:39:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.443+0000 7f1abfa75740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab81b1a80 con 0x55cbbd09b8a0 2024-04-25T01:39:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.443+0000 7f1abfa75740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab81aecc0 con 0x55cbbd09f580 2024-04-25T01:39:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.443+0000 7f1abfa75740 10 client.0 _ll_register_callbacks cb 0x7f1aa8009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.444+0000 7f1ab6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09fea0 0x55cbbd0a49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.444+0000 7f1ab6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09fea0 0x55cbbd0a49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51784/0 (socket says 172.21.15.28:51784) 2024-04-25T01:39:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.444+0000 7f1ab6d9d640 1 -- 172.21.15.28:0/2233386639 learned_addr learned my addr 172.21.15.28:0/2233386639 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.444+0000 7f1aaffff640 1 --2- 172.21.15.28:0/2233386639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09f580 0x55cbbd0a4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.444+0000 7f1ab659c640 1 --2- 172.21.15.28:0/2233386639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd09b8a0 0x55cbbd0a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.444+0000 7f1ab6d9d640 1 -- 172.21.15.28:0/2233386639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09f580 msgr2=0x55cbbd0a4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.444+0000 7f1ab6d9d640 1 --2- 172.21.15.28:0/2233386639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09f580 0x55cbbd0a4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.444+0000 7f1ab6d9d640 1 -- 172.21.15.28:0/2233386639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd09b8a0 msgr2=0x55cbbd0a3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.444+0000 7f1ab6d9d640 1 --2- 172.21.15.28:0/2233386639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd09b8a0 0x55cbbd0a3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.444+0000 7f1ab6d9d640 1 -- 172.21.15.28:0/2233386639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbbd11f9c0 con 0x55cbbd09fea0 2024-04-25T01:39:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.444+0000 7f1ab6d9d640 1 --2- 172.21.15.28:0/2233386639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09fea0 0x55cbbd0a49d0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0020470 tx=0x7f1ab0005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f71fa2061f11b75b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.444+0000 7f1a9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.444+0000 7f1a9dffb640 1 -- 172.21.15.28:0/2233386639 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0011930 con 0x55cbbd09fea0 2024-04-25T01:39:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.444+0000 7f1a9dffb640 1 -- 172.21.15.28:0/2233386639 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1ab0011ad0 con 0x55cbbd09fea0 2024-04-25T01:39:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.444+0000 7f1abfa75740 1 -- 172.21.15.28:0/2233386639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1aa80bc840 con 0x55cbbd09fea0 2024-04-25T01:39:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.444+0000 7f1a9dffb640 1 -- 172.21.15.28:0/2233386639 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1ab0031050 con 0x55cbbd09fea0 2024-04-25T01:39:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.444+0000 7f1abfa75740 10 client.17901 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.444+0000 7f1abfa75740 1 -- 172.21.15.28:0/2233386639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cbbd120c70 con 0x55cbbd09fea0 2024-04-25T01:39:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.445+0000 7f1a9dffb640 1 -- 172.21.15.28:0/2233386639 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1aa80bc840 con 0x55cbbd09fea0 2024-04-25T01:39:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.445+0000 7f1a9dffb640 1 -- 172.21.15.28:0/2233386639 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1ab000f040 con 0x55cbbd09fea0 2024-04-25T01:39:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.445+0000 7f1abfa75740 10 client.17901 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.445+0000 7f1abfa75740 10 client.17901 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:39:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.445+0000 7f1abfa75740 20 client.17901 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.445+0000 7f1abfa75740 1 --2- 172.21.15.28:0/2233386639 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cbbd0aa570 0x55cbbd123480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.445+0000 7f1abfa75740 4 client.17901 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.445+0000 7f1abfa75740 1 -- 172.21.15.28:0/2233386639 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cbbd00bf00 con 0x55cbbd0aa570 2024-04-25T01:39:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.445+0000 7f1ab659c640 1 --2- 172.21.15.28:0/2233386639 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cbbd0aa570 0x55cbbd123480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.445+0000 7f1ab659c640 1 --2- 172.21.15.28:0/2233386639 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cbbd0aa570 0x55cbbd123480 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=126347239e1a0f46 server_cookie=4c508f93c8c35003 in_seq=0 out_seq=0 2024-04-25T01:39:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.445+0000 7f1a9dffb640 10 client.17901 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:39:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.445+0000 7f1a9dffb640 1 -- 172.21.15.28:0/2233386639 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cbbd00bf00 con 0x55cbbd0aa570 2024-04-25T01:39:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.445+0000 7f1a9dffb640 10 client.17901 handle_command_reply: tid=0 2024-04-25T01:39:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abfa75740 1 client.17901 shutdown 2024-04-25T01:39:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abfa75740 1 -- 172.21.15.28:0/2233386639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09fea0 msgr2=0x55cbbd0a49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abfa75740 1 --2- 172.21.15.28:0/2233386639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09fea0 0x55cbbd0a49d0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0020470 tx=0x7f1ab0005bd0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abfa75740 1 -- 172.21.15.28:0/2233386639 shutdown_connections 2024-04-25T01:39:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abfa75740 1 --2- 172.21.15.28:0/2233386639 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cbbd0aa570 0x55cbbd123480 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abfa75740 1 --2- 172.21.15.28:0/2233386639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbbd09fea0 0x55cbbd0a49d0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abfa75740 1 --2- 172.21.15.28:0/2233386639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cbbd09f580 0x55cbbd0a4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abfa75740 1 --2- 172.21.15.28:0/2233386639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cbbd09b8a0 0x55cbbd0a3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1a9dffb640 0 client.17901 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:39:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abfa75740 1 -- 172.21.15.28:0/2233386639 >> 172.21.15.28:0/2233386639 conn(0x55cbbd0a7840 msgr2=0x55cbbd0a1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abfa75740 1 -- 172.21.15.28:0/2233386639 shutdown_connections 2024-04-25T01:39:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abfa75740 1 -- 172.21.15.28:0/2233386639 wait complete. 2024-04-25T01:39:46.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abfa75740 20 client.17901 trim_cache size 0 max 16384 2024-04-25T01:39:46.436 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:46.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abcd63640 1 -- 172.21.15.28:0/4189006611 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1aa40a2db0 msgr2=0x7f1aa40a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abcd63640 1 --2- 172.21.15.28:0/4189006611 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1aa40a2db0 0x7f1aa40a5270 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0005c40 tx=0x7f1ab0004a40 comp rx=0 tx=0).stop 2024-04-25T01:39:46.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abcd63640 1 -- 172.21.15.28:0/4189006611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab8101710 msgr2=0x7f1ab8073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abcd63640 1 --2- 172.21.15.28:0/4189006611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab8101710 0x7f1ab8073a40 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f1aa802a940 tx=0x7f1aa8003b90 comp rx=0 tx=0).stop 2024-04-25T01:39:46.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abcd63640 1 -- 172.21.15.28:0/4189006611 shutdown_connections 2024-04-25T01:39:46.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abcd63640 1 --2- 172.21.15.28:0/4189006611 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1aa40a2db0 0x7f1aa40a5270 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abcd63640 1 --2- 172.21.15.28:0/4189006611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ab8107720 0x7f1ab80744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abcd63640 1 --2- 172.21.15.28:0/4189006611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1ab81020d0 0x7f1ab8073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abcd63640 1 --2- 172.21.15.28:0/4189006611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab8101710 0x7f1ab8073a40 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abcd63640 1 -- 172.21.15.28:0/4189006611 >> 172.21.15.28:0/4189006611 conn(0x7f1ab8078b90 msgr2=0x7f1ab80ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abcd63640 1 -- 172.21.15.28:0/4189006611 shutdown_connections 2024-04-25T01:39:46.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.446+0000 7f1abcd63640 1 -- 172.21.15.28:0/4189006611 wait complete. 2024-04-25T01:39:46.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:39:46.464 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:46 smithi130 ceph-mon[32702]: pgmap v205: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:39:46.489 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:46 smithi032 ceph-mon[32461]: pgmap v205: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:39:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.562+0000 7f373cc4d640 1 -- 172.21.15.28:0/2952429912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3738100fd0 msgr2=0x7f373810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.562+0000 7f373cc4d640 1 --2- 172.21.15.28:0/2952429912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3738100fd0 0x7f373810d4e0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f3730009cd0 tx=0x7f373002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.562+0000 7f373cc4d640 1 -- 172.21.15.28:0/2952429912 shutdown_connections 2024-04-25T01:39:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.562+0000 7f373cc4d640 1 --2- 172.21.15.28:0/2952429912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3738100fd0 0x7f373810d4e0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.562+0000 7f373cc4d640 1 --2- 172.21.15.28:0/2952429912 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3738100610 0x7f3738100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.562+0000 7f373cc4d640 1 --2- 172.21.15.28:0/2952429912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37381066b0 0x7f3738106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.562+0000 7f373cc4d640 1 -- 172.21.15.28:0/2952429912 >> 172.21.15.28:0/2952429912 conn(0x7f3738077740 msgr2=0x7f3738077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.562+0000 7f373cc4d640 1 -- 172.21.15.28:0/2952429912 shutdown_connections 2024-04-25T01:39:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.563+0000 7f373cc4d640 1 -- 172.21.15.28:0/2952429912 wait complete. 2024-04-25T01:39:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.563+0000 7f373cc4d640 1 Processor -- start 2024-04-25T01:39:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.563+0000 7f373cc4d640 1 -- start start 2024-04-25T01:39:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.564+0000 7f373cc4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3738100610 0x7f3738197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.564+0000 7f373cc4d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3738100fd0 0x7f37381984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.564+0000 7f373cc4d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37381066b0 0x7f373819c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.564+0000 7f373cc4d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3738110a40 con 0x7f3738100610 2024-04-25T01:39:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.564+0000 7f373cc4d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f37381108c0 con 0x7f3738100fd0 2024-04-25T01:39:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.564+0000 7f373cc4d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f3738110bc0 con 0x7f37381066b0 2024-04-25T01:39:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.564+0000 7f373659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3738100610 0x7f3738197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.564+0000 7f373659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3738100610 0x7f3738197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51806/0 (socket says 172.21.15.28:51806) 2024-04-25T01:39:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.564+0000 7f373659c640 1 -- 172.21.15.28:0/770861551 learned_addr learned my addr 172.21.15.28:0/770861551 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.564+0000 7f3736d9d640 1 --2- 172.21.15.28:0/770861551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37381066b0 0x7f373819c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.564+0000 7f372dd9b640 1 --2- 172.21.15.28:0/770861551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3738100fd0 0x7f37381984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.564+0000 7f373659c640 1 -- 172.21.15.28:0/770861551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37381066b0 msgr2=0x7f373819c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.564+0000 7f373659c640 1 --2- 172.21.15.28:0/770861551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37381066b0 0x7f373819c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.564+0000 7f373659c640 1 -- 172.21.15.28:0/770861551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3738100fd0 msgr2=0x7f37381984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.564+0000 7f373659c640 1 --2- 172.21.15.28:0/770861551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3738100fd0 0x7f37381984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.564+0000 7f373659c640 1 -- 172.21.15.28:0/770861551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f373819cf20 con 0x7f3738100610 2024-04-25T01:39:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.564+0000 7f373659c640 1 --2- 172.21.15.28:0/770861551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3738100610 0x7f3738197f80 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f372800c970 tx=0x7f372800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.564+0000 7f372ffff640 1 -- 172.21.15.28:0/770861551 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3728004220 con 0x7f3738100610 2024-04-25T01:39:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.565+0000 7f373cc4d640 1 -- 172.21.15.28:0/770861551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f373819d210 con 0x7f3738100610 2024-04-25T01:39:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.565+0000 7f372ffff640 1 -- 172.21.15.28:0/770861551 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37280043c0 con 0x7f3738100610 2024-04-25T01:39:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.565+0000 7f372ffff640 1 -- 172.21.15.28:0/770861551 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372800f450 con 0x7f3738100610 2024-04-25T01:39:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.565+0000 7f373cc4d640 1 -- 172.21.15.28:0/770861551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37381a4aa0 con 0x7f3738100610 2024-04-25T01:39:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f372ffff640 1 -- 172.21.15.28:0/770861551 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f37280079b0 con 0x7f3738100610 2024-04-25T01:39:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f373f5fd740 1 Processor -- start 2024-04-25T01:39:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f373f5fd740 1 -- start start 2024-04-25T01:39:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f373f5fd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a9664b580 0x561a9664b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f373f5fd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a9664bea0 0x561a96656030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f373f5fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a96656570 0x561a96658960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f373f5fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561a965b0670 con 0x561a96656570 2024-04-25T01:39:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f373f5fd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561a96558970 con 0x561a9664b580 2024-04-25T01:39:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f373f5fd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x561a965ac630 con 0x561a9664bea0 2024-04-25T01:39:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f372dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a96656570 0x561a96658960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f372dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a96656570 0x561a96658960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51822/0 (socket says 172.21.15.28:51822) 2024-04-25T01:39:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f372dd9b640 1 -- 172.21.15.28:0/2621202752 learned_addr learned my addr 172.21.15.28:0/2621202752 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f3736d9d640 1 --2- 172.21.15.28:0/2621202752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a9664b580 0x561a9664b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f373659c640 1 --2- 172.21.15.28:0/2621202752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a9664bea0 0x561a96656030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f372ffff640 1 --2- 172.21.15.28:0/770861551 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f37240a2d00 0x7f37240a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f372dd9b640 1 -- 172.21.15.28:0/2621202752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a9664bea0 msgr2=0x561a96656030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f372dd9b640 1 --2- 172.21.15.28:0/2621202752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a9664bea0 0x561a96656030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f372dd9b640 1 -- 172.21.15.28:0/2621202752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a9664b580 msgr2=0x561a9664b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f372dd9b640 1 --2- 172.21.15.28:0/2621202752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a9664b580 0x561a9664b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f372dd9b640 1 -- 172.21.15.28:0/2621202752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a9657c400 con 0x561a96656570 2024-04-25T01:39:46.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.568+0000 7f373659c640 1 --2- 172.21.15.28:0/2621202752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a9664bea0 0x561a96656030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:46.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.569+0000 7f3736d9d640 1 --2- 172.21.15.28:0/770861551 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f37240a2d00 0x7f37240a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.569+0000 7f372ffff640 1 -- 172.21.15.28:0/770861551 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f37280cf400 con 0x7f3738100610 2024-04-25T01:39:46.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.569+0000 7f372dd9b640 1 --2- 172.21.15.28:0/2621202752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a96656570 0x561a96658960 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f372000bce0 tx=0x7f372000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc920bd27e479516 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.569+0000 7f372cd99640 1 -- 172.21.15.28:0/2621202752 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3720015070 con 0x561a96656570 2024-04-25T01:39:46.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.569+0000 7f372cd99640 1 -- 172.21.15.28:0/2621202752 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f372001a940 con 0x561a96656570 2024-04-25T01:39:46.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.569+0000 7f372cd99640 1 -- 172.21.15.28:0/2621202752 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3720011a70 con 0x561a96656570 2024-04-25T01:39:46.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.570+0000 7f373f5fd740 1 -- 172.21.15.28:0/2621202752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a96656570 msgr2=0x561a96658960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.570+0000 7f3736d9d640 1 --2- 172.21.15.28:0/770861551 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f37240a2d00 0x7f37240a51c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3730005e30 tx=0x7f373003f040 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.570+0000 7f373f5fd740 1 --2- 172.21.15.28:0/2621202752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a96656570 0x561a96658960 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f372000bce0 tx=0x7f372000f730 comp rx=0 tx=0).stop 2024-04-25T01:39:46.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.570+0000 7f373f5fd740 1 -- 172.21.15.28:0/2621202752 shutdown_connections 2024-04-25T01:39:46.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.570+0000 7f373f5fd740 1 --2- 172.21.15.28:0/2621202752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a96656570 0x561a96658960 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.570+0000 7f373f5fd740 1 --2- 172.21.15.28:0/2621202752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a9664bea0 0x561a96656030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.570+0000 7f373f5fd740 1 --2- 172.21.15.28:0/2621202752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a9664b580 0x561a9664b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.570+0000 7f373f5fd740 1 -- 172.21.15.28:0/2621202752 >> 172.21.15.28:0/2621202752 conn(0x561a966478a0 msgr2=0x561a96647c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.570+0000 7f373f5fd740 1 -- 172.21.15.28:0/2621202752 shutdown_connections 2024-04-25T01:39:46.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.570+0000 7f373f5fd740 1 -- 172.21.15.28:0/2621202752 wait complete. 2024-04-25T01:39:46.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.570+0000 7f373f5fd740 1 -- start start 2024-04-25T01:39:46.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.570+0000 7f373f5fd740 1 Processor -- start 2024-04-25T01:39:46.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.570+0000 7f373f5fd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a9664b580 0x561a96655eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.570+0000 7f373f5fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664bea0 0x561a966531a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.570+0000 7f373f5fd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a96656570 0x561a966536e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.570+0000 7f373f5fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561a96659360 con 0x561a9664bea0 2024-04-25T01:39:46.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.570+0000 7f373f5fd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561a9656c050 con 0x561a9664b580 2024-04-25T01:39:46.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.570+0000 7f373f5fd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x561a966594e0 con 0x561a96656570 2024-04-25T01:39:46.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.571+0000 7f373f5fd740 10 client.0 _ll_register_callbacks cb 0x561a962c3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:46.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.571+0000 7f372dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664bea0 0x561a966531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.571+0000 7f372dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664bea0 0x561a966531a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51828/0 (socket says 172.21.15.28:51828) 2024-04-25T01:39:46.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.571+0000 7f372dd9b640 1 -- 172.21.15.28:0/1447899019 learned_addr learned my addr 172.21.15.28:0/1447899019 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.571+0000 7f373659c640 1 --2- 172.21.15.28:0/1447899019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a9664b580 0x561a96655eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.571+0000 7f372dd9b640 1 -- 172.21.15.28:0/1447899019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a96656570 msgr2=0x561a966536e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.571+0000 7f3736d9d640 1 --2- 172.21.15.28:0/1447899019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a96656570 0x561a966536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.571+0000 7f372dd9b640 1 --2- 172.21.15.28:0/1447899019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a96656570 0x561a966536e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.571+0000 7f372dd9b640 1 -- 172.21.15.28:0/1447899019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a9664b580 msgr2=0x561a96655eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.571+0000 7f372dd9b640 1 --2- 172.21.15.28:0/1447899019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a9664b580 0x561a96655eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.571+0000 7f372dd9b640 1 -- 172.21.15.28:0/1447899019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a9653ddc0 con 0x561a9664bea0 2024-04-25T01:39:46.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.571+0000 7f372dd9b640 1 --2- 172.21.15.28:0/1447899019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664bea0 0x561a966531a0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f3720011710 tx=0x7f372001a690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd4a1c14e402a0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.571+0000 7f371dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:46.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.571+0000 7f371dffb640 1 -- 172.21.15.28:0/1447899019 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3720015070 con 0x561a9664bea0 2024-04-25T01:39:46.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.571+0000 7f371dffb640 1 -- 172.21.15.28:0/1447899019 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f372000bdd0 con 0x561a9664bea0 2024-04-25T01:39:46.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.571+0000 7f373f5fd740 1 -- 172.21.15.28:0/1447899019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f372001a940 con 0x561a9664bea0 2024-04-25T01:39:46.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.571+0000 7f371dffb640 1 -- 172.21.15.28:0/1447899019 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f372001ad00 con 0x561a9664bea0 2024-04-25T01:39:46.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.571+0000 7f373f5fd740 10 client.17913 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:46.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.571+0000 7f373f5fd740 1 -- 172.21.15.28:0/1447899019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561a962c43f0 con 0x561a9664bea0 2024-04-25T01:39:46.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.572+0000 7f371dffb640 1 -- 172.21.15.28:0/1447899019 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f372001a940 con 0x561a9664bea0 2024-04-25T01:39:46.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.572+0000 7f371dffb640 1 -- 172.21.15.28:0/1447899019 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f37200117b0 con 0x561a9664bea0 2024-04-25T01:39:46.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.572+0000 7f373f5fd740 10 client.17913 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:46.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.572+0000 7f373f5fd740 10 client.17913 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:39:46.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.572+0000 7f373f5fd740 20 client.17913 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:46.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.572+0000 7f373f5fd740 1 --2- 172.21.15.28:0/1447899019 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x561a96666400 0x561a966687f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.572+0000 7f373f5fd740 4 client.17913 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:46.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.572+0000 7f373f5fd740 1 -- 172.21.15.28:0/1447899019 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a965bdbb0 con 0x561a96666400 2024-04-25T01:39:46.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.572+0000 7f373659c640 1 --2- 172.21.15.28:0/1447899019 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x561a96666400 0x561a966687f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.573+0000 7f371dffb640 10 client.17913 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:39:46.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.573+0000 7f373659c640 1 --2- 172.21.15.28:0/1447899019 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x561a96666400 0x561a966687f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=737df13a54da691f server_cookie=b10e884d9df9273 in_seq=0 out_seq=0 2024-04-25T01:39:46.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.573+0000 7f371dffb640 1 -- 172.21.15.28:0/1447899019 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561a965bdbb0 con 0x561a96666400 2024-04-25T01:39:46.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.573+0000 7f371dffb640 10 client.17913 handle_command_reply: tid=0 2024-04-25T01:39:46.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.573+0000 7f373f5fd740 1 client.17913 shutdown 2024-04-25T01:39:46.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.574+0000 7f373f5fd740 1 -- 172.21.15.28:0/1447899019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664bea0 msgr2=0x561a966531a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.574+0000 7f373f5fd740 1 --2- 172.21.15.28:0/1447899019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664bea0 0x561a966531a0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f3720011710 tx=0x7f372001a690 comp rx=0 tx=0).stop 2024-04-25T01:39:46.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.574+0000 7f373f5fd740 1 -- 172.21.15.28:0/1447899019 shutdown_connections 2024-04-25T01:39:46.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.574+0000 7f373f5fd740 1 --2- 172.21.15.28:0/1447899019 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x561a96666400 0x561a966687f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.574+0000 7f373f5fd740 1 --2- 172.21.15.28:0/1447899019 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a96656570 0x561a966536e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.574+0000 7f373f5fd740 1 --2- 172.21.15.28:0/1447899019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664bea0 0x561a966531a0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.574+0000 7f371dffb640 0 client.17913 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:39:46.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.574+0000 7f373f5fd740 1 --2- 172.21.15.28:0/1447899019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a9664b580 0x561a96655eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.574+0000 7f373f5fd740 1 -- 172.21.15.28:0/1447899019 >> 172.21.15.28:0/1447899019 conn(0x561a966478a0 msgr2=0x561a96646b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.574+0000 7f373f5fd740 1 -- 172.21.15.28:0/1447899019 shutdown_connections 2024-04-25T01:39:46.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.574+0000 7f373f5fd740 1 -- 172.21.15.28:0/1447899019 wait complete. 2024-04-25T01:39:46.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.574+0000 7f373f5fd740 20 client.17913 trim_cache size 0 max 16384 2024-04-25T01:39:46.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.587+0000 7f373f5fd740 1 Processor -- start 2024-04-25T01:39:46.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.587+0000 7f373f5fd740 1 -- start start 2024-04-25T01:39:46.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.587+0000 7f373f5fd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a966478a0 0x561a96662220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.587+0000 7f373f5fd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a9664b580 0x561a9666dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.587+0000 7f373f5fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664bea0 0x561a9666e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.587+0000 7f373f5fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f37381aeab0 con 0x561a9664bea0 2024-04-25T01:39:46.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.587+0000 7f373f5fd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f37381b05f0 con 0x561a966478a0 2024-04-25T01:39:46.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.587+0000 7f373f5fd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f37381af180 con 0x561a9664b580 2024-04-25T01:39:46.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.587+0000 7f3736d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664bea0 0x561a9666e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.587+0000 7f3736d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664bea0 0x561a9666e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51834/0 (socket says 172.21.15.28:51834) 2024-04-25T01:39:46.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.587+0000 7f3736d9d640 1 -- 172.21.15.28:0/2875234591 learned_addr learned my addr 172.21.15.28:0/2875234591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.588+0000 7f373659c640 1 --2- 172.21.15.28:0/2875234591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a966478a0 0x561a96662220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.588+0000 7f372dd9b640 1 --2- 172.21.15.28:0/2875234591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a9664b580 0x561a9666dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.588+0000 7f3736d9d640 1 -- 172.21.15.28:0/2875234591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a9664b580 msgr2=0x561a9666dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.588+0000 7f3736d9d640 1 --2- 172.21.15.28:0/2875234591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a9664b580 0x561a9666dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.588+0000 7f3736d9d640 1 -- 172.21.15.28:0/2875234591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a966478a0 msgr2=0x561a96662220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.588+0000 7f3736d9d640 1 --2- 172.21.15.28:0/2875234591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a966478a0 0x561a96662220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.588+0000 7f3736d9d640 1 -- 172.21.15.28:0/2875234591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a96668b60 con 0x561a9664bea0 2024-04-25T01:39:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.588+0000 7f373659c640 1 --2- 172.21.15.28:0/2875234591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a966478a0 0x561a96662220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.588+0000 7f3736d9d640 1 --2- 172.21.15.28:0/2875234591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664bea0 0x561a9666e530 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f373002aed0 tx=0x7f3730042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed10d58aa4275ed7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.588+0000 7f371ffff640 1 -- 172.21.15.28:0/2875234591 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3730004240 con 0x561a9664bea0 2024-04-25T01:39:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.588+0000 7f371ffff640 1 -- 172.21.15.28:0/2875234591 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3730036040 con 0x561a9664bea0 2024-04-25T01:39:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.588+0000 7f371ffff640 1 -- 172.21.15.28:0/2875234591 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3730030df0 con 0x561a9664bea0 2024-04-25T01:39:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f373f5fd740 1 -- 172.21.15.28:0/2875234591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664bea0 msgr2=0x561a9666e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f373f5fd740 1 --2- 172.21.15.28:0/2875234591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664bea0 0x561a9666e530 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f373002aed0 tx=0x7f3730042000 comp rx=0 tx=0).stop 2024-04-25T01:39:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f373f5fd740 1 -- 172.21.15.28:0/2875234591 shutdown_connections 2024-04-25T01:39:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f373f5fd740 1 --2- 172.21.15.28:0/2875234591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664bea0 0x561a9666e530 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f373f5fd740 1 --2- 172.21.15.28:0/2875234591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a9664b580 0x561a9666dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f373f5fd740 1 --2- 172.21.15.28:0/2875234591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a966478a0 0x561a96662220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f373f5fd740 1 -- 172.21.15.28:0/2875234591 >> 172.21.15.28:0/2875234591 conn(0x561a96653840 msgr2=0x561a966462b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f373f5fd740 1 -- 172.21.15.28:0/2875234591 shutdown_connections 2024-04-25T01:39:46.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f373f5fd740 1 -- 172.21.15.28:0/2875234591 wait complete. 2024-04-25T01:39:46.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f373f5fd740 1 -- start start 2024-04-25T01:39:46.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f373f5fd740 1 Processor -- start 2024-04-25T01:39:46.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f373f5fd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a966478a0 0x561a9665e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f373f5fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664b580 0x561a9665bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f373f5fd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a9664bea0 0x561a9665c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f373f5fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f37381aff20 con 0x561a9664b580 2024-04-25T01:39:46.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f373f5fd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f37381af850 con 0x561a9664bea0 2024-04-25T01:39:46.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f373f5fd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f37381aca90 con 0x561a966478a0 2024-04-25T01:39:46.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f373f5fd740 10 client.0 _ll_register_callbacks cb 0x7f372000bb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:46.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f372dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664b580 0x561a9665bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f372dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664b580 0x561a9665bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51846/0 (socket says 172.21.15.28:51846) 2024-04-25T01:39:46.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.589+0000 7f372dd9b640 1 -- 172.21.15.28:0/2640593847 learned_addr learned my addr 172.21.15.28:0/2640593847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.590+0000 7f373659c640 1 --2- 172.21.15.28:0/2640593847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a966478a0 0x561a9665e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.590+0000 7f3736d9d640 1 --2- 172.21.15.28:0/2640593847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a9664bea0 0x561a9665c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.590+0000 7f372dd9b640 1 -- 172.21.15.28:0/2640593847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a966478a0 msgr2=0x561a9665e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.590+0000 7f372dd9b640 1 --2- 172.21.15.28:0/2640593847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a966478a0 0x561a9665e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.590+0000 7f372dd9b640 1 -- 172.21.15.28:0/2640593847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a9664bea0 msgr2=0x561a9665c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.590+0000 7f372dd9b640 1 --2- 172.21.15.28:0/2640593847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a9664bea0 0x561a9665c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.590+0000 7f372dd9b640 1 -- 172.21.15.28:0/2640593847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a9638e7d0 con 0x561a9664b580 2024-04-25T01:39:46.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.590+0000 7f3736d9d640 1 --2- 172.21.15.28:0/2640593847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a9664bea0 0x561a9665c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:46.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.590+0000 7f372dd9b640 1 --2- 172.21.15.28:0/2640593847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664b580 0x561a9665bdf0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x561a96654770 tx=0x7f372000b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d99f53ab7002f33 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.590+0000 7f371dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:46.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.590+0000 7f371dffb640 1 -- 172.21.15.28:0/2640593847 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3720015040 con 0x561a9664b580 2024-04-25T01:39:46.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.590+0000 7f373f5fd740 1 -- 172.21.15.28:0/2640593847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3730036040 con 0x561a9664b580 2024-04-25T01:39:46.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.590+0000 7f371dffb640 1 -- 172.21.15.28:0/2640593847 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f372000bdd0 con 0x561a9664b580 2024-04-25T01:39:46.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.590+0000 7f371dffb640 1 -- 172.21.15.28:0/2640593847 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f372001a940 con 0x561a9664b580 2024-04-25T01:39:46.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.590+0000 7f373f5fd740 10 client.17919 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:46.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.590+0000 7f373f5fd740 1 -- 172.21.15.28:0/2640593847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561a966cc560 con 0x561a9664b580 2024-04-25T01:39:46.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.590+0000 7f371dffb640 1 -- 172.21.15.28:0/2640593847 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3730036040 con 0x561a9664b580 2024-04-25T01:39:46.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.591+0000 7f371dffb640 1 -- 172.21.15.28:0/2640593847 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3720024420 con 0x561a9664b580 2024-04-25T01:39:46.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.591+0000 7f373f5fd740 10 client.17919 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:46.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.591+0000 7f373f5fd740 10 client.17919 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:39:46.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.591+0000 7f373f5fd740 20 client.17919 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:46.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.591+0000 7f373f5fd740 1 --2- 172.21.15.28:0/2640593847 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x561a96656570 0x561a966cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.591+0000 7f373f5fd740 4 client.17919 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:46.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.591+0000 7f373f5fd740 1 -- 172.21.15.28:0/2640593847 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561a965b7f00 con 0x561a96656570 2024-04-25T01:39:46.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.591+0000 7f373659c640 1 --2- 172.21.15.28:0/2640593847 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x561a96656570 0x561a966cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f371dffb640 10 client.17919 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:39:46.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f373659c640 1 --2- 172.21.15.28:0/2640593847 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x561a96656570 0x561a966cebb0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=268a25bad53e023d server_cookie=32404a26ec881b1e in_seq=0 out_seq=0 2024-04-25T01:39:46.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f371dffb640 1 -- 172.21.15.28:0/2640593847 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561a965b7f00 con 0x561a96656570 2024-04-25T01:39:46.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f371dffb640 10 client.17919 handle_command_reply: tid=0 2024-04-25T01:39:46.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f373f5fd740 1 client.17919 shutdown 2024-04-25T01:39:46.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f373f5fd740 1 -- 172.21.15.28:0/2640593847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664b580 msgr2=0x561a9665bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f373f5fd740 1 --2- 172.21.15.28:0/2640593847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664b580 0x561a9665bdf0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x561a96654770 tx=0x7f372000b560 comp rx=0 tx=0).stop 2024-04-25T01:39:46.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f373f5fd740 1 -- 172.21.15.28:0/2640593847 shutdown_connections 2024-04-25T01:39:46.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f373f5fd740 1 --2- 172.21.15.28:0/2640593847 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561a9664bea0 0x561a9665c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f373f5fd740 1 --2- 172.21.15.28:0/2640593847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a9664b580 0x561a9665bdf0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f373f5fd740 1 --2- 172.21.15.28:0/2640593847 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x561a96656570 0x561a966cebb0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f373f5fd740 1 --2- 172.21.15.28:0/2640593847 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x561a966478a0 0x561a9665e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f373f5fd740 1 -- 172.21.15.28:0/2640593847 >> 172.21.15.28:0/2640593847 conn(0x561a96653840 msgr2=0x561a96646b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f371dffb640 0 client.17919 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:39:46.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f373f5fd740 1 -- 172.21.15.28:0/2640593847 shutdown_connections 2024-04-25T01:39:46.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f373f5fd740 1 -- 172.21.15.28:0/2640593847 wait complete. 2024-04-25T01:39:46.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f373f5fd740 20 client.17919 trim_cache size 0 max 16384 2024-04-25T01:39:46.583 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:46.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f373cc4d640 1 -- 172.21.15.28:0/770861551 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f37240a2d00 msgr2=0x7f37240a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f373cc4d640 1 --2- 172.21.15.28:0/770861551 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f37240a2d00 0x7f37240a51c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3730005e30 tx=0x7f373003f040 comp rx=0 tx=0).stop 2024-04-25T01:39:46.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f373cc4d640 1 -- 172.21.15.28:0/770861551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3738100610 msgr2=0x7f3738197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.592+0000 7f373cc4d640 1 --2- 172.21.15.28:0/770861551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3738100610 0x7f3738197f80 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f372800c970 tx=0x7f372800ce30 comp rx=0 tx=0).stop 2024-04-25T01:39:46.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.593+0000 7f373cc4d640 1 -- 172.21.15.28:0/770861551 shutdown_connections 2024-04-25T01:39:46.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.593+0000 7f373cc4d640 1 --2- 172.21.15.28:0/770861551 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f37240a2d00 0x7f37240a51c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.593+0000 7f373cc4d640 1 --2- 172.21.15.28:0/770861551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f37381066b0 0x7f373819c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.593+0000 7f373cc4d640 1 --2- 172.21.15.28:0/770861551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3738100fd0 0x7f37381984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.593+0000 7f373cc4d640 1 --2- 172.21.15.28:0/770861551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3738100610 0x7f3738197f80 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.593+0000 7f373cc4d640 1 -- 172.21.15.28:0/770861551 >> 172.21.15.28:0/770861551 conn(0x7f3738077740 msgr2=0x7f37380ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.593+0000 7f373cc4d640 1 -- 172.21.15.28:0/770861551 shutdown_connections 2024-04-25T01:39:46.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.593+0000 7f373cc4d640 1 -- 172.21.15.28:0/770861551 wait complete. 2024-04-25T01:39:46.587 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:39:46.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.708+0000 7f11e5314640 1 -- 172.21.15.28:0/2297717155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11e01020d0 msgr2=0x7f11e010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.708+0000 7f11e5314640 1 --2- 172.21.15.28:0/2297717155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11e01020d0 0x7f11e010e5e0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f11d8009cd0 tx=0x7f11d802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.708+0000 7f11e5314640 1 -- 172.21.15.28:0/2297717155 shutdown_connections 2024-04-25T01:39:46.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.708+0000 7f11e5314640 1 --2- 172.21.15.28:0/2297717155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11e01020d0 0x7f11e010e5e0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.708+0000 7f11e5314640 1 --2- 172.21.15.28:0/2297717155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e0101710 0x7f11e0101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.708+0000 7f11e5314640 1 --2- 172.21.15.28:0/2297717155 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e0107720 0x7f11e0107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.708+0000 7f11e5314640 1 -- 172.21.15.28:0/2297717155 >> 172.21.15.28:0/2297717155 conn(0x7f11e0078b90 msgr2=0x7f11e0078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.708+0000 7f11e5314640 1 -- 172.21.15.28:0/2297717155 shutdown_connections 2024-04-25T01:39:46.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.708+0000 7f11e5314640 1 -- 172.21.15.28:0/2297717155 wait complete. 2024-04-25T01:39:46.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.709+0000 7f11e5314640 1 Processor -- start 2024-04-25T01:39:46.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.709+0000 7f11e5314640 1 -- start start 2024-04-25T01:39:46.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.709+0000 7f11e5314640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11e0101710 0x7f11e0198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.709+0000 7f11e5314640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e01020d0 0x7f11e0198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.709+0000 7f11e5314640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e0107720 0x7f11e019c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.709+0000 7f11e5314640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11e0111b80 con 0x7f11e0101710 2024-04-25T01:39:46.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.709+0000 7f11e5314640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f11e0111a00 con 0x7f11e01020d0 2024-04-25T01:39:46.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.709+0000 7f11e5314640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f11e0111d00 con 0x7f11e0107720 2024-04-25T01:39:46.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.709+0000 7f11deffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11e0101710 0x7f11e0198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.709+0000 7f11deffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11e0101710 0x7f11e0198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39862/0 (socket says 172.21.15.28:39862) 2024-04-25T01:39:46.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.709+0000 7f11deffd640 1 -- 172.21.15.28:0/913963268 learned_addr learned my addr 172.21.15.28:0/913963268 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.709+0000 7f11d67fc640 1 --2- 172.21.15.28:0/913963268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e01020d0 0x7f11e0198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.709+0000 7f11df7fe640 1 --2- 172.21.15.28:0/913963268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e0107720 0x7f11e019c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.709+0000 7f11deffd640 1 -- 172.21.15.28:0/913963268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e0107720 msgr2=0x7f11e019c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.709+0000 7f11deffd640 1 --2- 172.21.15.28:0/913963268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e0107720 0x7f11e019c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.709+0000 7f11deffd640 1 -- 172.21.15.28:0/913963268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e01020d0 msgr2=0x7f11e0198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.709+0000 7f11deffd640 1 --2- 172.21.15.28:0/913963268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e01020d0 0x7f11e0198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.709+0000 7f11deffd640 1 -- 172.21.15.28:0/913963268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e019d030 con 0x7f11e0101710 2024-04-25T01:39:46.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.709+0000 7f11df7fe640 1 --2- 172.21.15.28:0/913963268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e0107720 0x7f11e019c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:46.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.710+0000 7f11deffd640 1 --2- 172.21.15.28:0/913963268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11e0101710 0x7f11e0198040 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f11d000c970 tx=0x7f11d000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.710+0000 7f11dcff9640 1 -- 172.21.15.28:0/913963268 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0007bb0 con 0x7f11e0101710 2024-04-25T01:39:46.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.710+0000 7f11dcff9640 1 -- 172.21.15.28:0/913963268 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11d0007d50 con 0x7f11e0101710 2024-04-25T01:39:46.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.710+0000 7f11e5314640 1 -- 172.21.15.28:0/913963268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11e019d320 con 0x7f11e0101710 2024-04-25T01:39:46.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.710+0000 7f11dcff9640 1 -- 172.21.15.28:0/913963268 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d000f450 con 0x7f11e0101710 2024-04-25T01:39:46.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.710+0000 7f11e5314640 1 -- 172.21.15.28:0/913963268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11e01a4bb0 con 0x7f11e0101710 2024-04-25T01:39:46.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.712+0000 7f11dcff9640 1 -- 172.21.15.28:0/913963268 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f11d0005020 con 0x7f11e0101710 2024-04-25T01:39:46.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.712+0000 7f11e8026740 1 Processor -- start 2024-04-25T01:39:46.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.712+0000 7f11e8026740 1 -- start start 2024-04-25T01:39:46.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11e8026740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba1832990 0x55fba1832d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11e8026740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba18332b0 0x55fba183d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11e8026740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba183d980 0x55fba183fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11e8026740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fba1793670 con 0x55fba183d980 2024-04-25T01:39:46.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11e8026740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fba173b970 con 0x55fba1832990 2024-04-25T01:39:46.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11e8026740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55fba178f630 con 0x55fba18332b0 2024-04-25T01:39:46.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11d67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba183d980 0x55fba183fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11d67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba183d980 0x55fba183fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39872/0 (socket says 172.21.15.28:39872) 2024-04-25T01:39:46.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11d67fc640 1 -- 172.21.15.28:0/2948420845 learned_addr learned my addr 172.21.15.28:0/2948420845 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11deffd640 1 --2- 172.21.15.28:0/2948420845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba18332b0 0x55fba183d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11df7fe640 1 --2- 172.21.15.28:0/2948420845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba1832990 0x55fba1832d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11dcff9640 1 --2- 172.21.15.28:0/913963268 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f11cc0a2d80 0x7f11cc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11d67fc640 1 -- 172.21.15.28:0/2948420845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba18332b0 msgr2=0x55fba183d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11d67fc640 1 --2- 172.21.15.28:0/2948420845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba18332b0 0x55fba183d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11d67fc640 1 -- 172.21.15.28:0/2948420845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba1832990 msgr2=0x55fba1832d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11d67fc640 1 --2- 172.21.15.28:0/2948420845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba1832990 0x55fba1832d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11d67fc640 1 -- 172.21.15.28:0/2948420845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fba175f400 con 0x55fba183d980 2024-04-25T01:39:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11deffd640 1 --2- 172.21.15.28:0/2948420845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba18332b0 0x55fba183d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11df7fe640 1 --2- 172.21.15.28:0/913963268 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f11cc0a2d80 0x7f11cc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11d67fc640 1 --2- 172.21.15.28:0/2948420845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba183d980 0x55fba183fd70 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009c20 tx=0x7f11c80117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af3259aa6872ebaf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11dcff9640 1 -- 172.21.15.28:0/913963268 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f11d00cf560 con 0x7f11e0101710 2024-04-25T01:39:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.713+0000 7f11d57fa640 1 -- 172.21.15.28:0/2948420845 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c8012450 con 0x55fba183d980 2024-04-25T01:39:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.714+0000 7f11d57fa640 1 -- 172.21.15.28:0/2948420845 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11c8012ad0 con 0x55fba183d980 2024-04-25T01:39:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.714+0000 7f11d57fa640 1 -- 172.21.15.28:0/2948420845 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c800fbc0 con 0x55fba183d980 2024-04-25T01:39:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.714+0000 7f11df7fe640 1 --2- 172.21.15.28:0/913963268 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f11cc0a2d80 0x7f11cc0a5240 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f11d8006010 tx=0x7f11d80095c0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.714+0000 7f11e8026740 1 -- 172.21.15.28:0/2948420845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba183d980 msgr2=0x55fba183fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.714+0000 7f11e8026740 1 --2- 172.21.15.28:0/2948420845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba183d980 0x55fba183fd70 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009c20 tx=0x7f11c80117b0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.714+0000 7f11e8026740 1 -- 172.21.15.28:0/2948420845 shutdown_connections 2024-04-25T01:39:46.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.714+0000 7f11e8026740 1 --2- 172.21.15.28:0/2948420845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba183d980 0x55fba183fd70 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.714+0000 7f11e8026740 1 --2- 172.21.15.28:0/2948420845 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba18332b0 0x55fba183d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.714+0000 7f11e8026740 1 --2- 172.21.15.28:0/2948420845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba1832990 0x55fba1832d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.714+0000 7f11e8026740 1 -- 172.21.15.28:0/2948420845 >> 172.21.15.28:0/2948420845 conn(0x55fba182ecb0 msgr2=0x55fba182f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.714+0000 7f11e8026740 1 -- 172.21.15.28:0/2948420845 shutdown_connections 2024-04-25T01:39:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.714+0000 7f11e8026740 1 -- 172.21.15.28:0/2948420845 wait complete. 2024-04-25T01:39:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.715+0000 7f11e8026740 1 -- start start 2024-04-25T01:39:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.715+0000 7f11e8026740 1 Processor -- start 2024-04-25T01:39:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.715+0000 7f11e8026740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba1832990 0x55fba183d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.715+0000 7f11e8026740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba18332b0 0x55fba183a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.715+0000 7f11e8026740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba183d980 0x55fba183aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.715+0000 7f11e8026740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fba1840770 con 0x55fba183d980 2024-04-25T01:39:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.715+0000 7f11e8026740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fba174f050 con 0x55fba1832990 2024-04-25T01:39:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.715+0000 7f11e8026740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55fba18408f0 con 0x55fba18332b0 2024-04-25T01:39:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.715+0000 7f11e8026740 10 client.0 _ll_register_callbacks cb 0x55fba14a6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.715+0000 7f11df7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba183d980 0x55fba183aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.715+0000 7f11df7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba183d980 0x55fba183aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39876/0 (socket says 172.21.15.28:39876) 2024-04-25T01:39:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.715+0000 7f11df7fe640 1 -- 172.21.15.28:0/3695393893 learned_addr learned my addr 172.21.15.28:0/3695393893 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.715+0000 7f11deffd640 1 --2- 172.21.15.28:0/3695393893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba1832990 0x55fba183d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.715+0000 7f11d67fc640 1 --2- 172.21.15.28:0/3695393893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba18332b0 0x55fba183a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.715+0000 7f11df7fe640 1 -- 172.21.15.28:0/3695393893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba18332b0 msgr2=0x55fba183a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.715+0000 7f11df7fe640 1 --2- 172.21.15.28:0/3695393893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba18332b0 0x55fba183a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.715+0000 7f11df7fe640 1 -- 172.21.15.28:0/3695393893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba1832990 msgr2=0x55fba183d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.715+0000 7f11df7fe640 1 --2- 172.21.15.28:0/3695393893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba1832990 0x55fba183d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.715+0000 7f11df7fe640 1 -- 172.21.15.28:0/3695393893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fba1720dc0 con 0x55fba183d980 2024-04-25T01:39:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.716+0000 7f11df7fe640 1 --2- 172.21.15.28:0/3695393893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba183d980 0x55fba183aa20 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f11d8003bf0 tx=0x7f11d802a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0a26c5aac7ef52 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.716+0000 7f11c67fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.716+0000 7f11c67fc640 1 -- 172.21.15.28:0/3695393893 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d80047c0 con 0x55fba183d980 2024-04-25T01:39:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.716+0000 7f11e8026740 1 -- 172.21.15.28:0/3695393893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11c8012ad0 con 0x55fba183d980 2024-04-25T01:39:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.716+0000 7f11c67fc640 1 -- 172.21.15.28:0/3695393893 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f11d80305a0 con 0x55fba183d980 2024-04-25T01:39:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.716+0000 7f11c67fc640 1 -- 172.21.15.28:0/3695393893 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f11d8031d50 con 0x55fba183d980 2024-04-25T01:39:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.716+0000 7f11e8026740 10 client.17931 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.716+0000 7f11e8026740 1 -- 172.21.15.28:0/3695393893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fba14a73f0 con 0x55fba183d980 2024-04-25T01:39:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.716+0000 7f11c67fc640 1 -- 172.21.15.28:0/3695393893 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11c8012ad0 con 0x55fba183d980 2024-04-25T01:39:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.716+0000 7f11c67fc640 1 -- 172.21.15.28:0/3695393893 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f11d8043070 con 0x55fba183d980 2024-04-25T01:39:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.716+0000 7f11e8026740 10 client.17931 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.716+0000 7f11e8026740 10 client.17931 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:39:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.716+0000 7f11e8026740 20 client.17931 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.716+0000 7f11e8026740 1 --2- 172.21.15.28:0/3695393893 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55fba184d920 0x55fba184fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.716+0000 7f11e8026740 4 client.17931 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:46.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.716+0000 7f11e8026740 1 -- 172.21.15.28:0/3695393893 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fba17a0bb0 con 0x55fba184d920 2024-04-25T01:39:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.717+0000 7f11deffd640 1 --2- 172.21.15.28:0/3695393893 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55fba184d920 0x55fba184fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.717+0000 7f11deffd640 1 --2- 172.21.15.28:0/3695393893 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55fba184d920 0x55fba184fd70 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7ec938161ddf844d server_cookie=54d7c0fc6e205c1d in_seq=0 out_seq=0 2024-04-25T01:39:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.717+0000 7f11c67fc640 10 client.17931 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:39:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.718+0000 7f11c67fc640 1 -- 172.21.15.28:0/3695393893 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fba17a0bb0 con 0x55fba184d920 2024-04-25T01:39:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.718+0000 7f11c67fc640 10 client.17931 handle_command_reply: tid=0 2024-04-25T01:39:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.718+0000 7f11e8026740 1 client.17931 shutdown 2024-04-25T01:39:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.718+0000 7f11e8026740 1 -- 172.21.15.28:0/3695393893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba183d980 msgr2=0x55fba183aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.718+0000 7f11e8026740 1 --2- 172.21.15.28:0/3695393893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba183d980 0x55fba183aa20 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f11d8003bf0 tx=0x7f11d802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.718+0000 7f11e8026740 1 -- 172.21.15.28:0/3695393893 shutdown_connections 2024-04-25T01:39:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.718+0000 7f11e8026740 1 --2- 172.21.15.28:0/3695393893 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55fba184d920 0x55fba184fd70 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.718+0000 7f11e8026740 1 --2- 172.21.15.28:0/3695393893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba183d980 0x55fba183aa20 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.718+0000 7f11e8026740 1 --2- 172.21.15.28:0/3695393893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba18332b0 0x55fba183a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.718+0000 7f11c67fc640 0 client.17931 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:39:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.718+0000 7f11e8026740 1 --2- 172.21.15.28:0/3695393893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba1832990 0x55fba183d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.718+0000 7f11e8026740 1 -- 172.21.15.28:0/3695393893 >> 172.21.15.28:0/3695393893 conn(0x55fba182ecb0 msgr2=0x55fba1829b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.718+0000 7f11e8026740 1 -- 172.21.15.28:0/3695393893 shutdown_connections 2024-04-25T01:39:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.718+0000 7f11e8026740 1 -- 172.21.15.28:0/3695393893 wait complete. 2024-04-25T01:39:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.718+0000 7f11e8026740 20 client.17931 trim_cache size 0 max 16384 2024-04-25T01:39:46.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.731+0000 7f11e8026740 1 Processor -- start 2024-04-25T01:39:46.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.731+0000 7f11e8026740 1 -- start start 2024-04-25T01:39:46.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.732+0000 7f11e8026740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba1832990 0x55fba1849630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.732+0000 7f11e8026740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba18332b0 0x55fba1855640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.732+0000 7f11e8026740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba183d980 0x55fba1855b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.732+0000 7f11e8026740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11e01ac8b0 con 0x55fba18332b0 2024-04-25T01:39:46.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.732+0000 7f11e8026740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f11e01ae3f0 con 0x55fba183d980 2024-04-25T01:39:46.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.732+0000 7f11e8026740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f11e01acf80 con 0x55fba1832990 2024-04-25T01:39:46.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.732+0000 7f11d67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba18332b0 0x55fba1855640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.732+0000 7f11d67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba18332b0 0x55fba1855640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39878/0 (socket says 172.21.15.28:39878) 2024-04-25T01:39:46.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.732+0000 7f11d67fc640 1 -- 172.21.15.28:0/2390107573 learned_addr learned my addr 172.21.15.28:0/2390107573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.732+0000 7f11deffd640 1 --2- 172.21.15.28:0/2390107573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba1832990 0x55fba1849630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.732+0000 7f11df7fe640 1 --2- 172.21.15.28:0/2390107573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba183d980 0x55fba1855b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.732+0000 7f11d67fc640 1 -- 172.21.15.28:0/2390107573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba1832990 msgr2=0x55fba1849630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.732+0000 7f11d67fc640 1 --2- 172.21.15.28:0/2390107573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba1832990 0x55fba1849630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.732+0000 7f11d67fc640 1 -- 172.21.15.28:0/2390107573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba183d980 msgr2=0x55fba1855b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.732+0000 7f11d67fc640 1 --2- 172.21.15.28:0/2390107573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba183d980 0x55fba1855b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.732+0000 7f11d67fc640 1 -- 172.21.15.28:0/2390107573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fba15717d0 con 0x55fba18332b0 2024-04-25T01:39:46.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.732+0000 7f11d67fc640 1 --2- 172.21.15.28:0/2390107573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba18332b0 0x55fba1855640 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009c20 tx=0x7f11c8009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56280ebf6721a0d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.733+0000 7f11d4ff9640 1 -- 172.21.15.28:0/2390107573 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c80052b0 con 0x55fba18332b0 2024-04-25T01:39:46.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.733+0000 7f11d4ff9640 1 -- 172.21.15.28:0/2390107573 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11c8009dd0 con 0x55fba18332b0 2024-04-25T01:39:46.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.733+0000 7f11d4ff9640 1 -- 172.21.15.28:0/2390107573 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c800f710 con 0x55fba18332b0 2024-04-25T01:39:46.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.733+0000 7f11e8026740 1 -- 172.21.15.28:0/2390107573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba18332b0 msgr2=0x55fba1855640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.733+0000 7f11e8026740 1 --2- 172.21.15.28:0/2390107573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba18332b0 0x55fba1855640 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009c20 tx=0x7f11c8009930 comp rx=0 tx=0).stop 2024-04-25T01:39:46.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.733+0000 7f11e8026740 1 -- 172.21.15.28:0/2390107573 shutdown_connections 2024-04-25T01:39:46.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.733+0000 7f11e8026740 1 --2- 172.21.15.28:0/2390107573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba183d980 0x55fba1855b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.733+0000 7f11e8026740 1 --2- 172.21.15.28:0/2390107573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba18332b0 0x55fba1855640 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.733+0000 7f11e8026740 1 --2- 172.21.15.28:0/2390107573 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba1832990 0x55fba1849630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.733+0000 7f11e8026740 1 -- 172.21.15.28:0/2390107573 >> 172.21.15.28:0/2390107573 conn(0x55fba182ecb0 msgr2=0x55fba1829b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.733+0000 7f11e8026740 1 -- 172.21.15.28:0/2390107573 shutdown_connections 2024-04-25T01:39:46.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.733+0000 7f11e8026740 1 -- 172.21.15.28:0/2390107573 wait complete. 2024-04-25T01:39:46.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11e8026740 1 -- start start 2024-04-25T01:39:46.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11e8026740 1 Processor -- start 2024-04-25T01:39:46.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11e8026740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba1832990 0x55fba18437a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11e8026740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba18332b0 0x55fba1843ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11e8026740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba183d980 0x55fba1844220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11e8026740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11e01add20 con 0x55fba1832990 2024-04-25T01:39:46.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11e8026740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f11e01ad650 con 0x55fba183d980 2024-04-25T01:39:46.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11e8026740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f11e01aa890 con 0x55fba18332b0 2024-04-25T01:39:46.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11e8026740 10 client.0 _ll_register_callbacks cb 0x7f11c8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11deffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba1832990 0x55fba18437a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11deffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba1832990 0x55fba18437a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39886/0 (socket says 172.21.15.28:39886) 2024-04-25T01:39:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11deffd640 1 -- 172.21.15.28:0/475454344 learned_addr learned my addr 172.21.15.28:0/475454344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11df7fe640 1 --2- 172.21.15.28:0/475454344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba183d980 0x55fba1844220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11deffd640 1 -- 172.21.15.28:0/475454344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba18332b0 msgr2=0x55fba1843ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11deffd640 1 --2- 172.21.15.28:0/475454344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba18332b0 0x55fba1843ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11deffd640 1 -- 172.21.15.28:0/475454344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba183d980 msgr2=0x55fba1844220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11deffd640 1 --2- 172.21.15.28:0/475454344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba183d980 0x55fba1844220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11deffd640 1 -- 172.21.15.28:0/475454344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fba18b30c0 con 0x55fba1832990 2024-04-25T01:39:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11df7fe640 1 --2- 172.21.15.28:0/475454344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba183d980 0x55fba1844220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11deffd640 1 --2- 172.21.15.28:0/475454344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba1832990 0x55fba18437a0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f11d009b6d0 tx=0x7f11d0086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d9963136c4a8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.734+0000 7f11c67fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.735+0000 7f11c67fc640 1 -- 172.21.15.28:0/475454344 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d001d040 con 0x55fba1832990 2024-04-25T01:39:46.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.735+0000 7f11c67fc640 1 -- 172.21.15.28:0/475454344 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f11d0099a40 con 0x55fba1832990 2024-04-25T01:39:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.735+0000 7f11e8026740 1 -- 172.21.15.28:0/475454344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11c8009dd0 con 0x55fba1832990 2024-04-25T01:39:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.735+0000 7f11c67fc640 1 -- 172.21.15.28:0/475454344 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f11d009b740 con 0x55fba1832990 2024-04-25T01:39:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.735+0000 7f11e8026740 10 client.17937 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.735+0000 7f11e8026740 1 -- 172.21.15.28:0/475454344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fba18b43e0 con 0x55fba1832990 2024-04-25T01:39:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.735+0000 7f11c67fc640 1 -- 172.21.15.28:0/475454344 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11c8009dd0 con 0x55fba1832990 2024-04-25T01:39:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.735+0000 7f11c67fc640 1 -- 172.21.15.28:0/475454344 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f11d0005310 con 0x55fba1832990 2024-04-25T01:39:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.735+0000 7f11e8026740 10 client.17937 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.735+0000 7f11e8026740 10 client.17937 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:39:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.735+0000 7f11e8026740 20 client.17937 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.735+0000 7f11e8026740 1 --2- 172.21.15.28:0/475454344 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55fba18b4ae0 0x55fba18b6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.735+0000 7f11e8026740 4 client.17937 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.735+0000 7f11e8026740 1 -- 172.21.15.28:0/475454344 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fba183adb0 con 0x55fba18b4ae0 2024-04-25T01:39:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.735+0000 7f11d67fc640 1 --2- 172.21.15.28:0/475454344 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55fba18b4ae0 0x55fba18b6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.736+0000 7f11d67fc640 1 --2- 172.21.15.28:0/475454344 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55fba18b4ae0 0x55fba18b6f40 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cf4ab0ada4033fc5 server_cookie=696a580fe3d22b3 in_seq=0 out_seq=0 2024-04-25T01:39:46.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.736+0000 7f11c67fc640 10 client.17937 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:39:46.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.736+0000 7f11c67fc640 1 -- 172.21.15.28:0/475454344 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fba183adb0 con 0x55fba18b4ae0 2024-04-25T01:39:46.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.736+0000 7f11c67fc640 10 client.17937 handle_command_reply: tid=0 2024-04-25T01:39:46.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.736+0000 7f11e8026740 1 client.17937 shutdown 2024-04-25T01:39:46.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.736+0000 7f11e8026740 1 -- 172.21.15.28:0/475454344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba1832990 msgr2=0x55fba18437a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.736+0000 7f11e8026740 1 --2- 172.21.15.28:0/475454344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba1832990 0x55fba18437a0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f11d009b6d0 tx=0x7f11d0086040 comp rx=0 tx=0).stop 2024-04-25T01:39:46.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.736+0000 7f11e8026740 1 -- 172.21.15.28:0/475454344 shutdown_connections 2024-04-25T01:39:46.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.736+0000 7f11e8026740 1 --2- 172.21.15.28:0/475454344 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55fba18b4ae0 0x55fba18b6f40 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.736+0000 7f11e8026740 1 --2- 172.21.15.28:0/475454344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba183d980 0x55fba1844220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.736+0000 7f11e8026740 1 --2- 172.21.15.28:0/475454344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55fba18332b0 0x55fba1843ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.736+0000 7f11e8026740 1 --2- 172.21.15.28:0/475454344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fba1832990 0x55fba18437a0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.736+0000 7f11e8026740 1 -- 172.21.15.28:0/475454344 >> 172.21.15.28:0/475454344 conn(0x55fba182ecb0 msgr2=0x55fba1848620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.736+0000 7f11c67fc640 0 client.17937 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:39:46.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.736+0000 7f11e8026740 1 -- 172.21.15.28:0/475454344 shutdown_connections 2024-04-25T01:39:46.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.736+0000 7f11e8026740 1 -- 172.21.15.28:0/475454344 wait complete. 2024-04-25T01:39:46.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.736+0000 7f11e8026740 20 client.17937 trim_cache size 0 max 16384 2024-04-25T01:39:46.727 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:46.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.737+0000 7f11e5314640 1 -- 172.21.15.28:0/913963268 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f11cc0a2d80 msgr2=0x7f11cc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.737+0000 7f11e5314640 1 --2- 172.21.15.28:0/913963268 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f11cc0a2d80 0x7f11cc0a5240 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f11d8006010 tx=0x7f11d80095c0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.737+0000 7f11e5314640 1 -- 172.21.15.28:0/913963268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11e0101710 msgr2=0x7f11e0198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.737+0000 7f11e5314640 1 --2- 172.21.15.28:0/913963268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11e0101710 0x7f11e0198040 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f11d000c970 tx=0x7f11d000ce30 comp rx=0 tx=0).stop 2024-04-25T01:39:46.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.737+0000 7f11e5314640 1 -- 172.21.15.28:0/913963268 shutdown_connections 2024-04-25T01:39:46.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.737+0000 7f11e5314640 1 --2- 172.21.15.28:0/913963268 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f11cc0a2d80 0x7f11cc0a5240 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.737+0000 7f11e5314640 1 --2- 172.21.15.28:0/913963268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f11e0107720 0x7f11e019c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.737+0000 7f11e5314640 1 --2- 172.21.15.28:0/913963268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f11e01020d0 0x7f11e0198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.737+0000 7f11e5314640 1 --2- 172.21.15.28:0/913963268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11e0101710 0x7f11e0198040 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.737+0000 7f11e5314640 1 -- 172.21.15.28:0/913963268 >> 172.21.15.28:0/913963268 conn(0x7f11e0078b90 msgr2=0x7f11e00ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.737+0000 7f11e5314640 1 -- 172.21.15.28:0/913963268 shutdown_connections 2024-04-25T01:39:46.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.737+0000 7f11e5314640 1 -- 172.21.15.28:0/913963268 wait complete. 2024-04-25T01:39:46.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:39:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.852+0000 7fc009691640 1 -- 172.21.15.28:0/1381359114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc004106630 msgr2=0x7fc004106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.852+0000 7fc009691640 1 --2- 172.21.15.28:0/1381359114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc004106630 0x7fc004106a10 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fbff4009cd0 tx=0x7fbff402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.852+0000 7fc009691640 1 -- 172.21.15.28:0/1381359114 shutdown_connections 2024-04-25T01:39:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.852+0000 7fc009691640 1 --2- 172.21.15.28:0/1381359114 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc004100fe0 0x7fc00410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.852+0000 7fc009691640 1 --2- 172.21.15.28:0/1381359114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc004100620 0x7fc004100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.852+0000 7fc009691640 1 --2- 172.21.15.28:0/1381359114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc004106630 0x7fc004106a10 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.852+0000 7fc009691640 1 -- 172.21.15.28:0/1381359114 >> 172.21.15.28:0/1381359114 conn(0x7fc0040776a0 msgr2=0x7fc004077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.852+0000 7fc009691640 1 -- 172.21.15.28:0/1381359114 shutdown_connections 2024-04-25T01:39:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.852+0000 7fc009691640 1 -- 172.21.15.28:0/1381359114 wait complete. 2024-04-25T01:39:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.853+0000 7fc009691640 1 Processor -- start 2024-04-25T01:39:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.853+0000 7fc009691640 1 -- start start 2024-04-25T01:39:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.853+0000 7fc009691640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc004100620 0x7fc0041980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.853+0000 7fc009691640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc004100fe0 0x7fc004198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.853+0000 7fc009691640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc004106630 0x7fc00419c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.853+0000 7fc009691640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc004110a00 con 0x7fc004106630 2024-04-25T01:39:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.853+0000 7fc009691640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc004110880 con 0x7fc004100620 2024-04-25T01:39:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.853+0000 7fc009691640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc004110b80 con 0x7fc004100fe0 2024-04-25T01:39:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.853+0000 7fc0037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc004106630 0x7fc00419c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.853+0000 7fc0037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc004106630 0x7fc00419c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39910/0 (socket says 172.21.15.28:39910) 2024-04-25T01:39:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.853+0000 7fc0037fe640 1 -- 172.21.15.28:0/1241560050 learned_addr learned my addr 172.21.15.28:0/1241560050 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.854+0000 7fc0027fc640 1 --2- 172.21.15.28:0/1241560050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc004100fe0 0x7fc004198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.854+0000 7fc002ffd640 1 --2- 172.21.15.28:0/1241560050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc004100620 0x7fc0041980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.854+0000 7fc0037fe640 1 -- 172.21.15.28:0/1241560050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc004100fe0 msgr2=0x7fc004198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.854+0000 7fc0037fe640 1 --2- 172.21.15.28:0/1241560050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc004100fe0 0x7fc004198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.854+0000 7fc0037fe640 1 -- 172.21.15.28:0/1241560050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc004100620 msgr2=0x7fc0041980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.854+0000 7fc0037fe640 1 --2- 172.21.15.28:0/1241560050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc004100620 0x7fc0041980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.854+0000 7fc0037fe640 1 -- 172.21.15.28:0/1241560050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc00419d0c0 con 0x7fc004106630 2024-04-25T01:39:46.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.854+0000 7fc002ffd640 1 --2- 172.21.15.28:0/1241560050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc004100620 0x7fc0041980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:46.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.854+0000 7fc0037fe640 1 --2- 172.21.15.28:0/1241560050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc004106630 0x7fc00419c990 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fbffc00d950 tx=0x7fbffc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.854+0000 7fbfebfff640 1 -- 172.21.15.28:0/1241560050 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbffc017030 con 0x7fc004106630 2024-04-25T01:39:46.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.854+0000 7fc009691640 1 -- 172.21.15.28:0/1241560050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc00419d3b0 con 0x7fc004106630 2024-04-25T01:39:46.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.854+0000 7fbfebfff640 1 -- 172.21.15.28:0/1241560050 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbffc004020 con 0x7fc004106630 2024-04-25T01:39:46.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.854+0000 7fbfebfff640 1 -- 172.21.15.28:0/1241560050 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbffc003c00 con 0x7fc004106630 2024-04-25T01:39:46.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.854+0000 7fc009691640 1 -- 172.21.15.28:0/1241560050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0041a4c40 con 0x7fc004106630 2024-04-25T01:39:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.856+0000 7fbfebfff640 1 -- 172.21.15.28:0/1241560050 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fbffc01a070 con 0x7fc004106630 2024-04-25T01:39:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.856+0000 7fc00c395740 1 Processor -- start 2024-04-25T01:39:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc00c395740 1 -- start start 2024-04-25T01:39:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc00c395740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c57580 0x5595c5c57960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc00c395740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 0x5595c5c62030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc00c395740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c62570 0x5595c5c64960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc00c395740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5595c5bbc670 con 0x5595c5c57ea0 2024-04-25T01:39:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc00c395740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5595c5b64970 con 0x5595c5c62570 2024-04-25T01:39:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc00c395740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5595c5bb8630 con 0x5595c5c57580 2024-04-25T01:39:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fbfebfff640 1 --2- 172.21.15.28:0/1241560050 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbfec0a2d00 0x7fbfec0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc002ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 0x5595c5c62030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc002ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 0x5595c5c62030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39926/0 (socket says 172.21.15.28:39926) 2024-04-25T01:39:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc002ffd640 1 -- 172.21.15.28:0/3602918936 learned_addr learned my addr 172.21.15.28:0/3602918936 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc0037fe640 1 --2- 172.21.15.28:0/1241560050 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbfec0a2d00 0x7fbfec0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc0027fc640 1 --2- 172.21.15.28:0/3602918936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c57580 0x5595c5c57960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fbfebfff640 1 -- 172.21.15.28:0/1241560050 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fbffc0cefe0 con 0x7fc004106630 2024-04-25T01:39:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc0027fc640 1 --2- 172.21.15.28:0/3602918936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c62570 0x5595c5c64960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc002ffd640 1 -- 172.21.15.28:0/3602918936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c57580 msgr2=0x5595c5c57960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc002ffd640 1 --2- 172.21.15.28:0/3602918936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c57580 0x5595c5c57960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc002ffd640 1 -- 172.21.15.28:0/3602918936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c62570 msgr2=0x5595c5c64960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc002ffd640 1 --2- 172.21.15.28:0/3602918936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c62570 0x5595c5c64960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc002ffd640 1 -- 172.21.15.28:0/3602918936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595c5b88400 con 0x5595c5c57ea0 2024-04-25T01:39:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc0027fc640 1 --2- 172.21.15.28:0/3602918936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c57580 0x5595c5c57960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.857+0000 7fc0037fe640 1 --2- 172.21.15.28:0/1241560050 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbfec0a2d00 0x7fbfec0a51c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0cc4a0 tx=0x7fbffc06d180 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.858+0000 7fc002ffd640 1 --2- 172.21.15.28:0/3602918936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 0x5595c5c62030 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fbff4000f80 tx=0x7fbff4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ecea89aa8067c3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.858+0000 7fbfe97fa640 1 -- 172.21.15.28:0/3602918936 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff40318c0 con 0x5595c5c57ea0 2024-04-25T01:39:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.858+0000 7fbfe97fa640 1 -- 172.21.15.28:0/3602918936 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbff4036040 con 0x5595c5c57ea0 2024-04-25T01:39:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.858+0000 7fbfe97fa640 1 -- 172.21.15.28:0/3602918936 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff40045d0 con 0x5595c5c57ea0 2024-04-25T01:39:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.858+0000 7fc00c395740 1 -- 172.21.15.28:0/3602918936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 msgr2=0x5595c5c62030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.858+0000 7fc00c395740 1 --2- 172.21.15.28:0/3602918936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 0x5595c5c62030 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fbff4000f80 tx=0x7fbff4005f90 comp rx=0 tx=0).stop 2024-04-25T01:39:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.858+0000 7fc00c395740 1 -- 172.21.15.28:0/3602918936 shutdown_connections 2024-04-25T01:39:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.858+0000 7fc00c395740 1 --2- 172.21.15.28:0/3602918936 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c62570 0x5595c5c64960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.858+0000 7fc00c395740 1 --2- 172.21.15.28:0/3602918936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 0x5595c5c62030 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.858+0000 7fc00c395740 1 --2- 172.21.15.28:0/3602918936 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c57580 0x5595c5c57960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.858+0000 7fc00c395740 1 -- 172.21.15.28:0/3602918936 >> 172.21.15.28:0/3602918936 conn(0x5595c5c538a0 msgr2=0x5595c5c53c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.858+0000 7fc00c395740 1 -- 172.21.15.28:0/3602918936 shutdown_connections 2024-04-25T01:39:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.858+0000 7fc00c395740 1 -- 172.21.15.28:0/3602918936 wait complete. 2024-04-25T01:39:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.859+0000 7fc00c395740 1 -- start start 2024-04-25T01:39:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.859+0000 7fc00c395740 1 Processor -- start 2024-04-25T01:39:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.859+0000 7fc00c395740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c57580 0x5595c5c61e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.859+0000 7fc00c395740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 0x5595c5c5f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.859+0000 7fc00c395740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c62570 0x5595c5c5f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.859+0000 7fc00c395740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5595c5c65300 con 0x5595c5c57ea0 2024-04-25T01:39:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.859+0000 7fc00c395740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5595c5b78050 con 0x5595c5c57580 2024-04-25T01:39:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.859+0000 7fc00c395740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5595c5c65480 con 0x5595c5c62570 2024-04-25T01:39:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.859+0000 7fc00c395740 10 client.0 _ll_register_callbacks cb 0x5595c58cfbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.859+0000 7fc002ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 0x5595c5c5f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.859+0000 7fc002ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 0x5595c5c5f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39932/0 (socket says 172.21.15.28:39932) 2024-04-25T01:39:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.859+0000 7fc002ffd640 1 -- 172.21.15.28:0/883909496 learned_addr learned my addr 172.21.15.28:0/883909496 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.859+0000 7fc0027fc640 1 --2- 172.21.15.28:0/883909496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c57580 0x5595c5c61e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.859+0000 7fc0027fc640 1 --2- 172.21.15.28:0/883909496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c62570 0x5595c5c5f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fc002ffd640 1 -- 172.21.15.28:0/883909496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c62570 msgr2=0x5595c5c5f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fc002ffd640 1 --2- 172.21.15.28:0/883909496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c62570 0x5595c5c5f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fc002ffd640 1 -- 172.21.15.28:0/883909496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c57580 msgr2=0x5595c5c61e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fc002ffd640 1 --2- 172.21.15.28:0/883909496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c57580 0x5595c5c61e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fc002ffd640 1 -- 172.21.15.28:0/883909496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595c5b49dc0 con 0x5595c5c57ea0 2024-04-25T01:39:46.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fc0027fc640 1 --2- 172.21.15.28:0/883909496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c57580 0x5595c5c61e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:46.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fc002ffd640 1 --2- 172.21.15.28:0/883909496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 0x5595c5c5f180 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fbff4031d70 tx=0x7fbff402a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79876ad36cd0e08a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fbfe6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:46.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fbfe6ff5640 1 -- 172.21.15.28:0/883909496 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff4054060 con 0x5595c5c57ea0 2024-04-25T01:39:46.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fbfe6ff5640 1 -- 172.21.15.28:0/883909496 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fbff402e040 con 0x5595c5c57ea0 2024-04-25T01:39:46.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fc00c395740 1 -- 172.21.15.28:0/883909496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbff4036040 con 0x5595c5c57ea0 2024-04-25T01:39:46.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fbfe6ff5640 1 -- 172.21.15.28:0/883909496 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbff405b050 con 0x5595c5c57ea0 2024-04-25T01:39:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fc00c395740 10 client.17949 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fc00c395740 1 -- 172.21.15.28:0/883909496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5595c58d03f0 con 0x5595c5c57ea0 2024-04-25T01:39:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fbfe6ff5640 1 -- 172.21.15.28:0/883909496 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbff4036040 con 0x5595c5c57ea0 2024-04-25T01:39:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fbfe6ff5640 1 -- 172.21.15.28:0/883909496 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbff4030070 con 0x5595c5c57ea0 2024-04-25T01:39:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fc00c395740 10 client.17949 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fc00c395740 10 client.17949 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fc00c395740 20 client.17949 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fc00c395740 1 --2- 172.21.15.28:0/883909496 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5595c5c723d0 0x5595c5c747c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fc00c395740 4 client.17949 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.860+0000 7fc00c395740 1 -- 172.21.15.28:0/883909496 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595c5bc9bb0 con 0x5595c5c723d0 2024-04-25T01:39:46.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.861+0000 7fc0027fc640 1 --2- 172.21.15.28:0/883909496 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5595c5c723d0 0x5595c5c747c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.861+0000 7fbfe6ff5640 10 client.17949 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:46.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.861+0000 7fc0027fc640 1 --2- 172.21.15.28:0/883909496 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5595c5c723d0 0x5595c5c747c0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fc4625e6d1b8f86 server_cookie=333421847a39359f in_seq=0 out_seq=0 2024-04-25T01:39:46.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.862+0000 7fbfe6ff5640 1 -- 172.21.15.28:0/883909496 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5595c5bc9bb0 con 0x5595c5c723d0 2024-04-25T01:39:46.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.862+0000 7fbfe6ff5640 10 client.17949 handle_command_reply: tid=0 2024-04-25T01:39:46.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.862+0000 7fc00c395740 1 client.17949 shutdown 2024-04-25T01:39:46.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.862+0000 7fc00c395740 1 -- 172.21.15.28:0/883909496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 msgr2=0x5595c5c5f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.862+0000 7fc00c395740 1 --2- 172.21.15.28:0/883909496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 0x5595c5c5f180 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fbff4031d70 tx=0x7fbff402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.862+0000 7fc00c395740 1 -- 172.21.15.28:0/883909496 shutdown_connections 2024-04-25T01:39:46.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.862+0000 7fc00c395740 1 --2- 172.21.15.28:0/883909496 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5595c5c723d0 0x5595c5c747c0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.862+0000 7fc00c395740 1 --2- 172.21.15.28:0/883909496 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c62570 0x5595c5c5f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.862+0000 7fc00c395740 1 --2- 172.21.15.28:0/883909496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 0x5595c5c5f180 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.862+0000 7fc00c395740 1 --2- 172.21.15.28:0/883909496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c57580 0x5595c5c61e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.862+0000 7fc00c395740 1 -- 172.21.15.28:0/883909496 >> 172.21.15.28:0/883909496 conn(0x5595c5c538a0 msgr2=0x5595c5c52b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.862+0000 7fbfe6ff5640 0 client.17949 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:46.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.862+0000 7fc00c395740 1 -- 172.21.15.28:0/883909496 shutdown_connections 2024-04-25T01:39:46.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.862+0000 7fc00c395740 1 -- 172.21.15.28:0/883909496 wait complete. 2024-04-25T01:39:46.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.862+0000 7fc00c395740 20 client.17949 trim_cache size 0 max 16384 2024-04-25T01:39:46.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.876+0000 7fc00c395740 1 Processor -- start 2024-04-25T01:39:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.876+0000 7fc00c395740 1 -- start start 2024-04-25T01:39:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.876+0000 7fc00c395740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c57580 0x5595c5c619c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.876+0000 7fc00c395740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c57ea0 0x5595c5c79d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.876+0000 7fc00c395740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c62570 0x5595c5c7a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.876+0000 7fc00c395740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0041aea80 con 0x5595c5c62570 2024-04-25T01:39:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.876+0000 7fc00c395740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc0041b05c0 con 0x5595c5c57580 2024-04-25T01:39:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.876+0000 7fc00c395740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc0041af150 con 0x5595c5c57ea0 2024-04-25T01:39:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.876+0000 7fc0027fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c62570 0x5595c5c7a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.876+0000 7fc0027fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c62570 0x5595c5c7a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39934/0 (socket says 172.21.15.28:39934) 2024-04-25T01:39:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.876+0000 7fc0027fc640 1 -- 172.21.15.28:0/2664529786 learned_addr learned my addr 172.21.15.28:0/2664529786 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.876+0000 7fc002ffd640 1 --2- 172.21.15.28:0/2664529786 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c57ea0 0x5595c5c79d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.876+0000 7fc0027fc640 1 --2- 172.21.15.28:0/2664529786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c57580 0x5595c5c619c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.876+0000 7fc0027fc640 1 -- 172.21.15.28:0/2664529786 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c57ea0 msgr2=0x5595c5c79d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.876+0000 7fc0027fc640 1 --2- 172.21.15.28:0/2664529786 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c57ea0 0x5595c5c79d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.876+0000 7fc0027fc640 1 -- 172.21.15.28:0/2664529786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c57580 msgr2=0x5595c5c619c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.876+0000 7fc0027fc640 1 --2- 172.21.15.28:0/2664529786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c57580 0x5595c5c619c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.876+0000 7fc0027fc640 1 -- 172.21.15.28:0/2664529786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595c599a7d0 con 0x5595c5c62570 2024-04-25T01:39:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.877+0000 7fc0027fc640 1 --2- 172.21.15.28:0/2664529786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c62570 0x5595c5c7a2d0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fbff8009730 tx=0x7fbff8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b658d50e2acb0c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.877+0000 7fbfe8ff9640 1 -- 172.21.15.28:0/2664529786 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8027030 con 0x5595c5c62570 2024-04-25T01:39:46.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.877+0000 7fbfe8ff9640 1 -- 172.21.15.28:0/2664529786 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbff80217e0 con 0x5595c5c62570 2024-04-25T01:39:46.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.877+0000 7fbfe8ff9640 1 -- 172.21.15.28:0/2664529786 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8000e60 con 0x5595c5c62570 2024-04-25T01:39:46.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.877+0000 7fc00c395740 1 -- 172.21.15.28:0/2664529786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c62570 msgr2=0x5595c5c7a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.877+0000 7fc00c395740 1 --2- 172.21.15.28:0/2664529786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c62570 0x5595c5c7a2d0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fbff8009730 tx=0x7fbff8004730 comp rx=0 tx=0).stop 2024-04-25T01:39:46.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.877+0000 7fc00c395740 1 -- 172.21.15.28:0/2664529786 shutdown_connections 2024-04-25T01:39:46.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.877+0000 7fc00c395740 1 --2- 172.21.15.28:0/2664529786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c62570 0x5595c5c7a2d0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.877+0000 7fc00c395740 1 --2- 172.21.15.28:0/2664529786 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c57ea0 0x5595c5c79d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.877+0000 7fc00c395740 1 --2- 172.21.15.28:0/2664529786 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c57580 0x5595c5c619c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.877+0000 7fc00c395740 1 -- 172.21.15.28:0/2664529786 >> 172.21.15.28:0/2664529786 conn(0x5595c5c538a0 msgr2=0x5595c5c5e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.877+0000 7fc00c395740 1 -- 172.21.15.28:0/2664529786 shutdown_connections 2024-04-25T01:39:46.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.877+0000 7fc00c395740 1 -- 172.21.15.28:0/2664529786 wait complete. 2024-04-25T01:39:46.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc00c395740 1 -- start start 2024-04-25T01:39:46.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc00c395740 1 Processor -- start 2024-04-25T01:39:46.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc00c395740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c57580 0x5595c5c676d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc00c395740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 0x5595c5c67c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc00c395740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c62570 0x5595c5cd6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc00c395740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0041afef0 con 0x5595c5c57ea0 2024-04-25T01:39:46.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc00c395740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc0041af820 con 0x5595c5c62570 2024-04-25T01:39:46.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc00c395740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc0041aca60 con 0x5595c5c57580 2024-04-25T01:39:46.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc00c395740 10 client.0 _ll_register_callbacks cb 0x7fbff4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:46.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc002ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 0x5595c5c67c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc002ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 0x5595c5c67c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39946/0 (socket says 172.21.15.28:39946) 2024-04-25T01:39:46.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc002ffd640 1 -- 172.21.15.28:0/1495618420 learned_addr learned my addr 172.21.15.28:0/1495618420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:46.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc0027fc640 1 --2- 172.21.15.28:0/1495618420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c57580 0x5595c5c676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc0027fc640 1 --2- 172.21.15.28:0/1495618420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c62570 0x5595c5cd6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc002ffd640 1 -- 172.21.15.28:0/1495618420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c57580 msgr2=0x5595c5c676d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc002ffd640 1 --2- 172.21.15.28:0/1495618420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c57580 0x5595c5c676d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc002ffd640 1 -- 172.21.15.28:0/1495618420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c62570 msgr2=0x5595c5cd6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:46.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc002ffd640 1 --2- 172.21.15.28:0/1495618420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c62570 0x5595c5cd6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc002ffd640 1 -- 172.21.15.28:0/1495618420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595c5c684a0 con 0x5595c5c57ea0 2024-04-25T01:39:46.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.878+0000 7fc0027fc640 1 --2- 172.21.15.28:0/1495618420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c57580 0x5595c5c676d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:46.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.879+0000 7fc002ffd640 1 --2- 172.21.15.28:0/1495618420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 0x5595c5c67c10 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fbff402ae00 tx=0x7fbff4009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8abe1fb9e533b3a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:46.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.879+0000 7fbfe6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:46.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.879+0000 7fbfe6ff5640 1 -- 172.21.15.28:0/1495618420 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff4054070 con 0x5595c5c57ea0 2024-04-25T01:39:46.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.879+0000 7fbfe6ff5640 1 -- 172.21.15.28:0/1495618420 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fbff404c040 con 0x5595c5c57ea0 2024-04-25T01:39:46.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.879+0000 7fc00c395740 1 -- 172.21.15.28:0/1495618420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbff80217e0 con 0x5595c5c57ea0 2024-04-25T01:39:46.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.879+0000 7fbfe6ff5640 1 -- 172.21.15.28:0/1495618420 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbff40047d0 con 0x5595c5c57ea0 2024-04-25T01:39:46.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.879+0000 7fc00c395740 10 client.17955 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:46.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.879+0000 7fc00c395740 1 -- 172.21.15.28:0/1495618420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5595c5cd7c10 con 0x5595c5c57ea0 2024-04-25T01:39:46.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.879+0000 7fbfe6ff5640 1 -- 172.21.15.28:0/1495618420 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbff80217e0 con 0x5595c5c57ea0 2024-04-25T01:39:46.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.879+0000 7fbfe6ff5640 1 -- 172.21.15.28:0/1495618420 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbff4003cd0 con 0x5595c5c57ea0 2024-04-25T01:39:46.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.879+0000 7fc00c395740 10 client.17955 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:46.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.879+0000 7fc00c395740 10 client.17955 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:46.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.879+0000 7fc00c395740 20 client.17955 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:46.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.879+0000 7fc00c395740 1 --2- 172.21.15.28:0/1495618420 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5595c5cd8330 0x5595c5cda750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:46.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.879+0000 7fc00c395740 4 client.17955 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:46.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.879+0000 7fc00c395740 1 -- 172.21.15.28:0/1495618420 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5595c5bc3f00 con 0x5595c5cd8330 2024-04-25T01:39:46.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.880+0000 7fc0027fc640 1 --2- 172.21.15.28:0/1495618420 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5595c5cd8330 0x5595c5cda750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:46.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.880+0000 7fc0027fc640 1 --2- 172.21.15.28:0/1495618420 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5595c5cd8330 0x5595c5cda750 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df6cdfee4c133c1c server_cookie=dad52ef3d8a9df7d in_seq=0 out_seq=0 2024-04-25T01:39:46.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.880+0000 7fbfe6ff5640 10 client.17955 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:46.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.880+0000 7fbfe6ff5640 1 -- 172.21.15.28:0/1495618420 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5595c5bc3f00 con 0x5595c5cd8330 2024-04-25T01:39:46.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.880+0000 7fbfe6ff5640 10 client.17955 handle_command_reply: tid=0 2024-04-25T01:39:46.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.880+0000 7fc00c395740 1 client.17955 shutdown 2024-04-25T01:39:46.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.880+0000 7fc00c395740 1 -- 172.21.15.28:0/1495618420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 msgr2=0x5595c5c67c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.880+0000 7fc00c395740 1 --2- 172.21.15.28:0/1495618420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 0x5595c5c67c10 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fbff402ae00 tx=0x7fbff4009a20 comp rx=0 tx=0).stop 2024-04-25T01:39:46.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.880+0000 7fc00c395740 1 -- 172.21.15.28:0/1495618420 shutdown_connections 2024-04-25T01:39:46.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.880+0000 7fc00c395740 1 --2- 172.21.15.28:0/1495618420 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5595c5cd8330 0x5595c5cda750 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.880+0000 7fc00c395740 1 --2- 172.21.15.28:0/1495618420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c5c62570 0x5595c5cd6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.880+0000 7fc00c395740 1 --2- 172.21.15.28:0/1495618420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c5c57ea0 0x5595c5c67c10 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.880+0000 7fbfe6ff5640 0 client.17955 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:46.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.880+0000 7fc00c395740 1 --2- 172.21.15.28:0/1495618420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c5c57580 0x5595c5c676d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.880+0000 7fc00c395740 1 -- 172.21.15.28:0/1495618420 >> 172.21.15.28:0/1495618420 conn(0x5595c5c538a0 msgr2=0x5595c5c522b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.880+0000 7fc00c395740 1 -- 172.21.15.28:0/1495618420 shutdown_connections 2024-04-25T01:39:46.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.880+0000 7fc00c395740 1 -- 172.21.15.28:0/1495618420 wait complete. 2024-04-25T01:39:46.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.880+0000 7fc00c395740 20 client.17955 trim_cache size 0 max 16384 2024-04-25T01:39:46.878 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:46.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.881+0000 7fc009691640 1 -- 172.21.15.28:0/1241560050 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbfec0a2d00 msgr2=0x7fbfec0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.881+0000 7fc009691640 1 --2- 172.21.15.28:0/1241560050 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbfec0a2d00 0x7fbfec0a51c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0cc4a0 tx=0x7fbffc06d180 comp rx=0 tx=0).stop 2024-04-25T01:39:46.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.881+0000 7fc009691640 1 -- 172.21.15.28:0/1241560050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc004106630 msgr2=0x7fc00419c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:46.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.881+0000 7fc009691640 1 --2- 172.21.15.28:0/1241560050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc004106630 0x7fc00419c990 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fbffc00d950 tx=0x7fbffc00de10 comp rx=0 tx=0).stop 2024-04-25T01:39:46.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.881+0000 7fc009691640 1 -- 172.21.15.28:0/1241560050 shutdown_connections 2024-04-25T01:39:46.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.881+0000 7fc009691640 1 --2- 172.21.15.28:0/1241560050 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbfec0a2d00 0x7fbfec0a51c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.881+0000 7fc009691640 1 --2- 172.21.15.28:0/1241560050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc004106630 0x7fc00419c990 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.881+0000 7fc009691640 1 --2- 172.21.15.28:0/1241560050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc004100fe0 0x7fc004198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.881+0000 7fc009691640 1 --2- 172.21.15.28:0/1241560050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc004100620 0x7fc0041980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:46.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.881+0000 7fc009691640 1 -- 172.21.15.28:0/1241560050 >> 172.21.15.28:0/1241560050 conn(0x7fc0040776a0 msgr2=0x7fc00406a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:46.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.881+0000 7fc009691640 1 -- 172.21.15.28:0/1241560050 shutdown_connections 2024-04-25T01:39:46.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:46.881+0000 7fc009691640 1 -- 172.21.15.28:0/1241560050 wait complete. 2024-04-25T01:39:47.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:47 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2237886606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:39:47.464 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:47 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2237886606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:39:47.489 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:47 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2237886606' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:39:47.882 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:39:47.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:39:47.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:39:48.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.019+0000 7f7f7e116640 1 -- 172.21.15.28:0/962537405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f78102100 msgr2=0x7f7f7810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.019+0000 7f7f7e116640 1 --2- 172.21.15.28:0/962537405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f78102100 0x7f7f7810e610 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f7f70009cd0 tx=0x7f7f7002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.019+0000 7f7f7e116640 1 -- 172.21.15.28:0/962537405 shutdown_connections 2024-04-25T01:39:48.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.019+0000 7f7f7e116640 1 --2- 172.21.15.28:0/962537405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f78102100 0x7f7f7810e610 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.019+0000 7f7f7e116640 1 --2- 172.21.15.28:0/962537405 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f78101740 0x7f7f78101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.019+0000 7f7f7e116640 1 --2- 172.21.15.28:0/962537405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f78107750 0x7f7f78107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.019+0000 7f7f7e116640 1 -- 172.21.15.28:0/962537405 >> 172.21.15.28:0/962537405 conn(0x7f7f78078bd0 msgr2=0x7f7f78078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.019+0000 7f7f7e116640 1 -- 172.21.15.28:0/962537405 shutdown_connections 2024-04-25T01:39:48.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.019+0000 7f7f7e116640 1 -- 172.21.15.28:0/962537405 wait complete. 2024-04-25T01:39:48.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.019+0000 7f7f7e116640 1 Processor -- start 2024-04-25T01:39:48.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.020+0000 7f7f7e116640 1 -- start start 2024-04-25T01:39:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.020+0000 7f7f7e116640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f78101740 0x7f7f781980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.020+0000 7f7f7e116640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f78102100 0x7f7f78198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.020+0000 7f7f7e116640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f78107750 0x7f7f7819c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.020+0000 7f7f7e116640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f78111b60 con 0x7f7f78101740 2024-04-25T01:39:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.020+0000 7f7f7e116640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7f781119e0 con 0x7f7f78102100 2024-04-25T01:39:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.020+0000 7f7f7e116640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7f78111ce0 con 0x7f7f78107750 2024-04-25T01:39:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.020+0000 7f7f777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f78101740 0x7f7f781980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.020+0000 7f7f777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f78101740 0x7f7f781980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39960/0 (socket says 172.21.15.28:39960) 2024-04-25T01:39:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.020+0000 7f7f777fe640 1 -- 172.21.15.28:0/3261137630 learned_addr learned my addr 172.21.15.28:0/3261137630 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.020+0000 7f7f76ffd640 1 --2- 172.21.15.28:0/3261137630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f78102100 0x7f7f78198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.020+0000 7f7f77fff640 1 --2- 172.21.15.28:0/3261137630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f78107750 0x7f7f7819c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.020+0000 7f7f777fe640 1 -- 172.21.15.28:0/3261137630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f78107750 msgr2=0x7f7f7819c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.020+0000 7f7f777fe640 1 --2- 172.21.15.28:0/3261137630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f78107750 0x7f7f7819c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.020+0000 7f7f777fe640 1 -- 172.21.15.28:0/3261137630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f78102100 msgr2=0x7f7f78198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.020+0000 7f7f777fe640 1 --2- 172.21.15.28:0/3261137630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f78102100 0x7f7f78198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.020+0000 7f7f777fe640 1 -- 172.21.15.28:0/3261137630 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7819d0b0 con 0x7f7f78101740 2024-04-25T01:39:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.020+0000 7f7f77fff640 1 --2- 172.21.15.28:0/3261137630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f78107750 0x7f7f7819c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.020+0000 7f7f76ffd640 1 --2- 172.21.15.28:0/3261137630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f78102100 0x7f7f78198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.021+0000 7f7f777fe640 1 --2- 172.21.15.28:0/3261137630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f78101740 0x7f7f781980c0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f7f6800d950 tx=0x7f7f6800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.021+0000 7f7f74ff9640 1 -- 172.21.15.28:0/3261137630 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f68017070 con 0x7f7f78101740 2024-04-25T01:39:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.021+0000 7f7f7e116640 1 -- 172.21.15.28:0/3261137630 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f7819d3a0 con 0x7f7f78101740 2024-04-25T01:39:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.021+0000 7f7f74ff9640 1 -- 172.21.15.28:0/3261137630 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f68009c30 con 0x7f7f78101740 2024-04-25T01:39:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.021+0000 7f7f74ff9640 1 -- 172.21.15.28:0/3261137630 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6800f6b0 con 0x7f7f78101740 2024-04-25T01:39:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.021+0000 7f7f7e116640 1 -- 172.21.15.28:0/3261137630 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f781a4c30 con 0x7f7f78101740 2024-04-25T01:39:48.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.023+0000 7f7f74ff9640 1 -- 172.21.15.28:0/3261137630 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7f680109d0 con 0x7f7f78101740 2024-04-25T01:39:48.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f80e1c740 1 Processor -- start 2024-04-25T01:39:48.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f80e1c740 1 -- start start 2024-04-25T01:39:48.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f74ff9640 1 --2- 172.21.15.28:0/3261137630 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7f600a2d00 0x7f7f600a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f80e1c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d00980 0x5644f1cfecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f80e1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1cff290 0x5644f1cff710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f80e1c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1d0a010 0x5644f1d0c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f80e1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5644f1c632c0 con 0x5644f1cff290 2024-04-25T01:39:48.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f80e1c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5644f1c0b910 con 0x5644f1d00980 2024-04-25T01:39:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f80e1c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5644f1c5f280 con 0x5644f1d0a010 2024-04-25T01:39:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f77fff640 1 --2- 172.21.15.28:0/3261137630 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7f600a2d00 0x7f7f600a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f74ff9640 1 -- 172.21.15.28:0/3261137630 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f7f680cf3e0 con 0x7f7f78101740 2024-04-25T01:39:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f76ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1cff290 0x5644f1cff710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f76ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1cff290 0x5644f1cff710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39972/0 (socket says 172.21.15.28:39972) 2024-04-25T01:39:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f76ffd640 1 -- 172.21.15.28:0/1561545610 learned_addr learned my addr 172.21.15.28:0/1561545610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f777fe640 1 --2- 172.21.15.28:0/1561545610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d00980 0x5644f1cfecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f77fff640 1 --2- 172.21.15.28:0/1561545610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1d0a010 0x5644f1d0c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f76ffd640 1 -- 172.21.15.28:0/1561545610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1d0a010 msgr2=0x5644f1d0c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f76ffd640 1 --2- 172.21.15.28:0/1561545610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1d0a010 0x5644f1d0c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f76ffd640 1 -- 172.21.15.28:0/1561545610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d00980 msgr2=0x5644f1cfecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f76ffd640 1 --2- 172.21.15.28:0/1561545610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d00980 0x5644f1cfecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f76ffd640 1 -- 172.21.15.28:0/1561545610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644f1c1cfb0 con 0x5644f1cff290 2024-04-25T01:39:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.024+0000 7f7f77fff640 1 --2- 172.21.15.28:0/3261137630 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7f600a2d00 0x7f7f600a51c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7f70005e30 tx=0x7f7f700316e0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.025+0000 7f7f76ffd640 1 --2- 172.21.15.28:0/1561545610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1cff290 0x5644f1cff710 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c009c20 tx=0x7f7f6c00e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f0f80ea497ccfc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.025+0000 7f7f5dffb640 1 -- 172.21.15.28:0/1561545610 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c016070 con 0x5644f1cff290 2024-04-25T01:39:48.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.025+0000 7f7f5dffb640 1 -- 172.21.15.28:0/1561545610 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f6c009e50 con 0x5644f1cff290 2024-04-25T01:39:48.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.025+0000 7f7f5dffb640 1 -- 172.21.15.28:0/1561545610 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c0107f0 con 0x5644f1cff290 2024-04-25T01:39:48.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.025+0000 7f7f80e1c740 1 -- 172.21.15.28:0/1561545610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1cff290 msgr2=0x5644f1cff710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.025+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/1561545610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1cff290 0x5644f1cff710 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c009c20 tx=0x7f7f6c00e6c0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.025+0000 7f7f80e1c740 1 -- 172.21.15.28:0/1561545610 shutdown_connections 2024-04-25T01:39:48.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.025+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/1561545610 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1d0a010 0x5644f1d0c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.025+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/1561545610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1cff290 0x5644f1cff710 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.025+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/1561545610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d00980 0x5644f1cfecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.025+0000 7f7f80e1c740 1 -- 172.21.15.28:0/1561545610 >> 172.21.15.28:0/1561545610 conn(0x5644f1cc7950 msgr2=0x5644f1cfb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.025+0000 7f7f80e1c740 1 -- 172.21.15.28:0/1561545610 shutdown_connections 2024-04-25T01:39:48.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.025+0000 7f7f80e1c740 1 -- 172.21.15.28:0/1561545610 wait complete. 2024-04-25T01:39:48.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.026+0000 7f7f80e1c740 1 -- start start 2024-04-25T01:39:48.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.026+0000 7f7f80e1c740 1 Processor -- start 2024-04-25T01:39:48.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.026+0000 7f7f80e1c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1cff290 0x5644f1d06ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.026+0000 7f7f80e1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1d00980 0x5644f1d07410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.026+0000 7f7f80e1c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d0a010 0x5644f1d07950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.026+0000 7f7f80e1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5644f1d0cea0 con 0x5644f1d00980 2024-04-25T01:39:48.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.026+0000 7f7f80e1c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5644f1c1ee90 con 0x5644f1d0a010 2024-04-25T01:39:48.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.026+0000 7f7f80e1c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5644f1d0d020 con 0x5644f1cff290 2024-04-25T01:39:48.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.026+0000 7f7f80e1c740 10 client.0 _ll_register_callbacks cb 0x5644f1c5a970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:48.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.026+0000 7f7f76ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1d00980 0x5644f1d07410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.026+0000 7f7f76ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1d00980 0x5644f1d07410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39986/0 (socket says 172.21.15.28:39986) 2024-04-25T01:39:48.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.026+0000 7f7f76ffd640 1 -- 172.21.15.28:0/7365752 learned_addr learned my addr 172.21.15.28:0/7365752 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.027+0000 7f7f77fff640 1 --2- 172.21.15.28:0/7365752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d0a010 0x5644f1d07950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.027+0000 7f7f777fe640 1 --2- 172.21.15.28:0/7365752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1cff290 0x5644f1d06ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.027+0000 7f7f76ffd640 1 -- 172.21.15.28:0/7365752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1cff290 msgr2=0x5644f1d06ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.027+0000 7f7f76ffd640 1 --2- 172.21.15.28:0/7365752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1cff290 0x5644f1d06ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.027+0000 7f7f76ffd640 1 -- 172.21.15.28:0/7365752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d0a010 msgr2=0x5644f1d07950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.027+0000 7f7f76ffd640 1 --2- 172.21.15.28:0/7365752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d0a010 0x5644f1d07950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.027+0000 7f7f76ffd640 1 -- 172.21.15.28:0/7365752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644f1a3f7e0 con 0x5644f1d00980 2024-04-25T01:39:48.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.027+0000 7f7f76ffd640 1 --2- 172.21.15.28:0/7365752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1d00980 0x5644f1d07410 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c005880 tx=0x7f7f6c0058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b9663b24ea7850 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.027+0000 7f7f56ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:48.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.027+0000 7f7f56ffd640 1 -- 172.21.15.28:0/7365752 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c028060 con 0x5644f1d00980 2024-04-25T01:39:48.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.027+0000 7f7f56ffd640 1 -- 172.21.15.28:0/7365752 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f7f6c011670 con 0x5644f1d00980 2024-04-25T01:39:48.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.027+0000 7f7f80e1c740 1 -- 172.21.15.28:0/7365752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f6c009e50 con 0x5644f1d00980 2024-04-25T01:39:48.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.027+0000 7f7f56ffd640 1 -- 172.21.15.28:0/7365752 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7f6c02f050 con 0x5644f1d00980 2024-04-25T01:39:48.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.027+0000 7f7f80e1c740 10 client.17967 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:48.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.027+0000 7f7f80e1c740 1 -- 172.21.15.28:0/7365752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5644f1a9ff70 con 0x5644f1d00980 2024-04-25T01:39:48.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.028+0000 7f7f56ffd640 1 -- 172.21.15.28:0/7365752 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f6c009e50 con 0x5644f1d00980 2024-04-25T01:39:48.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.028+0000 7f7f56ffd640 1 -- 172.21.15.28:0/7365752 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7f6c00f040 con 0x5644f1d00980 2024-04-25T01:39:48.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.028+0000 7f7f80e1c740 10 client.17967 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:48.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.028+0000 7f7f80e1c740 10 client.17967 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:48.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.028+0000 7f7f80e1c740 20 client.17967 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:48.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.028+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/7365752 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5644f1d19fb0 0x5644f1d1c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.028+0000 7f7f80e1c740 4 client.17967 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:48.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.028+0000 7f7f80e1c740 1 -- 172.21.15.28:0/7365752 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644f1c375a0 con 0x5644f1d19fb0 2024-04-25T01:39:48.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.028+0000 7f7f777fe640 1 --2- 172.21.15.28:0/7365752 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5644f1d19fb0 0x5644f1d1c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.029+0000 7f7f777fe640 1 --2- 172.21.15.28:0/7365752 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5644f1d19fb0 0x5644f1d1c3a0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a90afc0d7c33fc0 server_cookie=139d049704bcee6b in_seq=0 out_seq=0 2024-04-25T01:39:48.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.029+0000 7f7f56ffd640 10 client.17967 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:48.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.029+0000 7f7f56ffd640 1 -- 172.21.15.28:0/7365752 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5644f1c375a0 con 0x5644f1d19fb0 2024-04-25T01:39:48.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.029+0000 7f7f56ffd640 10 client.17967 handle_command_reply: tid=0 2024-04-25T01:39:48.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.029+0000 7f7f80e1c740 1 client.17967 shutdown 2024-04-25T01:39:48.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.030+0000 7f7f80e1c740 1 -- 172.21.15.28:0/7365752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1d00980 msgr2=0x5644f1d07410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.030+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/7365752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1d00980 0x5644f1d07410 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c005880 tx=0x7f7f6c0058b0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.030+0000 7f7f80e1c740 1 -- 172.21.15.28:0/7365752 shutdown_connections 2024-04-25T01:39:48.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.030+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/7365752 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5644f1d19fb0 0x5644f1d1c3a0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.030+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/7365752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d0a010 0x5644f1d07950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.030+0000 7f7f56ffd640 0 client.17967 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:48.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.030+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/7365752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1d00980 0x5644f1d07410 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.030+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/7365752 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1cff290 0x5644f1d06ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.030+0000 7f7f80e1c740 1 -- 172.21.15.28:0/7365752 >> 172.21.15.28:0/7365752 conn(0x5644f1cc7950 msgr2=0x5644f1d01050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.030+0000 7f7f80e1c740 1 -- 172.21.15.28:0/7365752 shutdown_connections 2024-04-25T01:39:48.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.030+0000 7f7f80e1c740 1 -- 172.21.15.28:0/7365752 wait complete. 2024-04-25T01:39:48.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.030+0000 7f7f80e1c740 20 client.17967 trim_cache size 0 max 16384 2024-04-25T01:39:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.048+0000 7f7f80e1c740 1 Processor -- start 2024-04-25T01:39:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.048+0000 7f7f80e1c740 1 -- start start 2024-04-25T01:39:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.048+0000 7f7f80e1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1cff290 0x5644f1d15dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.048+0000 7f7f80e1c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d00980 0x5644f1d21970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.048+0000 7f7f80e1c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1d0a010 0x5644f1d21eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.048+0000 7f7f80e1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f781aeb20 con 0x5644f1cff290 2024-04-25T01:39:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.048+0000 7f7f80e1c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7f781b0660 con 0x5644f1d00980 2024-04-25T01:39:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.048+0000 7f7f80e1c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7f781af1f0 con 0x5644f1d0a010 2024-04-25T01:39:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.048+0000 7f7f777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1cff290 0x5644f1d15dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.048+0000 7f7f777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1cff290 0x5644f1d15dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39992/0 (socket says 172.21.15.28:39992) 2024-04-25T01:39:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.048+0000 7f7f777fe640 1 -- 172.21.15.28:0/1412106626 learned_addr learned my addr 172.21.15.28:0/1412106626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.048+0000 7f7f77fff640 1 --2- 172.21.15.28:0/1412106626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1d0a010 0x5644f1d21eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.048+0000 7f7f76ffd640 1 --2- 172.21.15.28:0/1412106626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d00980 0x5644f1d21970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.048+0000 7f7f777fe640 1 -- 172.21.15.28:0/1412106626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1d0a010 msgr2=0x5644f1d21eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.048+0000 7f7f777fe640 1 --2- 172.21.15.28:0/1412106626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1d0a010 0x5644f1d21eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.048+0000 7f7f777fe640 1 -- 172.21.15.28:0/1412106626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d00980 msgr2=0x5644f1d21970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.048+0000 7f7f777fe640 1 --2- 172.21.15.28:0/1412106626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d00980 0x5644f1d21970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.048+0000 7f7f777fe640 1 -- 172.21.15.28:0/1412106626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644f1937170 con 0x5644f1cff290 2024-04-25T01:39:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.049+0000 7f7f777fe640 1 --2- 172.21.15.28:0/1412106626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1cff290 0x5644f1d15dd0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f7f680cc850 tx=0x7f7f68005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56927188f66f4c00 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.051+0000 7f7f5d7fa640 1 -- 172.21.15.28:0/1412106626 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f68083050 con 0x5644f1cff290 2024-04-25T01:39:48.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.051+0000 7f7f5d7fa640 1 -- 172.21.15.28:0/1412106626 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f6809b5c0 con 0x5644f1cff290 2024-04-25T01:39:48.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.051+0000 7f7f5d7fa640 1 -- 172.21.15.28:0/1412106626 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f68020040 con 0x5644f1cff290 2024-04-25T01:39:48.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.052+0000 7f7f80e1c740 1 -- 172.21.15.28:0/1412106626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1cff290 msgr2=0x5644f1d15dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.052+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/1412106626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1cff290 0x5644f1d15dd0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f7f680cc850 tx=0x7f7f68005f90 comp rx=0 tx=0).stop 2024-04-25T01:39:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.052+0000 7f7f80e1c740 1 -- 172.21.15.28:0/1412106626 shutdown_connections 2024-04-25T01:39:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.052+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/1412106626 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1d0a010 0x5644f1d21eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.052+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/1412106626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d00980 0x5644f1d21970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.052+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/1412106626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1cff290 0x5644f1d15dd0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.052+0000 7f7f80e1c740 1 -- 172.21.15.28:0/1412106626 >> 172.21.15.28:0/1412106626 conn(0x5644f1cc7950 msgr2=0x5644f1cfa950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.052+0000 7f7f80e1c740 1 -- 172.21.15.28:0/1412106626 shutdown_connections 2024-04-25T01:39:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.052+0000 7f7f80e1c740 1 -- 172.21.15.28:0/1412106626 wait complete. 2024-04-25T01:39:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.052+0000 7f7f80e1c740 1 -- start start 2024-04-25T01:39:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.052+0000 7f7f80e1c740 1 Processor -- start 2024-04-25T01:39:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.053+0000 7f7f80e1c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1cff290 0x5644f1d127c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.053+0000 7f7f80e1c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1d00980 0x5644f1d042c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.053+0000 7f7f80e1c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d0a010 0x5644f1d04840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.053+0000 7f7f80e1c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f781aff90 con 0x5644f1d00980 2024-04-25T01:39:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.053+0000 7f7f80e1c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7f781af8c0 con 0x5644f1d0a010 2024-04-25T01:39:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.053+0000 7f7f80e1c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7f781acb00 con 0x5644f1cff290 2024-04-25T01:39:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.053+0000 7f7f80e1c740 10 client.0 _ll_register_callbacks cb 0x7f7f6c009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.053+0000 7f7f76ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1d00980 0x5644f1d042c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.053+0000 7f7f76ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1d00980 0x5644f1d042c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39996/0 (socket says 172.21.15.28:39996) 2024-04-25T01:39:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.053+0000 7f7f76ffd640 1 -- 172.21.15.28:0/1003170223 learned_addr learned my addr 172.21.15.28:0/1003170223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.053+0000 7f7f77fff640 1 --2- 172.21.15.28:0/1003170223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d0a010 0x5644f1d04840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.053+0000 7f7f777fe640 1 --2- 172.21.15.28:0/1003170223 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1cff290 0x5644f1d127c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.053+0000 7f7f76ffd640 1 -- 172.21.15.28:0/1003170223 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1cff290 msgr2=0x5644f1d127c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.053+0000 7f7f76ffd640 1 --2- 172.21.15.28:0/1003170223 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1cff290 0x5644f1d127c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.053+0000 7f7f76ffd640 1 -- 172.21.15.28:0/1003170223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d0a010 msgr2=0x5644f1d04840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.053+0000 7f7f76ffd640 1 --2- 172.21.15.28:0/1003170223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d0a010 0x5644f1d04840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.053+0000 7f7f76ffd640 1 -- 172.21.15.28:0/1003170223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644f1d03160 con 0x5644f1d00980 2024-04-25T01:39:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.053+0000 7f7f76ffd640 1 --2- 172.21.15.28:0/1003170223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1d00980 0x5644f1d042c0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c005850 tx=0x7f7f6c009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1be8df8dbbb6a754 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.053+0000 7f7f56ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.054+0000 7f7f56ffd640 1 -- 172.21.15.28:0/1003170223 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c028070 con 0x5644f1d00980 2024-04-25T01:39:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.054+0000 7f7f80e1c740 1 -- 172.21.15.28:0/1003170223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f6809b5c0 con 0x5644f1d00980 2024-04-25T01:39:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.054+0000 7f7f56ffd640 1 -- 172.21.15.28:0/1003170223 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f7f6c0109c0 con 0x5644f1d00980 2024-04-25T01:39:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.054+0000 7f7f56ffd640 1 -- 172.21.15.28:0/1003170223 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7f6c016070 con 0x5644f1d00980 2024-04-25T01:39:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.054+0000 7f7f80e1c740 10 client.17973 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.054+0000 7f7f80e1c740 1 -- 172.21.15.28:0/1003170223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5644f1d85840 con 0x5644f1d00980 2024-04-25T01:39:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.054+0000 7f7f56ffd640 1 -- 172.21.15.28:0/1003170223 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f6809b5c0 con 0x5644f1d00980 2024-04-25T01:39:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.054+0000 7f7f56ffd640 1 -- 172.21.15.28:0/1003170223 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7f6c011470 con 0x5644f1d00980 2024-04-25T01:39:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.054+0000 7f7f80e1c740 10 client.17973 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.054+0000 7f7f80e1c740 10 client.17973 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.054+0000 7f7f80e1c740 20 client.17973 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.054+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/1003170223 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5644f1d85e90 0x5644f1d88280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.054+0000 7f7f80e1c740 4 client.17973 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:39:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.054+0000 7f7f80e1c740 1 -- 172.21.15.28:0/1003170223 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5644f1c70800 con 0x5644f1d85e90 2024-04-25T01:39:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.054+0000 7f7f777fe640 1 --2- 172.21.15.28:0/1003170223 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5644f1d85e90 0x5644f1d88280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.055+0000 7f7f777fe640 1 --2- 172.21.15.28:0/1003170223 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5644f1d85e90 0x5644f1d88280 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=966d7f4d83cd578 server_cookie=864d16d1c4f8225c in_seq=0 out_seq=0 2024-04-25T01:39:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.055+0000 7f7f56ffd640 10 client.17973 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.056+0000 7f7f56ffd640 1 -- 172.21.15.28:0/1003170223 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5644f1c70800 con 0x5644f1d85e90 2024-04-25T01:39:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.056+0000 7f7f56ffd640 10 client.17973 handle_command_reply: tid=0 2024-04-25T01:39:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.056+0000 7f7f80e1c740 1 client.17973 shutdown 2024-04-25T01:39:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f80e1c740 1 -- 172.21.15.28:0/1003170223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1d00980 msgr2=0x5644f1d042c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/1003170223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1d00980 0x5644f1d042c0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c005850 tx=0x7f7f6c009ed0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f80e1c740 1 -- 172.21.15.28:0/1003170223 shutdown_connections 2024-04-25T01:39:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/1003170223 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5644f1d85e90 0x5644f1d88280 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/1003170223 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5644f1d0a010 0x5644f1d04840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/1003170223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5644f1d00980 0x5644f1d042c0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f80e1c740 1 --2- 172.21.15.28:0/1003170223 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5644f1cff290 0x5644f1d127c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f56ffd640 0 client.17973 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f80e1c740 1 -- 172.21.15.28:0/1003170223 >> 172.21.15.28:0/1003170223 conn(0x5644f1cc7950 msgr2=0x5644f1d00d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f80e1c740 1 -- 172.21.15.28:0/1003170223 shutdown_connections 2024-04-25T01:39:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f80e1c740 1 -- 172.21.15.28:0/1003170223 wait complete. 2024-04-25T01:39:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f80e1c740 20 client.17973 trim_cache size 0 max 16384 2024-04-25T01:39:48.044 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:39:48.044 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:39:48.044 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "0eeeb4c2-7c4a-4144-9e6d-cbcf092705bf", 2024-04-25T01:39:48.044 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:39:48.044 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:39:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f7e116640 1 -- 172.21.15.28:0/3261137630 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7f600a2d00 msgr2=0x7f7f600a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f7e116640 1 --2- 172.21.15.28:0/3261137630 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7f600a2d00 0x7f7f600a51c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7f70005e30 tx=0x7f7f700316e0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f7e116640 1 -- 172.21.15.28:0/3261137630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f78101740 msgr2=0x7f7f781980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f7e116640 1 --2- 172.21.15.28:0/3261137630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f78101740 0x7f7f781980c0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f7f6800d950 tx=0x7f7f6800de10 comp rx=0 tx=0).stop 2024-04-25T01:39:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f7e116640 1 -- 172.21.15.28:0/3261137630 shutdown_connections 2024-04-25T01:39:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f7e116640 1 --2- 172.21.15.28:0/3261137630 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7f600a2d00 0x7f7f600a51c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f7e116640 1 --2- 172.21.15.28:0/3261137630 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7f78107750 0x7f7f7819c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f7e116640 1 --2- 172.21.15.28:0/3261137630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7f78102100 0x7f7f78198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f7e116640 1 --2- 172.21.15.28:0/3261137630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f78101740 0x7f7f781980c0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.057+0000 7f7f7e116640 1 -- 172.21.15.28:0/3261137630 >> 172.21.15.28:0/3261137630 conn(0x7f7f78078bd0 msgr2=0x7f7f780ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.058+0000 7f7f7e116640 1 -- 172.21.15.28:0/3261137630 shutdown_connections 2024-04-25T01:39:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.058+0000 7f7f7e116640 1 -- 172.21.15.28:0/3261137630 wait complete. 2024-04-25T01:39:48.052 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:39:48.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:39:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.173+0000 7efede770640 1 -- 172.21.15.28:0/1215625486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8100620 msgr2=0x7efed8100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.173+0000 7efede770640 1 --2- 172.21.15.28:0/1215625486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8100620 0x7efed8100aa0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7efecc009930 tx=0x7efecc02a420 comp rx=0 tx=0).stop 2024-04-25T01:39:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.173+0000 7efede770640 1 -- 172.21.15.28:0/1215625486 shutdown_connections 2024-04-25T01:39:48.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.173+0000 7efede770640 1 --2- 172.21.15.28:0/1215625486 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efed8100fe0 0x7efed810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.173+0000 7efede770640 1 --2- 172.21.15.28:0/1215625486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8100620 0x7efed8100aa0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.173+0000 7efede770640 1 --2- 172.21.15.28:0/1215625486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed8106630 0x7efed8106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.173+0000 7efede770640 1 -- 172.21.15.28:0/1215625486 >> 172.21.15.28:0/1215625486 conn(0x7efed80776a0 msgr2=0x7efed8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.173+0000 7efede770640 1 -- 172.21.15.28:0/1215625486 shutdown_connections 2024-04-25T01:39:48.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.173+0000 7efede770640 1 -- 172.21.15.28:0/1215625486 wait complete. 2024-04-25T01:39:48.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.173+0000 7efede770640 1 Processor -- start 2024-04-25T01:39:48.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.173+0000 7efede770640 1 -- start start 2024-04-25T01:39:48.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efede770640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed8100620 0x7efed8198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efede770640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efed8100fe0 0x7efed81993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efede770640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8106630 0x7efed81a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efede770640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efed8110a40 con 0x7efed8106630 2024-04-25T01:39:48.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efede770640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efed81108c0 con 0x7efed8100620 2024-04-25T01:39:48.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efede770640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7efed8110bc0 con 0x7efed8100fe0 2024-04-25T01:39:48.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efedcd0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8106630 0x7efed81a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efedcd0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8106630 0x7efed81a3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40022/0 (socket says 172.21.15.28:40022) 2024-04-25T01:39:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efedcd0d640 1 -- 172.21.15.28:0/1563983563 learned_addr learned my addr 172.21.15.28:0/1563983563 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efed7fff640 1 --2- 172.21.15.28:0/1563983563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed8100620 0x7efed8198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efed77fe640 1 --2- 172.21.15.28:0/1563983563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efed8100fe0 0x7efed81993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efedcd0d640 1 -- 172.21.15.28:0/1563983563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efed8100fe0 msgr2=0x7efed81993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efedcd0d640 1 --2- 172.21.15.28:0/1563983563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efed8100fe0 0x7efed81993a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efedcd0d640 1 -- 172.21.15.28:0/1563983563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed8100620 msgr2=0x7efed8198e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efedcd0d640 1 --2- 172.21.15.28:0/1563983563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed8100620 0x7efed8198e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efedcd0d640 1 -- 172.21.15.28:0/1563983563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed81a4420 con 0x7efed8106630 2024-04-25T01:39:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efed77fe640 1 --2- 172.21.15.28:0/1563983563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efed8100fe0 0x7efed81993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efedcd0d640 1 --2- 172.21.15.28:0/1563983563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8106630 0x7efed81a3ee0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7efed000cad0 tx=0x7efed000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efed57fa640 1 -- 172.21.15.28:0/1563983563 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed000ae20 con 0x7efed8106630 2024-04-25T01:39:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efed57fa640 1 -- 172.21.15.28:0/1563983563 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efed0005020 con 0x7efed8106630 2024-04-25T01:39:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efede770640 1 -- 172.21.15.28:0/1563983563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efed81a45f0 con 0x7efed8106630 2024-04-25T01:39:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efed57fa640 1 -- 172.21.15.28:0/1563983563 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed000f6f0 con 0x7efed8106630 2024-04-25T01:39:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.174+0000 7efede770640 1 -- 172.21.15.28:0/1563983563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efed81a4b40 con 0x7efed8106630 2024-04-25T01:39:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.177+0000 7efed57fa640 1 -- 172.21.15.28:0/1563983563 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7efed00051d0 con 0x7efed8106630 2024-04-25T01:39:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.177+0000 7efee1484740 1 Processor -- start 2024-04-25T01:39:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.177+0000 7efee1484740 1 -- start start 2024-04-25T01:39:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.177+0000 7efee1484740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015630 0x55c4a4015a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.177+0000 7efee1484740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4015f50 0x55c4a40200e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.177+0000 7efee1484740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4a4020620 0x55c4a4022a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.177+0000 7efee1484740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c4a3f7a6b0 con 0x55c4a4015630 2024-04-25T01:39:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.177+0000 7efee1484740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c4a3f22880 con 0x55c4a4020620 2024-04-25T01:39:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.177+0000 7efee1484740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c4a3f76670 con 0x55c4a4015f50 2024-04-25T01:39:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.178+0000 7efed77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015630 0x55c4a4015a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.178+0000 7efed77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015630 0x55c4a4015a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40026/0 (socket says 172.21.15.28:40026) 2024-04-25T01:39:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.178+0000 7efed77fe640 1 -- 172.21.15.28:0/3062207002 learned_addr learned my addr 172.21.15.28:0/3062207002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.178+0000 7efed57fa640 1 --2- 172.21.15.28:0/1563983563 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7efec00a2d00 0x7efec00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.178+0000 7efed7fff640 1 --2- 172.21.15.28:0/3062207002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4015f50 0x55c4a40200e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.178+0000 7efedcd0d640 1 --2- 172.21.15.28:0/1563983563 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7efec00a2d00 0x7efec00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.178+0000 7efed77fe640 1 --2- 172.21.15.28:0/3062207002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4a4020620 0x55c4a4022a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.178+0000 7efed77fe640 1 -- 172.21.15.28:0/3062207002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4015f50 msgr2=0x55c4a40200e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.178+0000 7efed77fe640 1 --2- 172.21.15.28:0/3062207002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4015f50 0x55c4a40200e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.178+0000 7efed77fe640 1 -- 172.21.15.28:0/3062207002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4a4020620 msgr2=0x55c4a4022a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.178+0000 7efed57fa640 1 -- 172.21.15.28:0/1563983563 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7efed00cfaf0 con 0x7efed8106630 2024-04-25T01:39:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.178+0000 7efed77fe640 1 --2- 172.21.15.28:0/3062207002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4a4020620 0x55c4a4022a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.178+0000 7efed77fe640 1 -- 172.21.15.28:0/3062207002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4a3d587d0 con 0x55c4a4015630 2024-04-25T01:39:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.178+0000 7efed77fe640 1 --2- 172.21.15.28:0/3062207002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015630 0x55c4a4015a10 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7efecc005c20 tx=0x7efecc005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18db67407542354b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.178+0000 7efedcd0d640 1 --2- 172.21.15.28:0/1563983563 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7efec00a2d00 0x7efec00a51c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efed0098130 tx=0x7efed0061040 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.178+0000 7efebe7fc640 1 -- 172.21.15.28:0/3062207002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efecc035050 con 0x55c4a4015630 2024-04-25T01:39:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.178+0000 7efebe7fc640 1 -- 172.21.15.28:0/3062207002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efecc0317b0 con 0x55c4a4015630 2024-04-25T01:39:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.178+0000 7efebe7fc640 1 -- 172.21.15.28:0/3062207002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efecc0046c0 con 0x55c4a4015630 2024-04-25T01:39:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.179+0000 7efee1484740 1 -- 172.21.15.28:0/3062207002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015630 msgr2=0x55c4a4015a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.179+0000 7efee1484740 1 --2- 172.21.15.28:0/3062207002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015630 0x55c4a4015a10 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7efecc005c20 tx=0x7efecc005bb0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.179+0000 7efee1484740 1 -- 172.21.15.28:0/3062207002 shutdown_connections 2024-04-25T01:39:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.179+0000 7efee1484740 1 --2- 172.21.15.28:0/3062207002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4a4020620 0x55c4a4022a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.179+0000 7efee1484740 1 --2- 172.21.15.28:0/3062207002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4015f50 0x55c4a40200e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.179+0000 7efee1484740 1 --2- 172.21.15.28:0/3062207002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015630 0x55c4a4015a10 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.179+0000 7efee1484740 1 -- 172.21.15.28:0/3062207002 >> 172.21.15.28:0/3062207002 conn(0x55c4a4011950 msgr2=0x55c4a4011d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.179+0000 7efee1484740 1 -- 172.21.15.28:0/3062207002 shutdown_connections 2024-04-25T01:39:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.179+0000 7efee1484740 1 -- 172.21.15.28:0/3062207002 wait complete. 2024-04-25T01:39:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.179+0000 7efee1484740 1 -- start start 2024-04-25T01:39:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efee1484740 1 Processor -- start 2024-04-25T01:39:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efee1484740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4015630 0x55c4a401d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efee1484740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015f50 0x55c4a401dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efee1484740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4a4020620 0x55c4a401e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efee1484740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c4a40233b0 con 0x55c4a4015f50 2024-04-25T01:39:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efee1484740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c4a3f35f50 con 0x55c4a4020620 2024-04-25T01:39:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efee1484740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c4a4023530 con 0x55c4a4015630 2024-04-25T01:39:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efee1484740 10 client.0 _ll_register_callbacks cb 0x55c4a3f71000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efed7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015f50 0x55c4a401dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efed7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015f50 0x55c4a401dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40034/0 (socket says 172.21.15.28:40034) 2024-04-25T01:39:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efed7fff640 1 -- 172.21.15.28:0/230383747 learned_addr learned my addr 172.21.15.28:0/230383747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efed77fe640 1 --2- 172.21.15.28:0/230383747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4015630 0x55c4a401d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efed77fe640 1 --2- 172.21.15.28:0/230383747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4a4020620 0x55c4a401e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efed7fff640 1 -- 172.21.15.28:0/230383747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4015630 msgr2=0x55c4a401d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efed7fff640 1 --2- 172.21.15.28:0/230383747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4015630 0x55c4a401d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efed7fff640 1 -- 172.21.15.28:0/230383747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4a4020620 msgr2=0x55c4a401e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efed7fff640 1 --2- 172.21.15.28:0/230383747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4a4020620 0x55c4a401e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efed7fff640 1 -- 172.21.15.28:0/230383747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4a3c8e3f0 con 0x55c4a4015f50 2024-04-25T01:39:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efed77fe640 1 --2- 172.21.15.28:0/230383747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4a4020620 0x55c4a401e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efed7fff640 1 --2- 172.21.15.28:0/230383747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015f50 0x55c4a401dad0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7efec800bf40 tx=0x7efec80118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd619d8971d859d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.180+0000 7efeb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.181+0000 7efeb77fe640 1 -- 172.21.15.28:0/230383747 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec8004020 con 0x55c4a4015f50 2024-04-25T01:39:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.181+0000 7efeb77fe640 1 -- 172.21.15.28:0/230383747 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7efec80041c0 con 0x55c4a4015f50 2024-04-25T01:39:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.181+0000 7efee1484740 1 -- 172.21.15.28:0/230383747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efecc0317b0 con 0x55c4a4015f50 2024-04-25T01:39:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.181+0000 7efeb77fe640 1 -- 172.21.15.28:0/230383747 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7efec8024050 con 0x55c4a4015f50 2024-04-25T01:39:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.181+0000 7efee1484740 10 client.17985 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.181+0000 7efee1484740 1 -- 172.21.15.28:0/230383747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c4a3c4e0f0 con 0x55c4a4015f50 2024-04-25T01:39:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.181+0000 7efeb77fe640 1 -- 172.21.15.28:0/230383747 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efecc0317b0 con 0x55c4a4015f50 2024-04-25T01:39:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.181+0000 7efeb77fe640 1 -- 172.21.15.28:0/230383747 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7efec8004c60 con 0x55c4a4015f50 2024-04-25T01:39:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.181+0000 7efee1484740 10 client.17985 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.181+0000 7efee1484740 10 client.17985 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.181+0000 7efee1484740 20 client.17985 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.181+0000 7efee1484740 1 --2- 172.21.15.28:0/230383747 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c4a4030910 0x55c4a4032d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.181+0000 7efee1484740 4 client.17985 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.181+0000 7efee1484740 1 -- 172.21.15.28:0/230383747 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4a3f87bf0 con 0x55c4a4030910 2024-04-25T01:39:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.182+0000 7efed77fe640 1 --2- 172.21.15.28:0/230383747 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c4a4030910 0x55c4a4032d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.182+0000 7efed77fe640 1 --2- 172.21.15.28:0/230383747 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c4a4030910 0x55c4a4032d30 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db7ae676d578f505 server_cookie=ba6cff39aa1f412f in_seq=0 out_seq=0 2024-04-25T01:39:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.182+0000 7efeb77fe640 10 client.17985 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.183+0000 7efeb77fe640 1 -- 172.21.15.28:0/230383747 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c4a3f87bf0 con 0x55c4a4030910 2024-04-25T01:39:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.183+0000 7efeb77fe640 10 client.17985 handle_command_reply: tid=0 2024-04-25T01:39:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.183+0000 7efee1484740 1 client.17985 shutdown 2024-04-25T01:39:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.183+0000 7efee1484740 1 -- 172.21.15.28:0/230383747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015f50 msgr2=0x55c4a401dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.183+0000 7efee1484740 1 --2- 172.21.15.28:0/230383747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015f50 0x55c4a401dad0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7efec800bf40 tx=0x7efec80118d0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.183+0000 7efee1484740 1 -- 172.21.15.28:0/230383747 shutdown_connections 2024-04-25T01:39:48.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.183+0000 7efee1484740 1 --2- 172.21.15.28:0/230383747 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c4a4030910 0x55c4a4032d30 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.183+0000 7efee1484740 1 --2- 172.21.15.28:0/230383747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4a4020620 0x55c4a401e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.183+0000 7efee1484740 1 --2- 172.21.15.28:0/230383747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015f50 0x55c4a401dad0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.183+0000 7efeb77fe640 0 client.17985 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:48.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.183+0000 7efee1484740 1 --2- 172.21.15.28:0/230383747 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4015630 0x55c4a401d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.183+0000 7efee1484740 1 -- 172.21.15.28:0/230383747 >> 172.21.15.28:0/230383747 conn(0x55c4a4011950 msgr2=0x55c4a4017750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.183+0000 7efee1484740 1 -- 172.21.15.28:0/230383747 shutdown_connections 2024-04-25T01:39:48.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.183+0000 7efee1484740 1 -- 172.21.15.28:0/230383747 wait complete. 2024-04-25T01:39:48.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.183+0000 7efee1484740 20 client.17985 trim_cache size 0 max 16384 2024-04-25T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.200+0000 7efee1484740 1 Processor -- start 2024-04-25T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efee1484740 1 -- start start 2024-04-25T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efee1484740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4a4015630 0x55c4a401fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efee1484740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015f50 0x55c4a4010af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efee1484740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4020620 0x55c4a401de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efee1484740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efed81aeb00 con 0x55c4a4015f50 2024-04-25T01:39:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efee1484740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efed81b0640 con 0x55c4a4015630 2024-04-25T01:39:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efee1484740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7efed81af1d0 con 0x55c4a4020620 2024-04-25T01:39:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efed7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015f50 0x55c4a4010af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efed7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015f50 0x55c4a4010af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40040/0 (socket says 172.21.15.28:40040) 2024-04-25T01:39:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efed7fff640 1 -- 172.21.15.28:0/4014126647 learned_addr learned my addr 172.21.15.28:0/4014126647 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efed77fe640 1 --2- 172.21.15.28:0/4014126647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4020620 0x55c4a401de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efed7fff640 1 -- 172.21.15.28:0/4014126647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4020620 msgr2=0x55c4a401de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efed7fff640 1 --2- 172.21.15.28:0/4014126647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4020620 0x55c4a401de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efed7fff640 1 -- 172.21.15.28:0/4014126647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4a4015630 msgr2=0x55c4a401fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efed7fff640 1 --2- 172.21.15.28:0/4014126647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4a4015630 0x55c4a401fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efed7fff640 1 -- 172.21.15.28:0/4014126647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4a4019170 con 0x55c4a4015f50 2024-04-25T01:39:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efed77fe640 1 --2- 172.21.15.28:0/4014126647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4020620 0x55c4a401de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efed7fff640 1 --2- 172.21.15.28:0/4014126647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015f50 0x55c4a4010af0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7efec8010890 tx=0x7efec8011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3ea5bc98eebffb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efebdffb640 1 -- 172.21.15.28:0/4014126647 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec802c070 con 0x55c4a4015f50 2024-04-25T01:39:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efebdffb640 1 -- 172.21.15.28:0/4014126647 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efec8010df0 con 0x55c4a4015f50 2024-04-25T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.201+0000 7efebdffb640 1 -- 172.21.15.28:0/4014126647 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec80048a0 con 0x55c4a4015f50 2024-04-25T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.202+0000 7efee1484740 1 -- 172.21.15.28:0/4014126647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015f50 msgr2=0x55c4a4010af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.202+0000 7efee1484740 1 --2- 172.21.15.28:0/4014126647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015f50 0x55c4a4010af0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7efec8010890 tx=0x7efec8011dd0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.202+0000 7efee1484740 1 -- 172.21.15.28:0/4014126647 shutdown_connections 2024-04-25T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.202+0000 7efee1484740 1 --2- 172.21.15.28:0/4014126647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4020620 0x55c4a401de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.202+0000 7efee1484740 1 --2- 172.21.15.28:0/4014126647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4015f50 0x55c4a4010af0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.202+0000 7efee1484740 1 --2- 172.21.15.28:0/4014126647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4a4015630 0x55c4a401fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.202+0000 7efee1484740 1 -- 172.21.15.28:0/4014126647 >> 172.21.15.28:0/4014126647 conn(0x55c4a4011950 msgr2=0x55c4a4016d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.202+0000 7efee1484740 1 -- 172.21.15.28:0/4014126647 shutdown_connections 2024-04-25T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.202+0000 7efee1484740 1 -- 172.21.15.28:0/4014126647 wait complete. 2024-04-25T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.202+0000 7efee1484740 1 -- start start 2024-04-25T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.203+0000 7efee1484740 1 Processor -- start 2024-04-25T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.203+0000 7efee1484740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4015630 0x55c4a4024370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.203+0000 7efee1484740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4a4015f50 0x55c4a40248b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.203+0000 7efee1484740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4020620 0x55c4a4024df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.203+0000 7efee1484740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efed81aff70 con 0x55c4a4020620 2024-04-25T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.203+0000 7efee1484740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efed81af8a0 con 0x55c4a4015f50 2024-04-25T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.203+0000 7efee1484740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7efed81acae0 con 0x55c4a4015630 2024-04-25T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.203+0000 7efee1484740 10 client.0 _ll_register_callbacks cb 0x7efec800bb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.203+0000 7efed77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4020620 0x55c4a4024df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.203+0000 7efed77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4020620 0x55c4a4024df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40050/0 (socket says 172.21.15.28:40050) 2024-04-25T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.203+0000 7efed77fe640 1 -- 172.21.15.28:0/2895706154 learned_addr learned my addr 172.21.15.28:0/2895706154 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.203+0000 7efed77fe640 1 -- 172.21.15.28:0/2895706154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4015630 msgr2=0x55c4a4024370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.203+0000 7efed77fe640 1 --2- 172.21.15.28:0/2895706154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4015630 0x55c4a4024370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.203+0000 7efed77fe640 1 -- 172.21.15.28:0/2895706154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4a4015f50 msgr2=0x55c4a40248b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.203+0000 7efed77fe640 1 --2- 172.21.15.28:0/2895706154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4a4015f50 0x55c4a40248b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.203+0000 7efed77fe640 1 -- 172.21.15.28:0/2895706154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4a4015d80 con 0x55c4a4020620 2024-04-25T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.203+0000 7efed77fe640 1 --2- 172.21.15.28:0/2895706154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4020620 0x55c4a4024df0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7efecc005c20 tx=0x7efecc02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826c26bb0b17f96 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.203+0000 7efeb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.203+0000 7efeb77fe640 1 -- 172.21.15.28:0/2895706154 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efecc055060 con 0x55c4a4020620 2024-04-25T01:39:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.204+0000 7efeb77fe640 1 -- 172.21.15.28:0/2895706154 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7efecc004250 con 0x55c4a4020620 2024-04-25T01:39:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.204+0000 7efee1484740 1 -- 172.21.15.28:0/2895706154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efec8010df0 con 0x55c4a4020620 2024-04-25T01:39:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.204+0000 7efeb77fe640 1 -- 172.21.15.28:0/2895706154 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7efecc035040 con 0x55c4a4020620 2024-04-25T01:39:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.204+0000 7efee1484740 10 client.17991 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.204+0000 7efee1484740 1 -- 172.21.15.28:0/2895706154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c4a40970f0 con 0x55c4a4020620 2024-04-25T01:39:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.204+0000 7efeb77fe640 1 -- 172.21.15.28:0/2895706154 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efec8010df0 con 0x55c4a4020620 2024-04-25T01:39:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.204+0000 7efeb77fe640 1 -- 172.21.15.28:0/2895706154 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7efecc031940 con 0x55c4a4020620 2024-04-25T01:39:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.204+0000 7efee1484740 10 client.17991 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.204+0000 7efee1484740 10 client.17991 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.204+0000 7efee1484740 20 client.17991 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.204+0000 7efee1484740 1 --2- 172.21.15.28:0/2895706154 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c4a4097460 0x55c4a4099850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.204+0000 7efee1484740 4 client.17991 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:39:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.204+0000 7efee1484740 1 -- 172.21.15.28:0/2895706154 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c4a3f81f40 con 0x55c4a4097460 2024-04-25T01:39:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.204+0000 7efed7fff640 1 --2- 172.21.15.28:0/2895706154 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c4a4097460 0x55c4a4099850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.205+0000 7efed7fff640 1 --2- 172.21.15.28:0/2895706154 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c4a4097460 0x55c4a4099850 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d279b9c9215667a server_cookie=4114690841a9d18 in_seq=0 out_seq=0 2024-04-25T01:39:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.205+0000 7efeb77fe640 10 client.17991 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.205+0000 7efeb77fe640 1 -- 172.21.15.28:0/2895706154 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c4a3f81f40 con 0x55c4a4097460 2024-04-25T01:39:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.205+0000 7efeb77fe640 10 client.17991 handle_command_reply: tid=0 2024-04-25T01:39:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.205+0000 7efee1484740 1 client.17991 shutdown 2024-04-25T01:39:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.205+0000 7efee1484740 1 -- 172.21.15.28:0/2895706154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4020620 msgr2=0x55c4a4024df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.205+0000 7efee1484740 1 --2- 172.21.15.28:0/2895706154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4020620 0x55c4a4024df0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7efecc005c20 tx=0x7efecc02a420 comp rx=0 tx=0).stop 2024-04-25T01:39:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.205+0000 7efee1484740 1 -- 172.21.15.28:0/2895706154 shutdown_connections 2024-04-25T01:39:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.205+0000 7efee1484740 1 --2- 172.21.15.28:0/2895706154 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c4a4097460 0x55c4a4099850 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.205+0000 7efee1484740 1 --2- 172.21.15.28:0/2895706154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4a4020620 0x55c4a4024df0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.205+0000 7efee1484740 1 --2- 172.21.15.28:0/2895706154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4a4015f50 0x55c4a40248b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.205+0000 7efee1484740 1 --2- 172.21.15.28:0/2895706154 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c4a4015630 0x55c4a4024370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.205+0000 7efeb77fe640 0 client.17991 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.205+0000 7efee1484740 1 -- 172.21.15.28:0/2895706154 >> 172.21.15.28:0/2895706154 conn(0x55c4a4011950 msgr2=0x55c4a4010970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.205+0000 7efee1484740 1 -- 172.21.15.28:0/2895706154 shutdown_connections 2024-04-25T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.205+0000 7efee1484740 1 -- 172.21.15.28:0/2895706154 wait complete. 2024-04-25T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.205+0000 7efee1484740 20 client.17991 trim_cache size 0 max 16384 2024-04-25T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:39:48.195 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:39:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.206+0000 7efede770640 1 -- 172.21.15.28:0/1563983563 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7efec00a2d00 msgr2=0x7efec00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.206+0000 7efede770640 1 --2- 172.21.15.28:0/1563983563 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7efec00a2d00 0x7efec00a51c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efed0098130 tx=0x7efed0061040 comp rx=0 tx=0).stop 2024-04-25T01:39:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.206+0000 7efede770640 1 -- 172.21.15.28:0/1563983563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8106630 msgr2=0x7efed81a3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.206+0000 7efede770640 1 --2- 172.21.15.28:0/1563983563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8106630 0x7efed81a3ee0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7efed000cad0 tx=0x7efed000cf90 comp rx=0 tx=0).stop 2024-04-25T01:39:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.206+0000 7efede770640 1 -- 172.21.15.28:0/1563983563 shutdown_connections 2024-04-25T01:39:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.206+0000 7efede770640 1 --2- 172.21.15.28:0/1563983563 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7efec00a2d00 0x7efec00a51c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.206+0000 7efede770640 1 --2- 172.21.15.28:0/1563983563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efed8106630 0x7efed81a3ee0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.206+0000 7efede770640 1 --2- 172.21.15.28:0/1563983563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efed8100fe0 0x7efed81993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.206+0000 7efede770640 1 --2- 172.21.15.28:0/1563983563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efed8100620 0x7efed8198e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.206+0000 7efede770640 1 -- 172.21.15.28:0/1563983563 >> 172.21.15.28:0/1563983563 conn(0x7efed80776a0 msgr2=0x7efed81031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.206+0000 7efede770640 1 -- 172.21.15.28:0/1563983563 shutdown_connections 2024-04-25T01:39:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.206+0000 7efede770640 1 -- 172.21.15.28:0/1563983563 wait complete. 2024-04-25T01:39:48.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:48 smithi028 ceph-mon[29390]: pgmap v206: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:39:48.200 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:39:48.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:39:48.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.319+0000 7fa28c363640 1 -- 172.21.15.28:0/1686854126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa284100300 msgr2=0x7fa2841006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.319+0000 7fa28c363640 1 --2- 172.21.15.28:0/1686854126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa284100300 0x7fa2841006e0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fa280009900 tx=0x7fa28002a410 comp rx=0 tx=0).stop 2024-04-25T01:39:48.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.319+0000 7fa28c363640 1 -- 172.21.15.28:0/1686854126 shutdown_connections 2024-04-25T01:39:48.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.319+0000 7fa28c363640 1 --2- 172.21.15.28:0/1686854126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa284106ac0 0x7fa28410e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.319+0000 7fa28c363640 1 --2- 172.21.15.28:0/1686854126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa284100cb0 0x7fa2841063e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.319+0000 7fa28c363640 1 --2- 172.21.15.28:0/1686854126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa284100300 0x7fa2841006e0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.319+0000 7fa28c363640 1 -- 172.21.15.28:0/1686854126 >> 172.21.15.28:0/1686854126 conn(0x7fa284075920 msgr2=0x7fa284075d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.319+0000 7fa28c363640 1 -- 172.21.15.28:0/1686854126 shutdown_connections 2024-04-25T01:39:48.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.319+0000 7fa28c363640 1 -- 172.21.15.28:0/1686854126 wait complete. 2024-04-25T01:39:48.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa28c363640 1 Processor -- start 2024-04-25T01:39:48.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa28c363640 1 -- start start 2024-04-25T01:39:48.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa28c363640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa284100300 0x7fa2841980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa28c363640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa284106ac0 0x7fa2841985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa28c363640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa28419c970 0x7fa28419ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa28c363640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa284111b90 con 0x7fa284100300 2024-04-25T01:39:48.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa28c363640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa284111a10 con 0x7fa28419c970 2024-04-25T01:39:48.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa28c363640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa284111d10 con 0x7fa284106ac0 2024-04-25T01:39:48.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa28a0ff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa284100300 0x7fa2841980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa28a0ff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa284100300 0x7fa2841980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40058/0 (socket says 172.21.15.28:40058) 2024-04-25T01:39:48.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa28a0ff640 1 -- 172.21.15.28:0/3900255422 learned_addr learned my addr 172.21.15.28:0/3900255422 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa28a900640 1 --2- 172.21.15.28:0/3900255422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa28419c970 0x7fa28419ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa28a0ff640 1 -- 172.21.15.28:0/3900255422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa284106ac0 msgr2=0x7fa2841985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa2898fe640 1 --2- 172.21.15.28:0/3900255422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa284106ac0 0x7fa2841985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa28a0ff640 1 --2- 172.21.15.28:0/3900255422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa284106ac0 0x7fa2841985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa28a0ff640 1 -- 172.21.15.28:0/3900255422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa28419c970 msgr2=0x7fa28419ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa28a0ff640 1 --2- 172.21.15.28:0/3900255422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa28419c970 0x7fa28419ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa28a0ff640 1 -- 172.21.15.28:0/3900255422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa28419d460 con 0x7fa284100300 2024-04-25T01:39:48.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa2898fe640 1 --2- 172.21.15.28:0/3900255422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa284106ac0 0x7fa2841985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:48.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.320+0000 7fa28a900640 1 --2- 172.21.15.28:0/3900255422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa28419c970 0x7fa28419ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:48.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.321+0000 7fa28a0ff640 1 --2- 172.21.15.28:0/3900255422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa284100300 0x7fa2841980b0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fa280006010 tx=0x7fa2800042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.321+0000 7fa27b7fe640 1 -- 172.21.15.28:0/3900255422 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28003f030 con 0x7fa284100300 2024-04-25T01:39:48.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.321+0000 7fa28c363640 1 -- 172.21.15.28:0/3900255422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa28419d6f0 con 0x7fa284100300 2024-04-25T01:39:48.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.321+0000 7fa27b7fe640 1 -- 172.21.15.28:0/3900255422 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa280003ce0 con 0x7fa284100300 2024-04-25T01:39:48.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.321+0000 7fa27b7fe640 1 -- 172.21.15.28:0/3900255422 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa280035070 con 0x7fa284100300 2024-04-25T01:39:48.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.321+0000 7fa28c363640 1 -- 172.21.15.28:0/3900255422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2841a4f10 con 0x7fa284100300 2024-04-25T01:39:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.322+0000 7fa28c363640 1 -- 172.21.15.28:0/3900255422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2840689b0 con 0x7fa284100300 2024-04-25T01:39:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.323+0000 7fa27b7fe640 1 -- 172.21.15.28:0/3900255422 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa280042030 con 0x7fa284100300 2024-04-25T01:39:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.323+0000 7fa27b7fe640 1 --2- 172.21.15.28:0/3900255422 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa2700a2d00 0x7fa2700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.324+0000 7fa2898fe640 1 --2- 172.21.15.28:0/3900255422 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa2700a2d00 0x7fa2700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.324+0000 7fa27b7fe640 1 -- 172.21.15.28:0/3900255422 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fa2800efd50 con 0x7fa284100300 2024-04-25T01:39:48.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.324+0000 7fa2898fe640 1 --2- 172.21.15.28:0/3900255422 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa2700a2d00 0x7fa2700a51c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa2840ffcf0 tx=0x7fa274005e50 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.328+0000 7fa27b7fe640 1 -- 172.21.15.28:0/3900255422 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa2800f3030 con 0x7fa284100300 2024-04-25T01:39:48.464 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:48 smithi130 ceph-mon[32702]: pgmap v206: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:39:48.489 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:48 smithi032 ceph-mon[32461]: pgmap v206: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:39:48.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.552+0000 7fa28c363640 1 -- 172.21.15.28:0/3900255422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa284109c70 con 0x7fa284100300 2024-04-25T01:39:48.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.553+0000 7fa27b7fe640 1 -- 172.21.15.28:0/3900255422 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7fa2800bc6f0 con 0x7fa284100300 2024-04-25T01:39:48.537 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:39:48.537 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:39:48.538 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:39:48.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.555+0000 7fa28c363640 1 -- 172.21.15.28:0/3900255422 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa2700a2d00 msgr2=0x7fa2700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.555+0000 7fa28c363640 1 --2- 172.21.15.28:0/3900255422 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa2700a2d00 0x7fa2700a51c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa2840ffcf0 tx=0x7fa274005e50 comp rx=0 tx=0).stop 2024-04-25T01:39:48.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.555+0000 7fa28c363640 1 -- 172.21.15.28:0/3900255422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa284100300 msgr2=0x7fa2841980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.555+0000 7fa28c363640 1 --2- 172.21.15.28:0/3900255422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa284100300 0x7fa2841980b0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fa280006010 tx=0x7fa2800042f0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.555+0000 7fa28c363640 1 -- 172.21.15.28:0/3900255422 shutdown_connections 2024-04-25T01:39:48.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.555+0000 7fa28c363640 1 --2- 172.21.15.28:0/3900255422 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa2700a2d00 0x7fa2700a51c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.555+0000 7fa28c363640 1 --2- 172.21.15.28:0/3900255422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa28419c970 0x7fa28419ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.555+0000 7fa28c363640 1 --2- 172.21.15.28:0/3900255422 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa284106ac0 0x7fa2841985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.555+0000 7fa28c363640 1 --2- 172.21.15.28:0/3900255422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa284100300 0x7fa2841980b0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.555+0000 7fa28c363640 1 -- 172.21.15.28:0/3900255422 >> 172.21.15.28:0/3900255422 conn(0x7fa284075920 msgr2=0x7fa2840fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.555+0000 7fa28c363640 1 -- 172.21.15.28:0/3900255422 shutdown_connections 2024-04-25T01:39:48.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.555+0000 7fa28c363640 1 -- 172.21.15.28:0/3900255422 wait complete. 2024-04-25T01:39:48.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:39:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.671+0000 7fb897458640 1 -- 172.21.15.28:0/3667505166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb890102100 msgr2=0x7fb89010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.671+0000 7fb897458640 1 --2- 172.21.15.28:0/3667505166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb890102100 0x7fb89010e610 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fb888009cd0 tx=0x7fb88802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.671+0000 7fb897458640 1 -- 172.21.15.28:0/3667505166 shutdown_connections 2024-04-25T01:39:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.671+0000 7fb897458640 1 --2- 172.21.15.28:0/3667505166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb890102100 0x7fb89010e610 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.671+0000 7fb897458640 1 --2- 172.21.15.28:0/3667505166 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb890101740 0x7fb890101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.671+0000 7fb897458640 1 --2- 172.21.15.28:0/3667505166 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb890107750 0x7fb890107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.671+0000 7fb897458640 1 -- 172.21.15.28:0/3667505166 >> 172.21.15.28:0/3667505166 conn(0x7fb890078bd0 msgr2=0x7fb890078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.671+0000 7fb897458640 1 -- 172.21.15.28:0/3667505166 shutdown_connections 2024-04-25T01:39:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.671+0000 7fb897458640 1 -- 172.21.15.28:0/3667505166 wait complete. 2024-04-25T01:39:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.672+0000 7fb897458640 1 Processor -- start 2024-04-25T01:39:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.672+0000 7fb897458640 1 -- start start 2024-04-25T01:39:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.672+0000 7fb897458640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb890101740 0x7fb890198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.672+0000 7fb897458640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb890102100 0x7fb890198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.672+0000 7fb897458640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb890107750 0x7fb89019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.672+0000 7fb897458640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb890111b70 con 0x7fb890101740 2024-04-25T01:39:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.672+0000 7fb897458640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb8901119f0 con 0x7fb890107750 2024-04-25T01:39:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.672+0000 7fb897458640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb890111cf0 con 0x7fb890102100 2024-04-25T01:39:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.672+0000 7fb8951f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb890101740 0x7fb890198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.672+0000 7fb8951f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb890101740 0x7fb890198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40088/0 (socket says 172.21.15.28:40088) 2024-04-25T01:39:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.672+0000 7fb8951f4640 1 -- 172.21.15.28:0/2782538111 learned_addr learned my addr 172.21.15.28:0/2782538111 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.672+0000 7fb8949f3640 1 --2- 172.21.15.28:0/2782538111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb890102100 0x7fb890198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.672+0000 7fb8959f5640 1 --2- 172.21.15.28:0/2782538111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb890107750 0x7fb89019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.672+0000 7fb8951f4640 1 -- 172.21.15.28:0/2782538111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb890102100 msgr2=0x7fb890198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.672+0000 7fb8951f4640 1 --2- 172.21.15.28:0/2782538111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb890102100 0x7fb890198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.672+0000 7fb8951f4640 1 -- 172.21.15.28:0/2782538111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb890107750 msgr2=0x7fb89019c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.672+0000 7fb8951f4640 1 --2- 172.21.15.28:0/2782538111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb890107750 0x7fb89019c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.672+0000 7fb8951f4640 1 -- 172.21.15.28:0/2782538111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb89019d000 con 0x7fb890101740 2024-04-25T01:39:48.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.672+0000 7fb8959f5640 1 --2- 172.21.15.28:0/2782538111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb890107750 0x7fb89019c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:48.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.673+0000 7fb8951f4640 1 --2- 172.21.15.28:0/2782538111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb890101740 0x7fb890198010 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00e9f0 tx=0x7fb88c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.673+0000 7fb8867fc640 1 -- 172.21.15.28:0/2782538111 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c00cd80 con 0x7fb890101740 2024-04-25T01:39:48.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.673+0000 7fb8867fc640 1 -- 172.21.15.28:0/2782538111 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb88c010910 con 0x7fb890101740 2024-04-25T01:39:48.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.673+0000 7fb8867fc640 1 -- 172.21.15.28:0/2782538111 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c018990 con 0x7fb890101740 2024-04-25T01:39:48.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.673+0000 7fb897458640 1 -- 172.21.15.28:0/2782538111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb89019d2f0 con 0x7fb890101740 2024-04-25T01:39:48.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.673+0000 7fb897458640 1 -- 172.21.15.28:0/2782538111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8901a4b80 con 0x7fb890101740 2024-04-25T01:39:48.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.675+0000 7fb8867fc640 1 -- 172.21.15.28:0/2782538111 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb88c010ac0 con 0x7fb890101740 2024-04-25T01:39:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.675+0000 7fb89a166740 1 Processor -- start 2024-04-25T01:39:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.675+0000 7fb89a166740 1 -- start start 2024-04-25T01:39:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.675+0000 7fb89a166740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd75f480 0x564abd75f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb89a166740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 0x564abd76e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb89a166740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd76e980 0x564abd770d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb89a166740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564abd6c4670 con 0x564abd75fda0 2024-04-25T01:39:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb89a166740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564abd66c970 con 0x564abd75f480 2024-04-25T01:39:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb89a166740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x564abd6c0630 con 0x564abd76e980 2024-04-25T01:39:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb8867fc640 1 --2- 172.21.15.28:0/2782538111 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb87c0a2d80 0x7fb87c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb8951f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 0x564abd76e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb8951f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 0x564abd76e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40092/0 (socket says 172.21.15.28:40092) 2024-04-25T01:39:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb8951f4640 1 -- 172.21.15.28:0/2577656821 learned_addr learned my addr 172.21.15.28:0/2577656821 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb8959f5640 1 --2- 172.21.15.28:0/2782538111 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb87c0a2d80 0x7fb87c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb8949f3640 1 --2- 172.21.15.28:0/2577656821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd76e980 0x564abd770d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb8959f5640 1 --2- 172.21.15.28:0/2577656821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd75f480 0x564abd75f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb8867fc640 1 -- 172.21.15.28:0/2782538111 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fb88c0cfb20 con 0x7fb890101740 2024-04-25T01:39:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb8951f4640 1 -- 172.21.15.28:0/2577656821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd76e980 msgr2=0x564abd770d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb8951f4640 1 --2- 172.21.15.28:0/2577656821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd76e980 0x564abd770d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb8951f4640 1 -- 172.21.15.28:0/2577656821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd75f480 msgr2=0x564abd75f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb8951f4640 1 --2- 172.21.15.28:0/2577656821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd75f480 0x564abd75f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb8951f4640 1 -- 172.21.15.28:0/2577656821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564abd690400 con 0x564abd75fda0 2024-04-25T01:39:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb8959f5640 1 --2- 172.21.15.28:0/2577656821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd75f480 0x564abd75f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb8951f4640 1 --2- 172.21.15.28:0/2577656821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 0x564abd76e440 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fb88c0cce40 tx=0x7fb88c09b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a4ccc0d869e9317 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb8959f5640 1 --2- 172.21.15.28:0/2782538111 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb87c0a2d80 0x7fb87c0a5240 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb888004af0 tx=0x7fb888004a40 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.676+0000 7fb87b7fe640 1 -- 172.21.15.28:0/2577656821 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c0a0050 con 0x564abd75fda0 2024-04-25T01:39:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.677+0000 7fb87b7fe640 1 -- 172.21.15.28:0/2577656821 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb88c0a4910 con 0x564abd75fda0 2024-04-25T01:39:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.677+0000 7fb87b7fe640 1 -- 172.21.15.28:0/2577656821 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c018d70 con 0x564abd75fda0 2024-04-25T01:39:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.677+0000 7fb89a166740 1 -- 172.21.15.28:0/2577656821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 msgr2=0x564abd76e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.677+0000 7fb89a166740 1 --2- 172.21.15.28:0/2577656821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 0x564abd76e440 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fb88c0cce40 tx=0x7fb88c09b6c0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.677+0000 7fb89a166740 1 -- 172.21.15.28:0/2577656821 shutdown_connections 2024-04-25T01:39:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.677+0000 7fb89a166740 1 --2- 172.21.15.28:0/2577656821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd76e980 0x564abd770d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.677+0000 7fb89a166740 1 --2- 172.21.15.28:0/2577656821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 0x564abd76e440 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.677+0000 7fb89a166740 1 --2- 172.21.15.28:0/2577656821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd75f480 0x564abd75f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.677+0000 7fb89a166740 1 -- 172.21.15.28:0/2577656821 >> 172.21.15.28:0/2577656821 conn(0x564abd75b8a0 msgr2=0x564abd75bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.677+0000 7fb89a166740 1 -- 172.21.15.28:0/2577656821 shutdown_connections 2024-04-25T01:39:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.677+0000 7fb89a166740 1 -- 172.21.15.28:0/2577656821 wait complete. 2024-04-25T01:39:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.678+0000 7fb89a166740 1 -- start start 2024-04-25T01:39:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.678+0000 7fb89a166740 1 Processor -- start 2024-04-25T01:39:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.678+0000 7fb89a166740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd75f480 0x564abd76e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.678+0000 7fb89a166740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 0x564abd76b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.678+0000 7fb89a166740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd76e980 0x564abd76ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.678+0000 7fb89a166740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564abd771740 con 0x564abd75fda0 2024-04-25T01:39:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.678+0000 7fb89a166740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564abd680050 con 0x564abd76e980 2024-04-25T01:39:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.678+0000 7fb89a166740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x564abd7718c0 con 0x564abd75f480 2024-04-25T01:39:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.678+0000 7fb89a166740 10 client.0 _ll_register_callbacks cb 0x564abd3d7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.678+0000 7fb8949f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 0x564abd76b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.678+0000 7fb8949f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 0x564abd76b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40108/0 (socket says 172.21.15.28:40108) 2024-04-25T01:39:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.678+0000 7fb8949f3640 1 -- 172.21.15.28:0/3565606203 learned_addr learned my addr 172.21.15.28:0/3565606203 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.678+0000 7fb8951f4640 1 --2- 172.21.15.28:0/3565606203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd75f480 0x564abd76e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.678+0000 7fb8959f5640 1 --2- 172.21.15.28:0/3565606203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd76e980 0x564abd76ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.678+0000 7fb8949f3640 1 -- 172.21.15.28:0/3565606203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd75f480 msgr2=0x564abd76e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.678+0000 7fb8949f3640 1 --2- 172.21.15.28:0/3565606203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd75f480 0x564abd76e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.678+0000 7fb8949f3640 1 -- 172.21.15.28:0/3565606203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd76e980 msgr2=0x564abd76ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.678+0000 7fb8949f3640 1 --2- 172.21.15.28:0/3565606203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd76e980 0x564abd76ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.678+0000 7fb8949f3640 1 -- 172.21.15.28:0/3565606203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564abd651dc0 con 0x564abd75fda0 2024-04-25T01:39:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.679+0000 7fb8949f3640 1 --2- 172.21.15.28:0/3565606203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 0x564abd76b530 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fb880009dc0 tx=0x7fb8800117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f921aa4e3f65e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.679+0000 7fb878ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.679+0000 7fb878ff9640 1 -- 172.21.15.28:0/3565606203 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb880012860 con 0x564abd75fda0 2024-04-25T01:39:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.679+0000 7fb89a166740 1 -- 172.21.15.28:0/3565606203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb88c0a4910 con 0x564abd75fda0 2024-04-25T01:39:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.679+0000 7fb878ff9640 1 -- 172.21.15.28:0/3565606203 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fb880012a00 con 0x564abd75fda0 2024-04-25T01:39:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.679+0000 7fb878ff9640 1 -- 172.21.15.28:0/3565606203 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb880025050 con 0x564abd75fda0 2024-04-25T01:39:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.679+0000 7fb89a166740 10 client.18009 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.679+0000 7fb89a166740 1 -- 172.21.15.28:0/3565606203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564abd3d83f0 con 0x564abd75fda0 2024-04-25T01:39:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.679+0000 7fb878ff9640 1 -- 172.21.15.28:0/3565606203 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb88c0a4910 con 0x564abd75fda0 2024-04-25T01:39:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.679+0000 7fb878ff9640 1 -- 172.21.15.28:0/3565606203 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb88000f040 con 0x564abd75fda0 2024-04-25T01:39:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.679+0000 7fb89a166740 10 client.18009 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.679+0000 7fb89a166740 10 client.18009 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:39:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.679+0000 7fb89a166740 20 client.18009 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.679+0000 7fb89a166740 1 --2- 172.21.15.28:0/3565606203 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564abd77eb10 0x564abd780f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.679+0000 7fb89a166740 4 client.18009 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.679+0000 7fb89a166740 1 -- 172.21.15.28:0/3565606203 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564abd6d1bb0 con 0x564abd77eb10 2024-04-25T01:39:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.680+0000 7fb8951f4640 1 --2- 172.21.15.28:0/3565606203 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564abd77eb10 0x564abd780f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.680+0000 7fb8951f4640 1 --2- 172.21.15.28:0/3565606203 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564abd77eb10 0x564abd780f30 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b00ec2a9f1e404ec server_cookie=fa988d548071fef in_seq=0 out_seq=0 2024-04-25T01:39:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.680+0000 7fb878ff9640 10 client.18009 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:39:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.681+0000 7fb878ff9640 1 -- 172.21.15.28:0/3565606203 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564abd6d1bb0 con 0x564abd77eb10 2024-04-25T01:39:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.681+0000 7fb878ff9640 10 client.18009 handle_command_reply: tid=0 2024-04-25T01:39:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.681+0000 7fb89a166740 1 client.18009 shutdown 2024-04-25T01:39:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.681+0000 7fb89a166740 1 -- 172.21.15.28:0/3565606203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 msgr2=0x564abd76b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.681+0000 7fb89a166740 1 --2- 172.21.15.28:0/3565606203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 0x564abd76b530 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fb880009dc0 tx=0x7fb8800117b0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.681+0000 7fb89a166740 1 -- 172.21.15.28:0/3565606203 shutdown_connections 2024-04-25T01:39:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.681+0000 7fb89a166740 1 --2- 172.21.15.28:0/3565606203 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564abd77eb10 0x564abd780f30 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.681+0000 7fb89a166740 1 --2- 172.21.15.28:0/3565606203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd76e980 0x564abd76ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.681+0000 7fb878ff9640 0 client.18009 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:39:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.681+0000 7fb89a166740 1 --2- 172.21.15.28:0/3565606203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 0x564abd76b530 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.681+0000 7fb89a166740 1 --2- 172.21.15.28:0/3565606203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd75f480 0x564abd76e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.681+0000 7fb89a166740 1 -- 172.21.15.28:0/3565606203 >> 172.21.15.28:0/3565606203 conn(0x564abd75b8a0 msgr2=0x564abd75ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.681+0000 7fb89a166740 1 -- 172.21.15.28:0/3565606203 shutdown_connections 2024-04-25T01:39:48.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.681+0000 7fb89a166740 1 -- 172.21.15.28:0/3565606203 wait complete. 2024-04-25T01:39:48.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.681+0000 7fb89a166740 20 client.18009 trim_cache size 0 max 16384 2024-04-25T01:39:48.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.695+0000 7fb89a166740 1 Processor -- start 2024-04-25T01:39:48.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.695+0000 7fb89a166740 1 -- start start 2024-04-25T01:39:48.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.695+0000 7fb89a166740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd75f480 0x564abd761600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.695+0000 7fb89a166740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 0x564abd76dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.695+0000 7fb89a166740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd76e980 0x564abd786800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.695+0000 7fb89a166740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb8901ac830 con 0x564abd75fda0 2024-04-25T01:39:48.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.695+0000 7fb89a166740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb8901ae370 con 0x564abd76e980 2024-04-25T01:39:48.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.695+0000 7fb89a166740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb8901acf00 con 0x564abd75f480 2024-04-25T01:39:48.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.695+0000 7fb8949f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 0x564abd76dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.695+0000 7fb8949f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 0x564abd76dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40118/0 (socket says 172.21.15.28:40118) 2024-04-25T01:39:48.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.695+0000 7fb8949f3640 1 -- 172.21.15.28:0/1083070974 learned_addr learned my addr 172.21.15.28:0/1083070974 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.695+0000 7fb8949f3640 1 -- 172.21.15.28:0/1083070974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd75f480 msgr2=0x564abd761600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.695+0000 7fb8949f3640 1 --2- 172.21.15.28:0/1083070974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd75f480 0x564abd761600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.695+0000 7fb8949f3640 1 -- 172.21.15.28:0/1083070974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd76e980 msgr2=0x564abd786800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.695+0000 7fb8949f3640 1 --2- 172.21.15.28:0/1083070974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd76e980 0x564abd786800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.695+0000 7fb8949f3640 1 -- 172.21.15.28:0/1083070974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564abd4a27d0 con 0x564abd75fda0 2024-04-25T01:39:48.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.695+0000 7fb8949f3640 1 --2- 172.21.15.28:0/1083070974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 0x564abd76dd70 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fb880009d90 tx=0x7fb880009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ecdf2416a814556 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.695+0000 7fb87affd640 1 -- 172.21.15.28:0/1083070974 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb880005230 con 0x564abd75fda0 2024-04-25T01:39:48.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.696+0000 7fb87affd640 1 -- 172.21.15.28:0/1083070974 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb88002d070 con 0x564abd75fda0 2024-04-25T01:39:48.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.696+0000 7fb87affd640 1 -- 172.21.15.28:0/1083070974 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb880012660 con 0x564abd75fda0 2024-04-25T01:39:48.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.696+0000 7fb89a166740 1 -- 172.21.15.28:0/1083070974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 msgr2=0x564abd76dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.696+0000 7fb89a166740 1 --2- 172.21.15.28:0/1083070974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 0x564abd76dd70 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fb880009d90 tx=0x7fb880009eb0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.696+0000 7fb89a166740 1 -- 172.21.15.28:0/1083070974 shutdown_connections 2024-04-25T01:39:48.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.696+0000 7fb89a166740 1 --2- 172.21.15.28:0/1083070974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd76e980 0x564abd786800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.696+0000 7fb89a166740 1 --2- 172.21.15.28:0/1083070974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd75fda0 0x564abd76dd70 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.696+0000 7fb89a166740 1 --2- 172.21.15.28:0/1083070974 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd75f480 0x564abd761600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.696+0000 7fb89a166740 1 -- 172.21.15.28:0/1083070974 >> 172.21.15.28:0/1083070974 conn(0x564abd75b8a0 msgr2=0x564abd76b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.696+0000 7fb89a166740 1 -- 172.21.15.28:0/1083070974 shutdown_connections 2024-04-25T01:39:48.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.696+0000 7fb89a166740 1 -- 172.21.15.28:0/1083070974 wait complete. 2024-04-25T01:39:48.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.696+0000 7fb89a166740 1 -- start start 2024-04-25T01:39:48.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.697+0000 7fb89a166740 1 Processor -- start 2024-04-25T01:39:48.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.697+0000 7fb89a166740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd75f480 0x564abd772870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.697+0000 7fb89a166740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd75fda0 0x564abd772db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.697+0000 7fb89a166740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd76e980 0x564abd7732f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.697+0000 7fb89a166740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb8901adca0 con 0x564abd76e980 2024-04-25T01:39:48.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.697+0000 7fb89a166740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb8901ad5d0 con 0x564abd75fda0 2024-04-25T01:39:48.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.697+0000 7fb89a166740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb8901aa810 con 0x564abd75f480 2024-04-25T01:39:48.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.697+0000 7fb89a166740 10 client.0 _ll_register_callbacks cb 0x7fb880009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:48.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.697+0000 7fb8959f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd76e980 0x564abd7732f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.697+0000 7fb8959f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd76e980 0x564abd7732f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40132/0 (socket says 172.21.15.28:40132) 2024-04-25T01:39:48.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.697+0000 7fb8959f5640 1 -- 172.21.15.28:0/3631145627 learned_addr learned my addr 172.21.15.28:0/3631145627 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.697+0000 7fb8949f3640 1 --2- 172.21.15.28:0/3631145627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd75fda0 0x564abd772db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.697+0000 7fb8951f4640 1 --2- 172.21.15.28:0/3631145627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd75f480 0x564abd772870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.697+0000 7fb8959f5640 1 -- 172.21.15.28:0/3631145627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd75f480 msgr2=0x564abd772870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.697+0000 7fb8959f5640 1 --2- 172.21.15.28:0/3631145627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd75f480 0x564abd772870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.697+0000 7fb8959f5640 1 -- 172.21.15.28:0/3631145627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd75fda0 msgr2=0x564abd772db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.697+0000 7fb8959f5640 1 --2- 172.21.15.28:0/3631145627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd75fda0 0x564abd772db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.697+0000 7fb8959f5640 1 -- 172.21.15.28:0/3631145627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564abd7e4310 con 0x564abd76e980 2024-04-25T01:39:48.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.697+0000 7fb8951f4640 1 --2- 172.21.15.28:0/3631145627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd75f480 0x564abd772870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:48.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.698+0000 7fb8959f5640 1 --2- 172.21.15.28:0/3631145627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd76e980 0x564abd7732f0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fb888035000 tx=0x7fb888009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a15882146e288878 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.698+0000 7fb878ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:48.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.698+0000 7fb878ff9640 1 -- 172.21.15.28:0/3631145627 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb888004040 con 0x564abd76e980 2024-04-25T01:39:48.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.698+0000 7fb878ff9640 1 -- 172.21.15.28:0/3631145627 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fb8880041e0 con 0x564abd76e980 2024-04-25T01:39:48.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.698+0000 7fb89a166740 1 -- 172.21.15.28:0/3631145627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb88002d070 con 0x564abd76e980 2024-04-25T01:39:48.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.698+0000 7fb878ff9640 1 -- 172.21.15.28:0/3631145627 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb8880043e0 con 0x564abd76e980 2024-04-25T01:39:48.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.698+0000 7fb89a166740 10 client.18015 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:48.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.698+0000 7fb89a166740 1 -- 172.21.15.28:0/3631145627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564abd7e5610 con 0x564abd76e980 2024-04-25T01:39:48.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.698+0000 7fb878ff9640 1 -- 172.21.15.28:0/3631145627 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb88002d070 con 0x564abd76e980 2024-04-25T01:39:48.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.699+0000 7fb878ff9640 1 -- 172.21.15.28:0/3631145627 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb888048070 con 0x564abd76e980 2024-04-25T01:39:48.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.699+0000 7fb89a166740 10 client.18015 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:48.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.699+0000 7fb89a166740 10 client.18015 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:39:48.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.699+0000 7fb89a166740 20 client.18015 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:48.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.699+0000 7fb89a166740 1 --2- 172.21.15.28:0/3631145627 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564abd7e5e10 0x564abd7e8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.699+0000 7fb89a166740 4 client.18015 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:48.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.699+0000 7fb89a166740 1 -- 172.21.15.28:0/3631145627 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564abd6cbf00 con 0x564abd7e5e10 2024-04-25T01:39:48.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.699+0000 7fb8951f4640 1 --2- 172.21.15.28:0/3631145627 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564abd7e5e10 0x564abd7e8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.699+0000 7fb8951f4640 1 --2- 172.21.15.28:0/3631145627 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564abd7e5e10 0x564abd7e8230 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=16b9a2598fbd1355 server_cookie=cb746587df9d527f in_seq=0 out_seq=0 2024-04-25T01:39:48.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.699+0000 7fb878ff9640 10 client.18015 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:39:48.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.700+0000 7fb878ff9640 1 -- 172.21.15.28:0/3631145627 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564abd6cbf00 con 0x564abd7e5e10 2024-04-25T01:39:48.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.700+0000 7fb878ff9640 10 client.18015 handle_command_reply: tid=0 2024-04-25T01:39:48.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.700+0000 7fb89a166740 1 client.18015 shutdown 2024-04-25T01:39:48.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.700+0000 7fb89a166740 1 -- 172.21.15.28:0/3631145627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd76e980 msgr2=0x564abd7732f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.700+0000 7fb89a166740 1 --2- 172.21.15.28:0/3631145627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd76e980 0x564abd7732f0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fb888035000 tx=0x7fb888009a20 comp rx=0 tx=0).stop 2024-04-25T01:39:48.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.700+0000 7fb89a166740 1 -- 172.21.15.28:0/3631145627 shutdown_connections 2024-04-25T01:39:48.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.700+0000 7fb89a166740 1 --2- 172.21.15.28:0/3631145627 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564abd7e5e10 0x564abd7e8230 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.700+0000 7fb89a166740 1 --2- 172.21.15.28:0/3631145627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564abd76e980 0x564abd7732f0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.700+0000 7fb89a166740 1 --2- 172.21.15.28:0/3631145627 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564abd75fda0 0x564abd772db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.700+0000 7fb89a166740 1 --2- 172.21.15.28:0/3631145627 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564abd75f480 0x564abd772870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.700+0000 7fb878ff9640 0 client.18015 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:39:48.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.700+0000 7fb89a166740 1 -- 172.21.15.28:0/3631145627 >> 172.21.15.28:0/3631145627 conn(0x564abd75b8a0 msgr2=0x564abd761600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.700+0000 7fb89a166740 1 -- 172.21.15.28:0/3631145627 shutdown_connections 2024-04-25T01:39:48.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.700+0000 7fb89a166740 1 -- 172.21.15.28:0/3631145627 wait complete. 2024-04-25T01:39:48.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.700+0000 7fb89a166740 20 client.18015 trim_cache size 0 max 16384 2024-04-25T01:39:48.689 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:48.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.700+0000 7fb897458640 1 -- 172.21.15.28:0/2782538111 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb87c0a2d80 msgr2=0x7fb87c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.700+0000 7fb897458640 1 --2- 172.21.15.28:0/2782538111 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb87c0a2d80 0x7fb87c0a5240 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb888004af0 tx=0x7fb888004a40 comp rx=0 tx=0).stop 2024-04-25T01:39:48.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.700+0000 7fb897458640 1 -- 172.21.15.28:0/2782538111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb890101740 msgr2=0x7fb890198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.700+0000 7fb897458640 1 --2- 172.21.15.28:0/2782538111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb890101740 0x7fb890198010 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00e9f0 tx=0x7fb88c00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.701+0000 7fb897458640 1 -- 172.21.15.28:0/2782538111 shutdown_connections 2024-04-25T01:39:48.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.701+0000 7fb897458640 1 --2- 172.21.15.28:0/2782538111 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb87c0a2d80 0x7fb87c0a5240 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.701+0000 7fb897458640 1 --2- 172.21.15.28:0/2782538111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb890107750 0x7fb89019c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.701+0000 7fb897458640 1 --2- 172.21.15.28:0/2782538111 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb890102100 0x7fb890198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.701+0000 7fb897458640 1 --2- 172.21.15.28:0/2782538111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb890101740 0x7fb890198010 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.701+0000 7fb897458640 1 -- 172.21.15.28:0/2782538111 >> 172.21.15.28:0/2782538111 conn(0x7fb890078bd0 msgr2=0x7fb8900ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.701+0000 7fb897458640 1 -- 172.21.15.28:0/2782538111 shutdown_connections 2024-04-25T01:39:48.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.701+0000 7fb897458640 1 -- 172.21.15.28:0/2782538111 wait complete. 2024-04-25T01:39:48.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:39:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.816+0000 7f6b4f770640 1 -- 172.21.15.28:0/1593246654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b481029d0 msgr2=0x7f6b48102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.816+0000 7f6b4f770640 1 --2- 172.21.15.28:0/1593246654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b481029d0 0x7f6b48102db0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009ca0 tx=0x7f6b4402a9e0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.816+0000 7f6b4f770640 1 -- 172.21.15.28:0/1593246654 shutdown_connections 2024-04-25T01:39:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.816+0000 7f6b4f770640 1 --2- 172.21.15.28:0/1593246654 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b4810bd10 0x7f6b4810e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.816+0000 7f6b4f770640 1 --2- 172.21.15.28:0/1593246654 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b481032f0 0x7f6b4810b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.816+0000 7f6b4f770640 1 --2- 172.21.15.28:0/1593246654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b481029d0 0x7f6b48102db0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.816+0000 7f6b4f770640 1 -- 172.21.15.28:0/1593246654 >> 172.21.15.28:0/1593246654 conn(0x7f6b480769c0 msgr2=0x7f6b48076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.816+0000 7f6b4f770640 1 -- 172.21.15.28:0/1593246654 shutdown_connections 2024-04-25T01:39:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.816+0000 7f6b4f770640 1 -- 172.21.15.28:0/1593246654 wait complete. 2024-04-25T01:39:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.817+0000 7f6b4f770640 1 Processor -- start 2024-04-25T01:39:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.817+0000 7f6b4f770640 1 -- start start 2024-04-25T01:39:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.817+0000 7f6b4f770640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b481029d0 0x7f6b48198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.817+0000 7f6b4f770640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b481032f0 0x7f6b481985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.817+0000 7f6b4f770640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b4810bd10 0x7f6b4819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.817+0000 7f6b4f770640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b48111b60 con 0x7f6b481032f0 2024-04-25T01:39:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.817+0000 7f6b4f770640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481119e0 con 0x7f6b4810bd10 2024-04-25T01:39:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.817+0000 7f6b4f770640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6b48111ce0 con 0x7f6b481029d0 2024-04-25T01:39:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.817+0000 7f6b4cd0b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b481032f0 0x7f6b481985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.817+0000 7f6b4cd0b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b481032f0 0x7f6b481985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40142/0 (socket says 172.21.15.28:40142) 2024-04-25T01:39:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.817+0000 7f6b4cd0b640 1 -- 172.21.15.28:0/1952107536 learned_addr learned my addr 172.21.15.28:0/1952107536 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.817+0000 7f6b4d50c640 1 --2- 172.21.15.28:0/1952107536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b481029d0 0x7f6b48198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.817+0000 7f6b4dd0d640 1 --2- 172.21.15.28:0/1952107536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b4810bd10 0x7f6b4819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.817+0000 7f6b4cd0b640 1 -- 172.21.15.28:0/1952107536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b481029d0 msgr2=0x7f6b48198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.817+0000 7f6b4cd0b640 1 --2- 172.21.15.28:0/1952107536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b481029d0 0x7f6b48198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.817+0000 7f6b4cd0b640 1 -- 172.21.15.28:0/1952107536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b4810bd10 msgr2=0x7f6b4819c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.817+0000 7f6b4cd0b640 1 --2- 172.21.15.28:0/1952107536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b4810bd10 0x7f6b4819c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.817+0000 7f6b4cd0b640 1 -- 172.21.15.28:0/1952107536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b4819d050 con 0x7f6b481032f0 2024-04-25T01:39:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.817+0000 7f6b4cd0b640 1 --2- 172.21.15.28:0/1952107536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b481032f0 0x7f6b481985a0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f6b3800c970 tx=0x7f6b3800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.818+0000 7f6b3e7fc640 1 -- 172.21.15.28:0/1952107536 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b38007bb0 con 0x7f6b481032f0 2024-04-25T01:39:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.818+0000 7f6b3e7fc640 1 -- 172.21.15.28:0/1952107536 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b38007d50 con 0x7f6b481032f0 2024-04-25T01:39:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.818+0000 7f6b4f770640 1 -- 172.21.15.28:0/1952107536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b4819d340 con 0x7f6b481032f0 2024-04-25T01:39:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.818+0000 7f6b3e7fc640 1 -- 172.21.15.28:0/1952107536 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b3800f450 con 0x7f6b481032f0 2024-04-25T01:39:48.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.818+0000 7f6b4f770640 1 -- 172.21.15.28:0/1952107536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b481a4bd0 con 0x7f6b481032f0 2024-04-25T01:39:48.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.820+0000 7f6b3e7fc640 1 -- 172.21.15.28:0/1952107536 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6b38005020 con 0x7f6b481032f0 2024-04-25T01:39:48.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b52484740 1 Processor -- start 2024-04-25T01:39:48.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b52484740 1 -- start start 2024-04-25T01:39:48.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b52484740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37460 0x55aebed37840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b3e7fc640 1 --2- 172.21.15.28:0/1952107536 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6b340a2d00 0x7f6b340a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b52484740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed37d80 0x55aebed46440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b52484740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed46980 0x55aebed48d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b52484740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aebec9c670 con 0x55aebed46980 2024-04-25T01:39:48.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b52484740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55aebec44970 con 0x55aebed37d80 2024-04-25T01:39:48.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b52484740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55aebec98630 con 0x55aebed37460 2024-04-25T01:39:48.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b4dd0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed46980 0x55aebed48d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b4d50c640 1 --2- 172.21.15.28:0/1952107536 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6b340a2d00 0x7f6b340a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b4dd0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed46980 0x55aebed48d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40150/0 (socket says 172.21.15.28:40150) 2024-04-25T01:39:48.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b4dd0d640 1 -- 172.21.15.28:0/1503405344 learned_addr learned my addr 172.21.15.28:0/1503405344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b3e7fc640 1 -- 172.21.15.28:0/1952107536 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f6b380cf530 con 0x7f6b481032f0 2024-04-25T01:39:48.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b4dd0d640 1 --2- 172.21.15.28:0/1503405344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37460 0x55aebed37840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b4cd0b640 1 --2- 172.21.15.28:0/1503405344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed37d80 0x55aebed46440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b4dd0d640 1 -- 172.21.15.28:0/1503405344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37460 msgr2=0x55aebed37840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b4dd0d640 1 --2- 172.21.15.28:0/1503405344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37460 0x55aebed37840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b4dd0d640 1 -- 172.21.15.28:0/1503405344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed37d80 msgr2=0x55aebed46440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b4dd0d640 1 --2- 172.21.15.28:0/1503405344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed37d80 0x55aebed46440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.821+0000 7f6b4dd0d640 1 -- 172.21.15.28:0/1503405344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aebec68400 con 0x55aebed46980 2024-04-25T01:39:48.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.822+0000 7f6b4d50c640 1 --2- 172.21.15.28:0/1952107536 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6b340a2d00 0x7f6b340a51c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6b44006010 tx=0x7f6b4403a040 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.822+0000 7f6b4dd0d640 1 --2- 172.21.15.28:0/1503405344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed46980 0x55aebed48d70 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f6b400126b0 tx=0x7f6b4000b6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2adc097efee0ed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.822+0000 7f6b337fe640 1 -- 172.21.15.28:0/1503405344 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4001f420 con 0x55aebed46980 2024-04-25T01:39:48.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.822+0000 7f6b337fe640 1 -- 172.21.15.28:0/1503405344 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b4001e8a0 con 0x55aebed46980 2024-04-25T01:39:48.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.822+0000 7f6b337fe640 1 -- 172.21.15.28:0/1503405344 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4001d810 con 0x55aebed46980 2024-04-25T01:39:48.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.822+0000 7f6b52484740 1 -- 172.21.15.28:0/1503405344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed46980 msgr2=0x55aebed48d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.822+0000 7f6b52484740 1 --2- 172.21.15.28:0/1503405344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed46980 0x55aebed48d70 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f6b400126b0 tx=0x7f6b4000b6d0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.823+0000 7f6b52484740 1 -- 172.21.15.28:0/1503405344 shutdown_connections 2024-04-25T01:39:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.823+0000 7f6b52484740 1 --2- 172.21.15.28:0/1503405344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed46980 0x55aebed48d70 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.823+0000 7f6b52484740 1 --2- 172.21.15.28:0/1503405344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed37d80 0x55aebed46440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.823+0000 7f6b52484740 1 --2- 172.21.15.28:0/1503405344 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37460 0x55aebed37840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.823+0000 7f6b52484740 1 -- 172.21.15.28:0/1503405344 >> 172.21.15.28:0/1503405344 conn(0x55aebed338a0 msgr2=0x55aebed33c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.823+0000 7f6b52484740 1 -- 172.21.15.28:0/1503405344 shutdown_connections 2024-04-25T01:39:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.823+0000 7f6b52484740 1 -- 172.21.15.28:0/1503405344 wait complete. 2024-04-25T01:39:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.823+0000 7f6b52484740 1 -- start start 2024-04-25T01:39:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.823+0000 7f6b52484740 1 Processor -- start 2024-04-25T01:39:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.823+0000 7f6b52484740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed37460 0x55aebed46330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.823+0000 7f6b52484740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37d80 0x55aebed43620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.823+0000 7f6b52484740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed46980 0x55aebed43bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.823+0000 7f6b52484740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aebed497c0 con 0x55aebed37460 2024-04-25T01:39:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.823+0000 7f6b52484740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55aebec58050 con 0x55aebed46980 2024-04-25T01:39:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.823+0000 7f6b52484740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55aebed49940 con 0x55aebed37d80 2024-04-25T01:39:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.823+0000 7f6b52484740 10 client.0 _ll_register_callbacks cb 0x55aebe9afbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.823+0000 7f6b4d50c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed37460 0x55aebed46330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.824+0000 7f6b4d50c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed37460 0x55aebed46330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40158/0 (socket says 172.21.15.28:40158) 2024-04-25T01:39:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.824+0000 7f6b4d50c640 1 -- 172.21.15.28:0/1737792807 learned_addr learned my addr 172.21.15.28:0/1737792807 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.824+0000 7f6b4cd0b640 1 --2- 172.21.15.28:0/1737792807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37d80 0x55aebed43620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.824+0000 7f6b4dd0d640 1 --2- 172.21.15.28:0/1737792807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed46980 0x55aebed43bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.824+0000 7f6b4d50c640 1 -- 172.21.15.28:0/1737792807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37d80 msgr2=0x55aebed43620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.824+0000 7f6b4d50c640 1 --2- 172.21.15.28:0/1737792807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37d80 0x55aebed43620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.824+0000 7f6b4d50c640 1 -- 172.21.15.28:0/1737792807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed46980 msgr2=0x55aebed43bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.824+0000 7f6b4d50c640 1 --2- 172.21.15.28:0/1737792807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed46980 0x55aebed43bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.824+0000 7f6b4d50c640 1 -- 172.21.15.28:0/1737792807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aebec29dc0 con 0x55aebed37460 2024-04-25T01:39:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.824+0000 7f6b4d50c640 1 --2- 172.21.15.28:0/1737792807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed37460 0x55aebed46330 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009ca0 tx=0x7f6b440095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bef8c36a7feacd5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.824+0000 7f6b30ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.824+0000 7f6b30ff9640 1 -- 172.21.15.28:0/1737792807 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b44054060 con 0x55aebed37460 2024-04-25T01:39:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.824+0000 7f6b30ff9640 1 -- 172.21.15.28:0/1737792807 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f6b4402f640 con 0x55aebed37460 2024-04-25T01:39:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.824+0000 7f6b52484740 1 -- 172.21.15.28:0/1737792807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b4001e8a0 con 0x55aebed37460 2024-04-25T01:39:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.824+0000 7f6b30ff9640 1 -- 172.21.15.28:0/1737792807 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6b4405b050 con 0x55aebed37460 2024-04-25T01:39:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.824+0000 7f6b52484740 10 client.18027 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.824+0000 7f6b52484740 1 -- 172.21.15.28:0/1737792807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55aebe9b03f0 con 0x55aebed37460 2024-04-25T01:39:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.824+0000 7f6b30ff9640 1 -- 172.21.15.28:0/1737792807 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b4001e8a0 con 0x55aebed37460 2024-04-25T01:39:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.825+0000 7f6b30ff9640 1 -- 172.21.15.28:0/1737792807 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6b44004570 con 0x55aebed37460 2024-04-25T01:39:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.825+0000 7f6b52484740 10 client.18027 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.825+0000 7f6b52484740 10 client.18027 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:39:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.825+0000 7f6b52484740 20 client.18027 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.825+0000 7f6b52484740 1 --2- 172.21.15.28:0/1737792807 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55aebed56ad0 0x55aebed58ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.825+0000 7f6b52484740 4 client.18027 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.825+0000 7f6b52484740 1 -- 172.21.15.28:0/1737792807 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aebeca9bb0 con 0x55aebed56ad0 2024-04-25T01:39:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.825+0000 7f6b4cd0b640 1 --2- 172.21.15.28:0/1737792807 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55aebed56ad0 0x55aebed58ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.825+0000 7f6b30ff9640 10 client.18027 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:39:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.825+0000 7f6b4cd0b640 1 --2- 172.21.15.28:0/1737792807 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55aebed56ad0 0x55aebed58ee0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d291e0f7044cd4e1 server_cookie=3e4a95a32c22aef5 in_seq=0 out_seq=0 2024-04-25T01:39:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.826+0000 7f6b30ff9640 1 -- 172.21.15.28:0/1737792807 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aebeca9bb0 con 0x55aebed56ad0 2024-04-25T01:39:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.826+0000 7f6b30ff9640 10 client.18027 handle_command_reply: tid=0 2024-04-25T01:39:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.826+0000 7f6b52484740 1 client.18027 shutdown 2024-04-25T01:39:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.826+0000 7f6b52484740 1 -- 172.21.15.28:0/1737792807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed37460 msgr2=0x55aebed46330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.826+0000 7f6b52484740 1 --2- 172.21.15.28:0/1737792807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed37460 0x55aebed46330 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009ca0 tx=0x7f6b440095c0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.826+0000 7f6b52484740 1 -- 172.21.15.28:0/1737792807 shutdown_connections 2024-04-25T01:39:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.826+0000 7f6b52484740 1 --2- 172.21.15.28:0/1737792807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed46980 0x55aebed43bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.826+0000 7f6b52484740 1 --2- 172.21.15.28:0/1737792807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37d80 0x55aebed43620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.826+0000 7f6b52484740 1 --2- 172.21.15.28:0/1737792807 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55aebed56ad0 0x55aebed58ee0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.826+0000 7f6b52484740 1 --2- 172.21.15.28:0/1737792807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed37460 0x55aebed46330 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.826+0000 7f6b30ff9640 0 client.18027 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:39:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.826+0000 7f6b52484740 1 -- 172.21.15.28:0/1737792807 >> 172.21.15.28:0/1737792807 conn(0x55aebed338a0 msgr2=0x55aebed39650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.826+0000 7f6b52484740 1 -- 172.21.15.28:0/1737792807 shutdown_connections 2024-04-25T01:39:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.827+0000 7f6b52484740 1 -- 172.21.15.28:0/1737792807 wait complete. 2024-04-25T01:39:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.827+0000 7f6b52484740 20 client.18027 trim_cache size 0 max 16384 2024-04-25T01:39:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.840+0000 7f6b52484740 1 Processor -- start 2024-04-25T01:39:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.840+0000 7f6b52484740 1 -- start start 2024-04-25T01:39:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.840+0000 7f6b52484740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37460 0x55aebed526f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.840+0000 7f6b52484740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed37d80 0x55aebed43520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.840+0000 7f6b52484740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed46980 0x55aebed45e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.840+0000 7f6b52484740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481ac940 con 0x55aebed46980 2024-04-25T01:39:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.840+0000 7f6b52484740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481ae480 con 0x55aebed37d80 2024-04-25T01:39:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.840+0000 7f6b52484740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481ad010 con 0x55aebed37460 2024-04-25T01:39:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.841+0000 7f6b4dd0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed46980 0x55aebed45e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.841+0000 7f6b4dd0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed46980 0x55aebed45e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40170/0 (socket says 172.21.15.28:40170) 2024-04-25T01:39:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.841+0000 7f6b4dd0d640 1 -- 172.21.15.28:0/1137710315 learned_addr learned my addr 172.21.15.28:0/1137710315 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.841+0000 7f6b4d50c640 1 --2- 172.21.15.28:0/1137710315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37460 0x55aebed526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.841+0000 7f6b4cd0b640 1 --2- 172.21.15.28:0/1137710315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed37d80 0x55aebed43520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.841+0000 7f6b4dd0d640 1 -- 172.21.15.28:0/1137710315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37460 msgr2=0x55aebed526f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.841+0000 7f6b4dd0d640 1 --2- 172.21.15.28:0/1137710315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37460 0x55aebed526f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.841+0000 7f6b4dd0d640 1 -- 172.21.15.28:0/1137710315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed37d80 msgr2=0x55aebed43520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.841+0000 7f6b4dd0d640 1 --2- 172.21.15.28:0/1137710315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed37d80 0x55aebed43520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.841+0000 7f6b4dd0d640 1 -- 172.21.15.28:0/1137710315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aebea7a7d0 con 0x55aebed46980 2024-04-25T01:39:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.841+0000 7f6b4cd0b640 1 --2- 172.21.15.28:0/1137710315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed37d80 0x55aebed43520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.841+0000 7f6b4dd0d640 1 --2- 172.21.15.28:0/1137710315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed46980 0x55aebed45e60 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f6b400126b0 tx=0x7f6b40005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66ee9dcc5bbab2cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.841+0000 7f6b32ffd640 1 -- 172.21.15.28:0/1137710315 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b400050b0 con 0x55aebed46980 2024-04-25T01:39:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.841+0000 7f6b32ffd640 1 -- 172.21.15.28:0/1137710315 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b400035f0 con 0x55aebed46980 2024-04-25T01:39:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.841+0000 7f6b32ffd640 1 -- 172.21.15.28:0/1137710315 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4001e620 con 0x55aebed46980 2024-04-25T01:39:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.842+0000 7f6b52484740 1 -- 172.21.15.28:0/1137710315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed46980 msgr2=0x55aebed45e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.842+0000 7f6b52484740 1 --2- 172.21.15.28:0/1137710315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed46980 0x55aebed45e60 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f6b400126b0 tx=0x7f6b40005040 comp rx=0 tx=0).stop 2024-04-25T01:39:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.842+0000 7f6b52484740 1 -- 172.21.15.28:0/1137710315 shutdown_connections 2024-04-25T01:39:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.842+0000 7f6b52484740 1 --2- 172.21.15.28:0/1137710315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed46980 0x55aebed45e60 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.842+0000 7f6b52484740 1 --2- 172.21.15.28:0/1137710315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed37d80 0x55aebed43520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.842+0000 7f6b52484740 1 --2- 172.21.15.28:0/1137710315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37460 0x55aebed526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.842+0000 7f6b52484740 1 -- 172.21.15.28:0/1137710315 >> 172.21.15.28:0/1137710315 conn(0x55aebed338a0 msgr2=0x55aebed3ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.842+0000 7f6b52484740 1 -- 172.21.15.28:0/1137710315 shutdown_connections 2024-04-25T01:39:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.842+0000 7f6b52484740 1 -- 172.21.15.28:0/1137710315 wait complete. 2024-04-25T01:39:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.842+0000 7f6b52484740 1 -- start start 2024-04-25T01:39:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.842+0000 7f6b52484740 1 Processor -- start 2024-04-25T01:39:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b52484740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed37460 0x55aebed4f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b52484740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37d80 0x55aebed400b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b52484740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed46980 0x55aebed407f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b52484740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481addb0 con 0x55aebed37460 2024-04-25T01:39:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b52484740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481ad6e0 con 0x55aebed46980 2024-04-25T01:39:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b52484740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481aa920 con 0x55aebed37d80 2024-04-25T01:39:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b52484740 10 client.0 _ll_register_callbacks cb 0x7f6b440098a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b4d50c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed37460 0x55aebed4f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b4d50c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed37460 0x55aebed4f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40176/0 (socket says 172.21.15.28:40176) 2024-04-25T01:39:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b4d50c640 1 -- 172.21.15.28:0/928452437 learned_addr learned my addr 172.21.15.28:0/928452437 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b4cd0b640 1 --2- 172.21.15.28:0/928452437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37d80 0x55aebed400b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b4dd0d640 1 --2- 172.21.15.28:0/928452437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed46980 0x55aebed407f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b4d50c640 1 -- 172.21.15.28:0/928452437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37d80 msgr2=0x55aebed400b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b4d50c640 1 --2- 172.21.15.28:0/928452437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37d80 0x55aebed400b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b4d50c640 1 -- 172.21.15.28:0/928452437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed46980 msgr2=0x55aebed407f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b4d50c640 1 --2- 172.21.15.28:0/928452437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed46980 0x55aebed407f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b4d50c640 1 -- 172.21.15.28:0/928452437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aebedbc020 con 0x55aebed37460 2024-04-25T01:39:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b4cd0b640 1 --2- 172.21.15.28:0/928452437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37d80 0x55aebed400b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b4dd0d640 1 --2- 172.21.15.28:0/928452437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed46980 0x55aebed407f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b4d50c640 1 --2- 172.21.15.28:0/928452437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed37460 0x55aebed4f100 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f6b4402ead0 tx=0x7f6b4402b060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f996d5b4567792 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b30ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b30ff9640 1 -- 172.21.15.28:0/928452437 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b44054040 con 0x55aebed37460 2024-04-25T01:39:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b30ff9640 1 -- 172.21.15.28:0/928452437 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f6b440389e0 con 0x55aebed37460 2024-04-25T01:39:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b52484740 1 -- 172.21.15.28:0/928452437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b400035f0 con 0x55aebed37460 2024-04-25T01:39:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.843+0000 7f6b30ff9640 1 -- 172.21.15.28:0/928452437 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6b44033050 con 0x55aebed37460 2024-04-25T01:39:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.844+0000 7f6b52484740 10 client.18033 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.844+0000 7f6b52484740 1 -- 172.21.15.28:0/928452437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55aebedbd0c0 con 0x55aebed37460 2024-04-25T01:39:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.844+0000 7f6b30ff9640 1 -- 172.21.15.28:0/928452437 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b400035f0 con 0x55aebed37460 2024-04-25T01:39:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.844+0000 7f6b30ff9640 1 -- 172.21.15.28:0/928452437 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6b4402f640 con 0x55aebed37460 2024-04-25T01:39:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.844+0000 7f6b52484740 10 client.18033 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.844+0000 7f6b52484740 10 client.18033 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:39:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.844+0000 7f6b52484740 20 client.18033 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.844+0000 7f6b52484740 1 --2- 172.21.15.28:0/928452437 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55aebedbd830 0x55aebedbfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.844+0000 7f6b52484740 4 client.18033 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.844+0000 7f6b52484740 1 -- 172.21.15.28:0/928452437 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55aebeca3f00 con 0x55aebedbd830 2024-04-25T01:39:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.844+0000 7f6b4cd0b640 1 --2- 172.21.15.28:0/928452437 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55aebedbd830 0x55aebedbfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.845+0000 7f6b30ff9640 10 client.18033 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:39:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.845+0000 7f6b4cd0b640 1 --2- 172.21.15.28:0/928452437 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55aebedbd830 0x55aebedbfc50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e653ad3265d464d8 server_cookie=c16c3165c6908ebb in_seq=0 out_seq=0 2024-04-25T01:39:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.845+0000 7f6b30ff9640 1 -- 172.21.15.28:0/928452437 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55aebeca3f00 con 0x55aebedbd830 2024-04-25T01:39:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.845+0000 7f6b30ff9640 10 client.18033 handle_command_reply: tid=0 2024-04-25T01:39:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.845+0000 7f6b52484740 1 client.18033 shutdown 2024-04-25T01:39:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.845+0000 7f6b52484740 1 -- 172.21.15.28:0/928452437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed37460 msgr2=0x55aebed4f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.845+0000 7f6b52484740 1 --2- 172.21.15.28:0/928452437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed37460 0x55aebed4f100 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f6b4402ead0 tx=0x7f6b4402b060 comp rx=0 tx=0).stop 2024-04-25T01:39:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.845+0000 7f6b52484740 1 -- 172.21.15.28:0/928452437 shutdown_connections 2024-04-25T01:39:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.845+0000 7f6b52484740 1 --2- 172.21.15.28:0/928452437 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55aebed46980 0x55aebed407f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.845+0000 7f6b52484740 1 --2- 172.21.15.28:0/928452437 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55aebed37d80 0x55aebed400b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.845+0000 7f6b52484740 1 --2- 172.21.15.28:0/928452437 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55aebedbd830 0x55aebedbfc50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.845+0000 7f6b52484740 1 --2- 172.21.15.28:0/928452437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aebed37460 0x55aebed4f100 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.845+0000 7f6b30ff9640 0 client.18033 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:39:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.845+0000 7f6b52484740 1 -- 172.21.15.28:0/928452437 >> 172.21.15.28:0/928452437 conn(0x55aebed338a0 msgr2=0x55aebed322b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.845+0000 7f6b52484740 1 -- 172.21.15.28:0/928452437 shutdown_connections 2024-04-25T01:39:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.845+0000 7f6b52484740 1 -- 172.21.15.28:0/928452437 wait complete. 2024-04-25T01:39:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.845+0000 7f6b52484740 20 client.18033 trim_cache size 0 max 16384 2024-04-25T01:39:48.837 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:48.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.846+0000 7f6b4f770640 1 -- 172.21.15.28:0/1952107536 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6b340a2d00 msgr2=0x7f6b340a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.846+0000 7f6b4f770640 1 --2- 172.21.15.28:0/1952107536 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6b340a2d00 0x7f6b340a51c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6b44006010 tx=0x7f6b4403a040 comp rx=0 tx=0).stop 2024-04-25T01:39:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.846+0000 7f6b4f770640 1 -- 172.21.15.28:0/1952107536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b481032f0 msgr2=0x7f6b481985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.846+0000 7f6b4f770640 1 --2- 172.21.15.28:0/1952107536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b481032f0 0x7f6b481985a0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f6b3800c970 tx=0x7f6b3800ce30 comp rx=0 tx=0).stop 2024-04-25T01:39:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.846+0000 7f6b4f770640 1 -- 172.21.15.28:0/1952107536 shutdown_connections 2024-04-25T01:39:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.846+0000 7f6b4f770640 1 --2- 172.21.15.28:0/1952107536 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6b340a2d00 0x7f6b340a51c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.846+0000 7f6b4f770640 1 --2- 172.21.15.28:0/1952107536 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b4810bd10 0x7f6b4819c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.846+0000 7f6b4f770640 1 --2- 172.21.15.28:0/1952107536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b481032f0 0x7f6b481985a0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.846+0000 7f6b4f770640 1 --2- 172.21.15.28:0/1952107536 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b481029d0 0x7f6b48198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.846+0000 7f6b4f770640 1 -- 172.21.15.28:0/1952107536 >> 172.21.15.28:0/1952107536 conn(0x7f6b480769c0 msgr2=0x7f6b480fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.846+0000 7f6b4f770640 1 -- 172.21.15.28:0/1952107536 shutdown_connections 2024-04-25T01:39:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.846+0000 7f6b4f770640 1 -- 172.21.15.28:0/1952107536 wait complete. 2024-04-25T01:39:48.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:39:48.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.964+0000 7f6500f7e640 1 -- 172.21.15.28:0/3000186018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64fc107720 msgr2=0x7f64fc107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.964+0000 7f6500f7e640 1 --2- 172.21.15.28:0/3000186018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64fc107720 0x7f64fc107b00 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f64ec009930 tx=0x7f64ec02a420 comp rx=0 tx=0).stop 2024-04-25T01:39:48.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.964+0000 7f6500f7e640 1 -- 172.21.15.28:0/3000186018 shutdown_connections 2024-04-25T01:39:48.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.964+0000 7f6500f7e640 1 --2- 172.21.15.28:0/3000186018 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64fc1020d0 0x7f64fc10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.964+0000 7f6500f7e640 1 --2- 172.21.15.28:0/3000186018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc101710 0x7f64fc101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.964+0000 7f6500f7e640 1 --2- 172.21.15.28:0/3000186018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64fc107720 0x7f64fc107b00 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.964+0000 7f6500f7e640 1 -- 172.21.15.28:0/3000186018 >> 172.21.15.28:0/3000186018 conn(0x7f64fc0fd180 msgr2=0x7f64fc0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.964+0000 7f6500f7e640 1 -- 172.21.15.28:0/3000186018 shutdown_connections 2024-04-25T01:39:48.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.964+0000 7f6500f7e640 1 -- 172.21.15.28:0/3000186018 wait complete. 2024-04-25T01:39:48.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.965+0000 7f6500f7e640 1 Processor -- start 2024-04-25T01:39:48.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.965+0000 7f6500f7e640 1 -- start start 2024-04-25T01:39:48.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.965+0000 7f6500f7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64fc101710 0x7f64fc198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.965+0000 7f6500f7e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64fc1020d0 0x7f64fc1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.965+0000 7f6500f7e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc107720 0x7f64fc19c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.965+0000 7f6500f7e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64fc111b80 con 0x7f64fc101710 2024-04-25T01:39:48.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.965+0000 7f6500f7e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f64fc111a00 con 0x7f64fc107720 2024-04-25T01:39:48.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.965+0000 7f6500f7e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f64fc111d00 con 0x7f64fc1020d0 2024-04-25T01:39:48.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.965+0000 7f64fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64fc101710 0x7f64fc198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.965+0000 7f64fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64fc101710 0x7f64fc198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40190/0 (socket says 172.21.15.28:40190) 2024-04-25T01:39:48.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.965+0000 7f64fa59c640 1 -- 172.21.15.28:0/472168386 learned_addr learned my addr 172.21.15.28:0/472168386 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.965+0000 7f64fad9d640 1 --2- 172.21.15.28:0/472168386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc107720 0x7f64fc19c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.965+0000 7f64f1d9b640 1 --2- 172.21.15.28:0/472168386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64fc1020d0 0x7f64fc1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.966+0000 7f64fa59c640 1 -- 172.21.15.28:0/472168386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64fc1020d0 msgr2=0x7f64fc1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.966+0000 7f64fa59c640 1 --2- 172.21.15.28:0/472168386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64fc1020d0 0x7f64fc1985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.966+0000 7f64fa59c640 1 -- 172.21.15.28:0/472168386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc107720 msgr2=0x7f64fc19c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.966+0000 7f64fa59c640 1 --2- 172.21.15.28:0/472168386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc107720 0x7f64fc19c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.966+0000 7f64fa59c640 1 -- 172.21.15.28:0/472168386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64fc19d050 con 0x7f64fc101710 2024-04-25T01:39:48.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.966+0000 7f64fa59c640 1 --2- 172.21.15.28:0/472168386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64fc101710 0x7f64fc198060 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f64ec02a940 tx=0x7f64ec003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.966+0000 7f64f3fff640 1 -- 172.21.15.28:0/472168386 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec03f070 con 0x7f64fc101710 2024-04-25T01:39:48.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.966+0000 7f6500f7e640 1 -- 172.21.15.28:0/472168386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64fc19d2e0 con 0x7f64fc101710 2024-04-25T01:39:48.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.966+0000 7f64f3fff640 1 -- 172.21.15.28:0/472168386 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64ec031c90 con 0x7f64fc101710 2024-04-25T01:39:48.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.966+0000 7f64f3fff640 1 -- 172.21.15.28:0/472168386 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec035030 con 0x7f64fc101710 2024-04-25T01:39:48.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.966+0000 7f6500f7e640 1 -- 172.21.15.28:0/472168386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64fc1a4bd0 con 0x7f64fc101710 2024-04-25T01:39:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.968+0000 7f64f3fff640 1 -- 172.21.15.28:0/472168386 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f64ec042070 con 0x7f64fc101710 2024-04-25T01:39:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.968+0000 7f6503d6a740 1 Processor -- start 2024-04-25T01:39:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f6503d6a740 1 -- start start 2024-04-25T01:39:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f6503d6a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e14580 0x558053e14960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f6503d6a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e14ea0 0x558053e1f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f6503d6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e1f570 0x558053e21960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f6503d6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558053d79670 con 0x558053e1f570 2024-04-25T01:39:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f6503d6a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558053d21970 con 0x558053e14580 2024-04-25T01:39:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f6503d6a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x558053d75630 con 0x558053e14ea0 2024-04-25T01:39:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f64f1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e1f570 0x558053e21960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f64f1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e1f570 0x558053e21960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40206/0 (socket says 172.21.15.28:40206) 2024-04-25T01:39:48.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f64f1d9b640 1 -- 172.21.15.28:0/411717840 learned_addr learned my addr 172.21.15.28:0/411717840 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f64fa59c640 1 --2- 172.21.15.28:0/411717840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e14ea0 0x558053e1f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f64f3fff640 1 --2- 172.21.15.28:0/472168386 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f64e80a2d00 0x7f64e80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f64fad9d640 1 --2- 172.21.15.28:0/411717840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e14580 0x558053e14960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f64fad9d640 1 --2- 172.21.15.28:0/472168386 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f64e80a2d00 0x7f64e80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f64f1d9b640 1 -- 172.21.15.28:0/411717840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e14ea0 msgr2=0x558053e1f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f64f1d9b640 1 --2- 172.21.15.28:0/411717840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e14ea0 0x558053e1f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f64f1d9b640 1 -- 172.21.15.28:0/411717840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e14580 msgr2=0x558053e14960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f64f1d9b640 1 --2- 172.21.15.28:0/411717840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e14580 0x558053e14960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f64f1d9b640 1 -- 172.21.15.28:0/411717840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558053d45400 con 0x558053e1f570 2024-04-25T01:39:48.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.969+0000 7f64f3fff640 1 -- 172.21.15.28:0/472168386 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f64ec0efe80 con 0x7f64fc101710 2024-04-25T01:39:48.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.970+0000 7f64f1d9b640 1 --2- 172.21.15.28:0/411717840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e1f570 0x558053e21960 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f64e400abf0 tx=0x7f64e400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad7fc80e8a7bcb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.970+0000 7f64f0d99640 1 -- 172.21.15.28:0/411717840 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e401d070 con 0x558053e1f570 2024-04-25T01:39:48.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.970+0000 7f64fad9d640 1 --2- 172.21.15.28:0/472168386 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f64e80a2d00 0x7f64e80a51c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f64f4005fd0 tx=0x7f64f4004ab0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.970+0000 7f64f0d99640 1 -- 172.21.15.28:0/411717840 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64e4018b20 con 0x558053e1f570 2024-04-25T01:39:48.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.970+0000 7f64f0d99640 1 -- 172.21.15.28:0/411717840 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e4010ae0 con 0x558053e1f570 2024-04-25T01:39:48.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.970+0000 7f6503d6a740 1 -- 172.21.15.28:0/411717840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e1f570 msgr2=0x558053e21960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.970+0000 7f6503d6a740 1 --2- 172.21.15.28:0/411717840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e1f570 0x558053e21960 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f64e400abf0 tx=0x7f64e400f790 comp rx=0 tx=0).stop 2024-04-25T01:39:48.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f6503d6a740 1 -- 172.21.15.28:0/411717840 shutdown_connections 2024-04-25T01:39:48.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f6503d6a740 1 --2- 172.21.15.28:0/411717840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e1f570 0x558053e21960 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f6503d6a740 1 --2- 172.21.15.28:0/411717840 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e14ea0 0x558053e1f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f6503d6a740 1 --2- 172.21.15.28:0/411717840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e14580 0x558053e14960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f6503d6a740 1 -- 172.21.15.28:0/411717840 >> 172.21.15.28:0/411717840 conn(0x558053e108a0 msgr2=0x558053e10c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f6503d6a740 1 -- 172.21.15.28:0/411717840 shutdown_connections 2024-04-25T01:39:48.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f6503d6a740 1 -- 172.21.15.28:0/411717840 wait complete. 2024-04-25T01:39:48.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f6503d6a740 1 -- start start 2024-04-25T01:39:48.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f6503d6a740 1 Processor -- start 2024-04-25T01:39:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f6503d6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e14580 0x558053e1eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f6503d6a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e14ea0 0x558053e1c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f6503d6a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e1f570 0x558053e1c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f6503d6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558053e22360 con 0x558053e14580 2024-04-25T01:39:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f6503d6a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558053d35050 con 0x558053e1f570 2024-04-25T01:39:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f6503d6a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x558053e224e0 con 0x558053e14ea0 2024-04-25T01:39:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f6503d6a740 10 client.0 _ll_register_callbacks cb 0x558053a8cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f64fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e14580 0x558053e1eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f64fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e14580 0x558053e1eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40222/0 (socket says 172.21.15.28:40222) 2024-04-25T01:39:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f64fa59c640 1 -- 172.21.15.28:0/3512064553 learned_addr learned my addr 172.21.15.28:0/3512064553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f64f1d9b640 1 --2- 172.21.15.28:0/3512064553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e14ea0 0x558053e1c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.971+0000 7f64fad9d640 1 --2- 172.21.15.28:0/3512064553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e1f570 0x558053e1c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.972+0000 7f64fa59c640 1 -- 172.21.15.28:0/3512064553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e14ea0 msgr2=0x558053e1c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.972+0000 7f64fa59c640 1 --2- 172.21.15.28:0/3512064553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e14ea0 0x558053e1c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.972+0000 7f64fa59c640 1 -- 172.21.15.28:0/3512064553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e1f570 msgr2=0x558053e1c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.972+0000 7f64fa59c640 1 --2- 172.21.15.28:0/3512064553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e1f570 0x558053e1c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.972+0000 7f64fa59c640 1 -- 172.21.15.28:0/3512064553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558053d06dc0 con 0x558053e14580 2024-04-25T01:39:48.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.972+0000 7f64fa59c640 1 --2- 172.21.15.28:0/3512064553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e14580 0x558053e1eeb0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f64ec0b8570 tx=0x7f64ec02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36229be7dbc983e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.972+0000 7f64e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:48.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.972+0000 7f64e1ffb640 1 -- 172.21.15.28:0/3512064553 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec049070 con 0x558053e14580 2024-04-25T01:39:48.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.972+0000 7f64e1ffb640 1 -- 172.21.15.28:0/3512064553 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f64ec0ed510 con 0x558053e14580 2024-04-25T01:39:48.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.972+0000 7f6503d6a740 1 -- 172.21.15.28:0/3512064553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64e4018b20 con 0x558053e14580 2024-04-25T01:39:48.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.972+0000 7f64e1ffb640 1 -- 172.21.15.28:0/3512064553 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f64ec0bbc50 con 0x558053e14580 2024-04-25T01:39:48.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.972+0000 7f6503d6a740 10 client.18045 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:48.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.972+0000 7f6503d6a740 1 -- 172.21.15.28:0/3512064553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558053a8d3f0 con 0x558053e14580 2024-04-25T01:39:48.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.972+0000 7f64e1ffb640 1 -- 172.21.15.28:0/3512064553 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64e4018b20 con 0x558053e14580 2024-04-25T01:39:48.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.972+0000 7f64e1ffb640 1 -- 172.21.15.28:0/3512064553 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f64ec0b8780 con 0x558053e14580 2024-04-25T01:39:48.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.972+0000 7f6503d6a740 10 client.18045 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:48.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.972+0000 7f6503d6a740 10 client.18045 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:39:48.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.972+0000 7f6503d6a740 20 client.18045 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:48.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.973+0000 7f6503d6a740 1 --2- 172.21.15.28:0/3512064553 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x558053e2f590 0x558053e31a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.973+0000 7f6503d6a740 4 client.18045 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:48.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.973+0000 7f6503d6a740 1 -- 172.21.15.28:0/3512064553 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558053d86bb0 con 0x558053e2f590 2024-04-25T01:39:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.973+0000 7f64f1d9b640 1 --2- 172.21.15.28:0/3512064553 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x558053e2f590 0x558053e31a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.973+0000 7f64e1ffb640 10 client.18045 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:39:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.973+0000 7f64f1d9b640 1 --2- 172.21.15.28:0/3512064553 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x558053e2f590 0x558053e31a00 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c24cef4188af053a server_cookie=b2e20f46632c428f in_seq=0 out_seq=0 2024-04-25T01:39:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.974+0000 7f64e1ffb640 1 -- 172.21.15.28:0/3512064553 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558053d86bb0 con 0x558053e2f590 2024-04-25T01:39:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.974+0000 7f64e1ffb640 10 client.18045 handle_command_reply: tid=0 2024-04-25T01:39:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.974+0000 7f6503d6a740 1 client.18045 shutdown 2024-04-25T01:39:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.974+0000 7f6503d6a740 1 -- 172.21.15.28:0/3512064553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e14580 msgr2=0x558053e1eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.974+0000 7f6503d6a740 1 --2- 172.21.15.28:0/3512064553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e14580 0x558053e1eeb0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f64ec0b8570 tx=0x7f64ec02a420 comp rx=0 tx=0).stop 2024-04-25T01:39:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.974+0000 7f6503d6a740 1 -- 172.21.15.28:0/3512064553 shutdown_connections 2024-04-25T01:39:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.974+0000 7f6503d6a740 1 --2- 172.21.15.28:0/3512064553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e1f570 0x558053e1c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.974+0000 7f6503d6a740 1 --2- 172.21.15.28:0/3512064553 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x558053e2f590 0x558053e31a00 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.974+0000 7f6503d6a740 1 --2- 172.21.15.28:0/3512064553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e14ea0 0x558053e1c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.974+0000 7f6503d6a740 1 --2- 172.21.15.28:0/3512064553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e14580 0x558053e1eeb0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.974+0000 7f6503d6a740 1 -- 172.21.15.28:0/3512064553 >> 172.21.15.28:0/3512064553 conn(0x558053e108a0 msgr2=0x558053e0fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.974+0000 7f64e1ffb640 0 client.18045 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:39:48.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.974+0000 7f6503d6a740 1 -- 172.21.15.28:0/3512064553 shutdown_connections 2024-04-25T01:39:48.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.974+0000 7f6503d6a740 1 -- 172.21.15.28:0/3512064553 wait complete. 2024-04-25T01:39:48.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.974+0000 7f6503d6a740 20 client.18045 trim_cache size 0 max 16384 2024-04-25T01:39:48.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.988+0000 7f6503d6a740 1 Processor -- start 2024-04-25T01:39:48.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.988+0000 7f6503d6a740 1 -- start start 2024-04-25T01:39:48.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.988+0000 7f6503d6a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e108a0 0x558053e17860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.988+0000 7f6503d6a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e14580 0x558053e1e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.988+0000 7f6503d6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e14ea0 0x558053e372e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.988+0000 7f6503d6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64fc1aeaf0 con 0x558053e14ea0 2024-04-25T01:39:48.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.988+0000 7f6503d6a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f64fc1b0630 con 0x558053e14580 2024-04-25T01:39:48.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.988+0000 7f6503d6a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f64fc1af1c0 con 0x558053e108a0 2024-04-25T01:39:48.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.988+0000 7f64fad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e14ea0 0x558053e372e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.988+0000 7f64fad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e14ea0 0x558053e372e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40226/0 (socket says 172.21.15.28:40226) 2024-04-25T01:39:48.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.988+0000 7f64fad9d640 1 -- 172.21.15.28:0/1729240849 learned_addr learned my addr 172.21.15.28:0/1729240849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.988+0000 7f64f1d9b640 1 --2- 172.21.15.28:0/1729240849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e14580 0x558053e1e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.989+0000 7f64fa59c640 1 --2- 172.21.15.28:0/1729240849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e108a0 0x558053e17860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.989+0000 7f64fad9d640 1 -- 172.21.15.28:0/1729240849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e108a0 msgr2=0x558053e17860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.989+0000 7f64fad9d640 1 --2- 172.21.15.28:0/1729240849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e108a0 0x558053e17860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.989+0000 7f64fad9d640 1 -- 172.21.15.28:0/1729240849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e14580 msgr2=0x558053e1e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.989+0000 7f64fad9d640 1 --2- 172.21.15.28:0/1729240849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e14580 0x558053e1e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.989+0000 7f64fad9d640 1 -- 172.21.15.28:0/1729240849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558053b577d0 con 0x558053e14ea0 2024-04-25T01:39:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.989+0000 7f64fa59c640 1 --2- 172.21.15.28:0/1729240849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e108a0 0x558053e17860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.989+0000 7f64f1d9b640 1 --2- 172.21.15.28:0/1729240849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e14580 0x558053e1e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.989+0000 7f64fad9d640 1 --2- 172.21.15.28:0/1729240849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e14ea0 0x558053e372e0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f64f4004f10 tx=0x7f64f4009a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e416d8239f3cfe9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.989+0000 7f64e3fff640 1 -- 172.21.15.28:0/1729240849 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f4018070 con 0x558053e14ea0 2024-04-25T01:39:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.989+0000 7f64e3fff640 1 -- 172.21.15.28:0/1729240849 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64f40058a0 con 0x558053e14ea0 2024-04-25T01:39:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.989+0000 7f64e3fff640 1 -- 172.21.15.28:0/1729240849 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f401fd80 con 0x558053e14ea0 2024-04-25T01:39:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f6503d6a740 1 -- 172.21.15.28:0/1729240849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e14ea0 msgr2=0x558053e372e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f6503d6a740 1 --2- 172.21.15.28:0/1729240849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e14ea0 0x558053e372e0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f64f4004f10 tx=0x7f64f4009a00 comp rx=0 tx=0).stop 2024-04-25T01:39:48.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f6503d6a740 1 -- 172.21.15.28:0/1729240849 shutdown_connections 2024-04-25T01:39:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f6503d6a740 1 --2- 172.21.15.28:0/1729240849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e14ea0 0x558053e372e0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f6503d6a740 1 --2- 172.21.15.28:0/1729240849 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e14580 0x558053e1e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f6503d6a740 1 --2- 172.21.15.28:0/1729240849 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e108a0 0x558053e17860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f6503d6a740 1 -- 172.21.15.28:0/1729240849 >> 172.21.15.28:0/1729240849 conn(0x558053e1c840 msgr2=0x558053e0f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f6503d6a740 1 -- 172.21.15.28:0/1729240849 shutdown_connections 2024-04-25T01:39:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f6503d6a740 1 -- 172.21.15.28:0/1729240849 wait complete. 2024-04-25T01:39:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f6503d6a740 1 -- start start 2024-04-25T01:39:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f6503d6a740 1 Processor -- start 2024-04-25T01:39:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f6503d6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e108a0 0x558053e238b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f6503d6a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e14580 0x558053e23df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f6503d6a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e14ea0 0x558053e94b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f6503d6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64fc1aff60 con 0x558053e108a0 2024-04-25T01:39:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f6503d6a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f64fc1af890 con 0x558053e14ea0 2024-04-25T01:39:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f6503d6a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f64fc1acad0 con 0x558053e14580 2024-04-25T01:39:48.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f6503d6a740 10 client.0 _ll_register_callbacks cb 0x7f64ec009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:48.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f64fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e108a0 0x558053e238b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.990+0000 7f64fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e108a0 0x558053e238b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40234/0 (socket says 172.21.15.28:40234) 2024-04-25T01:39:48.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.991+0000 7f64fa59c640 1 -- 172.21.15.28:0/3767340970 learned_addr learned my addr 172.21.15.28:0/3767340970 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:48.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.991+0000 7f64f1d9b640 1 --2- 172.21.15.28:0/3767340970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e14580 0x558053e23df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.991+0000 7f64fad9d640 1 --2- 172.21.15.28:0/3767340970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e14ea0 0x558053e94b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.991+0000 7f64fa59c640 1 -- 172.21.15.28:0/3767340970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e14580 msgr2=0x558053e23df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.991+0000 7f64fa59c640 1 --2- 172.21.15.28:0/3767340970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e14580 0x558053e23df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.991+0000 7f64fa59c640 1 -- 172.21.15.28:0/3767340970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e14ea0 msgr2=0x558053e94b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:48.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.991+0000 7f64fa59c640 1 --2- 172.21.15.28:0/3767340970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e14ea0 0x558053e94b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.991+0000 7f64fa59c640 1 -- 172.21.15.28:0/3767340970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558053e950c0 con 0x558053e108a0 2024-04-25T01:39:48.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.991+0000 7f64fad9d640 1 --2- 172.21.15.28:0/3767340970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e14ea0 0x558053e94b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:48.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.991+0000 7f64fa59c640 1 --2- 172.21.15.28:0/3767340970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e108a0 0x558053e238b0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f64ec0dc010 tx=0x7f64ec03b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fff010f1d435a95 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:48.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.991+0000 7f64e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:48.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.991+0000 7f64e1ffb640 1 -- 172.21.15.28:0/3767340970 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec049070 con 0x558053e108a0 2024-04-25T01:39:48.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.991+0000 7f64e1ffb640 1 -- 172.21.15.28:0/3767340970 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f64ec0bb560 con 0x558053e108a0 2024-04-25T01:39:48.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.991+0000 7f6503d6a740 1 -- 172.21.15.28:0/3767340970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64f40058a0 con 0x558053e108a0 2024-04-25T01:39:48.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.991+0000 7f64e1ffb640 1 -- 172.21.15.28:0/3767340970 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f64ec0bba70 con 0x558053e108a0 2024-04-25T01:39:48.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.991+0000 7f6503d6a740 10 client.18051 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:48.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.991+0000 7f6503d6a740 1 -- 172.21.15.28:0/3767340970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558053e963c0 con 0x558053e108a0 2024-04-25T01:39:48.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.991+0000 7f64e1ffb640 1 -- 172.21.15.28:0/3767340970 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64f40058a0 con 0x558053e108a0 2024-04-25T01:39:48.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.991+0000 7f64e1ffb640 1 -- 172.21.15.28:0/3767340970 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f64ec0a5140 con 0x558053e108a0 2024-04-25T01:39:48.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.992+0000 7f6503d6a740 10 client.18051 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:48.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.992+0000 7f6503d6a740 10 client.18051 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:39:48.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.992+0000 7f6503d6a740 20 client.18051 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:48.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.992+0000 7f6503d6a740 1 --2- 172.21.15.28:0/3767340970 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x558053e1f570 0x558053e98950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:48.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.992+0000 7f6503d6a740 4 client.18051 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:48.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.992+0000 7f6503d6a740 1 -- 172.21.15.28:0/3767340970 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558053d80f00 con 0x558053e1f570 2024-04-25T01:39:48.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.992+0000 7f64f1d9b640 1 --2- 172.21.15.28:0/3767340970 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x558053e1f570 0x558053e98950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:48.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.992+0000 7f64f1d9b640 1 --2- 172.21.15.28:0/3767340970 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x558053e1f570 0x558053e98950 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4c0f341f44124293 server_cookie=a950c05b43c8e62b in_seq=0 out_seq=0 2024-04-25T01:39:48.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.992+0000 7f64e1ffb640 10 client.18051 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:39:48.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.992+0000 7f64e1ffb640 1 -- 172.21.15.28:0/3767340970 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558053d80f00 con 0x558053e1f570 2024-04-25T01:39:48.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.992+0000 7f64e1ffb640 10 client.18051 handle_command_reply: tid=0 2024-04-25T01:39:48.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.992+0000 7f6503d6a740 1 client.18051 shutdown 2024-04-25T01:39:48.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6503d6a740 1 -- 172.21.15.28:0/3767340970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e108a0 msgr2=0x558053e238b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6503d6a740 1 --2- 172.21.15.28:0/3767340970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e108a0 0x558053e238b0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f64ec0dc010 tx=0x7f64ec03b990 comp rx=0 tx=0).stop 2024-04-25T01:39:48.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6503d6a740 1 -- 172.21.15.28:0/3767340970 shutdown_connections 2024-04-25T01:39:48.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6503d6a740 1 --2- 172.21.15.28:0/3767340970 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558053e14ea0 0x558053e94b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6503d6a740 1 --2- 172.21.15.28:0/3767340970 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x558053e1f570 0x558053e98950 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6503d6a740 1 --2- 172.21.15.28:0/3767340970 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558053e14580 0x558053e23df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f64e1ffb640 0 client.18051 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:39:48.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6503d6a740 1 --2- 172.21.15.28:0/3767340970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558053e108a0 0x558053e238b0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6503d6a740 1 -- 172.21.15.28:0/3767340970 >> 172.21.15.28:0/3767340970 conn(0x558053e1c840 msgr2=0x558053e2a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6503d6a740 1 -- 172.21.15.28:0/3767340970 shutdown_connections 2024-04-25T01:39:48.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6503d6a740 1 -- 172.21.15.28:0/3767340970 wait complete. 2024-04-25T01:39:48.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6503d6a740 20 client.18051 trim_cache size 0 max 16384 2024-04-25T01:39:48.984 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:48.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6500f7e640 1 -- 172.21.15.28:0/472168386 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f64e80a2d00 msgr2=0x7f64e80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6500f7e640 1 --2- 172.21.15.28:0/472168386 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f64e80a2d00 0x7f64e80a51c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f64f4005fd0 tx=0x7f64f4004ab0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6500f7e640 1 -- 172.21.15.28:0/472168386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64fc101710 msgr2=0x7f64fc198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:48.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6500f7e640 1 --2- 172.21.15.28:0/472168386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64fc101710 0x7f64fc198060 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f64ec02a940 tx=0x7f64ec003b90 comp rx=0 tx=0).stop 2024-04-25T01:39:48.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6500f7e640 1 -- 172.21.15.28:0/472168386 shutdown_connections 2024-04-25T01:39:48.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6500f7e640 1 --2- 172.21.15.28:0/472168386 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f64e80a2d00 0x7f64e80a51c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6500f7e640 1 --2- 172.21.15.28:0/472168386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f64fc107720 0x7f64fc19c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6500f7e640 1 --2- 172.21.15.28:0/472168386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f64fc1020d0 0x7f64fc1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6500f7e640 1 --2- 172.21.15.28:0/472168386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64fc101710 0x7f64fc198060 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:48.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6500f7e640 1 -- 172.21.15.28:0/472168386 >> 172.21.15.28:0/472168386 conn(0x7f64fc0fd180 msgr2=0x7f64fc0fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:48.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6500f7e640 1 -- 172.21.15.28:0/472168386 shutdown_connections 2024-04-25T01:39:48.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:48.993+0000 7f6500f7e640 1 -- 172.21.15.28:0/472168386 wait complete. 2024-04-25T01:39:48.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:39:49.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.109+0000 7f6b4e883640 1 -- 172.21.15.28:0/487183824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b48107720 msgr2=0x7f6b48107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.109+0000 7f6b4e883640 1 --2- 172.21.15.28:0/487183824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b48107720 0x7f6b48107b00 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f6b38009930 tx=0x7f6b3802a420 comp rx=0 tx=0).stop 2024-04-25T01:39:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.109+0000 7f6b4e883640 1 -- 172.21.15.28:0/487183824 shutdown_connections 2024-04-25T01:39:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.109+0000 7f6b4e883640 1 --2- 172.21.15.28:0/487183824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b481020d0 0x7f6b4810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.109+0000 7f6b4e883640 1 --2- 172.21.15.28:0/487183824 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b48101710 0x7f6b48101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.109+0000 7f6b4e883640 1 --2- 172.21.15.28:0/487183824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b48107720 0x7f6b48107b00 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.109+0000 7f6b4e883640 1 -- 172.21.15.28:0/487183824 >> 172.21.15.28:0/487183824 conn(0x7f6b480fd180 msgr2=0x7f6b480ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.109+0000 7f6b4e883640 1 -- 172.21.15.28:0/487183824 shutdown_connections 2024-04-25T01:39:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.109+0000 7f6b4e883640 1 -- 172.21.15.28:0/487183824 wait complete. 2024-04-25T01:39:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.110+0000 7f6b4e883640 1 Processor -- start 2024-04-25T01:39:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.110+0000 7f6b4e883640 1 -- start start 2024-04-25T01:39:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.110+0000 7f6b4e883640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b48101710 0x7f6b48197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.110+0000 7f6b4e883640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b481020d0 0x7f6b481984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.110+0000 7f6b4e883640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b48107720 0x7f6b4819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.110+0000 7f6b4e883640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b48111b40 con 0x7f6b48101710 2024-04-25T01:39:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.110+0000 7f6b4e883640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481119c0 con 0x7f6b48107720 2024-04-25T01:39:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.110+0000 7f6b4e883640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6b48111cc0 con 0x7f6b481020d0 2024-04-25T01:39:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.111+0000 7f6b47fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b48101710 0x7f6b48197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.111+0000 7f6b47fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b48101710 0x7f6b48197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40242/0 (socket says 172.21.15.28:40242) 2024-04-25T01:39:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.111+0000 7f6b47fff640 1 -- 172.21.15.28:0/2499535361 learned_addr learned my addr 172.21.15.28:0/2499535361 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.111+0000 7f6b4ce20640 1 --2- 172.21.15.28:0/2499535361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b48107720 0x7f6b4819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.111+0000 7f6b477fe640 1 --2- 172.21.15.28:0/2499535361 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b481020d0 0x7f6b481984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.111+0000 7f6b47fff640 1 -- 172.21.15.28:0/2499535361 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b481020d0 msgr2=0x7f6b481984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.111+0000 7f6b47fff640 1 --2- 172.21.15.28:0/2499535361 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b481020d0 0x7f6b481984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.111+0000 7f6b47fff640 1 -- 172.21.15.28:0/2499535361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b48107720 msgr2=0x7f6b4819c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.111+0000 7f6b47fff640 1 --2- 172.21.15.28:0/2499535361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b48107720 0x7f6b4819c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.111+0000 7f6b47fff640 1 -- 172.21.15.28:0/2499535361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b4819cf90 con 0x7f6b48101710 2024-04-25T01:39:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.111+0000 7f6b47fff640 1 --2- 172.21.15.28:0/2499535361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b48101710 0x7f6b48197fa0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f6b3802a940 tx=0x7f6b38003c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.111+0000 7f6b457fa640 1 -- 172.21.15.28:0/2499535361 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b3803f070 con 0x7f6b48101710 2024-04-25T01:39:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.111+0000 7f6b457fa640 1 -- 172.21.15.28:0/2499535361 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b3803b460 con 0x7f6b48101710 2024-04-25T01:39:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.111+0000 7f6b4e883640 1 -- 172.21.15.28:0/2499535361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b4819d220 con 0x7f6b48101710 2024-04-25T01:39:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.111+0000 7f6b457fa640 1 -- 172.21.15.28:0/2499535361 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b38035030 con 0x7f6b48101710 2024-04-25T01:39:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.111+0000 7f6b4e883640 1 -- 172.21.15.28:0/2499535361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b481a4b10 con 0x7f6b48101710 2024-04-25T01:39:49.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.114+0000 7f6b457fa640 1 -- 172.21.15.28:0/2499535361 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6b38042070 con 0x7f6b48101710 2024-04-25T01:39:49.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.114+0000 7f6b5158a740 1 Processor -- start 2024-04-25T01:39:49.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.114+0000 7f6b5158a740 1 -- start start 2024-04-25T01:39:49.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.114+0000 7f6b5158a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb261580 0x55abfb261960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.114+0000 7f6b5158a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261ea0 0x55abfb26c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.114+0000 7f6b5158a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb26c570 0x55abfb26e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.114+0000 7f6b5158a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55abfb1c6670 con 0x55abfb261ea0 2024-04-25T01:39:49.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.114+0000 7f6b5158a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55abfb16e970 con 0x55abfb261580 2024-04-25T01:39:49.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.114+0000 7f6b5158a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55abfb1c2630 con 0x55abfb26c570 2024-04-25T01:39:49.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.114+0000 7f6b47fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261ea0 0x55abfb26c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.114+0000 7f6b457fa640 1 --2- 172.21.15.28:0/2499535361 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6b340a2d00 0x7f6b340a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.114+0000 7f6b47fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261ea0 0x55abfb26c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40250/0 (socket says 172.21.15.28:40250) 2024-04-25T01:39:49.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.114+0000 7f6b47fff640 1 -- 172.21.15.28:0/1910319778 learned_addr learned my addr 172.21.15.28:0/1910319778 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:49.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.114+0000 7f6b4ce20640 1 --2- 172.21.15.28:0/1910319778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb261580 0x55abfb261960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.115+0000 7f6b4ce20640 1 --2- 172.21.15.28:0/2499535361 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6b340a2d00 0x7f6b340a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.115+0000 7f6b477fe640 1 --2- 172.21.15.28:0/1910319778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb26c570 0x55abfb26e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.115+0000 7f6b457fa640 1 -- 172.21.15.28:0/2499535361 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f6b380f0030 con 0x7f6b48101710 2024-04-25T01:39:49.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.115+0000 7f6b47fff640 1 -- 172.21.15.28:0/1910319778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb26c570 msgr2=0x55abfb26e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:49.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.115+0000 7f6b47fff640 1 --2- 172.21.15.28:0/1910319778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb26c570 0x55abfb26e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.115+0000 7f6b47fff640 1 -- 172.21.15.28:0/1910319778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb261580 msgr2=0x55abfb261960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:49.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.115+0000 7f6b47fff640 1 --2- 172.21.15.28:0/1910319778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb261580 0x55abfb261960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.115+0000 7f6b47fff640 1 -- 172.21.15.28:0/1910319778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abfb192400 con 0x55abfb261ea0 2024-04-25T01:39:49.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.115+0000 7f6b47fff640 1 --2- 172.21.15.28:0/1910319778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261ea0 0x55abfb26c030 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f6b38000c00 tx=0x7f6b3803bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7162a93ec321e5c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:49.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.115+0000 7f6b4ce20640 1 --2- 172.21.15.28:0/2499535361 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6b340a2d00 0x7f6b340a51c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6b4000a780 tx=0x7f6b4000a4d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:49.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.115+0000 7f6b3e7fc640 1 -- 172.21.15.28:0/1910319778 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b380bc210 con 0x55abfb261ea0 2024-04-25T01:39:49.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.115+0000 7f6b3e7fc640 1 -- 172.21.15.28:0/1910319778 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b380dae50 con 0x55abfb261ea0 2024-04-25T01:39:49.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.115+0000 7f6b3e7fc640 1 -- 172.21.15.28:0/1910319778 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b380b8880 con 0x55abfb261ea0 2024-04-25T01:39:49.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.116+0000 7f6b5158a740 1 -- 172.21.15.28:0/1910319778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261ea0 msgr2=0x55abfb26c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:49.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.116+0000 7f6b5158a740 1 --2- 172.21.15.28:0/1910319778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261ea0 0x55abfb26c030 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f6b38000c00 tx=0x7f6b3803bd40 comp rx=0 tx=0).stop 2024-04-25T01:39:49.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.116+0000 7f6b5158a740 1 -- 172.21.15.28:0/1910319778 shutdown_connections 2024-04-25T01:39:49.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.116+0000 7f6b5158a740 1 --2- 172.21.15.28:0/1910319778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb26c570 0x55abfb26e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.116+0000 7f6b5158a740 1 --2- 172.21.15.28:0/1910319778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261ea0 0x55abfb26c030 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.116+0000 7f6b5158a740 1 --2- 172.21.15.28:0/1910319778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb261580 0x55abfb261960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.116+0000 7f6b5158a740 1 -- 172.21.15.28:0/1910319778 >> 172.21.15.28:0/1910319778 conn(0x55abfb25d8a0 msgr2=0x55abfb25dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:49.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.116+0000 7f6b5158a740 1 -- 172.21.15.28:0/1910319778 shutdown_connections 2024-04-25T01:39:49.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.116+0000 7f6b5158a740 1 -- 172.21.15.28:0/1910319778 wait complete. 2024-04-25T01:39:49.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.116+0000 7f6b5158a740 1 -- start start 2024-04-25T01:39:49.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.116+0000 7f6b5158a740 1 Processor -- start 2024-04-25T01:39:49.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b5158a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb261580 0x55abfb26be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b5158a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261ea0 0x55abfb269170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b5158a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb26c570 0x55abfb2696b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b5158a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55abfb26f120 con 0x55abfb261ea0 2024-04-25T01:39:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b5158a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55abfb182050 con 0x55abfb261580 2024-04-25T01:39:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b5158a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55abfb26f2a0 con 0x55abfb26c570 2024-04-25T01:39:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b5158a740 10 client.0 _ll_register_callbacks cb 0x55abfaed9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261ea0 0x55abfb269170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261ea0 0x55abfb269170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40256/0 (socket says 172.21.15.28:40256) 2024-04-25T01:39:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b477fe640 1 -- 172.21.15.28:0/2280621403 learned_addr learned my addr 172.21.15.28:0/2280621403 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b4ce20640 1 --2- 172.21.15.28:0/2280621403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb26c570 0x55abfb2696b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b47fff640 1 --2- 172.21.15.28:0/2280621403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb261580 0x55abfb26be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b477fe640 1 -- 172.21.15.28:0/2280621403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb26c570 msgr2=0x55abfb2696b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b477fe640 1 --2- 172.21.15.28:0/2280621403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb26c570 0x55abfb2696b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b477fe640 1 -- 172.21.15.28:0/2280621403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb261580 msgr2=0x55abfb26be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b477fe640 1 --2- 172.21.15.28:0/2280621403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb261580 0x55abfb26be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b477fe640 1 -- 172.21.15.28:0/2280621403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abfb153dc0 con 0x55abfb261ea0 2024-04-25T01:39:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b47fff640 1 --2- 172.21.15.28:0/2280621403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb261580 0x55abfb26be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b477fe640 1 --2- 172.21.15.28:0/2280621403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261ea0 0x55abfb269170 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f6b30010790 tx=0x7f6b30010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=404f651c37b1d6a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b277fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b277fe640 1 -- 172.21.15.28:0/2280621403 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b30010ea0 con 0x55abfb261ea0 2024-04-25T01:39:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b277fe640 1 -- 172.21.15.28:0/2280621403 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f6b3000cd20 con 0x55abfb261ea0 2024-04-25T01:39:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.117+0000 7f6b5158a740 1 -- 172.21.15.28:0/2280621403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b380dae50 con 0x55abfb261ea0 2024-04-25T01:39:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.118+0000 7f6b277fe640 1 -- 172.21.15.28:0/2280621403 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6b30024050 con 0x55abfb261ea0 2024-04-25T01:39:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.118+0000 7f6b5158a740 10 client.18063 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.118+0000 7f6b5158a740 1 -- 172.21.15.28:0/2280621403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55abfaeda3f0 con 0x55abfb261ea0 2024-04-25T01:39:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.118+0000 7f6b277fe640 1 -- 172.21.15.28:0/2280621403 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b380dae50 con 0x55abfb261ea0 2024-04-25T01:39:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.118+0000 7f6b277fe640 1 -- 172.21.15.28:0/2280621403 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6b3001c670 con 0x55abfb261ea0 2024-04-25T01:39:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.118+0000 7f6b5158a740 10 client.18063 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.118+0000 7f6b5158a740 10 client.18063 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:39:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.118+0000 7f6b5158a740 20 client.18063 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.118+0000 7f6b5158a740 1 --2- 172.21.15.28:0/2280621403 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55abfb27c320 0x55abfb27e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.118+0000 7f6b5158a740 4 client.18063 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.118+0000 7f6b5158a740 1 -- 172.21.15.28:0/2280621403 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55abfb1d3bb0 con 0x55abfb27c320 2024-04-25T01:39:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.118+0000 7f6b47fff640 1 --2- 172.21.15.28:0/2280621403 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55abfb27c320 0x55abfb27e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.119+0000 7f6b47fff640 1 --2- 172.21.15.28:0/2280621403 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55abfb27c320 0x55abfb27e7a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e309b9fa1afe6b49 server_cookie=31723d2b4053a718 in_seq=0 out_seq=0 2024-04-25T01:39:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.119+0000 7f6b277fe640 10 client.18063 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:39:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.119+0000 7f6b277fe640 1 -- 172.21.15.28:0/2280621403 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55abfb1d3bb0 con 0x55abfb27c320 2024-04-25T01:39:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.119+0000 7f6b277fe640 10 client.18063 handle_command_reply: tid=0 2024-04-25T01:39:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.120+0000 7f6b5158a740 1 client.18063 shutdown 2024-04-25T01:39:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.120+0000 7f6b5158a740 1 -- 172.21.15.28:0/2280621403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261ea0 msgr2=0x55abfb269170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.120+0000 7f6b5158a740 1 --2- 172.21.15.28:0/2280621403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261ea0 0x55abfb269170 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f6b30010790 tx=0x7f6b30010c50 comp rx=0 tx=0).stop 2024-04-25T01:39:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.120+0000 7f6b5158a740 1 -- 172.21.15.28:0/2280621403 shutdown_connections 2024-04-25T01:39:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.120+0000 7f6b5158a740 1 --2- 172.21.15.28:0/2280621403 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55abfb27c320 0x55abfb27e7a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.120+0000 7f6b5158a740 1 --2- 172.21.15.28:0/2280621403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb26c570 0x55abfb2696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.120+0000 7f6b277fe640 0 client.18063 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:39:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.120+0000 7f6b5158a740 1 --2- 172.21.15.28:0/2280621403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261ea0 0x55abfb269170 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.120+0000 7f6b5158a740 1 --2- 172.21.15.28:0/2280621403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb261580 0x55abfb26be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.120+0000 7f6b5158a740 1 -- 172.21.15.28:0/2280621403 >> 172.21.15.28:0/2280621403 conn(0x55abfb25d8a0 msgr2=0x55abfb25cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.120+0000 7f6b5158a740 1 -- 172.21.15.28:0/2280621403 shutdown_connections 2024-04-25T01:39:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.120+0000 7f6b5158a740 1 -- 172.21.15.28:0/2280621403 wait complete. 2024-04-25T01:39:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.120+0000 7f6b5158a740 20 client.18063 trim_cache size 0 max 16384 2024-04-25T01:39:49.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.133+0000 7f6b5158a740 1 Processor -- start 2024-04-25T01:39:49.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.133+0000 7f6b5158a740 1 -- start start 2024-04-25T01:39:49.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.133+0000 7f6b5158a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb261580 0x55abfb277e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.133+0000 7f6b5158a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb261ea0 0x55abfb26b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.133+0000 7f6b5158a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb26c570 0x55abfb284070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.133+0000 7f6b5158a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481aeab0 con 0x55abfb26c570 2024-04-25T01:39:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.133+0000 7f6b5158a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481b05f0 con 0x55abfb261580 2024-04-25T01:39:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.133+0000 7f6b5158a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481af180 con 0x55abfb261ea0 2024-04-25T01:39:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.133+0000 7f6b4ce20640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb26c570 0x55abfb284070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.133+0000 7f6b4ce20640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb26c570 0x55abfb284070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40268/0 (socket says 172.21.15.28:40268) 2024-04-25T01:39:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.133+0000 7f6b4ce20640 1 -- 172.21.15.28:0/1470218609 learned_addr learned my addr 172.21.15.28:0/1470218609 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.134+0000 7f6b477fe640 1 --2- 172.21.15.28:0/1470218609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb261ea0 0x55abfb26b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.134+0000 7f6b47fff640 1 --2- 172.21.15.28:0/1470218609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb261580 0x55abfb277e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.134+0000 7f6b4ce20640 1 -- 172.21.15.28:0/1470218609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb261ea0 msgr2=0x55abfb26b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.134+0000 7f6b4ce20640 1 --2- 172.21.15.28:0/1470218609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb261ea0 0x55abfb26b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.134+0000 7f6b4ce20640 1 -- 172.21.15.28:0/1470218609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb261580 msgr2=0x55abfb277e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.134+0000 7f6b4ce20640 1 --2- 172.21.15.28:0/1470218609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb261580 0x55abfb277e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.134+0000 7f6b4ce20640 1 -- 172.21.15.28:0/1470218609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abfafa47d0 con 0x55abfb26c570 2024-04-25T01:39:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.134+0000 7f6b4ce20640 1 --2- 172.21.15.28:0/1470218609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb26c570 0x55abfb284070 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f6b40011b30 tx=0x7f6b4000a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=276f06f136a9144 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.134+0000 7f6b3dffb640 1 -- 172.21.15.28:0/1470218609 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4000fd80 con 0x55abfb26c570 2024-04-25T01:39:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.134+0000 7f6b3dffb640 1 -- 172.21.15.28:0/1470218609 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b40003970 con 0x55abfb26c570 2024-04-25T01:39:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.134+0000 7f6b3dffb640 1 -- 172.21.15.28:0/1470218609 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b400044e0 con 0x55abfb26c570 2024-04-25T01:39:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.135+0000 7f6b5158a740 1 -- 172.21.15.28:0/1470218609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb26c570 msgr2=0x55abfb284070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.135+0000 7f6b5158a740 1 --2- 172.21.15.28:0/1470218609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb26c570 0x55abfb284070 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f6b40011b30 tx=0x7f6b4000a890 comp rx=0 tx=0).stop 2024-04-25T01:39:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.135+0000 7f6b5158a740 1 -- 172.21.15.28:0/1470218609 shutdown_connections 2024-04-25T01:39:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.135+0000 7f6b5158a740 1 --2- 172.21.15.28:0/1470218609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb26c570 0x55abfb284070 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.135+0000 7f6b5158a740 1 --2- 172.21.15.28:0/1470218609 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb261ea0 0x55abfb26b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.135+0000 7f6b5158a740 1 --2- 172.21.15.28:0/1470218609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb261580 0x55abfb277e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.135+0000 7f6b5158a740 1 -- 172.21.15.28:0/1470218609 >> 172.21.15.28:0/1470218609 conn(0x55abfb25d8a0 msgr2=0x55abfb25cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.135+0000 7f6b5158a740 1 -- 172.21.15.28:0/1470218609 shutdown_connections 2024-04-25T01:39:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.135+0000 7f6b5158a740 1 -- 172.21.15.28:0/1470218609 wait complete. 2024-04-25T01:39:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.135+0000 7f6b5158a740 1 -- start start 2024-04-25T01:39:49.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.135+0000 7f6b5158a740 1 Processor -- start 2024-04-25T01:39:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.135+0000 7f6b5158a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261580 0x55abfb2ded00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.135+0000 7f6b5158a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb261ea0 0x55abfb2df240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.135+0000 7f6b5158a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb26c570 0x55abfb265c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.135+0000 7f6b5158a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481aff20 con 0x55abfb261580 2024-04-25T01:39:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.135+0000 7f6b5158a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481af850 con 0x55abfb26c570 2024-04-25T01:39:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.135+0000 7f6b5158a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481aca90 con 0x55abfb261ea0 2024-04-25T01:39:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.135+0000 7f6b5158a740 10 client.0 _ll_register_callbacks cb 0x7f6b3000e3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b47fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261580 0x55abfb2ded00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b47fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261580 0x55abfb2ded00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40274/0 (socket says 172.21.15.28:40274) 2024-04-25T01:39:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b47fff640 1 -- 172.21.15.28:0/942573600 learned_addr learned my addr 172.21.15.28:0/942573600 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b477fe640 1 --2- 172.21.15.28:0/942573600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb261ea0 0x55abfb2df240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b4ce20640 1 --2- 172.21.15.28:0/942573600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb26c570 0x55abfb265c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b47fff640 1 -- 172.21.15.28:0/942573600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb261ea0 msgr2=0x55abfb2df240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b47fff640 1 --2- 172.21.15.28:0/942573600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb261ea0 0x55abfb2df240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b47fff640 1 -- 172.21.15.28:0/942573600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb26c570 msgr2=0x55abfb265c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b47fff640 1 --2- 172.21.15.28:0/942573600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb26c570 0x55abfb265c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b47fff640 1 -- 172.21.15.28:0/942573600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abfb266340 con 0x55abfb261580 2024-04-25T01:39:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b4ce20640 1 --2- 172.21.15.28:0/942573600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb26c570 0x55abfb265c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b47fff640 1 --2- 172.21.15.28:0/942573600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261580 0x55abfb2ded00 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f6b380eda40 tx=0x7f6b38034000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b02bc91f5842e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b277fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b277fe640 1 -- 172.21.15.28:0/942573600 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b380c7070 con 0x55abfb261580 2024-04-25T01:39:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b277fe640 1 -- 172.21.15.28:0/942573600 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f6b380ed6c0 con 0x55abfb261580 2024-04-25T01:39:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b5158a740 1 -- 172.21.15.28:0/942573600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b40003970 con 0x55abfb261580 2024-04-25T01:39:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b277fe640 1 -- 172.21.15.28:0/942573600 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6b38049040 con 0x55abfb261580 2024-04-25T01:39:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b5158a740 10 client.18069 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b5158a740 1 -- 172.21.15.28:0/942573600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55abfb2e6500 con 0x55abfb261580 2024-04-25T01:39:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.136+0000 7f6b277fe640 1 -- 172.21.15.28:0/942573600 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b40003970 con 0x55abfb261580 2024-04-25T01:39:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.137+0000 7f6b277fe640 1 -- 172.21.15.28:0/942573600 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6b380b8880 con 0x55abfb261580 2024-04-25T01:39:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.137+0000 7f6b5158a740 10 client.18069 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.137+0000 7f6b5158a740 10 client.18069 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:39:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.137+0000 7f6b5158a740 20 client.18069 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.137+0000 7f6b5158a740 1 --2- 172.21.15.28:0/942573600 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55abfb2e68b0 0x55abfb2e8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.137+0000 7f6b5158a740 4 client.18069 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.137+0000 7f6b5158a740 1 -- 172.21.15.28:0/942573600 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55abfb1cdf00 con 0x55abfb2e68b0 2024-04-25T01:39:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.137+0000 7f6b477fe640 1 --2- 172.21.15.28:0/942573600 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55abfb2e68b0 0x55abfb2e8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.137+0000 7f6b277fe640 10 client.18069 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:39:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.137+0000 7f6b477fe640 1 --2- 172.21.15.28:0/942573600 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55abfb2e68b0 0x55abfb2e8ca0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8aca19e689aeca09 server_cookie=bb1be2ad5b111bc5 in_seq=0 out_seq=0 2024-04-25T01:39:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.138+0000 7f6b277fe640 1 -- 172.21.15.28:0/942573600 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55abfb1cdf00 con 0x55abfb2e68b0 2024-04-25T01:39:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.138+0000 7f6b277fe640 10 client.18069 handle_command_reply: tid=0 2024-04-25T01:39:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.138+0000 7f6b5158a740 1 client.18069 shutdown 2024-04-25T01:39:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.138+0000 7f6b5158a740 1 -- 172.21.15.28:0/942573600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261580 msgr2=0x55abfb2ded00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.138+0000 7f6b5158a740 1 --2- 172.21.15.28:0/942573600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261580 0x55abfb2ded00 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f6b380eda40 tx=0x7f6b38034000 comp rx=0 tx=0).stop 2024-04-25T01:39:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.138+0000 7f6b5158a740 1 -- 172.21.15.28:0/942573600 shutdown_connections 2024-04-25T01:39:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.138+0000 7f6b5158a740 1 --2- 172.21.15.28:0/942573600 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55abfb2e68b0 0x55abfb2e8ca0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.138+0000 7f6b5158a740 1 --2- 172.21.15.28:0/942573600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55abfb26c570 0x55abfb265c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.138+0000 7f6b5158a740 1 --2- 172.21.15.28:0/942573600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55abfb261ea0 0x55abfb2df240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.138+0000 7f6b277fe640 0 client.18069 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:39:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.138+0000 7f6b5158a740 1 --2- 172.21.15.28:0/942573600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55abfb261580 0x55abfb2ded00 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.138+0000 7f6b5158a740 1 -- 172.21.15.28:0/942573600 >> 172.21.15.28:0/942573600 conn(0x55abfb25d8a0 msgr2=0x55abfb276e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.138+0000 7f6b5158a740 1 -- 172.21.15.28:0/942573600 shutdown_connections 2024-04-25T01:39:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.138+0000 7f6b5158a740 1 -- 172.21.15.28:0/942573600 wait complete. 2024-04-25T01:39:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.138+0000 7f6b5158a740 20 client.18069 trim_cache size 0 max 16384 2024-04-25T01:39:49.128 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.138+0000 7f6b4e883640 1 -- 172.21.15.28:0/2499535361 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6b340a2d00 msgr2=0x7f6b340a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.138+0000 7f6b4e883640 1 --2- 172.21.15.28:0/2499535361 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6b340a2d00 0x7f6b340a51c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6b4000a780 tx=0x7f6b4000a4d0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.138+0000 7f6b4e883640 1 -- 172.21.15.28:0/2499535361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b48101710 msgr2=0x7f6b48197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.138+0000 7f6b4e883640 1 --2- 172.21.15.28:0/2499535361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b48101710 0x7f6b48197fa0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f6b3802a940 tx=0x7f6b38003c00 comp rx=0 tx=0).stop 2024-04-25T01:39:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.139+0000 7f6b4e883640 1 -- 172.21.15.28:0/2499535361 shutdown_connections 2024-04-25T01:39:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.139+0000 7f6b4e883640 1 --2- 172.21.15.28:0/2499535361 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6b340a2d00 0x7f6b340a51c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.139+0000 7f6b4e883640 1 --2- 172.21.15.28:0/2499535361 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b48107720 0x7f6b4819c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.139+0000 7f6b4e883640 1 --2- 172.21.15.28:0/2499535361 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6b481020d0 0x7f6b481984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.139+0000 7f6b4e883640 1 --2- 172.21.15.28:0/2499535361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b48101710 0x7f6b48197fa0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.139+0000 7f6b4e883640 1 -- 172.21.15.28:0/2499535361 >> 172.21.15.28:0/2499535361 conn(0x7f6b480fd180 msgr2=0x7f6b480fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.139+0000 7f6b4e883640 1 -- 172.21.15.28:0/2499535361 shutdown_connections 2024-04-25T01:39:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.139+0000 7f6b4e883640 1 -- 172.21.15.28:0/2499535361 wait complete. 2024-04-25T01:39:49.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:49 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:39:49.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:49 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:49.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:49 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:39:49.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:49 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:49.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:49 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:39:49.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:49 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3900255422' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:39:49.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:39:49.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.253+0000 7fc990f9e640 1 -- 172.21.15.28:0/2542454868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc98c100fd0 msgr2=0x7fc98c10d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:49.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.253+0000 7fc990f9e640 1 --2- 172.21.15.28:0/2542454868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc98c100fd0 0x7fc98c10d4e0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fc984009cd0 tx=0x7fc98402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.253+0000 7fc990f9e640 1 -- 172.21.15.28:0/2542454868 shutdown_connections 2024-04-25T01:39:49.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.253+0000 7fc990f9e640 1 --2- 172.21.15.28:0/2542454868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc98c100fd0 0x7fc98c10d4e0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.253+0000 7fc990f9e640 1 --2- 172.21.15.28:0/2542454868 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc98c100610 0x7fc98c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.253+0000 7fc990f9e640 1 --2- 172.21.15.28:0/2542454868 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc98c1066b0 0x7fc98c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.253+0000 7fc990f9e640 1 -- 172.21.15.28:0/2542454868 >> 172.21.15.28:0/2542454868 conn(0x7fc98c077740 msgr2=0x7fc98c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.253+0000 7fc990f9e640 1 -- 172.21.15.28:0/2542454868 shutdown_connections 2024-04-25T01:39:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.254+0000 7fc990f9e640 1 -- 172.21.15.28:0/2542454868 wait complete. 2024-04-25T01:39:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.254+0000 7fc990f9e640 1 Processor -- start 2024-04-25T01:39:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.254+0000 7fc990f9e640 1 -- start start 2024-04-25T01:39:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc990f9e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc98c100610 0x7fc98c197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc990f9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc98c100fd0 0x7fc98c1983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc990f9e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc98c1066b0 0x7fc98c19c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc990f9e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c111b30 con 0x7fc98c100fd0 2024-04-25T01:39:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc990f9e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c1119b0 con 0x7fc98c100610 2024-04-25T01:39:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc990f9e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c111cb0 con 0x7fc98c1066b0 2024-04-25T01:39:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc981d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc98c100fd0 0x7fc98c1983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc981d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc98c100fd0 0x7fc98c1983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40288/0 (socket says 172.21.15.28:40288) 2024-04-25T01:39:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc981d9b640 1 -- 172.21.15.28:0/933413878 learned_addr learned my addr 172.21.15.28:0/933413878 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc98a59c640 1 --2- 172.21.15.28:0/933413878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc98c100610 0x7fc98c197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc98ad9d640 1 --2- 172.21.15.28:0/933413878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc98c1066b0 0x7fc98c19c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc981d9b640 1 -- 172.21.15.28:0/933413878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc98c1066b0 msgr2=0x7fc98c19c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc981d9b640 1 --2- 172.21.15.28:0/933413878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc98c1066b0 0x7fc98c19c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc981d9b640 1 -- 172.21.15.28:0/933413878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc98c100610 msgr2=0x7fc98c197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc981d9b640 1 --2- 172.21.15.28:0/933413878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc98c100610 0x7fc98c197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc981d9b640 1 -- 172.21.15.28:0/933413878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc98c19ce60 con 0x7fc98c100fd0 2024-04-25T01:39:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc98ad9d640 1 --2- 172.21.15.28:0/933413878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc98c1066b0 0x7fc98c19c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc98a59c640 1 --2- 172.21.15.28:0/933413878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc98c100610 0x7fc98c197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc981d9b640 1 --2- 172.21.15.28:0/933413878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc98c100fd0 0x7fc98c1983b0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fc97400c920 tx=0x7fc97400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc983fff640 1 -- 172.21.15.28:0/933413878 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974007c70 con 0x7fc98c100fd0 2024-04-25T01:39:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc983fff640 1 -- 172.21.15.28:0/933413878 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc974007e10 con 0x7fc98c100fd0 2024-04-25T01:39:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc990f9e640 1 -- 172.21.15.28:0/933413878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc98c19d150 con 0x7fc98c100fd0 2024-04-25T01:39:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.255+0000 7fc983fff640 1 -- 172.21.15.28:0/933413878 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974018890 con 0x7fc98c100fd0 2024-04-25T01:39:49.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.256+0000 7fc990f9e640 1 -- 172.21.15.28:0/933413878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc98c1a49e0 con 0x7fc98c100fd0 2024-04-25T01:39:49.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.258+0000 7fc983fff640 1 -- 172.21.15.28:0/933413878 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc974018a70 con 0x7fc98c100fd0 2024-04-25T01:39:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.258+0000 7fc993d84740 1 Processor -- start 2024-04-25T01:39:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc993d84740 1 -- start start 2024-04-25T01:39:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc993d84740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8cb580 0x55668f8cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc993d84740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 0x55668f8d6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc993d84740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8d6570 0x55668f8d8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc993d84740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55668f830670 con 0x55668f8cbea0 2024-04-25T01:39:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc993d84740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55668f7d8970 con 0x55668f8d6570 2024-04-25T01:39:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc993d84740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55668f82c630 con 0x55668f8cb580 2024-04-25T01:39:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc98a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 0x55668f8d6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc983fff640 1 --2- 172.21.15.28:0/933413878 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc9780a2d00 0x7fc9780a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc98a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 0x55668f8d6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40296/0 (socket says 172.21.15.28:40296) 2024-04-25T01:39:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc98a59c640 1 -- 172.21.15.28:0/3409122717 learned_addr learned my addr 172.21.15.28:0/3409122717 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc98ad9d640 1 --2- 172.21.15.28:0/3409122717 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8cb580 0x55668f8cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc981d9b640 1 --2- 172.21.15.28:0/3409122717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8d6570 0x55668f8d8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc98ad9d640 1 --2- 172.21.15.28:0/933413878 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc9780a2d00 0x7fc9780a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc98a59c640 1 -- 172.21.15.28:0/3409122717 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8cb580 msgr2=0x55668f8cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc98a59c640 1 --2- 172.21.15.28:0/3409122717 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8cb580 0x55668f8cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc983fff640 1 -- 172.21.15.28:0/933413878 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc9740cf9f0 con 0x7fc98c100fd0 2024-04-25T01:39:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc98a59c640 1 -- 172.21.15.28:0/3409122717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8d6570 msgr2=0x55668f8d8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc98a59c640 1 --2- 172.21.15.28:0/3409122717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8d6570 0x55668f8d8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc98a59c640 1 -- 172.21.15.28:0/3409122717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55668f7fc400 con 0x55668f8cbea0 2024-04-25T01:39:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.259+0000 7fc981d9b640 1 --2- 172.21.15.28:0/3409122717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8d6570 0x55668f8d8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.260+0000 7fc98a59c640 1 --2- 172.21.15.28:0/3409122717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 0x55668f8d6030 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fc97c009cf0 tx=0x7fc97c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc182d847e4b680d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.260+0000 7fc980d99640 1 -- 172.21.15.28:0/3409122717 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c012450 con 0x55668f8cbea0 2024-04-25T01:39:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.260+0000 7fc98ad9d640 1 --2- 172.21.15.28:0/933413878 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc9780a2d00 0x7fc9780a51c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc984006010 tx=0x7fc9840095c0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.260+0000 7fc980d99640 1 -- 172.21.15.28:0/3409122717 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc97c012ad0 con 0x55668f8cbea0 2024-04-25T01:39:49.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.260+0000 7fc980d99640 1 -- 172.21.15.28:0/3409122717 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c00fbf0 con 0x55668f8cbea0 2024-04-25T01:39:49.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.261+0000 7fc993d84740 1 -- 172.21.15.28:0/3409122717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 msgr2=0x55668f8d6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.261+0000 7fc993d84740 1 --2- 172.21.15.28:0/3409122717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 0x55668f8d6030 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fc97c009cf0 tx=0x7fc97c0117b0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.261+0000 7fc993d84740 1 -- 172.21.15.28:0/3409122717 shutdown_connections 2024-04-25T01:39:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.261+0000 7fc993d84740 1 --2- 172.21.15.28:0/3409122717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8d6570 0x55668f8d8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.261+0000 7fc993d84740 1 --2- 172.21.15.28:0/3409122717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 0x55668f8d6030 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.261+0000 7fc993d84740 1 --2- 172.21.15.28:0/3409122717 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8cb580 0x55668f8cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.261+0000 7fc993d84740 1 -- 172.21.15.28:0/3409122717 >> 172.21.15.28:0/3409122717 conn(0x55668f8c78a0 msgr2=0x55668f8c7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.261+0000 7fc993d84740 1 -- 172.21.15.28:0/3409122717 shutdown_connections 2024-04-25T01:39:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.261+0000 7fc993d84740 1 -- 172.21.15.28:0/3409122717 wait complete. 2024-04-25T01:39:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.261+0000 7fc993d84740 1 -- start start 2024-04-25T01:39:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.261+0000 7fc993d84740 1 Processor -- start 2024-04-25T01:39:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.261+0000 7fc993d84740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8cb580 0x55668f8d5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.261+0000 7fc993d84740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 0x55668f8d3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.261+0000 7fc993d84740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8d6570 0x55668f8d3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.261+0000 7fc993d84740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55668f8d9360 con 0x55668f8cbea0 2024-04-25T01:39:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.261+0000 7fc993d84740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55668f7ec050 con 0x55668f8cb580 2024-04-25T01:39:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.261+0000 7fc993d84740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55668f8d94e0 con 0x55668f8d6570 2024-04-25T01:39:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.261+0000 7fc993d84740 10 client.0 _ll_register_callbacks cb 0x55668f543be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.261+0000 7fc981d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 0x55668f8d3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.262+0000 7fc981d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 0x55668f8d3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40304/0 (socket says 172.21.15.28:40304) 2024-04-25T01:39:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.262+0000 7fc981d9b640 1 -- 172.21.15.28:0/3585295976 learned_addr learned my addr 172.21.15.28:0/3585295976 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.262+0000 7fc98ad9d640 1 --2- 172.21.15.28:0/3585295976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8d6570 0x55668f8d3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.262+0000 7fc98a59c640 1 --2- 172.21.15.28:0/3585295976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8cb580 0x55668f8d5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.262+0000 7fc981d9b640 1 -- 172.21.15.28:0/3585295976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8d6570 msgr2=0x55668f8d3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.262+0000 7fc981d9b640 1 --2- 172.21.15.28:0/3585295976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8d6570 0x55668f8d3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.262+0000 7fc981d9b640 1 -- 172.21.15.28:0/3585295976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8cb580 msgr2=0x55668f8d5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.262+0000 7fc981d9b640 1 --2- 172.21.15.28:0/3585295976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8cb580 0x55668f8d5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.262+0000 7fc981d9b640 1 -- 172.21.15.28:0/3585295976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55668f7bddc0 con 0x55668f8cbea0 2024-04-25T01:39:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.262+0000 7fc981d9b640 1 --2- 172.21.15.28:0/3585295976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 0x55668f8d3150 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fc97409b870 tx=0x7fc97401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f79d8b83b1f2b049 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.262+0000 7fc971ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.262+0000 7fc971ffb640 1 -- 172.21.15.28:0/3585295976 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974014210 con 0x55668f8cbea0 2024-04-25T01:39:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.262+0000 7fc971ffb640 1 -- 172.21.15.28:0/3585295976 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc97401dd40 con 0x55668f8cbea0 2024-04-25T01:39:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.262+0000 7fc993d84740 1 -- 172.21.15.28:0/3585295976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc97c012ad0 con 0x55668f8cbea0 2024-04-25T01:39:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.262+0000 7fc971ffb640 1 -- 172.21.15.28:0/3585295976 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc97409bb70 con 0x55668f8cbea0 2024-04-25T01:39:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.262+0000 7fc993d84740 10 client.18081 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.262+0000 7fc993d84740 1 -- 172.21.15.28:0/3585295976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55668f5443f0 con 0x55668f8cbea0 2024-04-25T01:39:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.262+0000 7fc971ffb640 1 -- 172.21.15.28:0/3585295976 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc97c012ad0 con 0x55668f8cbea0 2024-04-25T01:39:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.263+0000 7fc971ffb640 1 -- 172.21.15.28:0/3585295976 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc9740a4350 con 0x55668f8cbea0 2024-04-25T01:39:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.263+0000 7fc993d84740 10 client.18081 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.263+0000 7fc993d84740 10 client.18081 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.263+0000 7fc993d84740 20 client.18081 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.263+0000 7fc993d84740 1 --2- 172.21.15.28:0/3585295976 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55668f8e6590 0x55668f8e89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.263+0000 7fc993d84740 4 client.18081 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.263+0000 7fc993d84740 1 -- 172.21.15.28:0/3585295976 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55668f83dbb0 con 0x55668f8e6590 2024-04-25T01:39:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.263+0000 7fc98a59c640 1 --2- 172.21.15.28:0/3585295976 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55668f8e6590 0x55668f8e89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.264+0000 7fc971ffb640 10 client.18081 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.264+0000 7fc98a59c640 1 --2- 172.21.15.28:0/3585295976 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55668f8e6590 0x55668f8e89a0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c0096f4a56bc2ab server_cookie=22c80805e9edc910 in_seq=0 out_seq=0 2024-04-25T01:39:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.264+0000 7fc971ffb640 1 -- 172.21.15.28:0/3585295976 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55668f83dbb0 con 0x55668f8e6590 2024-04-25T01:39:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.264+0000 7fc971ffb640 10 client.18081 handle_command_reply: tid=0 2024-04-25T01:39:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.264+0000 7fc993d84740 1 client.18081 shutdown 2024-04-25T01:39:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.264+0000 7fc993d84740 1 -- 172.21.15.28:0/3585295976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 msgr2=0x55668f8d3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.264+0000 7fc993d84740 1 --2- 172.21.15.28:0/3585295976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 0x55668f8d3150 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fc97409b870 tx=0x7fc97401c000 comp rx=0 tx=0).stop 2024-04-25T01:39:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.264+0000 7fc993d84740 1 -- 172.21.15.28:0/3585295976 shutdown_connections 2024-04-25T01:39:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.264+0000 7fc993d84740 1 --2- 172.21.15.28:0/3585295976 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55668f8e6590 0x55668f8e89a0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.264+0000 7fc993d84740 1 --2- 172.21.15.28:0/3585295976 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8d6570 0x55668f8d3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.264+0000 7fc993d84740 1 --2- 172.21.15.28:0/3585295976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 0x55668f8d3150 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.264+0000 7fc993d84740 1 --2- 172.21.15.28:0/3585295976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8cb580 0x55668f8d5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.264+0000 7fc971ffb640 0 client.18081 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:49.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.264+0000 7fc993d84740 1 -- 172.21.15.28:0/3585295976 >> 172.21.15.28:0/3585295976 conn(0x55668f8c78a0 msgr2=0x55668f8c6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:49.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.264+0000 7fc993d84740 1 -- 172.21.15.28:0/3585295976 shutdown_connections 2024-04-25T01:39:49.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.264+0000 7fc993d84740 1 -- 172.21.15.28:0/3585295976 wait complete. 2024-04-25T01:39:49.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.265+0000 7fc993d84740 20 client.18081 trim_cache size 0 max 16384 2024-04-25T01:39:49.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.278+0000 7fc993d84740 1 Processor -- start 2024-04-25T01:39:49.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.278+0000 7fc993d84740 1 -- start start 2024-04-25T01:39:49.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.278+0000 7fc993d84740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8cb580 0x55668f8c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.278+0000 7fc993d84740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8cbea0 0x55668f8ee270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.278+0000 7fc993d84740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8d6570 0x55668f8ee7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.278+0000 7fc993d84740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c1ae940 con 0x55668f8d6570 2024-04-25T01:39:49.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.278+0000 7fc993d84740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c1b0480 con 0x55668f8cb580 2024-04-25T01:39:49.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.278+0000 7fc993d84740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c1af010 con 0x55668f8cbea0 2024-04-25T01:39:49.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.278+0000 7fc98ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8d6570 0x55668f8ee7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.278+0000 7fc98ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8d6570 0x55668f8ee7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40310/0 (socket says 172.21.15.28:40310) 2024-04-25T01:39:49.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.278+0000 7fc98ad9d640 1 -- 172.21.15.28:0/3388830891 learned_addr learned my addr 172.21.15.28:0/3388830891 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:49.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.278+0000 7fc981d9b640 1 --2- 172.21.15.28:0/3388830891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8cbea0 0x55668f8ee270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.278+0000 7fc98ad9d640 1 -- 172.21.15.28:0/3388830891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8cbea0 msgr2=0x55668f8ee270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:49.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.278+0000 7fc98a59c640 1 --2- 172.21.15.28:0/3388830891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8cb580 0x55668f8c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.278+0000 7fc98ad9d640 1 --2- 172.21.15.28:0/3388830891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8cbea0 0x55668f8ee270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.278+0000 7fc98ad9d640 1 -- 172.21.15.28:0/3388830891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8cb580 msgr2=0x55668f8c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:49.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.278+0000 7fc98ad9d640 1 --2- 172.21.15.28:0/3388830891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8cb580 0x55668f8c72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.278+0000 7fc98ad9d640 1 -- 172.21.15.28:0/3388830891 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55668f60e7d0 con 0x55668f8d6570 2024-04-25T01:39:49.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.278+0000 7fc98ad9d640 1 --2- 172.21.15.28:0/3388830891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8d6570 0x55668f8ee7b0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fc984009cd0 tx=0x7fc984042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7944c04e3c4986d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:49.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.279+0000 7fc973fff640 1 -- 172.21.15.28:0/3388830891 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984043070 con 0x55668f8d6570 2024-04-25T01:39:49.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.279+0000 7fc973fff640 1 -- 172.21.15.28:0/3388830891 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc984036040 con 0x55668f8d6570 2024-04-25T01:39:49.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.279+0000 7fc973fff640 1 -- 172.21.15.28:0/3388830891 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9840305a0 con 0x55668f8d6570 2024-04-25T01:39:49.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.279+0000 7fc993d84740 1 -- 172.21.15.28:0/3388830891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8d6570 msgr2=0x55668f8ee7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:49.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.279+0000 7fc993d84740 1 --2- 172.21.15.28:0/3388830891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8d6570 0x55668f8ee7b0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fc984009cd0 tx=0x7fc984042000 comp rx=0 tx=0).stop 2024-04-25T01:39:49.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.279+0000 7fc993d84740 1 -- 172.21.15.28:0/3388830891 shutdown_connections 2024-04-25T01:39:49.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.279+0000 7fc993d84740 1 --2- 172.21.15.28:0/3388830891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8d6570 0x55668f8ee7b0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.279+0000 7fc993d84740 1 --2- 172.21.15.28:0/3388830891 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8cbea0 0x55668f8ee270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.279+0000 7fc993d84740 1 --2- 172.21.15.28:0/3388830891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8cb580 0x55668f8c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.279+0000 7fc993d84740 1 -- 172.21.15.28:0/3388830891 >> 172.21.15.28:0/3388830891 conn(0x55668f8c78a0 msgr2=0x55668f8e1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:49.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.279+0000 7fc993d84740 1 -- 172.21.15.28:0/3388830891 shutdown_connections 2024-04-25T01:39:49.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.279+0000 7fc993d84740 1 -- 172.21.15.28:0/3388830891 wait complete. 2024-04-25T01:39:49.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.279+0000 7fc993d84740 1 -- start start 2024-04-25T01:39:49.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.279+0000 7fc993d84740 1 Processor -- start 2024-04-25T01:39:49.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc993d84740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8cb580 0x55668f8de800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc993d84740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 0x55668f8dbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc993d84740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8d6570 0x55668f8dc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc993d84740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c1afdb0 con 0x55668f8cbea0 2024-04-25T01:39:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc993d84740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c1af6e0 con 0x55668f8d6570 2024-04-25T01:39:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc993d84740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c1ac920 con 0x55668f8cb580 2024-04-25T01:39:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc993d84740 10 client.0 _ll_register_callbacks cb 0x7fc97400a560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc981d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 0x55668f8dbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc981d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 0x55668f8dbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40318/0 (socket says 172.21.15.28:40318) 2024-04-25T01:39:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc981d9b640 1 -- 172.21.15.28:0/2510089211 learned_addr learned my addr 172.21.15.28:0/2510089211 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc98a59c640 1 --2- 172.21.15.28:0/2510089211 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8cb580 0x55668f8de800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc981d9b640 1 -- 172.21.15.28:0/2510089211 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8cb580 msgr2=0x55668f8de800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc981d9b640 1 --2- 172.21.15.28:0/2510089211 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8cb580 0x55668f8de800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc981d9b640 1 -- 172.21.15.28:0/2510089211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8d6570 msgr2=0x55668f8dc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc981d9b640 1 --2- 172.21.15.28:0/2510089211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8d6570 0x55668f8dc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc981d9b640 1 -- 172.21.15.28:0/2510089211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55668f8dc5f0 con 0x55668f8cbea0 2024-04-25T01:39:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc981d9b640 1 --2- 172.21.15.28:0/2510089211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 0x55668f8dbb70 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fc9740984f0 tx=0x7fc97400de20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a158d38b17eff9df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc971ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc971ffb640 1 -- 172.21.15.28:0/2510089211 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974094020 con 0x55668f8cbea0 2024-04-25T01:39:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc971ffb640 1 -- 172.21.15.28:0/2510089211 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc9740cd020 con 0x55668f8cbea0 2024-04-25T01:39:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.280+0000 7fc993d84740 1 -- 172.21.15.28:0/2510089211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc984036040 con 0x55668f8cbea0 2024-04-25T01:39:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.281+0000 7fc971ffb640 1 -- 172.21.15.28:0/2510089211 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc97409b6b0 con 0x55668f8cbea0 2024-04-25T01:39:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.281+0000 7fc993d84740 10 client.18087 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.281+0000 7fc993d84740 1 -- 172.21.15.28:0/2510089211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55668f94c710 con 0x55668f8cbea0 2024-04-25T01:39:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.281+0000 7fc971ffb640 1 -- 172.21.15.28:0/2510089211 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc984036040 con 0x55668f8cbea0 2024-04-25T01:39:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.281+0000 7fc971ffb640 1 -- 172.21.15.28:0/2510089211 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc9740978c0 con 0x55668f8cbea0 2024-04-25T01:39:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.281+0000 7fc993d84740 10 client.18087 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.281+0000 7fc993d84740 10 client.18087 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.281+0000 7fc993d84740 20 client.18087 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:49.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.281+0000 7fc993d84740 1 --2- 172.21.15.28:0/2510089211 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55668f94cd30 0x55668f94f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:49.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.281+0000 7fc993d84740 4 client.18087 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:39:49.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.281+0000 7fc993d84740 1 -- 172.21.15.28:0/2510089211 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55668f837f00 con 0x55668f94cd30 2024-04-25T01:39:49.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.281+0000 7fc98a59c640 1 --2- 172.21.15.28:0/2510089211 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55668f94cd30 0x55668f94f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:49.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.281+0000 7fc971ffb640 10 client.18087 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:49.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.281+0000 7fc98a59c640 1 --2- 172.21.15.28:0/2510089211 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55668f94cd30 0x55668f94f150 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c14421583ac5022b server_cookie=e7e09a7375f97fdf in_seq=0 out_seq=0 2024-04-25T01:39:49.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc971ffb640 1 -- 172.21.15.28:0/2510089211 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55668f837f00 con 0x55668f94cd30 2024-04-25T01:39:49.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc971ffb640 10 client.18087 handle_command_reply: tid=0 2024-04-25T01:39:49.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc993d84740 1 client.18087 shutdown 2024-04-25T01:39:49.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc993d84740 1 -- 172.21.15.28:0/2510089211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 msgr2=0x55668f8dbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:49.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc993d84740 1 --2- 172.21.15.28:0/2510089211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 0x55668f8dbb70 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fc9740984f0 tx=0x7fc97400de20 comp rx=0 tx=0).stop 2024-04-25T01:39:49.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc993d84740 1 -- 172.21.15.28:0/2510089211 shutdown_connections 2024-04-25T01:39:49.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc993d84740 1 --2- 172.21.15.28:0/2510089211 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55668f94cd30 0x55668f94f150 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc993d84740 1 --2- 172.21.15.28:0/2510089211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55668f8d6570 0x55668f8dc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc993d84740 1 --2- 172.21.15.28:0/2510089211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55668f8cbea0 0x55668f8dbb70 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc971ffb640 0 client.18087 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:49.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc993d84740 1 --2- 172.21.15.28:0/2510089211 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55668f8cb580 0x55668f8de800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc993d84740 1 -- 172.21.15.28:0/2510089211 >> 172.21.15.28:0/2510089211 conn(0x55668f8c78a0 msgr2=0x55668f8e1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:49.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc993d84740 1 -- 172.21.15.28:0/2510089211 shutdown_connections 2024-04-25T01:39:49.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc993d84740 1 -- 172.21.15.28:0/2510089211 wait complete. 2024-04-25T01:39:49.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc993d84740 20 client.18087 trim_cache size 0 max 16384 2024-04-25T01:39:49.272 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:39:49.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc990f9e640 1 -- 172.21.15.28:0/933413878 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc9780a2d00 msgr2=0x7fc9780a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:49.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc990f9e640 1 --2- 172.21.15.28:0/933413878 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc9780a2d00 0x7fc9780a51c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc984006010 tx=0x7fc9840095c0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc990f9e640 1 -- 172.21.15.28:0/933413878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc98c100fd0 msgr2=0x7fc98c1983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:49.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc990f9e640 1 --2- 172.21.15.28:0/933413878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc98c100fd0 0x7fc98c1983b0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fc97400c920 tx=0x7fc97400cde0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.282+0000 7fc990f9e640 1 -- 172.21.15.28:0/933413878 shutdown_connections 2024-04-25T01:39:49.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.283+0000 7fc990f9e640 1 --2- 172.21.15.28:0/933413878 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc9780a2d00 0x7fc9780a51c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.283+0000 7fc990f9e640 1 --2- 172.21.15.28:0/933413878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc98c1066b0 0x7fc98c19c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.283+0000 7fc990f9e640 1 --2- 172.21.15.28:0/933413878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc98c100fd0 0x7fc98c1983b0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.283+0000 7fc990f9e640 1 --2- 172.21.15.28:0/933413878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc98c100610 0x7fc98c197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:49.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.283+0000 7fc990f9e640 1 -- 172.21.15.28:0/933413878 >> 172.21.15.28:0/933413878 conn(0x7fc98c077740 msgr2=0x7fc98c0feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:49.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.283+0000 7fc990f9e640 1 -- 172.21.15.28:0/933413878 shutdown_connections 2024-04-25T01:39:49.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:49.283+0000 7fc990f9e640 1 -- 172.21.15.28:0/933413878 wait complete. 2024-04-25T01:39:49.464 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:49 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:39:49.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:49 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:49.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:49 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:39:49.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:49 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:49.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:49 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:39:49.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:49 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3900255422' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:39:49.489 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:49 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:39:49.489 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:49 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:49.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:49 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:39:49.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:49 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:49.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:49 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:39:49.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:49 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3900255422' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:39:50.277 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:39:50.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:39:50.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:39:50.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:50 smithi028 ceph-mon[29390]: pgmap v207: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:39:50.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.415+0000 7fb0e7529640 1 -- 172.21.15.28:0/3714252436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e01020d0 msgr2=0x7fb0e010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.415+0000 7fb0e7529640 1 --2- 172.21.15.28:0/3714252436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e01020d0 0x7fb0e010e5e0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009cd0 tx=0x7fb0d802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.415+0000 7fb0e7529640 1 -- 172.21.15.28:0/3714252436 shutdown_connections 2024-04-25T01:39:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.415+0000 7fb0e7529640 1 --2- 172.21.15.28:0/3714252436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e01020d0 0x7fb0e010e5e0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.415+0000 7fb0e7529640 1 --2- 172.21.15.28:0/3714252436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0e0101710 0x7fb0e0101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.415+0000 7fb0e7529640 1 --2- 172.21.15.28:0/3714252436 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb0e0107720 0x7fb0e0107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.415+0000 7fb0e7529640 1 -- 172.21.15.28:0/3714252436 >> 172.21.15.28:0/3714252436 conn(0x7fb0e00fd180 msgr2=0x7fb0e00ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.415+0000 7fb0e7529640 1 -- 172.21.15.28:0/3714252436 shutdown_connections 2024-04-25T01:39:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.415+0000 7fb0e7529640 1 -- 172.21.15.28:0/3714252436 wait complete. 2024-04-25T01:39:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.416+0000 7fb0e7529640 1 Processor -- start 2024-04-25T01:39:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.416+0000 7fb0e7529640 1 -- start start 2024-04-25T01:39:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.416+0000 7fb0e7529640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e0101710 0x7fb0e0198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.416+0000 7fb0e7529640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0e01020d0 0x7fb0e0198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.416+0000 7fb0e7529640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb0e0107720 0x7fb0e019c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.416+0000 7fb0e7529640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e0111b80 con 0x7fb0e0101710 2024-04-25T01:39:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.416+0000 7fb0e7529640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e0111a00 con 0x7fb0e01020d0 2024-04-25T01:39:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.416+0000 7fb0e7529640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e0111d00 con 0x7fb0e0107720 2024-04-25T01:39:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.416+0000 7fb0e52c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e0101710 0x7fb0e0198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.416+0000 7fb0e52c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e0101710 0x7fb0e0198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40342/0 (socket says 172.21.15.28:40342) 2024-04-25T01:39:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.416+0000 7fb0e52c5640 1 -- 172.21.15.28:0/532728241 learned_addr learned my addr 172.21.15.28:0/532728241 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.416+0000 7fb0e4ac4640 1 --2- 172.21.15.28:0/532728241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0e01020d0 0x7fb0e0198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.416+0000 7fb0e5ac6640 1 --2- 172.21.15.28:0/532728241 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb0e0107720 0x7fb0e019c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.417+0000 7fb0e52c5640 1 -- 172.21.15.28:0/532728241 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb0e0107720 msgr2=0x7fb0e019c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.417+0000 7fb0e52c5640 1 --2- 172.21.15.28:0/532728241 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb0e0107720 0x7fb0e019c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.417+0000 7fb0e52c5640 1 -- 172.21.15.28:0/532728241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0e01020d0 msgr2=0x7fb0e0198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.417+0000 7fb0e52c5640 1 --2- 172.21.15.28:0/532728241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0e01020d0 0x7fb0e0198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.417+0000 7fb0e52c5640 1 -- 172.21.15.28:0/532728241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e019d030 con 0x7fb0e0101710 2024-04-25T01:39:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.417+0000 7fb0e4ac4640 1 --2- 172.21.15.28:0/532728241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0e01020d0 0x7fb0e0198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.417+0000 7fb0e5ac6640 1 --2- 172.21.15.28:0/532728241 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb0e0107720 0x7fb0e019c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.417+0000 7fb0e52c5640 1 --2- 172.21.15.28:0/532728241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e0101710 0x7fb0e0198040 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc00e9f0 tx=0x7fb0dc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.417+0000 7fb0d67fc640 1 -- 172.21.15.28:0/532728241 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc00cd80 con 0x7fb0e0101710 2024-04-25T01:39:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.417+0000 7fb0d67fc640 1 -- 172.21.15.28:0/532728241 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0dc005500 con 0x7fb0e0101710 2024-04-25T01:39:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.417+0000 7fb0e7529640 1 -- 172.21.15.28:0/532728241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e019d320 con 0x7fb0e0101710 2024-04-25T01:39:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.417+0000 7fb0d67fc640 1 -- 172.21.15.28:0/532728241 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc010690 con 0x7fb0e0101710 2024-04-25T01:39:50.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.417+0000 7fb0e7529640 1 -- 172.21.15.28:0/532728241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e01a4bb0 con 0x7fb0e0101710 2024-04-25T01:39:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.419+0000 7fb0d67fc640 1 -- 172.21.15.28:0/532728241 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb0dc0046c0 con 0x7fb0e0101710 2024-04-25T01:39:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.420+0000 7fb0ea239740 1 Processor -- start 2024-04-25T01:39:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.420+0000 7fb0ea239740 1 -- start start 2024-04-25T01:39:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.420+0000 7fb0ea239740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba5a980 0x55ea3ba58cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.420+0000 7fb0ea239740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba59290 0x55ea3ba59710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.420+0000 7fb0ea239740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba64010 0x55ea3ba66400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.420+0000 7fb0ea239740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ea3b9bd2c0 con 0x55ea3ba59290 2024-04-25T01:39:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.420+0000 7fb0ea239740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ea3b965910 con 0x55ea3ba64010 2024-04-25T01:39:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.420+0000 7fb0ea239740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ea3b9b9280 con 0x55ea3ba5a980 2024-04-25T01:39:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.420+0000 7fb0d67fc640 1 --2- 172.21.15.28:0/532728241 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb0cc0a2d00 0x7fb0cc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.421+0000 7fb0e5ac6640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba5a980 0x55ea3ba58cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.421+0000 7fb0e52c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba59290 0x55ea3ba59710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.421+0000 7fb0e5ac6640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba5a980 0x55ea3ba58cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.28:36356/0 (socket says 172.21.15.28:36356) 2024-04-25T01:39:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.421+0000 7fb0e5ac6640 1 -- 172.21.15.28:0/2580941705 learned_addr learned my addr 172.21.15.28:0/2580941705 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.421+0000 7fb0e5ac6640 1 --2- 172.21.15.28:0/532728241 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb0cc0a2d00 0x7fb0cc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.421+0000 7fb0e4ac4640 1 --2- 172.21.15.28:0/2580941705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba64010 0x55ea3ba66400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.421+0000 7fb0d67fc640 1 -- 172.21.15.28:0/532728241 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fb0dc0cfcd0 con 0x7fb0e0101710 2024-04-25T01:39:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.421+0000 7fb0e5ac6640 1 -- 172.21.15.28:0/2580941705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba64010 msgr2=0x55ea3ba66400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.421+0000 7fb0e5ac6640 1 --2- 172.21.15.28:0/2580941705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba64010 0x55ea3ba66400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.421+0000 7fb0e5ac6640 1 -- 172.21.15.28:0/2580941705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba59290 msgr2=0x55ea3ba59710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.421+0000 7fb0e5ac6640 1 --2- 172.21.15.28:0/2580941705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba59290 0x55ea3ba59710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.421+0000 7fb0e5ac6640 1 -- 172.21.15.28:0/2580941705 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea3b976fb0 con 0x55ea3ba5a980 2024-04-25T01:39:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.421+0000 7fb0e5ac6640 1 --2- 172.21.15.28:0/532728241 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb0cc0a2d00 0x7fb0cc0a51c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009a60 tx=0x7fb0d8048000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.421+0000 7fb0e5ac6640 1 --2- 172.21.15.28:0/2580941705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba5a980 0x55ea3ba58cc0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8004af0 tx=0x7fb0d8031c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=822f6e5ae73086bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.422+0000 7fb0cb7fe640 1 -- 172.21.15.28:0/2580941705 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d804e070 con 0x55ea3ba5a980 2024-04-25T01:39:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.422+0000 7fb0cb7fe640 1 -- 172.21.15.28:0/2580941705 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d8036040 con 0x55ea3ba5a980 2024-04-25T01:39:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.422+0000 7fb0cb7fe640 1 -- 172.21.15.28:0/2580941705 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8003ec0 con 0x55ea3ba5a980 2024-04-25T01:39:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.422+0000 7fb0ea239740 1 -- 172.21.15.28:0/2580941705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba5a980 msgr2=0x55ea3ba58cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.422+0000 7fb0ea239740 1 --2- 172.21.15.28:0/2580941705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba5a980 0x55ea3ba58cc0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8004af0 tx=0x7fb0d8031c30 comp rx=0 tx=0).stop 2024-04-25T01:39:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.422+0000 7fb0ea239740 1 -- 172.21.15.28:0/2580941705 shutdown_connections 2024-04-25T01:39:50.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.422+0000 7fb0ea239740 1 --2- 172.21.15.28:0/2580941705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba64010 0x55ea3ba66400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.422+0000 7fb0ea239740 1 --2- 172.21.15.28:0/2580941705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba59290 0x55ea3ba59710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.422+0000 7fb0ea239740 1 --2- 172.21.15.28:0/2580941705 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba5a980 0x55ea3ba58cc0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.422+0000 7fb0ea239740 1 -- 172.21.15.28:0/2580941705 >> 172.21.15.28:0/2580941705 conn(0x55ea3ba21950 msgr2=0x55ea3ba557f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:50.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.422+0000 7fb0ea239740 1 -- 172.21.15.28:0/2580941705 shutdown_connections 2024-04-25T01:39:50.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.422+0000 7fb0ea239740 1 -- 172.21.15.28:0/2580941705 wait complete. 2024-04-25T01:39:50.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.423+0000 7fb0ea239740 1 -- start start 2024-04-25T01:39:50.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.423+0000 7fb0ea239740 1 Processor -- start 2024-04-25T01:39:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.423+0000 7fb0ea239740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba59290 0x55ea3ba60f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.423+0000 7fb0ea239740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba5a980 0x55ea3ba614b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.423+0000 7fb0ea239740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba64010 0x55ea3ba619f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.423+0000 7fb0ea239740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ea3ba66e40 con 0x55ea3ba59290 2024-04-25T01:39:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.423+0000 7fb0ea239740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ea3b978e90 con 0x55ea3ba5a980 2024-04-25T01:39:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.423+0000 7fb0ea239740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ea3ba66fc0 con 0x55ea3ba64010 2024-04-25T01:39:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.423+0000 7fb0ea239740 10 client.0 _ll_register_callbacks cb 0x55ea3b9b4970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.423+0000 7fb0e52c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba59290 0x55ea3ba60f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.423+0000 7fb0e52c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba59290 0x55ea3ba60f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40366/0 (socket says 172.21.15.28:40366) 2024-04-25T01:39:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.423+0000 7fb0e52c5640 1 -- 172.21.15.28:0/657790209 learned_addr learned my addr 172.21.15.28:0/657790209 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.423+0000 7fb0e4ac4640 1 --2- 172.21.15.28:0/657790209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba5a980 0x55ea3ba614b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.424+0000 7fb0e5ac6640 1 --2- 172.21.15.28:0/657790209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba64010 0x55ea3ba619f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.424+0000 7fb0e52c5640 1 -- 172.21.15.28:0/657790209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba64010 msgr2=0x55ea3ba619f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:50.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.424+0000 7fb0e52c5640 1 --2- 172.21.15.28:0/657790209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba64010 0x55ea3ba619f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.424+0000 7fb0e52c5640 1 -- 172.21.15.28:0/657790209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba5a980 msgr2=0x55ea3ba614b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:50.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.424+0000 7fb0e52c5640 1 --2- 172.21.15.28:0/657790209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba5a980 0x55ea3ba614b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.424+0000 7fb0e52c5640 1 -- 172.21.15.28:0/657790209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea3b7997e0 con 0x55ea3ba59290 2024-04-25T01:39:50.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.424+0000 7fb0e52c5640 1 --2- 172.21.15.28:0/657790209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba59290 0x55ea3ba60f70 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc09bde0 tx=0x7fb0dc06bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c804740c98af4ec5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:50.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.424+0000 7fb0c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:50.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.424+0000 7fb0c8ff9640 1 -- 172.21.15.28:0/657790209 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc09a600 con 0x55ea3ba59290 2024-04-25T01:39:50.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.424+0000 7fb0c8ff9640 1 -- 172.21.15.28:0/657790209 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fb0dc0cd300 con 0x55ea3ba59290 2024-04-25T01:39:50.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.424+0000 7fb0ea239740 1 -- 172.21.15.28:0/657790209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0d8036040 con 0x55ea3ba59290 2024-04-25T01:39:50.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.424+0000 7fb0c8ff9640 1 -- 172.21.15.28:0/657790209 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb0dc09a410 con 0x55ea3ba59290 2024-04-25T01:39:50.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.424+0000 7fb0ea239740 10 client.18099 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:50.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.424+0000 7fb0ea239740 1 -- 172.21.15.28:0/657790209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ea3b7f9f70 con 0x55ea3ba59290 2024-04-25T01:39:50.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.425+0000 7fb0c8ff9640 1 -- 172.21.15.28:0/657790209 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0dc09e070 con 0x55ea3ba59290 2024-04-25T01:39:50.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.425+0000 7fb0c8ff9640 1 -- 172.21.15.28:0/657790209 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb0dc0a4630 con 0x55ea3ba59290 2024-04-25T01:39:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.425+0000 7fb0ea239740 10 client.18099 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.425+0000 7fb0ea239740 10 client.18099 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.425+0000 7fb0ea239740 20 client.18099 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.425+0000 7fb0ea239740 1 --2- 172.21.15.28:0/657790209 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ea3ba740d0 0x55ea3ba76520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.425+0000 7fb0ea239740 4 client.18099 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.425+0000 7fb0ea239740 1 -- 172.21.15.28:0/657790209 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea3b9915a0 con 0x55ea3ba740d0 2024-04-25T01:39:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.425+0000 7fb0e4ac4640 1 --2- 172.21.15.28:0/657790209 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ea3ba740d0 0x55ea3ba76520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.426+0000 7fb0e4ac4640 1 --2- 172.21.15.28:0/657790209 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ea3ba740d0 0x55ea3ba76520 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=888bac9af1b58d69 server_cookie=bf2d52e774893eb5 in_seq=0 out_seq=0 2024-04-25T01:39:50.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.426+0000 7fb0c8ff9640 10 client.18099 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:50.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.426+0000 7fb0c8ff9640 1 -- 172.21.15.28:0/657790209 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ea3b9915a0 con 0x55ea3ba740d0 2024-04-25T01:39:50.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.426+0000 7fb0c8ff9640 10 client.18099 handle_command_reply: tid=0 2024-04-25T01:39:50.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.426+0000 7fb0ea239740 1 client.18099 shutdown 2024-04-25T01:39:50.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.426+0000 7fb0ea239740 1 -- 172.21.15.28:0/657790209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba59290 msgr2=0x55ea3ba60f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:50.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.426+0000 7fb0ea239740 1 --2- 172.21.15.28:0/657790209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba59290 0x55ea3ba60f70 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc09bde0 tx=0x7fb0dc06bd70 comp rx=0 tx=0).stop 2024-04-25T01:39:50.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.426+0000 7fb0ea239740 1 -- 172.21.15.28:0/657790209 shutdown_connections 2024-04-25T01:39:50.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.426+0000 7fb0ea239740 1 --2- 172.21.15.28:0/657790209 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ea3ba740d0 0x55ea3ba76520 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.426+0000 7fb0ea239740 1 --2- 172.21.15.28:0/657790209 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba64010 0x55ea3ba619f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.426+0000 7fb0ea239740 1 --2- 172.21.15.28:0/657790209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba5a980 0x55ea3ba614b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.426+0000 7fb0c8ff9640 0 client.18099 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:50.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.426+0000 7fb0ea239740 1 --2- 172.21.15.28:0/657790209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba59290 0x55ea3ba60f70 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.426+0000 7fb0ea239740 1 -- 172.21.15.28:0/657790209 >> 172.21.15.28:0/657790209 conn(0x55ea3ba21950 msgr2=0x55ea3ba5b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.427+0000 7fb0ea239740 1 -- 172.21.15.28:0/657790209 shutdown_connections 2024-04-25T01:39:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.427+0000 7fb0ea239740 1 -- 172.21.15.28:0/657790209 wait complete. 2024-04-25T01:39:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.427+0000 7fb0ea239740 20 client.18099 trim_cache size 0 max 16384 2024-04-25T01:39:50.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.445+0000 7fb0ea239740 1 Processor -- start 2024-04-25T01:39:50.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.445+0000 7fb0ea239740 1 -- start start 2024-04-25T01:39:50.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.445+0000 7fb0ea239740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba59290 0x55ea3ba637b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.445+0000 7fb0ea239740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba5a980 0x55ea3ba6fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.445+0000 7fb0ea239740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba64010 0x55ea3ba547d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.445+0000 7fb0ea239740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e01aeb70 con 0x55ea3ba64010 2024-04-25T01:39:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.445+0000 7fb0ea239740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e01b06b0 con 0x55ea3ba59290 2024-04-25T01:39:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.445+0000 7fb0ea239740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e01af240 con 0x55ea3ba5a980 2024-04-25T01:39:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.445+0000 7fb0e5ac6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba64010 0x55ea3ba547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.445+0000 7fb0e5ac6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba64010 0x55ea3ba547d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40372/0 (socket says 172.21.15.28:40372) 2024-04-25T01:39:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.445+0000 7fb0e5ac6640 1 -- 172.21.15.28:0/2212244537 learned_addr learned my addr 172.21.15.28:0/2212244537 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.445+0000 7fb0e52c5640 1 --2- 172.21.15.28:0/2212244537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba59290 0x55ea3ba637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.445+0000 7fb0e4ac4640 1 --2- 172.21.15.28:0/2212244537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba5a980 0x55ea3ba6fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.445+0000 7fb0e5ac6640 1 -- 172.21.15.28:0/2212244537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba5a980 msgr2=0x55ea3ba6fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.445+0000 7fb0e5ac6640 1 --2- 172.21.15.28:0/2212244537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba5a980 0x55ea3ba6fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.445+0000 7fb0e5ac6640 1 -- 172.21.15.28:0/2212244537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba59290 msgr2=0x55ea3ba637b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.445+0000 7fb0e5ac6640 1 --2- 172.21.15.28:0/2212244537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba59290 0x55ea3ba637b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.445+0000 7fb0e5ac6640 1 -- 172.21.15.28:0/2212244537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea3b691170 con 0x55ea3ba64010 2024-04-25T01:39:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.445+0000 7fb0e52c5640 1 --2- 172.21.15.28:0/2212244537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba59290 0x55ea3ba637b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.446+0000 7fb0e5ac6640 1 --2- 172.21.15.28:0/2212244537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba64010 0x55ea3ba547d0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8004af0 tx=0x7fb0d8030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b27b6912bf0d37 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.446+0000 7fb0caffd640 1 -- 172.21.15.28:0/2212244537 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8046040 con 0x55ea3ba64010 2024-04-25T01:39:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.446+0000 7fb0caffd640 1 -- 172.21.15.28:0/2212244537 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d8057070 con 0x55ea3ba64010 2024-04-25T01:39:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.446+0000 7fb0caffd640 1 -- 172.21.15.28:0/2212244537 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d804e040 con 0x55ea3ba64010 2024-04-25T01:39:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.446+0000 7fb0ea239740 1 -- 172.21.15.28:0/2212244537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba64010 msgr2=0x55ea3ba547d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.446+0000 7fb0ea239740 1 --2- 172.21.15.28:0/2212244537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba64010 0x55ea3ba547d0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8004af0 tx=0x7fb0d8030ec0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.446+0000 7fb0ea239740 1 -- 172.21.15.28:0/2212244537 shutdown_connections 2024-04-25T01:39:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.446+0000 7fb0ea239740 1 --2- 172.21.15.28:0/2212244537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba64010 0x55ea3ba547d0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.446+0000 7fb0ea239740 1 --2- 172.21.15.28:0/2212244537 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba5a980 0x55ea3ba6fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.446+0000 7fb0ea239740 1 --2- 172.21.15.28:0/2212244537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba59290 0x55ea3ba637b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.446+0000 7fb0ea239740 1 -- 172.21.15.28:0/2212244537 >> 172.21.15.28:0/2212244537 conn(0x55ea3ba21950 msgr2=0x55ea3ba540c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.446+0000 7fb0ea239740 1 -- 172.21.15.28:0/2212244537 shutdown_connections 2024-04-25T01:39:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.446+0000 7fb0ea239740 1 -- 172.21.15.28:0/2212244537 wait complete. 2024-04-25T01:39:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.447+0000 7fb0ea239740 1 -- start start 2024-04-25T01:39:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.447+0000 7fb0ea239740 1 Processor -- start 2024-04-25T01:39:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.447+0000 7fb0ea239740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba59290 0x55ea3ba6c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.447+0000 7fb0ea239740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba5a980 0x55ea3ba6c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.447+0000 7fb0ea239740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba64010 0x55ea3ba69220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.447+0000 7fb0ea239740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e01affe0 con 0x55ea3ba59290 2024-04-25T01:39:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.447+0000 7fb0ea239740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e01af910 con 0x55ea3ba5a980 2024-04-25T01:39:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.447+0000 7fb0ea239740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e01acb50 con 0x55ea3ba64010 2024-04-25T01:39:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.447+0000 7fb0ea239740 10 client.0 _ll_register_callbacks cb 0x7fb0dc009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.447+0000 7fb0e52c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba59290 0x55ea3ba6c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.447+0000 7fb0e52c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba59290 0x55ea3ba6c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40374/0 (socket says 172.21.15.28:40374) 2024-04-25T01:39:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.447+0000 7fb0e52c5640 1 -- 172.21.15.28:0/3956585218 learned_addr learned my addr 172.21.15.28:0/3956585218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.447+0000 7fb0e4ac4640 1 --2- 172.21.15.28:0/3956585218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba5a980 0x55ea3ba6c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.448+0000 7fb0e5ac6640 1 --2- 172.21.15.28:0/3956585218 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba64010 0x55ea3ba69220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.448+0000 7fb0e52c5640 1 -- 172.21.15.28:0/3956585218 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba64010 msgr2=0x55ea3ba69220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.448+0000 7fb0e52c5640 1 --2- 172.21.15.28:0/3956585218 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba64010 0x55ea3ba69220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.448+0000 7fb0e52c5640 1 -- 172.21.15.28:0/3956585218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba5a980 msgr2=0x55ea3ba6c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.448+0000 7fb0e52c5640 1 --2- 172.21.15.28:0/3956585218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba5a980 0x55ea3ba6c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.448+0000 7fb0e52c5640 1 -- 172.21.15.28:0/3956585218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea3ba698c0 con 0x55ea3ba59290 2024-04-25T01:39:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.448+0000 7fb0e5ac6640 1 --2- 172.21.15.28:0/3956585218 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba64010 0x55ea3ba69220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.448+0000 7fb0e52c5640 1 --2- 172.21.15.28:0/3956585218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba59290 0x55ea3ba6c250 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc00e9c0 tx=0x7fb0dc0cd590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c864424f3af009b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.448+0000 7fb0c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.448+0000 7fb0c8ff9640 1 -- 172.21.15.28:0/3956585218 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc09b990 con 0x55ea3ba59290 2024-04-25T01:39:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.448+0000 7fb0c8ff9640 1 -- 172.21.15.28:0/3956585218 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fb0dc0ba670 con 0x55ea3ba59290 2024-04-25T01:39:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.448+0000 7fb0ea239740 1 -- 172.21.15.28:0/3956585218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0d8057070 con 0x55ea3ba59290 2024-04-25T01:39:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.448+0000 7fb0c8ff9640 1 -- 172.21.15.28:0/3956585218 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb0dc066050 con 0x55ea3ba59290 2024-04-25T01:39:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.448+0000 7fb0ea239740 10 client.18105 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.448+0000 7fb0ea239740 1 -- 172.21.15.28:0/3956585218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ea3badb050 con 0x55ea3ba59290 2024-04-25T01:39:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.448+0000 7fb0c8ff9640 1 -- 172.21.15.28:0/3956585218 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d8057070 con 0x55ea3ba59290 2024-04-25T01:39:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.448+0000 7fb0c8ff9640 1 -- 172.21.15.28:0/3956585218 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb0dc0ba280 con 0x55ea3ba59290 2024-04-25T01:39:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.449+0000 7fb0ea239740 10 client.18105 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.449+0000 7fb0ea239740 10 client.18105 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.449+0000 7fb0ea239740 20 client.18105 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.449+0000 7fb0ea239740 1 --2- 172.21.15.28:0/3956585218 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ea3badb7f0 0x55ea3baddc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.449+0000 7fb0ea239740 4 client.18105 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:39:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.449+0000 7fb0ea239740 1 -- 172.21.15.28:0/3956585218 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ea3b9ca800 con 0x55ea3badb7f0 2024-04-25T01:39:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.449+0000 7fb0e4ac4640 1 --2- 172.21.15.28:0/3956585218 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ea3badb7f0 0x55ea3baddc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.449+0000 7fb0e4ac4640 1 --2- 172.21.15.28:0/3956585218 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ea3badb7f0 0x55ea3baddc10 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc68b2e549d2996a server_cookie=a7ee56c1814cd1e1 in_seq=0 out_seq=0 2024-04-25T01:39:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.449+0000 7fb0c8ff9640 10 client.18105 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.450+0000 7fb0c8ff9640 1 -- 172.21.15.28:0/3956585218 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ea3b9ca800 con 0x55ea3badb7f0 2024-04-25T01:39:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.450+0000 7fb0c8ff9640 10 client.18105 handle_command_reply: tid=0 2024-04-25T01:39:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.450+0000 7fb0ea239740 1 client.18105 shutdown 2024-04-25T01:39:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.450+0000 7fb0ea239740 1 -- 172.21.15.28:0/3956585218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba59290 msgr2=0x55ea3ba6c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0ea239740 1 --2- 172.21.15.28:0/3956585218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba59290 0x55ea3ba6c250 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc00e9c0 tx=0x7fb0dc0cd590 comp rx=0 tx=0).stop 2024-04-25T01:39:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0ea239740 1 -- 172.21.15.28:0/3956585218 shutdown_connections 2024-04-25T01:39:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0ea239740 1 --2- 172.21.15.28:0/3956585218 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ea3badb7f0 0x55ea3baddc10 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0ea239740 1 --2- 172.21.15.28:0/3956585218 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea3ba64010 0x55ea3ba69220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0ea239740 1 --2- 172.21.15.28:0/3956585218 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea3ba5a980 0x55ea3ba6c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0ea239740 1 --2- 172.21.15.28:0/3956585218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea3ba59290 0x55ea3ba6c250 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0ea239740 1 -- 172.21.15.28:0/3956585218 >> 172.21.15.28:0/3956585218 conn(0x55ea3ba21950 msgr2=0x55ea3ba54950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0c8ff9640 0 client.18105 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0ea239740 1 -- 172.21.15.28:0/3956585218 shutdown_connections 2024-04-25T01:39:50.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0ea239740 1 -- 172.21.15.28:0/3956585218 wait complete. 2024-04-25T01:39:50.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0ea239740 20 client.18105 trim_cache size 0 max 16384 2024-04-25T01:39:50.442 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:39:50.442 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:39:50.442 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "7b68961b-e4d9-41e5-b9c1-d021123f4340", 2024-04-25T01:39:50.442 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:39:50.442 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:39:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0e7529640 1 -- 172.21.15.28:0/532728241 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb0cc0a2d00 msgr2=0x7fb0cc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0e7529640 1 --2- 172.21.15.28:0/532728241 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb0cc0a2d00 0x7fb0cc0a51c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009a60 tx=0x7fb0d8048000 comp rx=0 tx=0).stop 2024-04-25T01:39:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0e7529640 1 -- 172.21.15.28:0/532728241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e0101710 msgr2=0x7fb0e0198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0e7529640 1 --2- 172.21.15.28:0/532728241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e0101710 0x7fb0e0198040 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc00e9f0 tx=0x7fb0dc00edf0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0e7529640 1 -- 172.21.15.28:0/532728241 shutdown_connections 2024-04-25T01:39:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0e7529640 1 --2- 172.21.15.28:0/532728241 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb0cc0a2d00 0x7fb0cc0a51c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0e7529640 1 --2- 172.21.15.28:0/532728241 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb0e0107720 0x7fb0e019c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0e7529640 1 --2- 172.21.15.28:0/532728241 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0e01020d0 0x7fb0e0198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0e7529640 1 --2- 172.21.15.28:0/532728241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e0101710 0x7fb0e0198040 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.451+0000 7fb0e7529640 1 -- 172.21.15.28:0/532728241 >> 172.21.15.28:0/532728241 conn(0x7fb0e00fd180 msgr2=0x7fb0e00fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.452+0000 7fb0e7529640 1 -- 172.21.15.28:0/532728241 shutdown_connections 2024-04-25T01:39:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.452+0000 7fb0e7529640 1 -- 172.21.15.28:0/532728241 wait complete. 2024-04-25T01:39:50.446 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:39:50.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:39:50.464 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:50 smithi130 ceph-mon[32702]: pgmap v207: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:39:50.489 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:50 smithi032 ceph-mon[32461]: pgmap v207: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:39:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.608+0000 7fb7cd56a640 1 -- 172.21.15.28:0/3288739444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c8107730 msgr2=0x7fb7c8107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.608+0000 7fb7cd56a640 1 --2- 172.21.15.28:0/3288739444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c8107730 0x7fb7c8107b10 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009930 tx=0x7fb7b802a420 comp rx=0 tx=0).stop 2024-04-25T01:39:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.608+0000 7fb7cd56a640 1 -- 172.21.15.28:0/3288739444 shutdown_connections 2024-04-25T01:39:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.608+0000 7fb7cd56a640 1 --2- 172.21.15.28:0/3288739444 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c806a780 0x7fb7c810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.608+0000 7fb7cd56a640 1 --2- 172.21.15.28:0/3288739444 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8069dc0 0x7fb7c806a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.608+0000 7fb7cd56a640 1 --2- 172.21.15.28:0/3288739444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c8107730 0x7fb7c8107b10 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.608+0000 7fb7cd56a640 1 -- 172.21.15.28:0/3288739444 >> 172.21.15.28:0/3288739444 conn(0x7fb7c80776a0 msgr2=0x7fb7c8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.608+0000 7fb7cd56a640 1 -- 172.21.15.28:0/3288739444 shutdown_connections 2024-04-25T01:39:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.608+0000 7fb7cd56a640 1 -- 172.21.15.28:0/3288739444 wait complete. 2024-04-25T01:39:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.609+0000 7fb7cd56a640 1 Processor -- start 2024-04-25T01:39:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.609+0000 7fb7cd56a640 1 -- start start 2024-04-25T01:39:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.609+0000 7fb7cd56a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8069dc0 0x7fb7c81980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.609+0000 7fb7cd56a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c806a780 0x7fb7c81985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.609+0000 7fb7cd56a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c8107730 0x7fb7c819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.609+0000 7fb7cd56a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c8111b40 con 0x7fb7c806a780 2024-04-25T01:39:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.609+0000 7fb7cd56a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c81119c0 con 0x7fb7c8107730 2024-04-25T01:39:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.609+0000 7fb7cd56a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c8111cc0 con 0x7fb7c8069dc0 2024-04-25T01:39:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.609+0000 7fb7bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c806a780 0x7fb7c81985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.609+0000 7fb7bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c806a780 0x7fb7c81985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40378/0 (socket says 172.21.15.28:40378) 2024-04-25T01:39:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.609+0000 7fb7bffff640 1 -- 172.21.15.28:0/283189895 learned_addr learned my addr 172.21.15.28:0/283189895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.609+0000 7fb7c77fe640 1 --2- 172.21.15.28:0/283189895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c8107730 0x7fb7c819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.609+0000 7fb7c6ffd640 1 --2- 172.21.15.28:0/283189895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8069dc0 0x7fb7c81980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.609+0000 7fb7bffff640 1 -- 172.21.15.28:0/283189895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8069dc0 msgr2=0x7fb7c81980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.609+0000 7fb7bffff640 1 --2- 172.21.15.28:0/283189895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8069dc0 0x7fb7c81980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.609+0000 7fb7bffff640 1 -- 172.21.15.28:0/283189895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c8107730 msgr2=0x7fb7c819c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.609+0000 7fb7bffff640 1 --2- 172.21.15.28:0/283189895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c8107730 0x7fb7c819c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.609+0000 7fb7bffff640 1 -- 172.21.15.28:0/283189895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c819d090 con 0x7fb7c806a780 2024-04-25T01:39:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.610+0000 7fb7bffff640 1 --2- 172.21.15.28:0/283189895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c806a780 0x7fb7c81985e0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fb7b000d770 tx=0x7fb7b000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.610+0000 7fb7c4ff9640 1 -- 172.21.15.28:0/283189895 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0017070 con 0x7fb7c806a780 2024-04-25T01:39:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.610+0000 7fb7c4ff9640 1 -- 172.21.15.28:0/283189895 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b0005500 con 0x7fb7c806a780 2024-04-25T01:39:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.610+0000 7fb7c4ff9640 1 -- 172.21.15.28:0/283189895 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0004e00 con 0x7fb7c806a780 2024-04-25T01:39:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.610+0000 7fb7cd56a640 1 -- 172.21.15.28:0/283189895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7c819d380 con 0x7fb7c806a780 2024-04-25T01:39:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.610+0000 7fb7cd56a640 1 -- 172.21.15.28:0/283189895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7c81a4c10 con 0x7fb7c806a780 2024-04-25T01:39:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.612+0000 7fb7c4ff9640 1 -- 172.21.15.28:0/283189895 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb7b0004fa0 con 0x7fb7c806a780 2024-04-25T01:39:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.612+0000 7fb7d0271740 1 Processor -- start 2024-04-25T01:39:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.612+0000 7fb7d0271740 1 -- start start 2024-04-25T01:39:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7d0271740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef01630 0x5614cef01a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7d0271740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01f50 0x5614cef0c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7d0271740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef0c620 0x5614cef0ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7d0271740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5614cee666b0 con 0x5614cef01f50 2024-04-25T01:39:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7d0271740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5614cee0e880 con 0x5614cef01630 2024-04-25T01:39:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7d0271740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5614cee62670 con 0x5614cef0c620 2024-04-25T01:39:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7c6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01f50 0x5614cef0c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7c6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01f50 0x5614cef0c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40390/0 (socket says 172.21.15.28:40390) 2024-04-25T01:39:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7c6ffd640 1 -- 172.21.15.28:0/301251899 learned_addr learned my addr 172.21.15.28:0/301251899 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7c77fe640 1 --2- 172.21.15.28:0/301251899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef01630 0x5614cef01a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7bffff640 1 --2- 172.21.15.28:0/301251899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef0c620 0x5614cef0ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7c4ff9640 1 --2- 172.21.15.28:0/283189895 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb7b40a2d00 0x7fb7b40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7c6ffd640 1 -- 172.21.15.28:0/301251899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef0c620 msgr2=0x5614cef0ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7c6ffd640 1 --2- 172.21.15.28:0/301251899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef0c620 0x5614cef0ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7c6ffd640 1 -- 172.21.15.28:0/301251899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef01630 msgr2=0x5614cef01a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7c6ffd640 1 --2- 172.21.15.28:0/301251899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef01630 0x5614cef01a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7c77fe640 1 --2- 172.21.15.28:0/283189895 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb7b40a2d00 0x7fb7b40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7c6ffd640 1 -- 172.21.15.28:0/301251899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614cec447d0 con 0x5614cef01f50 2024-04-25T01:39:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7bffff640 1 --2- 172.21.15.28:0/301251899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef0c620 0x5614cef0ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.613+0000 7fb7c4ff9640 1 -- 172.21.15.28:0/283189895 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fb7b00cf2c0 con 0x7fb7c806a780 2024-04-25T01:39:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.614+0000 7fb7c6ffd640 1 --2- 172.21.15.28:0/301251899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01f50 0x5614cef0c0e0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fb7b802a940 tx=0x7fb7b8005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e144ff39ec8fec53 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.614+0000 7fb7bd7fa640 1 -- 172.21.15.28:0/301251899 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b803f070 con 0x5614cef01f50 2024-04-25T01:39:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.614+0000 7fb7c77fe640 1 --2- 172.21.15.28:0/283189895 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb7b40a2d00 0x7fb7b40a51c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0005fd0 tx=0x7fb7c0004ab0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.614+0000 7fb7bd7fa640 1 -- 172.21.15.28:0/301251899 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b8004030 con 0x5614cef01f50 2024-04-25T01:39:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.614+0000 7fb7bd7fa640 1 -- 172.21.15.28:0/301251899 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8035070 con 0x5614cef01f50 2024-04-25T01:39:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.614+0000 7fb7d0271740 1 -- 172.21.15.28:0/301251899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01f50 msgr2=0x5614cef0c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.614+0000 7fb7d0271740 1 --2- 172.21.15.28:0/301251899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01f50 0x5614cef0c0e0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fb7b802a940 tx=0x7fb7b8005e80 comp rx=0 tx=0).stop 2024-04-25T01:39:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7d0271740 1 -- 172.21.15.28:0/301251899 shutdown_connections 2024-04-25T01:39:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7d0271740 1 --2- 172.21.15.28:0/301251899 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef0c620 0x5614cef0ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7d0271740 1 --2- 172.21.15.28:0/301251899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01f50 0x5614cef0c0e0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7d0271740 1 --2- 172.21.15.28:0/301251899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef01630 0x5614cef01a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7d0271740 1 -- 172.21.15.28:0/301251899 >> 172.21.15.28:0/301251899 conn(0x5614ceefd950 msgr2=0x5614ceefdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7d0271740 1 -- 172.21.15.28:0/301251899 shutdown_connections 2024-04-25T01:39:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7d0271740 1 -- 172.21.15.28:0/301251899 wait complete. 2024-04-25T01:39:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7d0271740 1 -- start start 2024-04-25T01:39:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7d0271740 1 Processor -- start 2024-04-25T01:39:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7d0271740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef01630 0x5614cef166e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7d0271740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01f50 0x5614cef16c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7d0271740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef0c620 0x5614cef10af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7d0271740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5614cef0f3b0 con 0x5614cef01f50 2024-04-25T01:39:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7d0271740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5614cee21f50 con 0x5614cef01630 2024-04-25T01:39:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7d0271740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5614cef0f530 con 0x5614cef0c620 2024-04-25T01:39:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7d0271740 10 client.0 _ll_register_callbacks cb 0x5614cee5d000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01f50 0x5614cef16c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01f50 0x5614cef16c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40396/0 (socket says 172.21.15.28:40396) 2024-04-25T01:39:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7bffff640 1 -- 172.21.15.28:0/3639018339 learned_addr learned my addr 172.21.15.28:0/3639018339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.615+0000 7fb7c6ffd640 1 --2- 172.21.15.28:0/3639018339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef01630 0x5614cef166e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7c77fe640 1 --2- 172.21.15.28:0/3639018339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef0c620 0x5614cef10af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7bffff640 1 -- 172.21.15.28:0/3639018339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef0c620 msgr2=0x5614cef10af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7bffff640 1 --2- 172.21.15.28:0/3639018339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef0c620 0x5614cef10af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7bffff640 1 -- 172.21.15.28:0/3639018339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef01630 msgr2=0x5614cef166e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7bffff640 1 --2- 172.21.15.28:0/3639018339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef01630 0x5614cef166e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7bffff640 1 -- 172.21.15.28:0/3639018339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614ceb7a3f0 con 0x5614cef01f50 2024-04-25T01:39:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7c6ffd640 1 --2- 172.21.15.28:0/3639018339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef01630 0x5614cef166e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7bffff640 1 --2- 172.21.15.28:0/3639018339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01f50 0x5614cef16c20 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fb7b00cc730 tx=0x7fb7b009ae00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a3cd54036b602b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7ae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7ae7fc640 1 -- 172.21.15.28:0/3639018339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0020070 con 0x5614cef01f50 2024-04-25T01:39:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7ae7fc640 1 -- 172.21.15.28:0/3639018339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fb7b00cc950 con 0x5614cef01f50 2024-04-25T01:39:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7d0271740 1 -- 172.21.15.28:0/3639018339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7b8004030 con 0x5614cef01f50 2024-04-25T01:39:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7ae7fc640 1 -- 172.21.15.28:0/3639018339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb7b00b1e60 con 0x5614cef01f50 2024-04-25T01:39:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7d0271740 10 client.18117 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7d0271740 1 -- 172.21.15.28:0/3639018339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5614ceb3a0f0 con 0x5614cef01f50 2024-04-25T01:39:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7ae7fc640 1 -- 172.21.15.28:0/3639018339 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b8004030 con 0x5614cef01f50 2024-04-25T01:39:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7ae7fc640 1 -- 172.21.15.28:0/3639018339 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb7b0097b10 con 0x5614cef01f50 2024-04-25T01:39:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7d0271740 10 client.18117 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7d0271740 10 client.18117 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.616+0000 7fb7d0271740 20 client.18117 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.617+0000 7fb7d0271740 1 --2- 172.21.15.28:0/3639018339 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5614cef20d00 0x5614cef230f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.617+0000 7fb7d0271740 4 client.18117 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:39:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.617+0000 7fb7d0271740 1 -- 172.21.15.28:0/3639018339 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614cee73bf0 con 0x5614cef20d00 2024-04-25T01:39:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.617+0000 7fb7c6ffd640 1 --2- 172.21.15.28:0/3639018339 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5614cef20d00 0x5614cef230f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.617+0000 7fb7c6ffd640 1 --2- 172.21.15.28:0/3639018339 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5614cef20d00 0x5614cef230f0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f3ab0aa6f60af3f server_cookie=51667a00f80f2162 in_seq=0 out_seq=0 2024-04-25T01:39:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.617+0000 7fb7ae7fc640 10 client.18117 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.618+0000 7fb7ae7fc640 1 -- 172.21.15.28:0/3639018339 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5614cee73bf0 con 0x5614cef20d00 2024-04-25T01:39:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.618+0000 7fb7ae7fc640 10 client.18117 handle_command_reply: tid=0 2024-04-25T01:39:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.618+0000 7fb7d0271740 1 client.18117 shutdown 2024-04-25T01:39:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.618+0000 7fb7d0271740 1 -- 172.21.15.28:0/3639018339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01f50 msgr2=0x5614cef16c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.618+0000 7fb7d0271740 1 --2- 172.21.15.28:0/3639018339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01f50 0x5614cef16c20 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fb7b00cc730 tx=0x7fb7b009ae00 comp rx=0 tx=0).stop 2024-04-25T01:39:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.618+0000 7fb7d0271740 1 -- 172.21.15.28:0/3639018339 shutdown_connections 2024-04-25T01:39:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.618+0000 7fb7d0271740 1 --2- 172.21.15.28:0/3639018339 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5614cef20d00 0x5614cef230f0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.618+0000 7fb7d0271740 1 --2- 172.21.15.28:0/3639018339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef0c620 0x5614cef10af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.618+0000 7fb7ae7fc640 0 client.18117 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.618+0000 7fb7d0271740 1 --2- 172.21.15.28:0/3639018339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01f50 0x5614cef16c20 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.618+0000 7fb7d0271740 1 --2- 172.21.15.28:0/3639018339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef01630 0x5614cef166e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.618+0000 7fb7d0271740 1 -- 172.21.15.28:0/3639018339 >> 172.21.15.28:0/3639018339 conn(0x5614ceefd950 msgr2=0x5614ceefcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.618+0000 7fb7d0271740 1 -- 172.21.15.28:0/3639018339 shutdown_connections 2024-04-25T01:39:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.618+0000 7fb7d0271740 1 -- 172.21.15.28:0/3639018339 wait complete. 2024-04-25T01:39:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.618+0000 7fb7d0271740 20 client.18117 trim_cache size 0 max 16384 2024-04-25T01:39:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.636+0000 7fb7d0271740 1 Processor -- start 2024-04-25T01:39:50.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.636+0000 7fb7d0271740 1 -- start start 2024-04-25T01:39:50.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.636+0000 7fb7d0271740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01630 0x5614cef163a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.636+0000 7fb7d0271740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef01f50 0x5614cef168e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.636+0000 7fb7d0271740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef0c620 0x5614cef0abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.636+0000 7fb7d0271740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c81ac700 con 0x5614cef01630 2024-04-25T01:39:50.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.636+0000 7fb7d0271740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c81ae240 con 0x5614cef01f50 2024-04-25T01:39:50.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.636+0000 7fb7d0271740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c81acdd0 con 0x5614cef0c620 2024-04-25T01:39:50.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.636+0000 7fb7c6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01630 0x5614cef163a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.636+0000 7fb7c6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01630 0x5614cef163a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40408/0 (socket says 172.21.15.28:40408) 2024-04-25T01:39:50.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.636+0000 7fb7c6ffd640 1 -- 172.21.15.28:0/2677726755 learned_addr learned my addr 172.21.15.28:0/2677726755 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:50.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.637+0000 7fb7c6ffd640 1 -- 172.21.15.28:0/2677726755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef0c620 msgr2=0x5614cef0abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:50.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.637+0000 7fb7c6ffd640 1 --2- 172.21.15.28:0/2677726755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef0c620 0x5614cef0abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.637+0000 7fb7c6ffd640 1 -- 172.21.15.28:0/2677726755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef01f50 msgr2=0x5614cef168e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.637+0000 7fb7c6ffd640 1 --2- 172.21.15.28:0/2677726755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef01f50 0x5614cef168e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.637+0000 7fb7c6ffd640 1 -- 172.21.15.28:0/2677726755 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614cef23460 con 0x5614cef01630 2024-04-25T01:39:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.637+0000 7fb7c6ffd640 1 --2- 172.21.15.28:0/2677726755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01630 0x5614cef163a0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fb7b80052a0 tx=0x7fb7b80052d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bbeb6253c5645dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.637+0000 7fb7bcff9640 1 -- 172.21.15.28:0/2677726755 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8035070 con 0x5614cef01630 2024-04-25T01:39:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.637+0000 7fb7bcff9640 1 -- 172.21.15.28:0/2677726755 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b8009520 con 0x5614cef01630 2024-04-25T01:39:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.637+0000 7fb7bcff9640 1 -- 172.21.15.28:0/2677726755 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b803f040 con 0x5614cef01630 2024-04-25T01:39:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.637+0000 7fb7d0271740 1 -- 172.21.15.28:0/2677726755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01630 msgr2=0x5614cef163a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.637+0000 7fb7d0271740 1 --2- 172.21.15.28:0/2677726755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01630 0x5614cef163a0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fb7b80052a0 tx=0x7fb7b80052d0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.638+0000 7fb7d0271740 1 -- 172.21.15.28:0/2677726755 shutdown_connections 2024-04-25T01:39:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.638+0000 7fb7d0271740 1 --2- 172.21.15.28:0/2677726755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef0c620 0x5614cef0abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.638+0000 7fb7d0271740 1 --2- 172.21.15.28:0/2677726755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef01f50 0x5614cef168e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.638+0000 7fb7d0271740 1 --2- 172.21.15.28:0/2677726755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01630 0x5614cef163a0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.638+0000 7fb7d0271740 1 -- 172.21.15.28:0/2677726755 >> 172.21.15.28:0/2677726755 conn(0x5614ceefd950 msgr2=0x5614cef10380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:50.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.638+0000 7fb7d0271740 1 -- 172.21.15.28:0/2677726755 shutdown_connections 2024-04-25T01:39:50.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.638+0000 7fb7d0271740 1 -- 172.21.15.28:0/2677726755 wait complete. 2024-04-25T01:39:50.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.638+0000 7fb7d0271740 1 -- start start 2024-04-25T01:39:50.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.638+0000 7fb7d0271740 1 Processor -- start 2024-04-25T01:39:50.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.638+0000 7fb7d0271740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01630 0x5614cef1a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.638+0000 7fb7d0271740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef01f50 0x5614cef1ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.638+0000 7fb7d0271740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef0c620 0x5614cef1b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.638+0000 7fb7d0271740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c81adb70 con 0x5614cef01630 2024-04-25T01:39:50.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.638+0000 7fb7d0271740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c81ad4a0 con 0x5614cef0c620 2024-04-25T01:39:50.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.638+0000 7fb7d0271740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c81aa6e0 con 0x5614cef01f50 2024-04-25T01:39:50.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.638+0000 7fb7d0271740 10 client.0 _ll_register_callbacks cb 0x7fb7b00056b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:39:50.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.638+0000 7fb7c6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01630 0x5614cef1a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.638+0000 7fb7c6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01630 0x5614cef1a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40424/0 (socket says 172.21.15.28:40424) 2024-04-25T01:39:50.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.638+0000 7fb7c6ffd640 1 -- 172.21.15.28:0/4044917083 learned_addr learned my addr 172.21.15.28:0/4044917083 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:39:50.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.639+0000 7fb7c77fe640 1 --2- 172.21.15.28:0/4044917083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef0c620 0x5614cef1b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.639+0000 7fb7bffff640 1 --2- 172.21.15.28:0/4044917083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef01f50 0x5614cef1ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.639+0000 7fb7c6ffd640 1 -- 172.21.15.28:0/4044917083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef01f50 msgr2=0x5614cef1ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.639+0000 7fb7c6ffd640 1 --2- 172.21.15.28:0/4044917083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef01f50 0x5614cef1ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.639+0000 7fb7c6ffd640 1 -- 172.21.15.28:0/4044917083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef0c620 msgr2=0x5614cef1b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:39:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.639+0000 7fb7c6ffd640 1 --2- 172.21.15.28:0/4044917083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef0c620 0x5614cef1b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.639+0000 7fb7c6ffd640 1 -- 172.21.15.28:0/4044917083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614cef86260 con 0x5614cef01630 2024-04-25T01:39:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.639+0000 7fb7bffff640 1 --2- 172.21.15.28:0/4044917083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef01f50 0x5614cef1ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:39:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.639+0000 7fb7c77fe640 1 --2- 172.21.15.28:0/4044917083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef0c620 0x5614cef1b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:39:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.639+0000 7fb7c6ffd640 1 --2- 172.21.15.28:0/4044917083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01630 0x5614cef1a640 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009930 tx=0x7fb7b8005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a92b93ba33450ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:39:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.639+0000 7fb7ae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:39:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.639+0000 7fb7ae7fc640 1 -- 172.21.15.28:0/4044917083 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8055020 con 0x5614cef01630 2024-04-25T01:39:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.639+0000 7fb7ae7fc640 1 -- 172.21.15.28:0/4044917083 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fb7b8004040 con 0x5614cef01630 2024-04-25T01:39:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.639+0000 7fb7d0271740 1 -- 172.21.15.28:0/4044917083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7b8009520 con 0x5614cef01630 2024-04-25T01:39:50.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.639+0000 7fb7ae7fc640 1 -- 172.21.15.28:0/4044917083 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb7b8060050 con 0x5614cef01630 2024-04-25T01:39:50.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.639+0000 7fb7d0271740 10 client.18123 fetch_fsmap learned FSMap version 28 2024-04-25T01:39:50.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.639+0000 7fb7d0271740 1 -- 172.21.15.28:0/4044917083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5614cef87550 con 0x5614cef01630 2024-04-25T01:39:50.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.639+0000 7fb7ae7fc640 1 -- 172.21.15.28:0/4044917083 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7b8009520 con 0x5614cef01630 2024-04-25T01:39:50.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.640+0000 7fb7ae7fc640 1 -- 172.21.15.28:0/4044917083 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb7b80035f0 con 0x5614cef01630 2024-04-25T01:39:50.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.640+0000 7fb7d0271740 10 client.18123 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:39:50.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.640+0000 7fb7d0271740 10 client.18123 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:39:50.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.640+0000 7fb7d0271740 20 client.18123 populate_metadata read hostname 'smithi028' 2024-04-25T01:39:50.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.640+0000 7fb7d0271740 1 --2- 172.21.15.28:0/4044917083 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5614cef87840 0x5614cef89c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:39:50.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.640+0000 7fb7d0271740 4 client.18123 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:39:50.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.640+0000 7fb7d0271740 1 -- 172.21.15.28:0/4044917083 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5614cee6df40 con 0x5614cef87840 2024-04-25T01:39:50.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.640+0000 7fb7bffff640 1 --2- 172.21.15.28:0/4044917083 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5614cef87840 0x5614cef89c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:39:50.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.640+0000 7fb7bffff640 1 --2- 172.21.15.28:0/4044917083 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5614cef87840 0x5614cef89c30 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4222b66ee16491e0 server_cookie=6f7e1a78c2010af0 in_seq=0 out_seq=0 2024-04-25T01:39:50.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.640+0000 7fb7ae7fc640 10 client.18123 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:50.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7ae7fc640 1 -- 172.21.15.28:0/4044917083 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5614cee6df40 con 0x5614cef87840 2024-04-25T01:39:50.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7ae7fc640 10 client.18123 handle_command_reply: tid=0 2024-04-25T01:39:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7d0271740 1 client.18123 shutdown 2024-04-25T01:39:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7d0271740 1 -- 172.21.15.28:0/4044917083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01630 msgr2=0x5614cef1a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7d0271740 1 --2- 172.21.15.28:0/4044917083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01630 0x5614cef1a640 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8009930 tx=0x7fb7b8005e80 comp rx=0 tx=0).stop 2024-04-25T01:39:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7d0271740 1 -- 172.21.15.28:0/4044917083 shutdown_connections 2024-04-25T01:39:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7d0271740 1 --2- 172.21.15.28:0/4044917083 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5614cef87840 0x5614cef89c30 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7d0271740 1 --2- 172.21.15.28:0/4044917083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614cef0c620 0x5614cef1b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7d0271740 1 --2- 172.21.15.28:0/4044917083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614cef01f50 0x5614cef1ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7ae7fc640 0 client.18123 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:39:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7d0271740 1 --2- 172.21.15.28:0/4044917083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cef01630 0x5614cef1a640 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7d0271740 1 -- 172.21.15.28:0/4044917083 >> 172.21.15.28:0/4044917083 conn(0x5614ceefd950 msgr2=0x5614cef16230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7d0271740 1 -- 172.21.15.28:0/4044917083 shutdown_connections 2024-04-25T01:39:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7d0271740 1 -- 172.21.15.28:0/4044917083 wait complete. 2024-04-25T01:39:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7d0271740 20 client.18123 trim_cache size 0 max 16384 2024-04-25T01:39:50.632 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:39:50.632 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:39:50.632 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-25T01:39:50.632 INFO:teuthology.orchestra.run.smithi028.stdout: "7b68961b-e4d9-41e5-b9c1-d021123f4340": { 2024-04-25T01:39:50.632 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-25T01:39:50.632 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "7b68961b-e4d9-41e5-b9c1-d021123f4340", 2024-04-25T01:39:50.632 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-25T01:39:50.632 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:39:50.632 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:39:50.632 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:39:50.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7cd56a640 1 -- 172.21.15.28:0/283189895 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb7b40a2d00 msgr2=0x7fb7b40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:50.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7cd56a640 1 --2- 172.21.15.28:0/283189895 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb7b40a2d00 0x7fb7b40a51c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0005fd0 tx=0x7fb7c0004ab0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7cd56a640 1 -- 172.21.15.28:0/283189895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c806a780 msgr2=0x7fb7c81985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:39:50.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7cd56a640 1 --2- 172.21.15.28:0/283189895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c806a780 0x7fb7c81985e0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fb7b000d770 tx=0x7fb7b000dc30 comp rx=0 tx=0).stop 2024-04-25T01:39:50.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7cd56a640 1 -- 172.21.15.28:0/283189895 shutdown_connections 2024-04-25T01:39:50.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7cd56a640 1 --2- 172.21.15.28:0/283189895 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb7b40a2d00 0x7fb7b40a51c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7cd56a640 1 --2- 172.21.15.28:0/283189895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7c8107730 0x7fb7c819c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7cd56a640 1 --2- 172.21.15.28:0/283189895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7c806a780 0x7fb7c81985e0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7cd56a640 1 --2- 172.21.15.28:0/283189895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb7c8069dc0 0x7fb7c81980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:39:50.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.641+0000 7fb7cd56a640 1 -- 172.21.15.28:0/283189895 >> 172.21.15.28:0/283189895 conn(0x7fb7c80776a0 msgr2=0x7fb7c80fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:39:50.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.642+0000 7fb7cd56a640 1 -- 172.21.15.28:0/283189895 shutdown_connections 2024-04-25T01:39:50.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:39:50.642+0000 7fb7cd56a640 1 -- 172.21.15.28:0/283189895 wait complete. 2024-04-25T01:39:50.636 INFO:tasks.cephfs.filesystem:scrub status for tag:7b68961b-e4d9-41e5-b9c1-d021123f4340 - {'path': '/', 'tag': '7b68961b-e4d9-41e5-b9c1-d021123f4340', 'options': 'recursive,force'} 2024-04-25T01:39:52.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:52 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:39:52.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:52 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:52.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:52 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:39:52.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:52 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:52.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:52 smithi028 ceph-mon[29390]: pgmap v208: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:39:52.464 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:52 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:39:52.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:52 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:52.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:52 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:39:52.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:52 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:52.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:52 smithi130 ceph-mon[32702]: pgmap v208: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:39:52.489 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:52 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:39:52.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:52 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:52.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:52 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:39:52.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:52 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:39:52.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:52 smithi032 ceph-mon[32461]: pgmap v208: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:39:54.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:54 smithi028 ceph-mon[29390]: pgmap v209: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 12 op/s 2024-04-25T01:39:54.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:54 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:39:54.464 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:54 smithi130 ceph-mon[32702]: pgmap v209: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 12 op/s 2024-04-25T01:39:54.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:54 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:39:54.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:54 smithi032 ceph-mon[32461]: pgmap v209: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 12 op/s 2024-04-25T01:39:54.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:54 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:39:56.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:56 smithi028 ceph-mon[29390]: pgmap v210: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:39:56.464 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:56 smithi130 ceph-mon[32702]: pgmap v210: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:39:56.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:56 smithi032 ceph-mon[32461]: pgmap v210: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:39:58.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:39:58 smithi028 ceph-mon[29390]: pgmap v211: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-25T01:39:58.464 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:39:58 smithi130 ceph-mon[32702]: pgmap v211: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-25T01:39:58.489 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:39:58 smithi032 ceph-mon[32461]: pgmap v211: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-25T01:40:00.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:00 smithi028 ceph-mon[29390]: pgmap v212: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-04-25T01:40:00.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:00 smithi028 ceph-mon[29390]: overall HEALTH_OK 2024-04-25T01:40:00.368 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:00 smithi032 ceph-mon[32461]: pgmap v212: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-04-25T01:40:00.368 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:00 smithi032 ceph-mon[32461]: overall HEALTH_OK 2024-04-25T01:40:00.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:00 smithi130 ceph-mon[32702]: pgmap v212: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-04-25T01:40:00.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:00 smithi130 ceph-mon[32702]: overall HEALTH_OK 2024-04-25T01:40:02.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:02 smithi028 ceph-mon[29390]: pgmap v213: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:40:02.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:02 smithi130 ceph-mon[32702]: pgmap v213: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:40:02.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:02 smithi032 ceph-mon[32461]: pgmap v213: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:40:04.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:04 smithi028 ceph-mon[29390]: pgmap v214: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-25T01:40:04.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:04 smithi130 ceph-mon[32702]: pgmap v214: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-25T01:40:04.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:04 smithi032 ceph-mon[32461]: pgmap v214: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-25T01:40:06.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:06 smithi028 ceph-mon[29390]: pgmap v215: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-25T01:40:06.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:06 smithi130 ceph-mon[32702]: pgmap v215: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-25T01:40:06.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:06 smithi032 ceph-mon[32461]: pgmap v215: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-25T01:40:08.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:08 smithi028 ceph-mon[29390]: pgmap v216: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-04-25T01:40:08.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:08 smithi130 ceph-mon[32702]: pgmap v216: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-04-25T01:40:08.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:08 smithi032 ceph-mon[32461]: pgmap v216: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-04-25T01:40:10.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:10 smithi130 ceph-mon[32702]: pgmap v217: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-04-25T01:40:10.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:10 smithi130 ceph-mon[32702]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T01:40:10.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:10 smithi032 ceph-mon[32461]: pgmap v217: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-04-25T01:40:10.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:10 smithi032 ceph-mon[32461]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T01:40:10.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:10 smithi028 ceph-mon[29390]: pgmap v217: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-04-25T01:40:10.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:10 smithi028 ceph-mon[29390]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T01:40:12.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:12 smithi130 ceph-mon[32702]: pgmap v219: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-25T01:40:12.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:12 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:40:12.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:12 smithi032 ceph-mon[32461]: pgmap v219: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-25T01:40:12.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:12 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:40:12.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:12 smithi028 ceph-mon[29390]: pgmap v219: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-25T01:40:12.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:12 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:40:13.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:13 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:40:13.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:13 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:40:13.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:13 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:40:13.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:13 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:40:13.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:13 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:40:13.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:13 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:40:13.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:13 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:40:13.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:13 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:40:13.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:13 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:40:14.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:14 smithi130 ceph-mon[32702]: pgmap v220: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:40:14.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:14 smithi032 ceph-mon[32461]: pgmap v220: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:40:14.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:14 smithi028 ceph-mon[29390]: pgmap v220: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:40:16.465 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:16 smithi130 ceph-mon[32702]: pgmap v221: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:40:16.490 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:16 smithi032 ceph-mon[32461]: pgmap v221: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:40:16.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:16 smithi028 ceph-mon[29390]: pgmap v221: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:40:18.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:17 smithi130 ceph-mon[32702]: pgmap v222: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:40:18.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:17 smithi130 ceph-mon[32702]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T01:40:18.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:17 smithi032 ceph-mon[32461]: pgmap v222: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:40:18.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:17 smithi032 ceph-mon[32461]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T01:40:18.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:17 smithi028 ceph-mon[29390]: pgmap v222: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:40:18.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:17 smithi028 ceph-mon[29390]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T01:40:20.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:19 smithi130 ceph-mon[32702]: pgmap v224: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2024-04-25T01:40:20.240 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:19 smithi032 ceph-mon[32461]: pgmap v224: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2024-04-25T01:40:20.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:19 smithi028 ceph-mon[29390]: pgmap v224: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2024-04-25T01:40:20.637 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:40:20.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:40:20.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.774+0000 7fd681d89640 1 -- 172.21.15.28:0/553848230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c107730 msgr2=0x7fd67c107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:20.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.774+0000 7fd681d89640 1 --2- 172.21.15.28:0/553848230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c107730 0x7fd67c107b10 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009930 tx=0x7fd66c02a420 comp rx=0 tx=0).stop 2024-04-25T01:40:20.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.774+0000 7fd681d89640 1 -- 172.21.15.28:0/553848230 shutdown_connections 2024-04-25T01:40:20.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.774+0000 7fd681d89640 1 --2- 172.21.15.28:0/553848230 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd67c06a780 0x7fd67c10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.774+0000 7fd681d89640 1 --2- 172.21.15.28:0/553848230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd67c069dc0 0x7fd67c06a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.774+0000 7fd681d89640 1 --2- 172.21.15.28:0/553848230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c107730 0x7fd67c107b10 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.774+0000 7fd681d89640 1 -- 172.21.15.28:0/553848230 >> 172.21.15.28:0/553848230 conn(0x7fd67c0776a0 msgr2=0x7fd67c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:20.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.774+0000 7fd681d89640 1 -- 172.21.15.28:0/553848230 shutdown_connections 2024-04-25T01:40:20.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.774+0000 7fd681d89640 1 -- 172.21.15.28:0/553848230 wait complete. 2024-04-25T01:40:20.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.775+0000 7fd681d89640 1 Processor -- start 2024-04-25T01:40:20.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.775+0000 7fd681d89640 1 -- start start 2024-04-25T01:40:20.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.775+0000 7fd681d89640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd67c069dc0 0x7fd67c100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.775+0000 7fd681d89640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd67c06a780 0x7fd67c100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.775+0000 7fd681d89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c107730 0x7fd67c1010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.775+0000 7fd681d89640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c111b40 con 0x7fd67c107730 2024-04-25T01:40:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.775+0000 7fd681d89640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c1119c0 con 0x7fd67c069dc0 2024-04-25T01:40:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.775+0000 7fd681d89640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c111cc0 con 0x7fd67c06a780 2024-04-25T01:40:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.776+0000 7fd67bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c107730 0x7fd67c1010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.776+0000 7fd67bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c107730 0x7fd67c1010f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34486/0 (socket says 172.21.15.28:34486) 2024-04-25T01:40:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.776+0000 7fd67bfff640 1 -- 172.21.15.28:0/2957482768 learned_addr learned my addr 172.21.15.28:0/2957482768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.776+0000 7fd67b7fe640 1 --2- 172.21.15.28:0/2957482768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd67c069dc0 0x7fd67c100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.776+0000 7fd67affd640 1 --2- 172.21.15.28:0/2957482768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd67c06a780 0x7fd67c100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.776+0000 7fd67bfff640 1 -- 172.21.15.28:0/2957482768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd67c06a780 msgr2=0x7fd67c100bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.776+0000 7fd67bfff640 1 --2- 172.21.15.28:0/2957482768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd67c06a780 0x7fd67c100bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.776+0000 7fd67bfff640 1 -- 172.21.15.28:0/2957482768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd67c069dc0 msgr2=0x7fd67c100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:20.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.776+0000 7fd67bfff640 1 --2- 172.21.15.28:0/2957482768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd67c069dc0 0x7fd67c100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.776+0000 7fd67bfff640 1 -- 172.21.15.28:0/2957482768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67c1a64a0 con 0x7fd67c107730 2024-04-25T01:40:20.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.776+0000 7fd67bfff640 1 --2- 172.21.15.28:0/2957482768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c107730 0x7fd67c1010f0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fd67400cad0 tx=0x7fd67400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:20.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.776+0000 7fd678ff9640 1 -- 172.21.15.28:0/2957482768 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67400ae20 con 0x7fd67c107730 2024-04-25T01:40:20.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.776+0000 7fd678ff9640 1 -- 172.21.15.28:0/2957482768 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd674005020 con 0x7fd67c107730 2024-04-25T01:40:20.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.776+0000 7fd681d89640 1 -- 172.21.15.28:0/2957482768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd67c1a6790 con 0x7fd67c107730 2024-04-25T01:40:20.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.776+0000 7fd678ff9640 1 -- 172.21.15.28:0/2957482768 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67400f6f0 con 0x7fd67c107730 2024-04-25T01:40:20.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.776+0000 7fd681d89640 1 -- 172.21.15.28:0/2957482768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd67c1a6d30 con 0x7fd67c107730 2024-04-25T01:40:20.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.779+0000 7fd678ff9640 1 -- 172.21.15.28:0/2957482768 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd6740051d0 con 0x7fd67c107730 2024-04-25T01:40:20.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.779+0000 7fd684b74740 1 Processor -- start 2024-04-25T01:40:20.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.779+0000 7fd684b74740 1 -- start start 2024-04-25T01:40:20.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.779+0000 7fd684b74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228630 0x55b232228a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.779+0000 7fd684b74740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b232228f50 0x55b2322330e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.779+0000 7fd684b74740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232233620 0x55b232235a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.779+0000 7fd684b74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b23218d6b0 con 0x55b232228630 2024-04-25T01:40:20.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.779+0000 7fd684b74740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b232135880 con 0x55b232233620 2024-04-25T01:40:20.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.779+0000 7fd684b74740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b232189670 con 0x55b232228f50 2024-04-25T01:40:20.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.779+0000 7fd67affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228630 0x55b232228a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.779+0000 7fd67affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228630 0x55b232228a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34494/0 (socket says 172.21.15.28:34494) 2024-04-25T01:40:20.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.779+0000 7fd67affd640 1 -- 172.21.15.28:0/393990668 learned_addr learned my addr 172.21.15.28:0/393990668 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:20.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.780+0000 7fd67b7fe640 1 --2- 172.21.15.28:0/393990668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b232228f50 0x55b2322330e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.780+0000 7fd678ff9640 1 --2- 172.21.15.28:0/2957482768 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd6640a2d80 0x7fd6640a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.780+0000 7fd67affd640 1 --2- 172.21.15.28:0/393990668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232233620 0x55b232235a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.780+0000 7fd67bfff640 1 --2- 172.21.15.28:0/2957482768 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd6640a2d80 0x7fd6640a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.780+0000 7fd67affd640 1 -- 172.21.15.28:0/393990668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b232228f50 msgr2=0x55b2322330e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:20.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.780+0000 7fd67affd640 1 --2- 172.21.15.28:0/393990668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b232228f50 0x55b2322330e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.780+0000 7fd67affd640 1 -- 172.21.15.28:0/393990668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232233620 msgr2=0x55b232235a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:20.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.780+0000 7fd67affd640 1 --2- 172.21.15.28:0/393990668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232233620 0x55b232235a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.780+0000 7fd67affd640 1 -- 172.21.15.28:0/393990668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b231f6b7d0 con 0x55b232228630 2024-04-25T01:40:20.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.780+0000 7fd678ff9640 1 -- 172.21.15.28:0/2957482768 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fd6740cf9d0 con 0x7fd67c107730 2024-04-25T01:40:20.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.780+0000 7fd67affd640 1 --2- 172.21.15.28:0/393990668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228630 0x55b232228a10 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fd67000bfa0 tx=0x7fd67000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c44b577fcb90e331 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:20.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.780+0000 7fd661ffb640 1 -- 172.21.15.28:0/393990668 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd670004030 con 0x55b232228630 2024-04-25T01:40:20.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.780+0000 7fd67bfff640 1 --2- 172.21.15.28:0/2957482768 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd6640a2d80 0x7fd6640a5240 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd674098070 tx=0x7fd674066ce0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:20.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.780+0000 7fd661ffb640 1 -- 172.21.15.28:0/393990668 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6700041d0 con 0x55b232228630 2024-04-25T01:40:20.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.780+0000 7fd661ffb640 1 -- 172.21.15.28:0/393990668 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67000f6e0 con 0x55b232228630 2024-04-25T01:40:20.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.781+0000 7fd684b74740 1 -- 172.21.15.28:0/393990668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228630 msgr2=0x55b232228a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:20.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.781+0000 7fd684b74740 1 --2- 172.21.15.28:0/393990668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228630 0x55b232228a10 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fd67000bfa0 tx=0x7fd67000b840 comp rx=0 tx=0).stop 2024-04-25T01:40:20.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.781+0000 7fd684b74740 1 -- 172.21.15.28:0/393990668 shutdown_connections 2024-04-25T01:40:20.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.781+0000 7fd684b74740 1 --2- 172.21.15.28:0/393990668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232233620 0x55b232235a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.781+0000 7fd684b74740 1 --2- 172.21.15.28:0/393990668 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b232228f50 0x55b2322330e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.781+0000 7fd684b74740 1 --2- 172.21.15.28:0/393990668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228630 0x55b232228a10 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.781+0000 7fd684b74740 1 -- 172.21.15.28:0/393990668 >> 172.21.15.28:0/393990668 conn(0x55b232224950 msgr2=0x55b232224d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:20.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.781+0000 7fd684b74740 1 -- 172.21.15.28:0/393990668 shutdown_connections 2024-04-25T01:40:20.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.781+0000 7fd684b74740 1 -- 172.21.15.28:0/393990668 wait complete. 2024-04-25T01:40:20.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.781+0000 7fd684b74740 1 -- start start 2024-04-25T01:40:20.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd684b74740 1 Processor -- start 2024-04-25T01:40:20.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd684b74740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b232228630 0x55b232230600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd684b74740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232228f50 0x55b232230b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd684b74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232233620 0x55b232231080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd684b74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b232236410 con 0x55b232233620 2024-04-25T01:40:20.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd684b74740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b232148f50 con 0x55b232228f50 2024-04-25T01:40:20.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd684b74740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b232236590 con 0x55b232228630 2024-04-25T01:40:20.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd684b74740 10 client.0 _ll_register_callbacks cb 0x55b232184000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:20.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd67affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232233620 0x55b232231080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd67affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232233620 0x55b232231080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34500/0 (socket says 172.21.15.28:34500) 2024-04-25T01:40:20.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd67affd640 1 -- 172.21.15.28:0/2786991006 learned_addr learned my addr 172.21.15.28:0/2786991006 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:20.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd67b7fe640 1 --2- 172.21.15.28:0/2786991006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232228f50 0x55b232230b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd67affd640 1 --2- 172.21.15.28:0/2786991006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b232228630 0x55b232230600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd67affd640 1 -- 172.21.15.28:0/2786991006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b232228630 msgr2=0x55b232230600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:20.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd67affd640 1 --2- 172.21.15.28:0/2786991006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b232228630 0x55b232230600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd67affd640 1 -- 172.21.15.28:0/2786991006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232228f50 msgr2=0x55b232230b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:20.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd67affd640 1 --2- 172.21.15.28:0/2786991006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232228f50 0x55b232230b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd67affd640 1 -- 172.21.15.28:0/2786991006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b231ea13f0 con 0x55b232233620 2024-04-25T01:40:20.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd67affd640 1 --2- 172.21.15.28:0/2786991006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232233620 0x55b232231080 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fd670006010 tx=0x7fd670003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b97932aea1faff07 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:20.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.782+0000 7fd65affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:20.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.783+0000 7fd65affd640 1 -- 172.21.15.28:0/2786991006 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6700035f0 con 0x55b232233620 2024-04-25T01:40:20.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.783+0000 7fd65affd640 1 -- 172.21.15.28:0/2786991006 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fd670003790 con 0x55b232233620 2024-04-25T01:40:20.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.783+0000 7fd684b74740 1 -- 172.21.15.28:0/2786991006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6700041d0 con 0x55b232233620 2024-04-25T01:40:20.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.783+0000 7fd65affd640 1 -- 172.21.15.28:0/2786991006 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd670003990 con 0x55b232233620 2024-04-25T01:40:20.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.783+0000 7fd684b74740 10 client.18135 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:20.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.783+0000 7fd684b74740 1 -- 172.21.15.28:0/2786991006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b231e610f0 con 0x55b232233620 2024-04-25T01:40:20.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.783+0000 7fd65affd640 1 -- 172.21.15.28:0/2786991006 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6700041d0 con 0x55b232233620 2024-04-25T01:40:20.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.783+0000 7fd65affd640 1 -- 172.21.15.28:0/2786991006 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd67001d070 con 0x55b232233620 2024-04-25T01:40:20.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.783+0000 7fd684b74740 10 client.18135 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:20.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.783+0000 7fd684b74740 10 client.18135 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:20.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.783+0000 7fd684b74740 20 client.18135 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:20.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.783+0000 7fd684b74740 1 --2- 172.21.15.28:0/2786991006 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b232243cc0 0x55b2322460d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.783+0000 7fd684b74740 4 client.18135 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:20.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.783+0000 7fd684b74740 1 -- 172.21.15.28:0/2786991006 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b23219abf0 con 0x55b232243cc0 2024-04-25T01:40:20.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.784+0000 7fd67b7fe640 1 --2- 172.21.15.28:0/2786991006 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b232243cc0 0x55b2322460d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.784+0000 7fd65affd640 10 client.18135 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:20.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.784+0000 7fd67b7fe640 1 --2- 172.21.15.28:0/2786991006 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b232243cc0 0x55b2322460d0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9383f5442aa91483 server_cookie=dddaa6f184a54a5d in_seq=0 out_seq=0 2024-04-25T01:40:20.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.785+0000 7fd65affd640 1 -- 172.21.15.28:0/2786991006 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b23219abf0 con 0x55b232243cc0 2024-04-25T01:40:20.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.785+0000 7fd65affd640 10 client.18135 handle_command_reply: tid=0 2024-04-25T01:40:20.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.785+0000 7fd684b74740 1 client.18135 shutdown 2024-04-25T01:40:20.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.785+0000 7fd684b74740 1 -- 172.21.15.28:0/2786991006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232233620 msgr2=0x55b232231080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:20.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.785+0000 7fd684b74740 1 --2- 172.21.15.28:0/2786991006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232233620 0x55b232231080 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fd670006010 tx=0x7fd670003c10 comp rx=0 tx=0).stop 2024-04-25T01:40:20.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.785+0000 7fd684b74740 1 -- 172.21.15.28:0/2786991006 shutdown_connections 2024-04-25T01:40:20.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.785+0000 7fd684b74740 1 --2- 172.21.15.28:0/2786991006 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b232243cc0 0x55b2322460d0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.785+0000 7fd684b74740 1 --2- 172.21.15.28:0/2786991006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232233620 0x55b232231080 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.785+0000 7fd684b74740 1 --2- 172.21.15.28:0/2786991006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232228f50 0x55b232230b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.785+0000 7fd65affd640 0 client.18135 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:20.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.785+0000 7fd684b74740 1 --2- 172.21.15.28:0/2786991006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b232228630 0x55b232230600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.785+0000 7fd684b74740 1 -- 172.21.15.28:0/2786991006 >> 172.21.15.28:0/2786991006 conn(0x55b232224950 msgr2=0x55b23222a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:20.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.785+0000 7fd684b74740 1 -- 172.21.15.28:0/2786991006 shutdown_connections 2024-04-25T01:40:20.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.785+0000 7fd684b74740 1 -- 172.21.15.28:0/2786991006 wait complete. 2024-04-25T01:40:20.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.785+0000 7fd684b74740 20 client.18135 trim_cache size 0 max 16384 2024-04-25T01:40:20.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.803+0000 7fd684b74740 1 Processor -- start 2024-04-25T01:40:20.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.803+0000 7fd684b74740 1 -- start start 2024-04-25T01:40:20.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.803+0000 7fd684b74740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b232228630 0x55b232232d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.803+0000 7fd684b74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228f50 0x55b23223f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.803+0000 7fd684b74740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232233620 0x55b232223970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.803+0000 7fd684b74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c1b0dc0 con 0x55b232228f50 2024-04-25T01:40:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.803+0000 7fd684b74740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c1b2900 con 0x55b232233620 2024-04-25T01:40:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.803+0000 7fd684b74740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c1b1490 con 0x55b232228630 2024-04-25T01:40:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.803+0000 7fd67b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228f50 0x55b23223f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.803+0000 7fd67b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228f50 0x55b23223f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34504/0 (socket says 172.21.15.28:34504) 2024-04-25T01:40:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.803+0000 7fd67b7fe640 1 -- 172.21.15.28:0/1667464828 learned_addr learned my addr 172.21.15.28:0/1667464828 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.803+0000 7fd67b7fe640 1 -- 172.21.15.28:0/1667464828 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b232228630 msgr2=0x55b232232d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.803+0000 7fd67b7fe640 1 --2- 172.21.15.28:0/1667464828 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b232228630 0x55b232232d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.803+0000 7fd67b7fe640 1 -- 172.21.15.28:0/1667464828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232233620 msgr2=0x55b232223970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.803+0000 7fd67b7fe640 1 --2- 172.21.15.28:0/1667464828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232233620 0x55b232223970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.803+0000 7fd67b7fe640 1 -- 172.21.15.28:0/1667464828 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b232225290 con 0x55b232228f50 2024-04-25T01:40:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.803+0000 7fd67b7fe640 1 --2- 172.21.15.28:0/1667464828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228f50 0x55b23223f2d0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fd66c000c00 tx=0x7fd66c031b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a6fecfd8ad80b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:20.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.804+0000 7fd6617fa640 1 -- 172.21.15.28:0/1667464828 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66c04c070 con 0x55b232228f50 2024-04-25T01:40:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.804+0000 7fd6617fa640 1 -- 172.21.15.28:0/1667464828 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd66c048900 con 0x55b232228f50 2024-04-25T01:40:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.804+0000 7fd6617fa640 1 -- 172.21.15.28:0/1667464828 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66c035040 con 0x55b232228f50 2024-04-25T01:40:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.804+0000 7fd684b74740 1 -- 172.21.15.28:0/1667464828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228f50 msgr2=0x55b23223f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.804+0000 7fd684b74740 1 --2- 172.21.15.28:0/1667464828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228f50 0x55b23223f2d0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fd66c000c00 tx=0x7fd66c031b40 comp rx=0 tx=0).stop 2024-04-25T01:40:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.804+0000 7fd684b74740 1 -- 172.21.15.28:0/1667464828 shutdown_connections 2024-04-25T01:40:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.804+0000 7fd684b74740 1 --2- 172.21.15.28:0/1667464828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232233620 0x55b232223970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.804+0000 7fd684b74740 1 --2- 172.21.15.28:0/1667464828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228f50 0x55b23223f2d0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.804+0000 7fd684b74740 1 --2- 172.21.15.28:0/1667464828 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b232228630 0x55b232232d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.804+0000 7fd684b74740 1 -- 172.21.15.28:0/1667464828 >> 172.21.15.28:0/1667464828 conn(0x55b232224950 msgr2=0x55b232223260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.804+0000 7fd684b74740 1 -- 172.21.15.28:0/1667464828 shutdown_connections 2024-04-25T01:40:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.804+0000 7fd684b74740 1 -- 172.21.15.28:0/1667464828 wait complete. 2024-04-25T01:40:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.804+0000 7fd684b74740 1 -- start start 2024-04-25T01:40:20.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.805+0000 7fd684b74740 1 Processor -- start 2024-04-25T01:40:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.805+0000 7fd684b74740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b232228630 0x55b23223bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.805+0000 7fd684b74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228f50 0x55b23222dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.805+0000 7fd684b74740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232233620 0x55b23222c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.805+0000 7fd684b74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c1b2230 con 0x55b232228f50 2024-04-25T01:40:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.805+0000 7fd684b74740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c1b1b60 con 0x55b232233620 2024-04-25T01:40:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.805+0000 7fd684b74740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c1aedc0 con 0x55b232228630 2024-04-25T01:40:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.805+0000 7fd684b74740 10 client.0 _ll_register_callbacks cb 0x7fd670012480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.805+0000 7fd67b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228f50 0x55b23222dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.805+0000 7fd67b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228f50 0x55b23222dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34512/0 (socket says 172.21.15.28:34512) 2024-04-25T01:40:20.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.805+0000 7fd67b7fe640 1 -- 172.21.15.28:0/2835942016 learned_addr learned my addr 172.21.15.28:0/2835942016 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:20.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.805+0000 7fd67affd640 1 --2- 172.21.15.28:0/2835942016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232233620 0x55b23222c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.805+0000 7fd67affd640 1 --2- 172.21.15.28:0/2835942016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b232228630 0x55b23223bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.805+0000 7fd67b7fe640 1 -- 172.21.15.28:0/2835942016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b232228630 msgr2=0x55b23223bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:20.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.805+0000 7fd67b7fe640 1 --2- 172.21.15.28:0/2835942016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b232228630 0x55b23223bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.805+0000 7fd67b7fe640 1 -- 172.21.15.28:0/2835942016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232233620 msgr2=0x55b23222c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.805+0000 7fd67b7fe640 1 --2- 172.21.15.28:0/2835942016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232233620 0x55b23222c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.805+0000 7fd67b7fe640 1 -- 172.21.15.28:0/2835942016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b23222e640 con 0x55b232228f50 2024-04-25T01:40:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.805+0000 7fd67affd640 1 --2- 172.21.15.28:0/2835942016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232233620 0x55b23222c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.806+0000 7fd67b7fe640 1 --2- 172.21.15.28:0/2835942016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228f50 0x55b23222dd70 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fd66c005940 tx=0x7fd66c005180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba964802d64d2673 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.806+0000 7fd65affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:20.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.806+0000 7fd65affd640 1 -- 172.21.15.28:0/2835942016 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66c035070 con 0x55b232228f50 2024-04-25T01:40:20.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.806+0000 7fd684b74740 1 -- 172.21.15.28:0/2835942016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd66c048900 con 0x55b232228f50 2024-04-25T01:40:20.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.806+0000 7fd65affd640 1 -- 172.21.15.28:0/2835942016 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fd66c048620 con 0x55b232228f50 2024-04-25T01:40:20.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.806+0000 7fd65affd640 1 -- 172.21.15.28:0/2835942016 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd66c061050 con 0x55b232228f50 2024-04-25T01:40:20.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.806+0000 7fd684b74740 10 client.18141 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:20.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.806+0000 7fd684b74740 1 -- 172.21.15.28:0/2835942016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b2322a9820 con 0x55b232228f50 2024-04-25T01:40:20.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.806+0000 7fd65affd640 1 -- 172.21.15.28:0/2835942016 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd66c048900 con 0x55b232228f50 2024-04-25T01:40:20.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.806+0000 7fd65affd640 1 -- 172.21.15.28:0/2835942016 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd66c0035f0 con 0x55b232228f50 2024-04-25T01:40:20.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.806+0000 7fd684b74740 10 client.18141 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:20.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.806+0000 7fd684b74740 10 client.18141 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:20.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.806+0000 7fd684b74740 20 client.18141 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:20.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.806+0000 7fd684b74740 1 --2- 172.21.15.28:0/2835942016 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b2322a9fc0 0x55b2322ac3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.806+0000 7fd684b74740 4 client.18141 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:40:20.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.806+0000 7fd684b74740 1 -- 172.21.15.28:0/2835942016 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b232194f40 con 0x55b2322a9fc0 2024-04-25T01:40:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.807+0000 7fd67affd640 1 --2- 172.21.15.28:0/2835942016 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b2322a9fc0 0x55b2322ac3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.807+0000 7fd67affd640 1 --2- 172.21.15.28:0/2835942016 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b2322a9fc0 0x55b2322ac3e0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75dd211f39e072c0 server_cookie=dbc3d31dbba6d296 in_seq=0 out_seq=0 2024-04-25T01:40:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.807+0000 7fd65affd640 10 client.18141 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.807+0000 7fd65affd640 1 -- 172.21.15.28:0/2835942016 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b232194f40 con 0x55b2322a9fc0 2024-04-25T01:40:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.807+0000 7fd65affd640 10 client.18141 handle_command_reply: tid=0 2024-04-25T01:40:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.807+0000 7fd684b74740 1 client.18141 shutdown 2024-04-25T01:40:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.807+0000 7fd684b74740 1 -- 172.21.15.28:0/2835942016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228f50 msgr2=0x55b23222dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:20.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.807+0000 7fd684b74740 1 --2- 172.21.15.28:0/2835942016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228f50 0x55b23222dd70 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fd66c005940 tx=0x7fd66c005180 comp rx=0 tx=0).stop 2024-04-25T01:40:20.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.807+0000 7fd684b74740 1 -- 172.21.15.28:0/2835942016 shutdown_connections 2024-04-25T01:40:20.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.807+0000 7fd684b74740 1 --2- 172.21.15.28:0/2835942016 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b2322a9fc0 0x55b2322ac3e0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.807+0000 7fd684b74740 1 --2- 172.21.15.28:0/2835942016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b232233620 0x55b23222c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.807+0000 7fd65affd640 0 client.18141 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:20.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.807+0000 7fd684b74740 1 --2- 172.21.15.28:0/2835942016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b232228f50 0x55b23222dd70 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.807+0000 7fd684b74740 1 --2- 172.21.15.28:0/2835942016 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b232228630 0x55b23223bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.807+0000 7fd684b74740 1 -- 172.21.15.28:0/2835942016 >> 172.21.15.28:0/2835942016 conn(0x55b232224950 msgr2=0x55b232230e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.808+0000 7fd684b74740 1 -- 172.21.15.28:0/2835942016 shutdown_connections 2024-04-25T01:40:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.808+0000 7fd684b74740 1 -- 172.21.15.28:0/2835942016 wait complete. 2024-04-25T01:40:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.808+0000 7fd684b74740 20 client.18141 trim_cache size 0 max 16384 2024-04-25T01:40:20.806 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:40:20.807 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:40:20.807 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:40:20.807 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:40:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.808+0000 7fd681d89640 1 -- 172.21.15.28:0/2957482768 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd6640a2d80 msgr2=0x7fd6640a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.808+0000 7fd681d89640 1 --2- 172.21.15.28:0/2957482768 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd6640a2d80 0x7fd6640a5240 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd674098070 tx=0x7fd674066ce0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.808+0000 7fd681d89640 1 -- 172.21.15.28:0/2957482768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c107730 msgr2=0x7fd67c1010f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:20.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.808+0000 7fd681d89640 1 --2- 172.21.15.28:0/2957482768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c107730 0x7fd67c1010f0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fd67400cad0 tx=0x7fd67400cf90 comp rx=0 tx=0).stop 2024-04-25T01:40:20.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.808+0000 7fd681d89640 1 -- 172.21.15.28:0/2957482768 shutdown_connections 2024-04-25T01:40:20.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.808+0000 7fd681d89640 1 --2- 172.21.15.28:0/2957482768 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd6640a2d80 0x7fd6640a5240 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.808+0000 7fd681d89640 1 --2- 172.21.15.28:0/2957482768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd67c107730 0x7fd67c1010f0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.808+0000 7fd681d89640 1 --2- 172.21.15.28:0/2957482768 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd67c06a780 0x7fd67c100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.808+0000 7fd681d89640 1 --2- 172.21.15.28:0/2957482768 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd67c069dc0 0x7fd67c100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.808+0000 7fd681d89640 1 -- 172.21.15.28:0/2957482768 >> 172.21.15.28:0/2957482768 conn(0x7fd67c0776a0 msgr2=0x7fd67c0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.808+0000 7fd681d89640 1 -- 172.21.15.28:0/2957482768 shutdown_connections 2024-04-25T01:40:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.808+0000 7fd681d89640 1 -- 172.21.15.28:0/2957482768 wait complete. 2024-04-25T01:40:20.811 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:40:20.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:40:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.931+0000 7f3ac7cd4640 1 -- 172.21.15.28:0/3917986317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac01032c0 msgr2=0x7f3ac010b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.931+0000 7f3ac7cd4640 1 --2- 172.21.15.28:0/3917986317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac01032c0 0x7f3ac010b7a0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0009900 tx=0x7f3ab002a480 comp rx=0 tx=0).stop 2024-04-25T01:40:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.931+0000 7f3ac7cd4640 1 -- 172.21.15.28:0/3917986317 shutdown_connections 2024-04-25T01:40:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.931+0000 7f3ac7cd4640 1 --2- 172.21.15.28:0/3917986317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ac010bd70 0x7f3ac010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.931+0000 7f3ac7cd4640 1 --2- 172.21.15.28:0/3917986317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac01032c0 0x7f3ac010b7a0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.931+0000 7f3ac7cd4640 1 --2- 172.21.15.28:0/3917986317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ac01029a0 0x7f3ac0102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.931+0000 7f3ac7cd4640 1 -- 172.21.15.28:0/3917986317 >> 172.21.15.28:0/3917986317 conn(0x7f3ac00769c0 msgr2=0x7f3ac0076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.931+0000 7f3ac7cd4640 1 -- 172.21.15.28:0/3917986317 shutdown_connections 2024-04-25T01:40:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.931+0000 7f3ac7cd4640 1 -- 172.21.15.28:0/3917986317 wait complete. 2024-04-25T01:40:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.932+0000 7f3ac7cd4640 1 Processor -- start 2024-04-25T01:40:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.932+0000 7f3ac7cd4640 1 -- start start 2024-04-25T01:40:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.932+0000 7f3ac7cd4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ac01029a0 0x7f3ac0198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.932+0000 7f3ac7cd4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac01032c0 0x7f3ac0198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.932+0000 7f3ac7cd4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ac010bd70 0x7f3ac019c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.932+0000 7f3ac7cd4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac0111b50 con 0x7f3ac01032c0 2024-04-25T01:40:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.932+0000 7f3ac7cd4640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac01119d0 con 0x7f3ac010bd70 2024-04-25T01:40:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.932+0000 7f3ac7cd4640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac0111cd0 con 0x7f3ac01029a0 2024-04-25T01:40:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.932+0000 7f3ac526f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac01032c0 0x7f3ac0198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.932+0000 7f3ac526f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac01032c0 0x7f3ac0198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34538/0 (socket says 172.21.15.28:34538) 2024-04-25T01:40:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.932+0000 7f3ac526f640 1 -- 172.21.15.28:0/168441658 learned_addr learned my addr 172.21.15.28:0/168441658 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.932+0000 7f3ac5a70640 1 --2- 172.21.15.28:0/168441658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ac01029a0 0x7f3ac0198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.932+0000 7f3ac6271640 1 --2- 172.21.15.28:0/168441658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ac010bd70 0x7f3ac019c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.933+0000 7f3ac526f640 1 -- 172.21.15.28:0/168441658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ac01029a0 msgr2=0x7f3ac0198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.933+0000 7f3ac526f640 1 --2- 172.21.15.28:0/168441658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ac01029a0 0x7f3ac0198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.933+0000 7f3ac526f640 1 -- 172.21.15.28:0/168441658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ac010bd70 msgr2=0x7f3ac019c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.933+0000 7f3ac526f640 1 --2- 172.21.15.28:0/168441658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ac010bd70 0x7f3ac019c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.933+0000 7f3ac526f640 1 -- 172.21.15.28:0/168441658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac019cff0 con 0x7f3ac01032c0 2024-04-25T01:40:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.933+0000 7f3ac5a70640 1 --2- 172.21.15.28:0/168441658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ac01029a0 0x7f3ac0198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.933+0000 7f3ac6271640 1 --2- 172.21.15.28:0/168441658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ac010bd70 0x7f3ac019c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.933+0000 7f3ac526f640 1 --2- 172.21.15.28:0/168441658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac01032c0 0x7f3ac0198590 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0004770 tx=0x7f3ab0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.933+0000 7f3ab6ffd640 1 -- 172.21.15.28:0/168441658 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab003f070 con 0x7f3ac01032c0 2024-04-25T01:40:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.933+0000 7f3ab6ffd640 1 -- 172.21.15.28:0/168441658 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ab0031ca0 con 0x7f3ac01032c0 2024-04-25T01:40:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.933+0000 7f3ac7cd4640 1 -- 172.21.15.28:0/168441658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ac019d280 con 0x7f3ac01032c0 2024-04-25T01:40:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.933+0000 7f3ab6ffd640 1 -- 172.21.15.28:0/168441658 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0035030 con 0x7f3ac01032c0 2024-04-25T01:40:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.933+0000 7f3ac7cd4640 1 -- 172.21.15.28:0/168441658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ac01a4b70 con 0x7f3ac01032c0 2024-04-25T01:40:20.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.934+0000 7f3ac7cd4640 1 -- 172.21.15.28:0/168441658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ac006a5c0 con 0x7f3ac01032c0 2024-04-25T01:40:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.935+0000 7f3ab6ffd640 1 -- 172.21.15.28:0/168441658 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3ab0042070 con 0x7f3ac01032c0 2024-04-25T01:40:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.936+0000 7f3ab6ffd640 1 --2- 172.21.15.28:0/168441658 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3aac0a2d80 0x7f3aac0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.936+0000 7f3ac5a70640 1 --2- 172.21.15.28:0/168441658 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3aac0a2d80 0x7f3aac0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.936+0000 7f3ab6ffd640 1 -- 172.21.15.28:0/168441658 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f3ab00eff70 con 0x7f3ac01032c0 2024-04-25T01:40:20.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.936+0000 7f3ac5a70640 1 --2- 172.21.15.28:0/168441658 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3aac0a2d80 0x7f3aac0a5240 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3abc005fd0 tx=0x7f3abc00b5f0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:20.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:20.940+0000 7f3ab6ffd640 1 -- 172.21.15.28:0/168441658 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3ab00bc7a0 con 0x7f3ac01032c0 2024-04-25T01:40:21.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.163+0000 7f3ac7cd4640 1 -- 172.21.15.28:0/168441658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ac0106ec0 con 0x7f3ac01032c0 2024-04-25T01:40:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.164+0000 7f3ab6ffd640 1 -- 172.21.15.28:0/168441658 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7f3ab00bc0a0 con 0x7f3ac01032c0 2024-04-25T01:40:21.148 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:40:21.148 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:40:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:40:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.166+0000 7f3ac7cd4640 1 -- 172.21.15.28:0/168441658 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3aac0a2d80 msgr2=0x7f3aac0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.166+0000 7f3ac7cd4640 1 --2- 172.21.15.28:0/168441658 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3aac0a2d80 0x7f3aac0a5240 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3abc005fd0 tx=0x7f3abc00b5f0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.166+0000 7f3ac7cd4640 1 -- 172.21.15.28:0/168441658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac01032c0 msgr2=0x7f3ac0198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.166+0000 7f3ac7cd4640 1 --2- 172.21.15.28:0/168441658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac01032c0 0x7f3ac0198590 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0004770 tx=0x7f3ab0003c10 comp rx=0 tx=0).stop 2024-04-25T01:40:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.167+0000 7f3ac7cd4640 1 -- 172.21.15.28:0/168441658 shutdown_connections 2024-04-25T01:40:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.167+0000 7f3ac7cd4640 1 --2- 172.21.15.28:0/168441658 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3aac0a2d80 0x7f3aac0a5240 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.167+0000 7f3ac7cd4640 1 --2- 172.21.15.28:0/168441658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ac010bd70 0x7f3ac019c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.167+0000 7f3ac7cd4640 1 --2- 172.21.15.28:0/168441658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ac01032c0 0x7f3ac0198590 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.167+0000 7f3ac7cd4640 1 --2- 172.21.15.28:0/168441658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ac01029a0 0x7f3ac0198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.167+0000 7f3ac7cd4640 1 -- 172.21.15.28:0/168441658 >> 172.21.15.28:0/168441658 conn(0x7f3ac00769c0 msgr2=0x7f3ac00fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.167+0000 7f3ac7cd4640 1 -- 172.21.15.28:0/168441658 shutdown_connections 2024-04-25T01:40:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.167+0000 7f3ac7cd4640 1 -- 172.21.15.28:0/168441658 wait complete. 2024-04-25T01:40:21.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:40:21.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.283+0000 7f988f25d640 1 -- 172.21.15.28:0/1487159228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888107720 msgr2=0x7f9888107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.283+0000 7f988f25d640 1 --2- 172.21.15.28:0/1487159228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888107720 0x7f9888107b00 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f9878009930 tx=0x7f987802a420 comp rx=0 tx=0).stop 2024-04-25T01:40:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.283+0000 7f988f25d640 1 -- 172.21.15.28:0/1487159228 shutdown_connections 2024-04-25T01:40:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.283+0000 7f988f25d640 1 --2- 172.21.15.28:0/1487159228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f98881020d0 0x7f988810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.283+0000 7f988f25d640 1 --2- 172.21.15.28:0/1487159228 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9888101710 0x7f9888101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.283+0000 7f988f25d640 1 --2- 172.21.15.28:0/1487159228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888107720 0x7f9888107b00 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.283+0000 7f988f25d640 1 -- 172.21.15.28:0/1487159228 >> 172.21.15.28:0/1487159228 conn(0x7f9888078b90 msgr2=0x7f9888078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.283+0000 7f988f25d640 1 -- 172.21.15.28:0/1487159228 shutdown_connections 2024-04-25T01:40:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.283+0000 7f988f25d640 1 -- 172.21.15.28:0/1487159228 wait complete. 2024-04-25T01:40:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.283+0000 7f988f25d640 1 Processor -- start 2024-04-25T01:40:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.283+0000 7f988f25d640 1 -- start start 2024-04-25T01:40:21.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f988f25d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888101710 0x7f9888198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f988f25d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98881020d0 0x7f98881985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f988f25d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9888107720 0x7f988819c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f988f25d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9888111b30 con 0x7f9888101710 2024-04-25T01:40:21.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f988f25d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f98881119b0 con 0x7f9888107720 2024-04-25T01:40:21.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f988f25d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9888111cb0 con 0x7f98881020d0 2024-04-25T01:40:21.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f988cff9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888101710 0x7f9888198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f988cff9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888101710 0x7f9888198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34552/0 (socket says 172.21.15.28:34552) 2024-04-25T01:40:21.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f988cff9640 1 -- 172.21.15.28:0/2398838533 learned_addr learned my addr 172.21.15.28:0/2398838533 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f988d7fa640 1 --2- 172.21.15.28:0/2398838533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9888107720 0x7f988819c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f9887fff640 1 --2- 172.21.15.28:0/2398838533 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98881020d0 0x7f98881985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f988cff9640 1 -- 172.21.15.28:0/2398838533 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98881020d0 msgr2=0x7f98881985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f988cff9640 1 --2- 172.21.15.28:0/2398838533 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98881020d0 0x7f98881985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f988cff9640 1 -- 172.21.15.28:0/2398838533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9888107720 msgr2=0x7f988819c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f988cff9640 1 --2- 172.21.15.28:0/2398838533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9888107720 0x7f988819c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f988cff9640 1 -- 172.21.15.28:0/2398838533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988819d080 con 0x7f9888101710 2024-04-25T01:40:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f988d7fa640 1 --2- 172.21.15.28:0/2398838533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9888107720 0x7f988819c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f988cff9640 1 --2- 172.21.15.28:0/2398838533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888101710 0x7f9888198090 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f9878009230 tx=0x7f9878004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f9885ffb640 1 -- 172.21.15.28:0/2398838533 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987803f030 con 0x7f9888101710 2024-04-25T01:40:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f988f25d640 1 -- 172.21.15.28:0/2398838533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f988819d310 con 0x7f9888101710 2024-04-25T01:40:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f9885ffb640 1 -- 172.21.15.28:0/2398838533 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9878003ad0 con 0x7f9888101710 2024-04-25T01:40:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.284+0000 7f9885ffb640 1 -- 172.21.15.28:0/2398838533 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9878035070 con 0x7f9888101710 2024-04-25T01:40:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.285+0000 7f988f25d640 1 -- 172.21.15.28:0/2398838533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98881a4c00 con 0x7f9888101710 2024-04-25T01:40:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.287+0000 7f9885ffb640 1 -- 172.21.15.28:0/2398838533 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9878042030 con 0x7f9888101710 2024-04-25T01:40:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.287+0000 7f9891f6a740 1 Processor -- start 2024-04-25T01:40:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.287+0000 7f9891f6a740 1 -- start start 2024-04-25T01:40:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.287+0000 7f9891f6a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967bb580 0x556e967bb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.287+0000 7f9891f6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 0x556e967c6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.287+0000 7f9891f6a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967c6570 0x556e967c8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.287+0000 7f9891f6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556e96720670 con 0x556e967bbea0 2024-04-25T01:40:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.287+0000 7f9891f6a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556e966c8970 con 0x556e967bb580 2024-04-25T01:40:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.287+0000 7f9891f6a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x556e9671c630 con 0x556e967c6570 2024-04-25T01:40:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.287+0000 7f988cff9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 0x556e967c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.287+0000 7f988cff9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 0x556e967c6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34560/0 (socket says 172.21.15.28:34560) 2024-04-25T01:40:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.287+0000 7f988cff9640 1 -- 172.21.15.28:0/3112543017 learned_addr learned my addr 172.21.15.28:0/3112543017 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.287+0000 7f9885ffb640 1 --2- 172.21.15.28:0/2398838533 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f98700a2d00 0x7f98700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.288+0000 7f988d7fa640 1 --2- 172.21.15.28:0/3112543017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967bb580 0x556e967bb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.288+0000 7f9887fff640 1 --2- 172.21.15.28:0/3112543017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967c6570 0x556e967c8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.288+0000 7f988d7fa640 1 --2- 172.21.15.28:0/2398838533 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f98700a2d00 0x7f98700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.288+0000 7f9885ffb640 1 -- 172.21.15.28:0/2398838533 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f98780efd80 con 0x7f9888101710 2024-04-25T01:40:21.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.288+0000 7f988cff9640 1 -- 172.21.15.28:0/3112543017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967c6570 msgr2=0x556e967c8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.288+0000 7f988cff9640 1 --2- 172.21.15.28:0/3112543017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967c6570 0x556e967c8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.288+0000 7f988cff9640 1 -- 172.21.15.28:0/3112543017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967bb580 msgr2=0x556e967bb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.288+0000 7f988cff9640 1 --2- 172.21.15.28:0/3112543017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967bb580 0x556e967bb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.288+0000 7f988cff9640 1 -- 172.21.15.28:0/3112543017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e966ec400 con 0x556e967bbea0 2024-04-25T01:40:21.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.288+0000 7f9887fff640 1 --2- 172.21.15.28:0/3112543017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967c6570 0x556e967c8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:21.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.288+0000 7f988d7fa640 1 --2- 172.21.15.28:0/3112543017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967bb580 0x556e967bb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:21.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.288+0000 7f988cff9640 1 --2- 172.21.15.28:0/3112543017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 0x556e967c6030 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f98780b8370 tx=0x7f98780dc140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e234cd79686c0a24 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.288+0000 7f988d7fa640 1 --2- 172.21.15.28:0/2398838533 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f98700a2d00 0x7f98700a51c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9880005c40 tx=0x7f9880009280 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.288+0000 7f986effd640 1 -- 172.21.15.28:0/3112543017 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9878049070 con 0x556e967bbea0 2024-04-25T01:40:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.288+0000 7f986effd640 1 -- 172.21.15.28:0/3112543017 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98780bc610 con 0x556e967bbea0 2024-04-25T01:40:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.288+0000 7f986effd640 1 -- 172.21.15.28:0/3112543017 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9878003c80 con 0x556e967bbea0 2024-04-25T01:40:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.289+0000 7f9891f6a740 1 -- 172.21.15.28:0/3112543017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 msgr2=0x556e967c6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.289+0000 7f9891f6a740 1 --2- 172.21.15.28:0/3112543017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 0x556e967c6030 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f98780b8370 tx=0x7f98780dc140 comp rx=0 tx=0).stop 2024-04-25T01:40:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.289+0000 7f9891f6a740 1 -- 172.21.15.28:0/3112543017 shutdown_connections 2024-04-25T01:40:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.289+0000 7f9891f6a740 1 --2- 172.21.15.28:0/3112543017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967c6570 0x556e967c8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.289+0000 7f9891f6a740 1 --2- 172.21.15.28:0/3112543017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 0x556e967c6030 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.289+0000 7f9891f6a740 1 --2- 172.21.15.28:0/3112543017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967bb580 0x556e967bb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.289+0000 7f9891f6a740 1 -- 172.21.15.28:0/3112543017 >> 172.21.15.28:0/3112543017 conn(0x556e967b78a0 msgr2=0x556e967b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.289+0000 7f9891f6a740 1 -- 172.21.15.28:0/3112543017 shutdown_connections 2024-04-25T01:40:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.289+0000 7f9891f6a740 1 -- 172.21.15.28:0/3112543017 wait complete. 2024-04-25T01:40:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.289+0000 7f9891f6a740 1 -- start start 2024-04-25T01:40:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.289+0000 7f9891f6a740 1 Processor -- start 2024-04-25T01:40:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f9891f6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bb580 0x556e967c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f9891f6a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967bbea0 0x556e967c3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f9891f6a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967c6570 0x556e967c37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f9891f6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556e96720670 con 0x556e967bb580 2024-04-25T01:40:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f9891f6a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x556e966dc050 con 0x556e967c6570 2024-04-25T01:40:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f9891f6a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x556e967c9120 con 0x556e967bbea0 2024-04-25T01:40:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f9891f6a740 10 client.0 _ll_register_callbacks cb 0x556e96433be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f988cff9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bb580 0x556e967c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f988cff9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bb580 0x556e967c5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34570/0 (socket says 172.21.15.28:34570) 2024-04-25T01:40:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f988cff9640 1 -- 172.21.15.28:0/4116615623 learned_addr learned my addr 172.21.15.28:0/4116615623 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f988cff9640 1 -- 172.21.15.28:0/4116615623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967bbea0 msgr2=0x556e967c3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f9887fff640 1 --2- 172.21.15.28:0/4116615623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967bbea0 0x556e967c3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f988d7fa640 1 --2- 172.21.15.28:0/4116615623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967c6570 0x556e967c37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f988cff9640 1 --2- 172.21.15.28:0/4116615623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967bbea0 0x556e967c3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f988cff9640 1 -- 172.21.15.28:0/4116615623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967c6570 msgr2=0x556e967c37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f988cff9640 1 --2- 172.21.15.28:0/4116615623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967c6570 0x556e967c37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f988cff9640 1 -- 172.21.15.28:0/4116615623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e966addc0 con 0x556e967bb580 2024-04-25T01:40:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f9887fff640 1 --2- 172.21.15.28:0/4116615623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967bbea0 0x556e967c3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f988d7fa640 1 --2- 172.21.15.28:0/4116615623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967c6570 0x556e967c37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f988cff9640 1 --2- 172.21.15.28:0/4116615623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bb580 0x556e967c5f10 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f98780bc7c0 tx=0x7f9878009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f656142dc1392e1d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.290+0000 7f9867fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.291+0000 7f9867fff640 1 -- 172.21.15.28:0/4116615623 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9878049070 con 0x556e967bb580 2024-04-25T01:40:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.291+0000 7f9867fff640 1 -- 172.21.15.28:0/4116615623 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f98780ed3c0 con 0x556e967bb580 2024-04-25T01:40:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.291+0000 7f9891f6a740 1 -- 172.21.15.28:0/4116615623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98780bc610 con 0x556e967bb580 2024-04-25T01:40:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.291+0000 7f9867fff640 1 -- 172.21.15.28:0/4116615623 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f98780bbf10 con 0x556e967bb580 2024-04-25T01:40:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.291+0000 7f9891f6a740 10 client.18159 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:21.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.291+0000 7f9891f6a740 1 -- 172.21.15.28:0/4116615623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556e964343f0 con 0x556e967bb580 2024-04-25T01:40:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.291+0000 7f9867fff640 1 -- 172.21.15.28:0/4116615623 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98780bc610 con 0x556e967bb580 2024-04-25T01:40:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.291+0000 7f9867fff640 1 -- 172.21.15.28:0/4116615623 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f98780f3030 con 0x556e967bb580 2024-04-25T01:40:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.291+0000 7f9891f6a740 10 client.18159 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.291+0000 7f9891f6a740 10 client.18159 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:40:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.291+0000 7f9891f6a740 20 client.18159 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.291+0000 7f9891f6a740 1 --2- 172.21.15.28:0/4116615623 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x556e967d6150 0x556e967d8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.291+0000 7f9891f6a740 4 client.18159 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.291+0000 7f9891f6a740 1 -- 172.21.15.28:0/4116615623 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e9672dbb0 con 0x556e967d6150 2024-04-25T01:40:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.291+0000 7f9887fff640 1 --2- 172.21.15.28:0/4116615623 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x556e967d6150 0x556e967d8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.292+0000 7f9867fff640 10 client.18159 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:40:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.292+0000 7f9887fff640 1 --2- 172.21.15.28:0/4116615623 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x556e967d6150 0x556e967d8560 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2829e3a38b992442 server_cookie=d61538c4be32a408 in_seq=0 out_seq=0 2024-04-25T01:40:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.293+0000 7f9867fff640 1 -- 172.21.15.28:0/4116615623 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556e9672dbb0 con 0x556e967d6150 2024-04-25T01:40:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.293+0000 7f9867fff640 10 client.18159 handle_command_reply: tid=0 2024-04-25T01:40:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.293+0000 7f9891f6a740 1 client.18159 shutdown 2024-04-25T01:40:21.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.293+0000 7f9891f6a740 1 -- 172.21.15.28:0/4116615623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bb580 msgr2=0x556e967c5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.293+0000 7f9891f6a740 1 --2- 172.21.15.28:0/4116615623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bb580 0x556e967c5f10 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f98780bc7c0 tx=0x7f9878009260 comp rx=0 tx=0).stop 2024-04-25T01:40:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.293+0000 7f9891f6a740 1 -- 172.21.15.28:0/4116615623 shutdown_connections 2024-04-25T01:40:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.293+0000 7f9891f6a740 1 --2- 172.21.15.28:0/4116615623 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x556e967d6150 0x556e967d8560 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.293+0000 7f9891f6a740 1 --2- 172.21.15.28:0/4116615623 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967c6570 0x556e967c37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.293+0000 7f9867fff640 0 client.18159 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:40:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.293+0000 7f9891f6a740 1 --2- 172.21.15.28:0/4116615623 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967bbea0 0x556e967c3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.293+0000 7f9891f6a740 1 --2- 172.21.15.28:0/4116615623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bb580 0x556e967c5f10 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.293+0000 7f9891f6a740 1 -- 172.21.15.28:0/4116615623 >> 172.21.15.28:0/4116615623 conn(0x556e967b78a0 msgr2=0x556e967b6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.293+0000 7f9891f6a740 1 -- 172.21.15.28:0/4116615623 shutdown_connections 2024-04-25T01:40:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.293+0000 7f9891f6a740 1 -- 172.21.15.28:0/4116615623 wait complete. 2024-04-25T01:40:21.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.293+0000 7f9891f6a740 20 client.18159 trim_cache size 0 max 16384 2024-04-25T01:40:21.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.306+0000 7f9891f6a740 1 Processor -- start 2024-04-25T01:40:21.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.306+0000 7f9891f6a740 1 -- start start 2024-04-25T01:40:21.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f9891f6a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967bb580 0x556e967d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f9891f6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 0x556e967b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f9891f6a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967c6570 0x556e967c5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f9891f6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98881aece0 con 0x556e967bbea0 2024-04-25T01:40:21.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f9891f6a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f98881b0820 con 0x556e967c6570 2024-04-25T01:40:21.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f9891f6a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f98881af3b0 con 0x556e967bb580 2024-04-25T01:40:21.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f9887fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 0x556e967b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f9887fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 0x556e967b6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34578/0 (socket says 172.21.15.28:34578) 2024-04-25T01:40:21.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f9887fff640 1 -- 172.21.15.28:0/4038955423 learned_addr learned my addr 172.21.15.28:0/4038955423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f988d7fa640 1 --2- 172.21.15.28:0/4038955423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967c6570 0x556e967c5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f988cff9640 1 --2- 172.21.15.28:0/4038955423 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967bb580 0x556e967d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f9887fff640 1 -- 172.21.15.28:0/4038955423 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967bb580 msgr2=0x556e967d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f9887fff640 1 --2- 172.21.15.28:0/4038955423 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967bb580 0x556e967d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f9887fff640 1 -- 172.21.15.28:0/4038955423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967c6570 msgr2=0x556e967c5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f9887fff640 1 --2- 172.21.15.28:0/4038955423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967c6570 0x556e967c5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f9887fff640 1 -- 172.21.15.28:0/4038955423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e964fe7d0 con 0x556e967bbea0 2024-04-25T01:40:21.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f9887fff640 1 --2- 172.21.15.28:0/4038955423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 0x556e967b6b40 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x556e967b8a30 tx=0x7f987c00a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e93a3de5b8861635 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f986e7fc640 1 -- 172.21.15.28:0/4038955423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987c01f030 con 0x556e967bbea0 2024-04-25T01:40:21.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f986e7fc640 1 -- 172.21.15.28:0/4038955423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f987c01b880 con 0x556e967bbea0 2024-04-25T01:40:21.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.307+0000 7f986e7fc640 1 -- 172.21.15.28:0/4038955423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987c004a90 con 0x556e967bbea0 2024-04-25T01:40:21.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.308+0000 7f9891f6a740 1 -- 172.21.15.28:0/4038955423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 msgr2=0x556e967b6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.308+0000 7f9891f6a740 1 --2- 172.21.15.28:0/4038955423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 0x556e967b6b40 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x556e967b8a30 tx=0x7f987c00a610 comp rx=0 tx=0).stop 2024-04-25T01:40:21.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.308+0000 7f9891f6a740 1 -- 172.21.15.28:0/4038955423 shutdown_connections 2024-04-25T01:40:21.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.308+0000 7f9891f6a740 1 --2- 172.21.15.28:0/4038955423 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967c6570 0x556e967c5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.308+0000 7f9891f6a740 1 --2- 172.21.15.28:0/4038955423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 0x556e967b6b40 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.308+0000 7f9891f6a740 1 --2- 172.21.15.28:0/4038955423 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967bb580 0x556e967d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.308+0000 7f9891f6a740 1 -- 172.21.15.28:0/4038955423 >> 172.21.15.28:0/4038955423 conn(0x556e967b78a0 msgr2=0x556e967c2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.308+0000 7f9891f6a740 1 -- 172.21.15.28:0/4038955423 shutdown_connections 2024-04-25T01:40:21.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.308+0000 7f9891f6a740 1 -- 172.21.15.28:0/4038955423 wait complete. 2024-04-25T01:40:21.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.308+0000 7f9891f6a740 1 -- start start 2024-04-25T01:40:21.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.308+0000 7f9891f6a740 1 Processor -- start 2024-04-25T01:40:21.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f9891f6a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967bb580 0x556e967ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f9891f6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 0x556e967ce830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f9891f6a740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967c6570 0x556e967c9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f9891f6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98881b0150 con 0x556e967bbea0 2024-04-25T01:40:21.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f9891f6a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f98881afa80 con 0x556e967bb580 2024-04-25T01:40:21.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f9891f6a740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f98881acce0 con 0x556e967c6570 2024-04-25T01:40:21.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f9891f6a740 10 client.0 _ll_register_callbacks cb 0x7f9878005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:21.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f9887fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 0x556e967ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f9887fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 0x556e967ce830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34588/0 (socket says 172.21.15.28:34588) 2024-04-25T01:40:21.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f9887fff640 1 -- 172.21.15.28:0/1513697727 learned_addr learned my addr 172.21.15.28:0/1513697727 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f988d7fa640 1 --2- 172.21.15.28:0/1513697727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967c6570 0x556e967c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f988cff9640 1 --2- 172.21.15.28:0/1513697727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967bb580 0x556e967ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f9887fff640 1 -- 172.21.15.28:0/1513697727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967c6570 msgr2=0x556e967c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f9887fff640 1 --2- 172.21.15.28:0/1513697727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967c6570 0x556e967c9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f9887fff640 1 -- 172.21.15.28:0/1513697727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967bb580 msgr2=0x556e967ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f9887fff640 1 --2- 172.21.15.28:0/1513697727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967bb580 0x556e967ce2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f9887fff640 1 -- 172.21.15.28:0/1513697727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e967ca040 con 0x556e967bbea0 2024-04-25T01:40:21.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f9887fff640 1 --2- 172.21.15.28:0/1513697727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 0x556e967ce830 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f987c00abb0 tx=0x7f987c01b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68cdb87f10e20575 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f9867fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.309+0000 7f9867fff640 1 -- 172.21.15.28:0/1513697727 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987c0054f0 con 0x556e967bbea0 2024-04-25T01:40:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.310+0000 7f9867fff640 1 -- 172.21.15.28:0/1513697727 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f987c004730 con 0x556e967bbea0 2024-04-25T01:40:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.310+0000 7f9891f6a740 1 -- 172.21.15.28:0/1513697727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f987c01b880 con 0x556e967bbea0 2024-04-25T01:40:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.310+0000 7f9867fff640 1 -- 172.21.15.28:0/1513697727 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f987c004930 con 0x556e967bbea0 2024-04-25T01:40:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.310+0000 7f9891f6a740 10 client.18165 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.310+0000 7f9891f6a740 1 -- 172.21.15.28:0/1513697727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556e9683c1f0 con 0x556e967bbea0 2024-04-25T01:40:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.310+0000 7f9867fff640 1 -- 172.21.15.28:0/1513697727 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f987c01b880 con 0x556e967bbea0 2024-04-25T01:40:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.310+0000 7f9867fff640 1 -- 172.21.15.28:0/1513697727 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f987c024440 con 0x556e967bbea0 2024-04-25T01:40:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.310+0000 7f9891f6a740 10 client.18165 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.310+0000 7f9891f6a740 10 client.18165 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:40:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.310+0000 7f9891f6a740 20 client.18165 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.310+0000 7f9891f6a740 1 --2- 172.21.15.28:0/1513697727 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x556e9683c8e0 0x556e9683ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.310+0000 7f9891f6a740 4 client.18165 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.310+0000 7f9891f6a740 1 -- 172.21.15.28:0/1513697727 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556e96727f00 con 0x556e9683c8e0 2024-04-25T01:40:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.310+0000 7f988cff9640 1 --2- 172.21.15.28:0/1513697727 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x556e9683c8e0 0x556e9683ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.311+0000 7f9867fff640 10 client.18165 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:40:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.311+0000 7f988cff9640 1 --2- 172.21.15.28:0/1513697727 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x556e9683c8e0 0x556e9683ed00 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9f7c4480b0fc3f6d server_cookie=744df93e05168577 in_seq=0 out_seq=0 2024-04-25T01:40:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.311+0000 7f9867fff640 1 -- 172.21.15.28:0/1513697727 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556e96727f00 con 0x556e9683c8e0 2024-04-25T01:40:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.311+0000 7f9867fff640 10 client.18165 handle_command_reply: tid=0 2024-04-25T01:40:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.311+0000 7f9891f6a740 1 client.18165 shutdown 2024-04-25T01:40:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.311+0000 7f9891f6a740 1 -- 172.21.15.28:0/1513697727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 msgr2=0x556e967ce830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.311+0000 7f9891f6a740 1 --2- 172.21.15.28:0/1513697727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 0x556e967ce830 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f987c00abb0 tx=0x7f987c01b650 comp rx=0 tx=0).stop 2024-04-25T01:40:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.311+0000 7f9891f6a740 1 -- 172.21.15.28:0/1513697727 shutdown_connections 2024-04-25T01:40:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.311+0000 7f9891f6a740 1 --2- 172.21.15.28:0/1513697727 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x556e9683c8e0 0x556e9683ed00 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.311+0000 7f9891f6a740 1 --2- 172.21.15.28:0/1513697727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x556e967c6570 0x556e967c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.311+0000 7f9891f6a740 1 --2- 172.21.15.28:0/1513697727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e967bbea0 0x556e967ce830 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.311+0000 7f9891f6a740 1 --2- 172.21.15.28:0/1513697727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556e967bb580 0x556e967ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.311+0000 7f9867fff640 0 client.18165 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:40:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.311+0000 7f9891f6a740 1 -- 172.21.15.28:0/1513697727 >> 172.21.15.28:0/1513697727 conn(0x556e967b78a0 msgr2=0x556e967be690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.311+0000 7f9891f6a740 1 -- 172.21.15.28:0/1513697727 shutdown_connections 2024-04-25T01:40:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.311+0000 7f9891f6a740 1 -- 172.21.15.28:0/1513697727 wait complete. 2024-04-25T01:40:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.311+0000 7f9891f6a740 20 client.18165 trim_cache size 0 max 16384 2024-04-25T01:40:21.302 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.312+0000 7f988f25d640 1 -- 172.21.15.28:0/2398838533 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f98700a2d00 msgr2=0x7f98700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.312+0000 7f988f25d640 1 --2- 172.21.15.28:0/2398838533 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f98700a2d00 0x7f98700a51c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9880005c40 tx=0x7f9880009280 comp rx=0 tx=0).stop 2024-04-25T01:40:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.312+0000 7f988f25d640 1 -- 172.21.15.28:0/2398838533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888101710 msgr2=0x7f9888198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.312+0000 7f988f25d640 1 --2- 172.21.15.28:0/2398838533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888101710 0x7f9888198090 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f9878009230 tx=0x7f9878004270 comp rx=0 tx=0).stop 2024-04-25T01:40:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.312+0000 7f988f25d640 1 -- 172.21.15.28:0/2398838533 shutdown_connections 2024-04-25T01:40:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.312+0000 7f988f25d640 1 --2- 172.21.15.28:0/2398838533 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f98700a2d00 0x7f98700a51c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.312+0000 7f988f25d640 1 --2- 172.21.15.28:0/2398838533 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9888107720 0x7f988819c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.312+0000 7f988f25d640 1 --2- 172.21.15.28:0/2398838533 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f98881020d0 0x7f98881985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.312+0000 7f988f25d640 1 --2- 172.21.15.28:0/2398838533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9888101710 0x7f9888198090 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.312+0000 7f988f25d640 1 -- 172.21.15.28:0/2398838533 >> 172.21.15.28:0/2398838533 conn(0x7f9888078b90 msgr2=0x7f98880ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.312+0000 7f988f25d640 1 -- 172.21.15.28:0/2398838533 shutdown_connections 2024-04-25T01:40:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.312+0000 7f988f25d640 1 -- 172.21.15.28:0/2398838533 wait complete. 2024-04-25T01:40:21.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:40:21.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.426+0000 7fa13838b640 1 -- 172.21.15.28:0/4131650421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130107730 msgr2=0x7fa130107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.426+0000 7fa13838b640 1 --2- 172.21.15.28:0/4131650421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130107730 0x7fa130107b10 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fa12c009930 tx=0x7fa12c02a420 comp rx=0 tx=0).stop 2024-04-25T01:40:21.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.426+0000 7fa13838b640 1 -- 172.21.15.28:0/4131650421 shutdown_connections 2024-04-25T01:40:21.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.426+0000 7fa13838b640 1 --2- 172.21.15.28:0/4131650421 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa13006a780 0x7fa13010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.426+0000 7fa13838b640 1 --2- 172.21.15.28:0/4131650421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa130069dc0 0x7fa13006a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.426+0000 7fa13838b640 1 --2- 172.21.15.28:0/4131650421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130107730 0x7fa130107b10 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.426+0000 7fa13838b640 1 -- 172.21.15.28:0/4131650421 >> 172.21.15.28:0/4131650421 conn(0x7fa1300776a0 msgr2=0x7fa130077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.426+0000 7fa13838b640 1 -- 172.21.15.28:0/4131650421 shutdown_connections 2024-04-25T01:40:21.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.426+0000 7fa13838b640 1 -- 172.21.15.28:0/4131650421 wait complete. 2024-04-25T01:40:21.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa13838b640 1 Processor -- start 2024-04-25T01:40:21.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa13838b640 1 -- start start 2024-04-25T01:40:21.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa13838b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130069dc0 0x7fa1301980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa13838b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa13006a780 0x7fa130198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa13838b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa130107730 0x7fa13019c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa13838b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa130111b00 con 0x7fa130069dc0 2024-04-25T01:40:21.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa13838b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa130111980 con 0x7fa130107730 2024-04-25T01:40:21.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa13838b640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa130111c80 con 0x7fa13006a780 2024-04-25T01:40:21.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa136127640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130069dc0 0x7fa1301980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa136127640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130069dc0 0x7fa1301980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34598/0 (socket says 172.21.15.28:34598) 2024-04-25T01:40:21.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa136127640 1 -- 172.21.15.28:0/3150926571 learned_addr learned my addr 172.21.15.28:0/3150926571 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa135926640 1 --2- 172.21.15.28:0/3150926571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa13006a780 0x7fa130198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa136928640 1 --2- 172.21.15.28:0/3150926571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa130107730 0x7fa13019c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa136127640 1 -- 172.21.15.28:0/3150926571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa13006a780 msgr2=0x7fa130198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa136127640 1 --2- 172.21.15.28:0/3150926571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa13006a780 0x7fa130198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa136127640 1 -- 172.21.15.28:0/3150926571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa130107730 msgr2=0x7fa13019c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa136127640 1 --2- 172.21.15.28:0/3150926571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa130107730 0x7fa13019c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa136127640 1 -- 172.21.15.28:0/3150926571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13019d060 con 0x7fa130069dc0 2024-04-25T01:40:21.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa135926640 1 --2- 172.21.15.28:0/3150926571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa13006a780 0x7fa130198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:21.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.427+0000 7fa136928640 1 --2- 172.21.15.28:0/3150926571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa130107730 0x7fa13019c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:21.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.428+0000 7fa136127640 1 --2- 172.21.15.28:0/3150926571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130069dc0 0x7fa1301980c0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fa12c009230 tx=0x7fa12c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.428+0000 7fa1277fe640 1 -- 172.21.15.28:0/3150926571 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c03f030 con 0x7fa130069dc0 2024-04-25T01:40:21.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.428+0000 7fa1277fe640 1 -- 172.21.15.28:0/3150926571 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa12c003ad0 con 0x7fa130069dc0 2024-04-25T01:40:21.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.428+0000 7fa13838b640 1 -- 172.21.15.28:0/3150926571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa13019d2f0 con 0x7fa130069dc0 2024-04-25T01:40:21.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.428+0000 7fa1277fe640 1 -- 172.21.15.28:0/3150926571 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c035070 con 0x7fa130069dc0 2024-04-25T01:40:21.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.428+0000 7fa13838b640 1 -- 172.21.15.28:0/3150926571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1301a4be0 con 0x7fa130069dc0 2024-04-25T01:40:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.430+0000 7fa1277fe640 1 -- 172.21.15.28:0/3150926571 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa12c042030 con 0x7fa130069dc0 2024-04-25T01:40:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.431+0000 7fa13b170740 1 Processor -- start 2024-04-25T01:40:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.431+0000 7fa13b170740 1 -- start start 2024-04-25T01:40:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.431+0000 7fa13b170740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35db580 0x559ed35db960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.431+0000 7fa13b170740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35dbea0 0x559ed35e6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.431+0000 7fa13b170740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35e6570 0x559ed35e8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.431+0000 7fa13b170740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559ed3540670 con 0x559ed35dbea0 2024-04-25T01:40:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.431+0000 7fa13b170740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559ed34e8970 con 0x559ed35db580 2024-04-25T01:40:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.431+0000 7fa13b170740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x559ed353c630 con 0x559ed35e6570 2024-04-25T01:40:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.431+0000 7fa136127640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35dbea0 0x559ed35e6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.431+0000 7fa1277fe640 1 --2- 172.21.15.28:0/3150926571 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa11c0a2d80 0x7fa11c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.431+0000 7fa136127640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35dbea0 0x559ed35e6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34612/0 (socket says 172.21.15.28:34612) 2024-04-25T01:40:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.431+0000 7fa136127640 1 -- 172.21.15.28:0/3321905487 learned_addr learned my addr 172.21.15.28:0/3321905487 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.431+0000 7fa135926640 1 --2- 172.21.15.28:0/3321905487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35e6570 0x559ed35e8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.431+0000 7fa136928640 1 --2- 172.21.15.28:0/3321905487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35db580 0x559ed35db960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.432+0000 7fa136928640 1 --2- 172.21.15.28:0/3150926571 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa11c0a2d80 0x7fa11c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.432+0000 7fa1277fe640 1 -- 172.21.15.28:0/3150926571 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fa12c0efed0 con 0x7fa130069dc0 2024-04-25T01:40:21.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.432+0000 7fa136127640 1 -- 172.21.15.28:0/3321905487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35e6570 msgr2=0x559ed35e8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.432+0000 7fa136127640 1 --2- 172.21.15.28:0/3321905487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35e6570 0x559ed35e8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.432+0000 7fa136127640 1 -- 172.21.15.28:0/3321905487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35db580 msgr2=0x559ed35db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.432+0000 7fa136127640 1 --2- 172.21.15.28:0/3321905487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35db580 0x559ed35db960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.432+0000 7fa136127640 1 -- 172.21.15.28:0/3321905487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ed350c400 con 0x559ed35dbea0 2024-04-25T01:40:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.432+0000 7fa136127640 1 --2- 172.21.15.28:0/3321905487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35dbea0 0x559ed35e6030 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0bbd00 tx=0x7fa12c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba48f1af2cdf5e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.432+0000 7fa136928640 1 --2- 172.21.15.28:0/3150926571 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa11c0a2d80 0x7fa11c0a5240 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa128009370 tx=0x7fa128009240 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.432+0000 7fa124bfe640 1 -- 172.21.15.28:0/3321905487 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c049070 con 0x559ed35dbea0 2024-04-25T01:40:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.432+0000 7fa124bfe640 1 -- 172.21.15.28:0/3321905487 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa12c03b460 con 0x559ed35dbea0 2024-04-25T01:40:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.432+0000 7fa124bfe640 1 -- 172.21.15.28:0/3321905487 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c0b8670 con 0x559ed35dbea0 2024-04-25T01:40:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.433+0000 7fa13b170740 1 -- 172.21.15.28:0/3321905487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35dbea0 msgr2=0x559ed35e6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.433+0000 7fa13b170740 1 --2- 172.21.15.28:0/3321905487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35dbea0 0x559ed35e6030 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0bbd00 tx=0x7fa12c009260 comp rx=0 tx=0).stop 2024-04-25T01:40:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.433+0000 7fa13b170740 1 -- 172.21.15.28:0/3321905487 shutdown_connections 2024-04-25T01:40:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.433+0000 7fa13b170740 1 --2- 172.21.15.28:0/3321905487 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35e6570 0x559ed35e8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.433+0000 7fa13b170740 1 --2- 172.21.15.28:0/3321905487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35dbea0 0x559ed35e6030 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.433+0000 7fa13b170740 1 --2- 172.21.15.28:0/3321905487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35db580 0x559ed35db960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.433+0000 7fa13b170740 1 -- 172.21.15.28:0/3321905487 >> 172.21.15.28:0/3321905487 conn(0x559ed35d78a0 msgr2=0x559ed35d7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.433+0000 7fa13b170740 1 -- 172.21.15.28:0/3321905487 shutdown_connections 2024-04-25T01:40:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.433+0000 7fa13b170740 1 -- 172.21.15.28:0/3321905487 wait complete. 2024-04-25T01:40:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.433+0000 7fa13b170740 1 -- start start 2024-04-25T01:40:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.433+0000 7fa13b170740 1 Processor -- start 2024-04-25T01:40:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa13b170740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35db580 0x559ed35e5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa13b170740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35dbea0 0x559ed35e31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa13b170740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35e6570 0x559ed35e3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa13b170740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559ed35e9120 con 0x559ed35e6570 2024-04-25T01:40:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa13b170740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559ed34fc050 con 0x559ed35dbea0 2024-04-25T01:40:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa13b170740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x559ed35e92a0 con 0x559ed35db580 2024-04-25T01:40:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa13b170740 10 client.0 _ll_register_callbacks cb 0x559ed3253be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa136928640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35e6570 0x559ed35e3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa136928640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35e6570 0x559ed35e3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34620/0 (socket says 172.21.15.28:34620) 2024-04-25T01:40:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa136928640 1 -- 172.21.15.28:0/1013620727 learned_addr learned my addr 172.21.15.28:0/1013620727 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa135926640 1 --2- 172.21.15.28:0/1013620727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35dbea0 0x559ed35e31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa136127640 1 --2- 172.21.15.28:0/1013620727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35db580 0x559ed35e5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa136928640 1 -- 172.21.15.28:0/1013620727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35db580 msgr2=0x559ed35e5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa136928640 1 --2- 172.21.15.28:0/1013620727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35db580 0x559ed35e5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa136928640 1 -- 172.21.15.28:0/1013620727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35dbea0 msgr2=0x559ed35e31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa136928640 1 --2- 172.21.15.28:0/1013620727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35dbea0 0x559ed35e31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa136928640 1 -- 172.21.15.28:0/1013620727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ed34cddc0 con 0x559ed35e6570 2024-04-25T01:40:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa136127640 1 --2- 172.21.15.28:0/1013620727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35db580 0x559ed35e5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa135926640 1 --2- 172.21.15.28:0/1013620727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35dbea0 0x559ed35e31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa136928640 1 --2- 172.21.15.28:0/1013620727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35e6570 0x559ed35e3720 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fa128009df0 tx=0x7fa128004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b2d00b5a297391 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa119ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa119ffb640 1 -- 172.21.15.28:0/1013620727 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa128004b20 con 0x559ed35e6570 2024-04-25T01:40:21.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa119ffb640 1 -- 172.21.15.28:0/1013620727 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fa128004cc0 con 0x559ed35e6570 2024-04-25T01:40:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.434+0000 7fa13b170740 1 -- 172.21.15.28:0/1013620727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559ed354dbb0 con 0x559ed35e6570 2024-04-25T01:40:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.435+0000 7fa119ffb640 1 -- 172.21.15.28:0/1013620727 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa128003970 con 0x559ed35e6570 2024-04-25T01:40:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.435+0000 7fa13b170740 10 client.18177 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.435+0000 7fa13b170740 1 -- 172.21.15.28:0/1013620727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559ed32543f0 con 0x559ed35e6570 2024-04-25T01:40:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.435+0000 7fa119ffb640 1 -- 172.21.15.28:0/1013620727 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559ed354dbb0 con 0x559ed35e6570 2024-04-25T01:40:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.435+0000 7fa119ffb640 1 -- 172.21.15.28:0/1013620727 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa12801d040 con 0x559ed35e6570 2024-04-25T01:40:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.435+0000 7fa13b170740 10 client.18177 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.435+0000 7fa13b170740 10 client.18177 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:40:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.435+0000 7fa13b170740 20 client.18177 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.435+0000 7fa13b170740 1 --2- 172.21.15.28:0/1013620727 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x559ed35f62c0 0x559ed35f8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.435+0000 7fa13b170740 4 client.18177 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.435+0000 7fa13b170740 1 -- 172.21.15.28:0/1013620727 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ed3547f00 con 0x559ed35f62c0 2024-04-25T01:40:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.435+0000 7fa136127640 1 --2- 172.21.15.28:0/1013620727 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x559ed35f62c0 0x559ed35f8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.436+0000 7fa136127640 1 --2- 172.21.15.28:0/1013620727 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x559ed35f62c0 0x559ed35f8700 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6863b5c1b2d641c1 server_cookie=fb31a4300361d9ac in_seq=0 out_seq=0 2024-04-25T01:40:21.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.436+0000 7fa119ffb640 10 client.18177 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:40:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.436+0000 7fa119ffb640 1 -- 172.21.15.28:0/1013620727 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559ed3547f00 con 0x559ed35f62c0 2024-04-25T01:40:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.436+0000 7fa119ffb640 10 client.18177 handle_command_reply: tid=0 2024-04-25T01:40:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.436+0000 7fa13b170740 1 client.18177 shutdown 2024-04-25T01:40:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.437+0000 7fa13b170740 1 -- 172.21.15.28:0/1013620727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35e6570 msgr2=0x559ed35e3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.437+0000 7fa13b170740 1 --2- 172.21.15.28:0/1013620727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35e6570 0x559ed35e3720 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fa128009df0 tx=0x7fa128004a40 comp rx=0 tx=0).stop 2024-04-25T01:40:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.437+0000 7fa13b170740 1 -- 172.21.15.28:0/1013620727 shutdown_connections 2024-04-25T01:40:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.437+0000 7fa13b170740 1 --2- 172.21.15.28:0/1013620727 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x559ed35f62c0 0x559ed35f8700 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.437+0000 7fa13b170740 1 --2- 172.21.15.28:0/1013620727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35e6570 0x559ed35e3720 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.437+0000 7fa13b170740 1 --2- 172.21.15.28:0/1013620727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35dbea0 0x559ed35e31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.437+0000 7fa119ffb640 0 client.18177 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:40:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.437+0000 7fa13b170740 1 --2- 172.21.15.28:0/1013620727 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35db580 0x559ed35e5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.437+0000 7fa13b170740 1 -- 172.21.15.28:0/1013620727 >> 172.21.15.28:0/1013620727 conn(0x559ed35d78a0 msgr2=0x559ed35d6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.437+0000 7fa13b170740 1 -- 172.21.15.28:0/1013620727 shutdown_connections 2024-04-25T01:40:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.437+0000 7fa13b170740 1 -- 172.21.15.28:0/1013620727 wait complete. 2024-04-25T01:40:21.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.437+0000 7fa13b170740 20 client.18177 trim_cache size 0 max 16384 2024-04-25T01:40:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.449+0000 7fa13b170740 1 Processor -- start 2024-04-25T01:40:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.449+0000 7fa13b170740 1 -- start start 2024-04-25T01:40:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.449+0000 7fa13b170740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35db580 0x559ed35f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.449+0000 7fa13b170740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35dbea0 0x559ed35fdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.449+0000 7fa13b170740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35e6570 0x559ed35fe510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.449+0000 7fa13b170740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1301aebf0 con 0x559ed35db580 2024-04-25T01:40:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.449+0000 7fa13b170740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa1301b0730 con 0x559ed35e6570 2024-04-25T01:40:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.449+0000 7fa13b170740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa1301af2c0 con 0x559ed35dbea0 2024-04-25T01:40:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.449+0000 7fa136127640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35db580 0x559ed35f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.449+0000 7fa136127640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35db580 0x559ed35f1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34626/0 (socket says 172.21.15.28:34626) 2024-04-25T01:40:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.449+0000 7fa136127640 1 -- 172.21.15.28:0/3677779272 learned_addr learned my addr 172.21.15.28:0/3677779272 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.449+0000 7fa136928640 1 --2- 172.21.15.28:0/3677779272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35e6570 0x559ed35fe510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.450+0000 7fa135926640 1 --2- 172.21.15.28:0/3677779272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35dbea0 0x559ed35fdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.450+0000 7fa136127640 1 -- 172.21.15.28:0/3677779272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35dbea0 msgr2=0x559ed35fdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.450+0000 7fa136127640 1 --2- 172.21.15.28:0/3677779272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35dbea0 0x559ed35fdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.450+0000 7fa136127640 1 -- 172.21.15.28:0/3677779272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35e6570 msgr2=0x559ed35fe510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.450+0000 7fa136127640 1 --2- 172.21.15.28:0/3677779272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35e6570 0x559ed35fe510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.450+0000 7fa136127640 1 -- 172.21.15.28:0/3677779272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ed35e2170 con 0x559ed35db580 2024-04-25T01:40:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.450+0000 7fa135926640 1 --2- 172.21.15.28:0/3677779272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35dbea0 0x559ed35fdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.450+0000 7fa136127640 1 --2- 172.21.15.28:0/3677779272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35db580 0x559ed35f1ec0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0b89d0 tx=0x7fa12c005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80224ff818d954aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.450+0000 7fa11bfff640 1 -- 172.21.15.28:0/3677779272 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c049040 con 0x559ed35db580 2024-04-25T01:40:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.450+0000 7fa11bfff640 1 -- 172.21.15.28:0/3677779272 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa12c0bc000 con 0x559ed35db580 2024-04-25T01:40:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.450+0000 7fa11bfff640 1 -- 172.21.15.28:0/3677779272 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c0ed4b0 con 0x559ed35db580 2024-04-25T01:40:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.450+0000 7fa13b170740 1 -- 172.21.15.28:0/3677779272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35db580 msgr2=0x559ed35f1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.450+0000 7fa13b170740 1 --2- 172.21.15.28:0/3677779272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35db580 0x559ed35f1ec0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0b89d0 tx=0x7fa12c005b50 comp rx=0 tx=0).stop 2024-04-25T01:40:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.450+0000 7fa13b170740 1 -- 172.21.15.28:0/3677779272 shutdown_connections 2024-04-25T01:40:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.450+0000 7fa13b170740 1 --2- 172.21.15.28:0/3677779272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35e6570 0x559ed35fe510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.450+0000 7fa13b170740 1 --2- 172.21.15.28:0/3677779272 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35dbea0 0x559ed35fdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.450+0000 7fa13b170740 1 --2- 172.21.15.28:0/3677779272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35db580 0x559ed35f1ec0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.450+0000 7fa13b170740 1 -- 172.21.15.28:0/3677779272 >> 172.21.15.28:0/3677779272 conn(0x559ed35d78a0 msgr2=0x559ed35d6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa13b170740 1 -- 172.21.15.28:0/3677779272 shutdown_connections 2024-04-25T01:40:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa13b170740 1 -- 172.21.15.28:0/3677779272 wait complete. 2024-04-25T01:40:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa13b170740 1 -- start start 2024-04-25T01:40:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa13b170740 1 Processor -- start 2024-04-25T01:40:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa13b170740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35db580 0x559ed35ebff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa13b170740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35dbea0 0x559ed35ec530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa13b170740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35e6570 0x559ed35eca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa13b170740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1301b0060 con 0x559ed35db580 2024-04-25T01:40:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa13b170740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa1301af990 con 0x559ed35e6570 2024-04-25T01:40:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa13b170740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa1301acbf0 con 0x559ed35dbea0 2024-04-25T01:40:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa13b170740 10 client.0 _ll_register_callbacks cb 0x7fa128009d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa136127640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35db580 0x559ed35ebff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa136127640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35db580 0x559ed35ebff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34630/0 (socket says 172.21.15.28:34630) 2024-04-25T01:40:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa136127640 1 -- 172.21.15.28:0/3730033439 learned_addr learned my addr 172.21.15.28:0/3730033439 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa136928640 1 --2- 172.21.15.28:0/3730033439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35e6570 0x559ed35eca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa135926640 1 --2- 172.21.15.28:0/3730033439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35dbea0 0x559ed35ec530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa136127640 1 -- 172.21.15.28:0/3730033439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35dbea0 msgr2=0x559ed35ec530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa136127640 1 --2- 172.21.15.28:0/3730033439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35dbea0 0x559ed35ec530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa136127640 1 -- 172.21.15.28:0/3730033439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35e6570 msgr2=0x559ed35eca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.451+0000 7fa136127640 1 --2- 172.21.15.28:0/3730033439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35e6570 0x559ed35eca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.452+0000 7fa136127640 1 -- 172.21.15.28:0/3730033439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ed331e7d0 con 0x559ed35db580 2024-04-25T01:40:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.452+0000 7fa135926640 1 --2- 172.21.15.28:0/3730033439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35dbea0 0x559ed35ec530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.452+0000 7fa136127640 1 --2- 172.21.15.28:0/3730033439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35db580 0x559ed35ebff0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fa12c000c00 tx=0x7fa12c0b8320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d10824cf9a1269 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.452+0000 7fa119ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.452+0000 7fa119ffb640 1 -- 172.21.15.28:0/3730033439 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c049040 con 0x559ed35db580 2024-04-25T01:40:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.452+0000 7fa119ffb640 1 -- 172.21.15.28:0/3730033439 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fa12c03bd40 con 0x559ed35db580 2024-04-25T01:40:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.452+0000 7fa13b170740 1 -- 172.21.15.28:0/3730033439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa12c0bc000 con 0x559ed35db580 2024-04-25T01:40:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.452+0000 7fa119ffb640 1 -- 172.21.15.28:0/3730033439 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa12c0b8670 con 0x559ed35db580 2024-04-25T01:40:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.452+0000 7fa13b170740 10 client.18183 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.452+0000 7fa13b170740 1 -- 172.21.15.28:0/3730033439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559ed365cad0 con 0x559ed35db580 2024-04-25T01:40:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.452+0000 7fa119ffb640 1 -- 172.21.15.28:0/3730033439 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa12c0bc000 con 0x559ed35db580 2024-04-25T01:40:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.452+0000 7fa119ffb640 1 -- 172.21.15.28:0/3730033439 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa12c0c47e0 con 0x559ed35db580 2024-04-25T01:40:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.452+0000 7fa13b170740 10 client.18183 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.452+0000 7fa13b170740 10 client.18183 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:40:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.452+0000 7fa13b170740 20 client.18183 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.452+0000 7fa13b170740 1 --2- 172.21.15.28:0/3730033439 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x559ed365ceb0 0x559ed365f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.452+0000 7fa13b170740 4 client.18183 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.452+0000 7fa13b170740 1 -- 172.21.15.28:0/3730033439 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559ed35d30e0 con 0x559ed365ceb0 2024-04-25T01:40:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.453+0000 7fa135926640 1 --2- 172.21.15.28:0/3730033439 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x559ed365ceb0 0x559ed365f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.453+0000 7fa135926640 1 --2- 172.21.15.28:0/3730033439 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x559ed365ceb0 0x559ed365f2a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3901428b9f6b9e3e server_cookie=fcd4f6120f7e6e54 in_seq=0 out_seq=0 2024-04-25T01:40:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.453+0000 7fa119ffb640 10 client.18183 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:40:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.453+0000 7fa119ffb640 1 -- 172.21.15.28:0/3730033439 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559ed35d30e0 con 0x559ed365ceb0 2024-04-25T01:40:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.453+0000 7fa119ffb640 10 client.18183 handle_command_reply: tid=0 2024-04-25T01:40:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.453+0000 7fa13b170740 1 client.18183 shutdown 2024-04-25T01:40:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.453+0000 7fa13b170740 1 -- 172.21.15.28:0/3730033439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35db580 msgr2=0x559ed35ebff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.453+0000 7fa13b170740 1 --2- 172.21.15.28:0/3730033439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35db580 0x559ed35ebff0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fa12c000c00 tx=0x7fa12c0b8320 comp rx=0 tx=0).stop 2024-04-25T01:40:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13b170740 1 -- 172.21.15.28:0/3730033439 shutdown_connections 2024-04-25T01:40:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13b170740 1 --2- 172.21.15.28:0/3730033439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559ed35e6570 0x559ed35eca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13b170740 1 --2- 172.21.15.28:0/3730033439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x559ed35dbea0 0x559ed35ec530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13b170740 1 --2- 172.21.15.28:0/3730033439 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x559ed365ceb0 0x559ed365f2a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13b170740 1 --2- 172.21.15.28:0/3730033439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559ed35db580 0x559ed35ebff0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa119ffb640 0 client.18183 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:40:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13b170740 1 -- 172.21.15.28:0/3730033439 >> 172.21.15.28:0/3730033439 conn(0x559ed35d78a0 msgr2=0x559ed35f0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13b170740 1 -- 172.21.15.28:0/3730033439 shutdown_connections 2024-04-25T01:40:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13b170740 1 -- 172.21.15.28:0/3730033439 wait complete. 2024-04-25T01:40:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13b170740 20 client.18183 trim_cache size 0 max 16384 2024-04-25T01:40:21.446 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13838b640 1 -- 172.21.15.28:0/3150926571 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa11c0a2d80 msgr2=0x7fa11c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13838b640 1 --2- 172.21.15.28:0/3150926571 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa11c0a2d80 0x7fa11c0a5240 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa128009370 tx=0x7fa128009240 comp rx=0 tx=0).stop 2024-04-25T01:40:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13838b640 1 -- 172.21.15.28:0/3150926571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130069dc0 msgr2=0x7fa1301980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13838b640 1 --2- 172.21.15.28:0/3150926571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130069dc0 0x7fa1301980c0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fa12c009230 tx=0x7fa12c004270 comp rx=0 tx=0).stop 2024-04-25T01:40:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13838b640 1 -- 172.21.15.28:0/3150926571 shutdown_connections 2024-04-25T01:40:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13838b640 1 --2- 172.21.15.28:0/3150926571 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa11c0a2d80 0x7fa11c0a5240 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13838b640 1 --2- 172.21.15.28:0/3150926571 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa130107730 0x7fa13019c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13838b640 1 --2- 172.21.15.28:0/3150926571 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa13006a780 0x7fa130198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13838b640 1 --2- 172.21.15.28:0/3150926571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa130069dc0 0x7fa1301980c0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13838b640 1 -- 172.21.15.28:0/3150926571 >> 172.21.15.28:0/3150926571 conn(0x7fa1300776a0 msgr2=0x7fa1300fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13838b640 1 -- 172.21.15.28:0/3150926571 shutdown_connections 2024-04-25T01:40:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.454+0000 7fa13838b640 1 -- 172.21.15.28:0/3150926571 wait complete. 2024-04-25T01:40:21.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:40:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.568+0000 7f14f7989640 1 -- 172.21.15.28:0/2654717424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f010bd00 msgr2=0x7f14f010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.568+0000 7f14f7989640 1 --2- 172.21.15.28:0/2654717424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f010bd00 0x7f14f010e1a0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009ca0 tx=0x7f14e802a9e0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.568+0000 7f14f7989640 1 -- 172.21.15.28:0/2654717424 shutdown_connections 2024-04-25T01:40:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.568+0000 7f14f7989640 1 --2- 172.21.15.28:0/2654717424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f010bd00 0x7f14f010e1a0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.568+0000 7f14f7989640 1 --2- 172.21.15.28:0/2654717424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14f01032e0 0x7f14f010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.568+0000 7f14f7989640 1 --2- 172.21.15.28:0/2654717424 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14f01029c0 0x7f14f0102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.569+0000 7f14f7989640 1 -- 172.21.15.28:0/2654717424 >> 172.21.15.28:0/2654717424 conn(0x7f14f0077740 msgr2=0x7f14f0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.569+0000 7f14f7989640 1 -- 172.21.15.28:0/2654717424 shutdown_connections 2024-04-25T01:40:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.569+0000 7f14f7989640 1 -- 172.21.15.28:0/2654717424 wait complete. 2024-04-25T01:40:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.569+0000 7f14f7989640 1 Processor -- start 2024-04-25T01:40:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.569+0000 7f14f7989640 1 -- start start 2024-04-25T01:40:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.569+0000 7f14f7989640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14f01029c0 0x7f14f0100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.569+0000 7f14f7989640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14f01032e0 0x7f14f0100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.569+0000 7f14f7989640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f010bd00 0x7f14f0101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.569+0000 7f14f7989640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14f0111b50 con 0x7f14f010bd00 2024-04-25T01:40:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.569+0000 7f14f7989640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f14f01119d0 con 0x7f14f01029c0 2024-04-25T01:40:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.569+0000 7f14f7989640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f14f0111cd0 con 0x7f14f01032e0 2024-04-25T01:40:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.569+0000 7f14f5f26640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f010bd00 0x7f14f0101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.570+0000 7f14f5f26640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f010bd00 0x7f14f0101090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34644/0 (socket says 172.21.15.28:34644) 2024-04-25T01:40:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.570+0000 7f14f5f26640 1 -- 172.21.15.28:0/4170918794 learned_addr learned my addr 172.21.15.28:0/4170918794 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.570+0000 7f14f4f24640 1 --2- 172.21.15.28:0/4170918794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14f01032e0 0x7f14f0100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.570+0000 7f14f5725640 1 --2- 172.21.15.28:0/4170918794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14f01029c0 0x7f14f0100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.570+0000 7f14f5f26640 1 -- 172.21.15.28:0/4170918794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14f01032e0 msgr2=0x7f14f0100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.570+0000 7f14f5f26640 1 --2- 172.21.15.28:0/4170918794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14f01032e0 0x7f14f0100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.570+0000 7f14f5f26640 1 -- 172.21.15.28:0/4170918794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14f01029c0 msgr2=0x7f14f0100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.570+0000 7f14f5f26640 1 --2- 172.21.15.28:0/4170918794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14f01029c0 0x7f14f0100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.570+0000 7f14f5f26640 1 -- 172.21.15.28:0/4170918794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14f01a6430 con 0x7f14f010bd00 2024-04-25T01:40:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.570+0000 7f14f5f26640 1 --2- 172.21.15.28:0/4170918794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f010bd00 0x7f14f0101090 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f14e8004af0 tx=0x7f14e8003f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.570+0000 7f14e67fc640 1 -- 172.21.15.28:0/4170918794 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e8047070 con 0x7f14f010bd00 2024-04-25T01:40:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.570+0000 7f14e67fc640 1 -- 172.21.15.28:0/4170918794 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14e802e870 con 0x7f14f010bd00 2024-04-25T01:40:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.570+0000 7f14f7989640 1 -- 172.21.15.28:0/4170918794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14f01a66c0 con 0x7f14f010bd00 2024-04-25T01:40:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.570+0000 7f14e67fc640 1 -- 172.21.15.28:0/4170918794 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e8033050 con 0x7f14f010bd00 2024-04-25T01:40:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.570+0000 7f14f7989640 1 -- 172.21.15.28:0/4170918794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14f01a6bb0 con 0x7f14f010bd00 2024-04-25T01:40:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.573+0000 7f14e67fc640 1 -- 172.21.15.28:0/4170918794 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f14e804a070 con 0x7f14f010bd00 2024-04-25T01:40:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.573+0000 7f14fa772740 1 Processor -- start 2024-04-25T01:40:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.573+0000 7f14fa772740 1 -- start start 2024-04-25T01:40:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.573+0000 7f14fa772740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c015165580 0x55c015165960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.573+0000 7f14fa772740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 0x55c015170030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.573+0000 7f14fa772740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c015170570 0x55c015172960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.573+0000 7f14fa772740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c0150ca670 con 0x55c015165ea0 2024-04-25T01:40:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.573+0000 7f14fa772740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c015072970 con 0x55c015170570 2024-04-25T01:40:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.573+0000 7f14fa772740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c0150c6630 con 0x55c015165580 2024-04-25T01:40:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.573+0000 7f14e67fc640 1 --2- 172.21.15.28:0/4170918794 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f14dc0a2d00 0x7f14dc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.573+0000 7f14f5725640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 0x55c015170030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.573+0000 7f14f5725640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 0x55c015170030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34660/0 (socket says 172.21.15.28:34660) 2024-04-25T01:40:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.573+0000 7f14f5725640 1 -- 172.21.15.28:0/403151156 learned_addr learned my addr 172.21.15.28:0/403151156 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.573+0000 7f14f4f24640 1 --2- 172.21.15.28:0/403151156 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c015165580 0x55c015165960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.573+0000 7f14f5f26640 1 --2- 172.21.15.28:0/4170918794 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f14dc0a2d00 0x7f14dc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.573+0000 7f14f4f24640 1 --2- 172.21.15.28:0/403151156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c015170570 0x55c015172960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.574+0000 7f14e67fc640 1 -- 172.21.15.28:0/4170918794 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f14e80f0350 con 0x7f14f010bd00 2024-04-25T01:40:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.574+0000 7f14f5725640 1 -- 172.21.15.28:0/403151156 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c015165580 msgr2=0x55c015165960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.574+0000 7f14f5725640 1 --2- 172.21.15.28:0/403151156 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c015165580 0x55c015165960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.574+0000 7f14f5725640 1 -- 172.21.15.28:0/403151156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c015170570 msgr2=0x55c015172960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.574+0000 7f14f5725640 1 --2- 172.21.15.28:0/403151156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c015170570 0x55c015172960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.574+0000 7f14f5725640 1 -- 172.21.15.28:0/403151156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c015096400 con 0x55c015165ea0 2024-04-25T01:40:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.574+0000 7f14f5725640 1 --2- 172.21.15.28:0/403151156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 0x55c015170030 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00cc60 tx=0x7f14ec00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13063d12310e325e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.574+0000 7f14f5f26640 1 --2- 172.21.15.28:0/4170918794 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f14dc0a2d00 0x7f14dc0a51c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f14e80ed7c0 tx=0x7f14e80875b0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.574+0000 7f14db7fe640 1 -- 172.21.15.28:0/403151156 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec014070 con 0x55c015165ea0 2024-04-25T01:40:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.574+0000 7f14db7fe640 1 -- 172.21.15.28:0/403151156 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14ec0047a0 con 0x55c015165ea0 2024-04-25T01:40:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.574+0000 7f14db7fe640 1 -- 172.21.15.28:0/403151156 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec010720 con 0x55c015165ea0 2024-04-25T01:40:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.575+0000 7f14fa772740 1 -- 172.21.15.28:0/403151156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 msgr2=0x55c015170030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.575+0000 7f14fa772740 1 --2- 172.21.15.28:0/403151156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 0x55c015170030 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00cc60 tx=0x7f14ec00e720 comp rx=0 tx=0).stop 2024-04-25T01:40:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.575+0000 7f14fa772740 1 -- 172.21.15.28:0/403151156 shutdown_connections 2024-04-25T01:40:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.575+0000 7f14fa772740 1 --2- 172.21.15.28:0/403151156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c015170570 0x55c015172960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.575+0000 7f14fa772740 1 --2- 172.21.15.28:0/403151156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 0x55c015170030 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.575+0000 7f14fa772740 1 --2- 172.21.15.28:0/403151156 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c015165580 0x55c015165960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.575+0000 7f14fa772740 1 -- 172.21.15.28:0/403151156 >> 172.21.15.28:0/403151156 conn(0x55c0151618a0 msgr2=0x55c015161c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.575+0000 7f14fa772740 1 -- 172.21.15.28:0/403151156 shutdown_connections 2024-04-25T01:40:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.575+0000 7f14fa772740 1 -- 172.21.15.28:0/403151156 wait complete. 2024-04-25T01:40:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.575+0000 7f14fa772740 1 -- start start 2024-04-25T01:40:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.575+0000 7f14fa772740 1 Processor -- start 2024-04-25T01:40:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14fa772740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165580 0x55c01516ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14fa772740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c015165ea0 0x55c01516d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14fa772740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c015170570 0x55c01516d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14fa772740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c015173360 con 0x55c015165580 2024-04-25T01:40:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14fa772740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c015086050 con 0x55c015165ea0 2024-04-25T01:40:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14fa772740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c0151734e0 con 0x55c015170570 2024-04-25T01:40:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14fa772740 10 client.0 _ll_register_callbacks cb 0x55c014dddbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14f4f24640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165580 0x55c01516ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14f4f24640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165580 0x55c01516ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34666/0 (socket says 172.21.15.28:34666) 2024-04-25T01:40:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14f4f24640 1 -- 172.21.15.28:0/1168672620 learned_addr learned my addr 172.21.15.28:0/1168672620 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14f5725640 1 --2- 172.21.15.28:0/1168672620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c015165ea0 0x55c01516d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14f4f24640 1 --2- 172.21.15.28:0/1168672620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c015170570 0x55c01516d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14f4f24640 1 -- 172.21.15.28:0/1168672620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c015170570 msgr2=0x55c01516d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14f4f24640 1 --2- 172.21.15.28:0/1168672620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c015170570 0x55c01516d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14f4f24640 1 -- 172.21.15.28:0/1168672620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c015165ea0 msgr2=0x55c01516d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14f4f24640 1 --2- 172.21.15.28:0/1168672620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c015165ea0 0x55c01516d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14f4f24640 1 -- 172.21.15.28:0/1168672620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c015057dc0 con 0x55c015165580 2024-04-25T01:40:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14f4f24640 1 --2- 172.21.15.28:0/1168672620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165580 0x55c01516ff40 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f14e0010a10 tx=0x7f14e0009a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770145038cb07fd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14d8ff9640 1 -- 172.21.15.28:0/1168672620 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e0025060 con 0x55c015165580 2024-04-25T01:40:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14d8ff9640 1 -- 172.21.15.28:0/1168672620 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f14e000fb90 con 0x55c015165580 2024-04-25T01:40:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.576+0000 7f14fa772740 1 -- 172.21.15.28:0/1168672620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14ec0047a0 con 0x55c015165580 2024-04-25T01:40:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.577+0000 7f14d8ff9640 1 -- 172.21.15.28:0/1168672620 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f14e002c050 con 0x55c015165580 2024-04-25T01:40:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.577+0000 7f14fa772740 10 client.18195 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.577+0000 7f14fa772740 1 -- 172.21.15.28:0/1168672620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c014dde3f0 con 0x55c015165580 2024-04-25T01:40:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.577+0000 7f14d8ff9640 1 -- 172.21.15.28:0/1168672620 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14ec0047a0 con 0x55c015165580 2024-04-25T01:40:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.577+0000 7f14d8ff9640 1 -- 172.21.15.28:0/1168672620 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f14e000e3c0 con 0x55c015165580 2024-04-25T01:40:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.577+0000 7f14fa772740 10 client.18195 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.577+0000 7f14fa772740 10 client.18195 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:40:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.577+0000 7f14fa772740 20 client.18195 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.577+0000 7f14fa772740 1 --2- 172.21.15.28:0/1168672620 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55c015180680 0x55c015182a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.577+0000 7f14fa772740 4 client.18195 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.577+0000 7f14fa772740 1 -- 172.21.15.28:0/1168672620 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0150d7bb0 con 0x55c015180680 2024-04-25T01:40:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.577+0000 7f14f5725640 1 --2- 172.21.15.28:0/1168672620 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55c015180680 0x55c015182a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.578+0000 7f14f5725640 1 --2- 172.21.15.28:0/1168672620 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55c015180680 0x55c015182a90 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=257ad99e32df6b6a server_cookie=2a8b6ae4a400827d in_seq=0 out_seq=0 2024-04-25T01:40:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.578+0000 7f14d8ff9640 10 client.18195 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:40:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.578+0000 7f14d8ff9640 1 -- 172.21.15.28:0/1168672620 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c0150d7bb0 con 0x55c015180680 2024-04-25T01:40:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.578+0000 7f14d8ff9640 10 client.18195 handle_command_reply: tid=0 2024-04-25T01:40:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.578+0000 7f14fa772740 1 client.18195 shutdown 2024-04-25T01:40:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.579+0000 7f14fa772740 1 -- 172.21.15.28:0/1168672620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165580 msgr2=0x55c01516ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.579+0000 7f14fa772740 1 --2- 172.21.15.28:0/1168672620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165580 0x55c01516ff40 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f14e0010a10 tx=0x7f14e0009a40 comp rx=0 tx=0).stop 2024-04-25T01:40:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.579+0000 7f14fa772740 1 -- 172.21.15.28:0/1168672620 shutdown_connections 2024-04-25T01:40:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.579+0000 7f14fa772740 1 --2- 172.21.15.28:0/1168672620 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55c015180680 0x55c015182a90 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.579+0000 7f14fa772740 1 --2- 172.21.15.28:0/1168672620 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c015170570 0x55c01516d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.579+0000 7f14fa772740 1 --2- 172.21.15.28:0/1168672620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c015165ea0 0x55c01516d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.579+0000 7f14d8ff9640 0 client.18195 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:40:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.579+0000 7f14fa772740 1 --2- 172.21.15.28:0/1168672620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165580 0x55c01516ff40 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.579+0000 7f14fa772740 1 -- 172.21.15.28:0/1168672620 >> 172.21.15.28:0/1168672620 conn(0x55c0151618a0 msgr2=0x55c015160b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.579+0000 7f14fa772740 1 -- 172.21.15.28:0/1168672620 shutdown_connections 2024-04-25T01:40:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.579+0000 7f14fa772740 1 -- 172.21.15.28:0/1168672620 wait complete. 2024-04-25T01:40:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.579+0000 7f14fa772740 20 client.18195 trim_cache size 0 max 16384 2024-04-25T01:40:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.592+0000 7f14fa772740 1 Processor -- start 2024-04-25T01:40:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.592+0000 7f14fa772740 1 -- start start 2024-04-25T01:40:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.592+0000 7f14fa772740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c015165580 0x55c01517c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.592+0000 7f14fa772740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 0x55c01516c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.592+0000 7f14fa772740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c015170570 0x55c015188310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.592+0000 7f14fa772740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14f01b0cf0 con 0x55c015165ea0 2024-04-25T01:40:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.592+0000 7f14fa772740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f14f01b2830 con 0x55c015170570 2024-04-25T01:40:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.592+0000 7f14fa772740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f14f01b13c0 con 0x55c015165580 2024-04-25T01:40:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.592+0000 7f14f5725640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 0x55c01516c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.592+0000 7f14f5725640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 0x55c01516c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34674/0 (socket says 172.21.15.28:34674) 2024-04-25T01:40:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.592+0000 7f14f5725640 1 -- 172.21.15.28:0/4013581335 learned_addr learned my addr 172.21.15.28:0/4013581335 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.593+0000 7f14f5725640 1 -- 172.21.15.28:0/4013581335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c015165580 msgr2=0x55c01517c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.593+0000 7f14f5725640 1 --2- 172.21.15.28:0/4013581335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c015165580 0x55c01517c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.593+0000 7f14f5725640 1 -- 172.21.15.28:0/4013581335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c015170570 msgr2=0x55c015188310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.593+0000 7f14f5725640 1 --2- 172.21.15.28:0/4013581335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c015170570 0x55c015188310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.593+0000 7f14f5725640 1 -- 172.21.15.28:0/4013581335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c014ea87d0 con 0x55c015165ea0 2024-04-25T01:40:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.593+0000 7f14f5725640 1 --2- 172.21.15.28:0/4013581335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 0x55c01516c390 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00cfd0 tx=0x7f14ec0046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30371dae241f530d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.593+0000 7f14daffd640 1 -- 172.21.15.28:0/4013581335 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec014070 con 0x55c015165ea0 2024-04-25T01:40:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.593+0000 7f14daffd640 1 -- 172.21.15.28:0/4013581335 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14ec004520 con 0x55c015165ea0 2024-04-25T01:40:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.593+0000 7f14daffd640 1 -- 172.21.15.28:0/4013581335 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec0260d0 con 0x55c015165ea0 2024-04-25T01:40:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.593+0000 7f14fa772740 1 -- 172.21.15.28:0/4013581335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 msgr2=0x55c01516c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.593+0000 7f14fa772740 1 --2- 172.21.15.28:0/4013581335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 0x55c01516c390 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00cfd0 tx=0x7f14ec0046e0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.593+0000 7f14fa772740 1 -- 172.21.15.28:0/4013581335 shutdown_connections 2024-04-25T01:40:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.593+0000 7f14fa772740 1 --2- 172.21.15.28:0/4013581335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c015170570 0x55c015188310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.593+0000 7f14fa772740 1 --2- 172.21.15.28:0/4013581335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 0x55c01516c390 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.593+0000 7f14fa772740 1 --2- 172.21.15.28:0/4013581335 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c015165580 0x55c01517c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.593+0000 7f14fa772740 1 -- 172.21.15.28:0/4013581335 >> 172.21.15.28:0/4013581335 conn(0x55c0151618a0 msgr2=0x55c0151602b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.594+0000 7f14fa772740 1 -- 172.21.15.28:0/4013581335 shutdown_connections 2024-04-25T01:40:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.594+0000 7f14fa772740 1 -- 172.21.15.28:0/4013581335 wait complete. 2024-04-25T01:40:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.594+0000 7f14fa772740 1 -- start start 2024-04-25T01:40:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.594+0000 7f14fa772740 1 Processor -- start 2024-04-25T01:40:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.594+0000 7f14fa772740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c015165580 0x55c015178b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.594+0000 7f14fa772740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 0x55c015175f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.594+0000 7f14fa772740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c015170570 0x55c015176460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.594+0000 7f14fa772740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14f01b2160 con 0x55c015165ea0 2024-04-25T01:40:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.594+0000 7f14fa772740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f14f01b1a90 con 0x55c015170570 2024-04-25T01:40:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.594+0000 7f14fa772740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f14f01aecf0 con 0x55c015165580 2024-04-25T01:40:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.594+0000 7f14fa772740 10 client.0 _ll_register_callbacks cb 0x7f14e0010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.594+0000 7f14f5725640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 0x55c015175f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.594+0000 7f14f5725640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 0x55c015175f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34686/0 (socket says 172.21.15.28:34686) 2024-04-25T01:40:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.594+0000 7f14f5725640 1 -- 172.21.15.28:0/1072473395 learned_addr learned my addr 172.21.15.28:0/1072473395 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.594+0000 7f14f4f24640 1 --2- 172.21.15.28:0/1072473395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c015170570 0x55c015176460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.595+0000 7f14f4f24640 1 --2- 172.21.15.28:0/1072473395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c015165580 0x55c015178b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.595+0000 7f14f5725640 1 -- 172.21.15.28:0/1072473395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c015165580 msgr2=0x55c015178b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.595+0000 7f14f5725640 1 --2- 172.21.15.28:0/1072473395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c015165580 0x55c015178b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.595+0000 7f14f5725640 1 -- 172.21.15.28:0/1072473395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c015170570 msgr2=0x55c015176460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.595+0000 7f14f5725640 1 --2- 172.21.15.28:0/1072473395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c015170570 0x55c015176460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.595+0000 7f14f5725640 1 -- 172.21.15.28:0/1072473395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c015176c90 con 0x55c015165ea0 2024-04-25T01:40:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.595+0000 7f14f5725640 1 --2- 172.21.15.28:0/1072473395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 0x55c015175f20 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00cfa0 tx=0x7f14ec0040d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bafeda2f77542d3b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.595+0000 7f14d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.595+0000 7f14d8ff9640 1 -- 172.21.15.28:0/1072473395 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec014070 con 0x55c015165ea0 2024-04-25T01:40:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.595+0000 7f14d8ff9640 1 -- 172.21.15.28:0/1072473395 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f14ec0262d0 con 0x55c015165ea0 2024-04-25T01:40:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.595+0000 7f14fa772740 1 -- 172.21.15.28:0/1072473395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c0150d1f00 con 0x55c015165ea0 2024-04-25T01:40:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.595+0000 7f14d8ff9640 1 -- 172.21.15.28:0/1072473395 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f14ec0264f0 con 0x55c015165ea0 2024-04-25T01:40:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.595+0000 7f14fa772740 10 client.18201 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.595+0000 7f14fa772740 1 -- 172.21.15.28:0/1072473395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c0151e6bf0 con 0x55c015165ea0 2024-04-25T01:40:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.595+0000 7f14d8ff9640 1 -- 172.21.15.28:0/1072473395 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c0150d1f00 con 0x55c015165ea0 2024-04-25T01:40:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.595+0000 7f14d8ff9640 1 -- 172.21.15.28:0/1072473395 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f14ec018410 con 0x55c015165ea0 2024-04-25T01:40:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.596+0000 7f14fa772740 10 client.18201 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.596+0000 7f14fa772740 10 client.18201 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:40:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.596+0000 7f14fa772740 20 client.18201 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.596+0000 7f14fa772740 1 --2- 172.21.15.28:0/1072473395 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55c015180680 0x55c0151e90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.596+0000 7f14fa772740 4 client.18201 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.596+0000 7f14fa772740 1 -- 172.21.15.28:0/1072473395 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c01504f820 con 0x55c015180680 2024-04-25T01:40:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.596+0000 7f14f4f24640 1 --2- 172.21.15.28:0/1072473395 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55c015180680 0x55c0151e90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.596+0000 7f14d8ff9640 10 client.18201 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:40:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.596+0000 7f14f4f24640 1 --2- 172.21.15.28:0/1072473395 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55c015180680 0x55c0151e90d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c8517be199f591a6 server_cookie=9633eb777c829ca8 in_seq=0 out_seq=0 2024-04-25T01:40:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.596+0000 7f14d8ff9640 1 -- 172.21.15.28:0/1072473395 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c01504f820 con 0x55c015180680 2024-04-25T01:40:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.596+0000 7f14d8ff9640 10 client.18201 handle_command_reply: tid=0 2024-04-25T01:40:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.596+0000 7f14fa772740 1 client.18201 shutdown 2024-04-25T01:40:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.596+0000 7f14fa772740 1 -- 172.21.15.28:0/1072473395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 msgr2=0x55c015175f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.596+0000 7f14fa772740 1 --2- 172.21.15.28:0/1072473395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 0x55c015175f20 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00cfa0 tx=0x7f14ec0040d0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.596+0000 7f14fa772740 1 -- 172.21.15.28:0/1072473395 shutdown_connections 2024-04-25T01:40:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14fa772740 1 --2- 172.21.15.28:0/1072473395 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c015170570 0x55c015176460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14fa772740 1 --2- 172.21.15.28:0/1072473395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c015165ea0 0x55c015175f20 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14fa772740 1 --2- 172.21.15.28:0/1072473395 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55c015180680 0x55c0151e90d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14fa772740 1 --2- 172.21.15.28:0/1072473395 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c015165580 0x55c015178b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14d8ff9640 0 client.18201 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:40:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14fa772740 1 -- 172.21.15.28:0/1072473395 >> 172.21.15.28:0/1072473395 conn(0x55c0151618a0 msgr2=0x55c015160b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14fa772740 1 -- 172.21.15.28:0/1072473395 shutdown_connections 2024-04-25T01:40:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14fa772740 1 -- 172.21.15.28:0/1072473395 wait complete. 2024-04-25T01:40:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14fa772740 20 client.18201 trim_cache size 0 max 16384 2024-04-25T01:40:21.588 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14f7989640 1 -- 172.21.15.28:0/4170918794 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f14dc0a2d00 msgr2=0x7f14dc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14f7989640 1 --2- 172.21.15.28:0/4170918794 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f14dc0a2d00 0x7f14dc0a51c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f14e80ed7c0 tx=0x7f14e80875b0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14f7989640 1 -- 172.21.15.28:0/4170918794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f010bd00 msgr2=0x7f14f0101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14f7989640 1 --2- 172.21.15.28:0/4170918794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f010bd00 0x7f14f0101090 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f14e8004af0 tx=0x7f14e8003f30 comp rx=0 tx=0).stop 2024-04-25T01:40:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14f7989640 1 -- 172.21.15.28:0/4170918794 shutdown_connections 2024-04-25T01:40:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14f7989640 1 --2- 172.21.15.28:0/4170918794 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f14dc0a2d00 0x7f14dc0a51c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14f7989640 1 --2- 172.21.15.28:0/4170918794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f010bd00 0x7f14f0101090 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14f7989640 1 --2- 172.21.15.28:0/4170918794 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14f01032e0 0x7f14f0100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14f7989640 1 --2- 172.21.15.28:0/4170918794 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14f01029c0 0x7f14f0100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14f7989640 1 -- 172.21.15.28:0/4170918794 >> 172.21.15.28:0/4170918794 conn(0x7f14f0077740 msgr2=0x7f14f00fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14f7989640 1 -- 172.21.15.28:0/4170918794 shutdown_connections 2024-04-25T01:40:21.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.597+0000 7f14f7989640 1 -- 172.21.15.28:0/4170918794 wait complete. 2024-04-25T01:40:21.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:40:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.712+0000 7f84ea7e7640 1 -- 172.21.15.28:0/3087154546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84e4107730 msgr2=0x7f84e4107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.712+0000 7f84ea7e7640 1 --2- 172.21.15.28:0/3087154546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84e4107730 0x7f84e4107b10 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f84d4009930 tx=0x7f84d402a420 comp rx=0 tx=0).stop 2024-04-25T01:40:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.712+0000 7f84ea7e7640 1 -- 172.21.15.28:0/3087154546 shutdown_connections 2024-04-25T01:40:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.712+0000 7f84ea7e7640 1 --2- 172.21.15.28:0/3087154546 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84e406a780 0x7f84e410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.712+0000 7f84ea7e7640 1 --2- 172.21.15.28:0/3087154546 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84e4069dc0 0x7f84e406a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.712+0000 7f84ea7e7640 1 --2- 172.21.15.28:0/3087154546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84e4107730 0x7f84e4107b10 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.712+0000 7f84ea7e7640 1 -- 172.21.15.28:0/3087154546 >> 172.21.15.28:0/3087154546 conn(0x7f84e40776a0 msgr2=0x7f84e4077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.712+0000 7f84ea7e7640 1 -- 172.21.15.28:0/3087154546 shutdown_connections 2024-04-25T01:40:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.712+0000 7f84ea7e7640 1 -- 172.21.15.28:0/3087154546 wait complete. 2024-04-25T01:40:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.713+0000 7f84ea7e7640 1 Processor -- start 2024-04-25T01:40:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.713+0000 7f84ea7e7640 1 -- start start 2024-04-25T01:40:21.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.713+0000 7f84ea7e7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84e4069dc0 0x7f84e4198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.713+0000 7f84ea7e7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84e406a780 0x7f84e41985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.713+0000 7f84ea7e7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84e4107730 0x7f84e419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.713+0000 7f84ea7e7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84e4111b00 con 0x7f84e4107730 2024-04-25T01:40:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.713+0000 7f84ea7e7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f84e4111980 con 0x7f84e4069dc0 2024-04-25T01:40:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.713+0000 7f84ea7e7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f84e4111c80 con 0x7f84e406a780 2024-04-25T01:40:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.713+0000 7f84e8d84640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84e4107730 0x7f84e419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.714+0000 7f84e8d84640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84e4107730 0x7f84e419c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34718/0 (socket says 172.21.15.28:34718) 2024-04-25T01:40:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.714+0000 7f84e8d84640 1 -- 172.21.15.28:0/2950149127 learned_addr learned my addr 172.21.15.28:0/2950149127 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.714+0000 7f84e3fff640 1 --2- 172.21.15.28:0/2950149127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84e4069dc0 0x7f84e4198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.714+0000 7f84e37fe640 1 --2- 172.21.15.28:0/2950149127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84e406a780 0x7f84e41985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.714+0000 7f84e8d84640 1 -- 172.21.15.28:0/2950149127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84e406a780 msgr2=0x7f84e41985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.714+0000 7f84e8d84640 1 --2- 172.21.15.28:0/2950149127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84e406a780 0x7f84e41985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.714+0000 7f84e8d84640 1 -- 172.21.15.28:0/2950149127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84e4069dc0 msgr2=0x7f84e4198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.714+0000 7f84e8d84640 1 --2- 172.21.15.28:0/2950149127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84e4069dc0 0x7f84e4198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.714+0000 7f84e8d84640 1 -- 172.21.15.28:0/2950149127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e419d050 con 0x7f84e4107730 2024-04-25T01:40:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.714+0000 7f84e37fe640 1 --2- 172.21.15.28:0/2950149127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84e406a780 0x7f84e41985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:21.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.714+0000 7f84e8d84640 1 --2- 172.21.15.28:0/2950149127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84e4107730 0x7f84e419c920 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00efd0 tx=0x7f84dc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.714+0000 7f84e17fa640 1 -- 172.21.15.28:0/2950149127 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc003b70 con 0x7f84e4107730 2024-04-25T01:40:21.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.714+0000 7f84ea7e7640 1 -- 172.21.15.28:0/2950149127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84e419d340 con 0x7f84e4107730 2024-04-25T01:40:21.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.714+0000 7f84e17fa640 1 -- 172.21.15.28:0/2950149127 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84dc003d10 con 0x7f84e4107730 2024-04-25T01:40:21.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.714+0000 7f84e17fa640 1 -- 172.21.15.28:0/2950149127 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84dc0109f0 con 0x7f84e4107730 2024-04-25T01:40:21.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.714+0000 7f84ea7e7640 1 -- 172.21.15.28:0/2950149127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84e41a4bd0 con 0x7f84e4107730 2024-04-25T01:40:21.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.716+0000 7f84e17fa640 1 -- 172.21.15.28:0/2950149127 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f84dc004020 con 0x7f84e4107730 2024-04-25T01:40:21.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.716+0000 7f84ed5d5740 1 Processor -- start 2024-04-25T01:40:21.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84ed5d5740 1 -- start start 2024-04-25T01:40:21.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84ed5d5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836579480 0x562836579860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84ed5d5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579da0 0x562836588440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84ed5d5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836588980 0x56283658ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84ed5d5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5628364de670 con 0x562836579da0 2024-04-25T01:40:21.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84ed5d5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x562836486970 con 0x562836579480 2024-04-25T01:40:21.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84ed5d5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5628364da630 con 0x562836588980 2024-04-25T01:40:21.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84e3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579da0 0x562836588440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84e37fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836579480 0x562836579860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84e3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579da0 0x562836588440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34732/0 (socket says 172.21.15.28:34732) 2024-04-25T01:40:21.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84e3fff640 1 -- 172.21.15.28:0/3917398400 learned_addr learned my addr 172.21.15.28:0/3917398400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84e37fe640 1 --2- 172.21.15.28:0/3917398400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836588980 0x56283658ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84e3fff640 1 -- 172.21.15.28:0/3917398400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836588980 msgr2=0x56283658ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84e3fff640 1 --2- 172.21.15.28:0/3917398400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836588980 0x56283658ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84e17fa640 1 --2- 172.21.15.28:0/2950149127 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f84cc0a2d00 0x7f84cc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84e3fff640 1 -- 172.21.15.28:0/3917398400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836579480 msgr2=0x562836579860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84e3fff640 1 --2- 172.21.15.28:0/3917398400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836579480 0x562836579860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84e3fff640 1 -- 172.21.15.28:0/3917398400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628364aa400 con 0x562836579da0 2024-04-25T01:40:21.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84e37fe640 1 --2- 172.21.15.28:0/3917398400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836588980 0x56283658ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:21.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84e8d84640 1 --2- 172.21.15.28:0/2950149127 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f84cc0a2d00 0x7f84cc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.717+0000 7f84e3fff640 1 --2- 172.21.15.28:0/3917398400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579da0 0x562836588440 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f84d402a940 tx=0x7f84d4005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ae4872b173b66d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.718+0000 7f84e17fa640 1 -- 172.21.15.28:0/2950149127 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f84dc0cfc50 con 0x7f84e4107730 2024-04-25T01:40:21.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.718+0000 7f84ca7fc640 1 -- 172.21.15.28:0/3917398400 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d4040070 con 0x562836579da0 2024-04-25T01:40:21.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.718+0000 7f84ca7fc640 1 -- 172.21.15.28:0/3917398400 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84d40318f0 con 0x562836579da0 2024-04-25T01:40:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.718+0000 7f84ca7fc640 1 -- 172.21.15.28:0/3917398400 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d4035040 con 0x562836579da0 2024-04-25T01:40:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.718+0000 7f84ed5d5740 1 -- 172.21.15.28:0/3917398400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579da0 msgr2=0x562836588440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.718+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/3917398400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579da0 0x562836588440 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f84d402a940 tx=0x7f84d4005c00 comp rx=0 tx=0).stop 2024-04-25T01:40:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.718+0000 7f84e8d84640 1 --2- 172.21.15.28:0/2950149127 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f84cc0a2d00 0x7f84cc0a51c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f84dc0cd0c0 tx=0x7f84dc06dda0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.718+0000 7f84ed5d5740 1 -- 172.21.15.28:0/3917398400 shutdown_connections 2024-04-25T01:40:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.718+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/3917398400 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836588980 0x56283658ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.718+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/3917398400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579da0 0x562836588440 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.718+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/3917398400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836579480 0x562836579860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.718+0000 7f84ed5d5740 1 -- 172.21.15.28:0/3917398400 >> 172.21.15.28:0/3917398400 conn(0x5628365758a0 msgr2=0x562836575c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.718+0000 7f84ed5d5740 1 -- 172.21.15.28:0/3917398400 shutdown_connections 2024-04-25T01:40:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.718+0000 7f84ed5d5740 1 -- 172.21.15.28:0/3917398400 wait complete. 2024-04-25T01:40:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.719+0000 7f84ed5d5740 1 -- start start 2024-04-25T01:40:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.719+0000 7f84ed5d5740 1 Processor -- start 2024-04-25T01:40:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.719+0000 7f84ed5d5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579480 0x5628365882a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.719+0000 7f84ed5d5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836579da0 0x562836585590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.719+0000 7f84ed5d5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836588980 0x562836585ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.719+0000 7f84ed5d5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56283658b6e0 con 0x562836579480 2024-04-25T01:40:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.719+0000 7f84ed5d5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56283649a050 con 0x562836588980 2024-04-25T01:40:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.719+0000 7f84ed5d5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56283658b860 con 0x562836579da0 2024-04-25T01:40:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.719+0000 7f84ed5d5740 10 client.0 _ll_register_callbacks cb 0x5628361f1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.719+0000 7f84e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579480 0x5628365882a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.719+0000 7f84e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579480 0x5628365882a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34736/0 (socket says 172.21.15.28:34736) 2024-04-25T01:40:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.719+0000 7f84e37fe640 1 -- 172.21.15.28:0/2477047391 learned_addr learned my addr 172.21.15.28:0/2477047391 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.719+0000 7f84e3fff640 1 --2- 172.21.15.28:0/2477047391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836579da0 0x562836585590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.720+0000 7f84e37fe640 1 --2- 172.21.15.28:0/2477047391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836588980 0x562836585ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.720+0000 7f84e37fe640 1 -- 172.21.15.28:0/2477047391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836579da0 msgr2=0x562836585590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.720+0000 7f84e37fe640 1 --2- 172.21.15.28:0/2477047391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836579da0 0x562836585590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.720+0000 7f84e37fe640 1 -- 172.21.15.28:0/2477047391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836588980 msgr2=0x562836585ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.720+0000 7f84e37fe640 1 --2- 172.21.15.28:0/2477047391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836588980 0x562836585ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.720+0000 7f84e37fe640 1 -- 172.21.15.28:0/2477047391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56283646bdc0 con 0x562836579480 2024-04-25T01:40:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.720+0000 7f84e3fff640 1 --2- 172.21.15.28:0/2477047391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836579da0 0x562836585590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.720+0000 7f84e37fe640 1 --2- 172.21.15.28:0/2477047391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579480 0x5628365882a0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f84d8004630 tx=0x7f84d8016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e954923dbb7a07 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.720+0000 7f84c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.720+0000 7f84c37fe640 1 -- 172.21.15.28:0/2477047391 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d8012b60 con 0x562836579480 2024-04-25T01:40:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.720+0000 7f84c37fe640 1 -- 172.21.15.28:0/2477047391 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f84d8012d00 con 0x562836579480 2024-04-25T01:40:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.720+0000 7f84ed5d5740 1 -- 172.21.15.28:0/2477047391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84d40318f0 con 0x562836579480 2024-04-25T01:40:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.720+0000 7f84c37fe640 1 -- 172.21.15.28:0/2477047391 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f84d8005020 con 0x562836579480 2024-04-25T01:40:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.720+0000 7f84ed5d5740 10 client.18213 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.720+0000 7f84ed5d5740 1 -- 172.21.15.28:0/2477047391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5628361f23f0 con 0x562836579480 2024-04-25T01:40:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.720+0000 7f84c37fe640 1 -- 172.21.15.28:0/2477047391 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84d40318f0 con 0x562836579480 2024-04-25T01:40:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.720+0000 7f84c37fe640 1 -- 172.21.15.28:0/2477047391 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f84d8011470 con 0x562836579480 2024-04-25T01:40:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.721+0000 7f84ed5d5740 10 client.18213 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.721+0000 7f84ed5d5740 10 client.18213 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:40:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.721+0000 7f84ed5d5740 20 client.18213 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.721+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/2477047391 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x562836598980 0x56283659ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.721+0000 7f84ed5d5740 4 client.18213 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.721+0000 7f84ed5d5740 1 -- 172.21.15.28:0/2477047391 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628364ebbb0 con 0x562836598980 2024-04-25T01:40:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.721+0000 7f84e3fff640 1 --2- 172.21.15.28:0/2477047391 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x562836598980 0x56283659ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.721+0000 7f84e3fff640 1 --2- 172.21.15.28:0/2477047391 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x562836598980 0x56283659ae00 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=804aae5e4cd2d04f server_cookie=c13652fe262f4e19 in_seq=0 out_seq=0 2024-04-25T01:40:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.721+0000 7f84c37fe640 10 client.18213 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:40:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.722+0000 7f84c37fe640 1 -- 172.21.15.28:0/2477047391 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5628364ebbb0 con 0x562836598980 2024-04-25T01:40:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.722+0000 7f84c37fe640 10 client.18213 handle_command_reply: tid=0 2024-04-25T01:40:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.722+0000 7f84ed5d5740 1 client.18213 shutdown 2024-04-25T01:40:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.722+0000 7f84ed5d5740 1 -- 172.21.15.28:0/2477047391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579480 msgr2=0x5628365882a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.722+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/2477047391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579480 0x5628365882a0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f84d8004630 tx=0x7f84d8016040 comp rx=0 tx=0).stop 2024-04-25T01:40:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.722+0000 7f84ed5d5740 1 -- 172.21.15.28:0/2477047391 shutdown_connections 2024-04-25T01:40:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.722+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/2477047391 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x562836598980 0x56283659ae00 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.722+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/2477047391 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836588980 0x562836585ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.722+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/2477047391 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836579da0 0x562836585590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.722+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/2477047391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579480 0x5628365882a0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.722+0000 7f84c37fe640 0 client.18213 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:40:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.722+0000 7f84ed5d5740 1 -- 172.21.15.28:0/2477047391 >> 172.21.15.28:0/2477047391 conn(0x5628365758a0 msgr2=0x562836574b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.722+0000 7f84ed5d5740 1 -- 172.21.15.28:0/2477047391 shutdown_connections 2024-04-25T01:40:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.722+0000 7f84ed5d5740 1 -- 172.21.15.28:0/2477047391 wait complete. 2024-04-25T01:40:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.722+0000 7f84ed5d5740 20 client.18213 trim_cache size 0 max 16384 2024-04-25T01:40:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.736+0000 7f84ed5d5740 1 Processor -- start 2024-04-25T01:40:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.736+0000 7f84ed5d5740 1 -- start start 2024-04-25T01:40:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.736+0000 7f84ed5d5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579480 0x5628365846f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.736+0000 7f84ed5d5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836579da0 0x562836587dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.736+0000 7f84ed5d5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836588980 0x5628365a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.736+0000 7f84ed5d5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84e41ac970 con 0x562836579480 2024-04-25T01:40:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.736+0000 7f84ed5d5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f84e41ae4b0 con 0x562836579da0 2024-04-25T01:40:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.736+0000 7f84ed5d5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f84e41ad040 con 0x562836588980 2024-04-25T01:40:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.736+0000 7f84e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579480 0x5628365846f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.736+0000 7f84e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579480 0x5628365846f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34750/0 (socket says 172.21.15.28:34750) 2024-04-25T01:40:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.736+0000 7f84e37fe640 1 -- 172.21.15.28:0/527181336 learned_addr learned my addr 172.21.15.28:0/527181336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.736+0000 7f84e37fe640 1 -- 172.21.15.28:0/527181336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836588980 msgr2=0x5628365a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.736+0000 7f84e37fe640 1 --2- 172.21.15.28:0/527181336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836588980 0x5628365a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.736+0000 7f84e37fe640 1 -- 172.21.15.28:0/527181336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836579da0 msgr2=0x562836587dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.736+0000 7f84e3fff640 1 --2- 172.21.15.28:0/527181336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836579da0 0x562836587dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.737+0000 7f84e37fe640 1 --2- 172.21.15.28:0/527181336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836579da0 0x562836587dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.737+0000 7f84e37fe640 1 -- 172.21.15.28:0/527181336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628362bc7d0 con 0x562836579480 2024-04-25T01:40:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.737+0000 7f84e3fff640 1 --2- 172.21.15.28:0/527181336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836579da0 0x562836587dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.737+0000 7f84e37fe640 1 --2- 172.21.15.28:0/527181336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579480 0x5628365846f0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f84d80118b0 tx=0x7f84d8004460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5315b2511e865a3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.737+0000 7f84c9ffb640 1 -- 172.21.15.28:0/527181336 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d802d020 con 0x562836579480 2024-04-25T01:40:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.737+0000 7f84c9ffb640 1 -- 172.21.15.28:0/527181336 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84d8011470 con 0x562836579480 2024-04-25T01:40:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.737+0000 7f84c9ffb640 1 -- 172.21.15.28:0/527181336 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d80055c0 con 0x562836579480 2024-04-25T01:40:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.737+0000 7f84ed5d5740 1 -- 172.21.15.28:0/527181336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579480 msgr2=0x5628365846f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.737+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/527181336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579480 0x5628365846f0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f84d80118b0 tx=0x7f84d8004460 comp rx=0 tx=0).stop 2024-04-25T01:40:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.737+0000 7f84ed5d5740 1 -- 172.21.15.28:0/527181336 shutdown_connections 2024-04-25T01:40:21.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.737+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/527181336 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836588980 0x5628365a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.737+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/527181336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836579da0 0x562836587dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.737+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/527181336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579480 0x5628365846f0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.737+0000 7f84ed5d5740 1 -- 172.21.15.28:0/527181336 >> 172.21.15.28:0/527181336 conn(0x5628365758a0 msgr2=0x56283657ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.738+0000 7f84ed5d5740 1 -- 172.21.15.28:0/527181336 shutdown_connections 2024-04-25T01:40:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.738+0000 7f84ed5d5740 1 -- 172.21.15.28:0/527181336 wait complete. 2024-04-25T01:40:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.738+0000 7f84ed5d5740 1 -- start start 2024-04-25T01:40:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.738+0000 7f84ed5d5740 1 Processor -- start 2024-04-25T01:40:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.738+0000 7f84ed5d5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836579480 0x562836590ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.738+0000 7f84ed5d5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579da0 0x562836581de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.738+0000 7f84ed5d5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836588980 0x562836582320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.738+0000 7f84ed5d5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84e41adde0 con 0x562836579da0 2024-04-25T01:40:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.738+0000 7f84ed5d5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f84e41ad710 con 0x562836588980 2024-04-25T01:40:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.738+0000 7f84ed5d5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f84e41aa970 con 0x562836579480 2024-04-25T01:40:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.738+0000 7f84ed5d5740 10 client.0 _ll_register_callbacks cb 0x7f84d800b860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:21.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.738+0000 7f84e3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579da0 0x562836581de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.738+0000 7f84e3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579da0 0x562836581de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34764/0 (socket says 172.21.15.28:34764) 2024-04-25T01:40:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.738+0000 7f84e3fff640 1 -- 172.21.15.28:0/1476255099 learned_addr learned my addr 172.21.15.28:0/1476255099 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.738+0000 7f84e37fe640 1 --2- 172.21.15.28:0/1476255099 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836579480 0x562836590ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.739+0000 7f84e37fe640 1 --2- 172.21.15.28:0/1476255099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836588980 0x562836582320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.739+0000 7f84e3fff640 1 -- 172.21.15.28:0/1476255099 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836579480 msgr2=0x562836590ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.739+0000 7f84e3fff640 1 --2- 172.21.15.28:0/1476255099 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836579480 0x562836590ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.739+0000 7f84e3fff640 1 -- 172.21.15.28:0/1476255099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836588980 msgr2=0x562836582320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.739+0000 7f84e3fff640 1 --2- 172.21.15.28:0/1476255099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836588980 0x562836582320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.739+0000 7f84e3fff640 1 -- 172.21.15.28:0/1476255099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562836582960 con 0x562836579da0 2024-04-25T01:40:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.739+0000 7f84e37fe640 1 --2- 172.21.15.28:0/1476255099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836588980 0x562836582320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.739+0000 7f84e3fff640 1 --2- 172.21.15.28:0/1476255099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579da0 0x562836581de0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f84d402a940 tx=0x7f84d4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f377816087db2af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.739+0000 7f84c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.739+0000 7f84c37fe640 1 -- 172.21.15.28:0/1476255099 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d4040040 con 0x562836579da0 2024-04-25T01:40:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.739+0000 7f84c37fe640 1 -- 172.21.15.28:0/1476255099 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f84d40316f0 con 0x562836579da0 2024-04-25T01:40:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.739+0000 7f84ed5d5740 1 -- 172.21.15.28:0/1476255099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84d8011470 con 0x562836579da0 2024-04-25T01:40:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.739+0000 7f84c37fe640 1 -- 172.21.15.28:0/1476255099 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f84d405a050 con 0x562836579da0 2024-04-25T01:40:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.739+0000 7f84ed5d5740 10 client.18219 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.739+0000 7f84ed5d5740 1 -- 172.21.15.28:0/1476255099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5628365fed10 con 0x562836579da0 2024-04-25T01:40:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.739+0000 7f84c37fe640 1 -- 172.21.15.28:0/1476255099 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84d8011470 con 0x562836579da0 2024-04-25T01:40:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.740+0000 7f84c37fe640 1 -- 172.21.15.28:0/1476255099 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f84d40039f0 con 0x562836579da0 2024-04-25T01:40:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.740+0000 7f84ed5d5740 10 client.18219 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.740+0000 7f84ed5d5740 10 client.18219 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:40:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.740+0000 7f84ed5d5740 20 client.18219 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.740+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/1476255099 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5628365ff110 0x562836601500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.740+0000 7f84ed5d5740 4 client.18219 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.740+0000 7f84ed5d5740 1 -- 172.21.15.28:0/1476255099 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5628364e5f00 con 0x5628365ff110 2024-04-25T01:40:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.740+0000 7f84e37fe640 1 --2- 172.21.15.28:0/1476255099 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5628365ff110 0x562836601500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.740+0000 7f84e37fe640 1 --2- 172.21.15.28:0/1476255099 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5628365ff110 0x562836601500 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=76f130ddcdbb7c6c server_cookie=8ff5605d2d3fd117 in_seq=0 out_seq=0 2024-04-25T01:40:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.740+0000 7f84c37fe640 10 client.18219 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:40:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.740+0000 7f84c37fe640 1 -- 172.21.15.28:0/1476255099 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5628364e5f00 con 0x5628365ff110 2024-04-25T01:40:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.740+0000 7f84c37fe640 10 client.18219 handle_command_reply: tid=0 2024-04-25T01:40:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.740+0000 7f84ed5d5740 1 client.18219 shutdown 2024-04-25T01:40:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ed5d5740 1 -- 172.21.15.28:0/1476255099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579da0 msgr2=0x562836581de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/1476255099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579da0 0x562836581de0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f84d402a940 tx=0x7f84d4005d50 comp rx=0 tx=0).stop 2024-04-25T01:40:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ed5d5740 1 -- 172.21.15.28:0/1476255099 shutdown_connections 2024-04-25T01:40:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/1476255099 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5628365ff110 0x562836601500 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/1476255099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562836588980 0x562836582320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/1476255099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562836579da0 0x562836581de0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84c37fe640 0 client.18219 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:40:21.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ed5d5740 1 --2- 172.21.15.28:0/1476255099 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562836579480 0x562836590ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ed5d5740 1 -- 172.21.15.28:0/1476255099 >> 172.21.15.28:0/1476255099 conn(0x5628365758a0 msgr2=0x56283657b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ed5d5740 1 -- 172.21.15.28:0/1476255099 shutdown_connections 2024-04-25T01:40:21.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ed5d5740 1 -- 172.21.15.28:0/1476255099 wait complete. 2024-04-25T01:40:21.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ed5d5740 20 client.18219 trim_cache size 0 max 16384 2024-04-25T01:40:21.733 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:21.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ea7e7640 1 -- 172.21.15.28:0/2950149127 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f84cc0a2d00 msgr2=0x7f84cc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ea7e7640 1 --2- 172.21.15.28:0/2950149127 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f84cc0a2d00 0x7f84cc0a51c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f84dc0cd0c0 tx=0x7f84dc06dda0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ea7e7640 1 -- 172.21.15.28:0/2950149127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84e4107730 msgr2=0x7f84e419c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ea7e7640 1 --2- 172.21.15.28:0/2950149127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84e4107730 0x7f84e419c920 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00efd0 tx=0x7f84dc00c6a0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ea7e7640 1 -- 172.21.15.28:0/2950149127 shutdown_connections 2024-04-25T01:40:21.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ea7e7640 1 --2- 172.21.15.28:0/2950149127 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f84cc0a2d00 0x7f84cc0a51c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ea7e7640 1 --2- 172.21.15.28:0/2950149127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84e4107730 0x7f84e419c920 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ea7e7640 1 --2- 172.21.15.28:0/2950149127 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f84e406a780 0x7f84e41985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ea7e7640 1 --2- 172.21.15.28:0/2950149127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84e4069dc0 0x7f84e4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ea7e7640 1 -- 172.21.15.28:0/2950149127 >> 172.21.15.28:0/2950149127 conn(0x7f84e40776a0 msgr2=0x7f84e40fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ea7e7640 1 -- 172.21.15.28:0/2950149127 shutdown_connections 2024-04-25T01:40:21.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.741+0000 7f84ea7e7640 1 -- 172.21.15.28:0/2950149127 wait complete. 2024-04-25T01:40:21.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:40:21.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.856+0000 7f28b9112640 1 -- 172.21.15.28:0/643491887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28b41020c0 msgr2=0x7f28b410e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.856+0000 7f28b9112640 1 --2- 172.21.15.28:0/643491887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28b41020c0 0x7f28b410e5d0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f28ac009cd0 tx=0x7f28ac02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.856+0000 7f28b9112640 1 -- 172.21.15.28:0/643491887 shutdown_connections 2024-04-25T01:40:21.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.856+0000 7f28b9112640 1 --2- 172.21.15.28:0/643491887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28b41020c0 0x7f28b410e5d0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.856+0000 7f28b9112640 1 --2- 172.21.15.28:0/643491887 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28b4101700 0x7f28b4101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.856+0000 7f28b9112640 1 --2- 172.21.15.28:0/643491887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b4107710 0x7f28b4107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.856+0000 7f28b9112640 1 -- 172.21.15.28:0/643491887 >> 172.21.15.28:0/643491887 conn(0x7f28b40fd1b0 msgr2=0x7f28b40ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.856+0000 7f28b9112640 1 -- 172.21.15.28:0/643491887 shutdown_connections 2024-04-25T01:40:21.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.856+0000 7f28b9112640 1 -- 172.21.15.28:0/643491887 wait complete. 2024-04-25T01:40:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.857+0000 7f28b9112640 1 Processor -- start 2024-04-25T01:40:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.857+0000 7f28b9112640 1 -- start start 2024-04-25T01:40:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b9112640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28b4101700 0x7f28b4197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b9112640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b41020c0 0x7f28b41984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b9112640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28b4107710 0x7f28b419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b9112640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28b4111b20 con 0x7f28b4107710 2024-04-25T01:40:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b9112640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f28b41119a0 con 0x7f28b41020c0 2024-04-25T01:40:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b9112640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f28b4111ca0 con 0x7f28b4101700 2024-04-25T01:40:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28b4107710 0x7f28b419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28b4107710 0x7f28b419c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34778/0 (socket says 172.21.15.28:34778) 2024-04-25T01:40:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b359e640 1 -- 172.21.15.28:0/1302365929 learned_addr learned my addr 172.21.15.28:0/1302365929 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b259c640 1 --2- 172.21.15.28:0/1302365929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b41020c0 0x7f28b41984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b2d9d640 1 --2- 172.21.15.28:0/1302365929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28b4101700 0x7f28b4197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b359e640 1 -- 172.21.15.28:0/1302365929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28b4101700 msgr2=0x7f28b4197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b359e640 1 --2- 172.21.15.28:0/1302365929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28b4101700 0x7f28b4197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b359e640 1 -- 172.21.15.28:0/1302365929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b41020c0 msgr2=0x7f28b41984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b359e640 1 --2- 172.21.15.28:0/1302365929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b41020c0 0x7f28b41984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b359e640 1 -- 172.21.15.28:0/1302365929 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b419cf70 con 0x7f28b4107710 2024-04-25T01:40:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b259c640 1 --2- 172.21.15.28:0/1302365929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b41020c0 0x7f28b41984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b359e640 1 --2- 172.21.15.28:0/1302365929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28b4107710 0x7f28b419c840 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f28ac005cc0 tx=0x7f28ac032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f289bfff640 1 -- 172.21.15.28:0/1302365929 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ac004040 con 0x7f28b4107710 2024-04-25T01:40:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f289bfff640 1 -- 172.21.15.28:0/1302365929 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28ac0046c0 con 0x7f28b4107710 2024-04-25T01:40:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f289bfff640 1 -- 172.21.15.28:0/1302365929 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ac030e00 con 0x7f28b4107710 2024-04-25T01:40:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b9112640 1 -- 172.21.15.28:0/1302365929 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28b419d200 con 0x7f28b4107710 2024-04-25T01:40:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.858+0000 7f28b9112640 1 -- 172.21.15.28:0/1302365929 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28b41a4af0 con 0x7f28b4107710 2024-04-25T01:40:21.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.861+0000 7f289bfff640 1 -- 172.21.15.28:0/1302365929 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f28ac0305a0 con 0x7f28b4107710 2024-04-25T01:40:21.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.861+0000 7f28bbe16740 1 Processor -- start 2024-04-25T01:40:21.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.861+0000 7f28bbe16740 1 -- start start 2024-04-25T01:40:21.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.861+0000 7f28bbe16740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea69064580 0x55ea69064960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.861+0000 7f28bbe16740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea69064ea0 0x55ea6906f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.861+0000 7f28bbe16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea6906f570 0x55ea69071960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.861+0000 7f289bfff640 1 --2- 172.21.15.28:0/1302365929 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f289c0a2d00 0x7f289c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.861+0000 7f28bbe16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ea68fc9670 con 0x55ea6906f570 2024-04-25T01:40:21.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.861+0000 7f28bbe16740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ea68f71970 con 0x55ea69064ea0 2024-04-25T01:40:21.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.861+0000 7f28bbe16740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ea68fc5630 con 0x55ea69064580 2024-04-25T01:40:21.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.862+0000 7f28b259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea6906f570 0x55ea69071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.862+0000 7f28b359e640 1 --2- 172.21.15.28:0/1302365929 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f289c0a2d00 0x7f289c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.862+0000 7f28b259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea6906f570 0x55ea69071960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34780/0 (socket says 172.21.15.28:34780) 2024-04-25T01:40:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.862+0000 7f28b259c640 1 -- 172.21.15.28:0/1758068429 learned_addr learned my addr 172.21.15.28:0/1758068429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.862+0000 7f289bfff640 1 -- 172.21.15.28:0/1302365929 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f28ac02e030 con 0x7f28b4107710 2024-04-25T01:40:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.862+0000 7f28b2d9d640 1 --2- 172.21.15.28:0/1758068429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea69064ea0 0x55ea6906f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.862+0000 7f28b259c640 1 --2- 172.21.15.28:0/1758068429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea69064580 0x55ea69064960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.862+0000 7f28b259c640 1 -- 172.21.15.28:0/1758068429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea69064580 msgr2=0x55ea69064960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.862+0000 7f28b259c640 1 --2- 172.21.15.28:0/1758068429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea69064580 0x55ea69064960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.862+0000 7f28b259c640 1 -- 172.21.15.28:0/1758068429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea69064ea0 msgr2=0x55ea6906f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.862+0000 7f28b259c640 1 --2- 172.21.15.28:0/1758068429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea69064ea0 0x55ea6906f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.862+0000 7f28b259c640 1 -- 172.21.15.28:0/1758068429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea68f95400 con 0x55ea6906f570 2024-04-25T01:40:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.862+0000 7f28b359e640 1 --2- 172.21.15.28:0/1302365929 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f289c0a2d00 0x7f289c0a51c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f28ac0ee510 tx=0x7f28ac097040 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.862+0000 7f28b259c640 1 --2- 172.21.15.28:0/1758068429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea6906f570 0x55ea69071960 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f28a8012820 tx=0x7f28a8009720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbbe389b1c287e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.862+0000 7f28997fa640 1 -- 172.21.15.28:0/1758068429 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a8012e60 con 0x55ea6906f570 2024-04-25T01:40:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.862+0000 7f28997fa640 1 -- 172.21.15.28:0/1758068429 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28a8010cb0 con 0x55ea6906f570 2024-04-25T01:40:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.862+0000 7f28997fa640 1 -- 172.21.15.28:0/1758068429 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a801ebc0 con 0x55ea6906f570 2024-04-25T01:40:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.863+0000 7f28bbe16740 1 -- 172.21.15.28:0/1758068429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea6906f570 msgr2=0x55ea69071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.863+0000 7f28bbe16740 1 --2- 172.21.15.28:0/1758068429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea6906f570 0x55ea69071960 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f28a8012820 tx=0x7f28a8009720 comp rx=0 tx=0).stop 2024-04-25T01:40:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.863+0000 7f28bbe16740 1 -- 172.21.15.28:0/1758068429 shutdown_connections 2024-04-25T01:40:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.863+0000 7f28bbe16740 1 --2- 172.21.15.28:0/1758068429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea6906f570 0x55ea69071960 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.863+0000 7f28bbe16740 1 --2- 172.21.15.28:0/1758068429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea69064ea0 0x55ea6906f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.863+0000 7f28bbe16740 1 --2- 172.21.15.28:0/1758068429 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea69064580 0x55ea69064960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.863+0000 7f28bbe16740 1 -- 172.21.15.28:0/1758068429 >> 172.21.15.28:0/1758068429 conn(0x55ea690608a0 msgr2=0x55ea69060c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.863+0000 7f28bbe16740 1 -- 172.21.15.28:0/1758068429 shutdown_connections 2024-04-25T01:40:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.863+0000 7f28bbe16740 1 -- 172.21.15.28:0/1758068429 wait complete. 2024-04-25T01:40:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.863+0000 7f28bbe16740 1 -- start start 2024-04-25T01:40:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.863+0000 7f28bbe16740 1 Processor -- start 2024-04-25T01:40:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.863+0000 7f28bbe16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064580 0x55ea6906eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f28bbe16740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea69064ea0 0x55ea6906c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f28bbe16740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea6906f570 0x55ea6906c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f28bbe16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ea69072360 con 0x55ea69064580 2024-04-25T01:40:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f28bbe16740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ea68f85050 con 0x55ea6906f570 2024-04-25T01:40:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f28bbe16740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ea690724e0 con 0x55ea69064ea0 2024-04-25T01:40:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f28bbe16740 10 client.0 _ll_register_callbacks cb 0x55ea68cdcbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f28b259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064580 0x55ea6906eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f28b259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064580 0x55ea6906eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34796/0 (socket says 172.21.15.28:34796) 2024-04-25T01:40:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f28b259c640 1 -- 172.21.15.28:0/3805791990 learned_addr learned my addr 172.21.15.28:0/3805791990 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f28b2d9d640 1 --2- 172.21.15.28:0/3805791990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea69064ea0 0x55ea6906c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f28b259c640 1 --2- 172.21.15.28:0/3805791990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea6906f570 0x55ea6906c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f28b259c640 1 -- 172.21.15.28:0/3805791990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea69064ea0 msgr2=0x55ea6906c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f28b259c640 1 --2- 172.21.15.28:0/3805791990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea69064ea0 0x55ea6906c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f28b259c640 1 -- 172.21.15.28:0/3805791990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea6906f570 msgr2=0x55ea6906c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f28b259c640 1 --2- 172.21.15.28:0/3805791990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea6906f570 0x55ea6906c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f28b259c640 1 -- 172.21.15.28:0/3805791990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea68f56dc0 con 0x55ea69064580 2024-04-25T01:40:21.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f28b259c640 1 --2- 172.21.15.28:0/3805791990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064580 0x55ea6906eec0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f28a8011010 tx=0x7f28a8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b09541fbebebc92a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f2896ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:21.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f2896ff5640 1 -- 172.21.15.28:0/3805791990 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a8012e60 con 0x55ea69064580 2024-04-25T01:40:21.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f2896ff5640 1 -- 172.21.15.28:0/3805791990 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f28a8005050 con 0x55ea69064580 2024-04-25T01:40:21.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.864+0000 7f28bbe16740 1 -- 172.21.15.28:0/3805791990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28a8010cb0 con 0x55ea69064580 2024-04-25T01:40:21.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.865+0000 7f2896ff5640 1 -- 172.21.15.28:0/3805791990 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f28a8036050 con 0x55ea69064580 2024-04-25T01:40:21.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.865+0000 7f28bbe16740 10 client.18231 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:21.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.865+0000 7f28bbe16740 1 -- 172.21.15.28:0/3805791990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ea68cdd3f0 con 0x55ea69064580 2024-04-25T01:40:21.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.865+0000 7f2896ff5640 1 -- 172.21.15.28:0/3805791990 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28a8010cb0 con 0x55ea69064580 2024-04-25T01:40:21.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.865+0000 7f2896ff5640 1 -- 172.21.15.28:0/3805791990 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f28a8004c10 con 0x55ea69064580 2024-04-25T01:40:21.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.865+0000 7f28bbe16740 10 client.18231 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:21.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.865+0000 7f28bbe16740 10 client.18231 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:21.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.865+0000 7f28bbe16740 20 client.18231 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:21.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.865+0000 7f28bbe16740 1 --2- 172.21.15.28:0/3805791990 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ea6907f700 0x55ea69081af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.865+0000 7f28bbe16740 4 client.18231 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:21.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.865+0000 7f28bbe16740 1 -- 172.21.15.28:0/3805791990 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea68fd6bb0 con 0x55ea6907f700 2024-04-25T01:40:21.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.865+0000 7f28b2d9d640 1 --2- 172.21.15.28:0/3805791990 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ea6907f700 0x55ea69081af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.866+0000 7f28b2d9d640 1 --2- 172.21.15.28:0/3805791990 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ea6907f700 0x55ea69081af0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24cbe780fa689755 server_cookie=4eecb5bf716edf69 in_seq=0 out_seq=0 2024-04-25T01:40:21.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.866+0000 7f2896ff5640 10 client.18231 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:21.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.866+0000 7f2896ff5640 1 -- 172.21.15.28:0/3805791990 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ea68fd6bb0 con 0x55ea6907f700 2024-04-25T01:40:21.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.866+0000 7f2896ff5640 10 client.18231 handle_command_reply: tid=0 2024-04-25T01:40:21.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.866+0000 7f28bbe16740 1 client.18231 shutdown 2024-04-25T01:40:21.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.866+0000 7f28bbe16740 1 -- 172.21.15.28:0/3805791990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064580 msgr2=0x55ea6906eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.866+0000 7f28bbe16740 1 --2- 172.21.15.28:0/3805791990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064580 0x55ea6906eec0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f28a8011010 tx=0x7f28a8001e70 comp rx=0 tx=0).stop 2024-04-25T01:40:21.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.867+0000 7f28bbe16740 1 -- 172.21.15.28:0/3805791990 shutdown_connections 2024-04-25T01:40:21.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.867+0000 7f28bbe16740 1 --2- 172.21.15.28:0/3805791990 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ea6907f700 0x55ea69081af0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.867+0000 7f28bbe16740 1 --2- 172.21.15.28:0/3805791990 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea6906f570 0x55ea6906c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.867+0000 7f2896ff5640 0 client.18231 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:21.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.867+0000 7f28bbe16740 1 --2- 172.21.15.28:0/3805791990 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea69064ea0 0x55ea6906c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.867+0000 7f28bbe16740 1 --2- 172.21.15.28:0/3805791990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064580 0x55ea6906eec0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.867+0000 7f28bbe16740 1 -- 172.21.15.28:0/3805791990 >> 172.21.15.28:0/3805791990 conn(0x55ea690608a0 msgr2=0x55ea6905fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.867+0000 7f28bbe16740 1 -- 172.21.15.28:0/3805791990 shutdown_connections 2024-04-25T01:40:21.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.867+0000 7f28bbe16740 1 -- 172.21.15.28:0/3805791990 wait complete. 2024-04-25T01:40:21.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.867+0000 7f28bbe16740 20 client.18231 trim_cache size 0 max 16384 2024-04-25T01:40:21.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.880+0000 7f28bbe16740 1 Processor -- start 2024-04-25T01:40:21.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.880+0000 7f28bbe16740 1 -- start start 2024-04-25T01:40:21.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.880+0000 7f28bbe16740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea69064580 0x55ea69066880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.880+0000 7f28bbe16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064ea0 0x55ea6906e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.880+0000 7f28bbe16740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea6906f570 0x55ea69087120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.880+0000 7f28bbe16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28b41aeb10 con 0x55ea69064ea0 2024-04-25T01:40:21.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.880+0000 7f28bbe16740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f28b41b0650 con 0x55ea69064580 2024-04-25T01:40:21.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.880+0000 7f28bbe16740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f28b41af1e0 con 0x55ea6906f570 2024-04-25T01:40:21.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.880+0000 7f28b2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064ea0 0x55ea6906e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.880+0000 7f28b2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064ea0 0x55ea6906e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34806/0 (socket says 172.21.15.28:34806) 2024-04-25T01:40:21.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.880+0000 7f28b2d9d640 1 -- 172.21.15.28:0/3220298051 learned_addr learned my addr 172.21.15.28:0/3220298051 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.880+0000 7f28b259c640 1 --2- 172.21.15.28:0/3220298051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea69064580 0x55ea69066880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.880+0000 7f28b259c640 1 --2- 172.21.15.28:0/3220298051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea6906f570 0x55ea69087120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.880+0000 7f28b2d9d640 1 -- 172.21.15.28:0/3220298051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea6906f570 msgr2=0x55ea69087120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.880+0000 7f28b2d9d640 1 --2- 172.21.15.28:0/3220298051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea6906f570 0x55ea69087120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.880+0000 7f28b2d9d640 1 -- 172.21.15.28:0/3220298051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea69064580 msgr2=0x55ea69066880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.880+0000 7f28b2d9d640 1 --2- 172.21.15.28:0/3220298051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea69064580 0x55ea69066880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.880+0000 7f28b2d9d640 1 -- 172.21.15.28:0/3220298051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea68da77d0 con 0x55ea69064ea0 2024-04-25T01:40:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.881+0000 7f28b259c640 1 --2- 172.21.15.28:0/3220298051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea69064580 0x55ea69066880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.881+0000 7f28b2d9d640 1 --2- 172.21.15.28:0/3220298051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064ea0 0x55ea6906e9f0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f28a4005e10 tx=0x7f28a400cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f8c24a3b544a08c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.881+0000 7f2898ff9640 1 -- 172.21.15.28:0/3220298051 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a4003cc0 con 0x55ea69064ea0 2024-04-25T01:40:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.881+0000 7f2898ff9640 1 -- 172.21.15.28:0/3220298051 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28a4003e60 con 0x55ea69064ea0 2024-04-25T01:40:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.881+0000 7f2898ff9640 1 -- 172.21.15.28:0/3220298051 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a4010d30 con 0x55ea69064ea0 2024-04-25T01:40:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.881+0000 7f28bbe16740 1 -- 172.21.15.28:0/3220298051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064ea0 msgr2=0x55ea6906e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.881+0000 7f28bbe16740 1 --2- 172.21.15.28:0/3220298051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064ea0 0x55ea6906e9f0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f28a4005e10 tx=0x7f28a400cd20 comp rx=0 tx=0).stop 2024-04-25T01:40:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.881+0000 7f28bbe16740 1 -- 172.21.15.28:0/3220298051 shutdown_connections 2024-04-25T01:40:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.881+0000 7f28bbe16740 1 --2- 172.21.15.28:0/3220298051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea6906f570 0x55ea69087120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.881+0000 7f28bbe16740 1 --2- 172.21.15.28:0/3220298051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064ea0 0x55ea6906e9f0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.881+0000 7f28bbe16740 1 --2- 172.21.15.28:0/3220298051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea69064580 0x55ea69066880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.881+0000 7f28bbe16740 1 -- 172.21.15.28:0/3220298051 >> 172.21.15.28:0/3220298051 conn(0x55ea690608a0 msgr2=0x55ea6906b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.881+0000 7f28bbe16740 1 -- 172.21.15.28:0/3220298051 shutdown_connections 2024-04-25T01:40:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.881+0000 7f28bbe16740 1 -- 172.21.15.28:0/3220298051 wait complete. 2024-04-25T01:40:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.882+0000 7f28bbe16740 1 -- start start 2024-04-25T01:40:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.882+0000 7f28bbe16740 1 Processor -- start 2024-04-25T01:40:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.882+0000 7f28bbe16740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea69064580 0x55ea69068cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.882+0000 7f28bbe16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064ea0 0x55ea69069200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.882+0000 7f28bbe16740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea6906f570 0x55ea69069740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.882+0000 7f28bbe16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28b41aff80 con 0x55ea69064ea0 2024-04-25T01:40:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.882+0000 7f28bbe16740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f28b41af8b0 con 0x55ea69064580 2024-04-25T01:40:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.882+0000 7f28bbe16740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f28b41acb10 con 0x55ea6906f570 2024-04-25T01:40:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.882+0000 7f28bbe16740 10 client.0 _ll_register_callbacks cb 0x7f28a400c2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.882+0000 7f28b2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064ea0 0x55ea69069200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.882+0000 7f28b2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064ea0 0x55ea69069200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34808/0 (socket says 172.21.15.28:34808) 2024-04-25T01:40:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.882+0000 7f28b2d9d640 1 -- 172.21.15.28:0/2267340544 learned_addr learned my addr 172.21.15.28:0/2267340544 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.882+0000 7f28b259c640 1 --2- 172.21.15.28:0/2267340544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea69064580 0x55ea69068cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.882+0000 7f28b2d9d640 1 -- 172.21.15.28:0/2267340544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea6906f570 msgr2=0x55ea69069740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.882+0000 7f28b2d9d640 1 --2- 172.21.15.28:0/2267340544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea6906f570 0x55ea69069740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.882+0000 7f28b2d9d640 1 -- 172.21.15.28:0/2267340544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea69064580 msgr2=0x55ea69068cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.882+0000 7f28b2d9d640 1 --2- 172.21.15.28:0/2267340544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea69064580 0x55ea69068cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.882+0000 7f28b2d9d640 1 -- 172.21.15.28:0/2267340544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea690e4990 con 0x55ea69064ea0 2024-04-25T01:40:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.883+0000 7f28b2d9d640 1 --2- 172.21.15.28:0/2267340544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064ea0 0x55ea69069200 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f28a400c780 tx=0x7f28a4003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f2c4140282f3fc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.883+0000 7f2896ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.883+0000 7f2896ff5640 1 -- 172.21.15.28:0/2267340544 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a4010cb0 con 0x55ea69064ea0 2024-04-25T01:40:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.883+0000 7f2896ff5640 1 -- 172.21.15.28:0/2267340544 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f28a401c470 con 0x55ea69064ea0 2024-04-25T01:40:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.883+0000 7f28bbe16740 1 -- 172.21.15.28:0/2267340544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28a4003e60 con 0x55ea69064ea0 2024-04-25T01:40:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.883+0000 7f2896ff5640 1 -- 172.21.15.28:0/2267340544 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f28a4038050 con 0x55ea69064ea0 2024-04-25T01:40:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.883+0000 7f28bbe16740 10 client.18237 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.883+0000 7f28bbe16740 1 -- 172.21.15.28:0/2267340544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ea690e5ad0 con 0x55ea69064ea0 2024-04-25T01:40:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.883+0000 7f2896ff5640 1 -- 172.21.15.28:0/2267340544 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28a4003e60 con 0x55ea69064ea0 2024-04-25T01:40:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.883+0000 7f2896ff5640 1 -- 172.21.15.28:0/2267340544 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f28a40046c0 con 0x55ea69064ea0 2024-04-25T01:40:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.883+0000 7f28bbe16740 10 client.18237 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.883+0000 7f28bbe16740 10 client.18237 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.883+0000 7f28bbe16740 20 client.18237 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.883+0000 7f28bbe16740 1 --2- 172.21.15.28:0/2267340544 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ea690e61c0 0x55ea690e85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.883+0000 7f28bbe16740 4 client.18237 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.883+0000 7f28bbe16740 1 -- 172.21.15.28:0/2267340544 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ea68fd0f00 con 0x55ea690e61c0 2024-04-25T01:40:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.883+0000 7f28b259c640 1 --2- 172.21.15.28:0/2267340544 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ea690e61c0 0x55ea690e85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.884+0000 7f2896ff5640 10 client.18237 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.884+0000 7f28b259c640 1 --2- 172.21.15.28:0/2267340544 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ea690e61c0 0x55ea690e85e0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a4a47563bfb5621 server_cookie=29b95c5569d51f50 in_seq=0 out_seq=0 2024-04-25T01:40:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.884+0000 7f2896ff5640 1 -- 172.21.15.28:0/2267340544 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ea68fd0f00 con 0x55ea690e61c0 2024-04-25T01:40:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.884+0000 7f2896ff5640 10 client.18237 handle_command_reply: tid=0 2024-04-25T01:40:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.884+0000 7f28bbe16740 1 client.18237 shutdown 2024-04-25T01:40:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.884+0000 7f28bbe16740 1 -- 172.21.15.28:0/2267340544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064ea0 msgr2=0x55ea69069200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.884+0000 7f28bbe16740 1 --2- 172.21.15.28:0/2267340544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064ea0 0x55ea69069200 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f28a400c780 tx=0x7f28a4003d60 comp rx=0 tx=0).stop 2024-04-25T01:40:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.884+0000 7f28bbe16740 1 -- 172.21.15.28:0/2267340544 shutdown_connections 2024-04-25T01:40:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.884+0000 7f28bbe16740 1 --2- 172.21.15.28:0/2267340544 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ea690e61c0 0x55ea690e85e0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.884+0000 7f28bbe16740 1 --2- 172.21.15.28:0/2267340544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ea6906f570 0x55ea69069740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.884+0000 7f2896ff5640 0 client.18237 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.884+0000 7f28bbe16740 1 --2- 172.21.15.28:0/2267340544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ea69064ea0 0x55ea69069200 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.884+0000 7f28bbe16740 1 --2- 172.21.15.28:0/2267340544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ea69064580 0x55ea69068cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.884+0000 7f28bbe16740 1 -- 172.21.15.28:0/2267340544 >> 172.21.15.28:0/2267340544 conn(0x55ea690608a0 msgr2=0x55ea6905f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.884+0000 7f28bbe16740 1 -- 172.21.15.28:0/2267340544 shutdown_connections 2024-04-25T01:40:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.884+0000 7f28bbe16740 1 -- 172.21.15.28:0/2267340544 wait complete. 2024-04-25T01:40:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.884+0000 7f28bbe16740 20 client.18237 trim_cache size 0 max 16384 2024-04-25T01:40:21.876 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:21.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.885+0000 7f28b9112640 1 -- 172.21.15.28:0/1302365929 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f289c0a2d00 msgr2=0x7f289c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.885+0000 7f28b9112640 1 --2- 172.21.15.28:0/1302365929 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f289c0a2d00 0x7f289c0a51c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f28ac0ee510 tx=0x7f28ac097040 comp rx=0 tx=0).stop 2024-04-25T01:40:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.885+0000 7f28b9112640 1 -- 172.21.15.28:0/1302365929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28b4107710 msgr2=0x7f28b419c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.885+0000 7f28b9112640 1 --2- 172.21.15.28:0/1302365929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28b4107710 0x7f28b419c840 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f28ac005cc0 tx=0x7f28ac032f40 comp rx=0 tx=0).stop 2024-04-25T01:40:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.885+0000 7f28b9112640 1 -- 172.21.15.28:0/1302365929 shutdown_connections 2024-04-25T01:40:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.885+0000 7f28b9112640 1 --2- 172.21.15.28:0/1302365929 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f289c0a2d00 0x7f289c0a51c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.885+0000 7f28b9112640 1 --2- 172.21.15.28:0/1302365929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28b4107710 0x7f28b419c840 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.885+0000 7f28b9112640 1 --2- 172.21.15.28:0/1302365929 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f28b41020c0 0x7f28b41984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.885+0000 7f28b9112640 1 --2- 172.21.15.28:0/1302365929 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f28b4101700 0x7f28b4197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.885+0000 7f28b9112640 1 -- 172.21.15.28:0/1302365929 >> 172.21.15.28:0/1302365929 conn(0x7f28b40fd1b0 msgr2=0x7f28b40fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.885+0000 7f28b9112640 1 -- 172.21.15.28:0/1302365929 shutdown_connections 2024-04-25T01:40:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:21.885+0000 7f28b9112640 1 -- 172.21.15.28:0/1302365929 wait complete. 2024-04-25T01:40:22.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:21 smithi028 ceph-mon[29390]: pgmap v225: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:40:22.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:21 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/168441658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:40:22.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:21 smithi130 ceph-mon[32702]: pgmap v225: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:40:22.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:21 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/168441658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:40:22.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:21 smithi032 ceph-mon[32461]: pgmap v225: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:40:22.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:21 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/168441658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:40:22.880 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:40:22.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:40:22.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:40:23.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.016+0000 7ff8d3e53640 1 -- 172.21.15.28:0/637005492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8cc1020d0 msgr2=0x7ff8cc10e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.016+0000 7ff8d3e53640 1 --2- 172.21.15.28:0/637005492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8cc1020d0 0x7ff8cc10e5e0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4009930 tx=0x7ff8c402a420 comp rx=0 tx=0).stop 2024-04-25T01:40:23.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.016+0000 7ff8d3e53640 1 -- 172.21.15.28:0/637005492 shutdown_connections 2024-04-25T01:40:23.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.016+0000 7ff8d3e53640 1 --2- 172.21.15.28:0/637005492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8cc1020d0 0x7ff8cc10e5e0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.016+0000 7ff8d3e53640 1 --2- 172.21.15.28:0/637005492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8cc101710 0x7ff8cc101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.016+0000 7ff8d3e53640 1 --2- 172.21.15.28:0/637005492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8cc107720 0x7ff8cc107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.016+0000 7ff8d3e53640 1 -- 172.21.15.28:0/637005492 >> 172.21.15.28:0/637005492 conn(0x7ff8cc078b90 msgr2=0x7ff8cc078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.016+0000 7ff8d3e53640 1 -- 172.21.15.28:0/637005492 shutdown_connections 2024-04-25T01:40:23.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.016+0000 7ff8d3e53640 1 -- 172.21.15.28:0/637005492 wait complete. 2024-04-25T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.017+0000 7ff8d3e53640 1 Processor -- start 2024-04-25T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.017+0000 7ff8d3e53640 1 -- start start 2024-04-25T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.017+0000 7ff8d3e53640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8cc101710 0x7ff8cc198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.017+0000 7ff8d3e53640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8cc1020d0 0x7ff8cc198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.017+0000 7ff8d3e53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8cc107720 0x7ff8cc19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.017+0000 7ff8d3e53640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc111b30 con 0x7ff8cc107720 2024-04-25T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.017+0000 7ff8d3e53640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc1119b0 con 0x7ff8cc1020d0 2024-04-25T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.017+0000 7ff8d3e53640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc111cb0 con 0x7ff8cc101710 2024-04-25T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.017+0000 7ff8d23f0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8cc107720 0x7ff8cc19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.017+0000 7ff8d23f0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8cc107720 0x7ff8cc19c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34814/0 (socket says 172.21.15.28:34814) 2024-04-25T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.017+0000 7ff8d23f0640 1 -- 172.21.15.28:0/3197675594 learned_addr learned my addr 172.21.15.28:0/3197675594 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.017+0000 7ff8d13ee640 1 --2- 172.21.15.28:0/3197675594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8cc1020d0 0x7ff8cc198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.017+0000 7ff8d1bef640 1 --2- 172.21.15.28:0/3197675594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8cc101710 0x7ff8cc198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.017+0000 7ff8d23f0640 1 -- 172.21.15.28:0/3197675594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8cc101710 msgr2=0x7ff8cc198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.017+0000 7ff8d23f0640 1 --2- 172.21.15.28:0/3197675594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8cc101710 0x7ff8cc198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.017+0000 7ff8d23f0640 1 -- 172.21.15.28:0/3197675594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8cc1020d0 msgr2=0x7ff8cc198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.017+0000 7ff8d23f0640 1 --2- 172.21.15.28:0/3197675594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8cc1020d0 0x7ff8cc198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.017+0000 7ff8d23f0640 1 -- 172.21.15.28:0/3197675594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8cc19d040 con 0x7ff8cc107720 2024-04-25T01:40:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.017+0000 7ff8d13ee640 1 --2- 172.21.15.28:0/3197675594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8cc1020d0 0x7ff8cc198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.018+0000 7ff8d23f0640 1 --2- 172.21.15.28:0/3197675594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8cc107720 0x7ff8cc19c910 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4005ab0 tx=0x7ff8c40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.018+0000 7ff8c2ffd640 1 -- 172.21.15.28:0/3197675594 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c403f030 con 0x7ff8cc107720 2024-04-25T01:40:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.018+0000 7ff8c2ffd640 1 -- 172.21.15.28:0/3197675594 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8c4003ce0 con 0x7ff8cc107720 2024-04-25T01:40:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.018+0000 7ff8d3e53640 1 -- 172.21.15.28:0/3197675594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8cc19d2d0 con 0x7ff8cc107720 2024-04-25T01:40:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.018+0000 7ff8c2ffd640 1 -- 172.21.15.28:0/3197675594 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4035070 con 0x7ff8cc107720 2024-04-25T01:40:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.018+0000 7ff8d3e53640 1 -- 172.21.15.28:0/3197675594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8cc1a4bc0 con 0x7ff8cc107720 2024-04-25T01:40:23.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.020+0000 7ff8c2ffd640 1 -- 172.21.15.28:0/3197675594 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff8c4042030 con 0x7ff8cc107720 2024-04-25T01:40:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.020+0000 7ff8d6c33740 1 Processor -- start 2024-04-25T01:40:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.021+0000 7ff8d6c33740 1 -- start start 2024-04-25T01:40:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.021+0000 7ff8d6c33740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf7c980 0x563d3cf7acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.021+0000 7ff8d6c33740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7b290 0x563d3cf7b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.021+0000 7ff8d6c33740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf86010 0x563d3cf88400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.021+0000 7ff8d6c33740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563d3cedf2c0 con 0x563d3cf7b290 2024-04-25T01:40:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.021+0000 7ff8d6c33740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563d3ce87910 con 0x563d3cf86010 2024-04-25T01:40:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.021+0000 7ff8d6c33740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x563d3cedb280 con 0x563d3cf7c980 2024-04-25T01:40:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.021+0000 7ff8d1bef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7b290 0x563d3cf7b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.021+0000 7ff8d1bef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7b290 0x563d3cf7b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34820/0 (socket says 172.21.15.28:34820) 2024-04-25T01:40:23.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.021+0000 7ff8d1bef640 1 -- 172.21.15.28:0/630851012 learned_addr learned my addr 172.21.15.28:0/630851012 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.021+0000 7ff8d13ee640 1 --2- 172.21.15.28:0/630851012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf86010 0x563d3cf88400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.021+0000 7ff8c2ffd640 1 --2- 172.21.15.28:0/3197675594 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8b80a2d80 0x7ff8b80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.021+0000 7ff8d13ee640 1 --2- 172.21.15.28:0/630851012 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf7c980 0x563d3cf7acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.021+0000 7ff8d1bef640 1 -- 172.21.15.28:0/630851012 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf7c980 msgr2=0x563d3cf7acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.021+0000 7ff8d1bef640 1 --2- 172.21.15.28:0/630851012 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf7c980 0x563d3cf7acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.021+0000 7ff8d1bef640 1 -- 172.21.15.28:0/630851012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf86010 msgr2=0x563d3cf88400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.021+0000 7ff8d1bef640 1 --2- 172.21.15.28:0/630851012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf86010 0x563d3cf88400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.021+0000 7ff8d23f0640 1 --2- 172.21.15.28:0/3197675594 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8b80a2d80 0x7ff8b80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.021+0000 7ff8d1bef640 1 -- 172.21.15.28:0/630851012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d3ce98fb0 con 0x563d3cf7b290 2024-04-25T01:40:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.022+0000 7ff8c2ffd640 1 -- 172.21.15.28:0/3197675594 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7ff8c40efcf0 con 0x7ff8cc107720 2024-04-25T01:40:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.022+0000 7ff8d1bef640 1 --2- 172.21.15.28:0/630851012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7b290 0x563d3cf7b710 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8009df0 tx=0x7ff8c800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd5393067e68a95d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.022+0000 7ff8b7fff640 1 -- 172.21.15.28:0/630851012 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8003970 con 0x563d3cf7b290 2024-04-25T01:40:23.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.022+0000 7ff8d23f0640 1 --2- 172.21.15.28:0/3197675594 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8b80a2d80 0x7ff8b80a5240 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40bbb20 tx=0x7ff8c40b12b0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.022+0000 7ff8b7fff640 1 -- 172.21.15.28:0/630851012 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8c8004030 con 0x563d3cf7b290 2024-04-25T01:40:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.022+0000 7ff8b7fff640 1 -- 172.21.15.28:0/630851012 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c80117b0 con 0x563d3cf7b290 2024-04-25T01:40:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.023+0000 7ff8d6c33740 1 -- 172.21.15.28:0/630851012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7b290 msgr2=0x563d3cf7b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.023+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/630851012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7b290 0x563d3cf7b710 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8009df0 tx=0x7ff8c800f730 comp rx=0 tx=0).stop 2024-04-25T01:40:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.023+0000 7ff8d6c33740 1 -- 172.21.15.28:0/630851012 shutdown_connections 2024-04-25T01:40:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.023+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/630851012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf86010 0x563d3cf88400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.023+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/630851012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7b290 0x563d3cf7b710 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.023+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/630851012 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf7c980 0x563d3cf7acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.023+0000 7ff8d6c33740 1 -- 172.21.15.28:0/630851012 >> 172.21.15.28:0/630851012 conn(0x563d3cf43950 msgr2=0x563d3cf777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.023+0000 7ff8d6c33740 1 -- 172.21.15.28:0/630851012 shutdown_connections 2024-04-25T01:40:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.023+0000 7ff8d6c33740 1 -- 172.21.15.28:0/630851012 wait complete. 2024-04-25T01:40:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.023+0000 7ff8d6c33740 1 -- start start 2024-04-25T01:40:23.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.023+0000 7ff8d6c33740 1 Processor -- start 2024-04-25T01:40:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8d6c33740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf7b290 0x563d3cf82f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8d6c33740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7c980 0x563d3cf83460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8d6c33740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf86010 0x563d3cf839a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8d6c33740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563d3cf88ea0 con 0x563d3cf7c980 2024-04-25T01:40:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8d6c33740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563d3ce9ae90 con 0x563d3cf7b290 2024-04-25T01:40:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8d6c33740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x563d3cf89020 con 0x563d3cf86010 2024-04-25T01:40:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8d6c33740 10 client.0 _ll_register_callbacks cb 0x563d3ced6970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8d1bef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7c980 0x563d3cf83460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8d1bef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7c980 0x563d3cf83460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34826/0 (socket says 172.21.15.28:34826) 2024-04-25T01:40:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8d1bef640 1 -- 172.21.15.28:0/2912961892 learned_addr learned my addr 172.21.15.28:0/2912961892 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8d13ee640 1 --2- 172.21.15.28:0/2912961892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf7b290 0x563d3cf82f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8d13ee640 1 --2- 172.21.15.28:0/2912961892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf86010 0x563d3cf839a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8d1bef640 1 -- 172.21.15.28:0/2912961892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf86010 msgr2=0x563d3cf839a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8d1bef640 1 --2- 172.21.15.28:0/2912961892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf86010 0x563d3cf839a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8d1bef640 1 -- 172.21.15.28:0/2912961892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf7b290 msgr2=0x563d3cf82f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8d1bef640 1 --2- 172.21.15.28:0/2912961892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf7b290 0x563d3cf82f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8d1bef640 1 -- 172.21.15.28:0/2912961892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d3ccbb7e0 con 0x563d3cf7c980 2024-04-25T01:40:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8d1bef640 1 --2- 172.21.15.28:0/2912961892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7c980 0x563d3cf83460 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7ff8c800fc30 tx=0x7ff8c8004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46ccc3febf7d3249 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.024+0000 7ff8b57fa640 1 -- 172.21.15.28:0/2912961892 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c802a060 con 0x563d3cf7c980 2024-04-25T01:40:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.025+0000 7ff8b57fa640 1 -- 172.21.15.28:0/2912961892 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7ff8c800fe40 con 0x563d3cf7c980 2024-04-25T01:40:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.025+0000 7ff8d6c33740 1 -- 172.21.15.28:0/2912961892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8c8004030 con 0x563d3cf7c980 2024-04-25T01:40:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.025+0000 7ff8b57fa640 1 -- 172.21.15.28:0/2912961892 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff8c8031050 con 0x563d3cf7c980 2024-04-25T01:40:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.025+0000 7ff8d6c33740 10 client.18249 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.025+0000 7ff8d6c33740 1 -- 172.21.15.28:0/2912961892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563d3cd1bf70 con 0x563d3cf7c980 2024-04-25T01:40:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.025+0000 7ff8b57fa640 1 -- 172.21.15.28:0/2912961892 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8c8004030 con 0x563d3cf7c980 2024-04-25T01:40:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.025+0000 7ff8b57fa640 1 -- 172.21.15.28:0/2912961892 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff8c8011ab0 con 0x563d3cf7c980 2024-04-25T01:40:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.025+0000 7ff8d6c33740 10 client.18249 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.025+0000 7ff8d6c33740 10 client.18249 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.025+0000 7ff8d6c33740 20 client.18249 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.025+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/2912961892 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563d3cf96070 0x563d3cf98460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.025+0000 7ff8d6c33740 4 client.18249 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.025+0000 7ff8d6c33740 1 -- 172.21.15.28:0/2912961892 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563d3ceb35a0 con 0x563d3cf96070 2024-04-25T01:40:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.026+0000 7ff8d13ee640 1 --2- 172.21.15.28:0/2912961892 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563d3cf96070 0x563d3cf98460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.026+0000 7ff8d13ee640 1 --2- 172.21.15.28:0/2912961892 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563d3cf96070 0x563d3cf98460 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f26d2263ec28e7b6 server_cookie=b985c7e815fa4c8d in_seq=0 out_seq=0 2024-04-25T01:40:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.026+0000 7ff8b57fa640 10 client.18249 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.027+0000 7ff8b57fa640 1 -- 172.21.15.28:0/2912961892 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563d3ceb35a0 con 0x563d3cf96070 2024-04-25T01:40:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.027+0000 7ff8b57fa640 10 client.18249 handle_command_reply: tid=0 2024-04-25T01:40:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.027+0000 7ff8d6c33740 1 client.18249 shutdown 2024-04-25T01:40:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.027+0000 7ff8d6c33740 1 -- 172.21.15.28:0/2912961892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7c980 msgr2=0x563d3cf83460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.027+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/2912961892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7c980 0x563d3cf83460 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7ff8c800fc30 tx=0x7ff8c8004410 comp rx=0 tx=0).stop 2024-04-25T01:40:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.027+0000 7ff8d6c33740 1 -- 172.21.15.28:0/2912961892 shutdown_connections 2024-04-25T01:40:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.027+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/2912961892 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563d3cf96070 0x563d3cf98460 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.027+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/2912961892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf86010 0x563d3cf839a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.027+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/2912961892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7c980 0x563d3cf83460 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.027+0000 7ff8b57fa640 0 client.18249 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:23.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.027+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/2912961892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf7b290 0x563d3cf82f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.027+0000 7ff8d6c33740 1 -- 172.21.15.28:0/2912961892 >> 172.21.15.28:0/2912961892 conn(0x563d3cf43950 msgr2=0x563d3cf7d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.027+0000 7ff8d6c33740 1 -- 172.21.15.28:0/2912961892 shutdown_connections 2024-04-25T01:40:23.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.027+0000 7ff8d6c33740 1 -- 172.21.15.28:0/2912961892 wait complete. 2024-04-25T01:40:23.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.027+0000 7ff8d6c33740 20 client.18249 trim_cache size 0 max 16384 2024-04-25T01:40:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.045+0000 7ff8d6c33740 1 Processor -- start 2024-04-25T01:40:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.045+0000 7ff8d6c33740 1 -- start start 2024-04-25T01:40:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.045+0000 7ff8d6c33740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7b290 0x563d3cf91dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.045+0000 7ff8d6c33740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf7c980 0x563d3cf767d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.045+0000 7ff8d6c33740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf86010 0x563d3cf9da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.045+0000 7ff8d6c33740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc1aec00 con 0x563d3cf7b290 2024-04-25T01:40:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.045+0000 7ff8d6c33740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc1b0740 con 0x563d3cf86010 2024-04-25T01:40:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.045+0000 7ff8d6c33740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc1af2d0 con 0x563d3cf7c980 2024-04-25T01:40:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.045+0000 7ff8d13ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7b290 0x563d3cf91dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.045+0000 7ff8d13ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7b290 0x563d3cf91dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34828/0 (socket says 172.21.15.28:34828) 2024-04-25T01:40:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.045+0000 7ff8d13ee640 1 -- 172.21.15.28:0/934805951 learned_addr learned my addr 172.21.15.28:0/934805951 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.045+0000 7ff8d13ee640 1 --2- 172.21.15.28:0/934805951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf86010 0x563d3cf9da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.045+0000 7ff8d13ee640 1 -- 172.21.15.28:0/934805951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf7c980 msgr2=0x563d3cf767d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.045+0000 7ff8d1bef640 1 --2- 172.21.15.28:0/934805951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf7c980 0x563d3cf767d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.045+0000 7ff8d13ee640 1 --2- 172.21.15.28:0/934805951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf7c980 0x563d3cf767d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.045+0000 7ff8d13ee640 1 -- 172.21.15.28:0/934805951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf86010 msgr2=0x563d3cf9da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.045+0000 7ff8d13ee640 1 --2- 172.21.15.28:0/934805951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf86010 0x563d3cf9da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.045+0000 7ff8d13ee640 1 -- 172.21.15.28:0/934805951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d3cbb3170 con 0x563d3cf7b290 2024-04-25T01:40:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.045+0000 7ff8d1bef640 1 --2- 172.21.15.28:0/934805951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf7c980 0x563d3cf767d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.046+0000 7ff8d13ee640 1 --2- 172.21.15.28:0/934805951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7b290 0x563d3cf91dd0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc005d90 tx=0x7ff8bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c70777805a5b1fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.046+0000 7ff8b77fe640 1 -- 172.21.15.28:0/934805951 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc004220 con 0x563d3cf7b290 2024-04-25T01:40:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.046+0000 7ff8b77fe640 1 -- 172.21.15.28:0/934805951 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8bc0043c0 con 0x563d3cf7b290 2024-04-25T01:40:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.046+0000 7ff8b77fe640 1 -- 172.21.15.28:0/934805951 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc0047a0 con 0x563d3cf7b290 2024-04-25T01:40:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.046+0000 7ff8d6c33740 1 -- 172.21.15.28:0/934805951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7b290 msgr2=0x563d3cf91dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.046+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/934805951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7b290 0x563d3cf91dd0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc005d90 tx=0x7ff8bc0046c0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.046+0000 7ff8d6c33740 1 -- 172.21.15.28:0/934805951 shutdown_connections 2024-04-25T01:40:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.046+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/934805951 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf86010 0x563d3cf9da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.046+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/934805951 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf7c980 0x563d3cf767d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.046+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/934805951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7b290 0x563d3cf91dd0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.046+0000 7ff8d6c33740 1 -- 172.21.15.28:0/934805951 >> 172.21.15.28:0/934805951 conn(0x563d3cf43950 msgr2=0x563d3cf760c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.046+0000 7ff8d6c33740 1 -- 172.21.15.28:0/934805951 shutdown_connections 2024-04-25T01:40:23.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.046+0000 7ff8d6c33740 1 -- 172.21.15.28:0/934805951 wait complete. 2024-04-25T01:40:23.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.047+0000 7ff8d6c33740 1 -- start start 2024-04-25T01:40:23.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.047+0000 7ff8d6c33740 1 Processor -- start 2024-04-25T01:40:23.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.047+0000 7ff8d6c33740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf7b290 0x563d3cf87160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.047+0000 7ff8d6c33740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7c980 0x563d3cf876a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.047+0000 7ff8d6c33740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf86010 0x563d3cf8c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.047+0000 7ff8d6c33740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc1b0070 con 0x563d3cf7c980 2024-04-25T01:40:23.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.047+0000 7ff8d6c33740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc1af9a0 con 0x563d3cf7b290 2024-04-25T01:40:23.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.047+0000 7ff8d6c33740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc1acc00 con 0x563d3cf86010 2024-04-25T01:40:23.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.047+0000 7ff8d6c33740 10 client.0 _ll_register_callbacks cb 0x7ff8bc00ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:23.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.047+0000 7ff8d1bef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7c980 0x563d3cf876a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.047+0000 7ff8d1bef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7c980 0x563d3cf876a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34844/0 (socket says 172.21.15.28:34844) 2024-04-25T01:40:23.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.047+0000 7ff8d1bef640 1 -- 172.21.15.28:0/513541126 learned_addr learned my addr 172.21.15.28:0/513541126 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.047+0000 7ff8d13ee640 1 --2- 172.21.15.28:0/513541126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf7b290 0x563d3cf87160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.047+0000 7ff8d13ee640 1 --2- 172.21.15.28:0/513541126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf86010 0x563d3cf8c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.048+0000 7ff8d1bef640 1 -- 172.21.15.28:0/513541126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf86010 msgr2=0x563d3cf8c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.048+0000 7ff8d1bef640 1 --2- 172.21.15.28:0/513541126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf86010 0x563d3cf8c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.048+0000 7ff8d1bef640 1 -- 172.21.15.28:0/513541126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf7b290 msgr2=0x563d3cf87160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.048+0000 7ff8d1bef640 1 --2- 172.21.15.28:0/513541126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf7b290 0x563d3cf87160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.048+0000 7ff8d1bef640 1 -- 172.21.15.28:0/513541126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d3cf8ce70 con 0x563d3cf7c980 2024-04-25T01:40:23.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.048+0000 7ff8d1bef640 1 --2- 172.21.15.28:0/513541126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7c980 0x563d3cf876a0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x563d3cf844f0 tx=0x7ff8c8005d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e19140860804f23f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.048+0000 7ff8b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:23.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.048+0000 7ff8b57fa640 1 -- 172.21.15.28:0/513541126 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c802a040 con 0x563d3cf7c980 2024-04-25T01:40:23.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.048+0000 7ff8b57fa640 1 -- 172.21.15.28:0/513541126 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7ff8c800fc60 con 0x563d3cf7c980 2024-04-25T01:40:23.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.048+0000 7ff8d6c33740 1 -- 172.21.15.28:0/513541126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8bc0043c0 con 0x563d3cf7c980 2024-04-25T01:40:23.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.048+0000 7ff8b57fa640 1 -- 172.21.15.28:0/513541126 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff8c800fe80 con 0x563d3cf7c980 2024-04-25T01:40:23.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.048+0000 7ff8d6c33740 10 client.18255 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:23.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.048+0000 7ff8d6c33740 1 -- 172.21.15.28:0/513541126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563d3cffcab0 con 0x563d3cf7c980 2024-04-25T01:40:23.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.048+0000 7ff8b57fa640 1 -- 172.21.15.28:0/513541126 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8bc0043c0 con 0x563d3cf7c980 2024-04-25T01:40:23.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.048+0000 7ff8b57fa640 1 -- 172.21.15.28:0/513541126 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff8c80114d0 con 0x563d3cf7c980 2024-04-25T01:40:23.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.048+0000 7ff8d6c33740 10 client.18255 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:23.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.049+0000 7ff8d6c33740 10 client.18255 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:23.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.049+0000 7ff8d6c33740 20 client.18255 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:23.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.049+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/513541126 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563d3cffd230 0x563d3cfff650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.049+0000 7ff8d6c33740 4 client.18255 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:40:23.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.049+0000 7ff8d6c33740 1 -- 172.21.15.28:0/513541126 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563d3ceec800 con 0x563d3cffd230 2024-04-25T01:40:23.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.049+0000 7ff8d13ee640 1 --2- 172.21.15.28:0/513541126 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563d3cffd230 0x563d3cfff650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.049+0000 7ff8d13ee640 1 --2- 172.21.15.28:0/513541126 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563d3cffd230 0x563d3cfff650 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a0e813e7455ba8a server_cookie=da9b94c14e1b7529 in_seq=0 out_seq=0 2024-04-25T01:40:23.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.049+0000 7ff8b57fa640 10 client.18255 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:23.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.051+0000 7ff8b57fa640 1 -- 172.21.15.28:0/513541126 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563d3ceec800 con 0x563d3cffd230 2024-04-25T01:40:23.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.051+0000 7ff8b57fa640 10 client.18255 handle_command_reply: tid=0 2024-04-25T01:40:23.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.051+0000 7ff8d6c33740 1 client.18255 shutdown 2024-04-25T01:40:23.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.051+0000 7ff8d6c33740 1 -- 172.21.15.28:0/513541126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7c980 msgr2=0x563d3cf876a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.051+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/513541126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7c980 0x563d3cf876a0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x563d3cf844f0 tx=0x7ff8c8005d10 comp rx=0 tx=0).stop 2024-04-25T01:40:23.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.051+0000 7ff8d6c33740 1 -- 172.21.15.28:0/513541126 shutdown_connections 2024-04-25T01:40:23.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.051+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/513541126 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563d3cffd230 0x563d3cfff650 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.051+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/513541126 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563d3cf86010 0x563d3cf8c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.051+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/513541126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d3cf7c980 0x563d3cf876a0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.051+0000 7ff8d6c33740 1 --2- 172.21.15.28:0/513541126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563d3cf7b290 0x563d3cf87160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.051+0000 7ff8b57fa640 0 client.18255 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:23.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.051+0000 7ff8d6c33740 1 -- 172.21.15.28:0/513541126 >> 172.21.15.28:0/513541126 conn(0x563d3cf43950 msgr2=0x563d3cf76950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.051+0000 7ff8d6c33740 1 -- 172.21.15.28:0/513541126 shutdown_connections 2024-04-25T01:40:23.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.051+0000 7ff8d6c33740 1 -- 172.21.15.28:0/513541126 wait complete. 2024-04-25T01:40:23.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.051+0000 7ff8d6c33740 20 client.18255 trim_cache size 0 max 16384 2024-04-25T01:40:23.042 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:40:23.042 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:40:23.042 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "ed8d6522-5c63-4082-ac96-dc4c0335e963", 2024-04-25T01:40:23.042 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:40:23.042 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:40:23.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.052+0000 7ff8d3e53640 1 -- 172.21.15.28:0/3197675594 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8b80a2d80 msgr2=0x7ff8b80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.052+0000 7ff8d3e53640 1 --2- 172.21.15.28:0/3197675594 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8b80a2d80 0x7ff8b80a5240 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40bbb20 tx=0x7ff8c40b12b0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.052+0000 7ff8d3e53640 1 -- 172.21.15.28:0/3197675594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8cc107720 msgr2=0x7ff8cc19c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.052+0000 7ff8d3e53640 1 --2- 172.21.15.28:0/3197675594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8cc107720 0x7ff8cc19c910 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7ff8c4005ab0 tx=0x7ff8c40042f0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.052+0000 7ff8d3e53640 1 -- 172.21.15.28:0/3197675594 shutdown_connections 2024-04-25T01:40:23.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.052+0000 7ff8d3e53640 1 --2- 172.21.15.28:0/3197675594 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8b80a2d80 0x7ff8b80a5240 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.052+0000 7ff8d3e53640 1 --2- 172.21.15.28:0/3197675594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8cc107720 0x7ff8cc19c910 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.052+0000 7ff8d3e53640 1 --2- 172.21.15.28:0/3197675594 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff8cc1020d0 0x7ff8cc198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.052+0000 7ff8d3e53640 1 --2- 172.21.15.28:0/3197675594 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff8cc101710 0x7ff8cc198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.052+0000 7ff8d3e53640 1 -- 172.21.15.28:0/3197675594 >> 172.21.15.28:0/3197675594 conn(0x7ff8cc078b90 msgr2=0x7ff8cc0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.052+0000 7ff8d3e53640 1 -- 172.21.15.28:0/3197675594 shutdown_connections 2024-04-25T01:40:23.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.052+0000 7ff8d3e53640 1 -- 172.21.15.28:0/3197675594 wait complete. 2024-04-25T01:40:23.048 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:40:23.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:40:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.167+0000 7f67a26cd640 1 -- 172.21.15.28:0/3600287643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c100fd0 msgr2=0x7f679c10d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.167+0000 7f67a26cd640 1 --2- 172.21.15.28:0/3600287643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c100fd0 0x7f679c10d4e0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f6794009cd0 tx=0x7f679402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.168+0000 7f67a26cd640 1 -- 172.21.15.28:0/3600287643 shutdown_connections 2024-04-25T01:40:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.168+0000 7f67a26cd640 1 --2- 172.21.15.28:0/3600287643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c100fd0 0x7f679c10d4e0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.168+0000 7f67a26cd640 1 --2- 172.21.15.28:0/3600287643 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f679c100610 0x7f679c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.168+0000 7f67a26cd640 1 --2- 172.21.15.28:0/3600287643 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f679c1066b0 0x7f679c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.168+0000 7f67a26cd640 1 -- 172.21.15.28:0/3600287643 >> 172.21.15.28:0/3600287643 conn(0x7f679c0769c0 msgr2=0x7f679c076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.168+0000 7f67a26cd640 1 -- 172.21.15.28:0/3600287643 shutdown_connections 2024-04-25T01:40:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.168+0000 7f67a26cd640 1 -- 172.21.15.28:0/3600287643 wait complete. 2024-04-25T01:40:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.168+0000 7f67a26cd640 1 Processor -- start 2024-04-25T01:40:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.168+0000 7f67a26cd640 1 -- start start 2024-04-25T01:40:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.168+0000 7f67a26cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c100610 0x7f679c198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.168+0000 7f67a26cd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f679c100fd0 0x7f679c199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f67a26cd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f679c1066b0 0x7f679c1a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f67a26cd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f679c110a40 con 0x7f679c100610 2024-04-25T01:40:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f67a26cd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f679c1108c0 con 0x7f679c100fd0 2024-04-25T01:40:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f67a26cd640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f679c110bc0 con 0x7f679c1066b0 2024-04-25T01:40:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f679bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c100610 0x7f679c198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f679bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c100610 0x7f679c198d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34862/0 (socket says 172.21.15.28:34862) 2024-04-25T01:40:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f679bfff640 1 -- 172.21.15.28:0/2832984475 learned_addr learned my addr 172.21.15.28:0/2832984475 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f67a0c6a640 1 --2- 172.21.15.28:0/2832984475 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f679c1066b0 0x7f679c1a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f679bfff640 1 -- 172.21.15.28:0/2832984475 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f679c1066b0 msgr2=0x7f679c1a3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f679bfff640 1 --2- 172.21.15.28:0/2832984475 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f679c1066b0 0x7f679c1a3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f679b7fe640 1 --2- 172.21.15.28:0/2832984475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f679c100fd0 0x7f679c199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f679bfff640 1 -- 172.21.15.28:0/2832984475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f679c100fd0 msgr2=0x7f679c199260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f679bfff640 1 --2- 172.21.15.28:0/2832984475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f679c100fd0 0x7f679c199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f679bfff640 1 -- 172.21.15.28:0/2832984475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f679c1a42e0 con 0x7f679c100610 2024-04-25T01:40:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f67a0c6a640 1 --2- 172.21.15.28:0/2832984475 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f679c1066b0 0x7f679c1a3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f679bfff640 1 --2- 172.21.15.28:0/2832984475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c100610 0x7f679c198d20 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f678c00d770 tx=0x7f678c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f67997fa640 1 -- 172.21.15.28:0/2832984475 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f678c017070 con 0x7f679c100610 2024-04-25T01:40:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f67997fa640 1 -- 172.21.15.28:0/2832984475 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f678c009c30 con 0x7f679c100610 2024-04-25T01:40:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f67a26cd640 1 -- 172.21.15.28:0/2832984475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f679c1a44b0 con 0x7f679c100610 2024-04-25T01:40:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f67997fa640 1 -- 172.21.15.28:0/2832984475 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f678c003dd0 con 0x7f679c100610 2024-04-25T01:40:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.169+0000 7f67a26cd640 1 -- 172.21.15.28:0/2832984475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f679c1a4a00 con 0x7f679c100610 2024-04-25T01:40:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.172+0000 7f67997fa640 1 -- 172.21.15.28:0/2832984475 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f678c003fb0 con 0x7f679c100610 2024-04-25T01:40:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.172+0000 7f67a53da740 1 Processor -- start 2024-04-25T01:40:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.172+0000 7f67a53da740 1 -- start start 2024-04-25T01:40:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.172+0000 7f67a53da740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ac530 0x55bf6d0ac910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.172+0000 7f67a53da740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0ace50 0x55bf6d0bb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.172+0000 7f67a53da740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0bba30 0x55bf6d0bde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.172+0000 7f67a53da740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bf6d0116b0 con 0x55bf6d0ac530 2024-04-25T01:40:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.172+0000 7f67a53da740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55bf6cfb9880 con 0x55bf6d0bba30 2024-04-25T01:40:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.172+0000 7f67a53da740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55bf6d00d670 con 0x55bf6d0ace50 2024-04-25T01:40:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.172+0000 7f67997fa640 1 --2- 172.21.15.28:0/2832984475 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f67840a2d00 0x7f67840a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.172+0000 7f67a0c6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ac530 0x55bf6d0ac910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.172+0000 7f67a0c6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ac530 0x55bf6d0ac910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34876/0 (socket says 172.21.15.28:34876) 2024-04-25T01:40:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.172+0000 7f67a0c6a640 1 -- 172.21.15.28:0/989646289 learned_addr learned my addr 172.21.15.28:0/989646289 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.172+0000 7f67997fa640 1 -- 172.21.15.28:0/2832984475 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f678c0cf200 con 0x7f679c100610 2024-04-25T01:40:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.172+0000 7f679b7fe640 1 --2- 172.21.15.28:0/989646289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0bba30 0x55bf6d0bde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.172+0000 7f67a0c6a640 1 --2- 172.21.15.28:0/2832984475 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f67840a2d00 0x7f67840a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.173+0000 7f679bfff640 1 --2- 172.21.15.28:0/989646289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0ace50 0x55bf6d0bb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.173+0000 7f67a0c6a640 1 -- 172.21.15.28:0/989646289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0ace50 msgr2=0x55bf6d0bb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.173+0000 7f67a0c6a640 1 --2- 172.21.15.28:0/989646289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0ace50 0x55bf6d0bb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.173+0000 7f67a0c6a640 1 -- 172.21.15.28:0/989646289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0bba30 msgr2=0x55bf6d0bde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.173+0000 7f67a0c6a640 1 --2- 172.21.15.28:0/989646289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0bba30 0x55bf6d0bde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.173+0000 7f67a0c6a640 1 -- 172.21.15.28:0/989646289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf6cdef7d0 con 0x55bf6d0ac530 2024-04-25T01:40:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.173+0000 7f67a0c6a640 1 --2- 172.21.15.28:0/989646289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ac530 0x55bf6d0ac910 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f6794031970 tx=0x7f67940319a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43dab063057af1d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.173+0000 7f67827fc640 1 -- 172.21.15.28:0/989646289 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6794004410 con 0x55bf6d0ac530 2024-04-25T01:40:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.173+0000 7f67827fc640 1 -- 172.21.15.28:0/989646289 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6794036040 con 0x55bf6d0ac530 2024-04-25T01:40:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.173+0000 7f67827fc640 1 -- 172.21.15.28:0/989646289 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6794030e00 con 0x55bf6d0ac530 2024-04-25T01:40:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.174+0000 7f67a53da740 1 -- 172.21.15.28:0/989646289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ac530 msgr2=0x55bf6d0ac910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.174+0000 7f67a53da740 1 --2- 172.21.15.28:0/989646289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ac530 0x55bf6d0ac910 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f6794031970 tx=0x7f67940319a0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.174+0000 7f67a0c6a640 1 --2- 172.21.15.28:0/2832984475 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f67840a2d00 0x7f67840a51c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f679404b000 tx=0x7f679404c000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.174+0000 7f67a53da740 1 -- 172.21.15.28:0/989646289 shutdown_connections 2024-04-25T01:40:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.174+0000 7f67a53da740 1 --2- 172.21.15.28:0/989646289 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0bba30 0x55bf6d0bde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.174+0000 7f67a53da740 1 --2- 172.21.15.28:0/989646289 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0ace50 0x55bf6d0bb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.174+0000 7f67a53da740 1 --2- 172.21.15.28:0/989646289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ac530 0x55bf6d0ac910 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.174+0000 7f67a53da740 1 -- 172.21.15.28:0/989646289 >> 172.21.15.28:0/989646289 conn(0x55bf6d0a8950 msgr2=0x55bf6d0a8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.174+0000 7f67a53da740 1 -- 172.21.15.28:0/989646289 shutdown_connections 2024-04-25T01:40:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.174+0000 7f67a53da740 1 -- 172.21.15.28:0/989646289 wait complete. 2024-04-25T01:40:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.174+0000 7f67a53da740 1 -- start start 2024-04-25T01:40:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.174+0000 7f67a53da740 1 Processor -- start 2024-04-25T01:40:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f67a53da740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0ac530 0x55bf6d0b8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f67a53da740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ace50 0x55bf6d0b8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f67a53da740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0bba30 0x55bf6d0b9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f67a53da740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bf6d0be7f0 con 0x55bf6d0ace50 2024-04-25T01:40:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f67a53da740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55bf6cfccf50 con 0x55bf6d0ac530 2024-04-25T01:40:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f67a53da740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55bf6d0be970 con 0x55bf6d0bba30 2024-04-25T01:40:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f67a53da740 10 client.0 _ll_register_callbacks cb 0x55bf6d008000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f679b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ace50 0x55bf6d0b8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f679b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ace50 0x55bf6d0b8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34882/0 (socket says 172.21.15.28:34882) 2024-04-25T01:40:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f679b7fe640 1 -- 172.21.15.28:0/3323774962 learned_addr learned my addr 172.21.15.28:0/3323774962 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f679bfff640 1 --2- 172.21.15.28:0/3323774962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0ac530 0x55bf6d0b8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f67a0c6a640 1 --2- 172.21.15.28:0/3323774962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0bba30 0x55bf6d0b9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f679b7fe640 1 -- 172.21.15.28:0/3323774962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0bba30 msgr2=0x55bf6d0b9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f679b7fe640 1 --2- 172.21.15.28:0/3323774962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0bba30 0x55bf6d0b9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f679b7fe640 1 -- 172.21.15.28:0/3323774962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0ac530 msgr2=0x55bf6d0b8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f679b7fe640 1 --2- 172.21.15.28:0/3323774962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0ac530 0x55bf6d0b8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f679b7fe640 1 -- 172.21.15.28:0/3323774962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf6cd253f0 con 0x55bf6d0ace50 2024-04-25T01:40:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f679b7fe640 1 --2- 172.21.15.28:0/3323774962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ace50 0x55bf6d0b8ec0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f6790010820 tx=0x7f6790010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3ea138cc9ad104 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f677b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f677b7fe640 1 -- 172.21.15.28:0/3323774962 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f679000ed20 con 0x55bf6d0ace50 2024-04-25T01:40:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f677b7fe640 1 -- 172.21.15.28:0/3323774962 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f6790005020 con 0x55bf6d0ace50 2024-04-25T01:40:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.175+0000 7f67a53da740 1 -- 172.21.15.28:0/3323774962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6794036040 con 0x55bf6d0ace50 2024-04-25T01:40:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.176+0000 7f677b7fe640 1 -- 172.21.15.28:0/3323774962 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6790023050 con 0x55bf6d0ace50 2024-04-25T01:40:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.176+0000 7f67a53da740 10 client.18267 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.176+0000 7f67a53da740 1 -- 172.21.15.28:0/3323774962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bf6cce50f0 con 0x55bf6d0ace50 2024-04-25T01:40:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.176+0000 7f677b7fe640 1 -- 172.21.15.28:0/3323774962 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6794036040 con 0x55bf6d0ace50 2024-04-25T01:40:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.176+0000 7f677b7fe640 1 -- 172.21.15.28:0/3323774962 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6790018430 con 0x55bf6d0ace50 2024-04-25T01:40:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.176+0000 7f67a53da740 10 client.18267 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.176+0000 7f67a53da740 10 client.18267 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.176+0000 7f67a53da740 20 client.18267 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.176+0000 7f67a53da740 1 --2- 172.21.15.28:0/3323774962 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bf6d0cbcb0 0x55bf6d0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.176+0000 7f67a53da740 4 client.18267 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.176+0000 7f67a53da740 1 -- 172.21.15.28:0/3323774962 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf6d01ebf0 con 0x55bf6d0cbcb0 2024-04-25T01:40:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.176+0000 7f679bfff640 1 --2- 172.21.15.28:0/3323774962 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bf6d0cbcb0 0x55bf6d0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.177+0000 7f677b7fe640 10 client.18267 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.177+0000 7f679bfff640 1 --2- 172.21.15.28:0/3323774962 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bf6d0cbcb0 0x55bf6d0ce130 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a17cad81fad6606a server_cookie=c4ea5b74d88e67a6 in_seq=0 out_seq=0 2024-04-25T01:40:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.177+0000 7f677b7fe640 1 -- 172.21.15.28:0/3323774962 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bf6d01ebf0 con 0x55bf6d0cbcb0 2024-04-25T01:40:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.177+0000 7f677b7fe640 10 client.18267 handle_command_reply: tid=0 2024-04-25T01:40:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.177+0000 7f67a53da740 1 client.18267 shutdown 2024-04-25T01:40:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.177+0000 7f67a53da740 1 -- 172.21.15.28:0/3323774962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ace50 msgr2=0x55bf6d0b8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.177+0000 7f67a53da740 1 --2- 172.21.15.28:0/3323774962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ace50 0x55bf6d0b8ec0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f6790010820 tx=0x7f6790010ce0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.178+0000 7f67a53da740 1 -- 172.21.15.28:0/3323774962 shutdown_connections 2024-04-25T01:40:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.178+0000 7f67a53da740 1 --2- 172.21.15.28:0/3323774962 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bf6d0cbcb0 0x55bf6d0ce130 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.178+0000 7f67a53da740 1 --2- 172.21.15.28:0/3323774962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0bba30 0x55bf6d0b9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.178+0000 7f67a53da740 1 --2- 172.21.15.28:0/3323774962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ace50 0x55bf6d0b8ec0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.178+0000 7f67a53da740 1 --2- 172.21.15.28:0/3323774962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0ac530 0x55bf6d0b8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.178+0000 7f67a53da740 1 -- 172.21.15.28:0/3323774962 >> 172.21.15.28:0/3323774962 conn(0x55bf6d0a8950 msgr2=0x55bf6d0ae780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.178+0000 7f677b7fe640 0 client.18267 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.178+0000 7f67a53da740 1 -- 172.21.15.28:0/3323774962 shutdown_connections 2024-04-25T01:40:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.178+0000 7f67a53da740 1 -- 172.21.15.28:0/3323774962 wait complete. 2024-04-25T01:40:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.178+0000 7f67a53da740 20 client.18267 trim_cache size 0 max 16384 2024-04-25T01:40:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.195+0000 7f67a53da740 1 Processor -- start 2024-04-25T01:40:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.195+0000 7f67a53da740 1 -- start start 2024-04-25T01:40:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.195+0000 7f67a53da740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0ac530 0x55bf6d0c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.195+0000 7f67a53da740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ace50 0x55bf6d0b7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.195+0000 7f67a53da740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0bba30 0x55bf6d0ba7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.195+0000 7f67a53da740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f679c1ac9a0 con 0x55bf6d0ace50 2024-04-25T01:40:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.195+0000 7f67a53da740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f679c1ae4e0 con 0x55bf6d0ac530 2024-04-25T01:40:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.195+0000 7f67a53da740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f679c1ad070 con 0x55bf6d0bba30 2024-04-25T01:40:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.195+0000 7f679b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ace50 0x55bf6d0b7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.195+0000 7f679b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ace50 0x55bf6d0b7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34888/0 (socket says 172.21.15.28:34888) 2024-04-25T01:40:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.195+0000 7f679b7fe640 1 -- 172.21.15.28:0/910614937 learned_addr learned my addr 172.21.15.28:0/910614937 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.196+0000 7f679bfff640 1 --2- 172.21.15.28:0/910614937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0ac530 0x55bf6d0c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.196+0000 7f679b7fe640 1 -- 172.21.15.28:0/910614937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0bba30 msgr2=0x55bf6d0ba7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.196+0000 7f67a0c6a640 1 --2- 172.21.15.28:0/910614937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0bba30 0x55bf6d0ba7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.196+0000 7f679b7fe640 1 --2- 172.21.15.28:0/910614937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0bba30 0x55bf6d0ba7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.196+0000 7f679b7fe640 1 -- 172.21.15.28:0/910614937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0ac530 msgr2=0x55bf6d0c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.196+0000 7f679b7fe640 1 --2- 172.21.15.28:0/910614937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0ac530 0x55bf6d0c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.196+0000 7f679b7fe640 1 -- 172.21.15.28:0/910614937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf6d0a8780 con 0x55bf6d0ace50 2024-04-25T01:40:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.196+0000 7f67a0c6a640 1 --2- 172.21.15.28:0/910614937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0bba30 0x55bf6d0ba7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.196+0000 7f679b7fe640 1 --2- 172.21.15.28:0/910614937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ace50 0x55bf6d0b7ae0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f6790018850 tx=0x7f679000e800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2404cbf35dc0f57c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.196+0000 7f6781ffb640 1 -- 172.21.15.28:0/910614937 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f679002b070 con 0x55bf6d0ace50 2024-04-25T01:40:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.196+0000 7f6781ffb640 1 -- 172.21.15.28:0/910614937 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6790018cc0 con 0x55bf6d0ace50 2024-04-25T01:40:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.196+0000 7f6781ffb640 1 -- 172.21.15.28:0/910614937 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67900052d0 con 0x55bf6d0ace50 2024-04-25T01:40:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.197+0000 7f67a53da740 1 -- 172.21.15.28:0/910614937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ace50 msgr2=0x55bf6d0b7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.197+0000 7f67a53da740 1 --2- 172.21.15.28:0/910614937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ace50 0x55bf6d0b7ae0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f6790018850 tx=0x7f679000e800 comp rx=0 tx=0).stop 2024-04-25T01:40:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.197+0000 7f67a53da740 1 -- 172.21.15.28:0/910614937 shutdown_connections 2024-04-25T01:40:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.197+0000 7f67a53da740 1 --2- 172.21.15.28:0/910614937 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0bba30 0x55bf6d0ba7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.197+0000 7f67a53da740 1 --2- 172.21.15.28:0/910614937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0ace50 0x55bf6d0b7ae0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.197+0000 7f67a53da740 1 --2- 172.21.15.28:0/910614937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0ac530 0x55bf6d0c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.197+0000 7f67a53da740 1 -- 172.21.15.28:0/910614937 >> 172.21.15.28:0/910614937 conn(0x55bf6d0a8950 msgr2=0x55bf6d0ba0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.197+0000 7f67a53da740 1 -- 172.21.15.28:0/910614937 shutdown_connections 2024-04-25T01:40:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.197+0000 7f67a53da740 1 -- 172.21.15.28:0/910614937 wait complete. 2024-04-25T01:40:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.197+0000 7f67a53da740 1 -- start start 2024-04-25T01:40:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.197+0000 7f67a53da740 1 Processor -- start 2024-04-25T01:40:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.197+0000 7f67a53da740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0ac530 0x55bf6d0c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.197+0000 7f67a53da740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0ace50 0x55bf6d0b52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.197+0000 7f67a53da740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0bba30 0x55bf6d0b5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.197+0000 7f67a53da740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f679c1ade10 con 0x55bf6d0bba30 2024-04-25T01:40:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.197+0000 7f67a53da740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f679c1ad740 con 0x55bf6d0ac530 2024-04-25T01:40:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.197+0000 7f67a53da740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f679c1aa9a0 con 0x55bf6d0ace50 2024-04-25T01:40:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.197+0000 7f67a53da740 10 client.0 _ll_register_callbacks cb 0x7f679000b680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.198+0000 7f67a0c6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0bba30 0x55bf6d0b5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.198+0000 7f67a0c6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0bba30 0x55bf6d0b5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34904/0 (socket says 172.21.15.28:34904) 2024-04-25T01:40:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.198+0000 7f67a0c6a640 1 -- 172.21.15.28:0/95832956 learned_addr learned my addr 172.21.15.28:0/95832956 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.198+0000 7f679bfff640 1 --2- 172.21.15.28:0/95832956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0ac530 0x55bf6d0c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.198+0000 7f679b7fe640 1 --2- 172.21.15.28:0/95832956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0ace50 0x55bf6d0b52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.198+0000 7f67a0c6a640 1 -- 172.21.15.28:0/95832956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0ace50 msgr2=0x55bf6d0b52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.198+0000 7f67a0c6a640 1 --2- 172.21.15.28:0/95832956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0ace50 0x55bf6d0b52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.198+0000 7f67a0c6a640 1 -- 172.21.15.28:0/95832956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0ac530 msgr2=0x55bf6d0c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.198+0000 7f67a0c6a640 1 --2- 172.21.15.28:0/95832956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0ac530 0x55bf6d0c3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.198+0000 7f67a0c6a640 1 -- 172.21.15.28:0/95832956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf6d0b5ff0 con 0x55bf6d0bba30 2024-04-25T01:40:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.198+0000 7f67a0c6a640 1 --2- 172.21.15.28:0/95832956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0bba30 0x55bf6d0b5810 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f6794031040 tx=0x7f6794005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a4865f7e51c20c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.198+0000 7f677b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.198+0000 7f677b7fe640 1 -- 172.21.15.28:0/95832956 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6794003e00 con 0x55bf6d0bba30 2024-04-25T01:40:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.198+0000 7f677b7fe640 1 -- 172.21.15.28:0/95832956 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f6794004620 con 0x55bf6d0bba30 2024-04-25T01:40:23.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.198+0000 7f67a53da740 1 -- 172.21.15.28:0/95832956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6790018cc0 con 0x55bf6d0bba30 2024-04-25T01:40:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.198+0000 7f677b7fe640 1 -- 172.21.15.28:0/95832956 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f67940305a0 con 0x55bf6d0bba30 2024-04-25T01:40:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.198+0000 7f67a53da740 10 client.18273 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.198+0000 7f67a53da740 1 -- 172.21.15.28:0/95832956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bf6d132140 con 0x55bf6d0bba30 2024-04-25T01:40:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.199+0000 7f677b7fe640 1 -- 172.21.15.28:0/95832956 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6790018cc0 con 0x55bf6d0bba30 2024-04-25T01:40:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.199+0000 7f677b7fe640 1 -- 172.21.15.28:0/95832956 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f679405b070 con 0x55bf6d0bba30 2024-04-25T01:40:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.199+0000 7f67a53da740 10 client.18273 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.199+0000 7f67a53da740 10 client.18273 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.199+0000 7f67a53da740 20 client.18273 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.199+0000 7f67a53da740 1 --2- 172.21.15.28:0/95832956 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bf6d132420 0x55bf6d134810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.199+0000 7f67a53da740 4 client.18273 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:40:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.199+0000 7f67a53da740 1 -- 172.21.15.28:0/95832956 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bf6d018f40 con 0x55bf6d132420 2024-04-25T01:40:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.199+0000 7f679bfff640 1 --2- 172.21.15.28:0/95832956 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bf6d132420 0x55bf6d134810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.199+0000 7f677b7fe640 10 client.18273 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.199+0000 7f679bfff640 1 --2- 172.21.15.28:0/95832956 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bf6d132420 0x55bf6d134810 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6196f9e277e3b2 server_cookie=a8c2018c0ee1d0c3 in_seq=0 out_seq=0 2024-04-25T01:40:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.200+0000 7f677b7fe640 1 -- 172.21.15.28:0/95832956 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bf6d018f40 con 0x55bf6d132420 2024-04-25T01:40:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.200+0000 7f677b7fe640 10 client.18273 handle_command_reply: tid=0 2024-04-25T01:40:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.200+0000 7f67a53da740 1 client.18273 shutdown 2024-04-25T01:40:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.200+0000 7f67a53da740 1 -- 172.21.15.28:0/95832956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0bba30 msgr2=0x55bf6d0b5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.200+0000 7f67a53da740 1 --2- 172.21.15.28:0/95832956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0bba30 0x55bf6d0b5810 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f6794031040 tx=0x7f6794005da0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.200+0000 7f67a53da740 1 -- 172.21.15.28:0/95832956 shutdown_connections 2024-04-25T01:40:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.200+0000 7f67a53da740 1 --2- 172.21.15.28:0/95832956 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bf6d132420 0x55bf6d134810 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.200+0000 7f67a53da740 1 --2- 172.21.15.28:0/95832956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf6d0bba30 0x55bf6d0b5810 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.200+0000 7f67a53da740 1 --2- 172.21.15.28:0/95832956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf6d0ace50 0x55bf6d0b52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.200+0000 7f677b7fe640 0 client.18273 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.200+0000 7f67a53da740 1 --2- 172.21.15.28:0/95832956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf6d0ac530 0x55bf6d0c3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.200+0000 7f67a53da740 1 -- 172.21.15.28:0/95832956 >> 172.21.15.28:0/95832956 conn(0x55bf6d0a8950 msgr2=0x55bf6d0b91b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.200+0000 7f67a53da740 1 -- 172.21.15.28:0/95832956 shutdown_connections 2024-04-25T01:40:23.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.200+0000 7f67a53da740 1 -- 172.21.15.28:0/95832956 wait complete. 2024-04-25T01:40:23.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.200+0000 7f67a53da740 20 client.18273 trim_cache size 0 max 16384 2024-04-25T01:40:23.192 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:40:23.192 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:40:23.192 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:40:23.192 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:40:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.200+0000 7f67a26cd640 1 -- 172.21.15.28:0/2832984475 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f67840a2d00 msgr2=0x7f67840a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.200+0000 7f67a26cd640 1 --2- 172.21.15.28:0/2832984475 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f67840a2d00 0x7f67840a51c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f679404b000 tx=0x7f679404c000 comp rx=0 tx=0).stop 2024-04-25T01:40:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.200+0000 7f67a26cd640 1 -- 172.21.15.28:0/2832984475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c100610 msgr2=0x7f679c198d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.200+0000 7f67a26cd640 1 --2- 172.21.15.28:0/2832984475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c100610 0x7f679c198d20 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f678c00d770 tx=0x7f678c00dc30 comp rx=0 tx=0).stop 2024-04-25T01:40:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.201+0000 7f67a26cd640 1 -- 172.21.15.28:0/2832984475 shutdown_connections 2024-04-25T01:40:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.201+0000 7f67a26cd640 1 --2- 172.21.15.28:0/2832984475 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f67840a2d00 0x7f67840a51c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.201+0000 7f67a26cd640 1 --2- 172.21.15.28:0/2832984475 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f679c1066b0 0x7f679c1a3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.201+0000 7f67a26cd640 1 --2- 172.21.15.28:0/2832984475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f679c100fd0 0x7f679c199260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.201+0000 7f67a26cd640 1 --2- 172.21.15.28:0/2832984475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c100610 0x7f679c198d20 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.201+0000 7f67a26cd640 1 -- 172.21.15.28:0/2832984475 >> 172.21.15.28:0/2832984475 conn(0x7f679c0769c0 msgr2=0x7f679c0fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.201+0000 7f67a26cd640 1 -- 172.21.15.28:0/2832984475 shutdown_connections 2024-04-25T01:40:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.201+0000 7f67a26cd640 1 -- 172.21.15.28:0/2832984475 wait complete. 2024-04-25T01:40:23.196 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:40:23.196 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:40:23.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.314+0000 7f94ab316640 1 -- 172.21.15.28:0/121856759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94a4100690 msgr2=0x7f94a4100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.314+0000 7f94ab316640 1 --2- 172.21.15.28:0/121856759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94a4100690 0x7f94a4100b10 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f9494009930 tx=0x7f949402a420 comp rx=0 tx=0).stop 2024-04-25T01:40:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.314+0000 7f94ab316640 1 -- 172.21.15.28:0/121856759 shutdown_connections 2024-04-25T01:40:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.314+0000 7f94ab316640 1 --2- 172.21.15.28:0/121856759 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a4101050 0x7f94a410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.314+0000 7f94ab316640 1 --2- 172.21.15.28:0/121856759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94a4100690 0x7f94a4100b10 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.314+0000 7f94ab316640 1 --2- 172.21.15.28:0/121856759 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94a4106650 0x7f94a4106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.314+0000 7f94ab316640 1 -- 172.21.15.28:0/121856759 >> 172.21.15.28:0/121856759 conn(0x7f94a40777a0 msgr2=0x7f94a4077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.314+0000 7f94ab316640 1 -- 172.21.15.28:0/121856759 shutdown_connections 2024-04-25T01:40:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.314+0000 7f94ab316640 1 -- 172.21.15.28:0/121856759 wait complete. 2024-04-25T01:40:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.314+0000 7f94ab316640 1 Processor -- start 2024-04-25T01:40:23.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.314+0000 7f94ab316640 1 -- start start 2024-04-25T01:40:23.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94ab316640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a4100690 0x7f94a40fd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94ab316640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94a4101050 0x7f94a40fdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94ab316640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94a4106650 0x7f94a40fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94ab316640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94a4110a60 con 0x7f94a4101050 2024-04-25T01:40:23.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94ab316640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f94a41108e0 con 0x7f94a4100690 2024-04-25T01:40:23.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94ab316640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f94a4110be0 con 0x7f94a4106650 2024-04-25T01:40:23.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94a88b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94a4101050 0x7f94a40fdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94a88b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94a4101050 0x7f94a40fdb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34914/0 (socket says 172.21.15.28:34914) 2024-04-25T01:40:23.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94a88b1640 1 -- 172.21.15.28:0/2292144613 learned_addr learned my addr 172.21.15.28:0/2292144613 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94a90b2640 1 --2- 172.21.15.28:0/2292144613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a4100690 0x7f94a40fd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94a98b3640 1 --2- 172.21.15.28:0/2292144613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94a4106650 0x7f94a40fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94a88b1640 1 -- 172.21.15.28:0/2292144613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94a4106650 msgr2=0x7f94a40fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94a88b1640 1 --2- 172.21.15.28:0/2292144613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94a4106650 0x7f94a40fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94a88b1640 1 -- 172.21.15.28:0/2292144613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a4100690 msgr2=0x7f94a40fd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94a88b1640 1 --2- 172.21.15.28:0/2292144613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a4100690 0x7f94a40fd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94a88b1640 1 -- 172.21.15.28:0/2292144613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a41a6570 con 0x7f94a4101050 2024-04-25T01:40:23.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94a90b2640 1 --2- 172.21.15.28:0/2292144613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a4100690 0x7f94a40fd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:23.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94a88b1640 1 --2- 172.21.15.28:0/2292144613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94a4101050 0x7f94a40fdb90 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f9494009230 tx=0x7f9494004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f949a7fc640 1 -- 172.21.15.28:0/2292144613 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949403f030 con 0x7f94a4101050 2024-04-25T01:40:23.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f949a7fc640 1 -- 172.21.15.28:0/2292144613 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9494003ad0 con 0x7f94a4101050 2024-04-25T01:40:23.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94ab316640 1 -- 172.21.15.28:0/2292144613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94a41a6800 con 0x7f94a4101050 2024-04-25T01:40:23.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f949a7fc640 1 -- 172.21.15.28:0/2292144613 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9494035070 con 0x7f94a4101050 2024-04-25T01:40:23.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.315+0000 7f94ab316640 1 -- 172.21.15.28:0/2292144613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94a41a6cf0 con 0x7f94a4101050 2024-04-25T01:40:23.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.316+0000 7f94ab316640 1 -- 172.21.15.28:0/2292144613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94a4101d80 con 0x7f94a4101050 2024-04-25T01:40:23.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.317+0000 7f949a7fc640 1 -- 172.21.15.28:0/2292144613 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9494042030 con 0x7f94a4101050 2024-04-25T01:40:23.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.318+0000 7f949a7fc640 1 --2- 172.21.15.28:0/2292144613 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f94900a2d00 0x7f94900a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.318+0000 7f949a7fc640 1 -- 172.21.15.28:0/2292144613 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f94940eff00 con 0x7f94a4101050 2024-04-25T01:40:23.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.318+0000 7f94a90b2640 1 --2- 172.21.15.28:0/2292144613 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f94900a2d00 0x7f94900a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.319+0000 7f94a90b2640 1 --2- 172.21.15.28:0/2292144613 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f94900a2d00 0x7f94900a51c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f94a0005fd0 tx=0x7f94a000b5d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.323+0000 7f949a7fc640 1 -- 172.21.15.28:0/2292144613 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f94940bc730 con 0x7f94a4101050 2024-04-25T01:40:23.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.545+0000 7f94ab316640 1 -- 172.21.15.28:0/2292144613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94a4109be0 con 0x7f94a4101050 2024-04-25T01:40:23.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.546+0000 7f949a7fc640 1 -- 172.21.15.28:0/2292144613 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7f94940bc030 con 0x7f94a4101050 2024-04-25T01:40:23.530 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:40:23.531 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:40:23.531 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:40:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.548+0000 7f94ab316640 1 -- 172.21.15.28:0/2292144613 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f94900a2d00 msgr2=0x7f94900a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.548+0000 7f94ab316640 1 --2- 172.21.15.28:0/2292144613 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f94900a2d00 0x7f94900a51c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f94a0005fd0 tx=0x7f94a000b5d0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.548+0000 7f94ab316640 1 -- 172.21.15.28:0/2292144613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94a4101050 msgr2=0x7f94a40fdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.548+0000 7f94ab316640 1 --2- 172.21.15.28:0/2292144613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94a4101050 0x7f94a40fdb90 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f9494009230 tx=0x7f9494004270 comp rx=0 tx=0).stop 2024-04-25T01:40:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.548+0000 7f94ab316640 1 -- 172.21.15.28:0/2292144613 shutdown_connections 2024-04-25T01:40:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.548+0000 7f94ab316640 1 --2- 172.21.15.28:0/2292144613 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f94900a2d00 0x7f94900a51c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.548+0000 7f94ab316640 1 --2- 172.21.15.28:0/2292144613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f94a4106650 0x7f94a40fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.548+0000 7f94ab316640 1 --2- 172.21.15.28:0/2292144613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94a4101050 0x7f94a40fdb90 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.548+0000 7f94ab316640 1 --2- 172.21.15.28:0/2292144613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a4100690 0x7f94a40fd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.548+0000 7f94ab316640 1 -- 172.21.15.28:0/2292144613 >> 172.21.15.28:0/2292144613 conn(0x7f94a40777a0 msgr2=0x7f94a406a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.548+0000 7f94ab316640 1 -- 172.21.15.28:0/2292144613 shutdown_connections 2024-04-25T01:40:23.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.548+0000 7f94ab316640 1 -- 172.21.15.28:0/2292144613 wait complete. 2024-04-25T01:40:23.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:40:23.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.663+0000 7f9d5e71d640 1 -- 172.21.15.28:0/3503855002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d58069dc0 msgr2=0x7f9d5806a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.663+0000 7f9d5e71d640 1 --2- 172.21.15.28:0/3503855002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d58069dc0 0x7f9d5806a240 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f9d40009930 tx=0x7f9d4002a420 comp rx=0 tx=0).stop 2024-04-25T01:40:23.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.663+0000 7f9d5e71d640 1 -- 172.21.15.28:0/3503855002 shutdown_connections 2024-04-25T01:40:23.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.663+0000 7f9d5e71d640 1 --2- 172.21.15.28:0/3503855002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d5806a780 0x7f9d5810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.663+0000 7f9d5e71d640 1 --2- 172.21.15.28:0/3503855002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d58069dc0 0x7f9d5806a240 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.663+0000 7f9d5e71d640 1 --2- 172.21.15.28:0/3503855002 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d58107730 0x7f9d58107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.663+0000 7f9d5e71d640 1 -- 172.21.15.28:0/3503855002 >> 172.21.15.28:0/3503855002 conn(0x7f9d580776a0 msgr2=0x7f9d58077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.663+0000 7f9d5e71d640 1 -- 172.21.15.28:0/3503855002 shutdown_connections 2024-04-25T01:40:23.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.663+0000 7f9d5e71d640 1 -- 172.21.15.28:0/3503855002 wait complete. 2024-04-25T01:40:23.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.664+0000 7f9d5e71d640 1 Processor -- start 2024-04-25T01:40:23.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.664+0000 7f9d5e71d640 1 -- start start 2024-04-25T01:40:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.664+0000 7f9d5e71d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d58069dc0 0x7f9d58198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.664+0000 7f9d5e71d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d5806a780 0x7f9d581985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.664+0000 7f9d5e71d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d58107730 0x7f9d5819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.664+0000 7f9d5e71d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d58111b00 con 0x7f9d58107730 2024-04-25T01:40:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.664+0000 7f9d5e71d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9d58111980 con 0x7f9d5806a780 2024-04-25T01:40:23.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.664+0000 7f9d5e71d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9d58111c80 con 0x7f9d58069dc0 2024-04-25T01:40:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.664+0000 7f9d5ccba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d58107730 0x7f9d5819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.664+0000 7f9d5ccba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d58107730 0x7f9d5819c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34930/0 (socket says 172.21.15.28:34930) 2024-04-25T01:40:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.664+0000 7f9d5ccba640 1 -- 172.21.15.28:0/3577807014 learned_addr learned my addr 172.21.15.28:0/3577807014 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.665+0000 7f9d57fff640 1 --2- 172.21.15.28:0/3577807014 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d58069dc0 0x7f9d58198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.665+0000 7f9d577fe640 1 --2- 172.21.15.28:0/3577807014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d5806a780 0x7f9d581985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.665+0000 7f9d5ccba640 1 -- 172.21.15.28:0/3577807014 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d58069dc0 msgr2=0x7f9d58198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.665+0000 7f9d5ccba640 1 --2- 172.21.15.28:0/3577807014 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d58069dc0 0x7f9d58198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.665+0000 7f9d5ccba640 1 -- 172.21.15.28:0/3577807014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d5806a780 msgr2=0x7f9d581985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.665+0000 7f9d5ccba640 1 --2- 172.21.15.28:0/3577807014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d5806a780 0x7f9d581985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.665+0000 7f9d5ccba640 1 -- 172.21.15.28:0/3577807014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d5819d050 con 0x7f9d58107730 2024-04-25T01:40:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.665+0000 7f9d5ccba640 1 --2- 172.21.15.28:0/3577807014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d58107730 0x7f9d5819c920 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f9d5000eaa0 tx=0x7f9d5000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.665+0000 7f9d557fa640 1 -- 172.21.15.28:0/3577807014 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5000cdb0 con 0x7f9d58107730 2024-04-25T01:40:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.665+0000 7f9d557fa640 1 -- 172.21.15.28:0/3577807014 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d50005020 con 0x7f9d58107730 2024-04-25T01:40:23.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.665+0000 7f9d5e71d640 1 -- 172.21.15.28:0/3577807014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d5819d340 con 0x7f9d58107730 2024-04-25T01:40:23.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.665+0000 7f9d557fa640 1 -- 172.21.15.28:0/3577807014 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d500116b0 con 0x7f9d58107730 2024-04-25T01:40:23.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.665+0000 7f9d5e71d640 1 -- 172.21.15.28:0/3577807014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d581a4bd0 con 0x7f9d58107730 2024-04-25T01:40:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.667+0000 7f9d557fa640 1 -- 172.21.15.28:0/3577807014 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9d50005560 con 0x7f9d58107730 2024-04-25T01:40:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.668+0000 7f9d6141f740 1 Processor -- start 2024-04-25T01:40:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.668+0000 7f9d6141f740 1 -- start start 2024-04-25T01:40:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.668+0000 7f9d6141f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03d580 0x55744c03d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.668+0000 7f9d6141f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03dea0 0x55744c048030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.668+0000 7f9d557fa640 1 --2- 172.21.15.28:0/3577807014 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9d440a2d00 0x7f9d440a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.668+0000 7f9d6141f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 0x55744c04a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.668+0000 7f9d6141f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55744bfa2670 con 0x55744c03d580 2024-04-25T01:40:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.668+0000 7f9d6141f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55744bf4a970 con 0x55744c03dea0 2024-04-25T01:40:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.668+0000 7f9d6141f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55744bf9e630 con 0x55744c048570 2024-04-25T01:40:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.668+0000 7f9d577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03d580 0x55744c03d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.668+0000 7f9d57fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03dea0 0x55744c048030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.668+0000 7f9d577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03d580 0x55744c03d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34936/0 (socket says 172.21.15.28:34936) 2024-04-25T01:40:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.669+0000 7f9d57fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03dea0 0x55744c048030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:44732/0 (socket says 172.21.15.28:44732) 2024-04-25T01:40:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.669+0000 7f9d577fe640 1 -- 172.21.15.28:0/214961777 learned_addr learned my addr 172.21.15.28:0/214961777 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.669+0000 7f9d557fa640 1 -- 172.21.15.28:0/3577807014 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f9d500cfa90 con 0x7f9d58107730 2024-04-25T01:40:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.669+0000 7f9d577fe640 1 --2- 172.21.15.28:0/3577807014 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9d440a2d00 0x7f9d440a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.669+0000 7f9d5ccba640 1 --2- 172.21.15.28:0/214961777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 0x55744c04a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.669+0000 7f9d577fe640 1 -- 172.21.15.28:0/214961777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 msgr2=0x55744c04a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.669+0000 7f9d577fe640 1 --2- 172.21.15.28:0/214961777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 0x55744c04a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.669+0000 7f9d577fe640 1 -- 172.21.15.28:0/214961777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03dea0 msgr2=0x55744c048030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.669+0000 7f9d577fe640 1 --2- 172.21.15.28:0/214961777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03dea0 0x55744c048030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.669+0000 7f9d577fe640 1 -- 172.21.15.28:0/214961777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55744bf6e400 con 0x55744c03d580 2024-04-25T01:40:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.669+0000 7f9d5ccba640 1 --2- 172.21.15.28:0/214961777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 0x55744c04a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:40:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.669+0000 7f9d57fff640 1 --2- 172.21.15.28:0/214961777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03dea0 0x55744c048030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:40:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.669+0000 7f9d577fe640 1 --2- 172.21.15.28:0/214961777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03d580 0x55744c03d960 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f9d400096c0 tx=0x7f9d40031b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4144f82c28ef8a44 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.669+0000 7f9d577fe640 1 --2- 172.21.15.28:0/3577807014 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9d440a2d00 0x7f9d440a51c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9d4002a940 tx=0x7f9d40045000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.669+0000 7f9d4e7fc640 1 -- 172.21.15.28:0/214961777 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40035040 con 0x55744c03d580 2024-04-25T01:40:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.669+0000 7f9d4e7fc640 1 -- 172.21.15.28:0/214961777 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d40004400 con 0x55744c03d580 2024-04-25T01:40:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.669+0000 7f9d4e7fc640 1 -- 172.21.15.28:0/214961777 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40048a00 con 0x55744c03d580 2024-04-25T01:40:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.670+0000 7f9d6141f740 1 -- 172.21.15.28:0/214961777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03d580 msgr2=0x55744c03d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.670+0000 7f9d6141f740 1 --2- 172.21.15.28:0/214961777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03d580 0x55744c03d960 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f9d400096c0 tx=0x7f9d40031b00 comp rx=0 tx=0).stop 2024-04-25T01:40:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.670+0000 7f9d6141f740 1 -- 172.21.15.28:0/214961777 shutdown_connections 2024-04-25T01:40:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.670+0000 7f9d6141f740 1 --2- 172.21.15.28:0/214961777 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 0x55744c04a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.670+0000 7f9d6141f740 1 --2- 172.21.15.28:0/214961777 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03dea0 0x55744c048030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.670+0000 7f9d6141f740 1 --2- 172.21.15.28:0/214961777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03d580 0x55744c03d960 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.670+0000 7f9d6141f740 1 -- 172.21.15.28:0/214961777 >> 172.21.15.28:0/214961777 conn(0x55744c0398a0 msgr2=0x55744c039c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.670+0000 7f9d6141f740 1 -- 172.21.15.28:0/214961777 shutdown_connections 2024-04-25T01:40:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.670+0000 7f9d6141f740 1 -- 172.21.15.28:0/214961777 wait complete. 2024-04-25T01:40:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.670+0000 7f9d6141f740 1 -- start start 2024-04-25T01:40:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d6141f740 1 Processor -- start 2024-04-25T01:40:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d6141f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03d580 0x55744c047f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d6141f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03dea0 0x55744c045260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d6141f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 0x55744c045870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d6141f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55744c04b300 con 0x55744c03d580 2024-04-25T01:40:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d6141f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55744bf5e050 con 0x55744c03dea0 2024-04-25T01:40:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d6141f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55744c04b480 con 0x55744c048570 2024-04-25T01:40:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d6141f740 10 client.0 _ll_register_callbacks cb 0x55744bcb5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d57fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03d580 0x55744c047f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d57fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03d580 0x55744c047f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34940/0 (socket says 172.21.15.28:34940) 2024-04-25T01:40:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d57fff640 1 -- 172.21.15.28:0/2702589358 learned_addr learned my addr 172.21.15.28:0/2702589358 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d5ccba640 1 --2- 172.21.15.28:0/2702589358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 0x55744c045870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d577fe640 1 --2- 172.21.15.28:0/2702589358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03dea0 0x55744c045260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d57fff640 1 -- 172.21.15.28:0/2702589358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 msgr2=0x55744c045870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d57fff640 1 --2- 172.21.15.28:0/2702589358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 0x55744c045870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d57fff640 1 -- 172.21.15.28:0/2702589358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03dea0 msgr2=0x55744c045260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d57fff640 1 --2- 172.21.15.28:0/2702589358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03dea0 0x55744c045260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d57fff640 1 -- 172.21.15.28:0/2702589358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55744bf2fdc0 con 0x55744c03d580 2024-04-25T01:40:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d57fff640 1 --2- 172.21.15.28:0/2702589358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03d580 0x55744c047f70 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f9d4800b9f0 tx=0x7f9d4800beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d59edcf60cb28f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.671+0000 7f9d377fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.672+0000 7f9d377fe640 1 -- 172.21.15.28:0/2702589358 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d48003e20 con 0x55744c03d580 2024-04-25T01:40:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.672+0000 7f9d377fe640 1 -- 172.21.15.28:0/2702589358 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f9d48011440 con 0x55744c03d580 2024-04-25T01:40:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.672+0000 7f9d6141f740 1 -- 172.21.15.28:0/2702589358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d40004400 con 0x55744c03d580 2024-04-25T01:40:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.672+0000 7f9d377fe640 1 -- 172.21.15.28:0/2702589358 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9d48011640 con 0x55744c03d580 2024-04-25T01:40:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.672+0000 7f9d6141f740 10 client.18291 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.672+0000 7f9d6141f740 1 -- 172.21.15.28:0/2702589358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55744bcb63f0 con 0x55744c03d580 2024-04-25T01:40:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.672+0000 7f9d377fe640 1 -- 172.21.15.28:0/2702589358 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d40004400 con 0x55744c03d580 2024-04-25T01:40:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.672+0000 7f9d377fe640 1 -- 172.21.15.28:0/2702589358 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9d4801b960 con 0x55744c03d580 2024-04-25T01:40:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.672+0000 7f9d6141f740 10 client.18291 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.672+0000 7f9d6141f740 10 client.18291 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:40:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.672+0000 7f9d6141f740 20 client.18291 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.672+0000 7f9d6141f740 1 --2- 172.21.15.28:0/2702589358 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55744c0586e0 0x55744c05ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.672+0000 7f9d6141f740 4 client.18291 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.672+0000 7f9d6141f740 1 -- 172.21.15.28:0/2702589358 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55744bfafbb0 con 0x55744c0586e0 2024-04-25T01:40:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.673+0000 7f9d577fe640 1 --2- 172.21.15.28:0/2702589358 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55744c0586e0 0x55744c05ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.673+0000 7f9d577fe640 1 --2- 172.21.15.28:0/2702589358 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55744c0586e0 0x55744c05ab60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c468a9d8d52e3c09 server_cookie=ae8c7d2058a68d21 in_seq=0 out_seq=0 2024-04-25T01:40:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.673+0000 7f9d377fe640 10 client.18291 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:40:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.674+0000 7f9d377fe640 1 -- 172.21.15.28:0/2702589358 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55744bfafbb0 con 0x55744c0586e0 2024-04-25T01:40:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.674+0000 7f9d377fe640 10 client.18291 handle_command_reply: tid=0 2024-04-25T01:40:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.674+0000 7f9d6141f740 1 client.18291 shutdown 2024-04-25T01:40:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.674+0000 7f9d6141f740 1 -- 172.21.15.28:0/2702589358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03d580 msgr2=0x55744c047f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.674+0000 7f9d6141f740 1 --2- 172.21.15.28:0/2702589358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03d580 0x55744c047f70 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f9d4800b9f0 tx=0x7f9d4800beb0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.674+0000 7f9d6141f740 1 -- 172.21.15.28:0/2702589358 shutdown_connections 2024-04-25T01:40:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.674+0000 7f9d6141f740 1 --2- 172.21.15.28:0/2702589358 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55744c0586e0 0x55744c05ab60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.674+0000 7f9d6141f740 1 --2- 172.21.15.28:0/2702589358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 0x55744c045870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.674+0000 7f9d377fe640 0 client.18291 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:40:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.674+0000 7f9d6141f740 1 --2- 172.21.15.28:0/2702589358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03dea0 0x55744c045260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.674+0000 7f9d6141f740 1 --2- 172.21.15.28:0/2702589358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03d580 0x55744c047f70 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.674+0000 7f9d6141f740 1 -- 172.21.15.28:0/2702589358 >> 172.21.15.28:0/2702589358 conn(0x55744c0398a0 msgr2=0x55744c03f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.674+0000 7f9d6141f740 1 -- 172.21.15.28:0/2702589358 shutdown_connections 2024-04-25T01:40:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.674+0000 7f9d6141f740 1 -- 172.21.15.28:0/2702589358 wait complete. 2024-04-25T01:40:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.674+0000 7f9d6141f740 20 client.18291 trim_cache size 0 max 16384 2024-04-25T01:40:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.687+0000 7f9d6141f740 1 Processor -- start 2024-04-25T01:40:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.687+0000 7f9d6141f740 1 -- start start 2024-04-25T01:40:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.687+0000 7f9d6141f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03d580 0x55744c0541c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.687+0000 7f9d6141f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03dea0 0x55744c047aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.688+0000 7f9d6141f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 0x55744c060430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.688+0000 7f9d6141f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d581aebe0 con 0x55744c03dea0 2024-04-25T01:40:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.688+0000 7f9d6141f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9d581b0720 con 0x55744c03d580 2024-04-25T01:40:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.688+0000 7f9d6141f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9d581af2b0 con 0x55744c048570 2024-04-25T01:40:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.688+0000 7f9d577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03dea0 0x55744c047aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.688+0000 7f9d577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03dea0 0x55744c047aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34948/0 (socket says 172.21.15.28:34948) 2024-04-25T01:40:23.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.688+0000 7f9d577fe640 1 -- 172.21.15.28:0/4276349909 learned_addr learned my addr 172.21.15.28:0/4276349909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.688+0000 7f9d57fff640 1 --2- 172.21.15.28:0/4276349909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03d580 0x55744c0541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.688+0000 7f9d577fe640 1 -- 172.21.15.28:0/4276349909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 msgr2=0x55744c060430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.688+0000 7f9d5ccba640 1 --2- 172.21.15.28:0/4276349909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 0x55744c060430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.688+0000 7f9d577fe640 1 --2- 172.21.15.28:0/4276349909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 0x55744c060430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.688+0000 7f9d577fe640 1 -- 172.21.15.28:0/4276349909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03d580 msgr2=0x55744c0541c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.688+0000 7f9d577fe640 1 --2- 172.21.15.28:0/4276349909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03d580 0x55744c0541c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.688+0000 7f9d577fe640 1 -- 172.21.15.28:0/4276349909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55744c045be0 con 0x55744c03dea0 2024-04-25T01:40:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.688+0000 7f9d57fff640 1 --2- 172.21.15.28:0/4276349909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03d580 0x55744c0541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.688+0000 7f9d577fe640 1 --2- 172.21.15.28:0/4276349909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03dea0 0x55744c047aa0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f9d4004a040 tx=0x7f9d400485c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=729dbdce59bb4701 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.688+0000 7f9d4dffb640 1 -- 172.21.15.28:0/4276349909 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40035040 con 0x55744c03dea0 2024-04-25T01:40:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.688+0000 7f9d4dffb640 1 -- 172.21.15.28:0/4276349909 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d40004400 con 0x55744c03dea0 2024-04-25T01:40:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.688+0000 7f9d4dffb640 1 -- 172.21.15.28:0/4276349909 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40005460 con 0x55744c03dea0 2024-04-25T01:40:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.689+0000 7f9d6141f740 1 -- 172.21.15.28:0/4276349909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03dea0 msgr2=0x55744c047aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.689+0000 7f9d6141f740 1 --2- 172.21.15.28:0/4276349909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03dea0 0x55744c047aa0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f9d4004a040 tx=0x7f9d400485c0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.689+0000 7f9d6141f740 1 -- 172.21.15.28:0/4276349909 shutdown_connections 2024-04-25T01:40:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.689+0000 7f9d6141f740 1 --2- 172.21.15.28:0/4276349909 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 0x55744c060430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.689+0000 7f9d6141f740 1 --2- 172.21.15.28:0/4276349909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03dea0 0x55744c047aa0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.689+0000 7f9d6141f740 1 --2- 172.21.15.28:0/4276349909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03d580 0x55744c0541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.689+0000 7f9d6141f740 1 -- 172.21.15.28:0/4276349909 >> 172.21.15.28:0/4276349909 conn(0x55744c0398a0 msgr2=0x55744c039c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.689+0000 7f9d6141f740 1 -- 172.21.15.28:0/4276349909 shutdown_connections 2024-04-25T01:40:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.689+0000 7f9d6141f740 1 -- 172.21.15.28:0/4276349909 wait complete. 2024-04-25T01:40:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.689+0000 7f9d6141f740 1 -- start start 2024-04-25T01:40:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.689+0000 7f9d6141f740 1 Processor -- start 2024-04-25T01:40:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d6141f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03d580 0x55744c0bc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d6141f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03dea0 0x55744c0bc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d6141f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 0x55744c0b6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d6141f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d581b0050 con 0x55744c03dea0 2024-04-25T01:40:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d6141f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9d581af980 con 0x55744c03d580 2024-04-25T01:40:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d6141f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9d581acbe0 con 0x55744c048570 2024-04-25T01:40:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d6141f740 10 client.0 _ll_register_callbacks cb 0x7f9d48004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03dea0 0x55744c0bc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03dea0 0x55744c0bc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34958/0 (socket says 172.21.15.28:34958) 2024-04-25T01:40:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d577fe640 1 -- 172.21.15.28:0/1640255410 learned_addr learned my addr 172.21.15.28:0/1640255410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d5ccba640 1 --2- 172.21.15.28:0/1640255410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 0x55744c0b6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d57fff640 1 --2- 172.21.15.28:0/1640255410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03d580 0x55744c0bc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d577fe640 1 -- 172.21.15.28:0/1640255410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 msgr2=0x55744c0b6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d577fe640 1 --2- 172.21.15.28:0/1640255410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 0x55744c0b6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d577fe640 1 -- 172.21.15.28:0/1640255410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03d580 msgr2=0x55744c0bc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d577fe640 1 --2- 172.21.15.28:0/1640255410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03d580 0x55744c0bc120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d577fe640 1 -- 172.21.15.28:0/1640255410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55744c03dcd0 con 0x55744c03dea0 2024-04-25T01:40:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d57fff640 1 --2- 172.21.15.28:0/1640255410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03d580 0x55744c0bc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d577fe640 1 --2- 172.21.15.28:0/1640255410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03dea0 0x55744c0bc660 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f9d40005c40 tx=0x7f9d40005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46deea180e3b423b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d377fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d377fe640 1 -- 172.21.15.28:0/1640255410 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40035040 con 0x55744c03dea0 2024-04-25T01:40:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d377fe640 1 -- 172.21.15.28:0/1640255410 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f9d40031dd0 con 0x55744c03dea0 2024-04-25T01:40:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.690+0000 7f9d6141f740 1 -- 172.21.15.28:0/1640255410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d40004400 con 0x55744c03dea0 2024-04-25T01:40:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.691+0000 7f9d377fe640 1 -- 172.21.15.28:0/1640255410 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9d4006a050 con 0x55744c03dea0 2024-04-25T01:40:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.691+0000 7f9d6141f740 10 client.18297 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.691+0000 7f9d6141f740 1 -- 172.21.15.28:0/1640255410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55744bd807d0 con 0x55744c03dea0 2024-04-25T01:40:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.691+0000 7f9d377fe640 1 -- 172.21.15.28:0/1640255410 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d40004400 con 0x55744c03dea0 2024-04-25T01:40:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.691+0000 7f9d377fe640 1 -- 172.21.15.28:0/1640255410 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9d4003a070 con 0x55744c03dea0 2024-04-25T01:40:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.691+0000 7f9d6141f740 10 client.18297 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.691+0000 7f9d6141f740 10 client.18297 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:40:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.691+0000 7f9d6141f740 20 client.18297 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.691+0000 7f9d6141f740 1 --2- 172.21.15.28:0/1640255410 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55744c0c3780 0x55744c0c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.691+0000 7f9d6141f740 4 client.18297 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.691+0000 7f9d6141f740 1 -- 172.21.15.28:0/1640255410 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55744bfa9f00 con 0x55744c0c3780 2024-04-25T01:40:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.691+0000 7f9d57fff640 1 --2- 172.21.15.28:0/1640255410 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55744c0c3780 0x55744c0c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.692+0000 7f9d377fe640 10 client.18297 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:40:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.692+0000 7f9d57fff640 1 --2- 172.21.15.28:0/1640255410 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55744c0c3780 0x55744c0c5b70 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=edc73e27b069f3f4 server_cookie=8225f05808f6d27b in_seq=0 out_seq=0 2024-04-25T01:40:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.692+0000 7f9d377fe640 1 -- 172.21.15.28:0/1640255410 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55744bfa9f00 con 0x55744c0c3780 2024-04-25T01:40:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.692+0000 7f9d377fe640 10 client.18297 handle_command_reply: tid=0 2024-04-25T01:40:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.692+0000 7f9d6141f740 1 client.18297 shutdown 2024-04-25T01:40:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.692+0000 7f9d6141f740 1 -- 172.21.15.28:0/1640255410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03dea0 msgr2=0x55744c0bc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.692+0000 7f9d6141f740 1 --2- 172.21.15.28:0/1640255410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03dea0 0x55744c0bc660 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f9d40005c40 tx=0x7f9d40005fd0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.692+0000 7f9d6141f740 1 -- 172.21.15.28:0/1640255410 shutdown_connections 2024-04-25T01:40:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.692+0000 7f9d6141f740 1 --2- 172.21.15.28:0/1640255410 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55744c0c3780 0x55744c0c5b70 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.692+0000 7f9d6141f740 1 --2- 172.21.15.28:0/1640255410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55744c048570 0x55744c0b6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.692+0000 7f9d6141f740 1 --2- 172.21.15.28:0/1640255410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55744c03dea0 0x55744c0bc660 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.692+0000 7f9d377fe640 0 client.18297 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:40:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.692+0000 7f9d6141f740 1 --2- 172.21.15.28:0/1640255410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55744c03d580 0x55744c0bc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.692+0000 7f9d6141f740 1 -- 172.21.15.28:0/1640255410 >> 172.21.15.28:0/1640255410 conn(0x55744c0398a0 msgr2=0x55744c0531b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.692+0000 7f9d6141f740 1 -- 172.21.15.28:0/1640255410 shutdown_connections 2024-04-25T01:40:23.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.692+0000 7f9d6141f740 1 -- 172.21.15.28:0/1640255410 wait complete. 2024-04-25T01:40:23.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.692+0000 7f9d6141f740 20 client.18297 trim_cache size 0 max 16384 2024-04-25T01:40:23.684 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.693+0000 7f9d5e71d640 1 -- 172.21.15.28:0/3577807014 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9d440a2d00 msgr2=0x7f9d440a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.693+0000 7f9d5e71d640 1 --2- 172.21.15.28:0/3577807014 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9d440a2d00 0x7f9d440a51c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9d4002a940 tx=0x7f9d40045000 comp rx=0 tx=0).stop 2024-04-25T01:40:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.693+0000 7f9d5e71d640 1 -- 172.21.15.28:0/3577807014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d58107730 msgr2=0x7f9d5819c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.693+0000 7f9d5e71d640 1 --2- 172.21.15.28:0/3577807014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d58107730 0x7f9d5819c920 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f9d5000eaa0 tx=0x7f9d5000ef60 comp rx=0 tx=0).stop 2024-04-25T01:40:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.693+0000 7f9d5e71d640 1 -- 172.21.15.28:0/3577807014 shutdown_connections 2024-04-25T01:40:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.693+0000 7f9d5e71d640 1 --2- 172.21.15.28:0/3577807014 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9d440a2d00 0x7f9d440a51c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.693+0000 7f9d5e71d640 1 --2- 172.21.15.28:0/3577807014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d58107730 0x7f9d5819c920 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.693+0000 7f9d5e71d640 1 --2- 172.21.15.28:0/3577807014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d5806a780 0x7f9d581985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.693+0000 7f9d5e71d640 1 --2- 172.21.15.28:0/3577807014 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9d58069dc0 0x7f9d58198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.693+0000 7f9d5e71d640 1 -- 172.21.15.28:0/3577807014 >> 172.21.15.28:0/3577807014 conn(0x7f9d580776a0 msgr2=0x7f9d580fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.693+0000 7f9d5e71d640 1 -- 172.21.15.28:0/3577807014 shutdown_connections 2024-04-25T01:40:23.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.693+0000 7f9d5e71d640 1 -- 172.21.15.28:0/3577807014 wait complete. 2024-04-25T01:40:23.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:40:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.808+0000 7f1e8254f640 1 -- 172.21.15.28:0/1307144453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e7c069dc0 msgr2=0x7f1e7c06a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.808+0000 7f1e8254f640 1 --2- 172.21.15.28:0/1307144453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e7c069dc0 0x7f1e7c06a240 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f1e70009930 tx=0x7f1e7002a420 comp rx=0 tx=0).stop 2024-04-25T01:40:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.808+0000 7f1e8254f640 1 -- 172.21.15.28:0/1307144453 shutdown_connections 2024-04-25T01:40:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.808+0000 7f1e8254f640 1 --2- 172.21.15.28:0/1307144453 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e7c06a780 0x7f1e7c10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.808+0000 7f1e8254f640 1 --2- 172.21.15.28:0/1307144453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e7c069dc0 0x7f1e7c06a240 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.808+0000 7f1e8254f640 1 --2- 172.21.15.28:0/1307144453 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e7c107730 0x7f1e7c107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.808+0000 7f1e8254f640 1 -- 172.21.15.28:0/1307144453 >> 172.21.15.28:0/1307144453 conn(0x7f1e7c0776a0 msgr2=0x7f1e7c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.808+0000 7f1e8254f640 1 -- 172.21.15.28:0/1307144453 shutdown_connections 2024-04-25T01:40:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.808+0000 7f1e8254f640 1 -- 172.21.15.28:0/1307144453 wait complete. 2024-04-25T01:40:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.809+0000 7f1e8254f640 1 Processor -- start 2024-04-25T01:40:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.809+0000 7f1e8254f640 1 -- start start 2024-04-25T01:40:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.809+0000 7f1e8254f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e7c069dc0 0x7f1e7c100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.809+0000 7f1e8254f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e7c06a780 0x7f1e7c100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.809+0000 7f1e8254f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e7c107730 0x7f1e7c1010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.809+0000 7f1e8254f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c111b40 con 0x7f1e7c107730 2024-04-25T01:40:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.809+0000 7f1e8254f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c1119c0 con 0x7f1e7c069dc0 2024-04-25T01:40:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.809+0000 7f1e8254f640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c111cc0 con 0x7f1e7c06a780 2024-04-25T01:40:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.809+0000 7f1e80aec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e7c107730 0x7f1e7c1010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.809+0000 7f1e80aec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e7c107730 0x7f1e7c1010f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34980/0 (socket says 172.21.15.28:34980) 2024-04-25T01:40:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.809+0000 7f1e80aec640 1 -- 172.21.15.28:0/2261954893 learned_addr learned my addr 172.21.15.28:0/2261954893 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.810+0000 7f1e7bfff640 1 --2- 172.21.15.28:0/2261954893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e7c069dc0 0x7f1e7c100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.810+0000 7f1e7b7fe640 1 --2- 172.21.15.28:0/2261954893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e7c06a780 0x7f1e7c100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.810+0000 7f1e80aec640 1 -- 172.21.15.28:0/2261954893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e7c06a780 msgr2=0x7f1e7c100bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.810+0000 7f1e80aec640 1 --2- 172.21.15.28:0/2261954893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e7c06a780 0x7f1e7c100bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.810+0000 7f1e80aec640 1 -- 172.21.15.28:0/2261954893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e7c069dc0 msgr2=0x7f1e7c100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.810+0000 7f1e80aec640 1 --2- 172.21.15.28:0/2261954893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e7c069dc0 0x7f1e7c100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.810+0000 7f1e80aec640 1 -- 172.21.15.28:0/2261954893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e7c1a6560 con 0x7f1e7c107730 2024-04-25T01:40:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.810+0000 7f1e80aec640 1 --2- 172.21.15.28:0/2261954893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e7c107730 0x7f1e7c1010f0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f1e7400eb10 tx=0x7f1e7400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.810+0000 7f1e797fa640 1 -- 172.21.15.28:0/2261954893 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e74004b20 con 0x7f1e7c107730 2024-04-25T01:40:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.810+0000 7f1e797fa640 1 -- 172.21.15.28:0/2261954893 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e74004cc0 con 0x7f1e7c107730 2024-04-25T01:40:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.810+0000 7f1e8254f640 1 -- 172.21.15.28:0/2261954893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e7c1a6850 con 0x7f1e7c107730 2024-04-25T01:40:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.810+0000 7f1e797fa640 1 -- 172.21.15.28:0/2261954893 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e74010760 con 0x7f1e7c107730 2024-04-25T01:40:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.810+0000 7f1e8254f640 1 -- 172.21.15.28:0/2261954893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e7c1a6d70 con 0x7f1e7c107730 2024-04-25T01:40:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.812+0000 7f1e797fa640 1 -- 172.21.15.28:0/2261954893 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1e74005020 con 0x7f1e7c107730 2024-04-25T01:40:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.813+0000 7f1e8525c740 1 Processor -- start 2024-04-25T01:40:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.813+0000 7f1e8525c740 1 -- start start 2024-04-25T01:40:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.813+0000 7f1e8525c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfba7580 0x55cdbfba7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.813+0000 7f1e8525c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7ea0 0x55cdbfbb2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.813+0000 7f1e8525c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfbb2570 0x55cdbfbb4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.813+0000 7f1e8525c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cdbfb0c670 con 0x55cdbfba7580 2024-04-25T01:40:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.813+0000 7f1e8525c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cdbfab4970 con 0x55cdbfbb2570 2024-04-25T01:40:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.813+0000 7f1e8525c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55cdbfb08630 con 0x55cdbfba7ea0 2024-04-25T01:40:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.813+0000 7f1e797fa640 1 --2- 172.21.15.28:0/2261954893 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1e640a2db0 0x7f1e640a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.813+0000 7f1e7b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfba7580 0x55cdbfba7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.813+0000 7f1e7b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfba7580 0x55cdbfba7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34986/0 (socket says 172.21.15.28:34986) 2024-04-25T01:40:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.813+0000 7f1e7b7fe640 1 -- 172.21.15.28:0/316587953 learned_addr learned my addr 172.21.15.28:0/316587953 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.813+0000 7f1e80aec640 1 --2- 172.21.15.28:0/2261954893 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1e640a2db0 0x7f1e640a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.813+0000 7f1e7bfff640 1 --2- 172.21.15.28:0/316587953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7ea0 0x55cdbfbb2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.814+0000 7f1e7b7fe640 1 --2- 172.21.15.28:0/316587953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfbb2570 0x55cdbfbb4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.814+0000 7f1e797fa640 1 -- 172.21.15.28:0/2261954893 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f1e740cfcc0 con 0x7f1e7c107730 2024-04-25T01:40:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.814+0000 7f1e7b7fe640 1 -- 172.21.15.28:0/316587953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7ea0 msgr2=0x55cdbfbb2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.814+0000 7f1e7b7fe640 1 --2- 172.21.15.28:0/316587953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7ea0 0x55cdbfbb2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.814+0000 7f1e7b7fe640 1 -- 172.21.15.28:0/316587953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfbb2570 msgr2=0x55cdbfbb4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.814+0000 7f1e7b7fe640 1 --2- 172.21.15.28:0/316587953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfbb2570 0x55cdbfbb4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.814+0000 7f1e7b7fe640 1 -- 172.21.15.28:0/316587953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdbfad8400 con 0x55cdbfba7580 2024-04-25T01:40:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.814+0000 7f1e7b7fe640 1 --2- 172.21.15.28:0/316587953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfba7580 0x55cdbfba7960 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f1e70009930 tx=0x7f1e70000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4be597236cb62f36 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.814+0000 7f1e80aec640 1 --2- 172.21.15.28:0/2261954893 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1e640a2db0 0x7f1e640a5270 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1e7409bb50 tx=0x7f1e7400efd0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.814+0000 7f1e627fc640 1 -- 172.21.15.28:0/316587953 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e70044030 con 0x55cdbfba7580 2024-04-25T01:40:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.814+0000 7f1e627fc640 1 -- 172.21.15.28:0/316587953 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e70005020 con 0x55cdbfba7580 2024-04-25T01:40:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.814+0000 7f1e627fc640 1 -- 172.21.15.28:0/316587953 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e70035040 con 0x55cdbfba7580 2024-04-25T01:40:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.815+0000 7f1e8525c740 1 -- 172.21.15.28:0/316587953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfba7580 msgr2=0x55cdbfba7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.815+0000 7f1e8525c740 1 --2- 172.21.15.28:0/316587953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfba7580 0x55cdbfba7960 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f1e70009930 tx=0x7f1e70000bc0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.815+0000 7f1e8525c740 1 -- 172.21.15.28:0/316587953 shutdown_connections 2024-04-25T01:40:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.815+0000 7f1e8525c740 1 --2- 172.21.15.28:0/316587953 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfbb2570 0x55cdbfbb4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.815+0000 7f1e8525c740 1 --2- 172.21.15.28:0/316587953 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7ea0 0x55cdbfbb2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.815+0000 7f1e8525c740 1 --2- 172.21.15.28:0/316587953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfba7580 0x55cdbfba7960 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.815+0000 7f1e8525c740 1 -- 172.21.15.28:0/316587953 >> 172.21.15.28:0/316587953 conn(0x55cdbfba38a0 msgr2=0x55cdbfba3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.815+0000 7f1e8525c740 1 -- 172.21.15.28:0/316587953 shutdown_connections 2024-04-25T01:40:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.815+0000 7f1e8525c740 1 -- 172.21.15.28:0/316587953 wait complete. 2024-04-25T01:40:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.815+0000 7f1e8525c740 1 -- start start 2024-04-25T01:40:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.815+0000 7f1e8525c740 1 Processor -- start 2024-04-25T01:40:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e8525c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfba7580 0x55cdbfbb1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e8525c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7ea0 0x55cdbfbaf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e8525c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfbb2570 0x55cdbfbaf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e8525c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cdbfbb5300 con 0x55cdbfba7580 2024-04-25T01:40:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e8525c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cdbfac8050 con 0x55cdbfbb2570 2024-04-25T01:40:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e8525c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55cdbfbb5480 con 0x55cdbfba7ea0 2024-04-25T01:40:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e8525c740 10 client.0 _ll_register_callbacks cb 0x55cdbf81fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e7b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfba7580 0x55cdbfbb1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e7b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfba7580 0x55cdbfbb1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35000/0 (socket says 172.21.15.28:35000) 2024-04-25T01:40:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e7b7fe640 1 -- 172.21.15.28:0/969813120 learned_addr learned my addr 172.21.15.28:0/969813120 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e7bfff640 1 --2- 172.21.15.28:0/969813120 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7ea0 0x55cdbfbaf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e7b7fe640 1 -- 172.21.15.28:0/969813120 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7ea0 msgr2=0x55cdbfbaf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e7b7fe640 1 --2- 172.21.15.28:0/969813120 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7ea0 0x55cdbfbaf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e7b7fe640 1 -- 172.21.15.28:0/969813120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfbb2570 msgr2=0x55cdbfbaf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e7b7fe640 1 --2- 172.21.15.28:0/969813120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfbb2570 0x55cdbfbaf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e7b7fe640 1 -- 172.21.15.28:0/969813120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdbfa99dc0 con 0x55cdbfba7580 2024-04-25T01:40:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e7bfff640 1 --2- 172.21.15.28:0/969813120 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7ea0 0x55cdbfbaf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e7b7fe640 1 --2- 172.21.15.28:0/969813120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfba7580 0x55cdbfbb1e80 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f1e7002a940 tx=0x7f1e70009300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e1bf8204419a18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e5b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e5b7fe640 1 -- 172.21.15.28:0/969813120 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7004d070 con 0x55cdbfba7580 2024-04-25T01:40:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e5b7fe640 1 -- 172.21.15.28:0/969813120 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f1e70004770 con 0x55cdbfba7580 2024-04-25T01:40:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.816+0000 7f1e8525c740 1 -- 172.21.15.28:0/969813120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e70005020 con 0x55cdbfba7580 2024-04-25T01:40:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.817+0000 7f1e5b7fe640 1 -- 172.21.15.28:0/969813120 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1e70058050 con 0x55cdbfba7580 2024-04-25T01:40:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.817+0000 7f1e8525c740 10 client.18309 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.817+0000 7f1e8525c740 1 -- 172.21.15.28:0/969813120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cdbf8203f0 con 0x55cdbfba7580 2024-04-25T01:40:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.817+0000 7f1e5b7fe640 1 -- 172.21.15.28:0/969813120 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e70005020 con 0x55cdbfba7580 2024-04-25T01:40:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.817+0000 7f1e5b7fe640 1 -- 172.21.15.28:0/969813120 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1e7003ab20 con 0x55cdbfba7580 2024-04-25T01:40:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.817+0000 7f1e8525c740 10 client.18309 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.817+0000 7f1e8525c740 10 client.18309 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:40:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.817+0000 7f1e8525c740 20 client.18309 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.817+0000 7f1e8525c740 1 --2- 172.21.15.28:0/969813120 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cdbfbc25a0 0x55cdbfbc4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.817+0000 7f1e8525c740 4 client.18309 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.817+0000 7f1e8525c740 1 -- 172.21.15.28:0/969813120 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cdbfb19bb0 con 0x55cdbfbc25a0 2024-04-25T01:40:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.817+0000 7f1e7bfff640 1 --2- 172.21.15.28:0/969813120 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cdbfbc25a0 0x55cdbfbc4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.818+0000 7f1e7bfff640 1 --2- 172.21.15.28:0/969813120 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cdbfbc25a0 0x55cdbfbc4a20 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e4c5d5b35d8e1b8a server_cookie=900f40c97292360f in_seq=0 out_seq=0 2024-04-25T01:40:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.818+0000 7f1e5b7fe640 10 client.18309 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:40:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.818+0000 7f1e5b7fe640 1 -- 172.21.15.28:0/969813120 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cdbfb19bb0 con 0x55cdbfbc25a0 2024-04-25T01:40:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.818+0000 7f1e5b7fe640 10 client.18309 handle_command_reply: tid=0 2024-04-25T01:40:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.818+0000 7f1e8525c740 1 client.18309 shutdown 2024-04-25T01:40:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.819+0000 7f1e8525c740 1 -- 172.21.15.28:0/969813120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfba7580 msgr2=0x55cdbfbb1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.819+0000 7f1e8525c740 1 --2- 172.21.15.28:0/969813120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfba7580 0x55cdbfbb1e80 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f1e7002a940 tx=0x7f1e70009300 comp rx=0 tx=0).stop 2024-04-25T01:40:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.819+0000 7f1e8525c740 1 -- 172.21.15.28:0/969813120 shutdown_connections 2024-04-25T01:40:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.819+0000 7f1e8525c740 1 --2- 172.21.15.28:0/969813120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfbb2570 0x55cdbfbaf6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.819+0000 7f1e8525c740 1 --2- 172.21.15.28:0/969813120 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7ea0 0x55cdbfbaf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.819+0000 7f1e8525c740 1 --2- 172.21.15.28:0/969813120 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cdbfbc25a0 0x55cdbfbc4a20 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.819+0000 7f1e8525c740 1 --2- 172.21.15.28:0/969813120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfba7580 0x55cdbfbb1e80 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.819+0000 7f1e8525c740 1 -- 172.21.15.28:0/969813120 >> 172.21.15.28:0/969813120 conn(0x55cdbfba38a0 msgr2=0x55cdbfba96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.819+0000 7f1e5b7fe640 0 client.18309 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:40:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.819+0000 7f1e8525c740 1 -- 172.21.15.28:0/969813120 shutdown_connections 2024-04-25T01:40:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.819+0000 7f1e8525c740 1 -- 172.21.15.28:0/969813120 wait complete. 2024-04-25T01:40:23.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.819+0000 7f1e8525c740 20 client.18309 trim_cache size 0 max 16384 2024-04-25T01:40:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.832+0000 7f1e8525c740 1 Processor -- start 2024-04-25T01:40:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.832+0000 7f1e8525c740 1 -- start start 2024-04-25T01:40:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.832+0000 7f1e8525c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7580 0x55cdbfbbe1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.832+0000 7f1e8525c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfba7ea0 0x55cdbfbaa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.832+0000 7f1e8525c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfbb2570 0x55cdbfbb19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.832+0000 7f1e8525c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c1b0d40 con 0x55cdbfbb2570 2024-04-25T01:40:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.832+0000 7f1e8525c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c1b2880 con 0x55cdbfba7ea0 2024-04-25T01:40:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.832+0000 7f1e8525c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c1b1410 con 0x55cdbfba7580 2024-04-25T01:40:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.832+0000 7f1e7b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfbb2570 0x55cdbfbb19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.833+0000 7f1e7b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfbb2570 0x55cdbfbb19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35012/0 (socket says 172.21.15.28:35012) 2024-04-25T01:40:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.833+0000 7f1e7b7fe640 1 -- 172.21.15.28:0/1063777025 learned_addr learned my addr 172.21.15.28:0/1063777025 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.833+0000 7f1e7bfff640 1 --2- 172.21.15.28:0/1063777025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfba7ea0 0x55cdbfbaa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.833+0000 7f1e7b7fe640 1 -- 172.21.15.28:0/1063777025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7580 msgr2=0x55cdbfbbe1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.833+0000 7f1e7b7fe640 1 --2- 172.21.15.28:0/1063777025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7580 0x55cdbfbbe1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.833+0000 7f1e7b7fe640 1 -- 172.21.15.28:0/1063777025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfba7ea0 msgr2=0x55cdbfbaa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.833+0000 7f1e7b7fe640 1 --2- 172.21.15.28:0/1063777025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfba7ea0 0x55cdbfbaa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.833+0000 7f1e7b7fe640 1 -- 172.21.15.28:0/1063777025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdbf8ea7d0 con 0x55cdbfbb2570 2024-04-25T01:40:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.833+0000 7f1e7bfff640 1 --2- 172.21.15.28:0/1063777025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfba7ea0 0x55cdbfbaa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.833+0000 7f1e7b7fe640 1 --2- 172.21.15.28:0/1063777025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfbb2570 0x55cdbfbb19b0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f1e7003d440 tx=0x7f1e70009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e1557e19e2d679 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.833+0000 7f1e61ffb640 1 -- 172.21.15.28:0/1063777025 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7004d070 con 0x55cdbfbb2570 2024-04-25T01:40:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.833+0000 7f1e61ffb640 1 -- 172.21.15.28:0/1063777025 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e7003de20 con 0x55cdbfbb2570 2024-04-25T01:40:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.833+0000 7f1e61ffb640 1 -- 172.21.15.28:0/1063777025 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e70044050 con 0x55cdbfbb2570 2024-04-25T01:40:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.834+0000 7f1e8525c740 1 -- 172.21.15.28:0/1063777025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfbb2570 msgr2=0x55cdbfbb19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.834+0000 7f1e8525c740 1 --2- 172.21.15.28:0/1063777025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfbb2570 0x55cdbfbb19b0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f1e7003d440 tx=0x7f1e70009290 comp rx=0 tx=0).stop 2024-04-25T01:40:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.834+0000 7f1e8525c740 1 -- 172.21.15.28:0/1063777025 shutdown_connections 2024-04-25T01:40:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.834+0000 7f1e8525c740 1 --2- 172.21.15.28:0/1063777025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfbb2570 0x55cdbfbb19b0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.834+0000 7f1e8525c740 1 --2- 172.21.15.28:0/1063777025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfba7ea0 0x55cdbfbaa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.834+0000 7f1e8525c740 1 --2- 172.21.15.28:0/1063777025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7580 0x55cdbfbbe1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.834+0000 7f1e8525c740 1 -- 172.21.15.28:0/1063777025 >> 172.21.15.28:0/1063777025 conn(0x55cdbfba38a0 msgr2=0x55cdbfba9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.834+0000 7f1e8525c740 1 -- 172.21.15.28:0/1063777025 shutdown_connections 2024-04-25T01:40:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.834+0000 7f1e8525c740 1 -- 172.21.15.28:0/1063777025 wait complete. 2024-04-25T01:40:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.834+0000 7f1e8525c740 1 -- start start 2024-04-25T01:40:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.834+0000 7f1e8525c740 1 Processor -- start 2024-04-25T01:40:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.834+0000 7f1e8525c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfba7580 0x55cdbfbba6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.834+0000 7f1e8525c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7ea0 0x55cdbfbbac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.834+0000 7f1e8525c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfbb2570 0x55cdbfbab440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.834+0000 7f1e8525c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c1b21b0 con 0x55cdbfbb2570 2024-04-25T01:40:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.834+0000 7f1e8525c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c1b1ae0 con 0x55cdbfba7580 2024-04-25T01:40:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.834+0000 7f1e8525c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7c1aed40 con 0x55cdbfba7ea0 2024-04-25T01:40:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.835+0000 7f1e8525c740 10 client.0 _ll_register_callbacks cb 0x7f1e6c00a570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.835+0000 7f1e7b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfbb2570 0x55cdbfbab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.835+0000 7f1e7b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfbb2570 0x55cdbfbab440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35016/0 (socket says 172.21.15.28:35016) 2024-04-25T01:40:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.835+0000 7f1e7b7fe640 1 -- 172.21.15.28:0/2593276068 learned_addr learned my addr 172.21.15.28:0/2593276068 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.835+0000 7f1e7bfff640 1 --2- 172.21.15.28:0/2593276068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7ea0 0x55cdbfbbac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.835+0000 7f1e7b7fe640 1 --2- 172.21.15.28:0/2593276068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfba7580 0x55cdbfbba6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.835+0000 7f1e7b7fe640 1 -- 172.21.15.28:0/2593276068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7ea0 msgr2=0x55cdbfbbac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.835+0000 7f1e7b7fe640 1 --2- 172.21.15.28:0/2593276068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7ea0 0x55cdbfbbac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.835+0000 7f1e7b7fe640 1 -- 172.21.15.28:0/2593276068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfba7580 msgr2=0x55cdbfbba6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.835+0000 7f1e7b7fe640 1 --2- 172.21.15.28:0/2593276068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfba7580 0x55cdbfbba6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.835+0000 7f1e7b7fe640 1 -- 172.21.15.28:0/2593276068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdbfbaba80 con 0x55cdbfbb2570 2024-04-25T01:40:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.835+0000 7f1e7b7fe640 1 --2- 172.21.15.28:0/2593276068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfbb2570 0x55cdbfbab440 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f1e7002a940 tx=0x7f1e7003c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4239a832a5e686c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.835+0000 7f1e5b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.835+0000 7f1e5b7fe640 1 -- 172.21.15.28:0/2593276068 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7004d070 con 0x55cdbfbb2570 2024-04-25T01:40:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.835+0000 7f1e5b7fe640 1 -- 172.21.15.28:0/2593276068 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f1e70004a10 con 0x55cdbfbb2570 2024-04-25T01:40:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.835+0000 7f1e8525c740 1 -- 172.21.15.28:0/2593276068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e7003de20 con 0x55cdbfbb2570 2024-04-25T01:40:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.836+0000 7f1e5b7fe640 1 -- 172.21.15.28:0/2593276068 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1e70044040 con 0x55cdbfbb2570 2024-04-25T01:40:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.836+0000 7f1e8525c740 10 client.18315 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.836+0000 7f1e8525c740 1 -- 172.21.15.28:0/2593276068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cdbfc283b0 con 0x55cdbfbb2570 2024-04-25T01:40:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.836+0000 7f1e5b7fe640 1 -- 172.21.15.28:0/2593276068 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e7003de20 con 0x55cdbfbb2570 2024-04-25T01:40:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.836+0000 7f1e5b7fe640 1 -- 172.21.15.28:0/2593276068 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1e7003a6f0 con 0x55cdbfbb2570 2024-04-25T01:40:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.836+0000 7f1e8525c740 10 client.18315 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.836+0000 7f1e8525c740 10 client.18315 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:40:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.836+0000 7f1e8525c740 20 client.18315 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.836+0000 7f1e8525c740 1 --2- 172.21.15.28:0/2593276068 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cdbfc289c0 0x55cdbfc2ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.836+0000 7f1e8525c740 4 client.18315 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.836+0000 7f1e8525c740 1 -- 172.21.15.28:0/2593276068 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cdbfbafa40 con 0x55cdbfc289c0 2024-04-25T01:40:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.836+0000 7f1e7bfff640 1 --2- 172.21.15.28:0/2593276068 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cdbfc289c0 0x55cdbfc2ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.837+0000 7f1e5b7fe640 10 client.18315 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:40:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.837+0000 7f1e7bfff640 1 --2- 172.21.15.28:0/2593276068 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cdbfc289c0 0x55cdbfc2ade0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=752650f0f8807913 server_cookie=bbdcd2b14da889b3 in_seq=0 out_seq=0 2024-04-25T01:40:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.837+0000 7f1e5b7fe640 1 -- 172.21.15.28:0/2593276068 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cdbfbafa40 con 0x55cdbfc289c0 2024-04-25T01:40:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.837+0000 7f1e5b7fe640 10 client.18315 handle_command_reply: tid=0 2024-04-25T01:40:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.837+0000 7f1e8525c740 1 client.18315 shutdown 2024-04-25T01:40:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.837+0000 7f1e8525c740 1 -- 172.21.15.28:0/2593276068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfbb2570 msgr2=0x55cdbfbab440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.837+0000 7f1e8525c740 1 --2- 172.21.15.28:0/2593276068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfbb2570 0x55cdbfbab440 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f1e7002a940 tx=0x7f1e7003c040 comp rx=0 tx=0).stop 2024-04-25T01:40:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.837+0000 7f1e8525c740 1 -- 172.21.15.28:0/2593276068 shutdown_connections 2024-04-25T01:40:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.837+0000 7f1e8525c740 1 --2- 172.21.15.28:0/2593276068 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cdbfc289c0 0x55cdbfc2ade0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.837+0000 7f1e8525c740 1 --2- 172.21.15.28:0/2593276068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdbfbb2570 0x55cdbfbab440 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.837+0000 7f1e8525c740 1 --2- 172.21.15.28:0/2593276068 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cdbfba7ea0 0x55cdbfbbac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.837+0000 7f1e5b7fe640 0 client.18315 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:40:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.837+0000 7f1e8525c740 1 --2- 172.21.15.28:0/2593276068 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cdbfba7580 0x55cdbfbba6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.837+0000 7f1e8525c740 1 -- 172.21.15.28:0/2593276068 >> 172.21.15.28:0/2593276068 conn(0x55cdbfba38a0 msgr2=0x55cdbfbaa740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.837+0000 7f1e8525c740 1 -- 172.21.15.28:0/2593276068 shutdown_connections 2024-04-25T01:40:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.837+0000 7f1e8525c740 1 -- 172.21.15.28:0/2593276068 wait complete. 2024-04-25T01:40:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.837+0000 7f1e8525c740 20 client.18315 trim_cache size 0 max 16384 2024-04-25T01:40:23.829 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:23.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.838+0000 7f1e8254f640 1 -- 172.21.15.28:0/2261954893 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1e640a2db0 msgr2=0x7f1e640a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.838+0000 7f1e8254f640 1 --2- 172.21.15.28:0/2261954893 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1e640a2db0 0x7f1e640a5270 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1e7409bb50 tx=0x7f1e7400efd0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.838+0000 7f1e8254f640 1 -- 172.21.15.28:0/2261954893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e7c107730 msgr2=0x7f1e7c1010f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.838+0000 7f1e8254f640 1 --2- 172.21.15.28:0/2261954893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e7c107730 0x7f1e7c1010f0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f1e7400eb10 tx=0x7f1e7400c2d0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.838+0000 7f1e8254f640 1 -- 172.21.15.28:0/2261954893 shutdown_connections 2024-04-25T01:40:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.838+0000 7f1e8254f640 1 --2- 172.21.15.28:0/2261954893 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1e640a2db0 0x7f1e640a5270 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.838+0000 7f1e8254f640 1 --2- 172.21.15.28:0/2261954893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e7c107730 0x7f1e7c1010f0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.838+0000 7f1e8254f640 1 --2- 172.21.15.28:0/2261954893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e7c06a780 0x7f1e7c100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.838+0000 7f1e8254f640 1 --2- 172.21.15.28:0/2261954893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e7c069dc0 0x7f1e7c100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.838+0000 7f1e8254f640 1 -- 172.21.15.28:0/2261954893 >> 172.21.15.28:0/2261954893 conn(0x7f1e7c0776a0 msgr2=0x7f1e7c0fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.838+0000 7f1e8254f640 1 -- 172.21.15.28:0/2261954893 shutdown_connections 2024-04-25T01:40:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.838+0000 7f1e8254f640 1 -- 172.21.15.28:0/2261954893 wait complete. 2024-04-25T01:40:23.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:40:23.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.953+0000 7f8abe4ca640 1 -- 172.21.15.28:0/2818235321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab8107840 msgr2=0x7f8ab8107c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.953+0000 7f8abe4ca640 1 --2- 172.21.15.28:0/2818235321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab8107840 0x7f8ab8107c20 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f8aa8009900 tx=0x7f8aa802a480 comp rx=0 tx=0).stop 2024-04-25T01:40:23.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.953+0000 7f8abe4ca640 1 -- 172.21.15.28:0/2818235321 shutdown_connections 2024-04-25T01:40:23.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.953+0000 7f8abe4ca640 1 --2- 172.21.15.28:0/2818235321 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ab806a720 0x7f8ab810e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.953+0000 7f8abe4ca640 1 --2- 172.21.15.28:0/2818235321 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ab8069d60 0x7f8ab806a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.953+0000 7f8abe4ca640 1 --2- 172.21.15.28:0/2818235321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab8107840 0x7f8ab8107c20 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.953+0000 7f8abe4ca640 1 -- 172.21.15.28:0/2818235321 >> 172.21.15.28:0/2818235321 conn(0x7f8ab8077620 msgr2=0x7f8ab8077a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.953+0000 7f8abe4ca640 1 -- 172.21.15.28:0/2818235321 shutdown_connections 2024-04-25T01:40:23.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.953+0000 7f8abe4ca640 1 -- 172.21.15.28:0/2818235321 wait complete. 2024-04-25T01:40:23.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.953+0000 7f8abe4ca640 1 Processor -- start 2024-04-25T01:40:23.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.953+0000 7f8abe4ca640 1 -- start start 2024-04-25T01:40:23.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.954+0000 7f8abe4ca640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ab8069d60 0x7f8ab8197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.954+0000 7f8abe4ca640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ab806a720 0x7f8ab81984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.954+0000 7f8abe4ca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab8107840 0x7f8ab819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.954+0000 7f8abe4ca640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab8111b10 con 0x7f8ab8107840 2024-04-25T01:40:23.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.954+0000 7f8abe4ca640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab8111990 con 0x7f8ab8069d60 2024-04-25T01:40:23.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.954+0000 7f8abe4ca640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab8111c90 con 0x7f8ab806a720 2024-04-25T01:40:23.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.954+0000 7f8abca67640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab8107840 0x7f8ab819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.954+0000 7f8abca67640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab8107840 0x7f8ab819c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35026/0 (socket says 172.21.15.28:35026) 2024-04-25T01:40:23.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.954+0000 7f8abca67640 1 -- 172.21.15.28:0/503407219 learned_addr learned my addr 172.21.15.28:0/503407219 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.954+0000 7f8ab7fff640 1 --2- 172.21.15.28:0/503407219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ab8069d60 0x7f8ab8197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.954+0000 7f8ab77fe640 1 --2- 172.21.15.28:0/503407219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ab806a720 0x7f8ab81984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.954+0000 7f8abca67640 1 -- 172.21.15.28:0/503407219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ab806a720 msgr2=0x7f8ab81984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.954+0000 7f8abca67640 1 --2- 172.21.15.28:0/503407219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ab806a720 0x7f8ab81984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.954+0000 7f8abca67640 1 -- 172.21.15.28:0/503407219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ab8069d60 msgr2=0x7f8ab8197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.954+0000 7f8abca67640 1 --2- 172.21.15.28:0/503407219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ab8069d60 0x7f8ab8197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.954+0000 7f8abca67640 1 -- 172.21.15.28:0/503407219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ab819cfa0 con 0x7f8ab8107840 2024-04-25T01:40:23.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.954+0000 7f8ab77fe640 1 --2- 172.21.15.28:0/503407219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ab806a720 0x7f8ab81984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:23.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.954+0000 7f8abca67640 1 --2- 172.21.15.28:0/503407219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab8107840 0x7f8ab819c870 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0009fd0 tx=0x7f8ab000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.955+0000 7f8ab57fa640 1 -- 172.21.15.28:0/503407219 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab0017070 con 0x7f8ab8107840 2024-04-25T01:40:23.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.955+0000 7f8ab57fa640 1 -- 172.21.15.28:0/503407219 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ab0003e50 con 0x7f8ab8107840 2024-04-25T01:40:23.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.955+0000 7f8abe4ca640 1 -- 172.21.15.28:0/503407219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ab819d290 con 0x7f8ab8107840 2024-04-25T01:40:23.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.955+0000 7f8ab57fa640 1 -- 172.21.15.28:0/503407219 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab00106c0 con 0x7f8ab8107840 2024-04-25T01:40:23.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.955+0000 7f8abe4ca640 1 -- 172.21.15.28:0/503407219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ab81a4b20 con 0x7f8ab8107840 2024-04-25T01:40:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.957+0000 7f8ab57fa640 1 -- 172.21.15.28:0/503407219 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8ab0003970 con 0x7f8ab8107840 2024-04-25T01:40:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.957+0000 7f8ac11d7740 1 Processor -- start 2024-04-25T01:40:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.957+0000 7f8ac11d7740 1 -- start start 2024-04-25T01:40:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.957+0000 7f8ac11d7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64dc580 0x5585e64dc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.957+0000 7f8ac11d7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64dcea0 0x5585e64e7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.957+0000 7f8ac11d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64e7570 0x5585e64e9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.957+0000 7f8ac11d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5585e6441670 con 0x5585e64e7570 2024-04-25T01:40:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.957+0000 7f8ac11d7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5585e63e9970 con 0x5585e64dcea0 2024-04-25T01:40:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.957+0000 7f8ac11d7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5585e643d630 con 0x5585e64dc580 2024-04-25T01:40:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.957+0000 7f8ab77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64e7570 0x5585e64e9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.958+0000 7f8ab77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64e7570 0x5585e64e9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35042/0 (socket says 172.21.15.28:35042) 2024-04-25T01:40:23.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.958+0000 7f8ab57fa640 1 --2- 172.21.15.28:0/503407219 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8aa40a2d80 0x7f8aa40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.958+0000 7f8ab77fe640 1 -- 172.21.15.28:0/1989507363 learned_addr learned my addr 172.21.15.28:0/1989507363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.958+0000 7f8ab7fff640 1 --2- 172.21.15.28:0/1989507363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64dcea0 0x5585e64e7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.958+0000 7f8ab77fe640 1 --2- 172.21.15.28:0/1989507363 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64dc580 0x5585e64dc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.958+0000 7f8abca67640 1 --2- 172.21.15.28:0/503407219 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8aa40a2d80 0x7f8aa40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.958+0000 7f8ab77fe640 1 -- 172.21.15.28:0/1989507363 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64dc580 msgr2=0x5585e64dc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.958+0000 7f8ab77fe640 1 --2- 172.21.15.28:0/1989507363 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64dc580 0x5585e64dc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.958+0000 7f8ab77fe640 1 -- 172.21.15.28:0/1989507363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64dcea0 msgr2=0x5585e64e7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.958+0000 7f8ab77fe640 1 --2- 172.21.15.28:0/1989507363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64dcea0 0x5585e64e7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.958+0000 7f8ab77fe640 1 -- 172.21.15.28:0/1989507363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585e640d400 con 0x5585e64e7570 2024-04-25T01:40:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.958+0000 7f8ab7fff640 1 --2- 172.21.15.28:0/1989507363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64dcea0 0x5585e64e7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:40:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.958+0000 7f8ab57fa640 1 -- 172.21.15.28:0/503407219 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f8ab00cf550 con 0x7f8ab8107840 2024-04-25T01:40:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.958+0000 7f8ab77fe640 1 --2- 172.21.15.28:0/1989507363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64e7570 0x5585e64e9960 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0013890 tx=0x7f8aa000b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ae87bc2de8a4546 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.958+0000 7f8aae7fc640 1 -- 172.21.15.28:0/1989507363 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa000e8f0 con 0x5585e64e7570 2024-04-25T01:40:23.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.958+0000 7f8abca67640 1 --2- 172.21.15.28:0/503407219 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8aa40a2d80 0x7f8aa40a5240 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0097ae0 tx=0x7f8ab0004020 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.958+0000 7f8aae7fc640 1 -- 172.21.15.28:0/1989507363 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8aa00113e0 con 0x5585e64e7570 2024-04-25T01:40:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.958+0000 7f8aae7fc640 1 -- 172.21.15.28:0/1989507363 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0010630 con 0x5585e64e7570 2024-04-25T01:40:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.959+0000 7f8ac11d7740 1 -- 172.21.15.28:0/1989507363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64e7570 msgr2=0x5585e64e9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.959+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/1989507363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64e7570 0x5585e64e9960 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0013890 tx=0x7f8aa000b420 comp rx=0 tx=0).stop 2024-04-25T01:40:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.959+0000 7f8ac11d7740 1 -- 172.21.15.28:0/1989507363 shutdown_connections 2024-04-25T01:40:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.959+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/1989507363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64e7570 0x5585e64e9960 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.959+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/1989507363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64dcea0 0x5585e64e7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.959+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/1989507363 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64dc580 0x5585e64dc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.959+0000 7f8ac11d7740 1 -- 172.21.15.28:0/1989507363 >> 172.21.15.28:0/1989507363 conn(0x5585e64d88a0 msgr2=0x5585e64d8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.959+0000 7f8ac11d7740 1 -- 172.21.15.28:0/1989507363 shutdown_connections 2024-04-25T01:40:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.959+0000 7f8ac11d7740 1 -- 172.21.15.28:0/1989507363 wait complete. 2024-04-25T01:40:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.959+0000 7f8ac11d7740 1 -- start start 2024-04-25T01:40:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8ac11d7740 1 Processor -- start 2024-04-25T01:40:23.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8ac11d7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64dc580 0x5585e64e6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8ac11d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dcea0 0x5585e64e4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8ac11d7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64e7570 0x5585e64e4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8ac11d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5585e64ea360 con 0x5585e64dcea0 2024-04-25T01:40:23.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8ac11d7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5585e63fd050 con 0x5585e64e7570 2024-04-25T01:40:23.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8ac11d7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5585e64ea4e0 con 0x5585e64dc580 2024-04-25T01:40:23.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8ac11d7740 10 client.0 _ll_register_callbacks cb 0x5585e6154be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:23.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8ab7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dcea0 0x5585e64e4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8ab7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dcea0 0x5585e64e4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35054/0 (socket says 172.21.15.28:35054) 2024-04-25T01:40:23.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8ab7fff640 1 -- 172.21.15.28:0/3114992500 learned_addr learned my addr 172.21.15.28:0/3114992500 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8ab77fe640 1 --2- 172.21.15.28:0/3114992500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64dc580 0x5585e64e6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8ab7fff640 1 -- 172.21.15.28:0/3114992500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64dc580 msgr2=0x5585e64e6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8ab7fff640 1 --2- 172.21.15.28:0/3114992500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64dc580 0x5585e64e6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8ab7fff640 1 -- 172.21.15.28:0/3114992500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64e7570 msgr2=0x5585e64e4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8ab7fff640 1 --2- 172.21.15.28:0/3114992500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64e7570 0x5585e64e4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8ab7fff640 1 -- 172.21.15.28:0/3114992500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585e63cedc0 con 0x5585e64dcea0 2024-04-25T01:40:23.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8ab77fe640 1 --2- 172.21.15.28:0/3114992500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64dc580 0x5585e64e6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:23.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8ab7fff640 1 --2- 172.21.15.28:0/3114992500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dcea0 0x5585e64e4250 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f8aa80314e0 tx=0x7f8aa8004420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee0e92c0c552b377 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.960+0000 7f8a977fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:23.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.961+0000 7f8a977fe640 1 -- 172.21.15.28:0/3114992500 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa8035040 con 0x5585e64dcea0 2024-04-25T01:40:23.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.961+0000 7f8a977fe640 1 -- 172.21.15.28:0/3114992500 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f8aa8003ac0 con 0x5585e64dcea0 2024-04-25T01:40:23.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.961+0000 7f8ac11d7740 1 -- 172.21.15.28:0/3114992500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8aa00113e0 con 0x5585e64dcea0 2024-04-25T01:40:23.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.961+0000 7f8a977fe640 1 -- 172.21.15.28:0/3114992500 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8aa804e050 con 0x5585e64dcea0 2024-04-25T01:40:23.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.961+0000 7f8ac11d7740 10 client.18327 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:23.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.961+0000 7f8ac11d7740 1 -- 172.21.15.28:0/3114992500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5585e61553f0 con 0x5585e64dcea0 2024-04-25T01:40:23.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.961+0000 7f8a977fe640 1 -- 172.21.15.28:0/3114992500 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8aa00113e0 con 0x5585e64dcea0 2024-04-25T01:40:23.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.961+0000 7f8a977fe640 1 -- 172.21.15.28:0/3114992500 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8aa803c440 con 0x5585e64dcea0 2024-04-25T01:40:23.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.961+0000 7f8ac11d7740 10 client.18327 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:23.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.961+0000 7f8ac11d7740 10 client.18327 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:40:23.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.961+0000 7f8ac11d7740 20 client.18327 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:23.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.961+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/3114992500 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5585e64f7690 0x5585e64f9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.961+0000 7f8ac11d7740 4 client.18327 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:23.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.961+0000 7f8ac11d7740 1 -- 172.21.15.28:0/3114992500 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585e644ebb0 con 0x5585e64f7690 2024-04-25T01:40:23.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.961+0000 7f8ab77fe640 1 --2- 172.21.15.28:0/3114992500 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5585e64f7690 0x5585e64f9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.962+0000 7f8ab77fe640 1 --2- 172.21.15.28:0/3114992500 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5585e64f7690 0x5585e64f9b10 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ed2fe1f5907bb6e5 server_cookie=1aa4b8ee7e3fdd6c in_seq=0 out_seq=0 2024-04-25T01:40:23.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.962+0000 7f8a977fe640 10 client.18327 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:40:23.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.962+0000 7f8a977fe640 1 -- 172.21.15.28:0/3114992500 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5585e644ebb0 con 0x5585e64f7690 2024-04-25T01:40:23.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.962+0000 7f8a977fe640 10 client.18327 handle_command_reply: tid=0 2024-04-25T01:40:23.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.963+0000 7f8ac11d7740 1 client.18327 shutdown 2024-04-25T01:40:23.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.963+0000 7f8ac11d7740 1 -- 172.21.15.28:0/3114992500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dcea0 msgr2=0x5585e64e4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.963+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/3114992500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dcea0 0x5585e64e4250 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f8aa80314e0 tx=0x7f8aa8004420 comp rx=0 tx=0).stop 2024-04-25T01:40:23.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.963+0000 7f8ac11d7740 1 -- 172.21.15.28:0/3114992500 shutdown_connections 2024-04-25T01:40:23.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.963+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/3114992500 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64e7570 0x5585e64e4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.963+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/3114992500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dcea0 0x5585e64e4250 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.963+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/3114992500 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5585e64f7690 0x5585e64f9b10 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.963+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/3114992500 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64dc580 0x5585e64e6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.963+0000 7f8a977fe640 0 client.18327 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:40:23.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.963+0000 7f8ac11d7740 1 -- 172.21.15.28:0/3114992500 >> 172.21.15.28:0/3114992500 conn(0x5585e64d88a0 msgr2=0x5585e64d7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.963+0000 7f8ac11d7740 1 -- 172.21.15.28:0/3114992500 shutdown_connections 2024-04-25T01:40:23.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.963+0000 7f8ac11d7740 1 -- 172.21.15.28:0/3114992500 wait complete. 2024-04-25T01:40:23.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.963+0000 7f8ac11d7740 20 client.18327 trim_cache size 0 max 16384 2024-04-25T01:40:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.976+0000 7f8ac11d7740 1 Processor -- start 2024-04-25T01:40:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.976+0000 7f8ac11d7740 1 -- start start 2024-04-25T01:40:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8ac11d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dc580 0x5585e64f3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8ac11d7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64dcea0 0x5585e64ff3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8ac11d7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64e7570 0x5585e64ff920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8ac11d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab81aeac0 con 0x5585e64dc580 2024-04-25T01:40:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8ac11d7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab81b0600 con 0x5585e64dcea0 2024-04-25T01:40:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8ac11d7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab81af190 con 0x5585e64e7570 2024-04-25T01:40:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8ab77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dc580 0x5585e64f3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8ab77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dc580 0x5585e64f3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35060/0 (socket says 172.21.15.28:35060) 2024-04-25T01:40:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8ab77fe640 1 -- 172.21.15.28:0/710379167 learned_addr learned my addr 172.21.15.28:0/710379167 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8ab77fe640 1 --2- 172.21.15.28:0/710379167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64e7570 0x5585e64ff920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8ab7fff640 1 --2- 172.21.15.28:0/710379167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64dcea0 0x5585e64ff3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8ab77fe640 1 -- 172.21.15.28:0/710379167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64e7570 msgr2=0x5585e64ff920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8ab77fe640 1 --2- 172.21.15.28:0/710379167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64e7570 0x5585e64ff920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8ab77fe640 1 -- 172.21.15.28:0/710379167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64dcea0 msgr2=0x5585e64ff3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8ab77fe640 1 --2- 172.21.15.28:0/710379167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64dcea0 0x5585e64ff3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8ab77fe640 1 -- 172.21.15.28:0/710379167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585e64e4bd0 con 0x5585e64dc580 2024-04-25T01:40:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8ab7fff640 1 --2- 172.21.15.28:0/710379167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64dcea0 0x5585e64ff3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8ab77fe640 1 --2- 172.21.15.28:0/710379167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dc580 0x5585e64f3220 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0011ed0 tx=0x7f8aa0011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a9772b6ef9052e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8aadffb640 1 -- 172.21.15.28:0/710379167 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0031070 con 0x5585e64dc580 2024-04-25T01:40:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.977+0000 7f8aadffb640 1 -- 172.21.15.28:0/710379167 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8aa0013520 con 0x5585e64dc580 2024-04-25T01:40:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.978+0000 7f8aadffb640 1 -- 172.21.15.28:0/710379167 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0029870 con 0x5585e64dc580 2024-04-25T01:40:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.978+0000 7f8ac11d7740 1 -- 172.21.15.28:0/710379167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dc580 msgr2=0x5585e64f3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.978+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/710379167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dc580 0x5585e64f3220 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0011ed0 tx=0x7f8aa0011e00 comp rx=0 tx=0).stop 2024-04-25T01:40:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.978+0000 7f8ac11d7740 1 -- 172.21.15.28:0/710379167 shutdown_connections 2024-04-25T01:40:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.978+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/710379167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64e7570 0x5585e64ff920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.978+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/710379167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64dcea0 0x5585e64ff3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.978+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/710379167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dc580 0x5585e64f3220 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.978+0000 7f8ac11d7740 1 -- 172.21.15.28:0/710379167 >> 172.21.15.28:0/710379167 conn(0x5585e64d88a0 msgr2=0x5585e64d72b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.978+0000 7f8ac11d7740 1 -- 172.21.15.28:0/710379167 shutdown_connections 2024-04-25T01:40:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.978+0000 7f8ac11d7740 1 -- 172.21.15.28:0/710379167 wait complete. 2024-04-25T01:40:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.978+0000 7f8ac11d7740 1 -- start start 2024-04-25T01:40:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.979+0000 7f8ac11d7740 1 Processor -- start 2024-04-25T01:40:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.979+0000 7f8ac11d7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64dc580 0x5585e64ed1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.979+0000 7f8ac11d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dcea0 0x5585e64ed6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.979+0000 7f8ac11d7740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64e7570 0x5585e64edc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.979+0000 7f8ac11d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab81aff30 con 0x5585e64dcea0 2024-04-25T01:40:23.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.979+0000 7f8ac11d7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab81af860 con 0x5585e64dc580 2024-04-25T01:40:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.979+0000 7f8ac11d7740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab81acac0 con 0x5585e64e7570 2024-04-25T01:40:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.979+0000 7f8ac11d7740 10 client.0 _ll_register_callbacks cb 0x7f8aa8009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.979+0000 7f8ab7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dcea0 0x5585e64ed6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.979+0000 7f8ab7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dcea0 0x5585e64ed6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35062/0 (socket says 172.21.15.28:35062) 2024-04-25T01:40:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.979+0000 7f8ab7fff640 1 -- 172.21.15.28:0/1122796528 learned_addr learned my addr 172.21.15.28:0/1122796528 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.979+0000 7f8ab7fff640 1 -- 172.21.15.28:0/1122796528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64e7570 msgr2=0x5585e64edc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.979+0000 7f8ab7fff640 1 --2- 172.21.15.28:0/1122796528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64e7570 0x5585e64edc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.979+0000 7f8ab7fff640 1 -- 172.21.15.28:0/1122796528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64dc580 msgr2=0x5585e64ed1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.979+0000 7f8ab7fff640 1 --2- 172.21.15.28:0/1122796528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64dc580 0x5585e64ed1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.979+0000 7f8ab7fff640 1 -- 172.21.15.28:0/1122796528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585e621f7d0 con 0x5585e64dcea0 2024-04-25T01:40:23.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.979+0000 7f8ab7fff640 1 --2- 172.21.15.28:0/1122796528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dcea0 0x5585e64ed6e0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f8aa803c890 tx=0x7f8aa803c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c407d4a92c194d6b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.979+0000 7f8a977fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.980+0000 7f8a977fe640 1 -- 172.21.15.28:0/1122796528 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa8035070 con 0x5585e64dcea0 2024-04-25T01:40:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.980+0000 7f8a977fe640 1 -- 172.21.15.28:0/1122796528 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f8aa8005240 con 0x5585e64dcea0 2024-04-25T01:40:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.980+0000 7f8ac11d7740 1 -- 172.21.15.28:0/1122796528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8aa0013520 con 0x5585e64dcea0 2024-04-25T01:40:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.980+0000 7f8a977fe640 1 -- 172.21.15.28:0/1122796528 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8aa803cc90 con 0x5585e64dcea0 2024-04-25T01:40:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.980+0000 7f8ac11d7740 10 client.18333 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.980+0000 7f8ac11d7740 1 -- 172.21.15.28:0/1122796528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5585e655dd10 con 0x5585e64dcea0 2024-04-25T01:40:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.980+0000 7f8a977fe640 1 -- 172.21.15.28:0/1122796528 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8aa0013520 con 0x5585e64dcea0 2024-04-25T01:40:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.980+0000 7f8a977fe640 1 -- 172.21.15.28:0/1122796528 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8aa8003950 con 0x5585e64dcea0 2024-04-25T01:40:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.980+0000 7f8ac11d7740 10 client.18333 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.980+0000 7f8ac11d7740 10 client.18333 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:40:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.980+0000 7f8ac11d7740 20 client.18333 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.980+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/1122796528 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5585e655e430 0x5585e6560820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.980+0000 7f8ac11d7740 4 client.18333 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:23.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.980+0000 7f8ac11d7740 1 -- 172.21.15.28:0/1122796528 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5585e6448f00 con 0x5585e655e430 2024-04-25T01:40:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.980+0000 7f8ab77fe640 1 --2- 172.21.15.28:0/1122796528 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5585e655e430 0x5585e6560820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.981+0000 7f8ab77fe640 1 --2- 172.21.15.28:0/1122796528 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5585e655e430 0x5585e6560820 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=926505fd034693d7 server_cookie=e8dc1cbdf25949ea in_seq=0 out_seq=0 2024-04-25T01:40:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.981+0000 7f8a977fe640 10 client.18333 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:40:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.981+0000 7f8a977fe640 1 -- 172.21.15.28:0/1122796528 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5585e6448f00 con 0x5585e655e430 2024-04-25T01:40:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.981+0000 7f8a977fe640 10 client.18333 handle_command_reply: tid=0 2024-04-25T01:40:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.981+0000 7f8ac11d7740 1 client.18333 shutdown 2024-04-25T01:40:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.981+0000 7f8ac11d7740 1 -- 172.21.15.28:0/1122796528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dcea0 msgr2=0x5585e64ed6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.981+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/1122796528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dcea0 0x5585e64ed6e0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f8aa803c890 tx=0x7f8aa803c670 comp rx=0 tx=0).stop 2024-04-25T01:40:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.981+0000 7f8ac11d7740 1 -- 172.21.15.28:0/1122796528 shutdown_connections 2024-04-25T01:40:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.981+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/1122796528 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5585e655e430 0x5585e6560820 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.981+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/1122796528 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5585e64e7570 0x5585e64edc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.981+0000 7f8a977fe640 0 client.18333 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:40:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.981+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/1122796528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585e64dcea0 0x5585e64ed6e0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.981+0000 7f8ac11d7740 1 --2- 172.21.15.28:0/1122796528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5585e64dc580 0x5585e64ed1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.981+0000 7f8ac11d7740 1 -- 172.21.15.28:0/1122796528 >> 172.21.15.28:0/1122796528 conn(0x5585e64d88a0 msgr2=0x5585e64d7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.981+0000 7f8ac11d7740 1 -- 172.21.15.28:0/1122796528 shutdown_connections 2024-04-25T01:40:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.981+0000 7f8ac11d7740 1 -- 172.21.15.28:0/1122796528 wait complete. 2024-04-25T01:40:23.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.981+0000 7f8ac11d7740 20 client.18333 trim_cache size 0 max 16384 2024-04-25T01:40:23.973 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.982+0000 7f8abe4ca640 1 -- 172.21.15.28:0/503407219 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8aa40a2d80 msgr2=0x7f8aa40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.982+0000 7f8abe4ca640 1 --2- 172.21.15.28:0/503407219 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8aa40a2d80 0x7f8aa40a5240 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0097ae0 tx=0x7f8ab0004020 comp rx=0 tx=0).stop 2024-04-25T01:40:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.982+0000 7f8abe4ca640 1 -- 172.21.15.28:0/503407219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab8107840 msgr2=0x7f8ab819c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.982+0000 7f8abe4ca640 1 --2- 172.21.15.28:0/503407219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab8107840 0x7f8ab819c870 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0009fd0 tx=0x7f8ab000b260 comp rx=0 tx=0).stop 2024-04-25T01:40:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.982+0000 7f8abe4ca640 1 -- 172.21.15.28:0/503407219 shutdown_connections 2024-04-25T01:40:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.982+0000 7f8abe4ca640 1 --2- 172.21.15.28:0/503407219 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8aa40a2d80 0x7f8aa40a5240 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.982+0000 7f8abe4ca640 1 --2- 172.21.15.28:0/503407219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab8107840 0x7f8ab819c870 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.982+0000 7f8abe4ca640 1 --2- 172.21.15.28:0/503407219 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ab806a720 0x7f8ab81984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.982+0000 7f8abe4ca640 1 --2- 172.21.15.28:0/503407219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ab8069d60 0x7f8ab8197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.982+0000 7f8abe4ca640 1 -- 172.21.15.28:0/503407219 >> 172.21.15.28:0/503407219 conn(0x7f8ab8077620 msgr2=0x7f8ab80ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.982+0000 7f8abe4ca640 1 -- 172.21.15.28:0/503407219 shutdown_connections 2024-04-25T01:40:23.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:23.982+0000 7f8abe4ca640 1 -- 172.21.15.28:0/503407219 wait complete. 2024-04-25T01:40:23.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:40:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.096+0000 7ffbd3e95640 1 -- 172.21.15.28:0/4087303931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbcc06a300 msgr2=0x7ffbcc06a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.096+0000 7ffbd3e95640 1 --2- 172.21.15.28:0/4087303931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbcc06a300 0x7ffbcc06a760 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc009900 tx=0x7ffbbc02a410 comp rx=0 tx=0).stop 2024-04-25T01:40:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.096+0000 7ffbd3e95640 1 -- 172.21.15.28:0/4087303931 shutdown_connections 2024-04-25T01:40:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.096+0000 7ffbd3e95640 1 --2- 172.21.15.28:0/4087303931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbcc104170 0x7ffbcc1045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.096+0000 7ffbd3e95640 1 --2- 172.21.15.28:0/4087303931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbcc06a300 0x7ffbcc06a760 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.096+0000 7ffbd3e95640 1 --2- 172.21.15.28:0/4087303931 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffbcc106670 0x7ffbcc069dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.096+0000 7ffbd3e95640 1 -- 172.21.15.28:0/4087303931 >> 172.21.15.28:0/4087303931 conn(0x7ffbcc0776a0 msgr2=0x7ffbcc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.096+0000 7ffbd3e95640 1 -- 172.21.15.28:0/4087303931 shutdown_connections 2024-04-25T01:40:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.097+0000 7ffbd3e95640 1 -- 172.21.15.28:0/4087303931 wait complete. 2024-04-25T01:40:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.097+0000 7ffbd3e95640 1 Processor -- start 2024-04-25T01:40:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.097+0000 7ffbd3e95640 1 -- start start 2024-04-25T01:40:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.097+0000 7ffbd3e95640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbcc06a300 0x7ffbcc19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.097+0000 7ffbd3e95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbcc104170 0x7ffbcc19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.097+0000 7ffbd3e95640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffbcc106670 0x7ffbcc1a2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.097+0000 7ffbd3e95640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc118140 con 0x7ffbcc104170 2024-04-25T01:40:24.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.097+0000 7ffbd3e95640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc117fc0 con 0x7ffbcc06a300 2024-04-25T01:40:24.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.097+0000 7ffbd3e95640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1182c0 con 0x7ffbcc106670 2024-04-25T01:40:24.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.097+0000 7ffbd1430640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbcc104170 0x7ffbcc19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.097+0000 7ffbd1430640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbcc104170 0x7ffbcc19eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35076/0 (socket says 172.21.15.28:35076) 2024-04-25T01:40:24.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.097+0000 7ffbd1430640 1 -- 172.21.15.28:0/1342213778 learned_addr learned my addr 172.21.15.28:0/1342213778 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:24.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.098+0000 7ffbd1c31640 1 --2- 172.21.15.28:0/1342213778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbcc06a300 0x7ffbcc19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.098+0000 7ffbd2432640 1 --2- 172.21.15.28:0/1342213778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffbcc106670 0x7ffbcc1a2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.098+0000 7ffbd1430640 1 -- 172.21.15.28:0/1342213778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffbcc106670 msgr2=0x7ffbcc1a2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:24.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.098+0000 7ffbd1430640 1 --2- 172.21.15.28:0/1342213778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffbcc106670 0x7ffbcc1a2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.098+0000 7ffbd1430640 1 -- 172.21.15.28:0/1342213778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbcc06a300 msgr2=0x7ffbcc19e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:24.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.098+0000 7ffbd1430640 1 --2- 172.21.15.28:0/1342213778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbcc06a300 0x7ffbcc19e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.098+0000 7ffbd1430640 1 -- 172.21.15.28:0/1342213778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbcc1a35e0 con 0x7ffbcc104170 2024-04-25T01:40:24.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.098+0000 7ffbd2432640 1 --2- 172.21.15.28:0/1342213778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffbcc106670 0x7ffbcc1a2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:24.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.098+0000 7ffbd1430640 1 --2- 172.21.15.28:0/1342213778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbcc104170 0x7ffbcc19eb80 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc004770 tx=0x7ffbbc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:24.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.098+0000 7ffbc2ffd640 1 -- 172.21.15.28:0/1342213778 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc03f070 con 0x7ffbcc104170 2024-04-25T01:40:24.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.098+0000 7ffbc2ffd640 1 -- 172.21.15.28:0/1342213778 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbbc031ca0 con 0x7ffbcc104170 2024-04-25T01:40:24.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.098+0000 7ffbc2ffd640 1 -- 172.21.15.28:0/1342213778 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc035030 con 0x7ffbcc104170 2024-04-25T01:40:24.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.098+0000 7ffbd3e95640 1 -- 172.21.15.28:0/1342213778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbcc1a3870 con 0x7ffbcc104170 2024-04-25T01:40:24.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.098+0000 7ffbd3e95640 1 -- 172.21.15.28:0/1342213778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbcc1ab160 con 0x7ffbcc104170 2024-04-25T01:40:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.101+0000 7ffbc2ffd640 1 -- 172.21.15.28:0/1342213778 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ffbbc042070 con 0x7ffbcc104170 2024-04-25T01:40:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.101+0000 7ffbd6b9b740 1 Processor -- start 2024-04-25T01:40:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.101+0000 7ffbd6b9b740 1 -- start start 2024-04-25T01:40:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.101+0000 7ffbd6b9b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2580 0x558ace6e2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.101+0000 7ffbd6b9b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6e2ea0 0x558ace6ed030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.101+0000 7ffbd6b9b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558ace6ed570 0x558ace6ef960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.101+0000 7ffbd6b9b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558ace647670 con 0x558ace6e2580 2024-04-25T01:40:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.102+0000 7ffbd6b9b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558ace5ef970 con 0x558ace6ed570 2024-04-25T01:40:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.102+0000 7ffbd6b9b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x558ace643630 con 0x558ace6e2ea0 2024-04-25T01:40:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.102+0000 7ffbd2432640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2580 0x558ace6e2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.102+0000 7ffbd2432640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2580 0x558ace6e2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35080/0 (socket says 172.21.15.28:35080) 2024-04-25T01:40:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.102+0000 7ffbd2432640 1 -- 172.21.15.28:0/2143733503 learned_addr learned my addr 172.21.15.28:0/2143733503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.102+0000 7ffbc2ffd640 1 --2- 172.21.15.28:0/1342213778 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ffbb80a2d80 0x7ffbb80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.102+0000 7ffbd1c31640 1 --2- 172.21.15.28:0/2143733503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6e2ea0 0x558ace6ed030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.102+0000 7ffbd1430640 1 --2- 172.21.15.28:0/2143733503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558ace6ed570 0x558ace6ef960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.102+0000 7ffbd2432640 1 -- 172.21.15.28:0/2143733503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6e2ea0 msgr2=0x558ace6ed030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.102+0000 7ffbd2432640 1 --2- 172.21.15.28:0/2143733503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6e2ea0 0x558ace6ed030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.102+0000 7ffbd2432640 1 -- 172.21.15.28:0/2143733503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558ace6ed570 msgr2=0x558ace6ef960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.102+0000 7ffbd2432640 1 --2- 172.21.15.28:0/2143733503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558ace6ed570 0x558ace6ef960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.102+0000 7ffbd2432640 1 -- 172.21.15.28:0/2143733503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ace613400 con 0x558ace6e2580 2024-04-25T01:40:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.102+0000 7ffbc2ffd640 1 -- 172.21.15.28:0/1342213778 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7ffbbc0efb30 con 0x7ffbcc104170 2024-04-25T01:40:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.102+0000 7ffbd2432640 1 --2- 172.21.15.28:0/1342213778 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ffbb80a2d80 0x7ffbb80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.102+0000 7ffbd2432640 1 --2- 172.21.15.28:0/2143733503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2580 0x558ace6e2960 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7ffbc40105d0 tx=0x7ffbc4010a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b3c4d2bbaed36e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.102+0000 7ffbb7fff640 1 -- 172.21.15.28:0/2143733503 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4019070 con 0x558ace6e2580 2024-04-25T01:40:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.103+0000 7ffbb7fff640 1 -- 172.21.15.28:0/2143733503 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbc4026430 con 0x558ace6e2580 2024-04-25T01:40:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.103+0000 7ffbb7fff640 1 -- 172.21.15.28:0/2143733503 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc401d670 con 0x558ace6e2580 2024-04-25T01:40:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.103+0000 7ffbd2432640 1 --2- 172.21.15.28:0/1342213778 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ffbb80a2d80 0x7ffbb80a5240 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4020000 tx=0x7ffbc401e000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.103+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/2143733503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2580 msgr2=0x558ace6e2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.103+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/2143733503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2580 0x558ace6e2960 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7ffbc40105d0 tx=0x7ffbc4010a90 comp rx=0 tx=0).stop 2024-04-25T01:40:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.103+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/2143733503 shutdown_connections 2024-04-25T01:40:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.103+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/2143733503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558ace6ed570 0x558ace6ef960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.103+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/2143733503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6e2ea0 0x558ace6ed030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.103+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/2143733503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2580 0x558ace6e2960 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.103+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/2143733503 >> 172.21.15.28:0/2143733503 conn(0x558ace6de8a0 msgr2=0x558ace6dec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.103+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/2143733503 shutdown_connections 2024-04-25T01:40:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.103+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/2143733503 wait complete. 2024-04-25T01:40:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.104+0000 7ffbd6b9b740 1 -- start start 2024-04-25T01:40:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.104+0000 7ffbd6b9b740 1 Processor -- start 2024-04-25T01:40:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.104+0000 7ffbd6b9b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6e2580 0x558ace6eceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.104+0000 7ffbd6b9b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2ea0 0x558ace6ea1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.104+0000 7ffbd6b9b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558ace6ed570 0x558ace6ea6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.104+0000 7ffbd6b9b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558ace6f0360 con 0x558ace6e2ea0 2024-04-25T01:40:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.104+0000 7ffbd6b9b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558ace603050 con 0x558ace6ed570 2024-04-25T01:40:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.104+0000 7ffbd6b9b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x558ace6f04e0 con 0x558ace6e2580 2024-04-25T01:40:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.104+0000 7ffbd6b9b740 10 client.0 _ll_register_callbacks cb 0x558ace35abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.104+0000 7ffbd1430640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2ea0 0x558ace6ea1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.104+0000 7ffbd2432640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558ace6ed570 0x558ace6ea6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.104+0000 7ffbd1430640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2ea0 0x558ace6ea1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35090/0 (socket says 172.21.15.28:35090) 2024-04-25T01:40:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.104+0000 7ffbd1430640 1 -- 172.21.15.28:0/3556507197 learned_addr learned my addr 172.21.15.28:0/3556507197 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.104+0000 7ffbd1c31640 1 --2- 172.21.15.28:0/3556507197 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6e2580 0x558ace6eceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbd1430640 1 -- 172.21.15.28:0/3556507197 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6e2580 msgr2=0x558ace6eceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbd1430640 1 --2- 172.21.15.28:0/3556507197 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6e2580 0x558ace6eceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbd1430640 1 -- 172.21.15.28:0/3556507197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558ace6ed570 msgr2=0x558ace6ea6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbd1430640 1 --2- 172.21.15.28:0/3556507197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558ace6ed570 0x558ace6ea6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbd1430640 1 -- 172.21.15.28:0/3556507197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ace5d4dc0 con 0x558ace6e2ea0 2024-04-25T01:40:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbd1430640 1 --2- 172.21.15.28:0/3556507197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2ea0 0x558ace6ea1a0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc0ecf50 tx=0x7ffbbc02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a43b1033b6567a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbb57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbb57fa640 1 -- 172.21.15.28:0/3556507197 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc049070 con 0x558ace6e2ea0 2024-04-25T01:40:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbb57fa640 1 -- 172.21.15.28:0/3556507197 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7ffbbc0b8330 con 0x558ace6e2ea0 2024-04-25T01:40:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/3556507197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbc4026430 con 0x558ace6e2ea0 2024-04-25T01:40:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbb57fa640 1 -- 172.21.15.28:0/3556507197 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ffbbc0bc3c0 con 0x558ace6e2ea0 2024-04-25T01:40:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbd6b9b740 10 client.18345 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/3556507197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558ace35b3f0 con 0x558ace6e2ea0 2024-04-25T01:40:24.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbb57fa640 1 -- 172.21.15.28:0/3556507197 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbc4026430 con 0x558ace6e2ea0 2024-04-25T01:40:24.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbb57fa640 1 -- 172.21.15.28:0/3556507197 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ffbbc098b20 con 0x558ace6e2ea0 2024-04-25T01:40:24.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbd6b9b740 10 client.18345 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:24.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbd6b9b740 10 client.18345 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:40:24.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbd6b9b740 20 client.18345 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:24.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/3556507197 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x558ace6fd540 0x558ace6ff950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.105+0000 7ffbd6b9b740 4 client.18345 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:24.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.106+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/3556507197 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ace654bb0 con 0x558ace6fd540 2024-04-25T01:40:24.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.106+0000 7ffbd1c31640 1 --2- 172.21.15.28:0/3556507197 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x558ace6fd540 0x558ace6ff950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.106+0000 7ffbb57fa640 10 client.18345 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:40:24.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.106+0000 7ffbd1c31640 1 --2- 172.21.15.28:0/3556507197 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x558ace6fd540 0x558ace6ff950 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d3d9dab2b7e63a8a server_cookie=2581a34a2e08bb29 in_seq=0 out_seq=0 2024-04-25T01:40:24.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.107+0000 7ffbb57fa640 1 -- 172.21.15.28:0/3556507197 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558ace654bb0 con 0x558ace6fd540 2024-04-25T01:40:24.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.107+0000 7ffbb57fa640 10 client.18345 handle_command_reply: tid=0 2024-04-25T01:40:24.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.107+0000 7ffbd6b9b740 1 client.18345 shutdown 2024-04-25T01:40:24.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.107+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/3556507197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2ea0 msgr2=0x558ace6ea1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:24.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.107+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/3556507197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2ea0 0x558ace6ea1a0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc0ecf50 tx=0x7ffbbc02a410 comp rx=0 tx=0).stop 2024-04-25T01:40:24.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.107+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/3556507197 shutdown_connections 2024-04-25T01:40:24.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.107+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/3556507197 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x558ace6fd540 0x558ace6ff950 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.107+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/3556507197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558ace6ed570 0x558ace6ea6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.107+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/3556507197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2ea0 0x558ace6ea1a0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.107+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/3556507197 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6e2580 0x558ace6eceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.107+0000 7ffbb57fa640 0 client.18345 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:40:24.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.107+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/3556507197 >> 172.21.15.28:0/3556507197 conn(0x558ace6de8a0 msgr2=0x558ace6e46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:24.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.107+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/3556507197 shutdown_connections 2024-04-25T01:40:24.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.107+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/3556507197 wait complete. 2024-04-25T01:40:24.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.107+0000 7ffbd6b9b740 20 client.18345 trim_cache size 0 max 16384 2024-04-25T01:40:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.120+0000 7ffbd6b9b740 1 Processor -- start 2024-04-25T01:40:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.120+0000 7ffbd6b9b740 1 -- start start 2024-04-25T01:40:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.121+0000 7ffbd6b9b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6e2580 0x558ace6f9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.121+0000 7ffbd6b9b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2ea0 0x558ace705210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.121+0000 7ffbd6b9b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558ace6ed570 0x558ace705750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.121+0000 7ffbd6b9b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1b5130 con 0x558ace6e2ea0 2024-04-25T01:40:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.121+0000 7ffbd6b9b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1b6c70 con 0x558ace6ed570 2024-04-25T01:40:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.121+0000 7ffbd6b9b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1b5800 con 0x558ace6e2580 2024-04-25T01:40:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.121+0000 7ffbd1430640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2ea0 0x558ace705210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.121+0000 7ffbd1430640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2ea0 0x558ace705210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35102/0 (socket says 172.21.15.28:35102) 2024-04-25T01:40:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.121+0000 7ffbd1430640 1 -- 172.21.15.28:0/109776240 learned_addr learned my addr 172.21.15.28:0/109776240 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:24.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.121+0000 7ffbd1430640 1 -- 172.21.15.28:0/109776240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6e2580 msgr2=0x558ace6f9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.121+0000 7ffbd1430640 1 --2- 172.21.15.28:0/109776240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6e2580 0x558ace6f9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.121+0000 7ffbd1430640 1 -- 172.21.15.28:0/109776240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558ace6ed570 msgr2=0x558ace705750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.121+0000 7ffbd1430640 1 --2- 172.21.15.28:0/109776240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558ace6ed570 0x558ace705750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.121+0000 7ffbd1430640 1 -- 172.21.15.28:0/109776240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ace4257d0 con 0x558ace6e2ea0 2024-04-25T01:40:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.121+0000 7ffbd1430640 1 --2- 172.21.15.28:0/109776240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2ea0 0x558ace705210 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc031550 tx=0x7ffbbc0bbab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63cc307ed5d86ead server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.121+0000 7ffbb77fe640 1 -- 172.21.15.28:0/109776240 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc049050 con 0x558ace6e2ea0 2024-04-25T01:40:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.121+0000 7ffbb77fe640 1 -- 172.21.15.28:0/109776240 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbbc0d8070 con 0x558ace6e2ea0 2024-04-25T01:40:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.121+0000 7ffbb77fe640 1 -- 172.21.15.28:0/109776240 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc0bbcc0 con 0x558ace6e2ea0 2024-04-25T01:40:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.122+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/109776240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2ea0 msgr2=0x558ace705210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.122+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/109776240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2ea0 0x558ace705210 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc031550 tx=0x7ffbbc0bbab0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.122+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/109776240 shutdown_connections 2024-04-25T01:40:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.122+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/109776240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558ace6ed570 0x558ace705750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.122+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/109776240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2ea0 0x558ace705210 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.122+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/109776240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6e2580 0x558ace6f9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.122+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/109776240 >> 172.21.15.28:0/109776240 conn(0x558ace6de8a0 msgr2=0x558ace6dd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.122+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/109776240 shutdown_connections 2024-04-25T01:40:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.122+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/109776240 wait complete. 2024-04-25T01:40:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.122+0000 7ffbd6b9b740 1 -- start start 2024-04-25T01:40:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.122+0000 7ffbd6b9b740 1 Processor -- start 2024-04-25T01:40:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd6b9b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2580 0x558ace6f57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd6b9b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558ace6e2ea0 0x558ace6f2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd6b9b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6ed570 0x558ace6f3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd6b9b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1b65a0 con 0x558ace6e2580 2024-04-25T01:40:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd6b9b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1b5ed0 con 0x558ace6e2ea0 2024-04-25T01:40:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd6b9b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ffbcc1b3130 con 0x558ace6ed570 2024-04-25T01:40:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd6b9b740 10 client.0 _ll_register_callbacks cb 0x7ffbbc005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd1c31640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2580 0x558ace6f57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd1c31640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2580 0x558ace6f57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35112/0 (socket says 172.21.15.28:35112) 2024-04-25T01:40:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd1c31640 1 -- 172.21.15.28:0/3470515647 learned_addr learned my addr 172.21.15.28:0/3470515647 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd1430640 1 --2- 172.21.15.28:0/3470515647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558ace6e2ea0 0x558ace6f2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd2432640 1 --2- 172.21.15.28:0/3470515647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6ed570 0x558ace6f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd1c31640 1 -- 172.21.15.28:0/3470515647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6ed570 msgr2=0x558ace6f3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd1c31640 1 --2- 172.21.15.28:0/3470515647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6ed570 0x558ace6f3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd1c31640 1 -- 172.21.15.28:0/3470515647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558ace6e2ea0 msgr2=0x558ace6f2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd1c31640 1 --2- 172.21.15.28:0/3470515647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558ace6e2ea0 0x558ace6f2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd1c31640 1 -- 172.21.15.28:0/3470515647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ace6f3780 con 0x558ace6e2580 2024-04-25T01:40:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd2432640 1 --2- 172.21.15.28:0/3470515647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6ed570 0x558ace6f3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd1c31640 1 --2- 172.21.15.28:0/3470515647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2580 0x558ace6f57f0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x558ace700180 tx=0x7ffbc800bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd5a8e8c1352e47 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbb57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbb57fa640 1 -- 172.21.15.28:0/3470515647 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8003d10 con 0x558ace6e2580 2024-04-25T01:40:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbb57fa640 1 -- 172.21.15.28:0/3470515647 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7ffbc8005020 con 0x558ace6e2580 2024-04-25T01:40:24.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.123+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/3470515647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbbc0d8070 con 0x558ace6e2580 2024-04-25T01:40:24.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.124+0000 7ffbb57fa640 1 -- 172.21.15.28:0/3470515647 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ffbc802e050 con 0x558ace6e2580 2024-04-25T01:40:24.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.124+0000 7ffbd6b9b740 10 client.18351 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:24.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.124+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/3470515647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558ace7637a0 con 0x558ace6e2580 2024-04-25T01:40:24.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.124+0000 7ffbb57fa640 1 -- 172.21.15.28:0/3470515647 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbbc0d8070 con 0x558ace6e2580 2024-04-25T01:40:24.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.124+0000 7ffbb57fa640 1 -- 172.21.15.28:0/3470515647 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ffbc80201b0 con 0x558ace6e2580 2024-04-25T01:40:24.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.124+0000 7ffbd6b9b740 10 client.18351 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:24.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.124+0000 7ffbd6b9b740 10 client.18351 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:40:24.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.124+0000 7ffbd6b9b740 20 client.18351 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:24.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.124+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/3470515647 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x558ace763fa0 0x558ace7663c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.124+0000 7ffbd6b9b740 4 client.18351 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:24.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.124+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/3470515647 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558ace64ef00 con 0x558ace763fa0 2024-04-25T01:40:24.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.124+0000 7ffbd1430640 1 --2- 172.21.15.28:0/3470515647 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x558ace763fa0 0x558ace7663c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.125+0000 7ffbb57fa640 10 client.18351 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:40:24.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.125+0000 7ffbd1430640 1 --2- 172.21.15.28:0/3470515647 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x558ace763fa0 0x558ace7663c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=adb3c18531ffbecf server_cookie=9b8ddb040e16c642 in_seq=0 out_seq=0 2024-04-25T01:40:24.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.125+0000 7ffbb57fa640 1 -- 172.21.15.28:0/3470515647 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558ace64ef00 con 0x558ace763fa0 2024-04-25T01:40:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.125+0000 7ffbb57fa640 10 client.18351 handle_command_reply: tid=0 2024-04-25T01:40:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.125+0000 7ffbd6b9b740 1 client.18351 shutdown 2024-04-25T01:40:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.125+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/3470515647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2580 msgr2=0x558ace6f57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.125+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/3470515647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2580 0x558ace6f57f0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x558ace700180 tx=0x7ffbc800bdf0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.125+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/3470515647 shutdown_connections 2024-04-25T01:40:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.125+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/3470515647 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x558ace763fa0 0x558ace7663c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.125+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/3470515647 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558ace6ed570 0x558ace6f3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.125+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/3470515647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558ace6e2ea0 0x558ace6f2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.125+0000 7ffbd6b9b740 1 --2- 172.21.15.28:0/3470515647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ace6e2580 0x558ace6f57f0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.125+0000 7ffbb57fa640 0 client.18351 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:40:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.125+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/3470515647 >> 172.21.15.28:0/3470515647 conn(0x558ace6de8a0 msgr2=0x558ace6ddb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.125+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/3470515647 shutdown_connections 2024-04-25T01:40:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.125+0000 7ffbd6b9b740 1 -- 172.21.15.28:0/3470515647 wait complete. 2024-04-25T01:40:24.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.125+0000 7ffbd6b9b740 20 client.18351 trim_cache size 0 max 16384 2024-04-25T01:40:24.117 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:24.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.126+0000 7ffbd3e95640 1 -- 172.21.15.28:0/1342213778 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ffbb80a2d80 msgr2=0x7ffbb80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:24.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.126+0000 7ffbd3e95640 1 --2- 172.21.15.28:0/1342213778 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ffbb80a2d80 0x7ffbb80a5240 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4020000 tx=0x7ffbc401e000 comp rx=0 tx=0).stop 2024-04-25T01:40:24.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.126+0000 7ffbd3e95640 1 -- 172.21.15.28:0/1342213778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbcc104170 msgr2=0x7ffbcc19eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:24.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.126+0000 7ffbd3e95640 1 --2- 172.21.15.28:0/1342213778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbcc104170 0x7ffbcc19eb80 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc004770 tx=0x7ffbbc003c10 comp rx=0 tx=0).stop 2024-04-25T01:40:24.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.126+0000 7ffbd3e95640 1 -- 172.21.15.28:0/1342213778 shutdown_connections 2024-04-25T01:40:24.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.126+0000 7ffbd3e95640 1 --2- 172.21.15.28:0/1342213778 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ffbb80a2d80 0x7ffbb80a5240 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.126+0000 7ffbd3e95640 1 --2- 172.21.15.28:0/1342213778 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ffbcc106670 0x7ffbcc1a2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.126+0000 7ffbd3e95640 1 --2- 172.21.15.28:0/1342213778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbcc104170 0x7ffbcc19eb80 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.126+0000 7ffbd3e95640 1 --2- 172.21.15.28:0/1342213778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffbcc06a300 0x7ffbcc19e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.126+0000 7ffbd3e95640 1 -- 172.21.15.28:0/1342213778 >> 172.21.15.28:0/1342213778 conn(0x7ffbcc0776a0 msgr2=0x7ffbcc103150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:24.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.126+0000 7ffbd3e95640 1 -- 172.21.15.28:0/1342213778 shutdown_connections 2024-04-25T01:40:24.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.126+0000 7ffbd3e95640 1 -- 172.21.15.28:0/1342213778 wait complete. 2024-04-25T01:40:24.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:23 smithi028 ceph-mon[29390]: pgmap v226: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:40:24.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:23 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:40:24.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:23 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:40:24.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:23 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:40:24.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:23 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:40:24.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:23 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:40:24.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:23 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2292144613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:40:24.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:40:24.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:23 smithi130 ceph-mon[32702]: pgmap v226: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:40:24.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:23 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:40:24.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:23 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:40:24.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:23 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:40:24.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:23 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:40:24.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:23 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:40:24.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:23 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2292144613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:40:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.241+0000 7f33751a2640 1 -- 172.21.15.28:0/1023153025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3370107730 msgr2=0x7f3370107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.241+0000 7f33751a2640 1 --2- 172.21.15.28:0/1023153025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3370107730 0x7f3370107b10 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f3360009930 tx=0x7f336002a420 comp rx=0 tx=0).stop 2024-04-25T01:40:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.241+0000 7f33751a2640 1 -- 172.21.15.28:0/1023153025 shutdown_connections 2024-04-25T01:40:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.241+0000 7f33751a2640 1 --2- 172.21.15.28:0/1023153025 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f337006a780 0x7f337010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.241+0000 7f33751a2640 1 --2- 172.21.15.28:0/1023153025 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3370069dc0 0x7f337006a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.241+0000 7f33751a2640 1 --2- 172.21.15.28:0/1023153025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3370107730 0x7f3370107b10 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.241+0000 7f33751a2640 1 -- 172.21.15.28:0/1023153025 >> 172.21.15.28:0/1023153025 conn(0x7f33700776a0 msgr2=0x7f3370077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.241+0000 7f33751a2640 1 -- 172.21.15.28:0/1023153025 shutdown_connections 2024-04-25T01:40:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.241+0000 7f33751a2640 1 -- 172.21.15.28:0/1023153025 wait complete. 2024-04-25T01:40:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.242+0000 7f33751a2640 1 Processor -- start 2024-04-25T01:40:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.242+0000 7f33751a2640 1 -- start start 2024-04-25T01:40:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.242+0000 7f33751a2640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3370069dc0 0x7f33701980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.242+0000 7f33751a2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f337006a780 0x7f3370198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.242+0000 7f33751a2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3370107730 0x7f337019c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.242+0000 7f33751a2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3370111b40 con 0x7f3370107730 2024-04-25T01:40:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.242+0000 7f33751a2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f33701119c0 con 0x7f337006a780 2024-04-25T01:40:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.242+0000 7f33751a2640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f3370111cc0 con 0x7f3370069dc0 2024-04-25T01:40:24.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.242+0000 7f336f59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3370107730 0x7f337019c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.242+0000 7f336f59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3370107730 0x7f337019c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35136/0 (socket says 172.21.15.28:35136) 2024-04-25T01:40:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.242+0000 7f336f59e640 1 -- 172.21.15.28:0/2812129818 learned_addr learned my addr 172.21.15.28:0/2812129818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.242+0000 7f336ed9d640 1 --2- 172.21.15.28:0/2812129818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3370069dc0 0x7f33701980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.242+0000 7f3367fff640 1 --2- 172.21.15.28:0/2812129818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f337006a780 0x7f3370198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.242+0000 7f336f59e640 1 -- 172.21.15.28:0/2812129818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3370069dc0 msgr2=0x7f33701980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.242+0000 7f336f59e640 1 --2- 172.21.15.28:0/2812129818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3370069dc0 0x7f33701980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.242+0000 7f336f59e640 1 -- 172.21.15.28:0/2812129818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f337006a780 msgr2=0x7f3370198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.242+0000 7f336f59e640 1 --2- 172.21.15.28:0/2812129818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f337006a780 0x7f3370198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.242+0000 7f336f59e640 1 -- 172.21.15.28:0/2812129818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337019d0b0 con 0x7f3370107730 2024-04-25T01:40:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.242+0000 7f3367fff640 1 --2- 172.21.15.28:0/2812129818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f337006a780 0x7f3370198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.243+0000 7f336f59e640 1 --2- 172.21.15.28:0/2812129818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3370107730 0x7f337019c980 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f336800cad0 tx=0x7f336800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.243+0000 7f336cd99640 1 -- 172.21.15.28:0/2812129818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336800ae20 con 0x7f3370107730 2024-04-25T01:40:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.243+0000 7f336cd99640 1 -- 172.21.15.28:0/2812129818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3368005020 con 0x7f3370107730 2024-04-25T01:40:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.243+0000 7f336cd99640 1 -- 172.21.15.28:0/2812129818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336800f6f0 con 0x7f3370107730 2024-04-25T01:40:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.243+0000 7f33751a2640 1 -- 172.21.15.28:0/2812129818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f337019d3a0 con 0x7f3370107730 2024-04-25T01:40:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.243+0000 7f33751a2640 1 -- 172.21.15.28:0/2812129818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33701a4c30 con 0x7f3370107730 2024-04-25T01:40:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.245+0000 7f336cd99640 1 -- 172.21.15.28:0/2812129818 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f33680051d0 con 0x7f3370107730 2024-04-25T01:40:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.245+0000 7f3377f94740 1 Processor -- start 2024-04-25T01:40:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.245+0000 7f3377f94740 1 -- start start 2024-04-25T01:40:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f3377f94740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755fee580 0x55e755fee960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f3377f94740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755feeea0 0x55e755ff9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f336cd99640 1 --2- 172.21.15.28:0/2812129818 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f335c0a2d80 0x7f335c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f3377f94740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755ff9570 0x55e755ffb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f3377f94740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e755f53670 con 0x55e755feeea0 2024-04-25T01:40:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f3377f94740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e755efb970 con 0x55e755fee580 2024-04-25T01:40:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f3377f94740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55e755f4f630 con 0x55e755ff9570 2024-04-25T01:40:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f336ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755feeea0 0x55e755ff9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f3367fff640 1 --2- 172.21.15.28:0/2812129818 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f335c0a2d80 0x7f335c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f336ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755feeea0 0x55e755ff9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35138/0 (socket says 172.21.15.28:35138) 2024-04-25T01:40:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f336ed9d640 1 -- 172.21.15.28:0/162902939 learned_addr learned my addr 172.21.15.28:0/162902939 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f336cd99640 1 -- 172.21.15.28:0/2812129818 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f33680cfb50 con 0x7f3370107730 2024-04-25T01:40:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f3367fff640 1 --2- 172.21.15.28:0/162902939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755fee580 0x55e755fee960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f336f59e640 1 --2- 172.21.15.28:0/162902939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755ff9570 0x55e755ffb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f336ed9d640 1 -- 172.21.15.28:0/162902939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755ff9570 msgr2=0x55e755ffb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f336ed9d640 1 --2- 172.21.15.28:0/162902939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755ff9570 0x55e755ffb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f336ed9d640 1 -- 172.21.15.28:0/162902939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755fee580 msgr2=0x55e755fee960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f336ed9d640 1 --2- 172.21.15.28:0/162902939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755fee580 0x55e755fee960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f336ed9d640 1 -- 172.21.15.28:0/162902939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e755f1f400 con 0x55e755feeea0 2024-04-25T01:40:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f3367fff640 1 --2- 172.21.15.28:0/2812129818 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f335c0a2d80 0x7f335c0a5240 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3358005d00 tx=0x7f335800ba90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.246+0000 7f336ed9d640 1 --2- 172.21.15.28:0/162902939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755feeea0 0x55e755ff9030 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f3360009930 tx=0x7f3360032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d766d79ee4141e66 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.247+0000 7f33657fa640 1 -- 172.21.15.28:0/162902939 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336003f070 con 0x55e755feeea0 2024-04-25T01:40:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.247+0000 7f33657fa640 1 -- 172.21.15.28:0/162902939 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33600317b0 con 0x55e755feeea0 2024-04-25T01:40:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.247+0000 7f33657fa640 1 -- 172.21.15.28:0/162902939 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3360035070 con 0x55e755feeea0 2024-04-25T01:40:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.247+0000 7f3377f94740 1 -- 172.21.15.28:0/162902939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755feeea0 msgr2=0x55e755ff9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.247+0000 7f3377f94740 1 --2- 172.21.15.28:0/162902939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755feeea0 0x55e755ff9030 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f3360009930 tx=0x7f3360032ce0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.247+0000 7f3377f94740 1 -- 172.21.15.28:0/162902939 shutdown_connections 2024-04-25T01:40:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.247+0000 7f3377f94740 1 --2- 172.21.15.28:0/162902939 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755ff9570 0x55e755ffb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.247+0000 7f3377f94740 1 --2- 172.21.15.28:0/162902939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755feeea0 0x55e755ff9030 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.247+0000 7f3377f94740 1 --2- 172.21.15.28:0/162902939 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755fee580 0x55e755fee960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.247+0000 7f3377f94740 1 -- 172.21.15.28:0/162902939 >> 172.21.15.28:0/162902939 conn(0x55e755fea8a0 msgr2=0x55e755feac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.248+0000 7f3377f94740 1 -- 172.21.15.28:0/162902939 shutdown_connections 2024-04-25T01:40:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.248+0000 7f3377f94740 1 -- 172.21.15.28:0/162902939 wait complete. 2024-04-25T01:40:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.248+0000 7f3377f94740 1 -- start start 2024-04-25T01:40:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.248+0000 7f3377f94740 1 Processor -- start 2024-04-25T01:40:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.248+0000 7f3377f94740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755fee580 0x55e755ff8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.248+0000 7f3377f94740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755feeea0 0x55e755ff61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.248+0000 7f3377f94740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755ff9570 0x55e755ff6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.248+0000 7f3377f94740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e755ffc300 con 0x55e755feeea0 2024-04-25T01:40:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.248+0000 7f3377f94740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e755f0f050 con 0x55e755ff9570 2024-04-25T01:40:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.248+0000 7f3377f94740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55e755ffc480 con 0x55e755fee580 2024-04-25T01:40:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.248+0000 7f3377f94740 10 client.0 _ll_register_callbacks cb 0x55e755c66be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.248+0000 7f3367fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755feeea0 0x55e755ff61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.248+0000 7f3367fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755feeea0 0x55e755ff61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35152/0 (socket says 172.21.15.28:35152) 2024-04-25T01:40:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.248+0000 7f3367fff640 1 -- 172.21.15.28:0/3811827892 learned_addr learned my addr 172.21.15.28:0/3811827892 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.248+0000 7f336f59e640 1 --2- 172.21.15.28:0/3811827892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755ff9570 0x55e755ff6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.248+0000 7f336ed9d640 1 --2- 172.21.15.28:0/3811827892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755fee580 0x55e755ff8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.249+0000 7f3367fff640 1 -- 172.21.15.28:0/3811827892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755fee580 msgr2=0x55e755ff8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.249+0000 7f3367fff640 1 --2- 172.21.15.28:0/3811827892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755fee580 0x55e755ff8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.249+0000 7f3367fff640 1 -- 172.21.15.28:0/3811827892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755ff9570 msgr2=0x55e755ff6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.249+0000 7f3367fff640 1 --2- 172.21.15.28:0/3811827892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755ff9570 0x55e755ff6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.249+0000 7f3367fff640 1 -- 172.21.15.28:0/3811827892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e755ee0dc0 con 0x55e755feeea0 2024-04-25T01:40:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.249+0000 7f336ed9d640 1 --2- 172.21.15.28:0/3811827892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755fee580 0x55e755ff8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.249+0000 7f336f59e640 1 --2- 172.21.15.28:0/3811827892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755ff9570 0x55e755ff6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:40:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.249+0000 7f3367fff640 1 --2- 172.21.15.28:0/3811827892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755feeea0 0x55e755ff61c0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f335800b840 tx=0x7f3358005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5e59b4ef2db0932 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.249+0000 7f33567fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.249+0000 7f33567fc640 1 -- 172.21.15.28:0/3811827892 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3358004220 con 0x55e755feeea0 2024-04-25T01:40:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.249+0000 7f33567fc640 1 -- 172.21.15.28:0/3811827892 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f33580043c0 con 0x55e755feeea0 2024-04-25T01:40:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.249+0000 7f3377f94740 1 -- 172.21.15.28:0/3811827892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33600317b0 con 0x55e755feeea0 2024-04-25T01:40:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.249+0000 7f33567fc640 1 -- 172.21.15.28:0/3811827892 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3358032050 con 0x55e755feeea0 2024-04-25T01:40:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.249+0000 7f3377f94740 10 client.18363 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.249+0000 7f3377f94740 1 -- 172.21.15.28:0/3811827892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e755c673f0 con 0x55e755feeea0 2024-04-25T01:40:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.249+0000 7f33567fc640 1 -- 172.21.15.28:0/3811827892 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33600317b0 con 0x55e755feeea0 2024-04-25T01:40:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.250+0000 7f33567fc640 1 -- 172.21.15.28:0/3811827892 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f335800f040 con 0x55e755feeea0 2024-04-25T01:40:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.250+0000 7f3377f94740 10 client.18363 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.250+0000 7f3377f94740 10 client.18363 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.250+0000 7f3377f94740 20 client.18363 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.250+0000 7f3377f94740 1 --2- 172.21.15.28:0/3811827892 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e756009500 0x55e75600b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.250+0000 7f3377f94740 4 client.18363 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.250+0000 7f3377f94740 1 -- 172.21.15.28:0/3811827892 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e755f60bb0 con 0x55e756009500 2024-04-25T01:40:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.250+0000 7f336ed9d640 1 --2- 172.21.15.28:0/3811827892 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e756009500 0x55e75600b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.250+0000 7f33567fc640 10 client.18363 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.250+0000 7f336ed9d640 1 --2- 172.21.15.28:0/3811827892 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e756009500 0x55e75600b990 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef8609fed3e0288a server_cookie=481b58285605441f in_seq=0 out_seq=0 2024-04-25T01:40:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.251+0000 7f33567fc640 1 -- 172.21.15.28:0/3811827892 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e755f60bb0 con 0x55e756009500 2024-04-25T01:40:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.251+0000 7f33567fc640 10 client.18363 handle_command_reply: tid=0 2024-04-25T01:40:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.251+0000 7f3377f94740 1 client.18363 shutdown 2024-04-25T01:40:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.251+0000 7f3377f94740 1 -- 172.21.15.28:0/3811827892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755feeea0 msgr2=0x55e755ff61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.251+0000 7f3377f94740 1 --2- 172.21.15.28:0/3811827892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755feeea0 0x55e755ff61c0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f335800b840 tx=0x7f3358005850 comp rx=0 tx=0).stop 2024-04-25T01:40:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.251+0000 7f3377f94740 1 -- 172.21.15.28:0/3811827892 shutdown_connections 2024-04-25T01:40:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.251+0000 7f3377f94740 1 --2- 172.21.15.28:0/3811827892 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e756009500 0x55e75600b990 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.251+0000 7f3377f94740 1 --2- 172.21.15.28:0/3811827892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755ff9570 0x55e755ff6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.251+0000 7f3377f94740 1 --2- 172.21.15.28:0/3811827892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755feeea0 0x55e755ff61c0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.251+0000 7f33567fc640 0 client.18363 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.251+0000 7f3377f94740 1 --2- 172.21.15.28:0/3811827892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755fee580 0x55e755ff8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.251+0000 7f3377f94740 1 -- 172.21.15.28:0/3811827892 >> 172.21.15.28:0/3811827892 conn(0x55e755fea8a0 msgr2=0x55e755fe9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.251+0000 7f3377f94740 1 -- 172.21.15.28:0/3811827892 shutdown_connections 2024-04-25T01:40:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.251+0000 7f3377f94740 1 -- 172.21.15.28:0/3811827892 wait complete. 2024-04-25T01:40:24.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.251+0000 7f3377f94740 20 client.18363 trim_cache size 0 max 16384 2024-04-25T01:40:24.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:23 smithi032 ceph-mon[32461]: pgmap v226: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:40:24.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:23 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:40:24.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:23 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:40:24.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:23 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:40:24.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:23 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:40:24.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:23 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:40:24.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:23 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2292144613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:40:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.264+0000 7f3377f94740 1 Processor -- start 2024-04-25T01:40:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.265+0000 7f3377f94740 1 -- start start 2024-04-25T01:40:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.265+0000 7f3377f94740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755fee580 0x55e7560051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.265+0000 7f3377f94740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755feeea0 0x55e756001ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.265+0000 7f3377f94740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755ff9570 0x55e755ff8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.265+0000 7f3377f94740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f33701aeb10 con 0x55e755ff9570 2024-04-25T01:40:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.265+0000 7f3377f94740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f33701b0650 con 0x55e755feeea0 2024-04-25T01:40:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.265+0000 7f3377f94740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f33701af1e0 con 0x55e755fee580 2024-04-25T01:40:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.265+0000 7f336f59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755ff9570 0x55e755ff8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.265+0000 7f336f59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755ff9570 0x55e755ff8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35162/0 (socket says 172.21.15.28:35162) 2024-04-25T01:40:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.265+0000 7f336f59e640 1 -- 172.21.15.28:0/996975805 learned_addr learned my addr 172.21.15.28:0/996975805 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.265+0000 7f336ed9d640 1 --2- 172.21.15.28:0/996975805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755fee580 0x55e7560051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.265+0000 7f3367fff640 1 --2- 172.21.15.28:0/996975805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755feeea0 0x55e756001ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.265+0000 7f336f59e640 1 -- 172.21.15.28:0/996975805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755fee580 msgr2=0x55e7560051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.265+0000 7f336f59e640 1 --2- 172.21.15.28:0/996975805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755fee580 0x55e7560051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.265+0000 7f336f59e640 1 -- 172.21.15.28:0/996975805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755feeea0 msgr2=0x55e756001ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.265+0000 7f336f59e640 1 --2- 172.21.15.28:0/996975805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755feeea0 0x55e756001ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.265+0000 7f336f59e640 1 -- 172.21.15.28:0/996975805 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e755d317d0 con 0x55e755ff9570 2024-04-25T01:40:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.265+0000 7f336f59e640 1 --2- 172.21.15.28:0/996975805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755ff9570 0x55e755ff8a00 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f336809bc10 tx=0x7f3368097fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ceafd6ffbc28d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.266+0000 7f3364ff9640 1 -- 172.21.15.28:0/996975805 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336801d040 con 0x55e755ff9570 2024-04-25T01:40:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.266+0000 7f3364ff9640 1 -- 172.21.15.28:0/996975805 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33680a48f0 con 0x55e755ff9570 2024-04-25T01:40:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.266+0000 7f3364ff9640 1 -- 172.21.15.28:0/996975805 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336809c140 con 0x55e755ff9570 2024-04-25T01:40:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.266+0000 7f3377f94740 1 -- 172.21.15.28:0/996975805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755ff9570 msgr2=0x55e755ff8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.266+0000 7f3377f94740 1 --2- 172.21.15.28:0/996975805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755ff9570 0x55e755ff8a00 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f336809bc10 tx=0x7f3368097fa0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.266+0000 7f3377f94740 1 -- 172.21.15.28:0/996975805 shutdown_connections 2024-04-25T01:40:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.266+0000 7f3377f94740 1 --2- 172.21.15.28:0/996975805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755ff9570 0x55e755ff8a00 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.266+0000 7f3377f94740 1 --2- 172.21.15.28:0/996975805 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755feeea0 0x55e756001ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.266+0000 7f3377f94740 1 --2- 172.21.15.28:0/996975805 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755fee580 0x55e7560051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.266+0000 7f3377f94740 1 -- 172.21.15.28:0/996975805 >> 172.21.15.28:0/996975805 conn(0x55e755fea8a0 msgr2=0x55e755ff05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.266+0000 7f3377f94740 1 -- 172.21.15.28:0/996975805 shutdown_connections 2024-04-25T01:40:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.266+0000 7f3377f94740 1 -- 172.21.15.28:0/996975805 wait complete. 2024-04-25T01:40:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f3377f94740 1 -- start start 2024-04-25T01:40:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f3377f94740 1 Processor -- start 2024-04-25T01:40:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f3377f94740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755fee580 0x55e755ff2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f3377f94740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755feeea0 0x55e755ff2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f3377f94740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755ff9570 0x55e755ff2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f3377f94740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f33701aff80 con 0x55e755ff9570 2024-04-25T01:40:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f3377f94740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f33701af8b0 con 0x55e755fee580 2024-04-25T01:40:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f3377f94740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f33701acb10 con 0x55e755feeea0 2024-04-25T01:40:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f3377f94740 10 client.0 _ll_register_callbacks cb 0x7f335800bee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f336f59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755ff9570 0x55e755ff2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f336f59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755ff9570 0x55e755ff2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35174/0 (socket says 172.21.15.28:35174) 2024-04-25T01:40:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f336f59e640 1 -- 172.21.15.28:0/1673325938 learned_addr learned my addr 172.21.15.28:0/1673325938 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f336ed9d640 1 --2- 172.21.15.28:0/1673325938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755fee580 0x55e755ff2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f336f59e640 1 -- 172.21.15.28:0/1673325938 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755feeea0 msgr2=0x55e755ff2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f336f59e640 1 --2- 172.21.15.28:0/1673325938 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755feeea0 0x55e755ff2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f336f59e640 1 -- 172.21.15.28:0/1673325938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755fee580 msgr2=0x55e755ff2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f336f59e640 1 --2- 172.21.15.28:0/1673325938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755fee580 0x55e755ff2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f336f59e640 1 -- 172.21.15.28:0/1673325938 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e755ff36a0 con 0x55e755ff9570 2024-04-25T01:40:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f336ed9d640 1 --2- 172.21.15.28:0/1673325938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755fee580 0x55e755ff2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f336f59e640 1 --2- 172.21.15.28:0/1673325938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755ff9570 0x55e755ff2ed0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f336800a6e0 tx=0x7f33680b5020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=848c90013e5476cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.267+0000 7f33567fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.268+0000 7f33567fc640 1 -- 172.21.15.28:0/1673325938 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336806a020 con 0x55e755ff9570 2024-04-25T01:40:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.268+0000 7f33567fc640 1 -- 172.21.15.28:0/1673325938 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f33680ba640 con 0x55e755ff9570 2024-04-25T01:40:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.268+0000 7f3377f94740 1 -- 172.21.15.28:0/1673325938 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33680a48f0 con 0x55e755ff9570 2024-04-25T01:40:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.268+0000 7f33567fc640 1 -- 172.21.15.28:0/1673325938 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f336809c0e0 con 0x55e755ff9570 2024-04-25T01:40:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.268+0000 7f3377f94740 10 client.18369 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.268+0000 7f3377f94740 1 -- 172.21.15.28:0/1673325938 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e756074060 con 0x55e755ff9570 2024-04-25T01:40:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.268+0000 7f33567fc640 1 -- 172.21.15.28:0/1673325938 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33680a48f0 con 0x55e755ff9570 2024-04-25T01:40:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.268+0000 7f33567fc640 1 -- 172.21.15.28:0/1673325938 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f336809ab40 con 0x55e755ff9570 2024-04-25T01:40:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.268+0000 7f3377f94740 10 client.18369 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.268+0000 7f3377f94740 10 client.18369 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.268+0000 7f3377f94740 20 client.18369 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.268+0000 7f3377f94740 1 --2- 172.21.15.28:0/1673325938 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e756074590 0x55e756076980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:24.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.268+0000 7f3377f94740 4 client.18369 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:24.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.268+0000 7f3377f94740 1 -- 172.21.15.28:0/1673325938 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e755f5af00 con 0x55e756074590 2024-04-25T01:40:24.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.268+0000 7f336ed9d640 1 --2- 172.21.15.28:0/1673325938 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e756074590 0x55e756076980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:24.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.269+0000 7f33567fc640 10 client.18369 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:24.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.269+0000 7f336ed9d640 1 --2- 172.21.15.28:0/1673325938 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e756074590 0x55e756076980 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a58d4064a2cac18e server_cookie=faebee294315e804 in_seq=0 out_seq=0 2024-04-25T01:40:24.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.269+0000 7f33567fc640 1 -- 172.21.15.28:0/1673325938 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e755f5af00 con 0x55e756074590 2024-04-25T01:40:24.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.269+0000 7f33567fc640 10 client.18369 handle_command_reply: tid=0 2024-04-25T01:40:24.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.269+0000 7f3377f94740 1 client.18369 shutdown 2024-04-25T01:40:24.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.269+0000 7f3377f94740 1 -- 172.21.15.28:0/1673325938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755ff9570 msgr2=0x55e755ff2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:24.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.269+0000 7f3377f94740 1 --2- 172.21.15.28:0/1673325938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755ff9570 0x55e755ff2ed0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f336800a6e0 tx=0x7f33680b5020 comp rx=0 tx=0).stop 2024-04-25T01:40:24.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.269+0000 7f3377f94740 1 -- 172.21.15.28:0/1673325938 shutdown_connections 2024-04-25T01:40:24.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.269+0000 7f3377f94740 1 --2- 172.21.15.28:0/1673325938 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55e756074590 0x55e756076980 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.269+0000 7f3377f94740 1 --2- 172.21.15.28:0/1673325938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e755ff9570 0x55e755ff2ed0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.269+0000 7f33567fc640 0 client.18369 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:24.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.269+0000 7f3377f94740 1 --2- 172.21.15.28:0/1673325938 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e755feeea0 0x55e755ff2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.269+0000 7f3377f94740 1 --2- 172.21.15.28:0/1673325938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e755fee580 0x55e755ff2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.269+0000 7f3377f94740 1 -- 172.21.15.28:0/1673325938 >> 172.21.15.28:0/1673325938 conn(0x55e755fea8a0 msgr2=0x55e755ff0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:24.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.269+0000 7f3377f94740 1 -- 172.21.15.28:0/1673325938 shutdown_connections 2024-04-25T01:40:24.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.269+0000 7f3377f94740 1 -- 172.21.15.28:0/1673325938 wait complete. 2024-04-25T01:40:24.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.269+0000 7f3377f94740 20 client.18369 trim_cache size 0 max 16384 2024-04-25T01:40:24.261 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:24.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.270+0000 7f33751a2640 1 -- 172.21.15.28:0/2812129818 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f335c0a2d80 msgr2=0x7f335c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:24.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.270+0000 7f33751a2640 1 --2- 172.21.15.28:0/2812129818 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f335c0a2d80 0x7f335c0a5240 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3358005d00 tx=0x7f335800ba90 comp rx=0 tx=0).stop 2024-04-25T01:40:24.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.270+0000 7f33751a2640 1 -- 172.21.15.28:0/2812129818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3370107730 msgr2=0x7f337019c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:24.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.270+0000 7f33751a2640 1 --2- 172.21.15.28:0/2812129818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3370107730 0x7f337019c980 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f336800cad0 tx=0x7f336800cf90 comp rx=0 tx=0).stop 2024-04-25T01:40:24.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.270+0000 7f33751a2640 1 -- 172.21.15.28:0/2812129818 shutdown_connections 2024-04-25T01:40:24.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.270+0000 7f33751a2640 1 --2- 172.21.15.28:0/2812129818 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f335c0a2d80 0x7f335c0a5240 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.270+0000 7f33751a2640 1 --2- 172.21.15.28:0/2812129818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3370107730 0x7f337019c980 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.270+0000 7f33751a2640 1 --2- 172.21.15.28:0/2812129818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f337006a780 0x7f3370198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.270+0000 7f33751a2640 1 --2- 172.21.15.28:0/2812129818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3370069dc0 0x7f33701980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:24.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.270+0000 7f33751a2640 1 -- 172.21.15.28:0/2812129818 >> 172.21.15.28:0/2812129818 conn(0x7f33700776a0 msgr2=0x7f33700fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:24.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.270+0000 7f33751a2640 1 -- 172.21.15.28:0/2812129818 shutdown_connections 2024-04-25T01:40:24.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:24.270+0000 7f33751a2640 1 -- 172.21.15.28:0/2812129818 wait complete. 2024-04-25T01:40:25.267 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:40:25.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:40:25.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:40:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.402+0000 7f97dd4f5640 1 -- 172.21.15.28:0/621848733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d8100fd0 msgr2=0x7f97d810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.402+0000 7f97dd4f5640 1 --2- 172.21.15.28:0/621848733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d8100fd0 0x7f97d810d4e0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009cd0 tx=0x7f97d002a8f0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.402+0000 7f97dd4f5640 1 -- 172.21.15.28:0/621848733 shutdown_connections 2024-04-25T01:40:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.402+0000 7f97dd4f5640 1 --2- 172.21.15.28:0/621848733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d8100fd0 0x7f97d810d4e0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.402+0000 7f97dd4f5640 1 --2- 172.21.15.28:0/621848733 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97d8100610 0x7f97d8100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.402+0000 7f97dd4f5640 1 --2- 172.21.15.28:0/621848733 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97d81066b0 0x7f97d8106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.402+0000 7f97dd4f5640 1 -- 172.21.15.28:0/621848733 >> 172.21.15.28:0/621848733 conn(0x7f97d8077740 msgr2=0x7f97d8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:25.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.402+0000 7f97dd4f5640 1 -- 172.21.15.28:0/621848733 shutdown_connections 2024-04-25T01:40:25.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.402+0000 7f97dd4f5640 1 -- 172.21.15.28:0/621848733 wait complete. 2024-04-25T01:40:25.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.403+0000 7f97dd4f5640 1 Processor -- start 2024-04-25T01:40:25.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.403+0000 7f97dd4f5640 1 -- start start 2024-04-25T01:40:25.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.403+0000 7f97dd4f5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97d8100610 0x7f97d8102400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.403+0000 7f97dd4f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d8100fd0 0x7f97d8102940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.403+0000 7f97dd4f5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97d81066b0 0x7f97d8102e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.403+0000 7f97dd4f5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97d81109f0 con 0x7f97d8100fd0 2024-04-25T01:40:25.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.403+0000 7f97dd4f5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f97d8110870 con 0x7f97d8100610 2024-04-25T01:40:25.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.403+0000 7f97dd4f5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f97d8110b70 con 0x7f97d81066b0 2024-04-25T01:40:25.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.403+0000 7f97d67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d8100fd0 0x7f97d8102940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.403+0000 7f97d67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d8100fd0 0x7f97d8102940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35178/0 (socket says 172.21.15.28:35178) 2024-04-25T01:40:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.403+0000 7f97d67fc640 1 -- 172.21.15.28:0/4221525371 learned_addr learned my addr 172.21.15.28:0/4221525371 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.403+0000 7f97d6ffd640 1 --2- 172.21.15.28:0/4221525371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97d8100610 0x7f97d8102400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.403+0000 7f97d77fe640 1 --2- 172.21.15.28:0/4221525371 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97d81066b0 0x7f97d8102e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.404+0000 7f97d67fc640 1 -- 172.21.15.28:0/4221525371 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97d81066b0 msgr2=0x7f97d8102e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.404+0000 7f97d67fc640 1 --2- 172.21.15.28:0/4221525371 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97d81066b0 0x7f97d8102e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.404+0000 7f97d67fc640 1 -- 172.21.15.28:0/4221525371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97d8100610 msgr2=0x7f97d8102400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.404+0000 7f97d67fc640 1 --2- 172.21.15.28:0/4221525371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97d8100610 0x7f97d8102400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.404+0000 7f97d67fc640 1 -- 172.21.15.28:0/4221525371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d81a63a0 con 0x7f97d8100fd0 2024-04-25T01:40:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.404+0000 7f97d67fc640 1 --2- 172.21.15.28:0/4221525371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d8100fd0 0x7f97d8102940 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f97cc00c920 tx=0x7f97cc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.404+0000 7f97bffff640 1 -- 172.21.15.28:0/4221525371 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97cc007c70 con 0x7f97d8100fd0 2024-04-25T01:40:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.404+0000 7f97bffff640 1 -- 172.21.15.28:0/4221525371 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97cc007e10 con 0x7f97d8100fd0 2024-04-25T01:40:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.404+0000 7f97dd4f5640 1 -- 172.21.15.28:0/4221525371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97d81a6690 con 0x7f97d8100fd0 2024-04-25T01:40:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.404+0000 7f97bffff640 1 -- 172.21.15.28:0/4221525371 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97cc018890 con 0x7f97d8100fd0 2024-04-25T01:40:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.404+0000 7f97dd4f5640 1 -- 172.21.15.28:0/4221525371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97d81a6be0 con 0x7f97d8100fd0 2024-04-25T01:40:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.406+0000 7f97bffff640 1 -- 172.21.15.28:0/4221525371 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f97cc018a70 con 0x7f97d8100fd0 2024-04-25T01:40:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.406+0000 7f97e0203740 1 Processor -- start 2024-04-25T01:40:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97e0203740 1 -- start start 2024-04-25T01:40:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97e0203740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c621a30980 0x55c621a2ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97e0203740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a2f290 0x55c621a2f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97e0203740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a3a010 0x55c621a3c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97e0203740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c6219932c0 con 0x55c621a2f290 2024-04-25T01:40:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97e0203740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c62193b910 con 0x55c621a3a010 2024-04-25T01:40:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97e0203740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c62198f280 con 0x55c621a30980 2024-04-25T01:40:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97d6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a2f290 0x55c621a2f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97d6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a2f290 0x55c621a2f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35184/0 (socket says 172.21.15.28:35184) 2024-04-25T01:40:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97d6ffd640 1 -- 172.21.15.28:0/3037774519 learned_addr learned my addr 172.21.15.28:0/3037774519 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97d77fe640 1 --2- 172.21.15.28:0/3037774519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c621a30980 0x55c621a2ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97bffff640 1 --2- 172.21.15.28:0/4221525371 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97c00a2d00 0x7f97c00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97d67fc640 1 --2- 172.21.15.28:0/3037774519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a3a010 0x55c621a3c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97d6ffd640 1 -- 172.21.15.28:0/3037774519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c621a30980 msgr2=0x55c621a2ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97d6ffd640 1 --2- 172.21.15.28:0/3037774519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c621a30980 0x55c621a2ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97d6ffd640 1 -- 172.21.15.28:0/3037774519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a3a010 msgr2=0x55c621a3c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97d6ffd640 1 --2- 172.21.15.28:0/3037774519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a3a010 0x55c621a3c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97d77fe640 1 --2- 172.21.15.28:0/4221525371 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97c00a2d00 0x7f97c00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97d6ffd640 1 -- 172.21.15.28:0/3037774519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c62194cfb0 con 0x55c621a2f290 2024-04-25T01:40:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.407+0000 7f97bffff640 1 -- 172.21.15.28:0/4221525371 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f97cc0cf930 con 0x7f97d8100fd0 2024-04-25T01:40:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.408+0000 7f97d6ffd640 1 --2- 172.21.15.28:0/3037774519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a2f290 0x55c621a2f710 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f97c800bc30 tx=0x7f97c80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87b78b37274db9a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.408+0000 7f97bd7fa640 1 -- 172.21.15.28:0/3037774519 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c8004020 con 0x55c621a2f290 2024-04-25T01:40:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.408+0000 7f97bd7fa640 1 -- 172.21.15.28:0/3037774519 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97c80041c0 con 0x55c621a2f290 2024-04-25T01:40:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.408+0000 7f97d77fe640 1 --2- 172.21.15.28:0/4221525371 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97c00a2d00 0x7f97c00a51c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f97d0005e30 tx=0x7f97d002a8f0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.408+0000 7f97bd7fa640 1 -- 172.21.15.28:0/3037774519 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c8011840 con 0x55c621a2f290 2024-04-25T01:40:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.408+0000 7f97e0203740 1 -- 172.21.15.28:0/3037774519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a2f290 msgr2=0x55c621a2f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.408+0000 7f97e0203740 1 --2- 172.21.15.28:0/3037774519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a2f290 0x55c621a2f710 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f97c800bc30 tx=0x7f97c80107a0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.408+0000 7f97e0203740 1 -- 172.21.15.28:0/3037774519 shutdown_connections 2024-04-25T01:40:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.408+0000 7f97e0203740 1 --2- 172.21.15.28:0/3037774519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a3a010 0x55c621a3c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.408+0000 7f97e0203740 1 --2- 172.21.15.28:0/3037774519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a2f290 0x55c621a2f710 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.408+0000 7f97e0203740 1 --2- 172.21.15.28:0/3037774519 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c621a30980 0x55c621a2ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.409+0000 7f97e0203740 1 -- 172.21.15.28:0/3037774519 >> 172.21.15.28:0/3037774519 conn(0x55c6219f7950 msgr2=0x55c621a2b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.409+0000 7f97e0203740 1 -- 172.21.15.28:0/3037774519 shutdown_connections 2024-04-25T01:40:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.409+0000 7f97e0203740 1 -- 172.21.15.28:0/3037774519 wait complete. 2024-04-25T01:40:25.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.409+0000 7f97e0203740 1 -- start start 2024-04-25T01:40:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.409+0000 7f97e0203740 1 Processor -- start 2024-04-25T01:40:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.409+0000 7f97e0203740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a2f290 0x55c621a36ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.409+0000 7f97e0203740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a30980 0x55c621a37430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.409+0000 7f97e0203740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c621a3a010 0x55c621a37970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.409+0000 7f97e0203740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c621a3cea0 con 0x55c621a30980 2024-04-25T01:40:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.409+0000 7f97e0203740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c62194ee90 con 0x55c621a2f290 2024-04-25T01:40:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.409+0000 7f97e0203740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c621a3d020 con 0x55c621a3a010 2024-04-25T01:40:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.409+0000 7f97e0203740 10 client.0 _ll_register_callbacks cb 0x55c62198a970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.409+0000 7f97d67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a30980 0x55c621a37430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.409+0000 7f97d67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a30980 0x55c621a37430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35194/0 (socket says 172.21.15.28:35194) 2024-04-25T01:40:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.409+0000 7f97d67fc640 1 -- 172.21.15.28:0/1350346029 learned_addr learned my addr 172.21.15.28:0/1350346029 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.409+0000 7f97d6ffd640 1 --2- 172.21.15.28:0/1350346029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a2f290 0x55c621a36ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.410+0000 7f97d77fe640 1 --2- 172.21.15.28:0/1350346029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c621a3a010 0x55c621a37970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.410+0000 7f97d67fc640 1 -- 172.21.15.28:0/1350346029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c621a3a010 msgr2=0x55c621a37970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.410+0000 7f97d67fc640 1 --2- 172.21.15.28:0/1350346029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c621a3a010 0x55c621a37970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.410+0000 7f97d67fc640 1 -- 172.21.15.28:0/1350346029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a2f290 msgr2=0x55c621a36ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.410+0000 7f97d67fc640 1 --2- 172.21.15.28:0/1350346029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a2f290 0x55c621a36ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.410+0000 7f97d67fc640 1 -- 172.21.15.28:0/1350346029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c62176f7e0 con 0x55c621a30980 2024-04-25T01:40:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.410+0000 7f97d67fc640 1 --2- 172.21.15.28:0/1350346029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a30980 0x55c621a37430 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f97cc0ccd50 tx=0x7f97cc09b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d20b212269456f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.410+0000 7f97baff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.410+0000 7f97baff5640 1 -- 172.21.15.28:0/1350346029 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97cc09bac0 con 0x55c621a30980 2024-04-25T01:40:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.410+0000 7f97baff5640 1 -- 172.21.15.28:0/1350346029 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f97cc098130 con 0x55c621a30980 2024-04-25T01:40:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.410+0000 7f97e0203740 1 -- 172.21.15.28:0/1350346029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97c80041c0 con 0x55c621a30980 2024-04-25T01:40:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.410+0000 7f97baff5640 1 -- 172.21.15.28:0/1350346029 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f97cc0b9050 con 0x55c621a30980 2024-04-25T01:40:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.410+0000 7f97e0203740 10 client.18381 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.410+0000 7f97e0203740 1 -- 172.21.15.28:0/1350346029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c6217cff70 con 0x55c621a30980 2024-04-25T01:40:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.410+0000 7f97baff5640 1 -- 172.21.15.28:0/1350346029 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97c80041c0 con 0x55c621a30980 2024-04-25T01:40:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.411+0000 7f97baff5640 1 -- 172.21.15.28:0/1350346029 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f97cc0ba240 con 0x55c621a30980 2024-04-25T01:40:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.411+0000 7f97e0203740 10 client.18381 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.411+0000 7f97e0203740 10 client.18381 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.411+0000 7f97e0203740 20 client.18381 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.411+0000 7f97e0203740 1 --2- 172.21.15.28:0/1350346029 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c621a4a070 0x55c621a4c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.411+0000 7f97e0203740 4 client.18381 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.411+0000 7f97e0203740 1 -- 172.21.15.28:0/1350346029 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6219675a0 con 0x55c621a4a070 2024-04-25T01:40:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.411+0000 7f97d6ffd640 1 --2- 172.21.15.28:0/1350346029 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c621a4a070 0x55c621a4c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.411+0000 7f97baff5640 10 client.18381 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.411+0000 7f97d6ffd640 1 --2- 172.21.15.28:0/1350346029 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c621a4a070 0x55c621a4c460 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea77dd071cc43baa server_cookie=57367c9e17863fba in_seq=0 out_seq=0 2024-04-25T01:40:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.412+0000 7f97baff5640 1 -- 172.21.15.28:0/1350346029 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c6219675a0 con 0x55c621a4a070 2024-04-25T01:40:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.412+0000 7f97baff5640 10 client.18381 handle_command_reply: tid=0 2024-04-25T01:40:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.412+0000 7f97e0203740 1 client.18381 shutdown 2024-04-25T01:40:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.412+0000 7f97e0203740 1 -- 172.21.15.28:0/1350346029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a30980 msgr2=0x55c621a37430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.412+0000 7f97e0203740 1 --2- 172.21.15.28:0/1350346029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a30980 0x55c621a37430 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f97cc0ccd50 tx=0x7f97cc09b700 comp rx=0 tx=0).stop 2024-04-25T01:40:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.412+0000 7f97e0203740 1 -- 172.21.15.28:0/1350346029 shutdown_connections 2024-04-25T01:40:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.412+0000 7f97e0203740 1 --2- 172.21.15.28:0/1350346029 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c621a4a070 0x55c621a4c460 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.412+0000 7f97e0203740 1 --2- 172.21.15.28:0/1350346029 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c621a3a010 0x55c621a37970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.412+0000 7f97e0203740 1 --2- 172.21.15.28:0/1350346029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a30980 0x55c621a37430 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.412+0000 7f97e0203740 1 --2- 172.21.15.28:0/1350346029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a2f290 0x55c621a36ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.412+0000 7f97baff5640 0 client.18381 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.412+0000 7f97e0203740 1 -- 172.21.15.28:0/1350346029 >> 172.21.15.28:0/1350346029 conn(0x55c6219f7950 msgr2=0x55c621a315f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.412+0000 7f97e0203740 1 -- 172.21.15.28:0/1350346029 shutdown_connections 2024-04-25T01:40:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.412+0000 7f97e0203740 1 -- 172.21.15.28:0/1350346029 wait complete. 2024-04-25T01:40:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.412+0000 7f97e0203740 20 client.18381 trim_cache size 0 max 16384 2024-04-25T01:40:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.430+0000 7f97e0203740 1 Processor -- start 2024-04-25T01:40:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.430+0000 7f97e0203740 1 -- start start 2024-04-25T01:40:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97e0203740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c621a2f290 0x55c621a45da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97e0203740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a30980 0x55c621a2a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97e0203740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a3a010 0x55c621a51d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97e0203740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97d81b0c90 con 0x55c621a30980 2024-04-25T01:40:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97e0203740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f97d81b27d0 con 0x55c621a3a010 2024-04-25T01:40:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97e0203740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f97d81b1360 con 0x55c621a2f290 2024-04-25T01:40:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97d67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a30980 0x55c621a2a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97d67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a30980 0x55c621a2a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35210/0 (socket says 172.21.15.28:35210) 2024-04-25T01:40:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97d67fc640 1 -- 172.21.15.28:0/4213538207 learned_addr learned my addr 172.21.15.28:0/4213538207 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97d67fc640 1 -- 172.21.15.28:0/4213538207 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c621a2f290 msgr2=0x55c621a45da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97d67fc640 1 --2- 172.21.15.28:0/4213538207 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c621a2f290 0x55c621a45da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97d67fc640 1 -- 172.21.15.28:0/4213538207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a3a010 msgr2=0x55c621a51d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97d77fe640 1 --2- 172.21.15.28:0/4213538207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a3a010 0x55c621a51d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97d67fc640 1 --2- 172.21.15.28:0/4213538207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a3a010 0x55c621a51d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97d67fc640 1 -- 172.21.15.28:0/4213538207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c621a35ea0 con 0x55c621a30980 2024-04-25T01:40:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97d77fe640 1 --2- 172.21.15.28:0/4213538207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a3a010 0x55c621a51d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97d67fc640 1 --2- 172.21.15.28:0/4213538207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a30980 0x55c621a2a7d0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f97cc00cfd0 tx=0x7f97cc097d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=980fb8a9800c8504 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97bcff9640 1 -- 172.21.15.28:0/4213538207 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97cc09bac0 con 0x55c621a30980 2024-04-25T01:40:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97bcff9640 1 -- 172.21.15.28:0/4213538207 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97cc09fe70 con 0x55c621a30980 2024-04-25T01:40:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.431+0000 7f97bcff9640 1 -- 172.21.15.28:0/4213538207 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97cc0a42a0 con 0x55c621a30980 2024-04-25T01:40:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.432+0000 7f97e0203740 1 -- 172.21.15.28:0/4213538207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a30980 msgr2=0x55c621a2a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.432+0000 7f97e0203740 1 --2- 172.21.15.28:0/4213538207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a30980 0x55c621a2a7d0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f97cc00cfd0 tx=0x7f97cc097d60 comp rx=0 tx=0).stop 2024-04-25T01:40:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.432+0000 7f97e0203740 1 -- 172.21.15.28:0/4213538207 shutdown_connections 2024-04-25T01:40:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.432+0000 7f97e0203740 1 --2- 172.21.15.28:0/4213538207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a3a010 0x55c621a51d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.432+0000 7f97e0203740 1 --2- 172.21.15.28:0/4213538207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a30980 0x55c621a2a7d0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.432+0000 7f97e0203740 1 --2- 172.21.15.28:0/4213538207 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c621a2f290 0x55c621a45da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.432+0000 7f97e0203740 1 -- 172.21.15.28:0/4213538207 >> 172.21.15.28:0/4213538207 conn(0x55c6219f7950 msgr2=0x55c621a2a0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.432+0000 7f97e0203740 1 -- 172.21.15.28:0/4213538207 shutdown_connections 2024-04-25T01:40:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.432+0000 7f97e0203740 1 -- 172.21.15.28:0/4213538207 wait complete. 2024-04-25T01:40:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.432+0000 7f97e0203740 1 -- start start 2024-04-25T01:40:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.432+0000 7f97e0203740 1 Processor -- start 2024-04-25T01:40:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97e0203740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c621a2f290 0x55c621a42790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97e0203740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a30980 0x55c621a34330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97e0203740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a3a010 0x55c621a34960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97e0203740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97d81b2100 con 0x55c621a3a010 2024-04-25T01:40:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97e0203740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f97d81b1a30 con 0x55c621a30980 2024-04-25T01:40:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97e0203740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f97d81aec90 con 0x55c621a2f290 2024-04-25T01:40:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97e0203740 10 client.0 _ll_register_callbacks cb 0x7f97cc00c870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97d77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a3a010 0x55c621a34960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97d77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a3a010 0x55c621a34960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35226/0 (socket says 172.21.15.28:35226) 2024-04-25T01:40:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97d77fe640 1 -- 172.21.15.28:0/2464709408 learned_addr learned my addr 172.21.15.28:0/2464709408 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97d67fc640 1 --2- 172.21.15.28:0/2464709408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a30980 0x55c621a34330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97d6ffd640 1 --2- 172.21.15.28:0/2464709408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c621a2f290 0x55c621a42790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97d77fe640 1 -- 172.21.15.28:0/2464709408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c621a2f290 msgr2=0x55c621a42790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97d77fe640 1 --2- 172.21.15.28:0/2464709408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c621a2f290 0x55c621a42790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97d77fe640 1 -- 172.21.15.28:0/2464709408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a30980 msgr2=0x55c621a34330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97d77fe640 1 --2- 172.21.15.28:0/2464709408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a30980 0x55c621a34330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97d77fe640 1 -- 172.21.15.28:0/2464709408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c621a4c7d0 con 0x55c621a3a010 2024-04-25T01:40:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97d67fc640 1 --2- 172.21.15.28:0/2464709408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a30980 0x55c621a34330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:40:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97d77fe640 1 --2- 172.21.15.28:0/2464709408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a3a010 0x55c621a34960 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009cd0 tx=0x7f97d00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7bf2601edb7d2d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97baff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.433+0000 7f97baff5640 1 -- 172.21.15.28:0/2464709408 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d0044070 con 0x55c621a3a010 2024-04-25T01:40:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.434+0000 7f97baff5640 1 -- 172.21.15.28:0/2464709408 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f97d0004240 con 0x55c621a3a010 2024-04-25T01:40:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.434+0000 7f97e0203740 1 -- 172.21.15.28:0/2464709408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97cc09fe70 con 0x55c621a3a010 2024-04-25T01:40:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.434+0000 7f97baff5640 1 -- 172.21.15.28:0/2464709408 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f97d0004440 con 0x55c621a3a010 2024-04-25T01:40:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.434+0000 7f97e0203740 10 client.18387 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.434+0000 7f97e0203740 1 -- 172.21.15.28:0/2464709408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c621667170 con 0x55c621a3a010 2024-04-25T01:40:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.434+0000 7f97baff5640 1 -- 172.21.15.28:0/2464709408 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97cc09fe70 con 0x55c621a3a010 2024-04-25T01:40:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.434+0000 7f97baff5640 1 -- 172.21.15.28:0/2464709408 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f97d0047040 con 0x55c621a3a010 2024-04-25T01:40:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.434+0000 7f97e0203740 10 client.18387 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.434+0000 7f97e0203740 10 client.18387 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.434+0000 7f97e0203740 20 client.18387 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.434+0000 7f97e0203740 1 --2- 172.21.15.28:0/2464709408 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c621ab1c80 0x55c621ab40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.434+0000 7f97e0203740 4 client.18387 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:40:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.434+0000 7f97e0203740 1 -- 172.21.15.28:0/2464709408 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c6219a0800 con 0x55c621ab1c80 2024-04-25T01:40:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.434+0000 7f97d6ffd640 1 --2- 172.21.15.28:0/2464709408 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c621ab1c80 0x55c621ab40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.435+0000 7f97baff5640 10 client.18387 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.435+0000 7f97d6ffd640 1 --2- 172.21.15.28:0/2464709408 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c621ab1c80 0x55c621ab40a0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf831d7a73ff068b server_cookie=e13126bcbf0b611d in_seq=0 out_seq=0 2024-04-25T01:40:25.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.436+0000 7f97baff5640 1 -- 172.21.15.28:0/2464709408 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c6219a0800 con 0x55c621ab1c80 2024-04-25T01:40:25.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.436+0000 7f97baff5640 10 client.18387 handle_command_reply: tid=0 2024-04-25T01:40:25.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.436+0000 7f97e0203740 1 client.18387 shutdown 2024-04-25T01:40:25.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.436+0000 7f97e0203740 1 -- 172.21.15.28:0/2464709408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a3a010 msgr2=0x55c621a34960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:25.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.436+0000 7f97e0203740 1 --2- 172.21.15.28:0/2464709408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a3a010 0x55c621a34960 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009cd0 tx=0x7f97d00095c0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.436+0000 7f97e0203740 1 -- 172.21.15.28:0/2464709408 shutdown_connections 2024-04-25T01:40:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.436+0000 7f97e0203740 1 --2- 172.21.15.28:0/2464709408 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c621ab1c80 0x55c621ab40a0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.436+0000 7f97e0203740 1 --2- 172.21.15.28:0/2464709408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c621a3a010 0x55c621a34960 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.436+0000 7f97e0203740 1 --2- 172.21.15.28:0/2464709408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c621a30980 0x55c621a34330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.436+0000 7f97e0203740 1 --2- 172.21.15.28:0/2464709408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c621a2f290 0x55c621a42790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.436+0000 7f97e0203740 1 -- 172.21.15.28:0/2464709408 >> 172.21.15.28:0/2464709408 conn(0x55c6219f7950 msgr2=0x55c621a37740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.436+0000 7f97baff5640 0 client.18387 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.436+0000 7f97e0203740 1 -- 172.21.15.28:0/2464709408 shutdown_connections 2024-04-25T01:40:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.436+0000 7f97e0203740 1 -- 172.21.15.28:0/2464709408 wait complete. 2024-04-25T01:40:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.436+0000 7f97e0203740 20 client.18387 trim_cache size 0 max 16384 2024-04-25T01:40:25.427 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:40:25.427 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:40:25.428 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "a1cba845-4744-4cf2-9ffe-fe8753b39654", 2024-04-25T01:40:25.428 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:40:25.428 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:40:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.437+0000 7f97dd4f5640 1 -- 172.21.15.28:0/4221525371 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97c00a2d00 msgr2=0x7f97c00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.437+0000 7f97dd4f5640 1 --2- 172.21.15.28:0/4221525371 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97c00a2d00 0x7f97c00a51c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f97d0005e30 tx=0x7f97d002a8f0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.437+0000 7f97dd4f5640 1 -- 172.21.15.28:0/4221525371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d8100fd0 msgr2=0x7f97d8102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.437+0000 7f97dd4f5640 1 --2- 172.21.15.28:0/4221525371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d8100fd0 0x7f97d8102940 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f97cc00c920 tx=0x7f97cc00cde0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.437+0000 7f97dd4f5640 1 -- 172.21.15.28:0/4221525371 shutdown_connections 2024-04-25T01:40:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.437+0000 7f97dd4f5640 1 --2- 172.21.15.28:0/4221525371 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f97c00a2d00 0x7f97c00a51c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.437+0000 7f97dd4f5640 1 --2- 172.21.15.28:0/4221525371 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f97d81066b0 0x7f97d8102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.437+0000 7f97dd4f5640 1 --2- 172.21.15.28:0/4221525371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97d8100fd0 0x7f97d8102940 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.437+0000 7f97dd4f5640 1 --2- 172.21.15.28:0/4221525371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97d8100610 0x7f97d8102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.437+0000 7f97dd4f5640 1 -- 172.21.15.28:0/4221525371 >> 172.21.15.28:0/4221525371 conn(0x7f97d8077740 msgr2=0x7f97d8104540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.437+0000 7f97dd4f5640 1 -- 172.21.15.28:0/4221525371 shutdown_connections 2024-04-25T01:40:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.437+0000 7f97dd4f5640 1 -- 172.21.15.28:0/4221525371 wait complete. 2024-04-25T01:40:25.433 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:40:25.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:40:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.552+0000 7eff7b222640 1 -- 172.21.15.28:0/1664053076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff74100fe0 msgr2=0x7eff7410d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.552+0000 7eff7b222640 1 --2- 172.21.15.28:0/1664053076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff74100fe0 0x7eff7410d4f0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009cd0 tx=0x7eff6c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.552+0000 7eff7b222640 1 -- 172.21.15.28:0/1664053076 shutdown_connections 2024-04-25T01:40:25.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.552+0000 7eff7b222640 1 --2- 172.21.15.28:0/1664053076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff74100fe0 0x7eff7410d4f0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.552+0000 7eff7b222640 1 --2- 172.21.15.28:0/1664053076 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff74100620 0x7eff74100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.552+0000 7eff7b222640 1 --2- 172.21.15.28:0/1664053076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff74106630 0x7eff74106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.552+0000 7eff7b222640 1 -- 172.21.15.28:0/1664053076 >> 172.21.15.28:0/1664053076 conn(0x7eff740776a0 msgr2=0x7eff74077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.552+0000 7eff7b222640 1 -- 172.21.15.28:0/1664053076 shutdown_connections 2024-04-25T01:40:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.552+0000 7eff7b222640 1 -- 172.21.15.28:0/1664053076 wait complete. 2024-04-25T01:40:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.553+0000 7eff7b222640 1 Processor -- start 2024-04-25T01:40:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.553+0000 7eff7b222640 1 -- start start 2024-04-25T01:40:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.553+0000 7eff7b222640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff74100620 0x7eff74198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.553+0000 7eff7b222640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff74100fe0 0x7eff741985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.553+0000 7eff7b222640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff74106630 0x7eff7419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.553+0000 7eff7b222640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff74110a40 con 0x7eff74100620 2024-04-25T01:40:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.553+0000 7eff7b222640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7eff741108c0 con 0x7eff74100fe0 2024-04-25T01:40:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.553+0000 7eff7b222640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7eff74110bc0 con 0x7eff74106630 2024-04-25T01:40:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.553+0000 7eff78fbe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff74100620 0x7eff74198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.553+0000 7eff78fbe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff74100620 0x7eff74198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35240/0 (socket says 172.21.15.28:35240) 2024-04-25T01:40:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.553+0000 7eff78fbe640 1 -- 172.21.15.28:0/2902064006 learned_addr learned my addr 172.21.15.28:0/2902064006 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.553+0000 7eff73fff640 1 --2- 172.21.15.28:0/2902064006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff74100fe0 0x7eff741985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.554+0000 7eff797bf640 1 --2- 172.21.15.28:0/2902064006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff74106630 0x7eff7419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.554+0000 7eff78fbe640 1 -- 172.21.15.28:0/2902064006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff74106630 msgr2=0x7eff7419c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.554+0000 7eff78fbe640 1 --2- 172.21.15.28:0/2902064006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff74106630 0x7eff7419c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.554+0000 7eff78fbe640 1 -- 172.21.15.28:0/2902064006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff74100fe0 msgr2=0x7eff741985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.554+0000 7eff78fbe640 1 --2- 172.21.15.28:0/2902064006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff74100fe0 0x7eff741985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.554+0000 7eff78fbe640 1 -- 172.21.15.28:0/2902064006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff7419d050 con 0x7eff74100620 2024-04-25T01:40:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.554+0000 7eff797bf640 1 --2- 172.21.15.28:0/2902064006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff74106630 0x7eff7419c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.554+0000 7eff78fbe640 1 --2- 172.21.15.28:0/2902064006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff74100620 0x7eff74198060 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00d950 tx=0x7eff5c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.554+0000 7eff71ffb640 1 -- 172.21.15.28:0/2902064006 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c017070 con 0x7eff74100620 2024-04-25T01:40:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.554+0000 7eff7b222640 1 -- 172.21.15.28:0/2902064006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff7419d340 con 0x7eff74100620 2024-04-25T01:40:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.554+0000 7eff71ffb640 1 -- 172.21.15.28:0/2902064006 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff5c009c30 con 0x7eff74100620 2024-04-25T01:40:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.554+0000 7eff71ffb640 1 -- 172.21.15.28:0/2902064006 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c00f6b0 con 0x7eff74100620 2024-04-25T01:40:25.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.554+0000 7eff7b222640 1 -- 172.21.15.28:0/2902064006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff741a4bd0 con 0x7eff74100620 2024-04-25T01:40:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.556+0000 7eff71ffb640 1 -- 172.21.15.28:0/2902064006 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7eff5c0109d0 con 0x7eff74100620 2024-04-25T01:40:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.556+0000 7eff7e005740 1 Processor -- start 2024-04-25T01:40:25.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff7e005740 1 -- start start 2024-04-25T01:40:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff7e005740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55758ee49630 0x55758ee49a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff7e005740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49f50 0x55758ee540e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff71ffb640 1 --2- 172.21.15.28:0/2902064006 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7eff600a2d00 0x7eff600a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff7e005740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55758ee54620 0x55758ee56a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff7e005740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55758edae6b0 con 0x55758ee49f50 2024-04-25T01:40:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff7e005740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55758ed56880 con 0x55758ee49630 2024-04-25T01:40:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff7e005740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55758edaa670 con 0x55758ee54620 2024-04-25T01:40:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff73fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49f50 0x55758ee540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff73fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49f50 0x55758ee540e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35256/0 (socket says 172.21.15.28:35256) 2024-04-25T01:40:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff73fff640 1 -- 172.21.15.28:0/3394509112 learned_addr learned my addr 172.21.15.28:0/3394509112 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:25.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff78fbe640 1 --2- 172.21.15.28:0/2902064006 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7eff600a2d00 0x7eff600a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff797bf640 1 --2- 172.21.15.28:0/3394509112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55758ee49630 0x55758ee49a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff797bf640 1 --2- 172.21.15.28:0/3394509112 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55758ee54620 0x55758ee56a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff71ffb640 1 -- 172.21.15.28:0/2902064006 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7eff5c0cf3d0 con 0x7eff74100620 2024-04-25T01:40:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff73fff640 1 -- 172.21.15.28:0/3394509112 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55758ee54620 msgr2=0x55758ee56a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff73fff640 1 --2- 172.21.15.28:0/3394509112 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55758ee54620 0x55758ee56a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff73fff640 1 -- 172.21.15.28:0/3394509112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55758ee49630 msgr2=0x55758ee49a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff73fff640 1 --2- 172.21.15.28:0/3394509112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55758ee49630 0x55758ee49a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff73fff640 1 -- 172.21.15.28:0/3394509112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55758eb8c7d0 con 0x55758ee49f50 2024-04-25T01:40:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff73fff640 1 --2- 172.21.15.28:0/3394509112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49f50 0x55758ee540e0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7eff6400abf0 tx=0x7eff6400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8ad71a6c37feb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.557+0000 7eff78fbe640 1 --2- 172.21.15.28:0/2902064006 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7eff600a2d00 0x7eff600a51c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7eff5c09b490 tx=0x7eff5c090990 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.558+0000 7eff6affd640 1 -- 172.21.15.28:0/3394509112 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6401d070 con 0x55758ee49f50 2024-04-25T01:40:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.558+0000 7eff6affd640 1 -- 172.21.15.28:0/3394509112 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff64018b20 con 0x55758ee49f50 2024-04-25T01:40:25.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.558+0000 7eff6affd640 1 -- 172.21.15.28:0/3394509112 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff64010a70 con 0x55758ee49f50 2024-04-25T01:40:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.558+0000 7eff7e005740 1 -- 172.21.15.28:0/3394509112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49f50 msgr2=0x55758ee540e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.558+0000 7eff7e005740 1 --2- 172.21.15.28:0/3394509112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49f50 0x55758ee540e0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7eff6400abf0 tx=0x7eff6400f790 comp rx=0 tx=0).stop 2024-04-25T01:40:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.558+0000 7eff7e005740 1 -- 172.21.15.28:0/3394509112 shutdown_connections 2024-04-25T01:40:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.558+0000 7eff7e005740 1 --2- 172.21.15.28:0/3394509112 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55758ee54620 0x55758ee56a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.558+0000 7eff7e005740 1 --2- 172.21.15.28:0/3394509112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49f50 0x55758ee540e0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.558+0000 7eff7e005740 1 --2- 172.21.15.28:0/3394509112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55758ee49630 0x55758ee49a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.558+0000 7eff7e005740 1 -- 172.21.15.28:0/3394509112 >> 172.21.15.28:0/3394509112 conn(0x55758ee45950 msgr2=0x55758ee45d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.558+0000 7eff7e005740 1 -- 172.21.15.28:0/3394509112 shutdown_connections 2024-04-25T01:40:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.558+0000 7eff7e005740 1 -- 172.21.15.28:0/3394509112 wait complete. 2024-04-25T01:40:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.559+0000 7eff7e005740 1 -- start start 2024-04-25T01:40:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.559+0000 7eff7e005740 1 Processor -- start 2024-04-25T01:40:25.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.559+0000 7eff7e005740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55758ee49630 0x55758ee515a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.559+0000 7eff7e005740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49f50 0x55758ee51ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.559+0000 7eff7e005740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55758ee54620 0x55758ee52020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.559+0000 7eff7e005740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55758ee57410 con 0x55758ee49f50 2024-04-25T01:40:25.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.559+0000 7eff7e005740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55758ed69f50 con 0x55758ee54620 2024-04-25T01:40:25.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.559+0000 7eff7e005740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55758ee57590 con 0x55758ee49630 2024-04-25T01:40:25.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.559+0000 7eff73fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49f50 0x55758ee51ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.559+0000 7eff7e005740 10 client.0 _ll_register_callbacks cb 0x55758eda5000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:25.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.559+0000 7eff73fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49f50 0x55758ee51ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35268/0 (socket says 172.21.15.28:35268) 2024-04-25T01:40:25.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.559+0000 7eff73fff640 1 -- 172.21.15.28:0/174598567 learned_addr learned my addr 172.21.15.28:0/174598567 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:25.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.559+0000 7eff797bf640 1 --2- 172.21.15.28:0/174598567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55758ee54620 0x55758ee52020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.559+0000 7eff797bf640 1 --2- 172.21.15.28:0/174598567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55758ee49630 0x55758ee515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff73fff640 1 -- 172.21.15.28:0/174598567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55758ee49630 msgr2=0x55758ee515a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:25.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff73fff640 1 --2- 172.21.15.28:0/174598567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55758ee49630 0x55758ee515a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff73fff640 1 -- 172.21.15.28:0/174598567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55758ee54620 msgr2=0x55758ee52020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:25.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff73fff640 1 --2- 172.21.15.28:0/174598567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55758ee54620 0x55758ee52020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff73fff640 1 -- 172.21.15.28:0/174598567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55758eac23f0 con 0x55758ee49f50 2024-04-25T01:40:25.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff73fff640 1 --2- 172.21.15.28:0/174598567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49f50 0x55758ee51ae0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7eff640105f0 tx=0x7eff6400af50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4065ddbcbe2820a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:25.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff53fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:25.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff53fff640 1 -- 172.21.15.28:0/174598567 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff64027020 con 0x55758ee49f50 2024-04-25T01:40:25.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff53fff640 1 -- 172.21.15.28:0/174598567 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7eff640109f0 con 0x55758ee49f50 2024-04-25T01:40:25.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff7e005740 1 -- 172.21.15.28:0/174598567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff64018b20 con 0x55758ee49f50 2024-04-25T01:40:25.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff53fff640 1 -- 172.21.15.28:0/174598567 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7eff64032050 con 0x55758ee49f50 2024-04-25T01:40:25.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff7e005740 10 client.18399 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:25.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff7e005740 1 -- 172.21.15.28:0/174598567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55758ea820f0 con 0x55758ee49f50 2024-04-25T01:40:25.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff53fff640 1 -- 172.21.15.28:0/174598567 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff64018b20 con 0x55758ee49f50 2024-04-25T01:40:25.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff53fff640 1 -- 172.21.15.28:0/174598567 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7eff64023a50 con 0x55758ee49f50 2024-04-25T01:40:25.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff7e005740 10 client.18399 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:25.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff7e005740 10 client.18399 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:25.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff7e005740 20 client.18399 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:25.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff7e005740 1 --2- 172.21.15.28:0/174598567 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55758ee648b0 0x55758ee66ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff7e005740 4 client.18399 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:25.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.560+0000 7eff7e005740 1 -- 172.21.15.28:0/174598567 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55758edbbbf0 con 0x55758ee648b0 2024-04-25T01:40:25.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.561+0000 7eff797bf640 1 --2- 172.21.15.28:0/174598567 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55758ee648b0 0x55758ee66ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.561+0000 7eff797bf640 1 --2- 172.21.15.28:0/174598567 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55758ee648b0 0x55758ee66ca0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6893c4fdd8ad57e server_cookie=78a33e1667fad12 in_seq=0 out_seq=0 2024-04-25T01:40:25.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.561+0000 7eff53fff640 10 client.18399 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:25.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.562+0000 7eff53fff640 1 -- 172.21.15.28:0/174598567 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55758edbbbf0 con 0x55758ee648b0 2024-04-25T01:40:25.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.562+0000 7eff53fff640 10 client.18399 handle_command_reply: tid=0 2024-04-25T01:40:25.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.562+0000 7eff7e005740 1 client.18399 shutdown 2024-04-25T01:40:25.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.562+0000 7eff7e005740 1 -- 172.21.15.28:0/174598567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49f50 msgr2=0x55758ee51ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:25.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.562+0000 7eff7e005740 1 --2- 172.21.15.28:0/174598567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49f50 0x55758ee51ae0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7eff640105f0 tx=0x7eff6400af50 comp rx=0 tx=0).stop 2024-04-25T01:40:25.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.562+0000 7eff7e005740 1 -- 172.21.15.28:0/174598567 shutdown_connections 2024-04-25T01:40:25.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.562+0000 7eff7e005740 1 --2- 172.21.15.28:0/174598567 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55758ee648b0 0x55758ee66ca0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.562+0000 7eff7e005740 1 --2- 172.21.15.28:0/174598567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55758ee54620 0x55758ee52020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.562+0000 7eff7e005740 1 --2- 172.21.15.28:0/174598567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49f50 0x55758ee51ae0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.562+0000 7eff7e005740 1 --2- 172.21.15.28:0/174598567 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55758ee49630 0x55758ee515a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.562+0000 7eff53fff640 0 client.18399 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:25.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.562+0000 7eff7e005740 1 -- 172.21.15.28:0/174598567 >> 172.21.15.28:0/174598567 conn(0x55758ee45950 msgr2=0x55758ee4b630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:25.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.562+0000 7eff7e005740 1 -- 172.21.15.28:0/174598567 shutdown_connections 2024-04-25T01:40:25.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.562+0000 7eff7e005740 1 -- 172.21.15.28:0/174598567 wait complete. 2024-04-25T01:40:25.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.562+0000 7eff7e005740 20 client.18399 trim_cache size 0 max 16384 2024-04-25T01:40:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.580+0000 7eff7e005740 1 Processor -- start 2024-04-25T01:40:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.580+0000 7eff7e005740 1 -- start start 2024-04-25T01:40:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.580+0000 7eff7e005740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49630 0x55758ee60300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.580+0000 7eff7e005740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55758ee49f50 0x55758ee51c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.580+0000 7eff7e005740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55758ee54620 0x55758ee4bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.580+0000 7eff7e005740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff741aeaf0 con 0x55758ee49630 2024-04-25T01:40:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.580+0000 7eff7e005740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7eff741b0630 con 0x55758ee54620 2024-04-25T01:40:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.580+0000 7eff7e005740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7eff741af1c0 con 0x55758ee49f50 2024-04-25T01:40:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.580+0000 7eff797bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49630 0x55758ee60300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.580+0000 7eff797bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49630 0x55758ee60300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35276/0 (socket says 172.21.15.28:35276) 2024-04-25T01:40:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.580+0000 7eff797bf640 1 -- 172.21.15.28:0/3543255492 learned_addr learned my addr 172.21.15.28:0/3543255492 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.580+0000 7eff797bf640 1 -- 172.21.15.28:0/3543255492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55758ee49f50 msgr2=0x55758ee51c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.580+0000 7eff797bf640 1 --2- 172.21.15.28:0/3543255492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55758ee49f50 0x55758ee51c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.580+0000 7eff797bf640 1 -- 172.21.15.28:0/3543255492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55758ee54620 msgr2=0x55758ee4bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.580+0000 7eff797bf640 1 --2- 172.21.15.28:0/3543255492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55758ee54620 0x55758ee4bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.580+0000 7eff797bf640 1 -- 172.21.15.28:0/3543255492 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55758ee412a0 con 0x55758ee49630 2024-04-25T01:40:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.580+0000 7eff797bf640 1 --2- 172.21.15.28:0/3543255492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49630 0x55758ee60300 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009ca0 tx=0x7eff6c041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44087d6582a07c39 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.581+0000 7eff6a7fc640 1 -- 172.21.15.28:0/3543255492 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c054070 con 0x55758ee49630 2024-04-25T01:40:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.581+0000 7eff6a7fc640 1 -- 172.21.15.28:0/3543255492 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff6c036040 con 0x55758ee49630 2024-04-25T01:40:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.581+0000 7eff6a7fc640 1 -- 172.21.15.28:0/3543255492 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c04f3c0 con 0x55758ee49630 2024-04-25T01:40:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.581+0000 7eff7e005740 1 -- 172.21.15.28:0/3543255492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49630 msgr2=0x55758ee60300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.581+0000 7eff7e005740 1 --2- 172.21.15.28:0/3543255492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49630 0x55758ee60300 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009ca0 tx=0x7eff6c041040 comp rx=0 tx=0).stop 2024-04-25T01:40:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.581+0000 7eff7e005740 1 -- 172.21.15.28:0/3543255492 shutdown_connections 2024-04-25T01:40:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.581+0000 7eff7e005740 1 --2- 172.21.15.28:0/3543255492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55758ee54620 0x55758ee4bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.581+0000 7eff7e005740 1 --2- 172.21.15.28:0/3543255492 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55758ee49f50 0x55758ee51c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.581+0000 7eff7e005740 1 --2- 172.21.15.28:0/3543255492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49630 0x55758ee60300 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.581+0000 7eff7e005740 1 -- 172.21.15.28:0/3543255492 >> 172.21.15.28:0/3543255492 conn(0x55758ee45950 msgr2=0x55758ee51560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.581+0000 7eff7e005740 1 -- 172.21.15.28:0/3543255492 shutdown_connections 2024-04-25T01:40:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.581+0000 7eff7e005740 1 -- 172.21.15.28:0/3543255492 wait complete. 2024-04-25T01:40:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.581+0000 7eff7e005740 1 -- start start 2024-04-25T01:40:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.582+0000 7eff7e005740 1 Processor -- start 2024-04-25T01:40:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.582+0000 7eff7e005740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49630 0x55758ee5cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.582+0000 7eff7e005740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55758ee49f50 0x55758ee4df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.582+0000 7eff7e005740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55758ee54620 0x55758ee4e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.582+0000 7eff7e005740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff741aff60 con 0x55758ee49630 2024-04-25T01:40:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.582+0000 7eff7e005740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7eff741af890 con 0x55758ee54620 2024-04-25T01:40:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.582+0000 7eff7e005740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7eff741acaf0 con 0x55758ee49f50 2024-04-25T01:40:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.582+0000 7eff7e005740 10 client.0 _ll_register_callbacks cb 0x7eff6401b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.582+0000 7eff797bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49630 0x55758ee5cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.582+0000 7eff797bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49630 0x55758ee5cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35292/0 (socket says 172.21.15.28:35292) 2024-04-25T01:40:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.582+0000 7eff797bf640 1 -- 172.21.15.28:0/4137831408 learned_addr learned my addr 172.21.15.28:0/4137831408 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.582+0000 7eff797bf640 1 -- 172.21.15.28:0/4137831408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55758ee49f50 msgr2=0x55758ee4df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.582+0000 7eff797bf640 1 --2- 172.21.15.28:0/4137831408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55758ee49f50 0x55758ee4df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.582+0000 7eff797bf640 1 -- 172.21.15.28:0/4137831408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55758ee54620 msgr2=0x55758ee4e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.582+0000 7eff797bf640 1 --2- 172.21.15.28:0/4137831408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55758ee54620 0x55758ee4e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.582+0000 7eff797bf640 1 -- 172.21.15.28:0/4137831408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55758ee4eb90 con 0x55758ee49630 2024-04-25T01:40:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.583+0000 7eff797bf640 1 --2- 172.21.15.28:0/4137831408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49630 0x55758ee5cb00 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7eff6c0048c0 tx=0x7eff6c004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32be9bf834e9b509 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.583+0000 7eff53fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.583+0000 7eff53fff640 1 -- 172.21.15.28:0/4137831408 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c054040 con 0x55758ee49630 2024-04-25T01:40:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.583+0000 7eff53fff640 1 -- 172.21.15.28:0/4137831408 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7eff6c04f2d0 con 0x55758ee49630 2024-04-25T01:40:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.583+0000 7eff7e005740 1 -- 172.21.15.28:0/4137831408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff6c036040 con 0x55758ee49630 2024-04-25T01:40:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.583+0000 7eff53fff640 1 -- 172.21.15.28:0/4137831408 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7eff6c04f4d0 con 0x55758ee49630 2024-04-25T01:40:25.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.583+0000 7eff7e005740 10 client.18405 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.583+0000 7eff7e005740 1 -- 172.21.15.28:0/4137831408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55758eecab00 con 0x55758ee49630 2024-04-25T01:40:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.583+0000 7eff53fff640 1 -- 172.21.15.28:0/4137831408 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff6c036040 con 0x55758ee49630 2024-04-25T01:40:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.583+0000 7eff53fff640 1 -- 172.21.15.28:0/4137831408 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7eff6c030070 con 0x55758ee49630 2024-04-25T01:40:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.583+0000 7eff7e005740 10 client.18405 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.583+0000 7eff7e005740 10 client.18405 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.583+0000 7eff7e005740 20 client.18405 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.583+0000 7eff7e005740 1 --2- 172.21.15.28:0/4137831408 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55758eecb100 0x55758eecd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.583+0000 7eff7e005740 4 client.18405 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:40:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.583+0000 7eff7e005740 1 -- 172.21.15.28:0/4137831408 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55758edb5f40 con 0x55758eecb100 2024-04-25T01:40:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.584+0000 7eff73fff640 1 --2- 172.21.15.28:0/4137831408 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55758eecb100 0x55758eecd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.584+0000 7eff73fff640 1 --2- 172.21.15.28:0/4137831408 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55758eecb100 0x55758eecd520 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=850e872c6560eabe server_cookie=db424b394dab9c84 in_seq=0 out_seq=0 2024-04-25T01:40:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.584+0000 7eff53fff640 10 client.18405 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.584+0000 7eff53fff640 1 -- 172.21.15.28:0/4137831408 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55758edb5f40 con 0x55758eecb100 2024-04-25T01:40:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.584+0000 7eff53fff640 10 client.18405 handle_command_reply: tid=0 2024-04-25T01:40:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.584+0000 7eff7e005740 1 client.18405 shutdown 2024-04-25T01:40:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.584+0000 7eff7e005740 1 -- 172.21.15.28:0/4137831408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49630 msgr2=0x55758ee5cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.584+0000 7eff7e005740 1 --2- 172.21.15.28:0/4137831408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49630 0x55758ee5cb00 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7eff6c0048c0 tx=0x7eff6c004d30 comp rx=0 tx=0).stop 2024-04-25T01:40:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.584+0000 7eff7e005740 1 -- 172.21.15.28:0/4137831408 shutdown_connections 2024-04-25T01:40:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.584+0000 7eff7e005740 1 --2- 172.21.15.28:0/4137831408 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55758eecb100 0x55758eecd520 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.584+0000 7eff7e005740 1 --2- 172.21.15.28:0/4137831408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55758ee54620 0x55758ee4e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.584+0000 7eff7e005740 1 --2- 172.21.15.28:0/4137831408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55758ee49f50 0x55758ee4df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.584+0000 7eff7e005740 1 --2- 172.21.15.28:0/4137831408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55758ee49630 0x55758ee5cb00 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.584+0000 7eff53fff640 0 client.18405 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.584+0000 7eff7e005740 1 -- 172.21.15.28:0/4137831408 >> 172.21.15.28:0/4137831408 conn(0x55758ee45950 msgr2=0x55758ee44af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.584+0000 7eff7e005740 1 -- 172.21.15.28:0/4137831408 shutdown_connections 2024-04-25T01:40:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.584+0000 7eff7e005740 1 -- 172.21.15.28:0/4137831408 wait complete. 2024-04-25T01:40:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.584+0000 7eff7e005740 20 client.18405 trim_cache size 0 max 16384 2024-04-25T01:40:25.577 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:40:25.577 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:40:25.577 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-25T01:40:25.577 INFO:teuthology.orchestra.run.smithi028.stdout: "a1cba845-4744-4cf2-9ffe-fe8753b39654": { 2024-04-25T01:40:25.577 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-25T01:40:25.577 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "a1cba845-4744-4cf2-9ffe-fe8753b39654", 2024-04-25T01:40:25.577 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-25T01:40:25.577 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:40:25.578 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:40:25.578 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:40:25.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.585+0000 7eff7b222640 1 -- 172.21.15.28:0/2902064006 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7eff600a2d00 msgr2=0x7eff600a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.585+0000 7eff7b222640 1 --2- 172.21.15.28:0/2902064006 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7eff600a2d00 0x7eff600a51c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7eff5c09b490 tx=0x7eff5c090990 comp rx=0 tx=0).stop 2024-04-25T01:40:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.585+0000 7eff7b222640 1 -- 172.21.15.28:0/2902064006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff74100620 msgr2=0x7eff74198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.585+0000 7eff7b222640 1 --2- 172.21.15.28:0/2902064006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff74100620 0x7eff74198060 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00d950 tx=0x7eff5c00de10 comp rx=0 tx=0).stop 2024-04-25T01:40:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.585+0000 7eff7b222640 1 -- 172.21.15.28:0/2902064006 shutdown_connections 2024-04-25T01:40:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.585+0000 7eff7b222640 1 --2- 172.21.15.28:0/2902064006 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7eff600a2d00 0x7eff600a51c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.585+0000 7eff7b222640 1 --2- 172.21.15.28:0/2902064006 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7eff74106630 0x7eff7419c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.585+0000 7eff7b222640 1 --2- 172.21.15.28:0/2902064006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7eff74100fe0 0x7eff741985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.585+0000 7eff7b222640 1 --2- 172.21.15.28:0/2902064006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff74100620 0x7eff74198060 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.585+0000 7eff7b222640 1 -- 172.21.15.28:0/2902064006 >> 172.21.15.28:0/2902064006 conn(0x7eff740776a0 msgr2=0x7eff7406a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.585+0000 7eff7b222640 1 -- 172.21.15.28:0/2902064006 shutdown_connections 2024-04-25T01:40:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:25.585+0000 7eff7b222640 1 -- 172.21.15.28:0/2902064006 wait complete. 2024-04-25T01:40:25.581 INFO:tasks.cephfs.filesystem:scrub status for tag:a1cba845-4744-4cf2-9ffe-fe8753b39654 - {'path': '/', 'tag': 'a1cba845-4744-4cf2-9ffe-fe8753b39654', 'options': 'recursive,force'} 2024-04-25T01:40:26.215 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:25 smithi130 ceph-mon[32702]: pgmap v227: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:40:26.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:25 smithi032 ceph-mon[32461]: pgmap v227: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:40:26.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:25 smithi028 ceph-mon[29390]: pgmap v227: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:40:27.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:26 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:40:27.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:26 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:40:27.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:26 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:40:27.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:26 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:40:27.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:26 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:40:27.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:26 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:40:27.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:26 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:40:27.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:26 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:40:27.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:26 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:40:27.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:26 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:40:27.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:26 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:40:27.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:26 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:40:28.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:27 smithi130 ceph-mon[32702]: pgmap v228: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-25T01:40:28.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:27 smithi032 ceph-mon[32461]: pgmap v228: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-25T01:40:28.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:27 smithi028 ceph-mon[29390]: pgmap v228: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-25T01:40:29.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:28 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:40:29.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:28 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:40:29.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:28 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:40:30.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:29 smithi130 ceph-mon[32702]: pgmap v229: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-04-25T01:40:30.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:29 smithi032 ceph-mon[32461]: pgmap v229: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-04-25T01:40:30.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:29 smithi028 ceph-mon[29390]: pgmap v229: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-04-25T01:40:32.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:31 smithi130 ceph-mon[32702]: pgmap v230: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-25T01:40:32.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:31 smithi032 ceph-mon[32461]: pgmap v230: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-25T01:40:32.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:31 smithi028 ceph-mon[29390]: pgmap v230: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-25T01:40:34.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:33 smithi130 ceph-mon[32702]: pgmap v231: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 12 op/s 2024-04-25T01:40:34.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:33 smithi032 ceph-mon[32461]: pgmap v231: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 12 op/s 2024-04-25T01:40:34.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:33 smithi028 ceph-mon[29390]: pgmap v231: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 12 op/s 2024-04-25T01:40:36.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:35 smithi130 ceph-mon[32702]: pgmap v232: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-25T01:40:36.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:35 smithi032 ceph-mon[32461]: pgmap v232: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-25T01:40:36.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:35 smithi028 ceph-mon[29390]: pgmap v232: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-25T01:40:38.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:37 smithi130 ceph-mon[32702]: pgmap v233: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-25T01:40:38.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:37 smithi032 ceph-mon[32461]: pgmap v233: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-25T01:40:38.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:37 smithi028 ceph-mon[29390]: pgmap v233: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-25T01:40:40.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:39 smithi130 ceph-mon[32702]: pgmap v234: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-25T01:40:40.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:39 smithi032 ceph-mon[32461]: pgmap v234: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-25T01:40:40.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:39 smithi028 ceph-mon[29390]: pgmap v234: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-25T01:40:42.216 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:41 smithi130 ceph-mon[32702]: pgmap v235: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:40:42.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:41 smithi032 ceph-mon[32461]: pgmap v235: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:40:42.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:41 smithi028 ceph-mon[29390]: pgmap v235: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:40:44.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:43 smithi032 ceph-mon[32461]: pgmap v236: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:40:44.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:43 smithi028 ceph-mon[29390]: pgmap v236: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:40:44.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:43 smithi130 ceph-mon[32702]: pgmap v236: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:40:46.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:45 smithi032 ceph-mon[32461]: pgmap v237: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:40:46.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:45 smithi028 ceph-mon[29390]: pgmap v237: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:40:46.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:45 smithi130 ceph-mon[32702]: pgmap v237: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:40:48.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:47 smithi032 ceph-mon[32461]: pgmap v238: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:40:48.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:47 smithi028 ceph-mon[29390]: pgmap v238: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:40:48.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:47 smithi130 ceph-mon[32702]: pgmap v238: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:40:50.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:49 smithi032 ceph-mon[32461]: pgmap v239: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:40:50.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:49 smithi028 ceph-mon[29390]: pgmap v239: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:40:50.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:49 smithi130 ceph-mon[32702]: pgmap v239: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:40:52.241 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:51 smithi032 ceph-mon[32461]: pgmap v240: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:40:52.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:51 smithi028 ceph-mon[29390]: pgmap v240: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:40:52.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:51 smithi130 ceph-mon[32702]: pgmap v240: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:40:54.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:53 smithi032 ceph-mon[32461]: pgmap v241: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:40:54.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:53 smithi028 ceph-mon[29390]: pgmap v241: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:40:54.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:53 smithi130 ceph-mon[32702]: pgmap v241: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:40:55.582 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:40:55.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:40:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.715+0000 7fcc0c2d4640 1 -- 172.21.15.28:0/2158493850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc04069dc0 msgr2=0x7fcc0406a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.715+0000 7fcc0c2d4640 1 --2- 172.21.15.28:0/2158493850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc04069dc0 0x7fcc0406a240 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4009930 tx=0x7fcbf402a420 comp rx=0 tx=0).stop 2024-04-25T01:40:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.715+0000 7fcc0c2d4640 1 -- 172.21.15.28:0/2158493850 shutdown_connections 2024-04-25T01:40:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.715+0000 7fcc0c2d4640 1 --2- 172.21.15.28:0/2158493850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc0406a780 0x7fcc0410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.715+0000 7fcc0c2d4640 1 --2- 172.21.15.28:0/2158493850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc04069dc0 0x7fcc0406a240 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.715+0000 7fcc0c2d4640 1 --2- 172.21.15.28:0/2158493850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc04107730 0x7fcc04107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.715+0000 7fcc0c2d4640 1 -- 172.21.15.28:0/2158493850 >> 172.21.15.28:0/2158493850 conn(0x7fcc040776a0 msgr2=0x7fcc04077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.715+0000 7fcc0c2d4640 1 -- 172.21.15.28:0/2158493850 shutdown_connections 2024-04-25T01:40:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.715+0000 7fcc0c2d4640 1 -- 172.21.15.28:0/2158493850 wait complete. 2024-04-25T01:40:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.716+0000 7fcc0c2d4640 1 Processor -- start 2024-04-25T01:40:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.716+0000 7fcc0c2d4640 1 -- start start 2024-04-25T01:40:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.716+0000 7fcc0c2d4640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc04069dc0 0x7fcc04197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.716+0000 7fcc0c2d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0406a780 0x7fcc04198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.716+0000 7fcc0c2d4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc04107730 0x7fcc0419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.716+0000 7fcc0c2d4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc04111b40 con 0x7fcc0406a780 2024-04-25T01:40:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.716+0000 7fcc0c2d4640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041119c0 con 0x7fcc04107730 2024-04-25T01:40:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.716+0000 7fcc0c2d4640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fcc04111cc0 con 0x7fcc04069dc0 2024-04-25T01:40:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.716+0000 7fcc0986f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0406a780 0x7fcc04198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.716+0000 7fcc0986f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0406a780 0x7fcc04198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40788/0 (socket says 172.21.15.28:40788) 2024-04-25T01:40:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.716+0000 7fcc0986f640 1 -- 172.21.15.28:0/1494671105 learned_addr learned my addr 172.21.15.28:0/1494671105 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.716+0000 7fcc0a871640 1 --2- 172.21.15.28:0/1494671105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc04107730 0x7fcc0419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.717+0000 7fcc0a070640 1 --2- 172.21.15.28:0/1494671105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc04069dc0 0x7fcc04197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.717+0000 7fcc0986f640 1 -- 172.21.15.28:0/1494671105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc04069dc0 msgr2=0x7fcc04197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.717+0000 7fcc0986f640 1 --2- 172.21.15.28:0/1494671105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc04069dc0 0x7fcc04197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.717+0000 7fcc0986f640 1 -- 172.21.15.28:0/1494671105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc04107730 msgr2=0x7fcc0419c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.717+0000 7fcc0986f640 1 --2- 172.21.15.28:0/1494671105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc04107730 0x7fcc0419c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.717+0000 7fcc0986f640 1 -- 172.21.15.28:0/1494671105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0419cfd0 con 0x7fcc0406a780 2024-04-25T01:40:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.717+0000 7fcc0986f640 1 --2- 172.21.15.28:0/1494671105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0406a780 0x7fcc04198520 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4009230 tx=0x7fcbf4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:55.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.717+0000 7fcbfb7fe640 1 -- 172.21.15.28:0/1494671105 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf403f030 con 0x7fcc0406a780 2024-04-25T01:40:55.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.717+0000 7fcc0c2d4640 1 -- 172.21.15.28:0/1494671105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc0419d260 con 0x7fcc0406a780 2024-04-25T01:40:55.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.717+0000 7fcbfb7fe640 1 -- 172.21.15.28:0/1494671105 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbf4003ad0 con 0x7fcc0406a780 2024-04-25T01:40:55.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.717+0000 7fcbfb7fe640 1 -- 172.21.15.28:0/1494671105 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4035070 con 0x7fcc0406a780 2024-04-25T01:40:55.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.717+0000 7fcc0c2d4640 1 -- 172.21.15.28:0/1494671105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc041a4b50 con 0x7fcc0406a780 2024-04-25T01:40:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.719+0000 7fcbfb7fe640 1 -- 172.21.15.28:0/1494671105 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fcbf4042030 con 0x7fcc0406a780 2024-04-25T01:40:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.720+0000 7fcc0efe3740 1 Processor -- start 2024-04-25T01:40:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.720+0000 7fcc0efe3740 1 -- start start 2024-04-25T01:40:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.721+0000 7fcc0efe3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0630 0x558c298f0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.721+0000 7fcc0efe3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298f0f50 0x558c298fb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.721+0000 7fcc0efe3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 0x558c298fda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.721+0000 7fcc0efe3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558c298556b0 con 0x558c298fb620 2024-04-25T01:40:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.721+0000 7fcc0efe3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558c297fd880 con 0x558c298f0f50 2024-04-25T01:40:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.721+0000 7fcc0efe3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x558c29851670 con 0x558c298f0630 2024-04-25T01:40:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.721+0000 7fcbfb7fe640 1 --2- 172.21.15.28:0/1494671105 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcbf00a2d00 0x7fcbf00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.721+0000 7fcc0986f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 0x558c298fda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.721+0000 7fcc0986f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 0x558c298fda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40796/0 (socket says 172.21.15.28:40796) 2024-04-25T01:40:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.721+0000 7fcc0986f640 1 -- 172.21.15.28:0/2653735032 learned_addr learned my addr 172.21.15.28:0/2653735032 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.721+0000 7fcc0a871640 1 --2- 172.21.15.28:0/1494671105 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcbf00a2d00 0x7fcbf00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.721+0000 7fcc0a070640 1 --2- 172.21.15.28:0/2653735032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298f0f50 0x558c298fb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.721+0000 7fcbfb7fe640 1 -- 172.21.15.28:0/1494671105 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fcbf40efdd0 con 0x7fcc0406a780 2024-04-25T01:40:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.721+0000 7fcc0a871640 1 --2- 172.21.15.28:0/2653735032 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0630 0x558c298f0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.721+0000 7fcc0986f640 1 -- 172.21.15.28:0/2653735032 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0630 msgr2=0x558c298f0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.721+0000 7fcc0986f640 1 --2- 172.21.15.28:0/2653735032 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0630 0x558c298f0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.721+0000 7fcc0986f640 1 -- 172.21.15.28:0/2653735032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298f0f50 msgr2=0x558c298fb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.721+0000 7fcc0986f640 1 --2- 172.21.15.28:0/2653735032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298f0f50 0x558c298fb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.721+0000 7fcc0986f640 1 -- 172.21.15.28:0/2653735032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c296337d0 con 0x558c298fb620 2024-04-25T01:40:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.722+0000 7fcc0986f640 1 --2- 172.21.15.28:0/2653735032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 0x558c298fda10 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40ed190 tx=0x7fcbf408df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e05d9db242c563bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.722+0000 7fcc0a871640 1 --2- 172.21.15.28:0/1494671105 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcbf00a2d00 0x7fcbf00a51c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc005fd0 tx=0x7fcbfc009be0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.722+0000 7fcbf8bfe640 1 -- 172.21.15.28:0/2653735032 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4049070 con 0x558c298fb620 2024-04-25T01:40:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.722+0000 7fcbf8bfe640 1 -- 172.21.15.28:0/2653735032 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbf40c4b70 con 0x558c298fb620 2024-04-25T01:40:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.722+0000 7fcbf8bfe640 1 -- 172.21.15.28:0/2653735032 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf40b8620 con 0x558c298fb620 2024-04-25T01:40:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.722+0000 7fcc0efe3740 1 -- 172.21.15.28:0/2653735032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 msgr2=0x558c298fda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.723+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/2653735032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 0x558c298fda10 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40ed190 tx=0x7fcbf408df70 comp rx=0 tx=0).stop 2024-04-25T01:40:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.723+0000 7fcc0efe3740 1 -- 172.21.15.28:0/2653735032 shutdown_connections 2024-04-25T01:40:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.723+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/2653735032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 0x558c298fda10 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.723+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/2653735032 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298f0f50 0x558c298fb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.723+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/2653735032 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0630 0x558c298f0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.723+0000 7fcc0efe3740 1 -- 172.21.15.28:0/2653735032 >> 172.21.15.28:0/2653735032 conn(0x558c298ec950 msgr2=0x558c298ecd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.723+0000 7fcc0efe3740 1 -- 172.21.15.28:0/2653735032 shutdown_connections 2024-04-25T01:40:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.723+0000 7fcc0efe3740 1 -- 172.21.15.28:0/2653735032 wait complete. 2024-04-25T01:40:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.723+0000 7fcc0efe3740 1 -- start start 2024-04-25T01:40:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.723+0000 7fcc0efe3740 1 Processor -- start 2024-04-25T01:40:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.723+0000 7fcc0efe3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298f0630 0x558c298f8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.723+0000 7fcc0efe3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0f50 0x558c298f8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.723+0000 7fcc0efe3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 0x558c298f8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.723+0000 7fcc0efe3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558c298fe1d0 con 0x558c298fb620 2024-04-25T01:40:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.723+0000 7fcc0efe3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558c29810f50 con 0x558c298f0630 2024-04-25T01:40:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.723+0000 7fcc0efe3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x558c298fe350 con 0x558c298f0f50 2024-04-25T01:40:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.724+0000 7fcc0efe3740 10 client.0 _ll_register_callbacks cb 0x558c2984c000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.724+0000 7fcc0a871640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 0x558c298f8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.724+0000 7fcc0a871640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 0x558c298f8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40812/0 (socket says 172.21.15.28:40812) 2024-04-25T01:40:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.724+0000 7fcc0a070640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298f0630 0x558c298f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.724+0000 7fcc0a871640 1 -- 172.21.15.28:0/3452704550 learned_addr learned my addr 172.21.15.28:0/3452704550 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.724+0000 7fcc0986f640 1 --2- 172.21.15.28:0/3452704550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0f50 0x558c298f8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.724+0000 7fcc0a871640 1 -- 172.21.15.28:0/3452704550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0f50 msgr2=0x558c298f8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.724+0000 7fcc0a871640 1 --2- 172.21.15.28:0/3452704550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0f50 0x558c298f8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.724+0000 7fcc0a871640 1 -- 172.21.15.28:0/3452704550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298f0630 msgr2=0x558c298f8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.724+0000 7fcc0a871640 1 --2- 172.21.15.28:0/3452704550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298f0630 0x558c298f8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.724+0000 7fcc0a871640 1 -- 172.21.15.28:0/3452704550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c295693f0 con 0x558c298fb620 2024-04-25T01:40:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.724+0000 7fcc0a871640 1 --2- 172.21.15.28:0/3452704550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 0x558c298f8fc0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc011490 tx=0x7fcbfc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5b38a0e658e1b43 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.724+0000 7fcbedffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.724+0000 7fcbedffb640 1 -- 172.21.15.28:0/3452704550 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc004340 con 0x558c298fb620 2024-04-25T01:40:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.724+0000 7fcbedffb640 1 -- 172.21.15.28:0/3452704550 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fcbfc0044e0 con 0x558c298fb620 2024-04-25T01:40:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.724+0000 7fcc0efe3740 1 -- 172.21.15.28:0/3452704550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbf40c4b70 con 0x558c298fb620 2024-04-25T01:40:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.724+0000 7fcbedffb640 1 -- 172.21.15.28:0/3452704550 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcbfc0046e0 con 0x558c298fb620 2024-04-25T01:40:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.724+0000 7fcc0efe3740 10 client.18417 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.724+0000 7fcc0efe3740 1 -- 172.21.15.28:0/3452704550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558c295290f0 con 0x558c298fb620 2024-04-25T01:40:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.725+0000 7fcbedffb640 1 -- 172.21.15.28:0/3452704550 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbf40c4b70 con 0x558c298fb620 2024-04-25T01:40:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.725+0000 7fcbedffb640 1 -- 172.21.15.28:0/3452704550 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcbfc028520 con 0x558c298fb620 2024-04-25T01:40:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.725+0000 7fcc0efe3740 10 client.18417 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.725+0000 7fcc0efe3740 10 client.18417 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.725+0000 7fcc0efe3740 20 client.18417 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.725+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/3452704550 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558c2990b5e0 0x558c2990da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.725+0000 7fcc0efe3740 4 client.18417 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.725+0000 7fcc0efe3740 1 -- 172.21.15.28:0/3452704550 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c29862bf0 con 0x558c2990b5e0 2024-04-25T01:40:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.725+0000 7fcc0a070640 1 --2- 172.21.15.28:0/3452704550 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558c2990b5e0 0x558c2990da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.726+0000 7fcbedffb640 10 client.18417 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:55.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.726+0000 7fcc0a070640 1 --2- 172.21.15.28:0/3452704550 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558c2990b5e0 0x558c2990da30 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f481b2fc0c187dc8 server_cookie=45199f4495e964c7 in_seq=0 out_seq=0 2024-04-25T01:40:55.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.726+0000 7fcbedffb640 1 -- 172.21.15.28:0/3452704550 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558c29862bf0 con 0x558c2990b5e0 2024-04-25T01:40:55.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.726+0000 7fcbedffb640 10 client.18417 handle_command_reply: tid=0 2024-04-25T01:40:55.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.726+0000 7fcc0efe3740 1 client.18417 shutdown 2024-04-25T01:40:55.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.726+0000 7fcc0efe3740 1 -- 172.21.15.28:0/3452704550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 msgr2=0x558c298f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:55.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.726+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/3452704550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 0x558c298f8fc0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc011490 tx=0x7fcbfc004ab0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.726+0000 7fcc0efe3740 1 -- 172.21.15.28:0/3452704550 shutdown_connections 2024-04-25T01:40:55.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.726+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/3452704550 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558c2990b5e0 0x558c2990da30 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.727+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/3452704550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 0x558c298f8fc0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.727+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/3452704550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0f50 0x558c298f8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.727+0000 7fcbedffb640 0 client.18417 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:55.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.727+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/3452704550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298f0630 0x558c298f8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.727+0000 7fcc0efe3740 1 -- 172.21.15.28:0/3452704550 >> 172.21.15.28:0/3452704550 conn(0x558c298ec950 msgr2=0x558c298f2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:55.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.727+0000 7fcc0efe3740 1 -- 172.21.15.28:0/3452704550 shutdown_connections 2024-04-25T01:40:55.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.727+0000 7fcc0efe3740 1 -- 172.21.15.28:0/3452704550 wait complete. 2024-04-25T01:40:55.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.727+0000 7fcc0efe3740 20 client.18417 trim_cache size 0 max 16384 2024-04-25T01:40:55.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0efe3740 1 Processor -- start 2024-04-25T01:40:55.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0efe3740 1 -- start start 2024-04-25T01:40:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0efe3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0630 0x558c29907000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0efe3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298f0f50 0x558c298ecd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0efe3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298fb620 0x558c298f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0efe3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041aeb70 con 0x558c298f0f50 2024-04-25T01:40:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0efe3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041b06b0 con 0x558c298fb620 2024-04-25T01:40:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0efe3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041af240 con 0x558c298f0630 2024-04-25T01:40:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0986f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298f0f50 0x558c298ecd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0986f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298f0f50 0x558c298ecd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40818/0 (socket says 172.21.15.28:40818) 2024-04-25T01:40:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0a070640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0630 0x558c29907000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0986f640 1 -- 172.21.15.28:0/677802057 learned_addr learned my addr 172.21.15.28:0/677802057 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0a871640 1 --2- 172.21.15.28:0/677802057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298fb620 0x558c298f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0986f640 1 -- 172.21.15.28:0/677802057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0630 msgr2=0x558c29907000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0986f640 1 --2- 172.21.15.28:0/677802057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0630 0x558c29907000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0986f640 1 -- 172.21.15.28:0/677802057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298fb620 msgr2=0x558c298f2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0986f640 1 --2- 172.21.15.28:0/677802057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298fb620 0x558c298f2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0986f640 1 -- 172.21.15.28:0/677802057 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c298ec780 con 0x558c298f0f50 2024-04-25T01:40:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0a871640 1 --2- 172.21.15.28:0/677802057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298fb620 0x558c298f2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:40:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.745+0000 7fcc0986f640 1 --2- 172.21.15.28:0/677802057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298f0f50 0x558c298ecd30 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4009a50 tx=0x7fcbf402a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=908b0ae0eec4a7d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.746+0000 7fcbeffff640 1 -- 172.21.15.28:0/677802057 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf408d020 con 0x558c298f0f50 2024-04-25T01:40:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.746+0000 7fcbeffff640 1 -- 172.21.15.28:0/677802057 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbf40ba2b0 con 0x558c298f0f50 2024-04-25T01:40:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.746+0000 7fcbeffff640 1 -- 172.21.15.28:0/677802057 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4049040 con 0x558c298f0f50 2024-04-25T01:40:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.746+0000 7fcc0efe3740 1 -- 172.21.15.28:0/677802057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298f0f50 msgr2=0x558c298ecd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.746+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/677802057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298f0f50 0x558c298ecd30 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4009a50 tx=0x7fcbf402a420 comp rx=0 tx=0).stop 2024-04-25T01:40:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.746+0000 7fcc0efe3740 1 -- 172.21.15.28:0/677802057 shutdown_connections 2024-04-25T01:40:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.746+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/677802057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298fb620 0x558c298f2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.746+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/677802057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298f0f50 0x558c298ecd30 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.746+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/677802057 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0630 0x558c29907000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.746+0000 7fcc0efe3740 1 -- 172.21.15.28:0/677802057 >> 172.21.15.28:0/677802057 conn(0x558c298ec950 msgr2=0x558c298eb260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.746+0000 7fcc0efe3740 1 -- 172.21.15.28:0/677802057 shutdown_connections 2024-04-25T01:40:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.746+0000 7fcc0efe3740 1 -- 172.21.15.28:0/677802057 wait complete. 2024-04-25T01:40:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0efe3740 1 -- start start 2024-04-25T01:40:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0efe3740 1 Processor -- start 2024-04-25T01:40:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0efe3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298f0630 0x558c299037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0efe3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0f50 0x558c298f4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0efe3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 0x558c298f5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0efe3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041affe0 con 0x558c298fb620 2024-04-25T01:40:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0efe3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041af910 con 0x558c298f0630 2024-04-25T01:40:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0efe3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041acb70 con 0x558c298f0f50 2024-04-25T01:40:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0efe3740 10 client.0 _ll_register_callbacks cb 0x7fcbfc011050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0a871640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 0x558c298f5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0a070640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298f0630 0x558c299037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0a871640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 0x558c298f5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40824/0 (socket says 172.21.15.28:40824) 2024-04-25T01:40:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0a871640 1 -- 172.21.15.28:0/4001821561 learned_addr learned my addr 172.21.15.28:0/4001821561 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0986f640 1 --2- 172.21.15.28:0/4001821561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0f50 0x558c298f4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0a070640 1 -- 172.21.15.28:0/4001821561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0f50 msgr2=0x558c298f4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0a070640 1 --2- 172.21.15.28:0/4001821561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0f50 0x558c298f4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0a070640 1 -- 172.21.15.28:0/4001821561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 msgr2=0x558c298f5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0a070640 1 --2- 172.21.15.28:0/4001821561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 0x558c298f5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0a070640 1 -- 172.21.15.28:0/4001821561 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c298f5ad0 con 0x558c298f0630 2024-04-25T01:40:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.747+0000 7fcc0a871640 1 --2- 172.21.15.28:0/4001821561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 0x558c298f5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:40:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.748+0000 7fcc0a070640 1 --2- 172.21.15.28:0/4001821561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298f0630 0x558c299037b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x558c2990e240 tx=0x7fcc000039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df534ca52e504de8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.748+0000 7fcbedffb640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-25T01:40:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.748+0000 7fcbedffb640 1 -- 172.21.15.28:0/4001821561 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc00005220 con 0x558c298f0630 2024-04-25T01:40:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.748+0000 7fcbedffb640 1 -- 172.21.15.28:0/4001821561 <== mon.2 v2:172.21.15.32:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fcc000053c0 con 0x558c298f0630 2024-04-25T01:40:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.748+0000 7fcc0efe3740 1 -- 172.21.15.28:0/4001821561 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbf40ba2b0 con 0x558c298f0630 2024-04-25T01:40:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.748+0000 7fcbedffb640 1 -- 172.21.15.28:0/4001821561 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcc0002f050 con 0x558c298f0630 2024-04-25T01:40:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.748+0000 7fcc0efe3740 10 client.27332 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.748+0000 7fcc0efe3740 1 -- 172.21.15.28:0/4001821561 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558c29971a90 con 0x558c298f0630 2024-04-25T01:40:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.748+0000 7fcbedffb640 1 -- 172.21.15.28:0/4001821561 <== mon.2 v2:172.21.15.32:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbf40ba2b0 con 0x558c298f0630 2024-04-25T01:40:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcbedffb640 1 -- 172.21.15.28:0/4001821561 <== mon.2 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcc00004aa0 con 0x558c298f0630 2024-04-25T01:40:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcc0efe3740 10 client.27332 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcc0efe3740 10 client.27332 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcc0efe3740 20 client.27332 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/4001821561 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558c2990b5e0 0x558c29974250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcc0efe3740 4 client.27332 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:40:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcc0efe3740 1 -- 172.21.15.28:0/4001821561 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558c2985cf40 con 0x558c2990b5e0 2024-04-25T01:40:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcc0986f640 1 --2- 172.21.15.28:0/4001821561 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558c2990b5e0 0x558c29974250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcc0986f640 1 --2- 172.21.15.28:0/4001821561 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558c2990b5e0 0x558c29974250 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2e83f740f9d783c server_cookie=c9574022d90d907c in_seq=0 out_seq=0 2024-04-25T01:40:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcbedffb640 10 client.27332 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcbedffb640 1 -- 172.21.15.28:0/4001821561 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558c2985cf40 con 0x558c2990b5e0 2024-04-25T01:40:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcbedffb640 10 client.27332 handle_command_reply: tid=0 2024-04-25T01:40:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcc0efe3740 1 client.27332 shutdown 2024-04-25T01:40:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcc0efe3740 1 -- 172.21.15.28:0/4001821561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298f0630 msgr2=0x558c299037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/4001821561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298f0630 0x558c299037b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x558c2990e240 tx=0x7fcc000039c0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcc0efe3740 1 -- 172.21.15.28:0/4001821561 shutdown_connections 2024-04-25T01:40:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/4001821561 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558c2990b5e0 0x558c29974250 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/4001821561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c298fb620 0x558c298f5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcbedffb640 0 client.27332 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/4001821561 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c298f0f50 0x558c298f4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcc0efe3740 1 --2- 172.21.15.28:0/4001821561 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c298f0630 0x558c299037b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.749+0000 7fcc0efe3740 1 -- 172.21.15.28:0/4001821561 >> 172.21.15.28:0/4001821561 conn(0x558c298ec950 msgr2=0x558c298f8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.750+0000 7fcc0efe3740 1 -- 172.21.15.28:0/4001821561 shutdown_connections 2024-04-25T01:40:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.750+0000 7fcc0efe3740 1 -- 172.21.15.28:0/4001821561 wait complete. 2024-04-25T01:40:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.750+0000 7fcc0efe3740 20 client.27332 trim_cache size 0 max 16384 2024-04-25T01:40:55.743 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:40:55.744 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:40:55.744 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:40:55.744 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:40:55.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.750+0000 7fcc0c2d4640 1 -- 172.21.15.28:0/1494671105 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcbf00a2d00 msgr2=0x7fcbf00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:55.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.750+0000 7fcc0c2d4640 1 --2- 172.21.15.28:0/1494671105 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcbf00a2d00 0x7fcbf00a51c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc005fd0 tx=0x7fcbfc009be0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.750+0000 7fcc0c2d4640 1 -- 172.21.15.28:0/1494671105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0406a780 msgr2=0x7fcc04198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.750+0000 7fcc0c2d4640 1 --2- 172.21.15.28:0/1494671105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0406a780 0x7fcc04198520 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4009230 tx=0x7fcbf4004270 comp rx=0 tx=0).stop 2024-04-25T01:40:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.750+0000 7fcc0c2d4640 1 -- 172.21.15.28:0/1494671105 shutdown_connections 2024-04-25T01:40:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.750+0000 7fcc0c2d4640 1 --2- 172.21.15.28:0/1494671105 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcbf00a2d00 0x7fcbf00a51c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.750+0000 7fcc0c2d4640 1 --2- 172.21.15.28:0/1494671105 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc04107730 0x7fcc0419c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.750+0000 7fcc0c2d4640 1 --2- 172.21.15.28:0/1494671105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0406a780 0x7fcc04198520 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.750+0000 7fcc0c2d4640 1 --2- 172.21.15.28:0/1494671105 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcc04069dc0 0x7fcc04197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.750+0000 7fcc0c2d4640 1 -- 172.21.15.28:0/1494671105 >> 172.21.15.28:0/1494671105 conn(0x7fcc040776a0 msgr2=0x7fcc040fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.750+0000 7fcc0c2d4640 1 -- 172.21.15.28:0/1494671105 shutdown_connections 2024-04-25T01:40:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.750+0000 7fcc0c2d4640 1 -- 172.21.15.28:0/1494671105 wait complete. 2024-04-25T01:40:55.747 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:40:55.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:40:55.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.865+0000 7f8aead09640 1 -- 172.21.15.28:0/4157424157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ae4100690 msgr2=0x7f8ae4100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:55.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.865+0000 7f8aead09640 1 --2- 172.21.15.28:0/4157424157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ae4100690 0x7f8ae4100b10 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0009930 tx=0x7f8ad002a420 comp rx=0 tx=0).stop 2024-04-25T01:40:55.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.865+0000 7f8aead09640 1 -- 172.21.15.28:0/4157424157 shutdown_connections 2024-04-25T01:40:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.866+0000 7f8aead09640 1 --2- 172.21.15.28:0/4157424157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ae4101050 0x7f8ae410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.866+0000 7f8aead09640 1 --2- 172.21.15.28:0/4157424157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ae4100690 0x7f8ae4100b10 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.866+0000 7f8aead09640 1 --2- 172.21.15.28:0/4157424157 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ae4106650 0x7f8ae4106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.866+0000 7f8aead09640 1 -- 172.21.15.28:0/4157424157 >> 172.21.15.28:0/4157424157 conn(0x7f8ae40777a0 msgr2=0x7f8ae4077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.866+0000 7f8aead09640 1 -- 172.21.15.28:0/4157424157 shutdown_connections 2024-04-25T01:40:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.866+0000 7f8aead09640 1 -- 172.21.15.28:0/4157424157 wait complete. 2024-04-25T01:40:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.866+0000 7f8aead09640 1 Processor -- start 2024-04-25T01:40:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.866+0000 7f8aead09640 1 -- start start 2024-04-25T01:40:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.866+0000 7f8aead09640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ae4100690 0x7f8ae4198fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.866+0000 7f8aead09640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ae4101050 0x7f8ae41994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.866+0000 7f8aead09640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ae4106650 0x7f8ae41a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.866+0000 7f8aead09640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae4110a60 con 0x7f8ae4100690 2024-04-25T01:40:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.866+0000 7f8aead09640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae41108e0 con 0x7f8ae4101050 2024-04-25T01:40:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.866+0000 7f8aead09640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae4110be0 con 0x7f8ae4106650 2024-04-25T01:40:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.866+0000 7f8ae8aa5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ae4100690 0x7f8ae4198fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.867+0000 7f8ae8aa5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ae4100690 0x7f8ae4198fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40830/0 (socket says 172.21.15.28:40830) 2024-04-25T01:40:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.867+0000 7f8ae8aa5640 1 -- 172.21.15.28:0/2304088673 learned_addr learned my addr 172.21.15.28:0/2304088673 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.867+0000 7f8ae3fff640 1 --2- 172.21.15.28:0/2304088673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ae4101050 0x7f8ae41994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.867+0000 7f8ae92a6640 1 --2- 172.21.15.28:0/2304088673 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ae4106650 0x7f8ae41a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.867+0000 7f8ae8aa5640 1 -- 172.21.15.28:0/2304088673 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ae4106650 msgr2=0x7f8ae41a4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.867+0000 7f8ae8aa5640 1 --2- 172.21.15.28:0/2304088673 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ae4106650 0x7f8ae41a4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.867+0000 7f8ae8aa5640 1 -- 172.21.15.28:0/2304088673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ae4101050 msgr2=0x7f8ae41994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.867+0000 7f8ae8aa5640 1 --2- 172.21.15.28:0/2304088673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ae4101050 0x7f8ae41994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.867+0000 7f8ae8aa5640 1 -- 172.21.15.28:0/2304088673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ae41a4560 con 0x7f8ae4100690 2024-04-25T01:40:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.867+0000 7f8ae3fff640 1 --2- 172.21.15.28:0/2304088673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ae4101050 0x7f8ae41994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.867+0000 7f8ae92a6640 1 --2- 172.21.15.28:0/2304088673 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ae4106650 0x7f8ae41a4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:55.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.867+0000 7f8ae8aa5640 1 --2- 172.21.15.28:0/2304088673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ae4100690 0x7f8ae4198fa0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f8ad800b4a0 tx=0x7f8ad800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.867+0000 7f8ae1ffb640 1 -- 172.21.15.28:0/2304088673 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8014070 con 0x7f8ae4100690 2024-04-25T01:40:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.867+0000 7f8aead09640 1 -- 172.21.15.28:0/2304088673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ae41a4730 con 0x7f8ae4100690 2024-04-25T01:40:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.867+0000 7f8ae1ffb640 1 -- 172.21.15.28:0/2304088673 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ad8010af0 con 0x7f8ae4100690 2024-04-25T01:40:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.867+0000 7f8ae1ffb640 1 -- 172.21.15.28:0/2304088673 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8017d10 con 0x7f8ae4100690 2024-04-25T01:40:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.867+0000 7f8aead09640 1 -- 172.21.15.28:0/2304088673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ae41a4bf0 con 0x7f8ae4100690 2024-04-25T01:40:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.868+0000 7f8aead09640 1 -- 172.21.15.28:0/2304088673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ae4101d80 con 0x7f8ae4100690 2024-04-25T01:40:55.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.869+0000 7f8ae1ffb640 1 -- 172.21.15.28:0/2304088673 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8ad8017070 con 0x7f8ae4100690 2024-04-25T01:40:55.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.870+0000 7f8ae1ffb640 1 --2- 172.21.15.28:0/2304088673 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8ad40a2d00 0x7f8ad40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:55.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.870+0000 7f8ae1ffb640 1 -- 172.21.15.28:0/2304088673 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f8ad80ceaa0 con 0x7f8ae4100690 2024-04-25T01:40:55.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.870+0000 7f8ae3fff640 1 --2- 172.21.15.28:0/2304088673 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8ad40a2d00 0x7f8ad40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:55.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.870+0000 7f8ae3fff640 1 --2- 172.21.15.28:0/2304088673 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8ad40a2d00 0x7f8ad40a51c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0000c00 tx=0x7f8ad0005a20 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:55.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:55.874+0000 7f8ae1ffb640 1 -- 172.21.15.28:0/2304088673 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8ad809b440 con 0x7f8ae4100690 2024-04-25T01:40:56.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.097+0000 7f8aead09640 1 -- 172.21.15.28:0/2304088673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ae40689b0 con 0x7f8ae4100690 2024-04-25T01:40:56.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.097+0000 7f8ae1ffb640 1 -- 172.21.15.28:0/2304088673 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7f8ad809ad40 con 0x7f8ae4100690 2024-04-25T01:40:56.082 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:40:56.083 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:40:56.083 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:40:56.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.099+0000 7f8aead09640 1 -- 172.21.15.28:0/2304088673 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8ad40a2d00 msgr2=0x7f8ad40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.099+0000 7f8aead09640 1 --2- 172.21.15.28:0/2304088673 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8ad40a2d00 0x7f8ad40a51c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0000c00 tx=0x7f8ad0005a20 comp rx=0 tx=0).stop 2024-04-25T01:40:56.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.099+0000 7f8aead09640 1 -- 172.21.15.28:0/2304088673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ae4100690 msgr2=0x7f8ae4198fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.099+0000 7f8aead09640 1 --2- 172.21.15.28:0/2304088673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ae4100690 0x7f8ae4198fa0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f8ad800b4a0 tx=0x7f8ad800b960 comp rx=0 tx=0).stop 2024-04-25T01:40:56.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.099+0000 7f8aead09640 1 -- 172.21.15.28:0/2304088673 shutdown_connections 2024-04-25T01:40:56.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.099+0000 7f8aead09640 1 --2- 172.21.15.28:0/2304088673 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8ad40a2d00 0x7f8ad40a51c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.099+0000 7f8aead09640 1 --2- 172.21.15.28:0/2304088673 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8ae4106650 0x7f8ae41a4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.099+0000 7f8aead09640 1 --2- 172.21.15.28:0/2304088673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ae4101050 0x7f8ae41994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.099+0000 7f8aead09640 1 --2- 172.21.15.28:0/2304088673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ae4100690 0x7f8ae4198fa0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.099+0000 7f8aead09640 1 -- 172.21.15.28:0/2304088673 >> 172.21.15.28:0/2304088673 conn(0x7f8ae40777a0 msgr2=0x7f8ae4103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.100+0000 7f8aead09640 1 -- 172.21.15.28:0/2304088673 shutdown_connections 2024-04-25T01:40:56.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.100+0000 7f8aead09640 1 -- 172.21.15.28:0/2304088673 wait complete. 2024-04-25T01:40:56.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:40:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.215+0000 7f7e2c314640 1 -- 172.21.15.28:0/3740594167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e24107730 msgr2=0x7f7e24107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.215+0000 7f7e2c314640 1 --2- 172.21.15.28:0/3740594167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e24107730 0x7f7e24107b10 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009930 tx=0x7f7e2002a420 comp rx=0 tx=0).stop 2024-04-25T01:40:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.215+0000 7f7e2c314640 1 -- 172.21.15.28:0/3740594167 shutdown_connections 2024-04-25T01:40:56.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.215+0000 7f7e2c314640 1 --2- 172.21.15.28:0/3740594167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e2406a780 0x7f7e2410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.215+0000 7f7e2c314640 1 --2- 172.21.15.28:0/3740594167 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e24069dc0 0x7f7e2406a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.215+0000 7f7e2c314640 1 --2- 172.21.15.28:0/3740594167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e24107730 0x7f7e24107b10 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.215+0000 7f7e2c314640 1 -- 172.21.15.28:0/3740594167 >> 172.21.15.28:0/3740594167 conn(0x7f7e240776a0 msgr2=0x7f7e24077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.215+0000 7f7e2c314640 1 -- 172.21.15.28:0/3740594167 shutdown_connections 2024-04-25T01:40:56.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.215+0000 7f7e2c314640 1 -- 172.21.15.28:0/3740594167 wait complete. 2024-04-25T01:40:56.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.215+0000 7f7e2c314640 1 Processor -- start 2024-04-25T01:40:56.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.215+0000 7f7e2c314640 1 -- start start 2024-04-25T01:40:56.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e2c314640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e24069dc0 0x7f7e24100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e2c314640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e2406a780 0x7f7e24100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e2c314640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e24107730 0x7f7e241010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e2c314640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e24111b40 con 0x7f7e24069dc0 2024-04-25T01:40:56.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e2c314640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7e241119c0 con 0x7f7e24107730 2024-04-25T01:40:56.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e2c314640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7e24111cc0 con 0x7f7e2406a780 2024-04-25T01:40:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e2a0b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e24069dc0 0x7f7e24100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e2a0b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e24069dc0 0x7f7e24100670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40838/0 (socket says 172.21.15.28:40838) 2024-04-25T01:40:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e2a0b0640 1 -- 172.21.15.28:0/1432826373 learned_addr learned my addr 172.21.15.28:0/1432826373 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e2a8b1640 1 --2- 172.21.15.28:0/1432826373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e24107730 0x7f7e241010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e298af640 1 --2- 172.21.15.28:0/1432826373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e2406a780 0x7f7e24100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e2a0b0640 1 -- 172.21.15.28:0/1432826373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e2406a780 msgr2=0x7f7e24100bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e2a0b0640 1 --2- 172.21.15.28:0/1432826373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e2406a780 0x7f7e24100bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e2a0b0640 1 -- 172.21.15.28:0/1432826373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e24107730 msgr2=0x7f7e241010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e2a0b0640 1 --2- 172.21.15.28:0/1432826373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e24107730 0x7f7e241010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e2a0b0640 1 -- 172.21.15.28:0/1432826373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e241a6560 con 0x7f7e24069dc0 2024-04-25T01:40:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e298af640 1 --2- 172.21.15.28:0/1432826373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e2406a780 0x7f7e24100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e2a8b1640 1 --2- 172.21.15.28:0/1432826373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e24107730 0x7f7e241010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e2a0b0640 1 --2- 172.21.15.28:0/1432826373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e24069dc0 0x7f7e24100670 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009230 tx=0x7f7e20004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.216+0000 7f7e1b7fe640 1 -- 172.21.15.28:0/1432826373 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2003f030 con 0x7f7e24069dc0 2024-04-25T01:40:56.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.217+0000 7f7e1b7fe640 1 -- 172.21.15.28:0/1432826373 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e20003ad0 con 0x7f7e24069dc0 2024-04-25T01:40:56.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.217+0000 7f7e2c314640 1 -- 172.21.15.28:0/1432826373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e241a67f0 con 0x7f7e24069dc0 2024-04-25T01:40:56.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.217+0000 7f7e1b7fe640 1 -- 172.21.15.28:0/1432826373 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e20035070 con 0x7f7e24069dc0 2024-04-25T01:40:56.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.217+0000 7f7e2c314640 1 -- 172.21.15.28:0/1432826373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e241a6ce0 con 0x7f7e24069dc0 2024-04-25T01:40:56.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.219+0000 7f7e1b7fe640 1 -- 172.21.15.28:0/1432826373 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7e20042030 con 0x7f7e24069dc0 2024-04-25T01:40:56.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.219+0000 7f7e2f027740 1 Processor -- start 2024-04-25T01:40:56.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e2f027740 1 -- start start 2024-04-25T01:40:56.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e2f027740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27bad1580 0x55c27bad1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e2f027740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c27bad1ea0 0x55c27badc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e2f027740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27badc570 0x55c27bade960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e2f027740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c27ba36670 con 0x55c27badc570 2024-04-25T01:40:56.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e2f027740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c27b9de970 con 0x55c27bad1580 2024-04-25T01:40:56.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e2f027740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c27ba32630 con 0x55c27bad1ea0 2024-04-25T01:40:56.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e1b7fe640 1 --2- 172.21.15.28:0/1432826373 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7e100a2d00 0x7f7e100a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e298af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27badc570 0x55c27bade960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e298af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27badc570 0x55c27bade960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40844/0 (socket says 172.21.15.28:40844) 2024-04-25T01:40:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e298af640 1 -- 172.21.15.28:0/2872322600 learned_addr learned my addr 172.21.15.28:0/2872322600 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e2a8b1640 1 --2- 172.21.15.28:0/2872322600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27bad1580 0x55c27bad1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e2a8b1640 1 --2- 172.21.15.28:0/1432826373 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7e100a2d00 0x7f7e100a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e1b7fe640 1 -- 172.21.15.28:0/1432826373 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f7e200efe10 con 0x7f7e24069dc0 2024-04-25T01:40:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e2a0b0640 1 --2- 172.21.15.28:0/2872322600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c27bad1ea0 0x55c27badc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e298af640 1 -- 172.21.15.28:0/2872322600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c27bad1ea0 msgr2=0x55c27badc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e298af640 1 --2- 172.21.15.28:0/2872322600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c27bad1ea0 0x55c27badc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e298af640 1 -- 172.21.15.28:0/2872322600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27bad1580 msgr2=0x55c27bad1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e298af640 1 --2- 172.21.15.28:0/2872322600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27bad1580 0x55c27bad1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.220+0000 7f7e298af640 1 -- 172.21.15.28:0/2872322600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c27ba02400 con 0x55c27badc570 2024-04-25T01:40:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.221+0000 7f7e298af640 1 --2- 172.21.15.28:0/2872322600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27badc570 0x55c27bade960 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009c20 tx=0x7f7e1400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba01691c7d34822 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.221+0000 7f7e2a8b1640 1 --2- 172.21.15.28:0/1432826373 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7e100a2d00 0x7f7e100a51c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c013b30 tx=0x7f7e1c005c60 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.221+0000 7f7e18bfe640 1 -- 172.21.15.28:0/2872322600 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e14016070 con 0x55c27badc570 2024-04-25T01:40:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.221+0000 7f7e18bfe640 1 -- 172.21.15.28:0/2872322600 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e14011950 con 0x55c27badc570 2024-04-25T01:40:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.221+0000 7f7e18bfe640 1 -- 172.21.15.28:0/2872322600 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e14010800 con 0x55c27badc570 2024-04-25T01:40:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.222+0000 7f7e2f027740 1 -- 172.21.15.28:0/2872322600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27badc570 msgr2=0x55c27bade960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.222+0000 7f7e2f027740 1 --2- 172.21.15.28:0/2872322600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27badc570 0x55c27bade960 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009c20 tx=0x7f7e1400e6c0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.222+0000 7f7e2f027740 1 -- 172.21.15.28:0/2872322600 shutdown_connections 2024-04-25T01:40:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.222+0000 7f7e2f027740 1 --2- 172.21.15.28:0/2872322600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27badc570 0x55c27bade960 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.222+0000 7f7e2f027740 1 --2- 172.21.15.28:0/2872322600 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c27bad1ea0 0x55c27badc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.222+0000 7f7e2f027740 1 --2- 172.21.15.28:0/2872322600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27bad1580 0x55c27bad1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.222+0000 7f7e2f027740 1 -- 172.21.15.28:0/2872322600 >> 172.21.15.28:0/2872322600 conn(0x55c27bacd8a0 msgr2=0x55c27bacdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.222+0000 7f7e2f027740 1 -- 172.21.15.28:0/2872322600 shutdown_connections 2024-04-25T01:40:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.222+0000 7f7e2f027740 1 -- 172.21.15.28:0/2872322600 wait complete. 2024-04-25T01:40:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.222+0000 7f7e2f027740 1 -- start start 2024-04-25T01:40:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.222+0000 7f7e2f027740 1 Processor -- start 2024-04-25T01:40:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.222+0000 7f7e2f027740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1580 0x55c27badbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.222+0000 7f7e2f027740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c27bad1ea0 0x55c27bad9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.222+0000 7f7e2f027740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27badc570 0x55c27bad9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.222+0000 7f7e2f027740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c27badf360 con 0x55c27bad1580 2024-04-25T01:40:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.222+0000 7f7e2f027740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c27b9f2050 con 0x55c27badc570 2024-04-25T01:40:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.222+0000 7f7e2f027740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c27badf4e0 con 0x55c27bad1ea0 2024-04-25T01:40:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.222+0000 7f7e2f027740 10 client.0 _ll_register_callbacks cb 0x55c27b749be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.222+0000 7f7e2a0b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1580 0x55c27badbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.223+0000 7f7e2a0b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1580 0x55c27badbe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40850/0 (socket says 172.21.15.28:40850) 2024-04-25T01:40:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.223+0000 7f7e2a0b0640 1 -- 172.21.15.28:0/2335267383 learned_addr learned my addr 172.21.15.28:0/2335267383 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.223+0000 7f7e298af640 1 --2- 172.21.15.28:0/2335267383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c27bad1ea0 0x55c27bad9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.223+0000 7f7e2a8b1640 1 --2- 172.21.15.28:0/2335267383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27badc570 0x55c27bad9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.223+0000 7f7e2a0b0640 1 -- 172.21.15.28:0/2335267383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c27bad1ea0 msgr2=0x55c27bad9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.223+0000 7f7e2a0b0640 1 --2- 172.21.15.28:0/2335267383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c27bad1ea0 0x55c27bad9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.223+0000 7f7e2a0b0640 1 -- 172.21.15.28:0/2335267383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27badc570 msgr2=0x55c27bad9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.223+0000 7f7e2a0b0640 1 --2- 172.21.15.28:0/2335267383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27badc570 0x55c27bad9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.223+0000 7f7e2a0b0640 1 -- 172.21.15.28:0/2335267383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c27b9c3dc0 con 0x55c27bad1580 2024-04-25T01:40:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.223+0000 7f7e2a8b1640 1 --2- 172.21.15.28:0/2335267383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27badc570 0x55c27bad9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.223+0000 7f7e2a0b0640 1 --2- 172.21.15.28:0/2335267383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1580 0x55c27badbe60 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f7e200ed1d0 tx=0x7f7e20005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e68bbf8427794b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.223+0000 7f7e0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.223+0000 7f7e0dffb640 1 -- 172.21.15.28:0/2335267383 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e20049040 con 0x55c27bad1580 2024-04-25T01:40:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.223+0000 7f7e0dffb640 1 -- 172.21.15.28:0/2335267383 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f7e20003c80 con 0x55c27bad1580 2024-04-25T01:40:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.223+0000 7f7e2f027740 1 -- 172.21.15.28:0/2335267383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e14011950 con 0x55c27bad1580 2024-04-25T01:40:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.223+0000 7f7e0dffb640 1 -- 172.21.15.28:0/2335267383 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7e200c4bb0 con 0x55c27bad1580 2024-04-25T01:40:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.223+0000 7f7e2f027740 10 client.18441 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.223+0000 7f7e2f027740 1 -- 172.21.15.28:0/2335267383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c27b74a3f0 con 0x55c27bad1580 2024-04-25T01:40:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.223+0000 7f7e0dffb640 1 -- 172.21.15.28:0/2335267383 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e14011950 con 0x55c27bad1580 2024-04-25T01:40:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.224+0000 7f7e0dffb640 1 -- 172.21.15.28:0/2335267383 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7e200c4720 con 0x55c27bad1580 2024-04-25T01:40:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.224+0000 7f7e2f027740 10 client.18441 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.224+0000 7f7e2f027740 10 client.18441 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:40:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.224+0000 7f7e2f027740 20 client.18441 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.224+0000 7f7e2f027740 1 --2- 172.21.15.28:0/2335267383 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55c27baec550 0x55c27baee9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.224+0000 7f7e2f027740 4 client.18441 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.224+0000 7f7e2f027740 1 -- 172.21.15.28:0/2335267383 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c27ba43bb0 con 0x55c27baec550 2024-04-25T01:40:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.224+0000 7f7e298af640 1 --2- 172.21.15.28:0/2335267383 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55c27baec550 0x55c27baee9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.224+0000 7f7e0dffb640 10 client.18441 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:40:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.224+0000 7f7e298af640 1 --2- 172.21.15.28:0/2335267383 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55c27baec550 0x55c27baee9e0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dde6cc5fbf4fa7db server_cookie=5365edcf442948a0 in_seq=0 out_seq=0 2024-04-25T01:40:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.225+0000 7f7e0dffb640 1 -- 172.21.15.28:0/2335267383 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c27ba43bb0 con 0x55c27baec550 2024-04-25T01:40:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.225+0000 7f7e0dffb640 10 client.18441 handle_command_reply: tid=0 2024-04-25T01:40:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.225+0000 7f7e2f027740 1 client.18441 shutdown 2024-04-25T01:40:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.225+0000 7f7e2f027740 1 -- 172.21.15.28:0/2335267383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1580 msgr2=0x55c27badbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.225+0000 7f7e2f027740 1 --2- 172.21.15.28:0/2335267383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1580 0x55c27badbe60 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f7e200ed1d0 tx=0x7f7e20005e50 comp rx=0 tx=0).stop 2024-04-25T01:40:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.225+0000 7f7e2f027740 1 -- 172.21.15.28:0/2335267383 shutdown_connections 2024-04-25T01:40:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.225+0000 7f7e2f027740 1 --2- 172.21.15.28:0/2335267383 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55c27baec550 0x55c27baee9e0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.225+0000 7f7e2f027740 1 --2- 172.21.15.28:0/2335267383 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27badc570 0x55c27bad9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.225+0000 7f7e0dffb640 0 client.18441 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:40:56.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.225+0000 7f7e2f027740 1 --2- 172.21.15.28:0/2335267383 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c27bad1ea0 0x55c27bad9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.225+0000 7f7e2f027740 1 --2- 172.21.15.28:0/2335267383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1580 0x55c27badbe60 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.225+0000 7f7e2f027740 1 -- 172.21.15.28:0/2335267383 >> 172.21.15.28:0/2335267383 conn(0x55c27bacd8a0 msgr2=0x55c27baccb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.225+0000 7f7e2f027740 1 -- 172.21.15.28:0/2335267383 shutdown_connections 2024-04-25T01:40:56.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.225+0000 7f7e2f027740 1 -- 172.21.15.28:0/2335267383 wait complete. 2024-04-25T01:40:56.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.226+0000 7f7e2f027740 20 client.18441 trim_cache size 0 max 16384 2024-04-25T01:40:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.239+0000 7f7e2f027740 1 Processor -- start 2024-04-25T01:40:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.239+0000 7f7e2f027740 1 -- start start 2024-04-25T01:40:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.239+0000 7f7e2f027740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c27bad1580 0x55c27bad4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.239+0000 7f7e2f027740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1ea0 0x55c27badb990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.239+0000 7f7e2f027740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27badc570 0x55c27baf4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.239+0000 7f7e2f027740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e241b0d30 con 0x55c27bad1ea0 2024-04-25T01:40:56.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.239+0000 7f7e2f027740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7e241b2870 con 0x55c27badc570 2024-04-25T01:40:56.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.239+0000 7f7e2f027740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7e241b1400 con 0x55c27bad1580 2024-04-25T01:40:56.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.239+0000 7f7e298af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1ea0 0x55c27badb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.239+0000 7f7e298af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1ea0 0x55c27badb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40864/0 (socket says 172.21.15.28:40864) 2024-04-25T01:40:56.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.239+0000 7f7e298af640 1 -- 172.21.15.28:0/3286262986 learned_addr learned my addr 172.21.15.28:0/3286262986 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.239+0000 7f7e2a8b1640 1 --2- 172.21.15.28:0/3286262986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27badc570 0x55c27baf4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.239+0000 7f7e298af640 1 -- 172.21.15.28:0/3286262986 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c27bad1580 msgr2=0x55c27bad4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.239+0000 7f7e298af640 1 --2- 172.21.15.28:0/3286262986 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c27bad1580 0x55c27bad4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.239+0000 7f7e298af640 1 -- 172.21.15.28:0/3286262986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27badc570 msgr2=0x55c27baf4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.239+0000 7f7e298af640 1 --2- 172.21.15.28:0/3286262986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27badc570 0x55c27baf4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.239+0000 7f7e298af640 1 -- 172.21.15.28:0/3286262986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c27b8147d0 con 0x55c27bad1ea0 2024-04-25T01:40:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.240+0000 7f7e298af640 1 --2- 172.21.15.28:0/3286262986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1ea0 0x55c27badb990 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f7e140099e0 tx=0x7f7e14005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e32b7ac06a6576a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.240+0000 7f7e0ffff640 1 -- 172.21.15.28:0/3286262986 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e14016040 con 0x55c27bad1ea0 2024-04-25T01:40:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.240+0000 7f7e0ffff640 1 -- 172.21.15.28:0/3286262986 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e1402a110 con 0x55c27bad1ea0 2024-04-25T01:40:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.240+0000 7f7e0ffff640 1 -- 172.21.15.28:0/3286262986 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e14004880 con 0x55c27bad1ea0 2024-04-25T01:40:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.240+0000 7f7e2f027740 1 -- 172.21.15.28:0/3286262986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1ea0 msgr2=0x55c27badb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.240+0000 7f7e2f027740 1 --2- 172.21.15.28:0/3286262986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1ea0 0x55c27badb990 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f7e140099e0 tx=0x7f7e14005fb0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.240+0000 7f7e2f027740 1 -- 172.21.15.28:0/3286262986 shutdown_connections 2024-04-25T01:40:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.240+0000 7f7e2f027740 1 --2- 172.21.15.28:0/3286262986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27badc570 0x55c27baf4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.240+0000 7f7e2f027740 1 --2- 172.21.15.28:0/3286262986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1ea0 0x55c27badb990 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.240+0000 7f7e2f027740 1 --2- 172.21.15.28:0/3286262986 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c27bad1580 0x55c27bad4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.240+0000 7f7e2f027740 1 -- 172.21.15.28:0/3286262986 >> 172.21.15.28:0/3286262986 conn(0x55c27bacd8a0 msgr2=0x55c27bacc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.240+0000 7f7e2f027740 1 -- 172.21.15.28:0/3286262986 shutdown_connections 2024-04-25T01:40:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.240+0000 7f7e2f027740 1 -- 172.21.15.28:0/3286262986 wait complete. 2024-04-25T01:40:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.241+0000 7f7e2f027740 1 -- start start 2024-04-25T01:40:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.241+0000 7f7e2f027740 1 Processor -- start 2024-04-25T01:40:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.241+0000 7f7e2f027740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c27bad1580 0x55c27bae48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.241+0000 7f7e2f027740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1ea0 0x55c27badfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.241+0000 7f7e2f027740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27badc570 0x55c27bae0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.241+0000 7f7e2f027740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e241b21a0 con 0x55c27bad1ea0 2024-04-25T01:40:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.241+0000 7f7e2f027740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7e241b1ad0 con 0x55c27badc570 2024-04-25T01:40:56.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.241+0000 7f7e2f027740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7e241aed30 con 0x55c27bad1580 2024-04-25T01:40:56.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.241+0000 7f7e2f027740 10 client.0 _ll_register_callbacks cb 0x7f7e200b8290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:56.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.241+0000 7f7e298af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1ea0 0x55c27badfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.241+0000 7f7e298af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1ea0 0x55c27badfd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40866/0 (socket says 172.21.15.28:40866) 2024-04-25T01:40:56.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.241+0000 7f7e298af640 1 -- 172.21.15.28:0/538099170 learned_addr learned my addr 172.21.15.28:0/538099170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.241+0000 7f7e2a8b1640 1 --2- 172.21.15.28:0/538099170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27badc570 0x55c27bae0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.241+0000 7f7e298af640 1 -- 172.21.15.28:0/538099170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c27bad1580 msgr2=0x55c27bae48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.241+0000 7f7e298af640 1 --2- 172.21.15.28:0/538099170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c27bad1580 0x55c27bae48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.241+0000 7f7e298af640 1 -- 172.21.15.28:0/538099170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27badc570 msgr2=0x55c27bae0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.241+0000 7f7e298af640 1 --2- 172.21.15.28:0/538099170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27badc570 0x55c27bae0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.241+0000 7f7e298af640 1 -- 172.21.15.28:0/538099170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c27bae0850 con 0x55c27bad1ea0 2024-04-25T01:40:56.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.241+0000 7f7e2a8b1640 1 --2- 172.21.15.28:0/538099170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27badc570 0x55c27bae0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:56.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.242+0000 7f7e298af640 1 --2- 172.21.15.28:0/538099170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1ea0 0x55c27badfd50 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400e700 tx=0x7f7e14009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ace2c675051dfb59 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.242+0000 7f7e0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:56.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.242+0000 7f7e0dffb640 1 -- 172.21.15.28:0/538099170 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1403c060 con 0x55c27bad1ea0 2024-04-25T01:40:56.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.242+0000 7f7e0dffb640 1 -- 172.21.15.28:0/538099170 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f7e1402a6d0 con 0x55c27bad1ea0 2024-04-25T01:40:56.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.242+0000 7f7e2f027740 1 -- 172.21.15.28:0/538099170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e1402a110 con 0x55c27bad1ea0 2024-04-25T01:40:56.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.242+0000 7f7e0dffb640 1 -- 172.21.15.28:0/538099170 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7e14043050 con 0x55c27bad1ea0 2024-04-25T01:40:56.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.242+0000 7f7e2f027740 10 client.18447 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:56.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.242+0000 7f7e2f027740 1 -- 172.21.15.28:0/538099170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c27bb528d0 con 0x55c27bad1ea0 2024-04-25T01:40:56.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.242+0000 7f7e0dffb640 1 -- 172.21.15.28:0/538099170 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e1402a110 con 0x55c27bad1ea0 2024-04-25T01:40:56.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.242+0000 7f7e0dffb640 1 -- 172.21.15.28:0/538099170 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7e1400f040 con 0x55c27bad1ea0 2024-04-25T01:40:56.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.242+0000 7f7e2f027740 10 client.18447 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:56.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.242+0000 7f7e2f027740 10 client.18447 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:40:56.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.242+0000 7f7e2f027740 20 client.18447 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:56.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.242+0000 7f7e2f027740 1 --2- 172.21.15.28:0/538099170 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55c27bb52e50 0x55c27bb55270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.242+0000 7f7e2f027740 4 client.18447 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:56.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.242+0000 7f7e2f027740 1 -- 172.21.15.28:0/538099170 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c27ba3df00 con 0x55c27bb52e50 2024-04-25T01:40:56.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.243+0000 7f7e2a0b0640 1 --2- 172.21.15.28:0/538099170 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55c27bb52e50 0x55c27bb55270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.243+0000 7f7e2a0b0640 1 --2- 172.21.15.28:0/538099170 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55c27bb52e50 0x55c27bb55270 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=70085f812b4c5973 server_cookie=3cf5f749fca51b38 in_seq=0 out_seq=0 2024-04-25T01:40:56.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.243+0000 7f7e0dffb640 10 client.18447 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:40:56.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.243+0000 7f7e0dffb640 1 -- 172.21.15.28:0/538099170 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c27ba3df00 con 0x55c27bb52e50 2024-04-25T01:40:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.243+0000 7f7e0dffb640 10 client.18447 handle_command_reply: tid=0 2024-04-25T01:40:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.243+0000 7f7e2f027740 1 client.18447 shutdown 2024-04-25T01:40:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.243+0000 7f7e2f027740 1 -- 172.21.15.28:0/538099170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1ea0 msgr2=0x55c27badfd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.243+0000 7f7e2f027740 1 --2- 172.21.15.28:0/538099170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1ea0 0x55c27badfd50 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400e700 tx=0x7f7e14009f60 comp rx=0 tx=0).stop 2024-04-25T01:40:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.243+0000 7f7e2f027740 1 -- 172.21.15.28:0/538099170 shutdown_connections 2024-04-25T01:40:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.243+0000 7f7e2f027740 1 --2- 172.21.15.28:0/538099170 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55c27bb52e50 0x55c27bb55270 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.243+0000 7f7e2f027740 1 --2- 172.21.15.28:0/538099170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c27badc570 0x55c27bae0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.243+0000 7f7e0dffb640 0 client.18447 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:40:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.243+0000 7f7e2f027740 1 --2- 172.21.15.28:0/538099170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c27bad1ea0 0x55c27badfd50 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.243+0000 7f7e2f027740 1 --2- 172.21.15.28:0/538099170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c27bad1580 0x55c27bae48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.243+0000 7f7e2f027740 1 -- 172.21.15.28:0/538099170 >> 172.21.15.28:0/538099170 conn(0x55c27bacd8a0 msgr2=0x55c27bae7210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.243+0000 7f7e2f027740 1 -- 172.21.15.28:0/538099170 shutdown_connections 2024-04-25T01:40:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.243+0000 7f7e2f027740 1 -- 172.21.15.28:0/538099170 wait complete. 2024-04-25T01:40:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.243+0000 7f7e2f027740 20 client.18447 trim_cache size 0 max 16384 2024-04-25T01:40:56.239 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.244+0000 7f7e2c314640 1 -- 172.21.15.28:0/1432826373 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7e100a2d00 msgr2=0x7f7e100a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.244+0000 7f7e2c314640 1 --2- 172.21.15.28:0/1432826373 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7e100a2d00 0x7f7e100a51c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c013b30 tx=0x7f7e1c005c60 comp rx=0 tx=0).stop 2024-04-25T01:40:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.244+0000 7f7e2c314640 1 -- 172.21.15.28:0/1432826373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e24069dc0 msgr2=0x7f7e24100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.244+0000 7f7e2c314640 1 --2- 172.21.15.28:0/1432826373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e24069dc0 0x7f7e24100670 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009230 tx=0x7f7e20004270 comp rx=0 tx=0).stop 2024-04-25T01:40:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.244+0000 7f7e2c314640 1 -- 172.21.15.28:0/1432826373 shutdown_connections 2024-04-25T01:40:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.244+0000 7f7e2c314640 1 --2- 172.21.15.28:0/1432826373 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7e100a2d00 0x7f7e100a51c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.244+0000 7f7e2c314640 1 --2- 172.21.15.28:0/1432826373 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e24107730 0x7f7e241010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.244+0000 7f7e2c314640 1 --2- 172.21.15.28:0/1432826373 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e2406a780 0x7f7e24100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.244+0000 7f7e2c314640 1 --2- 172.21.15.28:0/1432826373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e24069dc0 0x7f7e24100670 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.244+0000 7f7e2c314640 1 -- 172.21.15.28:0/1432826373 >> 172.21.15.28:0/1432826373 conn(0x7f7e240776a0 msgr2=0x7f7e240fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.244+0000 7f7e2c314640 1 -- 172.21.15.28:0/1432826373 shutdown_connections 2024-04-25T01:40:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.244+0000 7f7e2c314640 1 -- 172.21.15.28:0/1432826373 wait complete. 2024-04-25T01:40:56.241 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:56 smithi028 ceph-mon[29390]: pgmap v242: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:40:56.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:40:56.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:56 smithi032 ceph-mon[32461]: pgmap v242: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:40:56.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.361+0000 7f23c0fe2640 1 -- 172.21.15.28:0/3049636770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23bc107730 msgr2=0x7f23bc107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.361+0000 7f23c0fe2640 1 --2- 172.21.15.28:0/3049636770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23bc107730 0x7f23bc107b10 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f23ac009cd0 tx=0x7f23ac02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.361+0000 7f23c0fe2640 1 -- 172.21.15.28:0/3049636770 shutdown_connections 2024-04-25T01:40:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.361+0000 7f23c0fe2640 1 --2- 172.21.15.28:0/3049636770 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23bc06a780 0x7f23bc10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.361+0000 7f23c0fe2640 1 --2- 172.21.15.28:0/3049636770 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23bc069dc0 0x7f23bc06a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.361+0000 7f23c0fe2640 1 --2- 172.21.15.28:0/3049636770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23bc107730 0x7f23bc107b10 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.361+0000 7f23c0fe2640 1 -- 172.21.15.28:0/3049636770 >> 172.21.15.28:0/3049636770 conn(0x7f23bc0776a0 msgr2=0x7f23bc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.361+0000 7f23c0fe2640 1 -- 172.21.15.28:0/3049636770 shutdown_connections 2024-04-25T01:40:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.361+0000 7f23c0fe2640 1 -- 172.21.15.28:0/3049636770 wait complete. 2024-04-25T01:40:56.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.362+0000 7f23c0fe2640 1 Processor -- start 2024-04-25T01:40:56.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.362+0000 7f23c0fe2640 1 -- start start 2024-04-25T01:40:56.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.362+0000 7f23c0fe2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23bc069dc0 0x7f23bc197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.362+0000 7f23c0fe2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23bc06a780 0x7f23bc198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.362+0000 7f23c0fe2640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23bc107730 0x7f23bc19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.362+0000 7f23c0fe2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f23bc111b00 con 0x7f23bc069dc0 2024-04-25T01:40:56.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.362+0000 7f23c0fe2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f23bc111980 con 0x7f23bc06a780 2024-04-25T01:40:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.362+0000 7f23c0fe2640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f23bc111c80 con 0x7f23bc107730 2024-04-25T01:40:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.362+0000 7f23ba59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23bc069dc0 0x7f23bc197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.362+0000 7f23ba59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23bc069dc0 0x7f23bc197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40878/0 (socket says 172.21.15.28:40878) 2024-04-25T01:40:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.362+0000 7f23ba59c640 1 -- 172.21.15.28:0/1374472244 learned_addr learned my addr 172.21.15.28:0/1374472244 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.362+0000 7f23b9d9b640 1 --2- 172.21.15.28:0/1374472244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23bc06a780 0x7f23bc198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.362+0000 7f23ba59c640 1 -- 172.21.15.28:0/1374472244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23bc107730 msgr2=0x7f23bc19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.362+0000 7f23ba59c640 1 --2- 172.21.15.28:0/1374472244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23bc107730 0x7f23bc19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.362+0000 7f23ba59c640 1 -- 172.21.15.28:0/1374472244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23bc06a780 msgr2=0x7f23bc198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.362+0000 7f23ba59c640 1 --2- 172.21.15.28:0/1374472244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23bc06a780 0x7f23bc198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.362+0000 7f23ba59c640 1 -- 172.21.15.28:0/1374472244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23bc19cef0 con 0x7f23bc069dc0 2024-04-25T01:40:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.362+0000 7f23b9d9b640 1 --2- 172.21.15.28:0/1374472244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23bc06a780 0x7f23bc198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.363+0000 7f23ba59c640 1 --2- 172.21.15.28:0/1374472244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23bc069dc0 0x7f23bc197f00 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f23ac02ae00 tx=0x7f23ac003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.363+0000 7f23ab7fe640 1 -- 172.21.15.28:0/1374472244 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ac004040 con 0x7f23bc069dc0 2024-04-25T01:40:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.363+0000 7f23c0fe2640 1 -- 172.21.15.28:0/1374472244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23bc19d180 con 0x7f23bc069dc0 2024-04-25T01:40:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.363+0000 7f23ab7fe640 1 -- 172.21.15.28:0/1374472244 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23ac0046c0 con 0x7f23bc069dc0 2024-04-25T01:40:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.363+0000 7f23ab7fe640 1 -- 172.21.15.28:0/1374472244 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ac046480 con 0x7f23bc069dc0 2024-04-25T01:40:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.363+0000 7f23c0fe2640 1 -- 172.21.15.28:0/1374472244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23bc1a4a70 con 0x7f23bc069dc0 2024-04-25T01:40:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.365+0000 7f23ab7fe640 1 -- 172.21.15.28:0/1374472244 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f23ac0305a0 con 0x7f23bc069dc0 2024-04-25T01:40:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.365+0000 7f23c3dc5740 1 Processor -- start 2024-04-25T01:40:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23c3dc5740 1 -- start start 2024-04-25T01:40:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23c3dc5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74b580 0x55cf9e74b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23c3dc5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9e74bea0 0x55cf9e756030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23c3dc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e756570 0x55cf9e758960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23c3dc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cf9e6b0670 con 0x55cf9e756570 2024-04-25T01:40:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23c3dc5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cf9e658970 con 0x55cf9e74b580 2024-04-25T01:40:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23c3dc5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55cf9e6ac630 con 0x55cf9e74bea0 2024-04-25T01:40:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23ab7fe640 1 --2- 172.21.15.28:0/1374472244 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23a40a2d00 0x7f23a40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23b9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e756570 0x55cf9e758960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23b9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e756570 0x55cf9e758960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40892/0 (socket says 172.21.15.28:40892) 2024-04-25T01:40:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23b9d9b640 1 -- 172.21.15.28:0/2629689956 learned_addr learned my addr 172.21.15.28:0/2629689956 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23ba59c640 1 --2- 172.21.15.28:0/2629689956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9e74bea0 0x55cf9e756030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23bad9d640 1 --2- 172.21.15.28:0/1374472244 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23a40a2d00 0x7f23a40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23bad9d640 1 --2- 172.21.15.28:0/2629689956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74b580 0x55cf9e74b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23ab7fe640 1 -- 172.21.15.28:0/1374472244 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f23ac02e030 con 0x7f23bc069dc0 2024-04-25T01:40:56.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23b9d9b640 1 -- 172.21.15.28:0/2629689956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9e74bea0 msgr2=0x55cf9e756030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23b9d9b640 1 --2- 172.21.15.28:0/2629689956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9e74bea0 0x55cf9e756030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23b9d9b640 1 -- 172.21.15.28:0/2629689956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74b580 msgr2=0x55cf9e74b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23b9d9b640 1 --2- 172.21.15.28:0/2629689956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74b580 0x55cf9e74b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.366+0000 7f23b9d9b640 1 -- 172.21.15.28:0/2629689956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf9e67c400 con 0x55cf9e756570 2024-04-25T01:40:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.367+0000 7f23b9d9b640 1 --2- 172.21.15.28:0/2629689956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e756570 0x55cf9e758960 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f23b0008c10 tx=0x7f23b000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c648040caae6ecfd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.367+0000 7f23bad9d640 1 --2- 172.21.15.28:0/1374472244 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23a40a2d00 0x7f23a40a51c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f23b40058c0 tx=0x7f23b4005e70 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.367+0000 7f23a8ff9640 1 -- 172.21.15.28:0/2629689956 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b0014070 con 0x55cf9e756570 2024-04-25T01:40:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.367+0000 7f23a8ff9640 1 -- 172.21.15.28:0/2629689956 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23b00047a0 con 0x55cf9e756570 2024-04-25T01:40:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.367+0000 7f23a8ff9640 1 -- 172.21.15.28:0/2629689956 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b0010690 con 0x55cf9e756570 2024-04-25T01:40:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.368+0000 7f23c3dc5740 1 -- 172.21.15.28:0/2629689956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e756570 msgr2=0x55cf9e758960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.368+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/2629689956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e756570 0x55cf9e758960 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f23b0008c10 tx=0x7f23b000e720 comp rx=0 tx=0).stop 2024-04-25T01:40:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.368+0000 7f23c3dc5740 1 -- 172.21.15.28:0/2629689956 shutdown_connections 2024-04-25T01:40:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.368+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/2629689956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e756570 0x55cf9e758960 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.368+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/2629689956 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9e74bea0 0x55cf9e756030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.368+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/2629689956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74b580 0x55cf9e74b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.368+0000 7f23c3dc5740 1 -- 172.21.15.28:0/2629689956 >> 172.21.15.28:0/2629689956 conn(0x55cf9e7478a0 msgr2=0x55cf9e747c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.368+0000 7f23c3dc5740 1 -- 172.21.15.28:0/2629689956 shutdown_connections 2024-04-25T01:40:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.368+0000 7f23c3dc5740 1 -- 172.21.15.28:0/2629689956 wait complete. 2024-04-25T01:40:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.368+0000 7f23c3dc5740 1 -- start start 2024-04-25T01:40:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.368+0000 7f23c3dc5740 1 Processor -- start 2024-04-25T01:40:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.368+0000 7f23c3dc5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74b580 0x55cf9e755ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.368+0000 7f23c3dc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74bea0 0x55cf9e753190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.368+0000 7f23c3dc5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9e756570 0x55cf9e7536d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.368+0000 7f23c3dc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cf9e759360 con 0x55cf9e74bea0 2024-04-25T01:40:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.368+0000 7f23c3dc5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55cf9e66c050 con 0x55cf9e74b580 2024-04-25T01:40:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.368+0000 7f23c3dc5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55cf9e7594e0 con 0x55cf9e756570 2024-04-25T01:40:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.368+0000 7f23c3dc5740 10 client.0 _ll_register_callbacks cb 0x55cf9e3c3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.368+0000 7f23b9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74bea0 0x55cf9e753190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.369+0000 7f23b9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74bea0 0x55cf9e753190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40896/0 (socket says 172.21.15.28:40896) 2024-04-25T01:40:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.369+0000 7f23b9d9b640 1 -- 172.21.15.28:0/2597610792 learned_addr learned my addr 172.21.15.28:0/2597610792 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.369+0000 7f23ba59c640 1 --2- 172.21.15.28:0/2597610792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74b580 0x55cf9e755ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.369+0000 7f23bad9d640 1 --2- 172.21.15.28:0/2597610792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9e756570 0x55cf9e7536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.369+0000 7f23b9d9b640 1 -- 172.21.15.28:0/2597610792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9e756570 msgr2=0x55cf9e7536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.369+0000 7f23b9d9b640 1 --2- 172.21.15.28:0/2597610792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9e756570 0x55cf9e7536d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.369+0000 7f23b9d9b640 1 -- 172.21.15.28:0/2597610792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74b580 msgr2=0x55cf9e755ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.369+0000 7f23b9d9b640 1 --2- 172.21.15.28:0/2597610792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74b580 0x55cf9e755ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.369+0000 7f23b9d9b640 1 -- 172.21.15.28:0/2597610792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf9e63ddc0 con 0x55cf9e74bea0 2024-04-25T01:40:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.369+0000 7f23b9d9b640 1 --2- 172.21.15.28:0/2597610792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74bea0 0x55cf9e753190 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f23b0008fd0 tx=0x7f23b0005160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=298d73d7652b70b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.369+0000 7f23a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.369+0000 7f23a1ffb640 1 -- 172.21.15.28:0/2597610792 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b0027060 con 0x55cf9e74bea0 2024-04-25T01:40:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.369+0000 7f23a1ffb640 1 -- 172.21.15.28:0/2597610792 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f23b00054e0 con 0x55cf9e74bea0 2024-04-25T01:40:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.369+0000 7f23c3dc5740 1 -- 172.21.15.28:0/2597610792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23b00047a0 con 0x55cf9e74bea0 2024-04-25T01:40:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.369+0000 7f23a1ffb640 1 -- 172.21.15.28:0/2597610792 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f23b002e050 con 0x55cf9e74bea0 2024-04-25T01:40:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.369+0000 7f23c3dc5740 10 client.18459 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.369+0000 7f23c3dc5740 1 -- 172.21.15.28:0/2597610792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cf9e3c43f0 con 0x55cf9e74bea0 2024-04-25T01:40:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.370+0000 7f23a1ffb640 1 -- 172.21.15.28:0/2597610792 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23b00047a0 con 0x55cf9e74bea0 2024-04-25T01:40:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.370+0000 7f23a1ffb640 1 -- 172.21.15.28:0/2597610792 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f23b0018530 con 0x55cf9e74bea0 2024-04-25T01:40:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.370+0000 7f23c3dc5740 10 client.18459 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.370+0000 7f23c3dc5740 10 client.18459 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:40:56.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.370+0000 7f23c3dc5740 20 client.18459 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:56.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.370+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/2597610792 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cf9e766570 0x55cf9e768960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.370+0000 7f23c3dc5740 4 client.18459 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:56.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.370+0000 7f23c3dc5740 1 -- 172.21.15.28:0/2597610792 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf9e6bdbb0 con 0x55cf9e766570 2024-04-25T01:40:56.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.370+0000 7f23ba59c640 1 --2- 172.21.15.28:0/2597610792 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cf9e766570 0x55cf9e768960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.371+0000 7f23a1ffb640 10 client.18459 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:40:56.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.371+0000 7f23ba59c640 1 --2- 172.21.15.28:0/2597610792 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cf9e766570 0x55cf9e768960 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ce3055bad5dadd0b server_cookie=af968673c852fad5 in_seq=0 out_seq=0 2024-04-25T01:40:56.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.371+0000 7f23a1ffb640 1 -- 172.21.15.28:0/2597610792 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cf9e6bdbb0 con 0x55cf9e766570 2024-04-25T01:40:56.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.371+0000 7f23a1ffb640 10 client.18459 handle_command_reply: tid=0 2024-04-25T01:40:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.371+0000 7f23c3dc5740 1 client.18459 shutdown 2024-04-25T01:40:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.371+0000 7f23c3dc5740 1 -- 172.21.15.28:0/2597610792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74bea0 msgr2=0x55cf9e753190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.371+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/2597610792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74bea0 0x55cf9e753190 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f23b0008fd0 tx=0x7f23b0005160 comp rx=0 tx=0).stop 2024-04-25T01:40:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.372+0000 7f23c3dc5740 1 -- 172.21.15.28:0/2597610792 shutdown_connections 2024-04-25T01:40:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.372+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/2597610792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9e756570 0x55cf9e7536d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.372+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/2597610792 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cf9e766570 0x55cf9e768960 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.372+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/2597610792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74bea0 0x55cf9e753190 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.372+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/2597610792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74b580 0x55cf9e755ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.372+0000 7f23a1ffb640 0 client.18459 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:40:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.372+0000 7f23c3dc5740 1 -- 172.21.15.28:0/2597610792 >> 172.21.15.28:0/2597610792 conn(0x55cf9e7478a0 msgr2=0x55cf9e74d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.372+0000 7f23c3dc5740 1 -- 172.21.15.28:0/2597610792 shutdown_connections 2024-04-25T01:40:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.372+0000 7f23c3dc5740 1 -- 172.21.15.28:0/2597610792 wait complete. 2024-04-25T01:40:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.372+0000 7f23c3dc5740 20 client.18459 trim_cache size 0 max 16384 2024-04-25T01:40:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.385+0000 7f23c3dc5740 1 Processor -- start 2024-04-25T01:40:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.385+0000 7f23c3dc5740 1 -- start start 2024-04-25T01:40:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.385+0000 7f23c3dc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74b580 0x55cf9e762250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.385+0000 7f23c3dc5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74bea0 0x55cf9e76dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.385+0000 7f23c3dc5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9e756570 0x55cf9e76e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.385+0000 7f23c3dc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f23bc1aea50 con 0x55cf9e74b580 2024-04-25T01:40:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.385+0000 7f23c3dc5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f23bc1b0590 con 0x55cf9e74bea0 2024-04-25T01:40:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.385+0000 7f23c3dc5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f23bc1af120 con 0x55cf9e756570 2024-04-25T01:40:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.385+0000 7f23ba59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74b580 0x55cf9e762250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.385+0000 7f23ba59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74b580 0x55cf9e762250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40904/0 (socket says 172.21.15.28:40904) 2024-04-25T01:40:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.385+0000 7f23ba59c640 1 -- 172.21.15.28:0/867656739 learned_addr learned my addr 172.21.15.28:0/867656739 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.385+0000 7f23ba59c640 1 -- 172.21.15.28:0/867656739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9e756570 msgr2=0x55cf9e76e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.385+0000 7f23ba59c640 1 --2- 172.21.15.28:0/867656739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9e756570 0x55cf9e76e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.385+0000 7f23ba59c640 1 -- 172.21.15.28:0/867656739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74bea0 msgr2=0x55cf9e76dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.385+0000 7f23ba59c640 1 --2- 172.21.15.28:0/867656739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74bea0 0x55cf9e76dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.385+0000 7f23ba59c640 1 -- 172.21.15.28:0/867656739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf9e48e7d0 con 0x55cf9e74b580 2024-04-25T01:40:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.386+0000 7f23ba59c640 1 --2- 172.21.15.28:0/867656739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74b580 0x55cf9e762250 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0b9650 tx=0x7f23ac043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e6e1b6fc3dfb63f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.386+0000 7f23a3fff640 1 -- 172.21.15.28:0/867656739 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ac0bd1f0 con 0x55cf9e74b580 2024-04-25T01:40:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.386+0000 7f23a3fff640 1 -- 172.21.15.28:0/867656739 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23ac036040 con 0x55cf9e74b580 2024-04-25T01:40:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.386+0000 7f23a3fff640 1 -- 172.21.15.28:0/867656739 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ac0b9860 con 0x55cf9e74b580 2024-04-25T01:40:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.386+0000 7f23c3dc5740 1 -- 172.21.15.28:0/867656739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74b580 msgr2=0x55cf9e762250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.386+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/867656739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74b580 0x55cf9e762250 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0b9650 tx=0x7f23ac043000 comp rx=0 tx=0).stop 2024-04-25T01:40:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.386+0000 7f23c3dc5740 1 -- 172.21.15.28:0/867656739 shutdown_connections 2024-04-25T01:40:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.386+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/867656739 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9e756570 0x55cf9e76e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.386+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/867656739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74bea0 0x55cf9e76dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.386+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/867656739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74b580 0x55cf9e762250 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.386+0000 7f23c3dc5740 1 -- 172.21.15.28:0/867656739 >> 172.21.15.28:0/867656739 conn(0x55cf9e7478a0 msgr2=0x55cf9e7462b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.386+0000 7f23c3dc5740 1 -- 172.21.15.28:0/867656739 shutdown_connections 2024-04-25T01:40:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.386+0000 7f23c3dc5740 1 -- 172.21.15.28:0/867656739 wait complete. 2024-04-25T01:40:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23c3dc5740 1 -- start start 2024-04-25T01:40:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23c3dc5740 1 Processor -- start 2024-04-25T01:40:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23c3dc5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74b580 0x55cf9e75e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23c3dc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74bea0 0x55cf9e75ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23c3dc5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9e756570 0x55cf9e75bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23c3dc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f23bc1afec0 con 0x55cf9e74bea0 2024-04-25T01:40:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23c3dc5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f23bc1af7f0 con 0x55cf9e74b580 2024-04-25T01:40:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23c3dc5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f23bc1aca50 con 0x55cf9e756570 2024-04-25T01:40:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23c3dc5740 10 client.0 _ll_register_callbacks cb 0x7f23b0008a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23b9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74bea0 0x55cf9e75ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23b9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74bea0 0x55cf9e75ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40920/0 (socket says 172.21.15.28:40920) 2024-04-25T01:40:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23b9d9b640 1 -- 172.21.15.28:0/3301862621 learned_addr learned my addr 172.21.15.28:0/3301862621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23bad9d640 1 --2- 172.21.15.28:0/3301862621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9e756570 0x55cf9e75bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23ba59c640 1 --2- 172.21.15.28:0/3301862621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74b580 0x55cf9e75e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23b9d9b640 1 -- 172.21.15.28:0/3301862621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9e756570 msgr2=0x55cf9e75bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23b9d9b640 1 --2- 172.21.15.28:0/3301862621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9e756570 0x55cf9e75bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23b9d9b640 1 -- 172.21.15.28:0/3301862621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74b580 msgr2=0x55cf9e75e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23b9d9b640 1 --2- 172.21.15.28:0/3301862621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74b580 0x55cf9e75e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23b9d9b640 1 -- 172.21.15.28:0/3301862621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf9e75c700 con 0x55cf9e74bea0 2024-04-25T01:40:56.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.387+0000 7f23ba59c640 1 --2- 172.21.15.28:0/3301862621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74b580 0x55cf9e75e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:56.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.388+0000 7f23b9d9b640 1 --2- 172.21.15.28:0/3301862621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74bea0 0x55cf9e75ece0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f23b0008b60 tx=0x7f23b000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab4b716f1edd53fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.388+0000 7f23a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:56.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.388+0000 7f23a1ffb640 1 -- 172.21.15.28:0/3301862621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b0027070 con 0x55cf9e74bea0 2024-04-25T01:40:56.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.388+0000 7f23c3dc5740 1 -- 172.21.15.28:0/3301862621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23ac036040 con 0x55cf9e74bea0 2024-04-25T01:40:56.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.388+0000 7f23a1ffb640 1 -- 172.21.15.28:0/3301862621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f23b00054e0 con 0x55cf9e74bea0 2024-04-25T01:40:56.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.388+0000 7f23a1ffb640 1 -- 172.21.15.28:0/3301862621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f23b003a050 con 0x55cf9e74bea0 2024-04-25T01:40:56.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.388+0000 7f23c3dc5740 10 client.18465 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:56.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.388+0000 7f23c3dc5740 1 -- 172.21.15.28:0/3301862621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cf9e7cc510 con 0x55cf9e74bea0 2024-04-25T01:40:56.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.388+0000 7f23a1ffb640 1 -- 172.21.15.28:0/3301862621 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23ac036040 con 0x55cf9e74bea0 2024-04-25T01:40:56.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.388+0000 7f23a1ffb640 1 -- 172.21.15.28:0/3301862621 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f23b00042a0 con 0x55cf9e74bea0 2024-04-25T01:40:56.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.388+0000 7f23c3dc5740 10 client.18465 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:56.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.388+0000 7f23c3dc5740 10 client.18465 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:40:56.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.388+0000 7f23c3dc5740 20 client.18465 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:56.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.388+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/3301862621 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cf9e7ccc30 0x55cf9e7cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.388+0000 7f23c3dc5740 4 client.18465 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:56.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.388+0000 7f23c3dc5740 1 -- 172.21.15.28:0/3301862621 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cf9e753a60 con 0x55cf9e7ccc30 2024-04-25T01:40:56.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.388+0000 7f23ba59c640 1 --2- 172.21.15.28:0/3301862621 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cf9e7ccc30 0x55cf9e7cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.389+0000 7f23a1ffb640 10 client.18465 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:40:56.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.389+0000 7f23ba59c640 1 --2- 172.21.15.28:0/3301862621 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cf9e7ccc30 0x55cf9e7cf050 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7b4e6c24c6043a2 server_cookie=dd6576dc6b2aa4d3 in_seq=0 out_seq=0 2024-04-25T01:40:56.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.389+0000 7f23a1ffb640 1 -- 172.21.15.28:0/3301862621 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cf9e753a60 con 0x55cf9e7ccc30 2024-04-25T01:40:56.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.389+0000 7f23a1ffb640 10 client.18465 handle_command_reply: tid=0 2024-04-25T01:40:56.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.389+0000 7f23c3dc5740 1 client.18465 shutdown 2024-04-25T01:40:56.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.389+0000 7f23c3dc5740 1 -- 172.21.15.28:0/3301862621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74bea0 msgr2=0x55cf9e75ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.389+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/3301862621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74bea0 0x55cf9e75ece0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f23b0008b60 tx=0x7f23b000ed50 comp rx=0 tx=0).stop 2024-04-25T01:40:56.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.389+0000 7f23c3dc5740 1 -- 172.21.15.28:0/3301862621 shutdown_connections 2024-04-25T01:40:56.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.389+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/3301862621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55cf9e756570 0x55cf9e75bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.389+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/3301862621 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55cf9e7ccc30 0x55cf9e7cf050 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.389+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/3301862621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf9e74bea0 0x55cf9e75ece0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.389+0000 7f23c3dc5740 1 --2- 172.21.15.28:0/3301862621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55cf9e74b580 0x55cf9e75e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.389+0000 7f23a1ffb640 0 client.18465 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:40:56.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.389+0000 7f23c3dc5740 1 -- 172.21.15.28:0/3301862621 >> 172.21.15.28:0/3301862621 conn(0x55cf9e7478a0 msgr2=0x55cf9e746b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.389+0000 7f23c3dc5740 1 -- 172.21.15.28:0/3301862621 shutdown_connections 2024-04-25T01:40:56.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.389+0000 7f23c3dc5740 1 -- 172.21.15.28:0/3301862621 wait complete. 2024-04-25T01:40:56.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.389+0000 7f23c3dc5740 20 client.18465 trim_cache size 0 max 16384 2024-04-25T01:40:56.382 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:56.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.390+0000 7f23c0fe2640 1 -- 172.21.15.28:0/1374472244 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23a40a2d00 msgr2=0x7f23a40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.390+0000 7f23c0fe2640 1 --2- 172.21.15.28:0/1374472244 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23a40a2d00 0x7f23a40a51c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f23b40058c0 tx=0x7f23b4005e70 comp rx=0 tx=0).stop 2024-04-25T01:40:56.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.390+0000 7f23c0fe2640 1 -- 172.21.15.28:0/1374472244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23bc069dc0 msgr2=0x7f23bc197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.390+0000 7f23c0fe2640 1 --2- 172.21.15.28:0/1374472244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23bc069dc0 0x7f23bc197f00 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f23ac02ae00 tx=0x7f23ac003970 comp rx=0 tx=0).stop 2024-04-25T01:40:56.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.390+0000 7f23c0fe2640 1 -- 172.21.15.28:0/1374472244 shutdown_connections 2024-04-25T01:40:56.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.390+0000 7f23c0fe2640 1 --2- 172.21.15.28:0/1374472244 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23a40a2d00 0x7f23a40a51c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.390+0000 7f23c0fe2640 1 --2- 172.21.15.28:0/1374472244 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f23bc107730 0x7f23bc19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.390+0000 7f23c0fe2640 1 --2- 172.21.15.28:0/1374472244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23bc06a780 0x7f23bc198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.390+0000 7f23c0fe2640 1 --2- 172.21.15.28:0/1374472244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23bc069dc0 0x7f23bc197f00 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.390+0000 7f23c0fe2640 1 -- 172.21.15.28:0/1374472244 >> 172.21.15.28:0/1374472244 conn(0x7f23bc0776a0 msgr2=0x7f23bc0fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.390+0000 7f23c0fe2640 1 -- 172.21.15.28:0/1374472244 shutdown_connections 2024-04-25T01:40:56.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.390+0000 7f23c0fe2640 1 -- 172.21.15.28:0/1374472244 wait complete. 2024-04-25T01:40:56.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:40:56.466 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:56 smithi130 ceph-mon[32702]: pgmap v242: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:40:56.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.505+0000 7f8e74f29640 1 -- 172.21.15.28:0/679397417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e7006a780 msgr2=0x7f8e7010e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.505+0000 7f8e74f29640 1 --2- 172.21.15.28:0/679397417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e7006a780 0x7f8e7010e5f0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f8e68009cd0 tx=0x7f8e6802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.505+0000 7f8e74f29640 1 -- 172.21.15.28:0/679397417 shutdown_connections 2024-04-25T01:40:56.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.505+0000 7f8e74f29640 1 --2- 172.21.15.28:0/679397417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e7006a780 0x7f8e7010e5f0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.505+0000 7f8e74f29640 1 --2- 172.21.15.28:0/679397417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e70069dc0 0x7f8e7006a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.505+0000 7f8e74f29640 1 --2- 172.21.15.28:0/679397417 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e70107730 0x7f8e70107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.505+0000 7f8e74f29640 1 -- 172.21.15.28:0/679397417 >> 172.21.15.28:0/679397417 conn(0x7f8e700776a0 msgr2=0x7f8e70077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.505+0000 7f8e74f29640 1 -- 172.21.15.28:0/679397417 shutdown_connections 2024-04-25T01:40:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.505+0000 7f8e74f29640 1 -- 172.21.15.28:0/679397417 wait complete. 2024-04-25T01:40:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.506+0000 7f8e74f29640 1 Processor -- start 2024-04-25T01:40:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.506+0000 7f8e74f29640 1 -- start start 2024-04-25T01:40:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.506+0000 7f8e74f29640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e70069dc0 0x7f8e70197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.506+0000 7f8e74f29640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e7006a780 0x7f8e701984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.506+0000 7f8e74f29640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e70107730 0x7f8e7019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.506+0000 7f8e74f29640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e70111b50 con 0x7f8e7006a780 2024-04-25T01:40:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.506+0000 7f8e74f29640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8e701119d0 con 0x7f8e70069dc0 2024-04-25T01:40:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.506+0000 7f8e74f29640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8e70111cd0 con 0x7f8e70107730 2024-04-25T01:40:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.506+0000 7f8e6dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e7006a780 0x7f8e701984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.506+0000 7f8e6dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e7006a780 0x7f8e701984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40946/0 (socket says 172.21.15.28:40946) 2024-04-25T01:40:56.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.506+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/1368218563 learned_addr learned my addr 172.21.15.28:0/1368218563 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.506+0000 7f8e6ed9d640 1 --2- 172.21.15.28:0/1368218563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e70107730 0x7f8e7019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.506+0000 7f8e6e59c640 1 --2- 172.21.15.28:0/1368218563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e70069dc0 0x7f8e70197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.506+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/1368218563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e70107730 msgr2=0x7f8e7019c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.506+0000 7f8e6dd9b640 1 --2- 172.21.15.28:0/1368218563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e70107730 0x7f8e7019c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.506+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/1368218563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e70069dc0 msgr2=0x7f8e70197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.506+0000 7f8e6dd9b640 1 --2- 172.21.15.28:0/1368218563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e70069dc0 0x7f8e70197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.506+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/1368218563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e7019cfa0 con 0x7f8e7006a780 2024-04-25T01:40:56.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.506+0000 7f8e6ed9d640 1 --2- 172.21.15.28:0/1368218563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e70107730 0x7f8e7019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:56.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.507+0000 7f8e6dd9b640 1 --2- 172.21.15.28:0/1368218563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e7006a780 0x7f8e701984f0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f8e6400e9f0 tx=0x7f8e6400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.507+0000 7f8e5f7fe640 1 -- 172.21.15.28:0/1368218563 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6400cd80 con 0x7f8e7006a780 2024-04-25T01:40:56.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.507+0000 7f8e5f7fe640 1 -- 172.21.15.28:0/1368218563 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e64010910 con 0x7f8e7006a780 2024-04-25T01:40:56.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.507+0000 7f8e74f29640 1 -- 172.21.15.28:0/1368218563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e7019d290 con 0x7f8e7006a780 2024-04-25T01:40:56.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.507+0000 7f8e5f7fe640 1 -- 172.21.15.28:0/1368218563 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e64018990 con 0x7f8e7006a780 2024-04-25T01:40:56.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.507+0000 7f8e74f29640 1 -- 172.21.15.28:0/1368218563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e701a4b20 con 0x7f8e7006a780 2024-04-25T01:40:56.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.510+0000 7f8e5f7fe640 1 -- 172.21.15.28:0/1368218563 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8e64010ac0 con 0x7f8e7006a780 2024-04-25T01:40:56.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.510+0000 7f8e77c38740 1 Processor -- start 2024-04-25T01:40:56.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.510+0000 7f8e77c38740 1 -- start start 2024-04-25T01:40:56.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.510+0000 7f8e77c38740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f4d66a8580 0x55f4d66a8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.510+0000 7f8e77c38740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66a8ea0 0x55f4d66b3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.510+0000 7f8e77c38740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66b3570 0x55f4d66b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.510+0000 7f8e77c38740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f4d660d670 con 0x55f4d66b3570 2024-04-25T01:40:56.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.510+0000 7f8e77c38740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f4d65b5970 con 0x55f4d66a8ea0 2024-04-25T01:40:56.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.510+0000 7f8e77c38740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55f4d6609630 con 0x55f4d66a8580 2024-04-25T01:40:56.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.510+0000 7f8e6dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66b3570 0x55f4d66b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.510+0000 7f8e6dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66b3570 0x55f4d66b5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40958/0 (socket says 172.21.15.28:40958) 2024-04-25T01:40:56.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.510+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/1543979962 learned_addr learned my addr 172.21.15.28:0/1543979962 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.510+0000 7f8e6e59c640 1 --2- 172.21.15.28:0/1543979962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66a8ea0 0x55f4d66b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.511+0000 7f8e6ed9d640 1 --2- 172.21.15.28:0/1543979962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f4d66a8580 0x55f4d66a8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.511+0000 7f8e5f7fe640 1 --2- 172.21.15.28:0/1368218563 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8e580a2d80 0x7f8e580a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.511+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/1543979962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f4d66a8580 msgr2=0x55f4d66a8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.511+0000 7f8e6dd9b640 1 --2- 172.21.15.28:0/1543979962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f4d66a8580 0x55f4d66a8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.511+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/1543979962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66a8ea0 msgr2=0x55f4d66b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.511+0000 7f8e6dd9b640 1 --2- 172.21.15.28:0/1543979962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66a8ea0 0x55f4d66b3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.511+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/1543979962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4d65d9400 con 0x55f4d66b3570 2024-04-25T01:40:56.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.511+0000 7f8e6ed9d640 1 --2- 172.21.15.28:0/1368218563 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8e580a2d80 0x7f8e580a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.511+0000 7f8e5f7fe640 1 -- 172.21.15.28:0/1368218563 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f8e640cfbb0 con 0x7f8e7006a780 2024-04-25T01:40:56.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.511+0000 7f8e6dd9b640 1 --2- 172.21.15.28:0/1543979962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66b3570 0x55f4d66b5960 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f8e640ccee0 tx=0x7f8e64066980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d187c25e09b7b096 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.511+0000 7f8e5cff9640 1 -- 172.21.15.28:0/1543979962 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6409b880 con 0x55f4d66b3570 2024-04-25T01:40:56.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.511+0000 7f8e5cff9640 1 -- 172.21.15.28:0/1543979962 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e64026050 con 0x55f4d66b3570 2024-04-25T01:40:56.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.511+0000 7f8e5cff9640 1 -- 172.21.15.28:0/1543979962 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6409b6a0 con 0x55f4d66b3570 2024-04-25T01:40:56.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.511+0000 7f8e6ed9d640 1 --2- 172.21.15.28:0/1368218563 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8e580a2d80 0x7f8e580a5240 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8e68004af0 tx=0x7f8e6803e040 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.512+0000 7f8e77c38740 1 -- 172.21.15.28:0/1543979962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66b3570 msgr2=0x55f4d66b5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.512+0000 7f8e77c38740 1 --2- 172.21.15.28:0/1543979962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66b3570 0x55f4d66b5960 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f8e640ccee0 tx=0x7f8e64066980 comp rx=0 tx=0).stop 2024-04-25T01:40:56.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.512+0000 7f8e77c38740 1 -- 172.21.15.28:0/1543979962 shutdown_connections 2024-04-25T01:40:56.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.512+0000 7f8e77c38740 1 --2- 172.21.15.28:0/1543979962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66b3570 0x55f4d66b5960 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.512+0000 7f8e77c38740 1 --2- 172.21.15.28:0/1543979962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66a8ea0 0x55f4d66b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.512+0000 7f8e77c38740 1 --2- 172.21.15.28:0/1543979962 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f4d66a8580 0x55f4d66a8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.512+0000 7f8e77c38740 1 -- 172.21.15.28:0/1543979962 >> 172.21.15.28:0/1543979962 conn(0x55f4d66a48a0 msgr2=0x55f4d66a4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.512+0000 7f8e77c38740 1 -- 172.21.15.28:0/1543979962 shutdown_connections 2024-04-25T01:40:56.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.512+0000 7f8e77c38740 1 -- 172.21.15.28:0/1543979962 wait complete. 2024-04-25T01:40:56.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.512+0000 7f8e77c38740 1 -- start start 2024-04-25T01:40:56.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.512+0000 7f8e77c38740 1 Processor -- start 2024-04-25T01:40:56.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e77c38740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66a8580 0x55f4d66b2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e77c38740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 0x55f4d66b01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e77c38740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f4d66b3570 0x55f4d66b0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e77c38740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f4d66b6300 con 0x55f4d66a8ea0 2024-04-25T01:40:56.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e77c38740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f4d65c9050 con 0x55f4d66a8580 2024-04-25T01:40:56.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e77c38740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55f4d66b6480 con 0x55f4d66b3570 2024-04-25T01:40:56.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e77c38740 10 client.0 _ll_register_callbacks cb 0x55f4d6320be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:56.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e6dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 0x55f4d66b01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e6dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 0x55f4d66b01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40960/0 (socket says 172.21.15.28:40960) 2024-04-25T01:40:56.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/3817973796 learned_addr learned my addr 172.21.15.28:0/3817973796 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e6e59c640 1 --2- 172.21.15.28:0/3817973796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66a8580 0x55f4d66b2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/3817973796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f4d66b3570 msgr2=0x55f4d66b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e6dd9b640 1 --2- 172.21.15.28:0/3817973796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f4d66b3570 0x55f4d66b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/3817973796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66a8580 msgr2=0x55f4d66b2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e6dd9b640 1 --2- 172.21.15.28:0/3817973796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66a8580 0x55f4d66b2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/3817973796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4d659adc0 con 0x55f4d66a8ea0 2024-04-25T01:40:56.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e6dd9b640 1 --2- 172.21.15.28:0/3817973796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 0x55f4d66b01b0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f8e640a4480 tx=0x7f8e640ccf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dcb165889266e21 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e55ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:56.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e55ffb640 1 -- 172.21.15.28:0/3817973796 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6409b6a0 con 0x55f4d66a8ea0 2024-04-25T01:40:56.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e55ffb640 1 -- 172.21.15.28:0/3817973796 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f8e640d3390 con 0x55f4d66a8ea0 2024-04-25T01:40:56.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.513+0000 7f8e77c38740 1 -- 172.21.15.28:0/3817973796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e64026050 con 0x55f4d66a8ea0 2024-04-25T01:40:56.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.514+0000 7f8e55ffb640 1 -- 172.21.15.28:0/3817973796 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8e6409b880 con 0x55f4d66a8ea0 2024-04-25T01:40:56.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.514+0000 7f8e77c38740 10 client.18477 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:56.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.514+0000 7f8e77c38740 1 -- 172.21.15.28:0/3817973796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f4d63213f0 con 0x55f4d66a8ea0 2024-04-25T01:40:56.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.514+0000 7f8e55ffb640 1 -- 172.21.15.28:0/3817973796 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e64026050 con 0x55f4d66a8ea0 2024-04-25T01:40:56.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.514+0000 7f8e55ffb640 1 -- 172.21.15.28:0/3817973796 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8e640983b0 con 0x55f4d66a8ea0 2024-04-25T01:40:56.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.514+0000 7f8e77c38740 10 client.18477 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:56.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.514+0000 7f8e77c38740 10 client.18477 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:40:56.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.514+0000 7f8e77c38740 20 client.18477 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:56.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.514+0000 7f8e77c38740 1 --2- 172.21.15.28:0/3817973796 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55f4d66c7830 0x55f4d66c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.514+0000 7f8e77c38740 4 client.18477 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:56.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.514+0000 7f8e77c38740 1 -- 172.21.15.28:0/3817973796 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4d661abb0 con 0x55f4d66c7830 2024-04-25T01:40:56.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.514+0000 7f8e6e59c640 1 --2- 172.21.15.28:0/3817973796 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55f4d66c7830 0x55f4d66c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.515+0000 7f8e55ffb640 10 client.18477 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:40:56.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.515+0000 7f8e6e59c640 1 --2- 172.21.15.28:0/3817973796 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55f4d66c7830 0x55f4d66c9c20 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9d79667124861e62 server_cookie=43d13d27acc83014 in_seq=0 out_seq=0 2024-04-25T01:40:56.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.515+0000 7f8e55ffb640 1 -- 172.21.15.28:0/3817973796 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f4d661abb0 con 0x55f4d66c7830 2024-04-25T01:40:56.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.515+0000 7f8e55ffb640 10 client.18477 handle_command_reply: tid=0 2024-04-25T01:40:56.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.515+0000 7f8e77c38740 1 client.18477 shutdown 2024-04-25T01:40:56.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.516+0000 7f8e77c38740 1 -- 172.21.15.28:0/3817973796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 msgr2=0x55f4d66b01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.516+0000 7f8e77c38740 1 --2- 172.21.15.28:0/3817973796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 0x55f4d66b01b0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f8e640a4480 tx=0x7f8e640ccf10 comp rx=0 tx=0).stop 2024-04-25T01:40:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.516+0000 7f8e77c38740 1 -- 172.21.15.28:0/3817973796 shutdown_connections 2024-04-25T01:40:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.516+0000 7f8e77c38740 1 --2- 172.21.15.28:0/3817973796 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55f4d66c7830 0x55f4d66c9c20 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.516+0000 7f8e77c38740 1 --2- 172.21.15.28:0/3817973796 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f4d66b3570 0x55f4d66b0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.516+0000 7f8e77c38740 1 --2- 172.21.15.28:0/3817973796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 0x55f4d66b01b0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.516+0000 7f8e77c38740 1 --2- 172.21.15.28:0/3817973796 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66a8580 0x55f4d66b2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.516+0000 7f8e77c38740 1 -- 172.21.15.28:0/3817973796 >> 172.21.15.28:0/3817973796 conn(0x55f4d66a48a0 msgr2=0x55f4d66a3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.516+0000 7f8e55ffb640 0 client.18477 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:40:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.516+0000 7f8e77c38740 1 -- 172.21.15.28:0/3817973796 shutdown_connections 2024-04-25T01:40:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.516+0000 7f8e77c38740 1 -- 172.21.15.28:0/3817973796 wait complete. 2024-04-25T01:40:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.516+0000 7f8e77c38740 20 client.18477 trim_cache size 0 max 16384 2024-04-25T01:40:56.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.529+0000 7f8e77c38740 1 Processor -- start 2024-04-25T01:40:56.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.529+0000 7f8e77c38740 1 -- start start 2024-04-25T01:40:56.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.529+0000 7f8e77c38740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f4d66a8580 0x55f4d66b0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.529+0000 7f8e77c38740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 0x55f4d66cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.529+0000 7f8e77c38740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66b3570 0x55f4d66cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.529+0000 7f8e77c38740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e701ac9f0 con 0x55f4d66a8ea0 2024-04-25T01:40:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.529+0000 7f8e77c38740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8e701ae530 con 0x55f4d66b3570 2024-04-25T01:40:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.529+0000 7f8e77c38740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8e701ad0c0 con 0x55f4d66a8580 2024-04-25T01:40:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.529+0000 7f8e6dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 0x55f4d66cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.529+0000 7f8e6dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 0x55f4d66cf220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40966/0 (socket says 172.21.15.28:40966) 2024-04-25T01:40:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.529+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/3631431488 learned_addr learned my addr 172.21.15.28:0/3631431488 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.530+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/3631431488 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f4d66a8580 msgr2=0x55f4d66b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.530+0000 7f8e6e59c640 1 --2- 172.21.15.28:0/3631431488 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f4d66a8580 0x55f4d66b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.530+0000 7f8e6dd9b640 1 --2- 172.21.15.28:0/3631431488 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f4d66a8580 0x55f4d66b0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.530+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/3631431488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66b3570 msgr2=0x55f4d66cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.530+0000 7f8e6ed9d640 1 --2- 172.21.15.28:0/3631431488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66b3570 0x55f4d66cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.530+0000 7f8e6dd9b640 1 --2- 172.21.15.28:0/3631431488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66b3570 0x55f4d66cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.530+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/3631431488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4d63eb7d0 con 0x55f4d66a8ea0 2024-04-25T01:40:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.530+0000 7f8e6e59c640 1 --2- 172.21.15.28:0/3631431488 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f4d66a8580 0x55f4d66b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.530+0000 7f8e6ed9d640 1 --2- 172.21.15.28:0/3631431488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66b3570 0x55f4d66cf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.530+0000 7f8e6dd9b640 1 --2- 172.21.15.28:0/3631431488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 0x55f4d66cf220 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f8e640b0170 tx=0x7f8e640ba7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f226bcedd1e43152 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.530+0000 7f8e57fff640 1 -- 172.21.15.28:0/3631431488 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6409b6a0 con 0x55f4d66a8ea0 2024-04-25T01:40:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.530+0000 7f8e57fff640 1 -- 172.21.15.28:0/3631431488 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e6406d070 con 0x55f4d66a8ea0 2024-04-25T01:40:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.530+0000 7f8e57fff640 1 -- 172.21.15.28:0/3631431488 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6409bb10 con 0x55f4d66a8ea0 2024-04-25T01:40:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.530+0000 7f8e77c38740 1 -- 172.21.15.28:0/3631431488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 msgr2=0x55f4d66cf220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.530+0000 7f8e77c38740 1 --2- 172.21.15.28:0/3631431488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 0x55f4d66cf220 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f8e640b0170 tx=0x7f8e640ba7f0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e77c38740 1 -- 172.21.15.28:0/3631431488 shutdown_connections 2024-04-25T01:40:56.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e77c38740 1 --2- 172.21.15.28:0/3631431488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66b3570 0x55f4d66cf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e77c38740 1 --2- 172.21.15.28:0/3631431488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 0x55f4d66cf220 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e77c38740 1 --2- 172.21.15.28:0/3631431488 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f4d66a8580 0x55f4d66b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e77c38740 1 -- 172.21.15.28:0/3631431488 >> 172.21.15.28:0/3631431488 conn(0x55f4d66a48a0 msgr2=0x55f4d66a3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e77c38740 1 -- 172.21.15.28:0/3631431488 shutdown_connections 2024-04-25T01:40:56.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e77c38740 1 -- 172.21.15.28:0/3631431488 wait complete. 2024-04-25T01:40:56.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e77c38740 1 -- start start 2024-04-25T01:40:56.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e77c38740 1 Processor -- start 2024-04-25T01:40:56.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e77c38740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f4d66a8580 0x55f4d66bfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e77c38740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 0x55f4d66c0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e77c38740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66b3570 0x55f4d66bd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e77c38740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e701ade60 con 0x55f4d66a8ea0 2024-04-25T01:40:56.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e77c38740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8e701ad790 con 0x55f4d66b3570 2024-04-25T01:40:56.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e77c38740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8e701aa9f0 con 0x55f4d66a8580 2024-04-25T01:40:56.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e77c38740 10 client.0 _ll_register_callbacks cb 0x7f8e6000d360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e6dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 0x55f4d66c0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e6dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 0x55f4d66c0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40972/0 (socket says 172.21.15.28:40972) 2024-04-25T01:40:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/3165351850 learned_addr learned my addr 172.21.15.28:0/3165351850 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e6ed9d640 1 --2- 172.21.15.28:0/3165351850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66b3570 0x55f4d66bd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/3165351850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f4d66a8580 msgr2=0x55f4d66bfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e6dd9b640 1 --2- 172.21.15.28:0/3165351850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f4d66a8580 0x55f4d66bfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.531+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/3165351850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66b3570 msgr2=0x55f4d66bd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.532+0000 7f8e6dd9b640 1 --2- 172.21.15.28:0/3165351850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66b3570 0x55f4d66bd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.532+0000 7f8e6dd9b640 1 -- 172.21.15.28:0/3165351850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4d66bd870 con 0x55f4d66a8ea0 2024-04-25T01:40:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.532+0000 7f8e6ed9d640 1 --2- 172.21.15.28:0/3165351850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66b3570 0x55f4d66bd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.532+0000 7f8e6dd9b640 1 --2- 172.21.15.28:0/3165351850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 0x55f4d66c0080 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f8e64098710 tx=0x7f8e640d35c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe56a45bdb2fd04 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.532+0000 7f8e55ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:56.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.532+0000 7f8e55ffb640 1 -- 172.21.15.28:0/3165351850 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6409bb10 con 0x55f4d66a8ea0 2024-04-25T01:40:56.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.532+0000 7f8e55ffb640 1 -- 172.21.15.28:0/3165351850 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f8e640a4730 con 0x55f4d66a8ea0 2024-04-25T01:40:56.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.532+0000 7f8e77c38740 1 -- 172.21.15.28:0/3165351850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e6406d070 con 0x55f4d66a8ea0 2024-04-25T01:40:56.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.532+0000 7f8e55ffb640 1 -- 172.21.15.28:0/3165351850 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8e6409b6a0 con 0x55f4d66a8ea0 2024-04-25T01:40:56.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.532+0000 7f8e77c38740 10 client.18483 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:56.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.532+0000 7f8e77c38740 1 -- 172.21.15.28:0/3165351850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f4d66bd870 con 0x55f4d66a8ea0 2024-04-25T01:40:56.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.532+0000 7f8e55ffb640 1 -- 172.21.15.28:0/3165351850 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e6406d070 con 0x55f4d66a8ea0 2024-04-25T01:40:56.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.532+0000 7f8e55ffb640 1 -- 172.21.15.28:0/3165351850 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8e6409bfa0 con 0x55f4d66a8ea0 2024-04-25T01:40:56.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.532+0000 7f8e77c38740 10 client.18483 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:56.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.532+0000 7f8e77c38740 10 client.18483 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:40:56.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.532+0000 7f8e77c38740 20 client.18483 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:56.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.532+0000 7f8e77c38740 1 --2- 172.21.15.28:0/3165351850 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55f4d672d910 0x55f4d672fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.532+0000 7f8e77c38740 4 client.18483 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:56.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.532+0000 7f8e77c38740 1 -- 172.21.15.28:0/3165351850 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f4d6614f00 con 0x55f4d672d910 2024-04-25T01:40:56.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.533+0000 7f8e6e59c640 1 --2- 172.21.15.28:0/3165351850 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55f4d672d910 0x55f4d672fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.533+0000 7f8e6e59c640 1 --2- 172.21.15.28:0/3165351850 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55f4d672d910 0x55f4d672fd00 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=215e7a3afddc2b95 server_cookie=d5d78c631d28a98 in_seq=0 out_seq=0 2024-04-25T01:40:56.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.533+0000 7f8e55ffb640 10 client.18483 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:40:56.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.533+0000 7f8e55ffb640 1 -- 172.21.15.28:0/3165351850 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f4d6614f00 con 0x55f4d672d910 2024-04-25T01:40:56.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.533+0000 7f8e55ffb640 10 client.18483 handle_command_reply: tid=0 2024-04-25T01:40:56.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.533+0000 7f8e77c38740 1 client.18483 shutdown 2024-04-25T01:40:56.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.533+0000 7f8e77c38740 1 -- 172.21.15.28:0/3165351850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 msgr2=0x55f4d66c0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.533+0000 7f8e77c38740 1 --2- 172.21.15.28:0/3165351850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 0x55f4d66c0080 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f8e64098710 tx=0x7f8e640d35c0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.533+0000 7f8e77c38740 1 -- 172.21.15.28:0/3165351850 shutdown_connections 2024-04-25T01:40:56.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.533+0000 7f8e77c38740 1 --2- 172.21.15.28:0/3165351850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f4d66b3570 0x55f4d66bd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.533+0000 7f8e77c38740 1 --2- 172.21.15.28:0/3165351850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d66a8ea0 0x55f4d66c0080 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.533+0000 7f8e77c38740 1 --2- 172.21.15.28:0/3165351850 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55f4d672d910 0x55f4d672fd00 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.533+0000 7f8e77c38740 1 --2- 172.21.15.28:0/3165351850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f4d66a8580 0x55f4d66bfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.533+0000 7f8e55ffb640 0 client.18483 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:40:56.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.533+0000 7f8e77c38740 1 -- 172.21.15.28:0/3165351850 >> 172.21.15.28:0/3165351850 conn(0x55f4d66a48a0 msgr2=0x55f4d66a3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.533+0000 7f8e77c38740 1 -- 172.21.15.28:0/3165351850 shutdown_connections 2024-04-25T01:40:56.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.533+0000 7f8e77c38740 1 -- 172.21.15.28:0/3165351850 wait complete. 2024-04-25T01:40:56.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.533+0000 7f8e77c38740 20 client.18483 trim_cache size 0 max 16384 2024-04-25T01:40:56.527 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:56.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.534+0000 7f8e74f29640 1 -- 172.21.15.28:0/1368218563 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8e580a2d80 msgr2=0x7f8e580a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.534+0000 7f8e74f29640 1 --2- 172.21.15.28:0/1368218563 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8e580a2d80 0x7f8e580a5240 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8e68004af0 tx=0x7f8e6803e040 comp rx=0 tx=0).stop 2024-04-25T01:40:56.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.534+0000 7f8e74f29640 1 -- 172.21.15.28:0/1368218563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e7006a780 msgr2=0x7f8e701984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.534+0000 7f8e74f29640 1 --2- 172.21.15.28:0/1368218563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e7006a780 0x7f8e701984f0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f8e6400e9f0 tx=0x7f8e6400eeb0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.534+0000 7f8e74f29640 1 -- 172.21.15.28:0/1368218563 shutdown_connections 2024-04-25T01:40:56.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.534+0000 7f8e74f29640 1 --2- 172.21.15.28:0/1368218563 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8e580a2d80 0x7f8e580a5240 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.534+0000 7f8e74f29640 1 --2- 172.21.15.28:0/1368218563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8e70107730 0x7f8e7019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.534+0000 7f8e74f29640 1 --2- 172.21.15.28:0/1368218563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e7006a780 0x7f8e701984f0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.534+0000 7f8e74f29640 1 --2- 172.21.15.28:0/1368218563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8e70069dc0 0x7f8e70197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.534+0000 7f8e74f29640 1 -- 172.21.15.28:0/1368218563 >> 172.21.15.28:0/1368218563 conn(0x7f8e700776a0 msgr2=0x7f8e700fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.534+0000 7f8e74f29640 1 -- 172.21.15.28:0/1368218563 shutdown_connections 2024-04-25T01:40:56.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.534+0000 7f8e74f29640 1 -- 172.21.15.28:0/1368218563 wait complete. 2024-04-25T01:40:56.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:40:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.649+0000 7f487f109640 1 -- 172.21.15.28:0/2628909264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48781002e0 msgr2=0x7f48781006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.649+0000 7f487f109640 1 --2- 172.21.15.28:0/2628909264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48781002e0 0x7f48781006c0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f486c00aa10 tx=0x7f486c02b720 comp rx=0 tx=0).stop 2024-04-25T01:40:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.649+0000 7f487f109640 1 -- 172.21.15.28:0/2628909264 shutdown_connections 2024-04-25T01:40:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.649+0000 7f487f109640 1 --2- 172.21.15.28:0/2628909264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4878105160 0x7f487810cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.649+0000 7f487f109640 1 --2- 172.21.15.28:0/2628909264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4878100c90 0x7f4878104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.649+0000 7f487f109640 1 --2- 172.21.15.28:0/2628909264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48781002e0 0x7f48781006c0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.649+0000 7f487f109640 1 -- 172.21.15.28:0/2628909264 >> 172.21.15.28:0/2628909264 conn(0x7f48780758d0 msgr2=0x7f4878075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.649+0000 7f487f109640 1 -- 172.21.15.28:0/2628909264 shutdown_connections 2024-04-25T01:40:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.649+0000 7f487f109640 1 -- 172.21.15.28:0/2628909264 wait complete. 2024-04-25T01:40:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.650+0000 7f487f109640 1 Processor -- start 2024-04-25T01:40:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.650+0000 7f487f109640 1 -- start start 2024-04-25T01:40:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.650+0000 7f487f109640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48781002e0 0x7f48781980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.650+0000 7f487f109640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4878105160 0x7f48781985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.650+0000 7f487f109640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f487819c910 0x7f487819cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.650+0000 7f487f109640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f487810f960 con 0x7f48781002e0 2024-04-25T01:40:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.650+0000 7f487f109640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f487810f7e0 con 0x7f487819c910 2024-04-25T01:40:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.650+0000 7f487f109640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f487810fae0 con 0x7f4878105160 2024-04-25T01:40:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.650+0000 7f487cea5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48781002e0 0x7f48781980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.650+0000 7f487cea5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48781002e0 0x7f48781980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40988/0 (socket says 172.21.15.28:40988) 2024-04-25T01:40:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.650+0000 7f487cea5640 1 -- 172.21.15.28:0/910117271 learned_addr learned my addr 172.21.15.28:0/910117271 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.651+0000 7f4877fff640 1 --2- 172.21.15.28:0/910117271 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4878105160 0x7f48781985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.651+0000 7f487d6a6640 1 --2- 172.21.15.28:0/910117271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f487819c910 0x7f487819cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.651+0000 7f487cea5640 1 -- 172.21.15.28:0/910117271 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4878105160 msgr2=0x7f48781985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.651+0000 7f487cea5640 1 --2- 172.21.15.28:0/910117271 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4878105160 0x7f48781985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.651+0000 7f487cea5640 1 -- 172.21.15.28:0/910117271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f487819c910 msgr2=0x7f487819cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.651+0000 7f487cea5640 1 --2- 172.21.15.28:0/910117271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f487819c910 0x7f487819cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.651+0000 7f487cea5640 1 -- 172.21.15.28:0/910117271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f487819d400 con 0x7f48781002e0 2024-04-25T01:40:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.651+0000 7f487d6a6640 1 --2- 172.21.15.28:0/910117271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f487819c910 0x7f487819cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.651+0000 7f487cea5640 1 --2- 172.21.15.28:0/910117271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48781002e0 0x7f48781980a0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f486c005940 tx=0x7f486c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.651+0000 7f4875ffb640 1 -- 172.21.15.28:0/910117271 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486c03f070 con 0x7f48781002e0 2024-04-25T01:40:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.651+0000 7f4875ffb640 1 -- 172.21.15.28:0/910117271 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f486c032400 con 0x7f48781002e0 2024-04-25T01:40:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.651+0000 7f487f109640 1 -- 172.21.15.28:0/910117271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f487819d690 con 0x7f48781002e0 2024-04-25T01:40:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.651+0000 7f4875ffb640 1 -- 172.21.15.28:0/910117271 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486c036040 con 0x7f48781002e0 2024-04-25T01:40:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.651+0000 7f487f109640 1 -- 172.21.15.28:0/910117271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48781a4eb0 con 0x7f48781002e0 2024-04-25T01:40:56.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.653+0000 7f4875ffb640 1 -- 172.21.15.28:0/910117271 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f486c042070 con 0x7f48781002e0 2024-04-25T01:40:56.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f4881e0d740 1 Processor -- start 2024-04-25T01:40:56.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f4881e0d740 1 -- start start 2024-04-25T01:40:56.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f4881e0d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5580 0x55e00ffb5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f4881e0d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffb5ea0 0x55e00ffc0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f4881e0d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffc0570 0x55e00ffc2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f4881e0d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e00ff1a670 con 0x55e00ffb5580 2024-04-25T01:40:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f4881e0d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e00fec2970 con 0x55e00ffc0570 2024-04-25T01:40:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f4881e0d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55e00ff16630 con 0x55e00ffb5ea0 2024-04-25T01:40:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f4875ffb640 1 --2- 172.21.15.28:0/910117271 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f48680a2d80 0x7f48680a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f487d6a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5580 0x55e00ffb5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f487d6a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5580 0x55e00ffb5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40996/0 (socket says 172.21.15.28:40996) 2024-04-25T01:40:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f487d6a6640 1 -- 172.21.15.28:0/870579833 learned_addr learned my addr 172.21.15.28:0/870579833 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f4875ffb640 1 -- 172.21.15.28:0/910117271 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f486c0efd20 con 0x7f48781002e0 2024-04-25T01:40:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f487d6a6640 1 --2- 172.21.15.28:0/910117271 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f48680a2d80 0x7f48680a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f4877fff640 1 --2- 172.21.15.28:0/870579833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffc0570 0x55e00ffc2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f487cea5640 1 --2- 172.21.15.28:0/870579833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffb5ea0 0x55e00ffc0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f487d6a6640 1 -- 172.21.15.28:0/870579833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffb5ea0 msgr2=0x55e00ffc0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f487d6a6640 1 --2- 172.21.15.28:0/870579833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffb5ea0 0x55e00ffc0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f487d6a6640 1 -- 172.21.15.28:0/870579833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffc0570 msgr2=0x55e00ffc2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f487d6a6640 1 --2- 172.21.15.28:0/870579833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffc0570 0x55e00ffc2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f487d6a6640 1 -- 172.21.15.28:0/870579833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e00fee6400 con 0x55e00ffb5580 2024-04-25T01:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.654+0000 7f4877fff640 1 --2- 172.21.15.28:0/870579833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffc0570 0x55e00ffc2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.655+0000 7f487d6a6640 1 --2- 172.21.15.28:0/870579833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5580 0x55e00ffb5960 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f4870014a90 tx=0x7f4870014f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80cf155d33cd138d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.655+0000 7f4862ffd640 1 -- 172.21.15.28:0/870579833 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4870014530 con 0x55e00ffb5580 2024-04-25T01:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.655+0000 7f4862ffd640 1 -- 172.21.15.28:0/870579833 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4870023070 con 0x55e00ffb5580 2024-04-25T01:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.655+0000 7f4862ffd640 1 -- 172.21.15.28:0/870579833 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487000ce20 con 0x55e00ffb5580 2024-04-25T01:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.655+0000 7f4881e0d740 1 -- 172.21.15.28:0/870579833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5580 msgr2=0x55e00ffb5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.655+0000 7f487d6a6640 1 --2- 172.21.15.28:0/910117271 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f48680a2d80 0x7f48680a5240 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4870022000 tx=0x7f4870020000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.655+0000 7f4881e0d740 1 --2- 172.21.15.28:0/870579833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5580 0x55e00ffb5960 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f4870014a90 tx=0x7f4870014f50 comp rx=0 tx=0).stop 2024-04-25T01:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.656+0000 7f4881e0d740 1 -- 172.21.15.28:0/870579833 shutdown_connections 2024-04-25T01:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.656+0000 7f4881e0d740 1 --2- 172.21.15.28:0/870579833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffc0570 0x55e00ffc2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.656+0000 7f4881e0d740 1 --2- 172.21.15.28:0/870579833 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffb5ea0 0x55e00ffc0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.656+0000 7f4881e0d740 1 --2- 172.21.15.28:0/870579833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5580 0x55e00ffb5960 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.656+0000 7f4881e0d740 1 -- 172.21.15.28:0/870579833 >> 172.21.15.28:0/870579833 conn(0x55e00ffb18a0 msgr2=0x55e00ffb1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.656+0000 7f4881e0d740 1 -- 172.21.15.28:0/870579833 shutdown_connections 2024-04-25T01:40:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.656+0000 7f4881e0d740 1 -- 172.21.15.28:0/870579833 wait complete. 2024-04-25T01:40:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.656+0000 7f4881e0d740 1 -- start start 2024-04-25T01:40:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.656+0000 7f4881e0d740 1 Processor -- start 2024-04-25T01:40:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.656+0000 7f4881e0d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5580 0x55e00ffbfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.656+0000 7f4881e0d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffb5ea0 0x55e00ffbd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.656+0000 7f4881e0d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffc0570 0x55e00ffbd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.656+0000 7f4881e0d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e00ffc3360 con 0x55e00ffb5580 2024-04-25T01:40:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.656+0000 7f4881e0d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55e00fed6050 con 0x55e00ffb5ea0 2024-04-25T01:40:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.656+0000 7f4881e0d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55e00ffc34e0 con 0x55e00ffc0570 2024-04-25T01:40:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.656+0000 7f4881e0d740 10 client.0 _ll_register_callbacks cb 0x55e00fc2dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:56.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.656+0000 7f487cea5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5580 0x55e00ffbfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.657+0000 7f487cea5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5580 0x55e00ffbfe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41010/0 (socket says 172.21.15.28:41010) 2024-04-25T01:40:56.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.657+0000 7f487cea5640 1 -- 172.21.15.28:0/1405334780 learned_addr learned my addr 172.21.15.28:0/1405334780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.657+0000 7f487d6a6640 1 --2- 172.21.15.28:0/1405334780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffc0570 0x55e00ffbd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.657+0000 7f4877fff640 1 --2- 172.21.15.28:0/1405334780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffb5ea0 0x55e00ffbd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.657+0000 7f487cea5640 1 -- 172.21.15.28:0/1405334780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffc0570 msgr2=0x55e00ffbd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.657+0000 7f487cea5640 1 --2- 172.21.15.28:0/1405334780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffc0570 0x55e00ffbd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.657+0000 7f487cea5640 1 -- 172.21.15.28:0/1405334780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffb5ea0 msgr2=0x55e00ffbd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.657+0000 7f487cea5640 1 --2- 172.21.15.28:0/1405334780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffb5ea0 0x55e00ffbd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.657+0000 7f487cea5640 1 -- 172.21.15.28:0/1405334780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e00fea7dc0 con 0x55e00ffb5580 2024-04-25T01:40:56.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.657+0000 7f487cea5640 1 --2- 172.21.15.28:0/1405334780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5580 0x55e00ffbfe30 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f486c0b8880 tx=0x7f486c0bbd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=453f023f425e30d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.657+0000 7f4857fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:56.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.657+0000 7f4857fff640 1 -- 172.21.15.28:0/1405334780 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486c049040 con 0x55e00ffb5580 2024-04-25T01:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.657+0000 7f4857fff640 1 -- 172.21.15.28:0/1405334780 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f486c0b8520 con 0x55e00ffb5580 2024-04-25T01:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.657+0000 7f4881e0d740 1 -- 172.21.15.28:0/1405334780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4870023070 con 0x55e00ffb5580 2024-04-25T01:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.657+0000 7f4857fff640 1 -- 172.21.15.28:0/1405334780 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f486c0bc5b0 con 0x55e00ffb5580 2024-04-25T01:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.657+0000 7f4881e0d740 10 client.18495 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.657+0000 7f4881e0d740 1 -- 172.21.15.28:0/1405334780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e00fc2e3f0 con 0x55e00ffb5580 2024-04-25T01:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.657+0000 7f4857fff640 1 -- 172.21.15.28:0/1405334780 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4870023070 con 0x55e00ffb5580 2024-04-25T01:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.658+0000 7f4857fff640 1 -- 172.21.15.28:0/1405334780 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f486c00b040 con 0x55e00ffb5580 2024-04-25T01:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.658+0000 7f4881e0d740 10 client.18495 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.658+0000 7f4881e0d740 10 client.18495 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.658+0000 7f4881e0d740 20 client.18495 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.658+0000 7f4881e0d740 1 --2- 172.21.15.28:0/1405334780 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55e00ffd0550 0x55e00ffd29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.658+0000 7f4881e0d740 4 client.18495 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.658+0000 7f4881e0d740 1 -- 172.21.15.28:0/1405334780 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e00ff27bb0 con 0x55e00ffd0550 2024-04-25T01:40:56.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.658+0000 7f4877fff640 1 --2- 172.21.15.28:0/1405334780 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55e00ffd0550 0x55e00ffd29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.658+0000 7f4857fff640 10 client.18495 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.658+0000 7f4877fff640 1 --2- 172.21.15.28:0/1405334780 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55e00ffd0550 0x55e00ffd29a0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e4633ec4cdd26eb4 server_cookie=22eaddfb477c42ab in_seq=0 out_seq=0 2024-04-25T01:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.659+0000 7f4857fff640 1 -- 172.21.15.28:0/1405334780 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e00ff27bb0 con 0x55e00ffd0550 2024-04-25T01:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.659+0000 7f4857fff640 10 client.18495 handle_command_reply: tid=0 2024-04-25T01:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.659+0000 7f4881e0d740 1 client.18495 shutdown 2024-04-25T01:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.659+0000 7f4881e0d740 1 -- 172.21.15.28:0/1405334780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5580 msgr2=0x55e00ffbfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.659+0000 7f4881e0d740 1 --2- 172.21.15.28:0/1405334780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5580 0x55e00ffbfe30 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f486c0b8880 tx=0x7f486c0bbd40 comp rx=0 tx=0).stop 2024-04-25T01:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.659+0000 7f4881e0d740 1 -- 172.21.15.28:0/1405334780 shutdown_connections 2024-04-25T01:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.659+0000 7f4881e0d740 1 --2- 172.21.15.28:0/1405334780 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55e00ffd0550 0x55e00ffd29a0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.659+0000 7f4881e0d740 1 --2- 172.21.15.28:0/1405334780 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffc0570 0x55e00ffbd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.659+0000 7f4881e0d740 1 --2- 172.21.15.28:0/1405334780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffb5ea0 0x55e00ffbd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.659+0000 7f4881e0d740 1 --2- 172.21.15.28:0/1405334780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5580 0x55e00ffbfe30 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.659+0000 7f4857fff640 0 client.18495 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:40:56.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.659+0000 7f4881e0d740 1 -- 172.21.15.28:0/1405334780 >> 172.21.15.28:0/1405334780 conn(0x55e00ffb18a0 msgr2=0x55e00ffb76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.659+0000 7f4881e0d740 1 -- 172.21.15.28:0/1405334780 shutdown_connections 2024-04-25T01:40:56.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.659+0000 7f4881e0d740 1 -- 172.21.15.28:0/1405334780 wait complete. 2024-04-25T01:40:56.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.660+0000 7f4881e0d740 20 client.18495 trim_cache size 0 max 16384 2024-04-25T01:40:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.673+0000 7f4881e0d740 1 Processor -- start 2024-04-25T01:40:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.673+0000 7f4881e0d740 1 -- start start 2024-04-25T01:40:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.673+0000 7f4881e0d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffb5580 0x55e00ffcc220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.673+0000 7f4881e0d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5ea0 0x55e00ffb86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.673+0000 7f4881e0d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffc0570 0x55e00ffbf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.673+0000 7f4881e0d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48781aebe0 con 0x55e00ffb5ea0 2024-04-25T01:40:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.673+0000 7f4881e0d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f48781b0720 con 0x55e00ffc0570 2024-04-25T01:40:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.673+0000 7f4881e0d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f48781af2b0 con 0x55e00ffb5580 2024-04-25T01:40:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.673+0000 7f4877fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5ea0 0x55e00ffb86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.673+0000 7f4877fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5ea0 0x55e00ffb86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47832/0 (socket says 172.21.15.28:47832) 2024-04-25T01:40:56.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.673+0000 7f4877fff640 1 -- 172.21.15.28:0/2259509769 learned_addr learned my addr 172.21.15.28:0/2259509769 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.673+0000 7f487d6a6640 1 --2- 172.21.15.28:0/2259509769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffc0570 0x55e00ffbf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.673+0000 7f487cea5640 1 --2- 172.21.15.28:0/2259509769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffb5580 0x55e00ffcc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.673+0000 7f4877fff640 1 -- 172.21.15.28:0/2259509769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffb5580 msgr2=0x55e00ffcc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.673+0000 7f4877fff640 1 --2- 172.21.15.28:0/2259509769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffb5580 0x55e00ffcc220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.673+0000 7f4877fff640 1 -- 172.21.15.28:0/2259509769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffc0570 msgr2=0x55e00ffbf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.673+0000 7f4877fff640 1 --2- 172.21.15.28:0/2259509769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffc0570 0x55e00ffbf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.673+0000 7f4877fff640 1 -- 172.21.15.28:0/2259509769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e00fcf87d0 con 0x55e00ffb5ea0 2024-04-25T01:40:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.674+0000 7f487cea5640 1 --2- 172.21.15.28:0/2259509769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffb5580 0x55e00ffcc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.674+0000 7f487d6a6640 1 --2- 172.21.15.28:0/2259509769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffc0570 0x55e00ffbf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.674+0000 7f4877fff640 1 --2- 172.21.15.28:0/2259509769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5ea0 0x55e00ffb86e0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f4864009460 tx=0x7f4864005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=396abffdb91b379e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.674+0000 7f48627fc640 1 -- 172.21.15.28:0/2259509769 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4864024070 con 0x55e00ffb5ea0 2024-04-25T01:40:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.674+0000 7f48627fc640 1 -- 172.21.15.28:0/2259509769 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48640042f0 con 0x55e00ffb5ea0 2024-04-25T01:40:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.674+0000 7f48627fc640 1 -- 172.21.15.28:0/2259509769 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486401f510 con 0x55e00ffb5ea0 2024-04-25T01:40:56.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.674+0000 7f4881e0d740 1 -- 172.21.15.28:0/2259509769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5ea0 msgr2=0x55e00ffb86e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.674+0000 7f4881e0d740 1 --2- 172.21.15.28:0/2259509769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5ea0 0x55e00ffb86e0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f4864009460 tx=0x7f4864005f90 comp rx=0 tx=0).stop 2024-04-25T01:40:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.674+0000 7f4881e0d740 1 -- 172.21.15.28:0/2259509769 shutdown_connections 2024-04-25T01:40:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.674+0000 7f4881e0d740 1 --2- 172.21.15.28:0/2259509769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffc0570 0x55e00ffbf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.675+0000 7f4881e0d740 1 --2- 172.21.15.28:0/2259509769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5ea0 0x55e00ffb86e0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.675+0000 7f4881e0d740 1 --2- 172.21.15.28:0/2259509769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffb5580 0x55e00ffcc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.675+0000 7f4881e0d740 1 -- 172.21.15.28:0/2259509769 >> 172.21.15.28:0/2259509769 conn(0x55e00ffb18a0 msgr2=0x55e00ffb7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.675+0000 7f4881e0d740 1 -- 172.21.15.28:0/2259509769 shutdown_connections 2024-04-25T01:40:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.675+0000 7f4881e0d740 1 -- 172.21.15.28:0/2259509769 wait complete. 2024-04-25T01:40:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.675+0000 7f4881e0d740 1 -- start start 2024-04-25T01:40:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.675+0000 7f4881e0d740 1 Processor -- start 2024-04-25T01:40:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.675+0000 7f4881e0d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffb5580 0x55e00ffc8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.675+0000 7f4881e0d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5ea0 0x55e00ffc8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.675+0000 7f4881e0d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffc0570 0x55e00ffc3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.675+0000 7f4881e0d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48781b0050 con 0x55e00ffb5ea0 2024-04-25T01:40:56.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.675+0000 7f4881e0d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f48781af980 con 0x55e00ffc0570 2024-04-25T01:40:56.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.675+0000 7f4881e0d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f48781acbe0 con 0x55e00ffb5580 2024-04-25T01:40:56.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.675+0000 7f4881e0d740 10 client.0 _ll_register_callbacks cb 0x7f486c030c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:56.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.675+0000 7f4877fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5ea0 0x55e00ffc8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.676+0000 7f4877fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5ea0 0x55e00ffc8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47846/0 (socket says 172.21.15.28:47846) 2024-04-25T01:40:56.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.676+0000 7f4877fff640 1 -- 172.21.15.28:0/3739314791 learned_addr learned my addr 172.21.15.28:0/3739314791 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.676+0000 7f487cea5640 1 --2- 172.21.15.28:0/3739314791 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffb5580 0x55e00ffc8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.676+0000 7f487d6a6640 1 --2- 172.21.15.28:0/3739314791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffc0570 0x55e00ffc3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.676+0000 7f4877fff640 1 -- 172.21.15.28:0/3739314791 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffb5580 msgr2=0x55e00ffc8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.676+0000 7f4877fff640 1 --2- 172.21.15.28:0/3739314791 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffb5580 0x55e00ffc8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.676+0000 7f4877fff640 1 -- 172.21.15.28:0/3739314791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffc0570 msgr2=0x55e00ffc3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.676+0000 7f4877fff640 1 --2- 172.21.15.28:0/3739314791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffc0570 0x55e00ffc3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.676+0000 7f4877fff640 1 -- 172.21.15.28:0/3739314791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e00ffc46b0 con 0x55e00ffb5ea0 2024-04-25T01:40:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.676+0000 7f4877fff640 1 --2- 172.21.15.28:0/3739314791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5ea0 0x55e00ffc8ca0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f4864000c00 tx=0x7f4864009ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88bfec743d7c0d58 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.676+0000 7f4857fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.676+0000 7f4857fff640 1 -- 172.21.15.28:0/3739314791 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486402e020 con 0x55e00ffb5ea0 2024-04-25T01:40:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.676+0000 7f4857fff640 1 -- 172.21.15.28:0/3739314791 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f486401f4a0 con 0x55e00ffb5ea0 2024-04-25T01:40:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.676+0000 7f4881e0d740 1 -- 172.21.15.28:0/3739314791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48640042f0 con 0x55e00ffb5ea0 2024-04-25T01:40:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.676+0000 7f4857fff640 1 -- 172.21.15.28:0/3739314791 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4864039050 con 0x55e00ffb5ea0 2024-04-25T01:40:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.676+0000 7f4881e0d740 10 client.18501 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.676+0000 7f4881e0d740 1 -- 172.21.15.28:0/3739314791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e010036760 con 0x55e00ffb5ea0 2024-04-25T01:40:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.677+0000 7f4857fff640 1 -- 172.21.15.28:0/3739314791 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48640042f0 con 0x55e00ffb5ea0 2024-04-25T01:40:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.677+0000 7f4857fff640 1 -- 172.21.15.28:0/3739314791 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f48640354b0 con 0x55e00ffb5ea0 2024-04-25T01:40:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.677+0000 7f4881e0d740 10 client.18501 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.677+0000 7f4881e0d740 10 client.18501 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:40:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.677+0000 7f4881e0d740 20 client.18501 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:56.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.677+0000 7f4881e0d740 1 --2- 172.21.15.28:0/3739314791 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55e010036e00 0x55e010039220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.677+0000 7f4881e0d740 4 client.18501 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:56.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.677+0000 7f4881e0d740 1 -- 172.21.15.28:0/3739314791 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e00ffbda60 con 0x55e010036e00 2024-04-25T01:40:56.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.677+0000 7f487cea5640 1 --2- 172.21.15.28:0/3739314791 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55e010036e00 0x55e010039220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.677+0000 7f487cea5640 1 --2- 172.21.15.28:0/3739314791 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55e010036e00 0x55e010039220 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=39d49a94828f6b0b server_cookie=efc1bc01f13926a5 in_seq=0 out_seq=0 2024-04-25T01:40:56.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.677+0000 7f4857fff640 10 client.18501 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:40:56.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.677+0000 7f4857fff640 1 -- 172.21.15.28:0/3739314791 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e00ffbda60 con 0x55e010036e00 2024-04-25T01:40:56.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.677+0000 7f4857fff640 10 client.18501 handle_command_reply: tid=0 2024-04-25T01:40:56.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.677+0000 7f4881e0d740 1 client.18501 shutdown 2024-04-25T01:40:56.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f4881e0d740 1 -- 172.21.15.28:0/3739314791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5ea0 msgr2=0x55e00ffc8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f4881e0d740 1 --2- 172.21.15.28:0/3739314791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5ea0 0x55e00ffc8ca0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f4864000c00 tx=0x7f4864009ae0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f4881e0d740 1 -- 172.21.15.28:0/3739314791 shutdown_connections 2024-04-25T01:40:56.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f4881e0d740 1 --2- 172.21.15.28:0/3739314791 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55e010036e00 0x55e010039220 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f4881e0d740 1 --2- 172.21.15.28:0/3739314791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55e00ffc0570 0x55e00ffc3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f4881e0d740 1 --2- 172.21.15.28:0/3739314791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e00ffb5ea0 0x55e00ffc8ca0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f4857fff640 0 client.18501 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:40:56.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f4881e0d740 1 --2- 172.21.15.28:0/3739314791 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55e00ffb5580 0x55e00ffc8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f4881e0d740 1 -- 172.21.15.28:0/3739314791 >> 172.21.15.28:0/3739314791 conn(0x55e00ffb18a0 msgr2=0x55e00ffb7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f4881e0d740 1 -- 172.21.15.28:0/3739314791 shutdown_connections 2024-04-25T01:40:56.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f4881e0d740 1 -- 172.21.15.28:0/3739314791 wait complete. 2024-04-25T01:40:56.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f4881e0d740 20 client.18501 trim_cache size 0 max 16384 2024-04-25T01:40:56.670 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:56.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f487f109640 1 -- 172.21.15.28:0/910117271 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f48680a2d80 msgr2=0x7f48680a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f487f109640 1 --2- 172.21.15.28:0/910117271 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f48680a2d80 0x7f48680a5240 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4870022000 tx=0x7f4870020000 comp rx=0 tx=0).stop 2024-04-25T01:40:56.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f487f109640 1 -- 172.21.15.28:0/910117271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48781002e0 msgr2=0x7f48781980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f487f109640 1 --2- 172.21.15.28:0/910117271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48781002e0 0x7f48781980a0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f486c005940 tx=0x7f486c0042f0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f487f109640 1 -- 172.21.15.28:0/910117271 shutdown_connections 2024-04-25T01:40:56.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f487f109640 1 --2- 172.21.15.28:0/910117271 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f48680a2d80 0x7f48680a5240 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f487f109640 1 --2- 172.21.15.28:0/910117271 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f487819c910 0x7f487819cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f487f109640 1 --2- 172.21.15.28:0/910117271 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4878105160 0x7f48781985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.678+0000 7f487f109640 1 --2- 172.21.15.28:0/910117271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48781002e0 0x7f48781980a0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.679+0000 7f487f109640 1 -- 172.21.15.28:0/910117271 >> 172.21.15.28:0/910117271 conn(0x7f48780758d0 msgr2=0x7f48781040c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.679+0000 7f487f109640 1 -- 172.21.15.28:0/910117271 shutdown_connections 2024-04-25T01:40:56.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.679+0000 7f487f109640 1 -- 172.21.15.28:0/910117271 wait complete. 2024-04-25T01:40:56.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:40:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.794+0000 7f14af42a640 1 -- 172.21.15.28:0/1665834966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a8106630 msgr2=0x7f14a8106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.794+0000 7f14af42a640 1 --2- 172.21.15.28:0/1665834966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a8106630 0x7f14a8106a10 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009930 tx=0x7f14a402a420 comp rx=0 tx=0).stop 2024-04-25T01:40:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.794+0000 7f14af42a640 1 -- 172.21.15.28:0/1665834966 shutdown_connections 2024-04-25T01:40:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.794+0000 7f14af42a640 1 --2- 172.21.15.28:0/1665834966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14a8100fe0 0x7f14a810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.794+0000 7f14af42a640 1 --2- 172.21.15.28:0/1665834966 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14a8100620 0x7f14a8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.794+0000 7f14af42a640 1 --2- 172.21.15.28:0/1665834966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a8106630 0x7f14a8106a10 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.794+0000 7f14af42a640 1 -- 172.21.15.28:0/1665834966 >> 172.21.15.28:0/1665834966 conn(0x7f14a80776a0 msgr2=0x7f14a8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.794+0000 7f14af42a640 1 -- 172.21.15.28:0/1665834966 shutdown_connections 2024-04-25T01:40:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.794+0000 7f14af42a640 1 -- 172.21.15.28:0/1665834966 wait complete. 2024-04-25T01:40:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.795+0000 7f14af42a640 1 Processor -- start 2024-04-25T01:40:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.795+0000 7f14af42a640 1 -- start start 2024-04-25T01:40:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.795+0000 7f14af42a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a8100620 0x7f14a8198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.795+0000 7f14af42a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14a8100fe0 0x7f14a81985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.795+0000 7f14af42a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14a8106630 0x7f14a819c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.795+0000 7f14af42a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14a8111b40 con 0x7f14a8100620 2024-04-25T01:40:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.795+0000 7f14af42a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f14a81119c0 con 0x7f14a8100fe0 2024-04-25T01:40:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.795+0000 7f14af42a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f14a8111cc0 con 0x7f14a8106630 2024-04-25T01:40:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.795+0000 7f14ad1c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a8100620 0x7f14a8198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.795+0000 7f14ad1c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a8100620 0x7f14a8198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47858/0 (socket says 172.21.15.28:47858) 2024-04-25T01:40:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.795+0000 7f14ad1c6640 1 -- 172.21.15.28:0/4209014757 learned_addr learned my addr 172.21.15.28:0/4209014757 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.795+0000 7f14ac9c5640 1 --2- 172.21.15.28:0/4209014757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14a8100fe0 0x7f14a81985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.795+0000 7f14ad1c6640 1 -- 172.21.15.28:0/4209014757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14a8106630 msgr2=0x7f14a819c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.795+0000 7f14ad1c6640 1 --2- 172.21.15.28:0/4209014757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14a8106630 0x7f14a819c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.795+0000 7f14ad1c6640 1 -- 172.21.15.28:0/4209014757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14a8100fe0 msgr2=0x7f14a81985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.795+0000 7f14ad1c6640 1 --2- 172.21.15.28:0/4209014757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14a8100fe0 0x7f14a81985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.795+0000 7f14ad1c6640 1 -- 172.21.15.28:0/4209014757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14a819d070 con 0x7f14a8100620 2024-04-25T01:40:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.795+0000 7f14ad1c6640 1 --2- 172.21.15.28:0/4209014757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a8100620 0x7f14a8198080 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009a50 tx=0x7f14a4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.795+0000 7f149e7fc640 1 -- 172.21.15.28:0/4209014757 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a403f030 con 0x7f14a8100620 2024-04-25T01:40:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.796+0000 7f149e7fc640 1 -- 172.21.15.28:0/4209014757 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14a4003ce0 con 0x7f14a8100620 2024-04-25T01:40:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.796+0000 7f14af42a640 1 -- 172.21.15.28:0/4209014757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14a819d300 con 0x7f14a8100620 2024-04-25T01:40:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.796+0000 7f149e7fc640 1 -- 172.21.15.28:0/4209014757 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a4035070 con 0x7f14a8100620 2024-04-25T01:40:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.796+0000 7f14af42a640 1 -- 172.21.15.28:0/4209014757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14a81a4bf0 con 0x7f14a8100620 2024-04-25T01:40:56.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.798+0000 7f149e7fc640 1 -- 172.21.15.28:0/4209014757 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f14a4042030 con 0x7f14a8100620 2024-04-25T01:40:56.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.798+0000 7f14b2212740 1 Processor -- start 2024-04-25T01:40:56.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.798+0000 7f14b2212740 1 -- start start 2024-04-25T01:40:56.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.798+0000 7f149e7fc640 1 --2- 172.21.15.28:0/4209014757 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f14940a2d00 0x7f14940a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.798+0000 7f14b2212740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d0b480 0x55b5e1d0b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.798+0000 7f14b2212740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d1a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.798+0000 7f14b2212740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d1a980 0x55b5e1d1cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.798+0000 7f14b2212740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b5e1c70670 con 0x55b5e1d0bda0 2024-04-25T01:40:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.798+0000 7f14b2212740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b5e1c18970 con 0x55b5e1d1a980 2024-04-25T01:40:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.798+0000 7f14b2212740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b5e1c6c630 con 0x55b5e1d0b480 2024-04-25T01:40:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.799+0000 7f14ad9c7640 1 --2- 172.21.15.28:0/4209014757 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f14940a2d00 0x7f14940a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.799+0000 7f149e7fc640 1 -- 172.21.15.28:0/4209014757 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f14a40efe10 con 0x7f14a8100620 2024-04-25T01:40:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.799+0000 7f14ac9c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d1a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.799+0000 7f14ac9c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d1a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47872/0 (socket says 172.21.15.28:47872) 2024-04-25T01:40:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.799+0000 7f14ad1c6640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d0b480 0x55b5e1d0b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.799+0000 7f14ac9c5640 1 -- 172.21.15.28:0/1371975117 learned_addr learned my addr 172.21.15.28:0/1371975117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.799+0000 7f14ad9c7640 1 --2- 172.21.15.28:0/1371975117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d1a980 0x55b5e1d1cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.799+0000 7f14ac9c5640 1 -- 172.21.15.28:0/1371975117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d0b480 msgr2=0x55b5e1d0b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.799+0000 7f14ac9c5640 1 --2- 172.21.15.28:0/1371975117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d0b480 0x55b5e1d0b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.799+0000 7f14ac9c5640 1 -- 172.21.15.28:0/1371975117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d1a980 msgr2=0x55b5e1d1cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.799+0000 7f14ac9c5640 1 --2- 172.21.15.28:0/1371975117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d1a980 0x55b5e1d1cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.799+0000 7f14ac9c5640 1 -- 172.21.15.28:0/1371975117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5e1c3c400 con 0x55b5e1d0bda0 2024-04-25T01:40:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.799+0000 7f14ad1c6640 1 --2- 172.21.15.28:0/1371975117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d0b480 0x55b5e1d0b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:40:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.799+0000 7f14ad9c7640 1 --2- 172.21.15.28:0/1371975117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d1a980 0x55b5e1d1cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.799+0000 7f14ad9c7640 1 --2- 172.21.15.28:0/4209014757 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f14940a2d00 0x7f14940a51c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f14a0005fd0 tx=0x7f14a0013820 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.799+0000 7f14ac9c5640 1 --2- 172.21.15.28:0/1371975117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d1a440 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f149800b7f0 tx=0x7f149800bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953318e246837d8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.799+0000 7f14937fe640 1 -- 172.21.15.28:0/1371975117 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f149800d950 con 0x55b5e1d0bda0 2024-04-25T01:40:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.799+0000 7f14937fe640 1 -- 172.21.15.28:0/1371975117 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1498018070 con 0x55b5e1d0bda0 2024-04-25T01:40:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.799+0000 7f14937fe640 1 -- 172.21.15.28:0/1371975117 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14980123e0 con 0x55b5e1d0bda0 2024-04-25T01:40:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.800+0000 7f14b2212740 1 -- 172.21.15.28:0/1371975117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d0bda0 msgr2=0x55b5e1d1a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.800+0000 7f14b2212740 1 --2- 172.21.15.28:0/1371975117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d1a440 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f149800b7f0 tx=0x7f149800bcb0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.800+0000 7f14b2212740 1 -- 172.21.15.28:0/1371975117 shutdown_connections 2024-04-25T01:40:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.800+0000 7f14b2212740 1 --2- 172.21.15.28:0/1371975117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d1a980 0x55b5e1d1cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.800+0000 7f14b2212740 1 --2- 172.21.15.28:0/1371975117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d1a440 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.800+0000 7f14b2212740 1 --2- 172.21.15.28:0/1371975117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d0b480 0x55b5e1d0b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.800+0000 7f14b2212740 1 -- 172.21.15.28:0/1371975117 >> 172.21.15.28:0/1371975117 conn(0x55b5e1d078a0 msgr2=0x55b5e1d07c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.800+0000 7f14b2212740 1 -- 172.21.15.28:0/1371975117 shutdown_connections 2024-04-25T01:40:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.800+0000 7f14b2212740 1 -- 172.21.15.28:0/1371975117 wait complete. 2024-04-25T01:40:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.800+0000 7f14b2212740 1 -- start start 2024-04-25T01:40:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.801+0000 7f14b2212740 1 Processor -- start 2024-04-25T01:40:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.801+0000 7f14b2212740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d0b480 0x55b5e1d1a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.801+0000 7f14b2212740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d17590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.801+0000 7f14b2212740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d1a980 0x55b5e1d17ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.801+0000 7f14b2212740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b5e1d1d740 con 0x55b5e1d1a980 2024-04-25T01:40:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.801+0000 7f14b2212740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b5e1c2c050 con 0x55b5e1d0b480 2024-04-25T01:40:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.801+0000 7f14b2212740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b5e1d1d8c0 con 0x55b5e1d0bda0 2024-04-25T01:40:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.801+0000 7f14b2212740 10 client.0 _ll_register_callbacks cb 0x55b5e1983be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.801+0000 7f14ad9c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d1a980 0x55b5e1d17ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.801+0000 7f14ad9c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d1a980 0x55b5e1d17ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47886/0 (socket says 172.21.15.28:47886) 2024-04-25T01:40:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.801+0000 7f14ad9c7640 1 -- 172.21.15.28:0/75411379 learned_addr learned my addr 172.21.15.28:0/75411379 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.801+0000 7f14ac9c5640 1 --2- 172.21.15.28:0/75411379 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d17590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.801+0000 7f14ad1c6640 1 --2- 172.21.15.28:0/75411379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d0b480 0x55b5e1d1a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.801+0000 7f14ad9c7640 1 -- 172.21.15.28:0/75411379 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d0bda0 msgr2=0x55b5e1d17590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.801+0000 7f14ad9c7640 1 --2- 172.21.15.28:0/75411379 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d17590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.801+0000 7f14ad9c7640 1 -- 172.21.15.28:0/75411379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d0b480 msgr2=0x55b5e1d1a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.801+0000 7f14ad9c7640 1 --2- 172.21.15.28:0/75411379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d0b480 0x55b5e1d1a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.801+0000 7f14ad9c7640 1 -- 172.21.15.28:0/75411379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5e1bfddc0 con 0x55b5e1d1a980 2024-04-25T01:40:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.801+0000 7f14ad1c6640 1 --2- 172.21.15.28:0/75411379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d0b480 0x55b5e1d1a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.802+0000 7f14ad9c7640 1 --2- 172.21.15.28:0/75411379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d1a980 0x55b5e1d17ad0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f14a0010d80 tx=0x7f14a0005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4afa031f9c1a12a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.802+0000 7f1490ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.802+0000 7f1490ff9640 1 -- 172.21.15.28:0/75411379 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a0003b70 con 0x55b5e1d1a980 2024-04-25T01:40:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.802+0000 7f14b2212740 1 -- 172.21.15.28:0/75411379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1498018070 con 0x55b5e1d1a980 2024-04-25T01:40:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.802+0000 7f1490ff9640 1 -- 172.21.15.28:0/75411379 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f14a0003d10 con 0x55b5e1d1a980 2024-04-25T01:40:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.802+0000 7f1490ff9640 1 -- 172.21.15.28:0/75411379 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f14a002f050 con 0x55b5e1d1a980 2024-04-25T01:40:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.802+0000 7f14b2212740 10 client.18513 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.802+0000 7f14b2212740 1 -- 172.21.15.28:0/75411379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b5e19843f0 con 0x55b5e1d1a980 2024-04-25T01:40:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.802+0000 7f1490ff9640 1 -- 172.21.15.28:0/75411379 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1498018070 con 0x55b5e1d1a980 2024-04-25T01:40:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.802+0000 7f1490ff9640 1 -- 172.21.15.28:0/75411379 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f14a001d040 con 0x55b5e1d1a980 2024-04-25T01:40:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.802+0000 7f14b2212740 10 client.18513 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.802+0000 7f14b2212740 10 client.18513 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.802+0000 7f14b2212740 20 client.18513 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.802+0000 7f14b2212740 1 --2- 172.21.15.28:0/75411379 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b5e1d2add0 0x55b5e1d2d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.802+0000 7f14b2212740 4 client.18513 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.802+0000 7f14b2212740 1 -- 172.21.15.28:0/75411379 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5e1c7dbb0 con 0x55b5e1d2add0 2024-04-25T01:40:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.803+0000 7f14ad1c6640 1 --2- 172.21.15.28:0/75411379 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b5e1d2add0 0x55b5e1d2d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.803+0000 7f1490ff9640 10 client.18513 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.803+0000 7f14ad1c6640 1 --2- 172.21.15.28:0/75411379 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b5e1d2add0 0x55b5e1d2d250 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ca9d6ae51f511a6 server_cookie=e6ec639db754b39 in_seq=0 out_seq=0 2024-04-25T01:40:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.804+0000 7f1490ff9640 1 -- 172.21.15.28:0/75411379 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b5e1c7dbb0 con 0x55b5e1d2add0 2024-04-25T01:40:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.804+0000 7f1490ff9640 10 client.18513 handle_command_reply: tid=0 2024-04-25T01:40:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.804+0000 7f14b2212740 1 client.18513 shutdown 2024-04-25T01:40:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.804+0000 7f14b2212740 1 -- 172.21.15.28:0/75411379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d1a980 msgr2=0x55b5e1d17ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.804+0000 7f14b2212740 1 --2- 172.21.15.28:0/75411379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d1a980 0x55b5e1d17ad0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f14a0010d80 tx=0x7f14a0005d90 comp rx=0 tx=0).stop 2024-04-25T01:40:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.804+0000 7f14b2212740 1 -- 172.21.15.28:0/75411379 shutdown_connections 2024-04-25T01:40:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.804+0000 7f14b2212740 1 --2- 172.21.15.28:0/75411379 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b5e1d2add0 0x55b5e1d2d250 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.804+0000 7f14b2212740 1 --2- 172.21.15.28:0/75411379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d1a980 0x55b5e1d17ad0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.804+0000 7f14b2212740 1 --2- 172.21.15.28:0/75411379 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d17590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.804+0000 7f14b2212740 1 --2- 172.21.15.28:0/75411379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d0b480 0x55b5e1d1a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.804+0000 7f1490ff9640 0 client.18513 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.804+0000 7f14b2212740 1 -- 172.21.15.28:0/75411379 >> 172.21.15.28:0/75411379 conn(0x55b5e1d078a0 msgr2=0x55b5e1d06b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.804+0000 7f14b2212740 1 -- 172.21.15.28:0/75411379 shutdown_connections 2024-04-25T01:40:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.804+0000 7f14b2212740 1 -- 172.21.15.28:0/75411379 wait complete. 2024-04-25T01:40:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.804+0000 7f14b2212740 20 client.18513 trim_cache size 0 max 16384 2024-04-25T01:40:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.818+0000 7f14b2212740 1 Processor -- start 2024-04-25T01:40:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.818+0000 7f14b2212740 1 -- start start 2024-04-25T01:40:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.818+0000 7f14b2212740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d0b480 0x55b5e1d26670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.818+0000 7f14b2212740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d07c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.818+0000 7f14b2212740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d1a980 0x55b5e1d19dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.818+0000 7f14b2212740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14a81ac980 con 0x55b5e1d0bda0 2024-04-25T01:40:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.818+0000 7f14b2212740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f14a81ae4c0 con 0x55b5e1d0b480 2024-04-25T01:40:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.818+0000 7f14b2212740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f14a81ad050 con 0x55b5e1d1a980 2024-04-25T01:40:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.818+0000 7f14ac9c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d07c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.818+0000 7f14ac9c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d07c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47890/0 (socket says 172.21.15.28:47890) 2024-04-25T01:40:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.818+0000 7f14ac9c5640 1 -- 172.21.15.28:0/1606301143 learned_addr learned my addr 172.21.15.28:0/1606301143 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.818+0000 7f14ad1c6640 1 --2- 172.21.15.28:0/1606301143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d0b480 0x55b5e1d26670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.818+0000 7f14ac9c5640 1 -- 172.21.15.28:0/1606301143 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d1a980 msgr2=0x55b5e1d19dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.818+0000 7f14ac9c5640 1 --2- 172.21.15.28:0/1606301143 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d1a980 0x55b5e1d19dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.818+0000 7f14ac9c5640 1 -- 172.21.15.28:0/1606301143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d0b480 msgr2=0x55b5e1d26670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.818+0000 7f14ac9c5640 1 --2- 172.21.15.28:0/1606301143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d0b480 0x55b5e1d26670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.818+0000 7f14ac9c5640 1 -- 172.21.15.28:0/1606301143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5e1d16540 con 0x55b5e1d0bda0 2024-04-25T01:40:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.818+0000 7f14ad1c6640 1 --2- 172.21.15.28:0/1606301143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d0b480 0x55b5e1d26670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.818+0000 7f14ac9c5640 1 --2- 172.21.15.28:0/1606301143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d07c80 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f149800b390 tx=0x7f149800b300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501b3ebc18fbaf1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.819+0000 7f1492ffd640 1 -- 172.21.15.28:0/1606301143 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1498022070 con 0x55b5e1d0bda0 2024-04-25T01:40:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.819+0000 7f1492ffd640 1 -- 172.21.15.28:0/1606301143 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1498018040 con 0x55b5e1d0bda0 2024-04-25T01:40:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.819+0000 7f1492ffd640 1 -- 172.21.15.28:0/1606301143 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f149801c950 con 0x55b5e1d0bda0 2024-04-25T01:40:56.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.819+0000 7f14b2212740 1 -- 172.21.15.28:0/1606301143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d0bda0 msgr2=0x55b5e1d07c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.819+0000 7f14b2212740 1 --2- 172.21.15.28:0/1606301143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d07c80 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f149800b390 tx=0x7f149800b300 comp rx=0 tx=0).stop 2024-04-25T01:40:56.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.819+0000 7f14b2212740 1 -- 172.21.15.28:0/1606301143 shutdown_connections 2024-04-25T01:40:56.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.819+0000 7f14b2212740 1 --2- 172.21.15.28:0/1606301143 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d1a980 0x55b5e1d19dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.819+0000 7f14b2212740 1 --2- 172.21.15.28:0/1606301143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d07c80 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.819+0000 7f14b2212740 1 --2- 172.21.15.28:0/1606301143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d0b480 0x55b5e1d26670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.819+0000 7f14b2212740 1 -- 172.21.15.28:0/1606301143 >> 172.21.15.28:0/1606301143 conn(0x55b5e1d078a0 msgr2=0x55b5e1d12c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.819+0000 7f14b2212740 1 -- 172.21.15.28:0/1606301143 shutdown_connections 2024-04-25T01:40:56.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.819+0000 7f14b2212740 1 -- 172.21.15.28:0/1606301143 wait complete. 2024-04-25T01:40:56.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.819+0000 7f14b2212740 1 -- start start 2024-04-25T01:40:56.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.820+0000 7f14b2212740 1 Processor -- start 2024-04-25T01:40:56.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.820+0000 7f14b2212740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d0b480 0x55b5e1d1e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.820+0000 7f14b2212740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d1ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.820+0000 7f14b2212740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d1a980 0x55b5e1d1efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.820+0000 7f14b2212740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14a81addf0 con 0x55b5e1d1a980 2024-04-25T01:40:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.820+0000 7f14b2212740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f14a81ad720 con 0x55b5e1d0bda0 2024-04-25T01:40:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.820+0000 7f14b2212740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f14a81aa980 con 0x55b5e1d0b480 2024-04-25T01:40:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.820+0000 7f14b2212740 10 client.0 _ll_register_callbacks cb 0x7f14a0007d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.820+0000 7f14ad9c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d1a980 0x55b5e1d1efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.820+0000 7f14ad9c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d1a980 0x55b5e1d1efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47892/0 (socket says 172.21.15.28:47892) 2024-04-25T01:40:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.820+0000 7f14ad9c7640 1 -- 172.21.15.28:0/2405805821 learned_addr learned my addr 172.21.15.28:0/2405805821 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.820+0000 7f14ad1c6640 1 --2- 172.21.15.28:0/2405805821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d0b480 0x55b5e1d1e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.820+0000 7f14ac9c5640 1 --2- 172.21.15.28:0/2405805821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d1ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.820+0000 7f14ad9c7640 1 -- 172.21.15.28:0/2405805821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d0b480 msgr2=0x55b5e1d1e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.820+0000 7f14ad9c7640 1 --2- 172.21.15.28:0/2405805821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d0b480 0x55b5e1d1e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.820+0000 7f14ad9c7640 1 -- 172.21.15.28:0/2405805821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d0bda0 msgr2=0x55b5e1d1ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.820+0000 7f14ad9c7640 1 --2- 172.21.15.28:0/2405805821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d1ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.820+0000 7f14ad9c7640 1 -- 172.21.15.28:0/2405805821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5e1d0bbd0 con 0x55b5e1d1a980 2024-04-25T01:40:56.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.821+0000 7f14ad9c7640 1 --2- 172.21.15.28:0/2405805821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d1a980 0x55b5e1d1efc0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f14a001e730 tx=0x7f14a001e610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d041880bbbae0c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:56.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.821+0000 7f1490ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:56.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.821+0000 7f1490ff9640 1 -- 172.21.15.28:0/2405805821 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a0011c40 con 0x55b5e1d1a980 2024-04-25T01:40:56.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.821+0000 7f1490ff9640 1 -- 172.21.15.28:0/2405805821 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f14a001e9a0 con 0x55b5e1d1a980 2024-04-25T01:40:56.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.821+0000 7f14b2212740 1 -- 172.21.15.28:0/2405805821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1498018040 con 0x55b5e1d1a980 2024-04-25T01:40:56.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.821+0000 7f1490ff9640 1 -- 172.21.15.28:0/2405805821 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f14a001eba0 con 0x55b5e1d1a980 2024-04-25T01:40:56.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.821+0000 7f14b2212740 10 client.18519 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:56.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.821+0000 7f14b2212740 1 -- 172.21.15.28:0/2405805821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b5e1a4e7d0 con 0x55b5e1d1a980 2024-04-25T01:40:56.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.821+0000 7f1490ff9640 1 -- 172.21.15.28:0/2405805821 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1498018040 con 0x55b5e1d1a980 2024-04-25T01:40:56.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.821+0000 7f1490ff9640 1 -- 172.21.15.28:0/2405805821 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f14a001d070 con 0x55b5e1d1a980 2024-04-25T01:40:56.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.821+0000 7f14b2212740 10 client.18519 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:56.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.821+0000 7f14b2212740 10 client.18519 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:56.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.821+0000 7f14b2212740 20 client.18519 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:56.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.821+0000 7f14b2212740 1 --2- 172.21.15.28:0/2405805821 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b5e1d91980 0x55b5e1d93da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:56.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.821+0000 7f14b2212740 4 client.18519 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:56.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.821+0000 7f14b2212740 1 -- 172.21.15.28:0/2405805821 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b5e1c77f00 con 0x55b5e1d91980 2024-04-25T01:40:56.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.822+0000 7f14ad1c6640 1 --2- 172.21.15.28:0/2405805821 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b5e1d91980 0x55b5e1d93da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:56.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.822+0000 7f14ad1c6640 1 --2- 172.21.15.28:0/2405805821 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b5e1d91980 0x55b5e1d93da0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1736ce97c1c4eeb server_cookie=57cea01c9daa3e67 in_seq=0 out_seq=0 2024-04-25T01:40:56.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.822+0000 7f1490ff9640 10 client.18519 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:56.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.822+0000 7f1490ff9640 1 -- 172.21.15.28:0/2405805821 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b5e1c77f00 con 0x55b5e1d91980 2024-04-25T01:40:56.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.822+0000 7f1490ff9640 10 client.18519 handle_command_reply: tid=0 2024-04-25T01:40:56.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.822+0000 7f14b2212740 1 client.18519 shutdown 2024-04-25T01:40:56.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.822+0000 7f14b2212740 1 -- 172.21.15.28:0/2405805821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d1a980 msgr2=0x55b5e1d1efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.822+0000 7f14b2212740 1 --2- 172.21.15.28:0/2405805821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d1a980 0x55b5e1d1efc0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f14a001e730 tx=0x7f14a001e610 comp rx=0 tx=0).stop 2024-04-25T01:40:56.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.822+0000 7f14b2212740 1 -- 172.21.15.28:0/2405805821 shutdown_connections 2024-04-25T01:40:56.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.822+0000 7f14b2212740 1 --2- 172.21.15.28:0/2405805821 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b5e1d91980 0x55b5e1d93da0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.822+0000 7f14b2212740 1 --2- 172.21.15.28:0/2405805821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e1d1a980 0x55b5e1d1efc0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.822+0000 7f14b2212740 1 --2- 172.21.15.28:0/2405805821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b5e1d0bda0 0x55b5e1d1ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.822+0000 7f14b2212740 1 --2- 172.21.15.28:0/2405805821 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b5e1d0b480 0x55b5e1d1e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.822+0000 7f1490ff9640 0 client.18519 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:56.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.822+0000 7f14b2212740 1 -- 172.21.15.28:0/2405805821 >> 172.21.15.28:0/2405805821 conn(0x55b5e1d078a0 msgr2=0x55b5e1d12c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.823+0000 7f14b2212740 1 -- 172.21.15.28:0/2405805821 shutdown_connections 2024-04-25T01:40:56.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.823+0000 7f14b2212740 1 -- 172.21.15.28:0/2405805821 wait complete. 2024-04-25T01:40:56.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.823+0000 7f14b2212740 20 client.18519 trim_cache size 0 max 16384 2024-04-25T01:40:56.814 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:56.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.823+0000 7f14af42a640 1 -- 172.21.15.28:0/4209014757 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f14940a2d00 msgr2=0x7f14940a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.823+0000 7f14af42a640 1 --2- 172.21.15.28:0/4209014757 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f14940a2d00 0x7f14940a51c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f14a0005fd0 tx=0x7f14a0013820 comp rx=0 tx=0).stop 2024-04-25T01:40:56.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.823+0000 7f14af42a640 1 -- 172.21.15.28:0/4209014757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a8100620 msgr2=0x7f14a8198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:56.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.823+0000 7f14af42a640 1 --2- 172.21.15.28:0/4209014757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a8100620 0x7f14a8198080 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009a50 tx=0x7f14a4004270 comp rx=0 tx=0).stop 2024-04-25T01:40:56.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.823+0000 7f14af42a640 1 -- 172.21.15.28:0/4209014757 shutdown_connections 2024-04-25T01:40:56.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.823+0000 7f14af42a640 1 --2- 172.21.15.28:0/4209014757 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f14940a2d00 0x7f14940a51c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.823+0000 7f14af42a640 1 --2- 172.21.15.28:0/4209014757 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f14a8106630 0x7f14a819c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.823+0000 7f14af42a640 1 --2- 172.21.15.28:0/4209014757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14a8100fe0 0x7f14a81985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.823+0000 7f14af42a640 1 --2- 172.21.15.28:0/4209014757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a8100620 0x7f14a8198080 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:56.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.823+0000 7f14af42a640 1 -- 172.21.15.28:0/4209014757 >> 172.21.15.28:0/4209014757 conn(0x7f14a80776a0 msgr2=0x7f14a81031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:56.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.823+0000 7f14af42a640 1 -- 172.21.15.28:0/4209014757 shutdown_connections 2024-04-25T01:40:56.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:56.823+0000 7f14af42a640 1 -- 172.21.15.28:0/4209014757 wait complete. 2024-04-25T01:40:57.069 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:57 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2304088673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:40:57.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:57 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2304088673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:40:57.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:57 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2304088673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:40:57.820 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:40:57.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:40:57.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:40:57.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.953+0000 7f8d3c3a5640 1 -- 172.21.15.28:0/3724248173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d34107720 msgr2=0x7f8d34107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.953+0000 7f8d3c3a5640 1 --2- 172.21.15.28:0/3724248173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d34107720 0x7f8d34107b00 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c009cd0 tx=0x7f8d2c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.953+0000 7f8d3c3a5640 1 -- 172.21.15.28:0/3724248173 shutdown_connections 2024-04-25T01:40:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.953+0000 7f8d3c3a5640 1 --2- 172.21.15.28:0/3724248173 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d341020d0 0x7f8d3410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.953+0000 7f8d3c3a5640 1 --2- 172.21.15.28:0/3724248173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d34101710 0x7f8d34101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.953+0000 7f8d3c3a5640 1 --2- 172.21.15.28:0/3724248173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d34107720 0x7f8d34107b00 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.953+0000 7f8d3c3a5640 1 -- 172.21.15.28:0/3724248173 >> 172.21.15.28:0/3724248173 conn(0x7f8d340fd180 msgr2=0x7f8d340ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.953+0000 7f8d3c3a5640 1 -- 172.21.15.28:0/3724248173 shutdown_connections 2024-04-25T01:40:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.953+0000 7f8d3c3a5640 1 -- 172.21.15.28:0/3724248173 wait complete. 2024-04-25T01:40:57.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.954+0000 7f8d3c3a5640 1 Processor -- start 2024-04-25T01:40:57.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.954+0000 7f8d3c3a5640 1 -- start start 2024-04-25T01:40:57.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.954+0000 7f8d3c3a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d34101710 0x7f8d34198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:57.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.954+0000 7f8d3c3a5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d341020d0 0x7f8d34198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:57.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.954+0000 7f8d3c3a5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d34107720 0x7f8d3419c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:57.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.954+0000 7f8d3c3a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d34111b40 con 0x7f8d34101710 2024-04-25T01:40:57.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.954+0000 7f8d3c3a5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8d341119c0 con 0x7f8d34107720 2024-04-25T01:40:57.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.954+0000 7f8d3c3a5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8d34111cc0 con 0x7f8d341020d0 2024-04-25T01:40:57.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.954+0000 7f8d3a141640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d34101710 0x7f8d34198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:57.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.954+0000 7f8d3a141640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d34101710 0x7f8d34198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47904/0 (socket says 172.21.15.28:47904) 2024-04-25T01:40:57.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.954+0000 7f8d3a141640 1 -- 172.21.15.28:0/3663797926 learned_addr learned my addr 172.21.15.28:0/3663797926 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:57.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.954+0000 7f8d3a942640 1 --2- 172.21.15.28:0/3663797926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d34107720 0x7f8d3419c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:57.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.954+0000 7f8d39940640 1 --2- 172.21.15.28:0/3663797926 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d341020d0 0x7f8d34198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:57.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.955+0000 7f8d3a141640 1 -- 172.21.15.28:0/3663797926 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d341020d0 msgr2=0x7f8d34198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:57.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.955+0000 7f8d3a141640 1 --2- 172.21.15.28:0/3663797926 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d341020d0 0x7f8d34198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.955+0000 7f8d3a141640 1 -- 172.21.15.28:0/3663797926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d34107720 msgr2=0x7f8d3419c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:57.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.955+0000 7f8d3a141640 1 --2- 172.21.15.28:0/3663797926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d34107720 0x7f8d3419c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.955+0000 7f8d3a141640 1 -- 172.21.15.28:0/3663797926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d3419d030 con 0x7f8d34101710 2024-04-25T01:40:57.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.955+0000 7f8d3a141640 1 --2- 172.21.15.28:0/3663797926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d34101710 0x7f8d34198040 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c033040 tx=0x7f8d2c0039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:57.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.955+0000 7f8d2b7fe640 1 -- 172.21.15.28:0/3663797926 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c004040 con 0x7f8d34101710 2024-04-25T01:40:57.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.955+0000 7f8d3c3a5640 1 -- 172.21.15.28:0/3663797926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d3419d2c0 con 0x7f8d34101710 2024-04-25T01:40:57.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.955+0000 7f8d2b7fe640 1 -- 172.21.15.28:0/3663797926 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d2c0046c0 con 0x7f8d34101710 2024-04-25T01:40:57.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.955+0000 7f8d2b7fe640 1 -- 172.21.15.28:0/3663797926 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c046480 con 0x7f8d34101710 2024-04-25T01:40:57.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.955+0000 7f8d3c3a5640 1 -- 172.21.15.28:0/3663797926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d341a4bb0 con 0x7f8d34101710 2024-04-25T01:40:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.957+0000 7f8d2b7fe640 1 -- 172.21.15.28:0/3663797926 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8d2c030710 con 0x7f8d34101710 2024-04-25T01:40:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.958+0000 7f8d3f18c740 1 Processor -- start 2024-04-25T01:40:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.958+0000 7f8d3f18c740 1 -- start start 2024-04-25T01:40:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.958+0000 7f8d3f18c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bce980 0x555ef6bcccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.958+0000 7f8d3f18c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bcd290 0x555ef6bcd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.958+0000 7f8d3f18c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bd8010 0x555ef6bda400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.958+0000 7f8d3f18c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555ef6b312c0 con 0x555ef6bd8010 2024-04-25T01:40:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.958+0000 7f8d3f18c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555ef6ad9910 con 0x555ef6bce980 2024-04-25T01:40:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.958+0000 7f8d3f18c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x555ef6b2d280 con 0x555ef6bcd290 2024-04-25T01:40:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.958+0000 7f8d2b7fe640 1 --2- 172.21.15.28:0/3663797926 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8d200a2d00 0x7f8d200a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.959+0000 7f8d39940640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bd8010 0x555ef6bda400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.959+0000 7f8d3a141640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bcd290 0x555ef6bcd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:57.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.959+0000 7f8d3a942640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bce980 0x555ef6bcccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.959+0000 7f8d3a942640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bce980 0x555ef6bcccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:44648/0 (socket says 172.21.15.28:44648) 2024-04-25T01:40:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.959+0000 7f8d39940640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bd8010 0x555ef6bda400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47910/0 (socket says 172.21.15.28:47910) 2024-04-25T01:40:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.959+0000 7f8d3a942640 1 -- 172.21.15.28:0/1471027605 learned_addr learned my addr 172.21.15.28:0/1471027605 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.959+0000 7f8d3a942640 1 --2- 172.21.15.28:0/3663797926 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8d200a2d00 0x7f8d200a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.959+0000 7f8d2b7fe640 1 -- 172.21.15.28:0/3663797926 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f8d2c02e030 con 0x7f8d34101710 2024-04-25T01:40:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.959+0000 7f8d3a942640 1 -- 172.21.15.28:0/1471027605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bcd290 msgr2=0x555ef6bcd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.959+0000 7f8d3a942640 1 --2- 172.21.15.28:0/1471027605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bcd290 0x555ef6bcd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.959+0000 7f8d3a942640 1 -- 172.21.15.28:0/1471027605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bd8010 msgr2=0x555ef6bda400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.959+0000 7f8d3a942640 1 --2- 172.21.15.28:0/1471027605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bd8010 0x555ef6bda400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.959+0000 7f8d3a942640 1 -- 172.21.15.28:0/1471027605 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ef6aeafb0 con 0x555ef6bce980 2024-04-25T01:40:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.959+0000 7f8d39940640 1 --2- 172.21.15.28:0/1471027605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bd8010 0x555ef6bda400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:40:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.959+0000 7f8d3a942640 1 --2- 172.21.15.28:0/3663797926 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8d200a2d00 0x7f8d200a51c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8d30010740 tx=0x7f8d30017000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.959+0000 7f8d3a942640 1 --2- 172.21.15.28:0/1471027605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bce980 0x555ef6bcccc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8d300129f0 tx=0x7f8d30012eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=632a7d79687eddf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.959+0000 7f8d28bfe640 1 -- 172.21.15.28:0/1471027605 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d30024070 con 0x555ef6bce980 2024-04-25T01:40:57.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.960+0000 7f8d28bfe640 1 -- 172.21.15.28:0/1471027605 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d3000b9c0 con 0x555ef6bce980 2024-04-25T01:40:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.960+0000 7f8d28bfe640 1 -- 172.21.15.28:0/1471027605 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d30004c80 con 0x555ef6bce980 2024-04-25T01:40:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.960+0000 7f8d3f18c740 1 -- 172.21.15.28:0/1471027605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bce980 msgr2=0x555ef6bcccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.960+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/1471027605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bce980 0x555ef6bcccc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8d300129f0 tx=0x7f8d30012eb0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.960+0000 7f8d3f18c740 1 -- 172.21.15.28:0/1471027605 shutdown_connections 2024-04-25T01:40:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.960+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/1471027605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bd8010 0x555ef6bda400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.960+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/1471027605 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bcd290 0x555ef6bcd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.960+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/1471027605 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bce980 0x555ef6bcccc0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.960+0000 7f8d3f18c740 1 -- 172.21.15.28:0/1471027605 >> 172.21.15.28:0/1471027605 conn(0x555ef6b95950 msgr2=0x555ef6bc97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.960+0000 7f8d3f18c740 1 -- 172.21.15.28:0/1471027605 shutdown_connections 2024-04-25T01:40:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.960+0000 7f8d3f18c740 1 -- 172.21.15.28:0/1471027605 wait complete. 2024-04-25T01:40:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.961+0000 7f8d3f18c740 1 -- start start 2024-04-25T01:40:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.961+0000 7f8d3f18c740 1 Processor -- start 2024-04-25T01:40:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.961+0000 7f8d3f18c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bcd290 0x555ef6bd4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.961+0000 7f8d3f18c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bce980 0x555ef6bd5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.961+0000 7f8d3f18c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bd8010 0x555ef6bd59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.961+0000 7f8d3f18c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555ef6bdae40 con 0x555ef6bcd290 2024-04-25T01:40:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.961+0000 7f8d3f18c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555ef6aece90 con 0x555ef6bd8010 2024-04-25T01:40:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.961+0000 7f8d3f18c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x555ef6bdafc0 con 0x555ef6bce980 2024-04-25T01:40:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.961+0000 7f8d3f18c740 10 client.0 _ll_register_callbacks cb 0x555ef6b28970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.961+0000 7f8d3a141640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bcd290 0x555ef6bd4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:57.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.961+0000 7f8d3a141640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bcd290 0x555ef6bd4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47916/0 (socket says 172.21.15.28:47916) 2024-04-25T01:40:57.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.961+0000 7f8d3a141640 1 -- 172.21.15.28:0/358623552 learned_addr learned my addr 172.21.15.28:0/358623552 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:57.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.961+0000 7f8d3a942640 1 --2- 172.21.15.28:0/358623552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bd8010 0x555ef6bd59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:57.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.961+0000 7f8d39940640 1 --2- 172.21.15.28:0/358623552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bce980 0x555ef6bd5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:57.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.962+0000 7f8d3a141640 1 -- 172.21.15.28:0/358623552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bce980 msgr2=0x555ef6bd5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:57.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.962+0000 7f8d3a141640 1 --2- 172.21.15.28:0/358623552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bce980 0x555ef6bd5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.962+0000 7f8d3a141640 1 -- 172.21.15.28:0/358623552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bd8010 msgr2=0x555ef6bd59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:57.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.962+0000 7f8d3a141640 1 --2- 172.21.15.28:0/358623552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bd8010 0x555ef6bd59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.962+0000 7f8d3a141640 1 -- 172.21.15.28:0/358623552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ef690d7e0 con 0x555ef6bcd290 2024-04-25T01:40:57.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.962+0000 7f8d39940640 1 --2- 172.21.15.28:0/358623552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bce980 0x555ef6bd5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:57.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.962+0000 7f8d3a141640 1 --2- 172.21.15.28:0/358623552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bcd290 0x555ef6bd4f30 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c000f80 tx=0x7f8d2c08cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0b4324fe9f0ec4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:57.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.962+0000 7f8d1dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:57.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.962+0000 7f8d1dffb640 1 -- 172.21.15.28:0/358623552 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c0bc850 con 0x555ef6bcd290 2024-04-25T01:40:57.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.962+0000 7f8d1dffb640 1 -- 172.21.15.28:0/358623552 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f8d2c0dbd30 con 0x555ef6bcd290 2024-04-25T01:40:57.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.962+0000 7f8d3f18c740 1 -- 172.21.15.28:0/358623552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d3000b9c0 con 0x555ef6bcd290 2024-04-25T01:40:57.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.962+0000 7f8d1dffb640 1 -- 172.21.15.28:0/358623552 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8d2c088210 con 0x555ef6bcd290 2024-04-25T01:40:57.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.962+0000 7f8d3f18c740 10 client.18531 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:57.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.962+0000 7f8d3f18c740 1 -- 172.21.15.28:0/358623552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555ef696df70 con 0x555ef6bcd290 2024-04-25T01:40:57.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.962+0000 7f8d1dffb640 1 -- 172.21.15.28:0/358623552 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d3000b9c0 con 0x555ef6bcd290 2024-04-25T01:40:57.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.962+0000 7f8d1dffb640 1 -- 172.21.15.28:0/358623552 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8d2c04e070 con 0x555ef6bcd290 2024-04-25T01:40:57.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.962+0000 7f8d3f18c740 10 client.18531 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:57.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.963+0000 7f8d3f18c740 10 client.18531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:57.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.963+0000 7f8d3f18c740 20 client.18531 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:57.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.963+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/358623552 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555ef6be8050 0x555ef6bea4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:57.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.963+0000 7f8d3f18c740 4 client.18531 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:57.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.963+0000 7f8d3f18c740 1 -- 172.21.15.28:0/358623552 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ef6b055a0 con 0x555ef6be8050 2024-04-25T01:40:57.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.963+0000 7f8d39940640 1 --2- 172.21.15.28:0/358623552 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555ef6be8050 0x555ef6bea4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:57.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.963+0000 7f8d39940640 1 --2- 172.21.15.28:0/358623552 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555ef6be8050 0x555ef6bea4d0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc9f3515be330d27 server_cookie=29e0afec14f94c79 in_seq=0 out_seq=0 2024-04-25T01:40:57.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.963+0000 7f8d1dffb640 10 client.18531 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:57.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.964+0000 7f8d1dffb640 1 -- 172.21.15.28:0/358623552 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555ef6b055a0 con 0x555ef6be8050 2024-04-25T01:40:57.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.964+0000 7f8d1dffb640 10 client.18531 handle_command_reply: tid=0 2024-04-25T01:40:57.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.964+0000 7f8d3f18c740 1 client.18531 shutdown 2024-04-25T01:40:57.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.965+0000 7f8d3f18c740 1 -- 172.21.15.28:0/358623552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bcd290 msgr2=0x555ef6bd4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:57.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.965+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/358623552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bcd290 0x555ef6bd4f30 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c000f80 tx=0x7f8d2c08cd50 comp rx=0 tx=0).stop 2024-04-25T01:40:57.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.965+0000 7f8d3f18c740 1 -- 172.21.15.28:0/358623552 shutdown_connections 2024-04-25T01:40:57.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.965+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/358623552 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555ef6be8050 0x555ef6bea4d0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.965+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/358623552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bd8010 0x555ef6bd59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.965+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/358623552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bce980 0x555ef6bd5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.965+0000 7f8d1dffb640 0 client.18531 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:57.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.965+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/358623552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bcd290 0x555ef6bd4f30 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.965+0000 7f8d3f18c740 1 -- 172.21.15.28:0/358623552 >> 172.21.15.28:0/358623552 conn(0x555ef6b95950 msgr2=0x555ef6bcf660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:57.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.965+0000 7f8d3f18c740 1 -- 172.21.15.28:0/358623552 shutdown_connections 2024-04-25T01:40:57.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.965+0000 7f8d3f18c740 1 -- 172.21.15.28:0/358623552 wait complete. 2024-04-25T01:40:57.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.965+0000 7f8d3f18c740 20 client.18531 trim_cache size 0 max 16384 2024-04-25T01:40:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.983+0000 7f8d3f18c740 1 Processor -- start 2024-04-25T01:40:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.983+0000 7f8d3f18c740 1 -- start start 2024-04-25T01:40:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.983+0000 7f8d3f18c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bcd290 0x555ef6be3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.983+0000 7f8d3f18c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bce980 0x555ef6befde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.983+0000 7f8d3f18c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bd8010 0x555ef6bf0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.983+0000 7f8d3f18c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d341ac870 con 0x555ef6bcd290 2024-04-25T01:40:57.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.983+0000 7f8d3f18c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8d341ae3b0 con 0x555ef6bce980 2024-04-25T01:40:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.983+0000 7f8d3f18c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8d341acf40 con 0x555ef6bd8010 2024-04-25T01:40:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.983+0000 7f8d3a141640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bcd290 0x555ef6be3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.983+0000 7f8d3a141640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bcd290 0x555ef6be3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47930/0 (socket says 172.21.15.28:47930) 2024-04-25T01:40:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.983+0000 7f8d3a141640 1 -- 172.21.15.28:0/3301138014 learned_addr learned my addr 172.21.15.28:0/3301138014 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.983+0000 7f8d3a942640 1 --2- 172.21.15.28:0/3301138014 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bd8010 0x555ef6bf0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.983+0000 7f8d3a141640 1 -- 172.21.15.28:0/3301138014 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bd8010 msgr2=0x555ef6bf0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.983+0000 7f8d3a141640 1 --2- 172.21.15.28:0/3301138014 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bd8010 0x555ef6bf0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.983+0000 7f8d3a141640 1 -- 172.21.15.28:0/3301138014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bce980 msgr2=0x555ef6befde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.983+0000 7f8d3a141640 1 --2- 172.21.15.28:0/3301138014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bce980 0x555ef6befde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.983+0000 7f8d3a141640 1 -- 172.21.15.28:0/3301138014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ef6805170 con 0x555ef6bcd290 2024-04-25T01:40:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.984+0000 7f8d3a141640 1 --2- 172.21.15.28:0/3301138014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bcd290 0x555ef6be3d70 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c0bb1a0 tx=0x7f8d2c030ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9376a5de576aaf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.984+0000 7f8d1ffff640 1 -- 172.21.15.28:0/3301138014 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c0bc850 con 0x555ef6bcd290 2024-04-25T01:40:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.984+0000 7f8d1ffff640 1 -- 172.21.15.28:0/3301138014 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d2c036020 con 0x555ef6bcd290 2024-04-25T01:40:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.984+0000 7f8d1ffff640 1 -- 172.21.15.28:0/3301138014 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c0b9190 con 0x555ef6bcd290 2024-04-25T01:40:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.984+0000 7f8d3f18c740 1 -- 172.21.15.28:0/3301138014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bcd290 msgr2=0x555ef6be3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.984+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/3301138014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bcd290 0x555ef6be3d70 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c0bb1a0 tx=0x7f8d2c030ab0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.984+0000 7f8d3f18c740 1 -- 172.21.15.28:0/3301138014 shutdown_connections 2024-04-25T01:40:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.984+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/3301138014 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bd8010 0x555ef6bf0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.984+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/3301138014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bce980 0x555ef6befde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.984+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/3301138014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bcd290 0x555ef6be3d70 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.984+0000 7f8d3f18c740 1 -- 172.21.15.28:0/3301138014 >> 172.21.15.28:0/3301138014 conn(0x555ef6b95950 msgr2=0x555ef6bc8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.984+0000 7f8d3f18c740 1 -- 172.21.15.28:0/3301138014 shutdown_connections 2024-04-25T01:40:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d3f18c740 1 -- 172.21.15.28:0/3301138014 wait complete. 2024-04-25T01:40:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d3f18c740 1 -- start start 2024-04-25T01:40:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d3f18c740 1 Processor -- start 2024-04-25T01:40:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d3f18c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bcd290 0x555ef6be0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d3f18c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bce980 0x555ef6bdd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d3f18c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bd8010 0x555ef6bdd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d3f18c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d341adce0 con 0x555ef6bce980 2024-04-25T01:40:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d3f18c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8d341ad610 con 0x555ef6bd8010 2024-04-25T01:40:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d3f18c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8d341aa870 con 0x555ef6bcd290 2024-04-25T01:40:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d3f18c740 10 client.0 _ll_register_callbacks cb 0x7f8d2c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d39940640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bce980 0x555ef6bdd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d39940640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bce980 0x555ef6bdd250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47946/0 (socket says 172.21.15.28:47946) 2024-04-25T01:40:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d39940640 1 -- 172.21.15.28:0/3032088303 learned_addr learned my addr 172.21.15.28:0/3032088303 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d3a141640 1 --2- 172.21.15.28:0/3032088303 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bcd290 0x555ef6be0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d3a942640 1 --2- 172.21.15.28:0/3032088303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bd8010 0x555ef6bdd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d39940640 1 -- 172.21.15.28:0/3032088303 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bcd290 msgr2=0x555ef6be0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d39940640 1 --2- 172.21.15.28:0/3032088303 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bcd290 0x555ef6be0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d39940640 1 -- 172.21.15.28:0/3032088303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bd8010 msgr2=0x555ef6bdd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d39940640 1 --2- 172.21.15.28:0/3032088303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bd8010 0x555ef6bdd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d39940640 1 -- 172.21.15.28:0/3032088303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ef6bde000 con 0x555ef6bce980 2024-04-25T01:40:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.985+0000 7f8d3a942640 1 --2- 172.21.15.28:0/3032088303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bd8010 0x555ef6bdd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.986+0000 7f8d39940640 1 --2- 172.21.15.28:0/3032088303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bce980 0x555ef6bdd250 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f8d240041f0 tx=0x7f8d24011f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=393a1be3c71279e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.986+0000 7f8d1dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.986+0000 7f8d1dffb640 1 -- 172.21.15.28:0/3032088303 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d24025070 con 0x555ef6bce980 2024-04-25T01:40:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.986+0000 7f8d1dffb640 1 -- 172.21.15.28:0/3032088303 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f8d24011020 con 0x555ef6bce980 2024-04-25T01:40:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.986+0000 7f8d3f18c740 1 -- 172.21.15.28:0/3032088303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d2c036020 con 0x555ef6bce980 2024-04-25T01:40:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.986+0000 7f8d1dffb640 1 -- 172.21.15.28:0/3032088303 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8d24030050 con 0x555ef6bce980 2024-04-25T01:40:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.986+0000 7f8d3f18c740 10 client.18537 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.986+0000 7f8d3f18c740 1 -- 172.21.15.28:0/3032088303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555ef6c4f7d0 con 0x555ef6bce980 2024-04-25T01:40:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.986+0000 7f8d1dffb640 1 -- 172.21.15.28:0/3032088303 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d2c036020 con 0x555ef6bce980 2024-04-25T01:40:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.986+0000 7f8d1dffb640 1 -- 172.21.15.28:0/3032088303 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8d24021070 con 0x555ef6bce980 2024-04-25T01:40:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.986+0000 7f8d3f18c740 10 client.18537 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.986+0000 7f8d3f18c740 10 client.18537 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.986+0000 7f8d3f18c740 20 client.18537 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.986+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/3032088303 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555ef6c4ff80 0x555ef6c523a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.986+0000 7f8d3f18c740 4 client.18537 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:40:57.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.986+0000 7f8d3f18c740 1 -- 172.21.15.28:0/3032088303 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555ef6b3e800 con 0x555ef6c4ff80 2024-04-25T01:40:57.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.987+0000 7f8d3a141640 1 --2- 172.21.15.28:0/3032088303 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555ef6c4ff80 0x555ef6c523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:57.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.987+0000 7f8d1dffb640 10 client.18537 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:57.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.987+0000 7f8d3a141640 1 --2- 172.21.15.28:0/3032088303 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555ef6c4ff80 0x555ef6c523a0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecfa1b04fc2323a9 server_cookie=588862da6203c412 in_seq=0 out_seq=0 2024-04-25T01:40:57.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.988+0000 7f8d1dffb640 1 -- 172.21.15.28:0/3032088303 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555ef6b3e800 con 0x555ef6c4ff80 2024-04-25T01:40:57.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.988+0000 7f8d1dffb640 10 client.18537 handle_command_reply: tid=0 2024-04-25T01:40:57.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.988+0000 7f8d3f18c740 1 client.18537 shutdown 2024-04-25T01:40:57.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.988+0000 7f8d3f18c740 1 -- 172.21.15.28:0/3032088303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bce980 msgr2=0x555ef6bdd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:57.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.988+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/3032088303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bce980 0x555ef6bdd250 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f8d240041f0 tx=0x7f8d24011f00 comp rx=0 tx=0).stop 2024-04-25T01:40:57.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.988+0000 7f8d3f18c740 1 -- 172.21.15.28:0/3032088303 shutdown_connections 2024-04-25T01:40:57.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.988+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/3032088303 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x555ef6c4ff80 0x555ef6c523a0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.988+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/3032088303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555ef6bd8010 0x555ef6bdd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.989+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/3032088303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ef6bce980 0x555ef6bdd250 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.989+0000 7f8d3f18c740 1 --2- 172.21.15.28:0/3032088303 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x555ef6bcd290 0x555ef6be0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.989+0000 7f8d3f18c740 1 -- 172.21.15.28:0/3032088303 >> 172.21.15.28:0/3032088303 conn(0x555ef6b95950 msgr2=0x555ef6bced60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.989+0000 7f8d1dffb640 0 client.18537 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.989+0000 7f8d3f18c740 1 -- 172.21.15.28:0/3032088303 shutdown_connections 2024-04-25T01:40:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.989+0000 7f8d3f18c740 1 -- 172.21.15.28:0/3032088303 wait complete. 2024-04-25T01:40:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.989+0000 7f8d3f18c740 20 client.18537 trim_cache size 0 max 16384 2024-04-25T01:40:57.980 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:40:57.980 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:40:57.980 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "d82d8e50-c76f-4508-97d0-a7d90c358a15", 2024-04-25T01:40:57.981 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:40:57.981 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:40:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.989+0000 7f8d3c3a5640 1 -- 172.21.15.28:0/3663797926 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8d200a2d00 msgr2=0x7f8d200a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.989+0000 7f8d3c3a5640 1 --2- 172.21.15.28:0/3663797926 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8d200a2d00 0x7f8d200a51c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8d30010740 tx=0x7f8d30017000 comp rx=0 tx=0).stop 2024-04-25T01:40:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.989+0000 7f8d3c3a5640 1 -- 172.21.15.28:0/3663797926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d34101710 msgr2=0x7f8d34198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.989+0000 7f8d3c3a5640 1 --2- 172.21.15.28:0/3663797926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d34101710 0x7f8d34198040 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c033040 tx=0x7f8d2c0039f0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.989+0000 7f8d3c3a5640 1 -- 172.21.15.28:0/3663797926 shutdown_connections 2024-04-25T01:40:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.989+0000 7f8d3c3a5640 1 --2- 172.21.15.28:0/3663797926 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8d200a2d00 0x7f8d200a51c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.989+0000 7f8d3c3a5640 1 --2- 172.21.15.28:0/3663797926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d34107720 0x7f8d3419c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.989+0000 7f8d3c3a5640 1 --2- 172.21.15.28:0/3663797926 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d341020d0 0x7f8d34198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.989+0000 7f8d3c3a5640 1 --2- 172.21.15.28:0/3663797926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d34101710 0x7f8d34198040 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.989+0000 7f8d3c3a5640 1 -- 172.21.15.28:0/3663797926 >> 172.21.15.28:0/3663797926 conn(0x7f8d340fd180 msgr2=0x7f8d340fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.989+0000 7f8d3c3a5640 1 -- 172.21.15.28:0/3663797926 shutdown_connections 2024-04-25T01:40:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:57.989+0000 7f8d3c3a5640 1 -- 172.21.15.28:0/3663797926 wait complete. 2024-04-25T01:40:57.986 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:40:57.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:40:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.105+0000 7fdd9cb09640 1 -- 172.21.15.28:0/3439417160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98069dc0 msgr2=0x7fdd9806a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.105+0000 7fdd9cb09640 1 --2- 172.21.15.28:0/3439417160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98069dc0 0x7fdd9806a240 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c009930 tx=0x7fdd8c02a420 comp rx=0 tx=0).stop 2024-04-25T01:40:58.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.105+0000 7fdd9cb09640 1 -- 172.21.15.28:0/3439417160 shutdown_connections 2024-04-25T01:40:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.105+0000 7fdd9cb09640 1 --2- 172.21.15.28:0/3439417160 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd9806a780 0x7fdd9810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.105+0000 7fdd9cb09640 1 --2- 172.21.15.28:0/3439417160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98069dc0 0x7fdd9806a240 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.105+0000 7fdd9cb09640 1 --2- 172.21.15.28:0/3439417160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd98107730 0x7fdd98107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.105+0000 7fdd9cb09640 1 -- 172.21.15.28:0/3439417160 >> 172.21.15.28:0/3439417160 conn(0x7fdd980776a0 msgr2=0x7fdd98077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.105+0000 7fdd9cb09640 1 -- 172.21.15.28:0/3439417160 shutdown_connections 2024-04-25T01:40:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.105+0000 7fdd9cb09640 1 -- 172.21.15.28:0/3439417160 wait complete. 2024-04-25T01:40:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.106+0000 7fdd9cb09640 1 Processor -- start 2024-04-25T01:40:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.106+0000 7fdd9cb09640 1 -- start start 2024-04-25T01:40:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.106+0000 7fdd9cb09640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98069dc0 0x7fdd98198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.106+0000 7fdd9cb09640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd9806a780 0x7fdd98198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.106+0000 7fdd9cb09640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd98107730 0x7fdd9819c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.106+0000 7fdd9cb09640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd98111b50 con 0x7fdd98069dc0 2024-04-25T01:40:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.106+0000 7fdd9cb09640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdd981119d0 con 0x7fdd98107730 2024-04-25T01:40:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.106+0000 7fdd9cb09640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fdd98111cd0 con 0x7fdd9806a780 2024-04-25T01:40:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.106+0000 7fdd9659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98069dc0 0x7fdd98198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.106+0000 7fdd9659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98069dc0 0x7fdd98198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47956/0 (socket says 172.21.15.28:47956) 2024-04-25T01:40:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.106+0000 7fdd9659c640 1 -- 172.21.15.28:0/2218925290 learned_addr learned my addr 172.21.15.28:0/2218925290 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.106+0000 7fdd96d9d640 1 --2- 172.21.15.28:0/2218925290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd98107730 0x7fdd9819c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.106+0000 7fdd95d9b640 1 --2- 172.21.15.28:0/2218925290 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd9806a780 0x7fdd98198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.106+0000 7fdd9659c640 1 -- 172.21.15.28:0/2218925290 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd9806a780 msgr2=0x7fdd98198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.106+0000 7fdd9659c640 1 --2- 172.21.15.28:0/2218925290 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd9806a780 0x7fdd98198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.106+0000 7fdd9659c640 1 -- 172.21.15.28:0/2218925290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd98107730 msgr2=0x7fdd9819c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.106+0000 7fdd9659c640 1 --2- 172.21.15.28:0/2218925290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd98107730 0x7fdd9819c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.106+0000 7fdd9659c640 1 -- 172.21.15.28:0/2218925290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd9819cfb0 con 0x7fdd98069dc0 2024-04-25T01:40:58.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.106+0000 7fdd96d9d640 1 --2- 172.21.15.28:0/2218925290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd98107730 0x7fdd9819c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:58.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.107+0000 7fdd9659c640 1 --2- 172.21.15.28:0/2218925290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98069dc0 0x7fdd98198010 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fdd8800db80 tx=0x7fdd8800b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.107+0000 7fdd877fe640 1 -- 172.21.15.28:0/2218925290 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd88017070 con 0x7fdd98069dc0 2024-04-25T01:40:58.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.107+0000 7fdd9cb09640 1 -- 172.21.15.28:0/2218925290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd9819d2a0 con 0x7fdd98069dc0 2024-04-25T01:40:58.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.107+0000 7fdd877fe640 1 -- 172.21.15.28:0/2218925290 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd88003b70 con 0x7fdd98069dc0 2024-04-25T01:40:58.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.107+0000 7fdd877fe640 1 -- 172.21.15.28:0/2218925290 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd880106c0 con 0x7fdd98069dc0 2024-04-25T01:40:58.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.107+0000 7fdd9cb09640 1 -- 172.21.15.28:0/2218925290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd981a4b30 con 0x7fdd98069dc0 2024-04-25T01:40:58.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.109+0000 7fdd877fe640 1 -- 172.21.15.28:0/2218925290 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fdd88004020 con 0x7fdd98069dc0 2024-04-25T01:40:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.109+0000 7fdd9f80e740 1 Processor -- start 2024-04-25T01:40:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd9f80e740 1 -- start start 2024-04-25T01:40:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd877fe640 1 --2- 172.21.15.28:0/2218925290 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdd800a2d00 0x7fdd800a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd9f80e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a66f630 0x558a5a66fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd9f80e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 0x558a5a67a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd9f80e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a67a620 0x558a5a67ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd9f80e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558a5a5d46b0 con 0x558a5a67a620 2024-04-25T01:40:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd9f80e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558a5a57c880 con 0x558a5a66f630 2024-04-25T01:40:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd9f80e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x558a5a5d0670 con 0x558a5a66ff50 2024-04-25T01:40:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd96d9d640 1 --2- 172.21.15.28:0/2218925290 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdd800a2d00 0x7fdd800a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd877fe640 1 -- 172.21.15.28:0/2218925290 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fdd880cf610 con 0x7fdd98069dc0 2024-04-25T01:40:58.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd9659c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a66f630 0x558a5a66fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd9659c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a66f630 0x558a5a66fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:44710/0 (socket says 172.21.15.28:44710) 2024-04-25T01:40:58.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd9659c640 1 -- 172.21.15.28:0/3958237613 learned_addr learned my addr 172.21.15.28:0/3958237613 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd96d9d640 1 --2- 172.21.15.28:0/3958237613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a67a620 0x558a5a67ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd95d9b640 1 --2- 172.21.15.28:0/3958237613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 0x558a5a67a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd9659c640 1 -- 172.21.15.28:0/3958237613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 msgr2=0x558a5a67a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd9659c640 1 --2- 172.21.15.28:0/3958237613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 0x558a5a67a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd9659c640 1 -- 172.21.15.28:0/3958237613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a67a620 msgr2=0x558a5a67ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd9659c640 1 --2- 172.21.15.28:0/3958237613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a67a620 0x558a5a67ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.110+0000 7fdd9659c640 1 -- 172.21.15.28:0/3958237613 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a5a3b27d0 con 0x558a5a66f630 2024-04-25T01:40:58.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.111+0000 7fdd96d9d640 1 --2- 172.21.15.28:0/2218925290 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdd800a2d00 0x7fdd800a51c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdd90013910 tx=0x7fdd9000baf0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.111+0000 7fdd9659c640 1 --2- 172.21.15.28:0/3958237613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a66f630 0x558a5a66fa10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdd880a3ec0 tx=0x7fdd8809b0e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=931473637af7a7ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.111+0000 7fdd84ff9640 1 -- 172.21.15.28:0/3958237613 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8809b780 con 0x558a5a66f630 2024-04-25T01:40:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.111+0000 7fdd84ff9640 1 -- 172.21.15.28:0/3958237613 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd8809fb30 con 0x558a5a66f630 2024-04-25T01:40:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.111+0000 7fdd84ff9640 1 -- 172.21.15.28:0/3958237613 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd88099b10 con 0x558a5a66f630 2024-04-25T01:40:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.112+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3958237613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a66f630 msgr2=0x558a5a66fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.112+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3958237613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a66f630 0x558a5a66fa10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdd880a3ec0 tx=0x7fdd8809b0e0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.112+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3958237613 shutdown_connections 2024-04-25T01:40:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.112+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3958237613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a67a620 0x558a5a67ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.112+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3958237613 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 0x558a5a67a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.112+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3958237613 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a66f630 0x558a5a66fa10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.112+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3958237613 >> 172.21.15.28:0/3958237613 conn(0x558a5a66b950 msgr2=0x558a5a66bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.112+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3958237613 shutdown_connections 2024-04-25T01:40:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.112+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3958237613 wait complete. 2024-04-25T01:40:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.112+0000 7fdd9f80e740 1 -- start start 2024-04-25T01:40:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.112+0000 7fdd9f80e740 1 Processor -- start 2024-04-25T01:40:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.112+0000 7fdd9f80e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a66f630 0x558a5a6775f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.112+0000 7fdd9f80e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 0x558a5a677b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.112+0000 7fdd9f80e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a67a620 0x558a5a678070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.112+0000 7fdd9f80e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558a5a67d410 con 0x558a5a66f630 2024-04-25T01:40:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.112+0000 7fdd9f80e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558a5a58ff50 con 0x558a5a67a620 2024-04-25T01:40:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.112+0000 7fdd9f80e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x558a5a67d590 con 0x558a5a66ff50 2024-04-25T01:40:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.112+0000 7fdd9f80e740 10 client.0 _ll_register_callbacks cb 0x558a5a5cb000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:58.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.112+0000 7fdd9659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a66f630 0x558a5a6775f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.113+0000 7fdd9659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a66f630 0x558a5a6775f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47984/0 (socket says 172.21.15.28:47984) 2024-04-25T01:40:58.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.113+0000 7fdd9659c640 1 -- 172.21.15.28:0/3352832740 learned_addr learned my addr 172.21.15.28:0/3352832740 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.113+0000 7fdd95d9b640 1 --2- 172.21.15.28:0/3352832740 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 0x558a5a677b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.113+0000 7fdd96d9d640 1 --2- 172.21.15.28:0/3352832740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a67a620 0x558a5a678070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.113+0000 7fdd9659c640 1 -- 172.21.15.28:0/3352832740 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 msgr2=0x558a5a677b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.113+0000 7fdd9659c640 1 --2- 172.21.15.28:0/3352832740 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 0x558a5a677b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.113+0000 7fdd9659c640 1 -- 172.21.15.28:0/3352832740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a67a620 msgr2=0x558a5a678070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.113+0000 7fdd9659c640 1 --2- 172.21.15.28:0/3352832740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a67a620 0x558a5a678070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.113+0000 7fdd9659c640 1 -- 172.21.15.28:0/3352832740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a5a2e83f0 con 0x558a5a66f630 2024-04-25T01:40:58.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.113+0000 7fdd9659c640 1 --2- 172.21.15.28:0/3352832740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a66f630 0x558a5a6775f0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fdd8809b710 tx=0x7fdd8809b210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ade380d9aecd244 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.113+0000 7fdd7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:58.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.113+0000 7fdd7dffb640 1 -- 172.21.15.28:0/3352832740 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8809b780 con 0x558a5a66f630 2024-04-25T01:40:58.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.113+0000 7fdd7dffb640 1 -- 172.21.15.28:0/3352832740 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fdd88003d20 con 0x558a5a66f630 2024-04-25T01:40:58.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.113+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3352832740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd8809fb30 con 0x558a5a66f630 2024-04-25T01:40:58.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.113+0000 7fdd7dffb640 1 -- 172.21.15.28:0/3352832740 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdd880ba080 con 0x558a5a66f630 2024-04-25T01:40:58.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.113+0000 7fdd9f80e740 10 client.18549 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:58.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.113+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3352832740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558a5a2a80f0 con 0x558a5a66f630 2024-04-25T01:40:58.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.113+0000 7fdd7dffb640 1 -- 172.21.15.28:0/3352832740 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd8809fb30 con 0x558a5a66f630 2024-04-25T01:40:58.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.114+0000 7fdd7dffb640 1 -- 172.21.15.28:0/3352832740 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fdd88097040 con 0x558a5a66f630 2024-04-25T01:40:58.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.114+0000 7fdd9f80e740 10 client.18549 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:58.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.114+0000 7fdd9f80e740 10 client.18549 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:58.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.114+0000 7fdd9f80e740 20 client.18549 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:58.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.114+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3352832740 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558a5a68a7d0 0x558a5a68cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.114+0000 7fdd9f80e740 4 client.18549 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:58.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.114+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3352832740 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a5a5e1bf0 con 0x558a5a68a7d0 2024-04-25T01:40:58.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.114+0000 7fdd95d9b640 1 --2- 172.21.15.28:0/3352832740 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558a5a68a7d0 0x558a5a68cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.114+0000 7fdd7dffb640 10 client.18549 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:58.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.114+0000 7fdd95d9b640 1 --2- 172.21.15.28:0/3352832740 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558a5a68a7d0 0x558a5a68cbc0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b4d1429d599afcc server_cookie=e14d2b1a1d607cc1 in_seq=0 out_seq=0 2024-04-25T01:40:58.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.115+0000 7fdd7dffb640 1 -- 172.21.15.28:0/3352832740 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558a5a5e1bf0 con 0x558a5a68a7d0 2024-04-25T01:40:58.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.115+0000 7fdd7dffb640 10 client.18549 handle_command_reply: tid=0 2024-04-25T01:40:58.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.115+0000 7fdd9f80e740 1 client.18549 shutdown 2024-04-25T01:40:58.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.115+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3352832740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a66f630 msgr2=0x558a5a6775f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.115+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3352832740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a66f630 0x558a5a6775f0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fdd8809b710 tx=0x7fdd8809b210 comp rx=0 tx=0).stop 2024-04-25T01:40:58.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.115+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3352832740 shutdown_connections 2024-04-25T01:40:58.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.115+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3352832740 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558a5a68a7d0 0x558a5a68cbc0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.115+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3352832740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a67a620 0x558a5a678070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.115+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3352832740 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 0x558a5a677b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.115+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3352832740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a66f630 0x558a5a6775f0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.115+0000 7fdd7dffb640 0 client.18549 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:58.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.115+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3352832740 >> 172.21.15.28:0/3352832740 conn(0x558a5a66b950 msgr2=0x558a5a671750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.115+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3352832740 shutdown_connections 2024-04-25T01:40:58.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.115+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3352832740 wait complete. 2024-04-25T01:40:58.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.115+0000 7fdd9f80e740 20 client.18549 trim_cache size 0 max 16384 2024-04-25T01:40:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd9f80e740 1 Processor -- start 2024-04-25T01:40:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd9f80e740 1 -- start start 2024-04-25T01:40:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd9f80e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a66f630 0x558a5a679d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd9f80e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 0x558a5a677e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd9f80e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a67a620 0x558a5a671e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd9f80e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd981aeb10 con 0x558a5a67a620 2024-04-25T01:40:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd9f80e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdd981b0650 con 0x558a5a66f630 2024-04-25T01:40:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd9f80e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fdd981af1e0 con 0x558a5a66ff50 2024-04-25T01:40:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd96d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a67a620 0x558a5a671e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd96d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a67a620 0x558a5a671e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47990/0 (socket says 172.21.15.28:47990) 2024-04-25T01:40:58.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd96d9d640 1 -- 172.21.15.28:0/3760962810 learned_addr learned my addr 172.21.15.28:0/3760962810 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd9659c640 1 --2- 172.21.15.28:0/3760962810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a66f630 0x558a5a679d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd95d9b640 1 --2- 172.21.15.28:0/3760962810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 0x558a5a677e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd96d9d640 1 -- 172.21.15.28:0/3760962810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 msgr2=0x558a5a677e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd96d9d640 1 --2- 172.21.15.28:0/3760962810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 0x558a5a677e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd96d9d640 1 -- 172.21.15.28:0/3760962810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a66f630 msgr2=0x558a5a679d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd96d9d640 1 --2- 172.21.15.28:0/3760962810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a66f630 0x558a5a679d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd96d9d640 1 -- 172.21.15.28:0/3760962810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a5a66fd80 con 0x558a5a67a620 2024-04-25T01:40:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd95d9b640 1 --2- 172.21.15.28:0/3760962810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 0x558a5a677e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.133+0000 7fdd96d9d640 1 --2- 172.21.15.28:0/3760962810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a67a620 0x558a5a671e50 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000b980 tx=0x7fdd90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9d88db909216e4c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.134+0000 7fdd7ffff640 1 -- 172.21.15.28:0/3760962810 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90024070 con 0x558a5a67a620 2024-04-25T01:40:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.134+0000 7fdd7ffff640 1 -- 172.21.15.28:0/3760962810 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd900047a0 con 0x558a5a67a620 2024-04-25T01:40:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.134+0000 7fdd7ffff640 1 -- 172.21.15.28:0/3760962810 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90011940 con 0x558a5a67a620 2024-04-25T01:40:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.134+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3760962810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a67a620 msgr2=0x558a5a671e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.134+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3760962810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a67a620 0x558a5a671e50 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000b980 tx=0x7fdd90004730 comp rx=0 tx=0).stop 2024-04-25T01:40:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.134+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3760962810 shutdown_connections 2024-04-25T01:40:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.134+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3760962810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a67a620 0x558a5a671e50 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.134+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3760962810 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 0x558a5a677e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.134+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3760962810 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a66f630 0x558a5a679d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.134+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3760962810 >> 172.21.15.28:0/3760962810 conn(0x558a5a66b950 msgr2=0x558a5a66a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.134+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3760962810 shutdown_connections 2024-04-25T01:40:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.134+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3760962810 wait complete. 2024-04-25T01:40:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd9f80e740 1 -- start start 2024-04-25T01:40:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd9f80e740 1 Processor -- start 2024-04-25T01:40:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd9f80e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a66f630 0x558a5a6ecd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd9f80e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 0x558a5a6ed2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd9f80e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a67a620 0x558a5a6edb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd9f80e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdd981aff80 con 0x558a5a66f630 2024-04-25T01:40:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd9f80e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdd981af8b0 con 0x558a5a67a620 2024-04-25T01:40:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd9f80e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fdd981acb10 con 0x558a5a66ff50 2024-04-25T01:40:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd9f80e740 10 client.0 _ll_register_callbacks cb 0x7fdd880c9070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd9659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a66f630 0x558a5a6ecd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd9659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a66f630 0x558a5a6ecd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48004/0 (socket says 172.21.15.28:48004) 2024-04-25T01:40:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd9659c640 1 -- 172.21.15.28:0/3798350878 learned_addr learned my addr 172.21.15.28:0/3798350878 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd96d9d640 1 --2- 172.21.15.28:0/3798350878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a67a620 0x558a5a6edb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd95d9b640 1 --2- 172.21.15.28:0/3798350878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 0x558a5a6ed2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd9659c640 1 -- 172.21.15.28:0/3798350878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 msgr2=0x558a5a6ed2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd9659c640 1 --2- 172.21.15.28:0/3798350878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 0x558a5a6ed2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd9659c640 1 -- 172.21.15.28:0/3798350878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a67a620 msgr2=0x558a5a6edb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd9659c640 1 --2- 172.21.15.28:0/3798350878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a67a620 0x558a5a6edb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd9659c640 1 -- 172.21.15.28:0/3798350878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a5a6ee200 con 0x558a5a66f630 2024-04-25T01:40:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.135+0000 7fdd9659c640 1 --2- 172.21.15.28:0/3798350878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a66f630 0x558a5a6ecd60 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fdd8800db50 tx=0x7fdd8809b0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43bf32826c57dcf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.136+0000 7fdd7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.136+0000 7fdd7dffb640 1 -- 172.21.15.28:0/3798350878 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd880cf450 con 0x558a5a66f630 2024-04-25T01:40:58.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.136+0000 7fdd7dffb640 1 -- 172.21.15.28:0/3798350878 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fdd880ccc30 con 0x558a5a66f630 2024-04-25T01:40:58.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.136+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3798350878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd900047a0 con 0x558a5a66f630 2024-04-25T01:40:58.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.136+0000 7fdd7dffb640 1 -- 172.21.15.28:0/3798350878 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdd8809fb30 con 0x558a5a66f630 2024-04-25T01:40:58.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.136+0000 7fdd9f80e740 10 client.18555 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:58.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.136+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3798350878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558a5a6f5600 con 0x558a5a66f630 2024-04-25T01:40:58.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.136+0000 7fdd7dffb640 1 -- 172.21.15.28:0/3798350878 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd900047a0 con 0x558a5a66f630 2024-04-25T01:40:58.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.136+0000 7fdd7dffb640 1 -- 172.21.15.28:0/3798350878 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fdd88097040 con 0x558a5a66f630 2024-04-25T01:40:58.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.136+0000 7fdd9f80e740 10 client.18555 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:58.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.136+0000 7fdd9f80e740 10 client.18555 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:58.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.136+0000 7fdd9f80e740 20 client.18555 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:58.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.136+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3798350878 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558a5a6f5b70 0x558a5a6f7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.136+0000 7fdd9f80e740 4 client.18555 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:40:58.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.136+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3798350878 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558a5a5dbf40 con 0x558a5a6f5b70 2024-04-25T01:40:58.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.136+0000 7fdd95d9b640 1 --2- 172.21.15.28:0/3798350878 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558a5a6f5b70 0x558a5a6f7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.137+0000 7fdd95d9b640 1 --2- 172.21.15.28:0/3798350878 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558a5a6f5b70 0x558a5a6f7f90 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc5d44dd6d36b30e server_cookie=aa136bef19db3bbb in_seq=0 out_seq=0 2024-04-25T01:40:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.137+0000 7fdd7dffb640 10 client.18555 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.137+0000 7fdd7dffb640 1 -- 172.21.15.28:0/3798350878 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558a5a5dbf40 con 0x558a5a6f5b70 2024-04-25T01:40:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.137+0000 7fdd7dffb640 10 client.18555 handle_command_reply: tid=0 2024-04-25T01:40:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.137+0000 7fdd9f80e740 1 client.18555 shutdown 2024-04-25T01:40:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.137+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3798350878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a66f630 msgr2=0x558a5a6ecd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.137+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3798350878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a66f630 0x558a5a6ecd60 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fdd8800db50 tx=0x7fdd8809b0e0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.137+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3798350878 shutdown_connections 2024-04-25T01:40:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.137+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3798350878 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558a5a6f5b70 0x558a5a6f7f90 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.137+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3798350878 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a5a67a620 0x558a5a6edb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.137+0000 7fdd7dffb640 0 client.18555 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.137+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3798350878 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558a5a66ff50 0x558a5a6ed2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.137+0000 7fdd9f80e740 1 --2- 172.21.15.28:0/3798350878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558a5a66f630 0x558a5a6ecd60 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.137+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3798350878 >> 172.21.15.28:0/3798350878 conn(0x558a5a66b950 msgr2=0x558a5a66a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.137+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3798350878 shutdown_connections 2024-04-25T01:40:58.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.137+0000 7fdd9f80e740 1 -- 172.21.15.28:0/3798350878 wait complete. 2024-04-25T01:40:58.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.137+0000 7fdd9f80e740 20 client.18555 trim_cache size 0 max 16384 2024-04-25T01:40:58.131 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:40:58.131 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:40:58.131 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:40:58.131 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:40:58.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.138+0000 7fdd9cb09640 1 -- 172.21.15.28:0/2218925290 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdd800a2d00 msgr2=0x7fdd800a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.138+0000 7fdd9cb09640 1 --2- 172.21.15.28:0/2218925290 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdd800a2d00 0x7fdd800a51c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdd90013910 tx=0x7fdd9000baf0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.138+0000 7fdd9cb09640 1 -- 172.21.15.28:0/2218925290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98069dc0 msgr2=0x7fdd98198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.138+0000 7fdd9cb09640 1 --2- 172.21.15.28:0/2218925290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98069dc0 0x7fdd98198010 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fdd8800db80 tx=0x7fdd8800b630 comp rx=0 tx=0).stop 2024-04-25T01:40:58.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.138+0000 7fdd9cb09640 1 -- 172.21.15.28:0/2218925290 shutdown_connections 2024-04-25T01:40:58.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.138+0000 7fdd9cb09640 1 --2- 172.21.15.28:0/2218925290 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdd800a2d00 0x7fdd800a51c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.138+0000 7fdd9cb09640 1 --2- 172.21.15.28:0/2218925290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdd98107730 0x7fdd9819c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.138+0000 7fdd9cb09640 1 --2- 172.21.15.28:0/2218925290 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdd9806a780 0x7fdd98198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.138+0000 7fdd9cb09640 1 --2- 172.21.15.28:0/2218925290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd98069dc0 0x7fdd98198010 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.138+0000 7fdd9cb09640 1 -- 172.21.15.28:0/2218925290 >> 172.21.15.28:0/2218925290 conn(0x7fdd980776a0 msgr2=0x7fdd980fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.138+0000 7fdd9cb09640 1 -- 172.21.15.28:0/2218925290 shutdown_connections 2024-04-25T01:40:58.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.138+0000 7fdd9cb09640 1 -- 172.21.15.28:0/2218925290 wait complete. 2024-04-25T01:40:58.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:58 smithi028 ceph-mon[29390]: pgmap v243: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:40:58.134 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:40:58.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:40:58.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.251+0000 7fcb95ba2640 1 -- 172.21.15.28:0/1851462458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb90102a20 msgr2=0x7fcb90102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.251+0000 7fcb95ba2640 1 --2- 172.21.15.28:0/1851462458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb90102a20 0x7fcb90102e00 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fcb80009ca0 tx=0x7fcb8002a950 comp rx=0 tx=0).stop 2024-04-25T01:40:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.251+0000 7fcb95ba2640 1 -- 172.21.15.28:0/1851462458 shutdown_connections 2024-04-25T01:40:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.251+0000 7fcb95ba2640 1 --2- 172.21.15.28:0/1851462458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb9010bd60 0x7fcb9010e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.251+0000 7fcb95ba2640 1 --2- 172.21.15.28:0/1851462458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb90103340 0x7fcb9010b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.251+0000 7fcb95ba2640 1 --2- 172.21.15.28:0/1851462458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb90102a20 0x7fcb90102e00 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.251+0000 7fcb95ba2640 1 -- 172.21.15.28:0/1851462458 >> 172.21.15.28:0/1851462458 conn(0x7fcb900777a0 msgr2=0x7fcb90077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.251+0000 7fcb95ba2640 1 -- 172.21.15.28:0/1851462458 shutdown_connections 2024-04-25T01:40:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.251+0000 7fcb95ba2640 1 -- 172.21.15.28:0/1851462458 wait complete. 2024-04-25T01:40:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.252+0000 7fcb95ba2640 1 Processor -- start 2024-04-25T01:40:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.252+0000 7fcb95ba2640 1 -- start start 2024-04-25T01:40:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.252+0000 7fcb95ba2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb90102a20 0x7fcb90198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.252+0000 7fcb95ba2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb90103340 0x7fcb90198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.252+0000 7fcb95ba2640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb9010bd60 0x7fcb9019c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.252+0000 7fcb95ba2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcb90111b80 con 0x7fcb90102a20 2024-04-25T01:40:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.252+0000 7fcb95ba2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcb90111a00 con 0x7fcb90103340 2024-04-25T01:40:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.252+0000 7fcb95ba2640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fcb90111d00 con 0x7fcb9010bd60 2024-04-25T01:40:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.253+0000 7fcb8f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb90102a20 0x7fcb90198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.253+0000 7fcb8f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb90102a20 0x7fcb90198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48030/0 (socket says 172.21.15.28:48030) 2024-04-25T01:40:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.253+0000 7fcb8f7fe640 1 -- 172.21.15.28:0/3421654448 learned_addr learned my addr 172.21.15.28:0/3421654448 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.253+0000 7fcb8ffff640 1 --2- 172.21.15.28:0/3421654448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb9010bd60 0x7fcb9019c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.253+0000 7fcb8f7fe640 1 -- 172.21.15.28:0/3421654448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb9010bd60 msgr2=0x7fcb9019c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.253+0000 7fcb8effd640 1 --2- 172.21.15.28:0/3421654448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb90103340 0x7fcb90198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.253+0000 7fcb8f7fe640 1 --2- 172.21.15.28:0/3421654448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb9010bd60 0x7fcb9019c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.253+0000 7fcb8f7fe640 1 -- 172.21.15.28:0/3421654448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb90103340 msgr2=0x7fcb90198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.253+0000 7fcb8f7fe640 1 --2- 172.21.15.28:0/3421654448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb90103340 0x7fcb90198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.253+0000 7fcb8f7fe640 1 -- 172.21.15.28:0/3421654448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb9019d010 con 0x7fcb90102a20 2024-04-25T01:40:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.253+0000 7fcb8ffff640 1 --2- 172.21.15.28:0/3421654448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb9010bd60 0x7fcb9019c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:58.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.253+0000 7fcb8effd640 1 --2- 172.21.15.28:0/3421654448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb90103340 0x7fcb90198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:58.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.253+0000 7fcb8f7fe640 1 --2- 172.21.15.28:0/3421654448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb90102a20 0x7fcb90198020 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fcb80005e30 tx=0x7fcb80003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.253+0000 7fcb8cff9640 1 -- 172.21.15.28:0/3421654448 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80034040 con 0x7fcb90102a20 2024-04-25T01:40:58.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.253+0000 7fcb95ba2640 1 -- 172.21.15.28:0/3421654448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb9019d2a0 con 0x7fcb90102a20 2024-04-25T01:40:58.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.253+0000 7fcb8cff9640 1 -- 172.21.15.28:0/3421654448 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb8002fe20 con 0x7fcb90102a20 2024-04-25T01:40:58.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.253+0000 7fcb8cff9640 1 -- 172.21.15.28:0/3421654448 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80045480 con 0x7fcb90102a20 2024-04-25T01:40:58.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.253+0000 7fcb95ba2640 1 -- 172.21.15.28:0/3421654448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb901a4b90 con 0x7fcb90102a20 2024-04-25T01:40:58.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.254+0000 7fcb95ba2640 1 -- 172.21.15.28:0/3421654448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb900689b0 con 0x7fcb90102a20 2024-04-25T01:40:58.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.256+0000 7fcb8cff9640 1 -- 172.21.15.28:0/3421654448 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fcb80037020 con 0x7fcb90102a20 2024-04-25T01:40:58.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.256+0000 7fcb8cff9640 1 --2- 172.21.15.28:0/3421654448 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcb780a2d00 0x7fcb780a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.256+0000 7fcb8effd640 1 --2- 172.21.15.28:0/3421654448 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcb780a2d00 0x7fcb780a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.256+0000 7fcb8cff9640 1 -- 172.21.15.28:0/3421654448 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fcb800f0760 con 0x7fcb90102a20 2024-04-25T01:40:58.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.257+0000 7fcb8effd640 1 --2- 172.21.15.28:0/3421654448 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcb780a2d00 0x7fcb780a51c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcb900ff750 tx=0x7fcb84005e50 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.260+0000 7fcb8cff9640 1 -- 172.21.15.28:0/3421654448 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcb800bd190 con 0x7fcb90102a20 2024-04-25T01:40:58.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:58 smithi130 ceph-mon[32702]: pgmap v243: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:40:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.487+0000 7fcb95ba2640 1 -- 172.21.15.28:0/3421654448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb901a4d60 con 0x7fcb90102a20 2024-04-25T01:40:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.488+0000 7fcb8cff9640 1 -- 172.21.15.28:0/3421654448 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7fcb800bca90 con 0x7fcb90102a20 2024-04-25T01:40:58.474 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:40:58.474 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:40:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:40:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.490+0000 7fcb95ba2640 1 -- 172.21.15.28:0/3421654448 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcb780a2d00 msgr2=0x7fcb780a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.490+0000 7fcb95ba2640 1 --2- 172.21.15.28:0/3421654448 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcb780a2d00 0x7fcb780a51c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcb900ff750 tx=0x7fcb84005e50 comp rx=0 tx=0).stop 2024-04-25T01:40:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.490+0000 7fcb95ba2640 1 -- 172.21.15.28:0/3421654448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb90102a20 msgr2=0x7fcb90198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.490+0000 7fcb95ba2640 1 --2- 172.21.15.28:0/3421654448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb90102a20 0x7fcb90198020 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fcb80005e30 tx=0x7fcb80003f90 comp rx=0 tx=0).stop 2024-04-25T01:40:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.490+0000 7fcb95ba2640 1 -- 172.21.15.28:0/3421654448 shutdown_connections 2024-04-25T01:40:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.490+0000 7fcb95ba2640 1 --2- 172.21.15.28:0/3421654448 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fcb780a2d00 0x7fcb780a51c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.490+0000 7fcb95ba2640 1 --2- 172.21.15.28:0/3421654448 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fcb9010bd60 0x7fcb9019c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.490+0000 7fcb95ba2640 1 --2- 172.21.15.28:0/3421654448 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcb90103340 0x7fcb90198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.490+0000 7fcb95ba2640 1 --2- 172.21.15.28:0/3421654448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb90102a20 0x7fcb90198020 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.490+0000 7fcb95ba2640 1 -- 172.21.15.28:0/3421654448 >> 172.21.15.28:0/3421654448 conn(0x7fcb900777a0 msgr2=0x7fcb900fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.490+0000 7fcb95ba2640 1 -- 172.21.15.28:0/3421654448 shutdown_connections 2024-04-25T01:40:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.490+0000 7fcb95ba2640 1 -- 172.21.15.28:0/3421654448 wait complete. 2024-04-25T01:40:58.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:40:58.492 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:58 smithi032 ceph-mon[32461]: pgmap v243: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:40:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.605+0000 7f4cd2329640 1 -- 172.21.15.28:0/2778010262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc107730 msgr2=0x7f4ccc107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.605+0000 7f4cd2329640 1 --2- 172.21.15.28:0/2778010262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc107730 0x7f4ccc107b10 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc009930 tx=0x7f4cbc02a420 comp rx=0 tx=0).stop 2024-04-25T01:40:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.605+0000 7f4cd2329640 1 -- 172.21.15.28:0/2778010262 shutdown_connections 2024-04-25T01:40:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.605+0000 7f4cd2329640 1 --2- 172.21.15.28:0/2778010262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ccc06a780 0x7f4ccc10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.605+0000 7f4cd2329640 1 --2- 172.21.15.28:0/2778010262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ccc069dc0 0x7f4ccc06a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.605+0000 7f4cd2329640 1 --2- 172.21.15.28:0/2778010262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc107730 0x7f4ccc107b10 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.605+0000 7f4cd2329640 1 -- 172.21.15.28:0/2778010262 >> 172.21.15.28:0/2778010262 conn(0x7f4ccc0776a0 msgr2=0x7f4ccc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.605+0000 7f4cd2329640 1 -- 172.21.15.28:0/2778010262 shutdown_connections 2024-04-25T01:40:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.605+0000 7f4cd2329640 1 -- 172.21.15.28:0/2778010262 wait complete. 2024-04-25T01:40:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.606+0000 7f4cd2329640 1 Processor -- start 2024-04-25T01:40:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.606+0000 7f4cd2329640 1 -- start start 2024-04-25T01:40:58.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.606+0000 7f4cd2329640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ccc069dc0 0x7f4ccc197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.606+0000 7f4cd2329640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ccc06a780 0x7f4ccc1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.606+0000 7f4cd2329640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc107730 0x7f4ccc19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.606+0000 7f4cd2329640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc111b00 con 0x7f4ccc107730 2024-04-25T01:40:58.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.606+0000 7f4cd2329640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc111980 con 0x7f4ccc06a780 2024-04-25T01:40:58.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.606+0000 7f4cd2329640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc111c80 con 0x7f4ccc069dc0 2024-04-25T01:40:58.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.606+0000 7f4cd08c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc107730 0x7f4ccc19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.606+0000 7f4cd08c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc107730 0x7f4ccc19c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48042/0 (socket says 172.21.15.28:48042) 2024-04-25T01:40:58.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.606+0000 7f4cd08c6640 1 -- 172.21.15.28:0/1019110280 learned_addr learned my addr 172.21.15.28:0/1019110280 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.606+0000 7f4ccb7fe640 1 --2- 172.21.15.28:0/1019110280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ccc06a780 0x7f4ccc1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.607+0000 7f4ccbfff640 1 --2- 172.21.15.28:0/1019110280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ccc069dc0 0x7f4ccc197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.607+0000 7f4cd08c6640 1 -- 172.21.15.28:0/1019110280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ccc069dc0 msgr2=0x7f4ccc197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.607+0000 7f4cd08c6640 1 --2- 172.21.15.28:0/1019110280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ccc069dc0 0x7f4ccc197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.607+0000 7f4cd08c6640 1 -- 172.21.15.28:0/1019110280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ccc06a780 msgr2=0x7f4ccc1984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.607+0000 7f4cd08c6640 1 --2- 172.21.15.28:0/1019110280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ccc06a780 0x7f4ccc1984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.607+0000 7f4cd08c6640 1 -- 172.21.15.28:0/1019110280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ccc19cf70 con 0x7f4ccc107730 2024-04-25T01:40:58.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.607+0000 7f4cd08c6640 1 --2- 172.21.15.28:0/1019110280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc107730 0x7f4ccc19c840 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400cce0 tx=0x7f4cc400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.607+0000 7f4cc97fa640 1 -- 172.21.15.28:0/1019110280 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc400f450 con 0x7f4ccc107730 2024-04-25T01:40:58.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.607+0000 7f4cc97fa640 1 -- 172.21.15.28:0/1019110280 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cc400fad0 con 0x7f4ccc107730 2024-04-25T01:40:58.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.607+0000 7f4cd2329640 1 -- 172.21.15.28:0/1019110280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ccc19d260 con 0x7f4ccc107730 2024-04-25T01:40:58.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.607+0000 7f4cc97fa640 1 -- 172.21.15.28:0/1019110280 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4018c10 con 0x7f4ccc107730 2024-04-25T01:40:58.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.607+0000 7f4cd2329640 1 -- 172.21.15.28:0/1019110280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ccc1a4af0 con 0x7f4ccc107730 2024-04-25T01:40:58.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.609+0000 7f4cc97fa640 1 -- 172.21.15.28:0/1019110280 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f4cc400f5f0 con 0x7f4ccc107730 2024-04-25T01:40:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.610+0000 7f4cd503c740 1 Processor -- start 2024-04-25T01:40:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.610+0000 7f4cd503c740 1 -- start start 2024-04-25T01:40:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.610+0000 7f4cd503c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913580 0x55f469913960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.610+0000 7f4cd503c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 0x55f46991e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.610+0000 7f4cd503c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f46991e570 0x55f469920960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.610+0000 7f4cd503c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f469878670 con 0x55f469913ea0 2024-04-25T01:40:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.610+0000 7f4cd503c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f469820970 con 0x55f46991e570 2024-04-25T01:40:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.610+0000 7f4cd503c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55f469874630 con 0x55f469913580 2024-04-25T01:40:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.610+0000 7f4cc97fa640 1 --2- 172.21.15.28:0/1019110280 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4cb80a2d00 0x7f4cb80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.610+0000 7f4ccbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 0x55f46991e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.610+0000 7f4ccbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 0x55f46991e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48050/0 (socket says 172.21.15.28:48050) 2024-04-25T01:40:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.610+0000 7f4ccbfff640 1 -- 172.21.15.28:0/2597837282 learned_addr learned my addr 172.21.15.28:0/2597837282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.610+0000 7f4ccb7fe640 1 --2- 172.21.15.28:0/2597837282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f46991e570 0x55f469920960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.610+0000 7f4cd08c6640 1 --2- 172.21.15.28:0/1019110280 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4cb80a2d00 0x7f4cb80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.610+0000 7f4ccb7fe640 1 --2- 172.21.15.28:0/2597837282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913580 0x55f469913960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.610+0000 7f4cc97fa640 1 -- 172.21.15.28:0/1019110280 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f4cc40cff90 con 0x7f4ccc107730 2024-04-25T01:40:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.611+0000 7f4ccbfff640 1 -- 172.21.15.28:0/2597837282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913580 msgr2=0x55f469913960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.611+0000 7f4ccbfff640 1 --2- 172.21.15.28:0/2597837282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913580 0x55f469913960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.611+0000 7f4ccbfff640 1 -- 172.21.15.28:0/2597837282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f46991e570 msgr2=0x55f469920960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.611+0000 7f4ccbfff640 1 --2- 172.21.15.28:0/2597837282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f46991e570 0x55f469920960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.611+0000 7f4ccbfff640 1 -- 172.21.15.28:0/2597837282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f469844400 con 0x55f469913ea0 2024-04-25T01:40:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.611+0000 7f4ccb7fe640 1 --2- 172.21.15.28:0/2597837282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913580 0x55f469913960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.611+0000 7f4ccbfff640 1 --2- 172.21.15.28:0/2597837282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 0x55f46991e030 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc006010 tx=0x7f4cbc005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38829d3aac376152 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.611+0000 7f4cd08c6640 1 --2- 172.21.15.28:0/1019110280 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4cb80a2d00 0x7f4cb80a51c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4cc409bdc0 tx=0x7f4cc4091550 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.611+0000 7f4cc27fc640 1 -- 172.21.15.28:0/2597837282 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cbc03f070 con 0x55f469913ea0 2024-04-25T01:40:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.611+0000 7f4cc27fc640 1 -- 172.21.15.28:0/2597837282 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cbc004030 con 0x55f469913ea0 2024-04-25T01:40:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.611+0000 7f4cc27fc640 1 -- 172.21.15.28:0/2597837282 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cbc035070 con 0x55f469913ea0 2024-04-25T01:40:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.612+0000 7f4cd503c740 1 -- 172.21.15.28:0/2597837282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 msgr2=0x55f46991e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.612+0000 7f4cd503c740 1 --2- 172.21.15.28:0/2597837282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 0x55f46991e030 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc006010 tx=0x7f4cbc005e80 comp rx=0 tx=0).stop 2024-04-25T01:40:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.612+0000 7f4cd503c740 1 -- 172.21.15.28:0/2597837282 shutdown_connections 2024-04-25T01:40:58.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.612+0000 7f4cd503c740 1 --2- 172.21.15.28:0/2597837282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f46991e570 0x55f469920960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.612+0000 7f4cd503c740 1 --2- 172.21.15.28:0/2597837282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 0x55f46991e030 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.612+0000 7f4cd503c740 1 --2- 172.21.15.28:0/2597837282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913580 0x55f469913960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.612+0000 7f4cd503c740 1 -- 172.21.15.28:0/2597837282 >> 172.21.15.28:0/2597837282 conn(0x55f46990f8a0 msgr2=0x55f46990fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.612+0000 7f4cd503c740 1 -- 172.21.15.28:0/2597837282 shutdown_connections 2024-04-25T01:40:58.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.612+0000 7f4cd503c740 1 -- 172.21.15.28:0/2597837282 wait complete. 2024-04-25T01:40:58.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.612+0000 7f4cd503c740 1 -- start start 2024-04-25T01:40:58.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.612+0000 7f4cd503c740 1 Processor -- start 2024-04-25T01:40:58.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4cd503c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913580 0x55f46991df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4cd503c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 0x55f46991b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4cd503c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f46991e570 0x55f46991b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4cd503c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f469921300 con 0x55f469913ea0 2024-04-25T01:40:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4cd503c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f469834050 con 0x55f46991e570 2024-04-25T01:40:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4cd503c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55f469921480 con 0x55f469913580 2024-04-25T01:40:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4cd503c740 10 client.0 _ll_register_callbacks cb 0x55f46958bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4ccbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 0x55f46991b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4ccbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 0x55f46991b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48052/0 (socket says 172.21.15.28:48052) 2024-04-25T01:40:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4ccbfff640 1 -- 172.21.15.28:0/2500145240 learned_addr learned my addr 172.21.15.28:0/2500145240 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4ccb7fe640 1 --2- 172.21.15.28:0/2500145240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913580 0x55f46991df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4ccbfff640 1 -- 172.21.15.28:0/2500145240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913580 msgr2=0x55f46991df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4ccbfff640 1 --2- 172.21.15.28:0/2500145240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913580 0x55f46991df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4ccbfff640 1 -- 172.21.15.28:0/2500145240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f46991e570 msgr2=0x55f46991b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4ccbfff640 1 --2- 172.21.15.28:0/2500145240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f46991e570 0x55f46991b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4ccbfff640 1 -- 172.21.15.28:0/2500145240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f469805dc0 con 0x55f469913ea0 2024-04-25T01:40:58.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4ccb7fe640 1 --2- 172.21.15.28:0/2500145240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913580 0x55f46991df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:58.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4ccbfff640 1 --2- 172.21.15.28:0/2500145240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 0x55f46991b220 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc032fe0 tx=0x7f4cbc0053f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e27fd3e2e289d57 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4cab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:58.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4cab7fe640 1 -- 172.21.15.28:0/2500145240 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cbc035030 con 0x55f469913ea0 2024-04-25T01:40:58.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4cab7fe640 1 -- 172.21.15.28:0/2500145240 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f4cbc0315b0 con 0x55f469913ea0 2024-04-25T01:40:58.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4cd503c740 1 -- 172.21.15.28:0/2500145240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cbc004030 con 0x55f469913ea0 2024-04-25T01:40:58.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4cab7fe640 1 -- 172.21.15.28:0/2500145240 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4cbc055050 con 0x55f469913ea0 2024-04-25T01:40:58.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.613+0000 7f4cd503c740 10 client.18573 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:58.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.614+0000 7f4cd503c740 1 -- 172.21.15.28:0/2500145240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f46958c3f0 con 0x55f469913ea0 2024-04-25T01:40:58.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.614+0000 7f4cab7fe640 1 -- 172.21.15.28:0/2500145240 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cbc004030 con 0x55f469913ea0 2024-04-25T01:40:58.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.614+0000 7f4cab7fe640 1 -- 172.21.15.28:0/2500145240 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4cbc0046c0 con 0x55f469913ea0 2024-04-25T01:40:58.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.614+0000 7f4cd503c740 10 client.18573 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:58.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.614+0000 7f4cd503c740 10 client.18573 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:40:58.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.614+0000 7f4cd503c740 20 client.18573 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:58.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.614+0000 7f4cd503c740 1 --2- 172.21.15.28:0/2500145240 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f46992e450 0x55f469930840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.614+0000 7f4cd503c740 4 client.18573 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:58.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.614+0000 7f4cd503c740 1 -- 172.21.15.28:0/2500145240 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f469885bb0 con 0x55f46992e450 2024-04-25T01:40:58.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.614+0000 7f4ccb7fe640 1 --2- 172.21.15.28:0/2500145240 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f46992e450 0x55f469930840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.615+0000 7f4cab7fe640 10 client.18573 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:40:58.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.615+0000 7f4ccb7fe640 1 --2- 172.21.15.28:0/2500145240 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f46992e450 0x55f469930840 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fb7cab577dbb21e4 server_cookie=4351f602e695afdf in_seq=0 out_seq=0 2024-04-25T01:40:58.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.615+0000 7f4cab7fe640 1 -- 172.21.15.28:0/2500145240 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f469885bb0 con 0x55f46992e450 2024-04-25T01:40:58.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.615+0000 7f4cab7fe640 10 client.18573 handle_command_reply: tid=0 2024-04-25T01:40:58.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.615+0000 7f4cd503c740 1 client.18573 shutdown 2024-04-25T01:40:58.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.616+0000 7f4cd503c740 1 -- 172.21.15.28:0/2500145240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 msgr2=0x55f46991b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.616+0000 7f4cd503c740 1 --2- 172.21.15.28:0/2500145240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 0x55f46991b220 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc032fe0 tx=0x7f4cbc0053f0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.616+0000 7f4cd503c740 1 -- 172.21.15.28:0/2500145240 shutdown_connections 2024-04-25T01:40:58.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.616+0000 7f4cd503c740 1 --2- 172.21.15.28:0/2500145240 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f46992e450 0x55f469930840 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.616+0000 7f4cd503c740 1 --2- 172.21.15.28:0/2500145240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f46991e570 0x55f46991b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.616+0000 7f4cab7fe640 0 client.18573 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:40:58.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.616+0000 7f4cd503c740 1 --2- 172.21.15.28:0/2500145240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 0x55f46991b220 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.616+0000 7f4cd503c740 1 --2- 172.21.15.28:0/2500145240 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913580 0x55f46991df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.616+0000 7f4cd503c740 1 -- 172.21.15.28:0/2500145240 >> 172.21.15.28:0/2500145240 conn(0x55f46990f8a0 msgr2=0x55f4699156a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.616+0000 7f4cd503c740 1 -- 172.21.15.28:0/2500145240 shutdown_connections 2024-04-25T01:40:58.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.616+0000 7f4cd503c740 1 -- 172.21.15.28:0/2500145240 wait complete. 2024-04-25T01:40:58.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.616+0000 7f4cd503c740 20 client.18573 trim_cache size 0 max 16384 2024-04-25T01:40:58.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.629+0000 7f4cd503c740 1 Processor -- start 2024-04-25T01:40:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.629+0000 7f4cd503c740 1 -- start start 2024-04-25T01:40:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.629+0000 7f4cd503c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913580 0x55f46991da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.629+0000 7f4cd503c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 0x55f4699266d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.629+0000 7f4cd503c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f46991e570 0x55f469926c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.629+0000 7f4cd503c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc1aeb40 con 0x55f469913ea0 2024-04-25T01:40:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.629+0000 7f4cd503c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc1b0680 con 0x55f46991e570 2024-04-25T01:40:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.629+0000 7f4cd503c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc1af210 con 0x55f469913580 2024-04-25T01:40:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.629+0000 7f4ccbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 0x55f4699266d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.629+0000 7f4ccbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 0x55f4699266d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48060/0 (socket says 172.21.15.28:48060) 2024-04-25T01:40:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.629+0000 7f4ccbfff640 1 -- 172.21.15.28:0/152356980 learned_addr learned my addr 172.21.15.28:0/152356980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.629+0000 7f4ccb7fe640 1 --2- 172.21.15.28:0/152356980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f46991e570 0x55f469926c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.629+0000 7f4ccbfff640 1 -- 172.21.15.28:0/152356980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913580 msgr2=0x55f46991da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.629+0000 7f4ccbfff640 1 --2- 172.21.15.28:0/152356980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913580 0x55f46991da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.629+0000 7f4ccbfff640 1 -- 172.21.15.28:0/152356980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f46991e570 msgr2=0x55f469926c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.629+0000 7f4ccbfff640 1 --2- 172.21.15.28:0/152356980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f46991e570 0x55f469926c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.630+0000 7f4ccbfff640 1 -- 172.21.15.28:0/152356980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4696567d0 con 0x55f469913ea0 2024-04-25T01:40:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.630+0000 7f4ccbfff640 1 --2- 172.21.15.28:0/152356980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 0x55f4699266d0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc000c00 tx=0x7f4cbc005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6fc778396259f66 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.630+0000 7f4cc1ffb640 1 -- 172.21.15.28:0/152356980 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cbc05e070 con 0x55f469913ea0 2024-04-25T01:40:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.630+0000 7f4cc1ffb640 1 -- 172.21.15.28:0/152356980 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cbc04c070 con 0x55f469913ea0 2024-04-25T01:40:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.630+0000 7f4cc1ffb640 1 -- 172.21.15.28:0/152356980 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cbc035040 con 0x55f469913ea0 2024-04-25T01:40:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.630+0000 7f4cd503c740 1 -- 172.21.15.28:0/152356980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 msgr2=0x55f4699266d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.630+0000 7f4cd503c740 1 --2- 172.21.15.28:0/152356980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 0x55f4699266d0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc000c00 tx=0x7f4cbc005e80 comp rx=0 tx=0).stop 2024-04-25T01:40:58.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.630+0000 7f4cd503c740 1 -- 172.21.15.28:0/152356980 shutdown_connections 2024-04-25T01:40:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.630+0000 7f4cd503c740 1 --2- 172.21.15.28:0/152356980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f46991e570 0x55f469926c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.630+0000 7f4cd503c740 1 --2- 172.21.15.28:0/152356980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f469913ea0 0x55f4699266d0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.630+0000 7f4cd503c740 1 --2- 172.21.15.28:0/152356980 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913580 0x55f46991da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.630+0000 7f4cd503c740 1 -- 172.21.15.28:0/152356980 >> 172.21.15.28:0/152356980 conn(0x55f46990f8a0 msgr2=0x55f4699291b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.630+0000 7f4cd503c740 1 -- 172.21.15.28:0/152356980 shutdown_connections 2024-04-25T01:40:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.630+0000 7f4cd503c740 1 -- 172.21.15.28:0/152356980 wait complete. 2024-04-25T01:40:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.631+0000 7f4cd503c740 1 -- start start 2024-04-25T01:40:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.631+0000 7f4cd503c740 1 Processor -- start 2024-04-25T01:40:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.631+0000 7f4cd503c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f469913580 0x55f469926390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.631+0000 7f4cd503c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913ea0 0x55f4699268d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.631+0000 7f4cd503c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f46991e570 0x55f469924830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.631+0000 7f4cd503c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc1affb0 con 0x55f46991e570 2024-04-25T01:40:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.631+0000 7f4cd503c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc1af8e0 con 0x55f469913580 2024-04-25T01:40:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.631+0000 7f4cd503c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4ccc1acb40 con 0x55f469913ea0 2024-04-25T01:40:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.631+0000 7f4cd503c740 10 client.0 _ll_register_callbacks cb 0x7f4cb400c9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.631+0000 7f4ccb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f46991e570 0x55f469924830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.631+0000 7f4ccb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f46991e570 0x55f469924830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48062/0 (socket says 172.21.15.28:48062) 2024-04-25T01:40:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.631+0000 7f4ccb7fe640 1 -- 172.21.15.28:0/2445324699 learned_addr learned my addr 172.21.15.28:0/2445324699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.631+0000 7f4ccbfff640 1 --2- 172.21.15.28:0/2445324699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913ea0 0x55f4699268d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.631+0000 7f4ccb7fe640 1 --2- 172.21.15.28:0/2445324699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f469913580 0x55f469926390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.632+0000 7f4ccb7fe640 1 -- 172.21.15.28:0/2445324699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913ea0 msgr2=0x55f4699268d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.632+0000 7f4ccb7fe640 1 --2- 172.21.15.28:0/2445324699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913ea0 0x55f4699268d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.632+0000 7f4ccb7fe640 1 -- 172.21.15.28:0/2445324699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f469913580 msgr2=0x55f469926390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.632+0000 7f4ccb7fe640 1 --2- 172.21.15.28:0/2445324699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f469913580 0x55f469926390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.632+0000 7f4ccb7fe640 1 -- 172.21.15.28:0/2445324699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f469924fc0 con 0x55f46991e570 2024-04-25T01:40:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.632+0000 7f4ccb7fe640 1 --2- 172.21.15.28:0/2445324699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f46991e570 0x55f469924830 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400d010 tx=0x7f4cb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a91cfd1ab80a0bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.632+0000 7f4cab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.632+0000 7f4cab7fe640 1 -- 172.21.15.28:0/2445324699 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4004230 con 0x55f46991e570 2024-04-25T01:40:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.632+0000 7f4cd503c740 1 -- 172.21.15.28:0/2445324699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cbc04c070 con 0x55f46991e570 2024-04-25T01:40:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.632+0000 7f4cab7fe640 1 -- 172.21.15.28:0/2445324699 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f4cb40043d0 con 0x55f46991e570 2024-04-25T01:40:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.632+0000 7f4cab7fe640 1 -- 172.21.15.28:0/2445324699 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4cb4013740 con 0x55f46991e570 2024-04-25T01:40:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.632+0000 7f4cd503c740 10 client.18579 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.632+0000 7f4cd503c740 1 -- 172.21.15.28:0/2445324699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f469995250 con 0x55f46991e570 2024-04-25T01:40:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.632+0000 7f4cab7fe640 1 -- 172.21.15.28:0/2445324699 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cbc04c070 con 0x55f46991e570 2024-04-25T01:40:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.632+0000 7f4cab7fe640 1 -- 172.21.15.28:0/2445324699 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4cb4003c70 con 0x55f46991e570 2024-04-25T01:40:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.633+0000 7f4cd503c740 10 client.18579 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.633+0000 7f4cd503c740 10 client.18579 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:40:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.633+0000 7f4cd503c740 20 client.18579 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.633+0000 7f4cd503c740 1 --2- 172.21.15.28:0/2445324699 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f469995980 0x55f469997da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.633+0000 7f4cd503c740 4 client.18579 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.633+0000 7f4cd503c740 1 -- 172.21.15.28:0/2445324699 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f46987ff00 con 0x55f469995980 2024-04-25T01:40:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.633+0000 7f4ccbfff640 1 --2- 172.21.15.28:0/2445324699 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f469995980 0x55f469997da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.633+0000 7f4cab7fe640 10 client.18579 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:40:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.633+0000 7f4ccbfff640 1 --2- 172.21.15.28:0/2445324699 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f469995980 0x55f469997da0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ba44cb960c13679d server_cookie=e1da226368aa536b in_seq=0 out_seq=0 2024-04-25T01:40:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.633+0000 7f4cab7fe640 1 -- 172.21.15.28:0/2445324699 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f46987ff00 con 0x55f469995980 2024-04-25T01:40:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.633+0000 7f4cab7fe640 10 client.18579 handle_command_reply: tid=0 2024-04-25T01:40:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.633+0000 7f4cd503c740 1 client.18579 shutdown 2024-04-25T01:40:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd503c740 1 -- 172.21.15.28:0/2445324699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f46991e570 msgr2=0x55f469924830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd503c740 1 --2- 172.21.15.28:0/2445324699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f46991e570 0x55f469924830 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400d010 tx=0x7f4cb40046c0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd503c740 1 -- 172.21.15.28:0/2445324699 shutdown_connections 2024-04-25T01:40:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd503c740 1 --2- 172.21.15.28:0/2445324699 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55f469995980 0x55f469997da0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd503c740 1 --2- 172.21.15.28:0/2445324699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f46991e570 0x55f469924830 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd503c740 1 --2- 172.21.15.28:0/2445324699 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55f469913ea0 0x55f4699268d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cab7fe640 0 client.18579 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:40:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd503c740 1 --2- 172.21.15.28:0/2445324699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f469913580 0x55f469926390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd503c740 1 -- 172.21.15.28:0/2445324699 >> 172.21.15.28:0/2445324699 conn(0x55f46990f8a0 msgr2=0x55f469929a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd503c740 1 -- 172.21.15.28:0/2445324699 shutdown_connections 2024-04-25T01:40:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd503c740 1 -- 172.21.15.28:0/2445324699 wait complete. 2024-04-25T01:40:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd503c740 20 client.18579 trim_cache size 0 max 16384 2024-04-25T01:40:58.629 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:58.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd2329640 1 -- 172.21.15.28:0/1019110280 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4cb80a2d00 msgr2=0x7f4cb80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd2329640 1 --2- 172.21.15.28:0/1019110280 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4cb80a2d00 0x7f4cb80a51c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4cc409bdc0 tx=0x7f4cc4091550 comp rx=0 tx=0).stop 2024-04-25T01:40:58.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd2329640 1 -- 172.21.15.28:0/1019110280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc107730 msgr2=0x7f4ccc19c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd2329640 1 --2- 172.21.15.28:0/1019110280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc107730 0x7f4ccc19c840 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400cce0 tx=0x7f4cc400a620 comp rx=0 tx=0).stop 2024-04-25T01:40:58.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd2329640 1 -- 172.21.15.28:0/1019110280 shutdown_connections 2024-04-25T01:40:58.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd2329640 1 --2- 172.21.15.28:0/1019110280 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4cb80a2d00 0x7f4cb80a51c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd2329640 1 --2- 172.21.15.28:0/1019110280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ccc107730 0x7f4ccc19c840 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd2329640 1 --2- 172.21.15.28:0/1019110280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ccc06a780 0x7f4ccc1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd2329640 1 --2- 172.21.15.28:0/1019110280 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4ccc069dc0 0x7f4ccc197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd2329640 1 -- 172.21.15.28:0/1019110280 >> 172.21.15.28:0/1019110280 conn(0x7f4ccc0776a0 msgr2=0x7f4ccc0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd2329640 1 -- 172.21.15.28:0/1019110280 shutdown_connections 2024-04-25T01:40:58.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.634+0000 7f4cd2329640 1 -- 172.21.15.28:0/1019110280 wait complete. 2024-04-25T01:40:58.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:40:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.749+0000 7f0f0869a640 1 -- 172.21.15.28:0/314733734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f00101330 msgr2=0x7f0f0010d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.749+0000 7f0f0869a640 1 --2- 172.21.15.28:0/314733734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f00101330 0x7f0f0010d460 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8009ca0 tx=0x7f0ef802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.749+0000 7f0f0869a640 1 -- 172.21.15.28:0/314733734 shutdown_connections 2024-04-25T01:40:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.749+0000 7f0f0869a640 1 --2- 172.21.15.28:0/314733734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f00101330 0x7f0f0010d460 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.749+0000 7f0f0869a640 1 --2- 172.21.15.28:0/314733734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f00100990 0x7f0f00100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.750+0000 7f0f0869a640 1 --2- 172.21.15.28:0/314733734 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f00106690 0x7f0f00106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.750+0000 7f0f0869a640 1 -- 172.21.15.28:0/314733734 >> 172.21.15.28:0/314733734 conn(0x7f0f00076440 msgr2=0x7f0f00076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.750+0000 7f0f0869a640 1 -- 172.21.15.28:0/314733734 shutdown_connections 2024-04-25T01:40:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.750+0000 7f0f0869a640 1 -- 172.21.15.28:0/314733734 wait complete. 2024-04-25T01:40:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.750+0000 7f0f0869a640 1 Processor -- start 2024-04-25T01:40:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.750+0000 7f0f0869a640 1 -- start start 2024-04-25T01:40:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.750+0000 7f0f0869a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f00100990 0x7f0f00197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.750+0000 7f0f0869a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f00101330 0x7f0f00198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.750+0000 7f0f0869a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f00106690 0x7f0f0019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.750+0000 7f0f0869a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f00111b20 con 0x7f0f00100990 2024-04-25T01:40:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.750+0000 7f0f0869a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0f001119a0 con 0x7f0f00106690 2024-04-25T01:40:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.750+0000 7f0f0869a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0f00111ca0 con 0x7f0f00101330 2024-04-25T01:40:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.750+0000 7f0f06436640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f00100990 0x7f0f00197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.750+0000 7f0f06436640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f00100990 0x7f0f00197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48072/0 (socket says 172.21.15.28:48072) 2024-04-25T01:40:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.750+0000 7f0f06436640 1 -- 172.21.15.28:0/4119006882 learned_addr learned my addr 172.21.15.28:0/4119006882 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.751+0000 7f0f06c37640 1 --2- 172.21.15.28:0/4119006882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f00106690 0x7f0f0019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.751+0000 7f0f05c35640 1 --2- 172.21.15.28:0/4119006882 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f00101330 0x7f0f00198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.751+0000 7f0f06436640 1 -- 172.21.15.28:0/4119006882 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f00101330 msgr2=0x7f0f00198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.751+0000 7f0f06436640 1 --2- 172.21.15.28:0/4119006882 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f00101330 0x7f0f00198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.751+0000 7f0f06436640 1 -- 172.21.15.28:0/4119006882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f00106690 msgr2=0x7f0f0019c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.751+0000 7f0f06436640 1 --2- 172.21.15.28:0/4119006882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f00106690 0x7f0f0019c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.751+0000 7f0f06436640 1 -- 172.21.15.28:0/4119006882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0019ceb0 con 0x7f0f00100990 2024-04-25T01:40:58.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.751+0000 7f0f05c35640 1 --2- 172.21.15.28:0/4119006882 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f00101330 0x7f0f00198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:58.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.751+0000 7f0f06436640 1 --2- 172.21.15.28:0/4119006882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f00100990 0x7f0f00197ec0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f0efc00c970 tx=0x7f0efc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.751+0000 7f0ef77fe640 1 -- 172.21.15.28:0/4119006882 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0efc007bb0 con 0x7f0f00100990 2024-04-25T01:40:58.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.751+0000 7f0ef77fe640 1 -- 172.21.15.28:0/4119006882 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0efc007d50 con 0x7f0f00100990 2024-04-25T01:40:58.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.751+0000 7f0f0869a640 1 -- 172.21.15.28:0/4119006882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f0019d1a0 con 0x7f0f00100990 2024-04-25T01:40:58.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.751+0000 7f0ef77fe640 1 -- 172.21.15.28:0/4119006882 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0efc00f450 con 0x7f0f00100990 2024-04-25T01:40:58.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.751+0000 7f0f0869a640 1 -- 172.21.15.28:0/4119006882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f001a4a30 con 0x7f0f00100990 2024-04-25T01:40:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.754+0000 7f0ef77fe640 1 -- 172.21.15.28:0/4119006882 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0efc005020 con 0x7f0f00100990 2024-04-25T01:40:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.754+0000 7f0f0b3a5740 1 Processor -- start 2024-04-25T01:40:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.754+0000 7f0f0b3a5740 1 -- start start 2024-04-25T01:40:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.754+0000 7f0f0b3a5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af0d480 0x55d36af0d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.754+0000 7f0f0b3a5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0dda0 0x55d36af1c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.754+0000 7f0f0b3a5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 0x55d36af1ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.755+0000 7f0f0b3a5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d36ae72670 con 0x55d36af1c980 2024-04-25T01:40:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.755+0000 7f0f0b3a5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d36ae1a970 con 0x55d36af0d480 2024-04-25T01:40:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.755+0000 7f0f0b3a5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55d36ae6e630 con 0x55d36af0dda0 2024-04-25T01:40:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.755+0000 7f0f05c35640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 0x55d36af1ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.755+0000 7f0f05c35640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 0x55d36af1ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48076/0 (socket says 172.21.15.28:48076) 2024-04-25T01:40:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.755+0000 7f0f05c35640 1 -- 172.21.15.28:0/248639665 learned_addr learned my addr 172.21.15.28:0/248639665 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.755+0000 7f0f06436640 1 --2- 172.21.15.28:0/248639665 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0dda0 0x55d36af1c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.755+0000 7f0f06c37640 1 --2- 172.21.15.28:0/248639665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af0d480 0x55d36af0d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.755+0000 7f0ef77fe640 1 --2- 172.21.15.28:0/4119006882 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0eec0a2d80 0x7f0eec0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.755+0000 7f0f06c37640 1 --2- 172.21.15.28:0/4119006882 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0eec0a2d80 0x7f0eec0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.755+0000 7f0f05c35640 1 -- 172.21.15.28:0/248639665 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0dda0 msgr2=0x55d36af1c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.755+0000 7f0f05c35640 1 --2- 172.21.15.28:0/248639665 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0dda0 0x55d36af1c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.755+0000 7f0f05c35640 1 -- 172.21.15.28:0/248639665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af0d480 msgr2=0x55d36af0d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.755+0000 7f0f05c35640 1 --2- 172.21.15.28:0/248639665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af0d480 0x55d36af0d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.755+0000 7f0ef77fe640 1 -- 172.21.15.28:0/4119006882 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f0efc0cf5b0 con 0x7f0f00100990 2024-04-25T01:40:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.755+0000 7f0f05c35640 1 -- 172.21.15.28:0/248639665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d36ae3e400 con 0x55d36af1c980 2024-04-25T01:40:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.755+0000 7f0f05c35640 1 --2- 172.21.15.28:0/248639665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 0x55d36af1ed70 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000bd30 tx=0x7f0ef000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b15cd3efc7165a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.755+0000 7f0f06c37640 1 --2- 172.21.15.28:0/4119006882 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0eec0a2d80 0x7f0eec0a5240 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8006010 tx=0x7f0ef802a8c0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.756+0000 7f0ef4ff9640 1 -- 172.21.15.28:0/248639665 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0015070 con 0x55d36af1c980 2024-04-25T01:40:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.756+0000 7f0ef4ff9640 1 -- 172.21.15.28:0/248639665 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ef000be20 con 0x55d36af1c980 2024-04-25T01:40:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.756+0000 7f0ef4ff9640 1 -- 172.21.15.28:0/248639665 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef00107f0 con 0x55d36af1c980 2024-04-25T01:40:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.756+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/248639665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 msgr2=0x55d36af1ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.756+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/248639665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 0x55d36af1ed70 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000bd30 tx=0x7f0ef000f730 comp rx=0 tx=0).stop 2024-04-25T01:40:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.756+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/248639665 shutdown_connections 2024-04-25T01:40:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.756+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/248639665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 0x55d36af1ed70 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.756+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/248639665 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0dda0 0x55d36af1c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.756+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/248639665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af0d480 0x55d36af0d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.756+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/248639665 >> 172.21.15.28:0/248639665 conn(0x55d36af098a0 msgr2=0x55d36af09c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.756+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/248639665 shutdown_connections 2024-04-25T01:40:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.756+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/248639665 wait complete. 2024-04-25T01:40:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.757+0000 7f0f0b3a5740 1 -- start start 2024-04-25T01:40:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.757+0000 7f0f0b3a5740 1 Processor -- start 2024-04-25T01:40:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.757+0000 7f0f0b3a5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0d480 0x55d36af1c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.757+0000 7f0f0b3a5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af0dda0 0x55d36af19580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.757+0000 7f0f0b3a5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 0x55d36af19ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.757+0000 7f0f0b3a5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d36af1f740 con 0x55d36af1c980 2024-04-25T01:40:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.757+0000 7f0f0b3a5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d36ae2e050 con 0x55d36af0dda0 2024-04-25T01:40:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.757+0000 7f0f0b3a5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55d36af1f8c0 con 0x55d36af0d480 2024-04-25T01:40:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.757+0000 7f0f0b3a5740 10 client.0 _ll_register_callbacks cb 0x55d36ab85be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.757+0000 7f0f06c37640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 0x55d36af19ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.757+0000 7f0f06c37640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 0x55d36af19ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48080/0 (socket says 172.21.15.28:48080) 2024-04-25T01:40:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.757+0000 7f0f06c37640 1 -- 172.21.15.28:0/3445241563 learned_addr learned my addr 172.21.15.28:0/3445241563 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.757+0000 7f0f05c35640 1 --2- 172.21.15.28:0/3445241563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af0dda0 0x55d36af19580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.757+0000 7f0f06436640 1 --2- 172.21.15.28:0/3445241563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0d480 0x55d36af1c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.758+0000 7f0f06c37640 1 -- 172.21.15.28:0/3445241563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0d480 msgr2=0x55d36af1c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.758+0000 7f0f06c37640 1 --2- 172.21.15.28:0/3445241563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0d480 0x55d36af1c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.758+0000 7f0f06c37640 1 -- 172.21.15.28:0/3445241563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af0dda0 msgr2=0x55d36af19580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.758+0000 7f0f06c37640 1 --2- 172.21.15.28:0/3445241563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af0dda0 0x55d36af19580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.758+0000 7f0f06c37640 1 -- 172.21.15.28:0/3445241563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d36adffdc0 con 0x55d36af1c980 2024-04-25T01:40:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.758+0000 7f0f05c35640 1 --2- 172.21.15.28:0/3445241563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af0dda0 0x55d36af19580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:40:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.758+0000 7f0f06c37640 1 --2- 172.21.15.28:0/3445241563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 0x55d36af19ac0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8035000 tx=0x7f0ef80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856f376e7b85e5db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.758+0000 7f0ee9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.758+0000 7f0ee9ffb640 1 -- 172.21.15.28:0/3445241563 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef804f070 con 0x55d36af1c980 2024-04-25T01:40:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.758+0000 7f0ee9ffb640 1 -- 172.21.15.28:0/3445241563 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f0ef80305c0 con 0x55d36af1c980 2024-04-25T01:40:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.758+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3445241563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ef000be20 con 0x55d36af1c980 2024-04-25T01:40:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.758+0000 7f0ee9ffb640 1 -- 172.21.15.28:0/3445241563 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0ef805b050 con 0x55d36af1c980 2024-04-25T01:40:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.758+0000 7f0f0b3a5740 10 client.18591 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.758+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3445241563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d36ab863f0 con 0x55d36af1c980 2024-04-25T01:40:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.758+0000 7f0ee9ffb640 1 -- 172.21.15.28:0/3445241563 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ef000be20 con 0x55d36af1c980 2024-04-25T01:40:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.758+0000 7f0ee9ffb640 1 -- 172.21.15.28:0/3445241563 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0ef804a040 con 0x55d36af1c980 2024-04-25T01:40:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.758+0000 7f0f0b3a5740 10 client.18591 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.758+0000 7f0f0b3a5740 10 client.18591 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:40:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.759+0000 7f0f0b3a5740 20 client.18591 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.759+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/3445241563 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55d36af2c930 0x55d36af2ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.759+0000 7f0f0b3a5740 4 client.18591 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.759+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3445241563 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d36ae7fbb0 con 0x55d36af2c930 2024-04-25T01:40:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.759+0000 7f0f06436640 1 --2- 172.21.15.28:0/3445241563 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55d36af2c930 0x55d36af2ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.759+0000 7f0ee9ffb640 10 client.18591 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:40:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.759+0000 7f0f06436640 1 --2- 172.21.15.28:0/3445241563 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55d36af2c930 0x55d36af2ed80 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5d9d1e1df5d5754b server_cookie=929ef6663a0c0f12 in_seq=0 out_seq=0 2024-04-25T01:40:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.760+0000 7f0ee9ffb640 1 -- 172.21.15.28:0/3445241563 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d36ae7fbb0 con 0x55d36af2c930 2024-04-25T01:40:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.760+0000 7f0ee9ffb640 10 client.18591 handle_command_reply: tid=0 2024-04-25T01:40:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.760+0000 7f0f0b3a5740 1 client.18591 shutdown 2024-04-25T01:40:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.760+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3445241563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 msgr2=0x55d36af19ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.760+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/3445241563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 0x55d36af19ac0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8035000 tx=0x7f0ef80095c0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.760+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3445241563 shutdown_connections 2024-04-25T01:40:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.760+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/3445241563 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55d36af2c930 0x55d36af2ed80 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.760+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/3445241563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 0x55d36af19ac0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.760+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/3445241563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af0dda0 0x55d36af19580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.760+0000 7f0ee9ffb640 0 client.18591 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:40:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.760+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/3445241563 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0d480 0x55d36af1c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.760+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3445241563 >> 172.21.15.28:0/3445241563 conn(0x55d36af098a0 msgr2=0x55d36af0f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.760+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3445241563 shutdown_connections 2024-04-25T01:40:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.760+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3445241563 wait complete. 2024-04-25T01:40:58.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.760+0000 7f0f0b3a5740 20 client.18591 trim_cache size 0 max 16384 2024-04-25T01:40:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.773+0000 7f0f0b3a5740 1 Processor -- start 2024-04-25T01:40:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.773+0000 7f0f0b3a5740 1 -- start start 2024-04-25T01:40:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0f0b3a5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af0d480 0x55d36af14c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0f0b3a5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0dda0 0x55d36af1bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0f0b3a5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 0x55d36af0f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0f0b3a5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f001ac8a0 con 0x55d36af1c980 2024-04-25T01:40:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0f0b3a5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0f001ae3e0 con 0x55d36af0d480 2024-04-25T01:40:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0f0b3a5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0f001acf70 con 0x55d36af0dda0 2024-04-25T01:40:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0f06c37640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 0x55d36af0f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0f06c37640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 0x55d36af0f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48082/0 (socket says 172.21.15.28:48082) 2024-04-25T01:40:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0f06c37640 1 -- 172.21.15.28:0/3590292789 learned_addr learned my addr 172.21.15.28:0/3590292789 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0f06436640 1 --2- 172.21.15.28:0/3590292789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af0d480 0x55d36af14c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0f05c35640 1 --2- 172.21.15.28:0/3590292789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0dda0 0x55d36af1bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0f06c37640 1 -- 172.21.15.28:0/3590292789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0dda0 msgr2=0x55d36af1bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0f06c37640 1 --2- 172.21.15.28:0/3590292789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0dda0 0x55d36af1bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0f06c37640 1 -- 172.21.15.28:0/3590292789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af0d480 msgr2=0x55d36af14c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0f06c37640 1 --2- 172.21.15.28:0/3590292789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af0d480 0x55d36af14c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0f06c37640 1 -- 172.21.15.28:0/3590292789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d36ac507d0 con 0x55d36af1c980 2024-04-25T01:40:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0f05c35640 1 --2- 172.21.15.28:0/3590292789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0dda0 0x55d36af1bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0f06c37640 1 --2- 172.21.15.28:0/3590292789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 0x55d36af0f480 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f0ef802ade0 tx=0x7f0ef8005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5fdde72c43a923c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0eebfff640 1 -- 172.21.15.28:0/3590292789 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8058070 con 0x55d36af1c980 2024-04-25T01:40:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0eebfff640 1 -- 172.21.15.28:0/3590292789 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ef8036050 con 0x55d36af1c980 2024-04-25T01:40:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.774+0000 7f0eebfff640 1 -- 172.21.15.28:0/3590292789 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef804f070 con 0x55d36af1c980 2024-04-25T01:40:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.775+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3590292789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 msgr2=0x55d36af0f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.775+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/3590292789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 0x55d36af0f480 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f0ef802ade0 tx=0x7f0ef8005bd0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.775+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3590292789 shutdown_connections 2024-04-25T01:40:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.775+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/3590292789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af1c980 0x55d36af0f480 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.775+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/3590292789 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0dda0 0x55d36af1bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.775+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/3590292789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af0d480 0x55d36af14c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.775+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3590292789 >> 172.21.15.28:0/3590292789 conn(0x55d36af098a0 msgr2=0x55d36af0ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.775+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3590292789 shutdown_connections 2024-04-25T01:40:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.775+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3590292789 wait complete. 2024-04-25T01:40:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.775+0000 7f0f0b3a5740 1 -- start start 2024-04-25T01:40:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.775+0000 7f0f0b3a5740 1 Processor -- start 2024-04-25T01:40:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f0b3a5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af0d480 0x55d36af24b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f0b3a5740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0dda0 0x55d36af25090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f0b3a5740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af1c980 0x55d36af15c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f0b3a5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f001add10 con 0x55d36af0d480 2024-04-25T01:40:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f0b3a5740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0f001ad640 con 0x55d36af1c980 2024-04-25T01:40:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f0b3a5740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0f001aa8a0 con 0x55d36af0dda0 2024-04-25T01:40:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f0b3a5740 10 client.0 _ll_register_callbacks cb 0x7f0ef8031830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f06436640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af0d480 0x55d36af24b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f06436640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af0d480 0x55d36af24b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48098/0 (socket says 172.21.15.28:48098) 2024-04-25T01:40:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f06436640 1 -- 172.21.15.28:0/3661145264 learned_addr learned my addr 172.21.15.28:0/3661145264 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f06c37640 1 --2- 172.21.15.28:0/3661145264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af1c980 0x55d36af15c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f05c35640 1 --2- 172.21.15.28:0/3661145264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0dda0 0x55d36af25090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f06436640 1 -- 172.21.15.28:0/3661145264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0dda0 msgr2=0x55d36af25090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f06436640 1 --2- 172.21.15.28:0/3661145264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0dda0 0x55d36af25090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f06436640 1 -- 172.21.15.28:0/3661145264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af1c980 msgr2=0x55d36af15c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f06436640 1 --2- 172.21.15.28:0/3661145264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af1c980 0x55d36af15c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f06436640 1 -- 172.21.15.28:0/3661145264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d36af16430 con 0x55d36af0d480 2024-04-25T01:40:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f06436640 1 --2- 172.21.15.28:0/3661145264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af0d480 0x55d36af24b50 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f0efc09b3e0 tx=0x7f0efc0d1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd74e82396995c87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0ee9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0ee9ffb640 1 -- 172.21.15.28:0/3661145264 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0efc01d040 con 0x55d36af0d480 2024-04-25T01:40:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0ee9ffb640 1 -- 172.21.15.28:0/3661145264 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f0efc097d50 con 0x55d36af0d480 2024-04-25T01:40:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3661145264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ef8036050 con 0x55d36af0d480 2024-04-25T01:40:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0ee9ffb640 1 -- 172.21.15.28:0/3661145264 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0efc09b040 con 0x55d36af0d480 2024-04-25T01:40:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f0b3a5740 10 client.18597 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.776+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3661145264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d36af92970 con 0x55d36af0d480 2024-04-25T01:40:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.777+0000 7f0ee9ffb640 1 -- 172.21.15.28:0/3661145264 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ef8036050 con 0x55d36af0d480 2024-04-25T01:40:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.777+0000 7f0ee9ffb640 1 -- 172.21.15.28:0/3661145264 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0efc005310 con 0x55d36af0d480 2024-04-25T01:40:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.777+0000 7f0f0b3a5740 10 client.18597 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.777+0000 7f0f0b3a5740 10 client.18597 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:40:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.777+0000 7f0f0b3a5740 20 client.18597 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:58.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.777+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/3661145264 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55d36af93040 0x55d36af95460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.777+0000 7f0f0b3a5740 4 client.18597 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.777+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3661145264 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d36ae79f00 con 0x55d36af93040 2024-04-25T01:40:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.777+0000 7f0f05c35640 1 --2- 172.21.15.28:0/3661145264 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55d36af93040 0x55d36af95460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.777+0000 7f0ee9ffb640 10 client.18597 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:40:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.777+0000 7f0f05c35640 1 --2- 172.21.15.28:0/3661145264 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55d36af93040 0x55d36af95460 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=eabc8ba1d5f805bf server_cookie=37464dc9f4219663 in_seq=0 out_seq=0 2024-04-25T01:40:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0ee9ffb640 1 -- 172.21.15.28:0/3661145264 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d36ae79f00 con 0x55d36af93040 2024-04-25T01:40:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0ee9ffb640 10 client.18597 handle_command_reply: tid=0 2024-04-25T01:40:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0b3a5740 1 client.18597 shutdown 2024-04-25T01:40:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3661145264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af0d480 msgr2=0x55d36af24b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/3661145264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af0d480 0x55d36af24b50 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f0efc09b3e0 tx=0x7f0efc0d1040 comp rx=0 tx=0).stop 2024-04-25T01:40:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3661145264 shutdown_connections 2024-04-25T01:40:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/3661145264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d36af1c980 0x55d36af15c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/3661145264 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d36af0dda0 0x55d36af25090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/3661145264 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55d36af93040 0x55d36af95460 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0b3a5740 1 --2- 172.21.15.28:0/3661145264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36af0d480 0x55d36af24b50 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0ee9ffb640 0 client.18597 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:40:58.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3661145264 >> 172.21.15.28:0/3661145264 conn(0x55d36af098a0 msgr2=0x55d36af14af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3661145264 shutdown_connections 2024-04-25T01:40:58.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0b3a5740 1 -- 172.21.15.28:0/3661145264 wait complete. 2024-04-25T01:40:58.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0b3a5740 20 client.18597 trim_cache size 0 max 16384 2024-04-25T01:40:58.771 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:58.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0869a640 1 -- 172.21.15.28:0/4119006882 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0eec0a2d80 msgr2=0x7f0eec0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0869a640 1 --2- 172.21.15.28:0/4119006882 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0eec0a2d80 0x7f0eec0a5240 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8006010 tx=0x7f0ef802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0869a640 1 -- 172.21.15.28:0/4119006882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f00100990 msgr2=0x7f0f00197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0869a640 1 --2- 172.21.15.28:0/4119006882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f00100990 0x7f0f00197ec0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f0efc00c970 tx=0x7f0efc00ce30 comp rx=0 tx=0).stop 2024-04-25T01:40:58.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0869a640 1 -- 172.21.15.28:0/4119006882 shutdown_connections 2024-04-25T01:40:58.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0869a640 1 --2- 172.21.15.28:0/4119006882 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0eec0a2d80 0x7f0eec0a5240 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0869a640 1 --2- 172.21.15.28:0/4119006882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0f00106690 0x7f0f0019c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0869a640 1 --2- 172.21.15.28:0/4119006882 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0f00101330 0x7f0f00198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0869a640 1 --2- 172.21.15.28:0/4119006882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f00100990 0x7f0f00197ec0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.778+0000 7f0f0869a640 1 -- 172.21.15.28:0/4119006882 >> 172.21.15.28:0/4119006882 conn(0x7f0f00076440 msgr2=0x7f0f000ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.779+0000 7f0f0869a640 1 -- 172.21.15.28:0/4119006882 shutdown_connections 2024-04-25T01:40:58.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.779+0000 7f0f0869a640 1 -- 172.21.15.28:0/4119006882 wait complete. 2024-04-25T01:40:58.776 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:40:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.895+0000 7f9eb1e93640 1 -- 172.21.15.28:0/431808834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eac107730 msgr2=0x7f9eac107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.895+0000 7f9eb1e93640 1 --2- 172.21.15.28:0/431808834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eac107730 0x7f9eac107b10 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c009930 tx=0x7f9e9c02a420 comp rx=0 tx=0).stop 2024-04-25T01:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.895+0000 7f9eb1e93640 1 -- 172.21.15.28:0/431808834 shutdown_connections 2024-04-25T01:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.895+0000 7f9eb1e93640 1 --2- 172.21.15.28:0/431808834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eac06a780 0x7f9eac10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.895+0000 7f9eb1e93640 1 --2- 172.21.15.28:0/431808834 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9eac069dc0 0x7f9eac06a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.895+0000 7f9eb1e93640 1 --2- 172.21.15.28:0/431808834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eac107730 0x7f9eac107b10 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.895+0000 7f9eb1e93640 1 -- 172.21.15.28:0/431808834 >> 172.21.15.28:0/431808834 conn(0x7f9eac0776a0 msgr2=0x7f9eac077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.895+0000 7f9eb1e93640 1 -- 172.21.15.28:0/431808834 shutdown_connections 2024-04-25T01:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.895+0000 7f9eb1e93640 1 -- 172.21.15.28:0/431808834 wait complete. 2024-04-25T01:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.895+0000 7f9eb1e93640 1 Processor -- start 2024-04-25T01:40:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eb1e93640 1 -- start start 2024-04-25T01:40:58.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eb1e93640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eac069dc0 0x7f9eac100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eb1e93640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9eac06a780 0x7f9eac100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eb1e93640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eac107730 0x7f9eac1010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eb1e93640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac111b00 con 0x7f9eac069dc0 2024-04-25T01:40:58.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eb1e93640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac111980 con 0x7f9eac107730 2024-04-25T01:40:58.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eb1e93640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac111c80 con 0x7f9eac06a780 2024-04-25T01:40:58.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eac069dc0 0x7f9eac100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eac069dc0 0x7f9eac100670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48114/0 (socket says 172.21.15.28:48114) 2024-04-25T01:40:58.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eab7fe640 1 -- 172.21.15.28:0/3452294047 learned_addr learned my addr 172.21.15.28:0/3452294047 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eabfff640 1 --2- 172.21.15.28:0/3452294047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eac107730 0x7f9eac1010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eaaffd640 1 --2- 172.21.15.28:0/3452294047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9eac06a780 0x7f9eac100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eab7fe640 1 -- 172.21.15.28:0/3452294047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9eac06a780 msgr2=0x7f9eac100bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eab7fe640 1 --2- 172.21.15.28:0/3452294047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9eac06a780 0x7f9eac100bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eab7fe640 1 -- 172.21.15.28:0/3452294047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eac107730 msgr2=0x7f9eac1010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eab7fe640 1 --2- 172.21.15.28:0/3452294047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eac107730 0x7f9eac1010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eab7fe640 1 -- 172.21.15.28:0/3452294047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eac1a6480 con 0x7f9eac069dc0 2024-04-25T01:40:58.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eabfff640 1 --2- 172.21.15.28:0/3452294047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eac107730 0x7f9eac1010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:58.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eaaffd640 1 --2- 172.21.15.28:0/3452294047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9eac06a780 0x7f9eac100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:58.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eab7fe640 1 --2- 172.21.15.28:0/3452294047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eac069dc0 0x7f9eac100670 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c009a50 tx=0x7f9e9c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9ea8ff9640 1 -- 172.21.15.28:0/3452294047 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c03f030 con 0x7f9eac069dc0 2024-04-25T01:40:58.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9ea8ff9640 1 -- 172.21.15.28:0/3452294047 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e9c003ce0 con 0x7f9eac069dc0 2024-04-25T01:40:58.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9eb1e93640 1 -- 172.21.15.28:0/3452294047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eac1a6710 con 0x7f9eac069dc0 2024-04-25T01:40:58.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.896+0000 7f9ea8ff9640 1 -- 172.21.15.28:0/3452294047 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c035070 con 0x7f9eac069dc0 2024-04-25T01:40:58.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.897+0000 7f9eb1e93640 1 -- 172.21.15.28:0/3452294047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eac1a6c00 con 0x7f9eac069dc0 2024-04-25T01:40:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.899+0000 7f9ea8ff9640 1 -- 172.21.15.28:0/3452294047 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9e9c042030 con 0x7f9eac069dc0 2024-04-25T01:40:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.899+0000 7f9eb4b9e740 1 Processor -- start 2024-04-25T01:40:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.899+0000 7f9eb4b9e740 1 -- start start 2024-04-25T01:40:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.899+0000 7f9eb4b9e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6c580 0x5610c6d6c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.899+0000 7f9eb4b9e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6cea0 0x5610c6d77030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.899+0000 7f9eb4b9e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d77570 0x5610c6d79960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.899+0000 7f9eb4b9e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5610c6cd1670 con 0x5610c6d6c580 2024-04-25T01:40:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.899+0000 7f9eb4b9e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5610c6c79970 con 0x5610c6d6cea0 2024-04-25T01:40:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.899+0000 7f9eb4b9e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5610c6ccd630 con 0x5610c6d77570 2024-04-25T01:40:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.899+0000 7f9eabfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6c580 0x5610c6d6c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.899+0000 7f9eabfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6c580 0x5610c6d6c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48128/0 (socket says 172.21.15.28:48128) 2024-04-25T01:40:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.899+0000 7f9eabfff640 1 -- 172.21.15.28:0/2828358948 learned_addr learned my addr 172.21.15.28:0/2828358948 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.899+0000 7f9eab7fe640 1 --2- 172.21.15.28:0/2828358948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6cea0 0x5610c6d77030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.899+0000 7f9ea8ff9640 1 --2- 172.21.15.28:0/3452294047 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9e940a2d00 0x7f9e940a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.899+0000 7f9eaaffd640 1 --2- 172.21.15.28:0/2828358948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d77570 0x5610c6d79960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.900+0000 7f9eabfff640 1 -- 172.21.15.28:0/2828358948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d77570 msgr2=0x5610c6d79960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.900+0000 7f9eabfff640 1 --2- 172.21.15.28:0/2828358948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d77570 0x5610c6d79960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.900+0000 7f9eabfff640 1 -- 172.21.15.28:0/2828358948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6cea0 msgr2=0x5610c6d77030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.900+0000 7f9eabfff640 1 --2- 172.21.15.28:0/2828358948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6cea0 0x5610c6d77030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.900+0000 7f9eabfff640 1 -- 172.21.15.28:0/2828358948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610c6c9d400 con 0x5610c6d6c580 2024-04-25T01:40:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.900+0000 7f9eaaffd640 1 --2- 172.21.15.28:0/2828358948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d77570 0x5610c6d79960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.900+0000 7f9ea8ff9640 1 -- 172.21.15.28:0/3452294047 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f9e9c0efe50 con 0x7f9eac069dc0 2024-04-25T01:40:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.900+0000 7f9eabfff640 1 --2- 172.21.15.28:0/3452294047 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9e940a2d00 0x7f9e940a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.900+0000 7f9eabfff640 1 --2- 172.21.15.28:0/2828358948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6c580 0x5610c6d6c960 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4008ed0 tx=0x7f9ea400d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=629a323368444920 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.900+0000 7f9e91ffb640 1 -- 172.21.15.28:0/2828358948 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4011e60 con 0x5610c6d6c580 2024-04-25T01:40:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.900+0000 7f9e91ffb640 1 -- 172.21.15.28:0/2828358948 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ea401adc0 con 0x5610c6d6c580 2024-04-25T01:40:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.900+0000 7f9e91ffb640 1 -- 172.21.15.28:0/2828358948 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4019c10 con 0x5610c6d6c580 2024-04-25T01:40:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.900+0000 7f9eabfff640 1 --2- 172.21.15.28:0/3452294047 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9e940a2d00 0x7f9e940a51c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9ea401e000 tx=0x7f9ea401f000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.901+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/2828358948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6c580 msgr2=0x5610c6d6c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.901+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/2828358948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6c580 0x5610c6d6c960 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4008ed0 tx=0x7f9ea400d710 comp rx=0 tx=0).stop 2024-04-25T01:40:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.901+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/2828358948 shutdown_connections 2024-04-25T01:40:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.901+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/2828358948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d77570 0x5610c6d79960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.901+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/2828358948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6cea0 0x5610c6d77030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.901+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/2828358948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6c580 0x5610c6d6c960 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.901+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/2828358948 >> 172.21.15.28:0/2828358948 conn(0x5610c6d688a0 msgr2=0x5610c6d68c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.901+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/2828358948 shutdown_connections 2024-04-25T01:40:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.901+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/2828358948 wait complete. 2024-04-25T01:40:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.901+0000 7f9eb4b9e740 1 -- start start 2024-04-25T01:40:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.901+0000 7f9eb4b9e740 1 Processor -- start 2024-04-25T01:40:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9eb4b9e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6c580 0x5610c6d76f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9eb4b9e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d6cea0 0x5610c6d74210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9eb4b9e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d77570 0x5610c6d747b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9eb4b9e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5610c6d7a360 con 0x5610c6d77570 2024-04-25T01:40:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9eb4b9e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5610c6c8d050 con 0x5610c6d6c580 2024-04-25T01:40:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9eb4b9e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5610c6d7a4e0 con 0x5610c6d6cea0 2024-04-25T01:40:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9eb4b9e740 10 client.0 _ll_register_callbacks cb 0x5610c69e4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9eabfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d77570 0x5610c6d747b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9eabfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d77570 0x5610c6d747b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48130/0 (socket says 172.21.15.28:48130) 2024-04-25T01:40:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9eabfff640 1 -- 172.21.15.28:0/3020604380 learned_addr learned my addr 172.21.15.28:0/3020604380 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9eab7fe640 1 --2- 172.21.15.28:0/3020604380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6c580 0x5610c6d76f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9eaaffd640 1 --2- 172.21.15.28:0/3020604380 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d6cea0 0x5610c6d74210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9eabfff640 1 -- 172.21.15.28:0/3020604380 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d6cea0 msgr2=0x5610c6d74210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9eabfff640 1 --2- 172.21.15.28:0/3020604380 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d6cea0 0x5610c6d74210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9eabfff640 1 -- 172.21.15.28:0/3020604380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6c580 msgr2=0x5610c6d76f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9eabfff640 1 --2- 172.21.15.28:0/3020604380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6c580 0x5610c6d76f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9eabfff640 1 -- 172.21.15.28:0/3020604380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610c6c5edc0 con 0x5610c6d77570 2024-04-25T01:40:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9eab7fe640 1 --2- 172.21.15.28:0/3020604380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6c580 0x5610c6d76f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9eabfff640 1 --2- 172.21.15.28:0/3020604380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d77570 0x5610c6d747b0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4019c70 tx=0x7f9ea4019ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7410cb87a502eaf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9e8affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.902+0000 7f9e8affd640 1 -- 172.21.15.28:0/3020604380 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4032060 con 0x5610c6d77570 2024-04-25T01:40:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.903+0000 7f9e8affd640 1 -- 172.21.15.28:0/3020604380 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f9ea4020070 con 0x5610c6d77570 2024-04-25T01:40:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.903+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/3020604380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ea401adc0 con 0x5610c6d77570 2024-04-25T01:40:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.903+0000 7f9e8affd640 1 -- 172.21.15.28:0/3020604380 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9ea4039050 con 0x5610c6d77570 2024-04-25T01:40:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.903+0000 7f9eb4b9e740 10 client.18609 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.903+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/3020604380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5610c69e53f0 con 0x5610c6d77570 2024-04-25T01:40:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.903+0000 7f9e8affd640 1 -- 172.21.15.28:0/3020604380 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ea401adc0 con 0x5610c6d77570 2024-04-25T01:40:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.903+0000 7f9e8affd640 1 -- 172.21.15.28:0/3020604380 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9ea4024960 con 0x5610c6d77570 2024-04-25T01:40:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.903+0000 7f9eb4b9e740 10 client.18609 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.903+0000 7f9eb4b9e740 10 client.18609 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:40:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.903+0000 7f9eb4b9e740 20 client.18609 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.903+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/3020604380 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5610c6d87600 0x5610c6d899f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.903+0000 7f9eb4b9e740 4 client.18609 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.903+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/3020604380 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5610c6cdebb0 con 0x5610c6d87600 2024-04-25T01:40:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.903+0000 7f9eab7fe640 1 --2- 172.21.15.28:0/3020604380 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5610c6d87600 0x5610c6d899f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.904+0000 7f9e8affd640 10 client.18609 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:40:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.904+0000 7f9eab7fe640 1 --2- 172.21.15.28:0/3020604380 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5610c6d87600 0x5610c6d899f0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=982ac38c5e6fd3be server_cookie=b15f86812773a76b in_seq=0 out_seq=0 2024-04-25T01:40:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.905+0000 7f9e8affd640 1 -- 172.21.15.28:0/3020604380 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5610c6cdebb0 con 0x5610c6d87600 2024-04-25T01:40:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.905+0000 7f9e8affd640 10 client.18609 handle_command_reply: tid=0 2024-04-25T01:40:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.905+0000 7f9eb4b9e740 1 client.18609 shutdown 2024-04-25T01:40:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.905+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/3020604380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d77570 msgr2=0x5610c6d747b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.905+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/3020604380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d77570 0x5610c6d747b0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4019c70 tx=0x7f9ea4019ca0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.905+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/3020604380 shutdown_connections 2024-04-25T01:40:58.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.905+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/3020604380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d77570 0x5610c6d747b0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.905+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/3020604380 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5610c6d87600 0x5610c6d899f0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.905+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/3020604380 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d6cea0 0x5610c6d74210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.905+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/3020604380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6c580 0x5610c6d76f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.905+0000 7f9e8affd640 0 client.18609 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:40:58.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.905+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/3020604380 >> 172.21.15.28:0/3020604380 conn(0x5610c6d688a0 msgr2=0x5610c6d6e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.905+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/3020604380 shutdown_connections 2024-04-25T01:40:58.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.905+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/3020604380 wait complete. 2024-04-25T01:40:58.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.905+0000 7f9eb4b9e740 20 client.18609 trim_cache size 0 max 16384 2024-04-25T01:40:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.918+0000 7f9eb4b9e740 1 Processor -- start 2024-04-25T01:40:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.918+0000 7f9eb4b9e740 1 -- start start 2024-04-25T01:40:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.918+0000 7f9eb4b9e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6c580 0x5610c6d83220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.918+0000 7f9eb4b9e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6cea0 0x5610c6d76a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.918+0000 7f9eb4b9e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d77570 0x5610c6d8f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.918+0000 7f9eb4b9e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac1b0d50 con 0x5610c6d6cea0 2024-04-25T01:40:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.918+0000 7f9eb4b9e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac1b2890 con 0x5610c6d6c580 2024-04-25T01:40:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.918+0000 7f9eb4b9e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac1b1420 con 0x5610c6d77570 2024-04-25T01:40:58.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.918+0000 7f9eaaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6cea0 0x5610c6d76a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.918+0000 7f9eaaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6cea0 0x5610c6d76a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48134/0 (socket says 172.21.15.28:48134) 2024-04-25T01:40:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.918+0000 7f9eaaffd640 1 -- 172.21.15.28:0/3616240978 learned_addr learned my addr 172.21.15.28:0/3616240978 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.919+0000 7f9eab7fe640 1 --2- 172.21.15.28:0/3616240978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6c580 0x5610c6d83220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.919+0000 7f9eabfff640 1 --2- 172.21.15.28:0/3616240978 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d77570 0x5610c6d8f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.919+0000 7f9eaaffd640 1 -- 172.21.15.28:0/3616240978 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d77570 msgr2=0x5610c6d8f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.919+0000 7f9eaaffd640 1 --2- 172.21.15.28:0/3616240978 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d77570 0x5610c6d8f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.919+0000 7f9eaaffd640 1 -- 172.21.15.28:0/3616240978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6c580 msgr2=0x5610c6d83220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.919+0000 7f9eaaffd640 1 --2- 172.21.15.28:0/3616240978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6c580 0x5610c6d83220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.919+0000 7f9eaaffd640 1 -- 172.21.15.28:0/3616240978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610c6aaf7d0 con 0x5610c6d6cea0 2024-04-25T01:40:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.919+0000 7f9eabfff640 1 --2- 172.21.15.28:0/3616240978 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d77570 0x5610c6d8f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.919+0000 7f9eaaffd640 1 --2- 172.21.15.28:0/3616240978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6cea0 0x5610c6d76a50 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000bb80 tx=0x7f9ea0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9018b1cbca212852 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.919+0000 7f9e917fa640 1 -- 172.21.15.28:0/3616240978 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0005220 con 0x5610c6d6cea0 2024-04-25T01:40:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.919+0000 7f9e917fa640 1 -- 172.21.15.28:0/3616240978 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ea0004c80 con 0x5610c6d6cea0 2024-04-25T01:40:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.919+0000 7f9e917fa640 1 -- 172.21.15.28:0/3616240978 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea000e9e0 con 0x5610c6d6cea0 2024-04-25T01:40:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.920+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/3616240978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6cea0 msgr2=0x5610c6d76a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.920+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/3616240978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6cea0 0x5610c6d76a50 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000bb80 tx=0x7f9ea0011810 comp rx=0 tx=0).stop 2024-04-25T01:40:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.920+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/3616240978 shutdown_connections 2024-04-25T01:40:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.920+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/3616240978 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d77570 0x5610c6d8f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.920+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/3616240978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6cea0 0x5610c6d76a50 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.920+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/3616240978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6c580 0x5610c6d83220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.920+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/3616240978 >> 172.21.15.28:0/3616240978 conn(0x5610c6d688a0 msgr2=0x5610c6d6f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.920+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/3616240978 shutdown_connections 2024-04-25T01:40:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.920+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/3616240978 wait complete. 2024-04-25T01:40:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.920+0000 7f9eb4b9e740 1 -- start start 2024-04-25T01:40:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.920+0000 7f9eb4b9e740 1 Processor -- start 2024-04-25T01:40:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.920+0000 7f9eb4b9e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6c580 0x5610c6d7fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.920+0000 7f9eb4b9e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6cea0 0x5610c6d7b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.920+0000 7f9eb4b9e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d77570 0x5610c6d7b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.920+0000 7f9eb4b9e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac1b21c0 con 0x5610c6d6cea0 2024-04-25T01:40:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.920+0000 7f9eb4b9e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac1b1af0 con 0x5610c6d6c580 2024-04-25T01:40:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.920+0000 7f9eb4b9e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac1aed50 con 0x5610c6d77570 2024-04-25T01:40:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.920+0000 7f9eb4b9e740 10 client.0 _ll_register_callbacks cb 0x7f9ea401d020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9eaaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6cea0 0x5610c6d7b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9eaaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6cea0 0x5610c6d7b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48148/0 (socket says 172.21.15.28:48148) 2024-04-25T01:40:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9eaaffd640 1 -- 172.21.15.28:0/1633201792 learned_addr learned my addr 172.21.15.28:0/1633201792 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9eabfff640 1 --2- 172.21.15.28:0/1633201792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d77570 0x5610c6d7b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9eab7fe640 1 --2- 172.21.15.28:0/1633201792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6c580 0x5610c6d7fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9eaaffd640 1 -- 172.21.15.28:0/1633201792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d77570 msgr2=0x5610c6d7b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9eaaffd640 1 --2- 172.21.15.28:0/1633201792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d77570 0x5610c6d7b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9eaaffd640 1 -- 172.21.15.28:0/1633201792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6c580 msgr2=0x5610c6d7fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9eaaffd640 1 --2- 172.21.15.28:0/1633201792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6c580 0x5610c6d7fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9eaaffd640 1 -- 172.21.15.28:0/1633201792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610c6dec8c0 con 0x5610c6d6cea0 2024-04-25T01:40:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9eabfff640 1 --2- 172.21.15.28:0/1633201792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d77570 0x5610c6d7b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9eab7fe640 1 --2- 172.21.15.28:0/1633201792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6c580 0x5610c6d7fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9eaaffd640 1 --2- 172.21.15.28:0/1633201792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6cea0 0x5610c6d7b020 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000e580 tx=0x7f9ea0005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=171fbb746c19268c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9e8affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9e8affd640 1 -- 172.21.15.28:0/1633201792 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0005440 con 0x5610c6d6cea0 2024-04-25T01:40:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9e8affd640 1 -- 172.21.15.28:0/1633201792 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f9ea000e8f0 con 0x5610c6d6cea0 2024-04-25T01:40:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/1633201792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ea0004c80 con 0x5610c6d6cea0 2024-04-25T01:40:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9e8affd640 1 -- 172.21.15.28:0/1633201792 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9ea0030050 con 0x5610c6d6cea0 2024-04-25T01:40:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9eb4b9e740 10 client.18615 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.921+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/1633201792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5610c6deda40 con 0x5610c6d6cea0 2024-04-25T01:40:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.922+0000 7f9e8affd640 1 -- 172.21.15.28:0/1633201792 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ea0004c80 con 0x5610c6d6cea0 2024-04-25T01:40:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.922+0000 7f9e8affd640 1 -- 172.21.15.28:0/1633201792 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9ea0018070 con 0x5610c6d6cea0 2024-04-25T01:40:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.922+0000 7f9eb4b9e740 10 client.18615 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.922+0000 7f9eb4b9e740 10 client.18615 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:40:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.922+0000 7f9eb4b9e740 20 client.18615 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.922+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/1633201792 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5610c6dee150 0x5610c6df0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.922+0000 7f9eb4b9e740 4 client.18615 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.922+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/1633201792 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5610c6cd8f00 con 0x5610c6dee150 2024-04-25T01:40:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.922+0000 7f9eab7fe640 1 --2- 172.21.15.28:0/1633201792 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5610c6dee150 0x5610c6df0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.922+0000 7f9eab7fe640 1 --2- 172.21.15.28:0/1633201792 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5610c6dee150 0x5610c6df0570 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=726b1ca4e2a0744e server_cookie=8da006d8d3c5628b in_seq=0 out_seq=0 2024-04-25T01:40:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.922+0000 7f9e8affd640 10 client.18615 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:40:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.923+0000 7f9e8affd640 1 -- 172.21.15.28:0/1633201792 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5610c6cd8f00 con 0x5610c6dee150 2024-04-25T01:40:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.923+0000 7f9e8affd640 10 client.18615 handle_command_reply: tid=0 2024-04-25T01:40:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.923+0000 7f9eb4b9e740 1 client.18615 shutdown 2024-04-25T01:40:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.923+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/1633201792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6cea0 msgr2=0x5610c6d7b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.923+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/1633201792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6cea0 0x5610c6d7b020 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000e580 tx=0x7f9ea0005250 comp rx=0 tx=0).stop 2024-04-25T01:40:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.923+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/1633201792 shutdown_connections 2024-04-25T01:40:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.923+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/1633201792 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x5610c6dee150 0x5610c6df0570 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.923+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/1633201792 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5610c6d77570 0x5610c6d7b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.923+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/1633201792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5610c6d6cea0 0x5610c6d7b020 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.923+0000 7f9e8affd640 0 client.18615 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:40:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.923+0000 7f9eb4b9e740 1 --2- 172.21.15.28:0/1633201792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5610c6d6c580 0x5610c6d7fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.923+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/1633201792 >> 172.21.15.28:0/1633201792 conn(0x5610c6d688a0 msgr2=0x5610c6d82210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.923+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/1633201792 shutdown_connections 2024-04-25T01:40:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.923+0000 7f9eb4b9e740 1 -- 172.21.15.28:0/1633201792 wait complete. 2024-04-25T01:40:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.923+0000 7f9eb4b9e740 20 client.18615 trim_cache size 0 max 16384 2024-04-25T01:40:58.916 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.923+0000 7f9eb1e93640 1 -- 172.21.15.28:0/3452294047 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9e940a2d00 msgr2=0x7f9e940a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.923+0000 7f9eb1e93640 1 --2- 172.21.15.28:0/3452294047 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9e940a2d00 0x7f9e940a51c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9ea401e000 tx=0x7f9ea401f000 comp rx=0 tx=0).stop 2024-04-25T01:40:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.923+0000 7f9eb1e93640 1 -- 172.21.15.28:0/3452294047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eac069dc0 msgr2=0x7f9eac100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.923+0000 7f9eb1e93640 1 --2- 172.21.15.28:0/3452294047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eac069dc0 0x7f9eac100670 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c009a50 tx=0x7f9e9c004270 comp rx=0 tx=0).stop 2024-04-25T01:40:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.924+0000 7f9eb1e93640 1 -- 172.21.15.28:0/3452294047 shutdown_connections 2024-04-25T01:40:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.924+0000 7f9eb1e93640 1 --2- 172.21.15.28:0/3452294047 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9e940a2d00 0x7f9e940a51c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.924+0000 7f9eb1e93640 1 --2- 172.21.15.28:0/3452294047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9eac107730 0x7f9eac1010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.924+0000 7f9eb1e93640 1 --2- 172.21.15.28:0/3452294047 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9eac06a780 0x7f9eac100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.924+0000 7f9eb1e93640 1 --2- 172.21.15.28:0/3452294047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9eac069dc0 0x7f9eac100670 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.924+0000 7f9eb1e93640 1 -- 172.21.15.28:0/3452294047 >> 172.21.15.28:0/3452294047 conn(0x7f9eac0776a0 msgr2=0x7f9eac0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:58.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.924+0000 7f9eb1e93640 1 -- 172.21.15.28:0/3452294047 shutdown_connections 2024-04-25T01:40:58.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:58.924+0000 7f9eb1e93640 1 -- 172.21.15.28:0/3452294047 wait complete. 2024-04-25T01:40:58.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:40:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.039+0000 7f63e3d96640 1 -- 172.21.15.28:0/574602144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63dc1029c0 msgr2=0x7f63dc102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.039+0000 7f63e3d96640 1 --2- 172.21.15.28:0/574602144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63dc1029c0 0x7f63dc102da0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f63d8009ca0 tx=0x7f63d802a9e0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.039+0000 7f63e3d96640 1 -- 172.21.15.28:0/574602144 shutdown_connections 2024-04-25T01:40:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.039+0000 7f63e3d96640 1 --2- 172.21.15.28:0/574602144 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63dc10bd00 0x7f63dc10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.039+0000 7f63e3d96640 1 --2- 172.21.15.28:0/574602144 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc1032e0 0x7f63dc10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.039+0000 7f63e3d96640 1 --2- 172.21.15.28:0/574602144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63dc1029c0 0x7f63dc102da0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.039+0000 7f63e3d96640 1 -- 172.21.15.28:0/574602144 >> 172.21.15.28:0/574602144 conn(0x7f63dc077740 msgr2=0x7f63dc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.040+0000 7f63e3d96640 1 -- 172.21.15.28:0/574602144 shutdown_connections 2024-04-25T01:40:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.040+0000 7f63e3d96640 1 -- 172.21.15.28:0/574602144 wait complete. 2024-04-25T01:40:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.040+0000 7f63e3d96640 1 Processor -- start 2024-04-25T01:40:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.040+0000 7f63e3d96640 1 -- start start 2024-04-25T01:40:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.040+0000 7f63e3d96640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63dc1029c0 0x7f63dc100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.040+0000 7f63e3d96640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc1032e0 0x7f63dc100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.040+0000 7f63e3d96640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63dc10bd00 0x7f63dc101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.040+0000 7f63e3d96640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc111b50 con 0x7f63dc10bd00 2024-04-25T01:40:59.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.040+0000 7f63e3d96640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc1119d0 con 0x7f63dc1032e0 2024-04-25T01:40:59.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.040+0000 7f63e3d96640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc111cd0 con 0x7f63dc1029c0 2024-04-25T01:40:59.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.040+0000 7f63e2333640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63dc10bd00 0x7f63dc101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.041+0000 7f63e2333640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63dc10bd00 0x7f63dc101090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48160/0 (socket says 172.21.15.28:48160) 2024-04-25T01:40:59.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.041+0000 7f63e2333640 1 -- 172.21.15.28:0/3312100701 learned_addr learned my addr 172.21.15.28:0/3312100701 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:59.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.041+0000 7f63e1b32640 1 --2- 172.21.15.28:0/3312100701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63dc1029c0 0x7f63dc100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.041+0000 7f63e1331640 1 --2- 172.21.15.28:0/3312100701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc1032e0 0x7f63dc100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.041+0000 7f63e2333640 1 -- 172.21.15.28:0/3312100701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63dc1029c0 msgr2=0x7f63dc100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:59.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.041+0000 7f63e2333640 1 --2- 172.21.15.28:0/3312100701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63dc1029c0 0x7f63dc100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.041+0000 7f63e2333640 1 -- 172.21.15.28:0/3312100701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc1032e0 msgr2=0x7f63dc100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:59.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.041+0000 7f63e2333640 1 --2- 172.21.15.28:0/3312100701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc1032e0 0x7f63dc100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.041+0000 7f63e2333640 1 -- 172.21.15.28:0/3312100701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63dc1a6450 con 0x7f63dc10bd00 2024-04-25T01:40:59.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.041+0000 7f63e2333640 1 --2- 172.21.15.28:0/3312100701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63dc10bd00 0x7f63dc101090 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f63d400d750 tx=0x7f63d400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:59.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.041+0000 7f63d2ffd640 1 -- 172.21.15.28:0/3312100701 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d4017070 con 0x7f63dc10bd00 2024-04-25T01:40:59.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.041+0000 7f63e3d96640 1 -- 172.21.15.28:0/3312100701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63dc1a6740 con 0x7f63dc10bd00 2024-04-25T01:40:59.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.041+0000 7f63d2ffd640 1 -- 172.21.15.28:0/3312100701 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63d4005500 con 0x7f63dc10bd00 2024-04-25T01:40:59.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.041+0000 7f63d2ffd640 1 -- 172.21.15.28:0/3312100701 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d4004d90 con 0x7f63dc10bd00 2024-04-25T01:40:59.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.041+0000 7f63e3d96640 1 -- 172.21.15.28:0/3312100701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63dc1a6ce0 con 0x7f63dc10bd00 2024-04-25T01:40:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.043+0000 7f63d2ffd640 1 -- 172.21.15.28:0/3312100701 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f63d4004f30 con 0x7f63dc10bd00 2024-04-25T01:40:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.044+0000 7f63e6b77740 1 Processor -- start 2024-04-25T01:40:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.044+0000 7f63e6b77740 1 -- start start 2024-04-25T01:40:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.044+0000 7f63e6b77740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99d580 0x558b8d99d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.044+0000 7f63e6b77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d99dea0 0x558b8d9a8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.044+0000 7f63e6b77740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b8d9a8570 0x558b8d9aa960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.044+0000 7f63e6b77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558b8d902670 con 0x558b8d99dea0 2024-04-25T01:40:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.044+0000 7f63e6b77740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558b8d8aa970 con 0x558b8d9a8570 2024-04-25T01:40:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.044+0000 7f63e6b77740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x558b8d8fe630 con 0x558b8d99d580 2024-04-25T01:40:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.044+0000 7f63e1b32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d99dea0 0x558b8d9a8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.044+0000 7f63e1b32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d99dea0 0x558b8d9a8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48166/0 (socket says 172.21.15.28:48166) 2024-04-25T01:40:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.044+0000 7f63e1b32640 1 -- 172.21.15.28:0/1344241669 learned_addr learned my addr 172.21.15.28:0/1344241669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.044+0000 7f63d2ffd640 1 --2- 172.21.15.28:0/3312100701 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f63c80a2d30 0x7f63c80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.044+0000 7f63e1331640 1 --2- 172.21.15.28:0/1344241669 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99d580 0x558b8d99d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.044+0000 7f63e2333640 1 --2- 172.21.15.28:0/3312100701 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f63c80a2d30 0x7f63c80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.044+0000 7f63e1331640 1 --2- 172.21.15.28:0/1344241669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b8d9a8570 0x558b8d9aa960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.045+0000 7f63e1b32640 1 -- 172.21.15.28:0/1344241669 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99d580 msgr2=0x558b8d99d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.045+0000 7f63e1b32640 1 --2- 172.21.15.28:0/1344241669 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99d580 0x558b8d99d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.045+0000 7f63e1b32640 1 -- 172.21.15.28:0/1344241669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b8d9a8570 msgr2=0x558b8d9aa960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.045+0000 7f63e1b32640 1 --2- 172.21.15.28:0/1344241669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b8d9a8570 0x558b8d9aa960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.045+0000 7f63e1b32640 1 -- 172.21.15.28:0/1344241669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b8d8ce400 con 0x558b8d99dea0 2024-04-25T01:40:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.045+0000 7f63e1331640 1 --2- 172.21.15.28:0/1344241669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b8d9a8570 0x558b8d9aa960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.045+0000 7f63d2ffd640 1 -- 172.21.15.28:0/3312100701 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f63d40cf120 con 0x7f63dc10bd00 2024-04-25T01:40:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.045+0000 7f63e1b32640 1 --2- 172.21.15.28:0/1344241669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d99dea0 0x558b8d9a8030 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f63d802af00 tx=0x7f63d8005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b23f196f50442cc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:59.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.045+0000 7f63e2333640 1 --2- 172.21.15.28:0/3312100701 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f63c80a2d30 0x7f63c80a51f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f63d40976b0 tx=0x7f63d4075040 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.045+0000 7f63c7fff640 1 -- 172.21.15.28:0/1344241669 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d8033050 con 0x558b8d99dea0 2024-04-25T01:40:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.045+0000 7f63c7fff640 1 -- 172.21.15.28:0/1344241669 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63d8047070 con 0x558b8d99dea0 2024-04-25T01:40:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.045+0000 7f63c7fff640 1 -- 172.21.15.28:0/1344241669 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d802f9c0 con 0x558b8d99dea0 2024-04-25T01:40:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.046+0000 7f63e6b77740 1 -- 172.21.15.28:0/1344241669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d99dea0 msgr2=0x558b8d9a8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.046+0000 7f63e6b77740 1 --2- 172.21.15.28:0/1344241669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d99dea0 0x558b8d9a8030 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f63d802af00 tx=0x7f63d8005d80 comp rx=0 tx=0).stop 2024-04-25T01:40:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.046+0000 7f63e6b77740 1 -- 172.21.15.28:0/1344241669 shutdown_connections 2024-04-25T01:40:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.046+0000 7f63e6b77740 1 --2- 172.21.15.28:0/1344241669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b8d9a8570 0x558b8d9aa960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.046+0000 7f63e6b77740 1 --2- 172.21.15.28:0/1344241669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d99dea0 0x558b8d9a8030 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.046+0000 7f63e6b77740 1 --2- 172.21.15.28:0/1344241669 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99d580 0x558b8d99d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.046+0000 7f63e6b77740 1 -- 172.21.15.28:0/1344241669 >> 172.21.15.28:0/1344241669 conn(0x558b8d9998a0 msgr2=0x558b8d999c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.046+0000 7f63e6b77740 1 -- 172.21.15.28:0/1344241669 shutdown_connections 2024-04-25T01:40:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.046+0000 7f63e6b77740 1 -- 172.21.15.28:0/1344241669 wait complete. 2024-04-25T01:40:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.046+0000 7f63e6b77740 1 -- start start 2024-04-25T01:40:59.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.046+0000 7f63e6b77740 1 Processor -- start 2024-04-25T01:40:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63e6b77740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b8d99d580 0x558b8d9a7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63e6b77740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99dea0 0x558b8d9a5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63e6b77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d9a8570 0x558b8d9a5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63e6b77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558b8d9ab300 con 0x558b8d9a8570 2024-04-25T01:40:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63e6b77740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558b8d8be050 con 0x558b8d99d580 2024-04-25T01:40:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63e6b77740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x558b8d9ab480 con 0x558b8d99dea0 2024-04-25T01:40:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63e6b77740 10 client.0 _ll_register_callbacks cb 0x558b8d615be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63e1331640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d9a8570 0x558b8d9a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63e1331640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d9a8570 0x558b8d9a5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48168/0 (socket says 172.21.15.28:48168) 2024-04-25T01:40:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63e1331640 1 -- 172.21.15.28:0/1839708416 learned_addr learned my addr 172.21.15.28:0/1839708416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63e1b32640 1 --2- 172.21.15.28:0/1839708416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99dea0 0x558b8d9a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63e1331640 1 --2- 172.21.15.28:0/1839708416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b8d99d580 0x558b8d9a7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63e1331640 1 -- 172.21.15.28:0/1839708416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99dea0 msgr2=0x558b8d9a5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:59.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63e1331640 1 --2- 172.21.15.28:0/1839708416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99dea0 0x558b8d9a5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63e1331640 1 -- 172.21.15.28:0/1839708416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b8d99d580 msgr2=0x558b8d9a7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:59.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63e1331640 1 --2- 172.21.15.28:0/1839708416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b8d99d580 0x558b8d9a7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63e1331640 1 -- 172.21.15.28:0/1839708416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b8d88fdc0 con 0x558b8d9a8570 2024-04-25T01:40:59.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63e1331640 1 --2- 172.21.15.28:0/1839708416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d9a8570 0x558b8d9a5830 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f63cc013bc0 tx=0x7f63cc00e810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f2cce8370caa79 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:59.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:59.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63c57fa640 1 -- 172.21.15.28:0/1839708416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63cc005020 con 0x558b8d9a8570 2024-04-25T01:40:59.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63c57fa640 1 -- 172.21.15.28:0/1839708416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f63cc0051c0 con 0x558b8d9a8570 2024-04-25T01:40:59.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.047+0000 7f63e6b77740 1 -- 172.21.15.28:0/1839708416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63d8047070 con 0x558b8d9a8570 2024-04-25T01:40:59.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.048+0000 7f63c57fa640 1 -- 172.21.15.28:0/1839708416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f63cc023050 con 0x558b8d9a8570 2024-04-25T01:40:59.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.048+0000 7f63e6b77740 10 client.18627 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:59.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.048+0000 7f63e6b77740 1 -- 172.21.15.28:0/1839708416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558b8d6163f0 con 0x558b8d9a8570 2024-04-25T01:40:59.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.048+0000 7f63c57fa640 1 -- 172.21.15.28:0/1839708416 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63d8047070 con 0x558b8d9a8570 2024-04-25T01:40:59.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.048+0000 7f63c57fa640 1 -- 172.21.15.28:0/1839708416 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f63cc00f5d0 con 0x558b8d9a8570 2024-04-25T01:40:59.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.048+0000 7f63e6b77740 10 client.18627 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:59.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.048+0000 7f63e6b77740 10 client.18627 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:40:59.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.048+0000 7f63e6b77740 20 client.18627 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:59.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.048+0000 7f63e6b77740 1 --2- 172.21.15.28:0/1839708416 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x558b8d9b85e0 0x558b8d9ba9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.048+0000 7f63e6b77740 4 client.18627 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:59.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.048+0000 7f63e6b77740 1 -- 172.21.15.28:0/1839708416 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b8d90fbb0 con 0x558b8d9b85e0 2024-04-25T01:40:59.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.048+0000 7f63e1b32640 1 --2- 172.21.15.28:0/1839708416 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x558b8d9b85e0 0x558b8d9ba9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.049+0000 7f63c57fa640 10 client.18627 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:40:59.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.049+0000 7f63e1b32640 1 --2- 172.21.15.28:0/1839708416 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x558b8d9b85e0 0x558b8d9ba9d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c4d5c6e24a947db8 server_cookie=9413441ba96e25e1 in_seq=0 out_seq=0 2024-04-25T01:40:59.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.049+0000 7f63c57fa640 1 -- 172.21.15.28:0/1839708416 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558b8d90fbb0 con 0x558b8d9b85e0 2024-04-25T01:40:59.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.050+0000 7f63c57fa640 10 client.18627 handle_command_reply: tid=0 2024-04-25T01:40:59.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.050+0000 7f63e6b77740 1 client.18627 shutdown 2024-04-25T01:40:59.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.050+0000 7f63e6b77740 1 -- 172.21.15.28:0/1839708416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d9a8570 msgr2=0x558b8d9a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:59.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.050+0000 7f63e6b77740 1 --2- 172.21.15.28:0/1839708416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d9a8570 0x558b8d9a5830 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f63cc013bc0 tx=0x7f63cc00e810 comp rx=0 tx=0).stop 2024-04-25T01:40:59.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.050+0000 7f63e6b77740 1 -- 172.21.15.28:0/1839708416 shutdown_connections 2024-04-25T01:40:59.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.050+0000 7f63e6b77740 1 --2- 172.21.15.28:0/1839708416 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x558b8d9b85e0 0x558b8d9ba9d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.050+0000 7f63e6b77740 1 --2- 172.21.15.28:0/1839708416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d9a8570 0x558b8d9a5830 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.050+0000 7f63e6b77740 1 --2- 172.21.15.28:0/1839708416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99dea0 0x558b8d9a5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.050+0000 7f63e6b77740 1 --2- 172.21.15.28:0/1839708416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b8d99d580 0x558b8d9a7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.050+0000 7f63c57fa640 0 client.18627 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:40:59.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.050+0000 7f63e6b77740 1 -- 172.21.15.28:0/1839708416 >> 172.21.15.28:0/1839708416 conn(0x558b8d9998a0 msgr2=0x558b8d99f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:59.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.050+0000 7f63e6b77740 1 -- 172.21.15.28:0/1839708416 shutdown_connections 2024-04-25T01:40:59.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.050+0000 7f63e6b77740 1 -- 172.21.15.28:0/1839708416 wait complete. 2024-04-25T01:40:59.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.050+0000 7f63e6b77740 20 client.18627 trim_cache size 0 max 16384 2024-04-25T01:40:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.063+0000 7f63e6b77740 1 Processor -- start 2024-04-25T01:40:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.063+0000 7f63e6b77740 1 -- start start 2024-04-25T01:40:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.063+0000 7f63e6b77740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b8d99d580 0x558b8d9a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.063+0000 7f63e6b77740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99dea0 0x558b8d9a7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.063+0000 7f63e6b77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d9a8570 0x558b8d9c02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.063+0000 7f63e6b77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc1b0cf0 con 0x558b8d9a8570 2024-04-25T01:40:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.063+0000 7f63e6b77740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc1b2830 con 0x558b8d99d580 2024-04-25T01:40:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.063+0000 7f63e6b77740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc1b13c0 con 0x558b8d99dea0 2024-04-25T01:40:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.064+0000 7f63e1331640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d9a8570 0x558b8d9c02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.064+0000 7f63e1331640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d9a8570 0x558b8d9c02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48176/0 (socket says 172.21.15.28:48176) 2024-04-25T01:40:59.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.064+0000 7f63e1331640 1 -- 172.21.15.28:0/4043630729 learned_addr learned my addr 172.21.15.28:0/4043630729 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.064+0000 7f63e1331640 1 -- 172.21.15.28:0/4043630729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99dea0 msgr2=0x558b8d9a7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.064+0000 7f63e1331640 1 --2- 172.21.15.28:0/4043630729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99dea0 0x558b8d9a7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.064+0000 7f63e1331640 1 -- 172.21.15.28:0/4043630729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b8d99d580 msgr2=0x558b8d9a4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.064+0000 7f63e1331640 1 --2- 172.21.15.28:0/4043630729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b8d99d580 0x558b8d9a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.064+0000 7f63e1331640 1 -- 172.21.15.28:0/4043630729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b8d99dcd0 con 0x558b8d9a8570 2024-04-25T01:40:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.064+0000 7f63e1331640 1 --2- 172.21.15.28:0/4043630729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d9a8570 0x558b8d9c02e0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f63cc00f9a0 tx=0x7f63cc00e930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a0f1036dda889ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.064+0000 7f63c77fe640 1 -- 172.21.15.28:0/4043630729 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63cc013310 con 0x558b8d9a8570 2024-04-25T01:40:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.064+0000 7f63c77fe640 1 -- 172.21.15.28:0/4043630729 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63cc00f790 con 0x558b8d9a8570 2024-04-25T01:40:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.064+0000 7f63c77fe640 1 -- 172.21.15.28:0/4043630729 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63cc005260 con 0x558b8d9a8570 2024-04-25T01:40:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.065+0000 7f63e6b77740 1 -- 172.21.15.28:0/4043630729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d9a8570 msgr2=0x558b8d9c02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.065+0000 7f63e6b77740 1 --2- 172.21.15.28:0/4043630729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d9a8570 0x558b8d9c02e0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f63cc00f9a0 tx=0x7f63cc00e930 comp rx=0 tx=0).stop 2024-04-25T01:40:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.065+0000 7f63e6b77740 1 -- 172.21.15.28:0/4043630729 shutdown_connections 2024-04-25T01:40:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.065+0000 7f63e6b77740 1 --2- 172.21.15.28:0/4043630729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d9a8570 0x558b8d9c02e0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.065+0000 7f63e6b77740 1 --2- 172.21.15.28:0/4043630729 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99dea0 0x558b8d9a7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.065+0000 7f63e6b77740 1 --2- 172.21.15.28:0/4043630729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b8d99d580 0x558b8d9a4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.065+0000 7f63e6b77740 1 -- 172.21.15.28:0/4043630729 >> 172.21.15.28:0/4043630729 conn(0x558b8d9998a0 msgr2=0x558b8d99f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.065+0000 7f63e6b77740 1 -- 172.21.15.28:0/4043630729 shutdown_connections 2024-04-25T01:40:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.065+0000 7f63e6b77740 1 -- 172.21.15.28:0/4043630729 wait complete. 2024-04-25T01:40:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.065+0000 7f63e6b77740 1 -- start start 2024-04-25T01:40:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.065+0000 7f63e6b77740 1 Processor -- start 2024-04-25T01:40:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63e6b77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d99d580 0x558b8d9b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63e6b77740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99dea0 0x558b8d9a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63e6b77740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b8d9a8570 0x558b8d9a24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63e6b77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc1b2160 con 0x558b8d99d580 2024-04-25T01:40:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63e6b77740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc1b1a90 con 0x558b8d9a8570 2024-04-25T01:40:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63e6b77740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc1aecf0 con 0x558b8d99dea0 2024-04-25T01:40:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63e6b77740 10 client.0 _ll_register_callbacks cb 0x7f63cc00e710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:59.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63e1331640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d99d580 0x558b8d9b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63e1331640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d99d580 0x558b8d9b0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48188/0 (socket says 172.21.15.28:48188) 2024-04-25T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63e1331640 1 -- 172.21.15.28:0/4026349819 learned_addr learned my addr 172.21.15.28:0/4026349819 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63e1b32640 1 --2- 172.21.15.28:0/4026349819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99dea0 0x558b8d9a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63e1331640 1 -- 172.21.15.28:0/4026349819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99dea0 msgr2=0x558b8d9a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63e1331640 1 --2- 172.21.15.28:0/4026349819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99dea0 0x558b8d9a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63e1331640 1 -- 172.21.15.28:0/4026349819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b8d9a8570 msgr2=0x558b8d9a24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63e1331640 1 --2- 172.21.15.28:0/4026349819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b8d9a8570 0x558b8d9a24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63e1331640 1 -- 172.21.15.28:0/4026349819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b8d6e07d0 con 0x558b8d99d580 2024-04-25T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63e1b32640 1 --2- 172.21.15.28:0/4026349819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99dea0 0x558b8d9a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63e1331640 1 --2- 172.21.15.28:0/4026349819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d99d580 0x558b8d9b0c60 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f63cc013bc0 tx=0x7f63cc00fe40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef384ad8f49c5b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63c57fa640 1 -- 172.21.15.28:0/4026349819 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63cc004fe0 con 0x558b8d99d580 2024-04-25T01:40:59.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63c57fa640 1 -- 172.21.15.28:0/4026349819 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f63cc005180 con 0x558b8d99d580 2024-04-25T01:40:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63e6b77740 1 -- 172.21.15.28:0/4026349819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558b8d909f00 con 0x558b8d99d580 2024-04-25T01:40:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.066+0000 7f63c57fa640 1 -- 172.21.15.28:0/4026349819 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f63cc005380 con 0x558b8d99d580 2024-04-25T01:40:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.067+0000 7f63e6b77740 10 client.18633 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.067+0000 7f63e6b77740 1 -- 172.21.15.28:0/4026349819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558b8da1eac0 con 0x558b8d99d580 2024-04-25T01:40:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.067+0000 7f63c57fa640 1 -- 172.21.15.28:0/4026349819 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558b8d909f00 con 0x558b8d99d580 2024-04-25T01:40:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.067+0000 7f63c57fa640 1 -- 172.21.15.28:0/4026349819 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f63cc011560 con 0x558b8d99d580 2024-04-25T01:40:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.067+0000 7f63e6b77740 10 client.18633 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.067+0000 7f63e6b77740 10 client.18633 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:40:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.067+0000 7f63e6b77740 20 client.18633 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.067+0000 7f63e6b77740 1 --2- 172.21.15.28:0/4026349819 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x558b8da1f140 0x558b8da21560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.067+0000 7f63e6b77740 4 client.18633 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.067+0000 7f63e6b77740 1 -- 172.21.15.28:0/4026349819 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558b8d887820 con 0x558b8da1f140 2024-04-25T01:40:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.067+0000 7f63e1b32640 1 --2- 172.21.15.28:0/4026349819 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x558b8da1f140 0x558b8da21560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.068+0000 7f63e1b32640 1 --2- 172.21.15.28:0/4026349819 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x558b8da1f140 0x558b8da21560 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1eb4966bc92d205e server_cookie=8c4d8094b73583e2 in_seq=0 out_seq=0 2024-04-25T01:40:59.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.068+0000 7f63c57fa640 10 client.18633 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:40:59.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.068+0000 7f63c57fa640 1 -- 172.21.15.28:0/4026349819 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558b8d887820 con 0x558b8da1f140 2024-04-25T01:40:59.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.068+0000 7f63c57fa640 10 client.18633 handle_command_reply: tid=0 2024-04-25T01:40:59.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.068+0000 7f63e6b77740 1 client.18633 shutdown 2024-04-25T01:40:59.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.068+0000 7f63e6b77740 1 -- 172.21.15.28:0/4026349819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d99d580 msgr2=0x558b8d9b0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:59.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.068+0000 7f63e6b77740 1 --2- 172.21.15.28:0/4026349819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d99d580 0x558b8d9b0c60 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f63cc013bc0 tx=0x7f63cc00fe40 comp rx=0 tx=0).stop 2024-04-25T01:40:59.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.068+0000 7f63e6b77740 1 -- 172.21.15.28:0/4026349819 shutdown_connections 2024-04-25T01:40:59.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.068+0000 7f63e6b77740 1 --2- 172.21.15.28:0/4026349819 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x558b8da1f140 0x558b8da21560 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.068+0000 7f63e6b77740 1 --2- 172.21.15.28:0/4026349819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558b8d9a8570 0x558b8d9a24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.068+0000 7f63e6b77740 1 --2- 172.21.15.28:0/4026349819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558b8d99dea0 0x558b8d9a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.068+0000 7f63e6b77740 1 --2- 172.21.15.28:0/4026349819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b8d99d580 0x558b8d9b0c60 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.068+0000 7f63c57fa640 0 client.18633 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:40:59.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.068+0000 7f63e6b77740 1 -- 172.21.15.28:0/4026349819 >> 172.21.15.28:0/4026349819 conn(0x558b8d9998a0 msgr2=0x558b8d9982b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:59.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.068+0000 7f63e6b77740 1 -- 172.21.15.28:0/4026349819 shutdown_connections 2024-04-25T01:40:59.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.068+0000 7f63e6b77740 1 -- 172.21.15.28:0/4026349819 wait complete. 2024-04-25T01:40:59.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.068+0000 7f63e6b77740 20 client.18633 trim_cache size 0 max 16384 2024-04-25T01:40:59.061 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:59.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.069+0000 7f63e3d96640 1 -- 172.21.15.28:0/3312100701 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f63c80a2d30 msgr2=0x7f63c80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:59.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.069+0000 7f63e3d96640 1 --2- 172.21.15.28:0/3312100701 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f63c80a2d30 0x7f63c80a51f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f63d40976b0 tx=0x7f63d4075040 comp rx=0 tx=0).stop 2024-04-25T01:40:59.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.069+0000 7f63e3d96640 1 -- 172.21.15.28:0/3312100701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63dc10bd00 msgr2=0x7f63dc101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:59.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.069+0000 7f63e3d96640 1 --2- 172.21.15.28:0/3312100701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63dc10bd00 0x7f63dc101090 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f63d400d750 tx=0x7f63d400dc10 comp rx=0 tx=0).stop 2024-04-25T01:40:59.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.069+0000 7f63e3d96640 1 -- 172.21.15.28:0/3312100701 shutdown_connections 2024-04-25T01:40:59.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.069+0000 7f63e3d96640 1 --2- 172.21.15.28:0/3312100701 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f63c80a2d30 0x7f63c80a51f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.069+0000 7f63e3d96640 1 --2- 172.21.15.28:0/3312100701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63dc10bd00 0x7f63dc101090 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.069+0000 7f63e3d96640 1 --2- 172.21.15.28:0/3312100701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63dc1032e0 0x7f63dc100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.069+0000 7f63e3d96640 1 --2- 172.21.15.28:0/3312100701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f63dc1029c0 0x7f63dc100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.069+0000 7f63e3d96640 1 -- 172.21.15.28:0/3312100701 >> 172.21.15.28:0/3312100701 conn(0x7f63dc077740 msgr2=0x7f63dc0fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:59.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.069+0000 7f63e3d96640 1 -- 172.21.15.28:0/3312100701 shutdown_connections 2024-04-25T01:40:59.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.069+0000 7f63e3d96640 1 -- 172.21.15.28:0/3312100701 wait complete. 2024-04-25T01:40:59.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:59 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:40:59.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:59 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:40:59.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:59 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:40:59.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:59 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:40:59.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:59 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:40:59.066 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:40:59 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3421654448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:40:59.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:40:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.185+0000 7ff9125b7640 1 -- 172.21.15.28:0/4262974210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c10bd00 msgr2=0x7ff90c10e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.185+0000 7ff9125b7640 1 --2- 172.21.15.28:0/4262974210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c10bd00 0x7ff90c10e1a0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7ff904009ca0 tx=0x7ff90402a9e0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.185+0000 7ff9125b7640 1 -- 172.21.15.28:0/4262974210 shutdown_connections 2024-04-25T01:40:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.185+0000 7ff9125b7640 1 --2- 172.21.15.28:0/4262974210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c10bd00 0x7ff90c10e1a0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.185+0000 7ff9125b7640 1 --2- 172.21.15.28:0/4262974210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff90c1032e0 0x7ff90c10b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.185+0000 7ff9125b7640 1 --2- 172.21.15.28:0/4262974210 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff90c1029c0 0x7ff90c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.185+0000 7ff9125b7640 1 -- 172.21.15.28:0/4262974210 >> 172.21.15.28:0/4262974210 conn(0x7ff90c077740 msgr2=0x7ff90c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.185+0000 7ff9125b7640 1 -- 172.21.15.28:0/4262974210 shutdown_connections 2024-04-25T01:40:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.185+0000 7ff9125b7640 1 -- 172.21.15.28:0/4262974210 wait complete. 2024-04-25T01:40:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.185+0000 7ff9125b7640 1 Processor -- start 2024-04-25T01:40:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.185+0000 7ff9125b7640 1 -- start start 2024-04-25T01:40:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff9125b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c1029c0 0x7ff90c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff9125b7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff90c1032e0 0x7ff90c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff9125b7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff90c10bd00 0x7ff90c19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff9125b7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c111b00 con 0x7ff90c1029c0 2024-04-25T01:40:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff9125b7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c111980 con 0x7ff90c1032e0 2024-04-25T01:40:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff9125b7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c111c80 con 0x7ff90c10bd00 2024-04-25T01:40:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff90bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c1029c0 0x7ff90c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff90bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c1029c0 0x7ff90c198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48204/0 (socket says 172.21.15.28:48204) 2024-04-25T01:40:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff90bfff640 1 -- 172.21.15.28:0/310330065 learned_addr learned my addr 172.21.15.28:0/310330065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff90b7fe640 1 --2- 172.21.15.28:0/310330065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff90c1032e0 0x7ff90c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff910b54640 1 --2- 172.21.15.28:0/310330065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff90c10bd00 0x7ff90c19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff90bfff640 1 -- 172.21.15.28:0/310330065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff90c10bd00 msgr2=0x7ff90c19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff90bfff640 1 --2- 172.21.15.28:0/310330065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff90c10bd00 0x7ff90c19c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff90bfff640 1 -- 172.21.15.28:0/310330065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff90c1032e0 msgr2=0x7ff90c198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff90bfff640 1 --2- 172.21.15.28:0/310330065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff90c1032e0 0x7ff90c198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff90bfff640 1 -- 172.21.15.28:0/310330065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff90c19cfa0 con 0x7ff90c1029c0 2024-04-25T01:40:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff90bfff640 1 --2- 172.21.15.28:0/310330065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c1029c0 0x7ff90c198000 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc00d770 tx=0x7ff8fc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff9097fa640 1 -- 172.21.15.28:0/310330065 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc017070 con 0x7ff90c1029c0 2024-04-25T01:40:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff9097fa640 1 -- 172.21.15.28:0/310330065 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8fc005500 con 0x7ff90c1029c0 2024-04-25T01:40:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff9125b7640 1 -- 172.21.15.28:0/310330065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff90c19d290 con 0x7ff90c1029c0 2024-04-25T01:40:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff9097fa640 1 -- 172.21.15.28:0/310330065 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc004e00 con 0x7ff90c1029c0 2024-04-25T01:40:59.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.186+0000 7ff9125b7640 1 -- 172.21.15.28:0/310330065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff90c1a4b20 con 0x7ff90c1029c0 2024-04-25T01:40:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.189+0000 7ff9097fa640 1 -- 172.21.15.28:0/310330065 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff8fc004fa0 con 0x7ff90c1029c0 2024-04-25T01:40:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.189+0000 7ff9153a6740 1 Processor -- start 2024-04-25T01:40:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.189+0000 7ff9153a6740 1 -- start start 2024-04-25T01:40:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.189+0000 7ff9153a6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70a580 0x56474b70a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.189+0000 7ff9153a6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b70aea0 0x56474b715030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.189+0000 7ff9153a6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b715570 0x56474b717960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.189+0000 7ff9153a6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56474b66f670 con 0x56474b70a580 2024-04-25T01:40:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.189+0000 7ff9153a6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56474b617970 con 0x56474b715570 2024-04-25T01:40:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.189+0000 7ff9153a6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56474b66b630 con 0x56474b70aea0 2024-04-25T01:40:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.189+0000 7ff9097fa640 1 --2- 172.21.15.28:0/310330065 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8f40a2d00 0x7ff8f40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.189+0000 7ff910b54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70a580 0x56474b70a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.189+0000 7ff910b54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70a580 0x56474b70a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48212/0 (socket says 172.21.15.28:48212) 2024-04-25T01:40:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.189+0000 7ff910b54640 1 -- 172.21.15.28:0/4209494485 learned_addr learned my addr 172.21.15.28:0/4209494485 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.189+0000 7ff910b54640 1 --2- 172.21.15.28:0/310330065 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8f40a2d00 0x7ff8f40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.189+0000 7ff9097fa640 1 -- 172.21.15.28:0/310330065 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7ff8fc0cf310 con 0x7ff90c1029c0 2024-04-25T01:40:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.189+0000 7ff90b7fe640 1 --2- 172.21.15.28:0/4209494485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b715570 0x56474b717960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.189+0000 7ff90bfff640 1 --2- 172.21.15.28:0/4209494485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b70aea0 0x56474b715030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.190+0000 7ff910b54640 1 -- 172.21.15.28:0/4209494485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b70aea0 msgr2=0x56474b715030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.190+0000 7ff910b54640 1 --2- 172.21.15.28:0/4209494485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b70aea0 0x56474b715030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.190+0000 7ff910b54640 1 -- 172.21.15.28:0/4209494485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b715570 msgr2=0x56474b717960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.190+0000 7ff910b54640 1 --2- 172.21.15.28:0/4209494485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b715570 0x56474b717960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.190+0000 7ff910b54640 1 -- 172.21.15.28:0/4209494485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56474b63b400 con 0x56474b70a580 2024-04-25T01:40:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.190+0000 7ff90b7fe640 1 --2- 172.21.15.28:0/4209494485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b715570 0x56474b717960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.190+0000 7ff90bfff640 1 --2- 172.21.15.28:0/4209494485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b70aea0 0x56474b715030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.190+0000 7ff910b54640 1 --2- 172.21.15.28:0/4209494485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70a580 0x56474b70a960 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7ff904004af0 tx=0x7ff90402ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe52fb757aada779 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.190+0000 7ff910b54640 1 --2- 172.21.15.28:0/310330065 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8f40a2d00 0x7ff8f40a51c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff904046000 tx=0x7ff904048000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.190+0000 7ff8f27fc640 1 -- 172.21.15.28:0/4209494485 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff904041050 con 0x56474b70a580 2024-04-25T01:40:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.190+0000 7ff8f27fc640 1 -- 172.21.15.28:0/4209494485 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff90404b020 con 0x56474b70a580 2024-04-25T01:40:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.190+0000 7ff8f27fc640 1 -- 172.21.15.28:0/4209494485 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff904033050 con 0x56474b70a580 2024-04-25T01:40:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.191+0000 7ff9153a6740 1 -- 172.21.15.28:0/4209494485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70a580 msgr2=0x56474b70a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.191+0000 7ff9153a6740 1 --2- 172.21.15.28:0/4209494485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70a580 0x56474b70a960 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7ff904004af0 tx=0x7ff90402ee70 comp rx=0 tx=0).stop 2024-04-25T01:40:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.191+0000 7ff9153a6740 1 -- 172.21.15.28:0/4209494485 shutdown_connections 2024-04-25T01:40:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.191+0000 7ff9153a6740 1 --2- 172.21.15.28:0/4209494485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b715570 0x56474b717960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.191+0000 7ff9153a6740 1 --2- 172.21.15.28:0/4209494485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b70aea0 0x56474b715030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.191+0000 7ff9153a6740 1 --2- 172.21.15.28:0/4209494485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70a580 0x56474b70a960 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.191+0000 7ff9153a6740 1 -- 172.21.15.28:0/4209494485 >> 172.21.15.28:0/4209494485 conn(0x56474b7068a0 msgr2=0x56474b706c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.191+0000 7ff9153a6740 1 -- 172.21.15.28:0/4209494485 shutdown_connections 2024-04-25T01:40:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.191+0000 7ff9153a6740 1 -- 172.21.15.28:0/4209494485 wait complete. 2024-04-25T01:40:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.191+0000 7ff9153a6740 1 -- start start 2024-04-25T01:40:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.191+0000 7ff9153a6740 1 Processor -- start 2024-04-25T01:40:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff9153a6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b70a580 0x56474b714f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff9153a6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70aea0 0x56474b712240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff9153a6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b715570 0x56474b712850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff9153a6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56474b718300 con 0x56474b70aea0 2024-04-25T01:40:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff9153a6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56474b62b050 con 0x56474b715570 2024-04-25T01:40:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff9153a6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56474b718480 con 0x56474b70a580 2024-04-25T01:40:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff9153a6740 10 client.0 _ll_register_callbacks cb 0x56474b382be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff90b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70aea0 0x56474b712240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff90b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70aea0 0x56474b712240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48218/0 (socket says 172.21.15.28:48218) 2024-04-25T01:40:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff90b7fe640 1 -- 172.21.15.28:0/3327260910 learned_addr learned my addr 172.21.15.28:0/3327260910 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff90bfff640 1 --2- 172.21.15.28:0/3327260910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b70a580 0x56474b714f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff910b54640 1 --2- 172.21.15.28:0/3327260910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b715570 0x56474b712850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff90b7fe640 1 -- 172.21.15.28:0/3327260910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b70a580 msgr2=0x56474b714f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff90b7fe640 1 --2- 172.21.15.28:0/3327260910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b70a580 0x56474b714f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff90b7fe640 1 -- 172.21.15.28:0/3327260910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b715570 msgr2=0x56474b712850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff90b7fe640 1 --2- 172.21.15.28:0/3327260910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b715570 0x56474b712850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff90b7fe640 1 -- 172.21.15.28:0/3327260910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56474b5fcdc0 con 0x56474b70aea0 2024-04-25T01:40:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff910b54640 1 --2- 172.21.15.28:0/3327260910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b715570 0x56474b712850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff90b7fe640 1 --2- 172.21.15.28:0/3327260910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70aea0 0x56474b712240 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7ff900013560 tx=0x7ff900013a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=129c95050184f730 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.192+0000 7ff8eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.193+0000 7ff8eb7fe640 1 -- 172.21.15.28:0/3327260910 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff900020060 con 0x56474b70aea0 2024-04-25T01:40:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.193+0000 7ff8eb7fe640 1 -- 172.21.15.28:0/3327260910 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7ff900004460 con 0x56474b70aea0 2024-04-25T01:40:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.193+0000 7ff9153a6740 1 -- 172.21.15.28:0/3327260910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff90404b020 con 0x56474b70aea0 2024-04-25T01:40:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.193+0000 7ff8eb7fe640 1 -- 172.21.15.28:0/3327260910 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff900027050 con 0x56474b70aea0 2024-04-25T01:40:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.193+0000 7ff9153a6740 10 client.18645 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.193+0000 7ff9153a6740 1 -- 172.21.15.28:0/3327260910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56474b3833f0 con 0x56474b70aea0 2024-04-25T01:40:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.193+0000 7ff8eb7fe640 1 -- 172.21.15.28:0/3327260910 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff90404b020 con 0x56474b70aea0 2024-04-25T01:40:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.193+0000 7ff8eb7fe640 1 -- 172.21.15.28:0/3327260910 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff90001a030 con 0x56474b70aea0 2024-04-25T01:40:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.193+0000 7ff9153a6740 10 client.18645 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.193+0000 7ff9153a6740 10 client.18645 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.193+0000 7ff9153a6740 20 client.18645 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.193+0000 7ff9153a6740 1 --2- 172.21.15.28:0/3327260910 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56474b725600 0x56474b727a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.193+0000 7ff9153a6740 4 client.18645 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:40:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.193+0000 7ff9153a6740 1 -- 172.21.15.28:0/3327260910 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56474b67cbb0 con 0x56474b725600 2024-04-25T01:40:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.193+0000 7ff90bfff640 1 --2- 172.21.15.28:0/3327260910 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56474b725600 0x56474b727a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.194+0000 7ff90bfff640 1 --2- 172.21.15.28:0/3327260910 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56474b725600 0x56474b727a10 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3f60aa82c967678 server_cookie=2356360f5495f4f in_seq=0 out_seq=0 2024-04-25T01:40:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.194+0000 7ff8eb7fe640 10 client.18645 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.195+0000 7ff8eb7fe640 1 -- 172.21.15.28:0/3327260910 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56474b67cbb0 con 0x56474b725600 2024-04-25T01:40:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.195+0000 7ff8eb7fe640 10 client.18645 handle_command_reply: tid=0 2024-04-25T01:40:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.195+0000 7ff9153a6740 1 client.18645 shutdown 2024-04-25T01:40:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.195+0000 7ff9153a6740 1 -- 172.21.15.28:0/3327260910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70aea0 msgr2=0x56474b712240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.195+0000 7ff9153a6740 1 --2- 172.21.15.28:0/3327260910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70aea0 0x56474b712240 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7ff900013560 tx=0x7ff900013a20 comp rx=0 tx=0).stop 2024-04-25T01:40:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.195+0000 7ff9153a6740 1 -- 172.21.15.28:0/3327260910 shutdown_connections 2024-04-25T01:40:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.195+0000 7ff9153a6740 1 --2- 172.21.15.28:0/3327260910 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56474b725600 0x56474b727a10 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.195+0000 7ff9153a6740 1 --2- 172.21.15.28:0/3327260910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b715570 0x56474b712850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.195+0000 7ff9153a6740 1 --2- 172.21.15.28:0/3327260910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70aea0 0x56474b712240 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.195+0000 7ff9153a6740 1 --2- 172.21.15.28:0/3327260910 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b70a580 0x56474b714f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.195+0000 7ff8eb7fe640 0 client.18645 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.195+0000 7ff9153a6740 1 -- 172.21.15.28:0/3327260910 >> 172.21.15.28:0/3327260910 conn(0x56474b7068a0 msgr2=0x56474b70c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.195+0000 7ff9153a6740 1 -- 172.21.15.28:0/3327260910 shutdown_connections 2024-04-25T01:40:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.195+0000 7ff9153a6740 1 -- 172.21.15.28:0/3327260910 wait complete. 2024-04-25T01:40:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.195+0000 7ff9153a6740 20 client.18645 trim_cache size 0 max 16384 2024-04-25T01:40:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.208+0000 7ff9153a6740 1 Processor -- start 2024-04-25T01:40:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.208+0000 7ff9153a6740 1 -- start start 2024-04-25T01:40:59.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.208+0000 7ff9153a6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b70a580 0x56474b7211c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.208+0000 7ff9153a6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70aea0 0x56474b714a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.208+0000 7ff9153a6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b715570 0x56474b72d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.208+0000 7ff9153a6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c1aeb20 con 0x56474b70aea0 2024-04-25T01:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.208+0000 7ff9153a6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c1b0660 con 0x56474b70a580 2024-04-25T01:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.208+0000 7ff9153a6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c1af1f0 con 0x56474b715570 2024-04-25T01:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.209+0000 7ff90b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70aea0 0x56474b714a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.209+0000 7ff90b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70aea0 0x56474b714a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48224/0 (socket says 172.21.15.28:48224) 2024-04-25T01:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.209+0000 7ff90b7fe640 1 -- 172.21.15.28:0/3566269161 learned_addr learned my addr 172.21.15.28:0/3566269161 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.209+0000 7ff90bfff640 1 --2- 172.21.15.28:0/3566269161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b70a580 0x56474b7211c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.209+0000 7ff910b54640 1 --2- 172.21.15.28:0/3566269161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b715570 0x56474b72d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.209+0000 7ff90b7fe640 1 -- 172.21.15.28:0/3566269161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b715570 msgr2=0x56474b72d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.209+0000 7ff90b7fe640 1 --2- 172.21.15.28:0/3566269161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b715570 0x56474b72d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.209+0000 7ff90b7fe640 1 -- 172.21.15.28:0/3566269161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b70a580 msgr2=0x56474b7211c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:59.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.209+0000 7ff90b7fe640 1 --2- 172.21.15.28:0/3566269161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b70a580 0x56474b7211c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.209+0000 7ff90b7fe640 1 -- 172.21.15.28:0/3566269161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56474b44d7d0 con 0x56474b70aea0 2024-04-25T01:40:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.209+0000 7ff90b7fe640 1 --2- 172.21.15.28:0/3566269161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70aea0 0x56474b714a80 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7ff90000a010 tx=0x7ff900003fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd6f5b043dc772e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.209+0000 7ff8f1ffb640 1 -- 172.21.15.28:0/3566269161 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9000035f0 con 0x56474b70aea0 2024-04-25T01:40:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.209+0000 7ff8f1ffb640 1 -- 172.21.15.28:0/3566269161 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff900005550 con 0x56474b70aea0 2024-04-25T01:40:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.209+0000 7ff8f1ffb640 1 -- 172.21.15.28:0/3566269161 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff90000fd30 con 0x56474b70aea0 2024-04-25T01:40:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.210+0000 7ff9153a6740 1 -- 172.21.15.28:0/3566269161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70aea0 msgr2=0x56474b714a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.210+0000 7ff9153a6740 1 --2- 172.21.15.28:0/3566269161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70aea0 0x56474b714a80 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7ff90000a010 tx=0x7ff900003fe0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.210+0000 7ff9153a6740 1 -- 172.21.15.28:0/3566269161 shutdown_connections 2024-04-25T01:40:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.210+0000 7ff9153a6740 1 --2- 172.21.15.28:0/3566269161 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b715570 0x56474b72d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.210+0000 7ff9153a6740 1 --2- 172.21.15.28:0/3566269161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70aea0 0x56474b714a80 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.210+0000 7ff9153a6740 1 --2- 172.21.15.28:0/3566269161 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b70a580 0x56474b7211c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.210+0000 7ff9153a6740 1 -- 172.21.15.28:0/3566269161 >> 172.21.15.28:0/3566269161 conn(0x56474b7068a0 msgr2=0x56474b705b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.210+0000 7ff9153a6740 1 -- 172.21.15.28:0/3566269161 shutdown_connections 2024-04-25T01:40:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.210+0000 7ff9153a6740 1 -- 172.21.15.28:0/3566269161 wait complete. 2024-04-25T01:40:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.210+0000 7ff9153a6740 1 -- start start 2024-04-25T01:40:59.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.210+0000 7ff9153a6740 1 Processor -- start 2024-04-25T01:40:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff9153a6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70a580 0x56474b7194b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff9153a6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b70aea0 0x56474b7199f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff9153a6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b715570 0x56474b78ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff9153a6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c1aff90 con 0x56474b70a580 2024-04-25T01:40:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff9153a6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c1af8c0 con 0x56474b70aea0 2024-04-25T01:40:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff9153a6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c1acb20 con 0x56474b715570 2024-04-25T01:40:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff9153a6740 10 client.0 _ll_register_callbacks cb 0x7ff900009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:40:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff90bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70a580 0x56474b7194b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff90bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70a580 0x56474b7194b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48230/0 (socket says 172.21.15.28:48230) 2024-04-25T01:40:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff90bfff640 1 -- 172.21.15.28:0/1928405730 learned_addr learned my addr 172.21.15.28:0/1928405730 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:40:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff90b7fe640 1 --2- 172.21.15.28:0/1928405730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b70aea0 0x56474b7199f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff910b54640 1 --2- 172.21.15.28:0/1928405730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b715570 0x56474b78ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff90bfff640 1 -- 172.21.15.28:0/1928405730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b715570 msgr2=0x56474b78ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:59.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff90bfff640 1 --2- 172.21.15.28:0/1928405730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b715570 0x56474b78ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff90bfff640 1 -- 172.21.15.28:0/1928405730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b70aea0 msgr2=0x56474b7199f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:40:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff90bfff640 1 --2- 172.21.15.28:0/1928405730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b70aea0 0x56474b7199f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff90bfff640 1 -- 172.21.15.28:0/1928405730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56474b78b0d0 con 0x56474b70a580 2024-04-25T01:40:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff910b54640 1 --2- 172.21.15.28:0/1928405730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b715570 0x56474b78ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:40:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff90bfff640 1 --2- 172.21.15.28:0/1928405730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70a580 0x56474b7194b0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc09b430 tx=0x7ff8fc0ce040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8994a9c457dab752 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:40:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff8eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:40:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff8eb7fe640 1 -- 172.21.15.28:0/1928405730 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc020040 con 0x56474b70a580 2024-04-25T01:40:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff8eb7fe640 1 -- 172.21.15.28:0/1928405730 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7ff8fc009cc0 con 0x56474b70a580 2024-04-25T01:40:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff9153a6740 1 -- 172.21.15.28:0/1928405730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff900005550 con 0x56474b70a580 2024-04-25T01:40:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff8eb7fe640 1 -- 172.21.15.28:0/1928405730 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff8fc09f850 con 0x56474b70a580 2024-04-25T01:40:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff9153a6740 10 client.18651 fetch_fsmap learned FSMap version 28 2024-04-25T01:40:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.211+0000 7ff9153a6740 1 -- 172.21.15.28:0/1928405730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56474b78c2b0 con 0x56474b70a580 2024-04-25T01:40:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.212+0000 7ff8eb7fe640 1 -- 172.21.15.28:0/1928405730 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff900005550 con 0x56474b70a580 2024-04-25T01:40:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.212+0000 7ff8eb7fe640 1 -- 172.21.15.28:0/1928405730 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff8fc097b10 con 0x56474b70a580 2024-04-25T01:40:59.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.212+0000 7ff9153a6740 10 client.18651 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:40:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.212+0000 7ff9153a6740 10 client.18651 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:40:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.212+0000 7ff9153a6740 20 client.18651 populate_metadata read hostname 'smithi028' 2024-04-25T01:40:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.212+0000 7ff9153a6740 1 --2- 172.21.15.28:0/1928405730 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56474b78c980 0x56474b78eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:40:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.212+0000 7ff9153a6740 4 client.18651 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:40:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.212+0000 7ff9153a6740 1 -- 172.21.15.28:0/1928405730 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56474b676f00 con 0x56474b78c980 2024-04-25T01:40:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.212+0000 7ff90b7fe640 1 --2- 172.21.15.28:0/1928405730 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56474b78c980 0x56474b78eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:40:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.213+0000 7ff8eb7fe640 10 client.18651 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.213+0000 7ff90b7fe640 1 --2- 172.21.15.28:0/1928405730 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56474b78c980 0x56474b78eda0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c43f921a4226e34e server_cookie=6a089f0e0fe3d90e in_seq=0 out_seq=0 2024-04-25T01:40:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.213+0000 7ff8eb7fe640 1 -- 172.21.15.28:0/1928405730 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56474b676f00 con 0x56474b78c980 2024-04-25T01:40:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.213+0000 7ff8eb7fe640 10 client.18651 handle_command_reply: tid=0 2024-04-25T01:40:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.213+0000 7ff9153a6740 1 client.18651 shutdown 2024-04-25T01:40:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.213+0000 7ff9153a6740 1 -- 172.21.15.28:0/1928405730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70a580 msgr2=0x56474b7194b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.213+0000 7ff9153a6740 1 --2- 172.21.15.28:0/1928405730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70a580 0x56474b7194b0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc09b430 tx=0x7ff8fc0ce040 comp rx=0 tx=0).stop 2024-04-25T01:40:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.213+0000 7ff9153a6740 1 -- 172.21.15.28:0/1928405730 shutdown_connections 2024-04-25T01:40:59.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.213+0000 7ff9153a6740 1 --2- 172.21.15.28:0/1928405730 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56474b78c980 0x56474b78eda0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.213+0000 7ff9153a6740 1 --2- 172.21.15.28:0/1928405730 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56474b715570 0x56474b78ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.213+0000 7ff9153a6740 1 --2- 172.21.15.28:0/1928405730 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56474b70aea0 0x56474b7199f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.213+0000 7ff8eb7fe640 0 client.18651 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:40:59.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.213+0000 7ff9153a6740 1 --2- 172.21.15.28:0/1928405730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56474b70a580 0x56474b7194b0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.213+0000 7ff9153a6740 1 -- 172.21.15.28:0/1928405730 >> 172.21.15.28:0/1928405730 conn(0x56474b7068a0 msgr2=0x56474b7201b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:59.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.213+0000 7ff9153a6740 1 -- 172.21.15.28:0/1928405730 shutdown_connections 2024-04-25T01:40:59.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.213+0000 7ff9153a6740 1 -- 172.21.15.28:0/1928405730 wait complete. 2024-04-25T01:40:59.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.213+0000 7ff9153a6740 20 client.18651 trim_cache size 0 max 16384 2024-04-25T01:40:59.205 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:40:59.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.214+0000 7ff9125b7640 1 -- 172.21.15.28:0/310330065 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8f40a2d00 msgr2=0x7ff8f40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:59.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.214+0000 7ff9125b7640 1 --2- 172.21.15.28:0/310330065 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8f40a2d00 0x7ff8f40a51c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff904046000 tx=0x7ff904048000 comp rx=0 tx=0).stop 2024-04-25T01:40:59.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.214+0000 7ff9125b7640 1 -- 172.21.15.28:0/310330065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c1029c0 msgr2=0x7ff90c198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:40:59.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.214+0000 7ff9125b7640 1 --2- 172.21.15.28:0/310330065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c1029c0 0x7ff90c198000 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc00d770 tx=0x7ff8fc00dc30 comp rx=0 tx=0).stop 2024-04-25T01:40:59.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.214+0000 7ff9125b7640 1 -- 172.21.15.28:0/310330065 shutdown_connections 2024-04-25T01:40:59.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.214+0000 7ff9125b7640 1 --2- 172.21.15.28:0/310330065 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8f40a2d00 0x7ff8f40a51c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.214+0000 7ff9125b7640 1 --2- 172.21.15.28:0/310330065 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff90c10bd00 0x7ff90c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.214+0000 7ff9125b7640 1 --2- 172.21.15.28:0/310330065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff90c1032e0 0x7ff90c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.214+0000 7ff9125b7640 1 --2- 172.21.15.28:0/310330065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c1029c0 0x7ff90c198000 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:40:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.214+0000 7ff9125b7640 1 -- 172.21.15.28:0/310330065 >> 172.21.15.28:0/310330065 conn(0x7ff90c077740 msgr2=0x7ff90c0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:40:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.214+0000 7ff9125b7640 1 -- 172.21.15.28:0/310330065 shutdown_connections 2024-04-25T01:40:59.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:40:59.214+0000 7ff9125b7640 1 -- 172.21.15.28:0/310330065 wait complete. 2024-04-25T01:40:59.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:59 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:40:59.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:59 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:40:59.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:59 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:40:59.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:59 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:40:59.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:59 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:40:59.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:40:59 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3421654448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:40:59.492 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:59 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:40:59.493 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:59 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:40:59.493 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:59 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:40:59.493 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:59 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:40:59.493 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:59 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:40:59.493 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:40:59 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3421654448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:41:00.211 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:41:00.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:41:00.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:41:00.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:00 smithi028 ceph-mon[29390]: pgmap v244: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:41:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.345+0000 7f32efb09640 1 -- 172.21.15.28:0/2111670460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32e81066b0 msgr2=0x7f32e8106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.345+0000 7f32efb09640 1 --2- 172.21.15.28:0/2111670460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32e81066b0 0x7f32e8106a90 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f32e0009930 tx=0x7f32e002a420 comp rx=0 tx=0).stop 2024-04-25T01:41:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.345+0000 7f32efb09640 1 -- 172.21.15.28:0/2111670460 shutdown_connections 2024-04-25T01:41:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.345+0000 7f32efb09640 1 --2- 172.21.15.28:0/2111670460 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f32e8100fd0 0x7f32e810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.345+0000 7f32efb09640 1 --2- 172.21.15.28:0/2111670460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32e8100610 0x7f32e8100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.345+0000 7f32efb09640 1 --2- 172.21.15.28:0/2111670460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32e81066b0 0x7f32e8106a90 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.345+0000 7f32efb09640 1 -- 172.21.15.28:0/2111670460 >> 172.21.15.28:0/2111670460 conn(0x7f32e8077740 msgr2=0x7f32e8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.345+0000 7f32efb09640 1 -- 172.21.15.28:0/2111670460 shutdown_connections 2024-04-25T01:41:00.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.345+0000 7f32efb09640 1 -- 172.21.15.28:0/2111670460 wait complete. 2024-04-25T01:41:00.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.345+0000 7f32efb09640 1 Processor -- start 2024-04-25T01:41:00.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32efb09640 1 -- start start 2024-04-25T01:41:00.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32efb09640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f32e8100610 0x7f32e81022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32efb09640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32e8100fd0 0x7f32e81027f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32efb09640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32e81066b0 0x7f32e8102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32efb09640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81109f0 con 0x7f32e81066b0 2024-04-25T01:41:00.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32efb09640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f32e8110870 con 0x7f32e8100fd0 2024-04-25T01:41:00.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32efb09640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f32e8110b70 con 0x7f32e8100610 2024-04-25T01:41:00.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32ee0a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32e81066b0 0x7f32e8102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32ee0a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32e81066b0 0x7f32e8102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48248/0 (socket says 172.21.15.28:48248) 2024-04-25T01:41:00.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32ee0a6640 1 -- 172.21.15.28:0/1532373653 learned_addr learned my addr 172.21.15.28:0/1532373653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:00.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32ed8a5640 1 --2- 172.21.15.28:0/1532373653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f32e8100610 0x7f32e81022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32ed0a4640 1 --2- 172.21.15.28:0/1532373653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32e8100fd0 0x7f32e81027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32ee0a6640 1 -- 172.21.15.28:0/1532373653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f32e8100610 msgr2=0x7f32e81022b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:00.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32ee0a6640 1 --2- 172.21.15.28:0/1532373653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f32e8100610 0x7f32e81022b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32ee0a6640 1 -- 172.21.15.28:0/1532373653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32e8100fd0 msgr2=0x7f32e81027f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:00.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32ee0a6640 1 --2- 172.21.15.28:0/1532373653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32e8100fd0 0x7f32e81027f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32ee0a6640 1 -- 172.21.15.28:0/1532373653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e81a6490 con 0x7f32e81066b0 2024-04-25T01:41:00.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32ed8a5640 1 --2- 172.21.15.28:0/1532373653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f32e8100610 0x7f32e81022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:00.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32ed0a4640 1 --2- 172.21.15.28:0/1532373653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32e8100fd0 0x7f32e81027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:41:00.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.346+0000 7f32ee0a6640 1 --2- 172.21.15.28:0/1532373653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32e81066b0 0x7f32e8102d30 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f32e400eaa0 tx=0x7f32e400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:00.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.347+0000 7f32deffd640 1 -- 172.21.15.28:0/1532373653 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e400cdb0 con 0x7f32e81066b0 2024-04-25T01:41:00.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.347+0000 7f32efb09640 1 -- 172.21.15.28:0/1532373653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32e81a6780 con 0x7f32e81066b0 2024-04-25T01:41:00.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.347+0000 7f32deffd640 1 -- 172.21.15.28:0/1532373653 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32e4005020 con 0x7f32e81066b0 2024-04-25T01:41:00.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.347+0000 7f32efb09640 1 -- 172.21.15.28:0/1532373653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32e81a6c80 con 0x7f32e81066b0 2024-04-25T01:41:00.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.347+0000 7f32deffd640 1 -- 172.21.15.28:0/1532373653 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e401f060 con 0x7f32e81066b0 2024-04-25T01:41:00.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.349+0000 7f32deffd640 1 -- 172.21.15.28:0/1532373653 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f32e4005560 con 0x7f32e81066b0 2024-04-25T01:41:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.350+0000 7f32f2811740 1 Processor -- start 2024-04-25T01:41:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.350+0000 7f32f2811740 1 -- start start 2024-04-25T01:41:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.350+0000 7f32f2811740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624dabbb980 0x5624dabb9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.350+0000 7f32f2811740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 0x5624dabba710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.350+0000 7f32f2811740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabc5010 0x5624dabc7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.350+0000 7f32f2811740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5624dab1e2c0 con 0x5624dabba290 2024-04-25T01:41:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.350+0000 7f32f2811740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5624daac6910 con 0x5624dabc5010 2024-04-25T01:41:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.350+0000 7f32f2811740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5624dab1a280 con 0x5624dabbb980 2024-04-25T01:41:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.350+0000 7f32deffd640 1 --2- 172.21.15.28:0/1532373653 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f32d40a2d00 0x7f32d40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.350+0000 7f32ed8a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 0x5624dabba710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.350+0000 7f32ed8a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 0x5624dabba710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48252/0 (socket says 172.21.15.28:48252) 2024-04-25T01:41:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.350+0000 7f32ed8a5640 1 -- 172.21.15.28:0/4184613880 learned_addr learned my addr 172.21.15.28:0/4184613880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.350+0000 7f32ed0a4640 1 --2- 172.21.15.28:0/4184613880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabc5010 0x5624dabc7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.350+0000 7f32ee0a6640 1 --2- 172.21.15.28:0/1532373653 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f32d40a2d00 0x7f32d40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.350+0000 7f32ed0a4640 1 --2- 172.21.15.28:0/4184613880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624dabbb980 0x5624dabb9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.351+0000 7f32deffd640 1 -- 172.21.15.28:0/1532373653 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f32e40cfa30 con 0x7f32e81066b0 2024-04-25T01:41:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.351+0000 7f32ed8a5640 1 -- 172.21.15.28:0/4184613880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624dabbb980 msgr2=0x5624dabb9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.351+0000 7f32ed8a5640 1 --2- 172.21.15.28:0/4184613880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624dabbb980 0x5624dabb9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.351+0000 7f32ed8a5640 1 -- 172.21.15.28:0/4184613880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabc5010 msgr2=0x5624dabc7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.351+0000 7f32ed8a5640 1 --2- 172.21.15.28:0/4184613880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabc5010 0x5624dabc7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.351+0000 7f32ed8a5640 1 -- 172.21.15.28:0/4184613880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624daad7fb0 con 0x5624dabba290 2024-04-25T01:41:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.351+0000 7f32ee0a6640 1 --2- 172.21.15.28:0/1532373653 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f32d40a2d00 0x7f32d40a51c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f32e40092f0 tx=0x7f32e4091050 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.351+0000 7f32ed8a5640 1 --2- 172.21.15.28:0/4184613880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 0x5624dabba710 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f32e0005c40 tx=0x7f32e0032cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b3921653a9c0c6c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.351+0000 7f32d3fff640 1 -- 172.21.15.28:0/4184613880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e003f070 con 0x5624dabba290 2024-04-25T01:41:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.351+0000 7f32d3fff640 1 -- 172.21.15.28:0/4184613880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32e0032e20 con 0x5624dabba290 2024-04-25T01:41:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.351+0000 7f32d3fff640 1 -- 172.21.15.28:0/4184613880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e0035070 con 0x5624dabba290 2024-04-25T01:41:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.352+0000 7f32f2811740 1 -- 172.21.15.28:0/4184613880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 msgr2=0x5624dabba710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.352+0000 7f32f2811740 1 --2- 172.21.15.28:0/4184613880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 0x5624dabba710 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f32e0005c40 tx=0x7f32e0032cb0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.352+0000 7f32f2811740 1 -- 172.21.15.28:0/4184613880 shutdown_connections 2024-04-25T01:41:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.352+0000 7f32f2811740 1 --2- 172.21.15.28:0/4184613880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabc5010 0x5624dabc7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.352+0000 7f32f2811740 1 --2- 172.21.15.28:0/4184613880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 0x5624dabba710 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.352+0000 7f32f2811740 1 --2- 172.21.15.28:0/4184613880 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624dabbb980 0x5624dabb9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.352+0000 7f32f2811740 1 -- 172.21.15.28:0/4184613880 >> 172.21.15.28:0/4184613880 conn(0x5624dab82950 msgr2=0x5624dabb67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.352+0000 7f32f2811740 1 -- 172.21.15.28:0/4184613880 shutdown_connections 2024-04-25T01:41:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.352+0000 7f32f2811740 1 -- 172.21.15.28:0/4184613880 wait complete. 2024-04-25T01:41:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.352+0000 7f32f2811740 1 -- start start 2024-04-25T01:41:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.353+0000 7f32f2811740 1 Processor -- start 2024-04-25T01:41:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.353+0000 7f32f2811740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624dabba290 0x5624dabc1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.353+0000 7f32f2811740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabbb980 0x5624dabc24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.353+0000 7f32f2811740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabc5010 0x5624dabc2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.353+0000 7f32f2811740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5624dabc7e40 con 0x5624dabc5010 2024-04-25T01:41:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.353+0000 7f32f2811740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5624daad9e90 con 0x5624dabbb980 2024-04-25T01:41:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.353+0000 7f32f2811740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5624dabc7fc0 con 0x5624dabba290 2024-04-25T01:41:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.353+0000 7f32f2811740 10 client.0 _ll_register_callbacks cb 0x5624dab15970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.353+0000 7f32ed0a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabc5010 0x5624dabc2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.353+0000 7f32ed0a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabc5010 0x5624dabc2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48268/0 (socket says 172.21.15.28:48268) 2024-04-25T01:41:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.353+0000 7f32ed0a4640 1 -- 172.21.15.28:0/4229518352 learned_addr learned my addr 172.21.15.28:0/4229518352 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.353+0000 7f32ed0a4640 1 --2- 172.21.15.28:0/4229518352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624dabba290 0x5624dabc1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.353+0000 7f32ed8a5640 1 --2- 172.21.15.28:0/4229518352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabbb980 0x5624dabc24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.353+0000 7f32ed0a4640 1 -- 172.21.15.28:0/4229518352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624dabba290 msgr2=0x5624dabc1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.353+0000 7f32ed0a4640 1 --2- 172.21.15.28:0/4229518352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624dabba290 0x5624dabc1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.353+0000 7f32ed0a4640 1 -- 172.21.15.28:0/4229518352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabbb980 msgr2=0x5624dabc24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.353+0000 7f32ed0a4640 1 --2- 172.21.15.28:0/4229518352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabbb980 0x5624dabc24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.353+0000 7f32ed0a4640 1 -- 172.21.15.28:0/4229518352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624da8fa7e0 con 0x5624dabc5010 2024-04-25T01:41:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.354+0000 7f32ed0a4640 1 --2- 172.21.15.28:0/4229518352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabc5010 0x5624dabc2a20 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f32d8013f10 tx=0x7f32d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f67aa0e6013243 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.354+0000 7f32d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.354+0000 7f32d17fa640 1 -- 172.21.15.28:0/4229518352 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8004020 con 0x5624dabc5010 2024-04-25T01:41:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.354+0000 7f32d17fa640 1 -- 172.21.15.28:0/4229518352 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f32d80041c0 con 0x5624dabc5010 2024-04-25T01:41:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.354+0000 7f32f2811740 1 -- 172.21.15.28:0/4229518352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32e0032e20 con 0x5624dabc5010 2024-04-25T01:41:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.354+0000 7f32d17fa640 1 -- 172.21.15.28:0/4229518352 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f32d802a050 con 0x5624dabc5010 2024-04-25T01:41:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.354+0000 7f32f2811740 10 client.18663 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.354+0000 7f32f2811740 1 -- 172.21.15.28:0/4229518352 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5624da95af70 con 0x5624dabc5010 2024-04-25T01:41:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.354+0000 7f32d17fa640 1 -- 172.21.15.28:0/4229518352 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32e0032e20 con 0x5624dabc5010 2024-04-25T01:41:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.354+0000 7f32d17fa640 1 -- 172.21.15.28:0/4229518352 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f32d800f040 con 0x5624dabc5010 2024-04-25T01:41:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.355+0000 7f32f2811740 10 client.18663 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.355+0000 7f32f2811740 10 client.18663 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:41:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.355+0000 7f32f2811740 20 client.18663 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.355+0000 7f32f2811740 1 --2- 172.21.15.28:0/4229518352 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5624dabd5170 0x5624dabd7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.355+0000 7f32f2811740 4 client.18663 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:41:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.355+0000 7f32f2811740 1 -- 172.21.15.28:0/4229518352 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624daaf25a0 con 0x5624dabd5170 2024-04-25T01:41:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.355+0000 7f32ed8a5640 1 --2- 172.21.15.28:0/4229518352 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5624dabd5170 0x5624dabd7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.355+0000 7f32ed8a5640 1 --2- 172.21.15.28:0/4229518352 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5624dabd5170 0x5624dabd7560 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cfb79180faaf727 server_cookie=611809c95df65af0 in_seq=0 out_seq=0 2024-04-25T01:41:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.355+0000 7f32d17fa640 10 client.18663 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.356+0000 7f32d17fa640 1 -- 172.21.15.28:0/4229518352 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5624daaf25a0 con 0x5624dabd5170 2024-04-25T01:41:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.356+0000 7f32d17fa640 10 client.18663 handle_command_reply: tid=0 2024-04-25T01:41:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.356+0000 7f32f2811740 1 client.18663 shutdown 2024-04-25T01:41:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.356+0000 7f32f2811740 1 -- 172.21.15.28:0/4229518352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabc5010 msgr2=0x5624dabc2a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.356+0000 7f32f2811740 1 --2- 172.21.15.28:0/4229518352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabc5010 0x5624dabc2a20 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f32d8013f10 tx=0x7f32d8004730 comp rx=0 tx=0).stop 2024-04-25T01:41:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.356+0000 7f32f2811740 1 -- 172.21.15.28:0/4229518352 shutdown_connections 2024-04-25T01:41:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.356+0000 7f32f2811740 1 --2- 172.21.15.28:0/4229518352 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5624dabd5170 0x5624dabd7560 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.356+0000 7f32f2811740 1 --2- 172.21.15.28:0/4229518352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabc5010 0x5624dabc2a20 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.356+0000 7f32f2811740 1 --2- 172.21.15.28:0/4229518352 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabbb980 0x5624dabc24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.356+0000 7f32f2811740 1 --2- 172.21.15.28:0/4229518352 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624dabba290 0x5624dabc1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.356+0000 7f32d17fa640 0 client.18663 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.356+0000 7f32f2811740 1 -- 172.21.15.28:0/4229518352 >> 172.21.15.28:0/4229518352 conn(0x5624dab82950 msgr2=0x5624dabbc660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.356+0000 7f32f2811740 1 -- 172.21.15.28:0/4229518352 shutdown_connections 2024-04-25T01:41:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.357+0000 7f32f2811740 1 -- 172.21.15.28:0/4229518352 wait complete. 2024-04-25T01:41:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.357+0000 7f32f2811740 20 client.18663 trim_cache size 0 max 16384 2024-04-25T01:41:00.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.375+0000 7f32f2811740 1 Processor -- start 2024-04-25T01:41:00.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.375+0000 7f32f2811740 1 -- start start 2024-04-25T01:41:00.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.375+0000 7f32f2811740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 0x5624dabd0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.375+0000 7f32f2811740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabbb980 0x5624dabb57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.375+0000 7f32f2811740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624dabc5010 0x5624dabdce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.375+0000 7f32f2811740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81b0d10 con 0x5624dabba290 2024-04-25T01:41:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.375+0000 7f32f2811740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81b2850 con 0x5624dabbb980 2024-04-25T01:41:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.375+0000 7f32f2811740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81b13e0 con 0x5624dabc5010 2024-04-25T01:41:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.375+0000 7f32ed0a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 0x5624dabd0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.375+0000 7f32ed0a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 0x5624dabd0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48282/0 (socket says 172.21.15.28:48282) 2024-04-25T01:41:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.375+0000 7f32ed0a4640 1 -- 172.21.15.28:0/1629266048 learned_addr learned my addr 172.21.15.28:0/1629266048 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.375+0000 7f32ed8a5640 1 --2- 172.21.15.28:0/1629266048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabbb980 0x5624dabb57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.375+0000 7f32ed0a4640 1 -- 172.21.15.28:0/1629266048 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624dabc5010 msgr2=0x5624dabdce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.375+0000 7f32ed0a4640 1 --2- 172.21.15.28:0/1629266048 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624dabc5010 0x5624dabdce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.375+0000 7f32ed0a4640 1 -- 172.21.15.28:0/1629266048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabbb980 msgr2=0x5624dabb57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.375+0000 7f32ed0a4640 1 --2- 172.21.15.28:0/1629266048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabbb980 0x5624dabb57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.375+0000 7f32ed0a4640 1 -- 172.21.15.28:0/1629266048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624dabd78d0 con 0x5624dabba290 2024-04-25T01:41:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.375+0000 7f32ed0a4640 1 --2- 172.21.15.28:0/1629266048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 0x5624dabd0d40 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f32d8013ce0 tx=0x7f32d8005cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=333073c09741047 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.375+0000 7f32d37fe640 1 -- 172.21.15.28:0/1629266048 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d80136e0 con 0x5624dabba290 2024-04-25T01:41:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.376+0000 7f32d37fe640 1 -- 172.21.15.28:0/1629266048 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32d8003880 con 0x5624dabba290 2024-04-25T01:41:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.376+0000 7f32d37fe640 1 -- 172.21.15.28:0/1629266048 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8003b80 con 0x5624dabba290 2024-04-25T01:41:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.376+0000 7f32f2811740 1 -- 172.21.15.28:0/1629266048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 msgr2=0x5624dabd0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.376+0000 7f32f2811740 1 --2- 172.21.15.28:0/1629266048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 0x5624dabd0d40 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f32d8013ce0 tx=0x7f32d8005cb0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.376+0000 7f32f2811740 1 -- 172.21.15.28:0/1629266048 shutdown_connections 2024-04-25T01:41:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.376+0000 7f32f2811740 1 --2- 172.21.15.28:0/1629266048 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624dabc5010 0x5624dabdce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.376+0000 7f32f2811740 1 --2- 172.21.15.28:0/1629266048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabbb980 0x5624dabb57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.376+0000 7f32f2811740 1 --2- 172.21.15.28:0/1629266048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 0x5624dabd0d40 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.376+0000 7f32f2811740 1 -- 172.21.15.28:0/1629266048 >> 172.21.15.28:0/1629266048 conn(0x5624dab82950 msgr2=0x5624dabb50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.376+0000 7f32f2811740 1 -- 172.21.15.28:0/1629266048 shutdown_connections 2024-04-25T01:41:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.376+0000 7f32f2811740 1 -- 172.21.15.28:0/1629266048 wait complete. 2024-04-25T01:41:00.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.376+0000 7f32f2811740 1 -- start start 2024-04-25T01:41:00.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.377+0000 7f32f2811740 1 Processor -- start 2024-04-25T01:41:00.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.377+0000 7f32f2811740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 0x5624dabc8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.377+0000 7f32f2811740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabbb980 0x5624dabc9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.377+0000 7f32f2811740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624dabc5010 0x5624dabc98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.377+0000 7f32f2811740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81b2180 con 0x5624dabba290 2024-04-25T01:41:00.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.377+0000 7f32f2811740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81b1ab0 con 0x5624dabbb980 2024-04-25T01:41:00.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.377+0000 7f32f2811740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81aed10 con 0x5624dabc5010 2024-04-25T01:41:00.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.377+0000 7f32f2811740 10 client.0 _ll_register_callbacks cb 0x7f32d8013460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:00.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.377+0000 7f32ed0a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 0x5624dabc8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.377+0000 7f32ed0a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 0x5624dabc8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48284/0 (socket says 172.21.15.28:48284) 2024-04-25T01:41:00.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.377+0000 7f32ed0a4640 1 -- 172.21.15.28:0/2870902644 learned_addr learned my addr 172.21.15.28:0/2870902644 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:00.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.377+0000 7f32ed0a4640 1 -- 172.21.15.28:0/2870902644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624dabc5010 msgr2=0x5624dabc98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:00.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.377+0000 7f32ed0a4640 1 --2- 172.21.15.28:0/2870902644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624dabc5010 0x5624dabc98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.377+0000 7f32ed8a5640 1 --2- 172.21.15.28:0/2870902644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabbb980 0x5624dabc9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.377+0000 7f32ed0a4640 1 -- 172.21.15.28:0/2870902644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabbb980 msgr2=0x5624dabc9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.377+0000 7f32ed0a4640 1 --2- 172.21.15.28:0/2870902644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabbb980 0x5624dabc9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.377+0000 7f32ed0a4640 1 -- 172.21.15.28:0/2870902644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624da7f2170 con 0x5624dabba290 2024-04-25T01:41:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.378+0000 7f32ed0a4640 1 --2- 172.21.15.28:0/2870902644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 0x5624dabc8e40 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f32d8012010 tx=0x7f32d8005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e2581b1a8375d18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.378+0000 7f32d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.378+0000 7f32d17fa640 1 -- 172.21.15.28:0/2870902644 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d80107f0 con 0x5624dabba290 2024-04-25T01:41:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.378+0000 7f32d17fa640 1 -- 172.21.15.28:0/2870902644 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7f32d8010990 con 0x5624dabba290 2024-04-25T01:41:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.378+0000 7f32f2811740 1 -- 172.21.15.28:0/2870902644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32d8003880 con 0x5624dabba290 2024-04-25T01:41:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.378+0000 7f32d17fa640 1 -- 172.21.15.28:0/2870902644 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f32d8013d80 con 0x5624dabba290 2024-04-25T01:41:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.378+0000 7f32f2811740 10 client.18669 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.378+0000 7f32f2811740 1 -- 172.21.15.28:0/2870902644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5624dac3b9d0 con 0x5624dabba290 2024-04-25T01:41:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.378+0000 7f32d17fa640 1 -- 172.21.15.28:0/2870902644 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32d8003880 con 0x5624dabba290 2024-04-25T01:41:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.378+0000 7f32d17fa640 1 -- 172.21.15.28:0/2870902644 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f32d8004290 con 0x5624dabba290 2024-04-25T01:41:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.378+0000 7f32f2811740 10 client.18669 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:00.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.378+0000 7f32f2811740 10 client.18669 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:41:00.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.378+0000 7f32f2811740 20 client.18669 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:00.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.378+0000 7f32f2811740 1 --2- 172.21.15.28:0/2870902644 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5624dac3bfd0 0x5624dac3e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.378+0000 7f32f2811740 4 client.18669 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:41:00.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.378+0000 7f32f2811740 1 -- 172.21.15.28:0/2870902644 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5624dab2b800 con 0x5624dac3bfd0 2024-04-25T01:41:00.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.379+0000 7f32ed8a5640 1 --2- 172.21.15.28:0/2870902644 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5624dac3bfd0 0x5624dac3e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.379+0000 7f32ed8a5640 1 --2- 172.21.15.28:0/2870902644 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5624dac3bfd0 0x5624dac3e3c0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=502a571256ccfbad server_cookie=4dcae250ce7634cf in_seq=0 out_seq=0 2024-04-25T01:41:00.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.379+0000 7f32d17fa640 10 client.18669 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:00.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.380+0000 7f32d17fa640 1 -- 172.21.15.28:0/2870902644 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5624dab2b800 con 0x5624dac3bfd0 2024-04-25T01:41:00.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.380+0000 7f32d17fa640 10 client.18669 handle_command_reply: tid=0 2024-04-25T01:41:00.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.380+0000 7f32f2811740 1 client.18669 shutdown 2024-04-25T01:41:00.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.380+0000 7f32f2811740 1 -- 172.21.15.28:0/2870902644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 msgr2=0x5624dabc8e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:00.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.380+0000 7f32f2811740 1 --2- 172.21.15.28:0/2870902644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 0x5624dabc8e40 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f32d8012010 tx=0x7f32d8005850 comp rx=0 tx=0).stop 2024-04-25T01:41:00.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.380+0000 7f32f2811740 1 -- 172.21.15.28:0/2870902644 shutdown_connections 2024-04-25T01:41:00.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.380+0000 7f32f2811740 1 --2- 172.21.15.28:0/2870902644 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5624dac3bfd0 0x5624dac3e3c0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.380+0000 7f32f2811740 1 --2- 172.21.15.28:0/2870902644 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5624dabc5010 0x5624dabc98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.380+0000 7f32f2811740 1 --2- 172.21.15.28:0/2870902644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5624dabbb980 0x5624dabc9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.380+0000 7f32d17fa640 0 client.18669 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:00.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.380+0000 7f32f2811740 1 --2- 172.21.15.28:0/2870902644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624dabba290 0x5624dabc8e40 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.380+0000 7f32f2811740 1 -- 172.21.15.28:0/2870902644 >> 172.21.15.28:0/2870902644 conn(0x5624dab82950 msgr2=0x5624dabb5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:00.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.380+0000 7f32f2811740 1 -- 172.21.15.28:0/2870902644 shutdown_connections 2024-04-25T01:41:00.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.380+0000 7f32f2811740 1 -- 172.21.15.28:0/2870902644 wait complete. 2024-04-25T01:41:00.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.381+0000 7f32f2811740 20 client.18669 trim_cache size 0 max 16384 2024-04-25T01:41:00.373 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:41:00.373 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:41:00.373 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "2fe0900b-b8b6-4a5c-bff3-421cc5d3103b", 2024-04-25T01:41:00.373 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:41:00.373 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:41:00.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.381+0000 7f32efb09640 1 -- 172.21.15.28:0/1532373653 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f32d40a2d00 msgr2=0x7f32d40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:00.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.381+0000 7f32efb09640 1 --2- 172.21.15.28:0/1532373653 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f32d40a2d00 0x7f32d40a51c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f32e40092f0 tx=0x7f32e4091050 comp rx=0 tx=0).stop 2024-04-25T01:41:00.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.381+0000 7f32efb09640 1 -- 172.21.15.28:0/1532373653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32e81066b0 msgr2=0x7f32e8102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:00.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.381+0000 7f32efb09640 1 --2- 172.21.15.28:0/1532373653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32e81066b0 0x7f32e8102d30 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f32e400eaa0 tx=0x7f32e400ef60 comp rx=0 tx=0).stop 2024-04-25T01:41:00.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.381+0000 7f32efb09640 1 -- 172.21.15.28:0/1532373653 shutdown_connections 2024-04-25T01:41:00.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.381+0000 7f32efb09640 1 --2- 172.21.15.28:0/1532373653 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f32d40a2d00 0x7f32d40a51c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.381+0000 7f32efb09640 1 --2- 172.21.15.28:0/1532373653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32e81066b0 0x7f32e8102d30 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.381+0000 7f32efb09640 1 --2- 172.21.15.28:0/1532373653 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32e8100fd0 0x7f32e81027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.381+0000 7f32efb09640 1 --2- 172.21.15.28:0/1532373653 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f32e8100610 0x7f32e81022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.381+0000 7f32efb09640 1 -- 172.21.15.28:0/1532373653 >> 172.21.15.28:0/1532373653 conn(0x7f32e8077740 msgr2=0x7f32e80ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:00.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.381+0000 7f32efb09640 1 -- 172.21.15.28:0/1532373653 shutdown_connections 2024-04-25T01:41:00.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.381+0000 7f32efb09640 1 -- 172.21.15.28:0/1532373653 wait complete. 2024-04-25T01:41:00.378 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:41:00.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:41:00.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:00 smithi130 ceph-mon[32702]: pgmap v244: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:41:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.498+0000 7fd7bdbee640 1 -- 172.21.15.28:0/4069691129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b81020d0 msgr2=0x7fd7b810e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.498+0000 7fd7bdbee640 1 --2- 172.21.15.28:0/4069691129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b81020d0 0x7fd7b810e5e0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0009930 tx=0x7fd7b002a420 comp rx=0 tx=0).stop 2024-04-25T01:41:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.498+0000 7fd7bdbee640 1 -- 172.21.15.28:0/4069691129 shutdown_connections 2024-04-25T01:41:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.498+0000 7fd7bdbee640 1 --2- 172.21.15.28:0/4069691129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b81020d0 0x7fd7b810e5e0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.498+0000 7fd7bdbee640 1 --2- 172.21.15.28:0/4069691129 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7b8101710 0x7fd7b8101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.498+0000 7fd7bdbee640 1 --2- 172.21.15.28:0/4069691129 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b8107720 0x7fd7b8107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.498+0000 7fd7bdbee640 1 -- 172.21.15.28:0/4069691129 >> 172.21.15.28:0/4069691129 conn(0x7fd7b80fd180 msgr2=0x7fd7b80ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.498+0000 7fd7bdbee640 1 -- 172.21.15.28:0/4069691129 shutdown_connections 2024-04-25T01:41:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.498+0000 7fd7bdbee640 1 -- 172.21.15.28:0/4069691129 wait complete. 2024-04-25T01:41:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.499+0000 7fd7bdbee640 1 Processor -- start 2024-04-25T01:41:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.499+0000 7fd7bdbee640 1 -- start start 2024-04-25T01:41:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.499+0000 7fd7bdbee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b8101710 0x7fd7b8198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.499+0000 7fd7bdbee640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b81020d0 0x7fd7b81985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.499+0000 7fd7bdbee640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7b8107720 0x7fd7b819c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.499+0000 7fd7bdbee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b8111b30 con 0x7fd7b8101710 2024-04-25T01:41:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.499+0000 7fd7bdbee640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b81119b0 con 0x7fd7b81020d0 2024-04-25T01:41:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.499+0000 7fd7bdbee640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b8111cb0 con 0x7fd7b8107720 2024-04-25T01:41:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.499+0000 7fd7b77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b8101710 0x7fd7b8198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.499+0000 7fd7b77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b8101710 0x7fd7b8198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48302/0 (socket says 172.21.15.28:48302) 2024-04-25T01:41:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.499+0000 7fd7b77fe640 1 -- 172.21.15.28:0/1492607799 learned_addr learned my addr 172.21.15.28:0/1492607799 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.499+0000 7fd7b7fff640 1 --2- 172.21.15.28:0/1492607799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7b8107720 0x7fd7b819c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.500+0000 7fd7b6ffd640 1 --2- 172.21.15.28:0/1492607799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b81020d0 0x7fd7b81985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.500+0000 7fd7b77fe640 1 -- 172.21.15.28:0/1492607799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7b8107720 msgr2=0x7fd7b819c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.500+0000 7fd7b77fe640 1 --2- 172.21.15.28:0/1492607799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7b8107720 0x7fd7b819c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.500+0000 7fd7b77fe640 1 -- 172.21.15.28:0/1492607799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b81020d0 msgr2=0x7fd7b81985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.500+0000 7fd7b77fe640 1 --2- 172.21.15.28:0/1492607799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b81020d0 0x7fd7b81985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.500+0000 7fd7b77fe640 1 -- 172.21.15.28:0/1492607799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b819d060 con 0x7fd7b8101710 2024-04-25T01:41:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.500+0000 7fd7b77fe640 1 --2- 172.21.15.28:0/1492607799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b8101710 0x7fd7b8198070 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800ebc0 tx=0x7fd7a800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:00.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.500+0000 7fd7b4ff9640 1 -- 172.21.15.28:0/1492607799 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8010630 con 0x7fd7b8101710 2024-04-25T01:41:00.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.500+0000 7fd7b4ff9640 1 -- 172.21.15.28:0/1492607799 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7a8010cb0 con 0x7fd7b8101710 2024-04-25T01:41:00.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.500+0000 7fd7bdbee640 1 -- 172.21.15.28:0/1492607799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7b819d350 con 0x7fd7b8101710 2024-04-25T01:41:00.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.500+0000 7fd7b4ff9640 1 -- 172.21.15.28:0/1492607799 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8018bd0 con 0x7fd7b8101710 2024-04-25T01:41:00.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.500+0000 7fd7bdbee640 1 -- 172.21.15.28:0/1492607799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7b81a4be0 con 0x7fd7b8101710 2024-04-25T01:41:00.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:00 smithi032 ceph-mon[32461]: pgmap v244: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:41:00.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.502+0000 7fd7b4ff9640 1 -- 172.21.15.28:0/1492607799 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd7a80107d0 con 0x7fd7b8101710 2024-04-25T01:41:00.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7c09dd740 1 Processor -- start 2024-04-25T01:41:00.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7c09dd740 1 -- start start 2024-04-25T01:41:00.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7c09dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafd630 0x55622aafda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7c09dd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622aafdf50 0x55622ab080e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7c09dd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622ab08620 0x55622ab0aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7c09dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55622aa626b0 con 0x55622aafd630 2024-04-25T01:41:00.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7c09dd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55622aa0a880 con 0x55622ab08620 2024-04-25T01:41:00.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7c09dd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55622aa5e670 con 0x55622aafdf50 2024-04-25T01:41:00.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7b7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafd630 0x55622aafda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7b7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafd630 0x55622aafda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48308/0 (socket says 172.21.15.28:48308) 2024-04-25T01:41:00.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7b7fff640 1 -- 172.21.15.28:0/3810664150 learned_addr learned my addr 172.21.15.28:0/3810664150 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:00.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7b4ff9640 1 --2- 172.21.15.28:0/1492607799 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd7a40a2d00 0x7fd7a40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7b6ffd640 1 --2- 172.21.15.28:0/3810664150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622ab08620 0x55622ab0aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7b77fe640 1 --2- 172.21.15.28:0/3810664150 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622aafdf50 0x55622ab080e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7b7fff640 1 -- 172.21.15.28:0/3810664150 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622aafdf50 msgr2=0x55622ab080e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:00.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7b7fff640 1 --2- 172.21.15.28:0/3810664150 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622aafdf50 0x55622ab080e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7b7fff640 1 -- 172.21.15.28:0/3810664150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622ab08620 msgr2=0x55622ab0aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:00.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7b7fff640 1 --2- 172.21.15.28:0/3810664150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622ab08620 0x55622ab0aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7b7fff640 1 -- 172.21.15.28:0/3810664150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55622a8407d0 con 0x55622aafd630 2024-04-25T01:41:00.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7b6ffd640 1 --2- 172.21.15.28:0/3810664150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622ab08620 0x55622ab0aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:00.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7b4ff9640 1 -- 172.21.15.28:0/1492607799 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fd7a80cfda0 con 0x7fd7b8101710 2024-04-25T01:41:00.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.503+0000 7fd7b7fff640 1 --2- 172.21.15.28:0/1492607799 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd7a40a2d00 0x7fd7a40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.504+0000 7fd7b7fff640 1 --2- 172.21.15.28:0/3810664150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafd630 0x55622aafda10 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0009930 tx=0x7fd7b00315b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e1625fb27d824b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:00.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.504+0000 7fd7adffb640 1 -- 172.21.15.28:0/3810664150 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b0035040 con 0x55622aafd630 2024-04-25T01:41:00.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.504+0000 7fd7b7fff640 1 --2- 172.21.15.28:0/1492607799 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd7a40a2d00 0x7fd7a40a51c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd7b004a000 tx=0x7fd7b0047000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:00.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.504+0000 7fd7adffb640 1 -- 172.21.15.28:0/3810664150 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7b00453d0 con 0x55622aafd630 2024-04-25T01:41:00.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.504+0000 7fd7adffb640 1 -- 172.21.15.28:0/3810664150 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b003b5d0 con 0x55622aafd630 2024-04-25T01:41:00.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.504+0000 7fd7c09dd740 1 -- 172.21.15.28:0/3810664150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafd630 msgr2=0x55622aafda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:00.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.504+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/3810664150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafd630 0x55622aafda10 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0009930 tx=0x7fd7b00315b0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.504+0000 7fd7c09dd740 1 -- 172.21.15.28:0/3810664150 shutdown_connections 2024-04-25T01:41:00.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.504+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/3810664150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622ab08620 0x55622ab0aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.504+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/3810664150 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622aafdf50 0x55622ab080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.504+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/3810664150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafd630 0x55622aafda10 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.505+0000 7fd7c09dd740 1 -- 172.21.15.28:0/3810664150 >> 172.21.15.28:0/3810664150 conn(0x55622aaf9950 msgr2=0x55622aaf9d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:00.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.505+0000 7fd7c09dd740 1 -- 172.21.15.28:0/3810664150 shutdown_connections 2024-04-25T01:41:00.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.505+0000 7fd7c09dd740 1 -- 172.21.15.28:0/3810664150 wait complete. 2024-04-25T01:41:00.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.505+0000 7fd7c09dd740 1 -- start start 2024-04-25T01:41:00.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.505+0000 7fd7c09dd740 1 Processor -- start 2024-04-25T01:41:00.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.505+0000 7fd7c09dd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622aafd630 0x55622ab05580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.505+0000 7fd7c09dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafdf50 0x55622ab05ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.505+0000 7fd7c09dd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622ab08620 0x55622ab06000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.505+0000 7fd7c09dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55622ab0b410 con 0x55622aafdf50 2024-04-25T01:41:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.505+0000 7fd7c09dd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55622aa1df50 con 0x55622aafd630 2024-04-25T01:41:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.505+0000 7fd7c09dd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55622ab0b590 con 0x55622ab08620 2024-04-25T01:41:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.505+0000 7fd7c09dd740 10 client.0 _ll_register_callbacks cb 0x55622aa59000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.505+0000 7fd7b6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafdf50 0x55622ab05ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.505+0000 7fd7b6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafdf50 0x55622ab05ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48314/0 (socket says 172.21.15.28:48314) 2024-04-25T01:41:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.505+0000 7fd7b6ffd640 1 -- 172.21.15.28:0/2752556943 learned_addr learned my addr 172.21.15.28:0/2752556943 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.506+0000 7fd7b77fe640 1 --2- 172.21.15.28:0/2752556943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622aafd630 0x55622ab05580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.506+0000 7fd7b7fff640 1 --2- 172.21.15.28:0/2752556943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622ab08620 0x55622ab06000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.506+0000 7fd7b6ffd640 1 -- 172.21.15.28:0/2752556943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622ab08620 msgr2=0x55622ab06000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:00.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.506+0000 7fd7b6ffd640 1 --2- 172.21.15.28:0/2752556943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622ab08620 0x55622ab06000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.506+0000 7fd7b6ffd640 1 -- 172.21.15.28:0/2752556943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622aafd630 msgr2=0x55622ab05580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:00.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.506+0000 7fd7b6ffd640 1 --2- 172.21.15.28:0/2752556943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622aafd630 0x55622ab05580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.506+0000 7fd7b6ffd640 1 -- 172.21.15.28:0/2752556943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55622a7763f0 con 0x55622aafdf50 2024-04-25T01:41:00.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.506+0000 7fd7b7fff640 1 --2- 172.21.15.28:0/2752556943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622ab08620 0x55622ab06000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:00.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.506+0000 7fd7b6ffd640 1 --2- 172.21.15.28:0/2752556943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafdf50 0x55622ab05ac0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fd7a000b9f0 tx=0x7fd7a000bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=931545b666d8008 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:00.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.506+0000 7fd796ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:00.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.506+0000 7fd796ffd640 1 -- 172.21.15.28:0/2752556943 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a001b440 con 0x55622aafdf50 2024-04-25T01:41:00.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.506+0000 7fd796ffd640 1 -- 172.21.15.28:0/2752556943 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fd7a0011440 con 0x55622aafdf50 2024-04-25T01:41:00.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.506+0000 7fd7c09dd740 1 -- 172.21.15.28:0/2752556943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7b00453d0 con 0x55622aafdf50 2024-04-25T01:41:00.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.506+0000 7fd796ffd640 1 -- 172.21.15.28:0/2752556943 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd7a0011640 con 0x55622aafdf50 2024-04-25T01:41:00.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.506+0000 7fd7c09dd740 10 client.18681 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:00.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.506+0000 7fd7c09dd740 1 -- 172.21.15.28:0/2752556943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55622a7360f0 con 0x55622aafdf50 2024-04-25T01:41:00.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.506+0000 7fd796ffd640 1 -- 172.21.15.28:0/2752556943 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7b00453d0 con 0x55622aafdf50 2024-04-25T01:41:00.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.507+0000 7fd796ffd640 1 -- 172.21.15.28:0/2752556943 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd7a0019020 con 0x55622aafdf50 2024-04-25T01:41:00.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.507+0000 7fd7c09dd740 10 client.18681 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:00.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.507+0000 7fd7c09dd740 10 client.18681 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:41:00.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.507+0000 7fd7c09dd740 20 client.18681 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:00.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.507+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/2752556943 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55622ab18a00 0x55622ab1adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.507+0000 7fd7c09dd740 4 client.18681 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:41:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.507+0000 7fd7c09dd740 1 -- 172.21.15.28:0/2752556943 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55622aa6fbf0 con 0x55622ab18a00 2024-04-25T01:41:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.507+0000 7fd7b77fe640 1 --2- 172.21.15.28:0/2752556943 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55622ab18a00 0x55622ab1adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.507+0000 7fd796ffd640 10 client.18681 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.507+0000 7fd7b77fe640 1 --2- 172.21.15.28:0/2752556943 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55622ab18a00 0x55622ab1adf0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2bd169ded3e2834 server_cookie=a5266730e3ea5c1c in_seq=0 out_seq=0 2024-04-25T01:41:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.508+0000 7fd796ffd640 1 -- 172.21.15.28:0/2752556943 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55622aa6fbf0 con 0x55622ab18a00 2024-04-25T01:41:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.508+0000 7fd796ffd640 10 client.18681 handle_command_reply: tid=0 2024-04-25T01:41:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.508+0000 7fd7c09dd740 1 client.18681 shutdown 2024-04-25T01:41:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.508+0000 7fd7c09dd740 1 -- 172.21.15.28:0/2752556943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafdf50 msgr2=0x55622ab05ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.508+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/2752556943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafdf50 0x55622ab05ac0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fd7a000b9f0 tx=0x7fd7a000bdf0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.508+0000 7fd7c09dd740 1 -- 172.21.15.28:0/2752556943 shutdown_connections 2024-04-25T01:41:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.508+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/2752556943 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55622ab18a00 0x55622ab1adf0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.508+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/2752556943 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622ab08620 0x55622ab06000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.508+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/2752556943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafdf50 0x55622ab05ac0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.508+0000 7fd796ffd640 0 client.18681 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:00.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.508+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/2752556943 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622aafd630 0x55622ab05580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.508+0000 7fd7c09dd740 1 -- 172.21.15.28:0/2752556943 >> 172.21.15.28:0/2752556943 conn(0x55622aaf9950 msgr2=0x55622aaff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:00.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.508+0000 7fd7c09dd740 1 -- 172.21.15.28:0/2752556943 shutdown_connections 2024-04-25T01:41:00.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.508+0000 7fd7c09dd740 1 -- 172.21.15.28:0/2752556943 wait complete. 2024-04-25T01:41:00.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.508+0000 7fd7c09dd740 20 client.18681 trim_cache size 0 max 16384 2024-04-25T01:41:00.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.526+0000 7fd7c09dd740 1 Processor -- start 2024-04-25T01:41:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.526+0000 7fd7c09dd740 1 -- start start 2024-04-25T01:41:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.526+0000 7fd7c09dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafd630 0x55622ab14300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.526+0000 7fd7c09dd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622aafdf50 0x55622ab03e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.526+0000 7fd7c09dd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622ab08620 0x55622ab05e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.526+0000 7fd7c09dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b81aeb60 con 0x55622aafd630 2024-04-25T01:41:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.526+0000 7fd7c09dd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b81b06a0 con 0x55622ab08620 2024-04-25T01:41:00.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.526+0000 7fd7c09dd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b81af230 con 0x55622aafdf50 2024-04-25T01:41:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.526+0000 7fd7b77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafd630 0x55622ab14300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.526+0000 7fd7b77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafd630 0x55622ab14300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48316/0 (socket says 172.21.15.28:48316) 2024-04-25T01:41:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.526+0000 7fd7b77fe640 1 -- 172.21.15.28:0/3143985138 learned_addr learned my addr 172.21.15.28:0/3143985138 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.526+0000 7fd7b77fe640 1 -- 172.21.15.28:0/3143985138 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622aafdf50 msgr2=0x55622ab03e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.526+0000 7fd7b77fe640 1 --2- 172.21.15.28:0/3143985138 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622aafdf50 0x55622ab03e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.526+0000 7fd7b77fe640 1 -- 172.21.15.28:0/3143985138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622ab08620 msgr2=0x55622ab05e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:00.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.526+0000 7fd7b77fe640 1 --2- 172.21.15.28:0/3143985138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622ab08620 0x55622ab05e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.526+0000 7fd7b77fe640 1 -- 172.21.15.28:0/3143985138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55622aaeb090 con 0x55622aafd630 2024-04-25T01:41:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.526+0000 7fd7b77fe640 1 --2- 172.21.15.28:0/3143985138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafd630 0x55622ab14300 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800cba0 tx=0x7fd7a80d21f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac132a070c0e78b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.527+0000 7fd7ad7fa640 1 -- 172.21.15.28:0/3143985138 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8010e60 con 0x55622aafd630 2024-04-25T01:41:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.527+0000 7fd7ad7fa640 1 -- 172.21.15.28:0/3143985138 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7a8010ac0 con 0x55622aafd630 2024-04-25T01:41:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.527+0000 7fd7ad7fa640 1 -- 172.21.15.28:0/3143985138 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a809b3d0 con 0x55622aafd630 2024-04-25T01:41:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.527+0000 7fd7c09dd740 1 -- 172.21.15.28:0/3143985138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafd630 msgr2=0x55622ab14300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.527+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/3143985138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafd630 0x55622ab14300 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800cba0 tx=0x7fd7a80d21f0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.527+0000 7fd7c09dd740 1 -- 172.21.15.28:0/3143985138 shutdown_connections 2024-04-25T01:41:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.527+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/3143985138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622ab08620 0x55622ab05e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.527+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/3143985138 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622aafdf50 0x55622ab03e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.527+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/3143985138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafd630 0x55622ab14300 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.527+0000 7fd7c09dd740 1 -- 172.21.15.28:0/3143985138 >> 172.21.15.28:0/3143985138 conn(0x55622aaf9950 msgr2=0x55622ab06ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.527+0000 7fd7c09dd740 1 -- 172.21.15.28:0/3143985138 shutdown_connections 2024-04-25T01:41:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.527+0000 7fd7c09dd740 1 -- 172.21.15.28:0/3143985138 wait complete. 2024-04-25T01:41:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.527+0000 7fd7c09dd740 1 -- start start 2024-04-25T01:41:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7c09dd740 1 Processor -- start 2024-04-25T01:41:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7c09dd740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622aafd630 0x55622ab0c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7c09dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafdf50 0x55622ab0ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7c09dd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622ab08620 0x55622ab0cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7c09dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b81affd0 con 0x55622aafdf50 2024-04-25T01:41:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7c09dd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b81af900 con 0x55622ab08620 2024-04-25T01:41:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7c09dd740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b81acb60 con 0x55622aafd630 2024-04-25T01:41:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7c09dd740 10 client.0 _ll_register_callbacks cb 0x7fd7a0004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7b6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafdf50 0x55622ab0ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7b6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafdf50 0x55622ab0ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48324/0 (socket says 172.21.15.28:48324) 2024-04-25T01:41:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7b6ffd640 1 -- 172.21.15.28:0/2474899143 learned_addr learned my addr 172.21.15.28:0/2474899143 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:00.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7b7fff640 1 --2- 172.21.15.28:0/2474899143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622ab08620 0x55622ab0cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7b6ffd640 1 -- 172.21.15.28:0/2474899143 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622aafd630 msgr2=0x55622ab0c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:00.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7b77fe640 1 --2- 172.21.15.28:0/2474899143 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622aafd630 0x55622ab0c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7b6ffd640 1 --2- 172.21.15.28:0/2474899143 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622aafd630 0x55622ab0c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7b6ffd640 1 -- 172.21.15.28:0/2474899143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622ab08620 msgr2=0x55622ab0cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:00.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7b6ffd640 1 --2- 172.21.15.28:0/2474899143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622ab08620 0x55622ab0cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7b6ffd640 1 -- 172.21.15.28:0/2474899143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55622ab7e210 con 0x55622aafdf50 2024-04-25T01:41:00.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7b7fff640 1 --2- 172.21.15.28:0/2474899143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622ab08620 0x55622ab0cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:41:00.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd7b6ffd640 1 --2- 172.21.15.28:0/2474899143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafdf50 0x55622ab0ca00 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0001eb0 tx=0x7fd7a0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8b8858e71bb621 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:00.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.528+0000 7fd796ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.529+0000 7fd796ffd640 1 -- 172.21.15.28:0/2474899143 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a0004360 con 0x55622aafdf50 2024-04-25T01:41:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.529+0000 7fd796ffd640 1 -- 172.21.15.28:0/2474899143 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9438+0+0 (secure 0 0 0) 0x7fd7a0010440 con 0x55622aafdf50 2024-04-25T01:41:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.529+0000 7fd7c09dd740 1 -- 172.21.15.28:0/2474899143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7a8010ac0 con 0x55622aafdf50 2024-04-25T01:41:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.529+0000 7fd796ffd640 1 -- 172.21.15.28:0/2474899143 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd7a0036050 con 0x55622aafdf50 2024-04-25T01:41:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.529+0000 7fd7c09dd740 10 client.18687 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.529+0000 7fd7c09dd740 1 -- 172.21.15.28:0/2474899143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55622ab7f520 con 0x55622aafdf50 2024-04-25T01:41:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.529+0000 7fd796ffd640 1 -- 172.21.15.28:0/2474899143 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7a8010ac0 con 0x55622aafdf50 2024-04-25T01:41:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.529+0000 7fd796ffd640 1 -- 172.21.15.28:0/2474899143 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd7a0019070 con 0x55622aafdf50 2024-04-25T01:41:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.529+0000 7fd7c09dd740 10 client.18687 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.529+0000 7fd7c09dd740 10 client.18687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:41:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.529+0000 7fd7c09dd740 20 client.18687 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:00.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.529+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/2474899143 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55622ab7fc60 0x55622ab82080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:00.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.529+0000 7fd7c09dd740 4 client.18687 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:41:00.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.529+0000 7fd7c09dd740 1 -- 172.21.15.28:0/2474899143 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55622aa69f40 con 0x55622ab7fc60 2024-04-25T01:41:00.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.529+0000 7fd7b77fe640 1 --2- 172.21.15.28:0/2474899143 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55622ab7fc60 0x55622ab82080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:00.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.530+0000 7fd7b77fe640 1 --2- 172.21.15.28:0/2474899143 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55622ab7fc60 0x55622ab82080 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c60529e9e1fae22 server_cookie=7e360ad023299b7a in_seq=0 out_seq=0 2024-04-25T01:41:00.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.530+0000 7fd796ffd640 10 client.18687 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:00.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.530+0000 7fd796ffd640 1 -- 172.21.15.28:0/2474899143 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55622aa69f40 con 0x55622ab7fc60 2024-04-25T01:41:00.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.530+0000 7fd796ffd640 10 client.18687 handle_command_reply: tid=0 2024-04-25T01:41:00.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.530+0000 7fd7c09dd740 1 client.18687 shutdown 2024-04-25T01:41:00.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.530+0000 7fd7c09dd740 1 -- 172.21.15.28:0/2474899143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafdf50 msgr2=0x55622ab0ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:00.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.530+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/2474899143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafdf50 0x55622ab0ca00 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0001eb0 tx=0x7fd7a0005fd0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.530+0000 7fd7c09dd740 1 -- 172.21.15.28:0/2474899143 shutdown_connections 2024-04-25T01:41:00.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.530+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/2474899143 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55622ab7fc60 0x55622ab82080 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.530+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/2474899143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55622ab08620 0x55622ab0cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.530+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/2474899143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55622aafdf50 0x55622ab0ca00 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.530+0000 7fd7c09dd740 1 --2- 172.21.15.28:0/2474899143 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55622aafd630 0x55622ab0c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.530+0000 7fd796ffd640 0 client.18687 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:00.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.530+0000 7fd7c09dd740 1 -- 172.21.15.28:0/2474899143 >> 172.21.15.28:0/2474899143 conn(0x55622aaf9950 msgr2=0x55622ab132f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:00.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.530+0000 7fd7c09dd740 1 -- 172.21.15.28:0/2474899143 shutdown_connections 2024-04-25T01:41:00.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.530+0000 7fd7c09dd740 1 -- 172.21.15.28:0/2474899143 wait complete. 2024-04-25T01:41:00.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.530+0000 7fd7c09dd740 20 client.18687 trim_cache size 0 max 16384 2024-04-25T01:41:00.532 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:41:00.532 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:41:00.532 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-25T01:41:00.532 INFO:teuthology.orchestra.run.smithi028.stdout: "2fe0900b-b8b6-4a5c-bff3-421cc5d3103b": { 2024-04-25T01:41:00.532 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-25T01:41:00.532 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "2fe0900b-b8b6-4a5c-bff3-421cc5d3103b", 2024-04-25T01:41:00.532 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-25T01:41:00.532 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:41:00.532 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:41:00.533 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:41:00.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.531+0000 7fd7bdbee640 1 -- 172.21.15.28:0/1492607799 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd7a40a2d00 msgr2=0x7fd7a40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:00.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.531+0000 7fd7bdbee640 1 --2- 172.21.15.28:0/1492607799 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd7a40a2d00 0x7fd7a40a51c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd7b004a000 tx=0x7fd7b0047000 comp rx=0 tx=0).stop 2024-04-25T01:41:00.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.531+0000 7fd7bdbee640 1 -- 172.21.15.28:0/1492607799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b8101710 msgr2=0x7fd7b8198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:00.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.531+0000 7fd7bdbee640 1 --2- 172.21.15.28:0/1492607799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b8101710 0x7fd7b8198070 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800ebc0 tx=0x7fd7a800c6a0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.531+0000 7fd7bdbee640 1 -- 172.21.15.28:0/1492607799 shutdown_connections 2024-04-25T01:41:00.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.531+0000 7fd7bdbee640 1 --2- 172.21.15.28:0/1492607799 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd7a40a2d00 0x7fd7a40a51c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.531+0000 7fd7bdbee640 1 --2- 172.21.15.28:0/1492607799 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd7b8107720 0x7fd7b819c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.531+0000 7fd7bdbee640 1 --2- 172.21.15.28:0/1492607799 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd7b81020d0 0x7fd7b81985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.531+0000 7fd7bdbee640 1 --2- 172.21.15.28:0/1492607799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b8101710 0x7fd7b8198070 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:00.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.531+0000 7fd7bdbee640 1 -- 172.21.15.28:0/1492607799 >> 172.21.15.28:0/1492607799 conn(0x7fd7b80fd180 msgr2=0x7fd7b80fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:00.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.531+0000 7fd7bdbee640 1 -- 172.21.15.28:0/1492607799 shutdown_connections 2024-04-25T01:41:00.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:00.531+0000 7fd7bdbee640 1 -- 172.21.15.28:0/1492607799 wait complete. 2024-04-25T01:41:00.536 INFO:tasks.cephfs.filesystem:scrub status for tag:2fe0900b-b8b6-4a5c-bff3-421cc5d3103b - {'path': '/', 'tag': '2fe0900b-b8b6-4a5c-bff3-421cc5d3103b', 'options': 'recursive,force'} 2024-04-25T01:41:02.269 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:02 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:41:02.269 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:02 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:41:02.269 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:02 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:41:02.269 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:02 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:41:02.269 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:02 smithi130 ceph-mon[32702]: pgmap v245: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:41:02.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:02 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:41:02.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:02 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:41:02.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:02 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:41:02.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:02 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:41:02.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:02 smithi028 ceph-mon[29390]: pgmap v245: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:41:02.492 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:02 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:41:02.492 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:02 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:41:02.492 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:02 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:41:02.492 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:02 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:41:02.492 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:02 smithi032 ceph-mon[32461]: pgmap v245: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:41:04.294 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:04 smithi032 ceph-mon[32461]: pgmap v246: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-25T01:41:04.295 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:04 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:41:04.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:04 smithi028 ceph-mon[29390]: pgmap v246: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-25T01:41:04.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:04 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:41:04.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:04 smithi130 ceph-mon[32702]: pgmap v246: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-25T01:41:04.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:04 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:41:06.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:06 smithi028 ceph-mon[29390]: pgmap v247: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-25T01:41:06.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:06 smithi130 ceph-mon[32702]: pgmap v247: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-25T01:41:06.492 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:06 smithi032 ceph-mon[32461]: pgmap v247: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-25T01:41:08.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:08 smithi028 ceph-mon[29390]: pgmap v248: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-25T01:41:08.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:08 smithi130 ceph-mon[32702]: pgmap v248: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-25T01:41:08.492 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:08 smithi032 ceph-mon[32461]: pgmap v248: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-25T01:41:10.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:10 smithi028 ceph-mon[29390]: pgmap v249: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-04-25T01:41:10.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:10 smithi028 ceph-mon[29390]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T01:41:10.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:10 smithi130 ceph-mon[32702]: pgmap v249: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-04-25T01:41:10.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:10 smithi130 ceph-mon[32702]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T01:41:10.492 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:10 smithi032 ceph-mon[32461]: pgmap v249: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-04-25T01:41:10.492 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:10 smithi032 ceph-mon[32461]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T01:41:12.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:12 smithi028 ceph-mon[29390]: pgmap v251: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-04-25T01:41:12.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:12 smithi130 ceph-mon[32702]: pgmap v251: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-04-25T01:41:12.492 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:12 smithi032 ceph-mon[32461]: pgmap v251: 129 pgs: 129 active+clean; 7.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 31 op/s 2024-04-25T01:41:13.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:13 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:41:13.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:13 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:41:13.492 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:13 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:41:14.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:14 smithi028 ceph-mon[29390]: pgmap v252: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:41:14.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:14 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:41:14.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:14 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:41:14.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:14 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:41:14.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:14 smithi130 ceph-mon[32702]: pgmap v252: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:41:14.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:14 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:41:14.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:14 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:41:14.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:14 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:41:14.493 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:14 smithi032 ceph-mon[32461]: pgmap v252: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:41:14.493 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:14 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:41:14.493 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:14 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:41:14.493 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:14 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:41:16.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:16 smithi028 ceph-mon[29390]: pgmap v253: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:41:16.467 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:16 smithi130 ceph-mon[32702]: pgmap v253: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:41:16.492 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:16 smithi032 ceph-mon[32461]: pgmap v253: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:41:18.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:17 smithi130 ceph-mon[32702]: pgmap v254: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:41:18.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:17 smithi130 ceph-mon[32702]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T01:41:18.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:17 smithi032 ceph-mon[32461]: pgmap v254: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:41:18.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:17 smithi032 ceph-mon[32461]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T01:41:18.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:17 smithi028 ceph-mon[29390]: pgmap v254: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:41:18.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:17 smithi028 ceph-mon[29390]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T01:41:20.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:19 smithi130 ceph-mon[32702]: pgmap v256: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2024-04-25T01:41:20.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:19 smithi032 ceph-mon[32461]: pgmap v256: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2024-04-25T01:41:20.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:19 smithi028 ceph-mon[29390]: pgmap v256: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2024-04-25T01:41:22.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:21 smithi130 ceph-mon[32702]: pgmap v257: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:41:22.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:21 smithi032 ceph-mon[32461]: pgmap v257: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:41:22.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:21 smithi028 ceph-mon[29390]: pgmap v257: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:41:24.217 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:23 smithi130 ceph-mon[32702]: pgmap v258: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:41:24.242 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:23 smithi032 ceph-mon[32461]: pgmap v258: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:41:24.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:23 smithi028 ceph-mon[29390]: pgmap v258: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:41:26.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:25 smithi130 ceph-mon[32702]: pgmap v259: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:41:26.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:25 smithi032 ceph-mon[32461]: pgmap v259: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:41:26.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:25 smithi028 ceph-mon[29390]: pgmap v259: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:41:28.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:27 smithi130 ceph-mon[32702]: pgmap v260: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:41:28.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:27 smithi032 ceph-mon[32461]: pgmap v260: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:41:28.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:27 smithi028 ceph-mon[29390]: pgmap v260: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:41:30.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:29 smithi130 ceph-mon[32702]: pgmap v261: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T01:41:30.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:29 smithi032 ceph-mon[32461]: pgmap v261: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T01:41:30.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:29 smithi028 ceph-mon[29390]: pgmap v261: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T01:41:30.537 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:41:30.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:41:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.668+0000 7fb1bf47d640 1 -- 172.21.15.28:0/2313202214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b8107720 msgr2=0x7fb1b8107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.668+0000 7fb1bf47d640 1 --2- 172.21.15.28:0/2313202214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b8107720 0x7fb1b8107b00 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4009930 tx=0x7fb1b402a420 comp rx=0 tx=0).stop 2024-04-25T01:41:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.668+0000 7fb1bf47d640 1 -- 172.21.15.28:0/2313202214 shutdown_connections 2024-04-25T01:41:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.668+0000 7fb1bf47d640 1 --2- 172.21.15.28:0/2313202214 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1b81020d0 0x7fb1b810e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.668+0000 7fb1bf47d640 1 --2- 172.21.15.28:0/2313202214 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1b8101710 0x7fb1b8101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.668+0000 7fb1bf47d640 1 --2- 172.21.15.28:0/2313202214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b8107720 0x7fb1b8107b00 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.668+0000 7fb1bf47d640 1 -- 172.21.15.28:0/2313202214 >> 172.21.15.28:0/2313202214 conn(0x7fb1b80fd180 msgr2=0x7fb1b80ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.668+0000 7fb1bf47d640 1 -- 172.21.15.28:0/2313202214 shutdown_connections 2024-04-25T01:41:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.668+0000 7fb1bf47d640 1 -- 172.21.15.28:0/2313202214 wait complete. 2024-04-25T01:41:30.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.669+0000 7fb1bf47d640 1 Processor -- start 2024-04-25T01:41:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.669+0000 7fb1bf47d640 1 -- start start 2024-04-25T01:41:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.669+0000 7fb1bf47d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b8101710 0x7fb1b81980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.669+0000 7fb1bf47d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1b81020d0 0x7fb1b8198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.669+0000 7fb1bf47d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1b8107720 0x7fb1b819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.669+0000 7fb1bf47d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b8111b30 con 0x7fb1b8101710 2024-04-25T01:41:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.669+0000 7fb1bf47d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b81119b0 con 0x7fb1b8107720 2024-04-25T01:41:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.669+0000 7fb1bf47d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b8111cb0 con 0x7fb1b81020d0 2024-04-25T01:41:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.669+0000 7fb1bd219640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b8101710 0x7fb1b81980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.669+0000 7fb1bd219640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b8101710 0x7fb1b81980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58232/0 (socket says 172.21.15.28:58232) 2024-04-25T01:41:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.669+0000 7fb1bd219640 1 -- 172.21.15.28:0/3555547843 learned_addr learned my addr 172.21.15.28:0/3555547843 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.669+0000 7fb1bca18640 1 --2- 172.21.15.28:0/3555547843 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1b81020d0 0x7fb1b8198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.669+0000 7fb1bda1a640 1 --2- 172.21.15.28:0/3555547843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1b8107720 0x7fb1b819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.670+0000 7fb1bd219640 1 -- 172.21.15.28:0/3555547843 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1b81020d0 msgr2=0x7fb1b8198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:30.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.670+0000 7fb1bd219640 1 --2- 172.21.15.28:0/3555547843 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1b81020d0 0x7fb1b8198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.670+0000 7fb1bd219640 1 -- 172.21.15.28:0/3555547843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1b8107720 msgr2=0x7fb1b819c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:30.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.670+0000 7fb1bd219640 1 --2- 172.21.15.28:0/3555547843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1b8107720 0x7fb1b819c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.670+0000 7fb1bd219640 1 -- 172.21.15.28:0/3555547843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b819d090 con 0x7fb1b8101710 2024-04-25T01:41:30.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.670+0000 7fb1bd219640 1 --2- 172.21.15.28:0/3555547843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b8101710 0x7fb1b81980f0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4009230 tx=0x7fb1b4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:30.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.670+0000 7fb1ae7fc640 1 -- 172.21.15.28:0/3555547843 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b403f030 con 0x7fb1b8101710 2024-04-25T01:41:30.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.670+0000 7fb1ae7fc640 1 -- 172.21.15.28:0/3555547843 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1b4003ad0 con 0x7fb1b8101710 2024-04-25T01:41:30.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.670+0000 7fb1bf47d640 1 -- 172.21.15.28:0/3555547843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1b819d320 con 0x7fb1b8101710 2024-04-25T01:41:30.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.670+0000 7fb1ae7fc640 1 -- 172.21.15.28:0/3555547843 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b4035070 con 0x7fb1b8101710 2024-04-25T01:41:30.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.670+0000 7fb1bf47d640 1 -- 172.21.15.28:0/3555547843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1b81a4c10 con 0x7fb1b8101710 2024-04-25T01:41:30.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.672+0000 7fb1ae7fc640 1 -- 172.21.15.28:0/3555547843 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb1b4042030 con 0x7fb1b8101710 2024-04-25T01:41:30.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.673+0000 7fb1c218e740 1 Processor -- start 2024-04-25T01:41:30.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.673+0000 7fb1c218e740 1 -- start start 2024-04-25T01:41:30.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.673+0000 7fb1c218e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12630 0x557981f12a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.673+0000 7fb1c218e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f12f50 0x557981f1d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.673+0000 7fb1c218e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f1d620 0x557981f1fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.673+0000 7fb1c218e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557981e776b0 con 0x557981f12630 2024-04-25T01:41:30.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.673+0000 7fb1c218e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557981e1f880 con 0x557981f1d620 2024-04-25T01:41:30.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.673+0000 7fb1c218e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x557981e73670 con 0x557981f12f50 2024-04-25T01:41:30.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.673+0000 7fb1ae7fc640 1 --2- 172.21.15.28:0/3555547843 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb1a40a2d80 0x7fb1a40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.673+0000 7fb1bda1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12630 0x557981f12a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.673+0000 7fb1bda1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12630 0x557981f12a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58246/0 (socket says 172.21.15.28:58246) 2024-04-25T01:41:30.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.673+0000 7fb1bda1a640 1 -- 172.21.15.28:0/701342463 learned_addr learned my addr 172.21.15.28:0/701342463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:30.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.673+0000 7fb1bda1a640 1 --2- 172.21.15.28:0/3555547843 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb1a40a2d80 0x7fb1a40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.673+0000 7fb1bd219640 1 --2- 172.21.15.28:0/701342463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f12f50 0x557981f1d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.673+0000 7fb1bca18640 1 --2- 172.21.15.28:0/701342463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f1d620 0x557981f1fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.673+0000 7fb1ae7fc640 1 -- 172.21.15.28:0/3555547843 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fb1b40efd20 con 0x7fb1b8101710 2024-04-25T01:41:30.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.674+0000 7fb1bda1a640 1 -- 172.21.15.28:0/701342463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f12f50 msgr2=0x557981f1d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:30.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.674+0000 7fb1bda1a640 1 --2- 172.21.15.28:0/701342463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f12f50 0x557981f1d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.674+0000 7fb1bda1a640 1 -- 172.21.15.28:0/701342463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f1d620 msgr2=0x557981f1fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:30.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.674+0000 7fb1bda1a640 1 --2- 172.21.15.28:0/701342463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f1d620 0x557981f1fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.674+0000 7fb1bda1a640 1 -- 172.21.15.28:0/701342463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557981c557d0 con 0x557981f12630 2024-04-25T01:41:30.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.674+0000 7fb1bca18640 1 --2- 172.21.15.28:0/701342463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f1d620 0x557981f1fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:41:30.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.674+0000 7fb1bda1a640 1 --2- 172.21.15.28:0/701342463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12630 0x557981f12a10 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0011e20 tx=0x7fb1b000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a32504819a044678 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:30.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.674+0000 7fb1bda1a640 1 --2- 172.21.15.28:0/3555547843 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb1a40a2d80 0x7fb1a40a5240 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb1b000fc90 tx=0x7fb1b001e000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:30.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.674+0000 7fb1a37fe640 1 -- 172.21.15.28:0/701342463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b001dcb0 con 0x557981f12630 2024-04-25T01:41:30.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.674+0000 7fb1a37fe640 1 -- 172.21.15.28:0/701342463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1b001de50 con 0x557981f12630 2024-04-25T01:41:30.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.674+0000 7fb1a37fe640 1 -- 172.21.15.28:0/701342463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b0004290 con 0x557981f12630 2024-04-25T01:41:30.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.675+0000 7fb1c218e740 1 -- 172.21.15.28:0/701342463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12630 msgr2=0x557981f12a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:30.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.675+0000 7fb1c218e740 1 --2- 172.21.15.28:0/701342463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12630 0x557981f12a10 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0011e20 tx=0x7fb1b000f790 comp rx=0 tx=0).stop 2024-04-25T01:41:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.675+0000 7fb1c218e740 1 -- 172.21.15.28:0/701342463 shutdown_connections 2024-04-25T01:41:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.675+0000 7fb1c218e740 1 --2- 172.21.15.28:0/701342463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f1d620 0x557981f1fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.675+0000 7fb1c218e740 1 --2- 172.21.15.28:0/701342463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f12f50 0x557981f1d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.675+0000 7fb1c218e740 1 --2- 172.21.15.28:0/701342463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12630 0x557981f12a10 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.675+0000 7fb1c218e740 1 -- 172.21.15.28:0/701342463 >> 172.21.15.28:0/701342463 conn(0x557981f0e950 msgr2=0x557981f0ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.675+0000 7fb1c218e740 1 -- 172.21.15.28:0/701342463 shutdown_connections 2024-04-25T01:41:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.675+0000 7fb1c218e740 1 -- 172.21.15.28:0/701342463 wait complete. 2024-04-25T01:41:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.675+0000 7fb1c218e740 1 -- start start 2024-04-25T01:41:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.675+0000 7fb1c218e740 1 Processor -- start 2024-04-25T01:41:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.676+0000 7fb1c218e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12630 0x557981f1a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.676+0000 7fb1c218e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f12f50 0x557981f1ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.676+0000 7fb1c218e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f1d620 0x557981f1b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.676+0000 7fb1c218e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557981f20350 con 0x557981f12630 2024-04-25T01:41:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.676+0000 7fb1c218e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557981e32f50 con 0x557981f1d620 2024-04-25T01:41:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.676+0000 7fb1c218e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x557981f204d0 con 0x557981f12f50 2024-04-25T01:41:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.676+0000 7fb1c218e740 10 client.0 _ll_register_callbacks cb 0x557981e6e000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.676+0000 7fb1bd219640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12630 0x557981f1a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.676+0000 7fb1bd219640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12630 0x557981f1a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58248/0 (socket says 172.21.15.28:58248) 2024-04-25T01:41:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.676+0000 7fb1bd219640 1 -- 172.21.15.28:0/312885498 learned_addr learned my addr 172.21.15.28:0/312885498 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.676+0000 7fb1bca18640 1 --2- 172.21.15.28:0/312885498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f12f50 0x557981f1ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.676+0000 7fb1bda1a640 1 --2- 172.21.15.28:0/312885498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f1d620 0x557981f1b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.676+0000 7fb1bd219640 1 -- 172.21.15.28:0/312885498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f12f50 msgr2=0x557981f1ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.676+0000 7fb1bd219640 1 --2- 172.21.15.28:0/312885498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f12f50 0x557981f1ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.676+0000 7fb1bd219640 1 -- 172.21.15.28:0/312885498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f1d620 msgr2=0x557981f1b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.676+0000 7fb1bd219640 1 --2- 172.21.15.28:0/312885498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f1d620 0x557981f1b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.676+0000 7fb1bd219640 1 -- 172.21.15.28:0/312885498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557981b8b3f0 con 0x557981f12630 2024-04-25T01:41:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.676+0000 7fb1bca18640 1 --2- 172.21.15.28:0/312885498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f12f50 0x557981f1ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.676+0000 7fb1bda1a640 1 --2- 172.21.15.28:0/312885498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f1d620 0x557981f1b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.677+0000 7fb1bd219640 1 --2- 172.21.15.28:0/312885498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12630 0x557981f1a640 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fb1b40ed6e0 tx=0x7fb1b402ffa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d51d9695bec41e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.677+0000 7fb1a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.677+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/312885498 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b4049040 con 0x557981f12630 2024-04-25T01:41:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.677+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/312885498 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fb1b4003c80 con 0x557981f12630 2024-04-25T01:41:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.677+0000 7fb1c218e740 1 -- 172.21.15.28:0/312885498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1b001de50 con 0x557981f12630 2024-04-25T01:41:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.677+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/312885498 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb1b40bb9f0 con 0x557981f12630 2024-04-25T01:41:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.677+0000 7fb1c218e740 10 client.18699 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.677+0000 7fb1c218e740 1 -- 172.21.15.28:0/312885498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557981b4b0f0 con 0x557981f12630 2024-04-25T01:41:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.677+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/312885498 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1b001de50 con 0x557981f12630 2024-04-25T01:41:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.677+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/312885498 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb1b40b8520 con 0x557981f12630 2024-04-25T01:41:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.677+0000 7fb1c218e740 10 client.18699 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.677+0000 7fb1c218e740 10 client.18699 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:41:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.677+0000 7fb1c218e740 20 client.18699 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.677+0000 7fb1c218e740 1 --2- 172.21.15.28:0/312885498 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x557981f2d860 0x557981f2fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.677+0000 7fb1c218e740 4 client.18699 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:41:30.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.677+0000 7fb1c218e740 1 -- 172.21.15.28:0/312885498 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557981e84bf0 con 0x557981f2d860 2024-04-25T01:41:30.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.678+0000 7fb1bca18640 1 --2- 172.21.15.28:0/312885498 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x557981f2d860 0x557981f2fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.678+0000 7fb1a0ff9640 10 client.18699 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:30.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.678+0000 7fb1bca18640 1 --2- 172.21.15.28:0/312885498 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x557981f2d860 0x557981f2fcb0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bff901cc69e8fcb8 server_cookie=874794a7bbc50539 in_seq=0 out_seq=0 2024-04-25T01:41:30.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.679+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/312885498 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557981e84bf0 con 0x557981f2d860 2024-04-25T01:41:30.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.679+0000 7fb1a0ff9640 10 client.18699 handle_command_reply: tid=0 2024-04-25T01:41:30.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.679+0000 7fb1c218e740 1 client.18699 shutdown 2024-04-25T01:41:30.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.679+0000 7fb1c218e740 1 -- 172.21.15.28:0/312885498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12630 msgr2=0x557981f1a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:30.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.679+0000 7fb1c218e740 1 --2- 172.21.15.28:0/312885498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12630 0x557981f1a640 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fb1b40ed6e0 tx=0x7fb1b402ffa0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.679+0000 7fb1c218e740 1 -- 172.21.15.28:0/312885498 shutdown_connections 2024-04-25T01:41:30.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.679+0000 7fb1c218e740 1 --2- 172.21.15.28:0/312885498 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x557981f2d860 0x557981f2fcb0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.679+0000 7fb1c218e740 1 --2- 172.21.15.28:0/312885498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f1d620 0x557981f1b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.679+0000 7fb1c218e740 1 --2- 172.21.15.28:0/312885498 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f12f50 0x557981f1ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.679+0000 7fb1c218e740 1 --2- 172.21.15.28:0/312885498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12630 0x557981f1a640 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.679+0000 7fb1a0ff9640 0 client.18699 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:30.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.679+0000 7fb1c218e740 1 -- 172.21.15.28:0/312885498 >> 172.21.15.28:0/312885498 conn(0x557981f0e950 msgr2=0x557981f14750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:30.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.679+0000 7fb1c218e740 1 -- 172.21.15.28:0/312885498 shutdown_connections 2024-04-25T01:41:30.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.679+0000 7fb1c218e740 1 -- 172.21.15.28:0/312885498 wait complete. 2024-04-25T01:41:30.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.680+0000 7fb1c218e740 20 client.18699 trim_cache size 0 max 16384 2024-04-25T01:41:30.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.697+0000 7fb1c218e740 1 Processor -- start 2024-04-25T01:41:30.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.697+0000 7fb1c218e740 1 -- start start 2024-04-25T01:41:30.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.697+0000 7fb1c218e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f12630 0x557981f1cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.697+0000 7fb1c218e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f12f50 0x557981f29210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.697+0000 7fb1c218e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f1d620 0x557981f145a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.697+0000 7fb1c218e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b81aebf0 con 0x557981f1d620 2024-04-25T01:41:30.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.697+0000 7fb1c218e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b81b0730 con 0x557981f12f50 2024-04-25T01:41:30.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.697+0000 7fb1c218e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b81af2c0 con 0x557981f12630 2024-04-25T01:41:30.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.697+0000 7fb1bda1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f1d620 0x557981f145a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.698+0000 7fb1bda1a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f1d620 0x557981f145a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58260/0 (socket says 172.21.15.28:58260) 2024-04-25T01:41:30.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.698+0000 7fb1bda1a640 1 -- 172.21.15.28:0/4231380741 learned_addr learned my addr 172.21.15.28:0/4231380741 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:30.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.698+0000 7fb1bca18640 1 --2- 172.21.15.28:0/4231380741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f12f50 0x557981f29210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.698+0000 7fb1bd219640 1 --2- 172.21.15.28:0/4231380741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f12630 0x557981f1cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.698+0000 7fb1bda1a640 1 -- 172.21.15.28:0/4231380741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f12630 msgr2=0x557981f1cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:30.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.698+0000 7fb1bda1a640 1 --2- 172.21.15.28:0/4231380741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f12630 0x557981f1cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.698+0000 7fb1bda1a640 1 -- 172.21.15.28:0/4231380741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f12f50 msgr2=0x557981f29210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:30.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.698+0000 7fb1bda1a640 1 --2- 172.21.15.28:0/4231380741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f12f50 0x557981f29210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.698+0000 7fb1bda1a640 1 -- 172.21.15.28:0/4231380741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557981f00090 con 0x557981f1d620 2024-04-25T01:41:30.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.698+0000 7fb1bd219640 1 --2- 172.21.15.28:0/4231380741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f12630 0x557981f1cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:30.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.698+0000 7fb1bca18640 1 --2- 172.21.15.28:0/4231380741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f12f50 0x557981f29210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:41:30.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.698+0000 7fb1bda1a640 1 --2- 172.21.15.28:0/4231380741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f1d620 0x557981f145a0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0011e20 tx=0x7fb1b0005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff0f79ee25e22235 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:30.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.698+0000 7fb1a2ffd640 1 -- 172.21.15.28:0/4231380741 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b0032070 con 0x557981f1d620 2024-04-25T01:41:30.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.698+0000 7fb1a2ffd640 1 -- 172.21.15.28:0/4231380741 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1b001dcb0 con 0x557981f1d620 2024-04-25T01:41:30.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.698+0000 7fb1a2ffd640 1 -- 172.21.15.28:0/4231380741 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b00052a0 con 0x557981f1d620 2024-04-25T01:41:30.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.699+0000 7fb1c218e740 1 -- 172.21.15.28:0/4231380741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f1d620 msgr2=0x557981f145a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:30.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.699+0000 7fb1c218e740 1 --2- 172.21.15.28:0/4231380741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f1d620 0x557981f145a0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0011e20 tx=0x7fb1b0005bd0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.699+0000 7fb1c218e740 1 -- 172.21.15.28:0/4231380741 shutdown_connections 2024-04-25T01:41:30.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.699+0000 7fb1c218e740 1 --2- 172.21.15.28:0/4231380741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f1d620 0x557981f145a0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.699+0000 7fb1c218e740 1 --2- 172.21.15.28:0/4231380741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f12f50 0x557981f29210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.699+0000 7fb1c218e740 1 --2- 172.21.15.28:0/4231380741 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f12630 0x557981f1cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.699+0000 7fb1c218e740 1 -- 172.21.15.28:0/4231380741 >> 172.21.15.28:0/4231380741 conn(0x557981f0e950 msgr2=0x557981f13e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:30.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.699+0000 7fb1c218e740 1 -- 172.21.15.28:0/4231380741 shutdown_connections 2024-04-25T01:41:30.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.699+0000 7fb1c218e740 1 -- 172.21.15.28:0/4231380741 wait complete. 2024-04-25T01:41:30.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.699+0000 7fb1c218e740 1 -- start start 2024-04-25T01:41:30.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.699+0000 7fb1c218e740 1 Processor -- start 2024-04-25T01:41:30.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1c218e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f12630 0x557981f25910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1c218e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12f50 0x557981f16e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1c218e740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f1d620 0x557981f173b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1c218e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b81b0060 con 0x557981f12f50 2024-04-25T01:41:30.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1c218e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b81af990 con 0x557981f12630 2024-04-25T01:41:30.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1c218e740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b81acbf0 con 0x557981f1d620 2024-04-25T01:41:30.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1c218e740 10 client.0 _ll_register_callbacks cb 0x7fb1b4009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:30.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1bca18640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12f50 0x557981f16e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1bca18640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12f50 0x557981f16e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58268/0 (socket says 172.21.15.28:58268) 2024-04-25T01:41:30.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1bca18640 1 -- 172.21.15.28:0/1860467046 learned_addr learned my addr 172.21.15.28:0/1860467046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:30.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1bda1a640 1 --2- 172.21.15.28:0/1860467046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f1d620 0x557981f173b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1bd219640 1 --2- 172.21.15.28:0/1860467046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f12630 0x557981f25910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1bca18640 1 -- 172.21.15.28:0/1860467046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f1d620 msgr2=0x557981f173b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:30.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1bca18640 1 --2- 172.21.15.28:0/1860467046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f1d620 0x557981f173b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1bca18640 1 -- 172.21.15.28:0/1860467046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f12630 msgr2=0x557981f25910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:30.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1bca18640 1 --2- 172.21.15.28:0/1860467046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f12630 0x557981f25910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1bca18640 1 -- 172.21.15.28:0/1860467046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557981f17920 con 0x557981f12f50 2024-04-25T01:41:30.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1bd219640 1 --2- 172.21.15.28:0/1860467046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f12630 0x557981f25910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:41:30.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1bca18640 1 --2- 172.21.15.28:0/1860467046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12f50 0x557981f16e70 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x557981f0fae0 tx=0x7fb1a8009110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd247de65937642 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:30.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:30.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/1860467046 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a8016640 con 0x557981f12f50 2024-04-25T01:41:30.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/1860467046 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fb1a80167e0 con 0x557981f12f50 2024-04-25T01:41:30.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.700+0000 7fb1c218e740 1 -- 172.21.15.28:0/1860467046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1b001dcb0 con 0x557981f12f50 2024-04-25T01:41:30.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.701+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/1860467046 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb1a802f050 con 0x557981f12f50 2024-04-25T01:41:30.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.701+0000 7fb1c218e740 10 client.18705 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:30.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.701+0000 7fb1c218e740 1 -- 172.21.15.28:0/1860467046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557981f93ba0 con 0x557981f12f50 2024-04-25T01:41:30.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.701+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/1860467046 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1b001dcb0 con 0x557981f12f50 2024-04-25T01:41:30.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.701+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/1860467046 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb1a8026ba0 con 0x557981f12f50 2024-04-25T01:41:30.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.701+0000 7fb1c218e740 10 client.18705 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:30.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.701+0000 7fb1c218e740 10 client.18705 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:41:30.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.701+0000 7fb1c218e740 20 client.18705 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.701+0000 7fb1c218e740 1 --2- 172.21.15.28:0/1860467046 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x557981f943a0 0x557981f967c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.701+0000 7fb1c218e740 4 client.18705 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:41:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.701+0000 7fb1c218e740 1 -- 172.21.15.28:0/1860467046 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557981e7ef40 con 0x557981f943a0 2024-04-25T01:41:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.701+0000 7fb1bd219640 1 --2- 172.21.15.28:0/1860467046 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x557981f943a0 0x557981f967c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.702+0000 7fb1bd219640 1 --2- 172.21.15.28:0/1860467046 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x557981f943a0 0x557981f967c0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c40abec42ac167f server_cookie=9004b7738b3f8128 in_seq=0 out_seq=0 2024-04-25T01:41:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.702+0000 7fb1a0ff9640 10 client.18705 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.702+0000 7fb1a0ff9640 1 -- 172.21.15.28:0/1860467046 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557981e7ef40 con 0x557981f943a0 2024-04-25T01:41:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.702+0000 7fb1a0ff9640 10 client.18705 handle_command_reply: tid=0 2024-04-25T01:41:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.702+0000 7fb1c218e740 1 client.18705 shutdown 2024-04-25T01:41:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.702+0000 7fb1c218e740 1 -- 172.21.15.28:0/1860467046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12f50 msgr2=0x557981f16e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.702+0000 7fb1c218e740 1 --2- 172.21.15.28:0/1860467046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12f50 0x557981f16e70 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x557981f0fae0 tx=0x7fb1a8009110 comp rx=0 tx=0).stop 2024-04-25T01:41:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.702+0000 7fb1c218e740 1 -- 172.21.15.28:0/1860467046 shutdown_connections 2024-04-25T01:41:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.702+0000 7fb1c218e740 1 --2- 172.21.15.28:0/1860467046 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x557981f943a0 0x557981f967c0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.702+0000 7fb1c218e740 1 --2- 172.21.15.28:0/1860467046 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x557981f1d620 0x557981f173b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.702+0000 7fb1c218e740 1 --2- 172.21.15.28:0/1860467046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557981f12f50 0x557981f16e70 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.702+0000 7fb1a0ff9640 0 client.18705 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:30.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.702+0000 7fb1c218e740 1 --2- 172.21.15.28:0/1860467046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557981f12630 0x557981f25910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.702+0000 7fb1c218e740 1 -- 172.21.15.28:0/1860467046 >> 172.21.15.28:0/1860467046 conn(0x557981f0e950 msgr2=0x557981f14610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:30.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.702+0000 7fb1c218e740 1 -- 172.21.15.28:0/1860467046 shutdown_connections 2024-04-25T01:41:30.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.702+0000 7fb1c218e740 1 -- 172.21.15.28:0/1860467046 wait complete. 2024-04-25T01:41:30.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.702+0000 7fb1c218e740 20 client.18705 trim_cache size 0 max 16384 2024-04-25T01:41:30.695 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:41:30.695 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:41:30.695 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:41:30.696 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:41:30.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.703+0000 7fb1bf47d640 1 -- 172.21.15.28:0/3555547843 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb1a40a2d80 msgr2=0x7fb1a40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:30.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.703+0000 7fb1bf47d640 1 --2- 172.21.15.28:0/3555547843 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb1a40a2d80 0x7fb1a40a5240 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb1b000fc90 tx=0x7fb1b001e000 comp rx=0 tx=0).stop 2024-04-25T01:41:30.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.703+0000 7fb1bf47d640 1 -- 172.21.15.28:0/3555547843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b8101710 msgr2=0x7fb1b81980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:30.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.703+0000 7fb1bf47d640 1 --2- 172.21.15.28:0/3555547843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b8101710 0x7fb1b81980f0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4009230 tx=0x7fb1b4004270 comp rx=0 tx=0).stop 2024-04-25T01:41:30.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.703+0000 7fb1bf47d640 1 -- 172.21.15.28:0/3555547843 shutdown_connections 2024-04-25T01:41:30.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.703+0000 7fb1bf47d640 1 --2- 172.21.15.28:0/3555547843 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb1a40a2d80 0x7fb1a40a5240 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.703+0000 7fb1bf47d640 1 --2- 172.21.15.28:0/3555547843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb1b8107720 0x7fb1b819c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.703+0000 7fb1bf47d640 1 --2- 172.21.15.28:0/3555547843 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb1b81020d0 0x7fb1b8198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.703+0000 7fb1bf47d640 1 --2- 172.21.15.28:0/3555547843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b8101710 0x7fb1b81980f0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.703+0000 7fb1bf47d640 1 -- 172.21.15.28:0/3555547843 >> 172.21.15.28:0/3555547843 conn(0x7fb1b80fd180 msgr2=0x7fb1b80fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:30.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.703+0000 7fb1bf47d640 1 -- 172.21.15.28:0/3555547843 shutdown_connections 2024-04-25T01:41:30.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.703+0000 7fb1bf47d640 1 -- 172.21.15.28:0/3555547843 wait complete. 2024-04-25T01:41:30.700 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:41:30.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:41:30.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.816+0000 7fe508ff9640 1 -- 172.21.15.28:0/439184552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5041032f0 msgr2=0x7fe50410b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:30.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.816+0000 7fe508ff9640 1 --2- 172.21.15.28:0/439184552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5041032f0 0x7fe50410b7d0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8009900 tx=0x7fe4f802a510 comp rx=0 tx=0).stop 2024-04-25T01:41:30.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.817+0000 7fe508ff9640 1 -- 172.21.15.28:0/439184552 shutdown_connections 2024-04-25T01:41:30.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.817+0000 7fe508ff9640 1 --2- 172.21.15.28:0/439184552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe50410bd10 0x7fe50410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.817+0000 7fe508ff9640 1 --2- 172.21.15.28:0/439184552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5041032f0 0x7fe50410b7d0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.817+0000 7fe508ff9640 1 --2- 172.21.15.28:0/439184552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5041029d0 0x7fe504102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.817+0000 7fe508ff9640 1 -- 172.21.15.28:0/439184552 >> 172.21.15.28:0/439184552 conn(0x7fe504077740 msgr2=0x7fe504077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:30.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.817+0000 7fe508ff9640 1 -- 172.21.15.28:0/439184552 shutdown_connections 2024-04-25T01:41:30.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.817+0000 7fe508ff9640 1 -- 172.21.15.28:0/439184552 wait complete. 2024-04-25T01:41:30.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.817+0000 7fe508ff9640 1 Processor -- start 2024-04-25T01:41:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.817+0000 7fe508ff9640 1 -- start start 2024-04-25T01:41:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe508ff9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5041029d0 0x7fe50419a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe508ff9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5041032f0 0x7fe50419a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe508ff9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe50410bd10 0x7fe50419eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe508ff9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe504111b10 con 0x7fe50410bd10 2024-04-25T01:41:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe508ff9640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe504111990 con 0x7fe5041032f0 2024-04-25T01:41:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe508ff9640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe504111c90 con 0x7fe5041029d0 2024-04-25T01:41:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe502d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe50410bd10 0x7fe50419eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe502d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe50410bd10 0x7fe50419eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58282/0 (socket says 172.21.15.28:58282) 2024-04-25T01:41:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe502d9d640 1 -- 172.21.15.28:0/727066701 learned_addr learned my addr 172.21.15.28:0/727066701 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe501d9b640 1 --2- 172.21.15.28:0/727066701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5041032f0 0x7fe50419a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe50259c640 1 --2- 172.21.15.28:0/727066701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5041029d0 0x7fe50419a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe502d9d640 1 -- 172.21.15.28:0/727066701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5041029d0 msgr2=0x7fe50419a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe502d9d640 1 --2- 172.21.15.28:0/727066701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5041029d0 0x7fe50419a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe502d9d640 1 -- 172.21.15.28:0/727066701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5041032f0 msgr2=0x7fe50419a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe502d9d640 1 --2- 172.21.15.28:0/727066701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5041032f0 0x7fe50419a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe502d9d640 1 -- 172.21.15.28:0/727066701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe50419f270 con 0x7fe50410bd10 2024-04-25T01:41:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe50259c640 1 --2- 172.21.15.28:0/727066701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5041029d0 0x7fe50419a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe502d9d640 1 --2- 172.21.15.28:0/727066701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe50410bd10 0x7fe50419eb40 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc009fd0 tx=0x7fe4fc00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe4f37fe640 1 -- 172.21.15.28:0/727066701 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc017070 con 0x7fe50410bd10 2024-04-25T01:41:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe4f37fe640 1 -- 172.21.15.28:0/727066701 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4fc003e50 con 0x7fe50410bd10 2024-04-25T01:41:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe4f37fe640 1 -- 172.21.15.28:0/727066701 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc0106c0 con 0x7fe50410bd10 2024-04-25T01:41:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe508ff9640 1 -- 172.21.15.28:0/727066701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe50419f560 con 0x7fe50410bd10 2024-04-25T01:41:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.818+0000 7fe508ff9640 1 -- 172.21.15.28:0/727066701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5041a6df0 con 0x7fe50410bd10 2024-04-25T01:41:30.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.819+0000 7fe508ff9640 1 -- 172.21.15.28:0/727066701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe504068560 con 0x7fe50410bd10 2024-04-25T01:41:30.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.821+0000 7fe4f37fe640 1 -- 172.21.15.28:0/727066701 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe4fc003970 con 0x7fe50410bd10 2024-04-25T01:41:30.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.821+0000 7fe4f37fe640 1 --2- 172.21.15.28:0/727066701 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe4ec0a2d80 0x7fe4ec0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:30.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.821+0000 7fe50259c640 1 --2- 172.21.15.28:0/727066701 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe4ec0a2d80 0x7fe4ec0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:30.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.821+0000 7fe4f37fe640 1 -- 172.21.15.28:0/727066701 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fe4fc0cf470 con 0x7fe50410bd10 2024-04-25T01:41:30.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.822+0000 7fe50259c640 1 --2- 172.21.15.28:0/727066701 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe4ec0a2d80 0x7fe4ec0a5240 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4009c20 tx=0x7fe4f4005f90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:30.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:30.825+0000 7fe4f37fe640 1 -- 172.21.15.28:0/727066701 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe4fc09bd80 con 0x7fe50410bd10 2024-04-25T01:41:31.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.047+0000 7fe508ff9640 1 -- 172.21.15.28:0/727066701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5041a7050 con 0x7fe50410bd10 2024-04-25T01:41:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.048+0000 7fe4f37fe640 1 -- 172.21.15.28:0/727066701 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7fe4fc09b680 con 0x7fe50410bd10 2024-04-25T01:41:31.034 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:41:31.034 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:41:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:41:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.050+0000 7fe508ff9640 1 -- 172.21.15.28:0/727066701 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe4ec0a2d80 msgr2=0x7fe4ec0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.050+0000 7fe508ff9640 1 --2- 172.21.15.28:0/727066701 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe4ec0a2d80 0x7fe4ec0a5240 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4009c20 tx=0x7fe4f4005f90 comp rx=0 tx=0).stop 2024-04-25T01:41:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.050+0000 7fe508ff9640 1 -- 172.21.15.28:0/727066701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe50410bd10 msgr2=0x7fe50419eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.050+0000 7fe508ff9640 1 --2- 172.21.15.28:0/727066701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe50410bd10 0x7fe50419eb40 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc009fd0 tx=0x7fe4fc00b260 comp rx=0 tx=0).stop 2024-04-25T01:41:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.050+0000 7fe508ff9640 1 -- 172.21.15.28:0/727066701 shutdown_connections 2024-04-25T01:41:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.050+0000 7fe508ff9640 1 --2- 172.21.15.28:0/727066701 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe4ec0a2d80 0x7fe4ec0a5240 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.050+0000 7fe508ff9640 1 --2- 172.21.15.28:0/727066701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe50410bd10 0x7fe50419eb40 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.050+0000 7fe508ff9640 1 --2- 172.21.15.28:0/727066701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5041032f0 0x7fe50419a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.050+0000 7fe508ff9640 1 --2- 172.21.15.28:0/727066701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5041029d0 0x7fe50419a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.050+0000 7fe508ff9640 1 -- 172.21.15.28:0/727066701 >> 172.21.15.28:0/727066701 conn(0x7fe504077740 msgr2=0x7fe504107cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.050+0000 7fe508ff9640 1 -- 172.21.15.28:0/727066701 shutdown_connections 2024-04-25T01:41:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.050+0000 7fe508ff9640 1 -- 172.21.15.28:0/727066701 wait complete. 2024-04-25T01:41:31.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:41:31.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.166+0000 7f07049c1640 1 -- 172.21.15.28:0/3594286646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07001020d0 msgr2=0x7f070010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.166+0000 7f07049c1640 1 --2- 172.21.15.28:0/3594286646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07001020d0 0x7f070010e5e0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f06f8009cd0 tx=0x7f06f802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.166+0000 7f07049c1640 1 -- 172.21.15.28:0/3594286646 shutdown_connections 2024-04-25T01:41:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.166+0000 7f07049c1640 1 --2- 172.21.15.28:0/3594286646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07001020d0 0x7f070010e5e0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.166+0000 7f07049c1640 1 --2- 172.21.15.28:0/3594286646 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0700101710 0x7f0700101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.166+0000 7f07049c1640 1 --2- 172.21.15.28:0/3594286646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0700107720 0x7f0700107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.166+0000 7f07049c1640 1 -- 172.21.15.28:0/3594286646 >> 172.21.15.28:0/3594286646 conn(0x7f07000fd180 msgr2=0x7f07000ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.166+0000 7f07049c1640 1 -- 172.21.15.28:0/3594286646 shutdown_connections 2024-04-25T01:41:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.166+0000 7f07049c1640 1 -- 172.21.15.28:0/3594286646 wait complete. 2024-04-25T01:41:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.167+0000 7f07049c1640 1 Processor -- start 2024-04-25T01:41:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.167+0000 7f07049c1640 1 -- start start 2024-04-25T01:41:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.167+0000 7f07049c1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0700101710 0x7f0700197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.167+0000 7f07049c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07001020d0 0x7f0700198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.167+0000 7f07049c1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0700107720 0x7f070019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.167+0000 7f07049c1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0700111af0 con 0x7f07001020d0 2024-04-25T01:41:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.167+0000 7f07049c1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0700111970 con 0x7f0700107720 2024-04-25T01:41:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.167+0000 7f07049c1640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0700111c70 con 0x7f0700101710 2024-04-25T01:41:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.167+0000 7f06f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07001020d0 0x7f0700198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.167+0000 7f06f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07001020d0 0x7f0700198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58300/0 (socket says 172.21.15.28:58300) 2024-04-25T01:41:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.167+0000 7f06f7fff640 1 -- 172.21.15.28:0/4033093485 learned_addr learned my addr 172.21.15.28:0/4033093485 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.167+0000 7f06fe59c640 1 --2- 172.21.15.28:0/4033093485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0700101710 0x7f0700197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.168+0000 7f06fed9d640 1 --2- 172.21.15.28:0/4033093485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0700107720 0x7f070019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.168+0000 7f06f7fff640 1 -- 172.21.15.28:0/4033093485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0700101710 msgr2=0x7f0700197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.168+0000 7f06f7fff640 1 --2- 172.21.15.28:0/4033093485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0700101710 0x7f0700197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.168+0000 7f06f7fff640 1 -- 172.21.15.28:0/4033093485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0700107720 msgr2=0x7f070019c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.168+0000 7f06f7fff640 1 --2- 172.21.15.28:0/4033093485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0700107720 0x7f070019c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.168+0000 7f06f7fff640 1 -- 172.21.15.28:0/4033093485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f070019cfe0 con 0x7f07001020d0 2024-04-25T01:41:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.168+0000 7f06f7fff640 1 --2- 172.21.15.28:0/4033093485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07001020d0 0x7f0700198530 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f06e800d950 tx=0x7f06e800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.168+0000 7f06f77fe640 1 -- 172.21.15.28:0/4033093485 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e8017070 con 0x7f07001020d0 2024-04-25T01:41:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.168+0000 7f07049c1640 1 -- 172.21.15.28:0/4033093485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f070019d2d0 con 0x7f07001020d0 2024-04-25T01:41:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.168+0000 7f06f77fe640 1 -- 172.21.15.28:0/4033093485 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06e8009c30 con 0x7f07001020d0 2024-04-25T01:41:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.168+0000 7f06f77fe640 1 -- 172.21.15.28:0/4033093485 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e800f6b0 con 0x7f07001020d0 2024-04-25T01:41:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.168+0000 7f07049c1640 1 -- 172.21.15.28:0/4033093485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07001a4b60 con 0x7f07001020d0 2024-04-25T01:41:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.170+0000 7f06f77fe640 1 -- 172.21.15.28:0/4033093485 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f06e80109d0 con 0x7f07001020d0 2024-04-25T01:41:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.171+0000 7f07077a9740 1 Processor -- start 2024-04-25T01:41:31.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.171+0000 7f07077a9740 1 -- start start 2024-04-25T01:41:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.171+0000 7f07077a9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9df6580 0x5642f9df6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.171+0000 7f07077a9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9df6ea0 0x5642f9e01030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.171+0000 7f07077a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9e01570 0x5642f9e03960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.171+0000 7f06f77fe640 1 --2- 172.21.15.28:0/4033093485 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f06ec0a2d00 0x7f06ec0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.171+0000 7f07077a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5642f9d5b670 con 0x5642f9e01570 2024-04-25T01:41:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.171+0000 7f07077a9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5642f9d03970 con 0x5642f9df6580 2024-04-25T01:41:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.171+0000 7f07077a9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5642f9d57630 con 0x5642f9df6ea0 2024-04-25T01:41:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.171+0000 7f06fed9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9df6580 0x5642f9df6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.171+0000 7f06f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9e01570 0x5642f9e03960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.171+0000 7f06fed9d640 1 --2- 172.21.15.28:0/4033093485 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f06ec0a2d00 0x7f06ec0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.171+0000 7f06f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9e01570 0x5642f9e03960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58306/0 (socket says 172.21.15.28:58306) 2024-04-25T01:41:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.171+0000 7f06f7fff640 1 -- 172.21.15.28:0/415076302 learned_addr learned my addr 172.21.15.28:0/415076302 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.171+0000 7f06fe59c640 1 --2- 172.21.15.28:0/415076302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9df6ea0 0x5642f9e01030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.172+0000 7f06f7fff640 1 -- 172.21.15.28:0/415076302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9df6ea0 msgr2=0x5642f9e01030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.172+0000 7f06f7fff640 1 --2- 172.21.15.28:0/415076302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9df6ea0 0x5642f9e01030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.172+0000 7f06f7fff640 1 -- 172.21.15.28:0/415076302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9df6580 msgr2=0x5642f9df6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.172+0000 7f06f7fff640 1 --2- 172.21.15.28:0/415076302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9df6580 0x5642f9df6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.172+0000 7f06f7fff640 1 -- 172.21.15.28:0/415076302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642f9d27400 con 0x5642f9e01570 2024-04-25T01:41:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.172+0000 7f06f77fe640 1 -- 172.21.15.28:0/4033093485 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f06e80cf3a0 con 0x7f07001020d0 2024-04-25T01:41:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.172+0000 7f06fed9d640 1 --2- 172.21.15.28:0/4033093485 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f06ec0a2d00 0x7f06ec0a51c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f06f8004ab0 tx=0x7f06f8004a40 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.172+0000 7f06f7fff640 1 --2- 172.21.15.28:0/415076302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9e01570 0x5642f9e03960 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f06e80979e0 tx=0x7f06e80c6060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f36e732fe304c30 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.172+0000 7f06f4ff9640 1 -- 172.21.15.28:0/415076302 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e8020030 con 0x5642f9e01570 2024-04-25T01:41:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.172+0000 7f06f4ff9640 1 -- 172.21.15.28:0/415076302 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06e8009de0 con 0x5642f9e01570 2024-04-25T01:41:31.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.172+0000 7f06f4ff9640 1 -- 172.21.15.28:0/415076302 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e809b580 con 0x5642f9e01570 2024-04-25T01:41:31.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.173+0000 7f07077a9740 1 -- 172.21.15.28:0/415076302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9e01570 msgr2=0x5642f9e03960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.173+0000 7f07077a9740 1 --2- 172.21.15.28:0/415076302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9e01570 0x5642f9e03960 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f06e80979e0 tx=0x7f06e80c6060 comp rx=0 tx=0).stop 2024-04-25T01:41:31.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.173+0000 7f07077a9740 1 -- 172.21.15.28:0/415076302 shutdown_connections 2024-04-25T01:41:31.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.173+0000 7f07077a9740 1 --2- 172.21.15.28:0/415076302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9e01570 0x5642f9e03960 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.173+0000 7f07077a9740 1 --2- 172.21.15.28:0/415076302 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9df6ea0 0x5642f9e01030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.173+0000 7f07077a9740 1 --2- 172.21.15.28:0/415076302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9df6580 0x5642f9df6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.173+0000 7f07077a9740 1 -- 172.21.15.28:0/415076302 >> 172.21.15.28:0/415076302 conn(0x5642f9df28a0 msgr2=0x5642f9df2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.173+0000 7f07077a9740 1 -- 172.21.15.28:0/415076302 shutdown_connections 2024-04-25T01:41:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.173+0000 7f07077a9740 1 -- 172.21.15.28:0/415076302 wait complete. 2024-04-25T01:41:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.173+0000 7f07077a9740 1 -- start start 2024-04-25T01:41:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.173+0000 7f07077a9740 1 Processor -- start 2024-04-25T01:41:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f07077a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6580 0x5642f9e00ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f07077a9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9df6ea0 0x5642f9dfe1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f07077a9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9e01570 0x5642f9dfe760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f07077a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5642f9e042a0 con 0x5642f9df6580 2024-04-25T01:41:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f07077a9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5642f9d17050 con 0x5642f9df6ea0 2024-04-25T01:41:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f07077a9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5642f9e04420 con 0x5642f9e01570 2024-04-25T01:41:31.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f07077a9740 10 client.0 _ll_register_callbacks cb 0x5642f9a6ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f06fe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6580 0x5642f9e00ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f06fe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6580 0x5642f9e00ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58322/0 (socket says 172.21.15.28:58322) 2024-04-25T01:41:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f06f7fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9df6ea0 0x5642f9dfe1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f06fe59c640 1 -- 172.21.15.28:0/2427099566 learned_addr learned my addr 172.21.15.28:0/2427099566 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f06fed9d640 1 --2- 172.21.15.28:0/2427099566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9e01570 0x5642f9dfe760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f06fe59c640 1 -- 172.21.15.28:0/2427099566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9e01570 msgr2=0x5642f9dfe760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f06fe59c640 1 --2- 172.21.15.28:0/2427099566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9e01570 0x5642f9dfe760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f06fe59c640 1 -- 172.21.15.28:0/2427099566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9df6ea0 msgr2=0x5642f9dfe1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f06fe59c640 1 --2- 172.21.15.28:0/2427099566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9df6ea0 0x5642f9dfe1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f06fe59c640 1 -- 172.21.15.28:0/2427099566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642f9ce8dc0 con 0x5642f9df6580 2024-04-25T01:41:31.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f06fe59c640 1 --2- 172.21.15.28:0/2427099566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6580 0x5642f9e00ef0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f06f000b940 tx=0x7f06f000be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d28815fc7e99f8ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f06e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.174+0000 7f06e5ffb640 1 -- 172.21.15.28:0/2427099566 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f001b440 con 0x5642f9df6580 2024-04-25T01:41:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.175+0000 7f06e5ffb640 1 -- 172.21.15.28:0/2427099566 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f06f001b5e0 con 0x5642f9df6580 2024-04-25T01:41:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.175+0000 7f07077a9740 1 -- 172.21.15.28:0/2427099566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06e8009de0 con 0x5642f9df6580 2024-04-25T01:41:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.175+0000 7f06e5ffb640 1 -- 172.21.15.28:0/2427099566 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f06f0024050 con 0x5642f9df6580 2024-04-25T01:41:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.175+0000 7f07077a9740 10 client.18723 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.175+0000 7f07077a9740 1 -- 172.21.15.28:0/2427099566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5642f9a6f3f0 con 0x5642f9df6580 2024-04-25T01:41:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.175+0000 7f06e5ffb640 1 -- 172.21.15.28:0/2427099566 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06e8009de0 con 0x5642f9df6580 2024-04-25T01:41:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.175+0000 7f06e5ffb640 1 -- 172.21.15.28:0/2427099566 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f06f0019020 con 0x5642f9df6580 2024-04-25T01:41:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.175+0000 7f07077a9740 10 client.18723 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:31.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.175+0000 7f07077a9740 10 client.18723 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:41:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.175+0000 7f07077a9740 20 client.18723 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.175+0000 7f07077a9740 1 --2- 172.21.15.28:0/2427099566 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5642f9e11580 0x5642f9e139b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.175+0000 7f07077a9740 4 client.18723 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:41:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.175+0000 7f07077a9740 1 -- 172.21.15.28:0/2427099566 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642f9d68bb0 con 0x5642f9e11580 2024-04-25T01:41:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.175+0000 7f06f7fff640 1 --2- 172.21.15.28:0/2427099566 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5642f9e11580 0x5642f9e139b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.176+0000 7f06f7fff640 1 --2- 172.21.15.28:0/2427099566 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5642f9e11580 0x5642f9e139b0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=246bb98327125186 server_cookie=efb2481e84743c52 in_seq=0 out_seq=0 2024-04-25T01:41:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.176+0000 7f06e5ffb640 10 client.18723 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:41:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.177+0000 7f06e5ffb640 1 -- 172.21.15.28:0/2427099566 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5642f9d68bb0 con 0x5642f9e11580 2024-04-25T01:41:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.177+0000 7f06e5ffb640 10 client.18723 handle_command_reply: tid=0 2024-04-25T01:41:31.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.177+0000 7f07077a9740 1 client.18723 shutdown 2024-04-25T01:41:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.177+0000 7f07077a9740 1 -- 172.21.15.28:0/2427099566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6580 msgr2=0x5642f9e00ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.177+0000 7f07077a9740 1 --2- 172.21.15.28:0/2427099566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6580 0x5642f9e00ef0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f06f000b940 tx=0x7f06f000be00 comp rx=0 tx=0).stop 2024-04-25T01:41:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.177+0000 7f07077a9740 1 -- 172.21.15.28:0/2427099566 shutdown_connections 2024-04-25T01:41:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.177+0000 7f07077a9740 1 --2- 172.21.15.28:0/2427099566 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5642f9e11580 0x5642f9e139b0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.177+0000 7f07077a9740 1 --2- 172.21.15.28:0/2427099566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9e01570 0x5642f9dfe760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.177+0000 7f06e5ffb640 0 client.18723 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:41:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.177+0000 7f07077a9740 1 --2- 172.21.15.28:0/2427099566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9df6ea0 0x5642f9dfe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.177+0000 7f07077a9740 1 --2- 172.21.15.28:0/2427099566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6580 0x5642f9e00ef0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.177+0000 7f07077a9740 1 -- 172.21.15.28:0/2427099566 >> 172.21.15.28:0/2427099566 conn(0x5642f9df28a0 msgr2=0x5642f9df86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.177+0000 7f07077a9740 1 -- 172.21.15.28:0/2427099566 shutdown_connections 2024-04-25T01:41:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.177+0000 7f07077a9740 1 -- 172.21.15.28:0/2427099566 wait complete. 2024-04-25T01:41:31.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.177+0000 7f07077a9740 20 client.18723 trim_cache size 0 max 16384 2024-04-25T01:41:31.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.190+0000 7f07077a9740 1 Processor -- start 2024-04-25T01:41:31.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.190+0000 7f07077a9740 1 -- start start 2024-04-25T01:41:31.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.190+0000 7f07077a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6580 0x5642f9e0d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f07077a9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9df6ea0 0x5642f9e19280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f07077a9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9e01570 0x5642f9e197c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f07077a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07001aeca0 con 0x5642f9df6580 2024-04-25T01:41:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f07077a9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f07001b07e0 con 0x5642f9e01570 2024-04-25T01:41:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f07077a9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f07001af370 con 0x5642f9df6ea0 2024-04-25T01:41:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f06fe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6580 0x5642f9e0d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f06fe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6580 0x5642f9e0d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58330/0 (socket says 172.21.15.28:58330) 2024-04-25T01:41:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f06fe59c640 1 -- 172.21.15.28:0/548030890 learned_addr learned my addr 172.21.15.28:0/548030890 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f06f7fff640 1 --2- 172.21.15.28:0/548030890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9df6ea0 0x5642f9e19280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f06fed9d640 1 --2- 172.21.15.28:0/548030890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9e01570 0x5642f9e197c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f06fe59c640 1 -- 172.21.15.28:0/548030890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9df6ea0 msgr2=0x5642f9e19280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f06fe59c640 1 --2- 172.21.15.28:0/548030890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9df6ea0 0x5642f9e19280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f06fe59c640 1 -- 172.21.15.28:0/548030890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9e01570 msgr2=0x5642f9e197c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f06fe59c640 1 --2- 172.21.15.28:0/548030890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9e01570 0x5642f9e197c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f06fe59c640 1 -- 172.21.15.28:0/548030890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642f9b397d0 con 0x5642f9df6580 2024-04-25T01:41:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f06f7fff640 1 --2- 172.21.15.28:0/548030890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9df6ea0 0x5642f9e19280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f06fe59c640 1 --2- 172.21.15.28:0/548030890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6580 0x5642f9e0d160 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f06f0000c00 tx=0x7f06f000bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab0f89a865f1c0dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f06e7fff640 1 -- 172.21.15.28:0/548030890 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f002c020 con 0x5642f9df6580 2024-04-25T01:41:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f06e7fff640 1 -- 172.21.15.28:0/548030890 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06f00039c0 con 0x5642f9df6580 2024-04-25T01:41:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.191+0000 7f06e7fff640 1 -- 172.21.15.28:0/548030890 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f0004660 con 0x5642f9df6580 2024-04-25T01:41:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.192+0000 7f07077a9740 1 -- 172.21.15.28:0/548030890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6580 msgr2=0x5642f9e0d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.192+0000 7f07077a9740 1 --2- 172.21.15.28:0/548030890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6580 0x5642f9e0d160 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f06f0000c00 tx=0x7f06f000bf30 comp rx=0 tx=0).stop 2024-04-25T01:41:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.192+0000 7f07077a9740 1 -- 172.21.15.28:0/548030890 shutdown_connections 2024-04-25T01:41:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.192+0000 7f07077a9740 1 --2- 172.21.15.28:0/548030890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9e01570 0x5642f9e197c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.192+0000 7f07077a9740 1 --2- 172.21.15.28:0/548030890 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9df6ea0 0x5642f9e19280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.192+0000 7f07077a9740 1 --2- 172.21.15.28:0/548030890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6580 0x5642f9e0d160 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.192+0000 7f07077a9740 1 -- 172.21.15.28:0/548030890 >> 172.21.15.28:0/548030890 conn(0x5642f9df28a0 msgr2=0x5642f9df12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.192+0000 7f07077a9740 1 -- 172.21.15.28:0/548030890 shutdown_connections 2024-04-25T01:41:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.192+0000 7f07077a9740 1 -- 172.21.15.28:0/548030890 wait complete. 2024-04-25T01:41:31.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.192+0000 7f07077a9740 1 -- start start 2024-04-25T01:41:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.192+0000 7f07077a9740 1 Processor -- start 2024-04-25T01:41:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.192+0000 7f07077a9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9df6580 0x5642f9e07160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f07077a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6ea0 0x5642f9e076a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f07077a9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9e01570 0x5642f9e07be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f07077a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07001b0110 con 0x5642f9df6ea0 2024-04-25T01:41:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f07077a9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f07001afa40 con 0x5642f9df6580 2024-04-25T01:41:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f07077a9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f07001acca0 con 0x5642f9e01570 2024-04-25T01:41:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f07077a9740 10 client.0 _ll_register_callbacks cb 0x7f06f0004080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f06f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6ea0 0x5642f9e076a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f06f7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6ea0 0x5642f9e076a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58344/0 (socket says 172.21.15.28:58344) 2024-04-25T01:41:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f06f7fff640 1 -- 172.21.15.28:0/3650872893 learned_addr learned my addr 172.21.15.28:0/3650872893 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f06fed9d640 1 --2- 172.21.15.28:0/3650872893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9e01570 0x5642f9e07be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f06fe59c640 1 --2- 172.21.15.28:0/3650872893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9df6580 0x5642f9e07160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f06f7fff640 1 -- 172.21.15.28:0/3650872893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9e01570 msgr2=0x5642f9e07be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f06f7fff640 1 --2- 172.21.15.28:0/3650872893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9e01570 0x5642f9e07be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f06f7fff640 1 -- 172.21.15.28:0/3650872893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9df6580 msgr2=0x5642f9e07160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f06f7fff640 1 --2- 172.21.15.28:0/3650872893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9df6580 0x5642f9e07160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f06f7fff640 1 -- 172.21.15.28:0/3650872893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642f9e76bb0 con 0x5642f9df6ea0 2024-04-25T01:41:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f06fe59c640 1 --2- 172.21.15.28:0/3650872893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9df6580 0x5642f9e07160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:41:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f06f7fff640 1 --2- 172.21.15.28:0/3650872893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6ea0 0x5642f9e076a0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f06e8099830 tx=0x7f06e8010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8134dafde55955da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f06e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f06e5ffb640 1 -- 172.21.15.28:0/3650872893 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e8020040 con 0x5642f9df6ea0 2024-04-25T01:41:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f06e5ffb640 1 -- 172.21.15.28:0/3650872893 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f06e80af6f0 con 0x5642f9df6ea0 2024-04-25T01:41:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f07077a9740 1 -- 172.21.15.28:0/3650872893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06f00039c0 con 0x5642f9df6ea0 2024-04-25T01:41:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f06e5ffb640 1 -- 172.21.15.28:0/3650872893 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f06e809aee0 con 0x5642f9df6ea0 2024-04-25T01:41:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f07077a9740 10 client.18729 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.193+0000 7f07077a9740 1 -- 172.21.15.28:0/3650872893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5642f9e77ef0 con 0x5642f9df6ea0 2024-04-25T01:41:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.194+0000 7f06e5ffb640 1 -- 172.21.15.28:0/3650872893 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06f00039c0 con 0x5642f9df6ea0 2024-04-25T01:41:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.194+0000 7f06e5ffb640 1 -- 172.21.15.28:0/3650872893 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f06e8097470 con 0x5642f9df6ea0 2024-04-25T01:41:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.194+0000 7f07077a9740 10 client.18729 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.194+0000 7f07077a9740 10 client.18729 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:41:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.194+0000 7f07077a9740 20 client.18729 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.194+0000 7f07077a9740 1 --2- 172.21.15.28:0/3650872893 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5642f9e11580 0x5642f9e7a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.194+0000 7f07077a9740 4 client.18729 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:41:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.194+0000 7f07077a9740 1 -- 172.21.15.28:0/3650872893 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5642f9d62f00 con 0x5642f9e11580 2024-04-25T01:41:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.194+0000 7f06fe59c640 1 --2- 172.21.15.28:0/3650872893 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5642f9e11580 0x5642f9e7a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.194+0000 7f06fe59c640 1 --2- 172.21.15.28:0/3650872893 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5642f9e11580 0x5642f9e7a3d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c3c37afd9e77714 server_cookie=8b22e7fb14cf9ed in_seq=0 out_seq=0 2024-04-25T01:41:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.194+0000 7f06e5ffb640 10 client.18729 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:41:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.194+0000 7f06e5ffb640 1 -- 172.21.15.28:0/3650872893 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5642f9d62f00 con 0x5642f9e11580 2024-04-25T01:41:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.194+0000 7f06e5ffb640 10 client.18729 handle_command_reply: tid=0 2024-04-25T01:41:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07077a9740 1 client.18729 shutdown 2024-04-25T01:41:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07077a9740 1 -- 172.21.15.28:0/3650872893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6ea0 msgr2=0x5642f9e076a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07077a9740 1 --2- 172.21.15.28:0/3650872893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6ea0 0x5642f9e076a0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f06e8099830 tx=0x7f06e8010d70 comp rx=0 tx=0).stop 2024-04-25T01:41:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07077a9740 1 -- 172.21.15.28:0/3650872893 shutdown_connections 2024-04-25T01:41:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07077a9740 1 --2- 172.21.15.28:0/3650872893 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5642f9e11580 0x5642f9e7a3d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07077a9740 1 --2- 172.21.15.28:0/3650872893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5642f9e01570 0x5642f9e07be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07077a9740 1 --2- 172.21.15.28:0/3650872893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642f9df6ea0 0x5642f9e076a0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f06e5ffb640 0 client.18729 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:41:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07077a9740 1 --2- 172.21.15.28:0/3650872893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5642f9df6580 0x5642f9e07160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07077a9740 1 -- 172.21.15.28:0/3650872893 >> 172.21.15.28:0/3650872893 conn(0x5642f9df28a0 msgr2=0x5642f9df1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07077a9740 1 -- 172.21.15.28:0/3650872893 shutdown_connections 2024-04-25T01:41:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07077a9740 1 -- 172.21.15.28:0/3650872893 wait complete. 2024-04-25T01:41:31.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07077a9740 20 client.18729 trim_cache size 0 max 16384 2024-04-25T01:41:31.188 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:41:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07049c1640 1 -- 172.21.15.28:0/4033093485 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f06ec0a2d00 msgr2=0x7f06ec0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07049c1640 1 --2- 172.21.15.28:0/4033093485 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f06ec0a2d00 0x7f06ec0a51c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f06f8004ab0 tx=0x7f06f8004a40 comp rx=0 tx=0).stop 2024-04-25T01:41:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07049c1640 1 -- 172.21.15.28:0/4033093485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07001020d0 msgr2=0x7f0700198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07049c1640 1 --2- 172.21.15.28:0/4033093485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07001020d0 0x7f0700198530 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f06e800d950 tx=0x7f06e800de10 comp rx=0 tx=0).stop 2024-04-25T01:41:31.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07049c1640 1 -- 172.21.15.28:0/4033093485 shutdown_connections 2024-04-25T01:41:31.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07049c1640 1 --2- 172.21.15.28:0/4033093485 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f06ec0a2d00 0x7f06ec0a51c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07049c1640 1 --2- 172.21.15.28:0/4033093485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0700107720 0x7f070019c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07049c1640 1 --2- 172.21.15.28:0/4033093485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07001020d0 0x7f0700198530 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07049c1640 1 --2- 172.21.15.28:0/4033093485 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0700101710 0x7f0700197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07049c1640 1 -- 172.21.15.28:0/4033093485 >> 172.21.15.28:0/4033093485 conn(0x7f07000fd180 msgr2=0x7f07000fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07049c1640 1 -- 172.21.15.28:0/4033093485 shutdown_connections 2024-04-25T01:41:31.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.195+0000 7f07049c1640 1 -- 172.21.15.28:0/4033093485 wait complete. 2024-04-25T01:41:31.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:41:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.310+0000 7f32fce38640 1 -- 172.21.15.28:0/3288676566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32f81029d0 msgr2=0x7f32f8102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.310+0000 7f32fce38640 1 --2- 172.21.15.28:0/3288676566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32f81029d0 0x7f32f8102db0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009900 tx=0x7f32e802a510 comp rx=0 tx=0).stop 2024-04-25T01:41:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.310+0000 7f32fce38640 1 -- 172.21.15.28:0/3288676566 shutdown_connections 2024-04-25T01:41:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.310+0000 7f32fce38640 1 --2- 172.21.15.28:0/3288676566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f810bd10 0x7f32f810e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.310+0000 7f32fce38640 1 --2- 172.21.15.28:0/3288676566 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f32f81032f0 0x7f32f810b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.310+0000 7f32fce38640 1 --2- 172.21.15.28:0/3288676566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32f81029d0 0x7f32f8102db0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.310+0000 7f32fce38640 1 -- 172.21.15.28:0/3288676566 >> 172.21.15.28:0/3288676566 conn(0x7f32f80769c0 msgr2=0x7f32f8076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.310+0000 7f32fce38640 1 -- 172.21.15.28:0/3288676566 shutdown_connections 2024-04-25T01:41:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.310+0000 7f32fce38640 1 -- 172.21.15.28:0/3288676566 wait complete. 2024-04-25T01:41:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.311+0000 7f32fce38640 1 Processor -- start 2024-04-25T01:41:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.311+0000 7f32fce38640 1 -- start start 2024-04-25T01:41:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.311+0000 7f32fce38640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f32f81029d0 0x7f32f8100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.311+0000 7f32fce38640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f81032f0 0x7f32f8100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.311+0000 7f32fce38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32f810bd10 0x7f32f81010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.311+0000 7f32fce38640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f32f8111b10 con 0x7f32f810bd10 2024-04-25T01:41:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.311+0000 7f32fce38640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f32f8111990 con 0x7f32f81032f0 2024-04-25T01:41:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.311+0000 7f32fce38640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f32f8111c90 con 0x7f32f81029d0 2024-04-25T01:41:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.312+0000 7f32f6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32f810bd10 0x7f32f81010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.312+0000 7f32f6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32f810bd10 0x7f32f81010a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58376/0 (socket says 172.21.15.28:58376) 2024-04-25T01:41:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.312+0000 7f32f6d9d640 1 -- 172.21.15.28:0/2106401203 learned_addr learned my addr 172.21.15.28:0/2106401203 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.312+0000 7f32f659c640 1 --2- 172.21.15.28:0/2106401203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f32f81029d0 0x7f32f8100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.312+0000 7f32f5d9b640 1 --2- 172.21.15.28:0/2106401203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f81032f0 0x7f32f8100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.312+0000 7f32f6d9d640 1 -- 172.21.15.28:0/2106401203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f32f81029d0 msgr2=0x7f32f8100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.312+0000 7f32f6d9d640 1 --2- 172.21.15.28:0/2106401203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f32f81029d0 0x7f32f8100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.312+0000 7f32f6d9d640 1 -- 172.21.15.28:0/2106401203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f81032f0 msgr2=0x7f32f8100b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.312+0000 7f32f6d9d640 1 --2- 172.21.15.28:0/2106401203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f81032f0 0x7f32f8100b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.312+0000 7f32f6d9d640 1 -- 172.21.15.28:0/2106401203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f81a6510 con 0x7f32f810bd10 2024-04-25T01:41:31.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.312+0000 7f32f6d9d640 1 --2- 172.21.15.28:0/2106401203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32f810bd10 0x7f32f81010a0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f32f000eaa0 tx=0x7f32f000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.312+0000 7f32e77fe640 1 -- 172.21.15.28:0/2106401203 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f000cdb0 con 0x7f32f810bd10 2024-04-25T01:41:31.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.312+0000 7f32fce38640 1 -- 172.21.15.28:0/2106401203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32f81a6800 con 0x7f32f810bd10 2024-04-25T01:41:31.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.312+0000 7f32e77fe640 1 -- 172.21.15.28:0/2106401203 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32f0005020 con 0x7f32f810bd10 2024-04-25T01:41:31.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.312+0000 7f32e77fe640 1 -- 172.21.15.28:0/2106401203 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f00116b0 con 0x7f32f810bd10 2024-04-25T01:41:31.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.312+0000 7f32fce38640 1 -- 172.21.15.28:0/2106401203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32f81a6d00 con 0x7f32f810bd10 2024-04-25T01:41:31.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.315+0000 7f32e77fe640 1 -- 172.21.15.28:0/2106401203 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f32f0005560 con 0x7f32f810bd10 2024-04-25T01:41:31.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.315+0000 7f32ffc25740 1 Processor -- start 2024-04-25T01:41:31.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.315+0000 7f32ffc25740 1 -- start start 2024-04-25T01:41:31.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.315+0000 7f32ffc25740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44580 0x5647ebf44960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.315+0000 7f32ffc25740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf44ea0 0x5647ebf4f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.315+0000 7f32ffc25740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 0x5647ebf51960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.315+0000 7f32ffc25740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5647ebea9670 con 0x5647ebf4f570 2024-04-25T01:41:31.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.315+0000 7f32ffc25740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5647ebe51970 con 0x5647ebf44580 2024-04-25T01:41:31.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.315+0000 7f32ffc25740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5647ebea5630 con 0x5647ebf44ea0 2024-04-25T01:41:31.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.315+0000 7f32f5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 0x5647ebf51960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.315+0000 7f32f5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 0x5647ebf51960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58388/0 (socket says 172.21.15.28:58388) 2024-04-25T01:41:31.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.315+0000 7f32f5d9b640 1 -- 172.21.15.28:0/1561942217 learned_addr learned my addr 172.21.15.28:0/1561942217 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.315+0000 7f32f5d9b640 1 --2- 172.21.15.28:0/1561942217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44580 0x5647ebf44960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.315+0000 7f32f659c640 1 --2- 172.21.15.28:0/1561942217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf44ea0 0x5647ebf4f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.315+0000 7f32e77fe640 1 --2- 172.21.15.28:0/2106401203 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f32e00a2d00 0x7f32e00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.316+0000 7f32f5d9b640 1 -- 172.21.15.28:0/1561942217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf44ea0 msgr2=0x5647ebf4f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.316+0000 7f32f5d9b640 1 --2- 172.21.15.28:0/1561942217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf44ea0 0x5647ebf4f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.316+0000 7f32f5d9b640 1 -- 172.21.15.28:0/1561942217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44580 msgr2=0x5647ebf44960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.316+0000 7f32f6d9d640 1 --2- 172.21.15.28:0/2106401203 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f32e00a2d00 0x7f32e00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.316+0000 7f32f5d9b640 1 --2- 172.21.15.28:0/1561942217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44580 0x5647ebf44960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.316+0000 7f32f5d9b640 1 -- 172.21.15.28:0/1561942217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647ebe75400 con 0x5647ebf4f570 2024-04-25T01:41:31.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.316+0000 7f32e77fe640 1 -- 172.21.15.28:0/2106401203 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f32f00cf980 con 0x7f32f810bd10 2024-04-25T01:41:31.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.316+0000 7f32f5d9b640 1 --2- 172.21.15.28:0/1561942217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 0x5647ebf51960 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f32ec011810 tx=0x7f32ec0097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec9aff0b35541f03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.316+0000 7f32e4ff9640 1 -- 172.21.15.28:0/1561942217 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec004030 con 0x5647ebf4f570 2024-04-25T01:41:31.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.316+0000 7f32f6d9d640 1 --2- 172.21.15.28:0/2106401203 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f32e00a2d00 0x7f32e00a51c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f32f006eba0 tx=0x7f32f006ebd0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.316+0000 7f32e4ff9640 1 -- 172.21.15.28:0/1561942217 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32ec011e50 con 0x5647ebf4f570 2024-04-25T01:41:31.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.316+0000 7f32e4ff9640 1 -- 172.21.15.28:0/1561942217 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec003920 con 0x5647ebf4f570 2024-04-25T01:41:31.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.317+0000 7f32ffc25740 1 -- 172.21.15.28:0/1561942217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 msgr2=0x5647ebf51960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.317+0000 7f32ffc25740 1 --2- 172.21.15.28:0/1561942217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 0x5647ebf51960 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f32ec011810 tx=0x7f32ec0097d0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.317+0000 7f32ffc25740 1 -- 172.21.15.28:0/1561942217 shutdown_connections 2024-04-25T01:41:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.317+0000 7f32ffc25740 1 --2- 172.21.15.28:0/1561942217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 0x5647ebf51960 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.317+0000 7f32ffc25740 1 --2- 172.21.15.28:0/1561942217 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf44ea0 0x5647ebf4f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.317+0000 7f32ffc25740 1 --2- 172.21.15.28:0/1561942217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44580 0x5647ebf44960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.317+0000 7f32ffc25740 1 -- 172.21.15.28:0/1561942217 >> 172.21.15.28:0/1561942217 conn(0x5647ebf408a0 msgr2=0x5647ebf40c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.317+0000 7f32ffc25740 1 -- 172.21.15.28:0/1561942217 shutdown_connections 2024-04-25T01:41:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.317+0000 7f32ffc25740 1 -- 172.21.15.28:0/1561942217 wait complete. 2024-04-25T01:41:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.317+0000 7f32ffc25740 1 -- start start 2024-04-25T01:41:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.317+0000 7f32ffc25740 1 Processor -- start 2024-04-25T01:41:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32ffc25740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf44580 0x5647ebf4eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32ffc25740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44ea0 0x5647ebf4c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32ffc25740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf4f570 0x5647ebf4c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32ffc25740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5647ebf52360 con 0x5647ebf44580 2024-04-25T01:41:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32ffc25740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5647ebe65050 con 0x5647ebf44ea0 2024-04-25T01:41:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32ffc25740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5647ebf524e0 con 0x5647ebf4f570 2024-04-25T01:41:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32ffc25740 10 client.0 _ll_register_callbacks cb 0x5647ebbbcbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32f5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf44580 0x5647ebf4eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32f5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf44580 0x5647ebf4eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58398/0 (socket says 172.21.15.28:58398) 2024-04-25T01:41:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32f5d9b640 1 -- 172.21.15.28:0/3420249045 learned_addr learned my addr 172.21.15.28:0/3420249045 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32f5d9b640 1 --2- 172.21.15.28:0/3420249045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf4f570 0x5647ebf4c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32f659c640 1 --2- 172.21.15.28:0/3420249045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44ea0 0x5647ebf4c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32f5d9b640 1 -- 172.21.15.28:0/3420249045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf4f570 msgr2=0x5647ebf4c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32f5d9b640 1 --2- 172.21.15.28:0/3420249045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf4f570 0x5647ebf4c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32f5d9b640 1 -- 172.21.15.28:0/3420249045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44ea0 msgr2=0x5647ebf4c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32f5d9b640 1 --2- 172.21.15.28:0/3420249045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44ea0 0x5647ebf4c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32f5d9b640 1 -- 172.21.15.28:0/3420249045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647ebe36dc0 con 0x5647ebf44580 2024-04-25T01:41:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32f659c640 1 --2- 172.21.15.28:0/3420249045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44ea0 0x5647ebf4c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:41:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32f5d9b640 1 --2- 172.21.15.28:0/3420249045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf44580 0x5647ebf4eec0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f32ec006010 tx=0x7f32ec009db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22aa03ab053b8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32ddffb640 1 -- 172.21.15.28:0/3420249045 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec028040 con 0x5647ebf44580 2024-04-25T01:41:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32ddffb640 1 -- 172.21.15.28:0/3420249045 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f32ec00f790 con 0x5647ebf44580 2024-04-25T01:41:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32ffc25740 1 -- 172.21.15.28:0/3420249045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32ec011e50 con 0x5647ebf44580 2024-04-25T01:41:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.318+0000 7f32ddffb640 1 -- 172.21.15.28:0/3420249045 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f32ec033050 con 0x5647ebf44580 2024-04-25T01:41:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.319+0000 7f32ffc25740 10 client.18741 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.319+0000 7f32ffc25740 1 -- 172.21.15.28:0/3420249045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5647ebbbd3f0 con 0x5647ebf44580 2024-04-25T01:41:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.319+0000 7f32ddffb640 1 -- 172.21.15.28:0/3420249045 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32ec011e50 con 0x5647ebf44580 2024-04-25T01:41:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.319+0000 7f32ddffb640 1 -- 172.21.15.28:0/3420249045 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f32ec004af0 con 0x5647ebf44580 2024-04-25T01:41:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.319+0000 7f32ffc25740 10 client.18741 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:31.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.319+0000 7f32ffc25740 10 client.18741 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:41:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.319+0000 7f32ffc25740 20 client.18741 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.319+0000 7f32ffc25740 1 --2- 172.21.15.28:0/3420249045 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5647ebf5f450 0x5647ebf61840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.319+0000 7f32ffc25740 4 client.18741 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:41:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.319+0000 7f32ffc25740 1 -- 172.21.15.28:0/3420249045 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647ebeb6bb0 con 0x5647ebf5f450 2024-04-25T01:41:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.319+0000 7f32f659c640 1 --2- 172.21.15.28:0/3420249045 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5647ebf5f450 0x5647ebf61840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.320+0000 7f32ddffb640 10 client.18741 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:41:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.320+0000 7f32f659c640 1 --2- 172.21.15.28:0/3420249045 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5647ebf5f450 0x5647ebf61840 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9b610fae95178083 server_cookie=ac08ef56cd113028 in_seq=0 out_seq=0 2024-04-25T01:41:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.320+0000 7f32ddffb640 1 -- 172.21.15.28:0/3420249045 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5647ebeb6bb0 con 0x5647ebf5f450 2024-04-25T01:41:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.320+0000 7f32ddffb640 10 client.18741 handle_command_reply: tid=0 2024-04-25T01:41:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.320+0000 7f32ffc25740 1 client.18741 shutdown 2024-04-25T01:41:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.320+0000 7f32ffc25740 1 -- 172.21.15.28:0/3420249045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf44580 msgr2=0x5647ebf4eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.320+0000 7f32ffc25740 1 --2- 172.21.15.28:0/3420249045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf44580 0x5647ebf4eec0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f32ec006010 tx=0x7f32ec009db0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.321+0000 7f32ffc25740 1 -- 172.21.15.28:0/3420249045 shutdown_connections 2024-04-25T01:41:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.321+0000 7f32ffc25740 1 --2- 172.21.15.28:0/3420249045 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf4f570 0x5647ebf4c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.321+0000 7f32ffc25740 1 --2- 172.21.15.28:0/3420249045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44ea0 0x5647ebf4c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.321+0000 7f32ffc25740 1 --2- 172.21.15.28:0/3420249045 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5647ebf5f450 0x5647ebf61840 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.321+0000 7f32ffc25740 1 --2- 172.21.15.28:0/3420249045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf44580 0x5647ebf4eec0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.321+0000 7f32ddffb640 0 client.18741 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:41:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.321+0000 7f32ffc25740 1 -- 172.21.15.28:0/3420249045 >> 172.21.15.28:0/3420249045 conn(0x5647ebf408a0 msgr2=0x5647ebf466a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.321+0000 7f32ffc25740 1 -- 172.21.15.28:0/3420249045 shutdown_connections 2024-04-25T01:41:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.321+0000 7f32ffc25740 1 -- 172.21.15.28:0/3420249045 wait complete. 2024-04-25T01:41:31.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.321+0000 7f32ffc25740 20 client.18741 trim_cache size 0 max 16384 2024-04-25T01:41:31.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.334+0000 7f32ffc25740 1 Processor -- start 2024-04-25T01:41:31.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.334+0000 7f32ffc25740 1 -- start start 2024-04-25T01:41:31.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.334+0000 7f32ffc25740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf44580 0x5647ebf5b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.334+0000 7f32ffc25740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44ea0 0x5647ebf4b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.334+0000 7f32ffc25740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 0x5647ebf670e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.334+0000 7f32ffc25740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f32f81b0e30 con 0x5647ebf4f570 2024-04-25T01:41:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.334+0000 7f32ffc25740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f32f81b2970 con 0x5647ebf44ea0 2024-04-25T01:41:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.334+0000 7f32ffc25740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f32f81b1500 con 0x5647ebf44580 2024-04-25T01:41:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.334+0000 7f32f5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 0x5647ebf670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.334+0000 7f32f5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 0x5647ebf670e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58412/0 (socket says 172.21.15.28:58412) 2024-04-25T01:41:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.334+0000 7f32f5d9b640 1 -- 172.21.15.28:0/1314385428 learned_addr learned my addr 172.21.15.28:0/1314385428 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.334+0000 7f32f659c640 1 --2- 172.21.15.28:0/1314385428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44ea0 0x5647ebf4b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.334+0000 7f32f5d9b640 1 --2- 172.21.15.28:0/1314385428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf44580 0x5647ebf5b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.335+0000 7f32f5d9b640 1 -- 172.21.15.28:0/1314385428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf44580 msgr2=0x5647ebf5b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.335+0000 7f32f5d9b640 1 --2- 172.21.15.28:0/1314385428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf44580 0x5647ebf5b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.335+0000 7f32f5d9b640 1 -- 172.21.15.28:0/1314385428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44ea0 msgr2=0x5647ebf4b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.335+0000 7f32f5d9b640 1 --2- 172.21.15.28:0/1314385428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44ea0 0x5647ebf4b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.335+0000 7f32f5d9b640 1 -- 172.21.15.28:0/1314385428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647ebf3f990 con 0x5647ebf4f570 2024-04-25T01:41:31.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.335+0000 7f32f659c640 1 --2- 172.21.15.28:0/1314385428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44ea0 0x5647ebf4b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:31.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.335+0000 7f32f5d9b640 1 --2- 172.21.15.28:0/1314385428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 0x5647ebf670e0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f32ec004ec0 tx=0x7f32ec00ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e2c5b40eb860afc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.335+0000 7f32dffff640 1 -- 172.21.15.28:0/1314385428 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec023070 con 0x5647ebf4f570 2024-04-25T01:41:31.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.335+0000 7f32dffff640 1 -- 172.21.15.28:0/1314385428 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32ec011e50 con 0x5647ebf4f570 2024-04-25T01:41:31.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.335+0000 7f32dffff640 1 -- 172.21.15.28:0/1314385428 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec00fdb0 con 0x5647ebf4f570 2024-04-25T01:41:31.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.335+0000 7f32ffc25740 1 -- 172.21.15.28:0/1314385428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 msgr2=0x5647ebf670e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.335+0000 7f32ffc25740 1 --2- 172.21.15.28:0/1314385428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 0x5647ebf670e0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f32ec004ec0 tx=0x7f32ec00ff90 comp rx=0 tx=0).stop 2024-04-25T01:41:31.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32ffc25740 1 -- 172.21.15.28:0/1314385428 shutdown_connections 2024-04-25T01:41:31.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32ffc25740 1 --2- 172.21.15.28:0/1314385428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 0x5647ebf670e0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32ffc25740 1 --2- 172.21.15.28:0/1314385428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44ea0 0x5647ebf4b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32ffc25740 1 --2- 172.21.15.28:0/1314385428 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf44580 0x5647ebf5b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32ffc25740 1 -- 172.21.15.28:0/1314385428 >> 172.21.15.28:0/1314385428 conn(0x5647ebf408a0 msgr2=0x5647ebf40c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32ffc25740 1 -- 172.21.15.28:0/1314385428 shutdown_connections 2024-04-25T01:41:31.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32ffc25740 1 -- 172.21.15.28:0/1314385428 wait complete. 2024-04-25T01:41:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32ffc25740 1 -- start start 2024-04-25T01:41:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32ffc25740 1 Processor -- start 2024-04-25T01:41:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32ffc25740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf44580 0x5647ebf57bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32ffc25740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44ea0 0x5647ebf54f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32ffc25740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 0x5647ebf55560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32ffc25740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f32f81b22a0 con 0x5647ebf4f570 2024-04-25T01:41:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32ffc25740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f32f81b1bd0 con 0x5647ebf44ea0 2024-04-25T01:41:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32ffc25740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f32f81aee30 con 0x5647ebf44580 2024-04-25T01:41:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32ffc25740 10 client.0 _ll_register_callbacks cb 0x7f32ec009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32f5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 0x5647ebf55560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32f5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 0x5647ebf55560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58418/0 (socket says 172.21.15.28:58418) 2024-04-25T01:41:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32f5d9b640 1 -- 172.21.15.28:0/2269442101 learned_addr learned my addr 172.21.15.28:0/2269442101 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32f659c640 1 --2- 172.21.15.28:0/2269442101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44ea0 0x5647ebf54f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.336+0000 7f32f5d9b640 1 --2- 172.21.15.28:0/2269442101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf44580 0x5647ebf57bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32f5d9b640 1 -- 172.21.15.28:0/2269442101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf44580 msgr2=0x5647ebf57bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32f5d9b640 1 --2- 172.21.15.28:0/2269442101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf44580 0x5647ebf57bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32f5d9b640 1 -- 172.21.15.28:0/2269442101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44ea0 msgr2=0x5647ebf54f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32f5d9b640 1 --2- 172.21.15.28:0/2269442101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44ea0 0x5647ebf54f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32f5d9b640 1 -- 172.21.15.28:0/2269442101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647ebf61bb0 con 0x5647ebf4f570 2024-04-25T01:41:31.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32f659c640 1 --2- 172.21.15.28:0/2269442101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44ea0 0x5647ebf54f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:31.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32f5d9b640 1 --2- 172.21.15.28:0/2269442101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 0x5647ebf55560 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f32ec0059f0 tx=0x7f32ec039040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4153779c988f1b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:31.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32ddffb640 1 -- 172.21.15.28:0/2269442101 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ec023070 con 0x5647ebf4f570 2024-04-25T01:41:31.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32ddffb640 1 -- 172.21.15.28:0/2269442101 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f32ec01fa00 con 0x5647ebf4f570 2024-04-25T01:41:31.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32ffc25740 1 -- 172.21.15.28:0/2269442101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32ec011e50 con 0x5647ebf4f570 2024-04-25T01:41:31.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32ddffb640 1 -- 172.21.15.28:0/2269442101 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f32ec009780 con 0x5647ebf4f570 2024-04-25T01:41:31.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32ffc25740 10 client.18747 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:31.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32ffc25740 1 -- 172.21.15.28:0/2269442101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5647ebf4ca80 con 0x5647ebf4f570 2024-04-25T01:41:31.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32ddffb640 1 -- 172.21.15.28:0/2269442101 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32ec011e50 con 0x5647ebf4f570 2024-04-25T01:41:31.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32ddffb640 1 -- 172.21.15.28:0/2269442101 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f32ec037400 con 0x5647ebf4f570 2024-04-25T01:41:31.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32ffc25740 10 client.18747 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:31.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32ffc25740 10 client.18747 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:41:31.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32ffc25740 20 client.18747 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:31.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32ffc25740 1 --2- 172.21.15.28:0/2269442101 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5647ebfc5c80 0x5647ebfc8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32ffc25740 4 client.18747 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:41:31.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.337+0000 7f32ffc25740 1 -- 172.21.15.28:0/2269442101 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5647ebeb0f00 con 0x5647ebfc5c80 2024-04-25T01:41:31.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.338+0000 7f32f659c640 1 --2- 172.21.15.28:0/2269442101 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5647ebfc5c80 0x5647ebfc8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.338+0000 7f32ddffb640 10 client.18747 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:41:31.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.338+0000 7f32f659c640 1 --2- 172.21.15.28:0/2269442101 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5647ebfc5c80 0x5647ebfc8070 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=89d37ea710f46a31 server_cookie=6012c24e5edbbf56 in_seq=0 out_seq=0 2024-04-25T01:41:31.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.338+0000 7f32ddffb640 1 -- 172.21.15.28:0/2269442101 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5647ebeb0f00 con 0x5647ebfc5c80 2024-04-25T01:41:31.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.338+0000 7f32ddffb640 10 client.18747 handle_command_reply: tid=0 2024-04-25T01:41:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.338+0000 7f32ffc25740 1 client.18747 shutdown 2024-04-25T01:41:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.338+0000 7f32ffc25740 1 -- 172.21.15.28:0/2269442101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 msgr2=0x5647ebf55560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.338+0000 7f32ffc25740 1 --2- 172.21.15.28:0/2269442101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 0x5647ebf55560 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f32ec0059f0 tx=0x7f32ec039040 comp rx=0 tx=0).stop 2024-04-25T01:41:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.338+0000 7f32ffc25740 1 -- 172.21.15.28:0/2269442101 shutdown_connections 2024-04-25T01:41:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.338+0000 7f32ffc25740 1 --2- 172.21.15.28:0/2269442101 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x5647ebfc5c80 0x5647ebfc8070 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.338+0000 7f32ffc25740 1 --2- 172.21.15.28:0/2269442101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647ebf4f570 0x5647ebf55560 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.338+0000 7f32ddffb640 0 client.18747 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:41:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.338+0000 7f32ffc25740 1 --2- 172.21.15.28:0/2269442101 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5647ebf44ea0 0x5647ebf54f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.338+0000 7f32ffc25740 1 --2- 172.21.15.28:0/2269442101 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5647ebf44580 0x5647ebf57bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.338+0000 7f32ffc25740 1 -- 172.21.15.28:0/2269442101 >> 172.21.15.28:0/2269442101 conn(0x5647ebf408a0 msgr2=0x5647ebf4b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.338+0000 7f32ffc25740 1 -- 172.21.15.28:0/2269442101 shutdown_connections 2024-04-25T01:41:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.338+0000 7f32ffc25740 1 -- 172.21.15.28:0/2269442101 wait complete. 2024-04-25T01:41:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.339+0000 7f32ffc25740 20 client.18747 trim_cache size 0 max 16384 2024-04-25T01:41:31.333 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:41:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.339+0000 7f32fce38640 1 -- 172.21.15.28:0/2106401203 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f32e00a2d00 msgr2=0x7f32e00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.339+0000 7f32fce38640 1 --2- 172.21.15.28:0/2106401203 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f32e00a2d00 0x7f32e00a51c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f32f006eba0 tx=0x7f32f006ebd0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.339+0000 7f32fce38640 1 -- 172.21.15.28:0/2106401203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32f810bd10 msgr2=0x7f32f81010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.339+0000 7f32fce38640 1 --2- 172.21.15.28:0/2106401203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32f810bd10 0x7f32f81010a0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f32f000eaa0 tx=0x7f32f000ef60 comp rx=0 tx=0).stop 2024-04-25T01:41:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.339+0000 7f32fce38640 1 -- 172.21.15.28:0/2106401203 shutdown_connections 2024-04-25T01:41:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.339+0000 7f32fce38640 1 --2- 172.21.15.28:0/2106401203 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f32e00a2d00 0x7f32e00a51c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.339+0000 7f32fce38640 1 --2- 172.21.15.28:0/2106401203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32f810bd10 0x7f32f81010a0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.339+0000 7f32fce38640 1 --2- 172.21.15.28:0/2106401203 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f32f81032f0 0x7f32f8100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.339+0000 7f32fce38640 1 --2- 172.21.15.28:0/2106401203 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f32f81029d0 0x7f32f8100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.339+0000 7f32fce38640 1 -- 172.21.15.28:0/2106401203 >> 172.21.15.28:0/2106401203 conn(0x7f32f80769c0 msgr2=0x7f32f80fd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.339+0000 7f32fce38640 1 -- 172.21.15.28:0/2106401203 shutdown_connections 2024-04-25T01:41:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.339+0000 7f32fce38640 1 -- 172.21.15.28:0/2106401203 wait complete. 2024-04-25T01:41:31.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:41:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.454+0000 7ff9871ce640 1 -- 172.21.15.28:0/875774473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9801020d0 msgr2=0x7ff98010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.454+0000 7ff9871ce640 1 --2- 172.21.15.28:0/875774473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9801020d0 0x7ff98010e5e0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7ff978009cd0 tx=0x7ff97802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.454+0000 7ff9871ce640 1 -- 172.21.15.28:0/875774473 shutdown_connections 2024-04-25T01:41:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.454+0000 7ff9871ce640 1 --2- 172.21.15.28:0/875774473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9801020d0 0x7ff98010e5e0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.454+0000 7ff9871ce640 1 --2- 172.21.15.28:0/875774473 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff980101710 0x7ff980101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.454+0000 7ff9871ce640 1 --2- 172.21.15.28:0/875774473 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff980107720 0x7ff980107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.454+0000 7ff9871ce640 1 -- 172.21.15.28:0/875774473 >> 172.21.15.28:0/875774473 conn(0x7ff9800fd180 msgr2=0x7ff9800ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.454+0000 7ff9871ce640 1 -- 172.21.15.28:0/875774473 shutdown_connections 2024-04-25T01:41:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.454+0000 7ff9871ce640 1 -- 172.21.15.28:0/875774473 wait complete. 2024-04-25T01:41:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.454+0000 7ff9871ce640 1 Processor -- start 2024-04-25T01:41:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.454+0000 7ff9871ce640 1 -- start start 2024-04-25T01:41:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.454+0000 7ff9871ce640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff980101710 0x7ff980198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.454+0000 7ff9871ce640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9801020d0 0x7ff9801985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.454+0000 7ff9871ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff980107720 0x7ff98019c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.454+0000 7ff9871ce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff980111b30 con 0x7ff980107720 2024-04-25T01:41:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.454+0000 7ff9871ce640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff9801119b0 con 0x7ff980101710 2024-04-25T01:41:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.454+0000 7ff9871ce640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff980111cb0 con 0x7ff9801020d0 2024-04-25T01:41:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.455+0000 7ff98576b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff980107720 0x7ff98019c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.455+0000 7ff98576b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff980107720 0x7ff98019c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58446/0 (socket says 172.21.15.28:58446) 2024-04-25T01:41:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.455+0000 7ff98576b640 1 -- 172.21.15.28:0/2187171816 learned_addr learned my addr 172.21.15.28:0/2187171816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.455+0000 7ff97ffff640 1 --2- 172.21.15.28:0/2187171816 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9801020d0 0x7ff9801985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.455+0000 7ff984f6a640 1 --2- 172.21.15.28:0/2187171816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff980101710 0x7ff980198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.455+0000 7ff98576b640 1 -- 172.21.15.28:0/2187171816 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9801020d0 msgr2=0x7ff9801985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.455+0000 7ff98576b640 1 --2- 172.21.15.28:0/2187171816 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9801020d0 0x7ff9801985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.455+0000 7ff98576b640 1 -- 172.21.15.28:0/2187171816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff980101710 msgr2=0x7ff980198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.455+0000 7ff98576b640 1 --2- 172.21.15.28:0/2187171816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff980101710 0x7ff980198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.455+0000 7ff98576b640 1 -- 172.21.15.28:0/2187171816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98019d060 con 0x7ff980107720 2024-04-25T01:41:31.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.455+0000 7ff98576b640 1 --2- 172.21.15.28:0/2187171816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff980107720 0x7ff98019c930 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7ff978005cc0 tx=0x7ff978003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.455+0000 7ff97dffb640 1 -- 172.21.15.28:0/2187171816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978004040 con 0x7ff980107720 2024-04-25T01:41:31.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.455+0000 7ff9871ce640 1 -- 172.21.15.28:0/2187171816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff98019d2f0 con 0x7ff980107720 2024-04-25T01:41:31.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.455+0000 7ff97dffb640 1 -- 172.21.15.28:0/2187171816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9780046c0 con 0x7ff980107720 2024-04-25T01:41:31.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.455+0000 7ff97dffb640 1 -- 172.21.15.28:0/2187171816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978046480 con 0x7ff980107720 2024-04-25T01:41:31.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.456+0000 7ff9871ce640 1 -- 172.21.15.28:0/2187171816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9801a4be0 con 0x7ff980107720 2024-04-25T01:41:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.458+0000 7ff97dffb640 1 -- 172.21.15.28:0/2187171816 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff9780305a0 con 0x7ff980107720 2024-04-25T01:41:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.458+0000 7ff989fba740 1 Processor -- start 2024-04-25T01:41:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.458+0000 7ff989fba740 1 -- start start 2024-04-25T01:41:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.458+0000 7ff989fba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 0x55b66cecf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.458+0000 7ff989fba740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecfea0 0x55b66ceda030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.458+0000 7ff989fba740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b66ceda570 0x55b66cedc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.458+0000 7ff989fba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b66ce34670 con 0x55b66cecf580 2024-04-25T01:41:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.458+0000 7ff989fba740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b66cddc970 con 0x55b66cecfea0 2024-04-25T01:41:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.458+0000 7ff989fba740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b66ce30630 con 0x55b66ceda570 2024-04-25T01:41:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.458+0000 7ff97ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 0x55b66cecf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.458+0000 7ff97dffb640 1 --2- 172.21.15.28:0/2187171816 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff9700a2d00 0x7ff9700a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.458+0000 7ff97ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 0x55b66cecf960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58460/0 (socket says 172.21.15.28:58460) 2024-04-25T01:41:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.458+0000 7ff97ffff640 1 -- 172.21.15.28:0/2586679054 learned_addr learned my addr 172.21.15.28:0/2586679054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.458+0000 7ff984f6a640 1 --2- 172.21.15.28:0/2586679054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecfea0 0x55b66ceda030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.459+0000 7ff98576b640 1 --2- 172.21.15.28:0/2187171816 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff9700a2d00 0x7ff9700a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.459+0000 7ff97ffff640 1 --2- 172.21.15.28:0/2586679054 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b66ceda570 0x55b66cedc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.459+0000 7ff97dffb640 1 -- 172.21.15.28:0/2187171816 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7ff97802e030 con 0x7ff980107720 2024-04-25T01:41:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.459+0000 7ff97ffff640 1 -- 172.21.15.28:0/2586679054 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b66ceda570 msgr2=0x55b66cedc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.459+0000 7ff97ffff640 1 --2- 172.21.15.28:0/2586679054 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b66ceda570 0x55b66cedc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.459+0000 7ff97ffff640 1 -- 172.21.15.28:0/2586679054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecfea0 msgr2=0x55b66ceda030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.459+0000 7ff97ffff640 1 --2- 172.21.15.28:0/2586679054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecfea0 0x55b66ceda030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.459+0000 7ff97ffff640 1 -- 172.21.15.28:0/2586679054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b66ce00400 con 0x55b66cecf580 2024-04-25T01:41:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.459+0000 7ff97ffff640 1 --2- 172.21.15.28:0/2586679054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 0x55b66cecf960 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7ff96c00bba0 tx=0x7ff96c004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=765e688586c848c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.459+0000 7ff98576b640 1 --2- 172.21.15.28:0/2187171816 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff9700a2d00 0x7ff9700a51c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff9780ee470 tx=0x7ff9780dd040 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.459+0000 7ff96affd640 1 -- 172.21.15.28:0/2586679054 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96c00be90 con 0x55b66cecf580 2024-04-25T01:41:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.459+0000 7ff96affd640 1 -- 172.21.15.28:0/2586679054 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff96c005500 con 0x55b66cecf580 2024-04-25T01:41:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.459+0000 7ff96affd640 1 -- 172.21.15.28:0/2586679054 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96c0106c0 con 0x55b66cecf580 2024-04-25T01:41:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.460+0000 7ff989fba740 1 -- 172.21.15.28:0/2586679054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 msgr2=0x55b66cecf960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.460+0000 7ff989fba740 1 --2- 172.21.15.28:0/2586679054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 0x55b66cecf960 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7ff96c00bba0 tx=0x7ff96c004250 comp rx=0 tx=0).stop 2024-04-25T01:41:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.460+0000 7ff989fba740 1 -- 172.21.15.28:0/2586679054 shutdown_connections 2024-04-25T01:41:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.460+0000 7ff989fba740 1 --2- 172.21.15.28:0/2586679054 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b66ceda570 0x55b66cedc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.460+0000 7ff989fba740 1 --2- 172.21.15.28:0/2586679054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecfea0 0x55b66ceda030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.460+0000 7ff989fba740 1 --2- 172.21.15.28:0/2586679054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 0x55b66cecf960 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.460+0000 7ff989fba740 1 -- 172.21.15.28:0/2586679054 >> 172.21.15.28:0/2586679054 conn(0x55b66cecb8a0 msgr2=0x55b66cecbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.460+0000 7ff989fba740 1 -- 172.21.15.28:0/2586679054 shutdown_connections 2024-04-25T01:41:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.460+0000 7ff989fba740 1 -- 172.21.15.28:0/2586679054 wait complete. 2024-04-25T01:41:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.460+0000 7ff989fba740 1 -- start start 2024-04-25T01:41:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.460+0000 7ff989fba740 1 Processor -- start 2024-04-25T01:41:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.461+0000 7ff989fba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 0x55b66ced9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.461+0000 7ff989fba740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecfea0 0x55b66ced7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.461+0000 7ff989fba740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b66ceda570 0x55b66ced7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.461+0000 7ff989fba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b66cedd300 con 0x55b66cecf580 2024-04-25T01:41:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.461+0000 7ff989fba740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b66cdf0050 con 0x55b66cecfea0 2024-04-25T01:41:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.461+0000 7ff989fba740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b66cedd480 con 0x55b66ceda570 2024-04-25T01:41:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.461+0000 7ff989fba740 10 client.0 _ll_register_callbacks cb 0x55b66cb47be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.461+0000 7ff97ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 0x55b66ced9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.461+0000 7ff97ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 0x55b66ced9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58470/0 (socket says 172.21.15.28:58470) 2024-04-25T01:41:31.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.461+0000 7ff97ffff640 1 -- 172.21.15.28:0/3653202992 learned_addr learned my addr 172.21.15.28:0/3653202992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.461+0000 7ff984f6a640 1 --2- 172.21.15.28:0/3653202992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecfea0 0x55b66ced7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.461+0000 7ff97ffff640 1 -- 172.21.15.28:0/3653202992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b66ceda570 msgr2=0x55b66ced7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.461+0000 7ff97ffff640 1 --2- 172.21.15.28:0/3653202992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b66ceda570 0x55b66ced7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.461+0000 7ff97ffff640 1 -- 172.21.15.28:0/3653202992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecfea0 msgr2=0x55b66ced7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.461+0000 7ff97ffff640 1 --2- 172.21.15.28:0/3653202992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecfea0 0x55b66ced7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.461+0000 7ff97ffff640 1 -- 172.21.15.28:0/3653202992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b66cdc1dc0 con 0x55b66cecf580 2024-04-25T01:41:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.461+0000 7ff97ffff640 1 --2- 172.21.15.28:0/3653202992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 0x55b66ced9e60 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7ff96c004d60 tx=0x7ff96c004350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31cb0305c73ba81c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.461+0000 7ff95ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.461+0000 7ff95ffff640 1 -- 172.21.15.28:0/3653202992 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96c0056b0 con 0x55b66cecf580 2024-04-25T01:41:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.462+0000 7ff95ffff640 1 -- 172.21.15.28:0/3653202992 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7ff96c005000 con 0x55b66cecf580 2024-04-25T01:41:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.462+0000 7ff989fba740 1 -- 172.21.15.28:0/3653202992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff96c005500 con 0x55b66cecf580 2024-04-25T01:41:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.462+0000 7ff95ffff640 1 -- 172.21.15.28:0/3653202992 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff96c005200 con 0x55b66cecf580 2024-04-25T01:41:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.462+0000 7ff989fba740 10 client.18759 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.462+0000 7ff989fba740 1 -- 172.21.15.28:0/3653202992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b66cb483f0 con 0x55b66cecf580 2024-04-25T01:41:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.462+0000 7ff95ffff640 1 -- 172.21.15.28:0/3653202992 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff96c005500 con 0x55b66cecf580 2024-04-25T01:41:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.462+0000 7ff95ffff640 1 -- 172.21.15.28:0/3653202992 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff96c020450 con 0x55b66cecf580 2024-04-25T01:41:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.462+0000 7ff989fba740 10 client.18759 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.462+0000 7ff989fba740 10 client.18759 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:41:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.462+0000 7ff989fba740 20 client.18759 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.462+0000 7ff989fba740 1 --2- 172.21.15.28:0/3653202992 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b66ceea4b0 0x55b66ceec8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.462+0000 7ff989fba740 4 client.18759 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:41:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.462+0000 7ff989fba740 1 -- 172.21.15.28:0/3653202992 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b66ce41bb0 con 0x55b66ceea4b0 2024-04-25T01:41:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.462+0000 7ff984f6a640 1 --2- 172.21.15.28:0/3653202992 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b66ceea4b0 0x55b66ceec8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.463+0000 7ff984f6a640 1 --2- 172.21.15.28:0/3653202992 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b66ceea4b0 0x55b66ceec8a0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5d709b8ba22700ea server_cookie=d3e4b610c94abdc1 in_seq=0 out_seq=0 2024-04-25T01:41:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.463+0000 7ff95ffff640 10 client.18759 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:41:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.463+0000 7ff95ffff640 1 -- 172.21.15.28:0/3653202992 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b66ce41bb0 con 0x55b66ceea4b0 2024-04-25T01:41:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.463+0000 7ff95ffff640 10 client.18759 handle_command_reply: tid=0 2024-04-25T01:41:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.464+0000 7ff989fba740 1 client.18759 shutdown 2024-04-25T01:41:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.464+0000 7ff989fba740 1 -- 172.21.15.28:0/3653202992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 msgr2=0x55b66ced9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.464+0000 7ff989fba740 1 --2- 172.21.15.28:0/3653202992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 0x55b66ced9e60 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7ff96c004d60 tx=0x7ff96c004350 comp rx=0 tx=0).stop 2024-04-25T01:41:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.464+0000 7ff989fba740 1 -- 172.21.15.28:0/3653202992 shutdown_connections 2024-04-25T01:41:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.464+0000 7ff989fba740 1 --2- 172.21.15.28:0/3653202992 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b66ceea4b0 0x55b66ceec8a0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.464+0000 7ff989fba740 1 --2- 172.21.15.28:0/3653202992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b66ceda570 0x55b66ced7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.464+0000 7ff989fba740 1 --2- 172.21.15.28:0/3653202992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecfea0 0x55b66ced7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.464+0000 7ff989fba740 1 --2- 172.21.15.28:0/3653202992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 0x55b66ced9e60 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.464+0000 7ff95ffff640 0 client.18759 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:41:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.464+0000 7ff989fba740 1 -- 172.21.15.28:0/3653202992 >> 172.21.15.28:0/3653202992 conn(0x55b66cecb8a0 msgr2=0x55b66cecab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.464+0000 7ff989fba740 1 -- 172.21.15.28:0/3653202992 shutdown_connections 2024-04-25T01:41:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.464+0000 7ff989fba740 1 -- 172.21.15.28:0/3653202992 wait complete. 2024-04-25T01:41:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.464+0000 7ff989fba740 20 client.18759 trim_cache size 0 max 16384 2024-04-25T01:41:31.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.477+0000 7ff989fba740 1 Processor -- start 2024-04-25T01:41:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.477+0000 7ff989fba740 1 -- start start 2024-04-25T01:41:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.477+0000 7ff989fba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 0x55b66cee6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.477+0000 7ff989fba740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecfea0 0x55b66ced9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.477+0000 7ff989fba740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b66ceda570 0x55b66cef20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.477+0000 7ff989fba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff9801aeaf0 con 0x55b66cecf580 2024-04-25T01:41:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.477+0000 7ff989fba740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff9801b0630 con 0x55b66cecfea0 2024-04-25T01:41:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.477+0000 7ff989fba740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff9801af1c0 con 0x55b66ceda570 2024-04-25T01:41:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.478+0000 7ff97ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 0x55b66cee6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.478+0000 7ff97ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 0x55b66cee6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58478/0 (socket says 172.21.15.28:58478) 2024-04-25T01:41:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.478+0000 7ff97ffff640 1 -- 172.21.15.28:0/1175110420 learned_addr learned my addr 172.21.15.28:0/1175110420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.478+0000 7ff984f6a640 1 --2- 172.21.15.28:0/1175110420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecfea0 0x55b66ced9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.478+0000 7ff97ffff640 1 -- 172.21.15.28:0/1175110420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b66ceda570 msgr2=0x55b66cef20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.478+0000 7ff97ffff640 1 --2- 172.21.15.28:0/1175110420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b66ceda570 0x55b66cef20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.478+0000 7ff97ffff640 1 -- 172.21.15.28:0/1175110420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecfea0 msgr2=0x55b66ced9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.478+0000 7ff97ffff640 1 --2- 172.21.15.28:0/1175110420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecfea0 0x55b66ced9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.478+0000 7ff97ffff640 1 -- 172.21.15.28:0/1175110420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b66cc127d0 con 0x55b66cecf580 2024-04-25T01:41:31.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.478+0000 7ff97ffff640 1 --2- 172.21.15.28:0/1175110420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 0x55b66cee6220 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7ff96c004690 tx=0x7ff96c004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83f66ba13e8a021 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.478+0000 7ff96a7fc640 1 -- 172.21.15.28:0/1175110420 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96c003e50 con 0x55b66cecf580 2024-04-25T01:41:31.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.478+0000 7ff96a7fc640 1 -- 172.21.15.28:0/1175110420 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff96c024070 con 0x55b66cecf580 2024-04-25T01:41:31.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.478+0000 7ff96a7fc640 1 -- 172.21.15.28:0/1175110420 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96c020cf0 con 0x55b66cecf580 2024-04-25T01:41:31.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.479+0000 7ff989fba740 1 -- 172.21.15.28:0/1175110420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 msgr2=0x55b66cee6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.479+0000 7ff989fba740 1 --2- 172.21.15.28:0/1175110420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 0x55b66cee6220 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7ff96c004690 tx=0x7ff96c004d90 comp rx=0 tx=0).stop 2024-04-25T01:41:31.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.479+0000 7ff989fba740 1 -- 172.21.15.28:0/1175110420 shutdown_connections 2024-04-25T01:41:31.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.479+0000 7ff989fba740 1 --2- 172.21.15.28:0/1175110420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b66ceda570 0x55b66cef20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.479+0000 7ff989fba740 1 --2- 172.21.15.28:0/1175110420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecfea0 0x55b66ced9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.479+0000 7ff989fba740 1 --2- 172.21.15.28:0/1175110420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecf580 0x55b66cee6220 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.479+0000 7ff989fba740 1 -- 172.21.15.28:0/1175110420 >> 172.21.15.28:0/1175110420 conn(0x55b66cecb8a0 msgr2=0x55b66ced2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.479+0000 7ff989fba740 1 -- 172.21.15.28:0/1175110420 shutdown_connections 2024-04-25T01:41:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.479+0000 7ff989fba740 1 -- 172.21.15.28:0/1175110420 wait complete. 2024-04-25T01:41:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.479+0000 7ff989fba740 1 -- start start 2024-04-25T01:41:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.479+0000 7ff989fba740 1 Processor -- start 2024-04-25T01:41:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.479+0000 7ff989fba740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecf580 0x55b66cee2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.479+0000 7ff989fba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecfea0 0x55b66ceddd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.479+0000 7ff989fba740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b66ceda570 0x55b66cede2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.479+0000 7ff989fba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff9801aff60 con 0x55b66cecfea0 2024-04-25T01:41:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.479+0000 7ff989fba740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff9801af890 con 0x55b66cecf580 2024-04-25T01:41:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.479+0000 7ff989fba740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff9801acaf0 con 0x55b66ceda570 2024-04-25T01:41:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.479+0000 7ff989fba740 10 client.0 _ll_register_callbacks cb 0x7ff97400d360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.480+0000 7ff984f6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecfea0 0x55b66ceddd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.480+0000 7ff984f6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecfea0 0x55b66ceddd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58490/0 (socket says 172.21.15.28:58490) 2024-04-25T01:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.480+0000 7ff984f6a640 1 -- 172.21.15.28:0/4215054275 learned_addr learned my addr 172.21.15.28:0/4215054275 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.480+0000 7ff97ffff640 1 --2- 172.21.15.28:0/4215054275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecf580 0x55b66cee2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.480+0000 7ff97ffff640 1 --2- 172.21.15.28:0/4215054275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b66ceda570 0x55b66cede2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.480+0000 7ff984f6a640 1 -- 172.21.15.28:0/4215054275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b66ceda570 msgr2=0x55b66cede2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.480+0000 7ff984f6a640 1 --2- 172.21.15.28:0/4215054275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b66ceda570 0x55b66cede2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.480+0000 7ff984f6a640 1 -- 172.21.15.28:0/4215054275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecf580 msgr2=0x55b66cee2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.480+0000 7ff984f6a640 1 --2- 172.21.15.28:0/4215054275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecf580 0x55b66cee2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.480+0000 7ff984f6a640 1 -- 172.21.15.28:0/4215054275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b66cedea10 con 0x55b66cecfea0 2024-04-25T01:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.480+0000 7ff97ffff640 1 --2- 172.21.15.28:0/4215054275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecf580 0x55b66cee2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.480+0000 7ff984f6a640 1 --2- 172.21.15.28:0/4215054275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecfea0 0x55b66ceddd90 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x55b66ced0b40 tx=0x7ff97400dfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a63a15ef46065bc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.480+0000 7ff95ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.480+0000 7ff95ffff640 1 -- 172.21.15.28:0/4215054275 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff974021070 con 0x55b66cecfea0 2024-04-25T01:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.480+0000 7ff95ffff640 1 -- 172.21.15.28:0/4215054275 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7ff97401d380 con 0x55b66cecfea0 2024-04-25T01:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.480+0000 7ff989fba740 1 -- 172.21.15.28:0/4215054275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff96c024070 con 0x55b66cecfea0 2024-04-25T01:41:31.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.480+0000 7ff95ffff640 1 -- 172.21.15.28:0/4215054275 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff97402d050 con 0x55b66cecfea0 2024-04-25T01:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.480+0000 7ff989fba740 10 client.18765 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.480+0000 7ff989fba740 1 -- 172.21.15.28:0/4215054275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b66cf508d0 con 0x55b66cecfea0 2024-04-25T01:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.481+0000 7ff95ffff640 1 -- 172.21.15.28:0/4215054275 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff96c024070 con 0x55b66cecfea0 2024-04-25T01:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.481+0000 7ff95ffff640 1 -- 172.21.15.28:0/4215054275 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff974010ca0 con 0x55b66cecfea0 2024-04-25T01:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.481+0000 7ff989fba740 10 client.18765 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.481+0000 7ff989fba740 10 client.18765 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.481+0000 7ff989fba740 20 client.18765 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.481+0000 7ff989fba740 1 --2- 172.21.15.28:0/4215054275 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b66cf51000 0x55b66cf53420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.481+0000 7ff989fba740 4 client.18765 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.481+0000 7ff989fba740 1 -- 172.21.15.28:0/4215054275 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b66ce3bf00 con 0x55b66cf51000 2024-04-25T01:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.481+0000 7ff97ffff640 1 --2- 172.21.15.28:0/4215054275 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b66cf51000 0x55b66cf53420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.481+0000 7ff97ffff640 1 --2- 172.21.15.28:0/4215054275 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b66cf51000 0x55b66cf53420 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=70875915c9b77c7 server_cookie=7b67ff3ae3acbee2 in_seq=0 out_seq=0 2024-04-25T01:41:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.481+0000 7ff95ffff640 10 client.18765 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:41:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.482+0000 7ff95ffff640 1 -- 172.21.15.28:0/4215054275 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b66ce3bf00 con 0x55b66cf51000 2024-04-25T01:41:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.482+0000 7ff95ffff640 10 client.18765 handle_command_reply: tid=0 2024-04-25T01:41:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.482+0000 7ff989fba740 1 client.18765 shutdown 2024-04-25T01:41:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.482+0000 7ff989fba740 1 -- 172.21.15.28:0/4215054275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecfea0 msgr2=0x55b66ceddd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.482+0000 7ff989fba740 1 --2- 172.21.15.28:0/4215054275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecfea0 0x55b66ceddd90 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x55b66ced0b40 tx=0x7ff97400dfd0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.482+0000 7ff989fba740 1 -- 172.21.15.28:0/4215054275 shutdown_connections 2024-04-25T01:41:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.482+0000 7ff989fba740 1 --2- 172.21.15.28:0/4215054275 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55b66cf51000 0x55b66cf53420 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.482+0000 7ff989fba740 1 --2- 172.21.15.28:0/4215054275 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b66ceda570 0x55b66cede2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.482+0000 7ff95ffff640 0 client.18765 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:41:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.482+0000 7ff989fba740 1 --2- 172.21.15.28:0/4215054275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66cecfea0 0x55b66ceddd90 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.482+0000 7ff989fba740 1 --2- 172.21.15.28:0/4215054275 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b66cecf580 0x55b66cee2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.482+0000 7ff989fba740 1 -- 172.21.15.28:0/4215054275 >> 172.21.15.28:0/4215054275 conn(0x55b66cecb8a0 msgr2=0x55b66ced0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.482+0000 7ff989fba740 1 -- 172.21.15.28:0/4215054275 shutdown_connections 2024-04-25T01:41:31.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.482+0000 7ff989fba740 1 -- 172.21.15.28:0/4215054275 wait complete. 2024-04-25T01:41:31.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.482+0000 7ff989fba740 20 client.18765 trim_cache size 0 max 16384 2024-04-25T01:41:31.476 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:41:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.482+0000 7ff9871ce640 1 -- 172.21.15.28:0/2187171816 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff9700a2d00 msgr2=0x7ff9700a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.482+0000 7ff9871ce640 1 --2- 172.21.15.28:0/2187171816 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff9700a2d00 0x7ff9700a51c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff9780ee470 tx=0x7ff9780dd040 comp rx=0 tx=0).stop 2024-04-25T01:41:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.482+0000 7ff9871ce640 1 -- 172.21.15.28:0/2187171816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff980107720 msgr2=0x7ff98019c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.482+0000 7ff9871ce640 1 --2- 172.21.15.28:0/2187171816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff980107720 0x7ff98019c930 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7ff978005cc0 tx=0x7ff978003970 comp rx=0 tx=0).stop 2024-04-25T01:41:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.483+0000 7ff9871ce640 1 -- 172.21.15.28:0/2187171816 shutdown_connections 2024-04-25T01:41:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.483+0000 7ff9871ce640 1 --2- 172.21.15.28:0/2187171816 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff9700a2d00 0x7ff9700a51c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.483+0000 7ff9871ce640 1 --2- 172.21.15.28:0/2187171816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff980107720 0x7ff98019c930 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.483+0000 7ff9871ce640 1 --2- 172.21.15.28:0/2187171816 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff9801020d0 0x7ff9801985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.483+0000 7ff9871ce640 1 --2- 172.21.15.28:0/2187171816 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff980101710 0x7ff980198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.483+0000 7ff9871ce640 1 -- 172.21.15.28:0/2187171816 >> 172.21.15.28:0/2187171816 conn(0x7ff9800fd180 msgr2=0x7ff9800fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.483+0000 7ff9871ce640 1 -- 172.21.15.28:0/2187171816 shutdown_connections 2024-04-25T01:41:31.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.483+0000 7ff9871ce640 1 -- 172.21.15.28:0/2187171816 wait complete. 2024-04-25T01:41:31.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:41:31.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.598+0000 7f814c2ba640 1 -- 172.21.15.28:0/967327334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f814406a780 msgr2=0x7f814410e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.598+0000 7f814c2ba640 1 --2- 172.21.15.28:0/967327334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f814406a780 0x7f814410e5f0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f813c009cd0 tx=0x7f813c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.598+0000 7f814c2ba640 1 -- 172.21.15.28:0/967327334 shutdown_connections 2024-04-25T01:41:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.598+0000 7f814c2ba640 1 --2- 172.21.15.28:0/967327334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f814406a780 0x7f814410e5f0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.598+0000 7f814c2ba640 1 --2- 172.21.15.28:0/967327334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8144069dc0 0x7f814406a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.598+0000 7f814c2ba640 1 --2- 172.21.15.28:0/967327334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8144107730 0x7f8144107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.598+0000 7f814c2ba640 1 -- 172.21.15.28:0/967327334 >> 172.21.15.28:0/967327334 conn(0x7f81440776a0 msgr2=0x7f8144077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.598+0000 7f814c2ba640 1 -- 172.21.15.28:0/967327334 shutdown_connections 2024-04-25T01:41:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.598+0000 7f814c2ba640 1 -- 172.21.15.28:0/967327334 wait complete. 2024-04-25T01:41:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f814c2ba640 1 Processor -- start 2024-04-25T01:41:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f814c2ba640 1 -- start start 2024-04-25T01:41:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f814c2ba640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8144069dc0 0x7f8144100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f814c2ba640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f814406a780 0x7f8144100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f814c2ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8144107730 0x7f81441010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f814c2ba640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8144111b40 con 0x7f8144107730 2024-04-25T01:41:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f814c2ba640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f81441119c0 con 0x7f8144069dc0 2024-04-25T01:41:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f814c2ba640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8144111cc0 con 0x7f814406a780 2024-04-25T01:41:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f814a857640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8144107730 0x7f81441010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f814a857640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8144107730 0x7f81441010f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58496/0 (socket says 172.21.15.28:58496) 2024-04-25T01:41:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f814a857640 1 -- 172.21.15.28:0/736424192 learned_addr learned my addr 172.21.15.28:0/736424192 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f8149855640 1 --2- 172.21.15.28:0/736424192 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f814406a780 0x7f8144100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f814a857640 1 -- 172.21.15.28:0/736424192 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f814406a780 msgr2=0x7f8144100bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f814a857640 1 --2- 172.21.15.28:0/736424192 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f814406a780 0x7f8144100bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f814a857640 1 -- 172.21.15.28:0/736424192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8144069dc0 msgr2=0x7f8144100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f814a056640 1 --2- 172.21.15.28:0/736424192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8144069dc0 0x7f8144100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f814a857640 1 --2- 172.21.15.28:0/736424192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8144069dc0 0x7f8144100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f814a857640 1 -- 172.21.15.28:0/736424192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81441a63e0 con 0x7f8144107730 2024-04-25T01:41:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f814a056640 1 --2- 172.21.15.28:0/736424192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8144069dc0 0x7f8144100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:41:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.599+0000 7f814a857640 1 --2- 172.21.15.28:0/736424192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8144107730 0x7f81441010f0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f813c006010 tx=0x7f813c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.600+0000 7f813b7fe640 1 -- 172.21.15.28:0/736424192 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813c004450 con 0x7f8144107730 2024-04-25T01:41:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.600+0000 7f813b7fe640 1 -- 172.21.15.28:0/736424192 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f813c003e10 con 0x7f8144107730 2024-04-25T01:41:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.600+0000 7f814c2ba640 1 -- 172.21.15.28:0/736424192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81441a6670 con 0x7f8144107730 2024-04-25T01:41:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.600+0000 7f813b7fe640 1 -- 172.21.15.28:0/736424192 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813c046480 con 0x7f8144107730 2024-04-25T01:41:31.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.600+0000 7f814c2ba640 1 -- 172.21.15.28:0/736424192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81441a6bc0 con 0x7f8144107730 2024-04-25T01:41:31.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.602+0000 7f813b7fe640 1 -- 172.21.15.28:0/736424192 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f813c0045f0 con 0x7f8144107730 2024-04-25T01:41:31.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.602+0000 7f814efc6740 1 Processor -- start 2024-04-25T01:41:31.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.602+0000 7f814efc6740 1 -- start start 2024-04-25T01:41:31.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f814efc6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efa2580 0x56400efa2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f814efc6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 0x56400efad030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f814efc6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efad570 0x56400efaf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f814efc6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56400ef07670 con 0x56400efad570 2024-04-25T01:41:31.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f814efc6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56400eeaf970 con 0x56400efa2ea0 2024-04-25T01:41:31.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f814efc6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56400ef03630 con 0x56400efa2580 2024-04-25T01:41:31.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f8149855640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efad570 0x56400efaf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f8149855640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efad570 0x56400efaf960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58506/0 (socket says 172.21.15.28:58506) 2024-04-25T01:41:31.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f8149855640 1 -- 172.21.15.28:0/350112551 learned_addr learned my addr 172.21.15.28:0/350112551 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f814a056640 1 --2- 172.21.15.28:0/350112551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 0x56400efad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f8149855640 1 --2- 172.21.15.28:0/350112551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efa2580 0x56400efa2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f813b7fe640 1 --2- 172.21.15.28:0/736424192 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f81300a2d00 0x7f81300a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f8149855640 1 -- 172.21.15.28:0/350112551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efa2580 msgr2=0x56400efa2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f8149855640 1 --2- 172.21.15.28:0/350112551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efa2580 0x56400efa2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f8149855640 1 -- 172.21.15.28:0/350112551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 msgr2=0x56400efad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f8149855640 1 --2- 172.21.15.28:0/350112551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 0x56400efad030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f814a857640 1 --2- 172.21.15.28:0/736424192 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f81300a2d00 0x7f81300a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f8149855640 1 -- 172.21.15.28:0/350112551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56400eed3400 con 0x56400efad570 2024-04-25T01:41:31.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f814a056640 1 --2- 172.21.15.28:0/350112551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 0x56400efad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:31.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f813b7fe640 1 -- 172.21.15.28:0/736424192 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f813c02e030 con 0x7f8144107730 2024-04-25T01:41:31.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.603+0000 7f8149855640 1 --2- 172.21.15.28:0/350112551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efad570 0x56400efaf960 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f8134011560 tx=0x7f813400a810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e193f2fdea1faf4c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.604+0000 7f8138bfe640 1 -- 172.21.15.28:0/350112551 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8134005220 con 0x56400efad570 2024-04-25T01:41:31.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.604+0000 7f8138bfe640 1 -- 172.21.15.28:0/350112551 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8134004c80 con 0x56400efad570 2024-04-25T01:41:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.604+0000 7f814a857640 1 --2- 172.21.15.28:0/736424192 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f81300a2d00 0x7f81300a51c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f813c0bd2b0 tx=0x7f813c08d180 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.604+0000 7f8138bfe640 1 -- 172.21.15.28:0/350112551 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813400fc40 con 0x56400efad570 2024-04-25T01:41:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.604+0000 7f814efc6740 1 -- 172.21.15.28:0/350112551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efad570 msgr2=0x56400efaf960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.604+0000 7f814efc6740 1 --2- 172.21.15.28:0/350112551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efad570 0x56400efaf960 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f8134011560 tx=0x7f813400a810 comp rx=0 tx=0).stop 2024-04-25T01:41:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.604+0000 7f814efc6740 1 -- 172.21.15.28:0/350112551 shutdown_connections 2024-04-25T01:41:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.604+0000 7f814efc6740 1 --2- 172.21.15.28:0/350112551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efad570 0x56400efaf960 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.604+0000 7f814efc6740 1 --2- 172.21.15.28:0/350112551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 0x56400efad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.604+0000 7f814efc6740 1 --2- 172.21.15.28:0/350112551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efa2580 0x56400efa2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.604+0000 7f814efc6740 1 -- 172.21.15.28:0/350112551 >> 172.21.15.28:0/350112551 conn(0x56400ef9e8a0 msgr2=0x56400ef9ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.604+0000 7f814efc6740 1 -- 172.21.15.28:0/350112551 shutdown_connections 2024-04-25T01:41:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.604+0000 7f814efc6740 1 -- 172.21.15.28:0/350112551 wait complete. 2024-04-25T01:41:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.605+0000 7f814efc6740 1 -- start start 2024-04-25T01:41:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.605+0000 7f814efc6740 1 Processor -- start 2024-04-25T01:41:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.605+0000 7f814efc6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efa2580 0x56400efacf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.605+0000 7f814efc6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 0x56400efaa230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.605+0000 7f814efc6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efad570 0x56400efaa840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.605+0000 7f814efc6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56400efb0360 con 0x56400efa2580 2024-04-25T01:41:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.605+0000 7f814efc6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56400eec3050 con 0x56400efa2ea0 2024-04-25T01:41:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.605+0000 7f814efc6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56400efb04e0 con 0x56400efad570 2024-04-25T01:41:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.605+0000 7f814efc6740 10 client.0 _ll_register_callbacks cb 0x56400ec1abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.605+0000 7f8149855640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efa2580 0x56400efacf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.605+0000 7f8149855640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efa2580 0x56400efacf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58518/0 (socket says 172.21.15.28:58518) 2024-04-25T01:41:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.605+0000 7f8149855640 1 -- 172.21.15.28:0/2844139148 learned_addr learned my addr 172.21.15.28:0/2844139148 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.605+0000 7f814a056640 1 --2- 172.21.15.28:0/2844139148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 0x56400efaa230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.605+0000 7f8149855640 1 --2- 172.21.15.28:0/2844139148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efad570 0x56400efaa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.605+0000 7f8149855640 1 -- 172.21.15.28:0/2844139148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efad570 msgr2=0x56400efaa840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.605+0000 7f8149855640 1 --2- 172.21.15.28:0/2844139148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efad570 0x56400efaa840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.605+0000 7f8149855640 1 -- 172.21.15.28:0/2844139148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 msgr2=0x56400efaa230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.605+0000 7f8149855640 1 --2- 172.21.15.28:0/2844139148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 0x56400efaa230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.605+0000 7f8149855640 1 -- 172.21.15.28:0/2844139148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56400ee94dc0 con 0x56400efa2580 2024-04-25T01:41:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.606+0000 7f8149855640 1 --2- 172.21.15.28:0/2844139148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efa2580 0x56400efacf40 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f8134011eb0 tx=0x7f8134005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ffcd9fd42a576e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.606+0000 7f812dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.606+0000 7f812dffb640 1 -- 172.21.15.28:0/2844139148 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813402e060 con 0x56400efa2580 2024-04-25T01:41:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.606+0000 7f812dffb640 1 -- 172.21.15.28:0/2844139148 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f813400f9c0 con 0x56400efa2580 2024-04-25T01:41:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.606+0000 7f814efc6740 1 -- 172.21.15.28:0/2844139148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8134004c80 con 0x56400efa2580 2024-04-25T01:41:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.606+0000 7f812dffb640 1 -- 172.21.15.28:0/2844139148 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8134035050 con 0x56400efa2580 2024-04-25T01:41:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.606+0000 7f814efc6740 10 client.18777 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.606+0000 7f814efc6740 1 -- 172.21.15.28:0/2844139148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56400ec1b3f0 con 0x56400efa2580 2024-04-25T01:41:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.606+0000 7f812dffb640 1 -- 172.21.15.28:0/2844139148 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8134004c80 con 0x56400efa2580 2024-04-25T01:41:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.606+0000 7f812dffb640 1 -- 172.21.15.28:0/2844139148 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f813401b070 con 0x56400efa2580 2024-04-25T01:41:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.606+0000 7f814efc6740 10 client.18777 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.606+0000 7f814efc6740 10 client.18777 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:41:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.606+0000 7f814efc6740 20 client.18777 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.606+0000 7f814efc6740 1 --2- 172.21.15.28:0/2844139148 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56400efbd560 0x56400efbf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.606+0000 7f814efc6740 4 client.18777 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:41:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.606+0000 7f814efc6740 1 -- 172.21.15.28:0/2844139148 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56400ef14bb0 con 0x56400efbd560 2024-04-25T01:41:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.607+0000 7f814a056640 1 --2- 172.21.15.28:0/2844139148 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56400efbd560 0x56400efbf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.607+0000 7f814a056640 1 --2- 172.21.15.28:0/2844139148 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56400efbd560 0x56400efbf950 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6f79e6b99a4f58b6 server_cookie=2e1fa050842e3b61 in_seq=0 out_seq=0 2024-04-25T01:41:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.607+0000 7f812dffb640 10 client.18777 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:41:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.608+0000 7f812dffb640 1 -- 172.21.15.28:0/2844139148 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56400ef14bb0 con 0x56400efbd560 2024-04-25T01:41:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.608+0000 7f812dffb640 10 client.18777 handle_command_reply: tid=0 2024-04-25T01:41:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.608+0000 7f814efc6740 1 client.18777 shutdown 2024-04-25T01:41:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.608+0000 7f814efc6740 1 -- 172.21.15.28:0/2844139148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efa2580 msgr2=0x56400efacf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.608+0000 7f814efc6740 1 --2- 172.21.15.28:0/2844139148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efa2580 0x56400efacf40 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f8134011eb0 tx=0x7f8134005850 comp rx=0 tx=0).stop 2024-04-25T01:41:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.608+0000 7f814efc6740 1 -- 172.21.15.28:0/2844139148 shutdown_connections 2024-04-25T01:41:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.608+0000 7f814efc6740 1 --2- 172.21.15.28:0/2844139148 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56400efbd560 0x56400efbf950 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.608+0000 7f814efc6740 1 --2- 172.21.15.28:0/2844139148 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efad570 0x56400efaa840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.608+0000 7f814efc6740 1 --2- 172.21.15.28:0/2844139148 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 0x56400efaa230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.608+0000 7f814efc6740 1 --2- 172.21.15.28:0/2844139148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efa2580 0x56400efacf40 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.608+0000 7f814efc6740 1 -- 172.21.15.28:0/2844139148 >> 172.21.15.28:0/2844139148 conn(0x56400ef9e8a0 msgr2=0x56400efa46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.608+0000 7f812dffb640 0 client.18777 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:41:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.608+0000 7f814efc6740 1 -- 172.21.15.28:0/2844139148 shutdown_connections 2024-04-25T01:41:31.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.608+0000 7f814efc6740 1 -- 172.21.15.28:0/2844139148 wait complete. 2024-04-25T01:41:31.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.608+0000 7f814efc6740 20 client.18777 trim_cache size 0 max 16384 2024-04-25T01:41:31.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.621+0000 7f814efc6740 1 Processor -- start 2024-04-25T01:41:31.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.621+0000 7f814efc6740 1 -- start start 2024-04-25T01:41:31.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.621+0000 7f814efc6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efa2580 0x56400efbf570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.621+0000 7f814efc6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 0x56400efa9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.621+0000 7f814efc6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efad570 0x56400efaca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.621+0000 7f814efc6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81441b0bf0 con 0x56400efad570 2024-04-25T01:41:31.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.621+0000 7f814efc6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f81441b2730 con 0x56400efa2ea0 2024-04-25T01:41:31.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.621+0000 7f814efc6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f81441b12c0 con 0x56400efa2580 2024-04-25T01:41:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.622+0000 7f8149855640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efad570 0x56400efaca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.622+0000 7f8149855640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efad570 0x56400efaca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58532/0 (socket says 172.21.15.28:58532) 2024-04-25T01:41:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.622+0000 7f8149855640 1 -- 172.21.15.28:0/2630202389 learned_addr learned my addr 172.21.15.28:0/2630202389 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.622+0000 7f8149855640 1 --2- 172.21.15.28:0/2630202389 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efa2580 0x56400efbf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.622+0000 7f814a056640 1 --2- 172.21.15.28:0/2630202389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 0x56400efa9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.622+0000 7f8149855640 1 -- 172.21.15.28:0/2630202389 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efa2580 msgr2=0x56400efbf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.622+0000 7f8149855640 1 --2- 172.21.15.28:0/2630202389 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efa2580 0x56400efbf570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.622+0000 7f8149855640 1 -- 172.21.15.28:0/2630202389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 msgr2=0x56400efa9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.622+0000 7f8149855640 1 --2- 172.21.15.28:0/2630202389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 0x56400efa9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.622+0000 7f8149855640 1 -- 172.21.15.28:0/2630202389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56400ece57d0 con 0x56400efad570 2024-04-25T01:41:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.622+0000 7f814a056640 1 --2- 172.21.15.28:0/2630202389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 0x56400efa9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.622+0000 7f8149855640 1 --2- 172.21.15.28:0/2630202389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efad570 0x56400efaca70 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f813400a480 tx=0x7f813400a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7545b3d4912d7976 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.622+0000 7f812ffff640 1 -- 172.21.15.28:0/2630202389 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813402e070 con 0x56400efad570 2024-04-25T01:41:31.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.622+0000 7f812ffff640 1 -- 172.21.15.28:0/2630202389 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8134005500 con 0x56400efad570 2024-04-25T01:41:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.622+0000 7f812ffff640 1 -- 172.21.15.28:0/2630202389 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8134028db0 con 0x56400efad570 2024-04-25T01:41:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f814efc6740 1 -- 172.21.15.28:0/2630202389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efad570 msgr2=0x56400efaca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f814efc6740 1 --2- 172.21.15.28:0/2630202389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efad570 0x56400efaca70 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f813400a480 tx=0x7f813400a630 comp rx=0 tx=0).stop 2024-04-25T01:41:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f814efc6740 1 -- 172.21.15.28:0/2630202389 shutdown_connections 2024-04-25T01:41:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f814efc6740 1 --2- 172.21.15.28:0/2630202389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efad570 0x56400efaca70 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f814efc6740 1 --2- 172.21.15.28:0/2630202389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 0x56400efa9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f814efc6740 1 --2- 172.21.15.28:0/2630202389 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efa2580 0x56400efbf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f814efc6740 1 -- 172.21.15.28:0/2630202389 >> 172.21.15.28:0/2630202389 conn(0x56400ef9e8a0 msgr2=0x56400efb8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f814efc6740 1 -- 172.21.15.28:0/2630202389 shutdown_connections 2024-04-25T01:41:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f814efc6740 1 -- 172.21.15.28:0/2630202389 wait complete. 2024-04-25T01:41:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f814efc6740 1 -- start start 2024-04-25T01:41:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f814efc6740 1 Processor -- start 2024-04-25T01:41:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f814efc6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efa2580 0x56400efb2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f814efc6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 0x56400efb2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f814efc6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efad570 0x56400f021900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f814efc6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81441b2060 con 0x56400efa2580 2024-04-25T01:41:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f814efc6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f81441b1990 con 0x56400efa2ea0 2024-04-25T01:41:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f814efc6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f81441aebf0 con 0x56400efad570 2024-04-25T01:41:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f814efc6740 10 client.0 _ll_register_callbacks cb 0x7f8134011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f8149855640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efa2580 0x56400efb2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f8149855640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efa2580 0x56400efb2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58544/0 (socket says 172.21.15.28:58544) 2024-04-25T01:41:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f8149855640 1 -- 172.21.15.28:0/48185017 learned_addr learned my addr 172.21.15.28:0/48185017 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.623+0000 7f814a056640 1 --2- 172.21.15.28:0/48185017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 0x56400efb2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f8149855640 1 --2- 172.21.15.28:0/48185017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efad570 0x56400f021900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f8149855640 1 -- 172.21.15.28:0/48185017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efad570 msgr2=0x56400f021900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f8149855640 1 --2- 172.21.15.28:0/48185017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efad570 0x56400f021900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f8149855640 1 -- 172.21.15.28:0/48185017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 msgr2=0x56400efb2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f8149855640 1 --2- 172.21.15.28:0/48185017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 0x56400efb2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f8149855640 1 -- 172.21.15.28:0/48185017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56400efb33b0 con 0x56400efa2580 2024-04-25T01:41:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f8149855640 1 --2- 172.21.15.28:0/48185017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efa2580 0x56400efb2570 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f8134027a70 tx=0x7f8134005350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce511cff254c42d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f812dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f812dffb640 1 -- 172.21.15.28:0/48185017 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813402e070 con 0x56400efa2580 2024-04-25T01:41:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f812dffb640 1 -- 172.21.15.28:0/48185017 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f8134027420 con 0x56400efa2580 2024-04-25T01:41:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f814efc6740 1 -- 172.21.15.28:0/48185017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8134005500 con 0x56400efa2580 2024-04-25T01:41:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f812dffb640 1 -- 172.21.15.28:0/48185017 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8134049050 con 0x56400efa2580 2024-04-25T01:41:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f814efc6740 10 client.18783 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f814efc6740 1 -- 172.21.15.28:0/48185017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56400f022f70 con 0x56400efa2580 2024-04-25T01:41:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f812dffb640 1 -- 172.21.15.28:0/48185017 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8134005500 con 0x56400efa2580 2024-04-25T01:41:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f812dffb640 1 -- 172.21.15.28:0/48185017 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f813401b050 con 0x56400efa2580 2024-04-25T01:41:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f814efc6740 10 client.18783 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f814efc6740 10 client.18783 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:41:31.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f814efc6740 20 client.18783 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f814efc6740 1 --2- 172.21.15.28:0/48185017 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56400f023520 0x56400f025910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f814efc6740 4 client.18783 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:41:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.624+0000 7f814efc6740 1 -- 172.21.15.28:0/48185017 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56400ef0ef00 con 0x56400f023520 2024-04-25T01:41:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.625+0000 7f814a056640 1 --2- 172.21.15.28:0/48185017 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56400f023520 0x56400f025910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.625+0000 7f812dffb640 10 client.18783 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:41:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.625+0000 7f814a056640 1 --2- 172.21.15.28:0/48185017 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56400f023520 0x56400f025910 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bde8ea53953f8846 server_cookie=f5d5e34283228706 in_seq=0 out_seq=0 2024-04-25T01:41:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.625+0000 7f812dffb640 1 -- 172.21.15.28:0/48185017 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56400ef0ef00 con 0x56400f023520 2024-04-25T01:41:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.625+0000 7f812dffb640 10 client.18783 handle_command_reply: tid=0 2024-04-25T01:41:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.625+0000 7f814efc6740 1 client.18783 shutdown 2024-04-25T01:41:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.625+0000 7f814efc6740 1 -- 172.21.15.28:0/48185017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efa2580 msgr2=0x56400efb2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.625+0000 7f814efc6740 1 --2- 172.21.15.28:0/48185017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efa2580 0x56400efb2570 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f8134027a70 tx=0x7f8134005350 comp rx=0 tx=0).stop 2024-04-25T01:41:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.625+0000 7f814efc6740 1 -- 172.21.15.28:0/48185017 shutdown_connections 2024-04-25T01:41:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.625+0000 7f814efc6740 1 --2- 172.21.15.28:0/48185017 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x56400f023520 0x56400f025910 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.625+0000 7f814efc6740 1 --2- 172.21.15.28:0/48185017 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56400efad570 0x56400f021900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.625+0000 7f814efc6740 1 --2- 172.21.15.28:0/48185017 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56400efa2ea0 0x56400efb2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.625+0000 7f814efc6740 1 --2- 172.21.15.28:0/48185017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400efa2580 0x56400efb2570 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.625+0000 7f812dffb640 0 client.18783 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:41:31.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.625+0000 7f814efc6740 1 -- 172.21.15.28:0/48185017 >> 172.21.15.28:0/48185017 conn(0x56400ef9e8a0 msgr2=0x56400efa9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.626+0000 7f814efc6740 1 -- 172.21.15.28:0/48185017 shutdown_connections 2024-04-25T01:41:31.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.626+0000 7f814efc6740 1 -- 172.21.15.28:0/48185017 wait complete. 2024-04-25T01:41:31.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.626+0000 7f814efc6740 20 client.18783 trim_cache size 0 max 16384 2024-04-25T01:41:31.619 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:41:31.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.626+0000 7f814c2ba640 1 -- 172.21.15.28:0/736424192 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f81300a2d00 msgr2=0x7f81300a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.626+0000 7f814c2ba640 1 --2- 172.21.15.28:0/736424192 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f81300a2d00 0x7f81300a51c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f813c0bd2b0 tx=0x7f813c08d180 comp rx=0 tx=0).stop 2024-04-25T01:41:31.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.626+0000 7f814c2ba640 1 -- 172.21.15.28:0/736424192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8144107730 msgr2=0x7f81441010f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.626+0000 7f814c2ba640 1 --2- 172.21.15.28:0/736424192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8144107730 0x7f81441010f0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f813c006010 tx=0x7f813c003c20 comp rx=0 tx=0).stop 2024-04-25T01:41:31.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.626+0000 7f814c2ba640 1 -- 172.21.15.28:0/736424192 shutdown_connections 2024-04-25T01:41:31.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.626+0000 7f814c2ba640 1 --2- 172.21.15.28:0/736424192 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f81300a2d00 0x7f81300a51c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.626+0000 7f814c2ba640 1 --2- 172.21.15.28:0/736424192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8144107730 0x7f81441010f0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.626+0000 7f814c2ba640 1 --2- 172.21.15.28:0/736424192 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f814406a780 0x7f8144100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.626+0000 7f814c2ba640 1 --2- 172.21.15.28:0/736424192 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8144069dc0 0x7f8144100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.626+0000 7f814c2ba640 1 -- 172.21.15.28:0/736424192 >> 172.21.15.28:0/736424192 conn(0x7f81440776a0 msgr2=0x7f81440fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.626+0000 7f814c2ba640 1 -- 172.21.15.28:0/736424192 shutdown_connections 2024-04-25T01:41:31.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.626+0000 7f814c2ba640 1 -- 172.21.15.28:0/736424192 wait complete. 2024-04-25T01:41:31.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:41:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.741+0000 7f10afc38640 1 -- 172.21.15.28:0/1576791245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10a81029c0 msgr2=0x7f10a8102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.741+0000 7f10afc38640 1 --2- 172.21.15.28:0/1576791245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10a81029c0 0x7f10a8102da0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f1098009900 tx=0x7f109802a510 comp rx=0 tx=0).stop 2024-04-25T01:41:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.741+0000 7f10afc38640 1 -- 172.21.15.28:0/1576791245 shutdown_connections 2024-04-25T01:41:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.741+0000 7f10afc38640 1 --2- 172.21.15.28:0/1576791245 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10a810bd00 0x7f10a810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.741+0000 7f10afc38640 1 --2- 172.21.15.28:0/1576791245 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10a81032e0 0x7f10a810b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.741+0000 7f10afc38640 1 --2- 172.21.15.28:0/1576791245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10a81029c0 0x7f10a8102da0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.741+0000 7f10afc38640 1 -- 172.21.15.28:0/1576791245 >> 172.21.15.28:0/1576791245 conn(0x7f10a8077740 msgr2=0x7f10a8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.742+0000 7f10afc38640 1 -- 172.21.15.28:0/1576791245 shutdown_connections 2024-04-25T01:41:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.742+0000 7f10afc38640 1 -- 172.21.15.28:0/1576791245 wait complete. 2024-04-25T01:41:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.742+0000 7f10afc38640 1 Processor -- start 2024-04-25T01:41:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.742+0000 7f10afc38640 1 -- start start 2024-04-25T01:41:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.742+0000 7f10afc38640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10a81029c0 0x7f10a8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.742+0000 7f10afc38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10a81032e0 0x7f10a8198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.742+0000 7f10afc38640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10a810bd00 0x7f10a819c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.742+0000 7f10afc38640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f10a8111b50 con 0x7f10a81032e0 2024-04-25T01:41:31.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.742+0000 7f10afc38640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f10a81119d0 con 0x7f10a810bd00 2024-04-25T01:41:31.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.742+0000 7f10afc38640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f10a8111cd0 con 0x7f10a81029c0 2024-04-25T01:41:31.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.742+0000 7f10ad1d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10a81032e0 0x7f10a8198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.742+0000 7f10ad1d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10a81032e0 0x7f10a8198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58562/0 (socket says 172.21.15.28:58562) 2024-04-25T01:41:31.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.742+0000 7f10ad1d3640 1 -- 172.21.15.28:0/2395740416 learned_addr learned my addr 172.21.15.28:0/2395740416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.743+0000 7f10ad9d4640 1 --2- 172.21.15.28:0/2395740416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10a81029c0 0x7f10a8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.743+0000 7f10ae1d5640 1 --2- 172.21.15.28:0/2395740416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10a810bd00 0x7f10a819c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.743+0000 7f10ad1d3640 1 -- 172.21.15.28:0/2395740416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10a81029c0 msgr2=0x7f10a8197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.743+0000 7f10ad1d3640 1 --2- 172.21.15.28:0/2395740416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10a81029c0 0x7f10a8197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.743+0000 7f10ad1d3640 1 -- 172.21.15.28:0/2395740416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10a810bd00 msgr2=0x7f10a819c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.743+0000 7f10ad1d3640 1 --2- 172.21.15.28:0/2395740416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10a810bd00 0x7f10a819c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.743+0000 7f10ad1d3640 1 -- 172.21.15.28:0/2395740416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10a819cfc0 con 0x7f10a81032e0 2024-04-25T01:41:31.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.743+0000 7f10ae1d5640 1 --2- 172.21.15.28:0/2395740416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10a810bd00 0x7f10a819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:41:31.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.743+0000 7f10ad1d3640 1 --2- 172.21.15.28:0/2395740416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10a81032e0 0x7f10a8198510 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f109c00d950 tx=0x7f109c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.743+0000 7f10a6ffd640 1 -- 172.21.15.28:0/2395740416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109c017070 con 0x7f10a81032e0 2024-04-25T01:41:31.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.743+0000 7f10a6ffd640 1 -- 172.21.15.28:0/2395740416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f109c009c30 con 0x7f10a81032e0 2024-04-25T01:41:31.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.743+0000 7f10a6ffd640 1 -- 172.21.15.28:0/2395740416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109c00f6b0 con 0x7f10a81032e0 2024-04-25T01:41:31.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.743+0000 7f10afc38640 1 -- 172.21.15.28:0/2395740416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10a819d2b0 con 0x7f10a81032e0 2024-04-25T01:41:31.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.743+0000 7f10afc38640 1 -- 172.21.15.28:0/2395740416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10a81a4b40 con 0x7f10a81032e0 2024-04-25T01:41:31.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.745+0000 7f10a6ffd640 1 -- 172.21.15.28:0/2395740416 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f109c0109d0 con 0x7f10a81032e0 2024-04-25T01:41:31.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.746+0000 7f10b2a26740 1 Processor -- start 2024-04-25T01:41:31.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.746+0000 7f10b2a26740 1 -- start start 2024-04-25T01:41:31.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.746+0000 7f10b2a26740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e4656d580 0x564e4656d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.746+0000 7f10b2a26740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e4656dea0 0x564e46578030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.746+0000 7f10b2a26740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e46578570 0x564e4657a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.746+0000 7f10b2a26740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564e464d2670 con 0x564e46578570 2024-04-25T01:41:31.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.746+0000 7f10b2a26740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564e4647a970 con 0x564e4656dea0 2024-04-25T01:41:31.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.746+0000 7f10b2a26740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x564e464ce630 con 0x564e4656d580 2024-04-25T01:41:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.746+0000 7f10a6ffd640 1 --2- 172.21.15.28:0/2395740416 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f10900a2d80 0x7f10900a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.746+0000 7f10ad1d3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e46578570 0x564e4657a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.746+0000 7f10ad9d4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e4656dea0 0x564e46578030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.747+0000 7f10ae1d5640 1 --2- 172.21.15.28:0/2395740416 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f10900a2d80 0x7f10900a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.747+0000 7f10ad9d4640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e4656dea0 0x564e46578030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:46710/0 (socket says 172.21.15.28:46710) 2024-04-25T01:41:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.747+0000 7f10ad9d4640 1 -- 172.21.15.28:0/3519995318 learned_addr learned my addr 172.21.15.28:0/3519995318 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.747+0000 7f10ae1d5640 1 --2- 172.21.15.28:0/3519995318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e4656d580 0x564e4656d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.747+0000 7f10a6ffd640 1 -- 172.21.15.28:0/2395740416 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f109c0cf3d0 con 0x7f10a81032e0 2024-04-25T01:41:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.747+0000 7f10ad9d4640 1 -- 172.21.15.28:0/3519995318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e4656d580 msgr2=0x564e4656d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.747+0000 7f10ad9d4640 1 --2- 172.21.15.28:0/3519995318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e4656d580 0x564e4656d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.747+0000 7f10ad9d4640 1 -- 172.21.15.28:0/3519995318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e46578570 msgr2=0x564e4657a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.747+0000 7f10ad9d4640 1 --2- 172.21.15.28:0/3519995318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e46578570 0x564e4657a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.747+0000 7f10ad9d4640 1 -- 172.21.15.28:0/3519995318 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e4649e400 con 0x564e4656dea0 2024-04-25T01:41:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.747+0000 7f10ae1d5640 1 --2- 172.21.15.28:0/2395740416 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f10900a2d80 0x7f10900a5240 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f10a0005c40 tx=0x7f10a0009cd0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.747+0000 7f10ad9d4640 1 --2- 172.21.15.28:0/3519995318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e4656dea0 0x564e46578030 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1098009900 tx=0x7f10980040f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac34ed525cc08f2e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.747+0000 7f108ffff640 1 -- 172.21.15.28:0/3519995318 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1098035040 con 0x564e4656dea0 2024-04-25T01:41:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.747+0000 7f108ffff640 1 -- 172.21.15.28:0/3519995318 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1098005520 con 0x564e4656dea0 2024-04-25T01:41:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.747+0000 7f108ffff640 1 -- 172.21.15.28:0/3519995318 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10980035f0 con 0x564e4656dea0 2024-04-25T01:41:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.748+0000 7f10b2a26740 1 -- 172.21.15.28:0/3519995318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e4656dea0 msgr2=0x564e46578030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.748+0000 7f10b2a26740 1 --2- 172.21.15.28:0/3519995318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e4656dea0 0x564e46578030 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1098009900 tx=0x7f10980040f0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.748+0000 7f10b2a26740 1 -- 172.21.15.28:0/3519995318 shutdown_connections 2024-04-25T01:41:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.748+0000 7f10b2a26740 1 --2- 172.21.15.28:0/3519995318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e46578570 0x564e4657a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.748+0000 7f10b2a26740 1 --2- 172.21.15.28:0/3519995318 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e4656dea0 0x564e46578030 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.748+0000 7f10b2a26740 1 --2- 172.21.15.28:0/3519995318 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e4656d580 0x564e4656d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.748+0000 7f10b2a26740 1 -- 172.21.15.28:0/3519995318 >> 172.21.15.28:0/3519995318 conn(0x564e465698a0 msgr2=0x564e46569c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.748+0000 7f10b2a26740 1 -- 172.21.15.28:0/3519995318 shutdown_connections 2024-04-25T01:41:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.748+0000 7f10b2a26740 1 -- 172.21.15.28:0/3519995318 wait complete. 2024-04-25T01:41:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.748+0000 7f10b2a26740 1 -- start start 2024-04-25T01:41:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.749+0000 7f10b2a26740 1 Processor -- start 2024-04-25T01:41:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.749+0000 7f10b2a26740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 0x564e46577e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.749+0000 7f10b2a26740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e4656dea0 0x564e46575120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.749+0000 7f10b2a26740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e46578570 0x564e46575660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.749+0000 7f10b2a26740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564e4657b300 con 0x564e4656d580 2024-04-25T01:41:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.749+0000 7f10b2a26740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564e4648e050 con 0x564e46578570 2024-04-25T01:41:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.749+0000 7f10b2a26740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x564e4657b480 con 0x564e4656dea0 2024-04-25T01:41:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.749+0000 7f10b2a26740 10 client.0 _ll_register_callbacks cb 0x564e461e5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.749+0000 7f10ad9d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 0x564e46577e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.749+0000 7f10ad9d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 0x564e46577e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58584/0 (socket says 172.21.15.28:58584) 2024-04-25T01:41:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.749+0000 7f10ad9d4640 1 -- 172.21.15.28:0/1822712202 learned_addr learned my addr 172.21.15.28:0/1822712202 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.749+0000 7f10ad1d3640 1 --2- 172.21.15.28:0/1822712202 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e4656dea0 0x564e46575120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.749+0000 7f10ae1d5640 1 --2- 172.21.15.28:0/1822712202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e46578570 0x564e46575660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.749+0000 7f10ad9d4640 1 -- 172.21.15.28:0/1822712202 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e4656dea0 msgr2=0x564e46575120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.749+0000 7f10ad9d4640 1 --2- 172.21.15.28:0/1822712202 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e4656dea0 0x564e46575120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.749+0000 7f10ad9d4640 1 -- 172.21.15.28:0/1822712202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e46578570 msgr2=0x564e46575660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.749+0000 7f10ad9d4640 1 --2- 172.21.15.28:0/1822712202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e46578570 0x564e46575660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.749+0000 7f10ad9d4640 1 -- 172.21.15.28:0/1822712202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e4645fdc0 con 0x564e4656d580 2024-04-25T01:41:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.750+0000 7f10ad9d4640 1 --2- 172.21.15.28:0/1822712202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 0x564e46577e30 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f109802aa30 tx=0x7f1098004bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=730e151176ddd71 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.750+0000 7f108d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:31.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.750+0000 7f108d7fa640 1 -- 172.21.15.28:0/1822712202 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1098035050 con 0x564e4656d580 2024-04-25T01:41:31.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.750+0000 7f108d7fa640 1 -- 172.21.15.28:0/1822712202 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f10980048d0 con 0x564e4656d580 2024-04-25T01:41:31.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.750+0000 7f10b2a26740 1 -- 172.21.15.28:0/1822712202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1098005520 con 0x564e4656d580 2024-04-25T01:41:31.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.750+0000 7f108d7fa640 1 -- 172.21.15.28:0/1822712202 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1098059050 con 0x564e4656d580 2024-04-25T01:41:31.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.750+0000 7f10b2a26740 10 client.18795 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:31.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.750+0000 7f10b2a26740 1 -- 172.21.15.28:0/1822712202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564e461e63f0 con 0x564e4656d580 2024-04-25T01:41:31.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.750+0000 7f108d7fa640 1 -- 172.21.15.28:0/1822712202 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1098005520 con 0x564e4656d580 2024-04-25T01:41:31.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.750+0000 7f108d7fa640 1 -- 172.21.15.28:0/1822712202 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1098003b60 con 0x564e4656d580 2024-04-25T01:41:31.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.750+0000 7f10b2a26740 10 client.18795 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:31.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.750+0000 7f10b2a26740 10 client.18795 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:41:31.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.750+0000 7f10b2a26740 20 client.18795 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:31.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.750+0000 7f10b2a26740 1 --2- 172.21.15.28:0/1822712202 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564e46588350 0x564e4658a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.750+0000 7f10b2a26740 4 client.18795 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:41:31.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.750+0000 7f10b2a26740 1 -- 172.21.15.28:0/1822712202 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e464dfbb0 con 0x564e46588350 2024-04-25T01:41:31.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.751+0000 7f10ad1d3640 1 --2- 172.21.15.28:0/1822712202 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564e46588350 0x564e4658a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.751+0000 7f108d7fa640 10 client.18795 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:31.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.751+0000 7f10ad1d3640 1 --2- 172.21.15.28:0/1822712202 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564e46588350 0x564e4658a740 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dc664c180ea8167 server_cookie=3226aab048865f43 in_seq=0 out_seq=0 2024-04-25T01:41:31.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.752+0000 7f108d7fa640 1 -- 172.21.15.28:0/1822712202 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564e464dfbb0 con 0x564e46588350 2024-04-25T01:41:31.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.752+0000 7f108d7fa640 10 client.18795 handle_command_reply: tid=0 2024-04-25T01:41:31.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.752+0000 7f10b2a26740 1 client.18795 shutdown 2024-04-25T01:41:31.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.752+0000 7f10b2a26740 1 -- 172.21.15.28:0/1822712202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 msgr2=0x564e46577e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.752+0000 7f10b2a26740 1 --2- 172.21.15.28:0/1822712202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 0x564e46577e30 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f109802aa30 tx=0x7f1098004bd0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.752+0000 7f10b2a26740 1 -- 172.21.15.28:0/1822712202 shutdown_connections 2024-04-25T01:41:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.752+0000 7f10b2a26740 1 --2- 172.21.15.28:0/1822712202 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564e46588350 0x564e4658a740 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.752+0000 7f10b2a26740 1 --2- 172.21.15.28:0/1822712202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e46578570 0x564e46575660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.752+0000 7f10b2a26740 1 --2- 172.21.15.28:0/1822712202 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e4656dea0 0x564e46575120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.752+0000 7f10b2a26740 1 --2- 172.21.15.28:0/1822712202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 0x564e46577e30 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.752+0000 7f108d7fa640 0 client.18795 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.752+0000 7f10b2a26740 1 -- 172.21.15.28:0/1822712202 >> 172.21.15.28:0/1822712202 conn(0x564e465698a0 msgr2=0x564e46568b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.752+0000 7f10b2a26740 1 -- 172.21.15.28:0/1822712202 shutdown_connections 2024-04-25T01:41:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.752+0000 7f10b2a26740 1 -- 172.21.15.28:0/1822712202 wait complete. 2024-04-25T01:41:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.752+0000 7f10b2a26740 20 client.18795 trim_cache size 0 max 16384 2024-04-25T01:41:31.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f10b2a26740 1 Processor -- start 2024-04-25T01:41:31.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f10b2a26740 1 -- start start 2024-04-25T01:41:31.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f10b2a26740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 0x564e465841c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f10b2a26740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e4656dea0 0x564e465689c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f10b2a26740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e46578570 0x564e46577960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f10b2a26740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f10a81aeb10 con 0x564e4656d580 2024-04-25T01:41:31.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f10b2a26740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f10a81b0650 con 0x564e4656dea0 2024-04-25T01:41:31.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f10b2a26740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f10a81af1e0 con 0x564e46578570 2024-04-25T01:41:31.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f10ad9d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 0x564e465841c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f10ad9d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 0x564e465841c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58598/0 (socket says 172.21.15.28:58598) 2024-04-25T01:41:31.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f10ad9d4640 1 -- 172.21.15.28:0/846564947 learned_addr learned my addr 172.21.15.28:0/846564947 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f10ad1d3640 1 --2- 172.21.15.28:0/846564947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e4656dea0 0x564e465689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f10ad9d4640 1 -- 172.21.15.28:0/846564947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e46578570 msgr2=0x564e46577960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f10ad9d4640 1 --2- 172.21.15.28:0/846564947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e46578570 0x564e46577960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f10ad9d4640 1 -- 172.21.15.28:0/846564947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e4656dea0 msgr2=0x564e465689c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f10ad9d4640 1 --2- 172.21.15.28:0/846564947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e4656dea0 0x564e465689c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f10ad9d4640 1 -- 172.21.15.28:0/846564947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e4656dcd0 con 0x564e4656d580 2024-04-25T01:41:31.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f10ad9d4640 1 --2- 172.21.15.28:0/846564947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 0x564e465841c0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f1098031550 tx=0x7f1098031990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b18b8285b620ce65 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f108f7fe640 1 -- 172.21.15.28:0/846564947 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1098035070 con 0x564e4656d580 2024-04-25T01:41:31.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f108f7fe640 1 -- 172.21.15.28:0/846564947 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1098005850 con 0x564e4656d580 2024-04-25T01:41:31.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.766+0000 7f108f7fe640 1 -- 172.21.15.28:0/846564947 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1098003bb0 con 0x564e4656d580 2024-04-25T01:41:31.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.767+0000 7f10b2a26740 1 -- 172.21.15.28:0/846564947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 msgr2=0x564e465841c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.767+0000 7f10b2a26740 1 --2- 172.21.15.28:0/846564947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 0x564e465841c0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f1098031550 tx=0x7f1098031990 comp rx=0 tx=0).stop 2024-04-25T01:41:31.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.767+0000 7f10b2a26740 1 -- 172.21.15.28:0/846564947 shutdown_connections 2024-04-25T01:41:31.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.767+0000 7f10b2a26740 1 --2- 172.21.15.28:0/846564947 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e46578570 0x564e46577960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.767+0000 7f10b2a26740 1 --2- 172.21.15.28:0/846564947 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e4656dea0 0x564e465689c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.767+0000 7f10b2a26740 1 --2- 172.21.15.28:0/846564947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 0x564e465841c0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.767+0000 7f10b2a26740 1 -- 172.21.15.28:0/846564947 >> 172.21.15.28:0/846564947 conn(0x564e465698a0 msgr2=0x564e465682b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.767+0000 7f10b2a26740 1 -- 172.21.15.28:0/846564947 shutdown_connections 2024-04-25T01:41:31.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.767+0000 7f10b2a26740 1 -- 172.21.15.28:0/846564947 wait complete. 2024-04-25T01:41:31.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.767+0000 7f10b2a26740 1 -- start start 2024-04-25T01:41:31.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.767+0000 7f10b2a26740 1 Processor -- start 2024-04-25T01:41:31.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.768+0000 7f10b2a26740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 0x564e465eb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.768+0000 7f10b2a26740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e4656dea0 0x564e465eb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.768+0000 7f10b2a26740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e46578570 0x564e465e54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.768+0000 7f10b2a26740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f10a81aff80 con 0x564e4656d580 2024-04-25T01:41:31.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.768+0000 7f10b2a26740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f10a81af8b0 con 0x564e4656dea0 2024-04-25T01:41:31.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.768+0000 7f10b2a26740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f10a81acb10 con 0x564e46578570 2024-04-25T01:41:31.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.768+0000 7f10b2a26740 10 client.0 _ll_register_callbacks cb 0x7f109c097160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:31.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.768+0000 7f10ad9d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 0x564e465eb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.768+0000 7f10ad9d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 0x564e465eb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58610/0 (socket says 172.21.15.28:58610) 2024-04-25T01:41:31.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.768+0000 7f10ad9d4640 1 -- 172.21.15.28:0/2154622526 learned_addr learned my addr 172.21.15.28:0/2154622526 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:31.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.768+0000 7f10ae1d5640 1 --2- 172.21.15.28:0/2154622526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e46578570 0x564e465e54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.768+0000 7f10ad1d3640 1 --2- 172.21.15.28:0/2154622526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e4656dea0 0x564e465eb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.768+0000 7f10ad9d4640 1 -- 172.21.15.28:0/2154622526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e46578570 msgr2=0x564e465e54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.768+0000 7f10ad9d4640 1 --2- 172.21.15.28:0/2154622526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e46578570 0x564e465e54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.768+0000 7f10ad9d4640 1 -- 172.21.15.28:0/2154622526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e4656dea0 msgr2=0x564e465eb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.768+0000 7f10ad9d4640 1 --2- 172.21.15.28:0/2154622526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e4656dea0 0x564e465eb5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.768+0000 7f10ad9d4640 1 -- 172.21.15.28:0/2154622526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e462b07d0 con 0x564e4656d580 2024-04-25T01:41:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.768+0000 7f10ad9d4640 1 --2- 172.21.15.28:0/2154622526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 0x564e465eb090 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f1098003fb0 tx=0x7f10980040f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24018fb734fed83 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.768+0000 7f108d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.769+0000 7f108d7fa640 1 -- 172.21.15.28:0/2154622526 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1098035040 con 0x564e4656d580 2024-04-25T01:41:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.769+0000 7f108d7fa640 1 -- 172.21.15.28:0/2154622526 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f1098031d30 con 0x564e4656d580 2024-04-25T01:41:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.769+0000 7f10b2a26740 1 -- 172.21.15.28:0/2154622526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1098005850 con 0x564e4656d580 2024-04-25T01:41:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.769+0000 7f108d7fa640 1 -- 172.21.15.28:0/2154622526 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f109806a050 con 0x564e4656d580 2024-04-25T01:41:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.769+0000 7f10b2a26740 10 client.18801 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.769+0000 7f10b2a26740 1 -- 172.21.15.28:0/2154622526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564e465f26d0 con 0x564e4656d580 2024-04-25T01:41:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.769+0000 7f108d7fa640 1 -- 172.21.15.28:0/2154622526 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1098005850 con 0x564e4656d580 2024-04-25T01:41:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.769+0000 7f108d7fa640 1 -- 172.21.15.28:0/2154622526 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1098065030 con 0x564e4656d580 2024-04-25T01:41:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.769+0000 7f10b2a26740 10 client.18801 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.769+0000 7f10b2a26740 10 client.18801 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:41:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.769+0000 7f10b2a26740 20 client.18801 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.769+0000 7f10b2a26740 1 --2- 172.21.15.28:0/2154622526 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564e465f2d30 0x564e465f5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.769+0000 7f10b2a26740 4 client.18801 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:41:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.769+0000 7f10b2a26740 1 -- 172.21.15.28:0/2154622526 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564e465759f0 con 0x564e465f2d30 2024-04-25T01:41:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.769+0000 7f10ad1d3640 1 --2- 172.21.15.28:0/2154622526 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564e465f2d30 0x564e465f5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.770+0000 7f108d7fa640 10 client.18801 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.770+0000 7f10ad1d3640 1 --2- 172.21.15.28:0/2154622526 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564e465f2d30 0x564e465f5150 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd303a82fde555e server_cookie=383d697a2ebae9e6 in_seq=0 out_seq=0 2024-04-25T01:41:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.770+0000 7f108d7fa640 1 -- 172.21.15.28:0/2154622526 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564e465759f0 con 0x564e465f2d30 2024-04-25T01:41:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.770+0000 7f108d7fa640 10 client.18801 handle_command_reply: tid=0 2024-04-25T01:41:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.770+0000 7f10b2a26740 1 client.18801 shutdown 2024-04-25T01:41:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.770+0000 7f10b2a26740 1 -- 172.21.15.28:0/2154622526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 msgr2=0x564e465eb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.770+0000 7f10b2a26740 1 --2- 172.21.15.28:0/2154622526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 0x564e465eb090 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f1098003fb0 tx=0x7f10980040f0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.770+0000 7f10b2a26740 1 -- 172.21.15.28:0/2154622526 shutdown_connections 2024-04-25T01:41:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.770+0000 7f10b2a26740 1 --2- 172.21.15.28:0/2154622526 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x564e465f2d30 0x564e465f5150 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.770+0000 7f10b2a26740 1 --2- 172.21.15.28:0/2154622526 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e46578570 0x564e465e54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.770+0000 7f10b2a26740 1 --2- 172.21.15.28:0/2154622526 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e4656dea0 0x564e465eb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.770+0000 7f108d7fa640 0 client.18801 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.770+0000 7f10b2a26740 1 --2- 172.21.15.28:0/2154622526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e4656d580 0x564e465eb090 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.770+0000 7f10b2a26740 1 -- 172.21.15.28:0/2154622526 >> 172.21.15.28:0/2154622526 conn(0x564e465698a0 msgr2=0x564e46568b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.770+0000 7f10b2a26740 1 -- 172.21.15.28:0/2154622526 shutdown_connections 2024-04-25T01:41:31.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.770+0000 7f10b2a26740 1 -- 172.21.15.28:0/2154622526 wait complete. 2024-04-25T01:41:31.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.770+0000 7f10b2a26740 20 client.18801 trim_cache size 0 max 16384 2024-04-25T01:41:31.795 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:41:31.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.771+0000 7f10afc38640 1 -- 172.21.15.28:0/2395740416 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f10900a2d80 msgr2=0x7f10900a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.771+0000 7f10afc38640 1 --2- 172.21.15.28:0/2395740416 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f10900a2d80 0x7f10900a5240 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f10a0005c40 tx=0x7f10a0009cd0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.771+0000 7f10afc38640 1 -- 172.21.15.28:0/2395740416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10a81032e0 msgr2=0x7f10a8198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:31.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.771+0000 7f10afc38640 1 --2- 172.21.15.28:0/2395740416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10a81032e0 0x7f10a8198510 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f109c00d950 tx=0x7f109c00de10 comp rx=0 tx=0).stop 2024-04-25T01:41:31.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.771+0000 7f10afc38640 1 -- 172.21.15.28:0/2395740416 shutdown_connections 2024-04-25T01:41:31.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.771+0000 7f10afc38640 1 --2- 172.21.15.28:0/2395740416 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f10900a2d80 0x7f10900a5240 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.771+0000 7f10afc38640 1 --2- 172.21.15.28:0/2395740416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10a810bd00 0x7f10a819c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.771+0000 7f10afc38640 1 --2- 172.21.15.28:0/2395740416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10a81032e0 0x7f10a8198510 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.771+0000 7f10afc38640 1 --2- 172.21.15.28:0/2395740416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f10a81029c0 0x7f10a8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:31.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.771+0000 7f10afc38640 1 -- 172.21.15.28:0/2395740416 >> 172.21.15.28:0/2395740416 conn(0x7f10a8077740 msgr2=0x7f10a80fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:31.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.771+0000 7f10afc38640 1 -- 172.21.15.28:0/2395740416 shutdown_connections 2024-04-25T01:41:31.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:31.771+0000 7f10afc38640 1 -- 172.21.15.28:0/2395740416 wait complete. 2024-04-25T01:41:32.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:31 smithi130 ceph-mon[32702]: pgmap v262: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:41:32.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:31 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/727066701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:41:32.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:31 smithi032 ceph-mon[32461]: pgmap v262: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:41:32.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:31 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/727066701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:41:32.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:31 smithi028 ceph-mon[29390]: pgmap v262: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:41:32.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:31 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/727066701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:41:32.798 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:41:32.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:41:32.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:41:32.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.931+0000 7f9cf1822640 1 -- 172.21.15.28:0/3089651542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec100610 msgr2=0x7f9cec100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:32.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.931+0000 7f9cf1822640 1 --2- 172.21.15.28:0/3089651542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec100610 0x7f9cec100a90 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0009930 tx=0x7f9ce002a420 comp rx=0 tx=0).stop 2024-04-25T01:41:32.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.931+0000 7f9cf1822640 1 -- 172.21.15.28:0/3089651542 shutdown_connections 2024-04-25T01:41:32.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.931+0000 7f9cf1822640 1 --2- 172.21.15.28:0/3089651542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cec100fd0 0x7f9cec10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.931+0000 7f9cf1822640 1 --2- 172.21.15.28:0/3089651542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec100610 0x7f9cec100a90 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.931+0000 7f9cf1822640 1 --2- 172.21.15.28:0/3089651542 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cec1066b0 0x7f9cec106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.931+0000 7f9cf1822640 1 -- 172.21.15.28:0/3089651542 >> 172.21.15.28:0/3089651542 conn(0x7f9cec077740 msgr2=0x7f9cec077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:32.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.931+0000 7f9cf1822640 1 -- 172.21.15.28:0/3089651542 shutdown_connections 2024-04-25T01:41:32.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.931+0000 7f9cf1822640 1 -- 172.21.15.28:0/3089651542 wait complete. 2024-04-25T01:41:32.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.931+0000 7f9cf1822640 1 Processor -- start 2024-04-25T01:41:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.931+0000 7f9cf1822640 1 -- start start 2024-04-25T01:41:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.932+0000 7f9cf1822640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec100610 0x7f9cec1022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.932+0000 7f9cf1822640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cec100fd0 0x7f9cec1027f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.932+0000 7f9cf1822640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cec1066b0 0x7f9cec102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.932+0000 7f9cf1822640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec110a40 con 0x7f9cec100610 2024-04-25T01:41:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.932+0000 7f9cf1822640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec1108c0 con 0x7f9cec100fd0 2024-04-25T01:41:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.932+0000 7f9cf1822640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec110bc0 con 0x7f9cec1066b0 2024-04-25T01:41:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.932+0000 7f9ceaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec100610 0x7f9cec1022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.932+0000 7f9ceaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec100610 0x7f9cec1022b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58640/0 (socket says 172.21.15.28:58640) 2024-04-25T01:41:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.932+0000 7f9ceaffd640 1 -- 172.21.15.28:0/3332105846 learned_addr learned my addr 172.21.15.28:0/3332105846 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.932+0000 7f9ceb7fe640 1 --2- 172.21.15.28:0/3332105846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cec1066b0 0x7f9cec102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.932+0000 7f9cea7fc640 1 --2- 172.21.15.28:0/3332105846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cec100fd0 0x7f9cec1027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.932+0000 7f9ceaffd640 1 -- 172.21.15.28:0/3332105846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cec1066b0 msgr2=0x7f9cec102d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.932+0000 7f9ceaffd640 1 --2- 172.21.15.28:0/3332105846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cec1066b0 0x7f9cec102d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.932+0000 7f9ceaffd640 1 -- 172.21.15.28:0/3332105846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cec100fd0 msgr2=0x7f9cec1027f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.932+0000 7f9ceaffd640 1 --2- 172.21.15.28:0/3332105846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cec100fd0 0x7f9cec1027f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.932+0000 7f9ceaffd640 1 -- 172.21.15.28:0/3332105846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cec1a64a0 con 0x7f9cec100610 2024-04-25T01:41:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.932+0000 7f9ceb7fe640 1 --2- 172.21.15.28:0/3332105846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cec1066b0 0x7f9cec102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.932+0000 7f9ceaffd640 1 --2- 172.21.15.28:0/3332105846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec100610 0x7f9cec1022b0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00ccb0 tx=0x7f9cdc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.933+0000 7f9cf0820640 1 -- 172.21.15.28:0/3332105846 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc003990 con 0x7f9cec100610 2024-04-25T01:41:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.933+0000 7f9cf0820640 1 -- 172.21.15.28:0/3332105846 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cdc003b30 con 0x7f9cec100610 2024-04-25T01:41:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.933+0000 7f9cf1822640 1 -- 172.21.15.28:0/3332105846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cec1a6790 con 0x7f9cec100610 2024-04-25T01:41:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.933+0000 7f9cf0820640 1 -- 172.21.15.28:0/3332105846 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc00f7b0 con 0x7f9cec100610 2024-04-25T01:41:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.933+0000 7f9cf1822640 1 -- 172.21.15.28:0/3332105846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cec1a6cb0 con 0x7f9cec100610 2024-04-25T01:41:32.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.935+0000 7f9cf0820640 1 -- 172.21.15.28:0/3332105846 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f9cdc00f950 con 0x7f9cec100610 2024-04-25T01:41:32.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.935+0000 7f9cf4605740 1 Processor -- start 2024-04-25T01:41:32.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.935+0000 7f9cf4605740 1 -- start start 2024-04-25T01:41:32.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9cf4605740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daef980 0x55ce1daedcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:32.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9cf4605740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daee290 0x55ce1daee710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:32.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9cf4605740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daf9010 0x55ce1dafb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:32.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9cf4605740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ce1da522c0 con 0x55ce1daee290 2024-04-25T01:41:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9cf4605740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ce1d9fa910 con 0x55ce1daef980 2024-04-25T01:41:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9cf4605740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ce1da4e280 con 0x55ce1daf9010 2024-04-25T01:41:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9cf0820640 1 --2- 172.21.15.28:0/3332105846 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9cd40a2d00 0x7f9cd40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9ceaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daee290 0x55ce1daee710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9ceaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daee290 0x55ce1daee710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58642/0 (socket says 172.21.15.28:58642) 2024-04-25T01:41:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9ceaffd640 1 -- 172.21.15.28:0/3153056602 learned_addr learned my addr 172.21.15.28:0/3153056602 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9ceb7fe640 1 --2- 172.21.15.28:0/3332105846 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9cd40a2d00 0x7f9cd40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9cea7fc640 1 --2- 172.21.15.28:0/3153056602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daf9010 0x55ce1dafb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9ceb7fe640 1 --2- 172.21.15.28:0/3153056602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daef980 0x55ce1daedcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9cf0820640 1 -- 172.21.15.28:0/3332105846 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f9cdc0cf910 con 0x7f9cec100610 2024-04-25T01:41:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9ceaffd640 1 -- 172.21.15.28:0/3153056602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daf9010 msgr2=0x55ce1dafb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9ceaffd640 1 --2- 172.21.15.28:0/3153056602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daf9010 0x55ce1dafb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9ceaffd640 1 -- 172.21.15.28:0/3153056602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daef980 msgr2=0x55ce1daedcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9ceaffd640 1 --2- 172.21.15.28:0/3153056602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daef980 0x55ce1daedcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9ceaffd640 1 -- 172.21.15.28:0/3153056602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce1da0bfb0 con 0x55ce1daee290 2024-04-25T01:41:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.936+0000 7f9ceb7fe640 1 --2- 172.21.15.28:0/3153056602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daef980 0x55ce1daedcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.937+0000 7f9ceaffd640 1 --2- 172.21.15.28:0/3153056602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daee290 0x55ce1daee710 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc0a41e0 tx=0x7f9cdc097e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1efab8ad40fa8c15 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.937+0000 7f9ceb7fe640 1 --2- 172.21.15.28:0/3332105846 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9cd40a2d00 0x7f9cd40a51c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9ce40136a0 tx=0x7f9ce4005f70 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.937+0000 7f9cd1ffb640 1 -- 172.21.15.28:0/3153056602 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc01d070 con 0x55ce1daee290 2024-04-25T01:41:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.937+0000 7f9cd1ffb640 1 -- 172.21.15.28:0/3153056602 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cdc09baa0 con 0x55ce1daee290 2024-04-25T01:41:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.937+0000 7f9cd1ffb640 1 -- 172.21.15.28:0/3153056602 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc098110 con 0x55ce1daee290 2024-04-25T01:41:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.937+0000 7f9cf4605740 1 -- 172.21.15.28:0/3153056602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daee290 msgr2=0x55ce1daee710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.937+0000 7f9cf4605740 1 --2- 172.21.15.28:0/3153056602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daee290 0x55ce1daee710 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc0a41e0 tx=0x7f9cdc097e70 comp rx=0 tx=0).stop 2024-04-25T01:41:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.937+0000 7f9cf4605740 1 -- 172.21.15.28:0/3153056602 shutdown_connections 2024-04-25T01:41:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.937+0000 7f9cf4605740 1 --2- 172.21.15.28:0/3153056602 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daf9010 0x55ce1dafb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.937+0000 7f9cf4605740 1 --2- 172.21.15.28:0/3153056602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daee290 0x55ce1daee710 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.937+0000 7f9cf4605740 1 --2- 172.21.15.28:0/3153056602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daef980 0x55ce1daedcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.937+0000 7f9cf4605740 1 -- 172.21.15.28:0/3153056602 >> 172.21.15.28:0/3153056602 conn(0x55ce1dab6950 msgr2=0x55ce1daea7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.937+0000 7f9cf4605740 1 -- 172.21.15.28:0/3153056602 shutdown_connections 2024-04-25T01:41:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.938+0000 7f9cf4605740 1 -- 172.21.15.28:0/3153056602 wait complete. 2024-04-25T01:41:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.938+0000 7f9cf4605740 1 -- start start 2024-04-25T01:41:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.938+0000 7f9cf4605740 1 Processor -- start 2024-04-25T01:41:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.938+0000 7f9cf4605740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daee290 0x55ce1daf5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.938+0000 7f9cf4605740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daef980 0x55ce1daf6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.938+0000 7f9cf4605740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daf9010 0x55ce1daf6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.938+0000 7f9cf4605740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ce1dafbe40 con 0x55ce1daef980 2024-04-25T01:41:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.938+0000 7f9cf4605740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ce1da0de90 con 0x55ce1daee290 2024-04-25T01:41:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.938+0000 7f9cf4605740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ce1dafbfc0 con 0x55ce1daf9010 2024-04-25T01:41:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.938+0000 7f9cf4605740 10 client.0 _ll_register_callbacks cb 0x55ce1da49970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.939+0000 7f9cea7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daef980 0x55ce1daf6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.939+0000 7f9cea7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daef980 0x55ce1daf6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58658/0 (socket says 172.21.15.28:58658) 2024-04-25T01:41:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.939+0000 7f9cea7fc640 1 -- 172.21.15.28:0/1210539753 learned_addr learned my addr 172.21.15.28:0/1210539753 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.939+0000 7f9ceaffd640 1 --2- 172.21.15.28:0/1210539753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daee290 0x55ce1daf5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.939+0000 7f9ceb7fe640 1 --2- 172.21.15.28:0/1210539753 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daf9010 0x55ce1daf6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.939+0000 7f9cea7fc640 1 -- 172.21.15.28:0/1210539753 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daf9010 msgr2=0x55ce1daf6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.939+0000 7f9cea7fc640 1 --2- 172.21.15.28:0/1210539753 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daf9010 0x55ce1daf6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.939+0000 7f9cea7fc640 1 -- 172.21.15.28:0/1210539753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daee290 msgr2=0x55ce1daf5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.939+0000 7f9cea7fc640 1 --2- 172.21.15.28:0/1210539753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daee290 0x55ce1daf5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.939+0000 7f9cea7fc640 1 -- 172.21.15.28:0/1210539753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce1d82e7e0 con 0x55ce1daef980 2024-04-25T01:41:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.939+0000 7f9ceaffd640 1 --2- 172.21.15.28:0/1210539753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daee290 0x55ce1daf5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.939+0000 7f9cea7fc640 1 --2- 172.21.15.28:0/1210539753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daef980 0x55ce1daf6520 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f9ce002aa10 tx=0x7f9ce0003800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba7173b3f31f0c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.939+0000 7f9ccaffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.939+0000 7f9ccaffd640 1 -- 172.21.15.28:0/1210539753 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0035040 con 0x55ce1daef980 2024-04-25T01:41:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.939+0000 7f9ccaffd640 1 -- 172.21.15.28:0/1210539753 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f9ce0003bf0 con 0x55ce1daef980 2024-04-25T01:41:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.939+0000 7f9cf4605740 1 -- 172.21.15.28:0/1210539753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9cdc09baa0 con 0x55ce1daef980 2024-04-25T01:41:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.939+0000 7f9ccaffd640 1 -- 172.21.15.28:0/1210539753 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9ce0003df0 con 0x55ce1daef980 2024-04-25T01:41:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.939+0000 7f9cf4605740 10 client.18813 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.939+0000 7f9cf4605740 1 -- 172.21.15.28:0/1210539753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ce1d88ef70 con 0x55ce1daef980 2024-04-25T01:41:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.940+0000 7f9ccaffd640 1 -- 172.21.15.28:0/1210539753 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cdc09baa0 con 0x55ce1daef980 2024-04-25T01:41:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.940+0000 7f9ccaffd640 1 -- 172.21.15.28:0/1210539753 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9ce003c740 con 0x55ce1daef980 2024-04-25T01:41:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.940+0000 7f9cf4605740 10 client.18813 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.940+0000 7f9cf4605740 10 client.18813 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:41:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.940+0000 7f9cf4605740 20 client.18813 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.940+0000 7f9cf4605740 1 --2- 172.21.15.28:0/1210539753 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ce1db09160 0x55ce1db0b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.940+0000 7f9cf4605740 4 client.18813 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:41:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.940+0000 7f9cf4605740 1 -- 172.21.15.28:0/1210539753 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce1da265a0 con 0x55ce1db09160 2024-04-25T01:41:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.940+0000 7f9ceaffd640 1 --2- 172.21.15.28:0/1210539753 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ce1db09160 0x55ce1db0b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.941+0000 7f9ceaffd640 1 --2- 172.21.15.28:0/1210539753 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ce1db09160 0x55ce1db0b5b0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=972564d8536c56aa server_cookie=10b4a68a45715653 in_seq=0 out_seq=0 2024-04-25T01:41:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.941+0000 7f9ccaffd640 10 client.18813 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.941+0000 7f9ccaffd640 1 -- 172.21.15.28:0/1210539753 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ce1da265a0 con 0x55ce1db09160 2024-04-25T01:41:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.941+0000 7f9ccaffd640 10 client.18813 handle_command_reply: tid=0 2024-04-25T01:41:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.942+0000 7f9cf4605740 1 client.18813 shutdown 2024-04-25T01:41:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.942+0000 7f9cf4605740 1 -- 172.21.15.28:0/1210539753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daef980 msgr2=0x55ce1daf6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.942+0000 7f9cf4605740 1 --2- 172.21.15.28:0/1210539753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daef980 0x55ce1daf6520 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f9ce002aa10 tx=0x7f9ce0003800 comp rx=0 tx=0).stop 2024-04-25T01:41:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.942+0000 7f9cf4605740 1 -- 172.21.15.28:0/1210539753 shutdown_connections 2024-04-25T01:41:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.942+0000 7f9cf4605740 1 --2- 172.21.15.28:0/1210539753 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ce1db09160 0x55ce1db0b5b0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.942+0000 7f9cf4605740 1 --2- 172.21.15.28:0/1210539753 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daf9010 0x55ce1daf6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.942+0000 7f9cf4605740 1 --2- 172.21.15.28:0/1210539753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daef980 0x55ce1daf6520 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.942+0000 7f9ccaffd640 0 client.18813 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.942+0000 7f9cf4605740 1 --2- 172.21.15.28:0/1210539753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daee290 0x55ce1daf5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.942+0000 7f9cf4605740 1 -- 172.21.15.28:0/1210539753 >> 172.21.15.28:0/1210539753 conn(0x55ce1dab6950 msgr2=0x55ce1daf05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.942+0000 7f9cf4605740 1 -- 172.21.15.28:0/1210539753 shutdown_connections 2024-04-25T01:41:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.942+0000 7f9cf4605740 1 -- 172.21.15.28:0/1210539753 wait complete. 2024-04-25T01:41:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.942+0000 7f9cf4605740 20 client.18813 trim_cache size 0 max 16384 2024-04-25T01:41:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.960+0000 7f9cf4605740 1 Processor -- start 2024-04-25T01:41:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.960+0000 7f9cf4605740 1 -- start start 2024-04-25T01:41:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.960+0000 7f9cf4605740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daee290 0x55ce1db04d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.960+0000 7f9cf4605740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daef980 0x55ce1daf7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.960+0000 7f9cf4605740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daf9010 0x55ce1db10ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.960+0000 7f9cf4605740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec1b0da0 con 0x55ce1daef980 2024-04-25T01:41:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.960+0000 7f9cf4605740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec1b28e0 con 0x55ce1daee290 2024-04-25T01:41:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.960+0000 7f9cf4605740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec1b1470 con 0x55ce1daf9010 2024-04-25T01:41:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.960+0000 7f9cea7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daef980 0x55ce1daf7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.960+0000 7f9cea7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daef980 0x55ce1daf7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58660/0 (socket says 172.21.15.28:58660) 2024-04-25T01:41:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.960+0000 7f9cea7fc640 1 -- 172.21.15.28:0/3257703555 learned_addr learned my addr 172.21.15.28:0/3257703555 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.961+0000 7f9ceaffd640 1 --2- 172.21.15.28:0/3257703555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daee290 0x55ce1db04d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.961+0000 7f9ceb7fe640 1 --2- 172.21.15.28:0/3257703555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daf9010 0x55ce1db10ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.961+0000 7f9cea7fc640 1 -- 172.21.15.28:0/3257703555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daf9010 msgr2=0x55ce1db10ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.961+0000 7f9cea7fc640 1 --2- 172.21.15.28:0/3257703555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daf9010 0x55ce1db10ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.961+0000 7f9cea7fc640 1 -- 172.21.15.28:0/3257703555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daee290 msgr2=0x55ce1db04d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.961+0000 7f9cea7fc640 1 --2- 172.21.15.28:0/3257703555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daee290 0x55ce1db04d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.961+0000 7f9cea7fc640 1 -- 172.21.15.28:0/3257703555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce1d726170 con 0x55ce1daef980 2024-04-25T01:41:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.961+0000 7f9ceaffd640 1 --2- 172.21.15.28:0/3257703555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daee290 0x55ce1db04d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.961+0000 7f9cea7fc640 1 --2- 172.21.15.28:0/3257703555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daef980 0x55ce1daf7f20 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f9ce003cd00 tx=0x7f9ce0005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2dc20ae185de513 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.961+0000 7f9cd17fa640 1 -- 172.21.15.28:0/3257703555 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0055020 con 0x55ce1daef980 2024-04-25T01:41:32.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.961+0000 7f9cd17fa640 1 -- 172.21.15.28:0/3257703555 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ce003c440 con 0x55ce1daef980 2024-04-25T01:41:32.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.961+0000 7f9cd17fa640 1 -- 172.21.15.28:0/3257703555 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0035050 con 0x55ce1daef980 2024-04-25T01:41:32.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.962+0000 7f9cf4605740 1 -- 172.21.15.28:0/3257703555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daef980 msgr2=0x55ce1daf7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:32.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.962+0000 7f9cf4605740 1 --2- 172.21.15.28:0/3257703555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daef980 0x55ce1daf7f20 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f9ce003cd00 tx=0x7f9ce0005e00 comp rx=0 tx=0).stop 2024-04-25T01:41:32.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.962+0000 7f9cf4605740 1 -- 172.21.15.28:0/3257703555 shutdown_connections 2024-04-25T01:41:32.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.962+0000 7f9cf4605740 1 --2- 172.21.15.28:0/3257703555 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daf9010 0x55ce1db10ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.962+0000 7f9cf4605740 1 --2- 172.21.15.28:0/3257703555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daef980 0x55ce1daf7f20 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.962+0000 7f9cf4605740 1 --2- 172.21.15.28:0/3257703555 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daee290 0x55ce1db04d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.962+0000 7f9cf4605740 1 -- 172.21.15.28:0/3257703555 >> 172.21.15.28:0/3257703555 conn(0x55ce1dab6950 msgr2=0x55ce1daf7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:32.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.962+0000 7f9cf4605740 1 -- 172.21.15.28:0/3257703555 shutdown_connections 2024-04-25T01:41:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.962+0000 7f9cf4605740 1 -- 172.21.15.28:0/3257703555 wait complete. 2024-04-25T01:41:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.962+0000 7f9cf4605740 1 -- start start 2024-04-25T01:41:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.962+0000 7f9cf4605740 1 Processor -- start 2024-04-25T01:41:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.962+0000 7f9cf4605740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daee290 0x55ce1db01250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.962+0000 7f9cf4605740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daef980 0x55ce1db01790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.962+0000 7f9cf4605740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daf9010 0x55ce1dafe220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9cf4605740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec1b2210 con 0x55ce1daee290 2024-04-25T01:41:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9cf4605740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec1b1b40 con 0x55ce1daf9010 2024-04-25T01:41:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9cf4605740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec1aeda0 con 0x55ce1daef980 2024-04-25T01:41:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9cf4605740 10 client.0 _ll_register_callbacks cb 0x7f9ce0005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9ceaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daee290 0x55ce1db01250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9ceaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daee290 0x55ce1db01250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58676/0 (socket says 172.21.15.28:58676) 2024-04-25T01:41:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9ceaffd640 1 -- 172.21.15.28:0/2725712658 learned_addr learned my addr 172.21.15.28:0/2725712658 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9cea7fc640 1 --2- 172.21.15.28:0/2725712658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daef980 0x55ce1db01790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:32.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9ceb7fe640 1 --2- 172.21.15.28:0/2725712658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daf9010 0x55ce1dafe220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:32.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9ceaffd640 1 -- 172.21.15.28:0/2725712658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daef980 msgr2=0x55ce1db01790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:32.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9ceaffd640 1 --2- 172.21.15.28:0/2725712658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daef980 0x55ce1db01790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9ceaffd640 1 -- 172.21.15.28:0/2725712658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daf9010 msgr2=0x55ce1dafe220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:32.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9ceaffd640 1 --2- 172.21.15.28:0/2725712658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daf9010 0x55ce1dafe220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9ceaffd640 1 -- 172.21.15.28:0/2725712658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce1dafe960 con 0x55ce1daee290 2024-04-25T01:41:32.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9ceb7fe640 1 --2- 172.21.15.28:0/2725712658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daf9010 0x55ce1dafe220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:32.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9cea7fc640 1 --2- 172.21.15.28:0/2725712658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daef980 0x55ce1db01790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:32.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9ceaffd640 1 --2- 172.21.15.28:0/2725712658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daee290 0x55ce1db01250 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc0cd2d0 tx=0x7f9cdc097d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b5a6a076c653cd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:32.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9ccaffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:32.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9ccaffd640 1 -- 172.21.15.28:0/2725712658 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc01d040 con 0x55ce1daee290 2024-04-25T01:41:32.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9ccaffd640 1 -- 172.21.15.28:0/2725712658 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f9cdc0ccf50 con 0x55ce1daee290 2024-04-25T01:41:32.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9cf4605740 1 -- 172.21.15.28:0/2725712658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ce003c440 con 0x55ce1daee290 2024-04-25T01:41:32.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9ccaffd640 1 -- 172.21.15.28:0/2725712658 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9cdc09baa0 con 0x55ce1daee290 2024-04-25T01:41:32.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9cf4605740 10 client.18819 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:32.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.963+0000 7f9cf4605740 1 -- 172.21.15.28:0/2725712658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ce1db701c0 con 0x55ce1daee290 2024-04-25T01:41:32.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.964+0000 7f9ccaffd640 1 -- 172.21.15.28:0/2725712658 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ce003c440 con 0x55ce1daee290 2024-04-25T01:41:32.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.964+0000 7f9ccaffd640 1 -- 172.21.15.28:0/2725712658 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9cdc00fc40 con 0x55ce1daee290 2024-04-25T01:41:32.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.964+0000 7f9cf4605740 10 client.18819 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:32.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.964+0000 7f9cf4605740 10 client.18819 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:41:32.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.964+0000 7f9cf4605740 20 client.18819 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:32.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.964+0000 7f9cf4605740 1 --2- 172.21.15.28:0/2725712658 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ce1db704b0 0x55ce1db728a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:32.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.964+0000 7f9cf4605740 4 client.18819 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:41:32.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.964+0000 7f9cf4605740 1 -- 172.21.15.28:0/2725712658 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ce1da5f800 con 0x55ce1db704b0 2024-04-25T01:41:32.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.964+0000 7f9cea7fc640 1 --2- 172.21.15.28:0/2725712658 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ce1db704b0 0x55ce1db728a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:32.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.965+0000 7f9cea7fc640 1 --2- 172.21.15.28:0/2725712658 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ce1db704b0 0x55ce1db728a0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d8a7b66672e027 server_cookie=4303058e6ab89061 in_seq=0 out_seq=0 2024-04-25T01:41:32.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.965+0000 7f9ccaffd640 10 client.18819 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:32.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.966+0000 7f9ccaffd640 1 -- 172.21.15.28:0/2725712658 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ce1da5f800 con 0x55ce1db704b0 2024-04-25T01:41:32.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.966+0000 7f9ccaffd640 10 client.18819 handle_command_reply: tid=0 2024-04-25T01:41:32.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.966+0000 7f9cf4605740 1 client.18819 shutdown 2024-04-25T01:41:32.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.966+0000 7f9cf4605740 1 -- 172.21.15.28:0/2725712658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daee290 msgr2=0x55ce1db01250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:32.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.966+0000 7f9cf4605740 1 --2- 172.21.15.28:0/2725712658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daee290 0x55ce1db01250 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc0cd2d0 tx=0x7f9cdc097d40 comp rx=0 tx=0).stop 2024-04-25T01:41:32.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.966+0000 7f9cf4605740 1 -- 172.21.15.28:0/2725712658 shutdown_connections 2024-04-25T01:41:32.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.966+0000 7f9cf4605740 1 --2- 172.21.15.28:0/2725712658 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ce1db704b0 0x55ce1db728a0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.966+0000 7f9cf4605740 1 --2- 172.21.15.28:0/2725712658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ce1daf9010 0x55ce1dafe220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.966+0000 7f9cf4605740 1 --2- 172.21.15.28:0/2725712658 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ce1daef980 0x55ce1db01790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.966+0000 7f9cf4605740 1 --2- 172.21.15.28:0/2725712658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce1daee290 0x55ce1db01250 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.966+0000 7f9ccaffd640 0 client.18819 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:32.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.966+0000 7f9cf4605740 1 -- 172.21.15.28:0/2725712658 >> 172.21.15.28:0/2725712658 conn(0x55ce1dab6950 msgr2=0x55ce1dae9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:32.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.966+0000 7f9cf4605740 1 -- 172.21.15.28:0/2725712658 shutdown_connections 2024-04-25T01:41:32.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.966+0000 7f9cf4605740 1 -- 172.21.15.28:0/2725712658 wait complete. 2024-04-25T01:41:32.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.966+0000 7f9cf4605740 20 client.18819 trim_cache size 0 max 16384 2024-04-25T01:41:32.962 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:41:32.962 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:41:32.962 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "0657ac0c-ad74-4b49-902a-abc291c11e5b", 2024-04-25T01:41:32.962 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:41:32.962 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:41:32.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.967+0000 7f9cf1822640 1 -- 172.21.15.28:0/3332105846 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9cd40a2d00 msgr2=0x7f9cd40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:32.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.967+0000 7f9cf1822640 1 --2- 172.21.15.28:0/3332105846 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9cd40a2d00 0x7f9cd40a51c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9ce40136a0 tx=0x7f9ce4005f70 comp rx=0 tx=0).stop 2024-04-25T01:41:32.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.967+0000 7f9cf1822640 1 -- 172.21.15.28:0/3332105846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec100610 msgr2=0x7f9cec1022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:32.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.967+0000 7f9cf1822640 1 --2- 172.21.15.28:0/3332105846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec100610 0x7f9cec1022b0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00ccb0 tx=0x7f9cdc00a620 comp rx=0 tx=0).stop 2024-04-25T01:41:32.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.967+0000 7f9cf1822640 1 -- 172.21.15.28:0/3332105846 shutdown_connections 2024-04-25T01:41:32.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.967+0000 7f9cf1822640 1 --2- 172.21.15.28:0/3332105846 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f9cd40a2d00 0x7f9cd40a51c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.967+0000 7f9cf1822640 1 --2- 172.21.15.28:0/3332105846 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9cec1066b0 0x7f9cec102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.967+0000 7f9cf1822640 1 --2- 172.21.15.28:0/3332105846 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9cec100fd0 0x7f9cec1027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.967+0000 7f9cf1822640 1 --2- 172.21.15.28:0/3332105846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cec100610 0x7f9cec1022b0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:32.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.967+0000 7f9cf1822640 1 -- 172.21.15.28:0/3332105846 >> 172.21.15.28:0/3332105846 conn(0x7f9cec077740 msgr2=0x7f9cec0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:32.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.967+0000 7f9cf1822640 1 -- 172.21.15.28:0/3332105846 shutdown_connections 2024-04-25T01:41:32.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:32.967+0000 7f9cf1822640 1 -- 172.21.15.28:0/3332105846 wait complete. 2024-04-25T01:41:32.966 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:41:32.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:41:33.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.083+0000 7f1e7694f640 1 -- 172.21.15.28:0/4207849050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e7006a780 msgr2=0x7f1e7010e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:33.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.083+0000 7f1e7694f640 1 --2- 172.21.15.28:0/4207849050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e7006a780 0x7f1e7010e5f0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f1e68009cd0 tx=0x7f1e6802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.083+0000 7f1e7694f640 1 -- 172.21.15.28:0/4207849050 shutdown_connections 2024-04-25T01:41:33.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.083+0000 7f1e7694f640 1 --2- 172.21.15.28:0/4207849050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e7006a780 0x7f1e7010e5f0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.083+0000 7f1e7694f640 1 --2- 172.21.15.28:0/4207849050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e70069dc0 0x7f1e7006a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.083+0000 7f1e7694f640 1 --2- 172.21.15.28:0/4207849050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e70107730 0x7f1e70107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.083+0000 7f1e7694f640 1 -- 172.21.15.28:0/4207849050 >> 172.21.15.28:0/4207849050 conn(0x7f1e700776a0 msgr2=0x7f1e70077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:33.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.083+0000 7f1e7694f640 1 -- 172.21.15.28:0/4207849050 shutdown_connections 2024-04-25T01:41:33.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.083+0000 7f1e7694f640 1 -- 172.21.15.28:0/4207849050 wait complete. 2024-04-25T01:41:33.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.084+0000 7f1e7694f640 1 Processor -- start 2024-04-25T01:41:33.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.084+0000 7f1e7694f640 1 -- start start 2024-04-25T01:41:33.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.084+0000 7f1e7694f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e70069dc0 0x7f1e70197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:33.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.084+0000 7f1e7694f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e7006a780 0x7f1e701984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:33.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.084+0000 7f1e7694f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e70107730 0x7f1e7019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.084+0000 7f1e7694f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e70111b00 con 0x7f1e70107730 2024-04-25T01:41:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.084+0000 7f1e7694f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1e70111980 con 0x7f1e7006a780 2024-04-25T01:41:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.084+0000 7f1e7694f640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1e70111c80 con 0x7f1e70069dc0 2024-04-25T01:41:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.084+0000 7f1e74eec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e70107730 0x7f1e7019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.084+0000 7f1e74eec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e70107730 0x7f1e7019c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58684/0 (socket says 172.21.15.28:58684) 2024-04-25T01:41:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.084+0000 7f1e74eec640 1 -- 172.21.15.28:0/2423374959 learned_addr learned my addr 172.21.15.28:0/2423374959 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.084+0000 7f1e6ffff640 1 --2- 172.21.15.28:0/2423374959 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e70069dc0 0x7f1e70197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.084+0000 7f1e6f7fe640 1 --2- 172.21.15.28:0/2423374959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e7006a780 0x7f1e701984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.085+0000 7f1e74eec640 1 -- 172.21.15.28:0/2423374959 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e70069dc0 msgr2=0x7f1e70197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.085+0000 7f1e74eec640 1 --2- 172.21.15.28:0/2423374959 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e70069dc0 0x7f1e70197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.085+0000 7f1e74eec640 1 -- 172.21.15.28:0/2423374959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e7006a780 msgr2=0x7f1e701984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.085+0000 7f1e74eec640 1 --2- 172.21.15.28:0/2423374959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e7006a780 0x7f1e701984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.085+0000 7f1e74eec640 1 -- 172.21.15.28:0/2423374959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e7019cf50 con 0x7f1e70107730 2024-04-25T01:41:33.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.085+0000 7f1e74eec640 1 --2- 172.21.15.28:0/2423374959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e70107730 0x7f1e7019c820 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f1e68005cc0 tx=0x7f1e68003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.085+0000 7f1e6d7fa640 1 -- 172.21.15.28:0/2423374959 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68004040 con 0x7f1e70107730 2024-04-25T01:41:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.085+0000 7f1e7694f640 1 -- 172.21.15.28:0/2423374959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e7019d1e0 con 0x7f1e70107730 2024-04-25T01:41:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.085+0000 7f1e6d7fa640 1 -- 172.21.15.28:0/2423374959 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e680046c0 con 0x7f1e70107730 2024-04-25T01:41:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.085+0000 7f1e6d7fa640 1 -- 172.21.15.28:0/2423374959 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68046480 con 0x7f1e70107730 2024-04-25T01:41:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.085+0000 7f1e7694f640 1 -- 172.21.15.28:0/2423374959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e701a4ad0 con 0x7f1e70107730 2024-04-25T01:41:33.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.087+0000 7f1e6d7fa640 1 -- 172.21.15.28:0/2423374959 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1e680305a0 con 0x7f1e70107730 2024-04-25T01:41:33.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.088+0000 7f1e7965c740 1 Processor -- start 2024-04-25T01:41:33.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.088+0000 7f1e7965c740 1 -- start start 2024-04-25T01:41:33.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.088+0000 7f1e7965c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edf630 0x563ef3edfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:33.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.088+0000 7f1e6d7fa640 1 --2- 172.21.15.28:0/2423374959 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1e580a2d00 0x7f1e580a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:33.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.088+0000 7f1e7965c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edff50 0x563ef3eea0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:33.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.088+0000 7f1e7965c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3eea620 0x563ef3eeca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:33.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.088+0000 7f1e7965c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563ef3e446b0 con 0x563ef3edff50 2024-04-25T01:41:33.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.088+0000 7f1e7965c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563ef3dec880 con 0x563ef3eea620 2024-04-25T01:41:33.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.088+0000 7f1e7965c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x563ef3e40670 con 0x563ef3edf630 2024-04-25T01:41:33.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.089+0000 7f1e6ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edff50 0x563ef3eea0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:33.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.089+0000 7f1e6f7fe640 1 --2- 172.21.15.28:0/2423374959 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1e580a2d00 0x7f1e580a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:33.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.089+0000 7f1e6ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edff50 0x563ef3eea0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58698/0 (socket says 172.21.15.28:58698) 2024-04-25T01:41:33.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.089+0000 7f1e6ffff640 1 -- 172.21.15.28:0/3701048908 learned_addr learned my addr 172.21.15.28:0/3701048908 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:33.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.089+0000 7f1e6d7fa640 1 -- 172.21.15.28:0/2423374959 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f1e6802e030 con 0x7f1e70107730 2024-04-25T01:41:33.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.089+0000 7f1e6f7fe640 1 --2- 172.21.15.28:0/3701048908 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edf630 0x563ef3edfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:33.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.089+0000 7f1e74eec640 1 --2- 172.21.15.28:0/3701048908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3eea620 0x563ef3eeca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:33.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.089+0000 7f1e6ffff640 1 -- 172.21.15.28:0/3701048908 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edf630 msgr2=0x563ef3edfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:33.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.089+0000 7f1e6ffff640 1 --2- 172.21.15.28:0/3701048908 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edf630 0x563ef3edfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.089+0000 7f1e6ffff640 1 -- 172.21.15.28:0/3701048908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3eea620 msgr2=0x563ef3eeca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:33.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.089+0000 7f1e6ffff640 1 --2- 172.21.15.28:0/3701048908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3eea620 0x563ef3eeca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.089+0000 7f1e6ffff640 1 -- 172.21.15.28:0/3701048908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ef3c227d0 con 0x563ef3edff50 2024-04-25T01:41:33.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.089+0000 7f1e74eec640 1 --2- 172.21.15.28:0/3701048908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3eea620 0x563ef3eeca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:33.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.089+0000 7f1e6f7fe640 1 --2- 172.21.15.28:0/2423374959 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1e580a2d00 0x7f1e580a51c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1e70077c10 tx=0x7f1e6400a830 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:33.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.089+0000 7f1e6ffff640 1 --2- 172.21.15.28:0/3701048908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edff50 0x563ef3eea0e0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f1e6000bc50 tx=0x7f1e600107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ccbbf83d63523 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:33.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.089+0000 7f1e567fc640 1 -- 172.21.15.28:0/3701048908 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6001c480 con 0x563ef3edff50 2024-04-25T01:41:33.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.089+0000 7f1e567fc640 1 -- 172.21.15.28:0/3701048908 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e6001cb00 con 0x563ef3edff50 2024-04-25T01:41:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.089+0000 7f1e567fc640 1 -- 172.21.15.28:0/3701048908 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e60011ba0 con 0x563ef3edff50 2024-04-25T01:41:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.090+0000 7f1e7965c740 1 -- 172.21.15.28:0/3701048908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edff50 msgr2=0x563ef3eea0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.090+0000 7f1e7965c740 1 --2- 172.21.15.28:0/3701048908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edff50 0x563ef3eea0e0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f1e6000bc50 tx=0x7f1e600107a0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.090+0000 7f1e7965c740 1 -- 172.21.15.28:0/3701048908 shutdown_connections 2024-04-25T01:41:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.090+0000 7f1e7965c740 1 --2- 172.21.15.28:0/3701048908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3eea620 0x563ef3eeca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.090+0000 7f1e7965c740 1 --2- 172.21.15.28:0/3701048908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edff50 0x563ef3eea0e0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.090+0000 7f1e7965c740 1 --2- 172.21.15.28:0/3701048908 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edf630 0x563ef3edfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.090+0000 7f1e7965c740 1 -- 172.21.15.28:0/3701048908 >> 172.21.15.28:0/3701048908 conn(0x563ef3edb950 msgr2=0x563ef3edbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.090+0000 7f1e7965c740 1 -- 172.21.15.28:0/3701048908 shutdown_connections 2024-04-25T01:41:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.090+0000 7f1e7965c740 1 -- 172.21.15.28:0/3701048908 wait complete. 2024-04-25T01:41:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.090+0000 7f1e7965c740 1 -- start start 2024-04-25T01:41:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e7965c740 1 Processor -- start 2024-04-25T01:41:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e7965c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edf630 0x563ef3ee75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e7965c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3edff50 0x563ef3ee7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e7965c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3eea620 0x563ef3ee8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e7965c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563ef3eed410 con 0x563ef3eea620 2024-04-25T01:41:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e7965c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563ef3dfff50 con 0x563ef3edff50 2024-04-25T01:41:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e7965c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x563ef3eed590 con 0x563ef3edf630 2024-04-25T01:41:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e7965c740 10 client.0 _ll_register_callbacks cb 0x563ef3e3b000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e74eec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3eea620 0x563ef3ee8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e74eec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3eea620 0x563ef3ee8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58704/0 (socket says 172.21.15.28:58704) 2024-04-25T01:41:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e74eec640 1 -- 172.21.15.28:0/3364021540 learned_addr learned my addr 172.21.15.28:0/3364021540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e6ffff640 1 --2- 172.21.15.28:0/3364021540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edf630 0x563ef3ee75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e6f7fe640 1 --2- 172.21.15.28:0/3364021540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3edff50 0x563ef3ee7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e74eec640 1 -- 172.21.15.28:0/3364021540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edf630 msgr2=0x563ef3ee75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e74eec640 1 --2- 172.21.15.28:0/3364021540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edf630 0x563ef3ee75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e74eec640 1 -- 172.21.15.28:0/3364021540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3edff50 msgr2=0x563ef3ee7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e74eec640 1 --2- 172.21.15.28:0/3364021540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3edff50 0x563ef3ee7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e74eec640 1 -- 172.21.15.28:0/3364021540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ef3b583f0 con 0x563ef3eea620 2024-04-25T01:41:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e6ffff640 1 --2- 172.21.15.28:0/3364021540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edf630 0x563ef3ee75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e74eec640 1 --2- 172.21.15.28:0/3364021540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3eea620 0x563ef3ee8050 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f1e680b9ab0 tx=0x7f1e680095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf752e43ce8efd3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.091+0000 7f1e4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.092+0000 7f1e4f7fe640 1 -- 172.21.15.28:0/3364021540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e680d62e0 con 0x563ef3eea620 2024-04-25T01:41:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.092+0000 7f1e4f7fe640 1 -- 172.21.15.28:0/3364021540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f1e680ee480 con 0x563ef3eea620 2024-04-25T01:41:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.092+0000 7f1e7965c740 1 -- 172.21.15.28:0/3364021540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e6001cb00 con 0x563ef3eea620 2024-04-25T01:41:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.092+0000 7f1e4f7fe640 1 -- 172.21.15.28:0/3364021540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1e680862d0 con 0x563ef3eea620 2024-04-25T01:41:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.092+0000 7f1e7965c740 10 client.18831 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.092+0000 7f1e7965c740 1 -- 172.21.15.28:0/3364021540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563ef3b180f0 con 0x563ef3eea620 2024-04-25T01:41:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.092+0000 7f1e4f7fe640 1 -- 172.21.15.28:0/3364021540 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e6001cb00 con 0x563ef3eea620 2024-04-25T01:41:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.092+0000 7f1e4f7fe640 1 -- 172.21.15.28:0/3364021540 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1e6804e070 con 0x563ef3eea620 2024-04-25T01:41:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.092+0000 7f1e7965c740 10 client.18831 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.092+0000 7f1e7965c740 10 client.18831 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:41:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.092+0000 7f1e7965c740 20 client.18831 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.092+0000 7f1e7965c740 1 --2- 172.21.15.28:0/3364021540 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563ef3efa8c0 0x563ef3efcd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.092+0000 7f1e7965c740 4 client.18831 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:41:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.092+0000 7f1e7965c740 1 -- 172.21.15.28:0/3364021540 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ef3e51bf0 con 0x563ef3efa8c0 2024-04-25T01:41:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.092+0000 7f1e6ffff640 1 --2- 172.21.15.28:0/3364021540 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563ef3efa8c0 0x563ef3efcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.093+0000 7f1e4f7fe640 10 client.18831 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.093+0000 7f1e6ffff640 1 --2- 172.21.15.28:0/3364021540 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563ef3efa8c0 0x563ef3efcd60 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5312c0789c25e4f server_cookie=64c6654c2730db95 in_seq=0 out_seq=0 2024-04-25T01:41:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.094+0000 7f1e4f7fe640 1 -- 172.21.15.28:0/3364021540 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563ef3e51bf0 con 0x563ef3efa8c0 2024-04-25T01:41:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.094+0000 7f1e4f7fe640 10 client.18831 handle_command_reply: tid=0 2024-04-25T01:41:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.094+0000 7f1e7965c740 1 client.18831 shutdown 2024-04-25T01:41:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.094+0000 7f1e7965c740 1 -- 172.21.15.28:0/3364021540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3eea620 msgr2=0x563ef3ee8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.094+0000 7f1e7965c740 1 --2- 172.21.15.28:0/3364021540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3eea620 0x563ef3ee8050 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f1e680b9ab0 tx=0x7f1e680095c0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.094+0000 7f1e7965c740 1 -- 172.21.15.28:0/3364021540 shutdown_connections 2024-04-25T01:41:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.094+0000 7f1e7965c740 1 --2- 172.21.15.28:0/3364021540 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563ef3efa8c0 0x563ef3efcd60 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.094+0000 7f1e7965c740 1 --2- 172.21.15.28:0/3364021540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3eea620 0x563ef3ee8050 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.094+0000 7f1e7965c740 1 --2- 172.21.15.28:0/3364021540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3edff50 0x563ef3ee7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.094+0000 7f1e7965c740 1 --2- 172.21.15.28:0/3364021540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edf630 0x563ef3ee75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.094+0000 7f1e4f7fe640 0 client.18831 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:33.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.094+0000 7f1e7965c740 1 -- 172.21.15.28:0/3364021540 >> 172.21.15.28:0/3364021540 conn(0x563ef3edb950 msgr2=0x563ef3ee1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:33.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.094+0000 7f1e7965c740 1 -- 172.21.15.28:0/3364021540 shutdown_connections 2024-04-25T01:41:33.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.094+0000 7f1e7965c740 1 -- 172.21.15.28:0/3364021540 wait complete. 2024-04-25T01:41:33.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.094+0000 7f1e7965c740 20 client.18831 trim_cache size 0 max 16384 2024-04-25T01:41:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.111+0000 7f1e7965c740 1 Processor -- start 2024-04-25T01:41:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.111+0000 7f1e7965c740 1 -- start start 2024-04-25T01:41:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.111+0000 7f1e7965c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edf630 0x563ef3ee6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.111+0000 7f1e7965c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edff50 0x563ef3ef62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.111+0000 7f1e7965c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3eea620 0x563ef3ee1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.111+0000 7f1e7965c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e701aea50 con 0x563ef3edff50 2024-04-25T01:41:33.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.111+0000 7f1e7965c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1e701b0590 con 0x563ef3eea620 2024-04-25T01:41:33.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.111+0000 7f1e7965c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1e701af120 con 0x563ef3edf630 2024-04-25T01:41:33.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.112+0000 7f1e6f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edff50 0x563ef3ef62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:33.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.112+0000 7f1e6f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edff50 0x563ef3ef62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58708/0 (socket says 172.21.15.28:58708) 2024-04-25T01:41:33.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.112+0000 7f1e6f7fe640 1 -- 172.21.15.28:0/708158867 learned_addr learned my addr 172.21.15.28:0/708158867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:33.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.112+0000 7f1e74eec640 1 --2- 172.21.15.28:0/708158867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3eea620 0x563ef3ee1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:33.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.112+0000 7f1e6f7fe640 1 -- 172.21.15.28:0/708158867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edf630 msgr2=0x563ef3ee6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:33.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.112+0000 7f1e6f7fe640 1 --2- 172.21.15.28:0/708158867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edf630 0x563ef3ee6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.112+0000 7f1e6f7fe640 1 -- 172.21.15.28:0/708158867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3eea620 msgr2=0x563ef3ee1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:33.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.112+0000 7f1e6f7fe640 1 --2- 172.21.15.28:0/708158867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3eea620 0x563ef3ee1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.112+0000 7f1e6f7fe640 1 -- 172.21.15.28:0/708158867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ef3edf430 con 0x563ef3edff50 2024-04-25T01:41:33.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.112+0000 7f1e74eec640 1 --2- 172.21.15.28:0/708158867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3eea620 0x563ef3ee1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:41:33.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.112+0000 7f1e6f7fe640 1 --2- 172.21.15.28:0/708158867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edff50 0x563ef3ef62d0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400f7b0 tx=0x7f1e64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a87689b3dabd6260 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:33.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.112+0000 7f1e55ffb640 1 -- 172.21.15.28:0/708158867 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e64005020 con 0x563ef3edff50 2024-04-25T01:41:33.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.112+0000 7f1e55ffb640 1 -- 172.21.15.28:0/708158867 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e640047a0 con 0x563ef3edff50 2024-04-25T01:41:33.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.112+0000 7f1e55ffb640 1 -- 172.21.15.28:0/708158867 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6401eb10 con 0x563ef3edff50 2024-04-25T01:41:33.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.113+0000 7f1e7965c740 1 -- 172.21.15.28:0/708158867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edff50 msgr2=0x563ef3ef62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:33.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.113+0000 7f1e7965c740 1 --2- 172.21.15.28:0/708158867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edff50 0x563ef3ef62d0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400f7b0 tx=0x7f1e64004730 comp rx=0 tx=0).stop 2024-04-25T01:41:33.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.113+0000 7f1e7965c740 1 -- 172.21.15.28:0/708158867 shutdown_connections 2024-04-25T01:41:33.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.113+0000 7f1e7965c740 1 --2- 172.21.15.28:0/708158867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3eea620 0x563ef3ee1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.113+0000 7f1e7965c740 1 --2- 172.21.15.28:0/708158867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edff50 0x563ef3ef62d0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.113+0000 7f1e7965c740 1 --2- 172.21.15.28:0/708158867 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edf630 0x563ef3ee6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.113+0000 7f1e7965c740 1 -- 172.21.15.28:0/708158867 >> 172.21.15.28:0/708158867 conn(0x563ef3edb950 msgr2=0x563ef3ee7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:33.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.113+0000 7f1e7965c740 1 -- 172.21.15.28:0/708158867 shutdown_connections 2024-04-25T01:41:33.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.113+0000 7f1e7965c740 1 -- 172.21.15.28:0/708158867 wait complete. 2024-04-25T01:41:33.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.113+0000 7f1e7965c740 1 -- start start 2024-04-25T01:41:33.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.113+0000 7f1e7965c740 1 Processor -- start 2024-04-25T01:41:33.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e7965c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edf630 0x563ef3ee3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:33.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e7965c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edff50 0x563ef3ee3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:33.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e7965c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3eea620 0x563ef3ee4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:33.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e7965c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e701afec0 con 0x563ef3edf630 2024-04-25T01:41:33.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e7965c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1e701af7f0 con 0x563ef3eea620 2024-04-25T01:41:33.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e7965c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1e701aca50 con 0x563ef3edff50 2024-04-25T01:41:33.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e7965c740 10 client.0 _ll_register_callbacks cb 0x7f1e68009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:41:33.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e6ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edf630 0x563ef3ee3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:33.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e6ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edf630 0x563ef3ee3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58716/0 (socket says 172.21.15.28:58716) 2024-04-25T01:41:33.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e6ffff640 1 -- 172.21.15.28:0/3383505297 learned_addr learned my addr 172.21.15.28:0/3383505297 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:41:33.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e6f7fe640 1 --2- 172.21.15.28:0/3383505297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edff50 0x563ef3ee3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:33.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e74eec640 1 --2- 172.21.15.28:0/3383505297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3eea620 0x563ef3ee4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:33.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e6ffff640 1 -- 172.21.15.28:0/3383505297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edff50 msgr2=0x563ef3ee3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:33.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e6ffff640 1 --2- 172.21.15.28:0/3383505297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edff50 0x563ef3ee3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e6ffff640 1 -- 172.21.15.28:0/3383505297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3eea620 msgr2=0x563ef3ee4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:41:33.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e6ffff640 1 --2- 172.21.15.28:0/3383505297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3eea620 0x563ef3ee4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e6ffff640 1 -- 172.21.15.28:0/3383505297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ef3ee4b30 con 0x563ef3edf630 2024-04-25T01:41:33.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e74eec640 1 --2- 172.21.15.28:0/3383505297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3eea620 0x563ef3ee4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:41:33.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e6ffff640 1 --2- 172.21.15.28:0/3383505297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edf630 0x563ef3ee3840 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x563ef3ee0bf0 tx=0x7f1e6000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad6af0975a09d15e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:41:33.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:41:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e4f7fe640 1 -- 172.21.15.28:0/3383505297 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e60005240 con 0x563ef3edf630 2024-04-25T01:41:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e4f7fe640 1 -- 172.21.15.28:0/3383505297 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f1e600053e0 con 0x563ef3edf630 2024-04-25T01:41:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.114+0000 7f1e7965c740 1 -- 172.21.15.28:0/3383505297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e640047a0 con 0x563ef3edf630 2024-04-25T01:41:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.115+0000 7f1e4f7fe640 1 -- 172.21.15.28:0/3383505297 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1e60037050 con 0x563ef3edf630 2024-04-25T01:41:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.115+0000 7f1e7965c740 10 client.18837 fetch_fsmap learned FSMap version 28 2024-04-25T01:41:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.115+0000 7f1e7965c740 1 -- 172.21.15.28:0/3383505297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563ef3f65520 con 0x563ef3edf630 2024-04-25T01:41:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.115+0000 7f1e4f7fe640 1 -- 172.21.15.28:0/3383505297 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e640047a0 con 0x563ef3edf630 2024-04-25T01:41:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.115+0000 7f1e4f7fe640 1 -- 172.21.15.28:0/3383505297 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1e60032030 con 0x563ef3edf630 2024-04-25T01:41:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.115+0000 7f1e7965c740 10 client.18837 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:41:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.115+0000 7f1e7965c740 10 client.18837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:41:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.115+0000 7f1e7965c740 20 client.18837 populate_metadata read hostname 'smithi028' 2024-04-25T01:41:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.115+0000 7f1e7965c740 1 --2- 172.21.15.28:0/3383505297 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563ef3f659c0 0x563ef3f67db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:41:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.115+0000 7f1e7965c740 4 client.18837 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:41:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.115+0000 7f1e7965c740 1 -- 172.21.15.28:0/3383505297 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563ef3e4bf40 con 0x563ef3f659c0 2024-04-25T01:41:33.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.115+0000 7f1e6f7fe640 1 --2- 172.21.15.28:0/3383505297 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563ef3f659c0 0x563ef3f67db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:41:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.116+0000 7f1e4f7fe640 10 client.18837 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.116+0000 7f1e6f7fe640 1 --2- 172.21.15.28:0/3383505297 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563ef3f659c0 0x563ef3f67db0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8f6ad4b71ff295d server_cookie=f2e19994a6d12db2 in_seq=0 out_seq=0 2024-04-25T01:41:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.116+0000 7f1e4f7fe640 1 -- 172.21.15.28:0/3383505297 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563ef3e4bf40 con 0x563ef3f659c0 2024-04-25T01:41:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.116+0000 7f1e4f7fe640 10 client.18837 handle_command_reply: tid=0 2024-04-25T01:41:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.116+0000 7f1e7965c740 1 client.18837 shutdown 2024-04-25T01:41:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.116+0000 7f1e7965c740 1 -- 172.21.15.28:0/3383505297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edf630 msgr2=0x563ef3ee3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.116+0000 7f1e7965c740 1 --2- 172.21.15.28:0/3383505297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edf630 0x563ef3ee3840 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x563ef3ee0bf0 tx=0x7f1e6000b8e0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.116+0000 7f1e7965c740 1 -- 172.21.15.28:0/3383505297 shutdown_connections 2024-04-25T01:41:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.116+0000 7f1e7965c740 1 --2- 172.21.15.28:0/3383505297 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x563ef3f659c0 0x563ef3f67db0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.116+0000 7f1e7965c740 1 --2- 172.21.15.28:0/3383505297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ef3eea620 0x563ef3ee4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.116+0000 7f1e4f7fe640 0 client.18837 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:41:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.116+0000 7f1e7965c740 1 --2- 172.21.15.28:0/3383505297 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x563ef3edff50 0x563ef3ee3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.116+0000 7f1e7965c740 1 --2- 172.21.15.28:0/3383505297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563ef3edf630 0x563ef3ee3840 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.116+0000 7f1e7965c740 1 -- 172.21.15.28:0/3383505297 >> 172.21.15.28:0/3383505297 conn(0x563ef3edb950 msgr2=0x563ef3eda260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.116+0000 7f1e7965c740 1 -- 172.21.15.28:0/3383505297 shutdown_connections 2024-04-25T01:41:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.116+0000 7f1e7965c740 1 -- 172.21.15.28:0/3383505297 wait complete. 2024-04-25T01:41:33.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.116+0000 7f1e7965c740 20 client.18837 trim_cache size 0 max 16384 2024-04-25T01:41:33.111 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:41:33.111 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:41:33.111 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-25T01:41:33.111 INFO:teuthology.orchestra.run.smithi028.stdout: "0657ac0c-ad74-4b49-902a-abc291c11e5b": { 2024-04-25T01:41:33.111 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-25T01:41:33.111 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "0657ac0c-ad74-4b49-902a-abc291c11e5b", 2024-04-25T01:41:33.111 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-25T01:41:33.111 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:41:33.112 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:41:33.112 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:41:33.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.117+0000 7f1e7694f640 1 -- 172.21.15.28:0/2423374959 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1e580a2d00 msgr2=0x7f1e580a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:33.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.117+0000 7f1e7694f640 1 --2- 172.21.15.28:0/2423374959 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1e580a2d00 0x7f1e580a51c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1e70077c10 tx=0x7f1e6400a830 comp rx=0 tx=0).stop 2024-04-25T01:41:33.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.117+0000 7f1e7694f640 1 -- 172.21.15.28:0/2423374959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e70107730 msgr2=0x7f1e7019c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:41:33.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.117+0000 7f1e7694f640 1 --2- 172.21.15.28:0/2423374959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e70107730 0x7f1e7019c820 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f1e68005cc0 tx=0x7f1e68003970 comp rx=0 tx=0).stop 2024-04-25T01:41:33.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.117+0000 7f1e7694f640 1 -- 172.21.15.28:0/2423374959 shutdown_connections 2024-04-25T01:41:33.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.117+0000 7f1e7694f640 1 --2- 172.21.15.28:0/2423374959 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1e580a2d00 0x7f1e580a51c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.117+0000 7f1e7694f640 1 --2- 172.21.15.28:0/2423374959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e70107730 0x7f1e7019c820 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.117+0000 7f1e7694f640 1 --2- 172.21.15.28:0/2423374959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e7006a780 0x7f1e701984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.117+0000 7f1e7694f640 1 --2- 172.21.15.28:0/2423374959 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1e70069dc0 0x7f1e70197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:41:33.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.117+0000 7f1e7694f640 1 -- 172.21.15.28:0/2423374959 >> 172.21.15.28:0/2423374959 conn(0x7f1e700776a0 msgr2=0x7f1e700fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:41:33.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.117+0000 7f1e7694f640 1 -- 172.21.15.28:0/2423374959 shutdown_connections 2024-04-25T01:41:33.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:41:33.117+0000 7f1e7694f640 1 -- 172.21.15.28:0/2423374959 wait complete. 2024-04-25T01:41:33.115 INFO:tasks.cephfs.filesystem:scrub status for tag:0657ac0c-ad74-4b49-902a-abc291c11e5b - {'path': '/', 'tag': '0657ac0c-ad74-4b49-902a-abc291c11e5b', 'options': 'recursive,force'} 2024-04-25T01:41:34.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:33 smithi130 ceph-mon[32702]: pgmap v263: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:41:34.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:33 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:41:34.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:33 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:41:34.218 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:33 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:41:34.219 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:33 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:41:34.219 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:33 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:41:34.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:33 smithi032 ceph-mon[32461]: pgmap v263: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:41:34.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:33 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:41:34.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:33 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:41:34.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:33 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:41:34.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:33 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:41:34.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:33 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:41:34.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:33 smithi028 ceph-mon[29390]: pgmap v263: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:41:34.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:33 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:41:34.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:33 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:41:34.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:33 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:41:34.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:33 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:41:34.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:33 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:41:36.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:35 smithi032 ceph-mon[32461]: pgmap v264: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:41:36.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:35 smithi028 ceph-mon[29390]: pgmap v264: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:41:36.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:35 smithi130 ceph-mon[32702]: pgmap v264: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:41:38.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:37 smithi032 ceph-mon[32461]: pgmap v265: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-04-25T01:41:38.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:37 smithi028 ceph-mon[29390]: pgmap v265: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-04-25T01:41:38.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:37 smithi130 ceph-mon[32702]: pgmap v265: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-04-25T01:41:40.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:39 smithi032 ceph-mon[32461]: pgmap v266: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T01:41:40.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:39 smithi028 ceph-mon[29390]: pgmap v266: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T01:41:40.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:39 smithi130 ceph-mon[32702]: pgmap v266: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-04-25T01:41:42.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:41 smithi032 ceph-mon[32461]: pgmap v267: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:41:42.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:41 smithi028 ceph-mon[29390]: pgmap v267: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:41:42.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:41 smithi130 ceph-mon[32702]: pgmap v267: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:41:44.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:43 smithi032 ceph-mon[32461]: pgmap v268: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-04-25T01:41:44.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:43 smithi028 ceph-mon[29390]: pgmap v268: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-04-25T01:41:44.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:43 smithi130 ceph-mon[32702]: pgmap v268: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-04-25T01:41:46.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:45 smithi032 ceph-mon[32461]: pgmap v269: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:41:46.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:45 smithi028 ceph-mon[29390]: pgmap v269: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:41:46.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:45 smithi130 ceph-mon[32702]: pgmap v269: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-04-25T01:41:48.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:47 smithi032 ceph-mon[32461]: pgmap v270: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-04-25T01:41:48.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:47 smithi028 ceph-mon[29390]: pgmap v270: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-04-25T01:41:48.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:47 smithi130 ceph-mon[32702]: pgmap v270: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-04-25T01:41:50.243 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:50 smithi032 ceph-mon[32461]: pgmap v271: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-25T01:41:50.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:50 smithi028 ceph-mon[29390]: pgmap v271: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-25T01:41:50.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:50 smithi130 ceph-mon[32702]: pgmap v271: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-25T01:41:52.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:52 smithi032 ceph-mon[32461]: pgmap v272: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:41:52.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:52 smithi028 ceph-mon[29390]: pgmap v272: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:41:52.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:52 smithi130 ceph-mon[32702]: pgmap v272: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:41:54.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:54 smithi028 ceph-mon[29390]: pgmap v273: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:41:54.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:54 smithi130 ceph-mon[32702]: pgmap v273: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:41:54.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:54 smithi032 ceph-mon[32461]: pgmap v273: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:41:56.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:56 smithi028 ceph-mon[29390]: pgmap v274: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:41:56.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:56 smithi130 ceph-mon[32702]: pgmap v274: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:41:56.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:56 smithi032 ceph-mon[32461]: pgmap v274: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:41:58.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:41:58 smithi028 ceph-mon[29390]: pgmap v275: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:41:58.468 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:41:58 smithi130 ceph-mon[32702]: pgmap v275: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:41:58.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:41:58 smithi032 ceph-mon[32461]: pgmap v275: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:42:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:00 smithi028 ceph-mon[29390]: pgmap v276: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:42:00.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:00 smithi130 ceph-mon[32702]: pgmap v276: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:42:00.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:00 smithi032 ceph-mon[32461]: pgmap v276: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:42:02.271 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:02 smithi130 ceph-mon[32702]: pgmap v277: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:42:02.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:02 smithi028 ceph-mon[29390]: pgmap v277: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:42:02.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:02 smithi032 ceph-mon[32461]: pgmap v277: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:42:03.116 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:42:03.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:42:03.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.246+0000 7ff104231640 1 -- 172.21.15.28:0/2509033854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0fc1066b0 msgr2=0x7ff0fc106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.246+0000 7ff104231640 1 --2- 172.21.15.28:0/2509033854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0fc1066b0 0x7ff0fc106a90 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8009930 tx=0x7ff0f802a420 comp rx=0 tx=0).stop 2024-04-25T01:42:03.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.247+0000 7ff104231640 1 -- 172.21.15.28:0/2509033854 shutdown_connections 2024-04-25T01:42:03.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.247+0000 7ff104231640 1 --2- 172.21.15.28:0/2509033854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0fc100fd0 0x7ff0fc10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.247+0000 7ff104231640 1 --2- 172.21.15.28:0/2509033854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0fc100610 0x7ff0fc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.247+0000 7ff104231640 1 --2- 172.21.15.28:0/2509033854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0fc1066b0 0x7ff0fc106a90 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.247+0000 7ff104231640 1 -- 172.21.15.28:0/2509033854 >> 172.21.15.28:0/2509033854 conn(0x7ff0fc077740 msgr2=0x7ff0fc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.247+0000 7ff104231640 1 -- 172.21.15.28:0/2509033854 shutdown_connections 2024-04-25T01:42:03.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.247+0000 7ff104231640 1 -- 172.21.15.28:0/2509033854 wait complete. 2024-04-25T01:42:03.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.247+0000 7ff104231640 1 Processor -- start 2024-04-25T01:42:03.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.247+0000 7ff104231640 1 -- start start 2024-04-25T01:42:03.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.247+0000 7ff104231640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0fc100610 0x7ff0fc198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff104231640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0fc100fd0 0x7ff0fc1992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff104231640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0fc1066b0 0x7ff0fc1a3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff104231640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc110a30 con 0x7ff0fc1066b0 2024-04-25T01:42:03.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff104231640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc1108b0 con 0x7ff0fc100610 2024-04-25T01:42:03.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff104231640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc110bb0 con 0x7ff0fc100fd0 2024-04-25T01:42:03.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff1027ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0fc1066b0 0x7ff0fc1a3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff1027ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0fc1066b0 0x7ff0fc1a3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45696/0 (socket says 172.21.15.28:45696) 2024-04-25T01:42:03.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff1027ce640 1 -- 172.21.15.28:0/2605095755 learned_addr learned my addr 172.21.15.28:0/2605095755 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:03.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff1017cc640 1 --2- 172.21.15.28:0/2605095755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0fc100fd0 0x7ff0fc1992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff101fcd640 1 --2- 172.21.15.28:0/2605095755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0fc100610 0x7ff0fc198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff1027ce640 1 -- 172.21.15.28:0/2605095755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0fc100fd0 msgr2=0x7ff0fc1992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff1027ce640 1 --2- 172.21.15.28:0/2605095755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0fc100fd0 0x7ff0fc1992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff1027ce640 1 -- 172.21.15.28:0/2605095755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0fc100610 msgr2=0x7ff0fc198d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff1027ce640 1 --2- 172.21.15.28:0/2605095755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0fc100610 0x7ff0fc198d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff1027ce640 1 -- 172.21.15.28:0/2605095755 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0fc1a4330 con 0x7ff0fc1066b0 2024-04-25T01:42:03.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff101fcd640 1 --2- 172.21.15.28:0/2605095755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0fc100610 0x7ff0fc198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:03.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff1027ce640 1 --2- 172.21.15.28:0/2605095755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0fc1066b0 0x7ff0fc1a3df0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400eaa0 tx=0x7ff0f400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff0f2ffd640 1 -- 172.21.15.28:0/2605095755 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4004d20 con 0x7ff0fc1066b0 2024-04-25T01:42:03.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff0f2ffd640 1 -- 172.21.15.28:0/2605095755 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0f4005020 con 0x7ff0fc1066b0 2024-04-25T01:42:03.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff104231640 1 -- 172.21.15.28:0/2605095755 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0fc1a4500 con 0x7ff0fc1066b0 2024-04-25T01:42:03.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff0f2ffd640 1 -- 172.21.15.28:0/2605095755 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f40116d0 con 0x7ff0fc1066b0 2024-04-25T01:42:03.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.248+0000 7ff104231640 1 -- 172.21.15.28:0/2605095755 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0fc1a4a20 con 0x7ff0fc1066b0 2024-04-25T01:42:03.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.250+0000 7ff0f2ffd640 1 -- 172.21.15.28:0/2605095755 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff0f40051d0 con 0x7ff0fc1066b0 2024-04-25T01:42:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.251+0000 7ff10701c740 1 Processor -- start 2024-04-25T01:42:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.251+0000 7ff10701c740 1 -- start start 2024-04-25T01:42:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.251+0000 7ff10701c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581edde2510 0x5581edde28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.251+0000 7ff10701c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2e30 0x5581eddf14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.251+0000 7ff10701c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 0x5581eddf3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.251+0000 7ff10701c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5581edd476b0 con 0x5581eddf1a30 2024-04-25T01:42:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.251+0000 7ff10701c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5581edcef880 con 0x5581edde2e30 2024-04-25T01:42:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.251+0000 7ff10701c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5581edd43670 con 0x5581edde2510 2024-04-25T01:42:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.251+0000 7ff0f2ffd640 1 --2- 172.21.15.28:0/2605095755 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff0e80a2d00 0x7ff0e80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.251+0000 7ff1017cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 0x5581eddf3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.251+0000 7ff1017cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 0x5581eddf3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45708/0 (socket says 172.21.15.28:45708) 2024-04-25T01:42:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.251+0000 7ff1017cc640 1 -- 172.21.15.28:0/296825701 learned_addr learned my addr 172.21.15.28:0/296825701 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.252+0000 7ff101fcd640 1 --2- 172.21.15.28:0/296825701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2e30 0x5581eddf14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.252+0000 7ff1017cc640 1 --2- 172.21.15.28:0/296825701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581edde2510 0x5581edde28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.252+0000 7ff1027ce640 1 --2- 172.21.15.28:0/2605095755 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff0e80a2d00 0x7ff0e80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.252+0000 7ff0f2ffd640 1 -- 172.21.15.28:0/2605095755 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7ff0f40cf9d0 con 0x7ff0fc1066b0 2024-04-25T01:42:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.252+0000 7ff1017cc640 1 -- 172.21.15.28:0/296825701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581edde2510 msgr2=0x5581edde28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.252+0000 7ff1017cc640 1 --2- 172.21.15.28:0/296825701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581edde2510 0x5581edde28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.252+0000 7ff1017cc640 1 -- 172.21.15.28:0/296825701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2e30 msgr2=0x5581eddf14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.252+0000 7ff1017cc640 1 --2- 172.21.15.28:0/296825701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2e30 0x5581eddf14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.252+0000 7ff1017cc640 1 -- 172.21.15.28:0/296825701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581edb257d0 con 0x5581eddf1a30 2024-04-25T01:42:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.252+0000 7ff1017cc640 1 --2- 172.21.15.28:0/296825701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 0x5581eddf3e20 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec0117e0 tx=0x7ff0ec00a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68ed735bd6acded2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.252+0000 7ff1027ce640 1 --2- 172.21.15.28:0/2605095755 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff0e80a2d00 0x7ff0e80a51c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff0f40619e0 tx=0x7ff0f4086040 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.252+0000 7ff0e7fff640 1 -- 172.21.15.28:0/296825701 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec011e20 con 0x5581eddf1a30 2024-04-25T01:42:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.252+0000 7ff0e7fff640 1 -- 172.21.15.28:0/296825701 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0ec005020 con 0x5581eddf1a30 2024-04-25T01:42:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.252+0000 7ff0e7fff640 1 -- 172.21.15.28:0/296825701 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec004b20 con 0x5581eddf1a30 2024-04-25T01:42:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.253+0000 7ff10701c740 1 -- 172.21.15.28:0/296825701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 msgr2=0x5581eddf3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.253+0000 7ff10701c740 1 --2- 172.21.15.28:0/296825701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 0x5581eddf3e20 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec0117e0 tx=0x7ff0ec00a760 comp rx=0 tx=0).stop 2024-04-25T01:42:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.253+0000 7ff10701c740 1 -- 172.21.15.28:0/296825701 shutdown_connections 2024-04-25T01:42:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.253+0000 7ff10701c740 1 --2- 172.21.15.28:0/296825701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 0x5581eddf3e20 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.253+0000 7ff10701c740 1 --2- 172.21.15.28:0/296825701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2e30 0x5581eddf14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.253+0000 7ff10701c740 1 --2- 172.21.15.28:0/296825701 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581edde2510 0x5581edde28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.253+0000 7ff10701c740 1 -- 172.21.15.28:0/296825701 >> 172.21.15.28:0/296825701 conn(0x5581eddde950 msgr2=0x5581eddded30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.253+0000 7ff10701c740 1 -- 172.21.15.28:0/296825701 shutdown_connections 2024-04-25T01:42:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.253+0000 7ff10701c740 1 -- 172.21.15.28:0/296825701 wait complete. 2024-04-25T01:42:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.253+0000 7ff10701c740 1 -- start start 2024-04-25T01:42:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.254+0000 7ff10701c740 1 Processor -- start 2024-04-25T01:42:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.254+0000 7ff10701c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2510 0x5581eddee9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.254+0000 7ff10701c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581edde2e30 0x5581eddeef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.254+0000 7ff10701c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 0x5581eddef440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.254+0000 7ff10701c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5581eddf4870 con 0x5581eddf1a30 2024-04-25T01:42:03.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.254+0000 7ff10701c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5581edd02f50 con 0x5581edde2510 2024-04-25T01:42:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.254+0000 7ff10701c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5581eddf49f0 con 0x5581edde2e30 2024-04-25T01:42:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.254+0000 7ff10701c740 10 client.0 _ll_register_callbacks cb 0x5581edd3e000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.254+0000 7ff1017cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 0x5581eddef440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.254+0000 7ff1017cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 0x5581eddef440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45710/0 (socket says 172.21.15.28:45710) 2024-04-25T01:42:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.254+0000 7ff1017cc640 1 -- 172.21.15.28:0/4122729269 learned_addr learned my addr 172.21.15.28:0/4122729269 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.254+0000 7ff1017cc640 1 --2- 172.21.15.28:0/4122729269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2510 0x5581eddee9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.254+0000 7ff101fcd640 1 --2- 172.21.15.28:0/4122729269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581edde2e30 0x5581eddeef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.255+0000 7ff1017cc640 1 -- 172.21.15.28:0/4122729269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581edde2e30 msgr2=0x5581eddeef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.255+0000 7ff1017cc640 1 --2- 172.21.15.28:0/4122729269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581edde2e30 0x5581eddeef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.255+0000 7ff1017cc640 1 -- 172.21.15.28:0/4122729269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2510 msgr2=0x5581eddee9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.255+0000 7ff1017cc640 1 --2- 172.21.15.28:0/4122729269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2510 0x5581eddee9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.255+0000 7ff1017cc640 1 -- 172.21.15.28:0/4122729269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581eda5b3f0 con 0x5581eddf1a30 2024-04-25T01:42:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.255+0000 7ff1017cc640 1 --2- 172.21.15.28:0/4122729269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 0x5581eddef440 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec011f10 tx=0x7ff0ec013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bd456a353b76605 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.255+0000 7ff0e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.255+0000 7ff0e57fa640 1 -- 172.21.15.28:0/4122729269 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec00f9d0 con 0x5581eddf1a30 2024-04-25T01:42:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.255+0000 7ff0e57fa640 1 -- 172.21.15.28:0/4122729269 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7ff0ec00fb70 con 0x5581eddf1a30 2024-04-25T01:42:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.255+0000 7ff10701c740 1 -- 172.21.15.28:0/4122729269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0ec005020 con 0x5581eddf1a30 2024-04-25T01:42:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.255+0000 7ff0e57fa640 1 -- 172.21.15.28:0/4122729269 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff0ec00fd70 con 0x5581eddf1a30 2024-04-25T01:42:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.255+0000 7ff10701c740 10 client.18849 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.255+0000 7ff10701c740 1 -- 172.21.15.28:0/4122729269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5581eda1b0f0 con 0x5581eddf1a30 2024-04-25T01:42:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.255+0000 7ff0e57fa640 1 -- 172.21.15.28:0/4122729269 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0ec005020 con 0x5581eddf1a30 2024-04-25T01:42:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.256+0000 7ff0e57fa640 1 -- 172.21.15.28:0/4122729269 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff0ec01e740 con 0x5581eddf1a30 2024-04-25T01:42:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.256+0000 7ff10701c740 10 client.18849 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.256+0000 7ff10701c740 10 client.18849 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.256+0000 7ff10701c740 20 client.18849 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.256+0000 7ff10701c740 1 --2- 172.21.15.28:0/4122729269 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5581ede01c60 0x5581ede04050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.256+0000 7ff10701c740 4 client.18849 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.256+0000 7ff10701c740 1 -- 172.21.15.28:0/4122729269 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581edd54bf0 con 0x5581ede01c60 2024-04-25T01:42:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.256+0000 7ff101fcd640 1 --2- 172.21.15.28:0/4122729269 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5581ede01c60 0x5581ede04050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.256+0000 7ff0e57fa640 10 client.18849 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.256+0000 7ff101fcd640 1 --2- 172.21.15.28:0/4122729269 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5581ede01c60 0x5581ede04050 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a260b4206c470bd server_cookie=f90a653bced2c8ef in_seq=0 out_seq=0 2024-04-25T01:42:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.257+0000 7ff0e57fa640 1 -- 172.21.15.28:0/4122729269 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5581edd54bf0 con 0x5581ede01c60 2024-04-25T01:42:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.257+0000 7ff0e57fa640 10 client.18849 handle_command_reply: tid=0 2024-04-25T01:42:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.257+0000 7ff10701c740 1 client.18849 shutdown 2024-04-25T01:42:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.257+0000 7ff10701c740 1 -- 172.21.15.28:0/4122729269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 msgr2=0x5581eddef440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.257+0000 7ff10701c740 1 --2- 172.21.15.28:0/4122729269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 0x5581eddef440 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec011f10 tx=0x7ff0ec013040 comp rx=0 tx=0).stop 2024-04-25T01:42:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.257+0000 7ff10701c740 1 -- 172.21.15.28:0/4122729269 shutdown_connections 2024-04-25T01:42:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.257+0000 7ff10701c740 1 --2- 172.21.15.28:0/4122729269 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5581ede01c60 0x5581ede04050 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.257+0000 7ff10701c740 1 --2- 172.21.15.28:0/4122729269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 0x5581eddef440 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.257+0000 7ff10701c740 1 --2- 172.21.15.28:0/4122729269 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581edde2e30 0x5581eddeef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.257+0000 7ff10701c740 1 --2- 172.21.15.28:0/4122729269 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2510 0x5581eddee9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.257+0000 7ff10701c740 1 -- 172.21.15.28:0/4122729269 >> 172.21.15.28:0/4122729269 conn(0x5581eddde950 msgr2=0x5581edde4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.257+0000 7ff0e57fa640 0 client.18849 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.257+0000 7ff10701c740 1 -- 172.21.15.28:0/4122729269 shutdown_connections 2024-04-25T01:42:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.257+0000 7ff10701c740 1 -- 172.21.15.28:0/4122729269 wait complete. 2024-04-25T01:42:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.258+0000 7ff10701c740 20 client.18849 trim_cache size 0 max 16384 2024-04-25T01:42:03.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.275+0000 7ff10701c740 1 Processor -- start 2024-04-25T01:42:03.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.275+0000 7ff10701c740 1 -- start start 2024-04-25T01:42:03.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.275+0000 7ff10701c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2510 0x5581eddf10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.275+0000 7ff10701c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581edde2e30 0x5581eddfd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.275+0000 7ff10701c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 0x5581eddddaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.275+0000 7ff10701c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc1ac9f0 con 0x5581eddf1a30 2024-04-25T01:42:03.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.275+0000 7ff10701c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc1ae530 con 0x5581edde2510 2024-04-25T01:42:03.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.275+0000 7ff10701c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc1ad0c0 con 0x5581edde2e30 2024-04-25T01:42:03.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.275+0000 7ff1017cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 0x5581eddddaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.275+0000 7ff1017cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 0x5581eddddaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45712/0 (socket says 172.21.15.28:45712) 2024-04-25T01:42:03.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.275+0000 7ff1017cc640 1 -- 172.21.15.28:0/2918049844 learned_addr learned my addr 172.21.15.28:0/2918049844 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:03.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.276+0000 7ff101fcd640 1 --2- 172.21.15.28:0/2918049844 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581edde2e30 0x5581eddfd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.276+0000 7ff1017cc640 1 --2- 172.21.15.28:0/2918049844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2510 0x5581eddf10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.276+0000 7ff1017cc640 1 -- 172.21.15.28:0/2918049844 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581edde2e30 msgr2=0x5581eddfd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.276+0000 7ff1017cc640 1 --2- 172.21.15.28:0/2918049844 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581edde2e30 0x5581eddfd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.276+0000 7ff1017cc640 1 -- 172.21.15.28:0/2918049844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2510 msgr2=0x5581eddf10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.276+0000 7ff1017cc640 1 --2- 172.21.15.28:0/2918049844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2510 0x5581eddf10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.276+0000 7ff1017cc640 1 -- 172.21.15.28:0/2918049844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581edde32f0 con 0x5581eddf1a30 2024-04-25T01:42:03.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.276+0000 7ff1017cc640 1 --2- 172.21.15.28:0/2918049844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 0x5581eddddaf0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x5581ede042f0 tx=0x7ff0ec00a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df1f914f267f24d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.276+0000 7ff0e77fe640 1 -- 172.21.15.28:0/2918049844 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec004ea0 con 0x5581eddf1a30 2024-04-25T01:42:03.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.276+0000 7ff0e77fe640 1 -- 172.21.15.28:0/2918049844 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0ec01e950 con 0x5581eddf1a30 2024-04-25T01:42:03.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.276+0000 7ff0e77fe640 1 -- 172.21.15.28:0/2918049844 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec0053e0 con 0x5581eddf1a30 2024-04-25T01:42:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.277+0000 7ff10701c740 1 -- 172.21.15.28:0/2918049844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 msgr2=0x5581eddddaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.277+0000 7ff10701c740 1 --2- 172.21.15.28:0/2918049844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 0x5581eddddaf0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x5581ede042f0 tx=0x7ff0ec00a2d0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.277+0000 7ff10701c740 1 -- 172.21.15.28:0/2918049844 shutdown_connections 2024-04-25T01:42:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.277+0000 7ff10701c740 1 --2- 172.21.15.28:0/2918049844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581eddf1a30 0x5581eddddaf0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.277+0000 7ff10701c740 1 --2- 172.21.15.28:0/2918049844 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581edde2e30 0x5581eddfd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.277+0000 7ff10701c740 1 --2- 172.21.15.28:0/2918049844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2510 0x5581eddf10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.277+0000 7ff10701c740 1 -- 172.21.15.28:0/2918049844 >> 172.21.15.28:0/2918049844 conn(0x5581eddde950 msgr2=0x5581eddee980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.277+0000 7ff10701c740 1 -- 172.21.15.28:0/2918049844 shutdown_connections 2024-04-25T01:42:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.277+0000 7ff10701c740 1 -- 172.21.15.28:0/2918049844 wait complete. 2024-04-25T01:42:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.277+0000 7ff10701c740 1 -- start start 2024-04-25T01:42:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.277+0000 7ff10701c740 1 Processor -- start 2024-04-25T01:42:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff10701c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2510 0x5581eddf9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff10701c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581edde2e30 0x5581eddfa240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff10701c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581eddf1a30 0x5581eddf5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff10701c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc1ade60 con 0x5581edde2e30 2024-04-25T01:42:03.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff10701c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc1ad790 con 0x5581edde2510 2024-04-25T01:42:03.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff10701c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc1aa9f0 con 0x5581eddf1a30 2024-04-25T01:42:03.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff10701c740 10 client.0 _ll_register_callbacks cb 0x7ff0f8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:03.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff101fcd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581edde2e30 0x5581eddfa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff101fcd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581edde2e30 0x5581eddfa240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45728/0 (socket says 172.21.15.28:45728) 2024-04-25T01:42:03.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff101fcd640 1 -- 172.21.15.28:0/1147540443 learned_addr learned my addr 172.21.15.28:0/1147540443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:03.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff1017cc640 1 --2- 172.21.15.28:0/1147540443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2510 0x5581eddf9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff101fcd640 1 -- 172.21.15.28:0/1147540443 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581eddf1a30 msgr2=0x5581eddf5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff101fcd640 1 --2- 172.21.15.28:0/1147540443 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581eddf1a30 0x5581eddf5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff101fcd640 1 -- 172.21.15.28:0/1147540443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2510 msgr2=0x5581eddf9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff101fcd640 1 --2- 172.21.15.28:0/1147540443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2510 0x5581eddf9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff101fcd640 1 -- 172.21.15.28:0/1147540443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581eddf3e20 con 0x5581edde2e30 2024-04-25T01:42:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff1017cc640 1 --2- 172.21.15.28:0/1147540443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2510 0x5581eddf9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff101fcd640 1 --2- 172.21.15.28:0/1147540443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581edde2e30 0x5581eddfa240 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8032fd0 tx=0x7ff0f8004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88b5a263769c3309 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff0e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff0e57fa640 1 -- 172.21.15.28:0/1147540443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8035040 con 0x5581edde2e30 2024-04-25T01:42:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff0e57fa640 1 -- 172.21.15.28:0/1147540443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7ff0f8005040 con 0x5581edde2e30 2024-04-25T01:42:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff10701c740 1 -- 172.21.15.28:0/1147540443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5581edd4ef40 con 0x5581edde2e30 2024-04-25T01:42:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff0e57fa640 1 -- 172.21.15.28:0/1147540443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff0f8054050 con 0x5581edde2e30 2024-04-25T01:42:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff10701c740 10 client.18855 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.278+0000 7ff10701c740 1 -- 172.21.15.28:0/1147540443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5581ede67af0 con 0x5581edde2e30 2024-04-25T01:42:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.279+0000 7ff0e57fa640 1 -- 172.21.15.28:0/1147540443 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5581edd4ef40 con 0x5581edde2e30 2024-04-25T01:42:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.279+0000 7ff0e57fa640 1 -- 172.21.15.28:0/1147540443 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff0f80046c0 con 0x5581edde2e30 2024-04-25T01:42:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.279+0000 7ff10701c740 10 client.18855 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.279+0000 7ff10701c740 10 client.18855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.279+0000 7ff10701c740 20 client.18855 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.279+0000 7ff10701c740 1 --2- 172.21.15.28:0/1147540443 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5581ede682a0 0x5581ede6a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.279+0000 7ff10701c740 4 client.18855 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:42:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.279+0000 7ff10701c740 1 -- 172.21.15.28:0/1147540443 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5581edccc6f0 con 0x5581ede682a0 2024-04-25T01:42:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.279+0000 7ff1017cc640 1 --2- 172.21.15.28:0/1147540443 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5581ede682a0 0x5581ede6a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.280+0000 7ff1017cc640 1 --2- 172.21.15.28:0/1147540443 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5581ede682a0 0x5581ede6a6c0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5366500cf68b26e7 server_cookie=c3d77f9ae29a4b88 in_seq=0 out_seq=0 2024-04-25T01:42:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.280+0000 7ff0e57fa640 10 client.18855 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.280+0000 7ff0e57fa640 1 -- 172.21.15.28:0/1147540443 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5581edccc6f0 con 0x5581ede682a0 2024-04-25T01:42:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.280+0000 7ff0e57fa640 10 client.18855 handle_command_reply: tid=0 2024-04-25T01:42:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.280+0000 7ff10701c740 1 client.18855 shutdown 2024-04-25T01:42:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.280+0000 7ff10701c740 1 -- 172.21.15.28:0/1147540443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581edde2e30 msgr2=0x5581eddfa240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.280+0000 7ff10701c740 1 --2- 172.21.15.28:0/1147540443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581edde2e30 0x5581eddfa240 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8032fd0 tx=0x7ff0f8004360 comp rx=0 tx=0).stop 2024-04-25T01:42:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.280+0000 7ff10701c740 1 -- 172.21.15.28:0/1147540443 shutdown_connections 2024-04-25T01:42:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.280+0000 7ff10701c740 1 --2- 172.21.15.28:0/1147540443 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5581ede682a0 0x5581ede6a6c0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.280+0000 7ff10701c740 1 --2- 172.21.15.28:0/1147540443 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5581eddf1a30 0x5581eddf5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.280+0000 7ff0e57fa640 0 client.18855 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:03.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.280+0000 7ff10701c740 1 --2- 172.21.15.28:0/1147540443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581edde2e30 0x5581eddfa240 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.280+0000 7ff10701c740 1 --2- 172.21.15.28:0/1147540443 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5581edde2510 0x5581eddf9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.280+0000 7ff10701c740 1 -- 172.21.15.28:0/1147540443 >> 172.21.15.28:0/1147540443 conn(0x5581eddde950 msgr2=0x5581edddd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.280+0000 7ff10701c740 1 -- 172.21.15.28:0/1147540443 shutdown_connections 2024-04-25T01:42:03.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.280+0000 7ff10701c740 1 -- 172.21.15.28:0/1147540443 wait complete. 2024-04-25T01:42:03.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.280+0000 7ff10701c740 20 client.18855 trim_cache size 0 max 16384 2024-04-25T01:42:03.275 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:42:03.275 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:42:03.276 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:42:03.276 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:42:03.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.281+0000 7ff104231640 1 -- 172.21.15.28:0/2605095755 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff0e80a2d00 msgr2=0x7ff0e80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.281+0000 7ff104231640 1 --2- 172.21.15.28:0/2605095755 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff0e80a2d00 0x7ff0e80a51c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff0f40619e0 tx=0x7ff0f4086040 comp rx=0 tx=0).stop 2024-04-25T01:42:03.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.281+0000 7ff104231640 1 -- 172.21.15.28:0/2605095755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0fc1066b0 msgr2=0x7ff0fc1a3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.281+0000 7ff104231640 1 --2- 172.21.15.28:0/2605095755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0fc1066b0 0x7ff0fc1a3df0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400eaa0 tx=0x7ff0f400eea0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.281+0000 7ff104231640 1 -- 172.21.15.28:0/2605095755 shutdown_connections 2024-04-25T01:42:03.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.281+0000 7ff104231640 1 --2- 172.21.15.28:0/2605095755 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff0e80a2d00 0x7ff0e80a51c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.281+0000 7ff104231640 1 --2- 172.21.15.28:0/2605095755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0fc1066b0 0x7ff0fc1a3df0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.281+0000 7ff104231640 1 --2- 172.21.15.28:0/2605095755 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff0fc100fd0 0x7ff0fc1992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.281+0000 7ff104231640 1 --2- 172.21.15.28:0/2605095755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff0fc100610 0x7ff0fc198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.281+0000 7ff104231640 1 -- 172.21.15.28:0/2605095755 >> 172.21.15.28:0/2605095755 conn(0x7ff0fc077740 msgr2=0x7ff0fc0ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.281+0000 7ff104231640 1 -- 172.21.15.28:0/2605095755 shutdown_connections 2024-04-25T01:42:03.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.281+0000 7ff104231640 1 -- 172.21.15.28:0/2605095755 wait complete. 2024-04-25T01:42:03.279 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:42:03.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:42:03.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.395+0000 7f9145c9e640 1 -- 172.21.15.28:0/475213509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91401017c0 msgr2=0x7f9140101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.395+0000 7f9145c9e640 1 --2- 172.21.15.28:0/475213509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91401017c0 0x7f9140101c20 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f9134009900 tx=0x7f913402a480 comp rx=0 tx=0).stop 2024-04-25T01:42:03.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.395+0000 7f9145c9e640 1 -- 172.21.15.28:0/475213509 shutdown_connections 2024-04-25T01:42:03.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.395+0000 7f9145c9e640 1 --2- 172.21.15.28:0/475213509 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9140102160 0x7f914010e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.395+0000 7f9145c9e640 1 --2- 172.21.15.28:0/475213509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91401017c0 0x7f9140101c20 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.395+0000 7f9145c9e640 1 --2- 172.21.15.28:0/475213509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9140107750 0x7f9140107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.395+0000 7f9145c9e640 1 -- 172.21.15.28:0/475213509 >> 172.21.15.28:0/475213509 conn(0x7f91400788d0 msgr2=0x7f9140078ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.395+0000 7f9145c9e640 1 -- 172.21.15.28:0/475213509 shutdown_connections 2024-04-25T01:42:03.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.395+0000 7f9145c9e640 1 -- 172.21.15.28:0/475213509 wait complete. 2024-04-25T01:42:03.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.395+0000 7f9145c9e640 1 Processor -- start 2024-04-25T01:42:03.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.396+0000 7f9145c9e640 1 -- start start 2024-04-25T01:42:03.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.396+0000 7f9145c9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91401017c0 0x7f9140073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.396+0000 7f9145c9e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9140102160 0x7f9140073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.396+0000 7f9145c9e640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9140107750 0x7f91400744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.396+0000 7f9145c9e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9140111b60 con 0x7f91401017c0 2024-04-25T01:42:03.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.396+0000 7f9145c9e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f91401119e0 con 0x7f9140102160 2024-04-25T01:42:03.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.396+0000 7f9145c9e640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f9140111ce0 con 0x7f9140107750 2024-04-25T01:42:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.396+0000 7f913f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91401017c0 0x7f9140073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.396+0000 7f913f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91401017c0 0x7f9140073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45740/0 (socket says 172.21.15.28:45740) 2024-04-25T01:42:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.396+0000 7f913f7fe640 1 -- 172.21.15.28:0/3772072508 learned_addr learned my addr 172.21.15.28:0/3772072508 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.396+0000 7f913effd640 1 --2- 172.21.15.28:0/3772072508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9140102160 0x7f9140073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.396+0000 7f913ffff640 1 --2- 172.21.15.28:0/3772072508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9140107750 0x7f91400744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.396+0000 7f913f7fe640 1 -- 172.21.15.28:0/3772072508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9140107750 msgr2=0x7f91400744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.396+0000 7f913f7fe640 1 --2- 172.21.15.28:0/3772072508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9140107750 0x7f91400744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.396+0000 7f913f7fe640 1 -- 172.21.15.28:0/3772072508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9140102160 msgr2=0x7f9140073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.396+0000 7f913f7fe640 1 --2- 172.21.15.28:0/3772072508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9140102160 0x7f9140073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.396+0000 7f913f7fe640 1 -- 172.21.15.28:0/3772072508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91401a6420 con 0x7f91401017c0 2024-04-25T01:42:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.396+0000 7f913effd640 1 --2- 172.21.15.28:0/3772072508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9140102160 0x7f9140073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.397+0000 7f913f7fe640 1 --2- 172.21.15.28:0/3772072508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91401017c0 0x7f9140073a40 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f913000d950 tx=0x7f913000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.397+0000 7f913cff9640 1 -- 172.21.15.28:0/3772072508 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9130017030 con 0x7f91401017c0 2024-04-25T01:42:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.397+0000 7f913cff9640 1 -- 172.21.15.28:0/3772072508 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9130004020 con 0x7f91401017c0 2024-04-25T01:42:03.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.397+0000 7f9145c9e640 1 -- 172.21.15.28:0/3772072508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91401a6710 con 0x7f91401017c0 2024-04-25T01:42:03.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.397+0000 7f913cff9640 1 -- 172.21.15.28:0/3772072508 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91300047a0 con 0x7f91401017c0 2024-04-25T01:42:03.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.397+0000 7f9145c9e640 1 -- 172.21.15.28:0/3772072508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91401a6c60 con 0x7f91401017c0 2024-04-25T01:42:03.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.398+0000 7f9145c9e640 1 -- 172.21.15.28:0/3772072508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91400689b0 con 0x7f91401017c0 2024-04-25T01:42:03.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.399+0000 7f913cff9640 1 -- 172.21.15.28:0/3772072508 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f91300041d0 con 0x7f91401017c0 2024-04-25T01:42:03.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.399+0000 7f913cff9640 1 --2- 172.21.15.28:0/3772072508 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f91280a2d30 0x7f91280a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.400+0000 7f913effd640 1 --2- 172.21.15.28:0/3772072508 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f91280a2d30 0x7f91280a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.400+0000 7f913cff9640 1 -- 172.21.15.28:0/3772072508 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f91300cf1f0 con 0x7f91401017c0 2024-04-25T01:42:03.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.400+0000 7f913effd640 1 --2- 172.21.15.28:0/3772072508 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f91280a2d30 0x7f91280a51f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9134005ab0 tx=0x7f9134000bc0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.404+0000 7f913cff9640 1 -- 172.21.15.28:0/3772072508 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f913009ba80 con 0x7f91401017c0 2024-04-25T01:42:03.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.627+0000 7f9145c9e640 1 -- 172.21.15.28:0/3772072508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9140074f20 con 0x7f91401017c0 2024-04-25T01:42:03.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.628+0000 7f913cff9640 1 -- 172.21.15.28:0/3772072508 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7f913009b380 con 0x7f91401017c0 2024-04-25T01:42:03.615 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:42:03.615 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:42:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:42:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.630+0000 7f9145c9e640 1 -- 172.21.15.28:0/3772072508 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f91280a2d30 msgr2=0x7f91280a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.630+0000 7f9145c9e640 1 --2- 172.21.15.28:0/3772072508 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f91280a2d30 0x7f91280a51f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9134005ab0 tx=0x7f9134000bc0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.630+0000 7f9145c9e640 1 -- 172.21.15.28:0/3772072508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91401017c0 msgr2=0x7f9140073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.630+0000 7f9145c9e640 1 --2- 172.21.15.28:0/3772072508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91401017c0 0x7f9140073a40 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f913000d950 tx=0x7f913000de10 comp rx=0 tx=0).stop 2024-04-25T01:42:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.630+0000 7f9145c9e640 1 -- 172.21.15.28:0/3772072508 shutdown_connections 2024-04-25T01:42:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.630+0000 7f9145c9e640 1 --2- 172.21.15.28:0/3772072508 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f91280a2d30 0x7f91280a51f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.630+0000 7f9145c9e640 1 --2- 172.21.15.28:0/3772072508 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f9140107750 0x7f91400744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.630+0000 7f9145c9e640 1 --2- 172.21.15.28:0/3772072508 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9140102160 0x7f9140073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.630+0000 7f9145c9e640 1 --2- 172.21.15.28:0/3772072508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91401017c0 0x7f9140073a40 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.630+0000 7f9145c9e640 1 -- 172.21.15.28:0/3772072508 >> 172.21.15.28:0/3772072508 conn(0x7f91400788d0 msgr2=0x7f91401003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.630+0000 7f9145c9e640 1 -- 172.21.15.28:0/3772072508 shutdown_connections 2024-04-25T01:42:03.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.630+0000 7f9145c9e640 1 -- 172.21.15.28:0/3772072508 wait complete. 2024-04-25T01:42:03.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:42:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.745+0000 7f2172f3b640 1 -- 172.21.15.28:0/148735043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f216c06a780 msgr2=0x7f216c10e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.745+0000 7f2172f3b640 1 --2- 172.21.15.28:0/148735043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f216c06a780 0x7f216c10e5f0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f2164009cd0 tx=0x7f216402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.745+0000 7f2172f3b640 1 -- 172.21.15.28:0/148735043 shutdown_connections 2024-04-25T01:42:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.745+0000 7f2172f3b640 1 --2- 172.21.15.28:0/148735043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f216c06a780 0x7f216c10e5f0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.745+0000 7f2172f3b640 1 --2- 172.21.15.28:0/148735043 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f216c069dc0 0x7f216c06a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.745+0000 7f2172f3b640 1 --2- 172.21.15.28:0/148735043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f216c107730 0x7f216c107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.745+0000 7f2172f3b640 1 -- 172.21.15.28:0/148735043 >> 172.21.15.28:0/148735043 conn(0x7f216c0776a0 msgr2=0x7f216c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.745+0000 7f2172f3b640 1 -- 172.21.15.28:0/148735043 shutdown_connections 2024-04-25T01:42:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.745+0000 7f2172f3b640 1 -- 172.21.15.28:0/148735043 wait complete. 2024-04-25T01:42:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.746+0000 7f2172f3b640 1 Processor -- start 2024-04-25T01:42:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.746+0000 7f2172f3b640 1 -- start start 2024-04-25T01:42:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.746+0000 7f2172f3b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f216c069dc0 0x7f216c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.746+0000 7f2172f3b640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f216c06a780 0x7f216c1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.746+0000 7f2172f3b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f216c107730 0x7f216c19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.746+0000 7f2172f3b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f216c111b90 con 0x7f216c107730 2024-04-25T01:42:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.746+0000 7f2172f3b640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f216c111a10 con 0x7f216c069dc0 2024-04-25T01:42:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.746+0000 7f2172f3b640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f216c111d10 con 0x7f216c06a780 2024-04-25T01:42:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.746+0000 7f21714d8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f216c107730 0x7f216c19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.746+0000 7f21714d8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f216c107730 0x7f216c19c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45762/0 (socket says 172.21.15.28:45762) 2024-04-25T01:42:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.746+0000 7f21714d8640 1 -- 172.21.15.28:0/3419177397 learned_addr learned my addr 172.21.15.28:0/3419177397 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.746+0000 7f2170cd7640 1 --2- 172.21.15.28:0/3419177397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f216c069dc0 0x7f216c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.746+0000 7f216bfff640 1 --2- 172.21.15.28:0/3419177397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f216c06a780 0x7f216c1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.746+0000 7f21714d8640 1 -- 172.21.15.28:0/3419177397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f216c06a780 msgr2=0x7f216c1984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.746+0000 7f21714d8640 1 --2- 172.21.15.28:0/3419177397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f216c06a780 0x7f216c1984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.747+0000 7f21714d8640 1 -- 172.21.15.28:0/3419177397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f216c069dc0 msgr2=0x7f216c197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.747+0000 7f21714d8640 1 --2- 172.21.15.28:0/3419177397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f216c069dc0 0x7f216c197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.747+0000 7f21714d8640 1 -- 172.21.15.28:0/3419177397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f216c19cf80 con 0x7f216c107730 2024-04-25T01:42:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.747+0000 7f2170cd7640 1 --2- 172.21.15.28:0/3419177397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f216c069dc0 0x7f216c197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:03.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.747+0000 7f21714d8640 1 --2- 172.21.15.28:0/3419177397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f216c107730 0x7f216c19c850 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f2164033040 tx=0x7f2164031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.747+0000 7f2169ffb640 1 -- 172.21.15.28:0/3419177397 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2164004040 con 0x7f216c107730 2024-04-25T01:42:03.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.747+0000 7f2169ffb640 1 -- 172.21.15.28:0/3419177397 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21640046c0 con 0x7f216c107730 2024-04-25T01:42:03.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.747+0000 7f2172f3b640 1 -- 172.21.15.28:0/3419177397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f216c19d210 con 0x7f216c107730 2024-04-25T01:42:03.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.747+0000 7f2169ffb640 1 -- 172.21.15.28:0/3419177397 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2164046480 con 0x7f216c107730 2024-04-25T01:42:03.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.747+0000 7f2172f3b640 1 -- 172.21.15.28:0/3419177397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f216c1a4b00 con 0x7f216c107730 2024-04-25T01:42:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.749+0000 7f2169ffb640 1 -- 172.21.15.28:0/3419177397 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f21640305a0 con 0x7f216c107730 2024-04-25T01:42:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.749+0000 7f2175c47740 1 Processor -- start 2024-04-25T01:42:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f2175c47740 1 -- start start 2024-04-25T01:42:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f2175c47740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a7601580 0x5576a7601960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f2175c47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a760c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f2175c47740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576a760c570 0x5576a760e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f2175c47740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5576a7566670 con 0x5576a7601ea0 2024-04-25T01:42:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f2175c47740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5576a750e970 con 0x5576a7601580 2024-04-25T01:42:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f2175c47740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5576a7562630 con 0x5576a760c570 2024-04-25T01:42:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f2170cd7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a760c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f2170cd7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a760c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45770/0 (socket says 172.21.15.28:45770) 2024-04-25T01:42:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f2170cd7640 1 -- 172.21.15.28:0/2218030419 learned_addr learned my addr 172.21.15.28:0/2218030419 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f216bfff640 1 --2- 172.21.15.28:0/2218030419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a7601580 0x5576a7601960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f2169ffb640 1 --2- 172.21.15.28:0/3419177397 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f21540a2d80 0x7f21540a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f2170cd7640 1 -- 172.21.15.28:0/2218030419 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576a760c570 msgr2=0x5576a760e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f2170cd7640 1 --2- 172.21.15.28:0/2218030419 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576a760c570 0x5576a760e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f2170cd7640 1 -- 172.21.15.28:0/2218030419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a7601580 msgr2=0x5576a7601960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f2170cd7640 1 --2- 172.21.15.28:0/2218030419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a7601580 0x5576a7601960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f2170cd7640 1 -- 172.21.15.28:0/2218030419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576a7532400 con 0x5576a7601ea0 2024-04-25T01:42:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f21714d8640 1 --2- 172.21.15.28:0/3419177397 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f21540a2d80 0x7f21540a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f216bfff640 1 --2- 172.21.15.28:0/2218030419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a7601580 0x5576a7601960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f2169ffb640 1 -- 172.21.15.28:0/3419177397 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f216402e030 con 0x7f216c107730 2024-04-25T01:42:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.750+0000 7f2170cd7640 1 --2- 172.21.15.28:0/2218030419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a760c030 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f215c009cd0 tx=0x7f215c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea9be923997758c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.751+0000 7f2152ffd640 1 -- 172.21.15.28:0/2218030419 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215c012450 con 0x5576a7601ea0 2024-04-25T01:42:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.751+0000 7f2152ffd640 1 -- 172.21.15.28:0/2218030419 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f215c012ad0 con 0x5576a7601ea0 2024-04-25T01:42:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.751+0000 7f2152ffd640 1 -- 172.21.15.28:0/2218030419 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215c00fbe0 con 0x5576a7601ea0 2024-04-25T01:42:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.751+0000 7f2175c47740 1 -- 172.21.15.28:0/2218030419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 msgr2=0x5576a760c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.751+0000 7f2175c47740 1 --2- 172.21.15.28:0/2218030419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a760c030 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f215c009cd0 tx=0x7f215c0117b0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.751+0000 7f21714d8640 1 --2- 172.21.15.28:0/3419177397 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f21540a2d80 0x7f21540a5240 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f21640bcd70 tx=0x7f21640dd040 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.751+0000 7f2175c47740 1 -- 172.21.15.28:0/2218030419 shutdown_connections 2024-04-25T01:42:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.751+0000 7f2175c47740 1 --2- 172.21.15.28:0/2218030419 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576a760c570 0x5576a760e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.751+0000 7f2175c47740 1 --2- 172.21.15.28:0/2218030419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a760c030 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.751+0000 7f2175c47740 1 --2- 172.21.15.28:0/2218030419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a7601580 0x5576a7601960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.751+0000 7f2175c47740 1 -- 172.21.15.28:0/2218030419 >> 172.21.15.28:0/2218030419 conn(0x5576a75fd8a0 msgr2=0x5576a75fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.751+0000 7f2175c47740 1 -- 172.21.15.28:0/2218030419 shutdown_connections 2024-04-25T01:42:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.751+0000 7f2175c47740 1 -- 172.21.15.28:0/2218030419 wait complete. 2024-04-25T01:42:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.752+0000 7f2175c47740 1 -- start start 2024-04-25T01:42:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.752+0000 7f2175c47740 1 Processor -- start 2024-04-25T01:42:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.752+0000 7f2175c47740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a7601580 0x5576a760bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.752+0000 7f2175c47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a76091e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.752+0000 7f2175c47740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576a760c570 0x5576a7609760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.752+0000 7f2175c47740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5576a760f360 con 0x5576a7601ea0 2024-04-25T01:42:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.752+0000 7f2175c47740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5576a7522050 con 0x5576a7601580 2024-04-25T01:42:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.752+0000 7f2175c47740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5576a760f4e0 con 0x5576a760c570 2024-04-25T01:42:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.752+0000 7f2175c47740 10 client.0 _ll_register_callbacks cb 0x5576a7279be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.752+0000 7f2170cd7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a76091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.752+0000 7f2170cd7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a76091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45780/0 (socket says 172.21.15.28:45780) 2024-04-25T01:42:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.752+0000 7f2170cd7640 1 -- 172.21.15.28:0/257834593 learned_addr learned my addr 172.21.15.28:0/257834593 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.752+0000 7f216bfff640 1 --2- 172.21.15.28:0/257834593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a7601580 0x5576a760bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.752+0000 7f216bfff640 1 --2- 172.21.15.28:0/257834593 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576a760c570 0x5576a7609760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.752+0000 7f2170cd7640 1 -- 172.21.15.28:0/257834593 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576a760c570 msgr2=0x5576a7609760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.753+0000 7f2170cd7640 1 --2- 172.21.15.28:0/257834593 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576a760c570 0x5576a7609760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.753+0000 7f2170cd7640 1 -- 172.21.15.28:0/257834593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a7601580 msgr2=0x5576a760bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.753+0000 7f2170cd7640 1 --2- 172.21.15.28:0/257834593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a7601580 0x5576a760bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.753+0000 7f2170cd7640 1 -- 172.21.15.28:0/257834593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576a74f3dc0 con 0x5576a7601ea0 2024-04-25T01:42:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.753+0000 7f2170cd7640 1 --2- 172.21.15.28:0/257834593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a76091e0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f215c011cb0 tx=0x7f215c004560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1207c9c732b2fae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.753+0000 7f214bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.753+0000 7f214bfff640 1 -- 172.21.15.28:0/257834593 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215c028030 con 0x5576a7601ea0 2024-04-25T01:42:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.753+0000 7f214bfff640 1 -- 172.21.15.28:0/257834593 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f215c012c80 con 0x5576a7601ea0 2024-04-25T01:42:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.753+0000 7f2175c47740 1 -- 172.21.15.28:0/257834593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f215c012ad0 con 0x5576a7601ea0 2024-04-25T01:42:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.753+0000 7f214bfff640 1 -- 172.21.15.28:0/257834593 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f215c032050 con 0x5576a7601ea0 2024-04-25T01:42:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.753+0000 7f2175c47740 10 client.18873 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.753+0000 7f2175c47740 1 -- 172.21.15.28:0/257834593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5576a727a3f0 con 0x5576a7601ea0 2024-04-25T01:42:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.753+0000 7f214bfff640 1 -- 172.21.15.28:0/257834593 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f215c012ad0 con 0x5576a7601ea0 2024-04-25T01:42:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.753+0000 7f214bfff640 1 -- 172.21.15.28:0/257834593 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f215c005020 con 0x5576a7601ea0 2024-04-25T01:42:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.753+0000 7f2175c47740 10 client.18873 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.753+0000 7f2175c47740 10 client.18873 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:42:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.754+0000 7f2175c47740 20 client.18873 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.754+0000 7f2175c47740 1 --2- 172.21.15.28:0/257834593 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5576a761c610 0x5576a761ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.754+0000 7f2175c47740 4 client.18873 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.754+0000 7f2175c47740 1 -- 172.21.15.28:0/257834593 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576a7573bb0 con 0x5576a761c610 2024-04-25T01:42:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.754+0000 7f216bfff640 1 --2- 172.21.15.28:0/257834593 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5576a761c610 0x5576a761ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.754+0000 7f214bfff640 10 client.18873 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:42:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.754+0000 7f216bfff640 1 --2- 172.21.15.28:0/257834593 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5576a761c610 0x5576a761ea00 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=73476e51601b568d server_cookie=da5987d5ee673652 in_seq=0 out_seq=0 2024-04-25T01:42:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.755+0000 7f214bfff640 1 -- 172.21.15.28:0/257834593 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5576a7573bb0 con 0x5576a761c610 2024-04-25T01:42:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.755+0000 7f214bfff640 10 client.18873 handle_command_reply: tid=0 2024-04-25T01:42:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.755+0000 7f2175c47740 1 client.18873 shutdown 2024-04-25T01:42:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.755+0000 7f2175c47740 1 -- 172.21.15.28:0/257834593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 msgr2=0x5576a76091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.755+0000 7f2175c47740 1 --2- 172.21.15.28:0/257834593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a76091e0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f215c011cb0 tx=0x7f215c004560 comp rx=0 tx=0).stop 2024-04-25T01:42:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.755+0000 7f2175c47740 1 -- 172.21.15.28:0/257834593 shutdown_connections 2024-04-25T01:42:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.755+0000 7f2175c47740 1 --2- 172.21.15.28:0/257834593 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5576a761c610 0x5576a761ea00 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.755+0000 7f2175c47740 1 --2- 172.21.15.28:0/257834593 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576a760c570 0x5576a7609760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.755+0000 7f2175c47740 1 --2- 172.21.15.28:0/257834593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a76091e0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.755+0000 7f214bfff640 0 client.18873 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:42:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.755+0000 7f2175c47740 1 --2- 172.21.15.28:0/257834593 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a7601580 0x5576a760bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.755+0000 7f2175c47740 1 -- 172.21.15.28:0/257834593 >> 172.21.15.28:0/257834593 conn(0x5576a75fd8a0 msgr2=0x5576a75fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.755+0000 7f2175c47740 1 -- 172.21.15.28:0/257834593 shutdown_connections 2024-04-25T01:42:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.755+0000 7f2175c47740 1 -- 172.21.15.28:0/257834593 wait complete. 2024-04-25T01:42:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.755+0000 7f2175c47740 20 client.18873 trim_cache size 0 max 16384 2024-04-25T01:42:03.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.768+0000 7f2175c47740 1 Processor -- start 2024-04-25T01:42:03.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.768+0000 7f2175c47740 1 -- start start 2024-04-25T01:42:03.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.769+0000 7f2175c47740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576a7601580 0x5576a7618250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.769+0000 7f2175c47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a760ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.769+0000 7f2175c47740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a760c570 0x5576a76240c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.769+0000 7f2175c47740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f216c1aead0 con 0x5576a7601ea0 2024-04-25T01:42:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.769+0000 7f2175c47740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f216c1b0610 con 0x5576a760c570 2024-04-25T01:42:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.769+0000 7f2175c47740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f216c1af1a0 con 0x5576a7601580 2024-04-25T01:42:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.769+0000 7f2170cd7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a760ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.769+0000 7f2170cd7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a760ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45786/0 (socket says 172.21.15.28:45786) 2024-04-25T01:42:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.769+0000 7f2170cd7640 1 -- 172.21.15.28:0/297030808 learned_addr learned my addr 172.21.15.28:0/297030808 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.769+0000 7f2170cd7640 1 -- 172.21.15.28:0/297030808 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576a7601580 msgr2=0x5576a7618250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.769+0000 7f2170cd7640 1 --2- 172.21.15.28:0/297030808 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576a7601580 0x5576a7618250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.769+0000 7f2170cd7640 1 -- 172.21.15.28:0/297030808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a760c570 msgr2=0x5576a76240c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.769+0000 7f2170cd7640 1 --2- 172.21.15.28:0/297030808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a760c570 0x5576a76240c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.769+0000 7f2170cd7640 1 -- 172.21.15.28:0/297030808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576a73447d0 con 0x5576a7601ea0 2024-04-25T01:42:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.769+0000 7f2170cd7640 1 --2- 172.21.15.28:0/297030808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a760ba20 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f215c005600 tx=0x7f215c009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d91eaa6f7431252 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.769+0000 7f21527fc640 1 -- 172.21.15.28:0/297030808 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215c03b070 con 0x5576a7601ea0 2024-04-25T01:42:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.769+0000 7f21527fc640 1 -- 172.21.15.28:0/297030808 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f215c0045d0 con 0x5576a7601ea0 2024-04-25T01:42:03.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.769+0000 7f21527fc640 1 -- 172.21.15.28:0/297030808 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215c028040 con 0x5576a7601ea0 2024-04-25T01:42:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.770+0000 7f2175c47740 1 -- 172.21.15.28:0/297030808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 msgr2=0x5576a760ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.770+0000 7f2175c47740 1 --2- 172.21.15.28:0/297030808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a760ba20 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f215c005600 tx=0x7f215c009fd0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.770+0000 7f2175c47740 1 -- 172.21.15.28:0/297030808 shutdown_connections 2024-04-25T01:42:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.770+0000 7f2175c47740 1 --2- 172.21.15.28:0/297030808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a760c570 0x5576a76240c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.770+0000 7f2175c47740 1 --2- 172.21.15.28:0/297030808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a760ba20 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.770+0000 7f2175c47740 1 --2- 172.21.15.28:0/297030808 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576a7601580 0x5576a7618250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.770+0000 7f2175c47740 1 -- 172.21.15.28:0/297030808 >> 172.21.15.28:0/297030808 conn(0x5576a75fd8a0 msgr2=0x5576a7603560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.770+0000 7f2175c47740 1 -- 172.21.15.28:0/297030808 shutdown_connections 2024-04-25T01:42:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.770+0000 7f2175c47740 1 -- 172.21.15.28:0/297030808 wait complete. 2024-04-25T01:42:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.770+0000 7f2175c47740 1 -- start start 2024-04-25T01:42:03.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.770+0000 7f2175c47740 1 Processor -- start 2024-04-25T01:42:03.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f2175c47740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576a7601580 0x5576a7614760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f2175c47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a7614ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f2175c47740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a760c570 0x5576a76055b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f2175c47740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f216c1aff40 con 0x5576a7601ea0 2024-04-25T01:42:03.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f2175c47740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f216c1af870 con 0x5576a760c570 2024-04-25T01:42:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f2175c47740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f216c1acad0 con 0x5576a7601580 2024-04-25T01:42:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f2175c47740 10 client.0 _ll_register_callbacks cb 0x7f215c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f2170cd7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a7614ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f2170cd7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a7614ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45796/0 (socket says 172.21.15.28:45796) 2024-04-25T01:42:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f2170cd7640 1 -- 172.21.15.28:0/1342361387 learned_addr learned my addr 172.21.15.28:0/1342361387 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f2170cd7640 1 -- 172.21.15.28:0/1342361387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576a7601580 msgr2=0x5576a7614760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f2170cd7640 1 --2- 172.21.15.28:0/1342361387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576a7601580 0x5576a7614760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f2170cd7640 1 -- 172.21.15.28:0/1342361387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a760c570 msgr2=0x5576a76055b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f2170cd7640 1 --2- 172.21.15.28:0/1342361387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a760c570 0x5576a76055b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f2170cd7640 1 -- 172.21.15.28:0/1342361387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576a7605db0 con 0x5576a7601ea0 2024-04-25T01:42:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f2170cd7640 1 --2- 172.21.15.28:0/1342361387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a7614ca0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f215c005410 tx=0x7f215c003b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b5038e9d25ccf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f214bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f214bfff640 1 -- 172.21.15.28:0/1342361387 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215c03b040 con 0x5576a7601ea0 2024-04-25T01:42:03.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f214bfff640 1 -- 172.21.15.28:0/1342361387 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f215c003d00 con 0x5576a7601ea0 2024-04-25T01:42:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f2175c47740 1 -- 172.21.15.28:0/1342361387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f215c0045d0 con 0x5576a7601ea0 2024-04-25T01:42:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.771+0000 7f214bfff640 1 -- 172.21.15.28:0/1342361387 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f215c028040 con 0x5576a7601ea0 2024-04-25T01:42:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.772+0000 7f2175c47740 10 client.18879 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.772+0000 7f2175c47740 1 -- 172.21.15.28:0/1342361387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5576a76823b0 con 0x5576a7601ea0 2024-04-25T01:42:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.772+0000 7f214bfff640 1 -- 172.21.15.28:0/1342361387 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f215c0045d0 con 0x5576a7601ea0 2024-04-25T01:42:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.772+0000 7f214bfff640 1 -- 172.21.15.28:0/1342361387 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f215c035070 con 0x5576a7601ea0 2024-04-25T01:42:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.772+0000 7f2175c47740 10 client.18879 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.772+0000 7f2175c47740 10 client.18879 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:42:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.772+0000 7f2175c47740 20 client.18879 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.772+0000 7f2175c47740 1 --2- 172.21.15.28:0/1342361387 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5576a7682a70 0x5576a7684e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.772+0000 7f2175c47740 4 client.18879 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:42:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.772+0000 7f2175c47740 1 -- 172.21.15.28:0/1342361387 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5576a756df00 con 0x5576a7682a70 2024-04-25T01:42:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.772+0000 7f216bfff640 1 --2- 172.21.15.28:0/1342361387 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5576a7682a70 0x5576a7684e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.773+0000 7f214bfff640 10 client.18879 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:42:03.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.773+0000 7f216bfff640 1 --2- 172.21.15.28:0/1342361387 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5576a7682a70 0x5576a7684e90 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5a2cbe1ef63dcc3 server_cookie=42d3e9b0c1d829aa in_seq=0 out_seq=0 2024-04-25T01:42:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.773+0000 7f214bfff640 1 -- 172.21.15.28:0/1342361387 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5576a756df00 con 0x5576a7682a70 2024-04-25T01:42:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.773+0000 7f214bfff640 10 client.18879 handle_command_reply: tid=0 2024-04-25T01:42:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.773+0000 7f2175c47740 1 client.18879 shutdown 2024-04-25T01:42:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.773+0000 7f2175c47740 1 -- 172.21.15.28:0/1342361387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 msgr2=0x5576a7614ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.773+0000 7f2175c47740 1 --2- 172.21.15.28:0/1342361387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a7614ca0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f215c005410 tx=0x7f215c003b10 comp rx=0 tx=0).stop 2024-04-25T01:42:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.773+0000 7f2175c47740 1 -- 172.21.15.28:0/1342361387 shutdown_connections 2024-04-25T01:42:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.773+0000 7f2175c47740 1 --2- 172.21.15.28:0/1342361387 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x5576a7682a70 0x5576a7684e90 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.773+0000 7f2175c47740 1 --2- 172.21.15.28:0/1342361387 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5576a760c570 0x5576a76055b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.773+0000 7f214bfff640 0 client.18879 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:42:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.773+0000 7f2175c47740 1 --2- 172.21.15.28:0/1342361387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a7601ea0 0x5576a7614ca0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.773+0000 7f2175c47740 1 --2- 172.21.15.28:0/1342361387 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5576a7601580 0x5576a7614760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.773+0000 7f2175c47740 1 -- 172.21.15.28:0/1342361387 >> 172.21.15.28:0/1342361387 conn(0x5576a75fd8a0 msgr2=0x5576a7602a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.773+0000 7f2175c47740 1 -- 172.21.15.28:0/1342361387 shutdown_connections 2024-04-25T01:42:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.773+0000 7f2175c47740 1 -- 172.21.15.28:0/1342361387 wait complete. 2024-04-25T01:42:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.773+0000 7f2175c47740 20 client.18879 trim_cache size 0 max 16384 2024-04-25T01:42:03.766 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:42:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.774+0000 7f2172f3b640 1 -- 172.21.15.28:0/3419177397 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f21540a2d80 msgr2=0x7f21540a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.774+0000 7f2172f3b640 1 --2- 172.21.15.28:0/3419177397 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f21540a2d80 0x7f21540a5240 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f21640bcd70 tx=0x7f21640dd040 comp rx=0 tx=0).stop 2024-04-25T01:42:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.774+0000 7f2172f3b640 1 -- 172.21.15.28:0/3419177397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f216c107730 msgr2=0x7f216c19c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.774+0000 7f2172f3b640 1 --2- 172.21.15.28:0/3419177397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f216c107730 0x7f216c19c850 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f2164033040 tx=0x7f2164031e60 comp rx=0 tx=0).stop 2024-04-25T01:42:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.774+0000 7f2172f3b640 1 -- 172.21.15.28:0/3419177397 shutdown_connections 2024-04-25T01:42:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.774+0000 7f2172f3b640 1 --2- 172.21.15.28:0/3419177397 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f21540a2d80 0x7f21540a5240 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.774+0000 7f2172f3b640 1 --2- 172.21.15.28:0/3419177397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f216c107730 0x7f216c19c850 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.774+0000 7f2172f3b640 1 --2- 172.21.15.28:0/3419177397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f216c06a780 0x7f216c1984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.774+0000 7f2172f3b640 1 --2- 172.21.15.28:0/3419177397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f216c069dc0 0x7f216c197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.774+0000 7f2172f3b640 1 -- 172.21.15.28:0/3419177397 >> 172.21.15.28:0/3419177397 conn(0x7f216c0776a0 msgr2=0x7f216c0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.774+0000 7f2172f3b640 1 -- 172.21.15.28:0/3419177397 shutdown_connections 2024-04-25T01:42:03.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.774+0000 7f2172f3b640 1 -- 172.21.15.28:0/3419177397 wait complete. 2024-04-25T01:42:03.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:42:03.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.888+0000 7f893afc8640 1 -- 172.21.15.28:0/2112276468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8934100620 msgr2=0x7f8934100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.888+0000 7f893afc8640 1 --2- 172.21.15.28:0/2112276468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8934100620 0x7f8934100aa0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f8920009930 tx=0x7f892002a420 comp rx=0 tx=0).stop 2024-04-25T01:42:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.888+0000 7f893afc8640 1 -- 172.21.15.28:0/2112276468 shutdown_connections 2024-04-25T01:42:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.888+0000 7f893afc8640 1 --2- 172.21.15.28:0/2112276468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8934100fe0 0x7f893410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.888+0000 7f893afc8640 1 --2- 172.21.15.28:0/2112276468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8934100620 0x7f8934100aa0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.888+0000 7f893afc8640 1 --2- 172.21.15.28:0/2112276468 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8934106630 0x7f8934106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.888+0000 7f893afc8640 1 -- 172.21.15.28:0/2112276468 >> 172.21.15.28:0/2112276468 conn(0x7f89340776a0 msgr2=0x7f8934077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.888+0000 7f893afc8640 1 -- 172.21.15.28:0/2112276468 shutdown_connections 2024-04-25T01:42:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.888+0000 7f893afc8640 1 -- 172.21.15.28:0/2112276468 wait complete. 2024-04-25T01:42:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.889+0000 7f893afc8640 1 Processor -- start 2024-04-25T01:42:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.889+0000 7f893afc8640 1 -- start start 2024-04-25T01:42:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.889+0000 7f893afc8640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8934100620 0x7f89340fd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.889+0000 7f893afc8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8934100fe0 0x7f89340fdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.889+0000 7f893afc8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8934106630 0x7f89340fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.889+0000 7f893afc8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8934110a40 con 0x7f8934106630 2024-04-25T01:42:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.889+0000 7f893afc8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f89341108c0 con 0x7f8934100fe0 2024-04-25T01:42:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.889+0000 7f893afc8640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8934110bc0 con 0x7f8934100620 2024-04-25T01:42:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.889+0000 7f8939565640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8934106630 0x7f89340fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.889+0000 7f8939565640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8934106630 0x7f89340fe0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45814/0 (socket says 172.21.15.28:45814) 2024-04-25T01:42:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.889+0000 7f8939565640 1 -- 172.21.15.28:0/3413752286 learned_addr learned my addr 172.21.15.28:0/3413752286 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.889+0000 7f8933fff640 1 --2- 172.21.15.28:0/3413752286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8934100fe0 0x7f89340fdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.890+0000 7f8938d64640 1 --2- 172.21.15.28:0/3413752286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8934100620 0x7f89340fd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.890+0000 7f8939565640 1 -- 172.21.15.28:0/3413752286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8934100620 msgr2=0x7f89340fd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.890+0000 7f8939565640 1 --2- 172.21.15.28:0/3413752286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8934100620 0x7f89340fd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.890+0000 7f8939565640 1 -- 172.21.15.28:0/3413752286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8934100fe0 msgr2=0x7f89340fdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.890+0000 7f8939565640 1 --2- 172.21.15.28:0/3413752286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8934100fe0 0x7f89340fdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.890+0000 7f8939565640 1 -- 172.21.15.28:0/3413752286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89341a6590 con 0x7f8934106630 2024-04-25T01:42:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.890+0000 7f8939565640 1 --2- 172.21.15.28:0/3413752286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8934106630 0x7f89340fe0d0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f892c00eaa0 tx=0x7f892c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.890+0000 7f8931ffb640 1 -- 172.21.15.28:0/3413752286 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c00cdb0 con 0x7f8934106630 2024-04-25T01:42:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.890+0000 7f8931ffb640 1 -- 172.21.15.28:0/3413752286 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f892c005020 con 0x7f8934106630 2024-04-25T01:42:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.890+0000 7f8931ffb640 1 -- 172.21.15.28:0/3413752286 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c0116b0 con 0x7f8934106630 2024-04-25T01:42:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.890+0000 7f893afc8640 1 -- 172.21.15.28:0/3413752286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89341a6880 con 0x7f8934106630 2024-04-25T01:42:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.890+0000 7f893afc8640 1 -- 172.21.15.28:0/3413752286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89341a6d50 con 0x7f8934106630 2024-04-25T01:42:03.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.892+0000 7f8931ffb640 1 -- 172.21.15.28:0/3413752286 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f892c005560 con 0x7f8934106630 2024-04-25T01:42:03.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f893ddb3740 1 Processor -- start 2024-04-25T01:42:03.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f893ddb3740 1 -- start start 2024-04-25T01:42:03.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f893ddb3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453579580 0x55a453579960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f893ddb3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579ea0 0x55a453584030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f893ddb3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453584570 0x55a453586960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f893ddb3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a4534de670 con 0x55a453584570 2024-04-25T01:42:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f893ddb3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a453486970 con 0x55a453579ea0 2024-04-25T01:42:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f893ddb3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55a4534da630 con 0x55a453579580 2024-04-25T01:42:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f8931ffb640 1 --2- 172.21.15.28:0/3413752286 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f89240a2d30 0x7f89240a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f8933fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453584570 0x55a453586960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f8933fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453584570 0x55a453586960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45824/0 (socket says 172.21.15.28:45824) 2024-04-25T01:42:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f8939565640 1 --2- 172.21.15.28:0/3413752286 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f89240a2d30 0x7f89240a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f8933fff640 1 -- 172.21.15.28:0/1312061592 learned_addr learned my addr 172.21.15.28:0/1312061592 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f8931ffb640 1 -- 172.21.15.28:0/3413752286 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f892c0cfa90 con 0x7f8934106630 2024-04-25T01:42:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f8938d64640 1 --2- 172.21.15.28:0/1312061592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579ea0 0x55a453584030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f8933fff640 1 --2- 172.21.15.28:0/1312061592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453579580 0x55a453579960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f8933fff640 1 -- 172.21.15.28:0/1312061592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453579580 msgr2=0x55a453579960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f8933fff640 1 --2- 172.21.15.28:0/1312061592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453579580 0x55a453579960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f8933fff640 1 -- 172.21.15.28:0/1312061592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579ea0 msgr2=0x55a453584030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f8933fff640 1 --2- 172.21.15.28:0/1312061592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579ea0 0x55a453584030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f8933fff640 1 -- 172.21.15.28:0/1312061592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4534aa400 con 0x55a453584570 2024-04-25T01:42:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.893+0000 7f8938d64640 1 --2- 172.21.15.28:0/1312061592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579ea0 0x55a453584030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.894+0000 7f8933fff640 1 --2- 172.21.15.28:0/1312061592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453584570 0x55a453586960 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f892002a940 tx=0x7f8920005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb947519311680c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.894+0000 7f8939565640 1 --2- 172.21.15.28:0/3413752286 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f89240a2d30 0x7f89240a51f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f892c098020 tx=0x7f892c0054d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.894+0000 7f891effd640 1 -- 172.21.15.28:0/1312061592 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8920035040 con 0x55a453584570 2024-04-25T01:42:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.894+0000 7f891effd640 1 -- 172.21.15.28:0/1312061592 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8920031a20 con 0x55a453584570 2024-04-25T01:42:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.894+0000 7f891effd640 1 -- 172.21.15.28:0/1312061592 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8920004a00 con 0x55a453584570 2024-04-25T01:42:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.894+0000 7f893ddb3740 1 -- 172.21.15.28:0/1312061592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453584570 msgr2=0x55a453586960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.894+0000 7f893ddb3740 1 --2- 172.21.15.28:0/1312061592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453584570 0x55a453586960 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f892002a940 tx=0x7f8920005fd0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.894+0000 7f893ddb3740 1 -- 172.21.15.28:0/1312061592 shutdown_connections 2024-04-25T01:42:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.894+0000 7f893ddb3740 1 --2- 172.21.15.28:0/1312061592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453584570 0x55a453586960 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.894+0000 7f893ddb3740 1 --2- 172.21.15.28:0/1312061592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579ea0 0x55a453584030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.894+0000 7f893ddb3740 1 --2- 172.21.15.28:0/1312061592 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453579580 0x55a453579960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.894+0000 7f893ddb3740 1 -- 172.21.15.28:0/1312061592 >> 172.21.15.28:0/1312061592 conn(0x55a4535758a0 msgr2=0x55a453575c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.894+0000 7f893ddb3740 1 -- 172.21.15.28:0/1312061592 shutdown_connections 2024-04-25T01:42:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.895+0000 7f893ddb3740 1 -- 172.21.15.28:0/1312061592 wait complete. 2024-04-25T01:42:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.895+0000 7f893ddb3740 1 -- start start 2024-04-25T01:42:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.895+0000 7f893ddb3740 1 Processor -- start 2024-04-25T01:42:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.895+0000 7f893ddb3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453579580 0x55a453583ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.895+0000 7f893ddb3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579ea0 0x55a4535811c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.895+0000 7f893ddb3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453584570 0x55a453581720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.895+0000 7f893ddb3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a453587300 con 0x55a453579580 2024-04-25T01:42:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.895+0000 7f893ddb3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a45349a050 con 0x55a453579ea0 2024-04-25T01:42:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.895+0000 7f893ddb3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55a453587480 con 0x55a453584570 2024-04-25T01:42:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.895+0000 7f893ddb3740 10 client.0 _ll_register_callbacks cb 0x55a4531f1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.895+0000 7f8933fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453579580 0x55a453583ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.895+0000 7f8933fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453579580 0x55a453583ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45840/0 (socket says 172.21.15.28:45840) 2024-04-25T01:42:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.895+0000 7f8933fff640 1 -- 172.21.15.28:0/3117453893 learned_addr learned my addr 172.21.15.28:0/3117453893 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.896+0000 7f8938d64640 1 --2- 172.21.15.28:0/3117453893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579ea0 0x55a4535811c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.896+0000 7f8933fff640 1 --2- 172.21.15.28:0/3117453893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453584570 0x55a453581720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.896+0000 7f8933fff640 1 -- 172.21.15.28:0/3117453893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453584570 msgr2=0x55a453581720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.896+0000 7f8933fff640 1 --2- 172.21.15.28:0/3117453893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453584570 0x55a453581720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.896+0000 7f8933fff640 1 -- 172.21.15.28:0/3117453893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579ea0 msgr2=0x55a4535811c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.896+0000 7f8933fff640 1 --2- 172.21.15.28:0/3117453893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579ea0 0x55a4535811c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.896+0000 7f8933fff640 1 -- 172.21.15.28:0/3117453893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a45346bdc0 con 0x55a453579580 2024-04-25T01:42:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.896+0000 7f8933fff640 1 --2- 172.21.15.28:0/3117453893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453579580 0x55a453583ed0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f8920005940 tx=0x7f89200055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ab41e0bc3aa916 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.896+0000 7f8913fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.896+0000 7f8913fff640 1 -- 172.21.15.28:0/3117453893 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8920035040 con 0x55a453579580 2024-04-25T01:42:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.896+0000 7f8913fff640 1 -- 172.21.15.28:0/3117453893 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f8920004980 con 0x55a453579580 2024-04-25T01:42:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.896+0000 7f893ddb3740 1 -- 172.21.15.28:0/3117453893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8920031a20 con 0x55a453579580 2024-04-25T01:42:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.896+0000 7f8913fff640 1 -- 172.21.15.28:0/3117453893 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8920058050 con 0x55a453579580 2024-04-25T01:42:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.896+0000 7f893ddb3740 10 client.18891 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.896+0000 7f893ddb3740 1 -- 172.21.15.28:0/3117453893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a4531f23f0 con 0x55a453579580 2024-04-25T01:42:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.896+0000 7f8913fff640 1 -- 172.21.15.28:0/3117453893 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8920031a20 con 0x55a453579580 2024-04-25T01:42:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.897+0000 7f8913fff640 1 -- 172.21.15.28:0/3117453893 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f892004ac60 con 0x55a453579580 2024-04-25T01:42:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.897+0000 7f893ddb3740 10 client.18891 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.897+0000 7f893ddb3740 10 client.18891 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:42:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.897+0000 7f893ddb3740 20 client.18891 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.897+0000 7f893ddb3740 1 --2- 172.21.15.28:0/3117453893 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55a4535943f0 0x55a4535967e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.897+0000 7f893ddb3740 4 client.18891 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.897+0000 7f893ddb3740 1 -- 172.21.15.28:0/3117453893 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4534ebbb0 con 0x55a4535943f0 2024-04-25T01:42:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.897+0000 7f8938d64640 1 --2- 172.21.15.28:0/3117453893 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55a4535943f0 0x55a4535967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.898+0000 7f8913fff640 10 client.18891 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:42:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.898+0000 7f8938d64640 1 --2- 172.21.15.28:0/3117453893 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55a4535943f0 0x55a4535967e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2c434bde3b4738df server_cookie=f906007f8e0134ed in_seq=0 out_seq=0 2024-04-25T01:42:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.898+0000 7f8913fff640 1 -- 172.21.15.28:0/3117453893 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a4534ebbb0 con 0x55a4535943f0 2024-04-25T01:42:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.898+0000 7f8913fff640 10 client.18891 handle_command_reply: tid=0 2024-04-25T01:42:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.898+0000 7f893ddb3740 1 client.18891 shutdown 2024-04-25T01:42:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.898+0000 7f893ddb3740 1 -- 172.21.15.28:0/3117453893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453579580 msgr2=0x55a453583ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.898+0000 7f893ddb3740 1 --2- 172.21.15.28:0/3117453893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453579580 0x55a453583ed0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f8920005940 tx=0x7f89200055a0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.898+0000 7f893ddb3740 1 -- 172.21.15.28:0/3117453893 shutdown_connections 2024-04-25T01:42:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.898+0000 7f893ddb3740 1 --2- 172.21.15.28:0/3117453893 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453584570 0x55a453581720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.898+0000 7f893ddb3740 1 --2- 172.21.15.28:0/3117453893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579ea0 0x55a4535811c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.898+0000 7f893ddb3740 1 --2- 172.21.15.28:0/3117453893 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55a4535943f0 0x55a4535967e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.898+0000 7f893ddb3740 1 --2- 172.21.15.28:0/3117453893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453579580 0x55a453583ed0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.898+0000 7f8913fff640 0 client.18891 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:42:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.898+0000 7f893ddb3740 1 -- 172.21.15.28:0/3117453893 >> 172.21.15.28:0/3117453893 conn(0x55a4535758a0 msgr2=0x55a45357b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.899+0000 7f893ddb3740 1 -- 172.21.15.28:0/3117453893 shutdown_connections 2024-04-25T01:42:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.899+0000 7f893ddb3740 1 -- 172.21.15.28:0/3117453893 wait complete. 2024-04-25T01:42:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.899+0000 7f893ddb3740 20 client.18891 trim_cache size 0 max 16384 2024-04-25T01:42:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.912+0000 7f893ddb3740 1 Processor -- start 2024-04-25T01:42:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.912+0000 7f893ddb3740 1 -- start start 2024-04-25T01:42:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.912+0000 7f893ddb3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579580 0x55a453580210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.912+0000 7f893ddb3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453579ea0 0x55a453599810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.912+0000 7f893ddb3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453584570 0x55a453599d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.912+0000 7f893ddb3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89341b0ce0 con 0x55a453579ea0 2024-04-25T01:42:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.912+0000 7f893ddb3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f89341b2820 con 0x55a453579580 2024-04-25T01:42:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.912+0000 7f893ddb3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f89341b13b0 con 0x55a453584570 2024-04-25T01:42:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.912+0000 7f8938d64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453579ea0 0x55a453599810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.912+0000 7f8938d64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453579ea0 0x55a453599810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45848/0 (socket says 172.21.15.28:45848) 2024-04-25T01:42:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.912+0000 7f8938d64640 1 -- 172.21.15.28:0/340892596 learned_addr learned my addr 172.21.15.28:0/340892596 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.912+0000 7f8933fff640 1 --2- 172.21.15.28:0/340892596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579580 0x55a453580210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.912+0000 7f8933fff640 1 --2- 172.21.15.28:0/340892596 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453584570 0x55a453599d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.913+0000 7f8938d64640 1 -- 172.21.15.28:0/340892596 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453584570 msgr2=0x55a453599d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.913+0000 7f8938d64640 1 --2- 172.21.15.28:0/340892596 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453584570 0x55a453599d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.913+0000 7f8938d64640 1 -- 172.21.15.28:0/340892596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579580 msgr2=0x55a453580210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.913+0000 7f8938d64640 1 --2- 172.21.15.28:0/340892596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579580 0x55a453580210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.913+0000 7f8938d64640 1 -- 172.21.15.28:0/340892596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4532bc7d0 con 0x55a453579ea0 2024-04-25T01:42:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.913+0000 7f8933fff640 1 --2- 172.21.15.28:0/340892596 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453584570 0x55a453599d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.913+0000 7f8938d64640 1 --2- 172.21.15.28:0/340892596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453579ea0 0x55a453599810 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f892800b370 tx=0x7f8928005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d81a439968328507 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.913+0000 7f891e7fc640 1 -- 172.21.15.28:0/340892596 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892801f020 con 0x55a453579ea0 2024-04-25T01:42:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.913+0000 7f891e7fc640 1 -- 172.21.15.28:0/340892596 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f892801bdc0 con 0x55a453579ea0 2024-04-25T01:42:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.913+0000 7f891e7fc640 1 -- 172.21.15.28:0/340892596 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8928004730 con 0x55a453579ea0 2024-04-25T01:42:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.913+0000 7f893ddb3740 1 -- 172.21.15.28:0/340892596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453579ea0 msgr2=0x55a453599810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.913+0000 7f893ddb3740 1 --2- 172.21.15.28:0/340892596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453579ea0 0x55a453599810 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f892800b370 tx=0x7f8928005c50 comp rx=0 tx=0).stop 2024-04-25T01:42:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.914+0000 7f893ddb3740 1 -- 172.21.15.28:0/340892596 shutdown_connections 2024-04-25T01:42:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.914+0000 7f893ddb3740 1 --2- 172.21.15.28:0/340892596 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453584570 0x55a453599d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.914+0000 7f893ddb3740 1 --2- 172.21.15.28:0/340892596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453579ea0 0x55a453599810 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.914+0000 7f893ddb3740 1 --2- 172.21.15.28:0/340892596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579580 0x55a453580210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.914+0000 7f893ddb3740 1 -- 172.21.15.28:0/340892596 >> 172.21.15.28:0/340892596 conn(0x55a4535758a0 msgr2=0x55a453575c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.914+0000 7f893ddb3740 1 -- 172.21.15.28:0/340892596 shutdown_connections 2024-04-25T01:42:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.914+0000 7f893ddb3740 1 -- 172.21.15.28:0/340892596 wait complete. 2024-04-25T01:42:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.914+0000 7f893ddb3740 1 -- start start 2024-04-25T01:42:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.914+0000 7f893ddb3740 1 Processor -- start 2024-04-25T01:42:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.914+0000 7f893ddb3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579580 0x55a4535f2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.914+0000 7f893ddb3740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453579ea0 0x55a4535f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.914+0000 7f893ddb3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453584570 0x55a453598e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.914+0000 7f893ddb3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89341b2150 con 0x55a453584570 2024-04-25T01:42:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.914+0000 7f893ddb3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f89341b1a80 con 0x55a453579580 2024-04-25T01:42:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.914+0000 7f893ddb3740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f89341aece0 con 0x55a453579ea0 2024-04-25T01:42:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.914+0000 7f893ddb3740 10 client.0 _ll_register_callbacks cb 0x7f8920005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.914+0000 7f8933fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453584570 0x55a453598e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.915+0000 7f8933fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453584570 0x55a453598e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45860/0 (socket says 172.21.15.28:45860) 2024-04-25T01:42:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.915+0000 7f8933fff640 1 -- 172.21.15.28:0/694218439 learned_addr learned my addr 172.21.15.28:0/694218439 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.915+0000 7f8938d64640 1 --2- 172.21.15.28:0/694218439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453579ea0 0x55a4535f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.915+0000 7f8933fff640 1 --2- 172.21.15.28:0/694218439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579580 0x55a4535f2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.915+0000 7f8933fff640 1 -- 172.21.15.28:0/694218439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453579ea0 msgr2=0x55a4535f31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.915+0000 7f8933fff640 1 --2- 172.21.15.28:0/694218439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453579ea0 0x55a4535f31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.915+0000 7f8933fff640 1 -- 172.21.15.28:0/694218439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579580 msgr2=0x55a4535f2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.915+0000 7f8933fff640 1 --2- 172.21.15.28:0/694218439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579580 0x55a4535f2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.915+0000 7f8933fff640 1 -- 172.21.15.28:0/694218439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a453599550 con 0x55a453584570 2024-04-25T01:42:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.915+0000 7f8933fff640 1 --2- 172.21.15.28:0/694218439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453584570 0x55a453598e10 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f8920009850 tx=0x7f8920005500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f936b4b2084b2338 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.915+0000 7f8913fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.915+0000 7f8913fff640 1 -- 172.21.15.28:0/694218439 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8920004980 con 0x55a453584570 2024-04-25T01:42:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.915+0000 7f8913fff640 1 -- 172.21.15.28:0/694218439 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f8920004b20 con 0x55a453584570 2024-04-25T01:42:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.915+0000 7f893ddb3740 1 -- 172.21.15.28:0/694218439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f892801bdc0 con 0x55a453584570 2024-04-25T01:42:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.915+0000 7f8913fff640 1 -- 172.21.15.28:0/694218439 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8920004d20 con 0x55a453584570 2024-04-25T01:42:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.915+0000 7f893ddb3740 10 client.18897 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:03.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.915+0000 7f893ddb3740 1 -- 172.21.15.28:0/694218439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a4535fa430 con 0x55a453584570 2024-04-25T01:42:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.916+0000 7f8913fff640 1 -- 172.21.15.28:0/694218439 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f892801bdc0 con 0x55a453584570 2024-04-25T01:42:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.916+0000 7f8913fff640 1 -- 172.21.15.28:0/694218439 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f892004a7d0 con 0x55a453584570 2024-04-25T01:42:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.916+0000 7f893ddb3740 10 client.18897 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.916+0000 7f893ddb3740 10 client.18897 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:42:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.916+0000 7f893ddb3740 20 client.18897 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.916+0000 7f893ddb3740 1 --2- 172.21.15.28:0/694218439 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55a4535faaa0 0x55a4535fce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.916+0000 7f893ddb3740 4 client.18897 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:42:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.916+0000 7f893ddb3740 1 -- 172.21.15.28:0/694218439 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a4534e5f00 con 0x55a4535faaa0 2024-04-25T01:42:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.916+0000 7f8938d64640 1 --2- 172.21.15.28:0/694218439 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55a4535faaa0 0x55a4535fce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.916+0000 7f8913fff640 10 client.18897 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:42:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.916+0000 7f8938d64640 1 --2- 172.21.15.28:0/694218439 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55a4535faaa0 0x55a4535fce90 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d221e384153b34f server_cookie=d9aca3f643d1f886 in_seq=0 out_seq=0 2024-04-25T01:42:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.917+0000 7f8913fff640 1 -- 172.21.15.28:0/694218439 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a4534e5f00 con 0x55a4535faaa0 2024-04-25T01:42:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.917+0000 7f8913fff640 10 client.18897 handle_command_reply: tid=0 2024-04-25T01:42:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.917+0000 7f893ddb3740 1 client.18897 shutdown 2024-04-25T01:42:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.917+0000 7f893ddb3740 1 -- 172.21.15.28:0/694218439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453584570 msgr2=0x55a453598e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.917+0000 7f893ddb3740 1 --2- 172.21.15.28:0/694218439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453584570 0x55a453598e10 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f8920009850 tx=0x7f8920005500 comp rx=0 tx=0).stop 2024-04-25T01:42:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.917+0000 7f893ddb3740 1 -- 172.21.15.28:0/694218439 shutdown_connections 2024-04-25T01:42:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.917+0000 7f893ddb3740 1 --2- 172.21.15.28:0/694218439 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x55a4535faaa0 0x55a4535fce90 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.917+0000 7f893ddb3740 1 --2- 172.21.15.28:0/694218439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a453584570 0x55a453598e10 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.917+0000 7f8913fff640 0 client.18897 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:42:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.917+0000 7f893ddb3740 1 --2- 172.21.15.28:0/694218439 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a453579ea0 0x55a4535f31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.917+0000 7f893ddb3740 1 --2- 172.21.15.28:0/694218439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a453579580 0x55a4535f2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.917+0000 7f893ddb3740 1 -- 172.21.15.28:0/694218439 >> 172.21.15.28:0/694218439 conn(0x55a4535758a0 msgr2=0x55a45358ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.917+0000 7f893ddb3740 1 -- 172.21.15.28:0/694218439 shutdown_connections 2024-04-25T01:42:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.917+0000 7f893ddb3740 1 -- 172.21.15.28:0/694218439 wait complete. 2024-04-25T01:42:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.917+0000 7f893ddb3740 20 client.18897 trim_cache size 0 max 16384 2024-04-25T01:42:03.911 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:42:03.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.917+0000 7f893afc8640 1 -- 172.21.15.28:0/3413752286 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f89240a2d30 msgr2=0x7f89240a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.917+0000 7f893afc8640 1 --2- 172.21.15.28:0/3413752286 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f89240a2d30 0x7f89240a51f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f892c098020 tx=0x7f892c0054d0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.917+0000 7f893afc8640 1 -- 172.21.15.28:0/3413752286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8934106630 msgr2=0x7f89340fe0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:03.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.917+0000 7f893afc8640 1 --2- 172.21.15.28:0/3413752286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8934106630 0x7f89340fe0d0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f892c00eaa0 tx=0x7f892c00ef60 comp rx=0 tx=0).stop 2024-04-25T01:42:03.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.918+0000 7f893afc8640 1 -- 172.21.15.28:0/3413752286 shutdown_connections 2024-04-25T01:42:03.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.918+0000 7f893afc8640 1 --2- 172.21.15.28:0/3413752286 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f89240a2d30 0x7f89240a51f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.918+0000 7f893afc8640 1 --2- 172.21.15.28:0/3413752286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8934106630 0x7f89340fe0d0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.918+0000 7f893afc8640 1 --2- 172.21.15.28:0/3413752286 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8934100fe0 0x7f89340fdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.918+0000 7f893afc8640 1 --2- 172.21.15.28:0/3413752286 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8934100620 0x7f89340fd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:03.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.918+0000 7f893afc8640 1 -- 172.21.15.28:0/3413752286 >> 172.21.15.28:0/3413752286 conn(0x7f89340776a0 msgr2=0x7f893406a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:03.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.918+0000 7f893afc8640 1 -- 172.21.15.28:0/3413752286 shutdown_connections 2024-04-25T01:42:03.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:03.918+0000 7f893afc8640 1 -- 172.21.15.28:0/3413752286 wait complete. 2024-04-25T01:42:03.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:42:04.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.032+0000 7fbf56e7d640 1 -- 172.21.15.28:0/1610793751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf501066b0 msgr2=0x7fbf50106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.032+0000 7fbf56e7d640 1 --2- 172.21.15.28:0/1610793751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf501066b0 0x7fbf50106a90 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009930 tx=0x7fbf4002a420 comp rx=0 tx=0).stop 2024-04-25T01:42:04.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.033+0000 7fbf56e7d640 1 -- 172.21.15.28:0/1610793751 shutdown_connections 2024-04-25T01:42:04.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.033+0000 7fbf56e7d640 1 --2- 172.21.15.28:0/1610793751 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf50100fd0 0x7fbf5010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.033+0000 7fbf56e7d640 1 --2- 172.21.15.28:0/1610793751 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf50100610 0x7fbf50100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.033+0000 7fbf56e7d640 1 --2- 172.21.15.28:0/1610793751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf501066b0 0x7fbf50106a90 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.033+0000 7fbf56e7d640 1 -- 172.21.15.28:0/1610793751 >> 172.21.15.28:0/1610793751 conn(0x7fbf50077740 msgr2=0x7fbf50077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:04.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.033+0000 7fbf56e7d640 1 -- 172.21.15.28:0/1610793751 shutdown_connections 2024-04-25T01:42:04.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.033+0000 7fbf56e7d640 1 -- 172.21.15.28:0/1610793751 wait complete. 2024-04-25T01:42:04.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.033+0000 7fbf56e7d640 1 Processor -- start 2024-04-25T01:42:04.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.033+0000 7fbf56e7d640 1 -- start start 2024-04-25T01:42:04.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.033+0000 7fbf56e7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf50100610 0x7fbf50197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.033+0000 7fbf56e7d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf50100fd0 0x7fbf501984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.033+0000 7fbf56e7d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf501066b0 0x7fbf5019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.033+0000 7fbf56e7d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf50111b30 con 0x7fbf50100610 2024-04-25T01:42:04.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.033+0000 7fbf56e7d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbf501119b0 con 0x7fbf501066b0 2024-04-25T01:42:04.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.033+0000 7fbf56e7d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbf50111cb0 con 0x7fbf50100fd0 2024-04-25T01:42:04.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.033+0000 7fbf54c19640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf50100610 0x7fbf50197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.034+0000 7fbf54c19640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf50100610 0x7fbf50197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45878/0 (socket says 172.21.15.28:45878) 2024-04-25T01:42:04.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.034+0000 7fbf54c19640 1 -- 172.21.15.28:0/3560630763 learned_addr learned my addr 172.21.15.28:0/3560630763 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:04.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.034+0000 7fbf4ffff640 1 --2- 172.21.15.28:0/3560630763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf50100fd0 0x7fbf501984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.034+0000 7fbf54c19640 1 -- 172.21.15.28:0/3560630763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf50100fd0 msgr2=0x7fbf501984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.034+0000 7fbf5541a640 1 --2- 172.21.15.28:0/3560630763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf501066b0 0x7fbf5019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.034+0000 7fbf54c19640 1 --2- 172.21.15.28:0/3560630763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf50100fd0 0x7fbf501984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.034+0000 7fbf54c19640 1 -- 172.21.15.28:0/3560630763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf501066b0 msgr2=0x7fbf5019c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.034+0000 7fbf54c19640 1 --2- 172.21.15.28:0/3560630763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf501066b0 0x7fbf5019c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.034+0000 7fbf54c19640 1 -- 172.21.15.28:0/3560630763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf5019cfa0 con 0x7fbf50100610 2024-04-25T01:42:04.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.034+0000 7fbf54c19640 1 --2- 172.21.15.28:0/3560630763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf50100610 0x7fbf50197fb0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fbf40005940 tx=0x7fbf400042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:04.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.034+0000 7fbf4dffb640 1 -- 172.21.15.28:0/3560630763 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4003f030 con 0x7fbf50100610 2024-04-25T01:42:04.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.034+0000 7fbf4dffb640 1 -- 172.21.15.28:0/3560630763 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf40003ce0 con 0x7fbf50100610 2024-04-25T01:42:04.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.034+0000 7fbf4dffb640 1 -- 172.21.15.28:0/3560630763 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf40035070 con 0x7fbf50100610 2024-04-25T01:42:04.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.034+0000 7fbf56e7d640 1 -- 172.21.15.28:0/3560630763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf5019d230 con 0x7fbf50100610 2024-04-25T01:42:04.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.034+0000 7fbf56e7d640 1 -- 172.21.15.28:0/3560630763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf501a4b20 con 0x7fbf50100610 2024-04-25T01:42:04.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.037+0000 7fbf4dffb640 1 -- 172.21.15.28:0/3560630763 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fbf40042030 con 0x7fbf50100610 2024-04-25T01:42:04.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.037+0000 7fbf59b8f740 1 Processor -- start 2024-04-25T01:42:04.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.037+0000 7fbf59b8f740 1 -- start start 2024-04-25T01:42:04.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf59b8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 0x55d40c85b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf59b8f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c85bea0 0x55d40c866030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf59b8f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c866570 0x55d40c868960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf59b8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d40c7c0670 con 0x55d40c85b580 2024-04-25T01:42:04.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf59b8f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d40c768970 con 0x55d40c85bea0 2024-04-25T01:42:04.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf59b8f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55d40c7bc630 con 0x55d40c866570 2024-04-25T01:42:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf5541a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 0x55d40c85b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf5541a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 0x55d40c85b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45884/0 (socket says 172.21.15.28:45884) 2024-04-25T01:42:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf5541a640 1 -- 172.21.15.28:0/828399763 learned_addr learned my addr 172.21.15.28:0/828399763 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf4dffb640 1 --2- 172.21.15.28:0/3560630763 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbf380a2d80 0x7fbf380a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf4ffff640 1 --2- 172.21.15.28:0/828399763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c866570 0x55d40c868960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf54c19640 1 --2- 172.21.15.28:0/828399763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c85bea0 0x55d40c866030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf5541a640 1 -- 172.21.15.28:0/828399763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c866570 msgr2=0x55d40c868960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf5541a640 1 --2- 172.21.15.28:0/828399763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c866570 0x55d40c868960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf5541a640 1 -- 172.21.15.28:0/828399763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c85bea0 msgr2=0x55d40c866030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf5541a640 1 --2- 172.21.15.28:0/828399763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c85bea0 0x55d40c866030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf5541a640 1 -- 172.21.15.28:0/828399763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d40c78c400 con 0x55d40c85b580 2024-04-25T01:42:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf54c19640 1 --2- 172.21.15.28:0/828399763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c85bea0 0x55d40c866030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf5541a640 1 --2- 172.21.15.28:0/3560630763 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbf380a2d80 0x7fbf380a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf4dffb640 1 -- 172.21.15.28:0/3560630763 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fbf400efcb0 con 0x7fbf50100610 2024-04-25T01:42:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.038+0000 7fbf5541a640 1 --2- 172.21.15.28:0/828399763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 0x55d40c85b960 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7fbf48009f30 tx=0x7fbf4800f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9bebce84be9c3a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.039+0000 7fbf36ffd640 1 -- 172.21.15.28:0/828399763 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf48004d20 con 0x55d40c85b580 2024-04-25T01:42:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.039+0000 7fbf36ffd640 1 -- 172.21.15.28:0/828399763 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf4801a8f0 con 0x55d40c85b580 2024-04-25T01:42:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.039+0000 7fbf36ffd640 1 -- 172.21.15.28:0/828399763 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf48010810 con 0x55d40c85b580 2024-04-25T01:42:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.039+0000 7fbf5541a640 1 --2- 172.21.15.28:0/3560630763 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbf380a2d80 0x7fbf380a5240 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbf48020000 tx=0x7fbf4801d000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.039+0000 7fbf59b8f740 1 -- 172.21.15.28:0/828399763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 msgr2=0x55d40c85b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.039+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/828399763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 0x55d40c85b960 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7fbf48009f30 tx=0x7fbf4800f7f0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.039+0000 7fbf59b8f740 1 -- 172.21.15.28:0/828399763 shutdown_connections 2024-04-25T01:42:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.039+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/828399763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c866570 0x55d40c868960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.039+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/828399763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c85bea0 0x55d40c866030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.039+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/828399763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 0x55d40c85b960 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.039+0000 7fbf59b8f740 1 -- 172.21.15.28:0/828399763 >> 172.21.15.28:0/828399763 conn(0x55d40c8578a0 msgr2=0x55d40c857c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.039+0000 7fbf59b8f740 1 -- 172.21.15.28:0/828399763 shutdown_connections 2024-04-25T01:42:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.039+0000 7fbf59b8f740 1 -- 172.21.15.28:0/828399763 wait complete. 2024-04-25T01:42:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.040+0000 7fbf59b8f740 1 -- start start 2024-04-25T01:42:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.040+0000 7fbf59b8f740 1 Processor -- start 2024-04-25T01:42:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.040+0000 7fbf59b8f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c85b580 0x55d40c865ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.040+0000 7fbf59b8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85bea0 0x55d40c8631c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.040+0000 7fbf59b8f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c866570 0x55d40c863700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.040+0000 7fbf59b8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d40c869360 con 0x55d40c85bea0 2024-04-25T01:42:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.040+0000 7fbf59b8f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d40c77c050 con 0x55d40c866570 2024-04-25T01:42:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.040+0000 7fbf59b8f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55d40c8694e0 con 0x55d40c85b580 2024-04-25T01:42:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.040+0000 7fbf59b8f740 10 client.0 _ll_register_callbacks cb 0x55d40c4d3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.040+0000 7fbf4ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85bea0 0x55d40c8631c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.040+0000 7fbf4ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85bea0 0x55d40c8631c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45892/0 (socket says 172.21.15.28:45892) 2024-04-25T01:42:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.040+0000 7fbf4ffff640 1 -- 172.21.15.28:0/680094164 learned_addr learned my addr 172.21.15.28:0/680094164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.040+0000 7fbf5541a640 1 --2- 172.21.15.28:0/680094164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c866570 0x55d40c863700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.040+0000 7fbf54c19640 1 --2- 172.21.15.28:0/680094164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c85b580 0x55d40c865ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.040+0000 7fbf4ffff640 1 -- 172.21.15.28:0/680094164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c85b580 msgr2=0x55d40c865ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf4ffff640 1 --2- 172.21.15.28:0/680094164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c85b580 0x55d40c865ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf4ffff640 1 -- 172.21.15.28:0/680094164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c866570 msgr2=0x55d40c863700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf4ffff640 1 --2- 172.21.15.28:0/680094164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c866570 0x55d40c863700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf4ffff640 1 -- 172.21.15.28:0/680094164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d40c74ddc0 con 0x55d40c85bea0 2024-04-25T01:42:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf5541a640 1 --2- 172.21.15.28:0/680094164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c866570 0x55d40c863700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf54c19640 1 --2- 172.21.15.28:0/680094164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c85b580 0x55d40c865ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf4ffff640 1 --2- 172.21.15.28:0/680094164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85bea0 0x55d40c8631c0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fbf4400b9f0 tx=0x7fbf4400beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b72742e00ab56c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf2ffff640 1 -- 172.21.15.28:0/680094164 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4401b640 con 0x55d40c85bea0 2024-04-25T01:42:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf2ffff640 1 -- 172.21.15.28:0/680094164 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fbf4401b7e0 con 0x55d40c85bea0 2024-04-25T01:42:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf59b8f740 1 -- 172.21.15.28:0/680094164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf4801a8f0 con 0x55d40c85bea0 2024-04-25T01:42:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf2ffff640 1 -- 172.21.15.28:0/680094164 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbf44024050 con 0x55d40c85bea0 2024-04-25T01:42:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf59b8f740 10 client.18909 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf59b8f740 1 -- 172.21.15.28:0/680094164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d40c4d43f0 con 0x55d40c85bea0 2024-04-25T01:42:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf2ffff640 1 -- 172.21.15.28:0/680094164 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf4801a8f0 con 0x55d40c85bea0 2024-04-25T01:42:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf2ffff640 1 -- 172.21.15.28:0/680094164 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbf44019020 con 0x55d40c85bea0 2024-04-25T01:42:04.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf59b8f740 10 client.18909 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:04.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf59b8f740 10 client.18909 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:42:04.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf59b8f740 20 client.18909 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:04.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/680094164 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55d40c8765a0 0x55d40c8789d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf59b8f740 4 client.18909 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:04.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.041+0000 7fbf59b8f740 1 -- 172.21.15.28:0/680094164 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d40c7cdbb0 con 0x55d40c8765a0 2024-04-25T01:42:04.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.042+0000 7fbf54c19640 1 --2- 172.21.15.28:0/680094164 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55d40c8765a0 0x55d40c8789d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.042+0000 7fbf54c19640 1 --2- 172.21.15.28:0/680094164 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55d40c8765a0 0x55d40c8789d0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=773d0e3db5f19a7a server_cookie=1cda41e386ced582 in_seq=0 out_seq=0 2024-04-25T01:42:04.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.042+0000 7fbf2ffff640 10 client.18909 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:42:04.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.043+0000 7fbf2ffff640 1 -- 172.21.15.28:0/680094164 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d40c7cdbb0 con 0x55d40c8765a0 2024-04-25T01:42:04.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.043+0000 7fbf2ffff640 10 client.18909 handle_command_reply: tid=0 2024-04-25T01:42:04.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.043+0000 7fbf59b8f740 1 client.18909 shutdown 2024-04-25T01:42:04.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.043+0000 7fbf59b8f740 1 -- 172.21.15.28:0/680094164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85bea0 msgr2=0x55d40c8631c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.043+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/680094164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85bea0 0x55d40c8631c0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fbf4400b9f0 tx=0x7fbf4400beb0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.043+0000 7fbf59b8f740 1 -- 172.21.15.28:0/680094164 shutdown_connections 2024-04-25T01:42:04.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.043+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/680094164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c866570 0x55d40c863700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.043+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/680094164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85bea0 0x55d40c8631c0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.043+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/680094164 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55d40c8765a0 0x55d40c8789d0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.043+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/680094164 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c85b580 0x55d40c865ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.043+0000 7fbf59b8f740 1 -- 172.21.15.28:0/680094164 >> 172.21.15.28:0/680094164 conn(0x55d40c8578a0 msgr2=0x55d40c856b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:04.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.043+0000 7fbf2ffff640 0 client.18909 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:42:04.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.043+0000 7fbf59b8f740 1 -- 172.21.15.28:0/680094164 shutdown_connections 2024-04-25T01:42:04.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.043+0000 7fbf59b8f740 1 -- 172.21.15.28:0/680094164 wait complete. 2024-04-25T01:42:04.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.044+0000 7fbf59b8f740 20 client.18909 trim_cache size 0 max 16384 2024-04-25T01:42:04.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf59b8f740 1 Processor -- start 2024-04-25T01:42:04.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf59b8f740 1 -- start start 2024-04-25T01:42:04.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf59b8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 0x55d40c87e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf59b8f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c85bea0 0x55d40c87e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf59b8f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c866570 0x55d40c86e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf59b8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf501aebe0 con 0x55d40c85b580 2024-04-25T01:42:04.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf59b8f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbf501b0720 con 0x55d40c866570 2024-04-25T01:42:04.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf59b8f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbf501af2b0 con 0x55d40c85bea0 2024-04-25T01:42:04.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf54c19640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 0x55d40c87e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf54c19640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 0x55d40c87e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45898/0 (socket says 172.21.15.28:45898) 2024-04-25T01:42:04.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf54c19640 1 -- 172.21.15.28:0/440337416 learned_addr learned my addr 172.21.15.28:0/440337416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:04.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf5541a640 1 --2- 172.21.15.28:0/440337416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c866570 0x55d40c86e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf4ffff640 1 --2- 172.21.15.28:0/440337416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c85bea0 0x55d40c87e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf54c19640 1 -- 172.21.15.28:0/440337416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c85bea0 msgr2=0x55d40c87e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf54c19640 1 --2- 172.21.15.28:0/440337416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c85bea0 0x55d40c87e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf54c19640 1 -- 172.21.15.28:0/440337416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c866570 msgr2=0x55d40c86e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf54c19640 1 --2- 172.21.15.28:0/440337416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c866570 0x55d40c86e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf54c19640 1 -- 172.21.15.28:0/440337416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d40c59e7d0 con 0x55d40c85b580 2024-04-25T01:42:04.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf5541a640 1 --2- 172.21.15.28:0/440337416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c866570 0x55d40c86e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:04.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf4ffff640 1 --2- 172.21.15.28:0/440337416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c85bea0 0x55d40c87e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:04.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.057+0000 7fbf54c19640 1 --2- 172.21.15.28:0/440337416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 0x55d40c87e2e0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fbf400b82a0 tx=0x7fbf400dc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6374cd2a04c17a67 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:04.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.058+0000 7fbf367fc640 1 -- 172.21.15.28:0/440337416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf400be030 con 0x55d40c85b580 2024-04-25T01:42:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.058+0000 7fbf367fc640 1 -- 172.21.15.28:0/440337416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf400bc540 con 0x55d40c85b580 2024-04-25T01:42:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.058+0000 7fbf367fc640 1 -- 172.21.15.28:0/440337416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf400ba140 con 0x55d40c85b580 2024-04-25T01:42:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.058+0000 7fbf59b8f740 1 -- 172.21.15.28:0/440337416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 msgr2=0x55d40c87e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.058+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/440337416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 0x55d40c87e2e0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fbf400b82a0 tx=0x7fbf400dc040 comp rx=0 tx=0).stop 2024-04-25T01:42:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.058+0000 7fbf59b8f740 1 -- 172.21.15.28:0/440337416 shutdown_connections 2024-04-25T01:42:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.058+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/440337416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c866570 0x55d40c86e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.058+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/440337416 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c85bea0 0x55d40c87e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.058+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/440337416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 0x55d40c87e2e0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.058+0000 7fbf59b8f740 1 -- 172.21.15.28:0/440337416 >> 172.21.15.28:0/440337416 conn(0x55d40c8578a0 msgr2=0x55d40c85d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.058+0000 7fbf59b8f740 1 -- 172.21.15.28:0/440337416 shutdown_connections 2024-04-25T01:42:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.058+0000 7fbf59b8f740 1 -- 172.21.15.28:0/440337416 wait complete. 2024-04-25T01:42:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.058+0000 7fbf59b8f740 1 -- start start 2024-04-25T01:42:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.059+0000 7fbf59b8f740 1 Processor -- start 2024-04-25T01:42:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.059+0000 7fbf59b8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 0x55d40c86e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.059+0000 7fbf59b8f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c85bea0 0x55d40c86eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.059+0000 7fbf59b8f740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c866570 0x55d40c86b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.059+0000 7fbf59b8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf501b0050 con 0x55d40c85b580 2024-04-25T01:42:04.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.059+0000 7fbf59b8f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbf501af980 con 0x55d40c85bea0 2024-04-25T01:42:04.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.059+0000 7fbf59b8f740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fbf501acbe0 con 0x55d40c866570 2024-04-25T01:42:04.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.059+0000 7fbf59b8f740 10 client.0 _ll_register_callbacks cb 0x7fbf44004140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:04.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.059+0000 7fbf54c19640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 0x55d40c86e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.059+0000 7fbf54c19640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 0x55d40c86e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45912/0 (socket says 172.21.15.28:45912) 2024-04-25T01:42:04.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.059+0000 7fbf54c19640 1 -- 172.21.15.28:0/720702082 learned_addr learned my addr 172.21.15.28:0/720702082 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:04.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.059+0000 7fbf4ffff640 1 --2- 172.21.15.28:0/720702082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c85bea0 0x55d40c86eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.059+0000 7fbf5541a640 1 --2- 172.21.15.28:0/720702082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c866570 0x55d40c86b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.059+0000 7fbf54c19640 1 -- 172.21.15.28:0/720702082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c866570 msgr2=0x55d40c86b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.059+0000 7fbf54c19640 1 --2- 172.21.15.28:0/720702082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c866570 0x55d40c86b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.059+0000 7fbf54c19640 1 -- 172.21.15.28:0/720702082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c85bea0 msgr2=0x55d40c86eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.059+0000 7fbf54c19640 1 --2- 172.21.15.28:0/720702082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c85bea0 0x55d40c86eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.059+0000 7fbf54c19640 1 -- 172.21.15.28:0/720702082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d40c86bda0 con 0x55d40c85b580 2024-04-25T01:42:04.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.060+0000 7fbf54c19640 1 --2- 172.21.15.28:0/720702082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 0x55d40c86e760 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fbf400b8810 tx=0x7fbf400bb870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cca101c58a7eefb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:04.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.060+0000 7fbf2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:04.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.060+0000 7fbf2ffff640 1 -- 172.21.15.28:0/720702082 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf400be070 con 0x55d40c85b580 2024-04-25T01:42:04.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.060+0000 7fbf2ffff640 1 -- 172.21.15.28:0/720702082 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fbf4003b460 con 0x55d40c85b580 2024-04-25T01:42:04.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.060+0000 7fbf59b8f740 1 -- 172.21.15.28:0/720702082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf400bc540 con 0x55d40c85b580 2024-04-25T01:42:04.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.060+0000 7fbf2ffff640 1 -- 172.21.15.28:0/720702082 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbf400c4ab0 con 0x55d40c85b580 2024-04-25T01:42:04.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.060+0000 7fbf59b8f740 10 client.18915 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:04.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.060+0000 7fbf59b8f740 1 -- 172.21.15.28:0/720702082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d40c8dd410 con 0x55d40c85b580 2024-04-25T01:42:04.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.060+0000 7fbf2ffff640 1 -- 172.21.15.28:0/720702082 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf400bc540 con 0x55d40c85b580 2024-04-25T01:42:04.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.060+0000 7fbf2ffff640 1 -- 172.21.15.28:0/720702082 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbf4003bc80 con 0x55d40c85b580 2024-04-25T01:42:04.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.060+0000 7fbf59b8f740 10 client.18915 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:04.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.060+0000 7fbf59b8f740 10 client.18915 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:42:04.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.060+0000 7fbf59b8f740 20 client.18915 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:04.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.060+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/720702082 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55d40c8ddb50 0x55d40c8dff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.060+0000 7fbf59b8f740 4 client.18915 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:42:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.060+0000 7fbf59b8f740 1 -- 172.21.15.28:0/720702082 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d40c7c7f00 con 0x55d40c8ddb50 2024-04-25T01:42:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.061+0000 7fbf4ffff640 1 --2- 172.21.15.28:0/720702082 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55d40c8ddb50 0x55d40c8dff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.061+0000 7fbf4ffff640 1 --2- 172.21.15.28:0/720702082 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55d40c8ddb50 0x55d40c8dff70 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5c7ebff118dd75c9 server_cookie=27f299b2b0c7715f in_seq=0 out_seq=0 2024-04-25T01:42:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.061+0000 7fbf2ffff640 10 client.18915 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:42:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.061+0000 7fbf2ffff640 1 -- 172.21.15.28:0/720702082 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d40c7c7f00 con 0x55d40c8ddb50 2024-04-25T01:42:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.061+0000 7fbf2ffff640 10 client.18915 handle_command_reply: tid=0 2024-04-25T01:42:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.061+0000 7fbf59b8f740 1 client.18915 shutdown 2024-04-25T01:42:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.061+0000 7fbf59b8f740 1 -- 172.21.15.28:0/720702082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 msgr2=0x55d40c86e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.061+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/720702082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 0x55d40c86e760 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fbf400b8810 tx=0x7fbf400bb870 comp rx=0 tx=0).stop 2024-04-25T01:42:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.061+0000 7fbf59b8f740 1 -- 172.21.15.28:0/720702082 shutdown_connections 2024-04-25T01:42:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.061+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/720702082 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55d40c8ddb50 0x55d40c8dff70 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.061+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/720702082 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d40c866570 0x55d40c86b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.061+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/720702082 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40c85bea0 0x55d40c86eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.061+0000 7fbf59b8f740 1 --2- 172.21.15.28:0/720702082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d40c85b580 0x55d40c86e760 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.061+0000 7fbf2ffff640 0 client.18915 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:42:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.061+0000 7fbf59b8f740 1 -- 172.21.15.28:0/720702082 >> 172.21.15.28:0/720702082 conn(0x55d40c8578a0 msgr2=0x55d40c8562b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.062+0000 7fbf59b8f740 1 -- 172.21.15.28:0/720702082 shutdown_connections 2024-04-25T01:42:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.062+0000 7fbf59b8f740 1 -- 172.21.15.28:0/720702082 wait complete. 2024-04-25T01:42:04.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.062+0000 7fbf59b8f740 20 client.18915 trim_cache size 0 max 16384 2024-04-25T01:42:04.055 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:42:04.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.062+0000 7fbf56e7d640 1 -- 172.21.15.28:0/3560630763 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbf380a2d80 msgr2=0x7fbf380a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.062+0000 7fbf56e7d640 1 --2- 172.21.15.28:0/3560630763 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbf380a2d80 0x7fbf380a5240 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbf48020000 tx=0x7fbf4801d000 comp rx=0 tx=0).stop 2024-04-25T01:42:04.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.062+0000 7fbf56e7d640 1 -- 172.21.15.28:0/3560630763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf50100610 msgr2=0x7fbf50197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.062+0000 7fbf56e7d640 1 --2- 172.21.15.28:0/3560630763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf50100610 0x7fbf50197fb0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fbf40005940 tx=0x7fbf400042f0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.062+0000 7fbf56e7d640 1 -- 172.21.15.28:0/3560630763 shutdown_connections 2024-04-25T01:42:04.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.062+0000 7fbf56e7d640 1 --2- 172.21.15.28:0/3560630763 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fbf380a2d80 0x7fbf380a5240 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.062+0000 7fbf56e7d640 1 --2- 172.21.15.28:0/3560630763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf501066b0 0x7fbf5019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.062+0000 7fbf56e7d640 1 --2- 172.21.15.28:0/3560630763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fbf50100fd0 0x7fbf501984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.062+0000 7fbf56e7d640 1 --2- 172.21.15.28:0/3560630763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf50100610 0x7fbf50197fb0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.062+0000 7fbf56e7d640 1 -- 172.21.15.28:0/3560630763 >> 172.21.15.28:0/3560630763 conn(0x7fbf50077740 msgr2=0x7fbf500ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:04.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.062+0000 7fbf56e7d640 1 -- 172.21.15.28:0/3560630763 shutdown_connections 2024-04-25T01:42:04.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.062+0000 7fbf56e7d640 1 -- 172.21.15.28:0/3560630763 wait complete. 2024-04-25T01:42:04.060 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:42:04.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.177+0000 7ff003409640 1 -- 172.21.15.28:0/2880283468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feffc100620 msgr2=0x7feffc100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.177+0000 7ff003409640 1 --2- 172.21.15.28:0/2880283468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feffc100620 0x7feffc100aa0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fefec009900 tx=0x7fefec02a410 comp rx=0 tx=0).stop 2024-04-25T01:42:04.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.177+0000 7ff003409640 1 -- 172.21.15.28:0/2880283468 shutdown_connections 2024-04-25T01:42:04.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.177+0000 7ff003409640 1 --2- 172.21.15.28:0/2880283468 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feffc100fe0 0x7feffc10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.177+0000 7ff003409640 1 --2- 172.21.15.28:0/2880283468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feffc100620 0x7feffc100aa0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.177+0000 7ff003409640 1 --2- 172.21.15.28:0/2880283468 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feffc106630 0x7feffc106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.177+0000 7ff003409640 1 -- 172.21.15.28:0/2880283468 >> 172.21.15.28:0/2880283468 conn(0x7feffc0776a0 msgr2=0x7feffc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:04.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.177+0000 7ff003409640 1 -- 172.21.15.28:0/2880283468 shutdown_connections 2024-04-25T01:42:04.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.177+0000 7ff003409640 1 -- 172.21.15.28:0/2880283468 wait complete. 2024-04-25T01:42:04.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.178+0000 7ff003409640 1 Processor -- start 2024-04-25T01:42:04.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.178+0000 7ff003409640 1 -- start start 2024-04-25T01:42:04.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.178+0000 7ff003409640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feffc100620 0x7feffc198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.178+0000 7ff003409640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feffc100fe0 0x7feffc198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.178+0000 7ff003409640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feffc106630 0x7feffc19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.178+0000 7ff003409640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feffc111b00 con 0x7feffc106630 2024-04-25T01:42:04.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.178+0000 7ff003409640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7feffc111980 con 0x7feffc100fe0 2024-04-25T01:42:04.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.178+0000 7ff003409640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7feffc111c80 con 0x7feffc100620 2024-04-25T01:42:04.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.178+0000 7ff0019a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feffc106630 0x7feffc19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.178+0000 7ff0019a6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feffc106630 0x7feffc19c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45918/0 (socket says 172.21.15.28:45918) 2024-04-25T01:42:04.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.178+0000 7ff0019a6640 1 -- 172.21.15.28:0/751323993 learned_addr learned my addr 172.21.15.28:0/751323993 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:04.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.178+0000 7ff0009a4640 1 --2- 172.21.15.28:0/751323993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feffc100fe0 0x7feffc198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.178+0000 7ff0011a5640 1 --2- 172.21.15.28:0/751323993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feffc100620 0x7feffc198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.178+0000 7ff0019a6640 1 -- 172.21.15.28:0/751323993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feffc100620 msgr2=0x7feffc198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.178+0000 7ff0019a6640 1 --2- 172.21.15.28:0/751323993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feffc100620 0x7feffc198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.178+0000 7ff0019a6640 1 -- 172.21.15.28:0/751323993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feffc100fe0 msgr2=0x7feffc198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.178+0000 7ff0019a6640 1 --2- 172.21.15.28:0/751323993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feffc100fe0 0x7feffc198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.178+0000 7ff0019a6640 1 -- 172.21.15.28:0/751323993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feffc19cff0 con 0x7feffc106630 2024-04-25T01:42:04.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.178+0000 7ff0009a4640 1 --2- 172.21.15.28:0/751323993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feffc100fe0 0x7feffc198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:04.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.179+0000 7ff0019a6640 1 --2- 172.21.15.28:0/751323993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feffc106630 0x7feffc19c8c0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7feff400eaa0 tx=0x7feff400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:04.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.179+0000 7feff27fc640 1 -- 172.21.15.28:0/751323993 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff400cdb0 con 0x7feffc106630 2024-04-25T01:42:04.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.179+0000 7feff27fc640 1 -- 172.21.15.28:0/751323993 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feff4005020 con 0x7feffc106630 2024-04-25T01:42:04.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.179+0000 7ff003409640 1 -- 172.21.15.28:0/751323993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feffc19d2e0 con 0x7feffc106630 2024-04-25T01:42:04.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.179+0000 7feff27fc640 1 -- 172.21.15.28:0/751323993 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff40116b0 con 0x7feffc106630 2024-04-25T01:42:04.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.179+0000 7ff003409640 1 -- 172.21.15.28:0/751323993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feffc1a4b70 con 0x7feffc106630 2024-04-25T01:42:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.181+0000 7feff27fc640 1 -- 172.21.15.28:0/751323993 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7feff4005560 con 0x7feffc106630 2024-04-25T01:42:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.182+0000 7ff0061f0740 1 Processor -- start 2024-04-25T01:42:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.182+0000 7ff0061f0740 1 -- start start 2024-04-25T01:42:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.182+0000 7ff0061f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a05580 0x5595c9a05960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.182+0000 7ff0061f0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c9a05ea0 0x5595c9a10030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.182+0000 7ff0061f0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a10570 0x5595c9a12960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.182+0000 7ff0061f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5595c996a670 con 0x5595c9a05580 2024-04-25T01:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.182+0000 7ff0061f0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5595c9912970 con 0x5595c9a05ea0 2024-04-25T01:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.182+0000 7ff0061f0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5595c9966630 con 0x5595c9a10570 2024-04-25T01:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.182+0000 7feff27fc640 1 --2- 172.21.15.28:0/751323993 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fefe80a2d80 0x7fefe80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.182+0000 7ff0009a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a05580 0x5595c9a05960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.182+0000 7ff0009a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a05580 0x5595c9a05960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45930/0 (socket says 172.21.15.28:45930) 2024-04-25T01:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.182+0000 7ff0009a4640 1 -- 172.21.15.28:0/2586415622 learned_addr learned my addr 172.21.15.28:0/2586415622 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.182+0000 7ff0019a6640 1 --2- 172.21.15.28:0/751323993 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fefe80a2d80 0x7fefe80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.182+0000 7ff0011a5640 1 --2- 172.21.15.28:0/2586415622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c9a05ea0 0x5595c9a10030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.182+0000 7ff0009a4640 1 --2- 172.21.15.28:0/2586415622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a10570 0x5595c9a12960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.182+0000 7feff27fc640 1 -- 172.21.15.28:0/751323993 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7feff40cfae0 con 0x7feffc106630 2024-04-25T01:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.183+0000 7ff0009a4640 1 -- 172.21.15.28:0/2586415622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a10570 msgr2=0x5595c9a12960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.183+0000 7ff0009a4640 1 --2- 172.21.15.28:0/2586415622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a10570 0x5595c9a12960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.183+0000 7ff0009a4640 1 -- 172.21.15.28:0/2586415622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c9a05ea0 msgr2=0x5595c9a10030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.183+0000 7ff0009a4640 1 --2- 172.21.15.28:0/2586415622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c9a05ea0 0x5595c9a10030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.183+0000 7ff0009a4640 1 -- 172.21.15.28:0/2586415622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595c9936400 con 0x5595c9a05580 2024-04-25T01:42:04.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.183+0000 7ff0009a4640 1 --2- 172.21.15.28:0/2586415622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a05580 0x5595c9a05960 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fefec031550 tx=0x7fefec005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=505318bd3daf369f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:04.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.183+0000 7ff0019a6640 1 --2- 172.21.15.28:0/751323993 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fefe80a2d80 0x7fefe80a5240 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7feff400efd0 tx=0x7feff40054d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:04.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.183+0000 7fefe77fe640 1 -- 172.21.15.28:0/2586415622 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec035050 con 0x5595c9a05580 2024-04-25T01:42:04.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.183+0000 7fefe77fe640 1 -- 172.21.15.28:0/2586415622 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefec031900 con 0x5595c9a05580 2024-04-25T01:42:04.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.183+0000 7fefe77fe640 1 -- 172.21.15.28:0/2586415622 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec0046c0 con 0x5595c9a05580 2024-04-25T01:42:04.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.184+0000 7ff0061f0740 1 -- 172.21.15.28:0/2586415622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a05580 msgr2=0x5595c9a05960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.184+0000 7ff0061f0740 1 --2- 172.21.15.28:0/2586415622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a05580 0x5595c9a05960 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fefec031550 tx=0x7fefec005bd0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.184+0000 7ff0061f0740 1 -- 172.21.15.28:0/2586415622 shutdown_connections 2024-04-25T01:42:04.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.184+0000 7ff0061f0740 1 --2- 172.21.15.28:0/2586415622 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a10570 0x5595c9a12960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.184+0000 7ff0061f0740 1 --2- 172.21.15.28:0/2586415622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c9a05ea0 0x5595c9a10030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.184+0000 7ff0061f0740 1 --2- 172.21.15.28:0/2586415622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a05580 0x5595c9a05960 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.184+0000 7ff0061f0740 1 -- 172.21.15.28:0/2586415622 >> 172.21.15.28:0/2586415622 conn(0x5595c9a018a0 msgr2=0x5595c9a01c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:04.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.184+0000 7ff0061f0740 1 -- 172.21.15.28:0/2586415622 shutdown_connections 2024-04-25T01:42:04.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.184+0000 7ff0061f0740 1 -- 172.21.15.28:0/2586415622 wait complete. 2024-04-25T01:42:04.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.184+0000 7ff0061f0740 1 -- start start 2024-04-25T01:42:04.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.184+0000 7ff0061f0740 1 Processor -- start 2024-04-25T01:42:04.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7ff0061f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a05580 0x5595c9a0fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7ff0061f0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a05ea0 0x5595c9a0d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7ff0061f0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c9a10570 0x5595c9a0d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7ff0061f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5595c9a13300 con 0x5595c9a05580 2024-04-25T01:42:04.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7ff0061f0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5595c9926050 con 0x5595c9a10570 2024-04-25T01:42:04.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7ff0061f0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5595c9a13480 con 0x5595c9a05ea0 2024-04-25T01:42:04.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7ff0061f0740 10 client.0 _ll_register_callbacks cb 0x5595c967dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:04.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7ff0009a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a05580 0x5595c9a0fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7ff0009a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a05580 0x5595c9a0fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45936/0 (socket says 172.21.15.28:45936) 2024-04-25T01:42:04.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7ff0009a4640 1 -- 172.21.15.28:0/869612673 learned_addr learned my addr 172.21.15.28:0/869612673 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:04.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7ff0011a5640 1 --2- 172.21.15.28:0/869612673 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a05ea0 0x5595c9a0d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7ff0009a4640 1 -- 172.21.15.28:0/869612673 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a05ea0 msgr2=0x5595c9a0d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7ff0009a4640 1 --2- 172.21.15.28:0/869612673 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a05ea0 0x5595c9a0d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7ff0009a4640 1 -- 172.21.15.28:0/869612673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c9a10570 msgr2=0x5595c9a0d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7ff0009a4640 1 --2- 172.21.15.28:0/869612673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c9a10570 0x5595c9a0d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7ff0009a4640 1 -- 172.21.15.28:0/869612673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595c98f7dc0 con 0x5595c9a05580 2024-04-25T01:42:04.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7ff0011a5640 1 --2- 172.21.15.28:0/869612673 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a05ea0 0x5595c9a0d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:04.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7ff0009a4640 1 --2- 172.21.15.28:0/869612673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a05580 0x5595c9a0fe50 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fefec0053f0 tx=0x7fefec009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e0e75cf76789325 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:04.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7fefe4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:04.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7fefe4ff9640 1 -- 172.21.15.28:0/869612673 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec04c070 con 0x5595c9a05580 2024-04-25T01:42:04.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7fefe4ff9640 1 -- 172.21.15.28:0/869612673 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fefec03da10 con 0x5595c9a05580 2024-04-25T01:42:04.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.185+0000 7ff0061f0740 1 -- 172.21.15.28:0/869612673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefec031900 con 0x5595c9a05580 2024-04-25T01:42:04.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.186+0000 7fefe4ff9640 1 -- 172.21.15.28:0/869612673 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fefec058050 con 0x5595c9a05580 2024-04-25T01:42:04.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.186+0000 7ff0061f0740 10 client.18927 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:04.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.186+0000 7ff0061f0740 1 -- 172.21.15.28:0/869612673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5595c967e3f0 con 0x5595c9a05580 2024-04-25T01:42:04.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.186+0000 7fefe4ff9640 1 -- 172.21.15.28:0/869612673 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefec031900 con 0x5595c9a05580 2024-04-25T01:42:04.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.186+0000 7fefe4ff9640 1 -- 172.21.15.28:0/869612673 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fefec039410 con 0x5595c9a05580 2024-04-25T01:42:04.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.186+0000 7ff0061f0740 10 client.18927 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:04.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.186+0000 7ff0061f0740 10 client.18927 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:42:04.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.186+0000 7ff0061f0740 20 client.18927 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:04.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.186+0000 7ff0061f0740 1 --2- 172.21.15.28:0/869612673 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5595c9a203f0 0x5595c9a227e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.186+0000 7ff0061f0740 4 client.18927 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:04.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.186+0000 7ff0061f0740 1 -- 172.21.15.28:0/869612673 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595c9977bb0 con 0x5595c9a203f0 2024-04-25T01:42:04.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.186+0000 7ff0011a5640 1 --2- 172.21.15.28:0/869612673 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5595c9a203f0 0x5595c9a227e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.187+0000 7ff0011a5640 1 --2- 172.21.15.28:0/869612673 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5595c9a203f0 0x5595c9a227e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2c87d04bded9caf2 server_cookie=93b3641f7e1c033 in_seq=0 out_seq=0 2024-04-25T01:42:04.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.187+0000 7fefe4ff9640 10 client.18927 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:42:04.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.188+0000 7fefe4ff9640 1 -- 172.21.15.28:0/869612673 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5595c9977bb0 con 0x5595c9a203f0 2024-04-25T01:42:04.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.188+0000 7fefe4ff9640 10 client.18927 handle_command_reply: tid=0 2024-04-25T01:42:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.188+0000 7ff0061f0740 1 client.18927 shutdown 2024-04-25T01:42:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.188+0000 7ff0061f0740 1 -- 172.21.15.28:0/869612673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a05580 msgr2=0x5595c9a0fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.188+0000 7ff0061f0740 1 --2- 172.21.15.28:0/869612673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a05580 0x5595c9a0fe50 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fefec0053f0 tx=0x7fefec009520 comp rx=0 tx=0).stop 2024-04-25T01:42:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.188+0000 7ff0061f0740 1 -- 172.21.15.28:0/869612673 shutdown_connections 2024-04-25T01:42:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.188+0000 7ff0061f0740 1 --2- 172.21.15.28:0/869612673 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5595c9a203f0 0x5595c9a227e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.188+0000 7ff0061f0740 1 --2- 172.21.15.28:0/869612673 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c9a10570 0x5595c9a0d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.188+0000 7fefe4ff9640 0 client.18927 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:42:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.188+0000 7ff0061f0740 1 --2- 172.21.15.28:0/869612673 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a05ea0 0x5595c9a0d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.188+0000 7ff0061f0740 1 --2- 172.21.15.28:0/869612673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a05580 0x5595c9a0fe50 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.188+0000 7ff0061f0740 1 -- 172.21.15.28:0/869612673 >> 172.21.15.28:0/869612673 conn(0x5595c9a018a0 msgr2=0x5595c9a00b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.188+0000 7ff0061f0740 1 -- 172.21.15.28:0/869612673 shutdown_connections 2024-04-25T01:42:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.188+0000 7ff0061f0740 1 -- 172.21.15.28:0/869612673 wait complete. 2024-04-25T01:42:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.188+0000 7ff0061f0740 20 client.18927 trim_cache size 0 max 16384 2024-04-25T01:42:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.201+0000 7ff0061f0740 1 Processor -- start 2024-04-25T01:42:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.201+0000 7ff0061f0740 1 -- start start 2024-04-25T01:42:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7ff0061f0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c9a05580 0x5595c9a1c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7ff0061f0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a05ea0 0x5595c99fc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7ff0061f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a10570 0x5595c9a0f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7ff0061f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feffc1aeae0 con 0x5595c9a10570 2024-04-25T01:42:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7ff0061f0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7feffc1b0620 con 0x5595c9a05580 2024-04-25T01:42:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7ff0061f0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7feffc1af1b0 con 0x5595c9a05ea0 2024-04-25T01:42:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7ff0009a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a10570 0x5595c9a0f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7ff0009a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a10570 0x5595c9a0f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45950/0 (socket says 172.21.15.28:45950) 2024-04-25T01:42:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7ff0009a4640 1 -- 172.21.15.28:0/309623948 learned_addr learned my addr 172.21.15.28:0/309623948 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7ff0011a5640 1 --2- 172.21.15.28:0/309623948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a05ea0 0x5595c99fc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7ff0009a4640 1 --2- 172.21.15.28:0/309623948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c9a05580 0x5595c9a1c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7ff0009a4640 1 -- 172.21.15.28:0/309623948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a05ea0 msgr2=0x5595c99fc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7ff0009a4640 1 --2- 172.21.15.28:0/309623948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a05ea0 0x5595c99fc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7ff0009a4640 1 -- 172.21.15.28:0/309623948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c9a05580 msgr2=0x5595c9a1c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7ff0009a4640 1 --2- 172.21.15.28:0/309623948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c9a05580 0x5595c9a1c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7ff0009a4640 1 -- 172.21.15.28:0/309623948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595c9a0c0f0 con 0x5595c9a10570 2024-04-25T01:42:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7ff0011a5640 1 --2- 172.21.15.28:0/309623948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a05ea0 0x5595c99fc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:04.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7ff0009a4640 1 --2- 172.21.15.28:0/309623948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a10570 0x5595c9a0f980 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fefec0397e0 tx=0x7fefec005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49a0ca87901f33ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:04.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7fefe6ffd640 1 -- 172.21.15.28:0/309623948 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec04c070 con 0x5595c9a10570 2024-04-25T01:42:04.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7fefe6ffd640 1 -- 172.21.15.28:0/309623948 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefec039c50 con 0x5595c9a10570 2024-04-25T01:42:04.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.202+0000 7fefe6ffd640 1 -- 172.21.15.28:0/309623948 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec035040 con 0x5595c9a10570 2024-04-25T01:42:04.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.203+0000 7ff0061f0740 1 -- 172.21.15.28:0/309623948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a10570 msgr2=0x5595c9a0f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.203+0000 7ff0061f0740 1 --2- 172.21.15.28:0/309623948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a10570 0x5595c9a0f980 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fefec0397e0 tx=0x7fefec005de0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.203+0000 7ff0061f0740 1 -- 172.21.15.28:0/309623948 shutdown_connections 2024-04-25T01:42:04.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.203+0000 7ff0061f0740 1 --2- 172.21.15.28:0/309623948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a10570 0x5595c9a0f980 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.203+0000 7ff0061f0740 1 --2- 172.21.15.28:0/309623948 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a05ea0 0x5595c99fc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.203+0000 7ff0061f0740 1 --2- 172.21.15.28:0/309623948 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c9a05580 0x5595c9a1c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.203+0000 7ff0061f0740 1 -- 172.21.15.28:0/309623948 >> 172.21.15.28:0/309623948 conn(0x5595c9a018a0 msgr2=0x5595c9a08740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:04.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.203+0000 7ff0061f0740 1 -- 172.21.15.28:0/309623948 shutdown_connections 2024-04-25T01:42:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.203+0000 7ff0061f0740 1 -- 172.21.15.28:0/309623948 wait complete. 2024-04-25T01:42:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.203+0000 7ff0061f0740 1 -- start start 2024-04-25T01:42:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.203+0000 7ff0061f0740 1 Processor -- start 2024-04-25T01:42:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7ff0061f0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c9a05580 0x5595c9a09dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7ff0061f0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a05ea0 0x5595c9a0a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7ff0061f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a10570 0x5595c9a0a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7ff0061f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feffc1aff50 con 0x5595c9a10570 2024-04-25T01:42:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7ff0061f0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7feffc1af880 con 0x5595c9a05580 2024-04-25T01:42:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7ff0061f0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7feffc1acae0 con 0x5595c9a05ea0 2024-04-25T01:42:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7ff0061f0740 10 client.0 _ll_register_callbacks cb 0x7fefec0315b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7ff0009a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a10570 0x5595c9a0a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7ff0009a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a10570 0x5595c9a0a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45962/0 (socket says 172.21.15.28:45962) 2024-04-25T01:42:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7ff0009a4640 1 -- 172.21.15.28:0/975400562 learned_addr learned my addr 172.21.15.28:0/975400562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7ff0011a5640 1 --2- 172.21.15.28:0/975400562 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a05ea0 0x5595c9a0a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7ff0009a4640 1 --2- 172.21.15.28:0/975400562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c9a05580 0x5595c9a09dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7ff0009a4640 1 -- 172.21.15.28:0/975400562 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a05ea0 msgr2=0x5595c9a0a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7ff0009a4640 1 --2- 172.21.15.28:0/975400562 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a05ea0 0x5595c9a0a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7ff0009a4640 1 -- 172.21.15.28:0/975400562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c9a05580 msgr2=0x5595c9a09dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7ff0009a4640 1 --2- 172.21.15.28:0/975400562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c9a05580 0x5595c9a09dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7ff0009a4640 1 -- 172.21.15.28:0/975400562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595c97487d0 con 0x5595c9a10570 2024-04-25T01:42:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7ff0009a4640 1 --2- 172.21.15.28:0/975400562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a10570 0x5595c9a0a850 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fefec03b010 tx=0x7fefec039510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f701bd495097a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7fefe4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7fefe4ff9640 1 -- 172.21.15.28:0/975400562 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec035070 con 0x5595c9a10570 2024-04-25T01:42:04.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7fefe4ff9640 1 -- 172.21.15.28:0/975400562 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fefec031890 con 0x5595c9a10570 2024-04-25T01:42:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.204+0000 7ff0061f0740 1 -- 172.21.15.28:0/975400562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefec039c50 con 0x5595c9a10570 2024-04-25T01:42:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.205+0000 7fefe4ff9640 1 -- 172.21.15.28:0/975400562 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fefec04c040 con 0x5595c9a10570 2024-04-25T01:42:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.205+0000 7ff0061f0740 10 client.18933 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.205+0000 7ff0061f0740 1 -- 172.21.15.28:0/975400562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5595c9a866d0 con 0x5595c9a10570 2024-04-25T01:42:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.205+0000 7fefe4ff9640 1 -- 172.21.15.28:0/975400562 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefec039c50 con 0x5595c9a10570 2024-04-25T01:42:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.205+0000 7fefe4ff9640 1 -- 172.21.15.28:0/975400562 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fefec0035f0 con 0x5595c9a10570 2024-04-25T01:42:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.205+0000 7ff0061f0740 10 client.18933 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.205+0000 7ff0061f0740 10 client.18933 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:42:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.205+0000 7ff0061f0740 20 client.18933 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.205+0000 7ff0061f0740 1 --2- 172.21.15.28:0/975400562 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5595c9a86d00 0x5595c9a89120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.205+0000 7ff0061f0740 4 client.18933 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:42:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.205+0000 7ff0061f0740 1 -- 172.21.15.28:0/975400562 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5595c9971f00 con 0x5595c9a86d00 2024-04-25T01:42:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.205+0000 7ff0011a5640 1 --2- 172.21.15.28:0/975400562 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5595c9a86d00 0x5595c9a89120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.206+0000 7fefe4ff9640 10 client.18933 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:42:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.206+0000 7ff0011a5640 1 --2- 172.21.15.28:0/975400562 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5595c9a86d00 0x5595c9a89120 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e7c40fc5d2c25f75 server_cookie=9c28d42a88617f82 in_seq=0 out_seq=0 2024-04-25T01:42:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.206+0000 7fefe4ff9640 1 -- 172.21.15.28:0/975400562 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5595c9971f00 con 0x5595c9a86d00 2024-04-25T01:42:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.206+0000 7fefe4ff9640 10 client.18933 handle_command_reply: tid=0 2024-04-25T01:42:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.206+0000 7ff0061f0740 1 client.18933 shutdown 2024-04-25T01:42:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.206+0000 7ff0061f0740 1 -- 172.21.15.28:0/975400562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a10570 msgr2=0x5595c9a0a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.206+0000 7ff0061f0740 1 --2- 172.21.15.28:0/975400562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a10570 0x5595c9a0a850 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fefec03b010 tx=0x7fefec039510 comp rx=0 tx=0).stop 2024-04-25T01:42:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.206+0000 7ff0061f0740 1 -- 172.21.15.28:0/975400562 shutdown_connections 2024-04-25T01:42:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.206+0000 7ff0061f0740 1 --2- 172.21.15.28:0/975400562 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5595c9a86d00 0x5595c9a89120 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.206+0000 7ff0061f0740 1 --2- 172.21.15.28:0/975400562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595c9a10570 0x5595c9a0a850 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.206+0000 7ff0061f0740 1 --2- 172.21.15.28:0/975400562 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5595c9a05ea0 0x5595c9a0a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.206+0000 7fefe4ff9640 0 client.18933 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:42:04.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.206+0000 7ff0061f0740 1 --2- 172.21.15.28:0/975400562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5595c9a05580 0x5595c9a09dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.206+0000 7ff0061f0740 1 -- 172.21.15.28:0/975400562 >> 172.21.15.28:0/975400562 conn(0x5595c9a018a0 msgr2=0x5595c9a06ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:04.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.206+0000 7ff0061f0740 1 -- 172.21.15.28:0/975400562 shutdown_connections 2024-04-25T01:42:04.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.206+0000 7ff0061f0740 1 -- 172.21.15.28:0/975400562 wait complete. 2024-04-25T01:42:04.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.206+0000 7ff0061f0740 20 client.18933 trim_cache size 0 max 16384 2024-04-25T01:42:04.202 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:42:04.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.207+0000 7ff003409640 1 -- 172.21.15.28:0/751323993 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fefe80a2d80 msgr2=0x7fefe80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.207+0000 7ff003409640 1 --2- 172.21.15.28:0/751323993 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fefe80a2d80 0x7fefe80a5240 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7feff400efd0 tx=0x7feff40054d0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.207+0000 7ff003409640 1 -- 172.21.15.28:0/751323993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feffc106630 msgr2=0x7feffc19c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.207+0000 7ff003409640 1 --2- 172.21.15.28:0/751323993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feffc106630 0x7feffc19c8c0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7feff400eaa0 tx=0x7feff400ef60 comp rx=0 tx=0).stop 2024-04-25T01:42:04.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.207+0000 7ff003409640 1 -- 172.21.15.28:0/751323993 shutdown_connections 2024-04-25T01:42:04.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.207+0000 7ff003409640 1 --2- 172.21.15.28:0/751323993 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fefe80a2d80 0x7fefe80a5240 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.207+0000 7ff003409640 1 --2- 172.21.15.28:0/751323993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feffc106630 0x7feffc19c8c0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.207+0000 7ff003409640 1 --2- 172.21.15.28:0/751323993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feffc100fe0 0x7feffc198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.207+0000 7ff003409640 1 --2- 172.21.15.28:0/751323993 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7feffc100620 0x7feffc198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.207+0000 7ff003409640 1 -- 172.21.15.28:0/751323993 >> 172.21.15.28:0/751323993 conn(0x7feffc0776a0 msgr2=0x7feffc1031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:04.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.207+0000 7ff003409640 1 -- 172.21.15.28:0/751323993 shutdown_connections 2024-04-25T01:42:04.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.207+0000 7ff003409640 1 -- 172.21.15.28:0/751323993 wait complete. 2024-04-25T01:42:04.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:04 smithi028 ceph-mon[29390]: pgmap v278: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-25T01:42:04.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:04 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3772072508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:42:04.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:42:04.296 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:04 smithi032 ceph-mon[32461]: pgmap v278: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-25T01:42:04.296 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:04 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3772072508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:42:04.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.321+0000 7fddfb35a640 1 -- 172.21.15.28:0/4192698997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddf4100540 msgr2=0x7fddf4100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.321+0000 7fddfb35a640 1 --2- 172.21.15.28:0/4192698997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddf4100540 0x7fddf4100920 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fddf0009ca0 tx=0x7fddf002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.321+0000 7fddfb35a640 1 -- 172.21.15.28:0/4192698997 shutdown_connections 2024-04-25T01:42:04.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.322+0000 7fddfb35a640 1 --2- 172.21.15.28:0/4192698997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddf410e0f0 0x7fddf41104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.322+0000 7fddfb35a640 1 --2- 172.21.15.28:0/4192698997 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddf4100e60 0x7fddf410dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.322+0000 7fddfb35a640 1 --2- 172.21.15.28:0/4192698997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddf4100540 0x7fddf4100920 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.322+0000 7fddfb35a640 1 -- 172.21.15.28:0/4192698997 >> 172.21.15.28:0/4192698997 conn(0x7fddf4077740 msgr2=0x7fddf4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:04.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.322+0000 7fddfb35a640 1 -- 172.21.15.28:0/4192698997 shutdown_connections 2024-04-25T01:42:04.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.322+0000 7fddfb35a640 1 -- 172.21.15.28:0/4192698997 wait complete. 2024-04-25T01:42:04.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.322+0000 7fddfb35a640 1 Processor -- start 2024-04-25T01:42:04.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.322+0000 7fddfb35a640 1 -- start start 2024-04-25T01:42:04.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.322+0000 7fddfb35a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddf4100540 0x7fddf419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.322+0000 7fddfb35a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddf4100e60 0x7fddf419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.322+0000 7fddfb35a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddf410e0f0 0x7fddf419ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.322+0000 7fddfb35a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fddf4113cf0 con 0x7fddf4100540 2024-04-25T01:42:04.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.322+0000 7fddfb35a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fddf4113b70 con 0x7fddf410e0f0 2024-04-25T01:42:04.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.322+0000 7fddfb35a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fddf4113e70 con 0x7fddf4100e60 2024-04-25T01:42:04.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.323+0000 7fddf90f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddf4100540 0x7fddf419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.323+0000 7fddf90f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddf4100540 0x7fddf419a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45988/0 (socket says 172.21.15.28:45988) 2024-04-25T01:42:04.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.323+0000 7fddf90f6640 1 -- 172.21.15.28:0/441824143 learned_addr learned my addr 172.21.15.28:0/441824143 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:04.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.323+0000 7fddf88f5640 1 --2- 172.21.15.28:0/441824143 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddf4100e60 0x7fddf419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.323+0000 7fddf98f7640 1 --2- 172.21.15.28:0/441824143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddf410e0f0 0x7fddf419ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.323+0000 7fddf90f6640 1 -- 172.21.15.28:0/441824143 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddf4100e60 msgr2=0x7fddf419a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.323+0000 7fddf90f6640 1 --2- 172.21.15.28:0/441824143 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddf4100e60 0x7fddf419a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.323+0000 7fddf90f6640 1 -- 172.21.15.28:0/441824143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddf410e0f0 msgr2=0x7fddf419ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.323+0000 7fddf90f6640 1 --2- 172.21.15.28:0/441824143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddf410e0f0 0x7fddf419ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.323+0000 7fddf90f6640 1 -- 172.21.15.28:0/441824143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddf419f150 con 0x7fddf4100540 2024-04-25T01:42:04.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.323+0000 7fddf90f6640 1 --2- 172.21.15.28:0/441824143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddf4100540 0x7fddf419a1b0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fddf002ade0 tx=0x7fddf00319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:04.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.323+0000 7fddea7fc640 1 -- 172.21.15.28:0/441824143 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf002f9e0 con 0x7fddf4100540 2024-04-25T01:42:04.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.323+0000 7fddea7fc640 1 -- 172.21.15.28:0/441824143 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddf00305c0 con 0x7fddf4100540 2024-04-25T01:42:04.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.323+0000 7fddfb35a640 1 -- 172.21.15.28:0/441824143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddf419f3e0 con 0x7fddf4100540 2024-04-25T01:42:04.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.323+0000 7fddea7fc640 1 -- 172.21.15.28:0/441824143 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf003a660 con 0x7fddf4100540 2024-04-25T01:42:04.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.323+0000 7fddfb35a640 1 -- 172.21.15.28:0/441824143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddf41a6cd0 con 0x7fddf4100540 2024-04-25T01:42:04.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.326+0000 7fddea7fc640 1 -- 172.21.15.28:0/441824143 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fddf0030770 con 0x7fddf4100540 2024-04-25T01:42:04.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.326+0000 7fddfe066740 1 Processor -- start 2024-04-25T01:42:04.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.326+0000 7fddfe066740 1 -- start start 2024-04-25T01:42:04.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddfe066740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55862175d580 0x55862175d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddfe066740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55862175dea0 0x558621768030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddea7fc640 1 --2- 172.21.15.28:0/441824143 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdde00a2d00 0x7fdde00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddfe066740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558621768570 0x55862176a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddfe066740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5586216c2670 con 0x55862175dea0 2024-04-25T01:42:04.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddfe066740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55862166a970 con 0x55862175d580 2024-04-25T01:42:04.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddfe066740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5586216be630 con 0x558621768570 2024-04-25T01:42:04.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddf90f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55862175dea0 0x558621768030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddf88f5640 1 --2- 172.21.15.28:0/441824143 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdde00a2d00 0x7fdde00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddf90f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55862175dea0 0x558621768030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45990/0 (socket says 172.21.15.28:45990) 2024-04-25T01:42:04.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddea7fc640 1 -- 172.21.15.28:0/441824143 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fddf00f0d80 con 0x7fddf4100540 2024-04-25T01:42:04.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddf90f6640 1 -- 172.21.15.28:0/2510190591 learned_addr learned my addr 172.21.15.28:0/2510190591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:04.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddf98f7640 1 --2- 172.21.15.28:0/2510190591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55862175d580 0x55862175d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddf98f7640 1 --2- 172.21.15.28:0/2510190591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558621768570 0x55862176a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddf90f6640 1 -- 172.21.15.28:0/2510190591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558621768570 msgr2=0x55862176a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddf90f6640 1 --2- 172.21.15.28:0/2510190591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558621768570 0x55862176a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddf90f6640 1 -- 172.21.15.28:0/2510190591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55862175d580 msgr2=0x55862175d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddf90f6640 1 --2- 172.21.15.28:0/2510190591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55862175d580 0x55862175d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddf90f6640 1 -- 172.21.15.28:0/2510190591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55862168e400 con 0x55862175dea0 2024-04-25T01:42:04.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddf98f7640 1 --2- 172.21.15.28:0/2510190591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55862175d580 0x55862175d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:42:04.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddf90f6640 1 --2- 172.21.15.28:0/2510190591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55862175dea0 0x558621768030 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fddf00bbee0 tx=0x7fddf008ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4211f4e2c981fb57 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:04.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fddf88f5640 1 --2- 172.21.15.28:0/441824143 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdde00a2d00 0x7fdde00a51c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fddf4100000 tx=0x7fdde4005cc0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:04.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.327+0000 7fdddf7fe640 1 -- 172.21.15.28:0/2510190591 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf00880b0 con 0x55862175dea0 2024-04-25T01:42:04.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.328+0000 7fdddf7fe640 1 -- 172.21.15.28:0/2510190591 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddf0036050 con 0x55862175dea0 2024-04-25T01:42:04.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.328+0000 7fdddf7fe640 1 -- 172.21.15.28:0/2510190591 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf00ee3c0 con 0x55862175dea0 2024-04-25T01:42:04.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.328+0000 7fddfe066740 1 -- 172.21.15.28:0/2510190591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55862175dea0 msgr2=0x558621768030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.328+0000 7fddfe066740 1 --2- 172.21.15.28:0/2510190591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55862175dea0 0x558621768030 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fddf00bbee0 tx=0x7fddf008ce80 comp rx=0 tx=0).stop 2024-04-25T01:42:04.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.328+0000 7fddfe066740 1 -- 172.21.15.28:0/2510190591 shutdown_connections 2024-04-25T01:42:04.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.328+0000 7fddfe066740 1 --2- 172.21.15.28:0/2510190591 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558621768570 0x55862176a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.328+0000 7fddfe066740 1 --2- 172.21.15.28:0/2510190591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55862175dea0 0x558621768030 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.328+0000 7fddfe066740 1 --2- 172.21.15.28:0/2510190591 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55862175d580 0x55862175d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.328+0000 7fddfe066740 1 -- 172.21.15.28:0/2510190591 >> 172.21.15.28:0/2510190591 conn(0x5586217598a0 msgr2=0x558621759c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:04.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.328+0000 7fddfe066740 1 -- 172.21.15.28:0/2510190591 shutdown_connections 2024-04-25T01:42:04.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.328+0000 7fddfe066740 1 -- 172.21.15.28:0/2510190591 wait complete. 2024-04-25T01:42:04.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.329+0000 7fddfe066740 1 -- start start 2024-04-25T01:42:04.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.329+0000 7fddfe066740 1 Processor -- start 2024-04-25T01:42:04.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.329+0000 7fddfe066740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55862175d580 0x558621767e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.329+0000 7fddfe066740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55862175dea0 0x558621765160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.329+0000 7fddfe066740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558621768570 0x5586217656a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.329+0000 7fddfe066740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5586216c2670 con 0x558621768570 2024-04-25T01:42:04.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.329+0000 7fddfe066740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55862167e050 con 0x55862175dea0 2024-04-25T01:42:04.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.329+0000 7fddfe066740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55862176b120 con 0x55862175d580 2024-04-25T01:42:04.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.329+0000 7fddfe066740 10 client.0 _ll_register_callbacks cb 0x5586213d5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:04.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.329+0000 7fddf98f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558621768570 0x5586217656a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.329+0000 7fddf98f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558621768570 0x5586217656a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45998/0 (socket says 172.21.15.28:45998) 2024-04-25T01:42:04.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.329+0000 7fddf98f7640 1 -- 172.21.15.28:0/3037129690 learned_addr learned my addr 172.21.15.28:0/3037129690 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:04.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.329+0000 7fddf88f5640 1 --2- 172.21.15.28:0/3037129690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55862175dea0 0x558621765160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.329+0000 7fddf90f6640 1 --2- 172.21.15.28:0/3037129690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55862175d580 0x558621767e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.330+0000 7fddf98f7640 1 -- 172.21.15.28:0/3037129690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55862175d580 msgr2=0x558621767e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.330+0000 7fddf98f7640 1 --2- 172.21.15.28:0/3037129690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55862175d580 0x558621767e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.330+0000 7fddf98f7640 1 -- 172.21.15.28:0/3037129690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55862175dea0 msgr2=0x558621765160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.330+0000 7fddf98f7640 1 --2- 172.21.15.28:0/3037129690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55862175dea0 0x558621765160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.330+0000 7fddf98f7640 1 -- 172.21.15.28:0/3037129690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55862164fdc0 con 0x558621768570 2024-04-25T01:42:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.330+0000 7fddf98f7640 1 --2- 172.21.15.28:0/3037129690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558621768570 0x5586217656a0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fddec012bb0 tx=0x7fddec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b47400c885f199bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.330+0000 7fdddcff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.330+0000 7fdddcff9640 1 -- 172.21.15.28:0/3037129690 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddec004050 con 0x558621768570 2024-04-25T01:42:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.330+0000 7fdddcff9640 1 -- 172.21.15.28:0/3037129690 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fddec0041f0 con 0x558621768570 2024-04-25T01:42:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.330+0000 7fddfe066740 1 -- 172.21.15.28:0/3037129690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddf0036050 con 0x558621768570 2024-04-25T01:42:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.330+0000 7fdddcff9640 1 -- 172.21.15.28:0/3037129690 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fddec026050 con 0x558621768570 2024-04-25T01:42:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.330+0000 7fddfe066740 10 client.18945 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.330+0000 7fddfe066740 1 -- 172.21.15.28:0/3037129690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5586213d63f0 con 0x558621768570 2024-04-25T01:42:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.330+0000 7fdddcff9640 1 -- 172.21.15.28:0/3037129690 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddf0036050 con 0x558621768570 2024-04-25T01:42:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.330+0000 7fdddcff9640 1 -- 172.21.15.28:0/3037129690 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fddec00f420 con 0x558621768570 2024-04-25T01:42:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.330+0000 7fddfe066740 10 client.18945 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.330+0000 7fddfe066740 10 client.18945 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.330+0000 7fddfe066740 20 client.18945 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.331+0000 7fddfe066740 1 --2- 172.21.15.28:0/3037129690 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5586217780c0 0x55862177a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.331+0000 7fddfe066740 4 client.18945 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.331+0000 7fddfe066740 1 -- 172.21.15.28:0/3037129690 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5586216cfbb0 con 0x5586217780c0 2024-04-25T01:42:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.331+0000 7fddf90f6640 1 --2- 172.21.15.28:0/3037129690 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5586217780c0 0x55862177a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.331+0000 7fddf90f6640 1 --2- 172.21.15.28:0/3037129690 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5586217780c0 0x55862177a4e0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=305e240627a6b40 server_cookie=ea697e733e3728d3 in_seq=0 out_seq=0 2024-04-25T01:42:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.331+0000 7fdddcff9640 10 client.18945 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.332+0000 7fdddcff9640 1 -- 172.21.15.28:0/3037129690 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5586216cfbb0 con 0x5586217780c0 2024-04-25T01:42:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.332+0000 7fdddcff9640 10 client.18945 handle_command_reply: tid=0 2024-04-25T01:42:04.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.332+0000 7fddfe066740 1 client.18945 shutdown 2024-04-25T01:42:04.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.332+0000 7fddfe066740 1 -- 172.21.15.28:0/3037129690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558621768570 msgr2=0x5586217656a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.332+0000 7fddfe066740 1 --2- 172.21.15.28:0/3037129690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558621768570 0x5586217656a0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fddec012bb0 tx=0x7fddec009e30 comp rx=0 tx=0).stop 2024-04-25T01:42:04.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.332+0000 7fddfe066740 1 -- 172.21.15.28:0/3037129690 shutdown_connections 2024-04-25T01:42:04.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.332+0000 7fddfe066740 1 --2- 172.21.15.28:0/3037129690 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5586217780c0 0x55862177a4e0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.332+0000 7fddfe066740 1 --2- 172.21.15.28:0/3037129690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558621768570 0x5586217656a0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.332+0000 7fdddcff9640 0 client.18945 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:04.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.332+0000 7fddfe066740 1 --2- 172.21.15.28:0/3037129690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55862175dea0 0x558621765160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.332+0000 7fddfe066740 1 --2- 172.21.15.28:0/3037129690 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55862175d580 0x558621767e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.332+0000 7fddfe066740 1 -- 172.21.15.28:0/3037129690 >> 172.21.15.28:0/3037129690 conn(0x5586217598a0 msgr2=0x558621758b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:04.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.332+0000 7fddfe066740 1 -- 172.21.15.28:0/3037129690 shutdown_connections 2024-04-25T01:42:04.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.333+0000 7fddfe066740 1 -- 172.21.15.28:0/3037129690 wait complete. 2024-04-25T01:42:04.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.333+0000 7fddfe066740 20 client.18945 trim_cache size 0 max 16384 2024-04-25T01:42:04.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.346+0000 7fddfe066740 1 Processor -- start 2024-04-25T01:42:04.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.346+0000 7fddfe066740 1 -- start start 2024-04-25T01:42:04.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.346+0000 7fddfe066740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55862175d580 0x55862175f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.346+0000 7fddfe066740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55862175dea0 0x5586217679a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.346+0000 7fddfe066740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558621768570 0x55862177fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.346+0000 7fddfe066740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fddf41b0cb0 con 0x55862175d580 2024-04-25T01:42:04.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.346+0000 7fddfe066740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fddf41b27f0 con 0x558621768570 2024-04-25T01:42:04.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.346+0000 7fddfe066740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fddf41b1380 con 0x55862175dea0 2024-04-25T01:42:04.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.346+0000 7fddf90f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55862175d580 0x55862175f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.346+0000 7fddf90f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55862175d580 0x55862175f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46000/0 (socket says 172.21.15.28:46000) 2024-04-25T01:42:04.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.346+0000 7fddf90f6640 1 -- 172.21.15.28:0/3540970414 learned_addr learned my addr 172.21.15.28:0/3540970414 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:04.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.346+0000 7fddf90f6640 1 -- 172.21.15.28:0/3540970414 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55862175dea0 msgr2=0x5586217679a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.346+0000 7fddf90f6640 1 --2- 172.21.15.28:0/3540970414 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55862175dea0 0x5586217679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.346+0000 7fddf90f6640 1 -- 172.21.15.28:0/3540970414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558621768570 msgr2=0x55862177fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.346+0000 7fddf90f6640 1 --2- 172.21.15.28:0/3540970414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558621768570 0x55862177fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.346+0000 7fddf90f6640 1 -- 172.21.15.28:0/3540970414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558621760d40 con 0x55862175d580 2024-04-25T01:42:04.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.347+0000 7fddf90f6640 1 --2- 172.21.15.28:0/3540970414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55862175d580 0x55862175f7e0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fddf0004af0 tx=0x7fddf002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb70224c0ffaa204 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:04.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.347+0000 7fdddeffd640 1 -- 172.21.15.28:0/3540970414 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf00c13d0 con 0x55862175d580 2024-04-25T01:42:04.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.347+0000 7fdddeffd640 1 -- 172.21.15.28:0/3540970414 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddf0036040 con 0x55862175d580 2024-04-25T01:42:04.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.347+0000 7fdddeffd640 1 -- 172.21.15.28:0/3540970414 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf00ee3c0 con 0x55862175d580 2024-04-25T01:42:04.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.347+0000 7fddfe066740 1 -- 172.21.15.28:0/3540970414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55862175d580 msgr2=0x55862175f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.347+0000 7fddfe066740 1 --2- 172.21.15.28:0/3540970414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55862175d580 0x55862175f7e0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fddf0004af0 tx=0x7fddf002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.347+0000 7fddfe066740 1 -- 172.21.15.28:0/3540970414 shutdown_connections 2024-04-25T01:42:04.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.347+0000 7fddfe066740 1 --2- 172.21.15.28:0/3540970414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558621768570 0x55862177fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.347+0000 7fddfe066740 1 --2- 172.21.15.28:0/3540970414 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55862175dea0 0x5586217679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.347+0000 7fddfe066740 1 --2- 172.21.15.28:0/3540970414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55862175d580 0x55862175f7e0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.347+0000 7fddfe066740 1 -- 172.21.15.28:0/3540970414 >> 172.21.15.28:0/3540970414 conn(0x5586217598a0 msgr2=0x558621765280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:04.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.347+0000 7fddfe066740 1 -- 172.21.15.28:0/3540970414 shutdown_connections 2024-04-25T01:42:04.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.347+0000 7fddfe066740 1 -- 172.21.15.28:0/3540970414 wait complete. 2024-04-25T01:42:04.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.348+0000 7fddfe066740 1 -- start start 2024-04-25T01:42:04.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.348+0000 7fddfe066740 1 Processor -- start 2024-04-25T01:42:04.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.348+0000 7fddfe066740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55862175d580 0x558621770710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.348+0000 7fddfe066740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55862175dea0 0x55862176bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.348+0000 7fddfe066740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558621768570 0x55862176c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.348+0000 7fddfe066740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fddf41b2120 con 0x558621768570 2024-04-25T01:42:04.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.348+0000 7fddfe066740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fddf41b1a50 con 0x55862175dea0 2024-04-25T01:42:04.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.348+0000 7fddfe066740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fddf41aecb0 con 0x55862175d580 2024-04-25T01:42:04.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.348+0000 7fddfe066740 10 client.0 _ll_register_callbacks cb 0x7fddec012760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:04.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.348+0000 7fddf98f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558621768570 0x55862176c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.348+0000 7fddf98f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558621768570 0x55862176c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46014/0 (socket says 172.21.15.28:46014) 2024-04-25T01:42:04.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.348+0000 7fddf98f7640 1 -- 172.21.15.28:0/1345835358 learned_addr learned my addr 172.21.15.28:0/1345835358 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:04.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.348+0000 7fddf88f5640 1 --2- 172.21.15.28:0/1345835358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55862175dea0 0x55862176bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.349+0000 7fddf90f6640 1 --2- 172.21.15.28:0/1345835358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55862175d580 0x558621770710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.349+0000 7fddf98f7640 1 -- 172.21.15.28:0/1345835358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55862175d580 msgr2=0x558621770710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.349+0000 7fddf98f7640 1 --2- 172.21.15.28:0/1345835358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55862175d580 0x558621770710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.349+0000 7fddf98f7640 1 -- 172.21.15.28:0/1345835358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55862175dea0 msgr2=0x55862176bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:04.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.349+0000 7fddf98f7640 1 --2- 172.21.15.28:0/1345835358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55862175dea0 0x55862176bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.349+0000 7fddf98f7640 1 -- 172.21.15.28:0/1345835358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586214a07d0 con 0x558621768570 2024-04-25T01:42:04.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.349+0000 7fddf98f7640 1 --2- 172.21.15.28:0/1345835358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558621768570 0x55862176c070 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fddec012b80 tx=0x7fddec009b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c38f518ade2aca9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:04.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.349+0000 7fdddcff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:04.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.349+0000 7fdddcff9640 1 -- 172.21.15.28:0/1345835358 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddec030060 con 0x558621768570 2024-04-25T01:42:04.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.349+0000 7fdddcff9640 1 -- 172.21.15.28:0/1345835358 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fddec00fa10 con 0x558621768570 2024-04-25T01:42:04.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.349+0000 7fddfe066740 1 -- 172.21.15.28:0/1345835358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddf0036040 con 0x558621768570 2024-04-25T01:42:04.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.349+0000 7fdddcff9640 1 -- 172.21.15.28:0/1345835358 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fddec037050 con 0x558621768570 2024-04-25T01:42:04.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.349+0000 7fddfe066740 10 client.18951 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:04.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.349+0000 7fddfe066740 1 -- 172.21.15.28:0/1345835358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5586217de4f0 con 0x558621768570 2024-04-25T01:42:04.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.349+0000 7fdddcff9640 1 -- 172.21.15.28:0/1345835358 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddf0036040 con 0x558621768570 2024-04-25T01:42:04.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.349+0000 7fdddcff9640 1 -- 172.21.15.28:0/1345835358 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fddec0035f0 con 0x558621768570 2024-04-25T01:42:04.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.349+0000 7fddfe066740 10 client.18951 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:04.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.349+0000 7fddfe066740 10 client.18951 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:04.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.349+0000 7fddfe066740 20 client.18951 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:04.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.350+0000 7fddfe066740 1 --2- 172.21.15.28:0/1345835358 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5586217dec40 0x5586217e1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:04.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.350+0000 7fddfe066740 4 client.18951 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:42:04.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.350+0000 7fddfe066740 1 -- 172.21.15.28:0/1345835358 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5586216c9f00 con 0x5586217dec40 2024-04-25T01:42:04.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.350+0000 7fddf90f6640 1 --2- 172.21.15.28:0/1345835358 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5586217dec40 0x5586217e1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:04.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.350+0000 7fddf90f6640 1 --2- 172.21.15.28:0/1345835358 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5586217dec40 0x5586217e1060 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5b40ada279cc0e server_cookie=26edf45404d04783 in_seq=0 out_seq=0 2024-04-25T01:42:04.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.350+0000 7fdddcff9640 10 client.18951 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:04.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.350+0000 7fdddcff9640 1 -- 172.21.15.28:0/1345835358 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5586216c9f00 con 0x5586217dec40 2024-04-25T01:42:04.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.350+0000 7fdddcff9640 10 client.18951 handle_command_reply: tid=0 2024-04-25T01:42:04.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.350+0000 7fddfe066740 1 client.18951 shutdown 2024-04-25T01:42:04.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfe066740 1 -- 172.21.15.28:0/1345835358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558621768570 msgr2=0x55862176c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfe066740 1 --2- 172.21.15.28:0/1345835358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558621768570 0x55862176c070 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fddec012b80 tx=0x7fddec009b70 comp rx=0 tx=0).stop 2024-04-25T01:42:04.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfe066740 1 -- 172.21.15.28:0/1345835358 shutdown_connections 2024-04-25T01:42:04.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfe066740 1 --2- 172.21.15.28:0/1345835358 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5586217dec40 0x5586217e1060 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfe066740 1 --2- 172.21.15.28:0/1345835358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558621768570 0x55862176c070 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfe066740 1 --2- 172.21.15.28:0/1345835358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55862175dea0 0x55862176bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfe066740 1 --2- 172.21.15.28:0/1345835358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55862175d580 0x558621770710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fdddcff9640 0 client.18951 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:04.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfe066740 1 -- 172.21.15.28:0/1345835358 >> 172.21.15.28:0/1345835358 conn(0x5586217598a0 msgr2=0x5586217582b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:04.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfe066740 1 -- 172.21.15.28:0/1345835358 shutdown_connections 2024-04-25T01:42:04.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfe066740 1 -- 172.21.15.28:0/1345835358 wait complete. 2024-04-25T01:42:04.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfe066740 20 client.18951 trim_cache size 0 max 16384 2024-04-25T01:42:04.345 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:42:04.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfb35a640 1 -- 172.21.15.28:0/441824143 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdde00a2d00 msgr2=0x7fdde00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfb35a640 1 --2- 172.21.15.28:0/441824143 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdde00a2d00 0x7fdde00a51c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fddf4100000 tx=0x7fdde4005cc0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfb35a640 1 -- 172.21.15.28:0/441824143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddf4100540 msgr2=0x7fddf419a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:04.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfb35a640 1 --2- 172.21.15.28:0/441824143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddf4100540 0x7fddf419a1b0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fddf002ade0 tx=0x7fddf00319f0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfb35a640 1 -- 172.21.15.28:0/441824143 shutdown_connections 2024-04-25T01:42:04.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfb35a640 1 --2- 172.21.15.28:0/441824143 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdde00a2d00 0x7fdde00a51c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfb35a640 1 --2- 172.21.15.28:0/441824143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fddf410e0f0 0x7fddf419ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfb35a640 1 --2- 172.21.15.28:0/441824143 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fddf4100e60 0x7fddf419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfb35a640 1 --2- 172.21.15.28:0/441824143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddf4100540 0x7fddf419a1b0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:04.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.351+0000 7fddfb35a640 1 -- 172.21.15.28:0/441824143 >> 172.21.15.28:0/441824143 conn(0x7fddf4077740 msgr2=0x7fddf410f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:04.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.352+0000 7fddfb35a640 1 -- 172.21.15.28:0/441824143 shutdown_connections 2024-04-25T01:42:04.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:04.352+0000 7fddfb35a640 1 -- 172.21.15.28:0/441824143 wait complete. 2024-04-25T01:42:04.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:04 smithi130 ceph-mon[32702]: pgmap v278: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-25T01:42:04.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:04 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3772072508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:42:05.350 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:42:05.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:42:05.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:42:05.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.482+0000 7f60f9996640 1 -- 172.21.15.28:0/441594346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60f4100ec0 msgr2=0x7f60f410b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:05.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.482+0000 7f60f9996640 1 --2- 172.21.15.28:0/441594346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60f4100ec0 0x7f60f410b620 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f60e8009900 tx=0x7f60e802a410 comp rx=0 tx=0).stop 2024-04-25T01:42:05.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.482+0000 7f60f9996640 1 -- 172.21.15.28:0/441594346 shutdown_connections 2024-04-25T01:42:05.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.482+0000 7f60f9996640 1 --2- 172.21.15.28:0/441594346 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60f410bce0 0x7f60f410e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.482+0000 7f60f9996640 1 --2- 172.21.15.28:0/441594346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60f4100ec0 0x7f60f410b620 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.482+0000 7f60f9996640 1 --2- 172.21.15.28:0/441594346 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60f4100510 0x7f60f41008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.482+0000 7f60f9996640 1 -- 172.21.15.28:0/441594346 >> 172.21.15.28:0/441594346 conn(0x7f60f4075940 msgr2=0x7f60f4075d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:05.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.482+0000 7f60f9996640 1 -- 172.21.15.28:0/441594346 shutdown_connections 2024-04-25T01:42:05.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.482+0000 7f60f9996640 1 -- 172.21.15.28:0/441594346 wait complete. 2024-04-25T01:42:05.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.483+0000 7f60f9996640 1 Processor -- start 2024-04-25T01:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.483+0000 7f60f9996640 1 -- start start 2024-04-25T01:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.483+0000 7f60f9996640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60f4100510 0x7f60f4197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.483+0000 7f60f9996640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60f410bce0 0x7f60f41984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.483+0000 7f60f9996640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60f419c820 0x7f60f419ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.483+0000 7f60f9996640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f60f4111ac0 con 0x7f60f419c820 2024-04-25T01:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.483+0000 7f60f9996640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f60f4111940 con 0x7f60f410bce0 2024-04-25T01:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.483+0000 7f60f9996640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f60f4111c40 con 0x7f60f4100510 2024-04-25T01:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.484+0000 7f60f37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60f419c820 0x7f60f419ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.484+0000 7f60f37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60f419c820 0x7f60f419ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46032/0 (socket says 172.21.15.28:46032) 2024-04-25T01:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.484+0000 7f60f37fe640 1 -- 172.21.15.28:0/1144140133 learned_addr learned my addr 172.21.15.28:0/1144140133 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.484+0000 7f60f2ffd640 1 --2- 172.21.15.28:0/1144140133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60f4100510 0x7f60f4197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.484+0000 7f60f27fc640 1 --2- 172.21.15.28:0/1144140133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60f410bce0 0x7f60f41984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.484+0000 7f60f37fe640 1 -- 172.21.15.28:0/1144140133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60f4100510 msgr2=0x7f60f4197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.484+0000 7f60f37fe640 1 --2- 172.21.15.28:0/1144140133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60f4100510 0x7f60f4197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.484+0000 7f60f37fe640 1 -- 172.21.15.28:0/1144140133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60f410bce0 msgr2=0x7f60f41984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.484+0000 7f60f37fe640 1 --2- 172.21.15.28:0/1144140133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60f410bce0 0x7f60f41984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.484+0000 7f60f37fe640 1 -- 172.21.15.28:0/1144140133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f419d310 con 0x7f60f419c820 2024-04-25T01:42:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.484+0000 7f60f27fc640 1 --2- 172.21.15.28:0/1144140133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60f410bce0 0x7f60f41984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.484+0000 7f60f37fe640 1 --2- 172.21.15.28:0/1144140133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60f419c820 0x7f60f419ccd0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f60ec00d750 tx=0x7f60ec00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.484+0000 7f60f8994640 1 -- 172.21.15.28:0/1144140133 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ec017070 con 0x7f60f419c820 2024-04-25T01:42:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.484+0000 7f60f9996640 1 -- 172.21.15.28:0/1144140133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60f419d600 con 0x7f60f419c820 2024-04-25T01:42:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.484+0000 7f60f8994640 1 -- 172.21.15.28:0/1144140133 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60ec005500 con 0x7f60f419c820 2024-04-25T01:42:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.484+0000 7f60f8994640 1 -- 172.21.15.28:0/1144140133 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ec004d90 con 0x7f60f419c820 2024-04-25T01:42:05.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.484+0000 7f60f9996640 1 -- 172.21.15.28:0/1144140133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60f41a4d70 con 0x7f60f419c820 2024-04-25T01:42:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.487+0000 7f60f8994640 1 -- 172.21.15.28:0/1144140133 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f60ec004f30 con 0x7f60f419c820 2024-04-25T01:42:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.487+0000 7f60fc777740 1 Processor -- start 2024-04-25T01:42:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.487+0000 7f60fc777740 1 -- start start 2024-04-25T01:42:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.487+0000 7f60fc777740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf87bf2980 0x55bf87bf0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.487+0000 7f60fc777740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf1290 0x55bf87bf1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.487+0000 7f60fc777740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 0x55bf87c02810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.487+0000 7f60fc777740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bf87b552c0 con 0x55bf87c00420 2024-04-25T01:42:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.487+0000 7f60fc777740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55bf87afd910 con 0x55bf87bf2980 2024-04-25T01:42:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.487+0000 7f60fc777740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55bf87b51280 con 0x55bf87bf1290 2024-04-25T01:42:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.487+0000 7f60f8994640 1 --2- 172.21.15.28:0/1144140133 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f60dc0a2d00 0x7f60dc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.487+0000 7f60f27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 0x55bf87c02810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.487+0000 7f60f27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 0x55bf87c02810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46044/0 (socket says 172.21.15.28:46044) 2024-04-25T01:42:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.487+0000 7f60f27fc640 1 -- 172.21.15.28:0/2354952586 learned_addr learned my addr 172.21.15.28:0/2354952586 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.487+0000 7f60f27fc640 1 --2- 172.21.15.28:0/2354952586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf87bf2980 0x55bf87bf0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.487+0000 7f60f2ffd640 1 --2- 172.21.15.28:0/2354952586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf1290 0x55bf87bf1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.487+0000 7f60f37fe640 1 --2- 172.21.15.28:0/1144140133 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f60dc0a2d00 0x7f60dc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.488+0000 7f60f8994640 1 -- 172.21.15.28:0/1144140133 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f60ec0cf220 con 0x7f60f419c820 2024-04-25T01:42:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.488+0000 7f60f27fc640 1 -- 172.21.15.28:0/2354952586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf1290 msgr2=0x55bf87bf1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.488+0000 7f60f27fc640 1 --2- 172.21.15.28:0/2354952586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf1290 0x55bf87bf1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.488+0000 7f60f27fc640 1 -- 172.21.15.28:0/2354952586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf87bf2980 msgr2=0x55bf87bf0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.488+0000 7f60f27fc640 1 --2- 172.21.15.28:0/2354952586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf87bf2980 0x55bf87bf0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.488+0000 7f60f27fc640 1 -- 172.21.15.28:0/2354952586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf87b0efb0 con 0x55bf87c00420 2024-04-25T01:42:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.488+0000 7f60f27fc640 1 --2- 172.21.15.28:0/2354952586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 0x55bf87c02810 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f60e80096c0 tx=0x7f60e80056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=959af52c0a42c7b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.488+0000 7f60f37fe640 1 --2- 172.21.15.28:0/1144140133 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f60dc0a2d00 0x7f60dc0a51c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f60ec09b340 tx=0x7f60ec06d370 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.488+0000 7f60d9ffb640 1 -- 172.21.15.28:0/2354952586 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e8035040 con 0x55bf87c00420 2024-04-25T01:42:05.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.488+0000 7f60d9ffb640 1 -- 172.21.15.28:0/2354952586 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60e80319e0 con 0x55bf87c00420 2024-04-25T01:42:05.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.488+0000 7f60d9ffb640 1 -- 172.21.15.28:0/2354952586 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e803d480 con 0x55bf87c00420 2024-04-25T01:42:05.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.489+0000 7f60fc777740 1 -- 172.21.15.28:0/2354952586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 msgr2=0x55bf87c02810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:05.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.489+0000 7f60fc777740 1 --2- 172.21.15.28:0/2354952586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 0x55bf87c02810 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f60e80096c0 tx=0x7f60e80056d0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.489+0000 7f60fc777740 1 -- 172.21.15.28:0/2354952586 shutdown_connections 2024-04-25T01:42:05.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.489+0000 7f60fc777740 1 --2- 172.21.15.28:0/2354952586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 0x55bf87c02810 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.489+0000 7f60fc777740 1 --2- 172.21.15.28:0/2354952586 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf1290 0x55bf87bf1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.489+0000 7f60fc777740 1 --2- 172.21.15.28:0/2354952586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf87bf2980 0x55bf87bf0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.489+0000 7f60fc777740 1 -- 172.21.15.28:0/2354952586 >> 172.21.15.28:0/2354952586 conn(0x55bf87bb9950 msgr2=0x55bf87bed7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:05.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.489+0000 7f60fc777740 1 -- 172.21.15.28:0/2354952586 shutdown_connections 2024-04-25T01:42:05.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.489+0000 7f60fc777740 1 -- 172.21.15.28:0/2354952586 wait complete. 2024-04-25T01:42:05.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.489+0000 7f60fc777740 1 -- start start 2024-04-25T01:42:05.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.490+0000 7f60fc777740 1 Processor -- start 2024-04-25T01:42:05.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.490+0000 7f60fc777740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87bf1290 0x55bf87bfd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.490+0000 7f60fc777740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf2980 0x55bf87bfd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.490+0000 7f60fc777740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf87c00420 0x55bf87bfde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.490+0000 7f60fc777740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bf87c03250 con 0x55bf87bf1290 2024-04-25T01:42:05.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.490+0000 7f60fc777740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55bf87b10e90 con 0x55bf87c00420 2024-04-25T01:42:05.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.490+0000 7f60fc777740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55bf87c033d0 con 0x55bf87bf2980 2024-04-25T01:42:05.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.490+0000 7f60fc777740 10 client.0 _ll_register_callbacks cb 0x55bf87b4c970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:05.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.490+0000 7f60f27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87bf1290 0x55bf87bfd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.490+0000 7f60f27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87bf1290 0x55bf87bfd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46052/0 (socket says 172.21.15.28:46052) 2024-04-25T01:42:05.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.490+0000 7f60f27fc640 1 -- 172.21.15.28:0/2072160149 learned_addr learned my addr 172.21.15.28:0/2072160149 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:05.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.490+0000 7f60f2ffd640 1 --2- 172.21.15.28:0/2072160149 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf2980 0x55bf87bfd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.490+0000 7f60f27fc640 1 -- 172.21.15.28:0/2072160149 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf2980 msgr2=0x55bf87bfd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:05.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.490+0000 7f60f27fc640 1 --2- 172.21.15.28:0/2072160149 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf2980 0x55bf87bfd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.490+0000 7f60f27fc640 1 -- 172.21.15.28:0/2072160149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf87c00420 msgr2=0x55bf87bfde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:05.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.490+0000 7f60f27fc640 1 --2- 172.21.15.28:0/2072160149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf87c00420 0x55bf87bfde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.490+0000 7f60f27fc640 1 -- 172.21.15.28:0/2072160149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf879317e0 con 0x55bf87bf1290 2024-04-25T01:42:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.490+0000 7f60f27fc640 1 --2- 172.21.15.28:0/2072160149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87bf1290 0x55bf87bfd380 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f60e8005ec0 tx=0x7f60e8005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906b4041a0e4771d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.490+0000 7f60d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.491+0000 7f60d2ffd640 1 -- 172.21.15.28:0/2072160149 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e804d020 con 0x55bf87bf1290 2024-04-25T01:42:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.491+0000 7f60d2ffd640 1 -- 172.21.15.28:0/2072160149 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f60e80042b0 con 0x55bf87bf1290 2024-04-25T01:42:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.491+0000 7f60fc777740 1 -- 172.21.15.28:0/2072160149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60e80319e0 con 0x55bf87bf1290 2024-04-25T01:42:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.491+0000 7f60d2ffd640 1 -- 172.21.15.28:0/2072160149 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f60e8058050 con 0x55bf87bf1290 2024-04-25T01:42:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.491+0000 7f60fc777740 10 client.18963 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.491+0000 7f60fc777740 1 -- 172.21.15.28:0/2072160149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bf87991f70 con 0x55bf87bf1290 2024-04-25T01:42:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.491+0000 7f60d2ffd640 1 -- 172.21.15.28:0/2072160149 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60e80319e0 con 0x55bf87bf1290 2024-04-25T01:42:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.491+0000 7f60d2ffd640 1 -- 172.21.15.28:0/2072160149 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f60e803a040 con 0x55bf87bf1290 2024-04-25T01:42:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.491+0000 7f60fc777740 10 client.18963 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.491+0000 7f60fc777740 10 client.18963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.491+0000 7f60fc777740 20 client.18963 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:05.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.491+0000 7f60fc777740 1 --2- 172.21.15.28:0/2072160149 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bf87c10380 0x55bf87c12770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.491+0000 7f60fc777740 4 client.18963 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.491+0000 7f60fc777740 1 -- 172.21.15.28:0/2072160149 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf87b295a0 con 0x55bf87c10380 2024-04-25T01:42:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.492+0000 7f60f2ffd640 1 --2- 172.21.15.28:0/2072160149 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bf87c10380 0x55bf87c12770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.492+0000 7f60f2ffd640 1 --2- 172.21.15.28:0/2072160149 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bf87c10380 0x55bf87c12770 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2c8fa5466340c72 server_cookie=860a8c0da4b9b970 in_seq=0 out_seq=0 2024-04-25T01:42:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.492+0000 7f60d2ffd640 10 client.18963 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.493+0000 7f60d2ffd640 1 -- 172.21.15.28:0/2072160149 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bf87b295a0 con 0x55bf87c10380 2024-04-25T01:42:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.493+0000 7f60d2ffd640 10 client.18963 handle_command_reply: tid=0 2024-04-25T01:42:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.493+0000 7f60fc777740 1 client.18963 shutdown 2024-04-25T01:42:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.493+0000 7f60fc777740 1 -- 172.21.15.28:0/2072160149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87bf1290 msgr2=0x55bf87bfd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.493+0000 7f60fc777740 1 --2- 172.21.15.28:0/2072160149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87bf1290 0x55bf87bfd380 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f60e8005ec0 tx=0x7f60e8005a20 comp rx=0 tx=0).stop 2024-04-25T01:42:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.493+0000 7f60fc777740 1 -- 172.21.15.28:0/2072160149 shutdown_connections 2024-04-25T01:42:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.493+0000 7f60fc777740 1 --2- 172.21.15.28:0/2072160149 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bf87c10380 0x55bf87c12770 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.493+0000 7f60fc777740 1 --2- 172.21.15.28:0/2072160149 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf87c00420 0x55bf87bfde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.493+0000 7f60fc777740 1 --2- 172.21.15.28:0/2072160149 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf2980 0x55bf87bfd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.493+0000 7f60fc777740 1 --2- 172.21.15.28:0/2072160149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87bf1290 0x55bf87bfd380 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.493+0000 7f60d2ffd640 0 client.18963 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.493+0000 7f60fc777740 1 -- 172.21.15.28:0/2072160149 >> 172.21.15.28:0/2072160149 conn(0x55bf87bb9950 msgr2=0x55bf87bf3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.493+0000 7f60fc777740 1 -- 172.21.15.28:0/2072160149 shutdown_connections 2024-04-25T01:42:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.493+0000 7f60fc777740 1 -- 172.21.15.28:0/2072160149 wait complete. 2024-04-25T01:42:05.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.493+0000 7f60fc777740 20 client.18963 trim_cache size 0 max 16384 2024-04-25T01:42:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.511+0000 7f60fc777740 1 Processor -- start 2024-04-25T01:42:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.511+0000 7f60fc777740 1 -- start start 2024-04-25T01:42:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.511+0000 7f60fc777740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf1290 0x55bf87c0c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.511+0000 7f60fc777740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf87bf2980 0x55bf87bfc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.511+0000 7f60fc777740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 0x55bf87bf34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.511+0000 7f60fc777740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f60f41ac7f0 con 0x55bf87c00420 2024-04-25T01:42:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.511+0000 7f60fc777740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f60f41ae330 con 0x55bf87bf2980 2024-04-25T01:42:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.511+0000 7f60fc777740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f60f41acec0 con 0x55bf87bf1290 2024-04-25T01:42:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.511+0000 7f60f27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 0x55bf87bf34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.511+0000 7f60f27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 0x55bf87bf34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46068/0 (socket says 172.21.15.28:46068) 2024-04-25T01:42:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.511+0000 7f60f27fc640 1 -- 172.21.15.28:0/1627872262 learned_addr learned my addr 172.21.15.28:0/1627872262 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.512+0000 7f60f2ffd640 1 --2- 172.21.15.28:0/1627872262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf87bf2980 0x55bf87bfc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.512+0000 7f60f27fc640 1 --2- 172.21.15.28:0/1627872262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf1290 0x55bf87c0c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.512+0000 7f60f27fc640 1 -- 172.21.15.28:0/1627872262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf1290 msgr2=0x55bf87c0c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.512+0000 7f60f27fc640 1 --2- 172.21.15.28:0/1627872262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf1290 0x55bf87c0c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.512+0000 7f60f27fc640 1 -- 172.21.15.28:0/1627872262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf87bf2980 msgr2=0x55bf87bfc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.512+0000 7f60f27fc640 1 --2- 172.21.15.28:0/1627872262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf87bf2980 0x55bf87bfc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.512+0000 7f60f27fc640 1 -- 172.21.15.28:0/1627872262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf87829170 con 0x55bf87c00420 2024-04-25T01:42:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.512+0000 7f60f27fc640 1 --2- 172.21.15.28:0/1627872262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 0x55bf87bf34e0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f60e8031550 tx=0x7f60e80056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c229d02588f1e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.512+0000 7f60d97fa640 1 -- 172.21.15.28:0/1627872262 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e804d040 con 0x55bf87c00420 2024-04-25T01:42:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.512+0000 7f60d97fa640 1 -- 172.21.15.28:0/1627872262 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60e8009780 con 0x55bf87c00420 2024-04-25T01:42:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.512+0000 7f60d97fa640 1 -- 172.21.15.28:0/1627872262 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e8035050 con 0x55bf87c00420 2024-04-25T01:42:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.513+0000 7f60fc777740 1 -- 172.21.15.28:0/1627872262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 msgr2=0x55bf87bf34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.513+0000 7f60fc777740 1 --2- 172.21.15.28:0/1627872262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 0x55bf87bf34e0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f60e8031550 tx=0x7f60e80056d0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.513+0000 7f60fc777740 1 -- 172.21.15.28:0/1627872262 shutdown_connections 2024-04-25T01:42:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.513+0000 7f60fc777740 1 --2- 172.21.15.28:0/1627872262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 0x55bf87bf34e0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.513+0000 7f60fc777740 1 --2- 172.21.15.28:0/1627872262 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf87bf2980 0x55bf87bfc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.513+0000 7f60fc777740 1 --2- 172.21.15.28:0/1627872262 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf1290 0x55bf87c0c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.513+0000 7f60fc777740 1 -- 172.21.15.28:0/1627872262 >> 172.21.15.28:0/1627872262 conn(0x55bf87bb9950 msgr2=0x55bf87bf2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.513+0000 7f60fc777740 1 -- 172.21.15.28:0/1627872262 shutdown_connections 2024-04-25T01:42:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.513+0000 7f60fc777740 1 -- 172.21.15.28:0/1627872262 wait complete. 2024-04-25T01:42:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.513+0000 7f60fc777740 1 -- start start 2024-04-25T01:42:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.513+0000 7f60fc777740 1 Processor -- start 2024-04-25T01:42:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.513+0000 7f60fc777740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf1290 0x55bf87bfa240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.513+0000 7f60fc777740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf87bf2980 0x55bf87bfa780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.513+0000 7f60fc777740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 0x55bf87bfacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.513+0000 7f60fc777740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f60f41adc60 con 0x55bf87c00420 2024-04-25T01:42:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.513+0000 7f60fc777740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f60f41ad590 con 0x55bf87bf2980 2024-04-25T01:42:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.513+0000 7f60fc777740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f60f41aa7f0 con 0x55bf87bf1290 2024-04-25T01:42:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.514+0000 7f60fc777740 10 client.0 _ll_register_callbacks cb 0x7f60e8032de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.514+0000 7f60f27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 0x55bf87bfacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.514+0000 7f60f27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 0x55bf87bfacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46084/0 (socket says 172.21.15.28:46084) 2024-04-25T01:42:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.514+0000 7f60f27fc640 1 -- 172.21.15.28:0/3908644568 learned_addr learned my addr 172.21.15.28:0/3908644568 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.514+0000 7f60f2ffd640 1 --2- 172.21.15.28:0/3908644568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf87bf2980 0x55bf87bfa780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.514+0000 7f60f27fc640 1 --2- 172.21.15.28:0/3908644568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf1290 0x55bf87bfa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.514+0000 7f60f27fc640 1 -- 172.21.15.28:0/3908644568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf1290 msgr2=0x55bf87bfa240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.514+0000 7f60f27fc640 1 --2- 172.21.15.28:0/3908644568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf1290 0x55bf87bfa240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.514+0000 7f60f27fc640 1 -- 172.21.15.28:0/3908644568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf87bf2980 msgr2=0x55bf87bfa780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.514+0000 7f60f27fc640 1 --2- 172.21.15.28:0/3908644568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf87bf2980 0x55bf87bfa780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.514+0000 7f60f27fc640 1 -- 172.21.15.28:0/3908644568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf87c757e0 con 0x55bf87c00420 2024-04-25T01:42:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.514+0000 7f60f27fc640 1 --2- 172.21.15.28:0/3908644568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 0x55bf87bfacc0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f60e802a930 tx=0x7f60e8049ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d1bfc011c0db793 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.514+0000 7f60d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.514+0000 7f60d2ffd640 1 -- 172.21.15.28:0/3908644568 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e804d070 con 0x55bf87c00420 2024-04-25T01:42:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.514+0000 7f60d2ffd640 1 -- 172.21.15.28:0/3908644568 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7f60e8049910 con 0x55bf87c00420 2024-04-25T01:42:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.514+0000 7f60fc777740 1 -- 172.21.15.28:0/3908644568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60e8009780 con 0x55bf87c00420 2024-04-25T01:42:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.515+0000 7f60d2ffd640 1 -- 172.21.15.28:0/3908644568 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f60e8035040 con 0x55bf87c00420 2024-04-25T01:42:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.515+0000 7f60fc777740 10 client.18969 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.515+0000 7f60fc777740 1 -- 172.21.15.28:0/3908644568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bf87c757e0 con 0x55bf87c00420 2024-04-25T01:42:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.515+0000 7f60d2ffd640 1 -- 172.21.15.28:0/3908644568 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60e8009780 con 0x55bf87c00420 2024-04-25T01:42:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.515+0000 7f60d2ffd640 1 -- 172.21.15.28:0/3908644568 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f60e803abe0 con 0x55bf87c00420 2024-04-25T01:42:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.515+0000 7f60fc777740 10 client.18969 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.515+0000 7f60fc777740 10 client.18969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.515+0000 7f60fc777740 20 client.18969 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.515+0000 7f60fc777740 1 --2- 172.21.15.28:0/3908644568 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bf87c76e00 0x55bf87c791f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.515+0000 7f60fc777740 4 client.18969 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:42:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.515+0000 7f60fc777740 1 -- 172.21.15.28:0/3908644568 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55bf87b62800 con 0x55bf87c76e00 2024-04-25T01:42:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.515+0000 7f60f2ffd640 1 --2- 172.21.15.28:0/3908644568 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bf87c76e00 0x55bf87c791f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.516+0000 7f60d2ffd640 10 client.18969 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:05.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.516+0000 7f60f2ffd640 1 --2- 172.21.15.28:0/3908644568 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bf87c76e00 0x55bf87c791f0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=436fcbc6b398c6a6 server_cookie=1a7b678b4da8ada4 in_seq=0 out_seq=0 2024-04-25T01:42:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.517+0000 7f60d2ffd640 1 -- 172.21.15.28:0/3908644568 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55bf87b62800 con 0x55bf87c76e00 2024-04-25T01:42:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.517+0000 7f60d2ffd640 10 client.18969 handle_command_reply: tid=0 2024-04-25T01:42:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.517+0000 7f60fc777740 1 client.18969 shutdown 2024-04-25T01:42:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.517+0000 7f60fc777740 1 -- 172.21.15.28:0/3908644568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 msgr2=0x55bf87bfacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.517+0000 7f60fc777740 1 --2- 172.21.15.28:0/3908644568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 0x55bf87bfacc0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f60e802a930 tx=0x7f60e8049ba0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.517+0000 7f60fc777740 1 -- 172.21.15.28:0/3908644568 shutdown_connections 2024-04-25T01:42:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.517+0000 7f60fc777740 1 --2- 172.21.15.28:0/3908644568 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55bf87c76e00 0x55bf87c791f0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.517+0000 7f60fc777740 1 --2- 172.21.15.28:0/3908644568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf87c00420 0x55bf87bfacc0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.517+0000 7f60fc777740 1 --2- 172.21.15.28:0/3908644568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf87bf2980 0x55bf87bfa780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.517+0000 7f60d2ffd640 0 client.18969 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.517+0000 7f60fc777740 1 --2- 172.21.15.28:0/3908644568 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf87bf1290 0x55bf87bfa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.517+0000 7f60fc777740 1 -- 172.21.15.28:0/3908644568 >> 172.21.15.28:0/3908644568 conn(0x55bf87bb9950 msgr2=0x55bf87bec950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.517+0000 7f60fc777740 1 -- 172.21.15.28:0/3908644568 shutdown_connections 2024-04-25T01:42:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.517+0000 7f60fc777740 1 -- 172.21.15.28:0/3908644568 wait complete. 2024-04-25T01:42:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.517+0000 7f60fc777740 20 client.18969 trim_cache size 0 max 16384 2024-04-25T01:42:05.510 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:42:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:42:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "e108bd65-7cb5-45cd-b13a-cdc2088664e9", 2024-04-25T01:42:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:42:05.510 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:42:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.518+0000 7f60f9996640 1 -- 172.21.15.28:0/1144140133 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f60dc0a2d00 msgr2=0x7f60dc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.518+0000 7f60f9996640 1 --2- 172.21.15.28:0/1144140133 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f60dc0a2d00 0x7f60dc0a51c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f60ec09b340 tx=0x7f60ec06d370 comp rx=0 tx=0).stop 2024-04-25T01:42:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.518+0000 7f60f9996640 1 -- 172.21.15.28:0/1144140133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60f419c820 msgr2=0x7f60f419ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.518+0000 7f60f9996640 1 --2- 172.21.15.28:0/1144140133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60f419c820 0x7f60f419ccd0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f60ec00d750 tx=0x7f60ec00dc10 comp rx=0 tx=0).stop 2024-04-25T01:42:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.518+0000 7f60f9996640 1 -- 172.21.15.28:0/1144140133 shutdown_connections 2024-04-25T01:42:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.518+0000 7f60f9996640 1 --2- 172.21.15.28:0/1144140133 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f60dc0a2d00 0x7f60dc0a51c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.518+0000 7f60f9996640 1 --2- 172.21.15.28:0/1144140133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60f419c820 0x7f60f419ccd0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.518+0000 7f60f9996640 1 --2- 172.21.15.28:0/1144140133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60f410bce0 0x7f60f41984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.518+0000 7f60f9996640 1 --2- 172.21.15.28:0/1144140133 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60f4100510 0x7f60f4197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.518+0000 7f60f9996640 1 -- 172.21.15.28:0/1144140133 >> 172.21.15.28:0/1144140133 conn(0x7f60f4075940 msgr2=0x7f60f40fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.518+0000 7f60f9996640 1 -- 172.21.15.28:0/1144140133 shutdown_connections 2024-04-25T01:42:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.518+0000 7f60f9996640 1 -- 172.21.15.28:0/1144140133 wait complete. 2024-04-25T01:42:05.516 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:42:05.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:42:05.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.633+0000 7fd94a95a640 1 -- 172.21.15.28:0/2186490282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd944069dc0 msgr2=0x7fd94406a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:05.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.633+0000 7fd94a95a640 1 --2- 172.21.15.28:0/2186490282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd944069dc0 0x7fd94406a240 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fd934009930 tx=0x7fd93402a420 comp rx=0 tx=0).stop 2024-04-25T01:42:05.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.633+0000 7fd94a95a640 1 -- 172.21.15.28:0/2186490282 shutdown_connections 2024-04-25T01:42:05.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.633+0000 7fd94a95a640 1 --2- 172.21.15.28:0/2186490282 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd94406a780 0x7fd94410e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.633+0000 7fd94a95a640 1 --2- 172.21.15.28:0/2186490282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd944069dc0 0x7fd94406a240 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.633+0000 7fd94a95a640 1 --2- 172.21.15.28:0/2186490282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd944107730 0x7fd944107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.633+0000 7fd94a95a640 1 -- 172.21.15.28:0/2186490282 >> 172.21.15.28:0/2186490282 conn(0x7fd9440776a0 msgr2=0x7fd944077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:05.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.633+0000 7fd94a95a640 1 -- 172.21.15.28:0/2186490282 shutdown_connections 2024-04-25T01:42:05.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.633+0000 7fd94a95a640 1 -- 172.21.15.28:0/2186490282 wait complete. 2024-04-25T01:42:05.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.634+0000 7fd94a95a640 1 Processor -- start 2024-04-25T01:42:05.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.634+0000 7fd94a95a640 1 -- start start 2024-04-25T01:42:05.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.634+0000 7fd94a95a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd944069dc0 0x7fd944198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.634+0000 7fd94a95a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd94406a780 0x7fd9441985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.634+0000 7fd94a95a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd944107730 0x7fd94419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.634+0000 7fd94a95a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd944111b40 con 0x7fd944107730 2024-04-25T01:42:05.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.634+0000 7fd94a95a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441119c0 con 0x7fd94406a780 2024-04-25T01:42:05.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.634+0000 7fd94a95a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd944111cc0 con 0x7fd944069dc0 2024-04-25T01:42:05.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.635+0000 7fd948ef7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd944107730 0x7fd94419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.635+0000 7fd948ef7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd944107730 0x7fd94419c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46108/0 (socket says 172.21.15.28:46108) 2024-04-25T01:42:05.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.635+0000 7fd948ef7640 1 -- 172.21.15.28:0/3592394895 learned_addr learned my addr 172.21.15.28:0/3592394895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:05.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.635+0000 7fd943fff640 1 --2- 172.21.15.28:0/3592394895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd944069dc0 0x7fd944198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.635+0000 7fd9437fe640 1 --2- 172.21.15.28:0/3592394895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd94406a780 0x7fd9441985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.635+0000 7fd948ef7640 1 -- 172.21.15.28:0/3592394895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd944069dc0 msgr2=0x7fd944198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:05.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.635+0000 7fd948ef7640 1 --2- 172.21.15.28:0/3592394895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd944069dc0 0x7fd944198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.635+0000 7fd948ef7640 1 -- 172.21.15.28:0/3592394895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd94406a780 msgr2=0x7fd9441985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:05.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.635+0000 7fd948ef7640 1 --2- 172.21.15.28:0/3592394895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd94406a780 0x7fd9441985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.635+0000 7fd948ef7640 1 -- 172.21.15.28:0/3592394895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd94419d000 con 0x7fd944107730 2024-04-25T01:42:05.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.635+0000 7fd948ef7640 1 --2- 172.21.15.28:0/3592394895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd944107730 0x7fd94419c8d0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fd93c00eaa0 tx=0x7fd93c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:05.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.635+0000 7fd9417fa640 1 -- 172.21.15.28:0/3592394895 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93c00cdb0 con 0x7fd944107730 2024-04-25T01:42:05.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.635+0000 7fd94a95a640 1 -- 172.21.15.28:0/3592394895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd94419d2f0 con 0x7fd944107730 2024-04-25T01:42:05.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.635+0000 7fd9417fa640 1 -- 172.21.15.28:0/3592394895 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd93c005020 con 0x7fd944107730 2024-04-25T01:42:05.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.635+0000 7fd9417fa640 1 -- 172.21.15.28:0/3592394895 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93c0116d0 con 0x7fd944107730 2024-04-25T01:42:05.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.635+0000 7fd94a95a640 1 -- 172.21.15.28:0/3592394895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9441a4b80 con 0x7fd944107730 2024-04-25T01:42:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.637+0000 7fd9417fa640 1 -- 172.21.15.28:0/3592394895 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd93c0054d0 con 0x7fd944107730 2024-04-25T01:42:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd94d665740 1 Processor -- start 2024-04-25T01:42:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd94d665740 1 -- start start 2024-04-25T01:42:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd94d665740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470360 0x55db21470740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd94d665740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470d10 0x55db2147f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd94d665740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db2147fa30 0x55db21481e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd94d665740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db213d56b0 con 0x55db21470360 2024-04-25T01:42:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd94d665740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55db2137d880 con 0x55db21470d10 2024-04-25T01:42:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd94d665740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55db213d1670 con 0x55db2147fa30 2024-04-25T01:42:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd9417fa640 1 --2- 172.21.15.28:0/3592394895 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd9300a2d00 0x7fd9300a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd9437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470360 0x55db21470740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd9437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470360 0x55db21470740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46110/0 (socket says 172.21.15.28:46110) 2024-04-25T01:42:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd9437fe640 1 -- 172.21.15.28:0/2606450618 learned_addr learned my addr 172.21.15.28:0/2606450618 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd948ef7640 1 --2- 172.21.15.28:0/3592394895 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd9300a2d00 0x7fd9300a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd9437fe640 1 --2- 172.21.15.28:0/2606450618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db2147fa30 0x55db21481e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd943fff640 1 --2- 172.21.15.28:0/2606450618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470d10 0x55db2147f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd9417fa640 1 -- 172.21.15.28:0/3592394895 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fd93c0cf950 con 0x7fd944107730 2024-04-25T01:42:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd9437fe640 1 -- 172.21.15.28:0/2606450618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db2147fa30 msgr2=0x55db21481e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd9437fe640 1 --2- 172.21.15.28:0/2606450618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db2147fa30 0x55db21481e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd9437fe640 1 -- 172.21.15.28:0/2606450618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470d10 msgr2=0x55db2147f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd9437fe640 1 --2- 172.21.15.28:0/2606450618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470d10 0x55db2147f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd9437fe640 1 -- 172.21.15.28:0/2606450618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db211b37d0 con 0x55db21470360 2024-04-25T01:42:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.638+0000 7fd943fff640 1 --2- 172.21.15.28:0/2606450618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470d10 0x55db2147f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.639+0000 7fd9437fe640 1 --2- 172.21.15.28:0/2606450618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470360 0x55db21470740 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fd934005c40 tx=0x7fd934005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=683c743f90f7cc05 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.639+0000 7fd948ef7640 1 --2- 172.21.15.28:0/3592394895 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd9300a2d00 0x7fd9300a51c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd93c009270 tx=0x7fd93c000f40 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.639+0000 7fd93a7fc640 1 -- 172.21.15.28:0/2606450618 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd934044030 con 0x55db21470360 2024-04-25T01:42:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.639+0000 7fd93a7fc640 1 -- 172.21.15.28:0/2606450618 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9340319c0 con 0x55db21470360 2024-04-25T01:42:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.639+0000 7fd93a7fc640 1 -- 172.21.15.28:0/2606450618 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd934035040 con 0x55db21470360 2024-04-25T01:42:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.639+0000 7fd94d665740 1 -- 172.21.15.28:0/2606450618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470360 msgr2=0x55db21470740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.639+0000 7fd94d665740 1 --2- 172.21.15.28:0/2606450618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470360 0x55db21470740 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fd934005c40 tx=0x7fd934005fd0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.639+0000 7fd94d665740 1 -- 172.21.15.28:0/2606450618 shutdown_connections 2024-04-25T01:42:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.639+0000 7fd94d665740 1 --2- 172.21.15.28:0/2606450618 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db2147fa30 0x55db21481e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.639+0000 7fd94d665740 1 --2- 172.21.15.28:0/2606450618 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470d10 0x55db2147f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.639+0000 7fd94d665740 1 --2- 172.21.15.28:0/2606450618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470360 0x55db21470740 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.639+0000 7fd94d665740 1 -- 172.21.15.28:0/2606450618 >> 172.21.15.28:0/2606450618 conn(0x55db2146c950 msgr2=0x55db2146cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.640+0000 7fd94d665740 1 -- 172.21.15.28:0/2606450618 shutdown_connections 2024-04-25T01:42:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.640+0000 7fd94d665740 1 -- 172.21.15.28:0/2606450618 wait complete. 2024-04-25T01:42:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.640+0000 7fd94d665740 1 -- start start 2024-04-25T01:42:05.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.640+0000 7fd94d665740 1 Processor -- start 2024-04-25T01:42:05.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.640+0000 7fd94d665740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470360 0x55db2147c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.640+0000 7fd94d665740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470d10 0x55db2147ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.640+0000 7fd94d665740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db2147fa30 0x55db2147d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.640+0000 7fd94d665740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db21482790 con 0x55db21470d10 2024-04-25T01:42:05.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.640+0000 7fd94d665740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55db21390f50 con 0x55db21470360 2024-04-25T01:42:05.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.640+0000 7fd94d665740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55db21482910 con 0x55db2147fa30 2024-04-25T01:42:05.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.640+0000 7fd94d665740 10 client.0 _ll_register_callbacks cb 0x55db213cc000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:05.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.640+0000 7fd943fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470d10 0x55db2147ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.640+0000 7fd943fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470d10 0x55db2147ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46120/0 (socket says 172.21.15.28:46120) 2024-04-25T01:42:05.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.641+0000 7fd943fff640 1 -- 172.21.15.28:0/1048235800 learned_addr learned my addr 172.21.15.28:0/1048235800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:05.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.641+0000 7fd9437fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470360 0x55db2147c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.641+0000 7fd9437fe640 1 --2- 172.21.15.28:0/1048235800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db2147fa30 0x55db2147d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.641+0000 7fd943fff640 1 -- 172.21.15.28:0/1048235800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db2147fa30 msgr2=0x55db2147d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:05.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.641+0000 7fd943fff640 1 --2- 172.21.15.28:0/1048235800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db2147fa30 0x55db2147d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.641+0000 7fd943fff640 1 -- 172.21.15.28:0/1048235800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470360 msgr2=0x55db2147c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:05.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.641+0000 7fd943fff640 1 --2- 172.21.15.28:0/1048235800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470360 0x55db2147c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.641+0000 7fd943fff640 1 -- 172.21.15.28:0/1048235800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db210e93f0 con 0x55db21470d10 2024-04-25T01:42:05.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.641+0000 7fd943fff640 1 --2- 172.21.15.28:0/1048235800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470d10 0x55db2147ceb0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fd92c012700 tx=0x7fd92c012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d85918892152e2a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:05.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.641+0000 7fd9237fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:05.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.641+0000 7fd9237fe640 1 -- 172.21.15.28:0/1048235800 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92c00f8c0 con 0x55db21470d10 2024-04-25T01:42:05.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.641+0000 7fd9237fe640 1 -- 172.21.15.28:0/1048235800 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fd92c00fa60 con 0x55db21470d10 2024-04-25T01:42:05.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.641+0000 7fd94d665740 1 -- 172.21.15.28:0/1048235800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9340319c0 con 0x55db21470d10 2024-04-25T01:42:05.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.641+0000 7fd9237fe640 1 -- 172.21.15.28:0/1048235800 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd92c025050 con 0x55db21470d10 2024-04-25T01:42:05.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.641+0000 7fd94d665740 10 client.18981 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:05.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.641+0000 7fd94d665740 1 -- 172.21.15.28:0/1048235800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55db210a90f0 con 0x55db21470d10 2024-04-25T01:42:05.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.642+0000 7fd9237fe640 1 -- 172.21.15.28:0/1048235800 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9340319c0 con 0x55db21470d10 2024-04-25T01:42:05.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.642+0000 7fd9237fe640 1 -- 172.21.15.28:0/1048235800 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd92c01cce0 con 0x55db21470d10 2024-04-25T01:42:05.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.642+0000 7fd94d665740 10 client.18981 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:05.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.642+0000 7fd94d665740 10 client.18981 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:05.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.642+0000 7fd94d665740 20 client.18981 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:05.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.642+0000 7fd94d665740 1 --2- 172.21.15.28:0/1048235800 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db2148fd10 0x55db21492170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.642+0000 7fd94d665740 4 client.18981 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:05.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.642+0000 7fd94d665740 1 -- 172.21.15.28:0/1048235800 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db213e2bf0 con 0x55db2148fd10 2024-04-25T01:42:05.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.642+0000 7fd9437fe640 1 --2- 172.21.15.28:0/1048235800 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db2148fd10 0x55db21492170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.643+0000 7fd9237fe640 10 client.18981 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:05.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.643+0000 7fd9437fe640 1 --2- 172.21.15.28:0/1048235800 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db2148fd10 0x55db21492170 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5091c2566095c9bf server_cookie=b9978149acf708f4 in_seq=0 out_seq=0 2024-04-25T01:42:05.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.643+0000 7fd9237fe640 1 -- 172.21.15.28:0/1048235800 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55db213e2bf0 con 0x55db2148fd10 2024-04-25T01:42:05.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.643+0000 7fd9237fe640 10 client.18981 handle_command_reply: tid=0 2024-04-25T01:42:05.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.643+0000 7fd94d665740 1 client.18981 shutdown 2024-04-25T01:42:05.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.643+0000 7fd94d665740 1 -- 172.21.15.28:0/1048235800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470d10 msgr2=0x55db2147ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:05.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.643+0000 7fd94d665740 1 --2- 172.21.15.28:0/1048235800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470d10 0x55db2147ceb0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fd92c012700 tx=0x7fd92c012bc0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.643+0000 7fd94d665740 1 -- 172.21.15.28:0/1048235800 shutdown_connections 2024-04-25T01:42:05.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.643+0000 7fd94d665740 1 --2- 172.21.15.28:0/1048235800 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db2148fd10 0x55db21492170 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.643+0000 7fd94d665740 1 --2- 172.21.15.28:0/1048235800 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db2147fa30 0x55db2147d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.643+0000 7fd94d665740 1 --2- 172.21.15.28:0/1048235800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470d10 0x55db2147ceb0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.643+0000 7fd9237fe640 0 client.18981 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.643+0000 7fd94d665740 1 --2- 172.21.15.28:0/1048235800 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470360 0x55db2147c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.643+0000 7fd94d665740 1 -- 172.21.15.28:0/1048235800 >> 172.21.15.28:0/1048235800 conn(0x55db2146c950 msgr2=0x55db214727c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.644+0000 7fd94d665740 1 -- 172.21.15.28:0/1048235800 shutdown_connections 2024-04-25T01:42:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.644+0000 7fd94d665740 1 -- 172.21.15.28:0/1048235800 wait complete. 2024-04-25T01:42:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.644+0000 7fd94d665740 20 client.18981 trim_cache size 0 max 16384 2024-04-25T01:42:05.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.661+0000 7fd94d665740 1 Processor -- start 2024-04-25T01:42:05.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.661+0000 7fd94d665740 1 -- start start 2024-04-25T01:42:05.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.661+0000 7fd94d665740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470360 0x55db2147f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.661+0000 7fd94d665740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470d10 0x55db2148b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.661+0000 7fd94d665740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db2147fa30 0x55db2146cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.661+0000 7fd94d665740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441ac9c0 con 0x55db21470360 2024-04-25T01:42:05.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.661+0000 7fd94d665740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441ae500 con 0x55db21470d10 2024-04-25T01:42:05.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.661+0000 7fd94d665740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441ad090 con 0x55db2147fa30 2024-04-25T01:42:05.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.661+0000 7fd9437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470360 0x55db2147f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.661+0000 7fd9437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470360 0x55db2147f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46122/0 (socket says 172.21.15.28:46122) 2024-04-25T01:42:05.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.661+0000 7fd9437fe640 1 -- 172.21.15.28:0/2327249229 learned_addr learned my addr 172.21.15.28:0/2327249229 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:05.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.662+0000 7fd943fff640 1 --2- 172.21.15.28:0/2327249229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470d10 0x55db2148b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.662+0000 7fd9437fe640 1 -- 172.21.15.28:0/2327249229 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db2147fa30 msgr2=0x55db2146cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:05.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.662+0000 7fd9437fe640 1 --2- 172.21.15.28:0/2327249229 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db2147fa30 0x55db2146cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.662+0000 7fd9437fe640 1 -- 172.21.15.28:0/2327249229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470d10 msgr2=0x55db2148b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:05.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.662+0000 7fd9437fe640 1 --2- 172.21.15.28:0/2327249229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470d10 0x55db2148b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.662+0000 7fd9437fe640 1 -- 172.21.15.28:0/2327249229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db2146c780 con 0x55db21470360 2024-04-25T01:42:05.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.662+0000 7fd9437fe640 1 --2- 172.21.15.28:0/2327249229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470360 0x55db2147f0a0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fd9340056a0 tx=0x7fd934031740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=326196c2005b910e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:05.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.662+0000 7fd939ffb640 1 -- 172.21.15.28:0/2327249229 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd934035040 con 0x55db21470360 2024-04-25T01:42:05.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.662+0000 7fd939ffb640 1 -- 172.21.15.28:0/2327249229 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd934004440 con 0x55db21470360 2024-04-25T01:42:05.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.662+0000 7fd939ffb640 1 -- 172.21.15.28:0/2327249229 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93404a720 con 0x55db21470360 2024-04-25T01:42:05.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.663+0000 7fd94d665740 1 -- 172.21.15.28:0/2327249229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470360 msgr2=0x55db2147f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:05.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.663+0000 7fd94d665740 1 --2- 172.21.15.28:0/2327249229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470360 0x55db2147f0a0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fd9340056a0 tx=0x7fd934031740 comp rx=0 tx=0).stop 2024-04-25T01:42:05.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.663+0000 7fd94d665740 1 -- 172.21.15.28:0/2327249229 shutdown_connections 2024-04-25T01:42:05.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.663+0000 7fd94d665740 1 --2- 172.21.15.28:0/2327249229 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db2147fa30 0x55db2146cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.663+0000 7fd94d665740 1 --2- 172.21.15.28:0/2327249229 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470d10 0x55db2148b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.663+0000 7fd94d665740 1 --2- 172.21.15.28:0/2327249229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db21470360 0x55db2147f0a0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.663+0000 7fd94d665740 1 -- 172.21.15.28:0/2327249229 >> 172.21.15.28:0/2327249229 conn(0x55db2146c950 msgr2=0x55db2146b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:05.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.663+0000 7fd94d665740 1 -- 172.21.15.28:0/2327249229 shutdown_connections 2024-04-25T01:42:05.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.663+0000 7fd94d665740 1 -- 172.21.15.28:0/2327249229 wait complete. 2024-04-25T01:42:05.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.663+0000 7fd94d665740 1 -- start start 2024-04-25T01:42:05.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.663+0000 7fd94d665740 1 Processor -- start 2024-04-25T01:42:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.663+0000 7fd94d665740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db21470360 0x55db21487ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.663+0000 7fd94d665740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470d10 0x55db21479260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.663+0000 7fd94d665740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2147fa30 0x55db214797a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.663+0000 7fd94d665740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441ade30 con 0x55db2147fa30 2024-04-25T01:42:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.663+0000 7fd94d665740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441ad760 con 0x55db21470d10 2024-04-25T01:42:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.663+0000 7fd94d665740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441aa9c0 con 0x55db21470360 2024-04-25T01:42:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.663+0000 7fd94d665740 10 client.0 _ll_register_callbacks cb 0x7fd92c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.664+0000 7fd9437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2147fa30 0x55db214797a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.664+0000 7fd9437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2147fa30 0x55db214797a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46130/0 (socket says 172.21.15.28:46130) 2024-04-25T01:42:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.664+0000 7fd9437fe640 1 -- 172.21.15.28:0/3091139073 learned_addr learned my addr 172.21.15.28:0/3091139073 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.664+0000 7fd9437fe640 1 --2- 172.21.15.28:0/3091139073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db21470360 0x55db21487ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.664+0000 7fd943fff640 1 --2- 172.21.15.28:0/3091139073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470d10 0x55db21479260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.664+0000 7fd9437fe640 1 -- 172.21.15.28:0/3091139073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db21470360 msgr2=0x55db21487ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.664+0000 7fd9437fe640 1 --2- 172.21.15.28:0/3091139073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db21470360 0x55db21487ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.664+0000 7fd9437fe640 1 -- 172.21.15.28:0/3091139073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470d10 msgr2=0x55db21479260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.664+0000 7fd9437fe640 1 --2- 172.21.15.28:0/3091139073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470d10 0x55db21479260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.664+0000 7fd9437fe640 1 -- 172.21.15.28:0/3091139073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db21479e60 con 0x55db2147fa30 2024-04-25T01:42:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.664+0000 7fd9437fe640 1 --2- 172.21.15.28:0/3091139073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2147fa30 0x55db214797a0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fd934004720 tx=0x7fd934040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e39e1fddb62bb44 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.664+0000 7fd9237fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.664+0000 7fd9237fe640 1 -- 172.21.15.28:0/3091139073 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd934035070 con 0x55db2147fa30 2024-04-25T01:42:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.664+0000 7fd9237fe640 1 -- 172.21.15.28:0/3091139073 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9438+0+0 (secure 0 0 0) 0x7fd934003a00 con 0x55db2147fa30 2024-04-25T01:42:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.664+0000 7fd94d665740 1 -- 172.21.15.28:0/3091139073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd934004440 con 0x55db2147fa30 2024-04-25T01:42:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.664+0000 7fd9237fe640 1 -- 172.21.15.28:0/3091139073 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd934004260 con 0x55db2147fa30 2024-04-25T01:42:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.664+0000 7fd94d665740 10 client.18987 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.664+0000 7fd94d665740 1 -- 172.21.15.28:0/3091139073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55db214f61b0 con 0x55db2147fa30 2024-04-25T01:42:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.664+0000 7fd9237fe640 1 -- 172.21.15.28:0/3091139073 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd934004440 con 0x55db2147fa30 2024-04-25T01:42:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.665+0000 7fd9237fe640 1 -- 172.21.15.28:0/3091139073 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd93404a450 con 0x55db2147fa30 2024-04-25T01:42:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.665+0000 7fd94d665740 10 client.18987 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.665+0000 7fd94d665740 10 client.18987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.665+0000 7fd94d665740 20 client.18987 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.665+0000 7fd94d665740 1 --2- 172.21.15.28:0/3091139073 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db214f6380 0x55db214f8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.665+0000 7fd94d665740 4 client.18987 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:42:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.665+0000 7fd94d665740 1 -- 172.21.15.28:0/3091139073 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55db213dcf40 con 0x55db214f6380 2024-04-25T01:42:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.665+0000 7fd943fff640 1 --2- 172.21.15.28:0/3091139073 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db214f6380 0x55db214f8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.665+0000 7fd9237fe640 10 client.18987 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.665+0000 7fd943fff640 1 --2- 172.21.15.28:0/3091139073 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db214f6380 0x55db214f8770 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e15c54b7640ddb32 server_cookie=974af54e17343947 in_seq=0 out_seq=0 2024-04-25T01:42:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.666+0000 7fd9237fe640 1 -- 172.21.15.28:0/3091139073 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55db213dcf40 con 0x55db214f6380 2024-04-25T01:42:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.666+0000 7fd9237fe640 10 client.18987 handle_command_reply: tid=0 2024-04-25T01:42:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.666+0000 7fd94d665740 1 client.18987 shutdown 2024-04-25T01:42:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.666+0000 7fd94d665740 1 -- 172.21.15.28:0/3091139073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2147fa30 msgr2=0x55db214797a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.666+0000 7fd94d665740 1 --2- 172.21.15.28:0/3091139073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2147fa30 0x55db214797a0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fd934004720 tx=0x7fd934040040 comp rx=0 tx=0).stop 2024-04-25T01:42:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.666+0000 7fd94d665740 1 -- 172.21.15.28:0/3091139073 shutdown_connections 2024-04-25T01:42:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.666+0000 7fd94d665740 1 --2- 172.21.15.28:0/3091139073 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55db214f6380 0x55db214f8770 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.666+0000 7fd94d665740 1 --2- 172.21.15.28:0/3091139073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db2147fa30 0x55db214797a0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.666+0000 7fd9237fe640 0 client.18987 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.666+0000 7fd94d665740 1 --2- 172.21.15.28:0/3091139073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55db21470d10 0x55db21479260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.666+0000 7fd94d665740 1 --2- 172.21.15.28:0/3091139073 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55db21470360 0x55db21487ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.666+0000 7fd94d665740 1 -- 172.21.15.28:0/3091139073 >> 172.21.15.28:0/3091139073 conn(0x55db2146c950 msgr2=0x55db21472730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.666+0000 7fd94d665740 1 -- 172.21.15.28:0/3091139073 shutdown_connections 2024-04-25T01:42:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.666+0000 7fd94d665740 1 -- 172.21.15.28:0/3091139073 wait complete. 2024-04-25T01:42:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.666+0000 7fd94d665740 20 client.18987 trim_cache size 0 max 16384 2024-04-25T01:42:05.660 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:42:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:42:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-25T01:42:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "e108bd65-7cb5-45cd-b13a-cdc2088664e9": { 2024-04-25T01:42:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-25T01:42:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "e108bd65-7cb5-45cd-b13a-cdc2088664e9", 2024-04-25T01:42:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-25T01:42:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:42:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:42:05.660 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:42:05.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.667+0000 7fd94a95a640 1 -- 172.21.15.28:0/3592394895 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd9300a2d00 msgr2=0x7fd9300a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:05.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.667+0000 7fd94a95a640 1 --2- 172.21.15.28:0/3592394895 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd9300a2d00 0x7fd9300a51c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd93c009270 tx=0x7fd93c000f40 comp rx=0 tx=0).stop 2024-04-25T01:42:05.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.667+0000 7fd94a95a640 1 -- 172.21.15.28:0/3592394895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd944107730 msgr2=0x7fd94419c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.667+0000 7fd94a95a640 1 --2- 172.21.15.28:0/3592394895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd944107730 0x7fd94419c8d0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fd93c00eaa0 tx=0x7fd93c00ef60 comp rx=0 tx=0).stop 2024-04-25T01:42:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.667+0000 7fd94a95a640 1 -- 172.21.15.28:0/3592394895 shutdown_connections 2024-04-25T01:42:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.667+0000 7fd94a95a640 1 --2- 172.21.15.28:0/3592394895 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd9300a2d00 0x7fd9300a51c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.667+0000 7fd94a95a640 1 --2- 172.21.15.28:0/3592394895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd944107730 0x7fd94419c8d0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.667+0000 7fd94a95a640 1 --2- 172.21.15.28:0/3592394895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd94406a780 0x7fd9441985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.667+0000 7fd94a95a640 1 --2- 172.21.15.28:0/3592394895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd944069dc0 0x7fd944198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.667+0000 7fd94a95a640 1 -- 172.21.15.28:0/3592394895 >> 172.21.15.28:0/3592394895 conn(0x7fd9440776a0 msgr2=0x7fd9440fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.667+0000 7fd94a95a640 1 -- 172.21.15.28:0/3592394895 shutdown_connections 2024-04-25T01:42:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:05.667+0000 7fd94a95a640 1 -- 172.21.15.28:0/3592394895 wait complete. 2024-04-25T01:42:05.665 INFO:tasks.cephfs.filesystem:scrub status for tag:e108bd65-7cb5-45cd-b13a-cdc2088664e9 - {'path': '/', 'tag': 'e108bd65-7cb5-45cd-b13a-cdc2088664e9', 'options': 'recursive,force'} 2024-04-25T01:42:06.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:06 smithi028 ceph-mon[29390]: pgmap v279: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-25T01:42:06.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:06 smithi130 ceph-mon[32702]: pgmap v279: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-25T01:42:06.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:06 smithi032 ceph-mon[32461]: pgmap v279: 129 pgs: 129 active+clean; 11 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-25T01:42:07.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:07 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:42:07.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:07 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:42:07.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:07 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:42:07.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:07 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:42:07.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:07 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:42:07.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:07 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:42:07.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:07 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:42:07.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:07 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:42:07.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:07 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:42:07.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:07 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:42:07.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:07 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:42:07.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:07 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:42:08.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:08 smithi028 ceph-mon[29390]: pgmap v280: 129 pgs: 129 active+clean; 11 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-25T01:42:08.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:08 smithi130 ceph-mon[32702]: pgmap v280: 129 pgs: 129 active+clean; 11 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-25T01:42:08.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:08 smithi032 ceph-mon[32461]: pgmap v280: 129 pgs: 129 active+clean; 11 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-25T01:42:09.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:09 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:42:09.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:09 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:42:09.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:09 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:42:10.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:10 smithi028 ceph-mon[29390]: pgmap v281: 129 pgs: 129 active+clean; 11 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-04-25T01:42:10.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:10 smithi028 ceph-mon[29390]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T01:42:10.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:10 smithi130 ceph-mon[32702]: pgmap v281: 129 pgs: 129 active+clean; 11 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-04-25T01:42:10.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:10 smithi130 ceph-mon[32702]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T01:42:10.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:10 smithi032 ceph-mon[32461]: pgmap v281: 129 pgs: 129 active+clean; 11 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-04-25T01:42:10.495 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:10 smithi032 ceph-mon[32461]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T01:42:12.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:12 smithi028 ceph-mon[29390]: pgmap v283: 129 pgs: 129 active+clean; 11 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-25T01:42:12.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:12 smithi130 ceph-mon[32702]: pgmap v283: 129 pgs: 129 active+clean; 11 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-25T01:42:12.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:12 smithi032 ceph-mon[32461]: pgmap v283: 129 pgs: 129 active+clean; 11 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-25T01:42:14.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:14 smithi028 ceph-mon[29390]: pgmap v284: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:42:14.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:14 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:42:14.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:14 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:42:14.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:14 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:42:14.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:14 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:42:14.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:14 smithi130 ceph-mon[32702]: pgmap v284: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:42:14.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:14 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:42:14.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:14 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:42:14.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:14 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:42:14.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:14 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:42:14.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:14 smithi032 ceph-mon[32461]: pgmap v284: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:42:14.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:14 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:42:14.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:14 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:42:14.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:14 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:42:14.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:14 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:42:16.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:16 smithi028 ceph-mon[29390]: pgmap v285: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:42:16.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:16 smithi130 ceph-mon[32702]: pgmap v285: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:42:16.494 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:16 smithi032 ceph-mon[32461]: pgmap v285: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:42:18.219 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:17 smithi130 ceph-mon[32702]: pgmap v286: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:42:18.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:17 smithi130 ceph-mon[32702]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T01:42:18.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:17 smithi032 ceph-mon[32461]: pgmap v286: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:42:18.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:17 smithi032 ceph-mon[32461]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T01:42:18.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:17 smithi028 ceph-mon[29390]: pgmap v286: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:42:18.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:17 smithi028 ceph-mon[29390]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T01:42:20.219 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:19 smithi130 ceph-mon[32702]: pgmap v288: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2024-04-25T01:42:20.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:19 smithi032 ceph-mon[32461]: pgmap v288: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2024-04-25T01:42:20.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:19 smithi028 ceph-mon[29390]: pgmap v288: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2024-04-25T01:42:22.219 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:21 smithi130 ceph-mon[32702]: pgmap v289: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:42:22.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:21 smithi032 ceph-mon[32461]: pgmap v289: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:42:22.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:21 smithi028 ceph-mon[29390]: pgmap v289: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-04-25T01:42:24.219 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:23 smithi130 ceph-mon[32702]: pgmap v290: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:42:24.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:23 smithi032 ceph-mon[32461]: pgmap v290: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:42:24.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:23 smithi028 ceph-mon[29390]: pgmap v290: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:42:26.219 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:25 smithi130 ceph-mon[32702]: pgmap v291: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:42:26.244 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:25 smithi032 ceph-mon[32461]: pgmap v291: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:42:26.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:25 smithi028 ceph-mon[29390]: pgmap v291: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:42:28.220 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:27 smithi130 ceph-mon[32702]: pgmap v292: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-04-25T01:42:28.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:27 smithi032 ceph-mon[32461]: pgmap v292: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-04-25T01:42:28.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:27 smithi028 ceph-mon[29390]: pgmap v292: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-04-25T01:42:30.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:29 smithi032 ceph-mon[32461]: pgmap v293: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T01:42:30.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:29 smithi028 ceph-mon[29390]: pgmap v293: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T01:42:30.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:29 smithi130 ceph-mon[32702]: pgmap v293: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T01:42:32.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:31 smithi032 ceph-mon[32461]: pgmap v294: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-25T01:42:32.272 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:31 smithi130 ceph-mon[32702]: pgmap v294: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-25T01:42:32.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:31 smithi028 ceph-mon[29390]: pgmap v294: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-04-25T01:42:34.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:33 smithi032 ceph-mon[32461]: pgmap v295: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:42:34.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:33 smithi028 ceph-mon[29390]: pgmap v295: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:42:34.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:33 smithi130 ceph-mon[32702]: pgmap v295: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:42:35.666 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:42:35.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:42:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.795+0000 7f233b002640 1 -- 172.21.15.28:0/2711628898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f233406a780 msgr2=0x7f233410e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.795+0000 7f233b002640 1 --2- 172.21.15.28:0/2711628898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f233406a780 0x7f233410e5f0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f232c009cd0 tx=0x7f232c02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.795+0000 7f233b002640 1 -- 172.21.15.28:0/2711628898 shutdown_connections 2024-04-25T01:42:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.795+0000 7f233b002640 1 --2- 172.21.15.28:0/2711628898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f233406a780 0x7f233410e5f0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.795+0000 7f233b002640 1 --2- 172.21.15.28:0/2711628898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2334069dc0 0x7f233406a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.795+0000 7f233b002640 1 --2- 172.21.15.28:0/2711628898 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2334107730 0x7f2334107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.795+0000 7f233b002640 1 -- 172.21.15.28:0/2711628898 >> 172.21.15.28:0/2711628898 conn(0x7f23340776a0 msgr2=0x7f2334077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.796+0000 7f233b002640 1 -- 172.21.15.28:0/2711628898 shutdown_connections 2024-04-25T01:42:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.796+0000 7f233b002640 1 -- 172.21.15.28:0/2711628898 wait complete. 2024-04-25T01:42:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.796+0000 7f233b002640 1 Processor -- start 2024-04-25T01:42:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.796+0000 7f233b002640 1 -- start start 2024-04-25T01:42:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.796+0000 7f233b002640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2334069dc0 0x7f2334100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.796+0000 7f233b002640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f233406a780 0x7f2334100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.796+0000 7f233b002640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2334107730 0x7f23341010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.796+0000 7f233b002640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2334111b40 con 0x7f233406a780 2024-04-25T01:42:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.796+0000 7f233b002640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f23341119c0 con 0x7f2334069dc0 2024-04-25T01:42:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.796+0000 7f233b002640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f2334111cc0 con 0x7f2334107730 2024-04-25T01:42:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.797+0000 7f2333fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f233406a780 0x7f2334100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.797+0000 7f2333fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f233406a780 0x7f2334100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49702/0 (socket says 172.21.15.28:49702) 2024-04-25T01:42:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.797+0000 7f2333fff640 1 -- 172.21.15.28:0/1174227797 learned_addr learned my addr 172.21.15.28:0/1174227797 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.797+0000 7f2338d9e640 1 --2- 172.21.15.28:0/1174227797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2334069dc0 0x7f2334100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.797+0000 7f233959f640 1 --2- 172.21.15.28:0/1174227797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2334107730 0x7f23341010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.797+0000 7f2333fff640 1 -- 172.21.15.28:0/1174227797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2334107730 msgr2=0x7f23341010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.797+0000 7f2333fff640 1 --2- 172.21.15.28:0/1174227797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2334107730 0x7f23341010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.797+0000 7f2333fff640 1 -- 172.21.15.28:0/1174227797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2334069dc0 msgr2=0x7f2334100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:35.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.797+0000 7f2333fff640 1 --2- 172.21.15.28:0/1174227797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2334069dc0 0x7f2334100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.797+0000 7f2333fff640 1 -- 172.21.15.28:0/1174227797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23341a6460 con 0x7f233406a780 2024-04-25T01:42:35.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.797+0000 7f2333fff640 1 --2- 172.21.15.28:0/1174227797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f233406a780 0x7f2334100bb0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f232000e9f0 tx=0x7f232000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:35.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.797+0000 7f2331ffb640 1 -- 172.21.15.28:0/1174227797 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232000cd80 con 0x7f233406a780 2024-04-25T01:42:35.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.797+0000 7f2331ffb640 1 -- 172.21.15.28:0/1174227797 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2320010910 con 0x7f233406a780 2024-04-25T01:42:35.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.797+0000 7f233b002640 1 -- 172.21.15.28:0/1174227797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23341a6750 con 0x7f233406a780 2024-04-25T01:42:35.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.797+0000 7f2331ffb640 1 -- 172.21.15.28:0/1174227797 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2320018990 con 0x7f233406a780 2024-04-25T01:42:35.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.798+0000 7f233b002640 1 -- 172.21.15.28:0/1174227797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23341a6ca0 con 0x7f233406a780 2024-04-25T01:42:35.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.800+0000 7f2331ffb640 1 -- 172.21.15.28:0/1174227797 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f2320010ac0 con 0x7f233406a780 2024-04-25T01:42:35.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.800+0000 7f233ddea740 1 Processor -- start 2024-04-25T01:42:35.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.800+0000 7f233ddea740 1 -- start start 2024-04-25T01:42:35.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.800+0000 7f233ddea740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55caa3191630 0x55caa3191a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.800+0000 7f233ddea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 0x55caa319c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.800+0000 7f233ddea740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa319c620 0x55caa319ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.800+0000 7f233ddea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55caa30f66b0 con 0x55caa3191f50 2024-04-25T01:42:35.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.800+0000 7f233ddea740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55caa309e880 con 0x55caa319c620 2024-04-25T01:42:35.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.800+0000 7f233ddea740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55caa30f2670 con 0x55caa3191630 2024-04-25T01:42:35.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.800+0000 7f2331ffb640 1 --2- 172.21.15.28:0/1174227797 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23240a2d00 0x7f23240a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.800+0000 7f2338d9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 0x55caa319c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.800+0000 7f233959f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55caa3191630 0x55caa3191a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.800+0000 7f2338d9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 0x55caa319c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49716/0 (socket says 172.21.15.28:49716) 2024-04-25T01:42:35.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.801+0000 7f2338d9e640 1 -- 172.21.15.28:0/2369020334 learned_addr learned my addr 172.21.15.28:0/2369020334 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:35.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.801+0000 7f233959f640 1 --2- 172.21.15.28:0/1174227797 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23240a2d00 0x7f23240a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.801+0000 7f2331ffb640 1 -- 172.21.15.28:0/1174227797 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f23200cfc50 con 0x7f233406a780 2024-04-25T01:42:35.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.801+0000 7f2333fff640 1 --2- 172.21.15.28:0/2369020334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa319c620 0x55caa319ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.801+0000 7f2338d9e640 1 -- 172.21.15.28:0/2369020334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55caa3191630 msgr2=0x55caa3191a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:35.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.801+0000 7f2338d9e640 1 --2- 172.21.15.28:0/2369020334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55caa3191630 0x55caa3191a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.801+0000 7f2338d9e640 1 -- 172.21.15.28:0/2369020334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa319c620 msgr2=0x55caa319ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:35.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.801+0000 7f2338d9e640 1 --2- 172.21.15.28:0/2369020334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa319c620 0x55caa319ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.801+0000 7f2338d9e640 1 -- 172.21.15.28:0/2369020334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55caa2ed47d0 con 0x55caa3191f50 2024-04-25T01:42:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.801+0000 7f2333fff640 1 --2- 172.21.15.28:0/2369020334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa319c620 0x55caa319ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.801+0000 7f2338d9e640 1 --2- 172.21.15.28:0/2369020334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 0x55caa319c0e0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f2328009c10 tx=0x7f23280117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e34bc01978e5a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.801+0000 7f233959f640 1 --2- 172.21.15.28:0/1174227797 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23240a2d00 0x7f23240a51c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f232c004af0 tx=0x7f232c0095c0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.801+0000 7f231effd640 1 -- 172.21.15.28:0/2369020334 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2328012450 con 0x55caa3191f50 2024-04-25T01:42:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.801+0000 7f231effd640 1 -- 172.21.15.28:0/2369020334 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2328012ad0 con 0x55caa3191f50 2024-04-25T01:42:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.801+0000 7f231effd640 1 -- 172.21.15.28:0/2369020334 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232800fb60 con 0x55caa3191f50 2024-04-25T01:42:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.802+0000 7f233ddea740 1 -- 172.21.15.28:0/2369020334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 msgr2=0x55caa319c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.802+0000 7f233ddea740 1 --2- 172.21.15.28:0/2369020334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 0x55caa319c0e0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f2328009c10 tx=0x7f23280117b0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.802+0000 7f233ddea740 1 -- 172.21.15.28:0/2369020334 shutdown_connections 2024-04-25T01:42:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.802+0000 7f233ddea740 1 --2- 172.21.15.28:0/2369020334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa319c620 0x55caa319ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.802+0000 7f233ddea740 1 --2- 172.21.15.28:0/2369020334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 0x55caa319c0e0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.802+0000 7f233ddea740 1 --2- 172.21.15.28:0/2369020334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55caa3191630 0x55caa3191a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.802+0000 7f233ddea740 1 -- 172.21.15.28:0/2369020334 >> 172.21.15.28:0/2369020334 conn(0x55caa318d950 msgr2=0x55caa318dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.802+0000 7f233ddea740 1 -- 172.21.15.28:0/2369020334 shutdown_connections 2024-04-25T01:42:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.802+0000 7f233ddea740 1 -- 172.21.15.28:0/2369020334 wait complete. 2024-04-25T01:42:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.802+0000 7f233ddea740 1 -- start start 2024-04-25T01:42:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.802+0000 7f233ddea740 1 Processor -- start 2024-04-25T01:42:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.803+0000 7f233ddea740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa3191630 0x55caa3199590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.803+0000 7f233ddea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 0x55caa3199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.803+0000 7f233ddea740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55caa319c620 0x55caa319a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.803+0000 7f233ddea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55caa319f410 con 0x55caa3191f50 2024-04-25T01:42:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.803+0000 7f233ddea740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55caa30b1f50 con 0x55caa3191630 2024-04-25T01:42:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.803+0000 7f233ddea740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55caa319f590 con 0x55caa319c620 2024-04-25T01:42:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.803+0000 7f233ddea740 10 client.0 _ll_register_callbacks cb 0x55caa30ed000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.803+0000 7f2333fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 0x55caa3199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.803+0000 7f2333fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 0x55caa3199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49730/0 (socket says 172.21.15.28:49730) 2024-04-25T01:42:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.803+0000 7f2333fff640 1 -- 172.21.15.28:0/3155551083 learned_addr learned my addr 172.21.15.28:0/3155551083 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.803+0000 7f233959f640 1 --2- 172.21.15.28:0/3155551083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55caa319c620 0x55caa319a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.803+0000 7f2338d9e640 1 --2- 172.21.15.28:0/3155551083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa3191630 0x55caa3199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.803+0000 7f2333fff640 1 -- 172.21.15.28:0/3155551083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55caa319c620 msgr2=0x55caa319a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.803+0000 7f2333fff640 1 --2- 172.21.15.28:0/3155551083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55caa319c620 0x55caa319a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.803+0000 7f2333fff640 1 -- 172.21.15.28:0/3155551083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa3191630 msgr2=0x55caa3199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.803+0000 7f2333fff640 1 --2- 172.21.15.28:0/3155551083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa3191630 0x55caa3199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.803+0000 7f2333fff640 1 -- 172.21.15.28:0/3155551083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55caa2e0a3f0 con 0x55caa3191f50 2024-04-25T01:42:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.803+0000 7f2338d9e640 1 --2- 172.21.15.28:0/3155551083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa3191630 0x55caa3199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.804+0000 7f2333fff640 1 --2- 172.21.15.28:0/3155551083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 0x55caa3199ad0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f23200cd610 tx=0x7f232009c5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7df9938b20a6d24a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.804+0000 7f2313fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.804+0000 7f2313fff640 1 -- 172.21.15.28:0/3155551083 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2320018d30 con 0x55caa3191f50 2024-04-25T01:42:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.804+0000 7f2313fff640 1 -- 172.21.15.28:0/3155551083 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f232009b230 con 0x55caa3191f50 2024-04-25T01:42:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.804+0000 7f233ddea740 1 -- 172.21.15.28:0/3155551083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2328012ad0 con 0x55caa3191f50 2024-04-25T01:42:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.804+0000 7f2313fff640 1 -- 172.21.15.28:0/3155551083 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f23200a4a50 con 0x55caa3191f50 2024-04-25T01:42:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.804+0000 7f233ddea740 10 client.18999 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.804+0000 7f233ddea740 1 -- 172.21.15.28:0/3155551083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55caa2dca0f0 con 0x55caa3191f50 2024-04-25T01:42:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.804+0000 7f2313fff640 1 -- 172.21.15.28:0/3155551083 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2328012ad0 con 0x55caa3191f50 2024-04-25T01:42:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.804+0000 7f2313fff640 1 -- 172.21.15.28:0/3155551083 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f232009aca0 con 0x55caa3191f50 2024-04-25T01:42:35.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.804+0000 7f233ddea740 10 client.18999 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:35.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.804+0000 7f233ddea740 10 client.18999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:35.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.804+0000 7f233ddea740 20 client.18999 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:35.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.804+0000 7f233ddea740 1 --2- 172.21.15.28:0/3155551083 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55caa31ac790 0x55caa31aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.804+0000 7f233ddea740 4 client.18999 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:35.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.804+0000 7f233ddea740 1 -- 172.21.15.28:0/3155551083 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55caa3103bf0 con 0x55caa31ac790 2024-04-25T01:42:35.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.805+0000 7f2338d9e640 1 --2- 172.21.15.28:0/3155551083 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55caa31ac790 0x55caa31aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.805+0000 7f2338d9e640 1 --2- 172.21.15.28:0/3155551083 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55caa31ac790 0x55caa31aeb80 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bed997b683a5790 server_cookie=faf085839ddcf7b2 in_seq=0 out_seq=0 2024-04-25T01:42:35.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.805+0000 7f2313fff640 10 client.18999 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:35.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.806+0000 7f2313fff640 1 -- 172.21.15.28:0/3155551083 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55caa3103bf0 con 0x55caa31ac790 2024-04-25T01:42:35.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.806+0000 7f2313fff640 10 client.18999 handle_command_reply: tid=0 2024-04-25T01:42:35.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.806+0000 7f233ddea740 1 client.18999 shutdown 2024-04-25T01:42:35.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.806+0000 7f233ddea740 1 -- 172.21.15.28:0/3155551083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 msgr2=0x55caa3199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:35.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.806+0000 7f233ddea740 1 --2- 172.21.15.28:0/3155551083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 0x55caa3199ad0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f23200cd610 tx=0x7f232009c5a0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.806+0000 7f233ddea740 1 -- 172.21.15.28:0/3155551083 shutdown_connections 2024-04-25T01:42:35.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.806+0000 7f233ddea740 1 --2- 172.21.15.28:0/3155551083 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55caa31ac790 0x55caa31aeb80 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.806+0000 7f233ddea740 1 --2- 172.21.15.28:0/3155551083 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55caa319c620 0x55caa319a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.806+0000 7f233ddea740 1 --2- 172.21.15.28:0/3155551083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 0x55caa3199ad0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.806+0000 7f2313fff640 0 client.18999 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:35.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.806+0000 7f233ddea740 1 --2- 172.21.15.28:0/3155551083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa3191630 0x55caa3199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.806+0000 7f233ddea740 1 -- 172.21.15.28:0/3155551083 >> 172.21.15.28:0/3155551083 conn(0x55caa318d950 msgr2=0x55caa3193750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:35.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.806+0000 7f233ddea740 1 -- 172.21.15.28:0/3155551083 shutdown_connections 2024-04-25T01:42:35.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.806+0000 7f233ddea740 1 -- 172.21.15.28:0/3155551083 wait complete. 2024-04-25T01:42:35.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.806+0000 7f233ddea740 20 client.18999 trim_cache size 0 max 16384 2024-04-25T01:42:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.824+0000 7f233ddea740 1 Processor -- start 2024-04-25T01:42:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.824+0000 7f233ddea740 1 -- start start 2024-04-25T01:42:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.824+0000 7f233ddea740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa3191630 0x55caa31a82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.824+0000 7f233ddea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 0x55caa31984d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.824+0000 7f233ddea740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55caa319c620 0x55caa3194900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.824+0000 7f233ddea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f23341b0dd0 con 0x55caa3191f50 2024-04-25T01:42:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.824+0000 7f233ddea740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f23341b2910 con 0x55caa3191630 2024-04-25T01:42:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.824+0000 7f233ddea740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f23341b14a0 con 0x55caa319c620 2024-04-25T01:42:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.824+0000 7f2333fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 0x55caa31984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.824+0000 7f2333fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 0x55caa31984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49738/0 (socket says 172.21.15.28:49738) 2024-04-25T01:42:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.824+0000 7f2333fff640 1 -- 172.21.15.28:0/1300471848 learned_addr learned my addr 172.21.15.28:0/1300471848 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.824+0000 7f2338d9e640 1 --2- 172.21.15.28:0/1300471848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa3191630 0x55caa31a82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.824+0000 7f2333fff640 1 -- 172.21.15.28:0/1300471848 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55caa319c620 msgr2=0x55caa3194900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:35.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.824+0000 7f2333fff640 1 --2- 172.21.15.28:0/1300471848 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55caa319c620 0x55caa3194900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.824+0000 7f2333fff640 1 -- 172.21.15.28:0/1300471848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa3191630 msgr2=0x55caa31a82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:35.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.824+0000 7f2333fff640 1 --2- 172.21.15.28:0/1300471848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa3191630 0x55caa31a82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.824+0000 7f2333fff640 1 -- 172.21.15.28:0/1300471848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55caa31892a0 con 0x55caa3191f50 2024-04-25T01:42:35.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.824+0000 7f2338d9e640 1 --2- 172.21.15.28:0/1300471848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa3191630 0x55caa31a82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:35.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.824+0000 7f2333fff640 1 --2- 172.21.15.28:0/1300471848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 0x55caa31984d0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f23200987b0 tx=0x7f23200a0310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b8b8a37be12b539 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:35.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.825+0000 7f231e7fc640 1 -- 172.21.15.28:0/1300471848 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2320018b30 con 0x55caa3191f50 2024-04-25T01:42:35.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.825+0000 7f231e7fc640 1 -- 172.21.15.28:0/1300471848 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2320018cd0 con 0x55caa3191f50 2024-04-25T01:42:35.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.825+0000 7f231e7fc640 1 -- 172.21.15.28:0/1300471848 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2320097d90 con 0x55caa3191f50 2024-04-25T01:42:35.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.825+0000 7f233ddea740 1 -- 172.21.15.28:0/1300471848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 msgr2=0x55caa31984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:35.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.825+0000 7f233ddea740 1 --2- 172.21.15.28:0/1300471848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 0x55caa31984d0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f23200987b0 tx=0x7f23200a0310 comp rx=0 tx=0).stop 2024-04-25T01:42:35.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.825+0000 7f233ddea740 1 -- 172.21.15.28:0/1300471848 shutdown_connections 2024-04-25T01:42:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.825+0000 7f233ddea740 1 --2- 172.21.15.28:0/1300471848 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55caa319c620 0x55caa3194900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.825+0000 7f233ddea740 1 --2- 172.21.15.28:0/1300471848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191f50 0x55caa31984d0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.825+0000 7f233ddea740 1 --2- 172.21.15.28:0/1300471848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa3191630 0x55caa31a82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.825+0000 7f233ddea740 1 -- 172.21.15.28:0/1300471848 >> 172.21.15.28:0/1300471848 conn(0x55caa318d950 msgr2=0x55caa318caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.825+0000 7f233ddea740 1 -- 172.21.15.28:0/1300471848 shutdown_connections 2024-04-25T01:42:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.825+0000 7f233ddea740 1 -- 172.21.15.28:0/1300471848 wait complete. 2024-04-25T01:42:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.826+0000 7f233ddea740 1 -- start start 2024-04-25T01:42:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.826+0000 7f233ddea740 1 Processor -- start 2024-04-25T01:42:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.826+0000 7f233ddea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191630 0x55caa31a49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.826+0000 7f233ddea740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa3191f50 0x55caa3195de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.826+0000 7f233ddea740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55caa319c620 0x55caa3196320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.826+0000 7f233ddea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f23341b2240 con 0x55caa3191630 2024-04-25T01:42:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.826+0000 7f233ddea740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f23341b1b70 con 0x55caa3191f50 2024-04-25T01:42:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.826+0000 7f233ddea740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f23341aedd0 con 0x55caa319c620 2024-04-25T01:42:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.826+0000 7f233ddea740 10 client.0 _ll_register_callbacks cb 0x7f232000e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.826+0000 7f2338d9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191630 0x55caa31a49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.826+0000 7f2338d9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191630 0x55caa31a49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49754/0 (socket says 172.21.15.28:49754) 2024-04-25T01:42:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.826+0000 7f2338d9e640 1 -- 172.21.15.28:0/2437685507 learned_addr learned my addr 172.21.15.28:0/2437685507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.826+0000 7f2333fff640 1 --2- 172.21.15.28:0/2437685507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa3191f50 0x55caa3195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.826+0000 7f233959f640 1 --2- 172.21.15.28:0/2437685507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55caa319c620 0x55caa3196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.826+0000 7f2338d9e640 1 -- 172.21.15.28:0/2437685507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55caa319c620 msgr2=0x55caa3196320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.826+0000 7f2338d9e640 1 --2- 172.21.15.28:0/2437685507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55caa319c620 0x55caa3196320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.826+0000 7f2338d9e640 1 -- 172.21.15.28:0/2437685507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa3191f50 msgr2=0x55caa3195de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.826+0000 7f2338d9e640 1 --2- 172.21.15.28:0/2437685507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa3191f50 0x55caa3195de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.827+0000 7f2338d9e640 1 -- 172.21.15.28:0/2437685507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55caa3196860 con 0x55caa3191630 2024-04-25T01:42:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.827+0000 7f2338d9e640 1 --2- 172.21.15.28:0/2437685507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191630 0x55caa31a49d0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f2328009c10 tx=0x7f2328009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45fb1ce43a605449 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.827+0000 7f2313fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.827+0000 7f2313fff640 1 -- 172.21.15.28:0/2437685507 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2328032060 con 0x55caa3191630 2024-04-25T01:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.827+0000 7f2313fff640 1 -- 172.21.15.28:0/2437685507 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f232800fae0 con 0x55caa3191630 2024-04-25T01:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.827+0000 7f233ddea740 1 -- 172.21.15.28:0/2437685507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2320018cd0 con 0x55caa3191630 2024-04-25T01:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.827+0000 7f2313fff640 1 -- 172.21.15.28:0/2437685507 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2328039050 con 0x55caa3191630 2024-04-25T01:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.827+0000 7f233ddea740 10 client.19005 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.827+0000 7f233ddea740 1 -- 172.21.15.28:0/2437685507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55caa3212980 con 0x55caa3191630 2024-04-25T01:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.827+0000 7f2313fff640 1 -- 172.21.15.28:0/2437685507 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2320018cd0 con 0x55caa3191630 2024-04-25T01:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.827+0000 7f2313fff640 1 -- 172.21.15.28:0/2437685507 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f23280035f0 con 0x55caa3191630 2024-04-25T01:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.827+0000 7f233ddea740 10 client.19005 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.827+0000 7f233ddea740 10 client.19005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.827+0000 7f233ddea740 20 client.19005 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.827+0000 7f233ddea740 1 --2- 172.21.15.28:0/2437685507 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55caa31ac790 0x55caa3214fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.827+0000 7f233ddea740 4 client.19005 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:42:35.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.827+0000 7f233ddea740 1 -- 172.21.15.28:0/2437685507 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55caa30fdf40 con 0x55caa31ac790 2024-04-25T01:42:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.828+0000 7f2333fff640 1 --2- 172.21.15.28:0/2437685507 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55caa31ac790 0x55caa3214fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.828+0000 7f2313fff640 10 client.19005 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.828+0000 7f2333fff640 1 --2- 172.21.15.28:0/2437685507 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55caa31ac790 0x55caa3214fb0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28aa3b61dd98fd57 server_cookie=52808e6f220d544f in_seq=0 out_seq=0 2024-04-25T01:42:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.828+0000 7f2313fff640 1 -- 172.21.15.28:0/2437685507 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55caa30fdf40 con 0x55caa31ac790 2024-04-25T01:42:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.828+0000 7f2313fff640 10 client.19005 handle_command_reply: tid=0 2024-04-25T01:42:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.828+0000 7f233ddea740 1 client.19005 shutdown 2024-04-25T01:42:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.828+0000 7f233ddea740 1 -- 172.21.15.28:0/2437685507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191630 msgr2=0x55caa31a49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.828+0000 7f233ddea740 1 --2- 172.21.15.28:0/2437685507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191630 0x55caa31a49d0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f2328009c10 tx=0x7f2328009680 comp rx=0 tx=0).stop 2024-04-25T01:42:35.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.828+0000 7f233ddea740 1 -- 172.21.15.28:0/2437685507 shutdown_connections 2024-04-25T01:42:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.828+0000 7f233ddea740 1 --2- 172.21.15.28:0/2437685507 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55caa31ac790 0x55caa3214fb0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.828+0000 7f233ddea740 1 --2- 172.21.15.28:0/2437685507 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55caa319c620 0x55caa3196320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.828+0000 7f2313fff640 0 client.19005 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.828+0000 7f233ddea740 1 --2- 172.21.15.28:0/2437685507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55caa3191f50 0x55caa3195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.828+0000 7f233ddea740 1 --2- 172.21.15.28:0/2437685507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55caa3191630 0x55caa31a49d0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.828+0000 7f233ddea740 1 -- 172.21.15.28:0/2437685507 >> 172.21.15.28:0/2437685507 conn(0x55caa318d950 msgr2=0x55caa3199550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.828+0000 7f233ddea740 1 -- 172.21.15.28:0/2437685507 shutdown_connections 2024-04-25T01:42:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.829+0000 7f233ddea740 1 -- 172.21.15.28:0/2437685507 wait complete. 2024-04-25T01:42:35.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.829+0000 7f233ddea740 20 client.19005 trim_cache size 0 max 16384 2024-04-25T01:42:35.825 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:42:35.825 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:42:35.825 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:42:35.825 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:42:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.829+0000 7f233b002640 1 -- 172.21.15.28:0/1174227797 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23240a2d00 msgr2=0x7f23240a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.829+0000 7f233b002640 1 --2- 172.21.15.28:0/1174227797 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23240a2d00 0x7f23240a51c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f232c004af0 tx=0x7f232c0095c0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.829+0000 7f233b002640 1 -- 172.21.15.28:0/1174227797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f233406a780 msgr2=0x7f2334100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.829+0000 7f233b002640 1 --2- 172.21.15.28:0/1174227797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f233406a780 0x7f2334100bb0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f232000e9f0 tx=0x7f232000eeb0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.829+0000 7f233b002640 1 -- 172.21.15.28:0/1174227797 shutdown_connections 2024-04-25T01:42:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.829+0000 7f233b002640 1 --2- 172.21.15.28:0/1174227797 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f23240a2d00 0x7f23240a51c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.829+0000 7f233b002640 1 --2- 172.21.15.28:0/1174227797 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f2334107730 0x7f23341010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.829+0000 7f233b002640 1 --2- 172.21.15.28:0/1174227797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f233406a780 0x7f2334100bb0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.829+0000 7f233b002640 1 --2- 172.21.15.28:0/1174227797 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2334069dc0 0x7f2334100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.829+0000 7f233b002640 1 -- 172.21.15.28:0/1174227797 >> 172.21.15.28:0/1174227797 conn(0x7f23340776a0 msgr2=0x7f23340fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.829+0000 7f233b002640 1 -- 172.21.15.28:0/1174227797 shutdown_connections 2024-04-25T01:42:35.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.829+0000 7f233b002640 1 -- 172.21.15.28:0/1174227797 wait complete. 2024-04-25T01:42:35.828 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:42:35.828 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:42:35.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.942+0000 7fb52fb7d640 1 -- 172.21.15.28:0/857183319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb52810bd60 msgr2=0x7fb52810e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.942+0000 7fb52fb7d640 1 --2- 172.21.15.28:0/857183319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb52810bd60 0x7fb52810e1d0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fb520009ca0 tx=0x7fb52002a950 comp rx=0 tx=0).stop 2024-04-25T01:42:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.942+0000 7fb52fb7d640 1 -- 172.21.15.28:0/857183319 shutdown_connections 2024-04-25T01:42:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.942+0000 7fb52fb7d640 1 --2- 172.21.15.28:0/857183319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb52810bd60 0x7fb52810e1d0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.942+0000 7fb52fb7d640 1 --2- 172.21.15.28:0/857183319 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb528103340 0x7fb52810b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.942+0000 7fb52fb7d640 1 --2- 172.21.15.28:0/857183319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb528102a20 0x7fb528102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.942+0000 7fb52fb7d640 1 -- 172.21.15.28:0/857183319 >> 172.21.15.28:0/857183319 conn(0x7fb5280777a0 msgr2=0x7fb528077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.942+0000 7fb52fb7d640 1 -- 172.21.15.28:0/857183319 shutdown_connections 2024-04-25T01:42:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.942+0000 7fb52fb7d640 1 -- 172.21.15.28:0/857183319 wait complete. 2024-04-25T01:42:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.943+0000 7fb52fb7d640 1 Processor -- start 2024-04-25T01:42:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.943+0000 7fb52fb7d640 1 -- start start 2024-04-25T01:42:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.943+0000 7fb52fb7d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb528102a20 0x7fb528197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.943+0000 7fb52fb7d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb528103340 0x7fb528198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.943+0000 7fb52fb7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb52810bd60 0x7fb52819c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.943+0000 7fb52fb7d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb528111b30 con 0x7fb52810bd60 2024-04-25T01:42:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.943+0000 7fb52fb7d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb5281119b0 con 0x7fb528102a20 2024-04-25T01:42:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.943+0000 7fb52fb7d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb528111cb0 con 0x7fb528103340 2024-04-25T01:42:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.943+0000 7fb52e11a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb52810bd60 0x7fb52819c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.943+0000 7fb52e11a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb52810bd60 0x7fb52819c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49774/0 (socket says 172.21.15.28:49774) 2024-04-25T01:42:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.943+0000 7fb52e11a640 1 -- 172.21.15.28:0/236939955 learned_addr learned my addr 172.21.15.28:0/236939955 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.943+0000 7fb52d118640 1 --2- 172.21.15.28:0/236939955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb528103340 0x7fb528198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.943+0000 7fb52d919640 1 --2- 172.21.15.28:0/236939955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb528102a20 0x7fb528197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.943+0000 7fb52e11a640 1 -- 172.21.15.28:0/236939955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb528103340 msgr2=0x7fb528198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.943+0000 7fb52e11a640 1 --2- 172.21.15.28:0/236939955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb528103340 0x7fb528198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.943+0000 7fb52e11a640 1 -- 172.21.15.28:0/236939955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb528102a20 msgr2=0x7fb528197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.943+0000 7fb52e11a640 1 --2- 172.21.15.28:0/236939955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb528102a20 0x7fb528197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.943+0000 7fb52e11a640 1 -- 172.21.15.28:0/236939955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb52819cf90 con 0x7fb52810bd60 2024-04-25T01:42:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.944+0000 7fb52e11a640 1 --2- 172.21.15.28:0/236939955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb52810bd60 0x7fb52819c860 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fb520004af0 tx=0x7fb52002f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.944+0000 7fb51effd640 1 -- 172.21.15.28:0/236939955 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520034040 con 0x7fb52810bd60 2024-04-25T01:42:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.944+0000 7fb52fb7d640 1 -- 172.21.15.28:0/236939955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb52819d220 con 0x7fb52810bd60 2024-04-25T01:42:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.944+0000 7fb51effd640 1 -- 172.21.15.28:0/236939955 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb520030920 con 0x7fb52810bd60 2024-04-25T01:42:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.944+0000 7fb51effd640 1 -- 172.21.15.28:0/236939955 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5200047b0 con 0x7fb52810bd60 2024-04-25T01:42:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.944+0000 7fb52fb7d640 1 -- 172.21.15.28:0/236939955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5281a4b10 con 0x7fb52810bd60 2024-04-25T01:42:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.945+0000 7fb52fb7d640 1 -- 172.21.15.28:0/236939955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5280689b0 con 0x7fb52810bd60 2024-04-25T01:42:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.947+0000 7fb51effd640 1 -- 172.21.15.28:0/236939955 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb520037020 con 0x7fb52810bd60 2024-04-25T01:42:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.947+0000 7fb51effd640 1 --2- 172.21.15.28:0/236939955 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb5140a2d00 0x7fb5140a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.947+0000 7fb52d919640 1 --2- 172.21.15.28:0/236939955 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb5140a2d00 0x7fb5140a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.947+0000 7fb51effd640 1 -- 172.21.15.28:0/236939955 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7fb5200f0490 con 0x7fb52810bd60 2024-04-25T01:42:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.948+0000 7fb52d919640 1 --2- 172.21.15.28:0/236939955 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb5140a2d00 0x7fb5140a51c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb524005d00 tx=0x7fb52400b410 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:35.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:35.951+0000 7fb51effd640 1 -- 172.21.15.28:0/236939955 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb5200bce30 con 0x7fb52810bd60 2024-04-25T01:42:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.177+0000 7fb52fb7d640 1 -- 172.21.15.28:0/236939955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5281a4d40 con 0x7fb52810bd60 2024-04-25T01:42:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.178+0000 7fb51effd640 1 -- 172.21.15.28:0/236939955 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7fb5200bc730 con 0x7fb52810bd60 2024-04-25T01:42:36.166 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:42:36.166 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:42:36.166 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:42:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.180+0000 7fb52fb7d640 1 -- 172.21.15.28:0/236939955 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb5140a2d00 msgr2=0x7fb5140a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.180+0000 7fb52fb7d640 1 --2- 172.21.15.28:0/236939955 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb5140a2d00 0x7fb5140a51c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb524005d00 tx=0x7fb52400b410 comp rx=0 tx=0).stop 2024-04-25T01:42:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.180+0000 7fb52fb7d640 1 -- 172.21.15.28:0/236939955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb52810bd60 msgr2=0x7fb52819c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.180+0000 7fb52fb7d640 1 --2- 172.21.15.28:0/236939955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb52810bd60 0x7fb52819c860 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fb520004af0 tx=0x7fb52002f5b0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.180+0000 7fb52fb7d640 1 -- 172.21.15.28:0/236939955 shutdown_connections 2024-04-25T01:42:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.180+0000 7fb52fb7d640 1 --2- 172.21.15.28:0/236939955 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb5140a2d00 0x7fb5140a51c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.180+0000 7fb52fb7d640 1 --2- 172.21.15.28:0/236939955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb52810bd60 0x7fb52819c860 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.180+0000 7fb52fb7d640 1 --2- 172.21.15.28:0/236939955 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb528103340 0x7fb528198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.180+0000 7fb52fb7d640 1 --2- 172.21.15.28:0/236939955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb528102a20 0x7fb528197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.180+0000 7fb52fb7d640 1 -- 172.21.15.28:0/236939955 >> 172.21.15.28:0/236939955 conn(0x7fb5280777a0 msgr2=0x7fb5280fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.180+0000 7fb52fb7d640 1 -- 172.21.15.28:0/236939955 shutdown_connections 2024-04-25T01:42:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.180+0000 7fb52fb7d640 1 -- 172.21.15.28:0/236939955 wait complete. 2024-04-25T01:42:36.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:42:36.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:35 smithi032 ceph-mon[32461]: pgmap v296: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:42:36.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.295+0000 7fd31af89640 1 -- 172.21.15.28:0/1428232135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314106690 msgr2=0x7fd314106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.295+0000 7fd31af89640 1 --2- 172.21.15.28:0/1428232135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314106690 0x7fd314106a70 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fd304009900 tx=0x7fd30402a410 comp rx=0 tx=0).stop 2024-04-25T01:42:36.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.295+0000 7fd31af89640 1 -- 172.21.15.28:0/1428232135 shutdown_connections 2024-04-25T01:42:36.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.295+0000 7fd31af89640 1 --2- 172.21.15.28:0/1428232135 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd314101330 0x7fd31410d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.295+0000 7fd31af89640 1 --2- 172.21.15.28:0/1428232135 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd314100990 0x7fd314100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.295+0000 7fd31af89640 1 --2- 172.21.15.28:0/1428232135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314106690 0x7fd314106a70 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.295+0000 7fd31af89640 1 -- 172.21.15.28:0/1428232135 >> 172.21.15.28:0/1428232135 conn(0x7fd314076440 msgr2=0x7fd314076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.295+0000 7fd31af89640 1 -- 172.21.15.28:0/1428232135 shutdown_connections 2024-04-25T01:42:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.295+0000 7fd31af89640 1 -- 172.21.15.28:0/1428232135 wait complete. 2024-04-25T01:42:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.295+0000 7fd31af89640 1 Processor -- start 2024-04-25T01:42:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd31af89640 1 -- start start 2024-04-25T01:42:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd31af89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314100990 0x7fd314198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd31af89640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd314101330 0x7fd3141985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd31af89640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd314106690 0x7fd31419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd31af89640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd314111bc0 con 0x7fd314100990 2024-04-25T01:42:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd31af89640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd314111a40 con 0x7fd314101330 2024-04-25T01:42:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd31af89640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd314111d40 con 0x7fd314106690 2024-04-25T01:42:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd318d25640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314100990 0x7fd314198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd318d25640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314100990 0x7fd314198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49798/0 (socket says 172.21.15.28:49798) 2024-04-25T01:42:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd318d25640 1 -- 172.21.15.28:0/2677349339 learned_addr learned my addr 172.21.15.28:0/2677349339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd313fff640 1 --2- 172.21.15.28:0/2677349339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd314101330 0x7fd3141985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd318d25640 1 -- 172.21.15.28:0/2677349339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd314106690 msgr2=0x7fd31419c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd318d25640 1 --2- 172.21.15.28:0/2677349339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd314106690 0x7fd31419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd318d25640 1 -- 172.21.15.28:0/2677349339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd314101330 msgr2=0x7fd3141985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd318d25640 1 --2- 172.21.15.28:0/2677349339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd314101330 0x7fd3141985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd318d25640 1 -- 172.21.15.28:0/2677349339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31419d050 con 0x7fd314100990 2024-04-25T01:42:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd318d25640 1 --2- 172.21.15.28:0/2677349339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314100990 0x7fd314198060 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fd304006010 tx=0x7fd304003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd311ffb640 1 -- 172.21.15.28:0/2677349339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30403f070 con 0x7fd314100990 2024-04-25T01:42:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd311ffb640 1 -- 172.21.15.28:0/2677349339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd304031c90 con 0x7fd314100990 2024-04-25T01:42:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.296+0000 7fd31af89640 1 -- 172.21.15.28:0/2677349339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd31419d2e0 con 0x7fd314100990 2024-04-25T01:42:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.297+0000 7fd311ffb640 1 -- 172.21.15.28:0/2677349339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd304035030 con 0x7fd314100990 2024-04-25T01:42:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.297+0000 7fd31af89640 1 -- 172.21.15.28:0/2677349339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3141a4bd0 con 0x7fd314100990 2024-04-25T01:42:36.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.299+0000 7fd311ffb640 1 -- 172.21.15.28:0/2677349339 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fd304042070 con 0x7fd314100990 2024-04-25T01:42:36.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.299+0000 7fd31dd71740 1 Processor -- start 2024-04-25T01:42:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.299+0000 7fd31dd71740 1 -- start start 2024-04-25T01:42:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.299+0000 7fd31dd71740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9d9580 0x55bf8a9d9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.299+0000 7fd31dd71740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9e4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd31dd71740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9e4570 0x55bf8a9e6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd31dd71740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bf8a93e670 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd31dd71740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55bf8a8e6970 con 0x55bf8a9e4570 2024-04-25T01:42:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd31dd71740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55bf8a93a630 con 0x55bf8a9d9580 2024-04-25T01:42:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd311ffb640 1 --2- 172.21.15.28:0/2677349339 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd3000a2d00 0x7fd3000a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd318d25640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9e4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd318d25640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9e4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49808/0 (socket says 172.21.15.28:49808) 2024-04-25T01:42:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd318d25640 1 -- 172.21.15.28:0/3285115051 learned_addr learned my addr 172.21.15.28:0/3285115051 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd319526640 1 --2- 172.21.15.28:0/2677349339 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd3000a2d00 0x7fd3000a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd313fff640 1 --2- 172.21.15.28:0/3285115051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9e4570 0x55bf8a9e6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd319526640 1 --2- 172.21.15.28:0/3285115051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9d9580 0x55bf8a9d9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd311ffb640 1 -- 172.21.15.28:0/2677349339 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7fd3040f01f0 con 0x7fd314100990 2024-04-25T01:42:36.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd318d25640 1 -- 172.21.15.28:0/3285115051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9d9580 msgr2=0x55bf8a9d9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd318d25640 1 --2- 172.21.15.28:0/3285115051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9d9580 0x55bf8a9d9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd318d25640 1 -- 172.21.15.28:0/3285115051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9e4570 msgr2=0x55bf8a9e6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd318d25640 1 --2- 172.21.15.28:0/3285115051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9e4570 0x55bf8a9e6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd318d25640 1 -- 172.21.15.28:0/3285115051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf8a90a400 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd318d25640 1 --2- 172.21.15.28:0/3285115051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9e4030 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fd3040b8730 tx=0x7fd304009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9bb5cb5309ccabc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd319526640 1 --2- 172.21.15.28:0/2677349339 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd3000a2d00 0x7fd3000a51c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd30c008fd0 tx=0x7fd30c00b250 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd30affd640 1 -- 172.21.15.28:0/3285115051 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd304049070 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd30affd640 1 -- 172.21.15.28:0/3285115051 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd304031e40 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.300+0000 7fd30affd640 1 -- 172.21.15.28:0/3285115051 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30403b5f0 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.301+0000 7fd31dd71740 1 -- 172.21.15.28:0/3285115051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 msgr2=0x55bf8a9e4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.301+0000 7fd31dd71740 1 --2- 172.21.15.28:0/3285115051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9e4030 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fd3040b8730 tx=0x7fd304009260 comp rx=0 tx=0).stop 2024-04-25T01:42:36.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.301+0000 7fd31dd71740 1 -- 172.21.15.28:0/3285115051 shutdown_connections 2024-04-25T01:42:36.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.301+0000 7fd31dd71740 1 --2- 172.21.15.28:0/3285115051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9e4570 0x55bf8a9e6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.301+0000 7fd31dd71740 1 --2- 172.21.15.28:0/3285115051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9e4030 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.301+0000 7fd31dd71740 1 --2- 172.21.15.28:0/3285115051 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9d9580 0x55bf8a9d9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.301+0000 7fd31dd71740 1 -- 172.21.15.28:0/3285115051 >> 172.21.15.28:0/3285115051 conn(0x55bf8a9d58a0 msgr2=0x55bf8a9d5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.301+0000 7fd31dd71740 1 -- 172.21.15.28:0/3285115051 shutdown_connections 2024-04-25T01:42:36.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.301+0000 7fd31dd71740 1 -- 172.21.15.28:0/3285115051 wait complete. 2024-04-25T01:42:36.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.301+0000 7fd31dd71740 1 -- start start 2024-04-25T01:42:36.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.302+0000 7fd31dd71740 1 Processor -- start 2024-04-25T01:42:36.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.302+0000 7fd31dd71740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9d9580 0x55bf8a9e3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.302+0000 7fd31dd71740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9e1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.302+0000 7fd31dd71740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9e4570 0x55bf8a9e17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.302+0000 7fd31dd71740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bf8a9e7120 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.302+0000 7fd31dd71740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55bf8a8fa050 con 0x55bf8a9d9580 2024-04-25T01:42:36.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.302+0000 7fd31dd71740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55bf8a9e72a0 con 0x55bf8a9e4570 2024-04-25T01:42:36.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.302+0000 7fd31dd71740 10 client.0 _ll_register_callbacks cb 0x55bf8a651be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:36.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.302+0000 7fd313fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9e1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.302+0000 7fd318d25640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9d9580 0x55bf8a9e3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.302+0000 7fd313fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9e1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49812/0 (socket says 172.21.15.28:49812) 2024-04-25T01:42:36.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.302+0000 7fd313fff640 1 -- 172.21.15.28:0/1952493311 learned_addr learned my addr 172.21.15.28:0/1952493311 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.302+0000 7fd319526640 1 --2- 172.21.15.28:0/1952493311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9e4570 0x55bf8a9e17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.302+0000 7fd318d25640 1 -- 172.21.15.28:0/1952493311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9e4570 msgr2=0x55bf8a9e17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.302+0000 7fd318d25640 1 --2- 172.21.15.28:0/1952493311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9e4570 0x55bf8a9e17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.302+0000 7fd318d25640 1 -- 172.21.15.28:0/1952493311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 msgr2=0x55bf8a9e1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.303+0000 7fd318d25640 1 --2- 172.21.15.28:0/1952493311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9e1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.303+0000 7fd318d25640 1 -- 172.21.15.28:0/1952493311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf8a8cbdc0 con 0x55bf8a9d9580 2024-04-25T01:42:36.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.303+0000 7fd313fff640 1 --2- 172.21.15.28:0/1952493311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9e1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:42:36.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.303+0000 7fd318d25640 1 --2- 172.21.15.28:0/1952493311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9d9580 0x55bf8a9e3f10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd3040c0890 tx=0x7fd304005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=757d65ca21578b26 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.303+0000 7fd2f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-25T01:42:36.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.303+0000 7fd2f3fff640 1 -- 172.21.15.28:0/1952493311 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd304049070 con 0x55bf8a9d9580 2024-04-25T01:42:36.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.303+0000 7fd2f3fff640 1 -- 172.21.15.28:0/1952493311 <== mon.2 v2:172.21.15.32:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7fd3040c4b00 con 0x55bf8a9d9580 2024-04-25T01:42:36.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.303+0000 7fd31dd71740 1 -- 172.21.15.28:0/1952493311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bf8a94bbb0 con 0x55bf8a9d9580 2024-04-25T01:42:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.303+0000 7fd2f3fff640 1 -- 172.21.15.28:0/1952493311 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd3040ed9e0 con 0x55bf8a9d9580 2024-04-25T01:42:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.303+0000 7fd31dd71740 10 client.27746 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.303+0000 7fd31dd71740 1 -- 172.21.15.28:0/1952493311 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bf8a6523f0 con 0x55bf8a9d9580 2024-04-25T01:42:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.303+0000 7fd2f3fff640 1 -- 172.21.15.28:0/1952493311 <== mon.2 v2:172.21.15.32:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bf8a94bbb0 con 0x55bf8a9d9580 2024-04-25T01:42:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.303+0000 7fd2f3fff640 1 -- 172.21.15.28:0/1952493311 <== mon.2 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd3040d50a0 con 0x55bf8a9d9580 2024-04-25T01:42:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.303+0000 7fd31dd71740 10 client.27746 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.304+0000 7fd31dd71740 10 client.27746 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:42:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.304+0000 7fd31dd71740 20 client.27746 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.304+0000 7fd31dd71740 1 --2- 172.21.15.28:0/1952493311 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55bf8a9f41c0 0x55bf8a9f65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.304+0000 7fd31dd71740 4 client.27746 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.304+0000 7fd31dd71740 1 -- 172.21.15.28:0/1952493311 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf8a945f00 con 0x55bf8a9f41c0 2024-04-25T01:42:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.304+0000 7fd313fff640 1 --2- 172.21.15.28:0/1952493311 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55bf8a9f41c0 0x55bf8a9f65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.304+0000 7fd313fff640 1 --2- 172.21.15.28:0/1952493311 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55bf8a9f41c0 0x55bf8a9f65d0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a68d77d189378f17 server_cookie=f4704f332101ca9c in_seq=0 out_seq=0 2024-04-25T01:42:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.304+0000 7fd2f3fff640 10 client.27746 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:42:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.305+0000 7fd2f3fff640 1 -- 172.21.15.28:0/1952493311 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bf8a945f00 con 0x55bf8a9f41c0 2024-04-25T01:42:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.305+0000 7fd2f3fff640 10 client.27746 handle_command_reply: tid=0 2024-04-25T01:42:36.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.305+0000 7fd31dd71740 1 client.27746 shutdown 2024-04-25T01:42:36.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.305+0000 7fd31dd71740 1 -- 172.21.15.28:0/1952493311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9d9580 msgr2=0x55bf8a9e3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.305+0000 7fd31dd71740 1 --2- 172.21.15.28:0/1952493311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9d9580 0x55bf8a9e3f10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd3040c0890 tx=0x7fd304005d50 comp rx=0 tx=0).stop 2024-04-25T01:42:36.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.305+0000 7fd31dd71740 1 -- 172.21.15.28:0/1952493311 shutdown_connections 2024-04-25T01:42:36.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.305+0000 7fd31dd71740 1 --2- 172.21.15.28:0/1952493311 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55bf8a9f41c0 0x55bf8a9f65d0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.305+0000 7fd31dd71740 1 --2- 172.21.15.28:0/1952493311 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9e4570 0x55bf8a9e17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.305+0000 7fd31dd71740 1 --2- 172.21.15.28:0/1952493311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9e1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.305+0000 7fd2f3fff640 0 client.27746 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:42:36.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.305+0000 7fd31dd71740 1 --2- 172.21.15.28:0/1952493311 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9d9580 0x55bf8a9e3f10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.305+0000 7fd31dd71740 1 -- 172.21.15.28:0/1952493311 >> 172.21.15.28:0/1952493311 conn(0x55bf8a9d58a0 msgr2=0x55bf8a9d4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.305+0000 7fd31dd71740 1 -- 172.21.15.28:0/1952493311 shutdown_connections 2024-04-25T01:42:36.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.305+0000 7fd31dd71740 1 -- 172.21.15.28:0/1952493311 wait complete. 2024-04-25T01:42:36.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.306+0000 7fd31dd71740 20 client.27746 trim_cache size 0 max 16384 2024-04-25T01:42:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd31dd71740 1 Processor -- start 2024-04-25T01:42:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd31dd71740 1 -- start start 2024-04-25T01:42:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd31dd71740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9d9580 0x55bf8a9eff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd31dd71740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9d4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd31dd71740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9e4570 0x55bf8a9e3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd31dd71740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd3141aec00 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd31dd71740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd3141b0740 con 0x55bf8a9d9580 2024-04-25T01:42:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd31dd71740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd3141af2d0 con 0x55bf8a9e4570 2024-04-25T01:42:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd313fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9d4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd313fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9d4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49820/0 (socket says 172.21.15.28:49820) 2024-04-25T01:42:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd313fff640 1 -- 172.21.15.28:0/2795325527 learned_addr learned my addr 172.21.15.28:0/2795325527 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd318d25640 1 --2- 172.21.15.28:0/2795325527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9d9580 0x55bf8a9eff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd319526640 1 --2- 172.21.15.28:0/2795325527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9e4570 0x55bf8a9e3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd313fff640 1 -- 172.21.15.28:0/2795325527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9e4570 msgr2=0x55bf8a9e3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd313fff640 1 --2- 172.21.15.28:0/2795325527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9e4570 0x55bf8a9e3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd313fff640 1 -- 172.21.15.28:0/2795325527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9d9580 msgr2=0x55bf8a9eff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd313fff640 1 --2- 172.21.15.28:0/2795325527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9d9580 0x55bf8a9eff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd313fff640 1 -- 172.21.15.28:0/2795325527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf8a9e01b0 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd318d25640 1 --2- 172.21.15.28:0/2795325527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9d9580 0x55bf8a9eff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:36.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd319526640 1 --2- 172.21.15.28:0/2795325527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9e4570 0x55bf8a9e3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:36.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.319+0000 7fd313fff640 1 --2- 172.21.15.28:0/2795325527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9d4b40 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x55bf8a9f6e10 tx=0x7fd2fc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40a4b4aaf849f15 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.320+0000 7fd30a7fc640 1 -- 172.21.15.28:0/2795325527 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc016920 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.320+0000 7fd30a7fc640 1 -- 172.21.15.28:0/2795325527 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2fc01c250 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.320+0000 7fd30a7fc640 1 -- 172.21.15.28:0/2795325527 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc025620 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.320+0000 7fd31dd71740 1 -- 172.21.15.28:0/2795325527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 msgr2=0x55bf8a9d4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.320+0000 7fd31dd71740 1 --2- 172.21.15.28:0/2795325527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9d4b40 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x55bf8a9f6e10 tx=0x7fd2fc008040 comp rx=0 tx=0).stop 2024-04-25T01:42:36.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.320+0000 7fd31dd71740 1 -- 172.21.15.28:0/2795325527 shutdown_connections 2024-04-25T01:42:36.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.320+0000 7fd31dd71740 1 --2- 172.21.15.28:0/2795325527 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9e4570 0x55bf8a9e3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.320+0000 7fd31dd71740 1 --2- 172.21.15.28:0/2795325527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9d4b40 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.320+0000 7fd31dd71740 1 --2- 172.21.15.28:0/2795325527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9d9580 0x55bf8a9eff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.320+0000 7fd31dd71740 1 -- 172.21.15.28:0/2795325527 >> 172.21.15.28:0/2795325527 conn(0x55bf8a9d58a0 msgr2=0x55bf8a9d5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.320+0000 7fd31dd71740 1 -- 172.21.15.28:0/2795325527 shutdown_connections 2024-04-25T01:42:36.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.320+0000 7fd31dd71740 1 -- 172.21.15.28:0/2795325527 wait complete. 2024-04-25T01:42:36.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.321+0000 7fd31dd71740 1 -- start start 2024-04-25T01:42:36.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.321+0000 7fd31dd71740 1 Processor -- start 2024-04-25T01:42:36.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.321+0000 7fd31dd71740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9d9580 0x55bf8a9ec7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.321+0000 7fd31dd71740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9e7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.321+0000 7fd31dd71740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9e4570 0x55bf8a9e8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.321+0000 7fd31dd71740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd3141b0070 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.321+0000 7fd31dd71740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd3141af9a0 con 0x55bf8a9e4570 2024-04-25T01:42:36.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.321+0000 7fd31dd71740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fd3141acc00 con 0x55bf8a9d9580 2024-04-25T01:42:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.321+0000 7fd31dd71740 10 client.0 _ll_register_callbacks cb 0x7fd2fc00b180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.321+0000 7fd313fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9e7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.321+0000 7fd313fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9e7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49824/0 (socket says 172.21.15.28:49824) 2024-04-25T01:42:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.321+0000 7fd313fff640 1 -- 172.21.15.28:0/862369708 learned_addr learned my addr 172.21.15.28:0/862369708 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.321+0000 7fd318d25640 1 --2- 172.21.15.28:0/862369708 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9d9580 0x55bf8a9ec7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.321+0000 7fd319526640 1 --2- 172.21.15.28:0/862369708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9e4570 0x55bf8a9e8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.321+0000 7fd313fff640 1 -- 172.21.15.28:0/862369708 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9d9580 msgr2=0x55bf8a9ec7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.321+0000 7fd313fff640 1 --2- 172.21.15.28:0/862369708 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9d9580 0x55bf8a9ec7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.321+0000 7fd313fff640 1 -- 172.21.15.28:0/862369708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9e4570 msgr2=0x55bf8a9e8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.321+0000 7fd313fff640 1 --2- 172.21.15.28:0/862369708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9e4570 0x55bf8a9e8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.321+0000 7fd313fff640 1 -- 172.21.15.28:0/862369708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf8a9d9cd0 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.322+0000 7fd313fff640 1 --2- 172.21.15.28:0/862369708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9e7b50 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc004810 tx=0x7fd2fc00bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61fb6d37fb92e6f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.322+0000 7fd2f3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.322+0000 7fd2f3fff640 1 -- 172.21.15.28:0/862369708 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc029020 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.322+0000 7fd31dd71740 1 -- 172.21.15.28:0/862369708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2fc01c250 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.322+0000 7fd2f3fff640 1 -- 172.21.15.28:0/862369708 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7fd2fc016920 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.322+0000 7fd2f3fff640 1 -- 172.21.15.28:0/862369708 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd2fc034050 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.322+0000 7fd31dd71740 10 client.19029 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.322+0000 7fd31dd71740 1 -- 172.21.15.28:0/862369708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bf8a71c7d0 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.322+0000 7fd2f3fff640 1 -- 172.21.15.28:0/862369708 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2fc01c250 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.322+0000 7fd2f3fff640 1 -- 172.21.15.28:0/862369708 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd2fc02d3a0 con 0x55bf8a9d9ea0 2024-04-25T01:42:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.322+0000 7fd31dd71740 10 client.19029 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.322+0000 7fd31dd71740 10 client.19029 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:42:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.322+0000 7fd31dd71740 20 client.19029 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.322+0000 7fd31dd71740 1 --2- 172.21.15.28:0/862369708 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55bf8aa5aa80 0x55bf8aa5cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.322+0000 7fd31dd71740 4 client.19029 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:42:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.322+0000 7fd31dd71740 1 -- 172.21.15.28:0/862369708 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bf8a8c3820 con 0x55bf8aa5aa80 2024-04-25T01:42:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.323+0000 7fd318d25640 1 --2- 172.21.15.28:0/862369708 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55bf8aa5aa80 0x55bf8aa5cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.323+0000 7fd318d25640 1 --2- 172.21.15.28:0/862369708 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55bf8aa5aa80 0x55bf8aa5cea0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c0a1f1be0d877101 server_cookie=f2839aef725722a0 in_seq=0 out_seq=0 2024-04-25T01:42:36.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.323+0000 7fd2f3fff640 10 client.19029 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:42:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.323+0000 7fd2f3fff640 1 -- 172.21.15.28:0/862369708 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bf8a8c3820 con 0x55bf8aa5aa80 2024-04-25T01:42:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.323+0000 7fd2f3fff640 10 client.19029 handle_command_reply: tid=0 2024-04-25T01:42:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.323+0000 7fd31dd71740 1 client.19029 shutdown 2024-04-25T01:42:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.323+0000 7fd31dd71740 1 -- 172.21.15.28:0/862369708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 msgr2=0x55bf8a9e7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.323+0000 7fd31dd71740 1 --2- 172.21.15.28:0/862369708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9e7b50 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc004810 tx=0x7fd2fc00bd00 comp rx=0 tx=0).stop 2024-04-25T01:42:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.323+0000 7fd31dd71740 1 -- 172.21.15.28:0/862369708 shutdown_connections 2024-04-25T01:42:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.323+0000 7fd31dd71740 1 --2- 172.21.15.28:0/862369708 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55bf8aa5aa80 0x55bf8aa5cea0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.323+0000 7fd31dd71740 1 --2- 172.21.15.28:0/862369708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55bf8a9e4570 0x55bf8a9e8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.323+0000 7fd2f3fff640 0 client.19029 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:42:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.323+0000 7fd31dd71740 1 --2- 172.21.15.28:0/862369708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf8a9d9ea0 0x55bf8a9e7b50 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.323+0000 7fd31dd71740 1 --2- 172.21.15.28:0/862369708 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55bf8a9d9580 0x55bf8a9ec7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.323+0000 7fd31dd71740 1 -- 172.21.15.28:0/862369708 >> 172.21.15.28:0/862369708 conn(0x55bf8a9d58a0 msgr2=0x55bf8a9dc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.324+0000 7fd31dd71740 1 -- 172.21.15.28:0/862369708 shutdown_connections 2024-04-25T01:42:36.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.324+0000 7fd31dd71740 1 -- 172.21.15.28:0/862369708 wait complete. 2024-04-25T01:42:36.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.324+0000 7fd31dd71740 20 client.19029 trim_cache size 0 max 16384 2024-04-25T01:42:36.319 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:42:36.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.324+0000 7fd31af89640 1 -- 172.21.15.28:0/2677349339 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd3000a2d00 msgr2=0x7fd3000a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.324+0000 7fd31af89640 1 --2- 172.21.15.28:0/2677349339 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd3000a2d00 0x7fd3000a51c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd30c008fd0 tx=0x7fd30c00b250 comp rx=0 tx=0).stop 2024-04-25T01:42:36.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.324+0000 7fd31af89640 1 -- 172.21.15.28:0/2677349339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314100990 msgr2=0x7fd314198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.324+0000 7fd31af89640 1 --2- 172.21.15.28:0/2677349339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314100990 0x7fd314198060 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fd304006010 tx=0x7fd304003ba0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.324+0000 7fd31af89640 1 -- 172.21.15.28:0/2677349339 shutdown_connections 2024-04-25T01:42:36.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.324+0000 7fd31af89640 1 --2- 172.21.15.28:0/2677349339 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fd3000a2d00 0x7fd3000a51c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.324+0000 7fd31af89640 1 --2- 172.21.15.28:0/2677349339 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fd314106690 0x7fd31419c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.324+0000 7fd31af89640 1 --2- 172.21.15.28:0/2677349339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd314101330 0x7fd3141985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.324+0000 7fd31af89640 1 --2- 172.21.15.28:0/2677349339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd314100990 0x7fd314198060 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.324+0000 7fd31af89640 1 -- 172.21.15.28:0/2677349339 >> 172.21.15.28:0/2677349339 conn(0x7fd314076440 msgr2=0x7fd3140ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.324+0000 7fd31af89640 1 -- 172.21.15.28:0/2677349339 shutdown_connections 2024-04-25T01:42:36.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.324+0000 7fd31af89640 1 -- 172.21.15.28:0/2677349339 wait complete. 2024-04-25T01:42:36.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:35 smithi028 ceph-mon[29390]: pgmap v296: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:42:36.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:42:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.439+0000 7fe6b29b7640 1 -- 172.21.15.28:0/2684194403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6ac1066b0 msgr2=0x7fe6ac106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.439+0000 7fe6b29b7640 1 --2- 172.21.15.28:0/2684194403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6ac1066b0 0x7fe6ac106a90 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fe69c009930 tx=0x7fe69c02a420 comp rx=0 tx=0).stop 2024-04-25T01:42:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.439+0000 7fe6b29b7640 1 -- 172.21.15.28:0/2684194403 shutdown_connections 2024-04-25T01:42:36.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.439+0000 7fe6b29b7640 1 --2- 172.21.15.28:0/2684194403 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6ac100fd0 0x7fe6ac10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.439+0000 7fe6b29b7640 1 --2- 172.21.15.28:0/2684194403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6ac100610 0x7fe6ac100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.439+0000 7fe6b29b7640 1 --2- 172.21.15.28:0/2684194403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6ac1066b0 0x7fe6ac106a90 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.439+0000 7fe6b29b7640 1 -- 172.21.15.28:0/2684194403 >> 172.21.15.28:0/2684194403 conn(0x7fe6ac077740 msgr2=0x7fe6ac077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.439+0000 7fe6b29b7640 1 -- 172.21.15.28:0/2684194403 shutdown_connections 2024-04-25T01:42:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.439+0000 7fe6b29b7640 1 -- 172.21.15.28:0/2684194403 wait complete. 2024-04-25T01:42:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.439+0000 7fe6b29b7640 1 Processor -- start 2024-04-25T01:42:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.439+0000 7fe6b29b7640 1 -- start start 2024-04-25T01:42:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.440+0000 7fe6b29b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6ac100610 0x7fe6ac197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.440+0000 7fe6b29b7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6ac100fd0 0x7fe6ac198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.440+0000 7fe6b29b7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6ac1066b0 0x7fe6ac19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.440+0000 7fe6b29b7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac111b30 con 0x7fe6ac100610 2024-04-25T01:42:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.440+0000 7fe6b29b7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac1119b0 con 0x7fe6ac1066b0 2024-04-25T01:42:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.440+0000 7fe6b29b7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac111cb0 con 0x7fe6ac100fd0 2024-04-25T01:42:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.440+0000 7fe6abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6ac100610 0x7fe6ac197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.440+0000 7fe6abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6ac100610 0x7fe6ac197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49834/0 (socket says 172.21.15.28:49834) 2024-04-25T01:42:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.440+0000 7fe6abfff640 1 -- 172.21.15.28:0/336868548 learned_addr learned my addr 172.21.15.28:0/336868548 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.440+0000 7fe6b0f54640 1 --2- 172.21.15.28:0/336868548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6ac1066b0 0x7fe6ac19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.440+0000 7fe6ab7fe640 1 --2- 172.21.15.28:0/336868548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6ac100fd0 0x7fe6ac198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.440+0000 7fe6abfff640 1 -- 172.21.15.28:0/336868548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6ac100fd0 msgr2=0x7fe6ac198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.440+0000 7fe6abfff640 1 --2- 172.21.15.28:0/336868548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6ac100fd0 0x7fe6ac198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.440+0000 7fe6abfff640 1 -- 172.21.15.28:0/336868548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6ac1066b0 msgr2=0x7fe6ac19c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.440+0000 7fe6abfff640 1 --2- 172.21.15.28:0/336868548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6ac1066b0 0x7fe6ac19c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.440+0000 7fe6abfff640 1 -- 172.21.15.28:0/336868548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6ac19cf70 con 0x7fe6ac100610 2024-04-25T01:42:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.440+0000 7fe6ab7fe640 1 --2- 172.21.15.28:0/336868548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6ac100fd0 0x7fe6ac198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.440+0000 7fe6abfff640 1 --2- 172.21.15.28:0/336868548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6ac100610 0x7fe6ac197fd0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fe69c005940 tx=0x7fe69c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.441+0000 7fe6a97fa640 1 -- 172.21.15.28:0/336868548 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c03f030 con 0x7fe6ac100610 2024-04-25T01:42:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.441+0000 7fe6a97fa640 1 -- 172.21.15.28:0/336868548 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe69c003ce0 con 0x7fe6ac100610 2024-04-25T01:42:36.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.441+0000 7fe6b29b7640 1 -- 172.21.15.28:0/336868548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6ac19d200 con 0x7fe6ac100610 2024-04-25T01:42:36.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.441+0000 7fe6a97fa640 1 -- 172.21.15.28:0/336868548 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c035070 con 0x7fe6ac100610 2024-04-25T01:42:36.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.441+0000 7fe6b29b7640 1 -- 172.21.15.28:0/336868548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6ac1a4af0 con 0x7fe6ac100610 2024-04-25T01:42:36.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.443+0000 7fe6a97fa640 1 -- 172.21.15.28:0/336868548 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe69c042030 con 0x7fe6ac100610 2024-04-25T01:42:36.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.443+0000 7fe6b57a2740 1 Processor -- start 2024-04-25T01:42:36.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.443+0000 7fe6b57a2740 1 -- start start 2024-04-25T01:42:36.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.443+0000 7fe6b57a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b62480 0x560957b62860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.443+0000 7fe6b57a2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b62da0 0x560957b71440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.443+0000 7fe6b57a2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b71980 0x560957b73d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.443+0000 7fe6b57a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560957ac7670 con 0x560957b62480 2024-04-25T01:42:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.443+0000 7fe6b57a2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560957a6f970 con 0x560957b62da0 2024-04-25T01:42:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.443+0000 7fe6b57a2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x560957ac3630 con 0x560957b71980 2024-04-25T01:42:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.443+0000 7fe6a97fa640 1 --2- 172.21.15.28:0/336868548 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe6940a2d80 0x7fe6940a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.443+0000 7fe6b0f54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b62480 0x560957b62860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.444+0000 7fe6abfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b62da0 0x560957b71440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.444+0000 7fe6b0f54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b62480 0x560957b62860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49850/0 (socket says 172.21.15.28:49850) 2024-04-25T01:42:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.444+0000 7fe6b0f54640 1 -- 172.21.15.28:0/2003517420 learned_addr learned my addr 172.21.15.28:0/2003517420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.444+0000 7fe6ab7fe640 1 --2- 172.21.15.28:0/2003517420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b71980 0x560957b73d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.444+0000 7fe6b0f54640 1 --2- 172.21.15.28:0/336868548 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe6940a2d80 0x7fe6940a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.444+0000 7fe6a97fa640 1 -- 172.21.15.28:0/336868548 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7fe69c0efcb0 con 0x7fe6ac100610 2024-04-25T01:42:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.444+0000 7fe6ab7fe640 1 -- 172.21.15.28:0/2003517420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b62da0 msgr2=0x560957b71440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.444+0000 7fe6ab7fe640 1 --2- 172.21.15.28:0/2003517420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b62da0 0x560957b71440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.444+0000 7fe6ab7fe640 1 -- 172.21.15.28:0/2003517420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b62480 msgr2=0x560957b62860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.444+0000 7fe6ab7fe640 1 --2- 172.21.15.28:0/2003517420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b62480 0x560957b62860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.444+0000 7fe6ab7fe640 1 -- 172.21.15.28:0/2003517420 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560957a93400 con 0x560957b71980 2024-04-25T01:42:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.444+0000 7fe6b0f54640 1 --2- 172.21.15.28:0/2003517420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b62480 0x560957b62860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:42:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.444+0000 7fe6b0f54640 1 --2- 172.21.15.28:0/336868548 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe6940a2d80 0x7fe6940a5240 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4005c20 tx=0x7fe6a4009600 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.444+0000 7fe6ab7fe640 1 --2- 172.21.15.28:0/2003517420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b71980 0x560957b73d70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000bc80 tx=0x7fe6a000f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cc7c679cc519581 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.444+0000 7fe6927fc640 1 -- 172.21.15.28:0/2003517420 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0015070 con 0x560957b71980 2024-04-25T01:42:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.444+0000 7fe6927fc640 1 -- 172.21.15.28:0/2003517420 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6a0011920 con 0x560957b71980 2024-04-25T01:42:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.444+0000 7fe6927fc640 1 -- 172.21.15.28:0/2003517420 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0010890 con 0x560957b71980 2024-04-25T01:42:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.445+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2003517420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b71980 msgr2=0x560957b73d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.445+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/2003517420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b71980 0x560957b73d70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000bc80 tx=0x7fe6a000f730 comp rx=0 tx=0).stop 2024-04-25T01:42:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.445+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2003517420 shutdown_connections 2024-04-25T01:42:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.445+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/2003517420 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b71980 0x560957b73d70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.445+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/2003517420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b62da0 0x560957b71440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.445+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/2003517420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b62480 0x560957b62860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.445+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2003517420 >> 172.21.15.28:0/2003517420 conn(0x560957b5e8a0 msgr2=0x560957b5ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.445+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2003517420 shutdown_connections 2024-04-25T01:42:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.445+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2003517420 wait complete. 2024-04-25T01:42:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.445+0000 7fe6b57a2740 1 -- start start 2024-04-25T01:42:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.446+0000 7fe6b57a2740 1 Processor -- start 2024-04-25T01:42:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.446+0000 7fe6b57a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b62480 0x560957b71260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.446+0000 7fe6b57a2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b62da0 0x560957b6e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.446+0000 7fe6b57a2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b71980 0x560957b6ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.446+0000 7fe6b57a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560957b74740 con 0x560957b62480 2024-04-25T01:42:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.446+0000 7fe6b57a2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560957a83050 con 0x560957b71980 2024-04-25T01:42:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.446+0000 7fe6b57a2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x560957b748c0 con 0x560957b62da0 2024-04-25T01:42:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.446+0000 7fe6b57a2740 10 client.0 _ll_register_callbacks cb 0x5609577dabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.446+0000 7fe6abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b62480 0x560957b71260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.446+0000 7fe6abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b62480 0x560957b71260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49856/0 (socket says 172.21.15.28:49856) 2024-04-25T01:42:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.446+0000 7fe6abfff640 1 -- 172.21.15.28:0/2472292836 learned_addr learned my addr 172.21.15.28:0/2472292836 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.446+0000 7fe6b0f54640 1 --2- 172.21.15.28:0/2472292836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b71980 0x560957b6ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.446+0000 7fe6ab7fe640 1 --2- 172.21.15.28:0/2472292836 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b62da0 0x560957b6e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.446+0000 7fe6abfff640 1 -- 172.21.15.28:0/2472292836 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b62da0 msgr2=0x560957b6e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.446+0000 7fe6abfff640 1 --2- 172.21.15.28:0/2472292836 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b62da0 0x560957b6e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.446+0000 7fe6abfff640 1 -- 172.21.15.28:0/2472292836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b71980 msgr2=0x560957b6ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.446+0000 7fe6abfff640 1 --2- 172.21.15.28:0/2472292836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b71980 0x560957b6ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.446+0000 7fe6abfff640 1 -- 172.21.15.28:0/2472292836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560957a54dc0 con 0x560957b62480 2024-04-25T01:42:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.446+0000 7fe6ab7fe640 1 --2- 172.21.15.28:0/2472292836 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b62da0 0x560957b6e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.447+0000 7fe6abfff640 1 --2- 172.21.15.28:0/2472292836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b62480 0x560957b71260 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fe69c0314b0 tx=0x7fe69c08ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8832f7d97393d29 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.447+0000 7fe68b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.447+0000 7fe68b7fe640 1 -- 172.21.15.28:0/2472292836 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c0be030 con 0x560957b62480 2024-04-25T01:42:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.447+0000 7fe68b7fe640 1 -- 172.21.15.28:0/2472292836 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7fe69c0ed2f0 con 0x560957b62480 2024-04-25T01:42:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.447+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2472292836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6a0011920 con 0x560957b62480 2024-04-25T01:42:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.447+0000 7fe68b7fe640 1 -- 172.21.15.28:0/2472292836 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe69c049050 con 0x560957b62480 2024-04-25T01:42:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.447+0000 7fe6b57a2740 10 client.19041 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.447+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2472292836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5609577db3f0 con 0x560957b62480 2024-04-25T01:42:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.447+0000 7fe68b7fe640 1 -- 172.21.15.28:0/2472292836 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6a0011920 con 0x560957b62480 2024-04-25T01:42:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.447+0000 7fe68b7fe640 1 -- 172.21.15.28:0/2472292836 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe69c03bc80 con 0x560957b62480 2024-04-25T01:42:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.447+0000 7fe6b57a2740 10 client.19041 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.447+0000 7fe6b57a2740 10 client.19041 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:42:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.447+0000 7fe6b57a2740 20 client.19041 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.447+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/2472292836 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x560957b81830 0x560957b83ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.447+0000 7fe6b57a2740 4 client.19041 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.447+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2472292836 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560957ad4bb0 con 0x560957b81830 2024-04-25T01:42:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.448+0000 7fe6ab7fe640 1 --2- 172.21.15.28:0/2472292836 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x560957b81830 0x560957b83ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.448+0000 7fe6ab7fe640 1 --2- 172.21.15.28:0/2472292836 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x560957b81830 0x560957b83ca0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bea4e3847c19628b server_cookie=eec64fe9803b5d0e in_seq=0 out_seq=0 2024-04-25T01:42:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.448+0000 7fe68b7fe640 10 client.19041 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:42:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.449+0000 7fe68b7fe640 1 -- 172.21.15.28:0/2472292836 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560957ad4bb0 con 0x560957b81830 2024-04-25T01:42:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.449+0000 7fe68b7fe640 10 client.19041 handle_command_reply: tid=0 2024-04-25T01:42:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.449+0000 7fe6b57a2740 1 client.19041 shutdown 2024-04-25T01:42:36.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.449+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2472292836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b62480 msgr2=0x560957b71260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.449+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/2472292836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b62480 0x560957b71260 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fe69c0314b0 tx=0x7fe69c08ad70 comp rx=0 tx=0).stop 2024-04-25T01:42:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.449+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2472292836 shutdown_connections 2024-04-25T01:42:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.449+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/2472292836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b71980 0x560957b6ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.449+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/2472292836 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b62da0 0x560957b6e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.449+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/2472292836 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x560957b81830 0x560957b83ca0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.449+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/2472292836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b62480 0x560957b71260 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.449+0000 7fe68b7fe640 0 client.19041 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:42:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.449+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2472292836 >> 172.21.15.28:0/2472292836 conn(0x560957b5e8a0 msgr2=0x560957b64710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.449+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2472292836 shutdown_connections 2024-04-25T01:42:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.449+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2472292836 wait complete. 2024-04-25T01:42:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.449+0000 7fe6b57a2740 20 client.19041 trim_cache size 0 max 16384 2024-04-25T01:42:36.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.462+0000 7fe6b57a2740 1 Processor -- start 2024-04-25T01:42:36.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.462+0000 7fe6b57a2740 1 -- start start 2024-04-25T01:42:36.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe6b57a2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b62480 0x560957b7d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe6b57a2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b62da0 0x560957b69af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe6b57a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b71980 0x560957b70d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe6b57a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac1ac940 con 0x560957b71980 2024-04-25T01:42:36.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe6b57a2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac1ae480 con 0x560957b62da0 2024-04-25T01:42:36.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe6b57a2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac1ad010 con 0x560957b62480 2024-04-25T01:42:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe6b0f54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b71980 0x560957b70d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe6b0f54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b71980 0x560957b70d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49870/0 (socket says 172.21.15.28:49870) 2024-04-25T01:42:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe6b0f54640 1 -- 172.21.15.28:0/3520663037 learned_addr learned my addr 172.21.15.28:0/3520663037 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe6ab7fe640 1 --2- 172.21.15.28:0/3520663037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b62da0 0x560957b69af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe6abfff640 1 --2- 172.21.15.28:0/3520663037 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b62480 0x560957b7d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe6b0f54640 1 -- 172.21.15.28:0/3520663037 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b62480 msgr2=0x560957b7d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe6b0f54640 1 --2- 172.21.15.28:0/3520663037 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b62480 0x560957b7d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe6b0f54640 1 -- 172.21.15.28:0/3520663037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b62da0 msgr2=0x560957b69af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe6b0f54640 1 --2- 172.21.15.28:0/3520663037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b62da0 0x560957b69af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe6b0f54640 1 -- 172.21.15.28:0/3520663037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560957b62bd0 con 0x560957b71980 2024-04-25T01:42:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe6b0f54640 1 --2- 172.21.15.28:0/3520663037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b71980 0x560957b70d90 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4009e70 tx=0x7fe6a4009a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d5d35164f31951 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe691ffb640 1 -- 172.21.15.28:0/3520663037 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4012df0 con 0x560957b71980 2024-04-25T01:42:36.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe691ffb640 1 -- 172.21.15.28:0/3520663037 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6a4020990 con 0x560957b71980 2024-04-25T01:42:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.463+0000 7fe691ffb640 1 -- 172.21.15.28:0/3520663037 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4005680 con 0x560957b71980 2024-04-25T01:42:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.464+0000 7fe6b57a2740 1 -- 172.21.15.28:0/3520663037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b71980 msgr2=0x560957b70d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.464+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/3520663037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b71980 0x560957b70d90 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4009e70 tx=0x7fe6a4009a80 comp rx=0 tx=0).stop 2024-04-25T01:42:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.464+0000 7fe6b57a2740 1 -- 172.21.15.28:0/3520663037 shutdown_connections 2024-04-25T01:42:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.464+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/3520663037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b71980 0x560957b70d90 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.464+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/3520663037 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b62da0 0x560957b69af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.464+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/3520663037 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b62480 0x560957b7d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.464+0000 7fe6b57a2740 1 -- 172.21.15.28:0/3520663037 >> 172.21.15.28:0/3520663037 conn(0x560957b5e8a0 msgr2=0x560957b693e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.464+0000 7fe6b57a2740 1 -- 172.21.15.28:0/3520663037 shutdown_connections 2024-04-25T01:42:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.464+0000 7fe6b57a2740 1 -- 172.21.15.28:0/3520663037 wait complete. 2024-04-25T01:42:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.464+0000 7fe6b57a2740 1 -- start start 2024-04-25T01:42:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.465+0000 7fe6b57a2740 1 Processor -- start 2024-04-25T01:42:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.465+0000 7fe6b57a2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b62480 0x560957b7a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.465+0000 7fe6b57a2740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b62da0 0x560957b6b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.465+0000 7fe6b57a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b71980 0x560957b6b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.465+0000 7fe6b57a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac1addb0 con 0x560957b71980 2024-04-25T01:42:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.465+0000 7fe6b57a2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac1ad6e0 con 0x560957b62480 2024-04-25T01:42:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.465+0000 7fe6b57a2740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac1aa940 con 0x560957b62da0 2024-04-25T01:42:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.465+0000 7fe6b57a2740 10 client.0 _ll_register_callbacks cb 0x7fe69c009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.465+0000 7fe6b0f54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b71980 0x560957b6b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.465+0000 7fe6b0f54640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b71980 0x560957b6b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49872/0 (socket says 172.21.15.28:49872) 2024-04-25T01:42:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.465+0000 7fe6b0f54640 1 -- 172.21.15.28:0/2315952365 learned_addr learned my addr 172.21.15.28:0/2315952365 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.465+0000 7fe6abfff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b62480 0x560957b7a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.465+0000 7fe6ab7fe640 1 --2- 172.21.15.28:0/2315952365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b62da0 0x560957b6b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.465+0000 7fe6b0f54640 1 -- 172.21.15.28:0/2315952365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b62da0 msgr2=0x560957b6b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.465+0000 7fe6b0f54640 1 --2- 172.21.15.28:0/2315952365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b62da0 0x560957b6b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.465+0000 7fe6b0f54640 1 -- 172.21.15.28:0/2315952365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b62480 msgr2=0x560957b7a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.465+0000 7fe6b0f54640 1 --2- 172.21.15.28:0/2315952365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b62480 0x560957b7a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.465+0000 7fe6b0f54640 1 -- 172.21.15.28:0/2315952365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609578a57d0 con 0x560957b71980 2024-04-25T01:42:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.466+0000 7fe6b0f54640 1 --2- 172.21.15.28:0/2315952365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b71980 0x560957b6b640 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4005360 tx=0x7fe6a4020d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5279c58ca74b5b4f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.466+0000 7fe68b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.466+0000 7fe68b7fe640 1 -- 172.21.15.28:0/2315952365 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4004c50 con 0x560957b71980 2024-04-25T01:42:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.466+0000 7fe68b7fe640 1 -- 172.21.15.28:0/2315952365 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7fe6a402f070 con 0x560957b71980 2024-04-25T01:42:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.466+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2315952365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6a4020990 con 0x560957b71980 2024-04-25T01:42:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.466+0000 7fe68b7fe640 1 -- 172.21.15.28:0/2315952365 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe6a4004df0 con 0x560957b71980 2024-04-25T01:42:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.466+0000 7fe6b57a2740 10 client.19047 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.466+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2315952365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560957be7d20 con 0x560957b71980 2024-04-25T01:42:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.466+0000 7fe68b7fe640 1 -- 172.21.15.28:0/2315952365 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6a4020990 con 0x560957b71980 2024-04-25T01:42:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.466+0000 7fe68b7fe640 1 -- 172.21.15.28:0/2315952365 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe6a400f040 con 0x560957b71980 2024-04-25T01:42:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.466+0000 7fe6b57a2740 10 client.19047 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.466+0000 7fe6b57a2740 10 client.19047 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:42:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.466+0000 7fe6b57a2740 20 client.19047 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.466+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/2315952365 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x560957be83d0 0x560957bea7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.466+0000 7fe6b57a2740 4 client.19047 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:42:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.466+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2315952365 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560957b6ee20 con 0x560957be83d0 2024-04-25T01:42:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.466+0000 7fe6abfff640 1 --2- 172.21.15.28:0/2315952365 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x560957be83d0 0x560957bea7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.467+0000 7fe6abfff640 1 --2- 172.21.15.28:0/2315952365 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x560957be83d0 0x560957bea7f0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=be6fe00dcc00be94 server_cookie=5c12b37e2f43031c in_seq=0 out_seq=0 2024-04-25T01:42:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.467+0000 7fe68b7fe640 10 client.19047 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:42:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.467+0000 7fe68b7fe640 1 -- 172.21.15.28:0/2315952365 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560957b6ee20 con 0x560957be83d0 2024-04-25T01:42:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.467+0000 7fe68b7fe640 10 client.19047 handle_command_reply: tid=0 2024-04-25T01:42:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.467+0000 7fe6b57a2740 1 client.19047 shutdown 2024-04-25T01:42:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.467+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2315952365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b71980 msgr2=0x560957b6b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.467+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/2315952365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b71980 0x560957b6b640 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4005360 tx=0x7fe6a4020d70 comp rx=0 tx=0).stop 2024-04-25T01:42:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2315952365 shutdown_connections 2024-04-25T01:42:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/2315952365 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x560957be83d0 0x560957bea7f0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/2315952365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560957b71980 0x560957b6b640 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/2315952365 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x560957b62da0 0x560957b6b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe68b7fe640 0 client.19047 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:42:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b57a2740 1 --2- 172.21.15.28:0/2315952365 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560957b62480 0x560957b7a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2315952365 >> 172.21.15.28:0/2315952365 conn(0x560957b5e8a0 msgr2=0x560957b63e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2315952365 shutdown_connections 2024-04-25T01:42:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b57a2740 1 -- 172.21.15.28:0/2315952365 wait complete. 2024-04-25T01:42:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b57a2740 20 client.19047 trim_cache size 0 max 16384 2024-04-25T01:42:36.462 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:42:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b29b7640 1 -- 172.21.15.28:0/336868548 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe6940a2d80 msgr2=0x7fe6940a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b29b7640 1 --2- 172.21.15.28:0/336868548 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe6940a2d80 0x7fe6940a5240 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4005c20 tx=0x7fe6a4009600 comp rx=0 tx=0).stop 2024-04-25T01:42:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b29b7640 1 -- 172.21.15.28:0/336868548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6ac100610 msgr2=0x7fe6ac197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b29b7640 1 --2- 172.21.15.28:0/336868548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6ac100610 0x7fe6ac197fd0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fe69c005940 tx=0x7fe69c0042f0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b29b7640 1 -- 172.21.15.28:0/336868548 shutdown_connections 2024-04-25T01:42:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b29b7640 1 --2- 172.21.15.28:0/336868548 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe6940a2d80 0x7fe6940a5240 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b29b7640 1 --2- 172.21.15.28:0/336868548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6ac1066b0 0x7fe6ac19c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b29b7640 1 --2- 172.21.15.28:0/336868548 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe6ac100fd0 0x7fe6ac198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b29b7640 1 --2- 172.21.15.28:0/336868548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6ac100610 0x7fe6ac197fd0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b29b7640 1 -- 172.21.15.28:0/336868548 >> 172.21.15.28:0/336868548 conn(0x7fe6ac077740 msgr2=0x7fe6ac0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b29b7640 1 -- 172.21.15.28:0/336868548 shutdown_connections 2024-04-25T01:42:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.468+0000 7fe6b29b7640 1 -- 172.21.15.28:0/336868548 wait complete. 2024-04-25T01:42:36.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:42:36.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:35 smithi130 ceph-mon[32702]: pgmap v296: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:42:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.582+0000 7ff11f9a5640 1 -- 172.21.15.28:0/4247283182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff118100600 msgr2=0x7ff11810e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.582+0000 7ff11f9a5640 1 --2- 172.21.15.28:0/4247283182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff118100600 0x7ff11810e5e0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7ff110009cd0 tx=0x7ff11002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.583+0000 7ff11f9a5640 1 -- 172.21.15.28:0/4247283182 shutdown_connections 2024-04-25T01:42:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.583+0000 7ff11f9a5640 1 --2- 172.21.15.28:0/4247283182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff118100600 0x7ff11810e5e0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.583+0000 7ff11f9a5640 1 --2- 172.21.15.28:0/4247283182 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1180ffc40 0x7ff1181000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.583+0000 7ff11f9a5640 1 --2- 172.21.15.28:0/4247283182 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff118107720 0x7ff118107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.583+0000 7ff11f9a5640 1 -- 172.21.15.28:0/4247283182 >> 172.21.15.28:0/4247283182 conn(0x7ff118076c00 msgr2=0x7ff1180fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.583+0000 7ff11f9a5640 1 -- 172.21.15.28:0/4247283182 shutdown_connections 2024-04-25T01:42:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.583+0000 7ff11f9a5640 1 -- 172.21.15.28:0/4247283182 wait complete. 2024-04-25T01:42:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.583+0000 7ff11f9a5640 1 Processor -- start 2024-04-25T01:42:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.583+0000 7ff11f9a5640 1 -- start start 2024-04-25T01:42:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.583+0000 7ff11f9a5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1180ffc40 0x7ff11819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.583+0000 7ff11f9a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff118100600 0x7ff11819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.583+0000 7ff11f9a5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff118107720 0x7ff11819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.583+0000 7ff11f9a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff118111b30 con 0x7ff118100600 2024-04-25T01:42:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.583+0000 7ff11f9a5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff1181119b0 con 0x7ff1180ffc40 2024-04-25T01:42:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.583+0000 7ff11f9a5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff118111cb0 con 0x7ff118107720 2024-04-25T01:42:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.584+0000 7ff11cf40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff118100600 0x7ff11819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.584+0000 7ff11cf40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff118100600 0x7ff11819a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49882/0 (socket says 172.21.15.28:49882) 2024-04-25T01:42:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.584+0000 7ff11cf40640 1 -- 172.21.15.28:0/2360021817 learned_addr learned my addr 172.21.15.28:0/2360021817 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.584+0000 7ff11d741640 1 --2- 172.21.15.28:0/2360021817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1180ffc40 0x7ff11819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.584+0000 7ff11df42640 1 --2- 172.21.15.28:0/2360021817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff118107720 0x7ff11819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.584+0000 7ff11cf40640 1 -- 172.21.15.28:0/2360021817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff118107720 msgr2=0x7ff11819eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.584+0000 7ff11cf40640 1 --2- 172.21.15.28:0/2360021817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff118107720 0x7ff11819eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.584+0000 7ff11cf40640 1 -- 172.21.15.28:0/2360021817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1180ffc40 msgr2=0x7ff11819a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.584+0000 7ff11cf40640 1 --2- 172.21.15.28:0/2360021817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1180ffc40 0x7ff11819a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.584+0000 7ff11cf40640 1 -- 172.21.15.28:0/2360021817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11819f1f0 con 0x7ff118100600 2024-04-25T01:42:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.584+0000 7ff11cf40640 1 --2- 172.21.15.28:0/2360021817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff118100600 0x7ff11819a740 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7ff10800d950 tx=0x7ff10800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.584+0000 7ff10e7fc640 1 -- 172.21.15.28:0/2360021817 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff108017070 con 0x7ff118100600 2024-04-25T01:42:36.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.584+0000 7ff10e7fc640 1 -- 172.21.15.28:0/2360021817 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff108009c30 con 0x7ff118100600 2024-04-25T01:42:36.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.584+0000 7ff11f9a5640 1 -- 172.21.15.28:0/2360021817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff11819f4e0 con 0x7ff118100600 2024-04-25T01:42:36.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.584+0000 7ff10e7fc640 1 -- 172.21.15.28:0/2360021817 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10800f6b0 con 0x7ff118100600 2024-04-25T01:42:36.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.584+0000 7ff11f9a5640 1 -- 172.21.15.28:0/2360021817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1181a6d70 con 0x7ff118100600 2024-04-25T01:42:36.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.586+0000 7ff10e7fc640 1 -- 172.21.15.28:0/2360021817 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff1080109d0 con 0x7ff118100600 2024-04-25T01:42:36.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.587+0000 7ff12278d740 1 Processor -- start 2024-04-25T01:42:36.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.587+0000 7ff12278d740 1 -- start start 2024-04-25T01:42:36.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.587+0000 7ff12278d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7580 0x562560ed7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.587+0000 7ff12278d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7ea0 0x562560ee2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.587+0000 7ff12278d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560ee4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.587+0000 7ff12278d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562560e3c670 con 0x562560ee2570 2024-04-25T01:42:36.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.587+0000 7ff12278d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x562560de4970 con 0x562560ed7580 2024-04-25T01:42:36.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.587+0000 7ff12278d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x562560e38630 con 0x562560ed7ea0 2024-04-25T01:42:36.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.587+0000 7ff11cf40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560ee4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.587+0000 7ff10e7fc640 1 --2- 172.21.15.28:0/2360021817 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff1040a2d80 0x7ff1040a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.587+0000 7ff11df42640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7580 0x562560ed7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.587+0000 7ff11cf40640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560ee4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49890/0 (socket says 172.21.15.28:49890) 2024-04-25T01:42:36.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.587+0000 7ff11cf40640 1 -- 172.21.15.28:0/3702703332 learned_addr learned my addr 172.21.15.28:0/3702703332 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.587+0000 7ff11d741640 1 --2- 172.21.15.28:0/3702703332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7ea0 0x562560ee2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.588+0000 7ff11df42640 1 --2- 172.21.15.28:0/2360021817 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff1040a2d80 0x7ff1040a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.588+0000 7ff10e7fc640 1 -- 172.21.15.28:0/2360021817 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7ff1080cf3c0 con 0x7ff118100600 2024-04-25T01:42:36.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.588+0000 7ff11df42640 1 -- 172.21.15.28:0/3702703332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7ea0 msgr2=0x562560ee2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.588+0000 7ff11df42640 1 --2- 172.21.15.28:0/3702703332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7ea0 0x562560ee2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.588+0000 7ff11df42640 1 -- 172.21.15.28:0/3702703332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 msgr2=0x562560ee4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.588+0000 7ff11df42640 1 --2- 172.21.15.28:0/3702703332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560ee4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.588+0000 7ff11df42640 1 -- 172.21.15.28:0/3702703332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562560e08400 con 0x562560ed7580 2024-04-25T01:42:36.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.588+0000 7ff11cf40640 1 --2- 172.21.15.28:0/3702703332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560ee4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:42:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.588+0000 7ff11df42640 1 --2- 172.21.15.28:0/3702703332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7580 0x562560ed7960 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff110005e30 tx=0x7ff110031820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0a51d0aed9a7d19 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.588+0000 7ff11df42640 1 --2- 172.21.15.28:0/2360021817 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff1040a2d80 0x7ff1040a5240 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff110045000 tx=0x7ff110048000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.588+0000 7ff1037fe640 1 -- 172.21.15.28:0/3702703332 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff110004410 con 0x562560ed7580 2024-04-25T01:42:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.588+0000 7ff1037fe640 1 -- 172.21.15.28:0/3702703332 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff110036040 con 0x562560ed7580 2024-04-25T01:42:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.588+0000 7ff1037fe640 1 -- 172.21.15.28:0/3702703332 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff110030e40 con 0x562560ed7580 2024-04-25T01:42:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.589+0000 7ff12278d740 1 -- 172.21.15.28:0/3702703332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7580 msgr2=0x562560ed7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.589+0000 7ff12278d740 1 --2- 172.21.15.28:0/3702703332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7580 0x562560ed7960 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff110005e30 tx=0x7ff110031820 comp rx=0 tx=0).stop 2024-04-25T01:42:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.589+0000 7ff12278d740 1 -- 172.21.15.28:0/3702703332 shutdown_connections 2024-04-25T01:42:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.589+0000 7ff12278d740 1 --2- 172.21.15.28:0/3702703332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560ee4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.589+0000 7ff12278d740 1 --2- 172.21.15.28:0/3702703332 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7ea0 0x562560ee2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.589+0000 7ff12278d740 1 --2- 172.21.15.28:0/3702703332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7580 0x562560ed7960 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.589+0000 7ff12278d740 1 -- 172.21.15.28:0/3702703332 >> 172.21.15.28:0/3702703332 conn(0x562560ed38a0 msgr2=0x562560ed3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.589+0000 7ff12278d740 1 -- 172.21.15.28:0/3702703332 shutdown_connections 2024-04-25T01:42:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.589+0000 7ff12278d740 1 -- 172.21.15.28:0/3702703332 wait complete. 2024-04-25T01:42:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.589+0000 7ff12278d740 1 -- start start 2024-04-25T01:42:36.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.589+0000 7ff12278d740 1 Processor -- start 2024-04-25T01:42:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff12278d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7580 0x562560ee1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff12278d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7ea0 0x562560edf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff12278d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560edf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff12278d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562560ee5300 con 0x562560ee2570 2024-04-25T01:42:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff12278d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x562560df8050 con 0x562560ed7ea0 2024-04-25T01:42:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff12278d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x562560ee5480 con 0x562560ed7580 2024-04-25T01:42:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff12278d740 10 client.0 _ll_register_callbacks cb 0x562560b4fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff11df42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560edf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff11df42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560edf7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49898/0 (socket says 172.21.15.28:49898) 2024-04-25T01:42:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff11df42640 1 -- 172.21.15.28:0/1888166050 learned_addr learned my addr 172.21.15.28:0/1888166050 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff11cf40640 1 --2- 172.21.15.28:0/1888166050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7ea0 0x562560edf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff11df42640 1 -- 172.21.15.28:0/1888166050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7580 msgr2=0x562560ee1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff11d741640 1 --2- 172.21.15.28:0/1888166050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7580 0x562560ee1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff11df42640 1 --2- 172.21.15.28:0/1888166050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7580 0x562560ee1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff11df42640 1 -- 172.21.15.28:0/1888166050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7ea0 msgr2=0x562560edf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff11df42640 1 --2- 172.21.15.28:0/1888166050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7ea0 0x562560edf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff11df42640 1 -- 172.21.15.28:0/1888166050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562560dc9dc0 con 0x562560ee2570 2024-04-25T01:42:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff11cf40640 1 --2- 172.21.15.28:0/1888166050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7ea0 0x562560edf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:42:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff11df42640 1 --2- 172.21.15.28:0/1888166050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560edf7b0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7ff110005c90 tx=0x7ff110030e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb4e7a8bcd67ddca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.590+0000 7ff100ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.591+0000 7ff100ff9640 1 -- 172.21.15.28:0/1888166050 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff110056020 con 0x562560ee2570 2024-04-25T01:42:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.591+0000 7ff12278d740 1 -- 172.21.15.28:0/1888166050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff110036040 con 0x562560ee2570 2024-04-25T01:42:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.591+0000 7ff100ff9640 1 -- 172.21.15.28:0/1888166050 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7ff1100307b0 con 0x562560ee2570 2024-04-25T01:42:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.591+0000 7ff100ff9640 1 -- 172.21.15.28:0/1888166050 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff110061050 con 0x562560ee2570 2024-04-25T01:42:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.591+0000 7ff12278d740 10 client.19059 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.591+0000 7ff12278d740 1 -- 172.21.15.28:0/1888166050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562560b503f0 con 0x562560ee2570 2024-04-25T01:42:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.591+0000 7ff100ff9640 1 -- 172.21.15.28:0/1888166050 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff110036040 con 0x562560ee2570 2024-04-25T01:42:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.591+0000 7ff100ff9640 1 -- 172.21.15.28:0/1888166050 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff110030070 con 0x562560ee2570 2024-04-25T01:42:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.591+0000 7ff12278d740 10 client.19059 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.591+0000 7ff12278d740 10 client.19059 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:42:36.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.591+0000 7ff12278d740 20 client.19059 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.591+0000 7ff12278d740 1 --2- 172.21.15.28:0/1888166050 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x562560ef2460 0x562560ef4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.591+0000 7ff12278d740 4 client.19059 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.591+0000 7ff12278d740 1 -- 172.21.15.28:0/1888166050 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562560e49bb0 con 0x562560ef2460 2024-04-25T01:42:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.592+0000 7ff11d741640 1 --2- 172.21.15.28:0/1888166050 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x562560ef2460 0x562560ef4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.592+0000 7ff11d741640 1 --2- 172.21.15.28:0/1888166050 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x562560ef2460 0x562560ef4850 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2a424c682443fcb5 server_cookie=2968a93686d87b07 in_seq=0 out_seq=0 2024-04-25T01:42:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.592+0000 7ff100ff9640 10 client.19059 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:42:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.593+0000 7ff100ff9640 1 -- 172.21.15.28:0/1888166050 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562560e49bb0 con 0x562560ef2460 2024-04-25T01:42:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.593+0000 7ff100ff9640 10 client.19059 handle_command_reply: tid=0 2024-04-25T01:42:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.593+0000 7ff12278d740 1 client.19059 shutdown 2024-04-25T01:42:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.593+0000 7ff12278d740 1 -- 172.21.15.28:0/1888166050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 msgr2=0x562560edf7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.593+0000 7ff12278d740 1 --2- 172.21.15.28:0/1888166050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560edf7b0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7ff110005c90 tx=0x7ff110030e30 comp rx=0 tx=0).stop 2024-04-25T01:42:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.593+0000 7ff12278d740 1 -- 172.21.15.28:0/1888166050 shutdown_connections 2024-04-25T01:42:36.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.593+0000 7ff12278d740 1 --2- 172.21.15.28:0/1888166050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560edf7b0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.593+0000 7ff12278d740 1 --2- 172.21.15.28:0/1888166050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7ea0 0x562560edf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.593+0000 7ff12278d740 1 --2- 172.21.15.28:0/1888166050 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x562560ef2460 0x562560ef4850 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.593+0000 7ff12278d740 1 --2- 172.21.15.28:0/1888166050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7580 0x562560ee1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.593+0000 7ff100ff9640 0 client.19059 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:42:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.593+0000 7ff12278d740 1 -- 172.21.15.28:0/1888166050 >> 172.21.15.28:0/1888166050 conn(0x562560ed38a0 msgr2=0x562560ed96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.593+0000 7ff12278d740 1 -- 172.21.15.28:0/1888166050 shutdown_connections 2024-04-25T01:42:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.593+0000 7ff12278d740 1 -- 172.21.15.28:0/1888166050 wait complete. 2024-04-25T01:42:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.593+0000 7ff12278d740 20 client.19059 trim_cache size 0 max 16384 2024-04-25T01:42:36.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.606+0000 7ff12278d740 1 Processor -- start 2024-04-25T01:42:36.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.606+0000 7ff12278d740 1 -- start start 2024-04-25T01:42:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.606+0000 7ff12278d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7580 0x562560ee1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.606+0000 7ff12278d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7ea0 0x562560ef9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.606+0000 7ff12278d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560efa330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.606+0000 7ff12278d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1181b0dc0 con 0x562560ee2570 2024-04-25T01:42:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.606+0000 7ff12278d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff1181b2900 con 0x562560ed7ea0 2024-04-25T01:42:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.606+0000 7ff12278d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff1181b1490 con 0x562560ed7580 2024-04-25T01:42:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.607+0000 7ff11df42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560efa330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.607+0000 7ff11df42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560efa330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49914/0 (socket says 172.21.15.28:49914) 2024-04-25T01:42:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.607+0000 7ff11df42640 1 -- 172.21.15.28:0/802102693 learned_addr learned my addr 172.21.15.28:0/802102693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.607+0000 7ff11cf40640 1 --2- 172.21.15.28:0/802102693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7ea0 0x562560ef9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.607+0000 7ff11d741640 1 --2- 172.21.15.28:0/802102693 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7580 0x562560ee1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.607+0000 7ff11df42640 1 -- 172.21.15.28:0/802102693 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7580 msgr2=0x562560ee1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.607+0000 7ff11df42640 1 --2- 172.21.15.28:0/802102693 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7580 0x562560ee1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.607+0000 7ff11df42640 1 -- 172.21.15.28:0/802102693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7ea0 msgr2=0x562560ef9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.607+0000 7ff11df42640 1 --2- 172.21.15.28:0/802102693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7ea0 0x562560ef9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.607+0000 7ff11df42640 1 -- 172.21.15.28:0/802102693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562560c1a7d0 con 0x562560ee2570 2024-04-25T01:42:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.607+0000 7ff11df42640 1 --2- 172.21.15.28:0/802102693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560efa330 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7ff110004af0 tx=0x7ff110005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72119320775fe625 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.607+0000 7ff102ffd640 1 -- 172.21.15.28:0/802102693 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff110056030 con 0x562560ee2570 2024-04-25T01:42:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.607+0000 7ff102ffd640 1 -- 172.21.15.28:0/802102693 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff110044070 con 0x562560ee2570 2024-04-25T01:42:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.607+0000 7ff102ffd640 1 -- 172.21.15.28:0/802102693 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1100307b0 con 0x562560ee2570 2024-04-25T01:42:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.608+0000 7ff12278d740 1 -- 172.21.15.28:0/802102693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 msgr2=0x562560efa330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.608+0000 7ff12278d740 1 --2- 172.21.15.28:0/802102693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560efa330 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7ff110004af0 tx=0x7ff110005df0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.608+0000 7ff12278d740 1 -- 172.21.15.28:0/802102693 shutdown_connections 2024-04-25T01:42:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.608+0000 7ff12278d740 1 --2- 172.21.15.28:0/802102693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560efa330 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.608+0000 7ff12278d740 1 --2- 172.21.15.28:0/802102693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7ea0 0x562560ef9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.608+0000 7ff12278d740 1 --2- 172.21.15.28:0/802102693 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7580 0x562560ee1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.608+0000 7ff12278d740 1 -- 172.21.15.28:0/802102693 >> 172.21.15.28:0/802102693 conn(0x562560ed38a0 msgr2=0x562560ed2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.608+0000 7ff12278d740 1 -- 172.21.15.28:0/802102693 shutdown_connections 2024-04-25T01:42:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.608+0000 7ff12278d740 1 -- 172.21.15.28:0/802102693 wait complete. 2024-04-25T01:42:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.608+0000 7ff12278d740 1 -- start start 2024-04-25T01:42:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.608+0000 7ff12278d740 1 Processor -- start 2024-04-25T01:42:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff12278d740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7580 0x562560ee78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff12278d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7ea0 0x562560ee7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff12278d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560f56660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff12278d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff1181b2230 con 0x562560ee2570 2024-04-25T01:42:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff12278d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff1181b1b60 con 0x562560ed7ea0 2024-04-25T01:42:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff12278d740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff1181aedc0 con 0x562560ed7580 2024-04-25T01:42:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff12278d740 10 client.0 _ll_register_callbacks cb 0x7ff10809b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff11df42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560f56660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff11df42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560f56660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49918/0 (socket says 172.21.15.28:49918) 2024-04-25T01:42:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff11df42640 1 -- 172.21.15.28:0/3148447852 learned_addr learned my addr 172.21.15.28:0/3148447852 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff11cf40640 1 --2- 172.21.15.28:0/3148447852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7ea0 0x562560ee7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff11d741640 1 --2- 172.21.15.28:0/3148447852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7580 0x562560ee78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff11df42640 1 -- 172.21.15.28:0/3148447852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7580 msgr2=0x562560ee78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff11df42640 1 --2- 172.21.15.28:0/3148447852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7580 0x562560ee78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff11df42640 1 -- 172.21.15.28:0/3148447852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7ea0 msgr2=0x562560ee7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff11df42640 1 --2- 172.21.15.28:0/3148447852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7ea0 0x562560ee7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff11df42640 1 -- 172.21.15.28:0/3148447852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562560f56ba0 con 0x562560ee2570 2024-04-25T01:42:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff11df42640 1 --2- 172.21.15.28:0/3148447852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560f56660 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7ff110005c90 tx=0x7ff1100095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4d2b54b97b1d796 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff100ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.609+0000 7ff100ff9640 1 -- 172.21.15.28:0/3148447852 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11006a020 con 0x562560ee2570 2024-04-25T01:42:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.610+0000 7ff100ff9640 1 -- 172.21.15.28:0/3148447852 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7ff110031940 con 0x562560ee2570 2024-04-25T01:42:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.610+0000 7ff12278d740 1 -- 172.21.15.28:0/3148447852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff110044070 con 0x562560ee2570 2024-04-25T01:42:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.610+0000 7ff100ff9640 1 -- 172.21.15.28:0/3148447852 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff110075050 con 0x562560ee2570 2024-04-25T01:42:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.610+0000 7ff12278d740 10 client.19065 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.610+0000 7ff12278d740 1 -- 172.21.15.28:0/3148447852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562560f57e10 con 0x562560ee2570 2024-04-25T01:42:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.610+0000 7ff100ff9640 1 -- 172.21.15.28:0/3148447852 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff110044070 con 0x562560ee2570 2024-04-25T01:42:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.610+0000 7ff100ff9640 1 -- 172.21.15.28:0/3148447852 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff1100667e0 con 0x562560ee2570 2024-04-25T01:42:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.610+0000 7ff12278d740 10 client.19065 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.610+0000 7ff12278d740 10 client.19065 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:42:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.610+0000 7ff12278d740 20 client.19065 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.610+0000 7ff12278d740 1 --2- 172.21.15.28:0/3148447852 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x562560f58460 0x562560f5a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.610+0000 7ff12278d740 4 client.19065 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:42:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.610+0000 7ff12278d740 1 -- 172.21.15.28:0/3148447852 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562560e43f00 con 0x562560f58460 2024-04-25T01:42:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.610+0000 7ff11d741640 1 --2- 172.21.15.28:0/3148447852 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x562560f58460 0x562560f5a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.611+0000 7ff100ff9640 10 client.19065 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:42:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.611+0000 7ff11d741640 1 --2- 172.21.15.28:0/3148447852 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x562560f58460 0x562560f5a850 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bf14aa684f2f8ce6 server_cookie=915f247db52f256f in_seq=0 out_seq=0 2024-04-25T01:42:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.611+0000 7ff100ff9640 1 -- 172.21.15.28:0/3148447852 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562560e43f00 con 0x562560f58460 2024-04-25T01:42:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.611+0000 7ff100ff9640 10 client.19065 handle_command_reply: tid=0 2024-04-25T01:42:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.611+0000 7ff12278d740 1 client.19065 shutdown 2024-04-25T01:42:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.611+0000 7ff12278d740 1 -- 172.21.15.28:0/3148447852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 msgr2=0x562560f56660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.611+0000 7ff12278d740 1 --2- 172.21.15.28:0/3148447852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560f56660 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7ff110005c90 tx=0x7ff1100095c0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.611+0000 7ff12278d740 1 -- 172.21.15.28:0/3148447852 shutdown_connections 2024-04-25T01:42:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.611+0000 7ff12278d740 1 --2- 172.21.15.28:0/3148447852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562560ee2570 0x562560f56660 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.611+0000 7ff12278d740 1 --2- 172.21.15.28:0/3148447852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562560ed7ea0 0x562560ee7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.611+0000 7ff12278d740 1 --2- 172.21.15.28:0/3148447852 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x562560f58460 0x562560f5a850 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.611+0000 7ff12278d740 1 --2- 172.21.15.28:0/3148447852 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x562560ed7580 0x562560ee78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.611+0000 7ff100ff9640 0 client.19065 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:42:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.611+0000 7ff12278d740 1 -- 172.21.15.28:0/3148447852 >> 172.21.15.28:0/3148447852 conn(0x562560ed38a0 msgr2=0x562560edea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.611+0000 7ff12278d740 1 -- 172.21.15.28:0/3148447852 shutdown_connections 2024-04-25T01:42:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.611+0000 7ff12278d740 1 -- 172.21.15.28:0/3148447852 wait complete. 2024-04-25T01:42:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.611+0000 7ff12278d740 20 client.19065 trim_cache size 0 max 16384 2024-04-25T01:42:36.608 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:42:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.612+0000 7ff11f9a5640 1 -- 172.21.15.28:0/2360021817 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff1040a2d80 msgr2=0x7ff1040a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.612+0000 7ff11f9a5640 1 --2- 172.21.15.28:0/2360021817 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff1040a2d80 0x7ff1040a5240 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff110045000 tx=0x7ff110048000 comp rx=0 tx=0).stop 2024-04-25T01:42:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.612+0000 7ff11f9a5640 1 -- 172.21.15.28:0/2360021817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff118100600 msgr2=0x7ff11819a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.612+0000 7ff11f9a5640 1 --2- 172.21.15.28:0/2360021817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff118100600 0x7ff11819a740 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7ff10800d950 tx=0x7ff10800de10 comp rx=0 tx=0).stop 2024-04-25T01:42:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.612+0000 7ff11f9a5640 1 -- 172.21.15.28:0/2360021817 shutdown_connections 2024-04-25T01:42:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.612+0000 7ff11f9a5640 1 --2- 172.21.15.28:0/2360021817 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff1040a2d80 0x7ff1040a5240 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.612+0000 7ff11f9a5640 1 --2- 172.21.15.28:0/2360021817 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff118107720 0x7ff11819eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.612+0000 7ff11f9a5640 1 --2- 172.21.15.28:0/2360021817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff118100600 0x7ff11819a740 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.612+0000 7ff11f9a5640 1 --2- 172.21.15.28:0/2360021817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1180ffc40 0x7ff11819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.612+0000 7ff11f9a5640 1 -- 172.21.15.28:0/2360021817 >> 172.21.15.28:0/2360021817 conn(0x7ff118076c00 msgr2=0x7ff11810d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.612+0000 7ff11f9a5640 1 -- 172.21.15.28:0/2360021817 shutdown_connections 2024-04-25T01:42:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.612+0000 7ff11f9a5640 1 -- 172.21.15.28:0/2360021817 wait complete. 2024-04-25T01:42:36.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:42:36.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.728+0000 7f3a3dc1c640 1 -- 172.21.15.28:0/903187113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a38100990 msgr2=0x7f3a38100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.728+0000 7f3a3dc1c640 1 --2- 172.21.15.28:0/903187113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a38100990 0x7f3a38100df0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c009900 tx=0x7f3a2c02a410 comp rx=0 tx=0).stop 2024-04-25T01:42:36.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.728+0000 7f3a3dc1c640 1 -- 172.21.15.28:0/903187113 shutdown_connections 2024-04-25T01:42:36.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.728+0000 7f3a3dc1c640 1 --2- 172.21.15.28:0/903187113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a38101330 0x7f3a3810d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.728+0000 7f3a3dc1c640 1 --2- 172.21.15.28:0/903187113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a38100990 0x7f3a38100df0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.728+0000 7f3a3dc1c640 1 --2- 172.21.15.28:0/903187113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3a38106690 0x7f3a38106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.728+0000 7f3a3dc1c640 1 -- 172.21.15.28:0/903187113 >> 172.21.15.28:0/903187113 conn(0x7f3a38076440 msgr2=0x7f3a38076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.728+0000 7f3a3dc1c640 1 -- 172.21.15.28:0/903187113 shutdown_connections 2024-04-25T01:42:36.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.728+0000 7f3a3dc1c640 1 -- 172.21.15.28:0/903187113 wait complete. 2024-04-25T01:42:36.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.728+0000 7f3a3dc1c640 1 Processor -- start 2024-04-25T01:42:36.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.729+0000 7f3a3dc1c640 1 -- start start 2024-04-25T01:42:36.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.729+0000 7f3a3dc1c640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3a38100990 0x7f3a38197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.729+0000 7f3a3dc1c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a38101330 0x7f3a38198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.729+0000 7f3a3dc1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a38106690 0x7f3a3819c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.729+0000 7f3a3dc1c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3a381109a0 con 0x7f3a38106690 2024-04-25T01:42:36.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.729+0000 7f3a3dc1c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3a38110820 con 0x7f3a38101330 2024-04-25T01:42:36.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.729+0000 7f3a3dc1c640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f3a38110b20 con 0x7f3a38100990 2024-04-25T01:42:36.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.729+0000 7f3a37fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a38106690 0x7f3a3819c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.729+0000 7f3a37fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a38106690 0x7f3a3819c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47576/0 (socket says 172.21.15.28:47576) 2024-04-25T01:42:36.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.729+0000 7f3a37fff640 1 -- 172.21.15.28:0/3646207550 learned_addr learned my addr 172.21.15.28:0/3646207550 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.729+0000 7f3a36ffd640 1 --2- 172.21.15.28:0/3646207550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a38101330 0x7f3a38198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.729+0000 7f3a377fe640 1 --2- 172.21.15.28:0/3646207550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3a38100990 0x7f3a38197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.729+0000 7f3a37fff640 1 -- 172.21.15.28:0/3646207550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3a38100990 msgr2=0x7f3a38197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.729+0000 7f3a37fff640 1 --2- 172.21.15.28:0/3646207550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3a38100990 0x7f3a38197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.729+0000 7f3a37fff640 1 -- 172.21.15.28:0/3646207550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a38101330 msgr2=0x7f3a38198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.729+0000 7f3a37fff640 1 --2- 172.21.15.28:0/3646207550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a38101330 0x7f3a38198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.729+0000 7f3a37fff640 1 -- 172.21.15.28:0/3646207550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3819cfb0 con 0x7f3a38106690 2024-04-25T01:42:36.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.729+0000 7f3a377fe640 1 --2- 172.21.15.28:0/3646207550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3a38100990 0x7f3a38197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:36.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.729+0000 7f3a37fff640 1 --2- 172.21.15.28:0/3646207550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a38106690 0x7f3a3819c880 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000eaa0 tx=0x7f3a3000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.730+0000 7f3a34ff9640 1 -- 172.21.15.28:0/3646207550 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3000cdb0 con 0x7f3a38106690 2024-04-25T01:42:36.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.730+0000 7f3a3dc1c640 1 -- 172.21.15.28:0/3646207550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a3819d2a0 con 0x7f3a38106690 2024-04-25T01:42:36.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.730+0000 7f3a34ff9640 1 -- 172.21.15.28:0/3646207550 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a30005020 con 0x7f3a38106690 2024-04-25T01:42:36.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.730+0000 7f3a34ff9640 1 -- 172.21.15.28:0/3646207550 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a300116b0 con 0x7f3a38106690 2024-04-25T01:42:36.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.730+0000 7f3a3dc1c640 1 -- 172.21.15.28:0/3646207550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a381a4b30 con 0x7f3a38106690 2024-04-25T01:42:36.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.732+0000 7f3a34ff9640 1 -- 172.21.15.28:0/3646207550 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3a30005560 con 0x7f3a38106690 2024-04-25T01:42:36.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.732+0000 7f3a40a06740 1 Processor -- start 2024-04-25T01:42:36.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.732+0000 7f3a40a06740 1 -- start start 2024-04-25T01:42:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.732+0000 7f3a40a06740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd609990 0x5625bd609d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.732+0000 7f3a40a06740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5625bd60a2b0 0x5625bd614440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.732+0000 7f3a34ff9640 1 --2- 172.21.15.28:0/3646207550 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3a200a2d00 0x7f3a200a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.732+0000 7f3a40a06740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd614980 0x5625bd616d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.732+0000 7f3a40a06740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5625bd56a670 con 0x5625bd614980 2024-04-25T01:42:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.732+0000 7f3a40a06740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5625bd512970 con 0x5625bd609990 2024-04-25T01:42:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.732+0000 7f3a40a06740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5625bd566630 con 0x5625bd60a2b0 2024-04-25T01:42:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.733+0000 7f3a37fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd614980 0x5625bd616d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.733+0000 7f3a36ffd640 1 --2- 172.21.15.28:0/3646207550 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3a200a2d00 0x7f3a200a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.733+0000 7f3a37fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd614980 0x5625bd616d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47588/0 (socket says 172.21.15.28:47588) 2024-04-25T01:42:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.733+0000 7f3a37fff640 1 -- 172.21.15.28:0/1774488656 learned_addr learned my addr 172.21.15.28:0/1774488656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.733+0000 7f3a377fe640 1 --2- 172.21.15.28:0/1774488656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5625bd60a2b0 0x5625bd614440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.733+0000 7f3a34ff9640 1 -- 172.21.15.28:0/3646207550 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f3a300cfa30 con 0x7f3a38106690 2024-04-25T01:42:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.733+0000 7f3a36ffd640 1 --2- 172.21.15.28:0/1774488656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd609990 0x5625bd609d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.733+0000 7f3a37fff640 1 -- 172.21.15.28:0/1774488656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5625bd60a2b0 msgr2=0x5625bd614440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.733+0000 7f3a37fff640 1 --2- 172.21.15.28:0/1774488656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5625bd60a2b0 0x5625bd614440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.733+0000 7f3a37fff640 1 -- 172.21.15.28:0/1774488656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd609990 msgr2=0x5625bd609d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.733+0000 7f3a37fff640 1 --2- 172.21.15.28:0/1774488656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd609990 0x5625bd609d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.733+0000 7f3a37fff640 1 -- 172.21.15.28:0/1774488656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625bd536400 con 0x5625bd614980 2024-04-25T01:42:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.733+0000 7f3a36ffd640 1 --2- 172.21.15.28:0/3646207550 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3a200a2d00 0x7f3a200a51c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c009230 tx=0x7f3a2c032f40 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.733+0000 7f3a37fff640 1 --2- 172.21.15.28:0/1774488656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd614980 0x5625bd616d70 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f3a30009270 tx=0x7f3a300a1830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18efd10d6d4cb1a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.733+0000 7f3a1dffb640 1 -- 172.21.15.28:0/1774488656 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3001f030 con 0x5625bd614980 2024-04-25T01:42:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.733+0000 7f3a1dffb640 1 -- 172.21.15.28:0/1774488656 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a300a2510 con 0x5625bd614980 2024-04-25T01:42:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.733+0000 7f3a1dffb640 1 -- 172.21.15.28:0/1774488656 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a300052d0 con 0x5625bd614980 2024-04-25T01:42:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.734+0000 7f3a40a06740 1 -- 172.21.15.28:0/1774488656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd614980 msgr2=0x5625bd616d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.734+0000 7f3a40a06740 1 --2- 172.21.15.28:0/1774488656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd614980 0x5625bd616d70 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f3a30009270 tx=0x7f3a300a1830 comp rx=0 tx=0).stop 2024-04-25T01:42:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.734+0000 7f3a40a06740 1 -- 172.21.15.28:0/1774488656 shutdown_connections 2024-04-25T01:42:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.734+0000 7f3a40a06740 1 --2- 172.21.15.28:0/1774488656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd614980 0x5625bd616d70 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.734+0000 7f3a40a06740 1 --2- 172.21.15.28:0/1774488656 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5625bd60a2b0 0x5625bd614440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.734+0000 7f3a40a06740 1 --2- 172.21.15.28:0/1774488656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd609990 0x5625bd609d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.734+0000 7f3a40a06740 1 -- 172.21.15.28:0/1774488656 >> 172.21.15.28:0/1774488656 conn(0x5625bd605cb0 msgr2=0x5625bd606090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.734+0000 7f3a40a06740 1 -- 172.21.15.28:0/1774488656 shutdown_connections 2024-04-25T01:42:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.734+0000 7f3a40a06740 1 -- 172.21.15.28:0/1774488656 wait complete. 2024-04-25T01:42:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.734+0000 7f3a40a06740 1 -- start start 2024-04-25T01:42:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.735+0000 7f3a40a06740 1 Processor -- start 2024-04-25T01:42:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.735+0000 7f3a40a06740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd609990 0x5625bd614300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.735+0000 7f3a40a06740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd60a2b0 0x5625bd6115f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.735+0000 7f3a40a06740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5625bd614980 0x5625bd611b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.735+0000 7f3a40a06740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5625bd617710 con 0x5625bd60a2b0 2024-04-25T01:42:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.735+0000 7f3a40a06740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5625bd526050 con 0x5625bd609990 2024-04-25T01:42:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.735+0000 7f3a40a06740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5625bd617890 con 0x5625bd614980 2024-04-25T01:42:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.735+0000 7f3a40a06740 10 client.0 _ll_register_callbacks cb 0x5625bd27dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.735+0000 7f3a36ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd60a2b0 0x5625bd6115f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.735+0000 7f3a36ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd60a2b0 0x5625bd6115f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47604/0 (socket says 172.21.15.28:47604) 2024-04-25T01:42:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.735+0000 7f3a36ffd640 1 -- 172.21.15.28:0/2297457406 learned_addr learned my addr 172.21.15.28:0/2297457406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.735+0000 7f3a377fe640 1 --2- 172.21.15.28:0/2297457406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd609990 0x5625bd614300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.735+0000 7f3a36ffd640 1 -- 172.21.15.28:0/2297457406 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5625bd614980 msgr2=0x5625bd611b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.735+0000 7f3a36ffd640 1 --2- 172.21.15.28:0/2297457406 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5625bd614980 0x5625bd611b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.735+0000 7f3a36ffd640 1 -- 172.21.15.28:0/2297457406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd609990 msgr2=0x5625bd614300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.735+0000 7f3a36ffd640 1 --2- 172.21.15.28:0/2297457406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd609990 0x5625bd614300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.735+0000 7f3a36ffd640 1 -- 172.21.15.28:0/2297457406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625bd4f7dc0 con 0x5625bd60a2b0 2024-04-25T01:42:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.735+0000 7f3a36ffd640 1 --2- 172.21.15.28:0/2297457406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd60a2b0 0x5625bd6115f0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c009900 tx=0x7f3a2c044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cafb8d1065e998f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.735+0000 7f3a16ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.736+0000 7f3a16ffd640 1 -- 172.21.15.28:0/2297457406 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c0035f0 con 0x5625bd60a2b0 2024-04-25T01:42:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.736+0000 7f3a16ffd640 1 -- 172.21.15.28:0/2297457406 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f3a2c003790 con 0x5625bd60a2b0 2024-04-25T01:42:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.736+0000 7f3a40a06740 1 -- 172.21.15.28:0/2297457406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a300a2510 con 0x5625bd60a2b0 2024-04-25T01:42:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.736+0000 7f3a16ffd640 1 -- 172.21.15.28:0/2297457406 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3a2c05a050 con 0x5625bd60a2b0 2024-04-25T01:42:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.736+0000 7f3a40a06740 10 client.19077 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.736+0000 7f3a40a06740 1 -- 172.21.15.28:0/2297457406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5625bd27e3f0 con 0x5625bd60a2b0 2024-04-25T01:42:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.736+0000 7f3a16ffd640 1 -- 172.21.15.28:0/2297457406 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a300a2510 con 0x5625bd60a2b0 2024-04-25T01:42:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.736+0000 7f3a16ffd640 1 -- 172.21.15.28:0/2297457406 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3a2c04c980 con 0x5625bd60a2b0 2024-04-25T01:42:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.736+0000 7f3a40a06740 10 client.19077 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.736+0000 7f3a40a06740 10 client.19077 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:42:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.736+0000 7f3a40a06740 20 client.19077 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.736+0000 7f3a40a06740 1 --2- 172.21.15.28:0/2297457406 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5625bd624a10 0x5625bd626e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.736+0000 7f3a40a06740 4 client.19077 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.736+0000 7f3a40a06740 1 -- 172.21.15.28:0/2297457406 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625bd577bb0 con 0x5625bd624a10 2024-04-25T01:42:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.737+0000 7f3a377fe640 1 --2- 172.21.15.28:0/2297457406 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5625bd624a10 0x5625bd626e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.737+0000 7f3a16ffd640 10 client.19077 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:42:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.737+0000 7f3a377fe640 1 --2- 172.21.15.28:0/2297457406 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5625bd624a10 0x5625bd626e80 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=26e52c2203d4d306 server_cookie=700b775cc166d30f in_seq=0 out_seq=0 2024-04-25T01:42:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.738+0000 7f3a16ffd640 1 -- 172.21.15.28:0/2297457406 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5625bd577bb0 con 0x5625bd624a10 2024-04-25T01:42:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.738+0000 7f3a16ffd640 10 client.19077 handle_command_reply: tid=0 2024-04-25T01:42:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.738+0000 7f3a40a06740 1 client.19077 shutdown 2024-04-25T01:42:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.738+0000 7f3a40a06740 1 -- 172.21.15.28:0/2297457406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd60a2b0 msgr2=0x5625bd6115f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.738+0000 7f3a40a06740 1 --2- 172.21.15.28:0/2297457406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd60a2b0 0x5625bd6115f0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c009900 tx=0x7f3a2c044000 comp rx=0 tx=0).stop 2024-04-25T01:42:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.738+0000 7f3a40a06740 1 -- 172.21.15.28:0/2297457406 shutdown_connections 2024-04-25T01:42:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.738+0000 7f3a40a06740 1 --2- 172.21.15.28:0/2297457406 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5625bd624a10 0x5625bd626e80 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.738+0000 7f3a40a06740 1 --2- 172.21.15.28:0/2297457406 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5625bd614980 0x5625bd611b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.738+0000 7f3a16ffd640 0 client.19077 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:42:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.738+0000 7f3a40a06740 1 --2- 172.21.15.28:0/2297457406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd60a2b0 0x5625bd6115f0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.738+0000 7f3a40a06740 1 --2- 172.21.15.28:0/2297457406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd609990 0x5625bd614300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.738+0000 7f3a40a06740 1 -- 172.21.15.28:0/2297457406 >> 172.21.15.28:0/2297457406 conn(0x5625bd605cb0 msgr2=0x5625bd60b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.738+0000 7f3a40a06740 1 -- 172.21.15.28:0/2297457406 shutdown_connections 2024-04-25T01:42:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.738+0000 7f3a40a06740 1 -- 172.21.15.28:0/2297457406 wait complete. 2024-04-25T01:42:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.738+0000 7f3a40a06740 20 client.19077 trim_cache size 0 max 16384 2024-04-25T01:42:36.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.751+0000 7f3a40a06740 1 Processor -- start 2024-04-25T01:42:36.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.751+0000 7f3a40a06740 1 -- start start 2024-04-25T01:42:36.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.752+0000 7f3a40a06740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd609990 0x5625bd600b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.752+0000 7f3a40a06740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5625bd60a2b0 0x5625bd613e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.752+0000 7f3a40a06740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd614980 0x5625bd61cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.752+0000 7f3a40a06740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3a381ac960 con 0x5625bd609990 2024-04-25T01:42:36.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.752+0000 7f3a40a06740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3a381ae4a0 con 0x5625bd614980 2024-04-25T01:42:36.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.752+0000 7f3a40a06740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f3a381ad030 con 0x5625bd60a2b0 2024-04-25T01:42:36.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.752+0000 7f3a377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd609990 0x5625bd600b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.752+0000 7f3a377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd609990 0x5625bd600b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47616/0 (socket says 172.21.15.28:47616) 2024-04-25T01:42:36.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.752+0000 7f3a377fe640 1 -- 172.21.15.28:0/431124551 learned_addr learned my addr 172.21.15.28:0/431124551 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.752+0000 7f3a37fff640 1 --2- 172.21.15.28:0/431124551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd614980 0x5625bd61cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.752+0000 7f3a377fe640 1 -- 172.21.15.28:0/431124551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5625bd60a2b0 msgr2=0x5625bd613e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.752+0000 7f3a377fe640 1 --2- 172.21.15.28:0/431124551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5625bd60a2b0 0x5625bd613e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.752+0000 7f3a377fe640 1 -- 172.21.15.28:0/431124551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd614980 msgr2=0x5625bd61cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.752+0000 7f3a377fe640 1 --2- 172.21.15.28:0/431124551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd614980 0x5625bd61cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.752+0000 7f3a377fe640 1 -- 172.21.15.28:0/431124551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625bd60a0e0 con 0x5625bd609990 2024-04-25T01:42:36.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.752+0000 7f3a377fe640 1 --2- 172.21.15.28:0/431124551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd609990 0x5625bd600b40 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f3a28004700 tx=0x7f3a2800f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff6803380de2b12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.752+0000 7f3a1d7fa640 1 -- 172.21.15.28:0/431124551 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a2801c470 con 0x5625bd609990 2024-04-25T01:42:36.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.752+0000 7f3a1d7fa640 1 -- 172.21.15.28:0/431124551 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a28005500 con 0x5625bd609990 2024-04-25T01:42:36.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.752+0000 7f3a1d7fa640 1 -- 172.21.15.28:0/431124551 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a28004bd0 con 0x5625bd609990 2024-04-25T01:42:36.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.753+0000 7f3a40a06740 1 -- 172.21.15.28:0/431124551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd609990 msgr2=0x5625bd600b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.753+0000 7f3a40a06740 1 --2- 172.21.15.28:0/431124551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd609990 0x5625bd600b40 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f3a28004700 tx=0x7f3a2800f7f0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.753+0000 7f3a40a06740 1 -- 172.21.15.28:0/431124551 shutdown_connections 2024-04-25T01:42:36.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.753+0000 7f3a40a06740 1 --2- 172.21.15.28:0/431124551 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd614980 0x5625bd61cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.753+0000 7f3a40a06740 1 --2- 172.21.15.28:0/431124551 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5625bd60a2b0 0x5625bd613e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.753+0000 7f3a40a06740 1 --2- 172.21.15.28:0/431124551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd609990 0x5625bd600b40 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.753+0000 7f3a40a06740 1 -- 172.21.15.28:0/431124551 >> 172.21.15.28:0/431124551 conn(0x5625bd605cb0 msgr2=0x5625bd61f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.753+0000 7f3a40a06740 1 -- 172.21.15.28:0/431124551 shutdown_connections 2024-04-25T01:42:36.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.753+0000 7f3a40a06740 1 -- 172.21.15.28:0/431124551 wait complete. 2024-04-25T01:42:36.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.753+0000 7f3a40a06740 1 -- start start 2024-04-25T01:42:36.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.753+0000 7f3a40a06740 1 Processor -- start 2024-04-25T01:42:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a40a06740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5625bd609990 0x5625bd61ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a40a06740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd60a2b0 0x5625bd61b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a40a06740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd614980 0x5625bd689fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a40a06740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3a381addd0 con 0x5625bd614980 2024-04-25T01:42:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a40a06740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3a381ad700 con 0x5625bd60a2b0 2024-04-25T01:42:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a40a06740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f3a381aa960 con 0x5625bd609990 2024-04-25T01:42:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a40a06740 10 client.0 _ll_register_callbacks cb 0x7f3a2c031780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a37fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd614980 0x5625bd689fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a37fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd614980 0x5625bd689fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47630/0 (socket says 172.21.15.28:47630) 2024-04-25T01:42:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a37fff640 1 -- 172.21.15.28:0/3668956093 learned_addr learned my addr 172.21.15.28:0/3668956093 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a36ffd640 1 --2- 172.21.15.28:0/3668956093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd60a2b0 0x5625bd61b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a377fe640 1 --2- 172.21.15.28:0/3668956093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5625bd609990 0x5625bd61ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a37fff640 1 -- 172.21.15.28:0/3668956093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5625bd609990 msgr2=0x5625bd61ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a37fff640 1 --2- 172.21.15.28:0/3668956093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5625bd609990 0x5625bd61ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a37fff640 1 -- 172.21.15.28:0/3668956093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd60a2b0 msgr2=0x5625bd61b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a37fff640 1 --2- 172.21.15.28:0/3668956093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd60a2b0 0x5625bd61b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a37fff640 1 -- 172.21.15.28:0/3668956093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625bd3487d0 con 0x5625bd614980 2024-04-25T01:42:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a37fff640 1 --2- 172.21.15.28:0/3668956093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd614980 0x5625bd689fd0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000efd0 tx=0x7f3a3009a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad01ab2bdd43e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a16ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a16ffd640 1 -- 172.21.15.28:0/3668956093 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3001f070 con 0x5625bd614980 2024-04-25T01:42:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a40a06740 1 -- 172.21.15.28:0/3668956093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a28005500 con 0x5625bd614980 2024-04-25T01:42:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.754+0000 7f3a16ffd640 1 -- 172.21.15.28:0/3668956093 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f3a300a3fa0 con 0x5625bd614980 2024-04-25T01:42:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.755+0000 7f3a16ffd640 1 -- 172.21.15.28:0/3668956093 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3a300a4830 con 0x5625bd614980 2024-04-25T01:42:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.755+0000 7f3a40a06740 10 client.19083 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.755+0000 7f3a40a06740 1 -- 172.21.15.28:0/3668956093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5625bd68b5b0 con 0x5625bd614980 2024-04-25T01:42:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.755+0000 7f3a16ffd640 1 -- 172.21.15.28:0/3668956093 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a28005500 con 0x5625bd614980 2024-04-25T01:42:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.755+0000 7f3a16ffd640 1 -- 172.21.15.28:0/3668956093 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3a3000f040 con 0x5625bd614980 2024-04-25T01:42:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.755+0000 7f3a40a06740 10 client.19083 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.755+0000 7f3a40a06740 10 client.19083 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:42:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.755+0000 7f3a40a06740 20 client.19083 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.755+0000 7f3a40a06740 1 --2- 172.21.15.28:0/3668956093 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5625bd68b8a0 0x5625bd68dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.755+0000 7f3a40a06740 4 client.19083 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:42:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.755+0000 7f3a40a06740 1 -- 172.21.15.28:0/3668956093 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5625bd571f00 con 0x5625bd68b8a0 2024-04-25T01:42:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.755+0000 7f3a377fe640 1 --2- 172.21.15.28:0/3668956093 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5625bd68b8a0 0x5625bd68dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.756+0000 7f3a377fe640 1 --2- 172.21.15.28:0/3668956093 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5625bd68b8a0 0x5625bd68dc90 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8977a9d1787fa39c server_cookie=a7da3b28b15d8c92 in_seq=0 out_seq=0 2024-04-25T01:42:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.756+0000 7f3a16ffd640 10 client.19083 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:42:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.756+0000 7f3a16ffd640 1 -- 172.21.15.28:0/3668956093 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5625bd571f00 con 0x5625bd68b8a0 2024-04-25T01:42:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.756+0000 7f3a16ffd640 10 client.19083 handle_command_reply: tid=0 2024-04-25T01:42:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.756+0000 7f3a40a06740 1 client.19083 shutdown 2024-04-25T01:42:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.756+0000 7f3a40a06740 1 -- 172.21.15.28:0/3668956093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd614980 msgr2=0x5625bd689fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.756+0000 7f3a40a06740 1 --2- 172.21.15.28:0/3668956093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd614980 0x5625bd689fd0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000efd0 tx=0x7f3a3009a0d0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.756+0000 7f3a40a06740 1 -- 172.21.15.28:0/3668956093 shutdown_connections 2024-04-25T01:42:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.756+0000 7f3a40a06740 1 --2- 172.21.15.28:0/3668956093 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5625bd68b8a0 0x5625bd68dc90 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.756+0000 7f3a40a06740 1 --2- 172.21.15.28:0/3668956093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5625bd614980 0x5625bd689fd0 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.756+0000 7f3a40a06740 1 --2- 172.21.15.28:0/3668956093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5625bd60a2b0 0x5625bd61b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.756+0000 7f3a16ffd640 0 client.19083 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:42:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.756+0000 7f3a40a06740 1 --2- 172.21.15.28:0/3668956093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5625bd609990 0x5625bd61ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.756+0000 7f3a40a06740 1 -- 172.21.15.28:0/3668956093 >> 172.21.15.28:0/3668956093 conn(0x5625bd605cb0 msgr2=0x5625bd6009c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.756+0000 7f3a40a06740 1 -- 172.21.15.28:0/3668956093 shutdown_connections 2024-04-25T01:42:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.756+0000 7f3a40a06740 1 -- 172.21.15.28:0/3668956093 wait complete. 2024-04-25T01:42:36.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.756+0000 7f3a40a06740 20 client.19083 trim_cache size 0 max 16384 2024-04-25T01:42:36.750 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:42:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.757+0000 7f3a3dc1c640 1 -- 172.21.15.28:0/3646207550 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3a200a2d00 msgr2=0x7f3a200a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.757+0000 7f3a3dc1c640 1 --2- 172.21.15.28:0/3646207550 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3a200a2d00 0x7f3a200a51c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c009230 tx=0x7f3a2c032f40 comp rx=0 tx=0).stop 2024-04-25T01:42:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.757+0000 7f3a3dc1c640 1 -- 172.21.15.28:0/3646207550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a38106690 msgr2=0x7f3a3819c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.757+0000 7f3a3dc1c640 1 --2- 172.21.15.28:0/3646207550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a38106690 0x7f3a3819c880 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000eaa0 tx=0x7f3a3000ef60 comp rx=0 tx=0).stop 2024-04-25T01:42:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.757+0000 7f3a3dc1c640 1 -- 172.21.15.28:0/3646207550 shutdown_connections 2024-04-25T01:42:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.757+0000 7f3a3dc1c640 1 --2- 172.21.15.28:0/3646207550 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3a200a2d00 0x7f3a200a51c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.757+0000 7f3a3dc1c640 1 --2- 172.21.15.28:0/3646207550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a38106690 0x7f3a3819c880 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.757+0000 7f3a3dc1c640 1 --2- 172.21.15.28:0/3646207550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a38101330 0x7f3a38198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.757+0000 7f3a3dc1c640 1 --2- 172.21.15.28:0/3646207550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3a38100990 0x7f3a38197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.757+0000 7f3a3dc1c640 1 -- 172.21.15.28:0/3646207550 >> 172.21.15.28:0/3646207550 conn(0x7f3a38076440 msgr2=0x7f3a3806a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.757+0000 7f3a3dc1c640 1 -- 172.21.15.28:0/3646207550 shutdown_connections 2024-04-25T01:42:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.757+0000 7f3a3dc1c640 1 -- 172.21.15.28:0/3646207550 wait complete. 2024-04-25T01:42:36.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:42:36.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.872+0000 7f38912a7640 1 -- 172.21.15.28:0/3612860093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388c100620 msgr2=0x7f388c100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.872+0000 7f38912a7640 1 --2- 172.21.15.28:0/3612860093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388c100620 0x7f388c100aa0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f3880009930 tx=0x7f388002a420 comp rx=0 tx=0).stop 2024-04-25T01:42:36.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.872+0000 7f38912a7640 1 -- 172.21.15.28:0/3612860093 shutdown_connections 2024-04-25T01:42:36.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.872+0000 7f38912a7640 1 --2- 172.21.15.28:0/3612860093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f388c100fe0 0x7f388c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.872+0000 7f38912a7640 1 --2- 172.21.15.28:0/3612860093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388c100620 0x7f388c100aa0 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.872+0000 7f38912a7640 1 --2- 172.21.15.28:0/3612860093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f388c106630 0x7f388c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.872+0000 7f38912a7640 1 -- 172.21.15.28:0/3612860093 >> 172.21.15.28:0/3612860093 conn(0x7f388c0776a0 msgr2=0x7f388c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.872+0000 7f38912a7640 1 -- 172.21.15.28:0/3612860093 shutdown_connections 2024-04-25T01:42:36.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.872+0000 7f38912a7640 1 -- 172.21.15.28:0/3612860093 wait complete. 2024-04-25T01:42:36.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.872+0000 7f38912a7640 1 Processor -- start 2024-04-25T01:42:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.873+0000 7f38912a7640 1 -- start start 2024-04-25T01:42:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.873+0000 7f38912a7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f388c100620 0x7f388c198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.873+0000 7f38912a7640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f388c100fe0 0x7f388c1992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.873+0000 7f38912a7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388c106630 0x7f388c1a3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.873+0000 7f38912a7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f388c110a40 con 0x7f388c106630 2024-04-25T01:42:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.873+0000 7f38912a7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f388c1108c0 con 0x7f388c100620 2024-04-25T01:42:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.873+0000 7f38912a7640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f388c110bc0 con 0x7f388c100fe0 2024-04-25T01:42:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.873+0000 7f388b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388c106630 0x7f388c1a3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.873+0000 7f388b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388c106630 0x7f388c1a3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47644/0 (socket says 172.21.15.28:47644) 2024-04-25T01:42:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.873+0000 7f388b7fe640 1 -- 172.21.15.28:0/4139358897 learned_addr learned my addr 172.21.15.28:0/4139358897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.873+0000 7f388affd640 1 --2- 172.21.15.28:0/4139358897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f388c100620 0x7f388c198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.873+0000 7f388a7fc640 1 --2- 172.21.15.28:0/4139358897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f388c100fe0 0x7f388c1992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.873+0000 7f388b7fe640 1 -- 172.21.15.28:0/4139358897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f388c100fe0 msgr2=0x7f388c1992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.873+0000 7f388b7fe640 1 --2- 172.21.15.28:0/4139358897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f388c100fe0 0x7f388c1992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.873+0000 7f388b7fe640 1 -- 172.21.15.28:0/4139358897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f388c100620 msgr2=0x7f388c198da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.873+0000 7f388b7fe640 1 --2- 172.21.15.28:0/4139358897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f388c100620 0x7f388c198da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.873+0000 7f388b7fe640 1 -- 172.21.15.28:0/4139358897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f388c1a4360 con 0x7f388c106630 2024-04-25T01:42:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.874+0000 7f388b7fe640 1 --2- 172.21.15.28:0/4139358897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388c106630 0x7f388c1a3e20 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f388400c970 tx=0x7f388400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.874+0000 7f3873fff640 1 -- 172.21.15.28:0/4139358897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3884007bb0 con 0x7f388c106630 2024-04-25T01:42:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.874+0000 7f3873fff640 1 -- 172.21.15.28:0/4139358897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3884007d50 con 0x7f388c106630 2024-04-25T01:42:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.874+0000 7f38912a7640 1 -- 172.21.15.28:0/4139358897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f388c1a4530 con 0x7f388c106630 2024-04-25T01:42:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.874+0000 7f3873fff640 1 -- 172.21.15.28:0/4139358897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3884018800 con 0x7f388c106630 2024-04-25T01:42:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.874+0000 7f38912a7640 1 -- 172.21.15.28:0/4139358897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f388c1a4a00 con 0x7f388c106630 2024-04-25T01:42:36.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.876+0000 7f3873fff640 1 -- 172.21.15.28:0/4139358897 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f388400fa50 con 0x7f388c106630 2024-04-25T01:42:36.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.876+0000 7f3893fb0740 1 Processor -- start 2024-04-25T01:42:36.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.876+0000 7f3893fb0740 1 -- start start 2024-04-25T01:42:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.876+0000 7f3893fb0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871eb480 0x5604871eb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.876+0000 7f3893fb0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871ebda0 0x5604871fa440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f3893fb0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 0x5604871fcd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f3893fb0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560487150670 con 0x5604871fa980 2024-04-25T01:42:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f3893fb0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5604870f8970 con 0x5604871ebda0 2024-04-25T01:42:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f3893fb0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56048714c630 con 0x5604871eb480 2024-04-25T01:42:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f388a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 0x5604871fcd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f388a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 0x5604871fcd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47646/0 (socket says 172.21.15.28:47646) 2024-04-25T01:42:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f388a7fc640 1 -- 172.21.15.28:0/4261919703 learned_addr learned my addr 172.21.15.28:0/4261919703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f3873fff640 1 --2- 172.21.15.28:0/4139358897 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f38740a2d00 0x7f38740a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f388affd640 1 --2- 172.21.15.28:0/4261919703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871ebda0 0x5604871fa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f388a7fc640 1 --2- 172.21.15.28:0/4261919703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871eb480 0x5604871eb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f388a7fc640 1 -- 172.21.15.28:0/4261919703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871eb480 msgr2=0x5604871eb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f388a7fc640 1 --2- 172.21.15.28:0/4261919703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871eb480 0x5604871eb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f388b7fe640 1 --2- 172.21.15.28:0/4139358897 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f38740a2d00 0x7f38740a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f388a7fc640 1 -- 172.21.15.28:0/4261919703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871ebda0 msgr2=0x5604871fa440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f388a7fc640 1 --2- 172.21.15.28:0/4261919703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871ebda0 0x5604871fa440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f388a7fc640 1 -- 172.21.15.28:0/4261919703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56048711c400 con 0x5604871fa980 2024-04-25T01:42:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f388affd640 1 --2- 172.21.15.28:0/4261919703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871ebda0 0x5604871fa440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f3873fff640 1 -- 172.21.15.28:0/4139358897 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f38840cf780 con 0x7f388c106630 2024-04-25T01:42:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f388a7fc640 1 --2- 172.21.15.28:0/4261919703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 0x5604871fcd70 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f3880006010 tx=0x7f3880009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c19d4937f3f1e7a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f38717fa640 1 -- 172.21.15.28:0/4261919703 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880035070 con 0x5604871fa980 2024-04-25T01:42:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.877+0000 7f388b7fe640 1 --2- 172.21.15.28:0/4139358897 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f38740a2d00 0x7f38740a51c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f388409b610 tx=0x7f388406d8d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.878+0000 7f38717fa640 1 -- 172.21.15.28:0/4261919703 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3880004030 con 0x5604871fa980 2024-04-25T01:42:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.878+0000 7f38717fa640 1 -- 172.21.15.28:0/4261919703 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38800047a0 con 0x5604871fa980 2024-04-25T01:42:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.878+0000 7f3893fb0740 1 -- 172.21.15.28:0/4261919703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 msgr2=0x5604871fcd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.878+0000 7f3893fb0740 1 --2- 172.21.15.28:0/4261919703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 0x5604871fcd70 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f3880006010 tx=0x7f3880009580 comp rx=0 tx=0).stop 2024-04-25T01:42:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.878+0000 7f3893fb0740 1 -- 172.21.15.28:0/4261919703 shutdown_connections 2024-04-25T01:42:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.878+0000 7f3893fb0740 1 --2- 172.21.15.28:0/4261919703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 0x5604871fcd70 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.878+0000 7f3893fb0740 1 --2- 172.21.15.28:0/4261919703 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871ebda0 0x5604871fa440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.878+0000 7f3893fb0740 1 --2- 172.21.15.28:0/4261919703 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871eb480 0x5604871eb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.878+0000 7f3893fb0740 1 -- 172.21.15.28:0/4261919703 >> 172.21.15.28:0/4261919703 conn(0x5604871e78a0 msgr2=0x5604871e7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.878+0000 7f3893fb0740 1 -- 172.21.15.28:0/4261919703 shutdown_connections 2024-04-25T01:42:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.878+0000 7f3893fb0740 1 -- 172.21.15.28:0/4261919703 wait complete. 2024-04-25T01:42:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f3893fb0740 1 -- start start 2024-04-25T01:42:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f3893fb0740 1 Processor -- start 2024-04-25T01:42:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f3893fb0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871eb480 0x5604871fa2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f3893fb0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871ebda0 0x5604871f75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f3893fb0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 0x5604871f7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f3893fb0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5604871fd6e0 con 0x5604871fa980 2024-04-25T01:42:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f3893fb0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56048710c050 con 0x5604871ebda0 2024-04-25T01:42:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f3893fb0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5604871fd860 con 0x5604871eb480 2024-04-25T01:42:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f3893fb0740 10 client.0 _ll_register_callbacks cb 0x560486e63be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f388a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 0x5604871f7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f388a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 0x5604871f7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47658/0 (socket says 172.21.15.28:47658) 2024-04-25T01:42:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f388a7fc640 1 -- 172.21.15.28:0/665360748 learned_addr learned my addr 172.21.15.28:0/665360748 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f388a7fc640 1 --2- 172.21.15.28:0/665360748 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871eb480 0x5604871fa2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f388affd640 1 --2- 172.21.15.28:0/665360748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871ebda0 0x5604871f75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f388a7fc640 1 -- 172.21.15.28:0/665360748 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871eb480 msgr2=0x5604871fa2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f388a7fc640 1 --2- 172.21.15.28:0/665360748 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871eb480 0x5604871fa2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f388a7fc640 1 -- 172.21.15.28:0/665360748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871ebda0 msgr2=0x5604871f75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f388a7fc640 1 --2- 172.21.15.28:0/665360748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871ebda0 0x5604871f75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f388a7fc640 1 -- 172.21.15.28:0/665360748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604870dddc0 con 0x5604871fa980 2024-04-25T01:42:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f388affd640 1 --2- 172.21.15.28:0/665360748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871ebda0 0x5604871f75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f388a7fc640 1 --2- 172.21.15.28:0/665360748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 0x5604871f7af0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f3880004210 tx=0x7f3880005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba07d70cc9529cc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.879+0000 7f386eff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.880+0000 7f386eff5640 1 -- 172.21.15.28:0/665360748 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880035040 con 0x5604871fa980 2024-04-25T01:42:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.880+0000 7f386eff5640 1 -- 172.21.15.28:0/665360748 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f3880004480 con 0x5604871fa980 2024-04-25T01:42:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.880+0000 7f3893fb0740 1 -- 172.21.15.28:0/665360748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3880004030 con 0x5604871fa980 2024-04-25T01:42:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.880+0000 7f386eff5640 1 -- 172.21.15.28:0/665360748 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f388005e050 con 0x5604871fa980 2024-04-25T01:42:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.880+0000 7f3893fb0740 10 client.19095 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.880+0000 7f3893fb0740 1 -- 172.21.15.28:0/665360748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560486e643f0 con 0x5604871fa980 2024-04-25T01:42:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.880+0000 7f386eff5640 1 -- 172.21.15.28:0/665360748 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3880004030 con 0x5604871fa980 2024-04-25T01:42:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.880+0000 7f386eff5640 1 -- 172.21.15.28:0/665360748 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f38800035f0 con 0x5604871fa980 2024-04-25T01:42:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.880+0000 7f3893fb0740 10 client.19095 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.880+0000 7f3893fb0740 10 client.19095 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.880+0000 7f3893fb0740 20 client.19095 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.880+0000 7f3893fb0740 1 --2- 172.21.15.28:0/665360748 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56048720a7f0 0x56048720cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.880+0000 7f3893fb0740 4 client.19095 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.880+0000 7f3893fb0740 1 -- 172.21.15.28:0/665360748 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56048715dbb0 con 0x56048720a7f0 2024-04-25T01:42:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.880+0000 7f388affd640 1 --2- 172.21.15.28:0/665360748 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56048720a7f0 0x56048720cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.881+0000 7f388affd640 1 --2- 172.21.15.28:0/665360748 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56048720a7f0 0x56048720cbe0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fdc1525b471a5da server_cookie=3eb0738aea224ac8 in_seq=0 out_seq=0 2024-04-25T01:42:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.881+0000 7f386eff5640 10 client.19095 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.882+0000 7f386eff5640 1 -- 172.21.15.28:0/665360748 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56048715dbb0 con 0x56048720a7f0 2024-04-25T01:42:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.882+0000 7f386eff5640 10 client.19095 handle_command_reply: tid=0 2024-04-25T01:42:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.882+0000 7f3893fb0740 1 client.19095 shutdown 2024-04-25T01:42:36.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.882+0000 7f3893fb0740 1 -- 172.21.15.28:0/665360748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 msgr2=0x5604871f7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.882+0000 7f3893fb0740 1 --2- 172.21.15.28:0/665360748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 0x5604871f7af0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f3880004210 tx=0x7f3880005390 comp rx=0 tx=0).stop 2024-04-25T01:42:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.882+0000 7f3893fb0740 1 -- 172.21.15.28:0/665360748 shutdown_connections 2024-04-25T01:42:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.882+0000 7f3893fb0740 1 --2- 172.21.15.28:0/665360748 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56048720a7f0 0x56048720cbe0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.882+0000 7f3893fb0740 1 --2- 172.21.15.28:0/665360748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 0x5604871f7af0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.882+0000 7f386eff5640 0 client.19095 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.882+0000 7f3893fb0740 1 --2- 172.21.15.28:0/665360748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871ebda0 0x5604871f75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.882+0000 7f3893fb0740 1 --2- 172.21.15.28:0/665360748 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871eb480 0x5604871fa2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.882+0000 7f3893fb0740 1 -- 172.21.15.28:0/665360748 >> 172.21.15.28:0/665360748 conn(0x5604871e78a0 msgr2=0x5604871ed6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.882+0000 7f3893fb0740 1 -- 172.21.15.28:0/665360748 shutdown_connections 2024-04-25T01:42:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.882+0000 7f3893fb0740 1 -- 172.21.15.28:0/665360748 wait complete. 2024-04-25T01:42:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.882+0000 7f3893fb0740 20 client.19095 trim_cache size 0 max 16384 2024-04-25T01:42:36.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.895+0000 7f3893fb0740 1 Processor -- start 2024-04-25T01:42:36.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.895+0000 7f3893fb0740 1 -- start start 2024-04-25T01:42:36.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.895+0000 7f3893fb0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871eb480 0x5604871f6590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.895+0000 7f3893fb0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871ebda0 0x5604871f9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.895+0000 7f3893fb0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871fa980 0x56048720fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.895+0000 7f3893fb0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f388c1ac8d0 con 0x5604871ebda0 2024-04-25T01:42:36.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.895+0000 7f3893fb0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f388c1ae410 con 0x5604871eb480 2024-04-25T01:42:36.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.895+0000 7f3893fb0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f388c1acfa0 con 0x5604871fa980 2024-04-25T01:42:36.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.896+0000 7f388affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871ebda0 0x5604871f9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.896+0000 7f388affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871ebda0 0x5604871f9df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47668/0 (socket says 172.21.15.28:47668) 2024-04-25T01:42:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.896+0000 7f388affd640 1 -- 172.21.15.28:0/1316113246 learned_addr learned my addr 172.21.15.28:0/1316113246 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.896+0000 7f388a7fc640 1 --2- 172.21.15.28:0/1316113246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871eb480 0x5604871f6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.896+0000 7f388a7fc640 1 --2- 172.21.15.28:0/1316113246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871fa980 0x56048720fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.896+0000 7f388affd640 1 -- 172.21.15.28:0/1316113246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871fa980 msgr2=0x56048720fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.896+0000 7f388affd640 1 --2- 172.21.15.28:0/1316113246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871fa980 0x56048720fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.896+0000 7f388affd640 1 -- 172.21.15.28:0/1316113246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871eb480 msgr2=0x5604871f6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.896+0000 7f388affd640 1 --2- 172.21.15.28:0/1316113246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871eb480 0x5604871f6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.896+0000 7f388affd640 1 -- 172.21.15.28:0/1316113246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604871ebbd0 con 0x5604871ebda0 2024-04-25T01:42:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.896+0000 7f388a7fc640 1 --2- 172.21.15.28:0/1316113246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871fa980 0x56048720fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.896+0000 7f388affd640 1 --2- 172.21.15.28:0/1316113246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871ebda0 0x5604871f9df0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x5604871edf80 tx=0x7f387c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9f9eb31859b0c5a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.896+0000 7f3870ff9640 1 -- 172.21.15.28:0/1316113246 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c004b20 con 0x5604871ebda0 2024-04-25T01:42:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.896+0000 7f3870ff9640 1 -- 172.21.15.28:0/1316113246 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f387c004cc0 con 0x5604871ebda0 2024-04-25T01:42:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.896+0000 7f3870ff9640 1 -- 172.21.15.28:0/1316113246 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c0058a0 con 0x5604871ebda0 2024-04-25T01:42:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.897+0000 7f3893fb0740 1 -- 172.21.15.28:0/1316113246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871ebda0 msgr2=0x5604871f9df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.897+0000 7f3893fb0740 1 --2- 172.21.15.28:0/1316113246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871ebda0 0x5604871f9df0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x5604871edf80 tx=0x7f387c004a40 comp rx=0 tx=0).stop 2024-04-25T01:42:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.897+0000 7f3893fb0740 1 -- 172.21.15.28:0/1316113246 shutdown_connections 2024-04-25T01:42:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.897+0000 7f3893fb0740 1 --2- 172.21.15.28:0/1316113246 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871fa980 0x56048720fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.897+0000 7f3893fb0740 1 --2- 172.21.15.28:0/1316113246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871ebda0 0x5604871f9df0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.897+0000 7f3893fb0740 1 --2- 172.21.15.28:0/1316113246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871eb480 0x5604871f6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.897+0000 7f3893fb0740 1 -- 172.21.15.28:0/1316113246 >> 172.21.15.28:0/1316113246 conn(0x5604871e78a0 msgr2=0x5604871f5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.897+0000 7f3893fb0740 1 -- 172.21.15.28:0/1316113246 shutdown_connections 2024-04-25T01:42:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.897+0000 7f3893fb0740 1 -- 172.21.15.28:0/1316113246 wait complete. 2024-04-25T01:42:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.897+0000 7f3893fb0740 1 -- start start 2024-04-25T01:42:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.897+0000 7f3893fb0740 1 Processor -- start 2024-04-25T01:42:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.897+0000 7f3893fb0740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871eb480 0x56048720fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.897+0000 7f3893fb0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871ebda0 0x56048720d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.897+0000 7f3893fb0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 0x56048720d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.897+0000 7f3893fb0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f388c1add40 con 0x5604871fa980 2024-04-25T01:42:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.897+0000 7f3893fb0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f388c1ad670 con 0x5604871ebda0 2024-04-25T01:42:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.897+0000 7f3893fb0740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f388c1aa8d0 con 0x5604871eb480 2024-04-25T01:42:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.897+0000 7f3893fb0740 10 client.0 _ll_register_callbacks cb 0x7f3880032e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.898+0000 7f388a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 0x56048720d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.898+0000 7f388a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 0x56048720d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47682/0 (socket says 172.21.15.28:47682) 2024-04-25T01:42:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.898+0000 7f388a7fc640 1 -- 172.21.15.28:0/1670249185 learned_addr learned my addr 172.21.15.28:0/1670249185 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.898+0000 7f388a7fc640 1 --2- 172.21.15.28:0/1670249185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871eb480 0x56048720fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.898+0000 7f388affd640 1 --2- 172.21.15.28:0/1670249185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871ebda0 0x56048720d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.898+0000 7f388a7fc640 1 -- 172.21.15.28:0/1670249185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871eb480 msgr2=0x56048720fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.898+0000 7f388a7fc640 1 --2- 172.21.15.28:0/1670249185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871eb480 0x56048720fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.898+0000 7f388a7fc640 1 -- 172.21.15.28:0/1670249185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871ebda0 msgr2=0x56048720d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.898+0000 7f388a7fc640 1 --2- 172.21.15.28:0/1670249185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871ebda0 0x56048720d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.898+0000 7f388a7fc640 1 -- 172.21.15.28:0/1670249185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560486f2e7d0 con 0x5604871fa980 2024-04-25T01:42:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.898+0000 7f388a7fc640 1 --2- 172.21.15.28:0/1670249185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 0x56048720d640 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f38800095c0 tx=0x7f38800052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89defecd76db28de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.898+0000 7f386eff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.898+0000 7f386eff5640 1 -- 172.21.15.28:0/1670249185 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880035070 con 0x5604871fa980 2024-04-25T01:42:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.898+0000 7f3893fb0740 1 -- 172.21.15.28:0/1670249185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f387c004cc0 con 0x5604871fa980 2024-04-25T01:42:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.898+0000 7f386eff5640 1 -- 172.21.15.28:0/1670249185 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f3880052040 con 0x5604871fa980 2024-04-25T01:42:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.898+0000 7f386eff5640 1 -- 172.21.15.28:0/1670249185 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3880005400 con 0x5604871fa980 2024-04-25T01:42:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.898+0000 7f3893fb0740 10 client.19101 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.898+0000 7f3893fb0740 1 -- 172.21.15.28:0/1670249185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560487270500 con 0x5604871fa980 2024-04-25T01:42:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.898+0000 7f386eff5640 1 -- 172.21.15.28:0/1670249185 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f387c004cc0 con 0x5604871fa980 2024-04-25T01:42:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.899+0000 7f386eff5640 1 -- 172.21.15.28:0/1670249185 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3880004770 con 0x5604871fa980 2024-04-25T01:42:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.899+0000 7f3893fb0740 10 client.19101 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.899+0000 7f3893fb0740 10 client.19101 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.899+0000 7f3893fb0740 20 client.19101 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.899+0000 7f3893fb0740 1 --2- 172.21.15.28:0/1670249185 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x560487270b10 0x560487272f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.899+0000 7f3893fb0740 4 client.19101 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:42:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.899+0000 7f3893fb0740 1 -- 172.21.15.28:0/1670249185 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560487157f00 con 0x560487270b10 2024-04-25T01:42:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.899+0000 7f388affd640 1 --2- 172.21.15.28:0/1670249185 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x560487270b10 0x560487272f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.899+0000 7f386eff5640 10 client.19101 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.899+0000 7f388affd640 1 --2- 172.21.15.28:0/1670249185 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x560487270b10 0x560487272f30 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6004167b58b11b49 server_cookie=b9a64d3c439b149c in_seq=0 out_seq=0 2024-04-25T01:42:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f386eff5640 1 -- 172.21.15.28:0/1670249185 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560487157f00 con 0x560487270b10 2024-04-25T01:42:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f386eff5640 10 client.19101 handle_command_reply: tid=0 2024-04-25T01:42:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f3893fb0740 1 client.19101 shutdown 2024-04-25T01:42:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f3893fb0740 1 -- 172.21.15.28:0/1670249185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 msgr2=0x56048720d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f3893fb0740 1 --2- 172.21.15.28:0/1670249185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 0x56048720d640 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f38800095c0 tx=0x7f38800052f0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f3893fb0740 1 -- 172.21.15.28:0/1670249185 shutdown_connections 2024-04-25T01:42:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f3893fb0740 1 --2- 172.21.15.28:0/1670249185 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x560487270b10 0x560487272f30 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f3893fb0740 1 --2- 172.21.15.28:0/1670249185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604871fa980 0x56048720d640 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f3893fb0740 1 --2- 172.21.15.28:0/1670249185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5604871ebda0 0x56048720d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f3893fb0740 1 --2- 172.21.15.28:0/1670249185 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5604871eb480 0x56048720fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f3893fb0740 1 -- 172.21.15.28:0/1670249185 >> 172.21.15.28:0/1670249185 conn(0x5604871e78a0 msgr2=0x5604871f6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f386eff5640 0 client.19101 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f3893fb0740 1 -- 172.21.15.28:0/1670249185 shutdown_connections 2024-04-25T01:42:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f3893fb0740 1 -- 172.21.15.28:0/1670249185 wait complete. 2024-04-25T01:42:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f3893fb0740 20 client.19101 trim_cache size 0 max 16384 2024-04-25T01:42:36.895 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:42:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f38912a7640 1 -- 172.21.15.28:0/4139358897 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f38740a2d00 msgr2=0x7f38740a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f38912a7640 1 --2- 172.21.15.28:0/4139358897 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f38740a2d00 0x7f38740a51c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f388409b610 tx=0x7f388406d8d0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f38912a7640 1 -- 172.21.15.28:0/4139358897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388c106630 msgr2=0x7f388c1a3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f38912a7640 1 --2- 172.21.15.28:0/4139358897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388c106630 0x7f388c1a3e20 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f388400c970 tx=0x7f388400ce30 comp rx=0 tx=0).stop 2024-04-25T01:42:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f38912a7640 1 -- 172.21.15.28:0/4139358897 shutdown_connections 2024-04-25T01:42:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f38912a7640 1 --2- 172.21.15.28:0/4139358897 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f38740a2d00 0x7f38740a51c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f38912a7640 1 --2- 172.21.15.28:0/4139358897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388c106630 0x7f388c1a3e20 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f38912a7640 1 --2- 172.21.15.28:0/4139358897 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f388c100fe0 0x7f388c1992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f38912a7640 1 --2- 172.21.15.28:0/4139358897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f388c100620 0x7f388c198da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.900+0000 7f38912a7640 1 -- 172.21.15.28:0/4139358897 >> 172.21.15.28:0/4139358897 conn(0x7f388c0776a0 msgr2=0x7f388c1030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.901+0000 7f38912a7640 1 -- 172.21.15.28:0/4139358897 shutdown_connections 2024-04-25T01:42:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:36.901+0000 7f38912a7640 1 -- 172.21.15.28:0/4139358897 wait complete. 2024-04-25T01:42:37.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:36 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/236939955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:42:37.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:36 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/236939955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:42:37.469 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:36 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/236939955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:42:37.900 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:42:37.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:42:37.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:42:38.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.031+0000 7f6ee20b3640 1 -- 172.21.15.28:0/2474656892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6edc107730 msgr2=0x7f6edc107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.031+0000 7f6ee20b3640 1 --2- 172.21.15.28:0/2474656892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6edc107730 0x7f6edc107b10 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc009930 tx=0x7f6ecc02a420 comp rx=0 tx=0).stop 2024-04-25T01:42:38.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.031+0000 7f6ee20b3640 1 -- 172.21.15.28:0/2474656892 shutdown_connections 2024-04-25T01:42:38.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.031+0000 7f6ee20b3640 1 --2- 172.21.15.28:0/2474656892 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6edc06a780 0x7f6edc10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.031+0000 7f6ee20b3640 1 --2- 172.21.15.28:0/2474656892 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6edc069dc0 0x7f6edc06a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.031+0000 7f6ee20b3640 1 --2- 172.21.15.28:0/2474656892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6edc107730 0x7f6edc107b10 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.031+0000 7f6ee20b3640 1 -- 172.21.15.28:0/2474656892 >> 172.21.15.28:0/2474656892 conn(0x7f6edc0776a0 msgr2=0x7f6edc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.031+0000 7f6ee20b3640 1 -- 172.21.15.28:0/2474656892 shutdown_connections 2024-04-25T01:42:38.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.031+0000 7f6ee20b3640 1 -- 172.21.15.28:0/2474656892 wait complete. 2024-04-25T01:42:38.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.032+0000 7f6ee20b3640 1 Processor -- start 2024-04-25T01:42:38.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.032+0000 7f6ee20b3640 1 -- start start 2024-04-25T01:42:38.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.032+0000 7f6ee20b3640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6edc069dc0 0x7f6edc198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.032+0000 7f6ee20b3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6edc06a780 0x7f6edc198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.032+0000 7f6ee20b3640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6edc107730 0x7f6edc19c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.032+0000 7f6ee20b3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc111b60 con 0x7f6edc06a780 2024-04-25T01:42:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.032+0000 7f6ee20b3640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc1119e0 con 0x7f6edc069dc0 2024-04-25T01:42:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.032+0000 7f6ee20b3640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc111ce0 con 0x7f6edc107730 2024-04-25T01:42:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.032+0000 7f6edaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6edc06a780 0x7f6edc198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.033+0000 7f6edaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6edc06a780 0x7f6edc198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47696/0 (socket says 172.21.15.28:47696) 2024-04-25T01:42:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.033+0000 7f6edaffd640 1 -- 172.21.15.28:0/1911606200 learned_addr learned my addr 172.21.15.28:0/1911606200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.033+0000 7f6edb7fe640 1 --2- 172.21.15.28:0/1911606200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6edc069dc0 0x7f6edc198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.033+0000 7f6edbfff640 1 --2- 172.21.15.28:0/1911606200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6edc107730 0x7f6edc19c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.033+0000 7f6edaffd640 1 -- 172.21.15.28:0/1911606200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6edc107730 msgr2=0x7f6edc19c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.033+0000 7f6edaffd640 1 --2- 172.21.15.28:0/1911606200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6edc107730 0x7f6edc19c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.033+0000 7f6edaffd640 1 -- 172.21.15.28:0/1911606200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6edc069dc0 msgr2=0x7f6edc198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.033+0000 7f6edaffd640 1 --2- 172.21.15.28:0/1911606200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6edc069dc0 0x7f6edc198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.033+0000 7f6edaffd640 1 -- 172.21.15.28:0/1911606200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6edc19d110 con 0x7f6edc06a780 2024-04-25T01:42:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.033+0000 7f6edaffd640 1 --2- 172.21.15.28:0/1911606200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6edc06a780 0x7f6edc198660 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000d770 tx=0x7f6ed000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.033+0000 7f6ed8ff9640 1 -- 172.21.15.28:0/1911606200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0017070 con 0x7f6edc06a780 2024-04-25T01:42:38.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.033+0000 7f6ed8ff9640 1 -- 172.21.15.28:0/1911606200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ed0005500 con 0x7f6edc06a780 2024-04-25T01:42:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.033+0000 7f6ee20b3640 1 -- 172.21.15.28:0/1911606200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6edc19d400 con 0x7f6edc06a780 2024-04-25T01:42:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.033+0000 7f6ed8ff9640 1 -- 172.21.15.28:0/1911606200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0004e00 con 0x7f6edc06a780 2024-04-25T01:42:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.033+0000 7f6ee20b3640 1 -- 172.21.15.28:0/1911606200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6edc1a4c90 con 0x7f6edc06a780 2024-04-25T01:42:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.036+0000 7f6ed8ff9640 1 -- 172.21.15.28:0/1911606200 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6ed0004fa0 con 0x7f6edc06a780 2024-04-25T01:42:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.036+0000 7f6ee4dbf740 1 Processor -- start 2024-04-25T01:42:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.036+0000 7f6ee4dbf740 1 -- start start 2024-04-25T01:42:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.036+0000 7f6ee4dbf740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbec980 0x55b11bbeacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.036+0000 7f6ee4dbf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbeb290 0x55b11bbeb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.036+0000 7f6ee4dbf740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbfa420 0x55b11bbfc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.036+0000 7f6ee4dbf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b11bb4f2c0 con 0x55b11bbeb290 2024-04-25T01:42:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.036+0000 7f6ee4dbf740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b11baf7910 con 0x55b11bbfa420 2024-04-25T01:42:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.036+0000 7f6ee4dbf740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b11bb4b280 con 0x55b11bbec980 2024-04-25T01:42:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.036+0000 7f6ed8ff9640 1 --2- 172.21.15.28:0/1911606200 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6ec40a2d00 0x7f6ec40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.036+0000 7f6edb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbeb290 0x55b11bbeb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.036+0000 7f6edb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbeb290 0x55b11bbeb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47708/0 (socket says 172.21.15.28:47708) 2024-04-25T01:42:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.036+0000 7f6edb7fe640 1 -- 172.21.15.28:0/1266132268 learned_addr learned my addr 172.21.15.28:0/1266132268 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.036+0000 7f6edaffd640 1 --2- 172.21.15.28:0/1266132268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbfa420 0x55b11bbfc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.036+0000 7f6edbfff640 1 --2- 172.21.15.28:0/1911606200 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6ec40a2d00 0x7f6ec40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.036+0000 7f6edbfff640 1 --2- 172.21.15.28:0/1266132268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbec980 0x55b11bbeacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.037+0000 7f6ed8ff9640 1 -- 172.21.15.28:0/1911606200 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f6ed00cf1b0 con 0x7f6edc06a780 2024-04-25T01:42:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.037+0000 7f6edb7fe640 1 -- 172.21.15.28:0/1266132268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbec980 msgr2=0x55b11bbeacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.037+0000 7f6edb7fe640 1 --2- 172.21.15.28:0/1266132268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbec980 0x55b11bbeacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.037+0000 7f6edb7fe640 1 -- 172.21.15.28:0/1266132268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbfa420 msgr2=0x55b11bbfc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.037+0000 7f6edb7fe640 1 --2- 172.21.15.28:0/1266132268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbfa420 0x55b11bbfc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.037+0000 7f6edb7fe640 1 -- 172.21.15.28:0/1266132268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b11bb08fb0 con 0x55b11bbeb290 2024-04-25T01:42:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.037+0000 7f6edb7fe640 1 --2- 172.21.15.28:0/1266132268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbeb290 0x55b11bbeb710 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc009930 tx=0x7f6ecc031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c326a30f2ec30ed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.037+0000 7f6edbfff640 1 --2- 172.21.15.28:0/1911606200 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6ec40a2d00 0x7f6ec40a51c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4009fd0 tx=0x7f6ed4009390 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.037+0000 7f6ec1ffb640 1 -- 172.21.15.28:0/1266132268 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc040070 con 0x55b11bbeb290 2024-04-25T01:42:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.037+0000 7f6ec1ffb640 1 -- 172.21.15.28:0/1266132268 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ecc005020 con 0x55b11bbeb290 2024-04-25T01:42:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.037+0000 7f6ec1ffb640 1 -- 172.21.15.28:0/1266132268 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc035040 con 0x55b11bbeb290 2024-04-25T01:42:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.038+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/1266132268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbeb290 msgr2=0x55b11bbeb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.038+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/1266132268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbeb290 0x55b11bbeb710 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc009930 tx=0x7f6ecc031630 comp rx=0 tx=0).stop 2024-04-25T01:42:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.038+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/1266132268 shutdown_connections 2024-04-25T01:42:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.038+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/1266132268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbfa420 0x55b11bbfc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.038+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/1266132268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbeb290 0x55b11bbeb710 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.038+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/1266132268 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbec980 0x55b11bbeacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.038+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/1266132268 >> 172.21.15.28:0/1266132268 conn(0x55b11bbb3950 msgr2=0x55b11bbe77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.038+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/1266132268 shutdown_connections 2024-04-25T01:42:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.038+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/1266132268 wait complete. 2024-04-25T01:42:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.038+0000 7f6ee4dbf740 1 -- start start 2024-04-25T01:42:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.038+0000 7f6ee4dbf740 1 Processor -- start 2024-04-25T01:42:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6ee4dbf740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbeb290 0x55b11bbf7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6ee4dbf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbec980 0x55b11bbf7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6ee4dbf740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbfa420 0x55b11bbf7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6ee4dbf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b11bbfd250 con 0x55b11bbec980 2024-04-25T01:42:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6ee4dbf740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b11bb0ae90 con 0x55b11bbeb290 2024-04-25T01:42:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6ee4dbf740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55b11bbfd3d0 con 0x55b11bbfa420 2024-04-25T01:42:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6ee4dbf740 10 client.0 _ll_register_callbacks cb 0x55b11bb46970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6edaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbec980 0x55b11bbf7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6edaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbec980 0x55b11bbf7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47718/0 (socket says 172.21.15.28:47718) 2024-04-25T01:42:38.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6edaffd640 1 -- 172.21.15.28:0/3752974504 learned_addr learned my addr 172.21.15.28:0/3752974504 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6edb7fe640 1 --2- 172.21.15.28:0/3752974504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbeb290 0x55b11bbf7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6edbfff640 1 --2- 172.21.15.28:0/3752974504 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbfa420 0x55b11bbf7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6edaffd640 1 -- 172.21.15.28:0/3752974504 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbfa420 msgr2=0x55b11bbf7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6edaffd640 1 --2- 172.21.15.28:0/3752974504 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbfa420 0x55b11bbf7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6edaffd640 1 -- 172.21.15.28:0/3752974504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbeb290 msgr2=0x55b11bbf7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6edaffd640 1 --2- 172.21.15.28:0/3752974504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbeb290 0x55b11bbf7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6edaffd640 1 -- 172.21.15.28:0/3752974504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b11b92b7e0 con 0x55b11bbec980 2024-04-25T01:42:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6edaffd640 1 --2- 172.21.15.28:0/3752974504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbec980 0x55b11bbf7780 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0097850 tx=0x7f6ed006b250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5c4b4ef937592f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6ebaffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6ebaffd640 1 -- 172.21.15.28:0/3752974504 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0020040 con 0x55b11bbec980 2024-04-25T01:42:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.039+0000 7f6ebaffd640 1 -- 172.21.15.28:0/3752974504 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f6ed00cc7f0 con 0x55b11bbec980 2024-04-25T01:42:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.040+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/3752974504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ecc005020 con 0x55b11bbec980 2024-04-25T01:42:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.040+0000 7f6ebaffd640 1 -- 172.21.15.28:0/3752974504 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6ed009b3f0 con 0x55b11bbec980 2024-04-25T01:42:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.040+0000 7f6ee4dbf740 10 client.19113 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.040+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/3752974504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b11b98bf70 con 0x55b11bbec980 2024-04-25T01:42:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.040+0000 7f6ebaffd640 1 -- 172.21.15.28:0/3752974504 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ecc005020 con 0x55b11bbec980 2024-04-25T01:42:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.040+0000 7f6ebaffd640 1 -- 172.21.15.28:0/3752974504 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6ed0097a60 con 0x55b11bbec980 2024-04-25T01:42:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.040+0000 7f6ee4dbf740 10 client.19113 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.040+0000 7f6ee4dbf740 10 client.19113 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.040+0000 7f6ee4dbf740 20 client.19113 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.040+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/3752974504 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b11bc0a3b0 0x55b11bc0c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.040+0000 7f6ee4dbf740 4 client.19113 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.040+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/3752974504 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b11bb235a0 con 0x55b11bc0a3b0 2024-04-25T01:42:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.040+0000 7f6edb7fe640 1 --2- 172.21.15.28:0/3752974504 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b11bc0a3b0 0x55b11bc0c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.041+0000 7f6edb7fe640 1 --2- 172.21.15.28:0/3752974504 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b11bc0a3b0 0x55b11bc0c7d0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dc11654483ec6a2 server_cookie=cf41720be64a808d in_seq=0 out_seq=0 2024-04-25T01:42:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.041+0000 7f6ebaffd640 10 client.19113 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.041+0000 7f6ebaffd640 1 -- 172.21.15.28:0/3752974504 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b11bb235a0 con 0x55b11bc0a3b0 2024-04-25T01:42:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.041+0000 7f6ebaffd640 10 client.19113 handle_command_reply: tid=0 2024-04-25T01:42:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.042+0000 7f6ee4dbf740 1 client.19113 shutdown 2024-04-25T01:42:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.042+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/3752974504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbec980 msgr2=0x55b11bbf7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.042+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/3752974504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbec980 0x55b11bbf7780 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0097850 tx=0x7f6ed006b250 comp rx=0 tx=0).stop 2024-04-25T01:42:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.042+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/3752974504 shutdown_connections 2024-04-25T01:42:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.042+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/3752974504 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b11bc0a3b0 0x55b11bc0c7d0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.042+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/3752974504 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbfa420 0x55b11bbf7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.042+0000 7f6ebaffd640 0 client.19113 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.042+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/3752974504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbec980 0x55b11bbf7780 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.042+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/3752974504 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbeb290 0x55b11bbf7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.042+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/3752974504 >> 172.21.15.28:0/3752974504 conn(0x55b11bbb3950 msgr2=0x55b11bbed660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.042+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/3752974504 shutdown_connections 2024-04-25T01:42:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.042+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/3752974504 wait complete. 2024-04-25T01:42:38.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.042+0000 7f6ee4dbf740 20 client.19113 trim_cache size 0 max 16384 2024-04-25T01:42:38.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.060+0000 7f6ee4dbf740 1 Processor -- start 2024-04-25T01:42:38.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.060+0000 7f6ee4dbf740 1 -- start start 2024-04-25T01:42:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.060+0000 7f6ee4dbf740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbeb290 0x55b11bc06150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.060+0000 7f6ee4dbf740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbec980 0x55b11bbf63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.060+0000 7f6ee4dbf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbfa420 0x55b11bbe67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.060+0000 7f6ee4dbf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc1ac940 con 0x55b11bbfa420 2024-04-25T01:42:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.060+0000 7f6ee4dbf740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc1ae480 con 0x55b11bbeb290 2024-04-25T01:42:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.060+0000 7f6ee4dbf740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc1ad010 con 0x55b11bbec980 2024-04-25T01:42:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.060+0000 7f6edbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbfa420 0x55b11bbe67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.060+0000 7f6edbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbfa420 0x55b11bbe67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47728/0 (socket says 172.21.15.28:47728) 2024-04-25T01:42:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.060+0000 7f6edbfff640 1 -- 172.21.15.28:0/4285148503 learned_addr learned my addr 172.21.15.28:0/4285148503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.060+0000 7f6edaffd640 1 --2- 172.21.15.28:0/4285148503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbec980 0x55b11bbf63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.060+0000 7f6edb7fe640 1 --2- 172.21.15.28:0/4285148503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbeb290 0x55b11bc06150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.060+0000 7f6edbfff640 1 -- 172.21.15.28:0/4285148503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbec980 msgr2=0x55b11bbf63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.060+0000 7f6edbfff640 1 --2- 172.21.15.28:0/4285148503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbec980 0x55b11bbf63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.060+0000 7f6edbfff640 1 -- 172.21.15.28:0/4285148503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbeb290 msgr2=0x55b11bc06150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.060+0000 7f6edbfff640 1 --2- 172.21.15.28:0/4285148503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbeb290 0x55b11bc06150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.060+0000 7f6edbfff640 1 -- 172.21.15.28:0/4285148503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b11bbf2e40 con 0x55b11bbfa420 2024-04-25T01:42:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.061+0000 7f6edbfff640 1 --2- 172.21.15.28:0/4285148503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbfa420 0x55b11bbe67d0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4008910 tx=0x7f6ed4009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e66cbbff3fa59334 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.061+0000 7f6ec17fa640 1 -- 172.21.15.28:0/4285148503 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4008040 con 0x55b11bbfa420 2024-04-25T01:42:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.061+0000 7f6ec17fa640 1 -- 172.21.15.28:0/4285148503 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ed4028070 con 0x55b11bbfa420 2024-04-25T01:42:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.061+0000 7f6ec17fa640 1 -- 172.21.15.28:0/4285148503 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4014040 con 0x55b11bbfa420 2024-04-25T01:42:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.061+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/4285148503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbfa420 msgr2=0x55b11bbe67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.061+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/4285148503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbfa420 0x55b11bbe67d0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4008910 tx=0x7f6ed4009650 comp rx=0 tx=0).stop 2024-04-25T01:42:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.061+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/4285148503 shutdown_connections 2024-04-25T01:42:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.061+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/4285148503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbfa420 0x55b11bbe67d0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.061+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/4285148503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbec980 0x55b11bbf63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.061+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/4285148503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbeb290 0x55b11bc06150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.061+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/4285148503 >> 172.21.15.28:0/4285148503 conn(0x55b11bbb3950 msgr2=0x55b11bbe60c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.062+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/4285148503 shutdown_connections 2024-04-25T01:42:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.062+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/4285148503 wait complete. 2024-04-25T01:42:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.062+0000 7f6ee4dbf740 1 -- start start 2024-04-25T01:42:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.062+0000 7f6ee4dbf740 1 Processor -- start 2024-04-25T01:42:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.062+0000 7f6ee4dbf740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbeb290 0x55b11bc02ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.062+0000 7f6ee4dbf740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbec980 0x55b11bbf34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.062+0000 7f6ee4dbf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbfa420 0x55b11bbf3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.062+0000 7f6ee4dbf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc1addb0 con 0x55b11bbfa420 2024-04-25T01:42:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.062+0000 7f6ee4dbf740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc1ad6e0 con 0x55b11bbec980 2024-04-25T01:42:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.062+0000 7f6ee4dbf740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc1aa940 con 0x55b11bbeb290 2024-04-25T01:42:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.062+0000 7f6ee4dbf740 10 client.0 _ll_register_callbacks cb 0x7f6ed009b1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.062+0000 7f6edbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbfa420 0x55b11bbf3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.062+0000 7f6edbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbfa420 0x55b11bbf3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47738/0 (socket says 172.21.15.28:47738) 2024-04-25T01:42:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.062+0000 7f6edbfff640 1 -- 172.21.15.28:0/371931611 learned_addr learned my addr 172.21.15.28:0/371931611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.063+0000 7f6edb7fe640 1 --2- 172.21.15.28:0/371931611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbeb290 0x55b11bc02ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.063+0000 7f6edaffd640 1 --2- 172.21.15.28:0/371931611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbec980 0x55b11bbf34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.063+0000 7f6edbfff640 1 -- 172.21.15.28:0/371931611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbeb290 msgr2=0x55b11bc02ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.063+0000 7f6edbfff640 1 --2- 172.21.15.28:0/371931611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbeb290 0x55b11bc02ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.063+0000 7f6edbfff640 1 -- 172.21.15.28:0/371931611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbec980 msgr2=0x55b11bbf34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.063+0000 7f6edbfff640 1 --2- 172.21.15.28:0/371931611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbec980 0x55b11bbf34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.063+0000 7f6edbfff640 1 -- 172.21.15.28:0/371931611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b11b823170 con 0x55b11bbfa420 2024-04-25T01:42:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.063+0000 7f6edbfff640 1 --2- 172.21.15.28:0/371931611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbfa420 0x55b11bbf3a10 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4009f10 tx=0x7f6ed4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59dd88fc57617b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.063+0000 7f6ebaffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.063+0000 7f6ebaffd640 1 -- 172.21.15.28:0/371931611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4032070 con 0x55b11bbfa420 2024-04-25T01:42:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.063+0000 7f6ebaffd640 1 -- 172.21.15.28:0/371931611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f6ed4023860 con 0x55b11bbfa420 2024-04-25T01:42:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.063+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/371931611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ed4028070 con 0x55b11bbfa420 2024-04-25T01:42:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.063+0000 7f6ebaffd640 1 -- 172.21.15.28:0/371931611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6ed403e050 con 0x55b11bbfa420 2024-04-25T01:42:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.063+0000 7f6ee4dbf740 10 client.19119 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.063+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/371931611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b11bc718d0 con 0x55b11bbfa420 2024-04-25T01:42:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.064+0000 7f6ebaffd640 1 -- 172.21.15.28:0/371931611 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ed4028070 con 0x55b11bbfa420 2024-04-25T01:42:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.064+0000 7f6ebaffd640 1 -- 172.21.15.28:0/371931611 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6ed402b380 con 0x55b11bbfa420 2024-04-25T01:42:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.064+0000 7f6ee4dbf740 10 client.19119 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.064+0000 7f6ee4dbf740 10 client.19119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.064+0000 7f6ee4dbf740 20 client.19119 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.064+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/371931611 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b11bc71ec0 0x55b11bc742e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.064+0000 7f6ee4dbf740 4 client.19119 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:42:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.064+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/371931611 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b11bb5c800 con 0x55b11bc71ec0 2024-04-25T01:42:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.064+0000 7f6edb7fe640 1 --2- 172.21.15.28:0/371931611 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b11bc71ec0 0x55b11bc742e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.064+0000 7f6edb7fe640 1 --2- 172.21.15.28:0/371931611 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b11bc71ec0 0x55b11bc742e0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=191616c98e0c0e21 server_cookie=cf725864fa4ca179 in_seq=0 out_seq=0 2024-04-25T01:42:38.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.064+0000 7f6ebaffd640 10 client.19119 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.065+0000 7f6ebaffd640 1 -- 172.21.15.28:0/371931611 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b11bb5c800 con 0x55b11bc71ec0 2024-04-25T01:42:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.065+0000 7f6ebaffd640 10 client.19119 handle_command_reply: tid=0 2024-04-25T01:42:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.065+0000 7f6ee4dbf740 1 client.19119 shutdown 2024-04-25T01:42:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.066+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/371931611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbfa420 msgr2=0x55b11bbf3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.066+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/371931611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbfa420 0x55b11bbf3a10 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4009f10 tx=0x7f6ed4004a40 comp rx=0 tx=0).stop 2024-04-25T01:42:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.066+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/371931611 shutdown_connections 2024-04-25T01:42:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.066+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/371931611 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55b11bc71ec0 0x55b11bc742e0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.066+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/371931611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b11bbfa420 0x55b11bbf3a10 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.066+0000 7f6ebaffd640 0 client.19119 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.066+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/371931611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b11bbec980 0x55b11bbf34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.066+0000 7f6ee4dbf740 1 --2- 172.21.15.28:0/371931611 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55b11bbeb290 0x55b11bc02ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.066+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/371931611 >> 172.21.15.28:0/371931611 conn(0x55b11bbb3950 msgr2=0x55b11bbe6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.066+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/371931611 shutdown_connections 2024-04-25T01:42:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.066+0000 7f6ee4dbf740 1 -- 172.21.15.28:0/371931611 wait complete. 2024-04-25T01:42:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.066+0000 7f6ee4dbf740 20 client.19119 trim_cache size 0 max 16384 2024-04-25T01:42:38.060 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:42:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:42:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "3b2d0e4c-794d-4920-96a2-89c15b6a5eb8", 2024-04-25T01:42:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:42:38.060 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:42:38.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.066+0000 7f6ee20b3640 1 -- 172.21.15.28:0/1911606200 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6ec40a2d00 msgr2=0x7f6ec40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.067+0000 7f6ee20b3640 1 --2- 172.21.15.28:0/1911606200 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6ec40a2d00 0x7f6ec40a51c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4009fd0 tx=0x7f6ed4009390 comp rx=0 tx=0).stop 2024-04-25T01:42:38.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.067+0000 7f6ee20b3640 1 -- 172.21.15.28:0/1911606200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6edc06a780 msgr2=0x7f6edc198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.067+0000 7f6ee20b3640 1 --2- 172.21.15.28:0/1911606200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6edc06a780 0x7f6edc198660 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000d770 tx=0x7f6ed000dc30 comp rx=0 tx=0).stop 2024-04-25T01:42:38.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.067+0000 7f6ee20b3640 1 -- 172.21.15.28:0/1911606200 shutdown_connections 2024-04-25T01:42:38.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.067+0000 7f6ee20b3640 1 --2- 172.21.15.28:0/1911606200 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6ec40a2d00 0x7f6ec40a51c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.067+0000 7f6ee20b3640 1 --2- 172.21.15.28:0/1911606200 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6edc107730 0x7f6edc19c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.067+0000 7f6ee20b3640 1 --2- 172.21.15.28:0/1911606200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6edc06a780 0x7f6edc198660 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.067+0000 7f6ee20b3640 1 --2- 172.21.15.28:0/1911606200 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6edc069dc0 0x7f6edc198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.067+0000 7f6ee20b3640 1 -- 172.21.15.28:0/1911606200 >> 172.21.15.28:0/1911606200 conn(0x7f6edc0776a0 msgr2=0x7f6edc0fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.067+0000 7f6ee20b3640 1 -- 172.21.15.28:0/1911606200 shutdown_connections 2024-04-25T01:42:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.067+0000 7f6ee20b3640 1 -- 172.21.15.28:0/1911606200 wait complete. 2024-04-25T01:42:38.066 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:42:38.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:42:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.183+0000 7f126a058640 1 -- 172.21.15.28:0/3233271386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1264101740 msgr2=0x7f1264101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.183+0000 7f126a058640 1 --2- 172.21.15.28:0/3233271386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1264101740 0x7f1264101bc0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f1258009930 tx=0x7f125802a420 comp rx=0 tx=0).stop 2024-04-25T01:42:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.183+0000 7f126a058640 1 -- 172.21.15.28:0/3233271386 shutdown_connections 2024-04-25T01:42:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.183+0000 7f126a058640 1 --2- 172.21.15.28:0/3233271386 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1264102100 0x7f126410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.183+0000 7f126a058640 1 --2- 172.21.15.28:0/3233271386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1264101740 0x7f1264101bc0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.183+0000 7f126a058640 1 --2- 172.21.15.28:0/3233271386 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1264107750 0x7f1264107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.183+0000 7f126a058640 1 -- 172.21.15.28:0/3233271386 >> 172.21.15.28:0/3233271386 conn(0x7f1264078bd0 msgr2=0x7f1264078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.183+0000 7f126a058640 1 -- 172.21.15.28:0/3233271386 shutdown_connections 2024-04-25T01:42:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.183+0000 7f126a058640 1 -- 172.21.15.28:0/3233271386 wait complete. 2024-04-25T01:42:38.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.183+0000 7f126a058640 1 Processor -- start 2024-04-25T01:42:38.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.184+0000 7f126a058640 1 -- start start 2024-04-25T01:42:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.184+0000 7f126a058640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1264101740 0x7f1264073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.184+0000 7f126a058640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1264102100 0x7f1264073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.184+0000 7f126a058640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1264107750 0x7f12640744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.184+0000 7f126a058640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1264111b60 con 0x7f1264102100 2024-04-25T01:42:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.184+0000 7f126a058640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f12641119e0 con 0x7f1264107750 2024-04-25T01:42:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.184+0000 7f126a058640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1264111ce0 con 0x7f1264101740 2024-04-25T01:42:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.184+0000 7f1262ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1264102100 0x7f1264073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.184+0000 7f1262ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1264102100 0x7f1264073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47754/0 (socket says 172.21.15.28:47754) 2024-04-25T01:42:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.184+0000 7f1262ffd640 1 -- 172.21.15.28:0/2429935459 learned_addr learned my addr 172.21.15.28:0/2429935459 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.184+0000 7f1263fff640 1 --2- 172.21.15.28:0/2429935459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1264107750 0x7f12640744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.184+0000 7f12637fe640 1 --2- 172.21.15.28:0/2429935459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1264101740 0x7f1264073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.184+0000 7f1262ffd640 1 -- 172.21.15.28:0/2429935459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1264101740 msgr2=0x7f1264073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.184+0000 7f1262ffd640 1 --2- 172.21.15.28:0/2429935459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1264101740 0x7f1264073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.184+0000 7f1262ffd640 1 -- 172.21.15.28:0/2429935459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1264107750 msgr2=0x7f12640744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.184+0000 7f1262ffd640 1 --2- 172.21.15.28:0/2429935459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1264107750 0x7f12640744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.184+0000 7f1262ffd640 1 -- 172.21.15.28:0/2429935459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12641a6580 con 0x7f1264102100 2024-04-25T01:42:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.185+0000 7f1262ffd640 1 --2- 172.21.15.28:0/2429935459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1264102100 0x7f1264073f80 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f1258009230 tx=0x7f1258004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.185+0000 7f1260ff9640 1 -- 172.21.15.28:0/2429935459 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125803f030 con 0x7f1264102100 2024-04-25T01:42:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.185+0000 7f1260ff9640 1 -- 172.21.15.28:0/2429935459 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1258003ad0 con 0x7f1264102100 2024-04-25T01:42:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.185+0000 7f126a058640 1 -- 172.21.15.28:0/2429935459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12641a6810 con 0x7f1264102100 2024-04-25T01:42:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.185+0000 7f1260ff9640 1 -- 172.21.15.28:0/2429935459 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258035070 con 0x7f1264102100 2024-04-25T01:42:38.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.185+0000 7f126a058640 1 -- 172.21.15.28:0/2429935459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12641a6d00 con 0x7f1264102100 2024-04-25T01:42:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.187+0000 7f1260ff9640 1 -- 172.21.15.28:0/2429935459 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1258042030 con 0x7f1264102100 2024-04-25T01:42:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.187+0000 7f126cd64740 1 Processor -- start 2024-04-25T01:42:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.187+0000 7f126cd64740 1 -- start start 2024-04-25T01:42:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f126cd64740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3085630 0x55c1e3085a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f1260ff9640 1 --2- 172.21.15.28:0/2429935459 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f124c0a2d00 0x7f124c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f126cd64740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3085f50 0x55c1e30900e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f126cd64740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3090620 0x55c1e3092a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f126cd64740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c1e2fea6b0 con 0x55c1e3090620 2024-04-25T01:42:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f126cd64740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c1e2f92880 con 0x55c1e3085f50 2024-04-25T01:42:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f126cd64740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c1e2fe6670 con 0x55c1e3085630 2024-04-25T01:42:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f12637fe640 1 --2- 172.21.15.28:0/2429935459 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f124c0a2d00 0x7f124c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f1263fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3090620 0x55c1e3092a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f1260ff9640 1 -- 172.21.15.28:0/2429935459 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f12580efc60 con 0x7f1264102100 2024-04-25T01:42:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f1263fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3090620 0x55c1e3092a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47766/0 (socket says 172.21.15.28:47766) 2024-04-25T01:42:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f1263fff640 1 -- 172.21.15.28:0/1931419945 learned_addr learned my addr 172.21.15.28:0/1931419945 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f1262ffd640 1 --2- 172.21.15.28:0/1931419945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3085f50 0x55c1e30900e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f1263fff640 1 --2- 172.21.15.28:0/1931419945 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3085630 0x55c1e3085a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f1263fff640 1 -- 172.21.15.28:0/1931419945 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3085630 msgr2=0x55c1e3085a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f1263fff640 1 --2- 172.21.15.28:0/1931419945 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3085630 0x55c1e3085a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f1263fff640 1 -- 172.21.15.28:0/1931419945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3085f50 msgr2=0x55c1e30900e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f1263fff640 1 --2- 172.21.15.28:0/1931419945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3085f50 0x55c1e30900e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f1263fff640 1 -- 172.21.15.28:0/1931419945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1e2dc87d0 con 0x55c1e3090620 2024-04-25T01:42:38.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f12637fe640 1 --2- 172.21.15.28:0/2429935459 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f124c0a2d00 0x7f124c0a51c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1254005fd0 tx=0x7f12540097d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.188+0000 7f1263fff640 1 --2- 172.21.15.28:0/1931419945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3090620 0x55c1e3092a10 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f125c011440 tx=0x7f125c009870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=172d3d535975ce10 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.189+0000 7f1249ffb640 1 -- 172.21.15.28:0/1931419945 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125c01e420 con 0x55c1e3090620 2024-04-25T01:42:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.189+0000 7f1249ffb640 1 -- 172.21.15.28:0/1931419945 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f125c00f8a0 con 0x55c1e3090620 2024-04-25T01:42:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.189+0000 7f1249ffb640 1 -- 172.21.15.28:0/1931419945 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125c00e780 con 0x55c1e3090620 2024-04-25T01:42:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.189+0000 7f126cd64740 1 -- 172.21.15.28:0/1931419945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3090620 msgr2=0x55c1e3092a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.189+0000 7f126cd64740 1 --2- 172.21.15.28:0/1931419945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3090620 0x55c1e3092a10 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f125c011440 tx=0x7f125c009870 comp rx=0 tx=0).stop 2024-04-25T01:42:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.189+0000 7f126cd64740 1 -- 172.21.15.28:0/1931419945 shutdown_connections 2024-04-25T01:42:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.189+0000 7f126cd64740 1 --2- 172.21.15.28:0/1931419945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3090620 0x55c1e3092a10 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.189+0000 7f126cd64740 1 --2- 172.21.15.28:0/1931419945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3085f50 0x55c1e30900e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.189+0000 7f126cd64740 1 --2- 172.21.15.28:0/1931419945 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3085630 0x55c1e3085a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.189+0000 7f126cd64740 1 -- 172.21.15.28:0/1931419945 >> 172.21.15.28:0/1931419945 conn(0x55c1e3081950 msgr2=0x55c1e3081d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.189+0000 7f126cd64740 1 -- 172.21.15.28:0/1931419945 shutdown_connections 2024-04-25T01:42:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.189+0000 7f126cd64740 1 -- 172.21.15.28:0/1931419945 wait complete. 2024-04-25T01:42:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.190+0000 7f126cd64740 1 -- start start 2024-04-25T01:42:38.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.190+0000 7f126cd64740 1 Processor -- start 2024-04-25T01:42:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.190+0000 7f126cd64740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3085630 0x55c1e308d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.190+0000 7f126cd64740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 0x55c1e308db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.190+0000 7f126cd64740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3090620 0x55c1e308e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.190+0000 7f126cd64740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c1e3093410 con 0x55c1e3085f50 2024-04-25T01:42:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.190+0000 7f126cd64740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c1e2fa5f50 con 0x55c1e3085630 2024-04-25T01:42:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.190+0000 7f126cd64740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55c1e3093590 con 0x55c1e3090620 2024-04-25T01:42:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.190+0000 7f126cd64740 10 client.0 _ll_register_callbacks cb 0x55c1e2fe1000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.190+0000 7f1262ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 0x55c1e308db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.190+0000 7f1262ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 0x55c1e308db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47770/0 (socket says 172.21.15.28:47770) 2024-04-25T01:42:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.190+0000 7f1262ffd640 1 -- 172.21.15.28:0/2796037505 learned_addr learned my addr 172.21.15.28:0/2796037505 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.190+0000 7f12637fe640 1 --2- 172.21.15.28:0/2796037505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3085630 0x55c1e308d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.190+0000 7f1263fff640 1 --2- 172.21.15.28:0/2796037505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3090620 0x55c1e308e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.191+0000 7f1262ffd640 1 -- 172.21.15.28:0/2796037505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3090620 msgr2=0x55c1e308e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.191+0000 7f1262ffd640 1 --2- 172.21.15.28:0/2796037505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3090620 0x55c1e308e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.191+0000 7f1262ffd640 1 -- 172.21.15.28:0/2796037505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3085630 msgr2=0x55c1e308d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.191+0000 7f1262ffd640 1 --2- 172.21.15.28:0/2796037505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3085630 0x55c1e308d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.191+0000 7f1262ffd640 1 -- 172.21.15.28:0/2796037505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1e2cfe3f0 con 0x55c1e3085f50 2024-04-25T01:42:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.191+0000 7f1262ffd640 1 --2- 172.21.15.28:0/2796037505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 0x55c1e308db30 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f1258009a50 tx=0x7f12580bb6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dcce5f416e80d67 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.191+0000 7f1242ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.191+0000 7f1242ffd640 1 -- 172.21.15.28:0/2796037505 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258049040 con 0x55c1e3085f50 2024-04-25T01:42:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.191+0000 7f1242ffd640 1 -- 172.21.15.28:0/2796037505 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f12580ed240 con 0x55c1e3085f50 2024-04-25T01:42:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.191+0000 7f126cd64740 1 -- 172.21.15.28:0/2796037505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f125c00f8a0 con 0x55c1e3085f50 2024-04-25T01:42:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.191+0000 7f1242ffd640 1 -- 172.21.15.28:0/2796037505 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f12580ed440 con 0x55c1e3085f50 2024-04-25T01:42:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.191+0000 7f126cd64740 10 client.19131 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.191+0000 7f126cd64740 1 -- 172.21.15.28:0/2796037505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c1e2cbe0f0 con 0x55c1e3085f50 2024-04-25T01:42:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.191+0000 7f1242ffd640 1 -- 172.21.15.28:0/2796037505 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f125c00f8a0 con 0x55c1e3085f50 2024-04-25T01:42:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.191+0000 7f1242ffd640 1 -- 172.21.15.28:0/2796037505 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f12580be030 con 0x55c1e3085f50 2024-04-25T01:42:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.191+0000 7f126cd64740 10 client.19131 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.191+0000 7f126cd64740 10 client.19131 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.191+0000 7f126cd64740 20 client.19131 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.192+0000 7f126cd64740 1 --2- 172.21.15.28:0/2796037505 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c1e30a0820 0x55c1e30a2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.192+0000 7f126cd64740 4 client.19131 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.192+0000 7f126cd64740 1 -- 172.21.15.28:0/2796037505 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1e2ff7bf0 con 0x55c1e30a0820 2024-04-25T01:42:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.192+0000 7f12637fe640 1 --2- 172.21.15.28:0/2796037505 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c1e30a0820 0x55c1e30a2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.192+0000 7f12637fe640 1 --2- 172.21.15.28:0/2796037505 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c1e30a0820 0x55c1e30a2ca0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c2ab6193f8870e3 server_cookie=7bd999c2a69702a7 in_seq=0 out_seq=0 2024-04-25T01:42:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.192+0000 7f1242ffd640 10 client.19131 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.193+0000 7f1242ffd640 1 -- 172.21.15.28:0/2796037505 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c1e2ff7bf0 con 0x55c1e30a0820 2024-04-25T01:42:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.193+0000 7f1242ffd640 10 client.19131 handle_command_reply: tid=0 2024-04-25T01:42:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.193+0000 7f126cd64740 1 client.19131 shutdown 2024-04-25T01:42:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.193+0000 7f126cd64740 1 -- 172.21.15.28:0/2796037505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 msgr2=0x55c1e308db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.193+0000 7f126cd64740 1 --2- 172.21.15.28:0/2796037505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 0x55c1e308db30 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f1258009a50 tx=0x7f12580bb6f0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.193+0000 7f126cd64740 1 -- 172.21.15.28:0/2796037505 shutdown_connections 2024-04-25T01:42:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.193+0000 7f126cd64740 1 --2- 172.21.15.28:0/2796037505 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c1e30a0820 0x55c1e30a2ca0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.193+0000 7f126cd64740 1 --2- 172.21.15.28:0/2796037505 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3090620 0x55c1e308e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.193+0000 7f126cd64740 1 --2- 172.21.15.28:0/2796037505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 0x55c1e308db30 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.193+0000 7f1242ffd640 0 client.19131 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.193+0000 7f126cd64740 1 --2- 172.21.15.28:0/2796037505 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3085630 0x55c1e308d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.193+0000 7f126cd64740 1 -- 172.21.15.28:0/2796037505 >> 172.21.15.28:0/2796037505 conn(0x55c1e3081950 msgr2=0x55c1e3087750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.193+0000 7f126cd64740 1 -- 172.21.15.28:0/2796037505 shutdown_connections 2024-04-25T01:42:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.193+0000 7f126cd64740 1 -- 172.21.15.28:0/2796037505 wait complete. 2024-04-25T01:42:38.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.193+0000 7f126cd64740 20 client.19131 trim_cache size 0 max 16384 2024-04-25T01:42:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f126cd64740 1 Processor -- start 2024-04-25T01:42:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f126cd64740 1 -- start start 2024-04-25T01:42:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f126cd64740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3085630 0x55c1e309c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f126cd64740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 0x55c1e308de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f126cd64740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3090620 0x55c1e3087d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f126cd64740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f12641b0df0 con 0x55c1e3085f50 2024-04-25T01:42:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f126cd64740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f12641b2930 con 0x55c1e3090620 2024-04-25T01:42:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f126cd64740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f12641b14c0 con 0x55c1e3085630 2024-04-25T01:42:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f1262ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 0x55c1e308de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f1262ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 0x55c1e308de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47774/0 (socket says 172.21.15.28:47774) 2024-04-25T01:42:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f1262ffd640 1 -- 172.21.15.28:0/2794498779 learned_addr learned my addr 172.21.15.28:0/2794498779 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f1263fff640 1 --2- 172.21.15.28:0/2794498779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3090620 0x55c1e3087d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f12637fe640 1 --2- 172.21.15.28:0/2794498779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3085630 0x55c1e309c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f1262ffd640 1 -- 172.21.15.28:0/2794498779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3085630 msgr2=0x55c1e309c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f1262ffd640 1 --2- 172.21.15.28:0/2794498779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3085630 0x55c1e309c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f1262ffd640 1 -- 172.21.15.28:0/2794498779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3090620 msgr2=0x55c1e3087d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f1262ffd640 1 --2- 172.21.15.28:0/2794498779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3090620 0x55c1e3087d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f1262ffd640 1 -- 172.21.15.28:0/2794498779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1e3092d80 con 0x55c1e3085f50 2024-04-25T01:42:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f12637fe640 1 --2- 172.21.15.28:0/2794498779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3085630 0x55c1e309c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.211+0000 7f1262ffd640 1 --2- 172.21.15.28:0/2794498779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 0x55c1e308de40 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f12580b8760 tx=0x7f12580ba2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=746c78b7a9228b35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.212+0000 7f12497fa640 1 -- 172.21.15.28:0/2794498779 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12580bbb60 con 0x55c1e3085f50 2024-04-25T01:42:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.212+0000 7f12497fa640 1 -- 172.21.15.28:0/2794498779 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12580bbd90 con 0x55c1e3085f50 2024-04-25T01:42:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.212+0000 7f12497fa640 1 -- 172.21.15.28:0/2794498779 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258031e30 con 0x55c1e3085f50 2024-04-25T01:42:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.212+0000 7f126cd64740 1 -- 172.21.15.28:0/2794498779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 msgr2=0x55c1e308de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.212+0000 7f126cd64740 1 --2- 172.21.15.28:0/2794498779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 0x55c1e308de40 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f12580b8760 tx=0x7f12580ba2a0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.212+0000 7f126cd64740 1 -- 172.21.15.28:0/2794498779 shutdown_connections 2024-04-25T01:42:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.212+0000 7f126cd64740 1 --2- 172.21.15.28:0/2794498779 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3090620 0x55c1e3087d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.212+0000 7f126cd64740 1 --2- 172.21.15.28:0/2794498779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 0x55c1e308de40 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.212+0000 7f126cd64740 1 --2- 172.21.15.28:0/2794498779 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3085630 0x55c1e309c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.212+0000 7f126cd64740 1 -- 172.21.15.28:0/2794498779 >> 172.21.15.28:0/2794498779 conn(0x55c1e3081950 msgr2=0x55c1e308ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.212+0000 7f126cd64740 1 -- 172.21.15.28:0/2794498779 shutdown_connections 2024-04-25T01:42:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.212+0000 7f126cd64740 1 -- 172.21.15.28:0/2794498779 wait complete. 2024-04-25T01:42:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.213+0000 7f126cd64740 1 -- start start 2024-04-25T01:42:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.213+0000 7f126cd64740 1 Processor -- start 2024-04-25T01:42:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.213+0000 7f126cd64740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3085630 0x55c1e308a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.213+0000 7f126cd64740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 0x55c1e308aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.213+0000 7f126cd64740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3090620 0x55c1e3105e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.213+0000 7f126cd64740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f12641b2260 con 0x55c1e3085f50 2024-04-25T01:42:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.213+0000 7f126cd64740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f12641b1b90 con 0x55c1e3085630 2024-04-25T01:42:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.213+0000 7f126cd64740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f12641aedf0 con 0x55c1e3090620 2024-04-25T01:42:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.213+0000 7f126cd64740 10 client.0 _ll_register_callbacks cb 0x7f1258009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.213+0000 7f1262ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 0x55c1e308aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.213+0000 7f1262ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 0x55c1e308aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47790/0 (socket says 172.21.15.28:47790) 2024-04-25T01:42:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.213+0000 7f1262ffd640 1 -- 172.21.15.28:0/2387653141 learned_addr learned my addr 172.21.15.28:0/2387653141 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.213+0000 7f1263fff640 1 --2- 172.21.15.28:0/2387653141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3090620 0x55c1e3105e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.213+0000 7f12637fe640 1 --2- 172.21.15.28:0/2387653141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3085630 0x55c1e308a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.213+0000 7f1262ffd640 1 -- 172.21.15.28:0/2387653141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3090620 msgr2=0x55c1e3105e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.213+0000 7f1262ffd640 1 --2- 172.21.15.28:0/2387653141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3090620 0x55c1e3105e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.213+0000 7f1262ffd640 1 -- 172.21.15.28:0/2387653141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3085630 msgr2=0x55c1e308a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.213+0000 7f1262ffd640 1 --2- 172.21.15.28:0/2387653141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3085630 0x55c1e308a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.213+0000 7f1262ffd640 1 -- 172.21.15.28:0/2387653141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1e3106360 con 0x55c1e3085f50 2024-04-25T01:42:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.214+0000 7f1262ffd640 1 --2- 172.21.15.28:0/2387653141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 0x55c1e308aee0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f12580bbd20 tx=0x7f12580cbaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d966abc66197aa5e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.214+0000 7f1242ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.214+0000 7f1242ffd640 1 -- 172.21.15.28:0/2387653141 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12580c4a00 con 0x55c1e3085f50 2024-04-25T01:42:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.214+0000 7f1242ffd640 1 -- 172.21.15.28:0/2387653141 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f1258031de0 con 0x55c1e3085f50 2024-04-25T01:42:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.214+0000 7f126cd64740 1 -- 172.21.15.28:0/2387653141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12580bbd90 con 0x55c1e3085f50 2024-04-25T01:42:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.214+0000 7f1242ffd640 1 -- 172.21.15.28:0/2387653141 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f12580c4570 con 0x55c1e3085f50 2024-04-25T01:42:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.214+0000 7f126cd64740 10 client.19137 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.214+0000 7f126cd64740 1 -- 172.21.15.28:0/2387653141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c1e3106360 con 0x55c1e3085f50 2024-04-25T01:42:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.214+0000 7f1242ffd640 1 -- 172.21.15.28:0/2387653141 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12580bbd90 con 0x55c1e3085f50 2024-04-25T01:42:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.214+0000 7f1242ffd640 1 -- 172.21.15.28:0/2387653141 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f125803bbc0 con 0x55c1e3085f50 2024-04-25T01:42:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.214+0000 7f126cd64740 10 client.19137 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.214+0000 7f126cd64740 10 client.19137 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.214+0000 7f126cd64740 20 client.19137 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.214+0000 7f126cd64740 1 --2- 172.21.15.28:0/2387653141 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c1e31077a0 0x55c1e3109b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.214+0000 7f126cd64740 4 client.19137 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:42:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.214+0000 7f126cd64740 1 -- 172.21.15.28:0/2387653141 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c1e2ff1f40 con 0x55c1e31077a0 2024-04-25T01:42:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.214+0000 7f12637fe640 1 --2- 172.21.15.28:0/2387653141 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c1e31077a0 0x55c1e3109b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.215+0000 7f12637fe640 1 --2- 172.21.15.28:0/2387653141 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c1e31077a0 0x55c1e3109b90 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=656576b23b1315b4 server_cookie=488776144895aec6 in_seq=0 out_seq=0 2024-04-25T01:42:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.215+0000 7f1242ffd640 10 client.19137 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.215+0000 7f1242ffd640 1 -- 172.21.15.28:0/2387653141 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c1e2ff1f40 con 0x55c1e31077a0 2024-04-25T01:42:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.215+0000 7f1242ffd640 10 client.19137 handle_command_reply: tid=0 2024-04-25T01:42:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.215+0000 7f126cd64740 1 client.19137 shutdown 2024-04-25T01:42:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.215+0000 7f126cd64740 1 -- 172.21.15.28:0/2387653141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 msgr2=0x55c1e308aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.215+0000 7f126cd64740 1 --2- 172.21.15.28:0/2387653141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 0x55c1e308aee0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f12580bbd20 tx=0x7f12580cbaf0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.215+0000 7f126cd64740 1 -- 172.21.15.28:0/2387653141 shutdown_connections 2024-04-25T01:42:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.215+0000 7f126cd64740 1 --2- 172.21.15.28:0/2387653141 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55c1e31077a0 0x55c1e3109b90 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.215+0000 7f126cd64740 1 --2- 172.21.15.28:0/2387653141 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55c1e3090620 0x55c1e3105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.215+0000 7f126cd64740 1 --2- 172.21.15.28:0/2387653141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1e3085f50 0x55c1e308aee0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.215+0000 7f126cd64740 1 --2- 172.21.15.28:0/2387653141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c1e3085630 0x55c1e308a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.215+0000 7f126cd64740 1 -- 172.21.15.28:0/2387653141 >> 172.21.15.28:0/2387653141 conn(0x55c1e3081950 msgr2=0x55c1e308dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.215+0000 7f1242ffd640 0 client.19137 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.215+0000 7f126cd64740 1 -- 172.21.15.28:0/2387653141 shutdown_connections 2024-04-25T01:42:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.215+0000 7f126cd64740 1 -- 172.21.15.28:0/2387653141 wait complete. 2024-04-25T01:42:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.215+0000 7f126cd64740 20 client.19137 trim_cache size 0 max 16384 2024-04-25T01:42:38.211 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:42:38.211 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:42:38.211 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:42:38.211 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:42:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.216+0000 7f126a058640 1 -- 172.21.15.28:0/2429935459 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f124c0a2d00 msgr2=0x7f124c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.216+0000 7f126a058640 1 --2- 172.21.15.28:0/2429935459 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f124c0a2d00 0x7f124c0a51c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1254005fd0 tx=0x7f12540097d0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.216+0000 7f126a058640 1 -- 172.21.15.28:0/2429935459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1264102100 msgr2=0x7f1264073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.216+0000 7f126a058640 1 --2- 172.21.15.28:0/2429935459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1264102100 0x7f1264073f80 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f1258009230 tx=0x7f1258004270 comp rx=0 tx=0).stop 2024-04-25T01:42:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.216+0000 7f126a058640 1 -- 172.21.15.28:0/2429935459 shutdown_connections 2024-04-25T01:42:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.216+0000 7f126a058640 1 --2- 172.21.15.28:0/2429935459 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f124c0a2d00 0x7f124c0a51c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.216+0000 7f126a058640 1 --2- 172.21.15.28:0/2429935459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1264107750 0x7f12640744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.216+0000 7f126a058640 1 --2- 172.21.15.28:0/2429935459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1264102100 0x7f1264073f80 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.216+0000 7f126a058640 1 --2- 172.21.15.28:0/2429935459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1264101740 0x7f1264073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.216+0000 7f126a058640 1 -- 172.21.15.28:0/2429935459 >> 172.21.15.28:0/2429935459 conn(0x7f1264078bd0 msgr2=0x7f12640ff640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.216+0000 7f126a058640 1 -- 172.21.15.28:0/2429935459 shutdown_connections 2024-04-25T01:42:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.216+0000 7f126a058640 1 -- 172.21.15.28:0/2429935459 wait complete. 2024-04-25T01:42:38.214 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:37 smithi028 ceph-mon[29390]: pgmap v297: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:42:38.215 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:42:38.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:42:38.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:37 smithi032 ceph-mon[32461]: pgmap v297: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:42:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.328+0000 7f1c830a5640 1 -- 172.21.15.28:0/1692172211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c7c073af0 msgr2=0x7f1c7c073ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.328+0000 7f1c830a5640 1 --2- 172.21.15.28:0/1692172211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c7c073af0 0x7f1c7c073ed0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f1c74009900 tx=0x7f1c7402a480 comp rx=0 tx=0).stop 2024-04-25T01:42:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.329+0000 7f1c830a5640 1 -- 172.21.15.28:0/1692172211 shutdown_connections 2024-04-25T01:42:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.329+0000 7f1c830a5640 1 --2- 172.21.15.28:0/1692172211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c7c10dff0 0x7f1c7c1103e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.329+0000 7f1c830a5640 1 --2- 172.21.15.28:0/1692172211 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c7c074410 0x7f1c7c10dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.329+0000 7f1c830a5640 1 --2- 172.21.15.28:0/1692172211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c7c073af0 0x7f1c7c073ed0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.329+0000 7f1c830a5640 1 -- 172.21.15.28:0/1692172211 >> 172.21.15.28:0/1692172211 conn(0x7f1c7c0fd1e0 msgr2=0x7f1c7c0ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.329+0000 7f1c830a5640 1 -- 172.21.15.28:0/1692172211 shutdown_connections 2024-04-25T01:42:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.329+0000 7f1c830a5640 1 -- 172.21.15.28:0/1692172211 wait complete. 2024-04-25T01:42:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.329+0000 7f1c830a5640 1 Processor -- start 2024-04-25T01:42:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.329+0000 7f1c830a5640 1 -- start start 2024-04-25T01:42:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.329+0000 7f1c830a5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c7c073af0 0x7f1c7c19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.329+0000 7f1c830a5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c7c074410 0x7f1c7c19a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.329+0000 7f1c830a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c7c10dff0 0x7f1c7c19eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.329+0000 7f1c830a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7c113d10 con 0x7f1c7c10dff0 2024-04-25T01:42:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.329+0000 7f1c830a5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7c113b90 con 0x7f1c7c073af0 2024-04-25T01:42:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.329+0000 7f1c830a5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7c113e90 con 0x7f1c7c074410 2024-04-25T01:42:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.330+0000 7f1c81642640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c7c10dff0 0x7f1c7c19eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.330+0000 7f1c81642640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c7c10dff0 0x7f1c7c19eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47798/0 (socket says 172.21.15.28:47798) 2024-04-25T01:42:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.330+0000 7f1c81642640 1 -- 172.21.15.28:0/2937879858 learned_addr learned my addr 172.21.15.28:0/2937879858 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.330+0000 7f1c80e41640 1 --2- 172.21.15.28:0/2937879858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c7c073af0 0x7f1c7c19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.330+0000 7f1c7bfff640 1 --2- 172.21.15.28:0/2937879858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c7c074410 0x7f1c7c19a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.330+0000 7f1c81642640 1 -- 172.21.15.28:0/2937879858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c7c074410 msgr2=0x7f1c7c19a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.330+0000 7f1c81642640 1 --2- 172.21.15.28:0/2937879858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c7c074410 0x7f1c7c19a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.330+0000 7f1c81642640 1 -- 172.21.15.28:0/2937879858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c7c073af0 msgr2=0x7f1c7c19a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.330+0000 7f1c81642640 1 --2- 172.21.15.28:0/2937879858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c7c073af0 0x7f1c7c19a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.330+0000 7f1c81642640 1 -- 172.21.15.28:0/2937879858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c7c19f260 con 0x7f1c7c10dff0 2024-04-25T01:42:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.330+0000 7f1c7bfff640 1 --2- 172.21.15.28:0/2937879858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c7c074410 0x7f1c7c19a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.330+0000 7f1c81642640 1 --2- 172.21.15.28:0/2937879858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c7c10dff0 0x7f1c7c19eb30 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f1c6400d750 tx=0x7f1c6400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.330+0000 7f1c79ffb640 1 -- 172.21.15.28:0/2937879858 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c64017070 con 0x7f1c7c10dff0 2024-04-25T01:42:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.330+0000 7f1c79ffb640 1 -- 172.21.15.28:0/2937879858 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c64005500 con 0x7f1c7c10dff0 2024-04-25T01:42:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.330+0000 7f1c830a5640 1 -- 172.21.15.28:0/2937879858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c7c19f550 con 0x7f1c7c10dff0 2024-04-25T01:42:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.330+0000 7f1c79ffb640 1 -- 172.21.15.28:0/2937879858 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c64004d90 con 0x7f1c7c10dff0 2024-04-25T01:42:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.331+0000 7f1c830a5640 1 -- 172.21.15.28:0/2937879858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c7c1a6de0 con 0x7f1c7c10dff0 2024-04-25T01:42:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.331+0000 7f1c830a5640 1 -- 172.21.15.28:0/2937879858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c7c068560 con 0x7f1c7c10dff0 2024-04-25T01:42:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.332+0000 7f1c79ffb640 1 -- 172.21.15.28:0/2937879858 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1c64004f30 con 0x7f1c7c10dff0 2024-04-25T01:42:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.333+0000 7f1c79ffb640 1 --2- 172.21.15.28:0/2937879858 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1c680a2d00 0x7f1c680a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.333+0000 7f1c79ffb640 1 -- 172.21.15.28:0/2937879858 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f1c640cf1b0 con 0x7f1c7c10dff0 2024-04-25T01:42:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.333+0000 7f1c80e41640 1 --2- 172.21.15.28:0/2937879858 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1c680a2d00 0x7f1c680a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.334+0000 7f1c80e41640 1 --2- 172.21.15.28:0/2937879858 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1c680a2d00 0x7f1c680a51c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1c74009230 tx=0x7f1c740058d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.338+0000 7f1c79ffb640 1 -- 172.21.15.28:0/2937879858 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1c6409b9e0 con 0x7f1c7c10dff0 2024-04-25T01:42:38.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:37 smithi130 ceph-mon[32702]: pgmap v297: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:42:38.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.565+0000 7f1c830a5640 1 -- 172.21.15.28:0/2937879858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c7c1091d0 con 0x7f1c7c10dff0 2024-04-25T01:42:38.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.566+0000 7f1c79ffb640 1 -- 172.21.15.28:0/2937879858 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7f1c6409b2e0 con 0x7f1c7c10dff0 2024-04-25T01:42:38.554 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:42:38.554 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:42:38.555 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:42:38.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.568+0000 7f1c830a5640 1 -- 172.21.15.28:0/2937879858 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1c680a2d00 msgr2=0x7f1c680a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.568+0000 7f1c830a5640 1 --2- 172.21.15.28:0/2937879858 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1c680a2d00 0x7f1c680a51c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1c74009230 tx=0x7f1c740058d0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.568+0000 7f1c830a5640 1 -- 172.21.15.28:0/2937879858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c7c10dff0 msgr2=0x7f1c7c19eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.568+0000 7f1c830a5640 1 --2- 172.21.15.28:0/2937879858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c7c10dff0 0x7f1c7c19eb30 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f1c6400d750 tx=0x7f1c6400dc10 comp rx=0 tx=0).stop 2024-04-25T01:42:38.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.568+0000 7f1c830a5640 1 -- 172.21.15.28:0/2937879858 shutdown_connections 2024-04-25T01:42:38.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.568+0000 7f1c830a5640 1 --2- 172.21.15.28:0/2937879858 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1c680a2d00 0x7f1c680a51c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.568+0000 7f1c830a5640 1 --2- 172.21.15.28:0/2937879858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c7c10dff0 0x7f1c7c19eb30 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.568+0000 7f1c830a5640 1 --2- 172.21.15.28:0/2937879858 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1c7c074410 0x7f1c7c19a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.568+0000 7f1c830a5640 1 --2- 172.21.15.28:0/2937879858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c7c073af0 0x7f1c7c19a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.568+0000 7f1c830a5640 1 -- 172.21.15.28:0/2937879858 >> 172.21.15.28:0/2937879858 conn(0x7f1c7c0fd1e0 msgr2=0x7f1c7c0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.568+0000 7f1c830a5640 1 -- 172.21.15.28:0/2937879858 shutdown_connections 2024-04-25T01:42:38.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.568+0000 7f1c830a5640 1 -- 172.21.15.28:0/2937879858 wait complete. 2024-04-25T01:42:38.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:42:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.683+0000 7fe0dce38640 1 -- 172.21.15.28:0/928937621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0d8069dc0 msgr2=0x7fe0d806a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.683+0000 7fe0dce38640 1 --2- 172.21.15.28:0/928937621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0d8069dc0 0x7fe0d806a240 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc009930 tx=0x7fe0cc02a420 comp rx=0 tx=0).stop 2024-04-25T01:42:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.683+0000 7fe0dce38640 1 -- 172.21.15.28:0/928937621 shutdown_connections 2024-04-25T01:42:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.683+0000 7fe0dce38640 1 --2- 172.21.15.28:0/928937621 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe0d806a780 0x7fe0d810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.683+0000 7fe0dce38640 1 --2- 172.21.15.28:0/928937621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0d8069dc0 0x7fe0d806a240 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.683+0000 7fe0dce38640 1 --2- 172.21.15.28:0/928937621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d8107730 0x7fe0d8107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.683+0000 7fe0dce38640 1 -- 172.21.15.28:0/928937621 >> 172.21.15.28:0/928937621 conn(0x7fe0d80776a0 msgr2=0x7fe0d8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.683+0000 7fe0dce38640 1 -- 172.21.15.28:0/928937621 shutdown_connections 2024-04-25T01:42:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.683+0000 7fe0dce38640 1 -- 172.21.15.28:0/928937621 wait complete. 2024-04-25T01:42:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.684+0000 7fe0dce38640 1 Processor -- start 2024-04-25T01:42:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.684+0000 7fe0dce38640 1 -- start start 2024-04-25T01:42:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.684+0000 7fe0dce38640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d8069dc0 0x7fe0d8100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.684+0000 7fe0dce38640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe0d806a780 0x7fe0d8100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.684+0000 7fe0dce38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0d8107730 0x7fe0d81010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.684+0000 7fe0dce38640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0d8111b90 con 0x7fe0d8107730 2024-04-25T01:42:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.684+0000 7fe0dce38640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe0d8111a10 con 0x7fe0d8069dc0 2024-04-25T01:42:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.684+0000 7fe0dce38640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe0d8111d10 con 0x7fe0d806a780 2024-04-25T01:42:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.684+0000 7fe0d6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0d8107730 0x7fe0d81010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.685+0000 7fe0d6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0d8107730 0x7fe0d81010f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47810/0 (socket says 172.21.15.28:47810) 2024-04-25T01:42:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.685+0000 7fe0d6d9d640 1 -- 172.21.15.28:0/305863061 learned_addr learned my addr 172.21.15.28:0/305863061 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.685+0000 7fe0d659c640 1 --2- 172.21.15.28:0/305863061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d8069dc0 0x7fe0d8100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.685+0000 7fe0d5d9b640 1 --2- 172.21.15.28:0/305863061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe0d806a780 0x7fe0d8100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.685+0000 7fe0d6d9d640 1 -- 172.21.15.28:0/305863061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe0d806a780 msgr2=0x7fe0d8100bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.685+0000 7fe0d6d9d640 1 --2- 172.21.15.28:0/305863061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe0d806a780 0x7fe0d8100bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.685+0000 7fe0d6d9d640 1 -- 172.21.15.28:0/305863061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d8069dc0 msgr2=0x7fe0d8100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.685+0000 7fe0d6d9d640 1 --2- 172.21.15.28:0/305863061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d8069dc0 0x7fe0d8100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.685+0000 7fe0d6d9d640 1 -- 172.21.15.28:0/305863061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0d81a6430 con 0x7fe0d8107730 2024-04-25T01:42:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.685+0000 7fe0d6d9d640 1 --2- 172.21.15.28:0/305863061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0d8107730 0x7fe0d81010f0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fe0d000c970 tx=0x7fe0d000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.685+0000 7fe0c77fe640 1 -- 172.21.15.28:0/305863061 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d0007bb0 con 0x7fe0d8107730 2024-04-25T01:42:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.685+0000 7fe0c77fe640 1 -- 172.21.15.28:0/305863061 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0d0007d50 con 0x7fe0d8107730 2024-04-25T01:42:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.685+0000 7fe0dce38640 1 -- 172.21.15.28:0/305863061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0d81a6720 con 0x7fe0d8107730 2024-04-25T01:42:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.685+0000 7fe0c77fe640 1 -- 172.21.15.28:0/305863061 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d0018800 con 0x7fe0d8107730 2024-04-25T01:42:38.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.685+0000 7fe0dce38640 1 -- 172.21.15.28:0/305863061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0d81a6bf0 con 0x7fe0d8107730 2024-04-25T01:42:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.687+0000 7fe0c77fe640 1 -- 172.21.15.28:0/305863061 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe0d000fa50 con 0x7fe0d8107730 2024-04-25T01:42:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.688+0000 7fe0dfc27740 1 Processor -- start 2024-04-25T01:42:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.688+0000 7fe0dfc27740 1 -- start start 2024-04-25T01:42:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.688+0000 7fe0dfc27740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e92094580 0x564e92094960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.688+0000 7fe0dfc27740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e92094ea0 0x564e9209f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.688+0000 7fe0dfc27740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9209f570 0x564e920a1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.688+0000 7fe0dfc27740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564e91ff9670 con 0x564e9209f570 2024-04-25T01:42:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.688+0000 7fe0dfc27740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564e91fa1970 con 0x564e92094ea0 2024-04-25T01:42:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.688+0000 7fe0dfc27740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x564e91ff5630 con 0x564e92094580 2024-04-25T01:42:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.688+0000 7fe0c77fe640 1 --2- 172.21.15.28:0/305863061 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe0c00a2d30 0x7fe0c00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.688+0000 7fe0d5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9209f570 0x564e920a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.689+0000 7fe0d659c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e92094ea0 0x564e9209f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.689+0000 7fe0d5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9209f570 0x564e920a1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47818/0 (socket says 172.21.15.28:47818) 2024-04-25T01:42:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.689+0000 7fe0d5d9b640 1 -- 172.21.15.28:0/4215783113 learned_addr learned my addr 172.21.15.28:0/4215783113 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.689+0000 7fe0d6d9d640 1 --2- 172.21.15.28:0/305863061 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe0c00a2d30 0x7fe0c00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.689+0000 7fe0d5d9b640 1 --2- 172.21.15.28:0/4215783113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e92094580 0x564e92094960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.689+0000 7fe0c77fe640 1 -- 172.21.15.28:0/305863061 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7fe0d00cf850 con 0x7fe0d8107730 2024-04-25T01:42:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.689+0000 7fe0d659c640 1 -- 172.21.15.28:0/4215783113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e92094580 msgr2=0x564e92094960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.689+0000 7fe0d659c640 1 --2- 172.21.15.28:0/4215783113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e92094580 0x564e92094960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.689+0000 7fe0d659c640 1 -- 172.21.15.28:0/4215783113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9209f570 msgr2=0x564e920a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.689+0000 7fe0d659c640 1 --2- 172.21.15.28:0/4215783113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9209f570 0x564e920a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.689+0000 7fe0d659c640 1 -- 172.21.15.28:0/4215783113 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e91fc5400 con 0x564e92094ea0 2024-04-25T01:42:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.689+0000 7fe0d5d9b640 1 --2- 172.21.15.28:0/4215783113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9209f570 0x564e920a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:42:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.689+0000 7fe0d6d9d640 1 --2- 172.21.15.28:0/305863061 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe0c00a2d30 0x7fe0c00a51f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe0d00a40c0 tx=0x7fe0d006d940 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.689+0000 7fe0d659c640 1 --2- 172.21.15.28:0/4215783113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e92094ea0 0x564e9209f030 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8009f30 tx=0x7fe0c8010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42b05bf9060c4bfb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.689+0000 7fe0c4ff9640 1 -- 172.21.15.28:0/4215783113 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c8003c30 con 0x564e92094ea0 2024-04-25T01:42:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.689+0000 7fe0c4ff9640 1 -- 172.21.15.28:0/4215783113 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0c8015070 con 0x564e92094ea0 2024-04-25T01:42:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.689+0000 7fe0c4ff9640 1 -- 172.21.15.28:0/4215783113 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c8011880 con 0x564e92094ea0 2024-04-25T01:42:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.690+0000 7fe0dfc27740 1 -- 172.21.15.28:0/4215783113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e92094ea0 msgr2=0x564e9209f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.690+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/4215783113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e92094ea0 0x564e9209f030 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8009f30 tx=0x7fe0c8010860 comp rx=0 tx=0).stop 2024-04-25T01:42:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.690+0000 7fe0dfc27740 1 -- 172.21.15.28:0/4215783113 shutdown_connections 2024-04-25T01:42:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.690+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/4215783113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9209f570 0x564e920a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.690+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/4215783113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e92094ea0 0x564e9209f030 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.690+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/4215783113 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e92094580 0x564e92094960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.690+0000 7fe0dfc27740 1 -- 172.21.15.28:0/4215783113 >> 172.21.15.28:0/4215783113 conn(0x564e920908a0 msgr2=0x564e92090c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.690+0000 7fe0dfc27740 1 -- 172.21.15.28:0/4215783113 shutdown_connections 2024-04-25T01:42:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.690+0000 7fe0dfc27740 1 -- 172.21.15.28:0/4215783113 wait complete. 2024-04-25T01:42:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.690+0000 7fe0dfc27740 1 -- start start 2024-04-25T01:42:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0dfc27740 1 Processor -- start 2024-04-25T01:42:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0dfc27740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e92094580 0x564e9209ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0dfc27740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e92094ea0 0x564e9209c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0dfc27740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e9209f570 0x564e9209c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0dfc27740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564e920a2360 con 0x564e92094580 2024-04-25T01:42:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0dfc27740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564e91fb5050 con 0x564e92094ea0 2024-04-25T01:42:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0dfc27740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x564e920a24e0 con 0x564e9209f570 2024-04-25T01:42:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0d5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e92094580 0x564e9209ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0dfc27740 10 client.0 _ll_register_callbacks cb 0x564e91d0cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0d5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e92094580 0x564e9209ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47822/0 (socket says 172.21.15.28:47822) 2024-04-25T01:42:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0d5d9b640 1 -- 172.21.15.28:0/3244452380 learned_addr learned my addr 172.21.15.28:0/3244452380 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0d659c640 1 --2- 172.21.15.28:0/3244452380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e92094ea0 0x564e9209c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0d5d9b640 1 --2- 172.21.15.28:0/3244452380 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e9209f570 0x564e9209c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0d5d9b640 1 -- 172.21.15.28:0/3244452380 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e9209f570 msgr2=0x564e9209c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0d5d9b640 1 --2- 172.21.15.28:0/3244452380 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e9209f570 0x564e9209c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0d5d9b640 1 -- 172.21.15.28:0/3244452380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e92094ea0 msgr2=0x564e9209c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0d5d9b640 1 --2- 172.21.15.28:0/3244452380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e92094ea0 0x564e9209c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0d5d9b640 1 -- 172.21.15.28:0/3244452380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e91f86dc0 con 0x564e92094580 2024-04-25T01:42:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0d5d9b640 1 --2- 172.21.15.28:0/3244452380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e92094580 0x564e9209ef10 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc009230 tx=0x7fe0cc040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b8d25b73156b08 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.691+0000 7fe0bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.692+0000 7fe0bdffb640 1 -- 172.21.15.28:0/3244452380 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc035040 con 0x564e92094580 2024-04-25T01:42:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.692+0000 7fe0bdffb640 1 -- 172.21.15.28:0/3244452380 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7fe0cc004b00 con 0x564e92094580 2024-04-25T01:42:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.692+0000 7fe0dfc27740 1 -- 172.21.15.28:0/3244452380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0c8015070 con 0x564e92094580 2024-04-25T01:42:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.692+0000 7fe0bdffb640 1 -- 172.21.15.28:0/3244452380 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe0cc004d00 con 0x564e92094580 2024-04-25T01:42:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.692+0000 7fe0dfc27740 10 client.19155 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.692+0000 7fe0dfc27740 1 -- 172.21.15.28:0/3244452380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564e91d0d3f0 con 0x564e92094580 2024-04-25T01:42:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.692+0000 7fe0bdffb640 1 -- 172.21.15.28:0/3244452380 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0c8015070 con 0x564e92094580 2024-04-25T01:42:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.692+0000 7fe0bdffb640 1 -- 172.21.15.28:0/3244452380 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe0cc03a040 con 0x564e92094580 2024-04-25T01:42:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.692+0000 7fe0dfc27740 10 client.19155 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.692+0000 7fe0dfc27740 10 client.19155 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:42:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.692+0000 7fe0dfc27740 20 client.19155 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.692+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/3244452380 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564e920af550 0x564e920b1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.692+0000 7fe0dfc27740 4 client.19155 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.692+0000 7fe0dfc27740 1 -- 172.21.15.28:0/3244452380 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e92006bb0 con 0x564e920af550 2024-04-25T01:42:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.692+0000 7fe0d659c640 1 --2- 172.21.15.28:0/3244452380 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564e920af550 0x564e920b1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.693+0000 7fe0bdffb640 10 client.19155 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:42:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.693+0000 7fe0d659c640 1 --2- 172.21.15.28:0/3244452380 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564e920af550 0x564e920b1980 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b297c072475afcfb server_cookie=5a0ca987026a78fd in_seq=0 out_seq=0 2024-04-25T01:42:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.694+0000 7fe0bdffb640 1 -- 172.21.15.28:0/3244452380 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564e92006bb0 con 0x564e920af550 2024-04-25T01:42:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.694+0000 7fe0bdffb640 10 client.19155 handle_command_reply: tid=0 2024-04-25T01:42:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.694+0000 7fe0dfc27740 1 client.19155 shutdown 2024-04-25T01:42:38.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.694+0000 7fe0dfc27740 1 -- 172.21.15.28:0/3244452380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e92094580 msgr2=0x564e9209ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.694+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/3244452380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e92094580 0x564e9209ef10 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc009230 tx=0x7fe0cc040040 comp rx=0 tx=0).stop 2024-04-25T01:42:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.694+0000 7fe0dfc27740 1 -- 172.21.15.28:0/3244452380 shutdown_connections 2024-04-25T01:42:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.694+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/3244452380 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564e920af550 0x564e920b1980 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.694+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/3244452380 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e9209f570 0x564e9209c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.694+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/3244452380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e92094ea0 0x564e9209c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.694+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/3244452380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e92094580 0x564e9209ef10 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.694+0000 7fe0bdffb640 0 client.19155 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:42:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.694+0000 7fe0dfc27740 1 -- 172.21.15.28:0/3244452380 >> 172.21.15.28:0/3244452380 conn(0x564e920908a0 msgr2=0x564e920966a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.694+0000 7fe0dfc27740 1 -- 172.21.15.28:0/3244452380 shutdown_connections 2024-04-25T01:42:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.694+0000 7fe0dfc27740 1 -- 172.21.15.28:0/3244452380 wait complete. 2024-04-25T01:42:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.694+0000 7fe0dfc27740 20 client.19155 trim_cache size 0 max 16384 2024-04-25T01:42:38.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.707+0000 7fe0dfc27740 1 Processor -- start 2024-04-25T01:42:38.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.707+0000 7fe0dfc27740 1 -- start start 2024-04-25T01:42:38.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.708+0000 7fe0dfc27740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e92094580 0x564e920ab220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.708+0000 7fe0dfc27740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e92094ea0 0x564e920b7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.708+0000 7fe0dfc27740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9209f570 0x564e920b7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.708+0000 7fe0dfc27740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0d81b0d10 con 0x564e9209f570 2024-04-25T01:42:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.708+0000 7fe0dfc27740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe0d81b2850 con 0x564e92094580 2024-04-25T01:42:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.708+0000 7fe0dfc27740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe0d81b13e0 con 0x564e92094ea0 2024-04-25T01:42:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.708+0000 7fe0d5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9209f570 0x564e920b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.708+0000 7fe0d5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9209f570 0x564e920b7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47828/0 (socket says 172.21.15.28:47828) 2024-04-25T01:42:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.708+0000 7fe0d5d9b640 1 -- 172.21.15.28:0/1437797035 learned_addr learned my addr 172.21.15.28:0/1437797035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.708+0000 7fe0d5d9b640 1 -- 172.21.15.28:0/1437797035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e92094ea0 msgr2=0x564e920b7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.708+0000 7fe0d5d9b640 1 --2- 172.21.15.28:0/1437797035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e92094ea0 0x564e920b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.708+0000 7fe0d5d9b640 1 -- 172.21.15.28:0/1437797035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e92094580 msgr2=0x564e920ab220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.708+0000 7fe0d5d9b640 1 --2- 172.21.15.28:0/1437797035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e92094580 0x564e920ab220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.708+0000 7fe0d5d9b640 1 -- 172.21.15.28:0/1437797035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e92094cd0 con 0x564e9209f570 2024-04-25T01:42:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.708+0000 7fe0d5d9b640 1 --2- 172.21.15.28:0/1437797035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9209f570 0x564e920b7790 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc0059d0 tx=0x7fe0cc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e6a0e60f0052212 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.708+0000 7fe0bffff640 1 -- 172.21.15.28:0/1437797035 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc035070 con 0x564e9209f570 2024-04-25T01:42:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.708+0000 7fe0bffff640 1 -- 172.21.15.28:0/1437797035 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0cc004b00 con 0x564e9209f570 2024-04-25T01:42:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.708+0000 7fe0bffff640 1 -- 172.21.15.28:0/1437797035 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc005570 con 0x564e9209f570 2024-04-25T01:42:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.709+0000 7fe0dfc27740 1 -- 172.21.15.28:0/1437797035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9209f570 msgr2=0x564e920b7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.709+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/1437797035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9209f570 0x564e920b7790 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc0059d0 tx=0x7fe0cc009260 comp rx=0 tx=0).stop 2024-04-25T01:42:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.709+0000 7fe0dfc27740 1 -- 172.21.15.28:0/1437797035 shutdown_connections 2024-04-25T01:42:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.709+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/1437797035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9209f570 0x564e920b7790 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.709+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/1437797035 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e92094ea0 0x564e920b7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.709+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/1437797035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e92094580 0x564e920ab220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.709+0000 7fe0dfc27740 1 -- 172.21.15.28:0/1437797035 >> 172.21.15.28:0/1437797035 conn(0x564e920908a0 msgr2=0x564e92090c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.709+0000 7fe0dfc27740 1 -- 172.21.15.28:0/1437797035 shutdown_connections 2024-04-25T01:42:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.709+0000 7fe0dfc27740 1 -- 172.21.15.28:0/1437797035 wait complete. 2024-04-25T01:42:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.709+0000 7fe0dfc27740 1 -- start start 2024-04-25T01:42:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.709+0000 7fe0dfc27740 1 Processor -- start 2024-04-25T01:42:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.709+0000 7fe0dfc27740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e92094580 0x564e920a7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.709+0000 7fe0dfc27740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e92094ea0 0x564e920a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.709+0000 7fe0dfc27740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e9209f570 0x564e920a5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0dfc27740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0d81b2180 con 0x564e92094ea0 2024-04-25T01:42:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0dfc27740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe0d81b1ab0 con 0x564e9209f570 2024-04-25T01:42:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0dfc27740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe0d81aed10 con 0x564e92094580 2024-04-25T01:42:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0dfc27740 10 client.0 _ll_register_callbacks cb 0x7fe0cc009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0d659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e92094ea0 0x564e920a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0d659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e92094ea0 0x564e920a4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47832/0 (socket says 172.21.15.28:47832) 2024-04-25T01:42:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0d659c640 1 -- 172.21.15.28:0/1177161706 learned_addr learned my addr 172.21.15.28:0/1177161706 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0d5d9b640 1 --2- 172.21.15.28:0/1177161706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e9209f570 0x564e920a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0d5d9b640 1 --2- 172.21.15.28:0/1177161706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e92094580 0x564e920a7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0d659c640 1 -- 172.21.15.28:0/1177161706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e92094580 msgr2=0x564e920a7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0d659c640 1 --2- 172.21.15.28:0/1177161706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e92094580 0x564e920a7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0d659c640 1 -- 172.21.15.28:0/1177161706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e9209f570 msgr2=0x564e920a5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0d659c640 1 --2- 172.21.15.28:0/1177161706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e9209f570 0x564e920a5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0d659c640 1 -- 172.21.15.28:0/1177161706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e91dd77d0 con 0x564e92094ea0 2024-04-25T01:42:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0d659c640 1 --2- 172.21.15.28:0/1177161706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e92094ea0 0x564e920a4cc0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8000f80 tx=0x7fe0c8005cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0550807dbb2da51 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0bdffb640 1 -- 172.21.15.28:0/1177161706 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c802f060 con 0x564e92094ea0 2024-04-25T01:42:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0bdffb640 1 -- 172.21.15.28:0/1177161706 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7fe0c802a520 con 0x564e92094ea0 2024-04-25T01:42:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0dfc27740 1 -- 172.21.15.28:0/1177161706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0cc004b00 con 0x564e92094ea0 2024-04-25T01:42:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0bdffb640 1 -- 172.21.15.28:0/1177161706 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe0c8036050 con 0x564e92094ea0 2024-04-25T01:42:38.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0dfc27740 10 client.19161 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.710+0000 7fe0dfc27740 1 -- 172.21.15.28:0/1177161706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564e921157c0 con 0x564e92094ea0 2024-04-25T01:42:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.711+0000 7fe0bdffb640 1 -- 172.21.15.28:0/1177161706 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0cc004b00 con 0x564e92094ea0 2024-04-25T01:42:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.711+0000 7fe0bdffb640 1 -- 172.21.15.28:0/1177161706 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe0c8011cc0 con 0x564e92094ea0 2024-04-25T01:42:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.711+0000 7fe0dfc27740 10 client.19161 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.711+0000 7fe0dfc27740 10 client.19161 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:42:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.711+0000 7fe0dfc27740 20 client.19161 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.711+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/1177161706 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564e92115c90 0x564e921180b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.711+0000 7fe0dfc27740 4 client.19161 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:42:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.711+0000 7fe0dfc27740 1 -- 172.21.15.28:0/1177161706 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564e92000f00 con 0x564e92115c90 2024-04-25T01:42:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.711+0000 7fe0d5d9b640 1 --2- 172.21.15.28:0/1177161706 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564e92115c90 0x564e921180b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.712+0000 7fe0bdffb640 10 client.19161 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:42:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.712+0000 7fe0d5d9b640 1 --2- 172.21.15.28:0/1177161706 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564e92115c90 0x564e921180b0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4e98f17afbc915c6 server_cookie=ced1ba8f1b1a7ef8 in_seq=0 out_seq=0 2024-04-25T01:42:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.712+0000 7fe0bdffb640 1 -- 172.21.15.28:0/1177161706 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564e92000f00 con 0x564e92115c90 2024-04-25T01:42:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.712+0000 7fe0bdffb640 10 client.19161 handle_command_reply: tid=0 2024-04-25T01:42:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.712+0000 7fe0dfc27740 1 client.19161 shutdown 2024-04-25T01:42:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.712+0000 7fe0dfc27740 1 -- 172.21.15.28:0/1177161706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e92094ea0 msgr2=0x564e920a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.712+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/1177161706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e92094ea0 0x564e920a4cc0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8000f80 tx=0x7fe0c8005cb0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.712+0000 7fe0dfc27740 1 -- 172.21.15.28:0/1177161706 shutdown_connections 2024-04-25T01:42:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.712+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/1177161706 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x564e92115c90 0x564e921180b0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.712+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/1177161706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e9209f570 0x564e920a5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.712+0000 7fe0bdffb640 0 client.19161 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:42:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.712+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/1177161706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e92094ea0 0x564e920a4cc0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.712+0000 7fe0dfc27740 1 --2- 172.21.15.28:0/1177161706 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564e92094580 0x564e920a7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.712+0000 7fe0dfc27740 1 -- 172.21.15.28:0/1177161706 >> 172.21.15.28:0/1177161706 conn(0x564e920908a0 msgr2=0x564e9209aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.712+0000 7fe0dfc27740 1 -- 172.21.15.28:0/1177161706 shutdown_connections 2024-04-25T01:42:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.712+0000 7fe0dfc27740 1 -- 172.21.15.28:0/1177161706 wait complete. 2024-04-25T01:42:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.712+0000 7fe0dfc27740 20 client.19161 trim_cache size 0 max 16384 2024-04-25T01:42:38.707 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:42:38.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.713+0000 7fe0dce38640 1 -- 172.21.15.28:0/305863061 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe0c00a2d30 msgr2=0x7fe0c00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.713+0000 7fe0dce38640 1 --2- 172.21.15.28:0/305863061 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe0c00a2d30 0x7fe0c00a51f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe0d00a40c0 tx=0x7fe0d006d940 comp rx=0 tx=0).stop 2024-04-25T01:42:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.713+0000 7fe0dce38640 1 -- 172.21.15.28:0/305863061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0d8107730 msgr2=0x7fe0d81010f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.713+0000 7fe0dce38640 1 --2- 172.21.15.28:0/305863061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0d8107730 0x7fe0d81010f0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fe0d000c970 tx=0x7fe0d000ce30 comp rx=0 tx=0).stop 2024-04-25T01:42:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.713+0000 7fe0dce38640 1 -- 172.21.15.28:0/305863061 shutdown_connections 2024-04-25T01:42:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.713+0000 7fe0dce38640 1 --2- 172.21.15.28:0/305863061 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe0c00a2d30 0x7fe0c00a51f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.713+0000 7fe0dce38640 1 --2- 172.21.15.28:0/305863061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0d8107730 0x7fe0d81010f0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.713+0000 7fe0dce38640 1 --2- 172.21.15.28:0/305863061 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe0d806a780 0x7fe0d8100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.713+0000 7fe0dce38640 1 --2- 172.21.15.28:0/305863061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe0d8069dc0 0x7fe0d8100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.713+0000 7fe0dce38640 1 -- 172.21.15.28:0/305863061 >> 172.21.15.28:0/305863061 conn(0x7fe0d80776a0 msgr2=0x7fe0d80fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.713+0000 7fe0dce38640 1 -- 172.21.15.28:0/305863061 shutdown_connections 2024-04-25T01:42:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.713+0000 7fe0dce38640 1 -- 172.21.15.28:0/305863061 wait complete. 2024-04-25T01:42:38.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:42:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.827+0000 7f81ddb09640 1 -- 172.21.15.28:0/2992687197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81d8100610 msgr2=0x7f81d8100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.827+0000 7f81ddb09640 1 --2- 172.21.15.28:0/2992687197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81d8100610 0x7f81d8100a90 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f81cc009930 tx=0x7f81cc02a420 comp rx=0 tx=0).stop 2024-04-25T01:42:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.827+0000 7f81ddb09640 1 -- 172.21.15.28:0/2992687197 shutdown_connections 2024-04-25T01:42:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.827+0000 7f81ddb09640 1 --2- 172.21.15.28:0/2992687197 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f81d8100fd0 0x7f81d810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.827+0000 7f81ddb09640 1 --2- 172.21.15.28:0/2992687197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81d8100610 0x7f81d8100a90 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.827+0000 7f81ddb09640 1 --2- 172.21.15.28:0/2992687197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d81066b0 0x7f81d8106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.827+0000 7f81ddb09640 1 -- 172.21.15.28:0/2992687197 >> 172.21.15.28:0/2992687197 conn(0x7f81d8077740 msgr2=0x7f81d8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.827+0000 7f81ddb09640 1 -- 172.21.15.28:0/2992687197 shutdown_connections 2024-04-25T01:42:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.828+0000 7f81ddb09640 1 -- 172.21.15.28:0/2992687197 wait complete. 2024-04-25T01:42:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.828+0000 7f81ddb09640 1 Processor -- start 2024-04-25T01:42:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.828+0000 7f81ddb09640 1 -- start start 2024-04-25T01:42:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.828+0000 7f81ddb09640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81d8100610 0x7f81d8197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.828+0000 7f81ddb09640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f81d8100fd0 0x7f81d8198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.828+0000 7f81ddb09640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d81066b0 0x7f81d819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81ddb09640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81d8110a30 con 0x7f81d8100610 2024-04-25T01:42:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81ddb09640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f81d81108b0 con 0x7f81d81066b0 2024-04-25T01:42:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81ddb09640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f81d8110bb0 con 0x7f81d8100fd0 2024-04-25T01:42:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81d77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81d8100610 0x7f81d8197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81d77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81d8100610 0x7f81d8197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47854/0 (socket says 172.21.15.28:47854) 2024-04-25T01:42:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81d77fe640 1 -- 172.21.15.28:0/444368552 learned_addr learned my addr 172.21.15.28:0/444368552 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81d6ffd640 1 --2- 172.21.15.28:0/444368552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f81d8100fd0 0x7f81d8198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81d7fff640 1 --2- 172.21.15.28:0/444368552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d81066b0 0x7f81d819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81d77fe640 1 -- 172.21.15.28:0/444368552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f81d8100fd0 msgr2=0x7f81d8198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81d77fe640 1 --2- 172.21.15.28:0/444368552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f81d8100fd0 0x7f81d8198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81d77fe640 1 -- 172.21.15.28:0/444368552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d81066b0 msgr2=0x7f81d819c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81d77fe640 1 --2- 172.21.15.28:0/444368552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d81066b0 0x7f81d819c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81d77fe640 1 -- 172.21.15.28:0/444368552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d819cf00 con 0x7f81d8100610 2024-04-25T01:42:38.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81d6ffd640 1 --2- 172.21.15.28:0/444368552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f81d8100fd0 0x7f81d8198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:38.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81d7fff640 1 --2- 172.21.15.28:0/444368552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d81066b0 0x7f81d819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:38.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81d77fe640 1 --2- 172.21.15.28:0/444368552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81d8100610 0x7f81d8197f10 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f81c800c970 tx=0x7f81c800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81d4ff9640 1 -- 172.21.15.28:0/444368552 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c8007c10 con 0x7f81d8100610 2024-04-25T01:42:38.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81d4ff9640 1 -- 172.21.15.28:0/444368552 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81c8007db0 con 0x7f81d8100610 2024-04-25T01:42:38.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81ddb09640 1 -- 172.21.15.28:0/444368552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81d819d1f0 con 0x7f81d8100610 2024-04-25T01:42:38.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81d4ff9640 1 -- 172.21.15.28:0/444368552 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c800f940 con 0x7f81d8100610 2024-04-25T01:42:38.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.829+0000 7f81ddb09640 1 -- 172.21.15.28:0/444368552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81d81a4a80 con 0x7f81d8100610 2024-04-25T01:42:38.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.832+0000 7f81d4ff9640 1 -- 172.21.15.28:0/444368552 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f81c800fae0 con 0x7f81d8100610 2024-04-25T01:42:38.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.832+0000 7f81e0810740 1 Processor -- start 2024-04-25T01:42:38.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.832+0000 7f81e0810740 1 -- start start 2024-04-25T01:42:38.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.832+0000 7f81e0810740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62e580 0x558c6d62e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.832+0000 7f81e0810740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62eea0 0x558c6d639030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.832+0000 7f81e0810740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d639570 0x558c6d63b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.832+0000 7f81d4ff9640 1 --2- 172.21.15.28:0/444368552 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f81c00a2d00 0x7f81c00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.832+0000 7f81e0810740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558c6d593670 con 0x558c6d62e580 2024-04-25T01:42:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.832+0000 7f81e0810740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558c6d53b970 con 0x558c6d639570 2024-04-25T01:42:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.832+0000 7f81e0810740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x558c6d58f630 con 0x558c6d62eea0 2024-04-25T01:42:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.832+0000 7f81d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62e580 0x558c6d62e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.832+0000 7f81d4ff9640 1 -- 172.21.15.28:0/444368552 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f81c80cfc90 con 0x7f81d8100610 2024-04-25T01:42:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.832+0000 7f81d6ffd640 1 --2- 172.21.15.28:0/444368552 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f81c00a2d00 0x7f81c00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.832+0000 7f81d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62e580 0x558c6d62e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47860/0 (socket says 172.21.15.28:47860) 2024-04-25T01:42:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.832+0000 7f81d7fff640 1 -- 172.21.15.28:0/1737505408 learned_addr learned my addr 172.21.15.28:0/1737505408 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.833+0000 7f81d7fff640 1 --2- 172.21.15.28:0/1737505408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d639570 0x558c6d63b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.833+0000 7f81d77fe640 1 --2- 172.21.15.28:0/1737505408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62eea0 0x558c6d639030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.833+0000 7f81d7fff640 1 -- 172.21.15.28:0/1737505408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62eea0 msgr2=0x558c6d639030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.833+0000 7f81d7fff640 1 --2- 172.21.15.28:0/1737505408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62eea0 0x558c6d639030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.833+0000 7f81d7fff640 1 -- 172.21.15.28:0/1737505408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d639570 msgr2=0x558c6d63b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.833+0000 7f81d7fff640 1 --2- 172.21.15.28:0/1737505408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d639570 0x558c6d63b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.833+0000 7f81d7fff640 1 -- 172.21.15.28:0/1737505408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c6d55f400 con 0x558c6d62e580 2024-04-25T01:42:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.833+0000 7f81d6ffd640 1 --2- 172.21.15.28:0/444368552 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f81c00a2d00 0x7f81c00a51c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f81cc000c00 tx=0x7f81cc005a20 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.833+0000 7f81d7fff640 1 --2- 172.21.15.28:0/1737505408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62e580 0x558c6d62e960 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f81d00148d0 tx=0x7f81d0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9454e31e50c4a603 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.833+0000 7f81bdffb640 1 -- 172.21.15.28:0/1737505408 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d001a020 con 0x558c6d62e580 2024-04-25T01:42:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.833+0000 7f81bdffb640 1 -- 172.21.15.28:0/1737505408 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81d0012410 con 0x558c6d62e580 2024-04-25T01:42:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.833+0000 7f81bdffb640 1 -- 172.21.15.28:0/1737505408 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d0010600 con 0x558c6d62e580 2024-04-25T01:42:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.834+0000 7f81e0810740 1 -- 172.21.15.28:0/1737505408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62e580 msgr2=0x558c6d62e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.834+0000 7f81e0810740 1 --2- 172.21.15.28:0/1737505408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62e580 0x558c6d62e960 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f81d00148d0 tx=0x7f81d0009780 comp rx=0 tx=0).stop 2024-04-25T01:42:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.834+0000 7f81e0810740 1 -- 172.21.15.28:0/1737505408 shutdown_connections 2024-04-25T01:42:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.834+0000 7f81e0810740 1 --2- 172.21.15.28:0/1737505408 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d639570 0x558c6d63b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.834+0000 7f81e0810740 1 --2- 172.21.15.28:0/1737505408 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62eea0 0x558c6d639030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.834+0000 7f81e0810740 1 --2- 172.21.15.28:0/1737505408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62e580 0x558c6d62e960 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.834+0000 7f81e0810740 1 -- 172.21.15.28:0/1737505408 >> 172.21.15.28:0/1737505408 conn(0x558c6d62a8a0 msgr2=0x558c6d62ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.834+0000 7f81e0810740 1 -- 172.21.15.28:0/1737505408 shutdown_connections 2024-04-25T01:42:38.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.834+0000 7f81e0810740 1 -- 172.21.15.28:0/1737505408 wait complete. 2024-04-25T01:42:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.834+0000 7f81e0810740 1 -- start start 2024-04-25T01:42:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.834+0000 7f81e0810740 1 Processor -- start 2024-04-25T01:42:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81e0810740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62e580 0x558c6d638e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81e0810740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d62eea0 0x558c6d636120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81e0810740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d639570 0x558c6d636660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81e0810740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558c6d63c360 con 0x558c6d639570 2024-04-25T01:42:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81e0810740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558c6d54f050 con 0x558c6d62eea0 2024-04-25T01:42:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81e0810740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x558c6d63c4e0 con 0x558c6d62e580 2024-04-25T01:42:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81e0810740 10 client.0 _ll_register_callbacks cb 0x558c6d2a6be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d639570 0x558c6d636660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d639570 0x558c6d636660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47876/0 (socket says 172.21.15.28:47876) 2024-04-25T01:42:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81d7fff640 1 -- 172.21.15.28:0/2455281345 learned_addr learned my addr 172.21.15.28:0/2455281345 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81d77fe640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62e580 0x558c6d638e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81d6ffd640 1 --2- 172.21.15.28:0/2455281345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d62eea0 0x558c6d636120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81d7fff640 1 -- 172.21.15.28:0/2455281345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62e580 msgr2=0x558c6d638e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81d7fff640 1 --2- 172.21.15.28:0/2455281345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62e580 0x558c6d638e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81d7fff640 1 -- 172.21.15.28:0/2455281345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d62eea0 msgr2=0x558c6d636120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81d7fff640 1 --2- 172.21.15.28:0/2455281345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d62eea0 0x558c6d636120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81d7fff640 1 -- 172.21.15.28:0/2455281345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c6d520dc0 con 0x558c6d639570 2024-04-25T01:42:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81d7fff640 1 --2- 172.21.15.28:0/2455281345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d639570 0x558c6d636660 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f81d0009880 tx=0x7f81d0012af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5702e9bfa476315a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81b6ffd640 1 -- 172.21.15.28:0/2455281345 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d001a030 con 0x558c6d639570 2024-04-25T01:42:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81b6ffd640 1 -- 172.21.15.28:0/2455281345 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f81d0004c40 con 0x558c6d639570 2024-04-25T01:42:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.835+0000 7f81e0810740 1 -- 172.21.15.28:0/2455281345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81d0012410 con 0x558c6d639570 2024-04-25T01:42:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.836+0000 7f81b6ffd640 1 -- 172.21.15.28:0/2455281345 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f81d00095d0 con 0x558c6d639570 2024-04-25T01:42:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.836+0000 7f81e0810740 10 client.19173 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.836+0000 7f81e0810740 1 -- 172.21.15.28:0/2455281345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558c6d2a73f0 con 0x558c6d639570 2024-04-25T01:42:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.836+0000 7f81b6ffd640 1 -- 172.21.15.28:0/2455281345 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81d0012410 con 0x558c6d639570 2024-04-25T01:42:38.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.836+0000 7f81b6ffd640 1 -- 172.21.15.28:0/2455281345 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f81d000f040 con 0x558c6d639570 2024-04-25T01:42:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.836+0000 7f81e0810740 10 client.19173 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.836+0000 7f81e0810740 10 client.19173 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:42:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.836+0000 7f81e0810740 20 client.19173 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.836+0000 7f81e0810740 1 --2- 172.21.15.28:0/2455281345 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x558c6d649460 0x558c6d64b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.836+0000 7f81e0810740 4 client.19173 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.836+0000 7f81e0810740 1 -- 172.21.15.28:0/2455281345 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c6d5a0bb0 con 0x558c6d649460 2024-04-25T01:42:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.836+0000 7f81d77fe640 1 --2- 172.21.15.28:0/2455281345 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x558c6d649460 0x558c6d64b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.837+0000 7f81d77fe640 1 --2- 172.21.15.28:0/2455281345 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x558c6d649460 0x558c6d64b850 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=49f25dfbe628714b server_cookie=22e2239e4d98ccc0 in_seq=0 out_seq=0 2024-04-25T01:42:38.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.837+0000 7f81b6ffd640 10 client.19173 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:42:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.838+0000 7f81b6ffd640 1 -- 172.21.15.28:0/2455281345 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558c6d5a0bb0 con 0x558c6d649460 2024-04-25T01:42:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.838+0000 7f81b6ffd640 10 client.19173 handle_command_reply: tid=0 2024-04-25T01:42:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.838+0000 7f81e0810740 1 client.19173 shutdown 2024-04-25T01:42:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.838+0000 7f81e0810740 1 -- 172.21.15.28:0/2455281345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d639570 msgr2=0x558c6d636660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.838+0000 7f81e0810740 1 --2- 172.21.15.28:0/2455281345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d639570 0x558c6d636660 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f81d0009880 tx=0x7f81d0012af0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.838+0000 7f81e0810740 1 -- 172.21.15.28:0/2455281345 shutdown_connections 2024-04-25T01:42:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.838+0000 7f81e0810740 1 --2- 172.21.15.28:0/2455281345 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x558c6d649460 0x558c6d64b850 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.838+0000 7f81e0810740 1 --2- 172.21.15.28:0/2455281345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d639570 0x558c6d636660 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.838+0000 7f81e0810740 1 --2- 172.21.15.28:0/2455281345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d62eea0 0x558c6d636120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.838+0000 7f81b6ffd640 0 client.19173 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:42:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.838+0000 7f81e0810740 1 --2- 172.21.15.28:0/2455281345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62e580 0x558c6d638e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.838+0000 7f81e0810740 1 -- 172.21.15.28:0/2455281345 >> 172.21.15.28:0/2455281345 conn(0x558c6d62a8a0 msgr2=0x558c6d6306a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.838+0000 7f81e0810740 1 -- 172.21.15.28:0/2455281345 shutdown_connections 2024-04-25T01:42:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.838+0000 7f81e0810740 1 -- 172.21.15.28:0/2455281345 wait complete. 2024-04-25T01:42:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.838+0000 7f81e0810740 20 client.19173 trim_cache size 0 max 16384 2024-04-25T01:42:38.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.851+0000 7f81e0810740 1 Processor -- start 2024-04-25T01:42:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.851+0000 7f81e0810740 1 -- start start 2024-04-25T01:42:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.851+0000 7f81e0810740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62e580 0x558c6d631860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.851+0000 7f81e0810740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62eea0 0x558c6d6299c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.851+0000 7f81e0810740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d639570 0x558c6d638960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.851+0000 7f81e0810740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81d81aeb50 con 0x558c6d62e580 2024-04-25T01:42:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.851+0000 7f81e0810740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f81d81b0690 con 0x558c6d639570 2024-04-25T01:42:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.851+0000 7f81e0810740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f81d81af220 con 0x558c6d62eea0 2024-04-25T01:42:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.851+0000 7f81d77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62e580 0x558c6d631860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.852+0000 7f81d77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62e580 0x558c6d631860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47886/0 (socket says 172.21.15.28:47886) 2024-04-25T01:42:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.852+0000 7f81d77fe640 1 -- 172.21.15.28:0/1076670074 learned_addr learned my addr 172.21.15.28:0/1076670074 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.852+0000 7f81d6ffd640 1 --2- 172.21.15.28:0/1076670074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62eea0 0x558c6d6299c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.852+0000 7f81d77fe640 1 -- 172.21.15.28:0/1076670074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62eea0 msgr2=0x558c6d6299c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.852+0000 7f81d77fe640 1 --2- 172.21.15.28:0/1076670074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62eea0 0x558c6d6299c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.852+0000 7f81d77fe640 1 -- 172.21.15.28:0/1076670074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d639570 msgr2=0x558c6d638960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.852+0000 7f81d77fe640 1 --2- 172.21.15.28:0/1076670074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d639570 0x558c6d638960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.852+0000 7f81d77fe640 1 -- 172.21.15.28:0/1076670074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c6d3717d0 con 0x558c6d62e580 2024-04-25T01:42:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.852+0000 7f81d6ffd640 1 --2- 172.21.15.28:0/1076670074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62eea0 0x558c6d6299c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.852+0000 7f81d77fe640 1 --2- 172.21.15.28:0/1076670074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62e580 0x558c6d631860 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f81c80cd0b0 tx=0x7f81c80ab040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f588b4b8ee02b678 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.852+0000 7f81bd7fa640 1 -- 172.21.15.28:0/1076670074 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c809a1b0 con 0x558c6d62e580 2024-04-25T01:42:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.852+0000 7f81bd7fa640 1 -- 172.21.15.28:0/1076670074 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81c80af070 con 0x558c6d62e580 2024-04-25T01:42:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.852+0000 7f81bd7fa640 1 -- 172.21.15.28:0/1076670074 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c80a4600 con 0x558c6d62e580 2024-04-25T01:42:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.853+0000 7f81e0810740 1 -- 172.21.15.28:0/1076670074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62e580 msgr2=0x558c6d631860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.853+0000 7f81e0810740 1 --2- 172.21.15.28:0/1076670074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62e580 0x558c6d631860 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f81c80cd0b0 tx=0x7f81c80ab040 comp rx=0 tx=0).stop 2024-04-25T01:42:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.853+0000 7f81e0810740 1 -- 172.21.15.28:0/1076670074 shutdown_connections 2024-04-25T01:42:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.853+0000 7f81e0810740 1 --2- 172.21.15.28:0/1076670074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d639570 0x558c6d638960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.853+0000 7f81e0810740 1 --2- 172.21.15.28:0/1076670074 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62eea0 0x558c6d6299c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.853+0000 7f81e0810740 1 --2- 172.21.15.28:0/1076670074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62e580 0x558c6d631860 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.853+0000 7f81e0810740 1 -- 172.21.15.28:0/1076670074 >> 172.21.15.28:0/1076670074 conn(0x558c6d62a8a0 msgr2=0x558c6d6292b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.853+0000 7f81e0810740 1 -- 172.21.15.28:0/1076670074 shutdown_connections 2024-04-25T01:42:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.853+0000 7f81e0810740 1 -- 172.21.15.28:0/1076670074 wait complete. 2024-04-25T01:42:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.853+0000 7f81e0810740 1 -- start start 2024-04-25T01:42:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.853+0000 7f81e0810740 1 Processor -- start 2024-04-25T01:42:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.853+0000 7f81e0810740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62e580 0x558c6d64e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.853+0000 7f81e0810740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62eea0 0x558c6d64bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.853+0000 7f81e0810740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d639570 0x558c6d64c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.853+0000 7f81e0810740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81d81affc0 con 0x558c6d62eea0 2024-04-25T01:42:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.853+0000 7f81e0810740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f81d81af8f0 con 0x558c6d639570 2024-04-25T01:42:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.853+0000 7f81e0810740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f81d81acb50 con 0x558c6d62e580 2024-04-25T01:42:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81e0810740 10 client.0 _ll_register_callbacks cb 0x7f81c80046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81d6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62eea0 0x558c6d64bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81d6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62eea0 0x558c6d64bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47898/0 (socket says 172.21.15.28:47898) 2024-04-25T01:42:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81d6ffd640 1 -- 172.21.15.28:0/3974695285 learned_addr learned my addr 172.21.15.28:0/3974695285 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81d7fff640 1 --2- 172.21.15.28:0/3974695285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d639570 0x558c6d64c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81d77fe640 1 --2- 172.21.15.28:0/3974695285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62e580 0x558c6d64e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81d6ffd640 1 -- 172.21.15.28:0/3974695285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62e580 msgr2=0x558c6d64e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81d6ffd640 1 --2- 172.21.15.28:0/3974695285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62e580 0x558c6d64e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81d6ffd640 1 -- 172.21.15.28:0/3974695285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d639570 msgr2=0x558c6d64c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81d6ffd640 1 --2- 172.21.15.28:0/3974695285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d639570 0x558c6d64c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81d6ffd640 1 -- 172.21.15.28:0/3974695285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c6d64c770 con 0x558c6d62eea0 2024-04-25T01:42:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81d77fe640 1 --2- 172.21.15.28:0/3974695285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62e580 0x558c6d64e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81d7fff640 1 --2- 172.21.15.28:0/3974695285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d639570 0x558c6d64c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81d6ffd640 1 --2- 172.21.15.28:0/3974695285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62eea0 0x558c6d64bcf0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f81cc02a940 tx=0x7f81cc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4eb695c030efab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81b6ffd640 1 -- 172.21.15.28:0/3974695285 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81cc04c070 con 0x558c6d62eea0 2024-04-25T01:42:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81b6ffd640 1 -- 172.21.15.28:0/3974695285 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f81cc0046c0 con 0x558c6d62eea0 2024-04-25T01:42:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81e0810740 1 -- 172.21.15.28:0/3974695285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81c80af070 con 0x558c6d62eea0 2024-04-25T01:42:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81b6ffd640 1 -- 172.21.15.28:0/3974695285 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f81cc058050 con 0x558c6d62eea0 2024-04-25T01:42:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81e0810740 10 client.19179 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.854+0000 7f81e0810740 1 -- 172.21.15.28:0/3974695285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558c6d6af2f0 con 0x558c6d62eea0 2024-04-25T01:42:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.855+0000 7f81b6ffd640 1 -- 172.21.15.28:0/3974695285 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81c80af070 con 0x558c6d62eea0 2024-04-25T01:42:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.855+0000 7f81b6ffd640 1 -- 172.21.15.28:0/3974695285 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f81cc003aa0 con 0x558c6d62eea0 2024-04-25T01:42:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.855+0000 7f81e0810740 10 client.19179 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.855+0000 7f81e0810740 10 client.19179 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:42:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.855+0000 7f81e0810740 20 client.19179 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.855+0000 7f81e0810740 1 --2- 172.21.15.28:0/3974695285 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x558c6d6afaa0 0x558c6d6b1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.855+0000 7f81e0810740 4 client.19179 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:42:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.855+0000 7f81e0810740 1 -- 172.21.15.28:0/3974695285 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558c6d6369f0 con 0x558c6d6afaa0 2024-04-25T01:42:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.855+0000 7f81d77fe640 1 --2- 172.21.15.28:0/3974695285 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x558c6d6afaa0 0x558c6d6b1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.855+0000 7f81b6ffd640 10 client.19179 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:42:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.855+0000 7f81d77fe640 1 --2- 172.21.15.28:0/3974695285 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x558c6d6afaa0 0x558c6d6b1ec0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5de6749812492480 server_cookie=72a3aee4e5d17b08 in_seq=0 out_seq=0 2024-04-25T01:42:38.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81b6ffd640 1 -- 172.21.15.28:0/3974695285 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558c6d6369f0 con 0x558c6d6afaa0 2024-04-25T01:42:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81b6ffd640 10 client.19179 handle_command_reply: tid=0 2024-04-25T01:42:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81e0810740 1 client.19179 shutdown 2024-04-25T01:42:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81e0810740 1 -- 172.21.15.28:0/3974695285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62eea0 msgr2=0x558c6d64bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81e0810740 1 --2- 172.21.15.28:0/3974695285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62eea0 0x558c6d64bcf0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f81cc02a940 tx=0x7f81cc009260 comp rx=0 tx=0).stop 2024-04-25T01:42:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81e0810740 1 -- 172.21.15.28:0/3974695285 shutdown_connections 2024-04-25T01:42:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81e0810740 1 --2- 172.21.15.28:0/3974695285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558c6d639570 0x558c6d64c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81e0810740 1 --2- 172.21.15.28:0/3974695285 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x558c6d6afaa0 0x558c6d6b1ec0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81e0810740 1 --2- 172.21.15.28:0/3974695285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c6d62eea0 0x558c6d64bcf0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81e0810740 1 --2- 172.21.15.28:0/3974695285 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558c6d62e580 0x558c6d64e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81b6ffd640 0 client.19179 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:42:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81e0810740 1 -- 172.21.15.28:0/3974695285 >> 172.21.15.28:0/3974695285 conn(0x558c6d62a8a0 msgr2=0x558c6d6292b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81e0810740 1 -- 172.21.15.28:0/3974695285 shutdown_connections 2024-04-25T01:42:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81e0810740 1 -- 172.21.15.28:0/3974695285 wait complete. 2024-04-25T01:42:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81e0810740 20 client.19179 trim_cache size 0 max 16384 2024-04-25T01:42:38.851 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:42:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81ddb09640 1 -- 172.21.15.28:0/444368552 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f81c00a2d00 msgr2=0x7f81c00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81ddb09640 1 --2- 172.21.15.28:0/444368552 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f81c00a2d00 0x7f81c00a51c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f81cc000c00 tx=0x7f81cc005a20 comp rx=0 tx=0).stop 2024-04-25T01:42:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81ddb09640 1 -- 172.21.15.28:0/444368552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81d8100610 msgr2=0x7f81d8197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81ddb09640 1 --2- 172.21.15.28:0/444368552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81d8100610 0x7f81d8197f10 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f81c800c970 tx=0x7f81c800ce30 comp rx=0 tx=0).stop 2024-04-25T01:42:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81ddb09640 1 -- 172.21.15.28:0/444368552 shutdown_connections 2024-04-25T01:42:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81ddb09640 1 --2- 172.21.15.28:0/444368552 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f81c00a2d00 0x7f81c00a51c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.856+0000 7f81ddb09640 1 --2- 172.21.15.28:0/444368552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f81d81066b0 0x7f81d819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.857+0000 7f81ddb09640 1 --2- 172.21.15.28:0/444368552 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f81d8100fd0 0x7f81d8198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.857+0000 7f81ddb09640 1 --2- 172.21.15.28:0/444368552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81d8100610 0x7f81d8197f10 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.857+0000 7f81ddb09640 1 -- 172.21.15.28:0/444368552 >> 172.21.15.28:0/444368552 conn(0x7f81d8077740 msgr2=0x7f81d8104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.857+0000 7f81ddb09640 1 -- 172.21.15.28:0/444368552 shutdown_connections 2024-04-25T01:42:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.857+0000 7f81ddb09640 1 -- 172.21.15.28:0/444368552 wait complete. 2024-04-25T01:42:38.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:42:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.971+0000 7fb049163640 1 -- 172.21.15.28:0/3002448063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb044106630 msgr2=0x7fb044106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.971+0000 7fb049163640 1 --2- 172.21.15.28:0/3002448063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb044106630 0x7fb044106a10 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fb034009930 tx=0x7fb03402a420 comp rx=0 tx=0).stop 2024-04-25T01:42:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.971+0000 7fb049163640 1 -- 172.21.15.28:0/3002448063 shutdown_connections 2024-04-25T01:42:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.971+0000 7fb049163640 1 --2- 172.21.15.28:0/3002448063 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb044100fe0 0x7fb04410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.971+0000 7fb049163640 1 --2- 172.21.15.28:0/3002448063 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb044100620 0x7fb044100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.971+0000 7fb049163640 1 --2- 172.21.15.28:0/3002448063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb044106630 0x7fb044106a10 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.971+0000 7fb049163640 1 -- 172.21.15.28:0/3002448063 >> 172.21.15.28:0/3002448063 conn(0x7fb0440776a0 msgr2=0x7fb044077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.971+0000 7fb049163640 1 -- 172.21.15.28:0/3002448063 shutdown_connections 2024-04-25T01:42:38.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.971+0000 7fb049163640 1 -- 172.21.15.28:0/3002448063 wait complete. 2024-04-25T01:42:38.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.971+0000 7fb049163640 1 Processor -- start 2024-04-25T01:42:38.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.972+0000 7fb049163640 1 -- start start 2024-04-25T01:42:38.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.972+0000 7fb049163640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb044100620 0x7fb0440fd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.972+0000 7fb049163640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb044100fe0 0x7fb0440fdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.972+0000 7fb049163640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb044106630 0x7fb0440fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.972+0000 7fb049163640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb044110a40 con 0x7fb044100fe0 2024-04-25T01:42:38.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.972+0000 7fb049163640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb0441108c0 con 0x7fb044100620 2024-04-25T01:42:38.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.972+0000 7fb049163640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb044110bc0 con 0x7fb044106630 2024-04-25T01:42:38.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.972+0000 7fb04259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb044100fe0 0x7fb0440fdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.972+0000 7fb04259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb044100fe0 0x7fb0440fdb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47918/0 (socket says 172.21.15.28:47918) 2024-04-25T01:42:38.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.972+0000 7fb04259c640 1 -- 172.21.15.28:0/3137291503 learned_addr learned my addr 172.21.15.28:0/3137291503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.972+0000 7fb042d9d640 1 --2- 172.21.15.28:0/3137291503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb044100620 0x7fb0440fd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.972+0000 7fb04359e640 1 --2- 172.21.15.28:0/3137291503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb044106630 0x7fb0440fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.972+0000 7fb04259c640 1 -- 172.21.15.28:0/3137291503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb044106630 msgr2=0x7fb0440fe0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.972+0000 7fb04259c640 1 --2- 172.21.15.28:0/3137291503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb044106630 0x7fb0440fe0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.972+0000 7fb04259c640 1 -- 172.21.15.28:0/3137291503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb044100620 msgr2=0x7fb0440fd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.972+0000 7fb04259c640 1 --2- 172.21.15.28:0/3137291503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb044100620 0x7fb0440fd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.972+0000 7fb04259c640 1 -- 172.21.15.28:0/3137291503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0441a65f0 con 0x7fb044100fe0 2024-04-25T01:42:38.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.972+0000 7fb042d9d640 1 --2- 172.21.15.28:0/3137291503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb044100620 0x7fb0440fd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:38.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.973+0000 7fb04259c640 1 --2- 172.21.15.28:0/3137291503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb044100fe0 0x7fb0440fdb90 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fb03800d770 tx=0x7fb03800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.973+0000 7fb02bfff640 1 -- 172.21.15.28:0/3137291503 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb038017070 con 0x7fb044100fe0 2024-04-25T01:42:38.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.973+0000 7fb049163640 1 -- 172.21.15.28:0/3137291503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0441a68e0 con 0x7fb044100fe0 2024-04-25T01:42:38.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.973+0000 7fb02bfff640 1 -- 172.21.15.28:0/3137291503 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb038005500 con 0x7fb044100fe0 2024-04-25T01:42:38.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.973+0000 7fb02bfff640 1 -- 172.21.15.28:0/3137291503 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb038004e00 con 0x7fb044100fe0 2024-04-25T01:42:38.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.973+0000 7fb049163640 1 -- 172.21.15.28:0/3137291503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0441a6e30 con 0x7fb044100fe0 2024-04-25T01:42:38.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.975+0000 7fb02bfff640 1 -- 172.21.15.28:0/3137291503 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb038004fa0 con 0x7fb044100fe0 2024-04-25T01:42:38.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.975+0000 7fb04be7b740 1 Processor -- start 2024-04-25T01:42:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.975+0000 7fb04be7b740 1 -- start start 2024-04-25T01:42:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb02bfff640 1 --2- 172.21.15.28:0/3137291503 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb02c0a2d00 0x7fb02c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb04be7b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a731b01580 0x55a731b01960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb04be7b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01ea0 0x55a731b0c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb04be7b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b0c570 0x55a731b0e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb04be7b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a731a66670 con 0x55a731b01ea0 2024-04-25T01:42:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb04be7b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a731a0e970 con 0x55a731b01580 2024-04-25T01:42:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb04be7b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55a731a62630 con 0x55a731b0c570 2024-04-25T01:42:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb04359e640 1 --2- 172.21.15.28:0/3137291503 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb02c0a2d00 0x7fb02c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb04259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01ea0 0x55a731b0c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb02bfff640 1 -- 172.21.15.28:0/3137291503 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7fb0380cf370 con 0x7fb044100fe0 2024-04-25T01:42:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb04259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01ea0 0x55a731b0c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47920/0 (socket says 172.21.15.28:47920) 2024-04-25T01:42:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb04259c640 1 -- 172.21.15.28:0/898005463 learned_addr learned my addr 172.21.15.28:0/898005463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb042d9d640 1 --2- 172.21.15.28:0/898005463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a731b01580 0x55a731b01960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb04259c640 1 -- 172.21.15.28:0/898005463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b0c570 msgr2=0x55a731b0e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb04259c640 1 --2- 172.21.15.28:0/898005463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b0c570 0x55a731b0e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb04259c640 1 -- 172.21.15.28:0/898005463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a731b01580 msgr2=0x55a731b01960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb04259c640 1 --2- 172.21.15.28:0/898005463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a731b01580 0x55a731b01960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb04259c640 1 -- 172.21.15.28:0/898005463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a731a32400 con 0x55a731b01ea0 2024-04-25T01:42:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb04359e640 1 --2- 172.21.15.28:0/3137291503 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb02c0a2d00 0x7fb02c0a51c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb03c0092f0 tx=0x7fb03c0097f0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb04259c640 1 --2- 172.21.15.28:0/898005463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01ea0 0x55a731b0c030 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fb0380a3be0 tx=0x7fb038018cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae7e3c00893459d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.976+0000 7fb0297fa640 1 -- 172.21.15.28:0/898005463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb038020040 con 0x55a731b01ea0 2024-04-25T01:42:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.977+0000 7fb0297fa640 1 -- 172.21.15.28:0/898005463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0380046c0 con 0x55a731b01ea0 2024-04-25T01:42:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.977+0000 7fb0297fa640 1 -- 172.21.15.28:0/898005463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03801e120 con 0x55a731b01ea0 2024-04-25T01:42:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.977+0000 7fb04be7b740 1 -- 172.21.15.28:0/898005463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01ea0 msgr2=0x55a731b0c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.977+0000 7fb04be7b740 1 --2- 172.21.15.28:0/898005463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01ea0 0x55a731b0c030 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fb0380a3be0 tx=0x7fb038018cf0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.977+0000 7fb04be7b740 1 -- 172.21.15.28:0/898005463 shutdown_connections 2024-04-25T01:42:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.977+0000 7fb04be7b740 1 --2- 172.21.15.28:0/898005463 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b0c570 0x55a731b0e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.977+0000 7fb04be7b740 1 --2- 172.21.15.28:0/898005463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01ea0 0x55a731b0c030 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.977+0000 7fb04be7b740 1 --2- 172.21.15.28:0/898005463 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a731b01580 0x55a731b01960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.977+0000 7fb04be7b740 1 -- 172.21.15.28:0/898005463 >> 172.21.15.28:0/898005463 conn(0x55a731afd8a0 msgr2=0x55a731afdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.977+0000 7fb04be7b740 1 -- 172.21.15.28:0/898005463 shutdown_connections 2024-04-25T01:42:38.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.977+0000 7fb04be7b740 1 -- 172.21.15.28:0/898005463 wait complete. 2024-04-25T01:42:38.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb04be7b740 1 -- start start 2024-04-25T01:42:38.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb04be7b740 1 Processor -- start 2024-04-25T01:42:38.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb04be7b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a731b01580 0x55a731b0bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb04be7b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b01ea0 0x55a731b09250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb04be7b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b0c570 0x55a731b09860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb04be7b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a731b0f360 con 0x55a731b0c570 2024-04-25T01:42:38.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb04be7b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a731a22050 con 0x55a731b01580 2024-04-25T01:42:38.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb04be7b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55a731b0f4e0 con 0x55a731b01ea0 2024-04-25T01:42:38.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb04be7b740 10 client.0 _ll_register_callbacks cb 0x55a731779be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:38.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb04359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b0c570 0x55a731b09860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb04359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b0c570 0x55a731b09860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47936/0 (socket says 172.21.15.28:47936) 2024-04-25T01:42:38.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb04359e640 1 -- 172.21.15.28:0/1485638525 learned_addr learned my addr 172.21.15.28:0/1485638525 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb042d9d640 1 --2- 172.21.15.28:0/1485638525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a731b01580 0x55a731b0bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb04259c640 1 --2- 172.21.15.28:0/1485638525 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b01ea0 0x55a731b09250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb04359e640 1 -- 172.21.15.28:0/1485638525 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b01ea0 msgr2=0x55a731b09250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb04359e640 1 --2- 172.21.15.28:0/1485638525 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b01ea0 0x55a731b09250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb04359e640 1 -- 172.21.15.28:0/1485638525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a731b01580 msgr2=0x55a731b0bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb04359e640 1 --2- 172.21.15.28:0/1485638525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a731b01580 0x55a731b0bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb04359e640 1 -- 172.21.15.28:0/1485638525 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7319f3dc0 con 0x55a731b0c570 2024-04-25T01:42:38.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.978+0000 7fb04259c640 1 --2- 172.21.15.28:0/1485638525 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b01ea0 0x55a731b09250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:38.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.979+0000 7fb04359e640 1 --2- 172.21.15.28:0/1485638525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b0c570 0x55a731b09860 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fb03c0092c0 tx=0x7fb03c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa6b54354f44ee8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.979+0000 7fb026ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:38.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.979+0000 7fb026ff5640 1 -- 172.21.15.28:0/1485638525 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03c024020 con 0x55a731b0c570 2024-04-25T01:42:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.979+0000 7fb04be7b740 1 -- 172.21.15.28:0/1485638525 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a731a73bb0 con 0x55a731b0c570 2024-04-25T01:42:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.979+0000 7fb026ff5640 1 -- 172.21.15.28:0/1485638525 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7fb03c00fae0 con 0x55a731b0c570 2024-04-25T01:42:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.979+0000 7fb026ff5640 1 -- 172.21.15.28:0/1485638525 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb03c02f050 con 0x55a731b0c570 2024-04-25T01:42:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.979+0000 7fb04be7b740 10 client.19191 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.979+0000 7fb04be7b740 1 -- 172.21.15.28:0/1485638525 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a73177a3f0 con 0x55a731b0c570 2024-04-25T01:42:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.979+0000 7fb026ff5640 1 -- 172.21.15.28:0/1485638525 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a731a73bb0 con 0x55a731b0c570 2024-04-25T01:42:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.979+0000 7fb026ff5640 1 -- 172.21.15.28:0/1485638525 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb03c00f040 con 0x55a731b0c570 2024-04-25T01:42:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.979+0000 7fb04be7b740 10 client.19191 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.979+0000 7fb04be7b740 10 client.19191 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:42:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.979+0000 7fb04be7b740 20 client.19191 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.979+0000 7fb04be7b740 1 --2- 172.21.15.28:0/1485638525 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55a731b1c750 0x55a731b1eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.979+0000 7fb04be7b740 4 client.19191 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.979+0000 7fb04be7b740 1 -- 172.21.15.28:0/1485638525 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a731a6df00 con 0x55a731b1c750 2024-04-25T01:42:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.980+0000 7fb042d9d640 1 --2- 172.21.15.28:0/1485638525 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55a731b1c750 0x55a731b1eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.980+0000 7fb026ff5640 10 client.19191 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:42:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.980+0000 7fb042d9d640 1 --2- 172.21.15.28:0/1485638525 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55a731b1c750 0x55a731b1eb40 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bf8fe4620178fc03 server_cookie=4c3b930c85793be8 in_seq=0 out_seq=0 2024-04-25T01:42:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.981+0000 7fb026ff5640 1 -- 172.21.15.28:0/1485638525 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a731a6df00 con 0x55a731b1c750 2024-04-25T01:42:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.981+0000 7fb026ff5640 10 client.19191 handle_command_reply: tid=0 2024-04-25T01:42:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.981+0000 7fb04be7b740 1 client.19191 shutdown 2024-04-25T01:42:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.981+0000 7fb04be7b740 1 -- 172.21.15.28:0/1485638525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b0c570 msgr2=0x55a731b09860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.981+0000 7fb04be7b740 1 --2- 172.21.15.28:0/1485638525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b0c570 0x55a731b09860 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fb03c0092c0 tx=0x7fb03c004ab0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.981+0000 7fb04be7b740 1 -- 172.21.15.28:0/1485638525 shutdown_connections 2024-04-25T01:42:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.981+0000 7fb04be7b740 1 --2- 172.21.15.28:0/1485638525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b0c570 0x55a731b09860 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.981+0000 7fb04be7b740 1 --2- 172.21.15.28:0/1485638525 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55a731b1c750 0x55a731b1eb40 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.981+0000 7fb04be7b740 1 --2- 172.21.15.28:0/1485638525 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b01ea0 0x55a731b09250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.981+0000 7fb04be7b740 1 --2- 172.21.15.28:0/1485638525 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a731b01580 0x55a731b0bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.981+0000 7fb026ff5640 0 client.19191 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:42:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.981+0000 7fb04be7b740 1 -- 172.21.15.28:0/1485638525 >> 172.21.15.28:0/1485638525 conn(0x55a731afd8a0 msgr2=0x55a731afcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.981+0000 7fb04be7b740 1 -- 172.21.15.28:0/1485638525 shutdown_connections 2024-04-25T01:42:38.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.981+0000 7fb04be7b740 1 -- 172.21.15.28:0/1485638525 wait complete. 2024-04-25T01:42:38.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.981+0000 7fb04be7b740 20 client.19191 trim_cache size 0 max 16384 2024-04-25T01:42:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.995+0000 7fb04be7b740 1 Processor -- start 2024-04-25T01:42:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.995+0000 7fb04be7b740 1 -- start start 2024-04-25T01:42:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.995+0000 7fb04be7b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01580 0x55a731b04860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.995+0000 7fb04be7b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a731b01ea0 0x55a731afc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.995+0000 7fb04be7b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b0c570 0x55a731b0ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.995+0000 7fb04be7b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb0441b0dc0 con 0x55a731b01580 2024-04-25T01:42:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.995+0000 7fb04be7b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb0441b2900 con 0x55a731b01ea0 2024-04-25T01:42:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.995+0000 7fb04be7b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb0441b1490 con 0x55a731b0c570 2024-04-25T01:42:38.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.995+0000 7fb042d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01580 0x55a731b04860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.995+0000 7fb042d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01580 0x55a731b04860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47946/0 (socket says 172.21.15.28:47946) 2024-04-25T01:42:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.995+0000 7fb042d9d640 1 -- 172.21.15.28:0/1960154231 learned_addr learned my addr 172.21.15.28:0/1960154231 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.995+0000 7fb04359e640 1 --2- 172.21.15.28:0/1960154231 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b0c570 0x55a731b0ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.995+0000 7fb042d9d640 1 -- 172.21.15.28:0/1960154231 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b0c570 msgr2=0x55a731b0ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.995+0000 7fb042d9d640 1 --2- 172.21.15.28:0/1960154231 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b0c570 0x55a731b0ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.995+0000 7fb042d9d640 1 -- 172.21.15.28:0/1960154231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a731b01ea0 msgr2=0x55a731afc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.995+0000 7fb042d9d640 1 --2- 172.21.15.28:0/1960154231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a731b01ea0 0x55a731afc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.995+0000 7fb042d9d640 1 -- 172.21.15.28:0/1960154231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7318447d0 con 0x55a731b01580 2024-04-25T01:42:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.995+0000 7fb04359e640 1 --2- 172.21.15.28:0/1960154231 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b0c570 0x55a731b0ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.995+0000 7fb042d9d640 1 --2- 172.21.15.28:0/1960154231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01580 0x55a731b04860 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fb034032dc0 tx=0x7fb034005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3f07ceb5389515 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.996+0000 7fb028ff9640 1 -- 172.21.15.28:0/1960154231 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb034035040 con 0x55a731b01580 2024-04-25T01:42:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.996+0000 7fb028ff9640 1 -- 172.21.15.28:0/1960154231 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb03404a070 con 0x55a731b01580 2024-04-25T01:42:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.996+0000 7fb028ff9640 1 -- 172.21.15.28:0/1960154231 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb034005500 con 0x55a731b01580 2024-04-25T01:42:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.996+0000 7fb04be7b740 1 -- 172.21.15.28:0/1960154231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01580 msgr2=0x55a731b04860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.996+0000 7fb04be7b740 1 --2- 172.21.15.28:0/1960154231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01580 0x55a731b04860 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fb034032dc0 tx=0x7fb034005b50 comp rx=0 tx=0).stop 2024-04-25T01:42:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.996+0000 7fb04be7b740 1 -- 172.21.15.28:0/1960154231 shutdown_connections 2024-04-25T01:42:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.996+0000 7fb04be7b740 1 --2- 172.21.15.28:0/1960154231 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b0c570 0x55a731b0ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.996+0000 7fb04be7b740 1 --2- 172.21.15.28:0/1960154231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a731b01ea0 0x55a731afc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.996+0000 7fb04be7b740 1 --2- 172.21.15.28:0/1960154231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01580 0x55a731b04860 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.996+0000 7fb04be7b740 1 -- 172.21.15.28:0/1960154231 >> 172.21.15.28:0/1960154231 conn(0x55a731afd8a0 msgr2=0x55a731afc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.996+0000 7fb04be7b740 1 -- 172.21.15.28:0/1960154231 shutdown_connections 2024-04-25T01:42:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.996+0000 7fb04be7b740 1 -- 172.21.15.28:0/1960154231 wait complete. 2024-04-25T01:42:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.997+0000 7fb04be7b740 1 -- start start 2024-04-25T01:42:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.997+0000 7fb04be7b740 1 Processor -- start 2024-04-25T01:42:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.997+0000 7fb04be7b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a731b01580 0x55a731b10470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.997+0000 7fb04be7b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01ea0 0x55a731b109b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.997+0000 7fb04be7b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b0c570 0x55a731b10ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.997+0000 7fb04be7b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb0441b2230 con 0x55a731b01ea0 2024-04-25T01:42:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.997+0000 7fb04be7b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb0441b1b60 con 0x55a731b01580 2024-04-25T01:42:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.997+0000 7fb04be7b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb0441aedc0 con 0x55a731b0c570 2024-04-25T01:42:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.997+0000 7fb04be7b740 10 client.0 _ll_register_callbacks cb 0x7fb03c009b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.997+0000 7fb04259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01ea0 0x55a731b109b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.997+0000 7fb04259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01ea0 0x55a731b109b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47962/0 (socket says 172.21.15.28:47962) 2024-04-25T01:42:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.997+0000 7fb04259c640 1 -- 172.21.15.28:0/1526129369 learned_addr learned my addr 172.21.15.28:0/1526129369 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.997+0000 7fb042d9d640 1 --2- 172.21.15.28:0/1526129369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a731b01580 0x55a731b10470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.997+0000 7fb04359e640 1 --2- 172.21.15.28:0/1526129369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b0c570 0x55a731b10ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.997+0000 7fb04259c640 1 -- 172.21.15.28:0/1526129369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b0c570 msgr2=0x55a731b10ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.997+0000 7fb04259c640 1 --2- 172.21.15.28:0/1526129369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b0c570 0x55a731b10ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.997+0000 7fb04259c640 1 -- 172.21.15.28:0/1526129369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a731b01580 msgr2=0x55a731b10470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.997+0000 7fb04259c640 1 --2- 172.21.15.28:0/1526129369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a731b01580 0x55a731b10470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.997+0000 7fb04259c640 1 -- 172.21.15.28:0/1526129369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a731b81f20 con 0x55a731b01ea0 2024-04-25T01:42:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.998+0000 7fb04259c640 1 --2- 172.21.15.28:0/1526129369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01ea0 0x55a731b109b0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fb038097e70 tx=0x7fb038012f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e0aadcaa6078b56 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.998+0000 7fb026ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.999+0000 7fb026ff5640 1 -- 172.21.15.28:0/1526129369 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0380c4020 con 0x55a731b01ea0 2024-04-25T01:42:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.999+0000 7fb026ff5640 1 -- 172.21.15.28:0/1526129369 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7fb038009cc0 con 0x55a731b01ea0 2024-04-25T01:42:38.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.999+0000 7fb04be7b740 1 -- 172.21.15.28:0/1526129369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb03404a070 con 0x55a731b01ea0 2024-04-25T01:42:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.999+0000 7fb026ff5640 1 -- 172.21.15.28:0/1526129369 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb038020040 con 0x55a731b01ea0 2024-04-25T01:42:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.999+0000 7fb04be7b740 10 client.19197 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.999+0000 7fb04be7b740 1 -- 172.21.15.28:0/1526129369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a731b83170 con 0x55a731b01ea0 2024-04-25T01:42:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:38.999+0000 7fb026ff5640 1 -- 172.21.15.28:0/1526129369 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb03404a070 con 0x55a731b01ea0 2024-04-25T01:42:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.000+0000 7fb026ff5640 1 -- 172.21.15.28:0/1526129369 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb038097b10 con 0x55a731b01ea0 2024-04-25T01:42:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.000+0000 7fb04be7b740 10 client.19197 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.000+0000 7fb04be7b740 10 client.19197 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:42:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.000+0000 7fb04be7b740 20 client.19197 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.000+0000 7fb04be7b740 1 --2- 172.21.15.28:0/1526129369 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55a731b83470 0x55a731b85860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.000+0000 7fb04be7b740 4 client.19197 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:42:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.000+0000 7fb04be7b740 1 -- 172.21.15.28:0/1526129369 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a7319eb820 con 0x55a731b83470 2024-04-25T01:42:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.000+0000 7fb042d9d640 1 --2- 172.21.15.28:0/1526129369 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55a731b83470 0x55a731b85860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.000+0000 7fb026ff5640 10 client.19197 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:42:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.000+0000 7fb042d9d640 1 --2- 172.21.15.28:0/1526129369 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55a731b83470 0x55a731b85860 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8820b301950fda1c server_cookie=ae44f7ae2ab0623e in_seq=0 out_seq=0 2024-04-25T01:42:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.001+0000 7fb026ff5640 1 -- 172.21.15.28:0/1526129369 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a7319eb820 con 0x55a731b83470 2024-04-25T01:42:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.001+0000 7fb026ff5640 10 client.19197 handle_command_reply: tid=0 2024-04-25T01:42:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.001+0000 7fb04be7b740 1 client.19197 shutdown 2024-04-25T01:42:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.001+0000 7fb04be7b740 1 -- 172.21.15.28:0/1526129369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01ea0 msgr2=0x55a731b109b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.001+0000 7fb04be7b740 1 --2- 172.21.15.28:0/1526129369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01ea0 0x55a731b109b0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fb038097e70 tx=0x7fb038012f30 comp rx=0 tx=0).stop 2024-04-25T01:42:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.001+0000 7fb04be7b740 1 -- 172.21.15.28:0/1526129369 shutdown_connections 2024-04-25T01:42:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.001+0000 7fb04be7b740 1 --2- 172.21.15.28:0/1526129369 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x55a731b83470 0x55a731b85860 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.001+0000 7fb04be7b740 1 --2- 172.21.15.28:0/1526129369 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55a731b0c570 0x55a731b10ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.001+0000 7fb04be7b740 1 --2- 172.21.15.28:0/1526129369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a731b01ea0 0x55a731b109b0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.001+0000 7fb04be7b740 1 --2- 172.21.15.28:0/1526129369 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a731b01580 0x55a731b10470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.001+0000 7fb026ff5640 0 client.19197 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:42:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.001+0000 7fb04be7b740 1 -- 172.21.15.28:0/1526129369 >> 172.21.15.28:0/1526129369 conn(0x55a731afd8a0 msgr2=0x55a731b17270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.001+0000 7fb04be7b740 1 -- 172.21.15.28:0/1526129369 shutdown_connections 2024-04-25T01:42:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.001+0000 7fb04be7b740 1 -- 172.21.15.28:0/1526129369 wait complete. 2024-04-25T01:42:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.001+0000 7fb04be7b740 20 client.19197 trim_cache size 0 max 16384 2024-04-25T01:42:38.994 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:42:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.002+0000 7fb049163640 1 -- 172.21.15.28:0/3137291503 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb02c0a2d00 msgr2=0x7fb02c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.002+0000 7fb049163640 1 --2- 172.21.15.28:0/3137291503 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb02c0a2d00 0x7fb02c0a51c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb03c0092f0 tx=0x7fb03c0097f0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.002+0000 7fb049163640 1 -- 172.21.15.28:0/3137291503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb044100fe0 msgr2=0x7fb0440fdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.002+0000 7fb049163640 1 --2- 172.21.15.28:0/3137291503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb044100fe0 0x7fb0440fdb90 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fb03800d770 tx=0x7fb03800dc30 comp rx=0 tx=0).stop 2024-04-25T01:42:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.002+0000 7fb049163640 1 -- 172.21.15.28:0/3137291503 shutdown_connections 2024-04-25T01:42:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.002+0000 7fb049163640 1 --2- 172.21.15.28:0/3137291503 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb02c0a2d00 0x7fb02c0a51c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.002+0000 7fb049163640 1 --2- 172.21.15.28:0/3137291503 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb044106630 0x7fb0440fe0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.002+0000 7fb049163640 1 --2- 172.21.15.28:0/3137291503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb044100fe0 0x7fb0440fdb90 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.002+0000 7fb049163640 1 --2- 172.21.15.28:0/3137291503 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb044100620 0x7fb0440fd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.002+0000 7fb049163640 1 -- 172.21.15.28:0/3137291503 >> 172.21.15.28:0/3137291503 conn(0x7fb0440776a0 msgr2=0x7fb04406a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.002+0000 7fb049163640 1 -- 172.21.15.28:0/3137291503 shutdown_connections 2024-04-25T01:42:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.002+0000 7fb049163640 1 -- 172.21.15.28:0/3137291503 wait complete. 2024-04-25T01:42:39.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:42:39.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.116+0000 7f08485c1640 1 -- 172.21.15.28:0/3600041807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0840107720 msgr2=0x7f0840107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:39.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.116+0000 7f08485c1640 1 --2- 172.21.15.28:0/3600041807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0840107720 0x7f0840107b00 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f083c009930 tx=0x7f083c02a420 comp rx=0 tx=0).stop 2024-04-25T01:42:39.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.116+0000 7f08485c1640 1 -- 172.21.15.28:0/3600041807 shutdown_connections 2024-04-25T01:42:39.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.116+0000 7f08485c1640 1 --2- 172.21.15.28:0/3600041807 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f08401020d0 0x7f084010e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.116+0000 7f08485c1640 1 --2- 172.21.15.28:0/3600041807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0840101710 0x7f0840101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.116+0000 7f08485c1640 1 --2- 172.21.15.28:0/3600041807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0840107720 0x7f0840107b00 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.116+0000 7f08485c1640 1 -- 172.21.15.28:0/3600041807 >> 172.21.15.28:0/3600041807 conn(0x7f08400fd180 msgr2=0x7f08400ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:39.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.116+0000 7f08485c1640 1 -- 172.21.15.28:0/3600041807 shutdown_connections 2024-04-25T01:42:39.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.116+0000 7f08485c1640 1 -- 172.21.15.28:0/3600041807 wait complete. 2024-04-25T01:42:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.117+0000 7f08485c1640 1 Processor -- start 2024-04-25T01:42:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.117+0000 7f08485c1640 1 -- start start 2024-04-25T01:42:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.117+0000 7f08485c1640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0840101710 0x7f0840198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.117+0000 7f08485c1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08401020d0 0x7f0840198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.117+0000 7f08485c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0840107720 0x7f084019c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.117+0000 7f08485c1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0840111b30 con 0x7f0840107720 2024-04-25T01:42:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.117+0000 7f08485c1640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f08401119b0 con 0x7f08401020d0 2024-04-25T01:42:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.117+0000 7f08485c1640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0840111cb0 con 0x7f0840101710 2024-04-25T01:42:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.117+0000 7f0846b5e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0840107720 0x7f084019c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.117+0000 7f0846b5e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0840107720 0x7f084019c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47980/0 (socket says 172.21.15.28:47980) 2024-04-25T01:42:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.117+0000 7f0846b5e640 1 -- 172.21.15.28:0/343049345 learned_addr learned my addr 172.21.15.28:0/343049345 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.117+0000 7f084635d640 1 --2- 172.21.15.28:0/343049345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0840101710 0x7f0840198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.117+0000 7f0845b5c640 1 --2- 172.21.15.28:0/343049345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08401020d0 0x7f0840198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.117+0000 7f0846b5e640 1 -- 172.21.15.28:0/343049345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0840101710 msgr2=0x7f0840198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.117+0000 7f0846b5e640 1 --2- 172.21.15.28:0/343049345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0840101710 0x7f0840198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.118+0000 7f0846b5e640 1 -- 172.21.15.28:0/343049345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08401020d0 msgr2=0x7f0840198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.118+0000 7f0846b5e640 1 --2- 172.21.15.28:0/343049345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08401020d0 0x7f0840198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.118+0000 7f0846b5e640 1 -- 172.21.15.28:0/343049345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084019d000 con 0x7f0840107720 2024-04-25T01:42:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.118+0000 7f0845b5c640 1 --2- 172.21.15.28:0/343049345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08401020d0 0x7f0840198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.118+0000 7f0846b5e640 1 --2- 172.21.15.28:0/343049345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0840107720 0x7f084019c8d0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f083800eaa0 tx=0x7f083800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.118+0000 7f08377fe640 1 -- 172.21.15.28:0/343049345 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083800cdb0 con 0x7f0840107720 2024-04-25T01:42:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.118+0000 7f08485c1640 1 -- 172.21.15.28:0/343049345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f084019d2f0 con 0x7f0840107720 2024-04-25T01:42:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.118+0000 7f08377fe640 1 -- 172.21.15.28:0/343049345 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0838005020 con 0x7f0840107720 2024-04-25T01:42:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.118+0000 7f08377fe640 1 -- 172.21.15.28:0/343049345 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08380116d0 con 0x7f0840107720 2024-04-25T01:42:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.118+0000 7f08485c1640 1 -- 172.21.15.28:0/343049345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08401a4b80 con 0x7f0840107720 2024-04-25T01:42:39.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.121+0000 7f08377fe640 1 -- 172.21.15.28:0/343049345 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f08380054d0 con 0x7f0840107720 2024-04-25T01:42:39.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.121+0000 7f084b3a9740 1 Processor -- start 2024-04-25T01:42:39.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.121+0000 7f084b3a9740 1 -- start start 2024-04-25T01:42:39.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.121+0000 7f084b3a9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb813142d0 0x55eb813146b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.121+0000 7f084b3a9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eb81314c80 0x55eb81323440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.121+0000 7f084b3a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81325d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.121+0000 7f084b3a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55eb81279670 con 0x55eb81323980 2024-04-25T01:42:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.121+0000 7f084b3a9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55eb81221970 con 0x55eb81314c80 2024-04-25T01:42:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.121+0000 7f084b3a9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55eb81275630 con 0x55eb813142d0 2024-04-25T01:42:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.122+0000 7f0845b5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81325d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.122+0000 7f08377fe640 1 --2- 172.21.15.28:0/343049345 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f082c0a2d00 0x7f082c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.122+0000 7f0845b5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81325d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47982/0 (socket says 172.21.15.28:47982) 2024-04-25T01:42:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.122+0000 7f0845b5c640 1 -- 172.21.15.28:0/1401552544 learned_addr learned my addr 172.21.15.28:0/1401552544 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.122+0000 7f084635d640 1 --2- 172.21.15.28:0/1401552544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eb81314c80 0x55eb81323440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.122+0000 7f0845b5c640 1 --2- 172.21.15.28:0/1401552544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb813142d0 0x55eb813146b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.122+0000 7f0846b5e640 1 --2- 172.21.15.28:0/343049345 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f082c0a2d00 0x7f082c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.122+0000 7f0845b5c640 1 -- 172.21.15.28:0/1401552544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb813142d0 msgr2=0x55eb813146b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.122+0000 7f0845b5c640 1 --2- 172.21.15.28:0/1401552544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb813142d0 0x55eb813146b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.122+0000 7f0845b5c640 1 -- 172.21.15.28:0/1401552544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eb81314c80 msgr2=0x55eb81323440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.122+0000 7f0845b5c640 1 --2- 172.21.15.28:0/1401552544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eb81314c80 0x55eb81323440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.122+0000 7f0845b5c640 1 -- 172.21.15.28:0/1401552544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb81245400 con 0x55eb81323980 2024-04-25T01:42:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.122+0000 7f08377fe640 1 -- 172.21.15.28:0/343049345 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f08380cf9c0 con 0x7f0840107720 2024-04-25T01:42:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.122+0000 7f0845b5c640 1 --2- 172.21.15.28:0/1401552544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81325d70 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f0830013890 tx=0x7f083000b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d684665bc2ca66e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.122+0000 7f0846b5e640 1 --2- 172.21.15.28:0/343049345 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f082c0a2d00 0x7f082c0a51c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f08380cce30 tx=0x7f0838005440 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.122+0000 7f0834bfe640 1 -- 172.21.15.28:0/1401552544 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083000e8f0 con 0x55eb81323980 2024-04-25T01:42:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.122+0000 7f0834bfe640 1 -- 172.21.15.28:0/1401552544 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0830013e10 con 0x55eb81323980 2024-04-25T01:42:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.122+0000 7f0834bfe640 1 -- 172.21.15.28:0/1401552544 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083001ec60 con 0x55eb81323980 2024-04-25T01:42:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.123+0000 7f084b3a9740 1 -- 172.21.15.28:0/1401552544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 msgr2=0x55eb81325d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.123+0000 7f084b3a9740 1 --2- 172.21.15.28:0/1401552544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81325d70 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f0830013890 tx=0x7f083000b420 comp rx=0 tx=0).stop 2024-04-25T01:42:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.123+0000 7f084b3a9740 1 -- 172.21.15.28:0/1401552544 shutdown_connections 2024-04-25T01:42:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.123+0000 7f084b3a9740 1 --2- 172.21.15.28:0/1401552544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81325d70 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.123+0000 7f084b3a9740 1 --2- 172.21.15.28:0/1401552544 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eb81314c80 0x55eb81323440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.123+0000 7f084b3a9740 1 --2- 172.21.15.28:0/1401552544 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb813142d0 0x55eb813146b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.123+0000 7f084b3a9740 1 -- 172.21.15.28:0/1401552544 >> 172.21.15.28:0/1401552544 conn(0x55eb813108a0 msgr2=0x55eb81310c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.123+0000 7f084b3a9740 1 -- 172.21.15.28:0/1401552544 shutdown_connections 2024-04-25T01:42:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.123+0000 7f084b3a9740 1 -- 172.21.15.28:0/1401552544 wait complete. 2024-04-25T01:42:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.124+0000 7f084b3a9740 1 -- start start 2024-04-25T01:42:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.124+0000 7f084b3a9740 1 Processor -- start 2024-04-25T01:42:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.124+0000 7f084b3a9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eb813142d0 0x55eb813233a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.124+0000 7f084b3a9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb81314c80 0x55eb81320690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.124+0000 7f084b3a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81320ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.124+0000 7f084b3a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55eb81326740 con 0x55eb81323980 2024-04-25T01:42:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.124+0000 7f084b3a9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55eb81235050 con 0x55eb813142d0 2024-04-25T01:42:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.124+0000 7f084b3a9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55eb813268c0 con 0x55eb81314c80 2024-04-25T01:42:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.124+0000 7f084b3a9740 10 client.0 _ll_register_callbacks cb 0x55eb80f8cbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.124+0000 7f0845b5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81320ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.124+0000 7f0845b5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81320ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47994/0 (socket says 172.21.15.28:47994) 2024-04-25T01:42:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.124+0000 7f0845b5c640 1 -- 172.21.15.28:0/1775503334 learned_addr learned my addr 172.21.15.28:0/1775503334 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.124+0000 7f084635d640 1 --2- 172.21.15.28:0/1775503334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb81314c80 0x55eb81320690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.124+0000 7f0845b5c640 1 --2- 172.21.15.28:0/1775503334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eb813142d0 0x55eb813233a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.124+0000 7f0845b5c640 1 -- 172.21.15.28:0/1775503334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb81314c80 msgr2=0x55eb81320690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.124+0000 7f0845b5c640 1 --2- 172.21.15.28:0/1775503334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb81314c80 0x55eb81320690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.124+0000 7f0845b5c640 1 -- 172.21.15.28:0/1775503334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eb813142d0 msgr2=0x55eb813233a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.124+0000 7f0845b5c640 1 --2- 172.21.15.28:0/1775503334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eb813142d0 0x55eb813233a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.124+0000 7f0845b5c640 1 -- 172.21.15.28:0/1775503334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb81206dc0 con 0x55eb81323980 2024-04-25T01:42:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.125+0000 7f0845b5c640 1 --2- 172.21.15.28:0/1775503334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81320ca0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f083001e940 tx=0x7f0830001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19dba4145e74479a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.125+0000 7f0829ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.125+0000 7f0829ffb640 1 -- 172.21.15.28:0/1775503334 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0830022030 con 0x55eb81323980 2024-04-25T01:42:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.125+0000 7f084b3a9740 1 -- 172.21.15.28:0/1775503334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0830013e10 con 0x55eb81323980 2024-04-25T01:42:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.125+0000 7f0829ffb640 1 -- 172.21.15.28:0/1775503334 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f083001e430 con 0x55eb81323980 2024-04-25T01:42:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.125+0000 7f0829ffb640 1 -- 172.21.15.28:0/1775503334 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0830034050 con 0x55eb81323980 2024-04-25T01:42:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.125+0000 7f084b3a9740 10 client.19209 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.125+0000 7f084b3a9740 1 -- 172.21.15.28:0/1775503334 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eb80f8d3f0 con 0x55eb81323980 2024-04-25T01:42:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.125+0000 7f0829ffb640 1 -- 172.21.15.28:0/1775503334 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0830013e10 con 0x55eb81323980 2024-04-25T01:42:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.125+0000 7f0829ffb640 1 -- 172.21.15.28:0/1775503334 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f083000f040 con 0x55eb81323980 2024-04-25T01:42:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.125+0000 7f084b3a9740 10 client.19209 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.125+0000 7f084b3a9740 10 client.19209 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:42:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.125+0000 7f084b3a9740 20 client.19209 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.125+0000 7f084b3a9740 1 --2- 172.21.15.28:0/1775503334 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55eb81333930 0x55eb81335d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.125+0000 7f084b3a9740 4 client.19209 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.125+0000 7f084b3a9740 1 -- 172.21.15.28:0/1775503334 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb81286bb0 con 0x55eb81333930 2024-04-25T01:42:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.126+0000 7f084635d640 1 --2- 172.21.15.28:0/1775503334 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55eb81333930 0x55eb81335d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.126+0000 7f084635d640 1 --2- 172.21.15.28:0/1775503334 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55eb81333930 0x55eb81335d20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bacc5a55114e685e server_cookie=236217c46c84d23f in_seq=0 out_seq=0 2024-04-25T01:42:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.126+0000 7f0829ffb640 10 client.19209 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:42:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.127+0000 7f0829ffb640 1 -- 172.21.15.28:0/1775503334 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eb81286bb0 con 0x55eb81333930 2024-04-25T01:42:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.127+0000 7f0829ffb640 10 client.19209 handle_command_reply: tid=0 2024-04-25T01:42:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.127+0000 7f084b3a9740 1 client.19209 shutdown 2024-04-25T01:42:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.127+0000 7f084b3a9740 1 -- 172.21.15.28:0/1775503334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 msgr2=0x55eb81320ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.127+0000 7f084b3a9740 1 --2- 172.21.15.28:0/1775503334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81320ca0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f083001e940 tx=0x7f0830001e50 comp rx=0 tx=0).stop 2024-04-25T01:42:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.127+0000 7f084b3a9740 1 -- 172.21.15.28:0/1775503334 shutdown_connections 2024-04-25T01:42:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.127+0000 7f084b3a9740 1 --2- 172.21.15.28:0/1775503334 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55eb81333930 0x55eb81335d20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.127+0000 7f084b3a9740 1 --2- 172.21.15.28:0/1775503334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81320ca0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.127+0000 7f0829ffb640 0 client.19209 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:42:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.127+0000 7f084b3a9740 1 --2- 172.21.15.28:0/1775503334 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb81314c80 0x55eb81320690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.127+0000 7f084b3a9740 1 --2- 172.21.15.28:0/1775503334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eb813142d0 0x55eb813233a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.127+0000 7f084b3a9740 1 -- 172.21.15.28:0/1775503334 >> 172.21.15.28:0/1775503334 conn(0x55eb813108a0 msgr2=0x55eb81316710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.127+0000 7f084b3a9740 1 -- 172.21.15.28:0/1775503334 shutdown_connections 2024-04-25T01:42:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.127+0000 7f084b3a9740 1 -- 172.21.15.28:0/1775503334 wait complete. 2024-04-25T01:42:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.127+0000 7f084b3a9740 20 client.19209 trim_cache size 0 max 16384 2024-04-25T01:42:39.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.140+0000 7f084b3a9740 1 Processor -- start 2024-04-25T01:42:39.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.140+0000 7f084b3a9740 1 -- start start 2024-04-25T01:42:39.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.140+0000 7f084b3a9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eb813142d0 0x55eb8130fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.140+0000 7f084b3a9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb81314c80 0x55eb81322ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.140+0000 7f084b3a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81338d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.140+0000 7f084b3a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08401acaf0 con 0x55eb81323980 2024-04-25T01:42:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.140+0000 7f084b3a9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f08401ae630 con 0x55eb813142d0 2024-04-25T01:42:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.140+0000 7f084b3a9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f08401ad1c0 con 0x55eb81314c80 2024-04-25T01:42:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.141+0000 7f0845b5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81338d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.141+0000 7f0845b5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81338d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47996/0 (socket says 172.21.15.28:47996) 2024-04-25T01:42:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.141+0000 7f0845b5c640 1 -- 172.21.15.28:0/883358513 learned_addr learned my addr 172.21.15.28:0/883358513 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.141+0000 7f084635d640 1 --2- 172.21.15.28:0/883358513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb81314c80 0x55eb81322ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.141+0000 7f0845b5c640 1 -- 172.21.15.28:0/883358513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb81314c80 msgr2=0x55eb81322ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.141+0000 7f0845b5c640 1 --2- 172.21.15.28:0/883358513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb81314c80 0x55eb81322ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.141+0000 7f0845b5c640 1 -- 172.21.15.28:0/883358513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eb813142d0 msgr2=0x55eb8130fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.141+0000 7f0845b5c640 1 --2- 172.21.15.28:0/883358513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eb813142d0 0x55eb8130fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.141+0000 7f0845b5c640 1 -- 172.21.15.28:0/883358513 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb81314a20 con 0x55eb81323980 2024-04-25T01:42:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.141+0000 7f084635d640 1 --2- 172.21.15.28:0/883358513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb81314c80 0x55eb81322ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.141+0000 7f0845b5c640 1 --2- 172.21.15.28:0/883358513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81338d40 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f083000bda0 tx=0x7f083000bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e77980c396070295 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.141+0000 7f082bfff640 1 -- 172.21.15.28:0/883358513 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0830022030 con 0x55eb81323980 2024-04-25T01:42:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.141+0000 7f082bfff640 1 -- 172.21.15.28:0/883358513 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0830023040 con 0x55eb81323980 2024-04-25T01:42:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.141+0000 7f082bfff640 1 -- 172.21.15.28:0/883358513 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0830004020 con 0x55eb81323980 2024-04-25T01:42:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.142+0000 7f084b3a9740 1 -- 172.21.15.28:0/883358513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 msgr2=0x55eb81338d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.142+0000 7f084b3a9740 1 --2- 172.21.15.28:0/883358513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81338d40 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f083000bda0 tx=0x7f083000bdd0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.142+0000 7f084b3a9740 1 -- 172.21.15.28:0/883358513 shutdown_connections 2024-04-25T01:42:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.142+0000 7f084b3a9740 1 --2- 172.21.15.28:0/883358513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81338d40 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.142+0000 7f084b3a9740 1 --2- 172.21.15.28:0/883358513 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb81314c80 0x55eb81322ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.142+0000 7f084b3a9740 1 --2- 172.21.15.28:0/883358513 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eb813142d0 0x55eb8130fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.142+0000 7f084b3a9740 1 -- 172.21.15.28:0/883358513 >> 172.21.15.28:0/883358513 conn(0x55eb813108a0 msgr2=0x55eb81310c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.142+0000 7f084b3a9740 1 -- 172.21.15.28:0/883358513 shutdown_connections 2024-04-25T01:42:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.142+0000 7f084b3a9740 1 -- 172.21.15.28:0/883358513 wait complete. 2024-04-25T01:42:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.142+0000 7f084b3a9740 1 -- start start 2024-04-25T01:42:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.142+0000 7f084b3a9740 1 Processor -- start 2024-04-25T01:42:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.142+0000 7f084b3a9740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb813142d0 0x55eb81338e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.142+0000 7f084b3a9740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eb81314c80 0x55eb81336340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.142+0000 7f084b3a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81336880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.142+0000 7f084b3a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08401adf60 con 0x55eb81323980 2024-04-25T01:42:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.142+0000 7f084b3a9740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f08401ad890 con 0x55eb81314c80 2024-04-25T01:42:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.142+0000 7f084b3a9740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f08401aaaf0 con 0x55eb813142d0 2024-04-25T01:42:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.142+0000 7f084b3a9740 10 client.0 _ll_register_callbacks cb 0x7f083000be40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.143+0000 7f0845b5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81336880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.143+0000 7f0845b5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81336880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48000/0 (socket says 172.21.15.28:48000) 2024-04-25T01:42:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.143+0000 7f0845b5c640 1 -- 172.21.15.28:0/109658763 learned_addr learned my addr 172.21.15.28:0/109658763 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.143+0000 7f084635d640 1 --2- 172.21.15.28:0/109658763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eb81314c80 0x55eb81336340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.143+0000 7f0845b5c640 1 --2- 172.21.15.28:0/109658763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb813142d0 0x55eb81338e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.143+0000 7f0845b5c640 1 -- 172.21.15.28:0/109658763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb813142d0 msgr2=0x55eb81338e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.143+0000 7f0845b5c640 1 --2- 172.21.15.28:0/109658763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb813142d0 0x55eb81338e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.143+0000 7f0845b5c640 1 -- 172.21.15.28:0/109658763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eb81314c80 msgr2=0x55eb81336340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.143+0000 7f0845b5c640 1 --2- 172.21.15.28:0/109658763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eb81314c80 0x55eb81336340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.143+0000 7f0845b5c640 1 -- 172.21.15.28:0/109658763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb810577d0 con 0x55eb81323980 2024-04-25T01:42:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.143+0000 7f0845b5c640 1 --2- 172.21.15.28:0/109658763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81336880 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f0830005940 tx=0x7f0830013f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57a04450574d237 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.143+0000 7f0829ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.143+0000 7f0829ffb640 1 -- 172.21.15.28:0/109658763 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0830022070 con 0x55eb81323980 2024-04-25T01:42:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.143+0000 7f0829ffb640 1 -- 172.21.15.28:0/109658763 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f083000e8c0 con 0x55eb81323980 2024-04-25T01:42:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.143+0000 7f084b3a9740 1 -- 172.21.15.28:0/109658763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0830023040 con 0x55eb81323980 2024-04-25T01:42:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.143+0000 7f0829ffb640 1 -- 172.21.15.28:0/109658763 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0830047050 con 0x55eb81323980 2024-04-25T01:42:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.143+0000 7f084b3a9740 10 client.19215 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.143+0000 7f084b3a9740 1 -- 172.21.15.28:0/109658763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eb81399600 con 0x55eb81323980 2024-04-25T01:42:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.144+0000 7f0829ffb640 1 -- 172.21.15.28:0/109658763 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0830023040 con 0x55eb81323980 2024-04-25T01:42:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.144+0000 7f0829ffb640 1 -- 172.21.15.28:0/109658763 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0830005440 con 0x55eb81323980 2024-04-25T01:42:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.144+0000 7f084b3a9740 10 client.19215 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.144+0000 7f084b3a9740 10 client.19215 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:42:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.144+0000 7f084b3a9740 20 client.19215 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.144+0000 7f084b3a9740 1 --2- 172.21.15.28:0/109658763 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55eb81399c40 0x55eb8139c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.144+0000 7f084b3a9740 4 client.19215 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:42:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.144+0000 7f084b3a9740 1 -- 172.21.15.28:0/109658763 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eb81280f00 con 0x55eb81399c40 2024-04-25T01:42:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.144+0000 7f084635d640 1 --2- 172.21.15.28:0/109658763 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55eb81399c40 0x55eb8139c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.145+0000 7f0829ffb640 10 client.19215 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:42:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.145+0000 7f084635d640 1 --2- 172.21.15.28:0/109658763 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55eb81399c40 0x55eb8139c060 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3793142653d442ef server_cookie=ca5cbfeb75085ca3 in_seq=0 out_seq=0 2024-04-25T01:42:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.145+0000 7f0829ffb640 1 -- 172.21.15.28:0/109658763 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eb81280f00 con 0x55eb81399c40 2024-04-25T01:42:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.145+0000 7f0829ffb640 10 client.19215 handle_command_reply: tid=0 2024-04-25T01:42:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.145+0000 7f084b3a9740 1 client.19215 shutdown 2024-04-25T01:42:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.145+0000 7f084b3a9740 1 -- 172.21.15.28:0/109658763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 msgr2=0x55eb81336880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.145+0000 7f084b3a9740 1 --2- 172.21.15.28:0/109658763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81336880 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f0830005940 tx=0x7f0830013f50 comp rx=0 tx=0).stop 2024-04-25T01:42:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.145+0000 7f084b3a9740 1 -- 172.21.15.28:0/109658763 shutdown_connections 2024-04-25T01:42:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.145+0000 7f084b3a9740 1 --2- 172.21.15.28:0/109658763 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x55eb81399c40 0x55eb8139c060 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.145+0000 7f084b3a9740 1 --2- 172.21.15.28:0/109658763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb81323980 0x55eb81336880 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.145+0000 7f084b3a9740 1 --2- 172.21.15.28:0/109658763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55eb81314c80 0x55eb81336340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.145+0000 7f084b3a9740 1 --2- 172.21.15.28:0/109658763 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55eb813142d0 0x55eb81338e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.145+0000 7f0829ffb640 0 client.19215 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:42:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.145+0000 7f084b3a9740 1 -- 172.21.15.28:0/109658763 >> 172.21.15.28:0/109658763 conn(0x55eb813108a0 msgr2=0x55eb8131bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.145+0000 7f084b3a9740 1 -- 172.21.15.28:0/109658763 shutdown_connections 2024-04-25T01:42:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.145+0000 7f084b3a9740 1 -- 172.21.15.28:0/109658763 wait complete. 2024-04-25T01:42:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.145+0000 7f084b3a9740 20 client.19215 trim_cache size 0 max 16384 2024-04-25T01:42:39.139 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:42:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.146+0000 7f08485c1640 1 -- 172.21.15.28:0/343049345 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f082c0a2d00 msgr2=0x7f082c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.146+0000 7f08485c1640 1 --2- 172.21.15.28:0/343049345 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f082c0a2d00 0x7f082c0a51c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f08380cce30 tx=0x7f0838005440 comp rx=0 tx=0).stop 2024-04-25T01:42:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.146+0000 7f08485c1640 1 -- 172.21.15.28:0/343049345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0840107720 msgr2=0x7f084019c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.146+0000 7f08485c1640 1 --2- 172.21.15.28:0/343049345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0840107720 0x7f084019c8d0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f083800eaa0 tx=0x7f083800ef60 comp rx=0 tx=0).stop 2024-04-25T01:42:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.146+0000 7f08485c1640 1 -- 172.21.15.28:0/343049345 shutdown_connections 2024-04-25T01:42:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.146+0000 7f08485c1640 1 --2- 172.21.15.28:0/343049345 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f082c0a2d00 0x7f082c0a51c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.146+0000 7f08485c1640 1 --2- 172.21.15.28:0/343049345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0840107720 0x7f084019c8d0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.146+0000 7f08485c1640 1 --2- 172.21.15.28:0/343049345 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08401020d0 0x7f0840198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.146+0000 7f08485c1640 1 --2- 172.21.15.28:0/343049345 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0840101710 0x7f0840198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.146+0000 7f08485c1640 1 -- 172.21.15.28:0/343049345 >> 172.21.15.28:0/343049345 conn(0x7f08400fd180 msgr2=0x7f08400fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:39.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.146+0000 7f08485c1640 1 -- 172.21.15.28:0/343049345 shutdown_connections 2024-04-25T01:42:39.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.146+0000 7f08485c1640 1 -- 172.21.15.28:0/343049345 wait complete. 2024-04-25T01:42:39.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:39 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:42:39.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:39 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:42:39.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:39 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:42:39.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:39 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:42:39.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:39 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:42:39.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:39 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2937879858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:42:39.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:42:39.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:39 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:42:39.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:39 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:42:39.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:39 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:42:39.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:39 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:42:39.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:39 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:42:39.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:39 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2937879858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:42:39.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.260+0000 7f6ff2c5d640 1 -- 172.21.15.28:0/1482727337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fec101710 msgr2=0x7f6fec101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:39.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.260+0000 7f6ff2c5d640 1 --2- 172.21.15.28:0/1482727337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fec101710 0x7f6fec101b90 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0009930 tx=0x7f6fe002a420 comp rx=0 tx=0).stop 2024-04-25T01:42:39.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.260+0000 7f6ff2c5d640 1 -- 172.21.15.28:0/1482727337 shutdown_connections 2024-04-25T01:42:39.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.260+0000 7f6ff2c5d640 1 --2- 172.21.15.28:0/1482727337 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fec1020d0 0x7f6fec10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.260+0000 7f6ff2c5d640 1 --2- 172.21.15.28:0/1482727337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fec101710 0x7f6fec101b90 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.260+0000 7f6ff2c5d640 1 --2- 172.21.15.28:0/1482727337 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fec107720 0x7f6fec107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.260+0000 7f6ff2c5d640 1 -- 172.21.15.28:0/1482727337 >> 172.21.15.28:0/1482727337 conn(0x7f6fec078b90 msgr2=0x7f6fec078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:39.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.260+0000 7f6ff2c5d640 1 -- 172.21.15.28:0/1482727337 shutdown_connections 2024-04-25T01:42:39.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.260+0000 7f6ff2c5d640 1 -- 172.21.15.28:0/1482727337 wait complete. 2024-04-25T01:42:39.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.261+0000 7f6ff2c5d640 1 Processor -- start 2024-04-25T01:42:39.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.261+0000 7f6ff2c5d640 1 -- start start 2024-04-25T01:42:39.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.261+0000 7f6ff2c5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fec101710 0x7f6fec1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.261+0000 7f6ff2c5d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fec1020d0 0x7f6fec1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.261+0000 7f6ff2c5d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fec107720 0x7f6fec19c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.261+0000 7f6ff2c5d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec111b30 con 0x7f6fec101710 2024-04-25T01:42:39.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.261+0000 7f6ff2c5d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec1119b0 con 0x7f6fec107720 2024-04-25T01:42:39.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.261+0000 7f6ff2c5d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec111cb0 con 0x7f6fec1020d0 2024-04-25T01:42:39.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.261+0000 7f6ff09f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fec101710 0x7f6fec1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.261+0000 7f6ff09f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fec101710 0x7f6fec1980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48012/0 (socket says 172.21.15.28:48012) 2024-04-25T01:42:39.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.261+0000 7f6ff09f9640 1 -- 172.21.15.28:0/803346381 learned_addr learned my addr 172.21.15.28:0/803346381 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:39.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.261+0000 7f6ff11fa640 1 --2- 172.21.15.28:0/803346381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fec107720 0x7f6fec19c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.261+0000 7f6febfff640 1 --2- 172.21.15.28:0/803346381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fec1020d0 0x7f6fec1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.262+0000 7f6ff09f9640 1 -- 172.21.15.28:0/803346381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fec1020d0 msgr2=0x7f6fec1985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:39.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.262+0000 7f6ff09f9640 1 --2- 172.21.15.28:0/803346381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fec1020d0 0x7f6fec1985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.262+0000 7f6ff09f9640 1 -- 172.21.15.28:0/803346381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fec107720 msgr2=0x7f6fec19c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:39.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.262+0000 7f6ff09f9640 1 --2- 172.21.15.28:0/803346381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fec107720 0x7f6fec19c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.262+0000 7f6ff09f9640 1 -- 172.21.15.28:0/803346381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fec19d0a0 con 0x7f6fec101710 2024-04-25T01:42:39.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.262+0000 7f6ff09f9640 1 --2- 172.21.15.28:0/803346381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fec101710 0x7f6fec1980b0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc00d770 tx=0x7f6fdc00db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:39.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.262+0000 7f6fe9ffb640 1 -- 172.21.15.28:0/803346381 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc017070 con 0x7f6fec101710 2024-04-25T01:42:39.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.262+0000 7f6fe9ffb640 1 -- 172.21.15.28:0/803346381 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fdc009820 con 0x7f6fec101710 2024-04-25T01:42:39.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.262+0000 7f6ff2c5d640 1 -- 172.21.15.28:0/803346381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fec19d390 con 0x7f6fec101710 2024-04-25T01:42:39.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.262+0000 7f6fe9ffb640 1 -- 172.21.15.28:0/803346381 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc004cd0 con 0x7f6fec101710 2024-04-25T01:42:39.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.262+0000 7f6ff2c5d640 1 -- 172.21.15.28:0/803346381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fec1a4c20 con 0x7f6fec101710 2024-04-25T01:42:39.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.264+0000 7f6fe9ffb640 1 -- 172.21.15.28:0/803346381 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6fdc0099d0 con 0x7f6fec101710 2024-04-25T01:42:39.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.265+0000 7f6ff5969740 1 Processor -- start 2024-04-25T01:42:39.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.265+0000 7f6ff5969740 1 -- start start 2024-04-25T01:42:39.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.265+0000 7f6fe9ffb640 1 --2- 172.21.15.28:0/803346381 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6fd40a2d80 0x7f6fd40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.265+0000 7f6ff5969740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f026580 0x56282f026960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.265+0000 7f6ff5969740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f026ea0 0x56282f031030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.265+0000 7f6ff5969740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f031570 0x56282f033960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.265+0000 7f6ff5969740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56282ef8b670 con 0x56282f031570 2024-04-25T01:42:39.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.265+0000 7f6ff5969740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56282ef33970 con 0x56282f026ea0 2024-04-25T01:42:39.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.265+0000 7f6ff5969740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56282ef87630 con 0x56282f026580 2024-04-25T01:42:39.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.265+0000 7f6ff11fa640 1 --2- 172.21.15.28:0/803346381 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6fd40a2d80 0x7f6fd40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.265+0000 7f6fe9ffb640 1 -- 172.21.15.28:0/803346381 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f6fdc0cf100 con 0x7f6fec101710 2024-04-25T01:42:39.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.265+0000 7f6ff09f9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f026580 0x56282f026960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.265+0000 7f6ff11fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f031570 0x56282f033960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.265+0000 7f6ff09f9640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f026580 0x56282f026960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.130:3300/0 says I am v2:172.21.15.28:47380/0 (socket says 172.21.15.28:47380) 2024-04-25T01:42:39.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.266+0000 7f6ff09f9640 1 -- 172.21.15.28:0/3343674410 learned_addr learned my addr 172.21.15.28:0/3343674410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:39.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.266+0000 7f6febfff640 1 --2- 172.21.15.28:0/3343674410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f026ea0 0x56282f031030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.266+0000 7f6ff09f9640 1 -- 172.21.15.28:0/3343674410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f026ea0 msgr2=0x56282f031030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:39.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.266+0000 7f6ff09f9640 1 --2- 172.21.15.28:0/3343674410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f026ea0 0x56282f031030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.266+0000 7f6ff09f9640 1 -- 172.21.15.28:0/3343674410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f031570 msgr2=0x56282f033960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:39.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.266+0000 7f6ff09f9640 1 --2- 172.21.15.28:0/3343674410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f031570 0x56282f033960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.266+0000 7f6ff09f9640 1 -- 172.21.15.28:0/3343674410 --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56282ef57400 con 0x56282f026580 2024-04-25T01:42:39.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.266+0000 7f6ff11fa640 1 --2- 172.21.15.28:0/803346381 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6fd40a2d80 0x7f6fd40a5240 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4005fd0 tx=0x7f6fe4013b80 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:39.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.266+0000 7f6ff09f9640 1 --2- 172.21.15.28:0/3343674410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f026580 0x56282f026960 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc097c00 tx=0x7f6fdc097600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b2b00bc5bd116b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:39.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.266+0000 7f6fd2ffd640 1 -- 172.21.15.28:0/3343674410 <== mon.1 v2:172.21.15.130:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc020050 con 0x56282f026580 2024-04-25T01:42:39.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.266+0000 7f6fd2ffd640 1 -- 172.21.15.28:0/3343674410 <== mon.1 v2:172.21.15.130:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fdc0a3ea0 con 0x56282f026580 2024-04-25T01:42:39.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.266+0000 7f6fd2ffd640 1 -- 172.21.15.28:0/3343674410 <== mon.1 v2:172.21.15.130:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc091c50 con 0x56282f026580 2024-04-25T01:42:39.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.267+0000 7f6ff5969740 1 -- 172.21.15.28:0/3343674410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f026580 msgr2=0x56282f026960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.267+0000 7f6ff5969740 1 --2- 172.21.15.28:0/3343674410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f026580 0x56282f026960 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc097c00 tx=0x7f6fdc097600 comp rx=0 tx=0).stop 2024-04-25T01:42:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.267+0000 7f6ff5969740 1 -- 172.21.15.28:0/3343674410 shutdown_connections 2024-04-25T01:42:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.267+0000 7f6ff5969740 1 --2- 172.21.15.28:0/3343674410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f031570 0x56282f033960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.267+0000 7f6ff5969740 1 --2- 172.21.15.28:0/3343674410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f026ea0 0x56282f031030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.267+0000 7f6ff5969740 1 --2- 172.21.15.28:0/3343674410 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f026580 0x56282f026960 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.267+0000 7f6ff5969740 1 -- 172.21.15.28:0/3343674410 >> 172.21.15.28:0/3343674410 conn(0x56282f0228a0 msgr2=0x56282f022c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.267+0000 7f6ff5969740 1 -- 172.21.15.28:0/3343674410 shutdown_connections 2024-04-25T01:42:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.267+0000 7f6ff5969740 1 -- 172.21.15.28:0/3343674410 wait complete. 2024-04-25T01:42:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.267+0000 7f6ff5969740 1 -- start start 2024-04-25T01:42:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.267+0000 7f6ff5969740 1 Processor -- start 2024-04-25T01:42:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6ff5969740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f026580 0x56282f030f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6ff5969740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f026ea0 0x56282f02e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6ff5969740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f031570 0x56282f02e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6ff5969740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56282f0342a0 con 0x56282f026ea0 2024-04-25T01:42:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6ff5969740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56282ef47050 con 0x56282f031570 2024-04-25T01:42:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6ff5969740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56282f034420 con 0x56282f026580 2024-04-25T01:42:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6ff5969740 10 client.0 _ll_register_callbacks cb 0x56282ec9ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6febfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f026ea0 0x56282f02e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6febfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f026ea0 0x56282f02e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48024/0 (socket says 172.21.15.28:48024) 2024-04-25T01:42:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6febfff640 1 -- 172.21.15.28:0/1179522146 learned_addr learned my addr 172.21.15.28:0/1179522146 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6ff09f9640 1 --2- 172.21.15.28:0/1179522146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f026580 0x56282f030f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6ff11fa640 1 --2- 172.21.15.28:0/1179522146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f031570 0x56282f02e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6febfff640 1 -- 172.21.15.28:0/1179522146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f026580 msgr2=0x56282f030f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6febfff640 1 --2- 172.21.15.28:0/1179522146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f026580 0x56282f030f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6febfff640 1 -- 172.21.15.28:0/1179522146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f031570 msgr2=0x56282f02e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6febfff640 1 --2- 172.21.15.28:0/1179522146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f031570 0x56282f02e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6febfff640 1 -- 172.21.15.28:0/1179522146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56282ef18dc0 con 0x56282f026ea0 2024-04-25T01:42:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6ff09f9640 1 --2- 172.21.15.28:0/1179522146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f026580 0x56282f030f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:42:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6febfff640 1 --2- 172.21.15.28:0/1179522146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f026ea0 0x56282f02e200 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0000c00 tx=0x7f6fe0031920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8e6091e092bb71c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.268+0000 7f6fcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.269+0000 7f6fcbfff640 1 -- 172.21.15.28:0/1179522146 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0035040 con 0x56282f026ea0 2024-04-25T01:42:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.269+0000 7f6fcbfff640 1 -- 172.21.15.28:0/1179522146 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f6fe0004240 con 0x56282f026ea0 2024-04-25T01:42:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.269+0000 7f6ff5969740 1 -- 172.21.15.28:0/1179522146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fdc0a3ea0 con 0x56282f026ea0 2024-04-25T01:42:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.269+0000 7f6fcbfff640 1 -- 172.21.15.28:0/1179522146 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6fe004f050 con 0x56282f026ea0 2024-04-25T01:42:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.269+0000 7f6ff5969740 10 client.19227 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.269+0000 7f6ff5969740 1 -- 172.21.15.28:0/1179522146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56282ec9f3f0 con 0x56282f026ea0 2024-04-25T01:42:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.269+0000 7f6fcbfff640 1 -- 172.21.15.28:0/1179522146 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fdc0a3ea0 con 0x56282f026ea0 2024-04-25T01:42:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.269+0000 7f6fcbfff640 1 -- 172.21.15.28:0/1179522146 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6fe0004b00 con 0x56282f026ea0 2024-04-25T01:42:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.269+0000 7f6ff5969740 10 client.19227 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.269+0000 7f6ff5969740 10 client.19227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.269+0000 7f6ff5969740 20 client.19227 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.269+0000 7f6ff5969740 1 --2- 172.21.15.28:0/1179522146 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56282f041540 0x56282f0439a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.269+0000 7f6ff5969740 4 client.19227 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.269+0000 7f6ff5969740 1 -- 172.21.15.28:0/1179522146 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56282ef98bb0 con 0x56282f041540 2024-04-25T01:42:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.270+0000 7f6ff09f9640 1 --2- 172.21.15.28:0/1179522146 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56282f041540 0x56282f0439a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.270+0000 7f6ff09f9640 1 --2- 172.21.15.28:0/1179522146 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56282f041540 0x56282f0439a0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3eb5cd6428a8902d server_cookie=5eb18ccfba628bdb in_seq=0 out_seq=0 2024-04-25T01:42:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.270+0000 7f6fcbfff640 10 client.19227 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.271+0000 7f6fcbfff640 1 -- 172.21.15.28:0/1179522146 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56282ef98bb0 con 0x56282f041540 2024-04-25T01:42:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.271+0000 7f6fcbfff640 10 client.19227 handle_command_reply: tid=0 2024-04-25T01:42:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.271+0000 7f6ff5969740 1 client.19227 shutdown 2024-04-25T01:42:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.271+0000 7f6ff5969740 1 -- 172.21.15.28:0/1179522146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f026ea0 msgr2=0x56282f02e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.271+0000 7f6ff5969740 1 --2- 172.21.15.28:0/1179522146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f026ea0 0x56282f02e200 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0000c00 tx=0x7f6fe0031920 comp rx=0 tx=0).stop 2024-04-25T01:42:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.271+0000 7f6ff5969740 1 -- 172.21.15.28:0/1179522146 shutdown_connections 2024-04-25T01:42:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.271+0000 7f6ff5969740 1 --2- 172.21.15.28:0/1179522146 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56282f041540 0x56282f0439a0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.271+0000 7f6ff5969740 1 --2- 172.21.15.28:0/1179522146 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f031570 0x56282f02e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.271+0000 7f6ff5969740 1 --2- 172.21.15.28:0/1179522146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f026ea0 0x56282f02e200 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.271+0000 7f6ff5969740 1 --2- 172.21.15.28:0/1179522146 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f026580 0x56282f030f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.271+0000 7f6fcbfff640 0 client.19227 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.271+0000 7f6ff5969740 1 -- 172.21.15.28:0/1179522146 >> 172.21.15.28:0/1179522146 conn(0x56282f0228a0 msgr2=0x56282f0286a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.271+0000 7f6ff5969740 1 -- 172.21.15.28:0/1179522146 shutdown_connections 2024-04-25T01:42:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.271+0000 7f6ff5969740 1 -- 172.21.15.28:0/1179522146 wait complete. 2024-04-25T01:42:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.271+0000 7f6ff5969740 20 client.19227 trim_cache size 0 max 16384 2024-04-25T01:42:39.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.284+0000 7f6ff5969740 1 Processor -- start 2024-04-25T01:42:39.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.284+0000 7f6ff5969740 1 -- start start 2024-04-25T01:42:39.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.284+0000 7f6ff5969740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f026580 0x56282f03d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.284+0000 7f6ff5969740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f026ea0 0x56282f030a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.284+0000 7f6ff5969740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f031570 0x56282f049270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.284+0000 7f6ff5969740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec1aeb90 con 0x56282f031570 2024-04-25T01:42:39.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.284+0000 7f6ff5969740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec1b06d0 con 0x56282f026580 2024-04-25T01:42:39.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.284+0000 7f6ff5969740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec1af260 con 0x56282f026ea0 2024-04-25T01:42:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.285+0000 7f6ff11fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f031570 0x56282f049270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.285+0000 7f6ff09f9640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f026580 0x56282f03d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.285+0000 7f6ff11fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f031570 0x56282f049270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48030/0 (socket says 172.21.15.28:48030) 2024-04-25T01:42:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.285+0000 7f6ff11fa640 1 -- 172.21.15.28:0/4027111170 learned_addr learned my addr 172.21.15.28:0/4027111170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.285+0000 7f6febfff640 1 --2- 172.21.15.28:0/4027111170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f026ea0 0x56282f030a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.285+0000 7f6ff11fa640 1 -- 172.21.15.28:0/4027111170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f026ea0 msgr2=0x56282f030a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.285+0000 7f6ff11fa640 1 --2- 172.21.15.28:0/4027111170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f026ea0 0x56282f030a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.285+0000 7f6ff11fa640 1 -- 172.21.15.28:0/4027111170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f026580 msgr2=0x56282f03d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.285+0000 7f6ff11fa640 1 --2- 172.21.15.28:0/4027111170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f026580 0x56282f03d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.285+0000 7f6ff11fa640 1 -- 172.21.15.28:0/4027111170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56282f02eb10 con 0x56282f031570 2024-04-25T01:42:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.285+0000 7f6ff09f9640 1 --2- 172.21.15.28:0/4027111170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f026580 0x56282f03d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:42:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.285+0000 7f6ff11fa640 1 --2- 172.21.15.28:0/4027111170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f031570 0x56282f049270 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40122f0 tx=0x7f6fe401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f1c4aef58889612 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.285+0000 7f6fd27fc640 1 -- 172.21.15.28:0/4027111170 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4004230 con 0x56282f031570 2024-04-25T01:42:39.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.285+0000 7f6fd27fc640 1 -- 172.21.15.28:0/4027111170 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fe4020430 con 0x56282f031570 2024-04-25T01:42:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.285+0000 7f6fd27fc640 1 -- 172.21.15.28:0/4027111170 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4010670 con 0x56282f031570 2024-04-25T01:42:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.286+0000 7f6ff5969740 1 -- 172.21.15.28:0/4027111170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f031570 msgr2=0x56282f049270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.286+0000 7f6ff5969740 1 --2- 172.21.15.28:0/4027111170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f031570 0x56282f049270 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f6fe40122f0 tx=0x7f6fe401f000 comp rx=0 tx=0).stop 2024-04-25T01:42:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.286+0000 7f6ff5969740 1 -- 172.21.15.28:0/4027111170 shutdown_connections 2024-04-25T01:42:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.286+0000 7f6ff5969740 1 --2- 172.21.15.28:0/4027111170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f031570 0x56282f049270 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.286+0000 7f6ff5969740 1 --2- 172.21.15.28:0/4027111170 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f026ea0 0x56282f030a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.286+0000 7f6ff5969740 1 --2- 172.21.15.28:0/4027111170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f026580 0x56282f03d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.286+0000 7f6ff5969740 1 -- 172.21.15.28:0/4027111170 >> 172.21.15.28:0/4027111170 conn(0x56282f0228a0 msgr2=0x56282f02caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.286+0000 7f6ff5969740 1 -- 172.21.15.28:0/4027111170 shutdown_connections 2024-04-25T01:42:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.286+0000 7f6ff5969740 1 -- 172.21.15.28:0/4027111170 wait complete. 2024-04-25T01:42:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.286+0000 7f6ff5969740 1 -- start start 2024-04-25T01:42:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.286+0000 7f6ff5969740 1 Processor -- start 2024-04-25T01:42:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6ff5969740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f026580 0x56282f09fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6ff5969740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f026ea0 0x56282f0a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6ff5969740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f031570 0x56282f0a3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6ff5969740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec1b0000 con 0x56282f026580 2024-04-25T01:42:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6ff5969740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec1af930 con 0x56282f026ea0 2024-04-25T01:42:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6ff5969740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec1acb90 con 0x56282f031570 2024-04-25T01:42:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6ff5969740 10 client.0 _ll_register_callbacks cb 0x7f6fe0009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6ff09f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f026580 0x56282f09fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6ff09f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f026580 0x56282f09fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48044/0 (socket says 172.21.15.28:48044) 2024-04-25T01:42:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6ff09f9640 1 -- 172.21.15.28:0/3889970529 learned_addr learned my addr 172.21.15.28:0/3889970529 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6febfff640 1 --2- 172.21.15.28:0/3889970529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f026ea0 0x56282f0a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6ff09f9640 1 -- 172.21.15.28:0/3889970529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f031570 msgr2=0x56282f0a3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6ff09f9640 1 --2- 172.21.15.28:0/3889970529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f031570 0x56282f0a3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6ff09f9640 1 -- 172.21.15.28:0/3889970529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f026ea0 msgr2=0x56282f0a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6ff09f9640 1 --2- 172.21.15.28:0/3889970529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f026ea0 0x56282f0a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6ff09f9640 1 -- 172.21.15.28:0/3889970529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56282ed697d0 con 0x56282f026580 2024-04-25T01:42:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6ff09f9640 1 --2- 172.21.15.28:0/3889970529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f026580 0x56282f09fc90 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc012040 tx=0x7f6fdc09b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da2be2e005cba507 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6fcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6fcbfff640 1 -- 172.21.15.28:0/3889970529 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc091070 con 0x56282f026580 2024-04-25T01:42:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6fcbfff640 1 -- 172.21.15.28:0/3889970529 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f6fdc0cc6e0 con 0x56282f026580 2024-04-25T01:42:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.287+0000 7f6ff5969740 1 -- 172.21.15.28:0/3889970529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fe4020430 con 0x56282f026580 2024-04-25T01:42:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.288+0000 7f6fcbfff640 1 -- 172.21.15.28:0/3889970529 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6fdc0b9050 con 0x56282f026580 2024-04-25T01:42:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.288+0000 7f6ff5969740 10 client.19233 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.288+0000 7f6ff5969740 1 -- 172.21.15.28:0/3889970529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56282f0ac630 con 0x56282f026580 2024-04-25T01:42:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.288+0000 7f6fcbfff640 1 -- 172.21.15.28:0/3889970529 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fe4020430 con 0x56282f026580 2024-04-25T01:42:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.288+0000 7f6fcbfff640 1 -- 172.21.15.28:0/3889970529 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6fdc0ccbb0 con 0x56282f026580 2024-04-25T01:42:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.288+0000 7f6ff5969740 10 client.19233 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.288+0000 7f6ff5969740 10 client.19233 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.288+0000 7f6ff5969740 20 client.19233 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.288+0000 7f6ff5969740 1 --2- 172.21.15.28:0/3889970529 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56282f0ac9c0 0x56282f0aedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:39.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.288+0000 7f6ff5969740 4 client.19233 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:42:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.288+0000 7f6ff5969740 1 -- 172.21.15.28:0/3889970529 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56282ef92f00 con 0x56282f0ac9c0 2024-04-25T01:42:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.288+0000 7f6febfff640 1 --2- 172.21.15.28:0/3889970529 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56282f0ac9c0 0x56282f0aedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.289+0000 7f6febfff640 1 --2- 172.21.15.28:0/3889970529 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56282f0ac9c0 0x56282f0aedb0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6b78b6edd6b35c3 server_cookie=c881565424a750 in_seq=0 out_seq=0 2024-04-25T01:42:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.289+0000 7f6fcbfff640 10 client.19233 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.289+0000 7f6fcbfff640 1 -- 172.21.15.28:0/3889970529 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56282ef92f00 con 0x56282f0ac9c0 2024-04-25T01:42:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.289+0000 7f6fcbfff640 10 client.19233 handle_command_reply: tid=0 2024-04-25T01:42:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.289+0000 7f6ff5969740 1 client.19233 shutdown 2024-04-25T01:42:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.289+0000 7f6ff5969740 1 -- 172.21.15.28:0/3889970529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f026580 msgr2=0x56282f09fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.289+0000 7f6ff5969740 1 --2- 172.21.15.28:0/3889970529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f026580 0x56282f09fc90 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc012040 tx=0x7f6fdc09b000 comp rx=0 tx=0).stop 2024-04-25T01:42:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.289+0000 7f6ff5969740 1 -- 172.21.15.28:0/3889970529 shutdown_connections 2024-04-25T01:42:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.289+0000 7f6ff5969740 1 --2- 172.21.15.28:0/3889970529 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x56282f0ac9c0 0x56282f0aedb0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.289+0000 7f6ff5969740 1 --2- 172.21.15.28:0/3889970529 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56282f031570 0x56282f0a3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.289+0000 7f6ff5969740 1 --2- 172.21.15.28:0/3889970529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56282f026ea0 0x56282f0a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.289+0000 7f6fcbfff640 0 client.19233 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:39.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.289+0000 7f6ff5969740 1 --2- 172.21.15.28:0/3889970529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56282f026580 0x56282f09fc90 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.289+0000 7f6ff5969740 1 -- 172.21.15.28:0/3889970529 >> 172.21.15.28:0/3889970529 conn(0x56282f0228a0 msgr2=0x56282f03c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.289+0000 7f6ff5969740 1 -- 172.21.15.28:0/3889970529 shutdown_connections 2024-04-25T01:42:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.289+0000 7f6ff5969740 1 -- 172.21.15.28:0/3889970529 wait complete. 2024-04-25T01:42:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.289+0000 7f6ff5969740 20 client.19233 trim_cache size 0 max 16384 2024-04-25T01:42:39.284 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:42:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.290+0000 7f6ff2c5d640 1 -- 172.21.15.28:0/803346381 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6fd40a2d80 msgr2=0x7f6fd40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.290+0000 7f6ff2c5d640 1 --2- 172.21.15.28:0/803346381 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6fd40a2d80 0x7f6fd40a5240 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4005fd0 tx=0x7f6fe4013b80 comp rx=0 tx=0).stop 2024-04-25T01:42:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.290+0000 7f6ff2c5d640 1 -- 172.21.15.28:0/803346381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fec101710 msgr2=0x7f6fec1980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.290+0000 7f6ff2c5d640 1 --2- 172.21.15.28:0/803346381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fec101710 0x7f6fec1980b0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc00d770 tx=0x7f6fdc00db70 comp rx=0 tx=0).stop 2024-04-25T01:42:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.290+0000 7f6ff2c5d640 1 -- 172.21.15.28:0/803346381 shutdown_connections 2024-04-25T01:42:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.290+0000 7f6ff2c5d640 1 --2- 172.21.15.28:0/803346381 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6fd40a2d80 0x7f6fd40a5240 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.290+0000 7f6ff2c5d640 1 --2- 172.21.15.28:0/803346381 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fec107720 0x7f6fec19c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.290+0000 7f6ff2c5d640 1 --2- 172.21.15.28:0/803346381 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fec1020d0 0x7f6fec1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.290+0000 7f6ff2c5d640 1 --2- 172.21.15.28:0/803346381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fec101710 0x7f6fec1980b0 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:39.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.290+0000 7f6ff2c5d640 1 -- 172.21.15.28:0/803346381 >> 172.21.15.28:0/803346381 conn(0x7f6fec078b90 msgr2=0x7f6fec0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:39.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.290+0000 7f6ff2c5d640 1 -- 172.21.15.28:0/803346381 shutdown_connections 2024-04-25T01:42:39.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:39.290+0000 7f6ff2c5d640 1 -- 172.21.15.28:0/803346381 wait complete. 2024-04-25T01:42:39.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:39 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:42:39.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:39 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:42:39.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:39 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:42:39.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:39 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:42:39.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:39 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:42:39.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:39 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2937879858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:42:40.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:40 smithi032 ceph-mon[32461]: pgmap v298: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:42:40.290 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:42:40.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:42:40.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:42:40.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:40 smithi028 ceph-mon[29390]: pgmap v298: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:42:40.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.421+0000 7f8da2e64640 1 -- 172.21.15.28:0/466182854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9c1032e0 msgr2=0x7f8d9c10b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.421+0000 7f8da2e64640 1 --2- 172.21.15.28:0/466182854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9c1032e0 0x7f8d9c10b7c0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f8d84009900 tx=0x7f8d8402a510 comp rx=0 tx=0).stop 2024-04-25T01:42:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.421+0000 7f8da2e64640 1 -- 172.21.15.28:0/466182854 shutdown_connections 2024-04-25T01:42:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.421+0000 7f8da2e64640 1 --2- 172.21.15.28:0/466182854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d9c10bd00 0x7f8d9c10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.421+0000 7f8da2e64640 1 --2- 172.21.15.28:0/466182854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9c1032e0 0x7f8d9c10b7c0 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.421+0000 7f8da2e64640 1 --2- 172.21.15.28:0/466182854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d9c1029c0 0x7f8d9c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.421+0000 7f8da2e64640 1 -- 172.21.15.28:0/466182854 >> 172.21.15.28:0/466182854 conn(0x7f8d9c077740 msgr2=0x7f8d9c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.421+0000 7f8da2e64640 1 -- 172.21.15.28:0/466182854 shutdown_connections 2024-04-25T01:42:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.421+0000 7f8da2e64640 1 -- 172.21.15.28:0/466182854 wait complete. 2024-04-25T01:42:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.422+0000 7f8da2e64640 1 Processor -- start 2024-04-25T01:42:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.422+0000 7f8da2e64640 1 -- start start 2024-04-25T01:42:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.422+0000 7f8da2e64640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d9c1029c0 0x7f8d9c100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.422+0000 7f8da2e64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9c1032e0 0x7f8d9c100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.422+0000 7f8da2e64640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d9c10bd00 0x7f8d9c101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.422+0000 7f8da2e64640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9c111b00 con 0x7f8d9c1032e0 2024-04-25T01:42:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.422+0000 7f8da2e64640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9c111980 con 0x7f8d9c10bd00 2024-04-25T01:42:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.422+0000 7f8da2e64640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9c111c80 con 0x7f8d9c1029c0 2024-04-25T01:42:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.422+0000 7f8d9bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9c1032e0 0x7f8d9c100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.422+0000 7f8d9bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9c1032e0 0x7f8d9c100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48072/0 (socket says 172.21.15.28:48072) 2024-04-25T01:42:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.422+0000 7f8d9bfff640 1 -- 172.21.15.28:0/139238982 learned_addr learned my addr 172.21.15.28:0/139238982 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.422+0000 7f8da1401640 1 --2- 172.21.15.28:0/139238982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d9c10bd00 0x7f8d9c101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.422+0000 7f8da0c00640 1 --2- 172.21.15.28:0/139238982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d9c1029c0 0x7f8d9c100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.422+0000 7f8d9bfff640 1 -- 172.21.15.28:0/139238982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d9c1029c0 msgr2=0x7f8d9c100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.422+0000 7f8d9bfff640 1 --2- 172.21.15.28:0/139238982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d9c1029c0 0x7f8d9c100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.422+0000 7f8d9bfff640 1 -- 172.21.15.28:0/139238982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d9c10bd00 msgr2=0x7f8d9c101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.423+0000 7f8d9bfff640 1 --2- 172.21.15.28:0/139238982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d9c10bd00 0x7f8d9c101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.423+0000 7f8d9bfff640 1 -- 172.21.15.28:0/139238982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9c1a6500 con 0x7f8d9c1032e0 2024-04-25T01:42:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.423+0000 7f8da1401640 1 --2- 172.21.15.28:0/139238982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d9c10bd00 0x7f8d9c101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.423+0000 7f8da0c00640 1 --2- 172.21.15.28:0/139238982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d9c1029c0 0x7f8d9c100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.423+0000 7f8d9bfff640 1 --2- 172.21.15.28:0/139238982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9c1032e0 0x7f8d9c100b50 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f8d84006010 tx=0x7f8d840042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.423+0000 7f8d99ffb640 1 -- 172.21.15.28:0/139238982 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8403f030 con 0x7f8d9c1032e0 2024-04-25T01:42:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.423+0000 7f8d99ffb640 1 -- 172.21.15.28:0/139238982 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d84003ce0 con 0x7f8d9c1032e0 2024-04-25T01:42:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.423+0000 7f8da2e64640 1 -- 172.21.15.28:0/139238982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d9c1a6790 con 0x7f8d9c1032e0 2024-04-25T01:42:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.423+0000 7f8d99ffb640 1 -- 172.21.15.28:0/139238982 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84035070 con 0x7f8d9c1032e0 2024-04-25T01:42:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.423+0000 7f8da2e64640 1 -- 172.21.15.28:0/139238982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d9c1a6c80 con 0x7f8d9c1032e0 2024-04-25T01:42:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.426+0000 7f8d99ffb640 1 -- 172.21.15.28:0/139238982 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f8d84042030 con 0x7f8d9c1032e0 2024-04-25T01:42:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.426+0000 7f8da5b74740 1 Processor -- start 2024-04-25T01:42:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.426+0000 7f8da5b74740 1 -- start start 2024-04-25T01:42:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.426+0000 7f8da5b74740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc68980 0x55ff0fc66cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.426+0000 7f8da5b74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 0x55ff0fc67710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8da5b74740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc72010 0x55ff0fc74400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8da5b74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ff0fbcb2c0 con 0x55ff0fc67290 2024-04-25T01:42:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8da5b74740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ff0fb73910 con 0x55ff0fc72010 2024-04-25T01:42:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8da5b74740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ff0fbc7280 con 0x55ff0fc68980 2024-04-25T01:42:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8da0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 0x55ff0fc67710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8da0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 0x55ff0fc67710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48084/0 (socket says 172.21.15.28:48084) 2024-04-25T01:42:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8da0c00640 1 -- 172.21.15.28:0/684734055 learned_addr learned my addr 172.21.15.28:0/684734055 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8da1401640 1 --2- 172.21.15.28:0/684734055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc68980 0x55ff0fc66cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8d9bfff640 1 --2- 172.21.15.28:0/684734055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc72010 0x55ff0fc74400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8d99ffb640 1 --2- 172.21.15.28:0/139238982 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8d880a2d00 0x7f8d880a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8da0c00640 1 -- 172.21.15.28:0/684734055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc68980 msgr2=0x55ff0fc66cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8da0c00640 1 --2- 172.21.15.28:0/684734055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc68980 0x55ff0fc66cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8da0c00640 1 -- 172.21.15.28:0/684734055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc72010 msgr2=0x55ff0fc74400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8da0c00640 1 --2- 172.21.15.28:0/684734055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc72010 0x55ff0fc74400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8da0c00640 1 -- 172.21.15.28:0/684734055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff0fb84fb0 con 0x55ff0fc67290 2024-04-25T01:42:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8da1401640 1 --2- 172.21.15.28:0/139238982 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8d880a2d00 0x7f8d880a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8d99ffb640 1 -- 172.21.15.28:0/139238982 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f8d840effb0 con 0x7f8d9c1032e0 2024-04-25T01:42:40.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8da0c00640 1 --2- 172.21.15.28:0/684734055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 0x55ff0fc67710 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c009c50 tx=0x7f8d8c00e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=569cc013bf9b2139 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8d92ffd640 1 -- 172.21.15.28:0/684734055 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c016070 con 0x55ff0fc67290 2024-04-25T01:42:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8d92ffd640 1 -- 172.21.15.28:0/684734055 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d8c005500 con 0x55ff0fc67290 2024-04-25T01:42:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.427+0000 7f8d92ffd640 1 -- 172.21.15.28:0/684734055 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c0116d0 con 0x55ff0fc67290 2024-04-25T01:42:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.428+0000 7f8da5b74740 1 -- 172.21.15.28:0/684734055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 msgr2=0x55ff0fc67710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.428+0000 7f8da1401640 1 --2- 172.21.15.28:0/139238982 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8d880a2d00 0x7f8d880a51c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8d94005fd0 tx=0x7f8d94004a40 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.428+0000 7f8da5b74740 1 --2- 172.21.15.28:0/684734055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 0x55ff0fc67710 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c009c50 tx=0x7f8d8c00e6c0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.428+0000 7f8da5b74740 1 -- 172.21.15.28:0/684734055 shutdown_connections 2024-04-25T01:42:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.428+0000 7f8da5b74740 1 --2- 172.21.15.28:0/684734055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc72010 0x55ff0fc74400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.428+0000 7f8da5b74740 1 --2- 172.21.15.28:0/684734055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 0x55ff0fc67710 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.428+0000 7f8da5b74740 1 --2- 172.21.15.28:0/684734055 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc68980 0x55ff0fc66cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.428+0000 7f8da5b74740 1 -- 172.21.15.28:0/684734055 >> 172.21.15.28:0/684734055 conn(0x55ff0fc2f950 msgr2=0x55ff0fc637f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.428+0000 7f8da5b74740 1 -- 172.21.15.28:0/684734055 shutdown_connections 2024-04-25T01:42:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.428+0000 7f8da5b74740 1 -- 172.21.15.28:0/684734055 wait complete. 2024-04-25T01:42:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.429+0000 7f8da5b74740 1 -- start start 2024-04-25T01:42:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.429+0000 7f8da5b74740 1 Processor -- start 2024-04-25T01:42:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.429+0000 7f8da5b74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 0x55ff0fc6ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.429+0000 7f8da5b74740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc68980 0x55ff0fc6f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.429+0000 7f8da5b74740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc72010 0x55ff0fc6f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.429+0000 7f8da5b74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ff0fc74ea0 con 0x55ff0fc67290 2024-04-25T01:42:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.429+0000 7f8da5b74740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ff0fb86e90 con 0x55ff0fc68980 2024-04-25T01:42:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.429+0000 7f8da5b74740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55ff0fc75020 con 0x55ff0fc72010 2024-04-25T01:42:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.429+0000 7f8da5b74740 10 client.0 _ll_register_callbacks cb 0x55ff0fbc2970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.429+0000 7f8da0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 0x55ff0fc6ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.429+0000 7f8da0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 0x55ff0fc6ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48090/0 (socket says 172.21.15.28:48090) 2024-04-25T01:42:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.429+0000 7f8da0c00640 1 -- 172.21.15.28:0/2816671992 learned_addr learned my addr 172.21.15.28:0/2816671992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.429+0000 7f8d9bfff640 1 --2- 172.21.15.28:0/2816671992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc68980 0x55ff0fc6f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.429+0000 7f8da1401640 1 --2- 172.21.15.28:0/2816671992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc72010 0x55ff0fc6f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.429+0000 7f8da0c00640 1 -- 172.21.15.28:0/2816671992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc72010 msgr2=0x55ff0fc6f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.429+0000 7f8da0c00640 1 --2- 172.21.15.28:0/2816671992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc72010 0x55ff0fc6f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.429+0000 7f8da0c00640 1 -- 172.21.15.28:0/2816671992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc68980 msgr2=0x55ff0fc6f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.429+0000 7f8da0c00640 1 --2- 172.21.15.28:0/2816671992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc68980 0x55ff0fc6f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.429+0000 7f8da0c00640 1 -- 172.21.15.28:0/2816671992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff0f9a77e0 con 0x55ff0fc67290 2024-04-25T01:42:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.430+0000 7f8da0c00640 1 --2- 172.21.15.28:0/2816671992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 0x55ff0fc6ef10 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c009990 tx=0x7f8d8c00efc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6183ba95188fda57 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.430+0000 7f8d7bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.430+0000 7f8d7bfff640 1 -- 172.21.15.28:0/2816671992 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c016040 con 0x55ff0fc67290 2024-04-25T01:42:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.430+0000 7f8d7bfff640 1 -- 172.21.15.28:0/2816671992 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f8d8c004020 con 0x55ff0fc67290 2024-04-25T01:42:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.430+0000 7f8da5b74740 1 -- 172.21.15.28:0/2816671992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d8c005500 con 0x55ff0fc67290 2024-04-25T01:42:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.430+0000 7f8d7bfff640 1 -- 172.21.15.28:0/2816671992 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8d8c004220 con 0x55ff0fc67290 2024-04-25T01:42:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.430+0000 7f8da5b74740 10 client.19245 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.430+0000 7f8da5b74740 1 -- 172.21.15.28:0/2816671992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ff0fa07f70 con 0x55ff0fc67290 2024-04-25T01:42:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.430+0000 7f8d7bfff640 1 -- 172.21.15.28:0/2816671992 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d8c005500 con 0x55ff0fc67290 2024-04-25T01:42:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.430+0000 7f8d7bfff640 1 -- 172.21.15.28:0/2816671992 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8d8c00f040 con 0x55ff0fc67290 2024-04-25T01:42:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.431+0000 7f8da5b74740 10 client.19245 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.431+0000 7f8da5b74740 10 client.19245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.431+0000 7f8da5b74740 20 client.19245 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.431+0000 7f8da5b74740 1 --2- 172.21.15.28:0/2816671992 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ff0fc81fd0 0x55ff0fc843c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.431+0000 7f8da5b74740 4 client.19245 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.431+0000 7f8da5b74740 1 -- 172.21.15.28:0/2816671992 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff0fb9f5a0 con 0x55ff0fc81fd0 2024-04-25T01:42:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.431+0000 7f8d9bfff640 1 --2- 172.21.15.28:0/2816671992 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ff0fc81fd0 0x55ff0fc843c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.431+0000 7f8d9bfff640 1 --2- 172.21.15.28:0/2816671992 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ff0fc81fd0 0x55ff0fc843c0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b934b5ae82379b1b server_cookie=c648c1048bd074ae in_seq=0 out_seq=0 2024-04-25T01:42:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.431+0000 7f8d7bfff640 10 client.19245 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.432+0000 7f8d7bfff640 1 -- 172.21.15.28:0/2816671992 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ff0fb9f5a0 con 0x55ff0fc81fd0 2024-04-25T01:42:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.432+0000 7f8d7bfff640 10 client.19245 handle_command_reply: tid=0 2024-04-25T01:42:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.432+0000 7f8da5b74740 1 client.19245 shutdown 2024-04-25T01:42:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.432+0000 7f8da5b74740 1 -- 172.21.15.28:0/2816671992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 msgr2=0x55ff0fc6ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.432+0000 7f8da5b74740 1 --2- 172.21.15.28:0/2816671992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 0x55ff0fc6ef10 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c009990 tx=0x7f8d8c00efc0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.432+0000 7f8da5b74740 1 -- 172.21.15.28:0/2816671992 shutdown_connections 2024-04-25T01:42:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.432+0000 7f8da5b74740 1 --2- 172.21.15.28:0/2816671992 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ff0fc81fd0 0x55ff0fc843c0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.432+0000 7f8da5b74740 1 --2- 172.21.15.28:0/2816671992 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc72010 0x55ff0fc6f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.432+0000 7f8da5b74740 1 --2- 172.21.15.28:0/2816671992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc68980 0x55ff0fc6f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.432+0000 7f8da5b74740 1 --2- 172.21.15.28:0/2816671992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 0x55ff0fc6ef10 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.432+0000 7f8d7bfff640 0 client.19245 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.432+0000 7f8da5b74740 1 -- 172.21.15.28:0/2816671992 >> 172.21.15.28:0/2816671992 conn(0x55ff0fc2f950 msgr2=0x55ff0fc695f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.432+0000 7f8da5b74740 1 -- 172.21.15.28:0/2816671992 shutdown_connections 2024-04-25T01:42:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.433+0000 7f8da5b74740 1 -- 172.21.15.28:0/2816671992 wait complete. 2024-04-25T01:42:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.433+0000 7f8da5b74740 20 client.19245 trim_cache size 0 max 16384 2024-04-25T01:42:40.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.450+0000 7f8da5b74740 1 Processor -- start 2024-04-25T01:42:40.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.450+0000 7f8da5b74740 1 -- start start 2024-04-25T01:42:40.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8da5b74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 0x55ff0fc7dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8da5b74740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc68980 0x55ff0fc899f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8da5b74740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc72010 0x55ff0fc89f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8da5b74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9c1b0df0 con 0x55ff0fc67290 2024-04-25T01:42:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8da5b74740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9c1b2930 con 0x55ff0fc68980 2024-04-25T01:42:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8da5b74740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9c1b14c0 con 0x55ff0fc72010 2024-04-25T01:42:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8da0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 0x55ff0fc7dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8da0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 0x55ff0fc7dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48104/0 (socket says 172.21.15.28:48104) 2024-04-25T01:42:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8da0c00640 1 -- 172.21.15.28:0/1500674930 learned_addr learned my addr 172.21.15.28:0/1500674930 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8d9bfff640 1 --2- 172.21.15.28:0/1500674930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc68980 0x55ff0fc899f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8da1401640 1 --2- 172.21.15.28:0/1500674930 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc72010 0x55ff0fc89f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8da0c00640 1 -- 172.21.15.28:0/1500674930 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc72010 msgr2=0x55ff0fc89f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8da0c00640 1 --2- 172.21.15.28:0/1500674930 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc72010 0x55ff0fc89f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8da0c00640 1 -- 172.21.15.28:0/1500674930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc68980 msgr2=0x55ff0fc899f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8da0c00640 1 --2- 172.21.15.28:0/1500674930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc68980 0x55ff0fc899f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8da0c00640 1 -- 172.21.15.28:0/1500674930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff0f89f170 con 0x55ff0fc67290 2024-04-25T01:42:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8d9bfff640 1 --2- 172.21.15.28:0/1500674930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc68980 0x55ff0fc899f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8da1401640 1 --2- 172.21.15.28:0/1500674930 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc72010 0x55ff0fc89f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8da0c00640 1 --2- 172.21.15.28:0/1500674930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 0x55ff0fc7dda0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c0099e0 tx=0x7f8d8c003c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b5d303938b7702 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8d927fc640 1 -- 172.21.15.28:0/1500674930 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c0043c0 con 0x55ff0fc67290 2024-04-25T01:42:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8d927fc640 1 -- 172.21.15.28:0/1500674930 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d8c003e80 con 0x55ff0fc67290 2024-04-25T01:42:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.451+0000 7f8d927fc640 1 -- 172.21.15.28:0/1500674930 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c023410 con 0x55ff0fc67290 2024-04-25T01:42:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.452+0000 7f8da5b74740 1 -- 172.21.15.28:0/1500674930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 msgr2=0x55ff0fc7dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.452+0000 7f8da5b74740 1 --2- 172.21.15.28:0/1500674930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 0x55ff0fc7dda0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c0099e0 tx=0x7f8d8c003c00 comp rx=0 tx=0).stop 2024-04-25T01:42:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.452+0000 7f8da5b74740 1 -- 172.21.15.28:0/1500674930 shutdown_connections 2024-04-25T01:42:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.452+0000 7f8da5b74740 1 --2- 172.21.15.28:0/1500674930 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc72010 0x55ff0fc89f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.452+0000 7f8da5b74740 1 --2- 172.21.15.28:0/1500674930 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc68980 0x55ff0fc899f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.452+0000 7f8da5b74740 1 --2- 172.21.15.28:0/1500674930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc67290 0x55ff0fc7dda0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.452+0000 7f8da5b74740 1 -- 172.21.15.28:0/1500674930 >> 172.21.15.28:0/1500674930 conn(0x55ff0fc2f950 msgr2=0x55ff0fc6d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.452+0000 7f8da5b74740 1 -- 172.21.15.28:0/1500674930 shutdown_connections 2024-04-25T01:42:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.452+0000 7f8da5b74740 1 -- 172.21.15.28:0/1500674930 wait complete. 2024-04-25T01:42:40.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.452+0000 7f8da5b74740 1 -- start start 2024-04-25T01:42:40.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.452+0000 7f8da5b74740 1 Processor -- start 2024-04-25T01:42:40.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8da5b74740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc67290 0x55ff0fc6bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8da5b74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc68980 0x55ff0fc6c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8da5b74740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc72010 0x55ff0fc6c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8da5b74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9c1b2260 con 0x55ff0fc68980 2024-04-25T01:42:40.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8da5b74740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9c1b1b90 con 0x55ff0fc67290 2024-04-25T01:42:40.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8da5b74740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9c1aedf0 con 0x55ff0fc72010 2024-04-25T01:42:40.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8da5b74740 10 client.0 _ll_register_callbacks cb 0x7f8d8c009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:40.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8d9bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc68980 0x55ff0fc6c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8d9bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc68980 0x55ff0fc6c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48114/0 (socket says 172.21.15.28:48114) 2024-04-25T01:42:40.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8d9bfff640 1 -- 172.21.15.28:0/509068765 learned_addr learned my addr 172.21.15.28:0/509068765 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:40.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8da0c00640 1 --2- 172.21.15.28:0/509068765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc67290 0x55ff0fc6bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8da1401640 1 --2- 172.21.15.28:0/509068765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc72010 0x55ff0fc6c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8d9bfff640 1 -- 172.21.15.28:0/509068765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc72010 msgr2=0x55ff0fc6c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:40.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8d9bfff640 1 --2- 172.21.15.28:0/509068765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc72010 0x55ff0fc6c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8d9bfff640 1 -- 172.21.15.28:0/509068765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc67290 msgr2=0x55ff0fc6bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:40.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8d9bfff640 1 --2- 172.21.15.28:0/509068765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc67290 0x55ff0fc6bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8d9bfff640 1 -- 172.21.15.28:0/509068765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff0fce7310 con 0x55ff0fc68980 2024-04-25T01:42:40.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8da1401640 1 --2- 172.21.15.28:0/509068765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc72010 0x55ff0fc6c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8da0c00640 1 --2- 172.21.15.28:0/509068765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc67290 0x55ff0fc6bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8d9bfff640 1 --2- 172.21.15.28:0/509068765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc68980 0x55ff0fc6c1e0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f8d840b8600 tx=0x7f8d840bbb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a4673878590cd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.453+0000 7f8d7bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.454+0000 7f8d7bfff640 1 -- 172.21.15.28:0/509068765 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84049040 con 0x55ff0fc68980 2024-04-25T01:42:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.454+0000 7f8d7bfff640 1 -- 172.21.15.28:0/509068765 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f8d840b8810 con 0x55ff0fc68980 2024-04-25T01:42:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.454+0000 7f8da5b74740 1 -- 172.21.15.28:0/509068765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d8c003e80 con 0x55ff0fc68980 2024-04-25T01:42:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.454+0000 7f8d7bfff640 1 -- 172.21.15.28:0/509068765 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8d8403b580 con 0x55ff0fc68980 2024-04-25T01:42:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.454+0000 7f8da5b74740 10 client.19251 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.454+0000 7f8da5b74740 1 -- 172.21.15.28:0/509068765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ff0fce8600 con 0x55ff0fc68980 2024-04-25T01:42:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.454+0000 7f8d7bfff640 1 -- 172.21.15.28:0/509068765 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d8c003e80 con 0x55ff0fc68980 2024-04-25T01:42:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.454+0000 7f8d7bfff640 1 -- 172.21.15.28:0/509068765 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8d84087180 con 0x55ff0fc68980 2024-04-25T01:42:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.454+0000 7f8da5b74740 10 client.19251 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.454+0000 7f8da5b74740 10 client.19251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.454+0000 7f8da5b74740 20 client.19251 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.454+0000 7f8da5b74740 1 --2- 172.21.15.28:0/509068765 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ff0fce8d10 0x55ff0fceb130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.454+0000 7f8da5b74740 4 client.19251 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:42:40.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.454+0000 7f8da5b74740 1 -- 172.21.15.28:0/509068765 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ff0fbd8800 con 0x55ff0fce8d10 2024-04-25T01:42:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.454+0000 7f8da0c00640 1 --2- 172.21.15.28:0/509068765 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ff0fce8d10 0x55ff0fceb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.455+0000 7f8da0c00640 1 --2- 172.21.15.28:0/509068765 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ff0fce8d10 0x55ff0fceb130 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9c7eb93015a23d7 server_cookie=d564fbf95c24ee00 in_seq=0 out_seq=0 2024-04-25T01:42:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.455+0000 7f8d7bfff640 10 client.19251 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.456+0000 7f8d7bfff640 1 -- 172.21.15.28:0/509068765 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ff0fbd8800 con 0x55ff0fce8d10 2024-04-25T01:42:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.456+0000 7f8d7bfff640 10 client.19251 handle_command_reply: tid=0 2024-04-25T01:42:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.456+0000 7f8da5b74740 1 client.19251 shutdown 2024-04-25T01:42:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.456+0000 7f8da5b74740 1 -- 172.21.15.28:0/509068765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc68980 msgr2=0x55ff0fc6c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.456+0000 7f8da5b74740 1 --2- 172.21.15.28:0/509068765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc68980 0x55ff0fc6c1e0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f8d840b8600 tx=0x7f8d840bbb00 comp rx=0 tx=0).stop 2024-04-25T01:42:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.456+0000 7f8da5b74740 1 -- 172.21.15.28:0/509068765 shutdown_connections 2024-04-25T01:42:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.456+0000 7f8da5b74740 1 --2- 172.21.15.28:0/509068765 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x55ff0fce8d10 0x55ff0fceb130 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.456+0000 7f8da5b74740 1 --2- 172.21.15.28:0/509068765 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55ff0fc72010 0x55ff0fc6c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.456+0000 7f8da5b74740 1 --2- 172.21.15.28:0/509068765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ff0fc68980 0x55ff0fc6c1e0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.456+0000 7f8da5b74740 1 --2- 172.21.15.28:0/509068765 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ff0fc67290 0x55ff0fc6bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.457+0000 7f8da5b74740 1 -- 172.21.15.28:0/509068765 >> 172.21.15.28:0/509068765 conn(0x55ff0fc2f950 msgr2=0x55ff0fc620c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.457+0000 7f8d7bfff640 0 client.19251 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.457+0000 7f8da5b74740 1 -- 172.21.15.28:0/509068765 shutdown_connections 2024-04-25T01:42:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.457+0000 7f8da5b74740 1 -- 172.21.15.28:0/509068765 wait complete. 2024-04-25T01:42:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.457+0000 7f8da5b74740 20 client.19251 trim_cache size 0 max 16384 2024-04-25T01:42:40.450 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:42:40.450 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T01:42:40.450 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "6a1c001b-ee70-46fc-bd43-b286af1f4837", 2024-04-25T01:42:40.450 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T01:42:40.450 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:42:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.457+0000 7f8da2e64640 1 -- 172.21.15.28:0/139238982 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8d880a2d00 msgr2=0x7f8d880a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.457+0000 7f8da2e64640 1 --2- 172.21.15.28:0/139238982 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8d880a2d00 0x7f8d880a51c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8d94005fd0 tx=0x7f8d94004a40 comp rx=0 tx=0).stop 2024-04-25T01:42:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.457+0000 7f8da2e64640 1 -- 172.21.15.28:0/139238982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9c1032e0 msgr2=0x7f8d9c100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.457+0000 7f8da2e64640 1 --2- 172.21.15.28:0/139238982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9c1032e0 0x7f8d9c100b50 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f8d84006010 tx=0x7f8d840042f0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.457+0000 7f8da2e64640 1 -- 172.21.15.28:0/139238982 shutdown_connections 2024-04-25T01:42:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.457+0000 7f8da2e64640 1 --2- 172.21.15.28:0/139238982 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f8d880a2d00 0x7f8d880a51c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.457+0000 7f8da2e64640 1 --2- 172.21.15.28:0/139238982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d9c10bd00 0x7f8d9c101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.457+0000 7f8da2e64640 1 --2- 172.21.15.28:0/139238982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9c1032e0 0x7f8d9c100b50 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.457+0000 7f8da2e64640 1 --2- 172.21.15.28:0/139238982 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8d9c1029c0 0x7f8d9c100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.457+0000 7f8da2e64640 1 -- 172.21.15.28:0/139238982 >> 172.21.15.28:0/139238982 conn(0x7f8d9c077740 msgr2=0x7f8d9c0fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.458+0000 7f8da2e64640 1 -- 172.21.15.28:0/139238982 shutdown_connections 2024-04-25T01:42:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.458+0000 7f8da2e64640 1 -- 172.21.15.28:0/139238982 wait complete. 2024-04-25T01:42:40.457 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:42:40.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:42:40.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:40 smithi130 ceph-mon[32702]: pgmap v298: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:42:40.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.613+0000 7f5a033ab640 1 -- 172.21.15.28:0/667198092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59fc069dc0 msgr2=0x7f59fc06a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:40.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.613+0000 7f5a033ab640 1 --2- 172.21.15.28:0/667198092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59fc069dc0 0x7f59fc06a240 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009930 tx=0x7f59ec02a420 comp rx=0 tx=0).stop 2024-04-25T01:42:40.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.613+0000 7f5a033ab640 1 -- 172.21.15.28:0/667198092 shutdown_connections 2024-04-25T01:42:40.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.613+0000 7f5a033ab640 1 --2- 172.21.15.28:0/667198092 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f59fc06a780 0x7f59fc10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.613+0000 7f5a033ab640 1 --2- 172.21.15.28:0/667198092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59fc069dc0 0x7f59fc06a240 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.613+0000 7f5a033ab640 1 --2- 172.21.15.28:0/667198092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59fc107730 0x7f59fc107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.613+0000 7f5a033ab640 1 -- 172.21.15.28:0/667198092 >> 172.21.15.28:0/667198092 conn(0x7f59fc0776a0 msgr2=0x7f59fc077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:40.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.613+0000 7f5a033ab640 1 -- 172.21.15.28:0/667198092 shutdown_connections 2024-04-25T01:42:40.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.613+0000 7f5a033ab640 1 -- 172.21.15.28:0/667198092 wait complete. 2024-04-25T01:42:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.614+0000 7f5a033ab640 1 Processor -- start 2024-04-25T01:42:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.614+0000 7f5a033ab640 1 -- start start 2024-04-25T01:42:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.614+0000 7f5a033ab640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f59fc069dc0 0x7f59fc1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.614+0000 7f5a033ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59fc06a780 0x7f59fc198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.614+0000 7f5a033ab640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59fc107730 0x7f59fc19c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.614+0000 7f5a033ab640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc111b40 con 0x7f59fc06a780 2024-04-25T01:42:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.614+0000 7f5a033ab640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc1119c0 con 0x7f59fc107730 2024-04-25T01:42:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.614+0000 7f5a033ab640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc111cc0 con 0x7f59fc069dc0 2024-04-25T01:42:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.614+0000 7f5a00946640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59fc06a780 0x7f59fc198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.614+0000 7f5a00946640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59fc06a780 0x7f59fc198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48124/0 (socket says 172.21.15.28:48124) 2024-04-25T01:42:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.614+0000 7f5a00946640 1 -- 172.21.15.28:0/1637157315 learned_addr learned my addr 172.21.15.28:0/1637157315 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.614+0000 7f5a01948640 1 --2- 172.21.15.28:0/1637157315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59fc107730 0x7f59fc19c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.615+0000 7f5a01147640 1 --2- 172.21.15.28:0/1637157315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f59fc069dc0 0x7f59fc1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.615+0000 7f5a00946640 1 -- 172.21.15.28:0/1637157315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f59fc069dc0 msgr2=0x7f59fc1980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.615+0000 7f5a00946640 1 --2- 172.21.15.28:0/1637157315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f59fc069dc0 0x7f59fc1980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.615+0000 7f5a00946640 1 -- 172.21.15.28:0/1637157315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59fc107730 msgr2=0x7f59fc19c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.615+0000 7f5a00946640 1 --2- 172.21.15.28:0/1637157315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59fc107730 0x7f59fc19c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.615+0000 7f5a00946640 1 -- 172.21.15.28:0/1637157315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59fc19d0b0 con 0x7f59fc06a780 2024-04-25T01:42:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.615+0000 7f5a00946640 1 --2- 172.21.15.28:0/1637157315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59fc06a780 0x7f59fc198600 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009230 tx=0x7f59ec004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.615+0000 7f59f27fc640 1 -- 172.21.15.28:0/1637157315 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec03f030 con 0x7f59fc06a780 2024-04-25T01:42:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.615+0000 7f5a033ab640 1 -- 172.21.15.28:0/1637157315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59fc19d340 con 0x7f59fc06a780 2024-04-25T01:42:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.615+0000 7f59f27fc640 1 -- 172.21.15.28:0/1637157315 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59ec003ad0 con 0x7f59fc06a780 2024-04-25T01:42:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.615+0000 7f59f27fc640 1 -- 172.21.15.28:0/1637157315 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec035070 con 0x7f59fc06a780 2024-04-25T01:42:40.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.615+0000 7f5a033ab640 1 -- 172.21.15.28:0/1637157315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59fc1a4c30 con 0x7f59fc06a780 2024-04-25T01:42:40.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.618+0000 7f59f27fc640 1 -- 172.21.15.28:0/1637157315 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f59ec042030 con 0x7f59fc06a780 2024-04-25T01:42:40.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.618+0000 7f5a0618b740 1 Processor -- start 2024-04-25T01:42:40.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.618+0000 7f5a0618b740 1 -- start start 2024-04-25T01:42:40.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.618+0000 7f5a0618b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938c6630 0x5561938c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.618+0000 7f5a0618b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6f50 0x5561938d10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.618+0000 7f5a0618b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938d1620 0x5561938d3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.618+0000 7f5a0618b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55619382b6b0 con 0x5561938c6f50 2024-04-25T01:42:40.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.618+0000 7f5a0618b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5561937d3880 con 0x5561938c6630 2024-04-25T01:42:40.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.618+0000 7f5a0618b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x556193827670 con 0x5561938d1620 2024-04-25T01:42:40.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.619+0000 7f5a01147640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6f50 0x5561938d10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.619+0000 7f5a01147640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6f50 0x5561938d10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48138/0 (socket says 172.21.15.28:48138) 2024-04-25T01:42:40.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.619+0000 7f5a01147640 1 -- 172.21.15.28:0/2232276550 learned_addr learned my addr 172.21.15.28:0/2232276550 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:40.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.619+0000 7f59f27fc640 1 --2- 172.21.15.28:0/1637157315 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f59e80a2d80 0x7f59e80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.619+0000 7f5a00946640 1 --2- 172.21.15.28:0/2232276550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938d1620 0x5561938d3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.619+0000 7f5a01948640 1 --2- 172.21.15.28:0/2232276550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938c6630 0x5561938c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.619+0000 7f5a01948640 1 --2- 172.21.15.28:0/1637157315 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f59e80a2d80 0x7f59e80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.619+0000 7f5a01147640 1 -- 172.21.15.28:0/2232276550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938d1620 msgr2=0x5561938d3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.619+0000 7f5a01147640 1 --2- 172.21.15.28:0/2232276550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938d1620 0x5561938d3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.619+0000 7f5a01147640 1 -- 172.21.15.28:0/2232276550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938c6630 msgr2=0x5561938c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.619+0000 7f5a01147640 1 --2- 172.21.15.28:0/2232276550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938c6630 0x5561938c6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.619+0000 7f59f27fc640 1 -- 172.21.15.28:0/1637157315 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f59ec0efc30 con 0x7f59fc06a780 2024-04-25T01:42:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.619+0000 7f5a01147640 1 -- 172.21.15.28:0/2232276550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561936097d0 con 0x5561938c6f50 2024-04-25T01:42:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.619+0000 7f5a01948640 1 --2- 172.21.15.28:0/2232276550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938c6630 0x5561938c6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:42:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.619+0000 7f5a01147640 1 --2- 172.21.15.28:0/2232276550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6f50 0x5561938d10e0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009df0 tx=0x7f59f800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa85ebc2f25e80e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.619+0000 7f59e77fe640 1 -- 172.21.15.28:0/2232276550 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f801a480 con 0x5561938c6f50 2024-04-25T01:42:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.619+0000 7f5a01948640 1 --2- 172.21.15.28:0/1637157315 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f59e80a2d80 0x7f59e80a5240 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f59f4005fd0 tx=0x7f59f4005f60 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.619+0000 7f59e77fe640 1 -- 172.21.15.28:0/2232276550 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59f801ab00 con 0x5561938c6f50 2024-04-25T01:42:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.619+0000 7f59e77fe640 1 -- 172.21.15.28:0/2232276550 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8011c80 con 0x5561938c6f50 2024-04-25T01:42:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.620+0000 7f5a0618b740 1 -- 172.21.15.28:0/2232276550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6f50 msgr2=0x5561938d10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.620+0000 7f5a0618b740 1 --2- 172.21.15.28:0/2232276550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6f50 0x5561938d10e0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f59f8009df0 tx=0x7f59f800f730 comp rx=0 tx=0).stop 2024-04-25T01:42:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.620+0000 7f5a0618b740 1 -- 172.21.15.28:0/2232276550 shutdown_connections 2024-04-25T01:42:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.620+0000 7f5a0618b740 1 --2- 172.21.15.28:0/2232276550 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938d1620 0x5561938d3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.620+0000 7f5a0618b740 1 --2- 172.21.15.28:0/2232276550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6f50 0x5561938d10e0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.620+0000 7f5a0618b740 1 --2- 172.21.15.28:0/2232276550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938c6630 0x5561938c6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.620+0000 7f5a0618b740 1 -- 172.21.15.28:0/2232276550 >> 172.21.15.28:0/2232276550 conn(0x5561938c2950 msgr2=0x5561938c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.620+0000 7f5a0618b740 1 -- 172.21.15.28:0/2232276550 shutdown_connections 2024-04-25T01:42:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.620+0000 7f5a0618b740 1 -- 172.21.15.28:0/2232276550 wait complete. 2024-04-25T01:42:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.620+0000 7f5a0618b740 1 -- start start 2024-04-25T01:42:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.620+0000 7f5a0618b740 1 Processor -- start 2024-04-25T01:42:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.621+0000 7f5a0618b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6630 0x5561938ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.621+0000 7f5a0618b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938c6f50 0x5561938ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.621+0000 7f5a0618b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938d1620 0x5561938cef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.621+0000 7f5a0618b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5561938d4410 con 0x5561938c6630 2024-04-25T01:42:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.621+0000 7f5a0618b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5561937e6f50 con 0x5561938d1620 2024-04-25T01:42:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.621+0000 7f5a0618b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5561938d4590 con 0x5561938c6f50 2024-04-25T01:42:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.621+0000 7f5a0618b740 10 client.0 _ll_register_callbacks cb 0x556193822000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.621+0000 7f5a01147640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6630 0x5561938ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.621+0000 7f5a01147640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6630 0x5561938ce4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48146/0 (socket says 172.21.15.28:48146) 2024-04-25T01:42:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.621+0000 7f5a01147640 1 -- 172.21.15.28:0/2604237720 learned_addr learned my addr 172.21.15.28:0/2604237720 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.621+0000 7f5a00946640 1 --2- 172.21.15.28:0/2604237720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938c6f50 0x5561938ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.621+0000 7f5a01948640 1 --2- 172.21.15.28:0/2604237720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938d1620 0x5561938cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.621+0000 7f5a01147640 1 -- 172.21.15.28:0/2604237720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938c6f50 msgr2=0x5561938ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.621+0000 7f5a01147640 1 --2- 172.21.15.28:0/2604237720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938c6f50 0x5561938ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.621+0000 7f5a01147640 1 -- 172.21.15.28:0/2604237720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938d1620 msgr2=0x5561938cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.621+0000 7f5a01147640 1 --2- 172.21.15.28:0/2604237720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938d1620 0x5561938cef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.621+0000 7f5a01147640 1 -- 172.21.15.28:0/2604237720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55619353f3f0 con 0x5561938c6630 2024-04-25T01:42:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.621+0000 7f5a01948640 1 --2- 172.21.15.28:0/2604237720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938d1620 0x5561938cef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.622+0000 7f5a01147640 1 --2- 172.21.15.28:0/2604237720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6630 0x5561938ce4b0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f59f800fc30 tx=0x7f59f8009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=360f33a367d25283 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.622+0000 7f59e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.622+0000 7f59e4ff9640 1 -- 172.21.15.28:0/2604237720 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f802a060 con 0x5561938c6630 2024-04-25T01:42:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.622+0000 7f59e4ff9640 1 -- 172.21.15.28:0/2604237720 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f59f8004c50 con 0x5561938c6630 2024-04-25T01:42:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.622+0000 7f5a0618b740 1 -- 172.21.15.28:0/2604237720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59f801ab00 con 0x5561938c6630 2024-04-25T01:42:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.622+0000 7f59e4ff9640 1 -- 172.21.15.28:0/2604237720 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f59f8031050 con 0x5561938c6630 2024-04-25T01:42:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.622+0000 7f5a0618b740 10 client.19263 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.622+0000 7f5a0618b740 1 -- 172.21.15.28:0/2604237720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5561934ff0f0 con 0x5561938c6630 2024-04-25T01:42:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.622+0000 7f59e4ff9640 1 -- 172.21.15.28:0/2604237720 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59f801ab00 con 0x5561938c6630 2024-04-25T01:42:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.622+0000 7f59e4ff9640 1 -- 172.21.15.28:0/2604237720 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f59f8005040 con 0x5561938c6630 2024-04-25T01:42:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.622+0000 7f5a0618b740 10 client.19263 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.622+0000 7f5a0618b740 10 client.19263 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.622+0000 7f5a0618b740 20 client.19263 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.622+0000 7f5a0618b740 1 --2- 172.21.15.28:0/2604237720 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5561938e1820 0x5561938e3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.622+0000 7f5a0618b740 4 client.19263 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:42:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.622+0000 7f5a0618b740 1 -- 172.21.15.28:0/2604237720 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556193838bf0 con 0x5561938e1820 2024-04-25T01:42:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.623+0000 7f5a00946640 1 --2- 172.21.15.28:0/2604237720 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5561938e1820 0x5561938e3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.623+0000 7f59e4ff9640 10 client.19263 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.623+0000 7f5a00946640 1 --2- 172.21.15.28:0/2604237720 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5561938e1820 0x5561938e3c10 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=939ff94a88d19201 server_cookie=f0140c3d28128907 in_seq=0 out_seq=0 2024-04-25T01:42:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.624+0000 7f59e4ff9640 1 -- 172.21.15.28:0/2604237720 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556193838bf0 con 0x5561938e1820 2024-04-25T01:42:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.624+0000 7f59e4ff9640 10 client.19263 handle_command_reply: tid=0 2024-04-25T01:42:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.624+0000 7f5a0618b740 1 client.19263 shutdown 2024-04-25T01:42:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.624+0000 7f5a0618b740 1 -- 172.21.15.28:0/2604237720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6630 msgr2=0x5561938ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.624+0000 7f5a0618b740 1 --2- 172.21.15.28:0/2604237720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6630 0x5561938ce4b0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f59f800fc30 tx=0x7f59f8009b30 comp rx=0 tx=0).stop 2024-04-25T01:42:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.624+0000 7f5a0618b740 1 -- 172.21.15.28:0/2604237720 shutdown_connections 2024-04-25T01:42:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.624+0000 7f5a0618b740 1 --2- 172.21.15.28:0/2604237720 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5561938e1820 0x5561938e3c10 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.624+0000 7f5a0618b740 1 --2- 172.21.15.28:0/2604237720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938d1620 0x5561938cef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.624+0000 7f5a0618b740 1 --2- 172.21.15.28:0/2604237720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938c6f50 0x5561938ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.624+0000 7f59e4ff9640 0 client.19263 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.624+0000 7f5a0618b740 1 --2- 172.21.15.28:0/2604237720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6630 0x5561938ce4b0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.624+0000 7f5a0618b740 1 -- 172.21.15.28:0/2604237720 >> 172.21.15.28:0/2604237720 conn(0x5561938c2950 msgr2=0x5561938c8650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.624+0000 7f5a0618b740 1 -- 172.21.15.28:0/2604237720 shutdown_connections 2024-04-25T01:42:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.624+0000 7f5a0618b740 1 -- 172.21.15.28:0/2604237720 wait complete. 2024-04-25T01:42:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.624+0000 7f5a0618b740 20 client.19263 trim_cache size 0 max 16384 2024-04-25T01:42:40.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a0618b740 1 Processor -- start 2024-04-25T01:42:40.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a0618b740 1 -- start start 2024-04-25T01:42:40.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a0618b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6630 0x5561938dd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a0618b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938c6f50 0x5561938d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a0618b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938d1620 0x5561938e9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a0618b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc1aeab0 con 0x5561938c6630 2024-04-25T01:42:40.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a0618b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc1b05f0 con 0x5561938d1620 2024-04-25T01:42:40.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a0618b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc1af180 con 0x5561938c6f50 2024-04-25T01:42:40.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a01147640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6630 0x5561938dd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a01147640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6630 0x5561938dd300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48154/0 (socket says 172.21.15.28:48154) 2024-04-25T01:42:40.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a01147640 1 -- 172.21.15.28:0/4255986364 learned_addr learned my addr 172.21.15.28:0/4255986364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:40.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a00946640 1 --2- 172.21.15.28:0/4255986364 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938c6f50 0x5561938d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a01948640 1 --2- 172.21.15.28:0/4255986364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938d1620 0x5561938e9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a01147640 1 -- 172.21.15.28:0/4255986364 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938c6f50 msgr2=0x5561938d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:40.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a01147640 1 --2- 172.21.15.28:0/4255986364 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938c6f50 0x5561938d02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a01147640 1 -- 172.21.15.28:0/4255986364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938d1620 msgr2=0x5561938e9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:40.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a01147640 1 --2- 172.21.15.28:0/4255986364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938d1620 0x5561938e9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a01147640 1 -- 172.21.15.28:0/4255986364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561938c6d80 con 0x5561938c6630 2024-04-25T01:42:40.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a00946640 1 --2- 172.21.15.28:0/4255986364 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938c6f50 0x5561938d02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.642+0000 7f5a01948640 1 --2- 172.21.15.28:0/4255986364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938d1620 0x5561938e9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:42:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.643+0000 7f5a01147640 1 --2- 172.21.15.28:0/4255986364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6630 0x5561938dd300 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f59f8006010 tx=0x7f59f8005aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8ba5eba0462a97 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.643+0000 7f59e6ffd640 1 -- 172.21.15.28:0/4255986364 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f802a040 con 0x5561938c6630 2024-04-25T01:42:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.643+0000 7f59e6ffd640 1 -- 172.21.15.28:0/4255986364 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59f8005780 con 0x5561938c6630 2024-04-25T01:42:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.643+0000 7f59e6ffd640 1 -- 172.21.15.28:0/4255986364 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8004f40 con 0x5561938c6630 2024-04-25T01:42:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.643+0000 7f5a0618b740 1 -- 172.21.15.28:0/4255986364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6630 msgr2=0x5561938dd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.643+0000 7f5a0618b740 1 --2- 172.21.15.28:0/4255986364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6630 0x5561938dd300 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f59f8006010 tx=0x7f59f8005aa0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.643+0000 7f5a0618b740 1 -- 172.21.15.28:0/4255986364 shutdown_connections 2024-04-25T01:42:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.643+0000 7f5a0618b740 1 --2- 172.21.15.28:0/4255986364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938d1620 0x5561938e9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.643+0000 7f5a0618b740 1 --2- 172.21.15.28:0/4255986364 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938c6f50 0x5561938d02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.643+0000 7f5a0618b740 1 --2- 172.21.15.28:0/4255986364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938c6630 0x5561938dd300 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.643+0000 7f5a0618b740 1 -- 172.21.15.28:0/4255986364 >> 172.21.15.28:0/4255986364 conn(0x5561938c2950 msgr2=0x5561938cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.643+0000 7f5a0618b740 1 -- 172.21.15.28:0/4255986364 shutdown_connections 2024-04-25T01:42:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.643+0000 7f5a0618b740 1 -- 172.21.15.28:0/4255986364 wait complete. 2024-04-25T01:42:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.644+0000 7f5a0618b740 1 -- start start 2024-04-25T01:42:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.644+0000 7f5a0618b740 1 Processor -- start 2024-04-25T01:42:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.644+0000 7f5a0618b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938c6630 0x5561938d9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.644+0000 7f5a0618b740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938c6f50 0x5561938d9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.644+0000 7f5a0618b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938d1620 0x5561938d4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.644+0000 7f5a0618b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc1aff20 con 0x5561938d1620 2024-04-25T01:42:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.644+0000 7f5a0618b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc1af850 con 0x5561938c6630 2024-04-25T01:42:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.644+0000 7f5a0618b740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc1acab0 con 0x5561938c6f50 2024-04-25T01:42:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.644+0000 7f5a0618b740 10 client.0 _ll_register_callbacks cb 0x7f59f8009c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:42:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.644+0000 7f5a01948640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938d1620 0x5561938d4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.644+0000 7f5a01948640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938d1620 0x5561938d4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48160/0 (socket says 172.21.15.28:48160) 2024-04-25T01:42:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.644+0000 7f5a01147640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938c6630 0x5561938d9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.644+0000 7f5a01948640 1 -- 172.21.15.28:0/3732762117 learned_addr learned my addr 172.21.15.28:0/3732762117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:42:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.644+0000 7f5a00946640 1 --2- 172.21.15.28:0/3732762117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938c6f50 0x5561938d9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f5a01948640 1 -- 172.21.15.28:0/3732762117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938c6f50 msgr2=0x5561938d9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f5a01948640 1 --2- 172.21.15.28:0/3732762117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938c6f50 0x5561938d9d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f5a01948640 1 -- 172.21.15.28:0/3732762117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938c6630 msgr2=0x5561938d9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:42:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f5a01948640 1 --2- 172.21.15.28:0/3732762117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938c6630 0x5561938d9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f5a01948640 1 -- 172.21.15.28:0/3732762117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561938d57a0 con 0x5561938d1620 2024-04-25T01:42:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f5a01948640 1 --2- 172.21.15.28:0/3732762117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938d1620 0x5561938d4fd0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f59f400fde0 tx=0x7f59f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7cf662ec34166ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:42:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f59e4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:42:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f59e4ff9640 1 -- 172.21.15.28:0/3732762117 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f402b060 con 0x5561938d1620 2024-04-25T01:42:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f59e4ff9640 1 -- 172.21.15.28:0/3732762117 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f59f401bd40 con 0x5561938d1620 2024-04-25T01:42:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f5a0618b740 1 -- 172.21.15.28:0/3732762117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59f8005780 con 0x5561938d1620 2024-04-25T01:42:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f59e4ff9640 1 -- 172.21.15.28:0/3732762117 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f59f4032050 con 0x5561938d1620 2024-04-25T01:42:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f5a0618b740 10 client.19269 fetch_fsmap learned FSMap version 28 2024-04-25T01:42:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f5a0618b740 1 -- 172.21.15.28:0/3732762117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556193947730 con 0x5561938d1620 2024-04-25T01:42:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f59e4ff9640 1 -- 172.21.15.28:0/3732762117 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59f8005780 con 0x5561938d1620 2024-04-25T01:42:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f59e4ff9640 1 -- 172.21.15.28:0/3732762117 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f59f4025600 con 0x5561938d1620 2024-04-25T01:42:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f5a0618b740 10 client.19269 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:42:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f5a0618b740 10 client.19269 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:42:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f5a0618b740 20 client.19269 populate_metadata read hostname 'smithi028' 2024-04-25T01:42:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f5a0618b740 1 --2- 172.21.15.28:0/3732762117 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556193947ee0 0x55619394a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:42:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f5a0618b740 4 client.19269 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:42:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.645+0000 7f5a0618b740 1 -- 172.21.15.28:0/3732762117 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556193832f40 con 0x556193947ee0 2024-04-25T01:42:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.646+0000 7f5a01147640 1 --2- 172.21.15.28:0/3732762117 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556193947ee0 0x55619394a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:42:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.646+0000 7f5a01147640 1 --2- 172.21.15.28:0/3732762117 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556193947ee0 0x55619394a300 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8be902702d9c5f6e server_cookie=622c1c6a734c2195 in_seq=0 out_seq=0 2024-04-25T01:42:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.646+0000 7f59e4ff9640 10 client.19269 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.646+0000 7f59e4ff9640 1 -- 172.21.15.28:0/3732762117 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556193832f40 con 0x556193947ee0 2024-04-25T01:42:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.646+0000 7f59e4ff9640 10 client.19269 handle_command_reply: tid=0 2024-04-25T01:42:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.646+0000 7f5a0618b740 1 client.19269 shutdown 2024-04-25T01:42:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.646+0000 7f5a0618b740 1 -- 172.21.15.28:0/3732762117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938d1620 msgr2=0x5561938d4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.646+0000 7f5a0618b740 1 --2- 172.21.15.28:0/3732762117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938d1620 0x5561938d4fd0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f59f400fde0 tx=0x7f59f40046c0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.646+0000 7f5a0618b740 1 -- 172.21.15.28:0/3732762117 shutdown_connections 2024-04-25T01:42:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.646+0000 7f5a0618b740 1 --2- 172.21.15.28:0/3732762117 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x556193947ee0 0x55619394a300 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.646+0000 7f5a0618b740 1 --2- 172.21.15.28:0/3732762117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5561938d1620 0x5561938d4fd0 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.646+0000 7f5a0618b740 1 --2- 172.21.15.28:0/3732762117 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5561938c6f50 0x5561938d9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.646+0000 7f59e4ff9640 0 client.19269 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:42:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.646+0000 7f5a0618b740 1 --2- 172.21.15.28:0/3732762117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5561938c6630 0x5561938d9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.646+0000 7f5a0618b740 1 -- 172.21.15.28:0/3732762117 >> 172.21.15.28:0/3732762117 conn(0x5561938c2950 msgr2=0x5561938c1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.647+0000 7f5a0618b740 1 -- 172.21.15.28:0/3732762117 shutdown_connections 2024-04-25T01:42:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.647+0000 7f5a0618b740 1 -- 172.21.15.28:0/3732762117 wait complete. 2024-04-25T01:42:40.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.647+0000 7f5a0618b740 20 client.19269 trim_cache size 0 max 16384 2024-04-25T01:42:40.641 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:42:40.641 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:42:40.642 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-25T01:42:40.642 INFO:teuthology.orchestra.run.smithi028.stdout: "6a1c001b-ee70-46fc-bd43-b286af1f4837": { 2024-04-25T01:42:40.642 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-25T01:42:40.642 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "6a1c001b-ee70-46fc-bd43-b286af1f4837", 2024-04-25T01:42:40.642 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-25T01:42:40.642 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:42:40.642 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:42:40.642 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:42:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.647+0000 7f5a033ab640 1 -- 172.21.15.28:0/1637157315 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f59e80a2d80 msgr2=0x7f59e80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.647+0000 7f5a033ab640 1 --2- 172.21.15.28:0/1637157315 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f59e80a2d80 0x7f59e80a5240 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f59f4005fd0 tx=0x7f59f4005f60 comp rx=0 tx=0).stop 2024-04-25T01:42:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.647+0000 7f5a033ab640 1 -- 172.21.15.28:0/1637157315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59fc06a780 msgr2=0x7f59fc198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:42:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.647+0000 7f5a033ab640 1 --2- 172.21.15.28:0/1637157315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59fc06a780 0x7f59fc198600 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009230 tx=0x7f59ec004270 comp rx=0 tx=0).stop 2024-04-25T01:42:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.647+0000 7f5a033ab640 1 -- 172.21.15.28:0/1637157315 shutdown_connections 2024-04-25T01:42:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.647+0000 7f5a033ab640 1 --2- 172.21.15.28:0/1637157315 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f59e80a2d80 0x7f59e80a5240 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.647+0000 7f5a033ab640 1 --2- 172.21.15.28:0/1637157315 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f59fc107730 0x7f59fc19c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.647+0000 7f5a033ab640 1 --2- 172.21.15.28:0/1637157315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59fc06a780 0x7f59fc198600 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.647+0000 7f5a033ab640 1 --2- 172.21.15.28:0/1637157315 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f59fc069dc0 0x7f59fc1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:42:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.647+0000 7f5a033ab640 1 -- 172.21.15.28:0/1637157315 >> 172.21.15.28:0/1637157315 conn(0x7f59fc0776a0 msgr2=0x7f59fc0fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:42:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.647+0000 7f5a033ab640 1 -- 172.21.15.28:0/1637157315 shutdown_connections 2024-04-25T01:42:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:42:40.647+0000 7f5a033ab640 1 -- 172.21.15.28:0/1637157315 wait complete. 2024-04-25T01:42:40.646 INFO:tasks.cephfs.filesystem:scrub status for tag:6a1c001b-ee70-46fc-bd43-b286af1f4837 - {'path': '/', 'tag': '6a1c001b-ee70-46fc-bd43-b286af1f4837', 'options': 'recursive,force'} 2024-04-25T01:42:42.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:42 smithi032 ceph-mon[32461]: scrub queued for path: / 2024-04-25T01:42:42.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:42 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:42:42.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:42 smithi032 ceph-mon[32461]: scrub summary: active paths [/] 2024-04-25T01:42:42.245 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:42 smithi032 ceph-mon[32461]: scrub summary: idle+waiting paths [/] 2024-04-25T01:42:42.246 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:42 smithi032 ceph-mon[32461]: pgmap v299: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:42:42.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:42 smithi028 ceph-mon[29390]: scrub queued for path: / 2024-04-25T01:42:42.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:42 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:42:42.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:42 smithi028 ceph-mon[29390]: scrub summary: active paths [/] 2024-04-25T01:42:42.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:42 smithi028 ceph-mon[29390]: scrub summary: idle+waiting paths [/] 2024-04-25T01:42:42.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:42 smithi028 ceph-mon[29390]: pgmap v299: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:42:42.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:42 smithi130 ceph-mon[32702]: scrub queued for path: / 2024-04-25T01:42:42.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:42 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:42:42.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:42 smithi130 ceph-mon[32702]: scrub summary: active paths [/] 2024-04-25T01:42:42.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:42 smithi130 ceph-mon[32702]: scrub summary: idle+waiting paths [/] 2024-04-25T01:42:42.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:42 smithi130 ceph-mon[32702]: pgmap v299: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:42:44.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:44 smithi028 ceph-mon[29390]: pgmap v300: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 12 op/s 2024-04-25T01:42:44.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:44 smithi028 ceph-mon[29390]: scrub summary: idle 2024-04-25T01:42:44.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:44 smithi130 ceph-mon[32702]: pgmap v300: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 12 op/s 2024-04-25T01:42:44.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:44 smithi130 ceph-mon[32702]: scrub summary: idle 2024-04-25T01:42:44.495 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:44 smithi032 ceph-mon[32461]: pgmap v300: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 12 op/s 2024-04-25T01:42:44.495 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:44 smithi032 ceph-mon[32461]: scrub summary: idle 2024-04-25T01:42:46.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:46 smithi028 ceph-mon[29390]: pgmap v301: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:42:46.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:46 smithi130 ceph-mon[32702]: pgmap v301: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:42:46.495 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:46 smithi032 ceph-mon[32461]: pgmap v301: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:42:48.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:48 smithi028 ceph-mon[29390]: pgmap v302: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-25T01:42:48.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:48 smithi130 ceph-mon[32702]: pgmap v302: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-25T01:42:48.767 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:48 smithi032 ceph-mon[32461]: pgmap v302: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-25T01:42:50.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:50 smithi028 ceph-mon[29390]: pgmap v303: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-04-25T01:42:50.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:50 smithi130 ceph-mon[32702]: pgmap v303: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-04-25T01:42:50.495 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:50 smithi032 ceph-mon[32461]: pgmap v303: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-04-25T01:42:52.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:52 smithi028 ceph-mon[29390]: pgmap v304: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:42:52.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:52 smithi130 ceph-mon[32702]: pgmap v304: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:42:52.495 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:52 smithi032 ceph-mon[32461]: pgmap v304: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T01:42:54.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:54 smithi028 ceph-mon[29390]: pgmap v305: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 12 op/s 2024-04-25T01:42:54.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:54 smithi130 ceph-mon[32702]: pgmap v305: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 12 op/s 2024-04-25T01:42:54.495 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:54 smithi032 ceph-mon[32461]: pgmap v305: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 12 op/s 2024-04-25T01:42:56.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:56 smithi028 ceph-mon[29390]: pgmap v306: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:42:56.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:56 smithi130 ceph-mon[32702]: pgmap v306: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:42:56.495 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:56 smithi032 ceph-mon[32461]: pgmap v306: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:42:58.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:42:58 smithi028 ceph-mon[29390]: pgmap v307: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:42:58.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:42:58 smithi130 ceph-mon[32702]: pgmap v307: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:42:58.495 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:42:58 smithi032 ceph-mon[32461]: pgmap v307: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:43:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:00 smithi028 ceph-mon[29390]: pgmap v308: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:43:00.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:00 smithi130 ceph-mon[32702]: pgmap v308: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:43:00.495 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:00 smithi032 ceph-mon[32461]: pgmap v308: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:43:02.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:02 smithi028 ceph-mon[29390]: pgmap v309: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:02.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:02 smithi130 ceph-mon[32702]: pgmap v309: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:02.495 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:02 smithi032 ceph-mon[32461]: pgmap v309: 129 pgs: 129 active+clean; 11 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:04.298 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:04 smithi032 ceph-mon[32461]: pgmap v310: 129 pgs: 129 active+clean; 11 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-25T01:43:04.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:04 smithi028 ceph-mon[29390]: pgmap v310: 129 pgs: 129 active+clean; 11 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-25T01:43:04.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:04 smithi130 ceph-mon[32702]: pgmap v310: 129 pgs: 129 active+clean; 11 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-04-25T01:43:06.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:06 smithi028 ceph-mon[29390]: pgmap v311: 129 pgs: 129 active+clean; 11 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-25T01:43:06.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:06 smithi130 ceph-mon[32702]: pgmap v311: 129 pgs: 129 active+clean; 11 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-25T01:43:06.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:06 smithi032 ceph-mon[32461]: pgmap v311: 129 pgs: 129 active+clean; 11 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-04-25T01:43:07.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-25T01:43:08.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.092+0000 7f7a6db30640 1 -- 172.21.15.28:0/3480102530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a68107760 msgr2=0x7f7a68107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:08.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.092+0000 7f7a6db30640 1 --2- 172.21.15.28:0/3480102530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a68107760 0x7f7a68107b40 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f7a58009930 tx=0x7f7a5802a420 comp rx=0 tx=0).stop 2024-04-25T01:43:08.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.092+0000 7f7a6db30640 1 -- 172.21.15.28:0/3480102530 shutdown_connections 2024-04-25T01:43:08.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.092+0000 7f7a6db30640 1 --2- 172.21.15.28:0/3480102530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a68102110 0x7f7a6810e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.092+0000 7f7a6db30640 1 --2- 172.21.15.28:0/3480102530 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a68101750 0x7f7a68101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.092+0000 7f7a6db30640 1 --2- 172.21.15.28:0/3480102530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a68107760 0x7f7a68107b40 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.092+0000 7f7a6db30640 1 -- 172.21.15.28:0/3480102530 >> 172.21.15.28:0/3480102530 conn(0x7f7a68078bd0 msgr2=0x7f7a68078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.092+0000 7f7a6db30640 1 -- 172.21.15.28:0/3480102530 shutdown_connections 2024-04-25T01:43:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.092+0000 7f7a6db30640 1 -- 172.21.15.28:0/3480102530 wait complete. 2024-04-25T01:43:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.092+0000 7f7a6db30640 1 Processor -- start 2024-04-25T01:43:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.093+0000 7f7a6db30640 1 -- start start 2024-04-25T01:43:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.093+0000 7f7a6db30640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a68101750 0x7f7a68073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.093+0000 7f7a6db30640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a68102110 0x7f7a68073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.093+0000 7f7a6db30640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a68107760 0x7f7a680744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.093+0000 7f7a6db30640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a68111b70 con 0x7f7a68102110 2024-04-25T01:43:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.093+0000 7f7a6db30640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7a681119f0 con 0x7f7a68101750 2024-04-25T01:43:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.093+0000 7f7a6db30640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7a68111cf0 con 0x7f7a68107760 2024-04-25T01:43:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.093+0000 7f7a66ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a68102110 0x7f7a68073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.093+0000 7f7a66ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a68102110 0x7f7a68073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56484/0 (socket says 172.21.15.28:56484) 2024-04-25T01:43:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.093+0000 7f7a66ffd640 1 -- 172.21.15.28:0/3062298606 learned_addr learned my addr 172.21.15.28:0/3062298606 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.093+0000 7f7a677fe640 1 --2- 172.21.15.28:0/3062298606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a68101750 0x7f7a68073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.093+0000 7f7a67fff640 1 --2- 172.21.15.28:0/3062298606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a68107760 0x7f7a680744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.093+0000 7f7a66ffd640 1 -- 172.21.15.28:0/3062298606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a68107760 msgr2=0x7f7a680744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.093+0000 7f7a66ffd640 1 --2- 172.21.15.28:0/3062298606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a68107760 0x7f7a680744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.093+0000 7f7a66ffd640 1 -- 172.21.15.28:0/3062298606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a68101750 msgr2=0x7f7a68073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.093+0000 7f7a66ffd640 1 --2- 172.21.15.28:0/3062298606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a68101750 0x7f7a68073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.093+0000 7f7a66ffd640 1 -- 172.21.15.28:0/3062298606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a681a6470 con 0x7f7a68102110 2024-04-25T01:43:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.093+0000 7f7a677fe640 1 --2- 172.21.15.28:0/3062298606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a68101750 0x7f7a68073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.093+0000 7f7a66ffd640 1 --2- 172.21.15.28:0/3062298606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a68102110 0x7f7a68073f80 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c00e9f0 tx=0x7f7a5c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.094+0000 7f7a64ff9640 1 -- 172.21.15.28:0/3062298606 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c00cd80 con 0x7f7a68102110 2024-04-25T01:43:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.094+0000 7f7a6db30640 1 -- 172.21.15.28:0/3062298606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a681a6760 con 0x7f7a68102110 2024-04-25T01:43:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.094+0000 7f7a64ff9640 1 -- 172.21.15.28:0/3062298606 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a5c005500 con 0x7f7a68102110 2024-04-25T01:43:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.094+0000 7f7a64ff9640 1 -- 172.21.15.28:0/3062298606 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c010690 con 0x7f7a68102110 2024-04-25T01:43:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.094+0000 7f7a6db30640 1 -- 172.21.15.28:0/3062298606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a681a6c60 con 0x7f7a68102110 2024-04-25T01:43:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.095+0000 7f7a6db30640 1 -- 172.21.15.28:0/3062298606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a6804f500 con 0x7f7a68102110 2024-04-25T01:43:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.096+0000 7f7a64ff9640 1 -- 172.21.15.28:0/3062298606 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7a5c0046c0 con 0x7f7a68102110 2024-04-25T01:43:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.096+0000 7f7a64ff9640 1 --2- 172.21.15.28:0/3062298606 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7a500a2da0 0x7f7a500a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.097+0000 7f7a677fe640 1 --2- 172.21.15.28:0/3062298606 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7a500a2da0 0x7f7a500a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.097+0000 7f7a64ff9640 1 -- 172.21.15.28:0/3062298606 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f7a5c0cfda0 con 0x7f7a68102110 2024-04-25T01:43:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.097+0000 7f7a677fe640 1 --2- 172.21.15.28:0/3062298606 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7a500a2da0 0x7f7a500a5260 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7a58000c00 tx=0x7f7a58005a20 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.101+0000 7f7a64ff9640 1 -- 172.21.15.28:0/3062298606 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7a5c09c5d0 con 0x7f7a68102110 2024-04-25T01:43:08.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.308+0000 7f7a6db30640 1 -- 172.21.15.28:0/3062298606 --> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f7a680ff6a0 con 0x7f7a500a2da0 2024-04-25T01:43:08.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.316+0000 7f7a64ff9640 1 -- 172.21.15.28:0/3062298606 <== mgr.15561 v2:172.21.15.28:6800/712762371 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f7a680ff6a0 con 0x7f7a500a2da0 2024-04-25T01:43:08.305 INFO:teuthology.orchestra.run.smithi028.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-25T00:00:00", "created": "2024-04-25T01:33:06", "first": "2024-04-25T01:34:00", "last": "2024-04-25T01:43:00", "last_pruned": "2024-04-25T01:43:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-04-25T01:43:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.317+0000 7f7a6db30640 1 -- 172.21.15.28:0/3062298606 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7a500a2da0 msgr2=0x7f7a500a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.317+0000 7f7a6db30640 1 --2- 172.21.15.28:0/3062298606 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7a500a2da0 0x7f7a500a5260 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7a58000c00 tx=0x7f7a58005a20 comp rx=0 tx=0).stop 2024-04-25T01:43:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.317+0000 7f7a6db30640 1 -- 172.21.15.28:0/3062298606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a68102110 msgr2=0x7f7a68073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.317+0000 7f7a6db30640 1 --2- 172.21.15.28:0/3062298606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a68102110 0x7f7a68073f80 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c00e9f0 tx=0x7f7a5c00edf0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.318+0000 7f7a6db30640 1 -- 172.21.15.28:0/3062298606 shutdown_connections 2024-04-25T01:43:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.318+0000 7f7a6db30640 1 --2- 172.21.15.28:0/3062298606 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7a500a2da0 0x7f7a500a5260 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.318+0000 7f7a6db30640 1 --2- 172.21.15.28:0/3062298606 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7a68107760 0x7f7a680744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.318+0000 7f7a6db30640 1 --2- 172.21.15.28:0/3062298606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a68102110 0x7f7a68073f80 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.318+0000 7f7a6db30640 1 --2- 172.21.15.28:0/3062298606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7a68101750 0x7f7a68073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.318+0000 7f7a6db30640 1 -- 172.21.15.28:0/3062298606 >> 172.21.15.28:0/3062298606 conn(0x7f7a68078bd0 msgr2=0x7f7a680fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.318+0000 7f7a6db30640 1 -- 172.21.15.28:0/3062298606 shutdown_connections 2024-04-25T01:43:08.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.318+0000 7f7a6db30640 1 -- 172.21.15.28:0/3062298606 wait complete. 2024-04-25T01:43:08.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:08 smithi028 ceph-mon[29390]: pgmap v312: 129 pgs: 129 active+clean; 11 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-04-25T01:43:08.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-25T01:43:08.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.432+0000 7f26c129a640 1 -- 172.21.15.28:0/1309789033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26bc1029d0 msgr2=0x7f26bc102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:08.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.432+0000 7f26c129a640 1 --2- 172.21.15.28:0/1309789033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26bc1029d0 0x7f26bc102db0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f26ac009900 tx=0x7f26ac02a510 comp rx=0 tx=0).stop 2024-04-25T01:43:08.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.433+0000 7f26c129a640 1 -- 172.21.15.28:0/1309789033 shutdown_connections 2024-04-25T01:43:08.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.433+0000 7f26c129a640 1 --2- 172.21.15.28:0/1309789033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26bc10bd10 0x7f26bc10e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.433+0000 7f26c129a640 1 --2- 172.21.15.28:0/1309789033 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26bc1032f0 0x7f26bc10b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.433+0000 7f26c129a640 1 --2- 172.21.15.28:0/1309789033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26bc1029d0 0x7f26bc102db0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.433+0000 7f26c129a640 1 -- 172.21.15.28:0/1309789033 >> 172.21.15.28:0/1309789033 conn(0x7f26bc0777c0 msgr2=0x7f26bc077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:08.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.433+0000 7f26c129a640 1 -- 172.21.15.28:0/1309789033 shutdown_connections 2024-04-25T01:43:08.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.433+0000 7f26c129a640 1 -- 172.21.15.28:0/1309789033 wait complete. 2024-04-25T01:43:08.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.433+0000 7f26c129a640 1 Processor -- start 2024-04-25T01:43:08.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.433+0000 7f26c129a640 1 -- start start 2024-04-25T01:43:08.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.434+0000 7f26c129a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26bc1029d0 0x7f26bc198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:08.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.434+0000 7f26c129a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26bc1032f0 0x7f26bc198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:08.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.434+0000 7f26c129a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26bc10bd10 0x7f26bc19c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:08.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.434+0000 7f26c129a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f26bc111b50 con 0x7f26bc1029d0 2024-04-25T01:43:08.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.434+0000 7f26c129a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f26bc1119d0 con 0x7f26bc10bd10 2024-04-25T01:43:08.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.434+0000 7f26c129a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f26bc111cd0 con 0x7f26bc1032f0 2024-04-25T01:43:08.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.434+0000 7f26baffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26bc1029d0 0x7f26bc198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:08.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.434+0000 7f26baffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26bc1029d0 0x7f26bc198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56510/0 (socket says 172.21.15.28:56510) 2024-04-25T01:43:08.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.434+0000 7f26baffd640 1 -- 172.21.15.28:0/1371451935 learned_addr learned my addr 172.21.15.28:0/1371451935 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:08.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.434+0000 7f26ba7fc640 1 --2- 172.21.15.28:0/1371451935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26bc1032f0 0x7f26bc198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:08.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.434+0000 7f26bb7fe640 1 --2- 172.21.15.28:0/1371451935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26bc10bd10 0x7f26bc19c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:08.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.434+0000 7f26baffd640 1 -- 172.21.15.28:0/1371451935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26bc1032f0 msgr2=0x7f26bc198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:08.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.434+0000 7f26baffd640 1 --2- 172.21.15.28:0/1371451935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26bc1032f0 0x7f26bc198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.434+0000 7f26baffd640 1 -- 172.21.15.28:0/1371451935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26bc10bd10 msgr2=0x7f26bc19c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:08.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.434+0000 7f26baffd640 1 --2- 172.21.15.28:0/1371451935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26bc10bd10 0x7f26bc19c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.434+0000 7f26baffd640 1 -- 172.21.15.28:0/1371451935 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26bc19cff0 con 0x7f26bc1029d0 2024-04-25T01:43:08.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.434+0000 7f26bb7fe640 1 --2- 172.21.15.28:0/1371451935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26bc10bd10 0x7f26bc19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:43:08.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.434+0000 7f26baffd640 1 --2- 172.21.15.28:0/1371451935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26bc1029d0 0x7f26bc198050 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f26ac006010 tx=0x7f26ac0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:08.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.434+0000 7f26a3fff640 1 -- 172.21.15.28:0/1371451935 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ac03f030 con 0x7f26bc1029d0 2024-04-25T01:43:08.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.435+0000 7f26a3fff640 1 -- 172.21.15.28:0/1371451935 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26ac003ce0 con 0x7f26bc1029d0 2024-04-25T01:43:08.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.435+0000 7f26c129a640 1 -- 172.21.15.28:0/1371451935 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26bc19d280 con 0x7f26bc1029d0 2024-04-25T01:43:08.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.435+0000 7f26a3fff640 1 -- 172.21.15.28:0/1371451935 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ac035070 con 0x7f26bc1029d0 2024-04-25T01:43:08.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.435+0000 7f26c129a640 1 -- 172.21.15.28:0/1371451935 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26bc1a4b70 con 0x7f26bc1029d0 2024-04-25T01:43:08.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.436+0000 7f26c129a640 1 -- 172.21.15.28:0/1371451935 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26bc04f500 con 0x7f26bc1029d0 2024-04-25T01:43:08.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.437+0000 7f26a3fff640 1 -- 172.21.15.28:0/1371451935 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f26ac042030 con 0x7f26bc1029d0 2024-04-25T01:43:08.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.437+0000 7f26a3fff640 1 --2- 172.21.15.28:0/1371451935 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f26a40a2da0 0x7f26a40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:08.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.437+0000 7f26ba7fc640 1 --2- 172.21.15.28:0/1371451935 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f26a40a2da0 0x7f26a40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:08.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.437+0000 7f26a3fff640 1 -- 172.21.15.28:0/1371451935 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9438+0+0 (secure 0 0 0) 0x7f26ac0efc50 con 0x7f26bc1029d0 2024-04-25T01:43:08.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.438+0000 7f26ba7fc640 1 --2- 172.21.15.28:0/1371451935 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f26a40a2da0 0x7f26a40a5260 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f26bc10db10 tx=0x7f26b0009110 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:08.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.442+0000 7f26a3fff640 1 -- 172.21.15.28:0/1371451935 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f26ac0f5020 con 0x7f26bc1029d0 2024-04-25T01:43:08.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:08 smithi130 ceph-mon[32702]: pgmap v312: 129 pgs: 129 active+clean; 11 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-04-25T01:43:08.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:08 smithi032 ceph-mon[32461]: pgmap v312: 129 pgs: 129 active+clean; 11 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 28 op/s 2024-04-25T01:43:08.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.647+0000 7f26c129a640 1 -- 172.21.15.28:0/1371451935 --> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f26bc061b80 con 0x7f26a40a2da0 2024-04-25T01:43:08.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.653+0000 7f26a3fff640 1 -- 172.21.15.28:0/1371451935 <== mgr.15561 v2:172.21.15.28:6800/712762371 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f26bc061b80 con 0x7f26a40a2da0 2024-04-25T01:43:08.643 INFO:teuthology.orchestra.run.smithi028.stdout:/ 1m 6m3h 2024-04-25T01:43:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.655+0000 7f26c129a640 1 -- 172.21.15.28:0/1371451935 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f26a40a2da0 msgr2=0x7f26a40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.655+0000 7f26c129a640 1 --2- 172.21.15.28:0/1371451935 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f26a40a2da0 0x7f26a40a5260 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f26bc10db10 tx=0x7f26b0009110 comp rx=0 tx=0).stop 2024-04-25T01:43:08.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.655+0000 7f26c129a640 1 -- 172.21.15.28:0/1371451935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26bc1029d0 msgr2=0x7f26bc198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.655+0000 7f26c129a640 1 --2- 172.21.15.28:0/1371451935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26bc1029d0 0x7f26bc198050 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f26ac006010 tx=0x7f26ac0042f0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.655+0000 7f26c129a640 1 -- 172.21.15.28:0/1371451935 shutdown_connections 2024-04-25T01:43:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.655+0000 7f26c129a640 1 --2- 172.21.15.28:0/1371451935 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f26a40a2da0 0x7f26a40a5260 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.655+0000 7f26c129a640 1 --2- 172.21.15.28:0/1371451935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f26bc10bd10 0x7f26bc19c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.655+0000 7f26c129a640 1 --2- 172.21.15.28:0/1371451935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f26bc1032f0 0x7f26bc198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.655+0000 7f26c129a640 1 --2- 172.21.15.28:0/1371451935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26bc1029d0 0x7f26bc198050 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.655+0000 7f26c129a640 1 -- 172.21.15.28:0/1371451935 >> 172.21.15.28:0/1371451935 conn(0x7f26bc0777c0 msgr2=0x7f26bc0fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.655+0000 7f26c129a640 1 -- 172.21.15.28:0/1371451935 shutdown_connections 2024-04-25T01:43:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:08.655+0000 7f26c129a640 1 -- 172.21.15.28:0/1371451935 wait complete. 2024-04-25T01:43:08.656 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-25T01:43:08.667 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-25T01:43:09.037 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:09 smithi028 ceph-mon[29390]: from='client.19275 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:43:09.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:09 smithi130 ceph-mon[32702]: from='client.19275 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:43:09.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:09 smithi032 ceph-mon[32461]: from='client.19275 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:43:10.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:10 smithi028 ceph-mon[29390]: from='client.19281 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:43:10.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:10 smithi028 ceph-mon[29390]: pgmap v313: 129 pgs: 129 active+clean; 11 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-04-25T01:43:10.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:10 smithi028 ceph-mon[29390]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T01:43:10.470 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:10 smithi130 ceph-mon[32702]: from='client.19281 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:43:10.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:10 smithi130 ceph-mon[32702]: pgmap v313: 129 pgs: 129 active+clean; 11 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-04-25T01:43:10.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:10 smithi130 ceph-mon[32702]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T01:43:10.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:10 smithi032 ceph-mon[32461]: from='client.19281 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:43:10.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:10 smithi032 ceph-mon[32461]: pgmap v313: 129 pgs: 129 active+clean; 11 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2024-04-25T01:43:10.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:10 smithi032 ceph-mon[32461]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T01:43:10.647 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:43:10.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:43:10.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.774+0000 7fe9e8742640 1 -- 172.21.15.28:0/971878921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e0107730 msgr2=0x7fe9e0107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:10.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.774+0000 7fe9e8742640 1 --2- 172.21.15.28:0/971878921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e0107730 0x7fe9e0107b10 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009930 tx=0x7fe9dc02a420 comp rx=0 tx=0).stop 2024-04-25T01:43:10.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.774+0000 7fe9e8742640 1 -- 172.21.15.28:0/971878921 shutdown_connections 2024-04-25T01:43:10.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.774+0000 7fe9e8742640 1 --2- 172.21.15.28:0/971878921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9e006a780 0x7fe9e010e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.774+0000 7fe9e8742640 1 --2- 172.21.15.28:0/971878921 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9e0069dc0 0x7fe9e006a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.774+0000 7fe9e8742640 1 --2- 172.21.15.28:0/971878921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e0107730 0x7fe9e0107b10 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.774+0000 7fe9e8742640 1 -- 172.21.15.28:0/971878921 >> 172.21.15.28:0/971878921 conn(0x7fe9e00776a0 msgr2=0x7fe9e0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:10.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.774+0000 7fe9e8742640 1 -- 172.21.15.28:0/971878921 shutdown_connections 2024-04-25T01:43:10.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.774+0000 7fe9e8742640 1 -- 172.21.15.28:0/971878921 wait complete. 2024-04-25T01:43:10.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.775+0000 7fe9e8742640 1 Processor -- start 2024-04-25T01:43:10.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.775+0000 7fe9e8742640 1 -- start start 2024-04-25T01:43:10.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.775+0000 7fe9e8742640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9e0069dc0 0x7fe9e01980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.775+0000 7fe9e8742640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e006a780 0x7fe9e01985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.775+0000 7fe9e8742640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9e0107730 0x7fe9e019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.775+0000 7fe9e8742640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e0111b00 con 0x7fe9e006a780 2024-04-25T01:43:10.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.775+0000 7fe9e8742640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e0111980 con 0x7fe9e0107730 2024-04-25T01:43:10.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.775+0000 7fe9e8742640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e0111c80 con 0x7fe9e0069dc0 2024-04-25T01:43:10.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.775+0000 7fe9e5cdd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e006a780 0x7fe9e01985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.775+0000 7fe9e5cdd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e006a780 0x7fe9e01985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56520/0 (socket says 172.21.15.28:56520) 2024-04-25T01:43:10.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.775+0000 7fe9e5cdd640 1 -- 172.21.15.28:0/985135872 learned_addr learned my addr 172.21.15.28:0/985135872 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:10.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.776+0000 7fe9e64de640 1 --2- 172.21.15.28:0/985135872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9e0069dc0 0x7fe9e01980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.776+0000 7fe9e6cdf640 1 --2- 172.21.15.28:0/985135872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9e0107730 0x7fe9e019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.776+0000 7fe9e5cdd640 1 -- 172.21.15.28:0/985135872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9e0069dc0 msgr2=0x7fe9e01980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:10.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.776+0000 7fe9e5cdd640 1 --2- 172.21.15.28:0/985135872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9e0069dc0 0x7fe9e01980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.776+0000 7fe9e5cdd640 1 -- 172.21.15.28:0/985135872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9e0107730 msgr2=0x7fe9e019c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:10.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.776+0000 7fe9e5cdd640 1 --2- 172.21.15.28:0/985135872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9e0107730 0x7fe9e019c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.776+0000 7fe9e5cdd640 1 -- 172.21.15.28:0/985135872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e019d090 con 0x7fe9e006a780 2024-04-25T01:43:10.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.776+0000 7fe9e6cdf640 1 --2- 172.21.15.28:0/985135872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9e0107730 0x7fe9e019c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:10.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.776+0000 7fe9e5cdd640 1 --2- 172.21.15.28:0/985135872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e006a780 0x7fe9e01985e0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000d950 tx=0x7fe9d000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:10.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.776+0000 7fe9d77fe640 1 -- 172.21.15.28:0/985135872 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0017070 con 0x7fe9e006a780 2024-04-25T01:43:10.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.776+0000 7fe9d77fe640 1 -- 172.21.15.28:0/985135872 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9d0009c30 con 0x7fe9e006a780 2024-04-25T01:43:10.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.776+0000 7fe9e8742640 1 -- 172.21.15.28:0/985135872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9e019d380 con 0x7fe9e006a780 2024-04-25T01:43:10.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.776+0000 7fe9d77fe640 1 -- 172.21.15.28:0/985135872 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d000f6b0 con 0x7fe9e006a780 2024-04-25T01:43:10.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.776+0000 7fe9e8742640 1 -- 172.21.15.28:0/985135872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9e01a4c10 con 0x7fe9e006a780 2024-04-25T01:43:10.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.779+0000 7fe9d77fe640 1 -- 172.21.15.28:0/985135872 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe9d00109d0 con 0x7fe9e006a780 2024-04-25T01:43:10.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.779+0000 7fe9eb451740 1 Processor -- start 2024-04-25T01:43:10.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.779+0000 7fe9eb451740 1 -- start start 2024-04-25T01:43:10.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.779+0000 7fe9eb451740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcc4360 0x5556afcc4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.779+0000 7fe9eb451740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4d10 0x5556afcd34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.779+0000 7fe9eb451740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcd3a30 0x5556afcd5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.779+0000 7fe9eb451740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5556afc296b0 con 0x5556afcc4360 2024-04-25T01:43:10.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.779+0000 7fe9eb451740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5556afbd1880 con 0x5556afcc4d10 2024-04-25T01:43:10.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.779+0000 7fe9eb451740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5556afc25670 con 0x5556afcd3a30 2024-04-25T01:43:10.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.779+0000 7fe9d77fe640 1 --2- 172.21.15.28:0/985135872 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe9cc0a2d00 0x7fe9cc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.779+0000 7fe9e64de640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4d10 0x5556afcd34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.779+0000 7fe9e6cdf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcc4360 0x5556afcc4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.779+0000 7fe9e64de640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4d10 0x5556afcd34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.28:55290/0 (socket says 172.21.15.28:55290) 2024-04-25T01:43:10.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.779+0000 7fe9e64de640 1 -- 172.21.15.28:0/2924104401 learned_addr learned my addr 172.21.15.28:0/2924104401 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:10.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.780+0000 7fe9e5cdd640 1 --2- 172.21.15.28:0/2924104401 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcd3a30 0x5556afcd5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.780+0000 7fe9e6cdf640 1 --2- 172.21.15.28:0/985135872 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe9cc0a2d00 0x7fe9cc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.780+0000 7fe9d77fe640 1 -- 172.21.15.28:0/985135872 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7fe9d00cf490 con 0x7fe9e006a780 2024-04-25T01:43:10.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.780+0000 7fe9e64de640 1 -- 172.21.15.28:0/2924104401 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcd3a30 msgr2=0x5556afcd5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:10.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.780+0000 7fe9e64de640 1 --2- 172.21.15.28:0/2924104401 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcd3a30 0x5556afcd5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.780+0000 7fe9e64de640 1 -- 172.21.15.28:0/2924104401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcc4360 msgr2=0x5556afcc4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:10.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.780+0000 7fe9e64de640 1 --2- 172.21.15.28:0/2924104401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcc4360 0x5556afcc4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.780+0000 7fe9e64de640 1 -- 172.21.15.28:0/2924104401 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556afa077d0 con 0x5556afcc4d10 2024-04-25T01:43:10.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.780+0000 7fe9e5cdd640 1 --2- 172.21.15.28:0/2924104401 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcd3a30 0x5556afcd5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:10.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.780+0000 7fe9e6cdf640 1 --2- 172.21.15.28:0/2924104401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcc4360 0x5556afcc4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:43:10.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.780+0000 7fe9e64de640 1 --2- 172.21.15.28:0/2924104401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4d10 0x5556afcd34f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009930 tx=0x7fe9dc005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17460e5613968062 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:10.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.780+0000 7fe9e6cdf640 1 --2- 172.21.15.28:0/985135872 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe9cc0a2d00 0x7fe9cc0a51c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8005c40 tx=0x7fe9d8009280 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:10.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.780+0000 7fe9d4ff9640 1 -- 172.21.15.28:0/2924104401 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc040070 con 0x5556afcc4d10 2024-04-25T01:43:10.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.780+0000 7fe9d4ff9640 1 -- 172.21.15.28:0/2924104401 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9dc0319f0 con 0x5556afcc4d10 2024-04-25T01:43:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.780+0000 7fe9d4ff9640 1 -- 172.21.15.28:0/2924104401 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc035040 con 0x5556afcc4d10 2024-04-25T01:43:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.781+0000 7fe9eb451740 1 -- 172.21.15.28:0/2924104401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4d10 msgr2=0x5556afcd34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.781+0000 7fe9eb451740 1 --2- 172.21.15.28:0/2924104401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4d10 0x5556afcd34f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009930 tx=0x7fe9dc005c00 comp rx=0 tx=0).stop 2024-04-25T01:43:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.781+0000 7fe9eb451740 1 -- 172.21.15.28:0/2924104401 shutdown_connections 2024-04-25T01:43:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.781+0000 7fe9eb451740 1 --2- 172.21.15.28:0/2924104401 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcd3a30 0x5556afcd5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.781+0000 7fe9eb451740 1 --2- 172.21.15.28:0/2924104401 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4d10 0x5556afcd34f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.781+0000 7fe9eb451740 1 --2- 172.21.15.28:0/2924104401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcc4360 0x5556afcc4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.781+0000 7fe9eb451740 1 -- 172.21.15.28:0/2924104401 >> 172.21.15.28:0/2924104401 conn(0x5556afcc0950 msgr2=0x5556afcc0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.781+0000 7fe9eb451740 1 -- 172.21.15.28:0/2924104401 shutdown_connections 2024-04-25T01:43:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.781+0000 7fe9eb451740 1 -- 172.21.15.28:0/2924104401 wait complete. 2024-04-25T01:43:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9eb451740 1 -- start start 2024-04-25T01:43:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9eb451740 1 Processor -- start 2024-04-25T01:43:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9eb451740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4360 0x5556afcd34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9eb451740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcc4d10 0x5556afcd0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9eb451740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcd3a30 0x5556afcd0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9eb451740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5556afcd6790 con 0x5556afcc4d10 2024-04-25T01:43:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9eb451740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5556afbe4f50 con 0x5556afcc4360 2024-04-25T01:43:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9eb451740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5556afcd6910 con 0x5556afcd3a30 2024-04-25T01:43:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9eb451740 10 client.0 _ll_register_callbacks cb 0x5556afc20000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:43:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9e5cdd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcc4d10 0x5556afcd0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9e5cdd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcc4d10 0x5556afcd0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56534/0 (socket says 172.21.15.28:56534) 2024-04-25T01:43:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9e5cdd640 1 -- 172.21.15.28:0/1584544640 learned_addr learned my addr 172.21.15.28:0/1584544640 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9e64de640 1 --2- 172.21.15.28:0/1584544640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4360 0x5556afcd34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9e6cdf640 1 --2- 172.21.15.28:0/1584544640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcd3a30 0x5556afcd0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9e5cdd640 1 -- 172.21.15.28:0/1584544640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcd3a30 msgr2=0x5556afcd0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9e5cdd640 1 --2- 172.21.15.28:0/1584544640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcd3a30 0x5556afcd0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9e5cdd640 1 -- 172.21.15.28:0/1584544640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4360 msgr2=0x5556afcd34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9e5cdd640 1 --2- 172.21.15.28:0/1584544640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4360 0x5556afcd34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9e5cdd640 1 -- 172.21.15.28:0/1584544640 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556af93d3f0 con 0x5556afcc4d10 2024-04-25T01:43:10.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9e6cdf640 1 --2- 172.21.15.28:0/1584544640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcd3a30 0x5556afcd0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:43:10.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9e5cdd640 1 --2- 172.21.15.28:0/1584544640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcc4d10 0x5556afcd0850 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0097f90 tx=0x7fe9d0010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2a326e780157007 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:10.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:43:10.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9c9ffb640 1 -- 172.21.15.28:0/1584544640 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d0020040 con 0x5556afcc4d10 2024-04-25T01:43:10.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9eb451740 1 -- 172.21.15.28:0/1584544640 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9dc0319f0 con 0x5556afcc4d10 2024-04-25T01:43:10.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9c9ffb640 1 -- 172.21.15.28:0/1584544640 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7fe9d0097c30 con 0x5556afcc4d10 2024-04-25T01:43:10.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9c9ffb640 1 -- 172.21.15.28:0/1584544640 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe9d009bcc0 con 0x5556afcc4d10 2024-04-25T01:43:10.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9eb451740 10 client.19293 fetch_fsmap learned FSMap version 28 2024-04-25T01:43:10.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9eb451740 1 -- 172.21.15.28:0/1584544640 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5556af8fd0f0 con 0x5556afcc4d10 2024-04-25T01:43:10.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9c9ffb640 1 -- 172.21.15.28:0/1584544640 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9dc0319f0 con 0x5556afcc4d10 2024-04-25T01:43:10.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9c9ffb640 1 -- 172.21.15.28:0/1584544640 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe9d00520d0 con 0x5556afcc4d10 2024-04-25T01:43:10.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9eb451740 10 client.19293 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:43:10.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9eb451740 10 client.19293 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:43:10.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9eb451740 20 client.19293 populate_metadata read hostname 'smithi028' 2024-04-25T01:43:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9eb451740 1 --2- 172.21.15.28:0/1584544640 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5556afce3bd0 0x5556afce5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9eb451740 4 client.19293 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:43:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.783+0000 7fe9eb451740 1 -- 172.21.15.28:0/1584544640 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556afc36bf0 con 0x5556afce3bd0 2024-04-25T01:43:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.785+0000 7fe9e64de640 1 --2- 172.21.15.28:0/1584544640 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5556afce3bd0 0x5556afce5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.785+0000 7fe9e64de640 1 --2- 172.21.15.28:0/1584544640 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5556afce3bd0 0x5556afce5fc0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8106ebffd7003ad server_cookie=263a72eb75b46516 in_seq=0 out_seq=0 2024-04-25T01:43:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.785+0000 7fe9c9ffb640 10 client.19293 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:43:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.786+0000 7fe9c9ffb640 1 -- 172.21.15.28:0/1584544640 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5556afc36bf0 con 0x5556afce3bd0 2024-04-25T01:43:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.786+0000 7fe9c9ffb640 10 client.19293 handle_command_reply: tid=0 2024-04-25T01:43:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.786+0000 7fe9eb451740 1 client.19293 shutdown 2024-04-25T01:43:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.786+0000 7fe9eb451740 1 -- 172.21.15.28:0/1584544640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcc4d10 msgr2=0x5556afcd0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.786+0000 7fe9eb451740 1 --2- 172.21.15.28:0/1584544640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcc4d10 0x5556afcd0850 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0097f90 tx=0x7fe9d0010d70 comp rx=0 tx=0).stop 2024-04-25T01:43:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.786+0000 7fe9eb451740 1 -- 172.21.15.28:0/1584544640 shutdown_connections 2024-04-25T01:43:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.786+0000 7fe9eb451740 1 --2- 172.21.15.28:0/1584544640 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5556afce3bd0 0x5556afce5fc0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.786+0000 7fe9eb451740 1 --2- 172.21.15.28:0/1584544640 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcd3a30 0x5556afcd0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.786+0000 7fe9c9ffb640 0 client.19293 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:43:10.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.786+0000 7fe9eb451740 1 --2- 172.21.15.28:0/1584544640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcc4d10 0x5556afcd0850 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.786+0000 7fe9eb451740 1 --2- 172.21.15.28:0/1584544640 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4360 0x5556afcd34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.786+0000 7fe9eb451740 1 -- 172.21.15.28:0/1584544640 >> 172.21.15.28:0/1584544640 conn(0x5556afcc0950 msgr2=0x5556afcc67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:10.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.786+0000 7fe9eb451740 1 -- 172.21.15.28:0/1584544640 shutdown_connections 2024-04-25T01:43:10.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.786+0000 7fe9eb451740 1 -- 172.21.15.28:0/1584544640 wait complete. 2024-04-25T01:43:10.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.786+0000 7fe9eb451740 20 client.19293 trim_cache size 0 max 16384 2024-04-25T01:43:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9eb451740 1 Processor -- start 2024-04-25T01:43:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9eb451740 1 -- start start 2024-04-25T01:43:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9eb451740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4360 0x5556afccf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9eb451740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcc4d10 0x5556afcd2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9eb451740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcd3a30 0x5556afcd34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9eb451740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01aca50 con 0x5556afcd3a30 2024-04-25T01:43:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9eb451740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01ae590 con 0x5556afcc4360 2024-04-25T01:43:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9eb451740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01ad120 con 0x5556afcc4d10 2024-04-25T01:43:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9e6cdf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcd3a30 0x5556afcd34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9e6cdf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcd3a30 0x5556afcd34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56550/0 (socket says 172.21.15.28:56550) 2024-04-25T01:43:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9e6cdf640 1 -- 172.21.15.28:0/2089065583 learned_addr learned my addr 172.21.15.28:0/2089065583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9e64de640 1 --2- 172.21.15.28:0/2089065583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4360 0x5556afccf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9e6cdf640 1 -- 172.21.15.28:0/2089065583 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcc4d10 msgr2=0x5556afcd2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9e5cdd640 1 --2- 172.21.15.28:0/2089065583 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcc4d10 0x5556afcd2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9e6cdf640 1 --2- 172.21.15.28:0/2089065583 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcc4d10 0x5556afcd2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9e6cdf640 1 -- 172.21.15.28:0/2089065583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4360 msgr2=0x5556afccf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9e6cdf640 1 --2- 172.21.15.28:0/2089065583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4360 0x5556afccf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9e6cdf640 1 -- 172.21.15.28:0/2089065583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556afcc5a90 con 0x5556afcd3a30 2024-04-25T01:43:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9e64de640 1 --2- 172.21.15.28:0/2089065583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4360 0x5556afccf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:43:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.803+0000 7fe9e5cdd640 1 --2- 172.21.15.28:0/2089065583 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcc4d10 0x5556afcd2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:43:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.805+0000 7fe9e6cdf640 1 --2- 172.21.15.28:0/2089065583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcd3a30 0x5556afcd34c0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8012500 tx=0x7fe9d8009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd8b911f35b1a03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.805+0000 7fe9cbfff640 1 -- 172.21.15.28:0/2089065583 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8025070 con 0x5556afcd3a30 2024-04-25T01:43:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.805+0000 7fe9cbfff640 1 -- 172.21.15.28:0/2089065583 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9d8012d00 con 0x5556afcd3a30 2024-04-25T01:43:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.805+0000 7fe9cbfff640 1 -- 172.21.15.28:0/2089065583 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8021b50 con 0x5556afcd3a30 2024-04-25T01:43:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.805+0000 7fe9eb451740 1 -- 172.21.15.28:0/2089065583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcd3a30 msgr2=0x5556afcd34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:10.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.805+0000 7fe9eb451740 1 --2- 172.21.15.28:0/2089065583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcd3a30 0x5556afcd34c0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8012500 tx=0x7fe9d8009700 comp rx=0 tx=0).stop 2024-04-25T01:43:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.805+0000 7fe9eb451740 1 -- 172.21.15.28:0/2089065583 shutdown_connections 2024-04-25T01:43:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.805+0000 7fe9eb451740 1 --2- 172.21.15.28:0/2089065583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcd3a30 0x5556afcd34c0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.805+0000 7fe9eb451740 1 --2- 172.21.15.28:0/2089065583 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcc4d10 0x5556afcd2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.805+0000 7fe9eb451740 1 --2- 172.21.15.28:0/2089065583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4360 0x5556afccf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.805+0000 7fe9eb451740 1 -- 172.21.15.28:0/2089065583 >> 172.21.15.28:0/2089065583 conn(0x5556afcc0950 msgr2=0x5556afcc5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.805+0000 7fe9eb451740 1 -- 172.21.15.28:0/2089065583 shutdown_connections 2024-04-25T01:43:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.805+0000 7fe9eb451740 1 -- 172.21.15.28:0/2089065583 wait complete. 2024-04-25T01:43:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9eb451740 1 -- start start 2024-04-25T01:43:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9eb451740 1 Processor -- start 2024-04-25T01:43:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9eb451740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4360 0x5556afccd790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9eb451740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcc4d10 0x5556afccdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9eb451740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcd3a30 0x5556afcce210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9eb451740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01adec0 con 0x5556afcd3a30 2024-04-25T01:43:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9eb451740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01ad7f0 con 0x5556afcc4360 2024-04-25T01:43:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9eb451740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e01aaa50 con 0x5556afcc4d10 2024-04-25T01:43:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9eb451740 10 client.0 _ll_register_callbacks cb 0x7fe9d009b390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:43:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9e6cdf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcd3a30 0x5556afcce210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9e6cdf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcd3a30 0x5556afcce210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56556/0 (socket says 172.21.15.28:56556) 2024-04-25T01:43:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9e6cdf640 1 -- 172.21.15.28:0/897775477 learned_addr learned my addr 172.21.15.28:0/897775477 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9e5cdd640 1 --2- 172.21.15.28:0/897775477 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcc4d10 0x5556afccdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9e64de640 1 --2- 172.21.15.28:0/897775477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4360 0x5556afccd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9e6cdf640 1 -- 172.21.15.28:0/897775477 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcc4d10 msgr2=0x5556afccdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9e6cdf640 1 --2- 172.21.15.28:0/897775477 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcc4d10 0x5556afccdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9e6cdf640 1 -- 172.21.15.28:0/897775477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4360 msgr2=0x5556afccd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9e6cdf640 1 --2- 172.21.15.28:0/897775477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4360 0x5556afccd790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9e6cdf640 1 -- 172.21.15.28:0/897775477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556afcda2a0 con 0x5556afcd3a30 2024-04-25T01:43:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9e64de640 1 --2- 172.21.15.28:0/897775477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4360 0x5556afccd790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9e6cdf640 1 --2- 172.21.15.28:0/897775477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcd3a30 0x5556afcce210 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8006010 tx=0x7fe9d8004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44c74b165d094381 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:43:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9c9ffb640 1 -- 172.21.15.28:0/897775477 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8025030 con 0x5556afcd3a30 2024-04-25T01:43:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9c9ffb640 1 -- 172.21.15.28:0/897775477 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7fe9d8010be0 con 0x5556afcd3a30 2024-04-25T01:43:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9eb451740 1 -- 172.21.15.28:0/897775477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9d8012d00 con 0x5556afcd3a30 2024-04-25T01:43:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9c9ffb640 1 -- 172.21.15.28:0/897775477 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe9d801b040 con 0x5556afcd3a30 2024-04-25T01:43:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9eb451740 10 client.19299 fetch_fsmap learned FSMap version 28 2024-04-25T01:43:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9eb451740 1 -- 172.21.15.28:0/897775477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5556afd4a510 con 0x5556afcd3a30 2024-04-25T01:43:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9c9ffb640 1 -- 172.21.15.28:0/897775477 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9d8012d00 con 0x5556afcd3a30 2024-04-25T01:43:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9c9ffb640 1 -- 172.21.15.28:0/897775477 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe9d801e040 con 0x5556afcd3a30 2024-04-25T01:43:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9eb451740 10 client.19299 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:43:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9eb451740 10 client.19299 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:43:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9eb451740 20 client.19299 populate_metadata read hostname 'smithi028' 2024-04-25T01:43:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9eb451740 1 --2- 172.21.15.28:0/897775477 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5556afd4ab80 0x5556afd4cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9eb451740 4 client.19299 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:43:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.806+0000 7fe9eb451740 1 -- 172.21.15.28:0/897775477 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5556afc30f40 con 0x5556afd4ab80 2024-04-25T01:43:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.808+0000 7fe9e64de640 1 --2- 172.21.15.28:0/897775477 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5556afd4ab80 0x5556afd4cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.808+0000 7fe9e64de640 1 --2- 172.21.15.28:0/897775477 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5556afd4ab80 0x5556afd4cf70 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ee26b5b6a638e57 server_cookie=a0b481d4d29ff6f5 in_seq=0 out_seq=0 2024-04-25T01:43:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.808+0000 7fe9c9ffb640 10 client.19299 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:43:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.808+0000 7fe9c9ffb640 1 -- 172.21.15.28:0/897775477 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5556afc30f40 con 0x5556afd4ab80 2024-04-25T01:43:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.808+0000 7fe9c9ffb640 10 client.19299 handle_command_reply: tid=0 2024-04-25T01:43:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.808+0000 7fe9eb451740 1 client.19299 shutdown 2024-04-25T01:43:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.808+0000 7fe9eb451740 1 -- 172.21.15.28:0/897775477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcd3a30 msgr2=0x5556afcce210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.808+0000 7fe9eb451740 1 --2- 172.21.15.28:0/897775477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcd3a30 0x5556afcce210 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8006010 tx=0x7fe9d8004cf0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.808+0000 7fe9eb451740 1 -- 172.21.15.28:0/897775477 shutdown_connections 2024-04-25T01:43:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.808+0000 7fe9eb451740 1 --2- 172.21.15.28:0/897775477 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x5556afd4ab80 0x5556afd4cf70 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.808+0000 7fe9eb451740 1 --2- 172.21.15.28:0/897775477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556afcd3a30 0x5556afcce210 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.808+0000 7fe9eb451740 1 --2- 172.21.15.28:0/897775477 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5556afcc4d10 0x5556afccdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.808+0000 7fe9c9ffb640 0 client.19299 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:43:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.808+0000 7fe9eb451740 1 --2- 172.21.15.28:0/897775477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5556afcc4360 0x5556afccd790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.808+0000 7fe9eb451740 1 -- 172.21.15.28:0/897775477 >> 172.21.15.28:0/897775477 conn(0x5556afcc0950 msgr2=0x5556afcc6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.808+0000 7fe9eb451740 1 -- 172.21.15.28:0/897775477 shutdown_connections 2024-04-25T01:43:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.808+0000 7fe9eb451740 1 -- 172.21.15.28:0/897775477 wait complete. 2024-04-25T01:43:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.809+0000 7fe9eb451740 20 client.19299 trim_cache size 0 max 16384 2024-04-25T01:43:10.804 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:43:10.804 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-25T01:43:10.804 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-25T01:43:10.804 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:43:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.809+0000 7fe9e8742640 1 -- 172.21.15.28:0/985135872 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe9cc0a2d00 msgr2=0x7fe9cc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.809+0000 7fe9e8742640 1 --2- 172.21.15.28:0/985135872 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe9cc0a2d00 0x7fe9cc0a51c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8005c40 tx=0x7fe9d8009280 comp rx=0 tx=0).stop 2024-04-25T01:43:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.809+0000 7fe9e8742640 1 -- 172.21.15.28:0/985135872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e006a780 msgr2=0x7fe9e01985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.809+0000 7fe9e8742640 1 --2- 172.21.15.28:0/985135872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e006a780 0x7fe9e01985e0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fe9d000d950 tx=0x7fe9d000de10 comp rx=0 tx=0).stop 2024-04-25T01:43:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.809+0000 7fe9e8742640 1 -- 172.21.15.28:0/985135872 shutdown_connections 2024-04-25T01:43:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.809+0000 7fe9e8742640 1 --2- 172.21.15.28:0/985135872 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe9cc0a2d00 0x7fe9cc0a51c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.809+0000 7fe9e8742640 1 --2- 172.21.15.28:0/985135872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe9e0107730 0x7fe9e019c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.809+0000 7fe9e8742640 1 --2- 172.21.15.28:0/985135872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e006a780 0x7fe9e01985e0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.809+0000 7fe9e8742640 1 --2- 172.21.15.28:0/985135872 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe9e0069dc0 0x7fe9e01980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.809+0000 7fe9e8742640 1 -- 172.21.15.28:0/985135872 >> 172.21.15.28:0/985135872 conn(0x7fe9e00776a0 msgr2=0x7fe9e00fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.809+0000 7fe9e8742640 1 -- 172.21.15.28:0/985135872 shutdown_connections 2024-04-25T01:43:10.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.809+0000 7fe9e8742640 1 -- 172.21.15.28:0/985135872 wait complete. 2024-04-25T01:43:10.809 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:43:10.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:43:10.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.922+0000 7ff511fce640 1 -- 172.21.15.28:0/3716378933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c103340 msgr2=0x7ff50c10b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:10.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.922+0000 7ff511fce640 1 --2- 172.21.15.28:0/3716378933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c103340 0x7ff50c10b820 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7ff500009900 tx=0x7ff50002a480 comp rx=0 tx=0).stop 2024-04-25T01:43:10.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.922+0000 7ff511fce640 1 -- 172.21.15.28:0/3716378933 shutdown_connections 2024-04-25T01:43:10.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.922+0000 7ff511fce640 1 --2- 172.21.15.28:0/3716378933 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff50c10bd60 0x7ff50c10e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.922+0000 7ff511fce640 1 --2- 172.21.15.28:0/3716378933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c103340 0x7ff50c10b820 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.922+0000 7ff511fce640 1 --2- 172.21.15.28:0/3716378933 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff50c102a20 0x7ff50c102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.922+0000 7ff511fce640 1 -- 172.21.15.28:0/3716378933 >> 172.21.15.28:0/3716378933 conn(0x7ff50c0777a0 msgr2=0x7ff50c077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.922+0000 7ff511fce640 1 -- 172.21.15.28:0/3716378933 shutdown_connections 2024-04-25T01:43:10.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.922+0000 7ff511fce640 1 -- 172.21.15.28:0/3716378933 wait complete. 2024-04-25T01:43:10.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.923+0000 7ff511fce640 1 Processor -- start 2024-04-25T01:43:10.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.923+0000 7ff511fce640 1 -- start start 2024-04-25T01:43:10.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.923+0000 7ff511fce640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff50c102a20 0x7ff50c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.923+0000 7ff511fce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c103340 0x7ff50c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.923+0000 7ff511fce640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff50c10bd60 0x7ff50c19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.923+0000 7ff511fce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff50c111b30 con 0x7ff50c103340 2024-04-25T01:43:10.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.923+0000 7ff511fce640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff50c1119b0 con 0x7ff50c102a20 2024-04-25T01:43:10.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.923+0000 7ff511fce640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff50c111cb0 con 0x7ff50c10bd60 2024-04-25T01:43:10.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.923+0000 7ff50affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c103340 0x7ff50c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.923+0000 7ff50affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c103340 0x7ff50c198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56566/0 (socket says 172.21.15.28:56566) 2024-04-25T01:43:10.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.923+0000 7ff50affd640 1 -- 172.21.15.28:0/2197855521 learned_addr learned my addr 172.21.15.28:0/2197855521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.923+0000 7ff50b7fe640 1 --2- 172.21.15.28:0/2197855521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff50c102a20 0x7ff50c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.923+0000 7ff50bfff640 1 --2- 172.21.15.28:0/2197855521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff50c10bd60 0x7ff50c19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.924+0000 7ff50affd640 1 -- 172.21.15.28:0/2197855521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff50c10bd60 msgr2=0x7ff50c19c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.924+0000 7ff50affd640 1 --2- 172.21.15.28:0/2197855521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff50c10bd60 0x7ff50c19c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.924+0000 7ff50affd640 1 -- 172.21.15.28:0/2197855521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff50c102a20 msgr2=0x7ff50c198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.924+0000 7ff50affd640 1 --2- 172.21.15.28:0/2197855521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff50c102a20 0x7ff50c198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.924+0000 7ff50affd640 1 -- 172.21.15.28:0/2197855521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff50c19d020 con 0x7ff50c103340 2024-04-25T01:43:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.924+0000 7ff50affd640 1 --2- 172.21.15.28:0/2197855521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c103340 0x7ff50c198570 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7ff500006010 tx=0x7ff5000042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.924+0000 7ff508ff9640 1 -- 172.21.15.28:0/2197855521 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50003f030 con 0x7ff50c103340 2024-04-25T01:43:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.924+0000 7ff511fce640 1 -- 172.21.15.28:0/2197855521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff50c19d2b0 con 0x7ff50c103340 2024-04-25T01:43:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.924+0000 7ff508ff9640 1 -- 172.21.15.28:0/2197855521 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff500003ce0 con 0x7ff50c103340 2024-04-25T01:43:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.924+0000 7ff508ff9640 1 -- 172.21.15.28:0/2197855521 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff500035070 con 0x7ff50c103340 2024-04-25T01:43:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.924+0000 7ff511fce640 1 -- 172.21.15.28:0/2197855521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff50c1a4ba0 con 0x7ff50c103340 2024-04-25T01:43:10.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.925+0000 7ff511fce640 1 -- 172.21.15.28:0/2197855521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff50c0689b0 con 0x7ff50c103340 2024-04-25T01:43:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.926+0000 7ff508ff9640 1 -- 172.21.15.28:0/2197855521 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff500042030 con 0x7ff50c103340 2024-04-25T01:43:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.927+0000 7ff508ff9640 1 --2- 172.21.15.28:0/2197855521 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4f40a2d00 0x7ff4f40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.927+0000 7ff50b7fe640 1 --2- 172.21.15.28:0/2197855521 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4f40a2d00 0x7ff4f40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.927+0000 7ff508ff9640 1 -- 172.21.15.28:0/2197855521 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7ff5000f0050 con 0x7ff50c103340 2024-04-25T01:43:10.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.927+0000 7ff50b7fe640 1 --2- 172.21.15.28:0/2197855521 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4f40a2d00 0x7ff4f40a51c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc005d00 tx=0x7ff4fc00b410 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:10.930+0000 7ff508ff9640 1 -- 172.21.15.28:0/2197855521 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff5000bc290 con 0x7ff50c103340 2024-04-25T01:43:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.153+0000 7ff511fce640 1 -- 172.21.15.28:0/2197855521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff50c106f10 con 0x7ff50c103340 2024-04-25T01:43:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.153+0000 7ff508ff9640 1 -- 172.21.15.28:0/2197855521 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7ff5000c0590 con 0x7ff50c103340 2024-04-25T01:43:11.143 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:43:11.143 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:43:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:43:11.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.156+0000 7ff511fce640 1 -- 172.21.15.28:0/2197855521 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4f40a2d00 msgr2=0x7ff4f40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.156+0000 7ff511fce640 1 --2- 172.21.15.28:0/2197855521 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4f40a2d00 0x7ff4f40a51c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc005d00 tx=0x7ff4fc00b410 comp rx=0 tx=0).stop 2024-04-25T01:43:11.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.156+0000 7ff511fce640 1 -- 172.21.15.28:0/2197855521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c103340 msgr2=0x7ff50c198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.156+0000 7ff511fce640 1 --2- 172.21.15.28:0/2197855521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c103340 0x7ff50c198570 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7ff500006010 tx=0x7ff5000042f0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.156+0000 7ff511fce640 1 -- 172.21.15.28:0/2197855521 shutdown_connections 2024-04-25T01:43:11.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.156+0000 7ff511fce640 1 --2- 172.21.15.28:0/2197855521 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4f40a2d00 0x7ff4f40a51c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.156+0000 7ff511fce640 1 --2- 172.21.15.28:0/2197855521 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff50c10bd60 0x7ff50c19c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.156+0000 7ff511fce640 1 --2- 172.21.15.28:0/2197855521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c103340 0x7ff50c198570 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.156+0000 7ff511fce640 1 --2- 172.21.15.28:0/2197855521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff50c102a20 0x7ff50c198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.156+0000 7ff511fce640 1 -- 172.21.15.28:0/2197855521 >> 172.21.15.28:0/2197855521 conn(0x7ff50c0777a0 msgr2=0x7ff50c0fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.156+0000 7ff511fce640 1 -- 172.21.15.28:0/2197855521 shutdown_connections 2024-04-25T01:43:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.156+0000 7ff511fce640 1 -- 172.21.15.28:0/2197855521 wait complete. 2024-04-25T01:43:11.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T01:43:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.270+0000 7f7b922da640 1 -- 172.21.15.28:0/181960195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c069dc0 msgr2=0x7f7b8c06a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.270+0000 7f7b922da640 1 --2- 172.21.15.28:0/181960195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c069dc0 0x7f7b8c06a240 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c009930 tx=0x7f7b7c02a420 comp rx=0 tx=0).stop 2024-04-25T01:43:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.270+0000 7f7b922da640 1 -- 172.21.15.28:0/181960195 shutdown_connections 2024-04-25T01:43:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.270+0000 7f7b922da640 1 --2- 172.21.15.28:0/181960195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b8c06a780 0x7f7b8c10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.270+0000 7f7b922da640 1 --2- 172.21.15.28:0/181960195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c069dc0 0x7f7b8c06a240 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.270+0000 7f7b922da640 1 --2- 172.21.15.28:0/181960195 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b8c107730 0x7f7b8c107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.270+0000 7f7b922da640 1 -- 172.21.15.28:0/181960195 >> 172.21.15.28:0/181960195 conn(0x7f7b8c0776a0 msgr2=0x7f7b8c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.270+0000 7f7b922da640 1 -- 172.21.15.28:0/181960195 shutdown_connections 2024-04-25T01:43:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.270+0000 7f7b922da640 1 -- 172.21.15.28:0/181960195 wait complete. 2024-04-25T01:43:11.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.271+0000 7f7b922da640 1 Processor -- start 2024-04-25T01:43:11.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.271+0000 7f7b922da640 1 -- start start 2024-04-25T01:43:11.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.271+0000 7f7b922da640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b8c069dc0 0x7f7b8c100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.271+0000 7f7b922da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c06a780 0x7f7b8c100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.271+0000 7f7b922da640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b8c107730 0x7f7b8c1010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.271+0000 7f7b922da640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c111b40 con 0x7f7b8c06a780 2024-04-25T01:43:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.271+0000 7f7b922da640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c1119c0 con 0x7f7b8c069dc0 2024-04-25T01:43:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.271+0000 7f7b922da640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c111cc0 con 0x7f7b8c107730 2024-04-25T01:43:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.272+0000 7f7b8b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c06a780 0x7f7b8c100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.272+0000 7f7b8b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c06a780 0x7f7b8c100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56582/0 (socket says 172.21.15.28:56582) 2024-04-25T01:43:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.272+0000 7f7b8b7fe640 1 -- 172.21.15.28:0/4180671538 learned_addr learned my addr 172.21.15.28:0/4180671538 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.272+0000 7f7b90877640 1 --2- 172.21.15.28:0/4180671538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b8c107730 0x7f7b8c1010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.272+0000 7f7b8bfff640 1 --2- 172.21.15.28:0/4180671538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b8c069dc0 0x7f7b8c100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.272+0000 7f7b8b7fe640 1 -- 172.21.15.28:0/4180671538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b8c107730 msgr2=0x7f7b8c1010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.272+0000 7f7b8b7fe640 1 --2- 172.21.15.28:0/4180671538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b8c107730 0x7f7b8c1010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.272+0000 7f7b8b7fe640 1 -- 172.21.15.28:0/4180671538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b8c069dc0 msgr2=0x7f7b8c100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.272+0000 7f7b8b7fe640 1 --2- 172.21.15.28:0/4180671538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b8c069dc0 0x7f7b8c100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.272+0000 7f7b8b7fe640 1 -- 172.21.15.28:0/4180671538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b8c1a6580 con 0x7f7b8c06a780 2024-04-25T01:43:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.272+0000 7f7b8b7fe640 1 --2- 172.21.15.28:0/4180671538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c06a780 0x7f7b8c100bb0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c009230 tx=0x7f7b7c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.272+0000 7f7b897fa640 1 -- 172.21.15.28:0/4180671538 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c03f030 con 0x7f7b8c06a780 2024-04-25T01:43:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.272+0000 7f7b897fa640 1 -- 172.21.15.28:0/4180671538 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b7c003ad0 con 0x7f7b8c06a780 2024-04-25T01:43:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.272+0000 7f7b922da640 1 -- 172.21.15.28:0/4180671538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b8c1a6810 con 0x7f7b8c06a780 2024-04-25T01:43:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.272+0000 7f7b897fa640 1 -- 172.21.15.28:0/4180671538 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c035070 con 0x7f7b8c06a780 2024-04-25T01:43:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.272+0000 7f7b922da640 1 -- 172.21.15.28:0/4180671538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b8c1a6d00 con 0x7f7b8c06a780 2024-04-25T01:43:11.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b897fa640 1 -- 172.21.15.28:0/4180671538 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7b7c042030 con 0x7f7b8c06a780 2024-04-25T01:43:11.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b94fee740 1 Processor -- start 2024-04-25T01:43:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b94fee740 1 -- start start 2024-04-25T01:43:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b94fee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 0x55d97d4fb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b94fee740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d4fbea0 0x55d97d506030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b94fee740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d506570 0x55d97d508960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b94fee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d97d460670 con 0x55d97d4fb580 2024-04-25T01:43:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b94fee740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d97d408970 con 0x55d97d506570 2024-04-25T01:43:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b94fee740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55d97d45c630 con 0x55d97d4fbea0 2024-04-25T01:43:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b90877640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 0x55d97d4fb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b90877640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 0x55d97d4fb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56592/0 (socket says 172.21.15.28:56592) 2024-04-25T01:43:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b90877640 1 -- 172.21.15.28:0/3168335093 learned_addr learned my addr 172.21.15.28:0/3168335093 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b8b7fe640 1 --2- 172.21.15.28:0/3168335093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d506570 0x55d97d508960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b897fa640 1 --2- 172.21.15.28:0/4180671538 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7b780a2d00 0x7f7b780a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b8bfff640 1 --2- 172.21.15.28:0/3168335093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d4fbea0 0x55d97d506030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b90877640 1 -- 172.21.15.28:0/3168335093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d4fbea0 msgr2=0x55d97d506030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b90877640 1 --2- 172.21.15.28:0/3168335093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d4fbea0 0x55d97d506030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b90877640 1 -- 172.21.15.28:0/3168335093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d506570 msgr2=0x55d97d508960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b90877640 1 --2- 172.21.15.28:0/3168335093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d506570 0x55d97d508960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b90877640 1 -- 172.21.15.28:0/3168335093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d97d42c400 con 0x55d97d4fb580 2024-04-25T01:43:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.275+0000 7f7b8b7fe640 1 --2- 172.21.15.28:0/3168335093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d506570 0x55d97d508960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.276+0000 7f7b90877640 1 --2- 172.21.15.28:0/4180671538 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7b780a2d00 0x7f7b780a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.276+0000 7f7b897fa640 1 -- 172.21.15.28:0/4180671538 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f7b7c0efd30 con 0x7f7b8c06a780 2024-04-25T01:43:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.276+0000 7f7b90877640 1 --2- 172.21.15.28:0/3168335093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 0x55d97d4fb960 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f7b84003b10 tx=0x7f7b84009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a45b91405de8172 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.276+0000 7f7b827fc640 1 -- 172.21.15.28:0/3168335093 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b84024070 con 0x55d97d4fb580 2024-04-25T01:43:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.276+0000 7f7b827fc640 1 -- 172.21.15.28:0/3168335093 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b84010ce0 con 0x55d97d4fb580 2024-04-25T01:43:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.276+0000 7f7b827fc640 1 -- 172.21.15.28:0/3168335093 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8401ed40 con 0x55d97d4fb580 2024-04-25T01:43:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.276+0000 7f7b90877640 1 --2- 172.21.15.28:0/4180671538 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7b780a2d00 0x7f7b780a51c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7b84021000 tx=0x7f7b8401e000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.277+0000 7f7b94fee740 1 -- 172.21.15.28:0/3168335093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 msgr2=0x55d97d4fb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.277+0000 7f7b94fee740 1 --2- 172.21.15.28:0/3168335093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 0x55d97d4fb960 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f7b84003b10 tx=0x7f7b84009f90 comp rx=0 tx=0).stop 2024-04-25T01:43:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.277+0000 7f7b94fee740 1 -- 172.21.15.28:0/3168335093 shutdown_connections 2024-04-25T01:43:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.277+0000 7f7b94fee740 1 --2- 172.21.15.28:0/3168335093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d506570 0x55d97d508960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.277+0000 7f7b94fee740 1 --2- 172.21.15.28:0/3168335093 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d4fbea0 0x55d97d506030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.277+0000 7f7b94fee740 1 --2- 172.21.15.28:0/3168335093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 0x55d97d4fb960 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.277+0000 7f7b94fee740 1 -- 172.21.15.28:0/3168335093 >> 172.21.15.28:0/3168335093 conn(0x55d97d4f78a0 msgr2=0x55d97d4f7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.277+0000 7f7b94fee740 1 -- 172.21.15.28:0/3168335093 shutdown_connections 2024-04-25T01:43:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.277+0000 7f7b94fee740 1 -- 172.21.15.28:0/3168335093 wait complete. 2024-04-25T01:43:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.277+0000 7f7b94fee740 1 -- start start 2024-04-25T01:43:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.277+0000 7f7b94fee740 1 Processor -- start 2024-04-25T01:43:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.277+0000 7f7b94fee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 0x55d97d505f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.277+0000 7f7b94fee740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d4fbea0 0x55d97d503220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.277+0000 7f7b94fee740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d506570 0x55d97d503830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.277+0000 7f7b94fee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d97d509360 con 0x55d97d4fb580 2024-04-25T01:43:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.277+0000 7f7b94fee740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d97d41c050 con 0x55d97d4fbea0 2024-04-25T01:43:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.277+0000 7f7b94fee740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x55d97d5094e0 con 0x55d97d506570 2024-04-25T01:43:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.277+0000 7f7b94fee740 10 client.0 _ll_register_callbacks cb 0x55d97d173be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:43:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.278+0000 7f7b8bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 0x55d97d505f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.278+0000 7f7b8bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 0x55d97d505f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56606/0 (socket says 172.21.15.28:56606) 2024-04-25T01:43:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.278+0000 7f7b8bfff640 1 -- 172.21.15.28:0/3074123601 learned_addr learned my addr 172.21.15.28:0/3074123601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.278+0000 7f7b90877640 1 --2- 172.21.15.28:0/3074123601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d506570 0x55d97d503830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.278+0000 7f7b8b7fe640 1 --2- 172.21.15.28:0/3074123601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d4fbea0 0x55d97d503220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.278+0000 7f7b8bfff640 1 -- 172.21.15.28:0/3074123601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d506570 msgr2=0x55d97d503830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.278+0000 7f7b8bfff640 1 --2- 172.21.15.28:0/3074123601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d506570 0x55d97d503830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.278+0000 7f7b8bfff640 1 -- 172.21.15.28:0/3074123601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d4fbea0 msgr2=0x55d97d503220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.278+0000 7f7b8bfff640 1 --2- 172.21.15.28:0/3074123601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d4fbea0 0x55d97d503220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.278+0000 7f7b8bfff640 1 -- 172.21.15.28:0/3074123601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d97d3eddc0 con 0x55d97d4fb580 2024-04-25T01:43:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.278+0000 7f7b8bfff640 1 --2- 172.21.15.28:0/3074123601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 0x55d97d505f30 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f7b7400bfc0 tx=0x7f7b7400f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e332a5be97781fc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.278+0000 7f7b6b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:43:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.278+0000 7f7b6b7fe640 1 -- 172.21.15.28:0/3074123601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b74018020 con 0x55d97d4fb580 2024-04-25T01:43:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.278+0000 7f7b6b7fe640 1 -- 172.21.15.28:0/3074123601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f7b74014660 con 0x55d97d4fb580 2024-04-25T01:43:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.278+0000 7f7b94fee740 1 -- 172.21.15.28:0/3074123601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b84010ce0 con 0x55d97d4fb580 2024-04-25T01:43:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.278+0000 7f7b6b7fe640 1 -- 172.21.15.28:0/3074123601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7b74023050 con 0x55d97d4fb580 2024-04-25T01:43:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.278+0000 7f7b94fee740 10 client.19317 fetch_fsmap learned FSMap version 28 2024-04-25T01:43:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.278+0000 7f7b94fee740 1 -- 172.21.15.28:0/3074123601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d97d1743f0 con 0x55d97d4fb580 2024-04-25T01:43:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.279+0000 7f7b6b7fe640 1 -- 172.21.15.28:0/3074123601 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b84010ce0 con 0x55d97d4fb580 2024-04-25T01:43:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.279+0000 7f7b6b7fe640 1 -- 172.21.15.28:0/3074123601 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7b74013a70 con 0x55d97d4fb580 2024-04-25T01:43:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.279+0000 7f7b94fee740 10 client.19317 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:43:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.279+0000 7f7b94fee740 10 client.19317 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:43:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.279+0000 7f7b94fee740 20 client.19317 populate_metadata read hostname 'smithi028' 2024-04-25T01:43:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.279+0000 7f7b94fee740 1 --2- 172.21.15.28:0/3074123601 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55d97d5166d0 0x55d97d518b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.279+0000 7f7b94fee740 4 client.19317 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:43:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.279+0000 7f7b94fee740 1 -- 172.21.15.28:0/3074123601 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d97d46dbb0 con 0x55d97d5166d0 2024-04-25T01:43:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.279+0000 7f7b8b7fe640 1 --2- 172.21.15.28:0/3074123601 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55d97d5166d0 0x55d97d518b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.280+0000 7f7b8b7fe640 1 --2- 172.21.15.28:0/3074123601 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55d97d5166d0 0x55d97d518b00 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=36e1c985676511b server_cookie=3534aa83d47b9008 in_seq=0 out_seq=0 2024-04-25T01:43:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.280+0000 7f7b6b7fe640 10 client.19317 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:43:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.280+0000 7f7b6b7fe640 1 -- 172.21.15.28:0/3074123601 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d97d46dbb0 con 0x55d97d5166d0 2024-04-25T01:43:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.280+0000 7f7b6b7fe640 10 client.19317 handle_command_reply: tid=0 2024-04-25T01:43:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.281+0000 7f7b94fee740 1 client.19317 shutdown 2024-04-25T01:43:11.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.281+0000 7f7b94fee740 1 -- 172.21.15.28:0/3074123601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 msgr2=0x55d97d505f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.281+0000 7f7b94fee740 1 --2- 172.21.15.28:0/3074123601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 0x55d97d505f30 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f7b7400bfc0 tx=0x7f7b7400f5b0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.281+0000 7f7b94fee740 1 -- 172.21.15.28:0/3074123601 shutdown_connections 2024-04-25T01:43:11.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.281+0000 7f7b94fee740 1 --2- 172.21.15.28:0/3074123601 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55d97d5166d0 0x55d97d518b00 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.281+0000 7f7b94fee740 1 --2- 172.21.15.28:0/3074123601 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d506570 0x55d97d503830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.281+0000 7f7b94fee740 1 --2- 172.21.15.28:0/3074123601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d4fbea0 0x55d97d503220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.281+0000 7f7b6b7fe640 0 client.19317 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:43:11.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.281+0000 7f7b94fee740 1 --2- 172.21.15.28:0/3074123601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 0x55d97d505f30 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.281+0000 7f7b94fee740 1 -- 172.21.15.28:0/3074123601 >> 172.21.15.28:0/3074123601 conn(0x55d97d4f78a0 msgr2=0x55d97d4fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.281+0000 7f7b94fee740 1 -- 172.21.15.28:0/3074123601 shutdown_connections 2024-04-25T01:43:11.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.281+0000 7f7b94fee740 1 -- 172.21.15.28:0/3074123601 wait complete. 2024-04-25T01:43:11.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.281+0000 7f7b94fee740 20 client.19317 trim_cache size 0 max 16384 2024-04-25T01:43:11.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.294+0000 7f7b94fee740 1 Processor -- start 2024-04-25T01:43:11.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.294+0000 7f7b94fee740 1 -- start start 2024-04-25T01:43:11.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.294+0000 7f7b94fee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 0x55d97d505a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.294+0000 7f7b94fee740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d4fbea0 0x55d97d50e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.294+0000 7f7b94fee740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d506570 0x55d97d50ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.294+0000 7f7b94fee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c1b0e50 con 0x55d97d4fb580 2024-04-25T01:43:11.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.294+0000 7f7b94fee740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c1b2990 con 0x55d97d506570 2024-04-25T01:43:11.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.294+0000 7f7b94fee740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c1b1520 con 0x55d97d4fbea0 2024-04-25T01:43:11.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.295+0000 7f7b8bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 0x55d97d505a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.295+0000 7f7b8bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 0x55d97d505a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56618/0 (socket says 172.21.15.28:56618) 2024-04-25T01:43:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.295+0000 7f7b8bfff640 1 -- 172.21.15.28:0/4100788050 learned_addr learned my addr 172.21.15.28:0/4100788050 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.295+0000 7f7b8b7fe640 1 --2- 172.21.15.28:0/4100788050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d4fbea0 0x55d97d50e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.295+0000 7f7b8bfff640 1 -- 172.21.15.28:0/4100788050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d4fbea0 msgr2=0x55d97d50e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.295+0000 7f7b8bfff640 1 --2- 172.21.15.28:0/4100788050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d4fbea0 0x55d97d50e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.295+0000 7f7b8bfff640 1 -- 172.21.15.28:0/4100788050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d506570 msgr2=0x55d97d50ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.295+0000 7f7b90877640 1 --2- 172.21.15.28:0/4100788050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d506570 0x55d97d50ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.295+0000 7f7b8bfff640 1 --2- 172.21.15.28:0/4100788050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d506570 0x55d97d50ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.295+0000 7f7b8bfff640 1 -- 172.21.15.28:0/4100788050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d97d23e7d0 con 0x55d97d4fb580 2024-04-25T01:43:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.295+0000 7f7b8b7fe640 1 --2- 172.21.15.28:0/4100788050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d4fbea0 0x55d97d50e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:43:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.295+0000 7f7b8bfff640 1 --2- 172.21.15.28:0/4100788050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 0x55d97d505a60 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f7b74004810 tx=0x7f7b7400bb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6511a815401f04fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.295+0000 7f7b81ffb640 1 -- 172.21.15.28:0/4100788050 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7402b070 con 0x55d97d4fb580 2024-04-25T01:43:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.295+0000 7f7b81ffb640 1 -- 172.21.15.28:0/4100788050 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b7400fab0 con 0x55d97d4fb580 2024-04-25T01:43:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.295+0000 7f7b81ffb640 1 -- 172.21.15.28:0/4100788050 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b74018040 con 0x55d97d4fb580 2024-04-25T01:43:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.296+0000 7f7b94fee740 1 -- 172.21.15.28:0/4100788050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 msgr2=0x55d97d505a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.296+0000 7f7b94fee740 1 --2- 172.21.15.28:0/4100788050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 0x55d97d505a60 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f7b74004810 tx=0x7f7b7400bb60 comp rx=0 tx=0).stop 2024-04-25T01:43:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.296+0000 7f7b94fee740 1 -- 172.21.15.28:0/4100788050 shutdown_connections 2024-04-25T01:43:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.296+0000 7f7b94fee740 1 --2- 172.21.15.28:0/4100788050 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d506570 0x55d97d50ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.296+0000 7f7b94fee740 1 --2- 172.21.15.28:0/4100788050 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d4fbea0 0x55d97d50e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.296+0000 7f7b94fee740 1 --2- 172.21.15.28:0/4100788050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d4fb580 0x55d97d505a60 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.296+0000 7f7b94fee740 1 -- 172.21.15.28:0/4100788050 >> 172.21.15.28:0/4100788050 conn(0x55d97d4f78a0 msgr2=0x55d97d511210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.296+0000 7f7b94fee740 1 -- 172.21.15.28:0/4100788050 shutdown_connections 2024-04-25T01:43:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.296+0000 7f7b94fee740 1 -- 172.21.15.28:0/4100788050 wait complete. 2024-04-25T01:43:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.296+0000 7f7b94fee740 1 -- start start 2024-04-25T01:43:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.296+0000 7f7b94fee740 1 Processor -- start 2024-04-25T01:43:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b94fee740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d4fb580 0x55d97d50c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b94fee740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d4fbea0 0x55d97d50c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b94fee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d506570 0x55d97d50ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b94fee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c1b22c0 con 0x55d97d506570 2024-04-25T01:43:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b94fee740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c1b1bf0 con 0x55d97d4fb580 2024-04-25T01:43:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b94fee740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c1aee50 con 0x55d97d4fbea0 2024-04-25T01:43:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b94fee740 10 client.0 _ll_register_callbacks cb 0x7f7b7400b2e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:43:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b90877640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d506570 0x55d97d50ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b90877640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d506570 0x55d97d50ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56626/0 (socket says 172.21.15.28:56626) 2024-04-25T01:43:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b90877640 1 -- 172.21.15.28:0/3236206458 learned_addr learned my addr 172.21.15.28:0/3236206458 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b8bfff640 1 --2- 172.21.15.28:0/3236206458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d4fb580 0x55d97d50c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b8b7fe640 1 --2- 172.21.15.28:0/3236206458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d4fbea0 0x55d97d50c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b90877640 1 -- 172.21.15.28:0/3236206458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d4fbea0 msgr2=0x55d97d50c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b90877640 1 --2- 172.21.15.28:0/3236206458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d4fbea0 0x55d97d50c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b90877640 1 -- 172.21.15.28:0/3236206458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d4fb580 msgr2=0x55d97d50c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b90877640 1 --2- 172.21.15.28:0/3236206458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d4fb580 0x55d97d50c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b90877640 1 -- 172.21.15.28:0/3236206458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d97d57bea0 con 0x55d97d506570 2024-04-25T01:43:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b8bfff640 1 --2- 172.21.15.28:0/3236206458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d4fb580 0x55d97d50c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:43:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b90877640 1 --2- 172.21.15.28:0/3236206458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d506570 0x55d97d50ce90 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f7b84004010 tx=0x7f7b84009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=649e87f3b5669657 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b6b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:43:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b6b7fe640 1 -- 172.21.15.28:0/3236206458 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b84024040 con 0x55d97d506570 2024-04-25T01:43:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b94fee740 1 -- 172.21.15.28:0/3236206458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b7400fab0 con 0x55d97d506570 2024-04-25T01:43:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.297+0000 7f7b6b7fe640 1 -- 172.21.15.28:0/3236206458 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f7b8401eca0 con 0x55d97d506570 2024-04-25T01:43:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.298+0000 7f7b6b7fe640 1 -- 172.21.15.28:0/3236206458 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7b8403c050 con 0x55d97d506570 2024-04-25T01:43:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.298+0000 7f7b94fee740 10 client.19323 fetch_fsmap learned FSMap version 28 2024-04-25T01:43:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.298+0000 7f7b94fee740 1 -- 172.21.15.28:0/3236206458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d97d57d110 con 0x55d97d506570 2024-04-25T01:43:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.298+0000 7f7b6b7fe640 1 -- 172.21.15.28:0/3236206458 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b7400fab0 con 0x55d97d506570 2024-04-25T01:43:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.298+0000 7f7b6b7fe640 1 -- 172.21.15.28:0/3236206458 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7b84010810 con 0x55d97d506570 2024-04-25T01:43:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.298+0000 7f7b94fee740 10 client.19323 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:43:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.298+0000 7f7b94fee740 10 client.19323 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-04-25T01:43:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.298+0000 7f7b94fee740 20 client.19323 populate_metadata read hostname 'smithi028' 2024-04-25T01:43:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.298+0000 7f7b94fee740 1 --2- 172.21.15.28:0/3236206458 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55d97d57d5a0 0x55d97d57f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.298+0000 7f7b94fee740 4 client.19323 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:43:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.298+0000 7f7b94fee740 1 -- 172.21.15.28:0/3236206458 --> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d97d467f00 con 0x55d97d57d5a0 2024-04-25T01:43:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.298+0000 7f7b8bfff640 1 --2- 172.21.15.28:0/3236206458 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55d97d57d5a0 0x55d97d57f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.299+0000 7f7b8bfff640 1 --2- 172.21.15.28:0/3236206458 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55d97d57d5a0 0x55d97d57f990 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ce120bb04d361663 server_cookie=e97cbfbed431aa33 in_seq=0 out_seq=0 2024-04-25T01:43:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.299+0000 7f7b6b7fe640 10 client.19323 ms_handle_connect on v2:172.21.15.130:6834/348635896 2024-04-25T01:43:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.299+0000 7f7b6b7fe640 1 -- 172.21.15.28:0/3236206458 <== mds.4 v2:172.21.15.130:6834/348635896 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d97d467f00 con 0x55d97d57d5a0 2024-04-25T01:43:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.299+0000 7f7b6b7fe640 10 client.19323 handle_command_reply: tid=0 2024-04-25T01:43:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.299+0000 7f7b94fee740 1 client.19323 shutdown 2024-04-25T01:43:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.299+0000 7f7b94fee740 1 -- 172.21.15.28:0/3236206458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d506570 msgr2=0x55d97d50ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.299+0000 7f7b94fee740 1 --2- 172.21.15.28:0/3236206458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d506570 0x55d97d50ce90 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f7b84004010 tx=0x7f7b84009bf0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.299+0000 7f7b94fee740 1 -- 172.21.15.28:0/3236206458 shutdown_connections 2024-04-25T01:43:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.299+0000 7f7b94fee740 1 --2- 172.21.15.28:0/3236206458 >> [v2:172.21.15.130:6834/348635896,v1:172.21.15.130:6835/348635896] conn(0x55d97d57d5a0 0x55d97d57f990 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.299+0000 7f7b94fee740 1 --2- 172.21.15.28:0/3236206458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d97d506570 0x55d97d50ce90 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.299+0000 7f7b94fee740 1 --2- 172.21.15.28:0/3236206458 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x55d97d4fbea0 0x55d97d50c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.299+0000 7f7b6b7fe640 0 client.19323 ms_handle_reset on v2:172.21.15.130:6834/348635896 2024-04-25T01:43:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.299+0000 7f7b94fee740 1 --2- 172.21.15.28:0/3236206458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d97d4fb580 0x55d97d50c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.299+0000 7f7b94fee740 1 -- 172.21.15.28:0/3236206458 >> 172.21.15.28:0/3236206458 conn(0x55d97d4f78a0 msgr2=0x55d97d511aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.299+0000 7f7b94fee740 1 -- 172.21.15.28:0/3236206458 shutdown_connections 2024-04-25T01:43:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.299+0000 7f7b94fee740 1 -- 172.21.15.28:0/3236206458 wait complete. 2024-04-25T01:43:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.299+0000 7f7b94fee740 20 client.19323 trim_cache size 0 max 16384 2024-04-25T01:43:11.295 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:43:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.300+0000 7f7b922da640 1 -- 172.21.15.28:0/4180671538 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7b780a2d00 msgr2=0x7f7b780a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.300+0000 7f7b922da640 1 --2- 172.21.15.28:0/4180671538 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7b780a2d00 0x7f7b780a51c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7b84021000 tx=0x7f7b8401e000 comp rx=0 tx=0).stop 2024-04-25T01:43:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.300+0000 7f7b922da640 1 -- 172.21.15.28:0/4180671538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c06a780 msgr2=0x7f7b8c100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.300+0000 7f7b922da640 1 --2- 172.21.15.28:0/4180671538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c06a780 0x7f7b8c100bb0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c009230 tx=0x7f7b7c004270 comp rx=0 tx=0).stop 2024-04-25T01:43:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.300+0000 7f7b922da640 1 -- 172.21.15.28:0/4180671538 shutdown_connections 2024-04-25T01:43:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.300+0000 7f7b922da640 1 --2- 172.21.15.28:0/4180671538 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7b780a2d00 0x7f7b780a51c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.300+0000 7f7b922da640 1 --2- 172.21.15.28:0/4180671538 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7b8c107730 0x7f7b8c1010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.300+0000 7f7b922da640 1 --2- 172.21.15.28:0/4180671538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8c06a780 0x7f7b8c100bb0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.300+0000 7f7b922da640 1 --2- 172.21.15.28:0/4180671538 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b8c069dc0 0x7f7b8c100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.300+0000 7f7b922da640 1 -- 172.21.15.28:0/4180671538 >> 172.21.15.28:0/4180671538 conn(0x7f7b8c0776a0 msgr2=0x7f7b8c0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.300+0000 7f7b922da640 1 -- 172.21.15.28:0/4180671538 shutdown_connections 2024-04-25T01:43:11.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.300+0000 7f7b922da640 1 -- 172.21.15.28:0/4180671538 wait complete. 2024-04-25T01:43:11.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T01:43:11.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.414+0000 7f41b083f640 1 -- 172.21.15.28:0/966205823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41ac107720 msgr2=0x7f41ac107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.414+0000 7f41b083f640 1 --2- 172.21.15.28:0/966205823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41ac107720 0x7f41ac107b00 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f419c009930 tx=0x7f419c02a420 comp rx=0 tx=0).stop 2024-04-25T01:43:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.414+0000 7f41b083f640 1 -- 172.21.15.28:0/966205823 shutdown_connections 2024-04-25T01:43:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.414+0000 7f41b083f640 1 --2- 172.21.15.28:0/966205823 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41ac1020d0 0x7f41ac10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.414+0000 7f41b083f640 1 --2- 172.21.15.28:0/966205823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41ac101710 0x7f41ac101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.414+0000 7f41b083f640 1 --2- 172.21.15.28:0/966205823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41ac107720 0x7f41ac107b00 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.414+0000 7f41b083f640 1 -- 172.21.15.28:0/966205823 >> 172.21.15.28:0/966205823 conn(0x7f41ac0fd180 msgr2=0x7f41ac0ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.414+0000 7f41b083f640 1 -- 172.21.15.28:0/966205823 shutdown_connections 2024-04-25T01:43:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.414+0000 7f41b083f640 1 -- 172.21.15.28:0/966205823 wait complete. 2024-04-25T01:43:11.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.415+0000 7f41b083f640 1 Processor -- start 2024-04-25T01:43:11.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.415+0000 7f41b083f640 1 -- start start 2024-04-25T01:43:11.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.415+0000 7f41b083f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41ac101710 0x7f41ac19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.415+0000 7f41b083f640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41ac1020d0 0x7f41ac19a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.415+0000 7f41b083f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41ac107720 0x7f41ac19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.415+0000 7f41b083f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac111af0 con 0x7f41ac107720 2024-04-25T01:43:11.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.415+0000 7f41b083f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac111970 con 0x7f41ac101710 2024-04-25T01:43:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.415+0000 7f41b083f640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac111c70 con 0x7f41ac1020d0 2024-04-25T01:43:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.416+0000 7f41aad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41ac107720 0x7f41ac19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.416+0000 7f41aad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41ac107720 0x7f41ac19eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56646/0 (socket says 172.21.15.28:56646) 2024-04-25T01:43:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.416+0000 7f41aad9d640 1 -- 172.21.15.28:0/1358326427 learned_addr learned my addr 172.21.15.28:0/1358326427 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.416+0000 7f41a9d9b640 1 --2- 172.21.15.28:0/1358326427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41ac1020d0 0x7f41ac19a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.416+0000 7f41aa59c640 1 --2- 172.21.15.28:0/1358326427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41ac101710 0x7f41ac19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.416+0000 7f41aad9d640 1 -- 172.21.15.28:0/1358326427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41ac1020d0 msgr2=0x7f41ac19a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.416+0000 7f41aad9d640 1 --2- 172.21.15.28:0/1358326427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41ac1020d0 0x7f41ac19a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.416+0000 7f41aad9d640 1 -- 172.21.15.28:0/1358326427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41ac101710 msgr2=0x7f41ac19a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.416+0000 7f41aad9d640 1 --2- 172.21.15.28:0/1358326427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41ac101710 0x7f41ac19a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.416+0000 7f41aad9d640 1 -- 172.21.15.28:0/1358326427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ac19f210 con 0x7f41ac107720 2024-04-25T01:43:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.416+0000 7f41aa59c640 1 --2- 172.21.15.28:0/1358326427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41ac101710 0x7f41ac19a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.416+0000 7f41aad9d640 1 --2- 172.21.15.28:0/1358326427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41ac107720 0x7f41ac19eae0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f41a400cad0 tx=0x7f41a400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.416+0000 7f419b7fe640 1 -- 172.21.15.28:0/1358326427 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a400ae20 con 0x7f41ac107720 2024-04-25T01:43:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.416+0000 7f41b083f640 1 -- 172.21.15.28:0/1358326427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41ac19f500 con 0x7f41ac107720 2024-04-25T01:43:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.416+0000 7f419b7fe640 1 -- 172.21.15.28:0/1358326427 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41a4005020 con 0x7f41ac107720 2024-04-25T01:43:11.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.416+0000 7f41b083f640 1 -- 172.21.15.28:0/1358326427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41ac1a6d90 con 0x7f41ac107720 2024-04-25T01:43:11.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.416+0000 7f419b7fe640 1 -- 172.21.15.28:0/1358326427 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a401d060 con 0x7f41ac107720 2024-04-25T01:43:11.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.419+0000 7f419b7fe640 1 -- 172.21.15.28:0/1358326427 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f41a40051d0 con 0x7f41ac107720 2024-04-25T01:43:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.419+0000 7f41b362c740 1 Processor -- start 2024-04-25T01:43:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41b362c740 1 -- start start 2024-04-25T01:43:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41b362c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c00ceb580 0x564c00ceb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41b362c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 0x564c00cf6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41b362c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 0x564c00cf8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41b362c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564c00c50670 con 0x564c00cf6570 2024-04-25T01:43:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41b362c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564c00bf8970 con 0x564c00ceb580 2024-04-25T01:43:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41b362c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x564c00c4c630 con 0x564c00cebea0 2024-04-25T01:43:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 0x564c00cf8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 0x564c00cf8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56658/0 (socket says 172.21.15.28:56658) 2024-04-25T01:43:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41a9d9b640 1 -- 172.21.15.28:0/2621721726 learned_addr learned my addr 172.21.15.28:0/2621721726 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41aa59c640 1 --2- 172.21.15.28:0/2621721726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 0x564c00cf6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41a9d9b640 1 --2- 172.21.15.28:0/2621721726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c00ceb580 0x564c00ceb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f419b7fe640 1 --2- 172.21.15.28:0/1358326427 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f41940a2d00 0x7f41940a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41a9d9b640 1 -- 172.21.15.28:0/2621721726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 msgr2=0x564c00cf6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41a9d9b640 1 --2- 172.21.15.28:0/2621721726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 0x564c00cf6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41a9d9b640 1 -- 172.21.15.28:0/2621721726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c00ceb580 msgr2=0x564c00ceb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41a9d9b640 1 --2- 172.21.15.28:0/2621721726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c00ceb580 0x564c00ceb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41a9d9b640 1 -- 172.21.15.28:0/2621721726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c00c1c400 con 0x564c00cf6570 2024-04-25T01:43:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41aa59c640 1 --2- 172.21.15.28:0/2621721726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 0x564c00cf6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41aad9d640 1 --2- 172.21.15.28:0/1358326427 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f41940a2d00 0x7f41940a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.420+0000 7f41a9d9b640 1 --2- 172.21.15.28:0/2621721726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 0x564c00cf8960 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f41a00125b0 tx=0x7f41a000b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d6b968caf241c00 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.421+0000 7f419b7fe640 1 -- 172.21.15.28:0/1358326427 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f41a40cfb40 con 0x7f41ac107720 2024-04-25T01:43:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.421+0000 7f4198ff9640 1 -- 172.21.15.28:0/2621721726 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a0004220 con 0x564c00cf6570 2024-04-25T01:43:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.421+0000 7f4198ff9640 1 -- 172.21.15.28:0/2621721726 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41a00043c0 con 0x564c00cf6570 2024-04-25T01:43:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.421+0000 7f4198ff9640 1 -- 172.21.15.28:0/2621721726 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a0010a20 con 0x564c00cf6570 2024-04-25T01:43:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.421+0000 7f41b362c740 1 -- 172.21.15.28:0/2621721726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 msgr2=0x564c00cf8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.421+0000 7f41aad9d640 1 --2- 172.21.15.28:0/1358326427 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f41940a2d00 0x7f41940a51c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f41a40ccf50 tx=0x7f41a406dc30 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.421+0000 7f41b362c740 1 --2- 172.21.15.28:0/2621721726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 0x564c00cf8960 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f41a00125b0 tx=0x7f41a000b650 comp rx=0 tx=0).stop 2024-04-25T01:43:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.421+0000 7f41b362c740 1 -- 172.21.15.28:0/2621721726 shutdown_connections 2024-04-25T01:43:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.421+0000 7f41b362c740 1 --2- 172.21.15.28:0/2621721726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 0x564c00cf8960 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.421+0000 7f41b362c740 1 --2- 172.21.15.28:0/2621721726 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 0x564c00cf6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.421+0000 7f41b362c740 1 --2- 172.21.15.28:0/2621721726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c00ceb580 0x564c00ceb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.421+0000 7f41b362c740 1 -- 172.21.15.28:0/2621721726 >> 172.21.15.28:0/2621721726 conn(0x564c00ce78a0 msgr2=0x564c00ce7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.422+0000 7f41b362c740 1 -- 172.21.15.28:0/2621721726 shutdown_connections 2024-04-25T01:43:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.422+0000 7f41b362c740 1 -- 172.21.15.28:0/2621721726 wait complete. 2024-04-25T01:43:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.422+0000 7f41b362c740 1 -- start start 2024-04-25T01:43:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.422+0000 7f41b362c740 1 Processor -- start 2024-04-25T01:43:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.422+0000 7f41b362c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c00ceb580 0x564c00cf5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.422+0000 7f41b362c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 0x564c00cf31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.422+0000 7f41b362c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 0x564c00cf3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.422+0000 7f41b362c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564c00cf9360 con 0x564c00cf6570 2024-04-25T01:43:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.422+0000 7f41b362c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564c00c0c050 con 0x564c00ceb580 2024-04-25T01:43:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.422+0000 7f41b362c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x564c00cf94e0 con 0x564c00cebea0 2024-04-25T01:43:11.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.422+0000 7f41b362c740 10 client.0 _ll_register_callbacks cb 0x564c00963be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:43:11.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.422+0000 7f41a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 0x564c00cf3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.422+0000 7f41a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 0x564c00cf3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56672/0 (socket says 172.21.15.28:56672) 2024-04-25T01:43:11.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.422+0000 7f41a9d9b640 1 -- 172.21.15.28:0/2605182540 learned_addr learned my addr 172.21.15.28:0/2605182540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.422+0000 7f41a9d9b640 1 --2- 172.21.15.28:0/2605182540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c00ceb580 0x564c00cf5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.423+0000 7f41aa59c640 1 --2- 172.21.15.28:0/2605182540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 0x564c00cf31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.423+0000 7f41a9d9b640 1 -- 172.21.15.28:0/2605182540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 msgr2=0x564c00cf31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.423+0000 7f41a9d9b640 1 --2- 172.21.15.28:0/2605182540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 0x564c00cf31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.423+0000 7f41a9d9b640 1 -- 172.21.15.28:0/2605182540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c00ceb580 msgr2=0x564c00cf5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.423+0000 7f41a9d9b640 1 --2- 172.21.15.28:0/2605182540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c00ceb580 0x564c00cf5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.423+0000 7f41a9d9b640 1 -- 172.21.15.28:0/2605182540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c00bdddc0 con 0x564c00cf6570 2024-04-25T01:43:11.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.423+0000 7f41a9d9b640 1 --2- 172.21.15.28:0/2605182540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 0x564c00cf3710 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f41a0004610 tx=0x7f41a00104b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d7369123221471f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.423+0000 7f4191ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:43:11.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.423+0000 7f4191ffb640 1 -- 172.21.15.28:0/2605182540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a0005460 con 0x564c00cf6570 2024-04-25T01:43:11.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.423+0000 7f4191ffb640 1 -- 172.21.15.28:0/2605182540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f41a0010840 con 0x564c00cf6570 2024-04-25T01:43:11.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.423+0000 7f41b362c740 1 -- 172.21.15.28:0/2605182540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41a00043c0 con 0x564c00cf6570 2024-04-25T01:43:11.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.423+0000 7f4191ffb640 1 -- 172.21.15.28:0/2605182540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f41a0033050 con 0x564c00cf6570 2024-04-25T01:43:11.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.423+0000 7f41b362c740 10 client.19335 fetch_fsmap learned FSMap version 28 2024-04-25T01:43:11.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.423+0000 7f41b362c740 1 -- 172.21.15.28:0/2605182540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564c009643f0 con 0x564c00cf6570 2024-04-25T01:43:11.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.423+0000 7f4191ffb640 1 -- 172.21.15.28:0/2605182540 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41a00043c0 con 0x564c00cf6570 2024-04-25T01:43:11.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.424+0000 7f4191ffb640 1 -- 172.21.15.28:0/2605182540 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f41a001c070 con 0x564c00cf6570 2024-04-25T01:43:11.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.424+0000 7f41b362c740 10 client.19335 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:43:11.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.424+0000 7f41b362c740 10 client.19335 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:43:11.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.424+0000 7f41b362c740 20 client.19335 populate_metadata read hostname 'smithi028' 2024-04-25T01:43:11.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.424+0000 7f41b362c740 1 --2- 172.21.15.28:0/2605182540 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x564c00d06480 0x564c00d08870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.424+0000 7f41b362c740 4 client.19335 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:43:11.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.424+0000 7f41b362c740 1 -- 172.21.15.28:0/2605182540 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c00c5dbb0 con 0x564c00d06480 2024-04-25T01:43:11.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.424+0000 7f41aa59c640 1 --2- 172.21.15.28:0/2605182540 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x564c00d06480 0x564c00d08870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.424+0000 7f4191ffb640 10 client.19335 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:43:11.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.424+0000 7f41aa59c640 1 --2- 172.21.15.28:0/2605182540 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x564c00d06480 0x564c00d08870 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ba6a9323ec1cc0d0 server_cookie=2e8949fcb48c4b85 in_seq=0 out_seq=0 2024-04-25T01:43:11.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.425+0000 7f4191ffb640 1 -- 172.21.15.28:0/2605182540 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564c00c5dbb0 con 0x564c00d06480 2024-04-25T01:43:11.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.425+0000 7f4191ffb640 10 client.19335 handle_command_reply: tid=0 2024-04-25T01:43:11.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.425+0000 7f41b362c740 1 client.19335 shutdown 2024-04-25T01:43:11.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.425+0000 7f41b362c740 1 -- 172.21.15.28:0/2605182540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 msgr2=0x564c00cf3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.425+0000 7f41b362c740 1 --2- 172.21.15.28:0/2605182540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 0x564c00cf3710 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f41a0004610 tx=0x7f41a00104b0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.425+0000 7f41b362c740 1 -- 172.21.15.28:0/2605182540 shutdown_connections 2024-04-25T01:43:11.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.425+0000 7f41b362c740 1 --2- 172.21.15.28:0/2605182540 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x564c00d06480 0x564c00d08870 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.425+0000 7f41b362c740 1 --2- 172.21.15.28:0/2605182540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 0x564c00cf3710 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.425+0000 7f41b362c740 1 --2- 172.21.15.28:0/2605182540 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 0x564c00cf31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.425+0000 7f41b362c740 1 --2- 172.21.15.28:0/2605182540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c00ceb580 0x564c00cf5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.425+0000 7f41b362c740 1 -- 172.21.15.28:0/2605182540 >> 172.21.15.28:0/2605182540 conn(0x564c00ce78a0 msgr2=0x564c00ced6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.425+0000 7f4191ffb640 0 client.19335 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:43:11.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.425+0000 7f41b362c740 1 -- 172.21.15.28:0/2605182540 shutdown_connections 2024-04-25T01:43:11.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.425+0000 7f41b362c740 1 -- 172.21.15.28:0/2605182540 wait complete. 2024-04-25T01:43:11.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.425+0000 7f41b362c740 20 client.19335 trim_cache size 0 max 16384 2024-04-25T01:43:11.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.439+0000 7f41b362c740 1 Processor -- start 2024-04-25T01:43:11.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.439+0000 7f41b362c740 1 -- start start 2024-04-25T01:43:11.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.439+0000 7f41b362c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00ceb580 0x564c00d02250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.439+0000 7f41b362c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 0x564c00cf1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.439+0000 7f41b362c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c00cf6570 0x564c00d0df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.439+0000 7f41b362c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac1b0f20 con 0x564c00ceb580 2024-04-25T01:43:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.439+0000 7f41b362c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac1b2a60 con 0x564c00cf6570 2024-04-25T01:43:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.439+0000 7f41b362c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac1b15f0 con 0x564c00cebea0 2024-04-25T01:43:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.439+0000 7f41a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00ceb580 0x564c00d02250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.439+0000 7f41a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00ceb580 0x564c00d02250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56686/0 (socket says 172.21.15.28:56686) 2024-04-25T01:43:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.439+0000 7f41a9d9b640 1 -- 172.21.15.28:0/2559620818 learned_addr learned my addr 172.21.15.28:0/2559620818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.439+0000 7f41a9d9b640 1 -- 172.21.15.28:0/2559620818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 msgr2=0x564c00cf1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.439+0000 7f41a9d9b640 1 --2- 172.21.15.28:0/2559620818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 0x564c00cf1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.439+0000 7f41a9d9b640 1 -- 172.21.15.28:0/2559620818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c00cf6570 msgr2=0x564c00d0df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.439+0000 7f41a9d9b640 1 --2- 172.21.15.28:0/2559620818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c00cf6570 0x564c00d0df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.439+0000 7f41a9d9b640 1 -- 172.21.15.28:0/2559620818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c00cf3a80 con 0x564c00ceb580 2024-04-25T01:43:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.440+0000 7f41a9d9b640 1 --2- 172.21.15.28:0/2559620818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00ceb580 0x564c00d02250 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f41a0000c00 tx=0x7f41a000bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a60dca740e98990d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.440+0000 7f4193fff640 1 -- 172.21.15.28:0/2559620818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a000b410 con 0x564c00ceb580 2024-04-25T01:43:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.440+0000 7f4193fff640 1 -- 172.21.15.28:0/2559620818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41a0021030 con 0x564c00ceb580 2024-04-25T01:43:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.440+0000 7f4193fff640 1 -- 172.21.15.28:0/2559620818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a0004cc0 con 0x564c00ceb580 2024-04-25T01:43:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.440+0000 7f41b362c740 1 -- 172.21.15.28:0/2559620818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00ceb580 msgr2=0x564c00d02250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.440+0000 7f41b362c740 1 --2- 172.21.15.28:0/2559620818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00ceb580 0x564c00d02250 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f41a0000c00 tx=0x7f41a000bb00 comp rx=0 tx=0).stop 2024-04-25T01:43:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.440+0000 7f41b362c740 1 -- 172.21.15.28:0/2559620818 shutdown_connections 2024-04-25T01:43:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.440+0000 7f41b362c740 1 --2- 172.21.15.28:0/2559620818 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c00cf6570 0x564c00d0df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.440+0000 7f41b362c740 1 --2- 172.21.15.28:0/2559620818 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 0x564c00cf1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.440+0000 7f41b362c740 1 --2- 172.21.15.28:0/2559620818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00ceb580 0x564c00d02250 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.440+0000 7f41b362c740 1 -- 172.21.15.28:0/2559620818 >> 172.21.15.28:0/2559620818 conn(0x564c00ce78a0 msgr2=0x564c00ce6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.440+0000 7f41b362c740 1 -- 172.21.15.28:0/2559620818 shutdown_connections 2024-04-25T01:43:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.440+0000 7f41b362c740 1 -- 172.21.15.28:0/2559620818 wait complete. 2024-04-25T01:43:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.441+0000 7f41b362c740 1 -- start start 2024-04-25T01:43:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.441+0000 7f41b362c740 1 Processor -- start 2024-04-25T01:43:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.441+0000 7f41b362c740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c00ceb580 0x564c00cfb920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.441+0000 7f41b362c740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 0x564c00cfbe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.441+0000 7f41b362c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 0x564c00d6a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.441+0000 7f41b362c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac1b2390 con 0x564c00cf6570 2024-04-25T01:43:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.441+0000 7f41b362c740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac1b1cc0 con 0x564c00ceb580 2024-04-25T01:43:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.441+0000 7f41b362c740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac1aef20 con 0x564c00cebea0 2024-04-25T01:43:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.441+0000 7f41b362c740 10 client.0 _ll_register_callbacks cb 0x7f41a0012410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:43:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.441+0000 7f41a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 0x564c00d6a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.441+0000 7f41a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 0x564c00d6a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56692/0 (socket says 172.21.15.28:56692) 2024-04-25T01:43:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.441+0000 7f41a9d9b640 1 -- 172.21.15.28:0/2943757409 learned_addr learned my addr 172.21.15.28:0/2943757409 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.441+0000 7f41a9d9b640 1 --2- 172.21.15.28:0/2943757409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c00ceb580 0x564c00cfb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.441+0000 7f41aa59c640 1 --2- 172.21.15.28:0/2943757409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 0x564c00cfbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.441+0000 7f41a9d9b640 1 -- 172.21.15.28:0/2943757409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 msgr2=0x564c00cfbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.441+0000 7f41a9d9b640 1 --2- 172.21.15.28:0/2943757409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 0x564c00cfbe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.441+0000 7f41a9d9b640 1 -- 172.21.15.28:0/2943757409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c00ceb580 msgr2=0x564c00cfb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.441+0000 7f41a9d9b640 1 --2- 172.21.15.28:0/2943757409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c00ceb580 0x564c00cfb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.441+0000 7f41a9d9b640 1 -- 172.21.15.28:0/2943757409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c00a2e7d0 con 0x564c00cf6570 2024-04-25T01:43:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.442+0000 7f41a9d9b640 1 --2- 172.21.15.28:0/2943757409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 0x564c00d6a7d0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f41a000bc10 tx=0x7f41a0004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae7cc4ea48d7ab75 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.442+0000 7f4191ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:43:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.442+0000 7f4191ffb640 1 -- 172.21.15.28:0/2943757409 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a00120d0 con 0x564c00cf6570 2024-04-25T01:43:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.442+0000 7f4191ffb640 1 -- 172.21.15.28:0/2943757409 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f41a0004a60 con 0x564c00cf6570 2024-04-25T01:43:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.442+0000 7f41b362c740 1 -- 172.21.15.28:0/2943757409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41a0021030 con 0x564c00cf6570 2024-04-25T01:43:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.442+0000 7f4191ffb640 1 -- 172.21.15.28:0/2943757409 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f41a0012270 con 0x564c00cf6570 2024-04-25T01:43:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.442+0000 7f41b362c740 10 client.19341 fetch_fsmap learned FSMap version 28 2024-04-25T01:43:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.442+0000 7f41b362c740 1 -- 172.21.15.28:0/2943757409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564c00d6be40 con 0x564c00cf6570 2024-04-25T01:43:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.442+0000 7f4191ffb640 1 -- 172.21.15.28:0/2943757409 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41a0021030 con 0x564c00cf6570 2024-04-25T01:43:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.442+0000 7f4191ffb640 1 -- 172.21.15.28:0/2943757409 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f41a0042040 con 0x564c00cf6570 2024-04-25T01:43:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.442+0000 7f41b362c740 10 client.19341 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:43:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.442+0000 7f41b362c740 10 client.19341 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-04-25T01:43:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.442+0000 7f41b362c740 20 client.19341 populate_metadata read hostname 'smithi028' 2024-04-25T01:43:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.442+0000 7f41b362c740 1 --2- 172.21.15.28:0/2943757409 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x564c00d6c3f0 0x564c00d6e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.442+0000 7f41b362c740 4 client.19341 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:43:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.442+0000 7f41b362c740 1 -- 172.21.15.28:0/2943757409 --> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564c00c57f00 con 0x564c00d6c3f0 2024-04-25T01:43:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.443+0000 7f41aa59c640 1 --2- 172.21.15.28:0/2943757409 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x564c00d6c3f0 0x564c00d6e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.443+0000 7f4191ffb640 10 client.19341 ms_handle_connect on v2:172.21.15.130:6836/211524954 2024-04-25T01:43:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.443+0000 7f41aa59c640 1 --2- 172.21.15.28:0/2943757409 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x564c00d6c3f0 0x564c00d6e7e0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=63f95c85806ebdf server_cookie=8776487aaf49f967 in_seq=0 out_seq=0 2024-04-25T01:43:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.443+0000 7f4191ffb640 1 -- 172.21.15.28:0/2943757409 <== mds.3 v2:172.21.15.130:6836/211524954 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564c00c57f00 con 0x564c00d6c3f0 2024-04-25T01:43:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.443+0000 7f4191ffb640 10 client.19341 handle_command_reply: tid=0 2024-04-25T01:43:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.443+0000 7f41b362c740 1 client.19341 shutdown 2024-04-25T01:43:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.443+0000 7f41b362c740 1 -- 172.21.15.28:0/2943757409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 msgr2=0x564c00d6a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.443+0000 7f41b362c740 1 --2- 172.21.15.28:0/2943757409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 0x564c00d6a7d0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f41a000bc10 tx=0x7f41a0004340 comp rx=0 tx=0).stop 2024-04-25T01:43:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.443+0000 7f41b362c740 1 -- 172.21.15.28:0/2943757409 shutdown_connections 2024-04-25T01:43:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.443+0000 7f41b362c740 1 --2- 172.21.15.28:0/2943757409 >> [v2:172.21.15.130:6836/211524954,v1:172.21.15.130:6837/211524954] conn(0x564c00d6c3f0 0x564c00d6e7e0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.443+0000 7f41b362c740 1 --2- 172.21.15.28:0/2943757409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c00cf6570 0x564c00d6a7d0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.443+0000 7f41b362c740 1 --2- 172.21.15.28:0/2943757409 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x564c00cebea0 0x564c00cfbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.443+0000 7f4191ffb640 0 client.19341 ms_handle_reset on v2:172.21.15.130:6836/211524954 2024-04-25T01:43:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.443+0000 7f41b362c740 1 --2- 172.21.15.28:0/2943757409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564c00ceb580 0x564c00cfb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.443+0000 7f41b362c740 1 -- 172.21.15.28:0/2943757409 >> 172.21.15.28:0/2943757409 conn(0x564c00ce78a0 msgr2=0x564c00d01240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.443+0000 7f41b362c740 1 -- 172.21.15.28:0/2943757409 shutdown_connections 2024-04-25T01:43:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.443+0000 7f41b362c740 1 -- 172.21.15.28:0/2943757409 wait complete. 2024-04-25T01:43:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.443+0000 7f41b362c740 20 client.19341 trim_cache size 0 max 16384 2024-04-25T01:43:11.439 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:43:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.444+0000 7f41b083f640 1 -- 172.21.15.28:0/1358326427 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f41940a2d00 msgr2=0x7f41940a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.444+0000 7f41b083f640 1 --2- 172.21.15.28:0/1358326427 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f41940a2d00 0x7f41940a51c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f41a40ccf50 tx=0x7f41a406dc30 comp rx=0 tx=0).stop 2024-04-25T01:43:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.444+0000 7f41b083f640 1 -- 172.21.15.28:0/1358326427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41ac107720 msgr2=0x7f41ac19eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.444+0000 7f41b083f640 1 --2- 172.21.15.28:0/1358326427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41ac107720 0x7f41ac19eae0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f41a400cad0 tx=0x7f41a400cf90 comp rx=0 tx=0).stop 2024-04-25T01:43:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.444+0000 7f41b083f640 1 -- 172.21.15.28:0/1358326427 shutdown_connections 2024-04-25T01:43:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.444+0000 7f41b083f640 1 --2- 172.21.15.28:0/1358326427 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f41940a2d00 0x7f41940a51c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.444+0000 7f41b083f640 1 --2- 172.21.15.28:0/1358326427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41ac107720 0x7f41ac19eae0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.444+0000 7f41b083f640 1 --2- 172.21.15.28:0/1358326427 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f41ac1020d0 0x7f41ac19a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.444+0000 7f41b083f640 1 --2- 172.21.15.28:0/1358326427 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f41ac101710 0x7f41ac19a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.444+0000 7f41b083f640 1 -- 172.21.15.28:0/1358326427 >> 172.21.15.28:0/1358326427 conn(0x7f41ac0fd180 msgr2=0x7f41ac0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.444+0000 7f41b083f640 1 -- 172.21.15.28:0/1358326427 shutdown_connections 2024-04-25T01:43:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.444+0000 7f41b083f640 1 -- 172.21.15.28:0/1358326427 wait complete. 2024-04-25T01:43:11.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:43:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.559+0000 7fb3b3dd5640 1 -- 172.21.15.28:0/1786324719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3ac069dc0 msgr2=0x7fb3ac06a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.559+0000 7fb3b3dd5640 1 --2- 172.21.15.28:0/1786324719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3ac069dc0 0x7fb3ac06a240 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fb39c009930 tx=0x7fb39c02a420 comp rx=0 tx=0).stop 2024-04-25T01:43:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.559+0000 7fb3b3dd5640 1 -- 172.21.15.28:0/1786324719 shutdown_connections 2024-04-25T01:43:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.559+0000 7fb3b3dd5640 1 --2- 172.21.15.28:0/1786324719 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3ac06a780 0x7fb3ac10e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.559+0000 7fb3b3dd5640 1 --2- 172.21.15.28:0/1786324719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3ac069dc0 0x7fb3ac06a240 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.559+0000 7fb3b3dd5640 1 --2- 172.21.15.28:0/1786324719 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3ac107730 0x7fb3ac107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.559+0000 7fb3b3dd5640 1 -- 172.21.15.28:0/1786324719 >> 172.21.15.28:0/1786324719 conn(0x7fb3ac0776a0 msgr2=0x7fb3ac077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.559+0000 7fb3b3dd5640 1 -- 172.21.15.28:0/1786324719 shutdown_connections 2024-04-25T01:43:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.559+0000 7fb3b3dd5640 1 -- 172.21.15.28:0/1786324719 wait complete. 2024-04-25T01:43:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.560+0000 7fb3b3dd5640 1 Processor -- start 2024-04-25T01:43:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.560+0000 7fb3b3dd5640 1 -- start start 2024-04-25T01:43:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.560+0000 7fb3b3dd5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3ac069dc0 0x7fb3ac198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.560+0000 7fb3b3dd5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3ac06a780 0x7fb3ac1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.560+0000 7fb3b3dd5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3ac107730 0x7fb3ac19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.560+0000 7fb3b3dd5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac111b40 con 0x7fb3ac107730 2024-04-25T01:43:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.560+0000 7fb3b3dd5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1119c0 con 0x7fb3ac06a780 2024-04-25T01:43:11.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.560+0000 7fb3b3dd5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac111cc0 con 0x7fb3ac069dc0 2024-04-25T01:43:11.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.560+0000 7fb3b2372640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3ac107730 0x7fb3ac19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.560+0000 7fb3b2372640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3ac107730 0x7fb3ac19c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56718/0 (socket says 172.21.15.28:56718) 2024-04-25T01:43:11.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.560+0000 7fb3b2372640 1 -- 172.21.15.28:0/1197301762 learned_addr learned my addr 172.21.15.28:0/1197301762 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.560+0000 7fb3b1370640 1 --2- 172.21.15.28:0/1197301762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3ac06a780 0x7fb3ac1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.560+0000 7fb3b1b71640 1 --2- 172.21.15.28:0/1197301762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3ac069dc0 0x7fb3ac198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.560+0000 7fb3b2372640 1 -- 172.21.15.28:0/1197301762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3ac069dc0 msgr2=0x7fb3ac198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.560+0000 7fb3b2372640 1 --2- 172.21.15.28:0/1197301762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3ac069dc0 0x7fb3ac198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.560+0000 7fb3b2372640 1 -- 172.21.15.28:0/1197301762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3ac06a780 msgr2=0x7fb3ac1985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.560+0000 7fb3b2372640 1 --2- 172.21.15.28:0/1197301762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3ac06a780 0x7fb3ac1985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.560+0000 7fb3b2372640 1 -- 172.21.15.28:0/1197301762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ac19d020 con 0x7fb3ac107730 2024-04-25T01:43:11.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.561+0000 7fb3b2372640 1 --2- 172.21.15.28:0/1197301762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3ac107730 0x7fb3ac19c8f0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400cce0 tx=0x7fb3a400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.561+0000 7fb3a2ffd640 1 -- 172.21.15.28:0/1197301762 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a400f450 con 0x7fb3ac107730 2024-04-25T01:43:11.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.561+0000 7fb3b3dd5640 1 -- 172.21.15.28:0/1197301762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3ac19d310 con 0x7fb3ac107730 2024-04-25T01:43:11.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.561+0000 7fb3a2ffd640 1 -- 172.21.15.28:0/1197301762 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3a400fad0 con 0x7fb3ac107730 2024-04-25T01:43:11.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.561+0000 7fb3b3dd5640 1 -- 172.21.15.28:0/1197301762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3ac1a4ba0 con 0x7fb3ac107730 2024-04-25T01:43:11.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.561+0000 7fb3a2ffd640 1 -- 172.21.15.28:0/1197301762 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4018c10 con 0x7fb3ac107730 2024-04-25T01:43:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.563+0000 7fb3a2ffd640 1 -- 172.21.15.28:0/1197301762 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fb3a400f5f0 con 0x7fb3ac107730 2024-04-25T01:43:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.564+0000 7fb3b6bb6740 1 Processor -- start 2024-04-25T01:43:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.564+0000 7fb3b6bb6740 1 -- start start 2024-04-25T01:43:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.564+0000 7fb3a2ffd640 1 --2- 172.21.15.28:0/1197301762 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb3980a2d00 0x7fb3980a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.564+0000 7fb3b6bb6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56004025b480 0x56004025b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.564+0000 7fb3b6bb6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 0x56004026a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.564+0000 7fb3b6bb6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56004026a980 0x56004026cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.564+0000 7fb3b6bb6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5600401c0670 con 0x56004025bda0 2024-04-25T01:43:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.564+0000 7fb3b6bb6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560040168970 con 0x56004026a980 2024-04-25T01:43:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.564+0000 7fb3b6bb6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5600401bc630 con 0x56004025b480 2024-04-25T01:43:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.564+0000 7fb3b1b71640 1 --2- 172.21.15.28:0/1197301762 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb3980a2d00 0x7fb3980a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.564+0000 7fb3b1370640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 0x56004026a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.564+0000 7fb3a2ffd640 1 -- 172.21.15.28:0/1197301762 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7fb3a40cff90 con 0x7fb3ac107730 2024-04-25T01:43:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.564+0000 7fb3b1370640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 0x56004026a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56724/0 (socket says 172.21.15.28:56724) 2024-04-25T01:43:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.564+0000 7fb3b1370640 1 -- 172.21.15.28:0/1804797060 learned_addr learned my addr 172.21.15.28:0/1804797060 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.564+0000 7fb3b2372640 1 --2- 172.21.15.28:0/1804797060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56004026a980 0x56004026cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.564+0000 7fb3b1370640 1 --2- 172.21.15.28:0/1804797060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56004025b480 0x56004025b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.565+0000 7fb3b2372640 1 -- 172.21.15.28:0/1804797060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56004025b480 msgr2=0x56004025b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.565+0000 7fb3b2372640 1 --2- 172.21.15.28:0/1804797060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56004025b480 0x56004025b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.565+0000 7fb3b2372640 1 -- 172.21.15.28:0/1804797060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 msgr2=0x56004026a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.565+0000 7fb3b2372640 1 --2- 172.21.15.28:0/1804797060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 0x56004026a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.565+0000 7fb3b2372640 1 -- 172.21.15.28:0/1804797060 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56004018c400 con 0x56004026a980 2024-04-25T01:43:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.565+0000 7fb3b1b71640 1 --2- 172.21.15.28:0/1197301762 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb3980a2d00 0x7fb3980a51c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009ae0 tx=0x7fb3a8005f90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.565+0000 7fb3b2372640 1 --2- 172.21.15.28:0/1804797060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56004026a980 0x56004026cd70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb3a40cd260 tx=0x7fb3a409b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6b70c80bc530d06 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.565+0000 7fb397fff640 1 -- 172.21.15.28:0/1804797060 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a40aeba0 con 0x56004026a980 2024-04-25T01:43:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.565+0000 7fb397fff640 1 -- 172.21.15.28:0/1804797060 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3a409c760 con 0x56004026a980 2024-04-25T01:43:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.565+0000 7fb397fff640 1 -- 172.21.15.28:0/1804797060 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a409a100 con 0x56004026a980 2024-04-25T01:43:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.566+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/1804797060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56004026a980 msgr2=0x56004026cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.566+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/1804797060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56004026a980 0x56004026cd70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb3a40cd260 tx=0x7fb3a409b9c0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.566+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/1804797060 shutdown_connections 2024-04-25T01:43:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.566+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/1804797060 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56004026a980 0x56004026cd70 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.566+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/1804797060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 0x56004026a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.566+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/1804797060 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56004025b480 0x56004025b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.566+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/1804797060 >> 172.21.15.28:0/1804797060 conn(0x5600402578a0 msgr2=0x560040257c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.566+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/1804797060 shutdown_connections 2024-04-25T01:43:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.566+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/1804797060 wait complete. 2024-04-25T01:43:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.566+0000 7fb3b6bb6740 1 -- start start 2024-04-25T01:43:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.566+0000 7fb3b6bb6740 1 Processor -- start 2024-04-25T01:43:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3b6bb6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025b480 0x56004026a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3b6bb6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56004025bda0 0x560040267580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3b6bb6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56004026a980 0x560040267ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3b6bb6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56004026d6e0 con 0x56004025b480 2024-04-25T01:43:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3b6bb6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56004017c050 con 0x56004025bda0 2024-04-25T01:43:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3b6bb6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x56004026d860 con 0x56004026a980 2024-04-25T01:43:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3b6bb6740 10 client.0 _ll_register_callbacks cb 0x56003fed3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:43:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3b1b71640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025b480 0x56004026a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3b1b71640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025b480 0x56004026a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56730/0 (socket says 172.21.15.28:56730) 2024-04-25T01:43:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3b1b71640 1 -- 172.21.15.28:0/515228808 learned_addr learned my addr 172.21.15.28:0/515228808 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3b2372640 1 --2- 172.21.15.28:0/515228808 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56004026a980 0x560040267ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3b1370640 1 --2- 172.21.15.28:0/515228808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56004025bda0 0x560040267580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3b1b71640 1 -- 172.21.15.28:0/515228808 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56004026a980 msgr2=0x560040267ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3b1b71640 1 --2- 172.21.15.28:0/515228808 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56004026a980 0x560040267ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3b1b71640 1 -- 172.21.15.28:0/515228808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56004025bda0 msgr2=0x560040267580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3b1b71640 1 --2- 172.21.15.28:0/515228808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56004025bda0 0x560040267580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3b1b71640 1 -- 172.21.15.28:0/515228808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56004014ddc0 con 0x56004025b480 2024-04-25T01:43:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3b1b71640 1 --2- 172.21.15.28:0/515228808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025b480 0x56004026a290 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8004630 tx=0x7fb3a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e00773b8926294c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3957fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:43:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3957fa640 1 -- 172.21.15.28:0/515228808 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a800fad0 con 0x56004025b480 2024-04-25T01:43:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3957fa640 1 -- 172.21.15.28:0/515228808 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7fb3a800fc70 con 0x56004025b480 2024-04-25T01:43:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.567+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/515228808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3a409c760 con 0x56004025b480 2024-04-25T01:43:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.568+0000 7fb3957fa640 1 -- 172.21.15.28:0/515228808 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb3a8030050 con 0x56004025b480 2024-04-25T01:43:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.568+0000 7fb3b6bb6740 10 client.19353 fetch_fsmap learned FSMap version 28 2024-04-25T01:43:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.568+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/515228808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56003fed43f0 con 0x56004025b480 2024-04-25T01:43:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.568+0000 7fb3957fa640 1 -- 172.21.15.28:0/515228808 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3a409c760 con 0x56004025b480 2024-04-25T01:43:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.568+0000 7fb3957fa640 1 -- 172.21.15.28:0/515228808 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb3a8027520 con 0x56004025b480 2024-04-25T01:43:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.568+0000 7fb3b6bb6740 10 client.19353 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:43:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.568+0000 7fb3b6bb6740 10 client.19353 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:43:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.568+0000 7fb3b6bb6740 20 client.19353 populate_metadata read hostname 'smithi028' 2024-04-25T01:43:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.568+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/515228808 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x56004027a990 0x56004027cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.568+0000 7fb3b6bb6740 4 client.19353 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:43:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.568+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/515228808 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600401cdbb0 con 0x56004027a990 2024-04-25T01:43:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.568+0000 7fb3b1370640 1 --2- 172.21.15.28:0/515228808 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x56004027a990 0x56004027cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.569+0000 7fb3957fa640 10 client.19353 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:43:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.569+0000 7fb3b1370640 1 --2- 172.21.15.28:0/515228808 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x56004027a990 0x56004027cd80 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3e3ac3300edf00a7 server_cookie=5a005a36b2459bd1 in_seq=0 out_seq=0 2024-04-25T01:43:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.569+0000 7fb3957fa640 1 -- 172.21.15.28:0/515228808 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5600401cdbb0 con 0x56004027a990 2024-04-25T01:43:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.569+0000 7fb3957fa640 10 client.19353 handle_command_reply: tid=0 2024-04-25T01:43:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.569+0000 7fb3b6bb6740 1 client.19353 shutdown 2024-04-25T01:43:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.570+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/515228808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025b480 msgr2=0x56004026a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.570+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/515228808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025b480 0x56004026a290 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8004630 tx=0x7fb3a8004730 comp rx=0 tx=0).stop 2024-04-25T01:43:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.570+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/515228808 shutdown_connections 2024-04-25T01:43:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.570+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/515228808 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x56004027a990 0x56004027cd80 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.570+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/515228808 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56004026a980 0x560040267ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.570+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/515228808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56004025bda0 0x560040267580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.570+0000 7fb3957fa640 0 client.19353 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:43:11.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.570+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/515228808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025b480 0x56004026a290 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.570+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/515228808 >> 172.21.15.28:0/515228808 conn(0x5600402578a0 msgr2=0x560040256b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.570+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/515228808 shutdown_connections 2024-04-25T01:43:11.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.570+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/515228808 wait complete. 2024-04-25T01:43:11.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.570+0000 7fb3b6bb6740 20 client.19353 trim_cache size 0 max 16384 2024-04-25T01:43:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.583+0000 7fb3b6bb6740 1 Processor -- start 2024-04-25T01:43:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.583+0000 7fb3b6bb6740 1 -- start start 2024-04-25T01:43:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.583+0000 7fb3b6bb6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600402578a0 0x560040262c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.583+0000 7fb3b6bb6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56004025b480 0x560040269dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.583+0000 7fb3b6bb6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 0x560040282650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.583+0000 7fb3b6bb6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1acac0 con 0x56004025bda0 2024-04-25T01:43:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.583+0000 7fb3b6bb6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1ae600 con 0x5600402578a0 2024-04-25T01:43:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.583+0000 7fb3b6bb6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1ad190 con 0x56004025b480 2024-04-25T01:43:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.584+0000 7fb3b2372640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 0x560040282650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.584+0000 7fb3b2372640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 0x560040282650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56738/0 (socket says 172.21.15.28:56738) 2024-04-25T01:43:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.584+0000 7fb3b2372640 1 -- 172.21.15.28:0/1396151553 learned_addr learned my addr 172.21.15.28:0/1396151553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.584+0000 7fb3b1b71640 1 --2- 172.21.15.28:0/1396151553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600402578a0 0x560040262c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.584+0000 7fb3b2372640 1 -- 172.21.15.28:0/1396151553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56004025b480 msgr2=0x560040269dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.584+0000 7fb3b2372640 1 --2- 172.21.15.28:0/1396151553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56004025b480 0x560040269dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.584+0000 7fb3b2372640 1 -- 172.21.15.28:0/1396151553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600402578a0 msgr2=0x560040262c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.584+0000 7fb3b2372640 1 --2- 172.21.15.28:0/1396151553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600402578a0 0x560040262c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.584+0000 7fb3b2372640 1 -- 172.21.15.28:0/1396151553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56003ff9e7d0 con 0x56004025bda0 2024-04-25T01:43:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.584+0000 7fb3b2372640 1 --2- 172.21.15.28:0/1396151553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 0x560040282650 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400ab20 tx=0x7fb3a409baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641cc45fa8d264e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.584+0000 7fb3977fe640 1 -- 172.21.15.28:0/1396151553 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a409c060 con 0x56004025bda0 2024-04-25T01:43:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.584+0000 7fb3977fe640 1 -- 172.21.15.28:0/1396151553 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3a40a0410 con 0x56004025bda0 2024-04-25T01:43:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.584+0000 7fb3977fe640 1 -- 172.21.15.28:0/1396151553 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a400fcb0 con 0x56004025bda0 2024-04-25T01:43:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.585+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/1396151553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 msgr2=0x560040282650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.585+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/1396151553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 0x560040282650 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400ab20 tx=0x7fb3a409baf0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.585+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/1396151553 shutdown_connections 2024-04-25T01:43:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.585+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/1396151553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 0x560040282650 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.585+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/1396151553 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x56004025b480 0x560040269dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.585+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/1396151553 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5600402578a0 0x560040262c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.585+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/1396151553 >> 172.21.15.28:0/1396151553 conn(0x560040267c20 msgr2=0x56004025cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.585+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/1396151553 shutdown_connections 2024-04-25T01:43:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.585+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/1396151553 wait complete. 2024-04-25T01:43:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.585+0000 7fb3b6bb6740 1 -- start start 2024-04-25T01:43:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.585+0000 7fb3b6bb6740 1 Processor -- start 2024-04-25T01:43:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b6bb6740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600402578a0 0x56004026e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b6bb6740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56004025b480 0x56004026ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b6bb6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 0x56004026efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b6bb6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1adf30 con 0x56004025bda0 2024-04-25T01:43:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b6bb6740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1ad860 con 0x56004025b480 2024-04-25T01:43:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b6bb6740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac1aaac0 con 0x5600402578a0 2024-04-25T01:43:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b6bb6740 10 client.0 _ll_register_callbacks cb 0x7fb3a80098f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:43:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b2372640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 0x56004026efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b2372640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 0x56004026efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56752/0 (socket says 172.21.15.28:56752) 2024-04-25T01:43:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b2372640 1 -- 172.21.15.28:0/2233555121 learned_addr learned my addr 172.21.15.28:0/2233555121 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b1b71640 1 --2- 172.21.15.28:0/2233555121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600402578a0 0x56004026e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b1370640 1 --2- 172.21.15.28:0/2233555121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56004025b480 0x56004026ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b2372640 1 -- 172.21.15.28:0/2233555121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600402578a0 msgr2=0x56004026e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b2372640 1 --2- 172.21.15.28:0/2233555121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600402578a0 0x56004026e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b2372640 1 -- 172.21.15.28:0/2233555121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56004025b480 msgr2=0x56004026ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b2372640 1 --2- 172.21.15.28:0/2233555121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56004025b480 0x56004026ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b2372640 1 -- 172.21.15.28:0/2233555121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600402dff10 con 0x56004025bda0 2024-04-25T01:43:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b1b71640 1 --2- 172.21.15.28:0/2233555121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600402578a0 0x56004026e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b2372640 1 --2- 172.21.15.28:0/2233555121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 0x56004026efd0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fb3a40bc010 tx=0x7fb3a409a280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3c70fdf54adc14c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3957fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:43:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3957fa640 1 -- 172.21.15.28:0/2233555121 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a40b4070 con 0x56004025bda0 2024-04-25T01:43:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3957fa640 1 -- 172.21.15.28:0/2233555121 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7fb3a40986d0 con 0x56004025bda0 2024-04-25T01:43:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/2233555121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5600401c7f00 con 0x56004025bda0 2024-04-25T01:43:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3957fa640 1 -- 172.21.15.28:0/2233555121 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb3a4067120 con 0x56004025bda0 2024-04-25T01:43:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b6bb6740 10 client.19359 fetch_fsmap learned FSMap version 28 2024-04-25T01:43:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.586+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/2233555121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5600402e11f0 con 0x56004025bda0 2024-04-25T01:43:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.587+0000 7fb3957fa640 1 -- 172.21.15.28:0/2233555121 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5600401c7f00 con 0x56004025bda0 2024-04-25T01:43:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.587+0000 7fb3957fa640 1 -- 172.21.15.28:0/2233555121 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb3a40cd510 con 0x56004025bda0 2024-04-25T01:43:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.587+0000 7fb3b6bb6740 10 client.19359 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:43:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.587+0000 7fb3b6bb6740 10 client.19359 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:43:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.587+0000 7fb3b6bb6740 20 client.19359 populate_metadata read hostname 'smithi028' 2024-04-25T01:43:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.587+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/2233555121 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x56004026a980 0x5600402e3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.587+0000 7fb3b6bb6740 4 client.19359 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:43:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.587+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/2233555121 --> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560040145820 con 0x56004026a980 2024-04-25T01:43:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.587+0000 7fb3b1b71640 1 --2- 172.21.15.28:0/2233555121 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x56004026a980 0x5600402e3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.588+0000 7fb3b1b71640 1 --2- 172.21.15.28:0/2233555121 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x56004026a980 0x5600402e3480 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4e2dd4088cc0fabf server_cookie=69ab3003250f1e80 in_seq=0 out_seq=0 2024-04-25T01:43:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.588+0000 7fb3957fa640 10 client.19359 ms_handle_connect on v2:172.21.15.32:6836/2771586449 2024-04-25T01:43:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.588+0000 7fb3957fa640 1 -- 172.21.15.28:0/2233555121 <== mds.2 v2:172.21.15.32:6836/2771586449 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560040145820 con 0x56004026a980 2024-04-25T01:43:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.588+0000 7fb3957fa640 10 client.19359 handle_command_reply: tid=0 2024-04-25T01:43:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.588+0000 7fb3b6bb6740 1 client.19359 shutdown 2024-04-25T01:43:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.588+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/2233555121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 msgr2=0x56004026efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.588+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/2233555121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 0x56004026efd0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fb3a40bc010 tx=0x7fb3a409a280 comp rx=0 tx=0).stop 2024-04-25T01:43:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.588+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/2233555121 shutdown_connections 2024-04-25T01:43:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.588+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/2233555121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56004025bda0 0x56004026efd0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.588+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/2233555121 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56004025b480 0x56004026ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.588+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/2233555121 >> [v2:172.21.15.32:6836/2771586449,v1:172.21.15.32:6837/2771586449] conn(0x56004026a980 0x5600402e3480 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.588+0000 7fb3b6bb6740 1 --2- 172.21.15.28:0/2233555121 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5600402578a0 0x56004026e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.588+0000 7fb3957fa640 0 client.19359 ms_handle_reset on v2:172.21.15.32:6836/2771586449 2024-04-25T01:43:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.588+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/2233555121 >> 172.21.15.28:0/2233555121 conn(0x560040267c20 msgr2=0x560040262af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.588+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/2233555121 shutdown_connections 2024-04-25T01:43:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.588+0000 7fb3b6bb6740 1 -- 172.21.15.28:0/2233555121 wait complete. 2024-04-25T01:43:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.588+0000 7fb3b6bb6740 20 client.19359 trim_cache size 0 max 16384 2024-04-25T01:43:11.584 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:43:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.589+0000 7fb3b3dd5640 1 -- 172.21.15.28:0/1197301762 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb3980a2d00 msgr2=0x7fb3980a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.589+0000 7fb3b3dd5640 1 --2- 172.21.15.28:0/1197301762 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb3980a2d00 0x7fb3980a51c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009ae0 tx=0x7fb3a8005f90 comp rx=0 tx=0).stop 2024-04-25T01:43:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.589+0000 7fb3b3dd5640 1 -- 172.21.15.28:0/1197301762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3ac107730 msgr2=0x7fb3ac19c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.589+0000 7fb3b3dd5640 1 --2- 172.21.15.28:0/1197301762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3ac107730 0x7fb3ac19c8f0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400cce0 tx=0x7fb3a400a620 comp rx=0 tx=0).stop 2024-04-25T01:43:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.589+0000 7fb3b3dd5640 1 -- 172.21.15.28:0/1197301762 shutdown_connections 2024-04-25T01:43:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.589+0000 7fb3b3dd5640 1 --2- 172.21.15.28:0/1197301762 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fb3980a2d00 0x7fb3980a51c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.589+0000 7fb3b3dd5640 1 --2- 172.21.15.28:0/1197301762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3ac107730 0x7fb3ac19c8f0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.589+0000 7fb3b3dd5640 1 --2- 172.21.15.28:0/1197301762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3ac06a780 0x7fb3ac1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.589+0000 7fb3b3dd5640 1 --2- 172.21.15.28:0/1197301762 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fb3ac069dc0 0x7fb3ac198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.589+0000 7fb3b3dd5640 1 -- 172.21.15.28:0/1197301762 >> 172.21.15.28:0/1197301762 conn(0x7fb3ac0776a0 msgr2=0x7fb3ac0fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.589+0000 7fb3b3dd5640 1 -- 172.21.15.28:0/1197301762 shutdown_connections 2024-04-25T01:43:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.589+0000 7fb3b3dd5640 1 -- 172.21.15.28:0/1197301762 wait complete. 2024-04-25T01:43:11.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:43:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.703+0000 7fa6ea3a2640 1 -- 172.21.15.28:0/4258514523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6e4100fd0 msgr2=0x7fa6e410d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.703+0000 7fa6ea3a2640 1 --2- 172.21.15.28:0/4258514523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6e4100fd0 0x7fa6e410d4e0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc009cd0 tx=0x7fa6dc02a8e0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.704+0000 7fa6ea3a2640 1 -- 172.21.15.28:0/4258514523 shutdown_connections 2024-04-25T01:43:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.704+0000 7fa6ea3a2640 1 --2- 172.21.15.28:0/4258514523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6e4100fd0 0x7fa6e410d4e0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.704+0000 7fa6ea3a2640 1 --2- 172.21.15.28:0/4258514523 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6e4100610 0x7fa6e4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.704+0000 7fa6ea3a2640 1 --2- 172.21.15.28:0/4258514523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e41066b0 0x7fa6e4106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.704+0000 7fa6ea3a2640 1 -- 172.21.15.28:0/4258514523 >> 172.21.15.28:0/4258514523 conn(0x7fa6e4077740 msgr2=0x7fa6e4077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.704+0000 7fa6ea3a2640 1 -- 172.21.15.28:0/4258514523 shutdown_connections 2024-04-25T01:43:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.704+0000 7fa6ea3a2640 1 -- 172.21.15.28:0/4258514523 wait complete. 2024-04-25T01:43:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.704+0000 7fa6ea3a2640 1 Processor -- start 2024-04-25T01:43:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.704+0000 7fa6ea3a2640 1 -- start start 2024-04-25T01:43:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6ea3a2640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e4100610 0x7fa6e410df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6ea3a2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6e4100fd0 0x7fa6e410e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6ea3a2640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6e41066b0 0x7fa6e410e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6ea3a2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e4111b30 con 0x7fa6e4100fd0 2024-04-25T01:43:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6ea3a2640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e41119b0 con 0x7fa6e4100610 2024-04-25T01:43:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6ea3a2640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e4111cb0 con 0x7fa6e41066b0 2024-04-25T01:43:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6e4100fd0 0x7fa6e410e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6e4100fd0 0x7fa6e410e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56780/0 (socket says 172.21.15.28:56780) 2024-04-25T01:43:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6e37fe640 1 -- 172.21.15.28:0/1910322869 learned_addr learned my addr 172.21.15.28:0/1910322869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6e3fff640 1 --2- 172.21.15.28:0/1910322869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e4100610 0x7fa6e410df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6e893f640 1 --2- 172.21.15.28:0/1910322869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6e41066b0 0x7fa6e410e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6e37fe640 1 -- 172.21.15.28:0/1910322869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6e41066b0 msgr2=0x7fa6e410e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6e37fe640 1 --2- 172.21.15.28:0/1910322869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6e41066b0 0x7fa6e410e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6e37fe640 1 -- 172.21.15.28:0/1910322869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e4100610 msgr2=0x7fa6e410df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6e37fe640 1 --2- 172.21.15.28:0/1910322869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e4100610 0x7fa6e410df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6e37fe640 1 -- 172.21.15.28:0/1910322869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6e41a6430 con 0x7fa6e4100fd0 2024-04-25T01:43:11.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6e37fe640 1 --2- 172.21.15.28:0/1910322869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6e4100fd0 0x7fa6e410e4b0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800d950 tx=0x7fa6d800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6e17fa640 1 -- 172.21.15.28:0/1910322869 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8017070 con 0x7fa6e4100fd0 2024-04-25T01:43:11.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6e17fa640 1 -- 172.21.15.28:0/1910322869 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6d8009c30 con 0x7fa6e4100fd0 2024-04-25T01:43:11.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6ea3a2640 1 -- 172.21.15.28:0/1910322869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6e41a6720 con 0x7fa6e4100fd0 2024-04-25T01:43:11.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6e17fa640 1 -- 172.21.15.28:0/1910322869 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d800f6b0 con 0x7fa6e4100fd0 2024-04-25T01:43:11.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.705+0000 7fa6ea3a2640 1 -- 172.21.15.28:0/1910322869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6e41a6c70 con 0x7fa6e4100fd0 2024-04-25T01:43:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6e17fa640 1 -- 172.21.15.28:0/1910322869 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fa6d80109d0 con 0x7fa6e4100fd0 2024-04-25T01:43:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6ed192740 1 Processor -- start 2024-04-25T01:43:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6ed192740 1 -- start start 2024-04-25T01:43:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6ed192740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556580 0x5614ac556960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6ed192740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac556ea0 0x5614ac561030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6ed192740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac561570 0x5614ac563960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6ed192740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5614ac4bb670 con 0x5614ac556580 2024-04-25T01:43:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6ed192740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5614ac463970 con 0x5614ac561570 2024-04-25T01:43:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6ed192740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5614ac4b7630 con 0x5614ac556ea0 2024-04-25T01:43:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6e893f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556580 0x5614ac556960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6e893f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556580 0x5614ac556960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56788/0 (socket says 172.21.15.28:56788) 2024-04-25T01:43:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6e893f640 1 -- 172.21.15.28:0/813449397 learned_addr learned my addr 172.21.15.28:0/813449397 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6e3fff640 1 --2- 172.21.15.28:0/813449397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac556ea0 0x5614ac561030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6e37fe640 1 --2- 172.21.15.28:0/813449397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac561570 0x5614ac563960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6e893f640 1 -- 172.21.15.28:0/813449397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac556ea0 msgr2=0x5614ac561030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6e893f640 1 --2- 172.21.15.28:0/813449397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac556ea0 0x5614ac561030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6e17fa640 1 --2- 172.21.15.28:0/1910322869 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa6cc0a2d00 0x7fa6cc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6e893f640 1 -- 172.21.15.28:0/813449397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac561570 msgr2=0x5614ac563960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6e893f640 1 --2- 172.21.15.28:0/813449397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac561570 0x5614ac563960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.708+0000 7fa6e893f640 1 -- 172.21.15.28:0/813449397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614ac487400 con 0x5614ac556580 2024-04-25T01:43:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.709+0000 7fa6e37fe640 1 --2- 172.21.15.28:0/813449397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac561570 0x5614ac563960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:43:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.709+0000 7fa6e17fa640 1 -- 172.21.15.28:0/1910322869 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7fa6d80cf4d0 con 0x7fa6e4100fd0 2024-04-25T01:43:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.709+0000 7fa6e893f640 1 --2- 172.21.15.28:0/1910322869 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa6cc0a2d00 0x7fa6cc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.709+0000 7fa6e893f640 1 --2- 172.21.15.28:0/813449397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556580 0x5614ac556960 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc005cc0 tx=0x7fa6dc005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a76d7e116d338e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.709+0000 7fa6ca7fc640 1 -- 172.21.15.28:0/813449397 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc031b20 con 0x5614ac556580 2024-04-25T01:43:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.709+0000 7fa6ca7fc640 1 -- 172.21.15.28:0/813449397 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6dc036040 con 0x5614ac556580 2024-04-25T01:43:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.709+0000 7fa6ca7fc640 1 -- 172.21.15.28:0/813449397 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc030da0 con 0x5614ac556580 2024-04-25T01:43:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.710+0000 7fa6ed192740 1 -- 172.21.15.28:0/813449397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556580 msgr2=0x5614ac556960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.710+0000 7fa6e893f640 1 --2- 172.21.15.28:0/1910322869 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa6cc0a2d00 0x7fa6cc0a51c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc048000 tx=0x7fa6dc04b000 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.710+0000 7fa6ed192740 1 --2- 172.21.15.28:0/813449397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556580 0x5614ac556960 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc005cc0 tx=0x7fa6dc005ef0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.710+0000 7fa6ed192740 1 -- 172.21.15.28:0/813449397 shutdown_connections 2024-04-25T01:43:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.710+0000 7fa6ed192740 1 --2- 172.21.15.28:0/813449397 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac561570 0x5614ac563960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.710+0000 7fa6ed192740 1 --2- 172.21.15.28:0/813449397 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac556ea0 0x5614ac561030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.710+0000 7fa6ed192740 1 --2- 172.21.15.28:0/813449397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556580 0x5614ac556960 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.710+0000 7fa6ed192740 1 -- 172.21.15.28:0/813449397 >> 172.21.15.28:0/813449397 conn(0x5614ac5528a0 msgr2=0x5614ac552c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.710+0000 7fa6ed192740 1 -- 172.21.15.28:0/813449397 shutdown_connections 2024-04-25T01:43:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.710+0000 7fa6ed192740 1 -- 172.21.15.28:0/813449397 wait complete. 2024-04-25T01:43:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.710+0000 7fa6ed192740 1 -- start start 2024-04-25T01:43:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.710+0000 7fa6ed192740 1 Processor -- start 2024-04-25T01:43:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6ed192740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac556580 0x5614ac560ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6ed192740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac556ea0 0x5614ac55e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6ed192740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac561570 0x5614ac55e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6ed192740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5614ac564300 con 0x5614ac561570 2024-04-25T01:43:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6ed192740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5614ac477050 con 0x5614ac556580 2024-04-25T01:43:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6ed192740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5614ac564480 con 0x5614ac556ea0 2024-04-25T01:43:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6ed192740 10 client.0 _ll_register_callbacks cb 0x5614ac1cebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:43:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6e893f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac561570 0x5614ac55e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6e893f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac561570 0x5614ac55e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56794/0 (socket says 172.21.15.28:56794) 2024-04-25T01:43:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6e893f640 1 -- 172.21.15.28:0/4051595785 learned_addr learned my addr 172.21.15.28:0/4051595785 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6e3fff640 1 --2- 172.21.15.28:0/4051595785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac556580 0x5614ac560ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6e37fe640 1 --2- 172.21.15.28:0/4051595785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac556ea0 0x5614ac55e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6e893f640 1 -- 172.21.15.28:0/4051595785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac556ea0 msgr2=0x5614ac55e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6e893f640 1 --2- 172.21.15.28:0/4051595785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac556ea0 0x5614ac55e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6e893f640 1 -- 172.21.15.28:0/4051595785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac556580 msgr2=0x5614ac560ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6e893f640 1 --2- 172.21.15.28:0/4051595785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac556580 0x5614ac560ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6e893f640 1 -- 172.21.15.28:0/4051595785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614ac448dc0 con 0x5614ac561570 2024-04-25T01:43:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6e893f640 1 --2- 172.21.15.28:0/4051595785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac561570 0x5614ac55e6d0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc030a60 tx=0x7fa6dc02afa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=357ef935703b2423 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:43:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6c37fe640 1 -- 172.21.15.28:0/4051595785 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc060060 con 0x5614ac561570 2024-04-25T01:43:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6ed192740 1 -- 172.21.15.28:0/4051595785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6dc036040 con 0x5614ac561570 2024-04-25T01:43:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.711+0000 7fa6c37fe640 1 -- 172.21.15.28:0/4051595785 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7fa6dc003e00 con 0x5614ac561570 2024-04-25T01:43:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.712+0000 7fa6c37fe640 1 -- 172.21.15.28:0/4051595785 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa6dc067050 con 0x5614ac561570 2024-04-25T01:43:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.712+0000 7fa6ed192740 10 client.19371 fetch_fsmap learned FSMap version 28 2024-04-25T01:43:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.712+0000 7fa6ed192740 1 -- 172.21.15.28:0/4051595785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5614ac1cf3f0 con 0x5614ac561570 2024-04-25T01:43:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.712+0000 7fa6c37fe640 1 -- 172.21.15.28:0/4051595785 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6dc036040 con 0x5614ac561570 2024-04-25T01:43:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.712+0000 7fa6c37fe640 1 -- 172.21.15.28:0/4051595785 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa6dc030070 con 0x5614ac561570 2024-04-25T01:43:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.712+0000 7fa6ed192740 10 client.19371 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:43:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.712+0000 7fa6ed192740 10 client.19371 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:43:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.712+0000 7fa6ed192740 20 client.19371 populate_metadata read hostname 'smithi028' 2024-04-25T01:43:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.712+0000 7fa6ed192740 1 --2- 172.21.15.28:0/4051595785 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5614ac571340 0x5614ac573730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.712+0000 7fa6ed192740 4 client.19371 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:43:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.712+0000 7fa6ed192740 1 -- 172.21.15.28:0/4051595785 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614ac4c8bb0 con 0x5614ac571340 2024-04-25T01:43:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.712+0000 7fa6e3fff640 1 --2- 172.21.15.28:0/4051595785 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5614ac571340 0x5614ac573730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.713+0000 7fa6c37fe640 10 client.19371 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:43:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.713+0000 7fa6e3fff640 1 --2- 172.21.15.28:0/4051595785 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5614ac571340 0x5614ac573730 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=db95ea1530221c72 server_cookie=9767baab58841eec in_seq=0 out_seq=0 2024-04-25T01:43:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.714+0000 7fa6c37fe640 1 -- 172.21.15.28:0/4051595785 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5614ac4c8bb0 con 0x5614ac571340 2024-04-25T01:43:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.714+0000 7fa6c37fe640 10 client.19371 handle_command_reply: tid=0 2024-04-25T01:43:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.714+0000 7fa6ed192740 1 client.19371 shutdown 2024-04-25T01:43:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.714+0000 7fa6ed192740 1 -- 172.21.15.28:0/4051595785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac561570 msgr2=0x5614ac55e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.714+0000 7fa6ed192740 1 --2- 172.21.15.28:0/4051595785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac561570 0x5614ac55e6d0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc030a60 tx=0x7fa6dc02afa0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.714+0000 7fa6ed192740 1 -- 172.21.15.28:0/4051595785 shutdown_connections 2024-04-25T01:43:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.714+0000 7fa6ed192740 1 --2- 172.21.15.28:0/4051595785 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5614ac571340 0x5614ac573730 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.714+0000 7fa6ed192740 1 --2- 172.21.15.28:0/4051595785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac561570 0x5614ac55e6d0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.714+0000 7fa6ed192740 1 --2- 172.21.15.28:0/4051595785 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac556ea0 0x5614ac55e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.714+0000 7fa6c37fe640 0 client.19371 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:43:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.714+0000 7fa6ed192740 1 --2- 172.21.15.28:0/4051595785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac556580 0x5614ac560ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.714+0000 7fa6ed192740 1 -- 172.21.15.28:0/4051595785 >> 172.21.15.28:0/4051595785 conn(0x5614ac5528a0 msgr2=0x5614ac551b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.714+0000 7fa6ed192740 1 -- 172.21.15.28:0/4051595785 shutdown_connections 2024-04-25T01:43:11.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.714+0000 7fa6ed192740 1 -- 172.21.15.28:0/4051595785 wait complete. 2024-04-25T01:43:11.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.714+0000 7fa6ed192740 20 client.19371 trim_cache size 0 max 16384 2024-04-25T01:43:11.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.728+0000 7fa6ed192740 1 Processor -- start 2024-04-25T01:43:11.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.728+0000 7fa6ed192740 1 -- start start 2024-04-25T01:43:11.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.728+0000 7fa6ed192740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac556580 0x5614ac56d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.728+0000 7fa6ed192740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556ea0 0x5614ac5609d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.728+0000 7fa6ed192740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac561570 0x5614ac578cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.728+0000 7fa6ed192740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e41b0e30 con 0x5614ac556ea0 2024-04-25T01:43:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.728+0000 7fa6ed192740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e41b2970 con 0x5614ac556580 2024-04-25T01:43:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.728+0000 7fa6ed192740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e41b1500 con 0x5614ac561570 2024-04-25T01:43:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.728+0000 7fa6e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556ea0 0x5614ac5609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.728+0000 7fa6e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556ea0 0x5614ac5609d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56808/0 (socket says 172.21.15.28:56808) 2024-04-25T01:43:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.728+0000 7fa6e37fe640 1 -- 172.21.15.28:0/3770698819 learned_addr learned my addr 172.21.15.28:0/3770698819 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.728+0000 7fa6e3fff640 1 --2- 172.21.15.28:0/3770698819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac556580 0x5614ac56d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.728+0000 7fa6e893f640 1 --2- 172.21.15.28:0/3770698819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac561570 0x5614ac578cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.728+0000 7fa6e37fe640 1 -- 172.21.15.28:0/3770698819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac561570 msgr2=0x5614ac578cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.728+0000 7fa6e37fe640 1 --2- 172.21.15.28:0/3770698819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac561570 0x5614ac578cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.729+0000 7fa6e37fe640 1 -- 172.21.15.28:0/3770698819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac556580 msgr2=0x5614ac56d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.729+0000 7fa6e37fe640 1 --2- 172.21.15.28:0/3770698819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac556580 0x5614ac56d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.729+0000 7fa6e37fe640 1 -- 172.21.15.28:0/3770698819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614ac2997d0 con 0x5614ac556ea0 2024-04-25T01:43:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.729+0000 7fa6e3fff640 1 --2- 172.21.15.28:0/3770698819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac556580 0x5614ac56d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.729+0000 7fa6e893f640 1 --2- 172.21.15.28:0/3770698819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac561570 0x5614ac578cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:43:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.729+0000 7fa6e37fe640 1 --2- 172.21.15.28:0/3770698819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556ea0 0x5614ac5609d0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fa6d80cc890 tx=0x7fa6d8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa6ce2cdfb8bb58d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.729+0000 7fa6c9ffb640 1 -- 172.21.15.28:0/3770698819 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d80cc030 con 0x5614ac556ea0 2024-04-25T01:43:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.729+0000 7fa6c9ffb640 1 -- 172.21.15.28:0/3770698819 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6d809f9b0 con 0x5614ac556ea0 2024-04-25T01:43:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.729+0000 7fa6c9ffb640 1 -- 172.21.15.28:0/3770698819 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8020050 con 0x5614ac556ea0 2024-04-25T01:43:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.729+0000 7fa6ed192740 1 -- 172.21.15.28:0/3770698819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556ea0 msgr2=0x5614ac5609d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.729+0000 7fa6ed192740 1 --2- 172.21.15.28:0/3770698819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556ea0 0x5614ac5609d0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fa6d80cc890 tx=0x7fa6d8005880 comp rx=0 tx=0).stop 2024-04-25T01:43:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.729+0000 7fa6ed192740 1 -- 172.21.15.28:0/3770698819 shutdown_connections 2024-04-25T01:43:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.729+0000 7fa6ed192740 1 --2- 172.21.15.28:0/3770698819 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac561570 0x5614ac578cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.729+0000 7fa6ed192740 1 --2- 172.21.15.28:0/3770698819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556ea0 0x5614ac5609d0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.729+0000 7fa6ed192740 1 --2- 172.21.15.28:0/3770698819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac556580 0x5614ac56d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.729+0000 7fa6ed192740 1 -- 172.21.15.28:0/3770698819 >> 172.21.15.28:0/3770698819 conn(0x5614ac5528a0 msgr2=0x5614ac55d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.730+0000 7fa6ed192740 1 -- 172.21.15.28:0/3770698819 shutdown_connections 2024-04-25T01:43:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.730+0000 7fa6ed192740 1 -- 172.21.15.28:0/3770698819 wait complete. 2024-04-25T01:43:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.730+0000 7fa6ed192740 1 -- start start 2024-04-25T01:43:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.730+0000 7fa6ed192740 1 Processor -- start 2024-04-25T01:43:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.730+0000 7fa6ed192740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556580 0x5614ac565100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.730+0000 7fa6ed192740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac556ea0 0x5614ac565640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.730+0000 7fa6ed192740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac561570 0x5614ac565b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.730+0000 7fa6ed192740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e41b22a0 con 0x5614ac556580 2024-04-25T01:43:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.730+0000 7fa6ed192740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e41b1bd0 con 0x5614ac561570 2024-04-25T01:43:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.730+0000 7fa6ed192740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e41aee30 con 0x5614ac556ea0 2024-04-25T01:43:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.730+0000 7fa6ed192740 10 client.0 _ll_register_callbacks cb 0x7fa6dc03f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:43:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.730+0000 7fa6e3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556580 0x5614ac565100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.730+0000 7fa6e3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556580 0x5614ac565100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56824/0 (socket says 172.21.15.28:56824) 2024-04-25T01:43:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.730+0000 7fa6e3fff640 1 -- 172.21.15.28:0/4124136131 learned_addr learned my addr 172.21.15.28:0/4124136131 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.731+0000 7fa6e893f640 1 --2- 172.21.15.28:0/4124136131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac561570 0x5614ac565b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.731+0000 7fa6e37fe640 1 --2- 172.21.15.28:0/4124136131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac556ea0 0x5614ac565640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.731+0000 7fa6e3fff640 1 -- 172.21.15.28:0/4124136131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac556ea0 msgr2=0x5614ac565640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.731+0000 7fa6e3fff640 1 --2- 172.21.15.28:0/4124136131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac556ea0 0x5614ac565640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.731+0000 7fa6e3fff640 1 -- 172.21.15.28:0/4124136131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac561570 msgr2=0x5614ac565b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.731+0000 7fa6e3fff640 1 --2- 172.21.15.28:0/4124136131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac561570 0x5614ac565b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.731+0000 7fa6e3fff640 1 -- 172.21.15.28:0/4124136131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614ac5d6570 con 0x5614ac556580 2024-04-25T01:43:11.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.731+0000 7fa6e893f640 1 --2- 172.21.15.28:0/4124136131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac561570 0x5614ac565b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:11.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.731+0000 7fa6e3fff640 1 --2- 172.21.15.28:0/4124136131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556580 0x5614ac565100 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4000c00 tx=0x7fa6d4010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9bf5968bc13b123 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.731+0000 7fa6c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:43:11.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.731+0000 7fa6c37fe640 1 -- 172.21.15.28:0/4124136131 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4023070 con 0x5614ac556580 2024-04-25T01:43:11.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.731+0000 7fa6c37fe640 1 -- 172.21.15.28:0/4124136131 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7fa6d4004460 con 0x5614ac556580 2024-04-25T01:43:11.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.731+0000 7fa6ed192740 1 -- 172.21.15.28:0/4124136131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6d809f9b0 con 0x5614ac556580 2024-04-25T01:43:11.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.731+0000 7fa6c37fe640 1 -- 172.21.15.28:0/4124136131 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa6d402e050 con 0x5614ac556580 2024-04-25T01:43:11.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.731+0000 7fa6ed192740 10 client.19377 fetch_fsmap learned FSMap version 28 2024-04-25T01:43:11.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.731+0000 7fa6ed192740 1 -- 172.21.15.28:0/4124136131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5614ac5d7800 con 0x5614ac556580 2024-04-25T01:43:11.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.731+0000 7fa6c37fe640 1 -- 172.21.15.28:0/4124136131 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6d809f9b0 con 0x5614ac556580 2024-04-25T01:43:11.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.732+0000 7fa6c37fe640 1 -- 172.21.15.28:0/4124136131 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa6d401e390 con 0x5614ac556580 2024-04-25T01:43:11.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.732+0000 7fa6ed192740 10 client.19377 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:43:11.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.732+0000 7fa6ed192740 10 client.19377 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:43:11.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.732+0000 7fa6ed192740 20 client.19377 populate_metadata read hostname 'smithi028' 2024-04-25T01:43:11.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.732+0000 7fa6ed192740 1 --2- 172.21.15.28:0/4124136131 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5614ac5d7fb0 0x5614ac5da3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.732+0000 7fa6ed192740 4 client.19377 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:43:11.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.732+0000 7fa6ed192740 1 -- 172.21.15.28:0/4124136131 --> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5614ac4c2f00 con 0x5614ac5d7fb0 2024-04-25T01:43:11.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.732+0000 7fa6e37fe640 1 --2- 172.21.15.28:0/4124136131 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5614ac5d7fb0 0x5614ac5da3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.732+0000 7fa6e37fe640 1 --2- 172.21.15.28:0/4124136131 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5614ac5d7fb0 0x5614ac5da3d0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c39593e993f0971b server_cookie=8ddae11782767458 in_seq=0 out_seq=0 2024-04-25T01:43:11.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.732+0000 7fa6c37fe640 10 client.19377 ms_handle_connect on v2:172.21.15.130:6838/463478317 2024-04-25T01:43:11.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.732+0000 7fa6c37fe640 1 -- 172.21.15.28:0/4124136131 <== mds.1 v2:172.21.15.130:6838/463478317 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5614ac4c2f00 con 0x5614ac5d7fb0 2024-04-25T01:43:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.732+0000 7fa6c37fe640 10 client.19377 handle_command_reply: tid=0 2024-04-25T01:43:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.732+0000 7fa6ed192740 1 client.19377 shutdown 2024-04-25T01:43:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ed192740 1 -- 172.21.15.28:0/4124136131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556580 msgr2=0x5614ac565100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ed192740 1 --2- 172.21.15.28:0/4124136131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556580 0x5614ac565100 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4000c00 tx=0x7fa6d4010800 comp rx=0 tx=0).stop 2024-04-25T01:43:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ed192740 1 -- 172.21.15.28:0/4124136131 shutdown_connections 2024-04-25T01:43:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ed192740 1 --2- 172.21.15.28:0/4124136131 >> [v2:172.21.15.130:6838/463478317,v1:172.21.15.130:6839/463478317] conn(0x5614ac5d7fb0 0x5614ac5da3d0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ed192740 1 --2- 172.21.15.28:0/4124136131 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5614ac561570 0x5614ac565b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ed192740 1 --2- 172.21.15.28:0/4124136131 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x5614ac556ea0 0x5614ac565640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ed192740 1 --2- 172.21.15.28:0/4124136131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614ac556580 0x5614ac565100 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6c37fe640 0 client.19377 ms_handle_reset on v2:172.21.15.130:6838/463478317 2024-04-25T01:43:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ed192740 1 -- 172.21.15.28:0/4124136131 >> 172.21.15.28:0/4124136131 conn(0x5614ac5528a0 msgr2=0x5614ac55c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ed192740 1 -- 172.21.15.28:0/4124136131 shutdown_connections 2024-04-25T01:43:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ed192740 1 -- 172.21.15.28:0/4124136131 wait complete. 2024-04-25T01:43:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ed192740 20 client.19377 trim_cache size 0 max 16384 2024-04-25T01:43:11.729 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:43:11.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ea3a2640 1 -- 172.21.15.28:0/1910322869 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa6cc0a2d00 msgr2=0x7fa6cc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ea3a2640 1 --2- 172.21.15.28:0/1910322869 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa6cc0a2d00 0x7fa6cc0a51c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc048000 tx=0x7fa6dc04b000 comp rx=0 tx=0).stop 2024-04-25T01:43:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ea3a2640 1 -- 172.21.15.28:0/1910322869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6e4100fd0 msgr2=0x7fa6e410e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ea3a2640 1 --2- 172.21.15.28:0/1910322869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6e4100fd0 0x7fa6e410e4b0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800d950 tx=0x7fa6d800de10 comp rx=0 tx=0).stop 2024-04-25T01:43:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ea3a2640 1 -- 172.21.15.28:0/1910322869 shutdown_connections 2024-04-25T01:43:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ea3a2640 1 --2- 172.21.15.28:0/1910322869 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fa6cc0a2d00 0x7fa6cc0a51c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ea3a2640 1 --2- 172.21.15.28:0/1910322869 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fa6e41066b0 0x7fa6e410e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ea3a2640 1 --2- 172.21.15.28:0/1910322869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6e4100fd0 0x7fa6e410e4b0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ea3a2640 1 --2- 172.21.15.28:0/1910322869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6e4100610 0x7fa6e410df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.733+0000 7fa6ea3a2640 1 -- 172.21.15.28:0/1910322869 >> 172.21.15.28:0/1910322869 conn(0x7fa6e4077740 msgr2=0x7fa6e40ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.734+0000 7fa6ea3a2640 1 -- 172.21.15.28:0/1910322869 shutdown_connections 2024-04-25T01:43:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.734+0000 7fa6ea3a2640 1 -- 172.21.15.28:0/1910322869 wait complete. 2024-04-25T01:43:11.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:43:11.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.849+0000 7f6fddf89640 1 -- 172.21.15.28:0/954474250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd81066b0 msgr2=0x7f6fd8106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.849+0000 7f6fddf89640 1 --2- 172.21.15.28:0/954474250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd81066b0 0x7f6fd8106a90 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009930 tx=0x7f6fc802a420 comp rx=0 tx=0).stop 2024-04-25T01:43:11.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.849+0000 7f6fddf89640 1 -- 172.21.15.28:0/954474250 shutdown_connections 2024-04-25T01:43:11.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.849+0000 7f6fddf89640 1 --2- 172.21.15.28:0/954474250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fd8100fd0 0x7f6fd810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.849+0000 7f6fddf89640 1 --2- 172.21.15.28:0/954474250 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fd8100610 0x7f6fd8100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.849+0000 7f6fddf89640 1 --2- 172.21.15.28:0/954474250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd81066b0 0x7f6fd8106a90 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.849+0000 7f6fddf89640 1 -- 172.21.15.28:0/954474250 >> 172.21.15.28:0/954474250 conn(0x7f6fd8077740 msgr2=0x7f6fd8077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.849+0000 7f6fddf89640 1 -- 172.21.15.28:0/954474250 shutdown_connections 2024-04-25T01:43:11.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.849+0000 7f6fddf89640 1 -- 172.21.15.28:0/954474250 wait complete. 2024-04-25T01:43:11.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.849+0000 7f6fddf89640 1 Processor -- start 2024-04-25T01:43:11.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.849+0000 7f6fddf89640 1 -- start start 2024-04-25T01:43:11.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fddf89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd8100610 0x7f6fd8198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fddf89640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fd8100fd0 0x7f6fd8198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fddf89640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fd81066b0 0x7f6fd819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fddf89640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd8111b30 con 0x7f6fd8100610 2024-04-25T01:43:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fddf89640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd81119b0 con 0x7f6fd8100fd0 2024-04-25T01:43:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fddf89640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd8111cb0 con 0x7f6fd81066b0 2024-04-25T01:43:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fd77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd8100610 0x7f6fd8198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fd77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd8100610 0x7f6fd8198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56842/0 (socket says 172.21.15.28:56842) 2024-04-25T01:43:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fd77fe640 1 -- 172.21.15.28:0/1426961580 learned_addr learned my addr 172.21.15.28:0/1426961580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fd6ffd640 1 --2- 172.21.15.28:0/1426961580 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fd8100fd0 0x7f6fd8198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fd77fe640 1 -- 172.21.15.28:0/1426961580 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fd81066b0 msgr2=0x7f6fd819c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fd77fe640 1 --2- 172.21.15.28:0/1426961580 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fd81066b0 0x7f6fd819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fd77fe640 1 -- 172.21.15.28:0/1426961580 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fd8100fd0 msgr2=0x7f6fd8198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fd77fe640 1 --2- 172.21.15.28:0/1426961580 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fd8100fd0 0x7f6fd8198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fd77fe640 1 -- 172.21.15.28:0/1426961580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd819d020 con 0x7f6fd8100610 2024-04-25T01:43:11.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fd77fe640 1 --2- 172.21.15.28:0/1426961580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd8100610 0x7f6fd8198030 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009230 tx=0x7f6fc80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fd4ff9640 1 -- 172.21.15.28:0/1426961580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc803f030 con 0x7f6fd8100610 2024-04-25T01:43:11.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fd4ff9640 1 -- 172.21.15.28:0/1426961580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fc8003ce0 con 0x7f6fd8100610 2024-04-25T01:43:11.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fddf89640 1 -- 172.21.15.28:0/1426961580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fd819d2b0 con 0x7f6fd8100610 2024-04-25T01:43:11.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.850+0000 7f6fd4ff9640 1 -- 172.21.15.28:0/1426961580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8035070 con 0x7f6fd8100610 2024-04-25T01:43:11.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.851+0000 7f6fddf89640 1 -- 172.21.15.28:0/1426961580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fd81a4ba0 con 0x7f6fd8100610 2024-04-25T01:43:11.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.853+0000 7f6fd4ff9640 1 -- 172.21.15.28:0/1426961580 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6fc8042030 con 0x7f6fd8100610 2024-04-25T01:43:11.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.853+0000 7f6fe0d75740 1 Processor -- start 2024-04-25T01:43:11.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.853+0000 7f6fe0d75740 1 -- start start 2024-04-25T01:43:11.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.853+0000 7f6fe0d75740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a60580 0x558869a60960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.853+0000 7f6fe0d75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a60ea0 0x558869a6b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.853+0000 7f6fe0d75740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a6b570 0x558869a6d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.853+0000 7f6fe0d75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5588699c5670 con 0x558869a60ea0 2024-04-25T01:43:11.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.853+0000 7f6fe0d75740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55886996d970 con 0x558869a60580 2024-04-25T01:43:11.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.853+0000 7f6fe0d75740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x5588699c1630 con 0x558869a6b570 2024-04-25T01:43:11.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.853+0000 7f6fd77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a60ea0 0x558869a6b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.853+0000 7f6fd77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a60ea0 0x558869a6b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56856/0 (socket says 172.21.15.28:56856) 2024-04-25T01:43:11.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.853+0000 7f6fd77fe640 1 -- 172.21.15.28:0/3384820935 learned_addr learned my addr 172.21.15.28:0/3384820935 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.853+0000 7f6fd4ff9640 1 --2- 172.21.15.28:0/1426961580 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6fc00a2d00 0x7f6fc00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.853+0000 7f6fd6ffd640 1 --2- 172.21.15.28:0/3384820935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a6b570 0x558869a6d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.854+0000 7f6fd7fff640 1 --2- 172.21.15.28:0/3384820935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a60580 0x558869a60960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.854+0000 7f6fd7fff640 1 --2- 172.21.15.28:0/1426961580 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6fc00a2d00 0x7f6fc00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.854+0000 7f6fd4ff9640 1 -- 172.21.15.28:0/1426961580 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f6fc80efe60 con 0x7f6fd8100610 2024-04-25T01:43:11.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.854+0000 7f6fd77fe640 1 -- 172.21.15.28:0/3384820935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a6b570 msgr2=0x558869a6d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.854+0000 7f6fd77fe640 1 --2- 172.21.15.28:0/3384820935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a6b570 0x558869a6d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.854+0000 7f6fd77fe640 1 -- 172.21.15.28:0/3384820935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a60580 msgr2=0x558869a60960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.854+0000 7f6fd77fe640 1 --2- 172.21.15.28:0/3384820935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a60580 0x558869a60960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.854+0000 7f6fd77fe640 1 -- 172.21.15.28:0/3384820935 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558869991400 con 0x558869a60ea0 2024-04-25T01:43:11.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.854+0000 7f6fd77fe640 1 --2- 172.21.15.28:0/3384820935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a60ea0 0x558869a6b030 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f6fc80b83f0 tx=0x7f6fc8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11859582ec1f947e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.854+0000 7f6fd7fff640 1 --2- 172.21.15.28:0/1426961580 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6fc00a2d00 0x7f6fc00a51c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0013a00 tx=0x7f6fd0005e00 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.854+0000 7f6fbdffb640 1 -- 172.21.15.28:0/3384820935 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8049070 con 0x558869a60ea0 2024-04-25T01:43:11.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.854+0000 7f6fbdffb640 1 -- 172.21.15.28:0/3384820935 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fc80bc690 con 0x558869a60ea0 2024-04-25T01:43:11.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.854+0000 7f6fbdffb640 1 -- 172.21.15.28:0/3384820935 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc80c0340 con 0x558869a60ea0 2024-04-25T01:43:11.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.855+0000 7f6fe0d75740 1 -- 172.21.15.28:0/3384820935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a60ea0 msgr2=0x558869a6b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.855+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/3384820935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a60ea0 0x558869a6b030 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f6fc80b83f0 tx=0x7f6fc8009260 comp rx=0 tx=0).stop 2024-04-25T01:43:11.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.855+0000 7f6fe0d75740 1 -- 172.21.15.28:0/3384820935 shutdown_connections 2024-04-25T01:43:11.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.855+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/3384820935 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a6b570 0x558869a6d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.855+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/3384820935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a60ea0 0x558869a6b030 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.855+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/3384820935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a60580 0x558869a60960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.855+0000 7f6fe0d75740 1 -- 172.21.15.28:0/3384820935 >> 172.21.15.28:0/3384820935 conn(0x558869a5c8a0 msgr2=0x558869a5cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.855+0000 7f6fe0d75740 1 -- 172.21.15.28:0/3384820935 shutdown_connections 2024-04-25T01:43:11.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.855+0000 7f6fe0d75740 1 -- 172.21.15.28:0/3384820935 wait complete. 2024-04-25T01:43:11.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.855+0000 7f6fe0d75740 1 -- start start 2024-04-25T01:43:11.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.855+0000 7f6fe0d75740 1 Processor -- start 2024-04-25T01:43:11.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fe0d75740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a60580 0x558869a6aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fe0d75740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a60ea0 0x558869a681b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fe0d75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a6b570 0x558869a686f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fe0d75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558869a6e120 con 0x558869a6b570 2024-04-25T01:43:11.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fe0d75740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558869981050 con 0x558869a60580 2024-04-25T01:43:11.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fe0d75740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x558869a6e2a0 con 0x558869a60ea0 2024-04-25T01:43:11.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fe0d75740 10 client.0 _ll_register_callbacks cb 0x5588696d8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:43:11.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a6b570 0x558869a686f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a6b570 0x558869a686f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56864/0 (socket says 172.21.15.28:56864) 2024-04-25T01:43:11.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fd7fff640 1 -- 172.21.15.28:0/4072386317 learned_addr learned my addr 172.21.15.28:0/4072386317 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fd6ffd640 1 --2- 172.21.15.28:0/4072386317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a60ea0 0x558869a681b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fd77fe640 1 --2- 172.21.15.28:0/4072386317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a60580 0x558869a6aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fd7fff640 1 -- 172.21.15.28:0/4072386317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a60ea0 msgr2=0x558869a681b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fd7fff640 1 --2- 172.21.15.28:0/4072386317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a60ea0 0x558869a681b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fd7fff640 1 -- 172.21.15.28:0/4072386317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a60580 msgr2=0x558869a6aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fd7fff640 1 --2- 172.21.15.28:0/4072386317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a60580 0x558869a6aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fd7fff640 1 -- 172.21.15.28:0/4072386317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558869952dc0 con 0x558869a6b570 2024-04-25T01:43:11.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fd77fe640 1 --2- 172.21.15.28:0/4072386317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a60580 0x558869a6aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:43:11.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fd7fff640 1 --2- 172.21.15.28:0/4072386317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a6b570 0x558869a686f0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0011aa0 tx=0x7f6fd0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb013e5cb1486ae5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:43:11.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fb6ffd640 1 -- 172.21.15.28:0/4072386317 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0025070 con 0x558869a6b570 2024-04-25T01:43:11.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fe0d75740 1 -- 172.21.15.28:0/4072386317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fc80bc690 con 0x558869a6b570 2024-04-25T01:43:11.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.856+0000 7f6fb6ffd640 1 -- 172.21.15.28:0/4072386317 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f6fd000caf0 con 0x558869a6b570 2024-04-25T01:43:11.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.857+0000 7f6fb6ffd640 1 -- 172.21.15.28:0/4072386317 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6fd0030050 con 0x558869a6b570 2024-04-25T01:43:11.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.857+0000 7f6fe0d75740 10 client.19389 fetch_fsmap learned FSMap version 28 2024-04-25T01:43:11.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.857+0000 7f6fe0d75740 1 -- 172.21.15.28:0/4072386317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5588696d93f0 con 0x558869a6b570 2024-04-25T01:43:11.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.857+0000 7f6fb6ffd640 1 -- 172.21.15.28:0/4072386317 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fc80bc690 con 0x558869a6b570 2024-04-25T01:43:11.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.857+0000 7f6fb6ffd640 1 -- 172.21.15.28:0/4072386317 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6fd001d040 con 0x558869a6b570 2024-04-25T01:43:11.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.857+0000 7f6fe0d75740 10 client.19389 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:43:11.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.857+0000 7f6fe0d75740 10 client.19389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:43:11.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.857+0000 7f6fe0d75740 20 client.19389 populate_metadata read hostname 'smithi028' 2024-04-25T01:43:11.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.857+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/4072386317 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558869a7b280 0x558869a7d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.857+0000 7f6fe0d75740 4 client.19389 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:43:11.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.857+0000 7f6fe0d75740 1 -- 172.21.15.28:0/4072386317 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588699d2bb0 con 0x558869a7b280 2024-04-25T01:43:11.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.857+0000 7f6fd77fe640 1 --2- 172.21.15.28:0/4072386317 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558869a7b280 0x558869a7d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.858+0000 7f6fb6ffd640 10 client.19389 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:43:11.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.858+0000 7f6fd77fe640 1 --2- 172.21.15.28:0/4072386317 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558869a7b280 0x558869a7d700 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f93832ef4c531981 server_cookie=7b7102f903320dd7 in_seq=0 out_seq=0 2024-04-25T01:43:11.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.858+0000 7f6fb6ffd640 1 -- 172.21.15.28:0/4072386317 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5588699d2bb0 con 0x558869a7b280 2024-04-25T01:43:11.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.858+0000 7f6fb6ffd640 10 client.19389 handle_command_reply: tid=0 2024-04-25T01:43:11.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.858+0000 7f6fe0d75740 1 client.19389 shutdown 2024-04-25T01:43:11.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.859+0000 7f6fe0d75740 1 -- 172.21.15.28:0/4072386317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a6b570 msgr2=0x558869a686f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.859+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/4072386317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a6b570 0x558869a686f0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0011aa0 tx=0x7f6fd0004a40 comp rx=0 tx=0).stop 2024-04-25T01:43:11.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.859+0000 7f6fe0d75740 1 -- 172.21.15.28:0/4072386317 shutdown_connections 2024-04-25T01:43:11.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.859+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/4072386317 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558869a7b280 0x558869a7d700 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.859+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/4072386317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a6b570 0x558869a686f0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.859+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/4072386317 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a60ea0 0x558869a681b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.859+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/4072386317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a60580 0x558869a6aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.859+0000 7f6fb6ffd640 0 client.19389 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:43:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.859+0000 7f6fe0d75740 1 -- 172.21.15.28:0/4072386317 >> 172.21.15.28:0/4072386317 conn(0x558869a5c8a0 msgr2=0x558869a5bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.859+0000 7f6fe0d75740 1 -- 172.21.15.28:0/4072386317 shutdown_connections 2024-04-25T01:43:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.859+0000 7f6fe0d75740 1 -- 172.21.15.28:0/4072386317 wait complete. 2024-04-25T01:43:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.859+0000 7f6fe0d75740 20 client.19389 trim_cache size 0 max 16384 2024-04-25T01:43:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.872+0000 7f6fe0d75740 1 Processor -- start 2024-04-25T01:43:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.872+0000 7f6fe0d75740 1 -- start start 2024-04-25T01:43:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.872+0000 7f6fe0d75740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a60580 0x558869a76ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.872+0000 7f6fe0d75740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a60ea0 0x558869a82fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.872+0000 7f6fe0d75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a6b570 0x558869a83510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.872+0000 7f6fe0d75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd81aed80 con 0x558869a6b570 2024-04-25T01:43:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.872+0000 7f6fe0d75740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd81b08c0 con 0x558869a60ea0 2024-04-25T01:43:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.872+0000 7f6fe0d75740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd81af450 con 0x558869a60580 2024-04-25T01:43:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.872+0000 7f6fd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a6b570 0x558869a83510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.872+0000 7f6fd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a6b570 0x558869a83510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56878/0 (socket says 172.21.15.28:56878) 2024-04-25T01:43:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.872+0000 7f6fd7fff640 1 -- 172.21.15.28:0/294223366 learned_addr learned my addr 172.21.15.28:0/294223366 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.872+0000 7f6fd6ffd640 1 --2- 172.21.15.28:0/294223366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a60ea0 0x558869a82fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.872+0000 7f6fd7fff640 1 -- 172.21.15.28:0/294223366 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a60580 msgr2=0x558869a76ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.872+0000 7f6fd77fe640 1 --2- 172.21.15.28:0/294223366 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a60580 0x558869a76ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.873+0000 7f6fd7fff640 1 --2- 172.21.15.28:0/294223366 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a60580 0x558869a76ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.873+0000 7f6fd7fff640 1 -- 172.21.15.28:0/294223366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a60ea0 msgr2=0x558869a82fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.873+0000 7f6fd7fff640 1 --2- 172.21.15.28:0/294223366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a60ea0 0x558869a82fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.873+0000 7f6fd7fff640 1 -- 172.21.15.28:0/294223366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588697a37d0 con 0x558869a6b570 2024-04-25T01:43:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.873+0000 7f6fd6ffd640 1 --2- 172.21.15.28:0/294223366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a60ea0 0x558869a82fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.873+0000 7f6fd7fff640 1 --2- 172.21.15.28:0/294223366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a6b570 0x558869a83510 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0017040 tx=0x7f6fd0005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e34f9f177370be98 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.873+0000 7f6fbd7fa640 1 -- 172.21.15.28:0/294223366 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd002d030 con 0x558869a6b570 2024-04-25T01:43:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.873+0000 7f6fbd7fa640 1 -- 172.21.15.28:0/294223366 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fd0013770 con 0x558869a6b570 2024-04-25T01:43:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.873+0000 7f6fbd7fa640 1 -- 172.21.15.28:0/294223366 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0025040 con 0x558869a6b570 2024-04-25T01:43:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.873+0000 7f6fe0d75740 1 -- 172.21.15.28:0/294223366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a6b570 msgr2=0x558869a83510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.873+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/294223366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a6b570 0x558869a83510 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0017040 tx=0x7f6fd0005d90 comp rx=0 tx=0).stop 2024-04-25T01:43:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.873+0000 7f6fe0d75740 1 -- 172.21.15.28:0/294223366 shutdown_connections 2024-04-25T01:43:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.874+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/294223366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a6b570 0x558869a83510 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.874+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/294223366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a60ea0 0x558869a82fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.874+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/294223366 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a60580 0x558869a76ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.874+0000 7f6fe0d75740 1 -- 172.21.15.28:0/294223366 >> 172.21.15.28:0/294223366 conn(0x558869a5c8a0 msgr2=0x558869a62560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.874+0000 7f6fe0d75740 1 -- 172.21.15.28:0/294223366 shutdown_connections 2024-04-25T01:43:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.874+0000 7f6fe0d75740 1 -- 172.21.15.28:0/294223366 wait complete. 2024-04-25T01:43:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.874+0000 7f6fe0d75740 1 -- start start 2024-04-25T01:43:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.874+0000 7f6fe0d75740 1 Processor -- start 2024-04-25T01:43:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.874+0000 7f6fe0d75740 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a60580 0x558869a738d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.874+0000 7f6fe0d75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a60ea0 0x558869a64cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.874+0000 7f6fe0d75740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a6b570 0x558869a653c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.874+0000 7f6fe0d75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd81b01f0 con 0x558869a60ea0 2024-04-25T01:43:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.874+0000 7f6fe0d75740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd81afb20 con 0x558869a6b570 2024-04-25T01:43:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.874+0000 7f6fe0d75740 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd81acd80 con 0x558869a60580 2024-04-25T01:43:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.874+0000 7f6fe0d75740 10 client.0 _ll_register_callbacks cb 0x7f6fd00136c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:43:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.875+0000 7f6fd6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a60ea0 0x558869a64cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.875+0000 7f6fd6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a60ea0 0x558869a64cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56884/0 (socket says 172.21.15.28:56884) 2024-04-25T01:43:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.875+0000 7f6fd6ffd640 1 -- 172.21.15.28:0/1091204875 learned_addr learned my addr 172.21.15.28:0/1091204875 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.875+0000 7f6fd77fe640 1 --2- 172.21.15.28:0/1091204875 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a60580 0x558869a738d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.875+0000 7f6fd7fff640 1 --2- 172.21.15.28:0/1091204875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a6b570 0x558869a653c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.875+0000 7f6fd6ffd640 1 -- 172.21.15.28:0/1091204875 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a60580 msgr2=0x558869a738d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.875+0000 7f6fd6ffd640 1 --2- 172.21.15.28:0/1091204875 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a60580 0x558869a738d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.875+0000 7f6fd6ffd640 1 -- 172.21.15.28:0/1091204875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a6b570 msgr2=0x558869a653c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:43:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.875+0000 7f6fd6ffd640 1 --2- 172.21.15.28:0/1091204875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a6b570 0x558869a653c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.875+0000 7f6fd6ffd640 1 -- 172.21.15.28:0/1091204875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558869ae0840 con 0x558869a60ea0 2024-04-25T01:43:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.875+0000 7f6fd6ffd640 1 --2- 172.21.15.28:0/1091204875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a60ea0 0x558869a64cd0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x558869a5da30 tx=0x7f6fcc014850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=898a3307fa1230f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.875+0000 7f6fb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T01:43:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.875+0000 7f6fb6ffd640 1 -- 172.21.15.28:0/1091204875 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc016700 con 0x558869a60ea0 2024-04-25T01:43:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.875+0000 7f6fb6ffd640 1 -- 172.21.15.28:0/1091204875 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f6fcc0168a0 con 0x558869a60ea0 2024-04-25T01:43:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.875+0000 7f6fe0d75740 1 -- 172.21.15.28:0/1091204875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fd0013770 con 0x558869a60ea0 2024-04-25T01:43:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.875+0000 7f6fb6ffd640 1 -- 172.21.15.28:0/1091204875 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6fcc029050 con 0x558869a60ea0 2024-04-25T01:43:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.875+0000 7f6fe0d75740 10 client.19395 fetch_fsmap learned FSMap version 28 2024-04-25T01:43:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.875+0000 7f6fe0d75740 1 -- 172.21.15.28:0/1091204875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558869ae19b0 con 0x558869a60ea0 2024-04-25T01:43:11.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.876+0000 7f6fb6ffd640 1 -- 172.21.15.28:0/1091204875 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fd0013770 con 0x558869a60ea0 2024-04-25T01:43:11.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.876+0000 7f6fb6ffd640 1 -- 172.21.15.28:0/1091204875 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6fcc020c70 con 0x558869a60ea0 2024-04-25T01:43:11.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.876+0000 7f6fe0d75740 10 client.19395 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:43:11.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.876+0000 7f6fe0d75740 10 client.19395 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:43:11.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.876+0000 7f6fe0d75740 20 client.19395 populate_metadata read hostname 'smithi028' 2024-04-25T01:43:11.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.876+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/1091204875 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558869ae2190 0x558869ae45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:11.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.876+0000 7f6fe0d75740 4 client.19395 mds_command: new command op to 24512 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:43:11.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.876+0000 7f6fe0d75740 1 -- 172.21.15.28:0/1091204875 --> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5588699ccf00 con 0x558869ae2190 2024-04-25T01:43:11.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.876+0000 7f6fd77fe640 1 --2- 172.21.15.28:0/1091204875 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558869ae2190 0x558869ae45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:11.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.877+0000 7f6fb6ffd640 10 client.19395 ms_handle_connect on v2:172.21.15.32:6838/2835949714 2024-04-25T01:43:11.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.877+0000 7f6fd77fe640 1 --2- 172.21.15.28:0/1091204875 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558869ae2190 0x558869ae45b0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=439c152769f25e38 server_cookie=5fdae5ad38cf9593 in_seq=0 out_seq=0 2024-04-25T01:43:11.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.877+0000 7f6fb6ffd640 1 -- 172.21.15.28:0/1091204875 <== mds.0 v2:172.21.15.32:6838/2835949714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5588699ccf00 con 0x558869ae2190 2024-04-25T01:43:11.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.877+0000 7f6fb6ffd640 10 client.19395 handle_command_reply: tid=0 2024-04-25T01:43:11.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.877+0000 7f6fe0d75740 1 client.19395 shutdown 2024-04-25T01:43:11.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.877+0000 7f6fe0d75740 1 -- 172.21.15.28:0/1091204875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a60ea0 msgr2=0x558869a64cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.877+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/1091204875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a60ea0 0x558869a64cd0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x558869a5da30 tx=0x7f6fcc014850 comp rx=0 tx=0).stop 2024-04-25T01:43:11.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.877+0000 7f6fe0d75740 1 -- 172.21.15.28:0/1091204875 shutdown_connections 2024-04-25T01:43:11.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.877+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/1091204875 >> [v2:172.21.15.32:6838/2835949714,v1:172.21.15.32:6839/2835949714] conn(0x558869ae2190 0x558869ae45b0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.877+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/1091204875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558869a6b570 0x558869a653c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.877+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/1091204875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558869a60ea0 0x558869a64cd0 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.877+0000 7f6fe0d75740 1 --2- 172.21.15.28:0/1091204875 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x558869a60580 0x558869a738d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.877+0000 7f6fb6ffd640 0 client.19395 ms_handle_reset on v2:172.21.15.32:6838/2835949714 2024-04-25T01:43:11.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.877+0000 7f6fe0d75740 1 -- 172.21.15.28:0/1091204875 >> 172.21.15.28:0/1091204875 conn(0x558869a5c8a0 msgr2=0x558869a62f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.877+0000 7f6fe0d75740 1 -- 172.21.15.28:0/1091204875 shutdown_connections 2024-04-25T01:43:11.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.877+0000 7f6fe0d75740 1 -- 172.21.15.28:0/1091204875 wait complete. 2024-04-25T01:43:11.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.877+0000 7f6fe0d75740 20 client.19395 trim_cache size 0 max 16384 2024-04-25T01:43:11.872 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-25T01:43:11.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.878+0000 7f6fddf89640 1 -- 172.21.15.28:0/1426961580 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6fc00a2d00 msgr2=0x7f6fc00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.878+0000 7f6fddf89640 1 --2- 172.21.15.28:0/1426961580 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6fc00a2d00 0x7f6fc00a51c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0013a00 tx=0x7f6fd0005e00 comp rx=0 tx=0).stop 2024-04-25T01:43:11.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.878+0000 7f6fddf89640 1 -- 172.21.15.28:0/1426961580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd8100610 msgr2=0x7f6fd8198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:11.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.878+0000 7f6fddf89640 1 --2- 172.21.15.28:0/1426961580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd8100610 0x7f6fd8198030 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009230 tx=0x7f6fc80042f0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.878+0000 7f6fddf89640 1 -- 172.21.15.28:0/1426961580 shutdown_connections 2024-04-25T01:43:11.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.878+0000 7f6fddf89640 1 --2- 172.21.15.28:0/1426961580 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6fc00a2d00 0x7f6fc00a51c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.878+0000 7f6fddf89640 1 --2- 172.21.15.28:0/1426961580 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6fd81066b0 0x7f6fd819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.878+0000 7f6fddf89640 1 --2- 172.21.15.28:0/1426961580 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6fd8100fd0 0x7f6fd8198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.878+0000 7f6fddf89640 1 --2- 172.21.15.28:0/1426961580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd8100610 0x7f6fd8198030 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:11.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.878+0000 7f6fddf89640 1 -- 172.21.15.28:0/1426961580 >> 172.21.15.28:0/1426961580 conn(0x7f6fd8077740 msgr2=0x7f6fd80ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:11.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.878+0000 7f6fddf89640 1 -- 172.21.15.28:0/1426961580 shutdown_connections 2024-04-25T01:43:11.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:11.878+0000 7f6fddf89640 1 -- 172.21.15.28:0/1426961580 wait complete. 2024-04-25T01:43:12.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:12 smithi028 ceph-mon[29390]: pgmap v315: 129 pgs: 129 active+clean; 11 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-25T01:43:12.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:12 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2197855521' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:43:12.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:12 smithi130 ceph-mon[32702]: pgmap v315: 129 pgs: 129 active+clean; 11 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-25T01:43:12.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:12 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2197855521' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:43:12.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:12 smithi032 ceph-mon[32461]: pgmap v315: 129 pgs: 129 active+clean; 11 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-04-25T01:43:12.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:12 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2197855521' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:43:12.878 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-04-25T01:43:12.878 INFO:tasks.fwd_scrub:done joining 2024-04-25T01:43:12.879 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-25T01:43:12.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:43:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.015+0000 7f60cd09a640 1 -- 172.21.15.28:0/1176112188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c810bd60 msgr2=0x7f60c810e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.015+0000 7f60cd09a640 1 --2- 172.21.15.28:0/1176112188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c810bd60 0x7f60c810e1d0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f60c0009ca0 tx=0x7f60c002a950 comp rx=0 tx=0).stop 2024-04-25T01:43:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.015+0000 7f60cd09a640 1 -- 172.21.15.28:0/1176112188 shutdown_connections 2024-04-25T01:43:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.015+0000 7f60cd09a640 1 --2- 172.21.15.28:0/1176112188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c810bd60 0x7f60c810e1d0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.015+0000 7f60cd09a640 1 --2- 172.21.15.28:0/1176112188 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60c8103340 0x7f60c810b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.015+0000 7f60cd09a640 1 --2- 172.21.15.28:0/1176112188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c8102a20 0x7f60c8102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.015+0000 7f60cd09a640 1 -- 172.21.15.28:0/1176112188 >> 172.21.15.28:0/1176112188 conn(0x7f60c80777a0 msgr2=0x7f60c8077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:13.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.016+0000 7f60cd09a640 1 -- 172.21.15.28:0/1176112188 shutdown_connections 2024-04-25T01:43:13.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.016+0000 7f60cd09a640 1 -- 172.21.15.28:0/1176112188 wait complete. 2024-04-25T01:43:13.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.016+0000 7f60cd09a640 1 Processor -- start 2024-04-25T01:43:13.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.017+0000 7f60cd09a640 1 -- start start 2024-04-25T01:43:13.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.017+0000 7f60cd09a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60c8102a20 0x7f60c8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:13.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.017+0000 7f60cd09a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c8103340 0x7f60c8198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:13.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.017+0000 7f60cd09a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c810bd60 0x7f60c819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:13.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.017+0000 7f60cd09a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f60c8111b30 con 0x7f60c8103340 2024-04-25T01:43:13.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.017+0000 7f60cd09a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f60c81119b0 con 0x7f60c810bd60 2024-04-25T01:43:13.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.017+0000 7f60cd09a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f60c8111cb0 con 0x7f60c8102a20 2024-04-25T01:43:13.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.017+0000 7f60c659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c8103340 0x7f60c8198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:13.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.017+0000 7f60c659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c8103340 0x7f60c8198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56902/0 (socket says 172.21.15.28:56902) 2024-04-25T01:43:13.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.017+0000 7f60c659c640 1 -- 172.21.15.28:0/2089971646 learned_addr learned my addr 172.21.15.28:0/2089971646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:13.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.017+0000 7f60c6d9d640 1 --2- 172.21.15.28:0/2089971646 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60c8102a20 0x7f60c8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:13.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.017+0000 7f60c759e640 1 --2- 172.21.15.28:0/2089971646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c810bd60 0x7f60c819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:13.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.017+0000 7f60c659c640 1 -- 172.21.15.28:0/2089971646 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60c8102a20 msgr2=0x7f60c8197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:13.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.017+0000 7f60c659c640 1 --2- 172.21.15.28:0/2089971646 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60c8102a20 0x7f60c8197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.017+0000 7f60c659c640 1 -- 172.21.15.28:0/2089971646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c810bd60 msgr2=0x7f60c819c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:13.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.017+0000 7f60c659c640 1 --2- 172.21.15.28:0/2089971646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c810bd60 0x7f60c819c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.017+0000 7f60c659c640 1 -- 172.21.15.28:0/2089971646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c819cf70 con 0x7f60c8103340 2024-04-25T01:43:13.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.017+0000 7f60c759e640 1 --2- 172.21.15.28:0/2089971646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c810bd60 0x7f60c819c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:13.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.017+0000 7f60c659c640 1 --2- 172.21.15.28:0/2089971646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c8103340 0x7f60c8198510 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f60bc00d770 tx=0x7f60bc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:13.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.018+0000 7f60affff640 1 -- 172.21.15.28:0/2089971646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc017070 con 0x7f60c8103340 2024-04-25T01:43:13.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.018+0000 7f60cd09a640 1 -- 172.21.15.28:0/2089971646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60c819d260 con 0x7f60c8103340 2024-04-25T01:43:13.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.018+0000 7f60affff640 1 -- 172.21.15.28:0/2089971646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60bc005500 con 0x7f60c8103340 2024-04-25T01:43:13.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.018+0000 7f60affff640 1 -- 172.21.15.28:0/2089971646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc004e00 con 0x7f60c8103340 2024-04-25T01:43:13.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.018+0000 7f60cd09a640 1 -- 172.21.15.28:0/2089971646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60c81a4af0 con 0x7f60c8103340 2024-04-25T01:43:13.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.019+0000 7f60cd09a640 1 -- 172.21.15.28:0/2089971646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60c80689b0 con 0x7f60c8103340 2024-04-25T01:43:13.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.020+0000 7f60affff640 1 -- 172.21.15.28:0/2089971646 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f60bc004fa0 con 0x7f60c8103340 2024-04-25T01:43:13.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.021+0000 7f60affff640 1 --2- 172.21.15.28:0/2089971646 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f60b00a2d00 0x7f60b00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:13.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.021+0000 7f60c6d9d640 1 --2- 172.21.15.28:0/2089971646 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f60b00a2d00 0x7f60b00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:13.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.021+0000 7f60affff640 1 -- 172.21.15.28:0/2089971646 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f60bc0cf6a0 con 0x7f60c8103340 2024-04-25T01:43:13.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.021+0000 7f60c6d9d640 1 --2- 172.21.15.28:0/2089971646 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f60b00a2d00 0x7f60b00a51c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f60b800bbe0 tx=0x7f60b800b390 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:13.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.024+0000 7f60affff640 1 -- 172.21.15.28:0/2089971646 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f60bc09be70 con 0x7f60c8103340 2024-04-25T01:43:13.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.211+0000 7f60cd09a640 1 -- 172.21.15.28:0/2089971646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f60c81a4d20 con 0x7f60c8103340 2024-04-25T01:43:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.212+0000 7f60affff640 1 -- 172.21.15.28:0/2089971646 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) ==== 74+0+25262 (secure 0 0 0) 0x7f60bc09b770 con 0x7f60c8103340 2024-04-25T01:43:13.202 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:43:13.202 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":101,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:43:09.043478+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/1149553567":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/500268183":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/3499593842":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/3630315328":"2024-04-26T01:33:00.617174+0000","172.21.15.28:6800/1149553567":"2024-04-26T01:33:00.617174+0000","172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3812371257":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:43:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.214+0000 7f60cd09a640 1 -- 172.21.15.28:0/2089971646 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f60b00a2d00 msgr2=0x7f60b00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.214+0000 7f60cd09a640 1 --2- 172.21.15.28:0/2089971646 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f60b00a2d00 0x7f60b00a51c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f60b800bbe0 tx=0x7f60b800b390 comp rx=0 tx=0).stop 2024-04-25T01:43:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.214+0000 7f60cd09a640 1 -- 172.21.15.28:0/2089971646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c8103340 msgr2=0x7f60c8198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.214+0000 7f60cd09a640 1 --2- 172.21.15.28:0/2089971646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c8103340 0x7f60c8198510 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f60bc00d770 tx=0x7f60bc00dc30 comp rx=0 tx=0).stop 2024-04-25T01:43:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.214+0000 7f60cd09a640 1 -- 172.21.15.28:0/2089971646 shutdown_connections 2024-04-25T01:43:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.214+0000 7f60cd09a640 1 --2- 172.21.15.28:0/2089971646 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f60b00a2d00 0x7f60b00a51c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.214+0000 7f60cd09a640 1 --2- 172.21.15.28:0/2089971646 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f60c810bd60 0x7f60c819c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.214+0000 7f60cd09a640 1 --2- 172.21.15.28:0/2089971646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60c8103340 0x7f60c8198510 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.214+0000 7f60cd09a640 1 --2- 172.21.15.28:0/2089971646 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f60c8102a20 0x7f60c8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.214+0000 7f60cd09a640 1 -- 172.21.15.28:0/2089971646 >> 172.21.15.28:0/2089971646 conn(0x7f60c80777a0 msgr2=0x7f60c80fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.215+0000 7f60cd09a640 1 -- 172.21.15.28:0/2089971646 shutdown_connections 2024-04-25T01:43:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.215+0000 7f60cd09a640 1 -- 172.21.15.28:0/2089971646 wait complete. 2024-04-25T01:43:13.215 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:25:48.767414+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:30:32.282430+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:30:33.127049+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:30:36.796132+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:43:13.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:43:13.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.331+0000 7f6c3e4bd640 1 -- 172.21.15.28:0/4106021257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c38107730 msgr2=0x7f6c38107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:13.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.331+0000 7f6c3e4bd640 1 --2- 172.21.15.28:0/4106021257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c38107730 0x7f6c38107b10 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f6c28009930 tx=0x7f6c2802a420 comp rx=0 tx=0).stop 2024-04-25T01:43:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.331+0000 7f6c3e4bd640 1 -- 172.21.15.28:0/4106021257 shutdown_connections 2024-04-25T01:43:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.331+0000 7f6c3e4bd640 1 --2- 172.21.15.28:0/4106021257 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c3806a780 0x7f6c3810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.331+0000 7f6c3e4bd640 1 --2- 172.21.15.28:0/4106021257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c38069dc0 0x7f6c3806a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.331+0000 7f6c3e4bd640 1 --2- 172.21.15.28:0/4106021257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c38107730 0x7f6c38107b10 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.331+0000 7f6c3e4bd640 1 -- 172.21.15.28:0/4106021257 >> 172.21.15.28:0/4106021257 conn(0x7f6c380776a0 msgr2=0x7f6c38077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.331+0000 7f6c3e4bd640 1 -- 172.21.15.28:0/4106021257 shutdown_connections 2024-04-25T01:43:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.331+0000 7f6c3e4bd640 1 -- 172.21.15.28:0/4106021257 wait complete. 2024-04-25T01:43:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.331+0000 7f6c3e4bd640 1 Processor -- start 2024-04-25T01:43:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.331+0000 7f6c3e4bd640 1 -- start start 2024-04-25T01:43:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.332+0000 7f6c3e4bd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c38069dc0 0x7f6c38197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.332+0000 7f6c3e4bd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c3806a780 0x7f6c38198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.332+0000 7f6c3e4bd640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c38107730 0x7f6c3819c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.332+0000 7f6c3e4bd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6c38111b40 con 0x7f6c3806a780 2024-04-25T01:43:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.332+0000 7f6c3e4bd640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6c381119c0 con 0x7f6c38069dc0 2024-04-25T01:43:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.332+0000 7f6c3e4bd640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f6c38111cc0 con 0x7f6c38107730 2024-04-25T01:43:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.332+0000 7f6c377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c3806a780 0x7f6c38198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.332+0000 7f6c377fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c3806a780 0x7f6c38198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56932/0 (socket says 172.21.15.28:56932) 2024-04-25T01:43:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.332+0000 7f6c377fe640 1 -- 172.21.15.28:0/3796485803 learned_addr learned my addr 172.21.15.28:0/3796485803 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.332+0000 7f6c37fff640 1 --2- 172.21.15.28:0/3796485803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c38069dc0 0x7f6c38197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.332+0000 7f6c3ca5a640 1 --2- 172.21.15.28:0/3796485803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c38107730 0x7f6c3819c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.332+0000 7f6c377fe640 1 -- 172.21.15.28:0/3796485803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c38107730 msgr2=0x7f6c3819c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.332+0000 7f6c377fe640 1 --2- 172.21.15.28:0/3796485803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c38107730 0x7f6c3819c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.332+0000 7f6c377fe640 1 -- 172.21.15.28:0/3796485803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c38069dc0 msgr2=0x7f6c38197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.332+0000 7f6c377fe640 1 --2- 172.21.15.28:0/3796485803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c38069dc0 0x7f6c38197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.332+0000 7f6c377fe640 1 -- 172.21.15.28:0/3796485803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c3819cfb0 con 0x7f6c3806a780 2024-04-25T01:43:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.332+0000 7f6c3ca5a640 1 --2- 172.21.15.28:0/3796485803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c38107730 0x7f6c3819c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:43:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.332+0000 7f6c37fff640 1 --2- 172.21.15.28:0/3796485803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c38069dc0 0x7f6c38197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.332+0000 7f6c377fe640 1 --2- 172.21.15.28:0/3796485803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c3806a780 0x7f6c38198500 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c00c920 tx=0x7f6c2c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.333+0000 7f6c357fa640 1 -- 172.21.15.28:0/3796485803 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c007c70 con 0x7f6c3806a780 2024-04-25T01:43:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.333+0000 7f6c3e4bd640 1 -- 172.21.15.28:0/3796485803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c3819d2a0 con 0x7f6c3806a780 2024-04-25T01:43:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.333+0000 7f6c357fa640 1 -- 172.21.15.28:0/3796485803 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c2c007e10 con 0x7f6c3806a780 2024-04-25T01:43:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.333+0000 7f6c357fa640 1 -- 172.21.15.28:0/3796485803 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c018890 con 0x7f6c3806a780 2024-04-25T01:43:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.333+0000 7f6c3e4bd640 1 -- 172.21.15.28:0/3796485803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c381a4b30 con 0x7f6c3806a780 2024-04-25T01:43:13.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.334+0000 7f6c3e4bd640 1 -- 172.21.15.28:0/3796485803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c380689b0 con 0x7f6c3806a780 2024-04-25T01:43:13.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.335+0000 7f6c357fa640 1 -- 172.21.15.28:0/3796485803 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f6c2c018a50 con 0x7f6c3806a780 2024-04-25T01:43:13.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.335+0000 7f6c357fa640 1 --2- 172.21.15.28:0/3796485803 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6c200a2d00 0x7f6c200a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:13.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.336+0000 7f6c37fff640 1 --2- 172.21.15.28:0/3796485803 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6c200a2d00 0x7f6c200a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:13.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.336+0000 7f6c357fa640 1 -- 172.21.15.28:0/3796485803 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f6c2c0cfc70 con 0x7f6c3806a780 2024-04-25T01:43:13.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.336+0000 7f6c37fff640 1 --2- 172.21.15.28:0/3796485803 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6c200a2d00 0x7f6c200a51c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6c28000c00 tx=0x7f6c28005a20 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:13.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.339+0000 7f6c357fa640 1 -- 172.21.15.28:0/3796485803 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6c2c014030 con 0x7f6c3806a780 2024-04-25T01:43:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.525+0000 7f6c3e4bd640 1 -- 172.21.15.28:0/3796485803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6c3802d170 con 0x7f6c3806a780 2024-04-25T01:43:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.526+0000 7f6c357fa640 1 -- 172.21.15.28:0/3796485803 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) ==== 93+0+10 (secure 0 0 0) 0x7f6c2c09c530 con 0x7f6c3806a780 2024-04-25T01:43:13.515 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T01:43:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.528+0000 7f6c3e4bd640 1 -- 172.21.15.28:0/3796485803 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6c200a2d00 msgr2=0x7f6c200a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.528+0000 7f6c3e4bd640 1 --2- 172.21.15.28:0/3796485803 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6c200a2d00 0x7f6c200a51c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6c28000c00 tx=0x7f6c28005a20 comp rx=0 tx=0).stop 2024-04-25T01:43:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.528+0000 7f6c3e4bd640 1 -- 172.21.15.28:0/3796485803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c3806a780 msgr2=0x7f6c38198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.528+0000 7f6c3e4bd640 1 --2- 172.21.15.28:0/3796485803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c3806a780 0x7f6c38198500 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c00c920 tx=0x7f6c2c00cde0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.528+0000 7f6c3e4bd640 1 -- 172.21.15.28:0/3796485803 shutdown_connections 2024-04-25T01:43:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.528+0000 7f6c3e4bd640 1 --2- 172.21.15.28:0/3796485803 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f6c200a2d00 0x7f6c200a51c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.528+0000 7f6c3e4bd640 1 --2- 172.21.15.28:0/3796485803 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f6c38107730 0x7f6c3819c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.528+0000 7f6c3e4bd640 1 --2- 172.21.15.28:0/3796485803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c3806a780 0x7f6c38198500 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.528+0000 7f6c3e4bd640 1 --2- 172.21.15.28:0/3796485803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c38069dc0 0x7f6c38197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.528+0000 7f6c3e4bd640 1 -- 172.21.15.28:0/3796485803 >> 172.21.15.28:0/3796485803 conn(0x7f6c380776a0 msgr2=0x7f6c380fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.528+0000 7f6c3e4bd640 1 -- 172.21.15.28:0/3796485803 shutdown_connections 2024-04-25T01:43:13.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.528+0000 7f6c3e4bd640 1 -- 172.21.15.28:0/3796485803 wait complete. 2024-04-25T01:43:13.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:43:13.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.644+0000 7ff913b5a640 1 -- 172.21.15.28:0/1183123922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c1066b0 msgr2=0x7ff90c106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:13.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.644+0000 7ff913b5a640 1 --2- 172.21.15.28:0/1183123922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c1066b0 0x7ff90c106a90 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7ff908009cd0 tx=0x7ff90802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.644+0000 7ff913b5a640 1 -- 172.21.15.28:0/1183123922 shutdown_connections 2024-04-25T01:43:13.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.644+0000 7ff913b5a640 1 --2- 172.21.15.28:0/1183123922 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff90c100fd0 0x7ff90c10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.644+0000 7ff913b5a640 1 --2- 172.21.15.28:0/1183123922 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff90c100610 0x7ff90c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.644+0000 7ff913b5a640 1 --2- 172.21.15.28:0/1183123922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c1066b0 0x7ff90c106a90 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.644+0000 7ff913b5a640 1 -- 172.21.15.28:0/1183123922 >> 172.21.15.28:0/1183123922 conn(0x7ff90c077740 msgr2=0x7ff90c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:13.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.644+0000 7ff913b5a640 1 -- 172.21.15.28:0/1183123922 shutdown_connections 2024-04-25T01:43:13.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.644+0000 7ff913b5a640 1 -- 172.21.15.28:0/1183123922 wait complete. 2024-04-25T01:43:13.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.644+0000 7ff913b5a640 1 Processor -- start 2024-04-25T01:43:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.644+0000 7ff913b5a640 1 -- start start 2024-04-25T01:43:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff913b5a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c100610 0x7ff90c198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff913b5a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff90c100fd0 0x7ff90c199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff913b5a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff90c1066b0 0x7ff90c1a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff913b5a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c110a30 con 0x7ff90c100610 2024-04-25T01:43:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff913b5a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c1108b0 con 0x7ff90c1066b0 2024-04-25T01:43:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff913b5a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c110bb0 con 0x7ff90c100fd0 2024-04-25T01:43:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff9118f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c100610 0x7ff90c198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff9118f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c100610 0x7ff90c198e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56952/0 (socket says 172.21.15.28:56952) 2024-04-25T01:43:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff9118f6640 1 -- 172.21.15.28:0/3652841459 learned_addr learned my addr 172.21.15.28:0/3652841459 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff9110f5640 1 --2- 172.21.15.28:0/3652841459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff90c100fd0 0x7ff90c199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff9120f7640 1 --2- 172.21.15.28:0/3652841459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff90c1066b0 0x7ff90c1a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff9118f6640 1 -- 172.21.15.28:0/3652841459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff90c100fd0 msgr2=0x7ff90c199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff9118f6640 1 --2- 172.21.15.28:0/3652841459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff90c100fd0 0x7ff90c199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff9118f6640 1 -- 172.21.15.28:0/3652841459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff90c1066b0 msgr2=0x7ff90c1a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff9118f6640 1 --2- 172.21.15.28:0/3652841459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff90c1066b0 0x7ff90c1a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff9118f6640 1 -- 172.21.15.28:0/3652841459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff90c1a43f0 con 0x7ff90c100610 2024-04-25T01:43:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff9120f7640 1 --2- 172.21.15.28:0/3652841459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff90c1066b0 0x7ff90c1a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:43:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff9118f6640 1 --2- 172.21.15.28:0/3652841459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c100610 0x7ff90c198e30 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7ff90802ae00 tx=0x7ff908031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff902ffd640 1 -- 172.21.15.28:0/3652841459 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff908003e00 con 0x7ff90c100610 2024-04-25T01:43:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff902ffd640 1 -- 172.21.15.28:0/3652841459 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff908004520 con 0x7ff90c100610 2024-04-25T01:43:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff913b5a640 1 -- 172.21.15.28:0/3652841459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff90c1a45c0 con 0x7ff90c100610 2024-04-25T01:43:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff902ffd640 1 -- 172.21.15.28:0/3652841459 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff908030de0 con 0x7ff90c100610 2024-04-25T01:43:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.645+0000 7ff913b5a640 1 -- 172.21.15.28:0/3652841459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff90c1a4a20 con 0x7ff90c100610 2024-04-25T01:43:13.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.646+0000 7ff913b5a640 1 -- 172.21.15.28:0/3652841459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff90c068560 con 0x7ff90c100610 2024-04-25T01:43:13.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.647+0000 7ff902ffd640 1 -- 172.21.15.28:0/3652841459 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff9080046d0 con 0x7ff90c100610 2024-04-25T01:43:13.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.648+0000 7ff902ffd640 1 --2- 172.21.15.28:0/3652841459 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8f80a2d00 0x7ff8f80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:13.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.648+0000 7ff9110f5640 1 --2- 172.21.15.28:0/3652841459 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8f80a2d00 0x7ff8f80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:13.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.648+0000 7ff902ffd640 1 -- 172.21.15.28:0/3652841459 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7ff90802e030 con 0x7ff90c100610 2024-04-25T01:43:13.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.648+0000 7ff9110f5640 1 --2- 172.21.15.28:0/3652841459 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8f80a2d00 0x7ff8f80a51c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff90c0693d0 tx=0x7ff8fc005f90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:13.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.652+0000 7ff902ffd640 1 -- 172.21.15.28:0/3652841459 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff90804b060 con 0x7ff90c100610 2024-04-25T01:43:13.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.837+0000 7ff913b5a640 1 -- 172.21.15.28:0/3652841459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff90c02d170 con 0x7ff90c100610 2024-04-25T01:43:13.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.837+0000 7ff902ffd640 1 -- 172.21.15.28:0/3652841459 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v101) ==== 104+0+11 (secure 0 0 0) 0x7ff9080bd580 con 0x7ff90c100610 2024-04-25T01:43:13.827 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:43:13.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.839+0000 7ff913b5a640 1 -- 172.21.15.28:0/3652841459 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8f80a2d00 msgr2=0x7ff8f80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:13.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.839+0000 7ff913b5a640 1 --2- 172.21.15.28:0/3652841459 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8f80a2d00 0x7ff8f80a51c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff90c0693d0 tx=0x7ff8fc005f90 comp rx=0 tx=0).stop 2024-04-25T01:43:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.839+0000 7ff913b5a640 1 -- 172.21.15.28:0/3652841459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c100610 msgr2=0x7ff90c198e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.839+0000 7ff913b5a640 1 --2- 172.21.15.28:0/3652841459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c100610 0x7ff90c198e30 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7ff90802ae00 tx=0x7ff908031e60 comp rx=0 tx=0).stop 2024-04-25T01:43:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.839+0000 7ff913b5a640 1 -- 172.21.15.28:0/3652841459 shutdown_connections 2024-04-25T01:43:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.839+0000 7ff913b5a640 1 --2- 172.21.15.28:0/3652841459 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff8f80a2d00 0x7ff8f80a51c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.839+0000 7ff913b5a640 1 --2- 172.21.15.28:0/3652841459 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff90c1066b0 0x7ff90c1a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.839+0000 7ff913b5a640 1 --2- 172.21.15.28:0/3652841459 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff90c100fd0 0x7ff90c199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.839+0000 7ff913b5a640 1 --2- 172.21.15.28:0/3652841459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff90c100610 0x7ff90c198e30 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.839+0000 7ff913b5a640 1 -- 172.21.15.28:0/3652841459 >> 172.21.15.28:0/3652841459 conn(0x7ff90c077740 msgr2=0x7ff90c0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.839+0000 7ff913b5a640 1 -- 172.21.15.28:0/3652841459 shutdown_connections 2024-04-25T01:43:13.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.839+0000 7ff913b5a640 1 -- 172.21.15.28:0/3652841459 wait complete. 2024-04-25T01:43:13.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:43:13.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.955+0000 7f3ed0522640 1 -- 172.21.15.28:0/516577850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec81020d0 msgr2=0x7f3ec810e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:13.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.955+0000 7f3ed0522640 1 --2- 172.21.15.28:0/516577850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec81020d0 0x7f3ec810e5e0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f3ec0009cd0 tx=0x7f3ec002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.955+0000 7f3ed0522640 1 -- 172.21.15.28:0/516577850 shutdown_connections 2024-04-25T01:43:13.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.955+0000 7f3ed0522640 1 --2- 172.21.15.28:0/516577850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec81020d0 0x7f3ec810e5e0 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.955+0000 7f3ed0522640 1 --2- 172.21.15.28:0/516577850 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ec8101710 0x7f3ec8101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.955+0000 7f3ed0522640 1 --2- 172.21.15.28:0/516577850 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8107720 0x7f3ec8107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.955+0000 7f3ed0522640 1 -- 172.21.15.28:0/516577850 >> 172.21.15.28:0/516577850 conn(0x7f3ec80fd180 msgr2=0x7f3ec80ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:13.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.955+0000 7f3ed0522640 1 -- 172.21.15.28:0/516577850 shutdown_connections 2024-04-25T01:43:13.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.955+0000 7f3ed0522640 1 -- 172.21.15.28:0/516577850 wait complete. 2024-04-25T01:43:13.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.955+0000 7f3ed0522640 1 Processor -- start 2024-04-25T01:43:13.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3ed0522640 1 -- start start 2024-04-25T01:43:13.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3ed0522640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ec8101710 0x7f3ec8198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:13.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3ed0522640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec81020d0 0x7f3ec81985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:13.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3ed0522640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8107720 0x7f3ec819c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:13.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3ed0522640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec8111b30 con 0x7f3ec81020d0 2024-04-25T01:43:13.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3ed0522640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec81119b0 con 0x7f3ec8107720 2024-04-25T01:43:13.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3ed0522640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec8111cb0 con 0x7f3ec8101710 2024-04-25T01:43:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3ecdabd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec81020d0 0x7f3ec81985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3ecdabd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec81020d0 0x7f3ec81985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56974/0 (socket says 172.21.15.28:56974) 2024-04-25T01:43:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3ecdabd640 1 -- 172.21.15.28:0/272395102 learned_addr learned my addr 172.21.15.28:0/272395102 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3eceabf640 1 --2- 172.21.15.28:0/272395102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8107720 0x7f3ec819c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3ece2be640 1 --2- 172.21.15.28:0/272395102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ec8101710 0x7f3ec8198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3ecdabd640 1 -- 172.21.15.28:0/272395102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ec8101710 msgr2=0x7f3ec8198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3ecdabd640 1 --2- 172.21.15.28:0/272395102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ec8101710 0x7f3ec8198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3ecdabd640 1 -- 172.21.15.28:0/272395102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8107720 msgr2=0x7f3ec819c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3ecdabd640 1 --2- 172.21.15.28:0/272395102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8107720 0x7f3ec819c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3ecdabd640 1 -- 172.21.15.28:0/272395102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec819d060 con 0x7f3ec81020d0 2024-04-25T01:43:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3ece2be640 1 --2- 172.21.15.28:0/272395102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ec8101710 0x7f3ec8198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:43:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3eceabf640 1 --2- 172.21.15.28:0/272395102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8107720 0x7f3ec819c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:43:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.956+0000 7f3ecdabd640 1 --2- 172.21.15.28:0/272395102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec81020d0 0x7f3ec81985b0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f3eb800d770 tx=0x7f3eb800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:13.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.957+0000 7f3ebf7fe640 1 -- 172.21.15.28:0/272395102 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8017070 con 0x7f3ec81020d0 2024-04-25T01:43:13.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.957+0000 7f3ebf7fe640 1 -- 172.21.15.28:0/272395102 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3eb8005500 con 0x7f3ec81020d0 2024-04-25T01:43:13.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.957+0000 7f3ebf7fe640 1 -- 172.21.15.28:0/272395102 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8004e00 con 0x7f3ec81020d0 2024-04-25T01:43:13.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.957+0000 7f3ed0522640 1 -- 172.21.15.28:0/272395102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ec819d350 con 0x7f3ec81020d0 2024-04-25T01:43:13.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.957+0000 7f3ed0522640 1 -- 172.21.15.28:0/272395102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ec81a4be0 con 0x7f3ec81020d0 2024-04-25T01:43:13.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.957+0000 7f3ed0522640 1 -- 172.21.15.28:0/272395102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ec80689b0 con 0x7f3ec81020d0 2024-04-25T01:43:13.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.959+0000 7f3ebf7fe640 1 -- 172.21.15.28:0/272395102 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f3eb8004fa0 con 0x7f3ec81020d0 2024-04-25T01:43:13.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.959+0000 7f3ebf7fe640 1 --2- 172.21.15.28:0/272395102 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3eb40a2d00 0x7f3eb40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:13.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.959+0000 7f3ece2be640 1 --2- 172.21.15.28:0/272395102 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3eb40a2d00 0x7f3eb40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:13.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.960+0000 7f3ebf7fe640 1 -- 172.21.15.28:0/272395102 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f3eb80cf780 con 0x7f3ec81020d0 2024-04-25T01:43:13.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.960+0000 7f3ece2be640 1 --2- 172.21.15.28:0/272395102 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3eb40a2d00 0x7f3eb40a51c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3ec40058c0 tx=0x7f3ec4005cb0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:13.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:13.963+0000 7f3ebf7fe640 1 -- 172.21.15.28:0/272395102 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3eb809bf50 con 0x7f3ec81020d0 2024-04-25T01:43:14.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.148+0000 7f3ed0522640 1 -- 172.21.15.28:0/272395102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3ec8109cd0 con 0x7f3ec81020d0 2024-04-25T01:43:14.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.148+0000 7f3ebf7fe640 1 -- 172.21.15.28:0/272395102 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v101) ==== 100+0+11 (secure 0 0 0) 0x7f3eb809b850 con 0x7f3ec81020d0 2024-04-25T01:43:14.138 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T01:43:14.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.150+0000 7f3ed0522640 1 -- 172.21.15.28:0/272395102 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3eb40a2d00 msgr2=0x7f3eb40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.150+0000 7f3ed0522640 1 --2- 172.21.15.28:0/272395102 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3eb40a2d00 0x7f3eb40a51c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3ec40058c0 tx=0x7f3ec4005cb0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.150+0000 7f3ed0522640 1 -- 172.21.15.28:0/272395102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec81020d0 msgr2=0x7f3ec81985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.150+0000 7f3ed0522640 1 --2- 172.21.15.28:0/272395102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec81020d0 0x7f3ec81985b0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f3eb800d770 tx=0x7f3eb800dc30 comp rx=0 tx=0).stop 2024-04-25T01:43:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.150+0000 7f3ed0522640 1 -- 172.21.15.28:0/272395102 shutdown_connections 2024-04-25T01:43:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.150+0000 7f3ed0522640 1 --2- 172.21.15.28:0/272395102 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f3eb40a2d00 0x7f3eb40a51c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.150+0000 7f3ed0522640 1 --2- 172.21.15.28:0/272395102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8107720 0x7f3ec819c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.150+0000 7f3ed0522640 1 --2- 172.21.15.28:0/272395102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ec81020d0 0x7f3ec81985b0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.150+0000 7f3ed0522640 1 --2- 172.21.15.28:0/272395102 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f3ec8101710 0x7f3ec8198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.150+0000 7f3ed0522640 1 -- 172.21.15.28:0/272395102 >> 172.21.15.28:0/272395102 conn(0x7f3ec80fd180 msgr2=0x7f3ec80fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.150+0000 7f3ed0522640 1 -- 172.21.15.28:0/272395102 shutdown_connections 2024-04-25T01:43:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.150+0000 7f3ed0522640 1 -- 172.21.15.28:0/272395102 wait complete. 2024-04-25T01:43:14.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:14 smithi028 ceph-mon[29390]: pgmap v316: 129 pgs: 129 active+clean; 11 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-04-25T01:43:14.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:14 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2089971646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:43:14.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:14 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3796485803' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:43:14.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:14 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3652841459' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:43:14.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:43:14.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.269+0000 7f4f53e95640 1 -- 172.21.15.28:0/3005717988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c100610 msgr2=0x7f4f4c100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:14.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.269+0000 7f4f53e95640 1 --2- 172.21.15.28:0/3005717988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c100610 0x7f4f4c100a90 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f4f40009930 tx=0x7f4f4002a420 comp rx=0 tx=0).stop 2024-04-25T01:43:14.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.269+0000 7f4f53e95640 1 -- 172.21.15.28:0/3005717988 shutdown_connections 2024-04-25T01:43:14.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.269+0000 7f4f53e95640 1 --2- 172.21.15.28:0/3005717988 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f4c100fd0 0x7f4f4c10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.269+0000 7f4f53e95640 1 --2- 172.21.15.28:0/3005717988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c100610 0x7f4f4c100a90 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.269+0000 7f4f53e95640 1 --2- 172.21.15.28:0/3005717988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f4c1066b0 0x7f4f4c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.269+0000 7f4f53e95640 1 -- 172.21.15.28:0/3005717988 >> 172.21.15.28:0/3005717988 conn(0x7f4f4c077740 msgr2=0x7f4f4c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:14.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.269+0000 7f4f53e95640 1 -- 172.21.15.28:0/3005717988 shutdown_connections 2024-04-25T01:43:14.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.269+0000 7f4f53e95640 1 -- 172.21.15.28:0/3005717988 wait complete. 2024-04-25T01:43:14.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.270+0000 7f4f53e95640 1 Processor -- start 2024-04-25T01:43:14.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.270+0000 7f4f53e95640 1 -- start start 2024-04-25T01:43:14.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.270+0000 7f4f53e95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c100610 0x7f4f4c1022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:14.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.270+0000 7f4f53e95640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f4c100fd0 0x7f4f4c1027f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:14.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.270+0000 7f4f53e95640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f4c1066b0 0x7f4f4c102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:14.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.270+0000 7f4f53e95640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f4c110a30 con 0x7f4f4c100610 2024-04-25T01:43:14.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.270+0000 7f4f53e95640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4f4c1108b0 con 0x7f4f4c100fd0 2024-04-25T01:43:14.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.270+0000 7f4f53e95640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4f4c110bb0 con 0x7f4f4c1066b0 2024-04-25T01:43:14.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.270+0000 7f4f51c31640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c100610 0x7f4f4c1022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:14.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.270+0000 7f4f51c31640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c100610 0x7f4f4c1022b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56992/0 (socket says 172.21.15.28:56992) 2024-04-25T01:43:14.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.270+0000 7f4f51c31640 1 -- 172.21.15.28:0/3040981124 learned_addr learned my addr 172.21.15.28:0/3040981124 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:14.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.270+0000 7f4f51430640 1 --2- 172.21.15.28:0/3040981124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f4c100fd0 0x7f4f4c1027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:14.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.270+0000 7f4f52432640 1 --2- 172.21.15.28:0/3040981124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f4c1066b0 0x7f4f4c102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:14.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.270+0000 7f4f51c31640 1 -- 172.21.15.28:0/3040981124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f4c1066b0 msgr2=0x7f4f4c102d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:14.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.270+0000 7f4f51c31640 1 --2- 172.21.15.28:0/3040981124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f4c1066b0 0x7f4f4c102d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.270+0000 7f4f51c31640 1 -- 172.21.15.28:0/3040981124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f4c100fd0 msgr2=0x7f4f4c1027f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:14.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.270+0000 7f4f51c31640 1 --2- 172.21.15.28:0/3040981124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f4c100fd0 0x7f4f4c1027f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.270+0000 7f4f51c31640 1 -- 172.21.15.28:0/3040981124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f4c1a6480 con 0x7f4f4c100610 2024-04-25T01:43:14.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.271+0000 7f4f51c31640 1 --2- 172.21.15.28:0/3040981124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c100610 0x7f4f4c1022b0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c00e9c0 tx=0x7f4f3c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:14.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.271+0000 7f4f4affd640 1 -- 172.21.15.28:0/3040981124 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c00cd80 con 0x7f4f4c100610 2024-04-25T01:43:14.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.271+0000 7f4f4affd640 1 -- 172.21.15.28:0/3040981124 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f3c0035f0 con 0x7f4f4c100610 2024-04-25T01:43:14.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.271+0000 7f4f53e95640 1 -- 172.21.15.28:0/3040981124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f4c1a6770 con 0x7f4f4c100610 2024-04-25T01:43:14.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.271+0000 7f4f4affd640 1 -- 172.21.15.28:0/3040981124 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c010690 con 0x7f4f4c100610 2024-04-25T01:43:14.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.271+0000 7f4f53e95640 1 -- 172.21.15.28:0/3040981124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f4c1a6c90 con 0x7f4f4c100610 2024-04-25T01:43:14.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.272+0000 7f4f53e95640 1 -- 172.21.15.28:0/3040981124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f4c068560 con 0x7f4f4c100610 2024-04-25T01:43:14.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.274+0000 7f4f4affd640 1 -- 172.21.15.28:0/3040981124 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f4f3c0037a0 con 0x7f4f4c100610 2024-04-25T01:43:14.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.274+0000 7f4f4affd640 1 --2- 172.21.15.28:0/3040981124 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4f340a2d30 0x7f4f340a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:14.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.274+0000 7f4f4affd640 1 -- 172.21.15.28:0/3040981124 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f4f3c0cfb60 con 0x7f4f4c100610 2024-04-25T01:43:14.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.274+0000 7f4f51430640 1 --2- 172.21.15.28:0/3040981124 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4f340a2d30 0x7f4f340a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:14.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.275+0000 7f4f51430640 1 --2- 172.21.15.28:0/3040981124 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4f340a2d30 0x7f4f340a51f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4f40009900 tx=0x7f4f400058d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:14.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.279+0000 7f4f4affd640 1 -- 172.21.15.28:0/3040981124 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4f3c09c390 con 0x7f4f4c100610 2024-04-25T01:43:14.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.468+0000 7f4f53e95640 1 -- 172.21.15.28:0/3040981124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4f4c1a6f50 con 0x7f4f4c100610 2024-04-25T01:43:14.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.468+0000 7f4f4affd640 1 -- 172.21.15.28:0/3040981124 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v101) ==== 103+0+11 (secure 0 0 0) 0x7f4f3c09bc90 con 0x7f4f4c100610 2024-04-25T01:43:14.457 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:43:14.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.470+0000 7f4f53e95640 1 -- 172.21.15.28:0/3040981124 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4f340a2d30 msgr2=0x7f4f340a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.470+0000 7f4f53e95640 1 --2- 172.21.15.28:0/3040981124 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4f340a2d30 0x7f4f340a51f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4f40009900 tx=0x7f4f400058d0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.470+0000 7f4f53e95640 1 -- 172.21.15.28:0/3040981124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c100610 msgr2=0x7f4f4c1022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.470+0000 7f4f53e95640 1 --2- 172.21.15.28:0/3040981124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c100610 0x7f4f4c1022b0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c00e9c0 tx=0x7f4f3c00ee80 comp rx=0 tx=0).stop 2024-04-25T01:43:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.470+0000 7f4f53e95640 1 -- 172.21.15.28:0/3040981124 shutdown_connections 2024-04-25T01:43:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.470+0000 7f4f53e95640 1 --2- 172.21.15.28:0/3040981124 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4f340a2d30 0x7f4f340a51f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.470+0000 7f4f53e95640 1 --2- 172.21.15.28:0/3040981124 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4f4c1066b0 0x7f4f4c102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.470+0000 7f4f53e95640 1 --2- 172.21.15.28:0/3040981124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f4c100fd0 0x7f4f4c1027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.470+0000 7f4f53e95640 1 --2- 172.21.15.28:0/3040981124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f4c100610 0x7f4f4c1022b0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.470+0000 7f4f53e95640 1 -- 172.21.15.28:0/3040981124 >> 172.21.15.28:0/3040981124 conn(0x7f4f4c077740 msgr2=0x7f4f4c0ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.470+0000 7f4f53e95640 1 -- 172.21.15.28:0/3040981124 shutdown_connections 2024-04-25T01:43:14.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.470+0000 7f4f53e95640 1 -- 172.21.15.28:0/3040981124 wait complete. 2024-04-25T01:43:14.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-25T01:43:14.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:14 smithi130 ceph-mon[32702]: pgmap v316: 129 pgs: 129 active+clean; 11 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-04-25T01:43:14.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:14 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2089971646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:43:14.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:14 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3796485803' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:43:14.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:14 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3652841459' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:43:14.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:14 smithi032 ceph-mon[32461]: pgmap v316: 129 pgs: 129 active+clean; 11 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-04-25T01:43:14.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:14 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2089971646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:43:14.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:14 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3796485803' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:43:14.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:14 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3652841459' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:43:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.586+0000 7f067a909640 1 -- 172.21.15.28:0/4187440866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067410e100 msgr2=0x7f06741104f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.586+0000 7f067a909640 1 --2- 172.21.15.28:0/4187440866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067410e100 0x7f06741104f0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f066c009ca0 tx=0x7f066c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.586+0000 7f067a909640 1 -- 172.21.15.28:0/4187440866 shutdown_connections 2024-04-25T01:43:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.586+0000 7f067a909640 1 --2- 172.21.15.28:0/4187440866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067410e100 0x7f06741104f0 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.586+0000 7f067a909640 1 --2- 172.21.15.28:0/4187440866 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0674100e70 0x7f067410dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.586+0000 7f067a909640 1 --2- 172.21.15.28:0/4187440866 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0674100550 0x7f0674100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.586+0000 7f067a909640 1 -- 172.21.15.28:0/4187440866 >> 172.21.15.28:0/4187440866 conn(0x7f0674077740 msgr2=0x7f0674077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.586+0000 7f067a909640 1 -- 172.21.15.28:0/4187440866 shutdown_connections 2024-04-25T01:43:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.586+0000 7f067a909640 1 -- 172.21.15.28:0/4187440866 wait complete. 2024-04-25T01:43:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.586+0000 7f067a909640 1 Processor -- start 2024-04-25T01:43:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.586+0000 7f067a909640 1 -- start start 2024-04-25T01:43:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.587+0000 7f067a909640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0674100550 0x7f067419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.587+0000 7f067a909640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0674100e70 0x7f067419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.587+0000 7f067a909640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067410e100 0x7f067419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.587+0000 7f067a909640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0674113d30 con 0x7f067410e100 2024-04-25T01:43:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.587+0000 7f067a909640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0674113bb0 con 0x7f0674100550 2024-04-25T01:43:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.587+0000 7f067a909640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0674113eb0 con 0x7f0674100e70 2024-04-25T01:43:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.587+0000 7f0678ea6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067410e100 0x7f067419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.587+0000 7f0673fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0674100550 0x7f067419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.587+0000 7f0678ea6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067410e100 0x7f067419ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57006/0 (socket says 172.21.15.28:57006) 2024-04-25T01:43:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.587+0000 7f0678ea6640 1 -- 172.21.15.28:0/987333514 learned_addr learned my addr 172.21.15.28:0/987333514 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:14.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.587+0000 7f06737fe640 1 --2- 172.21.15.28:0/987333514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0674100e70 0x7f067419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.587+0000 7f0673fff640 1 -- 172.21.15.28:0/987333514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0674100e70 msgr2=0x7f067419a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.587+0000 7f0673fff640 1 --2- 172.21.15.28:0/987333514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0674100e70 0x7f067419a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.587+0000 7f0673fff640 1 -- 172.21.15.28:0/987333514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067410e100 msgr2=0x7f067419ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.587+0000 7f0673fff640 1 --2- 172.21.15.28:0/987333514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067410e100 0x7f067419ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.587+0000 7f0673fff640 1 -- 172.21.15.28:0/987333514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f067419f200 con 0x7f0674100550 2024-04-25T01:43:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.587+0000 7f0673fff640 1 --2- 172.21.15.28:0/987333514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0674100550 0x7f067419a210 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f066800e9f0 tx=0x7f066800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.588+0000 7f06717fa640 1 -- 172.21.15.28:0/987333514 <== mon.2 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066800ccc0 con 0x7f0674100550 2024-04-25T01:43:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.588+0000 7f067a909640 1 -- 172.21.15.28:0/987333514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f067419f4f0 con 0x7f0674100550 2024-04-25T01:43:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.588+0000 7f06717fa640 1 -- 172.21.15.28:0/987333514 <== mon.2 v2:172.21.15.32:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0668005020 con 0x7f0674100550 2024-04-25T01:43:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.588+0000 7f06717fa640 1 -- 172.21.15.28:0/987333514 <== mon.2 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06680049e0 con 0x7f0674100550 2024-04-25T01:43:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.588+0000 7f067a909640 1 -- 172.21.15.28:0/987333514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06741a6d80 con 0x7f0674100550 2024-04-25T01:43:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.589+0000 7f067a909640 1 -- 172.21.15.28:0/987333514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0674068560 con 0x7f0674100550 2024-04-25T01:43:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.590+0000 7f06717fa640 1 -- 172.21.15.28:0/987333514 <== mon.2 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0668005560 con 0x7f0674100550 2024-04-25T01:43:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.591+0000 7f06717fa640 1 --2- 172.21.15.28:0/987333514 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f065c0a2d00 0x7f065c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.591+0000 7f06737fe640 1 --2- 172.21.15.28:0/987333514 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f065c0a2d00 0x7f065c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:14.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.591+0000 7f06717fa640 1 -- 172.21.15.28:0/987333514 <== mon.2 v2:172.21.15.32:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9526+0+0 (secure 0 0 0) 0x7f06680cf8a0 con 0x7f0674100550 2024-04-25T01:43:14.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.591+0000 7f06737fe640 1 --2- 172.21.15.28:0/987333514 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f065c0a2d00 0x7f065c0a51c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f06640095f0 tx=0x7f0664005f90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.599+0000 7f06717fa640 1 -- 172.21.15.28:0/987333514 <== mon.2 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f06680051d0 con 0x7f0674100550 2024-04-25T01:43:14.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.864+0000 7f067a909640 1 -- 172.21.15.28:0/987333514 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f06741092c0 con 0x7f0674100550 2024-04-25T01:43:14.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.873+0000 7f06717fa640 1 -- 172.21.15.28:0/987333514 <== mon.2 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) ==== 81+0+740898 (secure 0 0 0) 0x7f066809c100 con 0x7f0674100550 2024-04-25T01:43:14.863 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:43:14.863 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:43:14.863 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 25, 2024-04-25T01:43:14.863 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.864 INFO:teuthology.orchestra.run.smithi028.stdout: "active_gid": 15561, 2024-04-25T01:43:14.864 INFO:teuthology.orchestra.run.smithi028.stdout: "active_name": "x", 2024-04-25T01:43:14.864 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addrs": { 2024-04-25T01:43:14.864 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-25T01:43:14.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:14.864 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-25T01:43:14.864 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6800", 2024-04-25T01:43:14.864 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 712762371 2024-04-25T01:43:14.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:14.864 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-04-25T01:43:14.864 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-04-25T01:43:14.864 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 712762371 2024-04-25T01:43:14.865 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:14.865 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-25T01:43:14.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.865 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addr": "172.21.15.28:6801/712762371", 2024-04-25T01:43:14.865 INFO:teuthology.orchestra.run.smithi028.stdout: "active_change": "2024-04-25T01:33:00.617403+0000", 2024-04-25T01:43:14.865 INFO:teuthology.orchestra.run.smithi028.stdout: "active_mgr_features": 4540701547738038271, 2024-04-25T01:43:14.865 INFO:teuthology.orchestra.run.smithi028.stdout: "available": true, 2024-04-25T01:43:14.865 INFO:teuthology.orchestra.run.smithi028.stdout: "standbys": [ 2024-04-25T01:43:14.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:14.865 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 25126, 2024-04-25T01:43:14.865 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "z", 2024-04-25T01:43:14.865 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540701547738038271, 2024-04-25T01:43:14.865 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-04-25T01:43:14.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:14.866 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-04-25T01:43:14.866 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:14.866 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:14.866 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:14.866 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-25T01:43:14.866 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-25T01:43:14.866 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:14.866 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.866 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.866 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-25T01:43:14.867 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.867 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.867 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.867 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T01:43:14.867 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.867 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.867 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.867 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:14.867 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:14.867 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.867 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.868 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.868 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:14.868 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.868 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.868 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:14.868 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:14.868 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:14.868 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:14.868 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:14.868 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:14.868 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:14.868 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:14.868 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:14.868 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.869 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.869 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.869 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:14.869 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:14.869 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.869 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.869 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.869 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:14.869 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.869 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.869 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.869 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:14.869 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.870 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.870 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.870 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:14.870 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:14.870 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.870 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.870 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.870 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:14.870 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.871 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.871 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:14.871 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:14.871 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:14.871 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:14.871 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:14.871 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:14.871 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:14.871 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:14.871 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:14.871 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.872 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.872 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.872 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:14.872 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:14.872 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.872 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.872 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.872 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:14.872 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.872 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.873 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.873 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:14.873 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.873 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.873 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.873 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-04-25T01:43:14.873 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-04-25T01:43:14.873 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.873 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.873 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.874 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:14.874 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.874 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.874 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.874 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-04-25T01:43:14.874 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.874 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.874 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.874 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-04-25T01:43:14.875 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-04-25T01:43:14.875 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.875 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.875 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.875 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-04-25T01:43:14.875 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.875 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.875 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.875 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-04-25T01:43:14.876 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.876 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.876 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.876 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-04-25T01:43:14.876 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-04-25T01:43:14.876 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.876 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.876 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.876 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:14.876 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.876 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.877 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.877 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-04-25T01:43:14.877 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.877 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.877 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.877 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-04-25T01:43:14.877 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-04-25T01:43:14.877 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.877 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.877 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.877 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:14.877 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.878 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.878 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.878 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-04-25T01:43:14.878 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.878 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.878 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.878 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-04-25T01:43:14.878 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-04-25T01:43:14.878 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:14.879 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.879 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.879 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-04-25T01:43:14.879 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.879 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.879 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.879 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-04-25T01:43:14.879 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.879 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.879 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.879 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-04-25T01:43:14.879 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-04-25T01:43:14.880 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.880 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.880 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.880 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:14.880 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.880 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.880 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.880 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-04-25T01:43:14.880 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.880 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.881 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.881 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-04-25T01:43:14.881 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-04-25T01:43:14.881 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.881 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.881 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.881 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:14.881 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.882 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.882 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.882 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T01:43:14.882 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.882 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.882 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.882 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-04-25T01:43:14.882 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-04-25T01:43:14.883 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.883 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.883 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.883 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:14.883 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.883 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.883 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.883 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-04-25T01:43:14.883 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.883 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.884 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.884 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:14.884 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:14.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:14.884 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-04-25T01:43:14.884 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:14.884 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:14.884 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:14.885 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-04-25T01:43:14.885 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-04-25T01:43:14.885 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.885 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.885 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.885 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:14.885 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.885 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.885 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.885 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T01:43:14.885 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.885 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.886 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.886 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-04-25T01:43:14.886 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-04-25T01:43:14.886 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.886 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.886 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.886 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-04-25T01:43:14.886 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.886 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.886 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.886 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T01:43:14.887 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T01:43:14.887 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.887 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.887 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-04-25T01:43:14.887 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-04-25T01:43:14.887 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-25T01:43:14.887 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.887 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.887 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-25T01:43:14.887 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-25T01:43:14.887 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-25T01:43:14.887 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.888 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T01:43:14.888 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T01:43:14.888 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.888 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.888 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-04-25T01:43:14.888 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-04-25T01:43:14.888 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-25T01:43:14.888 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.888 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.888 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-04-25T01:43:14.888 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-25T01:43:14.888 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-04-25T01:43:14.888 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.889 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T01:43:14.889 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.889 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.889 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.889 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-04-25T01:43:14.889 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-04-25T01:43:14.889 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.889 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.889 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.889 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T01:43:14.889 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.889 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.889 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.890 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T01:43:14.890 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T01:43:14.890 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.890 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.890 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-04-25T01:43:14.890 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-04-25T01:43:14.890 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:14.890 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.890 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.891 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-04-25T01:43:14.891 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-04-25T01:43:14.891 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-04-25T01:43:14.891 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.891 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-04-25T01:43:14.891 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T01:43:14.891 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.891 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.891 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-04-25T01:43:14.891 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-04-25T01:43:14.892 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.892 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.892 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.892 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-04-25T01:43:14.892 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.892 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.892 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.892 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T01:43:14.892 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T01:43:14.892 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.892 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.892 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-04-25T01:43:14.893 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-04-25T01:43:14.893 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-25T01:43:14.893 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.893 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.893 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-25T01:43:14.893 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-25T01:43:14.893 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-25T01:43:14.893 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.893 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T01:43:14.893 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T01:43:14.893 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.893 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.893 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:14.894 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.895 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:14.904 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.904 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.905 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-04-25T01:43:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-04-25T01:43:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.908 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-25T01:43:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T01:43:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-25T01:43:14.909 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-25T01:43:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-04-25T01:43:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-04-25T01:43:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-04-25T01:43:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-04-25T01:43:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-04-25T01:43:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-04-25T01:43:14.910 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-04-25T01:43:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-04-25T01:43:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-04-25T01:43:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:14.911 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.912 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.912 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.912 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T01:43:14.912 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.912 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.912 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.912 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-25T01:43:14.913 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-25T01:43:14.913 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:14.913 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.913 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.913 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-25T01:43:14.913 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.913 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.913 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.913 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T01:43:14.913 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-04-25T01:43:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-04-25T01:43:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-25T01:43:14.914 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-25T01:43:14.915 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.915 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.915 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T01:43:14.915 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T01:43:14.915 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.915 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.915 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-04-25T01:43:14.915 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-04-25T01:43:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-25T01:43:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T01:43:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.916 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-04-25T01:43:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:14.917 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-04-25T01:43:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-04-25T01:43:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-25T01:43:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T01:43:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-04-25T01:43:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-04-25T01:43:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.919 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-04-25T01:43:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T01:43:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-04-25T01:43:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-04-25T01:43:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-04-25T01:43:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.921 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T01:43:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-04-25T01:43:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-04-25T01:43:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T01:43:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T01:43:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-04-25T01:43:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-04-25T01:43:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:14.924 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-25T01:43:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T01:43:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-04-25T01:43:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T01:43:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:14.925 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-04-25T01:43:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T01:43:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-04-25T01:43:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-04-25T01:43:14.926 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-04-25T01:43:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-04-25T01:43:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-04-25T01:43:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T01:43:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.927 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-04-25T01:43:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-04-25T01:43:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T01:43:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.928 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-04-25T01:43:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-04-25T01:43:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T01:43:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-04-25T01:43:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-04-25T01:43:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:14.930 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T01:43:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-04-25T01:43:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-04-25T01:43:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.931 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T01:43:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-25T01:43:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-04-25T01:43:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-04-25T01:43:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.932 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T01:43:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-04-25T01:43:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.933 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-04-25T01:43:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-04-25T01:43:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T01:43:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.934 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-04-25T01:43:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-04-25T01:43:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-04-25T01:43:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.935 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T01:43:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-25T01:43:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-04-25T01:43:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-04-25T01:43:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.936 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T01:43:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-04-25T01:43:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-04-25T01:43:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-04-25T01:43:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.937 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T01:43:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-04-25T01:43:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-04-25T01:43:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-04-25T01:43:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.938 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T01:43:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-04-25T01:43:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-04-25T01:43:14.939 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-04-25T01:43:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T01:43:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-04-25T01:43:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-04-25T01:43:14.940 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-04-25T01:43:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T01:43:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-04-25T01:43:14.941 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-04-25T01:43:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-04-25T01:43:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T01:43:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.942 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-04-25T01:43:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-04-25T01:43:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-04-25T01:43:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T01:43:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.943 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-25T01:43:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-04-25T01:43:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-04-25T01:43:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T01:43:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.944 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-04-25T01:43:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-04-25T01:43:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-04-25T01:43:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T01:43:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.945 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-25T01:43:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-04-25T01:43:14.946 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-04-25T01:43:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T01:43:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-04-25T01:43:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-04-25T01:43:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-04-25T01:43:14.947 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T01:43:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.948 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-04-25T01:43:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-04-25T01:43:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-04-25T01:43:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.949 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T01:43:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-04-25T01:43:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.950 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-04-25T01:43:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-04-25T01:43:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.951 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T01:43:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-04-25T01:43:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-04-25T01:43:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-25T01:43:14.952 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T01:43:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-04-25T01:43:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T01:43:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-04-25T01:43:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.954 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T01:43:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-04-25T01:43:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-04-25T01:43:14.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-04-25T01:43:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T01:43:14.956 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-04-25T01:43:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-04-25T01:43:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-04-25T01:43:14.957 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-04-25T01:43:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-04-25T01:43:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-04-25T01:43:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.958 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T01:43:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-04-25T01:43:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-04-25T01:43:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-25T01:43:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-04-25T01:43:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-04-25T01:43:14.961 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-04-25T01:43:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T01:43:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T01:43:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-04-25T01:43:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-04-25T01:43:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-25T01:43:14.963 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-04-25T01:43:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.964 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-04-25T01:43:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-04-25T01:43:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T01:43:14.965 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-04-25T01:43:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-04-25T01:43:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.966 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T01:43:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.967 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-04-25T01:43:14.968 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-04-25T01:43:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T01:43:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.969 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.970 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T01:43:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.971 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.972 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T01:43:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T01:43:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T01:43:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.974 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T01:43:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T01:43:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-04-25T01:43:14.975 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T01:43:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-04-25T01:43:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-04-25T01:43:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.976 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-04-25T01:43:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-04-25T01:43:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-04-25T01:43:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:14.977 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T01:43:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-04-25T01:43:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-04-25T01:43:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.979 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-04-25T01:43:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-25T01:43:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-25T01:43:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.980 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-04-25T01:43:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-04-25T01:43:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-04-25T01:43:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T01:43:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-04-25T01:43:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-04-25T01:43:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-04-25T01:43:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.982 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T01:43:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-04-25T01:43:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-04-25T01:43:14.983 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T01:43:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T01:43:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-04-25T01:43:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-04-25T01:43:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.985 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T01:43:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-04-25T01:43:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-04-25T01:43:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.986 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T01:43:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-04-25T01:43:14.987 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-04-25T01:43:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T01:43:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.988 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-04-25T01:43:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-04-25T01:43:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T01:43:14.989 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-04-25T01:43:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-04-25T01:43:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.990 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T01:43:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.991 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-04-25T01:43:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-04-25T01:43:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-04-25T01:43:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.992 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-04-25T01:43:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-04-25T01:43:14.993 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-04-25T01:43:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T01:43:14.994 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-04-25T01:43:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-04-25T01:43:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.995 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T01:43:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-04-25T01:43:14.996 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-04-25T01:43:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T01:43:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-04-25T01:43:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-04-25T01:43:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:14.998 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T01:43:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-04-25T01:43:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-04-25T01:43:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:14.999 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T01:43:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-04-25T01:43:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-04-25T01:43:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.000 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T01:43:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-04-25T01:43:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.002 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.003 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.006 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.007 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.008 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-04-25T01:43:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-04-25T01:43:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-04-25T01:43:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.009 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T01:43:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-04-25T01:43:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-04-25T01:43:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-04-25T01:43:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T01:43:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.011 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-04-25T01:43:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T01:43:15.012 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T01:43:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.013 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T01:43:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T01:43:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.014 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.015 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T01:43:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T01:43:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.016 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-04-25T01:43:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T01:43:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.017 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T01:43:15.018 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T01:43:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.019 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T01:43:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T01:43:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T01:43:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T01:43:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.021 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T01:43:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T01:43:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.022 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.023 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T01:43:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T01:43:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.024 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T01:43:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T01:43:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.025 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.026 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.027 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.027 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T01:43:15.027 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T01:43:15.027 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.029 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.029 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.029 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.029 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.029 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.029 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.029 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T01:43:15.030 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T01:43:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T01:43:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T01:43:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.032 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T01:43:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T01:43:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T01:43:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T01:43:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-25T01:43:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T01:43:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T01:43:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.035 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-04-25T01:43:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-04-25T01:43:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.036 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T01:43:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T01:43:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-25T01:43:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.037 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.038 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.038 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T01:43:15.038 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T01:43:15.038 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.038 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.038 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.038 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.038 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.038 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.038 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.038 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T01:43:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T01:43:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.039 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T01:43:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T01:43:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.040 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T01:43:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T01:43:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.041 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T01:43:15.042 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T01:43:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.043 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T01:43:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T01:43:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.044 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T01:43:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T01:43:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.045 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T01:43:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T01:43:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.046 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T01:43:15.047 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T01:43:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.048 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T01:43:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T01:43:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.049 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T01:43:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T01:43:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.050 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.051 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.051 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.051 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.051 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.051 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T01:43:15.051 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T01:43:15.051 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.051 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.051 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.052 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.052 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.052 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.052 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.052 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.052 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.052 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.052 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T01:43:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T01:43:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.053 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T01:43:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T01:43:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.054 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T01:43:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T01:43:15.055 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T01:43:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.056 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T01:43:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T01:43:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.057 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T01:43:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T01:43:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.058 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-04-25T01:43:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.059 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T01:43:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T01:43:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-04-25T01:43:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T01:43:15.060 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T01:43:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.061 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T01:43:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T01:43:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-25T01:43:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.062 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T01:43:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T01:43:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.063 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T01:43:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T01:43:15.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.065 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T01:43:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T01:43:15.066 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-25T01:43:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.067 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T01:43:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T01:43:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.068 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T01:43:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T01:43:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.069 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-25T01:43:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-04-25T01:43:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-04-25T01:43:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.070 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.071 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.071 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.071 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.071 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.071 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.071 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.071 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.071 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.071 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.071 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-04-25T01:43:15.071 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-04-25T01:43:15.071 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-04-25T01:43:15.072 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-04-25T01:43:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-04-25T01:43:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.073 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-04-25T01:43:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-04-25T01:43:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-04-25T01:43:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.074 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-25T01:43:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-25T01:43:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.075 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.076 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.077 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.078 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.079 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-04-25T01:43:15.080 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-04-25T01:43:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-04-25T01:43:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-04-25T01:43:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T01:43:15.081 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-25T01:43:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-25T01:43:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.082 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-25T01:43:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-25T01:43:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-25T01:43:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.083 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-04-25T01:43:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-25T01:43:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-25T01:43:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.084 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-04-25T01:43:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-04-25T01:43:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-04-25T01:43:15.085 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-25T01:43:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-25T01:43:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.086 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-04-25T01:43:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-04-25T01:43:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-25T01:43:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-25T01:43:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.087 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-25T01:43:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-25T01:43:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-25T01:43:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-04-25T01:43:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-04-25T01:43:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.089 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-25T01:43:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-04-25T01:43:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-04-25T01:43:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.090 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-04-25T01:43:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.091 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.092 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.093 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.093 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.093 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.093 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.093 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.093 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.093 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.093 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.093 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.093 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.093 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.094 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.094 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.094 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.094 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.094 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.094 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.094 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.094 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.094 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.094 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.094 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.094 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.095 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.095 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.095 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.095 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.095 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.095 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.095 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.095 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.095 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.095 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.096 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.096 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.096 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.096 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.096 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.096 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.096 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.096 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.096 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.096 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.096 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.097 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.097 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.097 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-04-25T01:43:15.097 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-04-25T01:43:15.097 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.097 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.097 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.097 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-04-25T01:43:15.097 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.098 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.098 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.098 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T01:43:15.098 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.098 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.098 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.098 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-04-25T01:43:15.098 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-04-25T01:43:15.098 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.098 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.099 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-04-25T01:43:15.099 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.099 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.099 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.099 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T01:43:15.099 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.099 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.099 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.100 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-04-25T01:43:15.100 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-04-25T01:43:15.100 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.100 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.100 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.100 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-04-25T01:43:15.100 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.100 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.100 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.100 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-04-25T01:43:15.100 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.101 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.101 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.101 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-04-25T01:43:15.101 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-04-25T01:43:15.101 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.101 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.101 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.101 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-25T01:43:15.101 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.102 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.102 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.102 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T01:43:15.102 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.102 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.102 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.102 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-04-25T01:43:15.102 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-04-25T01:43:15.102 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.102 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.102 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.102 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.102 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.102 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.103 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.103 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T01:43:15.103 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.103 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.103 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.103 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-25T01:43:15.103 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-25T01:43:15.103 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.103 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.103 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.103 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-25T01:43:15.103 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.103 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.104 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.104 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T01:43:15.104 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.104 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.104 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.104 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-04-25T01:43:15.104 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-04-25T01:43:15.104 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.104 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.104 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.104 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-04-25T01:43:15.104 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.104 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.104 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.104 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T01:43:15.105 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.105 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.105 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.105 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.105 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.105 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-04-25T01:43:15.105 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.105 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.105 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.105 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.105 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.106 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.106 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.106 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.106 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.106 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.106 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.106 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.106 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.106 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.106 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.106 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.106 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.106 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.106 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.106 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.107 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.107 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.107 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.107 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.107 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.107 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.107 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.107 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.107 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.107 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.107 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.107 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.107 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.107 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.108 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.108 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.108 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.108 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.108 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.108 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.108 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.108 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.108 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.108 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.108 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.108 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.108 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.108 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.109 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.109 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.109 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.109 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.109 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.109 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.109 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.109 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.109 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.109 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.109 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.109 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.109 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.109 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.110 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.110 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.110 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.110 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.110 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.110 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.110 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-04-25T01:43:15.110 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-04-25T01:43:15.110 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.110 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.110 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.110 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-25T01:43:15.110 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.111 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.111 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.111 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.111 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.111 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.111 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.111 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-04-25T01:43:15.111 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-04-25T01:43:15.111 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.111 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.111 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.111 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-04-25T01:43:15.111 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.111 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.112 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.112 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.112 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.112 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.112 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.112 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-25T01:43:15.112 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-25T01:43:15.112 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.112 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.112 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.112 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-25T01:43:15.112 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.112 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.112 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.113 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.113 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.113 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.113 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.113 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.113 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.113 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-04-25T01:43:15.113 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-04-25T01:43:15.113 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-04-25T01:43:15.113 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.113 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-04-25T01:43:15.113 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-04-25T01:43:15.113 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.114 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.114 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.114 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-04-25T01:43:15.114 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.114 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.114 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.114 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T01:43:15.114 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.114 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.114 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.114 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-04-25T01:43:15.114 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-04-25T01:43:15.114 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.115 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.115 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.115 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-04-25T01:43:15.115 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.115 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.115 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.115 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T01:43:15.115 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.115 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.115 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.115 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-04-25T01:43:15.115 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-04-25T01:43:15.115 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.116 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.116 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.116 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.116 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.116 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.116 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.116 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-04-25T01:43:15.116 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.116 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.116 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.116 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-25T01:43:15.116 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-25T01:43:15.117 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.117 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.117 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.117 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-04-25T01:43:15.117 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-04-25T01:43:15.117 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.117 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.117 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T01:43:15.117 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.117 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.117 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.117 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.117 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.118 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.118 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.118 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.118 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.118 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.118 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.118 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.118 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.118 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.118 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.118 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.118 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.118 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.119 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.119 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.119 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.119 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.119 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.119 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.119 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.119 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.119 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.119 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.119 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.119 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.119 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.120 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.120 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.120 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.120 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.120 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.120 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.120 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.120 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.120 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.120 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.120 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.120 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.120 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.120 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.121 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.121 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.121 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.121 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.121 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.121 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.121 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.121 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.121 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.121 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.121 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.121 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.121 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.121 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.121 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.122 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.122 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.122 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.122 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.122 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.122 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.122 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.122 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.122 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-04-25T01:43:15.122 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-04-25T01:43:15.122 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.122 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.123 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.123 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.123 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.123 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.123 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.123 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-04-25T01:43:15.123 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.123 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.123 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.123 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-04-25T01:43:15.123 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-04-25T01:43:15.124 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.124 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.124 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.124 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-04-25T01:43:15.124 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.124 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.124 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.124 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-04-25T01:43:15.124 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.124 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.124 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.125 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-25T01:43:15.125 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-25T01:43:15.125 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.125 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.125 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.125 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-04-25T01:43:15.125 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.125 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.125 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.125 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T01:43:15.125 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.125 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.125 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.126 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-04-25T01:43:15.126 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-04-25T01:43:15.126 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.126 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.126 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.126 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-25T01:43:15.126 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-25T01:43:15.126 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-04-25T01:43:15.126 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.126 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T01:43:15.127 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.127 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.127 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.127 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-04-25T01:43:15.127 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-04-25T01:43:15.127 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.127 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.127 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.127 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.127 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.127 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.127 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.127 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-04-25T01:43:15.127 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.128 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.128 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.128 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-04-25T01:43:15.128 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-04-25T01:43:15.128 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.128 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.128 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.128 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-04-25T01:43:15.128 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.128 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.128 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.129 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T01:43:15.129 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.129 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.129 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.129 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.129 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.129 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-04-25T01:43:15.129 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.129 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.129 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.129 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.129 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.129 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.130 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.130 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.130 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.130 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.130 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.130 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.130 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.130 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.130 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.131 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.131 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.131 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.131 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.131 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.131 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.131 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.131 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.131 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.131 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.131 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.131 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.131 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.132 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.132 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.132 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.132 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.132 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.132 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.132 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.132 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.132 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.133 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.133 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.133 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.133 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.133 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.133 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.133 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.133 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.133 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.133 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.133 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.133 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.133 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.134 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.134 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.134 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.134 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.134 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.134 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.134 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.134 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.134 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.135 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.135 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.135 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.135 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.135 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.135 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.135 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.135 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.135 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.136 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.136 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.136 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-04-25T01:43:15.136 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.136 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.136 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.136 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.136 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.136 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.137 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.137 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.137 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.137 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.137 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.137 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.137 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.137 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.137 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.137 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.138 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.138 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.138 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.138 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.138 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.138 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.138 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.138 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.139 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.139 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.139 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.139 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.139 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.139 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.139 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.139 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.139 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.139 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.139 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.139 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.139 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.140 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.140 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.140 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.140 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.140 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.140 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.140 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.140 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.140 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.140 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.140 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.140 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.141 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.141 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.141 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.141 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.141 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.141 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.141 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.141 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.141 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.141 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.141 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.141 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.141 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.142 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.142 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.142 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.142 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.142 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.142 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.142 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.142 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-04-25T01:43:15.142 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.142 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.142 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.142 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-04-25T01:43:15.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-04-25T01:43:15.143 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.143 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.143 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.143 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-04-25T01:43:15.143 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.143 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.143 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.143 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T01:43:15.143 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.143 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.143 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.144 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-04-25T01:43:15.144 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-04-25T01:43:15.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.144 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.144 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.144 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:15.144 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-04-25T01:43:15.144 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.144 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.144 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T01:43:15.144 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.144 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.144 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.145 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.145 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.145 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.145 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.145 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.145 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.145 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.145 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.145 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.145 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.145 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.145 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.145 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.145 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.145 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.146 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.146 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.146 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.146 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.146 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.146 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.146 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.146 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.146 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.146 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.146 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.146 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.146 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.146 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.147 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.147 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.147 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.147 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.147 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.147 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.147 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.147 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.147 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.147 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.147 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.147 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.147 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.148 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.148 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.148 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.148 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.148 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.148 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.148 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.148 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.148 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.148 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.148 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.148 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.148 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.149 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.149 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.149 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.149 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.149 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.149 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.149 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.149 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.149 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.149 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.149 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.149 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.149 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-04-25T01:43:15.150 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.150 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.150 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.150 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-04-25T01:43:15.150 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-04-25T01:43:15.150 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.150 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.150 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-04-25T01:43:15.150 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.150 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.150 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.150 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-04-25T01:43:15.150 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T01:43:15.151 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.151 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.151 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.151 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.151 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.151 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.151 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.151 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.151 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.151 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.151 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.151 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.151 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.151 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.152 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.152 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.152 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.152 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.152 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.152 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.152 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.152 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.152 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.152 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.152 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.152 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.152 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.152 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.153 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.153 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.153 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.153 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.153 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.153 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.153 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.153 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.153 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.153 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.153 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.153 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.153 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.153 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.153 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.154 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.154 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.154 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.154 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.154 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.154 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.154 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.154 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.154 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.154 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.154 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.154 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.154 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.155 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.155 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.155 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.155 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.155 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.155 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.155 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.155 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.155 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.155 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.155 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.155 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.155 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-04-25T01:43:15.156 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-04-25T01:43:15.156 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.156 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.156 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.156 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.156 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.156 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.156 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.156 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-04-25T01:43:15.156 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T01:43:15.156 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.156 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.156 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-04-25T01:43:15.156 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-04-25T01:43:15.157 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.157 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.157 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.157 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-04-25T01:43:15.157 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.157 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.157 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.157 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-04-25T01:43:15.157 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.157 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.157 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.157 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-04-25T01:43:15.157 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-04-25T01:43:15.158 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.158 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.158 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.158 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-04-25T01:43:15.158 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.158 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.158 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.158 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T01:43:15.158 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.158 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.158 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.158 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-04-25T01:43:15.158 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-04-25T01:43:15.158 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.159 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.159 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.159 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.159 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.159 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.159 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.159 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-04-25T01:43:15.159 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.159 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.159 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.159 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-04-25T01:43:15.159 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-04-25T01:43:15.159 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.159 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.160 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.160 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-04-25T01:43:15.160 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.160 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.160 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.160 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T01:43:15.160 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T01:43:15.160 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.160 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.161 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.161 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.161 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-04-25T01:43:15.161 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.161 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.161 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.161 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.161 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.161 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.161 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.161 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.161 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.162 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.162 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.162 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.162 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.162 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.162 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.162 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.162 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.162 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.162 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.162 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.163 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.163 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.163 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.163 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.163 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.163 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.163 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.163 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.163 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.163 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.163 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.163 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.164 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.164 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.164 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.164 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.164 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.164 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.164 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.164 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.164 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.164 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.165 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.165 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.165 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.165 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.165 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.165 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.165 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.165 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.165 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.165 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.165 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.165 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.165 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.166 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.166 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.166 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.166 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.166 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.166 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.166 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.166 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.166 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.166 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.167 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.167 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.167 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.167 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.167 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.167 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-04-25T01:43:15.167 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.167 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.167 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.167 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.167 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.167 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.167 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.168 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.168 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.168 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.168 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.168 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.168 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.168 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.168 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.168 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.168 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.168 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.169 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.169 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.169 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.169 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.169 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.169 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.169 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.169 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.169 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.170 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.170 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.170 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.170 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.170 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.170 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.170 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.170 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.170 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.171 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.171 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.171 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.171 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.171 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.171 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.171 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.171 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.171 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.171 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.172 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.172 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.172 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.172 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.172 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.172 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.172 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.172 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.172 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.172 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.173 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.173 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.173 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.173 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.173 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.173 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.173 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.173 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.173 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.173 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.173 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.173 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.173 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.174 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.174 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.174 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-04-25T01:43:15.174 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.174 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.174 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.174 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.174 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.174 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.174 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.174 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.175 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.175 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.175 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.175 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.175 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.175 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.175 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.175 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.175 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.175 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.175 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.175 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.175 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.175 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.176 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.176 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.176 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.176 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.176 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.176 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.176 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.176 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.176 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.176 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.176 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.176 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.176 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.177 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.177 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.177 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.177 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.177 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.177 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.177 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.177 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.177 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.177 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.177 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.177 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.177 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.177 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.177 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.177 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.178 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.178 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.178 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.178 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.178 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.178 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.178 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.178 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.178 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.178 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.178 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.178 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.178 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.179 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.179 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.179 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.179 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.179 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.179 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.179 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-25T01:43:15.179 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.179 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.179 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.179 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-04-25T01:43:15.179 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-04-25T01:43:15.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.180 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.180 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.180 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.180 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.180 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.180 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.180 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T01:43:15.180 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.180 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.180 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.180 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.180 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.180 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.180 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.181 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.181 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.181 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.181 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.181 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.181 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.181 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.181 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.181 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.181 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.181 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.181 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.181 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.181 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.181 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.182 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.182 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.182 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.182 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.182 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.182 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.182 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.182 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.182 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.182 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.182 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.182 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.182 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.182 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.183 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.183 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.183 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.183 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.183 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.183 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.183 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.183 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.183 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.183 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.183 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.183 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.183 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.184 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.184 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.184 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.184 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.184 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.184 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.184 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.184 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.184 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.184 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.184 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.184 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.184 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.185 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.185 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.185 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.185 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.185 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.185 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.185 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.185 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-04-25T01:43:15.185 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-25T01:43:15.185 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.185 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.185 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.185 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.185 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.186 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.186 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.186 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-25T01:43:15.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-04-25T01:43:15.186 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-04-25T01:43:15.186 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.186 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-04-25T01:43:15.186 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.186 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.186 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.186 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.186 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.187 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-04-25T01:43:15.187 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.187 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.187 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.187 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.187 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.187 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.187 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.187 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.187 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.187 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.187 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.187 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.187 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.188 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.188 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.188 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.188 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.188 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.188 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.188 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.188 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.188 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.188 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.188 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.188 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.188 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.189 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.189 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.189 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.189 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.189 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.189 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.189 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.189 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.189 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.189 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.189 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.189 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.189 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.189 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.190 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.190 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.190 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.190 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.190 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.190 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.190 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.190 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.190 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.190 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.190 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.190 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.190 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.190 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.191 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.191 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.191 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.191 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.191 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.191 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.191 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.191 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.191 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.191 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.191 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.191 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.191 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.191 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.192 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.192 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.192 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.192 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-04-25T01:43:15.192 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.192 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.192 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.192 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.192 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.192 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.192 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.192 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.193 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.193 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.193 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.193 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.193 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.193 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.193 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.193 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.193 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.193 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.193 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.193 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.193 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.193 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.194 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.194 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.194 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.194 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.194 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.194 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.194 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.194 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.194 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.194 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.195 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.195 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.195 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.195 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.195 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.195 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.195 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.195 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.195 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.195 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.195 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.196 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.196 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.196 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.196 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.196 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.196 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.196 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.196 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.196 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.196 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.196 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.196 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.196 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.197 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.197 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.197 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.197 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.197 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.197 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.197 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.197 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.197 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.197 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.197 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.198 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.198 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.198 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-04-25T01:43:15.198 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.198 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.198 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.198 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.198 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.198 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.198 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.198 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.198 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.199 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.199 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.199 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.199 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.199 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.199 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.199 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.199 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.199 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.199 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.200 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.201 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.201 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.201 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.201 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.201 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.201 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.201 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.201 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.202 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.202 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.202 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.202 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.202 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.202 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.202 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.202 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.202 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.202 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.202 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.202 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.202 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.203 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.203 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.203 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.203 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.203 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.203 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.203 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.203 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.203 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.203 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.203 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.203 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.203 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.203 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.204 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.204 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-25T01:43:15.204 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-25T01:43:15.204 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.204 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.204 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.204 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-25T01:43:15.204 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.204 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.204 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.204 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.204 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.204 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.205 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.205 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-04-25T01:43:15.205 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-04-25T01:43:15.205 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:15.205 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.205 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.205 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-25T01:43:15.205 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-04-25T01:43:15.205 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.205 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.206 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-04-25T01:43:15.206 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T01:43:15.206 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.206 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.206 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.206 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.206 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-04-25T01:43:15.206 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.207 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.207 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.207 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-04-25T01:43:15.207 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-04-25T01:43:15.207 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.207 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.207 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.207 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.207 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.207 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.207 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.208 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T01:43:15.208 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.208 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.208 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.208 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-25T01:43:15.208 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-25T01:43:15.208 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.208 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.208 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.209 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.209 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.209 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.209 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.209 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.209 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.209 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.209 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.209 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.210 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.210 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.210 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.210 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.210 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.210 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.210 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.210 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.210 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.210 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.210 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.210 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.210 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.210 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.210 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.211 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.211 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.211 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.211 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.211 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.211 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.211 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.211 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.211 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.211 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.211 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.211 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.211 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.212 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.212 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.212 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.212 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.212 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.212 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.212 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.212 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.212 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.212 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.212 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.212 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.212 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.213 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.213 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.213 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.213 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.213 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.213 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.213 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.213 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.213 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.213 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.213 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.213 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.213 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.214 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.214 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.214 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.214 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.214 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.214 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.214 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.214 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.214 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.214 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-04-25T01:43:15.214 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-04-25T01:43:15.214 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.215 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.215 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-04-25T01:43:15.215 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.215 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.215 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.215 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-04-25T01:43:15.215 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.215 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.215 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.215 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-25T01:43:15.215 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-25T01:43:15.215 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.216 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.216 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.216 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-25T01:43:15.216 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.216 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.216 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.216 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-04-25T01:43:15.216 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.216 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.216 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.216 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.216 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.216 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-04-25T01:43:15.217 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.217 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.217 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.217 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-04-25T01:43:15.217 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-04-25T01:43:15.217 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.217 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.217 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.217 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.217 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.217 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.217 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.217 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.217 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.217 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.217 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.218 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-04-25T01:43:15.218 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-04-25T01:43:15.218 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.218 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.218 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.218 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.218 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.218 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.218 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.218 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T01:43:15.218 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T01:43:15.218 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.218 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.219 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.219 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.219 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.219 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.219 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.219 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.219 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.219 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.219 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.219 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.219 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.219 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.219 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.219 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.220 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.220 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.220 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.220 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.220 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.220 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.220 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.220 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.220 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.220 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.220 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.220 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.220 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.221 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.221 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.221 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.221 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.221 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.221 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.221 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.221 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.221 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.221 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.221 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.221 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.221 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.222 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.222 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.222 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.222 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.222 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.222 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.222 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.222 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.222 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.222 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.222 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.222 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.222 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.222 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.223 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.223 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.223 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.223 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.223 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.223 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.223 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.223 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.223 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.223 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.223 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.223 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.223 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-04-25T01:43:15.223 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-04-25T01:43:15.224 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.224 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.224 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.224 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.224 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.224 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.224 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.224 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.224 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.224 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.224 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.224 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T01:43:15.224 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T01:43:15.224 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.225 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.225 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.225 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-04-25T01:43:15.225 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.225 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.225 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.225 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.225 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.225 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.225 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.225 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-04-25T01:43:15.225 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-04-25T01:43:15.225 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:15.226 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.226 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.226 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-04-25T01:43:15.226 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.226 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.226 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.226 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.226 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.226 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.226 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.226 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-25T01:43:15.226 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-25T01:43:15.226 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.226 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.227 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-25T01:43:15.227 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.227 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.227 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.227 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T01:43:15.227 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.227 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.227 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.227 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-25T01:43:15.227 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-25T01:43:15.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.227 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.228 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-04-25T01:43:15.228 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.228 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.228 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.228 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T01:43:15.228 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.228 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.228 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.228 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-04-25T01:43:15.228 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-04-25T01:43:15.228 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.228 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.228 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-04-25T01:43:15.229 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.229 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.229 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.229 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.229 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.229 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.229 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.229 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-25T01:43:15.229 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-25T01:43:15.229 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.229 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.229 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.229 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-04-25T01:43:15.229 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.230 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.230 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.230 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-04-25T01:43:15.230 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-04-25T01:43:15.230 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.230 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.230 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.230 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.230 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.230 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-25T01:43:15.230 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-25T01:43:15.230 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.231 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.231 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.231 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-25T01:43:15.231 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-25T01:43:15.231 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-25T01:43:15.231 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.231 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.231 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.233 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.233 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.233 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.233 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.233 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.233 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-25T01:43:15.233 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.233 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.234 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.234 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.234 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.234 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.234 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.234 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.234 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.234 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.234 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.234 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.234 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.234 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.234 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.235 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.235 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.235 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.235 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.235 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.235 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.235 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.235 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.235 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.236 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.236 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.236 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.236 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.236 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.236 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.236 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.236 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.236 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.236 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.236 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.237 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.237 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.237 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.237 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.237 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.237 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.237 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.237 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.237 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.237 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.238 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.238 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.238 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.238 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.238 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.238 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.238 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.238 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.238 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.238 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.238 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.239 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.239 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.239 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.239 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.239 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.239 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.239 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.239 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.239 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.239 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.239 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.240 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-04-25T01:43:15.240 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-04-25T01:43:15.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.240 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.240 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:15.240 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.240 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.240 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.240 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.241 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.241 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.241 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.241 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-04-25T01:43:15.241 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T01:43:15.241 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.241 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.241 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.241 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.241 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.241 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.241 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.241 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.242 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.242 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.242 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.242 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-04-25T01:43:15.242 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-04-25T01:43:15.242 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.242 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.243 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.243 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.243 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.243 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.243 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.243 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.243 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.243 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.243 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.243 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.244 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.244 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.244 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-04-25T01:43:15.244 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.244 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.244 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.244 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-04-25T01:43:15.244 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-04-25T01:43:15.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.244 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.245 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.245 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.245 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.245 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.245 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.245 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.245 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.245 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.245 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-25T01:43:15.245 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-25T01:43:15.246 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.246 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.246 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.246 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.246 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.246 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.246 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.246 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.246 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.246 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.247 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.247 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.247 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.247 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.247 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.247 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.247 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.247 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.247 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.247 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.247 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.247 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.247 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.248 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.248 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.248 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.248 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.248 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.248 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.248 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.248 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.248 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.248 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.248 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.248 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.248 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.248 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.249 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.249 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.249 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.249 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.249 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.249 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.249 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.249 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.249 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.249 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.249 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.249 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.250 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.250 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.250 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.250 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.250 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.250 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.250 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.250 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.250 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.250 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.250 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.250 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.250 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.251 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.251 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.251 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.251 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.251 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.251 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.251 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.251 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.251 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.251 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.251 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.251 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.251 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.252 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-25T01:43:15.252 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-25T01:43:15.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.252 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.252 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.252 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.252 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.252 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.252 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.252 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.252 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.252 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.252 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.253 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-25T01:43:15.253 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-25T01:43:15.253 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.253 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.253 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.253 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.253 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.253 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.253 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.253 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.253 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.253 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.253 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.253 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.254 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.254 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.254 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-04-25T01:43:15.254 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.254 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.254 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.254 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.254 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.254 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.254 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.254 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.254 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.255 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.255 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.255 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.255 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.255 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.255 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.255 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.255 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.255 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.255 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.255 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.255 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.255 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.255 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.256 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.256 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.256 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.256 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.256 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.256 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.256 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.256 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.256 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.256 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.256 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.256 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.256 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.256 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.256 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.257 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.257 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.257 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.257 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.257 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.257 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.257 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.257 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.257 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.257 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.257 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.257 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.257 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.257 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.257 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.257 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.258 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.258 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.258 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.258 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.258 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.258 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.258 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.258 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.258 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.258 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.258 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.258 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.258 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.258 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.259 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-04-25T01:43:15.259 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.259 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.259 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.259 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.259 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.259 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.259 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.259 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.259 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.259 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.259 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.260 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.260 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.260 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.260 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.260 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.260 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.260 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.260 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.260 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.260 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.260 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.260 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.260 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.261 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.261 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.261 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.261 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.261 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.261 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.261 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.261 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.261 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.261 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.261 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.261 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.261 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.261 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.261 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.262 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.262 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.262 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.262 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.262 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.262 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.262 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.262 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.262 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.262 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.262 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.262 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.262 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.263 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.263 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.263 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.263 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.263 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.263 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.263 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.263 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.263 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.263 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.263 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.263 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.263 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.263 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.264 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.264 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-04-25T01:43:15.264 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-04-25T01:43:15.264 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.264 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.264 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.264 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-04-25T01:43:15.264 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.264 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.264 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.264 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T01:43:15.264 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.264 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.264 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.265 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.265 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.265 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-04-25T01:43:15.265 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.265 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.265 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.265 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.265 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.265 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.265 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.265 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.265 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.266 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.266 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.266 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.266 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.266 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.266 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.266 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.266 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.266 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.266 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.266 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.266 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.266 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.266 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.267 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.267 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.267 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.267 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.267 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.267 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.267 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.267 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.267 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.268 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.268 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.268 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.268 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.268 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.268 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.268 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.268 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.268 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.268 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.268 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.268 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.269 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.269 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.269 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.269 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.269 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.269 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.269 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.269 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.269 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.269 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.269 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.270 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.270 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.270 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.270 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.270 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.270 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.270 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.270 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.270 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.270 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.270 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.271 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.271 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.271 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-04-25T01:43:15.271 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-04-25T01:43:15.271 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.271 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.271 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.271 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.271 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.271 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.271 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.272 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.272 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.272 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.272 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.272 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-04-25T01:43:15.272 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-04-25T01:43:15.272 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.272 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.272 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.272 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-04-25T01:43:15.272 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.273 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.273 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.273 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.273 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.273 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.273 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.273 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-04-25T01:43:15.273 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-04-25T01:43:15.273 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.273 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.273 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.273 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.274 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.274 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.274 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.274 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.274 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.274 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.274 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.274 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-04-25T01:43:15.274 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-04-25T01:43:15.274 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.275 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.275 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.275 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.275 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.275 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.275 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.275 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.275 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.275 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.275 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.276 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-04-25T01:43:15.276 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-04-25T01:43:15.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:15.276 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.276 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.276 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.276 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.276 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.276 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.276 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.276 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.276 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.277 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.277 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-04-25T01:43:15.277 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-04-25T01:43:15.277 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.277 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.277 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.277 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.277 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.277 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.277 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.277 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.277 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.277 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.277 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.278 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-04-25T01:43:15.278 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-04-25T01:43:15.278 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.278 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.278 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.278 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.278 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.278 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.278 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.278 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.278 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.278 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.278 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.279 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-04-25T01:43:15.279 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-04-25T01:43:15.279 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.279 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.279 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.279 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.279 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.279 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.280 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.280 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.280 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.280 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.280 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.280 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-04-25T01:43:15.280 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-04-25T01:43:15.280 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.281 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.281 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.281 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.281 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-25T01:43:15.281 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-04-25T01:43:15.281 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.281 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.281 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.281 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.281 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.282 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-04-25T01:43:15.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-04-25T01:43:15.282 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.282 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.282 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.282 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.282 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.282 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.282 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.283 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.283 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.283 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.283 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.283 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-04-25T01:43:15.283 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-04-25T01:43:15.283 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.283 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.283 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.283 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.283 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.283 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.283 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.284 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.284 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.284 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.284 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.284 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-04-25T01:43:15.284 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-04-25T01:43:15.284 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.284 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.284 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.284 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.284 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.284 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.284 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.284 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.285 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.285 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.285 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.285 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.285 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.285 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-25T01:43:15.285 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.285 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.285 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.285 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-04-25T01:43:15.285 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-04-25T01:43:15.285 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.286 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.286 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.286 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.286 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.286 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.286 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.286 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T01:43:15.286 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.286 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.286 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.286 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-04-25T01:43:15.286 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-04-25T01:43:15.286 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.287 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.287 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.287 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.287 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.287 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.287 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.287 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-04-25T01:43:15.287 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.287 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.287 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.287 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.287 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.287 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.287 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.288 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.288 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.288 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.288 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.288 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.288 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.288 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.288 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.288 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.288 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.288 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.288 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.288 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.288 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.288 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.288 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.289 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.289 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.289 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.289 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.289 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.289 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.289 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.289 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.289 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.289 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.289 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.289 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.289 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.289 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.289 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.290 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.290 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.290 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.290 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.290 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.290 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.290 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.290 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.290 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.290 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.290 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.290 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.290 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.290 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.291 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.291 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.291 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.291 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.291 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.291 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.291 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.291 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.291 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.291 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.291 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.291 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.291 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.291 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.291 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.292 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.292 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.292 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.292 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.292 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.292 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-04-25T01:43:15.292 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.292 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.292 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.292 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.292 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.292 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.292 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.292 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.293 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.293 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.293 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.293 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.293 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.293 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.293 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.293 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.293 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.293 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.293 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.293 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.293 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.293 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.294 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.294 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.294 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.294 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.294 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.294 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.294 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.294 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.294 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.294 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.294 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.294 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.294 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.294 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.294 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.295 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.295 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.295 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.295 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.295 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.295 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.295 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.295 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.295 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.295 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.295 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.295 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.295 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.295 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.296 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.296 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.296 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.296 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.296 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.296 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.296 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.296 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.296 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.296 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.296 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.296 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.296 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.296 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.297 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.297 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.297 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.297 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.297 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.297 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.297 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.297 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-04-25T01:43:15.297 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.297 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.297 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.297 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.297 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.298 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.298 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.298 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.298 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.298 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.298 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.298 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.298 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.298 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.298 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.298 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.298 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.298 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.298 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.299 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.299 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.299 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.299 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.299 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.299 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.299 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.299 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.299 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.299 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.299 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.299 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.299 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.299 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.299 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.299 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.300 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.300 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.300 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.300 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.300 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.300 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.300 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.300 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.300 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.300 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.300 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.300 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.300 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.300 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.300 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.300 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.301 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.301 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.301 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.301 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.301 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.301 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.301 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.301 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.301 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.301 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.301 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.301 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.301 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.301 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.301 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.301 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.302 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.302 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.302 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.302 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-04-25T01:43:15.302 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.302 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.302 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.302 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-04-25T01:43:15.302 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-04-25T01:43:15.302 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.302 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.303 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.303 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T01:43:15.303 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.303 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.303 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.303 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.303 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.303 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.303 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.303 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-25T01:43:15.303 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-25T01:43:15.303 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.303 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.303 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.304 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-04-25T01:43:15.304 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.304 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.304 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.304 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.304 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.304 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.304 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.304 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.304 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.304 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.305 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.305 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.305 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.305 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.305 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.305 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.305 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.305 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.305 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.305 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.305 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.305 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.306 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.306 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.306 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.306 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.306 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.306 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.306 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.306 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.306 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.306 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.306 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.307 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.307 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.307 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.307 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.307 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.307 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.307 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.307 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.307 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.307 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.308 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.308 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.308 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.308 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.308 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.308 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.308 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.308 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.308 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.308 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.308 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.308 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.308 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.308 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.309 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.309 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.309 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.309 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.309 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.309 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.309 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.309 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.309 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.310 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.310 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.310 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.310 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.310 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.310 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.310 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.310 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.310 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.310 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-04-25T01:43:15.310 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.311 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.311 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.311 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-04-25T01:43:15.311 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-04-25T01:43:15.311 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.311 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.311 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.311 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.311 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.311 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.311 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.312 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T01:43:15.312 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.312 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.312 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.312 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-04-25T01:43:15.312 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-04-25T01:43:15.312 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.312 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.312 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.312 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.312 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.313 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.313 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.313 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T01:43:15.313 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.313 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.313 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.313 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-04-25T01:43:15.313 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-04-25T01:43:15.313 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.313 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.313 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.314 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.314 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.314 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.314 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.314 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T01:43:15.314 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.314 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.314 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.314 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-04-25T01:43:15.315 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-04-25T01:43:15.315 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.315 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.315 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.315 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.315 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.315 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.315 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.315 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T01:43:15.315 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.316 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.316 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.316 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.316 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-04-25T01:43:15.316 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-04-25T01:43:15.316 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.316 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.316 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.316 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.316 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.317 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.317 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.317 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T01:43:15.317 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.317 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.317 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.317 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-04-25T01:43:15.317 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-04-25T01:43:15.317 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.317 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.318 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.318 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.318 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.318 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.318 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.318 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.318 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.318 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.318 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.318 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-04-25T01:43:15.318 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-04-25T01:43:15.318 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.318 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.319 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.319 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.319 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.319 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.319 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.319 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.319 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.319 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.319 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.319 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-04-25T01:43:15.319 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-04-25T01:43:15.319 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.319 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.320 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.320 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T01:43:15.320 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.320 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.320 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.320 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.320 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.320 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.320 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.320 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-25T01:43:15.320 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-25T01:43:15.320 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.320 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.320 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.321 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.321 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.321 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.321 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.321 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.321 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.321 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.321 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-25T01:43:15.321 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-25T01:43:15.321 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.321 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.321 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.321 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-04-25T01:43:15.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-04-25T01:43:15.322 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.322 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.322 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.322 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.322 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.322 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.322 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-04-25T01:43:15.322 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-04-25T01:43:15.322 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.322 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.322 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.322 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-04-25T01:43:15.322 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.323 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.323 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.323 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.323 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.323 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.323 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.323 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-04-25T01:43:15.323 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-04-25T01:43:15.323 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.323 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.323 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.323 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.323 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.324 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.324 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.324 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.324 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.324 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.324 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.324 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-04-25T01:43:15.324 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-04-25T01:43:15.324 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.324 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.324 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.324 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.324 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.324 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.325 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.325 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.325 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.325 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.325 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.325 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.325 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.325 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.325 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.325 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.325 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.325 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.325 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.325 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.326 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.326 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.326 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.326 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.326 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.326 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.326 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.326 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.326 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.326 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.326 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.326 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.326 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.327 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.327 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.327 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.327 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.327 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.327 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.327 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.327 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.327 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.327 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.327 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.327 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.328 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.328 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.328 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.328 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.328 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.328 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.328 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.328 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.328 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.328 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.328 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.328 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.328 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.328 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.328 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.329 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.329 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.329 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.329 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.329 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.329 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.329 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.329 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.329 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.329 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.329 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.329 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.329 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.329 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.329 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.330 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-04-25T01:43:15.330 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-04-25T01:43:15.330 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.330 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.330 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.330 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.330 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.330 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.330 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.330 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.330 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.330 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.330 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.331 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-04-25T01:43:15.331 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-04-25T01:43:15.331 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.331 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.331 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.331 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.331 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.331 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.331 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.331 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.331 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.331 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.332 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.332 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-04-25T01:43:15.332 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-04-25T01:43:15.332 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.332 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.332 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.332 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T01:43:15.332 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.332 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.332 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.332 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.332 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.332 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.333 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.333 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.333 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.333 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-04-25T01:43:15.333 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.333 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.333 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.333 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.333 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.333 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.333 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.333 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.334 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.334 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.334 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.334 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.334 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.334 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.334 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.334 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.334 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.334 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.334 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.334 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.334 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.335 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.335 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.335 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.335 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.335 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.335 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.335 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.335 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.335 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.335 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.335 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.335 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.335 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.335 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.336 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.336 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.336 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.336 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.336 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.336 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.336 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.336 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.336 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.336 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.336 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.336 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.336 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.336 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.338 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.338 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.338 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.338 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.338 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.338 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.338 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.338 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-25T01:43:15.338 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.338 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.338 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.339 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.339 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.339 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.339 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.339 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.339 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.339 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.339 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.339 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.339 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.340 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.340 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.340 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.340 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.340 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.340 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.340 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.340 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.340 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.340 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.340 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.340 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.340 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.340 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.341 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.341 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.341 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.341 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.341 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.341 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.341 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.341 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.342 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.342 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.342 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.342 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.342 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.342 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.342 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.342 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.342 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.342 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.342 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.342 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.342 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.342 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.343 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.343 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.343 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.343 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.343 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.343 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.343 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.343 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.344 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.344 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.344 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.344 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.344 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.344 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.344 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.344 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.344 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.344 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.344 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.344 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-04-25T01:43:15.344 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-04-25T01:43:15.345 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.345 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.345 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.345 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-04-25T01:43:15.345 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.345 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.345 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.345 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T01:43:15.345 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.345 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.345 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.346 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-04-25T01:43:15.346 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-04-25T01:43:15.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.346 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.346 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.346 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.346 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.346 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.346 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-04-25T01:43:15.346 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.347 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.347 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.347 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-04-25T01:43:15.347 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-04-25T01:43:15.347 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.347 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.347 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.347 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-25T01:43:15.347 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.347 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.348 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.348 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T01:43:15.348 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.348 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.348 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.348 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-04-25T01:43:15.348 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T01:43:15.348 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.348 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.349 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.349 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.349 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.349 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.349 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T01:43:15.349 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.349 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.349 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.349 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.350 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.350 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-04-25T01:43:15.350 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.350 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.350 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.350 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-04-25T01:43:15.350 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-04-25T01:43:15.350 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-25T01:43:15.350 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.350 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-04-25T01:43:15.351 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.351 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.351 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.351 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.351 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.351 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.351 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.351 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-04-25T01:43:15.351 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-04-25T01:43:15.351 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.351 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.351 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.352 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.352 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.352 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.352 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.352 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.352 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.352 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.352 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.352 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-25T01:43:15.352 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-25T01:43:15.352 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.352 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.353 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.353 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-25T01:43:15.353 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.353 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.353 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.353 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.353 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.353 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.354 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.354 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.354 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.354 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.354 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.354 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.354 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.354 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.354 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.354 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.355 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.355 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.355 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.355 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.355 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.355 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.355 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.355 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.355 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.355 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.355 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.355 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.355 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.356 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.356 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.356 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.356 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.356 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.356 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.356 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.356 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.356 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.356 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.356 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.356 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.357 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.357 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.357 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.357 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.357 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.357 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.357 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.357 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.357 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.357 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.357 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.357 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.357 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.357 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.357 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.357 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.358 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.358 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.358 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.358 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.358 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.358 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.358 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.358 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.358 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.358 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.358 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.358 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.358 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-04-25T01:43:15.359 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-04-25T01:43:15.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.359 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.359 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.359 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.359 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.359 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.359 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.359 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.359 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.359 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.359 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.359 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-04-25T01:43:15.359 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-04-25T01:43:15.360 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.360 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.360 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.360 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-04-25T01:43:15.360 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.360 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.360 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.360 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.360 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.360 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.360 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.360 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-04-25T01:43:15.360 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-04-25T01:43:15.360 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.360 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.360 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.361 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T01:43:15.361 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.361 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.361 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.361 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.361 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.361 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.361 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.361 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.361 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.361 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.361 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-25T01:43:15.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.361 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 25151, 2024-04-25T01:43:15.362 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "y", 2024-04-25T01:43:15.362 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540701547738038271, 2024-04-25T01:43:15.362 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-04-25T01:43:15.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.362 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-04-25T01:43:15.362 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.362 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.362 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.362 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-25T01:43:15.362 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-25T01:43:15.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.362 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.362 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-25T01:43:15.362 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.363 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.363 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.363 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T01:43:15.363 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.363 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.363 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.363 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.363 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.363 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.363 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.363 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.363 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.363 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.363 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.364 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.364 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.364 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.364 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.364 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.364 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.364 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.364 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.364 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.364 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.364 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.364 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.365 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.365 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.365 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.365 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.365 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.365 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.365 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.365 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.365 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.365 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.365 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.365 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.365 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.365 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.366 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.366 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.366 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.366 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.366 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.366 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.366 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.366 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.366 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.366 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.366 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.366 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.366 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.367 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.367 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.367 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.367 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.367 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.367 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.367 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.367 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.367 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.367 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.367 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.367 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.367 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.367 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.368 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.368 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.368 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.368 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.368 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.368 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-04-25T01:43:15.368 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-04-25T01:43:15.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.368 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.368 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.368 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.368 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.368 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.368 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.368 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-04-25T01:43:15.369 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.369 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.369 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.369 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-04-25T01:43:15.369 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-04-25T01:43:15.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.369 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.369 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.369 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-04-25T01:43:15.369 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.369 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.369 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.369 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-04-25T01:43:15.370 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.370 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.370 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.370 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-04-25T01:43:15.370 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-04-25T01:43:15.370 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.370 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.370 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.370 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.370 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.370 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.370 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.370 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-04-25T01:43:15.371 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.371 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.371 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.371 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-04-25T01:43:15.371 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-04-25T01:43:15.371 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.371 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.371 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.371 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.371 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.371 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.371 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.371 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-04-25T01:43:15.371 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.371 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.372 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.372 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-04-25T01:43:15.372 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-04-25T01:43:15.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.372 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.372 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.372 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-04-25T01:43:15.372 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.372 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.372 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.372 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-04-25T01:43:15.372 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.372 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.373 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.373 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-04-25T01:43:15.373 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-04-25T01:43:15.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.373 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.373 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.373 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.373 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.373 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.373 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.373 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-04-25T01:43:15.373 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.373 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.373 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-04-25T01:43:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-04-25T01:43:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T01:43:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.375 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-04-25T01:43:15.375 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-04-25T01:43:15.375 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.375 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.375 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.375 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.375 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.375 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.375 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.375 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-04-25T01:43:15.376 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.376 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.376 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.376 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.376 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.376 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.376 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-04-25T01:43:15.376 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.376 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.376 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.376 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-04-25T01:43:15.376 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-04-25T01:43:15.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.376 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.377 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.377 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.377 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.377 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.377 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.377 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T01:43:15.377 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.377 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.377 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.377 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-04-25T01:43:15.377 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-04-25T01:43:15.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.378 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.378 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.378 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-04-25T01:43:15.378 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.378 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.378 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.378 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T01:43:15.378 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T01:43:15.378 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.378 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.379 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-04-25T01:43:15.379 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-04-25T01:43:15.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-25T01:43:15.379 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.379 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.379 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-25T01:43:15.379 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-25T01:43:15.379 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-25T01:43:15.379 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.379 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T01:43:15.379 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T01:43:15.379 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.380 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.380 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-04-25T01:43:15.380 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-04-25T01:43:15.380 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-25T01:43:15.380 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.380 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.380 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-04-25T01:43:15.380 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-25T01:43:15.380 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-04-25T01:43:15.380 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.381 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T01:43:15.381 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.381 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.381 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.381 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-04-25T01:43:15.381 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-04-25T01:43:15.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.381 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.381 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T01:43:15.381 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.382 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.382 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.382 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T01:43:15.382 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T01:43:15.382 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.382 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.382 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-04-25T01:43:15.382 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-04-25T01:43:15.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:15.382 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.382 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.383 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-04-25T01:43:15.383 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-04-25T01:43:15.383 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-04-25T01:43:15.383 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.383 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-04-25T01:43:15.383 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T01:43:15.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.383 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.383 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-04-25T01:43:15.383 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-04-25T01:43:15.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.384 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.384 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.384 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-04-25T01:43:15.384 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.384 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.384 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.384 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T01:43:15.384 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T01:43:15.384 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.384 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.385 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-04-25T01:43:15.385 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-04-25T01:43:15.385 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-25T01:43:15.385 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.385 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.385 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-25T01:43:15.385 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-25T01:43:15.385 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-25T01:43:15.385 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.385 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T01:43:15.386 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T01:43:15.386 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.386 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.386 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.386 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.386 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.386 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.386 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.387 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.387 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.387 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.387 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.387 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.387 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.387 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.387 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.387 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.387 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.388 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.388 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.388 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.388 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.388 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.388 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.388 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.388 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.389 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.389 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.389 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.389 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.389 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.389 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.389 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.389 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.389 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.389 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.389 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.389 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.390 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.390 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.390 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.390 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.390 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.390 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.390 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.390 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.390 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.390 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.390 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.390 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.390 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.390 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.391 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.391 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.391 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.391 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.391 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.391 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.391 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.391 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.391 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.391 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.391 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.391 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.391 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-04-25T01:43:15.392 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-04-25T01:43:15.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:15.392 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.392 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.392 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-25T01:43:15.392 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.392 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.392 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.392 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T01:43:15.392 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.392 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.392 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.392 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-25T01:43:15.393 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-25T01:43:15.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.393 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.393 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.393 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-04-25T01:43:15.393 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.393 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.393 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.393 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-04-25T01:43:15.393 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-04-25T01:43:15.393 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-04-25T01:43:15.393 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-04-25T01:43:15.393 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-04-25T01:43:15.393 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.393 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-04-25T01:43:15.394 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.394 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.394 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.394 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-04-25T01:43:15.394 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-04-25T01:43:15.394 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.394 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.394 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.394 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.394 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.394 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.394 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.394 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T01:43:15.395 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.395 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.395 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.395 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-25T01:43:15.395 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-25T01:43:15.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.395 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.395 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.395 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-25T01:43:15.395 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.395 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.395 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.395 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T01:43:15.395 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.396 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.396 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.396 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-04-25T01:43:15.396 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-04-25T01:43:15.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.396 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.396 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.396 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-25T01:43:15.396 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-25T01:43:15.396 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.396 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.396 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T01:43:15.396 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T01:43:15.397 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.397 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.397 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-04-25T01:43:15.397 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-04-25T01:43:15.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-25T01:43:15.397 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.397 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:15.397 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.397 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.397 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.397 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T01:43:15.397 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.397 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.397 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.398 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.398 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.398 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-04-25T01:43:15.398 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.398 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.398 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.398 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-04-25T01:43:15.398 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-04-25T01:43:15.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:15.398 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.398 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.398 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-25T01:43:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T01:43:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-04-25T01:43:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-04-25T01:43:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-04-25T01:43:15.399 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.400 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.400 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.400 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T01:43:15.400 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.400 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.400 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.400 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-04-25T01:43:15.400 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-04-25T01:43:15.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.400 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.400 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-04-25T01:43:15.400 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.401 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.401 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.401 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T01:43:15.401 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.401 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.401 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.401 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-04-25T01:43:15.401 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-04-25T01:43:15.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.401 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.401 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.401 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.401 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.401 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.402 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.402 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T01:43:15.402 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T01:43:15.402 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.402 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.402 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-04-25T01:43:15.402 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-04-25T01:43:15.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.402 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.402 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.402 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-25T01:43:15.402 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.402 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.403 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.403 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T01:43:15.403 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.403 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.403 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.403 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-04-25T01:43:15.403 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T01:43:15.403 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:15.403 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.403 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.403 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-04-25T01:43:15.403 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.403 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.404 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.404 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T01:43:15.404 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.404 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.404 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.404 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-04-25T01:43:15.404 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-04-25T01:43:15.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.404 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.404 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.404 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.404 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.404 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.404 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.405 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-04-25T01:43:15.405 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-04-25T01:43:15.405 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-04-25T01:43:15.405 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.405 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T01:43:15.405 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.405 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.405 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.405 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-04-25T01:43:15.405 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-04-25T01:43:15.405 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.405 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.405 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.405 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.406 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.406 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.406 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.406 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T01:43:15.406 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.406 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.406 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.406 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-04-25T01:43:15.406 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-04-25T01:43:15.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.406 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.406 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.406 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.407 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.407 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.407 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T01:43:15.407 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.407 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.407 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.407 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-04-25T01:43:15.407 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-04-25T01:43:15.407 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.407 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.407 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.407 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.407 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.407 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.408 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.408 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T01:43:15.408 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.408 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.408 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.408 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-04-25T01:43:15.408 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-04-25T01:43:15.408 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.408 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.408 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.408 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T01:43:15.408 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.408 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.408 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.408 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-25T01:43:15.409 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.409 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.409 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.409 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-04-25T01:43:15.409 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-04-25T01:43:15.409 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.409 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.409 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.409 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T01:43:15.409 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.409 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.409 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.409 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-04-25T01:43:15.409 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.410 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.410 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.410 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-04-25T01:43:15.410 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-04-25T01:43:15.410 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.410 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.410 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.410 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T01:43:15.411 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.411 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.411 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.411 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-04-25T01:43:15.411 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.411 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.411 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.411 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-04-25T01:43:15.411 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-04-25T01:43:15.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.411 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.411 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.411 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T01:43:15.411 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.411 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.411 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.412 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-25T01:43:15.412 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.412 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.412 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.412 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-04-25T01:43:15.412 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-04-25T01:43:15.412 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.412 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.412 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.413 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T01:43:15.413 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.413 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.413 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.413 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-04-25T01:43:15.413 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.413 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.413 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.413 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-04-25T01:43:15.413 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-04-25T01:43:15.413 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.413 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.414 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.414 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T01:43:15.414 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.414 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.414 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.414 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-04-25T01:43:15.414 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.414 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.414 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.415 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-04-25T01:43:15.415 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-04-25T01:43:15.415 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.415 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.415 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.415 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T01:43:15.415 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.415 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.415 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.415 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-04-25T01:43:15.415 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.415 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.415 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.415 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-04-25T01:43:15.416 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-04-25T01:43:15.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.416 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.416 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.416 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T01:43:15.416 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.416 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.416 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.416 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-04-25T01:43:15.417 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.417 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.417 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.417 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-04-25T01:43:15.417 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-04-25T01:43:15.417 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.417 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.417 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.417 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T01:43:15.417 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.417 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.417 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.418 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-04-25T01:43:15.418 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.418 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.418 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.418 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-04-25T01:43:15.418 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-04-25T01:43:15.418 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.418 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.419 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T01:43:15.419 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.419 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.419 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.419 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-04-25T01:43:15.419 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.419 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.419 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.419 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-04-25T01:43:15.420 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-04-25T01:43:15.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.420 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.420 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.420 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T01:43:15.420 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.420 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.420 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.420 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-25T01:43:15.420 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.420 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.421 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.421 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-04-25T01:43:15.421 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-04-25T01:43:15.421 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.421 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.421 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.421 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T01:43:15.421 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.421 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.422 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.422 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-04-25T01:43:15.422 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.422 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.422 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.422 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-04-25T01:43:15.422 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-04-25T01:43:15.422 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.422 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.422 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.423 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T01:43:15.423 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.423 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.423 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.423 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-25T01:43:15.423 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.423 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.423 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.423 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-04-25T01:43:15.423 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-04-25T01:43:15.423 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.423 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.423 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.423 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T01:43:15.424 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.424 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.424 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.424 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-04-25T01:43:15.424 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.424 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.424 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.424 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-04-25T01:43:15.424 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-04-25T01:43:15.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.424 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.425 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T01:43:15.425 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.425 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.425 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.425 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-04-25T01:43:15.425 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.425 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.425 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.425 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-04-25T01:43:15.425 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-04-25T01:43:15.425 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.426 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.426 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.426 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T01:43:15.426 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.426 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.426 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.426 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-04-25T01:43:15.426 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.426 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.426 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.426 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-04-25T01:43:15.426 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-04-25T01:43:15.427 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.427 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.427 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.427 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.427 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.427 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.427 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.427 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T01:43:15.427 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.427 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.427 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.427 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-04-25T01:43:15.427 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-04-25T01:43:15.427 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.428 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.428 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.428 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-25T01:43:15.428 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.428 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.428 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.428 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T01:43:15.428 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.428 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.428 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.428 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-04-25T01:43:15.428 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T01:43:15.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.429 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.429 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.429 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-04-25T01:43:15.429 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.429 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.429 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.429 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T01:43:15.429 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.429 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.429 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.429 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-04-25T01:43:15.429 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-04-25T01:43:15.429 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.430 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.430 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-04-25T01:43:15.430 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.430 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.430 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.430 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T01:43:15.430 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.430 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.430 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.430 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-04-25T01:43:15.430 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-04-25T01:43:15.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.430 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.431 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.431 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-04-25T01:43:15.431 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.431 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.431 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.431 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-04-25T01:43:15.431 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.431 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.431 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.431 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-04-25T01:43:15.431 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-04-25T01:43:15.431 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.431 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.432 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.432 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.432 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.432 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.432 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.432 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T01:43:15.432 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.432 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.432 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.432 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-04-25T01:43:15.432 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-04-25T01:43:15.432 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.432 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.432 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.432 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-25T01:43:15.433 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.433 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.433 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.433 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-04-25T01:43:15.433 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.433 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.433 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.433 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-04-25T01:43:15.433 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-04-25T01:43:15.433 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.433 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.433 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.433 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T01:43:15.434 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.434 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.434 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.434 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T01:43:15.434 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.434 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.434 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.434 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-04-25T01:43:15.434 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-04-25T01:43:15.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.434 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.434 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.434 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-25T01:43:15.435 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.435 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.435 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.435 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-04-25T01:43:15.435 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.435 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.435 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.435 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-04-25T01:43:15.435 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-04-25T01:43:15.435 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.435 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.435 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.435 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.435 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.436 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.436 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.436 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T01:43:15.436 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.436 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.436 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.436 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-04-25T01:43:15.436 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-04-25T01:43:15.436 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.436 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.436 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.436 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.436 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.436 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.436 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.437 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T01:43:15.437 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.437 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.437 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.437 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.438 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.438 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.438 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.438 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.438 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.438 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.438 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.438 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.438 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.438 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.438 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.438 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.438 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.439 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.439 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.439 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.439 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.439 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.439 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.439 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-04-25T01:43:15.439 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-04-25T01:43:15.439 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.439 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.439 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.439 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.439 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.440 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.440 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.440 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T01:43:15.440 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.440 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.440 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.440 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.440 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.440 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.440 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.440 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.440 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.440 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.441 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.441 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T01:43:15.441 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.441 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.441 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.441 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.441 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.441 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.441 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.441 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.441 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.441 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.441 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.442 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.442 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.442 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.442 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.442 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.442 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.442 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.442 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.442 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.442 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.442 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.442 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.442 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.443 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.443 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.443 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.443 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.443 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.443 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.443 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.443 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.443 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.443 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.443 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.443 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.443 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T01:43:15.444 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T01:43:15.444 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.444 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.444 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.444 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.444 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.444 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.444 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.444 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T01:43:15.444 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.444 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.444 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.444 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T01:43:15.444 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T01:43:15.444 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.445 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.445 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.445 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-04-25T01:43:15.445 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.445 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.445 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.445 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T01:43:15.445 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.445 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.445 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.445 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-04-25T01:43:15.445 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-04-25T01:43:15.445 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.445 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.445 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.446 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:15.446 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.446 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.446 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.446 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-04-25T01:43:15.446 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.446 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.446 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.446 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-04-25T01:43:15.446 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-04-25T01:43:15.446 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.446 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.446 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.447 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:15.447 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.447 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.447 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.447 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T01:43:15.447 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.447 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.447 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.447 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-04-25T01:43:15.448 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-04-25T01:43:15.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.448 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.448 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.448 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.448 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.448 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.448 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.448 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-04-25T01:43:15.448 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.448 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.448 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.448 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-25T01:43:15.449 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-25T01:43:15.449 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.449 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.449 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.449 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-04-25T01:43:15.449 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.449 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.449 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.449 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-04-25T01:43:15.449 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-04-25T01:43:15.450 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.450 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T01:43:15.450 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.450 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.450 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.450 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-04-25T01:43:15.450 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-04-25T01:43:15.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.450 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.450 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.450 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-04-25T01:43:15.451 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.451 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.451 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.451 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T01:43:15.451 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.451 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.451 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.451 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-04-25T01:43:15.451 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-04-25T01:43:15.451 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.451 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.451 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.452 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T01:43:15.452 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.452 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.452 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.452 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T01:43:15.452 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.452 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.452 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.452 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-04-25T01:43:15.453 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-04-25T01:43:15.453 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.453 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.453 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.453 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.453 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.453 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.453 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.453 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T01:43:15.453 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.453 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.453 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.454 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-04-25T01:43:15.454 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-04-25T01:43:15.454 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.454 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.454 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.454 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.454 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.454 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.454 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.454 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T01:43:15.454 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.455 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.455 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.455 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-04-25T01:43:15.455 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-04-25T01:43:15.455 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.455 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.455 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.455 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.455 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.456 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.456 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.456 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T01:43:15.456 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.456 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.456 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.456 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-04-25T01:43:15.456 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-04-25T01:43:15.456 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.456 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.456 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.457 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.457 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.457 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.457 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.457 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T01:43:15.457 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.457 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.457 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.457 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-04-25T01:43:15.457 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-04-25T01:43:15.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.458 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.458 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.458 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.458 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.458 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.458 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.458 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T01:43:15.458 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.458 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.459 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.459 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-04-25T01:43:15.459 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-04-25T01:43:15.459 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.459 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.459 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.459 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-04-25T01:43:15.459 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.460 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.460 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.460 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-04-25T01:43:15.460 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.460 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.460 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.460 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-04-25T01:43:15.460 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-04-25T01:43:15.460 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.460 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.460 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.461 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.461 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.461 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.461 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.461 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T01:43:15.461 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.461 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.461 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.461 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-04-25T01:43:15.461 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-04-25T01:43:15.461 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.461 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.461 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.461 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.461 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.462 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.462 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.462 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T01:43:15.462 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.462 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.462 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.462 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-04-25T01:43:15.462 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-04-25T01:43:15.462 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.462 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.462 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.462 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.462 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.462 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.463 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.463 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T01:43:15.463 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.463 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.463 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.463 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-04-25T01:43:15.463 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-04-25T01:43:15.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.463 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.463 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.463 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.463 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.464 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.464 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T01:43:15.464 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.464 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.464 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.464 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-04-25T01:43:15.464 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-04-25T01:43:15.464 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.464 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.464 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.464 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.464 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.464 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.464 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.464 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T01:43:15.465 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.465 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.465 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.465 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-04-25T01:43:15.465 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-04-25T01:43:15.465 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.465 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.465 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.465 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.465 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.465 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.465 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.465 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T01:43:15.465 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.466 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.466 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.466 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.466 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.466 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.466 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-04-25T01:43:15.466 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.466 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.466 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.466 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.466 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.466 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.466 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.466 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.467 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.467 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.467 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.467 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.467 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.467 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.467 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.467 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.467 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.467 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.467 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.467 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.467 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.467 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.467 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.468 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.468 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.468 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.468 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.468 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.468 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.468 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.468 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.468 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.468 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.468 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.468 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.468 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.469 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.469 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.469 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.469 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.469 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.469 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.469 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.469 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.469 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.469 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.469 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.469 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.469 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.469 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.470 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.470 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.470 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.470 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.470 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.470 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.470 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.470 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.470 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.470 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.470 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.470 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.470 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.470 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.471 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.471 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.471 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.471 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.471 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.471 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-04-25T01:43:15.471 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-04-25T01:43:15.471 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.471 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.471 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.471 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-04-25T01:43:15.471 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.471 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.471 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.472 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T01:43:15.472 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.472 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.472 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.472 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-04-25T01:43:15.472 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-04-25T01:43:15.472 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.472 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.472 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.472 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-04-25T01:43:15.472 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.472 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.472 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.472 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T01:43:15.472 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.473 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.473 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.473 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.473 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.473 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.473 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-04-25T01:43:15.473 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.473 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.473 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.473 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T01:43:15.473 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T01:43:15.474 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.474 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.474 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.474 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:15.474 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.474 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.474 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.474 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.474 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.474 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.474 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.474 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T01:43:15.474 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T01:43:15.474 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.475 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.475 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.475 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.475 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.475 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.475 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.475 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.475 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.475 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.475 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.475 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T01:43:15.475 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T01:43:15.475 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.475 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.476 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.476 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.476 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.476 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.476 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.476 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.476 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.476 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.476 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.476 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-04-25T01:43:15.476 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T01:43:15.476 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.476 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.476 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.476 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.477 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.477 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.477 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.477 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.477 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.477 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.477 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.477 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T01:43:15.477 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T01:43:15.477 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.477 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.477 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.477 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.477 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.477 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.478 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.478 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.478 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.478 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.478 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.478 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T01:43:15.478 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T01:43:15.478 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.478 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.478 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.478 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.478 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.478 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.479 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.479 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.479 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.479 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.479 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.479 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T01:43:15.479 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T01:43:15.479 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.479 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.479 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.479 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.479 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.479 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.479 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.480 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.480 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.480 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.480 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.480 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T01:43:15.480 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T01:43:15.480 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.480 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.480 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.480 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.480 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.480 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.480 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.480 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.480 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.481 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.481 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.481 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T01:43:15.481 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T01:43:15.481 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.481 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.481 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.481 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.481 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.481 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.481 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.481 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.482 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.482 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.482 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.482 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T01:43:15.482 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T01:43:15.482 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.482 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.482 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.483 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.483 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.483 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.483 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.483 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.483 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.483 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.483 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.483 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T01:43:15.483 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T01:43:15.483 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.483 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.483 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.483 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.484 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.484 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.484 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.484 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.484 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.484 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.484 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.484 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T01:43:15.484 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T01:43:15.484 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T01:43:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T01:43:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.487 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T01:43:15.487 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T01:43:15.487 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.487 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.487 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.487 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.487 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.487 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.487 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.487 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.487 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.487 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.488 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.488 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T01:43:15.488 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T01:43:15.488 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.488 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.488 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.488 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-25T01:43:15.488 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.488 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.488 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.489 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.489 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.489 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.489 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.489 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T01:43:15.489 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T01:43:15.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.489 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.489 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.489 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.489 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.489 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.490 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.490 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.490 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.490 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.490 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.490 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.490 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-04-25T01:43:15.490 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-04-25T01:43:15.490 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.490 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.490 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.491 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.491 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.491 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.491 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.491 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.491 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.491 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.491 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.491 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T01:43:15.491 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T01:43:15.492 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.492 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.492 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.492 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-25T01:43:15.492 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.492 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.492 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.492 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.493 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.493 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.493 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.493 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T01:43:15.493 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T01:43:15.493 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.493 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.493 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.493 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.494 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.494 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.494 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.494 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.494 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.494 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.494 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.495 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T01:43:15.495 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T01:43:15.495 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.495 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.495 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.495 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.495 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.495 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.495 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.495 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.495 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.495 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.495 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.496 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T01:43:15.496 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T01:43:15.496 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.496 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.496 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.496 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.496 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.496 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.496 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.496 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.496 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.496 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.496 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.496 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T01:43:15.496 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T01:43:15.497 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.497 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.497 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.497 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.497 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.497 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.497 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.497 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.497 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.497 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.497 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.497 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T01:43:15.497 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T01:43:15.497 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.498 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.498 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.498 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.498 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.498 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.498 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.498 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.498 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.498 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.498 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.498 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T01:43:15.498 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T01:43:15.498 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.498 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.499 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.499 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.499 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.499 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.499 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.499 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.499 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.499 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.499 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.499 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T01:43:15.499 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T01:43:15.499 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.499 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.499 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.500 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.500 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.500 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.500 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.500 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.500 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.500 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.500 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.500 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T01:43:15.500 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T01:43:15.500 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.500 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.500 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.500 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.501 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.501 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.501 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.501 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.501 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.501 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.501 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.501 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T01:43:15.501 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T01:43:15.501 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.501 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.501 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.501 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.501 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.502 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.502 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.502 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.502 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.502 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.502 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.502 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T01:43:15.502 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T01:43:15.502 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.502 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.502 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.502 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.502 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.502 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.503 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.503 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.503 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.503 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.503 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.503 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T01:43:15.503 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T01:43:15.503 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.503 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.503 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.503 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.503 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.503 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.504 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.504 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.504 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.504 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.504 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.504 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T01:43:15.504 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T01:43:15.504 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.504 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.504 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.504 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.504 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.504 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.505 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.505 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.505 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.505 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.505 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.505 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T01:43:15.505 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T01:43:15.505 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.505 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.505 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.505 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.505 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.505 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.505 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.506 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.506 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.506 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.506 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.506 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T01:43:15.506 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T01:43:15.506 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.506 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.506 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.506 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.506 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.506 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.506 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T01:43:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T01:43:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T01:43:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.508 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.508 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.508 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.508 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.508 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T01:43:15.508 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T01:43:15.508 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.508 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.508 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.508 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:15.508 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.508 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.508 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.509 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.509 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.509 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.509 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.509 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T01:43:15.509 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T01:43:15.509 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.509 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.509 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.509 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-04-25T01:43:15.509 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.509 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.509 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.510 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.510 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.510 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.510 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.510 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T01:43:15.510 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T01:43:15.510 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.510 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.510 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.510 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-04-25T01:43:15.510 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.510 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.510 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.511 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.511 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.511 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.511 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.511 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T01:43:15.511 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T01:43:15.511 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.511 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.511 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.511 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.511 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.511 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.511 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.511 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.511 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.512 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.512 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.512 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T01:43:15.512 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T01:43:15.512 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.512 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.512 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.512 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-25T01:43:15.512 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.512 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.512 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.512 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.512 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.512 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.513 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.513 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T01:43:15.513 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T01:43:15.513 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.513 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.513 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.513 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.513 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.513 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.513 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.513 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.513 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.513 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.514 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.514 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T01:43:15.514 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T01:43:15.514 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.514 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.514 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.514 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.514 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.514 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.514 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.514 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.514 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.515 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.515 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.515 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T01:43:15.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T01:43:15.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.515 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.515 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.515 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-25T01:43:15.515 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.515 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.515 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.515 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.515 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.515 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.516 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.516 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T01:43:15.516 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T01:43:15.516 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.516 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.516 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.516 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:15.516 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.516 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.516 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.516 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.516 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.517 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.517 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.517 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T01:43:15.517 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T01:43:15.517 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.517 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.517 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.517 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-25T01:43:15.517 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.517 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.517 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.517 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.518 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.518 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.518 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.518 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-04-25T01:43:15.518 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-04-25T01:43:15.518 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.518 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.518 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.518 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.518 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.518 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.518 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.518 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.518 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.519 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.519 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.519 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-04-25T01:43:15.519 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-04-25T01:43:15.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.519 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.519 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.519 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.519 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.519 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.519 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.519 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.519 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.519 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.519 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.520 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-04-25T01:43:15.520 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-04-25T01:43:15.520 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.520 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.520 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.520 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.520 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.520 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.520 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.520 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-04-25T01:43:15.520 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.520 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.520 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.521 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-04-25T01:43:15.521 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-04-25T01:43:15.521 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.521 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.521 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.521 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-04-25T01:43:15.521 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.521 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.521 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.521 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.522 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.522 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.522 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.522 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-25T01:43:15.522 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-25T01:43:15.522 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.522 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.522 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.522 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.522 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.522 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.523 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.523 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.523 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.523 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.523 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.523 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.523 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.523 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.523 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.524 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.524 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.524 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.524 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.524 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.524 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.524 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.524 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.524 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.524 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.524 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.524 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.524 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.524 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.524 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.525 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.525 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.525 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.525 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.525 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.525 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.525 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.525 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.526 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.526 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.526 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.526 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.526 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.526 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.526 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.526 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.526 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.526 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.526 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.527 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.527 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.527 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.527 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.527 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.527 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.527 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.527 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.527 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.527 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.528 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.528 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.528 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.528 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.528 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.528 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.528 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.528 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.528 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.528 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.529 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.529 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.529 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.529 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.529 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.529 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.529 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.529 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-04-25T01:43:15.530 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-04-25T01:43:15.530 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.530 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.530 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.530 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.530 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.530 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.530 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.530 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-04-25T01:43:15.530 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.530 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.531 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.531 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-04-25T01:43:15.531 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T01:43:15.531 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.531 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.531 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.531 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.531 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.531 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.532 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.532 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.532 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.532 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.532 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.532 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-25T01:43:15.532 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-25T01:43:15.532 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.532 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.532 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.532 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-25T01:43:15.532 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.532 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.533 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.533 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.533 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.533 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.533 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.533 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-25T01:43:15.533 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-25T01:43:15.533 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.533 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.533 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.533 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-04-25T01:43:15.533 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.533 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.534 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.534 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.534 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.534 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.534 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.534 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-25T01:43:15.534 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-25T01:43:15.534 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.534 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.534 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.534 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.534 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.534 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.535 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.535 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.535 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.535 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.535 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.535 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-04-25T01:43:15.535 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-04-25T01:43:15.535 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.535 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.535 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.535 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-04-25T01:43:15.535 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.535 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.535 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.536 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.536 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.536 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.536 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.536 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-25T01:43:15.536 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-25T01:43:15.536 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.536 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.536 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.536 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-04-25T01:43:15.536 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.536 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.536 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.537 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.537 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-04-25T01:43:15.537 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.537 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.537 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.537 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.537 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.537 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-25T01:43:15.537 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-25T01:43:15.537 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.537 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.537 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.537 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-25T01:43:15.537 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-25T01:43:15.538 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-25T01:43:15.538 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.538 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.538 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.538 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.538 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.538 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-04-25T01:43:15.538 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-04-25T01:43:15.538 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.538 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.538 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.538 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.538 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.539 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.539 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.539 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.539 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.539 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.539 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.539 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.539 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.539 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-25T01:43:15.539 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.539 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.540 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.540 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-04-25T01:43:15.540 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-04-25T01:43:15.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.540 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.540 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.540 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.540 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.540 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.540 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.540 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-04-25T01:43:15.540 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.540 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.540 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.540 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.541 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.541 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.541 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.541 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.541 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.541 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.541 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.541 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.541 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.541 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.541 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.541 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.541 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.541 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.541 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.542 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.542 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.542 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.542 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.542 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.542 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.542 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.542 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.542 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.542 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.542 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.542 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.543 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.543 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.543 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.543 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.543 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.543 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.543 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.543 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.543 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.543 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.543 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.543 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.543 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.544 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.544 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.544 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.544 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.544 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.544 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.544 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.544 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.544 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.544 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.544 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.544 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.544 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.544 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.545 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.545 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.545 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.545 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.545 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.545 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.545 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.545 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.545 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.545 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.545 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-04-25T01:43:15.545 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-04-25T01:43:15.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.545 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.546 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.546 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-04-25T01:43:15.546 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.546 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.546 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.546 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T01:43:15.546 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.546 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.546 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.546 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-04-25T01:43:15.546 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-04-25T01:43:15.547 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.547 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.547 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.547 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-04-25T01:43:15.547 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.547 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.547 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.547 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T01:43:15.547 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.547 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.547 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.547 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-04-25T01:43:15.547 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-04-25T01:43:15.547 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.548 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.548 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.548 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-04-25T01:43:15.548 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.548 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.548 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.548 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-04-25T01:43:15.548 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.548 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.548 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.548 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-04-25T01:43:15.548 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-04-25T01:43:15.548 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.548 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.549 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.549 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-25T01:43:15.549 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.549 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.549 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.549 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T01:43:15.549 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.549 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.549 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.549 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-04-25T01:43:15.549 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-04-25T01:43:15.549 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.549 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.549 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.550 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.550 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.550 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.550 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.550 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T01:43:15.550 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.550 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.550 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.550 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-25T01:43:15.550 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-25T01:43:15.550 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.550 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.551 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.551 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-25T01:43:15.551 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.551 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.551 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.551 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T01:43:15.551 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.551 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.551 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.551 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-04-25T01:43:15.551 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-04-25T01:43:15.551 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.551 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.551 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.552 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-04-25T01:43:15.552 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.552 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.552 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.552 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T01:43:15.552 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.552 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.552 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.552 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.552 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.552 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-04-25T01:43:15.552 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.553 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.553 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.553 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.553 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.553 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.553 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.553 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.553 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.553 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.553 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.553 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.553 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.553 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.554 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.554 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.554 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.554 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.554 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.554 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.554 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.554 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.554 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.555 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.555 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.555 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.555 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.555 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.555 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.555 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.555 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.555 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.555 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.555 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.555 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.555 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.556 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.556 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.556 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.556 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.556 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.556 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.556 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.556 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.556 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.556 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.557 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.557 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.557 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.557 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.557 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.557 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.557 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.557 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.557 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.557 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.557 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.557 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.557 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.557 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.558 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.558 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.558 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.558 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.558 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.558 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.558 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.558 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.558 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.559 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-04-25T01:43:15.559 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-04-25T01:43:15.559 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.559 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.559 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.559 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-25T01:43:15.559 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.559 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.559 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.559 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.559 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.559 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.559 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.560 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-04-25T01:43:15.560 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-04-25T01:43:15.560 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.560 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.560 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.560 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-04-25T01:43:15.560 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.560 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.560 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.560 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.560 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.560 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.561 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.561 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-25T01:43:15.561 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-25T01:43:15.561 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.561 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.561 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.561 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-25T01:43:15.561 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.561 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.561 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.561 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.562 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.562 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.562 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.562 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.562 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.562 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.562 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-04-25T01:43:15.562 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-04-25T01:43:15.562 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-04-25T01:43:15.562 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.562 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-04-25T01:43:15.563 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-04-25T01:43:15.563 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.563 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.563 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.563 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-04-25T01:43:15.563 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.563 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.563 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.563 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T01:43:15.563 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.564 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.564 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.564 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-04-25T01:43:15.564 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-04-25T01:43:15.564 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.564 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.564 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.564 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-04-25T01:43:15.564 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.564 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.564 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.565 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T01:43:15.565 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.565 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.565 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.565 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-04-25T01:43:15.565 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-04-25T01:43:15.565 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.565 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.566 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.566 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.566 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.566 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.566 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.566 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-04-25T01:43:15.566 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.566 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.566 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.567 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-25T01:43:15.567 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-25T01:43:15.567 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.567 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.567 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.567 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-04-25T01:43:15.567 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-04-25T01:43:15.567 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.567 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.567 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T01:43:15.567 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.567 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.567 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.568 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.568 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.568 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.568 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.568 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.568 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.568 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.568 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.568 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.568 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.568 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.568 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.568 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.568 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.569 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.569 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.569 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.569 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.569 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.569 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.569 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.569 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.569 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.569 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.569 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.569 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.569 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.570 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.570 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.570 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.570 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.570 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.570 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.570 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.570 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.570 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.570 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.570 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.570 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.572 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.572 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.572 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.572 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.572 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.572 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.572 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.572 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.572 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.572 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.572 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.572 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.572 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.573 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-04-25T01:43:15.573 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-04-25T01:43:15.573 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.573 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.573 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.573 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.573 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.573 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.573 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.573 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-04-25T01:43:15.573 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.573 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.573 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.574 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-04-25T01:43:15.574 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-04-25T01:43:15.574 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.574 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.574 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.574 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-04-25T01:43:15.574 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.574 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.574 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.574 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-04-25T01:43:15.574 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.574 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.574 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.575 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-25T01:43:15.575 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-25T01:43:15.575 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.575 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.575 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.575 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-04-25T01:43:15.575 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.575 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.575 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.575 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T01:43:15.575 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.575 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.575 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.575 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-04-25T01:43:15.576 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-04-25T01:43:15.576 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.576 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.576 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.576 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-25T01:43:15.576 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-25T01:43:15.576 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-04-25T01:43:15.576 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.576 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T01:43:15.576 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.576 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.576 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.576 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-04-25T01:43:15.576 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-04-25T01:43:15.577 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.577 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.577 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.577 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.577 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.577 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.577 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.577 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-04-25T01:43:15.577 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.577 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.577 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.577 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-04-25T01:43:15.577 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-04-25T01:43:15.577 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.577 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.578 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.578 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-04-25T01:43:15.578 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.578 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.578 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.578 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T01:43:15.578 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.578 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.578 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.578 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.578 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.578 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.579 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-04-25T01:43:15.579 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.579 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.579 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.579 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.579 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.579 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.579 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.579 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.579 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.579 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.579 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.579 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.579 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.579 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.580 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.580 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.580 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.580 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.580 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.580 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.580 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.580 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.580 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.580 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.580 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.580 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.580 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.580 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.580 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.581 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.581 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.581 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.581 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.581 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.581 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.581 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.581 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.581 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.581 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.581 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.581 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.581 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.581 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.582 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.582 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.582 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.582 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.582 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.582 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.582 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.582 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.582 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.582 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.582 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.582 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.582 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.583 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.583 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.583 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.583 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.583 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.583 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.583 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.583 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.583 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.583 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.583 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.583 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.583 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.584 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.584 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.584 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-04-25T01:43:15.584 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.584 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.584 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.584 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.584 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.584 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.584 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.584 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.584 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.584 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.584 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.585 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.585 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.585 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.585 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.585 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.585 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.585 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.585 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.585 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.585 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.585 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.585 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.585 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.585 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.585 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.586 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.586 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.586 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.586 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.586 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.586 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.586 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.586 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.586 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.586 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.586 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.586 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.586 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.586 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.587 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.587 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.587 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.587 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.587 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.587 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.587 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.587 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.587 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.587 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.587 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.587 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.587 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.587 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.587 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.587 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.588 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.588 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.588 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.588 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.588 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.588 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.588 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.588 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.589 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.589 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.589 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.589 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.589 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.589 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.589 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.589 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-04-25T01:43:15.589 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.589 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.589 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.589 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-04-25T01:43:15.590 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-04-25T01:43:15.590 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.590 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.590 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.590 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-04-25T01:43:15.590 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.590 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.590 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.590 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T01:43:15.590 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.590 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.591 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.591 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-04-25T01:43:15.591 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-04-25T01:43:15.591 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.591 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.591 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.591 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:15.591 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-04-25T01:43:15.591 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.591 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.591 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T01:43:15.591 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.591 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.592 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.592 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.592 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.592 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.592 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.592 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.592 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.592 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.592 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.592 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.592 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.592 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.592 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.592 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.593 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.593 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.593 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.593 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.593 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.593 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.593 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.593 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.593 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.593 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.593 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.593 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.594 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.594 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.594 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.594 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.594 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.594 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.594 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.594 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.594 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.594 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.595 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.595 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.595 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.595 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.595 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.595 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.595 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.595 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.595 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.595 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.596 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.596 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.596 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.596 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.596 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.596 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.596 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.596 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.596 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.596 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.597 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.597 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.597 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.597 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.597 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.597 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.597 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.597 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.597 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.597 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.597 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.597 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.598 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.598 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-04-25T01:43:15.598 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.598 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.598 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.598 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-04-25T01:43:15.598 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-04-25T01:43:15.599 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.599 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.599 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.599 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-04-25T01:43:15.599 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.599 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.599 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.599 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-04-25T01:43:15.599 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T01:43:15.599 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.599 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.599 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.600 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.600 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.600 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.600 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.600 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.600 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.600 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.600 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.600 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.600 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.600 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.601 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.601 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.601 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.601 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.601 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.601 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.601 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.601 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.601 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.601 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.602 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.602 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.602 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.602 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.602 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.602 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.602 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.602 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.602 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.603 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.603 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.603 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.603 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.603 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.603 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.603 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.603 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.603 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.603 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.603 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.604 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.604 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.604 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.604 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.604 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.604 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.604 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.604 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.604 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.604 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.604 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.604 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.604 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.605 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.605 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.605 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.605 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.605 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.605 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.605 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.605 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.605 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.605 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-04-25T01:43:15.605 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-04-25T01:43:15.606 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.606 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.606 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.606 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.606 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.606 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.606 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.606 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-04-25T01:43:15.606 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T01:43:15.606 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.606 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.606 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-04-25T01:43:15.606 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-04-25T01:43:15.607 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.607 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.607 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.607 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-04-25T01:43:15.607 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.607 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.607 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.607 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-04-25T01:43:15.607 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.607 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.607 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.607 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-04-25T01:43:15.607 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-04-25T01:43:15.608 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.608 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.608 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.608 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-04-25T01:43:15.608 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.608 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.608 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.608 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T01:43:15.608 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.608 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.609 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.609 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-04-25T01:43:15.609 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-04-25T01:43:15.609 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.609 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.609 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.609 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.609 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.609 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.609 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.609 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-04-25T01:43:15.609 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.609 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.610 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.610 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-04-25T01:43:15.610 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-04-25T01:43:15.610 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.610 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.610 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.610 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-04-25T01:43:15.610 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.610 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.610 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.610 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T01:43:15.610 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T01:43:15.611 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.611 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.611 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.611 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.611 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-04-25T01:43:15.611 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.611 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.611 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.611 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.611 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.611 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.611 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.612 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.612 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.612 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.612 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.612 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.612 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.612 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.612 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.612 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.612 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.612 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.612 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.612 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.612 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.612 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.613 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.613 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.613 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.613 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.613 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.613 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.613 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.613 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.613 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.613 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.613 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.613 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.613 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.614 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.615 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-04-25T01:43:15.616 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.617 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.618 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.619 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.620 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-04-25T01:43:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.622 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.624 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.625 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.626 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.627 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-25T01:43:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-04-25T01:43:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-04-25T01:43:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.628 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T01:43:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.629 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.630 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.630 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.630 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.630 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.630 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.630 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.630 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.630 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.630 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.630 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.630 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.631 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.631 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.631 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.631 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.631 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.631 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.631 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.631 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.631 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.631 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.631 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.631 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.632 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.633 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.634 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-04-25T01:43:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-25T01:43:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-25T01:43:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-04-25T01:43:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-04-25T01:43:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-04-25T01:43:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-04-25T01:43:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.637 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.638 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.639 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.640 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.641 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.642 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-04-25T01:43:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.644 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.646 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.647 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-04-25T01:43:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.649 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.650 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.651 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.652 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.654 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-25T01:43:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-25T01:43:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-25T01:43:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-04-25T01:43:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-04-25T01:43:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-25T01:43:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-04-25T01:43:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-04-25T01:43:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T01:43:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.656 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-04-25T01:43:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-04-25T01:43:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-04-25T01:43:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T01:43:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-25T01:43:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-25T01:43:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.658 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.659 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.661 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.662 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.663 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.664 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-04-25T01:43:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-04-25T01:43:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-04-25T01:43:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-04-25T01:43:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-25T01:43:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-25T01:43:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-25T01:43:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-04-25T01:43:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-04-25T01:43:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.667 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-04-25T01:43:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-04-25T01:43:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-04-25T01:43:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-04-25T01:43:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T01:43:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T01:43:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.669 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.670 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.671 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.672 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.673 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.674 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.675 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-04-25T01:43:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-04-25T01:43:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.676 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T01:43:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T01:43:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-04-25T01:43:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.677 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-04-25T01:43:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-04-25T01:43:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-04-25T01:43:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.678 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-25T01:43:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-25T01:43:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-25T01:43:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T01:43:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.679 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-25T01:43:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-25T01:43:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-04-25T01:43:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T01:43:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-04-25T01:43:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-04-25T01:43:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-04-25T01:43:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-25T01:43:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-25T01:43:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-04-25T01:43:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-04-25T01:43:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-04-25T01:43:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.682 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-25T01:43:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-25T01:43:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-25T01:43:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-25T01:43:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-25T01:43:15.683 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-25T01:43:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.684 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.685 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.686 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.688 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-04-25T01:43:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-04-25T01:43:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.689 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-04-25T01:43:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T01:43:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.690 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-04-25T01:43:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-04-25T01:43:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.691 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-04-25T01:43:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.692 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-04-25T01:43:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-04-25T01:43:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.693 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-25T01:43:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-25T01:43:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.695 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.696 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.697 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.698 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-25T01:43:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-25T01:43:15.700 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-25T01:43:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-25T01:43:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.702 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-04-25T01:43:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.703 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.704 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.705 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.707 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.709 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-04-25T01:43:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.710 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.711 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.712 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.713 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.714 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-04-25T01:43:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-04-25T01:43:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.715 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-04-25T01:43:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T01:43:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-04-25T01:43:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.716 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.717 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.718 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.719 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.720 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-04-25T01:43:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-04-25T01:43:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.721 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-04-25T01:43:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-04-25T01:43:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.722 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-04-25T01:43:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-04-25T01:43:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-04-25T01:43:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.723 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-04-25T01:43:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-04-25T01:43:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.724 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-04-25T01:43:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-04-25T01:43:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.725 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-04-25T01:43:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-04-25T01:43:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.726 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-04-25T01:43:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-04-25T01:43:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.727 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-04-25T01:43:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-04-25T01:43:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.728 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-04-25T01:43:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-04-25T01:43:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-25T01:43:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-04-25T01:43:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-04-25T01:43:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-04-25T01:43:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.730 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-04-25T01:43:15.731 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-04-25T01:43:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-04-25T01:43:15.732 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-04-25T01:43:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.733 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-25T01:43:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-04-25T01:43:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-04-25T01:43:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.734 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T01:43:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-04-25T01:43:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-04-25T01:43:15.735 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-04-25T01:43:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.737 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.738 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.739 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.739 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.739 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.739 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.739 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.739 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.739 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.739 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.739 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.739 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.740 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.740 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.740 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.740 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.740 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.740 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.740 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.740 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.740 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.740 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.740 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.740 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.740 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.740 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.741 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.741 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.741 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.741 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.741 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.741 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.741 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.741 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.741 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.741 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.741 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.741 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.741 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.742 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.742 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.742 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.742 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.742 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.742 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.742 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.742 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.742 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-04-25T01:43:15.742 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.743 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.743 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.743 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.743 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.743 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.743 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.743 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.743 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.743 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.744 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.744 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.744 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.744 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.744 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.744 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.744 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.744 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.744 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.745 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.745 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.745 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.745 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.745 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.745 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.745 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.745 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.746 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.746 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.746 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.746 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.746 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.746 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.746 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.746 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.746 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.746 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.746 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.747 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.747 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.747 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.747 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.747 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.747 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.747 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.747 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.747 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.747 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.747 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.747 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.747 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.747 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.747 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.747 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.748 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-04-25T01:43:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.749 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.750 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.751 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.751 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.751 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.751 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.751 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.751 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.751 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.751 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.751 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.751 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.751 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.751 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.751 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.751 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.752 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.752 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.752 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.752 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.752 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.752 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.752 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.752 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.752 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.752 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.752 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.752 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.752 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.752 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.753 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.753 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.753 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.753 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.753 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.753 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.753 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.753 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.753 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.753 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.753 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.753 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.753 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.753 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.754 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.754 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.754 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.754 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-04-25T01:43:15.754 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.754 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.754 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.754 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-04-25T01:43:15.754 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-04-25T01:43:15.754 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.754 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.754 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T01:43:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-25T01:43:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-25T01:43:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.755 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-04-25T01:43:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.756 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.758 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.759 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.760 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-04-25T01:43:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.761 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-04-25T01:43:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-04-25T01:43:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T01:43:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.762 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-04-25T01:43:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-04-25T01:43:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T01:43:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-04-25T01:43:15.763 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-04-25T01:43:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T01:43:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-04-25T01:43:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-04-25T01:43:15.764 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T01:43:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-04-25T01:43:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-04-25T01:43:15.765 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.766 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.766 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.766 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.766 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.766 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.766 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.766 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T01:43:15.766 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.766 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.766 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.766 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-04-25T01:43:15.767 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-04-25T01:43:15.767 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.767 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.767 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.767 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.767 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.767 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.767 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.767 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.767 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.767 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.767 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.768 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-04-25T01:43:15.768 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-04-25T01:43:15.768 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.768 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.768 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.768 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.768 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.768 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.768 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.768 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.768 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-04-25T01:43:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-04-25T01:43:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T01:43:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.769 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-25T01:43:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-25T01:43:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.770 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-25T01:43:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-25T01:43:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-04-25T01:43:15.772 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-04-25T01:43:15.772 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.772 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.772 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.772 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.772 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.772 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.772 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-04-25T01:43:15.772 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-04-25T01:43:15.773 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.773 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.773 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.773 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-04-25T01:43:15.773 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.773 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.773 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.773 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.773 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.773 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.773 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.773 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-04-25T01:43:15.774 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-04-25T01:43:15.774 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.774 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.774 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.774 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.774 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.774 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.774 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.774 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.774 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.775 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.775 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.775 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-04-25T01:43:15.775 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-04-25T01:43:15.775 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.775 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.775 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.775 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.775 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.775 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.775 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.775 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.776 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.776 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.776 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.776 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.776 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.776 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.776 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.776 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.776 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.776 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.778 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.778 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.778 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.778 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.778 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.778 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.778 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.778 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.778 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.779 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.779 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.779 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.779 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.779 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.779 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.779 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.779 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.779 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.779 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.780 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.780 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.780 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.780 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.780 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.780 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.780 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.780 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.780 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.780 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.780 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.780 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.781 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.781 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.781 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.781 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.781 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.781 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.781 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.781 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.781 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.781 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.781 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.781 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.781 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.782 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.782 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.782 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-04-25T01:43:15.782 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-04-25T01:43:15.782 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.782 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.782 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.782 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.782 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.782 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.782 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.782 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.782 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.783 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.783 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.783 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-04-25T01:43:15.783 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-04-25T01:43:15.783 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.783 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.783 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.783 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.783 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.783 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.783 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.783 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.783 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.783 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.784 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.784 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-04-25T01:43:15.784 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-04-25T01:43:15.784 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.784 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.784 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.784 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T01:43:15.784 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.784 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.784 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.784 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.784 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.784 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.784 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.785 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.785 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.785 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-04-25T01:43:15.785 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.785 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.785 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.785 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.785 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.785 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.785 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.785 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.785 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.786 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.786 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.786 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.786 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.786 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.786 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.786 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.786 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.786 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.786 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.786 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.786 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.786 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.786 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.787 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.787 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.787 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.787 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.787 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.787 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.787 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.787 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.787 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.787 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.787 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.787 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.787 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.788 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.788 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.788 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.788 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.788 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.788 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.788 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.788 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.788 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.788 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.788 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.788 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.788 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.788 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.788 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.789 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.789 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.789 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.789 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.789 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.789 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.789 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.789 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.789 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.789 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.789 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.789 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.789 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.790 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.790 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.790 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.790 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.790 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.790 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.790 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.790 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-25T01:43:15.790 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.790 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.790 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.790 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.791 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.791 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.791 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.791 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.791 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.791 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.791 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.791 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.791 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.791 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.791 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.791 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.791 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.791 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.791 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.792 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.792 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.792 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.792 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.792 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.792 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.792 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.792 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.792 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.792 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.792 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.792 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.792 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.792 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.793 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.793 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.793 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.793 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.793 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.793 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.793 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.793 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.793 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.793 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.793 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.793 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.793 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.794 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.794 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.794 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.794 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.794 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.794 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.794 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.794 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.794 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.794 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.794 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.794 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.795 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.795 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.795 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.795 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.795 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.795 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.795 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.795 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.795 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.795 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.795 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-04-25T01:43:15.795 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-04-25T01:43:15.795 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.795 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.795 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.796 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-04-25T01:43:15.796 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.796 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.796 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.796 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T01:43:15.796 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.796 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.796 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.796 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-04-25T01:43:15.796 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-04-25T01:43:15.796 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.796 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.796 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.796 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.797 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.797 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.797 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.797 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-04-25T01:43:15.797 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.797 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.797 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.797 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-04-25T01:43:15.797 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-04-25T01:43:15.797 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.797 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.797 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.797 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-25T01:43:15.797 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.798 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.798 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.798 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T01:43:15.798 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.798 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.798 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.798 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-04-25T01:43:15.798 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T01:43:15.798 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.798 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.798 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.799 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.799 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.799 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.799 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.799 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T01:43:15.799 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.799 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.799 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.799 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-04-25T01:43:15.799 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.800 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.800 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.800 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-04-25T01:43:15.800 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-04-25T01:43:15.800 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-25T01:43:15.800 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.800 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.800 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-04-25T01:43:15.800 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.800 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.800 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.800 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.800 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.800 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.801 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.801 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-04-25T01:43:15.801 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-04-25T01:43:15.801 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.801 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.801 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.801 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.801 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.801 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.801 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.801 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.802 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.802 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.802 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.802 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-25T01:43:15.802 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-25T01:43:15.802 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.802 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.802 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.802 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-25T01:43:15.802 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.803 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.803 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.803 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.803 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.803 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.803 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.803 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.803 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.803 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.803 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.803 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.803 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.803 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.804 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.804 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.804 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.804 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.804 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.804 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.804 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.804 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.804 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.804 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.804 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.805 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.805 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.805 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.805 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.805 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.805 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.805 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.805 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.805 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.805 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.805 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.806 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.806 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.806 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.806 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.806 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.806 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.806 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.806 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.807 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.807 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.807 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.807 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.807 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.807 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.807 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.807 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.807 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.807 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.807 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.807 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.807 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.807 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.808 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.808 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.808 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.808 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.808 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.808 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.808 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.808 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.808 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.808 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.808 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.809 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.809 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.809 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-04-25T01:43:15.809 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-04-25T01:43:15.809 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.809 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.809 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.809 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.809 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.809 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.809 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.809 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.809 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.810 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.810 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.810 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-04-25T01:43:15.810 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-04-25T01:43:15.810 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.810 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.810 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.810 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-04-25T01:43:15.810 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.811 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.811 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.811 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.811 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.811 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.811 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.811 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-04-25T01:43:15.811 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-04-25T01:43:15.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.811 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.812 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.812 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T01:43:15.812 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.812 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.812 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.812 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.812 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.812 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.812 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.812 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.812 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.812 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.813 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-25T01:43:15.813 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.813 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.813 INFO:teuthology.orchestra.run.smithi028.stdout: "modules": [ 2024-04-25T01:43:15.813 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-25T01:43:15.813 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard", 2024-04-25T01:43:15.813 INFO:teuthology.orchestra.run.smithi028.stdout: "iostat", 2024-04-25T01:43:15.813 INFO:teuthology.orchestra.run.smithi028.stdout: "nfs", 2024-04-25T01:43:15.813 INFO:teuthology.orchestra.run.smithi028.stdout: "restful", 2024-04-25T01:43:15.813 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_schedule" 2024-04-25T01:43:15.813 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.813 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-04-25T01:43:15.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.814 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-04-25T01:43:15.814 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.814 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.814 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.814 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-25T01:43:15.814 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-25T01:43:15.814 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.814 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.814 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.814 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-25T01:43:15.814 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.814 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.814 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.815 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T01:43:15.815 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.815 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.815 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.815 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.815 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.815 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.815 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.815 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.815 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.815 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.815 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.815 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.815 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.816 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.816 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.816 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.816 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.816 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.816 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.816 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.816 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.816 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.816 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.816 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.817 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.817 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.817 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.817 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.817 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.817 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.817 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.817 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.817 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.818 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.818 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.818 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.818 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.818 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.818 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.818 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.818 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.818 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.818 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.818 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.819 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.819 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.819 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.819 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.819 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.819 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.819 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.819 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.819 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.819 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.819 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.819 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.819 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.819 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.819 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.820 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.820 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.820 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.820 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.820 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.820 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.820 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.820 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.820 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.820 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.820 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-04-25T01:43:15.820 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-04-25T01:43:15.820 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.820 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.821 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.821 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.821 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.821 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.821 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.821 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-04-25T01:43:15.821 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.821 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.821 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.821 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-04-25T01:43:15.821 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-04-25T01:43:15.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.821 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.821 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.822 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-04-25T01:43:15.822 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.822 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.822 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.822 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-04-25T01:43:15.822 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.822 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.822 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.822 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-04-25T01:43:15.822 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-04-25T01:43:15.822 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.822 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.822 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.823 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.823 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.823 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.823 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.823 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-04-25T01:43:15.823 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.823 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.823 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.823 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-04-25T01:43:15.823 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-04-25T01:43:15.823 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.823 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.823 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.823 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.824 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.824 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.824 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.824 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-04-25T01:43:15.824 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.824 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.824 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.824 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-04-25T01:43:15.824 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-04-25T01:43:15.824 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.824 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.824 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.824 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-04-25T01:43:15.825 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.825 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.825 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.825 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-04-25T01:43:15.825 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.825 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.825 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.825 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-04-25T01:43:15.825 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-04-25T01:43:15.825 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.825 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.825 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.825 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.825 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.826 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.826 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.826 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-04-25T01:43:15.826 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.826 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.826 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.826 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-04-25T01:43:15.826 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-04-25T01:43:15.826 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.826 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.826 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.826 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.826 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.827 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.827 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.827 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T01:43:15.827 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.827 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.827 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.827 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-04-25T01:43:15.827 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-04-25T01:43:15.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.827 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.827 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.827 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.827 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.827 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.828 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.828 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-04-25T01:43:15.828 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.828 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.828 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.828 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.828 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.828 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-04-25T01:43:15.828 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.828 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.828 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.828 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-04-25T01:43:15.828 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-04-25T01:43:15.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.829 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.829 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.829 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.829 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.829 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.829 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T01:43:15.829 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.829 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.829 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.829 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-04-25T01:43:15.829 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-04-25T01:43:15.830 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.830 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.830 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.830 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-04-25T01:43:15.830 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.830 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.830 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.830 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T01:43:15.830 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T01:43:15.830 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.830 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.830 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-04-25T01:43:15.830 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-04-25T01:43:15.831 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-25T01:43:15.831 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.831 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.831 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-25T01:43:15.831 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-25T01:43:15.831 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-25T01:43:15.831 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.831 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T01:43:15.831 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T01:43:15.831 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.831 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.831 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-04-25T01:43:15.831 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-04-25T01:43:15.832 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-25T01:43:15.832 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.832 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.832 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-04-25T01:43:15.832 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-25T01:43:15.832 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-04-25T01:43:15.832 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.832 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T01:43:15.832 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.832 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.832 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.832 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-04-25T01:43:15.832 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-04-25T01:43:15.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.833 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.833 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.833 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T01:43:15.833 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.833 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.833 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.833 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T01:43:15.833 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T01:43:15.833 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.833 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.833 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-04-25T01:43:15.833 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-04-25T01:43:15.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:15.834 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.834 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.834 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-04-25T01:43:15.834 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-04-25T01:43:15.834 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-04-25T01:43:15.834 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.834 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-04-25T01:43:15.834 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T01:43:15.834 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.834 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.834 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-04-25T01:43:15.834 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-04-25T01:43:15.834 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.835 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.835 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.835 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-04-25T01:43:15.835 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.835 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.835 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.835 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T01:43:15.835 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T01:43:15.835 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.835 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.835 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-04-25T01:43:15.835 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-04-25T01:43:15.835 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-25T01:43:15.835 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.836 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.836 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-25T01:43:15.836 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-25T01:43:15.836 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-25T01:43:15.836 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.836 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T01:43:15.836 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T01:43:15.836 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.836 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.836 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.836 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.836 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.836 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.836 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.837 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.837 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.837 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.837 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.837 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.837 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.837 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.837 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.837 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.837 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.837 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.837 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.837 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.837 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.837 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.838 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.838 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.838 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.838 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.838 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.838 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.838 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.838 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.838 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.838 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.838 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.838 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.839 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.839 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.839 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.839 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.839 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.839 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.839 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.839 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.839 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.839 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.839 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.840 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.840 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.840 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.840 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.840 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.840 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.840 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.840 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.840 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.840 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.840 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.841 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.841 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.841 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.841 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.841 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.841 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.841 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.841 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.841 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.841 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.841 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.841 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.841 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-04-25T01:43:15.841 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-04-25T01:43:15.842 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:15.842 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.842 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.842 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-25T01:43:15.842 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.842 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.842 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.842 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T01:43:15.842 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.843 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.843 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.843 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-25T01:43:15.843 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-25T01:43:15.843 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.843 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.843 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.843 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-04-25T01:43:15.843 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.843 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.843 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.843 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-04-25T01:43:15.843 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-04-25T01:43:15.843 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-04-25T01:43:15.844 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-04-25T01:43:15.844 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-04-25T01:43:15.845 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.845 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-04-25T01:43:15.845 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.845 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.845 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.846 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-04-25T01:43:15.846 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-04-25T01:43:15.846 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.846 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.846 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.846 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.846 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.846 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.846 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.846 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T01:43:15.846 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.847 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.847 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.847 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-25T01:43:15.847 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-25T01:43:15.847 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.847 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.847 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.847 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-25T01:43:15.847 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.847 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.847 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.847 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T01:43:15.848 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.848 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.848 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.848 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-04-25T01:43:15.848 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-04-25T01:43:15.848 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.848 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.848 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.848 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-25T01:43:15.849 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-25T01:43:15.849 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.849 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.849 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T01:43:15.849 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T01:43:15.849 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.849 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.849 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-04-25T01:43:15.849 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-04-25T01:43:15.849 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-25T01:43:15.850 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.850 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.850 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:15.850 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.850 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.850 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.850 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T01:43:15.850 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.850 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.850 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.850 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.850 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.851 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-04-25T01:43:15.851 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.851 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.851 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.851 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-04-25T01:43:15.851 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-04-25T01:43:15.851 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:15.851 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.851 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.852 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-25T01:43:15.852 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.852 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.852 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.852 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T01:43:15.852 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.852 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.852 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.852 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-04-25T01:43:15.853 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-04-25T01:43:15.853 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.853 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.853 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-04-25T01:43:15.853 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.853 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.853 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.853 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T01:43:15.853 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.854 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.854 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.854 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-04-25T01:43:15.854 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-04-25T01:43:15.854 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.854 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.854 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.854 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-04-25T01:43:15.854 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.854 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.855 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.855 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T01:43:15.855 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.855 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.855 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.855 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-04-25T01:43:15.855 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-04-25T01:43:15.855 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.855 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.855 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.855 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.855 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.856 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.856 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.856 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T01:43:15.856 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T01:43:15.856 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.856 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.856 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-04-25T01:43:15.856 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-04-25T01:43:15.856 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.856 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.856 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.856 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-25T01:43:15.856 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.857 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.857 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.857 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T01:43:15.857 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.857 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.857 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.857 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-04-25T01:43:15.857 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T01:43:15.857 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:15.857 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.857 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.857 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-04-25T01:43:15.857 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.857 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.857 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.858 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T01:43:15.858 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.858 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.858 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.858 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-04-25T01:43:15.858 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-04-25T01:43:15.858 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.858 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.858 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.858 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.858 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.858 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.858 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.859 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-04-25T01:43:15.859 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-04-25T01:43:15.859 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-04-25T01:43:15.859 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.859 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T01:43:15.859 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.859 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.859 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.859 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-04-25T01:43:15.859 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-04-25T01:43:15.859 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.860 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.860 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.860 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.860 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.860 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.860 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.860 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T01:43:15.860 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.860 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.861 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.861 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-04-25T01:43:15.861 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-04-25T01:43:15.861 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.861 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.861 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.862 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.862 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.862 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.862 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.862 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T01:43:15.862 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.862 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.863 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.863 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-04-25T01:43:15.863 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-04-25T01:43:15.863 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.863 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.863 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.863 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.863 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.863 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.863 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.863 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T01:43:15.863 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-04-25T01:43:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-04-25T01:43:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T01:43:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-25T01:43:15.864 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-04-25T01:43:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-04-25T01:43:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T01:43:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-04-25T01:43:15.865 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.866 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.866 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.866 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-04-25T01:43:15.866 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-04-25T01:43:15.866 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.866 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.866 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.866 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T01:43:15.866 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.866 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.866 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.866 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-04-25T01:43:15.867 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.867 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.867 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.867 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-04-25T01:43:15.867 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-04-25T01:43:15.867 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.867 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.867 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.867 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T01:43:15.867 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.867 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.867 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.868 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-25T01:43:15.868 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.868 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.868 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.868 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-04-25T01:43:15.868 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-04-25T01:43:15.868 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.868 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.868 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.868 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T01:43:15.868 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.868 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.868 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.868 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-04-25T01:43:15.869 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.869 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.869 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.869 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-04-25T01:43:15.869 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-04-25T01:43:15.869 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.869 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.869 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.869 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T01:43:15.869 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.869 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.869 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.869 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-04-25T01:43:15.869 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.870 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.870 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.870 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-04-25T01:43:15.870 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-04-25T01:43:15.870 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.870 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.870 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.870 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T01:43:15.870 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.870 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.870 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.871 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-04-25T01:43:15.871 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.871 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.871 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.871 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-04-25T01:43:15.871 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-04-25T01:43:15.871 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.871 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.871 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.871 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T01:43:15.871 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.871 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.872 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.872 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-04-25T01:43:15.872 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.872 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.872 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.872 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-04-25T01:43:15.872 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-04-25T01:43:15.872 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.872 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.872 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.872 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T01:43:15.872 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.872 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.872 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.872 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-04-25T01:43:15.873 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.873 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.873 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.873 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-04-25T01:43:15.873 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-04-25T01:43:15.873 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.873 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.873 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.873 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T01:43:15.873 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.873 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.873 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.873 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-04-25T01:43:15.874 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.874 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.874 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.874 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-04-25T01:43:15.874 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-04-25T01:43:15.874 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.874 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.874 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.874 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T01:43:15.874 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.874 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.875 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.875 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-25T01:43:15.875 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.875 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.875 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.875 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-04-25T01:43:15.875 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-04-25T01:43:15.875 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.875 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.875 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.876 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T01:43:15.876 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.876 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.876 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.876 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-04-25T01:43:15.876 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.876 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.876 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.877 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-04-25T01:43:15.877 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-04-25T01:43:15.877 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.877 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.877 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.877 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T01:43:15.877 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.877 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.877 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.877 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-25T01:43:15.877 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.877 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.877 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.878 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-04-25T01:43:15.878 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-04-25T01:43:15.878 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.878 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.878 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.878 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T01:43:15.878 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.878 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.878 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.879 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-04-25T01:43:15.879 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.879 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.879 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.879 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-04-25T01:43:15.879 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-04-25T01:43:15.879 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.879 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.879 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.879 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T01:43:15.879 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.879 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.879 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.880 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-04-25T01:43:15.880 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.880 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.880 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.880 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-04-25T01:43:15.880 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-04-25T01:43:15.880 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.880 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.880 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.881 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T01:43:15.881 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.881 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.881 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.881 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-04-25T01:43:15.881 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.881 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.881 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.881 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-04-25T01:43:15.881 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-04-25T01:43:15.881 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.882 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.882 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.882 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.882 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.882 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.882 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.882 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T01:43:15.882 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.882 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.882 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.883 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-04-25T01:43:15.883 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-04-25T01:43:15.883 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.883 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.883 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.883 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-25T01:43:15.883 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.883 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.883 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.883 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T01:43:15.883 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.884 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.884 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.884 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-04-25T01:43:15.884 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T01:43:15.884 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.884 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.884 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.884 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-04-25T01:43:15.884 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.884 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.885 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.885 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T01:43:15.885 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.885 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.885 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.885 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-04-25T01:43:15.885 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-04-25T01:43:15.885 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.885 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.885 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.886 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-04-25T01:43:15.886 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.886 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.886 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.886 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T01:43:15.886 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.886 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.886 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.886 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-04-25T01:43:15.887 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-04-25T01:43:15.887 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.887 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.887 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.887 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-04-25T01:43:15.887 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.887 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.887 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.887 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-04-25T01:43:15.888 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.888 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.888 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.888 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-04-25T01:43:15.888 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-04-25T01:43:15.888 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.888 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.888 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.888 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.888 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.888 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.888 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.889 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T01:43:15.889 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.889 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.889 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.889 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-04-25T01:43:15.889 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-04-25T01:43:15.889 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.889 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.889 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.889 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-25T01:43:15.889 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.889 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.889 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.889 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-04-25T01:43:15.889 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.890 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.890 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.890 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-04-25T01:43:15.890 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-04-25T01:43:15.890 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.890 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.890 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.890 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T01:43:15.890 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.891 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.891 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.891 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T01:43:15.891 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.891 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.891 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.891 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-04-25T01:43:15.891 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-04-25T01:43:15.891 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.891 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.891 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.891 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-25T01:43:15.892 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.892 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.892 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.892 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-04-25T01:43:15.892 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.892 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.892 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.892 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-04-25T01:43:15.892 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-04-25T01:43:15.892 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.892 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.892 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.892 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.892 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.893 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.893 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.893 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T01:43:15.893 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.893 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.893 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.893 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-04-25T01:43:15.893 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-04-25T01:43:15.893 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.893 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.893 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.893 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.893 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.894 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.894 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.894 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T01:43:15.894 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.894 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.894 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.894 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.894 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.894 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.894 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.894 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.894 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.894 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.895 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.895 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.895 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.895 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.895 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.895 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.895 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.895 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.895 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.895 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.895 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.895 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.895 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.895 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-04-25T01:43:15.895 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-04-25T01:43:15.896 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.896 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.896 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.896 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.896 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.896 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.896 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.896 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T01:43:15.896 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.896 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.896 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.896 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.896 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.897 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.897 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.897 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.897 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.897 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.897 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.897 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.897 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T01:43:15.897 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.897 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.897 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.897 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.897 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.897 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.897 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.898 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.898 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.898 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.898 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.898 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.898 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.898 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.898 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.898 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.898 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.898 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.898 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.898 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.898 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.899 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.899 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.899 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.899 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.899 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.899 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.899 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.899 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.899 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.899 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.899 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.899 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.899 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.899 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.900 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.900 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T01:43:15.900 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T01:43:15.900 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.900 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.900 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.900 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.900 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.900 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.900 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.900 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T01:43:15.900 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.900 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.900 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.901 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T01:43:15.901 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T01:43:15.901 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.901 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.901 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.901 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-04-25T01:43:15.901 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.901 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.901 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.901 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T01:43:15.901 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.901 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.901 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.901 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-04-25T01:43:15.902 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-04-25T01:43:15.902 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.902 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.902 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.902 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:15.902 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.902 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.902 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.902 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-04-25T01:43:15.902 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.902 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.902 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.903 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-04-25T01:43:15.903 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-04-25T01:43:15.903 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.903 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.903 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.903 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:15.903 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.903 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.903 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.903 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T01:43:15.903 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.903 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.903 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.903 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-04-25T01:43:15.904 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-04-25T01:43:15.904 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.904 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.904 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.904 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.904 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.904 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.904 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.904 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-04-25T01:43:15.904 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.904 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.904 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.904 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-25T01:43:15.904 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-25T01:43:15.905 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.905 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.905 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.905 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-04-25T01:43:15.905 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.905 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.905 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.905 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-04-25T01:43:15.905 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-04-25T01:43:15.905 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.905 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T01:43:15.905 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.905 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.905 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.906 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-04-25T01:43:15.906 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-04-25T01:43:15.906 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.906 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.906 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.906 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-04-25T01:43:15.906 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.906 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.906 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.906 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T01:43:15.906 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.906 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.906 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.906 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-04-25T01:43:15.907 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-04-25T01:43:15.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.907 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.907 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.907 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T01:43:15.907 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.907 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.907 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.907 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T01:43:15.907 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.907 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.907 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.907 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-04-25T01:43:15.907 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-04-25T01:43:15.908 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.908 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.908 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.908 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.908 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.908 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.908 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.908 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T01:43:15.908 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.908 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.908 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.908 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-04-25T01:43:15.908 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-04-25T01:43:15.908 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.908 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.909 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.909 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.909 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.909 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.909 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.909 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T01:43:15.909 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.909 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.909 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.909 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-04-25T01:43:15.909 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-04-25T01:43:15.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.909 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.910 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.910 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.910 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.910 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.910 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.910 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T01:43:15.910 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.910 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.910 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.910 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-04-25T01:43:15.910 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-04-25T01:43:15.910 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.910 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.911 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.911 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.911 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.911 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.911 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.911 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T01:43:15.911 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.911 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.911 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.911 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-04-25T01:43:15.911 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-04-25T01:43:15.911 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.911 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.912 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.912 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.912 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.878+0000 7f067a909640 1 -- 172.21.15.28:0/987333514 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f065c0a2d00 msgr2=0x7f065c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:15.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.878+0000 7f067a909640 1 --2- 172.21.15.28:0/987333514 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f065c0a2d00 0x7f065c0a51c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f06640095f0 tx=0x7f0664005f90 comp rx=0 tx=0).stop 2024-04-25T01:43:15.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.878+0000 7f067a909640 1 -- 172.21.15.28:0/987333514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0674100550 msgr2=0x7f067419a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:15.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.878+0000 7f067a909640 1 --2- 172.21.15.28:0/987333514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0674100550 0x7f067419a210 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f066800e9f0 tx=0x7f066800eeb0 comp rx=0 tx=0).stop 2024-04-25T01:43:15.913 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.913 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.913 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T01:43:15.913 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.914 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.914 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.914 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-04-25T01:43:15.914 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-04-25T01:43:15.914 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.914 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.914 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.914 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-04-25T01:43:15.914 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.914 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.914 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.914 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-04-25T01:43:15.915 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.915 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.915 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.915 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-04-25T01:43:15.915 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-04-25T01:43:15.915 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.915 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.915 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.915 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.916 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.916 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.916 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.916 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T01:43:15.916 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.916 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.916 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.916 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-04-25T01:43:15.916 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-04-25T01:43:15.916 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.916 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.916 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.917 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.917 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.917 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.917 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.917 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T01:43:15.917 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.917 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.917 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.917 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-04-25T01:43:15.917 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-04-25T01:43:15.917 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.917 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.918 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.918 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.918 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.918 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.918 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.918 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T01:43:15.918 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.918 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.918 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.918 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-04-25T01:43:15.919 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-04-25T01:43:15.919 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.919 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.919 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.919 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.919 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.919 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.919 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.919 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T01:43:15.919 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.919 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.920 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.920 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-04-25T01:43:15.920 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-04-25T01:43:15.920 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.920 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.920 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.920 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.920 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.920 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.920 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.920 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T01:43:15.920 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.921 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.921 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.921 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-04-25T01:43:15.921 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-04-25T01:43:15.921 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.921 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.921 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.921 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.922 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.922 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.922 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.922 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T01:43:15.922 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.922 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.922 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.922 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.922 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.922 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-04-25T01:43:15.923 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.923 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.923 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.923 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.923 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.923 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.923 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.923 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.923 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.924 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.924 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.924 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.924 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.924 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.924 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.924 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.924 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.924 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.924 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.924 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.924 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.925 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.925 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.925 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.925 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.925 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.925 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.925 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.925 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.925 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.926 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.926 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.926 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.926 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.926 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.926 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.926 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.926 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.926 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.926 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.926 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.926 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.926 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.927 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.927 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.927 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.927 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.927 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.927 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.927 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.927 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.927 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.927 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.927 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.927 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.927 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.927 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.927 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.928 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.928 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.928 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.928 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.928 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.928 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.928 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.928 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.928 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.928 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.928 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.928 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-04-25T01:43:15.928 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-04-25T01:43:15.928 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.928 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.929 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.929 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-04-25T01:43:15.929 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.929 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.929 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.929 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T01:43:15.929 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.929 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.929 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.929 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-04-25T01:43:15.929 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-04-25T01:43:15.929 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:15.929 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.929 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.930 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-04-25T01:43:15.930 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.930 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.930 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.930 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T01:43:15.930 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.930 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.930 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.930 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.930 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.930 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-04-25T01:43:15.930 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.931 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.931 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.931 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T01:43:15.931 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T01:43:15.931 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.931 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.931 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.931 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:15.931 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.931 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.931 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.931 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.931 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.931 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.931 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.932 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T01:43:15.932 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T01:43:15.932 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.932 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.932 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.932 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.932 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.932 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.932 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.932 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.932 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.932 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.932 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.933 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T01:43:15.933 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T01:43:15.933 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.933 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.933 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.933 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.933 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.933 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.933 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.933 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.933 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.933 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.933 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.934 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-04-25T01:43:15.934 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T01:43:15.934 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.934 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.934 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.934 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.934 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.934 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.934 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.934 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.934 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.934 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.934 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.935 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T01:43:15.935 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T01:43:15.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.935 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.935 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.935 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.935 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.935 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.935 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.935 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.935 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.935 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.935 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.936 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T01:43:15.936 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T01:43:15.936 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.936 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.936 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.936 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.936 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.936 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.936 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.936 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.936 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.936 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.936 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.937 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T01:43:15.937 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T01:43:15.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.937 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.937 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.937 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.937 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.937 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.937 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.937 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.937 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.937 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.937 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.938 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T01:43:15.938 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T01:43:15.938 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.938 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.938 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.938 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.938 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.938 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.938 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.938 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.938 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.938 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.938 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.939 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T01:43:15.939 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T01:43:15.939 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.939 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.939 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.939 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.939 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.939 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.939 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.939 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.939 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.939 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.939 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T01:43:15.940 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T01:43:15.940 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.940 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.940 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.940 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.940 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.940 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.940 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.940 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.940 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.940 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.940 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.940 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T01:43:15.940 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T01:43:15.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.941 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.941 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.941 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.941 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.941 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.941 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.941 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.941 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.941 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.941 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.941 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T01:43:15.941 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T01:43:15.942 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.942 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.942 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.942 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.942 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.942 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.942 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.942 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.942 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.942 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.942 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.942 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T01:43:15.942 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T01:43:15.943 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.943 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.943 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.943 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.943 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.943 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.943 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.943 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.943 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.943 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.943 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.943 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T01:43:15.943 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T01:43:15.943 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.944 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.944 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.944 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.944 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.944 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.944 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.944 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.944 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.944 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.944 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.944 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T01:43:15.944 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T01:43:15.944 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.944 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.945 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.945 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-25T01:43:15.945 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.945 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.945 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.945 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.945 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.945 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.945 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.945 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T01:43:15.945 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T01:43:15.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.945 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.945 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.945 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.946 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.946 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.946 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.946 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.946 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.946 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.946 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.946 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-04-25T01:43:15.946 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-04-25T01:43:15.946 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.946 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.946 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.946 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.947 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.947 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.947 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.947 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.947 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.947 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.947 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.947 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T01:43:15.947 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T01:43:15.947 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.947 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.947 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.948 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-25T01:43:15.948 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.948 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.948 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.948 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.948 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.948 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.948 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.948 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T01:43:15.949 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T01:43:15.949 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.949 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.949 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.949 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.949 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.949 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.949 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.949 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.949 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.949 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.949 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.949 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T01:43:15.949 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T01:43:15.950 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.950 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.950 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.950 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.950 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.950 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.950 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.950 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.950 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.950 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.951 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.951 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T01:43:15.951 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T01:43:15.951 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.951 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.951 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.951 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.951 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.951 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.951 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.951 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.951 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.951 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.951 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.952 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T01:43:15.952 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T01:43:15.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.952 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.952 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.952 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.952 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.952 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.953 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.953 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.953 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.953 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.953 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.953 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T01:43:15.953 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T01:43:15.953 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.953 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.953 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.953 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.953 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.954 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.954 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.954 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.954 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.954 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.954 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.954 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T01:43:15.954 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T01:43:15.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.954 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.955 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.955 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.955 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.955 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.955 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.955 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.955 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.955 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.955 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T01:43:15.955 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T01:43:15.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.956 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.956 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.956 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.956 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.956 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.956 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.956 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.956 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.956 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.956 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.956 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T01:43:15.957 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T01:43:15.957 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.957 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.957 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.957 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.957 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.957 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.957 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.957 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.957 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.957 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.958 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.958 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T01:43:15.958 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T01:43:15.958 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.958 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.958 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.958 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.958 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.959 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.959 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.959 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.959 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.959 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.959 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.959 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T01:43:15.959 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T01:43:15.959 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.960 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.960 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.960 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.960 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.960 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.960 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.960 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.960 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.960 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.960 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.960 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T01:43:15.961 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T01:43:15.961 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.961 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.961 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.961 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.961 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.961 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.961 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.961 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.961 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.961 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.961 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.961 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T01:43:15.962 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T01:43:15.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.962 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.962 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.962 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.962 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.962 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.962 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.962 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.962 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.962 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.962 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.962 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T01:43:15.963 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T01:43:15.963 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.963 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.963 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.963 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.963 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.963 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.963 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.963 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.963 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.963 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.963 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.963 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T01:43:15.963 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T01:43:15.964 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.964 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.964 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.964 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.964 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.964 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.964 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.964 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.964 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.964 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.964 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.964 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T01:43:15.964 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T01:43:15.965 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.965 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.965 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.965 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T01:43:15.965 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.965 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.965 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.965 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.965 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.965 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.965 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.965 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T01:43:15.965 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T01:43:15.966 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.966 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.966 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.966 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:15.966 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.966 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.966 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.966 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.966 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.966 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.966 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.966 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T01:43:15.966 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T01:43:15.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.967 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.967 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.967 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-04-25T01:43:15.967 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.967 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.967 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.967 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.967 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.967 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.967 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.967 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T01:43:15.967 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T01:43:15.968 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.968 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.968 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.968 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-04-25T01:43:15.968 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.968 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.968 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.968 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.968 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.968 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.968 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.968 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T01:43:15.968 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T01:43:15.969 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.969 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.969 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.969 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.969 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.969 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.969 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.969 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.969 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.969 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.969 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.969 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T01:43:15.969 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T01:43:15.970 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.970 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.970 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.970 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-25T01:43:15.970 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.970 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.970 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.970 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.970 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.970 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.970 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.970 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T01:43:15.970 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T01:43:15.971 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.971 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.971 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.971 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.971 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.971 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.971 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.971 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.971 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.971 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.971 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.971 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T01:43:15.971 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T01:43:15.971 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.972 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.972 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.972 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.972 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.972 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.972 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.972 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.972 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.972 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.972 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.972 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T01:43:15.972 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T01:43:15.972 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.972 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.973 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.973 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-25T01:43:15.973 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.973 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.973 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.973 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.973 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.973 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.973 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.973 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T01:43:15.973 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T01:43:15.973 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.973 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.973 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.974 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:15.974 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.974 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.974 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.974 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.974 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.974 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.974 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.974 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T01:43:15.974 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T01:43:15.974 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.974 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.974 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.975 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-25T01:43:15.975 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.975 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.975 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.975 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.975 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.975 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.975 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.975 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-04-25T01:43:15.975 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-04-25T01:43:15.975 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.975 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.975 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.975 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.975 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.976 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.976 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.976 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.976 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.976 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.976 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.976 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-04-25T01:43:15.976 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-04-25T01:43:15.976 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.976 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.976 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.976 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.976 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.976 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.977 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.977 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.977 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.977 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.977 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.977 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-04-25T01:43:15.977 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-04-25T01:43:15.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.977 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.977 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.977 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.977 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.977 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.977 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.978 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-04-25T01:43:15.978 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.978 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.978 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.978 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-04-25T01:43:15.978 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-04-25T01:43:15.978 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.978 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.978 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.978 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-04-25T01:43:15.978 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.978 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.979 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.979 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.979 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.979 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.979 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.979 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-25T01:43:15.979 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-25T01:43:15.979 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.979 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.979 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.979 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.979 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.979 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.980 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.980 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.980 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.980 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.980 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.980 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.980 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.980 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.980 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.980 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.878+0000 7f067a909640 1 -- 172.21.15.28:0/987333514 shutdown_connections 2024-04-25T01:43:15.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.878+0000 7f067a909640 1 --2- 172.21.15.28:0/987333514 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f065c0a2d00 0x7f065c0a51c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:15.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.878+0000 7f067a909640 1 --2- 172.21.15.28:0/987333514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067410e100 0x7f067419ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:15.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.878+0000 7f067a909640 1 --2- 172.21.15.28:0/987333514 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0674100e70 0x7f067419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:15.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.878+0000 7f067a909640 1 --2- 172.21.15.28:0/987333514 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0674100550 0x7f067419a210 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:15.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.878+0000 7f067a909640 1 -- 172.21.15.28:0/987333514 >> 172.21.15.28:0/987333514 conn(0x7f0674077740 msgr2=0x7f067410f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:15.981 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.982 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.982 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.982 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.982 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.982 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.982 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.982 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.982 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.983 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.983 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.983 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.983 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.983 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.983 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.983 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:15.983 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:15.983 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.983 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.983 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.983 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.983 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.983 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.984 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.984 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.984 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.984 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.984 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.984 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:15.984 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:15.984 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.984 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.984 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.984 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:15.984 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.984 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.985 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.985 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:15.985 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:15.985 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:15.985 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.985 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:15.985 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:15.985 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.985 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.985 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.985 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.985 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.985 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:15.985 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:15.986 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.986 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.986 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.986 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.986 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.986 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.986 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.986 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.986 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.986 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.986 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.986 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-04-25T01:43:15.987 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-04-25T01:43:15.987 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.987 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.987 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.987 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.987 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.987 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.987 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.987 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-04-25T01:43:15.987 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.988 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.988 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.988 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-04-25T01:43:15.988 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T01:43:15.988 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.988 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.988 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.988 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:15.988 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.988 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.988 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.988 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.988 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.988 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.989 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.989 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-25T01:43:15.989 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-25T01:43:15.989 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.989 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.989 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.989 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-25T01:43:15.989 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.989 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.990 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.990 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.990 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.990 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.990 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.990 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-25T01:43:15.990 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-25T01:43:15.990 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.990 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.990 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-04-25T01:43:15.990 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.990 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.991 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.991 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.991 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.991 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.991 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.991 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-25T01:43:15.991 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-25T01:43:15.991 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.991 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.992 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.992 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.992 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.992 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.992 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.992 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.992 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.992 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.992 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-04-25T01:43:15.992 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-04-25T01:43:15.993 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.993 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.993 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.993 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-04-25T01:43:15.993 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.993 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.993 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.993 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.993 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.993 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.993 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.993 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-25T01:43:15.994 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-25T01:43:15.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.994 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.994 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.994 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-04-25T01:43:15.994 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.994 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.994 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:15.994 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:15.994 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-04-25T01:43:15.995 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:15.995 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.995 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.995 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.995 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.995 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-25T01:43:15.995 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-25T01:43:15.995 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:15.995 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.996 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.996 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-25T01:43:15.996 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-25T01:43:15.996 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-25T01:43:15.996 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.996 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.996 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.996 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.996 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.996 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-04-25T01:43:15.997 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-04-25T01:43:15.997 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.997 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.997 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:15.997 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:15.997 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.997 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.997 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.997 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:15.997 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.998 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.998 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.998 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.998 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:15.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:15.998 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-25T01:43:15.998 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:15.998 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:15.998 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:15.998 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-04-25T01:43:15.998 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-04-25T01:43:15.998 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:15.998 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.999 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:15.999 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:15.999 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:15.999 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:15.999 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:15.999 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-04-25T01:43:15.999 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:15.999 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:15.999 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:15.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:15.999 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:15.999 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:15.999 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:15.999 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:15.999 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.000 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.000 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.000 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.000 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.000 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.000 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.000 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.000 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.000 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.000 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.000 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.000 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.000 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.000 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.000 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.001 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.001 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.001 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.001 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.001 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.001 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.001 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.001 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.001 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.001 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.001 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.001 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.001 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.002 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.002 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.002 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.002 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.002 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.002 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.002 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.002 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.002 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.002 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.002 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.002 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.002 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.002 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.003 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.003 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.003 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.003 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.003 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.003 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.003 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.003 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.003 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.003 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.003 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.003 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.003 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.004 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.004 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.004 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.004 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.004 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.004 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.004 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-04-25T01:43:16.004 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-04-25T01:43:16.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:16.004 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.004 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.004 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-04-25T01:43:16.004 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.004 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.004 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.005 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T01:43:16.005 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.005 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.005 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.005 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-04-25T01:43:16.005 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-04-25T01:43:16.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.005 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.005 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-04-25T01:43:16.005 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.005 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.005 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.006 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T01:43:16.006 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.006 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.006 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.006 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-04-25T01:43:16.006 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-04-25T01:43:16.006 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:16.006 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.006 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.006 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-04-25T01:43:16.006 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.006 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.006 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.007 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-04-25T01:43:16.007 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.007 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.007 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.007 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-04-25T01:43:16.007 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-04-25T01:43:16.007 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:16.007 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.007 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.007 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-25T01:43:16.007 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.007 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.007 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.007 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T01:43:16.008 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.008 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.008 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.008 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-04-25T01:43:16.008 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-04-25T01:43:16.008 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.008 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.008 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.008 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:16.008 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.008 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.008 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.008 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T01:43:16.008 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.009 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.009 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-25T01:43:16.009 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-25T01:43:16.009 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:16.009 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.009 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.009 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-25T01:43:16.009 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.009 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.009 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.009 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T01:43:16.009 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.009 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.010 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-04-25T01:43:16.010 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-04-25T01:43:16.010 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:16.010 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.010 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-04-25T01:43:16.010 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.010 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.010 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.010 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T01:43:16.010 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.010 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.010 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.011 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.011 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.011 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-04-25T01:43:16.011 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.011 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.011 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.011 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.011 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.011 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.011 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.011 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.011 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.011 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.011 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.012 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.012 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.012 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.012 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.012 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.012 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.012 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.012 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.012 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.012 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.012 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.012 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.012 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.013 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.013 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.013 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.013 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.013 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.013 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.013 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.013 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.013 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.013 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.013 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.013 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.013 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.013 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.014 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.014 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.014 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.014 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.014 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.014 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.014 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.014 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.014 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.014 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.014 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.014 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.014 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.014 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.015 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.015 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.015 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.015 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.015 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.015 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.015 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.015 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.015 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.015 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.015 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.015 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.015 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.016 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.016 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.016 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.016 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-04-25T01:43:16.016 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-04-25T01:43:16.016 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.016 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.016 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-25T01:43:16.016 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.017 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.017 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.017 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.017 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.017 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.017 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.017 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-04-25T01:43:16.017 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-04-25T01:43:16.017 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.017 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.017 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.017 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-04-25T01:43:16.017 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.017 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.018 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.018 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.018 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.018 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.018 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.018 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-25T01:43:16.018 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-25T01:43:16.018 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.018 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.018 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.018 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-25T01:43:16.018 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.018 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.019 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.019 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.019 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.019 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.019 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.019 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.019 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.019 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-04-25T01:43:16.019 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-04-25T01:43:16.019 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-04-25T01:43:16.019 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.020 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-04-25T01:43:16.020 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-04-25T01:43:16.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.020 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.020 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.020 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-04-25T01:43:16.020 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.020 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.020 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.020 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T01:43:16.020 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.021 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.021 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.021 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-04-25T01:43:16.021 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-04-25T01:43:16.021 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.021 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.021 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.021 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-04-25T01:43:16.021 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.021 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.021 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.021 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T01:43:16.022 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.022 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.022 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.022 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-04-25T01:43:16.022 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-04-25T01:43:16.022 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.022 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.022 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.022 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.023 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.023 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.023 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.023 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-04-25T01:43:16.023 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.023 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.023 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.023 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-25T01:43:16.023 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-25T01:43:16.023 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:16.023 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.024 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-04-25T01:43:16.024 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-04-25T01:43:16.024 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.024 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.024 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T01:43:16.024 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.024 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.024 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.024 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.025 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.025 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.025 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.025 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.025 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.025 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.025 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.025 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.025 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.025 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.025 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.025 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.025 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.025 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.025 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.026 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.026 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.026 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.026 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.026 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.026 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.026 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.026 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.027 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.027 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.027 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.027 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.027 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.027 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.027 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.027 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.027 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.027 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.027 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.028 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.028 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.028 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.028 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.028 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.028 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.028 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.028 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.028 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.028 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.028 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.028 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.029 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.029 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.029 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.029 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.029 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.029 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.029 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.029 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.029 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.030 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.030 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.030 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.030 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.030 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.030 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.030 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.030 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.030 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.030 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.031 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-04-25T01:43:16.031 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-04-25T01:43:16.031 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.031 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.031 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.031 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.031 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.031 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.032 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.032 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-04-25T01:43:16.032 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.032 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.032 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.032 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-04-25T01:43:16.032 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-04-25T01:43:16.032 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.032 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.032 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.032 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-04-25T01:43:16.033 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.033 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.033 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.033 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-04-25T01:43:16.033 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.033 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.033 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.033 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-25T01:43:16.033 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-25T01:43:16.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.033 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.033 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.033 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-04-25T01:43:16.034 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.034 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.034 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.034 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T01:43:16.034 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.034 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.034 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.034 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-04-25T01:43:16.034 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-04-25T01:43:16.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.034 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.034 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.034 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-25T01:43:16.035 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-25T01:43:16.035 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-04-25T01:43:16.035 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.035 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T01:43:16.035 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.035 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.035 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.035 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-04-25T01:43:16.035 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-04-25T01:43:16.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.035 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.035 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.035 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.035 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.036 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.036 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.036 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-04-25T01:43:16.036 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.036 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.036 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.036 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-04-25T01:43:16.036 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-04-25T01:43:16.036 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.036 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.036 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.036 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-04-25T01:43:16.036 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.037 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.037 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.037 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T01:43:16.037 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.037 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.037 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.037 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.037 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.037 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-04-25T01:43:16.037 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.037 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.037 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.037 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.038 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.038 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.038 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.038 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.038 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.038 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.038 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.038 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.038 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.038 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.038 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.038 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.038 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.038 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.039 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.039 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.039 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.039 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.039 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.039 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.039 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.039 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.039 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.039 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.039 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.039 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.040 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.040 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.040 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.040 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.040 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.040 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.040 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.040 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.040 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.040 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.040 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.040 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.040 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.041 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.041 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.041 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.041 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.041 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.041 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.041 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.041 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.041 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.041 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.041 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.041 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.041 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.042 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.042 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.042 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.042 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.042 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.042 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.042 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.042 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.042 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.042 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.042 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.042 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.043 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-04-25T01:43:16.043 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.043 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.043 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.043 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.043 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.043 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.043 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.043 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.043 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.043 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.043 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.043 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.043 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.044 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.044 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.044 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.044 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.044 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.044 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.044 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.044 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.044 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.044 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.044 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.044 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.045 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.045 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.045 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.045 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.045 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.045 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.045 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.045 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.045 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.045 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.045 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.045 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.045 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.045 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.045 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.046 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.046 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.046 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.046 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.046 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.046 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.046 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.046 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.046 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.046 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.046 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.046 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.046 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.046 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.047 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.047 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.047 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.047 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.047 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.047 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.047 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.047 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.047 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.047 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.047 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.047 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.047 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.047 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.048 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.048 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.049 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-04-25T01:43:16.049 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.049 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.049 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.049 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-04-25T01:43:16.049 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-04-25T01:43:16.049 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.049 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.049 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.050 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-04-25T01:43:16.050 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.050 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.050 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.050 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T01:43:16.050 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.050 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.050 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.050 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-04-25T01:43:16.050 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-04-25T01:43:16.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.050 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.050 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.051 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:16.051 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-04-25T01:43:16.051 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.051 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.051 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T01:43:16.051 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.051 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.051 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.051 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.051 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.051 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.051 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.051 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.052 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.052 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.052 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.052 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.052 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.052 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.052 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.052 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.052 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.052 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.052 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.052 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.052 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.052 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.053 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.878+0000 7f067a909640 1 -- 172.21.15.28:0/987333514 shutdown_connections 2024-04-25T01:43:16.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:14.878+0000 7f067a909640 1 -- 172.21.15.28:0/987333514 wait complete. 2024-04-25T01:43:16.055 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.055 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.055 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.055 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.055 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.055 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.055 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.055 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.055 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.056 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.056 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.056 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.056 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.056 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.056 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.056 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.056 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.056 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.056 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.056 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.057 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.057 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.057 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.057 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.057 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.057 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.057 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.057 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.057 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.057 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.058 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.058 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.058 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.058 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.058 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.058 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.058 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.058 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.058 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.058 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.058 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.058 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.058 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.059 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.059 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.059 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.059 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.059 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-04-25T01:43:16.059 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.059 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.059 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.059 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-04-25T01:43:16.059 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-04-25T01:43:16.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.060 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.060 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.060 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-04-25T01:43:16.060 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.060 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.060 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.060 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-04-25T01:43:16.060 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T01:43:16.060 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.060 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.061 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.061 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.061 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.061 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.061 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.061 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.061 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.061 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.061 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.061 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.061 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.061 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.061 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.061 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.061 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.062 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.062 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.062 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.062 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.062 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.062 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.062 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.062 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.062 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.062 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.062 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.063 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.063 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.063 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.063 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.063 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.063 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.063 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.063 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.063 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.063 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.063 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.063 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.063 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.064 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.064 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.064 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.064 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.064 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.064 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.064 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.064 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.064 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.064 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.065 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.065 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.065 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.065 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.065 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.065 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.065 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.065 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.065 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.065 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.065 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.065 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.065 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.066 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.066 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.066 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.066 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-04-25T01:43:16.066 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-04-25T01:43:16.066 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.066 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.066 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.066 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.067 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.067 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.067 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.067 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-04-25T01:43:16.067 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T01:43:16.067 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.067 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.067 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-04-25T01:43:16.067 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-04-25T01:43:16.067 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.067 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.068 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.068 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-04-25T01:43:16.068 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.068 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.068 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.068 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-04-25T01:43:16.068 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.068 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.068 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.068 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-04-25T01:43:16.068 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-04-25T01:43:16.069 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.069 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.069 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.069 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-04-25T01:43:16.069 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.069 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.069 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.069 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T01:43:16.069 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.069 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.069 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.070 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-04-25T01:43:16.070 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-04-25T01:43:16.070 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.070 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.070 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.070 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.070 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.070 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.070 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.071 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-04-25T01:43:16.071 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.071 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.071 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.071 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-04-25T01:43:16.071 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-04-25T01:43:16.071 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.071 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.071 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.072 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-04-25T01:43:16.072 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.072 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.072 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.072 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T01:43:16.072 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T01:43:16.072 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.072 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.072 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.072 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.073 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-04-25T01:43:16.073 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.073 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.073 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.073 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.073 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.073 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.073 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.073 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.073 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.073 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.073 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.073 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.073 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.074 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.074 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.074 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.074 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.074 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.074 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.074 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.074 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.074 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.074 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.074 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.074 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.075 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.075 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.075 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.075 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.075 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.075 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.075 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.075 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.075 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.075 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.075 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.075 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.075 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.076 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.076 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.076 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.076 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.076 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.076 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.076 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.076 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.076 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.076 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.076 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.076 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.076 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.076 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.077 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.077 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.077 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.077 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.077 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.077 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.077 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.077 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.077 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.077 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.077 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.077 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.077 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.078 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.078 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.078 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.078 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.078 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.078 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-04-25T01:43:16.078 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.078 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.078 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.078 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.078 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.078 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.078 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.079 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.079 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.079 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.079 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.079 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.079 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.079 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.079 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.079 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.079 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.079 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.079 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.079 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.079 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.080 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.080 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.080 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.080 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.080 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.080 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.080 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.080 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.080 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.080 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.080 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.080 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.080 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.080 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.081 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.081 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.081 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.081 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.081 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.081 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.081 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.081 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.081 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.081 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.081 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.081 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.081 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.081 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.082 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.082 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.082 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.082 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.082 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.082 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.082 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.082 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.082 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.082 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.083 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.083 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.083 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.083 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.083 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.083 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.083 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.083 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.083 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.083 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.083 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-04-25T01:43:16.083 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.083 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.084 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.084 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.084 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.084 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.084 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.084 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.084 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.084 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.084 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.084 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.084 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.084 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.084 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.085 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.085 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.085 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.085 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.085 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.085 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.085 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.085 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.085 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.085 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.085 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.085 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.086 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.086 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.086 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.086 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.086 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.086 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.086 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.086 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.086 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.086 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.087 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.087 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.087 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.087 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.087 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.087 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.087 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.087 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.087 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.087 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.087 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.087 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.087 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.087 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.087 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.088 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.088 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.088 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.088 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.088 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.088 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.088 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.088 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.088 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.088 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.088 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.088 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.088 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.088 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.088 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.089 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.089 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.089 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.089 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-25T01:43:16.089 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.089 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.089 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.089 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-04-25T01:43:16.089 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-04-25T01:43:16.089 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.089 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.089 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.089 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.089 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.089 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.090 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.090 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T01:43:16.090 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.090 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.090 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.090 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.090 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.090 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.090 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.090 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.090 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.090 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.090 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.090 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.091 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.091 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.091 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.091 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.091 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.091 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.091 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.091 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.091 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.091 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.091 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.091 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.091 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.091 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.091 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.092 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.092 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.092 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.092 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.092 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.092 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.092 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.092 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.092 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.092 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.092 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.092 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.092 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.092 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.093 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.093 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.093 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.093 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.093 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.093 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.093 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.093 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.093 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.093 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.093 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.093 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.093 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.093 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.094 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.094 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.094 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.094 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.094 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.094 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.094 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.094 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.095 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.095 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.095 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.095 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.095 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.095 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.095 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-04-25T01:43:16.095 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-25T01:43:16.095 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.095 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.095 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.095 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.095 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.095 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.095 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.096 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-25T01:43:16.096 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-04-25T01:43:16.096 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-04-25T01:43:16.096 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.096 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-04-25T01:43:16.096 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.096 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.096 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.096 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.096 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.097 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-04-25T01:43:16.097 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.097 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.097 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.097 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.097 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.097 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.097 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.097 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.097 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.097 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.097 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.098 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.098 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.098 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.098 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.098 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.098 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.098 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.098 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.098 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.098 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.099 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.099 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.099 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.099 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.099 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.099 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.099 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.099 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.099 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.099 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.099 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.100 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.100 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.100 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.100 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.100 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.100 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.100 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.100 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.100 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.100 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.101 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.101 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.101 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.101 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.101 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.101 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.101 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.101 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.101 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.101 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.101 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.101 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.101 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.102 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.102 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.102 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.102 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.102 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.102 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.102 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.102 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.102 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.102 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.102 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.103 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.103 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.103 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.103 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.103 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-04-25T01:43:16.103 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.103 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.103 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.103 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.103 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.104 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.104 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.104 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.104 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.104 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.104 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.104 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.104 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.104 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.104 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.104 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.105 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.105 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.105 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.105 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.105 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.105 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.105 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.105 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.105 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.106 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.106 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.106 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.106 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.106 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.106 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.106 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.106 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.106 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.106 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.107 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.107 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.107 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.107 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.107 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.107 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.107 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.107 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.107 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.107 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.107 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.107 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.107 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.108 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.108 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.108 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.108 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.108 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.108 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.108 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.108 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.108 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.108 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.108 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.108 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.108 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.108 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.109 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.109 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.109 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.109 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.109 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.109 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.109 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.109 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.109 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.109 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-04-25T01:43:16.109 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.109 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.109 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.110 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.110 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.110 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.110 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.110 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.110 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.110 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.110 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.110 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.110 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.110 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.110 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.110 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.110 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.111 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.112 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.112 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.112 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.112 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.112 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.112 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.112 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.112 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.112 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.112 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.112 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.112 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.112 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.112 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.113 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.113 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.113 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.113 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.113 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.113 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.113 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.113 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.113 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.113 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.113 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.113 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.114 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.114 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.114 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.114 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.114 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.114 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.114 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.114 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.114 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.114 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.114 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.114 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.114 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-25T01:43:16.115 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-25T01:43:16.115 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:16.115 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.115 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.115 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-25T01:43:16.115 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.115 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.115 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.115 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.115 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.115 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.115 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.115 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-04-25T01:43:16.116 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-04-25T01:43:16.116 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:16.116 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.116 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.116 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-25T01:43:16.116 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-04-25T01:43:16.116 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.116 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.116 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-04-25T01:43:16.116 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T01:43:16.116 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.116 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.116 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.117 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.117 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-04-25T01:43:16.117 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.117 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.117 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.117 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-04-25T01:43:16.117 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-04-25T01:43:16.117 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.117 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.117 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.117 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.117 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.117 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.118 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.118 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T01:43:16.118 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.118 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.118 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.118 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-25T01:43:16.118 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-25T01:43:16.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.118 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.118 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.118 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:16.118 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.118 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.119 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.119 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.119 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.119 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.119 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.119 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.119 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.119 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.119 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.119 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.119 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.119 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.119 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.119 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.120 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.120 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.120 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.120 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.120 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.120 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.120 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.120 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.120 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.120 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.120 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.120 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.120 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.120 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.120 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.121 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.121 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.121 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.121 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.121 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.121 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.121 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.121 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.121 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.121 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.121 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.121 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.121 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.121 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.121 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.122 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.122 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.122 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.122 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.122 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.122 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.122 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.122 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.122 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.122 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.122 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.122 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.122 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.122 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.123 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.123 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.123 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.123 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.123 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.123 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.123 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.123 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.123 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.123 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.123 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.123 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.123 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.124 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-04-25T01:43:16.124 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-04-25T01:43:16.124 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.124 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.124 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.124 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-04-25T01:43:16.124 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.124 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.124 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.124 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-04-25T01:43:16.124 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.124 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.124 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.125 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-25T01:43:16.125 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-25T01:43:16.125 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:16.125 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.125 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.125 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-25T01:43:16.125 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.125 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.125 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.125 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-04-25T01:43:16.125 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.125 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.125 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.125 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.125 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.126 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-04-25T01:43:16.126 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.126 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.126 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.126 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-04-25T01:43:16.126 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-04-25T01:43:16.126 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.126 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.126 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.126 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:16.126 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.127 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.127 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.127 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.127 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.127 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.127 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.127 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-04-25T01:43:16.127 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-04-25T01:43:16.127 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.127 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.127 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.127 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:16.127 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.127 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.128 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.128 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T01:43:16.128 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T01:43:16.128 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.128 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.128 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.128 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.128 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.128 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.128 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.128 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.128 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.128 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.129 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.129 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.129 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.129 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.129 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.129 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.129 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.129 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.129 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.132 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.132 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.132 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.132 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.132 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.132 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.132 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.132 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.132 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.132 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.132 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.132 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.132 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.133 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.133 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.133 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.133 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.133 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.133 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.133 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.133 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.133 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.133 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.134 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.134 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.134 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.134 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.134 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-04-25T01:43:16.134 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-04-25T01:43:16.134 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.134 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.134 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.134 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.134 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.135 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.135 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.135 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.135 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.135 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.135 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.135 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T01:43:16.135 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T01:43:16.135 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.135 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.135 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.136 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-04-25T01:43:16.136 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.136 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.136 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.136 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.136 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.136 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.136 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.136 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-04-25T01:43:16.136 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-04-25T01:43:16.136 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:16.136 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.137 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.137 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-04-25T01:43:16.137 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.137 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.137 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.137 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.137 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.137 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.137 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.137 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-25T01:43:16.137 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-25T01:43:16.137 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.137 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.138 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.138 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-25T01:43:16.138 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.138 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.138 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.138 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T01:43:16.138 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.138 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.138 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.138 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-25T01:43:16.138 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-25T01:43:16.138 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.138 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.139 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.139 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-04-25T01:43:16.139 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.139 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.139 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.139 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T01:43:16.139 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.139 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.139 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.139 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-04-25T01:43:16.140 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-04-25T01:43:16.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.140 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.140 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-04-25T01:43:16.140 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.140 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.140 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.140 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.140 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.140 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.140 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.141 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-25T01:43:16.141 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-25T01:43:16.141 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.141 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.141 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.141 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-04-25T01:43:16.141 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.141 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.141 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.141 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-04-25T01:43:16.142 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-04-25T01:43:16.142 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.142 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.142 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.142 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.142 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.142 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-25T01:43:16.142 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-25T01:43:16.143 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.143 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.143 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.143 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-25T01:43:16.143 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-25T01:43:16.143 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-25T01:43:16.143 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.143 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.143 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.143 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-25T01:43:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-04-25T01:43:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-04-25T01:43:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-25T01:43:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-04-25T01:43:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T01:43:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-04-25T01:43:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-04-25T01:43:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.153 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-04-25T01:43:16.153 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.153 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.153 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.153 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-04-25T01:43:16.153 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-04-25T01:43:16.153 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.153 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.153 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.153 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:16.153 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.153 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.153 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.154 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.154 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.154 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.154 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.154 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-25T01:43:16.154 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-25T01:43:16.154 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.154 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.154 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.154 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.154 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.154 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.154 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.155 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.155 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.155 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.155 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.155 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.155 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.155 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.155 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.155 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.155 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.155 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.155 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.155 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.155 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.156 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.156 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.156 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.156 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.156 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.156 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.156 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.156 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.156 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.156 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.156 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.156 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.156 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.156 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.157 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.157 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.157 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.157 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.157 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.157 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.157 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.157 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.157 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.157 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.157 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.157 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.157 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.158 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.158 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.158 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.158 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.158 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.158 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.158 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.158 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.158 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.158 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.158 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.158 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.158 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.158 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.158 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.159 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.159 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.159 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.159 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.159 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.159 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.159 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.159 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.159 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.159 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.159 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.159 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.159 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.160 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.160 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-25T01:43:16.160 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-25T01:43:16.160 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.160 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.160 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.160 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.160 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.160 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.160 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.160 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.160 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.160 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.160 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.161 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-25T01:43:16.161 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-25T01:43:16.161 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.161 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.161 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.161 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.161 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.161 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.161 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.161 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.161 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.161 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.161 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.161 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.161 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.162 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-04-25T01:43:16.162 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.162 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.162 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.162 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.162 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.162 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.162 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.162 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.162 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.162 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.163 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.163 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.163 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.163 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.163 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.163 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.163 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.163 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.163 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.163 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.163 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.163 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.163 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.163 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.164 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.164 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.164 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.164 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.164 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.164 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.164 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.164 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.164 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.164 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.164 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.164 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.164 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.165 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.165 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.165 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.165 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.165 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.165 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.165 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.165 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.165 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.165 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.165 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.165 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.165 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.165 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.166 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.166 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.166 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.166 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.166 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.166 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.166 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.166 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.166 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.167 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.167 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.167 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.167 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.167 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.167 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.167 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.167 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.167 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.167 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.167 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-04-25T01:43:16.167 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.168 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.168 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.168 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.168 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.168 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.168 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.168 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.168 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.168 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.169 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.169 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.169 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.169 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.169 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.169 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.169 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.169 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.169 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.169 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.169 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.169 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.169 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.170 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.170 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.170 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.170 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.170 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.170 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.170 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.170 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.170 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.171 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.171 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.171 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.171 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.171 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.171 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.171 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.171 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.171 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.171 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.171 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.171 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.171 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.172 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.172 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.172 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.172 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.172 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.172 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.172 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.172 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.172 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.173 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.173 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.173 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.173 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.173 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.173 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.173 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.173 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.173 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.173 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.173 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.174 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.174 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.174 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.174 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.174 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-04-25T01:43:16.174 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-04-25T01:43:16.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.174 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.174 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.174 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-04-25T01:43:16.175 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.175 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.175 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.175 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T01:43:16.175 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.175 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.175 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.175 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.175 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.176 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-04-25T01:43:16.176 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.176 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.176 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.176 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.176 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.176 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.176 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.176 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.176 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.177 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.177 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.177 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.177 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.177 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.177 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.177 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.177 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.177 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.178 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.178 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.178 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.178 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.178 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.178 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.178 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.178 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.178 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.178 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.179 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.179 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.179 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.179 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.179 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.179 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.179 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.179 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.179 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.179 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.179 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.179 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.179 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.179 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.180 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.180 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.180 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.180 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.180 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.180 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.180 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.180 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.180 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.180 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.180 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.180 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.180 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.181 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.181 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.181 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.181 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.181 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.181 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.181 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.181 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.181 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.181 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.181 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.181 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.182 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.182 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-04-25T01:43:16.182 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-04-25T01:43:16.182 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.182 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.182 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.182 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.182 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.182 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.182 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.182 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.182 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.183 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.183 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.183 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-04-25T01:43:16.183 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-04-25T01:43:16.183 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.183 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.183 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.183 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-04-25T01:43:16.183 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.183 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.183 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.183 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.183 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.183 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.184 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.184 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-04-25T01:43:16.184 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-04-25T01:43:16.184 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.184 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.184 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.184 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.184 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.184 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.184 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.184 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.184 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.184 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.185 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.185 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-04-25T01:43:16.185 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-04-25T01:43:16.185 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.185 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.185 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.185 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.185 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.185 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.185 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.185 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.185 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.185 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.185 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.186 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-04-25T01:43:16.186 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-04-25T01:43:16.186 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-25T01:43:16.186 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.186 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.186 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.186 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.186 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.186 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.186 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.186 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.186 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.186 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.187 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-04-25T01:43:16.187 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-04-25T01:43:16.187 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.187 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.187 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.187 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.187 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.187 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.187 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.187 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.187 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.187 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.187 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.188 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-04-25T01:43:16.188 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-04-25T01:43:16.188 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.188 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.188 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.188 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.188 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.188 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.188 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.188 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.188 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.188 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.188 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-04-25T01:43:16.189 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-04-25T01:43:16.189 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.189 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.189 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.189 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:16.189 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.189 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.189 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.189 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.189 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.189 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.189 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.189 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-04-25T01:43:16.190 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-04-25T01:43:16.190 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.190 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.190 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.190 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.190 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-25T01:43:16.190 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-04-25T01:43:16.190 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.190 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.190 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.190 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.190 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.190 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-04-25T01:43:16.191 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-04-25T01:43:16.191 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.191 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.191 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.191 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.191 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.191 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.191 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.191 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.191 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.191 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.191 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.191 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-04-25T01:43:16.191 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-04-25T01:43:16.191 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.192 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.192 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.192 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.192 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.192 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.192 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.192 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.192 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.192 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.192 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.192 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-04-25T01:43:16.192 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-04-25T01:43:16.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.193 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.193 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.193 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.193 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.193 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.193 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.193 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.193 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.193 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.193 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.193 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.193 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.194 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-25T01:43:16.194 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.194 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.194 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.194 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-04-25T01:43:16.194 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-04-25T01:43:16.194 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.194 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.194 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.194 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.194 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.194 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.194 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.194 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T01:43:16.195 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.195 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.195 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.195 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-04-25T01:43:16.195 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-04-25T01:43:16.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.195 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.195 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.195 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.195 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.195 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.195 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.195 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-04-25T01:43:16.195 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.195 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.196 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.196 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.196 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.196 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.196 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.196 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.196 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.196 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.196 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.196 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.196 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.196 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.196 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.196 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.197 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.197 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.197 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.197 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.197 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.197 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.197 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.197 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.197 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.197 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.197 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.197 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.197 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.198 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.198 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.198 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.198 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.198 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.198 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.198 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.198 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.198 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.198 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.198 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.198 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.199 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.199 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.199 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.199 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.199 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.199 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.199 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.199 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.199 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.199 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.199 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.199 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.199 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.199 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.200 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.200 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.200 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.200 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.200 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.200 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.200 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.201 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.201 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.201 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.201 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.201 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.201 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.201 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.201 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.201 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-04-25T01:43:16.201 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.201 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.201 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.202 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.202 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.202 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.202 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.202 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.202 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.202 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.202 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.202 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.203 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.203 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.203 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.203 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.203 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.203 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.203 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.203 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.203 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.203 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.203 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.204 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.204 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.204 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.204 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.204 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.204 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.204 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.204 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.204 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.204 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.204 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.204 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.205 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.205 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.205 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.205 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.205 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.205 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.205 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.205 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.205 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.205 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.205 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.206 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.206 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.206 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.206 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.206 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.206 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.206 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.206 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.206 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.206 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.207 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.207 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.207 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.207 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.207 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.207 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.207 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.207 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.207 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.207 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.207 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.207 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.208 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.208 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.208 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.208 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-04-25T01:43:16.208 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.208 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.209 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.209 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.209 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.209 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.209 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.209 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.209 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.209 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.209 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.209 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.209 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.209 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.209 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.209 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.209 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.210 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.210 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.210 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.210 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.210 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.210 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.210 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.210 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.210 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.210 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.210 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.210 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.210 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.211 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.211 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.211 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.211 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.211 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.211 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.211 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.211 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.211 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.211 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.211 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.211 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.211 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.211 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.212 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.212 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.212 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.212 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.212 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.212 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.212 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.212 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.212 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.213 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.213 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.213 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.213 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.213 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.213 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.213 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.214 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.214 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.214 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.214 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.214 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.214 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.214 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.214 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.214 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.215 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.215 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-04-25T01:43:16.215 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.215 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.215 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.215 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-04-25T01:43:16.215 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-04-25T01:43:16.215 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.216 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.216 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.216 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T01:43:16.216 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.216 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.216 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.216 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.216 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.216 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.216 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.216 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-25T01:43:16.216 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-25T01:43:16.216 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:16.217 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.217 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.217 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-04-25T01:43:16.217 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.217 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.217 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.217 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.217 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.217 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.217 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.217 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.217 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.217 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.217 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.218 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.218 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.218 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.218 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.218 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.218 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.218 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.218 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.218 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.218 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.218 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.218 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.218 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.218 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.219 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.219 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.219 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.219 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.219 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.219 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.219 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.219 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.219 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.219 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.219 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.219 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.220 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.220 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.220 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.220 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.220 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.220 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.220 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.220 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.220 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.220 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.220 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.220 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.220 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.220 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.221 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.221 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.221 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.221 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.221 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.221 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.221 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.221 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.221 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.221 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.221 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.221 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.221 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.221 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.222 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.222 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.222 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.222 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.222 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.222 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.222 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.222 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.222 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.222 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.222 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-04-25T01:43:16.223 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.223 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.223 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.223 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-04-25T01:43:16.223 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-04-25T01:43:16.223 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.223 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.223 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.223 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:16.223 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.223 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.223 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.223 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T01:43:16.223 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.224 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.224 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.224 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-04-25T01:43:16.224 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-04-25T01:43:16.224 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.224 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.224 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.224 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:16.224 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.224 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.224 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.224 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T01:43:16.224 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.224 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.225 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.225 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-04-25T01:43:16.225 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-04-25T01:43:16.225 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.225 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.225 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.225 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:16.225 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.225 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.225 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.225 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T01:43:16.225 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.225 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.225 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.226 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-04-25T01:43:16.226 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-04-25T01:43:16.226 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.226 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.226 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.226 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.226 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.226 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.226 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.226 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T01:43:16.226 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.226 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.226 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.227 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-04-25T01:43:16.227 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-04-25T01:43:16.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.227 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.227 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.227 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.227 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.227 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.227 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T01:43:16.227 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.227 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.227 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.228 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-04-25T01:43:16.228 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-04-25T01:43:16.228 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.228 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.228 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.228 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.228 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.228 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.228 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.228 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.228 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.228 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.228 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-04-25T01:43:16.229 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-04-25T01:43:16.229 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.229 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.229 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.229 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.229 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.229 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.229 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.229 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.229 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.229 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.229 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.229 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-04-25T01:43:16.229 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-04-25T01:43:16.229 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.229 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.230 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.230 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T01:43:16.230 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.230 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.230 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.230 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.230 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.230 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.230 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.230 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-25T01:43:16.230 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-25T01:43:16.230 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.230 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.231 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.231 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.231 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.231 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.231 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.231 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.231 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.231 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.231 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.231 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-25T01:43:16.231 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-25T01:43:16.231 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.231 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.231 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.231 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-04-25T01:43:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-04-25T01:43:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-04-25T01:43:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-04-25T01:43:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.232 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-04-25T01:43:16.233 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.233 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.233 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.233 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.233 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.233 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.233 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.233 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-04-25T01:43:16.233 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-04-25T01:43:16.233 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.233 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.233 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.233 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.233 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.234 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.234 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.234 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.234 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.234 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.234 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.234 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-04-25T01:43:16.234 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-04-25T01:43:16.234 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.234 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.234 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.234 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.234 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.235 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.235 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.235 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.235 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.235 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.235 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.235 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.235 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.235 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.235 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.235 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.235 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.235 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.235 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.235 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.236 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.236 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.236 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.236 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.236 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.236 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.236 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.236 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.236 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.236 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.236 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.236 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.236 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.237 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.237 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.237 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.237 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.237 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.237 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.237 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.237 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.237 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.237 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.238 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.238 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.238 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.238 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.238 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.238 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.238 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.238 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.238 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.238 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.238 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.238 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.238 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.239 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.239 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.239 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.239 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.239 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.239 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.239 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.239 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.239 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.240 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.240 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.240 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.240 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.240 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.240 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.240 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.240 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.240 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.241 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-04-25T01:43:16.241 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-04-25T01:43:16.241 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.241 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.241 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.241 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.241 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.241 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.241 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.241 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.241 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.242 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.242 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.242 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-04-25T01:43:16.242 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-04-25T01:43:16.242 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.242 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.242 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.242 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.243 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.243 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.243 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.243 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.243 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.243 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.243 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.243 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-04-25T01:43:16.243 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-04-25T01:43:16.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.244 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.244 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.244 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T01:43:16.244 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.244 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.244 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.244 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.244 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.244 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.245 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.245 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.245 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.245 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-04-25T01:43:16.245 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.245 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.246 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.246 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.246 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.246 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.246 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.246 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.246 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.246 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.246 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.246 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.247 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.247 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.247 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.247 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.247 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.247 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.247 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.247 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.247 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.248 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.248 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.248 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.248 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.248 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.248 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.249 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.249 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.249 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.249 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.249 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.249 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.249 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.249 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.250 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.250 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.250 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.250 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.250 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.250 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.250 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.253 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.253 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.253 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.253 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.253 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.253 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.253 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.253 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.253 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.253 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.254 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.254 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.254 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.254 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.254 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.254 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.254 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.254 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.254 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.254 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.254 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.255 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.255 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.255 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.255 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.255 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.255 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-25T01:43:16.255 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.255 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.255 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.255 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.255 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.255 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.255 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.256 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.256 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.256 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.256 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.256 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.256 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.256 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.256 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.256 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.256 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.256 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.256 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.256 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.256 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.257 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.257 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.257 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.257 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.257 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.257 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.257 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.257 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.257 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.257 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.257 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.257 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.257 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.257 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.258 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.258 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.258 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.258 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.258 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.258 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.258 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.258 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.258 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.258 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.259 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.259 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.259 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.259 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.259 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.259 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.259 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.259 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.259 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.259 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.259 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.259 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.260 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.260 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.260 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.260 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.260 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.260 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.260 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.260 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.260 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.260 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.260 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.260 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-04-25T01:43:16.260 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-04-25T01:43:16.260 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.260 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.261 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.261 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-04-25T01:43:16.261 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.261 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.261 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.261 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T01:43:16.261 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.261 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.261 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.261 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-04-25T01:43:16.261 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-04-25T01:43:16.261 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.261 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.261 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.262 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.262 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.262 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.262 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.262 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-04-25T01:43:16.262 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.262 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.262 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.262 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-04-25T01:43:16.262 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-04-25T01:43:16.262 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.262 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.262 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.263 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-25T01:43:16.263 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.263 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.263 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.263 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T01:43:16.263 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.263 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.263 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.263 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-04-25T01:43:16.263 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T01:43:16.263 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.263 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.264 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.264 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-25T01:43:16.264 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.264 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.264 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.264 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T01:43:16.264 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.264 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.264 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.264 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.264 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.264 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-04-25T01:43:16.264 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-25T01:43:16.265 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-25T01:43:16.265 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-25T01:43:16.265 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-04-25T01:43:16.265 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-04-25T01:43:16.265 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-25T01:43:16.265 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.265 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.265 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-04-25T01:43:16.265 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.265 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.265 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.265 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.265 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.265 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.265 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.266 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-04-25T01:43:16.266 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-04-25T01:43:16.266 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.266 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.266 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.266 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.266 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.266 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.266 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.266 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.266 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.266 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.266 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.267 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-25T01:43:16.267 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-25T01:43:16.267 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-25T01:43:16.267 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.267 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.267 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-25T01:43:16.267 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.267 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.267 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.267 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.267 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.267 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.267 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.267 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-25T01:43:16.268 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-25T01:43:16.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.268 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.268 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.268 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.268 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.268 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.268 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.268 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.268 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.268 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.268 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.268 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.268 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.268 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.268 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.268 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.269 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.269 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.269 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-25T01:43:16.269 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-25T01:43:16.269 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.269 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.269 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.269 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.269 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.269 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.269 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.269 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.269 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.270 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.270 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.270 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-25T01:43:16.270 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-25T01:43:16.270 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.270 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.270 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.270 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-25T01:43:16.270 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.270 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.270 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-25T01:43:16.270 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-25T01:43:16.270 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-25T01:43:16.270 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-25T01:43:16.271 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-25T01:43:16.271 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-25T01:43:16.271 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-25T01:43:16.271 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.271 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.271 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.271 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.271 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.271 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-25T01:43:16.271 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-25T01:43:16.271 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-25T01:43:16.271 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.271 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-25T01:43:16.271 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-25T01:43:16.271 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.272 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.272 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.272 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.272 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.272 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.272 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.272 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-04-25T01:43:16.272 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-04-25T01:43:16.272 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.272 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.272 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.272 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-25T01:43:16.272 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.272 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.272 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.273 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.273 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.273 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.273 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.273 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-04-25T01:43:16.273 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-04-25T01:43:16.273 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-25T01:43:16.273 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.273 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.273 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-04-25T01:43:16.273 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.273 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.273 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.273 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.273 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.274 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.274 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.274 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-04-25T01:43:16.274 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-04-25T01:43:16.274 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-25T01:43:16.274 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-25T01:43:16.274 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-25T01:43:16.274 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T01:43:16.274 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-25T01:43:16.275 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-25T01:43:16.275 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-25T01:43:16.275 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-25T01:43:16.275 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-25T01:43:16.275 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-25T01:43:16.275 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-25T01:43:16.275 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.275 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.275 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.275 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.275 INFO:teuthology.orchestra.run.smithi028.stdout: "services": { 2024-04-25T01:43:16.275 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard": "https://172.21.15.28:8443/" 2024-04-25T01:43:16.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.276 INFO:teuthology.orchestra.run.smithi028.stdout: "always_on_modules": { 2024-04-25T01:43:16.276 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus": [ 2024-04-25T01:43:16.276 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-25T01:43:16.276 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-25T01:43:16.276 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-25T01:43:16.276 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-25T01:43:16.276 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-25T01:43:16.276 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-25T01:43:16.276 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-25T01:43:16.276 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-25T01:43:16.276 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-25T01:43:16.276 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-25T01:43:16.276 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.276 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific": [ 2024-04-25T01:43:16.277 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-25T01:43:16.277 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-25T01:43:16.277 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-25T01:43:16.277 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-25T01:43:16.277 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-25T01:43:16.277 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-25T01:43:16.277 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-25T01:43:16.277 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-25T01:43:16.277 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-25T01:43:16.277 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-25T01:43:16.278 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.278 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy": [ 2024-04-25T01:43:16.278 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-25T01:43:16.278 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-25T01:43:16.278 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-25T01:43:16.278 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-25T01:43:16.278 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-25T01:43:16.278 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-25T01:43:16.278 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-25T01:43:16.278 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-25T01:43:16.278 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-25T01:43:16.278 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-25T01:43:16.278 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.278 INFO:teuthology.orchestra.run.smithi028.stdout: "reef": [ 2024-04-25T01:43:16.279 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-25T01:43:16.279 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-25T01:43:16.279 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-25T01:43:16.279 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-25T01:43:16.279 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-25T01:43:16.279 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-25T01:43:16.279 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-25T01:43:16.279 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-25T01:43:16.280 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-25T01:43:16.280 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-25T01:43:16.280 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-25T01:43:16.280 INFO:teuthology.orchestra.run.smithi028.stdout: "squid": [ 2024-04-25T01:43:16.280 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-25T01:43:16.280 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-25T01:43:16.280 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-25T01:43:16.280 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-25T01:43:16.280 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-25T01:43:16.280 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-25T01:43:16.280 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-25T01:43:16.280 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-25T01:43:16.280 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-25T01:43:16.280 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-25T01:43:16.281 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-25T01:43:16.281 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.281 INFO:teuthology.orchestra.run.smithi028.stdout: "last_failure_osd_epoch": 94, 2024-04-25T01:43:16.281 INFO:teuthology.orchestra.run.smithi028.stdout: "active_clients": [ 2024-04-25T01:43:16.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.281 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-25T01:43:16.281 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-25T01:43:16.281 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.281 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-25T01:43:16.281 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-25T01:43:16.282 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1842037481 2024-04-25T01:43:16.282 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.282 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-25T01:43:16.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "libcephsqlite", 2024-04-25T01:43:16.282 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-25T01:43:16.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.282 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-25T01:43:16.282 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-25T01:43:16.282 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 572790062 2024-04-25T01:43:16.282 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.283 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-25T01:43:16.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.283 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-25T01:43:16.283 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-25T01:43:16.283 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.283 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-25T01:43:16.283 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-25T01:43:16.283 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2696077390 2024-04-25T01:43:16.284 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.284 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-25T01:43:16.284 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.284 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-25T01:43:16.284 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-25T01:43:16.284 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.284 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-25T01:43:16.284 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-25T01:43:16.284 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3011565900 2024-04-25T01:43:16.284 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.285 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-25T01:43:16.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.285 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-25T01:43:16.285 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-25T01:43:16.285 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.285 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-25T01:43:16.285 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-25T01:43:16.285 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 626750849 2024-04-25T01:43:16.285 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.286 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-25T01:43:16.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:16.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.286 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-25T01:43:16.286 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-25T01:43:16.286 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T01:43:16.286 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-25T01:43:16.286 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-25T01:43:16.286 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3294403401 2024-04-25T01:43:16.286 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.286 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-25T01:43:16.286 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:16.286 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-25T01:43:16.286 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:43:16.293 DEBUG:tasks.check_counter:Getting stats from a 2024-04-25T01:43:16.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-25T01:43:16.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:15 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/272395102' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:43:16.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:15 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:43:16.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:15 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3040981124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:43:16.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:15 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/987333514' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T01:43:16.295 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:15 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/272395102' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:43:16.295 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:15 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:43:16.295 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:15 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3040981124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:43:16.295 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:15 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/987333514' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T01:43:16.295 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:15 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/272395102' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:43:16.295 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:15 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:43:16.295 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:15 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3040981124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:43:16.295 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:15 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/987333514' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T01:43:16.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:16 smithi028 ceph-mon[29390]: pgmap v317: 129 pgs: 129 active+clean; 11 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-04-25T01:43:16.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:16 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:16.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:16 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:16.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:16 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:16.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:16 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:16.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:16 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:16.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:16 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:16.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:16 smithi130 ceph-mon[32702]: pgmap v317: 129 pgs: 129 active+clean; 11 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-04-25T01:43:16.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:16 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:16.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:16 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:16.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:16 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:16.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:16 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:16.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:16 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:16.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:16 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:16.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:16 smithi032 ceph-mon[32461]: pgmap v317: 129 pgs: 129 active+clean; 11 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-04-25T01:43:16.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:16 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:16.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:16 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:16.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:16 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:16.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:16 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:16.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:16 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:16.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:16 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:16.579 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:43:17.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:17 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:17 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:17 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:17 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:17 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:17 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:17 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:17 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:17 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:17 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:17 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:17 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:17 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:43:17.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:17 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:43:17.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:17 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:17.326 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:17 smithi028 ceph-mon[29390]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T01:43:17.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:17 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:17 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:17 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:17 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:17 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:17 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:17 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:17 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:17 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:17 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:17 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:17 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:17 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:43:17.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:17 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:43:17.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:17 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:17.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:17 smithi130 ceph-mon[32702]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T01:43:17.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:17 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:17 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:17 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:17 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:17 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:17 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:17 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:17 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:17 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:17 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:17 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:17 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:43:17.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:17 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:43:17.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:17 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:43:17.498 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:17 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' 2024-04-25T01:43:17.498 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:17 smithi032 ceph-mon[32461]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T01:43:18.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:18 smithi028 ceph-mon[29390]: Adjusting osd_memory_target on smithi130 to 176.2M 2024-04-25T01:43:18.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:18 smithi028 ceph-mon[29390]: Unable to set osd_memory_target on smithi130 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-25T01:43:18.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:18 smithi028 ceph-mon[29390]: Adjusting osd_memory_target on smithi032 to 177.2M 2024-04-25T01:43:18.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:18 smithi028 ceph-mon[29390]: Unable to set osd_memory_target on smithi032 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-25T01:43:18.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:18 smithi028 ceph-mon[29390]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-04-25T01:43:18.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:18 smithi028 ceph-mon[29390]: Unable to set osd_memory_target on smithi028 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-25T01:43:18.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:18 smithi028 ceph-mon[29390]: pgmap v318: 129 pgs: 129 active+clean; 11 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-04-25T01:43:18.410 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:43:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:43:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1476, 2024-04-25T01:43:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1478, 2024-04-25T01:43:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 442575, 2024-04-25T01:43:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 469369, 2024-04-25T01:43:18.410 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 6, 2024-04-25T01:43:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 2, 2024-04-25T01:43:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.156442430, 2024-04-25T01:43:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.051369402, 2024-04-25T01:43:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.040549387, 2024-04-25T01:43:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.049327038, 2024-04-25T01:43:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1476, 2024-04-25T01:43:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.063370222, 2024-04-25T01:43:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000042933 2024-04-25T01:43:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.412 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:43:18.412 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:43:18.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.412 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:43:18.412 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 666, 2024-04-25T01:43:18.412 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 821, 2024-04-25T01:43:18.412 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 166324, 2024-04-25T01:43:18.412 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 648156, 2024-04-25T01:43:18.412 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 7, 2024-04-25T01:43:18.412 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 5, 2024-04-25T01:43:18.412 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.084047191, 2024-04-25T01:43:18.412 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.028983650, 2024-04-25T01:43:18.412 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.018955492, 2024-04-25T01:43:18.412 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.027616251, 2024-04-25T01:43:18.412 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:18.413 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 819, 2024-04-25T01:43:18.413 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.046361635, 2024-04-25T01:43:18.413 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000056607 2024-04-25T01:43:18.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.413 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:18.413 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.413 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.413 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.413 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-25T01:43:18.413 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 438208 2024-04-25T01:43:18.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.413 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:43:18.413 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 237, 2024-04-25T01:43:18.413 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 226, 2024-04-25T01:43:18.414 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1474620, 2024-04-25T01:43:18.414 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 131150, 2024-04-25T01:43:18.414 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 3, 2024-04-25T01:43:18.414 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 2, 2024-04-25T01:43:18.414 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.043047074, 2024-04-25T01:43:18.414 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.014119779, 2024-04-25T01:43:18.414 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.032780615, 2024-04-25T01:43:18.414 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.004459135, 2024-04-25T01:43:18.414 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:18.414 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 225, 2024-04-25T01:43:18.414 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.011856395, 2024-04-25T01:43:18.414 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000052695 2024-04-25T01:43:18.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.414 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:18.415 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.415 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.415 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.415 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1474352, 2024-04-25T01:43:18.415 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 130832 2024-04-25T01:43:18.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.415 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-25T01:43:18.415 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-04-25T01:43:18.415 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-25T01:43:18.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.415 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-04-25T01:43:18.415 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-04-25T01:43:18.415 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-04-25T01:43:18.416 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3385, 2024-04-25T01:43:18.416 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.111784327, 2024-04-25T01:43:18.416 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000033023 2024-04-25T01:43:18.416 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:18.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.416 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-04-25T01:43:18.416 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-04-25T01:43:18.416 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-04-25T01:43:18.416 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.416 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.416 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.416 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:18.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.416 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-04-25T01:43:18.416 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 0, 2024-04-25T01:43:18.416 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 0, 2024-04-25T01:43:18.416 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-04-25T01:43:18.417 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.417 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.417 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.417 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-04-25T01:43:18.417 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 0, 2024-04-25T01:43:18.417 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-04-25T01:43:18.417 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-04-25T01:43:18.417 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 0, 2024-04-25T01:43:18.417 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 0, 2024-04-25T01:43:18.417 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-04-25T01:43:18.417 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 0, 2024-04-25T01:43:18.417 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 0, 2024-04-25T01:43:18.417 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 0, 2024-04-25T01:43:18.417 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 0, 2024-04-25T01:43:18.418 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 0, 2024-04-25T01:43:18.418 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 0, 2024-04-25T01:43:18.418 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 0, 2024-04-25T01:43:18.418 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 0, 2024-04-25T01:43:18.418 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 1, 2024-04-25T01:43:18.418 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 0, 2024-04-25T01:43:18.418 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 0, 2024-04-25T01:43:18.418 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 0, 2024-04-25T01:43:18.418 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 0, 2024-04-25T01:43:18.418 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:43:18.418 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-04-25T01:43:18.418 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 0, 2024-04-25T01:43:18.418 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-04-25T01:43:18.418 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-04-25T01:43:18.418 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 0, 2024-04-25T01:43:18.419 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 0, 2024-04-25T01:43:18.419 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 0, 2024-04-25T01:43:18.419 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 0, 2024-04-25T01:43:18.419 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-04-25T01:43:18.419 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 0, 2024-04-25T01:43:18.419 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-04-25T01:43:18.419 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 0, 2024-04-25T01:43:18.419 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 0, 2024-04-25T01:43:18.419 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 0, 2024-04-25T01:43:18.419 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T01:43:18.419 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-04-25T01:43:18.419 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:43:18.419 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:43:18.419 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:43:18.419 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:43:18.419 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:43:18.420 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 0, 2024-04-25T01:43:18.420 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:43:18.420 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T01:43:18.420 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 0, 2024-04-25T01:43:18.420 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:43:18.420 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:43:18.420 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:43:18.420 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 0, 2024-04-25T01:43:18.420 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 0, 2024-04-25T01:43:18.420 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 0, 2024-04-25T01:43:18.420 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 0 2024-04-25T01:43:18.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.420 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-04-25T01:43:18.420 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 0, 2024-04-25T01:43:18.420 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 0, 2024-04-25T01:43:18.420 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 0, 2024-04-25T01:43:18.421 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 0, 2024-04-25T01:43:18.421 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 0, 2024-04-25T01:43:18.421 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-04-25T01:43:18.421 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-04-25T01:43:18.421 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:43:18.421 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 0, 2024-04-25T01:43:18.421 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 0, 2024-04-25T01:43:18.421 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 0, 2024-04-25T01:43:18.421 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-04-25T01:43:18.421 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-04-25T01:43:18.421 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:43:18.421 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:43:18.421 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-04-25T01:43:18.421 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-04-25T01:43:18.421 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:43:18.421 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:43:18.421 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:43:18.422 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 0, 2024-04-25T01:43:18.422 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-04-25T01:43:18.422 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:43:18.422 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-04-25T01:43:18.422 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0 2024-04-25T01:43:18.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.422 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-04-25T01:43:18.422 INFO:teuthology.orchestra.run.smithi028.stdout: "evlrg": 0, 2024-04-25T01:43:18.422 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 0, 2024-04-25T01:43:18.422 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 0, 2024-04-25T01:43:18.422 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 0, 2024-04-25T01:43:18.422 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 5, 2024-04-25T01:43:18.422 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-04-25T01:43:18.422 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 0, 2024-04-25T01:43:18.422 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 0, 2024-04-25T01:43:18.422 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 0, 2024-04-25T01:43:18.423 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 0, 2024-04-25T01:43:18.423 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 2, 2024-04-25T01:43:18.423 INFO:teuthology.orchestra.run.smithi028.stdout: "segmjr": 2, 2024-04-25T01:43:18.423 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-04-25T01:43:18.423 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 0, 2024-04-25T01:43:18.423 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 4194304, 2024-04-25T01:43:18.423 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 4195732, 2024-04-25T01:43:18.423 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 4195424, 2024-04-25T01:43:18.423 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-04-25T01:43:18.423 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.423 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.423 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.423 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 5 2024-04-25T01:43:18.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.424 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-04-25T01:43:18.424 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 3, 2024-04-25T01:43:18.424 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 3, 2024-04-25T01:43:18.424 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 0, 2024-04-25T01:43:18.424 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 1, 2024-04-25T01:43:18.424 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 1, 2024-04-25T01:43:18.424 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 0, 2024-04-25T01:43:18.424 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 0, 2024-04-25T01:43:18.424 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 0, 2024-04-25T01:43:18.424 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 0, 2024-04-25T01:43:18.424 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 0, 2024-04-25T01:43:18.424 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 0, 2024-04-25T01:43:18.424 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 0, 2024-04-25T01:43:18.424 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 52804, 2024-04-25T01:43:18.424 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 215420 2024-04-25T01:43:18.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.425 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-04-25T01:43:18.425 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 0, 2024-04-25T01:43:18.425 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 0, 2024-04-25T01:43:18.425 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 0, 2024-04-25T01:43:18.425 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 0, 2024-04-25T01:43:18.425 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 0, 2024-04-25T01:43:18.425 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-04-25T01:43:18.425 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.425 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.425 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.425 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-04-25T01:43:18.425 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.425 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.425 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.426 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-04-25T01:43:18.426 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.426 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.426 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.426 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-04-25T01:43:18.426 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.426 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.426 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.426 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-04-25T01:43:18.426 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.426 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.426 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.427 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-04-25T01:43:18.427 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.427 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.427 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.427 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-04-25T01:43:18.427 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.427 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.427 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.427 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-04-25T01:43:18.427 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.427 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.427 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.428 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-04-25T01:43:18.428 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.428 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.428 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.428 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-04-25T01:43:18.428 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.428 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.428 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.429 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-04-25T01:43:18.429 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.429 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.429 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.429 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-04-25T01:43:18.429 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.430 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.430 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.430 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-04-25T01:43:18.430 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.430 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.430 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.430 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-04-25T01:43:18.430 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.430 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.430 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.431 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-04-25T01:43:18.431 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.431 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.431 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.431 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-04-25T01:43:18.431 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.431 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.431 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.431 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-04-25T01:43:18.431 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.431 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.431 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.432 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-04-25T01:43:18.432 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.432 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.432 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.432 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-04-25T01:43:18.432 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.432 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.432 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.432 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-04-25T01:43:18.432 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.432 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.432 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.433 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-04-25T01:43:18.433 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.433 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.433 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.433 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-04-25T01:43:18.433 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.433 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.433 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.433 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-04-25T01:43:18.433 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.433 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.434 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.434 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-04-25T01:43:18.434 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.434 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.434 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.434 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-04-25T01:43:18.434 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.434 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.434 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.434 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-04-25T01:43:18.434 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.435 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.435 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.435 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-04-25T01:43:18.435 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.435 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.435 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.435 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-04-25T01:43:18.435 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.435 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.435 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.435 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-04-25T01:43:18.435 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.436 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.436 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.436 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:43:18.436 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:43:18.436 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-04-25T01:43:18.436 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.436 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.436 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.436 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:18.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.436 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-04-25T01:43:18.436 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 1, 2024-04-25T01:43:18.436 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 2, 2024-04-25T01:43:18.436 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 1, 2024-04-25T01:43:18.436 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 1, 2024-04-25T01:43:18.437 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-04-25T01:43:18.437 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 0, 2024-04-25T01:43:18.437 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 0, 2024-04-25T01:43:18.437 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 1801, 2024-04-25T01:43:18.437 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-04-25T01:43:18.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.437 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-25T01:43:18.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:43:18.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-25T01:43:18.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:43:18.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:43:18.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:43:18.437 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:43:18.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:43:18.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:43:18.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:43:18.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:43:18.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:43:18.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:43:18.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:43:18.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:43:18.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:43:18.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-25T01:43:18.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:43:18.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-25T01:43:18.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:43:18.438 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:43:18.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:43:18.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:43:18.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:43:18.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:43:18.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:43:18.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-25T01:43:18.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:43:18.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:43:18.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:43:18.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-25T01:43:18.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-25T01:43:18.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-25T01:43:18.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:43:18.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:43:18.439 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:43:18.440 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:43:18.440 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 189371, 2024-04-25T01:43:18.440 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 51, 2024-04-25T01:43:18.440 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:43:18.440 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-04-25T01:43:18.440 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-25T01:43:18.440 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-25T01:43:18.440 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:43:18.440 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-25T01:43:18.440 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:43:18.440 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-25T01:43:18.440 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 9560, 2024-04-25T01:43:18.440 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 184, 2024-04-25T01:43:18.440 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:43:18.441 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:43:18.441 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-25T01:43:18.441 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-25T01:43:18.441 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 8464, 2024-04-25T01:43:18.441 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 13, 2024-04-25T01:43:18.441 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-25T01:43:18.441 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-25T01:43:18.441 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-25T01:43:18.441 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-25T01:43:18.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.441 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-04-25T01:43:18.441 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-04-25T01:43:18.441 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-04-25T01:43:18.441 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 2137, 2024-04-25T01:43:18.441 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 21, 2024-04-25T01:43:18.442 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 3, 2024-04-25T01:43:18.442 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 2137, 2024-04-25T01:43:18.442 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-04-25T01:43:18.442 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2137, 2024-04-25T01:43:18.442 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 1.378961064, 2024-04-25T01:43:18.442 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000645278 2024-04-25T01:43:18.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.442 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-04-25T01:43:18.442 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-04-25T01:43:18.442 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2137, 2024-04-25T01:43:18.442 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 2138 2024-04-25T01:43:18.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.442 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 2137, 2024-04-25T01:43:18.442 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 2137, 2024-04-25T01:43:18.443 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 0, 2024-04-25T01:43:18.443 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-04-25T01:43:18.443 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-04-25T01:43:18.443 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 1393, 2024-04-25T01:43:18.443 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 0, 2024-04-25T01:43:18.443 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 742, 2024-04-25T01:43:18.443 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 0, 2024-04-25T01:43:18.443 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 0, 2024-04-25T01:43:18.443 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-04-25T01:43:18.443 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-04-25T01:43:18.443 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 0, 2024-04-25T01:43:18.443 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 0, 2024-04-25T01:43:18.443 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 0, 2024-04-25T01:43:18.443 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-04-25T01:43:18.444 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-04-25T01:43:18.444 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-04-25T01:43:18.444 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 1, 2024-04-25T01:43:18.444 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 0, 2024-04-25T01:43:18.444 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:43:18.444 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-04-25T01:43:18.444 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:43:18.444 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-04-25T01:43:18.444 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-04-25T01:43:18.444 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-04-25T01:43:18.444 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:43:18.444 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-04-25T01:43:18.444 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:43:18.444 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-04-25T01:43:18.444 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-04-25T01:43:18.445 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-04-25T01:43:18.445 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-04-25T01:43:18.445 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-04-25T01:43:18.445 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-04-25T01:43:18.445 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-04-25T01:43:18.445 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-04-25T01:43:18.445 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-04-25T01:43:18.445 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-04-25T01:43:18.445 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-04-25T01:43:18.445 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-04-25T01:43:18.445 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-04-25T01:43:18.445 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-04-25T01:43:18.445 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-04-25T01:43:18.445 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-04-25T01:43:18.445 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-04-25T01:43:18.446 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 102, 2024-04-25T01:43:18.446 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-04-25T01:43:18.446 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 11, 2024-04-25T01:43:18.446 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 5, 2024-04-25T01:43:18.446 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 5, 2024-04-25T01:43:18.446 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 0, 2024-04-25T01:43:18.446 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-04-25T01:43:18.446 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 0, 2024-04-25T01:43:18.446 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 2, 2024-04-25T01:43:18.446 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 0 2024-04-25T01:43:18.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.446 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-04-25T01:43:18.446 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 0, 2024-04-25T01:43:18.446 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:43:18.447 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-04-25T01:43:18.447 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-04-25T01:43:18.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.448 INFO:teuthology.orchestra.run.smithi028.stdout: "osd-slow-ops": { 2024-04-25T01:43:18.448 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 0 2024-04-25T01:43:18.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.448 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-04-25T01:43:18.448 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-04-25T01:43:18.448 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T01:43:18.448 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-04-25T01:43:18.448 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 0, 2024-04-25T01:43:18.448 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 0, 2024-04-25T01:43:18.448 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 0, 2024-04-25T01:43:18.448 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-04-25T01:43:18.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.449 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:43:18.449 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-25T01:43:18.449 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-25T01:43:18.449 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-25T01:43:18.449 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 2375, 2024-04-25T01:43:18.449 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1854663, 2024-04-25T01:43:18.449 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:18.449 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 2375, 2024-04-25T01:43:18.449 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-25T01:43:18.449 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-25T01:43:18.449 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 2375, 2024-04-25T01:43:18.449 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1854663, 2024-04-25T01:43:18.449 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-25T01:43:18.449 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.449 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.450 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.450 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:18.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.450 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-04-25T01:43:18.450 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-25T01:43:18.450 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-25T01:43:18.450 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-25T01:43:18.450 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-25T01:43:18.450 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-25T01:43:18.450 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:18.450 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:18.450 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 2137, 2024-04-25T01:43:18.450 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 1433, 2024-04-25T01:43:18.450 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 2137, 2024-04-25T01:43:18.450 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1433, 2024-04-25T01:43:18.451 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-25T01:43:18.451 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.451 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.451 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.451 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:18.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.451 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-04-25T01:43:18.451 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-25T01:43:18.451 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-04-25T01:43:18.451 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-25T01:43:18.451 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-25T01:43:18.451 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-25T01:43:18.451 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:18.451 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:18.452 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 2137, 2024-04-25T01:43:18.452 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 2137, 2024-04-25T01:43:18.452 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 2137, 2024-04-25T01:43:18.452 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 2137, 2024-04-25T01:43:18.452 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-25T01:43:18.452 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.452 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.452 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.452 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:18.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.452 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:43:18.452 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-25T01:43:18.452 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-04-25T01:43:18.452 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-25T01:43:18.452 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-25T01:43:18.452 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-25T01:43:18.453 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:18.453 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:18.453 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-25T01:43:18.453 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-25T01:43:18.453 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-04-25T01:43:18.453 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-04-25T01:43:18.453 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-25T01:43:18.453 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.453 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.453 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.453 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:18.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:18.453 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x55b7d21246e0": { 2024-04-25T01:43:18.453 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-25T01:43:18.453 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-04-25T01:43:18.454 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-25T01:43:18.454 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-25T01:43:18.454 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-25T01:43:18.454 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:18.454 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:18.454 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-25T01:43:18.454 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-25T01:43:18.454 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-04-25T01:43:18.454 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-04-25T01:43:18.454 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-25T01:43:18.454 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:18.454 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:18.454 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:18.454 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:18.454 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:18.455 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:43:18.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:18 smithi130 ceph-mon[32702]: Adjusting osd_memory_target on smithi130 to 176.2M 2024-04-25T01:43:18.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:18 smithi130 ceph-mon[32702]: Unable to set osd_memory_target on smithi130 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-25T01:43:18.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:18 smithi130 ceph-mon[32702]: Adjusting osd_memory_target on smithi032 to 177.2M 2024-04-25T01:43:18.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:18 smithi130 ceph-mon[32702]: Unable to set osd_memory_target on smithi032 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-25T01:43:18.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:18 smithi130 ceph-mon[32702]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-04-25T01:43:18.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:18 smithi130 ceph-mon[32702]: Unable to set osd_memory_target on smithi028 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-25T01:43:18.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:18 smithi130 ceph-mon[32702]: pgmap v318: 129 pgs: 129 active+clean; 11 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-04-25T01:43:18.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:18 smithi032 ceph-mon[32461]: Adjusting osd_memory_target on smithi130 to 176.2M 2024-04-25T01:43:18.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:18 smithi032 ceph-mon[32461]: Unable to set osd_memory_target on smithi130 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-25T01:43:18.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:18 smithi032 ceph-mon[32461]: Adjusting osd_memory_target on smithi032 to 177.2M 2024-04-25T01:43:18.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:18 smithi032 ceph-mon[32461]: Unable to set osd_memory_target on smithi032 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-25T01:43:18.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:18 smithi032 ceph-mon[32461]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-04-25T01:43:18.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:18 smithi032 ceph-mon[32461]: Unable to set osd_memory_target on smithi028 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-25T01:43:18.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:18 smithi032 ceph-mon[32461]: pgmap v318: 129 pgs: 129 active+clean; 11 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-04-25T01:43:19.082 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-04-25T01:43:19.082 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-04-25T01:43:19.082 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-04-25T01:43:19.082 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-04-25T01:43:19.082 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T01:43:19.083 DEBUG:tasks.check_counter:Getting stats from d 2024-04-25T01:43:19.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-25T01:43:19.379 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:43:20.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:20 smithi028 ceph-mon[29390]: pgmap v320: 129 pgs: 129 active+clean; 11 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 209 B/s wr, 13 op/s 2024-04-25T01:43:20.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:20 smithi130 ceph-mon[32702]: pgmap v320: 129 pgs: 129 active+clean; 11 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 209 B/s wr, 13 op/s 2024-04-25T01:43:20.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:20 smithi032 ceph-mon[32461]: pgmap v320: 129 pgs: 129 active+clean; 11 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 209 B/s wr, 13 op/s 2024-04-25T01:43:21.154 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:43:21.154 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:43:21.155 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 4, 2024-04-25T01:43:21.155 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 4, 2024-04-25T01:43:21.155 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 2992, 2024-04-25T01:43:21.155 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 448, 2024-04-25T01:43:21.155 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-04-25T01:43:21.155 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-04-25T01:43:21.155 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.001702591, 2024-04-25T01:43:21.155 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000042036, 2024-04-25T01:43:21.155 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.000682795, 2024-04-25T01:43:21.155 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:43:21.155 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:21.155 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3, 2024-04-25T01:43:21.155 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000871023, 2024-04-25T01:43:21.155 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000290341 2024-04-25T01:43:21.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:21.156 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:21.156 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:21.156 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:21.156 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:21.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:21.156 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-25T01:43:21.156 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T01:43:21.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:21.156 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:43:21.156 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 2, 2024-04-25T01:43:21.156 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 163, 2024-04-25T01:43:21.156 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 320, 2024-04-25T01:43:21.156 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 183968, 2024-04-25T01:43:21.156 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-04-25T01:43:21.157 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-04-25T01:43:21.157 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.016377059, 2024-04-25T01:43:21.157 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.010111609, 2024-04-25T01:43:21.157 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.001688814, 2024-04-25T01:43:21.157 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:43:21.157 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:21.157 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 160, 2024-04-25T01:43:21.157 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.020744323, 2024-04-25T01:43:21.157 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000129652 2024-04-25T01:43:21.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:21.157 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:21.157 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:21.157 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:21.157 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:21.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:21.157 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T01:43:21.158 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 183968 2024-04-25T01:43:21.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:21.158 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:43:21.158 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 225, 2024-04-25T01:43:21.158 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 215, 2024-04-25T01:43:21.158 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1443920, 2024-04-25T01:43:21.158 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 131904, 2024-04-25T01:43:21.158 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-04-25T01:43:21.158 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-04-25T01:43:21.158 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.044725371, 2024-04-25T01:43:21.158 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.014868609, 2024-04-25T01:43:21.158 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.032714240, 2024-04-25T01:43:21.158 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.004499017, 2024-04-25T01:43:21.158 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:21.159 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 214, 2024-04-25T01:43:21.159 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.010650484, 2024-04-25T01:43:21.159 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000049768 2024-04-25T01:43:21.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:21.159 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:21.159 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:21.159 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:21.159 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:21.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:21.159 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1443920, 2024-04-25T01:43:21.159 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 131904 2024-04-25T01:43:21.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:21.159 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-25T01:43:21.160 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-04-25T01:43:21.160 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-25T01:43:21.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:21.160 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-25T01:43:21.160 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:43:21.160 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-25T01:43:21.160 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:43:21.160 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:43:21.160 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:43:21.160 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:43:21.160 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:43:21.160 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:43:21.160 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:43:21.160 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:43:21.161 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:43:21.161 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:43:21.161 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:43:21.161 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:43:21.161 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:43:21.161 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-25T01:43:21.161 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:43:21.161 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-25T01:43:21.161 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:43:21.161 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:43:21.161 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:43:21.161 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:43:21.161 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:43:21.161 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:43:21.161 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:43:21.161 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-25T01:43:21.162 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:43:21.162 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:43:21.162 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:43:21.162 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-25T01:43:21.162 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-25T01:43:21.162 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-25T01:43:21.162 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:43:21.162 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:43:21.162 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:43:21.162 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:43:21.162 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 148977, 2024-04-25T01:43:21.162 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 30, 2024-04-25T01:43:21.162 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:43:21.163 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-04-25T01:43:21.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-25T01:43:21.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-25T01:43:21.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:43:21.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-25T01:43:21.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:43:21.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-25T01:43:21.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-04-25T01:43:21.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-04-25T01:43:21.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:43:21.163 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:43:21.163 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-25T01:43:21.164 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-25T01:43:21.164 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-04-25T01:43:21.164 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-04-25T01:43:21.164 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-25T01:43:21.164 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-25T01:43:21.164 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-25T01:43:21.164 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-25T01:43:21.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:21.164 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:43:21.164 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-25T01:43:21.164 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-25T01:43:21.164 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-25T01:43:21.164 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 227, 2024-04-25T01:43:21.164 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1412190, 2024-04-25T01:43:21.164 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:21.165 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 227, 2024-04-25T01:43:21.165 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-25T01:43:21.165 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-25T01:43:21.165 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 227, 2024-04-25T01:43:21.165 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1412190, 2024-04-25T01:43:21.165 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-25T01:43:21.166 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:21.166 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:21.166 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:21.166 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:21.166 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:21.166 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:43:21.839 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-25T01:43:21.839 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-25T01:43:21.839 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-25T01:43:21.839 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-25T01:43:21.839 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-25T01:43:21.839 DEBUG:tasks.check_counter:Getting stats from g 2024-04-25T01:43:21.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-25T01:43:22.094 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:43:22.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:22 smithi028 ceph-mon[29390]: pgmap v321: 129 pgs: 129 active+clean; 11 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-04-25T01:43:22.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:22 smithi130 ceph-mon[32702]: pgmap v321: 129 pgs: 129 active+clean; 11 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-04-25T01:43:22.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:22 smithi032 ceph-mon[32461]: pgmap v321: 129 pgs: 129 active+clean; 11 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-04-25T01:43:23.952 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:43:23.952 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:43:23.953 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 976, 2024-04-25T01:43:23.953 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 965, 2024-04-25T01:43:23.953 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1722864, 2024-04-25T01:43:23.953 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 366796, 2024-04-25T01:43:23.953 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-04-25T01:43:23.953 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-04-25T01:43:23.953 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.131261642, 2024-04-25T01:43:23.953 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.051168736, 2024-04-25T01:43:23.953 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.051886765, 2024-04-25T01:43:23.953 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.027226437, 2024-04-25T01:43:23.953 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:23.953 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 964, 2024-04-25T01:43:23.953 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.049351992, 2024-04-25T01:43:23.953 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000051195 2024-04-25T01:43:23.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.954 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:23.954 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.954 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.954 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.954 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1474896, 2024-04-25T01:43:23.954 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 132112 2024-04-25T01:43:23.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.954 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:43:23.954 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 1302, 2024-04-25T01:43:23.954 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 1461, 2024-04-25T01:43:23.954 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 332971, 2024-04-25T01:43:23.954 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 854801, 2024-04-25T01:43:23.955 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 7, 2024-04-25T01:43:23.955 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 5, 2024-04-25T01:43:23.955 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.105090845, 2024-04-25T01:43:23.955 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.034656479, 2024-04-25T01:43:23.955 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.026028690, 2024-04-25T01:43:23.955 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.035300754, 2024-04-25T01:43:23.955 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:23.955 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1458, 2024-04-25T01:43:23.955 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.061632570, 2024-04-25T01:43:23.955 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000042271 2024-04-25T01:43:23.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.955 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:23.955 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.955 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.956 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.956 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T01:43:23.956 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 441344 2024-04-25T01:43:23.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.956 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:43:23.956 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 8, 2024-04-25T01:43:23.956 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 8, 2024-04-25T01:43:23.956 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 4621, 2024-04-25T01:43:23.956 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 1732, 2024-04-25T01:43:23.956 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-04-25T01:43:23.956 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 4, 2024-04-25T01:43:23.956 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.001589325, 2024-04-25T01:43:23.956 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000111264, 2024-04-25T01:43:23.956 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.001355997, 2024-04-25T01:43:23.957 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000082530, 2024-04-25T01:43:23.957 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:23.957 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 7, 2024-04-25T01:43:23.957 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.002951565, 2024-04-25T01:43:23.957 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000421652 2024-04-25T01:43:23.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.957 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:23.957 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.957 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.957 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.957 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-25T01:43:23.957 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T01:43:23.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.957 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-25T01:43:23.958 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-04-25T01:43:23.958 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-25T01:43:23.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.958 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-04-25T01:43:23.958 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-04-25T01:43:23.958 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-04-25T01:43:23.958 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3503, 2024-04-25T01:43:23.958 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.090973188, 2024-04-25T01:43:23.958 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000025970 2024-04-25T01:43:23.958 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:23.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.958 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-04-25T01:43:23.958 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-04-25T01:43:23.958 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-04-25T01:43:23.959 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.959 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.959 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.959 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:23.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.959 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-04-25T01:43:23.959 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 0, 2024-04-25T01:43:23.959 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 0, 2024-04-25T01:43:23.959 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-04-25T01:43:23.959 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.959 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.959 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.959 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-04-25T01:43:23.959 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 0, 2024-04-25T01:43:23.960 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-04-25T01:43:23.960 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-04-25T01:43:23.960 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 0, 2024-04-25T01:43:23.960 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 0, 2024-04-25T01:43:23.960 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-04-25T01:43:23.960 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 0, 2024-04-25T01:43:23.960 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 0, 2024-04-25T01:43:23.960 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 0, 2024-04-25T01:43:23.960 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 0, 2024-04-25T01:43:23.960 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 0, 2024-04-25T01:43:23.960 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 0, 2024-04-25T01:43:23.960 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 0, 2024-04-25T01:43:23.960 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 0, 2024-04-25T01:43:23.960 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 1, 2024-04-25T01:43:23.960 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 0, 2024-04-25T01:43:23.961 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 0, 2024-04-25T01:43:23.961 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 0, 2024-04-25T01:43:23.961 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 0, 2024-04-25T01:43:23.961 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:43:23.961 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-04-25T01:43:23.961 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 0, 2024-04-25T01:43:23.961 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-04-25T01:43:23.961 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-04-25T01:43:23.961 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 0, 2024-04-25T01:43:23.961 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 0, 2024-04-25T01:43:23.961 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 0, 2024-04-25T01:43:23.962 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 0, 2024-04-25T01:43:23.962 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-04-25T01:43:23.962 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 0, 2024-04-25T01:43:23.962 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-04-25T01:43:23.962 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 0, 2024-04-25T01:43:23.962 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 0, 2024-04-25T01:43:23.962 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 0, 2024-04-25T01:43:23.962 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T01:43:23.962 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-04-25T01:43:23.962 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:43:23.962 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:43:23.962 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:43:23.962 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:43:23.962 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:43:23.963 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 0, 2024-04-25T01:43:23.963 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:43:23.963 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T01:43:23.963 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 0, 2024-04-25T01:43:23.963 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:43:23.963 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:43:23.963 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:43:23.963 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 0, 2024-04-25T01:43:23.963 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 0, 2024-04-25T01:43:23.963 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 0, 2024-04-25T01:43:23.963 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 0 2024-04-25T01:43:23.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.963 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-04-25T01:43:23.963 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 0, 2024-04-25T01:43:23.963 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 0, 2024-04-25T01:43:23.964 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 0, 2024-04-25T01:43:23.964 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 0, 2024-04-25T01:43:23.964 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 0, 2024-04-25T01:43:23.964 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-04-25T01:43:23.964 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-04-25T01:43:23.964 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:43:23.964 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 0, 2024-04-25T01:43:23.964 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 0, 2024-04-25T01:43:23.964 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 0, 2024-04-25T01:43:23.964 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-04-25T01:43:23.964 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-04-25T01:43:23.964 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:43:23.964 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:43:23.964 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-04-25T01:43:23.964 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-04-25T01:43:23.965 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:43:23.965 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:43:23.965 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:43:23.965 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 0, 2024-04-25T01:43:23.965 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-04-25T01:43:23.965 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:43:23.965 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-04-25T01:43:23.965 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0 2024-04-25T01:43:23.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.965 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-04-25T01:43:23.965 INFO:teuthology.orchestra.run.smithi028.stdout: "evlrg": 0, 2024-04-25T01:43:23.965 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 0, 2024-04-25T01:43:23.965 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 0, 2024-04-25T01:43:23.965 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 0, 2024-04-25T01:43:23.965 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 3, 2024-04-25T01:43:23.966 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-04-25T01:43:23.966 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 0, 2024-04-25T01:43:23.966 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 0, 2024-04-25T01:43:23.966 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 0, 2024-04-25T01:43:23.966 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 0, 2024-04-25T01:43:23.966 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 2, 2024-04-25T01:43:23.966 INFO:teuthology.orchestra.run.smithi028.stdout: "segmjr": 2, 2024-04-25T01:43:23.966 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-04-25T01:43:23.966 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 0, 2024-04-25T01:43:23.966 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 4194304, 2024-04-25T01:43:23.967 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 4195283, 2024-04-25T01:43:23.967 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 4194892, 2024-04-25T01:43:23.967 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-04-25T01:43:23.967 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.967 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.967 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.967 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 3 2024-04-25T01:43:23.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-04-25T01:43:23.967 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 3, 2024-04-25T01:43:23.967 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 3, 2024-04-25T01:43:23.967 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 0, 2024-04-25T01:43:23.967 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 1, 2024-04-25T01:43:23.967 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 1, 2024-04-25T01:43:23.968 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 0, 2024-04-25T01:43:23.968 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 0, 2024-04-25T01:43:23.968 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 0, 2024-04-25T01:43:23.968 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 0, 2024-04-25T01:43:23.968 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 0, 2024-04-25T01:43:23.968 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 0, 2024-04-25T01:43:23.968 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 0, 2024-04-25T01:43:23.968 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 52912, 2024-04-25T01:43:23.968 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 215420 2024-04-25T01:43:23.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.968 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-04-25T01:43:23.968 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 0, 2024-04-25T01:43:23.968 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 0, 2024-04-25T01:43:23.968 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 0, 2024-04-25T01:43:23.968 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 0, 2024-04-25T01:43:23.969 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 0, 2024-04-25T01:43:23.969 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-04-25T01:43:23.969 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.969 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.969 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.969 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-04-25T01:43:23.969 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.969 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.969 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.969 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-04-25T01:43:23.969 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.969 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.969 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.970 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-04-25T01:43:23.970 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.970 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.970 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.970 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-04-25T01:43:23.970 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.970 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.970 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.970 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-04-25T01:43:23.970 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.970 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.971 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.971 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-04-25T01:43:23.971 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.971 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.971 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.971 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-04-25T01:43:23.971 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.971 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.971 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.971 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-04-25T01:43:23.971 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.971 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.972 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.972 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-04-25T01:43:23.972 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.972 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.972 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.972 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-04-25T01:43:23.972 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.972 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.972 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.972 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-04-25T01:43:23.972 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.973 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.973 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.973 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-04-25T01:43:23.973 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.973 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.973 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.974 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-04-25T01:43:23.974 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.974 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.974 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.974 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-04-25T01:43:23.974 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.974 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.974 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.974 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-04-25T01:43:23.974 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.974 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.974 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.975 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-04-25T01:43:23.975 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.975 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.975 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.975 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-04-25T01:43:23.975 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.975 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.975 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.975 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-04-25T01:43:23.975 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.975 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.975 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.976 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-04-25T01:43:23.976 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.976 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.976 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.976 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-04-25T01:43:23.976 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.976 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.976 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.976 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-04-25T01:43:23.976 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.976 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.977 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.977 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-04-25T01:43:23.977 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.977 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.977 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.977 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-04-25T01:43:23.977 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.977 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.977 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.977 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-04-25T01:43:23.977 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.977 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.978 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.978 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-04-25T01:43:23.978 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.978 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.978 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.978 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-04-25T01:43:23.978 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.978 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.978 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.978 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-04-25T01:43:23.978 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.979 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.979 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.979 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-04-25T01:43:23.979 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.979 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.979 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.979 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:43:23.979 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:43:23.979 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-04-25T01:43:23.979 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.979 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.979 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.979 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:23.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.980 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-04-25T01:43:23.980 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 1, 2024-04-25T01:43:23.980 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 1, 2024-04-25T01:43:23.980 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 0, 2024-04-25T01:43:23.980 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 1, 2024-04-25T01:43:23.980 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-04-25T01:43:23.980 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 0, 2024-04-25T01:43:23.980 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 0, 2024-04-25T01:43:23.980 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 1808, 2024-04-25T01:43:23.980 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-04-25T01:43:23.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.980 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-25T01:43:23.980 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:43:23.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-25T01:43:23.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:43:23.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:43:23.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:43:23.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:43:23.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:43:23.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:43:23.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:43:23.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:43:23.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:43:23.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:43:23.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:43:23.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:43:23.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:43:23.981 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-25T01:43:23.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:43:23.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-25T01:43:23.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:43:23.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:43:23.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:43:23.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:43:23.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:43:23.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:43:23.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:43:23.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-25T01:43:23.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:43:23.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:43:23.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:43:23.982 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-25T01:43:23.983 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-25T01:43:23.983 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-25T01:43:23.983 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:43:23.983 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:43:23.983 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:43:23.983 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:43:23.983 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 197473, 2024-04-25T01:43:23.983 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 55, 2024-04-25T01:43:23.983 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:43:23.983 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-04-25T01:43:23.983 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-25T01:43:23.983 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-25T01:43:23.983 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:43:23.983 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-25T01:43:23.983 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:43:23.984 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-25T01:43:23.984 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 9320, 2024-04-25T01:43:23.984 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 178, 2024-04-25T01:43:23.984 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:43:23.984 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:43:23.984 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-25T01:43:23.984 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-25T01:43:23.984 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 8464, 2024-04-25T01:43:23.984 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 13, 2024-04-25T01:43:23.984 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-25T01:43:23.984 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-25T01:43:23.984 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-25T01:43:23.984 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-25T01:43:23.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.984 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-04-25T01:43:23.985 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-04-25T01:43:23.985 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-04-25T01:43:23.985 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 2042, 2024-04-25T01:43:23.985 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 21, 2024-04-25T01:43:23.985 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 3, 2024-04-25T01:43:23.985 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 2042, 2024-04-25T01:43:23.985 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-04-25T01:43:23.985 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2042, 2024-04-25T01:43:23.985 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 4.619869510, 2024-04-25T01:43:23.985 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002262423 2024-04-25T01:43:23.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.985 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-04-25T01:43:23.985 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-04-25T01:43:23.985 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2042, 2024-04-25T01:43:23.985 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 2043 2024-04-25T01:43:23.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.986 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 2042, 2024-04-25T01:43:23.986 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 2042, 2024-04-25T01:43:23.986 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 0, 2024-04-25T01:43:23.986 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-04-25T01:43:23.986 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-04-25T01:43:23.986 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 1297, 2024-04-25T01:43:23.986 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 0, 2024-04-25T01:43:23.986 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 743, 2024-04-25T01:43:23.986 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 0, 2024-04-25T01:43:23.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 0, 2024-04-25T01:43:23.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-04-25T01:43:23.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-04-25T01:43:23.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 0, 2024-04-25T01:43:23.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 0, 2024-04-25T01:43:23.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 0, 2024-04-25T01:43:23.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-04-25T01:43:23.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-04-25T01:43:23.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-04-25T01:43:23.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 1, 2024-04-25T01:43:23.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 0, 2024-04-25T01:43:23.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:43:23.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-04-25T01:43:23.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:43:23.987 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-04-25T01:43:23.988 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-04-25T01:43:23.988 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-04-25T01:43:23.988 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:43:23.988 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-04-25T01:43:23.988 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:43:23.988 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-04-25T01:43:23.988 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-04-25T01:43:23.988 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-04-25T01:43:23.988 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-04-25T01:43:23.988 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-04-25T01:43:23.988 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-04-25T01:43:23.988 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-04-25T01:43:23.988 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-04-25T01:43:23.988 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-04-25T01:43:23.988 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-04-25T01:43:23.989 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-04-25T01:43:23.989 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-04-25T01:43:23.989 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-04-25T01:43:23.989 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-04-25T01:43:23.989 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-04-25T01:43:23.989 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-04-25T01:43:23.989 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-04-25T01:43:23.989 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 102, 2024-04-25T01:43:23.989 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-04-25T01:43:23.989 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 11, 2024-04-25T01:43:23.989 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 6, 2024-04-25T01:43:23.989 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 6, 2024-04-25T01:43:23.989 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 0, 2024-04-25T01:43:23.989 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-04-25T01:43:23.989 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 0, 2024-04-25T01:43:23.989 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 2, 2024-04-25T01:43:23.990 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 0 2024-04-25T01:43:23.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.990 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-04-25T01:43:23.990 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 0, 2024-04-25T01:43:23.990 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:43:23.990 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-04-25T01:43:23.990 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-04-25T01:43:23.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.990 INFO:teuthology.orchestra.run.smithi028.stdout: "osd-slow-ops": { 2024-04-25T01:43:23.990 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 0 2024-04-25T01:43:23.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.990 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-04-25T01:43:23.990 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-04-25T01:43:23.990 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T01:43:23.991 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-04-25T01:43:23.991 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 0, 2024-04-25T01:43:23.991 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 0, 2024-04-25T01:43:23.991 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 0, 2024-04-25T01:43:23.991 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-04-25T01:43:23.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.991 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:43:23.991 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-25T01:43:23.991 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-25T01:43:23.991 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-25T01:43:23.991 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 2282, 2024-04-25T01:43:23.991 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1839866, 2024-04-25T01:43:23.991 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:23.991 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 2282, 2024-04-25T01:43:23.991 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-25T01:43:23.991 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-25T01:43:23.992 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 2282, 2024-04-25T01:43:23.992 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1839866, 2024-04-25T01:43:23.992 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-25T01:43:23.992 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.992 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.992 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.992 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:23.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.992 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-04-25T01:43:23.992 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-25T01:43:23.992 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-25T01:43:23.992 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-25T01:43:23.992 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-25T01:43:23.992 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-25T01:43:23.992 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:23.993 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:23.993 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 2042, 2024-04-25T01:43:23.993 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 984, 2024-04-25T01:43:23.993 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 2042, 2024-04-25T01:43:23.993 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 984, 2024-04-25T01:43:23.993 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-25T01:43:23.993 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.993 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.993 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.993 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:23.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.993 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-04-25T01:43:23.993 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-25T01:43:23.993 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-04-25T01:43:23.993 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-25T01:43:23.994 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-25T01:43:23.994 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-25T01:43:23.994 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:23.994 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:23.994 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 2042, 2024-04-25T01:43:23.994 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 2042, 2024-04-25T01:43:23.994 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 2042, 2024-04-25T01:43:23.994 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 2042, 2024-04-25T01:43:23.994 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-25T01:43:23.994 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.994 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.994 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.994 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:23.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.994 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:43:23.995 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-25T01:43:23.995 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-04-25T01:43:23.995 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-25T01:43:23.995 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-25T01:43:23.995 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-25T01:43:23.995 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:23.995 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:23.995 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-25T01:43:23.995 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-25T01:43:23.995 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-04-25T01:43:23.995 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-04-25T01:43:23.995 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-25T01:43:23.995 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.995 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.995 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.996 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:23.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:23.996 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x562d4e7266e0": { 2024-04-25T01:43:23.996 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-25T01:43:23.996 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-04-25T01:43:23.996 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-25T01:43:23.996 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-25T01:43:23.996 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-25T01:43:23.996 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:23.996 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:23.996 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-25T01:43:23.996 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-25T01:43:23.996 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 0, 2024-04-25T01:43:23.996 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 0, 2024-04-25T01:43:23.996 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-25T01:43:23.997 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:23.997 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:23.997 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:23.997 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:23.997 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:23.997 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:43:24.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:24 smithi028 ceph-mon[29390]: pgmap v322: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:43:24.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:24 smithi130 ceph-mon[32702]: pgmap v322: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:43:24.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:24 smithi032 ceph-mon[32461]: pgmap v322: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:43:24.699 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-04-25T01:43:24.699 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-04-25T01:43:24.699 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-04-25T01:43:24.699 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-04-25T01:43:24.699 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T01:43:24.699 DEBUG:tasks.check_counter:Getting stats from j 2024-04-25T01:43:24.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-25T01:43:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:43:26.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:26 smithi028 ceph-mon[29390]: pgmap v323: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:43:26.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:26 smithi130 ceph-mon[32702]: pgmap v323: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:43:26.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:26 smithi032 ceph-mon[32461]: pgmap v323: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-04-25T01:43:26.938 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T01:43:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:43:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 0, 2024-04-25T01:43:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 2, 2024-04-25T01:43:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 0, 2024-04-25T01:43:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 0, 2024-04-25T01:43:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-04-25T01:43:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-04-25T01:43:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.001466547, 2024-04-25T01:43:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T01:43:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.000331575, 2024-04-25T01:43:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:43:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:26.939 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:43:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:43:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:43:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 227, 2024-04-25T01:43:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 217, 2024-04-25T01:43:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1444464, 2024-04-25T01:43:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 133184, 2024-04-25T01:43:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-04-25T01:43:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-04-25T01:43:26.940 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.046607800, 2024-04-25T01:43:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.014731218, 2024-04-25T01:43:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.034829503, 2024-04-25T01:43:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.004701138, 2024-04-25T01:43:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 216, 2024-04-25T01:43:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.014892192, 2024-04-25T01:43:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000068945 2024-04-25T01:43:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1444464, 2024-04-25T01:43:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 133184 2024-04-25T01:43:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:43:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 6, 2024-04-25T01:43:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 168, 2024-04-25T01:43:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 3312, 2024-04-25T01:43:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 184944, 2024-04-25T01:43:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-04-25T01:43:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-04-25T01:43:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.016123285, 2024-04-25T01:43:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.010602893, 2024-04-25T01:43:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.002867107, 2024-04-25T01:43:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T01:43:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 166, 2024-04-25T01:43:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.011705328, 2024-04-25T01:43:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000070514 2024-04-25T01:43:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-25T01:43:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 184944 2024-04-25T01:43:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-25T01:43:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-04-25T01:43:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-25T01:43:26.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-25T01:43:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:43:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-25T01:43:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:43:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:43:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:43:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:43:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:43:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:43:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:43:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:43:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:43:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:43:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:43:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:43:26.944 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:43:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-25T01:43:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:43:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-25T01:43:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:43:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:43:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:43:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:43:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:43:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:43:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:43:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-25T01:43:26.945 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:43:26.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:43:26.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:43:26.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-25T01:43:26.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-25T01:43:26.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-25T01:43:26.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:43:26.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:43:26.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:43:26.946 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:43:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 148977, 2024-04-25T01:43:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 30, 2024-04-25T01:43:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:43:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-04-25T01:43:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-25T01:43:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-25T01:43:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:43:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-25T01:43:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:43:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-25T01:43:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-04-25T01:43:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-04-25T01:43:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:43:26.947 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:43:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-25T01:43:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-25T01:43:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-04-25T01:43:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-04-25T01:43:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-25T01:43:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-25T01:43:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-25T01:43:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-25T01:43:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T01:43:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:43:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-25T01:43:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-25T01:43:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-25T01:43:26.948 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 229, 2024-04-25T01:43:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1412450, 2024-04-25T01:43:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 229, 2024-04-25T01:43:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-25T01:43:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-25T01:43:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 229, 2024-04-25T01:43:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1412450, 2024-04-25T01:43:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-25T01:43:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-25T01:43:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-25T01:43:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-25T01:43:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:26.949 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T01:43:26.949 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T01:43:27.848 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-25T01:43:27.848 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-25T01:43:27.848 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-25T01:43:27.848 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-25T01:43:27.848 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-25T01:43:27.849 DEBUG:tasks.check_counter:Getting stats from b 2024-04-25T01:43:27.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-25T01:43:28.111 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:43:28.234 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:28 smithi032 ceph-mon[32461]: pgmap v324: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:43:28.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:28 smithi028 ceph-mon[29390]: pgmap v324: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:43:28.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:28 smithi130 ceph-mon[32702]: pgmap v324: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:43:30.014 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-25T01:43:30.015 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:43:30.015 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 4672, 2024-04-25T01:43:30.015 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 4492, 2024-04-25T01:43:30.015 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 2713531, 2024-04-25T01:43:30.015 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 5958679, 2024-04-25T01:43:30.015 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 10, 2024-04-25T01:43:30.015 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 5, 2024-04-25T01:43:30.015 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.442325922, 2024-04-25T01:43:30.015 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.134541470, 2024-04-25T01:43:30.015 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.145449329, 2024-04-25T01:43:30.015 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.130746976, 2024-04-25T01:43:30.015 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:30.016 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 4494, 2024-04-25T01:43:30.016 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.670475186, 2024-04-25T01:43:30.016 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000149193 2024-04-25T01:43:30.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.016 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:30.016 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 1914, 2024-04-25T01:43:30.016 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000394666, 2024-04-25T01:43:30.016 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000206 2024-04-25T01:43:30.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.016 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 1517296, 2024-04-25T01:43:30.016 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 481680 2024-04-25T01:43:30.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.016 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:43:30.016 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 8821, 2024-04-25T01:43:30.017 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 9298, 2024-04-25T01:43:30.017 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 2858167, 2024-04-25T01:43:30.017 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 5766422, 2024-04-25T01:43:30.017 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 24, 2024-04-25T01:43:30.017 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 6, 2024-04-25T01:43:30.017 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.753506167, 2024-04-25T01:43:30.017 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.246226569, 2024-04-25T01:43:30.017 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.459603243, 2024-04-25T01:43:30.017 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.166598195, 2024-04-25T01:43:30.017 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:30.017 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 9299, 2024-04-25T01:43:30.017 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 1.607602837, 2024-04-25T01:43:30.017 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000172879 2024-04-25T01:43:30.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.017 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:30.018 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 10549, 2024-04-25T01:43:30.018 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.003172838, 2024-04-25T01:43:30.018 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000300 2024-04-25T01:43:30.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.018 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 299520, 2024-04-25T01:43:30.018 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 222144 2024-04-25T01:43:30.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.018 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:43:30.018 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 5085, 2024-04-25T01:43:30.018 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 4700, 2024-04-25T01:43:30.018 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 1406499, 2024-04-25T01:43:30.018 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 7780991, 2024-04-25T01:43:30.018 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 195, 2024-04-25T01:43:30.018 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 11, 2024-04-25T01:43:30.019 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.515717401, 2024-04-25T01:43:30.019 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.149876369, 2024-04-25T01:43:30.019 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.179484743, 2024-04-25T01:43:30.019 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.144303528, 2024-04-25T01:43:30.019 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:30.019 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 4692, 2024-04-25T01:43:30.019 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.791187979, 2024-04-25T01:43:30.019 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000168624 2024-04-25T01:43:30.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.019 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:30.019 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 1544, 2024-04-25T01:43:30.019 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000325352, 2024-04-25T01:43:30.019 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000210 2024-04-25T01:43:30.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.019 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 140816, 2024-04-25T01:43:30.020 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 32304 2024-04-25T01:43:30.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.020 INFO:teuthology.orchestra.run.smithi032.stdout: "cct": { 2024-04-25T01:43:30.020 INFO:teuthology.orchestra.run.smithi032.stdout: "total_workers": 1, 2024-04-25T01:43:30.020 INFO:teuthology.orchestra.run.smithi032.stdout: "unhealthy_workers": 0 2024-04-25T01:43:30.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.020 INFO:teuthology.orchestra.run.smithi032.stdout: "finisher-MDSRank": { 2024-04-25T01:43:30.020 INFO:teuthology.orchestra.run.smithi032.stdout: "queue_len": 0, 2024-04-25T01:43:30.020 INFO:teuthology.orchestra.run.smithi032.stdout: "complete_latency": { 2024-04-25T01:43:30.020 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 4507, 2024-04-25T01:43:30.020 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 3.182066335, 2024-04-25T01:43:30.020 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000706027 2024-04-25T01:43:30.020 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:30.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.020 INFO:teuthology.orchestra.run.smithi032.stdout: "finisher-PurgeQueue": { 2024-04-25T01:43:30.021 INFO:teuthology.orchestra.run.smithi032.stdout: "queue_len": 0, 2024-04-25T01:43:30.021 INFO:teuthology.orchestra.run.smithi032.stdout: "complete_latency": { 2024-04-25T01:43:30.021 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 4256, 2024-04-25T01:43:30.021 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.138471926, 2024-04-25T01:43:30.021 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000032535 2024-04-25T01:43:30.021 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:30.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.021 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-25T01:43:30.021 INFO:teuthology.orchestra.run.smithi032.stdout: "request": 169, 2024-04-25T01:43:30.021 INFO:teuthology.orchestra.run.smithi032.stdout: "reply": 169, 2024-04-25T01:43:30.021 INFO:teuthology.orchestra.run.smithi032.stdout: "reply_latency": { 2024-04-25T01:43:30.021 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 169, 2024-04-25T01:43:30.021 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.242323760, 2024-04-25T01:43:30.021 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.001433868 2024-04-25T01:43:30.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.022 INFO:teuthology.orchestra.run.smithi032.stdout: "slow_reply": 0, 2024-04-25T01:43:30.022 INFO:teuthology.orchestra.run.smithi032.stdout: "forward": 0, 2024-04-25T01:43:30.022 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_fetch_complete": 0, 2024-04-25T01:43:30.022 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_fetch_keys": 0, 2024-04-25T01:43:30.022 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_commit": 12, 2024-04-25T01:43:30.022 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_split": 0, 2024-04-25T01:43:30.022 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_merge": 0, 2024-04-25T01:43:30.022 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes": 35, 2024-04-25T01:43:30.022 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_top": 2, 2024-04-25T01:43:30.022 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_bottom": 33, 2024-04-25T01:43:30.022 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_pin_tail": 0, 2024-04-25T01:43:30.022 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_pinned": 32, 2024-04-25T01:43:30.022 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_expired": 0, 2024-04-25T01:43:30.022 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_with_caps": 3, 2024-04-25T01:43:30.023 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": 3, 2024-04-25T01:43:30.023 INFO:teuthology.orchestra.run.smithi032.stdout: "subtrees": 2, 2024-04-25T01:43:30.023 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse": 241, 2024-04-25T01:43:30.023 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_hit": 230, 2024-04-25T01:43:30.023 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_forward": 0, 2024-04-25T01:43:30.023 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_discover": 0, 2024-04-25T01:43:30.023 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:43:30.023 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_remote_ino": 0, 2024-04-25T01:43:30.023 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_lock": 0, 2024-04-25T01:43:30.023 INFO:teuthology.orchestra.run.smithi032.stdout: "load_cent": 0, 2024-04-25T01:43:30.023 INFO:teuthology.orchestra.run.smithi032.stdout: "q": 0, 2024-04-25T01:43:30.023 INFO:teuthology.orchestra.run.smithi032.stdout: "exported": 0, 2024-04-25T01:43:30.023 INFO:teuthology.orchestra.run.smithi032.stdout: "exported_inodes": 0, 2024-04-25T01:43:30.023 INFO:teuthology.orchestra.run.smithi032.stdout: "imported": 0, 2024-04-25T01:43:30.023 INFO:teuthology.orchestra.run.smithi032.stdout: "imported_inodes": 0, 2024-04-25T01:43:30.024 INFO:teuthology.orchestra.run.smithi032.stdout: "openino_dir_fetch": 0, 2024-04-25T01:43:30.024 INFO:teuthology.orchestra.run.smithi032.stdout: "openino_backtrace_fetch": 283, 2024-04-25T01:43:30.024 INFO:teuthology.orchestra.run.smithi032.stdout: "openino_peer_discover": 0, 2024-04-25T01:43:30.024 INFO:teuthology.orchestra.run.smithi032.stdout: "root_rfiles": 4, 2024-04-25T01:43:30.024 INFO:teuthology.orchestra.run.smithi032.stdout: "root_rbytes": 234, 2024-04-25T01:43:30.024 INFO:teuthology.orchestra.run.smithi032.stdout: "root_rsnaps": 6, 2024-04-25T01:43:30.024 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_backtrace_fetch": 283, 2024-04-25T01:43:30.024 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_set_tag": 0, 2024-04-25T01:43:30.024 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:43:30.024 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:43:30.024 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_dir_inodes": 181, 2024-04-25T01:43:30.024 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_dir_base_inodes": 25, 2024-04-25T01:43:30.024 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_dirfrag_rstats": 156, 2024-04-25T01:43:30.024 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_file_inodes": 102, 2024-04-25T01:43:30.024 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_inode_file_caps": 1, 2024-04-25T01:43:30.025 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_revoke": 12, 2024-04-25T01:43:30.025 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_grant": 1725, 2024-04-25T01:43:30.025 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:43:30.025 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:43:30.025 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:43:30.025 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_caps": 4252, 2024-04-25T01:43:30.025 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_caps_dirty": 4236, 2024-04-25T01:43:30.025 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_cap_release": 4, 2024-04-25T01:43:30.025 INFO:teuthology.orchestra.run.smithi032.stdout: "process_request_cap_release": 26 2024-04-25T01:43:30.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.025 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache": { 2024-04-25T01:43:30.025 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_update": 0, 2024-04-25T01:43:30.025 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_update_receipt": 0, 2024-04-25T01:43:30.025 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_try_discover": 0, 2024-04-25T01:43:30.025 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_send_discover": 0, 2024-04-25T01:43:30.026 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_handle_discover": 7, 2024-04-25T01:43:30.026 INFO:teuthology.orchestra.run.smithi032.stdout: "num_strays": 2, 2024-04-25T01:43:30.026 INFO:teuthology.orchestra.run.smithi032.stdout: "num_strays_delayed": 0, 2024-04-25T01:43:30.026 INFO:teuthology.orchestra.run.smithi032.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:43:30.026 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_created": 5, 2024-04-25T01:43:30.026 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_enqueued": 3, 2024-04-25T01:43:30.026 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_reintegrated": 0, 2024-04-25T01:43:30.026 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_migrated": 0, 2024-04-25T01:43:30.026 INFO:teuthology.orchestra.run.smithi032.stdout: "num_recovering_processing": 0, 2024-04-25T01:43:30.026 INFO:teuthology.orchestra.run.smithi032.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:43:30.026 INFO:teuthology.orchestra.run.smithi032.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:43:30.026 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_started": 0, 2024-04-25T01:43:30.026 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_completed": 0, 2024-04-25T01:43:30.026 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:43:30.027 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:43:30.027 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_enqueue_scrub": 25, 2024-04-25T01:43:30.027 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_exportdir": 0, 2024-04-25T01:43:30.027 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_flush": 0, 2024-04-25T01:43:30.027 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:43:30.027 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_fragstats": 0, 2024-04-25T01:43:30.027 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_inodestats": 0 2024-04-25T01:43:30.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.027 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log": { 2024-04-25T01:43:30.027 INFO:teuthology.orchestra.run.smithi032.stdout: "evlrg": 0, 2024-04-25T01:43:30.027 INFO:teuthology.orchestra.run.smithi032.stdout: "evadd": 4372, 2024-04-25T01:43:30.027 INFO:teuthology.orchestra.run.smithi032.stdout: "evex": 0, 2024-04-25T01:43:30.027 INFO:teuthology.orchestra.run.smithi032.stdout: "evtrm": 0, 2024-04-25T01:43:30.027 INFO:teuthology.orchestra.run.smithi032.stdout: "ev": 4372, 2024-04-25T01:43:30.027 INFO:teuthology.orchestra.run.smithi032.stdout: "evexg": 0, 2024-04-25T01:43:30.027 INFO:teuthology.orchestra.run.smithi032.stdout: "evexd": 0, 2024-04-25T01:43:30.028 INFO:teuthology.orchestra.run.smithi032.stdout: "segadd": 6, 2024-04-25T01:43:30.028 INFO:teuthology.orchestra.run.smithi032.stdout: "segex": 0, 2024-04-25T01:43:30.028 INFO:teuthology.orchestra.run.smithi032.stdout: "segtrm": 0, 2024-04-25T01:43:30.028 INFO:teuthology.orchestra.run.smithi032.stdout: "seg": 6, 2024-04-25T01:43:30.028 INFO:teuthology.orchestra.run.smithi032.stdout: "segmjr": 2, 2024-04-25T01:43:30.028 INFO:teuthology.orchestra.run.smithi032.stdout: "segexg": 0, 2024-04-25T01:43:30.028 INFO:teuthology.orchestra.run.smithi032.stdout: "segexd": 0, 2024-04-25T01:43:30.028 INFO:teuthology.orchestra.run.smithi032.stdout: "expos": 4194304, 2024-04-25T01:43:30.028 INFO:teuthology.orchestra.run.smithi032.stdout: "wrpos": 14882987, 2024-04-25T01:43:30.028 INFO:teuthology.orchestra.run.smithi032.stdout: "rdpos": 0, 2024-04-25T01:43:30.028 INFO:teuthology.orchestra.run.smithi032.stdout: "jlat": { 2024-04-25T01:43:30.028 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 4343, 2024-04-25T01:43:30.028 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 149.099517068, 2024-04-25T01:43:30.028 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.034330996 2024-04-25T01:43:30.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.029 INFO:teuthology.orchestra.run.smithi032.stdout: "replayed": 0 2024-04-25T01:43:30.029 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.029 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_mem": { 2024-04-25T01:43:30.029 INFO:teuthology.orchestra.run.smithi032.stdout: "ino": 26, 2024-04-25T01:43:30.029 INFO:teuthology.orchestra.run.smithi032.stdout: "ino+": 54, 2024-04-25T01:43:30.029 INFO:teuthology.orchestra.run.smithi032.stdout: "ino-": 28, 2024-04-25T01:43:30.029 INFO:teuthology.orchestra.run.smithi032.stdout: "dir": 20, 2024-04-25T01:43:30.029 INFO:teuthology.orchestra.run.smithi032.stdout: "dir+": 20, 2024-04-25T01:43:30.029 INFO:teuthology.orchestra.run.smithi032.stdout: "dir-": 0, 2024-04-25T01:43:30.029 INFO:teuthology.orchestra.run.smithi032.stdout: "dn": 35, 2024-04-25T01:43:30.029 INFO:teuthology.orchestra.run.smithi032.stdout: "dn+": 38, 2024-04-25T01:43:30.029 INFO:teuthology.orchestra.run.smithi032.stdout: "dn-": 3, 2024-04-25T01:43:30.029 INFO:teuthology.orchestra.run.smithi032.stdout: "cap": 3, 2024-04-25T01:43:30.029 INFO:teuthology.orchestra.run.smithi032.stdout: "cap+": 20, 2024-04-25T01:43:30.029 INFO:teuthology.orchestra.run.smithi032.stdout: "cap-": 17, 2024-04-25T01:43:30.030 INFO:teuthology.orchestra.run.smithi032.stdout: "rss": 59284, 2024-04-25T01:43:30.030 INFO:teuthology.orchestra.run.smithi032.stdout: "heap": 231804 2024-04-25T01:43:30.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.030 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_server": { 2024-04-25T01:43:30.030 INFO:teuthology.orchestra.run.smithi032.stdout: "dispatch_client_request": 216, 2024-04-25T01:43:30.030 INFO:teuthology.orchestra.run.smithi032.stdout: "dispatch_server_request": 1, 2024-04-25T01:43:30.030 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_request": 169, 2024-04-25T01:43:30.030 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_session": 38, 2024-04-25T01:43:30.030 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_peer_request": 2, 2024-04-25T01:43:30.030 INFO:teuthology.orchestra.run.smithi032.stdout: "req_create_latency": { 2024-04-25T01:43:30.030 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 7, 2024-04-25T01:43:30.030 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.008414389, 2024-04-25T01:43:30.030 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.001202055 2024-04-25T01:43:30.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.031 INFO:teuthology.orchestra.run.smithi032.stdout: "req_getattr_latency": { 2024-04-25T01:43:30.031 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 108, 2024-04-25T01:43:30.031 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.029159025, 2024-04-25T01:43:30.031 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000269990 2024-04-25T01:43:30.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.031 INFO:teuthology.orchestra.run.smithi032.stdout: "req_getfilelock_latency": { 2024-04-25T01:43:30.031 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.031 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.031 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.031 INFO:teuthology.orchestra.run.smithi032.stdout: "req_link_latency": { 2024-04-25T01:43:30.031 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.031 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.031 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.032 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookup_latency": { 2024-04-25T01:43:30.032 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 1, 2024-04-25T01:43:30.032 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000129185, 2024-04-25T01:43:30.032 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000129185 2024-04-25T01:43:30.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.032 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookuphash_latency": { 2024-04-25T01:43:30.032 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.032 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.032 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.032 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupino_latency": { 2024-04-25T01:43:30.032 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 1, 2024-04-25T01:43:30.032 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000208756, 2024-04-25T01:43:30.032 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000208756 2024-04-25T01:43:30.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.033 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupname_latency": { 2024-04-25T01:43:30.033 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.033 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.033 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.033 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupparent_latency": { 2024-04-25T01:43:30.033 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.033 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.033 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.033 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupsnap_latency": { 2024-04-25T01:43:30.033 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.033 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.033 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.033 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lssnap_latency": { 2024-04-25T01:43:30.034 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 10, 2024-04-25T01:43:30.034 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.002058745, 2024-04-25T01:43:30.034 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000205874 2024-04-25T01:43:30.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.034 INFO:teuthology.orchestra.run.smithi032.stdout: "req_mkdir_latency": { 2024-04-25T01:43:30.034 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 8, 2024-04-25T01:43:30.034 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.013706752, 2024-04-25T01:43:30.034 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.001713344 2024-04-25T01:43:30.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.034 INFO:teuthology.orchestra.run.smithi032.stdout: "req_mknod_latency": { 2024-04-25T01:43:30.034 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.034 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.034 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.035 INFO:teuthology.orchestra.run.smithi032.stdout: "req_mksnap_latency": { 2024-04-25T01:43:30.035 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 10, 2024-04-25T01:43:30.035 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.126890929, 2024-04-25T01:43:30.035 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.012689092 2024-04-25T01:43:30.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.035 INFO:teuthology.orchestra.run.smithi032.stdout: "req_open_latency": { 2024-04-25T01:43:30.035 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 1, 2024-04-25T01:43:30.035 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000184853, 2024-04-25T01:43:30.035 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000184853 2024-04-25T01:43:30.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.035 INFO:teuthology.orchestra.run.smithi032.stdout: "req_readdir_latency": { 2024-04-25T01:43:30.035 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 2, 2024-04-25T01:43:30.035 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.004385182, 2024-04-25T01:43:30.035 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.002192591 2024-04-25T01:43:30.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.036 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rename_latency": { 2024-04-25T01:43:30.036 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 2, 2024-04-25T01:43:30.036 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.002140870, 2024-04-25T01:43:30.036 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.001070435 2024-04-25T01:43:30.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.036 INFO:teuthology.orchestra.run.smithi032.stdout: "req_renamesnap_latency": { 2024-04-25T01:43:30.036 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.036 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.036 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.036 INFO:teuthology.orchestra.run.smithi032.stdout: "req_snapdiff_latency": { 2024-04-25T01:43:30.036 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.036 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.036 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.037 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rmdir_latency": { 2024-04-25T01:43:30.037 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 2, 2024-04-25T01:43:30.037 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.006773226, 2024-04-25T01:43:30.037 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.003386613 2024-04-25T01:43:30.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.037 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rmsnap_latency": { 2024-04-25T01:43:30.037 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 4, 2024-04-25T01:43:30.037 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.019429128, 2024-04-25T01:43:30.037 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.004857282 2024-04-25T01:43:30.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.037 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rmxattr_latency": { 2024-04-25T01:43:30.037 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.037 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.038 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.038 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setattr_latency": { 2024-04-25T01:43:30.038 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 2, 2024-04-25T01:43:30.038 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.001534450, 2024-04-25T01:43:30.038 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000767225 2024-04-25T01:43:30.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.038 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setdirlayout_latency": { 2024-04-25T01:43:30.038 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.038 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.038 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.038 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setfilelock_latency": { 2024-04-25T01:43:30.038 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.039 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.039 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.039 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setlayout_latency": { 2024-04-25T01:43:30.039 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.039 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.039 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.039 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setxattr_latency": { 2024-04-25T01:43:30.039 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 8, 2024-04-25T01:43:30.039 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.020586819, 2024-04-25T01:43:30.039 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.002573352 2024-04-25T01:43:30.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.039 INFO:teuthology.orchestra.run.smithi032.stdout: "req_symlink_latency": { 2024-04-25T01:43:30.039 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.040 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.040 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.040 INFO:teuthology.orchestra.run.smithi032.stdout: "req_unlink_latency": { 2024-04-25T01:43:30.040 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 3, 2024-04-25T01:43:30.040 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.006721451, 2024-04-25T01:43:30.040 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.002240483 2024-04-25T01:43:30.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.040 INFO:teuthology.orchestra.run.smithi032.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:43:30.040 INFO:teuthology.orchestra.run.smithi032.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:43:30.040 INFO:teuthology.orchestra.run.smithi032.stdout: "req_getvxattr_latency": { 2024-04-25T01:43:30.040 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.040 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.040 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.040 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:30.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.041 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_sessions": { 2024-04-25T01:43:30.041 INFO:teuthology.orchestra.run.smithi032.stdout: "session_count": 2, 2024-04-25T01:43:30.041 INFO:teuthology.orchestra.run.smithi032.stdout: "session_add": 6, 2024-04-25T01:43:30.041 INFO:teuthology.orchestra.run.smithi032.stdout: "session_remove": 4, 2024-04-25T01:43:30.041 INFO:teuthology.orchestra.run.smithi032.stdout: "sessions_open": 2, 2024-04-25T01:43:30.041 INFO:teuthology.orchestra.run.smithi032.stdout: "sessions_stale": 0, 2024-04-25T01:43:30.041 INFO:teuthology.orchestra.run.smithi032.stdout: "total_load": 27, 2024-04-25T01:43:30.041 INFO:teuthology.orchestra.run.smithi032.stdout: "average_load": 13, 2024-04-25T01:43:30.041 INFO:teuthology.orchestra.run.smithi032.stdout: "avg_session_uptime": 599, 2024-04-25T01:43:30.041 INFO:teuthology.orchestra.run.smithi032.stdout: "mdthresh_evicted": 0 2024-04-25T01:43:30.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.041 INFO:teuthology.orchestra.run.smithi032.stdout: "mempool": { 2024-04-25T01:43:30.041 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:43:30.041 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_items": 0, 2024-04-25T01:43:30.042 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:43:30.042 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:43:30.042 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:43:30.042 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:43:30.042 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:43:30.042 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:43:30.042 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:43:30.042 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:43:30.042 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:43:30.042 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:43:30.042 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:43:30.042 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:43:30.042 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:43:30.042 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_items": 0, 2024-04-25T01:43:30.043 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:43:30.043 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_items": 0, 2024-04-25T01:43:30.043 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:43:30.043 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:43:30.043 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:43:30.043 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:43:30.043 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:43:30.043 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:43:30.043 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:43:30.043 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_items": 0, 2024-04-25T01:43:30.043 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:43:30.043 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:43:30.043 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:43:30.043 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_items": 0, 2024-04-25T01:43:30.043 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_bytes": 0, 2024-04-25T01:43:30.044 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_items": 0, 2024-04-25T01:43:30.044 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:43:30.044 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:43:30.044 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:43:30.044 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:43:30.044 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_bytes": 350936, 2024-04-25T01:43:30.044 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_items": 150, 2024-04-25T01:43:30.044 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_bytes": 792, 2024-04-25T01:43:30.044 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_items": 9, 2024-04-25T01:43:30.044 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bytes": 0, 2024-04-25T01:43:30.044 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_items": 0, 2024-04-25T01:43:30.044 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:43:30.044 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_items": 0, 2024-04-25T01:43:30.044 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:43:30.044 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_items": 0, 2024-04-25T01:43:30.045 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_bytes": 9320, 2024-04-25T01:43:30.045 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_items": 178, 2024-04-25T01:43:30.045 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:43:30.045 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:43:30.045 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_bytes": 0, 2024-04-25T01:43:30.045 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_items": 0, 2024-04-25T01:43:30.045 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_bytes": 125490, 2024-04-25T01:43:30.045 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_items": 1014, 2024-04-25T01:43:30.045 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_bytes": 0, 2024-04-25T01:43:30.045 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_items": 0, 2024-04-25T01:43:30.045 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_bytes": 0, 2024-04-25T01:43:30.045 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_items": 0 2024-04-25T01:43:30.045 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.045 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter": { 2024-04-25T01:43:30.045 INFO:teuthology.orchestra.run.smithi032.stdout: "op_active": 0, 2024-04-25T01:43:30.046 INFO:teuthology.orchestra.run.smithi032.stdout: "op_laggy": 0, 2024-04-25T01:43:30.046 INFO:teuthology.orchestra.run.smithi032.stdout: "op_send": 9888, 2024-04-25T01:43:30.046 INFO:teuthology.orchestra.run.smithi032.stdout: "op_send_bytes": 10708076, 2024-04-25T01:43:30.046 INFO:teuthology.orchestra.run.smithi032.stdout: "op_resend": 31, 2024-04-25T01:43:30.046 INFO:teuthology.orchestra.run.smithi032.stdout: "op_reply": 9888, 2024-04-25T01:43:30.046 INFO:teuthology.orchestra.run.smithi032.stdout: "op_latency": { 2024-04-25T01:43:30.046 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 9888, 2024-04-25T01:43:30.046 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 192.074107494, 2024-04-25T01:43:30.046 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.019424970 2024-04-25T01:43:30.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.046 INFO:teuthology.orchestra.run.smithi032.stdout: "op_inflight": 0, 2024-04-25T01:43:30.046 INFO:teuthology.orchestra.run.smithi032.stdout: "oplen_avg": { 2024-04-25T01:43:30.046 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 9888, 2024-04-25T01:43:30.046 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 9933 2024-04-25T01:43:30.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.047 INFO:teuthology.orchestra.run.smithi032.stdout: "op": 9888, 2024-04-25T01:43:30.047 INFO:teuthology.orchestra.run.smithi032.stdout: "op_r": 336, 2024-04-25T01:43:30.047 INFO:teuthology.orchestra.run.smithi032.stdout: "op_w": 9552, 2024-04-25T01:43:30.047 INFO:teuthology.orchestra.run.smithi032.stdout: "op_rmw": 0, 2024-04-25T01:43:30.047 INFO:teuthology.orchestra.run.smithi032.stdout: "op_pg": 0, 2024-04-25T01:43:30.047 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_stat": 0, 2024-04-25T01:43:30.047 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_create": 11, 2024-04-25T01:43:30.047 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_read": 28, 2024-04-25T01:43:30.047 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_write": 4346, 2024-04-25T01:43:30.047 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_writefull": 31, 2024-04-25T01:43:30.047 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_writesame": 0, 2024-04-25T01:43:30.047 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_append": 0, 2024-04-25T01:43:30.047 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_zero": 0, 2024-04-25T01:43:30.048 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_truncate": 0, 2024-04-25T01:43:30.048 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_delete": 5127, 2024-04-25T01:43:30.048 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_mapext": 0, 2024-04-25T01:43:30.048 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_sparse_read": 0, 2024-04-25T01:43:30.048 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_clonerange": 0, 2024-04-25T01:43:30.048 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_getxattr": 308, 2024-04-25T01:43:30.048 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_setxattr": 22, 2024-04-25T01:43:30.048 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:43:30.048 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_rmxattr": 0, 2024-04-25T01:43:30.048 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:43:30.048 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_call": 0, 2024-04-25T01:43:30.048 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_watch": 0, 2024-04-25T01:43:30.048 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_notify": 0, 2024-04-25T01:43:30.048 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:43:30.049 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_pgls": 0, 2024-04-25T01:43:30.049 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:43:30.049 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_other": 0, 2024-04-25T01:43:30.049 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_active": 0, 2024-04-25T01:43:30.049 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_send": 0, 2024-04-25T01:43:30.049 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_resend": 0, 2024-04-25T01:43:30.049 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_ping": 0, 2024-04-25T01:43:30.049 INFO:teuthology.orchestra.run.smithi032.stdout: "poolop_active": 0, 2024-04-25T01:43:30.049 INFO:teuthology.orchestra.run.smithi032.stdout: "poolop_send": 0, 2024-04-25T01:43:30.049 INFO:teuthology.orchestra.run.smithi032.stdout: "poolop_resend": 0, 2024-04-25T01:43:30.049 INFO:teuthology.orchestra.run.smithi032.stdout: "poolstat_active": 0, 2024-04-25T01:43:30.049 INFO:teuthology.orchestra.run.smithi032.stdout: "poolstat_send": 0, 2024-04-25T01:43:30.049 INFO:teuthology.orchestra.run.smithi032.stdout: "poolstat_resend": 0, 2024-04-25T01:43:30.049 INFO:teuthology.orchestra.run.smithi032.stdout: "statfs_active": 0, 2024-04-25T01:43:30.049 INFO:teuthology.orchestra.run.smithi032.stdout: "statfs_send": 0, 2024-04-25T01:43:30.050 INFO:teuthology.orchestra.run.smithi032.stdout: "statfs_resend": 0, 2024-04-25T01:43:30.050 INFO:teuthology.orchestra.run.smithi032.stdout: "command_active": 0, 2024-04-25T01:43:30.050 INFO:teuthology.orchestra.run.smithi032.stdout: "command_send": 0, 2024-04-25T01:43:30.050 INFO:teuthology.orchestra.run.smithi032.stdout: "command_resend": 0, 2024-04-25T01:43:30.050 INFO:teuthology.orchestra.run.smithi032.stdout: "map_epoch": 102, 2024-04-25T01:43:30.050 INFO:teuthology.orchestra.run.smithi032.stdout: "map_full": 0, 2024-04-25T01:43:30.050 INFO:teuthology.orchestra.run.smithi032.stdout: "map_inc": 16, 2024-04-25T01:43:30.050 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_sessions": 12, 2024-04-25T01:43:30.050 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_session_open": 12, 2024-04-25T01:43:30.050 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_session_close": 0, 2024-04-25T01:43:30.050 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_laggy": 0, 2024-04-25T01:43:30.050 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_wr": 51, 2024-04-25T01:43:30.050 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_rd": 0, 2024-04-25T01:43:30.050 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_del": 9 2024-04-25T01:43:30.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.051 INFO:teuthology.orchestra.run.smithi032.stdout: "oft": { 2024-04-25T01:43:30.051 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_objs": 1, 2024-04-25T01:43:30.051 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:43:30.051 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_updates": 0, 2024-04-25T01:43:30.051 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_removes": 0 2024-04-25T01:43:30.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.051 INFO:teuthology.orchestra.run.smithi032.stdout: "osd-slow-ops": { 2024-04-25T01:43:30.051 INFO:teuthology.orchestra.run.smithi032.stdout: "slow_ops_count": 0 2024-04-25T01:43:30.051 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.051 INFO:teuthology.orchestra.run.smithi032.stdout: "purge_queue": { 2024-04-25T01:43:30.051 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing_ops": 0, 2024-04-25T01:43:30.051 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing_ops_high_water": 3585, 2024-04-25T01:43:30.051 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing": 0, 2024-04-25T01:43:30.051 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing_high_water": 1, 2024-04-25T01:43:30.051 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executed_ops": 4613, 2024-04-25T01:43:30.052 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executed": 3, 2024-04-25T01:43:30.052 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_item_in_journal": 0 2024-04-25T01:43:30.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.052 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:43:30.052 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:30.052 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 104857600, 2024-04-25T01:43:30.052 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:30.052 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 18574, 2024-04-25T01:43:30.052 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 5241682, 2024-04-25T01:43:30.052 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:30.052 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 18574, 2024-04-25T01:43:30.052 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-25T01:43:30.052 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-25T01:43:30.052 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 18574, 2024-04-25T01:43:30.052 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 5241682, 2024-04-25T01:43:30.053 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:30.053 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.053 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.053 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.053 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:30.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.053 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-objecter_bytes": { 2024-04-25T01:43:30.053 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:30.053 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 104857600, 2024-04-25T01:43:30.053 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:30.053 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-25T01:43:30.053 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-25T01:43:30.053 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:30.053 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:30.053 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 9888, 2024-04-25T01:43:30.054 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 10708411, 2024-04-25T01:43:30.054 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 9888, 2024-04-25T01:43:30.054 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 10708411, 2024-04-25T01:43:30.054 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:30.054 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.054 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.054 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.054 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:30.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.054 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-objecter_ops": { 2024-04-25T01:43:30.054 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:30.054 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 1024, 2024-04-25T01:43:30.054 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:30.054 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-25T01:43:30.055 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-25T01:43:30.055 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:30.055 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:30.055 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 9888, 2024-04-25T01:43:30.055 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 9888, 2024-04-25T01:43:30.055 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 9888, 2024-04-25T01:43:30.055 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 9888, 2024-04-25T01:43:30.055 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:30.055 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.055 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.055 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.055 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:30.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.055 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:43:30.055 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:30.056 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 3758096384, 2024-04-25T01:43:30.056 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:30.056 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 3, 2024-04-25T01:43:30.056 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 335, 2024-04-25T01:43:30.056 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:30.056 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 3, 2024-04-25T01:43:30.056 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-25T01:43:30.056 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-25T01:43:30.056 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 3, 2024-04-25T01:43:30.056 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 335, 2024-04-25T01:43:30.056 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:30.056 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.056 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.056 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.056 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:30.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:30.057 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-write_buf_throttle-0x5583dbe68500": { 2024-04-25T01:43:30.057 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:30.057 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 3758096384, 2024-04-25T01:43:30.057 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:30.057 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 4372, 2024-04-25T01:43:30.057 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 10688683, 2024-04-25T01:43:30.057 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:30.057 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 4372, 2024-04-25T01:43:30.057 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-25T01:43:30.057 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-25T01:43:30.057 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 4343, 2024-04-25T01:43:30.057 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 10688683, 2024-04-25T01:43:30.057 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:30.057 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:30.058 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:30.058 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:30.058 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:30.058 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:30.058 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-25T01:43:30.246 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:30 smithi032 ceph-mon[32461]: pgmap v325: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T01:43:30.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:30 smithi028 ceph-mon[29390]: pgmap v325: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T01:43:30.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:30 smithi130 ceph-mon[32702]: pgmap v325: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T01:43:30.955 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-04-25T01:43:30.955 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-04-25T01:43:30.956 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-25T01:43:30.956 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2024-04-25T01:43:30.956 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2024-04-25T01:43:30.956 DEBUG:tasks.check_counter:Getting stats from e 2024-04-25T01:43:30.956 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-25T01:43:31.215 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:43:32.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:32 smithi028 ceph-mon[29390]: pgmap v326: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:32.367 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:32 smithi032 ceph-mon[32461]: pgmap v326: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:32.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:32 smithi130 ceph-mon[32702]: pgmap v326: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:32.936 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-25T01:43:32.936 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:43:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 1036, 2024-04-25T01:43:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 1038, 2024-04-25T01:43:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 4521265, 2024-04-25T01:43:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 329458, 2024-04-25T01:43:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 6, 2024-04-25T01:43:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 2, 2024-04-25T01:43:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.108599982, 2024-04-25T01:43:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.042226127, 2024-04-25T01:43:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.025886047, 2024-04-25T01:43:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.028623160, 2024-04-25T01:43:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 1036, 2024-04-25T01:43:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.049873716, 2024-04-25T01:43:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000048140 2024-04-25T01:43:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.937 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:43:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:43:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:43:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 1026, 2024-04-25T01:43:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 1190, 2024-04-25T01:43:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 2560955, 2024-04-25T01:43:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 771154, 2024-04-25T01:43:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 7, 2024-04-25T01:43:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 5, 2024-04-25T01:43:32.938 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.084321673, 2024-04-25T01:43:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.024923143, 2024-04-25T01:43:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.023934047, 2024-04-25T01:43:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.028108122, 2024-04-25T01:43:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 1187, 2024-04-25T01:43:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.045814168, 2024-04-25T01:43:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000038596 2024-04-25T01:43:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T01:43:32.939 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 445456 2024-04-25T01:43:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:43:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 528, 2024-04-25T01:43:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 515, 2024-04-25T01:43:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 5743201, 2024-04-25T01:43:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 225283, 2024-04-25T01:43:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 3, 2024-04-25T01:43:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 2, 2024-04-25T01:43:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.075718760, 2024-04-25T01:43:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.021610508, 2024-04-25T01:43:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.043654094, 2024-04-25T01:43:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.015270195, 2024-04-25T01:43:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 515, 2024-04-25T01:43:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.024886200, 2024-04-25T01:43:32.940 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000048322 2024-04-25T01:43:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 1479680, 2024-04-25T01:43:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 136560 2024-04-25T01:43:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "cct": { 2024-04-25T01:43:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "total_workers": 1, 2024-04-25T01:43:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "unhealthy_workers": 0 2024-04-25T01:43:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "finisher-MDSRank": { 2024-04-25T01:43:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "queue_len": 0, 2024-04-25T01:43:32.941 INFO:teuthology.orchestra.run.smithi032.stdout: "complete_latency": { 2024-04-25T01:43:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 3875, 2024-04-25T01:43:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.098519812, 2024-04-25T01:43:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000025424 2024-04-25T01:43:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "finisher-PurgeQueue": { 2024-04-25T01:43:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "queue_len": 0, 2024-04-25T01:43:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "complete_latency": { 2024-04-25T01:43:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-25T01:43:32.942 INFO:teuthology.orchestra.run.smithi032.stdout: "request": 0, 2024-04-25T01:43:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "reply": 0, 2024-04-25T01:43:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "reply_latency": { 2024-04-25T01:43:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "slow_reply": 0, 2024-04-25T01:43:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "forward": 0, 2024-04-25T01:43:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_fetch_complete": 0, 2024-04-25T01:43:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_fetch_keys": 0, 2024-04-25T01:43:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_commit": 0, 2024-04-25T01:43:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_split": 0, 2024-04-25T01:43:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_merge": 0, 2024-04-25T01:43:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes": 27, 2024-04-25T01:43:32.943 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_top": 0, 2024-04-25T01:43:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_bottom": 27, 2024-04-25T01:43:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_pin_tail": 0, 2024-04-25T01:43:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_pinned": 27, 2024-04-25T01:43:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_expired": 0, 2024-04-25T01:43:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_with_caps": 0, 2024-04-25T01:43:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": 0, 2024-04-25T01:43:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "subtrees": 2, 2024-04-25T01:43:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse": 0, 2024-04-25T01:43:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_hit": 0, 2024-04-25T01:43:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_forward": 0, 2024-04-25T01:43:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_discover": 0, 2024-04-25T01:43:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:43:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_remote_ino": 0, 2024-04-25T01:43:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_lock": 0, 2024-04-25T01:43:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "load_cent": 0, 2024-04-25T01:43:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "q": 0, 2024-04-25T01:43:32.944 INFO:teuthology.orchestra.run.smithi032.stdout: "exported": 0, 2024-04-25T01:43:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "exported_inodes": 0, 2024-04-25T01:43:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "imported": 0, 2024-04-25T01:43:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "imported_inodes": 0, 2024-04-25T01:43:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "openino_dir_fetch": 0, 2024-04-25T01:43:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "openino_backtrace_fetch": 0, 2024-04-25T01:43:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "openino_peer_discover": 0, 2024-04-25T01:43:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "root_rfiles": 4, 2024-04-25T01:43:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "root_rbytes": 234, 2024-04-25T01:43:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "root_rsnaps": 6, 2024-04-25T01:43:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T01:43:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_set_tag": 0, 2024-04-25T01:43:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:43:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:43:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:43:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:43:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:43:32.945 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_file_inodes": 0, 2024-04-25T01:43:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:43:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T01:43:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_grant": 0, 2024-04-25T01:43:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:43:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:43:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:43:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_caps": 0, 2024-04-25T01:43:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_caps_dirty": 0, 2024-04-25T01:43:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_cap_release": 0, 2024-04-25T01:43:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "process_request_cap_release": 0 2024-04-25T01:43:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache": { 2024-04-25T01:43:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_update": 0, 2024-04-25T01:43:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_update_receipt": 0, 2024-04-25T01:43:32.946 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_try_discover": 0, 2024-04-25T01:43:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_send_discover": 0, 2024-04-25T01:43:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_handle_discover": 0, 2024-04-25T01:43:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "num_strays": 0, 2024-04-25T01:43:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "num_strays_delayed": 0, 2024-04-25T01:43:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:43:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_created": 0, 2024-04-25T01:43:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_enqueued": 0, 2024-04-25T01:43:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_reintegrated": 0, 2024-04-25T01:43:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_migrated": 0, 2024-04-25T01:43:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "num_recovering_processing": 0, 2024-04-25T01:43:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:43:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:43:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_started": 0, 2024-04-25T01:43:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_completed": 0, 2024-04-25T01:43:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:43:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:43:32.947 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:43:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_exportdir": 0, 2024-04-25T01:43:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_flush": 0, 2024-04-25T01:43:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:43:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_fragstats": 0, 2024-04-25T01:43:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_inodestats": 0 2024-04-25T01:43:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log": { 2024-04-25T01:43:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "evlrg": 0, 2024-04-25T01:43:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "evadd": 0, 2024-04-25T01:43:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "evex": 0, 2024-04-25T01:43:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "evtrm": 0, 2024-04-25T01:43:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "ev": 4372, 2024-04-25T01:43:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "evexg": 0, 2024-04-25T01:43:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "evexd": 0, 2024-04-25T01:43:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "segadd": 0, 2024-04-25T01:43:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "segex": 0, 2024-04-25T01:43:32.948 INFO:teuthology.orchestra.run.smithi032.stdout: "segtrm": 0, 2024-04-25T01:43:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "seg": 6, 2024-04-25T01:43:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "segmjr": 2, 2024-04-25T01:43:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "segexg": 0, 2024-04-25T01:43:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "segexd": 0, 2024-04-25T01:43:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "expos": 4194304, 2024-04-25T01:43:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "wrpos": 14882987, 2024-04-25T01:43:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "rdpos": 14882851, 2024-04-25T01:43:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "jlat": { 2024-04-25T01:43:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: "replayed": 4372 2024-04-25T01:43:32.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_mem": { 2024-04-25T01:43:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "ino": 18, 2024-04-25T01:43:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "ino+": 21, 2024-04-25T01:43:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "ino-": 3, 2024-04-25T01:43:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "dir": 12, 2024-04-25T01:43:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "dir+": 12, 2024-04-25T01:43:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "dir-": 0, 2024-04-25T01:43:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "dn": 27, 2024-04-25T01:43:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "dn+": 27, 2024-04-25T01:43:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "dn-": 0, 2024-04-25T01:43:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "cap": 0, 2024-04-25T01:43:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "cap+": 0, 2024-04-25T01:43:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "cap-": 0, 2024-04-25T01:43:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "rss": 55424, 2024-04-25T01:43:32.950 INFO:teuthology.orchestra.run.smithi032.stdout: "heap": 215420 2024-04-25T01:43:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_server": { 2024-04-25T01:43:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "dispatch_client_request": 0, 2024-04-25T01:43:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "dispatch_server_request": 0, 2024-04-25T01:43:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_request": 0, 2024-04-25T01:43:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_session": 0, 2024-04-25T01:43:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_peer_request": 0, 2024-04-25T01:43:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "req_create_latency": { 2024-04-25T01:43:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "req_getattr_latency": { 2024-04-25T01:43:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.951 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "req_getfilelock_latency": { 2024-04-25T01:43:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "req_link_latency": { 2024-04-25T01:43:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookup_latency": { 2024-04-25T01:43:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.952 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookuphash_latency": { 2024-04-25T01:43:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupino_latency": { 2024-04-25T01:43:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupname_latency": { 2024-04-25T01:43:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.953 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupparent_latency": { 2024-04-25T01:43:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupsnap_latency": { 2024-04-25T01:43:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lssnap_latency": { 2024-04-25T01:43:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.954 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: "req_mkdir_latency": { 2024-04-25T01:43:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: "req_mknod_latency": { 2024-04-25T01:43:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.955 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "req_mksnap_latency": { 2024-04-25T01:43:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "req_open_latency": { 2024-04-25T01:43:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "req_readdir_latency": { 2024-04-25T01:43:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.956 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rename_latency": { 2024-04-25T01:43:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "req_renamesnap_latency": { 2024-04-25T01:43:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "req_snapdiff_latency": { 2024-04-25T01:43:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rmdir_latency": { 2024-04-25T01:43:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rmsnap_latency": { 2024-04-25T01:43:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rmxattr_latency": { 2024-04-25T01:43:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.958 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setattr_latency": { 2024-04-25T01:43:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setdirlayout_latency": { 2024-04-25T01:43:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setfilelock_latency": { 2024-04-25T01:43:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.959 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setlayout_latency": { 2024-04-25T01:43:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setxattr_latency": { 2024-04-25T01:43:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "req_symlink_latency": { 2024-04-25T01:43:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.960 INFO:teuthology.orchestra.run.smithi032.stdout: "req_unlink_latency": { 2024-04-25T01:43:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:43:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:43:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "req_getvxattr_latency": { 2024-04-25T01:43:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_sessions": { 2024-04-25T01:43:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "session_count": 2, 2024-04-25T01:43:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "session_add": 6, 2024-04-25T01:43:32.961 INFO:teuthology.orchestra.run.smithi032.stdout: "session_remove": 4, 2024-04-25T01:43:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "sessions_open": 4, 2024-04-25T01:43:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "sessions_stale": 0, 2024-04-25T01:43:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "total_load": 0, 2024-04-25T01:43:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "average_load": 0, 2024-04-25T01:43:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "avg_session_uptime": 3005, 2024-04-25T01:43:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "mdthresh_evicted": 0 2024-04-25T01:43:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "mempool": { 2024-04-25T01:43:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:43:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_items": 0, 2024-04-25T01:43:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:43:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:43:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:43:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:43:32.962 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:43:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:43:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:43:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:43:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:43:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:43:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:43:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:43:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:43:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_items": 0, 2024-04-25T01:43:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:43:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_items": 0, 2024-04-25T01:43:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:43:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:43:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:43:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:43:32.963 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:43:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:43:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:43:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_items": 0, 2024-04-25T01:43:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:43:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:43:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:43:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_items": 0, 2024-04-25T01:43:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_bytes": 0, 2024-04-25T01:43:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_items": 0, 2024-04-25T01:43:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:43:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:43:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:43:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:43:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_bytes": 189685, 2024-04-25T01:43:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_items": 58, 2024-04-25T01:43:32.964 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:43:32.965 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_items": 0, 2024-04-25T01:43:32.965 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bytes": 0, 2024-04-25T01:43:32.965 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_items": 0, 2024-04-25T01:43:32.965 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:43:32.965 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_items": 0, 2024-04-25T01:43:32.965 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:43:32.965 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_items": 0, 2024-04-25T01:43:32.965 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_bytes": 9320, 2024-04-25T01:43:32.965 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_items": 178, 2024-04-25T01:43:32.965 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:43:32.965 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:43:32.965 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_bytes": 0, 2024-04-25T01:43:32.965 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_items": 0, 2024-04-25T01:43:32.965 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_bytes": 87450, 2024-04-25T01:43:32.965 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_items": 601, 2024-04-25T01:43:32.965 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_bytes": 0, 2024-04-25T01:43:32.965 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_items": 0, 2024-04-25T01:43:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_bytes": 0, 2024-04-25T01:43:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_items": 0 2024-04-25T01:43:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter": { 2024-04-25T01:43:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: "op_active": 0, 2024-04-25T01:43:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: "op_laggy": 0, 2024-04-25T01:43:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: "op_send": 2339, 2024-04-25T01:43:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: "op_send_bytes": 26, 2024-04-25T01:43:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: "op_resend": 4, 2024-04-25T01:43:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: "op_reply": 2339, 2024-04-25T01:43:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: "op_latency": { 2024-04-25T01:43:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 2339, 2024-04-25T01:43:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 1.472970191, 2024-04-25T01:43:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000629743 2024-04-25T01:43:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.966 INFO:teuthology.orchestra.run.smithi032.stdout: "op_inflight": 0, 2024-04-25T01:43:32.967 INFO:teuthology.orchestra.run.smithi032.stdout: "oplen_avg": { 2024-04-25T01:43:32.967 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 2339, 2024-04-25T01:43:32.967 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 2340 2024-04-25T01:43:32.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.967 INFO:teuthology.orchestra.run.smithi032.stdout: "op": 2339, 2024-04-25T01:43:32.967 INFO:teuthology.orchestra.run.smithi032.stdout: "op_r": 2339, 2024-04-25T01:43:32.967 INFO:teuthology.orchestra.run.smithi032.stdout: "op_w": 0, 2024-04-25T01:43:32.967 INFO:teuthology.orchestra.run.smithi032.stdout: "op_rmw": 0, 2024-04-25T01:43:32.967 INFO:teuthology.orchestra.run.smithi032.stdout: "op_pg": 0, 2024-04-25T01:43:32.967 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_stat": 1500, 2024-04-25T01:43:32.967 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_create": 0, 2024-04-25T01:43:32.967 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_read": 836, 2024-04-25T01:43:32.967 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_write": 0, 2024-04-25T01:43:32.967 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_writefull": 0, 2024-04-25T01:43:32.967 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_writesame": 0, 2024-04-25T01:43:32.967 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_append": 0, 2024-04-25T01:43:32.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_zero": 0, 2024-04-25T01:43:32.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_truncate": 0, 2024-04-25T01:43:32.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_delete": 0, 2024-04-25T01:43:32.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_mapext": 0, 2024-04-25T01:43:32.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_sparse_read": 0, 2024-04-25T01:43:32.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_clonerange": 0, 2024-04-25T01:43:32.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_getxattr": 2, 2024-04-25T01:43:32.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_setxattr": 0, 2024-04-25T01:43:32.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:43:32.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_rmxattr": 0, 2024-04-25T01:43:32.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:43:32.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_call": 0, 2024-04-25T01:43:32.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_watch": 0, 2024-04-25T01:43:32.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_notify": 0, 2024-04-25T01:43:32.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:43:32.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_pgls": 0, 2024-04-25T01:43:32.968 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:43:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_other": 0, 2024-04-25T01:43:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_active": 0, 2024-04-25T01:43:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_send": 0, 2024-04-25T01:43:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_resend": 0, 2024-04-25T01:43:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_ping": 0, 2024-04-25T01:43:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: "poolop_active": 0, 2024-04-25T01:43:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: "poolop_send": 0, 2024-04-25T01:43:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: "poolop_resend": 0, 2024-04-25T01:43:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: "poolstat_active": 0, 2024-04-25T01:43:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: "poolstat_send": 0, 2024-04-25T01:43:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: "poolstat_resend": 0, 2024-04-25T01:43:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: "statfs_active": 0, 2024-04-25T01:43:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: "statfs_send": 0, 2024-04-25T01:43:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: "statfs_resend": 0, 2024-04-25T01:43:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: "command_active": 0, 2024-04-25T01:43:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: "command_send": 0, 2024-04-25T01:43:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: "command_resend": 0, 2024-04-25T01:43:32.969 INFO:teuthology.orchestra.run.smithi032.stdout: "map_epoch": 102, 2024-04-25T01:43:32.970 INFO:teuthology.orchestra.run.smithi032.stdout: "map_full": 0, 2024-04-25T01:43:32.970 INFO:teuthology.orchestra.run.smithi032.stdout: "map_inc": 11, 2024-04-25T01:43:32.970 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_sessions": 5, 2024-04-25T01:43:32.970 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_session_open": 5, 2024-04-25T01:43:32.970 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_session_close": 0, 2024-04-25T01:43:32.970 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_laggy": 0, 2024-04-25T01:43:32.970 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_wr": 0, 2024-04-25T01:43:32.970 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_rd": 2, 2024-04-25T01:43:32.970 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_del": 0 2024-04-25T01:43:32.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.970 INFO:teuthology.orchestra.run.smithi032.stdout: "oft": { 2024-04-25T01:43:32.970 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_objs": 0, 2024-04-25T01:43:32.970 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:43:32.970 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_updates": 0, 2024-04-25T01:43:32.970 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_removes": 0 2024-04-25T01:43:32.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.971 INFO:teuthology.orchestra.run.smithi032.stdout: "osd-slow-ops": { 2024-04-25T01:43:32.971 INFO:teuthology.orchestra.run.smithi032.stdout: "slow_ops_count": 0 2024-04-25T01:43:32.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.971 INFO:teuthology.orchestra.run.smithi032.stdout: "purge_queue": { 2024-04-25T01:43:32.971 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing_ops": 0, 2024-04-25T01:43:32.971 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T01:43:32.971 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing": 0, 2024-04-25T01:43:32.971 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing_high_water": 0, 2024-04-25T01:43:32.971 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executed_ops": 0, 2024-04-25T01:43:32.971 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executed": 0, 2024-04-25T01:43:32.971 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_item_in_journal": 0 2024-04-25T01:43:32.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.971 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:43:32.971 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:32.971 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 104857600, 2024-04-25T01:43:32.971 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:32.971 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 2586, 2024-04-25T01:43:32.972 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 12577113, 2024-04-25T01:43:32.972 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:32.972 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 2586, 2024-04-25T01:43:32.972 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-25T01:43:32.972 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-25T01:43:32.972 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 2586, 2024-04-25T01:43:32.972 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 12577113, 2024-04-25T01:43:32.972 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:32.972 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.972 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.972 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.972 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:32.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.972 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-objecter_bytes": { 2024-04-25T01:43:32.972 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:32.972 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 104857600, 2024-04-25T01:43:32.972 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:32.973 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-25T01:43:32.973 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-25T01:43:32.973 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:32.973 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:32.973 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 2339, 2024-04-25T01:43:32.973 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 10688693, 2024-04-25T01:43:32.973 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 2339, 2024-04-25T01:43:32.973 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 10688693, 2024-04-25T01:43:32.973 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:32.973 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.973 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.973 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.973 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:32.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.973 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-objecter_ops": { 2024-04-25T01:43:32.973 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:32.973 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 1024, 2024-04-25T01:43:32.974 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:32.974 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-25T01:43:32.974 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-25T01:43:32.974 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:32.974 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:32.974 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 2339, 2024-04-25T01:43:32.974 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 2339, 2024-04-25T01:43:32.974 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 2339, 2024-04-25T01:43:32.974 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 2339, 2024-04-25T01:43:32.974 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:32.974 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.974 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.974 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.974 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:32.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.975 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:43:32.975 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:32.975 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 3758096384, 2024-04-25T01:43:32.975 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:32.975 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-25T01:43:32.975 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-25T01:43:32.975 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:32.975 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:32.975 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-25T01:43:32.975 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-25T01:43:32.975 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 0, 2024-04-25T01:43:32.975 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 0, 2024-04-25T01:43:32.975 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:32.975 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.975 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.975 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.975 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:32.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:32.976 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-write_buf_throttle-0x55d0765506e0": { 2024-04-25T01:43:32.976 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:32.976 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 3758096384, 2024-04-25T01:43:32.976 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:32.976 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-25T01:43:32.976 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-25T01:43:32.976 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:32.976 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:32.976 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-25T01:43:32.976 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-25T01:43:32.976 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 0, 2024-04-25T01:43:32.976 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 0, 2024-04-25T01:43:32.976 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:32.976 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:32.976 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:32.976 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:32.977 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:32.977 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:32.977 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-25T01:43:33.848 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-25T01:43:33.848 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-25T01:43:33.848 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-04-25T01:43:33.848 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-25T01:43:33.848 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T01:43:33.848 DEBUG:tasks.check_counter:Getting stats from h 2024-04-25T01:43:33.848 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-25T01:43:34.097 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:43:34.349 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:34 smithi032 ceph-mon[32461]: pgmap v327: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:43:34.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:34 smithi130 ceph-mon[32702]: pgmap v327: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:43:34.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:34 smithi028 ceph-mon[29390]: pgmap v327: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:43:35.780 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-25T01:43:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:43:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 412, 2024-04-25T01:43:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 423, 2024-04-25T01:43:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 1547881, 2024-04-25T01:43:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 246951, 2024-04-25T01:43:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 15, 2024-04-25T01:43:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 3, 2024-04-25T01:43:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.071901865, 2024-04-25T01:43:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.021029482, 2024-04-25T01:43:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.047306203, 2024-04-25T01:43:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.006512065, 2024-04-25T01:43:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 425, 2024-04-25T01:43:35.781 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.033453721, 2024-04-25T01:43:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000078714 2024-04-25T01:43:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 231, 2024-04-25T01:43:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000051512, 2024-04-25T01:43:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000222 2024-04-25T01:43:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 1499040, 2024-04-25T01:43:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 140080 2024-04-25T01:43:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.782 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:43:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 84, 2024-04-25T01:43:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 125, 2024-04-25T01:43:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 16343, 2024-04-25T01:43:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 736306, 2024-04-25T01:43:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 61, 2024-04-25T01:43:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 6, 2024-04-25T01:43:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.027912936, 2024-04-25T01:43:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.003749616, 2024-04-25T01:43:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.021427066, 2024-04-25T01:43:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.000555429, 2024-04-25T01:43:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 120, 2024-04-25T01:43:35.783 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.018199351, 2024-04-25T01:43:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000151661 2024-04-25T01:43:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 82, 2024-04-25T01:43:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000009119, 2024-04-25T01:43:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000111 2024-04-25T01:43:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 4160, 2024-04-25T01:43:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 136048 2024-04-25T01:43:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.784 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:43:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 514, 2024-04-25T01:43:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 773, 2024-04-25T01:43:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 152649, 2024-04-25T01:43:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 581520, 2024-04-25T01:43:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 8, 2024-04-25T01:43:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 6, 2024-04-25T01:43:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.077307041, 2024-04-25T01:43:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.037868890, 2024-04-25T01:43:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.034300539, 2024-04-25T01:43:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.004442153, 2024-04-25T01:43:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:35.785 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 771, 2024-04-25T01:43:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.055973799, 2024-04-25T01:43:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000072598 2024-04-25T01:43:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 1112, 2024-04-25T01:43:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000253836, 2024-04-25T01:43:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000228 2024-04-25T01:43:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T01:43:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 314944 2024-04-25T01:43:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "cct": { 2024-04-25T01:43:35.786 INFO:teuthology.orchestra.run.smithi032.stdout: "total_workers": 1, 2024-04-25T01:43:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "unhealthy_workers": 0 2024-04-25T01:43:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "finisher-MDSRank": { 2024-04-25T01:43:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "queue_len": 0, 2024-04-25T01:43:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "complete_latency": { 2024-04-25T01:43:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 56, 2024-04-25T01:43:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.002148889, 2024-04-25T01:43:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000038373 2024-04-25T01:43:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "finisher-PurgeQueue": { 2024-04-25T01:43:35.787 INFO:teuthology.orchestra.run.smithi032.stdout: "queue_len": 0, 2024-04-25T01:43:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "complete_latency": { 2024-04-25T01:43:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 2, 2024-04-25T01:43:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000005691, 2024-04-25T01:43:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000002845 2024-04-25T01:43:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-25T01:43:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "request": 9, 2024-04-25T01:43:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "reply": 6, 2024-04-25T01:43:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "reply_latency": { 2024-04-25T01:43:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 6, 2024-04-25T01:43:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.004640092, 2024-04-25T01:43:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000773348 2024-04-25T01:43:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.788 INFO:teuthology.orchestra.run.smithi032.stdout: "slow_reply": 0, 2024-04-25T01:43:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "forward": 3, 2024-04-25T01:43:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_fetch_complete": 0, 2024-04-25T01:43:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_fetch_keys": 0, 2024-04-25T01:43:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_commit": 11, 2024-04-25T01:43:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_split": 0, 2024-04-25T01:43:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_merge": 0, 2024-04-25T01:43:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes": 14, 2024-04-25T01:43:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_top": 0, 2024-04-25T01:43:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_bottom": 14, 2024-04-25T01:43:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_pin_tail": 0, 2024-04-25T01:43:35.789 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_pinned": 14, 2024-04-25T01:43:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_expired": 1, 2024-04-25T01:43:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_with_caps": 1, 2024-04-25T01:43:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": 1, 2024-04-25T01:43:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "subtrees": 2, 2024-04-25T01:43:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse": 15, 2024-04-25T01:43:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_hit": 8, 2024-04-25T01:43:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_forward": 2, 2024-04-25T01:43:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_discover": 2, 2024-04-25T01:43:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:43:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_remote_ino": 0, 2024-04-25T01:43:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_lock": 0, 2024-04-25T01:43:35.790 INFO:teuthology.orchestra.run.smithi032.stdout: "load_cent": 0, 2024-04-25T01:43:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "q": 0, 2024-04-25T01:43:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "exported": 0, 2024-04-25T01:43:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "exported_inodes": 0, 2024-04-25T01:43:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "imported": 0, 2024-04-25T01:43:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "imported_inodes": 0, 2024-04-25T01:43:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "openino_dir_fetch": 0, 2024-04-25T01:43:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "openino_backtrace_fetch": 0, 2024-04-25T01:43:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "openino_peer_discover": 0, 2024-04-25T01:43:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "root_rfiles": 4, 2024-04-25T01:43:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "root_rbytes": 234, 2024-04-25T01:43:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "root_rsnaps": 6, 2024-04-25T01:43:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T01:43:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_set_tag": 0, 2024-04-25T01:43:35.791 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:43:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:43:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:43:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:43:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:43:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_file_inodes": 0, 2024-04-25T01:43:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:43:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_revoke": 2, 2024-04-25T01:43:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_grant": 3, 2024-04-25T01:43:35.792 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:43:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:43:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:43:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_caps": 2, 2024-04-25T01:43:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_caps_dirty": 0, 2024-04-25T01:43:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_cap_release": 0, 2024-04-25T01:43:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "process_request_cap_release": 1 2024-04-25T01:43:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache": { 2024-04-25T01:43:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_update": 0, 2024-04-25T01:43:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_update_receipt": 0, 2024-04-25T01:43:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_try_discover": 0, 2024-04-25T01:43:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_send_discover": 3, 2024-04-25T01:43:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_handle_discover": 0, 2024-04-25T01:43:35.793 INFO:teuthology.orchestra.run.smithi032.stdout: "num_strays": 0, 2024-04-25T01:43:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "num_strays_delayed": 0, 2024-04-25T01:43:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:43:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_created": 0, 2024-04-25T01:43:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_enqueued": 0, 2024-04-25T01:43:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_reintegrated": 0, 2024-04-25T01:43:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_migrated": 0, 2024-04-25T01:43:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "num_recovering_processing": 0, 2024-04-25T01:43:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:43:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:43:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_started": 0, 2024-04-25T01:43:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_completed": 0, 2024-04-25T01:43:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:43:35.794 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:43:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:43:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_exportdir": 0, 2024-04-25T01:43:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_flush": 0, 2024-04-25T01:43:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:43:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_fragstats": 0, 2024-04-25T01:43:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_inodestats": 0 2024-04-25T01:43:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log": { 2024-04-25T01:43:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: "evlrg": 0, 2024-04-25T01:43:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: "evadd": 7, 2024-04-25T01:43:35.795 INFO:teuthology.orchestra.run.smithi032.stdout: "evex": 0, 2024-04-25T01:43:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "evtrm": 0, 2024-04-25T01:43:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "ev": 7, 2024-04-25T01:43:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "evexg": 0, 2024-04-25T01:43:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "evexd": 0, 2024-04-25T01:43:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "segadd": 2, 2024-04-25T01:43:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "segex": 0, 2024-04-25T01:43:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "segtrm": 0, 2024-04-25T01:43:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "seg": 2, 2024-04-25T01:43:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "segmjr": 2, 2024-04-25T01:43:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "segexg": 0, 2024-04-25T01:43:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "segexd": 0, 2024-04-25T01:43:35.796 INFO:teuthology.orchestra.run.smithi032.stdout: "expos": 4194304, 2024-04-25T01:43:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "wrpos": 4196264, 2024-04-25T01:43:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "rdpos": 0, 2024-04-25T01:43:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "jlat": { 2024-04-25T01:43:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 6, 2024-04-25T01:43:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.009964164, 2024-04-25T01:43:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.001660694 2024-04-25T01:43:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "replayed": 0 2024-04-25T01:43:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_mem": { 2024-04-25T01:43:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ino": 17, 2024-04-25T01:43:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ino+": 17, 2024-04-25T01:43:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "ino-": 0, 2024-04-25T01:43:35.797 INFO:teuthology.orchestra.run.smithi032.stdout: "dir": 15, 2024-04-25T01:43:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "dir+": 16, 2024-04-25T01:43:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "dir-": 1, 2024-04-25T01:43:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "dn": 14, 2024-04-25T01:43:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "dn+": 15, 2024-04-25T01:43:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "dn-": 1, 2024-04-25T01:43:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "cap": 1, 2024-04-25T01:43:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "cap+": 5, 2024-04-25T01:43:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "cap-": 4, 2024-04-25T01:43:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "rss": 52584, 2024-04-25T01:43:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: "heap": 223612 2024-04-25T01:43:35.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_server": { 2024-04-25T01:43:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "dispatch_client_request": 14, 2024-04-25T01:43:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "dispatch_server_request": 0, 2024-04-25T01:43:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_request": 9, 2024-04-25T01:43:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_session": 17, 2024-04-25T01:43:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_peer_request": 1, 2024-04-25T01:43:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "req_create_latency": { 2024-04-25T01:43:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "req_getattr_latency": { 2024-04-25T01:43:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 3, 2024-04-25T01:43:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.003706392, 2024-04-25T01:43:35.799 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.001235464 2024-04-25T01:43:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "req_getfilelock_latency": { 2024-04-25T01:43:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "req_link_latency": { 2024-04-25T01:43:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookup_latency": { 2024-04-25T01:43:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 3, 2024-04-25T01:43:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000933700, 2024-04-25T01:43:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000311233 2024-04-25T01:43:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookuphash_latency": { 2024-04-25T01:43:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupino_latency": { 2024-04-25T01:43:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.801 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupname_latency": { 2024-04-25T01:43:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupparent_latency": { 2024-04-25T01:43:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupsnap_latency": { 2024-04-25T01:43:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.802 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lssnap_latency": { 2024-04-25T01:43:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: "req_mkdir_latency": { 2024-04-25T01:43:35.803 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: "req_mknod_latency": { 2024-04-25T01:43:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: "req_mksnap_latency": { 2024-04-25T01:43:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: "req_open_latency": { 2024-04-25T01:43:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: "req_readdir_latency": { 2024-04-25T01:43:35.805 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rename_latency": { 2024-04-25T01:43:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "req_renamesnap_latency": { 2024-04-25T01:43:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.806 INFO:teuthology.orchestra.run.smithi032.stdout: "req_snapdiff_latency": { 2024-04-25T01:43:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rmdir_latency": { 2024-04-25T01:43:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rmsnap_latency": { 2024-04-25T01:43:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.807 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rmxattr_latency": { 2024-04-25T01:43:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setattr_latency": { 2024-04-25T01:43:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setdirlayout_latency": { 2024-04-25T01:43:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.808 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setfilelock_latency": { 2024-04-25T01:43:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setlayout_latency": { 2024-04-25T01:43:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setxattr_latency": { 2024-04-25T01:43:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.809 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "req_symlink_latency": { 2024-04-25T01:43:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "req_unlink_latency": { 2024-04-25T01:43:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:43:35.810 INFO:teuthology.orchestra.run.smithi032.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:43:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "req_getvxattr_latency": { 2024-04-25T01:43:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_sessions": { 2024-04-25T01:43:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "session_count": 1, 2024-04-25T01:43:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "session_add": 3, 2024-04-25T01:43:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "session_remove": 2, 2024-04-25T01:43:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "sessions_open": 1, 2024-04-25T01:43:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "sessions_stale": 0, 2024-04-25T01:43:35.811 INFO:teuthology.orchestra.run.smithi032.stdout: "total_load": 5, 2024-04-25T01:43:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "average_load": 1, 2024-04-25T01:43:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "avg_session_uptime": 633, 2024-04-25T01:43:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "mdthresh_evicted": 0 2024-04-25T01:43:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "mempool": { 2024-04-25T01:43:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:43:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_items": 0, 2024-04-25T01:43:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:43:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:43:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:43:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:43:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:43:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:43:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:43:35.812 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:43:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:43:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:43:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:43:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:43:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:43:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_items": 0, 2024-04-25T01:43:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:43:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_items": 0, 2024-04-25T01:43:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:43:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:43:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:43:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:43:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:43:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:43:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:43:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_items": 0, 2024-04-25T01:43:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:43:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:43:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:43:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_items": 0, 2024-04-25T01:43:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_bytes": 0, 2024-04-25T01:43:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_items": 0, 2024-04-25T01:43:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:43:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:43:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:43:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:43:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_bytes": 272535, 2024-04-25T01:43:35.814 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_items": 104, 2024-04-25T01:43:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:43:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_items": 0, 2024-04-25T01:43:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bytes": 0, 2024-04-25T01:43:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_items": 0, 2024-04-25T01:43:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:43:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_items": 0, 2024-04-25T01:43:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:43:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_items": 0, 2024-04-25T01:43:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_bytes": 9320, 2024-04-25T01:43:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_items": 178, 2024-04-25T01:43:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:43:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:43:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_bytes": 0, 2024-04-25T01:43:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_items": 0, 2024-04-25T01:43:35.815 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_bytes": 67853, 2024-04-25T01:43:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_items": 299, 2024-04-25T01:43:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_bytes": 0, 2024-04-25T01:43:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_items": 0, 2024-04-25T01:43:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_bytes": 0, 2024-04-25T01:43:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_items": 0 2024-04-25T01:43:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter": { 2024-04-25T01:43:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "op_active": 0, 2024-04-25T01:43:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "op_laggy": 0, 2024-04-25T01:43:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "op_send": 46, 2024-04-25T01:43:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "op_send_bytes": 12851, 2024-04-25T01:43:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "op_resend": 30, 2024-04-25T01:43:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "op_reply": 46, 2024-04-25T01:43:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "op_latency": { 2024-04-25T01:43:35.816 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 46, 2024-04-25T01:43:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.238995210, 2024-04-25T01:43:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.005195548 2024-04-25T01:43:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "op_inflight": 0, 2024-04-25T01:43:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "oplen_avg": { 2024-04-25T01:43:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 46, 2024-04-25T01:43:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 72 2024-04-25T01:43:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "op": 46, 2024-04-25T01:43:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "op_r": 0, 2024-04-25T01:43:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "op_w": 46, 2024-04-25T01:43:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "op_rmw": 0, 2024-04-25T01:43:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "op_pg": 0, 2024-04-25T01:43:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_stat": 0, 2024-04-25T01:43:35.817 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_create": 10, 2024-04-25T01:43:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_read": 0, 2024-04-25T01:43:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_write": 6, 2024-04-25T01:43:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_writefull": 7, 2024-04-25T01:43:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_writesame": 0, 2024-04-25T01:43:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_append": 0, 2024-04-25T01:43:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_zero": 0, 2024-04-25T01:43:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_truncate": 0, 2024-04-25T01:43:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_delete": 6, 2024-04-25T01:43:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_mapext": 0, 2024-04-25T01:43:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_sparse_read": 0, 2024-04-25T01:43:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_clonerange": 0, 2024-04-25T01:43:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_getxattr": 0, 2024-04-25T01:43:35.818 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_setxattr": 20, 2024-04-25T01:43:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:43:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_rmxattr": 0, 2024-04-25T01:43:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:43:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_call": 0, 2024-04-25T01:43:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_watch": 0, 2024-04-25T01:43:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_notify": 0, 2024-04-25T01:43:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:43:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_pgls": 0, 2024-04-25T01:43:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:43:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_other": 0, 2024-04-25T01:43:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_active": 0, 2024-04-25T01:43:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_send": 0, 2024-04-25T01:43:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_resend": 0, 2024-04-25T01:43:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_ping": 0, 2024-04-25T01:43:35.819 INFO:teuthology.orchestra.run.smithi032.stdout: "poolop_active": 0, 2024-04-25T01:43:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "poolop_send": 0, 2024-04-25T01:43:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "poolop_resend": 0, 2024-04-25T01:43:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "poolstat_active": 0, 2024-04-25T01:43:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "poolstat_send": 0, 2024-04-25T01:43:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "poolstat_resend": 0, 2024-04-25T01:43:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "statfs_active": 0, 2024-04-25T01:43:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "statfs_send": 0, 2024-04-25T01:43:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "statfs_resend": 0, 2024-04-25T01:43:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "command_active": 0, 2024-04-25T01:43:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "command_send": 0, 2024-04-25T01:43:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "command_resend": 0, 2024-04-25T01:43:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "map_epoch": 102, 2024-04-25T01:43:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "map_full": 0, 2024-04-25T01:43:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "map_inc": 11, 2024-04-25T01:43:35.820 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_sessions": 8, 2024-04-25T01:43:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_session_open": 8, 2024-04-25T01:43:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_session_close": 0, 2024-04-25T01:43:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_laggy": 0, 2024-04-25T01:43:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_wr": 22, 2024-04-25T01:43:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_rd": 0, 2024-04-25T01:43:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_del": 1 2024-04-25T01:43:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: "oft": { 2024-04-25T01:43:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_objs": 1, 2024-04-25T01:43:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:43:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_updates": 0, 2024-04-25T01:43:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_removes": 0 2024-04-25T01:43:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "osd-slow-ops": { 2024-04-25T01:43:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "slow_ops_count": 0 2024-04-25T01:43:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "purge_queue": { 2024-04-25T01:43:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing_ops": 0, 2024-04-25T01:43:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T01:43:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing": 0, 2024-04-25T01:43:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing_high_water": 0, 2024-04-25T01:43:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executed_ops": 0, 2024-04-25T01:43:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executed": 0, 2024-04-25T01:43:35.822 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_item_in_journal": 0 2024-04-25T01:43:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:43:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 104857600, 2024-04-25T01:43:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 1006, 2024-04-25T01:43:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 1608750, 2024-04-25T01:43:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 1006, 2024-04-25T01:43:35.823 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-25T01:43:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-25T01:43:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 1006, 2024-04-25T01:43:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 1608750, 2024-04-25T01:43:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-objecter_bytes": { 2024-04-25T01:43:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 104857600, 2024-04-25T01:43:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-25T01:43:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-25T01:43:35.824 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 46, 2024-04-25T01:43:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 12851, 2024-04-25T01:43:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 46, 2024-04-25T01:43:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 12851, 2024-04-25T01:43:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-objecter_ops": { 2024-04-25T01:43:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 1024, 2024-04-25T01:43:35.825 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-25T01:43:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-25T01:43:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 46, 2024-04-25T01:43:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 46, 2024-04-25T01:43:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 46, 2024-04-25T01:43:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 46, 2024-04-25T01:43:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.826 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:43:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 3758096384, 2024-04-25T01:43:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-25T01:43:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-25T01:43:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-25T01:43:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-25T01:43:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 0, 2024-04-25T01:43:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 0, 2024-04-25T01:43:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.827 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-write_buf_throttle-0x561ac61be500": { 2024-04-25T01:43:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 3758096384, 2024-04-25T01:43:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 7, 2024-04-25T01:43:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 1960, 2024-04-25T01:43:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 7, 2024-04-25T01:43:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-25T01:43:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-25T01:43:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 6, 2024-04-25T01:43:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 1960, 2024-04-25T01:43:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:35.828 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:35.829 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:35.829 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:35.829 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:35.829 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-25T01:43:36.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:36 smithi130 ceph-mon[32702]: pgmap v328: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:36.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:36 smithi032 ceph-mon[32461]: pgmap v328: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:36.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:36 smithi028 ceph-mon[29390]: pgmap v328: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:36.702 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-25T01:43:36.702 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-25T01:43:36.702 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-04-25T01:43:36.703 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-25T01:43:36.703 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T01:43:36.703 DEBUG:tasks.check_counter:Getting stats from k 2024-04-25T01:43:36.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-25T01:43:36.953 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:43:38.414 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:38 smithi032 ceph-mon[32461]: pgmap v329: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:43:38.471 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:38 smithi130 ceph-mon[32702]: pgmap v329: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:43:38.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:38 smithi028 ceph-mon[29390]: pgmap v329: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:43:38.679 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-25T01:43:38.679 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:43:38.679 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 248, 2024-04-25T01:43:38.679 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 237, 2024-04-25T01:43:38.679 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 1478178, 2024-04-25T01:43:38.679 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 138196, 2024-04-25T01:43:38.679 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 5, 2024-04-25T01:43:38.679 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 1, 2024-04-25T01:43:38.679 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.048930325, 2024-04-25T01:43:38.680 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.014914572, 2024-04-25T01:43:38.680 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.035170195, 2024-04-25T01:43:38.680 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.006027790, 2024-04-25T01:43:38.680 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:38.680 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 236, 2024-04-25T01:43:38.680 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.013209183, 2024-04-25T01:43:38.680 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000055971 2024-04-25T01:43:38.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.680 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:38.680 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.680 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.680 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.680 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.680 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 1477344, 2024-04-25T01:43:38.680 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 137872 2024-04-25T01:43:38.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.681 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:43:38.681 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 643, 2024-04-25T01:43:38.681 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 809, 2024-04-25T01:43:38.681 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 158028, 2024-04-25T01:43:38.681 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 649651, 2024-04-25T01:43:38.681 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 7, 2024-04-25T01:43:38.681 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 5, 2024-04-25T01:43:38.681 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.073948201, 2024-04-25T01:43:38.681 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.026898759, 2024-04-25T01:43:38.681 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.015542571, 2024-04-25T01:43:38.681 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.022784916, 2024-04-25T01:43:38.681 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:38.681 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 806, 2024-04-25T01:43:38.682 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.038210574, 2024-04-25T01:43:38.682 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000047407 2024-04-25T01:43:38.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.682 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:38.682 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.682 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.682 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.682 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T01:43:38.682 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 445808 2024-04-25T01:43:38.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.682 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:43:38.682 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 1522, 2024-04-25T01:43:38.682 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 1522, 2024-04-25T01:43:38.683 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 458615, 2024-04-25T01:43:38.683 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 483230, 2024-04-25T01:43:38.683 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 4, 2024-04-25T01:43:38.683 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 3, 2024-04-25T01:43:38.683 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.164533383, 2024-04-25T01:43:38.683 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.058021413, 2024-04-25T01:43:38.683 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.041125219, 2024-04-25T01:43:38.683 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.049771362, 2024-04-25T01:43:38.683 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:38.683 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 1521, 2024-04-25T01:43:38.683 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.067795984, 2024-04-25T01:43:38.683 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000044573 2024-04-25T01:43:38.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.683 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:38.683 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.684 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.684 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.684 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-25T01:43:38.684 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T01:43:38.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.684 INFO:teuthology.orchestra.run.smithi032.stdout: "cct": { 2024-04-25T01:43:38.684 INFO:teuthology.orchestra.run.smithi032.stdout: "total_workers": 1, 2024-04-25T01:43:38.684 INFO:teuthology.orchestra.run.smithi032.stdout: "unhealthy_workers": 0 2024-04-25T01:43:38.684 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.684 INFO:teuthology.orchestra.run.smithi032.stdout: "finisher-MDSRank": { 2024-04-25T01:43:38.684 INFO:teuthology.orchestra.run.smithi032.stdout: "queue_len": 0, 2024-04-25T01:43:38.684 INFO:teuthology.orchestra.run.smithi032.stdout: "complete_latency": { 2024-04-25T01:43:38.684 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 3617, 2024-04-25T01:43:38.685 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.115979669, 2024-04-25T01:43:38.685 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000032065 2024-04-25T01:43:38.685 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:38.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.685 INFO:teuthology.orchestra.run.smithi032.stdout: "finisher-PurgeQueue": { 2024-04-25T01:43:38.685 INFO:teuthology.orchestra.run.smithi032.stdout: "queue_len": 0, 2024-04-25T01:43:38.685 INFO:teuthology.orchestra.run.smithi032.stdout: "complete_latency": { 2024-04-25T01:43:38.685 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.685 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.685 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.685 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:38.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.685 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-25T01:43:38.685 INFO:teuthology.orchestra.run.smithi032.stdout: "request": 0, 2024-04-25T01:43:38.686 INFO:teuthology.orchestra.run.smithi032.stdout: "reply": 0, 2024-04-25T01:43:38.686 INFO:teuthology.orchestra.run.smithi032.stdout: "reply_latency": { 2024-04-25T01:43:38.686 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.686 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.686 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.686 INFO:teuthology.orchestra.run.smithi032.stdout: "slow_reply": 0, 2024-04-25T01:43:38.686 INFO:teuthology.orchestra.run.smithi032.stdout: "forward": 0, 2024-04-25T01:43:38.686 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_fetch_complete": 0, 2024-04-25T01:43:38.686 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_fetch_keys": 0, 2024-04-25T01:43:38.686 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_commit": 0, 2024-04-25T01:43:38.686 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_split": 0, 2024-04-25T01:43:38.686 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_merge": 0, 2024-04-25T01:43:38.686 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes": 0, 2024-04-25T01:43:38.687 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_top": 0, 2024-04-25T01:43:38.687 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_bottom": 0, 2024-04-25T01:43:38.687 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_pin_tail": 0, 2024-04-25T01:43:38.687 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_pinned": 0, 2024-04-25T01:43:38.687 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_expired": 0, 2024-04-25T01:43:38.687 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_with_caps": 0, 2024-04-25T01:43:38.687 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": 0, 2024-04-25T01:43:38.687 INFO:teuthology.orchestra.run.smithi032.stdout: "subtrees": 1, 2024-04-25T01:43:38.687 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse": 0, 2024-04-25T01:43:38.687 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_hit": 0, 2024-04-25T01:43:38.687 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_forward": 0, 2024-04-25T01:43:38.687 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_discover": 0, 2024-04-25T01:43:38.687 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:43:38.687 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_remote_ino": 0, 2024-04-25T01:43:38.687 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_lock": 0, 2024-04-25T01:43:38.688 INFO:teuthology.orchestra.run.smithi032.stdout: "load_cent": 0, 2024-04-25T01:43:38.688 INFO:teuthology.orchestra.run.smithi032.stdout: "q": 0, 2024-04-25T01:43:38.688 INFO:teuthology.orchestra.run.smithi032.stdout: "exported": 0, 2024-04-25T01:43:38.688 INFO:teuthology.orchestra.run.smithi032.stdout: "exported_inodes": 0, 2024-04-25T01:43:38.688 INFO:teuthology.orchestra.run.smithi032.stdout: "imported": 0, 2024-04-25T01:43:38.688 INFO:teuthology.orchestra.run.smithi032.stdout: "imported_inodes": 0, 2024-04-25T01:43:38.688 INFO:teuthology.orchestra.run.smithi032.stdout: "openino_dir_fetch": 0, 2024-04-25T01:43:38.688 INFO:teuthology.orchestra.run.smithi032.stdout: "openino_backtrace_fetch": 0, 2024-04-25T01:43:38.688 INFO:teuthology.orchestra.run.smithi032.stdout: "openino_peer_discover": 0, 2024-04-25T01:43:38.688 INFO:teuthology.orchestra.run.smithi032.stdout: "root_rfiles": 0, 2024-04-25T01:43:38.688 INFO:teuthology.orchestra.run.smithi032.stdout: "root_rbytes": 0, 2024-04-25T01:43:38.688 INFO:teuthology.orchestra.run.smithi032.stdout: "root_rsnaps": 0, 2024-04-25T01:43:38.688 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T01:43:38.688 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_set_tag": 0, 2024-04-25T01:43:38.688 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:43:38.689 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:43:38.689 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:43:38.689 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:43:38.689 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:43:38.689 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_file_inodes": 0, 2024-04-25T01:43:38.689 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:43:38.689 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T01:43:38.689 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_grant": 0, 2024-04-25T01:43:38.689 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:43:38.689 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:43:38.689 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:43:38.689 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_caps": 0, 2024-04-25T01:43:38.689 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_caps_dirty": 0, 2024-04-25T01:43:38.689 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_cap_release": 0, 2024-04-25T01:43:38.690 INFO:teuthology.orchestra.run.smithi032.stdout: "process_request_cap_release": 0 2024-04-25T01:43:38.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.690 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache": { 2024-04-25T01:43:38.690 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_update": 0, 2024-04-25T01:43:38.690 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_update_receipt": 0, 2024-04-25T01:43:38.690 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_try_discover": 0, 2024-04-25T01:43:38.690 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_send_discover": 0, 2024-04-25T01:43:38.690 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_handle_discover": 0, 2024-04-25T01:43:38.690 INFO:teuthology.orchestra.run.smithi032.stdout: "num_strays": 0, 2024-04-25T01:43:38.690 INFO:teuthology.orchestra.run.smithi032.stdout: "num_strays_delayed": 0, 2024-04-25T01:43:38.690 INFO:teuthology.orchestra.run.smithi032.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:43:38.690 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_created": 0, 2024-04-25T01:43:38.690 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_enqueued": 0, 2024-04-25T01:43:38.691 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_reintegrated": 0, 2024-04-25T01:43:38.691 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_migrated": 0, 2024-04-25T01:43:38.691 INFO:teuthology.orchestra.run.smithi032.stdout: "num_recovering_processing": 0, 2024-04-25T01:43:38.691 INFO:teuthology.orchestra.run.smithi032.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:43:38.691 INFO:teuthology.orchestra.run.smithi032.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:43:38.691 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_started": 0, 2024-04-25T01:43:38.691 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_completed": 0, 2024-04-25T01:43:38.691 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:43:38.691 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:43:38.691 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:43:38.691 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_exportdir": 0, 2024-04-25T01:43:38.691 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_flush": 0, 2024-04-25T01:43:38.691 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:43:38.691 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_fragstats": 0, 2024-04-25T01:43:38.691 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_inodestats": 0 2024-04-25T01:43:38.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.692 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log": { 2024-04-25T01:43:38.692 INFO:teuthology.orchestra.run.smithi032.stdout: "evlrg": 0, 2024-04-25T01:43:38.692 INFO:teuthology.orchestra.run.smithi032.stdout: "evadd": 0, 2024-04-25T01:43:38.692 INFO:teuthology.orchestra.run.smithi032.stdout: "evex": 0, 2024-04-25T01:43:38.692 INFO:teuthology.orchestra.run.smithi032.stdout: "evtrm": 0, 2024-04-25T01:43:38.692 INFO:teuthology.orchestra.run.smithi032.stdout: "ev": 7, 2024-04-25T01:43:38.692 INFO:teuthology.orchestra.run.smithi032.stdout: "evexg": 0, 2024-04-25T01:43:38.692 INFO:teuthology.orchestra.run.smithi032.stdout: "evexd": 0, 2024-04-25T01:43:38.692 INFO:teuthology.orchestra.run.smithi032.stdout: "segadd": 0, 2024-04-25T01:43:38.692 INFO:teuthology.orchestra.run.smithi032.stdout: "segex": 0, 2024-04-25T01:43:38.692 INFO:teuthology.orchestra.run.smithi032.stdout: "segtrm": 0, 2024-04-25T01:43:38.692 INFO:teuthology.orchestra.run.smithi032.stdout: "seg": 2, 2024-04-25T01:43:38.692 INFO:teuthology.orchestra.run.smithi032.stdout: "segmjr": 2, 2024-04-25T01:43:38.692 INFO:teuthology.orchestra.run.smithi032.stdout: "segexg": 0, 2024-04-25T01:43:38.693 INFO:teuthology.orchestra.run.smithi032.stdout: "segexd": 0, 2024-04-25T01:43:38.693 INFO:teuthology.orchestra.run.smithi032.stdout: "expos": 4194304, 2024-04-25T01:43:38.693 INFO:teuthology.orchestra.run.smithi032.stdout: "wrpos": 4196264, 2024-04-25T01:43:38.693 INFO:teuthology.orchestra.run.smithi032.stdout: "rdpos": 4196123, 2024-04-25T01:43:38.693 INFO:teuthology.orchestra.run.smithi032.stdout: "jlat": { 2024-04-25T01:43:38.693 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.693 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.693 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.693 INFO:teuthology.orchestra.run.smithi032.stdout: "replayed": 7 2024-04-25T01:43:38.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.693 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_mem": { 2024-04-25T01:43:38.693 INFO:teuthology.orchestra.run.smithi032.stdout: "ino": 3, 2024-04-25T01:43:38.693 INFO:teuthology.orchestra.run.smithi032.stdout: "ino+": 3, 2024-04-25T01:43:38.693 INFO:teuthology.orchestra.run.smithi032.stdout: "ino-": 0, 2024-04-25T01:43:38.694 INFO:teuthology.orchestra.run.smithi032.stdout: "dir": 1, 2024-04-25T01:43:38.694 INFO:teuthology.orchestra.run.smithi032.stdout: "dir+": 1, 2024-04-25T01:43:38.694 INFO:teuthology.orchestra.run.smithi032.stdout: "dir-": 0, 2024-04-25T01:43:38.694 INFO:teuthology.orchestra.run.smithi032.stdout: "dn": 0, 2024-04-25T01:43:38.694 INFO:teuthology.orchestra.run.smithi032.stdout: "dn+": 0, 2024-04-25T01:43:38.694 INFO:teuthology.orchestra.run.smithi032.stdout: "dn-": 0, 2024-04-25T01:43:38.694 INFO:teuthology.orchestra.run.smithi032.stdout: "cap": 0, 2024-04-25T01:43:38.694 INFO:teuthology.orchestra.run.smithi032.stdout: "cap+": 0, 2024-04-25T01:43:38.694 INFO:teuthology.orchestra.run.smithi032.stdout: "cap-": 0, 2024-04-25T01:43:38.694 INFO:teuthology.orchestra.run.smithi032.stdout: "rss": 50160, 2024-04-25T01:43:38.694 INFO:teuthology.orchestra.run.smithi032.stdout: "heap": 215420 2024-04-25T01:43:38.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.694 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_server": { 2024-04-25T01:43:38.694 INFO:teuthology.orchestra.run.smithi032.stdout: "dispatch_client_request": 0, 2024-04-25T01:43:38.695 INFO:teuthology.orchestra.run.smithi032.stdout: "dispatch_server_request": 0, 2024-04-25T01:43:38.695 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_request": 0, 2024-04-25T01:43:38.695 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_session": 0, 2024-04-25T01:43:38.695 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_peer_request": 0, 2024-04-25T01:43:38.695 INFO:teuthology.orchestra.run.smithi032.stdout: "req_create_latency": { 2024-04-25T01:43:38.695 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.695 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.695 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.695 INFO:teuthology.orchestra.run.smithi032.stdout: "req_getattr_latency": { 2024-04-25T01:43:38.695 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.695 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.695 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.695 INFO:teuthology.orchestra.run.smithi032.stdout: "req_getfilelock_latency": { 2024-04-25T01:43:38.696 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.696 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.696 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.696 INFO:teuthology.orchestra.run.smithi032.stdout: "req_link_latency": { 2024-04-25T01:43:38.696 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.696 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.696 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.696 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookup_latency": { 2024-04-25T01:43:38.696 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.696 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.696 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.696 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookuphash_latency": { 2024-04-25T01:43:38.697 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.697 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.697 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.697 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupino_latency": { 2024-04-25T01:43:38.697 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.697 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.697 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.697 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupname_latency": { 2024-04-25T01:43:38.697 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.697 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.697 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.697 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupparent_latency": { 2024-04-25T01:43:38.697 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.698 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.698 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.698 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupsnap_latency": { 2024-04-25T01:43:38.698 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.698 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.698 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.698 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lssnap_latency": { 2024-04-25T01:43:38.698 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.698 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.698 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.699 INFO:teuthology.orchestra.run.smithi032.stdout: "req_mkdir_latency": { 2024-04-25T01:43:38.699 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.699 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.699 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.699 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.699 INFO:teuthology.orchestra.run.smithi032.stdout: "req_mknod_latency": { 2024-04-25T01:43:38.700 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.700 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.700 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.700 INFO:teuthology.orchestra.run.smithi032.stdout: "req_mksnap_latency": { 2024-04-25T01:43:38.700 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.700 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.700 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.700 INFO:teuthology.orchestra.run.smithi032.stdout: "req_open_latency": { 2024-04-25T01:43:38.700 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.700 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.700 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.700 INFO:teuthology.orchestra.run.smithi032.stdout: "req_readdir_latency": { 2024-04-25T01:43:38.701 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.701 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.701 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.701 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rename_latency": { 2024-04-25T01:43:38.701 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.701 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.701 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.701 INFO:teuthology.orchestra.run.smithi032.stdout: "req_renamesnap_latency": { 2024-04-25T01:43:38.701 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.701 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.701 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.701 INFO:teuthology.orchestra.run.smithi032.stdout: "req_snapdiff_latency": { 2024-04-25T01:43:38.702 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.702 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.702 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.702 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rmdir_latency": { 2024-04-25T01:43:38.702 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.702 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.702 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.702 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rmsnap_latency": { 2024-04-25T01:43:38.702 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.702 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.702 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.703 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rmxattr_latency": { 2024-04-25T01:43:38.703 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.703 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.703 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.703 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setattr_latency": { 2024-04-25T01:43:38.703 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.703 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.703 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.703 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setdirlayout_latency": { 2024-04-25T01:43:38.703 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.703 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.703 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.703 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setfilelock_latency": { 2024-04-25T01:43:38.704 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.704 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.704 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.704 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setlayout_latency": { 2024-04-25T01:43:38.704 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.704 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.704 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.704 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setxattr_latency": { 2024-04-25T01:43:38.704 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.704 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.704 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.704 INFO:teuthology.orchestra.run.smithi032.stdout: "req_symlink_latency": { 2024-04-25T01:43:38.705 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.705 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.705 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.705 INFO:teuthology.orchestra.run.smithi032.stdout: "req_unlink_latency": { 2024-04-25T01:43:38.705 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.705 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.705 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.705 INFO:teuthology.orchestra.run.smithi032.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:43:38.705 INFO:teuthology.orchestra.run.smithi032.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:43:38.705 INFO:teuthology.orchestra.run.smithi032.stdout: "req_getvxattr_latency": { 2024-04-25T01:43:38.706 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.706 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.706 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.706 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:38.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.706 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_sessions": { 2024-04-25T01:43:38.706 INFO:teuthology.orchestra.run.smithi032.stdout: "session_count": 1, 2024-04-25T01:43:38.706 INFO:teuthology.orchestra.run.smithi032.stdout: "session_add": 3, 2024-04-25T01:43:38.706 INFO:teuthology.orchestra.run.smithi032.stdout: "session_remove": 2, 2024-04-25T01:43:38.706 INFO:teuthology.orchestra.run.smithi032.stdout: "sessions_open": 3, 2024-04-25T01:43:38.706 INFO:teuthology.orchestra.run.smithi032.stdout: "sessions_stale": 0, 2024-04-25T01:43:38.706 INFO:teuthology.orchestra.run.smithi032.stdout: "total_load": 0, 2024-04-25T01:43:38.707 INFO:teuthology.orchestra.run.smithi032.stdout: "average_load": 0, 2024-04-25T01:43:38.707 INFO:teuthology.orchestra.run.smithi032.stdout: "avg_session_uptime": 4194, 2024-04-25T01:43:38.707 INFO:teuthology.orchestra.run.smithi032.stdout: "mdthresh_evicted": 0 2024-04-25T01:43:38.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.707 INFO:teuthology.orchestra.run.smithi032.stdout: "mempool": { 2024-04-25T01:43:38.707 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:43:38.707 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_items": 0, 2024-04-25T01:43:38.707 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:43:38.707 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:43:38.707 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:43:38.707 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:43:38.707 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:43:38.707 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:43:38.708 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:43:38.708 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:43:38.708 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:43:38.708 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:43:38.708 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:43:38.708 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:43:38.708 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:43:38.708 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_items": 0, 2024-04-25T01:43:38.708 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:43:38.708 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_items": 0, 2024-04-25T01:43:38.708 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:43:38.708 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:43:38.708 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:43:38.709 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:43:38.709 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:43:38.709 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:43:38.709 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:43:38.709 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_items": 0, 2024-04-25T01:43:38.709 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:43:38.709 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:43:38.709 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:43:38.709 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_items": 0, 2024-04-25T01:43:38.709 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_bytes": 0, 2024-04-25T01:43:38.709 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_items": 0, 2024-04-25T01:43:38.709 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:43:38.709 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:43:38.709 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:43:38.709 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:43:38.710 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_bytes": 190184, 2024-04-25T01:43:38.710 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_items": 53, 2024-04-25T01:43:38.710 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:43:38.710 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_items": 0, 2024-04-25T01:43:38.710 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bytes": 0, 2024-04-25T01:43:38.710 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_items": 0, 2024-04-25T01:43:38.710 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:43:38.710 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_items": 0, 2024-04-25T01:43:38.710 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:43:38.710 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_items": 0, 2024-04-25T01:43:38.710 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_bytes": 9320, 2024-04-25T01:43:38.710 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_items": 178, 2024-04-25T01:43:38.710 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:43:38.710 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:43:38.710 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_bytes": 0, 2024-04-25T01:43:38.711 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_items": 0, 2024-04-25T01:43:38.711 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_bytes": 8464, 2024-04-25T01:43:38.711 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_items": 13, 2024-04-25T01:43:38.711 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_bytes": 0, 2024-04-25T01:43:38.711 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_items": 0, 2024-04-25T01:43:38.711 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_bytes": 0, 2024-04-25T01:43:38.711 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_items": 0 2024-04-25T01:43:38.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.711 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter": { 2024-04-25T01:43:38.711 INFO:teuthology.orchestra.run.smithi032.stdout: "op_active": 0, 2024-04-25T01:43:38.711 INFO:teuthology.orchestra.run.smithi032.stdout: "op_laggy": 0, 2024-04-25T01:43:38.711 INFO:teuthology.orchestra.run.smithi032.stdout: "op_send": 2160, 2024-04-25T01:43:38.711 INFO:teuthology.orchestra.run.smithi032.stdout: "op_send_bytes": 21, 2024-04-25T01:43:38.711 INFO:teuthology.orchestra.run.smithi032.stdout: "op_resend": 3, 2024-04-25T01:43:38.711 INFO:teuthology.orchestra.run.smithi032.stdout: "op_reply": 2160, 2024-04-25T01:43:38.712 INFO:teuthology.orchestra.run.smithi032.stdout: "op_latency": { 2024-04-25T01:43:38.712 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 2160, 2024-04-25T01:43:38.712 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 1.430971040, 2024-04-25T01:43:38.712 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000662486 2024-04-25T01:43:38.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.712 INFO:teuthology.orchestra.run.smithi032.stdout: "op_inflight": 0, 2024-04-25T01:43:38.712 INFO:teuthology.orchestra.run.smithi032.stdout: "oplen_avg": { 2024-04-25T01:43:38.712 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 2160, 2024-04-25T01:43:38.712 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 2161 2024-04-25T01:43:38.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.712 INFO:teuthology.orchestra.run.smithi032.stdout: "op": 2160, 2024-04-25T01:43:38.712 INFO:teuthology.orchestra.run.smithi032.stdout: "op_r": 2160, 2024-04-25T01:43:38.712 INFO:teuthology.orchestra.run.smithi032.stdout: "op_w": 0, 2024-04-25T01:43:38.712 INFO:teuthology.orchestra.run.smithi032.stdout: "op_rmw": 0, 2024-04-25T01:43:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: "op_pg": 0, 2024-04-25T01:43:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_stat": 1395, 2024-04-25T01:43:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_create": 0, 2024-04-25T01:43:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_read": 763, 2024-04-25T01:43:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_write": 0, 2024-04-25T01:43:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_writefull": 0, 2024-04-25T01:43:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_writesame": 0, 2024-04-25T01:43:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_append": 0, 2024-04-25T01:43:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_zero": 0, 2024-04-25T01:43:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_truncate": 0, 2024-04-25T01:43:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_delete": 0, 2024-04-25T01:43:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_mapext": 0, 2024-04-25T01:43:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_sparse_read": 0, 2024-04-25T01:43:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_clonerange": 0, 2024-04-25T01:43:38.713 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_getxattr": 1, 2024-04-25T01:43:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_setxattr": 0, 2024-04-25T01:43:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:43:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_rmxattr": 0, 2024-04-25T01:43:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:43:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_call": 0, 2024-04-25T01:43:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_watch": 0, 2024-04-25T01:43:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_notify": 0, 2024-04-25T01:43:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:43:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_pgls": 0, 2024-04-25T01:43:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:43:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_other": 0, 2024-04-25T01:43:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_active": 0, 2024-04-25T01:43:38.714 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_send": 0, 2024-04-25T01:43:38.715 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_resend": 0, 2024-04-25T01:43:38.715 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_ping": 0, 2024-04-25T01:43:38.715 INFO:teuthology.orchestra.run.smithi032.stdout: "poolop_active": 0, 2024-04-25T01:43:38.715 INFO:teuthology.orchestra.run.smithi032.stdout: "poolop_send": 0, 2024-04-25T01:43:38.715 INFO:teuthology.orchestra.run.smithi032.stdout: "poolop_resend": 0, 2024-04-25T01:43:38.715 INFO:teuthology.orchestra.run.smithi032.stdout: "poolstat_active": 0, 2024-04-25T01:43:38.715 INFO:teuthology.orchestra.run.smithi032.stdout: "poolstat_send": 0, 2024-04-25T01:43:38.715 INFO:teuthology.orchestra.run.smithi032.stdout: "poolstat_resend": 0, 2024-04-25T01:43:38.715 INFO:teuthology.orchestra.run.smithi032.stdout: "statfs_active": 0, 2024-04-25T01:43:38.715 INFO:teuthology.orchestra.run.smithi032.stdout: "statfs_send": 0, 2024-04-25T01:43:38.715 INFO:teuthology.orchestra.run.smithi032.stdout: "statfs_resend": 0, 2024-04-25T01:43:38.715 INFO:teuthology.orchestra.run.smithi032.stdout: "command_active": 0, 2024-04-25T01:43:38.715 INFO:teuthology.orchestra.run.smithi032.stdout: "command_send": 0, 2024-04-25T01:43:38.716 INFO:teuthology.orchestra.run.smithi032.stdout: "command_resend": 0, 2024-04-25T01:43:38.716 INFO:teuthology.orchestra.run.smithi032.stdout: "map_epoch": 102, 2024-04-25T01:43:38.716 INFO:teuthology.orchestra.run.smithi032.stdout: "map_full": 0, 2024-04-25T01:43:38.716 INFO:teuthology.orchestra.run.smithi032.stdout: "map_inc": 11, 2024-04-25T01:43:38.716 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_sessions": 5, 2024-04-25T01:43:38.716 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_session_open": 5, 2024-04-25T01:43:38.716 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_session_close": 0, 2024-04-25T01:43:38.716 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_laggy": 0, 2024-04-25T01:43:38.716 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_wr": 0, 2024-04-25T01:43:38.716 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_rd": 2, 2024-04-25T01:43:38.716 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_del": 0 2024-04-25T01:43:38.716 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.716 INFO:teuthology.orchestra.run.smithi032.stdout: "oft": { 2024-04-25T01:43:38.716 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_objs": 0, 2024-04-25T01:43:38.717 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:43:38.717 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_updates": 0, 2024-04-25T01:43:38.717 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_removes": 0 2024-04-25T01:43:38.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.717 INFO:teuthology.orchestra.run.smithi032.stdout: "osd-slow-ops": { 2024-04-25T01:43:38.717 INFO:teuthology.orchestra.run.smithi032.stdout: "slow_ops_count": 0 2024-04-25T01:43:38.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.717 INFO:teuthology.orchestra.run.smithi032.stdout: "purge_queue": { 2024-04-25T01:43:38.717 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing_ops": 0, 2024-04-25T01:43:38.717 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T01:43:38.717 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing": 0, 2024-04-25T01:43:38.717 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing_high_water": 0, 2024-04-25T01:43:38.717 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executed_ops": 0, 2024-04-25T01:43:38.717 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executed": 0, 2024-04-25T01:43:38.717 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_item_in_journal": 0 2024-04-25T01:43:38.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.718 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:43:38.718 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:38.718 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 104857600, 2024-04-25T01:43:38.718 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:38.718 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 2409, 2024-04-25T01:43:38.718 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 1862333, 2024-04-25T01:43:38.718 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:38.718 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 2409, 2024-04-25T01:43:38.718 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-25T01:43:38.718 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-25T01:43:38.718 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 2409, 2024-04-25T01:43:38.718 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 1862333, 2024-04-25T01:43:38.718 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:38.718 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.718 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.719 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.719 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:38.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.719 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-objecter_bytes": { 2024-04-25T01:43:38.719 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:38.719 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 104857600, 2024-04-25T01:43:38.719 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:38.719 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-25T01:43:38.719 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-25T01:43:38.719 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:38.719 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:38.719 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 2160, 2024-04-25T01:43:38.719 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 1965, 2024-04-25T01:43:38.719 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 2160, 2024-04-25T01:43:38.719 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 1965, 2024-04-25T01:43:38.719 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:38.720 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.720 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.720 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.720 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:38.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.720 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-objecter_ops": { 2024-04-25T01:43:38.720 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:38.720 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 1024, 2024-04-25T01:43:38.720 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:38.720 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-25T01:43:38.720 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-25T01:43:38.720 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:38.720 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:38.720 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 2160, 2024-04-25T01:43:38.720 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 2160, 2024-04-25T01:43:38.720 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 2160, 2024-04-25T01:43:38.721 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 2160, 2024-04-25T01:43:38.721 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:38.721 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.721 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.721 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.721 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:38.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.721 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:43:38.721 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:38.721 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 3758096384, 2024-04-25T01:43:38.721 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:38.721 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-25T01:43:38.721 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-25T01:43:38.721 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:38.721 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:38.722 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-25T01:43:38.722 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-25T01:43:38.722 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 0, 2024-04-25T01:43:38.722 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 0, 2024-04-25T01:43:38.722 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:38.722 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.722 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.722 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.722 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:38.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-25T01:43:38.722 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-write_buf_throttle-0x55b875be06e0": { 2024-04-25T01:43:38.722 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-25T01:43:38.722 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 3758096384, 2024-04-25T01:43:38.722 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-25T01:43:38.723 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-25T01:43:38.723 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-25T01:43:38.723 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:38.723 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:38.723 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-25T01:43:38.723 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-25T01:43:38.723 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 0, 2024-04-25T01:43:38.723 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 0, 2024-04-25T01:43:38.723 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-25T01:43:38.723 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-25T01:43:38.723 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-25T01:43:38.724 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-25T01:43:38.724 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:38.724 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-25T01:43:38.724 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-25T01:43:39.582 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-04-25T01:43:39.582 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-04-25T01:43:39.582 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-04-25T01:43:39.582 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-04-25T01:43:39.583 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T01:43:39.583 DEBUG:tasks.check_counter:Getting stats from c 2024-04-25T01:43:39.583 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-25T01:43:39.833 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:43:40.222 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:40 smithi130 ceph-mon[32702]: pgmap v330: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:43:40.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:40 smithi032 ceph-mon[32461]: pgmap v330: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:43:40.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:40 smithi028 ceph-mon[29390]: pgmap v330: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:43:41.668 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-04-25T01:43:41.668 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:43:41.668 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 83, 2024-04-25T01:43:41.668 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 113, 2024-04-25T01:43:41.668 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 28455, 2024-04-25T01:43:41.668 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 39528, 2024-04-25T01:43:41.669 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 10, 2024-04-25T01:43:41.669 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 4, 2024-04-25T01:43:41.669 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.020718572, 2024-04-25T01:43:41.669 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.005974742, 2024-04-25T01:43:41.669 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.008812784, 2024-04-25T01:43:41.669 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.000649191, 2024-04-25T01:43:41.669 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:41.669 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 111, 2024-04-25T01:43:41.669 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.019446772, 2024-04-25T01:43:41.669 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000175196 2024-04-25T01:43:41.669 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.669 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:41.669 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 125, 2024-04-25T01:43:41.669 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000029628, 2024-04-25T01:43:41.669 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000237 2024-04-25T01:43:41.670 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.670 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:43:41.670 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:43:41.670 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.670 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:43:41.670 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 524, 2024-04-25T01:43:41.670 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 707, 2024-04-25T01:43:41.670 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 146799, 2024-04-25T01:43:41.670 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 956586, 2024-04-25T01:43:41.670 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 59, 2024-04-25T01:43:41.670 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 5, 2024-04-25T01:43:41.670 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.082335019, 2024-04-25T01:43:41.670 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.027626911, 2024-04-25T01:43:41.671 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.047899156, 2024-04-25T01:43:41.671 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.005030421, 2024-04-25T01:43:41.671 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:41.671 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 705, 2024-04-25T01:43:41.671 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.052095883, 2024-04-25T01:43:41.671 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000073894 2024-04-25T01:43:41.671 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.671 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:41.671 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1112, 2024-04-25T01:43:41.671 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000392749, 2024-04-25T01:43:41.671 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000353 2024-04-25T01:43:41.671 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.671 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T01:43:41.671 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 128544 2024-04-25T01:43:41.671 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.671 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:43:41.672 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 343, 2024-04-25T01:43:41.672 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 456, 2024-04-25T01:43:41.672 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 1518411, 2024-04-25T01:43:41.672 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 535696, 2024-04-25T01:43:41.672 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 13, 2024-04-25T01:43:41.672 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 4, 2024-04-25T01:43:41.672 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.061724367, 2024-04-25T01:43:41.672 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.022865594, 2024-04-25T01:43:41.672 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.039535059, 2024-04-25T01:43:41.672 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.005937080, 2024-04-25T01:43:41.672 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:41.672 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 455, 2024-04-25T01:43:41.672 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.029651559, 2024-04-25T01:43:41.672 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000065168 2024-04-25T01:43:41.672 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.673 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:41.673 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 102, 2024-04-25T01:43:41.673 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000022993, 2024-04-25T01:43:41.673 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000225 2024-04-25T01:43:41.673 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.673 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 1495136, 2024-04-25T01:43:41.673 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 458512 2024-04-25T01:43:41.673 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.673 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2024-04-25T01:43:41.673 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 1, 2024-04-25T01:43:41.673 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2024-04-25T01:43:41.673 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.673 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-MDSRank": { 2024-04-25T01:43:41.673 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-04-25T01:43:41.673 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-04-25T01:43:41.674 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 46, 2024-04-25T01:43:41.674 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.001867690, 2024-04-25T01:43:41.674 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000040601 2024-04-25T01:43:41.674 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:41.674 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.674 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-PurgeQueue": { 2024-04-25T01:43:41.674 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-04-25T01:43:41.674 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-04-25T01:43:41.674 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2, 2024-04-25T01:43:41.674 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000011187, 2024-04-25T01:43:41.674 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000005593 2024-04-25T01:43:41.674 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:41.674 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.675 INFO:teuthology.orchestra.run.smithi130.stdout: "mds": { 2024-04-25T01:43:41.675 INFO:teuthology.orchestra.run.smithi130.stdout: "request": 1, 2024-04-25T01:43:41.675 INFO:teuthology.orchestra.run.smithi130.stdout: "reply": 1, 2024-04-25T01:43:41.675 INFO:teuthology.orchestra.run.smithi130.stdout: "reply_latency": { 2024-04-25T01:43:41.675 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1, 2024-04-25T01:43:41.675 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000113845, 2024-04-25T01:43:41.675 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000113845 2024-04-25T01:43:41.675 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.675 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_reply": 0, 2024-04-25T01:43:41.675 INFO:teuthology.orchestra.run.smithi130.stdout: "forward": 0, 2024-04-25T01:43:41.675 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_complete": 0, 2024-04-25T01:43:41.675 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_keys": 0, 2024-04-25T01:43:41.675 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_commit": 11, 2024-04-25T01:43:41.675 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_split": 0, 2024-04-25T01:43:41.675 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_merge": 0, 2024-04-25T01:43:41.676 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes": 10, 2024-04-25T01:43:41.676 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_top": 0, 2024-04-25T01:43:41.676 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_bottom": 10, 2024-04-25T01:43:41.676 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pin_tail": 0, 2024-04-25T01:43:41.676 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pinned": 10, 2024-04-25T01:43:41.676 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_expired": 0, 2024-04-25T01:43:41.676 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_with_caps": 1, 2024-04-25T01:43:41.676 INFO:teuthology.orchestra.run.smithi130.stdout: "caps": 1, 2024-04-25T01:43:41.676 INFO:teuthology.orchestra.run.smithi130.stdout: "subtrees": 1, 2024-04-25T01:43:41.676 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse": 0, 2024-04-25T01:43:41.676 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_hit": 0, 2024-04-25T01:43:41.676 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_forward": 0, 2024-04-25T01:43:41.676 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_discover": 0, 2024-04-25T01:43:41.676 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:43:41.676 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_remote_ino": 0, 2024-04-25T01:43:41.677 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_lock": 0, 2024-04-25T01:43:41.677 INFO:teuthology.orchestra.run.smithi130.stdout: "load_cent": 0, 2024-04-25T01:43:41.677 INFO:teuthology.orchestra.run.smithi130.stdout: "q": 0, 2024-04-25T01:43:41.677 INFO:teuthology.orchestra.run.smithi130.stdout: "exported": 0, 2024-04-25T01:43:41.677 INFO:teuthology.orchestra.run.smithi130.stdout: "exported_inodes": 0, 2024-04-25T01:43:41.677 INFO:teuthology.orchestra.run.smithi130.stdout: "imported": 0, 2024-04-25T01:43:41.677 INFO:teuthology.orchestra.run.smithi130.stdout: "imported_inodes": 0, 2024-04-25T01:43:41.677 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_dir_fetch": 0, 2024-04-25T01:43:41.677 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_backtrace_fetch": 0, 2024-04-25T01:43:41.677 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_peer_discover": 0, 2024-04-25T01:43:41.677 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rfiles": 4, 2024-04-25T01:43:41.677 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rbytes": 234, 2024-04-25T01:43:41.677 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rsnaps": 6, 2024-04-25T01:43:41.677 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T01:43:41.677 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_set_tag": 0, 2024-04-25T01:43:41.678 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:43:41.678 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:43:41.678 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:43:41.678 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:43:41.678 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:43:41.678 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_file_inodes": 0, 2024-04-25T01:43:41.678 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:43:41.678 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T01:43:41.678 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_grant": 0, 2024-04-25T01:43:41.678 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:43:41.678 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:43:41.678 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:43:41.678 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps": 0, 2024-04-25T01:43:41.678 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps_dirty": 0, 2024-04-25T01:43:41.679 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_cap_release": 0, 2024-04-25T01:43:41.679 INFO:teuthology.orchestra.run.smithi130.stdout: "process_request_cap_release": 0 2024-04-25T01:43:41.679 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.679 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_cache": { 2024-04-25T01:43:41.679 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update": 0, 2024-04-25T01:43:41.679 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update_receipt": 0, 2024-04-25T01:43:41.679 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_try_discover": 0, 2024-04-25T01:43:41.679 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_send_discover": 1, 2024-04-25T01:43:41.679 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_handle_discover": 0, 2024-04-25T01:43:41.679 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays": 0, 2024-04-25T01:43:41.679 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_delayed": 0, 2024-04-25T01:43:41.679 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:43:41.679 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_created": 0, 2024-04-25T01:43:41.679 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_enqueued": 0, 2024-04-25T01:43:41.679 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_reintegrated": 0, 2024-04-25T01:43:41.680 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_migrated": 0, 2024-04-25T01:43:41.680 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_processing": 0, 2024-04-25T01:43:41.680 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:43:41.680 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:43:41.680 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_started": 0, 2024-04-25T01:43:41.680 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_completed": 0, 2024-04-25T01:43:41.680 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:43:41.680 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:43:41.680 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:43:41.680 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_exportdir": 0, 2024-04-25T01:43:41.680 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_flush": 0, 2024-04-25T01:43:41.680 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:43:41.680 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragstats": 0, 2024-04-25T01:43:41.680 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_inodestats": 0 2024-04-25T01:43:41.681 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.681 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_log": { 2024-04-25T01:43:41.681 INFO:teuthology.orchestra.run.smithi130.stdout: "evlrg": 0, 2024-04-25T01:43:41.681 INFO:teuthology.orchestra.run.smithi130.stdout: "evadd": 5, 2024-04-25T01:43:41.681 INFO:teuthology.orchestra.run.smithi130.stdout: "evex": 0, 2024-04-25T01:43:41.681 INFO:teuthology.orchestra.run.smithi130.stdout: "evtrm": 0, 2024-04-25T01:43:41.681 INFO:teuthology.orchestra.run.smithi130.stdout: "ev": 5, 2024-04-25T01:43:41.681 INFO:teuthology.orchestra.run.smithi130.stdout: "evexg": 0, 2024-04-25T01:43:41.681 INFO:teuthology.orchestra.run.smithi130.stdout: "evexd": 0, 2024-04-25T01:43:41.681 INFO:teuthology.orchestra.run.smithi130.stdout: "segadd": 2, 2024-04-25T01:43:41.681 INFO:teuthology.orchestra.run.smithi130.stdout: "segex": 0, 2024-04-25T01:43:41.681 INFO:teuthology.orchestra.run.smithi130.stdout: "segtrm": 0, 2024-04-25T01:43:41.681 INFO:teuthology.orchestra.run.smithi130.stdout: "seg": 2, 2024-04-25T01:43:41.681 INFO:teuthology.orchestra.run.smithi130.stdout: "segmjr": 2, 2024-04-25T01:43:41.681 INFO:teuthology.orchestra.run.smithi130.stdout: "segexg": 0, 2024-04-25T01:43:41.682 INFO:teuthology.orchestra.run.smithi130.stdout: "segexd": 0, 2024-04-25T01:43:41.682 INFO:teuthology.orchestra.run.smithi130.stdout: "expos": 4194304, 2024-04-25T01:43:41.682 INFO:teuthology.orchestra.run.smithi130.stdout: "wrpos": 4195732, 2024-04-25T01:43:41.682 INFO:teuthology.orchestra.run.smithi130.stdout: "rdpos": 0, 2024-04-25T01:43:41.682 INFO:teuthology.orchestra.run.smithi130.stdout: "jlat": { 2024-04-25T01:43:41.682 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 4, 2024-04-25T01:43:41.682 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.007150849, 2024-04-25T01:43:41.682 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001787712 2024-04-25T01:43:41.682 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.682 INFO:teuthology.orchestra.run.smithi130.stdout: "replayed": 0 2024-04-25T01:43:41.682 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.682 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_mem": { 2024-04-25T01:43:41.682 INFO:teuthology.orchestra.run.smithi130.stdout: "ino": 13, 2024-04-25T01:43:41.683 INFO:teuthology.orchestra.run.smithi130.stdout: "ino+": 13, 2024-04-25T01:43:41.683 INFO:teuthology.orchestra.run.smithi130.stdout: "ino-": 0, 2024-04-25T01:43:41.683 INFO:teuthology.orchestra.run.smithi130.stdout: "dir": 11, 2024-04-25T01:43:41.683 INFO:teuthology.orchestra.run.smithi130.stdout: "dir+": 12, 2024-04-25T01:43:41.683 INFO:teuthology.orchestra.run.smithi130.stdout: "dir-": 1, 2024-04-25T01:43:41.683 INFO:teuthology.orchestra.run.smithi130.stdout: "dn": 10, 2024-04-25T01:43:41.683 INFO:teuthology.orchestra.run.smithi130.stdout: "dn+": 10, 2024-04-25T01:43:41.683 INFO:teuthology.orchestra.run.smithi130.stdout: "dn-": 0, 2024-04-25T01:43:41.683 INFO:teuthology.orchestra.run.smithi130.stdout: "cap": 1, 2024-04-25T01:43:41.683 INFO:teuthology.orchestra.run.smithi130.stdout: "cap+": 1, 2024-04-25T01:43:41.683 INFO:teuthology.orchestra.run.smithi130.stdout: "cap-": 0, 2024-04-25T01:43:41.683 INFO:teuthology.orchestra.run.smithi130.stdout: "rss": 54976, 2024-04-25T01:43:41.683 INFO:teuthology.orchestra.run.smithi130.stdout: "heap": 223612 2024-04-25T01:43:41.683 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.683 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_server": { 2024-04-25T01:43:41.684 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_client_request": 1, 2024-04-25T01:43:41.684 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_server_request": 0, 2024-04-25T01:43:41.684 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_request": 1, 2024-04-25T01:43:41.684 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_session": 13, 2024-04-25T01:43:41.684 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_peer_request": 0, 2024-04-25T01:43:41.684 INFO:teuthology.orchestra.run.smithi130.stdout: "req_create_latency": { 2024-04-25T01:43:41.684 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.684 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.684 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.684 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.684 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getattr_latency": { 2024-04-25T01:43:41.684 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.684 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.684 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.684 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.685 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getfilelock_latency": { 2024-04-25T01:43:41.685 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.685 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.685 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.685 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.685 INFO:teuthology.orchestra.run.smithi130.stdout: "req_link_latency": { 2024-04-25T01:43:41.685 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.685 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.685 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.685 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.685 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookup_latency": { 2024-04-25T01:43:41.685 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.685 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.685 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.685 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.686 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookuphash_latency": { 2024-04-25T01:43:41.686 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.686 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.686 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.686 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.686 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupino_latency": { 2024-04-25T01:43:41.686 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1, 2024-04-25T01:43:41.686 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000113845, 2024-04-25T01:43:41.686 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000113845 2024-04-25T01:43:41.686 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.686 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupname_latency": { 2024-04-25T01:43:41.686 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.686 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.686 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.687 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.687 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupparent_latency": { 2024-04-25T01:43:41.687 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.687 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.687 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.687 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.687 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupsnap_latency": { 2024-04-25T01:43:41.687 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.687 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.687 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.687 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.687 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lssnap_latency": { 2024-04-25T01:43:41.687 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.687 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.687 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.688 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.688 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mkdir_latency": { 2024-04-25T01:43:41.688 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.688 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.688 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.688 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.688 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mknod_latency": { 2024-04-25T01:43:41.688 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.688 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.688 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.689 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.689 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mksnap_latency": { 2024-04-25T01:43:41.689 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.689 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.689 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.689 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.689 INFO:teuthology.orchestra.run.smithi130.stdout: "req_open_latency": { 2024-04-25T01:43:41.689 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.689 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.689 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.689 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.689 INFO:teuthology.orchestra.run.smithi130.stdout: "req_readdir_latency": { 2024-04-25T01:43:41.689 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.689 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.689 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.690 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.690 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rename_latency": { 2024-04-25T01:43:41.690 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.690 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.690 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.690 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.690 INFO:teuthology.orchestra.run.smithi130.stdout: "req_renamesnap_latency": { 2024-04-25T01:43:41.690 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.690 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.690 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.690 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.690 INFO:teuthology.orchestra.run.smithi130.stdout: "req_snapdiff_latency": { 2024-04-25T01:43:41.690 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.691 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.691 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.691 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.691 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmdir_latency": { 2024-04-25T01:43:41.691 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.691 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.691 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.691 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.691 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmsnap_latency": { 2024-04-25T01:43:41.691 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.691 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.691 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.691 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.691 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmxattr_latency": { 2024-04-25T01:43:41.692 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.692 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.692 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.692 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.692 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setattr_latency": { 2024-04-25T01:43:41.692 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.692 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.692 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.692 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.692 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setdirlayout_latency": { 2024-04-25T01:43:41.692 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.692 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.692 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.692 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.693 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setfilelock_latency": { 2024-04-25T01:43:41.693 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.693 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.693 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.693 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.693 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setlayout_latency": { 2024-04-25T01:43:41.693 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.693 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.693 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.693 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.693 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setxattr_latency": { 2024-04-25T01:43:41.693 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.693 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.693 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.693 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.693 INFO:teuthology.orchestra.run.smithi130.stdout: "req_symlink_latency": { 2024-04-25T01:43:41.694 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.694 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.694 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.694 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.694 INFO:teuthology.orchestra.run.smithi130.stdout: "req_unlink_latency": { 2024-04-25T01:43:41.694 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.694 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.694 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.694 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.694 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:43:41.694 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:43:41.694 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getvxattr_latency": { 2024-04-25T01:43:41.694 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.695 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.695 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.695 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:41.695 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.695 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_sessions": { 2024-04-25T01:43:41.695 INFO:teuthology.orchestra.run.smithi130.stdout: "session_count": 1, 2024-04-25T01:43:41.695 INFO:teuthology.orchestra.run.smithi130.stdout: "session_add": 2, 2024-04-25T01:43:41.695 INFO:teuthology.orchestra.run.smithi130.stdout: "session_remove": 1, 2024-04-25T01:43:41.695 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_open": 1, 2024-04-25T01:43:41.695 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_stale": 0, 2024-04-25T01:43:41.695 INFO:teuthology.orchestra.run.smithi130.stdout: "total_load": 1, 2024-04-25T01:43:41.695 INFO:teuthology.orchestra.run.smithi130.stdout: "average_load": 1, 2024-04-25T01:43:41.695 INFO:teuthology.orchestra.run.smithi130.stdout: "avg_session_uptime": 630, 2024-04-25T01:43:41.695 INFO:teuthology.orchestra.run.smithi130.stdout: "mdthresh_evicted": 0 2024-04-25T01:43:41.695 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.696 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2024-04-25T01:43:41.696 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:43:41.696 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2024-04-25T01:43:41.696 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:43:41.696 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:43:41.696 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:43:41.696 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:43:41.696 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:43:41.696 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:43:41.696 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:43:41.696 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:43:41.696 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:43:41.696 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:43:41.697 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:43:41.697 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:43:41.697 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:43:41.697 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2024-04-25T01:43:41.697 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:43:41.697 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2024-04-25T01:43:41.697 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:43:41.697 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:43:41.697 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:43:41.697 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:43:41.697 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:43:41.697 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:43:41.697 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:43:41.697 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2024-04-25T01:43:41.697 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:43:41.697 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:43:41.698 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:43:41.698 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2024-04-25T01:43:41.698 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2024-04-25T01:43:41.698 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2024-04-25T01:43:41.698 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:43:41.698 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:43:41.698 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:43:41.698 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:43:41.698 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 254439, 2024-04-25T01:43:41.698 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 86, 2024-04-25T01:43:41.698 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:43:41.698 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 0, 2024-04-25T01:43:41.698 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2024-04-25T01:43:41.698 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2024-04-25T01:43:41.699 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:43:41.699 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2024-04-25T01:43:41.699 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:43:41.699 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2024-04-25T01:43:41.699 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 9320, 2024-04-25T01:43:41.699 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 178, 2024-04-25T01:43:41.699 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:43:41.699 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:43:41.699 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2024-04-25T01:43:41.699 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2024-04-25T01:43:41.699 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 51032, 2024-04-25T01:43:41.699 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 200, 2024-04-25T01:43:41.699 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2024-04-25T01:43:41.699 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2024-04-25T01:43:41.699 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2024-04-25T01:43:41.700 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2024-04-25T01:43:41.700 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.700 INFO:teuthology.orchestra.run.smithi130.stdout: "objecter": { 2024-04-25T01:43:41.700 INFO:teuthology.orchestra.run.smithi130.stdout: "op_active": 0, 2024-04-25T01:43:41.700 INFO:teuthology.orchestra.run.smithi130.stdout: "op_laggy": 0, 2024-04-25T01:43:41.700 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send": 40, 2024-04-25T01:43:41.700 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send_bytes": 11730, 2024-04-25T01:43:41.700 INFO:teuthology.orchestra.run.smithi130.stdout: "op_resend": 28, 2024-04-25T01:43:41.700 INFO:teuthology.orchestra.run.smithi130.stdout: "op_reply": 40, 2024-04-25T01:43:41.700 INFO:teuthology.orchestra.run.smithi130.stdout: "op_latency": { 2024-04-25T01:43:41.700 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 40, 2024-04-25T01:43:41.700 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.218995990, 2024-04-25T01:43:41.700 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.005474899 2024-04-25T01:43:41.700 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.700 INFO:teuthology.orchestra.run.smithi130.stdout: "op_inflight": 0, 2024-04-25T01:43:41.701 INFO:teuthology.orchestra.run.smithi130.stdout: "oplen_avg": { 2024-04-25T01:43:41.701 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 40, 2024-04-25T01:43:41.701 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 62 2024-04-25T01:43:41.701 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.701 INFO:teuthology.orchestra.run.smithi130.stdout: "op": 40, 2024-04-25T01:43:41.701 INFO:teuthology.orchestra.run.smithi130.stdout: "op_r": 0, 2024-04-25T01:43:41.701 INFO:teuthology.orchestra.run.smithi130.stdout: "op_w": 40, 2024-04-25T01:43:41.701 INFO:teuthology.orchestra.run.smithi130.stdout: "op_rmw": 0, 2024-04-25T01:43:41.701 INFO:teuthology.orchestra.run.smithi130.stdout: "op_pg": 0, 2024-04-25T01:43:41.701 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_stat": 0, 2024-04-25T01:43:41.701 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_create": 10, 2024-04-25T01:43:41.701 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_read": 0, 2024-04-25T01:43:41.701 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_write": 4, 2024-04-25T01:43:41.701 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writefull": 7, 2024-04-25T01:43:41.702 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writesame": 0, 2024-04-25T01:43:41.702 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_append": 0, 2024-04-25T01:43:41.702 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_zero": 0, 2024-04-25T01:43:41.702 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_truncate": 0, 2024-04-25T01:43:41.702 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_delete": 6, 2024-04-25T01:43:41.702 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_mapext": 0, 2024-04-25T01:43:41.702 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_sparse_read": 0, 2024-04-25T01:43:41.702 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_clonerange": 0, 2024-04-25T01:43:41.702 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_getxattr": 0, 2024-04-25T01:43:41.702 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_setxattr": 20, 2024-04-25T01:43:41.702 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:43:41.702 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_rmxattr": 0, 2024-04-25T01:43:41.702 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:43:41.702 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_call": 0, 2024-04-25T01:43:41.703 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_watch": 0, 2024-04-25T01:43:41.703 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_notify": 0, 2024-04-25T01:43:41.703 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:43:41.703 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls": 0, 2024-04-25T01:43:41.703 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:43:41.703 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_other": 0, 2024-04-25T01:43:41.703 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_active": 0, 2024-04-25T01:43:41.703 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_send": 0, 2024-04-25T01:43:41.703 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_resend": 0, 2024-04-25T01:43:41.703 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_ping": 0, 2024-04-25T01:43:41.703 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_active": 0, 2024-04-25T01:43:41.703 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_send": 0, 2024-04-25T01:43:41.703 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_resend": 0, 2024-04-25T01:43:41.703 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_active": 0, 2024-04-25T01:43:41.703 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_send": 0, 2024-04-25T01:43:41.703 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_resend": 0, 2024-04-25T01:43:41.704 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_active": 0, 2024-04-25T01:43:41.704 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_send": 0, 2024-04-25T01:43:41.704 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_resend": 0, 2024-04-25T01:43:41.704 INFO:teuthology.orchestra.run.smithi130.stdout: "command_active": 0, 2024-04-25T01:43:41.704 INFO:teuthology.orchestra.run.smithi130.stdout: "command_send": 0, 2024-04-25T01:43:41.704 INFO:teuthology.orchestra.run.smithi130.stdout: "command_resend": 0, 2024-04-25T01:43:41.704 INFO:teuthology.orchestra.run.smithi130.stdout: "map_epoch": 102, 2024-04-25T01:43:41.704 INFO:teuthology.orchestra.run.smithi130.stdout: "map_full": 0, 2024-04-25T01:43:41.704 INFO:teuthology.orchestra.run.smithi130.stdout: "map_inc": 11, 2024-04-25T01:43:41.704 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_sessions": 8, 2024-04-25T01:43:41.704 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_open": 8, 2024-04-25T01:43:41.704 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_close": 0, 2024-04-25T01:43:41.704 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_laggy": 0, 2024-04-25T01:43:41.704 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_wr": 14, 2024-04-25T01:43:41.704 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_rd": 0, 2024-04-25T01:43:41.705 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_del": 1 2024-04-25T01:43:41.705 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.705 INFO:teuthology.orchestra.run.smithi130.stdout: "oft": { 2024-04-25T01:43:41.705 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_objs": 1, 2024-04-25T01:43:41.705 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:43:41.705 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_updates": 0, 2024-04-25T01:43:41.705 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_removes": 0 2024-04-25T01:43:41.705 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.705 INFO:teuthology.orchestra.run.smithi130.stdout: "osd-slow-ops": { 2024-04-25T01:43:41.705 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_ops_count": 0 2024-04-25T01:43:41.705 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.705 INFO:teuthology.orchestra.run.smithi130.stdout: "purge_queue": { 2024-04-25T01:43:41.705 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops": 0, 2024-04-25T01:43:41.705 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T01:43:41.706 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing": 0, 2024-04-25T01:43:41.706 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_high_water": 0, 2024-04-25T01:43:41.706 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed_ops": 0, 2024-04-25T01:43:41.706 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed": 0, 2024-04-25T01:43:41.706 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_item_in_journal": 0 2024-04-25T01:43:41.706 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.706 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:43:41.706 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:41.706 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-04-25T01:43:41.706 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:41.706 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 946, 2024-04-25T01:43:41.706 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 1591501, 2024-04-25T01:43:41.706 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:41.706 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 946, 2024-04-25T01:43:41.707 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-25T01:43:41.707 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-25T01:43:41.707 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 946, 2024-04-25T01:43:41.707 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 1591501, 2024-04-25T01:43:41.707 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:41.707 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.707 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.707 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.707 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:41.707 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.707 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_bytes": { 2024-04-25T01:43:41.707 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:41.707 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-04-25T01:43:41.707 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:41.707 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-25T01:43:41.708 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-25T01:43:41.708 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:41.708 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:41.708 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 40, 2024-04-25T01:43:41.708 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 11730, 2024-04-25T01:43:41.708 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 40, 2024-04-25T01:43:41.708 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 11730, 2024-04-25T01:43:41.708 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:41.708 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.708 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.708 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.708 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:41.708 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.708 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_ops": { 2024-04-25T01:43:41.708 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:41.709 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 1024, 2024-04-25T01:43:41.709 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:41.709 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-25T01:43:41.709 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-25T01:43:41.709 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:41.709 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:41.709 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 40, 2024-04-25T01:43:41.709 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 40, 2024-04-25T01:43:41.709 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 40, 2024-04-25T01:43:41.709 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 40, 2024-04-25T01:43:41.709 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:41.709 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.709 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.709 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.709 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:41.710 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.710 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:43:41.710 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:41.710 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-04-25T01:43:41.710 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:41.710 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-25T01:43:41.710 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-25T01:43:41.710 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:41.710 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:41.710 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-25T01:43:41.710 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-25T01:43:41.710 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 0, 2024-04-25T01:43:41.710 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 0, 2024-04-25T01:43:41.710 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:41.711 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.711 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.711 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.711 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:41.711 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:41.711 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle-0x556bb5f5c500": { 2024-04-25T01:43:41.711 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:41.711 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-04-25T01:43:41.711 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:41.711 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 5, 2024-04-25T01:43:41.711 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 1428, 2024-04-25T01:43:41.711 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:41.711 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 5, 2024-04-25T01:43:41.711 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-25T01:43:41.711 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-25T01:43:41.712 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 4, 2024-04-25T01:43:41.712 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 1428, 2024-04-25T01:43:41.712 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:41.712 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:41.712 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:41.712 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:41.712 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:41.712 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:41.712 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-04-25T01:43:42.458 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:42 smithi130 ceph-mon[32702]: pgmap v331: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:42.488 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-25T01:43:42.488 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-25T01:43:42.488 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-04-25T01:43:42.488 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-25T01:43:42.488 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T01:43:42.488 DEBUG:tasks.check_counter:Getting stats from f 2024-04-25T01:43:42.488 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-25T01:43:42.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:42 smithi032 ceph-mon[32461]: pgmap v331: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:42.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:42 smithi028 ceph-mon[29390]: pgmap v331: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:42.734 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:43:44.307 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:44 smithi130 ceph-mon[32702]: pgmap v332: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:43:44.409 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:44 smithi032 ceph-mon[32461]: pgmap v332: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:43:44.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:44 smithi028 ceph-mon[29390]: pgmap v332: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:43:44.579 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-04-25T01:43:44.579 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:43:44.579 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 347, 2024-04-25T01:43:44.579 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 467, 2024-04-25T01:43:44.579 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 1513227, 2024-04-25T01:43:44.579 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 535162, 2024-04-25T01:43:44.579 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 14, 2024-04-25T01:43:44.579 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 5, 2024-04-25T01:43:44.579 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.066506231, 2024-04-25T01:43:44.579 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.022575322, 2024-04-25T01:43:44.579 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.041297432, 2024-04-25T01:43:44.580 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.006541613, 2024-04-25T01:43:44.580 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:44.580 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 462, 2024-04-25T01:43:44.580 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.037950151, 2024-04-25T01:43:44.580 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000082143 2024-04-25T01:43:44.580 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.580 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:44.580 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 94, 2024-04-25T01:43:44.580 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000045095, 2024-04-25T01:43:44.580 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000479 2024-04-25T01:43:44.580 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.580 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 1488848, 2024-04-25T01:43:44.580 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 461008 2024-04-25T01:43:44.580 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.580 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:43:44.581 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 474, 2024-04-25T01:43:44.581 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 667, 2024-04-25T01:43:44.581 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 132466, 2024-04-25T01:43:44.581 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 374002, 2024-04-25T01:43:44.581 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 12, 2024-04-25T01:43:44.581 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 5, 2024-04-25T01:43:44.581 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.067244847, 2024-04-25T01:43:44.581 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.026473552, 2024-04-25T01:43:44.581 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.035396217, 2024-04-25T01:43:44.581 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.005014581, 2024-04-25T01:43:44.581 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:44.581 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 669, 2024-04-25T01:43:44.581 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.053816159, 2024-04-25T01:43:44.581 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000080442 2024-04-25T01:43:44.581 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.581 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:44.582 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1082, 2024-04-25T01:43:44.582 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000330354, 2024-04-25T01:43:44.582 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000305 2024-04-25T01:43:44.582 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.582 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 1728, 2024-04-25T01:43:44.582 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 133872 2024-04-25T01:43:44.582 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.582 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:43:44.582 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 117, 2024-04-25T01:43:44.582 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 119, 2024-04-25T01:43:44.582 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 32850, 2024-04-25T01:43:44.582 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 592274, 2024-04-25T01:43:44.582 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 56, 2024-04-25T01:43:44.582 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 5, 2024-04-25T01:43:44.582 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.025895440, 2024-04-25T01:43:44.583 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.004277871, 2024-04-25T01:43:44.583 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.019789826, 2024-04-25T01:43:44.583 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.000448224, 2024-04-25T01:43:44.583 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:44.583 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 117, 2024-04-25T01:43:44.583 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.030860814, 2024-04-25T01:43:44.583 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000263767 2024-04-25T01:43:44.583 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.583 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:44.583 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 123, 2024-04-25T01:43:44.583 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000025504, 2024-04-25T01:43:44.583 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000207 2024-04-25T01:43:44.583 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.583 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:43:44.583 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:43:44.584 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.584 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2024-04-25T01:43:44.584 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 1, 2024-04-25T01:43:44.584 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2024-04-25T01:43:44.584 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.584 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-MDSRank": { 2024-04-25T01:43:44.584 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-04-25T01:43:44.584 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-04-25T01:43:44.584 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 39, 2024-04-25T01:43:44.584 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.001960463, 2024-04-25T01:43:44.584 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000050268 2024-04-25T01:43:44.584 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:44.584 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.584 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-PurgeQueue": { 2024-04-25T01:43:44.584 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-04-25T01:43:44.584 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-04-25T01:43:44.584 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2, 2024-04-25T01:43:44.585 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000008693, 2024-04-25T01:43:44.585 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000004346 2024-04-25T01:43:44.585 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:44.585 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.585 INFO:teuthology.orchestra.run.smithi130.stdout: "mds": { 2024-04-25T01:43:44.585 INFO:teuthology.orchestra.run.smithi130.stdout: "request": 0, 2024-04-25T01:43:44.585 INFO:teuthology.orchestra.run.smithi130.stdout: "reply": 0, 2024-04-25T01:43:44.585 INFO:teuthology.orchestra.run.smithi130.stdout: "reply_latency": { 2024-04-25T01:43:44.585 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.585 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.585 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.585 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.585 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_reply": 0, 2024-04-25T01:43:44.585 INFO:teuthology.orchestra.run.smithi130.stdout: "forward": 0, 2024-04-25T01:43:44.585 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_complete": 0, 2024-04-25T01:43:44.586 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_keys": 0, 2024-04-25T01:43:44.586 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_commit": 11, 2024-04-25T01:43:44.586 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_split": 0, 2024-04-25T01:43:44.586 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_merge": 0, 2024-04-25T01:43:44.586 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes": 10, 2024-04-25T01:43:44.586 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_top": 0, 2024-04-25T01:43:44.586 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_bottom": 10, 2024-04-25T01:43:44.586 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pin_tail": 0, 2024-04-25T01:43:44.586 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pinned": 10, 2024-04-25T01:43:44.586 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_expired": 0, 2024-04-25T01:43:44.586 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_with_caps": 0, 2024-04-25T01:43:44.586 INFO:teuthology.orchestra.run.smithi130.stdout: "caps": 0, 2024-04-25T01:43:44.586 INFO:teuthology.orchestra.run.smithi130.stdout: "subtrees": 1, 2024-04-25T01:43:44.586 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse": 0, 2024-04-25T01:43:44.587 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_hit": 0, 2024-04-25T01:43:44.587 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_forward": 0, 2024-04-25T01:43:44.587 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_discover": 0, 2024-04-25T01:43:44.587 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:43:44.587 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_remote_ino": 0, 2024-04-25T01:43:44.587 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_lock": 0, 2024-04-25T01:43:44.587 INFO:teuthology.orchestra.run.smithi130.stdout: "load_cent": 0, 2024-04-25T01:43:44.587 INFO:teuthology.orchestra.run.smithi130.stdout: "q": 0, 2024-04-25T01:43:44.587 INFO:teuthology.orchestra.run.smithi130.stdout: "exported": 0, 2024-04-25T01:43:44.587 INFO:teuthology.orchestra.run.smithi130.stdout: "exported_inodes": 0, 2024-04-25T01:43:44.587 INFO:teuthology.orchestra.run.smithi130.stdout: "imported": 0, 2024-04-25T01:43:44.587 INFO:teuthology.orchestra.run.smithi130.stdout: "imported_inodes": 0, 2024-04-25T01:43:44.587 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_dir_fetch": 0, 2024-04-25T01:43:44.587 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_backtrace_fetch": 0, 2024-04-25T01:43:44.588 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_peer_discover": 0, 2024-04-25T01:43:44.588 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rfiles": 4, 2024-04-25T01:43:44.588 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rbytes": 234, 2024-04-25T01:43:44.588 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rsnaps": 6, 2024-04-25T01:43:44.588 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T01:43:44.588 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_set_tag": 0, 2024-04-25T01:43:44.588 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:43:44.588 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:43:44.588 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:43:44.588 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:43:44.588 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:43:44.588 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_file_inodes": 0, 2024-04-25T01:43:44.588 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:43:44.588 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T01:43:44.588 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_grant": 0, 2024-04-25T01:43:44.588 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:43:44.589 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:43:44.589 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:43:44.589 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps": 0, 2024-04-25T01:43:44.589 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps_dirty": 0, 2024-04-25T01:43:44.589 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_cap_release": 0, 2024-04-25T01:43:44.589 INFO:teuthology.orchestra.run.smithi130.stdout: "process_request_cap_release": 0 2024-04-25T01:43:44.589 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.589 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_cache": { 2024-04-25T01:43:44.589 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update": 0, 2024-04-25T01:43:44.589 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update_receipt": 0, 2024-04-25T01:43:44.589 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_try_discover": 0, 2024-04-25T01:43:44.589 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_send_discover": 1, 2024-04-25T01:43:44.589 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_handle_discover": 0, 2024-04-25T01:43:44.589 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays": 0, 2024-04-25T01:43:44.589 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_delayed": 0, 2024-04-25T01:43:44.589 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:43:44.590 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_created": 0, 2024-04-25T01:43:44.590 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_enqueued": 0, 2024-04-25T01:43:44.590 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_reintegrated": 0, 2024-04-25T01:43:44.590 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_migrated": 0, 2024-04-25T01:43:44.590 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_processing": 0, 2024-04-25T01:43:44.590 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:43:44.590 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:43:44.590 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_started": 0, 2024-04-25T01:43:44.590 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_completed": 0, 2024-04-25T01:43:44.590 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:43:44.590 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:43:44.590 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:43:44.590 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_exportdir": 0, 2024-04-25T01:43:44.590 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_flush": 0, 2024-04-25T01:43:44.590 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:43:44.591 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragstats": 0, 2024-04-25T01:43:44.591 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_inodestats": 0 2024-04-25T01:43:44.591 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.591 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_log": { 2024-04-25T01:43:44.591 INFO:teuthology.orchestra.run.smithi130.stdout: "evlrg": 0, 2024-04-25T01:43:44.591 INFO:teuthology.orchestra.run.smithi130.stdout: "evadd": 3, 2024-04-25T01:43:44.591 INFO:teuthology.orchestra.run.smithi130.stdout: "evex": 0, 2024-04-25T01:43:44.591 INFO:teuthology.orchestra.run.smithi130.stdout: "evtrm": 0, 2024-04-25T01:43:44.591 INFO:teuthology.orchestra.run.smithi130.stdout: "ev": 3, 2024-04-25T01:43:44.591 INFO:teuthology.orchestra.run.smithi130.stdout: "evexg": 0, 2024-04-25T01:43:44.591 INFO:teuthology.orchestra.run.smithi130.stdout: "evexd": 0, 2024-04-25T01:43:44.591 INFO:teuthology.orchestra.run.smithi130.stdout: "segadd": 2, 2024-04-25T01:43:44.591 INFO:teuthology.orchestra.run.smithi130.stdout: "segex": 0, 2024-04-25T01:43:44.591 INFO:teuthology.orchestra.run.smithi130.stdout: "segtrm": 0, 2024-04-25T01:43:44.591 INFO:teuthology.orchestra.run.smithi130.stdout: "seg": 2, 2024-04-25T01:43:44.591 INFO:teuthology.orchestra.run.smithi130.stdout: "segmjr": 2, 2024-04-25T01:43:44.592 INFO:teuthology.orchestra.run.smithi130.stdout: "segexg": 0, 2024-04-25T01:43:44.592 INFO:teuthology.orchestra.run.smithi130.stdout: "segexd": 0, 2024-04-25T01:43:44.592 INFO:teuthology.orchestra.run.smithi130.stdout: "expos": 4194304, 2024-04-25T01:43:44.592 INFO:teuthology.orchestra.run.smithi130.stdout: "wrpos": 4195283, 2024-04-25T01:43:44.592 INFO:teuthology.orchestra.run.smithi130.stdout: "rdpos": 0, 2024-04-25T01:43:44.592 INFO:teuthology.orchestra.run.smithi130.stdout: "jlat": { 2024-04-25T01:43:44.592 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2, 2024-04-25T01:43:44.592 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.003440973, 2024-04-25T01:43:44.592 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001720486 2024-04-25T01:43:44.592 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.592 INFO:teuthology.orchestra.run.smithi130.stdout: "replayed": 0 2024-04-25T01:43:44.592 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.592 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_mem": { 2024-04-25T01:43:44.592 INFO:teuthology.orchestra.run.smithi130.stdout: "ino": 13, 2024-04-25T01:43:44.592 INFO:teuthology.orchestra.run.smithi130.stdout: "ino+": 13, 2024-04-25T01:43:44.592 INFO:teuthology.orchestra.run.smithi130.stdout: "ino-": 0, 2024-04-25T01:43:44.592 INFO:teuthology.orchestra.run.smithi130.stdout: "dir": 11, 2024-04-25T01:43:44.593 INFO:teuthology.orchestra.run.smithi130.stdout: "dir+": 12, 2024-04-25T01:43:44.593 INFO:teuthology.orchestra.run.smithi130.stdout: "dir-": 1, 2024-04-25T01:43:44.593 INFO:teuthology.orchestra.run.smithi130.stdout: "dn": 10, 2024-04-25T01:43:44.593 INFO:teuthology.orchestra.run.smithi130.stdout: "dn+": 10, 2024-04-25T01:43:44.593 INFO:teuthology.orchestra.run.smithi130.stdout: "dn-": 0, 2024-04-25T01:43:44.593 INFO:teuthology.orchestra.run.smithi130.stdout: "cap": 0, 2024-04-25T01:43:44.593 INFO:teuthology.orchestra.run.smithi130.stdout: "cap+": 0, 2024-04-25T01:43:44.593 INFO:teuthology.orchestra.run.smithi130.stdout: "cap-": 0, 2024-04-25T01:43:44.593 INFO:teuthology.orchestra.run.smithi130.stdout: "rss": 54468, 2024-04-25T01:43:44.593 INFO:teuthology.orchestra.run.smithi130.stdout: "heap": 223612 2024-04-25T01:43:44.593 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.593 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_server": { 2024-04-25T01:43:44.593 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_client_request": 0, 2024-04-25T01:43:44.593 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_server_request": 0, 2024-04-25T01:43:44.593 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_request": 0, 2024-04-25T01:43:44.594 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_session": 8, 2024-04-25T01:43:44.594 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_peer_request": 0, 2024-04-25T01:43:44.594 INFO:teuthology.orchestra.run.smithi130.stdout: "req_create_latency": { 2024-04-25T01:43:44.594 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.594 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.594 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.594 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.594 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getattr_latency": { 2024-04-25T01:43:44.594 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.594 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.594 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.594 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.594 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getfilelock_latency": { 2024-04-25T01:43:44.594 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.594 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.594 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.595 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.595 INFO:teuthology.orchestra.run.smithi130.stdout: "req_link_latency": { 2024-04-25T01:43:44.595 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.595 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.595 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.595 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.595 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookup_latency": { 2024-04-25T01:43:44.595 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.595 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.595 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.595 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.595 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookuphash_latency": { 2024-04-25T01:43:44.595 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.595 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.595 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.595 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.596 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupino_latency": { 2024-04-25T01:43:44.596 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.596 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.596 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.596 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.596 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupname_latency": { 2024-04-25T01:43:44.596 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.596 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.596 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.596 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.596 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupparent_latency": { 2024-04-25T01:43:44.596 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.596 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.596 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.596 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.596 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupsnap_latency": { 2024-04-25T01:43:44.597 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.597 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.597 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.597 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.597 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lssnap_latency": { 2024-04-25T01:43:44.597 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.597 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.597 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.597 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.597 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mkdir_latency": { 2024-04-25T01:43:44.597 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.598 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.598 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.598 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.598 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mknod_latency": { 2024-04-25T01:43:44.598 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.598 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.598 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.598 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.598 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mksnap_latency": { 2024-04-25T01:43:44.598 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.598 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.598 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.598 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.599 INFO:teuthology.orchestra.run.smithi130.stdout: "req_open_latency": { 2024-04-25T01:43:44.599 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.599 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.599 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.599 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.599 INFO:teuthology.orchestra.run.smithi130.stdout: "req_readdir_latency": { 2024-04-25T01:43:44.599 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.599 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.599 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.599 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.599 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rename_latency": { 2024-04-25T01:43:44.599 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.599 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.599 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.599 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.600 INFO:teuthology.orchestra.run.smithi130.stdout: "req_renamesnap_latency": { 2024-04-25T01:43:44.600 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.600 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.600 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.600 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.600 INFO:teuthology.orchestra.run.smithi130.stdout: "req_snapdiff_latency": { 2024-04-25T01:43:44.600 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.600 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.600 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.600 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.600 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmdir_latency": { 2024-04-25T01:43:44.600 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.600 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.600 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.600 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.600 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmsnap_latency": { 2024-04-25T01:43:44.601 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.601 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.601 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.601 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.601 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmxattr_latency": { 2024-04-25T01:43:44.601 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.601 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.601 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.601 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.601 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setattr_latency": { 2024-04-25T01:43:44.601 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.601 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.601 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.601 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.601 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setdirlayout_latency": { 2024-04-25T01:43:44.602 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.602 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.602 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.602 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.602 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setfilelock_latency": { 2024-04-25T01:43:44.602 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.602 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.602 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.602 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.602 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setlayout_latency": { 2024-04-25T01:43:44.602 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.602 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.602 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.602 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.603 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setxattr_latency": { 2024-04-25T01:43:44.603 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.603 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.603 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.603 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.603 INFO:teuthology.orchestra.run.smithi130.stdout: "req_symlink_latency": { 2024-04-25T01:43:44.603 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.603 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.603 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.603 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.603 INFO:teuthology.orchestra.run.smithi130.stdout: "req_unlink_latency": { 2024-04-25T01:43:44.603 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.603 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.603 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.603 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.604 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:43:44.604 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:43:44.604 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getvxattr_latency": { 2024-04-25T01:43:44.604 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.604 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.604 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.604 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:44.604 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.604 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_sessions": { 2024-04-25T01:43:44.604 INFO:teuthology.orchestra.run.smithi130.stdout: "session_count": 1, 2024-04-25T01:43:44.604 INFO:teuthology.orchestra.run.smithi130.stdout: "session_add": 1, 2024-04-25T01:43:44.604 INFO:teuthology.orchestra.run.smithi130.stdout: "session_remove": 0, 2024-04-25T01:43:44.604 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_open": 1, 2024-04-25T01:43:44.604 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_stale": 0, 2024-04-25T01:43:44.605 INFO:teuthology.orchestra.run.smithi130.stdout: "total_load": 0, 2024-04-25T01:43:44.605 INFO:teuthology.orchestra.run.smithi130.stdout: "average_load": 0, 2024-04-25T01:43:44.605 INFO:teuthology.orchestra.run.smithi130.stdout: "avg_session_uptime": 580, 2024-04-25T01:43:44.605 INFO:teuthology.orchestra.run.smithi130.stdout: "mdthresh_evicted": 0 2024-04-25T01:43:44.605 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.605 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2024-04-25T01:43:44.605 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:43:44.605 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2024-04-25T01:43:44.605 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:43:44.605 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:43:44.605 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:43:44.605 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:43:44.605 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:43:44.605 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:43:44.605 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:43:44.606 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:43:44.606 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:43:44.606 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:43:44.606 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:43:44.606 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:43:44.606 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:43:44.606 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2024-04-25T01:43:44.606 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:43:44.606 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2024-04-25T01:43:44.606 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:43:44.606 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:43:44.606 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:43:44.606 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:43:44.606 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:43:44.607 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:43:44.607 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:43:44.607 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2024-04-25T01:43:44.607 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:43:44.607 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:43:44.607 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:43:44.607 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2024-04-25T01:43:44.607 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2024-04-25T01:43:44.607 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2024-04-25T01:43:44.607 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:43:44.607 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:43:44.607 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:43:44.607 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:43:44.607 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 246087, 2024-04-25T01:43:44.607 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 79, 2024-04-25T01:43:44.608 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:43:44.608 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 0, 2024-04-25T01:43:44.608 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2024-04-25T01:43:44.608 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2024-04-25T01:43:44.608 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:43:44.608 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2024-04-25T01:43:44.608 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:43:44.608 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2024-04-25T01:43:44.608 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 9320, 2024-04-25T01:43:44.608 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 178, 2024-04-25T01:43:44.608 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:43:44.608 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:43:44.608 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2024-04-25T01:43:44.608 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2024-04-25T01:43:44.608 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 50664, 2024-04-25T01:43:44.609 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 198, 2024-04-25T01:43:44.609 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2024-04-25T01:43:44.609 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2024-04-25T01:43:44.609 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2024-04-25T01:43:44.609 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2024-04-25T01:43:44.609 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.609 INFO:teuthology.orchestra.run.smithi130.stdout: "objecter": { 2024-04-25T01:43:44.609 INFO:teuthology.orchestra.run.smithi130.stdout: "op_active": 0, 2024-04-25T01:43:44.609 INFO:teuthology.orchestra.run.smithi130.stdout: "op_laggy": 0, 2024-04-25T01:43:44.609 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send": 37, 2024-04-25T01:43:44.609 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send_bytes": 11191, 2024-04-25T01:43:44.609 INFO:teuthology.orchestra.run.smithi130.stdout: "op_resend": 33, 2024-04-25T01:43:44.609 INFO:teuthology.orchestra.run.smithi130.stdout: "op_reply": 37, 2024-04-25T01:43:44.609 INFO:teuthology.orchestra.run.smithi130.stdout: "op_latency": { 2024-04-25T01:43:44.609 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 37, 2024-04-25T01:43:44.610 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.221995952, 2024-04-25T01:43:44.610 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.005999890 2024-04-25T01:43:44.610 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.610 INFO:teuthology.orchestra.run.smithi130.stdout: "op_inflight": 0, 2024-04-25T01:43:44.610 INFO:teuthology.orchestra.run.smithi130.stdout: "oplen_avg": { 2024-04-25T01:43:44.610 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 37, 2024-04-25T01:43:44.610 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 59 2024-04-25T01:43:44.610 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.610 INFO:teuthology.orchestra.run.smithi130.stdout: "op": 37, 2024-04-25T01:43:44.610 INFO:teuthology.orchestra.run.smithi130.stdout: "op_r": 0, 2024-04-25T01:43:44.610 INFO:teuthology.orchestra.run.smithi130.stdout: "op_w": 37, 2024-04-25T01:43:44.610 INFO:teuthology.orchestra.run.smithi130.stdout: "op_rmw": 0, 2024-04-25T01:43:44.610 INFO:teuthology.orchestra.run.smithi130.stdout: "op_pg": 0, 2024-04-25T01:43:44.610 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_stat": 0, 2024-04-25T01:43:44.610 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_create": 10, 2024-04-25T01:43:44.611 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_read": 0, 2024-04-25T01:43:44.611 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_write": 2, 2024-04-25T01:43:44.611 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writefull": 6, 2024-04-25T01:43:44.611 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writesame": 0, 2024-04-25T01:43:44.611 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_append": 0, 2024-04-25T01:43:44.611 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_zero": 0, 2024-04-25T01:43:44.611 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_truncate": 0, 2024-04-25T01:43:44.611 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_delete": 6, 2024-04-25T01:43:44.611 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_mapext": 0, 2024-04-25T01:43:44.611 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_sparse_read": 0, 2024-04-25T01:43:44.611 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_clonerange": 0, 2024-04-25T01:43:44.611 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_getxattr": 0, 2024-04-25T01:43:44.611 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_setxattr": 20, 2024-04-25T01:43:44.611 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:43:44.612 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_rmxattr": 0, 2024-04-25T01:43:44.612 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:43:44.612 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_call": 0, 2024-04-25T01:43:44.612 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_watch": 0, 2024-04-25T01:43:44.612 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_notify": 0, 2024-04-25T01:43:44.612 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:43:44.612 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls": 0, 2024-04-25T01:43:44.612 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:43:44.612 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_other": 0, 2024-04-25T01:43:44.612 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_active": 0, 2024-04-25T01:43:44.612 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_send": 0, 2024-04-25T01:43:44.612 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_resend": 0, 2024-04-25T01:43:44.612 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_ping": 0, 2024-04-25T01:43:44.612 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_active": 0, 2024-04-25T01:43:44.612 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_send": 0, 2024-04-25T01:43:44.612 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_resend": 0, 2024-04-25T01:43:44.613 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_active": 0, 2024-04-25T01:43:44.613 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_send": 0, 2024-04-25T01:43:44.613 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_resend": 0, 2024-04-25T01:43:44.613 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_active": 0, 2024-04-25T01:43:44.613 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_send": 0, 2024-04-25T01:43:44.613 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_resend": 0, 2024-04-25T01:43:44.613 INFO:teuthology.orchestra.run.smithi130.stdout: "command_active": 0, 2024-04-25T01:43:44.613 INFO:teuthology.orchestra.run.smithi130.stdout: "command_send": 0, 2024-04-25T01:43:44.613 INFO:teuthology.orchestra.run.smithi130.stdout: "command_resend": 0, 2024-04-25T01:43:44.613 INFO:teuthology.orchestra.run.smithi130.stdout: "map_epoch": 102, 2024-04-25T01:43:44.613 INFO:teuthology.orchestra.run.smithi130.stdout: "map_full": 0, 2024-04-25T01:43:44.613 INFO:teuthology.orchestra.run.smithi130.stdout: "map_inc": 11, 2024-04-25T01:43:44.613 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_sessions": 9, 2024-04-25T01:43:44.613 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_open": 9, 2024-04-25T01:43:44.614 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_close": 0, 2024-04-25T01:43:44.614 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_laggy": 0, 2024-04-25T01:43:44.614 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_wr": 14, 2024-04-25T01:43:44.614 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_rd": 0, 2024-04-25T01:43:44.614 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_del": 1 2024-04-25T01:43:44.614 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.614 INFO:teuthology.orchestra.run.smithi130.stdout: "oft": { 2024-04-25T01:43:44.614 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_objs": 1, 2024-04-25T01:43:44.614 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:43:44.614 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_updates": 0, 2024-04-25T01:43:44.614 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_removes": 0 2024-04-25T01:43:44.614 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.614 INFO:teuthology.orchestra.run.smithi130.stdout: "osd-slow-ops": { 2024-04-25T01:43:44.614 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_ops_count": 0 2024-04-25T01:43:44.614 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.615 INFO:teuthology.orchestra.run.smithi130.stdout: "purge_queue": { 2024-04-25T01:43:44.615 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops": 0, 2024-04-25T01:43:44.615 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T01:43:44.615 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing": 0, 2024-04-25T01:43:44.615 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_high_water": 0, 2024-04-25T01:43:44.615 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed_ops": 0, 2024-04-25T01:43:44.615 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed": 0, 2024-04-25T01:43:44.615 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_item_in_journal": 0 2024-04-25T01:43:44.615 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.615 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:43:44.615 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:44.615 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-04-25T01:43:44.615 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:44.615 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 934, 2024-04-25T01:43:44.616 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 1577095, 2024-04-25T01:43:44.616 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:44.616 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 934, 2024-04-25T01:43:44.616 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-25T01:43:44.616 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-25T01:43:44.616 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 934, 2024-04-25T01:43:44.616 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 1577095, 2024-04-25T01:43:44.616 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:44.616 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.616 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.616 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.616 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:44.616 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.616 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_bytes": { 2024-04-25T01:43:44.616 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:44.616 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-04-25T01:43:44.617 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:44.617 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-25T01:43:44.617 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-25T01:43:44.617 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:44.617 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:44.617 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 37, 2024-04-25T01:43:44.617 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 11191, 2024-04-25T01:43:44.617 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 37, 2024-04-25T01:43:44.617 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 11191, 2024-04-25T01:43:44.617 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:44.617 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.617 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.617 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.617 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:44.618 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.618 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_ops": { 2024-04-25T01:43:44.618 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:44.618 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 1024, 2024-04-25T01:43:44.618 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:44.618 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-25T01:43:44.618 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-25T01:43:44.618 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:44.618 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:44.618 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 37, 2024-04-25T01:43:44.618 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 37, 2024-04-25T01:43:44.618 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 37, 2024-04-25T01:43:44.618 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 37, 2024-04-25T01:43:44.618 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:44.618 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.619 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.619 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.619 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:44.619 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.619 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:43:44.619 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:44.619 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-04-25T01:43:44.619 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:44.619 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-25T01:43:44.619 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-25T01:43:44.619 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:44.619 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:44.619 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-25T01:43:44.619 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-25T01:43:44.619 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 0, 2024-04-25T01:43:44.620 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 0, 2024-04-25T01:43:44.620 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:44.620 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.620 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.620 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.620 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:44.620 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:44.620 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle-0x559681300500": { 2024-04-25T01:43:44.620 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:44.620 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-04-25T01:43:44.620 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:44.620 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 3, 2024-04-25T01:43:44.620 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 979, 2024-04-25T01:43:44.620 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:44.620 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 3, 2024-04-25T01:43:44.620 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-25T01:43:44.621 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-25T01:43:44.621 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 2, 2024-04-25T01:43:44.621 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 979, 2024-04-25T01:43:44.621 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:44.621 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:44.621 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:44.621 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:44.621 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:44.621 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:44.621 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-04-25T01:43:45.480 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-25T01:43:45.480 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-25T01:43:45.480 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-04-25T01:43:45.480 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-25T01:43:45.480 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T01:43:45.480 DEBUG:tasks.check_counter:Getting stats from i 2024-04-25T01:43:45.481 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-25T01:43:45.721 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:43:46.138 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:46 smithi130 ceph-mon[32702]: pgmap v333: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:46.496 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:46 smithi032 ceph-mon[32461]: pgmap v333: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:46.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:46 smithi028 ceph-mon[29390]: pgmap v333: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:47.510 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-04-25T01:43:47.510 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:43:47.510 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 471, 2024-04-25T01:43:47.510 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 646, 2024-04-25T01:43:47.510 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 129082, 2024-04-25T01:43:47.511 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 255073, 2024-04-25T01:43:47.511 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 13, 2024-04-25T01:43:47.511 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 2, 2024-04-25T01:43:47.511 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.072152694, 2024-04-25T01:43:47.511 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.025986366, 2024-04-25T01:43:47.511 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.035139214, 2024-04-25T01:43:47.511 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.005543449, 2024-04-25T01:43:47.511 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:47.511 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 644, 2024-04-25T01:43:47.511 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.052576714, 2024-04-25T01:43:47.511 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000081640 2024-04-25T01:43:47.511 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.511 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:47.511 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 1052, 2024-04-25T01:43:47.511 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000320848, 2024-04-25T01:43:47.512 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000304 2024-04-25T01:43:47.512 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.512 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 3136, 2024-04-25T01:43:47.512 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 10912 2024-04-25T01:43:47.512 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.512 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:43:47.512 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 126, 2024-04-25T01:43:47.512 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 288, 2024-04-25T01:43:47.512 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 36647, 2024-04-25T01:43:47.512 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 1088666, 2024-04-25T01:43:47.512 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 64, 2024-04-25T01:43:47.512 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 5, 2024-04-25T01:43:47.512 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.036644475, 2024-04-25T01:43:47.513 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.009996016, 2024-04-25T01:43:47.513 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.023667226, 2024-04-25T01:43:47.513 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.000316409, 2024-04-25T01:43:47.513 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:47.513 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 286, 2024-04-25T01:43:47.513 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.030983342, 2024-04-25T01:43:47.513 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000108333 2024-04-25T01:43:47.513 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.513 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:47.513 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 166, 2024-04-25T01:43:47.513 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000045913, 2024-04-25T01:43:47.513 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000276 2024-04-25T01:43:47.513 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.513 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 5248, 2024-04-25T01:43:47.514 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 452336 2024-04-25T01:43:47.514 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.514 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:43:47.514 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 392, 2024-04-25T01:43:47.514 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 374, 2024-04-25T01:43:47.514 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 1545473, 2024-04-25T01:43:47.514 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 192641, 2024-04-25T01:43:47.514 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 7, 2024-04-25T01:43:47.514 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 6, 2024-04-25T01:43:47.514 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.060199201, 2024-04-25T01:43:47.514 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.020882239, 2024-04-25T01:43:47.514 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.042575524, 2024-04-25T01:43:47.514 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.005869250, 2024-04-25T01:43:47.514 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:47.515 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 373, 2024-04-25T01:43:47.515 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.054925434, 2024-04-25T01:43:47.515 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000147253 2024-04-25T01:43:47.515 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.515 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:47.515 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 200, 2024-04-25T01:43:47.515 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000046280, 2024-04-25T01:43:47.515 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000231 2024-04-25T01:43:47.515 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.515 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 1501056, 2024-04-25T01:43:47.515 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 141392 2024-04-25T01:43:47.515 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.515 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2024-04-25T01:43:47.515 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 1, 2024-04-25T01:43:47.516 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2024-04-25T01:43:47.516 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.516 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-MDSRank": { 2024-04-25T01:43:47.516 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-04-25T01:43:47.516 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-04-25T01:43:47.516 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 52, 2024-04-25T01:43:47.516 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.001517555, 2024-04-25T01:43:47.516 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000029183 2024-04-25T01:43:47.516 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:47.516 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.516 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-PurgeQueue": { 2024-04-25T01:43:47.516 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-04-25T01:43:47.516 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-04-25T01:43:47.516 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2, 2024-04-25T01:43:47.516 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000006715, 2024-04-25T01:43:47.517 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000003357 2024-04-25T01:43:47.517 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:47.517 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.517 INFO:teuthology.orchestra.run.smithi130.stdout: "mds": { 2024-04-25T01:43:47.517 INFO:teuthology.orchestra.run.smithi130.stdout: "request": 8, 2024-04-25T01:43:47.517 INFO:teuthology.orchestra.run.smithi130.stdout: "reply": 4, 2024-04-25T01:43:47.517 INFO:teuthology.orchestra.run.smithi130.stdout: "reply_latency": { 2024-04-25T01:43:47.517 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 4, 2024-04-25T01:43:47.517 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000865872, 2024-04-25T01:43:47.517 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000216468 2024-04-25T01:43:47.517 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.517 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_reply": 0, 2024-04-25T01:43:47.517 INFO:teuthology.orchestra.run.smithi130.stdout: "forward": 4, 2024-04-25T01:43:47.517 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_complete": 0, 2024-04-25T01:43:47.518 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_keys": 0, 2024-04-25T01:43:47.518 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_commit": 11, 2024-04-25T01:43:47.518 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_split": 0, 2024-04-25T01:43:47.518 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_merge": 0, 2024-04-25T01:43:47.518 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes": 11, 2024-04-25T01:43:47.518 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_top": 0, 2024-04-25T01:43:47.518 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_bottom": 11, 2024-04-25T01:43:47.518 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pin_tail": 0, 2024-04-25T01:43:47.518 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pinned": 10, 2024-04-25T01:43:47.518 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_expired": 0, 2024-04-25T01:43:47.518 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_with_caps": 1, 2024-04-25T01:43:47.518 INFO:teuthology.orchestra.run.smithi130.stdout: "caps": 1, 2024-04-25T01:43:47.518 INFO:teuthology.orchestra.run.smithi130.stdout: "subtrees": 2, 2024-04-25T01:43:47.518 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse": 12, 2024-04-25T01:43:47.519 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_hit": 8, 2024-04-25T01:43:47.519 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_forward": 0, 2024-04-25T01:43:47.519 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_discover": 1, 2024-04-25T01:43:47.519 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:43:47.519 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_remote_ino": 0, 2024-04-25T01:43:47.519 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_lock": 0, 2024-04-25T01:43:47.519 INFO:teuthology.orchestra.run.smithi130.stdout: "load_cent": 0, 2024-04-25T01:43:47.519 INFO:teuthology.orchestra.run.smithi130.stdout: "q": 0, 2024-04-25T01:43:47.519 INFO:teuthology.orchestra.run.smithi130.stdout: "exported": 0, 2024-04-25T01:43:47.519 INFO:teuthology.orchestra.run.smithi130.stdout: "exported_inodes": 0, 2024-04-25T01:43:47.519 INFO:teuthology.orchestra.run.smithi130.stdout: "imported": 0, 2024-04-25T01:43:47.519 INFO:teuthology.orchestra.run.smithi130.stdout: "imported_inodes": 0, 2024-04-25T01:43:47.519 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_dir_fetch": 0, 2024-04-25T01:43:47.519 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_backtrace_fetch": 0, 2024-04-25T01:43:47.520 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_peer_discover": 0, 2024-04-25T01:43:47.520 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rfiles": 4, 2024-04-25T01:43:47.520 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rbytes": 234, 2024-04-25T01:43:47.520 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rsnaps": 6, 2024-04-25T01:43:47.520 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T01:43:47.520 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_set_tag": 0, 2024-04-25T01:43:47.520 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:43:47.520 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:43:47.520 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:43:47.520 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:43:47.520 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:43:47.521 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_file_inodes": 0, 2024-04-25T01:43:47.521 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:43:47.521 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T01:43:47.521 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_grant": 0, 2024-04-25T01:43:47.521 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:43:47.521 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:43:47.521 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:43:47.521 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps": 0, 2024-04-25T01:43:47.521 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps_dirty": 0, 2024-04-25T01:43:47.521 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_cap_release": 0, 2024-04-25T01:43:47.521 INFO:teuthology.orchestra.run.smithi130.stdout: "process_request_cap_release": 2 2024-04-25T01:43:47.521 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.521 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_cache": { 2024-04-25T01:43:47.521 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update": 0, 2024-04-25T01:43:47.522 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update_receipt": 0, 2024-04-25T01:43:47.522 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_try_discover": 0, 2024-04-25T01:43:47.522 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_send_discover": 2, 2024-04-25T01:43:47.522 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_handle_discover": 0, 2024-04-25T01:43:47.522 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays": 0, 2024-04-25T01:43:47.522 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_delayed": 0, 2024-04-25T01:43:47.522 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:43:47.522 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_created": 0, 2024-04-25T01:43:47.522 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_enqueued": 0, 2024-04-25T01:43:47.522 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_reintegrated": 0, 2024-04-25T01:43:47.522 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_migrated": 0, 2024-04-25T01:43:47.522 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_processing": 0, 2024-04-25T01:43:47.522 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:43:47.522 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:43:47.522 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_started": 0, 2024-04-25T01:43:47.523 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_completed": 0, 2024-04-25T01:43:47.523 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:43:47.523 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:43:47.523 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:43:47.523 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_exportdir": 0, 2024-04-25T01:43:47.523 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_flush": 0, 2024-04-25T01:43:47.523 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:43:47.523 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragstats": 0, 2024-04-25T01:43:47.523 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_inodestats": 0 2024-04-25T01:43:47.523 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.523 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_log": { 2024-04-25T01:43:47.523 INFO:teuthology.orchestra.run.smithi130.stdout: "evlrg": 0, 2024-04-25T01:43:47.524 INFO:teuthology.orchestra.run.smithi130.stdout: "evadd": 5, 2024-04-25T01:43:47.524 INFO:teuthology.orchestra.run.smithi130.stdout: "evex": 0, 2024-04-25T01:43:47.524 INFO:teuthology.orchestra.run.smithi130.stdout: "evtrm": 0, 2024-04-25T01:43:47.524 INFO:teuthology.orchestra.run.smithi130.stdout: "ev": 5, 2024-04-25T01:43:47.524 INFO:teuthology.orchestra.run.smithi130.stdout: "evexg": 0, 2024-04-25T01:43:47.524 INFO:teuthology.orchestra.run.smithi130.stdout: "evexd": 0, 2024-04-25T01:43:47.524 INFO:teuthology.orchestra.run.smithi130.stdout: "segadd": 2, 2024-04-25T01:43:47.524 INFO:teuthology.orchestra.run.smithi130.stdout: "segex": 0, 2024-04-25T01:43:47.524 INFO:teuthology.orchestra.run.smithi130.stdout: "segtrm": 0, 2024-04-25T01:43:47.524 INFO:teuthology.orchestra.run.smithi130.stdout: "seg": 2, 2024-04-25T01:43:47.524 INFO:teuthology.orchestra.run.smithi130.stdout: "segmjr": 2, 2024-04-25T01:43:47.524 INFO:teuthology.orchestra.run.smithi130.stdout: "segexg": 0, 2024-04-25T01:43:47.524 INFO:teuthology.orchestra.run.smithi130.stdout: "segexd": 0, 2024-04-25T01:43:47.524 INFO:teuthology.orchestra.run.smithi130.stdout: "expos": 4194304, 2024-04-25T01:43:47.525 INFO:teuthology.orchestra.run.smithi130.stdout: "wrpos": 4195815, 2024-04-25T01:43:47.525 INFO:teuthology.orchestra.run.smithi130.stdout: "rdpos": 0, 2024-04-25T01:43:47.525 INFO:teuthology.orchestra.run.smithi130.stdout: "jlat": { 2024-04-25T01:43:47.525 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 4, 2024-04-25T01:43:47.525 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.007434632, 2024-04-25T01:43:47.525 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.001858658 2024-04-25T01:43:47.525 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.525 INFO:teuthology.orchestra.run.smithi130.stdout: "replayed": 0 2024-04-25T01:43:47.525 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.525 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_mem": { 2024-04-25T01:43:47.525 INFO:teuthology.orchestra.run.smithi130.stdout: "ino": 14, 2024-04-25T01:43:47.525 INFO:teuthology.orchestra.run.smithi130.stdout: "ino+": 14, 2024-04-25T01:43:47.525 INFO:teuthology.orchestra.run.smithi130.stdout: "ino-": 0, 2024-04-25T01:43:47.525 INFO:teuthology.orchestra.run.smithi130.stdout: "dir": 12, 2024-04-25T01:43:47.526 INFO:teuthology.orchestra.run.smithi130.stdout: "dir+": 13, 2024-04-25T01:43:47.526 INFO:teuthology.orchestra.run.smithi130.stdout: "dir-": 1, 2024-04-25T01:43:47.526 INFO:teuthology.orchestra.run.smithi130.stdout: "dn": 11, 2024-04-25T01:43:47.526 INFO:teuthology.orchestra.run.smithi130.stdout: "dn+": 11, 2024-04-25T01:43:47.526 INFO:teuthology.orchestra.run.smithi130.stdout: "dn-": 0, 2024-04-25T01:43:47.526 INFO:teuthology.orchestra.run.smithi130.stdout: "cap": 1, 2024-04-25T01:43:47.526 INFO:teuthology.orchestra.run.smithi130.stdout: "cap+": 2, 2024-04-25T01:43:47.526 INFO:teuthology.orchestra.run.smithi130.stdout: "cap-": 1, 2024-04-25T01:43:47.526 INFO:teuthology.orchestra.run.smithi130.stdout: "rss": 53108, 2024-04-25T01:43:47.526 INFO:teuthology.orchestra.run.smithi130.stdout: "heap": 223612 2024-04-25T01:43:47.526 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.526 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_server": { 2024-04-25T01:43:47.526 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_client_request": 9, 2024-04-25T01:43:47.527 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_server_request": 0, 2024-04-25T01:43:47.527 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_request": 8, 2024-04-25T01:43:47.527 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_session": 12, 2024-04-25T01:43:47.527 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_peer_request": 0, 2024-04-25T01:43:47.527 INFO:teuthology.orchestra.run.smithi130.stdout: "req_create_latency": { 2024-04-25T01:43:47.527 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.527 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.527 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.527 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.527 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getattr_latency": { 2024-04-25T01:43:47.527 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 4, 2024-04-25T01:43:47.527 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000865872, 2024-04-25T01:43:47.527 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000216468 2024-04-25T01:43:47.527 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.527 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getfilelock_latency": { 2024-04-25T01:43:47.528 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.528 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.528 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.528 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.528 INFO:teuthology.orchestra.run.smithi130.stdout: "req_link_latency": { 2024-04-25T01:43:47.528 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.528 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.528 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.528 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.528 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookup_latency": { 2024-04-25T01:43:47.528 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.528 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.528 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.528 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.529 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookuphash_latency": { 2024-04-25T01:43:47.529 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.529 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.529 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.529 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.529 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupino_latency": { 2024-04-25T01:43:47.529 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.529 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.529 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.529 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.529 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupname_latency": { 2024-04-25T01:43:47.529 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.529 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.529 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.529 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.530 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupparent_latency": { 2024-04-25T01:43:47.530 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.530 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.530 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.530 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.530 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupsnap_latency": { 2024-04-25T01:43:47.530 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.530 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.530 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.530 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.530 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lssnap_latency": { 2024-04-25T01:43:47.530 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.530 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.530 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.530 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.531 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mkdir_latency": { 2024-04-25T01:43:47.531 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.531 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.531 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.531 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.532 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mknod_latency": { 2024-04-25T01:43:47.532 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.532 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.532 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.532 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.532 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mksnap_latency": { 2024-04-25T01:43:47.532 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.532 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.532 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.532 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.532 INFO:teuthology.orchestra.run.smithi130.stdout: "req_open_latency": { 2024-04-25T01:43:47.532 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.532 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.532 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.533 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.533 INFO:teuthology.orchestra.run.smithi130.stdout: "req_readdir_latency": { 2024-04-25T01:43:47.533 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.533 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.533 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.533 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.533 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rename_latency": { 2024-04-25T01:43:47.533 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.533 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.533 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.533 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.533 INFO:teuthology.orchestra.run.smithi130.stdout: "req_renamesnap_latency": { 2024-04-25T01:43:47.533 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.533 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.533 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.534 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.534 INFO:teuthology.orchestra.run.smithi130.stdout: "req_snapdiff_latency": { 2024-04-25T01:43:47.534 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.534 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.534 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.534 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.534 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmdir_latency": { 2024-04-25T01:43:47.534 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.534 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.534 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.534 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.534 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmsnap_latency": { 2024-04-25T01:43:47.534 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.534 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.535 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.535 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.535 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmxattr_latency": { 2024-04-25T01:43:47.535 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.535 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.535 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.535 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.535 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setattr_latency": { 2024-04-25T01:43:47.535 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.535 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.535 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.535 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.535 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setdirlayout_latency": { 2024-04-25T01:43:47.535 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.535 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.536 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.536 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.536 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setfilelock_latency": { 2024-04-25T01:43:47.536 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.536 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.536 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.536 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.536 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setlayout_latency": { 2024-04-25T01:43:47.536 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.536 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.536 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.536 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.536 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setxattr_latency": { 2024-04-25T01:43:47.536 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.536 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.537 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.537 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.537 INFO:teuthology.orchestra.run.smithi130.stdout: "req_symlink_latency": { 2024-04-25T01:43:47.537 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.537 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.537 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.537 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.537 INFO:teuthology.orchestra.run.smithi130.stdout: "req_unlink_latency": { 2024-04-25T01:43:47.537 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.537 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.537 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.537 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.537 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:43:47.537 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:43:47.537 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getvxattr_latency": { 2024-04-25T01:43:47.538 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.538 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.538 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.538 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:47.538 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.538 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_sessions": { 2024-04-25T01:43:47.538 INFO:teuthology.orchestra.run.smithi130.stdout: "session_count": 1, 2024-04-25T01:43:47.538 INFO:teuthology.orchestra.run.smithi130.stdout: "session_add": 2, 2024-04-25T01:43:47.538 INFO:teuthology.orchestra.run.smithi130.stdout: "session_remove": 1, 2024-04-25T01:43:47.538 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_open": 1, 2024-04-25T01:43:47.538 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_stale": 0, 2024-04-25T01:43:47.538 INFO:teuthology.orchestra.run.smithi130.stdout: "total_load": 3, 2024-04-25T01:43:47.538 INFO:teuthology.orchestra.run.smithi130.stdout: "average_load": 3, 2024-04-25T01:43:47.538 INFO:teuthology.orchestra.run.smithi130.stdout: "avg_session_uptime": 584, 2024-04-25T01:43:47.539 INFO:teuthology.orchestra.run.smithi130.stdout: "mdthresh_evicted": 0 2024-04-25T01:43:47.539 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.539 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2024-04-25T01:43:47.539 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:43:47.539 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2024-04-25T01:43:47.539 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:43:47.539 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:43:47.539 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:43:47.539 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:43:47.539 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:43:47.539 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:43:47.539 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:43:47.539 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:43:47.539 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:43:47.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:43:47.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:43:47.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:43:47.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:43:47.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2024-04-25T01:43:47.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:43:47.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2024-04-25T01:43:47.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:43:47.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:43:47.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:43:47.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:43:47.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:43:47.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:43:47.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:43:47.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2024-04-25T01:43:47.540 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:43:47.541 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:43:47.541 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:43:47.541 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2024-04-25T01:43:47.541 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2024-04-25T01:43:47.541 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2024-04-25T01:43:47.541 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:43:47.541 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:43:47.541 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:43:47.541 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:43:47.541 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 280295, 2024-04-25T01:43:47.541 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 105, 2024-04-25T01:43:47.541 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 88, 2024-04-25T01:43:47.541 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 1, 2024-04-25T01:43:47.541 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2024-04-25T01:43:47.542 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2024-04-25T01:43:47.542 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:43:47.542 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2024-04-25T01:43:47.542 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:43:47.542 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2024-04-25T01:43:47.542 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 9320, 2024-04-25T01:43:47.542 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 178, 2024-04-25T01:43:47.542 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:43:47.542 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:43:47.542 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2024-04-25T01:43:47.542 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2024-04-25T01:43:47.542 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 55216, 2024-04-25T01:43:47.542 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 214, 2024-04-25T01:43:47.542 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2024-04-25T01:43:47.542 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2024-04-25T01:43:47.543 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2024-04-25T01:43:47.543 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2024-04-25T01:43:47.543 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.543 INFO:teuthology.orchestra.run.smithi130.stdout: "objecter": { 2024-04-25T01:43:47.543 INFO:teuthology.orchestra.run.smithi130.stdout: "op_active": 0, 2024-04-25T01:43:47.543 INFO:teuthology.orchestra.run.smithi130.stdout: "op_laggy": 0, 2024-04-25T01:43:47.543 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send": 40, 2024-04-25T01:43:47.543 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send_bytes": 11813, 2024-04-25T01:43:47.543 INFO:teuthology.orchestra.run.smithi130.stdout: "op_resend": 33, 2024-04-25T01:43:47.543 INFO:teuthology.orchestra.run.smithi130.stdout: "op_reply": 40, 2024-04-25T01:43:47.543 INFO:teuthology.orchestra.run.smithi130.stdout: "op_latency": { 2024-04-25T01:43:47.543 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 40, 2024-04-25T01:43:47.543 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.294994605, 2024-04-25T01:43:47.543 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.007374865 2024-04-25T01:43:47.543 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.544 INFO:teuthology.orchestra.run.smithi130.stdout: "op_inflight": 0, 2024-04-25T01:43:47.544 INFO:teuthology.orchestra.run.smithi130.stdout: "oplen_avg": { 2024-04-25T01:43:47.544 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 40, 2024-04-25T01:43:47.544 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 62 2024-04-25T01:43:47.544 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.544 INFO:teuthology.orchestra.run.smithi130.stdout: "op": 40, 2024-04-25T01:43:47.544 INFO:teuthology.orchestra.run.smithi130.stdout: "op_r": 0, 2024-04-25T01:43:47.544 INFO:teuthology.orchestra.run.smithi130.stdout: "op_w": 40, 2024-04-25T01:43:47.544 INFO:teuthology.orchestra.run.smithi130.stdout: "op_rmw": 0, 2024-04-25T01:43:47.544 INFO:teuthology.orchestra.run.smithi130.stdout: "op_pg": 0, 2024-04-25T01:43:47.544 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_stat": 0, 2024-04-25T01:43:47.544 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_create": 10, 2024-04-25T01:43:47.544 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_read": 0, 2024-04-25T01:43:47.544 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_write": 4, 2024-04-25T01:43:47.544 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writefull": 7, 2024-04-25T01:43:47.545 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writesame": 0, 2024-04-25T01:43:47.545 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_append": 0, 2024-04-25T01:43:47.545 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_zero": 0, 2024-04-25T01:43:47.545 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_truncate": 0, 2024-04-25T01:43:47.545 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_delete": 6, 2024-04-25T01:43:47.545 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_mapext": 0, 2024-04-25T01:43:47.545 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_sparse_read": 0, 2024-04-25T01:43:47.545 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_clonerange": 0, 2024-04-25T01:43:47.545 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_getxattr": 0, 2024-04-25T01:43:47.545 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_setxattr": 20, 2024-04-25T01:43:47.545 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:43:47.545 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_rmxattr": 0, 2024-04-25T01:43:47.545 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:43:47.545 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_call": 0, 2024-04-25T01:43:47.545 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_watch": 0, 2024-04-25T01:43:47.546 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_notify": 0, 2024-04-25T01:43:47.546 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:43:47.546 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls": 0, 2024-04-25T01:43:47.546 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:43:47.546 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_other": 0, 2024-04-25T01:43:47.546 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_active": 0, 2024-04-25T01:43:47.546 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_send": 0, 2024-04-25T01:43:47.546 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_resend": 0, 2024-04-25T01:43:47.546 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_ping": 0, 2024-04-25T01:43:47.546 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_active": 0, 2024-04-25T01:43:47.546 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_send": 0, 2024-04-25T01:43:47.546 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_resend": 0, 2024-04-25T01:43:47.546 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_active": 0, 2024-04-25T01:43:47.546 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_send": 0, 2024-04-25T01:43:47.547 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_resend": 0, 2024-04-25T01:43:47.547 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_active": 0, 2024-04-25T01:43:47.547 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_send": 0, 2024-04-25T01:43:47.547 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_resend": 0, 2024-04-25T01:43:47.547 INFO:teuthology.orchestra.run.smithi130.stdout: "command_active": 0, 2024-04-25T01:43:47.547 INFO:teuthology.orchestra.run.smithi130.stdout: "command_send": 0, 2024-04-25T01:43:47.547 INFO:teuthology.orchestra.run.smithi130.stdout: "command_resend": 0, 2024-04-25T01:43:47.547 INFO:teuthology.orchestra.run.smithi130.stdout: "map_epoch": 102, 2024-04-25T01:43:47.547 INFO:teuthology.orchestra.run.smithi130.stdout: "map_full": 0, 2024-04-25T01:43:47.547 INFO:teuthology.orchestra.run.smithi130.stdout: "map_inc": 11, 2024-04-25T01:43:47.547 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_sessions": 9, 2024-04-25T01:43:47.547 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_open": 9, 2024-04-25T01:43:47.547 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_close": 0, 2024-04-25T01:43:47.547 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_laggy": 0, 2024-04-25T01:43:47.547 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_wr": 14, 2024-04-25T01:43:47.548 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_rd": 0, 2024-04-25T01:43:47.548 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_del": 1 2024-04-25T01:43:47.548 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.548 INFO:teuthology.orchestra.run.smithi130.stdout: "oft": { 2024-04-25T01:43:47.548 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_objs": 1, 2024-04-25T01:43:47.548 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:43:47.548 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_updates": 0, 2024-04-25T01:43:47.548 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_removes": 0 2024-04-25T01:43:47.548 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.548 INFO:teuthology.orchestra.run.smithi130.stdout: "osd-slow-ops": { 2024-04-25T01:43:47.548 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_ops_count": 0 2024-04-25T01:43:47.548 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.548 INFO:teuthology.orchestra.run.smithi130.stdout: "purge_queue": { 2024-04-25T01:43:47.548 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops": 0, 2024-04-25T01:43:47.549 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T01:43:47.549 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing": 0, 2024-04-25T01:43:47.549 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_high_water": 0, 2024-04-25T01:43:47.549 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed_ops": 0, 2024-04-25T01:43:47.549 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed": 0, 2024-04-25T01:43:47.549 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_item_in_journal": 0 2024-04-25T01:43:47.549 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.549 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:43:47.549 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:47.549 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-04-25T01:43:47.549 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:47.549 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 985, 2024-04-25T01:43:47.549 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 1604406, 2024-04-25T01:43:47.549 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:47.549 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 985, 2024-04-25T01:43:47.550 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-25T01:43:47.550 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-25T01:43:47.550 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 985, 2024-04-25T01:43:47.550 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 1604406, 2024-04-25T01:43:47.550 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:47.550 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.550 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.550 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.550 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:47.550 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.550 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_bytes": { 2024-04-25T01:43:47.550 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:47.550 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-04-25T01:43:47.550 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:47.551 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-25T01:43:47.551 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-25T01:43:47.551 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:47.551 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:47.551 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 40, 2024-04-25T01:43:47.551 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 11813, 2024-04-25T01:43:47.551 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 40, 2024-04-25T01:43:47.551 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 11813, 2024-04-25T01:43:47.551 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:47.551 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.551 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.551 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.551 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:47.551 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.552 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_ops": { 2024-04-25T01:43:47.552 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:47.552 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 1024, 2024-04-25T01:43:47.552 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:47.552 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-25T01:43:47.552 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-25T01:43:47.552 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:47.552 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:47.552 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 40, 2024-04-25T01:43:47.552 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 40, 2024-04-25T01:43:47.552 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 40, 2024-04-25T01:43:47.552 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 40, 2024-04-25T01:43:47.552 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:47.552 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.552 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.553 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.553 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:47.553 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.553 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:43:47.553 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:47.553 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-04-25T01:43:47.553 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:47.553 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-25T01:43:47.553 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-25T01:43:47.553 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:47.553 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:47.553 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-25T01:43:47.553 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-25T01:43:47.553 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 0, 2024-04-25T01:43:47.554 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 0, 2024-04-25T01:43:47.554 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:47.554 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.554 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.554 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.554 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:47.554 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:47.554 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle-0x55f1b8774500": { 2024-04-25T01:43:47.554 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:47.554 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-04-25T01:43:47.554 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:47.554 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 5, 2024-04-25T01:43:47.554 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 1511, 2024-04-25T01:43:47.554 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:47.555 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 5, 2024-04-25T01:43:47.555 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-25T01:43:47.555 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-25T01:43:47.555 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 4, 2024-04-25T01:43:47.555 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 1511, 2024-04-25T01:43:47.555 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:47.555 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:47.555 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:47.555 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:47.555 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:47.555 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:47.555 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-04-25T01:43:48.357 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:48 smithi130 ceph-mon[32702]: pgmap v334: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:43:48.386 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-25T01:43:48.386 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-04-25T01:43:48.386 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-25T01:43:48.386 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-25T01:43:48.386 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T01:43:48.387 DEBUG:tasks.check_counter:Getting stats from l 2024-04-25T01:43:48.387 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-25T01:43:48.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:48 smithi032 ceph-mon[32461]: pgmap v334: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:43:48.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:48 smithi028 ceph-mon[29390]: pgmap v334: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T01:43:48.625 INFO:teuthology.orchestra.run.smithi130.stderr:Inferring config /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:43:50.377 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:50 smithi028 ceph-mon[29390]: pgmap v335: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:43:50.404 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:50 smithi130 ceph-mon[32702]: pgmap v335: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:43:50.450 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2024-04-25T01:43:50.450 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T01:43:50.450 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 2230, 2024-04-25T01:43:50.450 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 2232, 2024-04-25T01:43:50.451 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 632122, 2024-04-25T01:43:50.451 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 709147, 2024-04-25T01:43:50.451 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 6, 2024-04-25T01:43:50.451 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 2, 2024-04-25T01:43:50.451 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.205178236, 2024-04-25T01:43:50.451 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.066709412, 2024-04-25T01:43:50.451 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.050806625, 2024-04-25T01:43:50.451 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.072210607, 2024-04-25T01:43:50.451 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:50.451 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2230, 2024-04-25T01:43:50.451 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.093452496, 2024-04-25T01:43:50.451 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000041906 2024-04-25T01:43:50.451 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.451 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:50.452 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.452 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.452 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.452 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.452 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T01:43:50.452 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T01:43:50.452 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.452 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T01:43:50.452 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 4, 2024-04-25T01:43:50.452 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 173, 2024-04-25T01:43:50.452 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 876, 2024-04-25T01:43:50.452 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 450266, 2024-04-25T01:43:50.452 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 6, 2024-04-25T01:43:50.452 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 4, 2024-04-25T01:43:50.452 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.010859204, 2024-04-25T01:43:50.453 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.007055080, 2024-04-25T01:43:50.453 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.001949054, 2024-04-25T01:43:50.453 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.000060823, 2024-04-25T01:43:50.453 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:50.453 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 170, 2024-04-25T01:43:50.453 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.017278393, 2024-04-25T01:43:50.453 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000101637 2024-04-25T01:43:50.453 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.453 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:50.453 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.453 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.453 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.453 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.453 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T01:43:50.454 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 449568 2024-04-25T01:43:50.454 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.454 INFO:teuthology.orchestra.run.smithi130.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T01:43:50.454 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_messages": 256, 2024-04-25T01:43:50.454 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages": 243, 2024-04-25T01:43:50.454 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_bytes": 1481692, 2024-04-25T01:43:50.454 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_bytes": 141198, 2024-04-25T01:43:50.454 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_created_connections": 3, 2024-04-25T01:43:50.454 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_active_connections": 2, 2024-04-25T01:43:50.454 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_total_time": 0.049250296, 2024-04-25T01:43:50.454 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_send_time": 0.015305252, 2024-04-25T01:43:50.454 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_recv_time": 0.036698580, 2024-04-25T01:43:50.454 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_running_fast_dispatch_time": 0.006080452, 2024-04-25T01:43:50.454 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T01:43:50.455 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 243, 2024-04-25T01:43:50.455 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.014491633, 2024-04-25T01:43:50.455 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000059636 2024-04-25T01:43:50.455 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.455 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_handle_ack_lat": { 2024-04-25T01:43:50.455 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.455 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.455 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.455 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.455 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_recv_encrypted_bytes": 1481424, 2024-04-25T01:43:50.455 INFO:teuthology.orchestra.run.smithi130.stdout: "msgr_send_encrypted_bytes": 140880 2024-04-25T01:43:50.455 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.455 INFO:teuthology.orchestra.run.smithi130.stdout: "cct": { 2024-04-25T01:43:50.455 INFO:teuthology.orchestra.run.smithi130.stdout: "total_workers": 1, 2024-04-25T01:43:50.455 INFO:teuthology.orchestra.run.smithi130.stdout: "unhealthy_workers": 0 2024-04-25T01:43:50.456 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.456 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-MDSRank": { 2024-04-25T01:43:50.456 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-04-25T01:43:50.456 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-04-25T01:43:50.456 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 3767, 2024-04-25T01:43:50.456 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.112228734, 2024-04-25T01:43:50.456 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000029792 2024-04-25T01:43:50.456 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:50.456 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.456 INFO:teuthology.orchestra.run.smithi130.stdout: "finisher-PurgeQueue": { 2024-04-25T01:43:50.456 INFO:teuthology.orchestra.run.smithi130.stdout: "queue_len": 0, 2024-04-25T01:43:50.456 INFO:teuthology.orchestra.run.smithi130.stdout: "complete_latency": { 2024-04-25T01:43:50.456 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.456 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.457 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.457 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:50.457 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.457 INFO:teuthology.orchestra.run.smithi130.stdout: "mds": { 2024-04-25T01:43:50.457 INFO:teuthology.orchestra.run.smithi130.stdout: "request": 0, 2024-04-25T01:43:50.457 INFO:teuthology.orchestra.run.smithi130.stdout: "reply": 0, 2024-04-25T01:43:50.457 INFO:teuthology.orchestra.run.smithi130.stdout: "reply_latency": { 2024-04-25T01:43:50.457 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.457 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.457 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.457 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.457 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_reply": 0, 2024-04-25T01:43:50.457 INFO:teuthology.orchestra.run.smithi130.stdout: "forward": 0, 2024-04-25T01:43:50.458 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_complete": 0, 2024-04-25T01:43:50.458 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_fetch_keys": 0, 2024-04-25T01:43:50.458 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_commit": 0, 2024-04-25T01:43:50.458 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_split": 0, 2024-04-25T01:43:50.458 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_merge": 0, 2024-04-25T01:43:50.458 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes": 0, 2024-04-25T01:43:50.458 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_top": 0, 2024-04-25T01:43:50.458 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_bottom": 0, 2024-04-25T01:43:50.458 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pin_tail": 0, 2024-04-25T01:43:50.458 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_pinned": 0, 2024-04-25T01:43:50.458 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_expired": 0, 2024-04-25T01:43:50.459 INFO:teuthology.orchestra.run.smithi130.stdout: "inodes_with_caps": 0, 2024-04-25T01:43:50.459 INFO:teuthology.orchestra.run.smithi130.stdout: "caps": 0, 2024-04-25T01:43:50.459 INFO:teuthology.orchestra.run.smithi130.stdout: "subtrees": 1, 2024-04-25T01:43:50.459 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse": 0, 2024-04-25T01:43:50.459 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_hit": 0, 2024-04-25T01:43:50.459 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_forward": 0, 2024-04-25T01:43:50.459 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_discover": 0, 2024-04-25T01:43:50.459 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_dir_fetch": 0, 2024-04-25T01:43:50.459 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_remote_ino": 0, 2024-04-25T01:43:50.459 INFO:teuthology.orchestra.run.smithi130.stdout: "traverse_lock": 0, 2024-04-25T01:43:50.459 INFO:teuthology.orchestra.run.smithi130.stdout: "load_cent": 0, 2024-04-25T01:43:50.460 INFO:teuthology.orchestra.run.smithi130.stdout: "q": 0, 2024-04-25T01:43:50.460 INFO:teuthology.orchestra.run.smithi130.stdout: "exported": 0, 2024-04-25T01:43:50.460 INFO:teuthology.orchestra.run.smithi130.stdout: "exported_inodes": 0, 2024-04-25T01:43:50.460 INFO:teuthology.orchestra.run.smithi130.stdout: "imported": 0, 2024-04-25T01:43:50.460 INFO:teuthology.orchestra.run.smithi130.stdout: "imported_inodes": 0, 2024-04-25T01:43:50.460 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_dir_fetch": 0, 2024-04-25T01:43:50.460 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_backtrace_fetch": 0, 2024-04-25T01:43:50.460 INFO:teuthology.orchestra.run.smithi130.stdout: "openino_peer_discover": 0, 2024-04-25T01:43:50.460 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rfiles": 0, 2024-04-25T01:43:50.460 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rbytes": 0, 2024-04-25T01:43:50.460 INFO:teuthology.orchestra.run.smithi130.stdout: "root_rsnaps": 0, 2024-04-25T01:43:50.460 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T01:43:50.460 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_set_tag": 0, 2024-04-25T01:43:50.461 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T01:43:50.461 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_inotable_repaired": 0, 2024-04-25T01:43:50.461 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_inodes": 0, 2024-04-25T01:43:50.461 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T01:43:50.461 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T01:43:50.461 INFO:teuthology.orchestra.run.smithi130.stdout: "scrub_file_inodes": 0, 2024-04-25T01:43:50.461 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_inode_file_caps": 0, 2024-04-25T01:43:50.461 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T01:43:50.461 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_grant": 0, 2024-04-25T01:43:50.461 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T01:43:50.462 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T01:43:50.462 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T01:43:50.462 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps": 0, 2024-04-25T01:43:50.462 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_caps_dirty": 0, 2024-04-25T01:43:50.462 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_cap_release": 0, 2024-04-25T01:43:50.462 INFO:teuthology.orchestra.run.smithi130.stdout: "process_request_cap_release": 0 2024-04-25T01:43:50.462 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.462 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_cache": { 2024-04-25T01:43:50.462 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update": 0, 2024-04-25T01:43:50.462 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_update_receipt": 0, 2024-04-25T01:43:50.462 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_try_discover": 0, 2024-04-25T01:43:50.462 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_send_discover": 0, 2024-04-25T01:43:50.463 INFO:teuthology.orchestra.run.smithi130.stdout: "dir_handle_discover": 0, 2024-04-25T01:43:50.463 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays": 0, 2024-04-25T01:43:50.463 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_delayed": 0, 2024-04-25T01:43:50.463 INFO:teuthology.orchestra.run.smithi130.stdout: "num_strays_enqueuing": 0, 2024-04-25T01:43:50.463 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_created": 0, 2024-04-25T01:43:50.463 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_enqueued": 0, 2024-04-25T01:43:50.463 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_reintegrated": 0, 2024-04-25T01:43:50.463 INFO:teuthology.orchestra.run.smithi130.stdout: "strays_migrated": 0, 2024-04-25T01:43:50.463 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_processing": 0, 2024-04-25T01:43:50.463 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_enqueued": 0, 2024-04-25T01:43:50.463 INFO:teuthology.orchestra.run.smithi130.stdout: "num_recovering_prioritized": 0, 2024-04-25T01:43:50.463 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_started": 0, 2024-04-25T01:43:50.464 INFO:teuthology.orchestra.run.smithi130.stdout: "recovery_completed": 0, 2024-04-25T01:43:50.464 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_quiesce_path": 0, 2024-04-25T01:43:50.464 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_quiesce_inode": 0, 2024-04-25T01:43:50.464 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T01:43:50.464 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_exportdir": 0, 2024-04-25T01:43:50.464 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_flush": 0, 2024-04-25T01:43:50.464 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragmentdir": 0, 2024-04-25T01:43:50.464 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_fragstats": 0, 2024-04-25T01:43:50.464 INFO:teuthology.orchestra.run.smithi130.stdout: "ireq_inodestats": 0 2024-04-25T01:43:50.464 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.464 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_log": { 2024-04-25T01:43:50.464 INFO:teuthology.orchestra.run.smithi130.stdout: "evlrg": 0, 2024-04-25T01:43:50.464 INFO:teuthology.orchestra.run.smithi130.stdout: "evadd": 0, 2024-04-25T01:43:50.464 INFO:teuthology.orchestra.run.smithi130.stdout: "evex": 0, 2024-04-25T01:43:50.464 INFO:teuthology.orchestra.run.smithi130.stdout: "evtrm": 0, 2024-04-25T01:43:50.465 INFO:teuthology.orchestra.run.smithi130.stdout: "ev": 5, 2024-04-25T01:43:50.465 INFO:teuthology.orchestra.run.smithi130.stdout: "evexg": 0, 2024-04-25T01:43:50.465 INFO:teuthology.orchestra.run.smithi130.stdout: "evexd": 0, 2024-04-25T01:43:50.465 INFO:teuthology.orchestra.run.smithi130.stdout: "segadd": 0, 2024-04-25T01:43:50.465 INFO:teuthology.orchestra.run.smithi130.stdout: "segex": 0, 2024-04-25T01:43:50.465 INFO:teuthology.orchestra.run.smithi130.stdout: "segtrm": 0, 2024-04-25T01:43:50.465 INFO:teuthology.orchestra.run.smithi130.stdout: "seg": 2, 2024-04-25T01:43:50.465 INFO:teuthology.orchestra.run.smithi130.stdout: "segmjr": 2, 2024-04-25T01:43:50.465 INFO:teuthology.orchestra.run.smithi130.stdout: "segexg": 0, 2024-04-25T01:43:50.465 INFO:teuthology.orchestra.run.smithi130.stdout: "segexd": 0, 2024-04-25T01:43:50.465 INFO:teuthology.orchestra.run.smithi130.stdout: "expos": 4194304, 2024-04-25T01:43:50.465 INFO:teuthology.orchestra.run.smithi130.stdout: "wrpos": 4195815, 2024-04-25T01:43:50.465 INFO:teuthology.orchestra.run.smithi130.stdout: "rdpos": 4195424, 2024-04-25T01:43:50.465 INFO:teuthology.orchestra.run.smithi130.stdout: "jlat": { 2024-04-25T01:43:50.465 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.466 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.466 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.466 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.466 INFO:teuthology.orchestra.run.smithi130.stdout: "replayed": 5 2024-04-25T01:43:50.466 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.466 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_mem": { 2024-04-25T01:43:50.466 INFO:teuthology.orchestra.run.smithi130.stdout: "ino": 3, 2024-04-25T01:43:50.466 INFO:teuthology.orchestra.run.smithi130.stdout: "ino+": 3, 2024-04-25T01:43:50.466 INFO:teuthology.orchestra.run.smithi130.stdout: "ino-": 0, 2024-04-25T01:43:50.466 INFO:teuthology.orchestra.run.smithi130.stdout: "dir": 1, 2024-04-25T01:43:50.466 INFO:teuthology.orchestra.run.smithi130.stdout: "dir+": 1, 2024-04-25T01:43:50.466 INFO:teuthology.orchestra.run.smithi130.stdout: "dir-": 0, 2024-04-25T01:43:50.466 INFO:teuthology.orchestra.run.smithi130.stdout: "dn": 0, 2024-04-25T01:43:50.466 INFO:teuthology.orchestra.run.smithi130.stdout: "dn+": 0, 2024-04-25T01:43:50.467 INFO:teuthology.orchestra.run.smithi130.stdout: "dn-": 0, 2024-04-25T01:43:50.467 INFO:teuthology.orchestra.run.smithi130.stdout: "cap": 0, 2024-04-25T01:43:50.467 INFO:teuthology.orchestra.run.smithi130.stdout: "cap+": 0, 2024-04-25T01:43:50.467 INFO:teuthology.orchestra.run.smithi130.stdout: "cap-": 0, 2024-04-25T01:43:50.467 INFO:teuthology.orchestra.run.smithi130.stdout: "rss": 52804, 2024-04-25T01:43:50.467 INFO:teuthology.orchestra.run.smithi130.stdout: "heap": 215420 2024-04-25T01:43:50.467 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.467 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_server": { 2024-04-25T01:43:50.467 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_client_request": 0, 2024-04-25T01:43:50.467 INFO:teuthology.orchestra.run.smithi130.stdout: "dispatch_server_request": 0, 2024-04-25T01:43:50.467 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_request": 0, 2024-04-25T01:43:50.467 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_client_session": 0, 2024-04-25T01:43:50.467 INFO:teuthology.orchestra.run.smithi130.stdout: "handle_peer_request": 0, 2024-04-25T01:43:50.467 INFO:teuthology.orchestra.run.smithi130.stdout: "req_create_latency": { 2024-04-25T01:43:50.467 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.468 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.468 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.468 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.468 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getattr_latency": { 2024-04-25T01:43:50.468 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.468 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.468 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.468 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.468 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getfilelock_latency": { 2024-04-25T01:43:50.468 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.468 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.468 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.468 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.468 INFO:teuthology.orchestra.run.smithi130.stdout: "req_link_latency": { 2024-04-25T01:43:50.469 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.469 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.469 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.469 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.469 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookup_latency": { 2024-04-25T01:43:50.469 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.469 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.469 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.469 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.469 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookuphash_latency": { 2024-04-25T01:43:50.469 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.469 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.469 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.469 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.469 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupino_latency": { 2024-04-25T01:43:50.469 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.470 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.470 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.470 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.470 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupname_latency": { 2024-04-25T01:43:50.470 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.470 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.470 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.470 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.470 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupparent_latency": { 2024-04-25T01:43:50.470 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.470 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.470 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.470 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.470 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lookupsnap_latency": { 2024-04-25T01:43:50.471 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.471 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.471 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.471 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.471 INFO:teuthology.orchestra.run.smithi130.stdout: "req_lssnap_latency": { 2024-04-25T01:43:50.471 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.471 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.471 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.471 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.471 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mkdir_latency": { 2024-04-25T01:43:50.471 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.472 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.472 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.472 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.472 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mknod_latency": { 2024-04-25T01:43:50.472 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.472 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.472 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.472 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.472 INFO:teuthology.orchestra.run.smithi130.stdout: "req_mksnap_latency": { 2024-04-25T01:43:50.472 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.472 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.473 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.473 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.473 INFO:teuthology.orchestra.run.smithi130.stdout: "req_open_latency": { 2024-04-25T01:43:50.473 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.473 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.473 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.473 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.473 INFO:teuthology.orchestra.run.smithi130.stdout: "req_readdir_latency": { 2024-04-25T01:43:50.473 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.473 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.473 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.473 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.473 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rename_latency": { 2024-04-25T01:43:50.473 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.473 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.474 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.474 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.474 INFO:teuthology.orchestra.run.smithi130.stdout: "req_renamesnap_latency": { 2024-04-25T01:43:50.474 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.474 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.474 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.474 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.474 INFO:teuthology.orchestra.run.smithi130.stdout: "req_snapdiff_latency": { 2024-04-25T01:43:50.474 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.474 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.474 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.474 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.474 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmdir_latency": { 2024-04-25T01:43:50.474 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.475 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.475 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.475 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.475 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmsnap_latency": { 2024-04-25T01:43:50.475 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.475 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.475 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.475 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.475 INFO:teuthology.orchestra.run.smithi130.stdout: "req_rmxattr_latency": { 2024-04-25T01:43:50.475 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.475 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.475 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.475 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.475 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setattr_latency": { 2024-04-25T01:43:50.475 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.476 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.476 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.476 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.476 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setdirlayout_latency": { 2024-04-25T01:43:50.476 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.476 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.476 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.476 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.476 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setfilelock_latency": { 2024-04-25T01:43:50.476 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.476 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.476 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.476 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.476 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setlayout_latency": { 2024-04-25T01:43:50.477 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.477 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.477 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.477 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.477 INFO:teuthology.orchestra.run.smithi130.stdout: "req_setxattr_latency": { 2024-04-25T01:43:50.477 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.477 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.477 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.477 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.477 INFO:teuthology.orchestra.run.smithi130.stdout: "req_symlink_latency": { 2024-04-25T01:43:50.477 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.477 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.477 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.477 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.477 INFO:teuthology.orchestra.run.smithi130.stdout: "req_unlink_latency": { 2024-04-25T01:43:50.477 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.478 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.478 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.478 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.478 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_revoke_eviction": 0, 2024-04-25T01:43:50.478 INFO:teuthology.orchestra.run.smithi130.stdout: "cap_acquisition_throttle": 0, 2024-04-25T01:43:50.478 INFO:teuthology.orchestra.run.smithi130.stdout: "req_getvxattr_latency": { 2024-04-25T01:43:50.478 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.478 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.478 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.478 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:50.478 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.478 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_sessions": { 2024-04-25T01:43:50.478 INFO:teuthology.orchestra.run.smithi130.stdout: "session_count": 1, 2024-04-25T01:43:50.478 INFO:teuthology.orchestra.run.smithi130.stdout: "session_add": 2, 2024-04-25T01:43:50.479 INFO:teuthology.orchestra.run.smithi130.stdout: "session_remove": 1, 2024-04-25T01:43:50.479 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_open": 1, 2024-04-25T01:43:50.479 INFO:teuthology.orchestra.run.smithi130.stdout: "sessions_stale": 0, 2024-04-25T01:43:50.479 INFO:teuthology.orchestra.run.smithi130.stdout: "total_load": 0, 2024-04-25T01:43:50.479 INFO:teuthology.orchestra.run.smithi130.stdout: "average_load": 0, 2024-04-25T01:43:50.479 INFO:teuthology.orchestra.run.smithi130.stdout: "avg_session_uptime": 1808, 2024-04-25T01:43:50.479 INFO:teuthology.orchestra.run.smithi130.stdout: "mdthresh_evicted": 0 2024-04-25T01:43:50.479 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.479 INFO:teuthology.orchestra.run.smithi130.stdout: "mempool": { 2024-04-25T01:43:50.479 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_bytes": 0, 2024-04-25T01:43:50.479 INFO:teuthology.orchestra.run.smithi130.stdout: "bloom_filter_items": 0, 2024-04-25T01:43:50.479 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T01:43:50.479 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_alloc_items": 0, 2024-04-25T01:43:50.479 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T01:43:50.479 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_data_items": 0, 2024-04-25T01:43:50.480 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T01:43:50.480 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T01:43:50.480 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T01:43:50.480 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T01:43:50.480 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T01:43:50.480 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_other_items": 0, 2024-04-25T01:43:50.480 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T01:43:50.480 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T01:43:50.480 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_bytes": 0, 2024-04-25T01:43:50.480 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_extent_items": 0, 2024-04-25T01:43:50.480 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_bytes": 0, 2024-04-25T01:43:50.480 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_blob_items": 0, 2024-04-25T01:43:50.480 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T01:43:50.480 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T01:43:50.480 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T01:43:50.481 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T01:43:50.481 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T01:43:50.481 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_fsck_items": 0, 2024-04-25T01:43:50.481 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_bytes": 0, 2024-04-25T01:43:50.481 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_txc_items": 0, 2024-04-25T01:43:50.481 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T01:43:50.481 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T01:43:50.481 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_bytes": 0, 2024-04-25T01:43:50.481 INFO:teuthology.orchestra.run.smithi130.stdout: "bluestore_writing_items": 0, 2024-04-25T01:43:50.481 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_bytes": 0, 2024-04-25T01:43:50.481 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_items": 0, 2024-04-25T01:43:50.481 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T01:43:50.481 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_reader_items": 0, 2024-04-25T01:43:50.481 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T01:43:50.481 INFO:teuthology.orchestra.run.smithi130.stdout: "bluefs_file_writer_items": 0, 2024-04-25T01:43:50.481 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_bytes": 182138, 2024-04-25T01:43:50.482 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_anon_items": 50, 2024-04-25T01:43:50.482 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_bytes": 0, 2024-04-25T01:43:50.482 INFO:teuthology.orchestra.run.smithi130.stdout: "buffer_meta_items": 0, 2024-04-25T01:43:50.482 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_bytes": 0, 2024-04-25T01:43:50.482 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_items": 0, 2024-04-25T01:43:50.482 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_bytes": 0, 2024-04-25T01:43:50.482 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_mapbl_items": 0, 2024-04-25T01:43:50.482 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_bytes": 0, 2024-04-25T01:43:50.482 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_pglog_items": 0, 2024-04-25T01:43:50.482 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_bytes": 9320, 2024-04-25T01:43:50.482 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_items": 178, 2024-04-25T01:43:50.482 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T01:43:50.482 INFO:teuthology.orchestra.run.smithi130.stdout: "osdmap_mapping_items": 0, 2024-04-25T01:43:50.482 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_bytes": 0, 2024-04-25T01:43:50.483 INFO:teuthology.orchestra.run.smithi130.stdout: "pgmap_items": 0, 2024-04-25T01:43:50.483 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_bytes": 8464, 2024-04-25T01:43:50.483 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_co_items": 13, 2024-04-25T01:43:50.483 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_bytes": 0, 2024-04-25T01:43:50.483 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_1_items": 0, 2024-04-25T01:43:50.483 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_bytes": 0, 2024-04-25T01:43:50.483 INFO:teuthology.orchestra.run.smithi130.stdout: "unittest_2_items": 0 2024-04-25T01:43:50.483 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.483 INFO:teuthology.orchestra.run.smithi130.stdout: "objecter": { 2024-04-25T01:43:50.483 INFO:teuthology.orchestra.run.smithi130.stdout: "op_active": 0, 2024-04-25T01:43:50.483 INFO:teuthology.orchestra.run.smithi130.stdout: "op_laggy": 0, 2024-04-25T01:43:50.483 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send": 2233, 2024-04-25T01:43:50.483 INFO:teuthology.orchestra.run.smithi130.stdout: "op_send_bytes": 21, 2024-04-25T01:43:50.483 INFO:teuthology.orchestra.run.smithi130.stdout: "op_resend": 3, 2024-04-25T01:43:50.483 INFO:teuthology.orchestra.run.smithi130.stdout: "op_reply": 2233, 2024-04-25T01:43:50.483 INFO:teuthology.orchestra.run.smithi130.stdout: "op_latency": { 2024-04-25T01:43:50.484 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2233, 2024-04-25T01:43:50.484 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 1.353975224, 2024-04-25T01:43:50.484 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000606348 2024-04-25T01:43:50.484 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.484 INFO:teuthology.orchestra.run.smithi130.stdout: "op_inflight": 0, 2024-04-25T01:43:50.484 INFO:teuthology.orchestra.run.smithi130.stdout: "oplen_avg": { 2024-04-25T01:43:50.484 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 2233, 2024-04-25T01:43:50.484 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 2234 2024-04-25T01:43:50.484 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.484 INFO:teuthology.orchestra.run.smithi130.stdout: "op": 2233, 2024-04-25T01:43:50.484 INFO:teuthology.orchestra.run.smithi130.stdout: "op_r": 2233, 2024-04-25T01:43:50.484 INFO:teuthology.orchestra.run.smithi130.stdout: "op_w": 0, 2024-04-25T01:43:50.484 INFO:teuthology.orchestra.run.smithi130.stdout: "op_rmw": 0, 2024-04-25T01:43:50.484 INFO:teuthology.orchestra.run.smithi130.stdout: "op_pg": 0, 2024-04-25T01:43:50.484 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_stat": 1457, 2024-04-25T01:43:50.485 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_create": 0, 2024-04-25T01:43:50.485 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_read": 774, 2024-04-25T01:43:50.485 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_write": 0, 2024-04-25T01:43:50.485 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writefull": 0, 2024-04-25T01:43:50.485 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_writesame": 0, 2024-04-25T01:43:50.485 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_append": 0, 2024-04-25T01:43:50.485 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_zero": 0, 2024-04-25T01:43:50.485 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_truncate": 0, 2024-04-25T01:43:50.485 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_delete": 0, 2024-04-25T01:43:50.485 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_mapext": 0, 2024-04-25T01:43:50.485 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_sparse_read": 0, 2024-04-25T01:43:50.485 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_clonerange": 0, 2024-04-25T01:43:50.485 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_getxattr": 1, 2024-04-25T01:43:50.485 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_setxattr": 0, 2024-04-25T01:43:50.485 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_cmpxattr": 0, 2024-04-25T01:43:50.486 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_rmxattr": 0, 2024-04-25T01:43:50.486 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_resetxattrs": 0, 2024-04-25T01:43:50.486 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_call": 0, 2024-04-25T01:43:50.486 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_watch": 0, 2024-04-25T01:43:50.486 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_notify": 0, 2024-04-25T01:43:50.486 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T01:43:50.486 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls": 0, 2024-04-25T01:43:50.486 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_pgls_filter": 0, 2024-04-25T01:43:50.486 INFO:teuthology.orchestra.run.smithi130.stdout: "osdop_other": 0, 2024-04-25T01:43:50.486 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_active": 0, 2024-04-25T01:43:50.486 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_send": 0, 2024-04-25T01:43:50.486 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_resend": 0, 2024-04-25T01:43:50.486 INFO:teuthology.orchestra.run.smithi130.stdout: "linger_ping": 0, 2024-04-25T01:43:50.486 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_active": 0, 2024-04-25T01:43:50.486 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_send": 0, 2024-04-25T01:43:50.487 INFO:teuthology.orchestra.run.smithi130.stdout: "poolop_resend": 0, 2024-04-25T01:43:50.487 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_active": 0, 2024-04-25T01:43:50.487 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_send": 0, 2024-04-25T01:43:50.487 INFO:teuthology.orchestra.run.smithi130.stdout: "poolstat_resend": 0, 2024-04-25T01:43:50.487 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_active": 0, 2024-04-25T01:43:50.487 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_send": 0, 2024-04-25T01:43:50.487 INFO:teuthology.orchestra.run.smithi130.stdout: "statfs_resend": 0, 2024-04-25T01:43:50.487 INFO:teuthology.orchestra.run.smithi130.stdout: "command_active": 0, 2024-04-25T01:43:50.487 INFO:teuthology.orchestra.run.smithi130.stdout: "command_send": 0, 2024-04-25T01:43:50.487 INFO:teuthology.orchestra.run.smithi130.stdout: "command_resend": 0, 2024-04-25T01:43:50.487 INFO:teuthology.orchestra.run.smithi130.stdout: "map_epoch": 102, 2024-04-25T01:43:50.487 INFO:teuthology.orchestra.run.smithi130.stdout: "map_full": 0, 2024-04-25T01:43:50.487 INFO:teuthology.orchestra.run.smithi130.stdout: "map_inc": 11, 2024-04-25T01:43:50.487 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_sessions": 4, 2024-04-25T01:43:50.488 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_open": 4, 2024-04-25T01:43:50.488 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_session_close": 0, 2024-04-25T01:43:50.488 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_laggy": 0, 2024-04-25T01:43:50.488 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_wr": 0, 2024-04-25T01:43:50.488 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_rd": 2, 2024-04-25T01:43:50.488 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_del": 0 2024-04-25T01:43:50.488 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.488 INFO:teuthology.orchestra.run.smithi130.stdout: "oft": { 2024-04-25T01:43:50.488 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_objs": 0, 2024-04-25T01:43:50.488 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_kv_pairs": 0, 2024-04-25T01:43:50.488 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_updates": 0, 2024-04-25T01:43:50.488 INFO:teuthology.orchestra.run.smithi130.stdout: "omap_total_removes": 0 2024-04-25T01:43:50.488 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.488 INFO:teuthology.orchestra.run.smithi130.stdout: "osd-slow-ops": { 2024-04-25T01:43:50.488 INFO:teuthology.orchestra.run.smithi130.stdout: "slow_ops_count": 0 2024-04-25T01:43:50.488 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.489 INFO:teuthology.orchestra.run.smithi130.stdout: "purge_queue": { 2024-04-25T01:43:50.489 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops": 0, 2024-04-25T01:43:50.489 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T01:43:50.489 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing": 0, 2024-04-25T01:43:50.489 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executing_high_water": 0, 2024-04-25T01:43:50.489 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed_ops": 0, 2024-04-25T01:43:50.489 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_executed": 0, 2024-04-25T01:43:50.489 INFO:teuthology.orchestra.run.smithi130.stdout: "pq_item_in_journal": 0 2024-04-25T01:43:50.489 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.489 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T01:43:50.489 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:50.489 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-04-25T01:43:50.490 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:50.490 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 2486, 2024-04-25T01:43:50.490 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 1875064, 2024-04-25T01:43:50.490 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:50.490 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 2486, 2024-04-25T01:43:50.490 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-25T01:43:50.490 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-25T01:43:50.490 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 2486, 2024-04-25T01:43:50.490 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 1875064, 2024-04-25T01:43:50.490 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:50.490 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.490 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.490 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.490 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:50.490 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.491 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_bytes": { 2024-04-25T01:43:50.491 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:50.491 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 104857600, 2024-04-25T01:43:50.491 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:50.491 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-25T01:43:50.491 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-25T01:43:50.491 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:50.491 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:50.491 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 2233, 2024-04-25T01:43:50.491 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 1516, 2024-04-25T01:43:50.492 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 2233, 2024-04-25T01:43:50.492 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 1516, 2024-04-25T01:43:50.492 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:50.492 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.492 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.492 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.492 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:50.492 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.492 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-objecter_ops": { 2024-04-25T01:43:50.492 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:50.492 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 1024, 2024-04-25T01:43:50.492 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:50.492 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-25T01:43:50.492 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-25T01:43:50.492 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:50.493 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:50.493 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 2233, 2024-04-25T01:43:50.493 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 2233, 2024-04-25T01:43:50.493 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 2233, 2024-04-25T01:43:50.493 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 2233, 2024-04-25T01:43:50.493 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:50.493 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.493 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.493 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.493 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:50.493 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.493 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle": { 2024-04-25T01:43:50.493 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:50.493 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-04-25T01:43:50.493 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:50.494 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-25T01:43:50.494 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-25T01:43:50.494 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:50.494 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:50.494 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-25T01:43:50.494 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-25T01:43:50.494 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 0, 2024-04-25T01:43:50.494 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 0, 2024-04-25T01:43:50.494 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:50.494 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.494 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.494 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.494 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:50.494 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2024-04-25T01:43:50.495 INFO:teuthology.orchestra.run.smithi130.stdout: "throttle-write_buf_throttle-0x55ae59cf66e0": { 2024-04-25T01:43:50.495 INFO:teuthology.orchestra.run.smithi130.stdout: "val": 0, 2024-04-25T01:43:50.495 INFO:teuthology.orchestra.run.smithi130.stdout: "max": 3758096384, 2024-04-25T01:43:50.495 INFO:teuthology.orchestra.run.smithi130.stdout: "get_started": 0, 2024-04-25T01:43:50.495 INFO:teuthology.orchestra.run.smithi130.stdout: "get": 0, 2024-04-25T01:43:50.495 INFO:teuthology.orchestra.run.smithi130.stdout: "get_sum": 0, 2024-04-25T01:43:50.495 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_fail": 0, 2024-04-25T01:43:50.495 INFO:teuthology.orchestra.run.smithi130.stdout: "get_or_fail_success": 0, 2024-04-25T01:43:50.495 INFO:teuthology.orchestra.run.smithi130.stdout: "take": 0, 2024-04-25T01:43:50.495 INFO:teuthology.orchestra.run.smithi130.stdout: "take_sum": 0, 2024-04-25T01:43:50.495 INFO:teuthology.orchestra.run.smithi130.stdout: "put": 0, 2024-04-25T01:43:50.495 INFO:teuthology.orchestra.run.smithi130.stdout: "put_sum": 0, 2024-04-25T01:43:50.495 INFO:teuthology.orchestra.run.smithi130.stdout: "wait": { 2024-04-25T01:43:50.495 INFO:teuthology.orchestra.run.smithi130.stdout: "avgcount": 0, 2024-04-25T01:43:50.495 INFO:teuthology.orchestra.run.smithi130.stdout: "sum": 0.000000000, 2024-04-25T01:43:50.495 INFO:teuthology.orchestra.run.smithi130.stdout: "avgtime": 0.000000000 2024-04-25T01:43:50.496 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:50.496 INFO:teuthology.orchestra.run.smithi130.stdout: } 2024-04-25T01:43:50.496 INFO:teuthology.orchestra.run.smithi130.stdout:} 2024-04-25T01:43:50.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:50 smithi032 ceph-mon[32461]: pgmap v335: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T01:43:51.444 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-04-25T01:43:51.445 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-04-25T01:43:51.445 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-04-25T01:43:51.445 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-04-25T01:43:51.445 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T01:43:51.445 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-25T01:43:51.454 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-25T01:43:51.455 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:43:51.455 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:43:51.474 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T01:43:51.474 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:43:51.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:43:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.686+0000 7fe4289a5640 1 -- 172.21.15.28:0/3802220488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe424100fe0 msgr2=0x7fe42410d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.686+0000 7fe4289a5640 1 --2- 172.21.15.28:0/3802220488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe424100fe0 0x7fe42410d4f0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fe414009930 tx=0x7fe41402a420 comp rx=0 tx=0).stop 2024-04-25T01:43:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.687+0000 7fe4289a5640 1 -- 172.21.15.28:0/3802220488 shutdown_connections 2024-04-25T01:43:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.687+0000 7fe4289a5640 1 --2- 172.21.15.28:0/3802220488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe424100fe0 0x7fe42410d4f0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.687+0000 7fe4289a5640 1 --2- 172.21.15.28:0/3802220488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe424100620 0x7fe424100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.687+0000 7fe4289a5640 1 --2- 172.21.15.28:0/3802220488 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe424106630 0x7fe424106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.687+0000 7fe4289a5640 1 -- 172.21.15.28:0/3802220488 >> 172.21.15.28:0/3802220488 conn(0x7fe424077740 msgr2=0x7fe424077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.687+0000 7fe4289a5640 1 -- 172.21.15.28:0/3802220488 shutdown_connections 2024-04-25T01:43:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.687+0000 7fe4289a5640 1 -- 172.21.15.28:0/3802220488 wait complete. 2024-04-25T01:43:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.687+0000 7fe4289a5640 1 Processor -- start 2024-04-25T01:43:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.688+0000 7fe4289a5640 1 -- start start 2024-04-25T01:43:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.688+0000 7fe4289a5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe424100620 0x7fe424102ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.688+0000 7fe4289a5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe424100fe0 0x7fe4241030e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.688+0000 7fe4289a5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe424106630 0x7fe424103620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.688+0000 7fe4289a5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe424110a00 con 0x7fe424100620 2024-04-25T01:43:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.688+0000 7fe4289a5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe424110880 con 0x7fe424106630 2024-04-25T01:43:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.688+0000 7fe4289a5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe424110b80 con 0x7fe424100fe0 2024-04-25T01:43:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.688+0000 7fe42259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe424100620 0x7fe424102ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.688+0000 7fe42259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe424100620 0x7fe424102ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52412/0 (socket says 172.21.15.28:52412) 2024-04-25T01:43:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.688+0000 7fe42259c640 1 -- 172.21.15.28:0/1245836720 learned_addr learned my addr 172.21.15.28:0/1245836720 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.688+0000 7fe422d9d640 1 --2- 172.21.15.28:0/1245836720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe424106630 0x7fe424103620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.688+0000 7fe419d9b640 1 --2- 172.21.15.28:0/1245836720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe424100fe0 0x7fe4241030e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.688+0000 7fe42259c640 1 -- 172.21.15.28:0/1245836720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe424100fe0 msgr2=0x7fe4241030e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.688+0000 7fe42259c640 1 --2- 172.21.15.28:0/1245836720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe424100fe0 0x7fe4241030e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.688+0000 7fe42259c640 1 -- 172.21.15.28:0/1245836720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe424106630 msgr2=0x7fe424103620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:51.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.688+0000 7fe42259c640 1 --2- 172.21.15.28:0/1245836720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe424106630 0x7fe424103620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.688+0000 7fe42259c640 1 -- 172.21.15.28:0/1245836720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4241a6450 con 0x7fe424100620 2024-04-25T01:43:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.688+0000 7fe419d9b640 1 --2- 172.21.15.28:0/1245836720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe424100fe0 0x7fe4241030e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.689+0000 7fe42259c640 1 --2- 172.21.15.28:0/1245836720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe424100620 0x7fe424102ba0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00cce0 tx=0x7fe41c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.689+0000 7fe41bfff640 1 -- 172.21.15.28:0/1245836720 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c00f650 con 0x7fe424100620 2024-04-25T01:43:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.689+0000 7fe4289a5640 1 -- 172.21.15.28:0/1245836720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4241a6740 con 0x7fe424100620 2024-04-25T01:43:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.689+0000 7fe41bfff640 1 -- 172.21.15.28:0/1245836720 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe41c00fcd0 con 0x7fe424100620 2024-04-25T01:43:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.689+0000 7fe41bfff640 1 -- 172.21.15.28:0/1245836720 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c018c70 con 0x7fe424100620 2024-04-25T01:43:51.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.689+0000 7fe4289a5640 1 -- 172.21.15.28:0/1245836720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4241a6ce0 con 0x7fe424100620 2024-04-25T01:43:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.690+0000 7fe4289a5640 1 -- 172.21.15.28:0/1245836720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe424068560 con 0x7fe424100620 2024-04-25T01:43:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.691+0000 7fe41bfff640 1 -- 172.21.15.28:0/1245836720 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe41c00f7f0 con 0x7fe424100620 2024-04-25T01:43:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.692+0000 7fe41bfff640 1 --2- 172.21.15.28:0/1245836720 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe4100a2d00 0x7fe4100a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.692+0000 7fe41bfff640 1 -- 172.21.15.28:0/1245836720 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9438+0+0 (secure 0 0 0) 0x7fe41c0cfc00 con 0x7fe424100620 2024-04-25T01:43:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.692+0000 7fe419d9b640 1 --2- 172.21.15.28:0/1245836720 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe4100a2d00 0x7fe4100a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.692+0000 7fe419d9b640 1 --2- 172.21.15.28:0/1245836720 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe4100a2d00 0x7fe4100a51c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe424104630 tx=0x7fe40c005f90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:51.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.697+0000 7fe41bfff640 1 -- 172.21.15.28:0/1245836720 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe41c0d6080 con 0x7fe424100620 2024-04-25T01:43:51.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.887+0000 7fe4289a5640 1 -- 172.21.15.28:0/1245836720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe424109bc0 con 0x7fe424100620 2024-04-25T01:43:51.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.888+0000 7fe41bfff640 1 -- 172.21.15.28:0/1245836720 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+25127 (secure 0 0 0) 0x7fe41c014030 con 0x7fe424100620 2024-04-25T01:43:51.879 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:43:51.879 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":102,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:43:16.975088+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/1149553567":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/500268183":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/3499593842":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/3630315328":"2024-04-26T01:33:00.617174+0000","172.21.15.28:6800/1149553567":"2024-04-26T01:33:00.617174+0000","172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3812371257":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:43:51.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.891+0000 7fe4289a5640 1 -- 172.21.15.28:0/1245836720 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe4100a2d00 msgr2=0x7fe4100a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:51.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.891+0000 7fe4289a5640 1 --2- 172.21.15.28:0/1245836720 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe4100a2d00 0x7fe4100a51c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe424104630 tx=0x7fe40c005f90 comp rx=0 tx=0).stop 2024-04-25T01:43:51.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.891+0000 7fe4289a5640 1 -- 172.21.15.28:0/1245836720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe424100620 msgr2=0x7fe424102ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:51.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.891+0000 7fe4289a5640 1 --2- 172.21.15.28:0/1245836720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe424100620 0x7fe424102ba0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00cce0 tx=0x7fe41c00a620 comp rx=0 tx=0).stop 2024-04-25T01:43:51.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.891+0000 7fe4289a5640 1 -- 172.21.15.28:0/1245836720 shutdown_connections 2024-04-25T01:43:51.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.891+0000 7fe4289a5640 1 --2- 172.21.15.28:0/1245836720 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe4100a2d00 0x7fe4100a51c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:51.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.891+0000 7fe4289a5640 1 --2- 172.21.15.28:0/1245836720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe424106630 0x7fe424103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:51.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.891+0000 7fe4289a5640 1 --2- 172.21.15.28:0/1245836720 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe424100fe0 0x7fe4241030e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:51.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.891+0000 7fe4289a5640 1 --2- 172.21.15.28:0/1245836720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe424100620 0x7fe424102ba0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:51.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.891+0000 7fe4289a5640 1 -- 172.21.15.28:0/1245836720 >> 172.21.15.28:0/1245836720 conn(0x7fe424077740 msgr2=0x7fe424068b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:51.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.891+0000 7fe4289a5640 1 -- 172.21.15.28:0/1245836720 shutdown_connections 2024-04-25T01:43:51.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:51.891+0000 7fe4289a5640 1 -- 172.21.15.28:0/1245836720 wait complete. 2024-04-25T01:43:51.893 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:25:48.767414+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:30:32.282430+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:30:33.127049+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T01:30:36.796132+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T01:43:51.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:43:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.011+0000 7fe556b10640 1 -- 172.21.15.28:0/3075615657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe55010bd00 msgr2=0x7fe55010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.011+0000 7fe556b10640 1 --2- 172.21.15.28:0/3075615657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe55010bd00 0x7fe55010e1a0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fe548009ca0 tx=0x7fe54802a9e0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.011+0000 7fe556b10640 1 -- 172.21.15.28:0/3075615657 shutdown_connections 2024-04-25T01:43:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.011+0000 7fe556b10640 1 --2- 172.21.15.28:0/3075615657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe55010bd00 0x7fe55010e1a0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.011+0000 7fe556b10640 1 --2- 172.21.15.28:0/3075615657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5501032e0 0x7fe55010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.011+0000 7fe556b10640 1 --2- 172.21.15.28:0/3075615657 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe5501029c0 0x7fe550102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.011+0000 7fe556b10640 1 -- 172.21.15.28:0/3075615657 >> 172.21.15.28:0/3075615657 conn(0x7fe550077740 msgr2=0x7fe550077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.011+0000 7fe556b10640 1 -- 172.21.15.28:0/3075615657 shutdown_connections 2024-04-25T01:43:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.011+0000 7fe556b10640 1 -- 172.21.15.28:0/3075615657 wait complete. 2024-04-25T01:43:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe556b10640 1 Processor -- start 2024-04-25T01:43:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe556b10640 1 -- start start 2024-04-25T01:43:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe556b10640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5501029c0 0x7fe550198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe556b10640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5501032e0 0x7fe550198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe556b10640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe55010bd00 0x7fe55019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe556b10640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe550111b00 con 0x7fe5501032e0 2024-04-25T01:43:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe556b10640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe550111980 con 0x7fe5501029c0 2024-04-25T01:43:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe556b10640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe550111c80 con 0x7fe55010bd00 2024-04-25T01:43:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe54ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5501032e0 0x7fe550198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe54ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5501032e0 0x7fe550198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52426/0 (socket says 172.21.15.28:52426) 2024-04-25T01:43:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe54ffff640 1 -- 172.21.15.28:0/4079189388 learned_addr learned my addr 172.21.15.28:0/4079189388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe5550ad640 1 --2- 172.21.15.28:0/4079189388 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe55010bd00 0x7fe55019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe5548ac640 1 --2- 172.21.15.28:0/4079189388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5501029c0 0x7fe550198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe54ffff640 1 -- 172.21.15.28:0/4079189388 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe55010bd00 msgr2=0x7fe55019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe54ffff640 1 --2- 172.21.15.28:0/4079189388 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe55010bd00 0x7fe55019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe54ffff640 1 -- 172.21.15.28:0/4079189388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5501029c0 msgr2=0x7fe550198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe54ffff640 1 --2- 172.21.15.28:0/4079189388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5501029c0 0x7fe550198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe54ffff640 1 -- 172.21.15.28:0/4079189388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe55019cfc0 con 0x7fe5501032e0 2024-04-25T01:43:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe5550ad640 1 --2- 172.21.15.28:0/4079189388 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe55010bd00 0x7fe55019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe5548ac640 1 --2- 172.21.15.28:0/4079189388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5501029c0 0x7fe550198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.012+0000 7fe54ffff640 1 --2- 172.21.15.28:0/4079189388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5501032e0 0x7fe550198560 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00d950 tx=0x7fe53c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.013+0000 7fe54dffb640 1 -- 172.21.15.28:0/4079189388 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53c017030 con 0x7fe5501032e0 2024-04-25T01:43:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.013+0000 7fe54dffb640 1 -- 172.21.15.28:0/4079189388 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe53c004020 con 0x7fe5501032e0 2024-04-25T01:43:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.013+0000 7fe556b10640 1 -- 172.21.15.28:0/4079189388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe55019d2b0 con 0x7fe5501032e0 2024-04-25T01:43:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.013+0000 7fe54dffb640 1 -- 172.21.15.28:0/4079189388 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53c0047a0 con 0x7fe5501032e0 2024-04-25T01:43:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.013+0000 7fe556b10640 1 -- 172.21.15.28:0/4079189388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5501a4b40 con 0x7fe5501032e0 2024-04-25T01:43:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.014+0000 7fe556b10640 1 -- 172.21.15.28:0/4079189388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe550068560 con 0x7fe5501032e0 2024-04-25T01:43:52.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.016+0000 7fe54dffb640 1 -- 172.21.15.28:0/4079189388 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe53c0041d0 con 0x7fe5501032e0 2024-04-25T01:43:52.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.016+0000 7fe54dffb640 1 --2- 172.21.15.28:0/4079189388 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe5400a2d00 0x7fe5400a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:52.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.016+0000 7fe54dffb640 1 -- 172.21.15.28:0/4079189388 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9438+0+0 (secure 0 0 0) 0x7fe53c0cf1f0 con 0x7fe5501032e0 2024-04-25T01:43:52.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.016+0000 7fe5548ac640 1 --2- 172.21.15.28:0/4079189388 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe5400a2d00 0x7fe5400a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:52.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.017+0000 7fe5548ac640 1 --2- 172.21.15.28:0/4079189388 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe5400a2d00 0x7fe5400a51c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe5440058c0 tx=0x7fe544005850 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:52.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.021+0000 7fe54dffb640 1 -- 172.21.15.28:0/4079189388 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe53c09ba80 con 0x7fe5501032e0 2024-04-25T01:43:52.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.208+0000 7fe556b10640 1 -- 172.21.15.28:0/4079189388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe550106e80 con 0x7fe5501032e0 2024-04-25T01:43:52.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.208+0000 7fe54dffb640 1 -- 172.21.15.28:0/4079189388 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) ==== 93+0+10 (secure 0 0 0) 0x7fe53c09b380 con 0x7fe5501032e0 2024-04-25T01:43:52.199 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T01:43:52.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.210+0000 7fe556b10640 1 -- 172.21.15.28:0/4079189388 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe5400a2d00 msgr2=0x7fe5400a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:52.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.210+0000 7fe556b10640 1 --2- 172.21.15.28:0/4079189388 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe5400a2d00 0x7fe5400a51c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe5440058c0 tx=0x7fe544005850 comp rx=0 tx=0).stop 2024-04-25T01:43:52.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.210+0000 7fe556b10640 1 -- 172.21.15.28:0/4079189388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5501032e0 msgr2=0x7fe550198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:52.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.210+0000 7fe556b10640 1 --2- 172.21.15.28:0/4079189388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5501032e0 0x7fe550198560 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00d950 tx=0x7fe53c00de10 comp rx=0 tx=0).stop 2024-04-25T01:43:52.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.210+0000 7fe556b10640 1 -- 172.21.15.28:0/4079189388 shutdown_connections 2024-04-25T01:43:52.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.210+0000 7fe556b10640 1 --2- 172.21.15.28:0/4079189388 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe5400a2d00 0x7fe5400a51c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.210+0000 7fe556b10640 1 --2- 172.21.15.28:0/4079189388 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe55010bd00 0x7fe55019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.210+0000 7fe556b10640 1 --2- 172.21.15.28:0/4079189388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5501032e0 0x7fe550198560 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.211+0000 7fe556b10640 1 --2- 172.21.15.28:0/4079189388 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5501029c0 0x7fe550198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.211+0000 7fe556b10640 1 -- 172.21.15.28:0/4079189388 >> 172.21.15.28:0/4079189388 conn(0x7fe550077740 msgr2=0x7fe5500fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:52.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.211+0000 7fe556b10640 1 -- 172.21.15.28:0/4079189388 shutdown_connections 2024-04-25T01:43:52.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.211+0000 7fe556b10640 1 -- 172.21.15.28:0/4079189388 wait complete. 2024-04-25T01:43:52.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:43:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.326+0000 7fdee1dce640 1 -- 172.21.15.28:0/1446347662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdedc1032e0 msgr2=0x7fdedc10b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.327+0000 7fdee1dce640 1 --2- 172.21.15.28:0/1446347662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdedc1032e0 0x7fdedc10b7c0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fdecc009900 tx=0x7fdecc02a510 comp rx=0 tx=0).stop 2024-04-25T01:43:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.327+0000 7fdee1dce640 1 -- 172.21.15.28:0/1446347662 shutdown_connections 2024-04-25T01:43:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.327+0000 7fdee1dce640 1 --2- 172.21.15.28:0/1446347662 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdedc10bd00 0x7fdedc10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.327+0000 7fdee1dce640 1 --2- 172.21.15.28:0/1446347662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdedc1032e0 0x7fdedc10b7c0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.327+0000 7fdee1dce640 1 --2- 172.21.15.28:0/1446347662 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdedc1029c0 0x7fdedc102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.327+0000 7fdee1dce640 1 -- 172.21.15.28:0/1446347662 >> 172.21.15.28:0/1446347662 conn(0x7fdedc077740 msgr2=0x7fdedc077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:52.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.327+0000 7fdee1dce640 1 -- 172.21.15.28:0/1446347662 shutdown_connections 2024-04-25T01:43:52.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.327+0000 7fdee1dce640 1 -- 172.21.15.28:0/1446347662 wait complete. 2024-04-25T01:43:52.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.327+0000 7fdee1dce640 1 Processor -- start 2024-04-25T01:43:52.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.327+0000 7fdee1dce640 1 -- start start 2024-04-25T01:43:52.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.328+0000 7fdee1dce640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdedc1029c0 0x7fdedc100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:52.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.328+0000 7fdee1dce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdedc1032e0 0x7fdedc100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:52.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.328+0000 7fdee1dce640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdedc10bd00 0x7fdedc101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:52.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.328+0000 7fdee1dce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc111b00 con 0x7fdedc1032e0 2024-04-25T01:43:52.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.328+0000 7fdee1dce640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc111980 con 0x7fdedc1029c0 2024-04-25T01:43:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.328+0000 7fdee1dce640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc111c80 con 0x7fdedc10bd00 2024-04-25T01:43:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.328+0000 7fdedaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdedc1032e0 0x7fdedc100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.328+0000 7fdedaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdedc1032e0 0x7fdedc100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52442/0 (socket says 172.21.15.28:52442) 2024-04-25T01:43:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.328+0000 7fdedaffd640 1 -- 172.21.15.28:0/3378138895 learned_addr learned my addr 172.21.15.28:0/3378138895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.328+0000 7fdedbfff640 1 --2- 172.21.15.28:0/3378138895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdedc10bd00 0x7fdedc101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.328+0000 7fdedb7fe640 1 --2- 172.21.15.28:0/3378138895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdedc1029c0 0x7fdedc100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.328+0000 7fdedaffd640 1 -- 172.21.15.28:0/3378138895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdedc10bd00 msgr2=0x7fdedc101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.328+0000 7fdedaffd640 1 --2- 172.21.15.28:0/3378138895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdedc10bd00 0x7fdedc101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.328+0000 7fdedaffd640 1 -- 172.21.15.28:0/3378138895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdedc1029c0 msgr2=0x7fdedc100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.328+0000 7fdedaffd640 1 --2- 172.21.15.28:0/3378138895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdedc1029c0 0x7fdedc100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.328+0000 7fdedaffd640 1 -- 172.21.15.28:0/3378138895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdedc1a64c0 con 0x7fdedc1032e0 2024-04-25T01:43:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.328+0000 7fdedb7fe640 1 --2- 172.21.15.28:0/3378138895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdedc1029c0 0x7fdedc100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:43:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.328+0000 7fdedaffd640 1 --2- 172.21.15.28:0/3378138895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdedc1032e0 0x7fdedc100b50 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fdecc006010 tx=0x7fdecc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.328+0000 7fded8ff9640 1 -- 172.21.15.28:0/3378138895 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc03f030 con 0x7fdedc1032e0 2024-04-25T01:43:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.329+0000 7fdee1dce640 1 -- 172.21.15.28:0/3378138895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdedc1a6750 con 0x7fdedc1032e0 2024-04-25T01:43:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.329+0000 7fded8ff9640 1 -- 172.21.15.28:0/3378138895 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdecc003ce0 con 0x7fdedc1032e0 2024-04-25T01:43:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.329+0000 7fded8ff9640 1 -- 172.21.15.28:0/3378138895 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc035070 con 0x7fdedc1032e0 2024-04-25T01:43:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.329+0000 7fdee1dce640 1 -- 172.21.15.28:0/3378138895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdedc1a6c90 con 0x7fdedc1032e0 2024-04-25T01:43:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.330+0000 7fdee1dce640 1 -- 172.21.15.28:0/3378138895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdedc068560 con 0x7fdedc1032e0 2024-04-25T01:43:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.331+0000 7fded8ff9640 1 -- 172.21.15.28:0/3378138895 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fdecc042030 con 0x7fdedc1032e0 2024-04-25T01:43:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.331+0000 7fded8ff9640 1 --2- 172.21.15.28:0/3378138895 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdec40a2d00 0x7fdec40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.331+0000 7fded8ff9640 1 -- 172.21.15.28:0/3378138895 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9438+0+0 (secure 0 0 0) 0x7fdecc0efe70 con 0x7fdedc1032e0 2024-04-25T01:43:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.331+0000 7fdedb7fe640 1 --2- 172.21.15.28:0/3378138895 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdec40a2d00 0x7fdec40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.332+0000 7fdedb7fe640 1 --2- 172.21.15.28:0/3378138895 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdec40a2d00 0x7fdec40a51c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fded0009b50 tx=0x7fded0005f90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:52.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.336+0000 7fded8ff9640 1 -- 172.21.15.28:0/3378138895 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdecc0bc7c0 con 0x7fdedc1032e0 2024-04-25T01:43:52.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:52 smithi130 ceph-mon[32702]: pgmap v336: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:52.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:52 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1245836720' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:43:52.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:52 smithi032 ceph-mon[32461]: pgmap v336: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:52.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:52 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1245836720' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:43:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.520+0000 7fdee1dce640 1 -- 172.21.15.28:0/3378138895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdedc106e80 con 0x7fdedc1032e0 2024-04-25T01:43:52.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.520+0000 7fded8ff9640 1 -- 172.21.15.28:0/3378138895 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) ==== 104+0+11 (secure 0 0 0) 0x7fdecc0bc0c0 con 0x7fdedc1032e0 2024-04-25T01:43:52.510 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:43:52.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.522+0000 7fdee1dce640 1 -- 172.21.15.28:0/3378138895 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdec40a2d00 msgr2=0x7fdec40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:52.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.522+0000 7fdee1dce640 1 --2- 172.21.15.28:0/3378138895 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdec40a2d00 0x7fdec40a51c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fded0009b50 tx=0x7fded0005f90 comp rx=0 tx=0).stop 2024-04-25T01:43:52.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.522+0000 7fdee1dce640 1 -- 172.21.15.28:0/3378138895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdedc1032e0 msgr2=0x7fdedc100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:52.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.522+0000 7fdee1dce640 1 --2- 172.21.15.28:0/3378138895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdedc1032e0 0x7fdedc100b50 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fdecc006010 tx=0x7fdecc0042f0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.522+0000 7fdee1dce640 1 -- 172.21.15.28:0/3378138895 shutdown_connections 2024-04-25T01:43:52.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.522+0000 7fdee1dce640 1 --2- 172.21.15.28:0/3378138895 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fdec40a2d00 0x7fdec40a51c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.522+0000 7fdee1dce640 1 --2- 172.21.15.28:0/3378138895 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fdedc10bd00 0x7fdedc101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.522+0000 7fdee1dce640 1 --2- 172.21.15.28:0/3378138895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdedc1032e0 0x7fdedc100b50 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.522+0000 7fdee1dce640 1 --2- 172.21.15.28:0/3378138895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdedc1029c0 0x7fdedc100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.522+0000 7fdee1dce640 1 -- 172.21.15.28:0/3378138895 >> 172.21.15.28:0/3378138895 conn(0x7fdedc077740 msgr2=0x7fdedc0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:52.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.522+0000 7fdee1dce640 1 -- 172.21.15.28:0/3378138895 shutdown_connections 2024-04-25T01:43:52.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.522+0000 7fdee1dce640 1 -- 172.21.15.28:0/3378138895 wait complete. 2024-04-25T01:43:52.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:52 smithi028 ceph-mon[29390]: pgmap v336: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:52.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:52 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1245836720' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:43:52.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:43:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.638+0000 7faf272ee640 1 -- 172.21.15.28:0/1122118377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf201066b0 msgr2=0x7faf20106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.638+0000 7faf272ee640 1 --2- 172.21.15.28:0/1122118377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf201066b0 0x7faf20106a90 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7faf1c009930 tx=0x7faf1c02a420 comp rx=0 tx=0).stop 2024-04-25T01:43:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.638+0000 7faf272ee640 1 -- 172.21.15.28:0/1122118377 shutdown_connections 2024-04-25T01:43:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.638+0000 7faf272ee640 1 --2- 172.21.15.28:0/1122118377 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf20100fd0 0x7faf2010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.638+0000 7faf272ee640 1 --2- 172.21.15.28:0/1122118377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf20100610 0x7faf20100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.638+0000 7faf272ee640 1 --2- 172.21.15.28:0/1122118377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf201066b0 0x7faf20106a90 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.638+0000 7faf272ee640 1 -- 172.21.15.28:0/1122118377 >> 172.21.15.28:0/1122118377 conn(0x7faf20077740 msgr2=0x7faf20077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.638+0000 7faf272ee640 1 -- 172.21.15.28:0/1122118377 shutdown_connections 2024-04-25T01:43:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.638+0000 7faf272ee640 1 -- 172.21.15.28:0/1122118377 wait complete. 2024-04-25T01:43:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.639+0000 7faf272ee640 1 Processor -- start 2024-04-25T01:43:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.639+0000 7faf272ee640 1 -- start start 2024-04-25T01:43:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.639+0000 7faf272ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf20100610 0x7faf201022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.639+0000 7faf272ee640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf20100fd0 0x7faf201027f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.639+0000 7faf272ee640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf201066b0 0x7faf20102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.639+0000 7faf272ee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faf20110a30 con 0x7faf20100610 2024-04-25T01:43:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.639+0000 7faf272ee640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faf201108b0 con 0x7faf20100fd0 2024-04-25T01:43:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.639+0000 7faf272ee640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7faf20110bb0 con 0x7faf201066b0 2024-04-25T01:43:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.639+0000 7faf2508a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf20100610 0x7faf201022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.639+0000 7faf2508a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf20100610 0x7faf201022b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52470/0 (socket says 172.21.15.28:52470) 2024-04-25T01:43:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.639+0000 7faf2508a640 1 -- 172.21.15.28:0/1084083048 learned_addr learned my addr 172.21.15.28:0/1084083048 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.639+0000 7faf24889640 1 --2- 172.21.15.28:0/1084083048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf20100fd0 0x7faf201027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.639+0000 7faf2588b640 1 --2- 172.21.15.28:0/1084083048 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf201066b0 0x7faf20102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.640+0000 7faf2508a640 1 -- 172.21.15.28:0/1084083048 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf201066b0 msgr2=0x7faf20102d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.640+0000 7faf2508a640 1 --2- 172.21.15.28:0/1084083048 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf201066b0 0x7faf20102d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.640+0000 7faf2508a640 1 -- 172.21.15.28:0/1084083048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf20100fd0 msgr2=0x7faf201027f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:52.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.640+0000 7faf2508a640 1 --2- 172.21.15.28:0/1084083048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf20100fd0 0x7faf201027f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.640+0000 7faf2508a640 1 -- 172.21.15.28:0/1084083048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf201a6440 con 0x7faf20100610 2024-04-25T01:43:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.640+0000 7faf2588b640 1 --2- 172.21.15.28:0/1084083048 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf201066b0 0x7faf20102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.640+0000 7faf2508a640 1 --2- 172.21.15.28:0/1084083048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf20100610 0x7faf201022b0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7faf1c005940 tx=0x7faf1c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.640+0000 7faf167fc640 1 -- 172.21.15.28:0/1084083048 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c03f030 con 0x7faf20100610 2024-04-25T01:43:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.640+0000 7faf167fc640 1 -- 172.21.15.28:0/1084083048 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf1c003ad0 con 0x7faf20100610 2024-04-25T01:43:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.640+0000 7faf272ee640 1 -- 172.21.15.28:0/1084083048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf201a66d0 con 0x7faf20100610 2024-04-25T01:43:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.640+0000 7faf167fc640 1 -- 172.21.15.28:0/1084083048 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c035070 con 0x7faf20100610 2024-04-25T01:43:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.640+0000 7faf272ee640 1 -- 172.21.15.28:0/1084083048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf201a6bc0 con 0x7faf20100610 2024-04-25T01:43:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.641+0000 7faf272ee640 1 -- 172.21.15.28:0/1084083048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf20068560 con 0x7faf20100610 2024-04-25T01:43:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.642+0000 7faf167fc640 1 -- 172.21.15.28:0/1084083048 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7faf1c042030 con 0x7faf20100610 2024-04-25T01:43:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.643+0000 7faf167fc640 1 --2- 172.21.15.28:0/1084083048 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7faf0c0a2d30 0x7faf0c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.643+0000 7faf24889640 1 --2- 172.21.15.28:0/1084083048 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7faf0c0a2d30 0x7faf0c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.643+0000 7faf167fc640 1 -- 172.21.15.28:0/1084083048 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9438+0+0 (secure 0 0 0) 0x7faf1c0f00a0 con 0x7faf20100610 2024-04-25T01:43:52.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.644+0000 7faf24889640 1 --2- 172.21.15.28:0/1084083048 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7faf0c0a2d30 0x7faf0c0a51f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7faf20103cf0 tx=0x7faf10005f90 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.647+0000 7faf167fc640 1 -- 172.21.15.28:0/1084083048 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faf1c0bc930 con 0x7faf20100610 2024-04-25T01:43:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.847+0000 7faf272ee640 1 -- 172.21.15.28:0/1084083048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7faf201a6e20 con 0x7faf20100610 2024-04-25T01:43:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.848+0000 7faf167fc640 1 -- 172.21.15.28:0/1084083048 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) ==== 100+0+11 (secure 0 0 0) 0x7faf1c0bc230 con 0x7faf20100610 2024-04-25T01:43:52.838 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T01:43:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.850+0000 7faf272ee640 1 -- 172.21.15.28:0/1084083048 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7faf0c0a2d30 msgr2=0x7faf0c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.850+0000 7faf272ee640 1 --2- 172.21.15.28:0/1084083048 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7faf0c0a2d30 0x7faf0c0a51f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7faf20103cf0 tx=0x7faf10005f90 comp rx=0 tx=0).stop 2024-04-25T01:43:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.850+0000 7faf272ee640 1 -- 172.21.15.28:0/1084083048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf20100610 msgr2=0x7faf201022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.850+0000 7faf272ee640 1 --2- 172.21.15.28:0/1084083048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf20100610 0x7faf201022b0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7faf1c005940 tx=0x7faf1c004270 comp rx=0 tx=0).stop 2024-04-25T01:43:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.850+0000 7faf272ee640 1 -- 172.21.15.28:0/1084083048 shutdown_connections 2024-04-25T01:43:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.850+0000 7faf272ee640 1 --2- 172.21.15.28:0/1084083048 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7faf0c0a2d30 0x7faf0c0a51f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.850+0000 7faf272ee640 1 --2- 172.21.15.28:0/1084083048 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7faf201066b0 0x7faf20102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.850+0000 7faf272ee640 1 --2- 172.21.15.28:0/1084083048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf20100fd0 0x7faf201027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.850+0000 7faf272ee640 1 --2- 172.21.15.28:0/1084083048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf20100610 0x7faf201022b0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.850+0000 7faf272ee640 1 -- 172.21.15.28:0/1084083048 >> 172.21.15.28:0/1084083048 conn(0x7faf20077740 msgr2=0x7faf200ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.850+0000 7faf272ee640 1 -- 172.21.15.28:0/1084083048 shutdown_connections 2024-04-25T01:43:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.850+0000 7faf272ee640 1 -- 172.21.15.28:0/1084083048 wait complete. 2024-04-25T01:43:52.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T01:43:52.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.966+0000 7f8004e4d640 1 -- 172.21.15.28:0/3810425639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8000100fe0 msgr2=0x7f800010d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:52.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.966+0000 7f8004e4d640 1 --2- 172.21.15.28:0/3810425639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8000100fe0 0x7f800010d4f0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8009cd0 tx=0x7f7ff802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.966+0000 7f8004e4d640 1 -- 172.21.15.28:0/3810425639 shutdown_connections 2024-04-25T01:43:52.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.966+0000 7f8004e4d640 1 --2- 172.21.15.28:0/3810425639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8000100fe0 0x7f800010d4f0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.966+0000 7f8004e4d640 1 --2- 172.21.15.28:0/3810425639 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8000100620 0x7f8000100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.966+0000 7f8004e4d640 1 --2- 172.21.15.28:0/3810425639 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8000106630 0x7f8000106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.966+0000 7f8004e4d640 1 -- 172.21.15.28:0/3810425639 >> 172.21.15.28:0/3810425639 conn(0x7f80000776a0 msgr2=0x7f8000077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:52.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.966+0000 7f8004e4d640 1 -- 172.21.15.28:0/3810425639 shutdown_connections 2024-04-25T01:43:52.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.966+0000 7f8004e4d640 1 -- 172.21.15.28:0/3810425639 wait complete. 2024-04-25T01:43:52.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.967+0000 7f8004e4d640 1 Processor -- start 2024-04-25T01:43:52.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.967+0000 7f8004e4d640 1 -- start start 2024-04-25T01:43:52.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.967+0000 7f8004e4d640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8000100620 0x7f80000fd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:52.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.967+0000 7f8004e4d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8000100fe0 0x7f80000fdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:52.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.967+0000 7f8004e4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8000106630 0x7f80000fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:52.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.967+0000 7f8004e4d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8000110a40 con 0x7f8000106630 2024-04-25T01:43:52.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.967+0000 7f8004e4d640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f80001108c0 con 0x7f8000100fe0 2024-04-25T01:43:52.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.967+0000 7f8004e4d640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f8000110bc0 con 0x7f8000100620 2024-04-25T01:43:52.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.967+0000 7f7ffed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8000106630 0x7f80000fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:52.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.967+0000 7f7ffed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8000106630 0x7f80000fe0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52496/0 (socket says 172.21.15.28:52496) 2024-04-25T01:43:52.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.967+0000 7f7ffed9d640 1 -- 172.21.15.28:0/2654413625 learned_addr learned my addr 172.21.15.28:0/2654413625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:52.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.967+0000 7f7ff7fff640 1 --2- 172.21.15.28:0/2654413625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8000100fe0 0x7f80000fdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:52.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.967+0000 7f7ffe59c640 1 --2- 172.21.15.28:0/2654413625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8000100620 0x7f80000fd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:52.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.967+0000 7f7ffed9d640 1 -- 172.21.15.28:0/2654413625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8000100620 msgr2=0x7f80000fd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:52.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.967+0000 7f7ffed9d640 1 --2- 172.21.15.28:0/2654413625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8000100620 0x7f80000fd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.967+0000 7f7ffed9d640 1 -- 172.21.15.28:0/2654413625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8000100fe0 msgr2=0x7f80000fdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:52.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.967+0000 7f7ffed9d640 1 --2- 172.21.15.28:0/2654413625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8000100fe0 0x7f80000fdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:52.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.967+0000 7f7ffed9d640 1 -- 172.21.15.28:0/2654413625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80001a65d0 con 0x7f8000106630 2024-04-25T01:43:52.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.967+0000 7f7ffe59c640 1 --2- 172.21.15.28:0/2654413625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8000100620 0x7f80000fd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:43:52.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.968+0000 7f7ffed9d640 1 --2- 172.21.15.28:0/2654413625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8000106630 0x7f80000fe0d0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8006010 tx=0x7f7ff8003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:52.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.968+0000 7f7ff77fe640 1 -- 172.21.15.28:0/2654413625 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff8004450 con 0x7f8000106630 2024-04-25T01:43:52.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.968+0000 7f8004e4d640 1 -- 172.21.15.28:0/2654413625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80001a6860 con 0x7f8000106630 2024-04-25T01:43:52.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.968+0000 7f7ff77fe640 1 -- 172.21.15.28:0/2654413625 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ff8003e10 con 0x7f8000106630 2024-04-25T01:43:52.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.968+0000 7f7ff77fe640 1 -- 172.21.15.28:0/2654413625 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff80466e0 con 0x7f8000106630 2024-04-25T01:43:52.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.968+0000 7f8004e4d640 1 -- 172.21.15.28:0/2654413625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80001a6db0 con 0x7f8000106630 2024-04-25T01:43:52.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.969+0000 7f8004e4d640 1 -- 172.21.15.28:0/2654413625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80000689b0 con 0x7f8000106630 2024-04-25T01:43:52.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.970+0000 7f7ff77fe640 1 -- 172.21.15.28:0/2654413625 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7ff80305a0 con 0x7f8000106630 2024-04-25T01:43:52.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.971+0000 7f7ff77fe640 1 --2- 172.21.15.28:0/2654413625 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7fec0a2d00 0x7f7fec0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:52.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.971+0000 7f7ffe59c640 1 --2- 172.21.15.28:0/2654413625 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7fec0a2d00 0x7f7fec0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:52.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.971+0000 7f7ff77fe640 1 -- 172.21.15.28:0/2654413625 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9438+0+0 (secure 0 0 0) 0x7f7ff802e030 con 0x7f8000106630 2024-04-25T01:43:52.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.971+0000 7f7ffe59c640 1 --2- 172.21.15.28:0/2654413625 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7fec0a2d00 0x7f7fec0a51c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0006fd0 tx=0x7f7ff0008040 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:52.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:52.974+0000 7f7ff77fe640 1 -- 172.21.15.28:0/2654413625 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7ff804b060 con 0x7f8000106630 2024-04-25T01:43:53.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.157+0000 7f8004e4d640 1 -- 172.21.15.28:0/2654413625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8000109bc0 con 0x7f8000106630 2024-04-25T01:43:53.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.158+0000 7f7ff77fe640 1 -- 172.21.15.28:0/2654413625 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v102) ==== 103+0+11 (secure 0 0 0) 0x7f7ff8044380 con 0x7f8000106630 2024-04-25T01:43:53.148 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T01:43:53.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.160+0000 7f8004e4d640 1 -- 172.21.15.28:0/2654413625 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7fec0a2d00 msgr2=0x7f7fec0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:53.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.160+0000 7f8004e4d640 1 --2- 172.21.15.28:0/2654413625 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7fec0a2d00 0x7f7fec0a51c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0006fd0 tx=0x7f7ff0008040 comp rx=0 tx=0).stop 2024-04-25T01:43:53.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.160+0000 7f8004e4d640 1 -- 172.21.15.28:0/2654413625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8000106630 msgr2=0x7f80000fe0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:53.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.160+0000 7f8004e4d640 1 --2- 172.21.15.28:0/2654413625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8000106630 0x7f80000fe0d0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8006010 tx=0x7f7ff8003c20 comp rx=0 tx=0).stop 2024-04-25T01:43:53.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.160+0000 7f8004e4d640 1 -- 172.21.15.28:0/2654413625 shutdown_connections 2024-04-25T01:43:53.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.160+0000 7f8004e4d640 1 --2- 172.21.15.28:0/2654413625 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7fec0a2d00 0x7f7fec0a51c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.160+0000 7f8004e4d640 1 --2- 172.21.15.28:0/2654413625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8000106630 0x7f80000fe0d0 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.160+0000 7f8004e4d640 1 --2- 172.21.15.28:0/2654413625 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8000100fe0 0x7f80000fdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.160+0000 7f8004e4d640 1 --2- 172.21.15.28:0/2654413625 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f8000100620 0x7f80000fd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.160+0000 7f8004e4d640 1 -- 172.21.15.28:0/2654413625 >> 172.21.15.28:0/2654413625 conn(0x7f80000776a0 msgr2=0x7f800006a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:53.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.160+0000 7f8004e4d640 1 -- 172.21.15.28:0/2654413625 shutdown_connections 2024-04-25T01:43:53.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.160+0000 7f8004e4d640 1 -- 172.21.15.28:0/2654413625 wait complete. 2024-04-25T01:43:53.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T01:43:53.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.274+0000 7f4ded858640 1 -- 172.21.15.28:0/2463457324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de8106aa0 msgr2=0x7f4de810e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:53.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.274+0000 7f4ded858640 1 --2- 172.21.15.28:0/2463457324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de8106aa0 0x7f4de810e6b0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f4de0009ca0 tx=0x7f4de002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.274+0000 7f4ded858640 1 -- 172.21.15.28:0/2463457324 shutdown_connections 2024-04-25T01:43:53.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.274+0000 7f4ded858640 1 --2- 172.21.15.28:0/2463457324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de8106aa0 0x7f4de810e6b0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.274+0000 7f4ded858640 1 --2- 172.21.15.28:0/2463457324 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4de8100c90 0x7f4de81063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.275+0000 7f4ded858640 1 --2- 172.21.15.28:0/2463457324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4de81002e0 0x7f4de81006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.275+0000 7f4ded858640 1 -- 172.21.15.28:0/2463457324 >> 172.21.15.28:0/2463457324 conn(0x7f4de80758d0 msgr2=0x7f4de8075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:53.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.275+0000 7f4ded858640 1 -- 172.21.15.28:0/2463457324 shutdown_connections 2024-04-25T01:43:53.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.275+0000 7f4ded858640 1 -- 172.21.15.28:0/2463457324 wait complete. 2024-04-25T01:43:53.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.275+0000 7f4ded858640 1 Processor -- start 2024-04-25T01:43:53.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.275+0000 7f4ded858640 1 -- start start 2024-04-25T01:43:53.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.275+0000 7f4ded858640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de81002e0 0x7f4de81980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:53.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.275+0000 7f4ded858640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4de8106aa0 0x7f4de81985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:53.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.275+0000 7f4ded858640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4de819c920 0x7f4de819cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:53.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.275+0000 7f4ded858640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4de8111b90 con 0x7f4de81002e0 2024-04-25T01:43:53.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.275+0000 7f4ded858640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4de8111a10 con 0x7f4de8106aa0 2024-04-25T01:43:53.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.275+0000 7f4ded858640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f4de8111d10 con 0x7f4de819c920 2024-04-25T01:43:53.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.276+0000 7f4de6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de81002e0 0x7f4de81980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:53.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.276+0000 7f4de6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de81002e0 0x7f4de81980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52510/0 (socket says 172.21.15.28:52510) 2024-04-25T01:43:53.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.276+0000 7f4de6ffd640 1 -- 172.21.15.28:0/1033159599 learned_addr learned my addr 172.21.15.28:0/1033159599 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:53.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.276+0000 7f4de77fe640 1 --2- 172.21.15.28:0/1033159599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4de819c920 0x7f4de819cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:53.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.276+0000 7f4de67fc640 1 --2- 172.21.15.28:0/1033159599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4de8106aa0 0x7f4de81985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:53.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.276+0000 7f4de6ffd640 1 -- 172.21.15.28:0/1033159599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4de819c920 msgr2=0x7f4de819cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:53.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.276+0000 7f4de6ffd640 1 --2- 172.21.15.28:0/1033159599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4de819c920 0x7f4de819cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.276+0000 7f4de6ffd640 1 -- 172.21.15.28:0/1033159599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4de8106aa0 msgr2=0x7f4de81985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:53.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.276+0000 7f4de6ffd640 1 --2- 172.21.15.28:0/1033159599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4de8106aa0 0x7f4de81985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.276+0000 7f4de6ffd640 1 -- 172.21.15.28:0/1033159599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4de819d410 con 0x7f4de81002e0 2024-04-25T01:43:53.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.276+0000 7f4de67fc640 1 --2- 172.21.15.28:0/1033159599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4de8106aa0 0x7f4de81985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:43:53.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.276+0000 7f4de77fe640 1 --2- 172.21.15.28:0/1033159599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4de819c920 0x7f4de819cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:53.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.276+0000 7f4de6ffd640 1 --2- 172.21.15.28:0/1033159599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de81002e0 0x7f4de81980b0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f4dd800d950 tx=0x7f4dd800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:53.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.276+0000 7f4dec856640 1 -- 172.21.15.28:0/1033159599 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd8017030 con 0x7f4de81002e0 2024-04-25T01:43:53.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.276+0000 7f4dec856640 1 -- 172.21.15.28:0/1033159599 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dd8004020 con 0x7f4de81002e0 2024-04-25T01:43:53.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.276+0000 7f4ded858640 1 -- 172.21.15.28:0/1033159599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4de819d700 con 0x7f4de81002e0 2024-04-25T01:43:53.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.276+0000 7f4dec856640 1 -- 172.21.15.28:0/1033159599 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd80047a0 con 0x7f4de81002e0 2024-04-25T01:43:53.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.276+0000 7f4ded858640 1 -- 172.21.15.28:0/1033159599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4de81a4ef0 con 0x7f4de81002e0 2024-04-25T01:43:53.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.277+0000 7f4ded858640 1 -- 172.21.15.28:0/1033159599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4de80689b0 con 0x7f4de81002e0 2024-04-25T01:43:53.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.279+0000 7f4dec856640 1 -- 172.21.15.28:0/1033159599 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f4dd80041d0 con 0x7f4de81002e0 2024-04-25T01:43:53.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.280+0000 7f4dec856640 1 --2- 172.21.15.28:0/1033159599 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4dd00a2d00 0x7f4dd00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:53.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.280+0000 7f4de67fc640 1 --2- 172.21.15.28:0/1033159599 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4dd00a2d00 0x7f4dd00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:53.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.280+0000 7f4dec856640 1 -- 172.21.15.28:0/1033159599 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9438+0+0 (secure 0 0 0) 0x7f4dd80cf900 con 0x7f4de81002e0 2024-04-25T01:43:53.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.280+0000 7f4de67fc640 1 --2- 172.21.15.28:0/1033159599 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4dd00a2d00 0x7f4dd00a51c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc005fd0 tx=0x7f4ddc0094d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:53.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.284+0000 7f4dec856640 1 -- 172.21.15.28:0/1033159599 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4dd809c190 con 0x7f4de81002e0 2024-04-25T01:43:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.481+0000 7f4ded858640 1 -- 172.21.15.28:0/1033159599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4de81993b0 con 0x7f4de81002e0 2024-04-25T01:43:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.482+0000 7f4dec856640 1 -- 172.21.15.28:0/1033159599 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v102) ==== 81+0+944 (secure 0 0 0) 0x7f4de81993b0 con 0x7f4de81002e0 2024-04-25T01:43:53.472 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/1149553567 2024-04-26T01:33:00.617174+0000 2024-04-25T01:43:53.472 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/500268183 2024-04-26T01:33:00.617174+0000 2024-04-25T01:43:53.472 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3499593842 2024-04-26T01:33:00.617174+0000 2024-04-25T01:43:53.472 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3630315328 2024-04-26T01:33:00.617174+0000 2024-04-25T01:43:53.472 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/1149553567 2024-04-26T01:33:00.617174+0000 2024-04-25T01:43:53.473 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/2407469300 2024-04-26T01:23:20.244799+0000 2024-04-25T01:43:53.473 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/4112717790 2024-04-26T01:23:20.244799+0000 2024-04-25T01:43:53.473 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/2407469300 2024-04-26T01:23:20.244799+0000 2024-04-25T01:43:53.473 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2222781993 2024-04-26T01:23:20.244799+0000 2024-04-25T01:43:53.473 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/3973667708 2024-04-26T01:22:53.256620+0000 2024-04-25T01:43:53.473 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3812371257 2024-04-26T01:33:00.617174+0000 2024-04-25T01:43:53.473 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2339219943 2024-04-26T01:22:53.256620+0000 2024-04-25T01:43:53.473 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/3973667708 2024-04-26T01:22:53.256620+0000 2024-04-25T01:43:53.473 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/747955335 2024-04-26T01:22:53.256620+0000 2024-04-25T01:43:53.473 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/4131336597 2024-04-26T01:22:53.256620+0000 2024-04-25T01:43:53.473 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3232269852 2024-04-26T01:23:20.244799+0000 2024-04-25T01:43:53.474 INFO:teuthology.orchestra.run.smithi028.stderr:listed 16 entries 2024-04-25T01:43:53.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.484+0000 7f4ded858640 1 -- 172.21.15.28:0/1033159599 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4dd00a2d00 msgr2=0x7f4dd00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:53.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.484+0000 7f4ded858640 1 --2- 172.21.15.28:0/1033159599 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4dd00a2d00 0x7f4dd00a51c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc005fd0 tx=0x7f4ddc0094d0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.484+0000 7f4ded858640 1 -- 172.21.15.28:0/1033159599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de81002e0 msgr2=0x7f4de81980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:53.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.484+0000 7f4ded858640 1 --2- 172.21.15.28:0/1033159599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de81002e0 0x7f4de81980b0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f4dd800d950 tx=0x7f4dd800de10 comp rx=0 tx=0).stop 2024-04-25T01:43:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.484+0000 7f4ded858640 1 -- 172.21.15.28:0/1033159599 shutdown_connections 2024-04-25T01:43:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.484+0000 7f4ded858640 1 --2- 172.21.15.28:0/1033159599 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f4dd00a2d00 0x7f4dd00a51c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.484+0000 7f4ded858640 1 --2- 172.21.15.28:0/1033159599 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f4de819c920 0x7f4de819cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.484+0000 7f4ded858640 1 --2- 172.21.15.28:0/1033159599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4de8106aa0 0x7f4de81985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.484+0000 7f4ded858640 1 --2- 172.21.15.28:0/1033159599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de81002e0 0x7f4de81980b0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.484+0000 7f4ded858640 1 -- 172.21.15.28:0/1033159599 >> 172.21.15.28:0/1033159599 conn(0x7f4de80758d0 msgr2=0x7f4de80fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.484+0000 7f4ded858640 1 -- 172.21.15.28:0/1033159599 shutdown_connections 2024-04-25T01:43:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.484+0000 7f4ded858640 1 -- 172.21.15.28:0/1033159599 wait complete. 2024-04-25T01:43:53.475 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:53 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/4079189388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:43:53.476 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:53 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3378138895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:43:53.476 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:53 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1084083048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:43:53.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:53 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/4079189388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:43:53.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:53 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3378138895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:43:53.484 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:53 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1084083048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:43:53.485 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T01:43:53.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:43:53.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T01:43:53.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:53 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/4079189388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:43:53.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:53 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3378138895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:43:53.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:53 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1084083048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:43:53.525 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T01:43:53.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:43:53.525 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T01:43:53.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:43:53.580 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-25T01:43:53.636 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T01:43:53.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:43:53.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T01:43:53.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:43:53.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-04-25T01:43:53.798 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-25T01:43:53.810 INFO:tasks.cephadm:Teardown begin 2024-04-25T01:43:53.810 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T01:43:53.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:43:53.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.923+0000 7fe2c0031640 1 -- 172.21.15.28:0/703647559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b810e150 msgr2=0x7fe2b8110540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:53.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.923+0000 7fe2c0031640 1 --2- 172.21.15.28:0/703647559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b810e150 0x7fe2b8110540 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fe2b4009900 tx=0x7fe2b402a410 comp rx=0 tx=0).stop 2024-04-25T01:43:53.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.923+0000 7fe2c0031640 1 -- 172.21.15.28:0/703647559 shutdown_connections 2024-04-25T01:43:53.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.923+0000 7fe2c0031640 1 --2- 172.21.15.28:0/703647559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b810e150 0x7fe2b8110540 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.923+0000 7fe2c0031640 1 --2- 172.21.15.28:0/703647559 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2b8100ec0 0x7fe2b810dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.923+0000 7fe2c0031640 1 --2- 172.21.15.28:0/703647559 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2b81005a0 0x7fe2b8100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.923+0000 7fe2c0031640 1 -- 172.21.15.28:0/703647559 >> 172.21.15.28:0/703647559 conn(0x7fe2b80777a0 msgr2=0x7fe2b8077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:53.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.923+0000 7fe2c0031640 1 -- 172.21.15.28:0/703647559 shutdown_connections 2024-04-25T01:43:53.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.923+0000 7fe2c0031640 1 -- 172.21.15.28:0/703647559 wait complete. 2024-04-25T01:43:53.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2c0031640 1 Processor -- start 2024-04-25T01:43:53.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2c0031640 1 -- start start 2024-04-25T01:43:53.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2c0031640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2b81005a0 0x7fe2b819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:53.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2c0031640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b8100ec0 0x7fe2b819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:53.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2c0031640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2b810e150 0x7fe2b819ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:53.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2c0031640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b8113d60 con 0x7fe2b8100ec0 2024-04-25T01:43:53.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2c0031640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b8113be0 con 0x7fe2b81005a0 2024-04-25T01:43:53.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2c0031640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b8113ee0 con 0x7fe2b810e150 2024-04-25T01:43:53.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2bd5cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b8100ec0 0x7fe2b819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:53.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2bd5cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b8100ec0 0x7fe2b819a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52518/0 (socket says 172.21.15.28:52518) 2024-04-25T01:43:53.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2bd5cc640 1 -- 172.21.15.28:0/1518430854 learned_addr learned my addr 172.21.15.28:0/1518430854 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:53.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2be5ce640 1 --2- 172.21.15.28:0/1518430854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2b810e150 0x7fe2b819ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:53.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2bddcd640 1 --2- 172.21.15.28:0/1518430854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2b81005a0 0x7fe2b819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:53.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2bd5cc640 1 -- 172.21.15.28:0/1518430854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2b810e150 msgr2=0x7fe2b819ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:53.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2bd5cc640 1 --2- 172.21.15.28:0/1518430854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2b810e150 0x7fe2b819ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2bd5cc640 1 -- 172.21.15.28:0/1518430854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2b81005a0 msgr2=0x7fe2b819a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:53.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2bd5cc640 1 --2- 172.21.15.28:0/1518430854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2b81005a0 0x7fe2b819a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2bd5cc640 1 -- 172.21.15.28:0/1518430854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2b819f190 con 0x7fe2b8100ec0 2024-04-25T01:43:53.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2bddcd640 1 --2- 172.21.15.28:0/1518430854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2b81005a0 0x7fe2b819a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:43:53.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.924+0000 7fe2bd5cc640 1 --2- 172.21.15.28:0/1518430854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b8100ec0 0x7fe2b819a6e0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fe2a800c920 tx=0x7fe2a800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:53.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.925+0000 7fe2aeffd640 1 -- 172.21.15.28:0/1518430854 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a8007c70 con 0x7fe2b8100ec0 2024-04-25T01:43:53.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.925+0000 7fe2aeffd640 1 -- 172.21.15.28:0/1518430854 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2a8007e10 con 0x7fe2b8100ec0 2024-04-25T01:43:53.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.925+0000 7fe2c0031640 1 -- 172.21.15.28:0/1518430854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2b819f480 con 0x7fe2b8100ec0 2024-04-25T01:43:53.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.925+0000 7fe2aeffd640 1 -- 172.21.15.28:0/1518430854 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a8018890 con 0x7fe2b8100ec0 2024-04-25T01:43:53.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.925+0000 7fe2c0031640 1 -- 172.21.15.28:0/1518430854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2b81a6d10 con 0x7fe2b8100ec0 2024-04-25T01:43:53.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.926+0000 7fe2c0031640 1 -- 172.21.15.28:0/1518430854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2b80689b0 con 0x7fe2b8100ec0 2024-04-25T01:43:53.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.927+0000 7fe2aeffd640 1 -- 172.21.15.28:0/1518430854 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fe2a8018a50 con 0x7fe2b8100ec0 2024-04-25T01:43:53.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.927+0000 7fe2aeffd640 1 --2- 172.21.15.28:0/1518430854 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe2a40a2d00 0x7fe2a40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:53.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.927+0000 7fe2aeffd640 1 -- 172.21.15.28:0/1518430854 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9438+0+0 (secure 0 0 0) 0x7fe2a80cfbd0 con 0x7fe2b8100ec0 2024-04-25T01:43:53.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.928+0000 7fe2bddcd640 1 --2- 172.21.15.28:0/1518430854 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe2a40a2d00 0x7fe2a40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:53.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.928+0000 7fe2bddcd640 1 --2- 172.21.15.28:0/1518430854 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe2a40a2d00 0x7fe2a40a51c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0005c40 tx=0x7fe2b00082f0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:53.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:53.931+0000 7fe2aeffd640 1 -- 172.21.15.28:0/1518430854 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe2a8014030 con 0x7fe2b8100ec0 2024-04-25T01:43:54.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.155+0000 7fe2c0031640 1 -- 172.21.15.28:0/1518430854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2b819b4f0 con 0x7fe2b8100ec0 2024-04-25T01:43:54.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.155+0000 7fe2aeffd640 1 -- 172.21.15.28:0/1518430854 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7fe2b819b4f0 con 0x7fe2b8100ec0 2024-04-25T01:43:54.146 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:43:54.146 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:43:54.146 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:43:54.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.157+0000 7fe2c0031640 1 -- 172.21.15.28:0/1518430854 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe2a40a2d00 msgr2=0x7fe2a40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:54.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.157+0000 7fe2c0031640 1 --2- 172.21.15.28:0/1518430854 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe2a40a2d00 0x7fe2a40a51c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe2b0005c40 tx=0x7fe2b00082f0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.157+0000 7fe2c0031640 1 -- 172.21.15.28:0/1518430854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b8100ec0 msgr2=0x7fe2b819a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:54.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.157+0000 7fe2c0031640 1 --2- 172.21.15.28:0/1518430854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b8100ec0 0x7fe2b819a6e0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fe2a800c920 tx=0x7fe2a800cde0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.157+0000 7fe2c0031640 1 -- 172.21.15.28:0/1518430854 shutdown_connections 2024-04-25T01:43:54.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.157+0000 7fe2c0031640 1 --2- 172.21.15.28:0/1518430854 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fe2a40a2d00 0x7fe2a40a51c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.157+0000 7fe2c0031640 1 --2- 172.21.15.28:0/1518430854 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fe2b810e150 0x7fe2b819ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.157+0000 7fe2c0031640 1 --2- 172.21.15.28:0/1518430854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b8100ec0 0x7fe2b819a6e0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.157+0000 7fe2c0031640 1 --2- 172.21.15.28:0/1518430854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe2b81005a0 0x7fe2b819a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.157+0000 7fe2c0031640 1 -- 172.21.15.28:0/1518430854 >> 172.21.15.28:0/1518430854 conn(0x7fe2b80777a0 msgr2=0x7fe2b810f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:54.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.158+0000 7fe2c0031640 1 -- 172.21.15.28:0/1518430854 shutdown_connections 2024-04-25T01:43:54.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.158+0000 7fe2c0031640 1 -- 172.21.15.28:0/1518430854 wait complete. 2024-04-25T01:43:54.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:43:54.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.271+0000 7f70bb55a640 1 -- 172.21.15.28:0/648311985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70b41066b0 msgr2=0x7f70b4106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:54.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.271+0000 7f70bb55a640 1 --2- 172.21.15.28:0/648311985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70b41066b0 0x7f70b4106a90 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f70b0009930 tx=0x7f70b002a420 comp rx=0 tx=0).stop 2024-04-25T01:43:54.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.271+0000 7f70bb55a640 1 -- 172.21.15.28:0/648311985 shutdown_connections 2024-04-25T01:43:54.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.271+0000 7f70bb55a640 1 --2- 172.21.15.28:0/648311985 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70b4100fd0 0x7f70b410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.271+0000 7f70bb55a640 1 --2- 172.21.15.28:0/648311985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70b4100610 0x7f70b4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.271+0000 7f70bb55a640 1 --2- 172.21.15.28:0/648311985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70b41066b0 0x7f70b4106a90 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.271+0000 7f70bb55a640 1 -- 172.21.15.28:0/648311985 >> 172.21.15.28:0/648311985 conn(0x7f70b40769c0 msgr2=0x7f70b4076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:54.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.271+0000 7f70bb55a640 1 -- 172.21.15.28:0/648311985 shutdown_connections 2024-04-25T01:43:54.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.271+0000 7f70bb55a640 1 -- 172.21.15.28:0/648311985 wait complete. 2024-04-25T01:43:54.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.272+0000 7f70bb55a640 1 Processor -- start 2024-04-25T01:43:54.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.272+0000 7f70bb55a640 1 -- start start 2024-04-25T01:43:54.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.272+0000 7f70bb55a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70b4100610 0x7f70b4197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:54.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.272+0000 7f70bb55a640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70b4100fd0 0x7f70b4198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:54.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.272+0000 7f70bb55a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70b41066b0 0x7f70b419c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:54.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.272+0000 7f70bb55a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f70b4110a40 con 0x7f70b41066b0 2024-04-25T01:43:54.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.272+0000 7f70bb55a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f70b41108c0 con 0x7f70b4100610 2024-04-25T01:43:54.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.272+0000 7f70bb55a640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f70b4110bc0 con 0x7f70b4100fd0 2024-04-25T01:43:54.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.272+0000 7f70b9af7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70b41066b0 0x7f70b419c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:54.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.272+0000 7f70b9af7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70b41066b0 0x7f70b419c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52538/0 (socket says 172.21.15.28:52538) 2024-04-25T01:43:54.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.272+0000 7f70b9af7640 1 -- 172.21.15.28:0/2237700687 learned_addr learned my addr 172.21.15.28:0/2237700687 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:54.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.272+0000 7f70b92f6640 1 --2- 172.21.15.28:0/2237700687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70b4100610 0x7f70b4197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:54.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.272+0000 7f70b8af5640 1 --2- 172.21.15.28:0/2237700687 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70b4100fd0 0x7f70b4198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:54.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.273+0000 7f70b9af7640 1 -- 172.21.15.28:0/2237700687 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70b4100fd0 msgr2=0x7f70b4198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:54.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.273+0000 7f70b9af7640 1 --2- 172.21.15.28:0/2237700687 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70b4100fd0 0x7f70b4198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.273+0000 7f70b9af7640 1 -- 172.21.15.28:0/2237700687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70b4100610 msgr2=0x7f70b4197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:54.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.273+0000 7f70b9af7640 1 --2- 172.21.15.28:0/2237700687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70b4100610 0x7f70b4197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.273+0000 7f70b9af7640 1 -- 172.21.15.28:0/2237700687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70b419ce70 con 0x7f70b41066b0 2024-04-25T01:43:54.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.273+0000 7f70b8af5640 1 --2- 172.21.15.28:0/2237700687 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70b4100fd0 0x7f70b4198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:43:54.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.273+0000 7f70b9af7640 1 --2- 172.21.15.28:0/2237700687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70b41066b0 0x7f70b419c740 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f70ac00cce0 tx=0x7f70ac00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:54.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.273+0000 7f70aa7fc640 1 -- 172.21.15.28:0/2237700687 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ac00f450 con 0x7f70b41066b0 2024-04-25T01:43:54.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.273+0000 7f70aa7fc640 1 -- 172.21.15.28:0/2237700687 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70ac00fad0 con 0x7f70b41066b0 2024-04-25T01:43:54.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.273+0000 7f70bb55a640 1 -- 172.21.15.28:0/2237700687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70b419d160 con 0x7f70b41066b0 2024-04-25T01:43:54.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.273+0000 7f70aa7fc640 1 -- 172.21.15.28:0/2237700687 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ac018c10 con 0x7f70b41066b0 2024-04-25T01:43:54.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.273+0000 7f70bb55a640 1 -- 172.21.15.28:0/2237700687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70b41a49f0 con 0x7f70b41066b0 2024-04-25T01:43:54.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.274+0000 7f70bb55a640 1 -- 172.21.15.28:0/2237700687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70b4101d50 con 0x7f70b41066b0 2024-04-25T01:43:54.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.275+0000 7f70aa7fc640 1 -- 172.21.15.28:0/2237700687 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f70ac00f5f0 con 0x7f70b41066b0 2024-04-25T01:43:54.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.275+0000 7f70aa7fc640 1 --2- 172.21.15.28:0/2237700687 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f70a00a2d00 0x7f70a00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:54.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.276+0000 7f70b92f6640 1 --2- 172.21.15.28:0/2237700687 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f70a00a2d00 0x7f70a00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:54.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.276+0000 7f70aa7fc640 1 -- 172.21.15.28:0/2237700687 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9438+0+0 (secure 0 0 0) 0x7f70ac0cff50 con 0x7f70b41066b0 2024-04-25T01:43:54.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.276+0000 7f70b92f6640 1 --2- 172.21.15.28:0/2237700687 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f70a00a2d00 0x7f70a00a51c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f70b0009900 tx=0x7f70b0000bc0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.280+0000 7f70aa7fc640 1 -- 172.21.15.28:0/2237700687 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f70ac014030 con 0x7f70b41066b0 2024-04-25T01:43:54.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:54 smithi130 ceph-mon[32702]: pgmap v337: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:43:54.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:54 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2654413625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:43:54.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:54 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1033159599' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:43:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.505+0000 7f70bb55a640 1 -- 172.21.15.28:0/2237700687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70b41991d0 con 0x7f70b41066b0 2024-04-25T01:43:54.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.506+0000 7f70aa7fc640 1 -- 172.21.15.28:0/2237700687 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11923 (secure 0 0 0) 0x7f70ac09c7e0 con 0x7f70b41066b0 2024-04-25T01:43:54.496 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:43:54.496 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2774244884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2774244884},{"type":"v1","addr":"172.21.15.28:6835","nonce":2774244884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14583,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2622429747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":2622429747},{"type":"v1","addr":"172.21.15.28:6837","nonce":2622429747}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:30:33.696870+0000","modified":"2024-04-25T01:32:58.810131+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24512,"mds_1":24478,"mds_2":24473,"mds_3":24454,"mds_4":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14643":{"gid":14643,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6839/3206225061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":3206225061},{"type":"v1","addr":"172.21.15.28:6839","nonce":3206225061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14655":{"gid":14655,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6841/3666045393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":3666045393},{"type":"v1","addr":"172.21.15.28:6841","nonce":3666045393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24401":{"gid":24401,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6833/983600975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":983600975},{"type":"v1","addr":"172.21.15.32:6833","nonce":983600975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.130:6833/1080169582","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6832","nonce":1080169582},{"type":"v1","addr":"172.21.15.130:6833","nonce":1080169582}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24440":{"gid":24440,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6835/356243399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":356243399},{"type":"v1","addr":"172.21.15.32:6835","nonce":356243399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":14,"addr":"172.21.15.130:6835/348635896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6834","nonce":348635896},{"type":"v1","addr":"172.21.15.130:6835","nonce":348635896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.130:6837/211524954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6836","nonce":211524954},{"type":"v1","addr":"172.21.15.130:6837","nonce":211524954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.32:6837/2771586449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2771586449},{"type":"v1","addr":"172.21.15.32:6837","nonce":2771586449}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.130:6839/463478317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6838","nonce":463478317},{"type":"v1","addr":"172.21.15.130:6839","nonce":463478317}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/2835949714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":2835949714},{"type":"v1","addr":"172.21.15.32:6839","nonce":2835949714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24512,"qdb_cluster":[24473,24478,24454,24512,24442]},"id":1}]} 2024-04-25T01:43:54.496 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 28 2024-04-25T01:43:54.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:54 smithi032 ceph-mon[32461]: pgmap v337: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:43:54.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:54 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2654413625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:43:54.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:54 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1033159599' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:43:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.508+0000 7f70bb55a640 1 -- 172.21.15.28:0/2237700687 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f70a00a2d00 msgr2=0x7f70a00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.508+0000 7f70bb55a640 1 --2- 172.21.15.28:0/2237700687 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f70a00a2d00 0x7f70a00a51c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f70b0009900 tx=0x7f70b0000bc0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.508+0000 7f70bb55a640 1 -- 172.21.15.28:0/2237700687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70b41066b0 msgr2=0x7f70b419c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.508+0000 7f70bb55a640 1 --2- 172.21.15.28:0/2237700687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70b41066b0 0x7f70b419c740 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f70ac00cce0 tx=0x7f70ac00a620 comp rx=0 tx=0).stop 2024-04-25T01:43:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.508+0000 7f70bb55a640 1 -- 172.21.15.28:0/2237700687 shutdown_connections 2024-04-25T01:43:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.508+0000 7f70bb55a640 1 --2- 172.21.15.28:0/2237700687 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f70a00a2d00 0x7f70a00a51c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.508+0000 7f70bb55a640 1 --2- 172.21.15.28:0/2237700687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70b41066b0 0x7f70b419c740 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.508+0000 7f70bb55a640 1 --2- 172.21.15.28:0/2237700687 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f70b4100fd0 0x7f70b4198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.508+0000 7f70bb55a640 1 --2- 172.21.15.28:0/2237700687 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f70b4100610 0x7f70b4197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.508+0000 7f70bb55a640 1 -- 172.21.15.28:0/2237700687 >> 172.21.15.28:0/2237700687 conn(0x7f70b40769c0 msgr2=0x7f70b40ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.508+0000 7f70bb55a640 1 -- 172.21.15.28:0/2237700687 shutdown_connections 2024-04-25T01:43:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.508+0000 7f70bb55a640 1 -- 172.21.15.28:0/2237700687 wait complete. 2024-04-25T01:43:54.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:54 smithi028 ceph-mon[29390]: pgmap v337: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-04-25T01:43:54.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:54 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2654413625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T01:43:54.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:54 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1033159599' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T01:43:54.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:43:54.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.619+0000 7ff46f942640 1 -- 172.21.15.28:0/3046816700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff46810df60 msgr2=0x7ff4681103b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:54.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.620+0000 7ff46f942640 1 --2- 172.21.15.28:0/3046816700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff46810df60 0x7ff4681103b0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7ff464009ca0 tx=0x7ff46402a950 comp rx=0 tx=0).stop 2024-04-25T01:43:54.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.620+0000 7ff46f942640 1 -- 172.21.15.28:0/3046816700 shutdown_connections 2024-04-25T01:43:54.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.620+0000 7ff46f942640 1 --2- 172.21.15.28:0/3046816700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff46810df60 0x7ff4681103b0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.620+0000 7ff46f942640 1 --2- 172.21.15.28:0/3046816700 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff468074490 0x7ff46810da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.620+0000 7ff46f942640 1 --2- 172.21.15.28:0/3046816700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff468073b70 0x7ff468073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.620+0000 7ff46f942640 1 -- 172.21.15.28:0/3046816700 >> 172.21.15.28:0/3046816700 conn(0x7ff4680fd1d0 msgr2=0x7ff4680ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:54.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.620+0000 7ff46f942640 1 -- 172.21.15.28:0/3046816700 shutdown_connections 2024-04-25T01:43:54.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.620+0000 7ff46f942640 1 -- 172.21.15.28:0/3046816700 wait complete. 2024-04-25T01:43:54.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.620+0000 7ff46f942640 1 Processor -- start 2024-04-25T01:43:54.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.620+0000 7ff46f942640 1 -- start start 2024-04-25T01:43:54.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.620+0000 7ff46f942640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff468073b70 0x7ff46819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:54.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.620+0000 7ff46f942640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff468074490 0x7ff46819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:54.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff46f942640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff46810df60 0x7ff46819ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:54.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff46f942640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff468113d00 con 0x7ff46810df60 2024-04-25T01:43:54.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff46f942640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff468113b80 con 0x7ff468074490 2024-04-25T01:43:54.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff46f942640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7ff468113e80 con 0x7ff468073b70 2024-04-25T01:43:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff46dedf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff46810df60 0x7ff46819ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff46dedf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff46810df60 0x7ff46819ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52554/0 (socket says 172.21.15.28:52554) 2024-04-25T01:43:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff46dedf640 1 -- 172.21.15.28:0/2063837769 learned_addr learned my addr 172.21.15.28:0/2063837769 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff46d6de640 1 --2- 172.21.15.28:0/2063837769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff468073b70 0x7ff46819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff46cedd640 1 --2- 172.21.15.28:0/2063837769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff468074490 0x7ff46819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff46dedf640 1 -- 172.21.15.28:0/2063837769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff468073b70 msgr2=0x7ff46819a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff46dedf640 1 --2- 172.21.15.28:0/2063837769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff468073b70 0x7ff46819a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff46dedf640 1 -- 172.21.15.28:0/2063837769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff468074490 msgr2=0x7ff46819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff46dedf640 1 --2- 172.21.15.28:0/2063837769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff468074490 0x7ff46819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff46dedf640 1 -- 172.21.15.28:0/2063837769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff46819f1a0 con 0x7ff46810df60 2024-04-25T01:43:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff46d6de640 1 --2- 172.21.15.28:0/2063837769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff468073b70 0x7ff46819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff46dedf640 1 --2- 172.21.15.28:0/2063837769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff46810df60 0x7ff46819ea70 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7ff464006010 tx=0x7ff46402f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff45e7fc640 1 -- 172.21.15.28:0/2063837769 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff464034040 con 0x7ff46810df60 2024-04-25T01:43:54.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff45e7fc640 1 -- 172.21.15.28:0/2063837769 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff464003e00 con 0x7ff46810df60 2024-04-25T01:43:54.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff45e7fc640 1 -- 172.21.15.28:0/2063837769 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4640456e0 con 0x7ff46810df60 2024-04-25T01:43:54.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff46f942640 1 -- 172.21.15.28:0/2063837769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff46819f430 con 0x7ff46810df60 2024-04-25T01:43:54.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.621+0000 7ff46f942640 1 -- 172.21.15.28:0/2063837769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4681a6d20 con 0x7ff46810df60 2024-04-25T01:43:54.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.622+0000 7ff46f942640 1 -- 172.21.15.28:0/2063837769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff468068560 con 0x7ff46810df60 2024-04-25T01:43:54.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.623+0000 7ff45e7fc640 1 -- 172.21.15.28:0/2063837769 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7ff464037020 con 0x7ff46810df60 2024-04-25T01:43:54.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.624+0000 7ff45e7fc640 1 --2- 172.21.15.28:0/2063837769 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4540a2d80 0x7ff4540a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:54.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.624+0000 7ff46d6de640 1 --2- 172.21.15.28:0/2063837769 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4540a2d80 0x7ff4540a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:54.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.624+0000 7ff45e7fc640 1 -- 172.21.15.28:0/2063837769 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9438+0+0 (secure 0 0 0) 0x7ff4640f08a0 con 0x7ff46810df60 2024-04-25T01:43:54.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.624+0000 7ff46d6de640 1 --2- 172.21.15.28:0/2063837769 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4540a2d80 0x7ff4540a5240 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff4600058c0 tx=0x7ff460005ec0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.628+0000 7ff45e7fc640 1 -- 172.21.15.28:0/2063837769 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff4640bd240 con 0x7ff46810df60 2024-04-25T01:43:54.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.814+0000 7ff46f942640 1 -- 172.21.15.28:0/2063837769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff468109140 con 0x7ff46810df60 2024-04-25T01:43:54.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.815+0000 7ff45e7fc640 1 -- 172.21.15.28:0/2063837769 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+25127 (secure 0 0 0) 0x7ff4640bcb40 con 0x7ff46810df60 2024-04-25T01:43:54.807 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T01:43:54.807 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":102,"fsid":"0e88f60a-02a2-11ef-bc93-c7b262605968","created":"2024-04-25T01:22:26.299667+0000","modified":"2024-04-25T01:43:16.975088+0000","last_up_change":"2024-04-25T01:29:40.962366+0000","last_in_change":"2024-04-25T01:29:20.012917+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:25:48.767414+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:30:32.282430+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:30:33.127049+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T01:30:36.796132+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a052b7a1-e3a4-4d03-a39c-b4c93bca5a8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6803","nonce":2903822772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6805","nonce":2903822772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6809","nonce":2903822772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2903822772},{"type":"v1","addr":"172.21.15.28:6807","nonce":2903822772}]},"public_addr":"172.21.15.28:6803/2903822772","cluster_addr":"172.21.15.28:6805/2903822772","heartbeat_back_addr":"172.21.15.28:6809/2903822772","heartbeat_front_addr":"172.21.15.28:6807/2903822772","state":["exists","up"]},{"osd":1,"uuid":"0a457e4c-75ec-4b92-9aaa-f66036d4b492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6801","nonce":2516196299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6803","nonce":2516196299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6807","nonce":2516196299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2516196299},{"type":"v1","addr":"172.21.15.32:6805","nonce":2516196299}]},"public_addr":"172.21.15.32:6801/2516196299","cluster_addr":"172.21.15.32:6803/2516196299","heartbeat_back_addr":"172.21.15.32:6807/2516196299","heartbeat_front_addr":"172.21.15.32:6805/2516196299","state":["exists","up"]},{"osd":2,"uuid":"2b6e267e-fe68-4971-8f49-3e51b82d1e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6800","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6801","nonce":3497329111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6802","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6803","nonce":3497329111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6806","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6807","nonce":3497329111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6804","nonce":3497329111},{"type":"v1","addr":"172.21.15.130:6805","nonce":3497329111}]},"public_addr":"172.21.15.130:6801/3497329111","cluster_addr":"172.21.15.130:6803/3497329111","heartbeat_back_addr":"172.21.15.130:6807/3497329111","heartbeat_front_addr":"172.21.15.130:6805/3497329111","state":["exists","up"]},{"osd":3,"uuid":"fd309ba8-012b-4058-bcc0-57f61f41c810","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6811","nonce":4005405927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6813","nonce":4005405927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6817","nonce":4005405927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":4005405927},{"type":"v1","addr":"172.21.15.28:6815","nonce":4005405927}]},"public_addr":"172.21.15.28:6811/4005405927","cluster_addr":"172.21.15.28:6813/4005405927","heartbeat_back_addr":"172.21.15.28:6817/4005405927","heartbeat_front_addr":"172.21.15.28:6815/4005405927","state":["exists","up"]},{"osd":4,"uuid":"80f71a46-76da-45eb-b59b-6af112d2e0aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6809","nonce":3679896058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6811","nonce":3679896058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6815","nonce":3679896058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3679896058},{"type":"v1","addr":"172.21.15.32:6813","nonce":3679896058}]},"public_addr":"172.21.15.32:6809/3679896058","cluster_addr":"172.21.15.32:6811/3679896058","heartbeat_back_addr":"172.21.15.32:6815/3679896058","heartbeat_front_addr":"172.21.15.32:6813/3679896058","state":["exists","up"]},{"osd":5,"uuid":"0394f4a0-297e-470a-b491-6da6616a7ed8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6808","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6809","nonce":4253601662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6810","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6811","nonce":4253601662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6814","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6815","nonce":4253601662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6812","nonce":4253601662},{"type":"v1","addr":"172.21.15.130:6813","nonce":4253601662}]},"public_addr":"172.21.15.130:6809/4253601662","cluster_addr":"172.21.15.130:6811/4253601662","heartbeat_back_addr":"172.21.15.130:6815/4253601662","heartbeat_front_addr":"172.21.15.130:6813/4253601662","state":["exists","up"]},{"osd":6,"uuid":"84cb9069-9d19-4d0e-8db4-cb0d41d7e48a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6819","nonce":2330487523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6821","nonce":2330487523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6825","nonce":2330487523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2330487523},{"type":"v1","addr":"172.21.15.28:6823","nonce":2330487523}]},"public_addr":"172.21.15.28:6819/2330487523","cluster_addr":"172.21.15.28:6821/2330487523","heartbeat_back_addr":"172.21.15.28:6825/2330487523","heartbeat_front_addr":"172.21.15.28:6823/2330487523","state":["exists","up"]},{"osd":7,"uuid":"ea0a98a6-80dc-4c7d-b335-b22f9326de2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6817","nonce":1373301181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6819","nonce":1373301181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6823","nonce":1373301181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1373301181},{"type":"v1","addr":"172.21.15.32:6821","nonce":1373301181}]},"public_addr":"172.21.15.32:6817/1373301181","cluster_addr":"172.21.15.32:6819/1373301181","heartbeat_back_addr":"172.21.15.32:6823/1373301181","heartbeat_front_addr":"172.21.15.32:6821/1373301181","state":["exists","up"]},{"osd":8,"uuid":"d5beef33-36c9-4759-a168-c104ee1b5f75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6816","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6817","nonce":2911117678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6818","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6819","nonce":2911117678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6822","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6823","nonce":2911117678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6820","nonce":2911117678},{"type":"v1","addr":"172.21.15.130:6821","nonce":2911117678}]},"public_addr":"172.21.15.130:6817/2911117678","cluster_addr":"172.21.15.130:6819/2911117678","heartbeat_back_addr":"172.21.15.130:6823/2911117678","heartbeat_front_addr":"172.21.15.130:6821/2911117678","state":["exists","up"]},{"osd":9,"uuid":"18835e9d-8405-4586-89b2-75d3e565fad9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6827","nonce":2946375206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6829","nonce":2946375206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6833","nonce":2946375206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2946375206},{"type":"v1","addr":"172.21.15.28:6831","nonce":2946375206}]},"public_addr":"172.21.15.28:6827/2946375206","cluster_addr":"172.21.15.28:6829/2946375206","heartbeat_back_addr":"172.21.15.28:6833/2946375206","heartbeat_front_addr":"172.21.15.28:6831/2946375206","state":["exists","up"]},{"osd":10,"uuid":"22b3f803-fccb-43d3-941e-b7cfd9b11461","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6825","nonce":13541811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6827","nonce":13541811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6831","nonce":13541811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":13541811},{"type":"v1","addr":"172.21.15.32:6829","nonce":13541811}]},"public_addr":"172.21.15.32:6825/13541811","cluster_addr":"172.21.15.32:6827/13541811","heartbeat_back_addr":"172.21.15.32:6831/13541811","heartbeat_front_addr":"172.21.15.32:6829/13541811","state":["exists","up"]},{"osd":11,"uuid":"3c129f89-41b2-4ba9-8e22-0863133613c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6824","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6825","nonce":3995209547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6826","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6827","nonce":3995209547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6830","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6831","nonce":3995209547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.130:6828","nonce":3995209547},{"type":"v1","addr":"172.21.15.130:6829","nonce":3995209547}]},"public_addr":"172.21.15.130:6825/3995209547","cluster_addr":"172.21.15.130:6827/3995209547","heartbeat_back_addr":"172.21.15.130:6831/3995209547","heartbeat_front_addr":"172.21.15.130:6829/3995209547","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:24:55.899781+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:20.643746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:25:44.626025+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:10.557753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:26:36.550045+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:01.291207+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:27.470845+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:27:54.120093+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:19.604094+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:28:45.933830+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:12.542659+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:29:38.342691+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"4.f","mappings":[{"from":5,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/1149553567":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/500268183":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/3499593842":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/3630315328":"2024-04-26T01:33:00.617174+0000","172.21.15.28:6800/1149553567":"2024-04-26T01:33:00.617174+0000","172.21.15.28:6801/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/4112717790":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6800/2407469300":"2024-04-26T01:23:20.244799+0000","172.21.15.28:0/2222781993":"2024-04-26T01:23:20.244799+0000","172.21.15.28:6801/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3812371257":"2024-04-26T01:33:00.617174+0000","172.21.15.28:0/2339219943":"2024-04-26T01:22:53.256620+0000","172.21.15.28:6800/3973667708":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/747955335":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/4131336597":"2024-04-26T01:22:53.256620+0000","172.21.15.28:0/3232269852":"2024-04-26T01:23:20.244799+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:43:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.818+0000 7ff46f942640 1 -- 172.21.15.28:0/2063837769 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4540a2d80 msgr2=0x7ff4540a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.818+0000 7ff46f942640 1 --2- 172.21.15.28:0/2063837769 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4540a2d80 0x7ff4540a5240 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff4600058c0 tx=0x7ff460005ec0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.818+0000 7ff46f942640 1 -- 172.21.15.28:0/2063837769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff46810df60 msgr2=0x7ff46819ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.818+0000 7ff46f942640 1 --2- 172.21.15.28:0/2063837769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff46810df60 0x7ff46819ea70 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7ff464006010 tx=0x7ff46402f710 comp rx=0 tx=0).stop 2024-04-25T01:43:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.818+0000 7ff46f942640 1 -- 172.21.15.28:0/2063837769 shutdown_connections 2024-04-25T01:43:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.818+0000 7ff46f942640 1 --2- 172.21.15.28:0/2063837769 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7ff4540a2d80 0x7ff4540a5240 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.818+0000 7ff46f942640 1 --2- 172.21.15.28:0/2063837769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff46810df60 0x7ff46819ea70 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.818+0000 7ff46f942640 1 --2- 172.21.15.28:0/2063837769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff468074490 0x7ff46819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.818+0000 7ff46f942640 1 --2- 172.21.15.28:0/2063837769 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7ff468073b70 0x7ff46819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.818+0000 7ff46f942640 1 -- 172.21.15.28:0/2063837769 >> 172.21.15.28:0/2063837769 conn(0x7ff4680fd1d0 msgr2=0x7ff4680fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.818+0000 7ff46f942640 1 -- 172.21.15.28:0/2063837769 shutdown_connections 2024-04-25T01:43:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.818+0000 7ff46f942640 1 -- 172.21.15.28:0/2063837769 wait complete. 2024-04-25T01:43:54.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T01:43:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.932+0000 7f1feb295640 1 -- 172.21.15.28:0/3038602661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fe4106630 msgr2=0x7f1fe4106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.932+0000 7f1feb295640 1 --2- 172.21.15.28:0/3038602661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fe4106630 0x7f1fe4106a10 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0009900 tx=0x7f1fe002a410 comp rx=0 tx=0).stop 2024-04-25T01:43:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.932+0000 7f1feb295640 1 -- 172.21.15.28:0/3038602661 shutdown_connections 2024-04-25T01:43:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.932+0000 7f1feb295640 1 --2- 172.21.15.28:0/3038602661 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fe4100fe0 0x7f1fe410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.932+0000 7f1feb295640 1 --2- 172.21.15.28:0/3038602661 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fe4100620 0x7f1fe4100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.932+0000 7f1feb295640 1 --2- 172.21.15.28:0/3038602661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fe4106630 0x7f1fe4106a10 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.932+0000 7f1feb295640 1 -- 172.21.15.28:0/3038602661 >> 172.21.15.28:0/3038602661 conn(0x7f1fe40776a0 msgr2=0x7f1fe4077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.932+0000 7f1feb295640 1 -- 172.21.15.28:0/3038602661 shutdown_connections 2024-04-25T01:43:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.932+0000 7f1feb295640 1 -- 172.21.15.28:0/3038602661 wait complete. 2024-04-25T01:43:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.933+0000 7f1feb295640 1 Processor -- start 2024-04-25T01:43:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.933+0000 7f1feb295640 1 -- start start 2024-04-25T01:43:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.933+0000 7f1feb295640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fe4100620 0x7f1fe41980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.933+0000 7f1feb295640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fe4100fe0 0x7f1fe4198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.933+0000 7f1feb295640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fe4106630 0x7f1fe419c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.933+0000 7f1feb295640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe4111b40 con 0x7f1fe4100fe0 2024-04-25T01:43:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.933+0000 7f1feb295640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe41119c0 con 0x7f1fe4100620 2024-04-25T01:43:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.933+0000 7f1feb295640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe4111cc0 con 0x7f1fe4106630 2024-04-25T01:43:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.933+0000 7f1fe8830640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fe4100fe0 0x7f1fe4198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.933+0000 7f1fe8830640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fe4100fe0 0x7f1fe4198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52572/0 (socket says 172.21.15.28:52572) 2024-04-25T01:43:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.933+0000 7f1fe8830640 1 -- 172.21.15.28:0/3002940358 learned_addr learned my addr 172.21.15.28:0/3002940358 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.933+0000 7f1fe9832640 1 --2- 172.21.15.28:0/3002940358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fe4106630 0x7f1fe419c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.933+0000 7f1fe9031640 1 --2- 172.21.15.28:0/3002940358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fe4100620 0x7f1fe41980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.933+0000 7f1fe8830640 1 -- 172.21.15.28:0/3002940358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fe4106630 msgr2=0x7f1fe419c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.933+0000 7f1fe8830640 1 --2- 172.21.15.28:0/3002940358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fe4106630 0x7f1fe419c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.933+0000 7f1fe8830640 1 -- 172.21.15.28:0/3002940358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fe4100620 msgr2=0x7f1fe41980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.933+0000 7f1fe8830640 1 --2- 172.21.15.28:0/3002940358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fe4100620 0x7f1fe41980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.933+0000 7f1fe8830640 1 -- 172.21.15.28:0/3002940358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe419d0b0 con 0x7f1fe4100fe0 2024-04-25T01:43:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.934+0000 7f1fe8830640 1 --2- 172.21.15.28:0/3002940358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fe4100fe0 0x7f1fe4198600 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f1fd400d770 tx=0x7f1fd400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.934+0000 7f1fda7fc640 1 -- 172.21.15.28:0/3002940358 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd4017070 con 0x7f1fe4100fe0 2024-04-25T01:43:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.934+0000 7f1fda7fc640 1 -- 172.21.15.28:0/3002940358 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fd4005500 con 0x7f1fe4100fe0 2024-04-25T01:43:54.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.934+0000 7f1feb295640 1 -- 172.21.15.28:0/3002940358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe419d3a0 con 0x7f1fe4100fe0 2024-04-25T01:43:54.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.934+0000 7f1fda7fc640 1 -- 172.21.15.28:0/3002940358 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd4004e00 con 0x7f1fe4100fe0 2024-04-25T01:43:54.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.934+0000 7f1feb295640 1 -- 172.21.15.28:0/3002940358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fe41a4c30 con 0x7f1fe4100fe0 2024-04-25T01:43:54.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.935+0000 7f1feb295640 1 -- 172.21.15.28:0/3002940358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fe40689b0 con 0x7f1fe4100fe0 2024-04-25T01:43:54.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.936+0000 7f1fda7fc640 1 -- 172.21.15.28:0/3002940358 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f1fd4004fa0 con 0x7f1fe4100fe0 2024-04-25T01:43:54.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.937+0000 7f1fda7fc640 1 --2- 172.21.15.28:0/3002940358 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1fd00a2d80 0x7f1fd00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:54.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.937+0000 7f1fe9031640 1 --2- 172.21.15.28:0/3002940358 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1fd00a2d80 0x7f1fd00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:54.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.937+0000 7f1fda7fc640 1 -- 172.21.15.28:0/3002940358 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9438+0+0 (secure 0 0 0) 0x7f1fd40cf1f0 con 0x7f1fe4100fe0 2024-04-25T01:43:54.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.937+0000 7f1fe9031640 1 --2- 172.21.15.28:0/3002940358 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1fd00a2d80 0x7f1fd00a5240 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0005ab0 tx=0x7f1fe0005a20 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:54.941+0000 7f1fda7fc640 1 -- 172.21.15.28:0/3002940358 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1fd409ba80 con 0x7f1fe4100fe0 2024-04-25T01:43:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.164+0000 7f1feb295640 1 -- 172.21.15.28:0/3002940358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f1fe4108bf0 con 0x7f1fe4100fe0 2024-04-25T01:43:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.166+0000 7f1fda7fc640 1 -- 172.21.15.28:0/3002940358 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v29) ==== 161+0+0 (secure 0 0 0) 0x7f1fd409b380 con 0x7f1fe4100fe0 2024-04-25T01:43:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T01:43:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.170+0000 7f1feb295640 1 -- 172.21.15.28:0/3002940358 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1fd00a2d80 msgr2=0x7f1fd00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.170+0000 7f1feb295640 1 --2- 172.21.15.28:0/3002940358 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1fd00a2d80 0x7f1fd00a5240 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0005ab0 tx=0x7f1fe0005a20 comp rx=0 tx=0).stop 2024-04-25T01:43:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.170+0000 7f1feb295640 1 -- 172.21.15.28:0/3002940358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fe4100fe0 msgr2=0x7f1fe4198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.170+0000 7f1feb295640 1 --2- 172.21.15.28:0/3002940358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fe4100fe0 0x7f1fe4198600 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f1fd400d770 tx=0x7f1fd400dc30 comp rx=0 tx=0).stop 2024-04-25T01:43:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.170+0000 7f1feb295640 1 -- 172.21.15.28:0/3002940358 shutdown_connections 2024-04-25T01:43:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.170+0000 7f1feb295640 1 --2- 172.21.15.28:0/3002940358 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f1fd00a2d80 0x7f1fd00a5240 secure :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0005ab0 tx=0x7f1fe0005a20 comp rx=0 tx=0).stop 2024-04-25T01:43:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.170+0000 7f1feb295640 1 --2- 172.21.15.28:0/3002940358 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f1fe4106630 0x7f1fe419c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.170+0000 7f1feb295640 1 --2- 172.21.15.28:0/3002940358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fe4100fe0 0x7f1fe4198600 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.170+0000 7f1feb295640 1 --2- 172.21.15.28:0/3002940358 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1fe4100620 0x7f1fe41980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.170+0000 7f1feb295640 1 -- 172.21.15.28:0/3002940358 >> 172.21.15.28:0/3002940358 conn(0x7f1fe40776a0 msgr2=0x7f1fe4103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.171+0000 7f1feb295640 1 -- 172.21.15.28:0/3002940358 shutdown_connections 2024-04-25T01:43:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.171+0000 7f1feb295640 1 -- 172.21.15.28:0/3002940358 wait complete. 2024-04-25T01:43:55.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T01:43:55.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.297+0000 7f0d9e596640 1 -- 172.21.15.28:0/1494589972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d98106630 msgr2=0x7f0d98106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:55.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.297+0000 7f0d9e596640 1 --2- 172.21.15.28:0/1494589972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d98106630 0x7f0d98106a10 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f0d88009930 tx=0x7f0d8802a420 comp rx=0 tx=0).stop 2024-04-25T01:43:55.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.297+0000 7f0d9e596640 1 -- 172.21.15.28:0/1494589972 shutdown_connections 2024-04-25T01:43:55.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.297+0000 7f0d9e596640 1 --2- 172.21.15.28:0/1494589972 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d98100fe0 0x7f0d9810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:55.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.297+0000 7f0d9e596640 1 --2- 172.21.15.28:0/1494589972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d98100620 0x7f0d98100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:55.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.297+0000 7f0d9e596640 1 --2- 172.21.15.28:0/1494589972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d98106630 0x7f0d98106a10 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:55.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.297+0000 7f0d9e596640 1 -- 172.21.15.28:0/1494589972 >> 172.21.15.28:0/1494589972 conn(0x7f0d980776a0 msgr2=0x7f0d98077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:55.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.297+0000 7f0d9e596640 1 -- 172.21.15.28:0/1494589972 shutdown_connections 2024-04-25T01:43:55.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.297+0000 7f0d9e596640 1 -- 172.21.15.28:0/1494589972 wait complete. 2024-04-25T01:43:55.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.297+0000 7f0d9e596640 1 Processor -- start 2024-04-25T01:43:55.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.297+0000 7f0d9e596640 1 -- start start 2024-04-25T01:43:55.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.298+0000 7f0d9e596640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d98100620 0x7f0d98198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:55.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.298+0000 7f0d9e596640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d98100fe0 0x7f0d98199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:55.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.298+0000 7f0d9e596640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d98106630 0x7f0d981a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:55.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.298+0000 7f0d9e596640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d98110a00 con 0x7f0d98100fe0 2024-04-25T01:43:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.298+0000 7f0d9e596640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0d98110880 con 0x7f0d98100620 2024-04-25T01:43:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.298+0000 7f0d9e596640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f0d98110b80 con 0x7f0d98106630 2024-04-25T01:43:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.298+0000 7f0d977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d98100fe0 0x7f0d98199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.298+0000 7f0d977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d98100fe0 0x7f0d98199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52624/0 (socket says 172.21.15.28:52624) 2024-04-25T01:43:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.298+0000 7f0d977fe640 1 -- 172.21.15.28:0/3932719119 learned_addr learned my addr 172.21.15.28:0/3932719119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.298+0000 7f0d9cb33640 1 --2- 172.21.15.28:0/3932719119 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d98106630 0x7f0d981a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.298+0000 7f0d97fff640 1 --2- 172.21.15.28:0/3932719119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d98100620 0x7f0d98198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.298+0000 7f0d977fe640 1 -- 172.21.15.28:0/3932719119 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d98106630 msgr2=0x7f0d981a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.298+0000 7f0d977fe640 1 --2- 172.21.15.28:0/3932719119 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d98106630 0x7f0d981a3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.298+0000 7f0d977fe640 1 -- 172.21.15.28:0/3932719119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d98100620 msgr2=0x7f0d98198ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.298+0000 7f0d977fe640 1 --2- 172.21.15.28:0/3932719119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d98100620 0x7f0d98198ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.298+0000 7f0d977fe640 1 -- 172.21.15.28:0/3932719119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d981a44a0 con 0x7f0d98100fe0 2024-04-25T01:43:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.298+0000 7f0d977fe640 1 --2- 172.21.15.28:0/3932719119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d98100fe0 0x7f0d98199420 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f0d8000d950 tx=0x7f0d8000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.299+0000 7f0d957fa640 1 -- 172.21.15.28:0/3932719119 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d80017030 con 0x7f0d98100fe0 2024-04-25T01:43:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.299+0000 7f0d957fa640 1 -- 172.21.15.28:0/3932719119 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d80004020 con 0x7f0d98100fe0 2024-04-25T01:43:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.299+0000 7f0d957fa640 1 -- 172.21.15.28:0/3932719119 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d800047a0 con 0x7f0d98100fe0 2024-04-25T01:43:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.299+0000 7f0d9e596640 1 -- 172.21.15.28:0/3932719119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d981a4670 con 0x7f0d98100fe0 2024-04-25T01:43:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.299+0000 7f0d9e596640 1 -- 172.21.15.28:0/3932719119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d981a4b30 con 0x7f0d98100fe0 2024-04-25T01:43:55.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.300+0000 7f0d9e596640 1 -- 172.21.15.28:0/3932719119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d980689b0 con 0x7f0d98100fe0 2024-04-25T01:43:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.301+0000 7f0d957fa640 1 -- 172.21.15.28:0/3932719119 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f0d800041d0 con 0x7f0d98100fe0 2024-04-25T01:43:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.301+0000 7f0d957fa640 1 --2- 172.21.15.28:0/3932719119 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0d840a2d00 0x7f0d840a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.302+0000 7f0d97fff640 1 --2- 172.21.15.28:0/3932719119 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0d840a2d00 0x7f0d840a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.302+0000 7f0d957fa640 1 -- 172.21.15.28:0/3932719119 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9780+0+0 (secure 0 0 0) 0x7f0d800cf320 con 0x7f0d98100fe0 2024-04-25T01:43:55.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.302+0000 7f0d97fff640 1 --2- 172.21.15.28:0/3932719119 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0d840a2d00 0x7f0d840a51c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0d88000c00 tx=0x7f0d88005a20 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:55.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.306+0000 7f0d957fa640 1 -- 172.21.15.28:0/3932719119 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0d8009ba50 con 0x7f0d98100fe0 2024-04-25T01:43:55.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:55 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1518430854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:43:55.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:55 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2237700687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:43:55.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:55 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/2063837769' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:43:55.472 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 25 01:43:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-c[62762]: ignoring --setuser ceph since I am not root 2024-04-25T01:43:55.472 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 25 01:43:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-c[62762]: ignoring --setgroup ceph since I am not root 2024-04-25T01:43:55.472 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 25 01:43:55 smithi130 ceph-mds[62766]: -- 172.21.15.130:0/2496685757 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bd19fdba40 con 0x55bd19faec00 2024-04-25T01:43:55.473 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 25 01:43:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-c[62762]: starting mds.c at 2024-04-25T01:43:55.473 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 25 01:43:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-l[62135]: ignoring --setuser ceph since I am not root 2024-04-25T01:43:55.473 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 25 01:43:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-l[62135]: ignoring --setgroup ceph since I am not root 2024-04-25T01:43:55.473 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 25 01:43:55 smithi130 ceph-mds[62139]: -- 172.21.15.130:0/3005791528 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55db308c3a40 con 0x55db30896c00 2024-04-25T01:43:55.473 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 25 01:43:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-l[62135]: starting mds.l at 2024-04-25T01:43:55.473 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:43:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-f[62451]: ignoring --setuser ceph since I am not root 2024-04-25T01:43:55.473 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:43:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-f[62451]: ignoring --setgroup ceph since I am not root 2024-04-25T01:43:55.474 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:43:55 smithi130 ceph-mds[62455]: -- 172.21.15.130:0/1958554938 <== mon.1 v2:172.21.15.130:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56523cee9a40 con 0x56523cebc800 2024-04-25T01:43:55.474 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:43:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-f[62451]: starting mds.f at 2024-04-25T01:43:55.474 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 25 01:43:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-i[63068]: ignoring --setuser ceph since I am not root 2024-04-25T01:43:55.474 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 25 01:43:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-i[63068]: ignoring --setgroup ceph since I am not root 2024-04-25T01:43:55.474 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 25 01:43:55 smithi130 ceph-mds[63072]: -- 172.21.15.130:0/1509170937 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562058fd3a40 con 0x562058fa6800 2024-04-25T01:43:55.474 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 25 01:43:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-i[63068]: starting mds.i at 2024-04-25T01:43:55.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:55 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1518430854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:43:55.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:55 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2237700687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:43:55.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:55 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/2063837769' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:43:55.497 INFO:journalctl@ceph.mds.e.smithi032.stdout:Apr 25 01:43:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-e[62540]: ignoring --setuser ceph since I am not root 2024-04-25T01:43:55.497 INFO:journalctl@ceph.mds.e.smithi032.stdout:Apr 25 01:43:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-e[62540]: ignoring --setgroup ceph since I am not root 2024-04-25T01:43:55.498 INFO:journalctl@ceph.mds.e.smithi032.stdout:Apr 25 01:43:55 smithi032 ceph-mds[62544]: -- 172.21.15.32:0/1764625927 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563fc0a6fa40 con 0x563fc0a43000 2024-04-25T01:43:55.498 INFO:journalctl@ceph.mds.e.smithi032.stdout:Apr 25 01:43:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-e[62540]: starting mds.e at 2024-04-25T01:43:55.498 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:43:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-h[62852]: ignoring --setuser ceph since I am not root 2024-04-25T01:43:55.498 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:43:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-h[62852]: ignoring --setgroup ceph since I am not root 2024-04-25T01:43:55.498 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:43:55 smithi032 ceph-mds[62856]: -- 172.21.15.32:0/1342813313 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55619f079a40 con 0x55619f04c800 2024-04-25T01:43:55.498 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:43:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-h[62852]: starting mds.h at 2024-04-25T01:43:55.498 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:43:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-b[63160]: ignoring --setuser ceph since I am not root 2024-04-25T01:43:55.499 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:43:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-b[63160]: ignoring --setgroup ceph since I am not root 2024-04-25T01:43:55.499 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:43:55 smithi032 ceph-mds[63164]: -- 172.21.15.32:0/3685381907 <== mon.2 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558189f69a40 con 0x558189f3cc00 2024-04-25T01:43:55.499 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:43:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-b[63160]: starting mds.b at 2024-04-25T01:43:55.499 INFO:journalctl@ceph.mds.k.smithi032.stdout:Apr 25 01:43:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-k[62215]: ignoring --setuser ceph since I am not root 2024-04-25T01:43:55.499 INFO:journalctl@ceph.mds.k.smithi032.stdout:Apr 25 01:43:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-k[62215]: ignoring --setgroup ceph since I am not root 2024-04-25T01:43:55.499 INFO:journalctl@ceph.mds.k.smithi032.stdout:Apr 25 01:43:55 smithi032 ceph-mds[62219]: -- 172.21.15.32:0/1569865998 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bddf745a40 con 0x55bddf718800 2024-04-25T01:43:55.499 INFO:journalctl@ceph.mds.k.smithi032.stdout:Apr 25 01:43:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-k[62215]: starting mds.k at 2024-04-25T01:43:55.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:55.529+0000 7f0d9e596640 1 -- 172.21.15.28:0/3932719119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f0d9802d170 con 0x7f0d98100fe0 2024-04-25T01:43:55.575 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 25 01:43:55 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-g[68909]: ignoring --setuser ceph since I am not root 2024-04-25T01:43:55.575 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 25 01:43:55 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-g[68909]: ignoring --setgroup ceph since I am not root 2024-04-25T01:43:55.575 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 25 01:43:55 smithi028 ceph-mds[68913]: -- 172.21.15.28:0/1037051400 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d172177a40 con 0x55d17214ac00 2024-04-25T01:43:55.575 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 25 01:43:55 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-g[68909]: starting mds.g at 2024-04-25T01:43:55.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:55 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1518430854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:43:55.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:55 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2237700687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:43:55.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:55 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/2063837769' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:43:55.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:55 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-a[29386]: 2024-04-25T01:43:55.165+0000 7f2915b73700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:43:55.576 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:43:55 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-a[69257]: ignoring --setuser ceph since I am not root 2024-04-25T01:43:55.576 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:43:55 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-a[69257]: ignoring --setgroup ceph since I am not root 2024-04-25T01:43:55.576 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:43:55 smithi028 ceph-mds[69261]: -- 172.21.15.28:0/1148012642 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55be7ccada40 con 0x55be7cc80800 2024-04-25T01:43:55.576 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:43:55 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-a[69257]: starting mds.a at 2024-04-25T01:43:56.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.168+0000 7f0d957fa640 1 -- 172.21.15.28:0/3932719119 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v30) ==== 104+0+0 (secure 0 0 0) 0x7f0d8009b350 con 0x7f0d98100fe0 2024-04-25T01:43:56.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.171+0000 7f0d9e596640 1 -- 172.21.15.28:0/3932719119 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0d840a2d00 msgr2=0x7f0d840a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:56.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.171+0000 7f0d9e596640 1 --2- 172.21.15.28:0/3932719119 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0d840a2d00 0x7f0d840a51c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0d88000c00 tx=0x7f0d88005a20 comp rx=0 tx=0).stop 2024-04-25T01:43:56.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.171+0000 7f0d9e596640 1 -- 172.21.15.28:0/3932719119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d98100fe0 msgr2=0x7f0d98199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:56.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.171+0000 7f0d9e596640 1 --2- 172.21.15.28:0/3932719119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d98100fe0 0x7f0d98199420 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f0d8000d950 tx=0x7f0d8000de10 comp rx=0 tx=0).stop 2024-04-25T01:43:56.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.171+0000 7f0d9e596640 1 -- 172.21.15.28:0/3932719119 shutdown_connections 2024-04-25T01:43:56.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.171+0000 7f0d9e596640 1 --2- 172.21.15.28:0/3932719119 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f0d840a2d00 0x7f0d840a51c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:56.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.171+0000 7f0d9e596640 1 --2- 172.21.15.28:0/3932719119 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f0d98106630 0x7f0d981a3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:56.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.171+0000 7f0d9e596640 1 --2- 172.21.15.28:0/3932719119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d98100fe0 0x7f0d98199420 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:56.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.171+0000 7f0d9e596640 1 --2- 172.21.15.28:0/3932719119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0d98100620 0x7f0d98198ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:56.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.171+0000 7f0d9e596640 1 -- 172.21.15.28:0/3932719119 >> 172.21.15.28:0/3932719119 conn(0x7f0d980776a0 msgr2=0x7f0d98103230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:56.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.171+0000 7f0d9e596640 1 -- 172.21.15.28:0/3932719119 shutdown_connections 2024-04-25T01:43:56.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.171+0000 7f0d9e596640 1 -- 172.21.15.28:0/3932719119 wait complete. 2024-04-25T01:43:56.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T01:43:56.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.301+0000 7fc39f742640 1 -- 172.21.15.28:0/3989729323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc398100610 msgr2=0x7fc398100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:56.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.301+0000 7fc39f742640 1 --2- 172.21.15.28:0/3989729323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc398100610 0x7fc398100a90 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fc388009930 tx=0x7fc38802a420 comp rx=0 tx=0).stop 2024-04-25T01:43:56.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.301+0000 7fc39f742640 1 -- 172.21.15.28:0/3989729323 shutdown_connections 2024-04-25T01:43:56.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.301+0000 7fc39f742640 1 --2- 172.21.15.28:0/3989729323 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc398100fd0 0x7fc39810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:56.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.301+0000 7fc39f742640 1 --2- 172.21.15.28:0/3989729323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc398100610 0x7fc398100a90 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:56.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.301+0000 7fc39f742640 1 --2- 172.21.15.28:0/3989729323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3981066b0 0x7fc398106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:56.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.301+0000 7fc39f742640 1 -- 172.21.15.28:0/3989729323 >> 172.21.15.28:0/3989729323 conn(0x7fc398077740 msgr2=0x7fc398077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:56.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.301+0000 7fc39f742640 1 -- 172.21.15.28:0/3989729323 shutdown_connections 2024-04-25T01:43:56.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.301+0000 7fc39f742640 1 -- 172.21.15.28:0/3989729323 wait complete. 2024-04-25T01:43:56.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.301+0000 7fc39f742640 1 Processor -- start 2024-04-25T01:43:56.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.301+0000 7fc39f742640 1 -- start start 2024-04-25T01:43:56.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc39f742640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc398100610 0x7fc3981024e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:56.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc39f742640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc398100fd0 0x7fc398102a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:56.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc39f742640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3981066b0 0x7fc398102f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:56.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc39f742640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3981109f0 con 0x7fc398100610 2024-04-25T01:43:56.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc39f742640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc398110870 con 0x7fc3981066b0 2024-04-25T01:43:56.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc39f742640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7fc398110b70 con 0x7fc398100fd0 2024-04-25T01:43:56.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc39d4de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc398100610 0x7fc3981024e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:56.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc39d4de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc398100610 0x7fc3981024e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52650/0 (socket says 172.21.15.28:52650) 2024-04-25T01:43:56.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc39d4de640 1 -- 172.21.15.28:0/1011259171 learned_addr learned my addr 172.21.15.28:0/1011259171 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:56.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc39ccdd640 1 --2- 172.21.15.28:0/1011259171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc398100fd0 0x7fc398102a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:56.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc39d4de640 1 -- 172.21.15.28:0/1011259171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc398100fd0 msgr2=0x7fc398102a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:56.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc39d4de640 1 --2- 172.21.15.28:0/1011259171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc398100fd0 0x7fc398102a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:56.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc39d4de640 1 -- 172.21.15.28:0/1011259171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3981066b0 msgr2=0x7fc398102f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:56.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc39d4de640 1 --2- 172.21.15.28:0/1011259171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3981066b0 0x7fc398102f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:56.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc39d4de640 1 -- 172.21.15.28:0/1011259171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3981a6440 con 0x7fc398100610 2024-04-25T01:43:56.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc39ccdd640 1 --2- 172.21.15.28:0/1011259171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc398100fd0 0x7fc398102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:43:56.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc39d4de640 1 --2- 172.21.15.28:0/1011259171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc398100610 0x7fc3981024e0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fc39400d950 tx=0x7fc39400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:56.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc38e7fc640 1 -- 172.21.15.28:0/1011259171 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc394017020 con 0x7fc398100610 2024-04-25T01:43:56.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc38e7fc640 1 -- 172.21.15.28:0/1011259171 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc394009c30 con 0x7fc398100610 2024-04-25T01:43:56.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc39f742640 1 -- 172.21.15.28:0/1011259171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3981a6730 con 0x7fc398100610 2024-04-25T01:43:56.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc38e7fc640 1 -- 172.21.15.28:0/1011259171 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc39400f740 con 0x7fc398100610 2024-04-25T01:43:56.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.302+0000 7fc39f742640 1 -- 172.21.15.28:0/1011259171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3981a6ca0 con 0x7fc398100610 2024-04-25T01:43:56.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.303+0000 7fc39f742640 1 -- 172.21.15.28:0/1011259171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc398068560 con 0x7fc398100610 2024-04-25T01:43:56.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.304+0000 7fc38e7fc640 1 -- 172.21.15.28:0/1011259171 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7fc39401a020 con 0x7fc398100610 2024-04-25T01:43:56.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.305+0000 7fc38e7fc640 1 --2- 172.21.15.28:0/1011259171 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc3840a2d80 0x7fc3840a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:56.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.305+0000 7fc38e7fc640 1 -- 172.21.15.28:0/1011259171 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9780+0+0 (secure 0 0 0) 0x7fc3940cf320 con 0x7fc398100610 2024-04-25T01:43:56.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.305+0000 7fc39ccdd640 1 --2- 172.21.15.28:0/1011259171 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc3840a2d80 0x7fc3840a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:56.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.306+0000 7fc39ccdd640 1 --2- 172.21.15.28:0/1011259171 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc3840a2d80 0x7fc3840a5240 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc388000c00 tx=0x7fc388005a20 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:56.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.310+0000 7fc38e7fc640 1 -- 172.21.15.28:0/1011259171 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc394021070 con 0x7fc398100610 2024-04-25T01:43:56.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:56 smithi130 ceph-mon[32702]: pgmap v338: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:56.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:56 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3002940358' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T01:43:56.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:56 smithi130 ceph-mon[32702]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T01:43:56.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:56 smithi130 ceph-mon[32702]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:43:56.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:56 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3002940358' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T01:43:56.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:56 smithi130 ceph-mon[32702]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T01:43:56.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:56 smithi130 ceph-mon[32702]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-25T01:43:56.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:56 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3932719119' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:43:56.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:56.493+0000 7fc39f742640 1 -- 172.21.15.28:0/1011259171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc3981a70b0 con 0x7fc398100610 2024-04-25T01:43:56.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:56 smithi032 ceph-mon[32461]: pgmap v338: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:56.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:56 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3002940358' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T01:43:56.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:56 smithi032 ceph-mon[32461]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T01:43:56.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:56 smithi032 ceph-mon[32461]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:43:56.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:56 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3002940358' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T01:43:56.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:56 smithi032 ceph-mon[32461]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T01:43:56.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:56 smithi032 ceph-mon[32461]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-25T01:43:56.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:56 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3932719119' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:43:56.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:56 smithi028 ceph-mon[29390]: pgmap v338: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T01:43:56.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:56 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3002940358' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T01:43:56.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:56 smithi028 ceph-mon[29390]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T01:43:56.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:56 smithi028 ceph-mon[29390]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T01:43:56.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:56 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3002940358' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T01:43:56.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:56 smithi028 ceph-mon[29390]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T01:43:56.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:56 smithi028 ceph-mon[29390]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-25T01:43:56.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:56 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3932719119' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:43:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.172+0000 7fc38e7fc640 1 -- 172.21.15.28:0/1011259171 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v104) ==== 181+0+0 (secure 0 0 0) 0x7fc394009de0 con 0x7fc398100610 2024-04-25T01:43:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-04-25T01:43:57.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.176+0000 7fc39f742640 1 -- 172.21.15.28:0/1011259171 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc3840a2d80 msgr2=0x7fc3840a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.176+0000 7fc39f742640 1 --2- 172.21.15.28:0/1011259171 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc3840a2d80 0x7fc3840a5240 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc388000c00 tx=0x7fc388005a20 comp rx=0 tx=0).stop 2024-04-25T01:43:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.176+0000 7fc39f742640 1 -- 172.21.15.28:0/1011259171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc398100610 msgr2=0x7fc3981024e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.176+0000 7fc39f742640 1 --2- 172.21.15.28:0/1011259171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc398100610 0x7fc3981024e0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fc39400d950 tx=0x7fc39400de10 comp rx=0 tx=0).stop 2024-04-25T01:43:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.177+0000 7fc39f742640 1 -- 172.21.15.28:0/1011259171 shutdown_connections 2024-04-25T01:43:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.177+0000 7fc39f742640 1 --2- 172.21.15.28:0/1011259171 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7fc3840a2d80 0x7fc3840a5240 secure :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc388000c00 tx=0x7fc388005a20 comp rx=0 tx=0).stop 2024-04-25T01:43:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.177+0000 7fc39f742640 1 --2- 172.21.15.28:0/1011259171 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc3981066b0 0x7fc398102f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.177+0000 7fc39f742640 1 --2- 172.21.15.28:0/1011259171 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7fc398100fd0 0x7fc398102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.177+0000 7fc39f742640 1 --2- 172.21.15.28:0/1011259171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc398100610 0x7fc3981024e0 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.177+0000 7fc39f742640 1 -- 172.21.15.28:0/1011259171 >> 172.21.15.28:0/1011259171 conn(0x7fc398077740 msgr2=0x7fc398104680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.177+0000 7fc39f742640 1 -- 172.21.15.28:0/1011259171 shutdown_connections 2024-04-25T01:43:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.177+0000 7fc39f742640 1 -- 172.21.15.28:0/1011259171 wait complete. 2024-04-25T01:43:57.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T01:43:57.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.303+0000 7efefcfa5640 1 -- 172.21.15.28:0/3761527482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef810dfd0 msgr2=0x7efef81103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:57.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.303+0000 7efefcfa5640 1 --2- 172.21.15.28:0/3761527482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef810dfd0 0x7efef81103c0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7efef0009ca0 tx=0x7efef002a950 comp rx=0 tx=0).stop 2024-04-25T01:43:57.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.304+0000 7efefcfa5640 1 -- 172.21.15.28:0/3761527482 shutdown_connections 2024-04-25T01:43:57.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.304+0000 7efefcfa5640 1 --2- 172.21.15.28:0/3761527482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef810dfd0 0x7efef81103c0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:57.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.304+0000 7efefcfa5640 1 --2- 172.21.15.28:0/3761527482 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efef8074410 0x7efef810da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:57.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.304+0000 7efefcfa5640 1 --2- 172.21.15.28:0/3761527482 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efef8073af0 0x7efef8073ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:57.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.304+0000 7efefcfa5640 1 -- 172.21.15.28:0/3761527482 >> 172.21.15.28:0/3761527482 conn(0x7efef80fd180 msgr2=0x7efef80ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:57.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.304+0000 7efefcfa5640 1 -- 172.21.15.28:0/3761527482 shutdown_connections 2024-04-25T01:43:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.304+0000 7efefcfa5640 1 -- 172.21.15.28:0/3761527482 wait complete. 2024-04-25T01:43:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.304+0000 7efefcfa5640 1 Processor -- start 2024-04-25T01:43:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.304+0000 7efefcfa5640 1 -- start start 2024-04-25T01:43:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.305+0000 7efefcfa5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efef8073af0 0x7efef819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.305+0000 7efefcfa5640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efef8074410 0x7efef819a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.305+0000 7efefcfa5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef810dfd0 0x7efef819ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.305+0000 7efefcfa5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efef8113cf0 con 0x7efef810dfd0 2024-04-25T01:43:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.305+0000 7efefcfa5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efef8113b70 con 0x7efef8073af0 2024-04-25T01:43:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.305+0000 7efefcfa5640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7efef8113e70 con 0x7efef8074410 2024-04-25T01:43:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.305+0000 7efef6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef810dfd0 0x7efef819ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.305+0000 7efef6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef810dfd0 0x7efef819ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41628/0 (socket says 172.21.15.28:41628) 2024-04-25T01:43:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.305+0000 7efef6d9d640 1 -- 172.21.15.28:0/1916407745 learned_addr learned my addr 172.21.15.28:0/1916407745 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:57.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.305+0000 7efef5d9b640 1 --2- 172.21.15.28:0/1916407745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efef8074410 0x7efef819a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.305+0000 7efef659c640 1 --2- 172.21.15.28:0/1916407745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efef8073af0 0x7efef819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.305+0000 7efef6d9d640 1 -- 172.21.15.28:0/1916407745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efef8074410 msgr2=0x7efef819a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.305+0000 7efef6d9d640 1 --2- 172.21.15.28:0/1916407745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efef8074410 0x7efef819a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.305+0000 7efef6d9d640 1 -- 172.21.15.28:0/1916407745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efef8073af0 msgr2=0x7efef819a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.305+0000 7efef6d9d640 1 --2- 172.21.15.28:0/1916407745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efef8073af0 0x7efef819a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.305+0000 7efef6d9d640 1 -- 172.21.15.28:0/1916407745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef819f170 con 0x7efef810dfd0 2024-04-25T01:43:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.305+0000 7efef6d9d640 1 --2- 172.21.15.28:0/1916407745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef810dfd0 0x7efef819ea40 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7efef0004af0 tx=0x7efef002f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.305+0000 7efee77fe640 1 -- 172.21.15.28:0/1916407745 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef0034040 con 0x7efef810dfd0 2024-04-25T01:43:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.306+0000 7efee77fe640 1 -- 172.21.15.28:0/1916407745 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efef0030920 con 0x7efef810dfd0 2024-04-25T01:43:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.306+0000 7efefcfa5640 1 -- 172.21.15.28:0/1916407745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef819f400 con 0x7efef810dfd0 2024-04-25T01:43:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.306+0000 7efee77fe640 1 -- 172.21.15.28:0/1916407745 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef0004740 con 0x7efef810dfd0 2024-04-25T01:43:57.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.306+0000 7efefcfa5640 1 -- 172.21.15.28:0/1916407745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef81a6cf0 con 0x7efef810dfd0 2024-04-25T01:43:57.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.307+0000 7efefcfa5640 1 -- 172.21.15.28:0/1916407745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efef80689b0 con 0x7efef810dfd0 2024-04-25T01:43:57.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.308+0000 7efee77fe640 1 -- 172.21.15.28:0/1916407745 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7efef0037020 con 0x7efef810dfd0 2024-04-25T01:43:57.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.308+0000 7efee77fe640 1 --2- 172.21.15.28:0/1916407745 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7efee00a2d80 0x7efee00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:57.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.309+0000 7efee77fe640 1 -- 172.21.15.28:0/1916407745 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9284+0+0 (secure 0 0 0) 0x7efef00f0380 con 0x7efef810dfd0 2024-04-25T01:43:57.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.309+0000 7efef659c640 1 --2- 172.21.15.28:0/1916407745 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7efee00a2d80 0x7efee00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:57.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.309+0000 7efef659c640 1 --2- 172.21.15.28:0/1916407745 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7efee00a2d80 0x7efee00a5240 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7efee800abe0 tx=0x7efee800a2d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.313+0000 7efee77fe640 1 -- 172.21.15.28:0/1916407745 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efef00bcdb0 con 0x7efef810dfd0 2024-04-25T01:43:57.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T01:43:57.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T01:43:57.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: Cluster is now healthy 2024-04-25T01:43:57.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/3932719119' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:43:57.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.28:6838/900895474,v1:172.21.15.28:6839/900895474] up:boot 2024-04-25T01:43:57.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.32:6832/4102034891,v1:172.21.15.32:6834/4102034891] up:boot 2024-04-25T01:43:57.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.32:6833/1697509273,v1:172.21.15.32:6835/1697509273] up:boot 2024-04-25T01:43:57.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.28:6840/946855408,v1:172.21.15.28:6841/946855408] up:boot 2024-04-25T01:43:57.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.32:6837/2709135835,v1:172.21.15.32:6838/2709135835] up:boot 2024-04-25T01:43:57.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.32:6836/2492790063,v1:172.21.15.32:6839/2492790063] up:boot 2024-04-25T01:43:57.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.130:6832/1262837528,v1:172.21.15.130:6834/1262837528] up:boot 2024-04-25T01:43:57.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.130:6833/1500501097,v1:172.21.15.130:6835/1500501097] up:boot 2024-04-25T01:43:57.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.130:6836/4261593498,v1:172.21.15.130:6838/4261593498] up:boot 2024-04-25T01:43:57.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: mds.? [v2:172.21.15.130:6837/876409782,v1:172.21.15.130:6839/876409782] up:boot 2024-04-25T01:43:57.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: fsmap 12 up:standby 2024-04-25T01:43:57.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:43:57.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:43:57.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:43:57.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:43:57.473 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:43:57.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:43:57.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:43:57.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:43:57.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:43:57.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:43:57.474 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:57 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1011259171' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:43:57.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:57.497+0000 7efefcfa5640 1 -- 172.21.15.28:0/1916407745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7efef8109180 con 0x7efef810dfd0 2024-04-25T01:43:57.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T01:43:57.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T01:43:57.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: Cluster is now healthy 2024-04-25T01:43:57.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/3932719119' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:43:57.498 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.28:6838/900895474,v1:172.21.15.28:6839/900895474] up:boot 2024-04-25T01:43:57.498 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.32:6832/4102034891,v1:172.21.15.32:6834/4102034891] up:boot 2024-04-25T01:43:57.498 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.32:6833/1697509273,v1:172.21.15.32:6835/1697509273] up:boot 2024-04-25T01:43:57.498 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.28:6840/946855408,v1:172.21.15.28:6841/946855408] up:boot 2024-04-25T01:43:57.498 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.32:6837/2709135835,v1:172.21.15.32:6838/2709135835] up:boot 2024-04-25T01:43:57.498 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.32:6836/2492790063,v1:172.21.15.32:6839/2492790063] up:boot 2024-04-25T01:43:57.498 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.130:6832/1262837528,v1:172.21.15.130:6834/1262837528] up:boot 2024-04-25T01:43:57.498 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.130:6833/1500501097,v1:172.21.15.130:6835/1500501097] up:boot 2024-04-25T01:43:57.498 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.130:6836/4261593498,v1:172.21.15.130:6838/4261593498] up:boot 2024-04-25T01:43:57.498 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: mds.? [v2:172.21.15.130:6837/876409782,v1:172.21.15.130:6839/876409782] up:boot 2024-04-25T01:43:57.498 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: fsmap 12 up:standby 2024-04-25T01:43:57.499 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:43:57.499 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:43:57.499 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:43:57.499 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:43:57.499 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:43:57.499 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:43:57.499 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:43:57.499 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:43:57.499 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:43:57.499 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:43:57.499 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:57 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1011259171' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:43:57.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T01:43:57.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T01:43:57.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: Cluster is now healthy 2024-04-25T01:43:57.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/3932719119' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:43:57.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.28:6838/900895474,v1:172.21.15.28:6839/900895474] up:boot 2024-04-25T01:43:57.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.32:6832/4102034891,v1:172.21.15.32:6834/4102034891] up:boot 2024-04-25T01:43:57.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.32:6833/1697509273,v1:172.21.15.32:6835/1697509273] up:boot 2024-04-25T01:43:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.28:6840/946855408,v1:172.21.15.28:6841/946855408] up:boot 2024-04-25T01:43:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.32:6837/2709135835,v1:172.21.15.32:6838/2709135835] up:boot 2024-04-25T01:43:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.32:6836/2492790063,v1:172.21.15.32:6839/2492790063] up:boot 2024-04-25T01:43:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.130:6832/1262837528,v1:172.21.15.130:6834/1262837528] up:boot 2024-04-25T01:43:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.130:6833/1500501097,v1:172.21.15.130:6835/1500501097] up:boot 2024-04-25T01:43:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.130:6836/4261593498,v1:172.21.15.130:6838/4261593498] up:boot 2024-04-25T01:43:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: mds.? [v2:172.21.15.130:6837/876409782,v1:172.21.15.130:6839/876409782] up:boot 2024-04-25T01:43:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: fsmap 12 up:standby 2024-04-25T01:43:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:43:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:43:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:43:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:43:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:43:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:43:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:43:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:43:57.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:43:57.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: from='mgr.15561 172.21.15.28:0/3661424458' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:43:57.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:57 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1011259171' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:43:58.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.174+0000 7efee77fe640 1 -- 172.21.15.28:0/1916407745 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v105) ==== 169+0+0 (secure 0 0 0) 0x7efef00bc6b0 con 0x7efef810dfd0 2024-04-25T01:43:58.166 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-04-25T01:43:58.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.178+0000 7efefcfa5640 1 -- 172.21.15.28:0/1916407745 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7efee00a2d80 msgr2=0x7efee00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:58.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.178+0000 7efefcfa5640 1 --2- 172.21.15.28:0/1916407745 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7efee00a2d80 0x7efee00a5240 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7efee800abe0 tx=0x7efee800a2d0 comp rx=0 tx=0).stop 2024-04-25T01:43:58.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.178+0000 7efefcfa5640 1 -- 172.21.15.28:0/1916407745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef810dfd0 msgr2=0x7efef819ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:58.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.178+0000 7efefcfa5640 1 --2- 172.21.15.28:0/1916407745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef810dfd0 0x7efef819ea40 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7efef0004af0 tx=0x7efef002f5b0 comp rx=0 tx=0).stop 2024-04-25T01:43:58.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.178+0000 7efefcfa5640 1 -- 172.21.15.28:0/1916407745 shutdown_connections 2024-04-25T01:43:58.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.178+0000 7efefcfa5640 1 --2- 172.21.15.28:0/1916407745 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7efee00a2d80 0x7efee00a5240 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:58.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.178+0000 7efefcfa5640 1 --2- 172.21.15.28:0/1916407745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef810dfd0 0x7efef819ea40 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:58.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.178+0000 7efefcfa5640 1 --2- 172.21.15.28:0/1916407745 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7efef8074410 0x7efef819a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:58.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.178+0000 7efefcfa5640 1 --2- 172.21.15.28:0/1916407745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efef8073af0 0x7efef819a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:58.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.178+0000 7efefcfa5640 1 -- 172.21.15.28:0/1916407745 >> 172.21.15.28:0/1916407745 conn(0x7efef80fd180 msgr2=0x7efef80fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:58.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.178+0000 7efefcfa5640 1 -- 172.21.15.28:0/1916407745 shutdown_connections 2024-04-25T01:43:58.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.178+0000 7efefcfa5640 1 -- 172.21.15.28:0/1916407745 wait complete. 2024-04-25T01:43:58.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-25T01:43:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.315+0000 7f7e87142640 1 -- 172.21.15.28:0/2564758589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e801020d0 msgr2=0x7f7e8010e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.315+0000 7f7e87142640 1 --2- 172.21.15.28:0/2564758589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e801020d0 0x7f7e8010e5e0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009930 tx=0x7f7e6802a420 comp rx=0 tx=0).stop 2024-04-25T01:43:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.316+0000 7f7e87142640 1 -- 172.21.15.28:0/2564758589 shutdown_connections 2024-04-25T01:43:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.316+0000 7f7e87142640 1 --2- 172.21.15.28:0/2564758589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e801020d0 0x7f7e8010e5e0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.316+0000 7f7e87142640 1 --2- 172.21.15.28:0/2564758589 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e80101710 0x7f7e80101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.316+0000 7f7e87142640 1 --2- 172.21.15.28:0/2564758589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e80107720 0x7f7e80107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:58.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.316+0000 7f7e87142640 1 -- 172.21.15.28:0/2564758589 >> 172.21.15.28:0/2564758589 conn(0x7f7e80078b90 msgr2=0x7f7e80078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:58.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.316+0000 7f7e87142640 1 -- 172.21.15.28:0/2564758589 shutdown_connections 2024-04-25T01:43:58.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.316+0000 7f7e87142640 1 -- 172.21.15.28:0/2564758589 wait complete. 2024-04-25T01:43:58.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.316+0000 7f7e87142640 1 Processor -- start 2024-04-25T01:43:58.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.316+0000 7f7e87142640 1 -- start start 2024-04-25T01:43:58.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.316+0000 7f7e87142640 1 --2- >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e80101710 0x7f7e80073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:58.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.316+0000 7f7e87142640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e801020d0 0x7f7e80073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:58.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.316+0000 7f7e87142640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e80107720 0x7f7e800744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:58.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.316+0000 7f7e87142640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e80111b30 con 0x7f7e801020d0 2024-04-25T01:43:58.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.316+0000 7f7e87142640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7e801119b0 con 0x7f7e80107720 2024-04-25T01:43:58.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.316+0000 7f7e87142640 1 -- --> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] -- mon_getmap magic: 0 -- 0x7f7e80111cb0 con 0x7f7e80101710 2024-04-25T01:43:58.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.316+0000 7f7e7ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e801020d0 0x7f7e80073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:58.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.317+0000 7f7e7ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e801020d0 0x7f7e80073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41650/0 (socket says 172.21.15.28:41650) 2024-04-25T01:43:58.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.317+0000 7f7e7ffff640 1 -- 172.21.15.28:0/1357041781 learned_addr learned my addr 172.21.15.28:0/1357041781 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T01:43:58.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.317+0000 7f7e84ede640 1 --2- 172.21.15.28:0/1357041781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e80101710 0x7f7e80073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:58.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.317+0000 7f7e856df640 1 --2- 172.21.15.28:0/1357041781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e80107720 0x7f7e800744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:58.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.317+0000 7f7e7ffff640 1 -- 172.21.15.28:0/1357041781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e80101710 msgr2=0x7f7e80073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:58.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.317+0000 7f7e7ffff640 1 --2- 172.21.15.28:0/1357041781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e80101710 0x7f7e80073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:58.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.317+0000 7f7e7ffff640 1 -- 172.21.15.28:0/1357041781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e80107720 msgr2=0x7f7e800744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:58.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.317+0000 7f7e7ffff640 1 --2- 172.21.15.28:0/1357041781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e80107720 0x7f7e800744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:58.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.317+0000 7f7e7ffff640 1 -- 172.21.15.28:0/1357041781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e801a6510 con 0x7f7e801020d0 2024-04-25T01:43:58.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.317+0000 7f7e7ffff640 1 --2- 172.21.15.28:0/1357041781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e801020d0 0x7f7e80073f80 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f7e7000eb90 tx=0x7f7e7000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:58.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.317+0000 7f7e7dffb640 1 -- 172.21.15.28:0/1357041781 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e70004d20 con 0x7f7e801020d0 2024-04-25T01:43:58.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.317+0000 7f7e7dffb640 1 -- 172.21.15.28:0/1357041781 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e7000ee90 con 0x7f7e801020d0 2024-04-25T01:43:58.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.317+0000 7f7e87142640 1 -- 172.21.15.28:0/1357041781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e801a6800 con 0x7f7e801020d0 2024-04-25T01:43:58.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.317+0000 7f7e7dffb640 1 -- 172.21.15.28:0/1357041781 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e70010830 con 0x7f7e801020d0 2024-04-25T01:43:58.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.317+0000 7f7e87142640 1 -- 172.21.15.28:0/1357041781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e801a6d50 con 0x7f7e801020d0 2024-04-25T01:43:58.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.318+0000 7f7e87142640 1 -- 172.21.15.28:0/1357041781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e800689b0 con 0x7f7e801020d0 2024-04-25T01:43:58.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.319+0000 7f7e7dffb640 1 -- 172.21.15.28:0/1357041781 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 137964+0+0 (secure 0 0 0) 0x7f7e70005020 con 0x7f7e801020d0 2024-04-25T01:43:58.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.320+0000 7f7e7dffb640 1 --2- 172.21.15.28:0/1357041781 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7e6c0a2db0 0x7f7e6c0a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:58.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.320+0000 7f7e84ede640 1 --2- 172.21.15.28:0/1357041781 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7e6c0a2db0 0x7f7e6c0a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:58.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.320+0000 7f7e7dffb640 1 -- 172.21.15.28:0/1357041781 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8754+0+0 (secure 0 0 0) 0x7f7e700cfb90 con 0x7f7e801020d0 2024-04-25T01:43:58.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.321+0000 7f7e84ede640 1 --2- 172.21.15.28:0/1357041781 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7e6c0a2db0 0x7f7e6c0a5270 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7e780058c0 tx=0x7f7e7800a2d0 comp rx=0 tx=0).ready entity=mgr.15561 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:58.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.325+0000 7f7e7dffb640 1 -- 172.21.15.28:0/1357041781 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7e7009c6c0 con 0x7f7e801020d0 2024-04-25T01:43:58.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:58 smithi130 ceph-mon[32702]: pgmap v340: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 614 B/s wr, 9 op/s 2024-04-25T01:43:58.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:58 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1011259171' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T01:43:58.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:58 smithi130 ceph-mon[32702]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T01:43:58.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:58 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1916407745' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:43:58.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:58 smithi032 ceph-mon[32461]: pgmap v340: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 614 B/s wr, 9 op/s 2024-04-25T01:43:58.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:58 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1011259171' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T01:43:58.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:58 smithi032 ceph-mon[32461]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T01:43:58.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:58 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1916407745' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:43:58.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:58.513+0000 7f7e87142640 1 -- 172.21.15.28:0/1357041781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7e80109ca0 con 0x7f7e801020d0 2024-04-25T01:43:58.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:58 smithi028 ceph-mon[29390]: pgmap v340: 129 pgs: 129 active+clean; 11 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 614 B/s wr, 9 op/s 2024-04-25T01:43:58.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:58 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1011259171' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T01:43:58.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:58 smithi028 ceph-mon[29390]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T01:43:58.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:58 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1916407745' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:43:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:59.185+0000 7f7e7dffb640 1 -- 172.21.15.28:0/1357041781 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v106) ==== 178+0+0 (secure 0 0 0) 0x7f7e7009bfc0 con 0x7f7e801020d0 2024-04-25T01:43:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data_ec' removed 2024-04-25T01:43:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:59.189+0000 7f7e87142640 1 -- 172.21.15.28:0/1357041781 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7e6c0a2db0 msgr2=0x7f7e6c0a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:59.189+0000 7f7e87142640 1 --2- 172.21.15.28:0/1357041781 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7e6c0a2db0 0x7f7e6c0a5270 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7e780058c0 tx=0x7f7e7800a2d0 comp rx=0 tx=0).stop 2024-04-25T01:43:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:59.189+0000 7f7e87142640 1 -- 172.21.15.28:0/1357041781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e801020d0 msgr2=0x7f7e80073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:59.189+0000 7f7e87142640 1 --2- 172.21.15.28:0/1357041781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e801020d0 0x7f7e80073f80 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f7e7000eb90 tx=0x7f7e7000c6a0 comp rx=0 tx=0).stop 2024-04-25T01:43:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:59.189+0000 7f7e87142640 1 -- 172.21.15.28:0/1357041781 shutdown_connections 2024-04-25T01:43:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:59.189+0000 7f7e87142640 1 --2- 172.21.15.28:0/1357041781 >> [v2:172.21.15.28:6800/712762371,v1:172.21.15.28:6801/712762371] conn(0x7f7e6c0a2db0 0x7f7e6c0a5270 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:59.189+0000 7f7e87142640 1 --2- 172.21.15.28:0/1357041781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7e80107720 0x7f7e800744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:59.189+0000 7f7e87142640 1 --2- 172.21.15.28:0/1357041781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e801020d0 0x7f7e80073f80 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:59.189+0000 7f7e87142640 1 --2- 172.21.15.28:0/1357041781 >> [v2:172.21.15.130:3300/0,v1:172.21.15.130:6789/0] conn(0x7f7e80101710 0x7f7e80073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:59.189+0000 7f7e87142640 1 -- 172.21.15.28:0/1357041781 >> 172.21.15.28:0/1357041781 conn(0x7f7e80078b90 msgr2=0x7f7e800ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:59.190+0000 7f7e87142640 1 -- 172.21.15.28:0/1357041781 shutdown_connections 2024-04-25T01:43:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T01:43:59.190+0000 7f7e87142640 1 -- 172.21.15.28:0/1357041781 wait complete. 2024-04-25T01:43:59.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:43:59.227 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:43:59.258 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:43:59.286 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-25T01:43:59.287 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T01:43:59.303 DEBUG:teuthology.orchestra.run.smithi032:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T01:43:59.320 DEBUG:teuthology.orchestra.run.smithi130:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T01:43:59.342 INFO:tasks.cephadm:Stopping all daemons... 2024-04-25T01:43:59.342 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-25T01:43:59.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mon.a 2024-04-25T01:43:59.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:59 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1916407745' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T01:43:59.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:59 smithi130 ceph-mon[32702]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T01:43:59.472 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:43:59 smithi130 ceph-mon[32702]: from='client.? 172.21.15.28:0/1357041781' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:43:59.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:59 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1916407745' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T01:43:59.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:59 smithi032 ceph-mon[32461]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T01:43:59.497 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:43:59 smithi032 ceph-mon[32461]: from='client.? 172.21.15.28:0/1357041781' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:43:59.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:59 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1916407745' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T01:43:59.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:59 smithi028 ceph-mon[29390]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T01:43:59.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:59 smithi028 ceph-mon[29390]: from='client.? 172.21.15.28:0/1357041781' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T01:43:59.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:59 smithi028 systemd[1]: Stopping Ceph mon.a for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:43:59.994 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:59 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-a[29386]: 2024-04-25T01:43:59.660+0000 7f291b37e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:43:59.994 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:59 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-a[29386]: 2024-04-25T01:43:59.660+0000 7f291b37e700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-25T01:43:59.994 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:43:59 smithi028 podman[102020]: 2024-04-25 01:43:59.722860452 +0000 UTC m=+0.176420445 container died 8b32243128b3bd12db0cb949a7f1e4f95331c5318b93c36d3ba29cd333fb85b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-25T01:44:00.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:44:00 smithi028 podman[102020]: 2024-04-25 01:44:00.112256123 +0000 UTC m=+0.565816115 container cleanup 8b32243128b3bd12db0cb949a7f1e4f95331c5318b93c36d3ba29cd333fb85b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-a, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T01:44:00.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 25 01:44:00 smithi028 bash[102020]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-a 2024-04-25T01:44:00.325 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:44:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:44:00.144+0000 7fae408a6700 -1 client.15621: SimpleRADOSStriper: lock: snap_db_v0.db: lock failed: (2) No such file or directory 2024-04-25T01:44:00.325 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:44:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:44:00.145+0000 7fae408a6700 -1 client.15621: SimpleRADOSStriper: lock: snap_db_v0.db: lock failed: (2) No such file or directory 2024-04-25T01:44:00.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mon.a.service' 2024-04-25T01:44:00.566 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:44:00.566 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-25T01:44:00.567 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-25T01:44:00.567 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mon.b 2024-04-25T01:44:00.900 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:44:00 smithi032 systemd[1]: Stopping Ceph mon.b for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:44:01.247 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:44:00 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-b[32457]: 2024-04-25T01:44:00.908+0000 7f3c92fe2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:44:01.247 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:44:00 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-b[32457]: 2024-04-25T01:44:00.908+0000 7f3c92fe2700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-25T01:44:01.248 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:44:00 smithi032 podman[68560]: 2024-04-25 01:44:00.97056271 +0000 UTC m=+0.187770724 container died 9ebdfa9017ebf6ce6cd4a7130fa2d450123d4fb625ee7f7885f2ed05d69a5fdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-25T01:44:01.543 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:44:01 smithi032 podman[68560]: 2024-04-25 01:44:01.359670318 +0000 UTC m=+0.576878329 container cleanup 9ebdfa9017ebf6ce6cd4a7130fa2d450123d4fb625ee7f7885f2ed05d69a5fdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:44:01.543 INFO:journalctl@ceph.mon.b.smithi032.stdout:Apr 25 01:44:01 smithi032 bash[68560]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-b 2024-04-25T01:44:01.702 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mon.b.service' 2024-04-25T01:44:01.744 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:44:01.744 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-25T01:44:01.744 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-25T01:44:01.744 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mon.c 2024-04-25T01:44:02.059 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:44:01 smithi130 systemd[1]: Stopping Ceph mon.c for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:44:02.360 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:44:02 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-c[32690]: 2024-04-25T01:44:02.068+0000 7f4879931700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:44:02.360 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:44:02 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-c[32690]: 2024-04-25T01:44:02.068+0000 7f4879931700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-25T01:44:02.360 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:44:02 smithi130 podman[68438]: 2024-04-25 01:44:02.089145542 +0000 UTC m=+0.138016443 container died 3cdb8931f7c82603268fc913bd01e49c3b7f81de11887c1ba5a213ef879ac0ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-c, RELEASE=squid-d2c45d9, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-25T01:44:02.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:44:02 smithi130 podman[68438]: 2024-04-25 01:44:02.477805792 +0000 UTC m=+0.526676701 container cleanup 3cdb8931f7c82603268fc913bd01e49c3b7f81de11887c1ba5a213ef879ac0ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T01:44:02.716 INFO:journalctl@ceph.mon.c.smithi130.stdout:Apr 25 01:44:02 smithi130 bash[68438]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mon-c 2024-04-25T01:44:02.859 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mon.c.service' 2024-04-25T01:44:02.908 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:44:02.908 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-25T01:44:02.909 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-25T01:44:02.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mgr.x 2024-04-25T01:44:03.216 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:44:02 smithi028 systemd[1]: Stopping Ceph mgr.x for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:44:03.525 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:44:03 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x[29616]: 2024-04-25T01:44:03.224+0000 7faeda9bf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:44:03.525 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:44:03 smithi028 podman[102142]: 2024-04-25 01:44:03.256589263 +0000 UTC m=+0.142809611 container died a137f351b2ca9b22bd1d139d127a8e8d14ff825ef5912f4adb93c7c9a7f55162 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:44:03.825 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:44:03 smithi028 podman[102142]: 2024-04-25 01:44:03.63457438 +0000 UTC m=+0.520794738 container cleanup a137f351b2ca9b22bd1d139d127a8e8d14ff825ef5912f4adb93c7c9a7f55162 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-25T01:44:03.825 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:44:03 smithi028 bash[102142]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x 2024-04-25T01:44:04.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-25T01:44:04.080 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:44:03 smithi028 podman[102156]: 2024-04-25 01:44:03.895058842 +0000 UTC m=+0.632718421 container remove a137f351b2ca9b22bd1d139d127a8e8d14ff825ef5912f4adb93c7c9a7f55162 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-x, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-25T01:44:04.080 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:44:04 smithi028 systemd[1]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mgr.x.service: Deactivated successfully. 2024-04-25T01:44:04.081 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:44:04 smithi028 systemd[1]: Stopped Ceph mgr.x for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:44:04.081 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 25 01:44:04 smithi028 systemd[1]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mgr.x.service: Consumed 1min 28.697s CPU time. 2024-04-25T01:44:04.094 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:44:04.094 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-25T01:44:04.094 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-25T01:44:04.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mgr.y 2024-04-25T01:44:04.399 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:44:04 smithi032 systemd[1]: Stopping Ceph mgr.y for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:44:04.685 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:44:04 smithi032 podman[68684]: 2024-04-25 01:44:04.434960092 +0000 UTC m=+0.140206210 container died 4db81abec7f938c2b2d232ef3132c2734a39401aa06db2e024424c5e7bd7d707 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:44:04.997 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:44:04 smithi032 podman[68684]: 2024-04-25 01:44:04.806744606 +0000 UTC m=+0.511990730 container cleanup 4db81abec7f938c2b2d232ef3132c2734a39401aa06db2e024424c5e7bd7d707 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-25T01:44:04.998 INFO:journalctl@ceph.mgr.y.smithi032.stdout:Apr 25 01:44:04 smithi032 bash[68684]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-y 2024-04-25T01:44:05.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-25T01:44:05.279 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:44:05.279 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-25T01:44:05.279 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-25T01:44:05.279 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mgr.z 2024-04-25T01:44:05.627 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:44:05 smithi130 systemd[1]: Stopping Ceph mgr.z for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:44:05.916 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:44:05 smithi130 podman[68571]: 2024-04-25 01:44:05.636495775 +0000 UTC m=+0.152159793 container died 0e6b2e060ce6cbc264438c176c1ecbeac6b206e41a7cb2e568213e6b5e9db962 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2) 2024-04-25T01:44:06.222 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:44:06 smithi130 podman[68571]: 2024-04-25 01:44:06.025026987 +0000 UTC m=+0.540691056 container cleanup 0e6b2e060ce6cbc264438c176c1ecbeac6b206e41a7cb2e568213e6b5e9db962 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-25T01:44:06.222 INFO:journalctl@ceph.mgr.z.smithi130.stdout:Apr 25 01:44:06 smithi130 bash[68571]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mgr-z 2024-04-25T01:44:06.445 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-25T01:44:06.478 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:44:06.479 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-25T01:44:06.479 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-25T01:44:06.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.0 2024-04-25T01:44:06.805 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:44:06 smithi028 systemd[1]: Stopping Ceph osd.0 for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:44:07.075 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:44:06 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-0[41941]: 2024-04-25T01:44:06.814+0000 7fabd1b63700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:44:07.076 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:44:06 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-0[41941]: 2024-04-25T01:44:06.814+0000 7fabd1b63700 -1 osd.0 106 *** Got signal Terminated *** 2024-04-25T01:44:07.076 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:44:06 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-0[41941]: 2024-04-25T01:44:06.814+0000 7fabd1b63700 -1 osd.0 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:44:12.490 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:44:12 smithi028 podman[102264]: 2024-04-25 01:44:12.237655817 +0000 UTC m=+5.551584720 container died fff12e04c30aa496af30d0f070d07e92e70f5279b70d3935b8a2322c840f8750 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.license=GPLv2) 2024-04-25T01:44:12.825 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:44:12 smithi028 podman[102264]: 2024-04-25 01:44:12.623580089 +0000 UTC m=+5.937509000 container cleanup fff12e04c30aa496af30d0f070d07e92e70f5279b70d3935b8a2322c840f8750 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-0, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:44:12.825 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:44:12 smithi028 bash[102264]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-0 2024-04-25T01:44:13.253 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:44:12 smithi028 podman[102281]: 2024-04-25 01:44:12.891812257 +0000 UTC m=+0.647483418 container remove fff12e04c30aa496af30d0f070d07e92e70f5279b70d3935b8a2322c840f8750 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-0, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424) 2024-04-25T01:44:13.553 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:44:13 smithi028 podman[102369]: 2024-04-25 01:44:13.165748371 +0000 UTC m=+0.018605989 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:44:13.553 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:44:13 smithi028 podman[102369]: 2024-04-25 01:44:13.287136654 +0000 UTC m=+0.139994298 container create b248c90ca1693055c3c799d53e82dbfd81babcd0bab98d126e06848faabc1d6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:44:13.825 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:44:13 smithi028 podman[102369]: 2024-04-25 01:44:13.562587473 +0000 UTC m=+0.415445084 container init b248c90ca1693055c3c799d53e82dbfd81babcd0bab98d126e06848faabc1d6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T01:44:13.825 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:44:13 smithi028 podman[102369]: 2024-04-25 01:44:13.565948753 +0000 UTC m=+0.418806355 container start b248c90ca1693055c3c799d53e82dbfd81babcd0bab98d126e06848faabc1d6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T01:44:13.826 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:44:13 smithi028 podman[102369]: 2024-04-25 01:44:13.671219994 +0000 UTC m=+0.524077608 container attach b248c90ca1693055c3c799d53e82dbfd81babcd0bab98d126e06848faabc1d6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-25T01:44:14.181 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 25 01:44:13 smithi028 podman[102369]: 2024-04-25 01:44:13.886263964 +0000 UTC m=+0.739121578 container died b248c90ca1693055c3c799d53e82dbfd81babcd0bab98d126e06848faabc1d6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:44:14.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.0.service' 2024-04-25T01:44:14.519 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:44:14.519 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-25T01:44:14.520 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-25T01:44:14.520 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.1 2024-04-25T01:44:14.861 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:44:14 smithi032 systemd[1]: Stopping Ceph osd.1 for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:44:15.139 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:44:14 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-1[37059]: 2024-04-25T01:44:14.869+0000 7f552fce5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:44:15.139 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:44:14 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-1[37059]: 2024-04-25T01:44:14.869+0000 7f552fce5700 -1 osd.1 106 *** Got signal Terminated *** 2024-04-25T01:44:15.139 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:44:14 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-1[37059]: 2024-04-25T01:44:14.869+0000 7f552fce5700 -1 osd.1 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:44:20.653 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:44:20 smithi032 podman[68818]: 2024-04-25 01:44:20.361756674 +0000 UTC m=+5.627678891 container died b0bf2dcffea2e8fe325f4989663b818998624014cdd625239c2180f2bc123149 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9) 2024-04-25T01:44:20.979 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:44:20 smithi032 podman[68818]: 2024-04-25 01:44:20.769990109 +0000 UTC m=+6.035912329 container cleanup b0bf2dcffea2e8fe325f4989663b818998624014cdd625239c2180f2bc123149 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-1, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-25T01:44:20.979 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:44:20 smithi032 bash[68818]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-1 2024-04-25T01:44:21.247 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:44:20 smithi032 podman[68836]: 2024-04-25 01:44:20.988562915 +0000 UTC m=+0.628795352 container remove b0bf2dcffea2e8fe325f4989663b818998624014cdd625239c2180f2bc123149 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9) 2024-04-25T01:44:21.684 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:44:21 smithi032 podman[68925]: 2024-04-25 01:44:21.308849712 +0000 UTC m=+0.018848917 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:44:21.685 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:44:21 smithi032 podman[68925]: 2024-04-25 01:44:21.431973866 +0000 UTC m=+0.141973054 container create 0924d64ff1a9f03f4e4a79945acd3be72007c39f66b8d402bac6f82a0f96a29b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS) 2024-04-25T01:44:21.994 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:44:21 smithi032 podman[68925]: 2024-04-25 01:44:21.693789689 +0000 UTC m=+0.403788888 container init 0924d64ff1a9f03f4e4a79945acd3be72007c39f66b8d402bac6f82a0f96a29b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=squid-d2c45d9, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:44:21.994 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:44:21 smithi032 podman[68925]: 2024-04-25 01:44:21.697336663 +0000 UTC m=+0.407335854 container start 0924d64ff1a9f03f4e4a79945acd3be72007c39f66b8d402bac6f82a0f96a29b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-25T01:44:21.994 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:44:21 smithi032 podman[68925]: 2024-04-25 01:44:21.793962276 +0000 UTC m=+0.503961488 container attach 0924d64ff1a9f03f4e4a79945acd3be72007c39f66b8d402bac6f82a0f96a29b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-25T01:44:22.248 INFO:journalctl@ceph.osd.1.smithi032.stdout:Apr 25 01:44:22 smithi032 podman[68925]: 2024-04-25 01:44:22.00378963 +0000 UTC m=+0.713788839 container died 0924d64ff1a9f03f4e4a79945acd3be72007c39f66b8d402bac6f82a0f96a29b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T01:44:22.639 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.1.service' 2024-04-25T01:44:22.678 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:44:22.678 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-25T01:44:22.678 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-25T01:44:22.679 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.2 2024-04-25T01:44:22.972 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:44:22 smithi130 systemd[1]: Stopping Ceph osd.2 for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:44:23.472 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:44:22 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-2[37842]: 2024-04-25T01:44:22.986+0000 7f4b4d539700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:44:23.473 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:44:22 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-2[37842]: 2024-04-25T01:44:22.986+0000 7f4b4d539700 -1 osd.2 106 *** Got signal Terminated *** 2024-04-25T01:44:23.473 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:44:22 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-2[37842]: 2024-04-25T01:44:22.986+0000 7f4b4d539700 -1 osd.2 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:44:28.612 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:44:28 smithi130 podman[68695]: 2024-04-25 01:44:28.352571854 +0000 UTC m=+5.475217047 container died 6bd6c9047a9cbaf914d1593f5b4212b2f7674edef5c0df6481cae667f8ea44f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-25T01:44:28.973 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:44:28 smithi130 podman[68695]: 2024-04-25 01:44:28.729629107 +0000 UTC m=+5.852274318 container cleanup 6bd6c9047a9cbaf914d1593f5b4212b2f7674edef5c0df6481cae667f8ea44f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-25T01:44:28.973 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:44:28 smithi130 bash[68695]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-2 2024-04-25T01:44:29.350 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:44:29 smithi130 podman[68712]: 2024-04-25 01:44:29.031107283 +0000 UTC m=+0.678436290 container remove 6bd6c9047a9cbaf914d1593f5b4212b2f7674edef5c0df6481cae667f8ea44f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:44:29.642 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:44:29 smithi130 podman[68801]: 2024-04-25 01:44:29.262416486 +0000 UTC m=+0.017937969 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:44:29.642 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:44:29 smithi130 podman[68801]: 2024-04-25 01:44:29.382190459 +0000 UTC m=+0.137711936 container create 15600a91f6aee5537eec3b44e3df22fced8d6086a85179dce695c899fd1a341f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424) 2024-04-25T01:44:29.940 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:44:29 smithi130 podman[68801]: 2024-04-25 01:44:29.651616644 +0000 UTC m=+0.407138120 container init 15600a91f6aee5537eec3b44e3df22fced8d6086a85179dce695c899fd1a341f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-25T01:44:29.940 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:44:29 smithi130 podman[68801]: 2024-04-25 01:44:29.654857807 +0000 UTC m=+0.410379284 container start 15600a91f6aee5537eec3b44e3df22fced8d6086a85179dce695c899fd1a341f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:44:29.940 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:44:29 smithi130 podman[68801]: 2024-04-25 01:44:29.760144799 +0000 UTC m=+0.515666282 container attach 15600a91f6aee5537eec3b44e3df22fced8d6086a85179dce695c899fd1a341f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-d2c45d9) 2024-04-25T01:44:30.222 INFO:journalctl@ceph.osd.2.smithi130.stdout:Apr 25 01:44:29 smithi130 podman[68801]: 2024-04-25 01:44:29.94992923 +0000 UTC m=+0.705450714 container died 15600a91f6aee5537eec3b44e3df22fced8d6086a85179dce695c899fd1a341f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T01:44:30.632 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.2.service' 2024-04-25T01:44:30.674 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:44:30.674 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-25T01:44:30.674 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-25T01:44:30.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.3 2024-04-25T01:44:31.027 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:44:30 smithi028 systemd[1]: Stopping Ceph osd.3 for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:44:31.306 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:44:31 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-3[48485]: 2024-04-25T01:44:31.035+0000 7fbe5ac38700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:44:31.307 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:44:31 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-3[48485]: 2024-04-25T01:44:31.035+0000 7fbe5ac38700 -1 osd.3 106 *** Got signal Terminated *** 2024-04-25T01:44:31.307 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:44:31 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-3[48485]: 2024-04-25T01:44:31.035+0000 7fbe5ac38700 -1 osd.3 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:44:33.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:33 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:33.179+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:34.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:34 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:34.139+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:34.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:34 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:34.169+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:35.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:35 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:35.092+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:35.575 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:35 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:35.166+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:35.748 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:35 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:35.321+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:36.100 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:35 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:35.833+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.259648+0000 front 2024-04-25T01:44:10.259821+0000 (oldest deadline 2024-04-25T01:44:35.559727+0000) 2024-04-25T01:44:36.429 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:36 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:36.186+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:36.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:36 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:36.108+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:36.713 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:44:36 smithi028 podman[102477]: 2024-04-25 01:44:36.438337979 +0000 UTC m=+5.554041969 container died 7e189f64276b762e0690bd3d257f27365b0d4c355bb436b6537c0f568354332e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T01:44:36.748 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:36 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:36.345+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:37.037 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:44:36 smithi028 podman[102477]: 2024-04-25 01:44:36.811648976 +0000 UTC m=+5.927353039 container cleanup 7e189f64276b762e0690bd3d257f27365b0d4c355bb436b6537c0f568354332e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-25T01:44:37.037 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:44:36 smithi028 bash[102477]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-3 2024-04-25T01:44:37.086 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:36 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:36.792+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.259648+0000 front 2024-04-25T01:44:10.259821+0000 (oldest deadline 2024-04-25T01:44:35.559727+0000) 2024-04-25T01:44:37.326 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:44:37 smithi028 podman[102506]: 2024-04-25 01:44:37.04607598 +0000 UTC m=+0.602784514 container remove 7e189f64276b762e0690bd3d257f27365b0d4c355bb436b6537c0f568354332e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-25T01:44:37.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:37 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:37.174+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:37.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:37 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:37.095+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:37.700 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:44:37 smithi028 podman[102597]: 2024-04-25 01:44:37.273622796 +0000 UTC m=+0.018463472 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:44:37.700 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:44:37 smithi028 podman[102597]: 2024-04-25 01:44:37.422126185 +0000 UTC m=+0.166966850 container create d5d0920651d8d4d11137d6c12c44dcd39e267a4f73a65dad6d54d69b7fb7d6ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS) 2024-04-25T01:44:37.700 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:44:37 smithi028 podman[102597]: 2024-04-25 01:44:37.708446538 +0000 UTC m=+0.453287205 container init d5d0920651d8d4d11137d6c12c44dcd39e267a4f73a65dad6d54d69b7fb7d6ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-25T01:44:37.748 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:37 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:37.357+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:38.010 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:44:37 smithi028 podman[102597]: 2024-04-25 01:44:37.71222677 +0000 UTC m=+0.457067444 container start d5d0920651d8d4d11137d6c12c44dcd39e267a4f73a65dad6d54d69b7fb7d6ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:44:38.010 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:44:37 smithi028 podman[102597]: 2024-04-25 01:44:37.783383425 +0000 UTC m=+0.528224092 container attach d5d0920651d8d4d11137d6c12c44dcd39e267a4f73a65dad6d54d69b7fb7d6ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-25T01:44:38.124 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:37 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:37.837+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.259648+0000 front 2024-04-25T01:44:10.259821+0000 (oldest deadline 2024-04-25T01:44:35.559727+0000) 2024-04-25T01:44:38.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:38 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:38.144+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:38.326 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:44:38 smithi028 podman[102597]: 2024-04-25 01:44:38.018487175 +0000 UTC m=+0.763327841 container died d5d0920651d8d4d11137d6c12c44dcd39e267a4f73a65dad6d54d69b7fb7d6ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2) 2024-04-25T01:44:38.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:38 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:38.132+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:38.620 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:44:38 smithi028 podman[102623]: 2024-04-25 01:44:38.604081768 +0000 UTC m=+0.579789225 container remove d5d0920651d8d4d11137d6c12c44dcd39e267a4f73a65dad6d54d69b7fb7d6ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424) 2024-04-25T01:44:38.621 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:44:38 smithi028 systemd[1]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-25T01:44:38.621 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:44:38 smithi028 systemd[1]: Stopped Ceph osd.3 for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:44:38.621 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 25 01:44:38 smithi028 systemd[1]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.3.service: Consumed 41.955s CPU time. 2024-04-25T01:44:38.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.3.service' 2024-04-25T01:44:38.669 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:44:38.670 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-25T01:44:38.670 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-25T01:44:38.670 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.4 2024-04-25T01:44:38.697 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:38 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:38.348+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:38.998 INFO:journalctl@ceph.osd.4.smithi032.stdout:Apr 25 01:44:38 smithi032 systemd[1]: Stopping Ceph osd.4 for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:44:39.100 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:38 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:38.858+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.259648+0000 front 2024-04-25T01:44:10.259821+0000 (oldest deadline 2024-04-25T01:44:35.559727+0000) 2024-04-25T01:44:39.307 INFO:journalctl@ceph.osd.4.smithi032.stdout:Apr 25 01:44:39 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-4[44020]: 2024-04-25T01:44:39.010+0000 7f80604c2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:44:39.307 INFO:journalctl@ceph.osd.4.smithi032.stdout:Apr 25 01:44:39 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-4[44020]: 2024-04-25T01:44:39.010+0000 7f80604c2700 -1 osd.4 106 *** Got signal Terminated *** 2024-04-25T01:44:39.307 INFO:journalctl@ceph.osd.4.smithi032.stdout:Apr 25 01:44:39 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-4[44020]: 2024-04-25T01:44:39.010+0000 7f80604c2700 -1 osd.4 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:44:39.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:39 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:39.108+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:39.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:39 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:39.165+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:39.748 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:39 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:39.315+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:40.223 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:39 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:39.888+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.259648+0000 front 2024-04-25T01:44:10.259821+0000 (oldest deadline 2024-04-25T01:44:35.559727+0000) 2024-04-25T01:44:40.223 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:40 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:40.136+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:40.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:40 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:40.142+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:40.748 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:40 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:40.305+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:41.223 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:40 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:40.906+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.259648+0000 front 2024-04-25T01:44:10.259821+0000 (oldest deadline 2024-04-25T01:44:35.559727+0000) 2024-04-25T01:44:41.224 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:41 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:41.120+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:41.498 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:41 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:41.256+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:41.498 INFO:journalctl@ceph.osd.4.smithi032.stdout:Apr 25 01:44:41 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-4[44020]: 2024-04-25T01:44:41.114+0000 7f805cab6700 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.931876+0000 front 2024-04-25T01:44:18.931963+0000 (oldest deadline 2024-04-25T01:44:40.532795+0000) 2024-04-25T01:44:41.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:41 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:41.138+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:42.143 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:41 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:41.868+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.259648+0000 front 2024-04-25T01:44:10.259821+0000 (oldest deadline 2024-04-25T01:44:35.559727+0000) 2024-04-25T01:44:42.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:42 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:42.109+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:42.326 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:41 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:41.995+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:15.239423+0000 front 2024-04-25T01:44:15.239207+0000 (oldest deadline 2024-04-25T01:44:41.139214+0000) 2024-04-25T01:44:42.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:42 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:42.151+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:42.498 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:42 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:42.282+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:42.498 INFO:journalctl@ceph.osd.4.smithi032.stdout:Apr 25 01:44:42 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-4[44020]: 2024-04-25T01:44:42.141+0000 7f805cab6700 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.931876+0000 front 2024-04-25T01:44:18.931963+0000 (oldest deadline 2024-04-25T01:44:40.532795+0000) 2024-04-25T01:44:42.998 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:42 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:42.561+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:44:43.171 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:42 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:42.915+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.259648+0000 front 2024-04-25T01:44:10.259821+0000 (oldest deadline 2024-04-25T01:44:35.559727+0000) 2024-04-25T01:44:43.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:43 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:43.062+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:43.326 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:43 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:43.008+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:15.239423+0000 front 2024-04-25T01:44:15.239207+0000 (oldest deadline 2024-04-25T01:44:41.139214+0000) 2024-04-25T01:44:43.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:43 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:43.179+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:43.498 INFO:journalctl@ceph.osd.4.smithi032.stdout:Apr 25 01:44:43 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-4[44020]: 2024-04-25T01:44:43.166+0000 7f805cab6700 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.931876+0000 front 2024-04-25T01:44:18.931963+0000 (oldest deadline 2024-04-25T01:44:40.532795+0000) 2024-04-25T01:44:43.498 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:43 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:43.306+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:43.998 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:43 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:43.596+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:44:44.223 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:43 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:43.928+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.259648+0000 front 2024-04-25T01:44:10.259821+0000 (oldest deadline 2024-04-25T01:44:35.559727+0000) 2024-04-25T01:44:44.223 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:44 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:44.133+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:44.326 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:44 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:44.006+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:15.239423+0000 front 2024-04-25T01:44:15.239207+0000 (oldest deadline 2024-04-25T01:44:41.139214+0000) 2024-04-25T01:44:44.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:44 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:44.020+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:44.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:44 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:44.020+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:44:44.614 INFO:journalctl@ceph.osd.4.smithi032.stdout:Apr 25 01:44:44 smithi032 podman[69054]: 2024-04-25 01:44:44.542032862 +0000 UTC m=+5.651060908 container died 1ac9e92d035a66d75c40adeab0ab3aa88c52ab64a076077ece090a38029eb5f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-4, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-25T01:44:44.614 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:44 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:44.317+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:44.614 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:44 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:44.317+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:44:44.919 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:44 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:44.622+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:44:45.223 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:44 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:44.904+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.259648+0000 front 2024-04-25T01:44:10.259821+0000 (oldest deadline 2024-04-25T01:44:35.559727+0000) 2024-04-25T01:44:45.223 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:44 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:44.904+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.462260+0000 front 2024-04-25T01:44:19.462207+0000 (oldest deadline 2024-04-25T01:44:44.762000+0000) 2024-04-25T01:44:45.223 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:45 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:45.095+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:45.248 INFO:journalctl@ceph.osd.4.smithi032.stdout:Apr 25 01:44:44 smithi032 podman[69054]: 2024-04-25 01:44:44.927953767 +0000 UTC m=+6.036981810 container cleanup 1ac9e92d035a66d75c40adeab0ab3aa88c52ab64a076077ece090a38029eb5f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9) 2024-04-25T01:44:45.248 INFO:journalctl@ceph.osd.4.smithi032.stdout:Apr 25 01:44:44 smithi032 bash[69054]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-4 2024-04-25T01:44:45.248 INFO:journalctl@ceph.osd.4.smithi032.stdout:Apr 25 01:44:45 smithi032 podman[69072]: 2024-04-25 01:44:45.113145642 +0000 UTC m=+0.571005640 container remove 1ac9e92d035a66d75c40adeab0ab3aa88c52ab64a076077ece090a38029eb5f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-4, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-25T01:44:45.326 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:44 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:44.958+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:15.239423+0000 front 2024-04-25T01:44:15.239207+0000 (oldest deadline 2024-04-25T01:44:41.139214+0000) 2024-04-25T01:44:45.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:45 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:45.020+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:45.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:45 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:45.020+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:44:45.548 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:45 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:45.268+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:45.548 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:45 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:45.268+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:44:45.833 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:45 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:45.648+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:44:45.833 INFO:journalctl@ceph.osd.4.smithi032.stdout:Apr 25 01:44:45 smithi032 podman[69156]: 2024-04-25 01:44:45.460299336 +0000 UTC m=+0.018834017 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:44:45.834 INFO:journalctl@ceph.osd.4.smithi032.stdout:Apr 25 01:44:45 smithi032 podman[69156]: 2024-04-25 01:44:45.597432745 +0000 UTC m=+0.155967410 container create 0be1040c21486542f73e7a87c976c599c5012cf3f40d8932c72ba3590688c181 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-25T01:44:45.834 INFO:journalctl@ceph.osd.4.smithi032.stdout:Apr 25 01:44:45 smithi032 podman[69156]: 2024-04-25 01:44:45.842284877 +0000 UTC m=+0.400819554 container init 0be1040c21486542f73e7a87c976c599c5012cf3f40d8932c72ba3590688c181 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424) 2024-04-25T01:44:46.134 INFO:journalctl@ceph.osd.4.smithi032.stdout:Apr 25 01:44:45 smithi032 podman[69156]: 2024-04-25 01:44:45.84572491 +0000 UTC m=+0.404259577 container start 0be1040c21486542f73e7a87c976c599c5012cf3f40d8932c72ba3590688c181 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-25T01:44:46.134 INFO:journalctl@ceph.osd.4.smithi032.stdout:Apr 25 01:44:45 smithi032 podman[69156]: 2024-04-25 01:44:45.934333213 +0000 UTC m=+0.492867888 container attach 0be1040c21486542f73e7a87c976c599c5012cf3f40d8932c72ba3590688c181 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T01:44:46.223 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:46 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:46.077+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:46.224 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:45 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:45.896+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.259648+0000 front 2024-04-25T01:44:10.259821+0000 (oldest deadline 2024-04-25T01:44:35.559727+0000) 2024-04-25T01:44:46.224 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:45 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:45.896+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.462260+0000 front 2024-04-25T01:44:19.462207+0000 (oldest deadline 2024-04-25T01:44:44.762000+0000) 2024-04-25T01:44:46.224 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:46 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:46.047+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:44:46.326 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:46 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:46.008+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:15.239423+0000 front 2024-04-25T01:44:15.239207+0000 (oldest deadline 2024-04-25T01:44:41.139214+0000) 2024-04-25T01:44:46.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:46 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:46.062+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:46.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:46 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:46.062+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:44:46.487 INFO:journalctl@ceph.osd.4.smithi032.stdout:Apr 25 01:44:46 smithi032 podman[69156]: 2024-04-25 01:44:46.142730101 +0000 UTC m=+0.701264771 container died 0be1040c21486542f73e7a87c976c599c5012cf3f40d8932c72ba3590688c181 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-25T01:44:46.487 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:46 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:46.313+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:46.488 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:46 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:46.313+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:44:46.748 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:46 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:46.644+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:44:46.819 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.4.service' 2024-04-25T01:44:46.903 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:44:46.903 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-25T01:44:46.903 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-25T01:44:46.903 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.5 2024-04-25T01:44:47.223 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:47 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:47.123+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:47.224 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:47 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:47.079+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:44:47.224 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:46 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:46.930+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.259648+0000 front 2024-04-25T01:44:10.259821+0000 (oldest deadline 2024-04-25T01:44:35.559727+0000) 2024-04-25T01:44:47.224 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:46 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:46.930+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.462260+0000 front 2024-04-25T01:44:19.462207+0000 (oldest deadline 2024-04-25T01:44:44.762000+0000) 2024-04-25T01:44:47.224 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:46 smithi130 systemd[1]: Stopping Ceph osd.5 for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:44:47.326 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:46 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:46.983+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:15.239423+0000 front 2024-04-25T01:44:15.239207+0000 (oldest deadline 2024-04-25T01:44:41.139214+0000) 2024-04-25T01:44:47.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:47 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:47.046+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:47.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:47 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:47.047+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:44:47.675 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:47 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:47.327+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:47.675 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:47 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:47.327+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:44:47.723 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:47 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:47.241+0000 7f47f3391700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:44:47.723 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:47 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:47.241+0000 7f47f3391700 -1 osd.5 106 *** Got signal Terminated *** 2024-04-25T01:44:47.723 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:47 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:47.241+0000 7f47f3391700 -1 osd.5 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:44:47.998 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:47 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:47.683+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:44:48.223 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:48 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:48.136+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:48.223 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:48 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:48.033+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:44:48.224 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:47 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:47.935+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.259648+0000 front 2024-04-25T01:44:10.259821+0000 (oldest deadline 2024-04-25T01:44:35.559727+0000) 2024-04-25T01:44:48.224 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:47 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:47.935+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.462260+0000 front 2024-04-25T01:44:19.462207+0000 (oldest deadline 2024-04-25T01:44:44.762000+0000) 2024-04-25T01:44:48.326 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:48 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:48.022+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:15.239423+0000 front 2024-04-25T01:44:15.239207+0000 (oldest deadline 2024-04-25T01:44:41.139214+0000) 2024-04-25T01:44:48.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:48 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:48.087+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:48.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:48 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:48.087+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:44:48.691 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:48 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:48.299+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:48.691 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:48 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:48.299+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:44:48.998 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:48 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:48.699+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:44:49.223 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:49 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:49.116+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:49.223 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:48 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:48.892+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.259648+0000 front 2024-04-25T01:44:10.259821+0000 (oldest deadline 2024-04-25T01:44:35.559727+0000) 2024-04-25T01:44:49.224 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:48 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:48.892+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.462260+0000 front 2024-04-25T01:44:19.462207+0000 (oldest deadline 2024-04-25T01:44:44.762000+0000) 2024-04-25T01:44:49.224 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:49 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:49.029+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:44:49.224 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:49 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:49.029+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:44:49.326 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:49 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:49.067+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:15.239423+0000 front 2024-04-25T01:44:15.239207+0000 (oldest deadline 2024-04-25T01:44:41.139214+0000) 2024-04-25T01:44:49.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:49 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:49.047+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:49.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:49 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:49.047+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:44:49.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:49 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:49.047+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:44:49.732 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:49 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:49.267+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:49.733 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:49 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:49.267+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:44:49.998 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:49 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:49.740+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:44:50.001 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:49 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:49.740+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.353457+0000 front 2024-04-25T01:44:27.353433+0000 (oldest deadline 2024-04-25T01:44:49.053519+0000) 2024-04-25T01:44:50.130 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:49 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:49.873+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.259648+0000 front 2024-04-25T01:44:10.259821+0000 (oldest deadline 2024-04-25T01:44:35.559727+0000) 2024-04-25T01:44:50.130 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:49 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:49.874+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.462260+0000 front 2024-04-25T01:44:19.462207+0000 (oldest deadline 2024-04-25T01:44:44.762000+0000) 2024-04-25T01:44:50.131 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:49 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:49.991+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:44:50.131 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:49 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:49.991+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:44:50.326 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:50 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:50.022+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:15.239423+0000 front 2024-04-25T01:44:15.239207+0000 (oldest deadline 2024-04-25T01:44:41.139214+0000) 2024-04-25T01:44:50.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:50 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:50.080+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:50.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:50 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:50.080+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:44:50.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:50 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:50.080+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:44:50.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:50 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:50.138+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:50.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:50 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:50.138+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:44:50.690 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:50 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:50.257+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:50.690 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:50 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:50.257+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:44:50.690 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:50 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:50.257+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:44:50.998 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:50 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:50.698+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:44:50.999 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:50 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:50.698+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.353457+0000 front 2024-04-25T01:44:27.353433+0000 (oldest deadline 2024-04-25T01:44:49.053519+0000) 2024-04-25T01:44:51.145 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:50 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:50.942+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:44:51.145 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:50 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:50.943+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:44:51.145 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:50 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:50.849+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.259648+0000 front 2024-04-25T01:44:10.259821+0000 (oldest deadline 2024-04-25T01:44:35.559727+0000) 2024-04-25T01:44:51.145 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:50 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:50.849+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.462260+0000 front 2024-04-25T01:44:19.462207+0000 (oldest deadline 2024-04-25T01:44:44.762000+0000) 2024-04-25T01:44:51.145 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:50 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:50.849+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:24.762300+0000 front 2024-04-25T01:44:24.762326+0000 (oldest deadline 2024-04-25T01:44:50.662480+0000) 2024-04-25T01:44:51.326 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:50 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:50.984+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:15.239423+0000 front 2024-04-25T01:44:15.239207+0000 (oldest deadline 2024-04-25T01:44:41.139214+0000) 2024-04-25T01:44:51.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:51 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:51.117+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:51.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:51 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:51.117+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:44:51.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:51 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:51.117+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:44:51.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:51 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:51.153+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:51.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:51 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:51.153+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:44:51.724 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:51 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:51.303+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:51.724 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:51 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:51.303+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:44:51.724 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:51 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:51.303+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:44:51.998 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:51 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:51.732+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:44:51.998 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:51 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:51.732+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.353457+0000 front 2024-04-25T01:44:27.353433+0000 (oldest deadline 2024-04-25T01:44:49.053519+0000) 2024-04-25T01:44:52.113 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:51 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:51.935+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:44:52.113 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:51 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:51.935+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:44:52.114 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:51 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:51.870+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.259648+0000 front 2024-04-25T01:44:10.259821+0000 (oldest deadline 2024-04-25T01:44:35.559727+0000) 2024-04-25T01:44:52.114 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:51 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:51.870+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.462260+0000 front 2024-04-25T01:44:19.462207+0000 (oldest deadline 2024-04-25T01:44:44.762000+0000) 2024-04-25T01:44:52.114 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:51 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5[44337]: 2024-04-25T01:44:51.870+0000 7f47eb97d700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:24.762300+0000 front 2024-04-25T01:44:24.762326+0000 (oldest deadline 2024-04-25T01:44:50.662480+0000) 2024-04-25T01:44:52.326 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:51 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:51.942+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:15.239423+0000 front 2024-04-25T01:44:15.239207+0000 (oldest deadline 2024-04-25T01:44:41.139214+0000) 2024-04-25T01:44:52.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:52 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:52.134+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:52.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:52 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:52.134+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:44:52.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:52 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:52.134+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:44:52.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:52 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:52.121+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:52.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:52 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:52.121+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:44:52.748 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:52 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:52.261+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:52.749 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:52 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:52.261+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:44:52.749 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:52 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:52.261+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:44:52.926 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:52 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:52.887+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:44:52.926 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:52 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:52.887+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:44:52.926 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:52 smithi130 podman[68918]: 2024-04-25 01:44:52.643131345 +0000 UTC m=+5.536238984 container died 1d62fdc627f4356865e135732c92da8cea10b11d66c67fe42740894e411e3052 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:44:53.221 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:52 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:52.758+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:44:53.221 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:52 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:52.758+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.353457+0000 front 2024-04-25T01:44:27.353433+0000 (oldest deadline 2024-04-25T01:44:49.053519+0000) 2024-04-25T01:44:53.223 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:53 smithi130 podman[68918]: 2024-04-25 01:44:53.025678134 +0000 UTC m=+5.918785763 container cleanup 1d62fdc627f4356865e135732c92da8cea10b11d66c67fe42740894e411e3052 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-25T01:44:53.223 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:53 smithi130 bash[68918]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5 2024-04-25T01:44:53.223 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:53 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:53.168+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:53.223 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:53 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:53.168+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:44:53.326 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:52 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:52.944+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:15.239423+0000 front 2024-04-25T01:44:15.239207+0000 (oldest deadline 2024-04-25T01:44:41.139214+0000) 2024-04-25T01:44:53.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:53 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:53.157+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:53.327 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:53 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:53.157+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:44:53.327 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:53 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:53.157+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:44:53.498 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:53 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:53.229+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:53.499 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:53 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:53.229+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:44:53.499 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:53 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:53.229+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:44:53.638 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:53 smithi130 podman[68939]: 2024-04-25 01:44:53.302093577 +0000 UTC m=+0.657664515 container remove 1d62fdc627f4356865e135732c92da8cea10b11d66c67fe42740894e411e3052 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-04-25T01:44:53.906 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:53 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:53.888+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:44:53.906 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:53 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:53.888+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:44:53.906 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:53 smithi130 podman[69032]: 2024-04-25 01:44:53.550038388 +0000 UTC m=+0.017802397 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:44:53.906 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:53 smithi130 podman[69032]: 2024-04-25 01:44:53.669667395 +0000 UTC m=+0.137431403 container create 1a6fe7ea9ab1da7060e4652f8fe7453a934cc7c9188af047eaddeff2e9dd28d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:44:53.906 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:53 smithi130 podman[69032]: 2024-04-25 01:44:53.914258933 +0000 UTC m=+0.382022947 container init 1a6fe7ea9ab1da7060e4652f8fe7453a934cc7c9188af047eaddeff2e9dd28d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T01:44:54.188 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:53 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:53.793+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:44:54.189 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:53 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:53.793+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.353457+0000 front 2024-04-25T01:44:27.353433+0000 (oldest deadline 2024-04-25T01:44:49.053519+0000) 2024-04-25T01:44:54.198 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:54 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:54.143+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:54.198 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:54 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:54.143+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:44:54.199 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:53 smithi130 podman[69032]: 2024-04-25 01:44:53.917716813 +0000 UTC m=+0.385480827 container start 1a6fe7ea9ab1da7060e4652f8fe7453a934cc7c9188af047eaddeff2e9dd28d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:44:54.199 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:54 smithi130 podman[69032]: 2024-04-25 01:44:54.013837632 +0000 UTC m=+0.481601660 container attach 1a6fe7ea9ab1da7060e4652f8fe7453a934cc7c9188af047eaddeff2e9dd28d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-25T01:44:54.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:54 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:54.188+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:54.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:54 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:54.188+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:44:54.326 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:54 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:54.188+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:44:54.326 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:53 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:53.986+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:15.239423+0000 front 2024-04-25T01:44:15.239207+0000 (oldest deadline 2024-04-25T01:44:41.139214+0000) 2024-04-25T01:44:54.327 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:53 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:53.986+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.342615+0000 front 2024-04-25T01:44:27.342460+0000 (oldest deadline 2024-04-25T01:44:53.242564+0000) 2024-04-25T01:44:54.473 INFO:journalctl@ceph.osd.5.smithi130.stdout:Apr 25 01:44:54 smithi130 podman[69032]: 2024-04-25 01:44:54.206592088 +0000 UTC m=+0.674356105 container died 1a6fe7ea9ab1da7060e4652f8fe7453a934cc7c9188af047eaddeff2e9dd28d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:44:54.498 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:54 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:54.197+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:54.498 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:54 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:54.197+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:44:54.498 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:54 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:54.197+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:44:54.880 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.5.service' 2024-04-25T01:44:54.917 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:44:54.917 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-25T01:44:54.917 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-25T01:44:54.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.6 2024-04-25T01:44:55.164 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:54 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:54.762+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:44:55.164 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:54 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:54.762+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.353457+0000 front 2024-04-25T01:44:27.353433+0000 (oldest deadline 2024-04-25T01:44:49.053519+0000) 2024-04-25T01:44:55.165 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:54 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:54.859+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:44:55.165 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:54 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:54.859+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:44:55.215 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:54 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:54.950+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:15.239423+0000 front 2024-04-25T01:44:15.239207+0000 (oldest deadline 2024-04-25T01:44:41.139214+0000) 2024-04-25T01:44:55.215 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:54 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:54.950+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.342615+0000 front 2024-04-25T01:44:27.342460+0000 (oldest deadline 2024-04-25T01:44:53.242564+0000) 2024-04-25T01:44:55.215 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:54 smithi028 systemd[1]: Stopping Ceph osd.6 for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:44:55.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:55.173+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:55.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:55.173+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:44:55.498 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:55.172+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:55.498 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:55.172+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:44:55.498 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:55.172+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:44:55.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:55 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:55.222+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:55.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:55 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:55.222+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:44:55.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:55 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:55.222+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:44:55.577 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:55 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:55.247+0000 7fc3e7b7f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:44:55.577 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:55 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:55.247+0000 7fc3e7b7f700 -1 osd.6 106 *** Got signal Terminated *** 2024-04-25T01:44:55.577 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:55 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:55.247+0000 7fc3e7b7f700 -1 osd.6 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:44:55.998 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:55.739+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:44:55.999 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:55.739+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.353457+0000 front 2024-04-25T01:44:27.353433+0000 (oldest deadline 2024-04-25T01:44:49.053519+0000) 2024-04-25T01:44:56.175 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:55.816+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:44:56.175 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:55 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:55.816+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:44:56.233 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:55 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:55.987+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:15.239423+0000 front 2024-04-25T01:44:15.239207+0000 (oldest deadline 2024-04-25T01:44:41.139214+0000) 2024-04-25T01:44:56.233 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:55 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:55.987+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.342615+0000 front 2024-04-25T01:44:27.342460+0000 (oldest deadline 2024-04-25T01:44:53.242564+0000) 2024-04-25T01:44:56.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:56 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:56.183+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:56.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:56 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:56.183+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:44:56.498 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:56 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:56.179+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:56.498 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:56 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:56.179+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:44:56.499 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:56 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:56.179+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:44:56.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:56 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:56.241+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:56.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:56 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:56.241+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:44:56.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:56 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:56.241+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:44:57.186 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:56 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:56.767+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:44:57.186 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:56 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:56.767+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.353457+0000 front 2024-04-25T01:44:27.353433+0000 (oldest deadline 2024-04-25T01:44:49.053519+0000) 2024-04-25T01:44:57.209 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:56 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:56.829+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:44:57.209 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:56 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:56.829+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:44:57.280 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:57 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:57.011+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:15.239423+0000 front 2024-04-25T01:44:15.239207+0000 (oldest deadline 2024-04-25T01:44:41.139214+0000) 2024-04-25T01:44:57.280 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:57 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:57.011+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.342615+0000 front 2024-04-25T01:44:27.342460+0000 (oldest deadline 2024-04-25T01:44:53.242564+0000) 2024-04-25T01:44:57.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:57 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:57.216+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:57.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:57 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:57.216+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:44:57.498 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:57 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:57.194+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:57.499 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:57 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:57.194+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:44:57.499 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:57 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:57.194+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:44:57.499 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:57 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:57.194+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:44:57.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:57 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:57.288+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:57.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:57 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:57.288+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:44:57.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:57 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:57.288+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:44:58.164 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:57 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:57.812+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:44:58.164 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:57 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:57.813+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.353457+0000 front 2024-04-25T01:44:27.353433+0000 (oldest deadline 2024-04-25T01:44:49.053519+0000) 2024-04-25T01:44:58.214 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:57 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:57.862+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:44:58.215 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:57 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:57.862+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:44:58.215 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:57 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:57.862+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.388697+0000 front 2024-04-25T01:44:33.388707+0000 (oldest deadline 2024-04-25T01:44:56.888519+0000) 2024-04-25T01:44:58.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:58.035+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:15.239423+0000 front 2024-04-25T01:44:15.239207+0000 (oldest deadline 2024-04-25T01:44:41.139214+0000) 2024-04-25T01:44:58.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:58.035+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.342615+0000 front 2024-04-25T01:44:27.342460+0000 (oldest deadline 2024-04-25T01:44:53.242564+0000) 2024-04-25T01:44:58.311 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:58.035+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.242898+0000 front 2024-04-25T01:44:33.242949+0000 (oldest deadline 2024-04-25T01:44:57.343058+0000) 2024-04-25T01:44:58.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:58 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:58.222+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:58.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:58 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:58.222+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:44:58.499 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:58 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:58.172+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:58.499 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:58 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:58.172+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:44:58.499 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:58 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:58.172+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:44:58.499 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:58 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:58.172+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:44:58.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:58.319+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:58.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:58.319+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:44:58.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:58.319+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:44:58.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:58 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:58.319+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:44:59.115 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:58 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:58.773+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:44:59.115 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:58 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:58.773+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.353457+0000 front 2024-04-25T01:44:27.353433+0000 (oldest deadline 2024-04-25T01:44:49.053519+0000) 2024-04-25T01:44:59.223 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:58 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:58.880+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:44:59.223 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:58 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:58.880+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:44:59.223 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:58 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:58.880+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.388697+0000 front 2024-04-25T01:44:33.388707+0000 (oldest deadline 2024-04-25T01:44:56.888519+0000) 2024-04-25T01:44:59.295 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:59 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:59.034+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:15.239423+0000 front 2024-04-25T01:44:15.239207+0000 (oldest deadline 2024-04-25T01:44:41.139214+0000) 2024-04-25T01:44:59.295 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:59 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:59.034+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.342615+0000 front 2024-04-25T01:44:27.342460+0000 (oldest deadline 2024-04-25T01:44:53.242564+0000) 2024-04-25T01:44:59.295 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:44:59 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:44:59.034+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.242898+0000 front 2024-04-25T01:44:33.242949+0000 (oldest deadline 2024-04-25T01:44:57.343058+0000) 2024-04-25T01:44:59.498 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:59 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:59.123+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:44:59.499 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:59 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:59.123+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:44:59.499 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:59 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:59.123+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:44:59.499 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:44:59 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:44:59.123+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:44:59.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:59 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:59.303+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:44:59.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:59 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:59.303+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:44:59.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:59 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:59.303+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:44:59.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:44:59 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:44:59.303+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:44:59.723 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:59 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:59.255+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:44:59.723 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:44:59 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:44:59.255+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:00.066 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:59 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:59.814+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:45:00.066 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:59 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:59.814+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.353457+0000 front 2024-04-25T01:44:27.353433+0000 (oldest deadline 2024-04-25T01:44:49.053519+0000) 2024-04-25T01:45:00.066 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:44:59 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:44:59.814+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.455124+0000 front 2024-04-25T01:44:35.455256+0000 (oldest deadline 2024-04-25T01:44:59.555057+0000) 2024-04-25T01:45:00.208 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:59 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:59.858+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:45:00.208 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:59 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:59.858+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:45:00.209 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:44:59 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:44:59.858+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.388697+0000 front 2024-04-25T01:44:33.388707+0000 (oldest deadline 2024-04-25T01:44:56.888519+0000) 2024-04-25T01:45:00.326 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:45:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:45:00.036+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:15.239423+0000 front 2024-04-25T01:44:15.239207+0000 (oldest deadline 2024-04-25T01:44:41.139214+0000) 2024-04-25T01:45:00.326 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:45:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:45:00.036+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.342615+0000 front 2024-04-25T01:44:27.342460+0000 (oldest deadline 2024-04-25T01:44:53.242564+0000) 2024-04-25T01:45:00.326 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:45:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6[55503]: 2024-04-25T01:45:00.036+0000 7fc3e4173700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.242898+0000 front 2024-04-25T01:44:33.242949+0000 (oldest deadline 2024-04-25T01:44:57.343058+0000) 2024-04-25T01:45:00.327 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:00.264+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:00.327 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:00.264+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:00.327 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:00.264+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:00.327 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:00 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:00.264+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:00.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:00 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:00.216+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:00.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:00 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:00.216+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:00.498 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:00 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:00.073+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:00.498 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:00 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:00.073+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:00.498 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:00 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:00.073+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:00.499 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:00 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:00.073+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:00.991 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:45:00 smithi028 podman[102703]: 2024-04-25 01:45:00.597825317 +0000 UTC m=+5.465997584 container died 9917501fa82bd8d8797bf8a018e4591f9276b663c7e2b1b8f20a109f4e120d02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, ceph=True) 2024-04-25T01:45:01.165 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:00 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:00.823+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:45:01.166 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:00 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:00.823+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:45:01.166 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:00 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:00.823+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.388697+0000 front 2024-04-25T01:44:33.388707+0000 (oldest deadline 2024-04-25T01:44:56.888519+0000) 2024-04-25T01:45:01.249 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:00 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:00.850+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:45:01.249 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:00 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:00.850+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.353457+0000 front 2024-04-25T01:44:27.353433+0000 (oldest deadline 2024-04-25T01:44:49.053519+0000) 2024-04-25T01:45:01.249 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:00 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:00.850+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.455124+0000 front 2024-04-25T01:44:35.455256+0000 (oldest deadline 2024-04-25T01:44:59.555057+0000) 2024-04-25T01:45:01.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:01 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:01.054+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:01.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:01 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:01.054+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:01.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:01 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:01.054+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:01.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:01 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:01.054+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:01.255 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:45:01 smithi028 podman[102703]: 2024-04-25 01:45:01.09037113 +0000 UTC m=+5.958543388 container cleanup 9917501fa82bd8d8797bf8a018e4591f9276b663c7e2b1b8f20a109f4e120d02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-25T01:45:01.255 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:45:01 smithi028 bash[102703]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6 2024-04-25T01:45:01.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:01 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:01.173+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:01.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:01 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:01.173+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:01.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:01 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:01.173+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:01.576 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:45:01 smithi028 podman[102720]: 2024-04-25 01:45:01.366836997 +0000 UTC m=+0.765111572 container remove 9917501fa82bd8d8797bf8a018e4591f9276b663c7e2b1b8f20a109f4e120d02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T01:45:01.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:01 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:01.263+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:01.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:01 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:01.263+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:01.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:01 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:01.263+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:01.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:01 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:01.263+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:01.920 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:45:01 smithi028 podman[102808]: 2024-04-25 01:45:01.579218599 +0000 UTC m=+0.018098736 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:45:01.920 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:45:01 smithi028 podman[102808]: 2024-04-25 01:45:01.69259532 +0000 UTC m=+0.131475428 container create 383f28fea26f47b531d87ad52dbe68f597dd24289ec2b49fa441236ab5a63495 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-25T01:45:02.073 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:01 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:01.809+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:45:02.073 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:01 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:01.809+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.353457+0000 front 2024-04-25T01:44:27.353433+0000 (oldest deadline 2024-04-25T01:44:49.053519+0000) 2024-04-25T01:45:02.074 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:01 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:01.809+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.455124+0000 front 2024-04-25T01:44:35.455256+0000 (oldest deadline 2024-04-25T01:44:59.555057+0000) 2024-04-25T01:45:02.140 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:01 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:01.836+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:45:02.140 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:01 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:01.836+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:45:02.140 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:01 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:01.836+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.388697+0000 front 2024-04-25T01:44:33.388707+0000 (oldest deadline 2024-04-25T01:44:56.888519+0000) 2024-04-25T01:45:02.219 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:45:01 smithi028 podman[102808]: 2024-04-25 01:45:01.928594898 +0000 UTC m=+0.367475007 container init 383f28fea26f47b531d87ad52dbe68f597dd24289ec2b49fa441236ab5a63495 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-25T01:45:02.219 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:45:01 smithi028 podman[102808]: 2024-04-25 01:45:01.931841905 +0000 UTC m=+0.370722016 container start 383f28fea26f47b531d87ad52dbe68f597dd24289ec2b49fa441236ab5a63495 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:45:02.220 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:45:02 smithi028 podman[102808]: 2024-04-25 01:45:02.011880945 +0000 UTC m=+0.450761082 container attach 383f28fea26f47b531d87ad52dbe68f597dd24289ec2b49fa441236ab5a63495 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-25T01:45:02.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:02 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:02.147+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:02.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:02 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:02.147+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:02.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:02 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:02.147+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:02.498 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:02 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:02.081+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:02.499 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:02 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:02.081+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:02.499 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:02 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:02.081+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:02.499 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:02 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:02.081+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:02.500 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 25 01:45:02 smithi028 podman[102808]: 2024-04-25 01:45:02.228021575 +0000 UTC m=+0.666901680 container died 383f28fea26f47b531d87ad52dbe68f597dd24289ec2b49fa441236ab5a63495 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-25T01:45:02.500 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:02 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:02.266+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:02.500 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:02 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:02.266+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:02.500 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:02 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:02.266+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:02.500 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:02 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:02.266+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:02.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.6.service' 2024-04-25T01:45:02.929 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:45:02.929 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-25T01:45:02.929 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-25T01:45:02.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.7 2024-04-25T01:45:03.096 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:02 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:02.802+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:45:03.097 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:02 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:02.802+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:45:03.097 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:02 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:02.802+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.388697+0000 front 2024-04-25T01:44:33.388707+0000 (oldest deadline 2024-04-25T01:44:56.888519+0000) 2024-04-25T01:45:03.249 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:02 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:02.802+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:45:03.249 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:02 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:02.802+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.353457+0000 front 2024-04-25T01:44:27.353433+0000 (oldest deadline 2024-04-25T01:44:49.053519+0000) 2024-04-25T01:45:03.249 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:02 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:02.802+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.455124+0000 front 2024-04-25T01:44:35.455256+0000 (oldest deadline 2024-04-25T01:44:59.555057+0000) 2024-04-25T01:45:03.249 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:02 smithi032 systemd[1]: Stopping Ceph osd.7 for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:45:03.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:03 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:03.050+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:03.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:03 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:03.050+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:03.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:03 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:03.050+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:03.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:03 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:03.050+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:03.411 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:03 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:03.104+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:03.411 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:03 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:03.104+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:03.411 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:03 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:03.104+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:03.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:03 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:03.307+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:03.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:03 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:03.307+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:03.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:03 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:03.307+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:03.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:03 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:03.307+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:03.749 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:03 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:03.284+0000 7fe007556700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:45:03.749 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:03 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:03.284+0000 7fe007556700 -1 osd.7 106 *** Got signal Terminated *** 2024-04-25T01:45:03.749 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:03 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:03.284+0000 7fe007556700 -1 osd.7 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:45:04.127 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:03 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:03.789+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:45:04.127 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:03 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:03.789+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:45:04.127 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:03 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:03.789+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.388697+0000 front 2024-04-25T01:44:33.388707+0000 (oldest deadline 2024-04-25T01:44:56.888519+0000) 2024-04-25T01:45:04.249 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:03 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:03.781+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:45:04.249 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:03 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:03.781+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.353457+0000 front 2024-04-25T01:44:27.353433+0000 (oldest deadline 2024-04-25T01:44:49.053519+0000) 2024-04-25T01:45:04.249 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:03 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:03.781+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.455124+0000 front 2024-04-25T01:44:35.455256+0000 (oldest deadline 2024-04-25T01:44:59.555057+0000) 2024-04-25T01:45:04.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:04 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:04.029+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:04.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:04 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:04.029+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:04.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:04 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:04.029+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:04.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:04 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:04.029+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:04.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:04 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:04.134+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:04.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:04 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:04.134+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:04.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:04 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:04.134+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:04.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:04 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:04.294+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:04.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:04 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:04.294+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:04.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:04 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:04.294+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:04.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:04 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:04.294+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:05.014 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:04 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:04.771+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:45:05.014 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:04 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:04.771+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.353457+0000 front 2024-04-25T01:44:27.353433+0000 (oldest deadline 2024-04-25T01:44:49.053519+0000) 2024-04-25T01:45:05.014 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:04 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:04.771+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.455124+0000 front 2024-04-25T01:44:35.455256+0000 (oldest deadline 2024-04-25T01:44:59.555057+0000) 2024-04-25T01:45:05.158 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:04 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:04.762+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:45:05.158 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:04 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:04.762+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:45:05.158 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:04 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:04.762+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.388697+0000 front 2024-04-25T01:44:33.388707+0000 (oldest deadline 2024-04-25T01:44:56.888519+0000) 2024-04-25T01:45:05.300 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:05 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:05.021+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:05.301 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:05 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:05.021+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:05.301 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:05 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:05.021+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:05.301 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:05 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:05.021+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:05.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:05 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:05.165+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:05.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:05 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:05.165+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:05.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:05 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:05.165+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:05.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:05 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:05.275+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:05.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:05 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:05.275+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:05.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:05 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:05.275+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:05.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:05 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:05.275+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:05.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:05 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:05.275+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.781433+0000 front 2024-04-25T01:44:42.781480+0000 (oldest deadline 2024-04-25T01:45:05.081357+0000) 2024-04-25T01:45:05.998 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:05 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:05.735+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:45:05.999 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:05 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:05.735+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.353457+0000 front 2024-04-25T01:44:27.353433+0000 (oldest deadline 2024-04-25T01:44:49.053519+0000) 2024-04-25T01:45:05.999 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:05 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:05.735+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.455124+0000 front 2024-04-25T01:44:35.455256+0000 (oldest deadline 2024-04-25T01:44:59.555057+0000) 2024-04-25T01:45:05.999 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:05 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:05.980+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:05.999 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:05 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:05.980+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:05.999 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:05 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:05.980+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:05.999 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:05 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:05.980+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:06.181 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:05 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:05.755+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:45:06.182 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:05 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:05.755+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:45:06.182 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:05 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:05.755+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.388697+0000 front 2024-04-25T01:44:33.388707+0000 (oldest deadline 2024-04-25T01:44:56.888519+0000) 2024-04-25T01:45:06.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:06 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:06.189+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:06.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:06 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:06.189+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:06.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:06 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:06.189+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:06.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:06 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:06.189+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:06.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:06 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:06.252+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:06.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:06 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:06.252+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:06.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:06 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:06.252+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:06.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:06 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:06.252+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:06.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:06 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:06.252+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.781433+0000 front 2024-04-25T01:44:42.781480+0000 (oldest deadline 2024-04-25T01:45:05.081357+0000) 2024-04-25T01:45:07.217 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:06 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:06.772+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:45:07.217 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:06 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:06.772+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:45:07.217 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:06 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:06.772+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.388697+0000 front 2024-04-25T01:44:33.388707+0000 (oldest deadline 2024-04-25T01:44:56.888519+0000) 2024-04-25T01:45:07.249 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:06 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:06.766+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:45:07.249 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:06 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:06.766+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.353457+0000 front 2024-04-25T01:44:27.353433+0000 (oldest deadline 2024-04-25T01:44:49.053519+0000) 2024-04-25T01:45:07.249 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:06 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:06.766+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.455124+0000 front 2024-04-25T01:44:35.455256+0000 (oldest deadline 2024-04-25T01:44:59.555057+0000) 2024-04-25T01:45:07.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:06 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:06.960+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:07.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:06 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:06.960+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:07.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:06 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:06.960+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:07.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:06 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:06.960+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:07.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:07 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:07.224+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:07.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:07 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:07.224+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:07.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:07 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:07.224+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:07.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:07 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:07.224+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:07.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:07 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:07.251+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:07.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:07 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:07.251+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:07.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:07 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:07.251+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:07.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:07 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:07.251+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:07.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:07 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:07.251+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.781433+0000 front 2024-04-25T01:44:42.781480+0000 (oldest deadline 2024-04-25T01:45:05.081357+0000) 2024-04-25T01:45:08.224 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:07 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:07.764+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:45:08.224 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:07 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:07.764+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:45:08.224 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:07 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:07.764+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.388697+0000 front 2024-04-25T01:44:33.388707+0000 (oldest deadline 2024-04-25T01:44:56.888519+0000) 2024-04-25T01:45:08.224 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:07 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:07.764+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:40.389977+0000 front 2024-04-25T01:44:40.389922+0000 (oldest deadline 2024-04-25T01:45:06.790169+0000) 2024-04-25T01:45:08.248 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:07 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:07.805+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.152117+0000 front 2024-04-25T01:44:19.152168+0000 (oldest deadline 2024-04-25T01:44:42.052230+0000) 2024-04-25T01:45:08.249 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:07 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:07.805+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.353457+0000 front 2024-04-25T01:44:27.353433+0000 (oldest deadline 2024-04-25T01:44:49.053519+0000) 2024-04-25T01:45:08.249 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:07 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:07.805+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.455124+0000 front 2024-04-25T01:44:35.455256+0000 (oldest deadline 2024-04-25T01:44:59.555057+0000) 2024-04-25T01:45:08.249 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:07 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7[50879]: 2024-04-25T01:45:07.805+0000 7fe003b4a700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.456119+0000 front 2024-04-25T01:44:42.456151+0000 (oldest deadline 2024-04-25T01:45:07.156342+0000) 2024-04-25T01:45:08.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:07 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:07.924+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:08.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:07 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:07.924+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:08.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:07 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:07.924+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:08.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:07 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:07.924+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:08.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:07 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:07.924+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:08.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:08 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:08.244+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:08.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:08 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:08.244+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:08.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:08 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:08.244+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:08.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:08 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:08.244+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:08.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:08 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:08.244+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.781433+0000 front 2024-04-25T01:44:42.781480+0000 (oldest deadline 2024-04-25T01:45:05.081357+0000) 2024-04-25T01:45:08.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:08 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:08.233+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:08.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:08 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:08.233+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:08.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:08 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:08.233+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:08.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:08 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:08.233+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:08.977 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:08 smithi032 podman[69260]: 2024-04-25 01:45:08.701772064 +0000 UTC m=+5.558507636 container died 5446ee14100d1df16ff86ede98045b2d4d45b8b7e2bb9bc83c936af0f745a8fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-25T01:45:08.977 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:08 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:08.877+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:08.978 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:08 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:08.877+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:08.978 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:08 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:08.877+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:08.978 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:08 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:08.877+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:08.978 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:08 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:08.877+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:09.187 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:08 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:08.738+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:45:09.187 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:08 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:08.738+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:45:09.187 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:08 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:08.738+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.388697+0000 front 2024-04-25T01:44:33.388707+0000 (oldest deadline 2024-04-25T01:44:56.888519+0000) 2024-04-25T01:45:09.187 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:08 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:08.738+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:40.389977+0000 front 2024-04-25T01:44:40.389922+0000 (oldest deadline 2024-04-25T01:45:06.790169+0000) 2024-04-25T01:45:09.249 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:09 smithi032 podman[69260]: 2024-04-25 01:45:09.075555118 +0000 UTC m=+5.932290689 container cleanup 5446ee14100d1df16ff86ede98045b2d4d45b8b7e2bb9bc83c936af0f745a8fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True) 2024-04-25T01:45:09.249 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:09 smithi032 bash[69260]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7 2024-04-25T01:45:09.473 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:09 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:09.194+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:09.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:09 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:09.194+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:09.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:09 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:09.194+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:09.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:09 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:09.194+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:09.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:09 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:09.259+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:09.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:09 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:09.259+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:09.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:09 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:09.259+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:09.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:09 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:09.259+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:09.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:09 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:09.259+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.781433+0000 front 2024-04-25T01:44:42.781480+0000 (oldest deadline 2024-04-25T01:45:05.081357+0000) 2024-04-25T01:45:09.636 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:09 smithi032 podman[69290]: 2024-04-25 01:45:09.36080003 +0000 UTC m=+0.653044642 container remove 5446ee14100d1df16ff86ede98045b2d4d45b8b7e2bb9bc83c936af0f745a8fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T01:45:09.919 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:09 smithi032 podman[69377]: 2024-04-25 01:45:09.644897384 +0000 UTC m=+0.100492591 container create 4e33d93e1a66fab9c031f542a5f05b3833f7605fc0cf48d3d7ae7f303619cb06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-25T01:45:09.920 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:09 smithi032 podman[69377]: 2024-04-25 01:45:09.563098346 +0000 UTC m=+0.018693520 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:45:09.920 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:09 smithi032 podman[69377]: 2024-04-25 01:45:09.88106521 +0000 UTC m=+0.336660382 container init 4e33d93e1a66fab9c031f542a5f05b3833f7605fc0cf48d3d7ae7f303619cb06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-25T01:45:09.920 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:09 smithi032 podman[69377]: 2024-04-25 01:45:09.884674304 +0000 UTC m=+0.340269467 container start 4e33d93e1a66fab9c031f542a5f05b3833f7605fc0cf48d3d7ae7f303619cb06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-25T01:45:10.157 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:09 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:09.754+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:45:10.157 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:09 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:09.754+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:45:10.157 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:09 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:09.754+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.388697+0000 front 2024-04-25T01:44:33.388707+0000 (oldest deadline 2024-04-25T01:44:56.888519+0000) 2024-04-25T01:45:10.157 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:09 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:09.754+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:40.389977+0000 front 2024-04-25T01:44:40.389922+0000 (oldest deadline 2024-04-25T01:45:06.790169+0000) 2024-04-25T01:45:10.179 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:09 smithi032 podman[69377]: 2024-04-25 01:45:09.98956589 +0000 UTC m=+0.445161053 container attach 4e33d93e1a66fab9c031f542a5f05b3833f7605fc0cf48d3d7ae7f303619cb06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS) 2024-04-25T01:45:10.180 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:09 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:09.926+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:10.180 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:09 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:09.926+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:10.180 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:09 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:09.926+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:10.180 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:09 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:09.926+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:10.180 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:09 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:09.926+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:10.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:10 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:10.164+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:10.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:10 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:10.164+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:10.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:10 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:10.164+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:10.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:10 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:10.164+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:10.498 INFO:journalctl@ceph.osd.7.smithi032.stdout:Apr 25 01:45:10 smithi032 podman[69377]: 2024-04-25 01:45:10.188008718 +0000 UTC m=+0.643603889 container died 4e33d93e1a66fab9c031f542a5f05b3833f7605fc0cf48d3d7ae7f303619cb06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:45:10.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:10 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:10.273+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:10.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:10 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:10.273+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:10.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:10 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:10.273+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:10.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:10 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:10.273+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:10.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:10 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:10.273+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.781433+0000 front 2024-04-25T01:44:42.781480+0000 (oldest deadline 2024-04-25T01:45:05.081357+0000) 2024-04-25T01:45:10.857 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.7.service' 2024-04-25T01:45:10.901 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:45:10.901 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-25T01:45:10.901 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-25T01:45:10.902 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.8 2024-04-25T01:45:11.128 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:10 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:10.737+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:45:11.129 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:10 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:10.737+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:45:11.129 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:10 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:10.737+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.388697+0000 front 2024-04-25T01:44:33.388707+0000 (oldest deadline 2024-04-25T01:44:56.888519+0000) 2024-04-25T01:45:11.129 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:10 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:10.737+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:40.389977+0000 front 2024-04-25T01:44:40.389922+0000 (oldest deadline 2024-04-25T01:45:06.790169+0000) 2024-04-25T01:45:11.129 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:10 smithi130 systemd[1]: Stopping Ceph osd.8 for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:45:11.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:10 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:10.893+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:11.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:10 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:10.893+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:11.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:10 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:10.893+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:11.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:10 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:10.893+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:11.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:10 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:10.893+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:11.474 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:11 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:11.227+0000 7fa724baa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:45:11.474 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:11 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:11.227+0000 7fa724baa700 -1 osd.8 106 *** Got signal Terminated *** 2024-04-25T01:45:11.474 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:11 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:11.227+0000 7fa724baa700 -1 osd.8 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:45:11.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:11 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:11.135+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:11.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:11 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:11.135+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:11.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:11 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:11.135+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:11.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:11 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:11.135+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:11.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:11 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:11.263+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:11.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:11 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:11.263+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:11.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:11 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:11.263+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:11.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:11 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:11.263+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:11.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:11 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:11.263+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.781433+0000 front 2024-04-25T01:44:42.781480+0000 (oldest deadline 2024-04-25T01:45:05.081357+0000) 2024-04-25T01:45:11.974 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:11 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:11.701+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:45:11.974 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:11 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:11.701+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:45:11.974 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:11 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:11.701+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.388697+0000 front 2024-04-25T01:44:33.388707+0000 (oldest deadline 2024-04-25T01:44:56.888519+0000) 2024-04-25T01:45:11.974 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:11 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:11.701+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:40.389977+0000 front 2024-04-25T01:44:40.389922+0000 (oldest deadline 2024-04-25T01:45:06.790169+0000) 2024-04-25T01:45:12.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:11 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:11.871+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:12.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:11 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:11.871+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:12.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:11 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:11.871+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:12.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:11 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:11.871+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:12.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:11 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:11.871+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:12.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:12 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:12.088+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:12.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:12 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:12.088+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:12.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:12 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:12.088+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:12.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:12 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:12.088+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:12.576 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:12 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:12.297+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:12.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:12 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:12.297+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:12.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:12 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:12.297+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:12.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:12 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:12.297+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:12.577 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:12 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:12.297+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.781433+0000 front 2024-04-25T01:44:42.781480+0000 (oldest deadline 2024-04-25T01:45:05.081357+0000) 2024-04-25T01:45:12.974 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:12 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:12.687+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:45:12.974 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:12 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:12.687+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:45:12.974 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:12 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:12.687+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.388697+0000 front 2024-04-25T01:44:33.388707+0000 (oldest deadline 2024-04-25T01:44:56.888519+0000) 2024-04-25T01:45:12.974 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:12 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:12.687+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:40.389977+0000 front 2024-04-25T01:44:40.389922+0000 (oldest deadline 2024-04-25T01:45:06.790169+0000) 2024-04-25T01:45:13.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:12 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:12.888+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:13.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:12 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:12.888+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:13.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:12 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:12.888+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:13.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:12 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:12.888+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:13.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:12 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:12.888+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:13.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:12 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:12.888+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:13.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:13 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:13.064+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:13.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:13 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:13.064+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:13.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:13 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:13.064+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:13.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:13 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:13.064+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:13.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:13 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:13.335+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:13.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:13 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:13.335+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:13.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:13 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:13.335+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:13.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:13 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:13.335+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:13.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:13 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:13.335+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.781433+0000 front 2024-04-25T01:44:42.781480+0000 (oldest deadline 2024-04-25T01:45:05.081357+0000) 2024-04-25T01:45:13.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:13 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:13.335+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:48.582703+0000 front 2024-04-25T01:44:48.582667+0000 (oldest deadline 2024-04-25T01:45:13.282568+0000) 2024-04-25T01:45:14.054 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:13 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:13.733+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:45:14.054 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:13 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:13.733+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:45:14.054 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:13 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:13.733+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.388697+0000 front 2024-04-25T01:44:33.388707+0000 (oldest deadline 2024-04-25T01:44:56.888519+0000) 2024-04-25T01:45:14.054 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:13 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:13.733+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:40.389977+0000 front 2024-04-25T01:44:40.389922+0000 (oldest deadline 2024-04-25T01:45:06.790169+0000) 2024-04-25T01:45:14.054 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:13 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:13.733+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:46.790571+0000 front 2024-04-25T01:44:46.790645+0000 (oldest deadline 2024-04-25T01:45:12.690720+0000) 2024-04-25T01:45:14.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:13 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:13.917+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:14.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:13 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:13.917+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:14.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:13 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:13.917+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:14.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:13 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:13.917+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:14.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:13 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:13.917+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:14.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:13 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:13.917+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:14.414 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:14 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:14.061+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:14.415 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:14 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:14.061+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:14.415 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:14 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:14.061+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:14.415 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:14 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:14.061+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:14.826 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:14 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:14.358+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:14.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:14 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:14.358+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:14.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:14 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:14.358+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:14.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:14 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:14.358+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:14.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:14 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:14.358+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.781433+0000 front 2024-04-25T01:44:42.781480+0000 (oldest deadline 2024-04-25T01:45:05.081357+0000) 2024-04-25T01:45:14.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:14 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:14.358+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:48.582703+0000 front 2024-04-25T01:44:48.582667+0000 (oldest deadline 2024-04-25T01:45:13.282568+0000) 2024-04-25T01:45:15.078 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:14 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:14.754+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:45:15.079 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:14 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:14.754+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:45:15.079 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:14 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:14.754+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.388697+0000 front 2024-04-25T01:44:33.388707+0000 (oldest deadline 2024-04-25T01:44:56.888519+0000) 2024-04-25T01:45:15.079 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:14 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:14.754+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:40.389977+0000 front 2024-04-25T01:44:40.389922+0000 (oldest deadline 2024-04-25T01:45:06.790169+0000) 2024-04-25T01:45:15.079 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:14 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:14.754+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:46.790571+0000 front 2024-04-25T01:44:46.790645+0000 (oldest deadline 2024-04-25T01:45:12.690720+0000) 2024-04-25T01:45:15.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:14 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:14.885+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:15.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:14 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:14.885+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:15.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:14 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:14.885+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:15.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:14 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:14.885+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:15.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:14 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:14.885+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:15.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:14 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:14.885+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:15.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:15 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:15.086+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:15.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:15 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:15.086+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:15.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:15 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:15.086+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:15.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:15 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:15.086+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:15.826 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:15 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:15.356+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:15.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:15 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:15.356+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:15.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:15 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:15.356+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:15.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:15 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:15.356+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:15.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:15 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:15.356+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.781433+0000 front 2024-04-25T01:44:42.781480+0000 (oldest deadline 2024-04-25T01:45:05.081357+0000) 2024-04-25T01:45:15.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:15 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:15.356+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:48.582703+0000 front 2024-04-25T01:44:48.582667+0000 (oldest deadline 2024-04-25T01:45:13.282568+0000) 2024-04-25T01:45:16.107 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:15 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:15.771+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.985390+0000 front 2024-04-25T01:44:18.985434+0000 (oldest deadline 2024-04-25T01:44:45.985925+0000) 2024-04-25T01:45:16.107 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:15 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:15.771+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:25.986293+0000 front 2024-04-25T01:44:25.986400+0000 (oldest deadline 2024-04-25T01:44:48.786965+0000) 2024-04-25T01:45:16.107 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:15 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:15.771+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:33.388697+0000 front 2024-04-25T01:44:33.388707+0000 (oldest deadline 2024-04-25T01:44:56.888519+0000) 2024-04-25T01:45:16.107 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:15 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:15.771+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:40.389977+0000 front 2024-04-25T01:44:40.389922+0000 (oldest deadline 2024-04-25T01:45:06.790169+0000) 2024-04-25T01:45:16.107 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:15 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8[51138]: 2024-04-25T01:45:15.771+0000 7fa72119e700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:46.790571+0000 front 2024-04-25T01:44:46.790645+0000 (oldest deadline 2024-04-25T01:45:12.690720+0000) 2024-04-25T01:45:16.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:15 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:15.874+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:16.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:15 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:15.874+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:16.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:15 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:15.874+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:16.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:15 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:15.874+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:16.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:15 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:15.874+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:16.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:15 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:15.874+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:16.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:16 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:16.114+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:16.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:16 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:16.114+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:16.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:16 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:16.114+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:16.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:16 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:16.114+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:16.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:16 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:16.114+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:16.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:16 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:16.398+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:16.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:16 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:16.398+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:16.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:16 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:16.398+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:16.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:16 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:16.398+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:16.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:16 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:16.398+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.781433+0000 front 2024-04-25T01:44:42.781480+0000 (oldest deadline 2024-04-25T01:45:05.081357+0000) 2024-04-25T01:45:16.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:16 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:16.398+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:48.582703+0000 front 2024-04-25T01:44:48.582667+0000 (oldest deadline 2024-04-25T01:45:13.282568+0000) 2024-04-25T01:45:16.871 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:16 smithi130 podman[69150]: 2024-04-25 01:45:16.614141867 +0000 UTC m=+5.514552448 container died 33660ba81c172ff6a15775a433cd623fd628db38f97df36423ba5f9d6b4197f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-25T01:45:17.127 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:16 smithi130 podman[69150]: 2024-04-25 01:45:16.978316862 +0000 UTC m=+5.878727441 container cleanup 33660ba81c172ff6a15775a433cd623fd628db38f97df36423ba5f9d6b4197f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T01:45:17.127 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:16 smithi130 bash[69150]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8 2024-04-25T01:45:17.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:16 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:16.890+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:17.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:16 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:16.890+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:17.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:16 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:16.890+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:17.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:16 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:16.890+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:17.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:16 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:16.890+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:17.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:16 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:16.890+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:17.473 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:17 smithi130 podman[69167]: 2024-04-25 01:45:17.288537348 +0000 UTC m=+0.670453789 container remove 33660ba81c172ff6a15775a433cd623fd628db38f97df36423ba5f9d6b4197f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-25T01:45:17.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:17 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:17.134+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:17.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:17 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:17.134+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:17.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:17 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:17.134+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:17.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:17 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:17.134+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:17.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:17 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:17.134+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:17.826 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:17 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:17.417+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:17.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:17 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:17.417+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:17.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:17 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:17.417+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:17.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:17 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:17.417+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:17.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:17 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:17.417+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.781433+0000 front 2024-04-25T01:44:42.781480+0000 (oldest deadline 2024-04-25T01:45:05.081357+0000) 2024-04-25T01:45:17.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:17 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:17.417+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:48.582703+0000 front 2024-04-25T01:44:48.582667+0000 (oldest deadline 2024-04-25T01:45:13.282568+0000) 2024-04-25T01:45:17.833 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:17 smithi130 podman[69260]: 2024-04-25 01:45:17.487791702 +0000 UTC m=+0.018146682 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:45:17.834 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:17 smithi130 podman[69260]: 2024-04-25 01:45:17.614238625 +0000 UTC m=+0.144593582 container create 2e25f742eeb60be441e052f130e5bc48775fade8d86cf2a8d6c23e1828915685 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-25T01:45:17.834 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:17 smithi130 podman[69260]: 2024-04-25 01:45:17.841730335 +0000 UTC m=+0.372085292 container init 2e25f742eeb60be441e052f130e5bc48775fade8d86cf2a8d6c23e1828915685 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:45:18.101 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:17 smithi130 podman[69260]: 2024-04-25 01:45:17.845563257 +0000 UTC m=+0.375918215 container start 2e25f742eeb60be441e052f130e5bc48775fade8d86cf2a8d6c23e1828915685 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-25T01:45:18.101 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:17 smithi130 podman[69260]: 2024-04-25 01:45:17.950442497 +0000 UTC m=+0.480797474 container attach 2e25f742eeb60be441e052f130e5bc48775fade8d86cf2a8d6c23e1828915685 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9) 2024-04-25T01:45:18.253 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:17 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:17.879+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:18.253 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:17 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:17.879+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:18.253 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:17 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:17.879+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:18.253 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:17 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:17.879+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:18.253 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:17 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:17.879+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:18.253 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:17 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:17.879+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:18.393 INFO:journalctl@ceph.osd.8.smithi130.stdout:Apr 25 01:45:18 smithi130 podman[69260]: 2024-04-25 01:45:18.134106601 +0000 UTC m=+0.664461571 container died 2e25f742eeb60be441e052f130e5bc48775fade8d86cf2a8d6c23e1828915685 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-25T01:45:18.394 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:18 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:18.108+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:18.394 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:18 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:18.108+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:18.394 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:18 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:18.108+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:18.394 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:18 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:18.108+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:18.394 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:18 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:18.108+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:18.820 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.8.service' 2024-04-25T01:45:18.826 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:18 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:18.429+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:18.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:18 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:18.429+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:18.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:18 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:18.429+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:18.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:18 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:18.429+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:18.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:18 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:18.429+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.781433+0000 front 2024-04-25T01:44:42.781480+0000 (oldest deadline 2024-04-25T01:45:05.081357+0000) 2024-04-25T01:45:18.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:18 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:18.429+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:48.582703+0000 front 2024-04-25T01:44:48.582667+0000 (oldest deadline 2024-04-25T01:45:13.282568+0000) 2024-04-25T01:45:18.857 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:45:18.857 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-25T01:45:18.857 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-25T01:45:18.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.9 2024-04-25T01:45:19.145 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:18 smithi028 systemd[1]: Stopping Ceph osd.9 for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:45:19.145 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:19 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:19.151+0000 7fcca1183700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:45:19.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:18 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:18.864+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:19.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:18 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:18.864+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:19.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:18 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:18.864+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:19.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:18 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:18.864+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:19.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:18 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:18.864+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:19.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:18 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:18.864+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:19.417 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:19 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:19.151+0000 7fcca1183700 -1 osd.9 106 *** Got signal Terminated *** 2024-04-25T01:45:19.417 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:19 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:19.151+0000 7fcca1183700 -1 osd.9 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:45:19.417 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:19 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:19.424+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:19.417 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:19 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:19.424+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:19.417 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:19 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:19.424+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:19.417 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:19 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:19.424+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:19.417 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:19 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:19.424+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.781433+0000 front 2024-04-25T01:44:42.781480+0000 (oldest deadline 2024-04-25T01:45:05.081357+0000) 2024-04-25T01:45:19.417 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:19 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:19.424+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:48.582703+0000 front 2024-04-25T01:44:48.582667+0000 (oldest deadline 2024-04-25T01:45:13.282568+0000) 2024-04-25T01:45:19.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:19 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:19.134+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:19.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:19 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:19.134+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:19.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:19 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:19.134+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:19.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:19 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:19.134+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:19.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:19 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:19.134+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:20.238 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:19 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:19.833+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:20.238 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:19 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:19.833+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:20.239 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:19 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:19.833+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:20.239 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:19 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:19.833+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:20.239 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:19 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:19.833+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:20.239 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:19 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:19.833+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:20.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:20 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:20.167+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:20.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:20 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:20.167+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:20.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:20 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:20.167+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:20.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:20 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:20.167+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:20.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:20 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:20.167+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:20.826 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:20 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:20.396+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:20.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:20 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:20.396+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:20.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:20 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:20.396+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:20.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:20 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:20.396+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:20.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:20 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:20.396+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.781433+0000 front 2024-04-25T01:44:42.781480+0000 (oldest deadline 2024-04-25T01:45:05.081357+0000) 2024-04-25T01:45:20.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:20 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:20.396+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:48.582703+0000 front 2024-04-25T01:44:48.582667+0000 (oldest deadline 2024-04-25T01:45:13.282568+0000) 2024-04-25T01:45:21.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:20 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:20.785+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:21.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:20 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:20.785+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:21.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:20 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:20.785+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:21.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:20 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:20.785+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:21.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:20 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:20.785+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:21.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:20 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:20.785+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:21.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:21 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:21.216+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:21.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:21 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:21.216+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:21.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:21 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:21.216+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:21.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:21 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:21.216+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:21.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:21 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:21.216+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:21.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:21 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:21.390+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:21.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:21 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:21.390+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:21.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:21 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:21.390+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:21.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:21 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:21.390+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:21.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:21 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:21.390+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.781433+0000 front 2024-04-25T01:44:42.781480+0000 (oldest deadline 2024-04-25T01:45:05.081357+0000) 2024-04-25T01:45:21.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:21 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:21.390+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:48.582703+0000 front 2024-04-25T01:44:48.582667+0000 (oldest deadline 2024-04-25T01:45:13.282568+0000) 2024-04-25T01:45:22.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:21 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:21.789+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:22.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:21 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:21.789+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:22.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:21 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:21.789+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:22.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:21 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:21.789+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:22.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:21 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:21.789+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:22.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:21 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:21.789+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:22.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:21 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:21.789+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:56.886337+0000 front 2024-04-25T01:44:56.886406+0000 (oldest deadline 2024-04-25T01:45:21.586131+0000) 2024-04-25T01:45:22.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:22 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:22.207+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:22.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:22 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:22.207+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:22.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:22 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:22.207+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:22.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:22 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:22.207+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:22.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:22 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:22.207+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:22.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:22 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:22.207+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:59.228913+0000 front 2024-04-25T01:44:59.228881+0000 (oldest deadline 2024-04-25T01:45:21.528858+0000) 2024-04-25T01:45:22.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:22 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:22.358+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:22.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:22 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:22.358+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:22.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:22 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:22.358+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:22.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:22 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:22.358+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:22.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:22 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:22.358+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.781433+0000 front 2024-04-25T01:44:42.781480+0000 (oldest deadline 2024-04-25T01:45:05.081357+0000) 2024-04-25T01:45:22.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:22 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:22.358+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:48.582703+0000 front 2024-04-25T01:44:48.582667+0000 (oldest deadline 2024-04-25T01:45:13.282568+0000) 2024-04-25T01:45:23.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:22 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:22.776+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:23.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:22 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:22.776+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:23.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:22 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:22.776+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:23.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:22 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:22.776+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:23.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:22 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:22.776+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:23.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:22 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:22.776+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:23.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:22 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:22.776+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:56.886337+0000 front 2024-04-25T01:44:56.886406+0000 (oldest deadline 2024-04-25T01:45:21.586131+0000) 2024-04-25T01:45:23.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:23 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:23.244+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:23.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:23 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:23.244+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:23.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:23 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:23.244+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:23.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:23 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:23.244+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:23.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:23 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:23.244+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:23.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:23 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:23.244+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:59.228913+0000 front 2024-04-25T01:44:59.228881+0000 (oldest deadline 2024-04-25T01:45:21.528858+0000) 2024-04-25T01:45:23.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:23 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:23.335+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:11.474795+0000 front 2024-04-25T01:44:11.474818+0000 (oldest deadline 2024-04-25T01:44:33.075612+0000) 2024-04-25T01:45:23.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:23 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:23.335+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:19.477548+0000 front 2024-04-25T01:44:19.477414+0000 (oldest deadline 2024-04-25T01:44:43.577477+0000) 2024-04-25T01:45:23.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:23 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:23.335+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.078626+0000 front 2024-04-25T01:44:27.078592+0000 (oldest deadline 2024-04-25T01:44:48.778758+0000) 2024-04-25T01:45:23.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:23 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:23.335+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.780322+0000 front 2024-04-25T01:44:35.780494+0000 (oldest deadline 2024-04-25T01:44:58.080412+0000) 2024-04-25T01:45:23.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:23 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:23.335+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:42.781433+0000 front 2024-04-25T01:44:42.781480+0000 (oldest deadline 2024-04-25T01:45:05.081357+0000) 2024-04-25T01:45:23.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:23 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9[62263]: 2024-04-25T01:45:23.335+0000 7fcc9d777700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:48.582703+0000 front 2024-04-25T01:44:48.582667+0000 (oldest deadline 2024-04-25T01:45:13.282568+0000) 2024-04-25T01:45:23.999 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:23 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:23.750+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:23.999 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:23 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:23.750+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:23.999 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:23 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:23.750+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:23.999 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:23 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:23.750+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:23.999 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:23 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:23.750+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:23.999 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:23 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:23.750+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:24.000 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:23 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:23.750+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:56.886337+0000 front 2024-04-25T01:44:56.886406+0000 (oldest deadline 2024-04-25T01:45:21.586131+0000) 2024-04-25T01:45:24.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:24 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:24.256+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:24.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:24 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:24.256+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:24.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:24 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:24.256+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:24.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:24 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:24.256+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:24.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:24 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:24.256+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:24.725 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:24 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:24.256+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:59.228913+0000 front 2024-04-25T01:44:59.228881+0000 (oldest deadline 2024-04-25T01:45:21.528858+0000) 2024-04-25T01:45:24.827 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:24 smithi028 podman[102926]: 2024-04-25 01:45:24.523864201 +0000 UTC m=+5.449974842 container died 7223c0482776ec5f27a178295ec098df3db047b4cd0424ca97456123f253fa6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-25T01:45:25.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:24 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:24.787+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:25.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:24 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:24.787+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:25.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:24 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:24.787+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:25.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:24 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:24.787+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:25.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:24 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:24.787+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:25.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:24 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:24.787+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:25.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:24 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:24.787+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:56.886337+0000 front 2024-04-25T01:44:56.886406+0000 (oldest deadline 2024-04-25T01:45:21.586131+0000) 2024-04-25T01:45:25.327 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:24 smithi028 podman[102926]: 2024-04-25 01:45:24.859952309 +0000 UTC m=+5.786062961 container cleanup 7223c0482776ec5f27a178295ec098df3db047b4cd0424ca97456123f253fa6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-25T01:45:25.327 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:24 smithi028 bash[102926]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9 2024-04-25T01:45:25.327 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:25 smithi028 podman[102946]: 2024-04-25 01:45:25.044682356 +0000 UTC m=+0.523143647 container remove 7223c0482776ec5f27a178295ec098df3db047b4cd0424ca97456123f253fa6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2) 2024-04-25T01:45:25.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:25 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:25.296+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:25.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:25 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:25.296+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:25.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:25 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:25.296+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:25.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:25 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:25.296+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:25.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:25 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:25.296+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:25.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:25 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:25.296+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:59.228913+0000 front 2024-04-25T01:44:59.228881+0000 (oldest deadline 2024-04-25T01:45:21.528858+0000) 2024-04-25T01:45:25.765 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:25 smithi028 podman[103037]: 2024-04-25 01:45:25.390076305 +0000 UTC m=+0.018539963 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:45:25.765 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:25 smithi028 podman[103037]: 2024-04-25 01:45:25.495515596 +0000 UTC m=+0.123979250 container create 23dc02a33c411b92f0e12298097fd998d8f72cfa34f2c367de2b38f1d925e0a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T01:45:25.765 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:25 smithi028 podman[103037]: 2024-04-25 01:45:25.697748534 +0000 UTC m=+0.326212190 container init 23dc02a33c411b92f0e12298097fd998d8f72cfa34f2c367de2b38f1d925e0a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0) 2024-04-25T01:45:25.765 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:25 smithi028 podman[103037]: 2024-04-25 01:45:25.701177702 +0000 UTC m=+0.329641360 container start 23dc02a33c411b92f0e12298097fd998d8f72cfa34f2c367de2b38f1d925e0a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T01:45:25.765 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:25 smithi028 podman[103037]: 2024-04-25 01:45:25.772596671 +0000 UTC m=+0.401060339 container attach 23dc02a33c411b92f0e12298097fd998d8f72cfa34f2c367de2b38f1d925e0a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-25T01:45:25.999 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:25 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:25.749+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:25.999 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:25 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:25.749+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:25.999 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:25 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:25.749+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:25.999 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:25 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:25.749+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:25.999 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:25 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:25.749+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:25.999 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:25 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:25.749+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:25.999 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:25 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:25.749+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:56.886337+0000 front 2024-04-25T01:44:56.886406+0000 (oldest deadline 2024-04-25T01:45:21.586131+0000) 2024-04-25T01:45:26.028 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 25 01:45:26 smithi028 podman[103037]: 2024-04-25 01:45:26.002563315 +0000 UTC m=+0.631026971 container died 23dc02a33c411b92f0e12298097fd998d8f72cfa34f2c367de2b38f1d925e0a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-25T01:45:26.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.9.service' 2024-04-25T01:45:26.623 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:45:26.623 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-25T01:45:26.624 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-25T01:45:26.624 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.10 2024-04-25T01:45:26.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:26 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:26.285+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:26.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:26 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:26.285+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:26.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:26 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:26.285+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:26.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:26 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:26.285+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:26.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:26 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:26.285+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:26.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:26 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:26.285+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:59.228913+0000 front 2024-04-25T01:44:59.228881+0000 (oldest deadline 2024-04-25T01:45:21.528858+0000) 2024-04-25T01:45:26.956 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:26 smithi032 systemd[1]: Stopping Ceph osd.10 for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:45:26.957 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:26 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:26.798+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:26.957 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:26 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:26.798+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:26.957 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:26 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:26.798+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:26.957 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:26 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:26.798+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:26.957 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:26 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:26.798+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:26.957 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:26 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:26.798+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:26.957 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:26 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:26.798+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:56.886337+0000 front 2024-04-25T01:44:56.886406+0000 (oldest deadline 2024-04-25T01:45:21.586131+0000) 2024-04-25T01:45:26.957 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:26 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:26.963+0000 7fa982ae3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:45:26.957 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:26 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:26.963+0000 7fa982ae3700 -1 osd.10 106 *** Got signal Terminated *** 2024-04-25T01:45:26.957 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:26 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:26.963+0000 7fa982ae3700 -1 osd.10 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:45:27.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:27 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:27.243+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:27.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:27 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:27.243+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:27.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:27 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:27.243+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:27.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:27 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:27.243+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:27.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:27 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:27.243+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:27.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:27 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:27.243+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:59.228913+0000 front 2024-04-25T01:44:59.228881+0000 (oldest deadline 2024-04-25T01:45:21.528858+0000) 2024-04-25T01:45:28.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:27 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:27.771+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:28.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:27 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:27.771+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:28.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:27 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:27.771+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:28.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:27 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:27.771+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:28.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:27 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:27.771+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:28.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:27 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:27.771+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:28.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:27 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:27.771+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:56.886337+0000 front 2024-04-25T01:44:56.886406+0000 (oldest deadline 2024-04-25T01:45:21.586131+0000) 2024-04-25T01:45:28.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:28 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:28.281+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:28.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:28 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:28.281+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:28.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:28 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:28.281+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:28.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:28 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:28.281+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:28.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:28 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:28.281+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:28.725 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:28 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:28.281+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:59.228913+0000 front 2024-04-25T01:44:59.228881+0000 (oldest deadline 2024-04-25T01:45:21.528858+0000) 2024-04-25T01:45:29.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:28 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:28.770+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:29.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:28 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:28.770+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:29.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:28 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:28.770+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:29.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:28 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:28.770+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:29.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:28 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:28.770+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:29.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:28 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:28.770+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:29.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:28 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:28.770+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:56.886337+0000 front 2024-04-25T01:44:56.886406+0000 (oldest deadline 2024-04-25T01:45:21.586131+0000) 2024-04-25T01:45:29.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:29 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:29.243+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:29.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:29 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:29.243+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:29.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:29 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:29.243+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:29.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:29 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:29.243+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:29.725 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:29 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:29.243+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:29.725 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:29 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:29.243+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:59.228913+0000 front 2024-04-25T01:44:59.228881+0000 (oldest deadline 2024-04-25T01:45:21.528858+0000) 2024-04-25T01:45:30.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:29 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:29.810+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:30.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:29 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:29.810+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:30.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:29 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:29.810+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:30.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:29 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:29.810+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:30.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:29 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:29.810+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:30.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:29 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:29.810+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:30.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:29 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:29.810+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:56.886337+0000 front 2024-04-25T01:44:56.886406+0000 (oldest deadline 2024-04-25T01:45:21.586131+0000) 2024-04-25T01:45:30.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:30 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:30.217+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:30.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:30 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:30.217+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:30.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:30 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:30.217+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:30.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:30 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:30.217+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:30.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:30 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:30.217+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:30.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:30 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:30.217+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:59.228913+0000 front 2024-04-25T01:44:59.228881+0000 (oldest deadline 2024-04-25T01:45:21.528858+0000) 2024-04-25T01:45:31.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:30 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:30.825+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:31.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:30 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:30.825+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:31.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:30 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:30.825+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:31.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:30 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:30.825+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:31.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:30 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:30.825+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:31.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:30 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:30.825+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:31.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:30 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:30.825+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:56.886337+0000 front 2024-04-25T01:44:56.886406+0000 (oldest deadline 2024-04-25T01:45:21.586131+0000) 2024-04-25T01:45:31.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:31 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:31.175+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:31.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:31 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:31.175+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:31.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:31 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:31.175+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:31.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:31 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:31.175+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:31.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:31 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:31.175+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:31.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:31 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:31.175+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:59.228913+0000 front 2024-04-25T01:44:59.228881+0000 (oldest deadline 2024-04-25T01:45:21.528858+0000) 2024-04-25T01:45:31.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:31 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:31.175+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6820 osd.7 since back 2024-04-25T01:45:07.429939+0000 front 2024-04-25T01:45:07.429859+0000 (oldest deadline 2024-04-25T01:45:30.929615+0000) 2024-04-25T01:45:32.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:31 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:31.869+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:10.779341+0000 front 2024-04-25T01:44:10.779396+0000 (oldest deadline 2024-04-25T01:44:34.879445+0000) 2024-04-25T01:45:32.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:31 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:31.869+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6804 osd.1 since back 2024-04-25T01:44:18.980419+0000 front 2024-04-25T01:44:18.980446+0000 (oldest deadline 2024-04-25T01:44:43.680567+0000) 2024-04-25T01:45:32.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:31 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:31.869+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:23.681118+0000 front 2024-04-25T01:44:23.681197+0000 (oldest deadline 2024-04-25T01:44:49.581129+0000) 2024-04-25T01:45:32.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:31 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:31.869+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:35.982697+0000 front 2024-04-25T01:44:35.982617+0000 (oldest deadline 2024-04-25T01:44:57.082626+0000) 2024-04-25T01:45:32.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:31 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:31.869+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.183387+0000 front 2024-04-25T01:44:41.183441+0000 (oldest deadline 2024-04-25T01:45:07.083605+0000) 2024-04-25T01:45:32.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:31 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:31.869+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:49.384963+0000 front 2024-04-25T01:44:49.385062+0000 (oldest deadline 2024-04-25T01:45:12.884644+0000) 2024-04-25T01:45:32.250 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:31 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10[57655]: 2024-04-25T01:45:31.869+0000 7fa97b0cf700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:56.886337+0000 front 2024-04-25T01:44:56.886406+0000 (oldest deadline 2024-04-25T01:45:21.586131+0000) 2024-04-25T01:45:32.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:32 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:32.156+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:32.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:32 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:32.156+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:32.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:32 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:32.156+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:32.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:32 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:32.156+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:32.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:32 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:32.156+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:32.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:32 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:32.156+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:59.228913+0000 front 2024-04-25T01:44:59.228881+0000 (oldest deadline 2024-04-25T01:45:21.528858+0000) 2024-04-25T01:45:32.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:32 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:32.156+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6820 osd.7 since back 2024-04-25T01:45:07.429939+0000 front 2024-04-25T01:45:07.429859+0000 (oldest deadline 2024-04-25T01:45:30.929615+0000) 2024-04-25T01:45:32.622 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:32 smithi032 podman[69483]: 2024-04-25 01:45:32.368725015 +0000 UTC m=+5.534195785 container died baf79060daa2e782216786cab0b6fdbc1207f96dd7ad581f351b8f8ce0848323 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:45:32.989 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:32 smithi032 podman[69483]: 2024-04-25 01:45:32.721441713 +0000 UTC m=+5.886912487 container cleanup baf79060daa2e782216786cab0b6fdbc1207f96dd7ad581f351b8f8ce0848323 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-25T01:45:32.989 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:32 smithi032 bash[69483]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10 2024-04-25T01:45:33.249 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:32 smithi032 podman[69500]: 2024-04-25 01:45:32.997241834 +0000 UTC m=+0.621599096 container remove baf79060daa2e782216786cab0b6fdbc1207f96dd7ad581f351b8f8ce0848323 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-25T01:45:33.396 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:33 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:33.129+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:33.396 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:33 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:33.129+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:33.396 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:33 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:33.129+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:33.397 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:33 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:33.129+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:33.397 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:33 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:33.129+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:33.397 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:33 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:33.129+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:59.228913+0000 front 2024-04-25T01:44:59.228881+0000 (oldest deadline 2024-04-25T01:45:21.528858+0000) 2024-04-25T01:45:33.397 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:33 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:33.129+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6820 osd.7 since back 2024-04-25T01:45:07.429939+0000 front 2024-04-25T01:45:07.429859+0000 (oldest deadline 2024-04-25T01:45:30.929615+0000) 2024-04-25T01:45:33.601 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:33 smithi032 podman[69594]: 2024-04-25 01:45:33.257799252 +0000 UTC m=+0.018506006 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:45:33.602 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:33 smithi032 podman[69594]: 2024-04-25 01:45:33.389971328 +0000 UTC m=+0.150678106 container create 6d7177dcbf1b2a8ac2e4c4c68c0320027d21a7c4b0eb9e7b1ee9bbe6aa48cde2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T01:45:33.602 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:33 smithi032 podman[69594]: 2024-04-25 01:45:33.609365219 +0000 UTC m=+0.370071951 container init 6d7177dcbf1b2a8ac2e4c4c68c0320027d21a7c4b0eb9e7b1ee9bbe6aa48cde2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:45:33.900 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:33 smithi032 podman[69594]: 2024-04-25 01:45:33.612713347 +0000 UTC m=+0.373420082 container start 6d7177dcbf1b2a8ac2e4c4c68c0320027d21a7c4b0eb9e7b1ee9bbe6aa48cde2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-25T01:45:33.900 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:33 smithi032 podman[69594]: 2024-04-25 01:45:33.692691106 +0000 UTC m=+0.453397839 container attach 6d7177dcbf1b2a8ac2e4c4c68c0320027d21a7c4b0eb9e7b1ee9bbe6aa48cde2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-25T01:45:34.188 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:33 smithi032 podman[69594]: 2024-04-25 01:45:33.908193715 +0000 UTC m=+0.668900438 container died 6d7177dcbf1b2a8ac2e4c4c68c0320027d21a7c4b0eb9e7b1ee9bbe6aa48cde2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-25T01:45:34.442 INFO:journalctl@ceph.osd.10.smithi032.stdout:Apr 25 01:45:34 smithi032 podman[69620]: 2024-04-25 01:45:34.43782078 +0000 UTC m=+0.519465886 container remove 6d7177dcbf1b2a8ac2e4c4c68c0320027d21a7c4b0eb9e7b1ee9bbe6aa48cde2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:45:34.448 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.10.service' 2024-04-25T01:45:34.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:34 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:34.123+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:34.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:34 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:34.123+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:34.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:34 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:34.123+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:34.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:34 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:34.123+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:34.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:34 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:34.123+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:34.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:34 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:34.123+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:59.228913+0000 front 2024-04-25T01:44:59.228881+0000 (oldest deadline 2024-04-25T01:45:21.528858+0000) 2024-04-25T01:45:34.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:34 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:34.123+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6820 osd.7 since back 2024-04-25T01:45:07.429939+0000 front 2024-04-25T01:45:07.429859+0000 (oldest deadline 2024-04-25T01:45:30.929615+0000) 2024-04-25T01:45:34.488 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:45:34.488 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-25T01:45:34.488 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-25T01:45:34.489 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.11 2024-04-25T01:45:34.783 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:34 smithi130 systemd[1]: Stopping Ceph osd.11 for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:45:34.783 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:34 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:34.789+0000 7f5fb382e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:45:34.783 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:34 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:34.789+0000 7f5fb382e700 -1 osd.11 106 *** Got signal Terminated *** 2024-04-25T01:45:34.783 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:34 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:34.789+0000 7f5fb382e700 -1 osd.11 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T01:45:35.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:35 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:35.164+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:35.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:35 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:35.164+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:35.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:35 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:35.164+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:35.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:35 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:35.164+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:35.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:35 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:35.164+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:35.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:35 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:35.164+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:59.228913+0000 front 2024-04-25T01:44:59.228881+0000 (oldest deadline 2024-04-25T01:45:21.528858+0000) 2024-04-25T01:45:35.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:35 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:35.164+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6820 osd.7 since back 2024-04-25T01:45:07.429939+0000 front 2024-04-25T01:45:07.429859+0000 (oldest deadline 2024-04-25T01:45:30.929615+0000) 2024-04-25T01:45:36.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:36 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:36.119+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:36.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:36 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:36.119+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:36.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:36 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:36.119+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:36.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:36 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:36.120+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:36.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:36 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:36.120+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:36.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:36 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:36.120+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:59.228913+0000 front 2024-04-25T01:44:59.228881+0000 (oldest deadline 2024-04-25T01:45:21.528858+0000) 2024-04-25T01:45:36.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:36 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:36.120+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6820 osd.7 since back 2024-04-25T01:45:07.429939+0000 front 2024-04-25T01:45:07.429859+0000 (oldest deadline 2024-04-25T01:45:30.929615+0000) 2024-04-25T01:45:37.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:37 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:37.111+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:37.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:37 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:37.111+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:37.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:37 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:37.111+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:37.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:37 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:37.111+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:37.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:37 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:37.111+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:37.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:37 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:37.111+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:59.228913+0000 front 2024-04-25T01:44:59.228881+0000 (oldest deadline 2024-04-25T01:45:21.528858+0000) 2024-04-25T01:45:37.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:37 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:37.111+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6820 osd.7 since back 2024-04-25T01:45:07.429939+0000 front 2024-04-25T01:45:07.429859+0000 (oldest deadline 2024-04-25T01:45:30.929615+0000) 2024-04-25T01:45:38.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:38 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:38.082+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:38.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:38 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:38.082+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:38.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:38 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:38.082+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:38.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:38 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:38.082+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:38.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:38 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:38.082+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:38.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:38 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:38.082+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:59.228913+0000 front 2024-04-25T01:44:59.228881+0000 (oldest deadline 2024-04-25T01:45:21.528858+0000) 2024-04-25T01:45:38.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:38 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:38.082+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6820 osd.7 since back 2024-04-25T01:45:07.429939+0000 front 2024-04-25T01:45:07.429859+0000 (oldest deadline 2024-04-25T01:45:30.929615+0000) 2024-04-25T01:45:39.474 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:39 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:39.108+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-25T01:44:09.120847+0000 front 2024-04-25T01:44:09.120923+0000 (oldest deadline 2024-04-25T01:44:33.220876+0000) 2024-04-25T01:45:39.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:39 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:39.108+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6804 osd.2 since back 2024-04-25T01:44:27.823523+0000 front 2024-04-25T01:44:27.823424+0000 (oldest deadline 2024-04-25T01:44:50.024097+0000) 2024-04-25T01:45:39.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:39 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:39.108+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-25T01:44:34.125183+0000 front 2024-04-25T01:44:34.125200+0000 (oldest deadline 2024-04-25T01:45:01.125796+0000) 2024-04-25T01:45:39.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:39 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:39.108+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6812 osd.4 since back 2024-04-25T01:44:41.126487+0000 front 2024-04-25T01:44:41.126340+0000 (oldest deadline 2024-04-25T01:45:05.826385+0000) 2024-04-25T01:45:39.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:39 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:39.108+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6812 osd.5 since back 2024-04-25T01:44:52.227741+0000 front 2024-04-25T01:44:52.227780+0000 (oldest deadline 2024-04-25T01:45:15.727843+0000) 2024-04-25T01:45:39.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:39 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:39.108+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-04-25T01:44:59.228913+0000 front 2024-04-25T01:44:59.228881+0000 (oldest deadline 2024-04-25T01:45:21.528858+0000) 2024-04-25T01:45:39.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:39 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:39.108+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.32:6820 osd.7 since back 2024-04-25T01:45:07.429939+0000 front 2024-04-25T01:45:07.429859+0000 (oldest deadline 2024-04-25T01:45:30.929615+0000) 2024-04-25T01:45:39.475 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:39 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11[57731]: 2024-04-25T01:45:39.108+0000 7f5fabe1a700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.130:6820 osd.8 since back 2024-04-25T01:45:13.830434+0000 front 2024-04-25T01:45:13.830386+0000 (oldest deadline 2024-04-25T01:45:38.530377+0000) 2024-04-25T01:45:40.459 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:40 smithi130 podman[69390]: 2024-04-25 01:45:40.194479827 +0000 UTC m=+5.515885304 container died 13318442a760da89d806c7370f1d0f85655e2181511c10caa50e4fef5d8d01d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-25T01:45:40.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:40 smithi130 podman[69390]: 2024-04-25 01:45:40.547946346 +0000 UTC m=+5.869351822 container cleanup 13318442a760da89d806c7370f1d0f85655e2181511c10caa50e4fef5d8d01d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9) 2024-04-25T01:45:40.725 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:40 smithi130 bash[69390]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11 2024-04-25T01:45:41.109 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:40 smithi130 podman[69409]: 2024-04-25 01:45:40.824367577 +0000 UTC m=+0.627091161 container remove 13318442a760da89d806c7370f1d0f85655e2181511c10caa50e4fef5d8d01d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-25T01:45:41.422 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:41 smithi130 podman[69499]: 2024-04-25 01:45:41.019166516 +0000 UTC m=+0.017442400 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:45:41.422 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:41 smithi130 podman[69499]: 2024-04-25 01:45:41.133442605 +0000 UTC m=+0.131718528 container create 0279224e7d50d1256ce645d0c8c0adb164320bf762e48a62f8ff7a0ae98ae311 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-25T01:45:41.422 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:41 smithi130 podman[69499]: 2024-04-25 01:45:41.345985058 +0000 UTC m=+0.344260939 container init 0279224e7d50d1256ce645d0c8c0adb164320bf762e48a62f8ff7a0ae98ae311 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9) 2024-04-25T01:45:41.422 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:41 smithi130 podman[69499]: 2024-04-25 01:45:41.350109573 +0000 UTC m=+0.348385462 container start 0279224e7d50d1256ce645d0c8c0adb164320bf762e48a62f8ff7a0ae98ae311 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T01:45:41.422 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:41 smithi130 podman[69499]: 2024-04-25 01:45:41.429274312 +0000 UTC m=+0.427550206 container attach 0279224e7d50d1256ce645d0c8c0adb164320bf762e48a62f8ff7a0ae98ae311 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-25T01:45:41.724 INFO:journalctl@ceph.osd.11.smithi130.stdout:Apr 25 01:45:41 smithi130 podman[69499]: 2024-04-25 01:45:41.642099721 +0000 UTC m=+0.640375605 container died 0279224e7d50d1256ce645d0c8c0adb164320bf762e48a62f8ff7a0ae98ae311 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-25T01:45:42.225 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@osd.11.service' 2024-04-25T01:45:42.263 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:45:42.263 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-25T01:45:42.263 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-25T01:45:42.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.a 2024-04-25T01:45:42.577 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:45:42 smithi028 systemd[1]: Stopping Ceph mds.a for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:45:43.077 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:45:42 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-a[69257]: 2024-04-25T01:45:42.594+0000 7f6082fef700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:45:43.077 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:45:42 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-a[69257]: 2024-04-25T01:45:42.594+0000 7f6082fef700 -1 mds.a *** got signal Terminated *** 2024-04-25T01:45:46.865 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:45:46 smithi028 podman[103157]: 2024-04-25 01:45:46.601305552 +0000 UTC m=+4.130504507 container died 739a81cfd2eb67a95fa51dd3de5787ef8439fcde5bb1117181135bc865997d10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-04-25T01:45:47.130 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:45:46 smithi028 podman[103157]: 2024-04-25 01:45:46.944734431 +0000 UTC m=+4.473933374 container cleanup 739a81cfd2eb67a95fa51dd3de5787ef8439fcde5bb1117181135bc865997d10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-25T01:45:47.130 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:45:46 smithi028 bash[103157]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-a 2024-04-25T01:45:47.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.a.service' 2024-04-25T01:45:47.393 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:45:47 smithi028 podman[103171]: 2024-04-25 01:45:47.137595425 +0000 UTC m=+0.526274221 container remove 739a81cfd2eb67a95fa51dd3de5787ef8439fcde5bb1117181135bc865997d10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-d2c45d9, GIT_CLEAN=True) 2024-04-25T01:45:47.393 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:45:47 smithi028 systemd[1]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.a.service: Deactivated successfully. 2024-04-25T01:45:47.393 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:45:47 smithi028 systemd[1]: Stopped Ceph mds.a for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:45:47.393 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 25 01:45:47 smithi028 systemd[1]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.a.service: Consumed 2.253s CPU time. 2024-04-25T01:45:47.406 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:45:47.407 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-25T01:45:47.407 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-25T01:45:47.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.d 2024-04-25T01:45:47.724 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 25 01:45:47 smithi028 systemd[1]: Stopping Ceph mds.d for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:45:48.077 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 25 01:45:47 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-d[68301]: 2024-04-25T01:45:47.730+0000 7f8bbc585700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:45:48.077 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 25 01:45:47 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-d[68301]: 2024-04-25T01:45:47.730+0000 7f8bbc585700 -1 mds.d *** got signal Terminated *** 2024-04-25T01:45:49.077 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 25 01:45:48 smithi028 podman[103278]: 2024-04-25 01:45:48.79853734 +0000 UTC m=+1.151348287 container died 7e9e51fb5540b0dbb5bc6eee5131b8584c69fa42c089632cfb00d34d3680571a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True) 2024-04-25T01:45:49.359 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 25 01:45:49 smithi028 podman[103278]: 2024-04-25 01:45:49.106807438 +0000 UTC m=+1.459618427 container cleanup 7e9e51fb5540b0dbb5bc6eee5131b8584c69fa42c089632cfb00d34d3680571a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-d, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-25T01:45:49.359 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 25 01:45:49 smithi028 bash[103278]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-d 2024-04-25T01:45:49.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.d.service' 2024-04-25T01:45:49.569 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:45:49.569 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-25T01:45:49.569 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-25T01:45:49.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.g 2024-04-25T01:45:49.946 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 25 01:45:49 smithi028 systemd[1]: Stopping Ceph mds.g for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:45:50.327 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 25 01:45:49 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-g[68909]: 2024-04-25T01:45:49.952+0000 7f925d81f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:45:50.327 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 25 01:45:49 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-g[68909]: 2024-04-25T01:45:49.952+0000 7f925d81f700 -1 mds.g *** got signal Terminated *** 2024-04-25T01:45:51.479 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 25 01:45:51 smithi028 podman[103401]: 2024-04-25 01:45:51.235711013 +0000 UTC m=+1.439538852 container died d92727996a48949f3f3d170c487efcbe2b81ed316195e203a43ae91e58078a91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-25T01:45:51.754 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 25 01:45:51 smithi028 podman[103401]: 2024-04-25 01:45:51.577712651 +0000 UTC m=+1.781540531 container cleanup d92727996a48949f3f3d170c487efcbe2b81ed316195e203a43ae91e58078a91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9) 2024-04-25T01:45:51.755 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 25 01:45:51 smithi028 bash[103401]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-g 2024-04-25T01:45:51.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.g.service' 2024-04-25T01:45:52.013 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:45:52.013 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-25T01:45:52.013 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-25T01:45:52.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.j 2024-04-25T01:45:52.327 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 25 01:45:52 smithi028 systemd[1]: Stopping Ceph mds.j for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:45:52.827 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 25 01:45:52 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-j[67915]: 2024-04-25T01:45:52.363+0000 7fc1eb380700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:45:52.828 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 25 01:45:52 smithi028 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-j[67915]: 2024-04-25T01:45:52.363+0000 7fc1eb380700 -1 mds.j *** got signal Terminated *** 2024-04-25T01:45:54.174 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 25 01:45:53 smithi028 podman[103522]: 2024-04-25 01:45:53.914824616 +0000 UTC m=+1.664885495 container died a369550f8aceb873d2f0ac2eee67efdc98b7943506460e0259f02cdfc4ac8ba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-j, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:45:54.460 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 25 01:45:54 smithi028 podman[103522]: 2024-04-25 01:45:54.323106209 +0000 UTC m=+2.073167088 container cleanup a369550f8aceb873d2f0ac2eee67efdc98b7943506460e0259f02cdfc4ac8ba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424) 2024-04-25T01:45:54.460 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 25 01:45:54 smithi028 bash[103522]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-j 2024-04-25T01:45:54.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.j.service' 2024-04-25T01:45:54.723 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 25 01:45:54 smithi028 podman[103537]: 2024-04-25 01:45:54.57397217 +0000 UTC m=+0.649387473 container remove a369550f8aceb873d2f0ac2eee67efdc98b7943506460e0259f02cdfc4ac8ba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-j, RELEASE=squid-d2c45d9, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:45:54.724 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 25 01:45:54 smithi028 systemd[1]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.j.service: Deactivated successfully. 2024-04-25T01:45:54.724 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 25 01:45:54 smithi028 systemd[1]: Stopped Ceph mds.j for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:45:54.724 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 25 01:45:54 smithi028 systemd[1]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.j.service: Consumed 1.086s CPU time. 2024-04-25T01:45:54.731 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:45:54.732 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-25T01:45:54.732 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-25T01:45:54.732 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.b 2024-04-25T01:45:55.040 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:45:54 smithi032 systemd[1]: Stopping Ceph mds.b for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:45:55.312 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:45:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-b[63160]: 2024-04-25T01:45:55.046+0000 7f59dbfaf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:45:55.312 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:45:55 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-b[63160]: 2024-04-25T01:45:55.046+0000 7f59dbfaf700 -1 mds.b *** got signal Terminated *** 2024-04-25T01:45:59.308 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:45:59 smithi032 podman[69700]: 2024-04-25 01:45:59.056221085 +0000 UTC m=+4.119915319 container died f3b2a1d13e2fead614dc5b171c74383f7c1ef20d1f1f0b438a3f7d78a2d184e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T01:45:59.616 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:45:59 smithi032 podman[69700]: 2024-04-25 01:45:59.405885354 +0000 UTC m=+4.469579589 container cleanup f3b2a1d13e2fead614dc5b171c74383f7c1ef20d1f1f0b438a3f7d78a2d184e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-25T01:45:59.616 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:45:59 smithi032 bash[69700]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-b 2024-04-25T01:45:59.872 INFO:journalctl@ceph.mds.b.smithi032.stdout:Apr 25 01:45:59 smithi032 podman[69726]: 2024-04-25 01:45:59.623433746 +0000 UTC m=+0.557205637 container remove f3b2a1d13e2fead614dc5b171c74383f7c1ef20d1f1f0b438a3f7d78a2d184e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-b, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:45:59.876 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.b.service' 2024-04-25T01:45:59.911 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:45:59.911 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-25T01:45:59.911 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-25T01:45:59.912 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.e 2024-04-25T01:46:00.236 INFO:journalctl@ceph.mds.e.smithi032.stdout:Apr 25 01:45:59 smithi032 systemd[1]: Stopping Ceph mds.e for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:46:00.500 INFO:journalctl@ceph.mds.e.smithi032.stdout:Apr 25 01:46:00 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-e[62540]: 2024-04-25T01:46:00.242+0000 7fab62dc3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:46:00.500 INFO:journalctl@ceph.mds.e.smithi032.stdout:Apr 25 01:46:00 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-e[62540]: 2024-04-25T01:46:00.242+0000 7fab62dc3700 -1 mds.e *** got signal Terminated *** 2024-04-25T01:46:03.500 INFO:journalctl@ceph.mds.e.smithi032.stdout:Apr 25 01:46:03 smithi032 podman[69834]: 2024-04-25 01:46:03.232062591 +0000 UTC m=+3.089634378 container died f70b176c5ce2c5bc2a63b08b81bea240ae0b10e4dc8fb0f838cc1a170092909e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-25T01:46:03.953 INFO:journalctl@ceph.mds.e.smithi032.stdout:Apr 25 01:46:03 smithi032 podman[69834]: 2024-04-25 01:46:03.701427865 +0000 UTC m=+3.558999643 container cleanup f70b176c5ce2c5bc2a63b08b81bea240ae0b10e4dc8fb0f838cc1a170092909e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-25T01:46:03.953 INFO:journalctl@ceph.mds.e.smithi032.stdout:Apr 25 01:46:03 smithi032 bash[69834]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-e 2024-04-25T01:46:04.082 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.e.service' 2024-04-25T01:46:04.115 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:46:04.116 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-25T01:46:04.116 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-25T01:46:04.116 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.h 2024-04-25T01:46:04.250 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:46:04 smithi032 systemd[1]: Stopping Ceph mds.h for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:46:04.750 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:46:04 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-h[62852]: 2024-04-25T01:46:04.453+0000 7fe419610700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:46:04.750 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:46:04 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-h[62852]: 2024-04-25T01:46:04.453+0000 7fe419610700 -1 mds.h *** got signal Terminated *** 2024-04-25T01:46:07.500 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:46:07 smithi032 podman[69956]: 2024-04-25 01:46:07.243492908 +0000 UTC m=+2.896664356 container died 4a375432e3bf098820e6c6480f9b03c0c450324344e507bb4ed2b799c96e0c78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-04-25T01:46:07.971 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:46:07 smithi032 podman[69956]: 2024-04-25 01:46:07.638334462 +0000 UTC m=+3.291505909 container cleanup 4a375432e3bf098820e6c6480f9b03c0c450324344e507bb4ed2b799c96e0c78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0) 2024-04-25T01:46:07.971 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:46:07 smithi032 bash[69956]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-h 2024-04-25T01:46:07.971 INFO:journalctl@ceph.mds.h.smithi032.stdout:Apr 25 01:46:07 smithi032 podman[69970]: 2024-04-25 01:46:07.872700728 +0000 UTC m=+0.622278163 container remove 4a375432e3bf098820e6c6480f9b03c0c450324344e507bb4ed2b799c96e0c78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-h, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:46:07.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.h.service' 2024-04-25T01:46:08.026 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:46:08.026 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-25T01:46:08.026 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-25T01:46:08.026 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.k 2024-04-25T01:46:08.250 INFO:journalctl@ceph.mds.k.smithi032.stdout:Apr 25 01:46:08 smithi032 systemd[1]: Stopping Ceph mds.k for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:46:08.750 INFO:journalctl@ceph.mds.k.smithi032.stdout:Apr 25 01:46:08 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-k[62215]: 2024-04-25T01:46:08.365+0000 7f45e06ea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:46:08.750 INFO:journalctl@ceph.mds.k.smithi032.stdout:Apr 25 01:46:08 smithi032 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-k[62215]: 2024-04-25T01:46:08.365+0000 7f45e06ea700 -1 mds.k *** got signal Terminated *** 2024-04-25T01:46:11.494 INFO:journalctl@ceph.mds.k.smithi032.stdout:Apr 25 01:46:11 smithi032 podman[70075]: 2024-04-25 01:46:11.231315904 +0000 UTC m=+2.981320018 container died 68adf427738ec563503f3ee9a580eb66e97e00032bf55239d5fc1b42da854137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-25T01:46:11.750 INFO:journalctl@ceph.mds.k.smithi032.stdout:Apr 25 01:46:11 smithi032 podman[70075]: 2024-04-25 01:46:11.583232619 +0000 UTC m=+3.333236730 container cleanup 68adf427738ec563503f3ee9a580eb66e97e00032bf55239d5fc1b42da854137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=squid-d2c45d9, ceph=True) 2024-04-25T01:46:11.750 INFO:journalctl@ceph.mds.k.smithi032.stdout:Apr 25 01:46:11 smithi032 bash[70075]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-k 2024-04-25T01:46:12.057 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.k.service' 2024-04-25T01:46:12.089 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:46:12.089 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-25T01:46:12.089 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-25T01:46:12.089 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.c 2024-04-25T01:46:12.401 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 25 01:46:12 smithi130 systemd[1]: Stopping Ceph mds.c for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:46:12.725 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 25 01:46:12 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-c[62762]: 2024-04-25T01:46:12.407+0000 7ff3e1f0d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:46:12.725 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 25 01:46:12 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-c[62762]: 2024-04-25T01:46:12.407+0000 7ff3e1f0d700 -1 mds.c *** got signal Terminated *** 2024-04-25T01:46:15.492 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 25 01:46:15 smithi130 podman[69603]: 2024-04-25 01:46:15.241698932 +0000 UTC m=+2.946271139 container died 9e1ca5001ff5d59dd0eef29d2e8a6c8431c4a1af183aaf472c06aec7b1157049 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:46:15.868 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 25 01:46:15 smithi130 podman[69603]: 2024-04-25 01:46:15.598817073 +0000 UTC m=+3.303389288 container cleanup 9e1ca5001ff5d59dd0eef29d2e8a6c8431c4a1af183aaf472c06aec7b1157049 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:46:15.868 INFO:journalctl@ceph.mds.c.smithi130.stdout:Apr 25 01:46:15 smithi130 bash[69603]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-c 2024-04-25T01:46:16.028 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.c.service' 2024-04-25T01:46:16.062 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:46:16.062 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-25T01:46:16.062 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-25T01:46:16.062 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.f 2024-04-25T01:46:16.396 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:46:16 smithi130 systemd[1]: Stopping Ceph mds.f for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:46:16.725 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:46:16 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-f[62451]: 2024-04-25T01:46:16.402+0000 7f71dbf9f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:46:16.725 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:46:16 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-f[62451]: 2024-04-25T01:46:16.402+0000 7f71dbf9f700 -1 mds.f *** got signal Terminated *** 2024-04-25T01:46:19.597 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:46:19 smithi130 podman[69736]: 2024-04-25 01:46:19.244481536 +0000 UTC m=+2.949169340 container died e9caaadfedf84525b5430331ea28aa186e8ce3d30e182da31e2eac3378fee612 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2) 2024-04-25T01:46:19.889 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:46:19 smithi130 podman[69736]: 2024-04-25 01:46:19.669818068 +0000 UTC m=+3.374505850 container cleanup e9caaadfedf84525b5430331ea28aa186e8ce3d30e182da31e2eac3378fee612 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-f, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-25T01:46:19.889 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:46:19 smithi130 bash[69736]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-f 2024-04-25T01:46:20.124 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.f.service' 2024-04-25T01:46:20.145 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:46:19 smithi130 podman[69750]: 2024-04-25 01:46:19.896232984 +0000 UTC m=+0.643409959 container remove e9caaadfedf84525b5430331ea28aa186e8ce3d30e182da31e2eac3378fee612 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-25T01:46:20.145 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:46:20 smithi130 systemd[1]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.f.service: Deactivated successfully. 2024-04-25T01:46:20.145 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:46:20 smithi130 systemd[1]: Stopped Ceph mds.f for 0e88f60a-02a2-11ef-bc93-c7b262605968. 2024-04-25T01:46:20.145 INFO:journalctl@ceph.mds.f.smithi130.stdout:Apr 25 01:46:20 smithi130 systemd[1]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.f.service: Consumed 1.954s CPU time. 2024-04-25T01:46:20.156 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:46:20.156 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-25T01:46:20.157 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-25T01:46:20.157 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.i 2024-04-25T01:46:20.475 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 25 01:46:20 smithi130 systemd[1]: Stopping Ceph mds.i for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:46:20.975 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 25 01:46:20 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-i[63068]: 2024-04-25T01:46:20.482+0000 7f53ad9cb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:46:20.976 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 25 01:46:20 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-i[63068]: 2024-04-25T01:46:20.482+0000 7f53ad9cb700 -1 mds.i *** got signal Terminated *** 2024-04-25T01:46:23.543 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 25 01:46:23 smithi130 podman[69856]: 2024-04-25 01:46:23.239655307 +0000 UTC m=+2.861491951 container died af61a3080c0dd2c8f8c6181b5c29ce650cb2a204d2c26e3af90fc322c364b2d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-i, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:46:23.861 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 25 01:46:23 smithi130 podman[69856]: 2024-04-25 01:46:23.632788618 +0000 UTC m=+3.254625243 container cleanup af61a3080c0dd2c8f8c6181b5c29ce650cb2a204d2c26e3af90fc322c364b2d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-04-25T01:46:23.861 INFO:journalctl@ceph.mds.i.smithi130.stdout:Apr 25 01:46:23 smithi130 bash[69856]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-i 2024-04-25T01:46:23.958 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.i.service' 2024-04-25T01:46:23.990 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:46:23.990 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-25T01:46:23.990 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-25T01:46:23.990 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.l 2024-04-25T01:46:24.225 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 25 01:46:24 smithi130 systemd[1]: Stopping Ceph mds.l for 0e88f60a-02a2-11ef-bc93-c7b262605968... 2024-04-25T01:46:24.659 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 25 01:46:24 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-l[62135]: 2024-04-25T01:46:24.319+0000 7fa52b806700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:46:24.659 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 25 01:46:24 smithi130 ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-l[62135]: 2024-04-25T01:46:24.319+0000 7fa52b806700 -1 mds.l *** got signal Terminated *** 2024-04-25T01:46:27.538 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 25 01:46:27 smithi130 podman[69981]: 2024-04-25 01:46:27.248742197 +0000 UTC m=+3.027432531 container died fac764c82e12f5bb0ab6f4fc228da4e249d3375c6712607d92fb1b72b1e1201e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-l, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:46:27.847 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 25 01:46:27 smithi130 podman[69981]: 2024-04-25 01:46:27.636221794 +0000 UTC m=+3.414912136 container cleanup fac764c82e12f5bb0ab6f4fc228da4e249d3375c6712607d92fb1b72b1e1201e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-l, RELEASE=squid-d2c45d9, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:46:27.847 INFO:journalctl@ceph.mds.l.smithi130.stdout:Apr 25 01:46:27 smithi130 bash[69981]: ceph-0e88f60a-02a2-11ef-bc93-c7b262605968-mds-l 2024-04-25T01:46:27.985 DEBUG:teuthology.orchestra.run.smithi130:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0e88f60a-02a2-11ef-bc93-c7b262605968@mds.l.service' 2024-04-25T01:46:28.016 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T01:46:28.016 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-25T01:46:28.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T01:46:28.215 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:46:30.715 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T01:46:30.918 INFO:teuthology.orchestra.run.smithi032.stdout:Deleting cluster with fsid: 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:46:33.478 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T01:46:33.666 INFO:teuthology.orchestra.run.smithi130.stdout:Deleting cluster with fsid: 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:46:35.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:46:36.029 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:46:36.062 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:46:36.088 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-25T01:46:36.090 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672318/remote/smithi028/crash 2024-04-25T01:46:36.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/crash -- . 2024-04-25T01:46:36.133 INFO:teuthology.orchestra.run.smithi028.stderr:tar: /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T01:46:36.133 INFO:teuthology.orchestra.run.smithi028.stderr:tar: Error is not recoverable: exiting now 2024-04-25T01:46:36.135 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672318/remote/smithi032/crash 2024-04-25T01:46:36.135 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/crash -- . 2024-04-25T01:46:36.171 INFO:teuthology.orchestra.run.smithi032.stderr:tar: /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T01:46:36.172 INFO:teuthology.orchestra.run.smithi032.stderr:tar: Error is not recoverable: exiting now 2024-04-25T01:46:36.173 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672318/remote/smithi130/crash 2024-04-25T01:46:36.174 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/crash -- . 2024-04-25T01:46:36.205 INFO:teuthology.orchestra.run.smithi130.stderr:tar: /var/lib/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T01:46:36.206 INFO:teuthology.orchestra.run.smithi130.stderr:tar: Error is not recoverable: exiting now 2024-04-25T01:46:36.207 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-25T01:46:36.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-25T01:46:36.249 INFO:tasks.cephadm:Compressing logs... 2024-04-25T01:46:36.249 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T01:46:36.292 DEBUG:teuthology.orchestra.run.smithi032:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T01:46:36.295 DEBUG:teuthology.orchestra.run.smithi130:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T01:46:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-25T01:46:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T01:46:36.318 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.log 2024-04-25T01:46:36.327 INFO:teuthology.orchestra.run.smithi130.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T01:46:36.327 INFO:teuthology.orchestra.run.smithi130.stderr:: No such file or directory 2024-04-25T01:46:36.328 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.log 2024-04-25T01:46:36.328 INFO:teuthology.orchestra.run.smithi130.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-25T01:46:36.328 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-25T01:46:36.328 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T01:46:36.328 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-25T01:46:36.328 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.f.log: /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-25T01:46:36.328 INFO:teuthology.orchestra.run.smithi130.stderr: 92.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T01:46:36.328 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-25T01:46:36.328 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-25T01:46:36.328 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-25T01:46:36.328 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mgr.z.log: 87.4% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T01:46:36.329 INFO:teuthology.orchestra.run.smithi130.stderr: 90.2% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T01:46:36.331 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose --/var/log/ceph/cephadm.log: /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T01:46:36.331 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-25T01:46:36.331 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-25T01:46:36.331 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-25T01:46:36.331 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-25T01:46:36.331 INFO:teuthology.orchestra.run.smithi028.stderr:gzip/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.3.log: -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-25T01:46:36.331 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-25T01:46:36.331 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.log.gz/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mgr.x.log: 2024-04-25T01:46:36.331 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-25T01:46:36.332 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mon.a.log: 90.0% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T01:46:36.332 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T01:46:36.332 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.6.log: 94.6% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-25T01:46:36.332 INFO:teuthology.orchestra.run.smithi028.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T01:46:36.335 INFO:teuthology.orchestra.run.smithi032.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T01:46:36.335 INFO:teuthology.orchestra.run.smithi032.stderr:: No such file or directory 2024-04-25T01:46:36.335 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.log 2024-04-25T01:46:36.335 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T01:46:36.336 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-25T01:46:36.336 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-25T01:46:36.336 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-25T01:46:36.336 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.7.log: /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-25T01:46:36.336 INFO:teuthology.orchestra.run.smithi032.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T01:46:36.336 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-25T01:46:36.336 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-25T01:46:36.336 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip 87.4% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T01:46:36.336 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-25T01:46:36.336 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 90.1% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T01:46:36.336 INFO:teuthology.orchestra.run.smithi032.stderr: --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T01:46:36.336 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.10.log: /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-25T01:46:36.336 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.9% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T01:46:36.336 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T01:46:36.336 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.e.log: 91.9% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-25T01:46:36.341 INFO:teuthology.orchestra.run.smithi130.stderr: 92.0% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-25T01:46:36.341 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T01:46:36.342 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-25T01:46:36.342 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-25T01:46:36.342 INFO:teuthology.orchestra.run.smithi130.stderr: 83.8% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T01:46:36.342 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-25T01:46:36.342 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T01:46:36.342 INFO:teuthology.orchestra.run.smithi028.stderr: 84.1% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T01:46:36.342 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-25T01:46:36.343 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-25T01:46:36.345 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T01:46:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-25T01:46:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.d.log: 91.1% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T01:46:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-25T01:46:36.354 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-25T01:46:36.355 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T01:46:36.357 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-25T01:46:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.9.log: 94.5% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-25T01:46:36.376 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-25T01:46:36.380 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.l.log: 92.6% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-25T01:46:36.384 INFO:teuthology.orchestra.run.smithi130.stderr: 91.0% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T01:46:36.386 INFO:teuthology.orchestra.run.smithi028.stderr: 94.5% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-25T01:46:36.387 INFO:teuthology.orchestra.run.smithi130.stderr: 92.5% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-25T01:46:36.401 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.k.log: 92.5% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-25T01:46:36.413 INFO:teuthology.orchestra.run.smithi028.stderr: 94.4% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-25T01:46:36.413 INFO:teuthology.orchestra.run.smithi130.stderr: 92.5% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-25T01:46:36.424 INFO:teuthology.orchestra.run.smithi032.stderr: 94.4% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-25T01:46:36.430 INFO:teuthology.orchestra.run.smithi130.stderr: 94.5% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-25T01:46:36.642 INFO:teuthology.orchestra.run.smithi032.stderr: 96.3% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-25T01:46:36.665 INFO:teuthology.orchestra.run.smithi130.stderr: 92.1% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-25T01:46:36.700 INFO:teuthology.orchestra.run.smithi028.stderr: 90.2% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-25T01:46:36.781 INFO:teuthology.orchestra.run.smithi032.stderr: 92.1% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-25T01:46:37.655 INFO:teuthology.orchestra.run.smithi028.stderr: 90.8% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-25T01:46:38.294 INFO:teuthology.orchestra.run.smithi032.stderr: 94.6% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-25T01:46:41.014 INFO:teuthology.orchestra.run.smithi130.stderr: 93.1% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-25T01:46:41.200 INFO:teuthology.orchestra.run.smithi130.stderr: 92.8% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-25T01:46:42.142 INFO:teuthology.orchestra.run.smithi032.stderr: 92.9% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-25T01:46:42.255 INFO:teuthology.orchestra.run.smithi032.stderr: 93.0% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-25T01:46:42.591 INFO:teuthology.orchestra.run.smithi028.stderr: 93.1% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-25T01:46:42.947 INFO:teuthology.orchestra.run.smithi028.stderr: 92.8% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-25T01:46:43.149 INFO:teuthology.orchestra.run.smithi032.stderr: 92.8% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-25T01:46:43.226 INFO:teuthology.orchestra.run.smithi028.stderr: 93.0% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-25T01:46:43.340 INFO:teuthology.orchestra.run.smithi130.stderr: 92.7% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-25T01:46:44.118 INFO:teuthology.orchestra.run.smithi032.stderr: 92.8% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-25T01:46:44.119 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-04-25T01:46:44.120 INFO:teuthology.orchestra.run.smithi032.stderr:real 0m7.809s 2024-04-25T01:46:44.120 INFO:teuthology.orchestra.run.smithi032.stderr:user 0m28.317s 2024-04-25T01:46:44.120 INFO:teuthology.orchestra.run.smithi032.stderr:sys 0m0.867s 2024-04-25T01:46:44.549 INFO:teuthology.orchestra.run.smithi028.stderr: 92.7% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-25T01:46:44.552 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-25T01:46:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:real 0m8.244s 2024-04-25T01:46:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:user 0m28.892s 2024-04-25T01:46:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m0.965s 2024-04-25T01:46:45.370 INFO:teuthology.orchestra.run.smithi130.stderr: 92.7% -- replaced with /var/log/ceph/0e88f60a-02a2-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-25T01:46:45.371 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-04-25T01:46:45.371 INFO:teuthology.orchestra.run.smithi130.stderr:real 0m9.062s 2024-04-25T01:46:45.371 INFO:teuthology.orchestra.run.smithi130.stderr:user 0m25.306s 2024-04-25T01:46:45.371 INFO:teuthology.orchestra.run.smithi130.stderr:sys 0m0.895s 2024-04-25T01:46:45.372 INFO:tasks.cephadm:Archiving logs... 2024-04-25T01:46:45.372 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672318/remote/smithi028/log 2024-04-25T01:46:45.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T01:46:50.346 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672318/remote/smithi032/log 2024-04-25T01:46:50.347 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T01:46:54.579 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672318/remote/smithi130/log 2024-04-25T01:46:54.580 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T01:46:58.920 INFO:tasks.cephadm:Removing cluster... 2024-04-25T01:46:58.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 --force 2024-04-25T01:46:59.120 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:46:59.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 --force 2024-04-25T01:46:59.617 INFO:teuthology.orchestra.run.smithi032.stdout:Deleting cluster with fsid: 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:46:59.921 DEBUG:teuthology.orchestra.run.smithi130:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0e88f60a-02a2-11ef-bc93-c7b262605968 --force 2024-04-25T01:47:00.114 INFO:teuthology.orchestra.run.smithi130.stdout:Deleting cluster with fsid: 0e88f60a-02a2-11ef-bc93-c7b262605968 2024-04-25T01:47:00.404 INFO:tasks.cephadm:Removing cephadm ... 2024-04-25T01:47:00.404 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T01:47:00.420 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T01:47:00.437 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T01:47:00.451 INFO:tasks.cephadm:Teardown complete 2024-04-25T01:47:00.452 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T01:47:00.464 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T01:47:00.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T01:47:00.466 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T01:47:00.480 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T01:47:00.588 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T01:47:00.599 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T01:47:00.599 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:47:00.601 DEBUG:teuthology.orchestra.run.smithi032:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:47:00.604 DEBUG:teuthology.orchestra.run.smithi130:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:47:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:47:00.616 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:47:00.617 INFO:teuthology.orchestra.run.smithi130.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:47:00.618 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:47:00.618 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-04-25T01:47:00.618 INFO:teuthology.orchestra.run.smithi028.stdout:^- hv01.front.sepia.ceph.com 2 6 377 60 +1554us[+1554us] +/- 585ms 2024-04-25T01:47:00.618 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv02.front.sepia.ceph.com 3 7 377 64 -172us[ -159us] +/- 67ms 2024-04-25T01:47:00.618 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 +291us[ +291us] +/- 74ms 2024-04-25T01:47:00.618 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:47:00.620 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:47:00.620 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-04-25T01:47:00.620 INFO:teuthology.orchestra.run.smithi032.stdout:^- hv01.front.sepia.ceph.com 2 6 377 0 +1543us[+1543us] +/- 586ms 2024-04-25T01:47:00.620 INFO:teuthology.orchestra.run.smithi032.stdout:^* hv02.front.sepia.ceph.com 3 6 377 63 -237us[ -239us] +/- 67ms 2024-04-25T01:47:00.620 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 +263us[ +262us] +/- 74ms 2024-04-25T01:47:00.620 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:47:00.622 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:47:00.622 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2024-04-25T01:47:00.622 INFO:teuthology.orchestra.run.smithi130.stdout:^- hv01.front.sepia.ceph.com 2 6 377 63 +1520us[+1521us] +/- 585ms 2024-04-25T01:47:00.622 INFO:teuthology.orchestra.run.smithi130.stdout:^* hv02.front.sepia.ceph.com 3 6 377 62 -238us[ -236us] +/- 67ms 2024-04-25T01:47:00.622 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 +268us[ +268us] +/- 74ms 2024-04-25T01:47:00.622 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:47:00.623 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T01:47:00.633 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T01:47:00.634 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T01:47:00.644 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T01:47:00.739 DEBUG:teuthology.orchestra.run.smithi032:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T01:47:00.806 DEBUG:teuthology.orchestra.run.smithi130:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T01:47:00.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:47:00.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:47:00.899 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:47:00.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:47:00.927 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:47:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:47:00.954 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T01:47:00.966 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T01:47:00.975 INFO:teuthology.task.internal:Duration was 1921.939635 seconds 2024-04-25T01:47:00.976 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T01:47:00.987 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T01:47:00.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T01:47:00.990 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T01:47:00.992 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T01:47:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:47:01.030 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:47:01.042 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:47:01.370 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T01:47:01.370 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T01:47:01.370 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T01:47:01.431 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi032.front.sepia.ceph.com 2024-04-25T01:47:01.431 DEBUG:teuthology.orchestra.run.smithi032:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T01:47:01.455 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi130.front.sepia.ceph.com 2024-04-25T01:47:01.456 DEBUG:teuthology.orchestra.run.smithi130:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T01:47:01.480 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T01:47:01.480 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T01:47:01.483 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T01:47:01.498 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T01:47:01.545 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T01:47:01.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T01:47:01.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T01:47:01.565 DEBUG:teuthology.orchestra.run.smithi130:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T01:47:01.965 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T01:47:01.977 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T01:47:01.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T01:47:02.002 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T01:47:02.027 DEBUG:teuthology.orchestra.run.smithi130:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T01:47:02.051 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T01:47:02.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:47:02.066 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:47:02.070 DEBUG:teuthology.orchestra.run.smithi130:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:47:02.086 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-04-25T01:47:02.090 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = core 2024-04-25T01:47:02.114 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = core 2024-04-25T01:47:02.130 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:47:02.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:47:02.154 DEBUG:teuthology.orchestra.run.smithi032:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:47:02.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:47:02.168 DEBUG:teuthology.orchestra.run.smithi130:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T01:47:02.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:47:02.186 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T01:47:02.204 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T01:47:02.205 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672318/remote/smithi028 2024-04-25T01:47:02.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T01:47:02.250 DEBUG:teuthology.misc:Transferring archived files from smithi032:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672318/remote/smithi032 2024-04-25T01:47:02.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T01:47:02.287 DEBUG:teuthology.misc:Transferring archived files from smithi130:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672318/remote/smithi130 2024-04-25T01:47:02.287 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T01:47:02.322 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T01:47:02.322 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T01:47:02.325 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T01:47:02.327 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T01:47:02.371 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T01:47:02.387 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T01:47:02.387 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T01:47:02.398 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T01:47:02.398 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T01:47:02.401 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T01:47:02.403 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T01:47:02.413 INFO:teuthology.orchestra.run.smithi028.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 01:47 /home/ubuntu/cephtest 2024-04-25T01:47:02.415 INFO:teuthology.orchestra.run.smithi032.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 01:47 /home/ubuntu/cephtest 2024-04-25T01:47:02.425 INFO:teuthology.orchestra.run.smithi130.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 01:47 /home/ubuntu/cephtest 2024-04-25T01:47:02.427 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T01:47:02.439 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T01:47:02.467 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} duration: 1921.9396350383759 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-25T01:47:02.468 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T01:47:02.550 INFO:teuthology.run:pass